fstests: use syncfs rather than sync

fstests: use syncfs rather than sync

sync(1) is a system wide sync and is implemented by iterating all
the superblocks in the system. In most cases, fstests require just
the filesystem under test to be synced - we require syncfs(2)
semantics but what we use is sync(2) semantics.

The result of this is that when running many concurrent fstests at
the same time, we can have *hundreds* of concurrent sync operations
in progress (thanks fsstress!) and this causes excessive
interference with other tests that are running on other filesystems.

For example, some tests try to specifically control extent layout
via specific write and fsync patterns. All these global syncs
perturb them and cause them to spuriously fail.

A random snapshot of running concurrent tests shows just how many
tests are explicitly blocked in sync(1):

check-parallel───check───077───077─┬─cut
      │                                    ├─du
      │                                    └─tail
      ├─check-parallel───check───311───xfs_scrub───{xfs_scrub}
      ├─check-parallel───check───531───128*[t_open_tmpfiles]
      ├─check-parallel───check───227
      ├─check-parallel───check───388
      ├─check-parallel───check───070───fsstress───fsstress───{fsstress+
      ├─check-parallel───check───232───fsstress───7*[fsstress───{fsstr+
      ├─check-parallel───check───648───sleep
      ├─check-parallel───check───409───sync
      ├─check-parallel───check───683───sync
      ├─check-parallel───check───013─┬─013───sleep
      │                              └─fsstress───2*[fsstress───{fsstr+
      ├─check-parallel───check───684───sync
      ├─check-parallel───check───673───sync
      ├─check-parallel───check───118───dd
      ├─check-parallel───check───467───open_by_handle
      ├─check-parallel───check─┬─622
      │                        └─check
      ├─check-parallel───check───685───sync
      ├─check-parallel───check───049───fsstress───fsstress───{fsstress+
      ├─check-parallel───check───599
      ├─check-parallel───check───426───open_by_handle
      ├─check-parallel───check───057───umount
      ├─check-parallel───check───390───fsstress─┬─18*[fsstress───{fsst+
      │                                         └─fsstress
      ├─check-parallel───check───158───fsstress───fsstress───{fsstress+
      ├─check-parallel───check───017
      ├─check-parallel───check───032───fsstress───fsstress
      ├─check-parallel───check───076
      ├─check-parallel───check───477───open_by_handle
      ├─check-parallel───check───170───2*[170───170]
      ├─check-parallel───check───112
      ├─check-parallel───check───686───sync
      ├─4*[check-parallel───check───check───xfs_scrub───{xfs_scrub}]
      ├─check-parallel───check───387───xfs_io───{xfs_io}
      ├─check-parallel───check───615───615
      ├─check-parallel───check─┬─051
      │                        └─check───xfs_repair
      ├─check-parallel───check───049
      ├─check-parallel───check───247
      ├─check-parallel───check───674───sync
      ├─check-parallel───check───040
      ├─check-parallel───check───560───fsstress───fsstress───{fsstress+
      ├─check-parallel───check───030─┬─030─┬─030───xfs_repair
      │                              │     └─030───perl
      │                              ├─sed
      │                              └─uniq
      ├─check-parallel───check───055───055
      ├─2*[check-parallel───check───check]
      ├─check-parallel───check───042
      ├─check-parallel───check───204
      ├─check-parallel───check───271─┬─271───sed
      │                              └─md5sum
      ├─check-parallel───check───091─┬─fsx
      │                              └─tee
      ├─check-parallel───check───063───sleep
      ├─check-parallel───check───026
      ├─check-parallel───check───459───lvm
      ├─check-parallel───check───495
      ├─check-parallel───check───141───fsstress───4*[fsstress]
      ├─check-parallel───check───011─┬─fsstress─┬─fsstress───{fsstress+
      │                              │          └─fsstress
      │                              └─sleep
      ├─check-parallel───check───328───sync
      ├─check-parallel───check───507───507
      ├─check-parallel───check
      ├─check-parallel───check───687───sync
      ├─check-parallel───check───109───mkfs.xfs
      ├─check-parallel───check───324
      ├─check-parallel───check───114───aio-dio-eof-rac
      └─check-parallel───check───503───xfs_scrub───2*[{xfs_scrub}]

There are ~10 sync(1) calls blocked and at least half of the 50-odd
fsstress processes currently running are also going to be stuck in
sync(2) calls.

They are stuck because the superblock iteration has to wait for
mount, unmount, freeze, thaw and any other operation that locks a
superblock exclusively. When running dozens of tests concurrently,
there can be tens of superblocks that are locked exclusively for
every second for significant lengths of time.

Hence the use of sync has impact on both performance and test
behaviour and we need to minimise the amount of sync(1) and
sync(2) usage as much as possible.

Introduce _test_sync() and _scratch_sync() so we can implement
a syncfs mechanism with a fallback to sync(1) if it is not supported
without dirtying all the test code unnecessarily. Then convert
fsstress to use syncfs(2) in preference to sync(2).

This commit changes all the generic and XFS tests to use the new
sync functions, other filesystem specific tests will eventually
need to be converted to avoid similar problems.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Zorro lang <zlang@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>