Darrick J. Wong [Thu, 21 Nov 2024 00:27:30 +0000 (16:27 -0800)]
xfs: skip tests if formatting small filesystem fails
There are a few tests that try to exercise XFS functionality with an
unusually small (< 500MB) filesystem. Formatting can fail if the test
configuration also specifies a very large realtime device because mkfs
hits ENOSPC when allocating the realtime metadata. The test proceeds
anyway (which causes an immediate mount failure) so we might as well
skip these.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Darrick J. Wong [Thu, 21 Nov 2024 00:27:30 +0000 (16:27 -0800)]
xfs/3{43,32}: adapt tests for rt extent size greater than 1
Both of these tests for the realtime volume can fail when the rt extent
size is larger than a single block.
332 is a read-write functionality test that encodes md5sum in the
output, so we need to skip it if $blksz isn't congruent with the extent
size, because the fcollapse call will fail.
343 is a test of the rmap btree, so the fix here is simpler -- make
$blksz the file allocation unit, and get rid of the md5sum in the
golden output.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Darrick J. Wong [Thu, 21 Nov 2024 00:27:30 +0000 (16:27 -0800)]
xfs/341: update test for rtgroup-based rmap
Now that we're sharding the realtime volume into multiple allocation
groups, update this test to reflect the new reality. The realtime rmap
btree record and key sizes have shrunk, and we can't guarantee that a
quick file write actually hits the same rt group as the one we fuzzed,
so eliminate the file write test since we're really only curious if
xfs_repair will fix the problem.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Darrick J. Wong [Thu, 21 Nov 2024 00:27:30 +0000 (16:27 -0800)]
xfs: fix various problems with fsmap detecting the data device
Various tests of realtime rmap functionality assumed that the data
device could be picked out from the GETFSMAP output by looking for
static fs metadata. This is no longer true, since rtgroups filesystems
write a superblock header at the start of the rt device, so update these
tests.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Darrick J. Wong [Thu, 21 Nov 2024 00:27:29 +0000 (16:27 -0800)]
fuzz: for fuzzing the rtrmapbt, find the path to the rt rmap btree file
The fs population code creates a realtime rmap btree in /some/ realtime
group with at least two levels. This rmapbt file isn't necessarily the
one for group 0, so we need to find it programmatically.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
This is due to the fact that SCRATCH_DEV is temporarily reassigned to
the regular file. That path is passed straight through _scratch_mount
to _xfs_prepare_for_eio_shutdown, but that helper _fails because the
"dev" argument isn't actually a path to a block device.
Fix this by porting it to the new common/metadump code that we merged
last year.
Cc: <fstests@vger.kernel.org> # v2024.12.08 Fixes: 1a49022fab9b4d ("fstests: always use fail-at-unmount semantics for XFS") Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Darrick J. Wong [Thu, 21 Nov 2024 00:27:29 +0000 (16:27 -0800)]
xfs: fix tests that try to access the realtime rmap inode
The realtime rmap tests were added to fstests a long time ago. Since
they were added, we decided to create a metadata file directory
structure instead of adding more fields to the superblock. Therefore,
fix all the tests that try to access these paths.
While we're at it, fix xfs/409 to run the *online* scrub program like
it's supposed to. xfs/408 is the fuzzer for xfs_repair testing.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Darrick J. Wong [Tue, 17 Dec 2024 18:32:00 +0000 (10:32 -0800)]
common: test statfs reporting with project quota
Create a test to check that statfs on a directory tree with a project
quota will report the quota limit and available blocks; and that the
available blocks reported doesn't exceed that of the whole filesystem.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Darrick J. Wong [Thu, 21 Nov 2024 00:27:28 +0000 (16:27 -0800)]
common: enable testing of realtime quota when supported
If the kernel advertises realtime quota support, test it.
However, this has a plot twist -- because rt quota only works if the xfs
is formatted with rtgroups, we have to mount a filesystem to see if
rtquota is actually supported. Since it's time consuming to format and
mount the scratch filesystem, we'll assume that the test and scratch
fses have the same support.
This will cause problems if one sets SCRATCH_RTDEV but not TEST_RTDEV.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Darrick J. Wong [Wed, 27 Nov 2024 21:01:15 +0000 (13:01 -0800)]
xfs: fix quota detection in fuzz tests
With metadir, quota options persist until they are changed by mount
options. Therefore, we can set the quota flags in MKFS_OPTIONS and
needn't supply them in MOUNT_OPTIONS. Unfortunately, this means that we
cannot grep the MOUNT_OPTIONS anymore; we must mount the fs and run
src/feature to determine if quotas are enabled.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Darrick J. Wong [Thu, 21 Nov 2024 00:27:26 +0000 (16:27 -0800)]
common/xfs: capture realtime devices during metadump/mdrestore
If xfs_metadump supports the -r switch to capture the contents of
realtime devices and there is a realtime device, add the option to the
command line to enable preservation.
Similarly, if the dump file could restore to an external scratch rtdev,
pass the -r option to mdrestore so that we can restore rtdev contents.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Darrick J. Wong [Thu, 21 Nov 2024 00:27:26 +0000 (16:27 -0800)]
xfs/449: update test to know about xfs_db -R
The realtime groups feature added a -R flag to xfs_db so that users can
pass in the realtime device. Since we've now modified the
_scratch_xfs_db to use this facility, we can update the test to do exact
comparisons of the xfs_db info command against the mkfs output.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Darrick J. Wong [Thu, 21 Nov 2024 00:27:25 +0000 (16:27 -0800)]
xfs/185: update for rtgroups
This old test is a bit too fixated on exact rt allocator behavior. With
rtgroups enabled, we can end up with one large contiguous region that's
split into multiple bmbt mappings to avoid crossing rtgroup boundaries.
The realtime superblock throws another twist into the mix because the
first rtx will always be in use, which can shift the start of the
physical space mappings by up to 1 rtx.
Also fix a bug where we'd try to fallocate the total number of rtx,
whereas we should be asking for the number of free rtx to avoid ENOSPC
errors.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Darrick J. Wong [Thu, 21 Nov 2024 00:27:25 +0000 (16:27 -0800)]
punch-alternating: detect xfs realtime files with large allocation units
For files on the XFS realtime volume, it's possible that the file
allocation unit (aka the minimum size we have to punch to deallocate
file blocks) could be greater than a single fs block. This utility
assumed that it's always possible to punch a single fs block, but for
these types of files, all that does is zeroes the page cache. While
that's what most *user applications* want, fstests uses punching to
fragment file mapping metadata and/or fragment free space, so adapt this
test for that purpose by detecting realtime files.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Darrick J. Wong [Thu, 21 Nov 2024 00:27:24 +0000 (16:27 -0800)]
common/populate: use metadump v2 format by default for fs metadata snapshots
When we're snapshotting filesystem metadata after creating a populated
filesystem, force the creation of metadump v2 files by default to
exercise the new format, since xfs_metadump continues to use the v1
format unless told otherwise.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Darrick J. Wong [Thu, 21 Nov 2024 00:27:24 +0000 (16:27 -0800)]
common/ext4: reformat external logs during mdrestore operations
The e2image file format doesn't support the capture of external log
devices, which means that mdrestore ought to reformat the external log
to get the restored filesystem to work again. The common/populate code
could already do this, so push it to the common ext4 helper.
While we're at it, fix the uncareful usage of SCRATCH_LOGDEV in the
populate code.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Darrick J. Wong [Thu, 21 Nov 2024 00:27:24 +0000 (16:27 -0800)]
common/populate: refactor caching of metadumps to a helper
Hoist out of _scratch_populate_cached all the code that we use to save a
metadump of the populated filesystem. We're going to make this more
involved for XFS in the next few patches so that we can take advantage
of the new support for external devices in metadump/mdrestore.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Darrick J. Wong [Fri, 22 Nov 2024 21:18:18 +0000 (13:18 -0800)]
xfs/122: disable this test for any codebase that knows about metadir
All of the ondisk structure size checks from this test were copied to
the build time checks in xfs_ondisk.h. This means that the kernel and
xfsprogs build processes check the structure sizes, which means that
fstests no longer needs to do that.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Darrick J. Wong [Thu, 21 Nov 2024 00:27:22 +0000 (16:27 -0800)]
xfs/509: adjust inumbers accounting for metadata directories
The INUMBERS ioctl exports data from the inode btree directly -- the
number of inodes it reports is taken from ir_freemask and includes all
the files in the metadata directory tree. BULKSTAT, on the other hand,
only reports non-metadata files. When metadir is enabled, this will
(eventually) cause a discrepancy in the inode counts that is large
enough to exceed the tolerances, thereby causing a test failure.
Correct this by counting the files in the metadata directory and
subtracting that from the INUMBERS totals.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Darrick J. Wong [Thu, 21 Nov 2024 00:27:22 +0000 (16:27 -0800)]
xfs/{050,144,153,299,330}: update quota reports to handle metadir trees
Prior to the new metadir feature in XFS, the rtbitmap and rtsummary
files were included in icount, though their bcount contribution is zero
due to rt and quota not being supported together. With the new metadir
feature in XFS, no files in the metadata directory tree are counted in
quota.
Hence we must adjust the icount of any quota report down by two to avoid
breaking golden outputs.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Darrick J. Wong [Thu, 21 Nov 2024 00:27:21 +0000 (16:27 -0800)]
common/repair: patch up repair sb inode value complaints
Now that we've refactored xfs_repair to be more consistent in how it
reports unexpected superblock inode pointer values, we have to fix up
the fstests repair filters to emulate the old golden output.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Darrick J. Wong [Thu, 21 Nov 2024 00:27:21 +0000 (16:27 -0800)]
xfs/{030,033,178}: forcibly disable metadata directory trees
The golden output for thests tests encode the xfs_repair output when we
fuzz various parts of the filesystem. With metadata directory trees
enabled, however, the golden output changes dramatically to reflect
reconstruction of the metadata directory tree.
To avoid regressions, add a helper to force metadata directories off via
MKFS_OPTIONS.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Darrick J. Wong [Thu, 21 Nov 2024 00:27:21 +0000 (16:27 -0800)]
various: fix finding metadata inode numbers when metadir is enabled
There are a number of tests that use xfs_db to examine the contents of
metadata inodes to check correct functioning. The logic is scattered
everywhere and won't work with metadata directory trees, so make a
shared helper to find these inodes.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Darrick J. Wong [Thu, 16 Jan 2025 20:18:05 +0000 (12:18 -0800)]
xfs/349: reclassify this test as not dangerous
This test creates a filesystem with all known types of metadata. It
doesn't fuzz anything, nor does it actually repair anything. Hence we
can put it in the auto group and drop the dangerous tags.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Darrick J. Wong [Thu, 23 Jan 2025 17:00:02 +0000 (09:00 -0800)]
xfs/28[56]: add to the auto group
Enable /some/ testing of online fsck for everybody by adding these two
stress tests to the autogroup. At this time I don't have any plans to
do the same for the other scrub stress tests.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Darrick J. Wong [Thu, 16 Jan 2025 19:44:03 +0000 (11:44 -0800)]
misc: remove the dangerous_scrub group
Now that online fsck has been in the upstream kernel for 8 months, I
think it's stabilized enough that the scrub functionality tests don't
need to hide behind the "dangerous" label anymore. Move them all to the
scrub group and delete the dangerous_scrub group.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Darrick J. Wong [Thu, 16 Jan 2025 20:02:09 +0000 (12:02 -0800)]
misc: fix misclassification of xfs_repair fuzz tests
All the tests in the "fuzzers_repair" group actually test xfs_repair,
not scrub. Therefore, we should remove them all from 'dangerous_scrub'
and put them in the 'repair' group.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Darrick J. Wong [Thu, 16 Jan 2025 19:47:17 +0000 (11:47 -0800)]
misc: rename the dangerous_norepair group to fuzzers_norepair
I've been running the norepair fuzzers (aka the verifier checks) for
most of the time that online fsck has been in development, and I haven't
seen any of the VMs crash in a couple of years. I think it's time that
these tests stopped hiding behind the "dangerous" label.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Darrick J. Wong [Thu, 16 Jan 2025 19:44:03 +0000 (11:44 -0800)]
misc: rename the dangerous_bothrepair group to fuzzers_bothrepair
Now that online fsck has been in the upstream kernel for 8 months, I
think it's stabilized enough that the scrub + repair functionality tests
don't need to hide behind the "dangerous" label anymore.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Darrick J. Wong [Thu, 16 Jan 2025 19:55:36 +0000 (11:55 -0800)]
misc: rename the dangerous_online_repair group to fuzzers_online_repair
Now that online fsck has been in the upstream kernel for 8 months, I
think it's stabilized enough that the scrub functionality tests don't
need to hide behind the "dangerous" label anymore. It's time for more
widespread testing.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Darrick J. Wong [Thu, 16 Jan 2025 18:07:44 +0000 (10:07 -0800)]
misc: drop the dangerous label from xfs_scrub fsstress tests
Now that online fsck has been in the upstream kernel for 8 months, I
think it's stabilized enough that we don't need to hide the stress tests
behind the "dangerous" label anymore.
Also rename fsstress_repair to fsstress_online_repair to be consistent
with the online_repair group.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Darrick J. Wong [Fri, 22 Nov 2024 16:28:04 +0000 (08:28 -0800)]
logwrites: only use BLKDISCARD if we know discard zeroes data
Building off the checks established in the previous patch, only enable
the use of BLKDISCARD if we know that the logwrites device guarantees
that reads after a discard return zeroes.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Darrick J. Wong [Thu, 21 Nov 2024 17:59:39 +0000 (09:59 -0800)]
logwrites: warn if we don't think read after discard returns zeroes
The logwrites replay program expects that it can issue a DISCARD against
the block device passed to _log_writes_init and that will cause all
subsequent reads to return zeroes. This is required for correct log
recovery on filesystems such as XFS that skip recovering buffers if
newer ones are found on disk.
Unfortunately, there's no way to discover if a device's discard
implementation actually guarantees zeroes. There used to be a sysfs
knob keyed to an allowlist, but it is now hardwired to return 0. So
either we need a magic device that does discard-and-zero, or we need to
do the zeroing ourselves. The logwrites program does its own zeroing if
there is no discard support, and some tests do their own zeroing.
The only devices we know to work reliably are the software defined ones
that are provided by the kernel itself -- which means dm-thinp. Warn if
we have a device that supports discard that isn't thinp and the test
fails.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Darrick J. Wong [Tue, 14 Jan 2025 17:47:07 +0000 (09:47 -0800)]
build: initialize stack variables to zero by default
Newer versions of gcc and clang can include the ability to zero stack
variables by default. Let's enable it so that we (a) reduce the risk of
writing stack contents to disk somewhere and (b) try to reduce
unpredictable program behavior based on random stack contents. The
kernel added this 6 years ago, so I think it's mature enough for
fstests.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Darrick J. Wong [Wed, 22 Jan 2025 05:06:32 +0000 (21:06 -0800)]
misc: don't put nr_cpus into the fsstress -n argument
fsstress -n is the number of fs operations per process, not the total
number of operations. There's no need to factor nr_cpus into the -n
argument because that causes excess runtime as core count increases.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Darrick J. Wong [Tue, 14 Jan 2025 02:10:26 +0000 (18:10 -0800)]
fsx: fix leaked log file pointer
Fix a resource leaks in fsx, where we fail to close the fsx logfile,
because the C library could have some buffered contents that aren't
flushed when the program terminates. glibc seems to do this for us, but
I wouldn't be so sure about the others.
Fixes: 3f742550dfed84 ("fsx: add support for recording operations to a file") Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Darrick J. Wong [Fri, 10 Jan 2025 06:06:52 +0000 (22:06 -0800)]
fix _require_scratch_duperemove ordering
Zorro complained that generic/559 stopped working, and I noticed that
the duperemove invocation in the _require_scratch_duperemove function
would fail with:
Error 2: No such file or directory while getting path to file /opt/file1. Skipping.
Error 2: No such file or directory while getting path to file /opt/file2. Skipping.
No dedupe candidates found.
Gathering file list...
The cause of this is the incorrect placement of _require_scratch_dedupe
after a _scratch_mount. _require_scratch_dedupe formats, mounts, tests,
and unmounts the scratch filesystem, which means that it should not come
between a _scratch_mount call and code that uses $SCRATCH_MNT.
Cc: <fstests@vger.kernel.org> # v2024.12.22 Fixes: 3b9f5fc7d7d853 ("common: call _require_scratch_dedupe from _require_scratch_duperemove") Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Darrick J. Wong [Wed, 8 Jan 2025 22:19:28 +0000 (14:19 -0800)]
fuzzy: port fsx and fsstress loop to use --duration
Quite a while ago, I added --duration= arguments to fsx and fsstress,
and apparently I forgot to update the scrub stress loops to use them.
Replace the usage of timeout(1) for the remount_period versions of the
loop to clean up that code; and convert the non-remount loop so that
they don't run over time.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Darrick J. Wong [Wed, 8 Jan 2025 22:18:15 +0000 (14:18 -0800)]
fuzzy: always stop the scrub fsstress loop on error
Always abort the non-remount scrub fsstress loop in
__stress_scrub_fsstress_loop if fsstress returns a nonzero exit code.
Cc: <fstests@vger.kernel.org> # v2023.01.15 Fixes: 20df87599f66d0 ("fuzzy: make scrub stress loop control more robust") Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Darrick J. Wong [Tue, 7 Jan 2025 20:28:20 +0000 (12:28 -0800)]
fuzzy: don't use readarray for xfsfind output
Some of the scrub stress tests (e.g. xfs/796) walk the directory tree to
find filepaths to scrub, and load the entire list of paths into a bash
array. On a large filesystem or a long-running test this is hugely
wasteful of memory because we use each path exactly once.
Fix __stress_one_scrub_loop to avoid this by reading lines directly from
the output of the xfsfind utility. However, we play some games with fd
77 so that the processes in the loop body will use the same stdin as the
test and /not/ the piped stdout of xfsfind.
To avoid read(1) becoming confused by newlines in the file paths, adapt
xfsfind to print nulls between pathnames, and the bash code to recognize
them.
This was a debugging patch while I was trying to figure out why xfs/286
and other scrub soak tests started OOMing after the v2024.12.08 changes,
though in the end the OOMs were the result of memory leaks in fsstress.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Darrick J. Wong [Wed, 8 Jan 2025 00:02:31 +0000 (16:02 -0800)]
generic/032: fix pinned mount failure
generic/032 now periodically fails with:
--- /tmp/fstests/tests/generic/032.out 2025-01-05 11:42:14.427388698 -0800
+++ /var/tmp/fstests/generic/032.out.bad 2025-01-06 18:20:17.122818195 -0800
@@ -1,5 +1,7 @@
QA output created by 032
100 iterations
-000000 cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd >................<
-*
-100000
+umount: /opt: target is busy.
+mount: /opt: /dev/sda4 already mounted on /opt.
+ dmesg(1) may have more information after failed mount system call.
+cycle mount failed
+(see /var/tmp/fstests/generic/032.full for details)
The root cause of this regression is the _syncloop subshell. This
background process runs _scratch_sync, which is actually an xfs_io
process that calls syncfs on the scratch mount.
Unfortunately, while the test kills the _syncloop subshell, it doesn't
actually kill the xfs_io process. If the xfs_io process is in D state
running the syncfs, it won't react to the signal, but it will pin the
mount. Then the _scratch_cycle_mount fails because the mount is pinned.
Prior to commit 8973af00ec212f the _syncloop ran sync(1) which avoided
pinning the scratch filesystem.
Fix this by pgrepping for the xfs_io process and killing and waiting for
it if necessary.
Cc: <fstests@vger.kernel.org> # v2024.12.08 Fixes: 8973af00ec212f ("fstests: cleanup fsstress process management") Signed-off-by: "Darrick J. Wong" <djwong@kernel.org> Reviewed-by: Dave Chinner <dchinner@redhat.com>
Darrick J. Wong [Sun, 5 Jan 2025 22:28:20 +0000 (14:28 -0800)]
generic/650: revert SOAK DURATION changes
Prior to commit 8973af00ec21, in the absence of an explicit
SOAK_DURATION, this test would run 2500 fsstress operations each of ten
times through the loop body. On the author's machines, this kept the
runtime to about 30s total. Oddly, this was changed to 30s per loop
body with no specific justification in the middle of an fsstress process
management change.
On the author's machine, this explodes the runtime from ~30s to 420s.
Put things back the way they were.
Cc: <fstests@vger.kernel.org> # v2024.12.08 Fixes: 8973af00ec212f ("fstests: cleanup fsstress process management") Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Darrick J. Wong [Mon, 6 Jan 2025 19:04:23 +0000 (11:04 -0800)]
preamble: fix missing _kill_fsstress
Commit 8973af00ec212f added a _kill_fsstress to the standard _cleanup
function. However, if something breaks during test program
initialization before it gets to sourcing common/rc, then you get
failures that look like this:
--- /tmp/fstests/tests/generic/556.out 2024-09-25 12:09:52.938797554 -0700
+++ /var/tmp/fstests/generic/556.out.bad 2025-01-04 22:34:01.268327003 -0800
@@ -1,16 +1,3 @@
QA output created by 556
-SCRATCH_MNT/basic Casefold
-SCRATCH_MNT/basic
-SCRATCH_MNT/casefold_flag_removal Casefold
-SCRATCH_MNT/casefold_flag_removal Casefold
-SCRATCH_MNT/flag_inheritance/d1/d2/d3 Casefold
-SCRATCH_MNT/symlink/ind1/TARGET
-mv: 'SCRATCH_MNT/rename/rename' and 'SCRATCH_MNT/rename/RENAME' are the same file
-# file: SCRATCH_MNT/xattrs/x
-user.foo="bar"
-
-# file: SCRATCH_MNT/xattrs/x/f1
-user.foo="bar"
-
-touch: 'SCRATCH_MNT/strict/corac'$'\314\247\303': Invalid argument
-touch: 'SCRATCH_MNT/strict/cora'$'\303\247\303': Invalid argument
+./tests/generic/556: 108: common/config: Syntax error: "&" unexpected
+./tests/generic/556: 10: _kill_fsstress: not found
It's that last line that's unnecessary. Fix this by checking for the
presence of a _kill_fsstress before invoking it.
Cc: <fstests@vger.kernel.org> # v2024.12.08 Fixes: 8973af00ec212f ("fstests: cleanup fsstress process management") Signed-off-by: "Darrick J. Wong" <djwong@kernel.org> Reviewed-by: Dave Chinner <dchinner@redhat.com>
Darrick J. Wong [Sat, 4 Jan 2025 21:39:29 +0000 (13:39 -0800)]
mkfs: don't hardcode log size
Commit 000813899afb46 hardcoded a log size of 256MB into xfs/501,
xfs/502, and generic/530. This seems to be an attempt to reduce test
run times by increasing the log size so that more background threads can
run in parallel. Unfortunately, this breaks a couple of my test
configurations:
- External logs smaller than 256MB
- Internal logs where the AG size is less than 256MB
For example, here's seqres.full from a failed xfs/501 invocation:
** mkfs failed with extra mkfs options added to " -m metadir=2,autofsck=1,uquota,gquota,pquota, -d rtinherit=1," by test 501 **
** attempting to mkfs using only test 501 options: -l size=256m **
size 256m specified for log subvolume is too large, maximum is 32768 blocks
<snip>
mount -ortdev=/dev/sdb4 -ologdev=/dev/sdb2 /dev/sda4 /opt failed
umount: /dev/sda4: not mounted.
Note that there's some formatting error here, so we jettison the entire
rt configuration to force the log size option, but then mount fails
because we didn't edit out the rtdev option there too.
Fortunately, mkfs.xfs already /has/ a few options to try to improve
parallelism in the filesystem by avoiding contention on the log grant
heads by scaling up the log size. These options are aware of log and AG
size constraints so they won't conflict with other geometry options.
Use them.
Cc: <fstests@vger.kernel.org> # v2024.12.08 Fixes: 000813899afb46 ("fstests: scale some tests for high CPU count sanity") Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Darrick J. Wong [Mon, 6 Jan 2025 19:19:37 +0000 (11:19 -0800)]
unmount: resume logging of stdout and stderr for filtering
There's a number of places where a test program calls a variant of
_unmount but then pipes the output through a _filter script or
something. The new _unmount helper redirects stdout and stderr to
seqres.full, which means that those error messages (some of which are
encoded in the golden outputs) are suppressed. This leads to test
regressions in generic/050 and other places, so let's resume logging.
This also undoes all the changes that removed /dev/null redirection of
unmount calls.
Cc: <fstests@vger.kernel.org> # v2024.12.08 Fixes: 4c6bc4565105e6 ("fstests: clean up mount and unmount operations") Signed-off-by: "Darrick J. Wong" <djwong@kernel.org> Reviewed-by: Dave Chinner <dchinner@redhat.com>
Darrick J. Wong [Thu, 9 Jan 2025 03:07:24 +0000 (19:07 -0800)]
common/rc: don't copy fsstress to $TEST_DIR
Now that we can pkill only processes that were started by this test, we
don't need to copy the fsstress binary to $TEST_DIR to avoid killing the
wrong program instances. This avoids a whole slew of ETXTBSY problems
with scrub stress tests that run multiple copies of fsstress in the
background.
Revert most of the changes to generic/270, because it wants to do
something fancy with the fsstress binary, so it needs to control the
process directly.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org> Reviewed-by: Dave Chinner <dchinner@redhat.com>
Darrick J. Wong [Thu, 9 Jan 2025 18:08:04 +0000 (10:08 -0800)]
common: fix pkill by running test program in a separate session
Run each test program with a separate session id so that we can tell
pkill to kill all processes of a given name, but only within our own
session id. This /should/ suffice to run multiple fstests on the same
machine without one instance shooting down processes of another
instance.
This fixes a general problem with using "pkill --parent" -- if the
process being targeted is not a direct descendant of the bash script
calling pkill, then pkill will not do anything. The scrub stress tests
make use of multiple background subshells, which is how a ^C in the
parent process fails to result in fsx/fsstress being killed.
This is necessary to fix SOAK_DURATION runtime constraints for all the
scrub stress tests. However, there is a cost -- the test program no
longer runs with the same controlling tty as ./check, which means that
^Z doesn't work and SIGINT/SIGQUIT are set to SIG_IGN. IOWs, if a test
wants to kill its subprocesses, it must use another signal such as
SIGPIPE. Fortunately, bash doesn't whine about children dying due to
fatal signals if the children run in a different session id.
I also explored alternate designs, and this was the least unsatisfying:
a) Setting the process group didn't work because background subshells
are assigned a new group id.
b) Constraining the pkill/pgrep search to a cgroup could work, but we'd
have to set up a cgroup in which to run the fstest.
c) Putting test subprocesses in a systemd sub-scope and telling systemd
to kill the sub-scope could work because ./check can already use it to
ensure that all child processes of a test are killed. However, this is
an *optional* feature, which means that we'd have to require systemd.
d) Constraining the pkill/pgrep search to a particular mount namespace
could work, but we already have tests that set up their own mount
namespaces, which means the constrained pgrep will not find all child
processes of a test.
e) Constraining to any other type of namespace (uts, pid, etc) might not
work because those namespaces might not be enabled.
f) Revert check-parallel and go back to one fstests instance per system.
Zorro already chose not to revert.
So. Change _run_seq to create a the ./$seq process with a new session
id, update _su calls to use the same session as the parent test, update
all the pkill sites to use a wrapper so that we only target processes
created by *this* instance of fstests, and update SIGINT to SIGPIPE.
Cc: <fstests@vger.kernel.org> # v2024.12.08 Fixes: 8973af00ec212f ("fstests: cleanup fsstress process management") Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Darrick J. Wong [Sun, 5 Jan 2025 19:44:20 +0000 (11:44 -0800)]
common/rc: create a wrapper for the su command
Create a _su wrapper around the su command so that the next patch can
fix all the pkill isolation code so that this test can only kill
processes started for this test.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Darrick J. Wong [Sat, 4 Jan 2025 20:57:35 +0000 (12:57 -0800)]
fuzzy: do not set _FSSTRESS_PID when exercising fsx
If we're not running fsstress as the scrub exerciser, don't set
_FSSTRESS_PID because the _kill_fsstress call in the cleanup function
will think that it has to wait for a nonexistant fsstress process.
This fixes the problem of xfs/565 runtime increasing from 30s to 800s
because it tries to kill a nonexistent "565.fsstress" process and then
waits for the fsx loop control process, which hasn't been sent any
signals.
Cc: <fstests@vger.kernel.org> # v2024.12.08 Fixes: 8973af00ec212f ("fstests: cleanup fsstress process management") Signed-off-by: "Darrick J. Wong" <djwong@kernel.org> Reviewed-by: Dave Chinner <dchinner@redhat.com>
Darrick J. Wong [Fri, 3 Jan 2025 23:30:21 +0000 (15:30 -0800)]
generic/019: don't fail if fio crashes while shutting down
My system (Debian 12) has fio 3.33. Once in a while, fio crashes while
shutting down after it receives a SIGBUS on account of the filesystem
going down. This causes the test to fail with:
Start fio..
Force SCRATCH_DEV device failure
+/tmp/fstests/tests/generic/019: line 112: 90841 Segmentation fault $FIO_PROG $fio_config >> $seqres.full 2>&1
Make SCRATCH_DEV device operable again
Disallow global fail_make_request feature
...
(Run 'diff -u /tmp/fstests/tests/generic/019.out /var/tmp/fstests/generic/019.out.bad' to see the entire diff)
because the wait command will dutifully report fatal signals that kill
the fio process. Unfortunately, a core dump shows that we blew up in
some library's exit handler somewhere:
(gdb) where
#0 unlink_chunk (p=p@entry=0x55b31cb9a430, av=0x7f8b4475ec60 <main_arena>) at ./malloc/malloc.c:1628
#1 0x00007f8b446222ff in _int_free (av=0x7f8b4475ec60 <main_arena>, p=0x55b31cb9a430, have_lock=<optimized out>, have_lock@entry=0) at ./malloc/malloc.c:4603
#2 0x00007f8b44624f1f in __GI___libc_free (mem=<optimized out>) at ./malloc/malloc.c:3385
#3 0x00007f8b3a71cf0e in ?? () from /lib/x86_64-linux-gnu/libtasn1.so.6
#4 0x00007f8b4426447c in ?? () from /lib/x86_64-linux-gnu/libgnutls.so.30
#5 0x00007f8b4542212a in _dl_call_fini (closure_map=closure_map@entry=0x7f8b44465620) at ./elf/dl-call_fini.c:43
#6 0x00007f8b4542581e in _dl_fini () at ./elf/dl-fini.c:114
#7 0x00007f8b445ca55d in __run_exit_handlers (status=0, listp=0x7f8b4475e820 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true)
at ./stdlib/exit.c:116
#8 0x00007f8b445ca69a in __GI_exit (status=<optimized out>) at ./stdlib/exit.c:146
#9 0x00007f8b445b3251 in __libc_start_call_main (main=main@entry=0x55b319278e10 <main>, argc=argc@entry=2, argv=argv@entry=0x7ffec6f8b468) at ../sysdeps/nptl/libc_start_call_main.h:74
#10 0x00007f8b445b3305 in __libc_start_main_impl (main=0x55b319278e10 <main>, argc=2, argv=0x7ffec6f8b468, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>,
stack_end=0x7ffec6f8b458) at ../csu/libc-start.c:360
#11 0x000055b319278ed1 in _start ()
This isn't a filesystem failure, so mask this by shovelling the output
to seqres.full.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org> Reviewed-by: Dave Chinner <dchinner@redhat.com>
Darrick J. Wong [Fri, 3 Jan 2025 22:24:36 +0000 (14:24 -0800)]
metadump: fix cleanup for v1 metadump testing
In commit ce79de11337e38, the metadump v2 tests were updated to leave
the names of loop devices in some global variables so that the cleanup
method can find them and remove the loop devices. Inexplicably, the
metadump v1 test function was not upgraded. Do so now.
Cc: <fstests@vger.kernel.org> # v2024.12.08 Fixes: ce79de11337e38 ("fstests: clean up loop device instantiation") Signed-off-by: "Darrick J. Wong" <djwong@kernel.org> Reviewed-by: Dave Chinner <dchinner@redhat.com>
Darrick J. Wong [Fri, 3 Jan 2025 22:22:04 +0000 (14:22 -0800)]
metadump: make non-local function variables more obvious
In _xfs_verify_metadump_v2(), we want to set up some loop devices,
record the names of those loop devices, and then leave the variables in
the global namespace so that _xfs_cleanup_verify_metadump can dispose of
them.
Elsewhere in fstests the convention for global variables is to put them
in all caps to make it obvious that they're global and not local
variables, so do that here too.
Cc: <fstests@vger.kernel.org> # v2024.12.08 Fixes: ce79de11337e38 ("fstests: clean up loop device instantiation") Signed-off-by: "Darrick J. Wong" <djwong@kernel.org> Reviewed-by: Dave Chinner <dchinner@redhat.com>
Darrick J. Wong [Fri, 3 Jan 2025 21:28:36 +0000 (13:28 -0800)]
generic/476: fix fsstress process management
generic/476 never used to run fsstress in the background, but 8973af00ec212f made it do that. This is incorrect, because now 476 runs
for three seconds (i.e. long enough to fall out the bottom of the test
and end up in _cleanup), ignoring any SOAK_DURATION/TIME_FACTOR
settings.
Cc: <fstests@vger.kernel.org> # v2024.12.08 Fixes: 8973af00ec212f ("fstests: cleanup fsstress process management") Signed-off-by: "Darrick J. Wong" <djwong@kernel.org> Reviewed-by: Dave Chinner <dchinner@redhat.com>
Filipe Manana [Tue, 7 Jan 2025 16:02:11 +0000 (16:02 +0000)]
btrfs: test cycle mounting a filesystem right after enabling simple quotas
Test that if we enable simple quotas on a filesystem and unmount it right
after without doing any other changes to the filesystem, we are able to
mount again the filesystem.
This is a regression test for the following kernel commit:
f2363e6fcc79 ("btrfs: fix transaction atomicity bug when enabling simple quotas")
Mark Harmstone [Mon, 6 Jan 2025 14:01:34 +0000 (14:01 +0000)]
btrfs: add test for encoded reads
Add btrfs/333 and its helper programs btrfs_encoded_read and
btrfs_encoded_write, in order to test encoded reads.
We use the BTRFS_IOC_ENCODED_WRITE ioctl to write random data into a
compressed extent, then use the BTRFS_IOC_ENCODED_READ ioctl to check
that it matches what we've written. If the new io_uring interface for
encoded reads is supported, we also check that that matches the ioctl.
Note that what we write isn't valid compressed data, so any non-encoded
reads on these files will fail.
Mark Harmstone [Mon, 6 Jan 2025 14:01:33 +0000 (14:01 +0000)]
configure: use pkg-config to find liburing
Change our autoconf macros so that instead of checking for the presence
of liburing.h, we use pkg-config.
The benefit of this is that we can then check the version of liburing,
and do conditional compilation based on this. There's a macro
IO_URING_CHECK_VERSION already, but it's only in relatively recent
versions of liburing.h.
This replaces HAVE_URING_H, defined by AC_CHECK_HEADERS, with
HAVE_URING. I also had to rename PKG_{MAJOR,MINOR,REVISION,BUILD} to
start with PACKAGE_, to avoid "possibly undefined macro" errors; it
looks like pkg-config assumes that anything called PKG_* is for its own
use.
Signed-off-by: Mark Harmstone <maharmstone@fb.com> Reviewed-by: Josef Bacik <josef@toxicpanda.com> Reviewed-by: Anand Jain <anand.jain@oracle.com> Signed-off-by: Anand Jain <anand.jain@oracle.com>
Su Yue [Mon, 6 Jan 2025 14:01:24 +0000 (22:01 +0800)]
common/quota: filter out option projquota in _qmount_option for ocfs2
ocfs2 doesn't support projquota but does support usrquota an grpquota.
For now, two tests generic/594 and generic/603 are for testing
usrquota,grpquota and projquota. The mount option 'projquota' causes
failure of ocfs2 mount.
To make things simple, just skip the tests for ocfs2.
However, we can't just put '_require_prjquota $SCRATCH_DEV' before
_qmount because f2fs and xfs need runtime after mount of SCRATCH_DEV.
For ocfs2, filter out option 'projquota' in _qmount_option() to make
_qmount successful. Then in _require_prjquota(), ocfs2 will fallthrough
as a no kernel projquota support fs type.
Signed-off-by: Su Yue <glass.su@suse.com> Reviewed-by: David Disseldorp <ddiss@suse.de> Signed-off-by: Zorro Lang <zlang@kernel.org>
Zhang Yi [Wed, 8 Jan 2025 08:44:07 +0000 (16:44 +0800)]
generic: add a partial pages zeroing out test
This addresses a data corruption issue encountered during partial page
zeroing in ext4 which the block size is smaller than the page size [1].
Add a new test which is expanded upon generic/567, this test performs a
zeroing range test that spans two partial pages to cover this case, and
also generalize it to work for non-4k page sizes.
Jens Axboe [Tue, 7 Jan 2025 16:05:15 +0000 (09:05 -0700)]
fsx: add support for RWF_DONTCACHE
Using RWF_DONTCACHE tells the kernel that any page cache instantiated
by this operation should get pruned once the operation completes. If
data is in cache prior to the operation it will remain there.
Add ops for testing both the read and write side of this. At startup,
kernel support for this feature is probed. If support isn't available,
uncached/dontcache IO is performed as regular buffered IO. If -Z is
used to turn on O_DIRECT, then uncached/dontcache IO isn't performed.
Defaults to on if available, and adds a -T parameter to turn it off.
Jens Axboe [Tue, 7 Jan 2025 16:05:14 +0000 (09:05 -0700)]
fsstress: add support for RWF_DONTCACHE
Using RWF_DONTCACHE tells the kernel that any page cache instantiated
by this operation should get pruned once the operation completes. If
data is in cache prior to the operation it will remain there.
Add ops for testing both the read and write side of this. If the kernel
being tested doesn't support RWF_DONTCACHE, then operations are performed
with regular read/write buffered IO.
Darrick J. Wong [Tue, 26 Nov 2024 21:32:50 +0000 (13:32 -0800)]
xfs/43[4-6]: implement impatient module reloading
These three tests try to reload the xfs module as a cheap way to detect
leaked inode and dquot objects when the slabs for those object are torn
down during rmmod. Removal might not succeed, and we don't really care
for that case because we still want to exercise the log recovery code.
However, if (say) the root filesystem is xfs, then removal will never
succeed. There's no way that waiting 50 seconds(!) per test is going
to change that. Add a silly helper to do it fast or go home.
Reported-by: sandeen@sandeen.net Signed-off-by: "Darrick J. Wong" <djwong@kernel.org> Reviewed-by: Zorro Lang <zlang@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Darrick J. Wong [Tue, 26 Nov 2024 18:42:40 +0000 (10:42 -0800)]
xfs/032: try running on blocksize > pagesize filesystems
Now that we're no longer limited to blocksize <= pagesize, let's make
sure that mkfs, fsstress, and copy work on such things. This is also a
subtle way to get more people running at least one test with that
config.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org> Reviewed-by: Zorro Lang <zlang@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Eric Biggers [Fri, 13 Dec 2024 05:28:39 +0000 (21:28 -0800)]
generic: verify ciphertext with hardware-wrapped keys
Add two tests which verify that encrypted files are encrypted correctly
when a hardware-wrapped inline encryption key is used. The two tests
are identical except that one uses FSCRYPT_POLICY_FLAG_IV_INO_LBLK_64
and the other uses FSCRYPT_POLICY_FLAG_IV_INO_LBLK_32. These cover both
of the settings where hardware-wrapped keys may be used.
Signed-off-by: Eric Biggers <ebiggers@google.com> Reviewed-by: Zorro Lang <zlang@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Eric Biggers [Fri, 13 Dec 2024 05:28:38 +0000 (21:28 -0800)]
common/encrypt: support hardware-wrapped key testing
To support testing the kernel's support for hardware-wrapped inline
encryption keys, update _verify_ciphertext_for_encryption_policy() to
support a hw_wrapped_key option.
Signed-off-by: Eric Biggers <ebiggers@google.com> Reviewed-by: Zorro Lang <zlang@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Eric Biggers [Fri, 13 Dec 2024 05:28:37 +0000 (21:28 -0800)]
fscrypt-crypt-util: add hardware KDF support
Add support to fscrypt-crypt-util for replicating the extra KDF (Key
Derivation Function) step that is required when a hardware-wrapped
inline encryption key is used. This step normally occurs in hardware,
but we need to replicate it for testing purposes.
Note, some care was needed to handle the fact that both inlinecrypt_key
and sw_secret can be needed in a single run of fscrypt-crypt-util.
Namely, with --iv-ino-lblk-32, inlinecrypt_key is needed for the
en/decryption while sw_secret is needed for hash_inode_number().
Signed-off-by: Eric Biggers <ebiggers@google.com> Reviewed-by: Zorro Lang <zlang@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Chao Yu [Thu, 26 Dec 2024 13:36:31 +0000 (21:36 +0800)]
f2fs/008: test snapshot creation/deletion on lvm device
This patch introduce a regression testcase to check whether
f2fs can handle discard correctly once underlying lvm device
changes to not support discard after user creates snapshot
on it.
Related bug was fixed by commit bc8aeb04fd80 ("f2fs: fix to
drop all discards after creating snapshot on lvm device")
Cc: Jaegeuk Kim <jaegeuk@kernel.org> Signed-off-by: Chao Yu <chao@kernel.org> Reviewed-by: Zorro Lang <zlang@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Theodore Ts'o [Sun, 15 Dec 2024 05:12:42 +0000 (00:12 -0500)]
generic/530: only use xfs-specific mkfs options when testing on xfs
This fixes a regression introduced by commit 000813899afb ("fstests:
scale some tests for high CPU count sanity") where this test
unconditionally tried to use the mkfs option "-l size=256m" which
would break when testing any file sytem other than xfs.
Fix this the same way commit 000813899afb dealt with this for
generic/531; so this was just an oversight.
Fixes: 000813899afb ("fstests: scale some tests for high CPU count sanity") Signed-off-by: Theodore Ts'o <tytso@mit.edu> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Theodore Ts'o [Sun, 15 Dec 2024 05:12:41 +0000 (00:12 -0500)]
generic/135: don't try to rm $SCRATCH_MNT/*
This fixes a regression for ext4 introduced by commit 32e14cb90b88
("fstests: don't use directory stacks"), which replaced a number of
files at the top-level of the scratch file system:
async_file sync_file direct_file trunc_file
with "rm $SCRATCH_MNT/*". This causes a test failure on ext4 file
systems caused by a failed attempt to unlink the lost+found directory.
The thing, is these files are all super small (4k or 16k) and the
scratch file system is going to get reformatted before it gets used
again. So just dropping the delete is the simplest way to solve the
problem.
Fixes: 32e14cb90b88 ("fstests: don't use directory stacks") Signed-off-by: Theodore Ts'o <tytso@mit.edu> Reviewed-by: Christoph Hellwig <hch@lst.de> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Filipe Manana [Wed, 11 Dec 2024 10:55:25 +0000 (10:55 +0000)]
generic/590: fix test failure when running against fs other than xfs
With commit ce79de11337e ("fstests: clean up loop device instantiation")
we started to always call _destroy_loop_device at the very end of the
test, but we only create a loop device if we are running against xfs,
so the call to _destroy_loop_device results in an error since no loop
device was setup.
For example running this test against btrfs or ext4 results in this
failure:
generic/590 29s ... [failed, exit status 1]- output mismatch (see /home/fdmanana/git/hub/xfstests/results//generic/590.out.bad)
--- tests/generic/590.out 2020-06-10 19:29:03.858520038 +0100
+++ /home/fdmanana/git/hub/xfstests/results//generic/590.out.bad 2024-12-11 10:48:43.946205346 +0000
@@ -1,2 +1,5 @@
QA output created by 590
-Silence is golden
+losetup: option requires an argument -- 'd'
+Try 'losetup --help' for more information.
+Cannot destroy loop device
+(see /home/fdmanana/git/hub/xfstests/results//generic/590.full for details)
...
(Run 'diff -u /home/fdmanana/git/hub/xfstests/tests/generic/590.out /home/fdmanana/git/hub/xfstests/results//generic/590.out.bad' to see the entire diff)
Ran: generic/590
Failures: generic/590
Failed 1 of 1 tests
Fix this by removing the call to _destroy_loop_device at the end of the
test, as it's unnecessary because we call it in the _cleanup function if
we have setup a loop device.
Fixes: ce79de11337e ("fstests: clean up loop device instantiation") Signed-off-by: Filipe Manana <fdmanana@suse.com> Reviewed-by: Dave Chinner <dchinner@redhat.com> Reviewed-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Zorro Lang <zlang@kernel.org>
Filipe Manana [Tue, 10 Dec 2024 17:38:32 +0000 (17:38 +0000)]
generic/442: fix failure due to missing test number argument for fsync-err
After commit 88c0291d297c ("fstests: per-test dmerror instances") the
script src/dmerror now has an extra argument, corresponding to a test's
sequence number, but generic/442 isn't passing that argument so the test
fails like this:
projects / users / hch / xfstests-dev.git / log