Eric Sandeen [Fri, 28 Feb 2025 15:13:19 +0000 (09:13 -0600)]
generic: test zero-byte writes to file
A bug was recently fixed in exfat where attempting to do a zero-byte
write would yield -EFAULT; test for that here.
Signed-off-by: Eric Sandeen <sandeen@redhat.com> Reviewed-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: Zorro Lang <zlang@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Zorro Lang [Mon, 10 Feb 2025 16:57:15 +0000 (00:57 +0800)]
fstests: use '-std=gnu11' to fix build errors with gcc-15
GCC-15 does a big change, it changes the default C compilation
language standard to -std=gnu23. That cause lots of "old style"
C codes hit building errors. On the other word, current xfstests
can't be used with GCC-15. So use a fixed language standard --
"-std=gnu11" (to match the kernel and xfsprogs).
Reviewed-by: Darrick J. Wong <djwong@kernel.org> Signed-off-by: Zorro Lang <zlang@kernel.org>
David Sterba [Wed, 19 Feb 2025 22:17:43 +0000 (23:17 +0100)]
fstests: remove reiserfs support
The linux kernel removed Reiserfs support in 6.13 so remove it from the
tests as well.
Signed-off-by: David Sterba <dsterba@suse.com> Reviewed-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: Zorro Lang <zlang@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
David Sterba [Wed, 19 Feb 2025 21:52:28 +0000 (22:52 +0100)]
fstests: remove ltp/growfiles
This utility is not used by any current test and seems that it's never
been used in xfstests, so remove it. Appending files can be simply done
by 'xfs_io' command too.
Signed-off-by: David Sterba <dsterba@suse.com> Reviewed-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: Zorro Lang <zlang@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Jeff Layton [Thu, 20 Feb 2025 11:26:38 +0000 (06:26 -0500)]
generic/126: run it inside its own subdirectory
I had been seeing some failures in generic/126 when running on NFS under
kdevops. The problem turned out to be that kdevops makes the root of the
NFS export mode 01777 by default (i.e. with the sticky bit set). This
causes some of these permission tests to fail because the open() is
called with the O_CREAT bit set.
Make the outcome of this test not depend on the permissions on
$TEST_DIR. Create a new directory to do these tests in, and clean it up
when done.
Signed-off-by: Jeff Layton <jlayton@kernel.org> Reviewed-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: Zorro Lang <zlang@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Filipe Manana [Wed, 19 Feb 2025 18:19:15 +0000 (18:19 +0000)]
btrfs/254: fix test failure in case scratch devices are larger than 50G
If the devices in the scratch pool have a size larger than 50G, then the
test fails due to chunk allocation failure when attempting to create a
multidevice filesystem on $lvdev and $scratch_dev2. This happens because
the $lvdev device has a size of 1G and metadata chunks have a size of 1G
for filesystems with a size greater than 50G, so mkfs fails when it
attempts to allocate chunks since it needs to allocate a 1G metadata
chunk plus a system chunk and a data chunk.
Filipe Manana [Wed, 19 Feb 2025 18:19:14 +0000 (18:19 +0000)]
btrfs/254: don't leave mount on test fs in case of failure/interruption
If the test fails or is interrupted after mounting $scratch_dev3 inside
the test filesystem and before unmounting at test_add_device(), we leave
without being unable to unmount the test filesystem since it has a mount
inside it. This results in the need to manually unmount $scratch_dev3,
otherwise a subsequent run of fstests fails since the unmount of the
test device fails with -EBUSY.
Fix this by unmounting $scratch_dev3 ($seq_mnt) in the _cleanup()
function.
Signed-off-by: Filipe Manana <fdmanana@suse.com> Reviewed-by: David Sterba <dsterba@suse.com> Reviewed-by: Anand Jain <anand.jain@oracle.com> Reviewed-by: Qu Wenruo <wqu@suse.com> Signed-off-by: Anand Jain <anand.jain@oracle.com>
Darrick J. Wong [Mon, 3 Feb 2025 22:00:32 +0000 (14:00 -0800)]
build: initialize stack variables to zero by default
Newer versions of gcc and clang can include the ability to zero stack
variables by default. Let's enable it so that we (a) reduce the risk of
writing stack contents to disk somewhere and (b) try to reduce
unpredictable program behavior based on random stack contents. The
kernel added this 6 years ago, so I think it's mature enough for
fstests.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Darrick J. Wong [Mon, 3 Feb 2025 22:00:31 +0000 (14:00 -0800)]
common/config: add $here to FSSTRESS_PROG
In general we're supposed to specify full paths to fstests binaries with
$here so that subtests can change the current working directory without
issues.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Darrick J. Wong [Mon, 3 Feb 2025 22:00:31 +0000 (14:00 -0800)]
misc: don't put nr_cpus into the fsstress -n argument
fsstress -n is the number of fs operations per process, not the total
number of operations. There's no need to factor nr_cpus into the -n
argument because that causes excess runtime as core count increases.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Darrick J. Wong [Mon, 3 Feb 2025 22:00:31 +0000 (14:00 -0800)]
fsx: fix leaked log file pointer
Fix a resource leaks in fsx, where we fail to close the fsx logfile,
because the C library could have some buffered contents that aren't
flushed when the program terminates. glibc seems to do this for us, but
I wouldn't be so sure about the others.
Fixes: 3f742550dfed84 ("fsx: add support for recording operations to a file") Signed-off-by: "Darrick J. Wong" <djwong@kernel.org> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Darrick J. Wong [Mon, 3 Feb 2025 22:00:31 +0000 (14:00 -0800)]
fsstress: fix a memory leak
Someone forgot to free the iovec that readv_f allocates.
Fixes: 80499d8f5f251e ("fsstress: new writev and readv operations test") Signed-off-by: "Darrick J. Wong" <djwong@kernel.org> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Darrick J. Wong [Mon, 3 Feb 2025 22:00:30 +0000 (14:00 -0800)]
fix _require_scratch_duperemove ordering
Zorro complained that generic/559 stopped working, and I noticed that
the duperemove invocation in the _require_scratch_duperemove function
would fail with:
Error 2: No such file or directory while getting path to file /opt/file1. Skipping.
Error 2: No such file or directory while getting path to file /opt/file2. Skipping.
No dedupe candidates found.
Gathering file list...
The cause of this is the incorrect placement of _require_scratch_dedupe
after a _scratch_mount. _require_scratch_dedupe formats, mounts, tests,
and unmounts the scratch filesystem, which means that it should not come
between a _scratch_mount call and code that uses $SCRATCH_MNT.
Cc: <fstests@vger.kernel.org> # v2024.12.22 Fixes: 3b9f5fc7d7d853 ("common: call _require_scratch_dedupe from _require_scratch_duperemove") Signed-off-by: "Darrick J. Wong" <djwong@kernel.org> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Darrick J. Wong [Mon, 3 Feb 2025 22:00:30 +0000 (14:00 -0800)]
fuzzy: port fsx and fsstress loop to use --duration
Quite a while ago, I added --duration= arguments to fsx and fsstress,
and apparently I forgot to update the scrub stress loops to use them.
Replace the usage of timeout(1) for the remount_period versions of the
loop to clean up that code; and convert the non-remount loop so that
they don't run over time.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Darrick J. Wong [Mon, 3 Feb 2025 22:00:30 +0000 (14:00 -0800)]
fuzzy: always stop the scrub fsstress loop on error
Always abort the non-remount scrub fsstress loop in
__stress_scrub_fsstress_loop if fsstress returns a nonzero exit code.
Cc: <fstests@vger.kernel.org> # v2023.01.15 Fixes: 20df87599f66d0 ("fuzzy: make scrub stress loop control more robust") Signed-off-by: "Darrick J. Wong" <djwong@kernel.org> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Darrick J. Wong [Mon, 3 Feb 2025 22:00:30 +0000 (14:00 -0800)]
fuzzy: don't use readarray for xfsfind output
Some of the scrub stress tests (e.g. xfs/796) walk the directory tree to
find filepaths to scrub, and load the entire list of paths into a bash
array. On a large filesystem or a long-running test this is hugely
wasteful of memory because we use each path exactly once.
Fix __stress_one_scrub_loop to avoid this by reading lines directly from
the output of the xfsfind utility. However, we play some games with fd
77 so that the processes in the loop body will use the same stdin as the
test and /not/ the piped stdout of xfsfind.
To avoid read(1) becoming confused by newlines in the file paths, adapt
xfsfind to print nulls between pathnames, and the bash code to recognize
them.
This was a debugging patch while I was trying to figure out why xfs/286
and other scrub soak tests started OOMing after the v2024.12.08 changes,
though in the end the OOMs were the result of memory leaks in fsstress.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org> Signed-off-by: Zorro Lang <zlang@kernel.org>
Darrick J. Wong [Mon, 3 Feb 2025 22:00:29 +0000 (14:00 -0800)]
generic/032: fix pinned mount failure
generic/032 now periodically fails with:
--- /tmp/fstests/tests/generic/032.out 2025-01-05 11:42:14.427388698 -0800
+++ /var/tmp/fstests/generic/032.out.bad 2025-01-06 18:20:17.122818195 -0800
@@ -1,5 +1,7 @@
QA output created by 032
100 iterations
-000000 cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd >................<
-*
-100000
+umount: /opt: target is busy.
+mount: /opt: /dev/sda4 already mounted on /opt.
+ dmesg(1) may have more information after failed mount system call.
+cycle mount failed
+(see /var/tmp/fstests/generic/032.full for details)
The root cause of this regression is the _syncloop subshell. This
background process runs _scratch_sync, which is actually an xfs_io
process that calls syncfs on the scratch mount.
Unfortunately, while the test kills the _syncloop subshell, it doesn't
actually kill the xfs_io process. If the xfs_io process is in D state
running the syncfs, it won't react to the signal, but it will pin the
mount. Then the _scratch_cycle_mount fails because the mount is pinned.
Prior to commit 8973af00ec212f the _syncloop ran sync(1) which avoided
pinning the scratch filesystem.
Fix this by pgrepping for the xfs_io process and killing and waiting for
it if necessary.
Cc: <fstests@vger.kernel.org> # v2024.12.08 Fixes: 8973af00ec212f ("fstests: cleanup fsstress process management") Signed-off-by: "Darrick J. Wong" <djwong@kernel.org> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Darrick J. Wong [Mon, 3 Feb 2025 22:00:29 +0000 (14:00 -0800)]
generic/650: revert SOAK DURATION changes
Prior to commit 8973af00ec21, in the absence of an explicit
SOAK_DURATION, this test would run 2500 fsstress operations each of ten
times through the loop body. On the author's machines, this kept the
runtime to about 30s total. Oddly, this was changed to 30s per loop
body with no specific justification in the middle of an fsstress process
management change.
On the author's machine, this explodes the runtime from ~30s to 420s.
Put things back the way they were.
Cc: <fstests@vger.kernel.org> # v2024.12.08 Fixes: 8973af00ec212f ("fstests: cleanup fsstress process management") Signed-off-by: "Darrick J. Wong" <djwong@kernel.org> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Darrick J. Wong [Mon, 3 Feb 2025 22:00:29 +0000 (14:00 -0800)]
preamble: fix missing _kill_fsstress
Commit 8973af00ec212f added a _kill_fsstress to the standard _cleanup
function. However, if something breaks during test program
initialization before it gets to sourcing common/rc, then you get
failures that look like this:
--- /tmp/fstests/tests/generic/556.out 2024-09-25 12:09:52.938797554 -0700
+++ /var/tmp/fstests/generic/556.out.bad 2025-01-04 22:34:01.268327003 -0800
@@ -1,16 +1,3 @@
QA output created by 556
-SCRATCH_MNT/basic Casefold
-SCRATCH_MNT/basic
-SCRATCH_MNT/casefold_flag_removal Casefold
-SCRATCH_MNT/casefold_flag_removal Casefold
-SCRATCH_MNT/flag_inheritance/d1/d2/d3 Casefold
-SCRATCH_MNT/symlink/ind1/TARGET
-mv: 'SCRATCH_MNT/rename/rename' and 'SCRATCH_MNT/rename/RENAME' are the same file
-# file: SCRATCH_MNT/xattrs/x
-user.foo="bar"
-
-# file: SCRATCH_MNT/xattrs/x/f1
-user.foo="bar"
-
-touch: 'SCRATCH_MNT/strict/corac'$'\314\247\303': Invalid argument
-touch: 'SCRATCH_MNT/strict/cora'$'\303\247\303': Invalid argument
+./tests/generic/556: 108: common/config: Syntax error: "&" unexpected
+./tests/generic/556: 10: _kill_fsstress: not found
It's that last line that's unnecessary. Fix this by checking for the
presence of a _kill_fsstress before invoking it.
Cc: <fstests@vger.kernel.org> # v2024.12.08 Fixes: 8973af00ec212f ("fstests: cleanup fsstress process management") Signed-off-by: "Darrick J. Wong" <djwong@kernel.org> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Darrick J. Wong [Mon, 3 Feb 2025 22:00:29 +0000 (14:00 -0800)]
mkfs: don't hardcode log size
Commit 000813899afb46 hardcoded a log size of 256MB into xfs/501,
xfs/502, and generic/530. This seems to be an attempt to reduce test
run times by increasing the log size so that more background threads can
run in parallel. Unfortunately, this breaks a couple of my test
configurations:
- External logs smaller than 256MB
- Internal logs where the AG size is less than 256MB
For example, here's seqres.full from a failed xfs/501 invocation:
** mkfs failed with extra mkfs options added to " -m metadir=2,autofsck=1,uquota,gquota,pquota, -d rtinherit=1," by test 501 **
** attempting to mkfs using only test 501 options: -l size=256m **
size 256m specified for log subvolume is too large, maximum is 32768 blocks
<snip>
mount -ortdev=/dev/sdb4 -ologdev=/dev/sdb2 /dev/sda4 /opt failed
umount: /dev/sda4: not mounted.
Note that there's some formatting error here, so we jettison the entire
rt configuration to force the log size option, but then mount fails
because we didn't edit out the rtdev option there too.
Fortunately, mkfs.xfs already /has/ a few options to try to improve
parallelism in the filesystem by avoiding contention on the log grant
heads by scaling up the log size. These options are aware of log and AG
size constraints so they won't conflict with other geometry options.
Use them.
Cc: <fstests@vger.kernel.org> # v2024.12.08 Fixes: 000813899afb46 ("fstests: scale some tests for high CPU count sanity") Signed-off-by: "Darrick J. Wong" <djwong@kernel.org> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Darrick J. Wong [Mon, 3 Feb 2025 22:00:28 +0000 (14:00 -0800)]
unmount: resume logging of stdout and stderr for filtering
There's a number of places where a test program calls a variant of
_unmount but then pipes the output through a _filter script or
something. The new _unmount helper redirects stdout and stderr to
seqres.full, which means that those error messages (some of which are
encoded in the golden outputs) are suppressed. This leads to test
regressions in generic/050 and other places, so let's resume logging.
This also undoes all the changes that removed /dev/null redirection of
unmount calls.
Cc: <fstests@vger.kernel.org> # v2024.12.08 Fixes: 4c6bc4565105e6 ("fstests: clean up mount and unmount operations") Signed-off-by: "Darrick J. Wong" <djwong@kernel.org> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Darrick J. Wong [Mon, 3 Feb 2025 22:00:28 +0000 (14:00 -0800)]
common/rc: don't copy fsstress to $TEST_DIR
Now that we can pkill only processes that were started by this test, we
don't need to copy the fsstress binary to $TEST_DIR to avoid killing the
wrong program instances. This avoids a whole slew of ETXTBSY problems
with scrub stress tests that run multiple copies of fsstress in the
background.
Revert most of the changes to generic/270, because it wants to do
something fancy with the fsstress binary, so it needs to control the
process directly.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Darrick J. Wong [Tue, 4 Feb 2025 20:17:11 +0000 (12:17 -0800)]
check: deprecate using process sessions to isolate test instances
As I've noted elsewhere, the use of process session ids to "isolate"
test instances from killing each other is kind of hacky and creates
other weird side effects. I'd rather everyone use the new code that
runs everything in proper isolation with private pid and mount
namespaces, but I don't know how many people this would break were it a
hard dependency.
Deprecate the process session handling immediately with a warning that
we're going to rip it out in a year.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Darrick J. Wong [Mon, 3 Feb 2025 22:00:28 +0000 (14:00 -0800)]
check: run tests in a private pid/mount namespace
As mentioned in the previous patch, trying to isolate processes from
separate test instances through the use of distinct Unix process
sessions is annoying due to the many complications with signal handling.
Instead, we could just use nsexec to run the test program with a private
pid namespace so that each test instance can only see its own processes;
and private mount namespace so that tests writing to /tmp cannot clobber
other tests or the stuff running on the main system. Further, the
process created by the clone(CLONE_NEWPID) call is considered the init
process of that pid namespace, so all processes will be SIGKILL'd when
the init process terminates, so we no longer need systemd scopes for
externally enforced cleanup.
However, it's not guaranteed that a particular kernel has pid and mount
namespaces enabled. Mount (2.4.19) and pid (2.6.24) namespaces have
been around for a long time, but there's no hard requirement for the
latter to be enabled in the kernel. Therefore, this bugfix slips
namespace support in alongside the session id thing.
Declaring CONFIG_PID_NS=n a deprecated configuration and removing
support should be a separate conversation, not something that I have to
do in a bug fix to get mainline QA back up.
Note that the new helper cannot unmount the /proc it inherits before
mounting a pidns-specific /proc because generic/504 relies on being able
to read the init_pid_ns (aka systemwide) version of /proc/locks to find
a file lock that was taken and leaked by a process.
Cc: <fstests@vger.kernel.org> # v2024.12.08 Fixes: 8973af00ec212f ("fstests: cleanup fsstress process management") Signed-off-by: "Darrick J. Wong" <djwong@kernel.org> Reviewed-by: Dave Chinner <dchinner@redhat.com> Reviewed-by: Zorro Lang <zlang@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Darrick J. Wong [Mon, 3 Feb 2025 22:00:28 +0000 (14:00 -0800)]
common: fix pkill by running test program in a separate session
Run each test program with a separate session id so that we can tell
pkill to kill all processes of a given name, but only within our own
session id. This /should/ suffice to run multiple fstests on the same
machine without one instance shooting down processes of another
instance.
This fixes a general problem with using "pkill --parent" -- if the
process being targeted is not a direct descendant of the bash script
calling pkill, then pkill will not do anything. The scrub stress tests
make use of multiple background subshells, which is how a ^C in the
parent process fails to result in fsx/fsstress being killed.
This is necessary to fix SOAK_DURATION runtime constraints for all the
scrub stress tests. However, there is a cost -- the test program no
longer runs with the same controlling tty as ./check, which means that
^Z doesn't work and SIGINT/SIGQUIT are set to SIG_IGN. IOWs, if a test
wants to kill its subprocesses, it must use another signal such as
SIGPIPE. Fortunately, bash doesn't whine about children dying due to
fatal signals if the children run in a different session id.
Unfortunately we have to let it run the test as a background process for
bash to handle SIGINT, and SIGSTOP no longer works properly.
This solution is a bit crap, and I have a better solution for it in the
next patch that uses private pid and mount namespaces. Unfortunately,
that solution adds new minimum requirements for running fstests and
removing previously supported configurations abruptly during a bug fix
is not appropriate behavior.
I also explored alternate designs, and this was the least unsatisfying:
a) Setting the process group didn't work because background subshells
are assigned a new group id.
b) Constraining the pkill/pgrep search to a cgroup could work, but it
seems that procps has only recently (~2023) gained the ability to filter
on a cgroup. Furthermore, we'd have to set up a cgroup in which to run
the fstest. The last decade has been rife with user bug reports
complaining about chaos resulting from multiple pieces of software (e.g.
Docker, systemd, etc.) deciding that they own the entire cgroup
structure without having any means to enforce that statement. We should
not wade into that mess.
c) Putting test subprocesses in a systemd sub-scope and telling systemd
to kill the sub-scope could work because ./check can already use it to
ensure that all child processes of a test are killed. However, this is
an *optional* feature, which means that we'd have to require systemd.
d) Constraining the pkill/pgrep search to a particular pid namespace
could work, but we already have tests that set up their own mount
namespaces, which means the constrained pgrep will not find all child
processes of a test. Though this hasn't been born out through testing?
e) Constraining to any other type of namespace (uts, pid, etc) might not
work because those namespaces might not be enabled. However, combining
a private pid and mount namespace to isolate tests from each other seems
to work better than session ids. This is coming in a subsequent patch,
but to avoid breaking older systems, we will use this as an immediately
deprecated fallback.
f) Revert check-parallel and go back to one fstests instance per system.
Zorro already chose not to revert.
So. Change _run_seq to create a the ./$seq process with a new session
id, update _su calls to use the same session as the parent test, update
all the pkill sites to use a wrapper so that we only target processes
created by *this* instance of fstests, and update SIGINT to SIGPIPE.
Cc: <fstests@vger.kernel.org> # v2024.12.08 Fixes: 8973af00ec212f ("fstests: cleanup fsstress process management") Signed-off-by: "Darrick J. Wong" <djwong@kernel.org> Reviewed-by: Dave Chinner <dchinner@redhat.com> Reviewed-by: Zorro Lang <zlang@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Darrick J. Wong [Fri, 14 Feb 2025 18:47:14 +0000 (10:47 -0800)]
tools: add a Makefile
Add a Makefile to the tools/ directory so that we can put helper scripts
in there and have them installed at make install time. The makefile
comes from a conversation that Zorro and I had over IRC, hence the RH
copyright.
Suggested-by: Zorro Lang <zlang@kernel.org> Signed-off-by: "Darrick J. Wong" <djwong@kernel.org> Reviewed-by: Zorro Lang <zlang@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Darrick J. Wong [Mon, 3 Feb 2025 22:00:27 +0000 (14:00 -0800)]
fuzzy: kill subprocesses with SIGPIPE, not SIGINT
The next patch in this series fixes various issues with the recently
added fstests process isolation scheme by running each new process in a
separate process group session. Unfortunately, the processes in the
session are created with SIGINT ignored by default because they are not
attached to the controlling terminal. Therefore, switch the kill signal
to SIGPIPE because that is usually fatal and not masked by default.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Darrick J. Wong [Mon, 3 Feb 2025 22:00:27 +0000 (14:00 -0800)]
common/rc: create a wrapper for the su command
Create a _su wrapper around the su command so that the next patch can
fix all the pkill isolation code so that this test can only kill
processes started for this test.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Darrick J. Wong [Tue, 4 Feb 2025 19:15:49 +0000 (11:15 -0800)]
generic/759,760: skip test if we can't set up a hugepage for IO
On an arm64 VM with 64k base pages and a paltry 8G of RAM, this test
will frequently fail like this:
> QA output created by 759
> fsx -N 10000 -l 500000 -h
> -fsx -N 10000 -o 8192 -l 500000 -h
> -fsx -N 10000 -o 128000 -l 500000 -h
> +Seed set to 1
> +madvise collapse for buf: Cannot allocate memory
> +init_hugepages_buf failed for good_buf: Cannot allocate memory
This system has a 512MB hugepage size, which means that there's a good
chance that memory is so fragmented that we won't be able to create a
huge page (in 1/16th the available DRAM). Create a _run_hugepage_fsx
helper that will detect this situation at the start of the test and skip
it, having refactored run_fsx into a properly namespaced version that
won't exit the test on failure.
Cc: <fstests@vger.kernel.org> # v2025.02.02 Cc: joannelkoong@gmail.com Fixes: 627289232371e3 ("generic: add tests for read/writes from hugepages-backed buffers") Signed-off-by: "Darrick J. Wong" <djwong@kernel.org> Reviewed-by: Joanne Koong <joannelkoong@gmail.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Darrick J. Wong [Mon, 3 Feb 2025 22:00:27 +0000 (14:00 -0800)]
generic/759,760: fix MADV_COLLAPSE detection and inclusion
On systems with "old" C libraries such as glibc 2.36 in Debian 12, the
MADV_COLLAPSE flag might not be defined in any of the header files
pulled in by sys/mman.h, which means that the fsx binary might not get
built with any of the MADV_COLLAPSE code. If the kernel supports THP,
the test will fail with:
Fix both tests to detect fsx binaries that don't support MADV_COLLAPSE,
then fix fsx.c to include the mman.h from the kernel headers (aka
linux/mman.h) so that we can actually test IOs to and from THPs if the
kernel is newer than the rest of userspace.
Cc: <fstests@vger.kernel.org> # v2025.02.02 Cc: joannelkoong@gmail.com Fixes: 627289232371e3 ("generic: add tests for read/writes from hugepages-backed buffers") Signed-off-by: "Darrick J. Wong" <djwong@kernel.org> Reviewed-by: Joanne Koong <joannelkoong@gmail.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Darrick J. Wong [Mon, 3 Feb 2025 22:00:27 +0000 (14:00 -0800)]
common/populate: correct the parent pointer name creation formulae
The formulae used to compute the number of parent pointers that we have
to create in a child file in order to generate a particular xattr
structure are not even close to correct -- the first one needs a bit of
adjustment, but the second one is way off and creates far too many
files.
Fix the computation, and document where the magic numbers come from.
Cc: <fstests@vger.kernel.org> # v2024.06.27 Fixes: 0c02207d61af9a ("populate: create hardlinks for parent pointers") Signed-off-by: "Darrick J. Wong" <djwong@kernel.org> Reviewed-by: Zorro Lang <zlang@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Darrick J. Wong [Mon, 3 Feb 2025 22:00:26 +0000 (14:00 -0800)]
common/dump: don't replace pids arbitrarily
In the next patch we'll run tests in a pid namespace, which means that
the test process will have a very low pid. This low pid situation
causes problems with the dump tests because they unconditionally replace
it with the word "PID", which causes unnecessary test failures.
Initially I was going to fix it by bracketing the regexp with a
whitespace/punctuation/eol/sol detector, but then I decided to remove it
see how many sheep came barreling through. None did, so I removed it
entirely. The commit adding it (linked below) was not insightful at
all.
Fixes: 19beb54c96e363 ("Extra filtering as part of IRIX/Linux xfstests reconciliation for dump.") Signed-off-by: "Darrick J. Wong" <djwong@kernel.org> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
This is not a 1:1 conversion here -- previously there was no
-R(ecursive) unmount option, and now there is. It looks as though
umount parses /proc/self/mountinfo to figure out what to unmount, and
maybe on Zorro's system it balks if the argument passed is not present
in that file? Debian 12's umount doesn't care.
Regardless, there was no justification for this change in behavior that
was buried in what is otherwise a hoist patch, so revert it. The author
can resubmit the change with proper documentation.
Cc: <fstests@vger.kernel.org> # v2024.12.08 Fixes: 4c6bc4565105e6 ("fstests: clean up mount and unmount operations") Signed-off-by: "Darrick J. Wong" <djwong@kernel.org> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Darrick J. Wong [Mon, 3 Feb 2025 22:00:26 +0000 (14:00 -0800)]
fuzzy: do not set _FSSTRESS_PID when exercising fsx
If we're not running fsstress as the scrub exerciser, don't set
_FSSTRESS_PID because the _kill_fsstress call in the cleanup function
will think that it has to wait for a nonexistant fsstress process.
This fixes the problem of xfs/565 runtime increasing from 30s to 800s
because it tries to kill a nonexistent "565.fsstress" process and then
waits for the fsx loop control process, which hasn't been sent any
signals.
Cc: <fstests@vger.kernel.org> # v2024.12.08 Fixes: 8973af00ec212f ("fstests: cleanup fsstress process management") Signed-off-by: "Darrick J. Wong" <djwong@kernel.org> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Darrick J. Wong [Mon, 3 Feb 2025 22:00:26 +0000 (14:00 -0800)]
generic/019: don't fail if fio crashes while shutting down
My system (Debian 12) has fio 3.33. Once in a while, fio crashes while
shutting down after it receives a SIGBUS on account of the filesystem
going down. This causes the test to fail with:
Start fio..
Force SCRATCH_DEV device failure
+/tmp/fstests/tests/generic/019: line 112: 90841 Segmentation fault $FIO_PROG $fio_config >> $seqres.full 2>&1
Make SCRATCH_DEV device operable again
Disallow global fail_make_request feature
...
(Run 'diff -u /tmp/fstests/tests/generic/019.out /var/tmp/fstests/generic/019.out.bad' to see the entire diff)
because the wait command will dutifully report fatal signals that kill
the fio process. Unfortunately, a core dump shows that we blew up in
some library's exit handler somewhere:
(gdb) where
#0 unlink_chunk (p=p@entry=0x55b31cb9a430, av=0x7f8b4475ec60 <main_arena>) at ./malloc/malloc.c:1628
#1 0x00007f8b446222ff in _int_free (av=0x7f8b4475ec60 <main_arena>, p=0x55b31cb9a430, have_lock=<optimized out>, have_lock@entry=0) at ./malloc/malloc.c:4603
#2 0x00007f8b44624f1f in __GI___libc_free (mem=<optimized out>) at ./malloc/malloc.c:3385
#3 0x00007f8b3a71cf0e in ?? () from /lib/x86_64-linux-gnu/libtasn1.so.6
#4 0x00007f8b4426447c in ?? () from /lib/x86_64-linux-gnu/libgnutls.so.30
#5 0x00007f8b4542212a in _dl_call_fini (closure_map=closure_map@entry=0x7f8b44465620) at ./elf/dl-call_fini.c:43
#6 0x00007f8b4542581e in _dl_fini () at ./elf/dl-fini.c:114
#7 0x00007f8b445ca55d in __run_exit_handlers (status=0, listp=0x7f8b4475e820 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true)
at ./stdlib/exit.c:116
#8 0x00007f8b445ca69a in __GI_exit (status=<optimized out>) at ./stdlib/exit.c:146
#9 0x00007f8b445b3251 in __libc_start_call_main (main=main@entry=0x55b319278e10 <main>, argc=argc@entry=2, argv=argv@entry=0x7ffec6f8b468) at ../sysdeps/nptl/libc_start_call_main.h:74
#10 0x00007f8b445b3305 in __libc_start_main_impl (main=0x55b319278e10 <main>, argc=2, argv=0x7ffec6f8b468, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>,
stack_end=0x7ffec6f8b458) at ../csu/libc-start.c:360
#11 0x000055b319278ed1 in _start ()
This isn't a filesystem failure, so mask this by shovelling the output
to seqres.full.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Darrick J. Wong [Mon, 3 Feb 2025 22:00:25 +0000 (14:00 -0800)]
metadump: fix cleanup for v1 metadump testing
In commit ce79de11337e38, the metadump v2 tests were updated to leave
the names of loop devices in some global variables so that the cleanup
method can find them and remove the loop devices. Inexplicably, the
metadump v1 test function was not upgraded. Do so now.
Cc: <fstests@vger.kernel.org> # v2024.12.08 Fixes: ce79de11337e38 ("fstests: clean up loop device instantiation") Signed-off-by: "Darrick J. Wong" <djwong@kernel.org> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Darrick J. Wong [Mon, 3 Feb 2025 22:00:25 +0000 (14:00 -0800)]
metadump: make non-local function variables more obvious
In _xfs_verify_metadump_v2(), we want to set up some loop devices,
record the names of those loop devices, and then leave the variables in
the global namespace so that _xfs_cleanup_verify_metadump can dispose of
them.
Elsewhere in fstests the convention for global variables is to put them
in all caps to make it obvious that they're global and not local
variables, so do that here too.
Cc: <fstests@vger.kernel.org> # v2024.12.08 Fixes: ce79de11337e38 ("fstests: clean up loop device instantiation") Signed-off-by: "Darrick J. Wong" <djwong@kernel.org> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Darrick J. Wong [Mon, 3 Feb 2025 22:00:25 +0000 (14:00 -0800)]
generic/476: fix fsstress process management
generic/476 never used to run fsstress in the background, but 8973af00ec212f made it do that. This is incorrect, because now 476 runs
for three seconds (i.e. long enough to fall out the bottom of the test
and end up in _cleanup), ignoring any SOAK_DURATION/TIME_FACTOR
settings.
Cc: <fstests@vger.kernel.org> # v2024.12.08 Fixes: 8973af00ec212f ("fstests: cleanup fsstress process management") Signed-off-by: "Darrick J. Wong" <djwong@kernel.org> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Filipe Manana [Wed, 12 Feb 2025 23:35:06 +0000 (23:35 +0000)]
btrfs: skip tests that exercise compression property when using nodatasum
A couple tests exercise the compression property and that fails when an
inode has the nodatasum flag set. So skip the tests when running under the
nodatasum mount option.
Signed-off-by: Filipe Manana <fdmanana@suse.com> Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com> Reviewed-by: David Sterba <dsterba@suse.com> Reviewed-by: Qu Wenruo <wqu@suse.com> Reviewed-by: Anand Jain <anand.jain@oracle.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Filipe Manana [Wed, 12 Feb 2025 23:35:05 +0000 (23:35 +0000)]
btrfs/281: skip test when running with nodatasum mount option
The test exercises compression and compression doesn't happen on inodes
with checksums disabled (nodatasum), making the test fail the expectations
if getting compressed extents. So skip the test if nodatasum is present.
Signed-off-by: Filipe Manana <fdmanana@suse.com> Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com> Reviewed-by: David Sterba <dsterba@suse.com> Reviewed-by: Qu Wenruo <wqu@suse.com> Reviewed-by: Anand Jain <anand.jain@oracle.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Filipe Manana [Wed, 12 Feb 2025 23:35:04 +0000 (23:35 +0000)]
btrfs: skip tests exercising data corruption and repair when using nodatasum
Several tests exercise corrupting data and then checking that on read the
data is repaired, but this requires using checksums, so the tests fail
when running with the nodatasum mount option.
So add a _require_btrfs_no_nodatasum call to these tests.
Signed-off-by: Filipe Manana <fdmanana@suse.com> Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com> Reviewed-by: Qu Wenruo <wqu@suse.com> Reviewed-by: Anand Jain <anand.jain@oracle.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Filipe Manana [Wed, 12 Feb 2025 23:35:03 +0000 (23:35 +0000)]
btrfs/205: skip test when running with nodatasum mount option
Currently the test fails when we pass "-o nodatasum" in MOUNT_OPTIONS and
the reason is because we enable compression, with "chattr +c", on a file
and then try to clone from it to a file with nodatasum inherited from the
mount options, which results in the clone ioctl to fail with -EINVAL since
it's not possible to clone from datasum to nodatasum and vice-versa.
So skip the test if nodatasum is a mount option.
Signed-off-by: Filipe Manana <fdmanana@suse.com> Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com> Reviewed-by: Qu Wenruo <wqu@suse.com> Reviewed-by: Anand Jain <anand.jain@oracle.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Filipe Manana [Wed, 12 Feb 2025 23:35:02 +0000 (23:35 +0000)]
btrfs/333: skip the test when running with nodatacow or nodatasum
Encoded writes are reject for inodes with the NODATASUM flag, so we must skip
the test if running with either the nodatasum or nodatacow (which implies
nodatasum) mount options.
So skip the test when under nodatacow and nodatasum mount options.
Signed-off-by: Filipe Manana <fdmanana@suse.com> Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com> Reviewed-by: Qu Wenruo <wqu@suse.com> Reviewed-by: Anand Jain <anand.jain@oracle.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Filipe Manana [Wed, 12 Feb 2025 23:35:01 +0000 (23:35 +0000)]
common/btrfs: add a _require_btrfs_no_nodatasum helper
Add a _require_btrfs_no_nodatasum helper to skip a test if the nodatasum
mount option is give, as we do have several tests that fail, for several
reasons, when that mount option is passed.
Signed-off-by: Filipe Manana <fdmanana@suse.com> Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com> Reviewed-by: Qu Wenruo <wqu@suse.com> Reviewed-by: Anand Jain <anand.jain@oracle.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Filipe Manana [Wed, 12 Feb 2025 23:35:00 +0000 (23:35 +0000)]
btrfs/290: skip test if we are running with nodatacow mount option
We exercise corrupting an inline extent and inline extents can't be created
with nodatacow, we get instead a regular file extent item and if we attempt
to corrupt its disk_bytenr field with btrfs-corrupt-block we fail tree-checker
validation at mount time resulting in failure to mount and the following in
dmesg:
[514127.759739] BTRFS critical (device sdc): corrupt leaf: root=5 \
block=30408704 slot=8 ino=257 file_offset=0, invalid disk_bytenr for \
file extent, have 7416089308958521981, should be aligned to 4096
[514127.762715] BTRFS error (device sdc): read time tree block corruption \
detected on logical 30408704 mirror 1
So add a _require_btrfs_no_nodatacow call to the test.
Signed-off-by: Filipe Manana <fdmanana@suse.com> Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com> Reviewed-by: Qu Wenruo <wqu@suse.com> Reviewed-by: Anand Jain <anand.jain@oracle.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Filipe Manana [Wed, 12 Feb 2025 23:34:59 +0000 (23:34 +0000)]
btrfs: skip tests incompatible with compression when compression is enabled
We have several tests that fail when compression is enabled (MOUNT_OPTIONS
has "-o compress" or "-o compress-force"") because they expect a fixed
extent size and they trigger corruption by writing directly to a device,
therefore making them incompatible with compression.
So add a _require_no_compress call to them.
Signed-off-by: Filipe Manana <fdmanana@suse.com> Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com> Reviewed-by: Qu Wenruo <wqu@suse.com> Reviewed-by: Anand Jain <anand.jain@oracle.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Filipe Manana [Thu, 6 Feb 2025 13:05:06 +0000 (13:05 +0000)]
generic: suggest fs specific fix only if the tested filesystem matches
It's odd when a test fails on a filesystem and a specific fix is suggested
for another filesystem. Some generic tests are suggesting filesystem
specific fixes without checking if the running filesystem matches, so
update them.
Signed-off-by: Filipe Manana <fdmanana@suse.com> Reviewed-by: Zorro Lang <zlang@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Qu Wenruo [Wed, 12 Feb 2025 03:04:06 +0000 (13:34 +1030)]
fstests: add a generic test to verify direct IO writes with buffer contents change
There is a long existing btrfs problem that if some one is modifying the
buffer of an on-going direct IO write, it has a very high chance causing
permanent data checksum mismatch.
This is caused by the following factors:
- Direct IO buffer is out of the control of filesystem
Thus user space can modify the contents during writeback.
- Btrfs generate its data checksum just before submitting the bio
This means if the contents happens after the checksum is generated,
the data written to disk will no longer match the checksum.
Btrfs later fixes the problem by forcing the direct IO to fallback to
buffered IO (if the inode requires data checksum), so that btrfs can
have a consistent view of the buffer.
This test case will verify the behavior by:
- Create a helper program 'dio-writeback-race'
Which does direct IO writes block-by-block, and the buffer is always
initialized to all 0xff before write,
Then starting two threads:
- One to submit the direct IO
- One to modify the buffer to 0x00
The program uses 4K as default block size, and 64MiB as the default
file size.
Which is more than enough to trigger tons of btrfs checksum errors
on unpatched kernels.
- New test case generic/761
Which will:
* Use above program to create a 64MiB file
* Do buffered read on that file
Since above program is doing direct IO, there is no page cache
populated.
And the buffered read will need to read out all data from the disk,
and if the filesystem supports data checksum, then the data checksum
will also be verified against the data.
The test case passes on the following fses:
- ext4
- xfs
- btrfs with "nodatasum" mount option
No data checksum to bother.
- btrfs with default "datasum" mount option and the fix "btrfs: always
fallback to buffered write if the inode requires checksum"
This makes btrfs to fallback on buffered IO so the contents won't
change during writeback of page cache.
And fails on the following fses:
- btrfs with default "datasum" mount option and without the fix
Expected.
Suggested-by: Christoph Hellwig <hch@infradead.org> Reviewed-by: David Disseldorp <ddiss@suse.de> Reviewed-by: Zorro Lang <zlang@redhat.com> Reviewed-by: "Darrick J. Wong" <djwong@kernel.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Qu Wenruo <wqu@suse.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Eric Biggers [Sat, 18 Jan 2025 07:23:36 +0000 (23:23 -0800)]
fscrypt-crypt-util: fix KDF contexts for SM8650
Update the KDF contexts to match those actually used on SM8650. This
turns out to be needed for the hardware-wrapped key tests generic/368
and generic/369 to pass on the SM8650 HDK (now that I have one to
actually test it). Apparently the contexts changed between the
prototype version I tested a couple years ago and the final version.
Signed-off-by: Eric Biggers <ebiggers@google.com> Reviewed-by: Zorro Lang <zlang@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Qu Wenruo [Wed, 5 Feb 2025 21:58:46 +0000 (08:28 +1030)]
fstests: btrfs/226: use nodatasum mount option to prevent false alerts
[BUG]
With recent kernel patch "btrfs: always fallback to buffered write if the
inode requires checksum", the test case btrfs/226 will fail with the
following error:
btrfs/226 1s ... - output mismatch (see /home/adam/xfstests/results//btrfs/226.out.bad)
--- tests/btrfs/226.out 2024-04-12 14:04:03.080000035 +0930
+++ /home/adam/xfstests/results//btrfs/226.out.bad 2025-02-06 08:23:42.564298585 +1030
@@ -39,14 +39,11 @@
Testing write against prealloc extent at eof
wrote 65536/65536 bytes at offset 0
XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-wrote 65536/65536 bytes at offset 65536
-XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+pwrite: Resource temporarily unavailable
File after write:
...
(Run 'diff -u /home/adam/xfstests/tests/btrfs/226.out /home/adam/xfstests/results//btrfs/226.out.bad' to see the entire diff)
Ran: btrfs/226
Failures: btrfs/226
Failed 1 of 1 tests
[CAUSE]
That kernel patch makes btrfs to always fallback to buffered IO if the
target inode requires data checksum.
This is to avoid more deadly problems of mismatched data checksum.
But this also means, for inodes with data checksum, RWF_NOWAIT will
always fail, because we will wait writing back the page cache, thus
breaking the RWF_NOWAIT requirement.
[FIX]
Update the test case to utilize nodatasum mount option, so that the
direct-IO will not fallback to buffered ones unconditionally.
Reported-by: Filipe Manana <fdmanana@kernel.org> Signed-off-by: Qu Wenruo <wqu@suse.com> Reviewed-by: Filipe Manana <fdmanana@suse.com> Reviewed-by: Anand Jain <anand.jain@oracle.com>
Christoph Hellwig [Tue, 4 Feb 2025 13:46:12 +0000 (14:46 +0100)]
common: fix a spelling error in _require_zoned_device
Signed-off-by: Christoph Hellwig <hch@lst.de> Reviewed-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: Zorro Lang <zlang@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Nirjhar Roy (IBM) [Tue, 28 Jan 2025 05:00:22 +0000 (05:00 +0000)]
check: Fix fs specfic imports when $FSTYPE!=$OLD_FSTYPE
Bug Description:
_test_mount function is failing with the following error:
./common/rc: line 4716: _xfs_prepare_for_eio_shutdown: command not found
check: failed to mount /dev/loop0 on /mnt1/test
when the second section in local.config file is xfs and the first section
is non-xfs.
It can be easily reproduced with the following local.config file
Root cause:
When _test_mount() is executed for the second section, the FSTYPE has
already changed but the new fs specific common/$FSTYP has not yet
been done. Hence _xfs_prepare_for_eio_shutdown() is not found and
the test run fails.
Fix:
Remove the additional _test_mount in check file just before ". common/rc"
since ". common/rc" is already sourcing fs specific imports and doing a
_test_mount.
Fixes: 1a49022fab9b4 ("fstests: always use fail-at-unmount semantics for XFS") Signed-off-by: Nirjhar Roy (IBM) <nirjhar.roy.lists@gmail.com> Reviewed-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: Zorro Lang <zlang@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Eric Sandeen [Mon, 27 Jan 2025 21:43:24 +0000 (15:43 -0600)]
fix vfs/utils.c for big-endian systems
generic/633 was failing with EINVAL on the fsxgetattr call on s390.
Looks like this is due to a failure to properly endian swap the
arguments to the syscall, so fix that, and the magic_etc compare
in expected_dummy_vfs_caps_uid() as well while we're at it.
Fixes: 0d1af68e ("generic: add fstests for idmapped mounts") Signed-off-by: Eric Sandeen <sandeen@redhat.com> Reviewed-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Zorro Lang <zlang@kernel.org>
Christoph Hellwig [Tue, 28 Jan 2025 07:16:05 +0000 (08:16 +0100)]
common: skip tests using LVM when the device is no known
LVM has a lot of elaborate code to make the users life painful. This
includes claiming the device type is unknown if it doesn't match it's
elaborately crafted internal list instead of just letting the user use
it. Skip tests using LVM if this is the case to avoid arcane failures
due to missing lvm devices when using null_blk.
Signed-off-by: Christoph Hellwig <hch@lst.de> Reviewed-by: Darrick J. Wong <djwong@kernel.org> Signed-off-by: Zorro Lang <zlang@kernel.org>
Christoph Hellwig [Tue, 28 Jan 2025 07:13:00 +0000 (08:13 +0100)]
replace _supported_fs with _exclude_fs
Tests don't require a list of supported file systems, as that is deducted
from the test directory name. Instead we exclude specific file systems
from a few common tests, and specify which of ext2 and ext3 should
actually also be tested after oddly multiplexing them into the ext4
directory full of tests only working for ext4.
Replace _supported_fs with a new _exclude_fs that takes only a single
file systems as the argument, making it easier to explain why the file
system is not supported for the common test. For ext4 this increases
the existing mess even further, but the maintainers have a plan to
move it to feature checks instead that are hopefully easier to
understand.
Signed-off-by: Christoph Hellwig <hch@lst.de> Reviewed-by: Darrick J. Wong <djwong@kernel.org> # common and tests/generic Acked-by: Darrick J. Wong <djwong@kernel.org> # tests/ext4 Signed-off-by: Zorro Lang <zlang@kernel.org>
Christoph Hellwig [Tue, 28 Jan 2025 07:12:58 +0000 (08:12 +0100)]
generic/363: remove _supported_fs xfs
Run this test for all file systems. Just because they are broken doesn't
mean that zeroing should not be tested.
Signed-off-by: Christoph Hellwig <hch@lst.de> Reviewed-by: Brian Foster <bfoster@redhat.com> Reviewed-by: Darrick J. Wong <djwong@kernel.org> Signed-off-by: Zorro Lang <zlang@kernel.org>
Joanne Koong [Tue, 21 Jan 2025 21:56:41 +0000 (13:56 -0800)]
generic: add tests for read/writes from hugepages-backed buffers
Add generic tests 758 and 759 for testing reads/writes from buffers
backed by hugepages.
Signed-off-by: Joanne Koong <joannelkoong@gmail.com> Reviewed-by: Brian Foster <bfoster@redhat.com> Reviewed-by: Darrick J. Wong <djwong@kernel.org> Signed-off-by: Zorro Lang <zlang@kernel.org>
Joanne Koong [Tue, 21 Jan 2025 21:56:40 +0000 (13:56 -0800)]
fsx: support reads/writes from buffers backed by hugepages
Add support for reads/writes from buffers backed by hugepages.
This can be enabled through the '-h' flag. This flag should only be used
on systems where THP capabilities are enabled.
This is motivated by a recent bug that was due to faulty handling of
userspace buffers backed by hugepages. This patch is a mitigation
against problems like this in the future.
Signed-off-by: Joanne Koong <joannelkoong@gmail.com> Reviewed-by: Brian Foster <bfoster@redhat.com> Reviewed-by: Darrick J. Wong <djwong@kernel.org> Signed-off-by: Zorro Lang <zlang@kernel.org>
Theodore Ts'o [Mon, 20 Jan 2025 17:01:09 +0000 (12:01 -0500)]
xfs/032: fix test failure on kernels which don't support bs > ps
When trying to mount a file system with a block size > page size on
kernel which doesn't support this, suppress the error messages from
showing up in the output file lest it cause test failures.
Fixes: 0b66f6efd669 ("xfs/032: try running on blocksize > pagesize filesystems") Signed-off-by: Theodore Ts'o <tytso@mit.edu> Reviewed-by: Zorro Lang <zlang@redhat.com> Reviewed-by: "Darrick J. Wong" <djwong@kernel.org> Signed-off-by: Zorro Lang <zlang@kernel.org>
Shin'ichiro Kawasaki [Tue, 14 Jan 2025 03:48:31 +0000 (12:48 +0900)]
fsx: fix compile error for preadv2()
Commit d6b9d8eff076 ("fsx: add support for RWF_DONTCACHE") introduced
preadv2() calls in ltp/fsx.c. However, sys/uio.h is not included to the
source file, which causes compile errors with the gcc option
-Werror-implicit-function-declaration:
fsx.c: In function 'test_dontcache_io':
fsx.c:1956:15: error: implicit declaration of function 'preadv2'; did you mean 'pread64'? [-Wimplicit-function-declaration]
1956 | ret = preadv2(fd, &iov, 1, 0, RWF_DONTCACHE);
| ^~~~~~~
| pread64
fsx.c: In function 'fsx_rw':
fsx.c:2836:31: error: implicit declaration of function 'pwritev2'; did you mean 'pwrite64'? [-Wimplicit-function-declaration]
2836 | ret = pwritev2(fd, &iov, 1, offset, flags);
| ^~~~~~~~
| pwrite64
To fix it, add the include directive.
Fixes: d6b9d8eff076 ("fsx: add support for RWF_DONTCACHE") Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com> Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com> Reviewed-by: Jens Axboe <axboe@kernel.dk> Reviewed-by: Zorro Lang <zlang@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Filipe Manana [Tue, 7 Jan 2025 16:02:11 +0000 (16:02 +0000)]
btrfs: test cycle mounting a filesystem right after enabling simple quotas
Test that if we enable simple quotas on a filesystem and unmount it right
after without doing any other changes to the filesystem, we are able to
mount again the filesystem.
This is a regression test for the following kernel commit:
f2363e6fcc79 ("btrfs: fix transaction atomicity bug when enabling simple quotas")
Mark Harmstone [Mon, 6 Jan 2025 14:01:34 +0000 (14:01 +0000)]
btrfs: add test for encoded reads
Add btrfs/333 and its helper programs btrfs_encoded_read and
btrfs_encoded_write, in order to test encoded reads.
We use the BTRFS_IOC_ENCODED_WRITE ioctl to write random data into a
compressed extent, then use the BTRFS_IOC_ENCODED_READ ioctl to check
that it matches what we've written. If the new io_uring interface for
encoded reads is supported, we also check that that matches the ioctl.
Note that what we write isn't valid compressed data, so any non-encoded
reads on these files will fail.
Mark Harmstone [Mon, 6 Jan 2025 14:01:33 +0000 (14:01 +0000)]
configure: use pkg-config to find liburing
Change our autoconf macros so that instead of checking for the presence
of liburing.h, we use pkg-config.
The benefit of this is that we can then check the version of liburing,
and do conditional compilation based on this. There's a macro
IO_URING_CHECK_VERSION already, but it's only in relatively recent
versions of liburing.h.
This replaces HAVE_URING_H, defined by AC_CHECK_HEADERS, with
HAVE_URING. I also had to rename PKG_{MAJOR,MINOR,REVISION,BUILD} to
start with PACKAGE_, to avoid "possibly undefined macro" errors; it
looks like pkg-config assumes that anything called PKG_* is for its own
use.
Signed-off-by: Mark Harmstone <maharmstone@fb.com> Reviewed-by: Josef Bacik <josef@toxicpanda.com> Reviewed-by: Anand Jain <anand.jain@oracle.com> Signed-off-by: Anand Jain <anand.jain@oracle.com>
Su Yue [Mon, 6 Jan 2025 14:01:24 +0000 (22:01 +0800)]
common/quota: filter out option projquota in _qmount_option for ocfs2
ocfs2 doesn't support projquota but does support usrquota an grpquota.
For now, two tests generic/594 and generic/603 are for testing
usrquota,grpquota and projquota. The mount option 'projquota' causes
failure of ocfs2 mount.
To make things simple, just skip the tests for ocfs2.
However, we can't just put '_require_prjquota $SCRATCH_DEV' before
_qmount because f2fs and xfs need runtime after mount of SCRATCH_DEV.
For ocfs2, filter out option 'projquota' in _qmount_option() to make
_qmount successful. Then in _require_prjquota(), ocfs2 will fallthrough
as a no kernel projquota support fs type.
Signed-off-by: Su Yue <glass.su@suse.com> Reviewed-by: David Disseldorp <ddiss@suse.de> Signed-off-by: Zorro Lang <zlang@kernel.org>
Zhang Yi [Wed, 8 Jan 2025 08:44:07 +0000 (16:44 +0800)]
generic: add a partial pages zeroing out test
This addresses a data corruption issue encountered during partial page
zeroing in ext4 which the block size is smaller than the page size [1].
Add a new test which is expanded upon generic/567, this test performs a
zeroing range test that spans two partial pages to cover this case, and
also generalize it to work for non-4k page sizes.
Jens Axboe [Tue, 7 Jan 2025 16:05:15 +0000 (09:05 -0700)]
fsx: add support for RWF_DONTCACHE
Using RWF_DONTCACHE tells the kernel that any page cache instantiated
by this operation should get pruned once the operation completes. If
data is in cache prior to the operation it will remain there.
Add ops for testing both the read and write side of this. At startup,
kernel support for this feature is probed. If support isn't available,
uncached/dontcache IO is performed as regular buffered IO. If -Z is
used to turn on O_DIRECT, then uncached/dontcache IO isn't performed.
Defaults to on if available, and adds a -T parameter to turn it off.
Jens Axboe [Tue, 7 Jan 2025 16:05:14 +0000 (09:05 -0700)]
fsstress: add support for RWF_DONTCACHE
Using RWF_DONTCACHE tells the kernel that any page cache instantiated
by this operation should get pruned once the operation completes. If
data is in cache prior to the operation it will remain there.
Add ops for testing both the read and write side of this. If the kernel
being tested doesn't support RWF_DONTCACHE, then operations are performed
with regular read/write buffered IO.
Darrick J. Wong [Tue, 26 Nov 2024 21:32:50 +0000 (13:32 -0800)]
xfs/43[4-6]: implement impatient module reloading
These three tests try to reload the xfs module as a cheap way to detect
leaked inode and dquot objects when the slabs for those object are torn
down during rmmod. Removal might not succeed, and we don't really care
for that case because we still want to exercise the log recovery code.
However, if (say) the root filesystem is xfs, then removal will never
succeed. There's no way that waiting 50 seconds(!) per test is going
to change that. Add a silly helper to do it fast or go home.
Reported-by: sandeen@sandeen.net Signed-off-by: "Darrick J. Wong" <djwong@kernel.org> Reviewed-by: Zorro Lang <zlang@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Darrick J. Wong [Tue, 26 Nov 2024 18:42:40 +0000 (10:42 -0800)]
xfs/032: try running on blocksize > pagesize filesystems
Now that we're no longer limited to blocksize <= pagesize, let's make
sure that mkfs, fsstress, and copy work on such things. This is also a
subtle way to get more people running at least one test with that
config.
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org> Reviewed-by: Zorro Lang <zlang@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Eric Biggers [Fri, 13 Dec 2024 05:28:39 +0000 (21:28 -0800)]
generic: verify ciphertext with hardware-wrapped keys
Add two tests which verify that encrypted files are encrypted correctly
when a hardware-wrapped inline encryption key is used. The two tests
are identical except that one uses FSCRYPT_POLICY_FLAG_IV_INO_LBLK_64
and the other uses FSCRYPT_POLICY_FLAG_IV_INO_LBLK_32. These cover both
of the settings where hardware-wrapped keys may be used.
Signed-off-by: Eric Biggers <ebiggers@google.com> Reviewed-by: Zorro Lang <zlang@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Eric Biggers [Fri, 13 Dec 2024 05:28:38 +0000 (21:28 -0800)]
common/encrypt: support hardware-wrapped key testing
To support testing the kernel's support for hardware-wrapped inline
encryption keys, update _verify_ciphertext_for_encryption_policy() to
support a hw_wrapped_key option.
Signed-off-by: Eric Biggers <ebiggers@google.com> Reviewed-by: Zorro Lang <zlang@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Eric Biggers [Fri, 13 Dec 2024 05:28:37 +0000 (21:28 -0800)]
fscrypt-crypt-util: add hardware KDF support
Add support to fscrypt-crypt-util for replicating the extra KDF (Key
Derivation Function) step that is required when a hardware-wrapped
inline encryption key is used. This step normally occurs in hardware,
but we need to replicate it for testing purposes.
Note, some care was needed to handle the fact that both inlinecrypt_key
and sw_secret can be needed in a single run of fscrypt-crypt-util.
Namely, with --iv-ino-lblk-32, inlinecrypt_key is needed for the
en/decryption while sw_secret is needed for hash_inode_number().
Signed-off-by: Eric Biggers <ebiggers@google.com> Reviewed-by: Zorro Lang <zlang@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Chao Yu [Thu, 26 Dec 2024 13:36:31 +0000 (21:36 +0800)]
f2fs/008: test snapshot creation/deletion on lvm device
This patch introduce a regression testcase to check whether
f2fs can handle discard correctly once underlying lvm device
changes to not support discard after user creates snapshot
on it.
Related bug was fixed by commit bc8aeb04fd80 ("f2fs: fix to
drop all discards after creating snapshot on lvm device")
Cc: Jaegeuk Kim <jaegeuk@kernel.org> Signed-off-by: Chao Yu <chao@kernel.org> Reviewed-by: Zorro Lang <zlang@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Theodore Ts'o [Sun, 15 Dec 2024 05:12:42 +0000 (00:12 -0500)]
generic/530: only use xfs-specific mkfs options when testing on xfs
This fixes a regression introduced by commit 000813899afb ("fstests:
scale some tests for high CPU count sanity") where this test
unconditionally tried to use the mkfs option "-l size=256m" which
would break when testing any file sytem other than xfs.
Fix this the same way commit 000813899afb dealt with this for
generic/531; so this was just an oversight.
Fixes: 000813899afb ("fstests: scale some tests for high CPU count sanity") Signed-off-by: Theodore Ts'o <tytso@mit.edu> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Theodore Ts'o [Sun, 15 Dec 2024 05:12:41 +0000 (00:12 -0500)]
generic/135: don't try to rm $SCRATCH_MNT/*
This fixes a regression for ext4 introduced by commit 32e14cb90b88
("fstests: don't use directory stacks"), which replaced a number of
files at the top-level of the scratch file system:
async_file sync_file direct_file trunc_file
with "rm $SCRATCH_MNT/*". This causes a test failure on ext4 file
systems caused by a failed attempt to unlink the lost+found directory.
The thing, is these files are all super small (4k or 16k) and the
scratch file system is going to get reformatted before it gets used
again. So just dropping the delete is the simplest way to solve the
problem.
Fixes: 32e14cb90b88 ("fstests: don't use directory stacks") Signed-off-by: Theodore Ts'o <tytso@mit.edu> Reviewed-by: Christoph Hellwig <hch@lst.de> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Filipe Manana [Wed, 11 Dec 2024 10:55:25 +0000 (10:55 +0000)]
generic/590: fix test failure when running against fs other than xfs
With commit ce79de11337e ("fstests: clean up loop device instantiation")
we started to always call _destroy_loop_device at the very end of the
test, but we only create a loop device if we are running against xfs,
so the call to _destroy_loop_device results in an error since no loop
device was setup.
For example running this test against btrfs or ext4 results in this
failure:
generic/590 29s ... [failed, exit status 1]- output mismatch (see /home/fdmanana/git/hub/xfstests/results//generic/590.out.bad)
--- tests/generic/590.out 2020-06-10 19:29:03.858520038 +0100
+++ /home/fdmanana/git/hub/xfstests/results//generic/590.out.bad 2024-12-11 10:48:43.946205346 +0000
@@ -1,2 +1,5 @@
QA output created by 590
-Silence is golden
+losetup: option requires an argument -- 'd'
+Try 'losetup --help' for more information.
+Cannot destroy loop device
+(see /home/fdmanana/git/hub/xfstests/results//generic/590.full for details)
...
(Run 'diff -u /home/fdmanana/git/hub/xfstests/tests/generic/590.out /home/fdmanana/git/hub/xfstests/results//generic/590.out.bad' to see the entire diff)
Ran: generic/590
Failures: generic/590
Failed 1 of 1 tests
Fix this by removing the call to _destroy_loop_device at the end of the
test, as it's unnecessary because we call it in the _cleanup function if
we have setup a loop device.
Fixes: ce79de11337e ("fstests: clean up loop device instantiation") Signed-off-by: Filipe Manana <fdmanana@suse.com> Reviewed-by: Dave Chinner <dchinner@redhat.com> Reviewed-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Zorro Lang <zlang@kernel.org>
Filipe Manana [Tue, 10 Dec 2024 17:38:32 +0000 (17:38 +0000)]
generic/442: fix failure due to missing test number argument for fsync-err
After commit 88c0291d297c ("fstests: per-test dmerror instances") the
script src/dmerror now has an extra argument, corresponding to a test's
sequence number, but generic/442 isn't passing that argument so the test
fails like this:
Filipe Manana [Tue, 10 Dec 2024 17:34:14 +0000 (17:34 +0000)]
btrfs/146: fix failure due to missing test number argument for fsync-err
After commit 88c0291d297c ("fstests: per-test dmerror instances") the
script src/dmerror now has an extra argument, corresponding to a test's
sequence number, but btrfs/146 isn't passing that argument so the test
fails like this:
Filipe Manana [Tue, 10 Dec 2024 17:33:47 +0000 (17:33 +0000)]
btrfs/142, btrfs/143: fix dmdust device names
After commit aaa132777476 ("fstests: per-test dmdust instances") the
dmdust device names are no longer a plain 'dust-test', they now have
a suffix matching '.N', where N is the test's sequence number.
The test cases btrf/142 and btrfs/143 are still using the old device
names, so they fail. Fix this my making them refer to 'dust-test.$seq'
instead of 'dust-test'.
Fixes: aaa132777476 ("fstests: per-test dmdust instances") Signed-off-by: Filipe Manana <fdmanana@suse.com> Reviewed-by: Qu Wenruo <wqu@suse.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Filipe Manana [Tue, 10 Dec 2024 17:33:22 +0000 (17:33 +0000)]
btrfs/100, btrfs/101: fix device name in their golden output
After commit 88c0291d297c ("fstests: per-test dmerror instances"), the dm
error device name changed so that it's no longer a plain 'error-test' but
instead it's that plus a prefix matching the test number, such as
'error-test.100' for example. The tests btrfs/100 and btrfs/101 are still
using the plain old name 'error-test' in their golden output, which makes
them fail. So update them to use 'error-test.100' and 'error-test.101'.
Fixes: 88c0291d297c ("fstests: per-test dmerror instances") Signed-off-by: Filipe Manana <fdmanana@suse.com> Reviewed-by: Qu Wenruo <wqu@suse.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
btrfs/007 1s ... [failed, exit status 1]- output mismatch (see /home/fdmanana/git/hub/xfstests/results//btrfs/007.out.bad)
--- tests/btrfs/007.out 2020-06-10 19:29:03.810518987 +0100
+++ /home/fdmanana/git/hub/xfstests/results//btrfs/007.out.bad 2024-12-10 16:09:56.345937619 +0000
@@ -1,3 +1,4 @@
QA output created by 007
*** test send / receive
-*** done
+failed: '2097152000 200'
+(see /home/fdmanana/git/hub/xfstests/results//btrfs/007.full for details)
...
(Run 'diff -u /home/fdmanana/git/hub/xfstests/tests/btrfs/007.out /home/fdmanana/git/hub/xfstests/results//btrfs/007.out.bad' to see the entire diff)
Ran: btrfs/007
The problem comes from _run_fsstress and _run_fsstress_bg using $*, which
splits the string argument for the -x command used by btrfs/007, so that
fsstress gets the argument for -x as simply:
Christoph Hellwig [Tue, 10 Dec 2024 06:53:15 +0000 (07:53 +0100)]
common: call _require_scratch_dedupe from _require_scratch_duperemove
_require_scratch_duperemove doesn't check if the scratch file system
actually supports dedup, so add the proper call for that.
Signed-off-by: Christoph Hellwig <hch@lst.de> Tested-by: Chuck Lever <chuck.lever@oracle.com> Reviewed-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: Zorro Lang <zlang@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
Christoph Hellwig [Tue, 10 Dec 2024 07:03:11 +0000 (08:03 +0100)]
common: loop device work on zoned file systems
Remove the call to _require_non_zoned_device from _require_loop because
there is nothing that prevents loop device to work on file systems on
zoned devices. But loop devices are not supported directly on top of
zoned block devices, so add a _require_non_zoned_device to generic/563
to not run that test on zoned block devices.
Signed-off-by: Christoph Hellwig <hch@lst.de> Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com> Signed-off-by: Zorro Lang <zlang@redhat.com>
Dave Chinner [Wed, 27 Nov 2024 00:36:08 +0000 (11:36 +1100)]
fstests: check-parallel
Runs tests in parallel runner threads. Each runner thread has it's
own set of tests to run, and runs a separate instance of check
to run those tests.
check-parallel sets up loop devices, mount points, results
directories, etc for each instance and divides the tests up between
the runner threads.
It currently hard codes the XFS and generic test lists, and then
gives each check invocation an explicit list of tests to run. It
also passes through exclusions so that test exclude filtering is
still done by check.
This is far from ideal, but I didn't want to have to embark on a
major refactoring of check to be able to run stuff in parallel.
It was quite the challenge just to get all the tests and test
infrastructure up to the point where they can run reliably in
parallel.
Hence I've left the actual factoring of test selection and setup
out of the patchset for the moment. The plan is to factor both the
test setup and the test list runner loop out of check and share them
between check and check-parallel, hence not requiring check-parallel
to run check directly. That is future work, however.
With the current test runner setup, it is not uncommon to see >5000%
cpu usage, 150-200kiops and 4-5GB/s of disk bandwidth being used
when running 64 runners. This is a serious stress load as it is
constantly mounting and unmounting dozens of filesystems, creating
and destroying devices, dropping caches, running sync, running CPU
hot plug, running page cache migration, etc.
The massive amount of IO that load generates causes qemu hosts to
abort (i.e. crash) because they run out of vm map segments. Hence
bumping up the max_map_count on the host like so:
There is no significant memory pressure to speak of from running the
tests like this. I've seen a maximum of about 50GB of RAM used when
running tests like this, so running on a 64p/64GB VM the additional
concurrency doesn't really stress memory capacity like it does CPU
and IO.
All the runners are executed in private mount namespaces. This is
to prevent ephemeral mount namespace clones from taking a reference
to every mounted filesystem in the machine and so causing random
"device busy after unmount" failures in the tests that are running
concurrently with the mount namespace setup and teardown.
total 0
crw-------. 1 root root 10, 236 Nov 27 09:27 control
lrwxrwxrwx. 1 root root 7 Nov 27 09:27 fast -> ../dm-0
/dev/mapper/fast 1.4T 192G 1.2T 14% /mnt/xfs
real 9m29.056s
user 0m0.005s
sys 0m0.022s
$
Yeah, that runtime is real - under 10 minutes for a full XFS auto
group test run. When running this normally (i.e. via check) on this
machine, it usually takes just under 4 hours to run the same set
of tests. i.e. I can run ./check-parallel roughly 25x times on this
machine in the same time it takes to run ./check.
Signed-off-by: Dave Chinner <dchinner@redhat.com> Reviewed-by: Zorro lang <zlang@redhat.com> Signed-off-by: Zorro Lang <zlang@kernel.org>
projects / users / hch / xfstests-dev.git / log