fix _require_scratch_duperemove ordering

Zorro complained that generic/559 stopped working, and I noticed that
the duperemove invocation in the _require_scratch_duperemove function
would fail with:

 Error 2: No such file or directory while getting path to file /opt/file1. Skipping.
 Error 2: No such file or directory while getting path to file /opt/file2. Skipping.
 No dedupe candidates found.
 Gathering file list...

The cause of this is the incorrect placement of _require_scratch_dedupe
after a _scratch_mount.  _require_scratch_dedupe formats, mounts, tests,
and unmounts the scratch filesystem, which means that it should not come
between a _scratch_mount call and code that uses $SCRATCH_MNT.

Cc: <fstests@vger.kernel.org> # v2024.12.22
Fixes: 3b9f5fc7d7d853 ("common: call _require_scratch_dedupe from _require_scratch_duperemove")
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

fuzzy: port fsx and fsstress loop to use --duration

Quite a while ago, I added --duration= arguments to fsx and fsstress,
and apparently I forgot to update the scrub stress loops to use them.
Replace the usage of timeout(1) for the remount_period versions of the
loop to clean up that code; and convert the non-remount loop so that
they don't run over time.

Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

fuzzy: always stop the scrub fsstress loop on error

Always abort the non-remount scrub fsstress loop in
__stress_scrub_fsstress_loop if fsstress returns a nonzero exit code.

Cc: <fstests@vger.kernel.org> # v2023.01.15
Fixes: 20df87599f66d0 ("fuzzy: make scrub stress loop control more robust")
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

fuzzy: don't use readarray for xfsfind output

Some of the scrub stress tests (e.g. xfs/796) walk the directory tree to
find filepaths to scrub, and load the entire list of paths into a bash
array.  On a large filesystem or a long-running test this is hugely
wasteful of memory because we use each path exactly once.

Fix __stress_one_scrub_loop to avoid this by reading lines directly from
the output of the xfsfind utility.  However, we play some games with fd
77 so that the processes in the loop body will use the same stdin as the
test and /not/ the piped stdout of xfsfind.

To avoid read(1) becoming confused by newlines in the file paths, adapt
xfsfind to print nulls between pathnames, and the bash code to recognize
them.

This was a debugging patch while I was trying to figure out why xfs/286
and other scrub soak tests started OOMing after the v2024.12.08 changes,
though in the end the OOMs were the result of memory leaks in fsstress.

Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Signed-off-by: Zorro Lang <zlang@kernel.org>

fuzzy: stop __stress_scrub_fsx_loop if fsx fails

Stop the fsx scrub stress loop if fsx returns a nonzero error code.

Cc: <fstests@vger.kernel.org> # v2023.01.15
Fixes: a2056ca8917bc8 ("fuzzy: enhance scrub stress testing to use fsx")
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

generic/032: fix pinned mount failure

generic/032 now periodically fails with:

 --- /tmp/fstests/tests/generic/032.out 2025-01-05 11:42:14.427388698 -0800
 +++ /var/tmp/fstests/generic/032.out.bad 2025-01-06 18:20:17.122818195 -0800
 @@ -1,5 +1,7 @@
  QA output created by 032
  100 iterations
 -000000 cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd  >................<
 -*
 -100000
 +umount: /opt: target is busy.
 +mount: /opt: /dev/sda4 already mounted on /opt.
 +       dmesg(1) may have more information after failed mount system call.
 +cycle mount failed
 +(see /var/tmp/fstests/generic/032.full for details)

The root cause of this regression is the _syncloop subshell.  This
background process runs _scratch_sync, which is actually an xfs_io
process that calls syncfs on the scratch mount.

Unfortunately, while the test kills the _syncloop subshell, it doesn't
actually kill the xfs_io process.  If the xfs_io process is in D state
running the syncfs, it won't react to the signal, but it will pin the
mount.  Then the _scratch_cycle_mount fails because the mount is pinned.

Prior to commit 8973af00ec212f the _syncloop ran sync(1) which avoided
pinning the scratch filesystem.

Fix this by pgrepping for the xfs_io process and killing and waiting for
it if necessary.

Cc: <fstests@vger.kernel.org> # v2024.12.08
Fixes: 8973af00ec212f ("fstests: cleanup fsstress process management")
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

generic/650: revert SOAK DURATION changes

Prior to commit 8973af00ec21, in the absence of an explicit
SOAK_DURATION, this test would run 2500 fsstress operations each of ten
times through the loop body.  On the author's machines, this kept the
runtime to about 30s total.  Oddly, this was changed to 30s per loop
body with no specific justification in the middle of an fsstress process
management change.

On the author's machine, this explodes the runtime from ~30s to 420s.
Put things back the way they were.

Cc: <fstests@vger.kernel.org> # v2024.12.08
Fixes: 8973af00ec212f ("fstests: cleanup fsstress process management")
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

preamble: fix missing _kill_fsstress

Commit 8973af00ec212f added a _kill_fsstress to the standard _cleanup
function.  However, if something breaks during test program
initialization before it gets to sourcing common/rc, then you get
failures that look like this:

 --- /tmp/fstests/tests/generic/556.out  2024-09-25 12:09:52.938797554 -0700
 +++ /var/tmp/fstests/generic/556.out.bad        2025-01-04 22:34:01.268327003 -0800
 @@ -1,16 +1,3 @@
  QA output created by 556
 -SCRATCH_MNT/basic Casefold
 -SCRATCH_MNT/basic
 -SCRATCH_MNT/casefold_flag_removal Casefold
 -SCRATCH_MNT/casefold_flag_removal Casefold
 -SCRATCH_MNT/flag_inheritance/d1/d2/d3 Casefold
 -SCRATCH_MNT/symlink/ind1/TARGET
 -mv: 'SCRATCH_MNT/rename/rename' and 'SCRATCH_MNT/rename/RENAME' are the same file
 -# file: SCRATCH_MNT/xattrs/x
 -user.foo="bar"
 -
 -# file: SCRATCH_MNT/xattrs/x/f1
 -user.foo="bar"
 -
 -touch: 'SCRATCH_MNT/strict/corac'$'\314\247\303': Invalid argument
 -touch: 'SCRATCH_MNT/strict/cora'$'\303\247\303': Invalid argument
 +./tests/generic/556: 108: common/config: Syntax error: "&" unexpected
 +./tests/generic/556: 10: _kill_fsstress: not found

It's that last line that's unnecessary.  Fix this by checking for the
presence of a _kill_fsstress before invoking it.

Cc: <fstests@vger.kernel.org> # v2024.12.08
Fixes: 8973af00ec212f ("fstests: cleanup fsstress process management")
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

common/rc: return mount_ret in _try_scratch_mount

Make the _try_scratch_mount and _test_mount helpers return the exit code
from mount, not _prepare_for_eio_shutdown.

Cc: <fstests@vger.kernel.org> # v2024.12.08
Fixes: 1a49022fab9b4d ("fstests: always use fail-at-unmount semantics for XFS")
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

mkfs: don't hardcode log size

Commit 000813899afb46 hardcoded a log size of 256MB into xfs/501,
xfs/502, and generic/530.  This seems to be an attempt to reduce test
run times by increasing the log size so that more background threads can
run in parallel.  Unfortunately, this breaks a couple of my test
configurations:

 - External logs smaller than 256MB
 - Internal logs where the AG size is less than 256MB

For example, here's seqres.full from a failed xfs/501 invocation:

** mkfs failed with extra mkfs options added to " -m metadir=2,autofsck=1,uquota,gquota,pquota, -d rtinherit=1," by test 501 **
** attempting to mkfs using only test 501 options: -l size=256m **
size 256m specified for log subvolume is too large, maximum is 32768 blocks
<snip>
mount -ortdev=/dev/sdb4 -ologdev=/dev/sdb2 /dev/sda4 /opt failed
umount: /dev/sda4: not mounted.

Note that there's some formatting error here, so we jettison the entire
rt configuration to force the log size option, but then mount fails
because we didn't edit out the rtdev option there too.

Fortunately, mkfs.xfs already /has/ a few options to try to improve
parallelism in the filesystem by avoiding contention on the log grant
heads by scaling up the log size.  These options are aware of log and AG
size constraints so they won't conflict with other geometry options.

Use them.

Cc: <fstests@vger.kernel.org> # v2024.12.08
Fixes: 000813899afb46 ("fstests: scale some tests for high CPU count sanity")
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

unmount: resume logging of stdout and stderr for filtering

There's a number of places where a test program calls a variant of
_unmount but then pipes the output through a _filter script or
something.  The new _unmount helper redirects stdout and stderr to
seqres.full, which means that those error messages (some of which are
encoded in the golden outputs) are suppressed.  This leads to test
regressions in generic/050 and other places, so let's resume logging.

This also undoes all the changes that removed /dev/null redirection of
unmount calls.

Cc: <fstests@vger.kernel.org> # v2024.12.08
Fixes: 4c6bc4565105e6 ("fstests: clean up mount and unmount operations")
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

common/rc: don't copy fsstress to $TEST_DIR

Now that we can pkill only processes that were started by this test, we
don't need to copy the fsstress binary to $TEST_DIR to avoid killing the
wrong program instances.  This avoids a whole slew of ETXTBSY problems
with scrub stress tests that run multiple copies of fsstress in the
background.

Revert most of the changes to generic/270, because it wants to do
something fancy with the fsstress binary, so it needs to control the
process directly.

Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

check: deprecate using process sessions to isolate test instances

As I've noted elsewhere, the use of process session ids to "isolate"
test instances from killing each other is kind of hacky and creates
other weird side effects.  I'd rather everyone use the new code that
runs everything in proper isolation with private pid and mount
namespaces, but I don't know how many people this would break were it a
hard dependency.

Deprecate the process session handling immediately with a warning that
we're going to rip it out in a year.

Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

check: run tests in a private pid/mount namespace

As mentioned in the previous patch, trying to isolate processes from
separate test instances through the use of distinct Unix process
sessions is annoying due to the many complications with signal handling.

Instead, we could just use nsexec to run the test program with a private
pid namespace so that each test instance can only see its own processes;
and private mount namespace so that tests writing to /tmp cannot clobber
other tests or the stuff running on the main system.  Further, the
process created by the clone(CLONE_NEWPID) call is considered the init
process of that pid namespace, so all processes will be SIGKILL'd when
the init process terminates, so we no longer need systemd scopes for
externally enforced cleanup.

However, it's not guaranteed that a particular kernel has pid and mount
namespaces enabled.  Mount (2.4.19) and pid (2.6.24) namespaces have
been around for a long time, but there's no hard requirement for the
latter to be enabled in the kernel.  Therefore, this bugfix slips
namespace support in alongside the session id thing.

Declaring CONFIG_PID_NS=n a deprecated configuration and removing
support should be a separate conversation, not something that I have to
do in a bug fix to get mainline QA back up.

Note that the new helper cannot unmount the /proc it inherits before
mounting a pidns-specific /proc because generic/504 relies on being able
to read the init_pid_ns (aka systemwide) version of /proc/locks to find
a file lock that was taken and leaked by a process.

Cc: <fstests@vger.kernel.org> # v2024.12.08
Fixes: 8973af00ec212f ("fstests: cleanup fsstress process management")
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Zorro Lang <zlang@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

common: fix pkill by running test program in a separate session

Run each test program with a separate session id so that we can tell
pkill to kill all processes of a given name, but only within our own
session id.  This /should/ suffice to run multiple fstests on the same
machine without one instance shooting down processes of another
instance.

This fixes a general problem with using "pkill --parent" -- if the
process being targeted is not a direct descendant of the bash script
calling pkill, then pkill will not do anything.  The scrub stress tests
make use of multiple background subshells, which is how a ^C in the
parent process fails to result in fsx/fsstress being killed.

This is necessary to fix SOAK_DURATION runtime constraints for all the
scrub stress tests.  However, there is a cost -- the test program no
longer runs with the same controlling tty as ./check, which means that
^Z doesn't work and SIGINT/SIGQUIT are set to SIG_IGN.  IOWs, if a test
wants to kill its subprocesses, it must use another signal such as
SIGPIPE.  Fortunately, bash doesn't whine about children dying due to
fatal signals if the children run in a different session id.
Unfortunately we have to let it run the test as a background process for
bash to handle SIGINT, and SIGSTOP no longer works properly.

This solution is a bit crap, and I have a better solution for it in the
next patch that uses private pid and mount namespaces.  Unfortunately,
that solution adds new minimum requirements for running fstests and
removing previously supported configurations abruptly during a bug fix
is not appropriate behavior.

I also explored alternate designs, and this was the least unsatisfying:

a) Setting the process group didn't work because background subshells
are assigned a new group id.

b) Constraining the pkill/pgrep search to a cgroup could work, but it
seems that procps has only recently (~2023) gained the ability to filter
on a cgroup.  Furthermore, we'd have to set up a cgroup in which to run
the fstest.  The last decade has been rife with user bug reports
complaining about chaos resulting from multiple pieces of software (e.g.
Docker, systemd, etc.) deciding that they own the entire cgroup
structure without having any means to enforce that statement.  We should
not wade into that mess.

c) Putting test subprocesses in a systemd sub-scope and telling systemd
to kill the sub-scope could work because ./check can already use it to
ensure that all child processes of a test are killed.  However, this is
an *optional* feature, which means that we'd have to require systemd.

d) Constraining the pkill/pgrep search to a particular pid namespace
could work, but we already have tests that set up their own mount
namespaces, which means the constrained pgrep will not find all child
processes of a test.  Though this hasn't been born out through testing?

e) Constraining to any other type of namespace (uts, pid, etc) might not
work because those namespaces might not be enabled.  However, combining
a private pid and mount namespace to isolate tests from each other seems
to work better than session ids.  This is coming in a subsequent patch,
but to avoid breaking older systems, we will use this as an immediately
deprecated fallback.

f) Revert check-parallel and go back to one fstests instance per system.
Zorro already chose not to revert.

So.  Change _run_seq to create a the ./$seq process with a new session
id, update _su calls to use the same session as the parent test, update
all the pkill sites to use a wrapper so that we only target processes
created by *this* instance of fstests, and update SIGINT to SIGPIPE.

Cc: <fstests@vger.kernel.org> # v2024.12.08
Fixes: 8973af00ec212f ("fstests: cleanup fsstress process management")
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Zorro Lang <zlang@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

tools: add a Makefile

Add a Makefile to the tools/ directory so that we can put helper scripts
in there and have them installed at make install time.  The makefile
comes from a conversation that Zorro and I had over IRC, hence the RH
copyright.

Suggested-by: Zorro Lang <zlang@kernel.org>
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Reviewed-by: Zorro Lang <zlang@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

common/rc: hoist pkill to a helper function

Create a helper function to wrap pkill in preparation for the next
patch.

Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

fuzzy: kill subprocesses with SIGPIPE, not SIGINT

The next patch in this series fixes various issues with the recently
added fstests process isolation scheme by running each new process in a
separate process group session.  Unfortunately, the processes in the
session are created with SIGINT ignored by default because they are not
attached to the controlling terminal.  Therefore, switch the kill signal
to SIGPIPE because that is usually fatal and not masked by default.

Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

common/rc: create a wrapper for the su command

Create a _su wrapper around the su command so that the next patch can
fix all the pkill isolation code so that this test can only kill
processes started for this test.

Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

generic/759,760: skip test if we can't set up a hugepage for IO

On an arm64 VM with 64k base pages and a paltry 8G of RAM, this test
will frequently fail like this:

>  QA output created by 759
>  fsx -N 10000 -l 500000 -h
> -fsx -N 10000 -o 8192 -l 500000 -h
> -fsx -N 10000 -o 128000 -l 500000 -h
> +Seed set to 1
> +madvise collapse for buf: Cannot allocate memory
> +init_hugepages_buf failed for good_buf: Cannot allocate memory

This system has a 512MB hugepage size, which means that there's a good
chance that memory is so fragmented that we won't be able to create a
huge page (in 1/16th the available DRAM).  Create a _run_hugepage_fsx
helper that will detect this situation at the start of the test and skip
it, having refactored run_fsx into a properly namespaced version that
won't exit the test on failure.

Cc: <fstests@vger.kernel.org> # v2025.02.02
Cc: joannelkoong@gmail.com
Fixes: 627289232371e3 ("generic: add tests for read/writes from hugepages-backed buffers")
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Reviewed-by: Joanne Koong <joannelkoong@gmail.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

generic/759,760: fix MADV_COLLAPSE detection and inclusion

On systems with "old" C libraries such as glibc 2.36 in Debian 12, the
MADV_COLLAPSE flag might not be defined in any of the header files
pulled in by sys/mman.h, which means that the fsx binary might not get
built with any of the MADV_COLLAPSE code.  If the kernel supports THP,
the test will fail with:

>  QA output created by 760
>  fsx -N 10000 -l 500000 -r PSIZE -t BSIZE -w BSIZE -Z -R -W -h
> -fsx -N 10000 -o 8192 -l 500000 -r PSIZE -t BSIZE -w BSIZE -Z -R -W -h
> -fsx -N 10000 -o 128000 -l 500000 -r PSIZE -t BSIZE -w BSIZE -Z -R -W -h
> +mapped writes DISABLED
> +MADV_COLLAPSE not supported. Can't support -h

Fix both tests to detect fsx binaries that don't support MADV_COLLAPSE,
then fix fsx.c to include the mman.h from the kernel headers (aka
linux/mman.h) so that we can actually test IOs to and from THPs if the
kernel is newer than the rest of userspace.

Cc: <fstests@vger.kernel.org> # v2025.02.02
Cc: joannelkoong@gmail.com
Fixes: 627289232371e3 ("generic: add tests for read/writes from hugepages-backed buffers")
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Reviewed-by: Joanne Koong <joannelkoong@gmail.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

common/populate: correct the parent pointer name creation formulae

The formulae used to compute the number of parent pointers that we have
to create in a child file in order to generate a particular xattr
structure are not even close to correct -- the first one needs a bit of
adjustment, but the second one is way off and creates far too many
files.

Fix the computation, and document where the magic numbers come from.

Cc: <fstests@vger.kernel.org> # v2024.06.27
Fixes: 0c02207d61af9a ("populate: create hardlinks for parent pointers")
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Reviewed-by: Zorro Lang <zlang@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

common/dump: don't replace pids arbitrarily

In the next patch we'll run tests in a pid namespace, which means that
the test process will have a very low pid.  This low pid situation
causes problems with the dump tests because they unconditionally replace
it with the word "PID", which causes unnecessary test failures.

Initially I was going to fix it by bracketing the regexp with a
whitespace/punctuation/eol/sol detector, but then I decided to remove it
see how many sheep came barreling through.  None did, so I removed it
entirely.  The commit adding it (linked below) was not insightful at
all.

Fixes: 19beb54c96e363 ("Extra filtering as part of IRIX/Linux xfstests reconciliation for dump.")
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

common/rc: revert recursive unmount in _clear_mount_stack

Zorro complained about the following regression in generic/589 that I
can't reproduce here on Debian 12:

>  --- /dev/fd/63  2025-01-08 19:53:49.621421512 -0500
>  +++ generic/589.out.bad 2025-01-08 19:53:49.244099127 -0500
>  @@ -30,6 +30,7 @@
>   mpC SCRATCH_DEV
>   mpC SCRATCH_DEV
>   ======
>  +umount: /mnt/xfstests/test/589-dst: not mounted
>
> It fails on generic/589 -> end_test -> _clear_mount_stack
>
>   ...
>   + end_test
>   + _clear_mount_stack
>   + '[' -n '/mnt/test/589-dst/201227_mpC /mnt/test/589-src/201227_mpA /mnt/test/589-dst /mnt/test/589-dst /mnt/test/589-src' ']'
>   + _unmount -R /mnt/test/589-dst/201227_mpC /mnt/test/589-src/201227_mpA /mnt/test/589-dst /mnt/test/589-dst /mnt/test/589-src
>   + local outlog=/tmp/201227.201227.umount
>   + local errlog=/tmp/201227.201227.umount.err
>   + rm -f /tmp/201227.201227.umount /tmp/201227.201227.umount.err
>   + /usr/bin/umount -R /mnt/test/589-dst/201227_mpC /mnt/test/589-src/201227_mpA /mnt/test/589-dst /mnt/test/589-dst /mnt/test/589-src
>   + local res=1
>   + '[' -s /tmp/201227.201227.umount ']'
>   + '[' -s /tmp/201227.201227.umount.err ']'
>   + cat /tmp/201227.201227.umount.err
>   + cat /tmp/201227.201227.umount.err
>   umount: /mnt/test/589-dst: not mounted
>
> The _get_mount already save all mount points into MOUNTED_POINT_STACK,
> MOUNTED_POINT_STACK="/mnt/test/589-dst/201227_mpC /mnt/test/589-src/201227_mpA /mnt/test/589-dst /mnt/test/589-dst /mnt/test/589-src"
>
> '-/mnt/test                                     /dev/sda5                                xfs        rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=64k,sunit=128,swidth=256,noquota
>   |-/mnt/test/589-src                           /dev/sda6                                xfs        rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=64k,sunit=128,swidth=256,noquota
>   | '-/mnt/test/589-src/223262_mpA              /dev/sda6                                xfs        rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=64k,sunit=128,swidth=256,noquota
>   '-/mnt/test/589-dst                           /dev/sda6                                xfs        rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=64k,sunit=128,swidth=256,noquota
>     |-/mnt/test/589-dst                         /dev/sda6                                xfs        rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=64k,sunit=128,swidth=256,noquota
>     | '-/mnt/test/589-dst/223262_mpC            /dev/sda6                                xfs        rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=64k,sunit=128,swidth=256,noquota
>     '-/mnt/test/589-dst/223262_mpC              /dev/sda6                                xfs        rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=64k,sunit=128,swidth=256,noquota
>
> Orignal _clear_mount_stack trys to umount all of them. But Dave gave -R option to
> umount command, so
>   `umount -R /mnt/test/589-dst/201227_mpC` and `umount -R /mnt/test/589-src/201227_mpA`
> already umount /mnt/test/589-dst and /mnt/test/589-src. recursively.
> Then `umount -R /mnt/test/589-dst` shows "not mounted".

I /think/ this is a result of commit 4c6bc4565105e6 performing this
"conversion" in _clear_mount_stack:

- $UMOUNT_PROG $MOUNTED_POINT_STACK
+ _unmount -R $MOUNTED_POINT_STACK

This is not a 1:1 conversion here -- previously there was no
-R(ecursive) unmount option, and now there is.  It looks as though
umount parses /proc/self/mountinfo to figure out what to unmount, and
maybe on Zorro's system it balks if the argument passed is not present
in that file?  Debian 12's umount doesn't care.

Regardless, there was no justification for this change in behavior that
was buried in what is otherwise a hoist patch, so revert it.  The author
can resubmit the change with proper documentation.

Cc: <fstests@vger.kernel.org> # v2024.12.08
Fixes: 4c6bc4565105e6 ("fstests: clean up mount and unmount operations")
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

fuzzy: do not set _FSSTRESS_PID when exercising fsx

If we're not running fsstress as the scrub exerciser, don't set
_FSSTRESS_PID because the _kill_fsstress call in the cleanup function
will think that it has to wait for a nonexistant fsstress process.
This fixes the problem of xfs/565 runtime increasing from 30s to 800s
because it tries to kill a nonexistent "565.fsstress" process and then
waits for the fsx loop control process, which hasn't been sent any
signals.

Cc: <fstests@vger.kernel.org> # v2024.12.08
Fixes: 8973af00ec212f ("fstests: cleanup fsstress process management")
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

generic/019: don't fail if fio crashes while shutting down

My system (Debian 12) has fio 3.33.  Once in a while, fio crashes while
shutting down after it receives a SIGBUS on account of the filesystem
going down.  This causes the test to fail with:

generic/019       - output mismatch (see /var/tmp/fstests/generic/019.out.bad)
    --- tests/generic/019.out   2024-02-28 16:20:24.130889521 -0800
    +++ /var/tmp/fstests/generic/019.out.bad    2025-01-03 15:00:35.903564431 -0800
    @@ -5,5 +5,6 @@

     Start fio..
     Force SCRATCH_DEV device failure
    +/tmp/fstests/tests/generic/019: line 112: 90841 Segmentation fault      $FIO_PROG $fio_config >> $seqres.full 2>&1
     Make SCRATCH_DEV device operable again
     Disallow global fail_make_request feature
    ...
    (Run 'diff -u /tmp/fstests/tests/generic/019.out /var/tmp/fstests/generic/019.out.bad'  to see the entire diff)

because the wait command will dutifully report fatal signals that kill
the fio process.  Unfortunately, a core dump shows that we blew up in
some library's exit handler somewhere:

(gdb) where
#0  unlink_chunk (p=p@entry=0x55b31cb9a430, av=0x7f8b4475ec60 <main_arena>) at ./malloc/malloc.c:1628
#1  0x00007f8b446222ff in _int_free (av=0x7f8b4475ec60 <main_arena>, p=0x55b31cb9a430, have_lock=<optimized out>, have_lock@entry=0) at ./malloc/malloc.c:4603
#2  0x00007f8b44624f1f in __GI___libc_free (mem=<optimized out>) at ./malloc/malloc.c:3385
#3  0x00007f8b3a71cf0e in ?? () from /lib/x86_64-linux-gnu/libtasn1.so.6
#4  0x00007f8b4426447c in ?? () from /lib/x86_64-linux-gnu/libgnutls.so.30
#5  0x00007f8b4542212a in _dl_call_fini (closure_map=closure_map@entry=0x7f8b44465620) at ./elf/dl-call_fini.c:43
#6  0x00007f8b4542581e in _dl_fini () at ./elf/dl-fini.c:114
#7  0x00007f8b445ca55d in __run_exit_handlers (status=0, listp=0x7f8b4475e820 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true)
    at ./stdlib/exit.c:116
#8  0x00007f8b445ca69a in __GI_exit (status=<optimized out>) at ./stdlib/exit.c:146
#9  0x00007f8b445b3251 in __libc_start_call_main (main=main@entry=0x55b319278e10 <main>, argc=argc@entry=2, argv=argv@entry=0x7ffec6f8b468) at ../sysdeps/nptl/libc_start_call_main.h:74
#10 0x00007f8b445b3305 in __libc_start_main_impl (main=0x55b319278e10 <main>, argc=2, argv=0x7ffec6f8b468, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>,
    stack_end=0x7ffec6f8b458) at ../csu/libc-start.c:360
#11 0x000055b319278ed1 in _start ()

This isn't a filesystem failure, so mask this by shovelling the output
to seqres.full.

Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

metadump: fix cleanup for v1 metadump testing

In commit ce79de11337e38, the metadump v2 tests were updated to leave
the names of loop devices in some global variables so that the cleanup
method can find them and remove the loop devices.  Inexplicably, the
metadump v1 test function was not upgraded.  Do so now.

Cc: <fstests@vger.kernel.org> # v2024.12.08
Fixes: ce79de11337e38 ("fstests: clean up loop device instantiation")
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

metadump: make non-local function variables more obvious

In _xfs_verify_metadump_v2(), we want to set up some loop devices,
record the names of those loop devices, and then leave the variables in
the global namespace so that _xfs_cleanup_verify_metadump can dispose of
them.

Elsewhere in fstests the convention for global variables is to put them
in all caps to make it obvious that they're global and not local
variables, so do that here too.

Cc: <fstests@vger.kernel.org> # v2024.12.08
Fixes: ce79de11337e38 ("fstests: clean up loop device instantiation")
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

generic/476: fix fsstress process management

generic/476 never used to run fsstress in the background, but
8973af00ec212f made it do that.  This is incorrect, because now 476 runs
for three seconds (i.e. long enough to fall out the bottom of the test
and end up in _cleanup), ignoring any SOAK_DURATION/TIME_FACTOR
settings.

Cc: <fstests@vger.kernel.org> # v2024.12.08
Fixes: 8973af00ec212f ("fstests: cleanup fsstress process management")
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

btrfs: skip tests that exercise compression property when using nodatasum

A couple tests exercise the compression property and that fails when an
inode has the nodatasum flag set. So skip the tests when running under the
nodatasum mount option.

Signed-off-by: Filipe Manana <fdmanana@suse.com>
Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
Reviewed-by: David Sterba <dsterba@suse.com>
Reviewed-by: Qu Wenruo <wqu@suse.com>
Reviewed-by: Anand Jain <anand.jain@oracle.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

btrfs/281: skip test when running with nodatasum mount option

The test exercises compression and compression doesn't happen on inodes
with checksums disabled (nodatasum), making the test fail the expectations
if getting compressed extents. So skip the test if nodatasum is present.

Signed-off-by: Filipe Manana <fdmanana@suse.com>
Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
Reviewed-by: David Sterba <dsterba@suse.com>
Reviewed-by: Qu Wenruo <wqu@suse.com>
Reviewed-by: Anand Jain <anand.jain@oracle.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

btrfs: skip tests exercising data corruption and repair when using nodatasum

Several tests exercise corrupting data and then checking that on read the
data is repaired, but this requires using checksums, so the tests fail
when running with the nodatasum mount option.

So add a _require_btrfs_no_nodatasum call to these tests.

Signed-off-by: Filipe Manana <fdmanana@suse.com>
Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
Reviewed-by: Qu Wenruo <wqu@suse.com>
Reviewed-by: Anand Jain <anand.jain@oracle.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

btrfs/205: skip test when running with nodatasum mount option

Currently the test fails when we pass "-o nodatasum" in MOUNT_OPTIONS and
the reason is because we enable compression, with "chattr +c", on a file
and then try to clone from it to a file with nodatasum inherited from the
mount options, which results in the clone ioctl to fail with -EINVAL since
it's not possible to clone from datasum to nodatasum and vice-versa.

So skip the test if nodatasum is a mount option.

Signed-off-by: Filipe Manana <fdmanana@suse.com>
Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
Reviewed-by: Qu Wenruo <wqu@suse.com>
Reviewed-by: Anand Jain <anand.jain@oracle.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

btrfs/333: skip the test when running with nodatacow or nodatasum

Encoded writes are reject for inodes with the NODATASUM flag, so we must skip
the test if running with either the nodatasum or nodatacow (which implies
nodatasum) mount options.

So skip the test when under nodatacow and nodatasum mount options.

Signed-off-by: Filipe Manana <fdmanana@suse.com>
Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
Reviewed-by: Qu Wenruo <wqu@suse.com>
Reviewed-by: Anand Jain <anand.jain@oracle.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

common/btrfs: add a _require_btrfs_no_nodatasum helper

Add a _require_btrfs_no_nodatasum helper to skip a test if the nodatasum
mount option is give, as we do have several tests that fail, for several
reasons, when that mount option is passed.

Signed-off-by: Filipe Manana <fdmanana@suse.com>
Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
Reviewed-by: Qu Wenruo <wqu@suse.com>
Reviewed-by: Anand Jain <anand.jain@oracle.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

btrfs/290: skip test if we are running with nodatacow mount option

We exercise corrupting an inline extent and inline extents can't be created
with nodatacow, we get instead a regular file extent item and if we attempt
to corrupt its disk_bytenr field with btrfs-corrupt-block we fail tree-checker
validation at mount time resulting in failure to mount and the following in
dmesg:

[514127.759739] BTRFS critical (device sdc): corrupt leaf: root=5 \
        block=30408704 slot=8 ino=257 file_offset=0, invalid disk_bytenr for \
        file extent, have 7416089308958521981, should be aligned to 4096
[514127.762715] BTRFS error (device sdc): read time tree block corruption \
        detected on logical 30408704 mirror 1

So add a _require_btrfs_no_nodatacow call to the test.

Signed-off-by: Filipe Manana <fdmanana@suse.com>
Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
Reviewed-by: Qu Wenruo <wqu@suse.com>
Reviewed-by: Anand Jain <anand.jain@oracle.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

btrfs: skip tests incompatible with compression when compression is enabled

We have several tests that fail when compression is enabled (MOUNT_OPTIONS
has "-o compress" or "-o compress-force"") because they expect a fixed
extent size and they trigger corruption by writing directly to a device,
therefore making them incompatible with compression.

So add a _require_no_compress call to them.

Signed-off-by: Filipe Manana <fdmanana@suse.com>
Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
Reviewed-by: Qu Wenruo <wqu@suse.com>
Reviewed-by: Anand Jain <anand.jain@oracle.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

generic: suggest fs specific fix only if the tested filesystem matches

It's odd when a test fails on a filesystem and a specific fix is suggested
for another filesystem. Some generic tests are suggesting filesystem
specific fixes without checking if the running filesystem matches, so
update them.

Signed-off-by: Filipe Manana <fdmanana@suse.com>
Reviewed-by: Zorro Lang <zlang@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

fstests: add a generic test to verify direct IO writes with buffer contents change

There is a long existing btrfs problem that if some one is modifying the
buffer of an on-going direct IO write, it has a very high chance causing
permanent data checksum mismatch.

This is caused by the following factors:

- Direct IO buffer is out of the control of filesystem
  Thus user space can modify the contents during writeback.

- Btrfs generate its data checksum just before submitting the bio
  This means if the contents happens after the checksum is generated,
  the data written to disk will no longer match the checksum.

Btrfs later fixes the problem by forcing the direct IO to fallback to
buffered IO (if the inode requires data checksum), so that btrfs can
have a consistent view of the buffer.

This test case will verify the behavior by:

- Create a helper program 'dio-writeback-race'
  Which does direct IO writes block-by-block, and the buffer is always
  initialized to all 0xff before write,
  Then starting two threads:
  - One to submit the direct IO
  - One to modify the buffer to 0x00

  The program uses 4K as default block size, and 64MiB as the default
  file size.
  Which is more than enough to trigger tons of btrfs checksum errors
  on unpatched kernels.

- New test case generic/761
  Which will:

  * Use above program to create a 64MiB file

  * Do buffered read on that file
    Since above program is doing direct IO, there is no page cache
    populated.
    And the buffered read will need to read out all data from the disk,
    and if the filesystem supports data checksum, then the data checksum
    will also be verified against the data.

The test case passes on the following fses:
- ext4
- xfs
- btrfs with "nodatasum" mount option
  No data checksum to bother.

- btrfs with default "datasum" mount option and the fix "btrfs: always
  fallback to buffered write if the inode requires checksum"
  This makes btrfs to fallback on buffered IO so the contents won't
  change during writeback of page cache.

And fails on the following fses:

- btrfs with default "datasum" mount option and without the fix
  Expected.

Suggested-by: Christoph Hellwig <hch@infradead.org>
Reviewed-by: David Disseldorp <ddiss@suse.de>
Reviewed-by: Zorro Lang <zlang@redhat.com>
Reviewed-by: "Darrick J. Wong" <djwong@kernel.org>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

fscrypt-crypt-util: fix KDF contexts for SM8650

Update the KDF contexts to match those actually used on SM8650.  This
turns out to be needed for the hardware-wrapped key tests generic/368
and generic/369 to pass on the SM8650 HDK (now that I have one to
actually test it).  Apparently the contexts changed between the
prototype version I tested a couple years ago and the final version.

Signed-off-by: Eric Biggers <ebiggers@google.com>
Reviewed-by: Zorro Lang <zlang@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

xfs/614: remove the _require_loop call

This test only creates file system images as regular files, but never
actually uses the kernel loop driver.  Remove the extra requirement.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Darrick J. Wong <djwong@kernel.org>
Signed-off-by: Zorro Lang <zlang@kernel.org>

fstests: btrfs/226: use nodatasum mount option to prevent false alerts

[BUG]
With recent kernel patch "btrfs: always fallback to buffered write if the
inode requires checksum", the test case btrfs/226 will fail with the
following error:

FSTYP         -- btrfs
PLATFORM      -- Linux/x86_64 btrfs-vm 6.13.0-rc6-custom+ #209 SMP PREEMPT_DYNAMIC Fri Jan 24 17:23:03 ACDT 2025
MKFS_OPTIONS  -- /dev/mapper/test-scratch1
MOUNT_OPTIONS -- /dev/mapper/test-scratch1 /mnt/scratch

btrfs/226 1s ... - output mismatch (see /home/adam/xfstests/results//btrfs/226.out.bad)
    --- tests/btrfs/226.out 2024-04-12 14:04:03.080000035 +0930
    +++ /home/adam/xfstests/results//btrfs/226.out.bad 2025-02-06 08:23:42.564298585 +1030
    @@ -39,14 +39,11 @@
     Testing write against prealloc extent at eof
     wrote 65536/65536 bytes at offset 0
     XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
    -wrote 65536/65536 bytes at offset 65536
    -XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
    +pwrite: Resource temporarily unavailable
     File after write:
    ...
    (Run 'diff -u /home/adam/xfstests/tests/btrfs/226.out /home/adam/xfstests/results//btrfs/226.out.bad'  to see the entire diff)
Ran: btrfs/226
Failures: btrfs/226
Failed 1 of 1 tests

[CAUSE]
That kernel patch makes btrfs to always fallback to buffered IO if the
target inode requires data checksum.

This is to avoid more deadly problems of mismatched data checksum.

But this also means, for inodes with data checksum, RWF_NOWAIT will
always fail, because we will wait writing back the page cache, thus
breaking the RWF_NOWAIT requirement.

[FIX]
Update the test case to utilize nodatasum mount option, so that the
direct-IO will not fallback to buffered ones unconditionally.

Reported-by: Filipe Manana <fdmanana@kernel.org>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Reviewed-by: Filipe Manana <fdmanana@suse.com>
Reviewed-by: Anand Jain <anand.jain@oracle.com>

common: fix a spelling error in _require_zoned_device

Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Darrick J. Wong <djwong@kernel.org>
Reviewed-by: Zorro Lang <zlang@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

check: Fix fs specfic imports when $FSTYPE!=$OLD_FSTYPE

Bug Description:

_test_mount function is failing with the following error:
./common/rc: line 4716: _xfs_prepare_for_eio_shutdown: command not found
check: failed to mount /dev/loop0 on /mnt1/test

when the second section in local.config file is xfs and the first section
is non-xfs.

It can be easily reproduced with the following local.config file

[s2]
export FSTYP=ext4
export TEST_DEV=/dev/loop0
export TEST_DIR=/mnt1/test
export SCRATCH_DEV=/dev/loop1
export SCRATCH_MNT=/mnt1/scratch

[s1]
export FSTYP=xfs
export TEST_DEV=/dev/loop0
export TEST_DIR=/mnt1/test
export SCRATCH_DEV=/dev/loop1
export SCRATCH_MNT=/mnt1/scratch

./check selftest/001

Root cause:
When _test_mount() is executed for the second section, the FSTYPE has
already changed but the new fs specific common/$FSTYP has not yet
been done. Hence _xfs_prepare_for_eio_shutdown() is not found and
the test run fails.

Fix:
Remove the additional _test_mount in check file just before ". common/rc"
since ". common/rc" is already sourcing fs specific imports and doing a
_test_mount.

Fixes: 1a49022fab9b4 ("fstests: always use fail-at-unmount semantics for XFS")
Signed-off-by: Nirjhar Roy (IBM) <nirjhar.roy.lists@gmail.com>
Reviewed-by: Darrick J. Wong <djwong@kernel.org>
Reviewed-by: Zorro Lang <zlang@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

fix vfs/utils.c for big-endian systems

generic/633 was failing with EINVAL on the fsxgetattr call on s390.
Looks like this is due to a failure to properly endian swap the
arguments to the syscall, so fix that, and the magic_etc compare
in expected_dummy_vfs_caps_uid() as well while we're at it.

Fixes: 0d1af68e ("generic: add fstests for idmapped mounts")
Signed-off-by: Eric Sandeen <sandeen@redhat.com>
Reviewed-by: Darrick J. Wong <djwong@kernel.org>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Zorro Lang <zlang@kernel.org>

common: skip tests using LVM when the device is no known

LVM has a lot of elaborate code to make the users life painful.  This
includes claiming the device type is unknown if it doesn't match it's
elaborately crafted internal list instead of just letting the user use
it.  Skip tests using LVM if this is the case to avoid arcane failures
due to missing lvm devices when using null_blk.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Darrick J. Wong <djwong@kernel.org>
Signed-off-by: Zorro Lang <zlang@kernel.org>

generic/418: use min_dio_alignment

Otherwise this test uses the wrong sector size on the RT device.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Darrick J. Wong <djwong@kernel.org>
Signed-off-by: Zorro Lang <zlang@kernel.org>

replace _supported_fs with _exclude_fs

Tests don't require a list of supported file systems, as that is deducted
from the test directory name.  Instead we exclude specific file systems
from a few common tests, and specify which of ext2 and ext3 should
actually also be tested after oddly multiplexing them into the ext4
directory full of tests only working for ext4.

Replace _supported_fs with a new _exclude_fs that takes only a single
file systems as the argument, making it easier to explain why the file
system is not supported for the common test.  For ext4 this increases
the existing mess even further, but the maintainers have a plan to
move it to feature checks instead that are hopefully easier to
understand.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Darrick J. Wong <djwong@kernel.org> # common and tests/generic
Acked-by: Darrick J. Wong <djwong@kernel.org> # tests/ext4
Signed-off-by: Zorro Lang <zlang@kernel.org>

common: remove the $FSYP check in _cleanup_dump

Despite the comment, _cleanup_dump is only called from xfs specific
tests, so this is superfluous.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Darrick J. Wong <djwong@kernel.org>
Signed-off-by: Zorro Lang <zlang@kernel.org>

generic/363: remove _supported_fs xfs

Run this test for all file systems.  Just because they are broken doesn't
mean that zeroing should not be tested.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Brian Foster <bfoster@redhat.com>
Reviewed-by: Darrick J. Wong <djwong@kernel.org>
Signed-off-by: Zorro Lang <zlang@kernel.org>

xfs/541: _notrun if the file system can't mount

A file system created without an RT section might not be able to mount
with an rtdev specified if the RT device has a larger LBA size.

Instead of letting the test fail, _notrun it for that case.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: "Darrick J. Wong" <djwong@kernel.org>
Signed-off-by: Zorro Lang <zlang@kernel.org>

generic: add tests for read/writes from hugepages-backed buffers

Add generic tests 758 and 759 for testing reads/writes from buffers
backed by hugepages.

Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
Reviewed-by: Brian Foster <bfoster@redhat.com>
Reviewed-by: Darrick J. Wong <djwong@kernel.org>
Signed-off-by: Zorro Lang <zlang@kernel.org>

fsx: support reads/writes from buffers backed by hugepages

Add support for reads/writes from buffers backed by hugepages.
This can be enabled through the '-h' flag. This flag should only be used
on systems where THP capabilities are enabled.

This is motivated by a recent bug that was due to faulty handling of
userspace buffers backed by hugepages. This patch is a mitigation
against problems like this in the future.

Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
Reviewed-by: Brian Foster <bfoster@redhat.com>
Reviewed-by: Darrick J. Wong <djwong@kernel.org>
Signed-off-by: Zorro Lang <zlang@kernel.org>

xfs/032: fix test failure on kernels which don't support bs > ps

When trying to mount a file system with a block size > page size on
kernel which doesn't support this, suppress the error messages from
showing up in the output file lest it cause test failures.

Fixes: 0b66f6efd669 ("xfs/032: try running on blocksize > pagesize filesystems")
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Reviewed-by: Zorro Lang <zlang@redhat.com>
Reviewed-by: "Darrick J. Wong" <djwong@kernel.org>
Signed-off-by: Zorro Lang <zlang@kernel.org>

generic: test swap activation on file that used to have clones

Test that we are able to activate a swap file on a file that used to have
its extents shared multiple times.

This exercises a bug on btrfs' extent sharedness detection during swap
file activation, which is fixed by the following kernel commit:

  03018e5d8508 ("btrfs: fix swap file activation failure due to extents that used to be shared")

The fails sporadically on XFS and the bug was already reported to the XFS
mailing list:

   https://lore.kernel.org/linux-xfs/CAL3q7H7cURmnkJfUUx44HM3q=xKmqHb80eRdisErD_x8rU4+0Q@mail.gmail.com/

   https://lore.kernel.org/fstests/dca49a16a7aacdab831b8895bdecbbb52c0e609c.1733928765.git.fdmanana@suse.com/

So for now skip the test on XFS and add comments with references to these
threads.

Signed-off-by: Filipe Manana <fdmanana@suse.com>
Reviewed-by: Zorro Lang <zlang@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

fsx: fix compile error for preadv2()

Commit d6b9d8eff076 ("fsx: add support for RWF_DONTCACHE") introduced
preadv2() calls in ltp/fsx.c. However, sys/uio.h is not included to the
source file, which causes compile errors with the gcc option
-Werror-implicit-function-declaration:

fsx.c: In function 'test_dontcache_io':
fsx.c:1956:15: error: implicit declaration of function 'preadv2'; did you mean 'pread64'? [-Wimplicit-function-declaration]
 1956 |         ret = preadv2(fd, &iov, 1, 0, RWF_DONTCACHE);
      |               ^~~~~~~
      |               pread64
fsx.c: In function 'fsx_rw':
fsx.c:2836:31: error: implicit declaration of function 'pwritev2'; did you mean 'pwrite64'? [-Wimplicit-function-declaration]
 2836 |                         ret = pwritev2(fd, &iov, 1, offset, flags);
      |                               ^~~~~~~~
      |                               pwrite64

To fix it, add the include directive.

Fixes: d6b9d8eff076 ("fsx: add support for RWF_DONTCACHE")
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
Reviewed-by: Jens Axboe <axboe@kernel.dk>
Reviewed-by: Zorro Lang <zlang@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

btrfs: test cycle mounting a filesystem right after enabling simple quotas

Test that if we enable simple quotas on a filesystem and unmount it right
after without doing any other changes to the filesystem, we are able to
mount again the filesystem.

This is a regression test for the following kernel commit:

  f2363e6fcc79 ("btrfs: fix transaction atomicity bug when enabling simple quotas")

Signed-off-by: Filipe Manana <fdmanana@suse.com>
Reviewed-by: Qu Wenruo <wqu@suse.com>
Reviewed-by: Anand Jain <anand.jain@oracle.com>
Signed-off-by: Anand Jain <anand.jain@oracle.com>

btrfs/326: update _fixed_by_kernel_commit

The original fix is already in the upstream, meanwhile a new but much
harder to hit bug is also exposed by this test case.

Add a new _fixed_by_kernel_commit for that bug, and special thanks to
Christian for pointing out the fix.

Cc: Christian Brauner <brauner@kernel.org>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Reviewed-by: David Sterba <dsterba@suse.com>
Reviewed-by: Anand Jain <anand.jain@oracle.com>
Signed-off-by: Anand Jain <anand.jain@oracle.com>

btrfs: add test for encoded reads

Add btrfs/333 and its helper programs btrfs_encoded_read and
btrfs_encoded_write, in order to test encoded reads.

We use the BTRFS_IOC_ENCODED_WRITE ioctl to write random data into a
compressed extent, then use the BTRFS_IOC_ENCODED_READ ioctl to check
that it matches what we've written. If the new io_uring interface for
encoded reads is supported, we also check that that matches the ioctl.

Note that what we write isn't valid compressed data, so any non-encoded
reads on these files will fail.

Reviewed-by: Anand Jain <anand.jain@oracle.com>
Signed-off-by: Mark Harmstone <maharmstone@fb.com>
Signed-off-by: Anand Jain <anand.jain@oracle.com>

configure: use pkg-config to find liburing

Change our autoconf macros so that instead of checking for the presence
of liburing.h, we use pkg-config.

The benefit of this is that we can then check the version of liburing,
and do conditional compilation based on this. There's a macro
IO_URING_CHECK_VERSION already, but it's only in relatively recent
versions of liburing.h.

This replaces HAVE_URING_H, defined by AC_CHECK_HEADERS, with
HAVE_URING. I also had to rename PKG_{MAJOR,MINOR,REVISION,BUILD} to
start with PACKAGE_, to avoid "possibly undefined macro" errors; it
looks like pkg-config assumes that anything called PKG_* is for its own
use.

Signed-off-by: Mark Harmstone <maharmstone@fb.com>
Reviewed-by: Josef Bacik <josef@toxicpanda.com>
Reviewed-by: Anand Jain <anand.jain@oracle.com>
Signed-off-by: Anand Jain <anand.jain@oracle.com>

common/quota: filter out option projquota in _qmount_option for ocfs2

ocfs2 doesn't support projquota but does support usrquota an grpquota.
For now, two tests generic/594 and generic/603 are for testing
usrquota,grpquota and projquota. The mount option 'projquota' causes
failure of ocfs2 mount.

To make things simple, just skip the tests for ocfs2.
However, we can't just put '_require_prjquota $SCRATCH_DEV' before
_qmount because f2fs and xfs need runtime after mount of SCRATCH_DEV.

For ocfs2, filter out option 'projquota' in _qmount_option() to make
_qmount successful. Then in _require_prjquota(), ocfs2 will fallthrough
as a no kernel projquota support fs type.

Signed-off-by: Su Yue <glass.su@suse.com>
Reviewed-by: David Disseldorp <ddiss@suse.de>
Signed-off-by: Zorro Lang <zlang@kernel.org>

common/rc: add ocfs2 supported timesptamp range

ocfs2 supports timestamp ranging from s64min to s64max.
Add it to _filesystem_timestamp_range then generic/402 runs.

Signed-off-by: Su Yue <glass.su@suse.com>
Reviewed-by: David Disseldorp <ddiss@suse.de>
Signed-off-by: Zorro Lang <zlang@kernel.org>

generic: add a partial pages zeroing out test

This addresses a data corruption issue encountered during partial page
zeroing in ext4 which the block size is smaller than the page size [1].
Add a new test which is expanded upon generic/567, this test performs a
zeroing range test that spans two partial pages to cover this case, and
also generalize it to work for non-4k page sizes.

Link: https://lore.kernel.org/linux-ext4/20241220011637.1157197-2-yi.zhang@huaweicloud.com/
Signed-off-by: Zhang Yi <yi.zhang@huawei.com>
Reviewed-by: Darrick J. Wong <djwong@kernel.org>
Reviewed-by: Ojaswin Mujoo <ojaswin@linux.ibm.com>
Reviewed-by: Zorro Lang <zlang@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

fsx: add support for RWF_DONTCACHE

Using RWF_DONTCACHE tells the kernel that any page cache instantiated
by this operation should get pruned once the operation completes. If
data is in cache prior to the operation it will remain there.

Add ops for testing both the read and write side of this. At startup,
kernel support for this feature is probed. If support isn't available,
uncached/dontcache IO is performed as regular buffered IO. If -Z is
used to turn on O_DIRECT, then uncached/dontcache IO isn't performed.
Defaults to on if available, and adds a -T parameter to turn it off.

See the kernel posting adding support:

https://lore.kernel.org/linux-fsdevel/20241220154831.1086649-1-axboe@kernel.dk/

Signed-off-by: Jens Axboe <axboe@kernel.dk>
Reviewed-by: Darrick J. Wong <djwong@kernel.org>
Signed-off-by: Zorro Lang <zlang@kernel.org>

fsstress: add support for RWF_DONTCACHE

Using RWF_DONTCACHE tells the kernel that any page cache instantiated
by this operation should get pruned once the operation completes. If
data is in cache prior to the operation it will remain there.

Add ops for testing both the read and write side of this. If the kernel
being tested doesn't support RWF_DONTCACHE, then operations are performed
with regular read/write buffered IO.

See the kernel posting adding support:

https://lore.kernel.org/linux-fsdevel/20241220154831.1086649-1-axboe@kernel.dk/

Signed-off-by: Jens Axboe <axboe@kernel.dk>
Reviewed-by: Darrick J. Wong <djwong@kernel.org>
Signed-off-by: Zorro Lang <zlang@kernel.org>

xfs/43[4-6]: implement impatient module reloading

These three tests try to reload the xfs module as a cheap way to detect
leaked inode and dquot objects when the slabs for those object are torn
down during rmmod.  Removal might not succeed, and we don't really care
for that case because we still want to exercise the log recovery code.

However, if (say) the root filesystem is xfs, then removal will never
succeed.  There's no way that waiting 50 seconds(!) per test is going
to change that.  Add a silly helper to do it fast or go home.

Reported-by: sandeen@sandeen.net
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Reviewed-by: Zorro Lang <zlang@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

xfs/032: try running on blocksize > pagesize filesystems

Now that we're no longer limited to blocksize <= pagesize, let's make
sure that mkfs, fsstress, and copy work on such things.  This is also a
subtle way to get more people running at least one test with that
config.

Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
Reviewed-by: Zorro Lang <zlang@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

generic: verify ciphertext with hardware-wrapped keys

Add two tests which verify that encrypted files are encrypted correctly
when a hardware-wrapped inline encryption key is used.  The two tests
are identical except that one uses FSCRYPT_POLICY_FLAG_IV_INO_LBLK_64
and the other uses FSCRYPT_POLICY_FLAG_IV_INO_LBLK_32.  These cover both
of the settings where hardware-wrapped keys may be used.

Signed-off-by: Eric Biggers <ebiggers@google.com>
Reviewed-by: Zorro Lang <zlang@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

common/encrypt: support hardware-wrapped key testing

To support testing the kernel's support for hardware-wrapped inline
encryption keys, update _verify_ciphertext_for_encryption_policy() to
support a hw_wrapped_key option.

Signed-off-by: Eric Biggers <ebiggers@google.com>
Reviewed-by: Zorro Lang <zlang@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

fscrypt-crypt-util: add hardware KDF support

Add support to fscrypt-crypt-util for replicating the extra KDF (Key
Derivation Function) step that is required when a hardware-wrapped
inline encryption key is used.  This step normally occurs in hardware,
but we need to replicate it for testing purposes.

Note, some care was needed to handle the fact that both inlinecrypt_key
and sw_secret can be needed in a single run of fscrypt-crypt-util.
Namely, with --iv-ino-lblk-32, inlinecrypt_key is needed for the
en/decryption while sw_secret is needed for hash_inode_number().

Signed-off-by: Eric Biggers <ebiggers@google.com>
Reviewed-by: Zorro Lang <zlang@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

f2fs/008: test snapshot creation/deletion on lvm device

This patch introduce a regression testcase to check whether
f2fs can handle discard correctly once underlying lvm device
changes to not support discard after user creates snapshot
on it.

Related bug was fixed by commit bc8aeb04fd80 ("f2fs: fix to
drop all discards after creating snapshot on lvm device")

Cc: Jaegeuk Kim <jaegeuk@kernel.org>
Signed-off-by: Chao Yu <chao@kernel.org>
Reviewed-by: Zorro Lang <zlang@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

common/rc: support f2fs in _mkfs_dev()

Then, f2fs/* testcases can use this wrapped common helper instead of
raw codes.

Suggested-by: Zorro Lang <zlang@kernel.org>
Signed-off-by: Chao Yu <chao@kernel.org>
Reviewed-by: Zorro Lang <zlang@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

generic/530: only use xfs-specific mkfs options when testing on xfs

This fixes a regression introduced by commit 000813899afb ("fstests:
scale some tests for high CPU count sanity") where this test
unconditionally tried to use the mkfs option "-l size=256m" which
would break when testing any file sytem other than xfs.

Fix this the same way commit 000813899afb dealt with this for
generic/531; so this was just an oversight.

Fixes: 000813899afb ("fstests: scale some tests for high CPU count sanity")
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

generic/135: don't try to rm $SCRATCH_MNT/*

This fixes a regression for ext4 introduced by commit 32e14cb90b88
("fstests: don't use directory stacks"), which replaced a number of
files at the top-level of the scratch file system:

    async_file sync_file direct_file trunc_file

with "rm $SCRATCH_MNT/*".  This causes a test failure on ext4 file
systems caused by a failed attempt to unlink the lost+found directory.
The thing, is these files are all super small (4k or 16k) and the
scratch file system is going to get reformatted before it gets used
again.  So just dropping the delete is the simplest way to solve the
problem.

Fixes: 32e14cb90b88 ("fstests: don't use directory stacks")
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

generic/590: fix test failure when running against fs other than xfs

With commit ce79de11337e ("fstests: clean up loop device instantiation")
we started to always call _destroy_loop_device at the very end of the
test, but we only create a loop device if we are running against xfs,
so the call to _destroy_loop_device results in an error since no loop
device was setup.

For example running this test against btrfs or ext4 results in this
failure:

   $ ./check generic/590
   FSTYP         -- btrfs
   PLATFORM      -- Linux/x86_64 debian0 6.13.0-rc1-btrfs-next-181+ #1 SMP PREEMPT_DYNAMIC Tue Dec  3 13:03:23 WET 2024
   MKFS_OPTIONS  -- /dev/sdc
   MOUNT_OPTIONS -- /dev/sdc /home/fdmanana/btrfs-tests/scratch_1

   generic/590 29s ... [failed, exit status 1]- output mismatch (see /home/fdmanana/git/hub/xfstests/results//generic/590.out.bad)
      --- tests/generic/590.out 2020-06-10 19:29:03.858520038 +0100
      +++ /home/fdmanana/git/hub/xfstests/results//generic/590.out.bad 2024-12-11 10:48:43.946205346 +0000
       @@ -1,2 +1,5 @@
       QA output created by 590
      -Silence is golden
      +losetup: option requires an argument -- 'd'
      +Try 'losetup --help' for more information.
      +Cannot destroy loop device
      +(see /home/fdmanana/git/hub/xfstests/results//generic/590.full for details)
      ...
      (Run 'diff -u /home/fdmanana/git/hub/xfstests/tests/generic/590.out /home/fdmanana/git/hub/xfstests/results//generic/590.out.bad'  to see the entire diff)
   Ran: generic/590
   Failures: generic/590
   Failed 1 of 1 tests

Fix this by removing the call to _destroy_loop_device at the end of the
test, as it's unnecessary because we call it in the _cleanup function if
we have setup a loop device.

Fixes: ce79de11337e ("fstests: clean up loop device instantiation")
Signed-off-by: Filipe Manana <fdmanana@suse.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Zorro Lang <zlang@kernel.org>

generic/442: fix failure due to missing test number argument for fsync-err

After commit 88c0291d297c ("fstests: per-test dmerror instances") the
script src/dmerror now has an extra argument, corresponding to a test's
sequence number, but generic/442 isn't passing that argument so the test
fails like this:

  $ ./check generic/442
  FSTYP         -- btrfs
  PLATFORM      -- Linux/x86_64 debian0 6.13.0-rc1-btrfs-next-181+ #1 SMP PREEMPT_DYNAMIC Tue Dec  3 13:03:23 WET 2024
  MKFS_OPTIONS  -- /dev/sdc
  MOUNT_OPTIONS -- /dev/sdc /home/fdmanana/btrfs-tests/scratch_1

  generic/442 4s ... - output mismatch (see /home/fdmanana/git/hub/xfstests/results//generic/442.out.bad)
      --- tests/generic/442.out 2020-06-10 19:29:03.850519863 +0100
      +++ /home/fdmanana/git/hub/xfstests/results//generic/442.out.bad 2024-12-10 17:35:59.746597468 +0000
      @@ -1,2 +1,3 @@
       QA output created by 442
      -Test passed!
      +Usage: /home/fdmanana/git/hub/xfstests/src/dmerror {load_error_table|load_working_table}
      +system: program exited: 1
      ...
      (Run 'diff -u /home/fdmanana/git/hub/xfstests/tests/generic/442.out /home/fdmanana/git/hub/xfstests/results//generic/442.out.bad'  to see the entire diff)
  Ran: generic/442
  Failures: generic/442
  Failed 1 of 1 tests

Fix this by passing the test's sequence number as an argument.

Fixes: 88c0291d297c ("fstests: per-test dmerror instances")
Signed-off-by: Filipe Manana <fdmanana@suse.com>
Reviewed-by: Qu Wenruo <wqu@suse.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Zorro Lang <zlang@kernel.org>

btrfs/146: fix failure due to missing test number argument for fsync-err

After commit 88c0291d297c ("fstests: per-test dmerror instances") the
script src/dmerror now has an extra argument, corresponding to a test's
sequence number, but btrfs/146 isn't passing that argument so the test
fails like this:

  $ ./check btrfs/146
  FSTYP         -- btrfs
  PLATFORM      -- Linux/x86_64 debian0 6.13.0-rc1-btrfs-next-181+ #1 SMP PREEMPT_DYNAMIC Tue Dec  3 13:03:23 WET 2024
  MKFS_OPTIONS  -- /dev/sdc
  MOUNT_OPTIONS -- /dev/sdc /home/fdmanana/btrfs-tests/scratch_1

  btrfs/146 3s ... - output mismatch (see /home/fdmanana/git/hub/xfstests/results//btrfs/146.out.bad)
      --- tests/btrfs/146.out 2020-06-10 19:29:03.818519162 +0100
      +++ /home/fdmanana/git/hub/xfstests/results//btrfs/146.out.bad 2024-12-10 17:19:40.363498130 +0000
      @@ -1,3 +1,4 @@
       QA output created by 146
       Format and mount
      -Test passed!
      +Usage: /home/fdmanana/git/hub/xfstests/src/dmerror {load_error_table|load_working_table}
      +system: program exited: 1
      ...
      (Run 'diff -u /home/fdmanana/git/hub/xfstests/tests/btrfs/146.out /home/fdmanana/git/hub/xfstests/results//btrfs/146.out.bad'  to see the entire diff)
  Ran: btrfs/146
  Failures: btrfs/146
  Failed 1 of 1 tests

Fix this by passing the test's sequence number as an argument.

Fixes: 88c0291d297c ("fstests: per-test dmerror instances")
Signed-off-by: Filipe Manana <fdmanana@suse.com>
Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

btrfs/142, btrfs/143: fix dmdust device names

After commit aaa132777476 ("fstests: per-test dmdust instances") the
dmdust device names are no longer a plain 'dust-test', they now have
a suffix matching '.N', where N is the test's sequence number.
The test cases btrf/142 and btrfs/143 are still using the old device
names, so they fail. Fix this my making them refer to 'dust-test.$seq'
instead of 'dust-test'.

Fixes: aaa132777476 ("fstests: per-test dmdust instances")
Signed-off-by: Filipe Manana <fdmanana@suse.com>
Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

btrfs/100, btrfs/101: fix device name in their golden output

After commit 88c0291d297c ("fstests: per-test dmerror instances"), the dm
error device name changed so that it's no longer a plain 'error-test' but
instead it's that plus a prefix matching the test number, such as
'error-test.100' for example. The tests btrfs/100 and btrfs/101 are still
using the plain old name 'error-test' in their golden output, which makes
them fail. So update them to use 'error-test.100' and 'error-test.101'.

Fixes: 88c0291d297c ("fstests: per-test dmerror instances")
Signed-off-by: Filipe Manana <fdmanana@suse.com>
Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

fstests: fix argument passing to _run_fsstress and _run_fsstress_bg

After commit 8973af00ec21 ("fstests: cleanup fsstress process management")
test cases btrfs/007 and btrfs/284 started to fail. For example:

  $ ./check btrfs/007
  FSTYP         -- btrfs
  PLATFORM      -- Linux/x86_64 debian0 6.13.0-rc1-btrfs-next-181+ #1 SMP PREEMPT_DYNAMIC Tue Dec  3 13:03:23 WET 2024
  MKFS_OPTIONS  -- /dev/sdc
  MOUNT_OPTIONS -- /dev/sdc /home/fdmanana/btrfs-tests/scratch_1

  btrfs/007 1s ... [failed, exit status 1]- output mismatch (see /home/fdmanana/git/hub/xfstests/results//btrfs/007.out.bad)
      --- tests/btrfs/007.out 2020-06-10 19:29:03.810518987 +0100
      +++ /home/fdmanana/git/hub/xfstests/results//btrfs/007.out.bad 2024-12-10 16:09:56.345937619 +0000
      @@ -1,3 +1,4 @@
       QA output created by 007
       *** test send / receive
      -*** done
      +failed: '2097152000 200'
      +(see /home/fdmanana/git/hub/xfstests/results//btrfs/007.full for details)
      ...
      (Run 'diff -u /home/fdmanana/git/hub/xfstests/tests/btrfs/007.out /home/fdmanana/git/hub/xfstests/results//btrfs/007.out.bad'  to see the entire diff)
  Ran: btrfs/007

The problem comes from _run_fsstress and _run_fsstress_bg using $*, which
splits the string argument for the -x command used by btrfs/007, so that
fsstress gets the argument for -x as simply:

   "btrfs"

Instead of:

   "btrfs subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/base"

Fix this by using "$@" instead of $*.

Fixes: 8973af00ec21 ("fstests: cleanup fsstress process management")
Signed-off-by: Filipe Manana <fdmanana@suse.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

common: call _require_scratch_dedupe from _require_scratch_duperemove

_require_scratch_duperemove doesn't check if the scratch file system
actually supports dedup, so add the proper call for that.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Tested-by: Chuck Lever <chuck.lever@oracle.com>
Reviewed-by: Darrick J. Wong <djwong@kernel.org>
Reviewed-by: Zorro Lang <zlang@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

common: cleanup scratch_mkfs_sized

Move the XFS RT specific code into the file system type switch
statement.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Darrick J. Wong <djwong@kernel.org>
Signed-off-by: Zorro Lang <zlang@kernel.org>

common: loop device work on zoned file systems

Remove the call to _require_non_zoned_device from _require_loop because
there is nothing that prevents loop device to work on file systems on
zoned devices.  But loop devices are not supported directly on top of
zoned block devices, so add a _require_non_zoned_device to generic/563
to not run that test on zoned block devices.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
Signed-off-by: Zorro Lang <zlang@redhat.com>

f2fs: add commit id to _fixed_by_kernel_commit

The bugs related to f2fs/00[5-7] regression testcases have been fixed
by below commits:

- d5c367ef8287 ("f2fs: fix f2fs_bug_on when uninstalling filesystem call
f2fs_evict_inode.")

- 1acd73edbbfe ("f2fs: fix to account dirty data in __get_secs_required()")

- 26413ce18e85 ("f2fs: compress: fix inconsistent update of i_blocks in
release_compress_blocks and reserve_compress_blocks")

Let's add commit id to _fixed_by_kernel_commit in f2fs/00[5-7].

Cc: Jaegeuk Kim <jaegeuk@kernel.org>
Signed-off-by: Chao Yu <chao@kernel.org>
Reviewed-by: Zorro Lang <zlang@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

fstests: check-parallel

Runs tests in parallel runner threads. Each runner thread has it's
own set of tests to run, and runs a separate instance of check
to run those tests.

check-parallel sets up loop devices, mount points, results
directories, etc for each instance and divides the tests up between
the runner threads.

It currently hard codes the XFS and generic test lists, and then
gives each check invocation an explicit list of tests to run. It
also passes through exclusions so that test exclude filtering is
still done by check.

This is far from ideal, but I didn't want to have to embark on a
major refactoring of check to be able to run stuff in parallel.
It was quite the challenge just to get all the tests and test
infrastructure up to the point where they can run reliably in
parallel.

Hence I've left the actual factoring of test selection and setup
out of the patchset for the moment. The plan is to factor both the
test setup and the test list runner loop out of check and share them
between check and check-parallel, hence not requiring check-parallel
to run check directly. That is future work, however.

With the current test runner setup, it is not uncommon to see >5000%
cpu usage, 150-200kiops and 4-5GB/s of disk bandwidth being used
when running 64 runners. This is a serious stress load as it is
constantly mounting and unmounting dozens of filesystems, creating
and destroying devices, dropping caches, running sync, running CPU
hot plug, running page cache migration, etc.

The massive amount of IO that load generates causes qemu hosts to
abort (i.e. crash) because they run out of vm map segments. Hence
bumping up the max_map_count on the host like so:

echo 1048576 > /proc/sys/vm/max_map_count

is necessary.

There is no significant memory pressure to speak of from running the
tests like this. I've seen a maximum of about 50GB of RAM used when
running tests like this, so running on a 64p/64GB VM the additional
concurrency doesn't really stress memory capacity like it does CPU
and IO.

All the runners are executed in private mount namespaces. This is
to prevent ephemeral mount namespace clones from taking a reference
to every mounted filesystem in the machine and so causing random
"device busy after unmount" failures in the tests that are running
concurrently with the mount namespace setup and teardown.

A typical `pstree -N mnt` looks like:

$ pstree -N mnt
[4026531841]
bash
bash───pstree
[0]
sudo───sudo───check-parallel─┬─check-parallel───nsexec───check───311─┬─cut
                             │                                       └─md5sum
                             ├─check-parallel───nsexec───check───750─┬─750───sleep
                             │                                       └─750.fsstress───4*[750.fsstress───{750.fsstress}]
                             ├─check-parallel───nsexec───check───013───013───sed
                             ├─check-parallel───nsexec───check───251───cp
                             ├─check-parallel───nsexec───check───467───open_by_handle
                             ├─check-parallel───nsexec───check───650─┬─650───sleep
                             │                                       └─650.fsstress─┬─61*[650.fsstress───{650.fsstress}]
                             │                                                      └─2*[650.fsstress]
                             ├─check-parallel───nsexec───check───707
                             ├─check-parallel───nsexec───check───705
                             ├─check-parallel───nsexec───check───416
                             ├─check-parallel───nsexec───check───477───2*[open_by_handle]
                             ├─check-parallel───nsexec───check───140───140
                             ├─check-parallel───nsexec───check───562
                             ├─check-parallel───nsexec───check───415───xfs_io───{xfs_io}
                             ├─check-parallel───nsexec───check───291
                             ├─check-parallel───nsexec───check───017
                             ├─check-parallel───nsexec───check───016
                             ├─check-parallel───nsexec───check───168───2*[168───168]
                             ├─check-parallel───nsexec───check───672───2*[672───672]
                             ├─check-parallel───nsexec───check───170─┬─170───170───170
                             │                                       └─170───170
                             ├─check-parallel───nsexec───check───531───122*[t_open_tmpfiles]
                             ├─check-parallel───nsexec───check───387
                             ├─check-parallel───nsexec───check───748
                             ├─check-parallel───nsexec───check───388─┬─388.fsstress───4*[388.fsstress───{388.fsstress}]
                             │                                       └─sleep
                             ├─check-parallel───nsexec───check───328───328
                             ├─check-parallel───nsexec───check───352
                             ├─check-parallel───nsexec───check───042
                             ├─check-parallel───nsexec───check───426───open_by_handle
                             ├─check-parallel───nsexec───check───756───2*[open_by_handle]
                             ├─check-parallel───nsexec───check───227
                             ├─check-parallel───nsexec───check───208───aio-dio-invalid───2*[aio-dio-invalid]
                             ├─check-parallel───nsexec───check───746───cp
                             ├─check-parallel───nsexec───check───187───187
                             ├─check-parallel───nsexec───check───027───8*[027]
                             ├─check-parallel───nsexec───check───045───xfs_io───{xfs_io}
                             ├─check-parallel───nsexec───check───044
                             ├─check-parallel───nsexec───check───204
                             ├─check-parallel───nsexec───check───186───186
                             ├─check-parallel───nsexec───check───449
                             ├─check-parallel───nsexec───check───231───su───fsx
                             ├─check-parallel───nsexec───check───509
                             ├─check-parallel───nsexec───check───127───5*[127───fsx]
                             ├─check-parallel───nsexec───check───047
                             ├─check-parallel───nsexec───check───043
                             ├─check-parallel───nsexec───check───475───pkill
                             ├─check-parallel───nsexec───check───299─┬─fio─┬─4*[fio]
                             │                                       │     ├─2*[fio───4*[{fio}]]
                             │                                       │     └─{fio}
                             │                                       └─pgrep
                             ├─check-parallel───nsexec───check───551───aio-dio-write-v
                             ├─check-parallel───nsexec───check───323───aio-last-ref-he───100*[{aio-last-ref-he}]
                             ├─check-parallel───nsexec───check───648───sleep
                             ├─check-parallel───nsexec───check───046
                             ├─check-parallel───nsexec───check───753─┬─753.fsstress───4*[753.fsstress]
                             │                                       └─pkill
                             ├─check-parallel───nsexec───check───507───507
                             ├─check-parallel───nsexec───check───629─┬─3*[629───xfs_io───{xfs_io}]
                             │                                       └─5*[629]
                             ├─check-parallel───nsexec───check───073───umount
                             ├─check-parallel───nsexec───check───615───615
                             ├─check-parallel───nsexec───check───176───punch-alternati
                             ├─check-parallel───nsexec───check───294
                             ├─check-parallel───nsexec───check───236───236
                             ├─check-parallel───nsexec───check───165─┬─165─┬─165─┬─cut
                             │                                       │     │     └─xfs_io───{xfs_io}
                             │                                       │     └─165───grep
                             │                                       └─165
                             ├─check-parallel───nsexec───check───259───sync
                             ├─check-parallel───nsexec───check───442───442.fsstress───4*[442.fsstress───{442.fsstress}]
                             ├─check-parallel───nsexec───check───558───255*[558]
                             ├─check-parallel───nsexec───check───358───358───358
                             ├─check-parallel───nsexec───check───169───169
                             └─check-parallel───nsexec───check───297─┬─297.fsstress─┬─284*[297.fsstress───{297.fsstress}]
                                                                     │              └─716*[297.fsstress]
                                                                     └─sleep

A typical test run looks like:

$ time sudo ./check-parallel /mnt/xfs -s xfs -x dump
Runner 63 Failures:  xfs/170
Runner 36 Failures:  xfs/050
Runner 30 Failures:  xfs/273
Runner 29 Failures:  generic/135
Runner 25 Failures:  generic/603
Tests run: 1140
Failure count: 5

Ten slowest tests - runtime in seconds:
xfs/013 454
generic/707 414
generic/017 398
generic/387 395
generic/748 390
xfs/140 351
generic/562 351
generic/705 347
generic/251 344
xfs/016 343

Cleanup on Aisle 5?

total 0
crw-------. 1 root root 10, 236 Nov 27 09:27 control
lrwxrwxrwx. 1 root root       7 Nov 27 09:27 fast -> ../dm-0
/dev/mapper/fast  1.4T  192G  1.2T  14% /mnt/xfs

real    9m29.056s
user    0m0.005s
sys     0m0.022s
$

Yeah, that runtime is real - under 10 minutes for a full XFS auto
group test run. When running this normally (i.e. via check) on this
machine, it usually takes just under 4 hours to run the same set
of tests. i.e. I can run ./check-parallel roughly 25x times on this
machine in the same time it takes to run ./check.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Zorro lang <zlang@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

fstests: quota grace periods unreliable under load

Starting the quota grace period doesn't necessary happen predictably
when the system is under heavy load. This results in random test
failures where grace period timeouts are expected.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Zorro lang <zlang@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

generic/062: don't leave debug files in $here on failure

Move them to $seqres.<debug-file> so they don't pollute the source
tree that fstests is running from.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Zorro lang <zlang@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

fstests: always use fail-at-unmount semantics for XFS

Rather than require every test that tests unmount in failure
conditions have to set up fail-at-unmount semantics for the
underlying filesystem, use these semantics for all test and scratch
device mounts.

This currently only affects XFS filesystems, and helps prevent
unexpected unmount hangs in EIO tests because metadata writes are
configured to try forever by default.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Zorro lang <zlang@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

fstests: capture some failures to seqres.full

Whilst trying to debug test failures, I found a few places where
errors needed to be redirected to $seqres.full rather than
/dev/null. This is a collection of all the conversions that haven't
been captured by some other bug fix patch.

Note that calling _check_filesystems() after removing the
require_test/scratch files means it is a no-op, so I removed that
call at the same time as capturing unmount failures after the test
has failed.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Zorro lang <zlang@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

xfs/076: fix broken mkfs filtering

The test does not do what it says on the packet - the mkfs output is
not actually passed to the mkfs filter, so it doesn't know what
inode size mkfs actually used. Hence CHUNK_SIZE ends up being
calculated as 0, and that means it enters an endless loop because
offset never decreases.

Fix it by adding the missing line continuation.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Zorro lang <zlang@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

xfs/629: single extent files should be within tolerance

The test passes if we have between 2 and 40 extents (despite what
the comment says!), with the target being 20. There is absolutely no
reason for considering a single extent file a failure - that
indicates the filesystem completely defeated the fragmentation
behaviour the test was trying to cause.  Hence expand the range of
"test pass" tolerance to 1-41 extents.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Zorro lang <zlang@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

vfstests: some tests require the testdir to be shared

This ensures that these tests will run successfully when the
parallel check infrastructure makes all the scratch and test
mounts private.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Zorro lang <zlang@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

fstests: clean up termination of various tests

Accumulated minor fixes to improve reliablity of the termination
of various tests when interrupted.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Zorro lang <zlang@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

fstests: clean up a couple of dm-flakey tests

Just little things I've found that should be cleaned up.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Zorro lang <zlang@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

fstests: don't use directory stacks

Using bash directory stacking (pushd, popd, etc) seems to be
somewhat unreliable. I've been seeing occasional random failures
from both pushd and popd commands that cause the test to fail, and
there does not appear to be any reason for the failures occurring.

Rather than wasting time chasing ghosts, just get rid of the
directory stacking altogether.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Zorro lang <zlang@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

generic/085: general cleanup for reliability and debugging

This test was quite unreliable during development of the parallel
check runner. It redirects all errors to /dev/null, so there was no
way to debug it when it failed.

Use common mount/unmount helpers, redirect errors to $seqres.full,
make sure the cleanup code is always run at test exit and only
attempt to kill processes if they are still running during cleanup.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Zorro lang <zlang@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

filters: add a filter that accepts EIO instead of other errors

Running a dm-flakey or dm-error test that loads a table that returns
EIO to all IO and then running a command that is expected to fail
with a specific error is racy.

If there is memory pressure at the same time that the table is
loaded, cached inodes can be turfed from memory and the command then
needs to read the inode it is about to act on from disk again. This
results in the inode read getting EIO and failing (e.g. xfs_io will
return a stat() error) rather than having the desired operation
fail.

This results in spurious test failures that look like this:

generic/331       - output mismatch (see /mnt/xfs/runner-41/results-2024-11-20-10:57:31/xfs/generic/331.out.bad)
    --- tests/generic/331.out   2022-12-21 15:53:25.487044098 +1100
    +++ /mnt/xfs/runner-41/results-2024-11-20-10:57:31/xfs/generic/331.out.bad  2024-11-20 11:02:12.123572607 +1100
    @@ -5,7 +5,8 @@
     1886e67cf8783e89ce6ddc5bb09a3944  SCRATCH_MNT/test-331/file1
     1886e67cf8783e89ce6ddc5bb09a3944  SCRATCH_MNT/test-331/file2
     CoW and unmount
    -fdatasync: Input/output error
    +/mnt/xfs/runner-41/scratch/test-331/file2: Input/output error
    +stat: Input/output error
     Compare files
    ...

Add a new "flakey EIO filter" that will catch -any- EIO error from
the command and change it to the error we expected to see.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Zorro lang <zlang@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

filter: handle mount errors from CONFIG_BLK_DEV_WRITE_MOUNTED=y

Kernels post 6.x may have CONFIG_BLK_DEV_WRITE_MOUNTED=y which
prevents mount from opening the block device on a mounted
filesystem. This results in an error such as:

mount: <dev>: Can't open blockdev

which is not the error that callers of _filter_error_mount() are
looking for. It is, however, a direct result of the test trying
to mount an alreayd mounted filesystem, so it is reflecting the same
error case. Hence this mismatch in errors should not fail the test.

Catch this mount error and convert it to the expected
"already mounted" error for the tests that exercise this behaviour.

There is also a minor test change here to push mount failure
information to $seqres.full in the cases where mount errors occur.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Zorro lang <zlang@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

generic/310: cleanup killing background processes

Use the trick we used with fsstress of copying the binary to a test
specific name so that we can simply use pkill to reliably kill the
background processes this test runs. Also use SIGPIPE to avoid
bash from throwing out "Killed" errors.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Zorro lang <zlang@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>

fstests: scale some tests for high CPU count sanity

Several tests use lots of processes to stress the filesystem. many
of them haven't really considered what this means for running the
test on high CPU machines (e.g. >32p) and the potential contention
and performance issues this might trigger.

Some of these tests simply need to increase the size of the journal.
Some need to run on filesystems with high inherent concurrency (e.g.
larger AG count). Some need more efficient/faster file creation. And
so on.

This commit is a collection of those sorts of changes to improve
runtimes on high CPU count machines.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Zorro lang <zlang@redhat.com>
Signed-off-by: Zorro Lang <zlang@kernel.org>