Darrick J. Wong [Wed, 2 Oct 2024 01:11:44 +0000 (18:11 -0700)]
libxfs: pass flags2 from parent to child when creating files
When mkfs creates a new file as a child of an existing directory, we
should propagate the flags2 field from parent to child like the kernel
does. This ensures that mkfs propagates cowextsize hints properly when
protofiles are in use.
Signed-off-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: Christoph Hellwig <hch@lst.de>
Darrick J. Wong [Wed, 2 Oct 2024 01:11:29 +0000 (18:11 -0700)]
libxfs: when creating a file in a directory, set the project id based on the parent
When we're creating a file as a child of an existing directory, use
xfs_get_initial_prid to have the child inherit the project id of the
directory if the directory has PROJINHERIT set, just like the kernel
does. This fixes mkfs project id propagation with -d projinherit=X when
protofiles are in use.
Signed-off-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: Christoph Hellwig <hch@lst.de>
Enable xfs_trans_ichgtime to change the inode access time so that we can
use this function to set inode times when allocating inodes instead of
open-coding it.
Signed-off-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: Christoph Hellwig <hch@lst.de>
Callers that want to create an inode currently pass all possible file
attribute values for the new inode into xfs_init_new_inode as ten
separate parameters. This causes two code maintenance issues: first, we
have large multi-line call sites which programmers must read carefully
to make sure they did not accidentally invert a value. Second, all
three file id parameters must be passed separately to the quota
functions; any discrepancy results in quota count errors.
Clean this up by creating a new icreate_args structure to hold all this
information, some helpers to initialize them properly, and make the
callers pass this structure through to the creation function, whose name
we shorten to xfs_icreate. This eliminates the issues, enables us to
keep the inode init code in sync with userspace via libxfs, and is
needed for future metadata directory tree management.
(A subsequent cleanup will also fix the quota alloc calls.)
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
Callers that want to create an inode currently pass all possible file
attribute values for the new inode into xfs_init_new_inode as ten
separate parameters. This causes two code maintenance issues: first, we
have large multi-line call sites which programmers must read carefully
to make sure they did not accidentally invert a value. Second, all
three file id parameters must be passed separately to the quota
functions; any discrepancy results in quota count errors.
Clean this up by creating a new icreate_args structure to hold all this
information, some helpers to initialize them properly, and make the
callers pass this structure through to the creation function, whose name
we shorten to xfs_icreate. This eliminates the issues, enables us to
keep the inode init code in sync with userspace via libxfs, and is
needed for future metadata directory tree management.
(A subsequent cleanup will also fix the quota alloc calls.)
Signed-off-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: Christoph Hellwig <hch@lst.de>
Darrick J. Wong [Wed, 2 Oct 2024 01:09:55 +0000 (18:09 -0700)]
libxfs: pass IGET flags through to xfs_iread
Change the lock_flags parameter to iget_flags so that we can supply
XFS_IGET_ flags in future patches. All callers of libxfs_iget and
libxfs_trans_iget pass zero for this parameter and there are no inode
locks in xfsprogs, so there's no behavior change here.
Port the kernel's version of the xfs_inode_from_disk callsite.
Signed-off-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: Christoph Hellwig <hch@lst.de>
Following warning is reported, so remove these duplicated header
including:
./fs/xfs/libxfs/xfs_trans_resv.c: xfs_da_format.h is included more than once.
./fs/xfs/scrub/quota_repair.c: xfs_format.h is included more than once.
./fs/xfs/xfs_handle.c: xfs_da_btree.h is included more than once.
./fs/xfs/xfs_qm_bhv.c: xfs_mount.h is included more than once.
./fs/xfs/xfs_trace.c: xfs_bmap.h is included more than once.
This is just a clean code, no logic changed.
Signed-off-by: Wenchao Hao <haowenchao22@gmail.com> Reviewed-by: Darrick J. Wong <djwong@kernel.org> Signed-off-by: Chandan Babu R <chandanbabu@kernel.org>
This adds sanity checks for xfs_dir2_data_unused and xfs_dir2_data_entry
to make sure don't stray beyond valid memory region. Before patching, the
loop simply checks that the start offset of the dup and dep is within the
range. So in a crafted image, if last entry is xfs_dir2_data_unused, we
can change dup->length to dup->length-1 and leave 1 byte of space. In the
next traversal, this space will be considered as dup or dep. We may
encounter an out of bound read when accessing the fixed members.
In the patch, we make sure that the remaining bytes large enough to hold
an unused entry before accessing xfs_dir2_data_unused and
xfs_dir2_data_unused is XFS_DIR2_DATA_ALIGN byte aligned. We also make
sure that the remaining bytes large enough to hold a dirent with a
single-byte name before accessing xfs_dir2_data_entry.
Signed-off-by: lei lu <llfamsec@gmail.com> Reviewed-by: Darrick J. Wong <djwong@kernel.org> Signed-off-by: Chandan Babu R <chandanbabu@kernel.org>
Currently AGFL blocks can be filled from the following three sources:
- allocbt free blocks, as in xfs_allocbt_free_block();
- rmapbt free blocks, as in xfs_rmapbt_free_block();
- refilled from freespace btrees, as in xfs_alloc_fix_freelist().
Originally, allocbt free blocks would be marked as stale only when they
put back in the general free space pool as Dave mentioned on IRC, "we
don't stale AGF metadata btree blocks when they are returned to the
AGFL .. but once they get put back in the general free space pool, we
have to make sure the buffers are marked stale as the next user of
those blocks might be user data...."
However, after commit ca250b1b3d71 ("xfs: invalidate allocbt blocks
moved to the free list") and commit edfd9dd54921 ("xfs: move buffer
invalidation to xfs_btree_free_block"), even allocbt / bmapbt free
blocks will be invalidated immediately since they may fail to pass
V5 format validation on writeback even writeback to free space would be
safe.
IOWs, IMHO currently there is actually no difference of free blocks
between AGFL freespace pool and the general free space pool. So let's
avoid extra redundant AGFL buffer invalidation, since otherwise we're
currently facing unnecessary xfs_log_force() due to xfs_trans_binval()
again on buffers already marked as stale before as below:
xfs_log_force() will take tens of milliseconds with AGF buffer locked.
It becomes an unnecessary long latency especially on our PMEM devices
with FSDAX enabled and fsops like xfs_reflink_find_shared() at the same
time are stuck due to the same AGF lock. Removing the double
invalidation on the AGFL blocks does not make this issue go away, but
this patch fixes for our workloads in reality and it should also work
by the code analysis.
Note that I'm not sure I need to remove another redundant one in
xfs_alloc_ag_vextent_small() since it's unrelated to our workloads.
Also fstests are passed with this patch.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Chandan Babu R <chandanbabu@kernel.org>
Catherine Hoang [Tue, 1 Oct 2024 18:28:49 +0000 (11:28 -0700)]
xfs_io: add RWF_ATOMIC support to pwrite
Enable testing write behavior with the per-io RWF_ATOMIC flag.
Signed-off-by: Catherine Hoang <catherine.hoang@oracle.com> Reviewed-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: John Garry <john.g.garry@oracle.com>
Darrick J. Wong [Tue, 1 Oct 2024 00:06:22 +0000 (17:06 -0700)]
libfrog: emulate deprecated attrlist functionality in libattr
Break our dependence on the (deprecated) libattr by emulating the
necessary pieces in libfrog. It's a little sketchy to open-code these
symbols, but they're originally from XFS so we trust ourselves.
Signed-off-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Reviewed-by: Carlos Maiolino <cmaiolino@redhat.com>
Bastian Germann [Tue, 1 Oct 2024 00:06:22 +0000 (17:06 -0700)]
debian: Modernize build script
- Use autoreconf template, this will now properly installs the
translation files
- Use $(CURDIR) to replace `pwd` syntax
Link: https://bugs.launchpad.net/ubuntu/+source/xfsprogs/+bug/2076309 Suggested-by: Zixing Liu <zixing.liu@canonical.com> Signed-off-by: Bastian Germann <bage@debian.org> Reviewed-by: Darrick J. Wong <djwong@kernel.org> Signed-off-by: Darrick J. Wong <djwong@kernel.org>
Bastian Germann [Tue, 1 Oct 2024 00:06:21 +0000 (17:06 -0700)]
debian: Add Build-Depends on pkg with systemd.pc
The detection for the systemd unit installation searches for systemd.pc.
This file was moved after the bookworm release, so we depend on two
alternative packages.
Fixes: 45cc055588 ("debian: enable xfs_scrub_all systemd timer services by default") Signed-off-by: Bastian Germann <bage@debian.org> Reviewed-by: Darrick J. Wong <djwong@kernel.org> Signed-off-by: Darrick J. Wong <djwong@kernel.org>
Bastian Germann [Tue, 1 Oct 2024 00:06:21 +0000 (17:06 -0700)]
debian: Update public release key
New key material, used for some releases in the Debian archive:
pub ed25519 2022-05-27 [C] 4020459E58C1A52511F5399113F703E6C11CF6F0
uid Carlos Eduardo Maiolino <carlos@maiolino.me>
uid Carlos Eduardo Maiolino <cem@kernel.org>
uid Carlos Eduardo Maiolino <cmaiolino@redhat.com>
sub ed25519 2022-05-27 [A]
sub ed25519 2022-05-27 [S]
sub nistp384 2024-02-15 [A]
sub nistp384 2024-02-15 [S]
sub nistp384 2024-02-15 [E]
sub cv25519 2022-05-27 [E]
Signed-off-by: Bastian Germann <bage@debian.org> Reviewed-by: Darrick J. Wong <djwong@kernel.org> Signed-off-by: Darrick J. Wong <djwong@kernel.org>
Gerald Yang [Tue, 13 Aug 2024 07:25:51 +0000 (15:25 +0800)]
fsck.xfs: fix fsck.xfs run by different shells when fsck.mode=force is set
When fsck.mode=force is specified in the kernel command line, fsck.xfs
is executed during the boot process. However, when the default shell is
not bash, $PS1 should be a different value, consider the following script:
cat ps1.sh
echo "$PS1"
run ps1.sh with different shells:
ash ./ps1.sh
$
bash ./ps1.sh
dash ./ps1.sh
$
ksh ./ps1.sh
zsh ./ps1.sh
On systems like Ubuntu, where dash is the default shell during the boot
process to improve startup speed. This results in FORCE being incorrectly
set to false and then xfs_repair is not invoked:
if [ -n "$PS1" -o -t 0 ]; then
FORCE=false
fi
Other distros may encounter this issue too if the default shell is set
to anoother shell.
Check "-t 0" is enough to determine if we are in interactive mode, and
xfs_repair is invoked as expected regardless of the shell used.
Fixes: 04a2d5dc ("fsck.xfs: allow forced repairs using xfs_repair") Signed-off-by: Gerald Yang <gerald.yang@canonical.com> Reviewed-by: Darrick J. Wong <djwong@kernel.org>
Julien Olivain [Sat, 17 Aug 2024 16:00:52 +0000 (18:00 +0200)]
libxfs: provide a memfd_create() wrapper if not present in libc
Commit 1cb2e387 "libxfs: add xfile support" introduced
a use of the memfd_create() system call, included in version
xfsprogs v6.9.0.
This system call was introduced in kernel commit [1], first included
in kernel v3.17 (released on 2014-10-05).
The memfd_create() glibc wrapper function was added much later in
commit [2], first included in glibc version 2.27 (released on
2018-02-01).
This direct use memfd_create() introduced a requirement on
Kernel >= 3.17 and glibc >= 2.27.
There is old toolchains like [3] for example (which ships gcc 7.3.1,
glibc 2.25 and includes kernel v4.10 headers), that can still be used
to build newer kernels. Even if such toolchains can be seen as
outdated, they are still claimed as supported by recent kernel.
For example, Kernel v6.10.5 has a requirement on gcc version 5.1 and
greater. See [4].
When compiling xfsprogs v6.9.0 with a toolchain not providing the
memfd_create() syscall wrapper, the compilation fail with message:
xfile.c: In function 'xfile_create_fd':
xfile.c:56:7: warning: implicit declaration of function 'memfd_create'; did you mean 'timer_create'? [-Wimplicit-function-declaration]
fd = memfd_create(description, MFD_CLOEXEC | MFD_NOEXEC_SEAL);
^~~~~~~~~~~~
../libxfs/.libs/libxfs.a(xfile.o): In function 'xfile_create_fd':
/build/xfsprogs-6.9.0/libxfs/xfile.c:56: undefined reference to 'memfd_create'
In order to let xfsprogs compile in a wider range of configurations,
this commit adds a memfd_create() function check in autoconf configure
script, and adds a system call wrapper which will be used if the
function is not available. With this commit, the environment
requirement is relaxed to only kernel >= v3.17.
Note: this issue was found in xfsprogs integration in Buildroot [5]
using the command "utils/test-pkg -a -p xfsprogs", which tests many
toolchain/arch combinations.
Carlos Maiolino [Sat, 17 Aug 2024 09:32:48 +0000 (11:32 +0200)]
xfs_io: Fix fscrypt macros ordering
We've been reported a failure to build xfsprogs within buildroot's
environment when they tried to upgrade xfsprogs from 6.4 to 6.9:
encrypt.c:53:36: error: 'FSCRYPT_KEY_IDENTIFIER_SIZE' undeclared
here (not in a function)
53 | __u8
master_key_identifier[FSCRYPT_KEY_IDENTIFIER_SIZE];
|
^~~~~~~~~~~~~~~~~~~~~~~~~~~
encrypt.c:61:42: error: field 'v1' has incomplete type
61 | struct fscrypt_policy_v1 v1;
| ^~
They were using a kernel version without FS_IOC_GET_ENCRYPTION_POLICY_EX
set and OVERRIDE_SYSTEM_FSCRYPT_POLICY_V2 was unset.
This combination caused xfsprogs to attempt to define fscrypt_policy_v2
locally, which uses:
__u8 master_key_identifier[FSCRYPT_KEY_IDENTIFIER_SIZE];
The problem is FSCRYPT_KEY_IDENTIFIER_SIZE is only after this block of
code, so we need to define it earlier.
This also attempts to use fscrypt_policy_v1, which is defined only
later.
To fix this, just reorder both ifdef blocks, but we need to move the
definition of FS_IOC_GET_ENCRYPTION_POLICY_EX to the later, otherwise,
the later definitions won't be enabled causing havoc.
Fixes: e97caf714697a ("xfs_io/encrypt: support specifying crypto data unit size") Reported-by: Waldemar Brodkorb <wbx@openadk.org> Reviewed-by: Bill O'Donnell <bodonnel@redhat.com> Signed-off-by: Carlos Maiolino <cmaiolino@redhat.com> Tested-by: Waldemar Brodkorb <wbx@openadk.org> Reviewed-by: Eric Biggers <ebiggers@google.com>
Darrick J. Wong [Mon, 16 Sep 2024 07:10:02 +0000 (09:10 +0200)]
libxfs: dirty buffers should be marked uptodate too
I started fuzz-testing the realtime rmap feature with a very large
number of realtime allocation groups. There were so many rt groups that
repair had to rebuild /realtime in the metadata directory tree, and that
directory was big enough to spur the creation of a block format
directory.
Unfortunately, repair then walks both directory trees to look for
unconnceted files. This part of phase 6 emits CRC errors on the newly
created buffers for the /realtime directory, declares the directory to
be garbage, and moves all the rt rmap inodes to /lost+found, resulting
in a corrupt fs.
Poking around in gdb, I noticed that the buffer contents were indeed
zero, and that UPTODATE was not set. This was very strange, until I
added a watch on bp->b_flags to watch for accesses. It turns out that
xfs_repair's prefetch code will _get a buffer and zero the contents if
UPTODATE is not set.
The directory tree code in libxfs will also _get a buffer, initialize
it, and log it to the coordinating transaction, which in this case is
the transactions used to reconnect the rmap btree inodes to /realtime.
At no point does any of that code ever set UPTODATE on the buffer, which
is why prefetch zaps the contents.
Hence change both buffer dirtying functions to set UPTODATE, since a
dirty buffer is by definition at least as recent as whatever's on disk.
Signed-off-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: Christoph Hellwig <hch@lst.de>
Several people reported C++ compilation errors due to things that C
compilers allow but C++ compilers do not. Fix both of these problems,
and hope there aren't more of these brown paper bags in 2 months when we
finally get these fixes through the process into a released xfsprogs.
NOTE: I am submitting this bugfix over the objections of a former
maintainer, who insists that we should remove this function from the
published userspace ABI instead of fixing the C++ compilation errors.
No deprecation period, no discussion, just a hard drop of an already
provided and correct C function, which would be in contravention of
Linus' rules. IOWs, removing ABI that have already shipped in a
released kernel requires a careful deprecation period, so I will let
that maintainer run that process.
Reported-by: kernel@mattwhitlock.name Reported-by: sam@gentoo.org Closes: https://bugzilla.kernel.org/show_bug.cgi?id=219203 Fixes: 233f4e12bbb2c ("xfs: add parent pointer ioctls") Signed-off-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: Christoph Hellwig <hch@lst.de>
Carlos Maiolino [Fri, 9 Aug 2024 08:25:11 +0000 (10:25 +0200)]
Merge tag 'debian-autofsck-6.10_2024-08-08' of https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev into for-next
debian: enable xfs_scrub_all by default [v30.11 2/2]
Update our packaging to enable the background xfs_scrub timer by default.
This won't do much unless the sysadmin sets the autofsck fs property or
formats a filesystem with backref metadata.
This has been running on the djcloud for months with no problems. Enjoy!
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
Carlos Maiolino [Fri, 9 Aug 2024 08:24:47 +0000 (10:24 +0200)]
Merge tag 'autofsck-6.10_2024-08-08' of https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev into for-next
xfs_scrub: admin control of automatic fsck [v30.11 1/2]
Now that we have the ability to set per-filesystem properties, teach the
background xfs_scrub service to pick up advice from the filesystem that it
wants to examine, and pick a mode from that. We're only going to enable this
by default for newer filesystems.
This has been running on the djcloud for months with no problems. Enjoy!
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
Darrick J. Wong [Thu, 8 Aug 2024 16:38:48 +0000 (09:38 -0700)]
debian: enable xfs_scrub_all systemd timer services by default
Now that we're finished building online fsck, enable the periodic
background scrub service by default. This involves the postinst script
starting the resource management slice and the timer.
No other sub-services need to be enabled or unmasked explicitly. They
also shouldn't be started or restarted because that might interrupt
background operation unnecessarily.
Although the xfs_scrub_all timer is activated by default, the individual
xfs_scrub@ services that it spawns will only do real work on filesystems
that are new enough to have back reference metadata available. This
avoids surprises for people who are upgrading Debian; only new installs
with new mkfs will get any automatic fsck behavior.
Signed-off-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: Christoph Hellwig <hch@lst.de>
Darrick J. Wong [Thu, 8 Aug 2024 16:38:48 +0000 (09:38 -0700)]
xfs_scrub: use the autofsck fsproperty to select mode
Now that we can set properties on xfs filesystems, make the xfs_scrub
background service query the autofsck property to figure out which
operating mode it should use.
Signed-off-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: Christoph Hellwig <hch@lst.de>
Darrick J. Wong [Thu, 8 Aug 2024 16:38:48 +0000 (09:38 -0700)]
libfrog: define a autofsck filesystem property
Now that we have the ability to set properties on filesystems, create an
"autofsck" property so that sysadmins can control background xfs_scrub
behaviors.
Signed-off-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: Christoph Hellwig <hch@lst.de>
Darrick J. Wong [Thu, 8 Aug 2024 16:38:47 +0000 (09:38 -0700)]
libxfs: pass a transaction context through listxattr
Pass a transaction context so that a new caller can walk the attr names
and query the values all in one go without deadlocking on nested buffer
access.
While we're at it, make the existing xfs_repair callers try to use
empty transactions so that we don't deadlock on cycles in the xattr
structure.
Signed-off-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Acked-by: Dave Chinner <dchinner@redhat.com>
Carlos Maiolino [Tue, 6 Aug 2024 13:50:18 +0000 (15:50 +0200)]
Merge tag 'vectorized-scrub-6.10_2024-07-29' of https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev into for-next
xfs_scrub: vectorize kernel calls [v30.9 22/28]
Create a vectorized version of the metadata scrub and repair ioctl, and
adapt xfs_scrub to use that. This is an experiment to measure overhead
and to try refactoring xfs_scrub.
This has been running on the djcloud for months with no problems. Enjoy!
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
Carlos Maiolino [Tue, 6 Aug 2024 13:50:08 +0000 (15:50 +0200)]
Merge tag 'scrub-directory-tree-6.10_2024-07-29' of https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev into for-next
xfsprogs: detect and correct directory tree problems [v13.8 21/28]
Historically, checking the tree-ness of the directory tree structure has
not been complete. Cycles of subdirectories break the tree properties,
as do subdirectories with multiple parents. It's easy enough for DFS to
detect problems as long as one of the participants is reachable from the
root, but this technique cannot find unconnected cycles.
Directory parent pointers change that, because we can discover all of
these problems from a simple walk from a subdirectory towards the root.
For each child we start with, if the walk terminates without reaching
the root, we know the path is disconnected and ought to be attached to
the lost and found. If we find ourselves, we know this is a cycle and
can delete an incoming edge. If we find multiple paths to the root, we
know to delete an incoming edge.
Even better, once we've finished walking paths, we've identified the
good ones and know which other path(s) to remove.
This has been running on the djcloud for months with no problems. Enjoy!
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
Carlos Maiolino [Tue, 6 Aug 2024 13:49:48 +0000 (15:49 +0200)]
Merge tag 'repair-pptrs-6.10_2024-07-29' of https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev into for-next
xfsprogs: offline repair for parent pointers [v13.8 20/28]
This series implements online checking and repair for directory parent
pointer metadata. The checking half is fairly straightforward -- for
each outgoing directory link (forward or backwards), grab the inode at
the other end, and confirm that there's a corresponding link. If we
can't grab an inode or lock it, we'll save that link for a slower loop
that cycles all the locks, confirms the continued existence of the link,
and rechecks the link if it's actually still there.
Repairs are a bit more involved -- for directories, we walk the entire
filesystem to rebuild the dirents from parent pointer information.
Parent pointer repairs do the same walk but rebuild the pptrs from the
dirent information, but with the added twist that it duplicates all the
xattrs so that it can use the atomic extent swapping code to commit the
repairs atomically.
This introduces an added twist to the xattr repair code -- we use dirent
hooks to detect a colliding update to the pptr data while we're not
holding the ILOCKs; if one is detected, we restart the xattr salvaging
process but this time hold all the ILOCKs until the end of the scan.
For offline repair, the phase6 directory connectivity scan generates an
index of all the expected parent pointers in the filesystem. Then it
walks each file and compares the parent pointers attached to that file
against the index generated, and resyncs the results as necessary.
The last patch teaches xfs_scrub to report pathnames of files that are
being repaired, when possible.
This has been running on the djcloud for months with no problems. Enjoy!
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
Carlos Maiolino [Tue, 6 Aug 2024 13:49:21 +0000 (15:49 +0200)]
Merge tag 'pptrs-6.10_2024-07-29' of https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev into for-next
xfsprogs: Parent Pointers [v13.8 18/28]
This is the latest parent pointer attributes for xfs. The goal of this
patch set is to add a parent pointer attribute to each inode. The
attribute name containing the parent inode, generation, and directory
offset, while the attribute value contains the file name. This feature
will enable future optimizations for online scrub, shrink, nfs handles,
verity, or any other feature that could make use of quickly deriving an
inodes path from the mount point.
Directory parent pointers are stored as namespaced extended attributes
of a file. Because parent pointers are an indivisible tuple of
(dirent_name, parent_ino, parent_gen) we cannot use the usual attr name
lookup functions to find a parent pointer. This is solvable by
introducing a new lookup mode that checks both the name and the value of
the xattr.
Therefore, introduce this new name-value lookup mode that's gated on the
XFS_ATTR_PARENT namespace. This requires the introduction of new
opcodes for the extended attribute update log intent items, which
actually means that parent pointers (itself an INCOMPAT feature) does
not depend on the LOGGED_XATTRS log incompat feature bit.
To reduce collisions on the dirent names of parent pointers, introduce a
new attr hash mode that is the dir2 namehash of the dirent name xor'd
with the parent inode number.
At this point, Allison has moved on to other things, so I've merged her
patchset into djwong-dev for merging.
Updates since v12 [djwong]:
Rebase on 6.9-rc and update the online fsck design document.
Redesign the ondisk format to use the name-value lookups to get us back
to the point where the attr is (dirent_name -> parent_ino/gen).
Updates since v11 [djwong]:
Rebase on 6.4-rc and make some tweaks and bugfixes to enable the repair
prototypes. Merge with djwong-dev and make online repair actually work.
Updates since v10 [djwong]:
Merge in the ondisk format changes to get rid of the diroffset conflicts
with the parent pointer repair code, rebase the entire series with the
attr vlookup changes first, and merge all the other random fixes.
Updates since v9:
Reordered patches 2 and 3 to be 6 and 7
xfs: Add xfs_verify_pptr
moved parent pointer validators to xfs_parent
xfs: Add parent pointer ioctl
Extra validation checks for fs id
added missing release for the inode
use GFP_KERNEL flags for malloc/realloc
reworked ioctl to use pptr listenty and flex array
NEW
xfs: don't remove the attr fork when parent pointers are enabled
NEW
directory lookups should return diroffsets too
NEW
xfs: move/add parent pointer validators to xfs_parent
Updates since v8:
xfs: parent pointer attribute creation
Fix xfs_parent_init to release log assist on alloc fail
Add slab cache for xfs_parent_defer
Fix xfs_create to release after unlock
Add xfs_parent_start and xfs_parent_finish wrappers
removed unused xfs_parent_name_irec and xfs_init_parent_name_irec
xfs: add parent attributes to link
Start/finish wrapper updates
Fix xfs_link to disallow reservationless quotas
xfs: add parent attributes to symlink
Fix xfs_symlink to release after unlock
Start/finish wrapper updates
xfs: Add parent pointers to rename
Start/finish wrapper updates
Fix rename to only grab logged xattr once
Fix xfs_rename to disallow reservationless quotas
Fix double unlock on dqattach fail
Move parent frees to out_release_wip
xfs: Add parent pointers to xfs_cross_rename
Hoist parent pointers into rename
Questions comments and feedback appreciated!
Thanks all!
Allison
This has been running on the djcloud for months with no problems. Enjoy!
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
Prior to introducing parent pointer extended attributes, let's spend
some time cleaning up the attr code and strengthening the validation
that it performs on attrs coming in from the disk.
This has been running on the djcloud for months with no problems. Enjoy!
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
Carlos Maiolino [Tue, 6 Aug 2024 13:48:39 +0000 (15:48 +0200)]
Merge tag 'scrub-media-scan-service-6.10_2024-07-29' of https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev into for-next
xfs_scrub_all: automatic media scan service [v30.9 15/28]
Now that we've completed the online fsck functionality, there are a few
things that could be improved in the automatic service. Specifically,
we would like to perform a more intensive metadata + media scan once per
month, to give the user confidence that the filesystem isn't losing data
silently. To accomplish this, enhance xfs_scrub_all to be able to
trigger media scans. Next, add a duplicate set of system services that
start the media scans automatically.
This has been running on the djcloud for months with no problems. Enjoy!
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
Carlos Maiolino [Tue, 6 Aug 2024 13:48:23 +0000 (15:48 +0200)]
Merge tag 'scrub-service-security-6.10_2024-07-29' of https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev into for-next
xfs_scrub: tighten security of systemd services [v30.9 14/28]
To reduce the risk of the online fsck service suffering some sort of
catastrophic breach that results in attackers reconfiguring the running
system, I embarked on a security audit of the systemd service files.
The result should be that all elements of the background service
(individual scrub jobs, the scrub_all initiator, and the failure
reporting) run with as few privileges and within as strong of a sandbox
as possible.
Granted, this does nothing about the potential for the /kernel/ screwing
up, but at least we could prevent obvious container escapes.
This has been running on the djcloud for months with no problems. Enjoy!
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
Carlos Maiolino [Tue, 6 Aug 2024 13:47:58 +0000 (15:47 +0200)]
Merge tag 'scrub-fstrim-minlen-freesp-histogram-6.10_2024-07-29' of https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev into for-next
xfs_scrub: use free space histograms to reduce fstrim runtime [v30.9 13/28]
This patchset dramatically reduces the runtime of the FITRIM calls made
during phase 8 of xfs_scrub. It turns out that phase 8 can really get
bogged down if the free space contains a large number of very small
extents. In these cases, the runtime can increase by an order of
magnitude to free less than 1%% of the free space. This is not worth the
time, since we're spending a lot of time to do very little work. The
FITRIM ioctl allows us to specify a minimum extent length, so we can use
statistical methods to compute a minlen parameter.
It turns out xfs_db/spaceman already have the code needed to create
histograms of free space extent lengths. We add the ability to compute
a CDF of the extent lengths, which make it easy to pick a minimum length
corresponding to 99%% of the free space. In most cases, this results in
dramatic reductions in phase 8 runtime. Hence, move the histogram code
to libfrog, and wire up xfs_scrub, since phase 7 already walks the
fsmap.
We also add a new -o suboption to xfs_scrub so that people who /do/ want
to examine every free extent can do so.
This has been running on the djcloud for months with no problems. Enjoy!
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
Carlos Maiolino [Tue, 6 Aug 2024 13:47:46 +0000 (15:47 +0200)]
Merge tag 'scrub-fstrim-phase-6.10_2024-07-29' of https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev into for-next
xfs_scrub: move fstrim to a separate phase [v30.9 12/28]
Back when I originally designed xfs_scrub, all filesystem metadata
checks were complete by the end of phase 3, and phase 4 was where all
the metadata repairs occurred. On the grounds that the filesystem
should be fully consistent by then, I made a call to FITRIM at the end
of phase 4 to discard empty space in the filesystem.
Unfortunately, that's no longer the case -- summary counters, link
counts, and quota counters are not checked until phase 7. It's not safe
to instruct the storage to unmap "empty" areas if we don't know where
those empty areas are, so we need to create a phase 8 to trim the fs.
While we're at it, make it more obvious that fstrim only gets to run if
there are no unfixed corruptions and no other runtime errors have
occurred.
Finally, reduce the latency impacts on the rest of the system by
breaking up the fstrim work into a loop that targets only 16GB per call.
This enables better progress reporting for interactive runs and cgroup
based resource constraints for background runs.
This has been running on the djcloud for months with no problems. Enjoy!
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
In early 2023, malware researchers disclosed a phishing attack that was
targeted at people running Linux workstations. The attack vector
involved the use of filenames containing what looked like a file
extension but instead contained a lookalike for the full stop (".")
and a common extension ("pdf"). Enhance xfs_scrub phase 5 to detect
these types of attacks and warn the system administrator.
This has been running on the djcloud for months with no problems. Enjoy!
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
Carlos Maiolino [Tue, 6 Aug 2024 13:47:09 +0000 (15:47 +0200)]
Merge tag 'scrub-repair-scheduling-6.10_2024-07-29' of https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev into for-next
xfs_scrub: improve scheduling of repair items [v30.9 10/28]
Currently, phase 4 of xfs_scrub uses per-AG repair item lists to
schedule repair work across a thread pool. This scheme is suboptimal
when most of the repairs involve a single AG because all the work gets
dumped on a single pool thread.
Instead, we should create a thread pool with the same number of workers
as CPUs, and dispatch individual repair tickets as separate work items
to maximize parallelization.
However, we also need to ensure that repairs to space metadata and file
metadata are kept in separate queues because file repairs generally
depend on correctness of space metadata.
This has been running on the djcloud for months with no problems. Enjoy!
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
Carlos Maiolino [Tue, 6 Aug 2024 13:46:57 +0000 (15:46 +0200)]
Merge tag 'scrub-object-tracking-6.10_2024-07-29' of https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev into for-next
xfs_scrub: use scrub_item to track check progress [v30.9 09/28]
Now that we've introduced tickets to track the status of repairs to a
specific principal XFS object (fs, ag, file), use them to track the
scrub state of those same objects. Ultimately, we want to make it easy
to introduce vectorized repair, where we send a batch of repair requests
to the kernel instead of making millions of ioctl calls. For now,
however, we'll settle for easier bookkeepping.
This has been running on the djcloud for months with no problems. Enjoy!
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
Carlos Maiolino [Tue, 6 Aug 2024 13:46:15 +0000 (15:46 +0200)]
Merge tag 'scrub-repair-data-deps-6.10_2024-07-29' of https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev into for-next
xfs_scrub: track data dependencies for repairs [v30.9 08/28]
Certain kinds of XFS metadata depend on the correctness of lower level
metadata. For example, directory indexes depends on the directory data
fork, which in turn depend on the directory inode to be correct. The
current scrub code does not strictly preserve these dependencies if it
has to defer a repair until phase 4, because phase 4 prioritizes repairs
by type (corruption, then cross referencing, and then preening) and
loses the ordering of in the previous phases. This leads to absurd
things like trying to repair a directory before repairing its corrupted
fork, which is absurd.
To solve this problem, introduce a repair ticket structure to track all
the repairs pending for a principal object (inode, AG, etc). This
reduces memory requirements if an object requires more than one type of
repair and makes it very easy to track the data dependencies between
sub-objects of a principal object. Repair dependencies between object
types (e.g. bnobt before inodes) must still be encoded statically into
phase 4.
A secondary benefit of this new ticket structure is that we can decide
to attempt a repair of an object A that was flagged for a cross
referencing error during the scan if a different object B depends on A
but only B showed definitive signs of corruption.
This has been running on the djcloud for months with no problems. Enjoy!
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
Carlos Maiolino [Tue, 6 Aug 2024 13:45:53 +0000 (15:45 +0200)]
Merge tag 'scrub-better-repair-warnings-6.10_2024-07-29' of https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev into for-next
xfs_scrub: improve warnings about difficult repairs [v30.9 07/28]
While I was poking through the QA results for xfs_scrub, I noticed that
it doesn't warn the user when the primary and secondary realtime
metadata are so out of whack that the chances of a successful repair are
not so high. I decided that it was worth refactoring the scrub code a
bit so that we could warn the user about these types of things, and
ended up refactoring unnecessary helpers out of existence and fixing
other reporting gaps.
This has been running on the djcloud for months with no problems. Enjoy!
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
Carlos Maiolino [Tue, 6 Aug 2024 13:45:40 +0000 (15:45 +0200)]
Merge tag 'scrub-repair-fixes-6.10_2024-07-29' of https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev into for-next
xfs_scrub: fixes to the repair code [v30.9 06/28]
Now that we've landed the new kernel code, it's time to reorganize the
xfs_scrub code that handles repairs. Clean up various naming warts and
misleading error messages. Move the repair code to scrub/repair.c as
the first step. Then, fix various issues in the repair code before we
start reorganizing things.
This has been running on the djcloud for months with no problems. Enjoy!
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
While doing QA of the online fsck code, I made a few observations:
First, nobody was checking that the di_onlink field is actually zero;
Second, that allocating a temporary file for repairs can fail (and
thus bring down the entire fs) if the inode cluster is corrupt; and
Third, that file link counts do not pin at ~0U to prevent integer
overflows.
This scattered patchset fixes those three problems.
This has been running on the djcloud for months with no problems. Enjoy!
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
Carlos Maiolino [Tue, 6 Aug 2024 13:45:05 +0000 (15:45 +0200)]
Merge tag 'dirattr-validate-owners-6.10_2024-07-29' of https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev into for-next
xfsprogs: set and validate dir/attr block owners [v30.9 04/28]
There are a couple of significatn changes that need to be made to the
directory and xattr code before we can support online repairs of those
data structures.
The first change is because online repair is designed to use libxfs to
create a replacement dir/xattr structure in a temporary file, and use
atomic extent swapping to commit the corrected structure. To avoid the
performance hit of walking every block of the new structure to rewrite
the owner number, we instead change libxfs to allow callers of the dir
and xattr code the ability to set an explicit owner number to be written
into the header fields of any new blocks that are created.
The second change is to update the dir/xattr code to actually *check*
the owner number in each block that is read off the disk, since we don't
currently do that.
This has been running on the djcloud for months with no problems. Enjoy!
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
Carlos Maiolino [Tue, 6 Aug 2024 13:44:20 +0000 (15:44 +0200)]
Merge tag 'atomic-file-updates-6.10_2024-07-29' of https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev into for-next
xfsprogs: atomic file updates [v30.9 03/28]
This series creates a new XFS_IOC_EXCHANGE_RANGE ioctl to exchange
ranges of bytes between two files atomically.
This new functionality enables data storage programs to stage and commit
file updates such that reader programs will see either the old contents
or the new contents in their entirety, with no chance of torn writes. A
successful call completion guarantees that the new contents will be seen
even if the system fails.
The ability to exchange file fork mappings between files in this manner
is critical to supporting online filesystem repair, which is built upon
the strategy of constructing a clean copy of a damaged structure and
committing the new structure into the metadata file atomically. The
ioctls exist to facilitate testing of the new functionality and to
enable future application program designs.
User programs will be able to update files atomically by opening an
O_TMPFILE, reflinking the source file to it, making whatever updates
they want to make, and exchange the relevant ranges of the temp file
with the original file. If the updates are aligned with the file block
size, a new (since v2) flag provides for exchanging only the written
areas. Note that application software must quiesce writes to the file
while it stages an atomic update. This will be addressed by a
subsequent series.
This mechanism solves the clunkiness of two existing atomic file update
mechanisms: for O_TRUNC + rewrite, this eliminates the brief period
where other programs can see an empty file. For create tempfile +
rename, the need to copy file attributes and extended attributes for
each file update is eliminated.
However, this method introduces its own awkwardness -- any program
initiating an exchange now needs to have a way to signal to other
programs that the file contents have changed. For file access mediated
via read and write, fanotify or inotify are probably sufficient. For
mmaped files, that may not be fast enough.
The reference implementation in XFS creates a new log incompat feature
and log intent items to track high level progress of swapping ranges of
two files and finish interrupted work if the system goes down. Sample
code can be found in the corresponding changes to xfs_io to exercise the
use case mentioned above.
Note that this function is /not/ the O_DIRECT atomic untorn file writes
concept that has also been floating around for years. It is also not
the RWF_ATOMIC patchset that has been shared. This RFC is constructed
entirely in software, which means that there are no limitations other
than the general filesystem limits.
As a side note, the original motivation behind the kernel functionality
is online repair of file-based metadata. The atomic file content
exchange is implemented as an atomic exchange of file fork mappings,
which means that we can implement online reconstruction of extended
attributes and directories by building a new one in another inode and
exchanging the contents.
Subsequent patchsets adapt the online filesystem repair code to use
atomic file exchanges. This enables repair functions to construct a
clean copy of a directory, xattr information, symbolic links, realtime
bitmaps, and realtime summary information in a temporary inode. If this
completes successfully, the new contents can be committed atomically
into the inode being repaired. This is essential to avoid making
corruption problems worse if the system goes down in the middle of
running repair.
For userspace, this series also includes the userspace pieces needed to
test the new functionality, and a sample implementation of atomic file
updates.
This has been running on the djcloud for months with no problems. Enjoy!
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
Darrick J. Wong [Mon, 29 Jul 2024 23:23:31 +0000 (16:23 -0700)]
xfs_repair: allow symlinks with short remote targets
Symbolic links can have extended attributes. If the attr fork consumes
enough space in the inode record, a shortform symlink can become a
remote symlink. However, if we delete those extended attributes, the
target is not moved back into the inode core.
IOWs, we can end up with a symlink inode that looks like this:
This is a symbolic link with a 297-byte target stored in a disk block,
which is to say this is a symlink with a remote target. The forkoff is
0, which is to say that there's 512 - 176 == 336 bytes in the inode core
to store the data fork.
Prior to kernel commit 1eb70f54c445f, the kernel was ok with this
arrangement, but the change to symlink validation in that patch now
produces corruption errors on filesystems written by older kernels that
are not otherwise inconsistent. Those changes were inspired by reports
of illegal memory accesses, which I think were a result of making data
fork access decisions based on symlink di_size and not on di_format.
Unfortunately, for a very long time xfs_repair has flagged these inodes
as being corrupt, even though the kernel has historically been willing
to read and write symlinks with these properties. Resolve the conflict
by adjusting the xfs_repair corruption tests to allow extents format.
This change matches the kernel patch "xfs: allow symlinks with short
remote targets".
While we're at it, fix a lurking bad symlink fork access.
Signed-off-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: Christoph Hellwig <hch@lst.de>
Darrick J. Wong [Mon, 29 Jul 2024 23:23:31 +0000 (16:23 -0700)]
xfs_scrub: try spot repairs of metadata items to make scrub progress
Now that we've enabled scrub dependency barriers, it's possible that a
scrub_item_check call will return with some of the scrub items still in
NEEDSCHECK state. If, for example, scrub type B depends on scrub type
A being clean and A is not clean, B will still be in NEEDSCHECK state.
In order to make as much scanning progress as possible during phase 2
and phase 3, allow ourselves to try some spot repairs in the hopes that
it will enable us to make progress towards at least scanning the whole
metadata item. If we can't make any forward progress, we'll queue the
scrub item for repair in phase 4, which means that anything still in in
NEEDSCHECK state becomes CORRUPT state. (At worst, the NEEDSCHECK item
will actually be clean by phase 4, and xfs_scrub will report that it
didn't need any work after all.)
Signed-off-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: Christoph Hellwig <hch@lst.de>
Darrick J. Wong [Mon, 29 Jul 2024 23:23:31 +0000 (16:23 -0700)]
xfs_scrub: use scrub barriers to reduce kernel calls
Use scrub barriers so that we can submit a single scrub request for a
bunch of things, and have the kernel stop midway through if it finds
anything broken.
Signed-off-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: Christoph Hellwig <hch@lst.de>
Darrick J. Wong [Mon, 29 Jul 2024 23:23:28 +0000 (16:23 -0700)]
xfs_scrub: detect and repair directory tree corruptions
Now that we have online fsck for directory tree structure problems, we
need to find a place to call it. The scanner requires that parent
pointers are enabled, that directory link counts are correct, and that
every directory entry has a corresponding parent pointer. Therefore, we
can only run it after phase 4 fixes every file, and phase 5 resets the
link counts.
In other words, we call it as part of the phase 5 file scan that we do
to warn about weird looking file names. This has the added benefit that
opening the directory by handle is less likely to fail if there are
loops in the directory structure. For now, only plumb in enough to try
to fix directory tree problems right away; the next patch will make
phase 5 retry the dirloop scanner until the problems are fixed or we
stop making forward progress.
Signed-off-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: Christoph Hellwig <hch@lst.de>
Darrick J. Wong [Mon, 29 Jul 2024 23:23:28 +0000 (16:23 -0700)]
xfs_scrub: fix erroring out of check_inode_names
The early exit logic in this function is a bit suboptimal -- we don't
need to close the @fd if we haven't even opened it, and since all errors
are fatal, we don't need to bump the progress counter. The logic in
this function is about to get more involved due to the addition of the
directory tree structure checker, so clean up these warts.
Signed-off-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: Christoph Hellwig <hch@lst.de>
Darrick J. Wong [Mon, 29 Jul 2024 23:23:28 +0000 (16:23 -0700)]
xfs_spaceman: report directory tree corruption in the health information
Report directories that are the source of corruption in the directory
tree. While we're at it, add the documentation updates for the new
reporting flags and scrub type.
Signed-off-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: Christoph Hellwig <hch@lst.de>
Darrick J. Wong [Mon, 29 Jul 2024 23:23:27 +0000 (16:23 -0700)]
libfrog: add directory tree structure scrubber to scrub library
Make it so that scrub clients can detect corruptions within the
directory tree structure itself. Update the documentation for the scrub
ioctl to mention this new functionality.
Signed-off-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: Christoph Hellwig <hch@lst.de>
Darrick J. Wong [Mon, 29 Jul 2024 23:23:30 +0000 (16:23 -0700)]
xfs_scrub: split the repair epilogue code into a separate function
Move all the code that updates the internal state in response to a
repair ioctl() call completion into a separate function. This will help
with vectorizing repair calls later on.
Signed-off-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: Christoph Hellwig <hch@lst.de>
Darrick J. Wong [Mon, 29 Jul 2024 23:23:29 +0000 (16:23 -0700)]
xfs_scrub: split the scrub epilogue code into a separate function
Move all the code that updates the internal state in response to a scrub
ioctl() call completion into a separate function. This will help with
vectorizing scrub calls later on.
Signed-off-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: Christoph Hellwig <hch@lst.de>
Darrick J. Wong [Mon, 29 Jul 2024 23:23:29 +0000 (16:23 -0700)]
man: document vectored scrub mode
Add a manpage to document XFS_IOC_SCRUBV_METADATA. From the kernel
patch:
Introduce a variant on XFS_SCRUB_METADATA that allows for a vectored
mode. The caller specifies the principal metadata object that they want
to scrub (allocation group, inode, etc.) once, followed by an array of
scrub types they want called on that object. The kernel runs the scrub
operations and writes the output flags and errno code to the
corresponding array element.
A new pseudo scrub type BARRIER is introduced to force the kernel to
return to userspace if any corruptions have been found when scrubbing
the previous scrub types in the array. This enables userspace to
schedule, for example, the sequence:
1. data fork
2. barrier
3. directory
If the data fork scrub is clean, then the kernel will perform the
directory scrub. If not, the barrier in 2 will exit back to userspace.
The alternative would have been an interface where userspace passes a
pointer to an empty buffer, and the kernel formats that with
xfs_scrub_vecs that tell userspace what it scrubbed and what the outcome
was. With that the kernel would have to communicate that the buffer
needed to have been at least X size, even though for our cases
XFS_SCRUB_TYPE_NR + 2 would always be enough.
Compared to that, this design keeps all the dependency policy and
ordering logic in userspace where it already resides instead of
duplicating it in the kernel. The downside of that is that it needs the
barrier logic.
When running fstests in "rebuild all metadata after each test" mode, I
observed a 10% reduction in runtime due to fewer transitions across the
system call boundary.
Signed-off-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: Christoph Hellwig <hch@lst.de>
Enable parent pointer support in mkfs via the '-n parent' parameter.
Signed-off-by: Allison Henderson <allison.henderson@oracle.com> Reviewed-by: Darrick J. Wong <djwong@kernel.org>
[djwong: move the no-V4 filesystem check to join the rest] Signed-off-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: Christoph Hellwig <hch@lst.de>
Darrick J. Wong [Mon, 29 Jul 2024 23:23:25 +0000 (16:23 -0700)]
xfs: create a blob array data structure
Create a simple 'blob array' data structure for storage of arbitrarily
sized metadata objects that will be used to reconstruct metadata. For
the intended usage (temporarily storing extended attribute names and
values) we only have to support storing objects and retrieving them.
Use the xfile abstraction to store the attribute information in memory
that can be swapped out.
Signed-off-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: Christoph Hellwig <hch@lst.de>
mkfs: Add parent pointers during protofile creation
Inodes created from protofile parsing will also need to add the
appropriate parent pointers.
Signed-off-by: Allison Henderson <allison.henderson@oracle.com> Reviewed-by: Darrick J. Wong <djwong@kernel.org>
[djwong: use xfs_parent_add from libxfs instead of open-coding xfs_attr_set] Signed-off-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: Christoph Hellwig <hch@lst.de>
Darrick J. Wong [Mon, 29 Jul 2024 23:23:27 +0000 (16:23 -0700)]
xfs_repair: check parent pointers
Use the parent pointer index that we constructed in the previous patch
to check that each file's parent pointer records exactly match the
directory entries that we recorded while walking directory entries.
Signed-off-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: Christoph Hellwig <hch@lst.de>
Darrick J. Wong [Mon, 29 Jul 2024 23:23:24 +0000 (16:23 -0700)]
libxfs: create new files with attr forks if necessary
Create new files with attr forks if they're going to have parent
pointers. In the next patch we'll fix mkfs to use the same parent
creation functions as the kernel, so we're going to need this.
Signed-off-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: Christoph Hellwig <hch@lst.de>
Darrick J. Wong [Mon, 29 Jul 2024 23:23:26 +0000 (16:23 -0700)]
xfs_repair: move the global dirent name store to a separate object
Abstract the main parent pointer dirent names xfblob object into a
separate data structure to hide implementation details.
The goals here are (a) reduce memory usage when we can by deduplicating
dirent names that exist in multiple directories; and (b) provide a
unique id for each name in the system so that sorting incore parent
pointer records can be done in a stable manner. Fast stable sorting of
records is required for the dirent <-> pptr matching algorithm.
Signed-off-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: Christoph Hellwig <hch@lst.de>
projects / users / hch / xfsprogs.git / log