common: allow to run all tests on idmapped mounts

In addition to the generic and filesystem-specific idmapped mount
testsuites that already exist upstream today add simple
infrastructure so any test can be run on idmapped mounts simply by
setting IDMAPPED_MOUNTS=true in the config file or section. The main
user for now will be overlay to verify it works correctly on
idmapped mounts.

Note that the infrastructure is completely generic so every
filesystem that supports idmapped mounts can simply run all of their
tests idmapped. But note that not all ways to create a mount have
been converted yet. That includes e.g. _dmthin_mount and direct
calls to _mount in various tests.

In addition, there will be corner-cases. For example, xfs doesn't
allow bulkstat on idmapped mounts because it is a filesystem wide
operation, i.e. you can retrieve information for any inode in the
filesystem so the operation cannot be scoped reasonably under a
single mount. So xfstests testing bulkstat will fail as it's
blocked. Similar for some btrfs ioctl()s.

While we could of course restrict this testmode to -overlay for
which we know things work correctly we should not do this. It would
mean that people won't start using it and so we won't see issues
unless someone sits down and goes through more than 1000 tests and
figures out for each individual one whether it needs to be skipped
or not.

So instead allow this mode but for all filesystems so people can
start running and reporting failures and we can fix them up or block
them as we detect them.

Cc: Eryu Guan <guaneryu@gmail.com>
Cc: Christoph Hellwig <hch@lst.de>
Cc: <fstests@vger.kernel.org>
Reviewed-by: Amir Goldstein <amir73il@gmail.com>
Signed-off-by: Christian Brauner (Microsoft) <brauner@kernel.org>
Signed-off-by: Eryu Guan <guaneryu@gmail.com>

diff --git a/README b/README
index 36ebdadd0f1236ecfa76512f633f992a9b14e3d5..7da66cb66a109fd8ebd1d0ea5edb633240e03438 100644 (file)
--- a/README
+++ b/README
@@ -237,6 +237,10 @@ Misc:
    filesystem less than MIN_FSSIZE.
  - Set DIFF_LENGTH to "number of diff lines to print from a failed test",
    by default 10, set to 0 to print the full diff
+ - set IDMAPPED_MOUNTS=true to run all tests on top of idmapped mounts. While
+   this option is supported for all filesystems currently only -overlay is
+   expected to run without issues. For other filesystems additional patches
+   and fixes to the test suite might be needed.
 
 ______________________
 USING THE FSQA SUITE

diff --git a/README.overlay b/README.overlay
index 39e25ada9117f25d999324c28685e68ead746795..ec4671c39698e48640132023183bccf426e213f5 100644 (file)
--- a/README.overlay
+++ b/README.overlay
@@ -31,7 +31,8 @@ partly supported with './check -overlay'. Only multi-section files that
 do not change FSTYP and MKFS_OPTIONS can be safely used with -overlay.
 
 For example, the following multi-section config file can be used to
-run overlay tests on the same base fs, but with different mount options:
+run overlay tests on the same base fs, but with different mount options, and on
+top of idmapped mounts:
 
  [xfs]
  TEST_DEV=/dev/sda5
@@ -46,6 +47,9 @@ run overlay tests on the same base fs, but with different mount options:
  OVERLAY_MOUNT_OPTIONS="-o redirect_dir=off"
  OVERLAY_FSCK_OPTIONS="-n -o redirect_dir=off"
 
+ [idmapped]
+ IDMAPPED_MOUNTS=true
+
 In the example above, MOUNT_OPTIONS will be used to mount the base scratch fs,
 TEST_FS_MOUNT_OPTS will be used to mount the base test fs,
 OVERLAY_MOUNT_OPTIONS will be used to mount both test and scratch overlay and

diff --git a/common/config b/common/config
index 479e50d1ad56a759ebf75283d569894453f1f76f..1033b8903ea12272e8c7477dd8ca2c54aa325a9a 100644 (file)
--- a/common/config
+++ b/common/config
@@ -647,6 +647,7 @@ _overlay_config_override()
        # Set fsck options, use default if user not set directly.
        export FSCK_OPTIONS="$OVERLAY_FSCK_OPTIONS"
        [ -z "$FSCK_OPTIONS" ] && _fsck_opts
+       export IDMAPPED_MOUNTS="$IDMAPPED_MOUNTS"
 }
 
 _overlay_config_restore()

diff --git a/common/overlay b/common/overlay
index 1ca37e291183bbb467b261631863104d72a8faaf..fff67ba12c1706fec29955e741f0b009df7c4c5b 100644 (file)
--- a/common/overlay
+++ b/common/overlay
@@ -73,6 +73,7 @@ _overlay_base_mount()
 
        if [ -z "$dev" -o -z "$mnt" ] || \
                _check_mounted_on $devname $dev $mntname $mnt; then
+               _idmapped_mount $dev $mnt
                # no base fs or already mounted
                return 0
        elif [ $? -ne 1 ]; then
@@ -81,6 +82,7 @@ _overlay_base_mount()
        fi
 
        _mount $* $dev $mnt
+       _idmapped_mount $dev $mnt
 }
 
 _overlay_base_test_mount()

diff --git a/common/rc b/common/rc
index d326572f08f407f33af27d032c45c8a300523ee6..176298011be593f1b9e316b7e04c81c034b86523 100644 (file)
--- a/common/rc
+++ b/common/rc
@@ -334,6 +334,7 @@ _try_scratch_mount()
                return $?
        fi
        _mount -t $FSTYP `_scratch_mount_options $*`
+       _idmapped_mount $SCRATCH_DEV $SCRATCH_MNT
 }
 
 # mount scratch device with given options and _fail if mount fails
@@ -444,6 +445,53 @@ _scratch_shutdown_handle()
        fi
 }
 
+_move_mount()
+{
+       local mnt=$1
+       local tmp=$2
+
+       # Replace $mnt with $tmp. Use a temporary bind-mount because
+       # mount --move will fail with certain mount propagation layouts.
+       $UMOUNT_PROG $mnt || _fail "Failed to unmount $mnt"
+       _mount --bind $tmp $mnt || _fail "Failed to bind-mount $tmp to $mnt"
+       $UMOUNT_PROG $tmp || _fail "Failed to unmount $tmp"
+       rmdir $tmp
+}
+
+_idmapped_mount()
+{
+       [ "$IDMAPPED_MOUNTS" = "true" ] || return 0
+
+       local dev=$1
+       local mnt=$2
+       local status=0
+       local tmp=`mktemp -d`
+
+       local mount_rec=`findmnt -rncv -S $dev -o OPTIONS`
+       if [[ "$mount_rec" == *"idmapped"* ]]; then
+               return 0
+       fi
+
+       # We create an idmapped mount where {g,u}id 0 writes to disk as
+       # {g,u}id 10000000 and $(id -u fsgqa) + 10000000. We change ownership
+        # of $mnt so {g,u} id 0 can actually create objects in there.
+       chown 10000000:10000000 $mnt || return 1
+       $here/src/idmapped-mounts/mount-idmapped \
+               --map-mount b:10000000:0:100000000000 \
+               $mnt $tmp
+       if [ $? -ne 0 ]; then
+               rmdir $tmp
+               return 1
+       fi
+
+       # The next call ensures we don't end up stacking an idmapped mount on
+       # top of the original mount. Instead we fully replace the original
+       # mount with the idmapped mount. This will not just allow a clean mount
+        # layout it also makes unmount and remounting way simpler.
+       _move_mount $mnt $tmp
+       return $?
+}
+
 _test_mount()
 {
     if [ "$FSTYP" == "overlay" ]; then
@@ -452,6 +500,7 @@ _test_mount()
     fi
     _test_options mount
     _mount -t $FSTYP $TEST_OPTIONS $TEST_FS_MOUNT_OPTS $SELINUX_MOUNT_OPTIONS $* $TEST_DEV $TEST_DIR
+    _idmapped_mount $TEST_DEV $TEST_DIR
 }
 
 _test_unmount()
@@ -3007,6 +3056,7 @@ _mount_or_remount_rw()
        if [ $USE_REMOUNT -eq 0 ]; then
                if [ "$FSTYP" != "overlay" ]; then
                        _mount -t $FSTYP $mount_opts $device $mountpoint
+                       _idmapped_mount $device $mountpoint
                else
                        _overlay_mount $device $mountpoint
                fi