selftests/mm: log run_vmtests.sh results in TAP format

When running tests on a CI system (e.g.  LAVA) it is useful to output test
results in TAP (Test Anything Protocol) format so that the CI can parse
the fine-grained results to show regressions.  Many of the mm selftest
binaries already output using the TAP format.  And the kselftests runner
(run_kselftest.sh) also uses the format.  CI systems such as LAVA can
already handle nested TAP reports.  However, with the mm selftests we have
3 levels of nesting (run_kselftest.sh -> run_vmtests.sh -> individual test
binaries) and the middle level did not previously support TAP, which
breaks the parser.

Let's fix that by teaching run_vmtests.sh to output using the TAP format.
Ideally this would be opt-in via a command line argument to avoid the
possibility of breaking anyone's existing scripts that might scrape the
output.  However, it is not possible to pass arguments to tests invoked
via run_kselftest.sh.  So I've implemented an opt-out option (-n), which
will revert to the existing output format.

Future changes to this file should be aware of 2 new conventions:

 - output that is part of the TAP reporting is piped through tap_output
 - general output is piped through tap_prefix

Link: https://lkml.kernel.org/r/20231214162434.3580009-1-ryan.roberts@arm.com
Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>
Reviewed-by: Mark Brown <broonie@kernel.org>
Tested-by: John Hubbard <jhubbard@nvidia.com>
Cc: Aishwarya TCV <aishwarya.tcv@arm.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Shuah Khan <shuah@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

diff --git a/tools/testing/selftests/mm/run_vmtests.sh b/tools/testing/selftests/mm/run_vmtests.sh
index 87f513f5cf91dd7c008873989dda920c19a439e1..246d53a5d7f287756795ca6fc73103018402ddfa 100755 (executable)
--- a/tools/testing/selftests/mm/run_vmtests.sh
+++ b/tools/testing/selftests/mm/run_vmtests.sh
@@ -5,6 +5,7 @@
 # Kselftest framework requirement - SKIP code is 4.
 ksft_skip=4
 
+count_total=0
 count_pass=0
 count_fail=0
 count_skip=0
@@ -17,6 +18,7 @@ usage: ${BASH_SOURCE[0]:-$0} [ options ]
   -a: run all tests, including extra ones
   -t: specify specific categories to tests to run
   -h: display this message
+  -n: disable TAP output
 
 The default behavior is to run required tests only.  If -a is specified,
 will run all tests.
@@ -77,12 +79,14 @@ EOF
 }
 
 RUN_ALL=false
+TAP_PREFIX="# "
 
-while getopts "aht:" OPT; do
+while getopts "aht:n" OPT; do
        case ${OPT} in
                "a") RUN_ALL=true ;;
                "h") usage ;;
                "t") VM_SELFTEST_ITEMS=${OPTARG} ;;
+               "n") TAP_PREFIX= ;;
        esac
 done
 shift $((OPTIND -1))
@@ -184,30 +188,52 @@ fi
 VADDR64=0
 echo "$ARCH64STR" | grep "$ARCH" &>/dev/null && VADDR64=1
 
+tap_prefix() {
+       sed -e "s/^/${TAP_PREFIX}/"
+}
+
+tap_output() {
+       if [[ ! -z "$TAP_PREFIX" ]]; then
+               read str
+               echo $str
+       fi
+}
+
+pretty_name() {
+       echo "$*" | sed -e 's/^\(bash \)\?\.\///'
+}
+
 # Usage: run_test [test binary] [arbitrary test arguments...]
 run_test() {
        if test_selected ${CATEGORY}; then
+               local test=$(pretty_name "$*")
                local title="running $*"
                local sep=$(echo -n "$title" | tr "[:graph:][:space:]" -)
-               printf "%s\n%s\n%s\n" "$sep" "$title" "$sep"
+               printf "%s\n%s\n%s\n" "$sep" "$title" "$sep" | tap_prefix
 
-               "$@"
-               local ret=$?
+               ("$@" 2>&1) | tap_prefix
+               local ret=${PIPESTATUS[0]}
+               count_total=$(( count_total + 1 ))
                if [ $ret -eq 0 ]; then
                        count_pass=$(( count_pass + 1 ))
-                       echo "[PASS]"
+                       echo "[PASS]" | tap_prefix
+                       echo "ok ${count_total} ${test}" | tap_output
                elif [ $ret -eq $ksft_skip ]; then
                        count_skip=$(( count_skip + 1 ))
-                       echo "[SKIP]"
+                       echo "[SKIP]" | tap_prefix
+                       echo "ok ${count_total} ${test} # SKIP" | tap_output
                        exitcode=$ksft_skip
                else
                        count_fail=$(( count_fail + 1 ))
-                       echo "[FAIL]"
+                       echo "[FAIL]" | tap_prefix
+                       echo "not ok ${count_total} ${test} # exit=$ret" | tap_output
                        exitcode=1
                fi
        fi # test_selected
 }
 
+echo "TAP version 13" | tap_output
+
 CATEGORY="hugetlb" run_test ./hugepage-mmap
 
 shmmax=$(cat /proc/sys/kernel/shmmax)
@@ -231,9 +257,9 @@ CATEGORY="hugetlb" run_test ./hugetlb_fault_after_madv
 echo "$nr_hugepages_tmp" > /proc/sys/vm/nr_hugepages
 
 if test_selected "hugetlb"; then
-       echo "NOTE: These hugetlb tests provide minimal coverage.  Use"
-       echo "      https://github.com/libhugetlbfs/libhugetlbfs.git for"
-       echo "      hugetlb regression testing."
+       echo "NOTE: These hugetlb tests provide minimal coverage.  Use"   | tap_prefix
+       echo "      https://github.com/libhugetlbfs/libhugetlbfs.git for" | tap_prefix
+       echo "      hugetlb regression testing."                          | tap_prefix
 fi
 
 CATEGORY="mmap" run_test ./map_fixed_noreplace
@@ -312,7 +338,7 @@ CATEGORY="hmm" run_test bash ./test_hmm.sh smoke
 # MADV_POPULATE_READ and MADV_POPULATE_WRITE tests
 CATEGORY="madv_populate" run_test ./madv_populate
 
-echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope
+(echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope 2>&1) | tap_prefix
 CATEGORY="memfd_secret" run_test ./memfd_secret
 
 # KSM KSM_MERGE_TIME_HUGE_PAGES test with size of 100
@@ -369,6 +395,7 @@ CATEGORY="mkdirty" run_test ./mkdirty
 
 CATEGORY="mdwe" run_test ./mdwe_test
 
-echo "SUMMARY: PASS=${count_pass} SKIP=${count_skip} FAIL=${count_fail}"
+echo "SUMMARY: PASS=${count_pass} SKIP=${count_skip} FAIL=${count_fail}" | tap_prefix
+echo "1..${count_total}" | tap_output
 
 exit $exitcode