bpf: Fix atomic probe zero-extension

Zero-extending results of atomic probe operations fails with:

    verifier bug. zext_dst is set, but no reg is defined

The problem is that insn_def_regno() handles BPF_ATOMICs, but not
BPF_PROBE_ATOMICs. Fix by adding the missing condition.

Fixes: d503a04f8bc0 ("bpf: Add support for certain atomics in bpf_arena to x86 JIT")
Signed-off-by: Ilya Leoshkevich <iii@linux.ibm.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20240701234304.14336-2-iii@linux.ibm.com

diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index d3927d819465e69aeb6a4a78e96d281303236c53..e25ad5fb9115d17b550caeb26daad1d7e9b6ba60 100644 (file)
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -3217,7 +3217,8 @@ static int insn_def_regno(const struct bpf_insn *insn)
        case BPF_ST:
                return -1;
        case BPF_STX:
-               if (BPF_MODE(insn->code) == BPF_ATOMIC &&
+               if ((BPF_MODE(insn->code) == BPF_ATOMIC ||
+                    BPF_MODE(insn->code) == BPF_PROBE_ATOMIC) &&
                    (insn->imm & BPF_FETCH)) {
                        if (insn->imm == BPF_CMPXCHG)
                                return BPF_REG_0;