mm: page_alloc: fix defrag_mode's retry & OOM path

Brendan points out that defrag_mode doesn't properly clear
ALLOC_NOFRAGMENT on its last-ditch attempt to allocate.  But looking
closer, the problem is actually more severe: it doesn't actually *check*
whether it's already retried, and keeps looping.  This means the OOM path
is never taken, and the thread can loop indefinitely.

This is verified with an intentional OOM test on defrag_mode=1, which
results in the machine hanging.  After this patch, it triggers the OOM
kill reliably and recovers.

Clear ALLOC_NOFRAGMENT properly, and only retry once.

Link: https://lkml.kernel.org/r/20250401041231.GA2117727@cmpxchg.org
Fixes: e3aa7df331bc ("mm: page_alloc: defrag_mode")
Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
Reported-by: Brendan Jackman <jackmanb@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index f51aa6051a99867d2d7d8c70aa7c30e523629951..37d111184eeebdadad987e376773a958442baa60 100644 (file)
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -4604,8 +4604,8 @@ retry:
                goto retry;
 
        /* Reclaim/compaction failed to prevent the fallback */
-       if (defrag_mode) {
-               alloc_flags &= ALLOC_NOFRAGMENT;
+       if (defrag_mode && (alloc_flags & ALLOC_NOFRAGMENT)) {
+               alloc_flags &= ~ALLOC_NOFRAGMENT;
                goto retry;
        }