objpool: fix to make percpu slot allocation more robust

Since gfp & GFP_ATOMIC == GFP_ATOMIC is true for GFP_KERNEL | GFP_HIGH, it
will use kmalloc if user specifies that combination.  Here the reason why
combining the __vmalloc_node() and kmalloc_node() is that the vmalloc does
not support all GFP flag, especially GFP_ATOMIC.  So we should check if
gfp & (GFP_ATOMIC | GFP_KERNEL) != GFP_ATOMIC for vmalloc first.  This
ensures caller can sleep.  And for the robustness, even if vmalloc fails,
it should retry with kmalloc to allocate it.

Link: https://lkml.kernel.org/r/173008598713.1262174.2959179484209897252.stgit@mhiramat.roam.corp.google.com
Fixes: aff1871bfc81 ("objpool: fix choosing allocation for percpu slots")
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Reported-by: Linus Torvalds <torvalds@linux-foundation.org>
Closes: https://lore.kernel.org/all/CAHk-=whO+vSH+XVRio8byJU8idAWES0SPGVZ7KAVdc4qrV0VUA@mail.gmail.com/
Cc: Leo Yan <leo.yan@arm.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Matt Wu <wuqiang.matt@bytedance.com>
Cc: Mikel Rychliski <mikel@mikelr.com>
Cc: Steven Rostedt (Google) <rostedt@goodmis.org>
Cc: Viktor Malik <vmalik@redhat.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

diff --git a/lib/objpool.c b/lib/objpool.c
index fd108fe0d095a70ae815ee15721117378f6b63d8..b998b720c7329d2907005972aa77fb94e66bb430 100644 (file)
--- a/lib/objpool.c
+++ b/lib/objpool.c
@@ -74,15 +74,21 @@ objpool_init_percpu_slots(struct objpool_head *pool, int nr_objs,
                 * warm caches and TLB hits. in default vmalloc is used to
                 * reduce the pressure of kernel slab system. as we know,
                 * mimimal size of vmalloc is one page since vmalloc would
-                * always align the requested size to page size
+                * always align the requested size to page size.
+                * but if vmalloc fails or it is not available (e.g. GFP_ATOMIC)
+                * allocate percpu slot with kmalloc.
                 */
-               if ((pool->gfp & GFP_ATOMIC) == GFP_ATOMIC)
-                       slot = kmalloc_node(size, pool->gfp, cpu_to_node(i));
-               else
+               slot = NULL;
+
+               if ((pool->gfp & (GFP_ATOMIC | GFP_KERNEL)) != GFP_ATOMIC)
                        slot = __vmalloc_node(size, sizeof(void *), pool->gfp,
                                cpu_to_node(i), __builtin_return_address(0));
-               if (!slot)
-                       return -ENOMEM;
+
+               if (!slot) {
+                       slot = kmalloc_node(size, pool->gfp, cpu_to_node(i));
+                       if (!slot)
+                               return -ENOMEM;
+               }
                memset(slot, 0, size);
                pool->cpu_slots[i] = slot;