mm, slub: simplify kmem_cache_cpu and tid setup

In slab_alloc_node() and do_slab_free() fastpaths we need to guarantee
that our kmem_cache_cpu pointer is from the same cpu as the tid value.
Currently that's done by reading the tid first using this_cpu_read(), then
the kmem_cache_cpu pointer and verifying we read the same tid using the
pointer and plain READ_ONCE().

This can be simplified to just fetching kmem_cache_cpu pointer and then
reading tid using the pointer.  That guarantees they are from the same
cpu.  We don't need to read the tid using this_cpu_read() because the
value will be validated by this_cpu_cmpxchg_double(), making sure we are
on the correct cpu and the freelist didn't change by anyone preempting us
since reading the tid.

Link: https://lkml.kernel.org/r/20210805152000.12817-12-vbabka@suse.cz
Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
Acked-by: Mel Gorman <mgorman@techsingularity.net>
Cc: Christoph Lameter <cl@linux.com>
Cc: David Rientjes <rientjes@google.com>
Cc: Jann Horn <jannh@google.com>
Cc: Jesper Dangaard Brouer <brouer@redhat.com>
Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Pekka Enberg <penberg@kernel.org>
Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Cc: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Stephen Rothwell <sfr@canb.auug.org.au>

diff --git a/mm/slub.c b/mm/slub.c
index b13424444caef03ac690ad763d4383382bd2f901..1f65d75077cebce24ca6f2b7c63c7bb4ec0cd1b4 100644 (file)
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -2868,15 +2868,14 @@ redo:
         * reading from one cpu area. That does not matter as long
         * as we end up on the original cpu again when doing the cmpxchg.
         *
-        * We should guarantee that tid and kmem_cache are retrieved on
-        * the same cpu. It could be different if CONFIG_PREEMPTION so we need
-        * to check if it is matched or not.
+        * We must guarantee that tid and kmem_cache_cpu are retrieved on the
+        * same cpu. We read first the kmem_cache_cpu pointer and use it to read
+        * the tid. If we are preempted and switched to another cpu between the
+        * two reads, it's OK as the two are still associated with the same cpu
+        * and cmpxchg later will validate the cpu.
         */
-       do {
-               tid = this_cpu_read(s->cpu_slab->tid);
-               c = raw_cpu_ptr(s->cpu_slab);
-       } while (IS_ENABLED(CONFIG_PREEMPTION) &&
-                unlikely(tid != READ_ONCE(c->tid)));
+       c = raw_cpu_ptr(s->cpu_slab);
+       tid = READ_ONCE(c->tid);
 
        /*
         * Irqless object alloc/free algorithm used here depends on sequence
@@ -3150,11 +3149,8 @@ redo:
         * data is retrieved via this pointer. If we are on the same cpu
         * during the cmpxchg then the free will succeed.
         */
-       do {
-               tid = this_cpu_read(s->cpu_slab->tid);
-               c = raw_cpu_ptr(s->cpu_slab);
-       } while (IS_ENABLED(CONFIG_PREEMPTION) &&
-                unlikely(tid != READ_ONCE(c->tid)));
+       c = raw_cpu_ptr(s->cpu_slab);
+       tid = READ_ONCE(c->tid);
 
        /* Same with comment on barrier() in slab_alloc_node() */
        barrier();