convert this_cpu_cmpxchg users to this_cpu_cmpxchg_local

this_cpu_cmpxchg was modified to atomic version, which can be more costly
than non-atomic version.

Switch users of this_cpu_cmpxchg to this_cpu_cmpxchg_local (which
preserves pre-non-atomic this_cpu_cmpxchg behaviour).

Link: https://lkml.kernel.org/r/20230320180745.709381791@redhat.com
Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
Acked-by: Peter Xu <peterx@redhat.com>
Cc: Aaron Tomlin <atomlin@atomlin.com>
Cc: Christoph Lameter <cl@linux.com>
Cc: Frederic Weisbecker <frederic@kernel.org>
Cc: Heiko Carstens <hca@linux.ibm.com>
Cc: Huacai Chen <chenhuacai@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: "Russell King (Oracle)" <linux@armlinux.org.uk>
Cc: Vlastimil Babka <vbabka@suse.cz>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

diff --git a/kernel/fork.c b/kernel/fork.c
index 639228be01151b0af894a1f2ceccd9cead18c374..2066a57786a896f89d7e3c91b837c5b7780f05c9 100644 (file)
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -203,7 +203,7 @@ static bool try_release_thread_stack_to_cache(struct vm_struct *vm)
        unsigned int i;
 
        for (i = 0; i < NR_CACHED_STACKS; i++) {
-               if (this_cpu_cmpxchg(cached_stacks[i], NULL, vm) != NULL)
+               if (this_cpu_cmpxchg_local(cached_stacks[i], NULL, vm) != NULL)
                        continue;
                return true;
        }

diff --git a/kernel/scs.c b/kernel/scs.c
index d7809affe740471eac9a9723f7df7e92353aa0a7..08b4af1a3f4d1e117c5ac2c64fdfb807087d5455 100644 (file)
--- a/kernel/scs.c
+++ b/kernel/scs.c
@@ -83,7 +83,7 @@ void scs_free(void *s)
         */
 
        for (i = 0; i < NR_CACHED_SCS; i++)
-               if (this_cpu_cmpxchg(scs_cache[i], 0, s) == NULL)
+               if (this_cpu_cmpxchg_local(scs_cache[i], 0, s) == NULL)
                        return;
 
        kasan_unpoison_vmalloc(s, SCS_SIZE, KASAN_VMALLOC_PROT_NORMAL);