]> www.infradead.org Git - users/dwmw2/linux.git/commit
cpuidle, sched: Use smp_mb__after_atomic() in current_clr_polling()
authorYujun Dong <yujundong@pascal-lab.net>
Mon, 30 Dec 2024 14:16:24 +0000 (22:16 +0800)
committerIngo Molnar <mingo@kernel.org>
Thu, 20 Mar 2025 09:03:52 +0000 (10:03 +0100)
commit3785c7dbae0f733f13f8857beaaada5d7dc63e02
tree85b00782c78e666eacf602e51ac3833860a75b14
parentb52173065e0aad82a31863bb5f63ebe46f7eb657
cpuidle, sched: Use smp_mb__after_atomic() in current_clr_polling()

In architectures that use the polling bit, current_clr_polling() employs
smp_mb() to ensure that the clearing of the polling bit is visible to
other cores before checking TIF_NEED_RESCHED.

However, smp_mb() can be costly. Given that clear_bit() is an atomic
operation, replacing smp_mb() with smp_mb__after_atomic() is appropriate.

Many architectures implement smp_mb__after_atomic() as a lighter-weight
barrier compared to smp_mb(), leading to performance improvements.
For instance, on x86, smp_mb__after_atomic() is a no-op. This change
eliminates a smp_mb() instruction in the cpuidle wake-up path, saving
several CPU cycles and thereby reducing wake-up latency.

Architectures that do not use the polling bit will retain the original
smp_mb() behavior to ensure that existing dependencies remain unaffected.

Signed-off-by: Yujun Dong <yujundong@pascal-lab.net>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Link: https://lore.kernel.org/r/20241230141624.155356-1-yujundong@pascal-lab.net
include/linux/sched/idle.h