]> www.infradead.org Git - users/dwmw2/linux.git/commit
locking/osq_lock: Use atomic_try_cmpxchg_release() in osq_unlock()
authorUros Bizjak <ubizjak@gmail.com>
Tue, 1 Oct 2024 11:45:57 +0000 (13:45 +0200)
committerPeter Zijlstra <peterz@infradead.org>
Fri, 25 Oct 2024 08:01:50 +0000 (10:01 +0200)
commit0d75e0c420e52b4057a2de274054a5274209a2ae
tree62a22db7cd0ec18bc0a0e4dc135f498001ac7ed3
parentf730fd535fc51573f982fad629f2fc6b4a0cde2f
locking/osq_lock: Use atomic_try_cmpxchg_release() in osq_unlock()

Replace this pattern in osq_unlock():

    atomic_cmpxchg(*ptr, old, new) == old

... with the simpler and faster:

    atomic_try_cmpxchg(*ptr, &old, new)

The x86 CMPXCHG instruction returns success in the ZF flag,
so this change saves a compare after the CMPXCHG.  The code
in the fast path of osq_unlock() improves from:

 11b: 31 c9                 xor    %ecx,%ecx
 11d: 8d 50 01              lea    0x1(%rax),%edx
 120: 89 d0                 mov    %edx,%eax
 122: f0 0f b1 0f           lock cmpxchg %ecx,(%rdi)
 126: 39 c2                 cmp    %eax,%edx
 128: 75 05                 jne    12f <...>

to:

 12b: 31 d2                 xor    %edx,%edx
 12d: 83 c0 01              add    $0x1,%eax
 130: f0 0f b1 17           lock cmpxchg %edx,(%rdi)
 134: 75 05                 jne    13b <...>

Signed-off-by: Uros Bizjak <ubizjak@gmail.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Waiman Long <longman@redhat.com>
Link: https://lore.kernel.org/r/20241001114606.820277-1-ubizjak@gmail.com
kernel/locking/osq_lock.c