locking/qspinlock: Avoid redundant read of next pointer

With optimistic prefetch of the next node cacheline, the next pointer
may have been properly inititalized. As a result, the reading
of node->next in the contended path may be redundant. This patch
eliminates the redundant read if the next pointer value is not NULL.

Signed-off-by: Waiman Long <Waiman.Long@hpe.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Davidlohr Bueso <dave@stgolabs.net>
Cc: Douglas Hatch <doug.hatch@hpe.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Scott J Norton <scott.norton@hpe.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Link: http://lkml.kernel.org/r/1447114167-47185-4-git-send-email-Waiman.Long@hpe.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
(cherry picked from commit aa68744f80bfb6f26fbe7f10e42876066f7dac1b)

Signed-off-by: Babu Moger <babu.moger@oracle.com>
Orabug: 26183741
Signed-off-by: Allen Pais <allen.pais@oracle.com>

diff --git a/kernel/locking/qspinlock.c b/kernel/locking/qspinlock.c
index 496d3631cfdf94a3669688ffe4fd10d693dfb4f2..a520eb1daa690d773a0a3505038cb0c9ec2d9e61 100644 (file)
--- a/kernel/locking/qspinlock.c
+++ b/kernel/locking/qspinlock.c
@@ -338,6 +338,7 @@ queue:
         * p,*,* -> n,*,*
         */
        old = xchg_tail(lock, tail);
+       next = NULL;
 
        /*
         * if there was a previous node; link it and wait until reaching the
@@ -398,10 +399,12 @@ queue:
        }
 
        /*
-        * contended path; wait for next, release.
+        * contended path; wait for next if not observed yet, release.
         */
-       while (!(next = READ_ONCE(node->next)))
-               cpu_relax();
+       if (!next) {
+               while (!(next = READ_ONCE(node->next)))
+                       cpu_relax();
+       }
 
        arch_mcs_spin_unlock_contended(&next->locked);