powerpc/64s/radix: Fix missing global invalidations when removing copro

With the optimizations for TLB invalidation from commit 0cef77c7798a
("powerpc/64s/radix: flush remote CPUs out of single-threaded
mm_cpumask"), the scope of a TLBI (global vs. local) can now be
influenced by the value of the 'copros' counter of the memory context.

When calling mm_context_remove_copro(), the 'copros' counter is
decremented first before flushing. It may have the unintended side
effect of sending local TLBIs when we explicitly need global
invalidations in this case. Thus breaking any nMMU user in a bad and
unpredictable way.

Fix it by flushing first, before updating the 'copros' counter, so
that invalidations will be global.

Fixes: 0cef77c7798a ("powerpc/64s/radix: flush remote CPUs out of single-threaded mm_cpumask")
Signed-off-by: Frederic Barrat <fbarrat@linux.ibm.com>
Reviewed-by: Nicholas Piggin <npiggin@gmail.com>
Tested-by: Vaibhav Jain <vaibhav@linux.ibm.com>
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>

diff --git a/arch/powerpc/include/asm/mmu_context.h b/arch/powerpc/include/asm/mmu_context.h
index 79d570cbf3325c0e2c67f648badf6edd3dbcd730..b2f89b621b159148c6d836e3c9c3d47a89aafe47 100644 (file)
--- a/arch/powerpc/include/asm/mmu_context.h
+++ b/arch/powerpc/include/asm/mmu_context.h
@@ -143,24 +143,33 @@ static inline void mm_context_remove_copro(struct mm_struct *mm)
 {
        int c;
 
-       c = atomic_dec_if_positive(&mm->context.copros);
-
-       /* Detect imbalance between add and remove */
-       WARN_ON(c < 0);
-
        /*
-        * Need to broadcast a global flush of the full mm before
-        * decrementing active_cpus count, as the next TLBI may be
-        * local and the nMMU and/or PSL need to be cleaned up.
-        * Should be rare enough so that it's acceptable.
+        * When removing the last copro, we need to broadcast a global
+        * flush of the full mm, as the next TLBI may be local and the
+        * nMMU and/or PSL need to be cleaned up.
+        *
+        * Both the 'copros' and 'active_cpus' counts are looked at in
+        * flush_all_mm() to determine the scope (local/global) of the
+        * TLBIs, so we need to flush first before decrementing
+        * 'copros'. If this API is used by several callers for the
+        * same context, it can lead to over-flushing. It's hopefully
+        * not common enough to be a problem.
         *
         * Skip on hash, as we don't know how to do the proper flush
         * for the time being. Invalidations will remain global if
-        * used on hash.
+        * used on hash. Note that we can't drop 'copros' either, as
+        * it could make some invalidations local with no flush
+        * in-between.
         */
-       if (c == 0 && radix_enabled()) {
+       if (radix_enabled()) {
                flush_all_mm(mm);
-               dec_mm_active_cpus(mm);
+
+               c = atomic_dec_if_positive(&mm->context.copros);
+               /* Detect imbalance between add and remove */
+               WARN_ON(c < 0);
+
+               if (c == 0)
+                       dec_mm_active_cpus(mm);
        }
 }
 #else