KVM: x86/mmu: Add lockdep assert to enforce safe usage of kvm_unmap_gfn_range()

Add a lockdep assertion in kvm_unmap_gfn_range() to ensure that either
mmu_invalidate_in_progress is elevated, or that the range is being zapped
due to memslot removal (loosely detected by slots_lock being held).
Zapping SPTEs without mmu_invalidate_{in_progress,seq} protection is unsafe
as KVM's page fault path snapshots state before acquiring mmu_lock, and
thus can create SPTEs with stale information if vCPUs aren't forced to
retry faults (due to seeing an in-progress or past MMU invalidation).

Memslot removal is a special case, as the memslot is retrieved outside of
mmu_invalidate_seq, i.e. doesn't use the "standard" protections, and
instead relies on SRCU synchronization to ensure any in-flight page faults
are fully resolved before zapping SPTEs.

Signed-off-by: Sean Christopherson <seanjc@google.com>
Message-ID: <20241009192345.1148353-3-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 09494d01c38edf1ddc0f9b5f32eb76d86267fb9d..8e853a5fc867b742321a78076d5bd81764abbc55 100644 (file)
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -1556,6 +1556,17 @@ bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)
 {
        bool flush = false;
 
+       /*
+        * To prevent races with vCPUs faulting in a gfn using stale data,
+        * zapping a gfn range must be protected by mmu_invalidate_in_progress
+        * (and mmu_invalidate_seq).  The only exception is memslot deletion;
+        * in that case, SRCU synchronization ensures that SPTEs are zapped
+        * after all vCPUs have unlocked SRCU, guaranteeing that vCPUs see the
+        * invalid slot.
+        */
+       lockdep_assert_once(kvm->mmu_invalidate_in_progress ||
+                           lockdep_is_held(&kvm->slots_lock));
+
        if (kvm_memslots_have_rmaps(kvm))
                flush = __kvm_rmap_zap_gfn_range(kvm, range->slot,
                                                 range->start, range->end,