mm: take a page reference when removing device exclusive entries

Device exclusive page table entries are used to prevent CPU access to a
page whilst it is being accessed from a device.  Typically this is used to
implement atomic operations when the underlying bus does not support
atomic access.  When a CPU thread encounters a device exclusive entry it
locks the page and restores the original entry after calling mmu notifiers
to signal drivers that exclusive access is no longer available.

The device exclusive entry holds a reference to the page making it safe to
access the struct page whilst the entry is present.  However the fault
handling code does not hold the PTL when taking the page lock.  This means
if there are multiple threads faulting concurrently on the device
exclusive entry one will remove the entry whilst others will wait on the
page lock without holding a reference.

This can lead to threads locking or waiting on a page with a zero
refcount.  Whilst mmap_lock prevents the pages getting freed via munmap()
they may still be freed by a migration.  This leads to warnings such as
PAGE_FLAGS_CHECK_AT_FREE due to the page being locked when the refcount
drops to zero.  Note that during removal of the device exclusive entry the
PTE is currently re-checked under the PTL so no futher bad page accesses
occur once it is locked.

Link: https://lkml.kernel.org/r/20230328021434.292971-1-apopple@nvidia.com
Fixes: b756a3b5e7ea ("mm: device exclusive memory access")
Signed-off-by: Alistair Popple <apopple@nvidia.com>
Reviewed-by: John Hubbard <jhubbard@nvidia.com>
Reviewed-by: Ralph Campbell <rcampbell@nvidia.com>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

diff --git a/mm/memory.c b/mm/memory.c
index f456f3b5049cf1545e875436cd9138b3c2b6ead4..a76e749bdf682ab142b2c5d8bfe535b73569fbf4 100644 (file)
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -3563,8 +3563,19 @@ static vm_fault_t remove_device_exclusive_entry(struct vm_fault *vmf)
        struct vm_area_struct *vma = vmf->vma;
        struct mmu_notifier_range range;
 
-       if (!folio_lock_or_retry(folio, vma->vm_mm, vmf->flags))
+       /*
+        * We need a page reference to lock the page because we don't hold the
+        * PTL so a racing thread can remove the device-exclusive entry and
+        * unmap the page. If the page is free the entry must have been
+        * removed already.
+        */
+       if (!get_page_unless_zero(vmf->page))
+               return 0;
+
+       if (!folio_lock_or_retry(folio, vma->vm_mm, vmf->flags)) {
+               put_page(vmf->page);
                return VM_FAULT_RETRY;
+       }
        mmu_notifier_range_init_owner(&range, MMU_NOTIFY_EXCLUSIVE, 0,
                                vma->vm_mm, vmf->address & PAGE_MASK,
                                (vmf->address & PAGE_MASK) + PAGE_SIZE, NULL);
@@ -3577,6 +3588,7 @@ static vm_fault_t remove_device_exclusive_entry(struct vm_fault *vmf)
 
        pte_unmap_unlock(vmf->pte, vmf->ptl);
        folio_unlock(folio);
+       put_page(vmf->page);
 
        mmu_notifier_invalidate_range_end(&range);
        return 0;