mm/hugetlb: hook page faults for uffd write protection

Hook up hugetlbfs_fault() with the capability to handle userfaultfd-wp
faults.

We do this slightly earlier than hugetlb_cow() so that we can avoid taking
some extra locks that we definitely don't need.

Link: https://lkml.kernel.org/r/20220405014901.14590-1-peterx@redhat.com
Signed-off-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Mike Kravetz <mike.kravetz@oracle.com>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: Jerome Glisse <jglisse@redhat.com>
Cc: "Kirill A . Shutemov" <kirill@shutemov.name>
Cc: Matthew Wilcox <willy@infradead.org>
Cc: Mike Rapoport <rppt@linux.vnet.ibm.com>
Cc: Nadav Amit <nadav.amit@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index fc31a8f49067d3843d991032a3f80aa610da88fa..0a2b3b8e765bb38cad37c791c120362a7b47484a 100644 (file)
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -5737,6 +5737,26 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,
        if (unlikely(!pte_same(entry, huge_ptep_get(ptep))))
                goto out_ptl;
 
+       /* Handle userfault-wp first, before trying to lock more pages */
+       if (userfaultfd_wp(vma) && huge_pte_uffd_wp(huge_ptep_get(ptep)) &&
+           (flags & FAULT_FLAG_WRITE) && !huge_pte_write(entry)) {
+               struct vm_fault vmf = {
+                       .vma = vma,
+                       .address = haddr,
+                       .real_address = address,
+                       .flags = flags,
+               };
+
+               spin_unlock(ptl);
+               if (pagecache_page) {
+                       unlock_page(pagecache_page);
+                       put_page(pagecache_page);
+               }
+               mutex_unlock(&hugetlb_fault_mutex_table[hash]);
+               i_mmap_unlock_read(mapping);
+               return handle_userfault(&vmf, VM_UFFD_WP);
+       }
+
        /*
         * hugetlb_wp() requires page locks of pte_page(entry) and
         * pagecache_page, so here we need take the former one