mm: add zone device coherent type memory support

Patch series "Add MEMORY_DEVICE_COHERENT for coherent device memory mapping", v5.

This patch series introduces MEMORY_DEVICE_COHERENT, a type of memory
owned by a device that can be mapped into CPU page tables like
MEMORY_DEVICE_GENERIC and can also be migrated like MEMORY_DEVICE_PRIVATE.

This patch series is mostly self-contained except for a few places where
it needs to update other subsystems to handle the new memory type.

System stability and performance are not affected according to our ongoing
testing, including xfstests.

How it works: The system BIOS advertises the GPU device memory (aka VRAM)
as SPM (special purpose memory) in the UEFI system address map.

The amdgpu driver registers the memory with devmap as
MEMORY_DEVICE_COHERENT using devm_memremap_pages.  The initial user for
this hardware page migration capability is the Frontier supercomputer
project.  This functionality is not AMD-specific.  We expect other GPU
vendors to find this functionality useful, and possibly other hardware
types in the future.

Our test nodes in the lab are similar to the Frontier configuration, with
.5 TB of system memory plus 256 GB of device memory split across 4 GPUs,
all in a single coherent address space.  Page migration is expected to
improve application efficiency significantly.  We will report empirical
results as they become available.

Coherent device type pages at gup are now migrated back to system memory
if they are being pinned long-term (FOLL_LONGTERM).  The reason is, that
long-term pinning would interfere with the device memory manager owning
the device-coherent pages (e.g.  evictions in TTM).  These series
incorporate Alistair Popple patches to do this migration from
pin_user_pages() calls.  hmm_gup_test has been added to hmm-test to test
different get user pages calls.

This series includes handling of device-managed anonymous pages returned
by vm_normal_pages.  Although they behave like normal pages for purposes
of mapping in CPU page tables and for COW, they do not support LRU lists,
NUMA migration or THP.

We also introduce a FOLL_LRU flag that adds the same behaviour to
follow_page and related APIs, to allow callers to specify that they expect
to put pages on an LRU list.

This patch (od 13):

Device memory that is cache coherent from device and CPU point of view.
This is used on platforms that have an advanced system bus (like CAPI or
CXL).  Any page of a process can be migrated to such memory.  However, no
one should be allowed to pin such memory so that it can always be evicted.

[hch@lst.de: rebased ontop of the refcount changes, removed is_dev_private_or_coherent_page]
Link: https://lkml.kernel.org/r/20220531200041.24904-1-alex.sierra@amd.com
Link: https://lkml.kernel.org/r/20220531200041.24904-2-alex.sierra@amd.com
Signed-off-by: Alex Sierra <alex.sierra@amd.com>
Signed-off-by: Christoph Hellwig <hch@lst.de>
Acked-by: Felix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: Alistair Popple <apopple@nvidia.com>
Cc: Jason Gunthorpe <jgg@nvidia.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: Ralph Campbell <rcampbell@nvidia.com>
Cc: Jerome Glisse <jglisse@redhat.com>
Cc: Matthew Wilcox <willy@infradead.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

diff --git a/include/linux/memremap.h b/include/linux/memremap.h
index 8af304f6b504727a9f3c8a8de3e95d7313aa1cb4..9f752ebed6137496fd9d89fc0113139135c2b800 100644 (file)
--- a/include/linux/memremap.h
+++ b/include/linux/memremap.h
@@ -41,6 +41,13 @@ struct vmem_altmap {
  * A more complete discussion of unaddressable memory may be found in
  * include/linux/hmm.h and Documentation/vm/hmm.rst.
  *
+ * MEMORY_DEVICE_COHERENT:
+ * Device memory that is cache coherent from device and CPU point of view. This
+ * is used on platforms that have an advanced system bus (like CAPI or CXL). A
+ * driver can hotplug the device memory using ZONE_DEVICE and with that memory
+ * type. Any page of a process can be migrated to such memory. However no one
+ * should be allowed to pin such memory so that it can always be evicted.
+ *
  * MEMORY_DEVICE_FS_DAX:
  * Host memory that has similar access semantics as System RAM i.e. DMA
  * coherent and supports page pinning. In support of coordinating page
@@ -61,6 +68,7 @@ struct vmem_altmap {
 enum memory_type {
        /* 0 is reserved to catch uninitialized type fields */
        MEMORY_DEVICE_PRIVATE = 1,
+       MEMORY_DEVICE_COHERENT,
        MEMORY_DEVICE_FS_DAX,
        MEMORY_DEVICE_GENERIC,
        MEMORY_DEVICE_PCI_P2PDMA,
@@ -143,6 +151,17 @@ static inline bool folio_is_device_private(const struct folio *folio)
        return is_device_private_page(&folio->page);
 }
 
+static inline bool is_device_coherent_page(const struct page *page)
+{
+       return is_zone_device_page(page) &&
+               page->pgmap->type == MEMORY_DEVICE_COHERENT;
+}
+
+static inline bool folio_is_device_coherent(const struct folio *folio)
+{
+       return is_device_coherent_page(&folio->page);
+}
+
 static inline bool is_pci_p2pdma_page(const struct page *page)
 {
        return IS_ENABLED(CONFIG_PCI_P2PDMA) &&

diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index d8e1b9ff72e6715d67280aa3a49e2ad5a38daa8c..370ef639110fa76bd24f357fc574dbfb9512c284 100644 (file)
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -5693,8 +5693,8 @@ out:
  *   2(MC_TARGET_SWAP): if the swap entry corresponding to this pte is a
  *     target for charge migration. if @target is not NULL, the entry is stored
  *     in target->ent.
- *   3(MC_TARGET_DEVICE): like MC_TARGET_PAGE  but page is MEMORY_DEVICE_PRIVATE
- *     (so ZONE_DEVICE page and thus not on the lru).
+ *   3(MC_TARGET_DEVICE): like MC_TARGET_PAGE  but page is device memory and
+ *   thus not on the lru.
  *     For now we such page is charge like a regular page would be as for all
  *     intent and purposes it is just special memory taking the place of a
  *     regular page.
@@ -5732,7 +5732,8 @@ static enum mc_target_type get_mctgt_type(struct vm_area_struct *vma,
                 */
                if (page_memcg(page) == mc.from) {
                        ret = MC_TARGET_PAGE;
-                       if (is_device_private_page(page))
+                       if (is_device_private_page(page) ||
+                           is_device_coherent_page(page))
                                ret = MC_TARGET_DEVICE;
                        if (target)
                                target->page = page;

diff --git a/mm/memory-failure.c b/mm/memory-failure.c
index 845369f839e1924df0690344d69227652a393f4c..16cbaa7b92ad1dcf929aebe6c18be68e5f9e7251 100644 (file)
--- a/mm/memory-failure.c
+++ b/mm/memory-failure.c
@@ -1685,12 +1685,16 @@ static int memory_failure_dev_pagemap(unsigned long pfn, int flags,
                goto unlock;
        }
 
-       if (pgmap->type == MEMORY_DEVICE_PRIVATE) {
+       switch (pgmap->type) {
+       case MEMORY_DEVICE_PRIVATE:
+       case MEMORY_DEVICE_COHERENT:
                /*
-                * TODO: Handle HMM pages which may need coordination
+                * TODO: Handle device pages which may need coordination
                 * with device-side memory.
                 */
                goto unlock;
+       default:
+               break;
        }
 
        /*

diff --git a/mm/memremap.c b/mm/memremap.c
index 8b5c8fd4ea8edd00b60499f6796d14a5d54719ac..f0955785150fe5d2c64f4e5fd2825eeafcd66577 100644 (file)
--- a/mm/memremap.c
+++ b/mm/memremap.c
@@ -315,6 +315,16 @@ void *memremap_pages(struct dev_pagemap *pgmap, int nid)
                        return ERR_PTR(-EINVAL);
                }
                break;
+       case MEMORY_DEVICE_COHERENT:
+               if (!pgmap->ops->page_free) {
+                       WARN(1, "Missing page_free method\n");
+                       return ERR_PTR(-EINVAL);
+               }
+               if (!pgmap->owner) {
+                       WARN(1, "Missing owner\n");
+                       return ERR_PTR(-EINVAL);
+               }
+               break;
        case MEMORY_DEVICE_FS_DAX:
                if (IS_ENABLED(CONFIG_FS_DAX_LIMITED)) {
                        WARN(1, "File system DAX not supported\n");

diff --git a/mm/migrate_device.c b/mm/migrate_device.c
index 5052093d0262d708eb9b573154d1038d07a462eb..a4847ad65da3c1f0186f8b1d6a9bc0482f81d0ec 100644 (file)
--- a/mm/migrate_device.c
+++ b/mm/migrate_device.c
@@ -518,7 +518,7 @@ EXPORT_SYMBOL(migrate_vma_setup);
  *     handle_pte_fault()
  *       do_anonymous_page()
  * to map in an anonymous zero page but the struct page will be a ZONE_DEVICE
- * private page.
+ * private or coherent page.
  */
 static void migrate_vma_insert_page(struct migrate_vma *migrate,
                                    unsigned long addr,
@@ -594,11 +594,8 @@ static void migrate_vma_insert_page(struct migrate_vma *migrate,
                                                page_to_pfn(page));
                entry = swp_entry_to_pte(swp_entry);
        } else {
-               /*
-                * For now we only support migrating to un-addressable device
-                * memory.
-                */
-               if (is_zone_device_page(page)) {
+               if (is_zone_device_page(page) &&
+                   !is_device_coherent_page(page)) {
                        pr_warn_once("Unsupported ZONE_DEVICE page type.\n");
                        goto abort;
                }
@@ -701,10 +698,11 @@ void migrate_vma_pages(struct migrate_vma *migrate)
 
                mapping = page_mapping(page);
 
-               if (is_device_private_page(newpage)) {
+               if (is_device_private_page(newpage) ||
+                   is_device_coherent_page(newpage)) {
                        /*
-                        * For now only support private anonymous when migrating
-                        * to un-addressable device memory.
+                        * For now only support anonymous memory migrating to
+                        * device private or coherent memory.
                         */
                        if (mapping) {
                                migrate->src[i] &= ~MIGRATE_PFN_MIGRATE;

diff --git a/mm/rmap.c b/mm/rmap.c
index 746c05acad2704ee2467820a3ed19e23047bbe8a..d28a0ab725b67581aa10d7357a93d7082a9e1e7e 100644 (file)
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -1972,7 +1972,7 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma,
                /* Update high watermark before we lower rss */
                update_hiwater_rss(mm);
 
-               if (folio_is_zone_device(folio)) {
+               if (folio_is_device_private(folio)) {
                        unsigned long pfn = folio_pfn(folio);
                        swp_entry_t entry;
                        pte_t swp_pte;
@@ -2138,7 +2138,8 @@ void try_to_migrate(struct folio *folio, enum ttu_flags flags)
                                        TTU_SYNC)))
                return;
 
-       if (folio_is_zone_device(folio) && !folio_is_device_private(folio))
+       if (folio_is_zone_device(folio) &&
+           (!folio_is_device_private(folio) && !folio_is_device_coherent(folio)))
                return;
 
        /*