mm, hwpoison, hugetlb: introduce SUBPAGE_INDEX_HWPOISON to save raw error page

This patchset enables memory error handling on 1GB hugepage.

"Save raw error page" patch (1/4 of patchset [1]) is necessary, so it's
included in this series (the remaining part of hotplug related things are
still in progress).  Patch 2/5 solves issues in a corner case of hugepage
handling, which might not be the main target of this patchset, but
slightly related.  It was posted separately [2] but depends on 1/5, so I
group them together.

Patches 3/5 to 5/5 are the main part of this series and fix a small issue
about handling 1GB hugepage, which I hope will be workable.

[1]: https://lore.kernel.org/linux-mm/20220427042841.678351-1-naoya.horiguchi@linux.dev/T/#u

[2]: https://lore.kernel.org/linux-mm/20220511151955.3951352-1-naoya.horiguchi@linux.dev/T/

This patch (of 5):

When handling memory error on a hugetlb page, the error handler tries to
dissolve and turn it into 4kB pages.  If it's successfully dissolved,
PageHWPoison flag is moved to the raw error page, so that's all right.
However, dissolve sometimes fails, then the error page is left as
hwpoisoned hugepage.  It's useful if we can retry to dissolve it to save
healthy pages, but that's not possible now because the information about
where the raw error page is lost.

Use the private field of a tail page to keep that information.  The code
path of shrinking hugepage pool used this info to try delayed dissolve.
This only keeps one hwpoison page for now, which might be OK because it's
simple and multiple hwpoison pages in a hugepage can be rare.  But it can
be extended in the future.

Link: https://lkml.kernel.org/r/20220602050631.771414-1-naoya.horiguchi@linux.dev
Link: https://lkml.kernel.org/r/20220602050631.771414-2-naoya.horiguchi@linux.dev
Signed-off-by: Naoya Horiguchi <naoya.horiguchi@nec.com>
Reviewed-by: Miaohe Lin <linmiaohe@huawei.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: Mike Kravetz <mike.kravetz@oracle.com>
Cc: Liu Shixin <liushixin2@huawei.com>
Cc: Yang Shi <shy828301@gmail.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Muchun Song <songmuchun@bytedance.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h
index 756b66ff025e5f8283d83df74e5bc51041d593a7..642a39016f9a4e48f0d41c58c0479b36c637aecf 100644 (file)
--- a/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -42,6 +42,9 @@ enum {
        SUBPAGE_INDEX_CGROUP,           /* reuse page->private */
        SUBPAGE_INDEX_CGROUP_RSVD,      /* reuse page->private */
        __MAX_CGROUP_SUBPAGE_INDEX = SUBPAGE_INDEX_CGROUP_RSVD,
+#endif
+#ifdef CONFIG_MEMORY_FAILURE
+       SUBPAGE_INDEX_HWPOISON,
 #endif
        __NR_USED_SUBPAGE,
 };
@@ -798,6 +801,27 @@ extern int dissolve_free_huge_page(struct page *page);
 extern int dissolve_free_huge_pages(unsigned long start_pfn,
                                    unsigned long end_pfn);
 
+#ifdef CONFIG_MEMORY_FAILURE
+/*
+ * pointer to raw error page is located in hpage[SUBPAGE_INDEX_HWPOISON].private
+ */
+static inline struct page *hugetlb_page_hwpoison(struct page *hpage)
+{
+       return (void *)page_private(hpage + SUBPAGE_INDEX_HWPOISON);
+}
+
+static inline void hugetlb_set_page_hwpoison(struct page *hpage,
+                                       struct page *page)
+{
+       set_page_private(hpage + SUBPAGE_INDEX_HWPOISON, (unsigned long)page);
+}
+#else
+static inline struct page *hugetlb_page_hwpoison(struct page *hpage)
+{
+       return NULL;
+}
+#endif
+
 #ifdef CONFIG_ARCH_ENABLE_HUGEPAGE_MIGRATION
 #ifndef arch_hugetlb_migration_supported
 static inline bool arch_hugetlb_migration_supported(struct hstate *h)

diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index a4e127484819f9ae93682ac732640b0d911e1760..7efe240311f6dfb8a7625c1a40436c26079a951d 100644 (file)
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -1553,6 +1553,15 @@ static void __update_and_free_page(struct hstate *h, struct page *page)
                return;
        }
 
+       if (unlikely(PageHWPoison(page))) {
+               struct page *raw_error = hugetlb_page_hwpoison(page);
+
+               if (raw_error && raw_error != page) {
+                       SetPageHWPoison(raw_error);
+                       ClearPageHWPoison(page);
+               }
+       }
+
        for (i = 0; i < pages_per_huge_page(h);
             i++, subpage = mem_map_next(subpage, page, i)) {
                subpage->flags &= ~(1 << PG_locked | 1 << PG_error |

diff --git a/mm/memory-failure.c b/mm/memory-failure.c
index 16cbaa7b92ad1dcf929aebe6c18be68e5f9e7251..1154c3da5c7e73583e5eb2e57c22004482705fdd 100644 (file)
--- a/mm/memory-failure.c
+++ b/mm/memory-failure.c
@@ -1537,6 +1537,8 @@ int __get_huge_page_for_hwpoison(unsigned long pfn, int flags)
                goto out;
        }
 
+       hugetlb_set_page_hwpoison(head, page);
+
        return ret;
 out:
        if (count_increased)