Matthew Wilcox (Oracle) [Tue, 9 Mar 2021 18:48:03 +0000 (13:48 -0500)]
mm/writeback: Add pageset_write_one
Transform write_one_page() into pageset_write_one() and add a compatibility
wrapper. Also move the declaration to pagemap.h as this is page cache
functionality that doesn't need to be used by the rest of the kernel.
Saves 58 bytes of kernel text. While pageset_write_one() is 101 bytes
smaller than write_one_page(), the inlined call to page_pageset() expands
each caller. There are fewer than ten callers so it doesn't seem worth
putting a wrapper in the core.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: David Howells <dhowells@redhat.com>
Matthew Wilcox (Oracle) [Thu, 24 Dec 2020 17:55:56 +0000 (12:55 -0500)]
mm/filemap: Add FGP_STABLE
Allow filemap_get_pageset() to wait for writeback to complete (if the
filesystem wants that behaviour). This is the pageset equivalent of
grab_cache_page_write_begin(), which is moved into the pageset-compat
file as a reminder to migrate all the code using it. This paves the
way for getting rid of AOP_FLAG_NOFS once grab_cache_page_write_begin()
is removed.
Kernel grows by 11 bytes. filemap_get_pageset() grows by 33 bytes but
grab_cache_page_write_begin() shrinks by 22 bytes to make up for it.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Vlastimil Babka <vbabka@suse.cz>
Matthew Wilcox (Oracle) [Mon, 8 Mar 2021 16:45:35 +0000 (11:45 -0500)]
mm/filemap: Add filemap_get_pageset
filemap_get_pageset() is a replacement for find_get_page().
Turn pagecache_get_page() into a wrapper around __filemap_get_pageset().
Remove find_lock_head() as this use case is now covered by
filemap_get_pageset().
Reduces overall kernel size by 209 bytes. __filemap_get_pageset() is
316 bytes shorter than pagecache_get_page() was, but the new
pagecache_get_page() wrapper is 99 bytes.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Vlastimil Babka <vbabka@suse.cz>
Matthew Wilcox (Oracle) [Wed, 16 Dec 2020 04:22:38 +0000 (23:22 -0500)]
mm/filemap: Convert mapping_get_entry to return a pageset
The pagecache only contains pagesets, so indicate that this is definitely
not a tail page. Shrinks mapping_get_entry() by 56 bytes, but grows
pagecache_get_page() by 21 bytes as gcc makes slightly different hot/cold
code decisions. A net reduction of 35 bytes of text.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Vlastimil Babka <vbabka@suse.cz>
Matthew Wilcox (Oracle) [Tue, 8 Dec 2020 13:56:28 +0000 (08:56 -0500)]
mm/filemap: Add filemap_add_pageset()
Convert __add_to_page_cache_locked() into __filemap_add_pageset().
Add an assertion to it that (for !hugetlbfs), the pageset is naturally
aligned within the file. Move the prototype from mm.h to pagemap.h.
Convert add_to_page_cache_lru() into filemap_add_pageset(). Add a
compatibility wrapper for unconverted callers.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Vlastimil Babka <vbabka@suse.cz>
Matthew Wilcox (Oracle) [Wed, 16 Dec 2020 04:11:07 +0000 (23:11 -0500)]
mm/filemap: Add filemap_alloc_pageset
Reimplement __page_cache_alloc as a wrapper around filemap_alloc_pageset
to allow filesystems to be converted at our leisure. Increases
kernel text size by 133 bytes, mostly in cachefiles_read_backing_file().
pagecache_get_page() shrinks by 32 bytes, though.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Vlastimil Babka <vbabka@suse.cz>
Matthew Wilcox (Oracle) [Wed, 16 Dec 2020 03:55:54 +0000 (22:55 -0500)]
mm/page_alloc: Add pageset allocation functions
The __pageset_alloc(), __pageset_alloc_node() and pageset_alloc() functions
are mostly for type safety, but they also ensure that the page allocator
allocates a compound page and initialises the deferred list if the page
is large enough to have one.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Acked-by: Vlastimil Babka <vbabka@suse.cz>
Matthew Wilcox (Oracle) [Fri, 14 May 2021 19:04:28 +0000 (15:04 -0400)]
mm: Add pageset_evictable()
This is the pageset equivalent of page_evictable(). Unfortunately, it's
different from !pageset_test_unevictable(), but I think it's used in places
where you have to be a VM expert and can reasonably be expected to know
the difference.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Acked-by: Vlastimil Babka <vbabka@suse.cz>
mm/workingset: Convert workingset_refault() to take a pageset
This nets us 178 bytes of savings from removing calls to compound_head.
The three callers all grow a little, but each of them will be converted
to use pagesets soon, so that's fine.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Vlastimil Babka <vbabka@suse.cz>
The pointers stored in the page cache are pagesets, by definition.
This change comes with a behaviour change -- callers of readahead_pageset()
are no longer required to put the page reference themselves. This matches
how readpage works, rather than matching how readpages used to work.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Vlastimil Babka <vbabka@suse.cz>
Matthew Wilcox (Oracle) [Mon, 3 May 2021 03:22:52 +0000 (23:22 -0400)]
mm/writeback: Add pageset_redirty_for_writepage()
Reimplement redirty_page_for_writepage() as a wrapper around
pageset_redirty_for_writepage(). Account the number of pages in the
pageset, add kernel-doc and move the prototype to writeback.h.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Vlastimil Babka <vbabka@suse.cz>
Matthew Wilcox (Oracle) [Mon, 3 May 2021 14:06:55 +0000 (10:06 -0400)]
mm/writeback: Add pageset_account_redirty()
Account the number of pages in the pageset that we're redirtying.
Turn account_page_dirty() into a wrapper around it. Also turn
the comment on pageset_account_redirty() into kernel-doc and
edit it slightly so it makes sense to its potential callers.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Vlastimil Babka <vbabka@suse.cz>
Matthew Wilcox (Oracle) [Sun, 28 Feb 2021 21:21:20 +0000 (16:21 -0500)]
mm/writeback: Add pageset_clear_dirty_for_io()
Transform clear_page_dirty_for_io() into pageset_clear_dirty_for_io()
and add a compatibility wrapper. Also move the declaration to pagemap.h
as this is page cache functionality that doesn't need to be used by the
rest of the kernel.
Increases the size of the kernel by 79 bytes. While we remove a few
calls to compound_head(), we add a call to pageset_nr_pages() to get the
stats correct for the eventual support of multi-page pagesets.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Vlastimil Babka <vbabka@suse.cz>
Matthew Wilcox (Oracle) [Mon, 8 Mar 2021 21:43:04 +0000 (16:43 -0500)]
mm/writeback: Add pageset_cancel_dirty()
Turn __cancel_dirty_page() into __pageset_cancel_dirty() and add wrappers.
Move the prototypes into pagemap.h since this is page cache functionality.
Saves 44 bytes of kernel text in total; 33 bytes from __pageset_cancel_dirty
and 11 from two callers of cancel_dirty_page().
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Vlastimil Babka <vbabka@suse.cz>
Matthew Wilcox (Oracle) [Tue, 4 May 2021 20:12:09 +0000 (16:12 -0400)]
mm/writeback: Add pageset_account_cleaned()
Get the statistics right; compound pages were being accounted as a
single page. This didn't matter before now as no filesystem which
supported compound pages did writeback. Also move the declaration
to pagemap.h since this is part of the page cache. Add a wrapper for
account_page_cleaned().
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Vlastimil Babka <vbabka@suse.cz>
Matthew Wilcox (Oracle) [Tue, 4 May 2021 03:30:44 +0000 (23:30 -0400)]
mm/writeback: Add filemap_dirty_pageset()
Reimplement __set_page_dirty_nobuffers() as a wrapper around
filemap_dirty_pageset(). Eventually pageset_mark_dirty() will pass
the pageset's mapping to the address space's ->dirty_pageset()
operation, so add the parameter to filemap_dirty_pageset() now.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Vlastimil Babka <vbabka@suse.cz>
Matthew Wilcox (Oracle) [Tue, 4 May 2021 15:01:10 +0000 (11:01 -0400)]
mm/writeback: Add __pageset_mark_dirty()
Turn __set_page_dirty() into a wrapper around __pageset_mark_dirty().
Convert account_page_dirtied() into pageset_account_dirtied() and account
the number of pages in the pageset to support multi-page pagesets.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Vlastimil Babka <vbabka@suse.cz>
Reimplement set_page_dirty() as a wrapper around pageset_mark_dirty().
There is no change to filesystems as they were already being called
with the compound_head of the page being marked dirty. We avoid
several calls to compound_head(), both statically (through
using pageset_test_dirty() instead of PageDirty() and dynamically by
calling pageset_mapping() instead of page_mapping().
Also return bool instead of int to show the range of values actually
returned, and add kernel-doc.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Vlastimil Babka <vbabka@suse.cz>
Rename set_page_writeback() to pageset_start_writeback() to match
pageset_end_writeback(). Do not bother with wrappers that return void;
callers are perfectly capable of ignoring return values.
Add wrappers for set_page_writeback(), set_page_writeback_keepwrite() and
test_set_page_writeback() for compatibililty with existing filesystems.
The main advantage of this patch is getting the statistics right,
although it does eliminate a couple of calls to compound_head().
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Vlastimil Babka <vbabka@suse.cz>
Matthew Wilcox (Oracle) [Sat, 16 Jan 2021 04:34:16 +0000 (23:34 -0500)]
mm/writeback: Add __pageset_end_writeback()
test_clear_page_writeback() is actually an mm-internal function, although
it's named as if it's a pagecache function. Move it to mm/internal.h,
rename it to __pageset_end_writeback() and change the return type to bool.
The conversion from page to pageset is mostly about accounting the number
of pages being written back, although it does eliminate a couple of
calls to compound_head().
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Vlastimil Babka <vbabka@suse.cz>
Matthew Wilcox (Oracle) [Sat, 20 Mar 2021 20:45:15 +0000 (16:45 -0400)]
mm/writeback: Change __wb_writeout_inc() to __wb_writeout_add()
Allow for accounting N pages at once instead of one page at a time.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Reviewed-by: Jan Kara <jack@suse.cz> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Vlastimil Babka <vbabka@suse.cz>
Matthew Wilcox (Oracle) [Sat, 20 Mar 2021 20:34:54 +0000 (16:34 -0400)]
flex_proportions: Allow N events instead of 1
When batching events (such as writing back N pages in a single I/O), it
is better to do one flex_proportion operation instead of N. There is
only one caller of __fprop_inc_percpu_max(), and it's the one we're
going to change in the next patch, so rename it instead of adding a
compatibility wrapper.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Reviewed-by: Jan Kara <jack@suse.cz>
Matthew Wilcox (Oracle) [Fri, 7 May 2021 19:05:06 +0000 (15:05 -0400)]
mm/migrate: Add pageset_migrate_copy()
This is the pageset equivalent of migrate_page_copy(), which is retained
as a wrapper for filesystems which are not yet converted to pagesets.
Also convert copy_huge_page() to pageset_copy().
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Zi Yan <ziy@nvidia.com> Acked-by: Vlastimil Babka <vbabka@suse.cz>
Matthew Wilcox (Oracle) [Fri, 7 May 2021 19:26:29 +0000 (15:26 -0400)]
mm/migrate: Add pageset_migrate_flags()
Turn migrate_page_states() into a wrapper around pageset_migrate_flags().
Also convert two functions only called from pageset_migrate_flags() to
be pageset-based. ksm_migrate_page() becomes pageset_migrate_ksm() and
copy_page_owner() becomes pageset_copy_owner(). pageset_migrate_flags()
alone shrinks by two thirds -- 1967 bytes down to 642 bytes.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Zi Yan <ziy@nvidia.com> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Vlastimil Babka <vbabka@suse.cz>
Matthew Wilcox (Oracle) [Sun, 28 Feb 2021 21:02:57 +0000 (16:02 -0500)]
mm/rmap: Add pageset_mkclean()
Transform page_mkclean() into pageset_mkclean() and add a page_mkclean()
wrapper around pageset_mkclean().
pageset_mkclean is 15 bytes smaller than page_mkclean, but the kernel
is enlarged by 33 bytes due to inlining page_pageset() into each caller.
This will go away once the callers are converted to use pageset_mkclean().
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Vlastimil Babka <vbabka@suse.cz>
Convert mark_page_accessed() to pageset_mark_accessed(). It already
operated on the entire compound page, but now we can avoid calling
compound_head quite so many times. Shrinks the function from 424 bytes
to 295 bytes (shrinking by 129 bytes). The compatibility wrapper is 30
bytes, plus the 8 bytes for the exported symbol means the kernel shrinks
by 91 bytes.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Vlastimil Babka <vbabka@suse.cz>
This replaces activate_page() and eliminates lots of calls to
compound_head(). Saves net 118 bytes of kernel text. There are still
some redundant calls to page_pageset() here which will be removed when
pagevec_lru_move_fn() is converted to use pagesets.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Vlastimil Babka <vbabka@suse.cz>
Idle page tracking is handled through page_ext on 32-bit architectures.
Add pageset equivalents for 32-bit and move all the page compatibility
parts to common code.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Acked-by: Vlastimil Babka <vbabka@suse.cz> Reviewed-by: William Kucharski <william.kucharski@oracle.com> Reviewed-by: Christoph Hellwig <hch@lst.de> Reviewed-by: David Howells <dhowells@redhat.com>
Matthew Wilcox (Oracle) [Wed, 16 Dec 2020 16:06:33 +0000 (11:06 -0500)]
mm: Add flush_dcache_pageset()
This is a default implementation which calls flush_dcache_page() on
each page in the pageset. If architectures can do better, they should
implement their own version of it.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Acked-by: Vlastimil Babka <vbabka@suse.cz>
Matthew Wilcox (Oracle) [Fri, 7 May 2021 15:17:34 +0000 (11:17 -0400)]
mm: Add pageset_raw_mapping()
Convert __page_rmapping to pageset_raw_mapping and move it to mm/internal.h.
It's only a couple of instructions (load and mask), so it's definitely
going to be cheaper to inline it than call it. Leave page_rmapping
out of line. Change page_anon_vma() to not call pageset_raw_mapping() --
it's more efficient to do the subtraction than the mask.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Vlastimil Babka <vbabka@suse.cz>
Matthew Wilcox (Oracle) [Tue, 4 May 2021 21:19:13 +0000 (17:19 -0400)]
mm/workingset: Convert workingset_activation to take a pageset
This function already assumed it was being passed a head page. No real
change here, except that thp_nr_pages() compiles away on kernels with
THP compiled out while pageset_nr_pages() is always present. Also convert
page_memcg_rcu() to pageset_memcg_rcu().
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Vlastimil Babka <vbabka@suse.cz>
Matthew Wilcox (Oracle) [Wed, 30 Jun 2021 02:27:31 +0000 (22:27 -0400)]
mm/memcg: Add pageset_lruvec_relock_irq() and pageset_lruvec_relock_irqsave()
These are the pageset equivalents of relock_page_lruvec_irq() and
pageset_lruvec_relock_irqsave(). Also convert page_matches_lruvec()
to pageset_matches_lruvec().
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Vlastimil Babka <vbabka@suse.cz>
Matthew Wilcox (Oracle) [Mon, 28 Jun 2021 21:26:00 +0000 (17:26 -0400)]
mm/memcg: Add pageset_memcg_lock() and pageset_memcg_unlock()
These are the pageset equivalents of lock_page_memcg() and
unlock_page_memcg().
lock_page_memcg() and unlock_page_memcg() have too many callers to be
easily replaced in a single patch, so reimplement them as wrappers for
now to be cleaned up later when enough callers have been converted to
use pagesets.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Acked-by: Mike Rapoport <rppt@linux.ibm.com> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Vlastimil Babka <vbabka@suse.cz>
Matthew Wilcox (Oracle) [Tue, 4 May 2021 15:43:01 +0000 (11:43 -0400)]
mm/memcg: Convert mem_cgroup_track_foreign_dirty_slowpath() to pageset
The page was only being used for the memcg and to gather trace
information, so this is a simple conversion. The only caller of
mem_cgroup_track_foreign_dirty() will be converted to pagesets in a later
patch, so doing this now makes that patch simpler.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Vlastimil Babka <vbabka@suse.cz>
Matthew Wilcox (Oracle) [Sun, 2 May 2021 00:42:23 +0000 (20:42 -0400)]
mm/memcg: Convert mem_cgroup_uncharge() to take a pageset
Convert all the callers to call page_pageset(). Most of them were already
using a head page, but a few of them I can't prove were, so this may
actually fix a bug.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Acked-by: Mike Rapoport <rppt@linux.ibm.com> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Vlastimil Babka <vbabka@suse.cz>
Matthew Wilcox (Oracle) [Fri, 25 Jun 2021 13:27:04 +0000 (09:27 -0400)]
mm/memcg: Convert mem_cgroup_charge() to take a pageset
Convert all callers of mem_cgroup_charge() to call page_pageset() on the
page they're currently passing in. Many of them will be converted to
use pagesets themselves soon.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Vlastimil Babka <vbabka@suse.cz>
Matthew Wilcox (Oracle) [Mon, 28 Jun 2021 18:59:26 +0000 (14:59 -0400)]
mm/memcg: Add pageset_memcg() and related functions
memcg information is only stored in the head page, so the memcg
subsystem needs to assure that all accesses are to the head page.
The first step is converting page_memcg() to pageset_memcg().
The callers of page_memcg() and PageMemcgKmem() are not yet ready to be
converted to use pagesets, so retain them as wrappers around pageset_memcg()
and pageset_memcg_kmem(). They will be converted in a later patch set.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Vlastimil Babka <vbabka@suse.cz>
Matthew Wilcox (Oracle) [Fri, 25 Jun 2021 12:57:02 +0000 (08:57 -0400)]
mm/memcg: Remove soft_limit_tree_node()
Opencode this one-line function in its three callers.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Acked-by: Michal Hocko <mhocko@suse.com> Acked-by: Johannes Weiner <hannes@cmpxchg.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Vlastimil Babka <vbabka@suse.cz>
mm/memcg: Use the node id in mem_cgroup_update_tree()
By using the node id in mem_cgroup_update_tree(), we can delete
soft_limit_tree_from_page() and mem_cgroup_page_nodeinfo(). Saves 42
bytes of kernel text on my config.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Acked-by: Michal Hocko <mhocko@suse.com> Acked-by: Johannes Weiner <hannes@cmpxchg.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Vlastimil Babka <vbabka@suse.cz>
mm/memcg: Remove 'page' parameter to mem_cgroup_charge_statistics()
The last use of 'page' was removed by commit 468c398233da ("mm:
memcontrol: switch to native NR_ANON_THPS counter"), so we can now remove
the parameter from the function.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Acked-by: Michal Hocko <mhocko@suse.com> Acked-by: Johannes Weiner <hannes@cmpxchg.org> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Vlastimil Babka <vbabka@suse.cz>
This function is the equivalent of page_mapped(). It is slightly
shorter as we do not need to handle the PageTail() case. Reimplement
page_mapped() as a wrapper around pageset_mapped(). pageset_mapped()
is 13 bytes smaller than page_mapped(), but the page_mapped() wrapper
is 30 bytes, for a net increase of 17 bytes of text.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Acked-by: Vlastimil Babka <vbabka@suse.cz> Reviewed-by: William Kucharski <william.kucharski@oracle.com> Reviewed-by: Christoph Hellwig <hch@lst.de> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com> Acked-by: Mike Rapoport <rppt@linux.ibm.com>
Match the page writeback functions by adding
pageset_start_fscache(), pageset_end_fscache(), pageset_wait_fscache() and
pageset_wait_fscache_killable(). Remove set_page_private_2(). Also rewrite
the kernel-doc to describe when to use the function rather than what the
function does, and include the kernel-doc in the appropriate rst file.
Saves 31 bytes of text in netfs_rreq_unlock() due to set_page_fscache()
calling page_pageset() once instead of three times.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Acked-by: Vlastimil Babka <vbabka@suse.cz> Reviewed-by: William Kucharski <william.kucharski@oracle.com> Reviewed-by: Christoph Hellwig <hch@lst.de> Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com> Acked-by: Mike Rapoport <rppt@linux.ibm.com> Reviewed-by: David Howells <dhowells@redhat.com>
end_page_private_2() becomes pageset_end_private_2(),
wait_on_page_private_2() becomes pageset_wait_private_2() and
wait_on_page_private_2_killable() becomes pageset_wait_private_2_killable().
Adjust the fscache equivalents to call page_pageset() before calling these
functions to avoid adding wrappers. Ends up costing 1 byte of text
in ceph & netfs, but the core shrinks by three calls to page_pageset().
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Acked-by: Vlastimil Babka <vbabka@suse.cz> Reviewed-by: William Kucharski <william.kucharski@oracle.com> Reviewed-by: Christoph Hellwig <hch@lst.de> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Matthew Wilcox (Oracle) [Sat, 16 Jan 2021 16:22:14 +0000 (11:22 -0500)]
mm/filemap: Convert page wait queues to be pagesets
Reinforce that page flags are actually in the head page by changing the
type from page to pageset. Increases the size of cachefiles by two bytes,
but the kernel core is unchanged in size.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Acked-by: Jeff Layton <jlayton@kernel.org> Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com> Acked-by: Vlastimil Babka <vbabka@suse.cz> Reviewed-by: William Kucharski <william.kucharski@oracle.com> Reviewed-by: David Howells <dhowells@redhat.com>
Matthew Wilcox (Oracle) [Fri, 15 Jan 2021 22:14:48 +0000 (17:14 -0500)]
mm/filemap: Add pageset_wake_bit()
Convert wake_up_page_bit() to pageset_wake_bit(). All callers have a pageset,
so use it directly. Saves 66 bytes of text in end_page_private_2().
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Acked-by: Jeff Layton <jlayton@kernel.org> Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com> Acked-by: Vlastimil Babka <vbabka@suse.cz> Reviewed-by: William Kucharski <william.kucharski@oracle.com> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Mike Rapoport <rppt@linux.ibm.com>
Matthew Wilcox (Oracle) [Thu, 4 Mar 2021 17:02:54 +0000 (12:02 -0500)]
mm/filemap: Add pageset_wait_bit()
Rename wait_on_page_bit() to pageset_wait_bit(). We must always wait on
the pageset, otherwise we won't be woken up due to the tail page hashing
to a different bucket from the head page.
This commit shrinks the kernel by 770 bytes, mostly due to moving
the page waitqueue lookup into pageset_wait_bit_common().
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Acked-by: Jeff Layton <jlayton@kernel.org> Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com> Acked-by: Vlastimil Babka <vbabka@suse.cz> Reviewed-by: William Kucharski <william.kucharski@oracle.com> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Mike Rapoport <rppt@linux.ibm.com>
Matthew Wilcox (Oracle) [Thu, 4 Mar 2021 16:25:25 +0000 (11:25 -0500)]
mm/writeback: Add pageset_wait_stable()
Move wait_for_stable_page() into the pageset compatibility file.
pageset_wait_stable() avoids a call to compound_head() and is 14 bytes
smaller than wait_for_stable_page() was. The net text size grows by 16
bytes as a result of this patch. We can also remove thp_head() as this
was the last user.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Acked-by: Jeff Layton <jlayton@kernel.org> Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com> Acked-by: Vlastimil Babka <vbabka@suse.cz> Reviewed-by: William Kucharski <william.kucharski@oracle.com> Reviewed-by: David Howells <dhowells@redhat.com>
Matthew Wilcox (Oracle) [Thu, 4 Mar 2021 16:09:17 +0000 (11:09 -0500)]
mm/writeback: Add pageset_wait_writeback()
wait_on_page_writeback_killable() only has one caller, so convert it to
call pageset_wait_writeback_killable(). For the wait_on_page_writeback()
callers, add a compatibility wrapper around pageset_wait_writeback().
Turning PageWriteback() into pageset_test_writeback() eliminates a call
to compound_head() which saves 8 bytes and 15 bytes in the two
functions. Unfortunately, that is more than offset by adding the
wait_on_page_writeback compatibility wrapper for a net increase in text
of 7 bytes.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Acked-by: Jeff Layton <jlayton@kernel.org> Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com> Acked-by: Vlastimil Babka <vbabka@suse.cz> Reviewed-by: William Kucharski <william.kucharski@oracle.com> Acked-by: Mike Rapoport <rppt@linux.ibm.com> Reviewed-by: David Howells <dhowells@redhat.com>
Matthew Wilcox (Oracle) [Wed, 3 Mar 2021 20:21:55 +0000 (15:21 -0500)]
mm/filemap: Add pageset_end_writeback()
Add an end_page_writeback() wrapper function for users that are not yet
converted to pagesets.
pageset_end_writeback() is less than half the size of end_page_writeback()
at just 105 bytes compared to 228 bytes, due to removing all the
compound_head() calls. The 30 byte wrapper function makes this a net
saving of 93 bytes.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Acked-by: Jeff Layton <jlayton@kernel.org> Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com> Acked-by: Vlastimil Babka <vbabka@suse.cz> Reviewed-by: William Kucharski <william.kucharski@oracle.com> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Mike Rapoport <rppt@linux.ibm.com>
Matthew Wilcox (Oracle) [Tue, 8 Dec 2020 06:25:39 +0000 (01:25 -0500)]
mm/swap: Add pageset_rotate_reclaimable()
Convert rotate_reclaimable_page() to pageset_rotate_reclaimable(). This
eliminates all five of the calls to compound_head() in this function,
saving 75 bytes at the cost of adding 15 bytes to its one caller,
end_page_writeback(). We also save 36 bytes from pagevec_move_tail_fn()
due to using pagesets there. Net 96 bytes savings.
Also move its declaration to mm/internal.h as it's only used by filemap.c.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Acked-by: Vlastimil Babka <vbabka@suse.cz> Reviewed-by: William Kucharski <william.kucharski@oracle.com> Reviewed-by: Christoph Hellwig <hch@lst.de> Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com> Acked-by: Mike Rapoport <rppt@linux.ibm.com> Reviewed-by: David Howells <dhowells@redhat.com>
Matthew Wilcox (Oracle) [Fri, 19 Mar 2021 01:39:45 +0000 (21:39 -0400)]
mm/filemap: Add __pageset_lock_or_retry()
Convert __lock_page_or_retry() to __pageset_lock_or_retry(). This actually
saves 4 bytes in the only caller of lock_page_or_retry() (due to better
register allocation) and saves the 14 byte cost of calling page_pageset()
in __pageset_lock_or_retry() for a total saving of 18 bytes. Also use
a bool for the return type.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Acked-by: Jeff Layton <jlayton@kernel.org> Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com> Reviewed-by: William Kucharski <william.kucharski@oracle.com> Acked-by: Mike Rapoport <rppt@linux.ibm.com> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Vlastimil Babka <vbabka@suse.cz>
Matthew Wilcox (Oracle) [Thu, 4 Mar 2021 15:21:02 +0000 (10:21 -0500)]
mm/filemap: Add pageset_wait_locked()
Also add pageset_wait_locked_killable(). Turn wait_on_page_locked() and
wait_on_page_locked_killable() into wrappers. This eliminates a call
to compound_head() from each call-site, reducing text size by 193 bytes
for me.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Acked-by: Jeff Layton <jlayton@kernel.org> Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com> Acked-by: Vlastimil Babka <vbabka@suse.cz> Reviewed-by: William Kucharski <william.kucharski@oracle.com> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Mike Rapoport <rppt@linux.ibm.com>
Matthew Wilcox (Oracle) [Wed, 30 Dec 2020 22:58:40 +0000 (17:58 -0500)]
mm/filemap: Add __pageset_lock_async()
There aren't any actual callers of lock_page_async(), so remove it.
Convert filemap_update_page() to call __pageset_lock_async().
__pageset_lock_async() is 21 bytes smaller than __lock_page_async(),
but the real savings come from using a pageset in filemap_update_page(),
shrinking it from 515 bytes to 404 bytes, saving 110 bytes. The text
shrinks by 132 bytes in total.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Acked-by: Jeff Layton <jlayton@kernel.org> Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com> Acked-by: Vlastimil Babka <vbabka@suse.cz> Reviewed-by: William Kucharski <william.kucharski@oracle.com> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Mike Rapoport <rppt@linux.ibm.com>
Matthew Wilcox (Oracle) [Tue, 8 Dec 2020 05:07:31 +0000 (00:07 -0500)]
mm/filemap: Add pageset_lock_killable()
This is like lock_page_killable() but for use by callers who
know they have a pageset. Convert __lock_page_killable() to be
__pageset_lock_killable(). This saves one call to compound_head() per
contended call to lock_page_killable().
__pageset_lock_killable() is 19 bytes smaller than __lock_page_killable()
was. filemap_fault() shrinks by 74 bytes and __lock_page_or_retry()
shrinks by 71 bytes. That's a total of 164 bytes of text saved.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Acked-by: Jeff Layton <jlayton@kernel.org> Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com> Acked-by: Vlastimil Babka <vbabka@suse.cz> Reviewed-by: William Kucharski <william.kucharski@oracle.com> Acked-by: Mike Rapoport <rppt@linux.ibm.com> Reviewed-by: David Howells <dhowells@redhat.com>
Matthew Wilcox (Oracle) [Tue, 2 Mar 2021 00:38:25 +0000 (19:38 -0500)]
mm/filemap: Add pageset_lock()
This is like lock_page() but for use by callers who know they have a pageset.
Convert __lock_page() to be __pageset_lock(). This saves one call to
compound_head() per contended call to lock_page().
Saves 455 bytes of text; mostly from improved register allocation and
inlining decisions. __pageset_lock is 59 bytes while __lock_page was 79.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Acked-by: Jeff Layton <jlayton@kernel.org> Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com> Acked-by: Vlastimil Babka <vbabka@suse.cz> Reviewed-by: William Kucharski <william.kucharski@oracle.com> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Mike Rapoport <rppt@linux.ibm.com>
Matthew Wilcox (Oracle) [Mon, 7 Dec 2020 20:44:35 +0000 (15:44 -0500)]
mm/filemap: Add pageset_unlock()
Convert unlock_page() to call pageset_unlock(). By using a pageset we
avoid a call to compound_head(). This shortens the function from 39
bytes to 25 and removes 4 instructions on x86-64. Because we still
have unlock_page(), it's a net increase of 16 bytes of text for the
kernel as a whole, but any path that uses pageset_unlock() will execute
4 fewer instructions.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Acked-by: Jeff Layton <jlayton@kernel.org> Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com> Reviewed-by: William Kucharski <william.kucharski@oracle.com> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Mike Rapoport <rppt@linux.ibm.com> Acked-by: Vlastimil Babka <vbabka@suse.cz>
Matthew Wilcox (Oracle) [Thu, 10 Dec 2020 15:55:05 +0000 (10:55 -0500)]
mm/util: Add pageset_mapping() and pageset_file_mapping()
These are the pageset equivalent of page_mapping() and page_file_mapping().
Add an out-of-line page_mapping() wrapper around pageset_mapping()
in order to prevent the page_pageset() call from bloating every caller
of page_mapping(). Adjust page_file_mapping() and page_mapping_file()
to use pagesets internally. Rename __page_file_mapping() to
swapcache_mapping() and change it to take a pageset.
This ends up saving 122 bytes of text overall. pageset_mapping() is
45 bytes shorter than page_mapping() was, but the new page_mapping()
wrapper is 30 bytes. The major reduction is a few bytes less in dozens
of nfs functions (which call page_file_mapping()). Most of these appear
to be a slight change in gcc's register allocation decisions, which allow:
for a reduction of a single byte. Once the NFS client is converted to
use pagesets, this entire sequence will disappear.
Also add pageset_mapping() documentation.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Acked-by: Jeff Layton <jlayton@kernel.org> Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com> Acked-by: Vlastimil Babka <vbabka@suse.cz> Reviewed-by: William Kucharski <william.kucharski@oracle.com> Reviewed-by: David Howells <dhowells@redhat.com>
Matthew Wilcox (Oracle) [Thu, 24 Dec 2020 12:25:19 +0000 (07:25 -0500)]
mm/filemap: Add pageset_pos() and pageset_file_pos()
These are just wrappers around page_offset() and page_file_offset()
respectively. No change to generated code.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Acked-by: Jeff Layton <jlayton@kernel.org> Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com> Acked-by: Vlastimil Babka <vbabka@suse.cz> Reviewed-by: William Kucharski <william.kucharski@oracle.com> Reviewed-by: David Howells <dhowells@redhat.com>
Matthew Wilcox (Oracle) [Sun, 21 Mar 2021 20:24:31 +0000 (16:24 -0400)]
mm/filemap: Add pageset_next_index()
This helper returns the page index of the next pageset in the file (ie
the end of this pageset, plus one).
No changes to generated code.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Acked-by: Jeff Layton <jlayton@kernel.org> Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com> Acked-by: Vlastimil Babka <vbabka@suse.cz> Reviewed-by: William Kucharski <william.kucharski@oracle.com> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Mike Rapoport <rppt@linux.ibm.com>
Matthew Wilcox (Oracle) [Sat, 16 Jan 2021 04:39:21 +0000 (23:39 -0500)]
mm/filemap: Add pageset_index(), pageset_file_page() and pageset_contains()
pageset_index() is the equivalent of page_index() for pagesets.
pageset_file_page() is the equivalent of find_subpage().
pageset_contains() is the equivalent of thp_contains().
No changes to generated code.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Acked-by: Jeff Layton <jlayton@kernel.org> Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com> Acked-by: Vlastimil Babka <vbabka@suse.cz> Reviewed-by: William Kucharski <william.kucharski@oracle.com> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Mike Rapoport <rppt@linux.ibm.com>
Matthew Wilcox (Oracle) [Mon, 11 Jan 2021 15:04:40 +0000 (10:04 -0500)]
mm: Handle per-pageset private data
Add pageset_get_private() which mirrors page_private() -- ie pageset private
data is the same as page private data. The only difference is that these
return a void * instead of an unsigned long, which matches the majority
of users.
Turn attach_page_private() into pageset_attach_private() and reimplement
attach_page_private() as a wrapper. No filesystem which uses page private
data currently supports compound pages, so we're free to define the rules.
attach_page_private() may only be called on a head page; if you want
to add private data to a tail page, you can call set_page_private()
directly (and shouldn't increment the page refcount! That should be
done when adding private data to the head page / pageset).
This saves 813 bytes of text with the distro-derived config that I'm
testing due to removing the calls to compound_head() in get_page()
& put_page().
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Acked-by: Jeff Layton <jlayton@kernel.org> Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com> Acked-by: Vlastimil Babka <vbabka@suse.cz> Reviewed-by: William Kucharski <william.kucharski@oracle.com> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Mike Rapoport <rppt@linux.ibm.com>
Matthew Wilcox (Oracle) [Thu, 25 Feb 2021 14:47:41 +0000 (09:47 -0500)]
mm/lru: Add pageset LRU functions
Handle arbitrary-order pagesets being added to the LRU. By definition,
all pages being added to the LRU were already head or base pages, but
call page_pageset() on them anyway to get the type right and avoid the
buried calls to compound_head().
Saves 783 bytes of kernel text; no functions grow.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Yu Zhao <yuzhao@google.com> Reviewed-by: Christoph Hellwig <hch@lst.de> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com> Acked-by: Mike Rapoport <rppt@linux.ibm.com> Acked-by: Vlastimil Babka <vbabka@suse.cz>
Matthew Wilcox (Oracle) [Mon, 7 Dec 2020 20:42:09 +0000 (15:42 -0500)]
mm: Add pageset flag manipulation functions
These new functions are the pageset analogues of the various PageFlags
functions. If CONFIG_DEBUG_VM_PGFLAGS is enabled, we check the pageset
is not a tail page at every invocation. This will also catch the
PagePoisoned case as a poisoned page has every bit set, which would
include PageTail.
This saves 1684 bytes of text with the distro-derived config that
I'm testing due to removing a double call to compound_head() in
PageSwapCache().
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Acked-by: Jeff Layton <jlayton@kernel.org> Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com> Acked-by: Vlastimil Babka <vbabka@suse.cz> Reviewed-by: William Kucharski <william.kucharski@oracle.com> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Mike Rapoport <rppt@linux.ibm.com>
Matthew Wilcox (Oracle) [Mon, 10 May 2021 20:33:22 +0000 (16:33 -0400)]
mm: Add pageset_try_get_rcu()
This is the equivalent of page_cache_get_speculative(). Also add
pageset_ref_try_add_rcu (the equivalent of page_cache_add_speculative)
and pageset_get_unless_zero() (the equivalent of get_page_unless_zero()).
The new kernel-doc attempts to explain from the user's point of view
when to use pageset_try_get_rcu() and when to use pageset_get_unless_zero(),
because there seems to be some confusion currently between the users of
page_cache_get_speculative() and get_page_unless_zero().
Reimplement page_cache_add_speculative() and page_cache_get_speculative()
as wrappers around the pageset equivalents, but leave get_page_unless_zero()
alone for now. This commit reduces text size by 3 bytes due to slightly
different register allocation & instruction selections.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Acked-by: Vlastimil Babka <vbabka@suse.cz> Reviewed-by: William Kucharski <william.kucharski@oracle.com> Reviewed-by: Christoph Hellwig <hch@lst.de> Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com> Acked-by: Mike Rapoport <rppt@linux.ibm.com>
Matthew Wilcox (Oracle) [Mon, 7 Dec 2020 04:04:57 +0000 (23:04 -0500)]
mm: Add pageset_get()
If we know we have a pageset, we can call pageset_get() instead
of get_page() and save the overhead of calling compound_head().
No change to generated code.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Zi Yan <ziy@nvidia.com> Reviewed-by: Christoph Hellwig <hch@lst.de> Acked-by: Jeff Layton <jlayton@kernel.org> Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com> Acked-by: Vlastimil Babka <vbabka@suse.cz> Reviewed-by: William Kucharski <william.kucharski@oracle.com> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Mike Rapoport <rppt@linux.ibm.com>
Matthew Wilcox (Oracle) [Mon, 7 Dec 2020 04:04:57 +0000 (23:04 -0500)]
mm: Add pageset_put()
If we know we have a pageset, we can call pageset_put() instead of put_page()
and save the overhead of calling compound_head(). Also skips the
devmap checks.
This commit looks like it should be a no-op, but actually saves 684 bytes
of text with the distro-derived config that I'm testing. Some functions
grow a little while others shrink. I presume the compiler is making
different inlining decisions.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Zi Yan <ziy@nvidia.com> Reviewed-by: Christoph Hellwig <hch@lst.de> Acked-by: Jeff Layton <jlayton@kernel.org> Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com> Acked-by: Vlastimil Babka <vbabka@suse.cz> Reviewed-by: William Kucharski <william.kucharski@oracle.com> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Mike Rapoport <rppt@linux.ibm.com>
Matthew Wilcox (Oracle) [Wed, 31 Mar 2021 14:39:55 +0000 (10:39 -0400)]
mm: Add pageset reference count functions
These functions mirror their page reference counterparts. Also add
the kernel-doc to the mm-api and correct the return type of
page_ref_add_unless() to bool. No change to generated code.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Acked-by: Jeff Layton <jlayton@kernel.org> Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com> Acked-by: Vlastimil Babka <vbabka@suse.cz> Reviewed-by: William Kucharski <william.kucharski@oracle.com> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Mike Rapoport <rppt@linux.ibm.com>
Matthew Wilcox (Oracle) [Fri, 15 Jan 2021 15:52:37 +0000 (10:52 -0500)]
mm/debug: Add VM_BUG_ON_PAGESET() and VM_WARN_ON_ONCE_PAGESET()
These are the pageset equivalents of VM_BUG_ON_PAGE and
VM_WARN_ON_ONCE_PAGE. No change to generated code.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Zi Yan <ziy@nvidia.com> Reviewed-by: Christoph Hellwig <hch@lst.de> Acked-by: Jeff Layton <jlayton@kernel.org> Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com> Acked-by: Vlastimil Babka <vbabka@suse.cz> Reviewed-by: William Kucharski <william.kucharski@oracle.com> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Mike Rapoport <rppt@linux.ibm.com>
Matthew Wilcox (Oracle) [Mon, 18 Jan 2021 13:14:00 +0000 (08:14 -0500)]
mm/vmstat: Add functions to account pageset statistics
Allow page counters to be more readily modified by callers which have
a pageset. Name these wrappers with 'stat' instead of 'state' as requested
by Linus here:
https://lore.kernel.org/linux-mm/CAHk-=wj847SudR-kt+46fT3+xFFgiwpgThvm7DJWGdi4cVrbnQ@mail.gmail.com/
No change to generated code.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Acked-by: Jeff Layton <jlayton@kernel.org> Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com> Acked-by: Vlastimil Babka <vbabka@suse.cz> Reviewed-by: William Kucharski <william.kucharski@oracle.com> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Mike Rapoport <rppt@linux.ibm.com>
Matthew Wilcox (Oracle) [Mon, 18 Jan 2021 12:40:36 +0000 (07:40 -0500)]
mm: Add pageset_pgdat(), pageset_zone() and pageset_zonenum()
These are just convenience wrappers for callers with pagesets; pgdat and
zone can be reached from tail pages as well as head pages. No change
to generated code.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Zi Yan <ziy@nvidia.com> Reviewed-by: Christoph Hellwig <hch@lst.de> Acked-by: Jeff Layton <jlayton@kernel.org> Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com> Acked-by: Vlastimil Babka <vbabka@suse.cz> Reviewed-by: William Kucharski <william.kucharski@oracle.com> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Mike Rapoport <rppt@linux.ibm.com>
Matthew Wilcox (Oracle) [Mon, 7 Dec 2020 03:22:48 +0000 (22:22 -0500)]
mm: Introduce struct pageset
A struct pageset is a new abstraction to replace the venerable struct page.
A function which takes a struct pageset argument declares that it will
operate on the entire (possibly compound) page, not just PAGE_SIZE bytes.
In return, the caller guarantees that the pointer it is passing does
not point to a tail page. No change to generated code.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Acked-by: Jeff Layton <jlayton@kernel.org> Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com> Acked-by: Vlastimil Babka <vbabka@suse.cz> Reviewed-by: William Kucharski <william.kucharski@oracle.com> Reviewed-by: Christoph Hellwig <hch@lst.de> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Mike Rapoport <rppt@linux.ibm.com>
Matthew Wilcox (Oracle) [Sun, 6 Jun 2021 03:20:17 +0000 (23:20 -0400)]
mm: Convert get_page_unless_zero() to return bool
atomic_add_unless() returns bool, so remove the widening casts to int
in page_ref_add_unless() and get_page_unless_zero(). This causes gcc
to produce slightly larger code in isolate_migratepages_block(), but
it's not clear that it's worse code. Net +19 bytes of text.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com> Reviewed-by: David Howells <dhowells@redhat.com> Acked-by: Vlastimil Babka <vbabka@suse.cz>
Merge tag 'xfs-5.14-fixes-1' of git://git.kernel.org/pub/scm/fs/xfs/xfs-linux
Pull xfs fixes from Darrick Wong:
"A few fixes for issues in the new online shrink code, additional
corrections for my recent bug-hunt w.r.t. extent size hints on
realtime, and improved input checking of the GROWFSRT ioctl.
IOW, the usual 'I somehow got bored during the merge window and
resumed auditing the farther reaches of xfs':
- Fix shrink eligibility checking when sparse inode clusters enabled
- Reset '..' directory entries when unlinking directories to prevent
verifier errors if fs is shrinked later
- Don't report unusable extent size hints to FSGETXATTR
- Don't warn when extent size hints are unusable because the sysadmin
configured them that way
- Fix insufficient parameter validation in GROWFSRT ioctl
- Fix integer overflow when adding rt volumes to filesystem"
* tag 'xfs-5.14-fixes-1' of git://git.kernel.org/pub/scm/fs/xfs/xfs-linux:
xfs: detect misaligned rtinherit directory extent size hints
xfs: fix an integer overflow error in xfs_growfs_rt
xfs: improve FSGROWFSRT precondition checking
xfs: don't expose misaligned extszinherit hints to userspace
xfs: correct the narrative around misaligned rtinherit/extszinherit dirs
xfs: reset child dir '..' entry when unlinking child
xfs: check for sparse inode clusters that cross new EOAG when shrinking
Merge tag 'iomap-5.14-fixes-1' of git://git.kernel.org/pub/scm/fs/xfs/xfs-linux
Pull iomap fixes from Darrick Wong:
"A handful of bugfixes for the iomap code.
There's nothing especially exciting here, just fixes for UBSAN (not
KASAN as I erroneously wrote in the tag message) warnings about
undefined behavior in the SEEK_DATA/SEEK_HOLE code, and some
reshuffling of per-page block state info to fix some problems with
gfs2.
- Fix KASAN warnings due to integer overflow in SEEK_DATA/SEEK_HOLE
- Fix assertion errors when using inlinedata files on gfs2"
* tag 'iomap-5.14-fixes-1' of git://git.kernel.org/pub/scm/fs/xfs/xfs-linux:
iomap: Don't create iomap_page objects in iomap_page_mkwrite_actor
iomap: Don't create iomap_page objects for inline files
iomap: Permit pages without an iop to enter writeback
iomap: remove the length variable in iomap_seek_hole
iomap: remove the length variable in iomap_seek_data
Merge tag 'kbuild-fixes-v5.14' of git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild
Pull Kbuild fixes from Masahiro Yamada:
- Restore the original behavior of scripts/setlocalversion when
LOCALVERSION is set to empty.
- Show Kconfig prompts even for 'make -s'
- Fix the combination of COFNIG_LTO_CLANG=y and CONFIG_MODVERSIONS=y
for older GNU Make versions
* tag 'kbuild-fixes-v5.14' of git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild:
Documentation: Fix intiramfs script name
Kbuild: lto: fix module versionings mismatch in GNU make 3.X
kbuild: do not suppress Kconfig prompts for silent build
scripts/setlocalversion: fix a bug when LOCALVERSION is empty
Robert Richter [Thu, 15 Jul 2021 09:26:02 +0000 (11:26 +0200)]
Documentation: Fix intiramfs script name
Documentation was not changed when renaming the script in commit 80e715a06c2d ("initramfs: rename gen_initramfs_list.sh to
gen_initramfs.sh"). Fixing this.
Basically does:
$ sed -i -e s/gen_initramfs_list.sh/gen_initramfs.sh/g $(git grep -l gen_initramfs_list.sh)
Fixes: 80e715a06c2d ("initramfs: rename gen_initramfs_list.sh to gen_initramfs.sh") Signed-off-by: Robert Richter <rrichter@amd.com> Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Kbuild: lto: fix module versionings mismatch in GNU make 3.X
When building modules(CONFIG_...=m), I found some of module versions
are incorrect and set to 0.
This can be found in build log for first clean build which shows
WARNING: EXPORT symbol "XXXX" [drivers/XXX/XXX.ko] version generation failed,
symbol will not be versioned.
But in second build(incremental build), the WARNING disappeared and the
module version becomes valid CRC and make someone who want to change
modules without updating kernel image can't insert their modules.
The problematic code is
+ $(foreach n, $(filter-out FORCE,$^), \
+ $(if $(wildcard $(n).symversions), \
+ ; cat $(n).symversions >> $@.symversions))
`foreach n` shows nothing to `cat` into $(n).symversions because
`if $(wildcard $(n).symversions)` return nothing, but actually
they do exist during this line was executed.
-rw-r--r-- 1 root root 168580 Jun 13 19:10 fs/notify/fsnotify.o
-rw-r--r-- 1 root root 111 Jun 13 19:10 fs/notify/fsnotify.o.symversions
The reason is the $(n).symversions are generated at runtime, but
Makefile wildcard function expends and checks the file exist or not
during parsing the Makefile.
Thus fix this by use `test` shell command to check the file
existence in runtime.
Rebase from both:
1. [https://lore.kernel.org/lkml/20210616080252.32046-1-lecopzer.chen@mediatek.com/]
2. [https://lore.kernel.org/lkml/20210702032943.7865-1-lecopzer.chen@mediatek.com/]
kbuild: do not suppress Kconfig prompts for silent build
When a new CONFIG option is available, Kbuild shows a prompt to get
the user input.
$ make
[ snip ]
Core Scheduling for SMT (SCHED_CORE) [N/y/?] (NEW)
This is the only interactive place in the build process.
Commit 174a1dcc9642 ("kbuild: sink stdout from cmd for silent build")
suppressed Kconfig prompts as well because syncconfig is invoked by
the 'cmd' macro. You cannot notice the fact that Kconfig is waiting
for the user input.
Use 'kecho' to show the equivalent short log without suppressing stdout
from sub-make.
scripts/setlocalversion: fix a bug when LOCALVERSION is empty
The commit 042da426f8eb ("scripts/setlocalversion: simplify the short
version part") reduces indentation. Unfortunately, it also changes behavior
in a subtle way - if the user has empty "LOCALVERSION" variable, the plus
sign is appended to the kernel version. It wasn't appended before.
This patch reverts to the old behavior - we append the plus sign only if
the LOCALVERSION variable is not set.
Fixes: 042da426f8eb ("scripts/setlocalversion: simplify the short version part") Signed-off-by: Mikulas Patocka <mpatocka@redhat.com> Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
It's not clear why, but it causes unexplained problems in entirely
unrelated xfs code. The most likely explanation is some slab
corruption, possibly triggered due to CONFIG_SLUB_DEBUG_ON. See [1].
It ends up having a few other problems too, like build errors on
arch/arc, and Geert reporting it using much more memory on m68k [3] (it
probably does so elsewhere too, but it is probably just more noticeable
on m68k).
The architecture issues (both build and memory use) are likely just
because this change effectively force-enabled STACKDEPOT (along with a
very bad default value for the stackdepot hash size). But together with
the xfs issue, this all smells like "this commit was not ready" to me.
Merge tag 'scsi-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi
Pull SCSI fixes from James Bottomley:
"One core fix for an oops which can occur if the error handling thread
fails to start for some reason and the driver is removed.
The other fixes are all minor ones in drivers"
* tag 'scsi-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi:
scsi: ufs: core: Add missing host_lock in ufshcd_vops_setup_xfer_req()
scsi: mpi3mr: Fix W=1 compilation warnings
scsi: pm8001: Clean up kernel-doc and comments
scsi: zfcp: Report port fc_security as unknown early during remote cable pull
scsi: core: Fix bad pointer dereference when ehandler kthread is invalid
scsi: fas216: Fix a build error
scsi: core: Fix the documentation of the scsi_execute() time parameter
projects / users / willy / pagecache.git / log