mm/mmap: Change do_mas_align_munmap() to static

Add static to the declaration as reported by sparse

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

lib/maple_tree: Fix rcu annotations

Add all the missing __rcu annotations reported by sparse.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>

mm/mmap: Lock maple tree when using mas_store_gfp()

When storing to the maple tree, take the write lock to avoid
entering/exiting the use of RCU during a write operation.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

lib/maple_tree: Fix mas_pop_node() zeroing.

When using a single allocation in the node, zero out slot 0 of the node
being returned.

Also, small whitespace fix in mas_push

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/ksm: Fix compile warning and whitespace for 3fd77bfa4f25fa

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

lib/maple_tree: Fix mas_store_gfp() on spanning store.

Spanning store detection needs to be more robust than checking if it spans the end of
a node.  Fix the detection by using mas_is_span_wr().  Also ensure it's not MAS_NONE.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

lib/maple_tree: Add check for dead nodes in mt_validate()

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: __insert_vm_struct() only needs to mas_reset()

vma_mas_store() will set the range again.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mprotect: Use vma_next() of prev.

The linked list used to use prev->next.  the maple state may be invalidated
by the mprotect_fixup call which may alter the tree.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mprotect: Use vma_next() instead of mas_next().

mas is in the wrong state and would potentially return the current vma twice.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Fix mmap_region() maple tree store.

mas_set_range() before vma_mas_link() as the address may have changed
as commented above in the code.

Do the same in the error path.

This forces a reset of the mas->node to MA_START.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: ULONG_MAX in exit_mmap

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: ULONG_MAX in do_brk_munmap

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: ULONG_MAX in detach_range

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: ULONG_MAX in remove_mt

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mprotect: Change do_mprotect_pkey()  -1 to ULONG_MAX

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mempolicy: Fix mbind_range -1 to ULONG_MAX

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

kernel/sched/fair: Change -1 to ULONG_MAX

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

kernel/acct: Change -1 to ULONG_MAX

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

task_mmu: Change -1 to ULONG_MAX

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

ppc book3s32: Change -1 to ULONG_MAX

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

lib/maple_tree: Reset request_count to 0 of old head on push

Just to be safe

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Add validate_mm() to __vma_adjust() start

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mprotect: Fix adcfbd7698558a linked list removal

Use mas_find() in do_mprotect_pkey to find the next vma to better align
with what happened with the old API

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

kernel/sched/fair: Fix linked list fallout 4acb2b9560cd8f

When searching for the vma, use mas_find, not mas_walk() to better do what was happening before

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mempolicy: fix adcfbd7698558a

When removing the linked list, a bug was introduced in maple state
iterator used for the mbind_range() function.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: do_brk_munmap() whitespace fix

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

maple_tree: use ma_is_root() in mas_replace().

Instead of using mte_is_root(), use ma_is_root() directly since the maple node
is known already.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Find next vma in do_brk_munmap() prior to updating tree.

The tree update may invalidate the location and cause a longer search if the VMA does not
preceed a NULL entry

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Fix do_brk_munmap() when munmapping an entire VMA or more

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Add more validate_mm() calls

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm: Return a bool from anon_vma_interval_tree_verify()

Added to allow printing which vma has the issue

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: expand_upwards() shouldn't call validate_mm()

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: mmap_region() fixes for when expanding vma as well as anon

When expanding a vma, call khugepaged_enter_vma_merge() with correct vma.

Also, set the vm_pgoff to the correct value when adding an anon mapping.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Fix detach_range() and do_mas_align_munmap()

When searching for the first entry overlapp in do_mas_align_munmap(), the mas->last
will be modified if one is found, and if the maple state isn't reset for splitting,
then the wrong end will be past to detach_range() in the maple state.

detach_range() no longer needs both vma and vm_area_struct last, so make that change
too.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: vma_expand() Cleanup code a bit

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Fix linked list damage from e40a951e09ed0e

Drop unnecessary while() after mas_for_each()

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

fs/proc/task_mmu: Fix linked list damage from c4e6b1dfcb6f99

Fix reset of mas and initialize vma_start.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/ksm: Fix error introduced in 3fd77bfa4f25fa

mm may be uninitialized during the first loop.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Set next to NULL in mmap_region()

When following cannot_expand goto, next would not be set

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Remove validate from expand() as the lock isn't held.

Add validate_mm() to many other locations.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Don't reset ma_neighbour in brk() as it won't be used

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Fix alignment mess of gap searching

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Assert lock is held before running validate_mm()

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Drop unused ma_lock from remap_file_pages

The variable is not used.

Reported-by: kernel test robot <lkp@intel.com>
Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Fix error path not checking anon_vma for null

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

maple_tree: Rework mas_replace()

Drop ma_set_slot() and mte_set_slot() in favour of just setting the slot
value in the parent.  This is the only function using those two
functions and they are not necessary.

Drop check for replacing the node with itself as that doesn't happen -
confirmed by running the test suite with a BUG_ON instead of the swap.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mempolicy: Fix mas_for_each() loop

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

lib/maple_tree: Fix allocation walk iterators.

Do not reset mas->node or mas->offset and advance the slot on
continuing.

This also drops mas_rev_awalk() and instead pulls it into
mas_empty_area_rev().

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Fix exit_mmap() maple state.

Do not reset the maple state prior to unmap_vmas().

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Fux do_mas_align_munmap() search for vma.

Use find instead of walk to search for the VMA to ensure a VMA is found
if the start does not fall on a VMA

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Change gap calculations

When calculating the gaps, make sure the alignment is okay.

This needs further analysis.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/memory: Fix zap_page_range

Use mas_for_each to avoid calling unmap_single_vma for the start VMA twice.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/memory: Whitespace fix

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Fix validate_mm() warning

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Remove rb and vma_link() comment on bug.

This bug does not exist for maple tree

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Use mtree_init() in mt_destroy_walk()

Don't use MTREE_INIT()

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Use mtree_init() in do_mas_align_munmap()

Don't use MTREE_INIT()

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Convert __insert_vm_struct to use mas, convert vma_link to use vma_mas_link()

Drop unused vma_store()

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm: Remove vma linked list.

The vma linked list has been replaced by the maple tree iterators and
vma_next() vma_prev() functions.

A part of this change is also the iterators free_pgd_range(),
zap_page_range(), and unmap_single_vma()

The internal _vma_next() has been dropped.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/util: Remove __vma_link_list() and __vma_unlink_list()

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/nommu: Stop inserting into the vma linked list

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/swapfile: Use maple tree iterator instead of vma linked list

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/pagewalk: Use vma_next() instead of vma linked list

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/oom_kill: Use maple tree iterators instead of vma linked list

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/nommu: Use maple tree iterators instead of vma linked list

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/msync: Use vma_next() instead of vma linked list

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mremap: Use vma_next() instead of vma linked list

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mprotect: Use maple tree navigation instead of vma linked list

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mlock: Use maple tree iterators instead of vma linked list

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mempolicy: Use maple tree iterators instead of vma linked list

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/memcontrol: Stop using mm->highest_vm_end

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/madvise: Use vma_next instead of vma linked list

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/ksm: Use maple tree iterators instead of vma linked list

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/khugepaged: Use maple tree iterators instead of vma linked list

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/huge_memory: Use vma_next() instead of vma linked list

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/gup: Use maple tree navigation instead of linked list

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

kernel/sys: Use maple tree iterators instead of linked list

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

kernel/sched/fair: Use maple tree iterators instead of linked list

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

kernel/events/uprobes: Use maple tree iterators instead of linked list

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

kernel/events/core: Use maple tree iterators instead of linked list

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

kernel/acct: Use maple tree iterators instead of linked list

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

ipc/shm: Stop using the vma linked list

When searching for a VMA, a maple state can be used instead of the linked list in
the mm_struct

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

fs/userfaultfd: Stop using vma linked list.

Don't use the mm_struct linked list or the vma->vm_next in prep for removal

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

fs/proc/task_mmu: Stop using linked list and highest_vm_end

Remove references to mm_struct linked list and highest_vm_end for when they are removed

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

fs/proc/base: Use maple tree iterators in place of linked list

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

fs/exec: Use vma_next() instead of linked list

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

fs/coredump: Use maple tree iterators in place of linked list

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

fs/binfmt_elf: Use maple tree iterators for fill_files_note()

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

drivers/tee/optee: Use maple tree iterators for __check_mem_type()

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

drivers/oprofile: Lookup address in tree instead of linked list.

Use the vma interface to find the vma if one exists instead of the linked list

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

drivers/misc/cxl: Use maple tree iterators for cxl_prefault_vma()

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

arch/xtensa: Use maple tree iterators for unmapped area

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

arch/x86: Use maple tree iterators for vdso/vma

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

arch/um: Use maple tree iterators instead of linked list

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

arch/s390: Use maple tree iterators instead of linked list.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

arch/powerpc: Optimize cell spu task sync.

Use the vma api to look up the spu reference instead of walking the linked list.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

arch/powerpc: Remove mmap linked list from mm/book2s32/subpage_prot

Start using the maple tree

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

arch/powerpc: Remove mmap linked list from mm/book2s32/tlb

Start using the maple tree

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

arch/parsic: Remove mmap linked list from kernel/cache

Start using the maple tree

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>