mm/mmap: Add validate_mm() to __vma_adjust() start

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mprotect: Fix adcfbd7698558a linked list removal

Use mas_find() in do_mprotect_pkey to find the next vma to better align
with what happened with the old API

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

kernel/sched/fair: Fix linked list fallout 4acb2b9560cd8f

When searching for the vma, use mas_find, not mas_walk() to better do what was happening before

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mempolicy: fix adcfbd7698558a

When removing the linked list, a bug was introduced in maple state
iterator used for the mbind_range() function.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: do_brk_munmap() whitespace fix

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

maple_tree: use ma_is_root() in mas_replace().

Instead of using mte_is_root(), use ma_is_root() directly since the maple node
is known already.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Find next vma in do_brk_munmap() prior to updating tree.

The tree update may invalidate the location and cause a longer search if the VMA does not
preceed a NULL entry

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Fix do_brk_munmap() when munmapping an entire VMA or more

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Add more validate_mm() calls

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm: Return a bool from anon_vma_interval_tree_verify()

Added to allow printing which vma has the issue

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: expand_upwards() shouldn't call validate_mm()

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: mmap_region() fixes for when expanding vma as well as anon

When expanding a vma, call khugepaged_enter_vma_merge() with correct vma.

Also, set the vm_pgoff to the correct value when adding an anon mapping.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Fix detach_range() and do_mas_align_munmap()

When searching for the first entry overlapp in do_mas_align_munmap(), the mas->last
will be modified if one is found, and if the maple state isn't reset for splitting,
then the wrong end will be past to detach_range() in the maple state.

detach_range() no longer needs both vma and vm_area_struct last, so make that change
too.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: vma_expand() Cleanup code a bit

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Fix linked list damage from e40a951e09ed0e

Drop unnecessary while() after mas_for_each()

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

fs/proc/task_mmu: Fix linked list damage from c4e6b1dfcb6f99

Fix reset of mas and initialize vma_start.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/ksm: Fix error introduced in 3fd77bfa4f25fa

mm may be uninitialized during the first loop.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Set next to NULL in mmap_region()

When following cannot_expand goto, next would not be set

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Remove validate from expand() as the lock isn't held.

Add validate_mm() to many other locations.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Don't reset ma_neighbour in brk() as it won't be used

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Fix alignment mess of gap searching

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Assert lock is held before running validate_mm()

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Drop unused ma_lock from remap_file_pages

The variable is not used.

Reported-by: kernel test robot <lkp@intel.com>
Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Fix error path not checking anon_vma for null

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

maple_tree: Rework mas_replace()

Drop ma_set_slot() and mte_set_slot() in favour of just setting the slot
value in the parent.  This is the only function using those two
functions and they are not necessary.

Drop check for replacing the node with itself as that doesn't happen -
confirmed by running the test suite with a BUG_ON instead of the swap.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mempolicy: Fix mas_for_each() loop

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

lib/maple_tree: Fix allocation walk iterators.

Do not reset mas->node or mas->offset and advance the slot on
continuing.

This also drops mas_rev_awalk() and instead pulls it into
mas_empty_area_rev().

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Fix exit_mmap() maple state.

Do not reset the maple state prior to unmap_vmas().

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Fux do_mas_align_munmap() search for vma.

Use find instead of walk to search for the VMA to ensure a VMA is found
if the start does not fall on a VMA

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Change gap calculations

When calculating the gaps, make sure the alignment is okay.

This needs further analysis.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/memory: Fix zap_page_range

Use mas_for_each to avoid calling unmap_single_vma for the start VMA twice.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/memory: Whitespace fix

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Fix validate_mm() warning

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Remove rb and vma_link() comment on bug.

This bug does not exist for maple tree

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Use mtree_init() in mt_destroy_walk()

Don't use MTREE_INIT()

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Use mtree_init() in do_mas_align_munmap()

Don't use MTREE_INIT()

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Convert __insert_vm_struct to use mas, convert vma_link to use vma_mas_link()

Drop unused vma_store()

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm: Remove vma linked list.

The vma linked list has been replaced by the maple tree iterators and
vma_next() vma_prev() functions.

A part of this change is also the iterators free_pgd_range(),
zap_page_range(), and unmap_single_vma()

The internal _vma_next() has been dropped.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/util: Remove __vma_link_list() and __vma_unlink_list()

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/nommu: Stop inserting into the vma linked list

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/swapfile: Use maple tree iterator instead of vma linked list

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/pagewalk: Use vma_next() instead of vma linked list

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/oom_kill: Use maple tree iterators instead of vma linked list

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/nommu: Use maple tree iterators instead of vma linked list

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/msync: Use vma_next() instead of vma linked list

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mremap: Use vma_next() instead of vma linked list

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mprotect: Use maple tree navigation instead of vma linked list

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mlock: Use maple tree iterators instead of vma linked list

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mempolicy: Use maple tree iterators instead of vma linked list

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/memcontrol: Stop using mm->highest_vm_end

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/madvise: Use vma_next instead of vma linked list

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/ksm: Use maple tree iterators instead of vma linked list

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/khugepaged: Use maple tree iterators instead of vma linked list

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/huge_memory: Use vma_next() instead of vma linked list

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/gup: Use maple tree navigation instead of linked list

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

kernel/sys: Use maple tree iterators instead of linked list

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

kernel/sched/fair: Use maple tree iterators instead of linked list

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

kernel/events/uprobes: Use maple tree iterators instead of linked list

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

kernel/events/core: Use maple tree iterators instead of linked list

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

kernel/acct: Use maple tree iterators instead of linked list

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

ipc/shm: Stop using the vma linked list

When searching for a VMA, a maple state can be used instead of the linked list in
the mm_struct

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

fs/userfaultfd: Stop using vma linked list.

Don't use the mm_struct linked list or the vma->vm_next in prep for removal

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

fs/proc/task_mmu: Stop using linked list and highest_vm_end

Remove references to mm_struct linked list and highest_vm_end for when they are removed

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

fs/proc/base: Use maple tree iterators in place of linked list

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

fs/exec: Use vma_next() instead of linked list

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

fs/coredump: Use maple tree iterators in place of linked list

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

fs/binfmt_elf: Use maple tree iterators for fill_files_note()

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

drivers/tee/optee: Use maple tree iterators for __check_mem_type()

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

drivers/oprofile: Lookup address in tree instead of linked list.

Use the vma interface to find the vma if one exists instead of the linked list

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

drivers/misc/cxl: Use maple tree iterators for cxl_prefault_vma()

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

arch/xtensa: Use maple tree iterators for unmapped area

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

arch/x86: Use maple tree iterators for vdso/vma

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

arch/um: Use maple tree iterators instead of linked list

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

arch/s390: Use maple tree iterators instead of linked list.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

arch/powerpc: Optimize cell spu task sync.

Use the vma api to look up the spu reference instead of walking the linked list.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

arch/powerpc: Remove mmap linked list from mm/book2s32/subpage_prot

Start using the maple tree

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

arch/powerpc: Remove mmap linked list from mm/book2s32/tlb

Start using the maple tree

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

arch/parsic: Remove mmap linked list from kernel/cache

Start using the maple tree

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

arch/arm64: Remove mmap linked list from vdso.

Start using the maple tree

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm: Introduce vma_next() and vma_prev()

Rename internam vma_next() to _vma_next().

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mmap: make remove_vma_list() inline

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Change do_brk_munmap() to use do_mas_align_munmap()

do_brk_munmap() already has aligned addresses.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mmap: Remove __do_munmap() in favour of do_mas_munmap()

Export new interface and use it in place of old interface.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mmap: Use find_vma_intersection in do_mmap() for overlap

When detecting a conflict with MAP_FIXED_NOREPLACE, using the new interface avoids
the need for a temp variable

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Add do_mas_munmap() and wraper for __do_munmap()

To avoid extra tree work, it is necessary to support passing in a maple state
to key functions.  Start this work with __do_munmap().

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Move mmap_region() below do_munmap()

Relocation of code for the next commit.  There should be no changes
here.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Change __do_munmap() to avoid unnecessary lookups.

As there is no longer a vmacache, find_vma() is more expensive and so
avoid doing them

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm: Remove vmacache

The maple tree is able to find a VMA quick enough to no longer need the
vma cache.  Remove the vmacache to reduce work in keeping it up to date
and code complexity.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Drop munmap_vma_range()

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Change mmap_region to use maple tree state

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm: Change find_vma_intersection to maple tree and make find_vma to
inline.

Move find_vma_intersection() to mmap.c and change implementation to
maple tree.

When searching for a vma within a range, it is easier to use the maple
tree interface.  This means the find_vma() call changes to a special
case of the find_vma_intersection().  Exported for kvm module.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Change vm_brk_flags() to use mm_populate_vma()

Skip searching for the vma to populate after creating it by passing in a pointer.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Change do_brk_flags() to expand existing VMA and add
do_brk_munmap()

Avoid allocating a new VMA when it is not necessary.  Expand or contract
the existing VMA instead.  This avoids unnecessary tree manipulations
and allocations.

Once the VMA is known, use it directly when populating to avoid
unnecessary lookup work.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/gup: Add mm_populate_vma() for use when the vma is known

When a vma is known, avoid calling mm_populate to search for the vma to
populate.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm: Remove rb tree.

Remove the RB tree and start using the maple tree for vm_area_struct
tracking.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

kernel/fork: Convert dup_mmap to use maple tree

Use the maple tree iterator to duplicate the mm_struct trees.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Change unmapped_area and unmapped_area_topdown to use maple
tree

Use the new maple tree data structure to find an unmapped area.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Change find_vma_prev() to use maple tree

Change the implementation of find_vma_prev to use the new maple tree
data structure.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Change find_vma() to use the maple tree

Start using the maple tree to find VMA entries in an mm_struct.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Introduce unlock_range() for code cleanup

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>