swapfile rcu fix

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

oom_kill rcu fix

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mlock rcu fix

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mempolicy rcu fix

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

memory rcu fix

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

ksm rcu fix

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

khugepaged rcu fix

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

sys rcu fix

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

sched/fair rcu fix

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

fork rcu fix

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

uprobes rcu fix

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

events/core rcu fix

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

kernel/acct rcu fix

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

userfaultfd rcu fix

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

task_mmu rcu fix

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

proc base rcu fix

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

binfmt_elf rcu fix

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

optee rcu fix

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

cxl rcu fix

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

xtensa rcu fix

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

x86 vdso rcu lock fix

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

um tlb rcu fix

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

s390: gmap, fix rcu lock

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

ppc book3s64: Fix subpage rcu lock

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

ppc book3s32: Fix tlb rcu lock

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

parisc: Fix cache rcu lock

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Fix remap_file_pages() locking to work with new do_mmap()

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Fix remap_file_pages() change.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Drop validations in do_mas_align_munmap()

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm: Move mas locking outside of munmap() path.

Now that there is a split variant that allows splitting to use a maple state,
move the locks to a more logical position.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

maple_tree: Rename b_node to bn in mab_calc_split()

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Add mas_split_vma() and use it for munmap()

Use the maple state when splitting a node to not have to rewalk/reset the state on splits.
This is also needed to clean the locks up

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

maple_tree: Fix spanning_rebalance() insufficient nodes and mas state.

Fix mas state when expanding a root, the offset on rebalance, and the node and offset
when doing a spanning store.

Note that spanning rebalance needed to do some tricky free/no free based on
where the node data comes from.  Next would skip the free of the next node
as it is picked up when mas_topiary() trims the parents children

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

maple_tree: Fix offset in mas_node_stre, find old node in mas_store_gfp.

Needs fixing, should restore the maple state during the operation by saving the leaf info

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Prepare detaching of VMAs for new split

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

maple_tree: Fix maple state offset on root expansion

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Optimize mmap_region() use of maple states

Only walk down once after the tree was modified with the munmap call.  Copy mas
to find prev and next.  Only reset mas if the address changed.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

Fix ipc/shm for nommu.

Define mas for nommu and mmu builds

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

maple_tree: Fix mas_find for a single entry, add test

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

lib/test_maple_tree: Rename zero() to not_empty(), use mas_set(), add more allocation tests

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

lib/maple_tree: Clean up node allocations

Remove extra parenthesis, etc

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mempolicy: Use vma_lookup() instead of find_vma_intersection()

vma_lookup() is more efficient

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/memory.c: Use vma_lookup() instead of find_vma_intersection()

vma_lookup() is more efficient

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

arch/powerpc/oprofile/cell/spu_task_sync: Use vma_lookup() instead of find_vma_intersection()

vma_lookup() is more efficient

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

arch/powerpc/kvm/book3s_hv_uvmem: Use vma_lookup() instead of find_vma_intersection()

vma_lookup() is more efficient

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

arch/mips/kernel/traps: Use vma_lookup() instead of find_vma_intersection()

vma_lookup() is more efficient

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

arch/arm64/kvm: Use vma_lookup() instead of find_vma_intersection()

vma_lookup() is more efficient

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

drivers/oprofile: Use vma_lookup() instead of find_vma_intersection()

vma_lookup() is more efficient

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

vfio: Use vma_lookup() instead of find_vma_intersection()

vma_lookup() is more efficient

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

virt/kvm: Use vma_lookup() instead of find_vma_intersection()

vma_lookup() is more efficient

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

arch/arc/kernel/troubleshoot: use vma_lookup() instead of find_vma_intersection()

vma_lookup() is more efficient

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/framge_vector: Use vma_lookup() instead of find_vma_intersection()

vma_lookup() is more efficient

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Fix mmap_region() maple state

When finding next, tmp must be reset to mas to ensure the correct slot is
used for writing

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Fix whitespace in expand_upwards()

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm: Add vma_lookup()

Walk the maple tree to find the requested address.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

i915_vma: Rename vma_lookup to i915_vma_lookup

Use i915 prefix to avoid name collision with future vma_lookup() in mm.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

maple_tree: Add 32 bit support

Add the defines for the size of nodes, etc to support 32 bit kernels.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

nommu: remove cache

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/nommu and fs/proc/task_nommu: convert to maple tree

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

nommu: linked list fix

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm: Restore rb_node to vm_region.

nommu uses the rb tree for overlapped shared regions

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm: Move mm_populate_vma to internal header.

mm_populate_vma is only internal to mm code

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm: Move vma_mas_store()/vma_mas_erase() to internal header.

These are used in nommu case as well

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

maple_tree: add mtree_store() to header

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

maple_tree: Add static where necessary.

Add static to anywhere sparse reports necessary.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Change do_mas_align_munmap() to static

Add static to the declaration as reported by sparse

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

lib/maple_tree: Fix rcu annotations

Add all the missing __rcu annotations reported by sparse.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>

mm/mmap: Lock maple tree when using mas_store_gfp()

When storing to the maple tree, take the write lock to avoid
entering/exiting the use of RCU during a write operation.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

lib/maple_tree: Fix mas_pop_node() zeroing.

When using a single allocation in the node, zero out slot 0 of the node
being returned.

Also, small whitespace fix in mas_push

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/ksm: Fix compile warning and whitespace for 3fd77bfa4f25fa

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

lib/maple_tree: Fix mas_store_gfp() on spanning store.

Spanning store detection needs to be more robust than checking if it spans the end of
a node.  Fix the detection by using mas_is_span_wr().  Also ensure it's not MAS_NONE.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

lib/maple_tree: Add check for dead nodes in mt_validate()

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: __insert_vm_struct() only needs to mas_reset()

vma_mas_store() will set the range again.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mprotect: Use vma_next() of prev.

The linked list used to use prev->next.  the maple state may be invalidated
by the mprotect_fixup call which may alter the tree.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mprotect: Use vma_next() instead of mas_next().

mas is in the wrong state and would potentially return the current vma twice.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Fix mmap_region() maple tree store.

mas_set_range() before vma_mas_link() as the address may have changed
as commented above in the code.

Do the same in the error path.

This forces a reset of the mas->node to MA_START.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: ULONG_MAX in exit_mmap

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: ULONG_MAX in do_brk_munmap

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: ULONG_MAX in detach_range

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: ULONG_MAX in remove_mt

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mprotect: Change do_mprotect_pkey()  -1 to ULONG_MAX

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mempolicy: Fix mbind_range -1 to ULONG_MAX

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

kernel/sched/fair: Change -1 to ULONG_MAX

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

kernel/acct: Change -1 to ULONG_MAX

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

task_mmu: Change -1 to ULONG_MAX

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

ppc book3s32: Change -1 to ULONG_MAX

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

lib/maple_tree: Reset request_count to 0 of old head on push

Just to be safe

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Add validate_mm() to __vma_adjust() start

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mprotect: Fix adcfbd7698558a linked list removal

Use mas_find() in do_mprotect_pkey to find the next vma to better align
with what happened with the old API

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

kernel/sched/fair: Fix linked list fallout 4acb2b9560cd8f

When searching for the vma, use mas_find, not mas_walk() to better do what was happening before

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mempolicy: fix adcfbd7698558a

When removing the linked list, a bug was introduced in maple state
iterator used for the mbind_range() function.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: do_brk_munmap() whitespace fix

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

maple_tree: use ma_is_root() in mas_replace().

Instead of using mte_is_root(), use ma_is_root() directly since the maple node
is known already.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Find next vma in do_brk_munmap() prior to updating tree.

The tree update may invalidate the location and cause a longer search if the VMA does not
preceed a NULL entry

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Fix do_brk_munmap() when munmapping an entire VMA or more

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Add more validate_mm() calls

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm: Return a bool from anon_vma_interval_tree_verify()

Added to allow printing which vma has the issue

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: expand_upwards() shouldn't call validate_mm()

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: mmap_region() fixes for when expanding vma as well as anon

When expanding a vma, call khugepaged_enter_vma_merge() with correct vma.

Also, set the vm_pgoff to the correct value when adding an anon mapping.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>

mm/mmap: Fix detach_range() and do_mas_align_munmap()

When searching for the first entry overlapp in do_mas_align_munmap(), the mas->last
will be modified if one is found, and if the maple state isn't reset for splitting,
then the wrong end will be past to detach_range() in the maple state.

detach_range() no longer needs both vma and vm_area_struct last, so make that change
too.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>