kmem_cache_setup_percpu_array() will allocate a per-cpu array for
caching alloc/free objects of given size for the cache. The cache
has to be created with SLAB_NO_MERGE flag.
When empty, half of the array is filled by an internal bulk alloc
operation. When full, half of the array is flushed by an internal bulk
free operation.
The bulk operations exposed to slab users also try to utilize the array
when possible, but leave the array empty or full and use the bulk
alloc/free only for the operation itself. If kmemcg is enabled and
active, bulk freeing skips the array as it would be less efficient.
The locking is copied from the page allocator's pcplists, based on
embedded spin locks. Interrupts are not disabled, only preemption (cpu
migration on RT). Trylock is attempted to avoid deadlock due to
an intnerrupt, trylock failure means the array is bypassed.
Sysfs stat counters alloc_cpu_cache and free_cpu_cache count operations
that used the percpu array.
kmem_cache_prefill_percpu_array() can be called to ensure the array on
the current cpu to at least the given number of objects. However this is
only opportunistic as there's no cpu pinning between the prefill and
usage, and trylocks may fail when the usage is in an irq handler.
Therefore allocations cannot rely on the array for success even after
the prefill. But misses should be rare enough that e.g. GFP_ATOMIC
allocations should be acceptable after the refill.
Mark SLAB_DEPRECATED as BROKEN so the new prefill call doesn't need to
be reimplemented there and the bots don't complain. SLAB has percpu
arrays by design but their sizes are determined internally and lack
prefill.
More TODO/FIXMEs:
- NUMA awareness - preferred node currently ignored, __GFP_THISNODE not
honored
- slub_debug - will not work for allocations from the array. Normally in
SLUB implementation the enabling slub_debug for a cache effectively
disables all the fast paths and makes every operation work with the
shared list, but that could lead to depleting the reserves if we ignore
the prefill and use GFP_ATOMIC. Needs more thought.