Page replacement is handled in the Linux Kernel in one of two ways:
1) Asynchronously via kswapd
2) Synchronously, via direct reclaim
At page allocation time the allocating task is immediately given a page
from the zone free list allowing it to go right back to work doing
whatever it was doing; Probably directly or indirectly executing
business logic.
Just prior to satisfying the allocation, free pages is checked to see if
it has reached the zone low watermark and if so, kswapd is awakened.
Kswapd will start scanning pages looking for inactive pages to evict to
make room for new page allocations. The work of kswapd allows tasks to
continue allocating memory from their respective zone free list without
incurring any delay.
When the demand for free pages exceeds the rate that kswapd tasks can
supply them, page allocation works differently. Once the allocating task
finds that the number of free pages is at or below the zone min
watermark, the task will no longer pull pages from the free list.
Instead, the task will run the same CPU-bound routines as kswapd to
satisfy its own allocation by scanning and evicting pages. This is
called a direct reclaim.
The time spent performing a direct reclaim can be substantial, often
taking tens to hundreds of milliseconds for small order0 allocations to
half a second or more for order9 huge-page allocations. In fact, kswapd
is not actually required on a linux system. It exists for the sole
purpose of optimizing performance by preventing direct reclaims.
When memory shortfall is sufficient to trigger direct reclaims, they can
occur in any task that is running on the system. A single aggressive
memory allocating task can set the stage for collateral damage to occur
in small tasks that rarely allocate additional memory. Consider the
impact of injecting an additional 100ms of latency when nscd allocates
memory to facilitate caching of a DNS query.
The presence of direct reclaims 10 years ago was a fairly reliable
indicator that too much was being asked of a Linux system. Kswapd was
likely wasting time scanning pages that were ineligible for eviction.
Adding RAM or reducing the working set size would usually make the
problem go away. Since then hardware has evolved to bring a new struggle
for kswapd. Storage speeds have increased by orders of magnitude while
CPU clock speeds have actually slowed down. This presents a throughput
problem for a single threaded kswapd that will get worse with each
generation of new hardware.
------------
Test Details
------------
The tests below were designed with the assumption that a kswapd
bottleneck is best demonstrated using filesystem reads. This way, the
inactive list will be full of clean pages, simplifying the analysis and
allowing kswapd to achieve the highest possible steal rate. Maximum
steal rates for kswapd are likely to be the same or lower for any other
mix of page types on the system.
Tests were run on a 2U Oracle X7-2L with 52 Intel Xeon Skylake 2GHz
cores, 756GB of RAM and 8 x 3.6 TB NVMe Solid State Disk drives. Each
drive has an XFS filesystem mounted separately as /d0 through /d7. NVMe
drives require multiple concurrent streams to show their potential, so I
created 11 250GB zero-filled files on each drive so that I could test
with parallel reads.
The test script runs in multiple stages. At each stage, the number of dd
tasks run concurrently is increased by 2. I did not include all of the
test output for brevity.
During each stage dd tasks are launched to read from each drive in a
round robin fashion until the specified number of tasks for the stage
has been reached. Then iostat, vmstat and top are started in the
background with 10 second intervals. After five minutes, all of the dd
tasks are killed and the iostat, vmstat and top output is parsed in
order to report the following:
CPU consumption
- sy: aggregate kernel mode CPU consumption from vmstat output. The
value doesn't tend to fluctuate much so I just grab the highest value.
Each sample is averaged over 10 seconds
- dd_cpu: for all of the dd tasks averaged across the top samples since
there is a lot of variation.
Throughput
- in Kbytes
- Command is iostat -x -d 10 -g total
This first test performs reads using O_DIRECT in order to show the peak
throughput that can be obtained using these drives. It also demonstrates
how rapidly throughput scales as the number of dd tasks are increased.
The dd command for this test looks like this:
Command Used: dd iflag=direct if=/d${i}/$n of=/dev/null bs=4M
dd sy dd_cpu throughput
6 1 4.52
14966994.50
10 1 4.94
21503269.37
16 1 4.70
25791251.00
22 1 5.02
26139553.00
28 1 4.85
26242989.00
34 2 4.53
26253264.20
40 2 3.82
26265978.60
46 2 3.39
26256091.80
52 2 3.06
26256913.60
58 2 2.74
26256988.40
64 2 2.50
26256534.20
70 2 2.27
26255088.00
76 2 2.12
26247909.00
80 2 1.99
26251164.80
Throughput peaked with 40 dd tasks at
26265978.60 KB/s. Very little
system CPU was consumed as expected the drives DMA directly into the
user address space when using O_DIRECT.
The remaining tests do not use O_DIRECT. We drop the page cache before
testing and stop the test as soon as kswapd wakes up.
dd sy dd_cpu throughput
6 2 30.34
5245348.50
10 3 32.53
7735288.00
16 5 32.78
11059243.20
22 6 30.77
13371912.80
28 8 31.52
16092092.00
34 10 30.12
18000076.80
40 11 29.34
19368494.40
46 11 26.45
20450313.60
52 13 25.47
21249290.40
58 13 23.75
22008188.80
64 13 21.38
22298248.80
70 15 21.39
22442940.80
76 14 18.82
22876260.80
80 15 19.45
23143716.80
Each read has to pause after the buffer in kernel space is populated
while that data is added to the pagecache and copied into the user
address space. For this reason, more parallel streams are required to
achieve peak throughput. The copy operation consumes substantially more
CPU than direct IO as expected.
The next test measures throughput after kswapd starts running. This is
the same test only we wait for kswapd to wake up before we start
collecting metrics.
The script actually keeps track of a few things that were not mentioned
earlier. It tracks direct reclaims and page scans by watching the
metrics in /proc/vmstat. CPU consumption for kswapd is tracked the same
way it is tracked for dd.
Since the test is 100% reads, you can assume that the page steal rate
for kswapd and direct reclaims is almost identical to the scan rate.
1 kswapd thread per node
dd sy dd_cpu kswapd0 kswapd1 throughput dr's pgscan_kswapd pgscan_direct
10 4 31.56 24.86 23.56
7828015.60 0
460668253 0
16 7 36.10 65.94 71.74
11149401.80 0
900894848 0
22 10 37.79 91.94 94.25
14271445.20 179
1149779893 4236610
28 14 46.04 86.71 84.39
14633873.80 14624
829782742 346638611
34 16 41.23 85.76 84.04
16058195.40 16303
927594668 386370834
40 20 47.65 69.78 68.79
15381517.80 28538
566746661 676222823
46 22 45.76 64.39 64.50
15941522.40 32567
510483237 771659039
52 25 47.40 60.56 63.15
15189850.20 34504
422051924 816932307
58 29 48.21 53.44 57.19
15191931.40 38313
330596133 907319630
64 32 49.88 51.08 51.10
15073485.20 41939
233133908 993009680
70 36 50.71 51.32 51.54
15265733.00 43348
209193894 1026357511
76 40 51.78 51.39 54.38
15091290.20 43804
192167798 1037072462
80 44 52.95 48.91 55.95
15009935.60 44218
177718893 1046802606
Look closely at the scan statistics and the CPU consumption numbers and
it should be clear that the bulk of the CPU consumption is occuring in
the context of the dd tasks due to direct reclaims, not kswapd.
Same test, more kswapd tasks:
6 kswapd threads per node
dd sy dd_cpu kswapd0 kswapd1 throughput dr's pgscan_kswapd pgscan_direct
10 4 33.19 6.98 6.44
8184050.97 0
460877355 0
16 10 41.61 27.99 28.26
11533556.80 0
941456735 0
22 12 39.00 28.12 29.67
14303265.20 10
1170356251 237431
28 15 37.53 38.01 40.42
16449001.40 30
1355387318 711292
34 19 38.87 49.81 51.33
18094928.20 0
1495622630 0
40 22 37.62 56.93 59.27
19562580.80 0
1618461307 0
46 25 36.51 64.00 66.34
20800868.60 0
1715162179 0
52 28 36.89 70.68 74.60
21650189.60 0
1787311285 0
58 34 37.44 80.59 81.43
22395273.00 1190
1794721827 28089474
64 44 50.22 67.36 76.96
21848111.20 18150
1105060289 429342513
70 46 55.57 56.59 64.22
18766118.20 27918
724659653 660301547
76 50 42.37 67.79 75.83
23688889.40 18603
1171174674 440088674
80 49 40.14 72.05 79.60
22350506.00 15680
1310470634 370843890
With 58 dd tasks, throughput is roughly the same as what we saw without
memory pressure. Ten additional kswapd tasks (5 per node) resulted in a
17% increase in aggregate kernel mode CPU consumption.
NOTE: The kswapd tasks were originally tracked with an array of task
structs in each pgdata structure. Sadly, any changes to the pg_data_t
resulted in KABI breakage. Look for the following definition that was
used as a workaround:
static struct task_struct *kswapd_list[MAX_NUMNODES][MAX_KSWAPD_THREADS];
Orabug:
27913411
Signed-off-by: Buddy Lumpkin <buddy.lumpkin@oracle.com>
Reviewed-by: Khalid Aziz <khalid.aziz@oracle.com>
Reviewed-by: Pavel Tatashin <pasha.tatashin@oracle.com>
Reviewed-by: John Sobecki <john.sobecki@oracle.com>
Reviewed by: Henry Willard <henry.willard@oracle.com
Signed-off-by: Brian Maly <brian.maly@oracle.com>
- extfrag_threshold
- hugepages_treat_as_movable
- hugetlb_shm_group
+- kswapd_threads
- laptop_mode
- legacy_va_layout
- lowmem_reserve_ratio
==============================================================
+kswapd_threads
+
+kswapd_threads allows you to control the number of kswapd threads per node
+running on the system. This provides the ability to devote more CPU in
+exchange for more aggressive page replacement. More aggressive page
+replacement can reduce direct reclaims which cause latency for tasks
+and decrease throughput when doing filesystem IO through the pagecache.
+Direct reclaims are recorded using the allocstall counter in /proc/vmstat.
+
+The default value is 1 and the range of acceptible values are 1-16.
+Always start with lower values in the 3-6 range. Higher values should
+be justified with testing. If direct reclaims occur in spite of high
+values, the cost of direct reclaims (in latency) that occur can be
+higher due to increased lock contention.
+
+==============================================================
+
laptop_mode
laptop_mode is a knob that controls "laptop mode". All the things that are
extern void set_dma_reserve(unsigned long new_dma_reserve);
extern void memmap_init_zone(unsigned long, int, unsigned long,
unsigned long, enum memmap_context);
+void update_kswapd_threads(void);
extern void setup_per_zone_wmarks(void);
extern int __meminit init_per_zone_wmark_min(void);
extern void mem_init(void);
extern void zone_pcp_reset(struct zone *zone);
/* page_alloc.c */
+extern int kswapd_threads;
extern int min_free_kbytes;
extern int watermark_scale_factor;
* will not.
*/
#define PAGE_ALLOC_COSTLY_ORDER 3
+#define MAX_KSWAPD_THREADS 16
enum {
MIGRATE_UNMOVABLE,
/* These two functions are used to setup the per zone pages min values */
struct ctl_table;
+int kswapd_threads_sysctl_handler(struct ctl_table *, int,
+ void __user *, size_t *, loff_t *);
int min_free_kbytes_sysctl_handler(struct ctl_table *, int,
void __user *, size_t *, loff_t *);
int watermark_scale_factor_sysctl_handler(struct ctl_table *, int,
#ifdef CONFIG_PRINTK
static int ten_thousand = 10000;
#endif
+static int max_kswapd_threads = MAX_KSWAPD_THREADS;
/* this is needed for the proc_doulongvec_minmax of vm_dirty_bytes */
static unsigned long dirty_bytes_min = 2 * PAGE_SIZE;
.proc_handler = min_free_kbytes_sysctl_handler,
.extra1 = &zero,
},
+ {
+ .procname = "kswapd_threads",
+ .data = &kswapd_threads,
+ .maxlen = sizeof(kswapd_threads),
+ .mode = 0644,
+ .proc_handler = kswapd_threads_sysctl_handler,
+ .extra1 = &one,
+ .extra2 = &max_kswapd_threads,
+ },
{
.procname = "watermark_scale_factor",
.data = &watermark_scale_factor,
return 0;
}
+int kswapd_threads_sysctl_handler(struct ctl_table *table, int write,
+ void __user *buffer, size_t *length,
+ loff_t *ppos)
+{
+ int rc;
+
+ rc = proc_dointvec_minmax(table, write, buffer, length, ppos);
+ if (rc)
+ return rc;
+
+ if (write)
+ update_kswapd_threads();
+ return 0;
+}
+
int watermark_scale_factor_sysctl_handler(struct ctl_table *table, int write,
void __user *buffer, size_t *length, loff_t *ppos)
{
#define lru_to_page(_head) (list_entry((_head)->prev, struct page, lru))
+/*
+ * Number of active kswapd threads
+ */
+#define DEF_KSWAPD_THREADS_PER_NODE 1
+int kswapd_threads = DEF_KSWAPD_THREADS_PER_NODE;
+static int kswapd_threads_current = DEF_KSWAPD_THREADS_PER_NODE;
+static struct task_struct *kswapd_list[MAX_NUMNODES][MAX_KSWAPD_THREADS];
+
#ifdef ARCH_HAS_PREFETCH
#define prefetch_prev_lru_page(_page, _base, _field) \
do { \
static int cpu_callback(struct notifier_block *nfb, unsigned long action,
void *hcpu)
{
- int nid;
+ int nid, hid;
+ int nr_threads = kswapd_threads_current;
- if (action == CPU_ONLINE || action == CPU_ONLINE_FROZEN) {
- for_each_node_state(nid, N_MEMORY) {
- pg_data_t *pgdat = NODE_DATA(nid);
- const struct cpumask *mask;
+ if (action != CPU_ONLINE && action != CPU_ONLINE_FROZEN)
+ return NOTIFY_OK;
- mask = cpumask_of_node(pgdat->node_id);
+ for_each_node_state(nid, N_MEMORY) {
+ pg_data_t *pgdat = NODE_DATA(nid);
+ const struct cpumask *mask = cpumask_of_node(pgdat->node_id);
+ struct task_struct *task;
- if (cpumask_any_and(cpu_online_mask, mask) < nr_cpu_ids)
- /* One of our CPUs online: restore mask */
- set_cpus_allowed_ptr(pgdat->kswapd, mask);
+ if (cpumask_any_and(cpu_online_mask, mask) < nr_cpu_ids) {
+ for (hid = 0; hid < nr_threads; hid++) {
+ struct task_struct *t = kswapd_list[nid][hid];
+ /* CPU of ours online: restore mask */
+ if (t)
+ set_cpus_allowed_ptr(t, mask);
+ }
}
}
return NOTIFY_OK;
}
+static void update_kswapd_threads_node(int nid)
+{
+ pg_data_t *pgdat;
+ int drop, increase;
+ int last_idx, start_idx, hid;
+ int nr_threads = kswapd_threads_current;
+
+ pgdat = NODE_DATA(nid);
+ last_idx = nr_threads - 1;
+ if (kswapd_threads < nr_threads) {
+ drop = nr_threads - kswapd_threads;
+ for (hid = last_idx; hid > (last_idx - drop); hid--) {
+ if (kswapd_list[nid][hid]) {
+ kthread_stop(kswapd_list[nid][hid]);
+ kswapd_list[nid][hid] = NULL;
+ }
+ }
+ } else {
+ increase = kswapd_threads - nr_threads;
+ start_idx = last_idx + 1;
+ for (hid = start_idx; hid < (start_idx + increase); hid++) {
+ kswapd_list[nid][hid] = kthread_run(kswapd, pgdat,
+ "kswapd%d:%d", nid, hid);
+ if (IS_ERR(kswapd_list[nid][hid])) {
+ pr_err("Failed to start kswapd%d on node %d\n",
+ hid, nid);
+ kswapd_list[nid][hid] = NULL;
+ /*
+ * We are out of resources. Do not start any
+ * more threads.
+ */
+ break;
+ }
+ }
+ }
+}
+
+/*
+ * When kswapd_threads is updated from userspace, this function is called
+ * to start required new kswapd threads or kill off extra kswapd threads
+ */
+void update_kswapd_threads(void)
+{
+ int nid;
+
+ if (kswapd_threads_current == kswapd_threads)
+ return;
+
+ /*
+ * This function updates the number of currently running kswapd
+ * threads and kills off or starts up kswapd to match what has
+ * been requested from userspace. Memory hotplug functions also
+ * start up and kill off kswapd threads and use
+ * kswapd_threads_current to determine the number of threads to
+ * start or kill. To avoid race condition between the two, take
+ * memory hotplug lock.
+ */
+ mem_hotplug_begin();
+ for_each_node_state(nid, N_MEMORY)
+ update_kswapd_threads_node(nid);
+
+ pr_info("kswapd_thread count changed, old:%d new:%d\n",
+ kswapd_threads_current, kswapd_threads);
+ kswapd_threads_current = kswapd_threads;
+ mem_hotplug_done();
+}
+
/*
* This kswapd start function will be called by init and node-hot-add.
* On node-hot-add, kswapd will moved to proper cpus if cpus are hot-added.
{
pg_data_t *pgdat = NODE_DATA(nid);
int ret = 0;
+ int hid, nr_threads;
- if (pgdat->kswapd)
+ if (kswapd_list[nid][0])
return 0;
- pgdat->kswapd = kthread_run(kswapd, pgdat, "kswapd%d", nid);
- if (IS_ERR(pgdat->kswapd)) {
- /* failure at boot is fatal */
- BUG_ON(system_state == SYSTEM_BOOTING);
- pr_err("Failed to start kswapd on node %d\n", nid);
- ret = PTR_ERR(pgdat->kswapd);
- pgdat->kswapd = NULL;
+ nr_threads = kswapd_threads;
+ for (hid = 0; hid < nr_threads; hid++) {
+ kswapd_list[nid][hid] = kthread_run(kswapd, pgdat,
+ "kswapd%d:%d",
+ nid, hid);
+ if (IS_ERR(kswapd_list[nid][hid])) {
+ /* failure at boot is fatal */
+ BUG_ON(system_state == SYSTEM_BOOTING);
+ pr_err("Failed to start kswapd%d on node %d\n",
+ hid, nid);
+ ret = PTR_ERR(kswapd_list[nid][hid]);
+ kswapd_list[nid][hid] = NULL;
+ break;
+ }
}
+ kswapd_threads_current = nr_threads;
return ret;
}
*/
void kswapd_stop(int nid)
{
- struct task_struct *kswapd = NODE_DATA(nid)->kswapd;
+ int hid;
+ int nr_threads = kswapd_threads_current;
- if (kswapd) {
- kthread_stop(kswapd);
- NODE_DATA(nid)->kswapd = NULL;
+ for (hid = 0; hid < nr_threads; hid++) {
+ if (kswapd_list[nid][hid]) {
+ kthread_stop(kswapd_list[nid][hid]);
+ kswapd_list[nid][hid] = NULL;
+ }
}
}
projects / users / jedix / linux-maple.git / commitdiff