* [PATCH 01/10] writeback: account per-bdi accumulated dirtied pages
2011-08-26 11:38 [PATCH 00/10] IO-less dirty throttling v10 Wu Fengguang
@ 2011-08-26 11:38 ` Wu Fengguang
2011-08-26 11:38 ` [PATCH 02/10] writeback: dirty position control Wu Fengguang
` (8 subsequent siblings)
9 siblings, 0 replies; 15+ messages in thread
From: Wu Fengguang @ 2011-08-26 11:38 UTC (permalink / raw)
To: linux-fsdevel
Cc: Peter Zijlstra, Jan Kara, Michael Rubin, Wu Fengguang,
Andrew Morton, Christoph Hellwig, Dave Chinner, Greg Thelen,
Minchan Kim, Vivek Goyal, Andrea Righi, linux-mm, LKML
[-- Attachment #1: writeback-bdi-dirtied.patch --]
[-- Type: text/plain, Size: 2322 bytes --]
Introduce the BDI_DIRTIED counter. It will be used for estimating the
bdi's dirty bandwidth.
CC: Jan Kara <jack@suse.cz>
CC: Michael Rubin <mrubin@google.com>
CC: Peter Zijlstra <a.p.zijlstra@chello.nl>
Signed-off-by: Wu Fengguang <fengguang.wu@intel.com>
---
include/linux/backing-dev.h | 1 +
mm/backing-dev.c | 2 ++
mm/page-writeback.c | 1 +
3 files changed, 4 insertions(+)
--- linux-next.orig/include/linux/backing-dev.h 2011-08-16 09:30:23.000000000 +0800
+++ linux-next/include/linux/backing-dev.h 2011-08-17 10:15:45.000000000 +0800
@@ -40,6 +40,7 @@ typedef int (congested_fn)(void *, int);
enum bdi_stat_item {
BDI_RECLAIMABLE,
BDI_WRITEBACK,
+ BDI_DIRTIED,
BDI_WRITTEN,
NR_BDI_STAT_ITEMS
};
--- linux-next.orig/mm/page-writeback.c 2011-08-16 09:30:23.000000000 +0800
+++ linux-next/mm/page-writeback.c 2011-08-17 20:35:22.000000000 +0800
@@ -1333,6 +1333,7 @@ void account_page_dirtied(struct page *p
__inc_zone_page_state(page, NR_FILE_DIRTY);
__inc_zone_page_state(page, NR_DIRTIED);
__inc_bdi_stat(mapping->backing_dev_info, BDI_RECLAIMABLE);
+ __inc_bdi_stat(mapping->backing_dev_info, BDI_DIRTIED);
task_dirty_inc(current);
task_io_account_write(PAGE_CACHE_SIZE);
}
--- linux-next.orig/mm/backing-dev.c 2011-08-16 09:30:23.000000000 +0800
+++ linux-next/mm/backing-dev.c 2011-08-17 10:15:45.000000000 +0800
@@ -97,6 +97,7 @@ static int bdi_debug_stats_show(struct s
"BdiDirtyThresh: %10lu kB\n"
"DirtyThresh: %10lu kB\n"
"BackgroundThresh: %10lu kB\n"
+ "BdiDirtied: %10lu kB\n"
"BdiWritten: %10lu kB\n"
"BdiWriteBandwidth: %10lu kBps\n"
"b_dirty: %10lu\n"
@@ -109,6 +110,7 @@ static int bdi_debug_stats_show(struct s
K(bdi_thresh),
K(dirty_thresh),
K(background_thresh),
+ (unsigned long) K(bdi_stat(bdi, BDI_DIRTIED)),
(unsigned long) K(bdi_stat(bdi, BDI_WRITTEN)),
(unsigned long) K(bdi->write_bandwidth),
nr_dirty,
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 15+ messages in thread* [PATCH 02/10] writeback: dirty position control
2011-08-26 11:38 [PATCH 00/10] IO-less dirty throttling v10 Wu Fengguang
2011-08-26 11:38 ` [PATCH 01/10] writeback: account per-bdi accumulated dirtied pages Wu Fengguang
@ 2011-08-26 11:38 ` Wu Fengguang
2011-08-26 11:38 ` [PATCH 03/10] writeback: dirty rate control Wu Fengguang
` (7 subsequent siblings)
9 siblings, 0 replies; 15+ messages in thread
From: Wu Fengguang @ 2011-08-26 11:38 UTC (permalink / raw)
To: linux-fsdevel
Cc: Peter Zijlstra, Jan Kara, Wu Fengguang, Andrew Morton,
Christoph Hellwig, Dave Chinner, Greg Thelen, Minchan Kim,
Vivek Goyal, Andrea Righi, linux-mm, LKML
[-- Attachment #1: writeback-control-algorithms.patch --]
[-- Type: text/plain, Size: 14625 bytes --]
bdi_position_ratio() provides a scale factor to bdi->dirty_ratelimit, so
that the resulted task rate limit can drive the dirty pages back to the
global/bdi setpoints.
Old scheme is,
|
free run area | throttle area
----------------------------------------+---------------------------->
thresh^ dirty pages
New scheme is,
^ task rate limit
|
| *
| *
| *
|[free run] * [smooth throttled]
| *
| *
| *
..bdi->dirty_ratelimit..........*
| . *
| . *
| . *
| . *
| . *
+-------------------------------.-----------------------*------------>
setpoint^ limit^ dirty pages
The slope of the bdi control line should be
1) large enough to pull the dirty pages to setpoint reasonably fast
2) small enough to avoid big fluctuations in the resulted pos_ratio and
hence task ratelimit
Since the fluctuation range of the bdi dirty pages is typically observed
to be within 1-second worth of data, the bdi control line's slope is
selected to be a linear function of bdi write bandwidth, so that it can
adapt to slow/fast storage devices well.
Assume the bdi control line
pos_ratio = 1.0 + k * (dirty - bdi_setpoint)
where k is the negative slope.
If targeting for 12.5% fluctuation range in pos_ratio when dirty pages
are fluctuating in range
[bdi_setpoint - write_bw/2, bdi_setpoint + write_bw/2],
we get slope
k = - 1 / (8 * write_bw)
Let pos_ratio(x_intercept) = 0, we get the parameter used in code:
x_intercept = bdi_setpoint + 8 * write_bw
The global/bdi slopes are nicely complementing each other when the
system has only one major bdi (indicated by bdi_thresh ~= thresh):
1) slope of global control line => scaling to the control scope size
2) slope of main bdi control line => scaling to the writeout bandwidth
so that
- in memory tight systems, (1) becomes strong enough to squeeze dirty
pages inside the control scope
- in large memory systems where the "gravity" of (1) for pulling the
dirty pages to setpoint is too weak, (2) can back (1) up and drive
dirty pages to bdi_setpoint ~= setpoint reasonably fast.
Unfortunately in JBOD setups, the fluctuation range of bdi threshold
is related to memory size due to the interferences between disks. In
this case, the bdi slope will be weighted sum of write_bw and bdi_thresh.
Given equations
span = x_intercept - bdi_setpoint
k = df/dx = - 1 / span
and the extremum values
span = bdi_thresh
dx = bdi_thresh
we get
df = - dx / span = - 1.0
That means, when bdi_dirty deviates bdi_thresh up, pos_ratio and hence
task ratelimit will fluctuate by -100%.
peter: use 3rd order polynomial for the global control line
CC: Peter Zijlstra <a.p.zijlstra@chello.nl>
Acked-by: Jan Kara <jack@suse.cz>
Signed-off-by: Wu Fengguang <fengguang.wu@intel.com>
---
fs/fs-writeback.c | 2
include/linux/writeback.h | 1
mm/page-writeback.c | 213 +++++++++++++++++++++++++++++++++++-
3 files changed, 210 insertions(+), 6 deletions(-)
--- linux-next.orig/mm/page-writeback.c 2011-08-26 15:57:18.000000000 +0800
+++ linux-next/mm/page-writeback.c 2011-08-26 15:57:34.000000000 +0800
@@ -46,6 +46,8 @@
*/
#define BANDWIDTH_INTERVAL max(HZ/5, 1)
+#define RATELIMIT_CALC_SHIFT 10
+
/*
* After a CPU has dirtied this many pages, balance_dirty_pages_ratelimited
* will look to see if it needs to force writeback or throttling.
@@ -409,6 +411,12 @@ int bdi_set_max_ratio(struct backing_dev
}
EXPORT_SYMBOL(bdi_set_max_ratio);
+static unsigned long dirty_freerun_ceiling(unsigned long thresh,
+ unsigned long bg_thresh)
+{
+ return (thresh + bg_thresh) / 2;
+}
+
static unsigned long hard_dirty_limit(unsigned long thresh)
{
return max(thresh, global_dirty_limit);
@@ -493,6 +501,197 @@ unsigned long bdi_dirty_limit(struct bac
return bdi_dirty;
}
+/*
+ * Dirty position control.
+ *
+ * (o) global/bdi setpoints
+ *
+ * We want the dirty pages be balanced around the global/bdi setpoints.
+ * When the number of dirty pages is higher/lower than the setpoint, the
+ * dirty position control ratio (and hence task dirty ratelimit) will be
+ * decreased/increased to bring the dirty pages back to the setpoint.
+ *
+ * pos_ratio = 1 << RATELIMIT_CALC_SHIFT
+ *
+ * if (dirty < setpoint) scale up pos_ratio
+ * if (dirty > setpoint) scale down pos_ratio
+ *
+ * if (bdi_dirty < bdi_setpoint) scale up pos_ratio
+ * if (bdi_dirty > bdi_setpoint) scale down pos_ratio
+ *
+ * task_ratelimit = dirty_ratelimit * pos_ratio >> RATELIMIT_CALC_SHIFT
+ *
+ * (o) global control line
+ *
+ * ^ pos_ratio
+ * |
+ * | |<===== global dirty control scope ======>|
+ * 2.0 .............*
+ * | .*
+ * | . *
+ * | . *
+ * | . *
+ * | . *
+ * | . *
+ * 1.0 ................................*
+ * | . . *
+ * | . . *
+ * | . . *
+ * | . . *
+ * | . . *
+ * 0 +------------.------------------.----------------------*------------->
+ * freerun^ setpoint^ limit^ dirty pages
+ *
+ * (o) bdi control lines
+ *
+ * The control lines for the global/bdi setpoints both stretch up to @limit.
+ * The below figure illustrates the main bdi control line with an auxiliary
+ * line extending it to @limit.
+ *
+ * o
+ * o
+ * o [o] main control line
+ * o [*] auxiliary control line
+ * o
+ * o
+ * o
+ * o
+ * o
+ * o
+ * o--------------------- balance point, rate scale = 1
+ * | o
+ * | o
+ * | o
+ * | o
+ * | o
+ * | o
+ * | o------- connect point, rate scale = 1/2
+ * | .*
+ * | . *
+ * | . *
+ * | . *
+ * | . *
+ * | . *
+ * | . *
+ * [--------------------+-----------------------------.--------------------*]
+ * 0 bdi_setpoint x_intercept limit
+ *
+ * The auxiliary control line allows smoothly throttling bdi_dirty down to
+ * normal if it starts high in situations like
+ * - start writing to a slow SD card and a fast disk at the same time. The SD
+ * card's bdi_dirty may rush to many times higher than bdi_setpoint.
+ * - the bdi dirty thresh drops quickly due to change of JBOD workload
+ */
+static unsigned long bdi_position_ratio(struct backing_dev_info *bdi,
+ unsigned long thresh,
+ unsigned long bg_thresh,
+ unsigned long dirty,
+ unsigned long bdi_thresh,
+ unsigned long bdi_dirty)
+{
+ unsigned long write_bw = bdi->avg_write_bandwidth;
+ unsigned long freerun = dirty_freerun_ceiling(thresh, bg_thresh);
+ unsigned long limit = hard_dirty_limit(thresh);
+ unsigned long x_intercept;
+ unsigned long setpoint; /* dirty pages' target balance point */
+ unsigned long bdi_setpoint;
+ unsigned long span;
+ long long pos_ratio; /* for scaling up/down the rate limit */
+ long x;
+
+ if (unlikely(dirty >= limit))
+ return 0;
+
+ /*
+ * global setpoint
+ *
+ * setpoint - dirty 3
+ * f(dirty) := 1.0 + (----------------)
+ * limit - setpoint
+ *
+ * it's a 3rd order polynomial that subjects to
+ *
+ * (1) f(freerun) = 2.0 => rampup dirty_ratelimit reasonably fast
+ * (2) f(setpoint) = 1.0 => the balance point
+ * (3) f(limit) = 0 => the hard limit
+ * (4) df/dx <= 0 => negative feedback control
+ * (5) the closer to setpoint, the smaller |df/dx| (and the reverse)
+ * => fast response on large errors; small oscillation near setpoint
+ */
+ setpoint = (freerun + limit) / 2;
+ x = div_s64((setpoint - dirty) << RATELIMIT_CALC_SHIFT,
+ limit - setpoint + 1);
+ pos_ratio = x;
+ pos_ratio = pos_ratio * x >> RATELIMIT_CALC_SHIFT;
+ pos_ratio = pos_ratio * x >> RATELIMIT_CALC_SHIFT;
+ pos_ratio += 1 << RATELIMIT_CALC_SHIFT;
+
+ /*
+ * We have computed basic pos_ratio above based on global situation. If
+ * the bdi is over/under its share of dirty pages, we want to scale
+ * pos_ratio further down/up. That is done by the following mechanism.
+ */
+
+ /*
+ * bdi setpoint
+ *
+ * f(bdi_dirty) := 1.0 + k * (bdi_dirty - bdi_setpoint)
+ *
+ * x_intercept - bdi_dirty
+ * := --------------------------
+ * x_intercept - bdi_setpoint
+ *
+ * The main bdi control line is a linear function that subjects to
+ *
+ * (1) f(bdi_setpoint) = 1.0
+ * (2) k = - 1 / (8 * write_bw) (in single bdi case)
+ * or equally: x_intercept = bdi_setpoint + 8 * write_bw
+ *
+ * For single bdi case, the dirty pages are observed to fluctuate
+ * regularly within range
+ * [bdi_setpoint - write_bw/2, bdi_setpoint + write_bw/2]
+ * for various filesystems, where (2) can yield in a reasonable 12.5%
+ * fluctuation range for pos_ratio.
+ *
+ * For JBOD case, bdi_thresh (not bdi_dirty!) could fluctuate up to its
+ * own size, so move the slope over accordingly and choose a slope that
+ * yields 100% pos_ratio fluctuation on suddenly doubled bdi_thresh.
+ */
+ if (unlikely(bdi_thresh > thresh))
+ bdi_thresh = thresh;
+ /*
+ * scale global setpoint to bdi's:
+ * bdi_setpoint = setpoint * bdi_thresh / thresh
+ */
+ x = div_u64((u64)bdi_thresh << 16, thresh + 1);
+ bdi_setpoint = setpoint * (u64)x >> 16;
+ /*
+ * Use span=(8*write_bw) in single bdi case as indicated by
+ * (thresh - bdi_thresh ~= 0) and transit to bdi_thresh in JBOD case.
+ *
+ * bdi_thresh thresh - bdi_thresh
+ * span = ---------- * (8 * write_bw) + ------------------- * bdi_thresh
+ * thresh thresh
+ */
+ span = (thresh - bdi_thresh + 8 * write_bw) * (u64)x >> 16;
+ x_intercept = bdi_setpoint + span;
+
+ span >>= 1;
+ if (unlikely(bdi_dirty > bdi_setpoint + span)) {
+ if (unlikely(bdi_dirty > limit))
+ return 0;
+ if (x_intercept < limit) {
+ x_intercept = limit; /* auxiliary control line */
+ bdi_setpoint += span;
+ pos_ratio >>= 1;
+ }
+ }
+ pos_ratio *= x_intercept - bdi_dirty;
+ do_div(pos_ratio, x_intercept - bdi_setpoint + 1);
+
+ return pos_ratio;
+}
+
static void bdi_update_write_bandwidth(struct backing_dev_info *bdi,
unsigned long elapsed,
unsigned long written)
@@ -591,6 +790,7 @@ static void global_update_bandwidth(unsi
void __bdi_update_bandwidth(struct backing_dev_info *bdi,
unsigned long thresh,
+ unsigned long bg_thresh,
unsigned long dirty,
unsigned long bdi_thresh,
unsigned long bdi_dirty,
@@ -627,6 +827,7 @@ snapshot:
static void bdi_update_bandwidth(struct backing_dev_info *bdi,
unsigned long thresh,
+ unsigned long bg_thresh,
unsigned long dirty,
unsigned long bdi_thresh,
unsigned long bdi_dirty,
@@ -635,8 +836,8 @@ static void bdi_update_bandwidth(struct
if (time_is_after_eq_jiffies(bdi->bw_time_stamp + BANDWIDTH_INTERVAL))
return;
spin_lock(&bdi->wb.list_lock);
- __bdi_update_bandwidth(bdi, thresh, dirty, bdi_thresh, bdi_dirty,
- start_time);
+ __bdi_update_bandwidth(bdi, thresh, bg_thresh, dirty,
+ bdi_thresh, bdi_dirty, start_time);
spin_unlock(&bdi->wb.list_lock);
}
@@ -677,7 +878,8 @@ static void balance_dirty_pages(struct a
* catch-up. This avoids (excessively) small writeouts
* when the bdi limits are ramping up.
*/
- if (nr_dirty <= (background_thresh + dirty_thresh) / 2)
+ if (nr_dirty <= dirty_freerun_ceiling(dirty_thresh,
+ background_thresh))
break;
bdi_thresh = bdi_dirty_limit(bdi, dirty_thresh);
@@ -721,8 +923,9 @@ static void balance_dirty_pages(struct a
if (!bdi->dirty_exceeded)
bdi->dirty_exceeded = 1;
- bdi_update_bandwidth(bdi, dirty_thresh, nr_dirty,
- bdi_thresh, bdi_dirty, start_time);
+ bdi_update_bandwidth(bdi, dirty_thresh, background_thresh,
+ nr_dirty, bdi_thresh, bdi_dirty,
+ start_time);
/* Note: nr_reclaimable denotes nr_dirty + nr_unstable.
* Unstable writes are a feature of certain networked
--- linux-next.orig/fs/fs-writeback.c 2011-08-26 15:57:18.000000000 +0800
+++ linux-next/fs/fs-writeback.c 2011-08-26 15:57:20.000000000 +0800
@@ -675,7 +675,7 @@ static inline bool over_bground_thresh(v
static void wb_update_bandwidth(struct bdi_writeback *wb,
unsigned long start_time)
{
- __bdi_update_bandwidth(wb->bdi, 0, 0, 0, 0, start_time);
+ __bdi_update_bandwidth(wb->bdi, 0, 0, 0, 0, 0, start_time);
}
/*
--- linux-next.orig/include/linux/writeback.h 2011-08-26 15:57:18.000000000 +0800
+++ linux-next/include/linux/writeback.h 2011-08-26 15:57:20.000000000 +0800
@@ -141,6 +141,7 @@ unsigned long bdi_dirty_limit(struct bac
void __bdi_update_bandwidth(struct backing_dev_info *bdi,
unsigned long thresh,
+ unsigned long bg_thresh,
unsigned long dirty,
unsigned long bdi_thresh,
unsigned long bdi_dirty,
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 15+ messages in thread* [PATCH 03/10] writeback: dirty rate control
2011-08-26 11:38 [PATCH 00/10] IO-less dirty throttling v10 Wu Fengguang
2011-08-26 11:38 ` [PATCH 01/10] writeback: account per-bdi accumulated dirtied pages Wu Fengguang
2011-08-26 11:38 ` [PATCH 02/10] writeback: dirty position control Wu Fengguang
@ 2011-08-26 11:38 ` Wu Fengguang
2011-08-26 11:38 ` [PATCH 04/10] writeback: stabilize bdi->dirty_ratelimit Wu Fengguang
` (6 subsequent siblings)
9 siblings, 0 replies; 15+ messages in thread
From: Wu Fengguang @ 2011-08-26 11:38 UTC (permalink / raw)
To: linux-fsdevel
Cc: Peter Zijlstra, Wu Fengguang, Andrew Morton, Jan Kara,
Christoph Hellwig, Dave Chinner, Greg Thelen, Minchan Kim,
Vivek Goyal, Andrea Righi, linux-mm, LKML
[-- Attachment #1: dirty-ratelimit --]
[-- Type: text/plain, Size: 10228 bytes --]
It's all about bdi->dirty_ratelimit, which aims to be (write_bw / N)
when there are N dd tasks.
On write() syscall, use bdi->dirty_ratelimit
============================================
balance_dirty_pages(pages_dirtied)
{
task_ratelimit = bdi->dirty_ratelimit * bdi_position_ratio();
pause = pages_dirtied / task_ratelimit;
sleep(pause);
}
On every 200ms, update bdi->dirty_ratelimit
===========================================
bdi_update_dirty_ratelimit()
{
task_ratelimit = bdi->dirty_ratelimit * bdi_position_ratio();
balanced_dirty_ratelimit = task_ratelimit * write_bw / dirty_rate;
bdi->dirty_ratelimit = balanced_dirty_ratelimit
}
Estimation of balanced bdi->dirty_ratelimit
===========================================
balanced task_ratelimit
-----------------------
balance_dirty_pages() needs to throttle tasks dirtying pages such that
the total amount of dirty pages stays below the specified dirty limit in
order to avoid memory deadlocks. Furthermore we desire fairness in that
tasks get throttled proportionally to the amount of pages they dirty.
IOW we want to throttle tasks such that we match the dirty rate to the
writeout bandwidth, this yields a stable amount of dirty pages:
dirty_rate == write_bw (1)
The fairness requirement gives us:
task_ratelimit = balanced_dirty_ratelimit
== write_bw / N (2)
where N is the number of dd tasks. We don't know N beforehand, but
still can estimate balanced_dirty_ratelimit within 200ms.
Start by throttling each dd task at rate
task_ratelimit = task_ratelimit_0 (3)
(any non-zero initial value is OK)
After 200ms, we measured
dirty_rate = # of pages dirtied by all dd's / 200ms
write_bw = # of pages written to the disk / 200ms
For the aggressive dd dirtiers, the equality holds
dirty_rate == N * task_rate
== N * task_ratelimit_0 (4)
Or
task_ratelimit_0 == dirty_rate / N (5)
Now we conclude that the balanced task ratelimit can be estimated by
write_bw
balanced_dirty_ratelimit = task_ratelimit_0 * ---------- (6)
dirty_rate
Because with (4) and (5) we can get the desired equality (1):
write_bw
balanced_dirty_ratelimit == (dirty_rate / N) * ----------
dirty_rate
== write_bw / N
Then using the balanced task ratelimit we can compute task pause times like:
task_pause = task->nr_dirtied / task_ratelimit
task_ratelimit with position control
------------------------------------
However, while the above gives us means of matching the dirty rate to
the writeout bandwidth, it at best provides us with a stable dirty page
count (assuming a static system). In order to control the dirty page
count such that it is high enough to provide performance, but does not
exceed the specified limit we need another control.
The dirty position control works by extending (2) to
task_ratelimit = balanced_dirty_ratelimit * pos_ratio (7)
where pos_ratio is a negative feedback function that subjects to
1) f(setpoint) = 1.0
2) df/dx < 0
That is, if the dirty pages are ABOVE the setpoint, we throttle each
task a bit more HEAVY than balanced_dirty_ratelimit, so that the dirty
pages are created less fast than they are cleaned, thus DROP to the
setpoints (and the reverse).
Based on (7) and the assumption that both dirty_ratelimit and pos_ratio
remains CONSTANT for the past 200ms, we get
task_ratelimit_0 = balanced_dirty_ratelimit * pos_ratio (8)
Putting (8) into (6), we get the formula used in
bdi_update_dirty_ratelimit():
write_bw
balanced_dirty_ratelimit *= pos_ratio * ---------- (9)
dirty_rate
Signed-off-by: Wu Fengguang <fengguang.wu@intel.com>
---
CC: Peter Zijlstra <a.p.zijlstra@chello.nl>
---
include/linux/backing-dev.h | 7 ++
mm/backing-dev.c | 1
mm/page-writeback.c | 82 +++++++++++++++++++++++++++++++++-
3 files changed, 88 insertions(+), 2 deletions(-)
--- linux-next.orig/include/linux/backing-dev.h 2011-08-26 13:53:40.000000000 +0800
+++ linux-next/include/linux/backing-dev.h 2011-08-26 13:54:13.000000000 +0800
@@ -75,10 +75,17 @@ struct backing_dev_info {
struct percpu_counter bdi_stat[NR_BDI_STAT_ITEMS];
unsigned long bw_time_stamp; /* last time write bw is updated */
+ unsigned long dirtied_stamp;
unsigned long written_stamp; /* pages written at bw_time_stamp */
unsigned long write_bandwidth; /* the estimated write bandwidth */
unsigned long avg_write_bandwidth; /* further smoothed write bw */
+ /*
+ * The base dirty throttle rate, re-calculated on every 200ms.
+ * All the bdi tasks' dirty rate will be curbed under it.
+ */
+ unsigned long dirty_ratelimit;
+
struct prop_local_percpu completions;
int dirty_exceeded;
--- linux-next.orig/mm/backing-dev.c 2011-08-26 13:53:40.000000000 +0800
+++ linux-next/mm/backing-dev.c 2011-08-26 13:54:13.000000000 +0800
@@ -670,6 +670,7 @@ int bdi_init(struct backing_dev_info *bd
bdi->bw_time_stamp = jiffies;
bdi->written_stamp = 0;
+ bdi->dirty_ratelimit = INIT_BW;
bdi->write_bandwidth = INIT_BW;
bdi->avg_write_bandwidth = INIT_BW;
--- linux-next.orig/mm/page-writeback.c 2011-08-26 13:52:42.000000000 +0800
+++ linux-next/mm/page-writeback.c 2011-08-26 15:52:42.000000000 +0800
@@ -787,6 +787,78 @@ static void global_update_bandwidth(unsi
spin_unlock(&dirty_lock);
}
+/*
+ * Maintain bdi->dirty_ratelimit, the base dirty throttle rate.
+ *
+ * Normal bdi tasks will be curbed at or below it in long term.
+ * Obviously it should be around (write_bw / N) when there are N dd tasks.
+ */
+static void bdi_update_dirty_ratelimit(struct backing_dev_info *bdi,
+ unsigned long thresh,
+ unsigned long bg_thresh,
+ unsigned long dirty,
+ unsigned long bdi_thresh,
+ unsigned long bdi_dirty,
+ unsigned long dirtied,
+ unsigned long elapsed)
+{
+ unsigned long write_bw = bdi->avg_write_bandwidth;
+ unsigned long dirty_ratelimit = bdi->dirty_ratelimit;
+ unsigned long dirty_rate;
+ unsigned long task_ratelimit;
+ unsigned long balanced_dirty_ratelimit;
+ unsigned long pos_ratio;
+
+ /*
+ * The dirty rate will match the writeout rate in long term, except
+ * when dirty pages are truncated by userspace or re-dirtied by FS.
+ */
+ dirty_rate = (dirtied - bdi->dirtied_stamp) * HZ / elapsed;
+
+ pos_ratio = bdi_position_ratio(bdi, thresh, bg_thresh, dirty,
+ bdi_thresh, bdi_dirty);
+ /*
+ * task_ratelimit reflects each dd's dirty rate for the past 200ms.
+ */
+ task_ratelimit = (u64)dirty_ratelimit *
+ pos_ratio >> RATELIMIT_CALC_SHIFT;
+
+ /*
+ * A linear estimation of the "balanced" throttle rate. The theory is,
+ * if there are N dd tasks, each throttled at task_ratelimit, the bdi's
+ * dirty_rate will be measured to be (N * task_ratelimit). So the below
+ * formula will yield the balanced rate limit (write_bw / N).
+ *
+ * Note that the expanded form is not a pure rate feedback:
+ * rate_(i+1) = rate_(i) * (write_bw / dirty_rate) (1)
+ * but also takes pos_ratio into account:
+ * rate_(i+1) = rate_(i) * (write_bw / dirty_rate) * pos_ratio (2)
+ *
+ * (1) is not realistic because pos_ratio also takes part in balancing
+ * the dirty rate. Consider the state
+ * pos_ratio = 0.5 (3)
+ * rate = 2 * (write_bw / N) (4)
+ * If (1) is used, it will stuck in that state! Because each dd will
+ * be throttled at
+ * task_ratelimit = pos_ratio * rate = (write_bw / N) (5)
+ * yielding
+ * dirty_rate = N * task_ratelimit = write_bw (6)
+ * put (6) into (1) we get
+ * rate_(i+1) = rate_(i) (7)
+ *
+ * So we end up using (2) to always keep
+ * rate_(i+1) ~= (write_bw / N) (8)
+ * regardless of the value of pos_ratio. As long as (8) is satisfied,
+ * pos_ratio is able to drive itself to 1.0, which is not only where
+ * the dirty count meet the setpoint, but also where the slope of
+ * pos_ratio is most flat and hence task_ratelimit is least fluctuated.
+ */
+ balanced_dirty_ratelimit = div_u64((u64)task_ratelimit * write_bw,
+ dirty_rate | 1);
+
+ bdi->dirty_ratelimit = max(balanced_dirty_ratelimit, 1UL);
+}
+
void __bdi_update_bandwidth(struct backing_dev_info *bdi,
unsigned long thresh,
unsigned long bg_thresh,
@@ -797,6 +869,7 @@ void __bdi_update_bandwidth(struct backi
{
unsigned long now = jiffies;
unsigned long elapsed = now - bdi->bw_time_stamp;
+ unsigned long dirtied;
unsigned long written;
/*
@@ -805,6 +878,7 @@ void __bdi_update_bandwidth(struct backi
if (elapsed < BANDWIDTH_INTERVAL)
return;
+ dirtied = percpu_counter_read(&bdi->bdi_stat[BDI_DIRTIED]);
written = percpu_counter_read(&bdi->bdi_stat[BDI_WRITTEN]);
/*
@@ -814,12 +888,16 @@ void __bdi_update_bandwidth(struct backi
if (elapsed > HZ && time_before(bdi->bw_time_stamp, start_time))
goto snapshot;
- if (thresh)
+ if (thresh) {
global_update_bandwidth(thresh, dirty, now);
-
+ bdi_update_dirty_ratelimit(bdi, thresh, bg_thresh, dirty,
+ bdi_thresh, bdi_dirty,
+ dirtied, elapsed);
+ }
bdi_update_write_bandwidth(bdi, elapsed, written);
snapshot:
+ bdi->dirtied_stamp = dirtied;
bdi->written_stamp = written;
bdi->bw_time_stamp = now;
}
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 15+ messages in thread* [PATCH 04/10] writeback: stabilize bdi->dirty_ratelimit
2011-08-26 11:38 [PATCH 00/10] IO-less dirty throttling v10 Wu Fengguang
` (2 preceding siblings ...)
2011-08-26 11:38 ` [PATCH 03/10] writeback: dirty rate control Wu Fengguang
@ 2011-08-26 11:38 ` Wu Fengguang
2011-08-26 11:38 ` [PATCH 05/10] writeback: per task dirty rate limit Wu Fengguang
` (5 subsequent siblings)
9 siblings, 0 replies; 15+ messages in thread
From: Wu Fengguang @ 2011-08-26 11:38 UTC (permalink / raw)
To: linux-fsdevel
Cc: Peter Zijlstra, Wu Fengguang, Andrew Morton, Jan Kara,
Christoph Hellwig, Dave Chinner, Greg Thelen, Minchan Kim,
Vivek Goyal, Andrea Righi, linux-mm, LKML
[-- Attachment #1: dirty-ratelimit-stablize --]
[-- Type: text/plain, Size: 6468 bytes --]
There are some imperfections in balanced_dirty_ratelimit.
1) large fluctuations
The dirty_rate used for computing balanced_dirty_ratelimit is merely
averaged in the past 200ms (very small comparing to the 3s estimation
period for write_bw), which makes rather dispersed distribution of
balanced_dirty_ratelimit.
It's pretty hard to average out the singular points by increasing the
estimation period. Considering that the averaging technique will
introduce very undesirable time lags, I give it up totally. (btw, the 3s
write_bw averaging time lag is much more acceptable because its impact
is one-way and therefore won't lead to oscillations.)
The more practical way is filtering -- most singular
balanced_dirty_ratelimit points can be filtered out by remembering some
prev_balanced_rate and prev_prev_balanced_rate. However the more
reliable way is to guard balanced_dirty_ratelimit with task_ratelimit.
2) due to truncates and fs redirties, the (write_bw <=> dirty_rate)
match could become unbalanced, which may lead to large systematical
errors in balanced_dirty_ratelimit. The truncates, due to its possibly
bumpy nature, can hardly be compensated smoothly. So let's face it. When
some over-estimated balanced_dirty_ratelimit brings dirty_ratelimit
high, dirty pages will go higher than the setpoint. task_ratelimit will
in turn become lower than dirty_ratelimit. So if we consider both
balanced_dirty_ratelimit and task_ratelimit and update dirty_ratelimit
only when they are on the same side of dirty_ratelimit, the systematical
errors in balanced_dirty_ratelimit won't be able to bring
dirty_ratelimit far away.
The balanced_dirty_ratelimit estimation may also be inaccurate near
@limit or @freerun, however is less an issue.
3) since we ultimately want to
- keep the fluctuations of task ratelimit as small as possible
- keep the dirty pages around the setpoint as long time as possible
the update policy used for (2) also serves the above goals nicely:
if for some reason the dirty pages are high (task_ratelimit < dirty_ratelimit),
and dirty_ratelimit is low (dirty_ratelimit < balanced_dirty_ratelimit),
there is no point to bring up dirty_ratelimit in a hurry only to hurt
both the above two goals.
So, we make use of task_ratelimit to limit the update of dirty_ratelimit
in two ways:
1) avoid changing dirty rate when it's against the position control target
(the adjusted rate will slow down the progress of dirty pages going
back to setpoint).
2) limit the step size. task_ratelimit is changing values step by step,
leaving a consistent trace comparing to the randomly jumping
balanced_dirty_ratelimit. task_ratelimit also has the nice smaller
errors in stable state and typically larger errors when there are big
errors in rate. So it's a pretty good limiting factor for the step
size of dirty_ratelimit.
Note that bdi->dirty_ratelimit is always tracking balanced_dirty_ratelimit.
task_ratelimit is merely used as a limiting factor.
Signed-off-by: Wu Fengguang <fengguang.wu@intel.com>
---
mm/page-writeback.c | 64 +++++++++++++++++++++++++++++++++++++++++-
1 file changed, 63 insertions(+), 1 deletion(-)
--- linux-next.orig/mm/page-writeback.c 2011-08-26 16:22:48.000000000 +0800
+++ linux-next/mm/page-writeback.c 2011-08-26 16:23:06.000000000 +0800
@@ -809,6 +809,7 @@ static void bdi_update_dirty_ratelimit(s
unsigned long task_ratelimit;
unsigned long balanced_dirty_ratelimit;
unsigned long pos_ratio;
+ unsigned long step;
/*
* The dirty rate will match the writeout rate in long term, except
@@ -857,7 +858,68 @@ static void bdi_update_dirty_ratelimit(s
balanced_dirty_ratelimit = div_u64((u64)task_ratelimit * write_bw,
dirty_rate | 1);
- bdi->dirty_ratelimit = max(balanced_dirty_ratelimit, 1UL);
+ /*
+ * We could safely do this and return immediately:
+ *
+ * bdi->dirty_ratelimit = balanced_dirty_ratelimit;
+ *
+ * However to get a more stable dirty_ratelimit, the below elaborated
+ * code makes use of task_ratelimit to filter out sigular points and
+ * limit the step size.
+ *
+ * The below code essentially only uses the relative value of
+ *
+ * task_ratelimit - dirty_ratelimit
+ * = (pos_ratio - 1) * dirty_ratelimit
+ *
+ * which reflects the direction and size of dirty position error.
+ */
+
+ /*
+ * dirty_ratelimit will follow balanced_dirty_ratelimit iff
+ * task_ratelimit is on the same side of dirty_ratelimit, too.
+ * For example, when
+ * - dirty_ratelimit > balanced_dirty_ratelimit
+ * - dirty_ratelimit > task_ratelimit (dirty pages are above setpoint)
+ * lowering dirty_ratelimit will help meet both the position and rate
+ * control targets. Otherwise, don't update dirty_ratelimit if it will
+ * only help meet the rate target. After all, what the users ultimately
+ * feel and care are stable dirty rate and small position error.
+ *
+ * |task_ratelimit - dirty_ratelimit| is used to limit the step size
+ * and filter out the sigular points of balanced_dirty_ratelimit. Which
+ * keeps jumping around randomly and can even leap far away at times
+ * due to the small 200ms estimation period of dirty_rate (we want to
+ * keep that period small to reduce time lags).
+ */
+ step = 0;
+ if (dirty_ratelimit < balanced_dirty_ratelimit) {
+ if (dirty_ratelimit < task_ratelimit)
+ step = min(balanced_dirty_ratelimit,
+ task_ratelimit) - dirty_ratelimit;
+ } else {
+ if (dirty_ratelimit > task_ratelimit)
+ step = dirty_ratelimit - max(balanced_dirty_ratelimit,
+ task_ratelimit);
+ }
+
+ /*
+ * Don't pursue 100% rate matching. It's impossible since the balanced
+ * rate itself is constantly fluctuating. So decrease the track speed
+ * when it gets close to the target. Helps eliminate pointless tremors.
+ */
+ step >>= dirty_ratelimit / (8 * step + 1);
+ /*
+ * Limit the tracking speed to avoid overshooting.
+ */
+ step = (step + 7) / 8;
+
+ if (dirty_ratelimit < balanced_dirty_ratelimit)
+ dirty_ratelimit += step;
+ else
+ dirty_ratelimit -= step;
+
+ bdi->dirty_ratelimit = max(dirty_ratelimit, 1UL);
}
void __bdi_update_bandwidth(struct backing_dev_info *bdi,
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 15+ messages in thread* [PATCH 05/10] writeback: per task dirty rate limit
2011-08-26 11:38 [PATCH 00/10] IO-less dirty throttling v10 Wu Fengguang
` (3 preceding siblings ...)
2011-08-26 11:38 ` [PATCH 04/10] writeback: stabilize bdi->dirty_ratelimit Wu Fengguang
@ 2011-08-26 11:38 ` Wu Fengguang
2011-08-26 12:51 ` Peter Zijlstra
2011-08-26 11:38 ` [PATCH 06/10] writeback: IO-less balance_dirty_pages() Wu Fengguang
` (4 subsequent siblings)
9 siblings, 1 reply; 15+ messages in thread
From: Wu Fengguang @ 2011-08-26 11:38 UTC (permalink / raw)
To: linux-fsdevel
Cc: Peter Zijlstra, Wu Fengguang, Andrew Morton, Jan Kara,
Christoph Hellwig, Dave Chinner, Greg Thelen, Minchan Kim,
Vivek Goyal, Andrea Righi, linux-mm, LKML
[-- Attachment #1: per-task-ratelimit --]
[-- Type: text/plain, Size: 7744 bytes --]
Add two fields to task_struct.
1) account dirtied pages in the individual tasks, for accuracy
2) per-task balance_dirty_pages() call intervals, for flexibility
The balance_dirty_pages() call interval (ie. nr_dirtied_pause) will
scale near-sqrt to the safety gap between dirty pages and threshold.
The main problem of per-task nr_dirtied is, if 1k+ tasks start dirtying
pages at exactly the same time, each task will be assigned a large
initial nr_dirtied_pause, so that the dirty threshold will be exceeded
long before each task reached its nr_dirtied_pause and hence call
balance_dirty_pages().
The solution is to watch for the number of pages dirtied on each CPU in
between the calls into balance_dirty_pages(). If it exceeds ratelimit_pages
(3% dirty threshold), force call balance_dirty_pages() for a chance to
set bdi->dirty_exceeded. In normal situations, this safeguarding
condition is not expected to trigger at all.
peter: keep the per-CPU ratelimit for safeguarding the 1k+ tasks case
CC: Peter Zijlstra <a.p.zijlstra@chello.nl>
Reviewed-by: Andrea Righi <andrea@betterlinux.com>
Signed-off-by: Wu Fengguang <fengguang.wu@intel.com>
---
include/linux/sched.h | 7 +++
kernel/fork.c | 3 +
mm/page-writeback.c | 89 ++++++++++++++++++++++------------------
3 files changed, 60 insertions(+), 39 deletions(-)
--- linux-next.orig/include/linux/sched.h 2011-08-16 10:43:00.000000000 +0800
+++ linux-next/include/linux/sched.h 2011-08-16 15:23:27.000000000 +0800
@@ -1525,6 +1525,13 @@ struct task_struct {
int make_it_fail;
#endif
struct prop_local_single dirties;
+ /*
+ * when (nr_dirtied >= nr_dirtied_pause), it's time to call
+ * balance_dirty_pages() for some dirty throttling pause
+ */
+ int nr_dirtied;
+ int nr_dirtied_pause;
+
#ifdef CONFIG_LATENCYTOP
int latency_record_count;
struct latency_record latency_record[LT_SAVECOUNT];
--- linux-next.orig/mm/page-writeback.c 2011-08-16 10:54:05.000000000 +0800
+++ linux-next/mm/page-writeback.c 2011-08-16 15:23:54.000000000 +0800
@@ -54,20 +54,6 @@
*/
static long ratelimit_pages = 32;
-/*
- * When balance_dirty_pages decides that the caller needs to perform some
- * non-background writeback, this is how many pages it will attempt to write.
- * It should be somewhat larger than dirtied pages to ensure that reasonably
- * large amounts of I/O are submitted.
- */
-static inline long sync_writeback_pages(unsigned long dirtied)
-{
- if (dirtied < ratelimit_pages)
- dirtied = ratelimit_pages;
-
- return dirtied + dirtied / 2;
-}
-
/* The following parameters are exported via /proc/sys/vm */
/*
@@ -169,6 +155,8 @@ static void update_completion_period(voi
int shift = calc_period_shift();
prop_change_shift(&vm_completions, shift);
prop_change_shift(&vm_dirties, shift);
+
+ writeback_set_ratelimit();
}
int dirty_background_ratio_handler(struct ctl_table *table, int write,
@@ -931,6 +919,23 @@ static void bdi_update_bandwidth(struct
}
/*
+ * After a task dirtied this many pages, balance_dirty_pages_ratelimited_nr()
+ * will look to see if it needs to start dirty throttling.
+ *
+ * If dirty_poll_interval is too low, big NUMA machines will call the expensive
+ * global_page_state() too often. So scale it near-sqrt to the safety margin
+ * (the number of pages we may dirty without exceeding the dirty limits).
+ */
+static unsigned long dirty_poll_interval(unsigned long dirty,
+ unsigned long thresh)
+{
+ if (thresh > dirty)
+ return 1UL << (ilog2(thresh - dirty) >> 1);
+
+ return 1;
+}
+
+/*
* balance_dirty_pages() must be called by processes which are generating dirty
* data. It looks at the number of dirty pages in the machine and will force
* the caller to perform writeback if the system is over `vm_dirty_ratio'.
@@ -1073,6 +1078,9 @@ static void balance_dirty_pages(struct a
if (clear_dirty_exceeded && bdi->dirty_exceeded)
bdi->dirty_exceeded = 0;
+ current->nr_dirtied = 0;
+ current->nr_dirtied_pause = dirty_poll_interval(nr_dirty, dirty_thresh);
+
if (writeback_in_progress(bdi))
return;
@@ -1099,7 +1107,7 @@ void set_page_dirty_balance(struct page
}
}
-static DEFINE_PER_CPU(unsigned long, bdp_ratelimits) = 0;
+static DEFINE_PER_CPU(int, bdp_ratelimits);
/**
* balance_dirty_pages_ratelimited_nr - balance dirty memory state
@@ -1119,31 +1127,39 @@ void balance_dirty_pages_ratelimited_nr(
unsigned long nr_pages_dirtied)
{
struct backing_dev_info *bdi = mapping->backing_dev_info;
- unsigned long ratelimit;
- unsigned long *p;
+ int ratelimit;
+ int *p;
if (!bdi_cap_account_dirty(bdi))
return;
- ratelimit = ratelimit_pages;
- if (mapping->backing_dev_info->dirty_exceeded)
- ratelimit = 8;
+ ratelimit = current->nr_dirtied_pause;
+ if (bdi->dirty_exceeded)
+ ratelimit = min(ratelimit, 32 >> (PAGE_SHIFT - 10));
+
+ current->nr_dirtied += nr_pages_dirtied;
+ preempt_disable();
/*
- * Check the rate limiting. Also, we do not want to throttle real-time
- * tasks in balance_dirty_pages(). Period.
+ * This prevents one CPU to accumulate too many dirtied pages without
+ * calling into balance_dirty_pages(), which can happen when there are
+ * 1000+ tasks, all of them start dirtying pages at exactly the same
+ * time, hence all honoured too large initial task->nr_dirtied_pause.
*/
- preempt_disable();
p = &__get_cpu_var(bdp_ratelimits);
- *p += nr_pages_dirtied;
- if (unlikely(*p >= ratelimit)) {
- ratelimit = sync_writeback_pages(*p);
+ if (unlikely(current->nr_dirtied >= ratelimit))
*p = 0;
- preempt_enable();
- balance_dirty_pages(mapping, ratelimit);
- return;
+ else {
+ *p += nr_pages_dirtied;
+ if (unlikely(*p >= ratelimit_pages)) {
+ *p = 0;
+ ratelimit = 0;
+ }
}
preempt_enable();
+
+ if (unlikely(current->nr_dirtied >= ratelimit))
+ balance_dirty_pages(mapping, current->nr_dirtied);
}
EXPORT_SYMBOL(balance_dirty_pages_ratelimited_nr);
@@ -1238,22 +1254,17 @@ void laptop_sync_completion(void)
*
* Here we set ratelimit_pages to a level which ensures that when all CPUs are
* dirtying in parallel, we cannot go more than 3% (1/32) over the dirty memory
- * thresholds before writeback cuts in.
- *
- * But the limit should not be set too high. Because it also controls the
- * amount of memory which the balance_dirty_pages() caller has to write back.
- * If this is too large then the caller will block on the IO queue all the
- * time. So limit it to four megabytes - the balance_dirty_pages() caller
- * will write six megabyte chunks, max.
+ * thresholds.
*/
void writeback_set_ratelimit(void)
{
- ratelimit_pages = vm_total_pages / (num_online_cpus() * 32);
+ unsigned long background_thresh;
+ unsigned long dirty_thresh;
+ global_dirty_limits(&background_thresh, &dirty_thresh);
+ ratelimit_pages = dirty_thresh / (num_online_cpus() * 32);
if (ratelimit_pages < 16)
ratelimit_pages = 16;
- if (ratelimit_pages * PAGE_CACHE_SIZE > 4096 * 1024)
- ratelimit_pages = (4096 * 1024) / PAGE_CACHE_SIZE;
}
static int __cpuinit
--- linux-next.orig/kernel/fork.c 2011-08-16 10:42:59.000000000 +0800
+++ linux-next/kernel/fork.c 2011-08-16 15:23:27.000000000 +0800
@@ -1301,6 +1301,9 @@ static struct task_struct *copy_process(
p->pdeath_signal = 0;
p->exit_state = 0;
+ p->nr_dirtied = 0;
+ p->nr_dirtied_pause = 128 >> (PAGE_SHIFT - 10);
+
/*
* Ok, make it visible to the rest of the system.
* We dont wake it up yet.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 15+ messages in thread* Re: [PATCH 05/10] writeback: per task dirty rate limit
2011-08-26 11:38 ` [PATCH 05/10] writeback: per task dirty rate limit Wu Fengguang
@ 2011-08-26 12:51 ` Peter Zijlstra
2011-08-26 12:59 ` Wu Fengguang
0 siblings, 1 reply; 15+ messages in thread
From: Peter Zijlstra @ 2011-08-26 12:51 UTC (permalink / raw)
To: Wu Fengguang
Cc: linux-fsdevel, Andrew Morton, Jan Kara, Christoph Hellwig,
Dave Chinner, Greg Thelen, Minchan Kim, Vivek Goyal, Andrea Righi,
linux-mm, LKML
On Fri, 2011-08-26 at 19:38 +0800, Wu Fengguang wrote:
> + preempt_disable();
> /*
> - * Check the rate limiting. Also, we do not want to throttle real-time
> - * tasks in balance_dirty_pages(). Period.
> + * This prevents one CPU to accumulate too many dirtied pages without
> + * calling into balance_dirty_pages(), which can happen when there are
> + * 1000+ tasks, all of them start dirtying pages at exactly the same
> + * time, hence all honoured too large initial task->nr_dirtied_pause.
> */
> - preempt_disable();
> p = &__get_cpu_var(bdp_ratelimits);
p = &get_cpu_var(bdp_ratelimits);
> - *p += nr_pages_dirtied;
> - if (unlikely(*p >= ratelimit)) {
> - ratelimit = sync_writeback_pages(*p);
> + if (unlikely(current->nr_dirtied >= ratelimit))
> *p = 0;
> - preempt_enable();
> - balance_dirty_pages(mapping, ratelimit);
> - return;
> + else {
> + *p += nr_pages_dirtied;
> + if (unlikely(*p >= ratelimit_pages)) {
> + *p = 0;
> + ratelimit = 0;
> + }
> }
> preempt_enable();
put_cpu_var(bdp_ratelimits);
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 15+ messages in thread* Re: [PATCH 05/10] writeback: per task dirty rate limit
2011-08-26 12:51 ` Peter Zijlstra
@ 2011-08-26 12:59 ` Wu Fengguang
0 siblings, 0 replies; 15+ messages in thread
From: Wu Fengguang @ 2011-08-26 12:59 UTC (permalink / raw)
To: Peter Zijlstra
Cc: linux-fsdevel@vger.kernel.org, Andrew Morton, Jan Kara,
Christoph Hellwig, Dave Chinner, Greg Thelen, Minchan Kim,
Vivek Goyal, Andrea Righi, linux-mm, LKML
On Fri, Aug 26, 2011 at 08:51:09PM +0800, Peter Zijlstra wrote:
> On Fri, 2011-08-26 at 19:38 +0800, Wu Fengguang wrote:
> > + preempt_disable();
> > /*
> > - * Check the rate limiting. Also, we do not want to throttle real-time
> > - * tasks in balance_dirty_pages(). Period.
> > + * This prevents one CPU to accumulate too many dirtied pages without
> > + * calling into balance_dirty_pages(), which can happen when there are
> > + * 1000+ tasks, all of them start dirtying pages at exactly the same
> > + * time, hence all honoured too large initial task->nr_dirtied_pause.
> > */
> > - preempt_disable();
> > p = &__get_cpu_var(bdp_ratelimits);
>
> p = &get_cpu_var(bdp_ratelimits);
Ah yeah.. I actually followed your suggestion, and then find we'll
eventually do two __get_cpu_var() calls here, one for bdp_ratelimits
and the other for dirty_leaks in a planned patch. So let's keep the
preempt_disable()/preempt_enable().
> > - *p += nr_pages_dirtied;
> > - if (unlikely(*p >= ratelimit)) {
> > - ratelimit = sync_writeback_pages(*p);
> > + if (unlikely(current->nr_dirtied >= ratelimit))
> > *p = 0;
> > - preempt_enable();
> > - balance_dirty_pages(mapping, ratelimit);
> > - return;
> > + else {
> > + *p += nr_pages_dirtied;
> > + if (unlikely(*p >= ratelimit_pages)) {
> > + *p = 0;
> > + ratelimit = 0;
> > + }
> > }
> > preempt_enable();
>
> put_cpu_var(bdp_ratelimits);
ditto.
Thanks,
Fengguang
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH 06/10] writeback: IO-less balance_dirty_pages()
2011-08-26 11:38 [PATCH 00/10] IO-less dirty throttling v10 Wu Fengguang
` (4 preceding siblings ...)
2011-08-26 11:38 ` [PATCH 05/10] writeback: per task dirty rate limit Wu Fengguang
@ 2011-08-26 11:38 ` Wu Fengguang
2011-08-26 11:38 ` [PATCH 07/10] writeback: dirty ratelimit - think time compensation Wu Fengguang
` (3 subsequent siblings)
9 siblings, 0 replies; 15+ messages in thread
From: Wu Fengguang @ 2011-08-26 11:38 UTC (permalink / raw)
To: linux-fsdevel
Cc: Peter Zijlstra, Wu Fengguang, Andrew Morton, Jan Kara,
Christoph Hellwig, Dave Chinner, Greg Thelen, Minchan Kim,
Vivek Goyal, Andrea Righi, linux-mm, LKML
[-- Attachment #1: writeback-ioless-balance_dirty_pages.patch --]
[-- Type: text/plain, Size: 15940 bytes --]
As proposed by Chris, Dave and Jan, don't start foreground writeback IO
inside balance_dirty_pages(). Instead, simply let it idle sleep for some
time to throttle the dirtying task. In the mean while, kick off the
per-bdi flusher thread to do background writeback IO.
RATIONALS
=========
- disk seeks on concurrent writeback of multiple inodes (Dave Chinner)
If every thread doing writes and being throttled start foreground
writeback, it leads to N IO submitters from at least N different
inodes at the same time, end up with N different sets of IO being
issued with potentially zero locality to each other, resulting in
much lower elevator sort/merge efficiency and hence we seek the disk
all over the place to service the different sets of IO.
OTOH, if there is only one submission thread, it doesn't jump between
inodes in the same way when congestion clears - it keeps writing to
the same inode, resulting in large related chunks of sequential IOs
being issued to the disk. This is more efficient than the above
foreground writeback because the elevator works better and the disk
seeks less.
- lock contention and cache bouncing on concurrent IO submitters (Dave Chinner)
With this patchset, the fs_mark benchmark on a 12-drive software RAID0 goes
from CPU bound to IO bound, freeing "3-4 CPUs worth of spinlock contention".
* "CPU usage has dropped by ~55%", "it certainly appears that most of
the CPU time saving comes from the removal of contention on the
inode_wb_list_lock" (IMHO at least 10% comes from the reduction of
cacheline bouncing, because the new code is able to call much less
frequently into balance_dirty_pages() and hence access the global
page states)
* the user space "App overhead" is reduced by 20%, by avoiding the
cacheline pollution by the complex writeback code path
* "for a ~5% throughput reduction", "the number of write IOs have
dropped by ~25%", and the elapsed time reduced from 41:42.17 to
40:53.23.
* On a simple test of 100 dd, it reduces the CPU %system time from 30% to 3%,
and improves IO throughput from 38MB/s to 42MB/s.
- IO size too small for fast arrays and too large for slow USB sticks
The write_chunk used by current balance_dirty_pages() cannot be
directly set to some large value (eg. 128MB) for better IO efficiency.
Because it could lead to more than 1 second user perceivable stalls.
Even the current 4MB write size may be too large for slow USB sticks.
The fact that balance_dirty_pages() starts IO on itself couples the
IO size to wait time, which makes it hard to do suitable IO size while
keeping the wait time under control.
Now it's possible to increase writeback chunk size proportional to the
disk bandwidth. In a simple test of 50 dd's on XFS, 1-HDD, 3GB ram,
the larger writeback size dramatically reduces the seek count to 1/10
(far beyond my expectation) and improves the write throughput by 24%.
- long block time in balance_dirty_pages() hurts desktop responsiveness
Many of us may have the experience: it often takes a couple of seconds
or even long time to stop a heavy writing dd/cp/tar command with
Ctrl-C or "kill -9".
- IO pipeline broken by bumpy write() progress
There are a broad class of "loop {read(buf); write(buf);}" applications
whose read() pipeline will be under-utilized or even come to a stop if
the write()s have long latencies _or_ don't progress in a constant rate.
The current threshold based throttling inherently transfers the large
low level IO completion fluctuations to bumpy application write()s,
and further deteriorates with increasing number of dirtiers and/or bdi's.
For example, when doing 50 dd's + 1 remote rsync to an XFS partition,
the rsync progresses very bumpy in legacy kernel, and throughput is
improved by 67% by this patchset. (plus the larger write chunk size,
it will be 93% speedup).
The new rate based throttling can support 1000+ dd's with excellent
smoothness, low latency and low overheads.
For the above reasons, it's much better to do IO-less and low latency
pauses in balance_dirty_pages().
Jan Kara, Dave Chinner and me explored the scheme to let
balance_dirty_pages() wait for enough writeback IO completions to
safeguard the dirty limit. However it's found to have two problems:
- in large NUMA systems, the per-cpu counters may have big accounting
errors, leading to big throttle wait time and jitters.
- NFS may kill large amount of unstable pages with one single COMMIT.
Because NFS server serves COMMIT with expensive fsync() IOs, it is
desirable to delay and reduce the number of COMMITs. So it's not
likely to optimize away such kind of bursty IO completions, and the
resulted large (and tiny) stall times in IO completion based throttling.
So here is a pause time oriented approach, which tries to control the
pause time in each balance_dirty_pages() invocations, by controlling
the number of pages dirtied before calling balance_dirty_pages(), for
smooth and efficient dirty throttling:
- avoid useless (eg. zero pause time) balance_dirty_pages() calls
- avoid too small pause time (less than 4ms, which burns CPU power)
- avoid too large pause time (more than 200ms, which hurts responsiveness)
- avoid big fluctuations of pause times
It can control pause times at will. The default policy (in a followup
patch) will be to do ~10ms pauses in 1-dd case, and increase to ~100ms
in 1000-dd case.
BEHAVIOR CHANGE
===============
(1) dirty threshold
Users will notice that the applications will get throttled once crossing
the global (background + dirty)/2=15% threshold, and then balanced around
17.5%. Before patch, the behavior is to just throttle it at 20% dirtyable
memory in 1-dd case.
Since the task will be soft throttled earlier than before, it may be
perceived by end users as performance "slow down" if his application
happens to dirty more than 15% dirtyable memory.
(2) smoothness/responsiveness
Users will notice a more responsive system during heavy writeback.
"killall dd" will take effect instantly.
Signed-off-by: Wu Fengguang <fengguang.wu@intel.com>
---
include/trace/events/writeback.h | 24 ----
mm/page-writeback.c | 161 +++++++++--------------------
2 files changed, 54 insertions(+), 131 deletions(-)
--- linux-next.orig/mm/page-writeback.c 2011-08-26 16:14:45.000000000 +0800
+++ linux-next/mm/page-writeback.c 2011-08-26 16:14:46.000000000 +0800
@@ -309,50 +309,6 @@ static void bdi_writeout_fraction(struct
numerator, denominator);
}
-static inline void task_dirties_fraction(struct task_struct *tsk,
- long *numerator, long *denominator)
-{
- prop_fraction_single(&vm_dirties, &tsk->dirties,
- numerator, denominator);
-}
-
-/*
- * task_dirty_limit - scale down dirty throttling threshold for one task
- *
- * task specific dirty limit:
- *
- * dirty -= (dirty/8) * p_{t}
- *
- * To protect light/slow dirtying tasks from heavier/fast ones, we start
- * throttling individual tasks before reaching the bdi dirty limit.
- * Relatively low thresholds will be allocated to heavy dirtiers. So when
- * dirty pages grow large, heavy dirtiers will be throttled first, which will
- * effectively curb the growth of dirty pages. Light dirtiers with high enough
- * dirty threshold may never get throttled.
- */
-#define TASK_LIMIT_FRACTION 8
-static unsigned long task_dirty_limit(struct task_struct *tsk,
- unsigned long bdi_dirty)
-{
- long numerator, denominator;
- unsigned long dirty = bdi_dirty;
- u64 inv = dirty / TASK_LIMIT_FRACTION;
-
- task_dirties_fraction(tsk, &numerator, &denominator);
- inv *= numerator;
- do_div(inv, denominator);
-
- dirty -= inv;
-
- return max(dirty, bdi_dirty/2);
-}
-
-/* Minimum limit for any task */
-static unsigned long task_min_dirty_limit(unsigned long bdi_dirty)
-{
- return bdi_dirty - bdi_dirty / TASK_LIMIT_FRACTION;
-}
-
/*
*
*/
@@ -989,29 +945,34 @@ static unsigned long dirty_poll_interval
/*
* balance_dirty_pages() must be called by processes which are generating dirty
* data. It looks at the number of dirty pages in the machine and will force
- * the caller to perform writeback if the system is over `vm_dirty_ratio'.
+ * the caller to wait once crossing the (background_thresh + dirty_thresh) / 2.
* If we're over `background_thresh' then the writeback threads are woken to
* perform some writeout.
*/
static void balance_dirty_pages(struct address_space *mapping,
- unsigned long write_chunk)
+ unsigned long pages_dirtied)
{
- unsigned long nr_reclaimable, bdi_nr_reclaimable;
+ unsigned long nr_reclaimable;
unsigned long nr_dirty; /* = file_dirty + writeback + unstable_nfs */
unsigned long bdi_dirty;
unsigned long background_thresh;
unsigned long dirty_thresh;
unsigned long bdi_thresh;
- unsigned long task_bdi_thresh;
- unsigned long min_task_bdi_thresh;
- unsigned long pages_written = 0;
- unsigned long pause = 1;
+ long pause = 0;
bool dirty_exceeded = false;
- bool clear_dirty_exceeded = true;
+ unsigned long task_ratelimit;
+ unsigned long base_rate;
+ unsigned long pos_ratio;
struct backing_dev_info *bdi = mapping->backing_dev_info;
unsigned long start_time = jiffies;
for (;;) {
+ /*
+ * Unstable writes are a feature of certain networked
+ * filesystems (i.e. NFS) in which data may have been
+ * written to the server's write cache, but has not yet
+ * been flushed to permanent storage.
+ */
nr_reclaimable = global_page_state(NR_FILE_DIRTY) +
global_page_state(NR_UNSTABLE_NFS);
nr_dirty = nr_reclaimable + global_page_state(NR_WRITEBACK);
@@ -1027,9 +988,23 @@ static void balance_dirty_pages(struct a
background_thresh))
break;
+ if (unlikely(!writeback_in_progress(bdi)))
+ bdi_start_background_writeback(bdi);
+
+ /*
+ * bdi_thresh is not treated as some limiting factor as
+ * dirty_thresh, due to reasons
+ * - in JBOD setup, bdi_thresh can fluctuate a lot
+ * - in a system with HDD and USB key, the USB key may somehow
+ * go into state (bdi_dirty >> bdi_thresh) either because
+ * bdi_dirty starts high, or because bdi_thresh drops low.
+ * In this case we don't want to hard throttle the USB key
+ * dirtiers for 100 seconds until bdi_dirty drops under
+ * bdi_thresh. Instead the auxiliary bdi control line in
+ * bdi_position_ratio() will let the dirtier task progress
+ * at some rate <= (write_bw / 2) for bringing down bdi_dirty.
+ */
bdi_thresh = bdi_dirty_limit(bdi, dirty_thresh);
- min_task_bdi_thresh = task_min_dirty_limit(bdi_thresh);
- task_bdi_thresh = task_dirty_limit(current, bdi_thresh);
/*
* In order to avoid the stacked BDI deadlock we need
@@ -1041,57 +1016,38 @@ static void balance_dirty_pages(struct a
* actually dirty; with m+n sitting in the percpu
* deltas.
*/
- if (task_bdi_thresh < 2 * bdi_stat_error(bdi)) {
- bdi_nr_reclaimable = bdi_stat_sum(bdi, BDI_RECLAIMABLE);
- bdi_dirty = bdi_nr_reclaimable +
+ if (bdi_thresh < 2 * bdi_stat_error(bdi))
+ bdi_dirty = bdi_stat_sum(bdi, BDI_RECLAIMABLE) +
bdi_stat_sum(bdi, BDI_WRITEBACK);
- } else {
- bdi_nr_reclaimable = bdi_stat(bdi, BDI_RECLAIMABLE);
- bdi_dirty = bdi_nr_reclaimable +
+ else
+ bdi_dirty = bdi_stat(bdi, BDI_RECLAIMABLE) +
bdi_stat(bdi, BDI_WRITEBACK);
- }
- /*
- * The bdi thresh is somehow "soft" limit derived from the
- * global "hard" limit. The former helps to prevent heavy IO
- * bdi or process from holding back light ones; The latter is
- * the last resort safeguard.
- */
- dirty_exceeded = (bdi_dirty > task_bdi_thresh) ||
+ dirty_exceeded = (bdi_dirty > bdi_thresh) ||
(nr_dirty > dirty_thresh);
- clear_dirty_exceeded = (bdi_dirty <= min_task_bdi_thresh) &&
- (nr_dirty <= dirty_thresh);
-
- if (!dirty_exceeded)
- break;
-
- if (!bdi->dirty_exceeded)
+ if (dirty_exceeded && !bdi->dirty_exceeded)
bdi->dirty_exceeded = 1;
bdi_update_bandwidth(bdi, dirty_thresh, background_thresh,
nr_dirty, bdi_thresh, bdi_dirty,
start_time);
- /* Note: nr_reclaimable denotes nr_dirty + nr_unstable.
- * Unstable writes are a feature of certain networked
- * filesystems (i.e. NFS) in which data may have been
- * written to the server's write cache, but has not yet
- * been flushed to permanent storage.
- * Only move pages to writeback if this bdi is over its
- * threshold otherwise wait until the disk writes catch
- * up.
- */
- trace_balance_dirty_start(bdi);
- if (bdi_nr_reclaimable > task_bdi_thresh) {
- pages_written += writeback_inodes_wb(&bdi->wb,
- write_chunk);
- trace_balance_dirty_written(bdi, pages_written);
- if (pages_written >= write_chunk)
- break; /* We've done our duty */
+ base_rate = bdi->dirty_ratelimit;
+ pos_ratio = bdi_position_ratio(bdi, dirty_thresh,
+ background_thresh, nr_dirty,
+ bdi_thresh, bdi_dirty);
+ if (unlikely(pos_ratio == 0)) {
+ pause = MAX_PAUSE;
+ goto pause;
}
+ task_ratelimit = (u64)base_rate *
+ pos_ratio >> RATELIMIT_CALC_SHIFT;
+ pause = (HZ * pages_dirtied) / (task_ratelimit | 1);
+ pause = min(pause, (long)MAX_PAUSE);
+
+pause:
__set_current_state(TASK_UNINTERRUPTIBLE);
io_schedule_timeout(pause);
- trace_balance_dirty_wait(bdi);
dirty_thresh = hard_dirty_limit(dirty_thresh);
/*
@@ -1100,22 +1056,11 @@ static void balance_dirty_pages(struct a
* 200ms is typically more than enough to curb heavy dirtiers;
* (b) the pause time limit makes the dirtiers more responsive.
*/
- if (nr_dirty < dirty_thresh &&
- bdi_dirty < (task_bdi_thresh + bdi_thresh) / 2 &&
- time_after(jiffies, start_time + MAX_PAUSE))
+ if (nr_dirty < dirty_thresh)
break;
-
- /*
- * Increase the delay for each loop, up to our previous
- * default of taking a 100ms nap.
- */
- pause <<= 1;
- if (pause > HZ / 10)
- pause = HZ / 10;
}
- /* Clear dirty_exceeded flag only when no task can exceed the limit */
- if (clear_dirty_exceeded && bdi->dirty_exceeded)
+ if (!dirty_exceeded && bdi->dirty_exceeded)
bdi->dirty_exceeded = 0;
current->nr_dirtied = 0;
@@ -1132,8 +1077,10 @@ static void balance_dirty_pages(struct a
* In normal mode, we start background writeout at the lower
* background_thresh, to keep the amount of dirty memory low.
*/
- if ((laptop_mode && pages_written) ||
- (!laptop_mode && (nr_reclaimable > background_thresh)))
+ if (laptop_mode)
+ return;
+
+ if (nr_reclaimable > background_thresh)
bdi_start_background_writeback(bdi);
}
--- linux-next.orig/include/trace/events/writeback.h 2011-08-26 16:14:45.000000000 +0800
+++ linux-next/include/trace/events/writeback.h 2011-08-26 16:14:46.000000000 +0800
@@ -104,30 +104,6 @@ DEFINE_WRITEBACK_EVENT(writeback_bdi_reg
DEFINE_WRITEBACK_EVENT(writeback_bdi_unregister);
DEFINE_WRITEBACK_EVENT(writeback_thread_start);
DEFINE_WRITEBACK_EVENT(writeback_thread_stop);
-DEFINE_WRITEBACK_EVENT(balance_dirty_start);
-DEFINE_WRITEBACK_EVENT(balance_dirty_wait);
-
-TRACE_EVENT(balance_dirty_written,
-
- TP_PROTO(struct backing_dev_info *bdi, int written),
-
- TP_ARGS(bdi, written),
-
- TP_STRUCT__entry(
- __array(char, name, 32)
- __field(int, written)
- ),
-
- TP_fast_assign(
- strncpy(__entry->name, dev_name(bdi->dev), 32);
- __entry->written = written;
- ),
-
- TP_printk("bdi %s written %d",
- __entry->name,
- __entry->written
- )
-);
DECLARE_EVENT_CLASS(wbc_class,
TP_PROTO(struct writeback_control *wbc, struct backing_dev_info *bdi),
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 15+ messages in thread* [PATCH 07/10] writeback: dirty ratelimit - think time compensation
2011-08-26 11:38 [PATCH 00/10] IO-less dirty throttling v10 Wu Fengguang
` (5 preceding siblings ...)
2011-08-26 11:38 ` [PATCH 06/10] writeback: IO-less balance_dirty_pages() Wu Fengguang
@ 2011-08-26 11:38 ` Wu Fengguang
2011-08-26 12:05 ` Wu Fengguang
2011-08-26 11:38 ` [PATCH 08/10] writeback: trace balance_dirty_pages Wu Fengguang
` (2 subsequent siblings)
9 siblings, 1 reply; 15+ messages in thread
From: Wu Fengguang @ 2011-08-26 11:38 UTC (permalink / raw)
To: linux-fsdevel
Cc: Peter Zijlstra, Wu Fengguang, Andrew Morton, Jan Kara,
Christoph Hellwig, Dave Chinner, Greg Thelen, Minchan Kim,
Vivek Goyal, Andrea Righi, linux-mm, LKML
[-- Attachment #1: think-time-compensation --]
[-- Type: text/plain, Size: 5525 bytes --]
Compensate the task's think time when computing the final pause time,
so that ->dirty_ratelimit can be executed accurately.
think time := time spend outside of balance_dirty_pages()
In the rare case that the task slept longer than the 200ms period time
(result in negative pause time), the sleep time will be compensated in
the following periods, too, if it's less than 1 second.
Accumulated errors are carefully avoided as long as the max pause area
is not hitted.
Pseudo code:
period = pages_dirtied / task_ratelimit;
think = jiffies - dirty_paused_when;
pause = period - think;
1) normal case: period > think
pause = period - think
dirty_paused_when = jiffies + pause
nr_dirtied = 0
period time
|===============================>|
think time pause time
|===============>|==============>|
------|----------------|---------------|------------------------
dirty_paused_when jiffies
2) no pause case: period <= think
don't pause; reduce future pause time by:
dirty_paused_when += period
nr_dirtied = 0
period time
|===============================>|
think time
|===================================================>|
------|--------------------------------+-------------------|----
dirty_paused_when jiffies
Signed-off-by: Wu Fengguang <fengguang.wu@intel.com>
---
include/linux/sched.h | 1 +
kernel/fork.c | 1 +
mm/page-writeback.c | 34 +++++++++++++++++++++++++++++++---
3 files changed, 33 insertions(+), 3 deletions(-)
--- linux-next.orig/include/linux/sched.h 2011-08-16 08:50:41.000000000 +0800
+++ linux-next/include/linux/sched.h 2011-08-16 08:54:13.000000000 +0800
@@ -1531,6 +1531,7 @@ struct task_struct {
*/
int nr_dirtied;
int nr_dirtied_pause;
+ unsigned long dirty_paused_when; /* start of a write-and-pause period */
#ifdef CONFIG_LATENCYTOP
int latency_record_count;
--- linux-next.orig/mm/page-writeback.c 2011-08-16 08:54:08.000000000 +0800
+++ linux-next/mm/page-writeback.c 2011-08-16 08:54:13.000000000 +0800
@@ -913,6 +913,7 @@ static void balance_dirty_pages(struct a
unsigned long background_thresh;
unsigned long dirty_thresh;
unsigned long bdi_thresh;
+ long period;
long pause = 0;
bool dirty_exceeded = false;
unsigned long task_ratelimit;
@@ -922,6 +923,8 @@ static void balance_dirty_pages(struct a
unsigned long start_time = jiffies;
for (;;) {
+ unsigned long now = jiffies;
+
/*
* Unstable writes are a feature of certain networked
* filesystems (i.e. NFS) in which data may have been
@@ -940,8 +943,11 @@ static void balance_dirty_pages(struct a
* when the bdi limits are ramping up.
*/
if (nr_dirty <= dirty_freerun_ceiling(dirty_thresh,
- background_thresh))
+ background_thresh)) {
+ current->dirty_paused_when = now;
+ current->nr_dirtied = 0;
break;
+ }
bdi_thresh = bdi_dirty_limit(bdi, dirty_thresh);
@@ -979,18 +985,41 @@ static void balance_dirty_pages(struct a
background_thresh, nr_dirty,
bdi_thresh, bdi_dirty);
if (unlikely(pos_ratio == 0)) {
+ period = MAX_PAUSE;
pause = MAX_PAUSE;
goto pause;
}
task_ratelimit = (u64)base_rate *
pos_ratio >> RATELIMIT_CALC_SHIFT;
- pause = (HZ * pages_dirtied) / (task_ratelimit | 1);
+ period = (HZ * pages_dirtied) / (task_ratelimit | 1);
+ pause = current->dirty_paused_when + period - now;
+ /*
+ * For less than 1s think time (ext3/4 may block the dirtier
+ * for up to 800ms from time to time on 1-HDD; so does xfs,
+ * however at much less frequency), try to compensate it in
+ * future periods by updating the virtual time; otherwise just
+ * do a reset, as it may be a light dirtier.
+ */
+ if (unlikely(pause <= 0)) {
+ if (pause < -HZ) {
+ current->dirty_paused_when = now;
+ current->nr_dirtied = 0;
+ } else if (period) {
+ current->dirty_paused_when += period;
+ current->nr_dirtied = 0;
+ }
+ pause = 1; /* avoid resetting nr_dirtied_pause below */
+ break;
+ }
pause = min(pause, (long)MAX_PAUSE);
pause:
__set_current_state(TASK_UNINTERRUPTIBLE);
io_schedule_timeout(pause);
+ current->dirty_paused_when = now + pause;
+ current->nr_dirtied = 0;
+
dirty_thresh = hard_dirty_limit(dirty_thresh);
/*
* max-pause area. If dirty exceeded but still within this
@@ -1017,7 +1046,6 @@ pause:
if (!dirty_exceeded && bdi->dirty_exceeded)
bdi->dirty_exceeded = 0;
- current->nr_dirtied = 0;
current->nr_dirtied_pause = dirty_poll_interval(nr_dirty, dirty_thresh);
if (writeback_in_progress(bdi))
--- linux-next.orig/kernel/fork.c 2011-08-16 08:50:41.000000000 +0800
+++ linux-next/kernel/fork.c 2011-08-16 08:54:13.000000000 +0800
@@ -1303,6 +1303,7 @@ static struct task_struct *copy_process(
p->nr_dirtied = 0;
p->nr_dirtied_pause = 128 >> (PAGE_SHIFT - 10);
+ p->dirty_paused_when = 0;
/*
* Ok, make it visible to the rest of the system.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 15+ messages in thread* Re: [PATCH 07/10] writeback: dirty ratelimit - think time compensation
2011-08-26 11:38 ` [PATCH 07/10] writeback: dirty ratelimit - think time compensation Wu Fengguang
@ 2011-08-26 12:05 ` Wu Fengguang
0 siblings, 0 replies; 15+ messages in thread
From: Wu Fengguang @ 2011-08-26 12:05 UTC (permalink / raw)
To: linux-fsdevel@vger.kernel.org
Cc: Peter Zijlstra, Andrew Morton, Jan Kara, Christoph Hellwig,
Dave Chinner, Greg Thelen, Minchan Kim, Vivek Goyal, Andrea Righi,
linux-mm, LKML
> - pause = (HZ * pages_dirtied) / (task_ratelimit | 1);
> + period = (HZ * pages_dirtied) / (task_ratelimit | 1);
> + pause = current->dirty_paused_when + period - now;
> + /*
> + * For less than 1s think time (ext3/4 may block the dirtier
> + * for up to 800ms from time to time on 1-HDD; so does xfs,
> + * however at much less frequency), try to compensate it in
> + * future periods by updating the virtual time; otherwise just
> + * do a reset, as it may be a light dirtier.
> + */
> + if (unlikely(pause <= 0)) {
> + if (pause < -HZ) {
> + current->dirty_paused_when = now;
> + current->nr_dirtied = 0;
> + } else if (period) {
> + current->dirty_paused_when += period;
> + current->nr_dirtied = 0;
> + }
> + pause = 1; /* avoid resetting nr_dirtied_pause below */
Note: the above comment is only effective with the planned max pause
time adaption patch.
Thanks,
Fengguang
> + break;
> + }
> pause = min(pause, (long)MAX_PAUSE);
>
> pause:
> __set_current_state(TASK_UNINTERRUPTIBLE);
> io_schedule_timeout(pause);
>
> + current->dirty_paused_when = now + pause;
> + current->nr_dirtied = 0;
> +
> dirty_thresh = hard_dirty_limit(dirty_thresh);
> /*
> * max-pause area. If dirty exceeded but still within this
> @@ -1017,7 +1046,6 @@ pause:
> if (!dirty_exceeded && bdi->dirty_exceeded)
> bdi->dirty_exceeded = 0;
>
> - current->nr_dirtied = 0;
> current->nr_dirtied_pause = dirty_poll_interval(nr_dirty, dirty_thresh);
>
> if (writeback_in_progress(bdi))
> --- linux-next.orig/kernel/fork.c 2011-08-16 08:50:41.000000000 +0800
> +++ linux-next/kernel/fork.c 2011-08-16 08:54:13.000000000 +0800
> @@ -1303,6 +1303,7 @@ static struct task_struct *copy_process(
>
> p->nr_dirtied = 0;
> p->nr_dirtied_pause = 128 >> (PAGE_SHIFT - 10);
> + p->dirty_paused_when = 0;
>
> /*
> * Ok, make it visible to the rest of the system.
>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH 08/10] writeback: trace balance_dirty_pages
2011-08-26 11:38 [PATCH 00/10] IO-less dirty throttling v10 Wu Fengguang
` (6 preceding siblings ...)
2011-08-26 11:38 ` [PATCH 07/10] writeback: dirty ratelimit - think time compensation Wu Fengguang
@ 2011-08-26 11:38 ` Wu Fengguang
2011-08-26 12:07 ` Wu Fengguang
2011-08-26 11:38 ` [PATCH 09/10] writeback: trace dirty_ratelimit Wu Fengguang
2011-08-26 11:38 ` [PATCH 10/10] trace task_io Wu Fengguang
9 siblings, 1 reply; 15+ messages in thread
From: Wu Fengguang @ 2011-08-26 11:38 UTC (permalink / raw)
To: linux-fsdevel
Cc: Peter Zijlstra, Wu Fengguang, Andrew Morton, Jan Kara,
Christoph Hellwig, Dave Chinner, Greg Thelen, Minchan Kim,
Vivek Goyal, Andrea Righi, linux-mm, LKML
[-- Attachment #1: writeback-trace-balance_dirty_pages.patch --]
[-- Type: text/plain, Size: 4542 bytes --]
Useful for analyzing the dynamics of the throttling algorithms and
debugging user reported problems.
Signed-off-by: Wu Fengguang <fengguang.wu@intel.com>
---
include/trace/events/writeback.h | 79 +++++++++++++++++++++++++++++
mm/page-writeback.c | 24 ++++++++
2 files changed, 103 insertions(+)
--- linux-next.orig/include/trace/events/writeback.h 2011-08-12 18:21:07.000000000 +0800
+++ linux-next/include/trace/events/writeback.h 2011-08-12 18:21:07.000000000 +0800
@@ -270,6 +270,85 @@ TRACE_EVENT(dirty_ratelimit,
)
);
+TRACE_EVENT(balance_dirty_pages,
+
+ TP_PROTO(struct backing_dev_info *bdi,
+ unsigned long thresh,
+ unsigned long bg_thresh,
+ unsigned long dirty,
+ unsigned long bdi_thresh,
+ unsigned long bdi_dirty,
+ unsigned long base_rate,
+ unsigned long task_ratelimit,
+ unsigned long dirtied,
+ unsigned long period,
+ long pause,
+ unsigned long start_time),
+
+ TP_ARGS(bdi, thresh, bg_thresh, dirty, bdi_thresh, bdi_dirty,
+ base_rate, task_ratelimit, dirtied, period, pause, start_time),
+
+ TP_STRUCT__entry(
+ __array( char, bdi, 32)
+ __field(unsigned long, limit)
+ __field(unsigned long, goal)
+ __field(unsigned long, dirty)
+ __field(unsigned long, bdi_goal)
+ __field(unsigned long, bdi_dirty)
+ __field(unsigned long, base_rate)
+ __field(unsigned long, task_ratelimit)
+ __field(unsigned int, dirtied)
+ __field(unsigned int, dirtied_pause)
+ __field(unsigned long, period)
+ __field( long, think)
+ __field( long, pause)
+ __field(unsigned long, paused)
+ ),
+
+ TP_fast_assign(
+ unsigned long freerun = (thresh + bg_thresh) / 2;
+ strlcpy(__entry->bdi, dev_name(bdi->dev), 32);
+
+ __entry->limit = global_dirty_limit;
+ __entry->goal = (global_dirty_limit + freerun) / 2;
+ __entry->dirty = dirty;
+ __entry->bdi_goal = __entry->goal * bdi_thresh / (thresh|1);
+ __entry->bdi_dirty = bdi_dirty;
+ __entry->base_rate = KBps(base_rate);
+ __entry->task_ratelimit = KBps(task_ratelimit);
+ __entry->dirtied = dirtied;
+ __entry->dirtied_pause = current->nr_dirtied_pause;
+ __entry->think = current->dirty_paused_when == 0 ? 0 :
+ (long)(jiffies - current->dirty_paused_when) * 1000/HZ;
+ __entry->period = period * 1000 / HZ;
+ __entry->pause = pause * 1000 / HZ;
+ __entry->paused = (jiffies - start_time) * 1000 / HZ;
+ ),
+
+
+ TP_printk("bdi %s: "
+ "limit=%lu goal=%lu dirty=%lu "
+ "bdi_goal=%lu bdi_dirty=%lu "
+ "base_rate=%lu task_ratelimit=%lu "
+ "dirtied=%u dirtied_pause=%u "
+ "period=%lu think=%ld pause=%ld paused=%lu",
+ __entry->bdi,
+ __entry->limit,
+ __entry->goal,
+ __entry->dirty,
+ __entry->bdi_goal,
+ __entry->bdi_dirty,
+ __entry->base_rate, /* bdi base throttle bandwidth */
+ __entry->task_ratelimit, /* task throttle bandwidth */
+ __entry->dirtied,
+ __entry->dirtied_pause,
+ __entry->period, /* ms */
+ __entry->think, /* ms */
+ __entry->pause, /* ms */
+ __entry->paused /* ms */
+ )
+);
+
DECLARE_EVENT_CLASS(writeback_congest_waited_template,
TP_PROTO(unsigned int usec_timeout, unsigned int usec_delayed),
--- linux-next.orig/mm/page-writeback.c 2011-08-12 18:21:07.000000000 +0800
+++ linux-next/mm/page-writeback.c 2011-08-12 18:22:01.000000000 +0800
@@ -946,6 +946,18 @@ static void balance_dirty_pages(struct a
* do a reset, as it may be a light dirtier.
*/
if (unlikely(pause <= 0)) {
+ trace_balance_dirty_pages(bdi,
+ dirty_thresh,
+ background_thresh,
+ nr_dirty,
+ bdi_thresh,
+ bdi_dirty,
+ base_rate,
+ task_ratelimit,
+ pages_dirtied,
+ period,
+ pause,
+ start_time);
if (pause < -HZ) {
current->dirty_paused_when = now;
current->nr_dirtied = 0;
@@ -959,6 +971,18 @@ static void balance_dirty_pages(struct a
pause = min(pause, MAX_PAUSE);
pause:
+ trace_balance_dirty_pages(bdi,
+ dirty_thresh,
+ background_thresh,
+ nr_dirty,
+ bdi_thresh,
+ bdi_dirty,
+ base_rate,
+ task_ratelimit,
+ pages_dirtied,
+ period,
+ pause,
+ start_time);
__set_current_state(TASK_UNINTERRUPTIBLE);
io_schedule_timeout(pause);
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 15+ messages in thread* Re: [PATCH 08/10] writeback: trace balance_dirty_pages
2011-08-26 11:38 ` [PATCH 08/10] writeback: trace balance_dirty_pages Wu Fengguang
@ 2011-08-26 12:07 ` Wu Fengguang
0 siblings, 0 replies; 15+ messages in thread
From: Wu Fengguang @ 2011-08-26 12:07 UTC (permalink / raw)
To: linux-fsdevel@vger.kernel.org
Cc: Peter Zijlstra, Andrew Morton, Jan Kara, Christoph Hellwig,
Dave Chinner, Greg Thelen, Minchan Kim, Vivek Goyal, Andrea Righi,
linux-mm, LKML
> + TP_printk("bdi %s: "
> + "limit=%lu goal=%lu dirty=%lu "
> + "bdi_goal=%lu bdi_dirty=%lu "
> + "base_rate=%lu task_ratelimit=%lu "
> + "dirtied=%u dirtied_pause=%u "
> + "period=%lu think=%ld pause=%ld paused=%lu",
Sorry, forgot the rename: base_rate => dirty_ratelimit.
Thanks,
Fengguang
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH 09/10] writeback: trace dirty_ratelimit
2011-08-26 11:38 [PATCH 00/10] IO-less dirty throttling v10 Wu Fengguang
` (7 preceding siblings ...)
2011-08-26 11:38 ` [PATCH 08/10] writeback: trace balance_dirty_pages Wu Fengguang
@ 2011-08-26 11:38 ` Wu Fengguang
2011-08-26 11:38 ` [PATCH 10/10] trace task_io Wu Fengguang
9 siblings, 0 replies; 15+ messages in thread
From: Wu Fengguang @ 2011-08-26 11:38 UTC (permalink / raw)
To: linux-fsdevel
Cc: Peter Zijlstra, Wu Fengguang, Andrew Morton, Jan Kara,
Christoph Hellwig, Dave Chinner, Greg Thelen, Minchan Kim,
Vivek Goyal, Andrea Righi, linux-mm, LKML
[-- Attachment #1: writeback-trace-throttle-bandwidth.patch --]
[-- Type: text/plain, Size: 2939 bytes --]
It helps understand how various throttle bandwidths are updated.
Signed-off-by: Wu Fengguang <fengguang.wu@intel.com>
---
include/trace/events/writeback.h | 46 +++++++++++++++++++++++++++++
mm/page-writeback.c | 3 +
2 files changed, 49 insertions(+)
--- linux-next.orig/mm/page-writeback.c 2011-08-26 19:27:21.000000000 +0800
+++ linux-next/mm/page-writeback.c 2011-08-26 19:27:26.000000000 +0800
@@ -864,6 +864,9 @@ static void bdi_update_dirty_ratelimit(s
dirty_ratelimit -= step;
bdi->dirty_ratelimit = max(dirty_ratelimit, 1UL);
+
+ trace_dirty_ratelimit(bdi, dirty_rate, task_ratelimit,
+ balanced_dirty_ratelimit);
}
void __bdi_update_bandwidth(struct backing_dev_info *bdi,
--- linux-next.orig/include/trace/events/writeback.h 2011-08-26 19:27:21.000000000 +0800
+++ linux-next/include/trace/events/writeback.h 2011-08-26 19:27:23.000000000 +0800
@@ -305,6 +305,52 @@ TRACE_EVENT(balance_dirty_pages,
)
);
+#define KBps(x) ((x) << (PAGE_SHIFT - 10))
+
+TRACE_EVENT(dirty_ratelimit,
+
+ TP_PROTO(struct backing_dev_info *bdi,
+ unsigned long dirty_rate,
+ unsigned long task_ratelimit,
+ unsigned long balanced_dirty_ratelimit),
+
+ TP_ARGS(bdi, dirty_rate, task_ratelimit, balanced_dirty_ratelimit),
+
+ TP_STRUCT__entry(
+ __array(char, bdi, 32)
+ __field(unsigned long, write_bw)
+ __field(unsigned long, avg_write_bw)
+ __field(unsigned long, dirty_rate)
+ __field(unsigned long, dirty_ratelimit)
+ __field(unsigned long, task_ratelimit)
+ __field(unsigned long, balanced_dirty_ratelimit)
+ ),
+
+ TP_fast_assign(
+ strlcpy(__entry->bdi, dev_name(bdi->dev), 32);
+ __entry->write_bw = KBps(bdi->write_bandwidth);
+ __entry->avg_write_bw = KBps(bdi->avg_write_bandwidth);
+ __entry->dirty_rate = KBps(dirty_rate);
+ __entry->dirty_ratelimit = KBps(bdi->dirty_ratelimit);
+ __entry->task_ratelimit = KBps(task_ratelimit);
+ __entry->balanced_dirty_ratelimit =
+ KBps(balanced_dirty_ratelimit);
+ ),
+
+ TP_printk("bdi %s: "
+ "write_bw=%lu awrite_bw=%lu dirty_rate=%lu "
+ "dirty_ratelimit=%lu task_ratelimit=%lu "
+ "balanced_dirty_ratelimit=%lu",
+ __entry->bdi,
+ __entry->write_bw, /* write bandwidth */
+ __entry->avg_write_bw, /* avg write bandwidth */
+ __entry->dirty_rate, /* bdi dirty rate */
+ __entry->dirty_ratelimit, /* base ratelimit */
+ __entry->task_ratelimit, /* ratelimit with position control */
+ __entry->balanced_dirty_ratelimit /* the balanced ratelimit */
+ )
+);
+
DECLARE_EVENT_CLASS(writeback_congest_waited_template,
TP_PROTO(unsigned int usec_timeout, unsigned int usec_delayed),
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 15+ messages in thread* [PATCH 10/10] trace task_io
2011-08-26 11:38 [PATCH 00/10] IO-less dirty throttling v10 Wu Fengguang
` (8 preceding siblings ...)
2011-08-26 11:38 ` [PATCH 09/10] writeback: trace dirty_ratelimit Wu Fengguang
@ 2011-08-26 11:38 ` Wu Fengguang
9 siblings, 0 replies; 15+ messages in thread
From: Wu Fengguang @ 2011-08-26 11:38 UTC (permalink / raw)
To: linux-fsdevel
Cc: Peter Zijlstra, Wu Fengguang, Andrew Morton, Jan Kara,
Christoph Hellwig, Dave Chinner, Greg Thelen, Minchan Kim,
Vivek Goyal, Andrea Righi, linux-mm, LKML
[-- Attachment #1: writeback-trace-task-total-dirtied.patch --]
[-- Type: text/plain, Size: 2259 bytes --]
The "write_bytes" curve over time is very handy for viewing the
smoothness and fairness of page dirties for each task.
XXX: It looks not a good fit for writeback traces, shall we create a
include/trace/events/vfs.h for this?
XXX: the other fields of struct task_io_accounting are not dumped.
Signed-off-by: Wu Fengguang <fengguang.wu@intel.com>
---
include/trace/events/writeback.h | 32 +++++++++++++++++++++++++++++
mm/page-writeback.c | 1
2 files changed, 33 insertions(+)
--- linux-next.orig/include/trace/events/writeback.h 2011-08-22 11:58:21.000000000 +0800
+++ linux-next/include/trace/events/writeback.h 2011-08-22 12:22:38.000000000 +0800
@@ -180,6 +180,38 @@ TRACE_EVENT(writeback_queue_io,
__entry->moved)
);
+TRACE_EVENT(task_io,
+ TP_PROTO(struct task_struct *task),
+ TP_ARGS(task),
+
+ TP_STRUCT__entry(
+ __field(unsigned long long, read_bytes)
+ __field(unsigned long long, write_bytes)
+ __field(unsigned long long, cancelled_write_bytes)
+ ),
+
+ TP_fast_assign(
+ struct task_io_accounting *ioac = &task->ioac;
+
+#ifdef CONFIG_TASK_IO_ACCOUNTING
+ __entry->read_bytes = ioac->read_bytes;
+ __entry->write_bytes = ioac->write_bytes;
+ __entry->cancelled_write_bytes = ioac->cancelled_write_bytes;
+#else
+ __entry->read_bytes = 0;
+ __entry->write_bytes = 0;
+ __entry->cancelled_write_bytes = 0;
+#endif
+ ),
+
+ TP_printk("read=%llu write=%llu cancelled_write=%llu",
+ __entry->read_bytes,
+ __entry->write_bytes,
+ __entry->cancelled_write_bytes
+ )
+);
+
+
TRACE_EVENT(global_dirty_state,
TP_PROTO(unsigned long background_thresh,
--- linux-next.orig/mm/page-writeback.c 2011-08-22 12:10:33.000000000 +0800
+++ linux-next/mm/page-writeback.c 2011-08-22 12:20:33.000000000 +0800
@@ -407,6 +407,7 @@ void global_dirty_limits(unsigned long *
*pbackground = background;
*pdirty = dirty;
trace_global_dirty_state(background, dirty);
+ trace_task_io(current);
}
/**
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 15+ messages in thread