wb-buf-throttle feedback

wb-buf-throttle feedback

[Ritesh Raj Sarraf]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Hi Jens,

[Please CC me in replies. I'm not subscribed to the list]

Based on your announcement about the writeback patches, I have picked up your
changes for testing.

I tested it on a 4.7.3 stable kernel with patches applied from your axboe/wb-
buf-throttle-v4.7 branch.


Instead of the infamous dd use case, I picked up the other generic one i.e. to
build the linux kernel.


Machine config:

Intel(R) Core(TM) i5-4210U CPU @ 1.70GHzRAM: 8 GiB
Disk: Rotational SATA HDD Hitachi - 7200 RPM


The machine has 4 logical CPUs. 


Unfortunately, the issue is still easily reproducible. Building a kernel with
defaults, which results in make -j4, ended up stalling the entire OS.

systemd-journald couldn't really log the messages because it too gets affected
by this writeback issue. 

https://github.com/systemd/systemd/issues/1353

The only message it logged are:

Sep 07 16:43:12 chutzpah systemd-journald[348]: Runtime journal
(/run/log/journal/) is 8.0M, max 73.8M, 65.8M free.
Sep 07 16:43:13 chutzpah systemd-journald[348]: System journal
(/var/log/journal/) is 3.9G, max 4.0G, 7.7M free.
Sep 07 16:43:14 chutzpah systemd-journald[348]: Time spent on flushing to /var
is 1.790678s for 2 entries.
Sep 07 16:43:20 chutzpah systemd-journald[348]: Journal started
Sep 08 00:28:55 chutzpah systemd-journald[348]: Suppressed 454 messages from
/system.slice/laptop-mode.service
Sep 08 23:07:29 chutzpah systemd-journald[348]: Suppressed 79 messages from
/user.slice/user-1000.slice
Sep 09 19:48:02 chutzpah systemd-journald[28710]: System journal
(/var/log/journal/) is 4.0G, max 4.0G, 0B free.
Sep 09 19:49:45 chutzpah systemd-journald[5986]: System journal
(/var/log/journal/) is 4.0G, max 4.0G, 0B free.
Sep 09 19:50:07 chutzpah systemd-journald[5986]: Journal started
Sep 09 19:43:30 chutzpah systemd[1]: systemd-journald.service: Watchdog timeout
(limit 3min)!
Sep 09 19:43:30 chutzpah systemd[1]: systemd-journald.service: Killing process
348 (systemd-journal) with signal SIGABRT.
- -- Reboot --

Which is self-explanatory.


At this time, the easy way to workaround the bug is to spare some resources.
In the above use case of mine, I've been working it around by consuming lesser
resources than what the machine has.

So my successful kernel builds have been firing `make -j3` instead. Not
efficient but definitely better than bringing the entire OS to a halt.


- -- 
Ritesh Raj Sarraf
RESEARCHUT - http://www.researchut.com
"Necessity is the mother of invention."
-----BEGIN PGP SIGNATURE-----

iQIcBAEBCgAGBQJX0wbqAAoJEKY6WKPy4XVpOZMP/jjSCCIwl5yekBnwtjITg1aO
2e6LLiNRn40H0+j6qwgfGU2ho+pptjxHPQSC0ViE/cICZ8QSmWzrNRhQgVnNj+ur
g8cADdBZT/0+GJ8pvbpmynL3LIK8rZuSbeXwdlKBYlZQiiMoCzUSA96l78IQ5R40
CyL8HP1JWmixxPkEPVX/xQ7cfdernrokFwLhwGkeWaqM9wiGutaZo9PY/4PBPXgn
A/P9Xrg3dAxu2OCiZNVIuiP3NC/AJ7S14FiUCCqOYGlvjlxwSBgCaNzHvYCzRcEs
xAV4XGu8DQMFOmHQ7PHXWjb5V2dwgYKDUc79sxzj8m9sF/UKFb53QFrA6S6z8jHO
hLXfiiimaYqI3qEwLw6lqkZ9EzUaIWxVW8gfFDqAV1xkYs1vcEpxkR6MpVPoK2UY
EaFNO+h3aWB6i9FxTn3x3pLKQHRoNEYORFEmMCPktlXZaqPMCORO0rKKfOPmr/DZ
wEDt8PHP+H0y74I6ZEwDv4jJCsqkz1IXc5narVmMAQ6bl64D6o5kUqNWLlfBqv24
R8ata+15oFsAD1LtVFo68ugR+7BFpHDL/1SClgfKrEECVefsQcY+ticO53tnQUoZ
6yHO2f7vqe1InVFoZh07g2jbEsSApBe/XX1jNffklMC2WQI1cHeRYJscI1izQjHO
S7rMxqUE3UhQYAtICn0b
=4zuC
-----END PGP SIGNATURE-----

Re: wb-buf-throttle feedback

[Jens Axboe]

On 09/09/2016 01:00 PM, Ritesh Raj Sarraf wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
>
> Hi Jens,
>
> [Please CC me in replies. I'm not subscribed to the list]
>
> Based on your announcement about the writeback patches, I have picked up your
> changes for testing.
>
> I tested it on a 4.7.3 stable kernel with patches applied from your axboe/wb-
> buf-throttle-v4.7 branch.
>
>
> Instead of the infamous dd use case, I picked up the other generic one i.e. to
> build the linux kernel.
>
>
> Machine config:
>
> Intel(R) Core(TM) i5-4210U CPU @ 1.70GHzRAM: 8 GiB
> Disk: Rotational SATA HDD Hitachi - 7200 RPM
>
>
> The machine has 4 logical CPUs.
>
>
> Unfortunately, the issue is still easily reproducible. Building a kernel with
> defaults, which results in make -j4, ended up stalling the entire OS.

What drive is hosting the system and kernel build? I think the current
branch isn't aggressive enough on SATA style QD=32 devices with write
back caching, so if that's what you are running, it'd be great if you
could try a patch or two so we can improve this situation for you.

The v4.7 branch is outdated. Let me spin a new one, that's probably the
best way to start.

-- 
Jens Axboe

Re: wb-buf-throttle feedback

[Jens Axboe]

On 09/09/2016 02:02 PM, Jens Axboe wrote:
> On 09/09/2016 01:00 PM, Ritesh Raj Sarraf wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA512
>>
>> Hi Jens,
>>
>> [Please CC me in replies. I'm not subscribed to the list]
>>
>> Based on your announcement about the writeback patches, I have picked
>> up your
>> changes for testing.
>>
>> I tested it on a 4.7.3 stable kernel with patches applied from your
>> axboe/wb-
>> buf-throttle-v4.7 branch.
>>
>>
>> Instead of the infamous dd use case, I picked up the other generic one
>> i.e. to
>> build the linux kernel.
>>
>>
>> Machine config:
>>
>> Intel(R) Core(TM) i5-4210U CPU @ 1.70GHzRAM: 8 GiB
>> Disk: Rotational SATA HDD Hitachi - 7200 RPM
>>
>>
>> The machine has 4 logical CPUs.
>>
>>
>> Unfortunately, the issue is still easily reproducible. Building a
>> kernel with
>> defaults, which results in make -j4, ended up stalling the entire OS.
>
> What drive is hosting the system and kernel build? I think the current
> branch isn't aggressive enough on SATA style QD=32 devices with write
> back caching, so if that's what you are running, it'd be great if you
> could try a patch or two so we can improve this situation for you.
>
> The v4.7 branch is outdated. Let me spin a new one, that's probably the
> best way to start.

The 4.7 branch has now been updated, it's fully uptodate with the
current 4.8 based branch. Would be great if you could try that. If you
still see slowdowns, try and change RWB_DEF_DEPTH. It's currently at 16,
scale it down to 4 or something like that.

-- 
Jens Axboe

Re: wb-buf-throttle feedback

[Ritesh Raj Sarraf]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Hello Jens,


On Fri, 2016-09-09 at 14:19 -0600, Jens Axboe wrote:
> > What drive is hosting the system and kernel build? I think the current
> > branch isn't aggressive enough on SATA style QD=32 devices with write
> > back caching, so if that's what you are running, it'd be great if you
> > could try a patch or two so we can improve this situation for you.
> >
> > The v4.7 branch is outdated. Let me spin a new one, that's probably the
> > best way to start.
> 

The drive is:

Sep 10 18:19:32 chutzpah kernel: ata1: SATA link up 6.0 Gbps (SStatus 133
SControl 300)
Sep 10 18:19:32 chutzpah kernel: ata1.00: ATA-9: HGST HTS725050A7E630, GH2OA910,
max UDMA/133
Sep 10 18:19:32 chutzpah kernel: ata1.00: 976773168 sectors, multi 16: LBA48 NCQ
(depth 31/32), AA
Sep 10 18:19:32 chutzpah kernel: ata1.00: configured for UDMA/133
Sep 10 18:19:32 chutzpah kernel: scsi 0:0:0:0: Direct-Access     ATA      HGST
HTS725050A7 A910 PQ: 0 ANSI: 5
Sep 10 18:19:32 chutzpah kernel: sd 0:0:0:0: [sda] 976773168 512-byte logical
blocks: (500 GB/466 GiB)
Sep 10 18:19:32 chutzpah kernel: sd 0:0:0:0: [sda] 4096-byte physical blocks
Sep 10 18:19:32 chutzpah kernel: sd 0:0:0:0: [sda] Write Protect is off
Sep 10 18:19:32 chutzpah kernel: sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
Sep 10 18:19:32 chutzpah kernel: sd 0:0:0:0: [sda] Write cache: enabled, read
cache: enabled, doesn't support DPO or FUA
Sep 10 18:19:32 chutzpah kernel: usb 1-4: new full-speed USB device number 2
using xhci_hcd
Sep 10 18:19:32 chutzpah kernel:  sda: sda1 sda2 sda3 sda4 < sda5 >
Sep 10 18:19:32 chutzpah kernel: sd 0:0:0:0: [sda] Attached SCSI disk


> The 4.7 branch has now been updated, it's fully uptodate with the
> current 4.8 based branch. Would be great if you could try that. If you
> still see slowdowns, try and change RWB_DEF_DEPTH. It's currently at 16,
> scale it down to 4 or something like that.


I pulled in your changes from the axboe/wb-buf-throttle-v4.7 branch.

I can confirm now that, with these changes, the machine is very well responsive.
I tried both, the kernel build and the dd tests. Both of them worked fine.

Thank you very much.

Ritesh

- -- 
Ritesh Raj Sarraf
RESEARCHUT - http://www.researchut.com
"Necessity is the mother of invention."
-----BEGIN PGP SIGNATURE-----

iQIcBAEBCgAGBQJX1SoIAAoJEKY6WKPy4XVpleIQAJJMvcOz/O64CFH9VhMsBiR0
Pu3CufAgy8rNOlNBXrGdTKwH3rRnQSZiHkKd/o3uq1BjVPSKHK4YQnigImQwgzbe
SJdWmpaKKX1Pw5eo3LMU7Q+TU/CT5jzmiMk1s1HHeuqdW4DNOmYWQi4CqsLEp3ry
CDP5b1vygWGprrRNeADTQo6NzTS/TG32pFDQjj+/07gXPz61YfBtAnOD56u9328N
N8OJaPjWcdPPERFSoFW3HZVfyMtKNQ8OGKUf9oFWe/0eX+Bqji1L8Om5NebJw/Yv
YTzNTsC7WXARoifCI8GnmgoGgf51o+7qwdWG60iAJ+IO9FI8AAgFM4zRd3PPItqw
kOJWj6zN1Zr1YeAU9m8Z0muhyZ5a+Gj4J+4AuIRHLOaXRm05jm4mnVovpnP/i/dE
N4eTs/I6UWgGkTb/8FAfzr/Ql5l07EiZ8HItWX3YT4RfyoA6R7GZy7kf4E3kueFq
v2HwoLfBqj2NuLN714X106pAMEfw2pIbkWdtTUq47ybeh5YRvlkz46WjcRB88YKq
piRLSXVYpVIfG16rYQArDlum6RfZXyrRVjEczpOMVr+ZqZNWmFSqNCuYOPqfD5Mg
s/gTUwgNdAf/J725/GUdfQVDjootIAFY96vyUbVWEQpKz3HedME+sebhF8/OveQg
ziTHWyJWU9t642fTeNLC
=mW9B
-----END PGP SIGNATURE-----