* Hangs on bcache on HEAD and bcache-testing when stacking LVM on top (?)
@ 2013-04-04 10:27 Heiko Wundram
[not found] ` <515D5592.60906-EqIAFqbRPK3NLxjTenLetw@public.gmane.org>
0 siblings, 1 reply; 2+ messages in thread
From: Heiko Wundram @ 2013-04-04 10:27 UTC (permalink / raw)
To: linux-bcache-u79uwXL29TY76Z2rM5mHXA
Hey!
I've already written previously about write hangs with bcache in "bcache
hangs with continuous write I/O to SSD device, bcache device stops
working", and I thought, that the problem was solved by statically
building bcache into the kernel, but that does not seem to be the case.
I've now encountered something similar, which seems to occur after
around 3-4 days of usage of the bcache device (I've had it twice now).
The bcache device (or rather, the devices stacked on the bcache device
through LVM) hang, and show 100% CPU utilization. All other devices
which are not on the bcache volume still work properly. There's no
obvious kernel thread which is taking CPU time and blocking the device
(as was with the last bug report I noted).
When the bcache volume is blocked, iostat reports:
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0,00 0,00 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,00 0,00
sdb 0,00 0,50 0,00 8,50 0,00 0,03
6,53 0,06 6,88 0,00 6,88 4,88 4,15
sdc 0,00 0,50 0,00 8,50 0,00 0,03
6,53 0,06 7,24 0,00 7,24 5,29 4,50
md127 0,00 0,00 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,00 0,00
md126 0,00 0,00 0,00 6,50 0,00 0,03
8,00 0,00 0,00 0,00 0,00 0,00 0,00
md125 0,00 0,00 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,00 0,00
dm-0 0,00 0,00 0,00 6,50 0,00 0,03
8,00 0,13 20,00 0,00 20,00 5,15 3,35
dm-1 0,00 0,00 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,00 0,00
dm-2 0,00 0,00 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,00 0,00
bcache0 0,00 0,00 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,00 0,00
dm-3 0,00 0,00 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,00 0,00
dm-4 0,00 0,00 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,00 0,00
dm-5 0,00 0,00 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,00 0,00
dm-6 0,00 0,00 0,00 0,00 0,00 0,00
0,00 1,00 0,00 0,00 0,00 0,00 100,00
dm-7 0,00 0,00 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,00 0,00
dm-8 0,00 0,00 0,00 0,00 0,00 0,00
0,00 1,00 0,00 0,00 0,00 0,00 100,00
dm-9 0,00 0,00 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,00 0,00
dm-10 0,00 0,00 0,00 0,00 0,00 0,00
0,00 3,00 0,00 0,00 0,00 0,00 100,00
The devices starting with dm-3 are logical volumes in a volume group
which resides on (the only) physical volume /dev/bcache0, which in turn
is made up of /dev/sda (SSD device) and /dev/md127 (backing volume,
kernel RAID1). /dev/md127 is made up of GPT partitions on /dev/sdb and
/dev/sdc.
The bcache volume and the cache is set to use 512byte logical sectors,
which both devices support (although they both have 4k physical sectors).
There's no real clue as to where I might start to look from here. Is
this a "known" problem when using bcache together with LVM? Memory usage
does not seem to be a culprit.
I'm now trying to boot the system using kernel 3.2 (i.e., the bcache-3.2
tree), and can report about that.
Thanks for any pointers!
--
--- Heiko.
^ permalink raw reply [flat|nested] 2+ messages in thread[parent not found: <515D5592.60906-EqIAFqbRPK3NLxjTenLetw@public.gmane.org>]
* Re: Hangs on bcache on HEAD and bcache-testing when stacking LVM on top (?) [not found] ` <515D5592.60906-EqIAFqbRPK3NLxjTenLetw@public.gmane.org> @ 2013-04-05 21:29 ` Kent Overstreet 0 siblings, 0 replies; 2+ messages in thread From: Kent Overstreet @ 2013-04-05 21:29 UTC (permalink / raw) To: Heiko Wundram; +Cc: linux-bcache-u79uwXL29TY76Z2rM5mHXA On Thu, Apr 04, 2013 at 12:27:30PM +0200, Heiko Wundram wrote: > Hey! > > I've already written previously about write hangs with bcache in > "bcache hangs with continuous write I/O to SSD device, bcache device > stops working", and I thought, that the problem was solved by > statically building bcache into the kernel, but that does not seem > to be the case. Hey - sorry for the delay; I started looking at your original bug report but I've been moving the past week so I still haven't been able to spend much time on it. I did a _bit_ of poking though, and I suspect it's related to background writeback - any chance you could confirm/deny? Might be worth flipping off writeback_running when it happens, and seeing if load returns to normal. Your bug reports were a big help in figuring that much out though - thanks! > I've now encountered something similar, which seems to occur after > around 3-4 days of usage of the bcache device (I've had it twice > now). The bcache device (or rather, the devices stacked on the > bcache device through LVM) hang, and show 100% CPU utilization. All > other devices which are not on the bcache volume still work > properly. There's no obvious kernel thread which is taking CPU time > and blocking the device (as was with the last bug report I noted). > > When the bcache volume is blocked, iostat reports: > > Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s > avgrq-sz avgqu-sz await r_await w_await svctm %util > sda 0,00 0,00 0,00 0,00 0,00 0,00 > 0,00 0,00 0,00 0,00 0,00 0,00 0,00 > sdb 0,00 0,50 0,00 8,50 0,00 0,03 > 6,53 0,06 6,88 0,00 6,88 4,88 4,15 > sdc 0,00 0,50 0,00 8,50 0,00 0,03 > 6,53 0,06 7,24 0,00 7,24 5,29 4,50 > md127 0,00 0,00 0,00 0,00 0,00 0,00 > 0,00 0,00 0,00 0,00 0,00 0,00 0,00 > md126 0,00 0,00 0,00 6,50 0,00 0,03 > 8,00 0,00 0,00 0,00 0,00 0,00 0,00 > md125 0,00 0,00 0,00 0,00 0,00 0,00 > 0,00 0,00 0,00 0,00 0,00 0,00 0,00 > dm-0 0,00 0,00 0,00 6,50 0,00 0,03 > 8,00 0,13 20,00 0,00 20,00 5,15 3,35 > dm-1 0,00 0,00 0,00 0,00 0,00 0,00 > 0,00 0,00 0,00 0,00 0,00 0,00 0,00 > dm-2 0,00 0,00 0,00 0,00 0,00 0,00 > 0,00 0,00 0,00 0,00 0,00 0,00 0,00 > bcache0 0,00 0,00 0,00 0,00 0,00 0,00 > 0,00 0,00 0,00 0,00 0,00 0,00 0,00 > dm-3 0,00 0,00 0,00 0,00 0,00 0,00 > 0,00 0,00 0,00 0,00 0,00 0,00 0,00 > dm-4 0,00 0,00 0,00 0,00 0,00 0,00 > 0,00 0,00 0,00 0,00 0,00 0,00 0,00 > dm-5 0,00 0,00 0,00 0,00 0,00 0,00 > 0,00 0,00 0,00 0,00 0,00 0,00 0,00 > dm-6 0,00 0,00 0,00 0,00 0,00 0,00 > 0,00 1,00 0,00 0,00 0,00 0,00 100,00 > dm-7 0,00 0,00 0,00 0,00 0,00 0,00 > 0,00 0,00 0,00 0,00 0,00 0,00 0,00 > dm-8 0,00 0,00 0,00 0,00 0,00 0,00 > 0,00 1,00 0,00 0,00 0,00 0,00 100,00 > dm-9 0,00 0,00 0,00 0,00 0,00 0,00 > 0,00 0,00 0,00 0,00 0,00 0,00 0,00 > dm-10 0,00 0,00 0,00 0,00 0,00 0,00 > 0,00 3,00 0,00 0,00 0,00 0,00 100,00 > > The devices starting with dm-3 are logical volumes in a volume group > which resides on (the only) physical volume /dev/bcache0, which in > turn is made up of /dev/sda (SSD device) and /dev/md127 (backing > volume, kernel RAID1). /dev/md127 is made up of GPT partitions on > /dev/sdb and /dev/sdc. > > The bcache volume and the cache is set to use 512byte logical > sectors, which both devices support (although they both have 4k > physical sectors). > > There's no real clue as to where I might start to look from here. Is > this a "known" problem when using bcache together with LVM? Memory > usage does not seem to be a culprit. > > I'm now trying to boot the system using kernel 3.2 (i.e., the > bcache-3.2 tree), and can report about that. > > Thanks for any pointers! > > -- > --- Heiko. > -- > To unsubscribe from this list: send the line "unsubscribe linux-bcache" in > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2013-04-05 21:29 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-04-04 10:27 Hangs on bcache on HEAD and bcache-testing when stacking LVM on top (?) Heiko Wundram
[not found] ` <515D5592.60906-EqIAFqbRPK3NLxjTenLetw@public.gmane.org>
2013-04-05 21:29 ` Kent Overstreet
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox