Issues dropping vm fs caches with kernel 3.5.2 + bcache master

Issues dropping vm fs caches with kernel 3.5.2 + bcache master

[Javier Marcet]

Hi,

I've recently set up a software RAID5 + bcache.

I'm using kernel 3.5.2 merged with bcache master branch. I'm quite
happy with the performance but
there is an important issue. I can't drop caches. Writing any number
to /proc/sys/vm/drop_caches
makes the system hungs.

Furthermore, so far I've been unable to curb bcache's memory
consumption using the available
sysctl knobs.

Is this problem known?


-- 
Javier Marcet <jmarcet-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>

Re: Issues dropping vm fs caches with kernel 3.5.2 + bcache master

[Kent Overstreet]

On Mon, Aug 20, 2012 at 09:03:08AM +0200, Javier Marcet wrote:
> Hi,
> 
> I've recently set up a software RAID5 + bcache.
> 
> I'm using kernel 3.5.2 merged with bcache master branch. I'm quite
> happy with the performance but
> there is an important issue. I can't drop caches. Writing any number
> to /proc/sys/vm/drop_caches
> makes the system hungs.
> 
> Furthermore, so far I've been unable to curb bcache's memory
> consumption using the available
> sysctl knobs.
> 
> Is this problem known?

No I didn't know about that - thanks, I'll look into it.

Just out of curiousity, what are you doing? drop_caches isn't something
you want to use in normal operation...

Re: Issues dropping vm fs caches with kernel 3.5.2 + bcache master

[Javier Marcet]

>> I've recently set up a software RAID5 + bcache.
>>
>> I'm using kernel 3.5.2 merged with bcache master branch. I'm quite
>> happy with the performance but
>> there is an important issue. I can't drop caches. Writing any number
>> to /proc/sys/vm/drop_caches
>> makes the system hung.
>>
>> Furthermore, so far I've been unable to curb bcache's memory
>> consumption using the available
>> sysctl knobs.
>>
>> Is this problem known?
>
> No I didn't know about that - thanks, I'll look into it.

Thanks :)

> Just out of curiousity, what are you doing? drop_caches isn't something
> you want to use in normal operation...

Yeah, not normal operation, but this machine has 8 GB of ram and nearly 7
were being used by caches. I tried to launch a virtual machine and I couldn't.

Seeing the state the memory was in, I went off to drop the caches, and... hang.


-- 
Javier Marcet <jmarcet-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>

Re: Issues dropping vm fs caches with kernel 3.5.2 + bcache master

[Kent Overstreet]

On Mon, Aug 20, 2012 at 11:06:30PM +0200, Javier Marcet wrote:
> >> I've recently set up a software RAID5 + bcache.
> >>
> >> I'm using kernel 3.5.2 merged with bcache master branch. I'm quite
> >> happy with the performance but
> >> there is an important issue. I can't drop caches. Writing any number
> >> to /proc/sys/vm/drop_caches
> >> makes the system hung.
> >>
> >> Furthermore, so far I've been unable to curb bcache's memory
> >> consumption using the available
> >> sysctl knobs.
> >>
> >> Is this problem known?
> >
> > No I didn't know about that - thanks, I'll look into it.

Just tried it - I can reproduce it, but it's gone in 3.6. Looks like a
3.5 bug, not a bcache bug.

I'm gonna reset the master branch to my current dev branch soon as I've
tested it a bit more, but I'd suggest the bcache-3.2 branch - that's the
best tested code.

Re: Issues dropping vm fs caches with kernel 3.5.2 + bcache master

[Javier Marcet]

>> >> I've recently set up a software RAID5 + bcache.
>> >>
>> >> I'm using kernel 3.5.2 merged with bcache master branch. I'm quite
>> >> happy with the performance but
>> >> there is an important issue. I can't drop caches. Writing any number
>> >> to /proc/sys/vm/drop_caches
>> >> makes the system hung.
>> >>
>> >> Furthermore, so far I've been unable to curb bcache's memory
>> >> consumption using the available
>> >> sysctl knobs.
>> >>
>> >> Is this problem known?
>> >
>> > No I didn't know about that - thanks, I'll look into it.
>
> Just tried it - I can reproduce it, but it's gone in 3.6. Looks like a
> 3.5 bug, not a bcache bug.

Oh, it was a coincidence then. I hadn't needed to use that for a long
time.

> I'm gonna reset the master branch to my current dev branch soon as I've
> tested it a bit more, but I'd suggest the bcache-3.2 branch - that's the
> best tested code.

I cannot use a 3.2 kernel. This an HTPC too, besides a NAS and host for
vm guests. The tuners I have are supported from 3.4 onwards.

Do you mean there might be more problems with the master branch?

By the way. I have read in the manual how to add new cache devices, but
can you remove completely the caches in a 'clean way'? Imagine to change
it to a different disk which for some reason you cannot use at the same time.


-- 
Javier Marcet <jmarcet-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>

Re: Issues dropping vm fs caches with kernel 3.5.2 + bcache master

[Kent Overstreet]

On Tue, Aug 21, 2012 at 02:31:18AM +0200, Javier Marcet wrote:
> >> >> I've recently set up a software RAID5 + bcache.
> >> >>
> >> >> I'm using kernel 3.5.2 merged with bcache master branch. I'm quite
> >> >> happy with the performance but
> >> >> there is an important issue. I can't drop caches. Writing any number
> >> >> to /proc/sys/vm/drop_caches
> >> >> makes the system hung.
> >> >>
> >> >> Furthermore, so far I've been unable to curb bcache's memory
> >> >> consumption using the available
> >> >> sysctl knobs.
> >> >>
> >> >> Is this problem known?
> >> >
> >> > No I didn't know about that - thanks, I'll look into it.
> >
> > Just tried it - I can reproduce it, but it's gone in 3.6. Looks like a
> > 3.5 bug, not a bcache bug.
> 
> Oh, it was a coincidence then. I hadn't needed to use that for a long
> time.

Yeah. Weird bug.

> > I'm gonna reset the master branch to my current dev branch soon as I've
> > tested it a bit more, but I'd suggest the bcache-3.2 branch - that's the
> > best tested code.
> 
> I cannot use a 3.2 kernel. This an HTPC too, besides a NAS and host for
> vm guests. The tuners I have are supported from 3.4 onwards.
> 
> Do you mean there might be more problems with the master branch?

Perhaps, but mainly it tracks mainline pretty closely. 

> By the way. I have read in the manual how to add new cache devices, but
> can you remove completely the caches in a 'clean way'? Imagine to change
> it to a different disk which for some reason you cannot use at the same time.

Yep. echo 1 > /sys/fs/bcache/<cache set>/unregister