scrub backing device

scrub backing device

[ching lu]

Hello,

i want to develop a script to perform data scrubbing periodically.

For my use case, i think it is meaningless to scrub cached data.

If i change the cache mode to "none", will it turn off the read cache too?

Furthermore, will this invalidate cached data? (i do not want to "warm
up" the cache again after the scrubbing)

Thanks in advance,
c

Re: scrub backing device

[Eric Wheeler]

On Fri, 29 Apr 2016, ching lu wrote:

> Hello,
> 
> i want to develop a script to perform data scrubbing periodically.

I think you want to set dirty_percent to 0 and cache_mode=writethrough.  
Then wait a long time for dirty_percent to actually reach 0.

--
Eric Wheeler


> 
> For my use case, i think it is meaningless to scrub cached data.
> 
> If i change the cache mode to "none", will it turn off the read cache too?
> 
> Furthermore, will this invalidate cached data? (i do not want to "warm
> up" the cache again after the scrubbing)
> 
> Thanks in advance,
> c
> --
> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: scrub backing device

[ching lu]

The read cache prevent the program to scrub the backing device directly.

if a few bits flipped in the backing device, the cache may still
return healthy data for a while.

Must i remove cache device before scrubbing?


On Wed, May 11, 2016 at 9:15 AM, Eric Wheeler <bcache@lists.ewheeler.net> wrote:
> On Fri, 29 Apr 2016, ching lu wrote:
>
>> Hello,
>>
>> i want to develop a script to perform data scrubbing periodically.
>
> I think you want to set dirty_percent to 0 and cache_mode=writethrough.
> Then wait a long time for dirty_percent to actually reach 0.
>
> --
> Eric Wheeler
>
>
>>
>> For my use case, i think it is meaningless to scrub cached data.
>>
>> If i change the cache mode to "none", will it turn off the read cache too?
>>
>> Furthermore, will this invalidate cached data? (i do not want to "warm
>> up" the cache again after the scrubbing)
>>
>> Thanks in advance,
>> c
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>

Re: scrub backing device

[Eric Wheeler]

On Wed, 11 May 2016, ching lu wrote:

> The read cache prevent the program to scrub the backing device directly.
> 
> if a few bits flipped in the backing device, the cache may still
> return healthy data for a while.
> 
> Must i remove cache device before scrubbing?

What do you mean by scrub?

--
Eric Wheeler

> 
> 
> On Wed, May 11, 2016 at 9:15 AM, Eric Wheeler <bcache@lists.ewheeler.net> wrote:
> > On Fri, 29 Apr 2016, ching lu wrote:
> >
> >> Hello,
> >>
> >> i want to develop a script to perform data scrubbing periodically.
> >
> > I think you want to set dirty_percent to 0 and cache_mode=writethrough.
> > Then wait a long time for dirty_percent to actually reach 0.
> >
> > --
> > Eric Wheeler
> >
> >
> >>
> >> For my use case, i think it is meaningless to scrub cached data.
> >>
> >> If i change the cache mode to "none", will it turn off the read cache too?
> >>
> >> Furthermore, will this invalidate cached data? (i do not want to "warm
> >> up" the cache again after the scrubbing)
> >>
> >> Thanks in advance,
> >> c
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: scrub backing device

[ching]

for example https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs-scrub

On May 12, 2016 3:13:22 AM GMT+08:00, Eric Wheeler <bcache@lists.ewheeler.net> wrote:
>On Wed, 11 May 2016, ching lu wrote:
>
>> The read cache prevent the program to scrub the backing device
>directly.
>> 
>> if a few bits flipped in the backing device, the cache may still
>> return healthy data for a while.
>> 
>> Must i remove cache device before scrubbing?
>
>What do you mean by scrub?
>
>--
>Eric Wheeler
>
>> 
>> 
>> On Wed, May 11, 2016 at 9:15 AM, Eric Wheeler
><bcache@lists.ewheeler.net> wrote:
>> > On Fri, 29 Apr 2016, ching lu wrote:
>> >
>> >> Hello,
>> >>
>> >> i want to develop a script to perform data scrubbing periodically.
>> >
>> > I think you want to set dirty_percent to 0 and
>cache_mode=writethrough.
>> > Then wait a long time for dirty_percent to actually reach 0.
>> >
>> > --
>> > Eric Wheeler
>> >
>> >
>> >>
>> >> For my use case, i think it is meaningless to scrub cached data.
>> >>
>> >> If i change the cache mode to "none", will it turn off the read
>cache too?
>> >>
>> >> Furthermore, will this invalidate cached data? (i do not want to
>"warm
>> >> up" the cache again after the scrubbing)
>> >>
>> >> Thanks in advance,
>> >> c
>> >> --
>> >> To unsubscribe from this list: send the line "unsubscribe
>linux-bcache" in
>> >> the body of a message to majordomo@vger.kernel.org
>> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> >>
>> --
>> To unsubscribe from this list: send the line "unsubscribe
>linux-bcache" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> 

Re: scrub backing device

[Marc MERLIN]

On Thu, May 12, 2016 at 07:34:26AM +0800, ching wrote:
> for example https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs-scrub
 
btrfs scrub is designed to make sure all the blocks and checksums are
consistent.
Ching makes a good point that it would be helpful to have a way to
access the underlying backing device directly to ensure it is
consistent.

I think currently you can use dmsetup with an offset to get the real
filesystem past the bcache header, mount that read only somewhere and
scrub that.
Not ideal because scrub will not be able to fix issues that are fixable,
but at least it should report errors.

Would that work?

Marc

> On May 12, 2016 3:13:22 AM GMT+08:00, Eric Wheeler <bcache@lists.ewheeler.net> wrote:
> >On Wed, 11 May 2016, ching lu wrote:
> >
> >> The read cache prevent the program to scrub the backing device
> >directly.
> >> 
> >> if a few bits flipped in the backing device, the cache may still
> >> return healthy data for a while.
> >> 
> >> Must i remove cache device before scrubbing?
> >
> >What do you mean by scrub?
> >
> >--
> >Eric Wheeler
> >
> >> 
> >> 
> >> On Wed, May 11, 2016 at 9:15 AM, Eric Wheeler
> ><bcache@lists.ewheeler.net> wrote:
> >> > On Fri, 29 Apr 2016, ching lu wrote:
> >> >
> >> >> Hello,
> >> >>
> >> >> i want to develop a script to perform data scrubbing periodically.
> >> >
> >> > I think you want to set dirty_percent to 0 and
> >cache_mode=writethrough.
> >> > Then wait a long time for dirty_percent to actually reach 0.
> >> >
> >> > --
> >> > Eric Wheeler
> >> >
> >> >
> >> >>
> >> >> For my use case, i think it is meaningless to scrub cached data.
> >> >>
> >> >> If i change the cache mode to "none", will it turn off the read
> >cache too?
> >> >>
> >> >> Furthermore, will this invalidate cached data? (i do not want to
> >"warm
> >> >> up" the cache again after the scrubbing)
> >> >>
> >> >> Thanks in advance,
> >> >> c
> >> >> --
> >> >> To unsubscribe from this list: send the line "unsubscribe
> >linux-bcache" in
> >> >> the body of a message to majordomo@vger.kernel.org
> >> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >> >>
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe
> >linux-bcache" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: scrub backing device

[ching]

it may be a mess that if there is 2 btrfs partition (1 via bcache, 1 via dm) with same UUID appear at the same time.

https://btrfs.wiki.kernel.org/index.php/Gotchas

Currently, i think removing cache device would be simpler.


On May 12, 2016 7:57:48 AM GMT+08:00, Marc MERLIN <marc@merlins.org> wrote:
>On Thu, May 12, 2016 at 07:34:26AM +0800, ching wrote:
>> for example
>https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs-scrub
> 
>btrfs scrub is designed to make sure all the blocks and checksums are
>consistent.
>Ching makes a good point that it would be helpful to have a way to
>access the underlying backing device directly to ensure it is
>consistent.
>
>I think currently you can use dmsetup with an offset to get the real
>filesystem past the bcache header, mount that read only somewhere and
>scrub that.
>Not ideal because scrub will not be able to fix issues that are
>fixable,
>but at least it should report errors.
>
>Would that work?
>
>Marc
>
>> On May 12, 2016 3:13:22 AM GMT+08:00, Eric Wheeler
><bcache@lists.ewheeler.net> wrote:
>> >On Wed, 11 May 2016, ching lu wrote:
>> >
>> >> The read cache prevent the program to scrub the backing device
>> >directly.
>> >> 
>> >> if a few bits flipped in the backing device, the cache may still
>> >> return healthy data for a while.
>> >> 
>> >> Must i remove cache device before scrubbing?
>> >
>> >What do you mean by scrub?
>> >
>> >--
>> >Eric Wheeler
>> >
>> >> 
>> >> 
>> >> On Wed, May 11, 2016 at 9:15 AM, Eric Wheeler
>> ><bcache@lists.ewheeler.net> wrote:
>> >> > On Fri, 29 Apr 2016, ching lu wrote:
>> >> >
>> >> >> Hello,
>> >> >>
>> >> >> i want to develop a script to perform data scrubbing
>periodically.
>> >> >
>> >> > I think you want to set dirty_percent to 0 and
>> >cache_mode=writethrough.
>> >> > Then wait a long time for dirty_percent to actually reach 0.
>> >> >
>> >> > --
>> >> > Eric Wheeler
>> >> >
>> >> >
>> >> >>
>> >> >> For my use case, i think it is meaningless to scrub cached
>data.
>> >> >>
>> >> >> If i change the cache mode to "none", will it turn off the read
>> >cache too?
>> >> >>
>> >> >> Furthermore, will this invalidate cached data? (i do not want
>to
>> >"warm
>> >> >> up" the cache again after the scrubbing)
>> >> >>
>> >> >> Thanks in advance,
>> >> >> c
>> >> >> --
>> >> >> To unsubscribe from this list: send the line "unsubscribe
>> >linux-bcache" in
>> >> >> the body of a message to majordomo@vger.kernel.org
>> >> >> More majordomo info at 
>http://vger.kernel.org/majordomo-info.html
>> >> >>
>> >> --
>> >> To unsubscribe from this list: send the line "unsubscribe
>> >linux-bcache" in
>> >> the body of a message to majordomo@vger.kernel.org
>> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> >> 
>> 
>> --
>> To unsubscribe from this list: send the line "unsubscribe
>linux-bcache" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> 

Re: scrub backing device

[Marc MERLIN]

On Thu, May 12, 2016 at 06:25:27PM +0800, ching wrote:
> it may be a mess that if there is 2 btrfs partition (1 via bcache, 1 via dm) with same UUID appear at the same time.
> 
> https://btrfs.wiki.kernel.org/index.php/Gotchas
> 
> Currently, i think removing cache device would be simpler.

You make a good point, and I agree with your assessment.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: scrub backing device

[Kent Overstreet]

On Wed, May 11, 2016 at 04:57:48PM -0700, Marc MERLIN wrote:
> On Thu, May 12, 2016 at 07:34:26AM +0800, ching wrote:
> > for example https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs-scrub
>  
> btrfs scrub is designed to make sure all the blocks and checksums are
> consistent.
> Ching makes a good point that it would be helpful to have a way to
> access the underlying backing device directly to ensure it is
> consistent.
> 
> I think currently you can use dmsetup with an offset to get the real
> filesystem past the bcache header, mount that read only somewhere and
> scrub that.
> Not ideal because scrub will not be able to fix issues that are fixable,
> but at least it should report errors.

Look at the verify code in drivers/md/bcache/debug.c, there's a
CONFIG_BCACHE_DEBUG mode where every time it reads clean cached data it reads
the data from the backing device and compares.

But for a scrub though you really want to walk all the _cached_ data, so it'd
take a bit more code than that.

Re: scrub backing device

[ching]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

it seems to be a kernel config, and the kernel panic cannot be handled by userspace program

On May 14, 2016 6:21:33 AM GMT+08:00, Kent Overstreet <kent.overstreet@gmail.com> wrote:
>On Wed, May 11, 2016 at 04:57:48PM -0700, Marc MERLIN wrote:
>> On Thu, May 12, 2016 at 07:34:26AM +0800, ching wrote:
>> > for example
>https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs-scrub
>>
>> btrfs scrub is designed to make sure all the blocks and checksums are
>> consistent.
>> Ching makes a good point that it would be helpful to have a way to
>> access the underlying backing device directly to ensure it is
>> consistent.
>>
>> I think currently you can use dmsetup with an offset to get the real
>> filesystem past the bcache header, mount that read only somewhere and
>> scrub that.
>> Not ideal because scrub will not be able to fix issues that are
>fixable,
>> but at least it should report errors.
>
>Look at the verify code in drivers/md/bcache/debug.c, there's a
>CONFIG_BCACHE_DEBUG mode where every time it reads clean cached data it
>reads
>the data from the backing device and compares.
>
>But for a scrub though you really want to walk all the _cached_ data,
>so it'd
>take a bit more code than that.
-----BEGIN PGP SIGNATURE-----
Version: APG v1.1.1

iQQ6BAEBCgAkBQJXOQy8HRxjaGluZyA8dG9tLmx1QGFsdW1uaS51c3QuaGs+AAoJ
EJh7yMFRxEtkK/cgAN9XBFqqZ7mhctx8IzSKKZx5GrP4xv7Ew7rITiX4J6J2JLqn
5NZUW9rGReRXKj1MaVz0MGFeD/Lnndzf5gRLwD6cwQtPQAydTm6krd0qsld1V4O+
o9plFThE9IjdKRhWoIr6yA11wKN04MmIInucGtjtNITVBx+vut3MHAcDuRxeIG/Y
sGmTrA4VbI34agltybanTn7XfnKULFdXOpfmf35dyqyrgShk22+5atcOW4hug112
5sPOpBo2S1jAwVZLKorwh4gIqeII4pFl+sfo5axJ60qLAZNXHwWeOGidmpcXFJMG
/NnSsWRmbrJvMlSsRk/hKTHjvkzZ/Ci7ZU9aB+ICkZQgqk58udRLXc990RhKMpvK
mKv9SHhltVYls+zoaaITxM6dS5rlIUwk++OhOU0qjjCOu9TI4yGmeBM5YmA29srb
hOc5mtwWnjORcWAf7uttfNS1ECxJ0AlYJIRL9rF5FFCvxEugfxd1UQnZXz3qBj6q
KDJo5JARiJvx+ayAw8I3cuh49Gc1U7VktUbRn5y/9Pm+Lj483Eb2Mp5kDyyAZTG5
X2baUtmkH0SkNID62ZzpBT7LE29oaw+PQ02czODIRYgAgJfuuCTEiCq0Q6zvRCDr
rG6SO6K4W4CsgN3Hm5W5b832vRuGK/pfWj/DaNmK2XJ30XlckQSGXXV92MitHrs+
bF82odaF2qFHQQA8/W3MX1CcaPazyulvZfVMh6n6+K0cehOd3Gb94J/yQSdcRIM6
ynVu/7DJ+Ig9aH/2MvpzH/y6X1UZZEPvhV+CUyQPcXIXfhm5VMbLEKRq8gpZaq24
3+z/Tf2Vcbw4Wa4+GShGssavHljNr7Q2s4D8Nf6Mfn2fLb6tjIciAms//t+ywXnV
DbLx34csTWTRWADq/Izsg2qJT5nAJ82HsAoY0Zg16pJHVgqCvg8gQPUnMvd06VK6
VPr9jcFv6yC3nijC/y78hkCcj2KVU4sx02hk4nxPo7mbbj2dJtrzdMW0HhT4iXhW
ELIuNnDYbRX7OzkkggDO9092Y+oCuRPyJdkfVp/kxtpHVBaZzI8BPmJHVusEzanx
6IvS4QgPDTVfmYfvyxHqfRL8d5jc7uvws2uwnsM3hEBoOvTq/bAPSdoM/wibsU9f
LmxEs2y5tX+AEHsT62o2FT1hc6ilvHzuRdJzNY+MPIsIFTBCPMF0wJFeL77tYYWE
71SUNDrBL8zT/eqeeneqfn1rbgEuDxucKOUuwLz0oYkWYGk0bhYieoWBrqyZGuq8
CKg2KPUB+2KJuS9Y9wXfL93QohzQOLc8RzOwf2oqCio7uVKn+DNyyE0gF66k0WOV
HfSvyfdcSb83CnQxXwVtQbEz9jHyUFe0J5Y6CK4=
=aTFB
-----END PGP SIGNATURE-----