blkid: excessive random reads probing for ZFS on NTFS filesystem

blkid: excessive random reads probing for ZFS on NTFS filesystem

[John Lindgren]

Hi,

I noticed blkid causing a noticeable amount of disk hits while
identifying an NTFS (Win7) partition, and did an strace to find out why
this was.  It seems that the probe_zfs() function does a
seek-and-read-41-bytes pattern 64 times in the 130 to 520 KB region of
the partition.  This of course is slow on a traditional hard disk, and
seems excessive.  I don't know the internals of NTFS or of ZFS.  Can
someone with a little more knowledge in this area provide some pointers
as to why this might be happening?

-- John Lindgren

Re: blkid: excessive random reads probing for ZFS on NTFS filesystem

[Karel Zak]

On Sun, Jun 12, 2011 at 03:02:48PM -0400, John Lindgren wrote:
> I noticed blkid causing a noticeable amount of disk hits while
> identifying an NTFS (Win7) partition, and did an strace to find out why
> this was.  It seems that the probe_zfs() function does a
> seek-and-read-41-bytes pattern 64 times in the 130 to 520 KB region of
> the partition.

 Is it really so big problem on traditional HDDs?

> This of course is slow on a traditional hard disk, and
> seems excessive.  I don't know the internals of NTFS or of ZFS.  Can
> someone with a little more knowledge in this area provide some pointers
> as to why this might be happening?

 CC: Andreas who is the code (co)author.

    Karel

-- 
 Karel Zak  <kzak@redhat.com>
 http://karelzak.blogspot.com

Re: blkid: excessive random reads probing for ZFS on NTFS filesystem

[John Lindgren]

On 06/20/2011 07:16 AM, Karel Zak wrote:
> On Sun, Jun 12, 2011 at 03:02:48PM -0400, John Lindgren wrote:
>> I noticed blkid causing a noticeable amount of disk hits while
>> identifying an NTFS (Win7) partition, and did an strace to find out why
>> this was.  It seems that the probe_zfs() function does a
>> seek-and-read-41-bytes pattern 64 times in the 130 to 520 KB region of
>> the partition.
> Is it really so big problem on traditional HDDs?
Many traditional HD's are rated with a seek time of ~10 ms (though I
work with older hardware mostly; newer ones may be better).  So by a
naive calculation that would make probe_zfs take 0.64 seconds.  Now, in
the case of probe_zfs, the seeks have good locality, so in practice it
will not take that long -- provided blkid is the only program accessing
the disk at that time.  If the system has some sort of read-ahead
process running during boot (as I do), then locality is lost for both
blkid and the read-ahead process, and you get thrashing.

Also take into consideration that while 64 seeks may not seem excessive
in itself, that is for only one filesystem, and blkid checks something
on the order of 40 different filesystems.  (Fortunately none of the
others take anywhere near the number of seeks that zfs_probe does.)

Andreas: As far as implementation, I think the ZFS code could be
improved by parsing the name-value pair list *before* searching for a
valid uberblock, rather than after.  I don't have any ZFS filesystems
around to test with, or I would try to do this myself.

-- John

Re: blkid: excessive random reads probing for ZFS on NTFS filesystem

[Karel Zak]

On Mon, Jun 20, 2011 at 07:59:15AM -0400, John Lindgren wrote:
> On 06/20/2011 07:16 AM, Karel Zak wrote:
> > On Sun, Jun 12, 2011 at 03:02:48PM -0400, John Lindgren wrote:
> >> I noticed blkid causing a noticeable amount of disk hits while
> >> identifying an NTFS (Win7) partition, and did an strace to find out why
> >> this was.  It seems that the probe_zfs() function does a
> >> seek-and-read-41-bytes pattern 64 times in the 130 to 520 KB region of
> >> the partition.
> > Is it really so big problem on traditional HDDs?
> Many traditional HD's are rated with a seek time of ~10 ms (though I
> work with older hardware mostly; newer ones may be better).  So by a
> naive calculation that would make probe_zfs take 0.64 seconds.  Now, in
> the case of probe_zfs, the seeks have good locality, so in practice it
> will not take that long -- provided blkid is the only program accessing
> the disk at that time.  If the system has some sort of read-ahead
> process running during boot (as I do), then locality is lost for both
> blkid and the read-ahead process, and you get thrashing.
>
> Also take into consideration that while 64 seeks may not seem excessive
> in itself, that is for only one filesystem, and blkid checks something
> on the order of 40 different filesystems.  (Fortunately none of the
> others take anywhere near the number of seeks that zfs_probe does.)

 Unfortunately, the old less expensive version of the ZFS detection
 code was insufficient.  See Andreas' patches:

 commit e54a76ca076625b1883ddf0595162eb8de81d5d1
 Author: Andreas Dilger <adilger@sun.com>
 Date:   Wed Feb 17 10:21:27 2010 +0100

 commit a1fbeb3df35d1441c4ef64ea7e04c2b1fda38ba2
 Author: Andreas Dilger <adilger@sun.com>
 Date:   Thu Mar 11 15:16:46 2010 +0100


> Andreas: As far as implementation, I think the ZFS code could be
> improved by parsing the name-value pair list *before* searching for a
> valid uberblock, rather than after.

 Doesn't depend the position of the list on the position of the valid
 uberblock?

> I don't have any ZFS filesystems
> around to test with, or I would try to do this myself.

 There is a test image, see tests/ts/blkid/images-fs/zfs.img.bz2

 Maybe you can try to move zfs to the end of the probed filesystems 
 in libblkid/src/superblocks/superblocks.c. It would be better to
 probe for exotic and "expensive" filesytems later.

    Karel

-- 
 Karel Zak  <kzak@redhat.com>
 http://karelzak.blogspot.com

Re: blkid: excessive random reads probing for ZFS on NTFS filesystem

[John Lindgren]

On 06/20/2011 08:46 AM, Karel Zak wrote:
>  Unfortunately, the old less expensive version of the ZFS detection
>  code was insufficient.  See Andreas' patches:
>
>  commit e54a76ca076625b1883ddf0595162eb8de81d5d1
>  Author: Andreas Dilger <adilger@sun.com>
>  Date:   Wed Feb 17 10:21:27 2010 +0100
>
>  commit a1fbeb3df35d1441c4ef64ea7e04c2b1fda38ba2
>  Author: Andreas Dilger <adilger@sun.com>
>  Date:   Thu Mar 11 15:16:46 2010 +0100

Understood.  The older implementation performed a less thorough search
for a valid uberblock, which I would expect to sometimes fail.

>> Andreas: As far as implementation, I think the ZFS code could be
>> improved by parsing the name-value pair list *before* searching for a
>> valid uberblock, rather than after.
> Doesn't depend the position of the list on the position of the valid
> uberblock?

No, it's at a fixed offset of 16k within the vdev label (before the
uberblock array, which is at 128k).

>> I don't have any ZFS filesystems
>> around to test with, or I would try to do this myself.
> There is a test image, see tests/ts/blkid/images-fs/zfs.img.bz2

Okay, thanks.

>  Maybe you can try to move zfs to the end of the probed filesystems 
>  in libblkid/src/superblocks/superblocks.c. It would be better to
>  probe for exotic and "expensive" filesytems later.

I did try that; however, blkid checks *all* filesystems, even after it
finds a match, just in case two probes turn up positive.  Even passing
"-n nozfs" doesn't actually prevent it from checking for ZFS.

Maybe it would be better to change this behavior and leave the ZFS code
as is?

-- John

Re: blkid: excessive random reads probing for ZFS on NTFS filesystem

[Karel Zak]

On Mon, Jun 20, 2011 at 09:05:36AM -0400, John Lindgren wrote:
> I did try that; however, blkid checks *all* filesystems, even after it
> finds a match, just in case two probes turn up positive.  Even passing
> "-n nozfs" doesn't actually prevent it from checking for ZFS.

 The name of the FS in libblkid is zfs_member :-)

    blkid -p -o udev -n nozfs_member

 Yeah, crazy name, see our TODO file:

   - add something like "blkid --list-known" to list all supported
     filesystems/raids
     
 Karel

-- 
 Karel Zak  <kzak@redhat.com>
 http://karelzak.blogspot.com

Re: blkid: excessive random reads probing for ZFS on NTFS filesystem

[John Lindgren]

On 06/20/2011 10:04 AM, Karel Zak wrote:
> On Mon, Jun 20, 2011 at 09:05:36AM -0400, John Lindgren wrote:
>> I did try that; however, blkid checks *all* filesystems, even after it
>> finds a match, just in case two probes turn up positive.  Even passing
>> "-n nozfs" doesn't actually prevent it from checking for ZFS.
>  The name of the FS in libblkid is zfs_member :-)
>
>     blkid -p -o udev -n nozfs_member
>
>  Yeah, crazy name, see our TODO file:
>
>    - add something like "blkid --list-known" to list all supported
>      filesystems/raids
>      
>  Karel

My bad.  "-n nozfs_member" does work, so I will add it to my udev rules
locally.

Perhaps a "-q/--quick" option could be added to blkid to allow an early
exit once the first positive match is found?  Then more difficult checks
such as ZFS could be moved to the end of the list, as you suggest, with
a performance benefit.

-- John

Re: blkid: excessive random reads probing for ZFS on NTFS filesystem

[Karel Zak]

On Mon, Jun 20, 2011 at 10:20:02AM -0400, John Lindgren wrote:
> On 06/20/2011 10:04 AM, Karel Zak wrote:
> > On Mon, Jun 20, 2011 at 09:05:36AM -0400, John Lindgren wrote:
> >> I did try that; however, blkid checks *all* filesystems, even after it
> >> finds a match, just in case two probes turn up positive.  Even passing
> >> "-n nozfs" doesn't actually prevent it from checking for ZFS.
> >  The name of the FS in libblkid is zfs_member :-)
> >
> >     blkid -p -o udev -n nozfs_member
> >
> >  Yeah, crazy name, see our TODO file:
> >
> >    - add something like "blkid --list-known" to list all supported
> >      filesystems/raids

 Implemented, 
 
    blkid -k

 lists all known filesystems/RAIDs now.

> Perhaps a "-q/--quick" option could be added to blkid to allow an early
> exit once the first positive match is found? 

 That's dangerous, and many times rejected by udev upstream. The
 mainstream distributions rely on the automatic FS detection on many
 places. We are not able to control mkfs and partitioning tools, so we
 are not sure that the device has been properly wiped and there is
 really only one valid superblock. We already have bad experience with
 FAT, linux swap, luks, ... some people already lost data. 

> Then more difficult checks
> such as ZFS could be moved to the end of the list, as you suggest, with
> a performance benefit.

 What about to set to blkid (during boot) greater I/O priority than to
 read-ahead?

    Karel

-- 
 Karel Zak  <kzak@redhat.com>
 http://karelzak.blogspot.com

Re: blkid: excessive random reads probing for ZFS on NTFS filesystem

[John Lindgren]

On 06/20/2011 07:16 AM, Karel Zak wrote:
> CC: Andreas who is the code (co)author.
His sun.com address is not active any more (as you have probably noticed).

-- John