[PATCH resend] android-xfstests: fix finding FS_TYPE when userdata is on dm device

[PATCH resend] android-xfstests: fix finding FS_TYPE when userdata is on dm device

[Eric Biggers]

From: Eric Biggers <ebiggers@google.com>

If the userdata filesystem is on a device-mapper device, then shrinking
the partition underneath it breaks 'blkid' probing, which
android-setup-partitions now uses find the filesystem type.  Use blkid's
'-p' and '-S' options to do a low-level probe with explicitly specified
size, which still works in this case.

Signed-off-by: Eric Biggers <ebiggers@google.com>
---
 kvm-xfstests/test-appliance/android-setup-partitions | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/kvm-xfstests/test-appliance/android-setup-partitions b/kvm-xfstests/test-appliance/android-setup-partitions
index 8d38303..ddaa252 100755
--- a/kvm-xfstests/test-appliance/android-setup-partitions
+++ b/kvm-xfstests/test-appliance/android-setup-partitions
@@ -259,7 +259,8 @@ get_fs_size()
 # userdata partition.  That isn't necessary, since resizing the underlying
 # partition to a size smaller than the dm device just causes I/O requests to the
 # truncated region to fail, and normally there should be no I/O occurring beyond
-# the end of the filesystem.
+# the end of the filesystem.  Exception: this makes 'blkid' stop reporting
+# information about the device, unless blkid's -p and -S options are used.
 shrink_userdata_partition()
 {
     local fs_size part_size
@@ -380,7 +381,9 @@ else
 fi
 
 # Type of the userdata filesystem, e.g. ext4 or f2fs
-USERDATA_FS_TYPE=$(blkid -s TYPE -o value "$USERDATA_FS_DEV")
+USERDATA_FS_TYPE=$(blkid -s TYPE -o value \
+		   -p -S $(get_partition_size "$USERDATA_RAW_DEV") \
+		   "$USERDATA_FS_DEV")
 
 if ! all_partitions_present ; then
     # Free up as much space as we can, then create the partitions.
-- 
2.19.0.rc2.392.g5ba43deb5a-goog

Re: [PATCH resend] android-xfstests: fix finding FS_TYPE when userdata is on dm device

[Theodore Y. Ts'o]

On Wed, Sep 05, 2018 at 12:07:00PM -0700, Eric Biggers wrote:
> From: Eric Biggers <ebiggers@google.com>
> 
> If the userdata filesystem is on a device-mapper device, then shrinking
> the partition underneath it breaks 'blkid' probing, which
> android-setup-partitions now uses find the filesystem type.  Use blkid's
> '-p' and '-S' options to do a low-level probe with explicitly specified
> size, which still works in this case.
> 
> Signed-off-by: Eric Biggers <ebiggers@google.com>

Thanks, applied.

					- Ted

Re: [PATCH resend] android-xfstests: fix finding FS_TYPE when userdata is on dm device

[Eric Biggers]

On Sat, Sep 15, 2018 at 03:36:57PM -0400, Theodore Y. Ts'o wrote:
> On Wed, Sep 05, 2018 at 12:07:00PM -0700, Eric Biggers wrote:
> > From: Eric Biggers <ebiggers@google.com>
> > 
> > If the userdata filesystem is on a device-mapper device, then shrinking
> > the partition underneath it breaks 'blkid' probing, which
> > android-setup-partitions now uses find the filesystem type.  Use blkid's
> > '-p' and '-S' options to do a low-level probe with explicitly specified
> > size, which still works in this case.
> > 
> > Signed-off-by: Eric Biggers <ebiggers@google.com>
> 
> Thanks, applied.
> 
> 					- Ted

Hi Ted, are you planning to push this out?

- Eric

Re: [PATCH resend] android-xfstests: fix finding FS_TYPE when userdata is on dm device

[Theodore Y. Ts'o]

On Wed, Oct 03, 2018 at 04:37:18PM -0700, Eric Biggers wrote:
> 
> Hi Ted, are you planning to push this out?
>

Oops, sorry.   Pushed out now.

				- Ted

Re: [PATCH resend] android-xfstests: fix finding FS_TYPE when userdata is on dm device

[Amir Goldstein]

On Thu, Oct 4, 2018 at 5:35 AM Theodore Y. Ts'o <tytso@mit.edu> wrote:
>
> On Wed, Oct 03, 2018 at 04:37:18PM -0700, Eric Biggers wrote:
> >
> > Hi Ted, are you planning to push this out?
> >
>
> Oops, sorry.   Pushed out now.
>

Ted,

FYI, this is pushed to github, but not kernel.org tree.
Intentional?

FYI2, I started trying to build a kvm-xfstests image with Debian buster.
First thing I ran into with setup-buildchroot is that golang-1.8-go
package is not available.
I suppose golang is for syzbot? Their wiki recommends to use
golang-go for debian based releases, which does work for buster.

Now I hit an error in ./gen-image:
W: Couldn't download package openssl (ver 1.1.0f-3+deb9u1 arch amd64) at
http://mirrors.kernel.org/debian/pool/main/o/openssl/openssl_1.1.0f-3+deb9u1_amd64.deb
I see that this pkg doesn't exist in the mirror, but this pkg does
exists (deb9u2):
http://mirrors.edge.kernel.org/debian/pool/main/o/openssl/openssl_1.1.0f-3+deb9u2_amd64.deb

Do you know what that is about?

Thanks,
Amir.

Re: [PATCH resend] android-xfstests: fix finding FS_TYPE when userdata is on dm device

[Theodore Y. Ts'o]

On Thu, Oct 11, 2018 at 03:01:36PM +0300, Amir Goldstein wrote:
> 
> FYI, this is pushed to github, but not kernel.org tree.
> Intentional?

Nope, just another oversight; fixed now.

> FYI2, I started trying to build a kvm-xfstests image with Debian buster.
> First thing I ran into with setup-buildchroot is that golang-1.8-go
> package is not available.
> I suppose golang is for syzbot? Their wiki recommends to use
> golang-go for debian based releases, which does work for buster.

At the time syzbot required golang 1.8.  I just checked, and according
to their docs[1] syzbot now requires golang 1.9.  I haven't been
bothering quite as much with syzkaller lately because there are now C
reproducers, and the syzkaller reproducers are not stable --- that is,
if you don't use *precisely* the same version of syzkaller, down to
the commit id, the syzkaller repro might not work at all.  I've tried
talking to Dmitry about providing at least *backwards* compatibility
(e.g., that a newer syzkaller can use a repro produced by an older
syzbot), but he's not willing to provide even that level of
compatibility.

[1] https://github.com/google/syzkaller/blob/master/docs/linux/setup.md

So building syzkaller is currently optional, and the only time I would
recommend it is that there apparently some cases where there is a
syzkaller reproducer, but no C reproducer, which is very said given
the complete lack of backwards compatibility.

I suppose we could include git, the syzkaller git tree and the go
compiler into the test VM, and build the appropriate version syzkaller
for each repro --- but it would really bloat the kvm-xfstests, and
that information isn't even encoded into the syzkaller repro file.  :-(

So it's all rather a mess at this point.  Maybe this is something we
could do just for the gce-xfstests test appliance.  Patches
appreciated.  :-)

On your other note, I haven't bothered using setup-buildchroot since
my laptop environment is buster.  Making setup-buildchroot install the
newest possible version of go available for buster or stretch sounds
like the best approach for now.


> Now I hit an error in ./gen-image:
> W: Couldn't download package openssl (ver 1.1.0f-3+deb9u1 arch amd64) at
> http://mirrors.kernel.org/debian/pool/main/o/openssl/openssl_1.1.0f-3+deb9u1_amd64.deb
> I see that this pkg doesn't exist in the mirror, but this pkg does
> exists (deb9u2):
> http://mirrors.edge.kernel.org/debian/pool/main/o/openssl/openssl_1.1.0f-3+deb9u2_amd64.deb
> 
> Do you know what that is about?

That's generally due to the mirror not being fully synced up so the
Debian distro metadata files weren't synced up with the debian package
files.  If you try again in 30 minutes or so, it should resolve
itself.

						- Ted

Re: [PATCH resend] android-xfstests: fix finding FS_TYPE when userdata is on dm device

[Amir Goldstein]

On Thu, Oct 11, 2018 at 5:56 PM Theodore Y. Ts'o <tytso@mit.edu> wrote:
>
> On Thu, Oct 11, 2018 at 03:01:36PM +0300, Amir Goldstein wrote:
> >
> > FYI, this is pushed to github, but not kernel.org tree.
> > Intentional?
>
> Nope, just another oversight; fixed now.
>
> > FYI2, I started trying to build a kvm-xfstests image with Debian buster.
> > First thing I ran into with setup-buildchroot is that golang-1.8-go
> > package is not available.
> > I suppose golang is for syzbot? Their wiki recommends to use
> > golang-go for debian based releases, which does work for buster.
>
> At the time syzbot required golang 1.8.  I just checked, and according
> to their docs[1] syzbot now requires golang 1.9.  I haven't been
> bothering quite as much with syzkaller lately because there are now C
> reproducers, and the syzkaller reproducers are not stable --- that is,
> if you don't use *precisely* the same version of syzkaller, down to
> the commit id, the syzkaller repro might not work at all.  I've tried
> talking to Dmitry about providing at least *backwards* compatibility
> (e.g., that a newer syzkaller can use a repro produced by an older
> syzbot), but he's not willing to provide even that level of
> compatibility.
>
> [1] https://github.com/google/syzkaller/blob/master/docs/linux/setup.md
>
> So building syzkaller is currently optional, and the only time I would
> recommend it is that there apparently some cases where there is a
> syzkaller reproducer, but no C reproducer, which is very said given
> the complete lack of backwards compatibility.
>
> I suppose we could include git, the syzkaller git tree and the go
> compiler into the test VM, and build the appropriate version syzkaller
> for each repro --- but it would really bloat the kvm-xfstests, and
> that information isn't even encoded into the syzkaller repro file.  :-(
>

That's a shame.

> So it's all rather a mess at this point.  Maybe this is something we
> could do just for the gce-xfstests test appliance.  Patches
> appreciated.  :-)
>
> On your other note, I haven't bothered using setup-buildchroot since
> my laptop environment is buster.  Making setup-buildchroot install the
> newest possible version of go available for buster or stretch sounds
> like the best approach for now.
>

That's what I did. will post one liner once everything is working.

>
> > Now I hit an error in ./gen-image:
> > W: Couldn't download package openssl (ver 1.1.0f-3+deb9u1 arch amd64) at
> > http://mirrors.kernel.org/debian/pool/main/o/openssl/openssl_1.1.0f-3+deb9u1_amd64.deb
> > I see that this pkg doesn't exist in the mirror, but this pkg does
> > exists (deb9u2):
> > http://mirrors.edge.kernel.org/debian/pool/main/o/openssl/openssl_1.1.0f-3+deb9u2_amd64.deb
> >
> > Do you know what that is about?
>
> That's generally due to the mirror not being fully synced up so the
> Debian distro metadata files weren't synced up with the debian package
> files.  If you try again in 30 minutes or so, it should resolve
> itself.
>

Note that the 2 packages I referred to above are different.
One is +deb9u1 (missing) the other is +deb9u2 (available)
I see that some packages downloaded have the +deb9u1 suffix
some have the +deb9u2 suffix and some no suffix at all.
I know absolutely zero about that distro black magic, so if you have
a lead for me, please let me know.

Thanks,
Amir.