ext4_mb_generate_buddy remounting read-only on domU

ext4_mb_generate_buddy remounting read-only on domU

[Sergi Seira]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

has anyone had ext4 filesystem remounting hours after a live-migration?

We've been having those 'read only remounts' always hours after a live-migration.
The more load a domU has the faster we get 'read only remount'.
We are suspecting something is getting corrupted after a live migration, and under load an error bumps:

 kernel: [3424760.190005] EXT4-fs error (device xvda3): ext4_mb_generate_buddy: EXT4-fs: group 55: 6102 blocks in bitmap, 6101 in gd
 kernel: [3424760.190059] Aborting journal on device xvda3-8.
 kernel: [3424760.191197] EXT4-fs (xvda3): Remounting filesystem read-only
 kernel: [3424760.204396] EXT4-fs (xvda3): delayed block allocation failed for inode 374665 at logical offset 318 with max blocks 1 with error -30
 kernel: [3424760.204447] This should not happen!!  Data will be lost
 kernel: [3424760.204483] EXT4-fs (xvda3): ext4_da_writepages: jbd2_start: 1015 pages, ino 374665; err -30

We have to fsck the volume from dom0 before re-creating domU.

Our system runs on :

 dom0 debian6 2.6.32-5-xen-amd64
 xen-hypervisor 4.0.1-2

DomU are:

 domU paravirtual debian6 2.6.32-5-xen-amd64
 3 iscsi volumes act as vxda's, swap, root and home (400G)
 7 vcpus
 8 GB RAM

Regards,
Sergi
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJO1f+8AAoJEN00VGSIizZ0zVIH/0KLyFgLAwJNmtXVbHo0VcsY
IicYbcv5CJNkJZ/WDskPK/33HsWH98epWzsRbr33qAibZrKfJig/1HuHs8SwGwGe
Xt1ii9+gnhy/oyNSnPbgmDpD6W81iEPkCTLm0fdHmndP9E5dbIWOzCMcIMvFB0RD
+2bPKQ3CcKEqnbXcriYzKVdfU2Y4/UbgZJYCQBJizK20A9KbqwAyLve/zRxJ7WP8
Q7+zWXkxefwcVq9rcBfkITvlrLQw9b9kK1KT7+7MrnxkG0ujIFYKVqkBxuIb4C3u
kILnz8Gh6vuiKhHy+5DKqEpGmhSsq/VVymLQCNerveluer2TF9QOTisv2kXw58Y=
=+hQj
-----END PGP SIGNATURE-----

Re: ext4_mb_generate_buddy remounting read-only on domU

[Konrad Rzeszutek Wilk]

On Wed, Nov 30, 2011 at 11:04:44AM +0100, Sergi Seira wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Hi,
> 
> has anyone had ext4 filesystem remounting hours after a live-migration?
> 
> We've been having those 'read only remounts' always hours after a live-migration.
> The more load a domU has the faster we get 'read only remount'.
> We are suspecting something is getting corrupted after a live migration, and under load an error bumps:
> 
>  kernel: [3424760.190005] EXT4-fs error (device xvda3): ext4_mb_generate_buddy: EXT4-fs: group 55: 6102 blocks in bitmap, 6101 in gd
>  kernel: [3424760.190059] Aborting journal on device xvda3-8.
>  kernel: [3424760.191197] EXT4-fs (xvda3): Remounting filesystem read-only
>  kernel: [3424760.204396] EXT4-fs (xvda3): delayed block allocation failed for inode 374665 at logical offset 318 with max blocks 1 with error -30
>  kernel: [3424760.204447] This should not happen!!  Data will be lost
>  kernel: [3424760.204483] EXT4-fs (xvda3): ext4_da_writepages: jbd2_start: 1015 pages, ino 374665; err -30
> 
> We have to fsck the volume from dom0 before re-creating domU.
> 
> Our system runs on :
> 
>  dom0 debian6 2.6.32-5-xen-amd64

This looks familiar,
http://xen.1045712.n5.nabble.com/ext4-BUG-in-dom0-Kernel-2-6-32-36-td4773487.html

Ah no. There was some fix to blktap for this I think. What exactly is
the drivery you are using to drive the root/home/swap of the guest? Is
it blkback or blktap?


>  xen-hypervisor 4.0.1-2
> 
> DomU are:
> 
>  domU paravirtual debian6 2.6.32-5-xen-amd64
>  3 iscsi volumes act as vxda's, swap, root and home (400G)
>  7 vcpus
>  8 GB RAM
> 
> Regards,
> Sergi
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.11 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
> 
> iQEcBAEBAgAGBQJO1f+8AAoJEN00VGSIizZ0zVIH/0KLyFgLAwJNmtXVbHo0VcsY
> IicYbcv5CJNkJZ/WDskPK/33HsWH98epWzsRbr33qAibZrKfJig/1HuHs8SwGwGe
> Xt1ii9+gnhy/oyNSnPbgmDpD6W81iEPkCTLm0fdHmndP9E5dbIWOzCMcIMvFB0RD
> +2bPKQ3CcKEqnbXcriYzKVdfU2Y4/UbgZJYCQBJizK20A9KbqwAyLve/zRxJ7WP8
> Q7+zWXkxefwcVq9rcBfkITvlrLQw9b9kK1KT7+7MrnxkG0ujIFYKVqkBxuIb4C3u
> kILnz8Gh6vuiKhHy+5DKqEpGmhSsq/VVymLQCNerveluer2TF9QOTisv2kXw58Y=
> =+hQj
> -----END PGP SIGNATURE-----
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel