puzzling error: XenServer, LVM and XFS

puzzling error: XenServer, LVM and XFS

[Michael Monnerie]

[-- Attachment #1.1: Type: Text/Plain, Size: 1084 bytes --]

I have a problem I can recreate, but have no idea where to ask/report 
it. I have a XenServer 5.5 based Virtual Machine with a running openSUSE 
11.2. Everything running fine.

Now I add another disk (which is an LVM volume on the host), there there 
I create a partition. In this partition, I create an LVM volume:
pvcreate ..
vgcreate ..
lvcreate ..
So far no problem.

Now I create an XFS:
mkfs.xfs /dev/test/test
mount /dev/test/test /1

I can reproduce and get tons of these errors on 2 different XenServers 
in 2 different VMs:

May 27 08:32:11 zdb13 kernel: [  738.046895] end_request: I/O error, dev 
xvdc, sector 0

This only happens after XFS is mounted from the LVM volume. On another 
machine where active I/O is happening, I get tons of these errors every 
some seconds.

-- 
mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc

it-management Internet Services
http://proteger.at [gesprochen: Prot-e-schee]
Tel: 0660 / 415 65 31

// Wir haben im Moment zwei Häuser zu verkaufen:
// http://zmi.at/langegg/
// http://zmi.at/haus2009/

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: puzzling error: XenServer, LVM and XFS

[Christoph Hellwig]

On Thu, May 27, 2010 at 08:57:38AM +0200, Michael Monnerie wrote:
> May 27 08:32:11 zdb13 kernel: [  738.046895] end_request: I/O error, dev 
> xvdc, sector 0
> 
> This only happens after XFS is mounted from the LVM volume. On another 
> machine where active I/O is happening, I get tons of these errors every 
> some seconds.

This looks like the Xen block device code is buggy and doesn't handle
the I/O pattern XFS throws at it.  A sector 0 error during mount usually
is the superblock barrier test write.  To verify that assumptions
try mounting the XFS filesystems with -o nobarrier and see if it goes
away.  Note that this is only for testing, and I would _not_ recommend
it for production use.  If it makes the error go away please file
a bug with the Xen guys to fix their barrier support by either
implementing it properly or not advertizing it.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: puzzling error: XenServer, LVM and XFS

[Michael Monnerie]

[-- Attachment #1.1: Type: Text/Plain, Size: 1224 bytes --]

On Donnerstag, 27. Mai 2010 Christoph Hellwig wrote:
> This looks like the Xen block device code is buggy and doesn't handle
> the I/O pattern XFS throws at it.  A sector 0 error during mount
>  usually is the superblock barrier test write.  To verify that
>  assumptions try mounting the XFS filesystems with -o nobarrier and
>  see if it goes away.  Note that this is only for testing, and I
>  would not recommend it for production use.  If it makes the error go
>  away please file a bug with the Xen guys to fix their barrier
>  support by either implementing it properly or not advertizing it.
 
You're right! As this does not happen with normal partitions but only 
LVM volumes, there must be some problem in the combination of XenServer, 
LVM and XFS.
There is nothing I can do to report this to Citrix, at least I didn't 
find a bug report facility for that, so it won't be fixed but now we 
know it.
This was XenServer 5.5

-- 
mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc

it-management Internet Services
http://proteger.at [gesprochen: Prot-e-schee]
Tel: 0660 / 415 65 31

// Wir haben im Moment zwei Häuser zu verkaufen:
// http://zmi.at/langegg/
// http://zmi.at/haus2009/

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: puzzling error: XenServer, LVM and XFS

[Michael Monnerie]

[-- Attachment #1.1: Type: Text/Plain, Size: 33078 bytes --]

On Donnerstag, 27. Mai 2010 Christoph Hellwig wrote:
> This looks like the Xen block device code is buggy and doesn't handle
> the I/O pattern XFS throws at it.  A sector 0 error during mount
>  usually is the superblock barrier test write.

One strange thing I want to mention: This doesn't only happen during 
mount, but also during runtime:

[272445.900691] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272445.900959] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272445.900966] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272445.901149] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272445.906411] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272445.906693] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272445.906700] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272445.906885] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272445.924679] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272445.924948] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272445.924956] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272445.925111] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272447.543881] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272447.544198] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272447.544205] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272447.544354] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272447.548822] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272447.549069] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272447.549077] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272447.549226] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272449.264465] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272449.264808] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272449.264816] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272449.265007] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272449.270708] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272449.270950] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272449.270958] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272449.271107] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272450.481089] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272450.481360] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272450.481368] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272450.481646] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272450.484787] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272450.485069] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272450.485077] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272450.485220] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272450.504376] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272450.504633] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272450.504642] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272450.504790] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272450.505045] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272450.505274] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272450.505281] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272450.505425] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272450.512789] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272450.513154] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272450.513162] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272450.513309] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272450.530959] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272450.531352] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272450.531361] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272450.531543] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272452.173596] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272452.173887] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272452.173896] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272452.174055] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272452.175007] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272452.175268] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272452.175275] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272452.175443] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272452.175815] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272452.176056] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272452.176063] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272452.176221] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272452.176793] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272452.177010] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272452.177016] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272452.177198] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272452.559967] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272452.560240] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272452.560247] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272452.560420] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272452.565146] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272452.565408] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272452.565415] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272452.565581] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272454.296089] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272454.296357] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272454.296365] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272454.296523] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272454.301814] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272454.302083] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272454.302091] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272454.302277] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272457.575577] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272457.575824] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272457.575832] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272457.575974] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272457.576165] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272457.576403] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272457.576410] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272457.576552] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272457.580690] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272457.580981] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272457.580989] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272457.581186] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272459.343053] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272459.343330] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272459.343338] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272459.343521] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272459.343750] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272459.344086] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272459.344093] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272459.344236] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272459.348876] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272459.349152] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272459.349159] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272459.349357] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272462.217232] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272462.217530] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272462.217539] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272462.217719] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272462.218443] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272462.218712] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272462.218720] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272462.218887] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272462.219677] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272462.219943] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272462.219982] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272462.220138] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272462.592004] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272462.592285] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272462.592293] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272462.592486] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272462.597265] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272462.597531] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272462.597539] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272462.597706] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272464.375703] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272464.375992] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272464.376000] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272464.376157] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272464.381090] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272464.381385] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272464.381392] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272464.381585] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272467.608382] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272467.608671] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272467.608678] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272467.608862] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         
[272467.613789] end_request: I/O error, dev xvdb, sector 0                                                                                                                                                                         

Does XFS always recheck for barriers? Or is it because I specified 
"barrier" as a mount option, but it doesn't work and XFS didn't detect 
it doesn't work?

-- 
mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc

it-management Internet Services
http://proteger.at [gesprochen: Prot-e-schee]
Tel: 0660 / 415 65 31

// Wir haben im Moment zwei Häuser zu verkaufen:
// http://zmi.at/langegg/
// http://zmi.at/haus2009/

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs