[Qemu-devel] Heads up: glibc preadv emulation breaks qemu on older kernels

[Qemu-devel] Heads up: glibc preadv emulation breaks qemu on older kernels

[Richard W.M. Jones]

This isn't a problem with qemu as such, but it may affect you if you
use qemu with a newer glibc on a Linux kernel which does not support
the preadv(2) syscall natively.  glibc will attempt to emulate
preadv(2) using pread(2) syscalls, but it doesn't align the user
buffer when it does this, and that can break things (if using
cache=off then the file is opened with O_DIRECT which has strict
alignment requirements).

The user visible effect of this bug is that blocks devices (of all
types) inside the guest throw large numbers of I/O errors.

  https://bugzilla.redhat.com/show_bug.cgi?id=563103#c8

One interesting thing is that qemu has its own preadv emulation (which
does the emulation correctly), but this is never used because qemu
never gets ENOSYS back from preadv.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming blog: http://rwmj.wordpress.com
Fedora now supports 80 OCaml packages (the OPEN alternative to F#)
http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora

Re: [Qemu-devel] Heads up: glibc preadv emulation breaks qemu on older kernels

[Christoph Hellwig]

On Thu, Feb 11, 2010 at 07:06:08PM +0000, Richard W.M. Jones wrote:
> One interesting thing is that qemu has its own preadv emulation (which
> does the emulation correctly), but this is never used because qemu
> never gets ENOSYS back from preadv.

At this point the amount of bugs in the glibc preadv/pwritev code really
make me want to go to use the raw system calls on Linux only.  Any
opinions from the maintainers if that is acceptable?

Re: [Qemu-devel] Heads up: glibc preadv emulation breaks qemu on older kernels

[malc]

On Fri, 12 Feb 2010, Christoph Hellwig wrote:

> On Thu, Feb 11, 2010 at 07:06:08PM +0000, Richard W.M. Jones wrote:
> > One interesting thing is that qemu has its own preadv emulation (which
> > does the emulation correctly), but this is never used because qemu
> > never gets ENOSYS back from preadv.
> 
> At this point the amount of bugs in the glibc preadv/pwritev code really
> make me want to go to use the raw system calls on Linux only.  Any
> opinions from the maintainers if that is acceptable?

There are more than one way to parse the first sentence, if it should be
read as:
  On linux and on linux only avoid using pread/write[v] and talk to the
  kernel directly.

Then i agree.

-- 
mailto:av1474@comtv.ru

Re: [Qemu-devel] Heads up: glibc preadv emulation breaks qemu on older kernels

[Christoph Hellwig]

On Fri, Feb 12, 2010 at 04:49:23PM +0300, malc wrote:
> On Fri, 12 Feb 2010, Christoph Hellwig wrote:
> 
> > On Thu, Feb 11, 2010 at 07:06:08PM +0000, Richard W.M. Jones wrote:
> > > One interesting thing is that qemu has its own preadv emulation (which
> > > does the emulation correctly), but this is never used because qemu
> > > never gets ENOSYS back from preadv.
> > 
> > At this point the amount of bugs in the glibc preadv/pwritev code really
> > make me want to go to use the raw system calls on Linux only.  Any
> > opinions from the maintainers if that is acceptable?
> 
> There are more than one way to parse the first sentence, if it should be
> read as:
>   On linux and on linux only avoid using pread/write[v] and talk to the
>   kernel directly.
> 
> Then i agree.

Yes.  The BSDs tend to not play stupid emulation games in the libc, so
changes of these kinds of messups to happen are far less.

Re: [Qemu-devel] Heads up: glibc preadv emulation breaks qemu on older kernels

[Anthony Liguori]

On 02/12/2010 07:50 AM, Christoph Hellwig wrote:
> On Fri, Feb 12, 2010 at 04:49:23PM +0300, malc wrote:
>    
>> On Fri, 12 Feb 2010, Christoph Hellwig wrote:
>>
>>      
>>> On Thu, Feb 11, 2010 at 07:06:08PM +0000, Richard W.M. Jones wrote:
>>>        
>>>> One interesting thing is that qemu has its own preadv emulation (which
>>>> does the emulation correctly), but this is never used because qemu
>>>> never gets ENOSYS back from preadv.
>>>>          
>>> At this point the amount of bugs in the glibc preadv/pwritev code really
>>> make me want to go to use the raw system calls on Linux only.  Any
>>> opinions from the maintainers if that is acceptable?
>>>        
>> There are more than one way to parse the first sentence, if it should be
>> read as:
>>    On linux and on linux only avoid using pread/write[v] and talk to the
>>    kernel directly.
>>
>> Then i agree.
>>      
> Yes.  The BSDs tend to not play stupid emulation games in the libc, so
> changes of these kinds of messups to happen are far less.
>    

In all fairness, I seem to recall there being a problem with the kernel 
implementation of preadv/pwritev too.

I think a configure option would be in order.  Forever avoiding glibc is 
probably a bit extreme.

Regards,

Anthony Liguori

>
>    

Re: [Qemu-devel] Heads up: glibc preadv emulation breaks qemu on older kernels

[Richard W.M. Jones]

On Fri, Feb 12, 2010 at 11:28:57AM -0600, Anthony Liguori wrote:
> I think a configure option would be in order.  Forever avoiding glibc is  
> probably a bit extreme.

I think a configure option won't work.  The issue here was that
the binary is being installed on a machine with an older
kernel.

Unfortunately it's pretty hard to test this at runtime too (preadv
doesn't return ENOSYS, and there isn't really a good way to test if
the preadv "syscall" is broken because the randomly aligned buffer
might be aligned OK).

I don't think there's a good solution for this at the qemu level.
However I have a working patch for glibc:

https://bugzilla.redhat.com/show_bug.cgi?id=563103#c13

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://et.redhat.com/~rjones/virt-df/

Re: [Qemu-devel] Heads up: glibc preadv emulation breaks qemu on older kernels

[Christoph Hellwig]

On Fri, Feb 12, 2010 at 06:47:55PM +0000, Richard W.M. Jones wrote:
> I don't think there's a good solution for this at the qemu level.
> However I have a working patch for glibc:
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=563103#c13

Any chance you could just persuade Uli to drop the optimization?  I
think users are generally better off to just do it themselves, as they
have to do that anyway for portable applications.

Re: [Qemu-devel] Heads up: glibc preadv emulation breaks qemu on older kernels

[Richard W.M. Jones]

On Mon, Feb 15, 2010 at 11:17:27PM +0100, Christoph Hellwig wrote:
> On Fri, Feb 12, 2010 at 06:47:55PM +0000, Richard W.M. Jones wrote:
> > I don't think there's a good solution for this at the qemu level.
> > However I have a working patch for glibc:
> > 
> > https://bugzilla.redhat.com/show_bug.cgi?id=563103#c13
> 
> Any chance you could just persuade Uli to drop the optimization?  I
> think users are generally better off to just do it themselves, as they
> have to do that anyway for portable applications.

I agree.  It seems like the sort of thing which is better in gnulib
than in glibc.  That way programs can choose to use gnulib for
portability, versus having to use it with glibc.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://et.redhat.com/~rjones/libguestfs/
See what it can do: http://et.redhat.com/~rjones/libguestfs/recipes.html

Re: [Qemu-devel] Heads up: glibc preadv emulation breaks qemu on older kernels

[Avi Kivity]

On 02/12/2010 07:28 PM, Anthony Liguori wrote:
> On 02/12/2010 07:50 AM, Christoph Hellwig wrote:
>> On Fri, Feb 12, 2010 at 04:49:23PM +0300, malc wrote:
>>> On Fri, 12 Feb 2010, Christoph Hellwig wrote:
>>>
>>>> On Thu, Feb 11, 2010 at 07:06:08PM +0000, Richard W.M. Jones wrote:
>>>>> One interesting thing is that qemu has its own preadv emulation 
>>>>> (which
>>>>> does the emulation correctly), but this is never used because qemu
>>>>> never gets ENOSYS back from preadv.
>>>> At this point the amount of bugs in the glibc preadv/pwritev code 
>>>> really
>>>> make me want to go to use the raw system calls on Linux only.  Any
>>>> opinions from the maintainers if that is acceptable?
>>> There are more than one way to parse the first sentence, if it 
>>> should be
>>> read as:
>>>    On linux and on linux only avoid using pread/write[v] and talk to 
>>> the
>>>    kernel directly.
>>>
>>> Then i agree.
>> Yes.  The BSDs tend to not play stupid emulation games in the libc, so
>> changes of these kinds of messups to happen are far less.
>
> In all fairness, I seem to recall there being a problem with the 
> kernel implementation of preadv/pwritev too.
>
> I think a configure option would be in order.  Forever avoiding glibc 
> is probably a bit extreme.
>

I'll go further and even avoid the configure options.  Users should run 
fully updated systems.

(how would a user know whether to enable or disable the option? by the 
time they figure out, they might as well update the system).

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: [Qemu-devel] Heads up: glibc preadv emulation breaks qemu on older kernels

[Christoph Hellwig]

On Fri, Feb 12, 2010 at 11:28:57AM -0600, Anthony Liguori wrote:
> >Yes.  The BSDs tend to not play stupid emulation games in the libc, so
> >changes of these kinds of messups to happen are far less.
> >   
> 
> In all fairness, I seem to recall there being a problem with the kernel 
> implementation of preadv/pwritev too.

I would be surprised by that as it's just an entirely trivial entry
point to long existant infrastructure.  I even checked the commit logs
and that code hasn't been touched at all since the initial commit.
So if people have problems with it, please report it..

Re: [Qemu-devel] Heads up: glibc preadv emulation breaks qemu on older kernels

[Avi Kivity]

On 02/16/2010 12:16 AM, Christoph Hellwig wrote:
> On Fri, Feb 12, 2010 at 11:28:57AM -0600, Anthony Liguori wrote:
>    
>>> Yes.  The BSDs tend to not play stupid emulation games in the libc, so
>>> changes of these kinds of messups to happen are far less.
>>>
>>>        
>> In all fairness, I seem to recall there being a problem with the kernel
>> implementation of preadv/pwritev too.
>>      
> I would be surprised by that as it's just an entirely trivial entry
> point to long existant infrastructure.  I even checked the commit logs
> and that code hasn't been touched at all since the initial commit.
> So if people have problems with it, please report it..
>    

IIRC there was a problem with wiring up glibc to use p*v on i386.  The 
large number and size of the arguments caused some confusion wrt the 
calling convention, or something.   So it's only fair that glibc 
emulation of the syscall would be broken too.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Heads up: glibc preadv emulation breaks qemu on older kernels

[Richard W.M. Jones]

On Tue, Feb 16, 2010 at 12:41:43PM +0200, Avi Kivity wrote:
> On 02/16/2010 12:16 AM, Christoph Hellwig wrote:
>> On Fri, Feb 12, 2010 at 11:28:57AM -0600, Anthony Liguori wrote:
>>    
>>>> Yes.  The BSDs tend to not play stupid emulation games in the libc, so
>>>> changes of these kinds of messups to happen are far less.
>>>>
>>>>        
>>> In all fairness, I seem to recall there being a problem with the kernel
>>> implementation of preadv/pwritev too.
>>>      
>> I would be surprised by that as it's just an entirely trivial entry
>> point to long existant infrastructure.  I even checked the commit logs
>> and that code hasn't been touched at all since the initial commit.
>> So if people have problems with it, please report it..
>>    
>
> IIRC there was a problem with wiring up glibc to use p*v on i386.  The  
> large number and size of the arguments caused some confusion wrt the  
> calling convention, or something.   So it's only fair that glibc  
> emulation of the syscall would be broken too.

It might be this bug:

https://bugzilla.redhat.com/show_bug.cgi?id=533063

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming blog: http://rwmj.wordpress.com
Fedora now supports 80 OCaml packages (the OPEN alternative to F#)
http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora

Re: [Qemu-devel] Heads up: glibc preadv emulation breaks qemu on older kernels

[Jamie Lokier]

malc wrote:
> On Fri, 12 Feb 2010, Christoph Hellwig wrote:
> 
> > On Thu, Feb 11, 2010 at 07:06:08PM +0000, Richard W.M. Jones wrote:
> > > One interesting thing is that qemu has its own preadv emulation (which
> > > does the emulation correctly), but this is never used because qemu
> > > never gets ENOSYS back from preadv.
> > 
> > At this point the amount of bugs in the glibc preadv/pwritev code really
> > make me want to go to use the raw system calls on Linux only.  Any
> > opinions from the maintainers if that is acceptable?
> 
> There are more than one way to parse the first sentence, if it should be
> read as:
>   On linux and on linux only avoid using pread/write[v] and talk to the
>   kernel directly.
> 
> Then i agree.

Do we know if BSDs have the same bug in some versions?

There have been (very ancient now) unixes where libc "emulated"
readv/writev, with the result that sending UDP packets broke, and TCP
streams were excessively segmented.  There have been more recently
very dodgy libc emulations of pselect (not signal safe like it's
entire point for existing).  For that matter pread/pwrite having
broken libc emulation too (not thread safe despite that being a main
reason for existing).

Did I mention I really, really hate libc "emulating" system calls badly.

-- Jamie