Re: [xen-discuss] possible memory-leak in qemu-dm

Re: [xen-discuss] possible memory-leak in qemu-dm

[Florian Manschwetus]

[-- Attachment #1.1: Type: text/plain, Size: 958 bytes --]

Am 14.10.2009 21:46, schrieb Mark Johnson:
> 
> Hi Florian,
> 
> Florian Manschwetus wrote:
>> Mark,
>> could you also make a short info howto update, a system I build just
>> yesterday, when the patch is inplace. The older Machine I would have to
>> re-bfu due to changed userspace, but the machine installed yesterday
>> should be fixed with rebuilding and reinstalling xvm packages?
>>
>> Btw, could you sync-up the onnv-3.4 with b124? It is difficult to build
>> b123 with b124, due to missing, with b124 obsolete, headerfiles.
> 
> I've sync'd up with the changes in qemu-unstable, and qemu-dm is
> still leaking a ton of memory for HVM guests which don't have PV
> drivers.  So it looks like they don't have a correct fix upstream
> for this.
Uh, if this could be confirmed it is really a huge and ultimately ugly
problem. If I could do anything to help you to fix it, give me some detail.

Florian
> 
> 
> 
> 
> MRJ
> 



[-- Attachment #1.2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3686 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Re: [xen-discuss] possible memory-leak in qemu-dm

[Pasi Kärkkäinen]

On Wed, Oct 14, 2009 at 11:58:14PM +0200, Florian Manschwetus wrote:
> Am 14.10.2009 21:46, schrieb Mark Johnson:
> > 
> > Hi Florian,
> > 
> > Florian Manschwetus wrote:
> >> Mark,
> >> could you also make a short info howto update, a system I build just
> >> yesterday, when the patch is inplace. The older Machine I would have to
> >> re-bfu due to changed userspace, but the machine installed yesterday
> >> should be fixed with rebuilding and reinstalling xvm packages?
> >>
> >> Btw, could you sync-up the onnv-3.4 with b124? It is difficult to build
> >> b123 with b124, due to missing, with b124 obsolete, headerfiles.
> > 
> > I've sync'd up with the changes in qemu-unstable, and qemu-dm is
> > still leaking a ton of memory for HVM guests which don't have PV
> > drivers.  So it looks like they don't have a correct fix upstream
> > for this.
> Uh, if this could be confirmed it is really a huge and ultimately ugly
> problem. If I could do anything to help you to fix it, give me some detail.
> 

Yeah.. what kind of HVM guest is triggering this bug? 
How fast it is leaking mem? 

-- Pasi

Re: Re: [xen-discuss] possible memory-leak in qemu-dm

[Florian Manschwetus]

[-- Attachment #1.1: Type: text/plain, Size: 1512 bytes --]

Am 15.10.2009 12:26, schrieb Pasi Kärkkäinen:
> On Wed, Oct 14, 2009 at 11:58:14PM +0200, Florian Manschwetus wrote:
>> Am 14.10.2009 21:46, schrieb Mark Johnson:
>>>
>>> Hi Florian,
>>>
>>> Florian Manschwetus wrote:
>>>> Mark,
>>>> could you also make a short info howto update, a system I build just
>>>> yesterday, when the patch is inplace. The older Machine I would have to
>>>> re-bfu due to changed userspace, but the machine installed yesterday
>>>> should be fixed with rebuilding and reinstalling xvm packages?
>>>>
>>>> Btw, could you sync-up the onnv-3.4 with b124? It is difficult to build
>>>> b123 with b124, due to missing, with b124 obsolete, headerfiles.
>>>
>>> I've sync'd up with the changes in qemu-unstable, and qemu-dm is
>>> still leaking a ton of memory for HVM guests which don't have PV
>>> drivers.  So it looks like they don't have a correct fix upstream
>>> for this.
>> Uh, if this could be confirmed it is really a huge and ultimately ugly
>> problem. If I could do anything to help you to fix it, give me some detail.
>>
> 
> Yeah.. what kind of HVM guest is triggering this bug? 
> How fast it is leaking mem? 
It is a win2003R2 with exchange2007

xm uptime tmpexch
Name                                ID Uptime
tmpexch                              7 9 days,  3:21:39

PID USERNAME NLWP PRI NICE  SIZE   RES STATE    TIME    CPU COMMAND
2106 xvm         3  13    0 7340M 3807M cpu/4   31.4H  6.57% qemu-dm

Florian

> 
> -- Pasi
> 



[-- Attachment #1.2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3686 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Re: [xen-discuss] possible memory-leak in qemu-dm

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 1776 bytes --]

On Thu, 15 Oct 2009, Florian Manschwetus wrote:
> Am 15.10.2009 12:26, schrieb Pasi Kärkkäinen:
> > On Wed, Oct 14, 2009 at 11:58:14PM +0200, Florian Manschwetus wrote:
> >> Am 14.10.2009 21:46, schrieb Mark Johnson:
> >>>
> >>> Hi Florian,
> >>>
> >>> Florian Manschwetus wrote:
> >>>> Mark,
> >>>> could you also make a short info howto update, a system I build just
> >>>> yesterday, when the patch is inplace. The older Machine I would have to
> >>>> re-bfu due to changed userspace, but the machine installed yesterday
> >>>> should be fixed with rebuilding and reinstalling xvm packages?
> >>>>
> >>>> Btw, could you sync-up the onnv-3.4 with b124? It is difficult to build
> >>>> b123 with b124, due to missing, with b124 obsolete, headerfiles.
> >>>
> >>> I've sync'd up with the changes in qemu-unstable, and qemu-dm is
> >>> still leaking a ton of memory for HVM guests which don't have PV
> >>> drivers.  So it looks like they don't have a correct fix upstream
> >>> for this.
> >> Uh, if this could be confirmed it is really a huge and ultimately ugly
> >> problem. If I could do anything to help you to fix it, give me some detail.
> >>
> > 
> > Yeah.. what kind of HVM guest is triggering this bug? 
> > How fast it is leaking mem? 
> It is a win2003R2 with exchange2007
> 
> xm uptime tmpexch
> Name                                ID Uptime
> tmpexch                              7 9 days,  3:21:39
> 
> PID USERNAME NLWP PRI NICE  SIZE   RES STATE    TIME    CPU COMMAND
> 2106 xvm         3  13    0 7340M 3807M cpu/4   31.4H  6.57% qemu-dm
> 
> Florian
> 

Was this happening before you applied my patches, but after "fix qemu
memory leak in block interface"?
I am trying to understand if this is a recent regression.

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Re: [xen-discuss] possible memory-leak in qemu-dm

[Florian Manschwetus]

[-- Attachment #1.1: Type: text/plain, Size: 2072 bytes --]

Am 15.10.2009 13:04, schrieb Stefano Stabellini:
> On Thu, 15 Oct 2009, Florian Manschwetus wrote:
>> Am 15.10.2009 12:26, schrieb Pasi Kärkkäinen:
>>> On Wed, Oct 14, 2009 at 11:58:14PM +0200, Florian Manschwetus wrote:
>>>> Am 14.10.2009 21:46, schrieb Mark Johnson:
>>>>>
>>>>> Hi Florian,
>>>>>
>>>>> Florian Manschwetus wrote:
>>>>>> Mark,
>>>>>> could you also make a short info howto update, a system I build just
>>>>>> yesterday, when the patch is inplace. The older Machine I would have to
>>>>>> re-bfu due to changed userspace, but the machine installed yesterday
>>>>>> should be fixed with rebuilding and reinstalling xvm packages?
>>>>>>
>>>>>> Btw, could you sync-up the onnv-3.4 with b124? It is difficult to build
>>>>>> b123 with b124, due to missing, with b124 obsolete, headerfiles.
>>>>>
>>>>> I've sync'd up with the changes in qemu-unstable, and qemu-dm is
>>>>> still leaking a ton of memory for HVM guests which don't have PV
>>>>> drivers.  So it looks like they don't have a correct fix upstream
>>>>> for this.
>>>> Uh, if this could be confirmed it is really a huge and ultimately ugly
>>>> problem. If I could do anything to help you to fix it, give me some detail.
>>>>
>>>
>>> Yeah.. what kind of HVM guest is triggering this bug? 
>>> How fast it is leaking mem? 
>> It is a win2003R2 with exchange2007
>>
>> xm uptime tmpexch
>> Name                                ID Uptime
>> tmpexch                              7 9 days,  3:21:39
>>
>> PID USERNAME NLWP PRI NICE  SIZE   RES STATE    TIME    CPU COMMAND
>> 2106 xvm         3  13    0 7340M 3807M cpu/4   31.4H  6.57% qemu-dm
>>
>> Florian
>>
> 
> Was this happening before you applied my patches, but after "fix qemu
> memory leak in block interface"?
> I am trying to understand if this is a recent regression.
It is an older build of xvm-3.4 gate
I have to review current patch state and will update accordingly if
possible.

It was Mark, who told that the problem isn't fixed by the Patch. So I
not rushed to update.

Florian


[-- Attachment #1.2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3686 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Re: [xen-discuss] possible memory-leak in qemu-dm

[Mark Johnson]

On Thu, Oct 15, 2009 at 7:04 AM, Stefano Stabellini
<stefano.stabellini@eu.citrix.com> wrote:
> Was this happening before you applied my patches, but after "fix qemu
> memory leak in block interface"?
> I am trying to understand if this is a recent regression.

I'm not sure when it first showed up...  It exists in current qemu-xen-3.4 bits.

I've updated the qemu bits to the latest qemu-xen-unstable bits (basically
running qemu-xen-unstable with xen 3.4) including your patches, and
qemu-dm is still leaking memory with a HVM guest with no PV drivers..

Tried a couple of different HVM guests with the same results..




MRJ

Re: Re: [xen-discuss] possible memory-leak in qemu-dm

[Pasi Kärkkäinen]

On Thu, Oct 15, 2009 at 01:24:29PM +0200, Florian Manschwetus wrote:
> >>>
> >>> Yeah.. what kind of HVM guest is triggering this bug? 
> >>> How fast it is leaking mem? 
> >> It is a win2003R2 with exchange2007
> >>
> >> xm uptime tmpexch
> >> Name                                ID Uptime
> >> tmpexch                              7 9 days,  3:21:39
> >>
> >> PID USERNAME NLWP PRI NICE  SIZE   RES STATE    TIME    CPU COMMAND
> >> 2106 xvm         3  13    0 7340M 3807M cpu/4   31.4H  6.57% qemu-dm
> >>
> >> Florian
> >>
> > 
> > Was this happening before you applied my patches, but after "fix qemu
> > memory leak in block interface"?
> > I am trying to understand if this is a recent regression.
> It is an older build of xvm-3.4 gate
> I have to review current patch state and will update accordingly if
> possible.
> 
> It was Mark, who told that the problem isn't fixed by the Patch. So I
> not rushed to update.
> 

Hmm.. wondering if this is opensolaris specific, or if it happens also
with Linux dom0s.

-- Pasi

Re: Re: [xen-discuss] possible memory-leak in qemu-dm

[Mark Johnson]

On Wed, Oct 21, 2009 at 7:24 AM, Pasi Kärkkäinen <pasik@iki.fi> wrote:
> On Thu, Oct 15, 2009 at 01:24:29PM +0200, Florian Manschwetus wrote:
>> >
>> > Was this happening before you applied my patches, but after "fix qemu
>> > memory leak in block interface"?
>> > I am trying to understand if this is a recent regression.
>> It is an older build of xvm-3.4 gate
>> I have to review current patch state and will update accordingly if
>> possible.
>>
>> It was Mark, who told that the problem isn't fixed by the Patch. So I
>> not rushed to update.
>>
>
> Hmm.. wondering if this is opensolaris specific,

It's possible... I looked through our qemu patches and we don't have
any changes in that code path..  So I would be surprised..


> or if it happens also with Linux dom0s.

Has anyone else seen this on 3.4 or unstable?  e.g. boot a rhel3 guest and watch
the memory usage of qemu-dm... You can't miss it if its leaking...



MRJ

Re: Re: [xen-discuss] possible memory-leak in qemu-dm

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 1201 bytes --]

On Wed, 21 Oct 2009, Mark Johnson wrote:
> On Wed, Oct 21, 2009 at 7:24 AM, Pasi Kärkkäinen <pasik@iki.fi> wrote:
> > On Thu, Oct 15, 2009 at 01:24:29PM +0200, Florian Manschwetus wrote:
> >> >
> >> > Was this happening before you applied my patches, but after "fix qemu
> >> > memory leak in block interface"?
> >> > I am trying to understand if this is a recent regression.
> >> It is an older build of xvm-3.4 gate
> >> I have to review current patch state and will update accordingly if
> >> possible.
> >>
> >> It was Mark, who told that the problem isn't fixed by the Patch. So I
> >> not rushed to update.
> >>
> >
> > Hmm.. wondering if this is opensolaris specific,
> 
> It's possible... I looked through our qemu patches and we don't have
> any changes in that code path..  So I would be surprised..
> 
> 
> > or if it happens also with Linux dom0s.
> 
> Has anyone else seen this on 3.4 or unstable?  e.g. boot a rhel3 guest and watch
> the memory usage of qemu-dm... You can't miss it if its leaking...
> 

What kind of operations are you doing in the guest?
Is there a simple benchmark that triggers this behaviour (for example a
disk or network benchmark)?

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Re: [xen-discuss] possible memory-leak in qemu-dm

[Mark Price]

On Wed, Oct 21, 2009 at 9:54 AM, Mark Johnson <johnson.nh@gmail.com> wrote:
> Has anyone else seen this on 3.4 or unstable?  e.g. boot a rhel3 guest and watch
> the memory usage of qemu-dm... You can't miss it if its leaking...

I just wanted to report that I also saw this memory leak problem on 3.4.1.

I applied the patch mentioned before:
http://lists.xensource.com/archives/html/xen-changelog/2009-10/msg00132.html

After applying the patch, the memory leak seems to be fixed.  This is
on a Linux 64-bit dom0 and freebsd HVM guest.  Memory leak of qemu-dm
was apparent just by extracting a ~5MB tarball of various text files.