[Qemu-devel] mem1 is in use, can not be deleted

[Qemu-devel] mem1 is in use, can not be deleted

[Eduardo Otubo]

[-- Attachment #1: Type: text/plain, Size: 797 bytes --]

Hello all,

I compiled the HEAD of the master branch and was testing memory
hotunplug and got to this issue. Note: I followed exactly what's written
on the docs/memory-hotplug.txt file.

QEMU 2.3.50 monitor - type 'help' for more information
(qemu) object_add memory-backend-ram,id=mem1,size=1G
object_add memory-backend-ram,id=mem1,size=1G
(qemu) device_add pc-dimm,id=dimm1,memdev=mem1
device_add pc-dimm,id=dimm1,memdev=mem1
(qemu) device_del dimm1
device_del dimm1
(qemu) object_del mem1
object_del mem1
mem1 is in use, can not be deleted

I tested with a Debian 7 running and on grub, in both situations I got
the same behavior. Anyone also got the same issue? I'll also try with a
different distro and will post back.

Regards,

-- 
Eduardo Otubo
ProfitBricks GmbH

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] mem1 is in use, can not be deleted

[Igor Mammedov]

On Tue, 30 Jun 2015 10:07:52 +0200
Eduardo Otubo <eduardo.otubo@profitbricks.com> wrote:

> Hello all,
> 
> I compiled the HEAD of the master branch and was testing memory
> hotunplug and got to this issue. Note: I followed exactly what's written
> on the docs/memory-hotplug.txt file.
> 
> QEMU 2.3.50 monitor - type 'help' for more information
> (qemu) object_add memory-backend-ram,id=mem1,size=1G
> object_add memory-backend-ram,id=mem1,size=1G
> (qemu) device_add pc-dimm,id=dimm1,memdev=mem1
> device_add pc-dimm,id=dimm1,memdev=mem1
> (qemu) device_del dimm1
> device_del dimm1
> (qemu) object_del mem1
> object_del mem1
> mem1 is in use, can not be deleted

probably because  dimm1 isn't deleted,
you can check it in monitor using command "info memory-devices"


> 
> I tested with a Debian 7 running and on grub, in both situations I got
> the same behavior. Anyone also got the same issue? I'll also try with a
> different distro and will post back.
> 
> Regards,
> 

Re: [Qemu-devel] mem1 is in use, can not be deleted

[Eduardo Otubo]

[-- Attachment #1: Type: text/plain, Size: 1488 bytes --]

On Tue, Jun 30, 2015 at 11=18=21AM +0200, Igor Mammedov wrote:
> On Tue, 30 Jun 2015 10:07:52 +0200
> Eduardo Otubo <eduardo.otubo@profitbricks.com> wrote:
> 
> > Hello all,
> > 
> > I compiled the HEAD of the master branch and was testing memory
> > hotunplug and got to this issue. Note: I followed exactly what's written
> > on the docs/memory-hotplug.txt file.
> > 
> > QEMU 2.3.50 monitor - type 'help' for more information
> > (qemu) object_add memory-backend-ram,id=mem1,size=1G
> > object_add memory-backend-ram,id=mem1,size=1G
> > (qemu) device_add pc-dimm,id=dimm1,memdev=mem1
> > device_add pc-dimm,id=dimm1,memdev=mem1
> > (qemu) device_del dimm1
> > device_del dimm1
> > (qemu) object_del mem1
> > object_del mem1
> > mem1 is in use, can not be deleted
> 
> probably because  dimm1 isn't deleted,
> you can check it in monitor using command "info memory-devices"

Yes, you're right. The reason is surely because dimm1 wasn't deleted --
and I think I didn't make my point very clear -- my question was more
about: Is there any reason for dimm1 not being deleted? The reason why I
tested with the guest OS fully running and on GRUB is because I guessed
the guest OS was using this memory and couldn't be deallocated. If
that's the case, and qemu did a best effort to remove and couldn't
because guest was using it, then Ok, I just need to adapt my tests.
Other than that perhaps I hit a bug.

Regards,

-- 
Eduardo Otubo
ProfitBricks GmbH

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] mem1 is in use, can not be deleted

[Igor Mammedov]

On Tue, 30 Jun 2015 15:56:13 +0200
Eduardo Otubo <eduardo.otubo@profitbricks.com> wrote:

> On Tue, Jun 30, 2015 at 11=18=21AM +0200, Igor Mammedov wrote:
> > On Tue, 30 Jun 2015 10:07:52 +0200
> > Eduardo Otubo <eduardo.otubo@profitbricks.com> wrote:
> > 
> > > Hello all,
> > > 
> > > I compiled the HEAD of the master branch and was testing memory
> > > hotunplug and got to this issue. Note: I followed exactly what's
> > > written on the docs/memory-hotplug.txt file.
> > > 
> > > QEMU 2.3.50 monitor - type 'help' for more information
> > > (qemu) object_add memory-backend-ram,id=mem1,size=1G
> > > object_add memory-backend-ram,id=mem1,size=1G
> > > (qemu) device_add pc-dimm,id=dimm1,memdev=mem1
> > > device_add pc-dimm,id=dimm1,memdev=mem1
> > > (qemu) device_del dimm1
> > > device_del dimm1
> > > (qemu) object_del mem1
> > > object_del mem1
> > > mem1 is in use, can not be deleted
> > 
> > probably because  dimm1 isn't deleted,
> > you can check it in monitor using command "info memory-devices"
> 
> Yes, you're right. The reason is surely because dimm1 wasn't deleted
> -- and I think I didn't make my point very clear -- my question was
> more about: Is there any reason for dimm1 not being deleted? The
> reason why I tested with the guest OS fully running and on GRUB is
> because I guessed the guest OS was using this memory and couldn't be
> deallocated. If that's the case, and qemu did a best effort to remove
> and couldn't because guest was using it, then Ok, I just need to
> adapt my tests. Other than that perhaps I hit a bug.
Guest OS has to:
 1. support memory hot remove
 2. eject memory device using ACPI _EJ0 method, once it has handled
 removal request, provided it is able to free corresponding memory pages
See docs they should have flows described for success and failure case.

If you are creating tests, it would be nice if you could integrate them
into QEMU's 'make check' infrastructure.
That could be hard to do for a full guest OS setup but it should be
easier to implement using MMIO interface QEMU exposes and simulating
ACPI part of memory hotplug in tests. It's not full coverage but it will
provide regression testing for the most parts of related code (sans
ACPI parts).


> 
> Regards,
> 

Re: [Qemu-devel] mem1 is in use, can not be deleted

[Eduardo Otubo]

[-- Attachment #1: Type: text/plain, Size: 3050 bytes --]

On Tue, Jun 30, 2015 at 05=56=02PM +0200, Igor Mammedov wrote:
> On Tue, 30 Jun 2015 15:56:13 +0200
> Eduardo Otubo <eduardo.otubo@profitbricks.com> wrote:
> 
> > On Tue, Jun 30, 2015 at 11=18=21AM +0200, Igor Mammedov wrote:
> > > On Tue, 30 Jun 2015 10:07:52 +0200
> > > Eduardo Otubo <eduardo.otubo@profitbricks.com> wrote:
> > > 
> > > > Hello all,
> > > > 
> > > > I compiled the HEAD of the master branch and was testing memory
> > > > hotunplug and got to this issue. Note: I followed exactly what's
> > > > written on the docs/memory-hotplug.txt file.
> > > > 
> > > > QEMU 2.3.50 monitor - type 'help' for more information
> > > > (qemu) object_add memory-backend-ram,id=mem1,size=1G
> > > > object_add memory-backend-ram,id=mem1,size=1G
> > > > (qemu) device_add pc-dimm,id=dimm1,memdev=mem1
> > > > device_add pc-dimm,id=dimm1,memdev=mem1
> > > > (qemu) device_del dimm1
> > > > device_del dimm1
> > > > (qemu) object_del mem1
> > > > object_del mem1
> > > > mem1 is in use, can not be deleted
> > > 
> > > probably because  dimm1 isn't deleted,
> > > you can check it in monitor using command "info memory-devices"
> > 
> > Yes, you're right. The reason is surely because dimm1 wasn't deleted
> > -- and I think I didn't make my point very clear -- my question was
> > more about: Is there any reason for dimm1 not being deleted? The
> > reason why I tested with the guest OS fully running and on GRUB is
> > because I guessed the guest OS was using this memory and couldn't be
> > deallocated. If that's the case, and qemu did a best effort to remove
> > and couldn't because guest was using it, then Ok, I just need to
> > adapt my tests. Other than that perhaps I hit a bug.
> Guest OS has to:
>  1. support memory hot remove

How do I know if guest OS supports memory hot remove? I'm testing on
Debian 8 with kernel 4.1. I start qemu with "-m 2G,slots=32,maxmem=8G".

>  2. eject memory device using ACPI _EJ0 method, once it has handled
>  removal request, provided it is able to free corresponding memory pages
> See docs they should have flows described for success and failure case.

When I issue the command "device_del dimm1" I see no output on dmesg on
the guest OS. I guess this is a sign that perhaps the guest does not
support it?

From the (very nice) diagram I found at docs/specs/acpi_mem_hotplug.txt,
Qemu QMP should output some sort of failure if Guest OS fails to
process ejection right? The only information I see is:

 (qemu) device_del dimm1
 device_del dimm1

 (qemu) info memory-devices
 info memory-devices
 Memory device [dimm]: "dimm1"
   addr: 0x100000000
   slot: 0
   node: 0
   size: 1073741824
   memdev: /objects/mem1
   hotplugged: true
   hotpluggable: true

 (qemu) info memdev
 info memdev
 memory backend: 0
   size:  1073741824
   merge: true
   dump: true
   prealloc: false
   policy: default
   host nodes: 

How was the environment when you tested this feature?

Regards,

-- 
Eduardo Otubo
ProfitBricks GmbH

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] mem1 is in use, can not be deleted

[Igor Mammedov]

On Thu, 9 Jul 2015 16:22:41 +0200
Eduardo Otubo <eduardo.otubo@profitbricks.com> wrote:

> On Tue, Jun 30, 2015 at 05=56=02PM +0200, Igor Mammedov wrote:
> > On Tue, 30 Jun 2015 15:56:13 +0200
> > Eduardo Otubo <eduardo.otubo@profitbricks.com> wrote:
> > 
> > > On Tue, Jun 30, 2015 at 11=18=21AM +0200, Igor Mammedov wrote:
> > > > On Tue, 30 Jun 2015 10:07:52 +0200
> > > > Eduardo Otubo <eduardo.otubo@profitbricks.com> wrote:
> > > > 
> > > > > Hello all,
> > > > > 
> > > > > I compiled the HEAD of the master branch and was testing memory
> > > > > hotunplug and got to this issue. Note: I followed exactly what's
> > > > > written on the docs/memory-hotplug.txt file.
> > > > > 
> > > > > QEMU 2.3.50 monitor - type 'help' for more information
> > > > > (qemu) object_add memory-backend-ram,id=mem1,size=1G
> > > > > object_add memory-backend-ram,id=mem1,size=1G
> > > > > (qemu) device_add pc-dimm,id=dimm1,memdev=mem1
> > > > > device_add pc-dimm,id=dimm1,memdev=mem1
> > > > > (qemu) device_del dimm1
> > > > > device_del dimm1
> > > > > (qemu) object_del mem1
> > > > > object_del mem1
> > > > > mem1 is in use, can not be deleted
> > > > 
> > > > probably because  dimm1 isn't deleted,
> > > > you can check it in monitor using command "info memory-devices"
> > > 
> > > Yes, you're right. The reason is surely because dimm1 wasn't deleted
> > > -- and I think I didn't make my point very clear -- my question was
> > > more about: Is there any reason for dimm1 not being deleted? The
> > > reason why I tested with the guest OS fully running and on GRUB is
> > > because I guessed the guest OS was using this memory and couldn't be
> > > deallocated. If that's the case, and qemu did a best effort to remove
> > > and couldn't because guest was using it, then Ok, I just need to
> > > adapt my tests. Other than that perhaps I hit a bug.
> > Guest OS has to:
> >  1. support memory hot remove
> 
> How do I know if guest OS supports memory hot remove? I'm testing on
> Debian 8 with kernel 4.1. I start qemu with "-m 2G,slots=32,maxmem=8G".
kernel should be compiled with memory remove options

Also memory removal is allowed to fail if guest kernel is not able
to offline corresponding memory sections but it probably should notify
QEMU via ACPI about failure.


> 
> >  2. eject memory device using ACPI _EJ0 method, once it has handled
> >  removal request, provided it is able to free corresponding memory pages
> > See docs they should have flows described for success and failure case.
> 
> When I issue the command "device_del dimm1" I see no output on dmesg on
> the guest OS. I guess this is a sign that perhaps the guest does not
> support it?
> 
> From the (very nice) diagram I found at docs/specs/acpi_mem_hotplug.txt,
> Qemu QMP should output some sort of failure if Guest OS fails to
> process ejection right? The only information I see is:
you need to use query-acpi-ospm-status command to see slot status.

> 
>  (qemu) device_del dimm1
>  device_del dimm1
> 
>  (qemu) info memory-devices
>  info memory-devices
>  Memory device [dimm]: "dimm1"
>    addr: 0x100000000
>    slot: 0
>    node: 0
>    size: 1073741824
>    memdev: /objects/mem1
>    hotplugged: true
>    hotpluggable: true
> 
>  (qemu) info memdev
>  info memdev
>  memory backend: 0
>    size:  1073741824
>    merge: true
>    dump: true
>    prealloc: false
>    policy: default
>    host nodes: 
> 
> How was the environment when you tested this feature?
Most likely I've used RHEL7.1 as guest with latest systemd
which onlines hotplugged memory automatically on hotplug.

> 
> Regards,
> 

Re: [Qemu-devel] mem1 is in use, can not be deleted

[Eduardo Otubo]

[-- Attachment #1: Type: text/plain, Size: 2949 bytes --]

> > > > Yes, you're right. The reason is surely because dimm1 wasn't deleted
> > > > -- and I think I didn't make my point very clear -- my question was
> > > > more about: Is there any reason for dimm1 not being deleted? The
> > > > reason why I tested with the guest OS fully running and on GRUB is
> > > > because I guessed the guest OS was using this memory and couldn't be
> > > > deallocated. If that's the case, and qemu did a best effort to remove
> > > > and couldn't because guest was using it, then Ok, I just need to
> > > > adapt my tests. Other than that perhaps I hit a bug.
> > > Guest OS has to:
> > >  1. support memory hot remove
> > 
> > How do I know if guest OS supports memory hot remove? I'm testing on
> > Debian 8 with kernel 4.1. I start qemu with "-m 2G,slots=32,maxmem=8G".
> kernel should be compiled with memory remove options

Double checked that and yes, my guest kernel has memory hotplug support.

> 
> Also memory removal is allowed to fail if guest kernel is not able
> to offline corresponding memory sections but it probably should notify
> QEMU via ACPI about failure.

How can I check this notification?

> 
> 
> > 
> > >  2. eject memory device using ACPI _EJ0 method, once it has handled
> > >  removal request, provided it is able to free corresponding memory pages
> > > See docs they should have flows described for success and failure case.
> > 
> > When I issue the command "device_del dimm1" I see no output on dmesg on
> > the guest OS. I guess this is a sign that perhaps the guest does not
> > support it?
> > 
> > From the (very nice) diagram I found at docs/specs/acpi_mem_hotplug.txt,
> > Qemu QMP should output some sort of failure if Guest OS fails to
> > process ejection right? The only information I see is:
> you need to use query-acpi-ospm-status command to see slot status.

Yep, this command outputs that the dimm is still there, no news :/

> 
> > 
> >  (qemu) device_del dimm1
> >  device_del dimm1
> > 
> >  (qemu) info memory-devices
> >  info memory-devices
> >  Memory device [dimm]: "dimm1"
> >    addr: 0x100000000
> >    slot: 0
> >    node: 0
> >    size: 1073741824
> >    memdev: /objects/mem1
> >    hotplugged: true
> >    hotpluggable: true
> > 
> >  (qemu) info memdev
> >  info memdev
> >  memory backend: 0
> >    size:  1073741824
> >    merge: true
> >    dump: true
> >    prealloc: false
> >    policy: default
> >    host nodes: 
> > 
> > How was the environment when you tested this feature?
> Most likely I've used RHEL7.1 as guest with latest systemd
> which onlines hotplugged memory automatically on hotplug.

I tried with Ubuntu 15.04, latest kernel 4.2 and systemd, still not
working. I'm downloading CentOS-7, I'll setup with systemd and proper
kernel configuration. I'll let you know the results.

Thanks a lot for the help so far! :)

-- 
Eduardo Otubo
ProfitBricks GmbH

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] mem1 is in use, can not be deleted

[Igor Mammedov]

On Fri, 10 Jul 2015 17:35:08 +0200
Eduardo Otubo <eduardo.otubo@profitbricks.com> wrote:

> > > > > Yes, you're right. The reason is surely because dimm1 wasn't deleted
> > > > > -- and I think I didn't make my point very clear -- my question was
> > > > > more about: Is there any reason for dimm1 not being deleted? The
> > > > > reason why I tested with the guest OS fully running and on GRUB is
> > > > > because I guessed the guest OS was using this memory and couldn't be
> > > > > deallocated. If that's the case, and qemu did a best effort to remove
> > > > > and couldn't because guest was using it, then Ok, I just need to
> > > > > adapt my tests. Other than that perhaps I hit a bug.
> > > > Guest OS has to:
> > > >  1. support memory hot remove
> > > 
> > > How do I know if guest OS supports memory hot remove? I'm testing on
> > > Debian 8 with kernel 4.1. I start qemu with "-m 2G,slots=32,maxmem=8G".
> > kernel should be compiled with memory remove options
> 
> Double checked that and yes, my guest kernel has memory hotplug support.
> 
> > 
> > Also memory removal is allowed to fail if guest kernel is not able
> > to offline corresponding memory sections but it probably should notify
> > QEMU via ACPI about failure.
> 
> How can I check this notification?
build QEMU with tracing enabled and you can use it to see what's going on
in QEMU<->guest ACPI interface.

$grep mhp trace-events 
mhp_acpi_invalid_slot_selected(uint32_t slot) "0x%"PRIx32
mhp_acpi_ejecting_invalid_slot(uint32_t slot) "0x%"PRIx32
mhp_acpi_read_addr_lo(uint32_t slot, uint32_t addr) "slot[0x%"PRIx32"] addr lo: 0x%"PRIx32
mhp_acpi_read_addr_hi(uint32_t slot, uint32_t addr) "slot[0x%"PRIx32"] addr hi: 0x%"PRIx32
mhp_acpi_read_size_lo(uint32_t slot, uint32_t size) "slot[0x%"PRIx32"] size lo: 0x%"PRIx32
mhp_acpi_read_size_hi(uint32_t slot, uint32_t size) "slot[0x%"PRIx32"] size hi: 0x%"PRIx32
mhp_acpi_read_pxm(uint32_t slot, uint32_t pxm) "slot[0x%"PRIx32"] proximity: 0x%"PRIx32
mhp_acpi_read_flags(uint32_t slot, uint32_t flags) "slot[0x%"PRIx32"] flags: 0x%"PRIx32
mhp_acpi_write_slot(uint32_t slot) "set active slot: 0x%"PRIx32
mhp_acpi_write_ost_ev(uint32_t slot, uint32_t ev) "slot[0x%"PRIx32"] OST EVENT: 0x%"PRIx32
mhp_acpi_write_ost_status(uint32_t slot, uint32_t st) "slot[0x%"PRIx32"] OST STATUS: 0x%"PRIx32
mhp_acpi_clear_insert_evt(uint32_t slot) "slot[0x%"PRIx32"] clear insert event"
mhp_acpi_clear_remove_evt(uint32_t slot) "slot[0x%"PRIx32"] clear remove event"
mhp_acpi_pc_dimm_deleted(uint32_t slot) "slot[0x%"PRIx32"] pc-dimm deleted"
mhp_acpi_pc_dimm_delete_failed(uint32_t slot) "slot[0x%"PRIx32"] pc-dimm delete failed"
mhp_pc_dimm_assigned_slot(int slot) "0x%d"
mhp_pc_dimm_assigned_address(uint64_t addr) "0x%"PRIx64

events of interest for you may be *_ost_*
you can check hw/acpi/memory_hotplug.c for events meaning

You also might need to enable ACPI debugging in guest to see what's happening
on guest side if there is no *pc_dimm_deleted event in the trace.

> 
> > 
> > 
> > > 
> > > >  2. eject memory device using ACPI _EJ0 method, once it has handled
> > > >  removal request, provided it is able to free corresponding memory pages
> > > > See docs they should have flows described for success and failure case.
> > > 
> > > When I issue the command "device_del dimm1" I see no output on dmesg on
> > > the guest OS. I guess this is a sign that perhaps the guest does not
> > > support it?
> > > 
> > > From the (very nice) diagram I found at docs/specs/acpi_mem_hotplug.txt,
> > > Qemu QMP should output some sort of failure if Guest OS fails to
> > > process ejection right? The only information I see is:
> > you need to use query-acpi-ospm-status command to see slot status.
> 
> Yep, this command outputs that the dimm is still there, no news :/
> 
> > 
> > > 
> > >  (qemu) device_del dimm1
> > >  device_del dimm1
> > > 
> > >  (qemu) info memory-devices
> > >  info memory-devices
> > >  Memory device [dimm]: "dimm1"
> > >    addr: 0x100000000
> > >    slot: 0
> > >    node: 0
> > >    size: 1073741824
> > >    memdev: /objects/mem1
> > >    hotplugged: true
> > >    hotpluggable: true
> > > 
> > >  (qemu) info memdev
> > >  info memdev
> > >  memory backend: 0
> > >    size:  1073741824
> > >    merge: true
> > >    dump: true
> > >    prealloc: false
> > >    policy: default
> > >    host nodes: 
> > > 
> > > How was the environment when you tested this feature?
> > Most likely I've used RHEL7.1 as guest with latest systemd
> > which onlines hotplugged memory automatically on hotplug.
> 
> I tried with Ubuntu 15.04, latest kernel 4.2 and systemd, still not
> working. I'm downloading CentOS-7, I'll setup with systemd and proper
> kernel configuration. I'll let you know the results.
> 
> Thanks a lot for the help so far! :)
>