Re: [PATCH v5 00/10] xen-block: multi hardware-queues/rings support

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

On Thu, Nov 26, 2015 at 10:28:10AM +0800, Bob Liu wrote:
> 
> On 11/26/2015 06:12 AM, Konrad Rzeszutek Wilk wrote:
> > On Wed, Nov 25, 2015 at 03:56:03PM -0500, Konrad Rzeszutek Wilk wrote:
> >> On Wed, Nov 25, 2015 at 02:25:07PM -0500, Konrad Rzeszutek Wilk wrote:
> >>>>   xen/blkback: separate ring information out of struct xen_blkif
> >>>>   xen/blkback: pseudo support for multi hardware queues/rings
> >>>>   xen/blkback: get the number of hardware queues/rings from blkfront
> >>>>   xen/blkback: make pool of persistent grants and free pages per-queue
> >>>
> >>> OK, got to those as well. I have put them in 'devel/for-jens-4.5' and
> >>> are going to test them overnight before pushing them out.
> >>>
> >>> I see two bugs in the code that we MUST deal with:
> >>>
> >>>  - print_stats () is going to show zero values.
> >>>  - the sysfs code (VBD_SHOW) aren't converted over to fetch data
> >>>    from all the rings.
> >>
> >> - kthread_run can't handle the two "name, i" arguments. I see:
> >>
> >> root      5101     2  0 20:47 ?        00:00:00 [blkback.3.xvda-]
> >> root      5102     2  0 20:47 ?        00:00:00 [blkback.3.xvda-]
> > 
> > And doing save/restore:
> > 
> > xl save <id> /tmp/A;
> > xl restore /tmp/A;
> > 
> > ends up us loosing the proper state and not getting the ring setup back.
> > I see this is backend:
> > 
> > [ 2719.448600] vbd vbd-22-51712: -1 guest requested 0 queues, exceeding the maximum of 3.
> > 
> > And XenStore agrees:
> > tool = ""
> >  xenstored = ""
> > local = ""
> >  domain = ""
> >   0 = ""
> >    domid = "0"
> >    name = "Domain-0"
> >    device-model = ""
> >     0 = ""
> >      state = "running"
> >    error = ""
> >     backend = ""
> >      vbd = ""
> >       2 = ""
> >        51712 = ""
> >         error = "-1 guest requested 0 queues, exceeding the maximum of 3."
> > 
> > .. which also leads to a memory leak as xen_blkbk_remove never gets
> > called.
> 
> I think which was already fix by your patch:
> [PATCH RFC 2/2] xen/blkback: Free resources if connect_ring failed.

Nope. I get that with or without the patch.

I pushed the patches in
git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git
#devel/for-jens-4.5 tree. It also has some extra patches that should be
soon going via the x86 tree.

With the xen-blkback compiled with #define DEBUG 1 I see:

[   63.887741] xen-blkback: xen_blkbk_probe ffff880026a8cc00 1
[   63.894302] xen-blkback: backend_changed ffff880026a8cc00 1
[   63.895748] xen-blkback: frontend_changed ffff880026a8cc00 Initialising
[   63.922700] xen-blkback: xen_blkbk_probe ffff8800269da800 1
[   63.927849] xen-blkback: backend_changed ffff8800269da800 1
[   63.929117] xen-blkback: Successful creation of handle=ca00 (dom=1)
[   63.930605] xen-blkback: frontend_changed ffff8800269da800 Initialising
[   64.097161] xen-blkback: backend_changed ffff880026a8cc00 1
[   64.098992] xen-blkback: Successful creation of handle=1600 (dom=1)
[   64.345913] device vif1.0 entered promiscuous mode
[   64.351469] IPv6: ADDRCONF(NETDEV_UP): vif1.0: link is not ready
[   64.538682] device vif1.0-emu entered promiscuous mode
[   64.546592] switch: port 3(vif1.0-emu) entered forwarding state
[   64.548357] switch: port 3(vif1.0-emu) entered forwarding state
[   79.544475] switch: port 3(vif1.0-emu) entered forwarding state
[   84.090637] switch: port 3(vif1.0-emu) entered disabled state
[   84.091545] device vif1.0-emu left promiscuous mode
[   84.092416] switch: port 3(vif1.0-emu) entered disabled state
[   89.286901] vif vif-1-0 vif1.0: Guest Rx ready
[   89.287921] IPv6: ADDRCONF(NETDEV_CHANGE): vif1.0: link becomes ready
[   89.288943] switch: port 2(vif1.0) entered forwarding state
[   89.289747] switch: port 2(vif1.0) entered forwarding state
[   89.456176] xen-blkback: frontend_changed ffff880026a8cc00 Closed
[   89.481945] xen-blkback: frontend_changed ffff8800269da800 Initialised
[   89.482802] xen-blkback: connect_ring /local/domain/1/device/vbd/51712
[   89.484068] xen-blkback: backend/vbd/1/51712: using 2 queues, protocol 2 (x86_32-abi) persistent grants
[   89.532755] xen-blkback: connect /local/domain/1/device/vbd/51712
[   89.541694] xen_update_blkif_status: name=[blkback.1.xvda-0]
[   89.542667] xen_update_blkif_status: name=[blkback.1.xvda-1]
[   89.561913] xen-blkback: frontend_changed ffff8800269da800 Connected

.. so here the guest booted and now we are suspending it.

[  104.300579] switch: port 2(vif1.0) entered forwarding state
[  208.057752] xen-blkback: frontend_changed ffff880026a8cc00 Unknown
[  208.061282] xen-blkback: xen_blkbk_remove ffff880026a8cc00 1
[  208.081888] xen-blkback: frontend_changed ffff8800269da800 Unknown
[  208.082759] xen-blkback: xen_blkbk_remove ffff8800269da800 1
[  208.102745] switch: port 2(vif1.0) entered disabled state
[  208.109089] switch: port 2(vif1.0) entered disabled state
[  208.109934] device vif1.0 left promiscuous mode
[  208.110734] switch: port 2(vif1.0) entered disabled state

We are done with suspend and we are resuming:

[  302.128132] xen-blkback: xen_blkbk_probe ffff880027fb1c00 2
[  302.134915] xen-blkback: backend_changed ffff880027fb1c00 2
[  302.136568] xen-blkback: frontend_changed ffff880027fb1c00 Initialising
[  302.165161] xen-blkback: xen_blkbk_probe ffff88001e2cac00 2
[  302.170157] xen-blkback: backend_changed ffff88001e2cac00 2
[  302.171420] xen-blkback: Successful creation of handle=ca00 (dom=2)
[  302.172873] xen-blkback: frontend_changed ffff88001e2cac00 Initialising
[  302.409007] xen-blkback: backend_changed ffff880027fb1c00 2
[  302.411237] xen-blkback: Successful creation of handle=1600 (dom=2)
[  302.634468] device vif2.0 entered promiscuous mode
[  302.639871] IPv6: ADDRCONF(NETDEV_UP): vif2.0: link is not ready
[  302.822459] device vif2.0-emu entered promiscuous mode
[  302.830021] switch: port 3(vif2.0-emu) entered forwarding state
[  302.831568] switch: port 3(vif2.0-emu) entered forwarding state
[  302.849206] switch: port 3(vif2.0-emu) entered disabled state
[  302.850952] device vif2.0-emu left promiscuous mode
[  302.852567] switch: port 3(vif2.0-emu) entered disabled state
[  302.895719] xen-blkback: frontend_changed ffff88002e2cac00 Initialised
[  302.897237] xen-blkback: connect_ring /local/domain/2/device/vbd/51712
[  302.899389] vbd vbd-2-51712: -1 guest requested 0 queues, exceeding the maximum of 3.
[  303.038086] xen-blkback: frontend_changed ffff88001e2cac00 Closing
[  303.073058] vif vif-2-0 vif2.0: Guest Rx ready
[  303.074057] IPv6: ADDRCONF(NETDEV_CHANGE): vif2.0: link becomes ready
[  303.075079] switch: port 2(vif2.0) entered forwarding state
[  303.075976] switch: port 2(vif2.0) entered forwarding state
[  306.204385] switch: port 2(vif2.0) entered disabled state
[  306.234820] xen-blkback: frontend_changed ffff880027fb1c00 Unknown
[  306.236307] xen-blkback: xen_blkbk_remove ffff880027fb1c00 2
[  306.248684] xen-blkback: frontend_changed ffff88001e2cac00 Unknown
[  306.249958] xen-blkback: xen_blkbk_remove ffff88001e2cac00 2
[  306.262379] switch: port 2(vif2.0) entered disabled state
[  306.263280] device vif2.0 left promiscuous mode
[  306.264210] switch: port 2(vif2.0) entered disabled state