Re: Hotplugged devices in Xen 4.5 and domain reboot

[Iurii Mykhalskyi <iurii.mykhalskyi@globallogic.com>]

[-- Attachment #1.1: Type: text/plain, Size: 6194 bytes --]

Thanks to all for a replays, please see my answers below:

On 12/01/2015 05:29 PM, Wei Liu wrote:
> On Tue, Dec 01, 2015 at 04:58:55PM +0200, Iurii Mykhalskyi wrote:
>> Our real usb mass-storage device are located at driver domain (DomD). So we
>> setup second block-device backend there.
>>
>> To hotplug usb mass-storage from DomD we use follow command:
>>
>> xl block-attach domU_id phy:/bla-bla,xvda10,w,backend="DomD"
>>
> What happens if you run this in Dom0? I guess DomD doesn't respond to
> the request?
     Yes, there is no responded from domD, because actual storage device
are located there, and toolstack stuck on real device existence check.
>> There was no support of attaching block-device in runtime from domain other
>> to Domain-0, so we have made some hacks to allow call block-attach command
>> from non-dom0 privileged domain.
> So this is a special use case. This is the first time I know people
> actually run xl block-attach in driver domain.
Yes, this is special case and we this by our solution design.
>
>> One of patches was - don't update
>> /var/lib/xen/userdata-d.$DOMID-$UUID.libxl-json during execution of this
>> command (because this log located on dom0 rootfs and we don't have any
>> access to it from DomD). So, there is no different in configs before and
>> after hotplug.
>>
> The state of $DOMID is recorded in libxl-json file. No wonder you lose
> all state.
>
> But even if you write those states, they are going to be inside driver
> domain.  There is no way at the moment to synthesise the state inside
> Dom0 and DomD into one. There is also difficulty in how you can split
> the synthesised and dispatch the states to multiple entities again when
> rebuilding a domain.
>
> So I think having multiple entities managing state of one single domain
> is bad. I think the proper way of making it work is to make hotplug
> device from domain other than Dom0 work.
>
> There is a daemon "xl devd" in driver domain. We might be able to teach
> it to response to Dom0 toostack request. I'm a bit surprised if it
> doesn't do that already. Did you forget to start that daemon?
We can't run devd in driver domain, because it failed on connect to
xenstored socket (/var/run/xenstored/socket - we have xenstored running
only in dom0).
>
> Roger, Ian and Ian, any thought?
>
> Wei.

On 12/01/2015 05:41 PM, Ian Campbell wrote:
> On Tue, 2015-12-01 at 15:29 +0000, Wei Liu wrote:
>> On Tue, Dec 01, 2015 at 04:58:55PM +0200, Iurii Mykhalskyi wrote:
>>> Our real usb mass-storage device are located at driver domain (DomD).
>>> So we
>>> setup second block-device backend there.
>>>
>>> To hotplug usb mass-storage from DomD we use follow command:
>>>
>>> xl block-attach domU_id phy:/bla-bla,xvda10,w,backend="DomD"
>>>
>> What happens if you run this in Dom0? I guess DomD doesn't respond to
>> the request?
>>
>>> There was no support of attaching block-device in runtime from domain
>>> other
>>> to Domain-0, so we have made some hacks to allow call block-attach
>>> command
>>> from non-dom0 privileged domain.
>> So this is a special use case. This is the first time I know people
>> actually run xl block-attach in driver domain.
> Toolstack commands (xl *) should be run in the toolstack domain, not in the
> driver domain.
>
> I don't think it should be expected that the latter work (at least not
> without a large amount of development work).
>
> In general a driver domain would not be expected to have sufficient
> privilege over e.g. a guest domain's /local/domain/domU/devices to create
> the f.e. dirs.
In our solution we have to create 2 full privileged domains - Dom0 and
DomD, so we need 2 toolstack domains.
    Any special privileges hack wasn't done - we need just to setup
additional permissions for DomD.
>> There is a daemon "xl devd" in driver domain. We might be able to teach
>> it to response to Dom0 toostack request. I'm a bit surprised if it
>> doesn't do that already. Did you forget to start that daemon?
> That's the entire purpose of that daemon, isn't it?
I can't find any valuable documentation or examples of use for this
daemon. Can you point me please to any documentation about it, please?
    Thank you.
>
>> Roger, Ian and Ian, any thought?
>>
>> Wei.

On 12/01/2015 05:56 PM, Roger Pau Monné wrote:
> Hello,
>
> El 01/12/15 a les 15.58, Iurii Mykhalskyi ha escrit:
>> Our real usb mass-storage device are located at driver domain (DomD). So we
>> setup second block-device backend there.
>>
>> To hotplug usb mass-storage from DomD we use follow command:
>>
>> xl block-attach domU_id phy:/bla-bla,xvda10,w,backend="DomD"
> This is not possible by design, you should only be able to execute `xl
> block-attach ...` from the control domain (Dom0). This is due to the
> fact that attaching a new device to a guest requires write permissions
> in the guest xenstore paths, which the driver domain should not have.
As I mention earlier, we need this case by design of our solution. And
DomD in our most of same privilegies as Dom0,
    so access to xenstore isn't point of the problem.
>
>> There was no support of attaching block-device in runtime from domain other
>> to Domain-0, so we have made some hacks to allow call block-attach command
>> from non-dom0 privileged domain.
> Do you have either the `xl devd` command running or udev rules
> correctly setup inside of the driver domain?
Yes, we have setup our own udev rules, that executes "xl block-attach
.." during usb stick insert.
> Does something like the following work? If not, could you paste the
> error when running it with -vvv.
>
> xl block-attach DomU format=raw,vdev=hdc,access=rw,backend=DomD,target=/path/to/dev
In dom0 we have next issue:
/libxl: error: libxl_device.c:283:libxl__device_disk_set_backend: Disk
vdev=xvda10 failed to stat: /dev/sda1: No such file or directory//-
/this issue occurs due to missing /dev/sda1 device (all hardware are
placed in DomD domain).

   In domD with our patches - all ok. /dev/sda1 successfully forwarded
to DomU.

>
> Roger.
>

With the best regards, Iurii.




[-- Attachment #1.2: Type: text/html, Size: 9155 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel