Re: udev PATH_ID for virtio devices

Re: udev PATH_ID for virtio devices

[Michael S. Tsirkin]

On Thu, Mar 27, 2014 at 07:43:34PM +0100, Kay Sievers wrote:
> On Thu, Mar 27, 2014 at 6:30 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
> 
> > If the virtio device is a PCI device, it is really best to
> > treat it like you treat any other PCI function (I guess you mean
> > function and not device, right? We support multifunction
> > devices and some people do pack multiple NICs in a single device)..
> >
> > At the moment many devices in a single pci function can not happen on a
> > PCI system (no multiport) but if we add multiport, we'll follow some
> > existing standard to expose this information to the guest.
> 
> This means, that there can currently never multiple devices below one
> and the same virtio parent device?
> 
> Kay

There's a single virtio device per pci function (you keep saying
device but I hope the distinction is clear and this is
just slip of the tongue).

For net devices under a pci function that is also currently the case,
but I can't yet tell you for sure ahead of the time how we'll present
multiport devices if we ever implement them.

I'm guessing there will be multiple net devices under
a single pci device and we'll present a sysfs attribute with the port
number in this case.

Hmm maybe we should go ahead and add a place-holder
attribute so that it's future-proof?

I'll write a patch like that and we'll see how it's accepted.

-- 
MST

Re: udev PATH_ID for virtio devices

[Kay Sievers]

On Thu, Mar 27, 2014 at 7:59 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
> On Thu, Mar 27, 2014 at 07:43:34PM +0100, Kay Sievers wrote:
>> On Thu, Mar 27, 2014 at 6:30 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
>>
>> > If the virtio device is a PCI device, it is really best to
>> > treat it like you treat any other PCI function (I guess you mean
>> > function and not device, right? We support multifunction
>> > devices and some people do pack multiple NICs in a single device)..
>> >
>> > At the moment many devices in a single pci function can not happen on a
>> > PCI system (no multiport) but if we add multiport, we'll follow some
>> > existing standard to expose this information to the guest.
>>
>> This means, that there can currently never multiple devices below one
>> and the same virtio parent device?

> There's a single virtio device per pci function (you keep saying
> device but I hope the distinction is clear and this is
> just slip of the tongue).

Right, we talks about sysfs directories and they are called "device",
we don't really care about the actual bus that is implemented,
userspace does not really know much about them. :)

> For net devices under a pci function that is also currently the case,
> but I can't yet tell you for sure ahead of the time how we'll present
> multiport devices if we ever implement them.
>
> I'm guessing there will be multiple net devices under
> a single pci device and we'll present a sysfs attribute with the port
> number in this case.
>
> Hmm maybe we should go ahead and add a place-holder
> attribute so that it's future-proof?
>
> I'll write a patch like that and we'll see how it's accepted.

Netdevs with multiple ports are represented with the standard "dev_id"
attribute identifying the instance of the driver per parent "device";
should all work already from the userspace side, if the virtio side
would use that too.

Kay

Re: udev PATH_ID for virtio devices

[Michael S. Tsirkin]

On Thu, Mar 27, 2014 at 08:06:15PM +0100, Kay Sievers wrote:
> On Thu, Mar 27, 2014 at 7:59 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
> > On Thu, Mar 27, 2014 at 07:43:34PM +0100, Kay Sievers wrote:
> >> On Thu, Mar 27, 2014 at 6:30 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
> >>
> >> > If the virtio device is a PCI device, it is really best to
> >> > treat it like you treat any other PCI function (I guess you mean
> >> > function and not device, right? We support multifunction
> >> > devices and some people do pack multiple NICs in a single device)..
> >> >
> >> > At the moment many devices in a single pci function can not happen on a
> >> > PCI system (no multiport) but if we add multiport, we'll follow some
> >> > existing standard to expose this information to the guest.
> >>
> >> This means, that there can currently never multiple devices below one
> >> and the same virtio parent device?
> 
> > There's a single virtio device per pci function (you keep saying
> > device but I hope the distinction is clear and this is
> > just slip of the tongue).
> 
> Right, we talks about sysfs directories and they are called "device",
> we don't really care about the actual bus that is implemented,
> userspace does not really know much about them. :)
> 
> > For net devices under a pci function that is also currently the case,
> > but I can't yet tell you for sure ahead of the time how we'll present
> > multiport devices if we ever implement them.
> >
> > I'm guessing there will be multiple net devices under
> > a single pci device and we'll present a sysfs attribute with the port
> > number in this case.
> >
> > Hmm maybe we should go ahead and add a place-holder
> > attribute so that it's future-proof?
> >
> > I'll write a patch like that and we'll see how it's accepted.
> 
> Netdevs with multiple ports are represented with the standard "dev_id"
> attribute identifying the instance of the driver per parent "device";
> should all work already from the userspace side, if the virtio side
> would use that too.
> 
> Kay

Aha. In that case it's easy - pls assume that if and when we implement
multiple we'll just follow standards and use dev_id.
For virtio pci devices specifically virtio<->pci 1:1 mapping is set
in stone in the spec.

Non pci ones need to be examined separately.

Re: udev PATH_ID for virtio devices

[Kay Sievers]

On Thu, Mar 27, 2014 at 8:13 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
> On Thu, Mar 27, 2014 at 08:06:15PM +0100, Kay Sievers wrote:
>> On Thu, Mar 27, 2014 at 7:59 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
>> > On Thu, Mar 27, 2014 at 07:43:34PM +0100, Kay Sievers wrote:
>> >> On Thu, Mar 27, 2014 at 6:30 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
>> >>
>> >> > If the virtio device is a PCI device, it is really best to
>> >> > treat it like you treat any other PCI function (I guess you mean
>> >> > function and not device, right? We support multifunction
>> >> > devices and some people do pack multiple NICs in a single device)..
>> >> >
>> >> > At the moment many devices in a single pci function can not happen on a
>> >> > PCI system (no multiport) but if we add multiport, we'll follow some
>> >> > existing standard to expose this information to the guest.
>> >>
>> >> This means, that there can currently never multiple devices below one
>> >> and the same virtio parent device?
>>
>> > There's a single virtio device per pci function (you keep saying
>> > device but I hope the distinction is clear and this is
>> > just slip of the tongue).
>>
>> Right, we talks about sysfs directories and they are called "device",
>> we don't really care about the actual bus that is implemented,
>> userspace does not really know much about them. :)
>>
>> > For net devices under a pci function that is also currently the case,
>> > but I can't yet tell you for sure ahead of the time how we'll present
>> > multiport devices if we ever implement them.
>> >
>> > I'm guessing there will be multiple net devices under
>> > a single pci device and we'll present a sysfs attribute with the port
>> > number in this case.
>> >
>> > Hmm maybe we should go ahead and add a place-holder
>> > attribute so that it's future-proof?
>> >
>> > I'll write a patch like that and we'll see how it's accepted.
>>
>> Netdevs with multiple ports are represented with the standard "dev_id"
>> attribute identifying the instance of the driver per parent "device";
>> should all work already from the userspace side, if the virtio side
>> would use that too.
>>
>> Kay
>
> Aha. In that case it's easy - pls assume that if and when we implement
> multiple we'll just follow standards and use dev_id.
> For virtio pci devices specifically virtio<->pci 1:1 mapping is set
> in stone in the spec.
>
> Non pci ones need to be examined separately.

Nice, thanks. That means we can just "jump over" the "virtio*" device
(device as in sysfs view), and let the pci parent let identify the
device.

Background: The logic for device naming generally refuses to touch
devices with unknown parent devices in the chain (the directories you
see for: ls -l /sys/class/net/), because the parents *could* offer
their own bus logic that exposes multiple devices below and we would
create clashing names for them. That's why the virt vs. non-virt case
needs to be handled explicitly (seems in this case by just skipping
it) and it is not necessarily by default the same behaviour.

Kay

Re: udev PATH_ID for virtio devices

[Tom Gundersen]

On Thu, Mar 27, 2014 at 8:25 PM, Kay Sievers <kay@vrfy.org> wrote:
> On Thu, Mar 27, 2014 at 8:13 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
>> On Thu, Mar 27, 2014 at 08:06:15PM +0100, Kay Sievers wrote:
>>> On Thu, Mar 27, 2014 at 7:59 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
>>> > On Thu, Mar 27, 2014 at 07:43:34PM +0100, Kay Sievers wrote:
>>> >> On Thu, Mar 27, 2014 at 6:30 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
>>> >>
>>> >> > If the virtio device is a PCI device, it is really best to
>>> >> > treat it like you treat any other PCI function (I guess you mean
>>> >> > function and not device, right? We support multifunction
>>> >> > devices and some people do pack multiple NICs in a single device)..
>>> >> >
>>> >> > At the moment many devices in a single pci function can not happen on a
>>> >> > PCI system (no multiport) but if we add multiport, we'll follow some
>>> >> > existing standard to expose this information to the guest.
>>> >>
>>> >> This means, that there can currently never multiple devices below one
>>> >> and the same virtio parent device?
>>>
>>> > There's a single virtio device per pci function (you keep saying
>>> > device but I hope the distinction is clear and this is
>>> > just slip of the tongue).
>>>
>>> Right, we talks about sysfs directories and they are called "device",
>>> we don't really care about the actual bus that is implemented,
>>> userspace does not really know much about them. :)
>>>
>>> > For net devices under a pci function that is also currently the case,
>>> > but I can't yet tell you for sure ahead of the time how we'll present
>>> > multiport devices if we ever implement them.
>>> >
>>> > I'm guessing there will be multiple net devices under
>>> > a single pci device and we'll present a sysfs attribute with the port
>>> > number in this case.
>>> >
>>> > Hmm maybe we should go ahead and add a place-holder
>>> > attribute so that it's future-proof?
>>> >
>>> > I'll write a patch like that and we'll see how it's accepted.
>>>
>>> Netdevs with multiple ports are represented with the standard "dev_id"
>>> attribute identifying the instance of the driver per parent "device";
>>> should all work already from the userspace side, if the virtio side
>>> would use that too.
>>>
>>> Kay
>>
>> Aha. In that case it's easy - pls assume that if and when we implement
>> multiple we'll just follow standards and use dev_id.
>> For virtio pci devices specifically virtio<->pci 1:1 mapping is set
>> in stone in the spec.
>>
>> Non pci ones need to be examined separately.
>
> Nice, thanks. That means we can just "jump over" the "virtio*" device
> (device as in sysfs view), and let the pci parent let identify the
> device.
>
> Background: The logic for device naming generally refuses to touch
> devices with unknown parent devices in the chain (the directories you
> see for: ls -l /sys/class/net/), because the parents *could* offer
> their own bus logic that exposes multiple devices below and we would
> create clashing names for them. That's why the virt vs. non-virt case
> needs to be handled explicitly (seems in this case by just skipping
> it) and it is not necessarily by default the same behaviour.

So Kay and I discussed this a bit more, and found that we likely
cannot handle virtio nic's nicely. As we would name them based on the
pci geo, we rely on this being stable between reboots and when
adding/removing hardware. Is there some way to make this work with
virtio, or will the 'fake' pci busses simply be enumerated as they are
created?

Cheers,

Tom

Re: udev PATH_ID for virtio devices

[Michael S. Tsirkin]

On Thu, Mar 27, 2014 at 09:24:10PM +0100, Tom Gundersen wrote:
> On Thu, Mar 27, 2014 at 8:25 PM, Kay Sievers <kay@vrfy.org> wrote:
> > On Thu, Mar 27, 2014 at 8:13 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
> >> On Thu, Mar 27, 2014 at 08:06:15PM +0100, Kay Sievers wrote:
> >>> On Thu, Mar 27, 2014 at 7:59 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
> >>> > On Thu, Mar 27, 2014 at 07:43:34PM +0100, Kay Sievers wrote:
> >>> >> On Thu, Mar 27, 2014 at 6:30 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
> >>> >>
> >>> >> > If the virtio device is a PCI device, it is really best to
> >>> >> > treat it like you treat any other PCI function (I guess you mean
> >>> >> > function and not device, right? We support multifunction
> >>> >> > devices and some people do pack multiple NICs in a single device)..
> >>> >> >
> >>> >> > At the moment many devices in a single pci function can not happen on a
> >>> >> > PCI system (no multiport) but if we add multiport, we'll follow some
> >>> >> > existing standard to expose this information to the guest.
> >>> >>
> >>> >> This means, that there can currently never multiple devices below one
> >>> >> and the same virtio parent device?
> >>>
> >>> > There's a single virtio device per pci function (you keep saying
> >>> > device but I hope the distinction is clear and this is
> >>> > just slip of the tongue).
> >>>
> >>> Right, we talks about sysfs directories and they are called "device",
> >>> we don't really care about the actual bus that is implemented,
> >>> userspace does not really know much about them. :)
> >>>
> >>> > For net devices under a pci function that is also currently the case,
> >>> > but I can't yet tell you for sure ahead of the time how we'll present
> >>> > multiport devices if we ever implement them.
> >>> >
> >>> > I'm guessing there will be multiple net devices under
> >>> > a single pci device and we'll present a sysfs attribute with the port
> >>> > number in this case.
> >>> >
> >>> > Hmm maybe we should go ahead and add a place-holder
> >>> > attribute so that it's future-proof?
> >>> >
> >>> > I'll write a patch like that and we'll see how it's accepted.
> >>>
> >>> Netdevs with multiple ports are represented with the standard "dev_id"
> >>> attribute identifying the instance of the driver per parent "device";
> >>> should all work already from the userspace side, if the virtio side
> >>> would use that too.
> >>>
> >>> Kay
> >>
> >> Aha. In that case it's easy - pls assume that if and when we implement
> >> multiple we'll just follow standards and use dev_id.
> >> For virtio pci devices specifically virtio<->pci 1:1 mapping is set
> >> in stone in the spec.
> >>
> >> Non pci ones need to be examined separately.
> >
> > Nice, thanks. That means we can just "jump over" the "virtio*" device
> > (device as in sysfs view), and let the pci parent let identify the
> > device.
> >
> > Background: The logic for device naming generally refuses to touch
> > devices with unknown parent devices in the chain (the directories you
> > see for: ls -l /sys/class/net/), because the parents *could* offer
> > their own bus logic that exposes multiple devices below and we would
> > create clashing names for them. That's why the virt vs. non-virt case
> > needs to be handled explicitly (seems in this case by just skipping
> > it) and it is not necessarily by default the same behaviour.
> 
> So Kay and I discussed this a bit more, and found that we likely
> cannot handle virtio nic's nicely. As we would name them based on the
> pci geo, we rely on this being stable between reboots and when
> adding/removing hardware. Is there some way to make this work with
> virtio, or will the 'fake' pci busses simply be enumerated as they are
> created?
> 
> Cheers,
> 
> Tom

I think it's standard PCI thing.
Some pci bridges have a slot id register - I'm assuming you
are using these if present?

If not you'll either need to rely on firmware enumerating buses
consistently, or use the mac of the NIC.

-- 
MST

Re: udev PATH_ID for virtio devices

[Tom Gundersen]

On Thu, Mar 27, 2014 at 9:58 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
> On Thu, Mar 27, 2014 at 09:24:10PM +0100, Tom Gundersen wrote:
>> On Thu, Mar 27, 2014 at 8:25 PM, Kay Sievers <kay@vrfy.org> wrote:
>> > On Thu, Mar 27, 2014 at 8:13 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
>> >> On Thu, Mar 27, 2014 at 08:06:15PM +0100, Kay Sievers wrote:
>> >>> On Thu, Mar 27, 2014 at 7:59 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
>> >>> > On Thu, Mar 27, 2014 at 07:43:34PM +0100, Kay Sievers wrote:
>> >>> >> On Thu, Mar 27, 2014 at 6:30 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
>> >>> >>
>> >>> >> > If the virtio device is a PCI device, it is really best to
>> >>> >> > treat it like you treat any other PCI function (I guess you mean
>> >>> >> > function and not device, right? We support multifunction
>> >>> >> > devices and some people do pack multiple NICs in a single device)..
>> >>> >> >
>> >>> >> > At the moment many devices in a single pci function can not happen on a
>> >>> >> > PCI system (no multiport) but if we add multiport, we'll follow some
>> >>> >> > existing standard to expose this information to the guest.
>> >>> >>
>> >>> >> This means, that there can currently never multiple devices below one
>> >>> >> and the same virtio parent device?
>> >>>
>> >>> > There's a single virtio device per pci function (you keep saying
>> >>> > device but I hope the distinction is clear and this is
>> >>> > just slip of the tongue).
>> >>>
>> >>> Right, we talks about sysfs directories and they are called "device",
>> >>> we don't really care about the actual bus that is implemented,
>> >>> userspace does not really know much about them. :)
>> >>>
>> >>> > For net devices under a pci function that is also currently the case,
>> >>> > but I can't yet tell you for sure ahead of the time how we'll present
>> >>> > multiport devices if we ever implement them.
>> >>> >
>> >>> > I'm guessing there will be multiple net devices under
>> >>> > a single pci device and we'll present a sysfs attribute with the port
>> >>> > number in this case.
>> >>> >
>> >>> > Hmm maybe we should go ahead and add a place-holder
>> >>> > attribute so that it's future-proof?
>> >>> >
>> >>> > I'll write a patch like that and we'll see how it's accepted.
>> >>>
>> >>> Netdevs with multiple ports are represented with the standard "dev_id"
>> >>> attribute identifying the instance of the driver per parent "device";
>> >>> should all work already from the userspace side, if the virtio side
>> >>> would use that too.
>> >>>
>> >>> Kay
>> >>
>> >> Aha. In that case it's easy - pls assume that if and when we implement
>> >> multiple we'll just follow standards and use dev_id.
>> >> For virtio pci devices specifically virtio<->pci 1:1 mapping is set
>> >> in stone in the spec.
>> >>
>> >> Non pci ones need to be examined separately.
>> >
>> > Nice, thanks. That means we can just "jump over" the "virtio*" device
>> > (device as in sysfs view), and let the pci parent let identify the
>> > device.
>> >
>> > Background: The logic for device naming generally refuses to touch
>> > devices with unknown parent devices in the chain (the directories you
>> > see for: ls -l /sys/class/net/), because the parents *could* offer
>> > their own bus logic that exposes multiple devices below and we would
>> > create clashing names for them. That's why the virt vs. non-virt case
>> > needs to be handled explicitly (seems in this case by just skipping
>> > it) and it is not necessarily by default the same behaviour.
>>
>> So Kay and I discussed this a bit more, and found that we likely
>> cannot handle virtio nic's nicely. As we would name them based on the
>> pci geo, we rely on this being stable between reboots and when
>> adding/removing hardware. Is there some way to make this work with
>> virtio, or will the 'fake' pci busses simply be enumerated as they are
>> created?
>>
>> Cheers,
>>
>> Tom
>
> I think it's standard PCI thing.
> Some pci bridges have a slot id register - I'm assuming you
> are using these if present?
>
> If not you'll either need to rely on firmware enumerating buses
> consistently, or use the mac of the NIC.

Hi Michael,

Sorry to resurrect such an old thread, but this suddenly became
relevant again and I see that we never really reached a conclusion.
Rereading the discussion it is still not really clear to me what the
API promise is, so let me try to rephrase what we need and hopefully
you can help me understand where we are at.

My understanding is that the enumeration of virtioX buses is global
rather than local to the direct parent device of a given bus. That
means that we cannot base deterministic naming of child devices
(functions) on virtio enumeration, as the names would then depend on
the presence or absence of unrelated virtio buses on the system.

We have two options that would allow us to get around this:

1) if there is a guarantee now and in the future (even if only
restricted to netdevs) that no virtio bus will have a direct sibling
bus (i.e., with the same parent device); or
2) if the enumeration of virtio buses was extended with a "port"
numebr which was guaranteed to be made local to their direct parent
device and moreover be made deterministic. I.e., if there are no
virtio sibling buses (but there may be other unrelated virtio buses)
the name would always be virtioX.0, and if there are two buses with
the same parent their names would always be virtioX.0 and virtioX.1
and always in the same order.

Is either of those things something that is already guaranteed or
something that could be made working?

Cheers,

Tom

Re: udev PATH_ID for virtio devices

[Michael S. Tsirkin]

On Mon, Aug 24, 2015 at 06:10:01PM +0200, Tom Gundersen wrote:
> On Thu, Mar 27, 2014 at 9:58 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
> > On Thu, Mar 27, 2014 at 09:24:10PM +0100, Tom Gundersen wrote:
> >> On Thu, Mar 27, 2014 at 8:25 PM, Kay Sievers <kay@vrfy.org> wrote:
> >> > On Thu, Mar 27, 2014 at 8:13 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
> >> >> On Thu, Mar 27, 2014 at 08:06:15PM +0100, Kay Sievers wrote:
> >> >>> On Thu, Mar 27, 2014 at 7:59 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
> >> >>> > On Thu, Mar 27, 2014 at 07:43:34PM +0100, Kay Sievers wrote:
> >> >>> >> On Thu, Mar 27, 2014 at 6:30 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
> >> >>> >>
> >> >>> >> > If the virtio device is a PCI device, it is really best to
> >> >>> >> > treat it like you treat any other PCI function (I guess you mean
> >> >>> >> > function and not device, right? We support multifunction
> >> >>> >> > devices and some people do pack multiple NICs in a single device)..
> >> >>> >> >
> >> >>> >> > At the moment many devices in a single pci function can not happen on a
> >> >>> >> > PCI system (no multiport) but if we add multiport, we'll follow some
> >> >>> >> > existing standard to expose this information to the guest.
> >> >>> >>
> >> >>> >> This means, that there can currently never multiple devices below one
> >> >>> >> and the same virtio parent device?
> >> >>>
> >> >>> > There's a single virtio device per pci function (you keep saying
> >> >>> > device but I hope the distinction is clear and this is
> >> >>> > just slip of the tongue).
> >> >>>
> >> >>> Right, we talks about sysfs directories and they are called "device",
> >> >>> we don't really care about the actual bus that is implemented,
> >> >>> userspace does not really know much about them. :)
> >> >>>
> >> >>> > For net devices under a pci function that is also currently the case,
> >> >>> > but I can't yet tell you for sure ahead of the time how we'll present
> >> >>> > multiport devices if we ever implement them.
> >> >>> >
> >> >>> > I'm guessing there will be multiple net devices under
> >> >>> > a single pci device and we'll present a sysfs attribute with the port
> >> >>> > number in this case.
> >> >>> >
> >> >>> > Hmm maybe we should go ahead and add a place-holder
> >> >>> > attribute so that it's future-proof?
> >> >>> >
> >> >>> > I'll write a patch like that and we'll see how it's accepted.
> >> >>>
> >> >>> Netdevs with multiple ports are represented with the standard "dev_id"
> >> >>> attribute identifying the instance of the driver per parent "device";
> >> >>> should all work already from the userspace side, if the virtio side
> >> >>> would use that too.
> >> >>>
> >> >>> Kay
> >> >>
> >> >> Aha. In that case it's easy - pls assume that if and when we implement
> >> >> multiple we'll just follow standards and use dev_id.
> >> >> For virtio pci devices specifically virtio<->pci 1:1 mapping is set
> >> >> in stone in the spec.
> >> >>
> >> >> Non pci ones need to be examined separately.
> >> >
> >> > Nice, thanks. That means we can just "jump over" the "virtio*" device
> >> > (device as in sysfs view), and let the pci parent let identify the
> >> > device.
> >> >
> >> > Background: The logic for device naming generally refuses to touch
> >> > devices with unknown parent devices in the chain (the directories you
> >> > see for: ls -l /sys/class/net/), because the parents *could* offer
> >> > their own bus logic that exposes multiple devices below and we would
> >> > create clashing names for them. That's why the virt vs. non-virt case
> >> > needs to be handled explicitly (seems in this case by just skipping
> >> > it) and it is not necessarily by default the same behaviour.
> >>
> >> So Kay and I discussed this a bit more, and found that we likely
> >> cannot handle virtio nic's nicely. As we would name them based on the
> >> pci geo, we rely on this being stable between reboots and when
> >> adding/removing hardware. Is there some way to make this work with
> >> virtio, or will the 'fake' pci busses simply be enumerated as they are
> >> created?
> >>
> >> Cheers,
> >>
> >> Tom
> >
> > I think it's standard PCI thing.
> > Some pci bridges have a slot id register - I'm assuming you
> > are using these if present?
> >
> > If not you'll either need to rely on firmware enumerating buses
> > consistently, or use the mac of the NIC.
> 
> Hi Michael,
> 
> Sorry to resurrect such an old thread, but this suddenly became
> relevant again and I see that we never really reached a conclusion.
> Rereading the discussion it is still not really clear to me what the
> API promise is, so let me try to rephrase what we need and hopefully
> you can help me understand where we are at.
> 
> My understanding is that the enumeration of virtioX buses is global
> rather than local to the direct parent device of a given bus. That
> means that we cannot base deterministic naming of child devices
> (functions) on virtio enumeration, as the names would then depend on
> the presence or absence of unrelated virtio buses on the system.
> 
> We have two options that would allow us to get around this:
> 
> 1) if there is a guarantee now and in the future (even if only
> restricted to netdevs) that no virtio bus will have a direct sibling
> bus (i.e., with the same parent device); or

I think this is the case. The virtio bus is an artifact.
There's always a single one behind each pci device.
Is this sufficient?

> 2) if the enumeration of virtio buses was extended with a "port"
> numebr which was guaranteed to be made local to their direct parent
> device and moreover be made deterministic. I.e., if there are no
> virtio sibling buses (but there may be other unrelated virtio buses)
> the name would always be virtioX.0, and if there are two buses with
> the same parent their names would always be virtioX.0 and virtioX.1
> and always in the same order.
> 
> Is either of those things something that is already guaranteed or
> something that could be made working?
> 
> Cheers,
> 
> Tom

Re: udev PATH_ID for virtio devices

[Kay Sievers]

On Tue, Aug 25, 2015 at 1:43 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
> On Mon, Aug 24, 2015 at 06:10:01PM +0200, Tom Gundersen wrote:
>> On Thu, Mar 27, 2014 at 9:58 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
>> > On Thu, Mar 27, 2014 at 09:24:10PM +0100, Tom Gundersen wrote:
>> >> On Thu, Mar 27, 2014 at 8:25 PM, Kay Sievers <kay@vrfy.org> wrote:
>> >> > On Thu, Mar 27, 2014 at 8:13 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
>> >> >> On Thu, Mar 27, 2014 at 08:06:15PM +0100, Kay Sievers wrote:
>> >> >>> On Thu, Mar 27, 2014 at 7:59 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
>> >> >>> > On Thu, Mar 27, 2014 at 07:43:34PM +0100, Kay Sievers wrote:
>> >> >>> >> On Thu, Mar 27, 2014 at 6:30 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
>> >> >>> >>
>> >> >>> >> > If the virtio device is a PCI device, it is really best to
>> >> >>> >> > treat it like you treat any other PCI function (I guess you mean
>> >> >>> >> > function and not device, right? We support multifunction
>> >> >>> >> > devices and some people do pack multiple NICs in a single device)..
>> >> >>> >> >
>> >> >>> >> > At the moment many devices in a single pci function can not happen on a
>> >> >>> >> > PCI system (no multiport) but if we add multiport, we'll follow some
>> >> >>> >> > existing standard to expose this information to the guest.
>> >> >>> >>
>> >> >>> >> This means, that there can currently never multiple devices below one
>> >> >>> >> and the same virtio parent device?
>> >> >>>
>> >> >>> > There's a single virtio device per pci function (you keep saying
>> >> >>> > device but I hope the distinction is clear and this is
>> >> >>> > just slip of the tongue).
>> >> >>>
>> >> >>> Right, we talks about sysfs directories and they are called "device",
>> >> >>> we don't really care about the actual bus that is implemented,
>> >> >>> userspace does not really know much about them. :)
>> >> >>>
>> >> >>> > For net devices under a pci function that is also currently the case,
>> >> >>> > but I can't yet tell you for sure ahead of the time how we'll present
>> >> >>> > multiport devices if we ever implement them.
>> >> >>> >
>> >> >>> > I'm guessing there will be multiple net devices under
>> >> >>> > a single pci device and we'll present a sysfs attribute with the port
>> >> >>> > number in this case.
>> >> >>> >
>> >> >>> > Hmm maybe we should go ahead and add a place-holder
>> >> >>> > attribute so that it's future-proof?
>> >> >>> >
>> >> >>> > I'll write a patch like that and we'll see how it's accepted.
>> >> >>>
>> >> >>> Netdevs with multiple ports are represented with the standard "dev_id"
>> >> >>> attribute identifying the instance of the driver per parent "device";
>> >> >>> should all work already from the userspace side, if the virtio side
>> >> >>> would use that too.
>> >> >>>
>> >> >>> Kay
>> >> >>
>> >> >> Aha. In that case it's easy - pls assume that if and when we implement
>> >> >> multiple we'll just follow standards and use dev_id.
>> >> >> For virtio pci devices specifically virtio<->pci 1:1 mapping is set
>> >> >> in stone in the spec.
>> >> >>
>> >> >> Non pci ones need to be examined separately.
>> >> >
>> >> > Nice, thanks. That means we can just "jump over" the "virtio*" device
>> >> > (device as in sysfs view), and let the pci parent let identify the
>> >> > device.
>> >> >
>> >> > Background: The logic for device naming generally refuses to touch
>> >> > devices with unknown parent devices in the chain (the directories you
>> >> > see for: ls -l /sys/class/net/), because the parents *could* offer
>> >> > their own bus logic that exposes multiple devices below and we would
>> >> > create clashing names for them. That's why the virt vs. non-virt case
>> >> > needs to be handled explicitly (seems in this case by just skipping
>> >> > it) and it is not necessarily by default the same behaviour.
>> >>
>> >> So Kay and I discussed this a bit more, and found that we likely
>> >> cannot handle virtio nic's nicely. As we would name them based on the
>> >> pci geo, we rely on this being stable between reboots and when
>> >> adding/removing hardware. Is there some way to make this work with
>> >> virtio, or will the 'fake' pci busses simply be enumerated as they are
>> >> created?
>> >>
>> >> Cheers,
>> >>
>> >> Tom
>> >
>> > I think it's standard PCI thing.
>> > Some pci bridges have a slot id register - I'm assuming you
>> > are using these if present?
>> >
>> > If not you'll either need to rely on firmware enumerating buses
>> > consistently, or use the mac of the NIC.
>>
>> Hi Michael,
>>
>> Sorry to resurrect such an old thread, but this suddenly became
>> relevant again and I see that we never really reached a conclusion.
>> Rereading the discussion it is still not really clear to me what the
>> API promise is, so let me try to rephrase what we need and hopefully
>> you can help me understand where we are at.
>>
>> My understanding is that the enumeration of virtioX buses is global
>> rather than local to the direct parent device of a given bus. That
>> means that we cannot base deterministic naming of child devices
>> (functions) on virtio enumeration, as the names would then depend on
>> the presence or absence of unrelated virtio buses on the system.
>>
>> We have two options that would allow us to get around this:
>>
>> 1) if there is a guarantee now and in the future (even if only
>> restricted to netdevs) that no virtio bus will have a direct sibling
>> bus (i.e., with the same parent device); or
>
> I think this is the case. The virtio bus is an artifact.
> There's always a single one behind each pci device.
> Is this sufficient?

I *think* is not good enough for udev to offer such functionality.

We need an authoritative answer that this cannot happen with today's
code, and also that there are no plans to ever make multiple virtio
devices per parent device.

Thanks,
Kay

Re: udev PATH_ID for virtio devices

[Michael S. Tsirkin]

On Tue, Aug 25, 2015 at 01:46:49PM +0200, Kay Sievers wrote:
> On Tue, Aug 25, 2015 at 1:43 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
> > On Mon, Aug 24, 2015 at 06:10:01PM +0200, Tom Gundersen wrote:
> >> On Thu, Mar 27, 2014 at 9:58 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
> >> > On Thu, Mar 27, 2014 at 09:24:10PM +0100, Tom Gundersen wrote:
> >> >> On Thu, Mar 27, 2014 at 8:25 PM, Kay Sievers <kay@vrfy.org> wrote:
> >> >> > On Thu, Mar 27, 2014 at 8:13 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
> >> >> >> On Thu, Mar 27, 2014 at 08:06:15PM +0100, Kay Sievers wrote:
> >> >> >>> On Thu, Mar 27, 2014 at 7:59 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
> >> >> >>> > On Thu, Mar 27, 2014 at 07:43:34PM +0100, Kay Sievers wrote:
> >> >> >>> >> On Thu, Mar 27, 2014 at 6:30 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
> >> >> >>> >>
> >> >> >>> >> > If the virtio device is a PCI device, it is really best to
> >> >> >>> >> > treat it like you treat any other PCI function (I guess you mean
> >> >> >>> >> > function and not device, right? We support multifunction
> >> >> >>> >> > devices and some people do pack multiple NICs in a single device)..
> >> >> >>> >> >
> >> >> >>> >> > At the moment many devices in a single pci function can not happen on a
> >> >> >>> >> > PCI system (no multiport) but if we add multiport, we'll follow some
> >> >> >>> >> > existing standard to expose this information to the guest.
> >> >> >>> >>
> >> >> >>> >> This means, that there can currently never multiple devices below one
> >> >> >>> >> and the same virtio parent device?
> >> >> >>>
> >> >> >>> > There's a single virtio device per pci function (you keep saying
> >> >> >>> > device but I hope the distinction is clear and this is
> >> >> >>> > just slip of the tongue).
> >> >> >>>
> >> >> >>> Right, we talks about sysfs directories and they are called "device",
> >> >> >>> we don't really care about the actual bus that is implemented,
> >> >> >>> userspace does not really know much about them. :)
> >> >> >>>
> >> >> >>> > For net devices under a pci function that is also currently the case,
> >> >> >>> > but I can't yet tell you for sure ahead of the time how we'll present
> >> >> >>> > multiport devices if we ever implement them.
> >> >> >>> >
> >> >> >>> > I'm guessing there will be multiple net devices under
> >> >> >>> > a single pci device and we'll present a sysfs attribute with the port
> >> >> >>> > number in this case.
> >> >> >>> >
> >> >> >>> > Hmm maybe we should go ahead and add a place-holder
> >> >> >>> > attribute so that it's future-proof?
> >> >> >>> >
> >> >> >>> > I'll write a patch like that and we'll see how it's accepted.
> >> >> >>>
> >> >> >>> Netdevs with multiple ports are represented with the standard "dev_id"
> >> >> >>> attribute identifying the instance of the driver per parent "device";
> >> >> >>> should all work already from the userspace side, if the virtio side
> >> >> >>> would use that too.
> >> >> >>>
> >> >> >>> Kay
> >> >> >>
> >> >> >> Aha. In that case it's easy - pls assume that if and when we implement
> >> >> >> multiple we'll just follow standards and use dev_id.
> >> >> >> For virtio pci devices specifically virtio<->pci 1:1 mapping is set
> >> >> >> in stone in the spec.
> >> >> >>
> >> >> >> Non pci ones need to be examined separately.
> >> >> >
> >> >> > Nice, thanks. That means we can just "jump over" the "virtio*" device
> >> >> > (device as in sysfs view), and let the pci parent let identify the
> >> >> > device.
> >> >> >
> >> >> > Background: The logic for device naming generally refuses to touch
> >> >> > devices with unknown parent devices in the chain (the directories you
> >> >> > see for: ls -l /sys/class/net/), because the parents *could* offer
> >> >> > their own bus logic that exposes multiple devices below and we would
> >> >> > create clashing names for them. That's why the virt vs. non-virt case
> >> >> > needs to be handled explicitly (seems in this case by just skipping
> >> >> > it) and it is not necessarily by default the same behaviour.
> >> >>
> >> >> So Kay and I discussed this a bit more, and found that we likely
> >> >> cannot handle virtio nic's nicely. As we would name them based on the
> >> >> pci geo, we rely on this being stable between reboots and when
> >> >> adding/removing hardware. Is there some way to make this work with
> >> >> virtio, or will the 'fake' pci busses simply be enumerated as they are
> >> >> created?
> >> >>
> >> >> Cheers,
> >> >>
> >> >> Tom
> >> >
> >> > I think it's standard PCI thing.
> >> > Some pci bridges have a slot id register - I'm assuming you
> >> > are using these if present?
> >> >
> >> > If not you'll either need to rely on firmware enumerating buses
> >> > consistently, or use the mac of the NIC.
> >>
> >> Hi Michael,
> >>
> >> Sorry to resurrect such an old thread, but this suddenly became
> >> relevant again and I see that we never really reached a conclusion.
> >> Rereading the discussion it is still not really clear to me what the
> >> API promise is, so let me try to rephrase what we need and hopefully
> >> you can help me understand where we are at.
> >>
> >> My understanding is that the enumeration of virtioX buses is global
> >> rather than local to the direct parent device of a given bus. That
> >> means that we cannot base deterministic naming of child devices
> >> (functions) on virtio enumeration, as the names would then depend on
> >> the presence or absence of unrelated virtio buses on the system.
> >>
> >> We have two options that would allow us to get around this:
> >>
> >> 1) if there is a guarantee now and in the future (even if only
> >> restricted to netdevs) that no virtio bus will have a direct sibling
> >> bus (i.e., with the same parent device); or
> >
> > I think this is the case. The virtio bus is an artifact.
> > There's always a single one behind each pci device.
> > Is this sufficient?
> 
> I *think* is not good enough for udev to offer such functionality.
> 
> We need an authoritative answer that this cannot happen with today's
> code, and also that there are no plans to ever make multiple virtio
> devices per parent device.
> 
> Thanks,
> Kay

But if virtio will make such a promise, will that be sufficient?

-- 
MST

Re: udev PATH_ID for virtio devices

[Kay Sievers]

On Tue, Aug 25, 2015 at 1:52 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
> On Tue, Aug 25, 2015 at 01:46:49PM +0200, Kay Sievers wrote:
>> On Tue, Aug 25, 2015 at 1:43 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
>> > On Mon, Aug 24, 2015 at 06:10:01PM +0200, Tom Gundersen wrote:

>> >> 1) if there is a guarantee now and in the future (even if only
>> >> restricted to netdevs) that no virtio bus will have a direct sibling
>> >> bus (i.e., with the same parent device); or
>> >
>> > I think this is the case. The virtio bus is an artifact.
>> > There's always a single one behind each pci device.
>> > Is this sufficient?
>>
>> I *think* is not good enough for udev to offer such functionality.
>>
>> We need an authoritative answer that this cannot happen with today's
>> code, and also that there are no plans to ever make multiple virtio
>> devices per parent device.

> But if virtio will make such a promise, will that be sufficient?

Sure. I mean we need to be reasonable sure, that there will never
be more than one virtio device per parent device. Otherwise an
innocent looking kernel change will just break existing userspace
setups which made assumptions which would then not be true anymore.

Kay

Re: udev PATH_ID for virtio devices

[Tom Gundersen]

On Tue, Aug 25, 2015 at 1:58 PM, Kay Sievers <kay@vrfy.org> wrote:
> On Tue, Aug 25, 2015 at 1:52 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
>> On Tue, Aug 25, 2015 at 01:46:49PM +0200, Kay Sievers wrote:
>>> On Tue, Aug 25, 2015 at 1:43 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
>>> > On Mon, Aug 24, 2015 at 06:10:01PM +0200, Tom Gundersen wrote:
>
>>> >> 1) if there is a guarantee now and in the future (even if only
>>> >> restricted to netdevs) that no virtio bus will have a direct sibling
>>> >> bus (i.e., with the same parent device); or
>>> >
>>> > I think this is the case. The virtio bus is an artifact.
>>> > There's always a single one behind each pci device.
>>> > Is this sufficient?
>>>
>>> I *think* is not good enough for udev to offer such functionality.
>>>
>>> We need an authoritative answer that this cannot happen with today's
>>> code, and also that there are no plans to ever make multiple virtio
>>> devices per parent device.
>
>> But if virtio will make such a promise, will that be sufficient?
>
> Sure.

Any chance we could get such a guarantee? Anything else needed from
our side? I have a pending patch that would be nice to get out, which
would depend on such a guarantee.

Cheers,

Tom

Re: udev PATH_ID for virtio devices

[Michael S. Tsirkin]

On Wed, Sep 02, 2015 at 01:53:33PM +0200, Tom Gundersen wrote:
> On Tue, Aug 25, 2015 at 1:58 PM, Kay Sievers <kay@vrfy.org> wrote:
> > On Tue, Aug 25, 2015 at 1:52 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
> >> On Tue, Aug 25, 2015 at 01:46:49PM +0200, Kay Sievers wrote:
> >>> On Tue, Aug 25, 2015 at 1:43 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
> >>> > On Mon, Aug 24, 2015 at 06:10:01PM +0200, Tom Gundersen wrote:
> >
> >>> >> 1) if there is a guarantee now and in the future (even if only
> >>> >> restricted to netdevs) that no virtio bus will have a direct sibling
> >>> >> bus (i.e., with the same parent device); or
> >>> >
> >>> > I think this is the case. The virtio bus is an artifact.
> >>> > There's always a single one behind each pci device.
> >>> > Is this sufficient?
> >>>
> >>> I *think* is not good enough for udev to offer such functionality.
> >>>
> >>> We need an authoritative answer that this cannot happen with today's
> >>> code, and also that there are no plans to ever make multiple virtio
> >>> devices per parent device.
> >
> >> But if virtio will make such a promise, will that be sufficient?
> >
> > Sure.
> 
> Any chance we could get such a guarantee? Anything else needed from
> our side? I have a pending patch that would be nice to get out, which
> would depend on such a guarantee.
> 
> Cheers,
> 
> Tom

I thought about this. The virtio spec makes it explicit for pci,
ccw and mmio devices. I think we can make this promise - if we
wanted to have many such buses, we can always make it something else,
not a virtio bus.

So please just check the type of the bus - if it's virtio,
you can assume there is no sibling.

-- 
MST

Re: udev PATH_ID for virtio devices

[Tom Gundersen]

On Wed, Sep 2, 2015 at 2:09 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
> On Wed, Sep 02, 2015 at 01:53:33PM +0200, Tom Gundersen wrote:
>> On Tue, Aug 25, 2015 at 1:58 PM, Kay Sievers <kay@vrfy.org> wrote:
>> > On Tue, Aug 25, 2015 at 1:52 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
>> >> On Tue, Aug 25, 2015 at 01:46:49PM +0200, Kay Sievers wrote:
>> >>> On Tue, Aug 25, 2015 at 1:43 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
>> >>> > On Mon, Aug 24, 2015 at 06:10:01PM +0200, Tom Gundersen wrote:
>> >
>> >>> >> 1) if there is a guarantee now and in the future (even if only
>> >>> >> restricted to netdevs) that no virtio bus will have a direct sibling
>> >>> >> bus (i.e., with the same parent device); or
>> >>> >
>> >>> > I think this is the case. The virtio bus is an artifact.
>> >>> > There's always a single one behind each pci device.
>> >>> > Is this sufficient?
>> >>>
>> >>> I *think* is not good enough for udev to offer such functionality.
>> >>>
>> >>> We need an authoritative answer that this cannot happen with today's
>> >>> code, and also that there are no plans to ever make multiple virtio
>> >>> devices per parent device.
>> >
>> >> But if virtio will make such a promise, will that be sufficient?
>> >
>> > Sure.
>>
>> Any chance we could get such a guarantee? Anything else needed from
>> our side? I have a pending patch that would be nice to get out, which
>> would depend on such a guarantee.
>>
>> Cheers,
>>
>> Tom
>
> I thought about this. The virtio spec makes it explicit for pci,
> ccw and mmio devices. I think we can make this promise - if we
> wanted to have many such buses, we can always make it something else,
> not a virtio bus.
>
> So please just check the type of the bus - if it's virtio,
> you can assume there is no sibling.

Thanks! Will do.

Cheers,

Tom