[Qemu-devel] RFC [v2]: vfio / device assignment -- layout of device fd files

[Qemu-devel] RFC [v2]: vfio / device assignment -- layout of device fd files

[Stuart Yoder]

Based on the discussions over the last couple of weeks
I have updated the device fd file layout proposal and
tried to specify it a bit more formally.

===============================================================

1.  Overview

  This specification describes the layout of device files
  used in the context of vfio, which gives user space
  direct access to I/O devices that have been bound to
  vfio.

  When a device fd is opened and read, offset 0x0 contains
  a fixed sized header followed by a number of variable length
  records that describe different characteristics
  of the device-- addressable regions, interrupts, etc.

  0x0  +-------------+-------------+
       |         magic             | u32  // identifies this as a vfio
device file
       +---------------------------+         and identifies the type of bus
       |         version           | u32  // specifies the version of this
       +---------------------------+
       |         flags             | u32  // encodes any flags
       +---------------------------+
       |  dev info record 0        |
       |    type                   | u32   // type of record
       |    rec_len                | u32   // length in bytes of record
       |                           |          (including record header)
       |    flags                  | u32   // type specific flags
       |    ...content...          |       // record content, which could
       +---------------------------+       // include sub-records
       |  dev info record 1        |
       +---------------------------+
       |  dev info record N        |
       +---------------------------+

  The device info records following the file header may have
  the following record types each with content encoded in
  a record specific way:

  ------------+-------+------------------------------------------------------
              |  type |
   Region     |  num  | Description
  ---------------------------------------------------------------------------
  REGION           1    describes an addressable address range for the device
  DTPATH           2    describes the device tree path for the device
  DTINDEX          3    describes the index into the related device tree
                          property (reg,ranges,interrupts,interrupt-map)
  INTERRUPT        4    describes an interrupt for the device
  PCI_CONFIG_SPACE 5    property identifying a region as PCI config space
  PCI_BAR_INDEX    6    describes the BAR index for a PCI region
  PHYS_ADDR        7    describes the physical address of the region
  ---------------------------------------------------------------------------

2. Header

The header is located at offset 0x0 in the device fd
and has the following format:

    struct devfd_header {
        __u32 magic;
        __u32 version;
        __u32 flags;
    };

    The 'magic' field contains a magic value that will
    identify the type bus the device is on.  Valid values
    are:

        0x70636900   // "pci" - PCI device
        0x64740000   // "dt" - device tree (system bus)

3. Region

  A REGION record an addressable address region for the device.

    struct devfd_region {
        __u32 type;   // must be 0x1
        __u32 record_len;
        __u32 flags;
        __u64 offset; // seek offset to region from beginning
                      // of file
        __u64 len   ; // length of the region
    };

  The 'flags' field supports one flag:

      IS_MMAPABLE

4. Device Tree Path (DTPATH)

  A DTPATH record is a sub-record of a REGION and describes
  the path to a device tree node for the region

    struct devfd_dtpath {
        __u32 type;   // must be 0x2
        __u32 record_len;
        __u64 char[]   ; // length of the region
    };

5. Device Tree Index (DTINDEX)

  A DTINDEX record is a sub-record of a REGION and specifies
  the index into the resource list encoded in the associated
  device tree property-- "reg", "ranges", "interrupts", or
  "interrupt-map".

    struct devfd_dtindex {
        __u32 type;   // must be 0x3
        __u32 record_len;
        __u32 prop_type;
        __u32 prop_index;  // index into the resource list
    };

    prop_type must have one of the follow values:
       1   // "reg" property
       2   // "ranges" property
       3   // "interrupts" property
       4   // "interrupts" property

    Note: prop_index is not the byte offset into the property,
    but the logical index.

6. Interrupts (INTERRUPT)

  An INTERRUPT record describes one of a device's interrupts.
  The handle field is an argument to VFIO_DEVICE_GET_IRQ_FD
  which user space can use to receive device interrupts.

    struct devfd_interrupts {
        __u32 type;   // must be 0x4
        __u32 record_len;
        __u32 flags;
        __u32 handle;  // parameter to VFIO_DEVICE_GET_IRQ_FD
    };

7.  PCI Config Space (PCI_CONFIG_SPACE)

    A PCI_CONFIG_SPACE record is a sub-record of a REGION record
    and identifies the region as PCI configuration space.

    struct devfd_cfgspace {
        __u32 type;   // must be 0x5
        __u32 record_len;
        __u32 flags;
    }

8.  PCI Bar Index (PCI_BAR_INDEX)

    A PCI_BAR_INDEX record is a sub-record of a REGION record
    and identifies the PCI BAR index for the region.

    struct devfd_barindex {
        __u32 type;   // must be 0x6
        __u32 record_len;
        __u32 flags;
        __u32 bar_index;
    }

9.  Physical Address (PHYS_ADDR)

    A PHYS_ADDR record is a sub-record of a REGION record
    and specifies the physical address of the region.

    struct devfd_physaddr {
        __u32 type;   // must be 0x7
        __u32 record_len;
        __u32 flags;
        __u64 phys_addr;
    }

Regards,
Stuart

Re: [Qemu-devel] RFC [v2]: vfio / device assignment -- layout of device fd files

[Stuart Yoder]

Meant to identify the changes in v2 of this proposal:

v2:
   -removed PCI_INFO record type
   -removed PCI_BAR_INFO record type
   -PCI_CONFIG_SPACE is now a sub-record/property of a REGION
   -removed physical address from region and made it
    a subrecord/property of a REGION
   -added PCI_BAR_INDEX sub-record type
   -updated magic numbers

Stuart

On Fri, Sep 9, 2011 at 8:11 AM, Stuart Yoder <b08248@gmail.com> wrote:
> Based on the discussions over the last couple of weeks
> I have updated the device fd file layout proposal and
> tried to specify it a bit more formally.
>
> ===============================================================
>
> 1.  Overview
>
>  This specification describes the layout of device files
>  used in the context of vfio, which gives user space
>  direct access to I/O devices that have been bound to
>  vfio.
>
>  When a device fd is opened and read, offset 0x0 contains
>  a fixed sized header followed by a number of variable length
>  records that describe different characteristics
>  of the device-- addressable regions, interrupts, etc.
>
>  0x0  +-------------+-------------+
>       |         magic             | u32  // identifies this as a vfio
> device file
>       +---------------------------+         and identifies the type of bus
>       |         version           | u32  // specifies the version of this
>       +---------------------------+
>       |         flags             | u32  // encodes any flags
>       +---------------------------+
>       |  dev info record 0        |
>       |    type                   | u32   // type of record
>       |    rec_len                | u32   // length in bytes of record
>       |                           |          (including record header)
>       |    flags                  | u32   // type specific flags
>       |    ...content...          |       // record content, which could
>       +---------------------------+       // include sub-records
>       |  dev info record 1        |
>       +---------------------------+
>       |  dev info record N        |
>       +---------------------------+
>
>  The device info records following the file header may have
>  the following record types each with content encoded in
>  a record specific way:
>
>  ------------+-------+------------------------------------------------------
>              |  type |
>   Region     |  num  | Description
>  ---------------------------------------------------------------------------
>  REGION           1    describes an addressable address range for the device
>  DTPATH           2    describes the device tree path for the device
>  DTINDEX          3    describes the index into the related device tree
>                          property (reg,ranges,interrupts,interrupt-map)
>  INTERRUPT        4    describes an interrupt for the device
>  PCI_CONFIG_SPACE 5    property identifying a region as PCI config space
>  PCI_BAR_INDEX    6    describes the BAR index for a PCI region
>  PHYS_ADDR        7    describes the physical address of the region
>  ---------------------------------------------------------------------------
>
> 2. Header
>
> The header is located at offset 0x0 in the device fd
> and has the following format:
>
>    struct devfd_header {
>        __u32 magic;
>        __u32 version;
>        __u32 flags;
>    };
>
>    The 'magic' field contains a magic value that will
>    identify the type bus the device is on.  Valid values
>    are:
>
>        0x70636900   // "pci" - PCI device
>        0x64740000   // "dt" - device tree (system bus)
>
> 3. Region
>
>  A REGION record an addressable address region for the device.
>
>    struct devfd_region {
>        __u32 type;   // must be 0x1
>        __u32 record_len;
>        __u32 flags;
>        __u64 offset; // seek offset to region from beginning
>                      // of file
>        __u64 len   ; // length of the region
>    };
>
>  The 'flags' field supports one flag:
>
>      IS_MMAPABLE
>
> 4. Device Tree Path (DTPATH)
>
>  A DTPATH record is a sub-record of a REGION and describes
>  the path to a device tree node for the region
>
>    struct devfd_dtpath {
>        __u32 type;   // must be 0x2
>        __u32 record_len;
>        __u64 char[]   ; // length of the region
>    };
>
> 5. Device Tree Index (DTINDEX)
>
>  A DTINDEX record is a sub-record of a REGION and specifies
>  the index into the resource list encoded in the associated
>  device tree property-- "reg", "ranges", "interrupts", or
>  "interrupt-map".
>
>    struct devfd_dtindex {
>        __u32 type;   // must be 0x3
>        __u32 record_len;
>        __u32 prop_type;
>        __u32 prop_index;  // index into the resource list
>    };
>
>    prop_type must have one of the follow values:
>       1   // "reg" property
>       2   // "ranges" property
>       3   // "interrupts" property
>       4   // "interrupts" property
>
>    Note: prop_index is not the byte offset into the property,
>    but the logical index.
>
> 6. Interrupts (INTERRUPT)
>
>  An INTERRUPT record describes one of a device's interrupts.
>  The handle field is an argument to VFIO_DEVICE_GET_IRQ_FD
>  which user space can use to receive device interrupts.
>
>    struct devfd_interrupts {
>        __u32 type;   // must be 0x4
>        __u32 record_len;
>        __u32 flags;
>        __u32 handle;  // parameter to VFIO_DEVICE_GET_IRQ_FD
>    };
>
> 7.  PCI Config Space (PCI_CONFIG_SPACE)
>
>    A PCI_CONFIG_SPACE record is a sub-record of a REGION record
>    and identifies the region as PCI configuration space.
>
>    struct devfd_cfgspace {
>        __u32 type;   // must be 0x5
>        __u32 record_len;
>        __u32 flags;
>    }
>
> 8.  PCI Bar Index (PCI_BAR_INDEX)
>
>    A PCI_BAR_INDEX record is a sub-record of a REGION record
>    and identifies the PCI BAR index for the region.
>
>    struct devfd_barindex {
>        __u32 type;   // must be 0x6
>        __u32 record_len;
>        __u32 flags;
>        __u32 bar_index;
>    }
>
> 9.  Physical Address (PHYS_ADDR)
>
>    A PHYS_ADDR record is a sub-record of a REGION record
>    and specifies the physical address of the region.
>
>    struct devfd_physaddr {
>        __u32 type;   // must be 0x7
>        __u32 record_len;
>        __u32 flags;
>        __u64 phys_addr;
>    }
>
> Regards,
> Stuart
>

Re: [Qemu-devel] RFC [v2]: vfio / device assignment -- layout of device fd files

[Alex Williamson]

On Fri, 2011-09-09 at 08:11 -0500, Stuart Yoder wrote:
> Based on the discussions over the last couple of weeks
> I have updated the device fd file layout proposal and
> tried to specify it a bit more formally.
> 
> ===============================================================
> 
> 1.  Overview
> 
>   This specification describes the layout of device files
>   used in the context of vfio, which gives user space
>   direct access to I/O devices that have been bound to
>   vfio.
> 
>   When a device fd is opened and read, offset 0x0 contains
>   a fixed sized header followed by a number of variable length
>   records that describe different characteristics
>   of the device-- addressable regions, interrupts, etc.
> 
>   0x0  +-------------+-------------+
>        |         magic             | u32  // identifies this as a vfio
> device file
>        +---------------------------+         and identifies the type of bus
>        |         version           | u32  // specifies the version of this
>        +---------------------------+
>        |         flags             | u32  // encodes any flags
>        +---------------------------+
>        |  dev info record 0        |
>        |    type                   | u32   // type of record
>        |    rec_len                | u32   // length in bytes of record
>        |                           |          (including record header)
>        |    flags                  | u32   // type specific flags
>        |    ...content...          |       // record content, which could
>        +---------------------------+       // include sub-records
>        |  dev info record 1        |
>        +---------------------------+
>        |  dev info record N        |
>        +---------------------------+
> 
>   The device info records following the file header may have
>   the following record types each with content encoded in
>   a record specific way:
> 
>   ------------+-------+------------------------------------------------------
>               |  type |
>    Region     |  num  | Description
>   ---------------------------------------------------------------------------
>   REGION           1    describes an addressable address range for the device
>   DTPATH           2    describes the device tree path for the device
>   DTINDEX          3    describes the index into the related device tree
>                           property (reg,ranges,interrupts,interrupt-map)
>   INTERRUPT        4    describes an interrupt for the device
>   PCI_CONFIG_SPACE 5    property identifying a region as PCI config space
>   PCI_BAR_INDEX    6    describes the BAR index for a PCI region
>   PHYS_ADDR        7    describes the physical address of the region
>   ---------------------------------------------------------------------------
> 
> 2. Header
> 
> The header is located at offset 0x0 in the device fd
> and has the following format:
> 
>     struct devfd_header {
>         __u32 magic;
>         __u32 version;
>         __u32 flags;
>     };
> 
>     The 'magic' field contains a magic value that will
>     identify the type bus the device is on.  Valid values
>     are:
> 
>         0x70636900   // "pci" - PCI device
>         0x64740000   // "dt" - device tree (system bus)
> 
> 3. Region
> 
>   A REGION record an addressable address region for the device.
> 
>     struct devfd_region {
>         __u32 type;   // must be 0x1
>         __u32 record_len;
>         __u32 flags;
>         __u64 offset; // seek offset to region from beginning
>                       // of file
>         __u64 len   ; // length of the region
>     };
> 
>   The 'flags' field supports one flag:
> 
>       IS_MMAPABLE
> 
> 4. Device Tree Path (DTPATH)
> 
>   A DTPATH record is a sub-record of a REGION and describes
>   the path to a device tree node for the region

Can we better distinguish sub-records from records?  I assume we're
trying to be as versatile as possible by having a single "type" address
space, but is this going to lead to implementation problems?  A DTPATH
as a record, an INTERRUPT as a sub-record, etc.  Should we instead have
a "subtype" address space per "type" and per device type?  For a "dt"
device, it looks like we really have:

      * REGION (type 0)
              * DTPATH (subtype 0)
              * DTINDEX (subtype 1)
              * PHYS_ADDR (subtype 2)
      * INTERRUPT (type 1)
              * DTPATH (subtype 0)
              * DTINDEX (subtype 1)

While "pci" is:

      * REGION (type 0)
              * PCI_CONFIG_SPACE (subtype 0)
              * PCI_BAR_INDEX (subtype 1)
      * INTERRUPT (type 1)

>     struct devfd_dtpath {
>         __u32 type;   // must be 0x2
>         __u32 record_len;
>         __u64 char[]   ; // length of the region
>     };
> 
> 5. Device Tree Index (DTINDEX)
> 
>   A DTINDEX record is a sub-record of a REGION and specifies
>   the index into the resource list encoded in the associated
>   device tree property-- "reg", "ranges", "interrupts", or
>   "interrupt-map".
> 
>     struct devfd_dtindex {
>         __u32 type;   // must be 0x3
>         __u32 record_len;
>         __u32 prop_type;
>         __u32 prop_index;  // index into the resource list
>     };
> 
>     prop_type must have one of the follow values:
>        1   // "reg" property
>        2   // "ranges" property
>        3   // "interrupts" property
>        4   // "interrupts" property
> 
>     Note: prop_index is not the byte offset into the property,
>     but the logical index.
> 
> 6. Interrupts (INTERRUPT)
> 
>   An INTERRUPT record describes one of a device's interrupts.
>   The handle field is an argument to VFIO_DEVICE_GET_IRQ_FD
>   which user space can use to receive device interrupts.
> 
>     struct devfd_interrupts {
>         __u32 type;   // must be 0x4
>         __u32 record_len;
>         __u32 flags;
>         __u32 handle;  // parameter to VFIO_DEVICE_GET_IRQ_FD
>     };

I'm still on the fence whether we should implement INTERRUPT for PCI or
only assume handle 0x0 or maybe assume handle == interrupt pin.

> 
> 7.  PCI Config Space (PCI_CONFIG_SPACE)
> 
>     A PCI_CONFIG_SPACE record is a sub-record of a REGION record
>     and identifies the region as PCI configuration space.
> 
>     struct devfd_cfgspace {
>         __u32 type;   // must be 0x5
>         __u32 record_len;
>         __u32 flags;
>     }
> 
> 8.  PCI Bar Index (PCI_BAR_INDEX)
> 
>     A PCI_BAR_INDEX record is a sub-record of a REGION record
>     and identifies the PCI BAR index for the region.
> 
>     struct devfd_barindex {
>         __u32 type;   // must be 0x6
>         __u32 record_len;
>         __u32 flags;
>         __u32 bar_index;
>     }

I suppose we're more concerned with easy parsing and alignment than
compactness, so a u32 to differentiate 6 BARS + 1 ROM is probably ok.

> 
> 9.  Physical Address (PHYS_ADDR)
> 
>     A PHYS_ADDR record is a sub-record of a REGION record
>     and specifies the physical address of the region.
> 
>     struct devfd_physaddr {
>         __u32 type;   // must be 0x7
>         __u32 record_len;
>         __u32 flags;
>         __u64 phys_addr;
>     }

Thanks,

Alex

Re: [Qemu-devel] RFC [v2]: vfio / device assignment -- layout of device fd files

[Scott Wood]

On 09/19/2011 10:16 AM, Alex Williamson wrote:
> On Fri, 2011-09-09 at 08:11 -0500, Stuart Yoder wrote:
>> 2. Header
>>
>> The header is located at offset 0x0 in the device fd
>> and has the following format:
>>
>>     struct devfd_header {
>>         __u32 magic;
>>         __u32 version;
>>         __u32 flags;
>>     };
>>
>>     The 'magic' field contains a magic value that will
>>     identify the type bus the device is on.  Valid values
>>     are:
>>
>>         0x70636900   // "pci" - PCI device
>>         0x64740000   // "dt" - device tree (system bus)

These look somewhat conflict-prone (even more so than "vfio") -- maybe
not ambiguous in context, but if we're going to use magic numbers we
might as well make them as unique as we can.  Can't we just generate a
couple random numbers?

Also, is the magic number specifically 0x70636900 in native endian, or
"pci" however it would be encoded on the platform?  Are there any
platforms in Linux where multiple endians are supported at once in
userspace (on a per-process basis)?

>> 3. Region
>>
>>   A REGION record an addressable address region for the device.
>>
>>     struct devfd_region {
>>         __u32 type;   // must be 0x1
>>         __u32 record_len;
>>         __u32 flags;
>>         __u64 offset; // seek offset to region from beginning
>>                       // of file
>>         __u64 len   ; // length of the region
>>     };
>>
>>   The 'flags' field supports one flag:
>>
>>       IS_MMAPABLE
>>
>> 4. Device Tree Path (DTPATH)
>>
>>   A DTPATH record is a sub-record of a REGION and describes
>>   the path to a device tree node for the region
> 
> Can we better distinguish sub-records from records?  I assume we're
> trying to be as versatile as possible by having a single "type" address
> space, but is this going to lead to implementation problems?

What kind of problems?

> A DTPATH as a record, an INTERRUPT as a sub-record, etc.

Same as any other unrecognized (sub)record type, you ignore it -- but
the kernel should not be generating this.

> Should we instead have
> a "subtype" address space per "type" and per device type?  For a "dt"
> device, it looks like we really have:
> 
>       * REGION (type 0)
>               * DTPATH (subtype 0)
>               * DTINDEX (subtype 1)
>               * PHYS_ADDR (subtype 2)
>       * INTERRUPT (type 1)
>               * DTPATH (subtype 0)
>               * DTINDEX (subtype 1)
> 
> While "pci" is:
> 
>       * REGION (type 0)
>               * PCI_CONFIG_SPACE (subtype 0)
>               * PCI_BAR_INDEX (subtype 1)
>       * INTERRUPT (type 1)

I'd prefer to keep one numberspace, with documented restrictions on what
types/subtypes are allowed in each context.  Makes it easier if we end
up in a situation where a (sub)record type is applicable to multiple
contexts, and easier to detect when an error has been made.

>> 8.  PCI Bar Index (PCI_BAR_INDEX)
>>
>>     A PCI_BAR_INDEX record is a sub-record of a REGION record
>>     and identifies the PCI BAR index for the region.
>>
>>     struct devfd_barindex {
>>         __u32 type;   // must be 0x6
>>         __u32 record_len;
>>         __u32 flags;
>>         __u32 bar_index;
>>     }
> 
> I suppose we're more concerned with easy parsing and alignment than
> compactness, so a u32 to differentiate 6 BARS + 1 ROM is probably ok.

Right.

-Scott

Re: [Qemu-devel] RFC [v2]: vfio / device assignment -- layout of device fd files

[Alex Williamson]

On Mon, 2011-09-19 at 14:37 -0500, Scott Wood wrote:
> On 09/19/2011 10:16 AM, Alex Williamson wrote:
> > On Fri, 2011-09-09 at 08:11 -0500, Stuart Yoder wrote:
> >> 2. Header
> >>
> >> The header is located at offset 0x0 in the device fd
> >> and has the following format:
> >>
> >>     struct devfd_header {
> >>         __u32 magic;
> >>         __u32 version;
> >>         __u32 flags;
> >>     };
> >>
> >>     The 'magic' field contains a magic value that will
> >>     identify the type bus the device is on.  Valid values
> >>     are:
> >>
> >>         0x70636900   // "pci" - PCI device
> >>         0x64740000   // "dt" - device tree (system bus)
> 
> These look somewhat conflict-prone (even more so than "vfio") -- maybe
> not ambiguous in context, but if we're going to use magic numbers we
> might as well make them as unique as we can.  Can't we just generate a
> couple random numbers?
> 
> Also, is the magic number specifically 0x70636900 in native endian, or
> "pci" however it would be encoded on the platform?  Are there any
> platforms in Linux where multiple endians are supported at once in
> userspace (on a per-process basis)?

A GUID would be fine w/ me.

> >> 3. Region
> >>
> >>   A REGION record an addressable address region for the device.
> >>
> >>     struct devfd_region {
> >>         __u32 type;   // must be 0x1
> >>         __u32 record_len;
> >>         __u32 flags;
> >>         __u64 offset; // seek offset to region from beginning
> >>                       // of file
> >>         __u64 len   ; // length of the region
> >>     };
> >>
> >>   The 'flags' field supports one flag:
> >>
> >>       IS_MMAPABLE
> >>
> >> 4. Device Tree Path (DTPATH)
> >>
> >>   A DTPATH record is a sub-record of a REGION and describes
> >>   the path to a device tree node for the region
> > 
> > Can we better distinguish sub-records from records?  I assume we're
> > trying to be as versatile as possible by having a single "type" address
> > space, but is this going to lead to implementation problems?
> 
> What kind of problems?

vvv Those kind.

> > A DTPATH as a record, an INTERRUPT as a sub-record, etc.
> 
> Same as any other unrecognized (sub)record type, you ignore it -- but
> the kernel should not be generating this.

I'm trying to express that I think the spec is unclear here.  It's easy
to hand wave that the code will do the right thing, but if the next
person comes along and doesn't understand that a DTPATH is only a
sub-type and a DTINDEX needs to be paired with a DTPATH, then we've
failed.

> > Should we instead have
> > a "subtype" address space per "type" and per device type?  For a "dt"
> > device, it looks like we really have:
> > 
> >       * REGION (type 0)
> >               * DTPATH (subtype 0)
> >               * DTINDEX (subtype 1)
> >               * PHYS_ADDR (subtype 2)
> >       * INTERRUPT (type 1)
> >               * DTPATH (subtype 0)
> >               * DTINDEX (subtype 1)
> > 
> > While "pci" is:
> > 
> >       * REGION (type 0)
> >               * PCI_CONFIG_SPACE (subtype 0)
> >               * PCI_BAR_INDEX (subtype 1)
> >       * INTERRUPT (type 1)
> 
> I'd prefer to keep one numberspace, with documented restrictions on what
> types/subtypes are allowed in each context.  Makes it easier if we end
> up in a situation where a (sub)record type is applicable to multiple
> contexts, and easier to detect when an error has been made.

Couldn't we accomplish the same with separate type and subtype number
spaces?

enum types {
	REGION,
	INTERRUPT,
};

enum subtypes {
	DTPATH,
	DTINDEX,
	PHYS_ADDR,
	PCI_CONFIG_SPACE,
	PCI_BAR_INDEX,
};

I just find it confusing that we intermix them when defining them.
Thanks,

Alex

> >> 8.  PCI Bar Index (PCI_BAR_INDEX)
> >>
> >>     A PCI_BAR_INDEX record is a sub-record of a REGION record
> >>     and identifies the PCI BAR index for the region.
> >>
> >>     struct devfd_barindex {
> >>         __u32 type;   // must be 0x6
> >>         __u32 record_len;
> >>         __u32 flags;
> >>         __u32 bar_index;
> >>     }
> > 
> > I suppose we're more concerned with easy parsing and alignment than
> > compactness, so a u32 to differentiate 6 BARS + 1 ROM is probably ok.
> 
> Right.
> 
> -Scott
> 

Re: [Qemu-devel] RFC [v2]: vfio / device assignment -- layout of device fd files

[Scott Wood]

On 09/19/2011 04:07 PM, Alex Williamson wrote:
> On Mon, 2011-09-19 at 14:37 -0500, Scott Wood wrote:
>>> A DTPATH as a record, an INTERRUPT as a sub-record, etc.
>>
>> Same as any other unrecognized (sub)record type, you ignore it -- but
>> the kernel should not be generating this.
> 
> I'm trying to express that I think the spec is unclear here.  It's easy
> to hand wave that the code will do the right thing, but if the next
> person comes along and doesn't understand that a DTPATH is only a
> sub-type and a DTINDEX needs to be paired with a DTPATH, then we've
> failed.

Sure, the spec needs to be clear about which types are expected in each
context.

>> I'd prefer to keep one numberspace, with documented restrictions on what
>> types/subtypes are allowed in each context.  Makes it easier if we end
>> up in a situation where a (sub)record type is applicable to multiple
>> contexts, and easier to detect when an error has been made.
> 
> Couldn't we accomplish the same with separate type and subtype number
> spaces?
> 
> enum types {
> 	REGION,
> 	INTERRUPT,
> };
> 
> enum subtypes {
> 	DTPATH,
> 	DTINDEX,
> 	PHYS_ADDR,
> 	PCI_CONFIG_SPACE,
> 	PCI_BAR_INDEX,
> };
> 
> I just find it confusing that we intermix them when defining them.
> Thanks,

As long as we don't have separate numberspaces for PCI/DT/future-stuff,
I'm reasonably happy.

-Scott

Re: [Qemu-devel] RFC [v2]: vfio / device assignment -- layout of device fd files

[David Gibson]

On Fri, Sep 09, 2011 at 08:11:54AM -0500, Stuart Yoder wrote:
> Based on the discussions over the last couple of weeks
> I have updated the device fd file layout proposal and
> tried to specify it a bit more formally.
> 
> ===============================================================
> 
> 1.  Overview
> 
>   This specification describes the layout of device files
>   used in the context of vfio, which gives user space
>   direct access to I/O devices that have been bound to
>   vfio.
> 
>   When a device fd is opened and read, offset 0x0 contains
>   a fixed sized header followed by a number of variable length
>   records that describe different characteristics
>   of the device-- addressable regions, interrupts, etc.
> 
>   0x0  +-------------+-------------+
>        |         magic             | u32  // identifies this as a vfio
> device file
>        +---------------------------+         and identifies the type of bus
>        |         version           | u32  // specifies the version of this
>        +---------------------------+
>        |         flags             | u32  // encodes any flags
>        +---------------------------+
>        |  dev info record 0        |
>        |    type                   | u32   // type of record
>        |    rec_len                | u32   // length in bytes of record
>        |                           |          (including record header)
>        |    flags                  | u32   // type specific flags
>        |    ...content...          |       // record content, which could
>        +---------------------------+       // include sub-records
>        |  dev info record 1        |
>        +---------------------------+
>        |  dev info record N        |
>        +---------------------------+

I really should have chimed in on this earlier, but I've been very
busy.

Um, not to put too fine a point on it, this is madness.

Yes, it's very flexible and can thereby cover a very wide range of
cases.  But it's much, much too complex.  Userspace has to parse a
complex, multilayered data structure, with variable length elements
just to get an address at which to do IO.  I can pretty much guarantee
that if we went with this, most userspace programs using this
interface would just ignore this metadata and directly map the
offsets at which they happen to know the kernel will put things for
the type of device they care about.

_At least_ for PCI, I think the original VFIO layout of each BAR at a
fixed, well known offset is much better.  Despite its limitations,
just advertising a "device type" ID which describes one of a few fixed
layouts would be preferable to this.  I'm still hoping, that we can do
a bit better than that.  But we should try really hard to at the very
least force the metadata into a simple array of resources each with a
fixed size record describing it, even if it means some space wastage
with occasionally-used fields.  Anything more complex than that and
userspace is just never going to use it properly.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

Re: [Qemu-devel] RFC [v2]: vfio / device assignment -- layout of device fd files

[Alexander Graf]

Am 26.09.2011 um 09:51 schrieb David Gibson <david@gibson.dropbear.id.au>:

> On Fri, Sep 09, 2011 at 08:11:54AM -0500, Stuart Yoder wrote:
>> Based on the discussions over the last couple of weeks
>> I have updated the device fd file layout proposal and
>> tried to specify it a bit more formally.
>> 
>> ===============================================================
>> 
>> 1.  Overview
>> 
>>  This specification describes the layout of device files
>>  used in the context of vfio, which gives user space
>>  direct access to I/O devices that have been bound to
>>  vfio.
>> 
>>  When a device fd is opened and read, offset 0x0 contains
>>  a fixed sized header followed by a number of variable length
>>  records that describe different characteristics
>>  of the device-- addressable regions, interrupts, etc.
>> 
>>  0x0  +-------------+-------------+
>>       |         magic             | u32  // identifies this as a vfio
>> device file
>>       +---------------------------+         and identifies the type of bus
>>       |         version           | u32  // specifies the version of this
>>       +---------------------------+
>>       |         flags             | u32  // encodes any flags
>>       +---------------------------+
>>       |  dev info record 0        |
>>       |    type                   | u32   // type of record
>>       |    rec_len                | u32   // length in bytes of record
>>       |                           |          (including record header)
>>       |    flags                  | u32   // type specific flags
>>       |    ...content...          |       // record content, which could
>>       +---------------------------+       // include sub-records
>>       |  dev info record 1        |
>>       +---------------------------+
>>       |  dev info record N        |
>>       +---------------------------+
> 
> I really should have chimed in on this earlier, but I've been very
> busy.
> 
> Um, not to put too fine a point on it, this is madness.
> 
> Yes, it's very flexible and can thereby cover a very wide range of
> cases.  But it's much, much too complex.  Userspace has to parse a
> complex, multilayered data structure, with variable length elements
> just to get an address at which to do IO.  I can pretty much guarantee
> that if we went with this, most userspace programs using this
> interface would just ignore this metadata and directly map the
> offsets at which they happen to know the kernel will put things for
> the type of device they care about.
> 
> _At least_ for PCI, I think the original VFIO layout of each BAR at a
> fixed, well known offset is much better.  Despite its limitations,
> just advertising a "device type" ID which describes one of a few fixed
> layouts would be preferable to this.  I'm still hoping, that we can do
> a bit better than that.  But we should try really hard to at the very
> least force the metadata into a simple array of resources each with a
> fixed size record describing it, even if it means some space wastage
> with occasionally-used fields.  Anything more complex than that and
> userspace is just never going to use it properly.

We will have 2 different types of user space. One wants to be as generic as possible and needs all this dynamic information. QEMU would fall into this category.

The other one is device specific drivers in user space. Here hardcoding might make sense.

For the generic interface, we need something that us as verbose as possible and lets us enumerate all the device properties, so we can properly map and forward them to the guest.

However, nothing keeps us from mapping BARs always at static offsets into the file. If you don't need the generic info, then you don't need it.

Also, if you can come up with an interface that does not have variable length descriptors but is still able to export all the required generic information, please send a proposal to the list :)


Alex
> 

Re: [Qemu-devel] RFC [v2]: vfio / device assignment -- layout of device fd files

[Alex Williamson]

On Mon, 2011-09-26 at 12:04 +0200, Alexander Graf wrote:
> Am 26.09.2011 um 09:51 schrieb David Gibson <david@gibson.dropbear.id.au>:
> 
> > On Fri, Sep 09, 2011 at 08:11:54AM -0500, Stuart Yoder wrote:
> >> Based on the discussions over the last couple of weeks
> >> I have updated the device fd file layout proposal and
> >> tried to specify it a bit more formally.
> >> 
> >> ===============================================================
> >> 
> >> 1.  Overview
> >> 
> >>  This specification describes the layout of device files
> >>  used in the context of vfio, which gives user space
> >>  direct access to I/O devices that have been bound to
> >>  vfio.
> >> 
> >>  When a device fd is opened and read, offset 0x0 contains
> >>  a fixed sized header followed by a number of variable length
> >>  records that describe different characteristics
> >>  of the device-- addressable regions, interrupts, etc.
> >> 
> >>  0x0  +-------------+-------------+
> >>       |         magic             | u32  // identifies this as a vfio
> >> device file
> >>       +---------------------------+         and identifies the type of bus
> >>       |         version           | u32  // specifies the version of this
> >>       +---------------------------+
> >>       |         flags             | u32  // encodes any flags
> >>       +---------------------------+
> >>       |  dev info record 0        |
> >>       |    type                   | u32   // type of record
> >>       |    rec_len                | u32   // length in bytes of record
> >>       |                           |          (including record header)
> >>       |    flags                  | u32   // type specific flags
> >>       |    ...content...          |       // record content, which could
> >>       +---------------------------+       // include sub-records
> >>       |  dev info record 1        |
> >>       +---------------------------+
> >>       |  dev info record N        |
> >>       +---------------------------+
> > 
> > I really should have chimed in on this earlier, but I've been very
> > busy.
> > 
> > Um, not to put too fine a point on it, this is madness.
> > 
> > Yes, it's very flexible and can thereby cover a very wide range of
> > cases.  But it's much, much too complex.  Userspace has to parse a
> > complex, multilayered data structure, with variable length elements
> > just to get an address at which to do IO.  I can pretty much guarantee
> > that if we went with this, most userspace programs using this
> > interface would just ignore this metadata and directly map the
> > offsets at which they happen to know the kernel will put things for
> > the type of device they care about.
> > 
> > _At least_ for PCI, I think the original VFIO layout of each BAR at a
> > fixed, well known offset is much better.  Despite its limitations,
> > just advertising a "device type" ID which describes one of a few fixed
> > layouts would be preferable to this.  I'm still hoping, that we can do
> > a bit better than that.  But we should try really hard to at the very
> > least force the metadata into a simple array of resources each with a
> > fixed size record describing it, even if it means some space wastage
> > with occasionally-used fields.  Anything more complex than that and
> > userspace is just never going to use it properly.
> 
> We will have 2 different types of user space. One wants to be as
> generic as possible and needs all this dynamic information. QEMU would
> fall into this category.
> 
> The other one is device specific drivers in user space. Here
> hardcoding might make sense.
> 
> For the generic interface, we need something that us as verbose as
> possible and lets us enumerate all the device properties, so we can
> properly map and forward them to the guest.
> 
> However, nothing keeps us from mapping BARs always at static offsets
> into the file. If you don't need the generic info, then you don't need
> it.
> 
> Also, if you can come up with an interface that does not have variable
> length descriptors but is still able to export all the required
> generic information, please send a proposal to the list :)
> 

Hi,

The other obvious possibility is a pure ioctl interface.  To match what
this proposal is trying to describe, plus the runtime interfaces, we'd
need something like:

/* :0 - PCI devices, :1 - Devices path device, 63:2 - reserved */
#define VFIO_DEVICE_GET_FLAGS			_IOR(, , u64)


/* Return number of mmio/iop/config regions.
 * For PCI this is always 8 (BAR0-5 + ROM + Config) */
#define VFIO_DEVICE_GET_NUM_REGIONS		_IOR(, , int)

/* Return length for region index (may be zero) */
#define VFIO_DEVICE_GET_REGION_LEN		_IOWR(, , u64)

/* Return flags for region index
 * :0 - mmap'able, :1 - read-only, 63:2 - reserved */
#define VFIO_DEVICE_GET_REGION_FLAGS		_IOR(, , u64)

/* Return file offset for region index */
#define VFIO_DEVICE_GET_REGION_OFFSET		_IOWR(, , u64)

/* Return physical address for region index - not implemented for PCI */
#define VFIO_DEVICE_GET_REGION_PHYS_ADDR	_IOWR(, , u64)



/* Return number of IRQs (Not including MSI/MSI-X for PCI) */
#define VFIO_DEVICE_GET_NUM_IRQ			_IOR(, , int)

/* Set IRQ eventfd for IRQ index, arg[0] = index, arg[1] = fd */
#define VFIO_DEVICE_SET_IRQ_EVENTFD		_IOW(, , int)

/* Unmask IRQ index */
#define VFIO_DEVICE_UNMASK_IRQ			_IOW(, , int)

/* Set unmask eventfd for index, arg[0] = index, arg[1] = fd */
#define VFIO_DEVICE_SET_UNMASK_IRQ_EVENTFD	_IOW(, , int)


/* Return the device tree path for type/index into the user
 * allocated buffer */
struct dtpath {
	u32	type; (0 = region, 1 = IRQ)
	u32	index;
	u32	buf_len;
	char	*buf;
};
#define VFIO_DEVICE_GET_DTPATH			_IOWR(, , struct dtpath)

/* Return the device tree index for type/index */
struct dtindex {
	u32	type; (0 = region, 1 = IRQ)
	u32	index;
	u32	prop_type;
	u32	prop_index;
};
#define VFIO_DEVICE_GET_DTINDEX			_IOWR(, , struct dtindex)


/* Reset the device */
#define VFIO_DEVICE_RESET			_IO(, ,)


/* PCI MSI setup, arg[0] = #, arg[1-n] = eventfds */
#define VFIO_DEVICE_PCI_SET_MSI_EVENTFDS	_IOW(, , int)
#define VFIO_DEVICE_PCI_SET_MSIX_EVENTFDS	_IOW(, , int)

Hope that covers it.  Something I prefer about this interface is that
everything can easily be generated on the fly, whereas reading out a
table from the device means we really need to have that table somewhere
in kernel memory to easily support reading random offsets.  Thoughts?
Thanks,

Alex

Re: [Qemu-devel] RFC [v2]: vfio / device assignment -- layout of device fd files

[Stuart Yoder]

>
> The other obvious possibility is a pure ioctl interface.  To match what
> this proposal is trying to describe, plus the runtime interfaces, we'd
> need something like:
>
> /* :0 - PCI devices, :1 - Devices path device, 63:2 - reserved */
> #define VFIO_DEVICE_GET_FLAGS                   _IOR(, , u64)
>
>
> /* Return number of mmio/iop/config regions.
>  * For PCI this is always 8 (BAR0-5 + ROM + Config) */
> #define VFIO_DEVICE_GET_NUM_REGIONS             _IOR(, , int)
>
> /* Return length for region index (may be zero) */
> #define VFIO_DEVICE_GET_REGION_LEN              _IOWR(, , u64)
>
> /* Return flags for region index
>  * :0 - mmap'able, :1 - read-only, 63:2 - reserved */
> #define VFIO_DEVICE_GET_REGION_FLAGS            _IOR(, , u64)
>
> /* Return file offset for region index */
> #define VFIO_DEVICE_GET_REGION_OFFSET           _IOWR(, , u64)
>
> /* Return physical address for region index - not implemented for PCI */
> #define VFIO_DEVICE_GET_REGION_PHYS_ADDR        _IOWR(, , u64)
>
>
>
> /* Return number of IRQs (Not including MSI/MSI-X for PCI) */
> #define VFIO_DEVICE_GET_NUM_IRQ                 _IOR(, , int)
>
> /* Set IRQ eventfd for IRQ index, arg[0] = index, arg[1] = fd */
> #define VFIO_DEVICE_SET_IRQ_EVENTFD             _IOW(, , int)
>
> /* Unmask IRQ index */
> #define VFIO_DEVICE_UNMASK_IRQ                  _IOW(, , int)
>
> /* Set unmask eventfd for index, arg[0] = index, arg[1] = fd */
> #define VFIO_DEVICE_SET_UNMASK_IRQ_EVENTFD      _IOW(, , int)
>
>
> /* Return the device tree path for type/index into the user
>  * allocated buffer */
> struct dtpath {
>        u32     type; (0 = region, 1 = IRQ)
>        u32     index;
>        u32     buf_len;
>        char    *buf;
> };
> #define VFIO_DEVICE_GET_DTPATH                  _IOWR(, , struct dtpath)
>
> /* Return the device tree index for type/index */
> struct dtindex {
>        u32     type; (0 = region, 1 = IRQ)
>        u32     index;
>        u32     prop_type;
>        u32     prop_index;
> };
> #define VFIO_DEVICE_GET_DTINDEX                 _IOWR(, , struct dtindex)
>
>
> /* Reset the device */
> #define VFIO_DEVICE_RESET                       _IO(, ,)
>
>
> /* PCI MSI setup, arg[0] = #, arg[1-n] = eventfds */
> #define VFIO_DEVICE_PCI_SET_MSI_EVENTFDS        _IOW(, , int)
> #define VFIO_DEVICE_PCI_SET_MSIX_EVENTFDS       _IOW(, , int)
>
> Hope that covers it.  Something I prefer about this interface is that
> everything can easily be generated on the fly, whereas reading out a
> table from the device means we really need to have that table somewhere
> in kernel memory to easily support reading random offsets.  Thoughts?

I think this could work, but I'm not sure it makes the problem David
had any better--  you substitute the complexity of parsing the
variable length regions with invoking a set of APIs.

Stuart

Re: [Qemu-devel] RFC [v2]: vfio / device assignment -- layout of device fd files

[Alex Williamson]

On Mon, 2011-09-26 at 15:03 -0500, Stuart Yoder wrote:
> >
> > The other obvious possibility is a pure ioctl interface.  To match what
> > this proposal is trying to describe, plus the runtime interfaces, we'd
> > need something like:
> >
> > /* :0 - PCI devices, :1 - Devices path device, 63:2 - reserved */
> > #define VFIO_DEVICE_GET_FLAGS                   _IOR(, , u64)
> >
> >
> > /* Return number of mmio/iop/config regions.
> >  * For PCI this is always 8 (BAR0-5 + ROM + Config) */
> > #define VFIO_DEVICE_GET_NUM_REGIONS             _IOR(, , int)
> >
> > /* Return length for region index (may be zero) */
> > #define VFIO_DEVICE_GET_REGION_LEN              _IOWR(, , u64)
> >
> > /* Return flags for region index
> >  * :0 - mmap'able, :1 - read-only, 63:2 - reserved */
> > #define VFIO_DEVICE_GET_REGION_FLAGS            _IOR(, , u64)
> >
> > /* Return file offset for region index */
> > #define VFIO_DEVICE_GET_REGION_OFFSET           _IOWR(, , u64)
> >
> > /* Return physical address for region index - not implemented for PCI */
> > #define VFIO_DEVICE_GET_REGION_PHYS_ADDR        _IOWR(, , u64)
> >
> >
> >
> > /* Return number of IRQs (Not including MSI/MSI-X for PCI) */
> > #define VFIO_DEVICE_GET_NUM_IRQ                 _IOR(, , int)
> >
> > /* Set IRQ eventfd for IRQ index, arg[0] = index, arg[1] = fd */
> > #define VFIO_DEVICE_SET_IRQ_EVENTFD             _IOW(, , int)
> >
> > /* Unmask IRQ index */
> > #define VFIO_DEVICE_UNMASK_IRQ                  _IOW(, , int)
> >
> > /* Set unmask eventfd for index, arg[0] = index, arg[1] = fd */
> > #define VFIO_DEVICE_SET_UNMASK_IRQ_EVENTFD      _IOW(, , int)
> >
> >
> > /* Return the device tree path for type/index into the user
> >  * allocated buffer */
> > struct dtpath {
> >        u32     type; (0 = region, 1 = IRQ)
> >        u32     index;
> >        u32     buf_len;
> >        char    *buf;
> > };
> > #define VFIO_DEVICE_GET_DTPATH                  _IOWR(, , struct dtpath)
> >
> > /* Return the device tree index for type/index */
> > struct dtindex {
> >        u32     type; (0 = region, 1 = IRQ)
> >        u32     index;
> >        u32     prop_type;
> >        u32     prop_index;
> > };
> > #define VFIO_DEVICE_GET_DTINDEX                 _IOWR(, , struct dtindex)
> >
> >
> > /* Reset the device */
> > #define VFIO_DEVICE_RESET                       _IO(, ,)
> >
> >
> > /* PCI MSI setup, arg[0] = #, arg[1-n] = eventfds */
> > #define VFIO_DEVICE_PCI_SET_MSI_EVENTFDS        _IOW(, , int)
> > #define VFIO_DEVICE_PCI_SET_MSIX_EVENTFDS       _IOW(, , int)
> >
> > Hope that covers it.  Something I prefer about this interface is that
> > everything can easily be generated on the fly, whereas reading out a
> > table from the device means we really need to have that table somewhere
> > in kernel memory to easily support reading random offsets.  Thoughts?
> 
> I think this could work, but I'm not sure it makes the problem David
> had any better--  you substitute the complexity of parsing the
> variable length regions with invoking a set of APIs.

I read it as mostly the complexity problem, which I think this makes
fairly trivial.  It also eliminates a lot of complexity on the kernel
side of supporting the table driven interface.  Thanks,

Alex

if (!(GET_FLAGS & PCI))
    return error;

if (GET_NUM_REGIONS < 8)
    return error;

GET_REGION_LEN(7)
GET_REGION_OFFSET(7) // setup config access

for (i = 0; i < 6; i++) {
    if (GET_REGION_LEN(i)) {
        GET_REGION_OFFSET(i)
        setup mmap/rw
    }
}

if (GET_REGION_LEN(6)) {
    GET_REGION_OFFSET(6)
    setup ROM access
}

Re: [Qemu-devel] RFC [v2]: vfio / device assignment -- layout of device fd files

[Scott Wood]

On 09/26/2011 01:34 PM, Alex Williamson wrote:
> The other obvious possibility is a pure ioctl interface.  To match what
> this proposal is trying to describe, plus the runtime interfaces, we'd
> need something like:
> 
> /* :0 - PCI devices, :1 - Devices path device, 63:2 - reserved */
> #define VFIO_DEVICE_GET_FLAGS			_IOR(, , u64)
> 
> 
> /* Return number of mmio/iop/config regions.
>  * For PCI this is always 8 (BAR0-5 + ROM + Config) */
> #define VFIO_DEVICE_GET_NUM_REGIONS		_IOR(, , int)

How do you handle BARs that a particular device doesn't use?  Zero-length?

> /* Return the device tree path for type/index into the user
>  * allocated buffer */
> struct dtpath {
> 	u32	type; (0 = region, 1 = IRQ)
> 	u32	index;
> 	u32	buf_len;
> 	char	*buf;
> };
> #define VFIO_DEVICE_GET_DTPATH			_IOWR(, , struct dtpath)

So now the user needs to guess a buffer length in advance... and what
happens if it's too small?

> /* Reset the device */
> #define VFIO_DEVICE_RESET			_IO(, ,)

What generic way do we have to do this?  We should probably have a way
to determine whether it's possible, without actually asking to do it.

> /* PCI MSI setup, arg[0] = #, arg[1-n] = eventfds */
> #define VFIO_DEVICE_PCI_SET_MSI_EVENTFDS	_IOW(, , int)
> #define VFIO_DEVICE_PCI_SET_MSIX_EVENTFDS	_IOW(, , int)
> 
> Hope that covers it.

It could be done this way, but I predict that the code (both kernel and
user side) will be larger.  Maybe not much more complex, but more
boilerplate.

How will you manage extensions to the interface?  With the table it's
simple, you see a new (sub)record type and you either understand it or
you skip it.  With ioctls you need to call every information-gathering
ioctl you know and care about (or are told is present via some feature
advertisement), and see if there's anything there.

> Something I prefer about this interface is that
> everything can easily be generated on the fly, whereas reading out a
> table from the device means we really need to have that table somewhere
> in kernel memory to easily support reading random offsets.  Thoughts?

The table should not be particularly large, and you'll need to keep the
information around in some form regardless.  Maybe in the PCI case you
could produce it dynamically (though I probably wouldn't), but it really
wouldn't make sense in the device tree case.

You also lose the ability to easily have a human look at the hexdump for
debugging; you'll need a special "lsvfio" tool.  You might want one
anyway to pretty-print the info, but with ioctls it's mandatory.

-Scott

Re: [Qemu-devel] RFC [v2]: vfio / device assignment -- layout of device fd files

[Alex Williamson]

On Mon, 2011-09-26 at 18:59 -0500, Scott Wood wrote:
> On 09/26/2011 01:34 PM, Alex Williamson wrote:
> > The other obvious possibility is a pure ioctl interface.  To match what
> > this proposal is trying to describe, plus the runtime interfaces, we'd
> > need something like:
> > 
> > /* :0 - PCI devices, :1 - Devices path device, 63:2 - reserved */
> > #define VFIO_DEVICE_GET_FLAGS			_IOR(, , u64)
> > 
> > 
> > /* Return number of mmio/iop/config regions.
> >  * For PCI this is always 8 (BAR0-5 + ROM + Config) */
> > #define VFIO_DEVICE_GET_NUM_REGIONS		_IOR(, , int)
> 
> How do you handle BARs that a particular device doesn't use?  Zero-length?

Yep

> > /* Return the device tree path for type/index into the user
> >  * allocated buffer */
> > struct dtpath {
> > 	u32	type; (0 = region, 1 = IRQ)
> > 	u32	index;
> > 	u32	buf_len;
> > 	char	*buf;
> > };
> > #define VFIO_DEVICE_GET_DTPATH			_IOWR(, , struct dtpath)
> 
> So now the user needs to guess a buffer length in advance... and what
> happens if it's too small?

-ENOSPC.  Call with buf_len = 0 and it could indicate the size.

> > /* Reset the device */
> > #define VFIO_DEVICE_RESET			_IO(, ,)
> 
> What generic way do we have to do this?  We should probably have a way
> to determine whether it's possible, without actually asking to do it.

It's not generic, it could be a VFIO_DEVICE_PCI_RESET or we could add a
bit to the device flags to indicate if it's available or we could add a
"probe" arg to the ioctl to either check for existence or do it.

> > /* PCI MSI setup, arg[0] = #, arg[1-n] = eventfds */
> > #define VFIO_DEVICE_PCI_SET_MSI_EVENTFDS	_IOW(, , int)
> > #define VFIO_DEVICE_PCI_SET_MSIX_EVENTFDS	_IOW(, , int)
> > 
> > Hope that covers it.
> 
> It could be done this way, but I predict that the code (both kernel and
> user side) will be larger.  Maybe not much more complex, but more
> boilerplate.
> 
> How will you manage extensions to the interface?

I would assume we'd do something similar to the kvm capabilities checks.
I don't know if that's just bits of GET_FLAGS or a different ioctl.

> With the table it's
> simple, you see a new (sub)record type and you either understand it or
> you skip it.  With ioctls you need to call every information-gathering
> ioctl you know and care about (or are told is present via some feature
> advertisement), and see if there's anything there.

I don't really see much difference between the interfaces here.  You'd
pick and choose which table entries you care about and pick and choose
ioctls.  For one you see it in the table, for the other there's a bit
indicating the capability exists.

> > Something I prefer about this interface is that
> > everything can easily be generated on the fly, whereas reading out a
> > table from the device means we really need to have that table somewhere
> > in kernel memory to easily support reading random offsets.  Thoughts?
> 
> The table should not be particularly large, and you'll need to keep the
> information around in some form regardless.  Maybe in the PCI case you
> could produce it dynamically (though I probably wouldn't), but it really
> wouldn't make sense in the device tree case.

It would be entirely dynamic for PCI, there's no advantage to caching
it.  Even for device tree, if you can't fetch it dynamically, you'd have
to duplicate it between an internal data structure and a buffer reading
the table.

> You also lose the ability to easily have a human look at the hexdump for
> debugging; you'll need a special "lsvfio" tool.  You might want one
> anyway to pretty-print the info, but with ioctls it's mandatory.

I don't think this alone justifies duplicating the data and making it
difficult to parse on both ends.  Chances are we won't need such a tool
for the ioctl interface because it's easier to get it right the first
time ;)

Note that I'm not stuck on this interface, I was just thinking about how
to generate the table last week, it seemed like a pain so I thought I'd
spend a few minutes outlining an ioctl interface... turns out it's not
so bad.  Thanks,

Alex

Re: [Qemu-devel] RFC [v2]: vfio / device assignment -- layout of device fd files

[Scott Wood]

On 09/26/2011 07:45 PM, Alex Williamson wrote:
> On Mon, 2011-09-26 at 18:59 -0500, Scott Wood wrote:
>> On 09/26/2011 01:34 PM, Alex Williamson wrote:
>>> /* Reset the device */
>>> #define VFIO_DEVICE_RESET			_IO(, ,)
>>
>> What generic way do we have to do this?  We should probably have a way
>> to determine whether it's possible, without actually asking to do it.
> 
> It's not generic, it could be a VFIO_DEVICE_PCI_RESET or we could add a
> bit to the device flags to indicate if it's available or we could add a
> "probe" arg to the ioctl to either check for existence or do it.

Even with PCI, isn't this only possible if function-level reset is
supported?  I think we need a flag.

For devices that can't be reset by the kernel, we'll want the ability to
stop/start DMA acccess through the IOMMU (or other bus-specific means),
separate from whether the fd is open.  If a device is assigned to a
partition and that partition gets reset, we'll want to disable DMA
before we re-use the memory, and enable it after the partition has reset
or quiesced the device (which requires the fd to be open).

>>> /* PCI MSI setup, arg[0] = #, arg[1-n] = eventfds */
>>> #define VFIO_DEVICE_PCI_SET_MSI_EVENTFDS	_IOW(, , int)
>>> #define VFIO_DEVICE_PCI_SET_MSIX_EVENTFDS	_IOW(, , int)
>>>
>>> Hope that covers it.
>>
>> It could be done this way, but I predict that the code (both kernel and
>> user side) will be larger.  Maybe not much more complex, but more
>> boilerplate.
>>
>> How will you manage extensions to the interface?
> 
> I would assume we'd do something similar to the kvm capabilities checks.

This information is already built into the data-structure approach.

>> The table should not be particularly large, and you'll need to keep the
>> information around in some form regardless.  Maybe in the PCI case you
>> could produce it dynamically (though I probably wouldn't), but it really
>> wouldn't make sense in the device tree case.
> 
> It would be entirely dynamic for PCI, there's no advantage to caching
> it.  Even for device tree, if you can't fetch it dynamically, you'd have
> to duplicate it between an internal data structure and a buffer reading
> the table.

I don't think we'd need to keep the device tree path/index info around
for anything but the table -- but really, this is a minor consideration.

>> You also lose the ability to easily have a human look at the hexdump for
>> debugging; you'll need a special "lsvfio" tool.  You might want one
>> anyway to pretty-print the info, but with ioctls it's mandatory.
> 
> I don't think this alone justifies duplicating the data and making it
> difficult to parse on both ends.  Chances are we won't need such a tool
> for the ioctl interface because it's easier to get it right the first
> time ;)

It's not just useful for getting the code right, but for e.g. sanity
checking that the devices were bound properly.  I think such a tool
would be generally useful, no matter what the kernel interface ends up
being.  I don't just use lspci to debug the PCI subsystem. :-)

> Note that I'm not stuck on this interface, I was just thinking about how
> to generate the table last week, it seemed like a pain so I thought I'd
> spend a few minutes outlining an ioctl interface... turns out it's not
> so bad.  Thanks,

Yeah, it can work either way, as long as the information's there and
there's a way to add new bits of information, or new bus types, down the
road.  Mainly a matter of aesthetics between the two.

-Scott

Re: [Qemu-devel] RFC [v2]: vfio / device assignment -- layout of device fd files

[Alex Williamson]

On Tue, 2011-09-27 at 16:28 -0500, Scott Wood wrote:
> On 09/26/2011 07:45 PM, Alex Williamson wrote:
> > On Mon, 2011-09-26 at 18:59 -0500, Scott Wood wrote:
> >> On 09/26/2011 01:34 PM, Alex Williamson wrote:
> >>> /* Reset the device */
> >>> #define VFIO_DEVICE_RESET			_IO(, ,)
> >>
> >> What generic way do we have to do this?  We should probably have a way
> >> to determine whether it's possible, without actually asking to do it.
> > 
> > It's not generic, it could be a VFIO_DEVICE_PCI_RESET or we could add a
> > bit to the device flags to indicate if it's available or we could add a
> > "probe" arg to the ioctl to either check for existence or do it.
> 
> Even with PCI, isn't this only possible if function-level reset is
> supported?

There are a couple other things we can do if FLR isn't present (D3hot
transition, secondary bus reset, device specific resets are possible).

>  I think we need a flag.

Ok, PCI has a pci_probe_reset_function() and pci_reset_function().  I'd
probably mimic those in the vfio device ops.  Common vfio code can probe
the reset and set the flag appropriately and we can have a common
VFIO_DEVICE_RESET ioctl that calls into the device ops reset function.

> For devices that can't be reset by the kernel, we'll want the ability to
> stop/start DMA acccess through the IOMMU (or other bus-specific means),
> separate from whether the fd is open.  If a device is assigned to a
> partition and that partition gets reset, we'll want to disable DMA
> before we re-use the memory, and enable it after the partition has reset
> or quiesced the device (which requires the fd to be open).

Maybe this can be accomplished via an iommu_detach_device() to
temporarily disassociate it from the domain.  We could also unmap all
the DMA.  Anyway, a few possibilities.

> >>> /* PCI MSI setup, arg[0] = #, arg[1-n] = eventfds */
> >>> #define VFIO_DEVICE_PCI_SET_MSI_EVENTFDS	_IOW(, , int)
> >>> #define VFIO_DEVICE_PCI_SET_MSIX_EVENTFDS	_IOW(, , int)
> >>>
> >>> Hope that covers it.
> >>
> >> It could be done this way, but I predict that the code (both kernel and
> >> user side) will be larger.  Maybe not much more complex, but more
> >> boilerplate.
> >>
> >> How will you manage extensions to the interface?
> > 
> > I would assume we'd do something similar to the kvm capabilities checks.
> 
> This information is already built into the data-structure approach.

If we define it to be part of the flags, then it's built-in to the ioctl
approach too...

> >> The table should not be particularly large, and you'll need to keep the
> >> information around in some form regardless.  Maybe in the PCI case you
> >> could produce it dynamically (though I probably wouldn't), but it really
> >> wouldn't make sense in the device tree case.
> > 
> > It would be entirely dynamic for PCI, there's no advantage to caching
> > it.  Even for device tree, if you can't fetch it dynamically, you'd have
> > to duplicate it between an internal data structure and a buffer reading
> > the table.
> 
> I don't think we'd need to keep the device tree path/index info around
> for anything but the table -- but really, this is a minor consideration.
> 
> >> You also lose the ability to easily have a human look at the hexdump for
> >> debugging; you'll need a special "lsvfio" tool.  You might want one
> >> anyway to pretty-print the info, but with ioctls it's mandatory.
> > 
> > I don't think this alone justifies duplicating the data and making it
> > difficult to parse on both ends.  Chances are we won't need such a tool
> > for the ioctl interface because it's easier to get it right the first
> > time ;)
> 
> It's not just useful for getting the code right, but for e.g. sanity
> checking that the devices were bound properly.  I think such a tool
> would be generally useful, no matter what the kernel interface ends up
> being.  I don't just use lspci to debug the PCI subsystem. :-)

This is also a minor consideration.  Looking at hexdumps isn't much to
rely on for debugging and if we take the step of writing a tool, it's
not much harder to write for either interface.  The table is more akin
to dumping the data, but I feel the ioctl is easier for how a driver
would probably make use of the data (linear vs random access).

> > Note that I'm not stuck on this interface, I was just thinking about how
> > to generate the table last week, it seemed like a pain so I thought I'd
> > spend a few minutes outlining an ioctl interface... turns out it's not
> > so bad.  Thanks,
> 
> Yeah, it can work either way, as long as the information's there and
> there's a way to add new bits of information, or new bus types, down the
> road.  Mainly a matter of aesthetics between the two.

It'd be nice if David would chime back in since he objected to the
table.  Does an ioctl interface look better?  Alex Graf, any opinions?
Thanks,

Alex

Re: [Qemu-devel] RFC [v2]: vfio / device assignment -- layout of device fd files

[Alexander Graf]

On 28.09.2011, at 04:40, Alex Williamson wrote:

> On Tue, 2011-09-27 at 16:28 -0500, Scott Wood wrote:
>> On 09/26/2011 07:45 PM, Alex Williamson wrote:
>>> On Mon, 2011-09-26 at 18:59 -0500, Scott Wood wrote:
>>>> On 09/26/2011 01:34 PM, Alex Williamson wrote:
>>>>> /* Reset the device */
>>>>> #define VFIO_DEVICE_RESET			_IO(, ,)
>>>> 
>>>> What generic way do we have to do this?  We should probably have a way
>>>> to determine whether it's possible, without actually asking to do it.
>>> 
>>> It's not generic, it could be a VFIO_DEVICE_PCI_RESET or we could add a
>>> bit to the device flags to indicate if it's available or we could add a
>>> "probe" arg to the ioctl to either check for existence or do it.
>> 
>> Even with PCI, isn't this only possible if function-level reset is
>> supported?
> 
> There are a couple other things we can do if FLR isn't present (D3hot
> transition, secondary bus reset, device specific resets are possible).
> 
>> I think we need a flag.
> 
> Ok, PCI has a pci_probe_reset_function() and pci_reset_function().  I'd
> probably mimic those in the vfio device ops.  Common vfio code can probe
> the reset and set the flag appropriately and we can have a common
> VFIO_DEVICE_RESET ioctl that calls into the device ops reset function.
> 
>> For devices that can't be reset by the kernel, we'll want the ability to
>> stop/start DMA acccess through the IOMMU (or other bus-specific means),
>> separate from whether the fd is open.  If a device is assigned to a
>> partition and that partition gets reset, we'll want to disable DMA
>> before we re-use the memory, and enable it after the partition has reset
>> or quiesced the device (which requires the fd to be open).
> 
> Maybe this can be accomplished via an iommu_detach_device() to
> temporarily disassociate it from the domain.  We could also unmap all
> the DMA.  Anyway, a few possibilities.
> 
>>>>> /* PCI MSI setup, arg[0] = #, arg[1-n] = eventfds */
>>>>> #define VFIO_DEVICE_PCI_SET_MSI_EVENTFDS	_IOW(, , int)
>>>>> #define VFIO_DEVICE_PCI_SET_MSIX_EVENTFDS	_IOW(, , int)
>>>>> 
>>>>> Hope that covers it.
>>>> 
>>>> It could be done this way, but I predict that the code (both kernel and
>>>> user side) will be larger.  Maybe not much more complex, but more
>>>> boilerplate.
>>>> 
>>>> How will you manage extensions to the interface?
>>> 
>>> I would assume we'd do something similar to the kvm capabilities checks.
>> 
>> This information is already built into the data-structure approach.
> 
> If we define it to be part of the flags, then it's built-in to the ioctl
> approach too...
> 
>>>> The table should not be particularly large, and you'll need to keep the
>>>> information around in some form regardless.  Maybe in the PCI case you
>>>> could produce it dynamically (though I probably wouldn't), but it really
>>>> wouldn't make sense in the device tree case.
>>> 
>>> It would be entirely dynamic for PCI, there's no advantage to caching
>>> it.  Even for device tree, if you can't fetch it dynamically, you'd have
>>> to duplicate it between an internal data structure and a buffer reading
>>> the table.
>> 
>> I don't think we'd need to keep the device tree path/index info around
>> for anything but the table -- but really, this is a minor consideration.
>> 
>>>> You also lose the ability to easily have a human look at the hexdump for
>>>> debugging; you'll need a special "lsvfio" tool.  You might want one
>>>> anyway to pretty-print the info, but with ioctls it's mandatory.
>>> 
>>> I don't think this alone justifies duplicating the data and making it
>>> difficult to parse on both ends.  Chances are we won't need such a tool
>>> for the ioctl interface because it's easier to get it right the first
>>> time ;)
>> 
>> It's not just useful for getting the code right, but for e.g. sanity
>> checking that the devices were bound properly.  I think such a tool
>> would be generally useful, no matter what the kernel interface ends up
>> being.  I don't just use lspci to debug the PCI subsystem. :-)
> 
> This is also a minor consideration.  Looking at hexdumps isn't much to
> rely on for debugging and if we take the step of writing a tool, it's
> not much harder to write for either interface.  The table is more akin
> to dumping the data, but I feel the ioctl is easier for how a driver
> would probably make use of the data (linear vs random access).
> 
>>> Note that I'm not stuck on this interface, I was just thinking about how
>>> to generate the table last week, it seemed like a pain so I thought I'd
>>> spend a few minutes outlining an ioctl interface... turns out it's not
>>> so bad.  Thanks,
>> 
>> Yeah, it can work either way, as long as the information's there and
>> there's a way to add new bits of information, or new bus types, down the
>> road.  Mainly a matter of aesthetics between the two.
> 
> It'd be nice if David would chime back in since he objected to the
> table.  Does an ioctl interface look better?  Alex Graf, any opinions?

I'm honestly pretty indifferent on ioctl vs. linear read. I got the impression that people dislike ioctls for whatever reason, so we went ahead and did the design based on read(). With KVM, ioctls are a constant pain to extend, but so are the constant sized fields here.

Whatever you do, please introduce a "flags" field to every struct you use and add some padding at the end, so it can possibly be extended.


Alex

Re: [Qemu-devel] RFC [v2]: vfio / device assignment -- layout of device fd files

[David Gibson]

On Wed, Sep 28, 2011 at 10:58:26AM +0200, Alexander Graf wrote:
> On 28.09.2011, at 04:40, Alex Williamson wrote:
> > On Tue, 2011-09-27 at 16:28 -0500, Scott Wood wrote:
[snip]
> I'm honestly pretty indifferent on ioctl vs. linear read. I got the
> impression that people dislike ioctls for whatever reason, so we
> went ahead and did the design based on read(). With KVM, ioctls are
> a constant pain to extend, but so are the constant sized fields
> here.
> 
> Whatever you do, please introduce a "flags" field to every struct
> you use and add some padding at the end, so it can possibly be
> extended.

Yes, those are good ideas.  Plus whenever creating an ioctl() that has
a structure "return value", it's a good idea for the size of the
structure to be specified as an input.  Again this allows later
extension.  Older userspace which only knows the old structure won't
get the new fields, but presumably it wouldn't know or care what to do
with them anyway.


-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

Re: [Qemu-devel] RFC [v2]: vfio / device assignment -- layout of device fd files

[David Gibson]

On Mon, Sep 26, 2011 at 06:59:33PM -0500, Scott Wood wrote:
> On 09/26/2011 01:34 PM, Alex Williamson wrote:
> > The other obvious possibility is a pure ioctl interface.  To match what
> > this proposal is trying to describe, plus the runtime interfaces, we'd
> > need something like:
> > 
> > /* :0 - PCI devices, :1 - Devices path device, 63:2 - reserved */
> > #define VFIO_DEVICE_GET_FLAGS			_IOR(, , u64)
> > 
> > 
> > /* Return number of mmio/iop/config regions.
> >  * For PCI this is always 8 (BAR0-5 + ROM + Config) */
> > #define VFIO_DEVICE_GET_NUM_REGIONS		_IOR(, , int)
> 
> How do you handle BARs that a particular device doesn't use?  Zero-length?
> 
> > /* Return the device tree path for type/index into the user
> >  * allocated buffer */
> > struct dtpath {
> > 	u32	type; (0 = region, 1 = IRQ)
> > 	u32	index;
> > 	u32	buf_len;
> > 	char	*buf;
> > };
> > #define VFIO_DEVICE_GET_DTPATH			_IOWR(, , struct dtpath)
> 
> So now the user needs to guess a buffer length in advance... and what
> happens if it's too small?
> 
> > /* Reset the device */
> > #define VFIO_DEVICE_RESET			_IO(, ,)
> 
> What generic way do we have to do this?  We should probably have a way
> to determine whether it's possible, without actually asking to do it.

That's a good point.  PCI devices have a standardized reset, but
embedded devices often won't.  Mind you we could just fail the call in
that cse.

> > /* PCI MSI setup, arg[0] = #, arg[1-n] = eventfds */
> > #define VFIO_DEVICE_PCI_SET_MSI_EVENTFDS	_IOW(, , int)
> > #define VFIO_DEVICE_PCI_SET_MSIX_EVENTFDS	_IOW(, , int)
> > 
> > Hope that covers it.
> 
> It could be done this way, but I predict that the code (both kernel and
> user side) will be larger.  Maybe not much more complex, but more
> boilerplate.
> 
> How will you manage extensions to the interface?  With the table it's
> simple, you see a new (sub)record type and you either understand it or
> you skip it.  With ioctls you need to call every information-gathering
> ioctl you know and care about (or are told is present via some feature
> advertisement), and see if there's anything there.

No.. quite the opposite.  With ioctl()s you call the ones your
userspace program cares about / can implement.  When an extended
interface is added, they keep working as is.  Newer userspace which
uses the new features will call the new ioctls() if it cares about
them.

> > Something I prefer about this interface is that
> > everything can easily be generated on the fly, whereas reading out a
> > table from the device means we really need to have that table somewhere
> > in kernel memory to easily support reading random offsets.  Thoughts?
> 
> The table should not be particularly large, and you'll need to keep the
> information around in some form regardless.  Maybe in the PCI case you
> could produce it dynamically (though I probably wouldn't), but it really
> wouldn't make sense in the device tree case.
> 
> You also lose the ability to easily have a human look at the hexdump for
> debugging; you'll need a special "lsvfio" tool.  You might want one
> anyway to pretty-print the info, but with ioctls it's mandatory.
> 
> -Scott
> 

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

Re: [Qemu-devel] RFC [v2]: vfio / device assignment -- layout of device fd files

[David Gibson]

On Mon, Sep 26, 2011 at 12:34:52PM -0600, Alex Williamson wrote:
> On Mon, 2011-09-26 at 12:04 +0200, Alexander Graf wrote:
> > Am 26.09.2011 um 09:51 schrieb David Gibson <david@gibson.dropbear.id.au>:
[snip]
> > Also, if you can come up with an interface that does not have variable
> > length descriptors but is still able to export all the required
> > generic information, please send a proposal to the list :)
> > 
> 
> Hi,
> 
> The other obvious possibility is a pure ioctl interface.  To match what
> this proposal is trying to describe, plus the runtime interfaces, we'd
> need something like:

Right, this also seems a reasonable possibility to me, depending on
the details.

> /* :0 - PCI devices, :1 - Devices path device, 63:2 - reserved */
> #define VFIO_DEVICE_GET_FLAGS			_IOR(, , u64)
> 
> 
> /* Return number of mmio/iop/config regions.
>  * For PCI this is always 8 (BAR0-5 + ROM + Config) */
> #define VFIO_DEVICE_GET_NUM_REGIONS		_IOR(, , int)
> 
> /* Return length for region index (may be zero) */
> #define VFIO_DEVICE_GET_REGION_LEN		_IOWR(, , u64)
> 
> /* Return flags for region index
>  * :0 - mmap'able, :1 - read-only, 63:2 - reserved */
> #define VFIO_DEVICE_GET_REGION_FLAGS		_IOR(, , u64)
> 
> /* Return file offset for region index */
> #define VFIO_DEVICE_GET_REGION_OFFSET		_IOWR(, , u64)

The above 3 can be be folded into one "getregioninfo" call.

> /* Return physical address for region index - not implemented for PCI */
> #define VFIO_DEVICE_GET_REGION_PHYS_ADDR	_IOWR(, , u64)
> 
> 
> 
> /* Return number of IRQs (Not including MSI/MSI-X for PCI) */
> #define VFIO_DEVICE_GET_NUM_IRQ			_IOR(, , int)
> 
> /* Set IRQ eventfd for IRQ index, arg[0] = index, arg[1] = fd */
> #define VFIO_DEVICE_SET_IRQ_EVENTFD		_IOW(, , int)
> 
> /* Unmask IRQ index */
> #define VFIO_DEVICE_UNMASK_IRQ			_IOW(, , int)
> 
> /* Set unmask eventfd for index, arg[0] = index, arg[1] = fd */
> #define VFIO_DEVICE_SET_UNMASK_IRQ_EVENTFD	_IOW(, , int)
> 
> 
> /* Return the device tree path for type/index into the user
>  * allocated buffer */
> struct dtpath {
> 	u32	type; (0 = region, 1 = IRQ)
> 	u32	index;
> 	u32	buf_len;
> 	char	*buf;
> };
> #define VFIO_DEVICE_GET_DTPATH			_IOWR(, , struct dtpath)
> 
> /* Return the device tree index for type/index */
> struct dtindex {
> 	u32	type; (0 = region, 1 = IRQ)
> 	u32	index;
> 	u32	prop_type;
> 	u32	prop_index;
> };
> #define VFIO_DEVICE_GET_DTINDEX			_IOWR(, , struct dtindex)

I think those need some work, but that doesn't impinge on the core
semantics.

> /* Reset the device */
> #define VFIO_DEVICE_RESET			_IO(, ,)
> 
> 
> /* PCI MSI setup, arg[0] = #, arg[1-n] = eventfds */
> #define VFIO_DEVICE_PCI_SET_MSI_EVENTFDS	_IOW(, , int)
> #define VFIO_DEVICE_PCI_SET_MSIX_EVENTFDS	_IOW(, , int)

Why does this need seperate controls, rather than just treating MSIs
as interrupts beyond the first for PCI devices?

> Hope that covers it.  Something I prefer about this interface is that
> everything can easily be generated on the fly, whereas reading out a
> table from the device means we really need to have that table somewhere
> in kernel memory to easily support reading random offsets.
> Thoughts?

I certainly prefer it to the previous proposal.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

Re: [Qemu-devel] RFC [v2]: vfio / device assignment -- layout of device fd files

[Alex Williamson]

On Fri, 2011-09-30 at 18:46 +1000, David Gibson wrote:
> On Mon, Sep 26, 2011 at 12:34:52PM -0600, Alex Williamson wrote:
> > On Mon, 2011-09-26 at 12:04 +0200, Alexander Graf wrote:
> > > Am 26.09.2011 um 09:51 schrieb David Gibson <david@gibson.dropbear.id.au>:
> [snip]
> > > Also, if you can come up with an interface that does not have variable
> > > length descriptors but is still able to export all the required
> > > generic information, please send a proposal to the list :)
> > > 
> > 
> > Hi,
> > 
> > The other obvious possibility is a pure ioctl interface.  To match what
> > this proposal is trying to describe, plus the runtime interfaces, we'd
> > need something like:
> 
> Right, this also seems a reasonable possibility to me, depending on
> the details.
> 
> > /* :0 - PCI devices, :1 - Devices path device, 63:2 - reserved */
> > #define VFIO_DEVICE_GET_FLAGS			_IOR(, , u64)
> > 
> > 
> > /* Return number of mmio/iop/config regions.
> >  * For PCI this is always 8 (BAR0-5 + ROM + Config) */
> > #define VFIO_DEVICE_GET_NUM_REGIONS		_IOR(, , int)
> > 
> > /* Return length for region index (may be zero) */
> > #define VFIO_DEVICE_GET_REGION_LEN		_IOWR(, , u64)
> > 
> > /* Return flags for region index
> >  * :0 - mmap'able, :1 - read-only, 63:2 - reserved */
> > #define VFIO_DEVICE_GET_REGION_FLAGS		_IOR(, , u64)
> > 
> > /* Return file offset for region index */
> > #define VFIO_DEVICE_GET_REGION_OFFSET		_IOWR(, , u64)
> 
> The above 3 can be be folded into one "getregioninfo" call.

Yep, and the phys addr one below.  We can use a flags bit to indicate
whether it's valid.

> > /* Return physical address for region index - not implemented for PCI */
> > #define VFIO_DEVICE_GET_REGION_PHYS_ADDR	_IOWR(, , u64)
> > 
> > 
> > 
> > /* Return number of IRQs (Not including MSI/MSI-X for PCI) */
> > #define VFIO_DEVICE_GET_NUM_IRQ			_IOR(, , int)
> > 
> > /* Set IRQ eventfd for IRQ index, arg[0] = index, arg[1] = fd */
> > #define VFIO_DEVICE_SET_IRQ_EVENTFD		_IOW(, , int)
> > 
> > /* Unmask IRQ index */
> > #define VFIO_DEVICE_UNMASK_IRQ			_IOW(, , int)
> > 
> > /* Set unmask eventfd for index, arg[0] = index, arg[1] = fd */
> > #define VFIO_DEVICE_SET_UNMASK_IRQ_EVENTFD	_IOW(, , int)
> > 
> > 
> > /* Return the device tree path for type/index into the user
> >  * allocated buffer */
> > struct dtpath {
> > 	u32	type; (0 = region, 1 = IRQ)
> > 	u32	index;
> > 	u32	buf_len;
> > 	char	*buf;
> > };
> > #define VFIO_DEVICE_GET_DTPATH			_IOWR(, , struct dtpath)
> > 
> > /* Return the device tree index for type/index */
> > struct dtindex {
> > 	u32	type; (0 = region, 1 = IRQ)
> > 	u32	index;
> > 	u32	prop_type;
> > 	u32	prop_index;
> > };
> > #define VFIO_DEVICE_GET_DTINDEX			_IOWR(, , struct dtindex)
> 
> I think those need some work, but that doesn't impinge on the core
> semantics.
> 
> > /* Reset the device */
> > #define VFIO_DEVICE_RESET			_IO(, ,)
> > 
> > 
> > /* PCI MSI setup, arg[0] = #, arg[1-n] = eventfds */
> > #define VFIO_DEVICE_PCI_SET_MSI_EVENTFDS	_IOW(, , int)
> > #define VFIO_DEVICE_PCI_SET_MSIX_EVENTFDS	_IOW(, , int)
> 
> Why does this need seperate controls, rather than just treating MSIs
> as interrupts beyond the first for PCI devices?

Well, we could say that PCI will always report 3 for
VFIO_DEVICE_GET_NUM_IRQ where 0 = legacy, 1 = MSI, 2 = MSI-X.  ioctls on
unimplemented IRQs will fail, UNMASK* ioctls on non-level triggered
interrupts will fail, and the parameter to SET_IRQ_EVENTFD becomes
arg[0] = index, arg[1] = count, arg[2-n] = fd.  Maybe we'd then have a
GET_IRQ_INFO that takes something like:

struct vfio_irq_info {
	int index;
	unsigned int count;
	u64 flags;
#define VFIO_IRQ_INFO_FLAGS_LEVEL	(1 << 0)
};

count would be 0 on PCI if the type of interrupt isn't supported.
Better?  Thanks,

Alex

> > Hope that covers it.  Something I prefer about this interface is that
> > everything can easily be generated on the fly, whereas reading out a
> > table from the device means we really need to have that table somewhere
> > in kernel memory to easily support reading random offsets.
> > Thoughts?
> 
> I certainly prefer it to the previous proposal.

Re: [Qemu-devel] RFC [v2]: vfio / device assignment -- layout of device fd files

[Alex Williamson]

On Fri, 2011-09-30 at 10:37 -0600, Alex Williamson wrote:
> On Fri, 2011-09-30 at 18:46 +1000, David Gibson wrote:
> > On Mon, Sep 26, 2011 at 12:34:52PM -0600, Alex Williamson wrote:
> > > On Mon, 2011-09-26 at 12:04 +0200, Alexander Graf wrote:
> > > > Am 26.09.2011 um 09:51 schrieb David Gibson <david@gibson.dropbear.id.au>:
> > [snip]
> > > > Also, if you can come up with an interface that does not have variable
> > > > length descriptors but is still able to export all the required
> > > > generic information, please send a proposal to the list :)
> > > > 
> > > 
> > > Hi,
> > > 
> > > The other obvious possibility is a pure ioctl interface.  To match what
> > > this proposal is trying to describe, plus the runtime interfaces, we'd
> > > need something like:
> > 
> > Right, this also seems a reasonable possibility to me, depending on
> > the details.
> > 
> > > /* :0 - PCI devices, :1 - Devices path device, 63:2 - reserved */
> > > #define VFIO_DEVICE_GET_FLAGS			_IOR(, , u64)
> > > 
> > > 
> > > /* Return number of mmio/iop/config regions.
> > >  * For PCI this is always 8 (BAR0-5 + ROM + Config) */
> > > #define VFIO_DEVICE_GET_NUM_REGIONS		_IOR(, , int)
> > > 
> > > /* Return length for region index (may be zero) */
> > > #define VFIO_DEVICE_GET_REGION_LEN		_IOWR(, , u64)
> > > 
> > > /* Return flags for region index
> > >  * :0 - mmap'able, :1 - read-only, 63:2 - reserved */
> > > #define VFIO_DEVICE_GET_REGION_FLAGS		_IOR(, , u64)
> > > 
> > > /* Return file offset for region index */
> > > #define VFIO_DEVICE_GET_REGION_OFFSET		_IOWR(, , u64)
> > 
> > The above 3 can be be folded into one "getregioninfo" call.
> 
> Yep, and the phys addr one below.  We can use a flags bit to indicate
> whether it's valid.
> 
> > > /* Return physical address for region index - not implemented for PCI */
> > > #define VFIO_DEVICE_GET_REGION_PHYS_ADDR	_IOWR(, , u64)
> > > 
> > > 
> > > 
> > > /* Return number of IRQs (Not including MSI/MSI-X for PCI) */
> > > #define VFIO_DEVICE_GET_NUM_IRQ			_IOR(, , int)
> > > 
> > > /* Set IRQ eventfd for IRQ index, arg[0] = index, arg[1] = fd */
> > > #define VFIO_DEVICE_SET_IRQ_EVENTFD		_IOW(, , int)
> > > 
> > > /* Unmask IRQ index */
> > > #define VFIO_DEVICE_UNMASK_IRQ			_IOW(, , int)
> > > 
> > > /* Set unmask eventfd for index, arg[0] = index, arg[1] = fd */
> > > #define VFIO_DEVICE_SET_UNMASK_IRQ_EVENTFD	_IOW(, , int)
> > > 
> > > 
> > > /* Return the device tree path for type/index into the user
> > >  * allocated buffer */
> > > struct dtpath {
> > > 	u32	type; (0 = region, 1 = IRQ)
> > > 	u32	index;
> > > 	u32	buf_len;
> > > 	char	*buf;
> > > };
> > > #define VFIO_DEVICE_GET_DTPATH			_IOWR(, , struct dtpath)
> > > 
> > > /* Return the device tree index for type/index */
> > > struct dtindex {
> > > 	u32	type; (0 = region, 1 = IRQ)
> > > 	u32	index;
> > > 	u32	prop_type;
> > > 	u32	prop_index;
> > > };
> > > #define VFIO_DEVICE_GET_DTINDEX			_IOWR(, , struct dtindex)
> > 
> > I think those need some work, but that doesn't impinge on the core
> > semantics.
> > 
> > > /* Reset the device */
> > > #define VFIO_DEVICE_RESET			_IO(, ,)
> > > 
> > > 
> > > /* PCI MSI setup, arg[0] = #, arg[1-n] = eventfds */
> > > #define VFIO_DEVICE_PCI_SET_MSI_EVENTFDS	_IOW(, , int)
> > > #define VFIO_DEVICE_PCI_SET_MSIX_EVENTFDS	_IOW(, , int)
> > 
> > Why does this need seperate controls, rather than just treating MSIs
> > as interrupts beyond the first for PCI devices?
> 
> Well, we could say that PCI will always report 3 for
> VFIO_DEVICE_GET_NUM_IRQ where 0 = legacy, 1 = MSI, 2 = MSI-X.  ioctls on
> unimplemented IRQs will fail, UNMASK* ioctls on non-level triggered
> interrupts will fail, and the parameter to SET_IRQ_EVENTFD becomes
> arg[0] = index, arg[1] = count, arg[2-n] = fd.  Maybe we'd then have a
> GET_IRQ_INFO that takes something like:
> 
> struct vfio_irq_info {
> 	int index;
> 	unsigned int count;
> 	u64 flags;
> #define VFIO_IRQ_INFO_FLAGS_LEVEL	(1 << 0)
> };
> 
> count would be 0 on PCI if the type of interrupt isn't supported.
> Better?  Thanks,

FYI for all, I've pushed a branch out to github with the current state
of the re-write.  You can find it here

https://awilliam@github.com/awilliam/linux-vfio.git
git://github.com/awilliam/linux-vfio.git

The vfio-ng branch is the latest.  The framework is quite a bit more
solid now, so I figure it's time to move into the device and iommu
implementation.  vfio-pci is now it's own module that depends on vfio, I
expect vfio-dt to be implemented the same.  The PCI ioctl is in place
and supports the interface described above.  I'll continue to port
pieces of the old vfio code into this new infrastructure.  Comments and
patches welcome.  Thanks,

Alex

Re: [Qemu-devel] RFC [v2]: vfio / device assignment -- layout of device fd files

[David Gibson]

On Mon, Sep 26, 2011 at 12:04:47PM +0200, Alexander Graf wrote:
> Am 26.09.2011 um 09:51 schrieb David Gibson <david@gibson.dropbear.id.au>:
[snip]
> > Um, not to put too fine a point on it, this is madness.
> > 
> > Yes, it's very flexible and can thereby cover a very wide range of
> > cases.  But it's much, much too complex.  Userspace has to parse a
> > complex, multilayered data structure, with variable length elements
> > just to get an address at which to do IO.  I can pretty much guarantee
> > that if we went with this, most userspace programs using this
> > interface would just ignore this metadata and directly map the
> > offsets at which they happen to know the kernel will put things for
> > the type of device they care about.
> > 
> > _At least_ for PCI, I think the original VFIO layout of each BAR at a
> > fixed, well known offset is much better.  Despite its limitations,
> > just advertising a "device type" ID which describes one of a few fixed
> > layouts would be preferable to this.  I'm still hoping, that we can do
> > a bit better than that.  But we should try really hard to at the very
> > least force the metadata into a simple array of resources each with a
> > fixed size record describing it, even if it means some space wastage
> > with occasionally-used fields.  Anything more complex than that and
> > userspace is just never going to use it properly.
> 
> We will have 2 different types of user space. One wants to be as
> generic as possible and needs all this dynamic information. QEMU
> would fall into this category.
> 
> The other one is device specific drivers in user space. Here
> hardcoding might make sense.
> 
> For the generic interface, we need something that us as verbose as
> possible and lets us enumerate all the device properties, so we can
> properly map and forward them to the guest.
> 
> However, nothing keeps us from mapping BARs always at static offsets
> into the file. If you don't need the generic info, then you don't
> need it.

This sounds dangerous to me.  I can just see some future kernel hacker
going "heey, Ican rearrange these, their locations are all advertised,
right".

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

Re: [Qemu-devel] RFC [v2]: vfio / device assignment -- layout of device fd files

[Stuart Yoder]

On Mon, Sep 26, 2011 at 2:51 AM, David Gibson
<david@gibson.dropbear.id.au> wrote:
> On Fri, Sep 09, 2011 at 08:11:54AM -0500, Stuart Yoder wrote:
>> Based on the discussions over the last couple of weeks
>> I have updated the device fd file layout proposal and
>> tried to specify it a bit more formally.
>>
>> ===============================================================
>>
>> 1.  Overview
>>
>>   This specification describes the layout of device files
>>   used in the context of vfio, which gives user space
>>   direct access to I/O devices that have been bound to
>>   vfio.
>>
>>   When a device fd is opened and read, offset 0x0 contains
>>   a fixed sized header followed by a number of variable length
>>   records that describe different characteristics
>>   of the device-- addressable regions, interrupts, etc.
>>
>>   0x0  +-------------+-------------+
>>        |         magic             | u32  // identifies this as a vfio
>> device file
>>        +---------------------------+         and identifies the type of bus
>>        |         version           | u32  // specifies the version of this
>>        +---------------------------+
>>        |         flags             | u32  // encodes any flags
>>        +---------------------------+
>>        |  dev info record 0        |
>>        |    type                   | u32   // type of record
>>        |    rec_len                | u32   // length in bytes of record
>>        |                           |          (including record header)
>>        |    flags                  | u32   // type specific flags
>>        |    ...content...          |       // record content, which could
>>        +---------------------------+       // include sub-records
>>        |  dev info record 1        |
>>        +---------------------------+
>>        |  dev info record N        |
>>        +---------------------------+
>
> I really should have chimed in on this earlier, but I've been very
> busy.
>
> Um, not to put too fine a point on it, this is madness.
>
> Yes, it's very flexible and can thereby cover a very wide range of
> cases.  But it's much, much too complex.  Userspace has to parse a
> complex, multilayered data structure, with variable length elements
> just to get an address at which to do IO.  I can pretty much guarantee
> that if we went with this, most userspace programs using this
> interface would just ignore this metadata and directly map the
> offsets at which they happen to know the kernel will put things for
> the type of device they care about.
>
> _At least_ for PCI, I think the original VFIO layout of each BAR at a
> fixed, well known offset is much better.  Despite its limitations,
> just advertising a "device type" ID which describes one of a few fixed
> layouts would be preferable to this.  I'm still hoping, that we can do
> a bit better than that.  But we should try really hard to at the very
> least force the metadata into a simple array of resources each with a
> fixed size record describing it, even if it means some space wastage
> with occasionally-used fields.  Anything more complex than that and
> userspace is just never going to use it properly.

So, is your issue really the variable length nature of what was
proposed?

I don't think it would be that hard to make the different resources
fixed length.   I think we have 2 types of resources now-- address
regions and interrupts.

The only thing that get's a bit tricky is device tree paths, which
are obviously variable length.

We could put a description of all the resources in an array with
each element being something like 4KB??

Stuart

Re: [Qemu-devel] RFC [v2]: vfio / device assignment -- layout of device fd files

[Scott Wood]

On 09/26/2011 02:57 PM, Stuart Yoder wrote:
> On Mon, Sep 26, 2011 at 2:51 AM, David Gibson
>> Um, not to put too fine a point on it, this is madness.
>>
>> Yes, it's very flexible and can thereby cover a very wide range of
>> cases.  But it's much, much too complex.  Userspace has to parse a
>> complex, multilayered data structure, with variable length elements
>> just to get an address at which to do IO.  I can pretty much guarantee
>> that if we went with this, most userspace programs using this
>> interface would just ignore this metadata and directly map the
>> offsets at which they happen to know the kernel will put things for
>> the type of device they care about.

Then they deserve to break when those offsets change, just like with all
the other bad assumptions poorly written code tends to make.

Really, it should not be that hard to parse this.  A loop with a switch
statement for toplevel records, and another loop with a switch statement
for each context in which subrecords can appear.

>> _At least_ for PCI, I think the original VFIO layout of each BAR at a
>> fixed, well known offset is much better.

This is what grew out of attempting to address the needs of assigning
non-PCI devices based on the device tree.  Personally, I'd rather be
dealing with this for PCI devices as well.

>>  Despite its limitations,
>> just advertising a "device type" ID which describes one of a few fixed
>> layouts would be preferable to this.

That's already more information than the original VFIO layout provided.

> So, is your issue really the variable length nature of what was
> proposed?
> 
> I don't think it would be that hard to make the different resources
> fixed length.   I think we have 2 types of resources now-- address
> regions and interrupts.
> 
> The only thing that get's a bit tricky is device tree paths, which
> are obviously variable length.
> 
> We could put a description of all the resources in an array with
> each element being something like 4KB??

I'm not sure what that would improve.  If the user wants to put a limit
on the size of a certain field it's willing to handle, so be it.  It'll
break in the same set of cases that would be unexpressable with such a
limitation in the interface, except that since the breakage is not in
the interface, it's more easily fixable.

-Scott