Re: [Qemu-devel] [PATCH] msi/msix: added API to set MSI message address and data

[Alexey Kardashevskiy <aik@ozlabs.ru>]

On 21/06/12 21:49, Jan Kiszka wrote:
> On 2012-06-21 13:39, Alexey Kardashevskiy wrote:
>> Added (msi|msix)_set_message() functions.
>>
>> Currently msi_notify()/msix_notify() write to these vectors to
>> signal the guest about an interrupt so the correct values have to
>> written there by the guest or QEMU.
>>
>> For example, POWER guest never initializes MSI/MSIX vectors, instead
>> it uses RTAS hypercalls. So in order to support MSIX for virtio-pci on
>> POWER we have to initialize MSI/MSIX message from QEMU.
>>
>> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
>> ---
>>  hw/msi.c  |   13 +++++++++++++
>>  hw/msi.h  |    1 +
>>  hw/msix.c |    9 +++++++++
>>  hw/msix.h |    2 ++
>>  4 files changed, 25 insertions(+)
>>
>> diff --git a/hw/msi.c b/hw/msi.c
>> index 5233204..cc6102f 100644
>> --- a/hw/msi.c
>> +++ b/hw/msi.c
>> @@ -105,6 +105,19 @@ static inline uint8_t msi_pending_off(const PCIDevice* dev, bool msi64bit)
>>      return dev->msi_cap + (msi64bit ? PCI_MSI_PENDING_64 : PCI_MSI_PENDING_32);
>>  }
>>  
>> +void msi_set_message(PCIDevice *dev, MSIMessage msg)
>> +{
>> +    uint16_t flags = pci_get_word(dev->config + msi_flags_off(dev));
>> +    bool msi64bit = flags & PCI_MSI_FLAGS_64BIT;
>> +
>> +    if (msi64bit) {
>> +        pci_set_quad(dev->config + msi_address_lo_off(dev), msg.address);
>> +    } else {
>> +        pci_set_long(dev->config + msi_address_lo_off(dev), msg.address);
>> +    }
>> +    pci_set_word(dev->config + msi_data_off(dev, msi64bit), msg.data);
>> +}
>> +
>>  bool msi_enabled(const PCIDevice *dev)
>>  {
>>      return msi_present(dev) &&
>> diff --git a/hw/msi.h b/hw/msi.h
>> index 75747ab..6ec1f99 100644
>> --- a/hw/msi.h
>> +++ b/hw/msi.h
>> @@ -31,6 +31,7 @@ struct MSIMessage {
>>  
>>  extern bool msi_supported;
>>  
>> +void msi_set_message(PCIDevice *dev, MSIMessage msg);
>>  bool msi_enabled(const PCIDevice *dev);
>>  int msi_init(struct PCIDevice *dev, uint8_t offset,
>>               unsigned int nr_vectors, bool msi64bit, bool msi_per_vector_mask);
>> diff --git a/hw/msix.c b/hw/msix.c
>> index ded3c55..5f7d6d3 100644
>> --- a/hw/msix.c
>> +++ b/hw/msix.c
>> @@ -45,6 +45,15 @@ static MSIMessage msix_get_message(PCIDevice *dev, unsigned vector)
>>      return msg;
>>  }
>>  
>> +void msix_set_message(PCIDevice *dev, int vector, struct MSIMessage msg)
>> +{
>> +    uint8_t *table_entry = dev->msix_table_page + vector * PCI_MSIX_ENTRY_SIZE;
>> +
>> +    pci_set_quad(table_entry + PCI_MSIX_ENTRY_LOWER_ADDR, msg.address);
>> +    pci_set_long(table_entry + PCI_MSIX_ENTRY_DATA, msg.data);
>> +    table_entry[PCI_MSIX_ENTRY_VECTOR_CTRL] &= ~PCI_MSIX_ENTRY_CTRL_MASKBIT;
>> +}
>> +
>>  /* Add MSI-X capability to the config space for the device. */
>>  /* Given a bar and its size, add MSI-X table on top of it
>>   * and fill MSI-X capability in the config space.
>> diff --git a/hw/msix.h b/hw/msix.h
>> index 50aee82..26a437e 100644
>> --- a/hw/msix.h
>> +++ b/hw/msix.h
>> @@ -4,6 +4,8 @@
>>  #include "qemu-common.h"
>>  #include "pci.h"
>>  
>> +void msix_set_message(PCIDevice *dev, int vector, MSIMessage msg);
>> +
>>  int msix_init(PCIDevice *pdev, unsigned short nentries,
>>                MemoryRegion *bar,
>>                unsigned bar_nr, unsigned bar_size);
>>
> 
> Interface looks good as fas as I can tell (can't asses the POWER need
> for clearing the mask bit on msix_set_message).


I do not know exactly how x86 works (who/how allocates addresses for MSI/MSIX). On POWER at the
moment I did the following thing in QEMU:

- registered memory_region_init_io at some big address which the guest won't use, it is just for QEMU
- put address from the previous step to the MSIX BAR via msix_set_message() when msi is being configured
- then the sequence looks like:
	- vfio_msi_interrupt() calls msix_notify()
	- msix_notify() checks if it is masked via msix_is_masked() - and here PCI_MSIX_ENTRY_CTRL_MASKBIT
must be unset
	- stl_le_phys() - here I get a notification in my MemoryRegionOps::write() and do qemu_irq_pulse()

2 reasons to do that:
1) I did not have to change either msix or vfio - cool for submitting patches;
2) neither POWER guest or qemu changes the msi or msix PCI config (it is done by different mechanism
called RTAS), so I have to do this myself to support 1) and I do not have to care about someone
breaking my settings


>> -- 
>> 1.7.10
>>
>> ps. double '-' and git version is an end-of-patch scissor as I read somewhere, cannot recall where exactly 
> 
> Check man git-am.

Ahhh. Confused end-of-message with end-of-patch. I'll repost it.



-- 
Alexey