From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from [140.186.70.92] (port=57578 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1ONXNd-0002lU-Sx for qemu-devel@nongnu.org; Sat, 12 Jun 2010 16:32:39 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1ONXNZ-0003SD-TT for qemu-devel@nongnu.org; Sat, 12 Jun 2010 16:32:35 -0400 Received: from mail-pw0-f45.google.com ([209.85.160.45]:41537) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1ONXNZ-0003S5-Kp for qemu-devel@nongnu.org; Sat, 12 Jun 2010 16:32:33 -0400 Received: by pwj8 with SMTP id 8so1695027pwj.4 for ; Sat, 12 Jun 2010 13:32:32 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: References: <201006121515.17695.paul@codesourcery.com> <201006121658.58928.paul@codesourcery.com> From: Blue Swirl Date: Sat, 12 Jun 2010 20:32:12 +0000 Message-ID: Subject: Re: [Qemu-devel] [PATCH 09/16] Enable message delivery via IRQs Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Paul Brook Cc: Jan Kiszka , Jan Kiszka , qemu-devel@nongnu.org, Gleb Natapov , Juan Quintela On Sat, Jun 12, 2010 at 7:33 PM, Blue Swirl wrote: > On Sat, Jun 12, 2010 at 3:58 PM, Paul Brook wrote= : >>> I think message passing interrupts >>> are only used in bus based systems, like PCI or UPA/JBUS etc. I don't >>> know how LAPIC/IOAPIC bus works, it could be similar. >> >> PCI Message Signalled Interrupts use a regular data write, and we model = it >> exactly that way. Under normal circumstances you program the device to w= rite >> to memory mapped APIC control registers, but there's no real reason why = you >> couldn't write to RAM. >> >> LAPIC/IOAPIC have their own bus. On some hardware this is multiplexed ov= er the >> main system bus, but logically it is a separate interconnect. >> >> The fact that these a bus based systems suggests against using qemu_irq,= which >> is an inherently point-point system. >> >>> > How is a >>> > receiver meant to know for format of the message being delivered, and= who >>> > it's intended for? >>> >>> This depends on the architecture. On Sparc64 the message is specified >>> by the guest or OF. >>>... >>> In real HW (MSI or Sparc64 mondo interrupts) a bus delivers the >>> message. But our buses only handle address and data lines. >> >> IIUC you're trying to use qemu_irq as a generic interconnect between dev= ices. >> I think that's going to cause more problems than it solves. =C2=A0If a b= us has >> additional message passing capabilities then this can be part of the bus >> interface. >> >> If two devices need a private communication channel then we probably wan= t some >> implementation agnostic way of defining this link. =C2=A0Connecting LAPI= C to IOAPIC >> is a potential example of this. However this needs to be done in a type-= safe >> manner. =C2=A0DMA channels may be another potential use of this linking.= We'd want >> to avoid connecting a DMA channel to an APIC. >> >> [*] A simple unidirectional dma request line is suitable for qmu_irq. A = DMA >> system that transfers data outside of memory read/write transactions is = not. >> e.g. ISA effectively defines a regular memory bus plus 8 bidirectional d= ata >> streams (aka DMA channels). These are multiplexed over the same physical= pins, >> but logically distinct. The DMA channels could either be bundled up into= the >> ISA bus interface, or modelled as links between devices and the DMA >> controller. > > Very very interesting. There's some out of band data related to DMA > (bus errors, IOMMU faults, ISA DREQ) which hasn't been covered in > previous Generic DMA proposals. Maybe all we need is a generic side > band link, which would be usable for point to point (qemu_irq, > coalescing, messages) and bus channels? I think we could solve all problems (well, maybe not world peace, yet) by switching to message based system for all of DMA and IRQs. Each device would have a message input port and way to output messages. Examples: Zero copy memory access from device D1 to D2 to host memory (D3) with access broken to page length units and errors occurring on the last byte: D1 send_msg(ID, MSG_MEM_WRITE, DMA address, length) -> D2 D2 send_msg(ID, MSG_MEM_WRITE, DMA address2, length) -> D3 D3 send_replymsg(ID, MSG_MEM_PTR, host address, 4096) -> D2 D2 send_replymsg(ID, MSG_MEM_PTR, host address, 4096) -> D1 D3 send_replymsg(ID, MSG_MEM_PTR, host address, length - 4096 - 1) -> D2 D2 send_replymsg(ID, MSG_MEM_PTR, host address, length - 4096 - 1) -> D1 D3 send_replymsg(ID, MSG_MEM_ERROR, DMA address2 + length - 1, 1, status) -= > D2 D2 send_replymsg(ID, MSG_MEM_ERROR, DMA address + length - 1, 1, status) ->= D1 IRQ delivery chain D1->D2->D3 with coalescing, messages, delivery reporting and EOI: D1 send_msg(ID, MSG_IRQ_RAISE, payload) -> D2 D2 send_msg(ID, MSG_IRQ_RAISE, payload) -> D3 D3 send_replymsg(ID, MSG_IRQ_STATUS, MI_COALESCED) -> D2 D2 send_replymsg(ID, MSG_IRQ_STATUS, MI_COALESCED) -> D1 D3 send_replymsg(ID, MSG_IRQ_STATUS, MI_DELIVERED) -> D2 D2 send_replymsg(ID, MSG_IRQ_STATUS, MI_DELIVERED) -> D1 D3 send_replymsg(ID, MSG_IRQ_STATUS, MI_EOI) -> D2 D2 send_replymsg(ID, MSG_IRQ_STATUS, MI_EOI) -> D1 What else do we want? :-)