Re: [Qemu-devel] Machine config files

Re: [Qemu-devel] Machine config files

[Hollis Blanchard]

On Thu, Nov 13, 2008 at 9:32 PM, Paul Brook <paul-qD8j1LwMmJjtCj0u4l0SBw@public.gmane.org> wrote:
> It's come up in other contexts recently, so I think it's worth mentioning that
> I am currently working on adding a machine config file support to qemu.
>
> I am focusing on the board setup side of things, rather than the user-level
> config side.  My goal is to come up with a system that will allow e.g. the
> entirety of realview.c and gumstix.c to be eliminated.  Currently this is
> based this round Flattened Device Trees (as used by ppc-linux).
>
> I have looked at the bits that Fabrice did a while ago. While that contains
> some good ideas (which I will probably steal!) it is approaching the problem
> from a somewhat different direction.  FTDs are a much better fit for some of
> my requirements (e.g. being able to pass the config through to the guest OS).
>
> I haven't entirely figured out how this will interact with user config (in
> particular user-pluggable devices like PCI NICs), but I think I can come up
> with something fairly sensible.

Sounds great! For what it's worth, in the PowerPC KVM patches that
I'll be posting here soon, we also use a flat device tree and pass it
to the guest. We start with a template .dtb file in pc-bios/, then
modify things like memory nodes at runtime based on qemu's command
line options.

One practical consideration is that libfdt isn't generally available
as a standalone package. I requested that Debian and Fedora create
packages for it, and they agreed, but those obviously aren't available
in older versions or other Linux distributions.

In the meantime, we could either include a libfdt snapshot in qemu
(which is how it was originally intended to be used), or require users
to build and install the package from source.

-Hollis

Re: [Qemu-devel] Machine config files

[David Gibson]

On Fri, Nov 14, 2008 at 11:54:24AM -0600, Hollis Blanchard wrote:
> On Thu, Nov 13, 2008 at 9:32 PM, Paul Brook <paul-qD8j1LwMmJjtCj0u4l0SBw@public.gmane.org> wrote:
> > It's come up in other contexts recently, so I think it's worth mentioning that
> > I am currently working on adding a machine config file support to qemu.
> >
> > I am focusing on the board setup side of things, rather than the user-level
> > config side.  My goal is to come up with a system that will allow e.g. the
> > entirety of realview.c and gumstix.c to be eliminated.  Currently this is
> > based this round Flattened Device Trees (as used by ppc-linux).
> >
> > I have looked at the bits that Fabrice did a while ago. While that contains
> > some good ideas (which I will probably steal!) it is approaching the problem
> > from a somewhat different direction.  FTDs are a much better fit for some of
> > my requirements (e.g. being able to pass the config through to the guest OS).
> >
> > I haven't entirely figured out how this will interact with user config (in
> > particular user-pluggable devices like PCI NICs), but I think I can come up
> > with something fairly sensible.
> 
> Sounds great! For what it's worth, in the PowerPC KVM patches that
> I'll be posting here soon, we also use a flat device tree and pass it
> to the guest. We start with a template .dtb file in pc-bios/, then
> modify things like memory nodes at runtime based on qemu's command
> line options.
> 
> One practical consideration is that libfdt isn't generally available
> as a standalone package. I requested that Debian and Fedora create
> packages for it, and they agreed, but those obviously aren't available
> in older versions or other Linux distributions.

Yeah, I probably should have been a bit more active to try to get dtc
and libfdt packaged for the various distros.  Oh well, they'll get it
eventually.

> In the meantime, we could either include a libfdt snapshot in qemu
> (which is how it was originally intended to be used), or require users
> to build and install the package from source.

Well.. libfdt is designed to be easy to embed in other projects so
that it can be incorporated into things like embedded bootloaders
which will usually have interesting linking requirements.  For a
userspace application like this it would be nicer to use a system
version, still, merging a snapshot is a reasonable approach for the
interim until libfdt is more widely packaged.

It would be good to make sure, though, that you merge it in such a way
that it's easy to update from upstream.  Again, libfdt is laid out so
that this should be easy.

If you want extra things in libfdt that would be convenient for you,
just ask.  The idea is that we build libfdt's features based on what
people want my way of fdt manipulation.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

Re: [Qemu-devel] Machine config files

[Hollis Blanchard]

On Friday 14 November 2008 13:29:12 Anthony Liguori wrote:
> Blue Swirl wrote:
> > On 11/14/08, Paul Brook <paul-qD8j1LwMmJjtCj0u4l0SBw@public.gmane.org> wrote:
> >   
> >> It's come up in other contexts recently, so I think it's worth mentioning 
that
> >>  I am currently working on adding a machine config file support to qemu.
> >>
> >>  I am focusing on the board setup side of things, rather than the 
user-level
> >>  config side.  My goal is to come up with a system that will allow e.g. 
the
> >>  entirety of realview.c and gumstix.c to be eliminated.  Currently this 
is
> >>  based this round Flattened Device Trees (as used by ppc-linux).
> >>
> >>  I have looked at the bits that Fabrice did a while ago. While that 
contains
> >>  some good ideas (which I will probably steal!) it is approaching the 
problem
> >>  from a somewhat different direction.  FTDs are a much better fit for 
some of
> >>  my requirements (e.g. being able to pass the config through to the guest 
OS).
> >>     
> >
> > libfdt looks like an excellent choice for both Sparc32 and Sparc64, I
> > think the whole OF tree can be represented with it.
> >
> > The config could be passed using the firmware configuration device.
> >   
> 
> I was looking through libfdt too.  I think it could also be used for 
> x86.  Hollis, I was looking through the Bamboo code, any reason you 
> don't describe the bus/device layouts in the DTS and then use that to 
> determine what bus/nic to add?

We do; PLB, OPB, and EBC are all on-chip busses found on IBM PowerPC SoCs. For 
example, you can see devices representing the UARTs and PCI controller in our 
bamboo.dts file. Qemu only modifies the device tree at runtime with 
information not known until then (memory size, clock frequency, etc).

You may mean why don't we enumerate *PCI* devices in the device tree. Since 
libfdt is used for embedded systems, this is a tradeoff that allows us to 
keep the device tree smaller -- why put lots of PCI nodes in the device tree, 
when the OS can easily discover them itself? That said, PCI devices are 
indeed found in the device tree in IBM Power servers; Linux just doesn't 
require them.

The Bamboo board in particular doesn't have any on-board PCI devices, but it 
does have several PCI slots. Therefore, a Bamboo device tree in qemu wouldn't 
contain any PCI devices anyways, but if the user says "I have an e1000", qemu 
could add a device node to represent the e1000 (though again, this is not a 
Linux requirement).

(Actually, I don't think the above addresses your specific question, but I 
went to the trouble of typing it so I figured I'd leave it in this mail for 
additional background. :)

Getting back to your original question, we could include an e1000 PCI device 
node in the DTS file, and have qemu walk the tree, find the node, and 
initialize the e1000 emulation driver. We didn't do this mostly to maintain 
compatibility with existing qemu practice, where users are required to 
specify all configuration on the command line.

-- 
Hollis Blanchard
IBM Linux Technology Center

Re: Machine config files

[Anthony Liguori]

Hollis Blanchard wrote:
> We do; PLB, OPB, and EBC are all on-chip busses found on IBM PowerPC SoCs. For 
> example, you can see devices representing the UARTs and PCI controller in our 
> bamboo.dts file. Qemu only modifies the device tree at runtime with 
> information not known until then (memory size, clock frequency, etc).
>   

Yes, but you don't build the machine based on the device tree.  What 
would be useful would be to be able to simply write a device tree 
description and that would fully represent the board such that you 
didn't need a ppc440_bamboo.c file at all.

So what I'm getting at is, what is preventing us from being able to do 
this today?

Regards,

Anthony Liguori

Re: [Qemu-devel] Machine config files

[Hollis Blanchard]

On Friday 14 November 2008 14:11:32 Anthony Liguori wrote:
> Hollis Blanchard wrote:
> > We do; PLB, OPB, and EBC are all on-chip busses found on IBM PowerPC SoCs. 
For 
> > example, you can see devices representing the UARTs and PCI controller in 
our 
> > bamboo.dts file. Qemu only modifies the device tree at runtime with 
> > information not known until then (memory size, clock frequency, etc).
> >   
> 
> Yes, but you don't build the machine based on the device tree.  What 
> would be useful would be to be able to simply write a device tree 
> description and that would fully represent the board such that you 
> didn't need a ppc440_bamboo.c file at all.
> 
> So what I'm getting at is, what is preventing us from being able to do 
> this today?

Theoretically, nothing. If qemu were a clean and modular place, it would 
probably be pretty straightforward.

Some translation code would be required for compatibility. For example, when 
the DTS has 128MB of memory, and the user invokes qemu with -m 256, now you 
need to update the device tree in order to pass it into the guest. That's 
pretty easy, but once you start talking about adding PCI devices it gets a 
little more difficult. I seem to recall this conversation was had on 
qemu-devel a little while back.

Of course, *some* code would still be needed *somewhere* to load the kernel, 
initrd, set initial register state to point to those memory locations, etc. 
In the case of KVM on 440, we also need to override the DTS with the real 
host clock frequency. (This probably isn't necessary for qemu+TCG.)

But yes, replacing all the pci_nic_init(), isa_mmio_init(), 
cpu_register_physical_memory(), etc could be automated by walking the device 
tree.

-- 
Hollis Blanchard
IBM Linux Technology Center

Re: Machine config files

[Anthony Liguori]

Hollis Blanchard wrote:
> On Friday 14 November 2008 14:11:32 Anthony Liguori wrote:
>   
> Theoretically, nothing. If qemu were a clean and modular place, it would 
> probably be pretty straightforward.
>
> Some translation code would be required for compatibility. For example, when 
> the DTS has 128MB of memory, and the user invokes qemu with -m 256, now you 
> need to update the device tree in order to pass it into the guest. That's 
> pretty easy, but once you start talking about adding PCI devices it gets a 
> little more difficult. I seem to recall this conversation was had on 
> qemu-devel a little while back.
>   

The way I would expect that it would work is that you could invoke QEMU 
with a device tree or it would by default use a device tree "template".  
The options like '-m' would modify the device tree template.  There's 
probably no need to differentiate between a user supplied device tree 
and a device tree template.

> Of course, *some* code would still be needed *somewhere* to load the kernel, 
> initrd, set initial register state to point to those memory locations, etc. 
> In the case of KVM on 440, we also need to override the DTS with the real 
> host clock frequency. (This probably isn't necessary for qemu+TCG.)
>
> But yes, replacing all the pci_nic_init(), isa_mmio_init(), 
> cpu_register_physical_memory(), etc could be automated by walking the device 
> tree.
>   

Ok.  This is starting to look interesting.  Paul, what do you think of DTS?

Regards,

Anthony Liguori

Re: Machine config files

[Paul Brook]

> > Some translation code would be required for compatibility. For example,
> > when the DTS has 128MB of memory, and the user invokes qemu with -m 256,
> > now you need to update the device tree in order to pass it into the
> > guest. That's pretty easy, but once you start talking about adding PCI
> > devices it gets a little more difficult. I seem to recall this
> > conversation was had on qemu-devel a little while back.
>...
> > Of course, *some* code would still be needed *somewhere* to load the
> > kernel, initrd, set initial register state to point to those memory
> > locations, etc. In the case of KVM on 440, we also need to override the
> > DTS with the real host clock frequency. (This probably isn't necessary
> > for qemu+TCG.)
> >
> > But yes, replacing all the pci_nic_init(), isa_mmio_init(),
> > cpu_register_physical_memory(), etc could be automated by walking the
> > device tree.
>
> Ok.  This is starting to look interesting.  Paul, what do you think of DTS?

Things like kernel loading are easy, you just add a special device (which 
linux will ignore) specifying the relevant parameters. IMO it should be 
possible to use the same device tree for both linux and qemu, but it's ok to 
require some additional information on top of what linux currently requires.

I agree with Hollis that the tricky bits are when you want to augment a board 
with use supplied options (e.g. RAM size or additional PCI devices). That 
either requires qemu to modify the supplied device tree, merging multiple 
trees, or some other magic. I'm ignoring this problem for now. Worst case we 
remove the existing options, provide a reasonable set of "standard" configs 
and anyone who wants exotic setups can modify the machine description.

I'm mainly interested in emulation, where any host information leaking through 
to the emulated machine is a bug :-)

Paul

Re: [Qemu-devel] Machine config files

[David Gibson]

On Sat, Nov 15, 2008 at 12:13:03AM +0000, Paul Brook wrote:
> > > Some translation code would be required for compatibility. For example,
> > > when the DTS has 128MB of memory, and the user invokes qemu with -m 256,
> > > now you need to update the device tree in order to pass it into the
> > > guest. That's pretty easy, but once you start talking about adding PCI
> > > devices it gets a little more difficult. I seem to recall this
> > > conversation was had on qemu-devel a little while back.
> >...
> > > Of course, *some* code would still be needed *somewhere* to load the
> > > kernel, initrd, set initial register state to point to those memory
> > > locations, etc. In the case of KVM on 440, we also need to override the
> > > DTS with the real host clock frequency. (This probably isn't necessary
> > > for qemu+TCG.)
> > >
> > > But yes, replacing all the pci_nic_init(), isa_mmio_init(),
> > > cpu_register_physical_memory(), etc could be automated by walking the
> > > device tree.
> >
> > Ok.  This is starting to look interesting.  Paul, what do you think of DTS?
> 
> Things like kernel loading are easy, you just add a special device (which 
> linux will ignore) specifying the relevant parameters. IMO it should be 
> possible to use the same device tree for both linux and qemu, but it's ok to 
> require some additional information on top of what linux currently requires.

Depending on exactly what you want to pass through, it may be
appropriate to use properties in /chosen for this.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

Re: [Qemu-devel] Machine config files

[David Gibson]

On Fri, Nov 14, 2008 at 01:51:40PM -0600, Hollis Blanchard wrote:
> On Friday 14 November 2008 13:29:12 Anthony Liguori wrote:
> > Blue Swirl wrote:
> > > On 11/14/08, Paul Brook <paul-qD8j1LwMmJjtCj0u4l0SBw@public.gmane.org> wrote:
> > >   
> > >> It's come up in other contexts recently, so I think it's worth mentioning 
> that
> > >>  I am currently working on adding a machine config file support to qemu.
> > >>
> > >>  I am focusing on the board setup side of things, rather than the 
> user-level
> > >>  config side.  My goal is to come up with a system that will allow e.g. 
> the
> > >>  entirety of realview.c and gumstix.c to be eliminated.  Currently this 
> is
> > >>  based this round Flattened Device Trees (as used by ppc-linux).
> > >>
> > >>  I have looked at the bits that Fabrice did a while ago. While that 
> contains
> > >>  some good ideas (which I will probably steal!) it is approaching the 
> problem
> > >>  from a somewhat different direction.  FTDs are a much better fit for 
> some of
> > >>  my requirements (e.g. being able to pass the config through to the guest 
> OS).
> > >>     
> > >
> > > libfdt looks like an excellent choice for both Sparc32 and Sparc64, I
> > > think the whole OF tree can be represented with it.
> > >
> > > The config could be passed using the firmware configuration device.
> > >   
> > 
> > I was looking through libfdt too.  I think it could also be used for 
> > x86.  Hollis, I was looking through the Bamboo code, any reason you 
> > don't describe the bus/device layouts in the DTS and then use that to 
> > determine what bus/nic to add?
> 
> We do; PLB, OPB, and EBC are all on-chip busses found on IBM PowerPC SoCs. For 
> example, you can see devices representing the UARTs and PCI controller in our 
> bamboo.dts file. Qemu only modifies the device tree at runtime with 
> information not known until then (memory size, clock frequency, etc).
> 
> You may mean why don't we enumerate *PCI* devices in the device tree. Since 
> libfdt is used for embedded systems, this is a tradeoff that allows us to 
> keep the device tree smaller -- why put lots of PCI nodes in the device tree, 
> when the OS can easily discover them itself? That said, PCI devices are 
> indeed found in the device tree in IBM Power servers; Linux just doesn't 
> require them.

Yes, in general the rule is that devices on busses that can be runtime
probed - PCI, USB and the like - don't typically need to appear in a
flattened device tree (though they would traditionally appear in a
full Open Firmware device tree).  Only the things the OS has to "just
know" are there need appear - e.g. on-chip devices, hardwarired
on-board devices, unprobeable legacy ISA devices, usually the PCI host
bridge, and so forth.

Nodes for probeable devices may optionally appear, and in certain
cases they must appear to supply bits of information that can't be
probed.  For example, on-board PCI devices often have interrupts
routed directly to the system PIC, rather than going across the PCI
bus.  In this case a device node would need to appear with the
interrupt routing information.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson