[ANNOUNCE] sunxi-babelfish (FEX-to-DT translator)

[ANNOUNCE] sunxi-babelfish (FEX-to-DT translator)

[Maxime Ripard]

Hi everyone,

One of the great strength of Allwinner's kernel is that it behaves
pretty much like DT does: their kernel image is mostly generic, and
all the device specific part is stored in a binary file (compiled from
a FEX file), that gets loaded together with the kernel by the
bootloader.

The kernel will then parse that file to get all the informations it
needs (which devices are enabled, which pins are muxed to what
function, etc.). Sounds familiar?

FEX and DT however are not compatible at all, so one board out there
might not have any support in mainline, while it already passes the
hardware description to the kernel.

Hence why I started to work on sunxi-babelfish. It aims at being a
very thin wrapper around the kernel image, that would generate an
uImage, with the babelfish code itself, plus the compiled DTSI we have
for the targetted SoCs, and finally the kernel image. During bootup,
babelfish will take one of these DTBs, and complete it using the
informations fetched from the compiled FEX file the board is feeding
it, before finally passing the DTB to Linux that will continue its
usual boot.

I've been working on this for a week, but we already have all the
pieces of infrastructure needed, plus some basic support for the
devices themselves. Currently, babelfish will only care about the
ATAGS and the UARTs, but I expect the list of devices it's aware of to
grow quite quickly.

The code is stored here:
  https://github.com/mripard/sunxi-babelfish/

There's a lot of room for improvements, so patches are welcome :)

Once downloaded, you can compile it using:
  CROSS_COMPILE=arm-linux- ZIMAGE=kernel/arch/arm/boot/zImage make uImage

It will obviously generate a uImage, that you will be able to boot
from u-boot.

This has been tested on the A20-olinuxino, so there might be some
glitches on the other SoCs, but I don't really expect any at the
moment, since stuff are pretty generic for now.

Another nice thing about it is that a FEX-to-DT compiler comes at no
cost afterwards. Since dtc is able to generate a DTS from
/proc/device-tree, you can easily generate a DTS for your previously
non-supported board using

  dtc -I fs -O dts -o my-board.dts /proc/device-tree/

Anyway, enough talking. Any feedback is, as usual, appreciated.

Maxime

-- 
Maxime Ripard, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20140107/46139cde/attachment.sig>

[ANNOUNCE] sunxi-babelfish (FEX-to-DT translator)

[Arnd Bergmann]

On Tuesday 07 January 2014, Maxime Ripard wrote:

> Anyway, enough talking. Any feedback is, as usual, appreciated.

I think it's a pretty cool idea. Have you considered integrating this with
the impedence-matcher project at

https://github.com/zonque/pxa-impedance-matcher ?

That one basically does the same thing for atags-based boot loaders and I
would guess that parts of the implementation can be shared as well, plus
the combination could support a wider range of devices.

	Arnd

[ANNOUNCE] sunxi-babelfish (FEX-to-DT translator)

[Maxime Ripard]

Hi Arnd,

On Wed, Jan 08, 2014 at 02:02:21PM +0100, Arnd Bergmann wrote:
> On Tuesday 07 January 2014, Maxime Ripard wrote:
> 
> > Anyway, enough talking. Any feedback is, as usual, appreciated.
> 
> I think it's a pretty cool idea. Have you considered integrating this with
> the impedence-matcher project at
> 
> https://github.com/zonque/pxa-impedance-matcher ?
> 
> That one basically does the same thing for atags-based boot loaders and I
> would guess that parts of the implementation can be shared as well, plus
> the combination could support a wider range of devices.

Yep, I've considered it at first. Actually, I even took some bits out
of it to use them.

But I thought this would be too far off the pxa-impedance-matcher to
bother trying to fit it in. IIRC, pxa-impedance-matcher is taking the
mach-id (mostly) to take an already existing DT and give it to the
kernel, while we have pretty much the opposite where we have a generic
mach-id, and build up the DTB from another source of information.

I also integrated some fexc bits to be able to parse the script.bin
file, which is highly sunxi specific.

But if Daniel's interested in merging all of this, I can totally
consider it more seriously.

Maxime

-- 
Maxime Ripard, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20140108/bd63bdf9/attachment.sig>

[ANNOUNCE] sunxi-babelfish (FEX-to-DT translator)

[Arnd Bergmann]

On Wednesday 08 January 2014 19:58:37 Maxime Ripard wrote:
> > I think it's a pretty cool idea. Have you considered integrating this with
> > the impedence-matcher project at
> > 
> > https://github.com/zonque/pxa-impedance-matcher ?
> > 
> > That one basically does the same thing for atags-based boot loaders and I
> > would guess that parts of the implementation can be shared as well, plus
> > the combination could support a wider range of devices.
> 
> Yep, I've considered it at first. Actually, I even took some bits out
> of it to use them.
> 
> But I thought this would be too far off the pxa-impedance-matcher to
> bother trying to fit it in. IIRC, pxa-impedance-matcher is taking the
> mach-id (mostly) to take an already existing DT and give it to the
> kernel, while we have pretty much the opposite where we have a generic
> mach-id, and build up the DTB from another source of information.

Yes, I understand the difference.

> I also integrated some fexc bits to be able to parse the script.bin
> file, which is highly sunxi specific.

I believe that the pxa-impedance-matcher now also contains some other
parts that go beyond just picking the right dtb file and interpreting
the atags, but I don't think any of them is as complex as the fex
parsing.

> But if Daniel's interested in merging all of this, I can totally
> consider it more seriously.

Let's see what he thinks then.

	Arnd

[linux-sunxi] [ANNOUNCE] sunxi-babelfish (FEX-to-DT translator)

[Ian Campbell]

On Tue, 2014-01-07 at 23:03 +0100, Maxime Ripard wrote:

> Hence why I started to work on sunxi-babelfish. It aims at being a
> very thin wrapper around the kernel image, that would generate an
> uImage, with the babelfish code itself, plus the compiled DTSI we have
> for the targetted SoCs, and finally the kernel image. During bootup,
> babelfish will take one of these DTBs, and complete it using the
> informations fetched from the compiled FEX file the board is feeding
> it, before finally passing the DTB to Linux that will continue its
> usual boot.

Very cool!

> [...]
> Another nice thing about it is that a FEX-to-DT compiler comes at no
> cost afterwards. Since dtc is able to generate a DTS from
> /proc/device-tree, you can easily generate a DTS for your previously
> non-supported board using
> 
>   dtc -I fs -O dts -o my-board.dts /proc/device-tree/

I wonder if it might also be useful to compile this as a normal user
space application which takes the fex and produces the dtb (or dts)?

I suppose producing something upstreamable would be different/harder
though, since it would need to not inline/merge the dtsi file but just
leave the include stmt. That's probably a different tool.

Ian.

[linux-sunxi] [ANNOUNCE] sunxi-babelfish (FEX-to-DT translator)

[Maxime Ripard]

On Wed, Jan 08, 2014 at 04:55:37PM +0000, Ian Campbell wrote:
> On Tue, 2014-01-07 at 23:03 +0100, Maxime Ripard wrote:
> > Another nice thing about it is that a FEX-to-DT compiler comes at no
> > cost afterwards. Since dtc is able to generate a DTS from
> > /proc/device-tree, you can easily generate a DTS for your previously
> > non-supported board using
> > 
> >   dtc -I fs -O dts -o my-board.dts /proc/device-tree/
> 
> I wonder if it might also be useful to compile this as a normal user
> space application which takes the fex and produces the dtb (or dts)?

At the moment, no, libfdt only operates on DTB, so we would have to
chain this somehow with dtc to at least do the DTB to DTS conversion I
guess.

It's not something that I'm pretty fond of either. Or, I don't really
see the benefit of it.

Let's say that you have such tool, I'm guessing one would want it to
mass-convert fex files to DT. But then, you'd still have to boot that
DT to be sure that everything is working, so you're pretty much losing
the mass-convertion thing, since you would have boot this with a
baremetal babelfish anyway.

> I suppose producing something upstreamable would be different/harder
> though, since it would need to not inline/merge the dtsi file but just
> leave the include stmt. That's probably a different tool.

Yeah, dtc outputs a "flat" DTS, without any include, so you still need
to edit it by hand to remove the common part, remove the duplicated
pinctrl nodes, fix the phandle, etc.

But most of that can be scripted I guess.

-- 
Maxime Ripard, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20140108/9d418e5e/attachment.sig>

[linux-sunxi] [ANNOUNCE] sunxi-babelfish (FEX-to-DT translator)

[Ian Campbell]

On Wed, 2014-01-08 at 19:29 +0100, Maxime Ripard wrote:
> On Wed, Jan 08, 2014 at 04:55:37PM +0000, Ian Campbell wrote:
> > On Tue, 2014-01-07 at 23:03 +0100, Maxime Ripard wrote:
> > > Another nice thing about it is that a FEX-to-DT compiler comes at no
> > > cost afterwards. Since dtc is able to generate a DTS from
> > > /proc/device-tree, you can easily generate a DTS for your previously
> > > non-supported board using
> > > 
> > >   dtc -I fs -O dts -o my-board.dts /proc/device-tree/
> > 
> > I wonder if it might also be useful to compile this as a normal user
> > space application which takes the fex and produces the dtb (or dts)?
> 
> At the moment, no, libfdt only operates on DTB, so we would have to
> chain this somehow with dtc to at least do the DTB to DTS conversion I
> guess.
> 
> It's not something that I'm pretty fond of either. Or, I don't really
> see the benefit of it.
> 
> Let's say that you have such tool, I'm guessing one would want it to
> mass-convert fex files to DT. But then, you'd still have to boot that
> DT to be sure that everything is working, so you're pretty much losing
> the mass-convertion thing, since you would have boot this with a
> baremetal babelfish anyway.

I wasn't really thinking of mass conversion. just an easy way for people
to produce a starting point for their platform to send upstream.

> > I suppose producing something upstreamable would be different/harder
> > though, since it would need to not inline/merge the dtsi file but just
> > leave the include stmt. That's probably a different tool.
> 
> Yeah, dtc outputs a "flat" DTS, without any include, so you still need
> to edit it by hand to remove the common part, remove the duplicated
> pinctrl nodes, fix the phandle, etc.
> 
> But most of that can be scripted I guess.

I think this could be done as a separate tool which read a FEX and
outputted the dts directly (i.e. with printf). The output would have the
#include for the right dtsi in place and the appropriate overrides
derived from the FEX. That's not really anything like your
sunxi-babelfish though.

Ian.

[linux-sunxi] [ANNOUNCE] sunxi-babelfish (FEX-to-DT translator)

[Julian Calaby]

Hi Ian,

On Thu, Jan 9, 2014 at 8:24 AM, Ian Campbell <ijc@hellion.org.uk> wrote:
> On Wed, 2014-01-08 at 19:29 +0100, Maxime Ripard wrote:
>> On Wed, Jan 08, 2014 at 04:55:37PM +0000, Ian Campbell wrote:
>> > On Tue, 2014-01-07 at 23:03 +0100, Maxime Ripard wrote:
>> > > Another nice thing about it is that a FEX-to-DT compiler comes at no
>> > > cost afterwards. Since dtc is able to generate a DTS from
>> > > /proc/device-tree, you can easily generate a DTS for your previously
>> > > non-supported board using
>> > >
>> > >   dtc -I fs -O dts -o my-board.dts /proc/device-tree/
>> >
>> > I wonder if it might also be useful to compile this as a normal user
>> > space application which takes the fex and produces the dtb (or dts)?
>>
>> At the moment, no, libfdt only operates on DTB, so we would have to
>> chain this somehow with dtc to at least do the DTB to DTS conversion I
>> guess.
>>
>> It's not something that I'm pretty fond of either. Or, I don't really
>> see the benefit of it.
>>
>> Let's say that you have such tool, I'm guessing one would want it to
>> mass-convert fex files to DT. But then, you'd still have to boot that
>> DT to be sure that everything is working, so you're pretty much losing
>> the mass-convertion thing, since you would have boot this with a
>> baremetal babelfish anyway.
>
> I wasn't really thinking of mass conversion. just an easy way for people
> to produce a starting point for their platform to send upstream.
>
>> > I suppose producing something upstreamable would be different/harder
>> > though, since it would need to not inline/merge the dtsi file but just
>> > leave the include stmt. That's probably a different tool.
>>
>> Yeah, dtc outputs a "flat" DTS, without any include, so you still need
>> to edit it by hand to remove the common part, remove the duplicated
>> pinctrl nodes, fix the phandle, etc.
>>
>> But most of that can be scripted I guess.
>
> I think this could be done as a separate tool which read a FEX and
> outputted the dts directly (i.e. with printf). The output would have the
> #include for the right dtsi in place and the appropriate overrides
> derived from the FEX. That's not really anything like your
> sunxi-babelfish though.

I was thinking about exactly this in the context of my (stalled)
online fex / board submission helper tool.

I.e. $RANDOM_USER submits their script.bin / script.fex file then it
produces (amongst other things) a dts based on it's contents using
sensible conversions based on what other knowledgeable people have
done in the past.

I was also thinking about adding a mass-converting interface to this
tool so we could regenerate the u-boot memory info and kernel dts
files from the contents of sunxi-boards.

Thanks,

-- 
Julian Calaby

Email: julian.calaby at gmail.com
Profile: http://www.google.com/profiles/julian.calaby/
.Plan: http://sites.google.com/site/juliancalaby/