Device tree with early buffer allocations and aliased memory

Device tree with early buffer allocations and aliased memory

[David VomLehn]

This is a very long question and appreciate your forebearance--I'm using
Linux and I have a pretty complicated memory structure for which I'm
trying to define a device tree and I need some help making sense out
it all. The basic scenario:
1	I'm using a MIPS processor, which has low memory from 0x0 to
	0x1fffffff.
2.	I have two banks of memory with the following DMA addresses:
		Bank 0	0x20000000-0x3fffffff
		Bank 1	0x60000000-0x7fffffff
	With a page map, the processor can also get to these so they
	are also physical addresses.
3.	All device registers are at physical addresses < 0x10000000.
4.	Since the processor needs memory to run, the first 0x0fc00000
	bytes also appear at 0x10000000. Our hardware guys call this
	a memory "alias". These addresses are physical, but not DMA
	addresses. (The hole is so we can alias the reset vector)
5.	We need huge buffers (up to 80 MiB) that are way too large to
	allocate with the usual functions, so we allocate them very
	early through the bootmem allocator. Some of these must be
	at a fixed address and some may be allocated in one or the
	other bank of memory. Dynamic address assignment is generally
	preferred to avoid overlapping static assignments.

Memory seems straight-forward-just use the process physical addresses
wherever possible:

	memories {
		memory@10000000 {
			device_type = "memory";
			reg = <10000000 0x0fc00000>;
		};
		memory@20000000 {
			device_type = "memory";
			reg = <2fc00000 10400000>;
		};
		memory@60000000 {
			default_type = "memory";
			reg = <60000000 20000000>;
		};
	};

For a device with buffers allocated at a static address in memory
without an alias I use a new property, cisco,memory-buffers, which
specifies triples for buffers where:
	item 1	Lowest allowable address of the first byte of the buffer
	item 2	Highest allowable address of the last byte of the buffer
	item 3	Number of bytes in the buffer.
If item 1 = item 2 - item 3 this buffer must be allocated at the address
given in item 1, i.e. it has a static address. Otherwise it may be
dynamically allocated within the given range:

	static-dev-noalias@08000000 {
		compatible = "cisco,static-dev-noalias";
		reg = <0x08000000>;
		cisco,memory-buffers=<0x60000000 0x60200000 0x00200000>;
	};

I think this is reasonable, though I'd prefer a standard way to specify
memory buffers allocation parameters. Things get complicated for a device
with buffers dynamically allocated in the part of memory with an alias.
The device registers are still in non-aliased memory, so this isn't a
completely independent bus:

	dyn-dev-alias@08001000 {
		compatible = "cisco,dyn-dev-alias";
		reg = <0x08001000>;
		cisco,memory-buffers = <0x10000000 0x1fb00000 0x00100000>;
	};

The problem is that I haven't supplied enough information to turn the
memory buffer physical addresses into DMA addresses, but if I wrap this
in something like a simple-bus, the address for the reg property will
be wrong. I could simply adjust it, but then it gets a lot harder to
read and maintain.

So, questions:
1.	Is there a better way to specify parameters for very early
	buffer allocations?
2.	Is there a better way to handle devices whose device registers
	live nicely in physical address space but whose buffers live
	in different physical and DMA address spaces?

I'm open to any and all suggestions.
-- 
David VL

Re: Device tree with early buffer allocations and aliased memory

[Grant Likely]

On Wed, Oct 27, 2010 at 2:49 PM, David VomLehn <dvomlehn-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> wrote:
> This is a very long question and appreciate your forebearance--I'm using
> Linux and I have a pretty complicated memory structure for which I'm
> trying to define a device tree and I need some help making sense out
> it all. The basic scenario:
> 1       I'm using a MIPS processor, which has low memory from 0x0 to
>        0x1fffffff.
> 2.      I have two banks of memory with the following DMA addresses:
>                Bank 0  0x20000000-0x3fffffff
>                Bank 1  0x60000000-0x7fffffff
>        With a page map, the processor can also get to these so they
>        are also physical addresses.
> 3.      All device registers are at physical addresses < 0x10000000.

Inside of the low memory region?

> 4.      Since the processor needs memory to run, the first 0x0fc00000
>        bytes also appear at 0x10000000. Our hardware guys call this
>        a memory "alias". These addresses are physical, but not DMA
>        addresses. (The hole is so we can alias the reset vector)

Not sure I'm clear.  How does the hole work?  It might be useful to
draw a map of the various ranges rather than trying to describe it in
prose.  So is phys address range 0x00000000-0x0fc00000 aliased to
0x10000000-0x1fc00000?

> 5.      We need huge buffers (up to 80 MiB) that are way too large to
>        allocate with the usual functions, so we allocate them very
>        early through the bootmem allocator. Some of these must be
>        at a fixed address and some may be allocated in one or the
>        other bank of memory. Dynamic address assignment is generally
>        preferred to avoid overlapping static assignments.
>
> Memory seems straight-forward-just use the process physical addresses
> wherever possible:
>
>        memories {
>                memory@10000000 {
>                        device_type = "memory";
>                        reg = <10000000 0x0fc00000>;
>                };
>                memory@20000000 {
>                        device_type = "memory";
>                        reg = <2fc00000 10400000>;
>                };
>                memory@60000000 {
>                        default_type = "memory";
>                        reg = <60000000 20000000>;
>                };
>        };

You don't need three separate nodes here.  Just do it thusly:

memory@10000000 {
        device_type = "memory";
        reg = <0x10000000 0x0fc00000
               0x2fc00000 0x10400000
               0x60000000 0x20000000>;
}

Since (I assume) both #address-cells and #size-cells are 1, the OS can
interpret three memory ranges from a single node.

> For a device with buffers allocated at a static address in memory
> without an alias I use a new property, cisco,memory-buffers, which
> specifies triples for buffers where:
>        item 1  Lowest allowable address of the first byte of the buffer
>        item 2  Highest allowable address of the last byte of the buffer
>        item 3  Number of bytes in the buffer.

This is a little odd.  Is the dma-ranges property not suitable to
describe which address ranges are actually available for DMA'ing into
for each bus?

> If item 1 = item 2 - item 3 this buffer must be allocated at the address
> given in item 1, i.e. it has a static address. Otherwise it may be
> dynamically allocated within the given range:
>
>        static-dev-noalias@08000000 {
>                compatible = "cisco,static-dev-noalias";
>                reg = <0x08000000>;

Unless the paren't #size-cells = 0, this reg property is broken
because it doesn't have the size field.

>                cisco,memory-buffers=<0x60000000 0x60200000 0x00200000>;
>        };
>
> I think this is reasonable, though I'd prefer a standard way to specify
> memory buffers allocation parameters. Things get complicated for a device
> with buffers dynamically allocated in the part of memory with an alias.
> The device registers are still in non-aliased memory, so this isn't a
> completely independent bus:
>
>        dyn-dev-alias@08001000 {
>                compatible = "cisco,dyn-dev-alias";
>                reg = <0x08001000>;
>                cisco,memory-buffers = <0x10000000 0x1fb00000 0x00100000>;
>        };
>
> The problem is that I haven't supplied enough information to turn the
> memory buffer physical addresses into DMA addresses, but if I wrap this
> in something like a simple-bus, the address for the reg property will
> be wrong.

How so?  Address translation should work if the properties are set up correctly.

> I could simply adjust it, but then it gets a lot harder to
> read and maintain.
>
> So, questions:
> 1.      Is there a better way to specify parameters for very early
>        buffer allocations?

Well, dma-ranges (see ePAPR) can set up the DMA constraints, and then
you can have a simple property just for specifying the required DMA
region size.

> 2.      Is there a better way to handle devices whose device registers
>        live nicely in physical address space but whose buffers live
>        in different physical and DMA address spaces?

dma-ranges is probably what you want.

g.

Re: Device tree with early buffer allocations and aliased memory

[David VomLehn]

This got eaten as junk mail, so pardon the slow response. Also, Stuart
Y. has been giving me some feedback, so I think I've made some
progress.

On Sat, Oct 30, 2010 at 10:57:34PM -0500, Grant Likely wrote:
> On Wed, Oct 27, 2010 at 2:49 PM, David VomLehn <dvomlehn-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> wrote:
...
> > 3.      All device registers are at physical addresses < 0x10000000.
> 
> Inside of the low memory region?

Yes. Those wacky hardware guys...

> > 4.      Since the processor needs memory to run, the first 0x0fc00000
> >        bytes also appear at 0x10000000. Our hardware guys call this
> >        a memory "alias". These addresses are physical, but not DMA
> >        addresses. (The hole is so we can alias the reset vector)
> 
> Not sure I'm clear.  How does the hole work?  It might be useful to
> draw a map of the various ranges rather than trying to describe it in
> prose.  So is phys address range 0x00000000-0x0fc00000 aliased to
> 0x10000000-0x1fc00000?

I can see I wasn't quite accurate here. The only things in the first
0x10000000 bytes of memory are device registers. Real memory starts at
0x10000000 and runs for 0x0fc00000 bytes. The same memory also appears
at 0x20000000-0x2fbfffff. Memory above 0x2fc00000 only appears at one
address.

> You don't need three separate nodes here.  Just do it thusly:
> 
> memory@10000000 {
>         device_type = "memory";
>         reg = <0x10000000 0x0fc00000
>                0x2fc00000 0x10400000
>                0x60000000 0x20000000>;
> }
> 
> Since (I assume) both #address-cells and #size-cells are 1, the OS can
> interpret three memory ranges from a single node.

This is good to know, though I might want to label each node. See below.

...
> dma-ranges is probably what you want.

I think this is a big part of the solution. My current thinking is to
have something like:

	platform-bus {
		compatible = "simple-bus";
		#address-cells = <1>;
		#size-cells = <1>;
		dma-ranges = <0x10000000 0x1000000 0x0fc00000
			0x20000000 0x10000000 0x0fc00000
			0x2fc00000 0x2fc00000 0x10400000
			0x60000000 0x60000000 0x20000000>;
		...

I'm not sure whether I need to specify all those 1:1 mappings or just
the range starting at 0x20000000 that doesn't map identically to the
physical address range. Assuming this is correct, my DMA mapping issue
is solved.

This still leaves the question of buffers. Buffers with statically and
dynamically assigned addresses need some sort of identifier so that
they can be referenced by device drivers. There may be more than one
buffer per device and a single device may use buffers with statically
assigned addresses and buffers with dynamically assigned addresses.
Buffers with statically assigned buffers addresses could be handled
with something like:

	device-s {
		compatible = "cisco,device-s";
		cisco,static-buffers = 
			"device-s-b1",<0x24000000 0x00100000>,
			"device-s-b2",<0x60000000 0x00200000>;
	};

The first buffer, device-s-b1, is in the area where physical and DMA
address differ. I'm using the DMA address to specify it, but the
the physical address may make more sense since the device tree is
more oriented around the processor's view.

Buffers with dynamically assigned addresses must be assigned within
a given range of memory. Using labeled memory nodes could help:

	device-d {
		compatible = "cisco,device-d";
		cisco,dynamic-buffers =
			"device-d-b1",<&node0-lowmem 0x00210000>,
			"device-d-b2",<&node0-highmem 0x00220000>,
			"device-d-b3",<&node1-highmem 0x00230000>;
	};

I could have separate memory nodes and label each:

	bank0_lowmem: memory@10000000 {
		device_type = "memory";
		reg = <0x10000000 0x0fc00000>;
	};
	bank0_highmem: memory@2c00000 {
		device_type = "memory";
		reg = <0x2fc00000 0x10400000>;
	};
	bank1_highmem: memory@60000000 {
		device_type = "memory";
		reg = <0x60000000 0x20000000>;
	};

or label the 2-tuple for each node:

	memory@10000000 {
		device_type = "memory";
		reg = <node0-lowmem: 0x10000000 0x0fc00000
		       node0-highmem: 0x2fc00000 0x10400000
		       node0-highlowmem: 0x60000000 0x20000000>;
	}

The only advantage to the former is that, should it be necessary to
decouple allocation restrictions from memory nodes, a second type
of node could be introduced to do that.

> g.
-- 
David VL

Re: Device tree with early buffer allocations and aliased memory

[Grant Likely]

On Wed, Nov 03, 2010 at 01:50:37PM -0700, David VomLehn wrote:
> This got eaten as junk mail, so pardon the slow response. Also, Stuart
> Y. has been giving me some feedback, so I think I've made some
> progress.
> 
> On Sat, Oct 30, 2010 at 10:57:34PM -0500, Grant Likely wrote:
> > On Wed, Oct 27, 2010 at 2:49 PM, David VomLehn <dvomlehn-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> wrote:
> ...
> > > 3.      All device registers are at physical addresses < 0x10000000.
> > 
> > Inside of the low memory region?
> 
> Yes. Those wacky hardware guys...
> 
> > > 4.      Since the processor needs memory to run, the first 0x0fc00000
> > >        bytes also appear at 0x10000000. Our hardware guys call this
> > >        a memory "alias". These addresses are physical, but not DMA
> > >        addresses. (The hole is so we can alias the reset vector)
> > 
> > Not sure I'm clear.  How does the hole work?  It might be useful to
> > draw a map of the various ranges rather than trying to describe it in
> > prose.  So is phys address range 0x00000000-0x0fc00000 aliased to
> > 0x10000000-0x1fc00000?
> 
> I can see I wasn't quite accurate here. The only things in the first
> 0x10000000 bytes of memory are device registers. Real memory starts at
> 0x10000000 and runs for 0x0fc00000 bytes. The same memory also appears
> at 0x20000000-0x2fbfffff. Memory above 0x2fc00000 only appears at one
> address.

Ah, okay.  So start of ram is at 0x10000000 instead of 0.  Not all
that uncommon.

> 
> > You don't need three separate nodes here.  Just do it thusly:
> > 
> > memory@10000000 {
> >         device_type = "memory";
> >         reg = <0x10000000 0x0fc00000
> >                0x2fc00000 0x10400000
> >                0x60000000 0x20000000>;
> > }
> > 
> > Since (I assume) both #address-cells and #size-cells are 1, the OS can
> > interpret three memory ranges from a single node.
> 
> This is good to know, though I might want to label each node. See below.

Further note:  The memory node must only contain memory ranges that
will actually be used by the operating system as system RAM.  If it is
dedicated to DMA buffers or the like, then you don't want to identify
it as a memory node.  You can create a new binding to describe your
DMA buffer ranges, but make sure it doesn't specify device_type="memory".

> 
> ...
> > dma-ranges is probably what you want.
> 
> I think this is a big part of the solution. My current thinking is to
> have something like:
> 
> 	platform-bus {

It *appears* that you're conflating Linux-kernel internal details (the
name "platform-bus") with the actual layout of the hardware.  I'd be
surprised if your peripheral bus is named "platform-bus" in the
hardware documentation.  Name things for what they are, with
preference for names in the generic names list from ePAPR.

> 		compatible = "simple-bus";
> 		#address-cells = <1>;
> 		#size-cells = <1>;
> 		dma-ranges = <0x10000000 0x1000000 0x0fc00000
> 			0x20000000 0x10000000 0x0fc00000
> 			0x2fc00000 0x2fc00000 0x10400000
> 			0x60000000 0x60000000 0x20000000>;

I suspect the length field in the 3rd line is wrong.  Should be
0x400000?  I also suspect the 2nd and 3rd lines could be merged to
give:
			0x20000000 0x20000000 0x10000000

> 		...
> 
> I'm not sure whether I need to specify all those 1:1 mappings or just
> the range starting at 0x20000000 that doesn't map identically to the
> physical address range. Assuming this is correct, my DMA mapping issue
> is solved.

Unless you *actually* want to perform DMAs to both the real and
aliased ranges, then you probably don't want specify both.

> 
> This still leaves the question of buffers. Buffers with statically and
> dynamically assigned addresses need some sort of identifier so that
> they can be referenced by device drivers. There may be more than one
> buffer per device and a single device may use buffers with statically
> assigned addresses and buffers with dynamically assigned addresses.
> Buffers with statically assigned buffers addresses could be handled
> with something like:
> 
> 	device-s {
> 		compatible = "cisco,device-s";
> 		cisco,static-buffers = 
> 			"device-s-b1",<0x24000000 0x00100000>,
> 			"device-s-b2",<0x60000000 0x00200000>;

Or, simply:
	cisco,static-buffer-b1 = <0x24000000 0x00100000>;
	cisco,static-buffer-b2 = <0x60000000 0x00200000>;

Try to avoid mixing string and cell values in the same property where
appropriate.  Sometimes doing so is the best binding, but those cases
are rare.

> 	};
> 
> The first buffer, device-s-b1, is in the area where physical and DMA
> address differ. I'm using the DMA address to specify it, but the
> the physical address may make more sense since the device tree is
> more oriented around the processor's view.

Since this sounds very device specific, doing custom properties are
just fine *providing* you document it in the "cisco,device-s" binding.

> 
> Buffers with dynamically assigned addresses must be assigned within
> a given range of memory. Using labeled memory nodes could help:
> 
> 	device-d {
> 		compatible = "cisco,device-d";
> 		cisco,dynamic-buffers =
> 			"device-d-b1",<&node0-lowmem 0x00210000>,
> 			"device-d-b2",<&node0-highmem 0x00220000>,
> 			"device-d-b3",<&node1-highmem 0x00230000>;

What is the meaning of the second cell in these properties
(0x00210000, 0x220000, and 0x230000)?  This doesn't seem to be an
optimal binding, although that is probably mainly due to the joint
string+cell property values.  Having a phandle to another node is
/okay/, but it might just be simpler and better to explicitly specify
the address ranges that the device is able to DMA to/from.

> 	};
> 
> I could have separate memory nodes and label each:
> 
> 	bank0_lowmem: memory@10000000 {
> 		device_type = "memory";
> 		reg = <0x10000000 0x0fc00000>;
> 	};
> 	bank0_highmem: memory@2c00000 {
> 		device_type = "memory";
> 		reg = <0x2fc00000 0x10400000>;
> 	};
> 	bank1_highmem: memory@60000000 {
> 		device_type = "memory";
> 		reg = <0x60000000 0x20000000>;
> 	};

This would be okay.

> 
> or label the 2-tuple for each node:
> 
> 	memory@10000000 {
> 		device_type = "memory";
> 		reg = <node0-lowmem: 0x10000000 0x0fc00000
> 		       node0-highmem: 0x2fc00000 0x10400000
> 		       node0-highlowmem: 0x60000000 0x20000000>;
> 	}

Don't do this.  You shouldn't ever need to use the labels inside a
property.  They are only used for some corner cases where a pointer is
needed to the data inside an fdt blob.

g.

Re: Device tree with early buffer allocations and aliased memory

[David VomLehn]

On Tue, Nov 09, 2010 at 10:25:37PM -0600, Grant Likely wrote:
> On Wed, Nov 03, 2010 at 01:50:37PM -0700, David VomLehn wrote:
...
> Further note:  The memory node must only contain memory ranges that
> will actually be used by the operating system as system RAM.  If it is
> dedicated to DMA buffers or the like, then you don't want to identify
> it as a memory node.  You can create a new binding to describe your
> DMA buffer ranges, but make sure it doesn't specify device_type="memory".

This one confuses me a bit. Devices using DMA share memory with device
drivers; does this mean that memory is used by the kernel as system RAM?
Also, the buffers whose addresses are dynamically assigned have exactly
the same usage pattern as those whose addresses are statically assigned,
but there is no way to know where in the memory nodes they should be.
It is thus not possible to exclude them from memory nodes. This seems
inconsistent.

> > 	platform-bus {
> 
> It *appears* that you're conflating Linux-kernel internal details (the
> name "platform-bus") with the actual layout of the hardware.  I'd be
> surprised if your peripheral bus is named "platform-bus" in the
> hardware documentation.  Name things for what they are, with
> preference for names in the generic names list from ePAPR.

It looks like naming this node "soc" would be in accordance with
convention. Is this better than "stbus", "simple-bus", or "simple-bus@0"?

> > 		compatible = "simple-bus";
> > 		#address-cells = <1>;
> > 		#size-cells = <1>;
> > 		dma-ranges = <0x10000000 0x1000000 0x0fc00000
> > 			0x20000000 0x10000000 0x0fc00000
> > 			0x2fc00000 0x2fc00000 0x10400000
> > 			0x60000000 0x60000000 0x20000000>;
> 
> I suspect the length field in the 3rd line is wrong.  Should be
> 0x400000?  I also suspect the 2nd and 3rd lines could be merged to
> give:
> 			0x20000000 0x20000000 0x10000000

The third line is for memory extending from 0x2fc00000-0x3fffffff, inclusive,
where the DMA and physical addresses are the same for the entire range. As
I understand it, the third parameter is the length, i.e. the length of the
area of memory shared between the device doing DMA and the device driver.
I don't see anything wrong with this, but would like to know if there is.

Though the third line has an identical mapping between physical and DMA
addresses, the second line has an offset of 0x10000000 between the two
addresses. Unless I misunderstand, I can't merge these.

> > I'm not sure whether I need to specify all those 1:1 mappings or just
> > the range starting at 0x20000000 that doesn't map identically to the
> > physical address range. Assuming this is correct, my DMA mapping issue
> > is solved.
> 
> Unless you *actually* want to perform DMAs to both the real and
> aliased ranges, then you probably don't want specify both.

All of the physical addresses can be used with DMA, after translation to their
corresponding DMA addresses, so it sounds like I need to specify all
physical addresses.

> Since this sounds very device specific, doing custom properties are
> just fine *providing* you document it in the "cisco,device-s" binding.

Once I get this all nailed down, is there a standard place to publish
custom bindings?

> > Buffers with dynamically assigned addresses must be assigned within
> > a given range of memory. Using labeled memory nodes could help:
> > 
> > 	device-d {
> > 		compatible = "cisco,device-d";
> > 		cisco,dynamic-buffers =
> > 			"device-d-b1",<&node0-lowmem 0x00210000>,
> > 			"device-d-b2",<&node0-highmem 0x00220000>,
> > 			"device-d-b3",<&node1-highmem 0x00230000>;
> 
> What is the meaning of the second cell in these properties
> (0x00210000, 0x220000, and 0x230000)?  This doesn't seem to be an
> optimal binding, although that is probably mainly due to the joint
> string+cell property values.  Having a phandle to another node is
> /okay/, but it might just be simpler and better to explicitly specify
> the address ranges that the device is able to DMA to/from.

Those second cells are the length of the DMA buffers. The phandles are
being used to indicate the range within which the beginning and ending
addresses of the buffers must lie.  I used phandles phandles because
all of the ranges will be within one of the three memory nodes and
this seemed like a nice way to simplify things and avoid potentially
subtle errors in specifying the range. But specifying ranges explicitly
for each buffer works fine.

> g.

-- 
David VL

Re: Device tree with early buffer allocations and aliased memory

[Grant Likely]

On Mon, Nov 15, 2010 at 03:10:08PM -0800, David VomLehn wrote:
> On Tue, Nov 09, 2010 at 10:25:37PM -0600, Grant Likely wrote:
> > On Wed, Nov 03, 2010 at 01:50:37PM -0700, David VomLehn wrote:
> ...
> > Further note:  The memory node must only contain memory ranges that
> > will actually be used by the operating system as system RAM.  If it is
> > dedicated to DMA buffers or the like, then you don't want to identify
> > it as a memory node.  You can create a new binding to describe your
> > DMA buffer ranges, but make sure it doesn't specify device_type="memory".
> 
> This one confuses me a bit. Devices using DMA share memory with device
> drivers; does this mean that memory is used by the kernel as system RAM?
> Also, the buffers whose addresses are dynamically assigned have exactly
> the same usage pattern as those whose addresses are statically assigned,
> but there is no way to know where in the memory nodes they should be.
> It is thus not possible to exclude them from memory nodes. This seems
> inconsistent.

I'm not sure I completely follow you.  Any memory described in a
device_type="memory" node will by default be used as system RAM by
Linux.  If some of that region needs to be used for DMA buffers, then
you'll need to make sure that when a driver allocates a buffer, that
the buffer is made valid for DMA operations.  How you do this is
architecture specific.

> > > 	platform-bus {
> > 
> > It *appears* that you're conflating Linux-kernel internal details (the
> > name "platform-bus") with the actual layout of the hardware.  I'd be
> > surprised if your peripheral bus is named "platform-bus" in the
> > hardware documentation.  Name things for what they are, with
> > preference for names in the generic names list from ePAPR.
> 
> It looks like naming this node "soc" would be in accordance with
> convention. Is this better than "stbus", "simple-bus", or "simple-bus@0"?

what does 'stbus' mean?  If it is the hardware name for the bus, then
it is probably a good choice.  'soc' is often used, but I'm not a big
fan of that convention because it doesn't reflect the actual internal
architecture of the chip.

> > > 		compatible = "simple-bus";
> > > 		#address-cells = <1>;
> > > 		#size-cells = <1>;
> > > 		dma-ranges = <0x10000000 0x1000000 0x0fc00000
> > > 			0x20000000 0x10000000 0x0fc00000
> > > 			0x2fc00000 0x2fc00000 0x10400000
> > > 			0x60000000 0x60000000 0x20000000>;
> > 
> > I suspect the length field in the 3rd line is wrong.  Should be
> > 0x400000?  I also suspect the 2nd and 3rd lines could be merged to
> > give:
> > 			0x20000000 0x20000000 0x10000000
> 
> The third line is for memory extending from 0x2fc00000-0x3fffffff, inclusive,
> where the DMA and physical addresses are the same for the entire range. As
> I understand it, the third parameter is the length, i.e. the length of the
> area of memory shared between the device doing DMA and the device driver.
> I don't see anything wrong with this, but would like to know if there is.

Sorry, I believe you're right.  I misread your example.

> Though the third line has an identical mapping between physical and DMA
> addresses, the second line has an offset of 0x10000000 between the two
> addresses. Unless I misunderstand, I can't merge these.

<digress into details of hardware design>
Is a device legally able to DMA into the address range
0x20000000..0x2fbfffff instead of 0x10000000..0x1fbfffff?  What is the
reason for dma'ing to base 0x10000000 for the first 0xfc00000, and to
base 0x20000000 for the second if the two ranges are simple aliases of
each other?

>From your earlier description it sounded like the 0x10000000 alias is
only to provide Linux with RAM to run out of, but the real ram base
address is 0x20000000.  As such, it still sounds like the dma-ranges
should cover the entire physical memory region from
0x20000000..0x3fffffff, and not reference the 0x10000000 alias at all.
</digress>

> > > I'm not sure whether I need to specify all those 1:1 mappings or just
> > > the range starting at 0x20000000 that doesn't map identically to the
> > > physical address range. Assuming this is correct, my DMA mapping issue
> > > is solved.
> > 
> > Unless you *actually* want to perform DMAs to both the real and
> > aliased ranges, then you probably don't want specify both.
> 
> All of the physical addresses can be used with DMA, after translation to their
> corresponding DMA addresses, so it sounds like I need to specify all
> physical addresses.
> 
> > Since this sounds very device specific, doing custom properties are
> > just fine *providing* you document it in the "cisco,device-s" binding.
> 
> Once I get this all nailed down, is there a standard place to publish
> custom bindings?

devicetree.org

> 
> > > Buffers with dynamically assigned addresses must be assigned within
> > > a given range of memory. Using labeled memory nodes could help:
> > > 
> > > 	device-d {
> > > 		compatible = "cisco,device-d";
> > > 		cisco,dynamic-buffers =
> > > 			"device-d-b1",<&node0-lowmem 0x00210000>,
> > > 			"device-d-b2",<&node0-highmem 0x00220000>,
> > > 			"device-d-b3",<&node1-highmem 0x00230000>;
> > 
> > What is the meaning of the second cell in these properties
> > (0x00210000, 0x220000, and 0x230000)?  This doesn't seem to be an
> > optimal binding, although that is probably mainly due to the joint
> > string+cell property values.  Having a phandle to another node is
> > /okay/, but it might just be simpler and better to explicitly specify
> > the address ranges that the device is able to DMA to/from.
> 
> Those second cells are the length of the DMA buffers. The phandles are
> being used to indicate the range within which the beginning and ending
> addresses of the buffers must lie.  I used phandles phandles because
> all of the ranges will be within one of the three memory nodes and
> this seemed like a nice way to simplify things and avoid potentially
> subtle errors in specifying the range. But specifying ranges explicitly
> for each buffer works fine.

I'd just go with the explicit ranges simply because it is simpler.  If
it was a more common use-case, then I might have a different opinion,
but since it is an oddball case I'd stick with familiar patterns for
specifying dma constraints.

g.

Re: Device tree with early buffer allocations and aliased memory

[David VomLehn]

On 11/15/2010 10:57 PM, Grant Likely wrote:

...
> I'm not sure I completely follow you.  Any memory described in a
> device_type="memory" node will by default be used as system RAM by
> Linux.  If some of that region needs to be used for DMA buffers, then
> you'll need to make sure that when a driver allocates a buffer, that
> the buffer is made valid for DMA operations.  How you do this is
> architecture specific.
Okay, I think I've got this down
...
> what does 'stbus' mean?  If it is the hardware name for the bus, then
> it is probably a good choice.  'soc' is often used, but I'm not a big
> fan of that convention because it doesn't reflect the actual internal
> architecture of the chip.
Yes, this is what STMicroelectronics call their bus, specifically 
"STBus". Which looks like what the device tree calls a simple- bus.
...
>>>> 		compatible = "simple-bus";
>>>> 		#address-cells =<1>;
>>>> 		#size-cells =<1>;
>>>> 		dma-ranges =<0x10000000 0x1000000 0x0fc00000
>>>> 			0x20000000 0x10000000 0x0fc00000
>>>> 			0x2fc00000 0x2fc00000 0x10400000
>>>> 			0x60000000 0x60000000 0x20000000>;
...
> <digress into details of hardware design>
> Is a device legally able to DMA into the address range
> 0x20000000..0x2fbfffff instead of 0x10000000..0x1fbfffff?  What is the
> reason for dma'ing to base 0x10000000 for the first 0xfc00000, and to
> base 0x20000000 for the second if the two ranges are simple aliases of
> each other?
>
>  From your earlier description it sounded like the 0x10000000 alias is
> only to provide Linux with RAM to run out of, but the real ram base
> address is 0x20000000.  As such, it still sounds like the dma-ranges
> should cover the entire physical memory region from
> 0x20000000..0x3fffffff, and not reference the 0x10000000 alias at all.
> </digress>
Think I've got it--the 0x10000000-0x1fc00000 is not visible to DMA 
devices, so it doesn't belong in the dma-ranges, only those address that 
can actually be given to DMA devices. So, I'll drop the first triplet 
and keep the others.
...
> I'd just go with the explicit ranges simply because it is simpler.  If
> it was a more common use-case, then I might have a different opinion,
> but since it is an oddball case I'd stick with familiar patterns for
> specifying dma constraints.
Sounds good.
> g.
Thanks!

Re: Device tree with early buffer allocations and aliased memory

[Grant Likely]

On Tue, Nov 16, 2010 at 4:21 PM, David VomLehn <dvomlehn-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> wrote:
> On 11/15/2010 10:57 PM, Grant Likely wrote:
>
> ...
>>
>> I'm not sure I completely follow you.  Any memory described in a
>> device_type="memory" node will by default be used as system RAM by
>> Linux.  If some of that region needs to be used for DMA buffers, then
>> you'll need to make sure that when a driver allocates a buffer, that
>> the buffer is made valid for DMA operations.  How you do this is
>> architecture specific.
>
> Okay, I think I've got this down
> ...
>>
>> what does 'stbus' mean?  If it is the hardware name for the bus, then
>> it is probably a good choice.  'soc' is often used, but I'm not a big
>> fan of that convention because it doesn't reflect the actual internal
>> architecture of the chip.
>
> Yes, this is what STMicroelectronics call their bus, specifically "STBus".
> Which looks like what the device tree calls a simple- bus.

To be specific, 'simple-bus' is a binding that means a simple memory
mapped bus without any configuration required, but it is not the name
of a bus.  See ePAPR for the actual binding definition (which is
pretty small).  For completeness, the compatible value for this node
should probably be:

compatbile = "<socvendor>,<socmodel>-stbus", "simple-bus";

so that both the specific device device is detailed, as well as the
fact that it is compatible with the simple-bus binding.

As for the node name, "bus" would be fine.  "stbus" would also be
okay, as would "soc" or "soc-bus".  The node name doesn't actually
matter much as operating systems shouldn't ever depend on node names
to determine behaviour, but the generic names recommended practice
should still be followed as much as possible (see section 2.2.2 in
ePAPR.  This is also why "simple-bus" isn't a good name, because
"simple-bus" represents the programming model, not a generic name of
what the node is).

.... and yes, I am aware that i get rather pedantic about this stuff.  :-)

g.

Re: Device tree with early buffer allocations and aliased memory

[David VomLehn]

On Tue, Nov 09, 2010 at 10:25:37PM -0600, Grant Likely wrote:
> On Wed, Nov 03, 2010 at 01:50:37PM -0700, David VomLehn wrote:
> > This got eaten as junk mail, so pardon the slow response. Also, Stuart
...
> > On Sat, Oct 30, 2010 at 10:57:34PM -0500, Grant Likely wrote:
> > > On Wed, Oct 27, 2010 at 2:49 PM, David VomLehn <dvomlehn-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> wrote:
...
> > This still leaves the question of buffers. Buffers with statically and
> > dynamically assigned addresses need some sort of identifier so that
> > they can be referenced by device drivers. There may be more than one
> > buffer per device and a single device may use buffers with statically
> > assigned addresses and buffers with dynamically assigned addresses.
> > Buffers with statically assigned buffers addresses could be handled
> > with something like:
> > 
> > 	device-s {
> > 		compatible = "cisco,device-s";
> > 		cisco,static-buffers = 
> > 			"device-s-b1",<0x24000000 0x00100000>,
> > 			"device-s-b2",<0x60000000 0x00200000>;
> 
> Or, simply:
> 	cisco,static-buffer-b1 = <0x24000000 0x00100000>;
> 	cisco,static-buffer-b2 = <0x60000000 0x00200000>;
> 
> Try to avoid mixing string and cell values in the same property where
> appropriate.  Sometimes doing so is the best binding, but those cases
> are rare.

I started implementing the 'cisco,static-buffer-b1' solution and it feels pretty awkward.
When I want to get the property value, I have to know not only the constant property
name but also a buffer name. The buffer name is really one of the pieces of data I
want to get. I can, of course, write code to scan the node and do a partial match on
the property name, but that doesn't feel like the rest of the DT API.

Not only does this have me doing odd things to property names, but I have a number of
devices. If I should want another buffer, I must create another property name.
This solution has me creating a bunch of property names, which all I really
want is two: "cisco,static-buffers" and "cisco,dynamic-buffers".

I think this is an occasion where it makes sense to mix string and cell values in the
same property.

> g.

-- 
David VL

Re: Device tree with early buffer allocations and aliased memory

[Scott Wood]

On Thu, 18 Nov 2010 14:08:27 -0800
David VomLehn <dvomlehn-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> wrote:

> On Tue, Nov 09, 2010 at 10:25:37PM -0600, Grant Likely wrote:
> > On Wed, Nov 03, 2010 at 01:50:37PM -0700, David VomLehn wrote:
> > > 	device-s {
> > > 		compatible = "cisco,device-s";
> > > 		cisco,static-buffers = 
> > > 			"device-s-b1",<0x24000000 0x00100000>,
> > > 			"device-s-b2",<0x60000000 0x00200000>;
> > 
> > Or, simply:
> > 	cisco,static-buffer-b1 = <0x24000000 0x00100000>;
> > 	cisco,static-buffer-b2 = <0x60000000 0x00200000>;
> > 
> > Try to avoid mixing string and cell values in the same property where
> > appropriate.  Sometimes doing so is the best binding, but those cases
> > are rare.
> 
> I started implementing the 'cisco,static-buffer-b1' solution and it feels pretty awkward.
> When I want to get the property value, I have to know not only the constant property
> name but also a buffer name. The buffer name is really one of the pieces of data I
> want to get. I can, of course, write code to scan the node and do a partial match on
> the property name, but that doesn't feel like the rest of the DT API.
> 
> Not only does this have me doing odd things to property names, but I have a number of
> devices. If I should want another buffer, I must create another property name.
> This solution has me creating a bunch of property names, which all I really
> want is two: "cisco,static-buffers" and "cisco,dynamic-buffers".
> 
> I think this is an occasion where it makes sense to mix string and cell values in the
> same property.

How about:

device-s {
	compatible = "cisco,device-s";
	...

	cisco,static-buffers {
		device-s-b1 = <0x24000000 0x00100000>;
		device-s-b2 = <0x60000000 0x00200000>;
	};
};


-Scott