UIO support for >32-bit physical addresses on 32-bit platforms

UIO support for >32-bit physical addresses on 32-bit platforms

[Kumar Gala]

Hans, Greg,

We are looking at using UIO for some driver work and noticed it assumes the address for MMIO regions is an 'unsigned long'.  This is a problem for the platforms we have in which we support a 36-bit physical address in a 32-bit machine.

Should we just change addr/size in struct uio_mem to u64 always?  At first I was thinking phys_addr_t but realized the addr could be PHYS, LOGICAL, or VIRTUAL.

- k

Re: UIO support for >32-bit physical addresses on 32-bit platforms

[Greg KH]

On Tue, Feb 23, 2010 at 08:54:16AM -0600, Kumar Gala wrote:
> Hans, Greg,
> 
> We are looking at using UIO for some driver work and noticed it
> assumes the address for MMIO regions is an 'unsigned long'.  This is a
> problem for the platforms we have in which we support a 36-bit
> physical address in a 32-bit machine.
> 
> Should we just change addr/size in struct uio_mem to u64 always?  At
> first I was thinking phys_addr_t but realized the addr could be PHYS,
> LOGICAL, or VIRTUAL.

I think that would work out fine, Hans, any ideas if this would cause
any problems with existing code or not?

thanks,

greg k-h

Re: UIO support for >32-bit physical addresses on 32-bit platforms

[Hans J. Koch]

On Tue, Feb 23, 2010 at 07:01:41AM -0800, Greg KH wrote:
> On Tue, Feb 23, 2010 at 08:54:16AM -0600, Kumar Gala wrote:
> > Hans, Greg,
> > 
> > We are looking at using UIO for some driver work and noticed it
> > assumes the address for MMIO regions is an 'unsigned long'.  This is a
> > problem for the platforms we have in which we support a 36-bit
> > physical address in a 32-bit machine.
> > 
> > Should we just change addr/size in struct uio_mem to u64 always?

No, if I didn't miss something important, it's not feasible IMHO.

> > At
> > first I was thinking phys_addr_t but realized the addr could be PHYS,
> > LOGICAL, or VIRTUAL.
> 
> I think that would work out fine, Hans, any ideas if this would cause
> any problems with existing code or not?

It won't work. This is not a UIO problem. UIO just passes addr to other
system functions, and all of them use "unsigned long" for these addresses.
You'd have to change the whole memory management code.

Note that members vm_start and vm_end in struct vm_area_struct are
also unsigned long. Having an u64 in the UIO core doesn't help at all.

Memory above the 4G border can only be accessed through himem mechanisms
or be mapped to an address below that border in lowlevel arch code.

In most cases, I'd say it's bad board design to have memory-mapped hardware
above the 4G border on a 32-bit machine.

Thanks,
Hans

Re: UIO support for >32-bit physical addresses on 32-bit platforms

[Kumar Gala]

On Feb 23, 2010, at 1:49 PM, Hans J. Koch wrote:

> On Tue, Feb 23, 2010 at 07:01:41AM -0800, Greg KH wrote:
>> On Tue, Feb 23, 2010 at 08:54:16AM -0600, Kumar Gala wrote:
>>> Hans, Greg,
>>> 
>>> We are looking at using UIO for some driver work and noticed it
>>> assumes the address for MMIO regions is an 'unsigned long'.  This is a
>>> problem for the platforms we have in which we support a 36-bit
>>> physical address in a 32-bit machine.
>>> 
>>> Should we just change addr/size in struct uio_mem to u64 always?
> 
> No, if I didn't miss something important, it's not feasible IMHO.
> 
>>> At
>>> first I was thinking phys_addr_t but realized the addr could be PHYS,
>>> LOGICAL, or VIRTUAL.
>> 
>> I think that would work out fine, Hans, any ideas if this would cause
>> any problems with existing code or not?
> 
> It won't work. This is not a UIO problem. UIO just passes addr to other
> system functions, and all of them use "unsigned long" for these addresses.
> You'd have to change the whole memory management code.
> 
> Note that members vm_start and vm_end in struct vm_area_struct are
> also unsigned long. Having an u64 in the UIO core doesn't help at all.

This already works for mmap since we pass down a PFN and not an address to remap_pfn_range.  While we can handle any physical address size we can handle one up to 44 bits on most architectures which is well beyond the 36-bit physical address case I'm dealing with on PPC.

> Memory above the 4G border can only be accessed through himem mechanisms
> or be mapped to an address below that border in lowlevel arch code.
> 
> In most cases, I'd say it's bad board design to have memory-mapped hardware
> above the 4G border on a 32-bit machine.

While I might agree with you its a reality on a number of embedded processors and I know three different PowerPC based embedded families that have 36-bit physical addressing to allow for >4G of DDR and thus IO ends up being above the 4G physical boundary and we already support these in the kernel.

- k

Re: UIO support for >32-bit physical addresses on 32-bit platforms

[Hans J. Koch]

On Tue, Feb 23, 2010 at 04:52:56PM -0600, Kumar Gala wrote:
> 
> On Feb 23, 2010, at 1:49 PM, Hans J. Koch wrote:
> 
> > On Tue, Feb 23, 2010 at 07:01:41AM -0800, Greg KH wrote:
> >> On Tue, Feb 23, 2010 at 08:54:16AM -0600, Kumar Gala wrote:
> >>> Hans, Greg,
> >>> 
> >>> We are looking at using UIO for some driver work and noticed it
> >>> assumes the address for MMIO regions is an 'unsigned long'.  This is a
> >>> problem for the platforms we have in which we support a 36-bit
> >>> physical address in a 32-bit machine.
> >>> 
> >>> Should we just change addr/size in struct uio_mem to u64 always?
> > 
> > No, if I didn't miss something important, it's not feasible IMHO.
> > 
> >>> At
> >>> first I was thinking phys_addr_t but realized the addr could be PHYS,
> >>> LOGICAL, or VIRTUAL.
> >> 
> >> I think that would work out fine, Hans, any ideas if this would cause
> >> any problems with existing code or not?
> > 
> > It won't work. This is not a UIO problem. UIO just passes addr to other
> > system functions, and all of them use "unsigned long" for these addresses.
> > You'd have to change the whole memory management code.
> > 
> > Note that members vm_start and vm_end in struct vm_area_struct are
> > also unsigned long. Having an u64 in the UIO core doesn't help at all.
> 
> This already works for mmap since we pass down a PFN and not an address to remap_pfn_range.  While we can handle any physical address size we can handle one up to 44 bits on most architectures which is well beyond the 36-bit physical address case I'm dealing with on PPC.
> 

Sure, there are always ways to handle physical addresses up to the buswidth
of the arch. But above the "natural" size (32 bits on 32-bit archs), it
becomes a bit clumsy. UIO has a documented interface to userspace. The user
is supposed to use the traditional mmap() call. mmap() returns void*, which
is also 32 bits on 32-bit archs.

Because of this, UIO already needs a 32-bit address. It might be possible to
map your I/O memory from above 4G down into the 32-bit space and then use
this address, probably not with UIO_MEM_PHYS but with UIO_MEM_VIRTUAL.
I understand PPC (is that an E500?) has some special features for doing
these kind of shifts in hardware. But I'm not really a PPC expert, I'll
ask a friend of mine.

> > Memory above the 4G border can only be accessed through himem mechanisms
> > or be mapped to an address below that border in lowlevel arch code.
> > 
> > In most cases, I'd say it's bad board design to have memory-mapped hardware
> > above the 4G border on a 32-bit machine.
> 
> While I might agree with you its a reality on a number of embedded processors and I know three different PowerPC based embedded families that have 36-bit physical addressing to allow for >4G of DDR and thus IO ends up being above the 4G physical boundary and we already support these in the kernel.

Well, UIO is (up to now) completely independent of the architecture. If you've
got a good idea how to implement such mappings, feel free to come up with your
suggestions/patches. Just changing struct uio_mem.addr to u64 is certainly not
enough.

I'll give it some thoughts myself, and will discuss the problem with friends,
maybe there's a nice solution. Help from your side is much appreciated.

Thanks,
Hans

> 
> - k

Re: UIO support for >32-bit physical addresses on 32-bit platforms

[Kumar Gala]

On Feb 23, 2010, at 5:39 PM, Hans J. Koch wrote:

> On Tue, Feb 23, 2010 at 04:52:56PM -0600, Kumar Gala wrote:
>> 
>> On Feb 23, 2010, at 1:49 PM, Hans J. Koch wrote:
>> 
>>> On Tue, Feb 23, 2010 at 07:01:41AM -0800, Greg KH wrote:
>>>> On Tue, Feb 23, 2010 at 08:54:16AM -0600, Kumar Gala wrote:
>>>>> Hans, Greg,
>>>>> 
>>>>> We are looking at using UIO for some driver work and noticed it
>>>>> assumes the address for MMIO regions is an 'unsigned long'.  This is a
>>>>> problem for the platforms we have in which we support a 36-bit
>>>>> physical address in a 32-bit machine.
>>>>> 
>>>>> Should we just change addr/size in struct uio_mem to u64 always?
>>> 
>>> No, if I didn't miss something important, it's not feasible IMHO.
>>> 
>>>>> At
>>>>> first I was thinking phys_addr_t but realized the addr could be PHYS,
>>>>> LOGICAL, or VIRTUAL.
>>>> 
>>>> I think that would work out fine, Hans, any ideas if this would cause
>>>> any problems with existing code or not?
>>> 
>>> It won't work. This is not a UIO problem. UIO just passes addr to other
>>> system functions, and all of them use "unsigned long" for these addresses.
>>> You'd have to change the whole memory management code.
>>> 
>>> Note that members vm_start and vm_end in struct vm_area_struct are
>>> also unsigned long. Having an u64 in the UIO core doesn't help at all.
>> 
>> This already works for mmap since we pass down a PFN and not an address to remap_pfn_range.  While we can handle any physical address size we can handle one up to 44 bits on most architectures which is well beyond the 36-bit physical address case I'm dealing with on PPC.
>> 
> 
> Sure, there are always ways to handle physical addresses up to the buswidth
> of the arch. But above the "natural" size (32 bits on 32-bit archs), it
> becomes a bit clumsy. UIO has a documented interface to userspace. The user
> is supposed to use the traditional mmap() call. mmap() returns void*, which
> is also 32 bits on 32-bit archs.
> 
> Because of this, UIO already needs a 32-bit address. It might be possible to
> map your I/O memory from above 4G down into the 32-bit space and then use
> this address, probably not with UIO_MEM_PHYS but with UIO_MEM_VIRTUAL.
> I understand PPC (is that an E500?) has some special features for doing
> these kind of shifts in hardware. But I'm not really a PPC expert, I'll
> ask a friend of mine.

(Yes this is on e500, but also true on the PPC 44x/46x and 745x processors)

Maybe I'm misunderstanding how UIO_MEM_PHYS works (or what it means).  I was under the assumption that it behaves similar to mmap of pci sysfs files.  In that the virtual address in the application is provided in the mmap call and the physical address is coming from the mem region.  As such the MMU is handling the process translation from the virtual address in the app to whatever the physical address might happen to be (if the HW supports that physical address being >32-bits so bit, the application isn't any the wiser).

>>> Memory above the 4G border can only be accessed through himem mechanisms
>>> or be mapped to an address below that border in lowlevel arch code.
>>> 
>>> In most cases, I'd say it's bad board design to have memory-mapped hardware
>>> above the 4G border on a 32-bit machine.
>> 
>> While I might agree with you its a reality on a number of embedded processors and I know three different PowerPC based embedded families that have 36-bit physical addressing to allow for >4G of DDR and thus IO ends up being above the 4G physical boundary and we already support these in the kernel.
> 
> Well, UIO is (up to now) completely independent of the architecture. If you've
> got a good idea how to implement such mappings, feel free to come up with your
> suggestions/patches. Just changing struct uio_mem.addr to u64 is certainly not
> enough.
> 
> I'll give it some thoughts myself, and will discuss the problem with friends,
> maybe there's a nice solution. Help from your side is much appreciated.

For my needs I believe changing uio_mem.addr to a u64 is sufficient for getting UIO_MEM_PHYS to work for the physical address being above 4G.  I need the type to be large to represent a wider PFN range.  Obviously there are limitations (ie, we can't make an IO region >4G in size, etc.)

- k

Re: UIO support for >32-bit physical addresses on 32-bit platforms

[Hans J. Koch]

On Tue, Feb 23, 2010 at 09:24:42PM -0600, Kumar Gala wrote:
> > Well, UIO is (up to now) completely independent of the architecture. If you've
> > got a good idea how to implement such mappings, feel free to come up with your
> > suggestions/patches. Just changing struct uio_mem.addr to u64 is certainly not
> > enough.
> > 
> > I'll give it some thoughts myself, and will discuss the problem with friends,
> > maybe there's a nice solution. Help from your side is much appreciated.
> 
> For my needs I believe changing uio_mem.addr to a u64 is sufficient for getting UIO_MEM_PHYS to work for the physical address being above 4G.  I need the type to be large to represent a wider PFN range.  Obviously there are limitations (ie, we can't make an IO region >4G in size, etc.)
> 

Hmm, after some research it seems you're right. But the type for the addr
member of struct uio_mem should be phys_addr_t, not u64. phys_addr_t is
an u64 on systems that support it, and an u32 otherwise.

It'll probably work with UIO_MEM_LOGICAL and UIO_MEM_VIRTUAL as well.

Could you hack up a patch to that effect, and test it? If it works for you,
I think we should apply it.

Thanks for pointing this out.

Hans

> - k