Question about DMA on 1:1 mapping dom0 of arm64

Question about DMA on 1:1 mapping dom0 of arm64

[Chen Baozi]

Hi all,

According to my recent experience, there might be some problems of swiotlb
dma map on 1:1 mapping arm64 dom0 with large memory. The issue is like below:

For those arm64 server with large memory, it is possible to set dom0_mem >
4G (e.g. I have one set with 16G). In this case, according to my understanding,
there is chance that the dom0 kernel needs to map some buffers above 4G to do
DMA operations (e.g. in snps,dwmac ethernet driver). However, most DMA engines
support only 32-bit physical address, thus aren't able to operate directly on
those memory. IIUC, swiotlb is implemented to solve this (using bounce buffer),
if there is no IOMMU or IOMMU is not enabled on the system. Sadly, it seems
that xen_swiotlb_map_page in my dom0 kernel allocates
(start_dma_addr = 0x944800000) the buffers for DMA above 4G which fails
dma_capable() checking and was then unable to return from xen_swiotlb_map_page()
successfully.

If I set dom0_mem to a small value (e.g. 512M), which makes all physical memory
of dom0 below 4G, everything goes fine.

I am not familiar with swiotlb-xen, so there would be misunderstanding about
the current situation. Fix me if I did/understood anything wrong.

Any ideas?

Cheers,

Chen Baozi

Re: Question about DMA on 1:1 mapping dom0 of arm64

[Ian Campbell]

On Fri, 2015-04-17 at 19:24 +0800, Chen Baozi wrote:
> Hi all,
> 
> According to my recent experience, there might be some problems of swiotlb
> dma map on 1:1 mapping arm64 dom0 with large memory. The issue is like below:
> 
> For those arm64 server with large memory, it is possible to set dom0_mem >
> 4G (e.g. I have one set with 16G). In this case, according to my understanding,
> there is chance that the dom0 kernel needs to map some buffers above 4G to do

                                                                 ^below?

> DMA operations (e.g. in snps,dwmac ethernet driver). However, most DMA engines
> support only 32-bit physical address, thus aren't able to operate directly on
> those memory.

Even on arm64 systems with RAM above 4GB? That seems.... short-sighted.
Oh well, I suppose we have to live with it.

>  IIUC, swiotlb is implemented to solve this (using bounce buffer),
> if there is no IOMMU or IOMMU is not enabled on the system. Sadly, it seems
> that xen_swiotlb_map_page in my dom0 kernel allocates
> (start_dma_addr = 0x944800000) the buffers for DMA above 4G which fails
> dma_capable() checking and was then unable to return from xen_swiotlb_map_page()
> successfully.

The swiotlb bounce buffer have been allocated below 4GB? I suspect that
xen_swiotlb_init is buggy for ARM -- it allocates some random pages and
then swizzles the backing pages for ones < 4G, but that won't work on an
ARM dom0 with a 1:1 mapping, I don't think. Do you see error messages
along those lines?

Essentially I think either xen_swiotlb_fixup is unable to work on ARM,
or the following:
        start_dma_addr = xen_virt_to_bus(xen_io_tlb_start);
is returning 1:1 and not reflecting the fixup.

> If I set dom0_mem to a small value (e.g. 512M), which makes all physical memory
> of dom0 below 4G, everything goes fine.

So you are getting allocated memory below 4G?

You message on IRC suggested you weren't, did you hack around this?

I think we have two options, either xen_swiotlb_init allocates pages
below 4GB (e.g. __GFP_DMA) or we do something to allow xen_swiotlb_fixup
to actually work even on a 1:1 dom0.

Although the first option seems preferable at first glance it has the
short coming that it requires dom0 to have some memory below 4GB, which
might not necessarily be the case. The second option seems like it might
be uglier but doesn't suffer from this issue.

Can you please look and find out if the IPA at 0x944800000 is actually
backed by 1:1 RAM or if xen_swiotlb_fixup has done it's job and updated
things such that the associated PAs are below 4GB?

Ian.

> I am not familiar with swiotlb-xen, so there would be misunderstanding about
> the current situation. Fix me if I did/understood anything wrong.
> 
> Any ideas?
> 
> Cheers,
> 
> Chen Baozi

Re: Question about DMA on 1:1 mapping dom0 of arm64

[Stefano Stabellini]

On Fri, 17 Apr 2015, Ian Campbell wrote:
> On Fri, 2015-04-17 at 19:24 +0800, Chen Baozi wrote:
> > Hi all,
> > 
> > According to my recent experience, there might be some problems of swiotlb
> > dma map on 1:1 mapping arm64 dom0 with large memory. The issue is like below:
> > 
> > For those arm64 server with large memory, it is possible to set dom0_mem >
> > 4G (e.g. I have one set with 16G). In this case, according to my understanding,
> > there is chance that the dom0 kernel needs to map some buffers above 4G to do
> 
>                                                                  ^below?
> 
> > DMA operations (e.g. in snps,dwmac ethernet driver). However, most DMA engines
> > support only 32-bit physical address, thus aren't able to operate directly on
> > those memory.
> 
> Even on arm64 systems with RAM above 4GB? That seems.... short-sighted.
> Oh well, I suppose we have to live with it.
> 
> >  IIUC, swiotlb is implemented to solve this (using bounce buffer),
> > if there is no IOMMU or IOMMU is not enabled on the system. Sadly, it seems
> > that xen_swiotlb_map_page in my dom0 kernel allocates
> > (start_dma_addr = 0x944800000) the buffers for DMA above 4G which fails
> > dma_capable() checking and was then unable to return from xen_swiotlb_map_page()
> > successfully.
> 
> The swiotlb bounce buffer have been allocated below 4GB? I suspect that
> xen_swiotlb_init is buggy for ARM -- it allocates some random pages and
> then swizzles the backing pages for ones < 4G, but that won't work on an
> ARM dom0 with a 1:1 mapping, I don't think. Do you see error messages
> along those lines?
> 
> Essentially I think either xen_swiotlb_fixup is unable to work on ARM,
> or the following:
>         start_dma_addr = xen_virt_to_bus(xen_io_tlb_start);
> is returning 1:1 and not reflecting the fixup.

The swiotlb on arm doesn't necessarily get memory under 4G, see my other reply.


> > If I set dom0_mem to a small value (e.g. 512M), which makes all physical memory
> > of dom0 below 4G, everything goes fine.
> 
> So you are getting allocated memory below 4G?
> 
> You message on IRC suggested you weren't, did you hack around this?
> 
> I think we have two options, either xen_swiotlb_init allocates pages
> below 4GB (e.g. __GFP_DMA) or we do something to allow xen_swiotlb_fixup
> to actually work even on a 1:1 dom0.

I don't think that making xen_swiotlb_fixup work on ARM is a good idea:
it would break the 1:1.


> Although the first option seems preferable at first glance it has the
> short coming that it requires dom0 to have some memory below 4GB, which
> might not necessarily be the case.

I think we should arrange dom0 to get some memory under 4G to begin
with, not necessarily all of it.


> The second option seems like it might
> be uglier but doesn't suffer from this issue.
> 
> Can you please look and find out if the IPA at 0x944800000 is actually
> backed by 1:1 RAM or if xen_swiotlb_fixup has done it's job and updated
> things such that the associated PAs are below 4GB?

Re: Question about DMA on 1:1 mapping dom0 of arm64

[Ian Campbell]

On Fri, 2015-04-17 at 15:34 +0100, Stefano Stabellini wrote:
> > > If I set dom0_mem to a small value (e.g. 512M), which makes all physical memory
> > > of dom0 below 4G, everything goes fine.
> > 
> > So you are getting allocated memory below 4G?
> > 
> > You message on IRC suggested you weren't, did you hack around this?
> > 
> > I think we have two options, either xen_swiotlb_init allocates pages
> > below 4GB (e.g. __GFP_DMA) or we do something to allow xen_swiotlb_fixup
> > to actually work even on a 1:1 dom0.
> 
> I don't think that making xen_swiotlb_fixup work on ARM is a good idea:
> it would break the 1:1.

This would actually work though, I think, because this is the swiotlb so
we definitely have the opportunity to return the actual DMA address
whenever we use this buffer and the device will use it in the right
places for sure.

The swiotlb buffer can't ever get reused for anything else so we don't
even need to worry about undoing the damage later.

> > Although the first option seems preferable at first glance it has the
> > short coming that it requires dom0 to have some memory below 4GB, which
> > might not necessarily be the case.
> 
> I think we should arrange dom0 to get some memory under 4G to begin
> with, not necessarily all of it.

It's another option for sure, the question is how to decide how much,
and how to make it configurable etc.

Ian.

Re: Question about DMA on 1:1 mapping dom0 of arm64

[Stefano Stabellini]

On Fri, 17 Apr 2015, Ian Campbell wrote:
> On Fri, 2015-04-17 at 15:34 +0100, Stefano Stabellini wrote:
> > > > If I set dom0_mem to a small value (e.g. 512M), which makes all physical memory
> > > > of dom0 below 4G, everything goes fine.
> > > 
> > > So you are getting allocated memory below 4G?
> > > 
> > > You message on IRC suggested you weren't, did you hack around this?
> > > 
> > > I think we have two options, either xen_swiotlb_init allocates pages
> > > below 4GB (e.g. __GFP_DMA) or we do something to allow xen_swiotlb_fixup
> > > to actually work even on a 1:1 dom0.
> > 
> > I don't think that making xen_swiotlb_fixup work on ARM is a good idea:
> > it would break the 1:1.
> 
> This would actually work though, I think, because this is the swiotlb so
> we definitely have the opportunity to return the actual DMA address
> whenever we use this buffer and the device will use it in the right
> places for sure.

The code is pretty complex as is -- I would rather avoid adding more
complexity to it.  For example we would need to bring back a mechanism
to track dma address -> pseudo-physical address mappings on arm, even
though it would be far simpler of course.

Also I think it makes sense to use the swiotlb buffer for its original
purpose.

If we could introduce a mechanism to get a lower than 4G buffer in dom0,
but matching the 1:1, I think it would make the maintenance much easier
on the linux side.


> The swiotlb buffer can't ever get reused for anything else so we don't
> even need to worry about undoing the damage later.
> 
> > > Although the first option seems preferable at first glance it has the
> > > short coming that it requires dom0 to have some memory below 4GB, which
> > > might not necessarily be the case.
> > 
> > I think we should arrange dom0 to get some memory under 4G to begin
> > with, not necessarily all of it.
> 
> It's another option for sure, the question is how to decide how much,
> and how to make it configurable etc.
 
Adding a memory region below 4G should be relatively easy, right?  Is it
sizing it up the problem? FYI the default size of the swiotlb buffer in
Linux is 64M, but it shoul be able to cope with less than that.

Re: Question about DMA on 1:1 mapping dom0 of arm64

[Ian Campbell]

On Fri, 2015-04-17 at 17:13 +0100, Stefano Stabellini wrote:
> On Fri, 17 Apr 2015, Ian Campbell wrote:
> > On Fri, 2015-04-17 at 15:34 +0100, Stefano Stabellini wrote:
> > > > > If I set dom0_mem to a small value (e.g. 512M), which makes all physical memory
> > > > > of dom0 below 4G, everything goes fine.
> > > > 
> > > > So you are getting allocated memory below 4G?
> > > > 
> > > > You message on IRC suggested you weren't, did you hack around this?
> > > > 
> > > > I think we have two options, either xen_swiotlb_init allocates pages
> > > > below 4GB (e.g. __GFP_DMA) or we do something to allow xen_swiotlb_fixup
> > > > to actually work even on a 1:1 dom0.
> > > 
> > > I don't think that making xen_swiotlb_fixup work on ARM is a good idea:
> > > it would break the 1:1.
> > 
> > This would actually work though, I think, because this is the swiotlb so
> > we definitely have the opportunity to return the actual DMA address
> > whenever we use this buffer and the device will use it in the right
> > places for sure.
> 
> The code is pretty complex as is -- I would rather avoid adding more
> complexity to it.  For example we would need to bring back a mechanism
> to track dma address -> pseudo-physical address mappings on arm, even
> though it would be far simpler of course.

There's no need for any of that, just initialise start_dma_addr with the
correct DMA addr when you do the fixup and the swiotlb code will already
take care of everything, won't it?.

That DMA addr would be given to us by Xen as we do the flip.

> Also I think it makes sense to use the swiotlb buffer for its original
> purpose.
> 
> If we could introduce a mechanism to get a lower than 4G buffer in dom0,
> but matching the 1:1, I think it would make the maintenance much easier
> on the linux side.
> 
> 
> > The swiotlb buffer can't ever get reused for anything else so we don't
> > even need to worry about undoing the damage later.
> > 
> > > > Although the first option seems preferable at first glance it has the
> > > > short coming that it requires dom0 to have some memory below 4GB, which
> > > > might not necessarily be the case.
> > > 
> > > I think we should arrange dom0 to get some memory under 4G to begin
> > > with, not necessarily all of it.
> > 
> > It's another option for sure, the question is how to decide how much,
> > and how to make it configurable etc.
>  
> Adding a memory region below 4G should be relatively easy, right?  Is it
> sizing it up the problem? FYI the default size of the swiotlb buffer in
> Linux is 64M, but it shoul be able to cope with less than that.

Picking a size might be tricky, you can't just say "Linux is 64M", what
if that 64M gets used for something else first.

Given a machine to 2GB of low RAM and 126GB of high RAM and a 27G dom0
which fraction do you put below 4GB?

But maybe it works to just say that dom0 gets as much lowmem as the host
memory map and our ability to allocate allows for. That's not entirely
trivial either though, in case Xen has (by chance) allocated any lowmem
for itself the bank bin packing stuff will get even more complex.
Likewise firmware reserved regions.

Ian.

Re: Question about DMA on 1:1 mapping dom0 of arm64

[Chen Baozi]

On Fri, Apr 17, 2015 at 05:13:16PM +0100, Stefano Stabellini wrote:
> On Fri, 17 Apr 2015, Ian Campbell wrote:
> > On Fri, 2015-04-17 at 15:34 +0100, Stefano Stabellini wrote:
> > > > > If I set dom0_mem to a small value (e.g. 512M), which makes all physical memory
> > > > > of dom0 below 4G, everything goes fine.
> > > > 
> > > > So you are getting allocated memory below 4G?
> > > > 
> > > > You message on IRC suggested you weren't, did you hack around this?
> > > > 
> > > > I think we have two options, either xen_swiotlb_init allocates pages
> > > > below 4GB (e.g. __GFP_DMA) or we do something to allow xen_swiotlb_fixup
> > > > to actually work even on a 1:1 dom0.
> > > 
> > > I don't think that making xen_swiotlb_fixup work on ARM is a good idea:
> > > it would break the 1:1.
> > 
> > This would actually work though, I think, because this is the swiotlb so
> > we definitely have the opportunity to return the actual DMA address
> > whenever we use this buffer and the device will use it in the right
> > places for sure.
> 
> The code is pretty complex as is -- I would rather avoid adding more
> complexity to it.  For example we would need to bring back a mechanism
> to track dma address -> pseudo-physical address mappings on arm, even
> though it would be far simpler of course.
> 
> Also I think it makes sense to use the swiotlb buffer for its original
> purpose.
> 
> If we could introduce a mechanism to get a lower than 4G buffer in dom0,
> but matching the 1:1, I think it would make the maintenance much easier
> on the linux side.

+1

Actually, we have already had the mechanism on arm32 to populate at least
one bank of memory below 4G. Thus, the only thing we have to do on the
hypervisor side is to make arm32 and arm64 share the same process in
allocate_memory_11(), removing the 'lowmem = is_32bit_domain(d)' related
conditions. If this is acceptable, the only thing we need to do in Linux
kernel is to add the __GFP_DMA flag when allocating pages for xen_io_tlb_start
in xen_swiotlb_init.

Baozi.

Re: Question about DMA on 1:1 mapping dom0 of arm64

[Chen Baozi]

On Sat, Apr 18, 2015 at 05:08:58PM +0800, Chen Baozi wrote:
> On Fri, Apr 17, 2015 at 05:13:16PM +0100, Stefano Stabellini wrote:
> > On Fri, 17 Apr 2015, Ian Campbell wrote:
> > > On Fri, 2015-04-17 at 15:34 +0100, Stefano Stabellini wrote:
> > > > > > If I set dom0_mem to a small value (e.g. 512M), which makes all physical memory
> > > > > > of dom0 below 4G, everything goes fine.
> > > > > 
> > > > > So you are getting allocated memory below 4G?
> > > > > 
> > > > > You message on IRC suggested you weren't, did you hack around this?
> > > > > 
> > > > > I think we have two options, either xen_swiotlb_init allocates pages
> > > > > below 4GB (e.g. __GFP_DMA) or we do something to allow xen_swiotlb_fixup
> > > > > to actually work even on a 1:1 dom0.
> > > > 
> > > > I don't think that making xen_swiotlb_fixup work on ARM is a good idea:
> > > > it would break the 1:1.
> > > 
> > > This would actually work though, I think, because this is the swiotlb so
> > > we definitely have the opportunity to return the actual DMA address
> > > whenever we use this buffer and the device will use it in the right
> > > places for sure.
> > 
> > The code is pretty complex as is -- I would rather avoid adding more
> > complexity to it.  For example we would need to bring back a mechanism
> > to track dma address -> pseudo-physical address mappings on arm, even
> > though it would be far simpler of course.
> > 
> > Also I think it makes sense to use the swiotlb buffer for its original
> > purpose.
> > 
> > If we could introduce a mechanism to get a lower than 4G buffer in dom0,
> > but matching the 1:1, I think it would make the maintenance much easier
> > on the linux side.
> 
> +1
> 
> Actually, we have already had the mechanism on arm32 to populate at least
> one bank of memory below 4G. Thus, the only thing we have to do on the
> hypervisor side is to make arm32 and arm64 share the same process in
> allocate_memory_11(), removing the 'lowmem = is_32bit_domain(d)' related
> conditions. If this is acceptable, the only thing we need to do in Linux
> kernel is to add the __GFP_DMA flag when allocating pages for xen_io_tlb_start
> in xen_swiotlb_init.
> 

This is the hacks I'm using:

diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 40a5303..d83a2b1 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -264,7 +264,7 @@ static void allocate_memory_11(struct domain *d, struct kernel_info *kinfo)
     unsigned int order = get_11_allocation_size(kinfo->unassigned_mem);
     int i;
 
-    bool_t lowmem = is_32bit_domain(d);
+    bool_t lowmem = true;
     unsigned int bits;
 
     printk("Allocating 1:1 mappings totalling %ldMB for dom0:\n",
@@ -279,7 +279,7 @@ static void allocate_memory_11(struct domain *d, struct kernel_info *kinfo)
      */
     while ( order >= min_low_order )
     {
-        for ( bits = order ; bits <= (lowmem ? 32 : PADDR_BITS); bits++ )
+        for ( bits = order ; bits <= 32 ; bits++ )
         {
             pg = alloc_domheap_pages(d, order, MEMF_bits(bits));
             if ( pg != NULL )

Re: Question about DMA on 1:1 mapping dom0 of arm64

[Ian Campbell]

On Sat, 2015-04-18 at 17:23 +0800, Chen Baozi wrote:
> On Sat, Apr 18, 2015 at 05:08:58PM +0800, Chen Baozi wrote:
> > On Fri, Apr 17, 2015 at 05:13:16PM +0100, Stefano Stabellini wrote:
> > > On Fri, 17 Apr 2015, Ian Campbell wrote:
> > > > On Fri, 2015-04-17 at 15:34 +0100, Stefano Stabellini wrote:
> > > > > > > If I set dom0_mem to a small value (e.g. 512M), which makes all physical memory
> > > > > > > of dom0 below 4G, everything goes fine.
> > > > > > 
> > > > > > So you are getting allocated memory below 4G?
> > > > > > 
> > > > > > You message on IRC suggested you weren't, did you hack around this?
> > > > > > 
> > > > > > I think we have two options, either xen_swiotlb_init allocates pages
> > > > > > below 4GB (e.g. __GFP_DMA) or we do something to allow xen_swiotlb_fixup
> > > > > > to actually work even on a 1:1 dom0.
> > > > > 
> > > > > I don't think that making xen_swiotlb_fixup work on ARM is a good idea:
> > > > > it would break the 1:1.
> > > > 
> > > > This would actually work though, I think, because this is the swiotlb so
> > > > we definitely have the opportunity to return the actual DMA address
> > > > whenever we use this buffer and the device will use it in the right
> > > > places for sure.
> > > 
> > > The code is pretty complex as is -- I would rather avoid adding more
> > > complexity to it.  For example we would need to bring back a mechanism
> > > to track dma address -> pseudo-physical address mappings on arm, even
> > > though it would be far simpler of course.
> > > 
> > > Also I think it makes sense to use the swiotlb buffer for its original
> > > purpose.
> > > 
> > > If we could introduce a mechanism to get a lower than 4G buffer in dom0,
> > > but matching the 1:1, I think it would make the maintenance much easier
> > > on the linux side.
> > 
> > +1
> > 
> > Actually, we have already had the mechanism on arm32 to populate at least
> > one bank of memory below 4G. Thus, the only thing we have to do on the
> > hypervisor side is to make arm32 and arm64 share the same process in
> > allocate_memory_11(), removing the 'lowmem = is_32bit_domain(d)' related
> > conditions. If this is acceptable,

Ah yes, I'd forgotten we already handled this for 32-bit, so enabling it
for 64-bit does indeed make sense.

However what you do is unfortunately not quite correct, since on a
64-bit system which has no RAM at all under 4GB it will now fail to
populate the first bank and error out.

I can think of two solutions off hand, either:

Either: initialise lowmem to (is_32bit_domain(d) ||
ram_exists_below_4g()) (which in practice is equivalent to
ram_exists_below_4g() since that is always going to be true for a 32-bit
system).

Or; Allow the initial bank0 filling to also fallback to non-lowmem (on
64-bit only) as the following stuff for subsequent banks does.

The first option is tricky because I'm not sure what
ram_exists_below_4g() would look like inside, i.e. whether it can be
queried from the page allocator or if we would need to make a note of
this fact when parsing/settingup memory early on.

The second one is tricky because it opens up the possibility of
ploughing on if lowmem can't be allocated, even if it is strictly
required on the platform.

I think on balance the first would be better.

>  the only thing we need to do in Linux
> > kernel is to add the __GFP_DMA flag when allocating pages for xen_io_tlb_start
> > in xen_swiotlb_init.
> > 
> 
> This is the hacks I'm using:
> 
> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
> index 40a5303..d83a2b1 100644
> --- a/xen/arch/arm/domain_build.c
> +++ b/xen/arch/arm/domain_build.c
> @@ -264,7 +264,7 @@ static void allocate_memory_11(struct domain *d, struct kernel_info *kinfo)
>      unsigned int order = get_11_allocation_size(kinfo->unassigned_mem);
>      int i;
>  
> -    bool_t lowmem = is_32bit_domain(d);
> +    bool_t lowmem = true;
>      unsigned int bits;
>  
>      printk("Allocating 1:1 mappings totalling %ldMB for dom0:\n",
> @@ -279,7 +279,7 @@ static void allocate_memory_11(struct domain *d, struct kernel_info *kinfo)
>       */
>      while ( order >= min_low_order )
>      {
> -        for ( bits = order ; bits <= (lowmem ? 32 : PADDR_BITS); bits++ )
> +        for ( bits = order ; bits <= 32 ; bits++ )

I think leave this as is, even if lowmem ends up being const.

Ian.

Re: Question about DMA on 1:1 mapping dom0 of arm64

[Stefano Stabellini]

On Sat, 18 Apr 2015, Chen Baozi wrote:
> On Fri, Apr 17, 2015 at 05:13:16PM +0100, Stefano Stabellini wrote:
> > On Fri, 17 Apr 2015, Ian Campbell wrote:
> > > On Fri, 2015-04-17 at 15:34 +0100, Stefano Stabellini wrote:
> > > > > > If I set dom0_mem to a small value (e.g. 512M), which makes all physical memory
> > > > > > of dom0 below 4G, everything goes fine.
> > > > > 
> > > > > So you are getting allocated memory below 4G?
> > > > > 
> > > > > You message on IRC suggested you weren't, did you hack around this?
> > > > > 
> > > > > I think we have two options, either xen_swiotlb_init allocates pages
> > > > > below 4GB (e.g. __GFP_DMA) or we do something to allow xen_swiotlb_fixup
> > > > > to actually work even on a 1:1 dom0.
> > > > 
> > > > I don't think that making xen_swiotlb_fixup work on ARM is a good idea:
> > > > it would break the 1:1.
> > > 
> > > This would actually work though, I think, because this is the swiotlb so
> > > we definitely have the opportunity to return the actual DMA address
> > > whenever we use this buffer and the device will use it in the right
> > > places for sure.
> > 
> > The code is pretty complex as is -- I would rather avoid adding more
> > complexity to it.  For example we would need to bring back a mechanism
> > to track dma address -> pseudo-physical address mappings on arm, even
> > though it would be far simpler of course.
> > 
> > Also I think it makes sense to use the swiotlb buffer for its original
> > purpose.
> > 
> > If we could introduce a mechanism to get a lower than 4G buffer in dom0,
> > but matching the 1:1, I think it would make the maintenance much easier
> > on the linux side.
> 
> +1
> 
> Actually, we have already had the mechanism on arm32 to populate at least
> one bank of memory below 4G. Thus, the only thing we have to do on the
> hypervisor side is to make arm32 and arm64 share the same process in
> allocate_memory_11(), removing the 'lowmem = is_32bit_domain(d)' related
> conditions. If this is acceptable, the only thing we need to do in Linux
> kernel is to add the __GFP_DMA flag when allocating pages for xen_io_tlb_start
> in xen_swiotlb_init.

Please send out the Linux patch using __GFP_DMA and I'll queue it up.

Re: Question about DMA on 1:1 mapping dom0 of arm64

[Ian Campbell]

On Mon, 2015-04-20 at 10:58 +0100, Stefano Stabellini wrote:
> On Sat, 18 Apr 2015, Chen Baozi wrote:
> > On Fri, Apr 17, 2015 at 05:13:16PM +0100, Stefano Stabellini wrote:
> > > On Fri, 17 Apr 2015, Ian Campbell wrote:
> > > > On Fri, 2015-04-17 at 15:34 +0100, Stefano Stabellini wrote:
> > > > > > > If I set dom0_mem to a small value (e.g. 512M), which makes all physical memory
> > > > > > > of dom0 below 4G, everything goes fine.
> > > > > > 
> > > > > > So you are getting allocated memory below 4G?
> > > > > > 
> > > > > > You message on IRC suggested you weren't, did you hack around this?
> > > > > > 
> > > > > > I think we have two options, either xen_swiotlb_init allocates pages
> > > > > > below 4GB (e.g. __GFP_DMA) or we do something to allow xen_swiotlb_fixup
> > > > > > to actually work even on a 1:1 dom0.
> > > > > 
> > > > > I don't think that making xen_swiotlb_fixup work on ARM is a good idea:
> > > > > it would break the 1:1.
> > > > 
> > > > This would actually work though, I think, because this is the swiotlb so
> > > > we definitely have the opportunity to return the actual DMA address
> > > > whenever we use this buffer and the device will use it in the right
> > > > places for sure.
> > > 
> > > The code is pretty complex as is -- I would rather avoid adding more
> > > complexity to it.  For example we would need to bring back a mechanism
> > > to track dma address -> pseudo-physical address mappings on arm, even
> > > though it would be far simpler of course.
> > > 
> > > Also I think it makes sense to use the swiotlb buffer for its original
> > > purpose.
> > > 
> > > If we could introduce a mechanism to get a lower than 4G buffer in dom0,
> > > but matching the 1:1, I think it would make the maintenance much easier
> > > on the linux side.
> > 
> > +1
> > 
> > Actually, we have already had the mechanism on arm32 to populate at least
> > one bank of memory below 4G. Thus, the only thing we have to do on the
> > hypervisor side is to make arm32 and arm64 share the same process in
> > allocate_memory_11(), removing the 'lowmem = is_32bit_domain(d)' related
> > conditions. If this is acceptable, the only thing we need to do in Linux
> > kernel is to add the __GFP_DMA flag when allocating pages for xen_io_tlb_start
> > in xen_swiotlb_init.
> 
> Please send out the Linux patch using __GFP_DMA and I'll queue it up.

What happens with __GFP_DMA if no suitable memory is available (i.e. all
of RAM is >4GB)?

Ian.

Re: Question about DMA on 1:1 mapping dom0 of arm64

[Stefano Stabellini]

On Mon, 20 Apr 2015, Ian Campbell wrote:
> On Mon, 2015-04-20 at 10:58 +0100, Stefano Stabellini wrote:
> > On Sat, 18 Apr 2015, Chen Baozi wrote:
> > > On Fri, Apr 17, 2015 at 05:13:16PM +0100, Stefano Stabellini wrote:
> > > > On Fri, 17 Apr 2015, Ian Campbell wrote:
> > > > > On Fri, 2015-04-17 at 15:34 +0100, Stefano Stabellini wrote:
> > > > > > > > If I set dom0_mem to a small value (e.g. 512M), which makes all physical memory
> > > > > > > > of dom0 below 4G, everything goes fine.
> > > > > > > 
> > > > > > > So you are getting allocated memory below 4G?
> > > > > > > 
> > > > > > > You message on IRC suggested you weren't, did you hack around this?
> > > > > > > 
> > > > > > > I think we have two options, either xen_swiotlb_init allocates pages
> > > > > > > below 4GB (e.g. __GFP_DMA) or we do something to allow xen_swiotlb_fixup
> > > > > > > to actually work even on a 1:1 dom0.
> > > > > > 
> > > > > > I don't think that making xen_swiotlb_fixup work on ARM is a good idea:
> > > > > > it would break the 1:1.
> > > > > 
> > > > > This would actually work though, I think, because this is the swiotlb so
> > > > > we definitely have the opportunity to return the actual DMA address
> > > > > whenever we use this buffer and the device will use it in the right
> > > > > places for sure.
> > > > 
> > > > The code is pretty complex as is -- I would rather avoid adding more
> > > > complexity to it.  For example we would need to bring back a mechanism
> > > > to track dma address -> pseudo-physical address mappings on arm, even
> > > > though it would be far simpler of course.
> > > > 
> > > > Also I think it makes sense to use the swiotlb buffer for its original
> > > > purpose.
> > > > 
> > > > If we could introduce a mechanism to get a lower than 4G buffer in dom0,
> > > > but matching the 1:1, I think it would make the maintenance much easier
> > > > on the linux side.
> > > 
> > > +1
> > > 
> > > Actually, we have already had the mechanism on arm32 to populate at least
> > > one bank of memory below 4G. Thus, the only thing we have to do on the
> > > hypervisor side is to make arm32 and arm64 share the same process in
> > > allocate_memory_11(), removing the 'lowmem = is_32bit_domain(d)' related
> > > conditions. If this is acceptable, the only thing we need to do in Linux
> > > kernel is to add the __GFP_DMA flag when allocating pages for xen_io_tlb_start
> > > in xen_swiotlb_init.
> > 
> > Please send out the Linux patch using __GFP_DMA and I'll queue it up.
> 
> What happens with __GFP_DMA if no suitable memory is available (i.e. all
> of RAM is >4GB)?

__get_free_pages would fail and xen_swiotlb_init will try again with a
smaller size and print a warning.

If no RAM under 4G is available, xen_swiotlb_init will fail with an
error. However it is probably better to fail explicitly with an error
message than failing with a stack trace at some point down the line when
DMA is actually done.

Re: Question about DMA on 1:1 mapping dom0 of arm64

[Ian Campbell]

On Mon, 2015-04-20 at 11:38 +0100, Stefano Stabellini wrote:
> On Mon, 20 Apr 2015, Ian Campbell wrote:
> > On Mon, 2015-04-20 at 10:58 +0100, Stefano Stabellini wrote:
> > > On Sat, 18 Apr 2015, Chen Baozi wrote:
> > > > On Fri, Apr 17, 2015 at 05:13:16PM +0100, Stefano Stabellini wrote:
> > > > > On Fri, 17 Apr 2015, Ian Campbell wrote:
> > > > > > On Fri, 2015-04-17 at 15:34 +0100, Stefano Stabellini wrote:
> > > > > > > > > If I set dom0_mem to a small value (e.g. 512M), which makes all physical memory
> > > > > > > > > of dom0 below 4G, everything goes fine.
> > > > > > > > 
> > > > > > > > So you are getting allocated memory below 4G?
> > > > > > > > 
> > > > > > > > You message on IRC suggested you weren't, did you hack around this?
> > > > > > > > 
> > > > > > > > I think we have two options, either xen_swiotlb_init allocates pages
> > > > > > > > below 4GB (e.g. __GFP_DMA) or we do something to allow xen_swiotlb_fixup
> > > > > > > > to actually work even on a 1:1 dom0.
> > > > > > > 
> > > > > > > I don't think that making xen_swiotlb_fixup work on ARM is a good idea:
> > > > > > > it would break the 1:1.
> > > > > > 
> > > > > > This would actually work though, I think, because this is the swiotlb so
> > > > > > we definitely have the opportunity to return the actual DMA address
> > > > > > whenever we use this buffer and the device will use it in the right
> > > > > > places for sure.
> > > > > 
> > > > > The code is pretty complex as is -- I would rather avoid adding more
> > > > > complexity to it.  For example we would need to bring back a mechanism
> > > > > to track dma address -> pseudo-physical address mappings on arm, even
> > > > > though it would be far simpler of course.
> > > > > 
> > > > > Also I think it makes sense to use the swiotlb buffer for its original
> > > > > purpose.
> > > > > 
> > > > > If we could introduce a mechanism to get a lower than 4G buffer in dom0,
> > > > > but matching the 1:1, I think it would make the maintenance much easier
> > > > > on the linux side.
> > > > 
> > > > +1
> > > > 
> > > > Actually, we have already had the mechanism on arm32 to populate at least
> > > > one bank of memory below 4G. Thus, the only thing we have to do on the
> > > > hypervisor side is to make arm32 and arm64 share the same process in
> > > > allocate_memory_11(), removing the 'lowmem = is_32bit_domain(d)' related
> > > > conditions. If this is acceptable, the only thing we need to do in Linux
> > > > kernel is to add the __GFP_DMA flag when allocating pages for xen_io_tlb_start
> > > > in xen_swiotlb_init.
> > > 
> > > Please send out the Linux patch using __GFP_DMA and I'll queue it up.
> > 
> > What happens with __GFP_DMA if no suitable memory is available (i.e. all
> > of RAM is >4GB)?
> 
> __get_free_pages would fail and xen_swiotlb_init will try again with a
> smaller size and print a warning.
> 
> If no RAM under 4G is available,

This is always going to be the case on e.g. X-Gene where all RAM is >4G
(starts at 128GB IIRC). IOW just doing it like this is going to break on
some arm64 platforms.

>  xen_swiotlb_init will fail with an
> error. However it is probably better to fail explicitly with an error
> message than failing with a stack trace at some point down the line when
> DMA is actually done.

Ian.

Re: Question about DMA on 1:1 mapping dom0 of arm64

[Stefano Stabellini]

On Mon, 20 Apr 2015, Ian Campbell wrote:
> On Mon, 2015-04-20 at 11:38 +0100, Stefano Stabellini wrote:
> > On Mon, 20 Apr 2015, Ian Campbell wrote:
> > > On Mon, 2015-04-20 at 10:58 +0100, Stefano Stabellini wrote:
> > > > On Sat, 18 Apr 2015, Chen Baozi wrote:
> > > > > On Fri, Apr 17, 2015 at 05:13:16PM +0100, Stefano Stabellini wrote:
> > > > > > On Fri, 17 Apr 2015, Ian Campbell wrote:
> > > > > > > On Fri, 2015-04-17 at 15:34 +0100, Stefano Stabellini wrote:
> > > > > > > > > > If I set dom0_mem to a small value (e.g. 512M), which makes all physical memory
> > > > > > > > > > of dom0 below 4G, everything goes fine.
> > > > > > > > > 
> > > > > > > > > So you are getting allocated memory below 4G?
> > > > > > > > > 
> > > > > > > > > You message on IRC suggested you weren't, did you hack around this?
> > > > > > > > > 
> > > > > > > > > I think we have two options, either xen_swiotlb_init allocates pages
> > > > > > > > > below 4GB (e.g. __GFP_DMA) or we do something to allow xen_swiotlb_fixup
> > > > > > > > > to actually work even on a 1:1 dom0.
> > > > > > > > 
> > > > > > > > I don't think that making xen_swiotlb_fixup work on ARM is a good idea:
> > > > > > > > it would break the 1:1.
> > > > > > > 
> > > > > > > This would actually work though, I think, because this is the swiotlb so
> > > > > > > we definitely have the opportunity to return the actual DMA address
> > > > > > > whenever we use this buffer and the device will use it in the right
> > > > > > > places for sure.
> > > > > > 
> > > > > > The code is pretty complex as is -- I would rather avoid adding more
> > > > > > complexity to it.  For example we would need to bring back a mechanism
> > > > > > to track dma address -> pseudo-physical address mappings on arm, even
> > > > > > though it would be far simpler of course.
> > > > > > 
> > > > > > Also I think it makes sense to use the swiotlb buffer for its original
> > > > > > purpose.
> > > > > > 
> > > > > > If we could introduce a mechanism to get a lower than 4G buffer in dom0,
> > > > > > but matching the 1:1, I think it would make the maintenance much easier
> > > > > > on the linux side.
> > > > > 
> > > > > +1
> > > > > 
> > > > > Actually, we have already had the mechanism on arm32 to populate at least
> > > > > one bank of memory below 4G. Thus, the only thing we have to do on the
> > > > > hypervisor side is to make arm32 and arm64 share the same process in
> > > > > allocate_memory_11(), removing the 'lowmem = is_32bit_domain(d)' related
> > > > > conditions. If this is acceptable, the only thing we need to do in Linux
> > > > > kernel is to add the __GFP_DMA flag when allocating pages for xen_io_tlb_start
> > > > > in xen_swiotlb_init.
> > > > 
> > > > Please send out the Linux patch using __GFP_DMA and I'll queue it up.
> > > 
> > > What happens with __GFP_DMA if no suitable memory is available (i.e. all
> > > of RAM is >4GB)?
> > 
> > __get_free_pages would fail and xen_swiotlb_init will try again with a
> > smaller size and print a warning.
> > 
> > If no RAM under 4G is available,
> 
> This is always going to be the case on e.g. X-Gene where all RAM is >4G
> (starts at 128GB IIRC). IOW just doing it like this is going to break on
> some arm64 platforms.

OK. Basically we need to find a way to retry without GFP_DMA if no ram
under 4G is present at all.


> >  xen_swiotlb_init will fail with an
> > error. However it is probably better to fail explicitly with an error
> > message than failing with a stack trace at some point down the line when
> > DMA is actually done.
> 
> Ian.
> 

Re: Question about DMA on 1:1 mapping dom0 of arm64

[Chen Baozi]

On Mon, Apr 20, 2015 at 10:58:52AM +0100, Stefano Stabellini wrote:
> On Sat, 18 Apr 2015, Chen Baozi wrote:
> > On Fri, Apr 17, 2015 at 05:13:16PM +0100, Stefano Stabellini wrote:
> > > On Fri, 17 Apr 2015, Ian Campbell wrote:
> > > > On Fri, 2015-04-17 at 15:34 +0100, Stefano Stabellini wrote:
> > > > > > > If I set dom0_mem to a small value (e.g. 512M), which makes all physical memory
> > > > > > > of dom0 below 4G, everything goes fine.
> > > > > > 
> > > > > > So you are getting allocated memory below 4G?
> > > > > > 
> > > > > > You message on IRC suggested you weren't, did you hack around this?
> > > > > > 
> > > > > > I think we have two options, either xen_swiotlb_init allocates pages
> > > > > > below 4GB (e.g. __GFP_DMA) or we do something to allow xen_swiotlb_fixup
> > > > > > to actually work even on a 1:1 dom0.
> > > > > 
> > > > > I don't think that making xen_swiotlb_fixup work on ARM is a good idea:
> > > > > it would break the 1:1.
> > > > 
> > > > This would actually work though, I think, because this is the swiotlb so
> > > > we definitely have the opportunity to return the actual DMA address
> > > > whenever we use this buffer and the device will use it in the right
> > > > places for sure.
> > > 
> > > The code is pretty complex as is -- I would rather avoid adding more
> > > complexity to it.  For example we would need to bring back a mechanism
> > > to track dma address -> pseudo-physical address mappings on arm, even
> > > though it would be far simpler of course.
> > > 
> > > Also I think it makes sense to use the swiotlb buffer for its original
> > > purpose.
> > > 
> > > If we could introduce a mechanism to get a lower than 4G buffer in dom0,
> > > but matching the 1:1, I think it would make the maintenance much easier
> > > on the linux side.
> > 
> > +1
> > 
> > Actually, we have already had the mechanism on arm32 to populate at least
> > one bank of memory below 4G. Thus, the only thing we have to do on the
> > hypervisor side is to make arm32 and arm64 share the same process in
> > allocate_memory_11(), removing the 'lowmem = is_32bit_domain(d)' related
> > conditions. If this is acceptable, the only thing we need to do in Linux
> > kernel is to add the __GFP_DMA flag when allocating pages for xen_io_tlb_start
> > in xen_swiotlb_init.
> 
> Please send out the Linux patch using __GFP_DMA and I'll queue it up.

I have sent it out in case that Ian would have no disagreement after the coming
discussion (hopefully) ;-)

Baozi.

Re: Question about DMA on 1:1 mapping dom0 of arm64

[Chen Baozi]

On Fri, Apr 17, 2015 at 02:21:45PM +0100, Ian Campbell wrote:
> On Fri, 2015-04-17 at 19:24 +0800, Chen Baozi wrote:
> > Hi all,
> > 
> > According to my recent experience, there might be some problems of swiotlb
> > dma map on 1:1 mapping arm64 dom0 with large memory. The issue is like below:
> > 
> > For those arm64 server with large memory, it is possible to set dom0_mem >
> > 4G (e.g. I have one set with 16G). In this case, according to my understanding,
> > there is chance that the dom0 kernel needs to map some buffers above 4G to do
> 
>                                                                  ^below?
> 
> > DMA operations (e.g. in snps,dwmac ethernet driver). However, most DMA engines
> > support only 32-bit physical address, thus aren't able to operate directly on
> > those memory.
> 
> Even on arm64 systems with RAM above 4GB? That seems.... short-sighted.
> Oh well, I suppose we have to live with it.

I understand for most ARM SoCs, the DMA engines come from third party IP companies
which is arm32/arm64 independent. Thus, 32-bit address DMA engine should be common
even on arm64 system. The preferred way is to use/enable SMMU(IOMMU). However, we
are focusing on 1:1 mapping right now...

> 
> >  IIUC, swiotlb is implemented to solve this (using bounce buffer),
> > if there is no IOMMU or IOMMU is not enabled on the system. Sadly, it seems
> > that xen_swiotlb_map_page in my dom0 kernel allocates
> > (start_dma_addr = 0x944800000) the buffers for DMA above 4G which fails
> > dma_capable() checking and was then unable to return from xen_swiotlb_map_page()
> > successfully.
> 
> The swiotlb bounce buffer have been allocated below 4GB? 

I have no idea (about the exact behavior of bounce buffer). But I don't think
it has been allocated below 4GB on my board, for in that case it won't fail
dma_capable() in the end of xen_swiotlb_map_page().

> I suspect that
> xen_swiotlb_init is buggy for ARM -- it allocates some random pages and
> then swizzles the backing pages for ones < 4G, but that won't work on an
> ARM dom0 with a 1:1 mapping, I don't think. Do you see error messages
> along those lines?
> 
> Essentially I think either xen_swiotlb_fixup is unable to work on ARM,
> or the following:
>         start_dma_addr = xen_virt_to_bus(xen_io_tlb_start);
> is returning 1:1 and not reflecting the fixup.

Yes. It seems very likely what happened in my system.

> 
> > If I set dom0_mem to a small value (e.g. 512M), which makes all physical memory
> > of dom0 below 4G, everything goes fine.
> 
> So you are getting allocated memory below 4G?

If all the banks of memory that xen populate to dom0 is below 4G, yes. However,
if some banks of memory for dom0 is above 4G, usually not.

> 
> You message on IRC suggested you weren't, did you hack around this?

Yes. I did some hacks to help understand my situation earlier. What I have done
and observed is as below:

1. At the very beginning, I used the default dom0_mem value to boot the system,
which is 128M. And I didn't realize the DMA buffer problem.

2. I started to try more dom0_mem (16G). Then the ethernet driver reported
that it cannot initiate rx buffers (DMA buffers). And I found out that
allocate_memory_11 didn't populate any banks of memory below 4G for dom0.
At that time, I guessed the failure might be introduced because there is no
memory banks below 4G was populated. (there is only a 2GB address space below 4G
for physical memory on my platform, and there is a hole for PCI memory address
space above 4G before the memory address space continue.)

3. So I did some hacks to let lowmem=true manually in allocate_memory_11, which
made xen on arm64 acts similar as it is on arm32 that populates at least one
bank of memory below 4G to dom0. (this is the point when I send you message
on IRC.) I thought that can solve the problem, but it doesn't.

4. Then I found out once xen populated any banks of memory which is above 4G,
the ethernet driver would have chances (very likely, almost every time if
dom0_mem=16G) to use buffers above 4G, regardless whether dom0 has banks of
memory below 4G.

> 
> I think we have two options, either xen_swiotlb_init allocates pages
> below 4GB (e.g. __GFP_DMA) or we do something to allow xen_swiotlb_fixup
> to actually work even on a 1:1 dom0.
> 
> Although the first option seems preferable at first glance it has the
> short coming that it requires dom0 to have some memory below 4GB, which
> might not necessarily be the case. The second option seems like it might
> be uglier but doesn't suffer from this issue.
> 
> Can you please look and find out if the IPA at 0x944800000 is actually
> backed by 1:1 RAM or if xen_swiotlb_fixup has done it's job and updated
> things such that the associated PAs are below 4GB?

I am at home now and will check it out tomorrow. But I guess it should be
the first situation you mentioned.

Cheers,

Baozi.

Re: Question about DMA on 1:1 mapping dom0 of arm64

[Ian Campbell]

On Sat, 2015-04-18 at 00:41 +0800, Chen Baozi wrote:
> On Fri, Apr 17, 2015 at 02:21:45PM +0100, Ian Campbell wrote:
> > On Fri, 2015-04-17 at 19:24 +0800, Chen Baozi wrote:
> > > Hi all,
> > > 
> > > According to my recent experience, there might be some problems of swiotlb
> > > dma map on 1:1 mapping arm64 dom0 with large memory. The issue is like below:
> > > 
> > > For those arm64 server with large memory, it is possible to set dom0_mem >
> > > 4G (e.g. I have one set with 16G). In this case, according to my understanding,
> > > there is chance that the dom0 kernel needs to map some buffers above 4G to do
> > 
> >                                                                  ^below?
> > 
> > > DMA operations (e.g. in snps,dwmac ethernet driver). However, most DMA engines
> > > support only 32-bit physical address, thus aren't able to operate directly on
> > > those memory.
> > 
> > Even on arm64 systems with RAM above 4GB? That seems.... short-sighted.
> > Oh well, I suppose we have to live with it.
> 
> I understand for most ARM SoCs, the DMA engines come from third party IP companies
> which is arm32/arm64 independent. Thus, 32-bit address DMA engine should be common
> even on arm64 system.

Yes, I suppose that will be true, but I stand by my "shortsighted"
comment ;-). (What's the point of a DMA engine which can only access 1/4
of the system's RAM and therefore requires bounce buffering before it
can be used...)

>  The preferred way is to use/enable SMMU(IOMMU). However, we
> are focusing on 1:1 mapping right now...
> 
> > 
> > >  IIUC, swiotlb is implemented to solve this (using bounce buffer),
> > > if there is no IOMMU or IOMMU is not enabled on the system. Sadly, it seems
> > > that xen_swiotlb_map_page in my dom0 kernel allocates
> > > (start_dma_addr = 0x944800000) the buffers for DMA above 4G which fails
> > > dma_capable() checking and was then unable to return from xen_swiotlb_map_page()
> > > successfully.
> > 
> > The swiotlb bounce buffer have been allocated below 4GB? 
> 
> I have no idea (about the exact behavior of bounce buffer). But I don't think
> it has been allocated below 4GB on my board, for in that case it won't fail
> dma_capable() in the end of xen_swiotlb_map_page().
> 
> > I suspect that
> > xen_swiotlb_init is buggy for ARM -- it allocates some random pages and
> > then swizzles the backing pages for ones < 4G, but that won't work on an
> > ARM dom0 with a 1:1 mapping, I don't think. Do you see error messages
> > along those lines?
> > 
> > Essentially I think either xen_swiotlb_fixup is unable to work on ARM,
> > or the following:
> >         start_dma_addr = xen_virt_to_bus(xen_io_tlb_start);
> > is returning 1:1 and not reflecting the fixup.
> 
> Yes. It seems very likely what happened in my system.

Stefano suggested that xen_swiotlb_fixup is a NOP on arm (which doesn't
surprise me, that made sense until this issue was identified), which
will be the root cause here.

> > > If I set dom0_mem to a small value (e.g. 512M), which makes all physical memory
> > > of dom0 below 4G, everything goes fine.
> > 
> > So you are getting allocated memory below 4G?
> 
> If all the banks of memory that xen populate to dom0 is below 4G, yes. However,
> if some banks of memory for dom0 is above 4G, usually not.
> 
> > 
> > You message on IRC suggested you weren't, did you hack around this?
> 
> Yes. I did some hacks to help understand my situation earlier. What I have done
> and observed is as below:
> 
> 1. At the very beginning, I used the default dom0_mem value to boot the system,
> which is 128M. And I didn't realize the DMA buffer problem.
> 
> 2. I started to try more dom0_mem (16G). Then the ethernet driver reported
> that it cannot initiate rx buffers (DMA buffers). And I found out that
> allocate_memory_11 didn't populate any banks of memory below 4G for dom0.
> At that time, I guessed the failure might be introduced because there is no
> memory banks below 4G was populated. (there is only a 2GB address space below 4G
> for physical memory on my platform, and there is a hole for PCI memory address
> space above 4G before the memory address space continue.)
> 
> 3. So I did some hacks to let lowmem=true manually in allocate_memory_11, which
> made xen on arm64 acts similar as it is on arm32 that populates at least one
> bank of memory below 4G to dom0. (this is the point when I send you message
> on IRC.) I thought that can solve the problem, but it doesn't.
> 
> 4. Then I found out once xen populated any banks of memory which is above 4G,
> the ethernet driver would have chances (very likely, almost every time if
> dom0_mem=16G) to use buffers above 4G, regardless whether dom0 has banks of
> memory below 4G.
> 
> > 
> > I think we have two options, either xen_swiotlb_init allocates pages
> > below 4GB (e.g. __GFP_DMA) or we do something to allow xen_swiotlb_fixup
> > to actually work even on a 1:1 dom0.
> > 
> > Although the first option seems preferable at first glance it has the
> > short coming that it requires dom0 to have some memory below 4GB, which
> > might not necessarily be the case. The second option seems like it might
> > be uglier but doesn't suffer from this issue.
> > 
> > Can you please look and find out if the IPA at 0x944800000 is actually
> > backed by 1:1 RAM or if xen_swiotlb_fixup has done it's job and updated
> > things such that the associated PAs are below 4GB?
> 
> I am at home now and will check it out tomorrow. But I guess it should be
> the first situation you mentioned.

I think given the information Stefano has provided there's likely not
much point in this experiment, since we now know that the fixup function
is a nop on ARM.

Ian.

Re: Question about DMA on 1:1 mapping dom0 of arm64

[Stefano Stabellini]

On Fri, 17 Apr 2015, Chen Baozi wrote:
> Hi all,
> 
> According to my recent experience, there might be some problems of swiotlb
> dma map on 1:1 mapping arm64 dom0 with large memory. The issue is like below:
> 
> For those arm64 server with large memory, it is possible to set dom0_mem >
> 4G (e.g. I have one set with 16G). In this case, according to my understanding,
> there is chance that the dom0 kernel needs to map some buffers above 4G to do
> DMA operations (e.g. in snps,dwmac ethernet driver). However, most DMA engines
> support only 32-bit physical address, thus aren't able to operate directly on
> those memory. IIUC, swiotlb is implemented to solve this (using bounce buffer),
> if there is no IOMMU or IOMMU is not enabled on the system. Sadly, it seems
> that xen_swiotlb_map_page in my dom0 kernel allocates
> (start_dma_addr = 0x944800000) the buffers for DMA above 4G which fails
> dma_capable() checking and was then unable to return from xen_swiotlb_map_page()
> successfully.
>
> If I set dom0_mem to a small value (e.g. 512M), which makes all physical memory
> of dom0 below 4G, everything goes fine.
> 
> I am not familiar with swiotlb-xen, so there would be misunderstanding about
> the current situation. Fix me if I did/understood anything wrong.
> 
> Any ideas?

I think that the problem is that xen_swiotlb_init doesn't necessarely allocate
memory under 4G on arm/arm64.

xen_swiotlb_init calls __get_free_pages to allocate memory, so the pages
could easily be above 4G.  Subsequently xen_swiotlb_fixup is called on
the allocated memory range, calling xen_create_contiguous_region and
passing an address_bits mask. However xen_create_contiguous_region
doesn't actually do anything at all on ARM.

I think that given that dom0 is mapped 1:1 on ARM, the easiest and best
fix would be to simply allocate memory under 4G to begin with. Something
like (maybe with an ifdef ARM around it):

diff --git a/drivers/xen/swiotlb-xen.c b/drivers/xen/swiotlb-xen.c
index 810ad41..22ac33a 100644
--- a/drivers/xen/swiotlb-xen.c
+++ b/drivers/xen/swiotlb-xen.c
@@ -235,7 +235,7 @@ retry:
 #define SLABS_PER_PAGE (1 << (PAGE_SHIFT - IO_TLB_SHIFT))
 #define IO_TLB_MIN_SLABS ((1<<20) >> IO_TLB_SHIFT)
 		while ((SLABS_PER_PAGE << order) > IO_TLB_MIN_SLABS) {
-			xen_io_tlb_start = (void *)__get_free_pages(__GFP_NOWARN, order);
+			xen_io_tlb_start = (void *)__get_free_pages(__GFP_NOWARN|__GFP_DMA32, order);
 			if (xen_io_tlb_start)
 				break;
 			order--;

Re: Question about DMA on 1:1 mapping dom0 of arm64

[Ian Campbell]

On Fri, 2015-04-17 at 15:32 +0100, Stefano Stabellini wrote:
> I think that given that dom0 is mapped 1:1 on ARM, the easiest and best
> fix would be to simply allocate memory under 4G to begin with.

Not necessarily best, see my reply (hint: dom0 might not have RAM under
4GB even if the host does).

Re: Question about DMA on 1:1 mapping dom0 of arm64

[Chen Baozi]

Hi Stefano,

On Fri, Apr 17, 2015 at 03:32:20PM +0100, Stefano Stabellini wrote:
> On Fri, 17 Apr 2015, Chen Baozi wrote:
> > Hi all,
> > 
> > According to my recent experience, there might be some problems of swiotlb
> > dma map on 1:1 mapping arm64 dom0 with large memory. The issue is like below:
> > 
> > For those arm64 server with large memory, it is possible to set dom0_mem >
> > 4G (e.g. I have one set with 16G). In this case, according to my understanding,
> > there is chance that the dom0 kernel needs to map some buffers above 4G to do
> > DMA operations (e.g. in snps,dwmac ethernet driver). However, most DMA engines
> > support only 32-bit physical address, thus aren't able to operate directly on
> > those memory. IIUC, swiotlb is implemented to solve this (using bounce buffer),
> > if there is no IOMMU or IOMMU is not enabled on the system. Sadly, it seems
> > that xen_swiotlb_map_page in my dom0 kernel allocates
> > (start_dma_addr = 0x944800000) the buffers for DMA above 4G which fails
> > dma_capable() checking and was then unable to return from xen_swiotlb_map_page()
> > successfully.
> >
> > If I set dom0_mem to a small value (e.g. 512M), which makes all physical memory
> > of dom0 below 4G, everything goes fine.
> > 
> > I am not familiar with swiotlb-xen, so there would be misunderstanding about
> > the current situation. Fix me if I did/understood anything wrong.
> > 
> > Any ideas?
> 
> I think that the problem is that xen_swiotlb_init doesn't necessarely allocate
> memory under 4G on arm/arm64.
> 
> xen_swiotlb_init calls __get_free_pages to allocate memory, so the pages
> could easily be above 4G.  Subsequently xen_swiotlb_fixup is called on
> the allocated memory range, calling xen_create_contiguous_region and
> passing an address_bits mask. However xen_create_contiguous_region
> doesn't actually do anything at all on ARM.
> 
> I think that given that dom0 is mapped 1:1 on ARM, the easiest and best
> fix would be to simply allocate memory under 4G to begin with. Something
> like (maybe with an ifdef ARM around it):
> 
> diff --git a/drivers/xen/swiotlb-xen.c b/drivers/xen/swiotlb-xen.c
> index 810ad41..22ac33a 100644
> --- a/drivers/xen/swiotlb-xen.c
> +++ b/drivers/xen/swiotlb-xen.c
> @@ -235,7 +235,7 @@ retry:
>  #define SLABS_PER_PAGE (1 << (PAGE_SHIFT - IO_TLB_SHIFT))
>  #define IO_TLB_MIN_SLABS ((1<<20) >> IO_TLB_SHIFT)
>  		while ((SLABS_PER_PAGE << order) > IO_TLB_MIN_SLABS) {
> -			xen_io_tlb_start = (void *)__get_free_pages(__GFP_NOWARN, order);
> +			xen_io_tlb_start = (void *)__get_free_pages(__GFP_NOWARN|__GFP_DMA32, order);
>  			if (xen_io_tlb_start)
>  				break;
>  			order--;

I have no idea if __GFP_DMA32 on arm64 has something wrong. But It looks like
that it doesn't help...

Here is the memory info about what xen has populated to dom0 (I did some hacks
to allocate_memory_11 to make it map some low memory banks to dom0):

(XEN) Allocating 1:1 mappings totalling 16384MB for dom0:
(XEN) BANK[0] 0x00000088000000-0x00000098000000 (256MB)
(XEN) BANK[1] 0x000000a0000000-0x000000f8000000 (1408MB)
(XEN) BANK[2] 0x00000400000000-0x00000600000000 (8192MB)
(XEN) BANK[3] 0x00000680000000-0x00000700000000 (2048MB)
(XEN) BANK[4] 0x00000800000000-0x00000900000000 (4096MB)
(XEN) BANK[5] 0x00000940000000-0x00000958000000 (384MB)

And Here is the printk info I got when trying to map a dma page:

enter xen_swiotlb_map_page.
phys = 0x9444e4042, dev_addr = 0x9444e4042, size = 0x600
start_dma_addr = 0x944800000
virt_to_phys(xen_io_tlb_start) = 0x944800000
Oh Well, have to allocate and map a bounce buffer.
map = 0x944800000
dev_addr = 0x944800000
*dev->dma_mask = 0xffffffff
!dma_capable(0xffffffc8bd384810, 0x944800000, 0x600)

And the patch I used for dom0 hacking:

diff --git a/drivers/xen/swiotlb-xen.c b/drivers/xen/swiotlb-xen.c
index 810ad41..96465cf 100644
--- a/drivers/xen/swiotlb-xen.c
+++ b/drivers/xen/swiotlb-xen.c
@@ -235,7 +235,7 @@ retry:
 #define SLABS_PER_PAGE (1 << (PAGE_SHIFT - IO_TLB_SHIFT))
 #define IO_TLB_MIN_SLABS ((1<<20) >> IO_TLB_SHIFT)
 		while ((SLABS_PER_PAGE << order) > IO_TLB_MIN_SLABS) {
-			xen_io_tlb_start = (void *)__get_free_pages(__GFP_NOWARN, order);
+			xen_io_tlb_start = (void *)__get_free_pages(__GFP_NOWARN|__GFP_DMA32, order);
 			if (xen_io_tlb_start)
 				break;
 			order--;
@@ -391,6 +391,13 @@ dma_addr_t xen_swiotlb_map_page(struct device *dev, struct page *page,
 	dma_addr_t dev_addr = xen_phys_to_bus(phys);
 
 	BUG_ON(dir == DMA_NONE);
+	printk("enter xen_swiotlb_map_page.\n");
+	printk("phys = 0x%lx, dev_addr = 0x%lx, size = 0x%lx\n",
+		phys, dev_addr, size);
+	printk("start_dma_addr = 0x%lx\n", start_dma_addr);
+	printk("virt_to_phys(xen_io_tlb_start) = 0x%lx\n",
+		virt_to_phys(xen_io_tlb_start));
+
 	/*
 	 * If the address happens to be in the device's DMA window,
 	 * we can safely return the device addr and not worry about bounce
@@ -403,27 +410,35 @@ dma_addr_t xen_swiotlb_map_page(struct device *dev, struct page *page,
 		/* we are not interested in the dma_addr returned by
 		 * xen_dma_map_page, only in the potential cache flushes executed
 		 * by the function. */
+		printk("The address happens to be in the device's DMA window\n");
 		xen_dma_map_page(dev, page, dev_addr, offset, size, dir, attrs);
 		return dev_addr;
 	}
 
+
 	/*
 	 * Oh well, have to allocate and map a bounce buffer.
 	 */
+	printk("Oh Well, have to allocate and map a bounce buffer.\n");
 	trace_swiotlb_bounced(dev, dev_addr, size, swiotlb_force);
 
 	map = swiotlb_tbl_map_single(dev, start_dma_addr, phys, size, dir);
 	if (map == SWIOTLB_MAP_ERROR)
 		return DMA_ERROR_CODE;
+	printk("map = 0x%lx\n", map);
 
 	xen_dma_map_page(dev, pfn_to_page(map >> PAGE_SHIFT),
 					dev_addr, map & ~PAGE_MASK, size, dir, attrs);
 	dev_addr = xen_phys_to_bus(map);
+	printk("dev_addr = 0x%lx\n", dev_addr);
+	printk("*dev->dma_mask = 0x%lx\n", *dev->dma_mask);
 
 	/*
 	 * Ensure that the address returned is DMA'ble
 	 */
 	if (!dma_capable(dev, dev_addr, size)) {
+		printk("!dma_capable(0x%p, 0x%lx, 0x%lx\n)",
+			dev, dev_addr, size);
 		swiotlb_tbl_unmap_single(dev, map, size, dir);
 		dev_addr = 0;
 	}

Re: Question about DMA on 1:1 mapping dom0 of arm64

[Chen Baozi]

On Fri, Apr 17, 2015 at 03:32:20PM +0100, Stefano Stabellini wrote:
> On Fri, 17 Apr 2015, Chen Baozi wrote:
> > Hi all,
> > 
> > According to my recent experience, there might be some problems of swiotlb
> > dma map on 1:1 mapping arm64 dom0 with large memory. The issue is like below:
> > 
> > For those arm64 server with large memory, it is possible to set dom0_mem >
> > 4G (e.g. I have one set with 16G). In this case, according to my understanding,
> > there is chance that the dom0 kernel needs to map some buffers above 4G to do
> > DMA operations (e.g. in snps,dwmac ethernet driver). However, most DMA engines
> > support only 32-bit physical address, thus aren't able to operate directly on
> > those memory. IIUC, swiotlb is implemented to solve this (using bounce buffer),
> > if there is no IOMMU or IOMMU is not enabled on the system. Sadly, it seems
> > that xen_swiotlb_map_page in my dom0 kernel allocates
> > (start_dma_addr = 0x944800000) the buffers for DMA above 4G which fails
> > dma_capable() checking and was then unable to return from xen_swiotlb_map_page()
> > successfully.
> >
> > If I set dom0_mem to a small value (e.g. 512M), which makes all physical memory
> > of dom0 below 4G, everything goes fine.
> > 
> > I am not familiar with swiotlb-xen, so there would be misunderstanding about
> > the current situation. Fix me if I did/understood anything wrong.
> > 
> > Any ideas?
> 
> I think that the problem is that xen_swiotlb_init doesn't necessarely allocate
> memory under 4G on arm/arm64.
> 
> xen_swiotlb_init calls __get_free_pages to allocate memory, so the pages
> could easily be above 4G.  Subsequently xen_swiotlb_fixup is called on
> the allocated memory range, calling xen_create_contiguous_region and
> passing an address_bits mask. However xen_create_contiguous_region
> doesn't actually do anything at all on ARM.
> 
> I think that given that dom0 is mapped 1:1 on ARM, the easiest and best
> fix would be to simply allocate memory under 4G to begin with. Something
> like (maybe with an ifdef ARM around it):
> 
> diff --git a/drivers/xen/swiotlb-xen.c b/drivers/xen/swiotlb-xen.c
> index 810ad41..22ac33a 100644
> --- a/drivers/xen/swiotlb-xen.c
> +++ b/drivers/xen/swiotlb-xen.c
> @@ -235,7 +235,7 @@ retry:
>  #define SLABS_PER_PAGE (1 << (PAGE_SHIFT - IO_TLB_SHIFT))
>  #define IO_TLB_MIN_SLABS ((1<<20) >> IO_TLB_SHIFT)
>  		while ((SLABS_PER_PAGE << order) > IO_TLB_MIN_SLABS) {
> -			xen_io_tlb_start = (void *)__get_free_pages(__GFP_NOWARN, order);
> +			xen_io_tlb_start = (void *)__get_free_pages(__GFP_NOWARN|__GFP_DMA32, order);
                                                                                 ^^ __GFP_DMA works on arm64

Cheers,

Baozi.