Re: swiotlb detection should be memory hotplug aware ?

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

On Thu, Jul 22, 2010 at 11:34:40AM -0700, Alok Kataria wrote:
> Hi,
> 
> On Wed, 2010-07-21 at 17:03 -0700, FUJITA Tomonori wrote:
> > On Thu, 22 Jul 2010 08:44:42 +0900
> > FUJITA Tomonori <fujita.tomonori@lab.ntt.co.jp> wrote:
> > 
> > > On Wed, 21 Jul 2010 10:13:34 -0700
> > > Alok Kataria <akataria@vmware.com> wrote:
> > > 
> > > > > Basically, you want to add hot-plug memory and enable swiotlb, right?
> > > > 
> > > > Not really, I am planning to do something like this, 
> > > > 
> > > > @@ -52,7 +52,7 @@ int __init pci_swiotlb_detect(void)
> > > >  
> > > >  	/* don't initialize swiotlb if iommu=off (no_iommu=1) */
> > > >  #ifdef CONFIG_X86_64
> > > > -	if (!no_iommu && max_pfn > MAX_DMA32_PFN)
> > > > +	if (!no_iommu && (max_pfn > MAX_DMA32_PFN || hotplug_possible()))
> > > >  		swiotlb = 1;
> > > 
> > > Always enable swiotlb with memory hotplug enabled?
> 
> yep though only on systems which have hotpluggable memory support.

What machines are there that have hotplug support and no hardware IOMMU?
I know of the IBM ones - but they use the Calgary IOMMU.
> 
> >  Wasting 64MB on a
> > > x86_64 system with 128MB doesn't look to be a good idea. I don't think
> > > that there is an easy solution for this issue though.
> 
> Good now that you agree that, that's the only feasible solution, do you
> have any suggestions for any interfaces that are available from SRAT for
> implementing hotplug_possible ?

I thought SRAT has NUMA affinity information - so for example my AMD
desktop box has that, but it does not support hotplug capability.

I think first your 'hotplug_possible' code needs to be more specific -
not just check if SRAT exists, but also if there are swaths of memory
that are non-populated. It would also help if there was some indication
of whether the box truly does a hardware hotplug - is there a way to do
this?

> 
> > 
> > btw, you need more work to enable switch on the fly.
> > 
> > You need to change the dma_ops pointer (see get_dma_ops()). It means
> > that you need to track outstanding dma operations per device, locking,
> > etc.
> 
> Yeah though if we are doing this during swiotlb_init time i.e. at bootup
> as suggested in the pseudo patch, we don't need to worry about all this,
> right ? 

Right..
> 
> Thanks,
> Alok