Re: Kexec, DMA, and SMP

[Corey Minyard <cminyard@mvista.com>]

Eric W. Biederman wrote:

>Corey Minyard <cminyard@mvista.com> writes:
>  
>
>>Eric,
>>
>>I saw that you are working on kexec.  I'm using and have hacked on a similar
>>piece of software named bootimg (and I'll be glad when yours is done and ready
>>and we can jettison bootimg).  From the looks of the code, it looks like you
>>have seen bootimg, too.  I looked through your patch, and I noticed a few
>>things.  Hopefully it's the newest version of the patch.
>>
>>First was that you don't turn of DMA bus masters.  There seemed to be some
>>discussion of this on lkml, but I didn't see anything in the patch for it.  We
>>are actually having problems with bootimg and DMA bus master devices, so the
>>problem is real.  And turning of DMA bus mastering for everything on the PCI bus
>>didn't seem to work, Ken Sumrall tried it, and at least the device in question
>>(a bcm5700) seemed to ignore the bit.  We are looking at adding an ioctl or a
>>notifier list that will allow devices to register non-blocking calls to shut off
>>DMA.  Is anything like that under consideration, or are you thinking of using
>>the reboot notifier for this, or what?
>>    
>>
>
>The reboot notifier + device->shutdown(), are called.  As you have noted
>the problem is not as easy as clearing the bus master bit, so I leave it
>up to the device driver.  The device driver is responsible for placing
>the device into a quiescent state.
>
>Generally that code is present in the driver somewhere already, as it
>is needed for the rmmod case.   
>
>In addition going through the normal shutdown path, downing interfaces
>etc, usually helps as well.
>
>So for when kexec is not used in a panic case this is easy.
>
The panic case is actually the most interesting for us.  We are using 
bootimg with the MCL coredump to take a kernel core to memory and pick 
it up on the next boot.  You cannot call most shutdown() functions from 
a panic, since they will block.

>>The patch doesn't make sure it is running on processor zero for SMP machines.
>>You must do this on x86 machines, the kernel assumes it comes up on processor
>>zero.  I assume this is true for other machines, too.
>>    
>>
>
>I have a secondary patch.  kexec-hwfixes, that does this.  I am I need to review
>it a little closer and make certain the code is clean enough to go into
>the general purpose kernel.  But I do have the code.
>
I have code that does this for bootimg, too, if you are interested, and 
it has received extensive testing.

>>Hopefully I'm not looking at an old version of the patch, but these are
>>important things you need to handle.
>>    
>>
>
>Yep.  I am a little scatter brained on the maintenance side but I am handling
>them all.
>
>If you are after the kexec on panic case that is much, more
>interesting because it is quite possible we cannot afford to call some
>of those functions.  But I am quite willing to discuss and work with
>people on what is really going on.  
>
>For any more conversation though can we please cc linux-kernel?
>
>I like to keep things public so I don't have to answer the same
>question too many times.
>
No problem.  As you have requested, lkml is copied.

Thanks,

-Corey