Re: collision between ZONE_MOVABLE and memblock allocations

[Ross Zwisler <zwisler@google.com>]

On Thu, Jul 20, 2023 at 02:13:25PM +0200, Michal Hocko wrote:
> On Wed 19-07-23 16:48:21, Ross Zwisler wrote:
> > On Wed, Jul 19, 2023 at 08:14:48AM +0200, Michal Hocko wrote:
> > > On Tue 18-07-23 16:01:06, Ross Zwisler wrote:
> > > [...]
> > > > I do think that we need to fix this collision between ZONE_MOVABLE and memmap
> > > > allocations, because this issue essentially makes the movablecore= kernel
> > > > command line parameter useless in many cases, as the ZONE_MOVABLE region it
> > > > creates will often actually be unmovable.
> > > 
> > > movablecore is kinda hack and I would be more inclined to get rid of it
> > > rather than build more into it. Could you be more specific about your
> > > use case?
> > 
> > The problem that I'm trying to solve is that I'd like to be able to get kernel
> > core dumps off machines (chromebooks) so that we can debug crashes.  Because
> > the memory used by the crash kernel ("crashkernel=" kernel command line
> > option) is consumed the entire time the machine is booted, there is a strong
> > motivation to keep the crash kernel as small and as simple as possible.  To
> > this end I'm trying to get away without SSD drivers, not having to worry about
> > encryption on the SSDs, etc.
> > 
> > So, the rough plan right now is:
> > 
> > 1) During boot set aside some memory that won't contain kernel allocations.
> > I'm trying to do this now with ZONE_MOVABLE, but I'm open to better ways.
> > 
> > We set aside memory for a crash kernel & arm it so that the ZONE_MOVABLE
> > region (or whatever non-kernel region) will be set aside as PMEM in the crash
> > kernel.  This is done with the memmap=nn[KMG]!ss[KMG] kernel command line
> > parameter passed to the crash kernel.
> > 
> > So, in my sample 4G VM system, I see:
> > 
> >   # lsmem --split ZONES --output-all
> >   RANGE                                  SIZE  STATE REMOVABLE BLOCK NODE   ZONES
> >   0x0000000000000000-0x0000000007ffffff  128M online       yes     0    0    None
> >   0x0000000008000000-0x00000000bfffffff  2.9G online       yes  1-23    0   DMA32
> >   0x0000000100000000-0x000000012fffffff  768M online       yes 32-37    0  Normal
> >   0x0000000130000000-0x000000013fffffff  256M online       yes 38-39    0 Movable
> >   
> >   Memory block size:       128M
> >   Total online memory:       4G
> >   Total offline memory:      0B
> > 
> > so I'll pass "memmap=256M!0x130000000" to the crash kernel.
> > 
> > 2) When we hit a kernel crash, we know (hope?) that the PMEM region we've set
> > aside only contains user data, which we don't want to store anyway.  We make a
> > filesystem in there, and create a kernel crash dump using 'makedumpfile':
> > 
> >   mkfs.ext4 /dev/pmem0
> >   mount /dev/pmem0 /mnt
> >   makedumpfile -c -d 31 /proc/vmcore /mnt/kdump
> > 
> > We then set up the next full kernel boot to also have this same PMEM region,
> > using the same memmap kernel parameter.  We reboot back into a full kernel.
> 
> Btw. How do you ensure that the address range doesn't get reinitialized
> by POST? Do you rely on kexec boot here?

I've been working under the assumption that I do need to do a full reboot (not
just another kexec boot) so that the devices in the system (NICs, disks, etc)
are all reinitialized and don't carry over bad state from the crash.

I do know about the 'reset_devices' kernel command line parameter, but wasn't
sure that would be enough.  From looking around it seems like this is very
driver + device dependent, so maybe I just need to test more.

In any case, you're right, if we do a full reboot and go through POST, it's
system dependent on whether BIOS/UEFI/Coreboot/etc will zero memory, and if it
does this feature won't work unless we kexec to the 3rd kernel.

I've also heard concerns around whether a full reboot will cause the memory
controller to reinitialize and potentially cause memory bit flips or similar,
though I haven't yet seen this myself.  Has anyone seen such bit flips /
memory corruption due to system reboot, or is this a non-issue in your
experience?

Lots to figure out, thanks for the help. :)