Re: [PATCH 0/5] arm64/mm: remap crash kernel with base pages even if rodata_full disabled

Re: [PATCH 0/5] arm64/mm: remap crash kernel with base pages even if rodata_full disabled

[Baoquan He]

Add kexec list in CC

On 08/19/22 at 07:11am, Mike Rapoport wrote:
> From: Mike Rapoport <mike.rapoport@gmail.com>
> 
> Hi,
> 
> There were several rounds of discussion how to remap with base pages only
> the crash kernel area, the latest one here:
> 
> https://lore.kernel.org/all/1656777473-73887-1-git-send-email-guanghuifeng@linux.alibaba.com
> 
> and this is my attempt to allow having both large pages in the linear map
> and protection for the crash kernel memory.
> 
> For server systems it is important to protect crash kernel memory for
> post-mortem analysis, and for that protection to work the crash kernel
> memory should be mapped with base pages in the linear map. 
> 
> On the systems with ZONE_DMA/DMA32 enabled, crash kernel reservation
> happens after the linear map is created and the current code forces using
> base pages for the entire linear map, which results in performance
> degradation.
> 
> These patches enable remapping of the crash kernel area with base pages
> while keeping large pages in the rest of the linear map.
> 
> The idea is to align crash kernel reservation to PUD boundaries, remap that
> PUD and then free the extra memory.

Hi Mike,

Thanks for the effort to work on this issue. While I have to say this
isnt's good because it can only be made relying on a prerequisite that
there's big enough memory. If on a system, say 2G memory, it's not easy
to succeed on getting one 1G memory. While we only require far smaller
region than 1G, e.g about 200M which should be easy to get. So the way
taken in this patchset is too quirky and will cause regression on
systemswith small memory. This kind of sytems with small memory exists
widely on virt guest instance.

The crashkernel reservation happens after linear map because the
reservation needs to know the dma zone boundary, arm64_dma_phys_limit.
If we can deduce that before bootmem_init(), the reservation can be
done before linear map. I will make an attempt on that. If still can't
be accepted, we would like to take off the crashkernel region protection
on arm64 for now.

Thanks
Baoquan


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH 0/5] arm64/mm: remap crash kernel with base pages even if rodata_full disabled

[Mike Rapoport]

Hi Baoquan,

On Thu, Aug 25, 2022 at 03:35:04PM +0800, Baoquan He wrote:
> Add kexec list in CC
> 
> On 08/19/22 at 07:11am, Mike Rapoport wrote:
> > From: Mike Rapoport <mike.rapoport@gmail.com>
> > 
> > Hi,
> > 
> > There were several rounds of discussion how to remap with base pages only
> > the crash kernel area, the latest one here:
> > 
> > https://lore.kernel.org/all/1656777473-73887-1-git-send-email-guanghuifeng@linux.alibaba.com
> > 
> > and this is my attempt to allow having both large pages in the linear map
> > and protection for the crash kernel memory.
> > 
> > For server systems it is important to protect crash kernel memory for
> > post-mortem analysis, and for that protection to work the crash kernel
> > memory should be mapped with base pages in the linear map. 
> > 
> > On the systems with ZONE_DMA/DMA32 enabled, crash kernel reservation
> > happens after the linear map is created and the current code forces using
> > base pages for the entire linear map, which results in performance
> > degradation.
> > 
> > These patches enable remapping of the crash kernel area with base pages
> > while keeping large pages in the rest of the linear map.
> > 
> > The idea is to align crash kernel reservation to PUD boundaries, remap that
> > PUD and then free the extra memory.
> 
> Hi Mike,
> 
> Thanks for the effort to work on this issue. While I have to say this
> isnt's good because it can only be made relying on a prerequisite that
> there's big enough memory. If on a system, say 2G memory, it's not easy
> to succeed on getting one 1G memory. While we only require far smaller
> region than 1G, e.g about 200M which should be easy to get. So the way
> taken in this patchset is too quirky and will cause regression on
> systemswith small memory. This kind of sytems with small memory exists
> widely on virt guest instance.

I don't agree there is a regression. If the PUD-aligned allocation fails,
there is a fallback to the allocation of the exact size requested for crash
kernel. This allocation just won't get protected.

Also please note, that the changes are only for the case when user didn't
force base-size pages in the linear map, so anything that works now will
work the same way with this set applied.
 
> The crashkernel reservation happens after linear map because the
> reservation needs to know the dma zone boundary, arm64_dma_phys_limit.
> If we can deduce that before bootmem_init(), the reservation can be
> done before linear map. I will make an attempt on that. If still can't
> be accepted, we would like to take off the crashkernel region protection
> on arm64 for now.

I doubt it would be easy because arm64_dma_phys_limit is determined after
parsing of the device tree and there might be memory allocations of
possibly unmapped memory during the parsing.
 
> Thanks
> Baoquan
> 

-- 
Sincerely yours,
Mike.

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH 0/5] arm64/mm: remap crash kernel with base pages even if rodata_full disabled

[Baoquan He]

On 08/25/22 at 10:48am, Mike Rapoport wrote:
...... 
> > > There were several rounds of discussion how to remap with base pages only
> > > the crash kernel area, the latest one here:
> > > 
> > > https://lore.kernel.org/all/1656777473-73887-1-git-send-email-guanghuifeng@linux.alibaba.com
> > > 
> > > and this is my attempt to allow having both large pages in the linear map
> > > and protection for the crash kernel memory.
> > > 
> > > For server systems it is important to protect crash kernel memory for
> > > post-mortem analysis, and for that protection to work the crash kernel
> > > memory should be mapped with base pages in the linear map. 
> > > 
> > > On the systems with ZONE_DMA/DMA32 enabled, crash kernel reservation
> > > happens after the linear map is created and the current code forces using
> > > base pages for the entire linear map, which results in performance
> > > degradation.
> > > 
> > > These patches enable remapping of the crash kernel area with base pages
> > > while keeping large pages in the rest of the linear map.
> > > 
> > > The idea is to align crash kernel reservation to PUD boundaries, remap that
> > > PUD and then free the extra memory.
> > 
> > Hi Mike,
> > 
> > Thanks for the effort to work on this issue. While I have to say this
> > isnt's good because it can only be made relying on a prerequisite that
> > there's big enough memory. If on a system, say 2G memory, it's not easy
> > to succeed on getting one 1G memory. While we only require far smaller
> > region than 1G, e.g about 200M which should be easy to get. So the way
> > taken in this patchset is too quirky and will cause regression on
> > systemswith small memory. This kind of sytems with small memory exists
> > widely on virt guest instance.
> 
> I don't agree there is a regression. If the PUD-aligned allocation fails,
> there is a fallback to the allocation of the exact size requested for crash
> kernel. This allocation just won't get protected.

Sorry, I misunderstood it. I just went through the log and didn't
look into codes.

But honestly, if we accept the fallback which doesn't do the protection,
we should be able to take off the protection completely, right?
Otherwise, the reservation code is a little complicated.

> 
> Also please note, that the changes are only for the case when user didn't
> force base-size pages in the linear map, so anything that works now will
> work the same way with this set applied.
>  
> > The crashkernel reservation happens after linear map because the
> > reservation needs to know the dma zone boundary, arm64_dma_phys_limit.
> > If we can deduce that before bootmem_init(), the reservation can be
> > done before linear map. I will make an attempt on that. If still can't
> > be accepted, we would like to take off the crashkernel region protection
> > on arm64 for now.
> 
> I doubt it would be easy because arm64_dma_phys_limit is determined after
> parsing of the device tree and there might be memory allocations of
> possibly unmapped memory during the parsing.

I have sent out the patches with an attempt, it's pretty straightforward
and simple. Because arm64 only has one exception, namely Raspberry Pi 4,
on which some peripherals can only address 30bit range. That is a corner
case, to be honest. And kdump is a necessary feature on server, but may
not be so expected on Raspberry Pi 4, a system for computer education
and hobbyists. And kdump only cares whether the dump target devices can
address 32bit range, namely storage device or network card on server.
If finally confirmed that storage devices can only address 30bit range
on Raspberry Pi 4, people still can have crashkernel=xM@yM method to
reserve crashkernel regions.

Thanks
Baoquan


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH 0/5] arm64/mm: remap crash kernel with base pages even if rodata_full disabled

[Mike Rapoport]

On Sun, Aug 28, 2022 at 04:37:29PM +0800, Baoquan He wrote:
> On 08/25/22 at 10:48am, Mike Rapoport wrote:
> ...... 
> > > > There were several rounds of discussion how to remap with base pages only
> > > > the crash kernel area, the latest one here:
> > > > 
> > > > https://lore.kernel.org/all/1656777473-73887-1-git-send-email-guanghuifeng@linux.alibaba.com
> > > > 
> > > > and this is my attempt to allow having both large pages in the linear map
> > > > and protection for the crash kernel memory.
> > > > 
> > > > For server systems it is important to protect crash kernel memory for
> > > > post-mortem analysis, and for that protection to work the crash kernel
> > > > memory should be mapped with base pages in the linear map. 
> > > > 
> > > > On the systems with ZONE_DMA/DMA32 enabled, crash kernel reservation
> > > > happens after the linear map is created and the current code forces using
> > > > base pages for the entire linear map, which results in performance
> > > > degradation.
> > > > 
> > > > These patches enable remapping of the crash kernel area with base pages
> > > > while keeping large pages in the rest of the linear map.
> > > > 
> > > > The idea is to align crash kernel reservation to PUD boundaries, remap that
> > > > PUD and then free the extra memory.
> > > 
> > > Hi Mike,
> > > 
> > > Thanks for the effort to work on this issue. While I have to say this
> > > isnt's good because it can only be made relying on a prerequisite that
> > > there's big enough memory. If on a system, say 2G memory, it's not easy
> > > to succeed on getting one 1G memory. While we only require far smaller
> > > region than 1G, e.g about 200M which should be easy to get. So the way
> > > taken in this patchset is too quirky and will cause regression on
> > > systemswith small memory. This kind of sytems with small memory exists
> > > widely on virt guest instance.
> > 
> > I don't agree there is a regression. If the PUD-aligned allocation fails,
> > there is a fallback to the allocation of the exact size requested for crash
> > kernel. This allocation just won't get protected.
> 
> Sorry, I misunderstood it. I just went through the log and didn't
> look into codes.
> 
> But honestly, if we accept the fallback which doesn't do the protection,
> we should be able to take off the protection completely, right?
> Otherwise, the reservation code is a little complicated.

We don't do protection of the crash kernel for most architectures
supporting kexec ;-)

My goal was to allow large systems with ZONE_DMA/DMA32 have block mappings
in the linear map and crash kernel protection without breaking backward
compatibility for the existing systems.

> > Also please note, that the changes are only for the case when user didn't
> > force base-size pages in the linear map, so anything that works now will
> > work the same way with this set applied.
> >  
> > > The crashkernel reservation happens after linear map because the
> > > reservation needs to know the dma zone boundary, arm64_dma_phys_limit.
> > > If we can deduce that before bootmem_init(), the reservation can be
> > > done before linear map. I will make an attempt on that. If still can't
> > > be accepted, we would like to take off the crashkernel region protection
> > > on arm64 for now.
> > 
> > I doubt it would be easy because arm64_dma_phys_limit is determined after
> > parsing of the device tree and there might be memory allocations of
> > possibly unmapped memory during the parsing.
> 
> I have sent out the patches with an attempt, it's pretty straightforward
> and simple. Because arm64 only has one exception, namely Raspberry Pi 4,
> on which some peripherals can only address 30bit range. That is a corner
> case, to be honest. And kdump is a necessary feature on server, but may
> not be so expected on Raspberry Pi 4, a system for computer education
> and hobbyists. And kdump only cares whether the dump target devices can
> address 32bit range, namely storage device or network card on server.
> If finally confirmed that storage devices can only address 30bit range
> on Raspberry Pi 4, people still can have crashkernel=xM@yM method to
> reserve crashkernel regions.

I hope you are right and Raspberry Pi 4 is the only system that limits
DMA'able range to 30 bits. But with diversity of arm64 chips and boards I
won't be surprised that there are other variants with a similar problem.
 
> Thanks
> Baoquan
> 

-- 
Sincerely yours,
Mike.

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH 0/5] arm64/mm: remap crash kernel with base pages even if rodata_full disabled

[Baoquan He]

On 08/29/22 at 05:31pm, Mike Rapoport wrote:
> On Sun, Aug 28, 2022 at 04:37:29PM +0800, Baoquan He wrote:
> > On 08/25/22 at 10:48am, Mike Rapoport wrote:
> > ...... 
> > > > > There were several rounds of discussion how to remap with base pages only
> > > > > the crash kernel area, the latest one here:
> > > > > 
> > > > > https://lore.kernel.org/all/1656777473-73887-1-git-send-email-guanghuifeng@linux.alibaba.com
> > > > > 
> > > > > and this is my attempt to allow having both large pages in the linear map
> > > > > and protection for the crash kernel memory.
> > > > > 
> > > > > For server systems it is important to protect crash kernel memory for
> > > > > post-mortem analysis, and for that protection to work the crash kernel
> > > > > memory should be mapped with base pages in the linear map. 
> > > > > 
> > > > > On the systems with ZONE_DMA/DMA32 enabled, crash kernel reservation
> > > > > happens after the linear map is created and the current code forces using
> > > > > base pages for the entire linear map, which results in performance
> > > > > degradation.
> > > > > 
> > > > > These patches enable remapping of the crash kernel area with base pages
> > > > > while keeping large pages in the rest of the linear map.
> > > > > 
> > > > > The idea is to align crash kernel reservation to PUD boundaries, remap that
> > > > > PUD and then free the extra memory.
> > > > 
> > > > Hi Mike,
> > > > 
> > > > Thanks for the effort to work on this issue. While I have to say this
> > > > isnt's good because it can only be made relying on a prerequisite that
> > > > there's big enough memory. If on a system, say 2G memory, it's not easy
> > > > to succeed on getting one 1G memory. While we only require far smaller
> > > > region than 1G, e.g about 200M which should be easy to get. So the way
> > > > taken in this patchset is too quirky and will cause regression on
> > > > systemswith small memory. This kind of sytems with small memory exists
> > > > widely on virt guest instance.
> > > 
> > > I don't agree there is a regression. If the PUD-aligned allocation fails,
> > > there is a fallback to the allocation of the exact size requested for crash
> > > kernel. This allocation just won't get protected.
> > 
> > Sorry, I misunderstood it. I just went through the log and didn't
> > look into codes.
> > 
> > But honestly, if we accept the fallback which doesn't do the protection,
> > we should be able to take off the protection completely, right?
> > Otherwise, the reservation code is a little complicated.
> 
> We don't do protection of the crash kernel for most architectures
> supporting kexec ;-)

Yeah. The protection was introduced into x86 firstly by my former
colleague of Redhat as an enhancement. Later people ported it to arm64.
We have signature verification mechanism to check if corruption on
loaded kdump kernel happened. In fact, panic is a small probability
event, and accidental corruption on kdump kernel data is a much smaller
probability event. The protection is an icing on the cake. But if it
brings mess, better take it away if no way to clean up the mess.

> 
> My goal was to allow large systems with ZONE_DMA/DMA32 have block mappings
> in the linear map and crash kernel protection without breaking backward
> compatibility for the existing systems.
> 
> > > Also please note, that the changes are only for the case when user didn't
> > > force base-size pages in the linear map, so anything that works now will
> > > work the same way with this set applied.
> > >  
> > > > The crashkernel reservation happens after linear map because the
> > > > reservation needs to know the dma zone boundary, arm64_dma_phys_limit.
> > > > If we can deduce that before bootmem_init(), the reservation can be
> > > > done before linear map. I will make an attempt on that. If still can't
> > > > be accepted, we would like to take off the crashkernel region protection
> > > > on arm64 for now.
> > > 
> > > I doubt it would be easy because arm64_dma_phys_limit is determined after
> > > parsing of the device tree and there might be memory allocations of
> > > possibly unmapped memory during the parsing.
> > 
> > I have sent out the patches with an attempt, it's pretty straightforward
> > and simple. Because arm64 only has one exception, namely Raspberry Pi 4,
> > on which some peripherals can only address 30bit range. That is a corner
> > case, to be honest. And kdump is a necessary feature on server, but may
> > not be so expected on Raspberry Pi 4, a system for computer education
> > and hobbyists. And kdump only cares whether the dump target devices can
> > address 32bit range, namely storage device or network card on server.
> > If finally confirmed that storage devices can only address 30bit range
> > on Raspberry Pi 4, people still can have crashkernel=xM@yM method to
> > reserve crashkernel regions.
> 
> I hope you are right and Raspberry Pi 4 is the only system that limits
> DMA'able range to 30 bits. But with diversity of arm64 chips and boards I
> won't be surprised that there are other variants with a similar problem.

We still need people to confirm if the storage disk or NIC on RPi4 is
able to address 32 bit range. From Nicalas's patch log and cover-letter,
he said not all devices on RPi4 are 30bit addressable.

That's possible a new arm64 chip comes out with devices of 30bit addresing,
even though those arm64 servers usually deployed with devices of wider than
32bit DMA addressing ability. And I don't think users of the chip will care
about kdump. Kdump is relied more on enterprise level system.

On x86, we ignore those ISA devices in kdump kernel at the beginning.
As you can see, the current kdump kernel has no available physical pages
in DMA zone on x86. If people have a ISA device in x86_64 system, and
want to set it as dump target, it doesn't work at all. We don't support
the corner case. If we want to cover everything, we can only limp with
patches all over us.


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec