* Finding kernel RAM consumers ? @ 2022-06-02 19:24 Joakim Tjernlund 2022-06-02 20:11 ` Matthew Wilcox 0 siblings, 1 reply; 17+ messages in thread From: Joakim Tjernlund @ 2022-06-02 19:24 UTC (permalink / raw) To: linux-mm@kvack.org We have this small embedded target(aarch64) with 32 MB of RAM where the kernel consumes 14420K: Memory: 22444K/36864K available (3584K kernel code, 698K rwdata, 936K rodata, 320K init, 255K bss, 14420K reserved, 0K cma-reserved) I want to track down were most of this RAM is consumed so I can trim away some MBs but I am having a hard time finding may way. Is there some tool/kernel config that can help me with that? Jocke ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Finding kernel RAM consumers ? 2022-06-02 19:24 Finding kernel RAM consumers ? Joakim Tjernlund @ 2022-06-02 20:11 ` Matthew Wilcox 2022-06-03 6:49 ` Joakim Tjernlund 0 siblings, 1 reply; 17+ messages in thread From: Matthew Wilcox @ 2022-06-02 20:11 UTC (permalink / raw) To: Joakim Tjernlund; +Cc: linux-mm@kvack.org On Thu, Jun 02, 2022 at 07:24:23PM +0000, Joakim Tjernlund wrote: > We have this small embedded target(aarch64) with 32 MB of RAM where the kernel consumes 14420K: > Memory: 22444K/36864K available (3584K kernel code, 698K rwdata, 936K rodata, 320K init, 255K bss, 14420K reserved, 0K cma-reserved) > > I want to track down were most of this RAM is consumed so I can trim away some MBs > but I am having a hard time finding may way. > Is there some tool/kernel config that can help me with that? You may find this series of articles interesting: https://lwn.net/Articles/741494/ https://lwn.net/Articles/744507/ https://lwn.net/Articles/746780/ https://lwn.net/Articles/748198/ While they're a little old and they're targetting a much smaller system than yours, they may give you some ideas of things you can try and tools you can use. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Finding kernel RAM consumers ? 2022-06-02 20:11 ` Matthew Wilcox @ 2022-06-03 6:49 ` Joakim Tjernlund 2022-06-03 17:26 ` Joakim Tjernlund 0 siblings, 1 reply; 17+ messages in thread From: Joakim Tjernlund @ 2022-06-03 6:49 UTC (permalink / raw) To: willy@infradead.org; +Cc: linux-mm@kvack.org On Thu, 2022-06-02 at 21:11 +0100, Matthew Wilcox wrote: > On Thu, Jun 02, 2022 at 07:24:23PM +0000, Joakim Tjernlund wrote: > > We have this small embedded target(aarch64) with 32 MB of RAM where the kernel consumes 14420K: > > Memory: 22444K/36864K available (3584K kernel code, 698K rwdata, 936K rodata, 320K init, 255K bss, 14420K reserved, 0K cma-reserved) > > > > I want to track down were most of this RAM is consumed so I can trim away some MBs > > but I am having a hard time finding may way. > > Is there some tool/kernel config that can help me with that? > > You may find this series of articles interesting: > > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flwn.net%2FArticles%2F741494%2F&data=05%7C01%7CJoakim.Tjernlund%40infinera.com%7C509eaa2f688b4612945208da44d40494%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C0%7C637897974658412129%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=zP2oazSeoVKiTrbPONqmQXEhzyIIL3pQdvsKfN8nOA0%3D&reserved=0 > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flwn.net%2FArticles%2F744507%2F&data=05%7C01%7CJoakim.Tjernlund%40infinera.com%7C509eaa2f688b4612945208da44d40494%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C0%7C637897974658412129%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=9kNmXDhuQ24tWlECGrYQOqaaTsc0sOO4TozcQBk5aYc%3D&reserved=0 > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flwn.net%2FArticles%2F746780%2F&data=05%7C01%7CJoakim.Tjernlund%40infinera.com%7C509eaa2f688b4612945208da44d40494%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C0%7C637897974658412129%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ZFu0q4tXX677JwkgBqK73M8WxDrhylw0XnuEw5Y8mA4%3D&reserved=0 > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flwn.net%2FArticles%2F748198%2F&data=05%7C01%7CJoakim.Tjernlund%40infinera.com%7C509eaa2f688b4612945208da44d40494%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C0%7C637897974658412129%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ws5Rlrhts80ndzr8aEv3xTu6EmCnND3L%2FmGXHLDaygw%3D&reserved=0 > > While they're a little old and they're targetting a much smaller system > than yours, they may give you some ideas of things you can try and tools > you can use. Those are interesting, thanks. In my case it is the amount of work space RAM allocated that is a bit much. The kernel code/data is 5+MB but the total need is 14MB, 9MB is buffer and similar. Jocke ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Finding kernel RAM consumers ? 2022-06-03 6:49 ` Joakim Tjernlund @ 2022-06-03 17:26 ` Joakim Tjernlund 2022-06-03 17:29 ` Matthew Wilcox 0 siblings, 1 reply; 17+ messages in thread From: Joakim Tjernlund @ 2022-06-03 17:26 UTC (permalink / raw) To: willy@infradead.org; +Cc: linux-mm@kvack.org On Fri, 2022-06-03 at 08:49 +0200, Joakim Tjernlund wrote: > On Thu, 2022-06-02 at 21:11 +0100, Matthew Wilcox wrote: > > On Thu, Jun 02, 2022 at 07:24:23PM +0000, Joakim Tjernlund wrote: > > > We have this small embedded target(aarch64) with 32 MB of RAM where the kernel consumes 14420K: > > > Memory: 22444K/36864K available (3584K kernel code, 698K rwdata, 936K rodata, 320K init, 255K bss, 14420K reserved, 0K cma-reserved) > > > > > > I want to track down were most of this RAM is consumed so I can trim away some MBs > > > but I am having a hard time finding may way. > > > Is there some tool/kernel config that can help me with that? > > > > You may find this series of articles interesting: > > > > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flwn.net%2FArticles%2F741494%2F&data=05%7C01%7CJoakim.Tjernlund%40infinera.com%7C509eaa2f688b4612945208da44d40494%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C0%7C637897974658412129%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=zP2oazSeoVKiTrbPONqmQXEhzyIIL3pQdvsKfN8nOA0%3D&reserved=0 > > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flwn.net%2FArticles%2F744507%2F&data=05%7C01%7CJoakim.Tjernlund%40infinera.com%7C509eaa2f688b4612945208da44d40494%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C0%7C637897974658412129%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=9kNmXDhuQ24tWlECGrYQOqaaTsc0sOO4TozcQBk5aYc%3D&reserved=0 > > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flwn.net%2FArticles%2F746780%2F&data=05%7C01%7CJoakim.Tjernlund%40infinera.com%7C509eaa2f688b4612945208da44d40494%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C0%7C637897974658412129%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ZFu0q4tXX677JwkgBqK73M8WxDrhylw0XnuEw5Y8mA4%3D&reserved=0 > > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flwn.net%2FArticles%2F748198%2F&data=05%7C01%7CJoakim.Tjernlund%40infinera.com%7C509eaa2f688b4612945208da44d40494%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C0%7C637897974658412129%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ws5Rlrhts80ndzr8aEv3xTu6EmCnND3L%2FmGXHLDaygw%3D&reserved=0 > > > > While they're a little old and they're targetting a much smaller system > > than yours, they may give you some ideas of things you can try and tools > > you can use. > > Those are interesting, thanks. > In my case it is the amount of work space RAM allocated that is a bit much. The kernel code/data > is 5+MB but the total need is 14MB, 9MB is buffer and similar. > > Jocke Found something, arm64 only supports mem model SPARSEMEM_EXTREME and it uses most of my memory. Tried to force SPARSEMEM_STATIC but that didn't boot, is STATIC even cheaper ? ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Finding kernel RAM consumers ? 2022-06-03 17:26 ` Joakim Tjernlund @ 2022-06-03 17:29 ` Matthew Wilcox 0 siblings, 0 replies; 17+ messages in thread From: Matthew Wilcox @ 2022-06-03 17:29 UTC (permalink / raw) To: Joakim Tjernlund; +Cc: linux-mm@kvack.org, linux-arm-kernel On Fri, Jun 03, 2022 at 05:26:33PM +0000, Joakim Tjernlund wrote: > On Fri, 2022-06-03 at 08:49 +0200, Joakim Tjernlund wrote: > > On Thu, 2022-06-02 at 21:11 +0100, Matthew Wilcox wrote: > > > On Thu, Jun 02, 2022 at 07:24:23PM +0000, Joakim Tjernlund wrote: > > > > We have this small embedded target(aarch64) with 32 MB of RAM where the kernel consumes 14420K: > > > > Memory: 22444K/36864K available (3584K kernel code, 698K rwdata, 936K rodata, 320K init, 255K bss, 14420K reserved, 0K cma-reserved) > > > > > > > > I want to track down were most of this RAM is consumed so I can trim away some MBs > > > > but I am having a hard time finding may way. > > > > Is there some tool/kernel config that can help me with that? > > > > Those are interesting, thanks. > > In my case it is the amount of work space RAM allocated that is a bit much. The kernel code/data > > is 5+MB but the total need is 14MB, 9MB is buffer and similar. > > > > Jocke > > Found something, arm64 only supports mem model SPARSEMEM_EXTREME and it uses most of my memory. > Tried to force SPARSEMEM_STATIC but that didn't boot, is STATIC even cheaper ? That sounds like a question for the arm64 maintainers _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Finding kernel RAM consumers ? @ 2022-06-03 17:29 ` Matthew Wilcox 0 siblings, 0 replies; 17+ messages in thread From: Matthew Wilcox @ 2022-06-03 17:29 UTC (permalink / raw) To: Joakim Tjernlund; +Cc: linux-mm@kvack.org, linux-arm-kernel On Fri, Jun 03, 2022 at 05:26:33PM +0000, Joakim Tjernlund wrote: > On Fri, 2022-06-03 at 08:49 +0200, Joakim Tjernlund wrote: > > On Thu, 2022-06-02 at 21:11 +0100, Matthew Wilcox wrote: > > > On Thu, Jun 02, 2022 at 07:24:23PM +0000, Joakim Tjernlund wrote: > > > > We have this small embedded target(aarch64) with 32 MB of RAM where the kernel consumes 14420K: > > > > Memory: 22444K/36864K available (3584K kernel code, 698K rwdata, 936K rodata, 320K init, 255K bss, 14420K reserved, 0K cma-reserved) > > > > > > > > I want to track down were most of this RAM is consumed so I can trim away some MBs > > > > but I am having a hard time finding may way. > > > > Is there some tool/kernel config that can help me with that? > > > > Those are interesting, thanks. > > In my case it is the amount of work space RAM allocated that is a bit much. The kernel code/data > > is 5+MB but the total need is 14MB, 9MB is buffer and similar. > > > > Jocke > > Found something, arm64 only supports mem model SPARSEMEM_EXTREME and it uses most of my memory. > Tried to force SPARSEMEM_STATIC but that didn't boot, is STATIC even cheaper ? That sounds like a question for the arm64 maintainers ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Finding kernel RAM consumers ? 2022-06-03 17:29 ` Matthew Wilcox @ 2022-06-03 18:11 ` Arnd Bergmann -1 siblings, 0 replies; 17+ messages in thread From: Arnd Bergmann @ 2022-06-03 18:11 UTC (permalink / raw) To: Matthew Wilcox; +Cc: Joakim Tjernlund, linux-mm@kvack.org, Linux ARM On Fri, Jun 3, 2022 at 7:29 PM Matthew Wilcox <willy@infradead.org> wrote: > On Fri, Jun 03, 2022 at 05:26:33PM +0000, Joakim Tjernlund wrote: > > On Fri, 2022-06-03 at 08:49 +0200, Joakim Tjernlund wrote: > > > On Thu, 2022-06-02 at 21:11 +0100, Matthew Wilcox wrote: > > > > On Thu, Jun 02, 2022 at 07:24:23PM +0000, Joakim Tjernlund wrote: > > > > > We have this small embedded target(aarch64) with 32 MB of RAM where the kernel consumes 14420K: > > > > > Memory: 22444K/36864K available (3584K kernel code, 698K rwdata, 936K rodata, 320K init, 255K bss, 14420K reserved, 0K cma-reserved) > > > > > > > > > > I want to track down were most of this RAM is consumed so I can trim away some MBs > > > > > but I am having a hard time finding may way. > > > > > Is there some tool/kernel config that can help me with that? > > > > > > Those are interesting, thanks. > > > In my case it is the amount of work space RAM allocated that is a bit much. The kernel code/data > > > is 5+MB but the total need is 14MB, 9MB is buffer and similar. I'm fairly sure you are in new territory here. The arm64 kernel is just not really optimized for small memory configurations. Even for 32-bit Arm, 32MB is considered too small for most workloads these days, though we do have some very limited support for machines with as little as 2MB of SRAM. The smallest arm64 machines that I can think of being supported in the kernel today have at least 256MB. > > Found something, arm64 only supports mem model SPARSEMEM_EXTREME and it uses most of my memory. > > Tried to force SPARSEMEM_STATIC but that didn't boot, is STATIC even cheaper ? > > That sounds like a question for the arm64 maintainers From what I understand, the 'static' variant uses a static allocation in .bss, so that may end up consuming most of your RAM even if it works normally. If all memory is contiguous, then using flatmem may be more appropriate. You may also be able to save some memory for sparsemem by reducing the size of the physical address space. CONFIG_ARM64_PA_BITS is normally '48', but reducing it to '40' may help (provided you can get that to boot). Most of the smaller CPUs only support a 40 bit physical address space, so having another option for this is probably sensible. I think this is a case of "patches welcome". Nobody has really needed this so far, but as even the smaller machines are slowly migrating from 32-bit to 64-bit cores, optimizing this will get interesting for more developers. There are probably other low-hanging fruit that you can address after figuring out. One observation I made is that modern kernels don't seem to deal as well as older ones with low-memory situations, so even if you manage to free up most of your 32MB, it might still not work reliably. Arnd _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Finding kernel RAM consumers ? @ 2022-06-03 18:11 ` Arnd Bergmann 0 siblings, 0 replies; 17+ messages in thread From: Arnd Bergmann @ 2022-06-03 18:11 UTC (permalink / raw) To: Matthew Wilcox; +Cc: Joakim Tjernlund, linux-mm@kvack.org, Linux ARM On Fri, Jun 3, 2022 at 7:29 PM Matthew Wilcox <willy@infradead.org> wrote: > On Fri, Jun 03, 2022 at 05:26:33PM +0000, Joakim Tjernlund wrote: > > On Fri, 2022-06-03 at 08:49 +0200, Joakim Tjernlund wrote: > > > On Thu, 2022-06-02 at 21:11 +0100, Matthew Wilcox wrote: > > > > On Thu, Jun 02, 2022 at 07:24:23PM +0000, Joakim Tjernlund wrote: > > > > > We have this small embedded target(aarch64) with 32 MB of RAM where the kernel consumes 14420K: > > > > > Memory: 22444K/36864K available (3584K kernel code, 698K rwdata, 936K rodata, 320K init, 255K bss, 14420K reserved, 0K cma-reserved) > > > > > > > > > > I want to track down were most of this RAM is consumed so I can trim away some MBs > > > > > but I am having a hard time finding may way. > > > > > Is there some tool/kernel config that can help me with that? > > > > > > Those are interesting, thanks. > > > In my case it is the amount of work space RAM allocated that is a bit much. The kernel code/data > > > is 5+MB but the total need is 14MB, 9MB is buffer and similar. I'm fairly sure you are in new territory here. The arm64 kernel is just not really optimized for small memory configurations. Even for 32-bit Arm, 32MB is considered too small for most workloads these days, though we do have some very limited support for machines with as little as 2MB of SRAM. The smallest arm64 machines that I can think of being supported in the kernel today have at least 256MB. > > Found something, arm64 only supports mem model SPARSEMEM_EXTREME and it uses most of my memory. > > Tried to force SPARSEMEM_STATIC but that didn't boot, is STATIC even cheaper ? > > That sounds like a question for the arm64 maintainers From what I understand, the 'static' variant uses a static allocation in .bss, so that may end up consuming most of your RAM even if it works normally. If all memory is contiguous, then using flatmem may be more appropriate. You may also be able to save some memory for sparsemem by reducing the size of the physical address space. CONFIG_ARM64_PA_BITS is normally '48', but reducing it to '40' may help (provided you can get that to boot). Most of the smaller CPUs only support a 40 bit physical address space, so having another option for this is probably sensible. I think this is a case of "patches welcome". Nobody has really needed this so far, but as even the smaller machines are slowly migrating from 32-bit to 64-bit cores, optimizing this will get interesting for more developers. There are probably other low-hanging fruit that you can address after figuring out. One observation I made is that modern kernels don't seem to deal as well as older ones with low-memory situations, so even if you manage to free up most of your 32MB, it might still not work reliably. Arnd ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Finding kernel RAM consumers ? 2022-06-03 18:11 ` Arnd Bergmann @ 2022-06-07 5:41 ` Alexander Dahl -1 siblings, 0 replies; 17+ messages in thread From: Alexander Dahl @ 2022-06-07 5:41 UTC (permalink / raw) To: Arnd Bergmann Cc: Matthew Wilcox, Joakim Tjernlund, linux-mm@kvack.org, Linux ARM Hei hei, Am Fri, Jun 03, 2022 at 08:11:31PM +0200 schrieb Arnd Bergmann: > On Fri, Jun 3, 2022 at 7:29 PM Matthew Wilcox <willy@infradead.org> wrote: > > On Fri, Jun 03, 2022 at 05:26:33PM +0000, Joakim Tjernlund wrote: > > > On Fri, 2022-06-03 at 08:49 +0200, Joakim Tjernlund wrote: > > > > On Thu, 2022-06-02 at 21:11 +0100, Matthew Wilcox wrote: > > > > > On Thu, Jun 02, 2022 at 07:24:23PM +0000, Joakim Tjernlund wrote: > > > > > > We have this small embedded target(aarch64) with 32 MB of RAM where the kernel consumes 14420K: > > > > > > Memory: 22444K/36864K available (3584K kernel code, 698K rwdata, 936K rodata, 320K init, 255K bss, 14420K reserved, 0K cma-reserved) > > > > > > > > > > > > I want to track down were most of this RAM is consumed so I can trim away some MBs > > > > > > but I am having a hard time finding may way. > > > > > > Is there some tool/kernel config that can help me with that? > > > > > > > > Those are interesting, thanks. > > > > In my case it is the amount of work space RAM allocated that is a bit much. The kernel code/data > > > > is 5+MB but the total need is 14MB, 9MB is buffer and similar. > > I'm fairly sure you are in new territory here. The arm64 kernel is > just not really optimized > for small memory configurations. Even for 32-bit Arm, 32MB is > considered too small for > most workloads these days, though we do have some very limited support > for machines > with as little as 2MB of SRAM. The smallest arm64 machines that I can > think of being > supported in the kernel today have at least 256MB. > > > > Found something, arm64 only supports mem model SPARSEMEM_EXTREME and it uses most of my memory. > > > Tried to force SPARSEMEM_STATIC but that didn't boot, is STATIC even cheaper ? > > > > That sounds like a question for the arm64 maintainers > > From what I understand, the 'static' variant uses a static allocation > in .bss, so that > may end up consuming most of your RAM even if it works normally. If all memory > is contiguous, then using flatmem may be more appropriate. You may also be able > to save some memory for sparsemem by reducing the size of the physical address > space. CONFIG_ARM64_PA_BITS is normally '48', but reducing it to '40' may > help (provided you can get that to boot). Most of the smaller CPUs only support > a 40 bit physical address space, so having another option for this is > probably sensible. > > I think this is a case of "patches welcome". Nobody has really needed > this so far, but > as even the smaller machines are slowly migrating from 32-bit to > 64-bit cores, optimizing > this will get interesting for more developers. There are probably > other low-hanging > fruit that you can address after figuring out. The SiP variants of at91 SAMA5D2 (armv7) or SAM9x60 (armv5) come with 64 MiB or 128 MiB, and given the latter is a new SoC announced only two or three years ago, requiring at least 256 MiB would be at best unfortunate. Given those SoCs are used in industrial applications with very long support times, I think 32bit ARM will stay for years, even with new products. > One observation I made is that modern kernels don't seem to deal as > well as older > ones with low-memory situations, so even if you manage to free up most of your > 32MB, it might still not work reliably. Since we are using the SAMA5D27C-D5M in production, I would also be interested in support for running 32 bit ARM with recent kernels on systems with 64 MiB or even 32 MiB of memory. If there are specific things to test, you can let me know. Greets Alex _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Finding kernel RAM consumers ? @ 2022-06-07 5:41 ` Alexander Dahl 0 siblings, 0 replies; 17+ messages in thread From: Alexander Dahl @ 2022-06-07 5:41 UTC (permalink / raw) To: Arnd Bergmann Cc: Matthew Wilcox, Joakim Tjernlund, linux-mm@kvack.org, Linux ARM Hei hei, Am Fri, Jun 03, 2022 at 08:11:31PM +0200 schrieb Arnd Bergmann: > On Fri, Jun 3, 2022 at 7:29 PM Matthew Wilcox <willy@infradead.org> wrote: > > On Fri, Jun 03, 2022 at 05:26:33PM +0000, Joakim Tjernlund wrote: > > > On Fri, 2022-06-03 at 08:49 +0200, Joakim Tjernlund wrote: > > > > On Thu, 2022-06-02 at 21:11 +0100, Matthew Wilcox wrote: > > > > > On Thu, Jun 02, 2022 at 07:24:23PM +0000, Joakim Tjernlund wrote: > > > > > > We have this small embedded target(aarch64) with 32 MB of RAM where the kernel consumes 14420K: > > > > > > Memory: 22444K/36864K available (3584K kernel code, 698K rwdata, 936K rodata, 320K init, 255K bss, 14420K reserved, 0K cma-reserved) > > > > > > > > > > > > I want to track down were most of this RAM is consumed so I can trim away some MBs > > > > > > but I am having a hard time finding may way. > > > > > > Is there some tool/kernel config that can help me with that? > > > > > > > > Those are interesting, thanks. > > > > In my case it is the amount of work space RAM allocated that is a bit much. The kernel code/data > > > > is 5+MB but the total need is 14MB, 9MB is buffer and similar. > > I'm fairly sure you are in new territory here. The arm64 kernel is > just not really optimized > for small memory configurations. Even for 32-bit Arm, 32MB is > considered too small for > most workloads these days, though we do have some very limited support > for machines > with as little as 2MB of SRAM. The smallest arm64 machines that I can > think of being > supported in the kernel today have at least 256MB. > > > > Found something, arm64 only supports mem model SPARSEMEM_EXTREME and it uses most of my memory. > > > Tried to force SPARSEMEM_STATIC but that didn't boot, is STATIC even cheaper ? > > > > That sounds like a question for the arm64 maintainers > > From what I understand, the 'static' variant uses a static allocation > in .bss, so that > may end up consuming most of your RAM even if it works normally. If all memory > is contiguous, then using flatmem may be more appropriate. You may also be able > to save some memory for sparsemem by reducing the size of the physical address > space. CONFIG_ARM64_PA_BITS is normally '48', but reducing it to '40' may > help (provided you can get that to boot). Most of the smaller CPUs only support > a 40 bit physical address space, so having another option for this is > probably sensible. > > I think this is a case of "patches welcome". Nobody has really needed > this so far, but > as even the smaller machines are slowly migrating from 32-bit to > 64-bit cores, optimizing > this will get interesting for more developers. There are probably > other low-hanging > fruit that you can address after figuring out. The SiP variants of at91 SAMA5D2 (armv7) or SAM9x60 (armv5) come with 64 MiB or 128 MiB, and given the latter is a new SoC announced only two or three years ago, requiring at least 256 MiB would be at best unfortunate. Given those SoCs are used in industrial applications with very long support times, I think 32bit ARM will stay for years, even with new products. > One observation I made is that modern kernels don't seem to deal as > well as older > ones with low-memory situations, so even if you manage to free up most of your > 32MB, it might still not work reliably. Since we are using the SAMA5D27C-D5M in production, I would also be interested in support for running 32 bit ARM with recent kernels on systems with 64 MiB or even 32 MiB of memory. If there are specific things to test, you can let me know. Greets Alex ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Finding kernel RAM consumers ? 2022-06-07 5:41 ` Alexander Dahl @ 2022-06-07 8:38 ` Arnd Bergmann -1 siblings, 0 replies; 17+ messages in thread From: Arnd Bergmann @ 2022-06-07 8:38 UTC (permalink / raw) To: Arnd Bergmann, Matthew Wilcox, Joakim Tjernlund, linux-mm@kvack.org, Linux ARM On Tue, Jun 7, 2022 at 7:41 AM Alexander Dahl <ada@thorsis.com> wrote: > Am Fri, Jun 03, 2022 at 08:11:31PM +0200 schrieb Arnd Bergmann: > > On Fri, Jun 3, 2022 at 7:29 PM Matthew Wilcox <willy@infradead.org> wrote: > > > > I think this is a case of "patches welcome". Nobody has really needed > > this so far, but as even the smaller machines are slowly migrating from > > 32-bit to 64-bit cores, optimizing this will get interesting for more > > developers. There are probably other low-hanging > > fruit that you can address after figuring out. > > The SiP variants of at91 SAMA5D2 (armv7) or SAM9x60 (armv5) come with > 64 MiB or 128 MiB, and given the latter is a new SoC announced only > two or three years ago, requiring at least 256 MiB would be at best > unfortunate. Given those SoCs are used in industrial applications > with very long support times, I think 32bit ARM will stay for years, > even with new products. Yes, of course, and there is nothing wrong with that. We already see Cortex-A7 cores down to 7nm, all running Linux, and I expect there will likely be another 5 to 10 years of new 32-bit chips, and then another 10 years of people putting the existing chips into production, and after that a slow decline of users updating their kernels before supporting 32-bit hardware becomes too expensive to support in the kernel. On hardware that can run both 32-bit and 64-bit kernels, there are pretty much only upsides to running 64-bit kernels (with 32-bit thumb user space), but there is a memory overhead for the kernel itself, usually some 20 to 30 MB. Reducing this difference can hopefully help get fewer users stuck on 32-bit kernels by the time that they get too painful to use. > > One observation I made is that modern kernels don't seem to deal as > > well as older ones with low-memory situations, so even if you manage > > to free up most of your 32MB, it might still not work reliably. > > Since we are using the SAMA5D27C-D5M in production, I would also be > interested in support for running 32 bit ARM with recent kernels on > systems with 64 MiB or even 32 MiB of memory. If there are specific > things to test, you can let me know. I don't have anything specific, just the general feeling that there is something wrong about memory reclaim in smaller configurations. A system that is fine after bootup can run for a long time without running out of memory, but one thing I've observed in the past is that after a process manages to consume all RAM and swap space, killing that one task does not restore the overall system back into the same state as before, and it remains sluggish. Another issue is that I think we have more broken error handling for failed in-kernel memory allocations. After you see the kernel fail an allocation, I would usually recommend a reboot, and my feeling is that this used to be better in the past. Both of these could of course just be side-effects of kernel bloat, where a particular workload is now worse than in the past because it now needs more RAM to do the same thing. Arnd _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Finding kernel RAM consumers ? @ 2022-06-07 8:38 ` Arnd Bergmann 0 siblings, 0 replies; 17+ messages in thread From: Arnd Bergmann @ 2022-06-07 8:38 UTC (permalink / raw) To: Arnd Bergmann, Matthew Wilcox, Joakim Tjernlund, linux-mm@kvack.org, Linux ARM On Tue, Jun 7, 2022 at 7:41 AM Alexander Dahl <ada@thorsis.com> wrote: > Am Fri, Jun 03, 2022 at 08:11:31PM +0200 schrieb Arnd Bergmann: > > On Fri, Jun 3, 2022 at 7:29 PM Matthew Wilcox <willy@infradead.org> wrote: > > > > I think this is a case of "patches welcome". Nobody has really needed > > this so far, but as even the smaller machines are slowly migrating from > > 32-bit to 64-bit cores, optimizing this will get interesting for more > > developers. There are probably other low-hanging > > fruit that you can address after figuring out. > > The SiP variants of at91 SAMA5D2 (armv7) or SAM9x60 (armv5) come with > 64 MiB or 128 MiB, and given the latter is a new SoC announced only > two or three years ago, requiring at least 256 MiB would be at best > unfortunate. Given those SoCs are used in industrial applications > with very long support times, I think 32bit ARM will stay for years, > even with new products. Yes, of course, and there is nothing wrong with that. We already see Cortex-A7 cores down to 7nm, all running Linux, and I expect there will likely be another 5 to 10 years of new 32-bit chips, and then another 10 years of people putting the existing chips into production, and after that a slow decline of users updating their kernels before supporting 32-bit hardware becomes too expensive to support in the kernel. On hardware that can run both 32-bit and 64-bit kernels, there are pretty much only upsides to running 64-bit kernels (with 32-bit thumb user space), but there is a memory overhead for the kernel itself, usually some 20 to 30 MB. Reducing this difference can hopefully help get fewer users stuck on 32-bit kernels by the time that they get too painful to use. > > One observation I made is that modern kernels don't seem to deal as > > well as older ones with low-memory situations, so even if you manage > > to free up most of your 32MB, it might still not work reliably. > > Since we are using the SAMA5D27C-D5M in production, I would also be > interested in support for running 32 bit ARM with recent kernels on > systems with 64 MiB or even 32 MiB of memory. If there are specific > things to test, you can let me know. I don't have anything specific, just the general feeling that there is something wrong about memory reclaim in smaller configurations. A system that is fine after bootup can run for a long time without running out of memory, but one thing I've observed in the past is that after a process manages to consume all RAM and swap space, killing that one task does not restore the overall system back into the same state as before, and it remains sluggish. Another issue is that I think we have more broken error handling for failed in-kernel memory allocations. After you see the kernel fail an allocation, I would usually recommend a reboot, and my feeling is that this used to be better in the past. Both of these could of course just be side-effects of kernel bloat, where a particular workload is now worse than in the past because it now needs more RAM to do the same thing. Arnd ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Finding kernel RAM consumers ? 2022-06-07 8:38 ` Arnd Bergmann @ 2022-06-07 8:54 ` Joakim Tjernlund -1 siblings, 0 replies; 17+ messages in thread From: Joakim Tjernlund @ 2022-06-07 8:54 UTC (permalink / raw) To: linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, willy@infradead.org, arnd@arndb.de On Tue, 2022-06-07 at 10:38 +0200, Arnd Bergmann wrote: > On Tue, Jun 7, 2022 at 7:41 AM Alexander Dahl <ada@thorsis.com> wrote: > > Am Fri, Jun 03, 2022 at 08:11:31PM +0200 schrieb Arnd Bergmann: > > > On Fri, Jun 3, 2022 at 7:29 PM Matthew Wilcox <willy@infradead.org> wrote: > > > > > > I think this is a case of "patches welcome". Nobody has really needed > > > this so far, but as even the smaller machines are slowly migrating from > > > 32-bit to 64-bit cores, optimizing this will get interesting for more > > > developers. There are probably other low-hanging > > > fruit that you can address after figuring out. > > > > The SiP variants of at91 SAMA5D2 (armv7) or SAM9x60 (armv5) come with > > 64 MiB or 128 MiB, and given the latter is a new SoC announced only > > two or three years ago, requiring at least 256 MiB would be at best > > unfortunate. Given those SoCs are used in industrial applications > > with very long support times, I think 32bit ARM will stay for years, > > even with new products. > > Yes, of course, and there is nothing wrong with that. We already see > Cortex-A7 cores down to 7nm, all running Linux, and I expect there > will likely be another 5 to 10 years of new 32-bit chips, and then another > 10 years of people putting the existing chips into production, and after > that a slow decline of users updating their kernels before supporting > 32-bit hardware becomes too expensive to support in the kernel. My aarch64 system with 36MB RAM just dropped from: Memory: 22444K/36864K available (3584K kernel code, 698K rwdata, 936K rodata, 320K init, 255K bss, 14420K reserved, 0K cma) to Memory: 29732K/36864K available (3648K kernel code, 698K rwdata, 936K rodata, 320K init, 255K bss, 7132K reserved, 0K cma) with a small hack by Ard Biesheuvel <ardb@kernel.org> ----------------------------------------------------------------- What you might try is changing the section size to 32 MB and mapping the vmemmap region down to pages. That way, the vmemmap region should only take up - 512 KiB for the struct page array[] itself - 4 KiB for the page table that replaces the 2 MB block mapping You could try the below and see if it makes any difference? diff --git a/arch/arm64/include/asm/sparsemem.h b/arch/arm64/include/asm/sparsemem.h index 4b73463423c3..a008f4342532 100644 --- a/arch/arm64/include/asm/sparsemem.h +++ b/arch/arm64/include/asm/sparsemem.h @@ -23,7 +23,7 @@ * entries could not be created for vmemmap mappings. * 16K follows 4K for simplicity. */ -#define SECTION_SIZE_BITS 27 +#define SECTION_SIZE_BITS 25 #endif /* CONFIG_ARM64_64K_PAGES */ #endif diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 5b1946f1805c..d25560a53a67 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -1196,7 +1196,7 @@ static void free_empty_tables(unsigned long addr, unsigned long end, } #endif -#if !ARM64_KERNEL_USES_PMD_MAPS +#if 1// !ARM64_KERNEL_USES_PMD_MAPS int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node, struct vmem_altmap *altmap) { So there is hope for systems with little RAM Jocke _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: Finding kernel RAM consumers ? @ 2022-06-07 8:54 ` Joakim Tjernlund 0 siblings, 0 replies; 17+ messages in thread From: Joakim Tjernlund @ 2022-06-07 8:54 UTC (permalink / raw) To: linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, willy@infradead.org, arnd@arndb.de On Tue, 2022-06-07 at 10:38 +0200, Arnd Bergmann wrote: > On Tue, Jun 7, 2022 at 7:41 AM Alexander Dahl <ada@thorsis.com> wrote: > > Am Fri, Jun 03, 2022 at 08:11:31PM +0200 schrieb Arnd Bergmann: > > > On Fri, Jun 3, 2022 at 7:29 PM Matthew Wilcox <willy@infradead.org> wrote: > > > > > > I think this is a case of "patches welcome". Nobody has really needed > > > this so far, but as even the smaller machines are slowly migrating from > > > 32-bit to 64-bit cores, optimizing this will get interesting for more > > > developers. There are probably other low-hanging > > > fruit that you can address after figuring out. > > > > The SiP variants of at91 SAMA5D2 (armv7) or SAM9x60 (armv5) come with > > 64 MiB or 128 MiB, and given the latter is a new SoC announced only > > two or three years ago, requiring at least 256 MiB would be at best > > unfortunate. Given those SoCs are used in industrial applications > > with very long support times, I think 32bit ARM will stay for years, > > even with new products. > > Yes, of course, and there is nothing wrong with that. We already see > Cortex-A7 cores down to 7nm, all running Linux, and I expect there > will likely be another 5 to 10 years of new 32-bit chips, and then another > 10 years of people putting the existing chips into production, and after > that a slow decline of users updating their kernels before supporting > 32-bit hardware becomes too expensive to support in the kernel. My aarch64 system with 36MB RAM just dropped from: Memory: 22444K/36864K available (3584K kernel code, 698K rwdata, 936K rodata, 320K init, 255K bss, 14420K reserved, 0K cma) to Memory: 29732K/36864K available (3648K kernel code, 698K rwdata, 936K rodata, 320K init, 255K bss, 7132K reserved, 0K cma) with a small hack by Ard Biesheuvel <ardb@kernel.org> ----------------------------------------------------------------- What you might try is changing the section size to 32 MB and mapping the vmemmap region down to pages. That way, the vmemmap region should only take up - 512 KiB for the struct page array[] itself - 4 KiB for the page table that replaces the 2 MB block mapping You could try the below and see if it makes any difference? diff --git a/arch/arm64/include/asm/sparsemem.h b/arch/arm64/include/asm/sparsemem.h index 4b73463423c3..a008f4342532 100644 --- a/arch/arm64/include/asm/sparsemem.h +++ b/arch/arm64/include/asm/sparsemem.h @@ -23,7 +23,7 @@ * entries could not be created for vmemmap mappings. * 16K follows 4K for simplicity. */ -#define SECTION_SIZE_BITS 27 +#define SECTION_SIZE_BITS 25 #endif /* CONFIG_ARM64_64K_PAGES */ #endif diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 5b1946f1805c..d25560a53a67 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -1196,7 +1196,7 @@ static void free_empty_tables(unsigned long addr, unsigned long end, } #endif -#if !ARM64_KERNEL_USES_PMD_MAPS +#if 1// !ARM64_KERNEL_USES_PMD_MAPS int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node, struct vmem_altmap *altmap) { So there is hope for systems with little RAM Jocke ^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: Finding kernel RAM consumers ? 2022-06-07 8:38 ` Arnd Bergmann (?) (?) @ 2022-06-07 9:44 ` Russell King (Oracle) 2022-06-07 10:22 ` Arnd Bergmann -1 siblings, 1 reply; 17+ messages in thread From: Russell King (Oracle) @ 2022-06-07 9:44 UTC (permalink / raw) To: Arnd Bergmann Cc: Matthew Wilcox, Joakim Tjernlund, linux-mm@kvack.org, Linux ARM On Tue, Jun 07, 2022 at 10:38:54AM +0200, Arnd Bergmann wrote: > On Tue, Jun 7, 2022 at 7:41 AM Alexander Dahl <ada@thorsis.com> wrote: > > Am Fri, Jun 03, 2022 at 08:11:31PM +0200 schrieb Arnd Bergmann: > > > On Fri, Jun 3, 2022 at 7:29 PM Matthew Wilcox <willy@infradead.org> wrote: > > > > > > I think this is a case of "patches welcome". Nobody has really needed > > > this so far, but as even the smaller machines are slowly migrating from > > > 32-bit to 64-bit cores, optimizing this will get interesting for more > > > developers. There are probably other low-hanging > > > fruit that you can address after figuring out. > > > > The SiP variants of at91 SAMA5D2 (armv7) or SAM9x60 (armv5) come with > > 64 MiB or 128 MiB, and given the latter is a new SoC announced only > > two or three years ago, requiring at least 256 MiB would be at best > > unfortunate. Given those SoCs are used in industrial applications > > with very long support times, I think 32bit ARM will stay for years, > > even with new products. > > Yes, of course, and there is nothing wrong with that. We already see > Cortex-A7 cores down to 7nm, all running Linux, and I expect there > will likely be another 5 to 10 years of new 32-bit chips, and then another > 10 years of people putting the existing chips into production, and after > that a slow decline of users updating their kernels before supporting > 32-bit hardware becomes too expensive to support in the kernel. It should be noted that 20 years puts us past the 2038 32-bit time_t wrap problem - and although there's been work to address that in the UAPI, that doesn't mean that userspace will cope. Anyone deploying a system that is expected to still be live beyond the end of 32-bit time_t had better be testing their userspace for that event now! -- RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last! _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Finding kernel RAM consumers ? 2022-06-07 9:44 ` Russell King (Oracle) @ 2022-06-07 10:22 ` Arnd Bergmann 0 siblings, 0 replies; 17+ messages in thread From: Arnd Bergmann @ 2022-06-07 10:22 UTC (permalink / raw) To: Russell King (Oracle) Cc: Arnd Bergmann, Matthew Wilcox, Joakim Tjernlund, linux-mm@kvack.org, Linux ARM On Tue, Jun 7, 2022 at 11:44 AM Russell King (Oracle) <linux@armlinux.org.uk> wrote: > On Tue, Jun 07, 2022 at 10:38:54AM +0200, Arnd Bergmann wrote: > > > > Yes, of course, and there is nothing wrong with that. We already see > > Cortex-A7 cores down to 7nm, all running Linux, and I expect there > > will likely be another 5 to 10 years of new 32-bit chips, and then another > > 10 years of people putting the existing chips into production, and after > > that a slow decline of users updating their kernels before supporting > > 32-bit hardware becomes too expensive to support in the kernel. > > It should be noted that 20 years puts us past the 2038 32-bit time_t > wrap problem - and although there's been work to address that in the > UAPI, that doesn't mean that userspace will cope. > > Anyone deploying a system that is expected to still be live beyond > the end of 32-bit time_t had better be testing their userspace for > that event now! That is absolutely true, but it's also independent of what kernel is being used. I assume that anyone who is this memory constrained is using 32-bit thumb userspace even on 64-bit kernels. The base distro support in embedded systems using openembedded with musl-1.2.x usually works fine beyond y2038, but of course each system should be tested for this before shipping as there are still bugs in less common code. arnd _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Finding kernel RAM consumers ? @ 2022-06-07 10:22 ` Arnd Bergmann 0 siblings, 0 replies; 17+ messages in thread From: Arnd Bergmann @ 2022-06-07 10:22 UTC (permalink / raw) To: Russell King (Oracle) Cc: Arnd Bergmann, Matthew Wilcox, Joakim Tjernlund, linux-mm@kvack.org, Linux ARM On Tue, Jun 7, 2022 at 11:44 AM Russell King (Oracle) <linux@armlinux.org.uk> wrote: > On Tue, Jun 07, 2022 at 10:38:54AM +0200, Arnd Bergmann wrote: > > > > Yes, of course, and there is nothing wrong with that. We already see > > Cortex-A7 cores down to 7nm, all running Linux, and I expect there > > will likely be another 5 to 10 years of new 32-bit chips, and then another > > 10 years of people putting the existing chips into production, and after > > that a slow decline of users updating their kernels before supporting > > 32-bit hardware becomes too expensive to support in the kernel. > > It should be noted that 20 years puts us past the 2038 32-bit time_t > wrap problem - and although there's been work to address that in the > UAPI, that doesn't mean that userspace will cope. > > Anyone deploying a system that is expected to still be live beyond > the end of 32-bit time_t had better be testing their userspace for > that event now! That is absolutely true, but it's also independent of what kernel is being used. I assume that anyone who is this memory constrained is using 32-bit thumb userspace even on 64-bit kernels. The base distro support in embedded systems using openembedded with musl-1.2.x usually works fine beyond y2038, but of course each system should be tested for this before shipping as there are still bugs in less common code. arnd ^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2022-06-07 10:30 UTC | newest] Thread overview: 17+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2022-06-02 19:24 Finding kernel RAM consumers ? Joakim Tjernlund 2022-06-02 20:11 ` Matthew Wilcox 2022-06-03 6:49 ` Joakim Tjernlund 2022-06-03 17:26 ` Joakim Tjernlund 2022-06-03 17:29 ` Matthew Wilcox 2022-06-03 17:29 ` Matthew Wilcox 2022-06-03 18:11 ` Arnd Bergmann 2022-06-03 18:11 ` Arnd Bergmann 2022-06-07 5:41 ` Alexander Dahl 2022-06-07 5:41 ` Alexander Dahl 2022-06-07 8:38 ` Arnd Bergmann 2022-06-07 8:38 ` Arnd Bergmann 2022-06-07 8:54 ` Joakim Tjernlund 2022-06-07 8:54 ` Joakim Tjernlund 2022-06-07 9:44 ` Russell King (Oracle) 2022-06-07 10:22 ` Arnd Bergmann 2022-06-07 10:22 ` Arnd Bergmann
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.