Packing data in kernel memory

Packing data in kernel memory

[John Richard Moser]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Is there a way to pack and store arbitrary data in the kernel, or do I
need to roll my own?

As a simple example, let's say I wanted to store 8 elements.  Assume
PAGE_SIZE is 4.  If I store these individually, their size and use is:

ELEM      SIZE    PAGES    USE
[A.]      4        1        4
[B...]    6        2        8
[C]       3        1        4
[D....]   7        2        8
[E]       3        1        4
[F.]      4        1        4
[G......] 9        3       12
[H.....]  8        2        8

TOTAL    44       13       52

2 excess pages, 8 excess units of memory wasted.

Now let's say I used groups of 4 pages:

E                 S   P   U
[A.][H.....]      12   4  16
[B...][C][D....]  16   4  16
[E][F.][G......]  16   4  16
                  44  12  48

1 excess pages, 4 units wasted memory.
- --
All content of all messages exchanged herein are left in the
Public Domain, unless otherwise explicitly stated.

    Creative brains are a valuable, limited resource. They shouldn't be
    wasted on re-inventing the wheel when there are so many fascinating
    new problems waiting out there.
                                                 -- Eric Steven Raymond

    We will enslave their women, eat their children and rape their
    cattle!
                  -- Bosc, Evil alien overlord from the fifth dimension
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iQIVAwUBRI8IkQs1xW0HCTEFAQJcCxAAnucDd+TBSS7hjHmYMdG64WRHwcidvGkS
prFXPSWKXZh0tTvV5+DGDwUvGeLew1VvEY5IJl7i6ak7V0nehAKuEIdMiL9x1RSI
OeZN2cA/eGl1Bv1r3QJdhutfCPF10yFMEUkkLwa2+z/bbPlN4zdoUaR0yAFRigwL
zuJbOTlrEQcY8lAIlPbtdm+c1mQ+FNfpe7dGEYWwOZZYeE8wglCQbFhzmda9nmP5
fJrod94dJnsGsD03MOs17+0+GaLEWz09KKCjMIXRi9fG4q1N9Kgi/GwH+kUR/Cnq
kT44JgcNIuHYMFVss4qUA+wBit3HuT7zL4U9BhxIMpg7BFaYPVCDD+Lpi4J50WXk
sYJs9GpOHwSAmeK1RQ68FU9iO7HI2g0LPKrAWxXj+nclaMhmBqQAF2Eue7LxnSbm
Cy2Ne3nWIPnIFA4y76L8VslsYgFhaBdrzydIHKTH/mqDHWvNMDmcCy6sSnueGXOR
qBopgebvKRpnCtwQvpQ+RgwvGx7/XyftP+W+Lag5BCC1yezP6vk7HDj1lvKrrKVf
kyGNcNx0t3yWnSqeJC1M8nOdpSfHsuDjoG03HdPcmdurQdaGtUmA5k37GpjvATBf
1JAst4FXf9LWJ8mwQakbp7r4vUrEyfMfub9J/j2u1zxq37bJCyAs6jxb5D7YmSno
zzV/7vqPdU0=
=kiK1
-----END PGP SIGNATURE-----

Re: Packing data in kernel memory

[Jan Engelhardt]

>Subject: Packing data in kernel memory
>

Can't you just use mlock(), if you want to keep it in RAM?

Or do you need it in kernel memory, because you need it in the lowmem area? 
Or for interaction with other kernel code?

>Is there a way to pack and store arbitrary data in the kernel, or do I
>need to roll my own?
>

Write a device driver, kmalloc some buffer, and copy data via a write 
function from userspace to that buffer. Should be trivial.

>1 excess pages, 4 units wasted memory.

Of course, kmalloc only works up to some boundary AFIACS.


Jan Engelhardt
-- 

Re: Packing data in kernel memory

[Randy.Dunlap]

On Tue, 13 Jun 2006 22:18:55 +0200 (MEST) Jan Engelhardt wrote:

> 
> >Subject: Packing data in kernel memory
> >
> 
> Can't you just use mlock(), if you want to keep it in RAM?
> 
> Or do you need it in kernel memory, because you need it in the lowmem area? 
> Or for interaction with other kernel code?
> 
> >Is there a way to pack and store arbitrary data in the kernel, or do I
> >need to roll my own?

Sounds a bit like a slab cache to me.

> Write a device driver, kmalloc some buffer, and copy data via a write 
> function from userspace to that buffer. Should be trivial.
> 
> >1 excess pages, 4 units wasted memory.
> 
> Of course, kmalloc only works up to some boundary AFIACS.

128 KB on some arches.  More on a few IIRC.

---
~Randy

Re: Packing data in kernel memory

[John Richard Moser]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



Randy.Dunlap wrote:
> On Tue, 13 Jun 2006 22:18:55 +0200 (MEST) Jan Engelhardt wrote:
> 
>>> Subject: Packing data in kernel memory
>>>
>> Can't you just use mlock(), if you want to keep it in RAM?
>>
>> Or do you need it in kernel memory, because you need it in the lowmem area? 
>> Or for interaction with other kernel code?
>>
>>> Is there a way to pack and store arbitrary data in the kernel, or do I
>>> need to roll my own?
> 
> Sounds a bit like a slab cache to me.
> 

OK cool, can I make that non-swappable?  I'm going to be trying to do
this between where kernel swaps a page out and swapped page actually is
written to disk.  The result will be a "Swap Zone," in-memory storage of
pages that the rest of the kernel thinks have been swapped to disk.
(Code here will use the swap interface, so the rest of the kernel thinks
it's just swapping; I'll handle whether to pull it out of memory or off
disk behind that)

The need for packing pages comes because eventually (using above
infrastructure) I'll be taking sets of 32KiB of data and compressing it;
I don't want to pad up to 4095 excess unused bytes if that stuff
compresses to 28KiB+1 :)  (more likely 16KiB+1 +/-8KiB)

This is all, of course, assuming I ever figure out how the heck to get
in the middle of the swapping process.  I'm looking at mm/page_io.c
swap_writepage() and friends and scratching my head.  I have no idea wtf...


>> Write a device driver, kmalloc some buffer, and copy data via a write 
>> function from userspace to that buffer. Should be trivial.
>>
>>> 1 excess pages, 4 units wasted memory.
>> Of course, kmalloc only works up to some boundary AFIACS.
> 
> 128 KB on some arches.  More on a few IIRC.
> 
> ---
> ~Randy
> 

- --
All content of all messages exchanged herein are left in the
Public Domain, unless otherwise explicitly stated.

    Creative brains are a valuable, limited resource. They shouldn't be
    wasted on re-inventing the wheel when there are so many fascinating
    new problems waiting out there.
                                                 -- Eric Steven Raymond

    We will enslave their women, eat their children and rape their
    cattle!
                  -- Bosc, Evil alien overlord from the fifth dimension
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iQIVAwUBRI8mAAs1xW0HCTEFAQJa4w/+Jrhnnp+DyOmuuQPL0A2QbydRlhvyeK6g
mixAd41AJPN8CmMqFZzWTPFbhN65BiM3oaKv+5YX8kvzJiKfhi8BabLlkapUgljx
qlFr2yOSTIBEkPiPaUTjjYSLFfVBqca1kAAcjO7qJGjcrCJK0AkVp11XKvF8xiLI
Hg8kEV1GzQKtMo65s+HQQR8XDTDRPuyTpGgWbVSwHyZnJY1pwFd2gVNpW63y52QM
pJw7WyLBa4NNDLLNRvX8/DbSjvaN3fYy223GItS67QaSOv5G9MNXQnhmUQV8dV0J
k4xiOhPBRoV1tDpIbdFTWajPp5facVZfLklsNv1uPyDUxdsrMDa8ETNsd6Kn3A5V
8Zp6EQWScyqoDa8u7aL2IZ0BCm69aJnaAXLm3miNheW1vUPmKYOZJn3+lmEX1vMh
JuXZzUYjgFgIss3djrpC2GoWqlMYgQ92ZBxecBoMQowPGkLwtcbz4J3qfwBalr1+
q9Ho85mH7eFUyod7ftWS8r6SQ1WtxCGTl9aPnwqlroq2RG1a/3bhJ2NIHyoLc+zt
y4IpzZ7B1JzTpBVKKkksOMv7B3XmxCwNzr4Qc1ilx+cLlqwsChiiAzv0IFXz6cBT
LVeTTHGyQOk7Yd5jO+Wi/s+9XsXEPFQGLDIimxqjqhXitcErAWOodg7tidYgMgFW
snfgQan4PZI=
=B4VW
-----END PGP SIGNATURE-----

Re: Packing data in kernel memory

[Randy.Dunlap]

On Tue, 13 Jun 2006 16:54:26 -0400 John Richard Moser wrote:

> Randy.Dunlap wrote:
> > On Tue, 13 Jun 2006 22:18:55 +0200 (MEST) Jan Engelhardt wrote:
> > 
> >>> Subject: Packing data in kernel memory
> >>>
> >> Can't you just use mlock(), if you want to keep it in RAM?
> >>
> >> Or do you need it in kernel memory, because you need it in the lowmem area? 
> >> Or for interaction with other kernel code?
> >>
> >>> Is there a way to pack and store arbitrary data in the kernel, or do I
> >>> need to roll my own?
> > 
> > Sounds a bit like a slab cache to me.
> > 
> 
> OK cool, can I make that non-swappable?

kernel (allocated) memory is non-swappable.

> I'm going to be trying to do
> this between where kernel swaps a page out and swapped page actually is
> written to disk.  The result will be a "Swap Zone," in-memory storage of
> pages that the rest of the kernel thinks have been swapped to disk.
> (Code here will use the swap interface, so the rest of the kernel thinks
> it's just swapping; I'll handle whether to pull it out of memory or off
> disk behind that)
> 
> The need for packing pages comes because eventually (using above
> infrastructure) I'll be taking sets of 32KiB of data and compressing it;
> I don't want to pad up to 4095 excess unused bytes if that stuff
> compresses to 28KiB+1 :)  (more likely 16KiB+1 +/-8KiB)
> 
> This is all, of course, assuming I ever figure out how the heck to get
> in the middle of the swapping process.  I'm looking at mm/page_io.c
> swap_writepage() and friends and scratching my head.  I have no idea wtf...
> 
> 
> >> Write a device driver, kmalloc some buffer, and copy data via a write 
> >> function from userspace to that buffer. Should be trivial.
> >>
> >>> 1 excess pages, 4 units wasted memory.
> >> Of course, kmalloc only works up to some boundary AFIACS.
> > 
> > 128 KB on some arches.  More on a few IIRC.

---
~Randy