* Re: [PATCH 00/14] Pramfs: Persistent and protected ram filesystem
From: Jamie Lokier @ 2009-06-24 17:47 UTC (permalink / raw)
To: Pavel Machek
Cc: Marco, Tim Bird, Linux Embedded, Linux Kernel, Linux FS Devel,
Daniel Walker
In-Reply-To: <20090623182941.GA2045@elf.ucw.cz>
Pavel Machek wrote:
> On Tue 2009-06-23 20:07:23, Marco wrote:
> > You are talked about journaling. This schema works well for a disk, but
> > what about a piece of ram? What about a crazy kernel that write in that
> > area for a bug? Do you remember for example the e1000e bug? It's not
>
> I believe you need both journaling *and* write protection. How do you
> handle power fault while writing data?
I think this is basically right.
write protection for the crazy kernels, and journalling for
powerfail/crash during updates.
Journalling can be extremely simple. It can be just one memory block
at a fixed location, double-buffering all writes.
Pramfs already has checksums, which makes that easier. You just write
to the buffer area first, with checksum, then write to the final area.
Mount looks at the buffer area, and if the checksum is fine, copies
the contents to the destination block.
That's all it takes to be resistant against power failures and crashes
during writes. Probably <100 lines of code.
-- Jamie
^ permalink raw reply
* Re: [PATCH 00/14] Pramfs: Persistent and protected ram filesystem
From: Pavel Machek @ 2009-06-24 17:59 UTC (permalink / raw)
To: Marco
Cc: tim.bird, jamie, Linux Embedded, Linux Kernel, Linux FS Devel,
Daniel Walker
In-Reply-To: <4A42649D.6080509@gmail.com>
On Wed 2009-06-24 19:38:37, Marco wrote:
> >>> Pavel Machek wrote:
> >>>> On Mon 2009-06-22 14:50:01, Tim Bird wrote:
> >>>>> Pavel Machek wrote:
> >>>>>>> block of fast non-volatile RAM that need to access data on it using a
> >>>>>>> standard filesytem interface."
> >>>>>> Turns a block of fast RAM into 13MB/sec disk. Hmm. I believe you are
> >>>>>> better with ext2.
> >>>>> Not if you want the RAM-based filesystem to persist over a kernel
> >>>>> invocation.
> >>>> Yes, you'll need to code Persistent, RAM-based _block_device_.
> >>> First of all I have to say that I'd like to update the site and make it
> >>> clearer but at the moment it's not possible because I'm not the admin
> >>> and I've already asked to the sourceforge support to have this possibility.
> >>>
> >>> About the comments: sincerely I don't understand the comments. We have
> >>> *already* a fs that takes care to remap a piace of ram (ram, sram,
> >>> nvram, etc.), takes care of caching problems, takes care of write
> >> Well, it looks pramfs design is confused. 13MB/sec shows that caching
> >> _is_ useful for pramfs. So...?
> >
> > caching problems means to avoid filesystem corruption, so dirty pages in
> > the page cache are not allowed to be written back to the backing-store
> > RAM. It's clear that there is a performance penalty. This penalty should
> > be reduced by the access speed of the RAM, however the performance are
> > not important for this special fs as Tim Bird said, so this question is
> > not relevant for me. If this issue is not clear enough on the web site,
> > I hope I can update the information in the future.
> >
> >>> You are talked about journaling. This schema works well for a disk, but
> >>> what about a piece of ram? What about a crazy kernel that write in that
> >>> area for a bug? Do you remember for example the e1000e bug? It's not
> >> I believe you need both journaling *and* write protection. How do you
> >> handle power fault while writing data?
> >> Pavel
> >
> > Ah now the write protection is a "needed feature", in your previous
> > comment you talked about why not use ext2/3.......
> >
> > Marco
> >
>
> Just for your information I tried the same test with pc in a virtual machine with 32MB of RAM:
>
> Version 1.03e ------Sequential Output------ --Sequential Input- --Random-
> -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
> Machine Size:chnk K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
> hostname 15M:1k 14156 99 128779 100 92240 100 11669 100 166242 99 80058 82
> ------Sequential Create------ --------Random Create--------
> -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
> 4 2842 99 133506 104 45088 101 2787 99 79581 101 58212 102
>
> These data are the proof of the importance of the environment, workload and so on when we talk
> about benchmark. Your consideration are really superficial.
Unfortunately, your numbers are meaningless.
Pavel
(PCs should have cca 3GB/sec RAM transfer rates; and you demosstrated
cca 166MB/sec read rate; disk is 80MB/sec, so that's too slow. If you
want to prove your filesystem the filesystem is reasonably fast,
compare it with ext2 on ramdisk.)
--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
^ permalink raw reply
* Status of bzip2 and lzma kernel compression for ARM?
From: Michael Opdenacker @ 2009-06-24 20:22 UTC (permalink / raw)
To: Alain Knaff
Cc: Thomas Petazzoni, Russell King - ARM Linux,
Linux Kernel Mailing List, linux-arm-kernel,
linux-embedded mailing list
Hi Alain,
I would like to test bzip2 and lzma compression on ARM.
Would you mind telling me what the status of your patches on ARM is, now
that the x86 and architecture independent code has been merged? Have
you or has anyone already updated them for 2.6.30, or shall I update
them by myself?
Thank you in advance,
Cheers,
Michael.
--
Michael Opdenacker, Free Electrons
Kernel, drivers and embedded Linux development,
consulting, training and support.
http://free-electrons.com
-------------------------------------------------------------------
List admin: http://lists.arm.linux.org.uk/mailman/listinfo/linux-arm-kernel
FAQ: http://www.arm.linux.org.uk/mailinglists/faq.php
Etiquette: http://www.arm.linux.org.uk/mailinglists/etiquette.php
^ permalink raw reply
* Re: How the kernel printk works before do console_setup.
From: Benjamin Herrenschmidt @ 2009-06-24 22:56 UTC (permalink / raw)
To: Tim Bird; +Cc: Johnny Hung, linuxppc-dev, linux-kernel, linux-embedded
In-Reply-To: <4A41B3A5.9010807@am.sony.com>
> Before the console is set up, the printk data is formatted
> and put into the kernel log buffer, but not sent to any console.
> Any messages printk'ed before that are buffered but do not
> appear. When the console is initialized, then all buffered
> messages are sent to the console, and subsequent printks cause
> the message to go to the log buffer, but then immediately
> get sent from there to the console.
>
> Under certain conditions you can examine the log buffer of
> a kernel that failed to initialize it's console, after a
> warm reset of the machine, using the firmware memory dump
> command.
On ppc, we have tricks to display things earlier :-)
We can initialize the serial ports way before console_setup() (and we do
in most cases) and we use what we call the "udbg" console until the real
one takes over. The "udbg" console is a very small layer which outputs
via a provided "putc" routine. Platforms can provide their own here, we
have a collection of standard ones for legacy UARTs (it should be
automatically setup in that case by the code in legacy_serial), Apple
ESCCs, etc... We even have compile time options that allow that stuff to
be initialized before start_kernel...
Cheers,
Ben.
> See http://elinux.org/Kernel_Debugging_Tips#Accessing_the_printk_buffer_after_a_silent_hang_on_boot
> for some tips on accessing the log buffer of a previous boot.
>
> Note also, that if the serial console does not come up,
> but the kernel successfully boots, and you can get a network
> login on the machine, you can always print out the kernel log
> buffer messages using 'dmesg' at a user-space shell prompt.
>
> Hope this helps!
> -- Tim
>
> =============================
> Tim Bird
> Architecture Group Chair, CE Linux Forum
> Senior Staff Engineer, Sony Corporation of America
> =============================
>
> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/linuxppc-dev
^ permalink raw reply
* Re: How the kernel printk works before do console_setup.
From: Johnny Hung @ 2009-06-25 6:27 UTC (permalink / raw)
To: Benjamin Herrenschmidt; +Cc: linuxppc-dev, linux-embedded, linux-kernel
In-Reply-To: <1245884203.21200.18.camel@pasglop>
2009/6/25 Benjamin Herrenschmidt <benh@kernel.crashing.org>:
>
>> Before the console is set up, the printk data is formatted
>> and put into the kernel log buffer, but not sent to any console.
>> Any messages printk'ed before that are buffered but do not
>> appear. When the console is initialized, then all buffered
>> messages are sent to the console, and subsequent printks cause
>> the message to go to the log buffer, but then immediately
>> get sent from there to the console.
>>
>> Under certain conditions you can examine the log buffer of
>> a kernel that failed to initialize it's console, after a
>> warm reset of the machine, using the firmware memory dump
>> command.
>
> On ppc, we have tricks to display things earlier :-)
>
> We can initialize the serial ports way before console_setup() (and we do
> in most cases) and we use what we call the "udbg" console until the real
> one takes over. The "udbg" console is a very small layer which outputs
> via a provided "putc" routine. Platforms can provide their own here, we
> have a collection of standard ones for legacy UARTs (it should be
> automatically setup in that case by the code in legacy_serial), Apple
> ESCCs, etc... We even have compile time options that allow that stuff to
> be initialized before start_kernel...
Thank you. This is what I described and want to understand. The
arch/powerpc/kernel/legacy_serial.c
do find_legacy_serial_ports then take a default serial port by using
open firmware device tree
information. The find_legacy_serial_ports() called form setup_arch but
I don't know who call
setup_arch (setup_32.c)function. Can you give me a hint ? Thanks in advanced.
BRs, H. Johnny
^ permalink raw reply
* Re: [PATCH 00/14] Pramfs: Persistent and protected ram filesystem
From: Marco Stornelli @ 2009-06-25 6:30 UTC (permalink / raw)
To: Pavel Machek
Cc: tim.bird, jamie, Linux Embedded, Linux Kernel, Linux FS Devel,
Daniel Walker
In-Reply-To: <20090624175943.GB6618@elf.ucw.cz>
2009/6/24 Pavel Machek <pavel@ucw.cz>:
> On Wed 2009-06-24 19:38:37, Marco wrote:
>> >>> Pavel Machek wrote:
>> >>>> On Mon 2009-06-22 14:50:01, Tim Bird wrote:
>> >>>>> Pavel Machek wrote:
>> >>>>>>> block of fast non-volatile RAM that need to access data on it using a
>> >>>>>>> standard filesytem interface."
>> >>>>>> Turns a block of fast RAM into 13MB/sec disk. Hmm. I believe you are
>> >>>>>> better with ext2.
>> >>>>> Not if you want the RAM-based filesystem to persist over a kernel
>> >>>>> invocation.
>> >>>> Yes, you'll need to code Persistent, RAM-based _block_device_.
>> >>> First of all I have to say that I'd like to update the site and make it
>> >>> clearer but at the moment it's not possible because I'm not the admin
>> >>> and I've already asked to the sourceforge support to have this possibility.
>> >>>
>> >>> About the comments: sincerely I don't understand the comments. We have
>> >>> *already* a fs that takes care to remap a piace of ram (ram, sram,
>> >>> nvram, etc.), takes care of caching problems, takes care of write
>> >> Well, it looks pramfs design is confused. 13MB/sec shows that caching
>> >> _is_ useful for pramfs. So...?
>> >
>> > caching problems means to avoid filesystem corruption, so dirty pages in
>> > the page cache are not allowed to be written back to the backing-store
>> > RAM. It's clear that there is a performance penalty. This penalty should
>> > be reduced by the access speed of the RAM, however the performance are
>> > not important for this special fs as Tim Bird said, so this question is
>> > not relevant for me. If this issue is not clear enough on the web site,
>> > I hope I can update the information in the future.
>> >
>> >>> You are talked about journaling. This schema works well for a disk, but
>> >>> what about a piece of ram? What about a crazy kernel that write in that
>> >>> area for a bug? Do you remember for example the e1000e bug? It's not
>> >> I believe you need both journaling *and* write protection. How do you
>> >> handle power fault while writing data?
>> >> Pavel
>> >
>> > Ah now the write protection is a "needed feature", in your previous
>> > comment you talked about why not use ext2/3.......
>> >
>> > Marco
>> >
>>
>> Just for your information I tried the same test with pc in a virtual machine with 32MB of RAM:
>>
>> Version 1.03e ------Sequential Output------ --Sequential Input- --Random-
>> -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
>> Machine Size:chnk K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
>> hostname 15M:1k 14156 99 128779 100 92240 100 11669 100 166242 99 80058 82
>> ------Sequential Create------ --------Random Create--------
>> -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
>> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
>> 4 2842 99 133506 104 45088 101 2787 99 79581 101 58212 102
>>
>> These data are the proof of the importance of the environment, workload and so on when we talk
>> about benchmark. Your consideration are really superficial.
>
> Unfortunately, your numbers are meaningless.
I don't think so.
> Pavel
>
> (PCs should have cca 3GB/sec RAM transfer rates; and you demosstrated
> cca 166MB/sec read rate; disk is 80MB/sec, so that's too slow. If you
> want to prove your filesystem the filesystem is reasonably fast,
> compare it with ext2 on ramdisk.)
>
This is the point. I don't want compare it with ext2 from performance
point of view. This comparison makes no sense for me. I've done this
test to prove that if you change environment you can change in a
purposeful way the results.
Marco
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH 00/14] Pramfs: Persistent and protected ram filesystem
From: Marco Stornelli @ 2009-06-25 6:32 UTC (permalink / raw)
To: Jamie Lokier
Cc: Pavel Machek, Tim Bird, Linux Embedded, Linux Kernel,
Linux FS Devel, Daniel Walker
In-Reply-To: <20090624174700.GI14121@shareable.org>
2009/6/24 Jamie Lokier <jamie@shareable.org>:
> Pavel Machek wrote:
>> On Tue 2009-06-23 20:07:23, Marco wrote:
>> > You are talked about journaling. This schema works well for a disk, but
>> > what about a piece of ram? What about a crazy kernel that write in that
>> > area for a bug? Do you remember for example the e1000e bug? It's not
>>
>> I believe you need both journaling *and* write protection. How do you
>> handle power fault while writing data?
>
> I think this is basically right.
>
> write protection for the crazy kernels, and journalling for
> powerfail/crash during updates.
>
> Journalling can be extremely simple. It can be just one memory block
> at a fixed location, double-buffering all writes.
>
> Pramfs already has checksums, which makes that easier. You just write
> to the buffer area first, with checksum, then write to the final area.
> Mount looks at the buffer area, and if the checksum is fine, copies
> the contents to the destination block.
>
> That's all it takes to be resistant against power failures and crashes
> during writes. Probably <100 lines of code.
>
> -- Jamie
>
It seems a reasonable request.
Marco
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: How the kernel printk works before do console_setup.
From: Michael Ellerman @ 2009-06-25 6:37 UTC (permalink / raw)
To: Johnny Hung
Cc: Benjamin Herrenschmidt, linuxppc-dev, linux-embedded,
linux-kernel
In-Reply-To: <cb9ecdfa0906242327j76d1e4fbib13bc7d9753aeacf@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 1885 bytes --]
On Thu, 2009-06-25 at 14:27 +0800, Johnny Hung wrote:
> 2009/6/25 Benjamin Herrenschmidt <benh@kernel.crashing.org>:
> >
> >> Before the console is set up, the printk data is formatted
> >> and put into the kernel log buffer, but not sent to any console.
> >> Any messages printk'ed before that are buffered but do not
> >> appear. When the console is initialized, then all buffered
> >> messages are sent to the console, and subsequent printks cause
> >> the message to go to the log buffer, but then immediately
> >> get sent from there to the console.
> >>
> >> Under certain conditions you can examine the log buffer of
> >> a kernel that failed to initialize it's console, after a
> >> warm reset of the machine, using the firmware memory dump
> >> command.
> >
> > On ppc, we have tricks to display things earlier :-)
> >
> > We can initialize the serial ports way before console_setup() (and we do
> > in most cases) and we use what we call the "udbg" console until the real
> > one takes over. The "udbg" console is a very small layer which outputs
> > via a provided "putc" routine. Platforms can provide their own here, we
> > have a collection of standard ones for legacy UARTs (it should be
> > automatically setup in that case by the code in legacy_serial), Apple
> > ESCCs, etc... We even have compile time options that allow that stuff to
> > be initialized before start_kernel...
>
> Thank you. This is what I described and want to understand. The
> arch/powerpc/kernel/legacy_serial.c
> do find_legacy_serial_ports then take a default serial port by using
> open firmware device tree
> information. The find_legacy_serial_ports() called form setup_arch but
> I don't know who call
> setup_arch (setup_32.c)function. Can you give me a hint ? Thanks in advanced.
setup_arch() is called from start_kernel() in init/main.c
cheers
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]
^ permalink raw reply
* Re: [PATCH 00/14] Pramfs: Persistent and protected ram filesystem
From: Marco Stornelli @ 2009-06-25 6:44 UTC (permalink / raw)
To: Jamie Lokier; +Cc: Linux Embedded, Linux Kernel, Linux FS Devel, Daniel Walker
In-Reply-To: <20090624174140.GH14121@shareable.org>
2009/6/24 Jamie Lokier <jamie@shareable.org>:
> Marco wrote:
>> > Second question: what happens if the system crashing _during_ a write
>> > to a file. Does it mean that file will fail it's checksum when it's
>> > read at the next boot?
>> >
>> > Maybe files aren't so important. What about when you write a file,
>> > and then rename it over an existing file to replace it. (E.g. a
>> > config file), and the system crashes _during_ the rename? At the next
>> > boot, is it guaranteed to see either the old or the new file, or can
>> > the directory be corrupt / fail it's checksum?
>>
>> First of all I have to explain better the current policy: the checksum
>> works at inode and superblock level and currently there isn't a recovery
>> function as the journaling. About the superblock it's easy to use a
>> redundant policy to be more robust.
>
> To be honest, superblock robustness is less of a concern. The real
> concern is losing file or directory contents, so it can't be used to
> store persistent configuration data, only debugging logs.
>
>> About the inode, at the moment when the checksum doesn't match the
>> inode it's marked as bad calling the function make_bad_inode().
>
> Let's see if I understand right.
>
> If it lose power when writing to a file, after boot the file is likely
> to be marked bad and so return -EIO instead of any file contents?
Depends on the checksum. If you lose power before the checksum update
of the inode
you'll have a bad inode and then an -EIO at the next access.
>
> If it loses power when doing atomic rename (to replace config files,
> for example), it's likely that the whole /pramfs/configs/ directory
> will be corrupt, because the rename is writing to the directory inode,
> so you lose access to all names in that directory?
>
> That sounds like it can't be used for persistent configuration data.
It's true from this point of view currently there is a lack for this
and it needs a bit of effort to resolve this problem.
From this point of view I'd like to point out that I know that there
was some aspects to study in a deeper way, so I'll need
of more then one review :) but since this fs has been abandoned since
2004 and it hadn't ever reviewed, it was important
to do a serious review with the kernel community to understand all the
problems.
>
> If a directory is marked as bad, or a file-inode in it is marked bad,
> can you even rmdir it to clean up and start again?
>
You can start again always. You can remount the fs with the init
option and then you'll have a new fs.
Marco
^ permalink raw reply
* Re: Status of bzip2 and lzma kernel compression for ARM?
From: Mike Rapoport @ 2009-06-25 8:56 UTC (permalink / raw)
To: Michael Opdenacker
Cc: Alain Knaff, Thomas Petazzoni, Russell King - ARM Linux,
Linux Kernel Mailing List, linux-arm-kernel,
linux-embedded mailing list
In-Reply-To: <4A428AFF.20307@free-electrons.com>
Hi Michael,
Michael Opdenacker wrote:
> Hi Alain,
>
> I would like to test bzip2 and lzma compression on ARM.
>
> Would you mind telling me what the status of your patches on ARM is, now
> that the x86 and architecture independent code has been merged? Have
> you or has anyone already updated them for 2.6.30, or shall I update
> them by myself?
I'm not sure what exactly do you mean by "test bzip2 and lzma compression on
ARM", but if you refer to internal initramfs compression, I can tell that I've
used lzma compression on PXA270 and it works fine.
> Thank you in advance,
>
> Cheers,
>
> Michael.
>
--
Sincerely yours,
Mike.
^ permalink raw reply
* Re: Status of bzip2 and lzma kernel compression for ARM?
From: Florian Fainelli @ 2009-06-25 9:04 UTC (permalink / raw)
To: Mike Rapoport
Cc: Michael Opdenacker, Alain Knaff, Thomas Petazzoni,
Russell King - ARM Linux, Linux Kernel Mailing List,
linux-arm-kernel, linux-embedded mailing list
In-Reply-To: <4A433BCD.2030506@compulab.co.il>
Hi Michael, Mike
Le Thursday 25 June 2009 10:56:45 Mike Rapoport, vous avez écrit :
> Hi Michael,
>
> Michael Opdenacker wrote:
> > Hi Alain,
> >
> > I would like to test bzip2 and lzma compression on ARM.
> >
> > Would you mind telling me what the status of your patches on ARM is, now
> > that the x86 and architecture independent code has been merged? Have
> > you or has anyone already updated them for 2.6.30, or shall I update
> > them by myself?
>
> I'm not sure what exactly do you mean by "test bzip2 and lzma compression
> on ARM", but if you refer to internal initramfs compression, I can tell
> that I've used lzma compression on PXA270 and it works fine.
I think Michael refers to the lzma decompressor which allows you to use a
lzma-compressed kernel as a (b)zImage with its architecture specific
decompressor piggy-backed in the (b)zImage.
Such thing would also be very useful on MIPS and PowerPC as well.
--
Best regards, Florian Fainelli
Email : florian@openwrt.org
http://openwrt.org
-------------------------------
^ permalink raw reply
* Saving the kernel log accross boots
From: Martin Meuli @ 2009-06-25 14:40 UTC (permalink / raw)
To: linux-embedded
Hi All
In case of a crash we would like to save the kernel log and retrieve it in
the next boot.
There are already several suggestions on how to do it on this and other
lists.
(Kernel crashing and log buffers: Robin Getz)
Unfortunately we don't have extra SRAM to use for it. We need to do it in
the 'normal' RAM.
ARCH=ARM
What we did:
- modify kernel/printk.c: use
__attribute__((__section__(".bss.kringbuf"... to put the log related stuf
into a separate section
- modify the linker script to link .bss.kringbuf at the start of the .bss
section
- modify arch/arm/kernel/head-common.S to exclude the new section from
being nulled (if it was a watchdog restart)
- modify kernel/printk.c to use cach_flush after prink
- make sure the bootloader does not overwrite that memory
This seems to work.
After a crash we get the messages from the new boot and the last crash.
It seems that the decompress code is using memory at the end of the kernel
and destroys the log. So we have to use the bootloader to decompress the
image into RAM.
Before digging any deeper I would like to know if someone as a better
(more general) idea.
Regards, Martin
^ permalink raw reply
* Re: Saving the kernel log accross boots
From: Bill Gatliff @ 2009-06-25 15:23 UTC (permalink / raw)
To: Martin Meuli; +Cc: linux-embedded
In-Reply-To: <op.uv238aaxq25ece@thur.netmodule.intranet>
Martin Meuli wrote:
>
> Before digging any deeper I would like to know if someone as a better
> (more general) idea.
Could you stuff the output into an MTD device?
What about netconsole, or putting the console on a serial port? Then
you could send the log out before and up to the point the crash happens.
^ permalink raw reply
* Re: Status of bzip2 and lzma kernel compression for ARM?
From: Michael Opdenacker @ 2009-06-25 15:24 UTC (permalink / raw)
To: Florian Fainelli
Cc: Mike Rapoport, Alain Knaff, Thomas Petazzoni,
Russell King - ARM Linux, Linux Kernel Mailing List,
linux-arm-kernel, linux-embedded mailing list
In-Reply-To: <200906251104.02830.florian@openwrt.org>
On 06/25/2009 11:04 AM, Florian Fainelli wrote:
> Le Thursday 25 June 2009 10:56:45 Mike Rapoport, vous avez écrit :
>> I'm not sure what exactly do you mean by "test bzip2 and lzma compression
>> on ARM", but if you refer to internal initramfs compression, I can tell
>> that I've used lzma compression on PXA270 and it works fine.
>>
>
> I think Michael refers to the lzma decompressor which allows you to use a
> lzma-compressed kernel as a (b)zImage with its architecture specific
> decompressor piggy-backed in the (b)zImage.
>
> Such thing would also be very useful on MIPS and PowerPC as well.
>
Yes, that's what I meant: using a bzip2 or lzma compressed kernel. I
would like to know the boot time impact on an ARM board.
If I don't get any answer from Alain in the next days, I will propose a
patch update for ARM.
Cheers,
Michael.
--
Michael Opdenacker, Free Electrons
Kernel, drivers and embedded Linux development,
consulting, training and support.
http://free-electrons.com
^ permalink raw reply
* Re: Saving the kernel log accross boots
From: Wolfram Sang @ 2009-06-25 15:29 UTC (permalink / raw)
To: Bill Gatliff; +Cc: Martin Meuli, linux-embedded
In-Reply-To: <4A43968A.2020905@billgatliff.com>
[-- Attachment #1: Type: text/plain, Size: 376 bytes --]
> > Before digging any deeper I would like to know if someone as a better
> > (more general) idea.
>
> Could you stuff the output into an MTD device?
For that, see CONFIG_MTD_OOPS.
Regards,
Wolfram
--
Pengutronix e.K. | Wolfram Sang |
Industrial Linux Solutions | http://www.pengutronix.de/ |
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]
^ permalink raw reply
* Re: Saving the kernel log accross boots
From: Marco @ 2009-06-25 17:02 UTC (permalink / raw)
To: Martin Meuli; +Cc: linux-embedded
In-Reply-To: <op.uv238aaxq25ece@thur.netmodule.intranet>
Martin Meuli wrote:
> Hi All
>
> In case of a crash we would like to save the kernel log and retrieve it
> in the next boot.
> There are already several suggestions on how to do it on this and other
> lists.
> (Kernel crashing and log buffers: Robin Getz)
> Unfortunately we don't have extra SRAM to use for it. We need to do it
> in the 'normal' RAM.
>
> ARCH=ARM
>
> What we did:
> - modify kernel/printk.c: use
> __attribute__((__section__(".bss.kringbuf"... to put the log related
> stuf into a separate section
> - modify the linker script to link .bss.kringbuf at the start of the
> .bss section
> - modify arch/arm/kernel/head-common.S to exclude the new section from
> being nulled (if it was a watchdog restart)
> - modify kernel/printk.c to use cach_flush after prink
> - make sure the bootloader does not overwrite that memory
>
> This seems to work.
> After a crash we get the messages from the new boot and the last crash.
> It seems that the decompress code is using memory at the end of the
> kernel and destroys the log. So we have to use the bootloader to
> decompress the image into RAM.
>
> Before digging any deeper I would like to know if someone as a better
> (more general) idea.
>
> Regards, Martin
I submitted for a review a new filesystem for 2.6.30 called pramfs that
it can be useful for your case. You could use a little piece of ram to
store the log.
Marco
^ permalink raw reply
* Re: Status of bzip2 and lzma kernel compression for ARM?
From: Paul Mundt @ 2009-06-25 17:08 UTC (permalink / raw)
To: Michael Opdenacker
Cc: Florian Fainelli, Mike Rapoport, Alain Knaff, Thomas Petazzoni,
Russell King - ARM Linux, Linux Kernel Mailing List,
linux-arm-kernel, linux-embedded mailing list
In-Reply-To: <4A4396CB.9000703@free-electrons.com>
On Thu, Jun 25, 2009 at 05:24:59PM +0200, Michael Opdenacker wrote:
> On 06/25/2009 11:04 AM, Florian Fainelli wrote:
> > Le Thursday 25 June 2009 10:56:45 Mike Rapoport, vous avez ?crit :
> >> I'm not sure what exactly do you mean by "test bzip2 and lzma compression
> >> on ARM", but if you refer to internal initramfs compression, I can tell
> >> that I've used lzma compression on PXA270 and it works fine.
> >>
> >
> > I think Michael refers to the lzma decompressor which allows you to use a
> > lzma-compressed kernel as a (b)zImage with its architecture specific
> > decompressor piggy-backed in the (b)zImage.
> >
> > Such thing would also be very useful on MIPS and PowerPC as well.
> >
> Yes, that's what I meant: using a bzip2 or lzma compressed kernel. I
> would like to know the boot time impact on an ARM board.
>
> If I don't get any answer from Alain in the next days, I will propose a
> patch update for ARM.
>
It's possible to use this generically now, yes. Anything building uImages
today can basically wire it up in a similar fashion to how blackfin has
and support it out of the box.
^ permalink raw reply
* Re: Saving the kernel log accross boots
From: Martin Meuli @ 2009-06-26 8:21 UTC (permalink / raw)
To: linux-embedded
In-Reply-To: <20090625152954.GB8445@pengutronix.de>
On Thu, 25 Jun 2009 17:29:54 +0200, Wolfram Sang <w.sang@pengutronix.de>
wrote:
>
>> > Before digging any deeper I would like to know if someone as a better
>> > (more general) idea.
>>
>> Could you stuff the output into an MTD device?
>
> For that, see CONFIG_MTD_OOPS.
Thak's looks great.
Right now we have a watchdog without OOPS.
>
> Regards,
>
> Wolfram
>
--
MM
^ permalink raw reply
* Re: [PATCH 00/14] Pramfs: Persistent and protected ram filesystem
From: Jamie Lokier @ 2009-06-26 11:30 UTC (permalink / raw)
To: Marco Stornelli
Cc: Linux Embedded, Linux Kernel, Linux FS Devel, Daniel Walker
In-Reply-To: <2ea1731b0906242344x5c8a6e58t5f82377be3d73411@mail.gmail.com>
Marco Stornelli wrote:
> 2009/6/24 Jamie Lokier <jamie@shareable.org>:
> > Marco wrote:
> >> > Second question: what happens if the system crashing _during_ a write
> >> > to a file. Does it mean that file will fail it's checksum when it's
> >> > read at the next boot?
> >> >
> >> > Maybe files aren't so important. What about when you write a file,
> >> > and then rename it over an existing file to replace it. (E.g. a
> >> > config file), and the system crashes _during_ the rename? At the next
> >> > boot, is it guaranteed to see either the old or the new file, or can
> >> > the directory be corrupt / fail it's checksum?
> >>
> >> First of all I have to explain better the current policy: the checksum
> >> works at inode and superblock level and currently there isn't a recovery
> >> function as the journaling. About the superblock it's easy to use a
> >> redundant policy to be more robust.
> >
> > To be honest, superblock robustness is less of a concern. The real
> > concern is losing file or directory contents, so it can't be used to
> > store persistent configuration data, only debugging logs.
> >
> >> About the inode, at the moment when the checksum doesn't match the
> >> inode it's marked as bad calling the function make_bad_inode().
> >
> > Let's see if I understand right.
> >
> > If it lose power when writing to a file, after boot the file is likely
> > to be marked bad and so return -EIO instead of any file contents?
>
> Depends on the checksum. If you lose power before the checksum update
> of the inode
> you'll have a bad inode and then an -EIO at the next access.
>
> >
> > If it loses power when doing atomic rename (to replace config files,
> > for example), it's likely that the whole /pramfs/configs/ directory
> > will be corrupt, because the rename is writing to the directory inode,
> > so you lose access to all names in that directory?
> >
> > That sounds like it can't be used for persistent configuration data.
>
> It's true from this point of view currently there is a lack for this
> and it needs a bit of effort to resolve this problem. >From this
> point of view I'd like to point out that I know that there was some
> aspects to study in a deeper way, so I'll need of more then one
> review :) but since this fs has been abandoned since 2004 and it
> hadn't ever reviewed, it was important to do a serious review with
> the kernel community to understand all the problems.
That's reasonable.
What do you think of my suggestion to double-buffer writes using a
single fixed position block, as explained elsewhere in this thread?
It should give the power fail safety with very little code. I don't
know how much it would slwo down writing. That probably depends on
whether it's the checksum which is slow (which only needs to be done
once when double-buffering), or the writing.
-- Jamie
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Kernel crashing and log buffers...
From: Robin Getz @ 2009-06-26 14:39 UTC (permalink / raw)
To: David VomLehn
Cc: Mike Frysinger, Greg Ungerer, Russell King, Paul Mundt, Tim Bird,
Wolfgang Denk, Grant Erickson, linux-embedded
In-Reply-To: <20090611175303.GA10058@cuplxvomd02.corp.sa.net>
OK - so after a bit more digging into things (and a poke by Mike) - I think
most of the stuff already exists.
Normal run time issues:
--------------------------
- MTD_OOPS - on 2007-06-28 this was added to Linus's tree:
http://lkml.org/lkml/2007/6/18/234
CONFIG_MTD_OOPS
tristate "Log panic/oops to an MTD buffer"
depends on MTD
help
This enables panic and oops messages to be logged to a circular
buffer in a flash partition where it can be read back at some
later point.
To use, add console=ttyMTDx to the kernel command line,
where x is the MTD device number to use.
Note - this currently _only_ does things when oops_in_progress is set, but
should be easy to extend to all - if that what people want.
- mtd devices can be backed by RAM.
CONFIG_MTD_RAM
- mtd devices can be backed by flash (if you want things to be persistant
across power cycles)...
early boot issues:
----------------------
- CONFIG_EARLY_PRINTK
- early printk just defined a console - and is supported by:
alpha, blackfin, microblaze, mips, powerpc, sh, x86
- it's pretty trivial to support a memory based buffer - some
archs already support it.
I think this only leaves Wolfgang's desire for memory buffers from the
bootloader to get (somehow) into the kernel's log buffer for syslog
processing...
Anyone else agree?
^ permalink raw reply
* Re: [PATCH 00/14] Pramfs: Persistent and protected ram filesystem
From: Marco @ 2009-06-26 16:56 UTC (permalink / raw)
To: Jamie Lokier; +Cc: Linux Embedded, Linux Kernel, Linux FS Devel, Daniel Walker
In-Reply-To: <20090626113016.GB17737@shareable.org>
Jamie Lokier wrote:
> Marco Stornelli wrote:
>> 2009/6/24 Jamie Lokier <jamie@shareable.org>:
>>> Marco wrote:
>>> If it loses power when doing atomic rename (to replace config files,
>>> for example), it's likely that the whole /pramfs/configs/ directory
>>> will be corrupt, because the rename is writing to the directory inode,
>>> so you lose access to all names in that directory?
>>>
>>> That sounds like it can't be used for persistent configuration data.
>> It's true from this point of view currently there is a lack for this
>> and it needs a bit of effort to resolve this problem. >From this
>> point of view I'd like to point out that I know that there was some
>> aspects to study in a deeper way, so I'll need of more then one
>> review :) but since this fs has been abandoned since 2004 and it
>> hadn't ever reviewed, it was important to do a serious review with
>> the kernel community to understand all the problems.
>
> That's reasonable.
>
> What do you think of my suggestion to double-buffer writes using a
> single fixed position block, as explained elsewhere in this thread?
>
> It should give the power fail safety with very little code. I don't
> know how much it would slwo down writing. That probably depends on
> whether it's the checksum which is slow (which only needs to be done
> once when double-buffering), or the writing.
>
> -- Jamie
>
Yeah it can be a choice. For this fs it's important to use "simple" but
useful mechanism. What do you exactly mean with "fixed position block"?
A fixed position in the fs? For example superblock+inodetable+in-use
bitmap+blocks+"double-buffering block"? Using a temp block of the same
size of blocks used, isn't it? I agree, but I think it needs more then
100 lines of code. Even with this simple schema it needs a mechanism
with a timeout to do the "commit" of the temp block, it needs a
mechanism to read the temp block instead of the "old" block or a
mechanism to write-back the temp block. So it can be implemented but it
needs a bit of effort. I think I'll implement it in the next release.
Marco
^ permalink raw reply
* Re: Kernel crashing and log buffers...
From: David VomLehn @ 2009-06-26 17:42 UTC (permalink / raw)
To: Robin Getz
Cc: Mike Frysinger, Greg Ungerer, Russell King, Paul Mundt, Tim Bird,
Wolfgang Denk, Grant Erickson, linux-embedded
In-Reply-To: <200906261039.50570.rgetz@blackfin.uclinux.org>
On Fri, Jun 26, 2009 at 10:39:50AM -0400, Robin Getz wrote:
> OK - so after a bit more digging into things (and a poke by Mike) - I think
> most of the stuff already exists.
>
> Normal run time issues:
> --------------------------
> - MTD_OOPS - on 2007-06-28 this was added to Linus's tree:
> http://lkml.org/lkml/2007/6/18/234
> CONFIG_MTD_OOPS
> tristate "Log panic/oops to an MTD buffer"
...
> early boot issues:
> ----------------------
> - CONFIG_EARLY_PRINTK
> - early printk just defined a console - and is supported by:
> alpha, blackfin, microblaze, mips, powerpc, sh, x86
> - it's pretty trivial to support a memory based buffer - some
> archs already support it.
>
> I think this only leaves Wolfgang's desire for memory buffers from the
> bootloader to get (somehow) into the kernel's log buffer for syslog
> processing...
>
> Anyone else agree?
Almost. A couple of us also want memory for "flight data record" FDR data
for doing continuous logging. This would, ideally, be either uncached or cached
in such a way that data is guaranteed to be written to memory in the event of
a watchdog timer-induced system reset.
David VL
^ permalink raw reply
* Re: Kernel crashing and log buffers...
From: Mike Frysinger @ 2009-06-26 17:50 UTC (permalink / raw)
To: David VomLehn
Cc: Robin Getz, Greg Ungerer, Russell King, Paul Mundt, Tim Bird,
Wolfgang Denk, Grant Erickson, linux-embedded
In-Reply-To: <20090626174219.GA19426@cuplxvomd02.corp.sa.net>
On Fri, Jun 26, 2009 at 13:42, David VomLehn wrote:
> On Fri, Jun 26, 2009 at 10:39:50AM -0400, Robin Getz wrote:
>> OK - so after a bit more digging into things (and a poke by Mike) - I think
>> most of the stuff already exists.
>>
>> Normal run time issues:
>> --------------------------
>> - MTD_OOPS - on 2007-06-28 this was added to Linus's tree:
>> http://lkml.org/lkml/2007/6/18/234
>> CONFIG_MTD_OOPS
>> tristate "Log panic/oops to an MTD buffer"
> ...
>> early boot issues:
>> ----------------------
>> - CONFIG_EARLY_PRINTK
>> - early printk just defined a console - and is supported by:
>> alpha, blackfin, microblaze, mips, powerpc, sh, x86
>> - it's pretty trivial to support a memory based buffer - some
>> archs already support it.
>>
>> I think this only leaves Wolfgang's desire for memory buffers from the
>> bootloader to get (somehow) into the kernel's log buffer for syslog
>> processing...
>>
>> Anyone else agree?
>
> Almost. A couple of us also want memory for "flight data record" FDR data
> for doing continuous logging. This would, ideally, be either uncached or cached
> in such a way that data is guaranteed to be written to memory in the event of
> a watchdog timer-induced system reset.
can none of the existing mtd devices provide uncached ram ?
otherwise, create a mtd map that simply calls data flush after every
write. then the existing mtd stack can provide what you want.
-mike
^ permalink raw reply
* Re: Kernel crashing and log buffers...
From: Robin Getz @ 2009-06-26 17:59 UTC (permalink / raw)
To: David VomLehn
Cc: Mike Frysinger, Greg Ungerer, Russell King, Paul Mundt, Tim Bird,
Wolfgang Denk, Grant Erickson, linux-embedded
In-Reply-To: <20090626174219.GA19426@cuplxvomd02.corp.sa.net>
On Fri 26 Jun 2009 13:42, David VomLehn pondered:
> On Fri, Jun 26, 2009 at 10:39:50AM -0400, Robin Getz wrote:
> > OK - so after a bit more digging into things (and a poke by Mike) - I
> think
> > most of the stuff already exists.
> >
> > Normal run time issues:
> > --------------------------
> > - MTD_OOPS - on 2007-06-28 this was added to Linus's tree:
> > http://lkml.org/lkml/2007/6/18/234
> > CONFIG_MTD_OOPS
> > tristate "Log panic/oops to an MTD buffer"
> ...
> > early boot issues:
> > ----------------------
> > - CONFIG_EARLY_PRINTK
> > - early printk just defined a console - and is supported by:
> > alpha, blackfin, microblaze, mips, powerpc, sh, x86
> > - it's pretty trivial to support a memory based buffer - some
> > archs already support it.
> >
> > I think this only leaves Wolfgang's desire for memory buffers from the
>
> > bootloader to get (somehow) into the kernel's log buffer for syslog
> > processing...
> >
> > Anyone else agree?
>
> Almost. A couple of us also want memory for "flight data record" FDR
> data for doing continuous logging. This would, ideally, be either uncached
> or cached in such a way that data is guaranteed to be written to memory in
> the event of a watchdog timer-induced system reset.
Many of the early_printk implementations accept a "keep" option (which does
not remove the console/buffer - and keeps it around forever) - I'm adding
that to the Blackfin implementation as we speak - but it is already there on
sh.
I expect that we could add something to the MTD_OOPS, as to allways write as
well (console=ttyMTDx,log_level) or something like that if the backing needed
to be flash.
Would that satisfy things?
-Robin
^ permalink raw reply
* Re: [RFC] transcendent memory for Linux
From: Linus Walleij @ 2009-06-27 13:18 UTC (permalink / raw)
To: Dan Magenheimer
Cc: linux-kernel, xen-devel, npiggin, chris.mason, kurt.hackel,
dave.mccracken, Avi Kivity, jeremy, Rik van Riel, alan,
Rusty Russell, Martin Schwidefsky, akpm, Marcelo Tosatti,
Balbir Singh, tmem-devel, sunil.mushran, linux-mm, Himanshu Raj,
linux-embedded
In-Reply-To: <cd40cd91-66e9-469d-b079-3a899a3ccadb@default>
2009/6/20 Dan Magenheimer <dan.magenheimer@oracle.com>:
> We call this latter class "transcendent memory" and it
> provides an interesting opportunity to more efficiently
> utilize RAM in a virtualized environment. However this
> "memory but not really memory" may also have applications
> in NON-virtualized environments, such as hotplug-memory
> deletion, SSDs, and page cache compression. Others have
> suggested ideas such as allowing use of highmem memory
> without a highmem kernel, or use of spare video memory.
Here is what I consider may be a use case from the embedded
world: we have to save power as much as possible, so we need
to shut off entire banks of memory.
Currently people do things like put memory into self-refresh
and then sleep, but for long lapses of time you would
want to compress memory towards lower addresses and
turn as many banks as possible off.
So we have something like 4x16MB banks of RAM = 64MB RAM,
and the most necessary stuff easily fits in one of them.
If we can shut down 3x16MB we save 3 x power supply of the
RAMs.
However in embedded we don't have any swap, so we'd need
some call that would attempt to remove a memory by paging
out code and data that has been demand-paged in
from the FS but no dirty pages, these should instead be
moved down to memory which will be retained, and the
call should fail if we didn't succeed to migrate all
dirty pages.
Would this be possible with transcendent memory?
Yours,
Linus Walleij
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox