* Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader
From: Rob Landley @ 2009-12-27 7:17 UTC (permalink / raw)
To: celinux-dev; +Cc: Andy Green, Robert Schwebel, linux-embedded
In-Reply-To: <4B3146E9.3010101@warmcat.com>
On Tuesday 22 December 2009 16:23:37 Andy Green wrote:
> On 12/22/09 11:12, Somebody in the thread at some point said:
>
> Hi Robert -
>
> >> (Personally I used Fedora ARM port and RPM, but any distro and
> >> packagesystem like Debian workable on ARM would be fine).
> >
> > Until now, we are using the "build it yourself" approach with ptxdist,
> > basically because of these reasons:
> >
> > - If something goes wrong, we want to be able to fix it, which means
> > that we must be able to recompile everything. Having the source is no
> > value by itself, if you are not able to build it.
>
> Fedora provides a whole solution there, with the restriction it's
> designed for native build, not cross.
QEMU: it's not just for breakfast anymore.
Rob
--
Latency is more important than throughput. It's that simple. - Linus Torvalds
^ permalink raw reply
* Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader
From: Rob Landley @ 2009-12-27 7:27 UTC (permalink / raw)
To: celinux-dev; +Cc: Andy Green, Robert Schwebel, linux-embedded
In-Reply-To: <4B31E2F2.4010206@warmcat.com>
On Wednesday 23 December 2009 03:29:22 Andy Green wrote:
> On 12/23/09 08:56, Somebody in the thread at some point said:
>
> Hi -
>
> >> yourself because it's the buildroot mindset, that whole task
> >> disappears with a distro basis.
> >
> > If you don't step into for example toolchain problems or other crazy
> > things...
>
> Again this is buildroot thinking. The distro provides both the native
> and cross toolchains for you. You're going to want to use the same
> distro as you normally use on your box so the cross toolchain installs
> as a package there.
Because boards that use things like uClibc and busybox just aren't interesting
to you?
Please don't confuse "development environment" with "build environment". A
development environment has xterms and IDEs and visual diff tools and a web
browser and PDF viewer and so on. A build environment just compiles stuff to
produce executables. (Even on x86, your fire breathing SMP build server in the
back room isn't necessarily something you're going to VNC into and boot a
desktop on.)
I agree it's nice to have a build environment compatible with your deployment
environment, and distros certainly have their advantages, but you may not want
to actually _deploy_ 48 megabytes of /var/lib/apt from Ubuntu in an embedded
device.
Rob
--
Latency is more important than throughput. It's that simple. - Linus Torvalds
^ permalink raw reply
* Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader
From: Andy Green @ 2009-12-27 9:54 UTC (permalink / raw)
To: Rob Landley; +Cc: celinux-dev, Robert Schwebel, linux-embedded
In-Reply-To: <200912270117.18130.rob@landley.net>
On 12/27/09 07:17, Somebody in the thread at some point said:
Hi Rob -
>> Fedora provides a whole solution there, with the restriction it's
>> designed for native build, not cross.
>
> QEMU: it's not just for breakfast anymore.
That's right Qemu often requires lunch, teatime and supper too to build
anything :-)
Newer ARM platforms like Cortex8+ and the Marvell Sheevaplug will
outstrip emulated performance on a normal PC. There are 2GHz multi-core
ARMs coming as well apparently. So I took the view I should ignore Qemu
and get an early start on the true native build that will be the future
of "native build" as opposed to cross due to that.
The point of the distro is you just let them build the bulk of it, just
installing binary packages. You're only rebuilding the bits you are
changing for your application. For a lot of cases that's a few small
app packages that are mainly linking against stuff from the distro and
they're not too bad to do natively. (In addition my workflow is to edit
on a host PC and use scripts to teleport a source tree tarball to the
device where it's built as a package every time and installed together
with its -devel, so everything is always under package control).
-Andy
^ permalink raw reply
* Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader
From: Andy Green @ 2009-12-27 10:09 UTC (permalink / raw)
To: Rob Landley; +Cc: celinux-dev, Robert Schwebel, linux-embedded
In-Reply-To: <200912270127.35414.rob@landley.net>
On 12/27/09 07:27, Somebody in the thread at some point said:
Hi Rob -
>> Again this is buildroot thinking. The distro provides both the native
>> and cross toolchains for you. You're going to want to use the same
>> distro as you normally use on your box so the cross toolchain installs
>> as a package there.
>
> Because boards that use things like uClibc and busybox just aren't interesting
> to you?
I used them both before, but I can say with confidence if the platform
will take glibc and bash, most people will expect more complete, in that
sense, more reliable, performance from those.
It breaks down at stock distro init because it's painfully slow. But
otherwise there are real advantages in having the full-strength versions
of everything.
> Please don't confuse "development environment" with "build environment". A
Since I didn't use either term, I don't know why you think I'm confusing
them.
> development environment has xterms and IDEs and visual diff tools and a web
> browser and PDF viewer and so on. A build environment just compiles stuff to
> produce executables. (Even on x86, your fire breathing SMP build server in the
> back room isn't necessarily something you're going to VNC into and boot a
> desktop on.)
As I said in the other reply, my workflow is to edit a package's source
tree on a host (so you can use any editor on your host not just kate /
fish:// ) and by host script with scp and ssh get the current tree
package-built and installed on the device in one step.
So I hope it's clear there is solid separation between what you're
calling "development environment" and "build environment" to the point
they have nothing to do with each other except ssh-based script to get
stuff built.
> I agree it's nice to have a build environment compatible with your deployment
> environment, and distros certainly have their advantages, but you may not want
> to actually _deploy_ 48 megabytes of /var/lib/apt from Ubuntu in an embedded
> device.
I did say in the thread you want ARM11+ basis and you need 100-200MBytes
rootfs space to get the advantages of the distro basis. If you have
something weaker (even ARM9 since stock Fedora is ARMv5+ instruction set
by default) then you have to do things the old way and recook everything
yourself one way or another.
Even now there are plenty of suitable platforms that will work with it,
and over time they will only increase. Nothing seems to totally die out
(8051-based micros are still in the market) but each time something new
comes in at the top it grabs some of the market and the older ones shrink.
It boils down to the point that if you just treat the ARM11+ platforms
like the previous generation and stick fat bootloaders and buildroot
blobs on them, you are going to miss out on an epochal simplification
where embedded Linux largely becomes like desktop Linux in workflow,
quality and reliability of update mechanisms, and effort needed to bring
up a box / device.
-Andy
^ permalink raw reply
* [ANNOUNCE] barebox-2009.12.0 has been released
From: Robert Schwebel @ 2009-12-27 17:10 UTC (permalink / raw)
To: linux-embedded
Revision 2009.12.0 of the barebox bootloader (formerly known as
u-boot-v2) has been released on http://www.barebox.org
This is the initial release since it has been renamed to barebox.
Here are some of the feature highlights:
* POSIX based file API
* Shell commands like ls/cd/mkdir/echo/cat,...
* The environment is not a variable store any more, but a file store.
You can use it for proper shell scripting now.
* Real filesystem support: mounts ramdisk on /, devfs on /dev, plus
cramfs support (more to come)
* device/driver model
* clocksource support
* Kconfig and Kernel build system
* Simulation target
* device parameter support
* initcalls
* getopt
* editor
More can be found in the online documentation at
http://www.barebox.org/documentation/barebox-2009.12.0/
Shortlog since u-boot-v2.0.0-rc10 follows below.
Anand Gadiyar (1):
OMAP3430-SDP: Compile fixes
Eric Benard (12):
CPUIMX27 : update dram declaration
imx-nand : remove wrong part of commit ca326e88827e08e723046a21b18e74d41d15ace7
Eukrea CPUIMX27 : add SDRAM size choice
Eukrea CPUIMX27 : add NOR flash size configuration
Eukrea CPUIMX27 : add console on Quad UART support
Eukrea CPUIMX27 : Add MMU support
Eukrea CPUIMX27 : update documentation
Eukrea CPUIMX27 : update defconfig
Fix imx_nand_set_layout for i.MX27
nand_imx.c : add reset command
nand_imx.c : remove unused code
cpuimx27 : use flash bbt
Jean-Christophe PLAGNIOL-VILLARD (39):
uboot_default_env: fix out of tree build
cfi new: fix new disabling buffer support
xyzModem: fix digit init in xyzModem_stream_open
at91sam9: move to at91
at91: add cpu devices common api
at91sam9: switch to the same api as the kernel
kbuild: support arch/$ARCH/include for tags, cscope
kbuild: asm symlink support for arch/$ARCH/include
kbuild: add arch/$ARCH/include to search path
[ARM] move include/asm-arm to arch/arm/include/asm
arm: merge proc-armv/ptrace.h and ptrace.h in one file
[ARM] Move include/asm-arm/arch-* to arch/arm/*/include/mach
[BLACKFIN] move include/asm-blackfin to arch/blackfin/include/asm
[SANDBOX] move include/asm-sandbox to arch/arm/include/sandbox
[SANDBOX] Move include/asm-sandbox/arch-* to arch/sandbox/*/include/mach
[PPC] move include/asm-ppc to arch/ppc/include/asm
[PPC] Move include/asm-ppc/arch-* to arch/ppc/*/include/mach
[M68K] move include/asm-m68k to arch/m68k/include/asm
[M68K] Move include/asm-m68k/{arch,proc}-* to arch/m68k/*/include/{mach,proc}
pm9263: add nand support
add bool types support
driver: add dev_name inline
at91: add support for the at91sam9263ek board of Atmel
arm/kconfig: inverse the board selection
at91/kconfig: use similar kconfig presentation
imx/kconfig: inverse the board selection
s3c/kconfig: inverse the board selection
netx/kconfig: inverse the board selection
arm: move config ARCH_TEXT_BASE and BOARDINFO to mach Kconfig
at91: Atmel Ref board fix config MACH to match with mach-types
arm: introduce CPU CONFIG from linux
arch/arm/Makefile: cleanup
arm: remove non used cpu-y
arm: use -marm as default when it's possible
arm: add endian config support
arm: update cpu tunning
arm: add Debug Communications Channel serial driver support
arm: add configurable AEABI support
arm: generate mach-types.h instead of static version
Juergen Beisert (31):
Adapt pcm043 platform to boot from NAND
A9M2440: Just clean up
S3C24xx: Provide a generic way to detect memory size
A9M2440: Fix documentation
A9M2440: Generic code should support generic hardware
A9M2440: Use external GPIO settings to setup the SDRAM
A9M2440: Be able to configure things baseboard dependent
A9M2440: Addapting the default environment
ENV_HANDLING requires CRC32
Remove deadlock message
Enable 'saveenv' for non eraseable media
Adding some documentation
Adding IPU clock query functions for i.MX31 and i.MX35
Unify driver's menu also for video
let linker create a link map
Move variables to bss
Include the 'size_t' declaration
Remove unused code
Move private settings and configuration into the C source
Use the native 'void' instead of 'Void_t'
Remove confusing macros that hide the real functions names
After moving all declarations to the C source, do the same with the doc
Use nowadays function prototypes
Remove debug code
Remove win32 support
Use the global string functions
Combine memory initialization with the main memory functions.
Re-format the file
Fix some exported names
Some doxygen related fixes
documentation updates
Juergen nogit Beisert (2):
fb: Add a i.MX IPU framebuffer driver
mx35 3stack display support
Luca Ceresoli (1):
Turn on CONFIG_CMD_MEMORY in Beagle Board defconfig.
Marc Kleine-Budde (26):
fec_imx: remove trailing whitespace
clock.h: uses uint?_t types, so include types.h
nand: print size parameter as unsinged not as signed
nand: ground dev_add_bb_dev if nand is disabled
spi: dev->id is an int, not a string, fix debug output
speed-imx35: the gptclk is connected to the ipgclk
speed-imx35: add function imx_get_i2cclk
imx-regs: define IMX_FEC_BASE
mx25-3stack, pcm043: use IMX_FEC_BASE define
imx35-regs: add base address for I2C1
imx35-regs: add CGR1 helper values
i2c: new framework
i2c-imx: ported to u-boot-v2
mc13892: driver added
mc9sdz60: driver added
clock.h: use types.h not linux/types.h
clock.c: use USECOND and MSECOND
i.MX35 3stack: several enhancements
i2c-imx: print errors with dev_err
i2c-imx: fix clear IFF race condition
i2c-imx: fix low bitrate problem
i2c-imx: i2c_imx_xfer return with err if sub-transfer isn't successfull
arm/Makefile: don't set ABI unconditionally to "apcs-gnu"
Cleanup some more collateral damage from renaming
arm/lib: add eabi names of helper functions
arm/lib: add missing abi helper functions __aeabi_idivmod, __aeabi_uidivmod
Peter Korsgaard (4):
image.h: amd64 support
sandbox: common: fix device names for environment files (-e)
cramfs: probe(): fix cdev lookup
Documentation: update reference to sandbox environment
Robert P. J. Day (9):
ARM: Correct Kconfig typo, "optimzed" -> "optimized".
NAND: Free asprintf()-allocated space upon mtd device deletion.
commands: correct "CONFIG_SIMPLE_PARSER" to "CONFIG_SHELL_SIMPLE".
scripts: Delete non-barebox content from scripts/.
commands: Remove reference to non-existent CONFIG_CMD_I2C.
Remove/adjust erroneous references to CONFIG_MODULE.
MTD: Correct typo in preprocessor directive.
Remove obsolete comment referring to CFG_CMD_JFFS2.
Remove PPC support for IDE.
Robert Schwebel (2):
README: rewrite some u-boot leftovers
README: add release rules
Sanjeev Premi (2):
omap3evm: Add basic support for the board
omap3evm: Minimal default configuration
Sascha Hauer (225):
return is not a function
arm: implement optimized string functions
imx_nand: Skip bbt scan during initialization
armlinux: remove unused atags
armlinux.c: refactor
arm: make 'bootz' configurable
arm: Add bootu command
Add MMU support
fec imx27: Add MMU support
pcm038: Add MMU support
i.MX: Do not dump clocks on startup.
imx nand: Make use of optimized string functions
ehci: Restore state after td timeout
i.MX: Use more accurate decode pll function from kernel
usb: when unregistering usb devices, remove them from device list
use _stext instead of TEXT_BASE
imx serial: add mx25 support
add iomux definitions for mx25
Add MX25 support
nand_base: We have to ignore the -EUCLEAN error
add Freescale MX25 3stack board support
add mx25 3ds defconfig
MX25/MX35 Nand support
imx27: Add usb defines
pca100: Add MMU support
pca100: Add USB host support
usb ehci driver: Get rid of echi_alloc/free
usb ehci driver: Add MMU support
usb: remove unnecessary code
usb: remove unused fields from struct usb_device
pcm038: Add USB support
ethact command: If called without arguments print current eth device
nand bb: fix check when erasing a bb device
nand bb: fix removal of bb devices
devfs: add open counter
ls: beautify output
make copy_file() globally available
add xstrdup function
devfs: fix return value for lseek in partitions
mx25: implement clko command
mx25: remove duplicate function, fix perclk values
console: make locally used function static
hush: Only run list if it's not empty
hush: pass return code from exit command
rename dma macros
console: partly revert 84688dfdb4aecc8296b4fef9bc657335d7b9ade5
Add USB device support
register env as env0 and not as env
readline: Allow only ascii and printable characters
fs: replace broken dev_protect with protect_file function
boards: use recently introduced protect_file function
imx: replace imx specific gpio functions with generic ones
imx spi: Use gpio as chip selects
mc13783: fix spi mode, remove now unneeded double read
nand_imx_v2: use xzalloc for data buffers
nand_imx_v2: make nand_boot_test command more flexible
i.MX introduce cpu_is_*() macros
nand_imx_v2: abstract i.MX25 specific stuff properly
nand_imx_v2: Use a two staged scan process
nand_imx_v2: Use generic bbt scan function
pca100: remove imx_ prefix from gpio access functions
pca100: add missing USB bits
fec_imx: initialize buffers on open time
add USB otg pin definitions
protect command: ignore -ENOSYS
sandbox: use libc ioctl for os part
sandbox tap driver: initialize device name
sandbox: add a default environment
sandbox: update defconfig
usb ehci: Fix braindamaged pointer-deref-cast-magic
pcm038 defconfig update
nand bb: fix handling of two subsequent bad blocks
ls: fix stat failure, print error
fec_imx: do not typedef struct
tftp: do not call NetStartAgain()
net: remove NetStartAgain()
net: remove unused state NETLOOP_RESTART
net: split NetLoop in NetLoop and NetLoopInit
return is not a function
net: check prerequisites in NetLoopIinit()
bootp: remove unnecessary counter
net: remove switch/case in NetLoop()
ping.c: refactor
net: do not use netboot_common for rarp
make netboot_common a global function
net: refactor dhcp support
rename bootp.c to dhcp.c
net: cleanup header file
net: remove dead code
only the ping code is interested in the ping ip
net: remove unnecessary if
do not abuse the received packet to send back another packet
imxfb: make framebuffer base address configurable
remove sntp support. Has been broken for long enough
stringlist: Fix printing on smaller consoles
make locally used funtion static
tftp: remove default filename. We always have a filename given
remove unused variable NetBootFileSize
tftp: remove unnecessary gateway information
tftp: consolidate different puts to printf
remove unused code
tftp: remove volatile from variables
tftp: remove never reached state STATE_TOO_LARGE
tftp: remove never reached state STATE_BAD_MAGIC
tftp: reorder to get rid of static function declarations
tftp: coding style
tftp: use debug macro
ping: do not return -1 in command
ping: shorten usage info
nfs: no default filename, we always have a filename
pass filename to nfs and tftp start functions
nfs: allow only serverip
remove global variable BootFile
consolidate command calling in execute_command
remove unused typedef
commands: add defines for command errors/success
commands: return COMMAND_ERROR_USAGE
call getopt_reset only once
cat: return error if file is not readable
remove obsolete and broken command 'splash'
remove unused header file
remove unused directory tools
miiphy: Add missing lseek function
printenv: do not return negative values in a command
commands: remove maxargs
hush: remove include/hush.h
nand_imx: fix largepage layout
nand_imx: merge send_read_page and send_prog_page
mxc_nand: introduce mxc_do_addr_cycle
mxc_nand: use buffers
mxc_nand: remove debug param
mxc_nand: simplify command processing
mxc nand: modify send_page to send all pages, not only one
mxc_nand: remove unused defines
mxc_nand: determine page and block size depending on NFMS bit
imx_nand: add booting from 2k NAND support
imx_nand: make oob read work on 512b page nands
mxc_nand: Make main/spare areas runtime configurable
mxc_nand: Add NFC V2 support
imx_nand: allow flash based bbt
IMX_NAND_BASE -> IMX_NFC_BASE
mxc_nand: set proper spas / ecc size
mxc_nand: add nand boot support for i.MX35
imx_nand: add i.MX25/i.MX35 to Kconfig
Merge branch 'for-sascha' of git://uboot.jcrosoft.org/u-boot-v2 into from-jean
a9m2410dev: compile fixes
i.MX21: fix compiler warning
i.MX1: do not try to print non existent reg CID
list: remove duplicated list.h
at91sam9260ek: Reset ethernet phy during init
i.MX Nand: Set correct datawidth/pagesize in CCM module
i.MX: remove obsolete V2 Nand driver
NAND: Allow to read oob data
i.MX flash header: make it work on i.MX35
nand: refuse to write data if not beginning on a page boundary
nand: do not write empty pages. Needed for writing UBI images
PCA100: initialize pll in assembler code
imxfb: show only for relevant i.MXs
fb: Add FB_SYNC_ defines from kernel
i.MX: split out iomux-v1 support
complete i.MX GPIO support
introduce GENERIC_GPIO label
command line gpio support
pcm043: add display support
pcm043: Add MMU support
omap: Add missing include
beagle board: Compile fixes
omap: Use correct label in arch/arm/Makefile
i.MX31: Fix CCM_UPCTL register offset
remove unused include files
pcm038: Use a flash based bbt
crc: Do not lseek if not necessary (bb devs do not allow lseek)
nand bb: Be sure that fd is at the beginning when erasing
i.MX Nand: fix bit clearing
pcm038: update defconfig
Merge branch 'for-sha-i2c-imx' of git://git.pengutronix.de/git/mkl/u-boot-v2
rename U-Boot-v2 project to barebox
Cleanup colleteral damage from renaming
Merge branch 'for-sha-collateral' of git://git.pengutronix.de/git/mkl/barebox
Merge branch 'for-sha-eabi-fixes' of git://git.pengutronix.de/git/mkl/barebox
ppc bitops: Fix compiler warning
i.MX35 3stack: select I2C drivers
fix arch_execute prototype
i.MX25 3stack: fixup flash header to new names/layout
ipe337: add missing include
scb9328: setup chipselect the correct way
read_key: Use ARRAY_SIZE instead of hardcoded value
Remove unnecessary casts in key defines
replace // style comments
netx: enable second ethernet device
pcm037: Add missing include
imx ipu fb: Make fb_videomode const
i.MX nand: mx21 has a v1 type controller
vsprintf: Only define PAGE_SIZE if not already defined
i2c: Add a no-op i2c_register_board_info if i2c is disabled
sandbox: update defconfig
pcm030: Update defconfig
mx21ads: update defconfig
pcm037: update defconfig
pca100: update defconfig
mx27ads: update defconfig
ipe337: update defconfig
scb9328: update defconfig
pcm043: update defconfig
pm9263: update defconfig
netx: update defconfig
mmccpu: update defconfig
eukrea cpuimx27: update defconfig
at91sam9260ek: update defconfig
at91sam9263ek: update defconfig
Digi a9m2410: update defconfig
Digi a9m2440: update defconfig
i.MX25 3stack: update defconfig
i.MX35 3stack: update defconfig
sdp3430: update defconfig
beagle board: update defconfig
omap3_evm: update defconfig
Makefile: clean barebox.ldr
Fix barebox Version number
commands/nand.c: Fix memory hole
pcm030: remove dead code
commands/nand.c: Fix memory hole
Merge branch 'pu'
barebox-2009.12.01
barebox-2009.12.0
Uwe Kleine-König (1):
Documentation/porting.txt: fix some spelling mistakes ...
rsc
--
Pengutronix e.K. | |
Industrial Linux Solutions | http://www.pengutronix.de/ |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0 |
Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917-5555 |
^ permalink raw reply
* Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader
From: Rob Landley @ 2009-12-27 23:15 UTC (permalink / raw)
To: Andy Green; +Cc: celinux-dev, Robert Schwebel, linux-embedded
In-Reply-To: <4B372EEB.2000902@warmcat.com>
On Sunday 27 December 2009 03:54:51 Andy Green wrote:
> On 12/27/09 07:17, Somebody in the thread at some point said:
>
> Hi Rob -
Before replying, I note that Mark Miller and I gave a presentation entitled
"Developing for non-x86 targets using QEMU" Ohio LinuxFest in October.
http://impactlinux.com/fwl/downloads/presentation.pdf
http://impactlinux.com/fwl/presentation.html
There's even horribly mangled audio of our rushed 1 hour presentation. (The
slides are for a day-long course and we had 55 minutes to give a talk, so we
skimmed like mad. Unfortunately, the netbook they used for audio recording
had the mother of all latency spikes whenever the cache flush did an actual
flash erase and write, so there are regular audio dropouts the whole way
through.) Still, it's somewhere under:
http://www.archive.org/details/OhioLinuxfest2009
I've also spent the last few years developing a project that produces native
built environments for various QEMU targets and documents how to bootstrap
various distros under them:
http://impactlinux.com/fwl
So I do have some firsthand experience here.
> >> Fedora provides a whole solution there, with the restriction it's
> >> designed for native build, not cross.
> >
> > QEMU: it's not just for breakfast anymore.
>
> That's right Qemu often requires lunch, teatime and supper too to build
> anything :-)
Which is why you hook it up to distcc so it can call out to the cross
compiler, which speeds up the build and lets you take advantage of SMP.
(Pages 217-226 of the above PDF.)
That's also why my FWL project uses a statically linked version of busybox,
because the static linking avoids the extra page retranslations on each exec
and thus sped up the ./configure stage by 20%. (Pages 235-236 of PDF.)
There's more things you can do to speed it up if you want to go down that
rabbit hole (which the presentation does), and there's more work being done in
qemu. (TCG was originally a performance hit but has improved since, although
it varies widely by platform and is its own rabbit hole. Also switching to
gigabit NIC emulation with jumbo frames helped distcc a lot.)
But in general, Moore's Law says that qemu on current PC hardware is about the
speed of current PC hardware seven years ago. (And obviously nobody ever
built anything before 2003. :)
> Newer ARM platforms like Cortex8+ and the Marvell Sheevaplug will
> outstrip emulated performance on a normal PC. There are 2GHz multi-core
> ARMs coming as well apparently. So I took the view I should ignore Qemu
> and get an early start on the true native build that will be the future
> of "native build" as opposed to cross due to that.
Pages 24-34 of the above PDF go over this. The first two pages are on the
advantages of native compiling on real hardware, the next eight pages are on
the disadvantages. It can certainly be made to work, especially in a large
corporation willing to spend a lot of money on hardware as a _prerequisite_ to
choosing a deployment platform.
For hobbyists, small businesses, and open source developers in general, there
are significant advantages to emulation. (Page 208 comes to mind.) And if you
_are_ going to throw money at hardware, x86-64 continues to have better
price/performance ratio, which was always its thing.
> The point of the distro is you just let them build the bulk of it, just
> installing binary packages. You're only rebuilding the bits you are
> changing for your application.
Pages 68-71. If your definition of embedded development is using off the shelf
hardware and installing prebuilt binary packages into it, life becomes a lot
easier, sure.
> For a lot of cases that's a few small
> app packages that are mainly linking against stuff from the distro and
> they're not too bad to do natively.
Pages 78-84
> (In addition my workflow is to edit
> on a host PC
Pages 178-180
> and use scripts to teleport a source tree tarball to the
> device where it's built as a package every time and installed together
> with its -devel,
Pages 181-202
> so everything is always under package control).
Package control or source control? (Different page ranges...)
> -Andy
Rob
--
Latency is more important than throughput. It's that simple. - Linus Torvalds
^ permalink raw reply
* Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader
From: Rob Landley @ 2009-12-28 0:21 UTC (permalink / raw)
To: Andy Green; +Cc: celinux-dev, Robert Schwebel, linux-embedded
In-Reply-To: <4B373253.9050006@warmcat.com>
On Sunday 27 December 2009 04:09:23 Andy Green wrote:
> > I agree it's nice to have a build environment compatible with your
> > deployment environment, and distros certainly have their advantages, but
> > you may not want to actually _deploy_ 48 megabytes of /var/lib/apt from
> > Ubuntu in an embedded device.
>
> I did say in the thread you want ARM11+ basis and you need 100-200MBytes
> rootfs space to get the advantages of the distro basis. If you have
> something weaker (even ARM9 since stock Fedora is ARMv5+ instruction set
> by default) then you have to do things the old way and recook everything
> yourself one way or another.
I started programming on a commodore 64. By modern standards, that system is
so far down into "embedded" territory it's barely a computer. And yet people
did development on it.
http://landley.net/history/catfire/wheniwasaboy.mp3
That said, you can follow Moore's Law in two directions: either it makes stuff
twice as powerful every 18 months or it makes the same amount of power half
the price.
What really interests me is disposable computing. Once upon a time swiss
watches were these amazingly valuable things (which Rolex and friends try to
cling to even today by gold-plating the suckers), but these days you can get a
cheap little LCD clock as a happy meal toy. The cheapest crappiest machines
capable of running Linux are 32-bit boxes with 4 gigs of ram, which were high-
end server systems circa 1987 that cost maybe about $5k (adjusted for inflation
anyway). These days, the cheapest low-end Linux boxes (of the "repurposed
router" variety) are what, about $35 new? Moore's Law says that 21 years is
14 doublings, which would be 1) $2500, 2) $1250, 3) $635, 4) $312, 5) $156, 6)
$87, 7) $39, 8) $19, 9) $9.76, 10) $4.88, 11) $2.44, 12) $1.22, 13) $0.61, 14)
$0.31.
So in 2009 that $5000 worth of computing power should actually cost about 30
cents, and should _be_ disposable. In reality, the CPU in that router is
clocked 20 times faster than a Compaq deskpro 386, you get 4 to 8 times the
memory, they added networking hardware, and so on. And there are fixed costs
for a case and power supply that don't care about Moore's Law, plus up-front
costs to any design that need to be amortized over a large production run to
become cheap, and so on.
And the real outstanding research problems include ergonomic UI issues for
tiny portable devices, batteries wearing out after too many cycles, and the
fact that making "disposable" devices out of materials like cadmium is dubious
from environmental standpoint. Oh, and of course there was the decade or two
companies like Intel lost going up a blind alley by bolting giant heat sinks
and fans onto abominations like the Pentium 4 and Itanic. They didn't care
about power consumption at all until fairly recently, and are still backing
out of that cul-de-sac even today...
Still, I forsee a day when cereal boxes have a display on the front that
changes every 30 seconds to attract passerby, driven by the same amount of
circuitry and battery that makes the "free toy inside" blink an LED today. (I
don't know what else that sort of thing will be used for, any more than people
predicted checking twitter from the iPhone,)
This I'm reluctant to abandond the low-end and say "oh we have more power now,
only machines with X and Y are interesting". The mainframe, minicomputer, and
micro (PC) guys each said that, and today the old PC form factor's getting
kicked into the server space by the iPhone and such. I want to follow Moore's
Law down into disruptive technology territory and find _out_ what it does.
> Even now there are plenty of suitable platforms that will work with it,
> and over time they will only increase.
You must be this tall to ride the computer.
> Nothing seems to totally die out
> (8051-based micros are still in the market)
Mainframes are still on the market too.
> but each time something new
> comes in at the top it grabs some of the market and the older ones shrink.
>
> It boils down to the point that if you just treat the ARM11+ platforms
> like the previous generation and stick fat bootloaders and buildroot
> blobs on them, you are going to miss out on an epochal simplification
> where embedded Linux largely becomes like desktop Linux in workflow,
> quality and reliability of update mechanisms, and effort needed to bring
> up a box / device.
New computing niches will develop new usage patterns. The iPhone is currently
doing this, and is unlikely to be the last cycle.
They'll also grow more powerful and expand into old niches the way "blade
servers" are constructed from laptop components and used for batch processing
today, but I personally find that less interesting.
> -Andy
Rob
--
Latency is more important than throughput. It's that simple. - Linus Torvalds
^ permalink raw reply
* Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader
From: Andy Green @ 2009-12-28 10:27 UTC (permalink / raw)
To: Rob Landley; +Cc: celinux-dev, Robert Schwebel, linux-embedded
In-Reply-To: <200912271715.54249.rob@landley.net>
On 12/27/09 23:15, Somebody in the thread at some point said:
Hi Rob -
> I've also spent the last few years developing a project that produces native
> built environments for various QEMU targets and documents how to bootstrap
> various distros under them:
>
> http://impactlinux.com/fwl
>
> So I do have some firsthand experience here.
I wasn't suggesting you don't have firsthand experience all over the
place, eg, busybox, I know it. What I do suggest in the principles I
have been bothering your Christmas with here can really pay off and are
in the opposite direction to Qemu, bloated bootloaders, and buildroot
style distro production.
>>>> Fedora provides a whole solution there, with the restriction it's
>>>> designed for native build, not cross.
>>>
>>> QEMU: it's not just for breakfast anymore.
>>
>> That's right Qemu often requires lunch, teatime and supper too to build
>> anything :-)
>
> Which is why you hook it up to distcc so it can call out to the cross
> compiler, which speeds up the build and lets you take advantage of SMP.
> (Pages 217-226 of the above PDF.)
Or you just do it native and you don't care about extreme rebuild case
because you're using a distro that built it already.
There's a critical advantage to build in specifically the execution
environment you do cover in three words on slide 14 of your PDF I expand
on at the end of the mail.
> That's also why my FWL project uses a statically linked version of busybox,
> because the static linking avoids the extra page retranslations on each exec
> and thus sped up the ./configure stage by 20%. (Pages 235-236 of PDF.)
>
> There's more things you can do to speed it up if you want to go down that
> rabbit hole (which the presentation does), and there's more work being done in
> qemu. (TCG was originally a performance hit but has improved since, although
> it varies widely by platform and is its own rabbit hole. Also switching to
> gigabit NIC emulation with jumbo frames helped distcc a lot.)
Just saying people don't do distribution rebuilds for their desktop or
server boxes unless they're Gentoo believers. So the need to consider
this heavy duty stuff only exists at the distro. In Fedora case, guys
at Marvell are running it and have access to lots of fast physical
hardware they hooked up to Koji (you touch on this in your PDF). I
don't think it's like they were waiting to hear about Qemu now they're
gonna drop that and move to emulation. They're in the business of
making fast ARM chips and they're going to outpace Qemu.
> But in general, Moore's Law says that qemu on current PC hardware is about the
> speed of current PC hardware seven years ago. (And obviously nobody ever
> built anything before 2003. :)
The speed surely depends on the architecture being emulated. I did try
Qemu on ARM on a nice desktop box here and it seemed unworkable to me.
I am perfectly happy to trust Fedora to take care of disribution
rebuilds for me.
>> Newer ARM platforms like Cortex8+ and the Marvell Sheevaplug will
>> outstrip emulated performance on a normal PC. There are 2GHz multi-core
>> ARMs coming as well apparently. So I took the view I should ignore Qemu
>> and get an early start on the true native build that will be the future
>> of "native build" as opposed to cross due to that.
>
> Pages 24-34 of the above PDF go over this. The first two pages are on the
> advantages of native compiling on real hardware, the next eight pages are on
> the disadvantages. It can certainly be made to work, especially in a large
> corporation willing to spend a lot of money on hardware as a _prerequisite_ to
> choosing a deployment platform.
This is what I have been calling "buildroot thinking" again. What do
you think you will be rebuilding exactly out of Fedora when you put it
on an ARM11+ system? Typically, it's going to be zero packages, nothing
at all. What you will need to build typically is this:
- bootloader (using Fedora's cross compiler package)
- kernel (using Fedora's cross compiler package)
- your own apps and libs (native build on device against libs there)
In addition you will need to take evasive action around the boot flow,
but to get started that's just init=/bin/my-init.sh
> For hobbyists, small businesses, and open source developers in general, there
> are significant advantages to emulation. (Page 208 comes to mind.) And if you
> _are_ going to throw money at hardware, x86-64 continues to have better
> price/performance ratio, which was always its thing.
The point is you will definitely be throwing lots of money at hardware
in this game, your ARM device platform. Since all products start with
zero users, and we get to be the first, we had best spend all our time
gaining experience with the hardware we plan to ship.
If there are problems with upstream unless it's something really in the
plumbing like cache behaviour which is a kernel issue, it can normally
be shown in the code easy enough.
> Pages 68-71. If your definition of embedded development is using off the shelf
> hardware and installing prebuilt binary packages into it, life becomes a lot
> easier, sure.
Well that's exactly my point, except the shelf the hardware came off is
the one with your own prototype ARM devices. No Qemu, no mass rebuilds,
no funky "it's embedded so we have to whip ourselves with nettles".
If life's a lot easier blowing away buildroot style and the
infrastructure for that, why is anyone considering doing it the hard way
on devices that can handle the faster, more reliable and well understood
packaged distro rootfs basis? It really does become like working on a
desktop Linux box with Fedora.
>> For a lot of cases that's a few small
>> app packages that are mainly linking against stuff from the distro and
>> they're not too bad to do natively.
>
> Pages 78-84
...
I have been actually living these principles the past couple of years,
consider this a report from a specific frontline rather than a review of
possibilities.
> Package control or source control? (Different page ranges...)
Package control is also source control, that is a whole other advantage
of using a packaged distro, license compliance becomes real easy. A
Source RPM is generated with the binary ones, you just need to get the
SRPMs of the packageset you are shipping (which you can get a list of
with rpm -qa) and publish it for GPL compliance. You can also grep the
packages for licence and automate other tasks that way (such as
excluding source).
Anyway here's the reason promised earlier that building out of the
execution environment will bite you, you need to keep the "-devel" stuff
in sync: includes and libraries. You can easily install a new version
of something with a -devel on your actual device and still be building
against the old version of it on this Qemu or Cross build environment.
Building on the device you are running on, and using packaged builds for
your own apps removes this issue completely.
This thread was meant to be about merits of Qi, it's kinda gone off into
embedded with distro rootfs because the philosophy is related. In both
cases burden on the developer is intended to be removed and effort
simplified to get the job done more reliably and quicker.
My experience has been this kind of talk makes people feel jumpy when
they're invested in the other philosophies. I hope what has been
discussed has left people at least wondering if there might be value in
disallowing the bootloader from doing anything but get you quickly into
Linux, so the development and support effort can focus just in Linux and
the bootloader was just something you spent a day tuning months ago and
never touched again (especially not in the factory).
-Andy
^ permalink raw reply
* Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader
From: Andy Green @ 2009-12-28 11:33 UTC (permalink / raw)
To: Rob Landley; +Cc: celinux-dev, Robert Schwebel, linux-embedded
In-Reply-To: <200912271821.38258.rob@landley.net>
On 12/28/09 00:21, Somebody in the thread at some point said:
Hi -
> I started programming on a commodore 64. By modern standards, that system is
> so far down into "embedded" territory it's barely a computer. And yet people
> did development on it.
My dear Rob I got started on a PET, I can understand your POV :-)
> kicked into the server space by the iPhone and such. I want to follow Moore's
> Law down into disruptive technology territory and find _out_ what it does.
The big challenge I see is delivering highly complex Linux devices with
insufficient developers in a way that won't disappear up its own ass and
kill the project / customer with delays or failure to perform.
It's nice if the device is efficient with every cycle, but that is a
geek preoccupation. Many customers in suits will tell to spend an extra
$1 to overcome it by hardware and gain back $5 from accelerated time to
market, so long as they can depend on the high quality of the software
basis to not kill that logic by delays.
-Andy
^ permalink raw reply
* How to store kernel pranic/oops
From: Marco Stornelli @ 2009-12-28 11:43 UTC (permalink / raw)
To: CeLinux, Linux Embedded
Hi,
I know the open project proposal 2010 is closed, but it's just to start
a discussion. It would be nice to save oops/panic automatically in a
structure/file in ram. At the moment there are two way to save
information: mtdoops (save the information in flash), with kdump/kexec
(we can extract the dmesg from vmcore file). With these tools there are
some drawbacks:
1) There are embedded systems without a flash where to save the information;
2) we could consider this kind of logs too volatile for a flash, I mean
there's no reason to store it for a long time, it's important to recover
and read them as soon as possible, at next boot for example.
3) kdump requires a lot of ram and resources for embedded systems.
4) kexec is available only for some archs.
It would be nice to have a "ramoops" to save in a circular buffer in a
persistent ram this kind of information. Any comments? Is there already
anything similar out-of-tree?
Marco
^ permalink raw reply
* Re: How to store kernel pranic/oops
From: David Woodhouse @ 2009-12-28 12:12 UTC (permalink / raw)
To: Marco Stornelli; +Cc: CeLinux, Linux Embedded
In-Reply-To: <4B3899DB.1020308@gmail.com>
On Mon, 2009-12-28 at 12:43 +0100, Marco Stornelli wrote:
> It would be nice to have a "ramoops" to save in a circular buffer in a
> persistent ram this kind of information. Any comments? Is there already
> anything similar out-of-tree?
Can't it be done with what's in the tree already? Just create an MTD
device using phram or something else, then point mtdoops at it.
--
David Woodhouse Open Source Technology Centre
David.Woodhouse@intel.com Intel Corporation
^ permalink raw reply
* Re: CELF Project Proposal - Feasibility analisys of Android introduction in a completely tested industrial device
From: Raffaele Recalcati @ 2009-12-28 16:26 UTC (permalink / raw)
To: Tim Bird; +Cc: CE Linux Developers List, linux-embedded
In-Reply-To: <4B30250E.5010505@am.sony.com>
Raffaele Recalcati wrote:
> 2009/12/18 Raffaele Recalcati <lamiaposta71@gmail.com>:
>> Summary: Feasibility analisys of Android introduction in a completely
>> tested industrial device.
>>
>> Description: By now Android has been ported to 600Mhz Cortex A8 cpu or similar.
>> The declared Android requirements are instead lower, about 200Mhz Arm9
>> cpu with 100Mhz Ram bus.
>> So I think the growing interest in this O.S. lacks some porting to
>> less powerful cpus.
>> The reasons to do this porting are commercial because of Google market
>> power, but are also technnical, because Android debugging environment
>> is very nice for not embedded developers.
>> This could help the diffusion of opensource embedded Linux.
> This is interesting. Can you let me know if the focus of this work
> is to experiment with the lower bounds of Android scalability, or
> whether the focus is on Android use in industrial devices?
>
> If the latter, than it would be good to hear more about what might
> be needed to extend (or reduce :-) ) Android to fit this market.
>
> I'll add a proposal for this, but I'd like to hear more to clarify
> the proposal.
>
> Thanks,
> -- Tim
>
> =============================
> Tim Bird
> Architecture Group Chair, CE Linux Forum
> Senior Staff Engineer, Sony Corporation of America
> =============================
>
>
Looking at
http://git.kernel.org/?p=linux%2Fkernel%2Fgit%2Ftorvalds%2Flinux-2.6.git&a=search&h=HEAD&st=commit&s=android
my idea seems to be ... not very good at all !!
What do you think?
I'm not a kernel mainline developer so I don't understand the real meaning.
Below there is my new proposal, but before accepting it please hear
this new idea, that could be better and possible to be used also in
2.6.33 kernel:
"Adding to Maemo or Debian or Gentoo embedded systems a debugging
environment similar to Android: I mean the possibility to connect to a
single debugging "server" and, in a graphical very easy (Eclipse or QT
Creator) way, connecting the gdb to the desired process, or, see
rootfs, or tracing cpu usage, ..."
What I really like of Android is the fantastique environment, with the
emulator that can do step by step debugging....
---------------------------------------------------------
Summary: Feasibility analisys of Android introduction in a completely
tested industrial device.
Description: Android is a new important graphic interface and a java
virtual machine optimized for
embedded devices.
The Android breaking feature in respect of traditional GNU Linux
embedded operating systems has
discouraged the possibility to enhance an already existing GNU Linux
embedded industrial device
with at least the Android's graphic interface.
By now Android has been fully ported to Cortex A8 cpus starting new
mobile phone projects from
scratch.
The reasons to do this work are commercial because of Google market
power, but are also technical,
because Android debugging environment is very nice for not embedded developers.
This could help the diffusion of opensource embedded GNU Linux in a
wider point of view.
The work should be maybe against the 2.6.31 kernel.
The idea is to preserve industrial tested development but adding
Android graphical interface and
debugging features.
The cpu dependency of the porting will be as small as possible.
The most important kernel interfaces will be investigated, at least
framebuffer for tft lcd,
touchscreen, audio ac97 interface, usb host for usb pen.
Related work:
* Android Porting - http://www.kandroid.org/<wbr></wbr>android_pdk/
* Android Pxa270 - http://android-pxa270.sourceforge.net/
Scope:
This should take more than 1 month for feasibility analysis.
------------------------------------------------------
--
www.opensurf.it
^ permalink raw reply
* Re: Re: How to store kernel panic/oops
From: Marco Stornelli @ 2009-12-28 18:03 UTC (permalink / raw)
To: dwmw2; +Cc: CeLinux, Linux Embedded
David Woodhouse wrote:
> Can't it be done with what's in the tree already? Just create an MTD
> device using phram or something else, then point mtdoops at it
Yes of course, if possible we shouldn't reinvent the wheel but I
wondered if there was something more specific. To add mtdoops (more or
less 1k) we have to add mtd subsys (more or less 14k) to the kernel to
achieve this and it's all overhead.
Marco
^ permalink raw reply
* Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader
From: Peter Korsgaard @ 2009-12-28 19:57 UTC (permalink / raw)
To: Andy Green; +Cc: Rob Landley, celinux-dev, Robert Schwebel, linux-embedded
In-Reply-To: <4B3887F8.5020509@warmcat.com>
>>>>> "Andy" == Andy Green <andy@warmcat.com> writes:
Hi,
Andy> This is what I have been calling "buildroot thinking" again.
Thanks for an interesting discussion and all your free buildroot
advertising ;)
(buildroot maintainer)
Andy> This thread was meant to be about merits of Qi, it's kinda gone
Andy> off into embedded with distro rootfs because the philosophy is
Andy> related. In both cases burden on the developer is intended to be
Andy> removed and effort simplified to get the job done more reliably
Andy> and quicker.
Sure, if your embedded device is very PC like (size/functionality) that
makes sense.
--
Bye, Peter Korsgaard
^ permalink raw reply
* Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader
From: Andy Green @ 2009-12-28 20:20 UTC (permalink / raw)
To: Peter Korsgaard; +Cc: Rob Landley, celinux-dev, Robert Schwebel, linux-embedded
In-Reply-To: <87ljgmvpkt.fsf@macbook.be.48ers.dk>
On 12/28/09 19:57, Somebody in the thread at some point said:
Hi Peter -
> Thanks for an interesting discussion and all your free buildroot
> advertising ;)
>
> (buildroot maintainer)
Hey you are welcome... in the situations where you can't play the distro
game then buildroot is the lifesaver. Same as U-Boot I have experienced
in detail the amount of work involved in what buildroot delivers and it
has to be respected for what it does.
I'm really saying there's a region where buildroot is the right answer
and a threshold beyond which a "real distro" is the right answer, we
should not just keep on doing what we have being doing.
(But you want to be careful you don't grow into OpenEmbedded's build
system. For me it failed partway through a 1000+ package build on
trying to build *host* dbus libs for me (yes, host dbus).
I just wanted to package "hello world" on Openmoko. They did fix it in
the end so it would build host dbus OK to be fair. But I felt that was
a very long way from the point.)
> Andy> This thread was meant to be about merits of Qi, it's kinda gone
> Andy> off into embedded with distro rootfs because the philosophy is
> Andy> related. In both cases burden on the developer is intended to be
> Andy> removed and effort simplified to get the job done more reliably
> Andy> and quicker.
>
> Sure, if your embedded device is very PC like (size/functionality) that
> makes sense.
Thanks. As written the threshold seems to be at ARM11+ and SD
particularly making it into enough of a "PC" you don't have to worry
about bash vs ash or where a few 100MB of storage is going to come from.
-Andy
^ permalink raw reply
* Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader
From: Rob Landley @ 2009-12-29 4:25 UTC (permalink / raw)
To: Andy Green; +Cc: celinux-dev, Robert Schwebel, linux-embedded
In-Reply-To: <4B3887F8.5020509@warmcat.com>
On Monday 28 December 2009 04:27:04 Andy Green wrote:
> I wasn't suggesting you don't have firsthand experience all over the
> place, eg, busybox, I know it. What I do suggest in the principles I
> have been bothering your Christmas with here can really pay off and are
> in the opposite direction to Qemu, bloated bootloaders, and buildroot
> style distro production.
In the PDF I spent a dozen pages pointing out that I think buildroot is silly
and how it diverged from its original goals. Why do you bring it up here?
Building natively is nice. I think cross compiling extensive amounts of stuff
is generally counterproductive, and you don't seem to disagree with that.
(However, if there weren't cases where cross compiling _does_ make sense I
wouldn't bother building the second stage cross compilers in my project and
packaging them up for other people to use. "I don't recommend it" != "people
who do this are stupid".)
You're conflating a bunch of issues. You can boot a full distro under qemu or
under real hardware, so "use a native distro" is orthogonal to "develop under
emulation vs develop on target hardware". And building a hand-rolled system
or assembling prebuilt distro bits can both be done natively (under either
kind of "native"), so that's orthogonal too.
I'm really not understanding what argument you're making. "The one specific
configuration I've chosen to use is better than any other possible
configuration"... Possibly there's a "for your needs" attached somewhere that
you're missing? You seem to be confusing "that's not an interesting option
for me" with "that's not an interesting option". You're conflating a bunch of
different issues into a One True Way, which is sad.
> >>>> Fedora provides a whole solution there, with the restriction it's
> >>>> designed for native build, not cross.
> >>>
> >>> QEMU: it's not just for breakfast anymore.
> >>
> >> That's right Qemu often requires lunch, teatime and supper too to build
> >> anything :-)
> >
> > Which is why you hook it up to distcc so it can call out to the cross
> > compiler, which speeds up the build and lets you take advantage of SMP.
> > (Pages 217-226 of the above PDF.)
>
> Or you just do it native and you don't care about extreme rebuild case
> because you're using a distro that built it already.
I find reproducibility to be kind of nice. (Remember when ubuntu dropped
Powerpc support? Or how Red hat Enterprise is finally dropping itanic? Even
today can you build a full distro natively on new hardware like microblaze or
blackfin?)
I also don't see any difference between your "let the distro handle everything"
with "the vendor supplies a BSP, why would you need anything else?" That's
certainly a point of view, and for you it may be fine. Anything can be
outsourced. But "I don't have to do it, therefore nobody does" is a
questionable position.
> > There's more things you can do to speed it up if you want to go down that
> > rabbit hole (which the presentation does), and there's more work being
> > done in qemu. (TCG was originally a performance hit but has improved
> > since, although it varies widely by platform and is its own rabbit hole.
> > Also switching to gigabit NIC emulation with jumbo frames helped distcc a
> > lot.)
>
> Just saying people don't do distribution rebuilds for their desktop or
> server boxes unless they're Gentoo believers.
Linux From Scratch does not exist? My friend Mark (who inherited a "Slackware
for PowerPC" cross-build system) is in reality unemployed? Nobody ever
actually works _on_ Fedora and needs to do a development build?
It's interesting to see someone who can imply these things with a straight
face on an embedded development list.
> So the need to consider
> this heavy duty stuff only exists at the distro. In Fedora case, guys
> at Marvell are running it and have access to lots of fast physical
> hardware they hooked up to Koji (you touch on this in your PDF). I
> don't think it's like they were waiting to hear about Qemu now they're
> gonna drop that and move to emulation.
You also seem to be unaware that QEMU was only started in 2003 (first commit to
the repository, Feb 18, 2003), and didn't even work on x86 all that well until
the end of 2005. Decent arm support started showing up in 2006 but it was
darn wonky, and the switch from dyngen to TCG didn't happen until Feb 1, 2008,
before which you couldn't even build qemu with gcc 4.x:
http://landley.net/qemu/2008-01-29.html#Feb_1,_2008_-_TCG
Therefore, they didn't "drop" support, QEMU wasn't ready for prime time back
in 2006 when my then-boss (Manas Saksena) quit Timesys to go launch Fedora for
Arm. (The boss I had after that, David Mandala, is now in charge of Ubuntu
Mobile. Timesys had some darn impressive alumni, pity the company screwed up
so badly none of us stayed around. I myself stayed through four bosses, three
CEOs, and outlasted 80% of my fellow engineers...)
By the way:
http://fedoraproject.org/wiki/Architectures/ARM/HowToQemu
You continue to conflate orthogonal issues.
There is a REASON QEMU project hasn't shipped its 1.0 release yet. Dismissing
it as irrelevant to the future based on your experiences with it years ago is
kind of hilarious, really. (And dismissing emulation in general as of
interest to nobody... Sigh.)
> They're in the business of
> making fast ARM chips and they're going to outpace Qemu.
They're not in the same business. They don't compete. You don't deploy on
qemu, and when you go down to Fry's to get a laptop the x86-64 is likely to
continue to be the main option for the forseeable future.
> > But in general, Moore's Law says that qemu on current PC hardware is
> > about the speed of current PC hardware seven years ago. (And obviously
> > nobody ever built anything before 2003. :)
>
> The speed surely depends on the architecture being emulated. I did try
> Qemu on ARM on a nice desktop box here and it seemed unworkable to me.
You couldn't make it work, therefore it can't be made to work?
> I am perfectly happy to trust Fedora to take care of disribution
> rebuilds for me.
Good for you.
Did it ever occur to you that there are people out there who do the bits you
don't personally do? That all these packages don't emerge from zeus's
forehead on a quarterly basis? That someone, somewhere, is actually
interested in how this stuff _does_ work?
> >> Newer ARM platforms like Cortex8+ and the Marvell Sheevaplug will
> >> outstrip emulated performance on a normal PC. There are 2GHz multi-core
> >> ARMs coming as well apparently. So I took the view I should ignore Qemu
> >> and get an early start on the true native build that will be the future
> >> of "native build" as opposed to cross due to that.
> >
> > Pages 24-34 of the above PDF go over this. The first two pages are on
> > the advantages of native compiling on real hardware, the next eight pages
> > are on the disadvantages. It can certainly be made to work, especially
> > in a large corporation willing to spend a lot of money on hardware as a
> > _prerequisite_ to choosing a deployment platform.
>
> This is what I have been calling "buildroot thinking" again. What do
> you think you will be rebuilding exactly out of Fedora when you put it
> on an ARM11+ system? Typically, it's going to be zero packages, nothing
> at all. What you will need to build typically is this:
"All the world's a Cortex A8, and always will be."
Look, go have fun. I'm sure what you've chosen to do is going to work just
great for you, and you'll never need to care about anything else. Personally,
I no interest in speaking to you ever again about anything, and I have a spam
filter for situations like this.
I'll even make you happy: "You win!" There you go.
*plonk*
Rob
--
Latency is more important than throughput. It's that simple. - Linus Torvalds
^ permalink raw reply
* Re: Re: How to store kernel panic/oops
From: Paul Mundt @ 2009-12-29 7:05 UTC (permalink / raw)
To: Marco Stornelli; +Cc: dwmw2, CeLinux, Linux Embedded
In-Reply-To: <4B38F304.4020801@gmail.com>
On Mon, Dec 28, 2009 at 07:03:48PM +0100, Marco Stornelli wrote:
> David Woodhouse wrote:
>
> > Can't it be done with what's in the tree already? Just create an MTD
> > device using phram or something else, then point mtdoops at it
>
> Yes of course, if possible we shouldn't reinvent the wheel but I
> wondered if there was something more specific. To add mtdoops (more or
> less 1k) we have to add mtd subsys (more or less 14k) to the kernel to
> achieve this and it's all overhead.
>
panic/oops information is punted down to a kmsg dumper, which mtdoops
just happens to be one of. If you wanted to do your own platform specific
hack for size constraints you could easily just have your own special
one instead. Take a look at linux/kmsg_dump.h.
^ permalink raw reply
* Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader
From: Andy Green @ 2009-12-29 11:11 UTC (permalink / raw)
To: Rob Landley; +Cc: celinux-dev, Robert Schwebel, linux-embedded
In-Reply-To: <200912282225.03019.rob@landley.net>
On 12/29/09 04:25, Somebody in the thread at some point said:
Hi -
(We're talking about ARM11+ case here, just to save a lot of qualifying)
>> I wasn't suggesting you don't have firsthand experience all over the
>> place, eg, busybox, I know it. What I do suggest in the principles I
>> have been bothering your Christmas with here can really pay off and are
>> in the opposite direction to Qemu, bloated bootloaders, and buildroot
>> style distro production.
>
> In the PDF I spent a dozen pages pointing out that I think buildroot is silly
> and how it diverged from its original goals. Why do you bring it up here?
Buildroot is the original "select a bunch of projects and I will sit
building them for you all day and give you a rootfs tarball". What I
talked about is "buildroot style distro production", where you sat there
building things being a bad way if it's possible to use a real distro
with precooked binaries.
> Building natively is nice. I think cross compiling extensive amounts of stuff
> is generally counterproductive, and you don't seem to disagree with that.
I completely agree with it.
> (However, if there weren't cases where cross compiling _does_ make sense I
> wouldn't bother building the second stage cross compilers in my project and
> packaging them up for other people to use. "I don't recommend it" != "people
> who do this are stupid".)
Earlier I pointed out that I use cross for bootloader and kernel.
The problems with cross generally come from having to continually fight
projects that don't want to build cross and maintaining and building the
hundreds of projects that make up your rootfs as your own personal
burden. Over time it seems to me it will be on a loser when it comes up
against established distro projects like Fedora that are inheriting the
quality of native built stuff and delivering binaries for your CPU ready
cooked. (Of course this is only true at this ARM11+ point, below that
cross like buildroot is the lifesaver).
So yes, I would tell people to avoid cross distros if they can find a
native built one that is workable for them because they can expect less
issues and more consistent quality. And they're not gonna be expected
to build anything themselves.
> You're conflating a bunch of issues. You can boot a full distro under qemu or
> under real hardware, so "use a native distro" is orthogonal to "develop under
> emulation vs develop on target hardware". And building a hand-rolled system
> or assembling prebuilt distro bits can both be done natively (under either
> kind of "native"), so that's orthogonal too.
See above. Qemu itself is an unnecessary burden for the developer, that
goes triple if the guy follows the way you suggested and makes a giant
Qemu build farm so he can follow that buildroot style and rebuild his
whole distro pointlessly with distributed make, instead of simply
downloading distro-built binaries and getting on with his life.
> I'm really not understanding what argument you're making. "The one specific
> configuration I've chosen to use is better than any other possible
> configuration"... Possibly there's a "for your needs" attached somewhere that
> you're missing? You seem to be confusing "that's not an interesting option
> for me" with "that's not an interesting option". You're conflating a bunch of
> different issues into a One True Way, which is sad.
Yes, I am sorry if it makes you uncomfortable but some things are
inherently the wrong direction. Bootloader bloat is the case in point,
if there's an OS like Linux on the box it's always preferable to do your
business in that, not add faux bash to your bootloader.
Sometimes you might get forced into going in the wrong direction, Robert
was making the case about needing to light up his display super quick at
boot. If that happens you may bend to it, but it doesn't change that it
was the wrong direction and you want to try to avoid it next time.
>> Or you just do it native and you don't care about extreme rebuild case
>> because you're using a distro that built it already.
>
> I find reproducibility to be kind of nice. (Remember when ubuntu dropped
> Powerpc support? Or how Red hat Enterprise is finally dropping itanic? Even
> today can you build a full distro natively on new hardware like microblaze or
> blackfin?)
>
> I also don't see any difference between your "let the distro handle everything"
> with "the vendor supplies a BSP, why would you need anything else?" That's
> certainly a point of view, and for you it may be fine. Anything can be
> outsourced. But "I don't have to do it, therefore nobody does" is a
> questionable position.
Hm, you were telling me I was conflating things earlier.
With RPM in terms of process you can easily rebuild the packages if you
have the SRPM. So it's nothing like a BSP blob, rebuild is always
easily possible package by package, and the packages explain their build
dependencies so it can be automated. So "reproducibility" is possible.
My point is that typically on desktop and server distro installs, you do
not go around rebuilding packages, Gentoo excepted because that's their
thing. I've done it to a couple of packages over the years and that's
it. The reason for going the distro route is I can rely on and exploit
the work they already did. (And in Fedora's case, their quality demands
are higher than mine so it's an automatic win).
So while reproducibility is possible, it's almost never needed by end
users. If they do need it, they're going to have to get some metal that
allows them to do it, maybe the mass Qemu farm thing or whatever. But
not typical users.
>>> There's more things you can do to speed it up if you want to go down that
>>> rabbit hole (which the presentation does), and there's more work being
>>> done in qemu. (TCG was originally a performance hit but has improved
>>> since, although it varies widely by platform and is its own rabbit hole.
>>> Also switching to gigabit NIC emulation with jumbo frames helped distcc a
>>> lot.)
>>
>> Just saying people don't do distribution rebuilds for their desktop or
>> server boxes unless they're Gentoo believers.
>
> Linux From Scratch does not exist? My friend Mark (who inherited a "Slackware
> for PowerPC" cross-build system) is in reality unemployed? Nobody ever
> actually works _on_ Fedora and needs to do a development build?
>
> It's interesting to see someone who can imply these things with a straight
> face on an embedded development list.
You are making straw men here. Normally, people who are using a distro
do not do whole distribution rebuilds for their desktop or server boxes,
it is a stone cold fact. It must be 99%+ of distro users never
customized one package let alone dist-rebuild. They use the binary
packages coming from the distro, that is the point of having a distro.
Yeah other people have buildroot-style build policies and do other
things, good luck to them. I'm suggesting they should be avoided for
use at this ARM11+ level.
>> So the need to consider
>> this heavy duty stuff only exists at the distro. In Fedora case, guys
>> at Marvell are running it and have access to lots of fast physical
>> hardware they hooked up to Koji (you touch on this in your PDF). I
>> don't think it's like they were waiting to hear about Qemu now they're
>> gonna drop that and move to emulation.
>
> You also seem to be unaware that QEMU was only started in 2003 (first commit to
...
> Therefore, they didn't "drop" support, QEMU wasn't ready for prime time back
Please read what I actually wrote above, which has nothing to do with
Qemu dropping anything. It seems you would call this kind of thing
"conflating orthogonal issues".
> By the way:
>
> http://fedoraproject.org/wiki/Architectures/ARM/HowToQemu
>
> You continue to conflate orthogonal issues.
Again I didn't say it's not possible to use Qemu, it's just slow. And
for the reasons of having unified dependencies in your build system, if
it is possible to build on your execution device that is preferable.
>> They're in the business of
>> making fast ARM chips and they're going to outpace Qemu.
>
> They're not in the same business. They don't compete. You don't deploy on
> qemu, and when you go down to Fry's to get a laptop the x86-64 is likely to
> continue to be the main option for the forseeable future.
Developers have to make a decision whether to use the native metal or
emulation. Qemu is slow, so much so you suggest having a build farm of
Qemus to work around that. Native metal is getting faster.
So I would advise people to use their fast native box if they have one,
because I think it's going to get faster quicker than Qemu basis over
the next years.
>>> But in general, Moore's Law says that qemu on current PC hardware is
>>> about the speed of current PC hardware seven years ago. (And obviously
>>> nobody ever built anything before 2003. :)
>>
>> The speed surely depends on the architecture being emulated. I did try
>> Qemu on ARM on a nice desktop box here and it seemed unworkable to me.
>
> You couldn't make it work, therefore it can't be made to work?
Oh no: it worked. It was "unworkable" as a basis for development
because it was slower than my native hardware. So I moved to that and
removed the burden of Qemu and syncing two rootfses from my development
thinking.
>> I am perfectly happy to trust Fedora to take care of disribution
>> rebuilds for me.
>
> Good for you.
>
> Did it ever occur to you that there are people out there who do the bits you
> don't personally do? That all these packages don't emerge from zeus's
> forehead on a quarterly basis? That someone, somewhere, is actually
> interested in how this stuff _does_ work?
Of course. Do you know about Koji? It's pretty cool.
http://arm.koji.fedoraproject.org/koji/
That level of infrastructure and investment is done by the distro, not
the users who can get high quality, immediately usable binaries for
free. We shouldn't be advising devs that they need to get into
reproducing the distro as the price of entry. (They can get into that
if they absolutely have to, they can even run their own koji which is
packaged in Fedora.)
>> This is what I have been calling "buildroot thinking" again. What do
>> you think you will be rebuilding exactly out of Fedora when you put it
>> on an ARM11+ system? Typically, it's going to be zero packages, nothing
>> at all. What you will need to build typically is this:
>
> "All the world's a Cortex A8, and always will be."
>
> Look, go have fun. I'm sure what you've chosen to do is going to work just
> great for you, and you'll never need to care about anything else. Personally,
> I no interest in speaking to you ever again about anything, and I have a spam
> filter for situations like this.
>
> I'll even make you happy: "You win!" There you go.
>
> *plonk*
I see.
Well, unfortunate as it is I now have to address the back of your head
with your hands clamped over your ears, I would like to thank you for an
interesting thread.
-Andy
^ permalink raw reply
* Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader
From: Jamie Lokier @ 2009-12-29 13:13 UTC (permalink / raw)
To: Andy Green; +Cc: Rob Landley, celinux-dev, Matt Hsu, linux-embedded
In-Reply-To: <4B31D956.9020907@warmcat.com>
Andy Green wrote:
> >No TCP/IP, no TFTP, not even BOOTP (but it's a nice bonus), no command
> >line interpreter (just a GPIO on board to boot into "unbrick me" mode
> >:-), and most strikingly _no_ flash driver for flash chip du jour.
> >
> >To flash it you send a kernel to boot from RAM which is capable of
> >flashing it.
>
> Sorry I missed where this kernel appears from and the bootloader that
> spawned it, since both could get trashed.
The kernel appeared from the network/serial/USB, as described. It's
not on the device. The point is you don't need a "spare" kernel for
unbrickability (and kernels are quite large); it's enough to be able
to send one in unbricking mode.
> That is actually a conundrum
> on a lot of systems and some of the solutions (write-once backup
> bootloader) in the long run lead to other issues.
I agree especially about the write-once backup bootloader.
> True SD Boot does truly deliver unbrickability if you are willing to
> swap out or reformat the SD card.
With the disadvantage that you need the board space, connector cost
etc. for an SD card. It's fine if you already have that.
I'd put it into a similar category to network/serial/USB: Using
whatever your board provides for bare-bones communication, bootstrap a
kernel for more advanced stuff involving fancier drivers.
> >>>http://wiki.openmoko.org/wiki/Qi
> >>
> >>Looking at the screen shot there, you've got code to parse ext2
> >>filesystems.
> >>What is your definition of "minimal"?
> >
> >Ew, ext2 doesn't even satisfy powerfail-during-kernel-upgrade safety.
>
> It's just misleading (but accurate). ext2 is the "lowest common
> denominator" read-only parsing that actually supports ext3 and ext4 if
> you are careful about the formatting options. So the actual filesystem
> is ext3 or ext4 typically (ext3 in GTA02 case), it's not that the
> bootloader is mandating specifically ext2.
If it reads an ext3/4 filesystem using an ext2 read-only parser, then
it will not be powerfail-safe. Partially written files and directory
updates will look like corruption on boot.
It's essential that it parses the journal as well. It does not have
to commit the journal (which would need device writing ability), but
in that case it must allow journal blocks to supercede blocks from
elsewhere in the filesystem.
> >It's possible to boot without parsing filesystems, but that is one
> >rather nice feature, and with the right filesystems it can make system
> >updates powerfail-safe.
>
> Bootloader is tricky, but actually on this iMX31 device Fedora is used,
> yum update keeps the last 3 kernels around and our kernel package
> follows that. So it's possible to have backup kernels automatically
> integrated into the bootloader and packaging system.
It's useless to have 3 kernels around if the directory containing them
looks corrupted due to not parsing the journal. :-) Then you won't see
any of the kernels at boot time.
-- Jamie
^ permalink raw reply
* Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader
From: Andy Green @ 2009-12-29 13:36 UTC (permalink / raw)
To: Jamie Lokier; +Cc: Rob Landley, celinux-dev, Matt Hsu, linux-embedded
In-Reply-To: <20091229131353.GA5736@shareable.org>
On 12/29/09 13:13, Somebody in the thread at some point said:
Hi -
>> Sorry I missed where this kernel appears from and the bootloader that
>> spawned it, since both could get trashed.
>
> The kernel appeared from the network/serial/USB, as described. It's
> not on the device. The point is you don't need a "spare" kernel for
> unbrickability (and kernels are quite large); it's enough to be able
> to send one in unbricking mode.
The bootloader that will load this kernel can't be updated or trashed
itself?
> With the disadvantage that you need the board space, connector cost
> etc. for an SD card. It's fine if you already have that.
Sure. Although the connector and pullups needed are really cheap. You
also need an SD interface on your CPU, but most of this kind of
processor have multiple SD interfaces already.
> I'd put it into a similar category to network/serial/USB: Using
> whatever your board provides for bare-bones communication, bootstrap a
> kernel for more advanced stuff involving fancier drivers.
Well my suggestion is SD is in a different category. Unlike network or
serial or USB it works for boot without a host to talk to.
>>> Ew, ext2 doesn't even satisfy powerfail-during-kernel-upgrade safety.
>>
>> It's just misleading (but accurate). ext2 is the "lowest common
>> denominator" read-only parsing that actually supports ext3 and ext4 if
>> you are careful about the formatting options. So the actual filesystem
>> is ext3 or ext4 typically (ext3 in GTA02 case), it's not that the
>> bootloader is mandating specifically ext2.
>
> If it reads an ext3/4 filesystem using an ext2 read-only parser, then
> it will not be powerfail-safe. Partially written files and directory
> updates will look like corruption on boot.
Yeah I think that is true.
> It's essential that it parses the journal as well. It does not have
> to commit the journal (which would need device writing ability), but
> in that case it must allow journal blocks to supercede blocks from
> elsewhere in the filesystem.
Personally I don't use ext2 for the containing the kernels, but VFAT on
SD. Qi supports ext2 because that's what was going on GTA02.
>>> It's possible to boot without parsing filesystems, but that is one
>>> rather nice feature, and with the right filesystems it can make system
>>> updates powerfail-safe.
>>
>> Bootloader is tricky, but actually on this iMX31 device Fedora is used,
>> yum update keeps the last 3 kernels around and our kernel package
>> follows that. So it's possible to have backup kernels automatically
>> integrated into the bootloader and packaging system.
>
> It's useless to have 3 kernels around if the directory containing them
> looks corrupted due to not parsing the journal. :-) Then you won't see
> any of the kernels at boot time.
You're assuming that those kernels are in ext2/3/4, as mentioned above
actually I have been using VFAT. But actually with Qi there's another
solution even for ext2 "/boot" partition that makes it safe.
Qi has a fallthru scheme for acquiring the kernels, you give a list of
partitions and filenames and it uses the first one that gets it a good
kernel image.
Therefore you can have a backup kernel and rootfs in a second partition
which is not normally updated. If the normal kernel is broken (plus or
minus the fs parsing code being good enough it won't infinitely loop or
whatever when given junk) it will select the backup one from the clean
partition.
-Andy
^ permalink raw reply
* Re: [PATCH 0/9] Squashfs: Add support for LZMA compressed filesystems
From: Bruno Wolff III @ 2009-12-31 12:06 UTC (permalink / raw)
To: Phillip Lougher
Cc: akpm, linux-embedded, linux-fsdevel, linux-kernel,
phillip.lougher, tim.bird
In-Reply-To: <4B300912.5060905@lougher.demon.co.uk>
On Mon, Dec 21, 2009 at 23:47:30 +0000,
Phillip Lougher <phillip@lougher.demon.co.uk> wrote:
>
> It's very unlikely. Andrew wanted it to spend a cycle in linux-next. I'll then
> try for 2.6.34 merge. I'm happy with that.
>
> Phillip
linux-next still hasn't picked these patches up. It would give me more leverage
in getting the patches into Fedora 13 if they were to be in linux-next.
^ permalink raw reply
* Re: [PATCH 0/9] Squashfs: Add support for LZMA compressed filesystems
From: Bruno Wolff III @ 2010-01-01 9:11 UTC (permalink / raw)
To: Phillip Lougher
Cc: akpm, linux-embedded, linux-fsdevel, linux-kernel,
phillip.lougher, tim.bird
In-Reply-To: <20091231120634.GC9930@wolff.to>
On Thu, Dec 31, 2009 at 06:06:34 -0600,
Bruno Wolff III <bruno@wolff.to> wrote:
> On Mon, Dec 21, 2009 at 23:47:30 +0000,
> Phillip Lougher <phillip@lougher.demon.co.uk> wrote:
> >
> > It's very unlikely. Andrew wanted it to spend a cycle in linux-next. I'll then
> > try for 2.6.34 merge. I'm happy with that.
> >
> > Phillip
>
> linux-next still hasn't picked these patches up. It would give me more leverage
> in getting the patches into Fedora 13 if they were to be in linux-next.
I see that they made it into the January 1st linux-next tree. Thanks.
^ permalink raw reply
* Re: [PATCH V2 6/8] lzma: Make lzma available to non initramfs/initrd code
From: Andrew Morton @ 2010-01-06 22:04 UTC (permalink / raw)
To: Phillip Lougher
Cc: linux-embedded, linux-fsdevel, linux-kernel, phillip.lougher,
tim.bird, Stephen Rothwell, Albin Tonnerre
In-Reply-To: <E1NIuOW-0001ol-RC@dylan.lougher.demon.co.uk>
On Fri, 11 Dec 2009 01:34:08 +0000
Phillip Lougher <phillip@lougher.demon.co.uk> wrote:
> Add a config option DECOMPRESS_LZMA_NEEDED which allows subsystems to
> specify they need the unlzma code. Normally decompress_unlzma.c is
> compiled with __init and unlzma is not exported to modules.
>
> Move INIT definition into separate header files for bzip2/lzma/inflate
> so it can be defined differently for each decompressor.
>
This patch (which is in linux-next) breaks
lib-add-support-for-lzo-compressed-kernels.patch, below. The
definition of INIT is no longer available in lib/decompress_unlzo.c, and
lib/decompress_unlzo.c: In function 'unlzo':
lib/decompress_unlzo.c:106: error: 'error' undeclared (first use in this function)
lib/decompress_unlzo.c:106: error: (Each undeclared identifier is reported only once
lib/decompress_unlzo.c:106: error: for each function it appears in.)
lib/decompress_unlzo.c:111: error: implicit declaration of function 'error'
I'm planning on merging
zlib-optimize-inffast-when-copying-direct-from-output.patch
lib-add-support-for-lzo-compressed-kernels.patch
arm-add-support-for-lzo-compressed-kernels.patch
x86-add-support-for-lzo-compressed-kernels.patch
add-lzo-compression-support-for-initramfs-and-old-style-initrd.patch
into 2.6.33. I don't immediately remember why I decided that - perhaps
because the patches did arrive in time for .33, but got stalled because
people were screwing around in other trees.
So if I go ahead with that merge, linux-next will need fixing. And I
didn't get down and work what the appropriate fix is, and I don't want
to break linux-next in serious ways.
So what to do? I guess I could go ahead with the mainline merge, and
Stephen drops <whatever that tree was> from linux-next until it has
been fixed up?
If we decide to hold the above five patches off until 2.6.34 then
they'll need to be fixed up to work against linux-next, and they will
be dependent upon your patch series.
From: Albin Tonnerre <albin.tonnerre@free-electrons.com>
This patch series adds generic support for creating and extracting
LZO-compressed kernel images, as well as support for using such images on
the x86 and ARM architectures, and support for creating and using
LZO-compressed initrd and initramfs images.
Russell King said:
: Testing on a Cortex A9 model:
: - lzo decompressor is 65% of the time gzip takes to decompress a kernel
: - lzo kernel is 9% larger than a gzip kernel
:
: which I'm happy to say confirms your figures when comparing the two.
:
: However, when comparing your new gzip code to the old gzip code:
: - new is 99% of the size of the old code
: - new takes 42% of the time to decompress than the old code
:
: What this means is that for a proper comparison, the results get even better:
: - lzo is 7.5% larger than the old gzip'd kernel image
: - lzo takes 28% of the time that the old gzip code took
:
: So the expense seems definitely worth the effort. The only reason I
: can think of ever using gzip would be if you needed the additional
: compression (eg, because you have limited flash to store the image.)
:
: I would argue that the default for ARM should therefore be LZO.
This patch:
The lzo compressor is worse than gzip at compression, but faster at
extraction. Here are some figures for an ARM board I'm working on:
Uncompressed size: 3.24Mo
gzip 1.61Mo 0.72s
lzo 1.75Mo 0.48s
So for a compression ratio that is still relatively close to gzip, it's
much faster to extract, at least in that case.
This part contains:
- Makefile routine to support lzo compression
- Fixes to the existing lzo compressor so that it can be used in
compressed kernels
- wrapper around the existing lzo1x_decompress, as it only extracts one
block at a time, while we need to extract a whole file here
- config dialog for kernel compression
[akpm@linux-foundation.org: coding-style fixes]
[akpm@linux-foundation.org: cleanup]
Signed-off-by: Albin Tonnerre <albin.tonnerre@free-electrons.com>
Tested-by: Wu Zhangjin <wuzhangjin@gmail.com>
Acked-by: "H. Peter Anvin" <hpa@zytor.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Thomas Gleixner <tglx@linutronix.de>
Tested-by: Russell King <rmk@arm.linux.org.uk>
Acked-by: Russell King <rmk@arm.linux.org.uk>
Cc: Ralf Baechle <ralf@linux-mips.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
include/linux/decompress/unlzo.h | 10 +
init/Kconfig | 18 +-
lib/decompress_unlzo.c | 209 +++++++++++++++++++++++++++++
lib/lzo/lzo1x_decompress.c | 9 -
scripts/Makefile.lib | 5
5 files changed, 244 insertions(+), 7 deletions(-)
diff -puN /dev/null include/linux/decompress/unlzo.h
--- /dev/null
+++ a/include/linux/decompress/unlzo.h
@@ -0,0 +1,10 @@
+#ifndef DECOMPRESS_UNLZO_H
+#define DECOMPRESS_UNLZO_H
+
+int unlzo(unsigned char *inbuf, int len,
+ int(*fill)(void*, unsigned int),
+ int(*flush)(void*, unsigned int),
+ unsigned char *output,
+ int *pos,
+ void(*error)(char *x));
+#endif
diff -puN init/Kconfig~lib-add-support-for-lzo-compressed-kernels init/Kconfig
--- a/init/Kconfig~lib-add-support-for-lzo-compressed-kernels
+++ a/init/Kconfig
@@ -123,10 +123,13 @@ config HAVE_KERNEL_BZIP2
config HAVE_KERNEL_LZMA
bool
+config HAVE_KERNEL_LZO
+ bool
+
choice
prompt "Kernel compression mode"
default KERNEL_GZIP
- depends on HAVE_KERNEL_GZIP || HAVE_KERNEL_BZIP2 || HAVE_KERNEL_LZMA
+ depends on HAVE_KERNEL_GZIP || HAVE_KERNEL_BZIP2 || HAVE_KERNEL_LZMA || HAVE_KERNEL_LZO
help
The linux kernel is a kind of self-extracting executable.
Several compression algorithms are available, which differ
@@ -149,9 +152,8 @@ config KERNEL_GZIP
bool "Gzip"
depends on HAVE_KERNEL_GZIP
help
- The old and tried gzip compression. Its compression ratio is
- the poorest among the 3 choices; however its speed (both
- compression and decompression) is the fastest.
+ The old and tried gzip compression. It provides a good balance
+ between compression ratio and decompression speed.
config KERNEL_BZIP2
bool "Bzip2"
@@ -172,6 +174,14 @@ config KERNEL_LZMA
two. Compression is slowest. The kernel size is about 33%
smaller with LZMA in comparison to gzip.
+config KERNEL_LZO
+ bool "LZO"
+ depends on HAVE_KERNEL_LZO
+ help
+ Its compression ratio is the poorest among the 4. The kernel
+ size is about about 10% bigger than gzip; however its speed
+ (both compression and decompression) is the fastest.
+
endchoice
config SWAP
diff -puN /dev/null lib/decompress_unlzo.c
--- /dev/null
+++ a/lib/decompress_unlzo.c
@@ -0,0 +1,209 @@
+/*
+ * LZO decompressor for the Linux kernel. Code borrowed from the lzo
+ * implementation by Markus Franz Xaver Johannes Oberhumer.
+ *
+ * Linux kernel adaptation:
+ * Copyright (C) 2009
+ * Albin Tonnerre, Free Electrons <albin.tonnerre@free-electrons.com>
+ *
+ * Original code:
+ * Copyright (C) 1996-2005 Markus Franz Xaver Johannes Oberhumer
+ * All Rights Reserved.
+ *
+ * lzop and the LZO library are free software; you can redistribute them
+ * and/or modify them under the terms of the GNU General Public License as
+ * published by the Free Software Foundation; either version 2 of
+ * the License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; see the file COPYING.
+ * If not, write to the Free Software Foundation, Inc.,
+ * 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
+ *
+ * Markus F.X.J. Oberhumer
+ * <markus@oberhumer.com>
+ * http://www.oberhumer.com/opensource/lzop/
+ */
+
+#ifdef STATIC
+#include "lzo/lzo1x_decompress.c"
+#else
+#include <linux/slab.h>
+#include <linux/decompress/unlzo.h>
+#endif
+
+#include <linux/types.h>
+#include <linux/lzo.h>
+#include <linux/decompress/mm.h>
+
+#include <linux/compiler.h>
+#include <asm/unaligned.h>
+
+static const unsigned char lzop_magic[] = {
+ 0x89, 0x4c, 0x5a, 0x4f, 0x00, 0x0d, 0x0a, 0x1a, 0x0a };
+
+#define LZO_BLOCK_SIZE (256*1024l)
+#define HEADER_HAS_FILTER 0x00000800L
+
+STATIC inline int INIT parse_header(u8 *input, u8 *skip)
+{
+ int l;
+ u8 *parse = input;
+ u8 level = 0;
+ u16 version;
+
+ /* read magic: 9 first bits */
+ for (l = 0; l < 9; l++) {
+ if (*parse++ != lzop_magic[l])
+ return 0;
+ }
+ /* get version (2bytes), skip library version (2),
+ * 'need to be extracted' version (2) and
+ * method (1) */
+ version = get_unaligned_be16(parse);
+ parse += 7;
+ if (version >= 0x0940)
+ level = *parse++;
+ if (get_unaligned_be32(parse) & HEADER_HAS_FILTER)
+ parse += 8; /* flags + filter info */
+ else
+ parse += 4; /* flags */
+
+ /* skip mode and mtime_low */
+ parse += 8;
+ if (version >= 0x0940)
+ parse += 4; /* skip mtime_high */
+
+ l = *parse++;
+ /* don't care about the file name, and skip checksum */
+ parse += l + 4;
+
+ *skip = parse - input;
+ return 1;
+}
+
+STATIC inline int INIT unlzo(u8 *input, int in_len,
+ int (*fill) (void *, unsigned int),
+ int (*flush) (void *, unsigned int),
+ u8 *output, int *posp,
+ void (*error_fn) (char *x))
+{
+ u8 skip = 0, r = 0;
+ u32 src_len, dst_len;
+ size_t tmp;
+ u8 *in_buf, *in_buf_save, *out_buf;
+ int obytes_processed = 0;
+
+ set_error_fn(error_fn);
+
+ if (output) {
+ out_buf = output;
+ } else if (!flush) {
+ error("NULL output pointer and no flush function provided");
+ goto exit;
+ } else {
+ out_buf = malloc(LZO_BLOCK_SIZE);
+ if (!out_buf) {
+ error("Could not allocate output buffer");
+ goto exit;
+ }
+ }
+
+ if (input && fill) {
+ error("Both input pointer and fill function provided, don't know what to do");
+ goto exit_1;
+ } else if (input) {
+ in_buf = input;
+ } else if (!fill || !posp) {
+ error("NULL input pointer and missing position pointer or fill function");
+ goto exit_1;
+ } else {
+ in_buf = malloc(lzo1x_worst_compress(LZO_BLOCK_SIZE));
+ if (!in_buf) {
+ error("Could not allocate input buffer");
+ goto exit_1;
+ }
+ }
+ in_buf_save = in_buf;
+
+ if (posp)
+ *posp = 0;
+
+ if (fill)
+ fill(in_buf, lzo1x_worst_compress(LZO_BLOCK_SIZE));
+
+ if (!parse_header(input, &skip)) {
+ error("invalid header");
+ goto exit_2;
+ }
+ in_buf += skip;
+
+ if (posp)
+ *posp = skip;
+
+ for (;;) {
+ /* read uncompressed block size */
+ dst_len = get_unaligned_be32(in_buf);
+ in_buf += 4;
+
+ /* exit if last block */
+ if (dst_len == 0) {
+ if (posp)
+ *posp += 4;
+ break;
+ }
+
+ if (dst_len > LZO_BLOCK_SIZE) {
+ error("dest len longer than block size");
+ goto exit_2;
+ }
+
+ /* read compressed block size, and skip block checksum info */
+ src_len = get_unaligned_be32(in_buf);
+ in_buf += 8;
+
+ if (src_len <= 0 || src_len > dst_len) {
+ error("file corrupted");
+ goto exit_2;
+ }
+
+ /* decompress */
+ tmp = dst_len;
+ r = lzo1x_decompress_safe((u8 *) in_buf, src_len,
+ out_buf, &tmp);
+
+ if (r != LZO_E_OK || dst_len != tmp) {
+ error("Compressed data violation");
+ goto exit_2;
+ }
+
+ obytes_processed += dst_len;
+ if (flush)
+ flush(out_buf, dst_len);
+ if (output)
+ out_buf += dst_len;
+ if (posp)
+ *posp += src_len + 12;
+ if (fill) {
+ in_buf = in_buf_save;
+ fill(in_buf, lzo1x_worst_compress(LZO_BLOCK_SIZE));
+ } else
+ in_buf += src_len;
+ }
+
+exit_2:
+ if (!input)
+ free(in_buf);
+exit_1:
+ if (!output)
+ free(out_buf);
+exit:
+ return obytes_processed;
+}
+
+#define decompress unlzo
diff -puN lib/lzo/lzo1x_decompress.c~lib-add-support-for-lzo-compressed-kernels lib/lzo/lzo1x_decompress.c
--- a/lib/lzo/lzo1x_decompress.c~lib-add-support-for-lzo-compressed-kernels
+++ a/lib/lzo/lzo1x_decompress.c
@@ -11,11 +11,13 @@
* Richard Purdie <rpurdie@openedhand.com>
*/
+#ifndef STATIC
#include <linux/module.h>
#include <linux/kernel.h>
-#include <linux/lzo.h>
-#include <asm/byteorder.h>
+#endif
+
#include <asm/unaligned.h>
+#include <linux/lzo.h>
#include "lzodefs.h"
#define HAVE_IP(x, ip_end, ip) ((size_t)(ip_end - ip) < (x))
@@ -244,9 +246,10 @@ lookbehind_overrun:
*out_len = op - out;
return LZO_E_LOOKBEHIND_OVERRUN;
}
-
+#ifndef STATIC
EXPORT_SYMBOL_GPL(lzo1x_decompress_safe);
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("LZO1X Decompressor");
+#endif
diff -puN scripts/Makefile.lib~lib-add-support-for-lzo-compressed-kernels scripts/Makefile.lib
--- a/scripts/Makefile.lib~lib-add-support-for-lzo-compressed-kernels
+++ a/scripts/Makefile.lib
@@ -235,3 +235,8 @@ quiet_cmd_lzma = LZMA $@
cmd_lzma = (cat $(filter-out FORCE,$^) | \
lzma -9 && $(call size_append, $(filter-out FORCE,$^))) > $@ || \
(rm -f $@ ; false)
+
+quiet_cmd_lzo = LZO $@
+cmd_lzo = (cat $(filter-out FORCE,$^) | \
+ lzop -9 && $(call size_append, $(filter-out FORCE,$^))) > $@ || \
+ (rm -f $@ ; false)
_
^ permalink raw reply
* Re: [PATCH V2 6/8] lzma: Make lzma available to non initramfs/initrd code
From: Albin Tonnerre @ 2010-01-06 22:45 UTC (permalink / raw)
To: Andrew Morton
Cc: Phillip Lougher, linux-embedded, linux-fsdevel, linux-kernel,
phillip.lougher, tim.bird, Stephen Rothwell
In-Reply-To: <20100106140409.ba0f63d1.akpm@linux-foundation.org>
On Wed, 06 Jan 2010 14:04 -0800, Andrew Morton wrote :
> On Fri, 11 Dec 2009 01:34:08 +0000
> Phillip Lougher <phillip@lougher.demon.co.uk> wrote:
>
> > Add a config option DECOMPRESS_LZMA_NEEDED which allows subsystems to
> > specify they need the unlzma code. Normally decompress_unlzma.c is
> > compiled with __init and unlzma is not exported to modules.
> >
> > Move INIT definition into separate header files for bzip2/lzma/inflate
> > so it can be defined differently for each decompressor.
> >
>
> This patch (which is in linux-next) breaks
> lib-add-support-for-lzo-compressed-kernels.patch, below. The
> definition of INIT is no longer available in lib/decompress_unlzo.c, and
>
> lib/decompress_unlzo.c: In function 'unlzo':
> lib/decompress_unlzo.c:106: error: 'error' undeclared (first use in this function)
> lib/decompress_unlzo.c:106: error: (Each undeclared identifier is reported only once
> lib/decompress_unlzo.c:106: error: for each function it appears in.)
> lib/decompress_unlzo.c:111: error: implicit declaration of function 'error'
>
>
> I'm planning on merging
>
> zlib-optimize-inffast-when-copying-direct-from-output.patch
> lib-add-support-for-lzo-compressed-kernels.patch
> arm-add-support-for-lzo-compressed-kernels.patch
> x86-add-support-for-lzo-compressed-kernels.patch
> add-lzo-compression-support-for-initramfs-and-old-style-initrd.patch
>
> into 2.6.33. I don't immediately remember why I decided that - perhaps
> because the patches did arrive in time for .33, but got stalled because
> people were screwing around in other trees.
>
> So if I go ahead with that merge, linux-next will need fixing. And I
> didn't get down and work what the appropriate fix is, and I don't want
> to break linux-next in serious ways.
>
>
> So what to do? I guess I could go ahead with the mainline merge, and
> Stephen drops <whatever that tree was> from linux-next until it has
> been fixed up?
I'd go for that one, although I'm obviously biaised. IMHO the linux-next fix
should be easy enough not to postpone the above five patches until 2.6.34.
Regards,
--
Albin Tonnerre, Free Electrons
Kernel, drivers and embedded Linux development,
consulting, training and support.
http://free-electrons.com
^ permalink raw reply
* Re: [PATCH V2 6/8] lzma: Make lzma available to non initramfs/initrd code
From: Stephen Rothwell @ 2010-01-07 0:04 UTC (permalink / raw)
To: Andrew Morton
Cc: Phillip Lougher, linux-embedded, linux-fsdevel, linux-kernel,
phillip.lougher, tim.bird, Albin Tonnerre
In-Reply-To: <20100106140409.ba0f63d1.akpm@linux-foundation.org>
[-- Attachment #1: Type: text/plain, Size: 730 bytes --]
Hi Andrew,
On Wed, 6 Jan 2010 14:04:09 -0800 Andrew Morton <akpm@linux-foundation.org> wrote:
>
> So if I go ahead with that merge, linux-next will need fixing. And I
> didn't get down and work what the appropriate fix is, and I don't want
> to break linux-next in serious ways.
Why not, everyone else does :-)
> So what to do? I guess I could go ahead with the mainline merge, and
> Stephen drops <whatever that tree was> from linux-next until it has
> been fixed up?
Just do the merge ... I will just need to drop the squashfs tree until
they do a merge/rebase with Linus' tree and fix it all up.
--
Cheers,
Stephen Rothwell sfr@canb.auug.org.au
http://www.canb.auug.org.au/~sfr/
[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox