* Re: 2.7% build time regression caused by enabling debug/printk.scc KERNEL_FEATURE] [not found] <1469620052.23580.115.camel@linuxfoundation.org> @ 2016-07-27 12:28 ` Ed Bartosh 2016-07-27 13:06 ` Bruce Ashfield 0 siblings, 1 reply; 7+ messages in thread From: Ed Bartosh @ 2016-07-27 12:28 UTC (permalink / raw) To: openembedded-core; +Cc: Bruce Ashfield Hi Bruce, Thank you for your suggestion to experiment with configuration options! > On 2016-07-26 10:32 AM, Ed Bartosh wrote: > >Hi all, > > > >We've noticed quite big increase of core-image-minimal build time caused by commit > >d55b7650d305ffad953dd36595ee6b9aa8df5a72 linux-yocto: Enablei debug/printk.scc KERNEL_FEATURE on qemuall machines. > > > > That commit only enables the following options: > > CONFIG_PRINTK=y > CONFIG_PRINTK_TIME=y > > CONFIG_EARLY_PRINTK=y > > CONFIG_EARLY_PRINTK_DBGP=y > CONFIG_EARLY_PRINTK_EFI=y > CONFIG_TTY_PRINTK=y > > And yes, that will add some size to the kernel, but I'm not seeing > similar size increases here. > > If you take a look through the kernel source and build, there are > relatively few additions to the actual kernel build that change > based on those options, and most of them are runtime changes versus > build-time changes. > The actuall difference in configuration is bigger as far as I can see: $ diff -u ../build-kernel/tmp/work/qemux86_64-poky-linux/linux-yocto/4.4.14+gitAUTOINC+4800a400d5_8361321fec-r0/image/boot/config-4.4.14-yocto-standard tmp/work/qemux86_64-poky-linux/linux-yocto/4.4.14+gitAUTOINC+4800a400d5_8361321fec-r0/image/boot/config-4.4.14-yocto-standard | grep '^+C' +CONFIG_CONSOLE_POLL=y +CONFIG_PRINTK_TIME=y +CONFIG_DEBUG_INFO=y +CONFIG_DEBUG_KERNEL=y +CONFIG_SCHED_DEBUG=y +CONFIG_DEBUG_PREEMPT=y +CONFIG_KGDB=y +CONFIG_KGDB_SERIAL_CONSOLE=y +CONFIG_KGDB_LOW_LEVEL_TRAP=y +CONFIG_KGDB_KDB=y +CONFIG_KDB_DEFAULT_ENABLE=0x1 +CONFIG_KDB_KEYBOARD=y +CONFIG_KDB_CONTINUE_CATASTROPHIC=0 +CONFIG_EARLY_PRINTK_DBGP=y +CONFIG_EARLY_PRINTK_EFI=y +CONFIG_DEBUG_RODATA=y +CONFIG_DEBUG_RODATA_TEST=y +CONFIG_X86_DEBUG_FPU=y I guess the reason of this is that printk.scc includes debug-kernel.scc, which brings more config options: CONFIG_DEBUG_KERNEL=y CONFIG_DEBUG_INFO=y CONFIG_DEBUG_PREEMPT=y That probably explains the difference in kernel size and compile time. After removing 'include debug-kernel.scc' the difference in configuration became more reasonable: +CONFIG_PRINTK_TIME=y +CONFIG_EARLY_PRINTK_DBGP=y +CONFIG_EARLY_PRINTK_EFI=y And the size of kernel and modules became almost the same as before enabling printk feature. Considering that the rest of the options from printk.scc don't appear in the result configuration even if debug-kernel.scc is included I hope it should be ok to remove 'include debug-kernel.scc' from printk.scc. Does it make sense for you? > >Here are results of /usr/bin/time -v bitbake core-image-minimal for this > >and previous commits: > > > >linux-yocto: Enable debug/printk.scc KERNEL_FEATURE on qemuall machines: d55b7650d305ffad953dd36595ee6b9aa8df5a72 > > Command being timed: "bitbake core-image-minimal" > > User time (seconds): 14966.24 > > System time (seconds): 3829.23 > > Percent of CPU this job got: 1212% > > Elapsed (wall clock) time (h:mm:ss or m:ss): 25:50.52 > > Average shared text size (kbytes): 0 > > Average unshared data size (kbytes): 0 > > Average stack size (kbytes): 0 > > Average total size (kbytes): 0 > > Maximum resident set size (kbytes): 917592 > > Average resident set size (kbytes): 0 > > Major (requiring I/O) page faults: 8105 > > Minor (reclaiming a frame) page faults: 628476540 > > Voluntary context switches: 26886617 > > Involuntary context switches: 2033535 > > Swaps: 0 > > File system inputs: 3160048 > > File system outputs: 78417720 > > Socket messages sent: 0 > > Socket messages received: 0 > > Signals delivered: 0 > > Page size (bytes): 4096 > > Exit status: 0 > > > >commit: nativesdk-packagegroup-sdk-host: Add wayland-scanner to the SDK native toolchain: e833508075bcb8f2d1ccddcf46f579aea7a2ab1d > > Command being timed: "bitbake core-image-minimal" > > User time (seconds): 14526.89 > > System time (seconds): 3770.74 > > Percent of CPU this job got: 1193% > > Elapsed (wall clock) time (h:mm:ss or m:ss): 25:33.44 > > Average shared text size (kbytes): 0 > > Average unshared data size (kbytes): 0 > > Average stack size (kbytes): 0 > > Average total size (kbytes): 0 > > Maximum resident set size (kbytes): 917792 > > Average resident set size (kbytes): 0 > > Major (requiring I/O) page faults: 8058 > > Minor (reclaiming a frame) page faults: 618283939 > > Voluntary context switches: 26538183 > > Involuntary context switches: 1859365 > > Swaps: 0 > > File system inputs: 3191584 > > File system outputs: 66411592 > > Socket messages sent: 0 > > Socket messages received: 0 > > Signals delivered: 0 > > Page size (bytes): 4096 > > Exit status: 0 > > > >The total time (user time + system time) incresed by 2.72% which seems quite a lot. > > > >diff of tasks utime shows biggest difference in do_package* tasks. This is probably > >due to increased size of packages: > > > >do_bundle_initramfs:utime: 21 | do_bundle_initramfs:utime: 32 > >do_compile:utime: 47 | do_compile:utime: 51 > >do_compile_kernelmodules:utime: 39 | do_compile_kernelmodules:utime: 42 > >do_configure:utime: 20 | do_configure:utime: 24 > >do_deploy:utime: 38 | do_deploy:utime: 35 > >do_fetch:utime: 25 | do_fetch:utime: 28 > >do_install:utime: 26 | do_install:utime: 22 > >do_kernel_checkout:utime: 21 | do_kernel_checkout:utime: 24 > >do_kernel_configcheck:utime: 21 | do_kernel_configcheck:utime: 21 > >do_kernel_configme:utime: 23 | do_kernel_configme:utime: 23 > >do_kernel_link_images:utime: 24 | do_kernel_link_images:utime: 22 > >do_kernel_metadata:utime: 25 | do_kernel_metadata:utime: 23 > >do_package:utime: 396 | do_package:utime: 378 > >do_packagedata:utime: 35 | do_packagedata:utime: 34 > >do_package_qa:utime: 460 | do_package_qa:utime: 492 > >do_package_write_rpm:utime: 431 | do_package_write_rpm:utime: 399 > >do_patch:utime: 25 | do_patch:utime: 25 > >do_populate_lic:utime: 30 | do_populate_lic:utime: 29 > >do_populate_sysroot:utime: 33 | do_populate_sysroot:utime: 30 > >do_shared_workdir:utime: 24 | do_shared_workdir:utime: 25 > >do_sizecheck:utime: 23 | do_sizecheck:utime: 21 > >do_strip:utime: 23 | do_strip:utime: 23 > >do_uboot_mkimage:utime: 25 | do_uboot_mkimage:utime: 24 > >do_unpack:utime: 21 | do_unpack:utime: 22 > >do_validate_branches:utime: 23 | do_validate_branches:utime: 24 > > > >There is also quite big increase in consumed disk space by the kernel > >and modules: > >$ ls -lh ./tmp.bad/work/qemux86-poky-linux/linux-yocto/4.4.14+gitAUTOINC+4800a400d5_8361321fec-r0/image/boot/vmlinux-4.4.14-yocto-standard > >-rw-r--r-- 1 ed ed 261M Jul 25 12:39 ./tmp.bad/work/qemux86-poky-linux/linux-yocto/4.4.14+gitAUTOINC+4800a400d5_8361321fec-r0/image/boot/vmlinux-4.4.14-yocto-standard > >$ ls -lh ./tmp.prev/work/qemux86-poky-linux/linux-yocto/4.4.14+gitAUTOINC+4800a400d5_8361321fec-r0/image/boot/vmlinux-4.4.14-yocto-standard > >-rw-r--r-- 1 ed ed 19M Jul 25 17:34 ./tmp.prev/work/qemux86-poky-linux/linux-yocto/4.4.14+gitAUTOINC+4800a400d5_8361321fec-r0/image/boot/vmlinux-4.4.14-yocto-standard > > > >$ ls -lh tmp.bad/deploy/images/qemux86/modules--4.4.14+git0+4800a400d5_8361321fec-r0-qemux86-20160725092744.tgz > >-rw-rw-r-- 2 ed ed 99M Jul 25 12:39 tmp.bad/deploy/images/qemux86/modules--4.4.14+git0+4800a400d5_8361321fec-r0-qemux86-20160725092744.tgz > >$ ls -lh tmp.prev/deploy/images/qemux86/modules--4.4.14+git0+4800a400d5_8361321fec-r0-qemux86-20160725105810.tgz > >-rw-rw-r-- 1 ed ed 4.3M Jul 25 17:34 tmp.prev/deploy/images/qemux86/modules--4.4.14+git0+4800a400d5_8361321fec-r0-qemux86-20160725105810.tgz > > > >Any thoughts on this? Is it normal or it should be tested further and fixed? > > I'm not sure that this is related to the debug printk build, but > anything that that could be increasing footprint and build time > like that should be looked into. > > Without messing with the oe-core commit that enabled the feature, has > someone done a build, gone into the kernel .config, disabled the options > and then rebuilt ? I'd start there to rule out that the option is > actually doing something that significant to the build itself. > > .. but I can't say much more than that, since as you can see by the > commit, I'm not on the sign-off list, so I don't have all the background. > > Bruce > > > > >-- > >Regards, > >Ed > > > > -- > _______________________________________________ > Openembedded-core mailing list > Openembedded-core@lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/openembedded-core -- -- Regards, Ed ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: 2.7% build time regression caused by enabling debug/printk.scc KERNEL_FEATURE] 2016-07-27 12:28 ` 2.7% build time regression caused by enabling debug/printk.scc KERNEL_FEATURE] Ed Bartosh @ 2016-07-27 13:06 ` Bruce Ashfield 2016-07-27 13:18 ` Ed Bartosh 0 siblings, 1 reply; 7+ messages in thread From: Bruce Ashfield @ 2016-07-27 13:06 UTC (permalink / raw) To: ed.bartosh, openembedded-core On 2016-07-27 08:28 AM, Ed Bartosh wrote: > Hi Bruce, > > Thank you for your suggestion to experiment with configuration options! > >> On 2016-07-26 10:32 AM, Ed Bartosh wrote: >>> Hi all, >>> >>> We've noticed quite big increase of core-image-minimal build time caused by commit >>> d55b7650d305ffad953dd36595ee6b9aa8df5a72 linux-yocto: Enablei debug/printk.scc KERNEL_FEATURE on qemuall machines. >>> >> >> That commit only enables the following options: >> >> CONFIG_PRINTK=y >> CONFIG_PRINTK_TIME=y >> >> CONFIG_EARLY_PRINTK=y >> >> CONFIG_EARLY_PRINTK_DBGP=y >> CONFIG_EARLY_PRINTK_EFI=y >> CONFIG_TTY_PRINTK=y >> >> And yes, that will add some size to the kernel, but I'm not seeing >> similar size increases here. >> >> If you take a look through the kernel source and build, there are >> relatively few additions to the actual kernel build that change >> based on those options, and most of them are runtime changes versus >> build-time changes. >> > > The actuall difference in configuration is bigger as far as I can see: > $ diff -u ../build-kernel/tmp/work/qemux86_64-poky-linux/linux-yocto/4.4.14+gitAUTOINC+4800a400d5_8361321fec-r0/image/boot/config-4.4.14-yocto-standard > tmp/work/qemux86_64-poky-linux/linux-yocto/4.4.14+gitAUTOINC+4800a400d5_8361321fec-r0/image/boot/config-4.4.14-yocto-standard | grep '^+C' > +CONFIG_CONSOLE_POLL=y > +CONFIG_PRINTK_TIME=y > +CONFIG_DEBUG_INFO=y > +CONFIG_DEBUG_KERNEL=y > +CONFIG_SCHED_DEBUG=y > +CONFIG_DEBUG_PREEMPT=y > +CONFIG_KGDB=y > +CONFIG_KGDB_SERIAL_CONSOLE=y > +CONFIG_KGDB_LOW_LEVEL_TRAP=y > +CONFIG_KGDB_KDB=y > +CONFIG_KDB_DEFAULT_ENABLE=0x1 > +CONFIG_KDB_KEYBOARD=y > +CONFIG_KDB_CONTINUE_CATASTROPHIC=0 > +CONFIG_EARLY_PRINTK_DBGP=y > +CONFIG_EARLY_PRINTK_EFI=y > +CONFIG_DEBUG_RODATA=y > +CONFIG_DEBUG_RODATA_TEST=y > +CONFIG_X86_DEBUG_FPU=y > > I guess the reason of this is that printk.scc includes debug-kernel.scc, > which brings more config options: > CONFIG_DEBUG_KERNEL=y > CONFIG_DEBUG_INFO=y > CONFIG_DEBUG_PREEMPT=y > > That probably explains the difference in kernel size and compile time. > Yup. > After removing 'include debug-kernel.scc' the difference in > configuration became more reasonable: > +CONFIG_PRINTK_TIME=y > +CONFIG_EARLY_PRINTK_DBGP=y > +CONFIG_EARLY_PRINTK_EFI=y > > And the size of kernel and modules became almost the same as before > enabling printk feature. > > Considering that the rest of the options from printk.scc don't appear in > the result configuration even if debug-kernel.scc is included I hope > it should be ok to remove 'include debug-kernel.scc' from printk.scc. > > Does it make sense for you? It does to me. Since we actually have a "developer" kernel type that is intended for this purpose. Anything that we add to the standard kernel type should be more surgical. Bruce > >>> Here are results of /usr/bin/time -v bitbake core-image-minimal for this >>> and previous commits: >>> >>> linux-yocto: Enable debug/printk.scc KERNEL_FEATURE on qemuall machines: d55b7650d305ffad953dd36595ee6b9aa8df5a72 >>> Command being timed: "bitbake core-image-minimal" >>> User time (seconds): 14966.24 >>> System time (seconds): 3829.23 >>> Percent of CPU this job got: 1212% >>> Elapsed (wall clock) time (h:mm:ss or m:ss): 25:50.52 >>> Average shared text size (kbytes): 0 >>> Average unshared data size (kbytes): 0 >>> Average stack size (kbytes): 0 >>> Average total size (kbytes): 0 >>> Maximum resident set size (kbytes): 917592 >>> Average resident set size (kbytes): 0 >>> Major (requiring I/O) page faults: 8105 >>> Minor (reclaiming a frame) page faults: 628476540 >>> Voluntary context switches: 26886617 >>> Involuntary context switches: 2033535 >>> Swaps: 0 >>> File system inputs: 3160048 >>> File system outputs: 78417720 >>> Socket messages sent: 0 >>> Socket messages received: 0 >>> Signals delivered: 0 >>> Page size (bytes): 4096 >>> Exit status: 0 >>> >>> commit: nativesdk-packagegroup-sdk-host: Add wayland-scanner to the SDK native toolchain: e833508075bcb8f2d1ccddcf46f579aea7a2ab1d >>> Command being timed: "bitbake core-image-minimal" >>> User time (seconds): 14526.89 >>> System time (seconds): 3770.74 >>> Percent of CPU this job got: 1193% >>> Elapsed (wall clock) time (h:mm:ss or m:ss): 25:33.44 >>> Average shared text size (kbytes): 0 >>> Average unshared data size (kbytes): 0 >>> Average stack size (kbytes): 0 >>> Average total size (kbytes): 0 >>> Maximum resident set size (kbytes): 917792 >>> Average resident set size (kbytes): 0 >>> Major (requiring I/O) page faults: 8058 >>> Minor (reclaiming a frame) page faults: 618283939 >>> Voluntary context switches: 26538183 >>> Involuntary context switches: 1859365 >>> Swaps: 0 >>> File system inputs: 3191584 >>> File system outputs: 66411592 >>> Socket messages sent: 0 >>> Socket messages received: 0 >>> Signals delivered: 0 >>> Page size (bytes): 4096 >>> Exit status: 0 >>> >>> The total time (user time + system time) incresed by 2.72% which seems quite a lot. >>> >>> diff of tasks utime shows biggest difference in do_package* tasks. This is probably >>> due to increased size of packages: >>> >>> do_bundle_initramfs:utime: 21 | do_bundle_initramfs:utime: 32 >>> do_compile:utime: 47 | do_compile:utime: 51 >>> do_compile_kernelmodules:utime: 39 | do_compile_kernelmodules:utime: 42 >>> do_configure:utime: 20 | do_configure:utime: 24 >>> do_deploy:utime: 38 | do_deploy:utime: 35 >>> do_fetch:utime: 25 | do_fetch:utime: 28 >>> do_install:utime: 26 | do_install:utime: 22 >>> do_kernel_checkout:utime: 21 | do_kernel_checkout:utime: 24 >>> do_kernel_configcheck:utime: 21 | do_kernel_configcheck:utime: 21 >>> do_kernel_configme:utime: 23 | do_kernel_configme:utime: 23 >>> do_kernel_link_images:utime: 24 | do_kernel_link_images:utime: 22 >>> do_kernel_metadata:utime: 25 | do_kernel_metadata:utime: 23 >>> do_package:utime: 396 | do_package:utime: 378 >>> do_packagedata:utime: 35 | do_packagedata:utime: 34 >>> do_package_qa:utime: 460 | do_package_qa:utime: 492 >>> do_package_write_rpm:utime: 431 | do_package_write_rpm:utime: 399 >>> do_patch:utime: 25 | do_patch:utime: 25 >>> do_populate_lic:utime: 30 | do_populate_lic:utime: 29 >>> do_populate_sysroot:utime: 33 | do_populate_sysroot:utime: 30 >>> do_shared_workdir:utime: 24 | do_shared_workdir:utime: 25 >>> do_sizecheck:utime: 23 | do_sizecheck:utime: 21 >>> do_strip:utime: 23 | do_strip:utime: 23 >>> do_uboot_mkimage:utime: 25 | do_uboot_mkimage:utime: 24 >>> do_unpack:utime: 21 | do_unpack:utime: 22 >>> do_validate_branches:utime: 23 | do_validate_branches:utime: 24 >>> >>> There is also quite big increase in consumed disk space by the kernel >>> and modules: >>> $ ls -lh ./tmp.bad/work/qemux86-poky-linux/linux-yocto/4.4.14+gitAUTOINC+4800a400d5_8361321fec-r0/image/boot/vmlinux-4.4.14-yocto-standard >>> -rw-r--r-- 1 ed ed 261M Jul 25 12:39 ./tmp.bad/work/qemux86-poky-linux/linux-yocto/4.4.14+gitAUTOINC+4800a400d5_8361321fec-r0/image/boot/vmlinux-4.4.14-yocto-standard >>> $ ls -lh ./tmp.prev/work/qemux86-poky-linux/linux-yocto/4.4.14+gitAUTOINC+4800a400d5_8361321fec-r0/image/boot/vmlinux-4.4.14-yocto-standard >>> -rw-r--r-- 1 ed ed 19M Jul 25 17:34 ./tmp.prev/work/qemux86-poky-linux/linux-yocto/4.4.14+gitAUTOINC+4800a400d5_8361321fec-r0/image/boot/vmlinux-4.4.14-yocto-standard >>> >>> $ ls -lh tmp.bad/deploy/images/qemux86/modules--4.4.14+git0+4800a400d5_8361321fec-r0-qemux86-20160725092744.tgz >>> -rw-rw-r-- 2 ed ed 99M Jul 25 12:39 tmp.bad/deploy/images/qemux86/modules--4.4.14+git0+4800a400d5_8361321fec-r0-qemux86-20160725092744.tgz >>> $ ls -lh tmp.prev/deploy/images/qemux86/modules--4.4.14+git0+4800a400d5_8361321fec-r0-qemux86-20160725105810.tgz >>> -rw-rw-r-- 1 ed ed 4.3M Jul 25 17:34 tmp.prev/deploy/images/qemux86/modules--4.4.14+git0+4800a400d5_8361321fec-r0-qemux86-20160725105810.tgz >>> >>> Any thoughts on this? Is it normal or it should be tested further and fixed? >> >> I'm not sure that this is related to the debug printk build, but >> anything that that could be increasing footprint and build time >> like that should be looked into. >> >> Without messing with the oe-core commit that enabled the feature, has >> someone done a build, gone into the kernel .config, disabled the options >> and then rebuilt ? I'd start there to rule out that the option is >> actually doing something that significant to the build itself. >> >> .. but I can't say much more than that, since as you can see by the >> commit, I'm not on the sign-off list, so I don't have all the background. >> >> Bruce >> >>> >>> -- >>> Regards, >>> Ed >>> >> >> -- >> _______________________________________________ >> Openembedded-core mailing list >> Openembedded-core@lists.openembedded.org >> http://lists.openembedded.org/mailman/listinfo/openembedded-core > > ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: 2.7% build time regression caused by enabling debug/printk.scc KERNEL_FEATURE] 2016-07-27 13:06 ` Bruce Ashfield @ 2016-07-27 13:18 ` Ed Bartosh 2016-07-27 14:31 ` Bruce Ashfield 0 siblings, 1 reply; 7+ messages in thread From: Ed Bartosh @ 2016-07-27 13:18 UTC (permalink / raw) To: Bruce Ashfield; +Cc: openembedded-core On Wed, Jul 27, 2016 at 09:06:44AM -0400, Bruce Ashfield wrote: > On 2016-07-27 08:28 AM, Ed Bartosh wrote: > >Hi Bruce, > > > >Thank you for your suggestion to experiment with configuration options! > > > >>On 2016-07-26 10:32 AM, Ed Bartosh wrote: > >>>Hi all, > >>> > >>>We've noticed quite big increase of core-image-minimal build time caused by commit > >>>d55b7650d305ffad953dd36595ee6b9aa8df5a72 linux-yocto: Enablei debug/printk.scc KERNEL_FEATURE on qemuall machines. > >>> > >> > >>That commit only enables the following options: > >> > >>CONFIG_PRINTK=y > >>CONFIG_PRINTK_TIME=y > >> > >>CONFIG_EARLY_PRINTK=y > >> > >>CONFIG_EARLY_PRINTK_DBGP=y > >>CONFIG_EARLY_PRINTK_EFI=y > >>CONFIG_TTY_PRINTK=y > >> > >>And yes, that will add some size to the kernel, but I'm not seeing > >>similar size increases here. > >> > >>If you take a look through the kernel source and build, there are > >>relatively few additions to the actual kernel build that change > >>based on those options, and most of them are runtime changes versus > >>build-time changes. > >> > > > >The actuall difference in configuration is bigger as far as I can see: > >$ diff -u ../build-kernel/tmp/work/qemux86_64-poky-linux/linux-yocto/4.4.14+gitAUTOINC+4800a400d5_8361321fec-r0/image/boot/config-4.4.14-yocto-standard > >tmp/work/qemux86_64-poky-linux/linux-yocto/4.4.14+gitAUTOINC+4800a400d5_8361321fec-r0/image/boot/config-4.4.14-yocto-standard | grep '^+C' > >+CONFIG_CONSOLE_POLL=y > >+CONFIG_PRINTK_TIME=y > >+CONFIG_DEBUG_INFO=y > >+CONFIG_DEBUG_KERNEL=y > >+CONFIG_SCHED_DEBUG=y > >+CONFIG_DEBUG_PREEMPT=y > >+CONFIG_KGDB=y > >+CONFIG_KGDB_SERIAL_CONSOLE=y > >+CONFIG_KGDB_LOW_LEVEL_TRAP=y > >+CONFIG_KGDB_KDB=y > >+CONFIG_KDB_DEFAULT_ENABLE=0x1 > >+CONFIG_KDB_KEYBOARD=y > >+CONFIG_KDB_CONTINUE_CATASTROPHIC=0 > >+CONFIG_EARLY_PRINTK_DBGP=y > >+CONFIG_EARLY_PRINTK_EFI=y > >+CONFIG_DEBUG_RODATA=y > >+CONFIG_DEBUG_RODATA_TEST=y > >+CONFIG_X86_DEBUG_FPU=y > > > >I guess the reason of this is that printk.scc includes debug-kernel.scc, > >which brings more config options: > >CONFIG_DEBUG_KERNEL=y > >CONFIG_DEBUG_INFO=y > >CONFIG_DEBUG_PREEMPT=y > > > >That probably explains the difference in kernel size and compile time. > > > > Yup. > > >After removing 'include debug-kernel.scc' the difference in > >configuration became more reasonable: > >+CONFIG_PRINTK_TIME=y > >+CONFIG_EARLY_PRINTK_DBGP=y > >+CONFIG_EARLY_PRINTK_EFI=y > > > >And the size of kernel and modules became almost the same as before > >enabling printk feature. > > > >Considering that the rest of the options from printk.scc don't appear in > >the result configuration even if debug-kernel.scc is included I hope > >it should be ok to remove 'include debug-kernel.scc' from printk.scc. > > > >Does it make sense for you? > > It does to me. Since we actually have a "developer" kernel type that > is intended for this purpose. Anything that we add to the standard > kernel type should be more surgical. > Looking at guilty commit https://git.yoctoproject.org/cgit/cgit.cgi/yocto-kernel-cache/commit/?h=yocto-4.4&id=fd8d90ca69f53d425fdb34fc9a9debac1d0c5f52 I'd say that the rest of scc files should be also tested the same way. We may not need to enable all 3 CONFIG_DEBUG options for each of them. -- Regards, Ed ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: 2.7% build time regression caused by enabling debug/printk.scc KERNEL_FEATURE] 2016-07-27 13:18 ` Ed Bartosh @ 2016-07-27 14:31 ` Bruce Ashfield 2016-07-27 20:06 ` Sullivan, California L 0 siblings, 1 reply; 7+ messages in thread From: Bruce Ashfield @ 2016-07-27 14:31 UTC (permalink / raw) To: ed.bartosh; +Cc: openembedded-core On 2016-07-27 09:18 AM, Ed Bartosh wrote: > On Wed, Jul 27, 2016 at 09:06:44AM -0400, Bruce Ashfield wrote: >> On 2016-07-27 08:28 AM, Ed Bartosh wrote: >>> Hi Bruce, >>> >>> Thank you for your suggestion to experiment with configuration options! >>> >>>> On 2016-07-26 10:32 AM, Ed Bartosh wrote: >>>>> Hi all, >>>>> >>>>> We've noticed quite big increase of core-image-minimal build time caused by commit >>>>> d55b7650d305ffad953dd36595ee6b9aa8df5a72 linux-yocto: Enablei debug/printk.scc KERNEL_FEATURE on qemuall machines. >>>>> >>>> >>>> That commit only enables the following options: >>>> >>>> CONFIG_PRINTK=y >>>> CONFIG_PRINTK_TIME=y >>>> >>>> CONFIG_EARLY_PRINTK=y >>>> >>>> CONFIG_EARLY_PRINTK_DBGP=y >>>> CONFIG_EARLY_PRINTK_EFI=y >>>> CONFIG_TTY_PRINTK=y >>>> >>>> And yes, that will add some size to the kernel, but I'm not seeing >>>> similar size increases here. >>>> >>>> If you take a look through the kernel source and build, there are >>>> relatively few additions to the actual kernel build that change >>>> based on those options, and most of them are runtime changes versus >>>> build-time changes. >>>> >>> >>> The actuall difference in configuration is bigger as far as I can see: >>> $ diff -u ../build-kernel/tmp/work/qemux86_64-poky-linux/linux-yocto/4.4.14+gitAUTOINC+4800a400d5_8361321fec-r0/image/boot/config-4.4.14-yocto-standard >>> tmp/work/qemux86_64-poky-linux/linux-yocto/4.4.14+gitAUTOINC+4800a400d5_8361321fec-r0/image/boot/config-4.4.14-yocto-standard | grep '^+C' >>> +CONFIG_CONSOLE_POLL=y >>> +CONFIG_PRINTK_TIME=y >>> +CONFIG_DEBUG_INFO=y >>> +CONFIG_DEBUG_KERNEL=y >>> +CONFIG_SCHED_DEBUG=y >>> +CONFIG_DEBUG_PREEMPT=y >>> +CONFIG_KGDB=y >>> +CONFIG_KGDB_SERIAL_CONSOLE=y >>> +CONFIG_KGDB_LOW_LEVEL_TRAP=y >>> +CONFIG_KGDB_KDB=y >>> +CONFIG_KDB_DEFAULT_ENABLE=0x1 >>> +CONFIG_KDB_KEYBOARD=y >>> +CONFIG_KDB_CONTINUE_CATASTROPHIC=0 >>> +CONFIG_EARLY_PRINTK_DBGP=y >>> +CONFIG_EARLY_PRINTK_EFI=y >>> +CONFIG_DEBUG_RODATA=y >>> +CONFIG_DEBUG_RODATA_TEST=y >>> +CONFIG_X86_DEBUG_FPU=y >>> >>> I guess the reason of this is that printk.scc includes debug-kernel.scc, >>> which brings more config options: >>> CONFIG_DEBUG_KERNEL=y >>> CONFIG_DEBUG_INFO=y >>> CONFIG_DEBUG_PREEMPT=y >>> >>> That probably explains the difference in kernel size and compile time. >>> >> >> Yup. >> >>> After removing 'include debug-kernel.scc' the difference in >>> configuration became more reasonable: >>> +CONFIG_PRINTK_TIME=y >>> +CONFIG_EARLY_PRINTK_DBGP=y >>> +CONFIG_EARLY_PRINTK_EFI=y >>> >>> And the size of kernel and modules became almost the same as before >>> enabling printk feature. >>> >>> Considering that the rest of the options from printk.scc don't appear in >>> the result configuration even if debug-kernel.scc is included I hope >>> it should be ok to remove 'include debug-kernel.scc' from printk.scc. >>> >>> Does it make sense for you? >> >> It does to me. Since we actually have a "developer" kernel type that >> is intended for this purpose. Anything that we add to the standard >> kernel type should be more surgical. >> > > Looking at guilty commit > https://git.yoctoproject.org/cgit/cgit.cgi/yocto-kernel-cache/commit/?h=yocto-4.4&id=fd8d90ca69f53d425fdb34fc9a9debac1d0c5f52 > I'd say that the rest of scc files should be also tested the same way. > We may not need to enable all 3 CONFIG_DEBUG options for each of them. Adding Cal, since he did the work on this split up. The profiling feature makes sense with debug kernel being included, but I can see some latency top use cases that would be valid with a non developer mode kernel (i.e. the standard kernel). But yes, we don't want fragments that could be included by the standard kernel to add typical functionality to add more than they need to function. Further tweaking can be done, its part of the iterative process on these fragments as we work through use cases. Bruce > > > -- > Regards, > Ed > ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: 2.7% build time regression caused by enabling debug/printk.scc KERNEL_FEATURE] 2016-07-27 14:31 ` Bruce Ashfield @ 2016-07-27 20:06 ` Sullivan, California L 2016-07-27 20:58 ` Aníbal Limón 0 siblings, 1 reply; 7+ messages in thread From: Sullivan, California L @ 2016-07-27 20:06 UTC (permalink / raw) To: Ashfield, Bruce (Wind River), ed.bartosh@linux.intel.com Cc: openembedded-core@lists.openembedded.org Adding the debug-kernel fragment to the printk fragment was probably a mistake on my part. I don't see anything that requires it. I'm also seeing another issue: TTY_PRINTK depends on EXPERT, which is only enabled on the developer kernel by default, so you currently won't be getting that option. From there, if we were to enable EXPERT, DEBUG_KERNEL then gets selected automatically and we essentially end up where we started and have the build time regression. Stuff to think about on my end I suppose. In the mean time I +1 removing the debug-kernel include from printk.scc. --- Cal On 07/27/2016 07:31 AM, Bruce Ashfield wrote: > On 2016-07-27 09:18 AM, Ed Bartosh wrote: >> On Wed, Jul 27, 2016 at 09:06:44AM -0400, Bruce Ashfield wrote: >>> On 2016-07-27 08:28 AM, Ed Bartosh wrote: >>>> Hi Bruce, >>>> >>>> Thank you for your suggestion to experiment with configuration options! >>>> >>>>> On 2016-07-26 10:32 AM, Ed Bartosh wrote: >>>>>> Hi all, >>>>>> >>>>>> We've noticed quite big increase of core-image-minimal build time caused by commit >>>>>> d55b7650d305ffad953dd36595ee6b9aa8df5a72 linux-yocto: Enablei debug/printk.scc KERNEL_FEATURE on qemuall machines. >>>>>> >>>>> That commit only enables the following options: >>>>> >>>>> CONFIG_PRINTK=y >>>>> CONFIG_PRINTK_TIME=y >>>>> >>>>> CONFIG_EARLY_PRINTK=y >>>>> >>>>> CONFIG_EARLY_PRINTK_DBGP=y >>>>> CONFIG_EARLY_PRINTK_EFI=y >>>>> CONFIG_TTY_PRINTK=y >>>>> >>>>> And yes, that will add some size to the kernel, but I'm not seeing >>>>> similar size increases here. >>>>> >>>>> If you take a look through the kernel source and build, there are >>>>> relatively few additions to the actual kernel build that change >>>>> based on those options, and most of them are runtime changes versus >>>>> build-time changes. >>>>> >>>> The actuall difference in configuration is bigger as far as I can see: >>>> $ diff -u ../build-kernel/tmp/work/qemux86_64-poky-linux/linux-yocto/4.4.14+gitAUTOINC+4800a400d5_8361321fec-r0/image/boot/config-4.4.14-yocto-standard >>>> tmp/work/qemux86_64-poky-linux/linux-yocto/4.4.14+gitAUTOINC+4800a400d5_8361321fec-r0/image/boot/config-4.4.14-yocto-standard | grep '^+C' >>>> +CONFIG_CONSOLE_POLL=y >>>> +CONFIG_PRINTK_TIME=y >>>> +CONFIG_DEBUG_INFO=y >>>> +CONFIG_DEBUG_KERNEL=y >>>> +CONFIG_SCHED_DEBUG=y >>>> +CONFIG_DEBUG_PREEMPT=y >>>> +CONFIG_KGDB=y >>>> +CONFIG_KGDB_SERIAL_CONSOLE=y >>>> +CONFIG_KGDB_LOW_LEVEL_TRAP=y >>>> +CONFIG_KGDB_KDB=y >>>> +CONFIG_KDB_DEFAULT_ENABLE=0x1 >>>> +CONFIG_KDB_KEYBOARD=y >>>> +CONFIG_KDB_CONTINUE_CATASTROPHIC=0 >>>> +CONFIG_EARLY_PRINTK_DBGP=y >>>> +CONFIG_EARLY_PRINTK_EFI=y >>>> +CONFIG_DEBUG_RODATA=y >>>> +CONFIG_DEBUG_RODATA_TEST=y >>>> +CONFIG_X86_DEBUG_FPU=y >>>> >>>> I guess the reason of this is that printk.scc includes debug-kernel.scc, >>>> which brings more config options: >>>> CONFIG_DEBUG_KERNEL=y >>>> CONFIG_DEBUG_INFO=y >>>> CONFIG_DEBUG_PREEMPT=y >>>> >>>> That probably explains the difference in kernel size and compile time. >>>> >>> Yup. >>> >>>> After removing 'include debug-kernel.scc' the difference in >>>> configuration became more reasonable: >>>> +CONFIG_PRINTK_TIME=y >>>> +CONFIG_EARLY_PRINTK_DBGP=y >>>> +CONFIG_EARLY_PRINTK_EFI=y >>>> >>>> And the size of kernel and modules became almost the same as before >>>> enabling printk feature. >>>> >>>> Considering that the rest of the options from printk.scc don't appear in >>>> the result configuration even if debug-kernel.scc is included I hope >>>> it should be ok to remove 'include debug-kernel.scc' from printk.scc. >>>> >>>> Does it make sense for you? >>> It does to me. Since we actually have a "developer" kernel type that >>> is intended for this purpose. Anything that we add to the standard >>> kernel type should be more surgical. >>> >> Looking at guilty commit >> https://git.yoctoproject.org/cgit/cgit.cgi/yocto-kernel-cache/commit/?h=yocto-4.4&id=fd8d90ca69f53d425fdb34fc9a9debac1d0c5f52 >> I'd say that the rest of scc files should be also tested the same way. >> We may not need to enable all 3 CONFIG_DEBUG options for each of them. > Adding Cal, since he did the work on this split up. > > The profiling feature makes sense with debug kernel being included, but > I can see some latency top use cases that would be valid with a non > developer mode kernel (i.e. the standard kernel). > > But yes, we don't want fragments that could be included by the standard > kernel to add typical functionality to add more than they need to > function. Further tweaking can be done, its part of the iterative > process on these fragments as we work through use cases. > > Bruce > >> >> -- >> Regards, >> Ed >> > ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: 2.7% build time regression caused by enabling debug/printk.scc KERNEL_FEATURE] 2016-07-27 20:06 ` Sullivan, California L @ 2016-07-27 20:58 ` Aníbal Limón 2016-07-28 19:26 ` Ed Bartosh 0 siblings, 1 reply; 7+ messages in thread From: Aníbal Limón @ 2016-07-27 20:58 UTC (permalink / raw) To: Sullivan, California L, Ashfield, Bruce (Wind River), ed.bartosh@linux.intel.com Cc: openembedded-core@lists.openembedded.org [-- Attachment #1: Type: text/plain, Size: 4995 bytes --] On 07/27/2016 03:06 PM, Sullivan, California L wrote: > Adding the debug-kernel fragment to the printk fragment was probably a > mistake on my part. I don't see anything that requires it. > > I'm also seeing another issue: TTY_PRINTK depends on EXPERT, which is > only enabled on the developer kernel by default, so you currently won't > be getting that option. From there, if we were to enable EXPERT, > DEBUG_KERNEL then gets selected automatically and we essentially end up > where we started and have the build time regression. > > Stuff to think about on my end I suppose. In the mean time I +1 removing > the debug-kernel include from printk.scc. I agree, alimon > > --- > Cal > > On 07/27/2016 07:31 AM, Bruce Ashfield wrote: >> On 2016-07-27 09:18 AM, Ed Bartosh wrote: >>> On Wed, Jul 27, 2016 at 09:06:44AM -0400, Bruce Ashfield wrote: >>>> On 2016-07-27 08:28 AM, Ed Bartosh wrote: >>>>> Hi Bruce, >>>>> >>>>> Thank you for your suggestion to experiment with configuration options! >>>>> >>>>>> On 2016-07-26 10:32 AM, Ed Bartosh wrote: >>>>>>> Hi all, >>>>>>> >>>>>>> We've noticed quite big increase of core-image-minimal build time caused by commit >>>>>>> d55b7650d305ffad953dd36595ee6b9aa8df5a72 linux-yocto: Enablei debug/printk.scc KERNEL_FEATURE on qemuall machines. >>>>>>> >>>>>> That commit only enables the following options: >>>>>> >>>>>> CONFIG_PRINTK=y >>>>>> CONFIG_PRINTK_TIME=y >>>>>> >>>>>> CONFIG_EARLY_PRINTK=y >>>>>> >>>>>> CONFIG_EARLY_PRINTK_DBGP=y >>>>>> CONFIG_EARLY_PRINTK_EFI=y >>>>>> CONFIG_TTY_PRINTK=y >>>>>> >>>>>> And yes, that will add some size to the kernel, but I'm not seeing >>>>>> similar size increases here. >>>>>> >>>>>> If you take a look through the kernel source and build, there are >>>>>> relatively few additions to the actual kernel build that change >>>>>> based on those options, and most of them are runtime changes versus >>>>>> build-time changes. >>>>>> >>>>> The actuall difference in configuration is bigger as far as I can see: >>>>> $ diff -u ../build-kernel/tmp/work/qemux86_64-poky-linux/linux-yocto/4.4.14+gitAUTOINC+4800a400d5_8361321fec-r0/image/boot/config-4.4.14-yocto-standard >>>>> tmp/work/qemux86_64-poky-linux/linux-yocto/4.4.14+gitAUTOINC+4800a400d5_8361321fec-r0/image/boot/config-4.4.14-yocto-standard | grep '^+C' >>>>> +CONFIG_CONSOLE_POLL=y >>>>> +CONFIG_PRINTK_TIME=y >>>>> +CONFIG_DEBUG_INFO=y >>>>> +CONFIG_DEBUG_KERNEL=y >>>>> +CONFIG_SCHED_DEBUG=y >>>>> +CONFIG_DEBUG_PREEMPT=y >>>>> +CONFIG_KGDB=y >>>>> +CONFIG_KGDB_SERIAL_CONSOLE=y >>>>> +CONFIG_KGDB_LOW_LEVEL_TRAP=y >>>>> +CONFIG_KGDB_KDB=y >>>>> +CONFIG_KDB_DEFAULT_ENABLE=0x1 >>>>> +CONFIG_KDB_KEYBOARD=y >>>>> +CONFIG_KDB_CONTINUE_CATASTROPHIC=0 >>>>> +CONFIG_EARLY_PRINTK_DBGP=y >>>>> +CONFIG_EARLY_PRINTK_EFI=y >>>>> +CONFIG_DEBUG_RODATA=y >>>>> +CONFIG_DEBUG_RODATA_TEST=y >>>>> +CONFIG_X86_DEBUG_FPU=y >>>>> >>>>> I guess the reason of this is that printk.scc includes debug-kernel.scc, >>>>> which brings more config options: >>>>> CONFIG_DEBUG_KERNEL=y >>>>> CONFIG_DEBUG_INFO=y >>>>> CONFIG_DEBUG_PREEMPT=y >>>>> >>>>> That probably explains the difference in kernel size and compile time. >>>>> >>>> Yup. >>>> >>>>> After removing 'include debug-kernel.scc' the difference in >>>>> configuration became more reasonable: >>>>> +CONFIG_PRINTK_TIME=y >>>>> +CONFIG_EARLY_PRINTK_DBGP=y >>>>> +CONFIG_EARLY_PRINTK_EFI=y >>>>> >>>>> And the size of kernel and modules became almost the same as before >>>>> enabling printk feature. >>>>> >>>>> Considering that the rest of the options from printk.scc don't appear in >>>>> the result configuration even if debug-kernel.scc is included I hope >>>>> it should be ok to remove 'include debug-kernel.scc' from printk.scc. >>>>> >>>>> Does it make sense for you? >>>> It does to me. Since we actually have a "developer" kernel type that >>>> is intended for this purpose. Anything that we add to the standard >>>> kernel type should be more surgical. >>>> >>> Looking at guilty commit >>> https://git.yoctoproject.org/cgit/cgit.cgi/yocto-kernel-cache/commit/?h=yocto-4.4&id=fd8d90ca69f53d425fdb34fc9a9debac1d0c5f52 >>> I'd say that the rest of scc files should be also tested the same way. >>> We may not need to enable all 3 CONFIG_DEBUG options for each of them. >> Adding Cal, since he did the work on this split up. >> >> The profiling feature makes sense with debug kernel being included, but >> I can see some latency top use cases that would be valid with a non >> developer mode kernel (i.e. the standard kernel). >> >> But yes, we don't want fragments that could be included by the standard >> kernel to add typical functionality to add more than they need to >> function. Further tweaking can be done, its part of the iterative >> process on these fragments as we work through use cases. >> >> Bruce >> >>> >>> -- >>> Regards, >>> Ed >>> >> > [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 836 bytes --] ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: 2.7% build time regression caused by enabling debug/printk.scc KERNEL_FEATURE] 2016-07-27 20:58 ` Aníbal Limón @ 2016-07-28 19:26 ` Ed Bartosh 0 siblings, 0 replies; 7+ messages in thread From: Ed Bartosh @ 2016-07-28 19:26 UTC (permalink / raw) To: Aníbal Limón Cc: openembedded-core@lists.openembedded.org, Ashfield, Bruce (Wind River) On Wed, Jul 27, 2016 at 03:58:16PM -0500, Aníbal Limón wrote: > > > On 07/27/2016 03:06 PM, Sullivan, California L wrote: > > Adding the debug-kernel fragment to the printk fragment was probably a > > mistake on my part. I don't see anything that requires it. > > > > I'm also seeing another issue: TTY_PRINTK depends on EXPERT, which is > > only enabled on the developer kernel by default, so you currently won't > > be getting that option. From there, if we were to enable EXPERT, > > DEBUG_KERNEL then gets selected automatically and we essentially end up > > where we started and have the build time regression. > > > > Stuff to think about on my end I suppose. In the mean time I +1 removing > > the debug-kernel include from printk.scc. > > I agree, > > alimon The fix has been merged to yocto-4.4 branch of yocto-kernel-cache: https://git.yoctoproject.org/cgit/cgit.cgi/yocto-kernel-cache/commit/?h=yocto-4.4&id=d3cc76c3cb20247b16c16b2367decf08b5fb90fa And included into last kernel update: http://lists.openembedded.org/pipermail/openembedded-core/2016-July/124466.html With the fix applied kernel build time decreased by 22.84%: Build Configuration: BB_VERSION = "1.31.0" BUILD_SYS = "x86_64-linux" NATIVELSBSTRING = "openSUSE-13.2" TARGET_SYS = "i586-poky-linux" MACHINE = "qemux86" DISTRO = "poky" DISTRO_VERSION = "2.1+snapshot-20160728" TUNE_FEATURES = "m32 i586" TARGET_FPU = "" meta meta-poky meta-yocto-bsp = "master:039f47ad197a9a53109c9f3deadd9c35e62c056d" before the fix: Command being timed: "bitbake virtual/kernel" User time (seconds): 1367.06 System time (seconds): 162.06 Percent of CPU this job got: 372% Elapsed (wall clock) time (h:mm:ss or m:ss): 6:50.56 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 627120 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 1292 Minor (reclaiming a frame) page faults: 61097551 Voluntary context switches: 885976 Involuntary context switches: 715312 Swaps: 0 File system inputs: 1024568 File system outputs: 13316088 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 0 after the fix: Command being timed: "bitbake virtual/kernel" User time (seconds): 1050.97 System time (seconds): 128.84 Percent of CPU this job got: 434% Elapsed (wall clock) time (h:mm:ss or m:ss): 4:31.35 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 387148 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 1264 Minor (reclaiming a frame) page faults: 52467379 Voluntary context switches: 589830 Involuntary context switches: 457987 Swaps: 0 File system inputs: 1107424 File system outputs: 3363464 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 0 And consumed disk space decreased by 89.62%: before the fix: $ du -sh tmp/work/qemux86-poky-linux/linux-yocto/4.4.14+gitAUTOINC+4800a400d5_8361321fec-r0/deploy-linux-yocto/ 106M after the fix: $ du -sh tmp/work/qemux86-poky-linux/linux-yocto/4.4.14+gitAUTOINC+86bf91f444_8361321fec-r0/deploy-linux-yocto/ 11M -- Regards, Ed ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2016-07-28 20:03 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <1469620052.23580.115.camel@linuxfoundation.org>
2016-07-27 12:28 ` 2.7% build time regression caused by enabling debug/printk.scc KERNEL_FEATURE] Ed Bartosh
2016-07-27 13:06 ` Bruce Ashfield
2016-07-27 13:18 ` Ed Bartosh
2016-07-27 14:31 ` Bruce Ashfield
2016-07-27 20:06 ` Sullivan, California L
2016-07-27 20:58 ` Aníbal Limón
2016-07-28 19:26 ` Ed Bartosh
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox