linuxppc-dev.lists.ozlabs.org archive mirror
 help / color / mirror / Atom feed
* [PATCH 00/34] Make kernel build deterministic
@ 2011-04-05 14:58 Michal Marek
  2011-04-05 14:58 ` [PATCH 03/34] powerpc: Call gzip with -n Michal Marek
                   ` (4 more replies)
  0 siblings, 5 replies; 14+ messages in thread
From: Michal Marek @ 2011-04-05 14:58 UTC (permalink / raw)
  To: linux-kbuild
  Cc: anil_ravindranath, mchehab, x86, mac, aacraid, linux-mtd,
	allan.stephens, hpa, netdev, t.sailer, gwingerde, IvDoorn, elf,
	cluster-devel, ccaulfie, mingo, dougthompson, linux-usb,
	linux-media, arnaud.giersch, teigland, tony.olech, apw,
	linux-hams, tglx, swhiteho, linux-arm-kernel, linux-edac,
	jon.maloy, linux-scsi, linuxppc-dev, gregkh, linux-wireless,
	linux-kernel, bluesmoke-devel, James.Bottomley, tipc-discussion,
	chuanxiao.dong, paulus, dwmw2, davem


Hi,

this series makes it possible to build bit-identical kernel image and
modules from identical sources. Of course the build is already
deterministic in terms of behavior of the code, but the various
timestamps embedded in the object files make it hard to compare two
builds, for instance to verify that a makefile cleanup didn't
accidentally change something. A prime example is /proc/config.gz, which
has both a timestamp in the gzip header and a timestamp in the payload
data. With this series applied, a script like this will produce
identical kernels each time:

#!/bin/bash
rm -rf /dev/shm/{source,build}{,1,2}
export KCONFIG_NOTIMESTAMP=1
export KBUILD_BUILD_TIMESTAMP='Sun May  1 12:00:00 CEST 2011'
export KBUILD_BUILD_USER=user
export KBUILD_BUILD_HOST=host
export ROOT_DEV=FLOPPY
for i in 1 2; do
	mkdir /dev/shm/source
	# randomize the inode order just for fun
	git ls-tree -r -z --name-only HEAD | sort -R -z | xargs -0 \
		cp --parents --target=/dev/shm/source
	pushd /dev/shm/source
	mkdir /dev/shm/build
	>/dev/shm/build/all.config
	for opt in GCOV_KERNEL UBIFS_FS_DEBUG; do
		echo "# CONFIG_$opt is not set" >>"/dev/shm/build"/all.config
	done
	make O="/dev/shm/build" KCONFIG_ALLCONFIG=1 allmodconfig
	make O="/dev/shm/build" -j64
	popd
	mv /dev/shm/source /dev/shm/source$i
	mv /dev/shm/build /dev/shm/build$i
done
diff -rq /dev/shm/build{1,2}

Unfortunatelly, this cannot be used to validate indentation-only
patches, even if CONFIG_DEBUG_INFO is turned off. This is because of the
__FILE__ and __LINE__ macros used in many places. For the same reason,
the source and build directory needs to be the same, otherwise the
results will differ. This was tested on
x86_64/{defconfig,allmodconfig,allyesconfig} and ppc64/defconfig. The
series is also available at

git://git.kernel.org/pub/scm/linux/kernel/git/mmarek/kbuild-2.6.git deterministic-build-v1

Michal


Michal Marek (34):
  kconfig: Do not record timestamp in auto.conf and autoconf.h
  kbuild: Call gzip with -n
  powerpc: Call gzip with -n
  kbuild: Use the deterministic mode of ar
  powerpc: Use the deterministic mode of ar
  kbuild: Drop unused LINUX_COMPILE_TIME and LINUX_COMPILE_DOMAIN
    macros
  kbuild: Allow to override LINUX_COMPILE_BY and LINUX_COMPILE_HOST
    macros
  initramfs: Use KBUILD_BUILD_TIMESTAMP for generated entries
  x86: Allow to override the ROOT_DEV variable
  cyclades: Drop __TIME__ usage
  nozomi: Drop __TIME__ usage
  isdn/diva: Drop __TIME__ usage
  media/radio-maxiradio: Drop __TIME__ usage
  media/cx231xx: Drop __TIME__ usage
  baycom: Drop __TIME__ usage
  nand/denali: Drop __TIME__ usage
  hdlcdrv: Drop __TIME__ usage
  wan/pc300: Drop __TIME__ usage
  rt2x00: Drop __TIME__ usage
  parport: Drop __TIME__ usage
  aacraid: Drop __TIME__ usage
  scsi/in2000: Drop __TIME__ usage
  scsi/wd33c93: Drop __TIME__ usage
  usb/u132-hcd: Drop __TIME__ usage
  usb/ftdi-elan: Drop __TIME__ usage
  dlm: Drop __TIME__ usage
  gfs2: Drop __TIME__ usage
  atm: Drop __TIME__ usage
  tipc: Drop __TIME__ usage
  rio: Drop __DATE__ usage
  edac: Drop __DATE__ usage
  pmcraid: Drop __DATE__ usage
  usb/lh7a40x_udc: Drop __DATE__ usage
  checkpatch: Warn about usage of __DATE__, __TIME__ and __TIMESTAMP__

 Documentation/kbuild/kbuild.txt              |   12 ++++++++++
 arch/powerpc/boot/Makefile                   |    2 +-
 arch/powerpc/boot/wrapper                    |    6 ++--
 arch/x86/boot/Makefile                       |    2 +-
 drivers/char/cyclades.c                      |    3 +-
 drivers/char/nozomi.c                        |    3 +-
 drivers/char/rio/rioinit.c                   |    2 +-
 drivers/edac/amd76x_edac.c                   |    2 +-
 drivers/edac/amd8111_edac.c                  |    2 +-
 drivers/edac/amd8131_edac.c                  |    2 +-
 drivers/edac/cpc925_edac.c                   |    2 +-
 drivers/edac/e752x_edac.c                    |    2 +-
 drivers/edac/e7xxx_edac.c                    |    2 +-
 drivers/edac/edac_module.c                   |    2 +-
 drivers/edac/i5000_edac.c                    |    2 +-
 drivers/edac/i5400_edac.c                    |    2 +-
 drivers/edac/i7300_edac.c                    |    2 +-
 drivers/edac/i7core_edac.c                   |    2 +-
 drivers/edac/i82860_edac.c                   |    2 +-
 drivers/edac/i82875p_edac.c                  |    2 +-
 drivers/edac/i82975x_edac.c                  |    2 +-
 drivers/edac/mpc85xx_edac.h                  |    2 +-
 drivers/edac/mv64x60_edac.h                  |    2 +-
 drivers/edac/ppc4xx_edac.c                   |    2 +-
 drivers/edac/r82600_edac.c                   |    2 +-
 drivers/isdn/hardware/eicon/divasfunc.c      |    5 +--
 drivers/media/radio/radio-maxiradio.c        |    3 +-
 drivers/media/video/cx231xx/cx231xx-avcore.c |    2 +-
 drivers/mtd/nand/denali.c                    |    3 +-
 drivers/net/hamradio/baycom_epp.c            |    2 +-
 drivers/net/hamradio/baycom_par.c            |    2 +-
 drivers/net/hamradio/baycom_ser_fdx.c        |    2 +-
 drivers/net/hamradio/baycom_ser_hdx.c        |    2 +-
 drivers/net/hamradio/hdlcdrv.c               |    2 +-
 drivers/net/wan/pc300_drv.c                  |    3 +-
 drivers/net/wireless/rt2x00/rt2x00debug.c    |    1 -
 drivers/parport/parport_ip32.c               |    1 -
 drivers/scsi/aacraid/linit.c                 |    3 +-
 drivers/scsi/in2000.c                        |    2 +-
 drivers/scsi/pmcraid.c                       |    9 +++----
 drivers/scsi/pmcraid.h                       |    1 -
 drivers/scsi/wd33c93.c                       |    7 ++---
 drivers/usb/gadget/lh7a40x_udc.c             |    7 ++---
 drivers/usb/host/u132-hcd.c                  |    3 +-
 drivers/usb/misc/ftdi-elan.c                 |    3 +-
 fs/dlm/main.c                                |    2 +-
 fs/gfs2/main.c                               |    2 +-
 net/atm/lec.c                                |    2 +-
 net/atm/mpc.c                                |    2 +-
 net/tipc/core.c                              |    3 +-
 scripts/Makefile.build                       |    4 +-
 scripts/Makefile.lib                         |    2 +-
 scripts/checkpatch.pl                        |    5 ++++
 scripts/gen_initramfs_list.sh                |    8 +++++-
 scripts/kconfig/confdata.c                   |    8 +-----
 scripts/mkcompile_h                          |   30 ++++++++++++-------------
 usr/gen_init_cpio.c                          |   17 +++++++++-----
 57 files changed, 109 insertions(+), 105 deletions(-)

-- 
1.7.4.1

^ permalink raw reply	[flat|nested] 14+ messages in thread

* [PATCH 03/34] powerpc: Call gzip with -n
  2011-04-05 14:58 [PATCH 00/34] Make kernel build deterministic Michal Marek
@ 2011-04-05 14:58 ` Michal Marek
  2011-04-05 14:58 ` [PATCH 05/34] powerpc: Use the deterministic mode of ar Michal Marek
                   ` (3 subsequent siblings)
  4 siblings, 0 replies; 14+ messages in thread
From: Michal Marek @ 2011-04-05 14:58 UTC (permalink / raw)
  To: linux-kbuild; +Cc: linuxppc-dev, Paul Mackerras, linux-kernel

The timestamps recorded in the .gz files add no value.

Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: linuxppc-dev@lists.ozlabs.org
Signed-off-by: Michal Marek <mmarek@suse.cz>
---
 arch/powerpc/boot/wrapper |    6 +++---
 1 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/arch/powerpc/boot/wrapper b/arch/powerpc/boot/wrapper
index cb97e75..854797b 100755
--- a/arch/powerpc/boot/wrapper
+++ b/arch/powerpc/boot/wrapper
@@ -251,7 +251,7 @@ if [ -z "$cacheit" -o ! -f "$vmz$gzip" -o "$vmz$gzip" -ot "$kernel" ]; then
     ${CROSS}objcopy $objflags "$kernel" "$vmz.$$"
 
     if [ -n "$gzip" ]; then
-        gzip -f -9 "$vmz.$$"
+        gzip -n -f -9 "$vmz.$$"
     fi
 
     if [ -n "$cacheit" ]; then
@@ -336,7 +336,7 @@ coff)
     $objbin/hack-coff "$ofile"
     ;;
 cuboot*)
-    gzip -f -9 "$ofile"
+    gzip -n -f -9 "$ofile"
     ${MKIMAGE} -A ppc -O linux -T kernel -C gzip -a "$base" -e "$entry" \
             $uboot_version -d "$ofile".gz "$ofile"
     ;;
@@ -383,6 +383,6 @@ ps3)
 
     odir="$(dirname "$ofile.bin")"
     rm -f "$odir/otheros.bld"
-    gzip --force -9 --stdout "$ofile.bin" > "$odir/otheros.bld"
+    gzip -n --force -9 --stdout "$ofile.bin" > "$odir/otheros.bld"
     ;;
 esac
-- 
1.7.4.1

^ permalink raw reply related	[flat|nested] 14+ messages in thread

* [PATCH 05/34] powerpc: Use the deterministic mode of ar
  2011-04-05 14:58 [PATCH 00/34] Make kernel build deterministic Michal Marek
  2011-04-05 14:58 ` [PATCH 03/34] powerpc: Call gzip with -n Michal Marek
@ 2011-04-05 14:58 ` Michal Marek
  2011-04-05 15:49 ` [PATCH 00/34] Make kernel build deterministic Greg KH
                   ` (2 subsequent siblings)
  4 siblings, 0 replies; 14+ messages in thread
From: Michal Marek @ 2011-04-05 14:58 UTC (permalink / raw)
  To: linux-kbuild; +Cc: linuxppc-dev, Paul Mackerras, linux-kernel

Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: linuxppc-dev@lists.ozlabs.org
Signed-off-by: Michal Marek <mmarek@suse.cz>
---
 arch/powerpc/boot/Makefile |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/arch/powerpc/boot/Makefile b/arch/powerpc/boot/Makefile
index 8917816..d028a65 100644
--- a/arch/powerpc/boot/Makefile
+++ b/arch/powerpc/boot/Makefile
@@ -127,7 +127,7 @@ quiet_cmd_bootas = BOOTAS  $@
       cmd_bootas = $(CROSS32CC) -Wp,-MD,$(depfile) $(BOOTAFLAGS) -c -o $@ $<
 
 quiet_cmd_bootar = BOOTAR  $@
-      cmd_bootar = $(CROSS32AR) -cr $@.$$$$ $(filter-out FORCE,$^); mv $@.$$$$ $@
+      cmd_bootar = $(CROSS32AR) -crD $@.$$$$ $(filter-out FORCE,$^); mv $@.$$$$ $@
 
 $(obj-libfdt): $(obj)/%.o: $(srctree)/scripts/dtc/libfdt/%.c FORCE
 	$(call if_changed_dep,bootcc)
-- 
1.7.4.1

^ permalink raw reply related	[flat|nested] 14+ messages in thread

* Re: [PATCH 00/34] Make kernel build deterministic
  2011-04-05 14:58 [PATCH 00/34] Make kernel build deterministic Michal Marek
  2011-04-05 14:58 ` [PATCH 03/34] powerpc: Call gzip with -n Michal Marek
  2011-04-05 14:58 ` [PATCH 05/34] powerpc: Use the deterministic mode of ar Michal Marek
@ 2011-04-05 15:49 ` Greg KH
  2011-04-05 18:16   ` James Bottomley
                     ` (2 more replies)
  2011-04-06  9:01 ` Ingo Molnar
  2011-05-05 15:22 ` [PATCH v2] powerpc: Use the deterministic mode of ar Michal Marek
  4 siblings, 3 replies; 14+ messages in thread
From: Greg KH @ 2011-04-05 15:49 UTC (permalink / raw)
  To: Michal Marek
  Cc: anil_ravindranath, mchehab, x86, mac, aacraid, linux-mtd,
	allan.stephens, hpa, netdev, t.sailer, gwingerde, IvDoorn, elf,
	cluster-devel, ccaulfie, mingo, dougthompson, linux-media,
	arnaud.giersch, linux-kbuild, teigland, tony.olech, apw,
	linux-hams, tglx, swhiteho, linux-arm-kernel, linux-edac,
	jon.maloy, linux-scsi, linuxppc-dev, linux-usb, linux-wireless,
	linux-kernel, bluesmoke-devel, James.Bottomley, tipc-discussion,
	chuanxiao.dong, paulus, dwmw2, davem

On Tue, Apr 05, 2011 at 04:58:47PM +0200, Michal Marek wrote:
> 
> Hi,
> 
> this series makes it possible to build bit-identical kernel image and
> modules from identical sources. Of course the build is already
> deterministic in terms of behavior of the code, but the various
> timestamps embedded in the object files make it hard to compare two
> builds, for instance to verify that a makefile cleanup didn't
> accidentally change something. A prime example is /proc/config.gz, which
> has both a timestamp in the gzip header and a timestamp in the payload
> data. With this series applied, a script like this will produce
> identical kernels each time:

Very nice stuff.  Do you want to take the individual patches through one
of your trees, or do you mind if the subsystem maintainers take them
through theirs?

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [PATCH 00/34] Make kernel build deterministic
  2011-04-05 15:49 ` [PATCH 00/34] Make kernel build deterministic Greg KH
@ 2011-04-05 18:16   ` James Bottomley
  2011-04-05 18:29     ` Mauro Carvalho Chehab
  2011-04-05 19:24   ` Artem Bityutskiy
  2011-04-06  9:23   ` Michal Marek
  2 siblings, 1 reply; 14+ messages in thread
From: James Bottomley @ 2011-04-05 18:16 UTC (permalink / raw)
  To: Greg KH
  Cc: anil_ravindranath, mchehab, x86, mac, aacraid, linux-mtd,
	allan.stephens, hpa, netdev, t.sailer, gwingerde, IvDoorn, elf,
	cluster-devel, ccaulfie, mingo, dougthompson, linux-media,
	jon.maloy, arnaud.giersch, linux-kbuild, teigland, tony.olech,
	apw, linux-hams, tglx, swhiteho, linux-arm-kernel, linux-edac,
	Michal Marek, linux-scsi, linuxppc-dev, linux-usb, linux-wireless,
	linux-kernel, bluesmoke-devel, tipc-discussion, chuanxiao.dong,
	paulus, dwmw2, davem

On Tue, 2011-04-05 at 08:49 -0700, Greg KH wrote:
> On Tue, Apr 05, 2011 at 04:58:47PM +0200, Michal Marek wrote:
> > 
> > Hi,
> > 
> > this series makes it possible to build bit-identical kernel image and
> > modules from identical sources. Of course the build is already
> > deterministic in terms of behavior of the code, but the various
> > timestamps embedded in the object files make it hard to compare two
> > builds, for instance to verify that a makefile cleanup didn't
> > accidentally change something. A prime example is /proc/config.gz, which
> > has both a timestamp in the gzip header and a timestamp in the payload
> > data. With this series applied, a script like this will produce
> > identical kernels each time:
> 
> Very nice stuff.  Do you want to take the individual patches through one
> of your trees, or do you mind if the subsystem maintainers take them
> through theirs?

I'm happy for this to go through a single tree.

James

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [PATCH 00/34] Make kernel build deterministic
  2011-04-05 18:16   ` James Bottomley
@ 2011-04-05 18:29     ` Mauro Carvalho Chehab
  2011-04-05 18:44       ` Greg KH
  0 siblings, 1 reply; 14+ messages in thread
From: Mauro Carvalho Chehab @ 2011-04-05 18:29 UTC (permalink / raw)
  To: James Bottomley
  Cc: anil_ravindranath, x86, mac, aacraid, linux-mtd, allan.stephens,
	hpa, netdev, t.sailer, gwingerde, IvDoorn, elf, cluster-devel,
	ccaulfie, mingo, dougthompson, linux-media, jon.maloy,
	arnaud.giersch, linux-kbuild, teigland, tony.olech, apw,
	linux-hams, tglx, swhiteho, linux-arm-kernel, linux-edac,
	Michal Marek, linux-scsi, linuxppc-dev, Greg KH, linux-wireless,
	linux-kernel, bluesmoke-devel, linux-usb, tipc-discussion,
	chuanxiao.dong, paulus, dwmw2, davem

Em 05-04-2011 15:16, James Bottomley escreveu:
> On Tue, 2011-04-05 at 08:49 -0700, Greg KH wrote:
>> On Tue, Apr 05, 2011 at 04:58:47PM +0200, Michal Marek wrote:
>>>
>>> Hi,
>>>
>>> this series makes it possible to build bit-identical kernel image and
>>> modules from identical sources. Of course the build is already
>>> deterministic in terms of behavior of the code, but the various
>>> timestamps embedded in the object files make it hard to compare two
>>> builds, for instance to verify that a makefile cleanup didn't
>>> accidentally change something. A prime example is /proc/config.gz, which
>>> has both a timestamp in the gzip header and a timestamp in the payload
>>> data. With this series applied, a script like this will produce
>>> identical kernels each time:
>>
>> Very nice stuff.  Do you want to take the individual patches through one
>> of your trees, or do you mind if the subsystem maintainers take them
>> through theirs?
> 
> I'm happy for this to go through a single tree.

Me too.

With respect to the patches I was c/c (patches 13, 14, 31):

Acked-by: Mauro Carvalho Chehab <mchehab@redhat.com>

Thanks,
Mauro.

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [PATCH 00/34] Make kernel build deterministic
  2011-04-05 18:29     ` Mauro Carvalho Chehab
@ 2011-04-05 18:44       ` Greg KH
  0 siblings, 0 replies; 14+ messages in thread
From: Greg KH @ 2011-04-05 18:44 UTC (permalink / raw)
  To: Mauro Carvalho Chehab
  Cc: anil_ravindranath, x86, mac, aacraid, linux-mtd, allan.stephens,
	hpa, netdev, t.sailer, gwingerde, IvDoorn, elf, cluster-devel,
	ccaulfie, mingo, dougthompson, linux-media, jon.maloy,
	arnaud.giersch, linux-kbuild, teigland, tony.olech, apw,
	linux-hams, tglx, swhiteho, linux-arm-kernel, linux-edac,
	Michal Marek, linux-scsi, linuxppc-dev, linux-usb, linux-wireless,
	linux-kernel, bluesmoke-devel, James Bottomley, tipc-discussion,
	chuanxiao.dong, paulus, dwmw2, davem

On Tue, Apr 05, 2011 at 03:29:19PM -0300, Mauro Carvalho Chehab wrote:
> Em 05-04-2011 15:16, James Bottomley escreveu:
> > On Tue, 2011-04-05 at 08:49 -0700, Greg KH wrote:
> >> On Tue, Apr 05, 2011 at 04:58:47PM +0200, Michal Marek wrote:
> >>>
> >>> Hi,
> >>>
> >>> this series makes it possible to build bit-identical kernel image and
> >>> modules from identical sources. Of course the build is already
> >>> deterministic in terms of behavior of the code, but the various
> >>> timestamps embedded in the object files make it hard to compare two
> >>> builds, for instance to verify that a makefile cleanup didn't
> >>> accidentally change something. A prime example is /proc/config.gz, which
> >>> has both a timestamp in the gzip header and a timestamp in the payload
> >>> data. With this series applied, a script like this will produce
> >>> identical kernels each time:
> >>
> >> Very nice stuff.  Do you want to take the individual patches through one
> >> of your trees, or do you mind if the subsystem maintainers take them
> >> through theirs?
> > 
> > I'm happy for this to go through a single tree.
> 
> Me too.
> 
> With respect to the patches I was c/c (patches 13, 14, 31):
> 
> Acked-by: Mauro Carvalho Chehab <mchehab@redhat.com>

Me too.

Acked-by: Greg Kroah-Hartman <gregkh@suse.de>

on the patches I was copied on.

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [PATCH 00/34] Make kernel build deterministic
  2011-04-05 15:49 ` [PATCH 00/34] Make kernel build deterministic Greg KH
  2011-04-05 18:16   ` James Bottomley
@ 2011-04-05 19:24   ` Artem Bityutskiy
  2011-04-06  8:57     ` Ingo Molnar
  2011-04-06  9:07     ` Michal Marek
  2011-04-06  9:23   ` Michal Marek
  2 siblings, 2 replies; 14+ messages in thread
From: Artem Bityutskiy @ 2011-04-05 19:24 UTC (permalink / raw)
  To: Greg KH
  Cc: anil_ravindranath, mchehab, mac, aacraid, linux-mtd,
	allan.stephens, hpa, t.sailer, gwingerde, x86, elf, cluster-devel,
	ccaulfie, mingo, dougthompson, dwmw2, linux-media, jon.maloy,
	arnaud.giersch, linux-kbuild, IvDoorn, teigland, tony.olech, apw,
	linux-hams, tglx, swhiteho, linux-arm-kernel, linux-edac,
	Michal Marek, linux-scsi, netdev, linux-usb, linux-wireless,
	linux-kernel, davem, James.Bottomley, tipc-discussion,
	chuanxiao.dong, paulus, linuxppc-dev, bluesmoke-devel

On Tue, 2011-04-05 at 08:49 -0700, Greg KH wrote:
> On Tue, Apr 05, 2011 at 04:58:47PM +0200, Michal Marek wrote:
> > 
> > Hi,
> > 
> > this series makes it possible to build bit-identical kernel image and
> > modules from identical sources. Of course the build is already
> > deterministic in terms of behavior of the code, but the various
> > timestamps embedded in the object files make it hard to compare two
> > builds, for instance to verify that a makefile cleanup didn't
> > accidentally change something. A prime example is /proc/config.gz, which
> > has both a timestamp in the gzip header and a timestamp in the payload
> > data. With this series applied, a script like this will produce
> > identical kernels each time:
> 
> Very nice stuff.  Do you want to take the individual patches through one
> of your trees, or do you mind if the subsystem maintainers take them
> through theirs?

But unfortunately, it is very easy to break this and for sure it'll be
broken very soon.

So additionally, I'd suggest:
1. Instrument checkpatch.pl and make it err or warn on timestamps.
2. Probably instrument linux-next to rise a warning when people break
   this.

-- 
Best Regards,
Artem Bityutskiy (Битюцкий Артём)

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [PATCH 00/34] Make kernel build deterministic
  2011-04-05 19:24   ` Artem Bityutskiy
@ 2011-04-06  8:57     ` Ingo Molnar
  2011-04-06  9:07     ` Michal Marek
  1 sibling, 0 replies; 14+ messages in thread
From: Ingo Molnar @ 2011-04-06  8:57 UTC (permalink / raw)
  To: Artem Bityutskiy
  Cc: anil_ravindranath, mchehab, mac, aacraid, linux-mtd,
	allan.stephens, hpa, t.sailer, gwingerde, x86, elf, cluster-devel,
	ccaulfie, mingo, dougthompson, tipc-discussion, dwmw2,
	linux-media, jon.maloy, arnaud.giersch, linux-kbuild, IvDoorn,
	teigland, tony.olech, apw, linux-hams, tglx, swhiteho,
	linux-arm-kernel, linux-edac, Michal Marek, linux-scsi, netdev,
	Greg KH, linux-wireless, linux-kernel, davem, linux-usb,
	James.Bottomley, chuanxiao.dong, paulus, linuxppc-dev,
	bluesmoke-devel


* Artem Bityutskiy <dedekind1@gmail.com> wrote:

> On Tue, 2011-04-05 at 08:49 -0700, Greg KH wrote:
> > On Tue, Apr 05, 2011 at 04:58:47PM +0200, Michal Marek wrote:
> > > 
> > > Hi,
> > > 
> > > this series makes it possible to build bit-identical kernel image and
> > > modules from identical sources. Of course the build is already
> > > deterministic in terms of behavior of the code, but the various
> > > timestamps embedded in the object files make it hard to compare two
> > > builds, for instance to verify that a makefile cleanup didn't
> > > accidentally change something. A prime example is /proc/config.gz, which
> > > has both a timestamp in the gzip header and a timestamp in the payload
> > > data. With this series applied, a script like this will produce
> > > identical kernels each time:
> > 
> > Very nice stuff.  Do you want to take the individual patches through one
> > of your trees, or do you mind if the subsystem maintainers take them
> > through theirs?
> 
> But unfortunately, it is very easy to break this and for sure it'll be
> broken very soon.
> 
> So additionally, I'd suggest:
> 1. Instrument checkpatch.pl and make it err or warn on timestamps.

See the grandparent mail:

  checkpatch: Warn about usage of __DATE__, __TIME__ and __TIMESTAMP__

Thanks,

	Ingo

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [PATCH 00/34] Make kernel build deterministic
  2011-04-05 14:58 [PATCH 00/34] Make kernel build deterministic Michal Marek
                   ` (2 preceding siblings ...)
  2011-04-05 15:49 ` [PATCH 00/34] Make kernel build deterministic Greg KH
@ 2011-04-06  9:01 ` Ingo Molnar
  2011-05-05 15:22 ` [PATCH v2] powerpc: Use the deterministic mode of ar Michal Marek
  4 siblings, 0 replies; 14+ messages in thread
From: Ingo Molnar @ 2011-04-06  9:01 UTC (permalink / raw)
  To: Michal Marek
  Cc: anil_ravindranath, mchehab, x86, mac, aacraid, linux-mtd,
	allan.stephens, hpa, netdev, t.sailer, gwingerde, IvDoorn, elf,
	cluster-devel, ccaulfie, mingo, dougthompson, linux-usb,
	linux-media, arnaud.giersch, linux-kbuild, teigland, tony.olech,
	apw, linux-hams, tglx, swhiteho, linux-arm-kernel, linux-edac,
	jon.maloy, linux-scsi, linuxppc-dev, gregkh, linux-wireless,
	linux-kernel, bluesmoke-devel, James.Bottomley, tipc-discussion,
	chuanxiao.dong, paulus, dwmw2, davem


* Michal Marek <mmarek@suse.cz> wrote:

> 
> Hi,
> 
> this series makes it possible to build bit-identical kernel image and
> modules from identical sources. Of course the build is already
> deterministic in terms of behavior of the code, but the various
> timestamps embedded in the object files make it hard to compare two
> builds, for instance to verify that a makefile cleanup didn't
> accidentally change something. A prime example is /proc/config.gz, which
> has both a timestamp in the gzip header and a timestamp in the payload
> data. With this series applied, a script like this will produce
> identical kernels each time:
> 
> #!/bin/bash
> rm -rf /dev/shm/{source,build}{,1,2}
> export KCONFIG_NOTIMESTAMP=1
> export KBUILD_BUILD_TIMESTAMP='Sun May  1 12:00:00 CEST 2011'
> export KBUILD_BUILD_USER=user
> export KBUILD_BUILD_HOST=host
> export ROOT_DEV=FLOPPY
> for i in 1 2; do
> 	mkdir /dev/shm/source
> 	# randomize the inode order just for fun
> 	git ls-tree -r -z --name-only HEAD | sort -R -z | xargs -0 \
> 		cp --parents --target=/dev/shm/source
> 	pushd /dev/shm/source
> 	mkdir /dev/shm/build
> 	>/dev/shm/build/all.config
> 	for opt in GCOV_KERNEL UBIFS_FS_DEBUG; do
> 		echo "# CONFIG_$opt is not set" >>"/dev/shm/build"/all.config
> 	done
> 	make O="/dev/shm/build" KCONFIG_ALLCONFIG=1 allmodconfig
> 	make O="/dev/shm/build" -j64
> 	popd
> 	mv /dev/shm/source /dev/shm/source$i
> 	mv /dev/shm/build /dev/shm/build$i
> done
> diff -rq /dev/shm/build{1,2}

Nice!

Acked-by: Ingo Molnar <mingo@elte.hu>

> Unfortunatelly, this cannot be used to validate indentation-only
> patches, even if CONFIG_DEBUG_INFO is turned off. This is because of the
> __FILE__ and __LINE__ macros used in many places. For the same reason,
> the source and build directory needs to be the same, otherwise the
> results will differ.

Nor can it be used to validate untrusted patches in general: a subtle change 
might be introduced in a piece of code that is dependent on a .config detail 
that is off for that particular build.

But in the common case it's nice to be able to have deterministic/reproducable 
builds.

Thanks,

	Ingo

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [PATCH 00/34] Make kernel build deterministic
  2011-04-05 19:24   ` Artem Bityutskiy
  2011-04-06  8:57     ` Ingo Molnar
@ 2011-04-06  9:07     ` Michal Marek
  2011-04-06  9:25       ` Artem Bityutskiy
  1 sibling, 1 reply; 14+ messages in thread
From: Michal Marek @ 2011-04-06  9:07 UTC (permalink / raw)
  To: dedekind1
  Cc: anil_ravindranath, mchehab, mac, aacraid, linux-mtd,
	allan.stephens, hpa, t.sailer, gwingerde, x86, elf, cluster-devel,
	ccaulfie, mingo, dougthompson, tipc-discussion, dwmw2,
	linux-media, arnaud.giersch, linux-kbuild, IvDoorn, teigland,
	tony.olech, apw, linux-hams, tglx, swhiteho, linux-arm-kernel,
	linux-edac, jon.maloy, linux-scsi, netdev, Greg KH,
	linux-wireless, linux-kernel, davem, linux-usb, James.Bottomley,
	chuanxiao.dong, paulus, linuxppc-dev, bluesmoke-devel

On 5.4.2011 21:24, Artem Bityutskiy wrote:
> On Tue, 2011-04-05 at 08:49 -0700, Greg KH wrote:
>> On Tue, Apr 05, 2011 at 04:58:47PM +0200, Michal Marek wrote:
>>>
>>> Hi,
>>>
>>> this series makes it possible to build bit-identical kernel image and
>>> modules from identical sources. Of course the build is already
>>> deterministic in terms of behavior of the code, but the various
>>> timestamps embedded in the object files make it hard to compare two
>>> builds, for instance to verify that a makefile cleanup didn't
>>> accidentally change something. A prime example is /proc/config.gz, which
>>> has both a timestamp in the gzip header and a timestamp in the payload
>>> data. With this series applied, a script like this will produce
>>> identical kernels each time:
>>
>> Very nice stuff.  Do you want to take the individual patches through one
>> of your trees, or do you mind if the subsystem maintainers take them
>> through theirs?
> 
> But unfortunately, it is very easy to break this and for sure it'll be
> broken very soon.

I'm not so pessimistic. 34 patches and 57 files might sound like a lot,
but given that this has been accumulating since day one, the cleanup
should last for some time.


> So additionally, I'd suggest:
> 1. Instrument checkpatch.pl and make it err or warn on timestamps.

This is patch 34/34 in this series: https://lkml.org/lkml/2011/4/5/198


> 2. Probably instrument linux-next to rise a warning when people break
>    this.

I'm not sure if Stephen has that much spare time, and I don't think it
is necessary. I think the checkpatch check is sufficient and I'll check
myself occasionally.

Michal

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [PATCH 00/34] Make kernel build deterministic
  2011-04-05 15:49 ` [PATCH 00/34] Make kernel build deterministic Greg KH
  2011-04-05 18:16   ` James Bottomley
  2011-04-05 19:24   ` Artem Bityutskiy
@ 2011-04-06  9:23   ` Michal Marek
  2 siblings, 0 replies; 14+ messages in thread
From: Michal Marek @ 2011-04-06  9:23 UTC (permalink / raw)
  To: Greg KH
  Cc: anil_ravindranath, mchehab, x86, mac, aacraid, linux-mtd,
	allan.stephens, hpa, netdev, t.sailer, gwingerde, IvDoorn, elf,
	cluster-devel, ccaulfie, mingo, dougthompson, linux-media,
	arnaud.giersch, linux-kbuild, teigland, tony.olech, apw,
	linux-hams, tglx, swhiteho, linux-arm-kernel, linux-edac,
	jon.maloy, linux-scsi, linuxppc-dev, linux-usb, linux-wireless,
	linux-kernel, bluesmoke-devel, James.Bottomley, tipc-discussion,
	chuanxiao.dong, paulus, dwmw2, davem

On 5.4.2011 17:49, Greg KH wrote:
> Very nice stuff.  Do you want to take the individual patches through one
> of your trees, or do you mind if the subsystem maintainers take them
> through theirs?

I'd leave it up to the subsystem maintainers. I'll check once the merge
window opens and send the remaining patches to Linus directly.

Michal

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [PATCH 00/34] Make kernel build deterministic
  2011-04-06  9:07     ` Michal Marek
@ 2011-04-06  9:25       ` Artem Bityutskiy
  0 siblings, 0 replies; 14+ messages in thread
From: Artem Bityutskiy @ 2011-04-06  9:25 UTC (permalink / raw)
  To: Michal Marek
  Cc: anil_ravindranath, mchehab, mac, aacraid, linux-mtd,
	allan.stephens, hpa, t.sailer, gwingerde, x86, elf, cluster-devel,
	ccaulfie, mingo, dougthompson, tipc-discussion, dwmw2,
	linux-media, arnaud.giersch, linux-kbuild, IvDoorn, teigland,
	tony.olech, apw, linux-hams, tglx, swhiteho, linux-arm-kernel,
	linux-edac, jon.maloy, linux-scsi, netdev, Greg KH,
	linux-wireless, linux-kernel, davem, linux-usb, James.Bottomley,
	chuanxiao.dong, paulus, linuxppc-dev, bluesmoke-devel

On Wed, 2011-04-06 at 11:07 +0200, Michal Marek wrote:
> > So additionally, I'd suggest:
> > 1. Instrument checkpatch.pl and make it err or warn on timestamps.
> 
> This is patch 34/34 in this series: https://lkml.org/lkml/2011/4/5/198

Yeah, great, did not notice, thanks!

> > 2. Probably instrument linux-next to rise a warning when people break
> >    this.
> 
> I'm not sure if Stephen has that much spare time, and I don't think it
> is necessary. I think the checkpatch check is sufficient and I'll check
> myself occasionally.

Yes, that's fine, I just wanted to speak this out - there is a
probability that someone gets excited and creates some instrumentation
to kbuild to automatically detect bad things and then Stephen could
easily use that.

WRT "not necessary" - well, I think it is always better to cut bad
things before they are merged, rather than afterwards.

But anyway, let's forget about this.

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

^ permalink raw reply	[flat|nested] 14+ messages in thread

* [PATCH v2] powerpc: Use the deterministic mode of ar
  2011-04-05 14:58 [PATCH 00/34] Make kernel build deterministic Michal Marek
                   ` (3 preceding siblings ...)
  2011-04-06  9:01 ` Ingo Molnar
@ 2011-05-05 15:22 ` Michal Marek
  4 siblings, 0 replies; 14+ messages in thread
From: Michal Marek @ 2011-05-05 15:22 UTC (permalink / raw)
  To: linux-kbuild; +Cc: linuxppc-dev, Paul Mackerras, linux-kernel

Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: linuxppc-dev@lists.ozlabs.org
Signed-off-by: Michal Marek <mmarek@suse.cz>

---

Hi,

this version fixes build with binutils <= 2.19 by first checking if ar(1)
supports the D option. It depends on a previous patch in the kbuild tree,
commit 40df759e (kbuild: Fix build with binutils <= 2.19) in linux-next,
so it will need to go through the kbuild tree. But feedback from the PPC
maintainers would be appreciated.

Thanks
Michal

---
 arch/powerpc/boot/Makefile |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/arch/powerpc/boot/Makefile b/arch/powerpc/boot/Makefile
index 8917816..f0543f8 100644
--- a/arch/powerpc/boot/Makefile
+++ b/arch/powerpc/boot/Makefile
@@ -127,7 +127,7 @@ quiet_cmd_bootas = BOOTAS  $@
       cmd_bootas = $(CROSS32CC) -Wp,-MD,$(depfile) $(BOOTAFLAGS) -c -o $@ $<
 
 quiet_cmd_bootar = BOOTAR  $@
-      cmd_bootar = $(CROSS32AR) -cr $@.$$$$ $(filter-out FORCE,$^); mv $@.$$$$ $@
+      cmd_bootar = $(CROSS32AR) -cr$(KBUILD_ARFLAGS) $@.$$$$ $(filter-out FORCE,$^); mv $@.$$$$ $@
 
 $(obj-libfdt): $(obj)/%.o: $(srctree)/scripts/dtc/libfdt/%.c FORCE
 	$(call if_changed_dep,bootcc)
-- 
1.7.4.1

^ permalink raw reply related	[flat|nested] 14+ messages in thread

end of thread, other threads:[~2011-05-05 15:23 UTC | newest]

Thread overview: 14+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2011-04-05 14:58 [PATCH 00/34] Make kernel build deterministic Michal Marek
2011-04-05 14:58 ` [PATCH 03/34] powerpc: Call gzip with -n Michal Marek
2011-04-05 14:58 ` [PATCH 05/34] powerpc: Use the deterministic mode of ar Michal Marek
2011-04-05 15:49 ` [PATCH 00/34] Make kernel build deterministic Greg KH
2011-04-05 18:16   ` James Bottomley
2011-04-05 18:29     ` Mauro Carvalho Chehab
2011-04-05 18:44       ` Greg KH
2011-04-05 19:24   ` Artem Bityutskiy
2011-04-06  8:57     ` Ingo Molnar
2011-04-06  9:07     ` Michal Marek
2011-04-06  9:25       ` Artem Bityutskiy
2011-04-06  9:23   ` Michal Marek
2011-04-06  9:01 ` Ingo Molnar
2011-05-05 15:22 ` [PATCH v2] powerpc: Use the deterministic mode of ar Michal Marek

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).