[PATCH 00/34] Make kernel build deterministic

[PATCH 00/34] Make kernel build deterministic

[Michal Marek]

Hi,

this series makes it possible to build bit-identical kernel image and
modules from identical sources. Of course the build is already
deterministic in terms of behavior of the code, but the various
timestamps embedded in the object files make it hard to compare two
builds, for instance to verify that a makefile cleanup didn't
accidentally change something. A prime example is /proc/config.gz, which
has both a timestamp in the gzip header and a timestamp in the payload
data. With this series applied, a script like this will produce
identical kernels each time:

#!/bin/bash
rm -rf /dev/shm/{source,build}{,1,2}
export KCONFIG_NOTIMESTAMP=1
export KBUILD_BUILD_TIMESTAMP='Sun May  1 12:00:00 CEST 2011'
export KBUILD_BUILD_USER=user
export KBUILD_BUILD_HOST=host
export ROOT_DEV=FLOPPY
for i in 1 2; do
	mkdir /dev/shm/source
	# randomize the inode order just for fun
	git ls-tree -r -z --name-only HEAD | sort -R -z | xargs -0 \
		cp --parents --target=/dev/shm/source
	pushd /dev/shm/source
	mkdir /dev/shm/build
	>/dev/shm/build/all.config
	for opt in GCOV_KERNEL UBIFS_FS_DEBUG; do
		echo "# CONFIG_$opt is not set" >>"/dev/shm/build"/all.config
	done
	make O="/dev/shm/build" KCONFIG_ALLCONFIG=1 allmodconfig
	make O="/dev/shm/build" -j64
	popd
	mv /dev/shm/source /dev/shm/source$i
	mv /dev/shm/build /dev/shm/build$i
done
diff -rq /dev/shm/build{1,2}

Unfortunatelly, this cannot be used to validate indentation-only
patches, even if CONFIG_DEBUG_INFO is turned off. This is because of the
__FILE__ and __LINE__ macros used in many places. For the same reason,
the source and build directory needs to be the same, otherwise the
results will differ. This was tested on
x86_64/{defconfig,allmodconfig,allyesconfig} and ppc64/defconfig. The
series is also available at

git://git.kernel.org/pub/scm/linux/kernel/git/mmarek/kbuild-2.6.git deterministic-build-v1

Michal


Michal Marek (34):
  kconfig: Do not record timestamp in auto.conf and autoconf.h
  kbuild: Call gzip with -n
  powerpc: Call gzip with -n
  kbuild: Use the deterministic mode of ar
  powerpc: Use the deterministic mode of ar
  kbuild: Drop unused LINUX_COMPILE_TIME and LINUX_COMPILE_DOMAIN
    macros
  kbuild: Allow to override LINUX_COMPILE_BY and LINUX_COMPILE_HOST
    macros
  initramfs: Use KBUILD_BUILD_TIMESTAMP for generated entries
  x86: Allow to override the ROOT_DEV variable
  cyclades: Drop __TIME__ usage
  nozomi: Drop __TIME__ usage
  isdn/diva: Drop __TIME__ usage
  media/radio-maxiradio: Drop __TIME__ usage
  media/cx231xx: Drop __TIME__ usage
  baycom: Drop __TIME__ usage
  nand/denali: Drop __TIME__ usage
  hdlcdrv: Drop __TIME__ usage
  wan/pc300: Drop __TIME__ usage
  rt2x00: Drop __TIME__ usage
  parport: Drop __TIME__ usage
  aacraid: Drop __TIME__ usage
  scsi/in2000: Drop __TIME__ usage
  scsi/wd33c93: Drop __TIME__ usage
  usb/u132-hcd: Drop __TIME__ usage
  usb/ftdi-elan: Drop __TIME__ usage
  dlm: Drop __TIME__ usage
  gfs2: Drop __TIME__ usage
  atm: Drop __TIME__ usage
  tipc: Drop __TIME__ usage
  rio: Drop __DATE__ usage
  edac: Drop __DATE__ usage
  pmcraid: Drop __DATE__ usage
  usb/lh7a40x_udc: Drop __DATE__ usage
  checkpatch: Warn about usage of __DATE__, __TIME__ and __TIMESTAMP__

 Documentation/kbuild/kbuild.txt              |   12 ++++++++++
 arch/powerpc/boot/Makefile                   |    2 +-
 arch/powerpc/boot/wrapper                    |    6 ++--
 arch/x86/boot/Makefile                       |    2 +-
 drivers/char/cyclades.c                      |    3 +-
 drivers/char/nozomi.c                        |    3 +-
 drivers/char/rio/rioinit.c                   |    2 +-
 drivers/edac/amd76x_edac.c                   |    2 +-
 drivers/edac/amd8111_edac.c                  |    2 +-
 drivers/edac/amd8131_edac.c                  |    2 +-
 drivers/edac/cpc925_edac.c                   |    2 +-
 drivers/edac/e752x_edac.c                    |    2 +-
 drivers/edac/e7xxx_edac.c                    |    2 +-
 drivers/edac/edac_module.c                   |    2 +-
 drivers/edac/i5000_edac.c                    |    2 +-
 drivers/edac/i5400_edac.c                    |    2 +-
 drivers/edac/i7300_edac.c                    |    2 +-
 drivers/edac/i7core_edac.c                   |    2 +-
 drivers/edac/i82860_edac.c                   |    2 +-
 drivers/edac/i82875p_edac.c                  |    2 +-
 drivers/edac/i82975x_edac.c                  |    2 +-
 drivers/edac/mpc85xx_edac.h                  |    2 +-
 drivers/edac/mv64x60_edac.h                  |    2 +-
 drivers/edac/ppc4xx_edac.c                   |    2 +-
 drivers/edac/r82600_edac.c                   |    2 +-
 drivers/isdn/hardware/eicon/divasfunc.c      |    5 +--
 drivers/media/radio/radio-maxiradio.c        |    3 +-
 drivers/media/video/cx231xx/cx231xx-avcore.c |    2 +-
 drivers/mtd/nand/denali.c                    |    3 +-
 drivers/net/hamradio/baycom_epp.c            |    2 +-
 drivers/net/hamradio/baycom_par.c            |    2 +-
 drivers/net/hamradio/baycom_ser_fdx.c        |    2 +-
 drivers/net/hamradio/baycom_ser_hdx.c        |    2 +-
 drivers/net/hamradio/hdlcdrv.c               |    2 +-
 drivers/net/wan/pc300_drv.c                  |    3 +-
 drivers/net/wireless/rt2x00/rt2x00debug.c    |    1 -
 drivers/parport/parport_ip32.c               |    1 -
 drivers/scsi/aacraid/linit.c                 |    3 +-
 drivers/scsi/in2000.c                        |    2 +-
 drivers/scsi/pmcraid.c                       |    9 +++----
 drivers/scsi/pmcraid.h                       |    1 -
 drivers/scsi/wd33c93.c                       |    7 ++---
 drivers/usb/gadget/lh7a40x_udc.c             |    7 ++---
 drivers/usb/host/u132-hcd.c                  |    3 +-
 drivers/usb/misc/ftdi-elan.c                 |    3 +-
 fs/dlm/main.c                                |    2 +-
 fs/gfs2/main.c                               |    2 +-
 net/atm/lec.c                                |    2 +-
 net/atm/mpc.c                                |    2 +-
 net/tipc/core.c                              |    3 +-
 scripts/Makefile.build                       |    4 +-
 scripts/Makefile.lib                         |    2 +-
 scripts/checkpatch.pl                        |    5 ++++
 scripts/gen_initramfs_list.sh                |    8 +++++-
 scripts/kconfig/confdata.c                   |    8 +-----
 scripts/mkcompile_h                          |   30 ++++++++++++-------------
 usr/gen_init_cpio.c                          |   17 +++++++++-----
 57 files changed, 109 insertions(+), 105 deletions(-)

-- 
1.7.4.1

[PATCH 21/34] aacraid: Drop __TIME__ usage

[Michal Marek]

The kernel already prints its build timestamp during boot, no need to
repeat it in random drivers and produce different object files each
time.

Cc: Adaptec OEM Raid Solutions <aacraid@adaptec.com>
Cc: linux-scsi@vger.kernel.org
Signed-off-by: Michal Marek <mmarek@suse.cz>
---
 drivers/scsi/aacraid/linit.c |    3 +--
 1 files changed, 1 insertions(+), 2 deletions(-)

diff --git a/drivers/scsi/aacraid/linit.c b/drivers/scsi/aacraid/linit.c
index 2c93d94..7babcbb 100644
--- a/drivers/scsi/aacraid/linit.c
+++ b/drivers/scsi/aacraid/linit.c
@@ -58,7 +58,6 @@
 #ifndef AAC_DRIVER_BRANCH
 #define AAC_DRIVER_BRANCH		""
 #endif
-#define AAC_DRIVER_BUILD_DATE		__DATE__ " " __TIME__
 #define AAC_DRIVERNAME			"aacraid"
 
 #ifdef AAC_DRIVER_BUILD
@@ -66,7 +65,7 @@
 #define str(x) _str(x)
 #define AAC_DRIVER_FULL_VERSION	AAC_DRIVER_VERSION "[" str(AAC_DRIVER_BUILD) "]" AAC_DRIVER_BRANCH
 #else
-#define AAC_DRIVER_FULL_VERSION	AAC_DRIVER_VERSION AAC_DRIVER_BRANCH " " AAC_DRIVER_BUILD_DATE
+#define AAC_DRIVER_FULL_VERSION	AAC_DRIVER_VERSION AAC_DRIVER_BRANCH
 #endif
 
 MODULE_AUTHOR("Red Hat Inc and Adaptec");
-- 
1.7.4.1

Re: [PATCH 21/34] aacraid: Drop __TIME__ usage

[Michal Marek]

On Tue, Apr 05, 2011 at 04:59:08PM +0200, Michal Marek wrote:
> The kernel already prints its build timestamp during boot, no need to
> repeat it in random drivers and produce different object files each
> time.
> 
> Cc: Adaptec OEM Raid Solutions <aacraid@adaptec.com>
> Cc: linux-scsi@vger.kernel.org
> Signed-off-by: Michal Marek <mmarek@suse.cz>
> ---
>  drivers/scsi/aacraid/linit.c |    3 +--
>  1 files changed, 1 insertions(+), 2 deletions(-)

Applied to kbuild-2.6.git#trivial.

Michal

[PATCH 22/34] scsi/in2000: Drop __TIME__ usage

[Michal Marek]

The kernel already prints its build timestamp during boot, no need to
repeat it in random drivers and produce different object files each
time.

Cc: "James E.J. Bottomley" <James.Bottomley@suse.de>
Cc: linux-scsi@vger.kernel.org
Signed-off-by: Michal Marek <mmarek@suse.cz>
---
 drivers/scsi/in2000.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/drivers/scsi/in2000.c b/drivers/scsi/in2000.c
index 6568aab..6096d93 100644
--- a/drivers/scsi/in2000.c
+++ b/drivers/scsi/in2000.c
@@ -2228,7 +2228,7 @@ static int in2000_proc_info(struct Scsi_Host *instance, char *buf, char **start,
 	bp = buf;
 	*bp = '\0';
 	if (hd->proc & PR_VERSION) {
-		sprintf(tbuf, "\nVersion %s - %s. Compiled %s %s", IN2000_VERSION, IN2000_DATE, __DATE__, __TIME__);
+		sprintf(tbuf, "\nVersion %s - %s.", IN2000_VERSION, IN2000_DATE);
 		strcat(bp, tbuf);
 	}
 	if (hd->proc & PR_INFO) {
-- 
1.7.4.1

Re: [PATCH 22/34] scsi/in2000: Drop __TIME__ usage

[Michal Marek]

On Tue, Apr 05, 2011 at 04:59:09PM +0200, Michal Marek wrote:
> The kernel already prints its build timestamp during boot, no need to
> repeat it in random drivers and produce different object files each
> time.
> 
> Cc: "James E.J. Bottomley" <James.Bottomley@suse.de>
> Cc: linux-scsi@vger.kernel.org
> Signed-off-by: Michal Marek <mmarek@suse.cz>
> ---
>  drivers/scsi/in2000.c |    2 +-
>  1 files changed, 1 insertions(+), 1 deletions(-)

Applied to kbuild-2.6.git#trivial.

Michal

[PATCH 23/34] scsi/wd33c93: Drop __TIME__ usage

[Michal Marek]

The kernel already prints its build timestamp during boot, no need to
repeat it in random drivers and produce different object files each
time.

Cc: "James E.J. Bottomley" <James.Bottomley@suse.de>
Cc: linux-scsi@vger.kernel.org
Signed-off-by: Michal Marek <mmarek@suse.cz>
---
 drivers/scsi/wd33c93.c |    7 +++----
 1 files changed, 3 insertions(+), 4 deletions(-)

diff --git a/drivers/scsi/wd33c93.c b/drivers/scsi/wd33c93.c
index 5f697e0..6c983a2 100644
--- a/drivers/scsi/wd33c93.c
+++ b/drivers/scsi/wd33c93.c
@@ -2052,8 +2052,7 @@ wd33c93_init(struct Scsi_Host *instance, const wd33c93_regs regs,
 	for (i = 0; i < MAX_SETUP_ARGS; i++)
 		printk("%s,", setup_args[i]);
 	printk("\n");
-	printk("           Version %s - %s, Compiled %s at %s\n",
-	       WD33C93_VERSION, WD33C93_DATE, __DATE__, __TIME__);
+	printk("           Version %s - %s\n", WD33C93_VERSION, WD33C93_DATE);
 }
 
 int
@@ -2133,8 +2132,8 @@ wd33c93_proc_info(struct Scsi_Host *instance, char *buf, char **start, off_t off
 	bp = buf;
 	*bp = '\0';
 	if (hd->proc & PR_VERSION) {
-		sprintf(tbuf, "\nVersion %s - %s. Compiled %s %s",
-			WD33C93_VERSION, WD33C93_DATE, __DATE__, __TIME__);
+		sprintf(tbuf, "\nVersion %s - %s.",
+			WD33C93_VERSION, WD33C93_DATE);
 		strcat(bp, tbuf);
 	}
 	if (hd->proc & PR_INFO) {
-- 
1.7.4.1

Re: [PATCH 23/34] scsi/wd33c93: Drop __TIME__ usage

[Michal Marek]

On Tue, Apr 05, 2011 at 04:59:10PM +0200, Michal Marek wrote:
> The kernel already prints its build timestamp during boot, no need to
> repeat it in random drivers and produce different object files each
> time.
> 
> Cc: "James E.J. Bottomley" <James.Bottomley@suse.de>
> Cc: linux-scsi@vger.kernel.org
> Signed-off-by: Michal Marek <mmarek@suse.cz>
> ---
>  drivers/scsi/wd33c93.c |    7 +++----
>  1 files changed, 3 insertions(+), 4 deletions(-)

Applied to kbuild-2.6.git#trivial.

Michal

[PATCH 32/34] pmcraid: Drop __DATE__ usage

[Michal Marek]

The kernel already prints its build timestamp during boot, no need to
repeat it in random drivers and produce different object files each
time.

Cc: Anil Ravindranath <anil_ravindranath@pmc-sierra.com>
Cc: linux-scsi@vger.kernel.org
Signed-off-by: Michal Marek <mmarek@suse.cz>
---
 drivers/scsi/pmcraid.c |    9 ++++-----
 drivers/scsi/pmcraid.h |    1 -
 2 files changed, 4 insertions(+), 6 deletions(-)

diff --git a/drivers/scsi/pmcraid.c b/drivers/scsi/pmcraid.c
index 321cf3a..7c13201 100644
--- a/drivers/scsi/pmcraid.c
+++ b/drivers/scsi/pmcraid.c
@@ -4249,8 +4249,8 @@ static ssize_t pmcraid_show_drv_version(
 	char *buf
 )
 {
-	return snprintf(buf, PAGE_SIZE, "version: %s, build date: %s\n",
-			PMCRAID_DRIVER_VERSION, PMCRAID_DRIVER_DATE);
+	return snprintf(buf, PAGE_SIZE, "version: %s\n",
+			PMCRAID_DRIVER_VERSION);
 }
 
 static struct device_attribute pmcraid_driver_version_attr = {
@@ -6093,9 +6093,8 @@ static int __init pmcraid_init(void)
 	dev_t dev;
 	int error;
 
-	pmcraid_info("%s Device Driver version: %s %s\n",
-			 PMCRAID_DRIVER_NAME,
-			 PMCRAID_DRIVER_VERSION, PMCRAID_DRIVER_DATE);
+	pmcraid_info("%s Device Driver version: %s\n",
+			 PMCRAID_DRIVER_NAME, PMCRAID_DRIVER_VERSION);
 
 	error = alloc_chrdev_region(&dev, 0,
 				    PMCRAID_MAX_ADAPTERS,
diff --git a/drivers/scsi/pmcraid.h b/drivers/scsi/pmcraid.h
index 4db210d..d302737 100644
--- a/drivers/scsi/pmcraid.h
+++ b/drivers/scsi/pmcraid.h
@@ -43,7 +43,6 @@
 #define PMCRAID_DRIVER_NAME		"PMC MaxRAID"
 #define PMCRAID_DEVFILE			"pmcsas"
 #define PMCRAID_DRIVER_VERSION		"1.0.3"
-#define PMCRAID_DRIVER_DATE		__DATE__
 
 #define PMCRAID_FW_VERSION_1		0x002
 
-- 
1.7.4.1

Re: [PATCH 32/34] pmcraid: Drop __DATE__ usage

[Michal Marek]

On Tue, Apr 05, 2011 at 04:59:19PM +0200, Michal Marek wrote:
> The kernel already prints its build timestamp during boot, no need to
> repeat it in random drivers and produce different object files each
> time.
> 
> Cc: Anil Ravindranath <anil_ravindranath@pmc-sierra.com>
> Cc: linux-scsi@vger.kernel.org
> Signed-off-by: Michal Marek <mmarek@suse.cz>
> ---
>  drivers/scsi/pmcraid.c |    9 ++++-----
>  drivers/scsi/pmcraid.h |    1 -
>  2 files changed, 4 insertions(+), 6 deletions(-)

Applied to kbuild-2.6.git#trivial.

Michal

Re: [PATCH 00/34] Make kernel build deterministic

[Greg KH]

On Tue, Apr 05, 2011 at 04:58:47PM +0200, Michal Marek wrote:
> 
> Hi,
> 
> this series makes it possible to build bit-identical kernel image and
> modules from identical sources. Of course the build is already
> deterministic in terms of behavior of the code, but the various
> timestamps embedded in the object files make it hard to compare two
> builds, for instance to verify that a makefile cleanup didn't
> accidentally change something. A prime example is /proc/config.gz, which
> has both a timestamp in the gzip header and a timestamp in the payload
> data. With this series applied, a script like this will produce
> identical kernels each time:

Very nice stuff.  Do you want to take the individual patches through one
of your trees, or do you mind if the subsystem maintainers take them
through theirs?

thanks,

greg k-h

Re: [PATCH 00/34] Make kernel build deterministic

[James Bottomley]

On Tue, 2011-04-05 at 08:49 -0700, Greg KH wrote:
> On Tue, Apr 05, 2011 at 04:58:47PM +0200, Michal Marek wrote:
> > 
> > Hi,
> > 
> > this series makes it possible to build bit-identical kernel image and
> > modules from identical sources. Of course the build is already
> > deterministic in terms of behavior of the code, but the various
> > timestamps embedded in the object files make it hard to compare two
> > builds, for instance to verify that a makefile cleanup didn't
> > accidentally change something. A prime example is /proc/config.gz, which
> > has both a timestamp in the gzip header and a timestamp in the payload
> > data. With this series applied, a script like this will produce
> > identical kernels each time:
> 
> Very nice stuff.  Do you want to take the individual patches through one
> of your trees, or do you mind if the subsystem maintainers take them
> through theirs?

I'm happy for this to go through a single tree.

James

Re: [PATCH 00/34] Make kernel build deterministic

[Mauro Carvalho Chehab]

Em 05-04-2011 15:16, James Bottomley escreveu:
> On Tue, 2011-04-05 at 08:49 -0700, Greg KH wrote:
>> On Tue, Apr 05, 2011 at 04:58:47PM +0200, Michal Marek wrote:
>>>
>>> Hi,
>>>
>>> this series makes it possible to build bit-identical kernel image and
>>> modules from identical sources. Of course the build is already
>>> deterministic in terms of behavior of the code, but the various
>>> timestamps embedded in the object files make it hard to compare two
>>> builds, for instance to verify that a makefile cleanup didn't
>>> accidentally change something. A prime example is /proc/config.gz, which
>>> has both a timestamp in the gzip header and a timestamp in the payload
>>> data. With this series applied, a script like this will produce
>>> identical kernels each time:
>>
>> Very nice stuff.  Do you want to take the individual patches through one
>> of your trees, or do you mind if the subsystem maintainers take them
>> through theirs?
> 
> I'm happy for this to go through a single tree.

Me too.

With respect to the patches I was c/c (patches 13, 14, 31):

Acked-by: Mauro Carvalho Chehab <mchehab@redhat.com>

Thanks,
Mauro.

Re: [PATCH 00/34] Make kernel build deterministic

[Greg KH]

On Tue, Apr 05, 2011 at 03:29:19PM -0300, Mauro Carvalho Chehab wrote:
> Em 05-04-2011 15:16, James Bottomley escreveu:
> > On Tue, 2011-04-05 at 08:49 -0700, Greg KH wrote:
> >> On Tue, Apr 05, 2011 at 04:58:47PM +0200, Michal Marek wrote:
> >>>
> >>> Hi,
> >>>
> >>> this series makes it possible to build bit-identical kernel image and
> >>> modules from identical sources. Of course the build is already
> >>> deterministic in terms of behavior of the code, but the various
> >>> timestamps embedded in the object files make it hard to compare two
> >>> builds, for instance to verify that a makefile cleanup didn't
> >>> accidentally change something. A prime example is /proc/config.gz, which
> >>> has both a timestamp in the gzip header and a timestamp in the payload
> >>> data. With this series applied, a script like this will produce
> >>> identical kernels each time:
> >>
> >> Very nice stuff.  Do you want to take the individual patches through one
> >> of your trees, or do you mind if the subsystem maintainers take them
> >> through theirs?
> > 
> > I'm happy for this to go through a single tree.
> 
> Me too.
> 
> With respect to the patches I was c/c (patches 13, 14, 31):
> 
> Acked-by: Mauro Carvalho Chehab <mchehab@redhat.com>

Me too.

Acked-by: Greg Kroah-Hartman <gregkh@suse.de>

on the patches I was copied on.

thanks,

greg k-h

Re: [PATCH 00/34] Make kernel build deterministic

[Artem Bityutskiy]

On Tue, 2011-04-05 at 08:49 -0700, Greg KH wrote:
> On Tue, Apr 05, 2011 at 04:58:47PM +0200, Michal Marek wrote:
> > 
> > Hi,
> > 
> > this series makes it possible to build bit-identical kernel image and
> > modules from identical sources. Of course the build is already
> > deterministic in terms of behavior of the code, but the various
> > timestamps embedded in the object files make it hard to compare two
> > builds, for instance to verify that a makefile cleanup didn't
> > accidentally change something. A prime example is /proc/config.gz, which
> > has both a timestamp in the gzip header and a timestamp in the payload
> > data. With this series applied, a script like this will produce
> > identical kernels each time:
> 
> Very nice stuff.  Do you want to take the individual patches through one
> of your trees, or do you mind if the subsystem maintainers take them
> through theirs?

But unfortunately, it is very easy to break this and for sure it'll be
broken very soon.

So additionally, I'd suggest:
1. Instrument checkpatch.pl and make it err or warn on timestamps.
2. Probably instrument linux-next to rise a warning when people break
   this.

-- 
Best Regards,
Artem Bityutskiy (Битюцкий Артём)


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 00/34] Make kernel build deterministic

[Ingo Molnar]

* Artem Bityutskiy <dedekind1@gmail.com> wrote:

> On Tue, 2011-04-05 at 08:49 -0700, Greg KH wrote:
> > On Tue, Apr 05, 2011 at 04:58:47PM +0200, Michal Marek wrote:
> > > 
> > > Hi,
> > > 
> > > this series makes it possible to build bit-identical kernel image and
> > > modules from identical sources. Of course the build is already
> > > deterministic in terms of behavior of the code, but the various
> > > timestamps embedded in the object files make it hard to compare two
> > > builds, for instance to verify that a makefile cleanup didn't
> > > accidentally change something. A prime example is /proc/config.gz, which
> > > has both a timestamp in the gzip header and a timestamp in the payload
> > > data. With this series applied, a script like this will produce
> > > identical kernels each time:
> > 
> > Very nice stuff.  Do you want to take the individual patches through one
> > of your trees, or do you mind if the subsystem maintainers take them
> > through theirs?
> 
> But unfortunately, it is very easy to break this and for sure it'll be
> broken very soon.
> 
> So additionally, I'd suggest:
> 1. Instrument checkpatch.pl and make it err or warn on timestamps.

See the grandparent mail:

  checkpatch: Warn about usage of __DATE__, __TIME__ and __TIMESTAMP__

Thanks,

	Ingo

Re: [PATCH 00/34] Make kernel build deterministic

[Michal Marek]

On 5.4.2011 21:24, Artem Bityutskiy wrote:
> On Tue, 2011-04-05 at 08:49 -0700, Greg KH wrote:
>> On Tue, Apr 05, 2011 at 04:58:47PM +0200, Michal Marek wrote:
>>>
>>> Hi,
>>>
>>> this series makes it possible to build bit-identical kernel image and
>>> modules from identical sources. Of course the build is already
>>> deterministic in terms of behavior of the code, but the various
>>> timestamps embedded in the object files make it hard to compare two
>>> builds, for instance to verify that a makefile cleanup didn't
>>> accidentally change something. A prime example is /proc/config.gz, which
>>> has both a timestamp in the gzip header and a timestamp in the payload
>>> data. With this series applied, a script like this will produce
>>> identical kernels each time:
>>
>> Very nice stuff.  Do you want to take the individual patches through one
>> of your trees, or do you mind if the subsystem maintainers take them
>> through theirs?
> 
> But unfortunately, it is very easy to break this and for sure it'll be
> broken very soon.

I'm not so pessimistic. 34 patches and 57 files might sound like a lot,
but given that this has been accumulating since day one, the cleanup
should last for some time.


> So additionally, I'd suggest:
> 1. Instrument checkpatch.pl and make it err or warn on timestamps.

This is patch 34/34 in this series: https://lkml.org/lkml/2011/4/5/198


> 2. Probably instrument linux-next to rise a warning when people break
>    this.

I'm not sure if Stephen has that much spare time, and I don't think it
is necessary. I think the checkpatch check is sufficient and I'll check
myself occasionally.

Michal

Re: [PATCH 00/34] Make kernel build deterministic

[Artem Bityutskiy]

On Wed, 2011-04-06 at 11:07 +0200, Michal Marek wrote:
> > So additionally, I'd suggest:
> > 1. Instrument checkpatch.pl and make it err or warn on timestamps.
> 
> This is patch 34/34 in this series: https://lkml.org/lkml/2011/4/5/198

Yeah, great, did not notice, thanks!

> > 2. Probably instrument linux-next to rise a warning when people break
> >    this.
> 
> I'm not sure if Stephen has that much spare time, and I don't think it
> is necessary. I think the checkpatch check is sufficient and I'll check
> myself occasionally.

Yes, that's fine, I just wanted to speak this out - there is a
probability that someone gets excited and creates some instrumentation
to kbuild to automatically detect bad things and then Stephen could
easily use that.

WRT "not necessary" - well, I think it is always better to cut bad
things before they are merged, rather than afterwards.

But anyway, let's forget about this.

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 00/34] Make kernel build deterministic

[Michal Marek]

On 5.4.2011 17:49, Greg KH wrote:
> Very nice stuff.  Do you want to take the individual patches through one
> of your trees, or do you mind if the subsystem maintainers take them
> through theirs?

I'd leave it up to the subsystem maintainers. I'll check once the merge
window opens and send the remaining patches to Linus directly.

Michal

Re: [PATCH 00/34] Make kernel build deterministic

[Ingo Molnar]

* Michal Marek <mmarek@suse.cz> wrote:

> 
> Hi,
> 
> this series makes it possible to build bit-identical kernel image and
> modules from identical sources. Of course the build is already
> deterministic in terms of behavior of the code, but the various
> timestamps embedded in the object files make it hard to compare two
> builds, for instance to verify that a makefile cleanup didn't
> accidentally change something. A prime example is /proc/config.gz, which
> has both a timestamp in the gzip header and a timestamp in the payload
> data. With this series applied, a script like this will produce
> identical kernels each time:
> 
> #!/bin/bash
> rm -rf /dev/shm/{source,build}{,1,2}
> export KCONFIG_NOTIMESTAMP=1
> export KBUILD_BUILD_TIMESTAMP='Sun May  1 12:00:00 CEST 2011'
> export KBUILD_BUILD_USER=user
> export KBUILD_BUILD_HOST=host
> export ROOT_DEV=FLOPPY
> for i in 1 2; do
> 	mkdir /dev/shm/source
> 	# randomize the inode order just for fun
> 	git ls-tree -r -z --name-only HEAD | sort -R -z | xargs -0 \
> 		cp --parents --target=/dev/shm/source
> 	pushd /dev/shm/source
> 	mkdir /dev/shm/build
> 	>/dev/shm/build/all.config
> 	for opt in GCOV_KERNEL UBIFS_FS_DEBUG; do
> 		echo "# CONFIG_$opt is not set" >>"/dev/shm/build"/all.config
> 	done
> 	make O="/dev/shm/build" KCONFIG_ALLCONFIG=1 allmodconfig
> 	make O="/dev/shm/build" -j64
> 	popd
> 	mv /dev/shm/source /dev/shm/source$i
> 	mv /dev/shm/build /dev/shm/build$i
> done
> diff -rq /dev/shm/build{1,2}

Nice!

Acked-by: Ingo Molnar <mingo@elte.hu>

> Unfortunatelly, this cannot be used to validate indentation-only
> patches, even if CONFIG_DEBUG_INFO is turned off. This is because of the
> __FILE__ and __LINE__ macros used in many places. For the same reason,
> the source and build directory needs to be the same, otherwise the
> results will differ.

Nor can it be used to validate untrusted patches in general: a subtle change 
might be introduced in a piece of code that is dependent on a .config detail 
that is off for that particular build.

But in the common case it's nice to be able to have deterministic/reproducable 
builds.

Thanks,

	Ingo