Re: Fwd: xenbusb_nop_confighook_cb timeout and cd issue

Re: Fwd: xenbusb_nop_confighook_cb timeout and cd issue

[Christoph Egger]

On 14.01.13 07:43, Steven Chamberlain wrote:

> Hi,
> 
>> kernel: run_interrupt_driven_hooks: still waiting after 60 seconds for xenbusb_nop_confighook_cb
> <boot hangs before mounting root fs>
> 
> It is not really a FreeBSD bug but some regression in Xen 4.1.3.  It
> only affects FreeBSD kernels if built with the XENHVM option.
> 
> On 06/12/12 11:04, Egoitz Aurrekoetxea Aurre wrote:
>> The patch of the URL is :  http://postfixquotareject.ramattack.net/xen-x86-interrupt-pointer-missmatch.diff
> 
> Thank you!  I just tested it on a NetBSD-6 Xen dom0 and it does indeed
> fix this (I started seeing this problem after an upgrade to
> pkgsrc-2012Q4;  Christoph:  would you be interested in this patch?)

> 

> The patch originally comes from:
> http://patch-tracker.debian.org/patch/series/view/xen/4.1.3-7/xen-x86-interrupt-pointer-missmatch.diff
> 
> A lot of Debian-FreeBSD-NetBSD collaboration seems to be happening here :)
> 
> Regards,


Thanks for this patch.
I am adding xen-devel for upstream.

Christoph

Re: Fwd: xenbusb_nop_confighook_cb timeout and cd issue

[Andrew Cooper]

On 17/01/13 07:36, Christoph Egger wrote:
> On 14.01.13 07:43, Steven Chamberlain wrote:
>
>> Hi,
>>
>>> kernel: run_interrupt_driven_hooks: still waiting after 60 seconds for xenbusb_nop_confighook_cb
>> <boot hangs before mounting root fs>
>>
>> It is not really a FreeBSD bug but some regression in Xen 4.1.3.  It
>> only affects FreeBSD kernels if built with the XENHVM option.
>>
>> On 06/12/12 11:04, Egoitz Aurrekoetxea Aurre wrote:
>>> The patch of the URL is :  http://postfixquotareject.ramattack.net/xen-x86-interrupt-pointer-missmatch.diff
>> Thank you!  I just tested it on a NetBSD-6 Xen dom0 and it does indeed
>> fix this (I started seeing this problem after an upgrade to
>> pkgsrc-2012Q4;  Christoph:  would you be interested in this patch?)
>> The patch originally comes from:
>> http://patch-tracker.debian.org/patch/series/view/xen/4.1.3-7/xen-x86-interrupt-pointer-missmatch.diff
>>
>> A lot of Debian-FreeBSD-NetBSD collaboration seems to be happening here :)
>>
>> Regards,
>
> Thanks for this patch.
> I am adding xen-devel for upstream.
>
> Christoph

config.h defines asmlinkage as nothing.

So I am really struggling to find how this code will alter the code in
Xen at all once the preprocessor has been completed.

~Andrew

>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: Fwd: xenbusb_nop_confighook_cb timeout and cd issue

[Steven Chamberlain]

Hi,

A little more background to this:

An error was first seen on Debian's i386 autobuilders when building Xen
4.1.2 (Debian package 4.1.2-7), but only after a switch from gcc-4.6 to
gcc-4.7 which seems to be what prompted this.  I'm not sure if it would
have affected amd64 builds as we don't have logs for those:

https://buildd.debian.org/status/fetch.php?pkg=xen&arch=i386&ver=4.1.3-7&stamp=1355254810
:
> gcc -O2 -fomit-frame-pointer -m32 -march=i686 -fno-strict-aliasing -std=gnu99 -Wall -Wstrict-prototypes -Wno-unused-value -Wdeclaration-after-statement -Wno-unused-but-set-variable  -DNDEBUG -nostdinc -fno-builtin -fno-common -Wredundant-decls -iwithprefix include -Werror -Wno-pointer-arith -pipe -I/build/buildd-xen_4.1.3~rc1+hg-20120614.a9c0a89c08f2-1-i386-iqa4wM/xen-4.1.3~rc1+hg-20120614.a9c0a89c08f2/debian/build/build-hypervisor_i386_i386/xen/include  -I/build/buildd-xen_4.1.3~rc1+hg-20120614.a9c0a89c08f2-1-i386-iqa4wM/xen-4.1.3~rc1+hg-20120614.a9c0a89c08f2/debian/build/build-hypervisor_i386_i386/xen/include/asm-x86/mach-generic -I/build/buildd-xen_4.1.3~rc1+hg-20120614.a9c0a89c08f2-1-i386-iqa4wM/xen-4.1.3~rc1+hg-20120614.a9c0a89c08f2/debian/build/build-hypervisor_i386_i386/xen/includ
 e/asm-x86/mach-default -msoft-float -fno-stack-protector -fno-exceptions -g -D__XEN__ -MMD -MF .i8259.o.d -c i8259.c -o i8259.o
> i8259.c:66:9: error: initialization from incompatible pointer type [-Werror]
> i8259.c:66:9: error: (near initialization for 'interrupt[0]') [-Werror]
> i8259.c:66:9: error: initialization from incompatible pointer type [-Werror]
> i8259.c:66:9: error: (near initialization for 'interrupt[1]') [-Werror]
> i8259.c:66:9: error: initialization from incompatible pointer type [-Werror]
> i8259.c:66:9: error: (near initialization for 'interrupt[2]') [-Werror]

In order to fix the build issue, this patch was written, and is still
used when building Debian's Xen 4.1.3 packages:
http://patch-tracker.debian.org/patch/series/view/xen/4.1.3-7/xen-x86-interrupt-pointer-missmatch.diff

Since October 2012 many FreeBSD (9.x) users reported an issue booting on
Xenserver 6.1 / XCP 1.6 (incl. -BETA) but not affecting XCP 1.5:
http://lists.freebsd.org/pipermail/freebsd-xen/2012-October/001374.html

There were prior reports of identical symptoms triggered by guest CD-ROM
drives without media, but this seems to be an unrelated bug.

Egoitz, who has done a lot of work porting FreeBSD to XCP recently,
found that Debian's (Wheezy) Xen kernel didn't have this bug;  realised
that the above patch fixes it.

I experienced the bug when booting the FreeBSD 9.0 kernel on a NetBSD
6.0.1 dom0 after upgrading to pkgsrc-2012Q4 (Xen 4.1.3, from a binary
package, unsure which compiler was used to build it).  Likewise the
patch fixed the issue for me;  I used gcc 4.5.3.

Regards,
-- 
Steven Chamberlain
steven@pyro.eu.org

Re: Fwd: xenbusb_nop_confighook_cb timeout and cd issue

[Christoph Egger]

Am 17.01.2013 12:19, schrieb Steven Chamberlain:
> Hi,
> 
> A little more background to this:
> 
> An error was first seen on Debian's i386 autobuilders when building Xen
> 4.1.2 (Debian package 4.1.2-7), but only after a switch from gcc-4.6 to
> gcc-4.7 which seems to be what prompted this.  I'm not sure if it would
> have affected amd64 builds as we don't have logs for those:
> 
> https://buildd.debian.org/status/fetch.php?pkg=xen&arch=i386&ver=4.1.3-7&stamp=1355254810
> :
>> gcc -O2 -fomit-frame-pointer -m32 -march=i686 -fno-strict-aliasing -std=gnu99 -Wall -Wstrict-prototypes -Wno-unused-value -Wdeclaration-after-statement -Wno-unused-but-set-variable  -DNDEBUG -nostdinc -fno-builtin -fno-common -Wredundant-decls -iwithprefix include -Werror -Wno-pointer-arith -pipe -I/build/buildd-xen_4.1.3~rc1+hg-20120614.a9c0a89c08f2-1-i386-iqa4wM/xen-4.1.3~rc1+hg-20120614.a9c0a89c08f2/debian/build/build-hypervisor_i386_i386/xen/include  -I/build/buildd-xen_4.1.3~rc1+hg-20120614.a9c0a89c08f2-1-i386-iqa4wM/xen-4.1.3~rc1+hg-20120614.a9c0a89c08f2/debian/build/build-hypervisor_i386_i386/xen/include/asm-x86/mach-generic -I/build/buildd-xen_4.1.3~rc1+hg-20120614.a9c0a89c08f2-1-i386-iqa4wM/xen-4.1.3~rc1+hg-20120614.a9c0a89c08f2/debian/build/build-hypervisor_i386_i386/xen/inclu
 de/asm-x86/mach-default -msoft-float -fno-stack-protector -fno-exceptions -g -D__XEN__ -MMD -MF .i8259.o.d -c i8259.c -o i8259.o
>> i8259.c:66:9: error: initialization from incompatible pointer type [-Werror]
>> i8259.c:66:9: error: (near initialization for 'interrupt[0]') [-Werror]
>> i8259.c:66:9: error: initialization from incompatible pointer type [-Werror]
>> i8259.c:66:9: error: (near initialization for 'interrupt[1]') [-Werror]
>> i8259.c:66:9: error: initialization from incompatible pointer type [-Werror]
>> i8259.c:66:9: error: (near initialization for 'interrupt[2]') [-Werror]
> 
> In order to fix the build issue, this patch was written, and is still
> used when building Debian's Xen 4.1.3 packages:
> http://patch-tracker.debian.org/patch/series/view/xen/4.1.3-7/xen-x86-interrupt-pointer-missmatch.diff
> 
> Since October 2012 many FreeBSD (9.x) users reported an issue booting on
> Xenserver 6.1 / XCP 1.6 (incl. -BETA) but not affecting XCP 1.5:
> http://lists.freebsd.org/pipermail/freebsd-xen/2012-October/001374.html
> 
> There were prior reports of identical symptoms triggered by guest CD-ROM
> drives without media, but this seems to be an unrelated bug.
> 
> Egoitz, who has done a lot of work porting FreeBSD to XCP recently,
> found that Debian's (Wheezy) Xen kernel didn't have this bug;  realised
> that the above patch fixes it.
> 
> I experienced the bug when booting the FreeBSD 9.0 kernel on a NetBSD
> 6.0.1 dom0 after upgrading to pkgsrc-2012Q4 (Xen 4.1.3, from a binary
> package, unsure which compiler was used to build it).  Likewise the
> patch fixed the issue for me;  I used gcc 4.5.3.
> 
> Regards,
> 

I compiled a clean checked out xen-4.1-testing.hg with and without the patch.

I created a objdump -S -D  from each xen-syms.
A diff -u between both objdump's shows this:

--- xen-syms-unpatched.objdump  2013-01-17 12:47:25.000000000 +0000
+++ xen-syms-patched.objdump    2013-01-17 12:52:26.000000000 +0000
@@ -1,5 +1,5 @@

-xen-syms-unpatched:     file format elf64-x86-64
+xen-syms-patched:     file format elf64-x86-64

 Disassembly of section .text:

@@ -435776,16 +435776,20 @@ ffff82c480215a03:   20 31
 ffff82c480215a05:      37                      (bad)
 ffff82c480215a06:      20 31                   and    %dh,(%rcx)
 ffff82c480215a08:      32 3a                   xor    (%rdx),%bh
-ffff82c480215a0a:      33 33                   xor    (%rbx),%esi
-ffff82c480215a0c:      3a 33                   cmp    (%rbx),%dh
-ffff82c480215a0e:      35 20 55 54 43          xor    $0x43545520,%eax
-ffff82c480215a13:      20 32                   and    %dh,(%rdx)
+ffff82c480215a0a:      33 30                   xor    (%rax),%esi
+ffff82c480215a0c:      3a 30                   cmp    (%rax),%dh
+ffff82c480215a0e:      39 20                   cmp    %esp,(%rax)
+ffff82c480215a10:      55                      push   %rbp
+ffff82c480215a11:      54                      push   %rsp
+ffff82c480215a12:      43 20 32                rexYZ and    %sil,(%r10)
 ffff82c480215a15:      30 31                   xor    %dh,(%rcx)
 ffff82c480215a17:      33 00                   xor    (%rax),%eax
 ffff82c480215a19:      31 32                   xor    %esi,(%rdx)
 ffff82c480215a1b:      3a 33                   cmp    (%rbx),%dh
-ffff82c480215a1d:      33 3a                   xor    (%rdx),%edi
-ffff82c480215a1f:      33 35 00 63 68 65       xor    1701339904(%rip),%esi        # ffff82c4e589bd25 <_end+0x655de725>
+ffff82c480215a1d:      30 3a                   xor    %bh,(%rdx)
+ffff82c480215a1f:      30 39                   xor    %bh,(%rcx)
+ffff82c480215a21:      00 63 68                add    %ah,0x68(%rbx)
+ffff82c480215a24:      65                      gs
 ffff82c480215a25:      67                      addr32
 ffff82c480215a26:      67                      addr32
 ffff82c480215a27:      65                      gs


The fact that this patch causes gcc (4.1.3 on a redhat box in my case)
to generate different assembly code makes me think that there is something
wrong with gcc. The fact that FreeBSD hangs even when Xen is compiled with
gcc 4.6 tells me that also gcc 4.6 is affected.

It is interesting to know that doing things semantically correct
works around it.
So I would take build fixes for gcc 4.7 more serious than just a "build fixes".

I think this should be shown to gcc dev folk to have a look into this.

In xen-unstable Jan Beulich moved the interrupt array into the
.init.data section. I would like to know if FreeBSD 9.1 hangs or
boots fine with xen-unstable.

Christoph

Re: Fwd: xenbusb_nop_confighook_cb timeout and cd issue

[Juergen Gross]

Am 17.01.2013 15:35, schrieb Christoph Egger:
> Am 17.01.2013 12:19, schrieb Steven Chamberlain:
>> Hi,
>>
>> A little more background to this:
>>
>> An error was first seen on Debian's i386 autobuilders when building Xen
>> 4.1.2 (Debian package 4.1.2-7), but only after a switch from gcc-4.6 to
>> gcc-4.7 which seems to be what prompted this.  I'm not sure if it would
>> have affected amd64 builds as we don't have logs for those:
>>
>> https://buildd.debian.org/status/fetch.php?pkg=xen&arch=i386&ver=4.1.3-7&stamp=1355254810
>> :
>>> gcc -O2 -fomit-frame-pointer -m32 -march=i686 -fno-strict-aliasing -std=gnu99 -Wall -Wstrict-prototypes -Wno-unused-value -Wdeclaration-after-statement -Wno-unused-but-set-variable  -DNDEBUG -nostdinc -fno-builtin -fno-common -Wredundant-decls -iwithprefix include -Werror -Wno-pointer-arith -pipe -I/build/buildd-xen_4.1.3~rc1+hg-20120614.a9c0a89c08f2-1-i386-iqa4wM/xen-4.1.3~rc1+hg-20120614.a9c0a89c08f2/debian/build/build-hypervisor_i386_i386/xen/include  -I/build/buildd-xen_4.1.3~rc1+hg-20120614.a9c0a89c08f2-1-i386-iqa4wM/xen-4.1.3~rc1+hg-20120614.a9c0a89c08f2/debian/build/build-hypervisor_i386_i386/xen/include/asm-x86/mach-generic -I/build/buildd-xen_4.1.3~rc1+hg-20120614.a9c0a89c08f2-1-i386-iqa4wM/xen-4.1.3~rc1+hg-20120614.a9c0a89c08f2/debian/build/build-hypervisor_i386_i386/xen/incl
 ude/asm-x86/mach-default -msoft-float -fno-stack-protector -fno-exceptions -g -D__XEN__ -MMD -MF .i8259.o.d -c i8259.c -o i8259.o
>>> i8259.c:66:9: error: initialization from incompatible pointer type [-Werror]
>>> i8259.c:66:9: error: (near initialization for 'interrupt[0]') [-Werror]
>>> i8259.c:66:9: error: initialization from incompatible pointer type [-Werror]
>>> i8259.c:66:9: error: (near initialization for 'interrupt[1]') [-Werror]
>>> i8259.c:66:9: error: initialization from incompatible pointer type [-Werror]
>>> i8259.c:66:9: error: (near initialization for 'interrupt[2]') [-Werror]
>>
>> In order to fix the build issue, this patch was written, and is still
>> used when building Debian's Xen 4.1.3 packages:
>> http://patch-tracker.debian.org/patch/series/view/xen/4.1.3-7/xen-x86-interrupt-pointer-missmatch.diff
>>
>> Since October 2012 many FreeBSD (9.x) users reported an issue booting on
>> Xenserver 6.1 / XCP 1.6 (incl. -BETA) but not affecting XCP 1.5:
>> http://lists.freebsd.org/pipermail/freebsd-xen/2012-October/001374.html
>>
>> There were prior reports of identical symptoms triggered by guest CD-ROM
>> drives without media, but this seems to be an unrelated bug.
>>
>> Egoitz, who has done a lot of work porting FreeBSD to XCP recently,
>> found that Debian's (Wheezy) Xen kernel didn't have this bug;  realised
>> that the above patch fixes it.
>>
>> I experienced the bug when booting the FreeBSD 9.0 kernel on a NetBSD
>> 6.0.1 dom0 after upgrading to pkgsrc-2012Q4 (Xen 4.1.3, from a binary
>> package, unsure which compiler was used to build it).  Likewise the
>> patch fixed the issue for me;  I used gcc 4.5.3.
>>
>> Regards,
>>
>
> I compiled a clean checked out xen-4.1-testing.hg with and without the patch.
>
> I created a objdump -S -D  from each xen-syms.
> A diff -u between both objdump's shows this:
>
> --- xen-syms-unpatched.objdump  2013-01-17 12:47:25.000000000 +0000
> +++ xen-syms-patched.objdump    2013-01-17 12:52:26.000000000 +0000
> @@ -1,5 +1,5 @@
>
> -xen-syms-unpatched:     file format elf64-x86-64
> +xen-syms-patched:     file format elf64-x86-64
>
>   Disassembly of section .text:
>
> @@ -435776,16 +435776,20 @@ ffff82c480215a03:   20 31
>   ffff82c480215a05:      37                      (bad)
>   ffff82c480215a06:      20 31                   and    %dh,(%rcx)
>   ffff82c480215a08:      32 3a                   xor    (%rdx),%bh
> -ffff82c480215a0a:      33 33                   xor    (%rbx),%esi
> -ffff82c480215a0c:      3a 33                   cmp    (%rbx),%dh
> -ffff82c480215a0e:      35 20 55 54 43          xor    $0x43545520,%eax
> -ffff82c480215a13:      20 32                   and    %dh,(%rdx)
> +ffff82c480215a0a:      33 30                   xor    (%rax),%esi
> +ffff82c480215a0c:      3a 30                   cmp    (%rax),%dh
> +ffff82c480215a0e:      39 20                   cmp    %esp,(%rax)
> +ffff82c480215a10:      55                      push   %rbp
> +ffff82c480215a11:      54                      push   %rsp
> +ffff82c480215a12:      43 20 32                rexYZ and    %sil,(%r10)
>   ffff82c480215a15:      30 31                   xor    %dh,(%rcx)
>   ffff82c480215a17:      33 00                   xor    (%rax),%eax
>   ffff82c480215a19:      31 32                   xor    %esi,(%rdx)
>   ffff82c480215a1b:      3a 33                   cmp    (%rbx),%dh
> -ffff82c480215a1d:      33 3a                   xor    (%rdx),%edi
> -ffff82c480215a1f:      33 35 00 63 68 65       xor    1701339904(%rip),%esi        # ffff82c4e589bd25<_end+0x655de725>
> +ffff82c480215a1d:      30 3a                   xor    %bh,(%rdx)
> +ffff82c480215a1f:      30 39                   xor    %bh,(%rcx)
> +ffff82c480215a21:      00 63 68                add    %ah,0x68(%rbx)
> +ffff82c480215a24:      65                      gs
>   ffff82c480215a25:      67                      addr32
>   ffff82c480215a26:      67                      addr32
>   ffff82c480215a27:      65                      gs

Are you sure? This looks like different ASCII texts to me:

-"12:33:35 UTC 2013"
-"12:33:35"
+"12:30:39 UTC 2013"
+"12:30:39"
  "chegge..."

Seems to be related to the time the file was generated.

Juergen

-- 
Juergen Gross                 Principal Developer Operating Systems
PBG PDG ES&S SWE OS6                   Telephone: +49 (0) 89 3222 2967
Fujitsu Technology Solutions              e-mail: juergen.gross@ts.fujitsu.com
Domagkstr. 28                           Internet: ts.fujitsu.com
D-80807 Muenchen                 Company details: ts.fujitsu.com/imprint.html

Re: Fwd: xenbusb_nop_confighook_cb timeout and cd issue

[Mark Felder]

Citrix has an open internal bug for this because of my incessant nagging.  
I'm going to direct them to this patch. If I can get their build manager  
to spin up a copy of XCP 1.6 with this patch included I'll make sure I get  
the ISO linked onto this mailing list.

Re: Fwd: xenbusb_nop_confighook_cb timeout and cd issue

[Andrew Cooper]

On 17/01/13 14:50, Juergen Gross wrote:
> Am 17.01.2013 15:35, schrieb Christoph Egger:
>> Am 17.01.2013 12:19, schrieb Steven Chamberlain:
>>> Hi,
>>>
>>> A little more background to this:
>>>
>>> An error was first seen on Debian's i386 autobuilders when building Xen
>>> 4.1.2 (Debian package 4.1.2-7), but only after a switch from gcc-4.6 to
>>> gcc-4.7 which seems to be what prompted this.  I'm not sure if it would
>>> have affected amd64 builds as we don't have logs for those:
>>>
>>> https://buildd.debian.org/status/fetch.php?pkg=xen&arch=i386&ver=4.1.3-7&stamp=1355254810
>>> :
>>>> gcc -O2 -fomit-frame-pointer -m32 -march=i686 -fno-strict-aliasing -std=gnu99 -Wall -Wstrict-prototypes -Wno-unused-value -Wdeclaration-after-statement -Wno-unused-but-set-variable  -DNDEBUG -nostdinc -fno-builtin -fno-common -Wredundant-decls -iwithprefix include -Werror -Wno-pointer-arith -pipe -I/build/buildd-xen_4.1.3~rc1+hg-20120614.a9c0a89c08f2-1-i386-iqa4wM/xen-4.1.3~rc1+hg-20120614.a9c0a89c08f2/debian/build/build-hypervisor_i386_i386/xen/include  -I/build/buildd-xen_4.1.3~rc1+hg-20120614.a9c0a89c08f2-1-i386-iqa4wM/xen-4.1.3~rc1+hg-20120614.a9c0a89c08f2/debian/build/build-hypervisor_i386_i386/xen/include/asm-x86/mach-generic -I/build/buildd-xen_4.1.3~rc1+hg-20120614.a9c0a89c08f2-1-i386-iqa4wM/xen-4.1.3~rc1+hg-20120614.a9c0a89c08f2/debian/build/build-hypervisor_i386_i386/xen/inc
 lude/asm-x86/mach-default -msoft-float -fno-stack-protector -fno-exceptions -g -D__XEN__ -MMD -MF .i8259.o.d -c i8259.c -o i8259.o
>>>> i8259.c:66:9: error: initialization from incompatible pointer type [-Werror]
>>>> i8259.c:66:9: error: (near initialization for 'interrupt[0]') [-Werror]
>>>> i8259.c:66:9: error: initialization from incompatible pointer type [-Werror]
>>>> i8259.c:66:9: error: (near initialization for 'interrupt[1]') [-Werror]
>>>> i8259.c:66:9: error: initialization from incompatible pointer type [-Werror]
>>>> i8259.c:66:9: error: (near initialization for 'interrupt[2]') [-Werror]
>>> In order to fix the build issue, this patch was written, and is still
>>> used when building Debian's Xen 4.1.3 packages:
>>> http://patch-tracker.debian.org/patch/series/view/xen/4.1.3-7/xen-x86-interrupt-pointer-missmatch.diff
>>>
>>> Since October 2012 many FreeBSD (9.x) users reported an issue booting on
>>> Xenserver 6.1 / XCP 1.6 (incl. -BETA) but not affecting XCP 1.5:
>>> http://lists.freebsd.org/pipermail/freebsd-xen/2012-October/001374.html
>>>
>>> There were prior reports of identical symptoms triggered by guest CD-ROM
>>> drives without media, but this seems to be an unrelated bug.
>>>
>>> Egoitz, who has done a lot of work porting FreeBSD to XCP recently,
>>> found that Debian's (Wheezy) Xen kernel didn't have this bug;  realised
>>> that the above patch fixes it.
>>>
>>> I experienced the bug when booting the FreeBSD 9.0 kernel on a NetBSD
>>> 6.0.1 dom0 after upgrading to pkgsrc-2012Q4 (Xen 4.1.3, from a binary
>>> package, unsure which compiler was used to build it).  Likewise the
>>> patch fixed the issue for me;  I used gcc 4.5.3.
>>>
>>> Regards,
>>>
>> I compiled a clean checked out xen-4.1-testing.hg with and without the patch.
>>
>> I created a objdump -S -D  from each xen-syms.
>> A diff -u between both objdump's shows this:
>>
>> --- xen-syms-unpatched.objdump  2013-01-17 12:47:25.000000000 +0000
>> +++ xen-syms-patched.objdump    2013-01-17 12:52:26.000000000 +0000
>> @@ -1,5 +1,5 @@
>>
>> -xen-syms-unpatched:     file format elf64-x86-64
>> +xen-syms-patched:     file format elf64-x86-64
>>
>>   Disassembly of section .text:
>>
>> @@ -435776,16 +435776,20 @@ ffff82c480215a03:   20 31
>>   ffff82c480215a05:      37                      (bad)
>>   ffff82c480215a06:      20 31                   and    %dh,(%rcx)
>>   ffff82c480215a08:      32 3a                   xor    (%rdx),%bh
>> -ffff82c480215a0a:      33 33                   xor    (%rbx),%esi
>> -ffff82c480215a0c:      3a 33                   cmp    (%rbx),%dh
>> -ffff82c480215a0e:      35 20 55 54 43          xor    $0x43545520,%eax
>> -ffff82c480215a13:      20 32                   and    %dh,(%rdx)
>> +ffff82c480215a0a:      33 30                   xor    (%rax),%esi
>> +ffff82c480215a0c:      3a 30                   cmp    (%rax),%dh
>> +ffff82c480215a0e:      39 20                   cmp    %esp,(%rax)
>> +ffff82c480215a10:      55                      push   %rbp
>> +ffff82c480215a11:      54                      push   %rsp
>> +ffff82c480215a12:      43 20 32                rexYZ and    %sil,(%r10)
>>   ffff82c480215a15:      30 31                   xor    %dh,(%rcx)
>>   ffff82c480215a17:      33 00                   xor    (%rax),%eax
>>   ffff82c480215a19:      31 32                   xor    %esi,(%rdx)
>>   ffff82c480215a1b:      3a 33                   cmp    (%rbx),%dh
>> -ffff82c480215a1d:      33 3a                   xor    (%rdx),%edi
>> -ffff82c480215a1f:      33 35 00 63 68 65       xor    1701339904(%rip),%esi        # ffff82c4e589bd25<_end+0x655de725>
>> +ffff82c480215a1d:      30 3a                   xor    %bh,(%rdx)
>> +ffff82c480215a1f:      30 39                   xor    %bh,(%rcx)
>> +ffff82c480215a21:      00 63 68                add    %ah,0x68(%rbx)
>> +ffff82c480215a24:      65                      gs
>>   ffff82c480215a25:      67                      addr32
>>   ffff82c480215a26:      67                      addr32
>>   ffff82c480215a27:      65                      gs
> Are you sure? This looks like different ASCII texts to me:
>
> -"12:33:35 UTC 2013"
> -"12:33:35"
> +"12:30:39 UTC 2013"
> +"12:30:39"
>   "chegge..."
>
> Seems to be related to the time the file was generated.
>
> Juergen
>

Yes - the timestamp is embedded into the image, and gets printed at the
start of day.

So given that this is the only difference across the two compiled Xens,
I fail to see how the patch is having any effect whatsoever on the
observed bug.

~Andrew

Re: Fwd: xenbusb_nop_confighook_cb timeout and cd issue

[Steven Chamberlain]

On 17/01/13 14:53, Mark Felder wrote:
> Citrix has an open internal bug for this because of my incessant
> nagging. I'm going to direct them to this patch.

I'm afraid the patch can't be what really fixed this.  Andrew Cooper is
right that the pre-processed output is identical (except for a space),
and so are all the object files except for version.o

My gcc 4.5 builds on NetBSD;  and gcc 4.7 on Debian Wheezy;  seem to be
the same with/without the patch.

Maybe it was something as trivial as the reboot that made it go away?

I notice now that gcc 4.7 was updated on Debian buildds, so a compiler
bugfix may be what really fixed the issue there.


My NetBSD pkgsrc build still differs from the distributed binary package
in a strange way, when the chroot build environments should be the same.

(This is a diff of 'strings')

--- netbsd-binary/xen41-kernel/xen.s   2013-01-17 13:45:08.000000000 +0000
+++ netbsd-local-unpatched/xen41-kernel/xen.s  2013-01-17
13:45:13.000000000 +0000
@@ -9641,10 +9636,10 @@
 hadow_
 pars
 allo
-page_
 ched_
 ister
 compa
+page_
 entr
 .clone.
 clone.
@@ -12250,7 +12245,6 @@
     PoD entries=%d cachesize=%d
 %s: Out of populate-on-demand memory! tot_pages %u pod_entries %i
 pg error: %s(): p2m already allocated for this domain
-%s: gfn_to_mfn returned type %d!
 <G><1>%s:%d:d%d Adding bad mfn to p2m map (%#lx -> %#lx)
 <G><0>%s:%d:d%d set_mmio_p2m_entry: set_p2m_entry failed! mfn=%08lx
 <G><0>%s:%d:d%d clear_mmio_p2m_entry: gfn_to_mfn failed! gfn=%08lx

Regards,
-- 
Steven Chamberlain
steven@pyro.eu.org

Re: Fwd: xenbusb_nop_confighook_cb timeout and cd issue

[Steven Chamberlain]

On 17/01/13 14:53, Mark Felder wrote:
> Citrix has an open internal bug for this because of my incessant
> nagging.

Actually - Mark I'm sure you've tried many reboots and are still able to
reproduce it?  So this must be something to do with the build Xen kernel
image after all.

Regards,
-- 
Steven Chamberlain
steven@pyro.eu.org

Re: Fwd: xenbusb_nop_confighook_cb timeout and cd issue

[Mark Felder]

I no longer have that environment that I was using to test the viability of upgrading to XCP 1.6 but it was reproducible every boot.

Re: Fwd: xenbusb_nop_confighook_cb timeout and cd issue

[Steven Chamberlain]

On 17/01/13 15:24, Mark Felder wrote:
> I no longer have that environment that I was using to test the viability of upgrading to XCP 1.6 but it was reproducible every boot.

Just to clarify - you rebooted the dom0/hypervisor also?

I know the bug is reproducible on every FreeBSD domU boot, at least.

Regards,
-- 
Steven Chamberlain
steven@pyro.eu.org

Re: Fwd: xenbusb_nop_confighook_cb timeout and cd issue

[Mark Felder]

For anyone running XCP 1.6 please try to use these RPMs which are patched.  
I'll be trying this myself later today.

http://downloads.xen.org/XCP/freebsd-xen-fix/


Cheers!

Re: Fwd: xenbusb_nop_confighook_cb timeout and cd issue

[Mark Felder]

On Thu, 17 Jan 2013 09:56:33 -0600, Steven Chamberlain  
<steven@pyro.eu.org> wrote:

> On 17/01/13 15:24, Mark Felder wrote:
>> I no longer have that environment that I was using to test the  
>> viability of upgrading to XCP 1.6 but it was reproducible every boot.
>
> Just to clarify - you rebooted the dom0/hypervisor also?
>
> I know the bug is reproducible on every FreeBSD domU boot, at least.
>

We did reboot the dom0 more than once but I didn't test that specifically  
to see if the problem would subside after a few reboots.

Re: Fwd: xenbusb_nop_confighook_cb timeout and cd issue

[Steven Chamberlain]

Hi Mark,

On 17/01/13 17:50, Mark Felder wrote:
> For anyone running XCP 1.6 please try to use these RPMs which are
> patched. I'll be trying this myself later today.
> 
> http://downloads.xen.org/XCP/freebsd-xen-fix/

Did you have chance to try it?  Did it make any difference for you?

Regards,
-- 
Steven Chamberlain
steven@pyro.eu.org

Re: Fwd: xenbusb_nop_confighook_cb timeout and cd issue

[Mark Felder]

On Wed, 30 Jan 2013 08:18:39 -0600, Steven Chamberlain  
<steven@pyro.eu.org> wrote:

> Hi Mark,
>
> On 17/01/13 17:50, Mark Felder wrote:
>> For anyone running XCP 1.6 please try to use these RPMs which are
>> patched. I'll be trying this myself later today.
>>
>> http://downloads.xen.org/XCP/freebsd-xen-fix/
>
> Did you have chance to try it?  Did it make any difference for you?
>
> Regards,

Hi Steve,

I did try it and it didn't seem to make a difference. I was going to check  
with my contact and have him reconfirm that the patches actually made it  
into those RPMs but I haven't had time :(

Re: Fwd: xenbusb_nop_confighook_cb timeout and cd issue

[Steven Chamberlain]

Hi,

On 10:14, Mark Felder wrote:
> I did try it and it didn't seem to make a difference. I was going to
> check with my contact and have him reconfirm that the patches
> actually made it into those RPMs but I haven't had time :(

Okay, thanks for the feedback.

Unfortunately as Andrew Cooper pointed out, that patch turns out to have
zero effect on the compiled code.

Whatever fixed this for me, with xenkernel41 4.1.3nb2 on NetBSD, is
some other, unexplained difference between NetBSD's builds and my own
local builds, of the same source, in what should be an essentially
identical chrooted build environment (GCC 4.5.3 (NetBSD nb2 20110806),
pkgsrc-2012Q4).

Somehow both of the XCP 1.6 builds had this problem too (GCC 4.1.2
20080704 (Red Hat 4.1.2-46)).

Debian's builds (GCC 4.7.1) seem to be unaffected, although they carry a
lot of patches as well.

Regards,
-- 
Steven Chamberlain
steven@pyro.eu.org