Problem with kexec on i386, linux-3.5

Problem with kexec on i386, linux-3.5

[Christian Schaubschläger]

Hello list,

I'm not sure if this is the correct place to post this; if it's not, I'd like to apologize.

Here's a short description of my problem:

I have a tiny protected-/real mode program, which I start using kexec (kexec-tools 2.0.3 released 05 April 2012). At some point this program makes a call to extended-int13 to read data from the disk. Now starting with linux-3.5-rc1 (and at least up to linux-3.5) this extended int13 call does not work any more. Apparently the call returns with error code 0x80, which means "timeout (not ready)".

I have two machines here, both with Intel chipsets (one CougarPoint, one older ICH7-M), and I see the same behaviour on both machines.

When I use older kernels (starting from 2.6.something up to 3.4.6), everything works fine.

Now I'm not sure if this is a kernel issue, or a kexec issue, or a mistake by myself. Maybe someone has a hint for me...

If required, of course, I can provide more detailed information about my hardware, kernel config, etc. (since I'm not sure if this is the correct place, I wanted to keep this message short for now).

Thank you
Christian Schaubschlaeger




_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: Problem with kexec on i386, linux-3.5

[Eric W. Biederman]

Christian Schaubschläger <christian.schaubschlaeger@gmx.at> writes:

> Hello list,
>
> I'm not sure if this is the correct place to post this; if it's not,
> I'd like to apologize.
>
> Here's a short description of my problem:
>
> I have a tiny protected-/real mode program, which I start using kexec
> (kexec-tools 2.0.3 released 05 April 2012). At some point this program
> makes a call to extended-int13 to read data from the disk. Now
> starting with linux-3.5-rc1 (and at least up to linux-3.5) this
> extended int13 call does not work any more. Apparently the call
> returns with error code 0x80, which means "timeout (not ready)".
>
> I have two machines here, both with Intel chipsets (one CougarPoint,
> one older ICH7-M), and I see the same behaviour on both machines.
>
> When I use older kernels (starting from 2.6.something up to 3.4.6),
> everything works fine.
>
> Now I'm not sure if this is a kernel issue, or a kexec issue, or a
> mistake by myself. Maybe someone has a hint for me...
>
> If required, of course, I can provide more detailed information about
> my hardware, kernel config, etc. (since I'm not sure if this is the
> correct place, I wanted to keep this message short for now).

That is a tricky issue.  Sometimes the slightest things can set
something like this off.

Somewhere someone changed something in one of the drivers that made it
so that the hardware winds up in a state the int 13 disk driver does not
like it after kexec.

If you want to track this down I would recommend a bisect between 3.4
and 3.5-rc1 to see which change breaks your setup.

Once we know what kernel change caused this we can start to think about
how fixable your failure is.

Ideally we fix these kinds of things, in practice many times things are
just too hard to track down and things like this break.

Eric

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: Problem with kexec on i386, linux-3.5

[Christian Schaubschläger]

> That is a tricky issue.  Sometimes the slightest things can set
> something like this off.
>
> Somewhere someone changed something in one of the drivers that made it
> so that the hardware winds up in a state the int 13 disk driver does not
> like it after kexec.
> .
> If you want to track this down I would recommend a bisect between 3.4
> and 3.5-rc1 to see which change breaks your setup.

I bistcted that down to this patch:

commit b566a22c23327f18ce941ffad0ca907e50a53d41
Author: Khalid Aziz <khalid.aziz@hp.com>
Date:   Fri Apr 27 13:00:33 2012 -0600

    PCI: disable Bus Master on PCI device shutdown
   
    Disable Bus Master bit on the device in pci_device_shutdown() to ensure PCI
    devices do not continue to DMA data after shutdown.  This can cause memory
    corruption in case of a kexec where the current kernel shuts down and
    transfers control to a new kernel while a PCI device continues to DMA to
    memory that does not belong to it any more in the new kernel.
   
    I have tested this code on two laptops, two workstations and a 16-socket
    server.  kexec worked correctly on all of them.
   
    Signed-off-by: Khalid Aziz <khalid.aziz@hp.com>
    Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>


Without this patch, int13 works fine here! If anyone needs more information, just let me know!


Greetings,
Christian


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: Problem with kexec on i386, linux-3.5

[Eric W. Biederman]

Christian Schaubschläger <christian.schaubschlaeger@gmx.at> writes:

>> That is a tricky issue.  Sometimes the slightest things can set
>> something like this off.
>>
>> Somewhere someone changed something in one of the drivers that made it
>> so that the hardware winds up in a state the int 13 disk driver does not
>> like it after kexec.
>> .
>> If you want to track this down I would recommend a bisect between 3.4
>> and 3.5-rc1 to see which change breaks your setup.
>
> I bistcted that down to this patch:
>
> commit b566a22c23327f18ce941ffad0ca907e50a53d41
> Author: Khalid Aziz <khalid.aziz@hp.com>
> Date:   Fri Apr 27 13:00:33 2012 -0600
>
>     PCI: disable Bus Master on PCI device shutdown
>    
>     Disable Bus Master bit on the device in pci_device_shutdown() to ensure PCI
>     devices do not continue to DMA data after shutdown.  This can cause memory
>     corruption in case of a kexec where the current kernel shuts down and
>     transfers control to a new kernel while a PCI device continues to DMA to
>     memory that does not belong to it any more in the new kernel.
>    
>     I have tested this code on two laptops, two workstations and a 16-socket
>     server.  kexec worked correctly on all of them.
>    
>     Signed-off-by: Khalid Aziz <khalid.aziz@hp.com>
>     Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
>
>
> Without this patch, int13 works fine here! If anyone needs more
> information, just let me know!

Which leads to an interesting conundrum.

kexec appears to be more reliable for booting another kernel with this
patch applied.  This patch does kill the entier use case of making BIOS
calls, and I suspect it also does nasty things to alpha bootloaders.

My gut feel is that the trampoline code should reenable bus mastering
on the devices that lie behind int13, but I don't know how practical
that suggestion is in reality.

Eric

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: Problem with kexec on i386, linux-3.5

[Christian Schaubschläger]

>> I bistcted that down to this patch:
>>
>> commit b566a22c23327f18ce941ffad0ca907e50a53d41
>> Author: Khalid Aziz <khalid.aziz@hp.com>
>> Date:   Fri Apr 27 13:00:33 2012 -0600
>>
>>     PCI: disable Bus Master on PCI device shutdown
>>    
>>     Disable Bus Master bit on the device in pci_device_shutdown() to ensure PCI
>>     devices do not continue to DMA data after shutdown.  This can cause memory
>>     corruption in case of a kexec where the current kernel shuts down and
>>     transfers control to a new kernel while a PCI device continues to DMA to
>>     memory that does not belong to it any more in the new kernel.
>>    
>>     I have tested this code on two laptops, two workstations and a 16-socket
>>     server.  kexec worked correctly on all of them.
>>    
>>     Signed-off-by: Khalid Aziz <khalid.aziz@hp.com>
>>     Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
>>
>>
>> Without this patch, int13 works fine here! If anyone needs more
>> information, just let me know!
> Which leads to an interesting conundrum.
>
> kexec appears to be more reliable for booting another kernel with this
> patch applied.  This patch does kill the entier use case of making BIOS
> calls, and I suspect it also does nasty things to alpha bootloaders.
>
> My gut feel is that the trampoline code should reenable bus mastering
> on the devices that lie behind int13, but I don't know how practical
> that suggestion is in reality.

One could make this patch optional... e.g. let kexec tell the running kernel whether or not to disable bus mastering before booting the new one (something like 'kexec --disable-bus-mastering -l /boot/new_kernel.bin'). Don't know if this would be practicable...

Christian


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: Problem with kexec on i386, linux-3.5

[Khalid Aziz]

On Fri, 2012-08-17 at 08:58 +0200, Christian Schaubschläger wrote:
> >> I bistcted that down to this patch:
> >>
> >> commit b566a22c23327f18ce941ffad0ca907e50a53d41
> >> Author: Khalid Aziz <khalid.aziz@hp.com>
> >> Date:   Fri Apr 27 13:00:33 2012 -0600
> >>
> >>     PCI: disable Bus Master on PCI device shutdown
> >>    
> >>     Disable Bus Master bit on the device in pci_device_shutdown() to ensure PCI
> >>     devices do not continue to DMA data after shutdown.  This can cause memory
> >>     corruption in case of a kexec where the current kernel shuts down and
> >>     transfers control to a new kernel while a PCI device continues to DMA to
> >>     memory that does not belong to it any more in the new kernel.
> >>    
> >>     I have tested this code on two laptops, two workstations and a 16-socket
> >>     server.  kexec worked correctly on all of them.
> >>    
> >>     Signed-off-by: Khalid Aziz <khalid.aziz@hp.com>
> >>     Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
> >>
> >>
> >> Without this patch, int13 works fine here! If anyone needs more
> >> information, just let me know!
> > Which leads to an interesting conundrum.
> >
> > kexec appears to be more reliable for booting another kernel with this
> > patch applied.  This patch does kill the entier use case of making BIOS
> > calls, and I suspect it also does nasty things to alpha bootloaders.
> >
> > My gut feel is that the trampoline code should reenable bus mastering
> > on the devices that lie behind int13, but I don't know how practical
> > that suggestion is in reality.
> 
> One could make this patch optional... e.g. let kexec tell the running kernel whether or not to disable bus mastering before booting the new one (something like 'kexec --disable-bus-mastering -l /boot/new_kernel.bin'). Don't know if this would be practicable...
> 
> Christian
> 

Sorry, I have been on vacation and hadn't been watching mailing list. I
would not want to make this patch optional. It will be difficult for
most end users to tell if they should kexec with this enabled or not. I
would rather fix issues caused by this patch.

I will spend some time understanding the problem you are seeing and get
back to you.

Thanks
-- 
Khalid Aziz <khalid.aziz@hp.com>


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: Problem with kexec on i386, linux-3.5

[Khalid Aziz]

On 07/24/2012 11:49 AM, Christian Schaubschläger wrote:
> Hello list,
>
> I'm not sure if this is the correct place to post this; if it's not, 
> I'd like to apologize.
>
> Here's a short description of my problem:
>
> I have a tiny protected-/real mode program, which I start using kexec 
> (kexec-tools 2.0.3 released 05 April 2012). At some point this program 
> makes a call to extended-int13 to read data from the disk. Now 
> starting with linux-3.5-rc1 (and at least up to linux-3.5) this 
> extended int13 call does not work any more. Apparently the call 
> returns with error code 0x80, which means "timeout (not ready)".
>
> I have two machines here, both with Intel chipsets (one CougarPoint, 
> one older ICH7-M), and I see the same behaviour on both machines.
>
> When I use older kernels (starting from 2.6.something up to 3.4.6), 
> everything works fine.
>
> Now I'm not sure if this is a kernel issue, or a kexec issue, or a 
> mistake by myself. Maybe someone has a hint for me...
>
> If required, of course, I can provide more detailed information about 
> my hardware, kernel config, etc. (since I'm not sure if this is the 
> correct place, I wanted to keep this message short for now).
>

Hello Christian,

Are you not loading the driver for your disk drive controller when the 
new kernel boots up, even though you are not using the driver for disk 
I/O? If yes, the driver should have re-enabled Bus Master bit in its 
init routine. If you are loading the driver, which driver is it? I can 
take a look at it and see if there is anything missing in the 
initialization routine. Can you also include output from "lspci -v" from 
your machine?

Thanks
-- 
Khalid

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: Problem with kexec on i386, linux-3.5

[Christian Schaubschläger]

[-- Attachment #1: Type: text/plain, Size: 797 bytes --]


Hello Khalid,

> Are you not loading the driver for your disk drive controller when the 
> new kernel boots up, even though you are not using the driver for disk 
> I/O? If yes, the driver should have re-enabled Bus Master bit in its 
> init routine. If you are loading the driver, which driver is it? I can 
> take a look at it and see if there is anything missing in the 
> initialization routine. Can you also include output from "lspci -v" from 
> your machine?

I'm not loading any drivers in my new kernel, I'm just doing pure BIOS disk I/O using Int13 calls (the program I start using kexec I would not actually call a 'kernel', it's just a very tiny piece of software which does some disk I/O and output on the screen).

Attached you find the output of lspci -v

Best regards
Christian



[-- Attachment #2: lspic_d520.txt --]
[-- Type: text/plain, Size: 7559 bytes --]

00:00.0 Host bridge: Intel Corporation Mobile 945GM/PM/GMS, 943/940GML and 945GT Express Memory Controller Hub (rev 03)
	Subsystem: Dell Device 01d4
	Flags: bus master, fast devsel, latency 0
	Capabilities: [e0] Vendor Specific Information: Len=09 <?>

00:02.0 VGA compatible controller: Intel Corporation Mobile 945GM/GMS, 943/940GML Express Integrated Graphics Controller (rev 03) (prog-if 00 [VGA controller])
	Subsystem: Dell Device 01d4
	Flags: bus master, fast devsel, latency 0, IRQ 11
	Memory at eff00000 (32-bit, non-prefetchable) [size=512K]
	I/O ports at eff8 [size=8]
	Memory at d0000000 (32-bit, prefetchable) [size=256M]
	Memory at efec0000 (32-bit, non-prefetchable) [size=256K]
	Expansion ROM at <unassigned> [disabled]
	Capabilities: [90] MSI: Enable- Count=1/1 Maskable- 64bit-
	Capabilities: [d0] Power Management version 2

00:02.1 Display controller: Intel Corporation Mobile 945GM/GMS/GME, 943/940GML Express Integrated Graphics Controller (rev 03)
	Subsystem: Dell Device 01d4
	Flags: bus master, fast devsel, latency 0
	Memory at eff80000 (32-bit, non-prefetchable) [size=512K]
	Capabilities: [d0] Power Management version 2

00:1b.0 Audio device: Intel Corporation 82801G (ICH7 Family) High Definition Audio Controller (rev 01)
	Subsystem: Dell Device 01d4
	Flags: bus master, fast devsel, latency 0, IRQ 10
	Memory at efebc000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [50] Power Management version 2
	Capabilities: [60] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [70] Express Root Complex Integrated Endpoint, MSI 00
	Capabilities: [100] Virtual Channel
	Capabilities: [130] Root Complex Link

00:1c.0 PCI bridge: Intel Corporation 82801G (ICH7 Family) PCI Express Port 1 (rev 01) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0
	Bus: primary=00, secondary=0b, subordinate=0b, sec-latency=0
	I/O behind bridge: 00002000-00002fff
	Memory behind bridge: c4000000-c41fffff
	Prefetchable memory behind bridge: 00000000c4200000-00000000c43fffff
	Capabilities: [40] Express Root Port (Slot+), MSI 00
	Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
	Capabilities: [90] Subsystem: Dell Device 01d4
	Capabilities: [a0] Power Management version 2
	Capabilities: [100] Virtual Channel
	Capabilities: [180] Root Complex Link
	Kernel driver in use: pcieport

00:1c.1 PCI bridge: Intel Corporation 82801G (ICH7 Family) PCI Express Port 2 (rev 01) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0
	Bus: primary=00, secondary=0c, subordinate=0c, sec-latency=0
	I/O behind bridge: 00003000-00003fff
	Memory behind bridge: efd00000-efdfffff
	Prefetchable memory behind bridge: 00000000c4400000-00000000c45fffff
	Capabilities: [40] Express Root Port (Slot+), MSI 00
	Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
	Capabilities: [90] Subsystem: Dell Device 01d4
	Capabilities: [a0] Power Management version 2
	Capabilities: [100] Virtual Channel
	Capabilities: [180] Root Complex Link
	Kernel driver in use: pcieport

00:1d.0 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI Controller #1 (rev 01) (prog-if 00 [UHCI])
	Subsystem: Dell Device 01d4
	Flags: bus master, medium devsel, latency 0, IRQ 9
	I/O ports at bf80 [size=32]
	Kernel driver in use: uhci_hcd
	Kernel modules: uhci-hcd

00:1d.1 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI Controller #2 (rev 01) (prog-if 00 [UHCI])
	Subsystem: Dell Device 01d4
	Flags: bus master, medium devsel, latency 0, IRQ 10
	I/O ports at bf60 [size=32]
	Kernel driver in use: uhci_hcd
	Kernel modules: uhci-hcd

00:1d.2 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI Controller #3 (rev 01) (prog-if 00 [UHCI])
	Subsystem: Dell Device 01d4
	Flags: bus master, medium devsel, latency 0, IRQ 5
	I/O ports at bf40 [size=32]
	Kernel driver in use: uhci_hcd
	Kernel modules: uhci-hcd

00:1d.3 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI Controller #4 (rev 01) (prog-if 00 [UHCI])
	Subsystem: Dell Device 01d4
	Flags: bus master, medium devsel, latency 0, IRQ 3
	I/O ports at bf20 [size=32]
	Kernel driver in use: uhci_hcd
	Kernel modules: uhci-hcd

00:1d.7 USB Controller: Intel Corporation 82801G (ICH7 Family) USB2 EHCI Controller (rev 01) (prog-if 20 [EHCI])
	Subsystem: Dell Device 01d4
	Flags: bus master, medium devsel, latency 0, IRQ 9
	Memory at ffa80000 (32-bit, non-prefetchable) [size=1K]
	Capabilities: [50] Power Management version 2
	Capabilities: [58] Debug port: BAR=1 offset=00a0
	Kernel driver in use: ehci_hcd
	Kernel modules: ehci-hcd

00:1e.0 PCI bridge: Intel Corporation 82801 Mobile PCI Bridge (rev e1) (prog-if 01 [Subtractive decode])
	Flags: bus master, fast devsel, latency 0
	Bus: primary=00, secondary=02, subordinate=06, sec-latency=32
	I/O behind bridge: 00004000-00004fff
	Memory behind bridge: efc00000-efcfffff
	Prefetchable memory behind bridge: 00000000c0000000-00000000c3ffffff
	Capabilities: [50] Subsystem: Dell Device 01d4

00:1f.0 ISA bridge: Intel Corporation 82801GBM (ICH7-M) LPC Interface Bridge (rev 01)
	Subsystem: Dell Device 01d4
	Flags: bus master, medium devsel, latency 0
	Capabilities: [e0] Vendor Specific Information: Len=0c <?>

00:1f.2 IDE interface: Intel Corporation 82801GBM/GHM (ICH7 Family) SATA IDE Controller (rev 01) (prog-if 80 [Master])
	Subsystem: Dell Device 01d4
	Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 7
	I/O ports at 01f0 [size=8]
	I/O ports at 03f4 [size=1]
	I/O ports at 0170 [size=8]
	I/O ports at 0374 [size=1]
	I/O ports at bfa0 [size=16]
	Capabilities: [70] Power Management version 2
	Kernel driver in use: ata_piix

00:1f.3 SMBus: Intel Corporation 82801G (ICH7 Family) SMBus Controller (rev 01)
	Subsystem: Dell Device 01d4
	Flags: medium devsel, IRQ 11
	I/O ports at 10c0 [size=32]

02:00.0 Ethernet controller: Broadcom Corporation BCM4401-B0 100Base-TX (rev 02)
	Subsystem: Dell Device 01d4
	Flags: bus master, fast devsel, latency 64, IRQ 7
	Memory at efcfe000 (32-bit, non-prefetchable) [size=8K]
	Capabilities: [40] Power Management version 2
	Kernel driver in use: b44
	Kernel modules: b44

02:01.0 CardBus bridge: O2 Micro, Inc. Cardbus bridge (rev 21)
	Subsystem: Dell Device 01d4
	Flags: bus master, stepping, slow devsel, latency 168, IRQ 5
	Memory at c8000000 (32-bit, non-prefetchable) [size=4K]
	Bus: primary=02, secondary=03, subordinate=06, sec-latency=176
	Memory window 0: c0000000-c3fff000 (prefetchable)
	Memory window 1: cc000000-cffff000
	I/O window 0: 00004000-000040ff
	I/O window 1: 00004400-000044ff
	16-bit legacy interface ports at 0001
	Kernel driver in use: yenta_cardbus
	Kernel modules: yenta_socket

02:01.4 FireWire (IEEE 1394): O2 Micro, Inc. Firewire (IEEE 1394) (rev 02) (prog-if 10 [OHCI])
	Subsystem: Dell Device 01d4
	Flags: bus master, medium devsel, latency 64, IRQ 5
	Memory at efcfd000 (32-bit, non-prefetchable) [size=4K]
	Memory at efcfc800 (32-bit, non-prefetchable) [size=2K]
	Capabilities: [60] Power Management version 2

0c:00.0 Network controller: Intel Corporation PRO/Wireless 3945ABG [Golan] Network Connection (rev 02)
	Subsystem: Intel Corporation Device 1021
	Flags: bus master, fast devsel, latency 0, IRQ 7
	Memory at efdff000 (32-bit, non-prefetchable) [size=4K]
	Capabilities: [c8] Power Management version 2
	Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [e0] Express Legacy Endpoint, MSI 00
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [140] Device Serial Number 00-18-de-ff-ff-94-bc-2f
	Kernel driver in use: iwl3945
	Kernel modules: iwl3945


[-- Attachment #3: Type: text/plain, Size: 143 bytes --]

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: Problem with kexec on i386, linux-3.5

[Khalid Aziz]

On Mon, 2012-09-10 at 08:00 +0200, Christian Schaubschläger wrote:
> Hello Khalid,
> 
> > Are you not loading the driver for your disk drive controller when the 
> > new kernel boots up, even though you are not using the driver for disk 
> > I/O? If yes, the driver should have re-enabled Bus Master bit in its 
> > init routine. If you are loading the driver, which driver is it? I can 
> > take a look at it and see if there is anything missing in the 
> > initialization routine. Can you also include output from "lspci -v" from 
> > your machine?
> 
> I'm not loading any drivers in my new kernel, I'm just doing pure BIOS disk I/O using Int13 calls (the program I start using kexec I would not actually call a 'kernel', it's just a very tiny piece of software which does some disk I/O and output on the screen).
> 
> Attached you find the output of lspci -v
> 
> Best regards
> Christian
> 
> 

Hello Christian,

You have a rather esoteric use case. The patch that clears Bus Master
bit relies upon drivers reinitializing the controllers which includes
setting the Bus Master bit as well. Can you access PCI config space in
your program that you kexec? If yes, can you set the Bus Master bit in
your program? You have a pretty standard IDE controller there which does
have Bus Master capability.

Thanks
-- 
Khalid Aziz <khalid.aziz@hp.com>


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec