Re: X86_64 + VIA + 4g problems

Re: X86_64 + VIA + 4g problems

[Robert Hancock]

Andi Kleen wrote:
> When you not compile in the SKGE network driver does everything else work?
> skge supports 64bit DMA, so it shouldn't use any IOMMU.

Are you sure this is skge? Anyway, even if the driver does support 
64-bit DMA, I would be surprised if a desktop motherboard had the 
Ethernet chip connected via a 64-bit PCI bus.

This brings up something I've been wondering. It's possible to run most 
64-bit capable PCI devices in a 32-bit slot (i.e. with the 64-bit part 
hanging out of the slot). In this configuration the device will not be 
able to use 64-bit DMA (unless it supports dual address cycle). However, 
who is supposed to detect this and know to not try to use DMA addresses 
above 4GB on that device? Is the driver supposed to know this and use a 
32-bit DMA mask, or is the PCI subsystem able to figure this out 
somehow? What if the driver has no way to detect this? I don't think 
I've seen any code in the kernel that would figure out if the current 
configuration of the card is actually capable of 64-bit DMA or not.

-- 
Robert Hancock      Saskatoon, SK, Canada
To email, remove "nospam" from hancockr@nospamshaw.ca
Home Page: http://www.roberthancock.com/

Re: X86_64 + VIA + 4g problems

[Andi Kleen]

On Monday 02 January 2006 20:18, Robert Hancock wrote:
> Andi Kleen wrote:
> > When you not compile in the SKGE network driver does everything else work?
> > skge supports 64bit DMA, so it shouldn't use any IOMMU.
> 
> Are you sure this is skge? 

I grepped for the output in that badly truncated logfile and skge hit.

> Anyway, even if the driver does support  
> 64-bit DMA, I would be surprised if a desktop motherboard had the 
> Ethernet chip connected via a 64-bit PCI bus.

I meant 64bit DMA, aka double address cycle, not a 64bit wide PCI-X bus.
That works on a 32bit PCI bus too. On modern boards it's likely PCI
Express anyways for on board devices.

> 
> This brings up something I've been wondering. It's possible to run most 
> 64-bit capable PCI devices in a 32-bit slot (i.e. with the 64-bit part 
> hanging out of the slot). In this configuration the device will not be 
> able to use 64-bit DMA (unless it supports dual address cycle). However, 
> who is supposed to detect this and know to not try to use DMA addresses 
> above 4GB on that device? 

64bit PCI-X bus has nothing to do with the addressing capability. It
just gives you more bandwidth.

-Andi

Re: X86_64 + VIA + 4g problems

[Robert Hancock]

Andi Kleen wrote:
>>This brings up something I've been wondering. It's possible to run most 
>>64-bit capable PCI devices in a 32-bit slot (i.e. with the 64-bit part 
>>hanging out of the slot). In this configuration the device will not be 
>>able to use 64-bit DMA (unless it supports dual address cycle). However, 
>>who is supposed to detect this and know to not try to use DMA addresses 
>>above 4GB on that device? 
> 
> 
> 64bit PCI-X bus has nothing to do with the addressing capability. It
> just gives you more bandwidth.

Indeed, I misunderstood how this worked.. Accesses to addresses >4GB 
must always use DAC addressing, the address can just be decoded faster 
if the extra 64-bit data lines are connected.

I think there could still be an issue if the PCI bus does not support 
DAC and the device tries to use PCI DAC addressing causing the bus to 
choke or the wrong RAM to get accessed. Apparently this is a real 
problem according to MS:

http://www.microsoft.com/whdc/system/platform/server/PAE/PAEdrv.mspx

"Unfortunately, Microsoft is finding that not all PCI buses on a system 
board support DAC, which is required for a 32-bit PCI adapter to address 
more than 4 GB of memory. Furthermore, there is no way for a DAC-capable 
PCI device (or its associated driver) to know that it is running on a 
non-DAC-capable bus."

Microsoft's solution is to disable all memory above 4GB if 
non-DAC-capable buses are present - largely due to their API limitations 
and the number of broken drivers that don't call proper DMA functions. 
Haha to them, I guess..

I'm not sure if this would be an issue with Linux. I would suspect that 
it could be, if there are indeed such buses that can't support DAC.. In 
this case the right thing to do would be to reject requests for DMA 
masks larger than 4GB for devices located on such a bus.

-- 
Robert Hancock      Saskatoon, SK, Canada
To email, remove "nospam" from hancockr@nospamshaw.ca
Home Page: http://www.roberthancock.com/

Re: X86_64 + VIA + 4g problems

[Andi Kleen]

On Monday 02 January 2006 21:15, Robert Hancock wrote:

> I'm not sure if this would be an issue with Linux. I would suspect that 
> it could be, if there are indeed such buses that can't support DAC.. In 
> this case the right thing to do would be to reject requests for DMA 
> masks larger than 4GB for devices located on such a bus.

Well, from the original messages in this thread which you so shamelessly
hijacked it is possible VIA PCI bridges might have this problem. If yes
it can be handled in the PCI-DMA API by disallowing DAC for devices
behind such bridges. But before taking such drastic action we need
first need more testing results.

-Andi

Re: X86_64 + VIA + 4g problems

[Robert Hancock]

Carsten Otto wrote:
> On Mon, Jan 02, 2006 at 08:01:08PM +0100, Andi Kleen wrote:
> 
>>Alternatively you can try with the appended patch. If that helps
>>then the chipset or the BIOS likely has some fundamental issue with >4GB.
> 
> 
> Based on the first few minutes of testing this works perfectly! Thanks a
> lot!
> 
> PS: Does it make sense to contact my shop and/or Abit and/or VIA
> regarding this problem? I don't want to pay money for broken hardware.

I suspect VIA may be the best place to complain to, it is most likely 
their breakage. I don't know how responsive they are to such things however.

-- 
Robert Hancock      Saskatoon, SK, Canada
To email, remove "nospam" from hancockr@nospamshaw.ca
Home Page: http://www.roberthancock.com/

X86_64 + VIA + 4g problems

[Dieter Stüken]

just gave 2.6.15-rc7 a try, but still fail when plugging 4g into the board :-(
Its an Asus sk8v (VIA chipset), thus I get:

  Linux version 2.6.15-rc7 (root@linux2) (gcc version 3.3.5 20050117 (prerelease) (SUSE Linux)) #9 Sun Jan 1 19:54:26 CET 2006
BIOS-provided physical RAM map:
  BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
  BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
  BIOS-e820: 00000000000e4000 - 0000000000100000 (reserved)
  BIOS-e820: 0000000000100000 - 00000000bff30000 (usable)
  BIOS-e820: 00000000bff30000 - 00000000bff40000 (ACPI data)
  BIOS-e820: 00000000bff40000 - 00000000bfff0000 (ACPI NVS)
  BIOS-e820: 00000000bfff0000 - 00000000c0000000 (reserved)
  BIOS-e820: 00000000fff80000 - 0000000100000000 (reserved)
  BIOS-e820: 0000000100000000 - 0000000140000000 (usable)
ACPI: RSDP (v002 ACPIAM                                ) @ 0x00000000000fa850
ACPI: XSDT (v001 A M I  OEMXSDT  0x06000517 MSFT 0x00000097) @ 0x00000000bff30100
ACPI: FADT (v003 A M I  OEMFACP  0x06000517 MSFT 0x00000097) @ 0x00000000bff30290
ACPI: MADT (v001 A M I  OEMAPIC  0x06000517 MSFT 0x00000097) @ 0x00000000bff30390
ACPI: OEMB (v001 A M I  OEMBIOS  0x06000517 MSFT 0x00000097) @ 0x00000000bff40040
ACPI: DSDT (v001  SK8V_ SK8V_033 0x00000033 MSFT 0x0100000d) @ 0x0000000000000000
On node 0 totalpages: 1029568
   DMA zone: 3160 pages, LIFO batch:0
   DMA32 zone: 767848 pages, LIFO batch:31
   Normal zone: 258560 pages, LIFO batch:31
   HighMem zone: 0 pages, LIFO batch:0
Looks like a VIA chipset. Disabling IOMMU. Overwrite with "iommu=allowed"
.....
GSI 19 sharing vector 0xC1 and IRQ 19
ACPI: PCI Interrupt 0000:00:0a.0[A] -> GSI 17 (level, low) -> IRQ 19
ACPI: PCI Interrupt 0000:00:0a.0[A] -> GSI 17 (level, low) -> IRQ 19
eth0: 3Com Gigabit LOM (3C940)
       PrefPort:A  RlmtMode:Check Link State

don't know, if it's related to that, but with 2G it runs stable since about a year.

The problem arises as soon as my network (3C940) gets enabled, the following
message is continuously repeated and nothing else works any more, not even
console switching.

eth0: -- ERROR --
Class: Hardware failure
Nr: 0x264
Msg: unexpected IRQ Status error
eth0: Adapter failed.

Any suggestion? Try different boot options acpi= pci= iommu= ... ?

Dieter.
-- 
Dieter Stüken, con terra GmbH, Münster
     stueken@conterra.de
     http://www.conterra.de/
     (0)251-7474-501

Re: X86_64 + VIA + 4g problems

[Carsten Otto]

I did not read (and find) your original posting, but it appears to me
that your problem is similar to mine. I upgraded to 4GB RAM on X86_64
and after that my Intel Gigabit card stopped working.
http://lkml.org/lkml/2005/12/19/144

Perhaps the higher class Gigabit cards need some memory area that is
(once) overwritten with 4GB main memory? I have no clue, actually :)

> eth0: 3Com Gigabit LOM (3C940)
>       PrefPort:A  RlmtMode:Check Link State

Did you try the ethtool tests? They show more information, at least with
my card.

> don't know, if it's related to that, but with 2G it runs stable since about 
> a year.

Does the problem still occur with 2GB (less)? Did you try running
Knoppix? The latter solves the problem here, but obviously Knoppix is no
final solution.

> The problem arises as soon as my network (3C940) gets enabled, the following
> message is continuously repeated and nothing else works any more, not even
> console switching.

Wow.
-- 
Carsten Otto
c-otto@gmx.de
www.c-otto.de

Re: X86_64 + VIA + 4g problems

[Andi Kleen]

Dieter Stüken <stueken@conterra.de> writes:

> just gave 2.6.15-rc7 a try, but still fail when plugging 4g into the board :-(
> Its an Asus sk8v (VIA chipset), thus I get:

Can you please post the full boot log? 

> ACPI: PCI Interrupt 0000:00:0a.0[A] -> GSI 17 (level, low) -> IRQ 19
> ACPI: PCI Interrupt 0000:00:0a.0[A] -> GSI 17 (level, low) -> IRQ 19
> eth0: 3Com Gigabit LOM (3C940)
>        PrefPort:A  RlmtMode:Check Link State
> 
> don't know, if it's related to that, but with 2G it runs stable since about a year.
> 
> The problem arises as soon as my network (3C940) gets enabled, the following
> message is continuously repeated and nothing else works any more, not even
> console switching.

When you not compile in the SKGE network driver does everything else work?
skge supports 64bit DMA, so it shouldn't use any IOMMU.  But I'm somewhat
suspicious of the >4GB support in the VIA chipset. We had problems with
that before. It's possible that it's just not supported in the hardware
or that the BIOS sets up the MTRRs wrong.

-Andi

Re: X86_64 + VIA + 4g problems

[Carsten Otto]

[-- Attachment #1: Type: text/plain, Size: 632 bytes --]

On Mon, Jan 02, 2006 at 05:22:12PM +0100, Andi Kleen wrote:
> When you not compile in the SKGE network driver does everything else work?
> skge supports 64bit DMA, so it shouldn't use any IOMMU.  But I'm somewhat
> suspicious of the >4GB support in the VIA chipset. We had problems with
> that before. It's possible that it's just not supported in the hardware
> or that the BIOS sets up the MTRRs wrong.

How can I find that out? Can I provide any useful material? VIA KT800Pro
and 4GB RAM here..

PS: I am not the original poster, but have a similar problem.

Thanks,
-- 
Carsten Otto
c-otto@gmx.de
www.c-otto.de

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: X86_64 + VIA + 4g problems

[Carsten Otto]

[-- Attachment #1: Type: text/plain, Size: 240 bytes --]

On Mon, Jan 02, 2006 at 05:52:31PM +0100, Carsten Otto wrote:
> How can I find that out? Can I provide any useful material? VIA KT800Pro
> and 4GB RAM here..

VIA K8T800 Pro is correct.
-- 
Carsten Otto
c-otto@gmx.de
www.c-otto.de

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: X86_64 + VIA + 4g problems

[Andi Kleen]

Carsten Otto <c-otto@gmx.de> writes:

[please don't strip cc lists like this - your email got nearly lost]

> On Mon, Jan 02, 2006 at 05:22:12PM +0100, Andi Kleen wrote:
> > When you not compile in the SKGE network driver does everything else work?
> > skge supports 64bit DMA, so it shouldn't use any IOMMU.  But I'm somewhat
> > suspicious of the >4GB support in the VIA chipset. We had problems with
> > that before. It's possible that it's just not supported in the hardware
> > or that the BIOS sets up the MTRRs wrong.
> 
> How can I find that out? Can I provide any useful material? VIA KT800Pro
> and 4GB RAM here..

What was unclear with my first question? Can you try that?
To repeat it again:

%% When you not compile in the SKGE network driver does everything else work?
%% skge supports 64bit DMA, so it shouldn't use any IOMMU. 

Alternatively you can try with the appended patch. If that helps
then the chipset or the BIOS likely has some fundamental issue with >4GB.

-Andi

Index: linux-2.6.15rc7-work/arch/x86_64/kernel/pci-gart.c
===================================================================
--- linux-2.6.15rc7-work.orig/arch/x86_64/kernel/pci-gart.c
+++ linux-2.6.15rc7-work/arch/x86_64/kernel/pci-gart.c
@@ -630,6 +630,13 @@ void dma_unmap_sg(struct device *dev, st
 
 int dma_supported(struct device *dev, u64 mask)
 {
+#if 1
+	/* test patch for VIA quirk */	
+	if (mask > 0xffffffff)
+		return 0;
+#endif
+
+
 	/* Copied from i386. Doesn't make much sense, because it will 
 	   only work for pci_alloc_coherent.
 	   The caller just has to use GFP_DMA in this case. */

Re: X86_64 + VIA + 4g problems

[Carsten Otto]

[-- Attachment #1: Type: text/plain, Size: 477 bytes --]

On Mon, Jan 02, 2006 at 08:01:08PM +0100, Andi Kleen wrote:
> Alternatively you can try with the appended patch. If that helps
> then the chipset or the BIOS likely has some fundamental issue with >4GB.

Based on the first few minutes of testing this works perfectly! Thanks a
lot!

PS: Does it make sense to contact my shop and/or Abit and/or VIA
regarding this problem? I don't want to pay money for broken hardware.
-- 
Carsten Otto
c-otto@gmx.de
www.c-otto.de

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: X86_64 + VIA + 4g problems

[Dieter Stüken]

Andi Kleen wrote:
> Can you please post the full boot log? 
> ...
> When you not compile in the SKGE network driver does everything else work?
> skge supports 64bit DMA, so it shouldn't use any IOMMU.  But I'm somewhat
> suspicious of the >4GB support in the VIA chipset. We had problems with
> that before. It's possible that it's just not supported in the hardware
> or that the BIOS sets up the MTRRs wrong.

sorry for the delay, we are a few hors off here...

here comes a more complete dmsg. Its a bit long, as its a large LVM system
with 7 SATA disks attached. There are lots of messages about USB stuff,
but I actually don't use any USB. But indeed I dicovered some mtrr message
missed in the snipplet before.

Unfortunately this is a production server, so I can't reboot and play with
any time. But may be this night, in about 12h, I can perform further tests.

Let me know, what kind of tests I may perform with all the SATA devices.
I have a PDC-20378 and a VIA-VT6420 on board and a PDC-20318 on a PCI slot.
Do those have any problems with 64-bit? For testing I would prefer to use
them R/only. Would a parallel dummy tar to /dev/zero be meaningful?.

Dieter.
-----------
Bootdata ok (command line is root=/dev/sda1 log_buf_len=32k video=matroxfb:vesa:0x1B8 s)
Linux version 2.6.15-rc7 (root@Rescue) (gcc version 3.3.5 20050117 (prerelease) (SUSE Linux)) #9 Sun Jan 1 19:54:26 CET 2006
BIOS-provided physical RAM map:
  BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
  BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
  BIOS-e820: 00000000000e4000 - 0000000000100000 (reserved)
  BIOS-e820: 0000000000100000 - 00000000bff30000 (usable)
  BIOS-e820: 00000000bff30000 - 00000000bff40000 (ACPI data)
  BIOS-e820: 00000000bff40000 - 00000000bfff0000 (ACPI NVS)
  BIOS-e820: 00000000bfff0000 - 00000000c0000000 (reserved)
  BIOS-e820: 00000000fff80000 - 0000000100000000 (reserved)
  BIOS-e820: 0000000100000000 - 0000000140000000 (usable)
ACPI: RSDP (v002 ACPIAM                                ) @ 0x00000000000fa850
ACPI: XSDT (v001 A M I  OEMXSDT  0x06000517 MSFT 0x00000097) @ 0x00000000bff30100
ACPI: FADT (v003 A M I  OEMFACP  0x06000517 MSFT 0x00000097) @ 0x00000000bff30290
ACPI: MADT (v001 A M I  OEMAPIC  0x06000517 MSFT 0x00000097) @ 0x00000000bff30390
ACPI: OEMB (v001 A M I  OEMBIOS  0x06000517 MSFT 0x00000097) @ 0x00000000bff40040
ACPI: DSDT (v001  SK8V_ SK8V_033 0x00000033 MSFT 0x0100000d) @ 0x0000000000000000
On node 0 totalpages: 1029568
   DMA zone: 3160 pages, LIFO batch:0
   DMA32 zone: 767848 pages, LIFO batch:31
   Normal zone: 258560 pages, LIFO batch:31
   HighMem zone: 0 pages, LIFO batch:0
Looks like a VIA chipset. Disabling IOMMU. Overwrite with "iommu=allowed"
ACPI: PM-Timer IO Port: 0x808
ACPI: Local APIC address 0xfee00000
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
Processor #0 15:5 APIC version 16
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x81] disabled)
ACPI: IOAPIC (id[0x01] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 1, version 3, address 0xfec00000, GSI 0-23
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
ACPI: IRQ0 used by override.
ACPI: IRQ2 used by override.
ACPI: IRQ9 used by override.
Setting APIC routing to flat
Using ACPI (MADT) for SMP configuration information
Allocating PCI resources starting at c4000000 (gap: c0000000:3ff80000)
Built 1 zonelists
Kernel command line: root=/dev/sda1 log_buf_len=32k video=matroxfb:vesa:0x1B8 s
Initializing CPU#0
PID hash table entries: 4096 (order: 12, 131072 bytes)
time.c: Using 3.579545 MHz PM timer.
time.c: Detected 2002.656 MHz processor.
time.c: Using PIT/TSC based timekeeping.
Console: colour VGA+ 80x25
Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes)
Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes)
Placing software IO TLB between 0x5c39000 - 0x9c39000
Memory: 4043548k/5242880k available (1809k kernel code, 148908k reserved, 770k data, 160k init)
Calibrating delay using timer specific routine.. 4014.56 BogoMIPS (lpj=8029129)
Mount-cache hash table entries: 256
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 1024K (64 bytes/line)
mtrr: v2.0 (20020519)
CPU: AMD Opteron(tm) Processor 146 stepping 0a
  tbxface-0109 [02] load_tables           : ACPI Tables successfully acquired
Parsing all Control 
Methods:...................................................................................................................................................
Table [DSDT](id 0005) - 538 Objects with 45 Devices 147 Methods 25 Regions
ACPI Namespace successfully loaded at root ffffffff803b8dc0
evxfevnt-0091 [03] enable                : Transition to ACPI mode successful
Using local APIC timer interrupts.
Detected 12.516 MHz APIC timer.
testing NMI watchdog ... OK.
checking if image is initramfs... it is
NET: Registered protocol family 16
ACPI: bus type pci registered
PCI: Using configuration type 1
ACPI: Subsystem revision 20050902
evgpeblk-0988 [06] ev_create_gpe_block   : GPE 00 to 0F [_GPE] 2 regs on int 0x9
evgpeblk-0996 [06] ev_create_gpe_block   : Found 7 Wake, Enabled 0 Runtime GPEs in this block
Completing Region/Field/Buffer/Package 
initialization:..................................................................................................................................
Initialized 24/25 Regions 44/44 Fields 47/47 Buffers 15/16 Packages (547 nodes)
Executing all Device _STA and_INI methods:.................................................
49 Devices found containing: 49 _STA, 0 _INI methods
ACPI: Interpreter enabled
ACPI: Using IOAPIC for interrupt routing
ACPI: PCI Root Bridge [PCI0] (0000:00)
PCI: Probing PCI hardware (bus 00)
Boot video device is 0000:01:00.0
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 7 10 *11 14 15)
ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 7 *10 11 14 15)
ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 *5 7 10 11 14 15)
ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 7 10 11 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 7 10 11 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 7 10 11 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 7 10 11 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 7 10 11 14 15) *0, disabled.
PCI: Using ACPI for IRQ routing
PCI: If a device doesn't work, try "pci=routeirq".  If it helps, post a report
agpgart: Detected AGP bridge 0
agpgart: AGP aperture is 64M @ 0xf4000000
PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
PCI: Bridge: 0000:00:01.0
   IO window: disabled.
   MEM window: fbb00000-fcefffff
   PREFETCH window: efa00000-f39fffff
acpi_bus-0201 [01] bus_set_power         : Device is not power manageable
PCI: Setting latency timer of device 0000:00:01.0 to 64
IA32 emulation $Id: sys_ia32.c,v 1.32 2002/03/24 13:02:28 ak Exp $
audit: initializing netlink socket (disabled)
audit(1136141830.556:1): initialized
Total HugeTLB memory allocated, 0
Initializing Cryptographic API
io scheduler noop registered
  0000:00:10.0: uhci_check_and_reset_hc: legsup = 0x8030
  0000:00:10.0: Performing full reset
  0000:00:10.1: uhci_check_and_reset_hc: legsup = 0x8030
  0000:00:10.1: Performing full reset
  0000:00:10.2: uhci_check_and_reset_hc: legsup = 0x8030
  0000:00:10.2: Performing full reset
  0000:00:10.3: uhci_check_and_reset_hc: legsup = 0x8030
  0000:00:10.3: Performing full reset
GSI 16 sharing vector 0xA9 and IRQ 16
ACPI: PCI Interrupt 0000:01:00.0[A] -> GSI 16 (level, low) -> IRQ 16
matroxfb: Matrox G400 (AGP) detected
PInS memtype = 0
mtrr: type mismatch for f0000000,1000000 old: write-back new: write-combining
matroxfb: MTRR's turned on
matroxfb: 1024x768x24bpp (virtual: 1024x5461)
matroxfb: framebuffer at 0xF0000000, mapped to 0xffffc20000080000, size 16777216
Console: switching to colour frame buffer device 128x48
fb0: MATROX frame buffer device
matroxfb_crtc2: secondary head of fb0 was registered as fb1
Real Time Clock Driver v1.12
Linux agpgart interface v0.101 (c) Dave Jones
serio: i8042 AUX port at 0x60,0x64 irq 12
serio: i8042 KBD port at 0x60,0x64 irq 1
RAMDISK driver initialized: 16 RAM disks of 64000K size 1024 blocksize
Intel(R) PRO/1000 Network Driver - version 6.1.16-k2-NAPI
Copyright (c) 1999-2005 Intel Corporation.
e100: Intel(R) PRO/100 Network Driver, 3.4.14-k4-NAPI
e100: Copyright(c) 1999-2005 Intel Corporation
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
VP_IDE: IDE controller at PCI slot 0000:00:0f.1
acpi_bus-0201 [01] bus_set_power         : Device is not power manageable
GSI 17 sharing vector 0xB1 and IRQ 17
ACPI: PCI Interrupt 0000:00:0f.1[A] -> GSI 20 (level, low) -> IRQ 17
PCI: Via IRQ fixup for 0000:00:0f.1, from 255 to 1
VP_IDE: chipset revision 6
VP_IDE: not 100% native mode: will probe irqs later
VP_IDE: VIA vt8237 (rev 00) IDE UDMA133 controller on pci0000:00:0f.1
     ide0: BM-DMA at 0xfc00-0xfc07, BIOS settings: hda:pio, hdb:pio
     ide1: BM-DMA at 0xfc08-0xfc0f, BIOS settings: hdc:pio, hdd:DMA
Probing IDE interface ide0...
Probing IDE interface ide1...
hdd: HL-DT-STDVD-ROM GDR8163B, ATAPI CD/DVD-ROM drive
ide1 at 0x170-0x177,0x376 on irq 15
mice: PS/2 mouse device common for all mice
NET: Registered protocol family 2
IP route cache hash table entries: 131072 (order: 8, 1048576 bytes)
TCP established hash table entries: 524288 (order: 10, 4194304 bytes)
TCP bind hash table entries: 65536 (order: 7, 524288 bytes)
TCP: Hash tables configured (established 524288 bind 65536)
TCP reno registered
input: AT Translated Set 2 keyboard as /class/input/input0
ACPI wakeup devices:
PCI0 PS2K PS2M UAR2 UAR1 AC97 USB1 USB2 USB3 USB4 EHCI PWRB SLPB
ACPI: (supports S0 S1 S3 S4 S5)
Freeing unused kernel memory: 160k freed
SCSI subsystem initialized
libata version 1.20 loaded.
sata_promise 0000:00:08.0: version 1.03
GSI 18 sharing vector 0xB9 and IRQ 18
ACPI: PCI Interrupt 0000:00:08.0[A] -> GSI 18 (level, low) -> IRQ 18
ata1: SATA max UDMA/133 cmd 0xFFFFC20000004200 ctl 0xFFFFC20000004238 bmdma 0x0 irq 18
ata2: SATA max UDMA/133 cmd 0xFFFFC20000004280 ctl 0xFFFFC200000042B8 bmdma 0x0 irq 18
ata1: dev 0 cfg 49:2f00 82:7c6b 83:7f09 84:4003 85:7c69 86:3e01 87:4003 88:407f
ata1: dev 0 ATA-7, max UDMA/133, 490234752 sectors: LBA48
ata1: dev 0 configured for UDMA/133
scsi0 : sata_promise
ata2: dev 0 cfg 49:2f00 82:346b 83:7fe9 84:4773 85:3469 86:3c01 87:4763 88:407f
ata2: dev 0 ATA-7, max UDMA/133, 976773168 sectors: LBA48
ata2: dev 0 configured for UDMA/133
scsi1 : sata_promise
isa bounce pool size: 16 pages
   Vendor: ATA       Model: Maxtor 7Y250M0    Rev: YAR5
   Type:   Direct-Access                      ANSI SCSI revision: 05
SCSI device sda: 490234752 512-byte hdwr sectors (251000 MB)
SCSI device sda: drive cache: write back
SCSI device sda: 490234752 512-byte hdwr sectors (251000 MB)
SCSI device sda: drive cache: write back
  sda: sda1 sda2
sd 0:0:0:0: Attached scsi disk sda
   Vendor: ATA       Model: HDS725050KLA360   Rev: K2AO
   Type:   Direct-Access                      ANSI SCSI revision: 05
SCSI device sdb: 976773168 512-byte hdwr sectors (500108 MB)
SCSI device sdb: drive cache: write back
SCSI device sdb: 976773168 512-byte hdwr sectors (500108 MB)
SCSI device sdb: drive cache: write back
  sdb: sdb1
sd 1:0:0:0: Attached scsi disk sdb
ACPI: PCI Interrupt 0000:00:09.0[A] -> GSI 16 (level, low) -> IRQ 16
ata3: SATA max UDMA/133 cmd 0xFFFFC20000006200 ctl 0xFFFFC20000006238 bmdma 0x0 irq 16
ata4: SATA max UDMA/133 cmd 0xFFFFC20000006280 ctl 0xFFFFC200000062B8 bmdma 0x0 irq 16
ata5: SATA max UDMA/133 cmd 0xFFFFC20000006300 ctl 0xFFFFC20000006338 bmdma 0x0 irq 16
ata6: SATA max UDMA/133 cmd 0xFFFFC20000006380 ctl 0xFFFFC200000063B8 bmdma 0x0 irq 16
ata3: dev 0 cfg 49:2f00 82:74eb 83:7fea 84:4023 85:74e9 86:3c02 87:4023 88:203f
ata3: dev 0 ATA-6, max UDMA/100, 488397168 sectors: LBA48
ata3: dev 0 configured for UDMA/100
scsi2 : sata_promise
ata4: dev 0 cfg 49:2f00 82:74eb 83:7fea 84:4023 85:74e9 86:3c02 87:4023 88:203f
ata4: dev 0 ATA-6, max UDMA/100, 488397168 sectors: LBA48
ata4: dev 0 configured for UDMA/100
scsi3 : sata_promise
ata5: dev 0 cfg 49:2f00 82:74eb 83:7fea 84:4023 85:74e9 86:3e02 87:4023 88:203f
ata5: dev 0 ATA-6, max UDMA/100, 488397168 sectors: LBA48
ata5: dev 0 configured for UDMA/100
scsi4 : sata_promise
ata6: dev 0 cfg 49:2f00 82:74eb 83:7fea 84:4023 85:74e9 86:3e02 87:4023 88:203f
ata6: dev 0 ATA-6, max UDMA/100, 488397168 sectors: LBA48
ata6: dev 0 configured for UDMA/100
scsi5 : sata_promise
   Vendor: ATA       Model: HDS722525VLSA80   Rev: V36O
   Type:   Direct-Access                      ANSI SCSI revision: 05
SCSI device sdc: 488397168 512-byte hdwr sectors (250059 MB)
SCSI device sdc: drive cache: write back
SCSI device sdc: 488397168 512-byte hdwr sectors (250059 MB)
SCSI device sdc: drive cache: write back
  sdc: sdc1
sd 2:0:0:0: Attached scsi disk sdc
   Vendor: ATA       Model: HDS722525VLSA80   Rev: V36O
   Type:   Direct-Access                      ANSI SCSI revision: 05
SCSI device sdd: 488397168 512-byte hdwr sectors (250059 MB)
SCSI device sdd: drive cache: write back
SCSI device sdd: 488397168 512-byte hdwr sectors (250059 MB)
SCSI device sdd: drive cache: write back
  sdd: sdd1
sd 3:0:0:0: Attached scsi disk sdd
   Vendor: ATA       Model: HDS722525VLSA80   Rev: V36O
   Type:   Direct-Access                      ANSI SCSI revision: 05
SCSI device sde: 488397168 512-byte hdwr sectors (250059 MB)
SCSI device sde: drive cache: write back
SCSI device sde: 488397168 512-byte hdwr sectors (250059 MB)
SCSI device sde: drive cache: write back
  sde: sde1
sd 4:0:0:0: Attached scsi disk sde
   Vendor: ATA       Model: HDS722525VLSA80   Rev: V36O
   Type:   Direct-Access                      ANSI SCSI revision: 05
SCSI device sdf: 488397168 512-byte hdwr sectors (250059 MB)
SCSI device sdf: drive cache: write back
SCSI device sdf: 488397168 512-byte hdwr sectors (250059 MB)
SCSI device sdf: drive cache: write back
  sdf: sdf1
sd 5:0:0:0: Attached scsi disk sdf
sata_via 0000:00:0f.0: version 1.1
ACPI: PCI Interrupt 0000:00:0f.0[B] -> GSI 20 (level, low) -> IRQ 17
PCI: Via IRQ fixup for 0000:00:0f.0, from 10 to 1
sata_via 0000:00:0f.0: routed to hard irq line 1
ata7: SATA max UDMA/133 cmd 0xE800 ctl 0xE402 bmdma 0xD400 irq 17
ata8: SATA max UDMA/133 cmd 0xE000 ctl 0xD802 bmdma 0xD408 irq 17
ata7: dev 0 cfg 49:2f00 82:74eb 83:7feb 84:4123 85:74e9 86:3c03 87:4123 88:407f
ata7: dev 0 ATA-7, max UDMA/133, 781422768 sectors: LBA48
ata7: dev 0 configured for UDMA/133
scsi6 : sata_via
ata8: dev 0 cfg 49:2f00 82:74eb 83:7feb 84:4123 85:74e9 86:3c03 87:4123 88:407f
ata8: dev 0 ATA-7, max UDMA/133, 781422768 sectors: LBA48
ata8: dev 0 configured for UDMA/133
scsi7 : sata_via
   Vendor: ATA       Model: HDS724040KLSA80   Rev: KFAO
   Type:   Direct-Access                      ANSI SCSI revision: 05
SCSI device sdg: 781422768 512-byte hdwr sectors (400088 MB)
SCSI device sdg: drive cache: write back
SCSI device sdg: 781422768 512-byte hdwr sectors (400088 MB)
SCSI device sdg: drive cache: write back
  sdg: sdg1
sd 6:0:0:0: Attached scsi disk sdg
   Vendor: ATA       Model: HDS724040KLSA80   Rev: KFAO
   Type:   Direct-Access                      ANSI SCSI revision: 05
SCSI device sdh: 781422768 512-byte hdwr sectors (400088 MB)
SCSI device sdh: drive cache: write back
SCSI device sdh: 781422768 512-byte hdwr sectors (400088 MB)
SCSI device sdh: drive cache: write back
  sdh: sdh1
sd 7:0:0:0: Attached scsi disk sdh
GSI 19 sharing vector 0xC1 and IRQ 19
ACPI: PCI Interrupt 0000:00:0a.0[A] -> GSI 17 (level, low) -> IRQ 19
ACPI: PCI Interrupt 0000:00:0a.0[A] -> GSI 17 (level, low) -> IRQ 19
eth0: 3Com Gigabit LOM (3C940)
       PrefPort:A  RlmtMode:Check Link State
kjournald starting.  Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
usbcore: registered new driver usbfs
usbcore: registered new driver hub
ehci_hcd: block sizes: qh 160 qtd 96 itd 192 sitd 96
acpi_bus-0201 [01] bus_set_power         : Device is not power manageable
GSI 20 sharing vector 0xC9 and IRQ 20
ACPI: PCI Interrupt 0000:00:10.4[C] -> GSI 21 (level, low) -> IRQ 20
PCI: Via IRQ fixup for 0000:00:10.4, from 5 to 4
ehci_hcd 0000:00:10.4: EHCI Host Controller
ehci_hcd 0000:00:10.4: reset hcs_params 0x4208 dbg=0 cc=4 pcc=2 ordered !ppc ports=8
ehci_hcd 0000:00:10.4: reset hcc_params 6872 thresh 7 uframes 256/512/1024
ehci_hcd 0000:00:10.4: capability 0001 at 68
ehci_hcd 0000:00:10.4: MWI active
drivers/usb/core/inode.c: creating file 'devices'
drivers/usb/core/inode.c: creating file '001'
ehci_hcd 0000:00:10.4: new USB bus registered, assigned bus number 1
ehci_hcd 0000:00:10.4: irq 20, io mem 0xfdf00000
ehci_hcd 0000:00:10.4: reset command 080002 (park)=0 ithresh=8 period=1024 Reset HALT
ehci_hcd 0000:00:10.4: init command 010009 (park)=0 ithresh=1 period=256 RUN
ehci_hcd 0000:00:10.4: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
usb usb1: default language 0x0409
usb usb1: new device strings: Mfr=3, Product=2, SerialNumber=1
usb usb1: Product: EHCI Host Controller
usb usb1: Manufacturer: Linux 2.6.15-rc7 ehci_hcd
usb usb1: SerialNumber: 0000:00:10.4
usb usb1: hotplug
usb usb1: adding 1-0:1.0 (config #1, interface 0)
usb 1-0:1.0: hotplug
hub 1-0:1.0: usb_probe_interface
hub 1-0:1.0: usb_probe_interface - got id
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 8 ports detected
hub 1-0:1.0: standalone hub
hub 1-0:1.0: no power switching (usb 1.0)
hub 1-0:1.0: individual port over-current protection
hub 1-0:1.0: Single TT
hub 1-0:1.0: TT requires at most 8 FS bit times (666 ns)
hub 1-0:1.0: power on to power good time: 20ms
hub 1-0:1.0: local power source is good
hub 1-0:1.0: state 5 ports 8 chg 0000 evt 0000
drivers/usb/core/inode.c: creating file '001'
ohci_hcd: 2005 April 22 USB 1.1 'Open' Host Controller (OHCI) Driver (PCI)
ohci_hcd: block sizes: ed 80 td 96
USB Universal Host Controller Interface driver v2.3
acpi_bus-0201 [01] bus_set_power         : Device is not power manageable
ACPI: PCI Interrupt 0000:00:10.0[A] -> GSI 21 (level, low) -> IRQ 20
PCI: Via IRQ fixup for 0000:00:10.0, from 11 to 4
uhci_hcd 0000:00:10.0: UHCI Host Controller
uhci_hcd 0000:00:10.0: detected 2 ports
uhci_hcd 0000:00:10.0: uhci_check_and_reset_hc: cmd = 0x0000
uhci_hcd 0000:00:10.0: Performing full reset
drivers/usb/core/inode.c: creating file '002'
uhci_hcd 0000:00:10.0: new USB bus registered, assigned bus number 2
uhci_hcd 0000:00:10.0: irq 20, io base 0x0000b400
uhci_hcd 0000:00:10.0: supports USB remote wakeup
usb usb2: default language 0x0409
usb usb2: new device strings: Mfr=3, Product=2, SerialNumber=1
usb usb2: Product: UHCI Host Controller
usb usb2: Manufacturer: Linux 2.6.15-rc7 uhci_hcd
usb usb2: SerialNumber: 0000:00:10.0
usb usb2: hotplug
usb usb2: adding 2-0:1.0 (config #1, interface 0)
usb 2-0:1.0: hotplug
hub 2-0:1.0: usb_probe_interface
hub 2-0:1.0: usb_probe_interface - got id
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 2 ports detected
hub 2-0:1.0: standalone hub
hub 2-0:1.0: no power switching (usb 1.0)
hub 2-0:1.0: individual port over-current protection
hub 2-0:1.0: power on to power good time: 2ms
hub 2-0:1.0: local power source is good
hub 2-0:1.0: state 5 ports 2 chg 0000 evt 0000
drivers/usb/core/inode.c: creating file '001'
uhci_hcd 0000:00:10.0: port 1 portsc 018a,00
hub 2-0:1.0: port 1, status 0300, change 0003, 1.5 Mb/s
acpi_bus-0201 [01] bus_set_power         : Device is not power manageable
ACPI: PCI Interrupt 0000:00:10.1[A] -> GSI 21 (level, low) -> IRQ 20
PCI: Via IRQ fixup for 0000:00:10.1, from 11 to 4
uhci_hcd 0000:00:10.1: UHCI Host Controller
uhci_hcd 0000:00:10.1: detected 2 ports
uhci_hcd 0000:00:10.1: uhci_check_and_reset_hc: cmd = 0x0000
uhci_hcd 0000:00:10.1: Performing full reset
drivers/usb/core/inode.c: creating file '003'
uhci_hcd 0000:00:10.1: new USB bus registered, assigned bus number 3
uhci_hcd 0000:00:10.1: irq 20, io base 0x0000b800
uhci_hcd 0000:00:10.1: supports USB remote wakeup
usb usb3: default language 0x0409
usb usb3: new device strings: Mfr=3, Product=2, SerialNumber=1
usb usb3: Product: UHCI Host Controller
usb usb3: Manufacturer: Linux 2.6.15-rc7 uhci_hcd
usb usb3: SerialNumber: 0000:00:10.1
usb usb3: hotplug
usb usb3: adding 3-0:1.0 (config #1, interface 0)
usb 3-0:1.0: hotplug
hub 3-0:1.0: usb_probe_interface
hub 3-0:1.0: usb_probe_interface - got id
hub 3-0:1.0: USB hub found
hub 3-0:1.0: 2 ports detected
hub 3-0:1.0: standalone hub
hub 3-0:1.0: no power switching (usb 1.0)
hub 3-0:1.0: individual port over-current protection
hub 3-0:1.0: power on to power good time: 2ms
hub 3-0:1.0: local power source is good
hub 2-0:1.0: debounce: port 1: total 100ms stable 100ms status 0x300
uhci_hcd 0000:00:10.0: port 2 portsc 008a,00
hub 2-0:1.0: port 2, status 0100, change 0003, 12 Mb/s
drivers/usb/core/inode.c: creating file '001'
acpi_bus-0201 [01] bus_set_power         : Device is not power manageable
ACPI: PCI Interrupt 0000:00:10.2[B] -> GSI 21 (level, low) -> IRQ 20
PCI: Via IRQ fixup for 0000:00:10.2, from 10 to 4
uhci_hcd 0000:00:10.2: UHCI Host Controller
uhci_hcd 0000:00:10.2: detected 2 ports
uhci_hcd 0000:00:10.2: uhci_check_and_reset_hc: cmd = 0x0000
uhci_hcd 0000:00:10.2: Performing full reset
drivers/usb/core/inode.c: creating file '004'
uhci_hcd 0000:00:10.2: new USB bus registered, assigned bus number 4
uhci_hcd 0000:00:10.2: irq 20, io base 0x0000c000
uhci_hcd 0000:00:10.2: supports USB remote wakeup
usb usb4: default language 0x0409
usb usb4: new device strings: Mfr=3, Product=2, SerialNumber=1
usb usb4: Product: UHCI Host Controller
usb usb4: Manufacturer: Linux 2.6.15-rc7 uhci_hcd
usb usb4: SerialNumber: 0000:00:10.2
usb usb4: hotplug
usb usb4: adding 4-0:1.0 (config #1, interface 0)
usb 4-0:1.0: hotplug
hub 4-0:1.0: usb_probe_interface
hub 4-0:1.0: usb_probe_interface - got id
hub 4-0:1.0: USB hub found
hub 4-0:1.0: 2 ports detected
hub 4-0:1.0: standalone hub
hub 4-0:1.0: no power switching (usb 1.0)
hub 4-0:1.0: individual port over-current protection
hub 4-0:1.0: power on to power good time: 2ms
hub 4-0:1.0: local power source is good
hub 2-0:1.0: debounce: port 2: total 100ms stable 100ms status 0x100
hub 2-0:1.0: state 5 ports 2 chg 0000 evt 0000
hub 3-0:1.0: state 5 ports 2 chg 0000 evt 0006
uhci_hcd 0000:00:10.1: port 1 portsc 018a,00
hub 3-0:1.0: port 1, status 0300, change 0003, 1.5 Mb/s
drivers/usb/core/inode.c: creating file '001'
acpi_bus-0201 [01] bus_set_power         : Device is not power manageable
ACPI: PCI Interrupt 0000:00:10.3[B] -> GSI 21 (level, low) -> IRQ 20
PCI: Via IRQ fixup for 0000:00:10.3, from 10 to 4
uhci_hcd 0000:00:10.3: UHCI Host Controller
uhci_hcd 0000:00:10.3: detected 2 ports
uhci_hcd 0000:00:10.3: uhci_check_and_reset_hc: cmd = 0x0000
uhci_hcd 0000:00:10.3: Performing full reset
drivers/usb/core/inode.c: creating file '005'
uhci_hcd 0000:00:10.3: new USB bus registered, assigned bus number 5
uhci_hcd 0000:00:10.3: irq 20, io base 0x0000c400
uhci_hcd 0000:00:10.3: supports USB remote wakeup
usb usb5: default language 0x0409
usb usb5: new device strings: Mfr=3, Product=2, SerialNumber=1
usb usb5: Product: UHCI Host Controller
usb usb5: Manufacturer: Linux 2.6.15-rc7 uhci_hcd
usb usb5: SerialNumber: 0000:00:10.3
usb usb5: hotplug
usb usb5: adding 5-0:1.0 (config #1, interface 0)
usb 5-0:1.0: hotplug
hub 5-0:1.0: usb_probe_interface
hub 5-0:1.0: usb_probe_interface - got id
hub 5-0:1.0: USB hub found
hub 5-0:1.0: 2 ports detected
hub 5-0:1.0: standalone hub
hub 5-0:1.0: no power switching (usb 1.0)
hub 5-0:1.0: individual port over-current protection
hub 5-0:1.0: power on to power good time: 2ms
hub 5-0:1.0: local power source is good
drivers/usb/core/inode.c: creating file '001'
hub 3-0:1.0: debounce: port 1: total 100ms stable 100ms status 0x300
uhci_hcd 0000:00:10.1: port 2 portsc 018a,00
hub 3-0:1.0: port 2, status 0300, change 0003, 1.5 Mb/s
hub 3-0:1.0: debounce: port 2: total 100ms stable 100ms status 0x300
hub 4-0:1.0: state 5 ports 2 chg 0000 evt 0006
uhci_hcd 0000:00:10.2: port 1 portsc 008a,00
hub 4-0:1.0: port 1, status 0100, change 0003, 12 Mb/s
hub 4-0:1.0: debounce: port 1: total 100ms stable 100ms status 0x100
uhci_hcd 0000:00:10.2: port 2 portsc 018a,00
hub 4-0:1.0: port 2, status 0300, change 0003, 1.5 Mb/s
hub 4-0:1.0: debounce: port 2: total 100ms stable 100ms status 0x300
hub 3-0:1.0: state 5 ports 2 chg 0000 evt 0000
hub 5-0:1.0: state 5 ports 2 chg 0000 evt 0006
uhci_hcd 0000:00:10.3: port 1 portsc 008a,00
hub 5-0:1.0: port 1, status 0100, change 0003, 12 Mb/s
hub 5-0:1.0: debounce: port 1: total 100ms stable 100ms status 0x100
uhci_hcd 0000:00:10.3: port 2 portsc 018a,00
hub 5-0:1.0: port 2, status 0300, change 0003, 1.5 Mb/s
hub 5-0:1.0: debounce: port 2: total 100ms stable 100ms status 0x300
hub 5-0:1.0: state 5 ports 2 chg 0000 evt 0000
uhci_hcd 0000:00:10.0: suspend_rh (auto-stop)
uhci_hcd 0000:00:10.1: suspend_rh (auto-stop)
uhci_hcd 0000:00:10.2: suspend_rh (auto-stop)
uhci_hcd 0000:00:10.3: suspend_rh (auto-stop)
usbcore: registered new driver hiddev
usbcore: registered new driver usbhid
drivers/usb/input/hid-core.c: v2.6:USB HID core driver
NET: Registered protocol family 1
EXT3 FS on sda1, internal journal

Re: X86_64 + VIA + 4g problems

[Andi Kleen]

Dieter Stüken <stueken@conterra.de> writes:

> Andi Kleen wrote:
> > Can you please post the full boot log? ...
> > When you not compile in the SKGE network driver does everything else work?
> > skge supports 64bit DMA, so it shouldn't use any IOMMU.  But I'm somewhat
> > suspicious of the >4GB support in the VIA chipset. We had problems with
> > that before. It's possible that it's just not supported in the hardware
> > or that the BIOS sets up the MTRRs wrong.
> 
> sorry for the delay, we are a few hors off here...

Does everything work (including the SKGE) driver
when you boot with swiotlb=force ? 

-Andi

Re: X86_64 + VIA + 4g problems

[Dieter Stüken]

Andi Kleen wrote:
> Does everything work (including the SKGE) driver
> when you boot with swiotlb=force ? 

Yes, it seems to work so far! I'm just reading some GB via NFS (while writing this
mail on the same host). I already performed some other test: without initializing the
network (and still without swiotlb=force) all my SATA controller seem to work properly.
I did a "dd bs=400M" on each in parallel. So I'm sure that each of my 4G was involved.
(also I'm not sure, if EACH of my 3 controllers really used something above 3G)

All this was still with an unpatched 2.6.15-rc7 and with IOMMU disabled.
So I could either try "iommu=allowed", as suggested by dmesg from
check_ioapic() or I may apply the suggested patch for pci-gart.c,
or both with and without network....

May be I report about soon, else tomorrow.

Dieter.
-- 
Dieter Stüken, con terra GmbH, Münster
     stueken@conterra.de
     http://www.conterra.de/
     (0)251-7474-501

Re: X86_64 + VIA + 4g problems

[Dieter Stüken]

OK, here are my last results for today:

using "iommu=allowed" did not work. System freezes during initialization
of the PDC20318, which is on the external PCI bus.

But swiotlb=force works well!

Using the patch for pci-gart.c currently works while reading data
on all 8 disks in parallel, reading data via NFS and writing this mail.

The pci-gart.c patch seems to disable dma. Is this the DMA my PCI devices
perform them self? As I learned, they may perform DMA even above 4g if all
works well. Thus I may be happy without any IOMMU. As I saw my system working
even without this patch, I will turn back to the original 2.6.15-rc7 an continue
running this torture test during this night.

Dieter.
-- 
Dieter Stüken, con terra GmbH, Münster
     stueken@conterra.de
     http://www.conterra.de/
     (0)251-7474-501

Re: X86_64 + VIA + 4g problems

[Andi Kleen]

Dieter Stüken <stueken@conterra.de> writes:
[can you please not always drop me from cc with each reply?]

Dieter Stüken <stueken@conterra.de> writes:

> OK, here are my last results for today:
>
> using "iommu=allowed" did not work. System freezes during initialization
> of the PDC20318, which is on the external PCI bus.
>
> But swiotlb=force works well!

This means your PCI bridge doesn't support addresses >4GB.

> The pci-gart.c patch seems to disable dma. 

Only DMA for addresses >4GB.

> Is this the DMA my PCI devices
> perform them self? As I learned, they may perform DMA even above 4g if all
> works well. Thus I may be happy without any IOMMU. As I saw my system working
> even without this patch, I will turn back to the original 2.6.15-rc7 an continue
> running this torture test during this night.

The patch should perform slightly better than swiotlb=force
because it will only force bounce buffering for addresses >4GB.

If your torture test involves more than 64MB of IO in flight
you might also need to increase the bounce buffer area
with swiotlb=128M or somesuch.

-Andi

Re: X86_64 + VIA + 4g problems

[Dieter Stüken]

Andi Kleen wrote:
> [can you please not always drop me from cc with each reply?] 

sorry, I thought the other way around: it would be annoying
to get each mail twice...

I just found a mailing list "discuss@x86-64.org". Would it be
better to turn over there, instead of polluting linux.kernel
by this?

>> But swiotlb=force works well! 
>
> This means your PCI bridge doesn't support addresses >4GB. 
>
>> The pci-gart.c patch seems to disable dma.
> 
> Only DMA for addresses >4GB.
> 
> If your torture test involves more than 64MB of IO in flight
> you might also need to increase the bounce buffer area
> with swiotlb=128M or somesuch. 

I'm about to understand whats going on. So should I use the
dma patch INSTEAD of "swiotlb=force"? I'll try that tonight.

> When you not compile in the SKGE network driver does everything else work? 

I may test it again, but all seemed to work without any patch or
swiotlb settings when running without the SKGE network driver.

So, how does this work, if my PCI bridge generally fails above 4G?

Problems arise not until I enable/use the network driver.
Does the dma-patch disable DMA for all PCI devices or only
for those which cause problems (like my SKGE)?

Dieter.
-- 
Dieter Stüken, con terra GmbH, Münster
     stueken@conterra.de
     http://www.conterra.de/
     (0)251-7474-501

Re: X86_64 + VIA + 4g problems

[Andi Kleen]

On Wednesday 04 January 2006 11:27, Dieter Stüken wrote:
> Andi Kleen wrote:
> > [can you please not always drop me from cc with each reply?] 
> 
> sorry, I thought the other way around: it would be annoying
> to get each mail twice...
> 
> I just found a mailing list "discuss@x86-64.org". Would it be
> better to turn over there, instead of polluting linux.kernel
> by this?

I already cc'ed it with the reply to your first mail, but you and your
colleague with the same problem persistently destroyed
the cc list with each reply so I gave up eventually.

No need to send it there again now.

> 
> >> But swiotlb=force works well! 
> >
> > This means your PCI bridge doesn't support addresses >4GB. 
> >
> >> The pci-gart.c patch seems to disable dma.
> > 
> > Only DMA for addresses >4GB.
> > 
> > If your torture test involves more than 64MB of IO in flight
> > you might also need to increase the bounce buffer area
> > with swiotlb=128M or somesuch. 
> 
> I'm about to understand whats going on. So should I use the
> dma patch INSTEAD of "swiotlb=force"? I'll try that tonight.
> 
> > When you not compile in the SKGE network driver does everything else work? 
> 
> I may test it again, but all seemed to work without any patch or
> swiotlb settings when running without the SKGE network driver.

That's likely because the SKGE driver is your only device
that can DMA >4GB.  Typical lowend IDE, sound etc. devices have
already 4GB limits and would be forced through bounce buffering
anyways. The patch just forces that for all devices.

Also it's only a hack and would need to be made VIA specific. But 
I'm trying to get confirmation of the problem from VIA first
because it's pretty drastic action. It's still possible that 
it's only a BIOS bug or somesuch that could be worked around
by setting the right magic bits

We had a similar problem some time ago with a small number
of NF4 boards where it turned out to be a broken BIOS not
programming the bridge correclty.

One contributing factor is likely that the cheap boards are
normally not tested with 4GB RAM, and VIA is normally only found
on cheap boards.

-Andi

Re: X86_64 + VIA + 4g problems

[Dieter Stüken]

Andi Kleen wrote:
> Does everything work (including the SKGE) driver
> when you boot with swiotlb=force ? 

I just realize there are TWO modules to get my 3c940 running:
skge and sk98lin. All my problems posted so far were related
to the sk98lin module, I used until now. I'll try to use the
skge module as soon as I'm able to reboot.

Dieter.
-- 
Dieter Stüken, con terra GmbH, Münster
     stueken@conterra.de
     http://www.conterra.de/
     (0)251-7474-501