[Linux-ia64] BitKeeper tree for 2.4.x

[Linux-ia64] BitKeeper tree for 2.4.x

[Bjorn Helgaas]

I am maintaining a BitKeeper tree that tracks Marcelo's 2.4.x
tree and the current ia64 patch.  My intent is that we will still
produce a plain patch against Marcelo's releases, just as David
has always done.  The BitKeeper tree merely provides visibility
into the current state of what will become the next ia64 patch.

I try to keep the BitKeeper tree working, but I don't do much
testing before updating it, so it should be regarded as unstable
work-in-progress.

The BitKeeper tools, licensing agreement, and documentation are
available here:

    http://www.bitkeeper.com

To get a local copy of the tree, use a BitKeeper command like
this:

    bk clone bk://lia64.bkbits.net/linux-ia64-2.4 linux-ia64-2.4

Subsequently, you can use "bk pull" in the linux-ia64-2.4/
directory to update your local repository.

The repository is hosted by BitMover, Inc., which provides a web
interface with a browsable source tree and ChangeSet history
here:

    http://lia64.bkbits.net:8080/linux-ia64-2.4

Here are the interesting changes since the 2.4.19-ia64-020821 patch:

  * Driver changes:
    - bcm: dropped (you may be able to use tg3 instead).
    - e1000: dropped 4.1.7, adopted upstream (currently 4.3.15-k1).
    - e100: RX_ALIGN now upstream; only last_rx_time fix in ia64 patch.
    - acenic: dropped ia64 patches; upstream has most or all.
    - forte: dropped 1.45, adopted upstream (currently 1.51).

  * SPCR/DBGP support removed (encumbered by Microsoft IP; use HCDP instead).
  * McKinley A-step config doc removed (code was already gone).
  * HP prototype code removed (Matthew Wilcox).
  * include/asm-ia64/offsets.h removed.
  * sim{eth,scsi,serial} (drivers for HP simulator) moved to arch/ia64/hp/sim/.
  * drivers/media/{radio,video}/dummy.c removed.

  * Support for write-coalescing mappings removed.
  * Support for non-cached mappings of main memory removed.
  * Support scatterlist page/offset in sba_iommu.
  * AGP/DRM rework to make it more presentable.
    - DRM: dropped obsolete #ifdef __alpha__ diffs.
    - DRM: r128, radeon: made all 460GX checks run-time, not compile-time.
  * ACPI CRS cleanup (Takayoshi Kochi).
  * ACPI debug fixes (Takayoshi Kochi).
  * Fix many warnings (Andreas Schwab).
  * Fix I/O macros (inb, outb, etc) (Andreas Schwab, David Mosberger).
  * If more than NR_CPUS found, ignore extras.
  * Add generic RAID xor routines with prefetch (Matthew Wilcox).
  * Discard *.text.exit and *.data.exit sections (Matthew Wilcox).
  * Fix edge-triggered IRQ handling (David Mosberger).
  * Alternate signal stack fixes (David Mosberger).
  * VFS extended attribute syscall numbers (Andreas Gruenbacher).
  * binfmt argv[1] preservation (David Mosberger).
  * Preserve FP registers around firmware calls (John Marvin, David Mosberger).
  * FPU load/save optimization (Fenghua Yu).
  * Syscalls for Extended Attribute VFS infrastructure (Andreas Gruenbacher).

The following major pieces of the ia64 patch are unchanged since
2.4.19-ia64-020821:

  * ACPI: CA version 20020517 (upstream has version 20011018).
  * qla1280: 3.23 Beta (upstream has 3.00-Beta).
  * qla2x00: 4.31.7b (not in upstream).

NOTE: I removed the mmap support for MAP_WRITECOMBINED and
MAP_NONCACHED to avoid issues with memory attribute aliasing.
The only user of these that I know about is XFree86, which still
seems functional when we ignore the attributes it requests.  I'm
very interested in any problems caused by this change.

Bjorn

Re: [Linux-ia64] BitKeeper tree for 2.4.x

[Bjorn Helgaas]

On Monday 14 October 2002 12:17 pm, Steffen Persvold wrote:
> Well, without MAP_WRITECOMBINED, streaming PIO accesses to the PCI bus 
> drops from 330MByte/sec to approx. 100MByte/s using the Dolphin SCI card. 
> 
> This is an rather significant drop which makes the IA64 platform kind of 
> uninteresting for high performance clustering (the Dolphin SCI cards are 
> used for shared memory message passing between nodes).
> 
> We can of course hack this in for our usage (in our kernel driver) if you 
> are totally against having it in the kernel headers, but I think since the 
> hardware allows different mapping types, the kernel should take advantage 
> of it.

I'm really not opposed to having it in the kernel headers.  The thing
is, we were allowing WC mappings regardless of whether the hardware
actually supported them.  What we need is some mechanism for
looking at the EFI memory map, which tells us what attributes are
supported.

We've got a mechanism for looking up the supported attributes
(efi_mem_attributes()); what I'd really like is for somebody to
hook that up with the mmap infrastructure so that we only allow
the supported attributes.

Bjorn

Re: [Linux-ia64] BitKeeper tree for 2.4.x

[Steffen Persvold]

On Thu, 10 Oct 2002, Bjorn Helgaas wrote:

> NOTE: I removed the mmap support for MAP_WRITECOMBINED and
> MAP_NONCACHED to avoid issues with memory attribute aliasing.
> The only user of these that I know about is XFree86, which still
> seems functional when we ignore the attributes it requests.  I'm
> very interested in any problems caused by this change.
> 

Well, without MAP_WRITECOMBINED, streaming PIO accesses to the PCI bus 
drops from 330MByte/sec to approx. 100MByte/s using the Dolphin SCI card. 

This is an rather significant drop which makes the IA64 platform kind of 
uninteresting for high performance clustering (the Dolphin SCI cards are 
used for shared memory message passing between nodes).

We can of course hack this in for our usage (in our kernel driver) if you 
are totally against having it in the kernel headers, but I think since the 
hardware allows different mapping types, the kernel should take advantage 
of it.

Regards,
 -- 
  Steffen Persvold   |       Scali AS      
 mailto:sp@scali.com |  http://www.scali.com
Tel: (+47) 2262 8950 |   Olaf Helsets vei 6
Fax: (+47) 2262 8951 |   N0621 Oslo, NORWAY

Re: [Linux-ia64] BitKeeper tree for 2.4.x

[Grant Grundler]

Bjorn Helgaas wrote:
> We've got a mechanism for looking up the supported attributes
> (efi_mem_attributes()); what I'd really like is for somebody to
> hook that up with the mmap infrastructure so that we only allow
> the supported attributes.

If MAP_WRITECOMBINED attribute isn't supported, is there any harm
in returning memory without that attribute?

I'd expect some platform specific support to know when it's
required vs a "performance hint". Forcing exact matches is one way
to deal with it but pushes the burden to the "application"
(may a device driver in this case) to find an exact match.

grant

Re: [Linux-ia64] BitKeeper tree for 2.4.x

[Bjorn Helgaas]

> Which IA64 hardware doesn't support _PAGE_MA_WC in the PTE ? My IA64 docs 
> doesn't mention anything about that.

The *processors* all support WC, no problem there.  The problem is
that the *chipset* may not.  The EFI memory descriptor table (from
GetMemoryMap()) tells us which attributes are supported for each
region of address space.

According to the EFI shell "memmap" command, the i2000 (BigSur)
supports WB or UC for memory, and UC for MMIO space.  No
mention of WC, so we have to assume it's not supported.

HP ZX1 machines report that they support WB for memory and
UC for MMIO space.  The ZX1 chipset is supposed to support WC
for MMIO space, so the fact that EFI doesn't report that looks
like a firmware defect.

> Actually I think _PAGE_MA_WC is only applicable to IO memory the same way 
> as _PAGE_MA_UC is. Can't it be handled in the same way ? (you've already 
> done the fix for UC, right ?)

WC could be used for memory as well as for MMIO space -- an
example is for AGP, where drivers like to have main memory
buffers mapped with WC, and the AGP engine can do non-coherent
DMA from the buffers.  That doesn't work on IA64 because we
use large TLB pages to map all of main memory with the WB
attribute, and there's no easy way to support WC mappings
at the same time.

The mmap support for UC currently cheats a little bit.  We
don't look at the EFI tables; we just use UC whenever we mmap
something that isn't main memory (this is in mmap_mem(), BTW).

Bjorn

Re: [Linux-ia64] BitKeeper tree for 2.4.x

[Steffen Persvold]

On Mon, 14 Oct 2002, Bjorn Helgaas wrote:

> On Monday 14 October 2002 12:17 pm, Steffen Persvold wrote:
> > Well, without MAP_WRITECOMBINED, streaming PIO accesses to the PCI bus 
> > drops from 330MByte/sec to approx. 100MByte/s using the Dolphin SCI card. 
> > 
> > This is an rather significant drop which makes the IA64 platform kind of 
> > uninteresting for high performance clustering (the Dolphin SCI cards are 
> > used for shared memory message passing between nodes).
> > 
> > We can of course hack this in for our usage (in our kernel driver) if you 
> > are totally against having it in the kernel headers, but I think since the 
> > hardware allows different mapping types, the kernel should take advantage 
> > of it.
> 
> I'm really not opposed to having it in the kernel headers.  The thing
> is, we were allowing WC mappings regardless of whether the hardware
> actually supported them.  What we need is some mechanism for
> looking at the EFI memory map, which tells us what attributes are
> supported.

Which IA64 hardware doesn't support _PAGE_MA_WC in the PTE ? My IA64 docs 
doesn't mention anything about that.

> 
> We've got a mechanism for looking up the supported attributes
> (efi_mem_attributes()); what I'd really like is for somebody to
> hook that up with the mmap infrastructure so that we only allow
> the supported attributes.

Actually I think _PAGE_MA_WC is only applicable to IO memory the same way 
as _PAGE_MA_UC is. Can't it be handled in the same way ? (you've already 
done the fix for UC, right ?)

Regards,
-- 
  Steffen Persvold   |       Scali AS      
 mailto:sp@scali.com |  http://www.scali.com
Tel: (+47) 2262 8950 |   Olaf Helsets vei 6
Fax: (+47) 2262 8951 |   N0621 Oslo, NORWAY

Re: [Linux-ia64] BitKeeper tree for 2.4.x

[Steffen Persvold]

On Thu, 17 Oct 2002, Bjorn Helgaas wrote:

> > Which IA64 hardware doesn't support _PAGE_MA_WC in the PTE ? My IA64 docs 
> > doesn't mention anything about that.
> 
> The *processors* all support WC, no problem there.  The problem is
> that the *chipset* may not.  The EFI memory descriptor table (from
> GetMemoryMap()) tells us which attributes are supported for each
> region of address space.
> 
> According to the EFI shell "memmap" command, the i2000 (BigSur)
> supports WB or UC for memory, and UC for MMIO space.  No
> mention of WC, so we have to assume it's not supported.
> 
> HP ZX1 machines report that they support WB for memory and
> UC for MMIO space.  The ZX1 chipset is supposed to support WC
> for MMIO space, so the fact that EFI doesn't report that looks
> like a firmware defect.
> 

So what does "supported" by the chipset actually mean ? I've tested the 
SCI cards in both BigSurs and ZX2000 and it works fine on IO memory 
(roughly 330 MByte on both platforms).

I was in under the impression that WC can be implemented in different 
sections of the system (processor->memorycontroller, 
memorycontroller->pci etc), and that the section we control with the 
WC attribute in the PTE is transactions on the 
processor->memorycontroller bus (as it is on the IA32 systems with MTRRs). 
Then the question of wether the memory contoller (chipset) is able to 
support large bursts of data on the processor->memorycontroller bus comes 
down to buffer space in the memory controller, and I can't really imagine 
that this isn't large enough (haven't checked though).

_But_ I agree that if the firmware has the ability to report this (and 
we can trust the information) we should take advantage of it (sort of the 
same thing which is done when someone tries to create a WC memory region 
on IA32, not all processors supports this).

> > Actually I think _PAGE_MA_WC is only applicable to IO memory the same way 
> > as _PAGE_MA_UC is. Can't it be handled in the same way ? (you've already 
> > done the fix for UC, right ?)
> 
> WC could be used for memory as well as for MMIO space -- an
> example is for AGP, where drivers like to have main memory
> buffers mapped with WC, and the AGP engine can do non-coherent
> DMA from the buffers.  That doesn't work on IA64 because we
> use large TLB pages to map all of main memory with the WB
> attribute, and there's no easy way to support WC mappings
> at the same time.
> 

I can't really see why mapping main memory with WC (processor) should 
affect the DMA performance of the AGP card. The other way around (mapping 
the AGP card's memory with WC) makes sense though and I know a lot of 
drivers that does this.

> The mmap support for UC currently cheats a little bit.  We
> don't look at the EFI tables; we just use UC whenever we mmap
> something that isn't main memory (this is in mmap_mem(), BTW).
> 

Some time ago I submitted a patch do David to allow mapping IO pages with 
WC in kernel space (ioremap()). Back then I remember this discussion about 
multiple processes having different PTE attributes on the same pages, so I 
guess this stuff with PTE attributes should be cleaned up (for all archs 
really).

Regards,
 -- 
  Steffen Persvold   | Scalable Linux Systems |   Try out the world's best
 mailto:sp@scali.com |  http://www.scali.com  | performing MPI implementation:
Tel: (+47) 2262 8950 |   Olaf Helsets vei 6   |      - ScaMPI 1.13.8 -
Fax: (+47) 2262 8951 |   N0621 Oslo, NORWAY   | >320MBytes/s and <4uS latency

Re: [Linux-ia64] BitKeeper tree for 2.4.x

[Bjorn Helgaas]

> So what does "supported" by the chipset actually mean ? I've tested the 
> SCI cards in both BigSurs and ZX2000 and it works fine on IO memory 
> (roughly 330 MByte on both platforms).

One thing it means is that supported attributes have been considered
in the hardware design, tested (hopefully), and if they don't work, the
vendor should be interested.  If we use unsupported attributes, we're
exposing ourselves to the possibility of MCAs and data corruption.

> I can't really see why mapping main memory with WC (processor) should 
> affect the DMA performance of the AGP card.

If the AGP card reads main memory with non-coherent DMA, those
transactions don't need to be snooped by the processor, and may
not need to appear on the processor bus.

Bjorn

Re: [Linux-ia64] BitKeeper tree for 2.4.x

[Steffen Persvold]

On Thu, 17 Oct 2002, Bjorn Helgaas wrote:

> > So what does "supported" by the chipset actually mean ? I've tested the
> > SCI cards in both BigSurs and ZX2000 and it works fine on IO memory
> > (roughly 330 MByte on both platforms).
>
> One thing it means is that supported attributes have been considered
> in the hardware design, tested (hopefully), and if they don't work, the
> vendor should be interested.  If we use unsupported attributes, we're
> exposing ourselves to the possibility of MCAs and data corruption.
>

I agree, I actually have some bad experiences on IA32 platforms where
data corruption happened with CPU WC enabled (in MTRRs). Sorry for the
fuzz.

> > I can't really see why mapping main memory with WC (processor) should
> > affect the DMA performance of the AGP card.
>
> If the AGP card reads main memory with non-coherent DMA, those
> transactions don't need to be snooped by the processor, and may
> not need to appear on the processor bus.
>

Ok, and then I also see what you mean by "supported by the chipset"
because this must be managed by the memory controller and therefore it
must know what type of mappings are used by the processor. Kinda
complicated though... I thought the IO system didn't use coherence
protocols, and it was because of that we needed API functions such as
pci_dma_sync_single() and pci_dma_sync_sg() to manually ensure coherence..

Regards,
 --
  Steffen Persvold   | Scalable Linux Systems |   Try out the world's best
 mailto:sp@scali.com |  http://www.scali.com  | performing MPI implementation:
Tel: (+47) 2262 8950 |   Olaf Helsets vei 6   |      - ScaMPI 1.13.8 -
Fax: (+47) 2262 8951 |   N0621 Oslo, NORWAY   | >320MBytes/s and <4uS latency

Re: [Linux-ia64] BitKeeper tree for 2.4.x

[Grant Grundler]

Steffen Persvold wrote:
> Kinda complicated though... I thought the IO system didn't use coherence
> protocols, and it was because of that we needed API functions such as
> pci_dma_sync_single() and pci_dma_sync_sg() to manually ensure coherence..

The original problem was some IOMMU's can not support a static translation
for all physical memory - ie virt_to_bus() wouldn't work.
"manual coherence" works because the design accomodates SW IOTLB
("bounce buffers") model.

Supported HP PARISC and IA64 IOMMU's participate in the coherency protocol.

grant

Re: [Linux-ia64] BitKeeper tree for 2.4.x

[Steffen Persvold]

On Thu, 17 Oct 2002, Grant Grundler wrote:

> Steffen Persvold wrote:
> > Kinda complicated though... I thought the IO system didn't use coherence
> > protocols, and it was because of that we needed API functions such as
> > pci_dma_sync_single() and pci_dma_sync_sg() to manually ensure coherence..
> 
> The original problem was some IOMMU's can not support a static translation
> for all physical memory - ie virt_to_bus() wouldn't work.
> "manual coherence" works because the design accomodates SW IOTLB
> ("bounce buffers") model.
> 
> Supported HP PARISC and IA64 IOMMU's participate in the coherency protocol.
> 

True, but I still can't see how a PTE with the WC attribute (in the CPU 
MMU) can affect the coherency protocol the IOMMU uses on DMA read 
transfers. I thought the WC attribute just controlled how the CPU buffered 
data.

Anyway, although I agree that the availability of the WC feature should be 
checked with the information from the firware, I don't agree that this 
should be done now on the 2.4 series. Atleast don't remove WC (or UC) 
support in any 2.4 kernels before the validation routines are ready and 
implemented.

Regards,
-- 
  Steffen Persvold   |       Scali AS      
 mailto:sp@scali.com |  http://www.scali.com
Tel: (+47) 2262 8950 |   Olaf Helsets vei 6
Fax: (+47) 2262 8951 |   N0621 Oslo, NORWAY