semaphore woes in 2.6, 32bit

semaphore woes in 2.6, 32bit

[Jun Sun]

I got a bunch of segfaults which are due to HAS_LLSCD cpu operating
on a semaphore which is aligned along 4-byte boundary instead of the
desired 8-byte boundary.

I traced down one such place at serial/serial_core.c:

int uart_register_driver(struct uart_driver *drv)
{
        struct tty_driver *normal = NULL;
        int i, retval;

        BUG_ON(drv->state);

        /*
         * Maybe we should be using a slab cache for this, especially if
         * we have a large number of ports to handle.
         */
        drv->state = kmalloc(sizeof(struct uart_state) * drv->nr, GFP_KERNEL);
...

where drv->state contains a semaphore variable, but apparently kmalloc() only
give 4-byte boundary alignment.

There are many other faults, which I did not bother to trace down.
Simply removing CPU_HAS_LLSCD makes the problem go away, which probably indicates
they are all of the same nature.

I wonder why this problem only shows up now while it did not show up
earlier when we introduced the new up().  Perhaps kmalloc() always
returns 8-byte aligned blocks?

I can't think of an immediate and good fix.  Hopefully someone else smarter
than me can find a solution before I come back to it on Monday.  :)

Jun

Re: semaphore woes in 2.6, 32bit

[Ralf Baechle]

On Fri, May 07, 2004 at 06:10:31PM -0700, Jun Sun wrote:

> I got a bunch of segfaults which are due to HAS_LLSCD cpu operating
> on a semaphore which is aligned along 4-byte boundary instead of the
> desired 8-byte boundary.

Dare to give a complete version number?  I've dumped 2.4 on all my systems
months ago and never have seen this problem except with slab debugging
enabled - but that side effect of slab debugging is known since years.

  Ralf

Re: semaphore woes in 2.6, 32bit

[Jun Sun]

On Sat, May 08, 2004 at 09:18:22AM +0200, Ralf Baechle wrote:
> On Fri, May 07, 2004 at 06:10:31PM -0700, Jun Sun wrote:
> 
> > I got a bunch of segfaults which are due to HAS_LLSCD cpu operating
> > on a semaphore which is aligned along 4-byte boundary instead of the
> > desired 8-byte boundary.
> 
> Dare to give a complete version number?  I've dumped 2.4 on all my systems
> months ago and never have seen this problem except with slab debugging
> enabled - but that side effect of slab debugging is known since years.
> 

Kernel is yesterday's CVS. gcc is 3.3.1.  config is ddb5477.  No 
additional patch.  See below.

In any case if you look at the uart code you should see there
is a problem already.  'state' is allocated through kmalloc() which only
gives 4-byte alignement.  The only puzzling thing is that why this
did not show up before.  Maybe kmalloc() was giving 8-byte aligned block?

VERSION = 2
PATCHLEVEL = 6
SUBLEVEL = 6
EXTRAVERSION =-rc3
NAME=Zonked Quokka

[jsun@orion linux]$ mipsel-linux-gcc -v
Reading specs from /opt/mvl-installs/pe040201/montavista/pro/devkit/mips/fp_le/bin/../lib/gcc-lib/mipsel-hardhat-linux/3.3.1/specs
Configured with: ../configure --host=i686-pc-linux-gnu --target=mipsel-hardhat-linux --prefix=/opt/montavista/devkit/mips/fp_le --exec-prefix=/opt/montavista/devkit/mips/fp_le --bindir=/opt/montavista/devkit/mips/fp_le/bin --sbindir=/opt/montavista/devkit/mips/fp_le/sbin --sysconfdir=/opt/montavista/devkit/mips/fp_le/etc --datadir=/opt/montavista/devkit/mips/fp_le/share --includedir=/opt/montavista/devkit/mips/fp_le/include --libdir=/opt/montavista/devkit/mips/fp_le/lib --libexecdir=/opt/montavista/devkit/mips/fp_le/libexec --localstatedir=/opt/montavista/devkit/mips/fp_le/var --sharedstatedir=/opt/montavista/devkit/mips/fp_le/share --mandir=/opt/montavista/devkit/mips/fp_le/man --infodir=/opt/montavista/devkit/mips/fp_le/info --program-transform-name=s,^,mips_fp_le-, --enable-cross --with-sysroot=/opt/montavista/devkit/mips/fp_le/target --enable-shared --enable-languages=c,c++ --enable-__cxa_atexit --enable-threads=posix --disable-multilib --with-gxx-include-dir='$'{gcc_tooldir}/../target/usr/include/c++/3.3.1
Thread model: posix
gcc version 3.3.1 (MontaVista 3.3.1-3.0.10.0300532 2003-12-24)
 
Jun

Re: semaphore woes in 2.6, 32bit

[Geert Uytterhoeven]

On Sat, 8 May 2004, Jun Sun wrote:
> On Sat, May 08, 2004 at 09:18:22AM +0200, Ralf Baechle wrote:
> > On Fri, May 07, 2004 at 06:10:31PM -0700, Jun Sun wrote:
> >
> > > I got a bunch of segfaults which are due to HAS_LLSCD cpu operating
> > > on a semaphore which is aligned along 4-byte boundary instead of the
> > > desired 8-byte boundary.
> >
> > Dare to give a complete version number?  I've dumped 2.4 on all my systems
> > months ago and never have seen this problem except with slab debugging
> > enabled - but that side effect of slab debugging is known since years.
> >
>
> Kernel is yesterday's CVS. gcc is 3.3.1.  config is ddb5477.  No
> additional patch.  See below.
>
> In any case if you look at the uart code you should see there
> is a problem already.  'state' is allocated through kmalloc() which only
> gives 4-byte alignement.  The only puzzling thing is that why this
> did not show up before.  Maybe kmalloc() was giving 8-byte aligned block?

AFAIK, kmaloc() always[*] returns 8-byte (or higher, for archs that need it)
aligned blocks.

[*] But I did see reports of this being false if slab debugging was enabled.

Gr{oetje,eeting}s,

						Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
							    -- Linus Torvalds

Re: semaphore woes in 2.6, 32bit

[Ralf Baechle]

On Sun, May 09, 2004 at 11:09:29AM +0200, Geert Uytterhoeven wrote:

> > Kernel is yesterday's CVS. gcc is 3.3.1.  config is ddb5477.  No
> > additional patch.  See below.
> >
> > In any case if you look at the uart code you should see there
> > is a problem already.  'state' is allocated through kmalloc() which only
> > gives 4-byte alignement.  The only puzzling thing is that why this
> > did not show up before.  Maybe kmalloc() was giving 8-byte aligned block?
> 
> AFAIK, kmaloc() always[*] returns 8-byte (or higher, for archs that need it)
> aligned blocks.

We got tripped by a change in 2.6.6-rc2.  Before that change the kmalloc
slab caches were being created with SLAB_HWCACHE_ALIGN which results in
L1_CACHE_SHIFT alignment for allocations of L1_CACHE_SHIFT for slab caches
that are at least that size.  For the sake of S390 this behaviour was
changed; new it defaults to BYTES_PER_WORD alignment which is four bytes.

Fixed by defining ARCH_KMALLOC_MINALIGN as 8.

  Ralf

Index: include/asm-mips/cache.h
===================================================================
RCS file: /home/cvs/linux/include/asm-mips/cache.h,v
retrieving revision 1.16
diff -u -r1.16 cache.h
--- include/asm-mips/cache.h	10 Oct 2003 20:37:35 -0000	1.16
+++ include/asm-mips/cache.h	9 May 2004 12:57:38 -0000
@@ -18,4 +18,6 @@
 #define SMP_CACHE_SHIFT		L1_CACHE_SHIFT
 #define SMP_CACHE_BYTES		L1_CACHE_BYTES
 
+#define ARCH_KMALLOC_MINALIGN	8
+
 #endif /* _ASM_CACHE_H */

Re: semaphore woes in 2.6, 32bit

[Atsushi Nemoto]

>>>>> On Sun, 9 May 2004 14:57:50 +0200, Ralf Baechle <ralf@linux-mips.org> said:

ralf> We got tripped by a change in 2.6.6-rc2.  Before that change the
ralf> kmalloc slab caches were being created with SLAB_HWCACHE_ALIGN
ralf> which results in L1_CACHE_SHIFT alignment for allocations of
ralf> L1_CACHE_SHIFT for slab caches that are at least that size.  For
ralf> the sake of S390 this behaviour was changed; new it defaults to
ralf> BYTES_PER_WORD alignment which is four bytes.

ralf> Fixed by defining ARCH_KMALLOC_MINALIGN as 8.

Hmm, many drivers use kmalloc and pci_map_single for DMA buffer.  So
ARCH_KMALLOC_MINALIGN should be L1_CACHE_BYTES for non-coherent
system?

---
Atsushi Nemoto

Re: semaphore woes in 2.6, 32bit

[Ralf Baechle]

On Sun, May 09, 2004 at 10:56:37PM +0900, Atsushi Nemoto wrote:

> ralf> We got tripped by a change in 2.6.6-rc2.  Before that change the
> ralf> kmalloc slab caches were being created with SLAB_HWCACHE_ALIGN
> ralf> which results in L1_CACHE_SHIFT alignment for allocations of
> ralf> L1_CACHE_SHIFT for slab caches that are at least that size.  For
> ralf> the sake of S390 this behaviour was changed; new it defaults to
> ralf> BYTES_PER_WORD alignment which is four bytes.
> 
> ralf> Fixed by defining ARCH_KMALLOC_MINALIGN as 8.
> 
> Hmm, many drivers use kmalloc and pci_map_single for DMA buffer.  So
> ARCH_KMALLOC_MINALIGN should be L1_CACHE_BYTES for non-coherent
> system?

No, those drivers are simply broken.  dma_get_cache_alignment() gives the
mimimum alignment and width for DMA mappings and that value is larger
than kmalloc alignment in almost all cases.

  Ralf

Re: semaphore woes in 2.6, 32bit

[Atsushi Nemoto]

>>>>> On Sun, 9 May 2004 18:48:35 +0200, Ralf Baechle <ralf@linux-mips.org> said:
>> Hmm, many drivers use kmalloc and pci_map_single for DMA buffer.
>> So ARCH_KMALLOC_MINALIGN should be L1_CACHE_BYTES for non-coherent
>> system?

ralf> No, those drivers are simply broken.  dma_get_cache_alignment()
ralf> gives the mimimum alignment and width for DMA mappings and that
ralf> value is larger than kmalloc alignment in almost all cases.

I see.  Thank you for pointing out it.  I must learn 2.6 DMA API
quickly ...

---
Atsushi Nemoto

Re: semaphore woes in 2.6, 32bit

[Ralf Baechle]

On Mon, May 10, 2004 at 10:28:45PM +0900, Atsushi Nemoto wrote:

> >> Hmm, many drivers use kmalloc and pci_map_single for DMA buffer.
> >> So ARCH_KMALLOC_MINALIGN should be L1_CACHE_BYTES for non-coherent
> >> system?
> 
> ralf> No, those drivers are simply broken.  dma_get_cache_alignment()
> ralf> gives the mimimum alignment and width for DMA mappings and that
> ralf> value is larger than kmalloc alignment in almost all cases.
> 
> I see.  Thank you for pointing out it.  I must learn 2.6 DMA API
> quickly ...

This also applies to the 2.4 PCI DMA API.

  Ralf

Re: semaphore woes in 2.6, 32bit

[Atsushi Nemoto]

>>>>> On Mon, 10 May 2004 16:06:06 +0200, Ralf Baechle <ralf@linux-mips.org> said:

>> I see.  Thank you for pointing out it.  I must learn 2.6 DMA API
>> quickly ...

ralf> This also applies to the 2.4 PCI DMA API.

But 2.4 PCI DMA API does not have dma_get_cache_alignment() (or
equivalent), or am I missing something?

---
Atsushi Nemoto

Re: semaphore woes in 2.6, 32bit

[Ralf Baechle]

On Tue, May 11, 2004 at 10:53:05PM +0900, Atsushi Nemoto wrote:

> >> I see.  Thank you for pointing out it.  I must learn 2.6 DMA API
> >> quickly ...
> 
> ralf> This also applies to the 2.4 PCI DMA API.
> 
> But 2.4 PCI DMA API does not have dma_get_cache_alignment() (or
> equivalent), or am I missing something?

No, you don't; that was a problem in the 2.4 DMA API which got fixed
in 2.5.  For 2.4 using L1_CACHE_BYTES from <asm/cache.h> is a reasonable
alternative.

  Ralf