Different behaviour when using "nosmp" parameter on SMP and UP

Different behaviour when using "nosmp" parameter on SMP and UP

[Jean-Michel Hautbois]

Hi,

I have a P2020 CPU (powerpc) and I compiled it with two different defconfigs.
The first one is a SMP, 2 cores, launched with the "nosmp" kernel
parameter, the other one is an UP kernel.

My driver behaviour is not the same whether launching one or the
other. It is hard to explain more precisely, as it deals only with
ioctl which only does ioread32/iowrite32 on a PCIe device.
But I can tell that it never works the same way when UP (not working
correctly), or SMP "nosmp" (working) or even SMP (not working).

AFAIK, the "nosmp" parameter should tell the kernel to act the same is
if it is an UP kernel, and it disables IO APIC, which is not an issue
in my case.

Can you think about anything that would explain it, or would help me
debugging it ?

Thanks in advance for your help.
Regards,
JM

Re: Different behaviour when using "nosmp" parameter on SMP and UP

[Kumar Gala]

On Dec 1, 2011, at 3:57 AM, Jean-Michel Hautbois wrote:

> Hi,
>=20
> I have a P2020 CPU (powerpc) and I compiled it with two different =
defconfigs.
> The first one is a SMP, 2 cores, launched with the "nosmp" kernel
> parameter, the other one is an UP kernel.
>=20
> My driver behaviour is not the same whether launching one or the
> other. It is hard to explain more precisely, as it deals only with
> ioctl which only does ioread32/iowrite32 on a PCIe device.
> But I can tell that it never works the same way when UP (not working
> correctly), or SMP "nosmp" (working) or even SMP (not working).
>=20
> AFAIK, the "nosmp" parameter should tell the kernel to act the same is
> if it is an UP kernel, and it disables IO APIC, which is not an issue
> in my case.
>=20
> Can you think about anything that would explain it, or would help me
> debugging it ?

This is a bit odd, hard to say w/o more details on what exactly your =
code is trying to do and wait the failure is.

The larger differences between SMP & UP build would be setting of memory =
attribute for cache coherency.  In UP case we don't set it.

Between SMP 'nosmp' and SMP 'on' cases you seem like you're hitting some =
locking condition.

Guessing your running into a timing & locking issue that you're getting =
lucky on the SMP 'nosmp' case such that it just happens to work.

- k=

Re: Different behaviour when using "nosmp" parameter on SMP and UP

[Jean-Michel Hautbois]

2011/12/1 Kumar Gala <galak@kernel.crashing.org>:
>
> On Dec 1, 2011, at 3:57 AM, Jean-Michel Hautbois wrote:
>
>> Hi,
>>
>> I have a P2020 CPU (powerpc) and I compiled it with two different defcon=
figs.
>> The first one is a SMP, 2 cores, launched with the "nosmp" kernel
>> parameter, the other one is an UP kernel.
>>
>> My driver behaviour is not the same whether launching one or the
>> other. It is hard to explain more precisely, as it deals only with
>> ioctl which only does ioread32/iowrite32 on a PCIe device.
>> But I can tell that it never works the same way when UP (not working
>> correctly), or SMP "nosmp" (working) or even SMP (not working).
>>
>> AFAIK, the "nosmp" parameter should tell the kernel to act the same is
>> if it is an UP kernel, and it disables IO APIC, which is not an issue
>> in my case.
>>
>> Can you think about anything that would explain it, or would help me
>> debugging it ?
>
> This is a bit odd, hard to say w/o more details on what exactly your code=
 is trying to do and wait the failure is.

I understand that it is difficult to give more details, but the driver
is not mine, and I can't give all the code associated.

> The larger differences between SMP & UP build would be setting of memory =
attribute for cache coherency. =C2=A0In UP case we don't set it.
>
> Between SMP 'nosmp' and SMP 'on' cases you seem like you're hitting some =
locking condition.
>
> Guessing your running into a timing & locking issue that you're getting l=
ucky on the SMP 'nosmp' case such that it just happens to work.

Any idea on how to debug this ? I am using a 2.6.35 kernel.

Thanks,
JM

Re: Different behaviour when using "nosmp" parameter on SMP and UP

[Tabi Timur-B04825]

On Thu, Dec 1, 2011 at 9:04 AM, Jean-Michel Hautbois
<jhautbois@gmail.com> wrote:
>
> Any idea on how to debug this ? I am using a 2.6.35 kernel.

There are a ton of Kconfig options for debugging various locking bugs.
 Try turning them on.

--=20
Timur Tabi
Linux kernel developer at Freescale=

Re: Different behaviour when using "nosmp" parameter on SMP and UP

[Jean-Michel Hautbois]

2011/12/1 Tabi Timur-B04825 <B04825@freescale.com>:
> On Thu, Dec 1, 2011 at 9:04 AM, Jean-Michel Hautbois
> <jhautbois@gmail.com> wrote:
>>
>> Any idea on how to debug this ? I am using a 2.6.35 kernel.
>
> There are a ton of Kconfig options for debugging various locking bugs.
> =C2=A0Try turning them on.
>

And when I do that, it works, even in SMP... :).

JM

Re: Different behaviour when using "nosmp" parameter on SMP and UP

[Jean-Michel Hautbois]

2011/12/1 Jean-Michel Hautbois <jhautbois@gmail.com>:
> 2011/12/1 Tabi Timur-B04825 <B04825@freescale.com>:
>> On Thu, Dec 1, 2011 at 9:04 AM, Jean-Michel Hautbois
>> <jhautbois@gmail.com> wrote:
>>>
>>> Any idea on how to debug this ? I am using a 2.6.35 kernel.
>>
>> There are a ton of Kconfig options for debugging various locking bugs.
>> =C2=A0Try turning them on.
>>

All Kconfig options do not help me with cache coherency problems, though.
Or, I didn't find any option related.
Any idea ?

JM

Re: Different behaviour when using "nosmp" parameter on SMP and UP

[Kumar Gala]

On Dec 1, 2011, at 11:52 AM, Jean-Michel Hautbois wrote:

> 2011/12/1 Jean-Michel Hautbois <jhautbois@gmail.com>:
>> 2011/12/1 Tabi Timur-B04825 <B04825@freescale.com>:
>>> On Thu, Dec 1, 2011 at 9:04 AM, Jean-Michel Hautbois
>>> <jhautbois@gmail.com> wrote:
>>>> 
>>>> Any idea on how to debug this ? I am using a 2.6.35 kernel.
>>> 
>>> There are a ton of Kconfig options for debugging various locking bugs.
>>>  Try turning them on.
>>> 
> 
> All Kconfig options do not help me with cache coherency problems, though.
> Or, I didn't find any option related.
> Any idea ?
> 
> JM

Nope, if you can't describe the code in any more detail we can't help.

- k

Re: Different behaviour when using "nosmp" parameter on SMP and UP

[Jean-Michel Hautbois]

2011/12/1 Kumar Gala <galak@kernel.crashing.org>:
>
> On Dec 1, 2011, at 11:52 AM, Jean-Michel Hautbois wrote:
>
>> 2011/12/1 Jean-Michel Hautbois <jhautbois@gmail.com>:
>>> 2011/12/1 Tabi Timur-B04825 <B04825@freescale.com>:
>>>> On Thu, Dec 1, 2011 at 9:04 AM, Jean-Michel Hautbois
>>>> <jhautbois@gmail.com> wrote:
>>>>>
>>>>> Any idea on how to debug this ? I am using a 2.6.35 kernel.
>>>>
>>>> There are a ton of Kconfig options for debugging various locking bugs.
>>>> =C2=A0Try turning them on.
>>>>
>>
>> All Kconfig options do not help me with cache coherency problems, though=
.
>> Or, I didn't find any option related.
>> Any idea ?
>>
>> JM
>
> Nope, if you can't describe the code in any more detail we can't help.
>

Here is what I can tell, I hope this will help you.

I have a userland process, which reads and writes data to a character
device thanks to ioctl.
Reads and Writes are performed with this scheme :
Lock Access -> ioctl -> down_interruptible(semaphore)
Read (or Write) -> ioctl -> ioread32
Unlock Access -> ioctl -> up(semaphore)

It reads (or writes) 4 bytes by 4 bytes, in a loop in the process
which will eventually get several bytes from the PCIe device.
The read (write) ioctl is really simple (and in fact, may be rewritten) :
- kmalloc a structure, then copy_from_user
- ioread32 using adress and offset from the structure copied previously
- copy_to_user and free the structure.

The userland process is currently doing a loop in order to read or
writes with a 4bytes step.
I am neither the writer of the driver nor the userland process, this
is why I can't give you the code.

Here is what I would do, at least :
1/ rewrite the read/write part of the ioctl in order to user a per_cpu
structure and not a "dynamically allocated at each call" structure
2/ rewrite the userland part in order to ask the driver for n bytes,
leaving the loop on ioread32/iowrite32 inside the ioctl

Thanks for your help.
JM