IRQs and memory consistency

IRQs and memory consistency

[Christopher Harvey]

This is a high level question about how IRQs work in the kernel.

I have a struct that was kmalloc'd into ram. Within this struct there
is an int, called devid. When I set this devid to a number that isn't
0, I print the following:

checking devid value....327683 in 0xcb6953d4

where 327683 is the value of the int and 0xcb6953d4 is the address of
the struct that the devid value is in.

Then an interrupt happens and when I print this value again at the
beginning of the interrupt handler I get the following printed text:

checking devid value....0 in 0xcb6953d4

notice that the devid has been set to 0. I can't find any code on my
end that would do this. Is there something, maybe related to memory
address spaces in IRQ handlers that I don't know about?

The IRQ and the setting of the devid value happen fairly close to
each other in time. (like less than a second, or closer)

Any ideas or guesses are appreciated.

-Chris

IRQs and memory consistency

[Wink Saville]

Sounds to me like there needs to be a flush of the processor cache
by using memory barriers.

I'm guessing that the IRQ is taken on a different thread and possibly
a different processor and the value needs to be flushed. You might
try having devid be an atomic_t and then use atomic_set
and atomic_read so that the "proper" memory barriers are used.

See: http://www.kernel.org/doc/Documentation/atomic_ops.txt

-- Wink


On Tue, Apr 10, 2012 at 1:22 PM, Christopher Harvey
<chris@basementcode.com>wrote:

> This is a high level question about how IRQs work in the kernel.
>
> I have a struct that was kmalloc'd into ram. Within this struct there
> is an int, called devid. When I set this devid to a number that isn't
> 0, I print the following:
>
> checking devid value....327683 in 0xcb6953d4
>
> where 327683 is the value of the int and 0xcb6953d4 is the address of
> the struct that the devid value is in.
>
> Then an interrupt happens and when I print this value again at the
> beginning of the interrupt handler I get the following printed text:
>
> checking devid value....0 in 0xcb6953d4
>
> notice that the devid has been set to 0. I can't find any code on my
> end that would do this. Is there something, maybe related to memory
> address spaces in IRQ handlers that I don't know about?
>
> The IRQ and the setting of the devid value happen fairly close to
> each other in time. (like less than a second, or closer)
>
> Any ideas or guesses are appreciated.
>
> -Chris
>
>
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20120410/885355e9/attachment.html 

IRQs and memory consistency

[Christopher Harvey]

On 10.04.2012 19:58, Wink Saville wrote:
> Sounds to me like there needs to be a flush of the processor cache
> by using memory barriers.

I used a wmb(); right after I set the value I wanted.

> I'm guessing that the IRQ is taken on a different thread and possibly
> a different processor and the value needs to be flushed. You might
> try having devid be an atomic_t and then use atomic_set
> and atomic_read so that the "proper" memory barriers are used.

should I use atomic ops instead of a wmb?
The interrupt can't happen until the value is assigned completely in 
one thread. I'm sure.

IRQs and memory consistency

[Wink Saville]

Do you have a read barrier in the IRQ?

See "SMP BARRIER PAIRING" in:

http://www.kernel.org/doc/Documentation/memory-barriers.txt

-- Wink

On Wed, Apr 11, 2012 at 4:59 AM, Christopher Harvey
<chris@basementcode.com>wrote:

> On 10.04.2012 19:58, Wink Saville wrote:
> > Sounds to me like there needs to be a flush of the processor cache
> > by using memory barriers.
>
> I used a wmb(); right after I set the value I wanted.
>
> > I'm guessing that the IRQ is taken on a different thread and possibly
> > a different processor and the value needs to be flushed. You might
> > try having devid be an atomic_t and then use atomic_set
> > and atomic_read so that the "proper" memory barriers are used.
>
> should I use atomic ops instead of a wmb?
> The interrupt can't happen until the value is assigned completely in
> one thread. I'm sure.
>
>
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20120411/2406c3e0/attachment.html 

IRQs and memory consistency

[Christopher Harvey]

On 11.04.2012 10:35, Wink Saville wrote:
> Do you have a read barrier in the IRQ?

Ah, no I don't.

> See "SMP BARRIER PAIRING" in:
>
> http://www.kernel.org/doc/Documentation/memory-barriers.txt [3]

I saw that but only skimmed it.

I'll let the mailing list know if it worked when I get a chance to try 
it.

thanks,
-Chris

IRQs and memory consistency

[Mulyadi Santosa]

Hi....

On Wed, Apr 11, 2012 at 03:22, Christopher Harvey
<chris@basementcode.com> wrote:
> The IRQ and the setting of the devid value happen fairly close to
> each other in time. (like less than a second, or closer)

Hmmm, and how about the order? which one do you guess go first?
setting value? or the IRQ handler?

I had a sense that your code flow might (in reality) goes like this

allocating struct RAM --> interrupted --> printing struct content -->
back to initialize struct

If that's true, then no wonder 0 (zero) is printed.

I suggest to grab a spin lock if you really need atomicity during
allocation and setting initial value, which in IRQ handler, you grab
the lock before printing. Oh and since memory allocation could go
slow, you might need to do allocation with GFP_ATOMIC.


-- 
regards,

Mulyadi Santosa
Freelance Linux trainer and consultant

blog: the-hydra.blogspot.com
training: mulyaditraining.blogspot.com

IRQs and memory consistency

[Christopher Harvey]

On 10.04.2012 22:18, Mulyadi Santosa wrote:
> Hi....
>
> On Wed, Apr 11, 2012 at 03:22, Christopher Harvey
> <chris@basementcode.com> wrote:
>> The IRQ and the setting of the devid value happen fairly close to
>> each other in time. (like less than a second, or closer)
>
> Hmmm, and how about the order? which one do you guess go first?
> setting value? or the IRQ handler?
>
> I had a sense that your code flow might (in reality) goes like this
>
> allocating struct RAM --> interrupted --> printing struct content -->
> back to initialize struct
>
> If that's true, then no wonder 0 (zero) is printed.
>
> I suggest to grab a spin lock if you really need atomicity during
> allocation and setting initial value, which in IRQ handler, you grab
> the lock before printing. Oh and since memory allocation could go
> slow, you might need to do allocation with GFP_ATOMIC.

well, actually my does does this:

set variable->init hardware->interrupt

the interrupt can't happen until 'init hardware'