Writing to BARs of PCIE device

Writing to BARs of PCIE device

[shiv prakash Agarwal]

Hi All,

I have a stress test which write a specific 4B pattern to whole BAR
space for all BARs of a pcie device.

Query is:

1. Is it allowed? Or will it have any side-effects?
2. If No, is it not allowed for all BARs or some BARs only like BAR0?

Re: Writing to BARs of PCIE device

[Alex Williamson]

On Thu, 2014-04-03 at 20:38 +0530, shiv prakash Agarwal wrote:
> Hi All,
> 
> I have a stress test which write a specific 4B pattern to whole BAR
> space for all BARs of a pcie device.
> 
> Query is:
> 
> 1. Is it allowed? Or will it have any side-effects?
> 2. If No, is it not allowed for all BARs or some BARs only like BAR0?

Of course it will have side-effects, it's conceptually the same as
'cat /dev/random > /dev/mem', but for a device instead of the kernel.
What you're overwriting is device dependent, but in any case you're
trashing something.  The test sounds completely invalid to me.  Thanks,

Alex

Re: Writing to BARs of PCIE device

[shiv prakash Agarwal]

Thanks Alex,

1. I am not concerned about device functionality but need to verify
PCIE interface only by stressing it.

2. But I don't see writes getting reflected for most of addresses.

3. Also very often, it hangs in middle with CPU soft lockup and rcu
preempt failures as below:

[ 9732.090081] BUG: soft lockup- CPU#0 stuck for 22s! [update-notifier:1319]
[ 9742.108839] INFO: rcu_preempt self-detected stall on CPU
[ 9742.108848] INFO: rcu_preempt detected stalls on CPUs/tasks: { 0}
(detected by 3, t=366016 jiffies, g=88556, c=88555, q=2037)

4. So query is why above 2 issues? I know it will affect device
functionality but at least stress write test suite should work, right?

On Thu, Apr 3, 2014 at 8:53 PM, Alex Williamson
<alex.williamson@redhat.com> wrote:
> On Thu, 2014-04-03 at 20:38 +0530, shiv prakash Agarwal wrote:
>> Hi All,
>>
>> I have a stress test which write a specific 4B pattern to whole BAR
>> space for all BARs of a pcie device.
>>
>> Query is:
>>
>> 1. Is it allowed? Or will it have any side-effects?
>> 2. If No, is it not allowed for all BARs or some BARs only like BAR0?
>
> Of course it will have side-effects, it's conceptually the same as
> 'cat /dev/random > /dev/mem', but for a device instead of the kernel.
> What you're overwriting is device dependent, but in any case you're
> trashing something.  The test sounds completely invalid to me.  Thanks,
>
> Alex
>

Re: Writing to BARs of PCIE device

[Andrew Murray]

On 4 April 2014 07:14, shiv prakash Agarwal <chhotu.shiv@gmail.com> wrote:
> Thanks Alex,
>
> 1. I am not concerned about device functionality but need to verify
> PCIE interface only by stressing it.
>
> 2. But I don't see writes getting reflected for most of addresses.

That's probably to be expected - You are treating the address space
provided by the BARs as RAM, rather than MMIO. It's entirely up to the
endpoint vendor to determine how this address space responds to reads
and writes. You'll find that the address space probably consists of
registers, some of which will likely be read only.

>
> 3. Also very often, it hangs in middle with CPU soft lockup and rcu
> preempt failures as below:

This may also be expected. The endpoint may consider some of your
reads/writes to be invalid, e.g. writing to a read-only register,
writing/reading at an incorrect time/location. This may result in the
endpoint raising error messages or even failing to respond to read
requests. (How do you know that you're not telling the endpoint to
stop responding to subsequent reads?)

>
> [ 9732.090081] BUG: soft lockup- CPU#0 stuck for 22s! [update-notifier:1319]
> [ 9742.108839] INFO: rcu_preempt self-detected stall on CPU
> [ 9742.108848] INFO: rcu_preempt detected stalls on CPUs/tasks: { 0}
> (detected by 3, t=366016 jiffies, g=88556, c=88555, q=2037)
>
> 4. So query is why above 2 issues? I know it will affect device
> functionality but at least stress write test suite should work, right?

Only where you know that the BAR memory for your particular endpoint
is designed to support such behavior. I can't imagine that any off the
shelf device would.

A better stress test would be to plug in a load of network / SATA
cards and heavily use them.

Thanks,

Andrew Murray

Re: Writing to BARs of PCIE device

[shiv prakash Agarwal]

Thanks Andrew,

1. I am looking for some kind of stress test which will stress pcie
interface links independent of device so that no device driver
required.
2. Stress Writes(which write whole BARs space) has above issues and
seems invalid.
3. Will Stress Reads fulfill my purpose? Or any other ways?

On Fri, Apr 4, 2014 at 5:27 PM, Andrew Murray
<amurray@embedded-bits.co.uk> wrote:
> On 4 April 2014 07:14, shiv prakash Agarwal <chhotu.shiv@gmail.com> wrote:
>> Thanks Alex,
>>
>> 1. I am not concerned about device functionality but need to verify
>> PCIE interface only by stressing it.
>>
>> 2. But I don't see writes getting reflected for most of addresses.
>
> That's probably to be expected - You are treating the address space
> provided by the BARs as RAM, rather than MMIO. It's entirely up to the
> endpoint vendor to determine how this address space responds to reads
> and writes. You'll find that the address space probably consists of
> registers, some of which will likely be read only.
>
>>
>> 3. Also very often, it hangs in middle with CPU soft lockup and rcu
>> preempt failures as below:
>
> This may also be expected. The endpoint may consider some of your
> reads/writes to be invalid, e.g. writing to a read-only register,
> writing/reading at an incorrect time/location. This may result in the
> endpoint raising error messages or even failing to respond to read
> requests. (How do you know that you're not telling the endpoint to
> stop responding to subsequent reads?)
>
>>
>> [ 9732.090081] BUG: soft lockup- CPU#0 stuck for 22s! [update-notifier:1319]
>> [ 9742.108839] INFO: rcu_preempt self-detected stall on CPU
>> [ 9742.108848] INFO: rcu_preempt detected stalls on CPUs/tasks: { 0}
>> (detected by 3, t=366016 jiffies, g=88556, c=88555, q=2037)
>>
>> 4. So query is why above 2 issues? I know it will affect device
>> functionality but at least stress write test suite should work, right?
>
> Only where you know that the BAR memory for your particular endpoint
> is designed to support such behavior. I can't imagine that any off the
> shelf device would.
>
> A better stress test would be to plug in a load of network / SATA
> cards and heavily use them.
>
> Thanks,
>
> Andrew Murray