RFC: Counters for PCI Express AERs

RFC: Counters for PCI Express AERs

[Rajat Jain]

Hello,

I have been thinking about adding counters for different kinds of AERs
and expose them via sysfs. IMHO this would help by giving some sense
of "link quality" for PCIe links (a lot of correctable AERs may
indicate system is workable, but may indicate some signal integrity
issues etc). Currently, on a correctable AER, we do log them, but
having them in sysfs would allow userspace tools to possibly
(periodically) poll them and raise an appropriate warning in case of
too many errors. I know that for my purposes, getting some idea of PCI
link quality or a way to quantize it, would help.

Do you think such counters make sense or would be helpful generically?
Also, please let me know if something like this already exists?

Thanks,

Rajat

Re: RFC: Counters for PCI Express AERs

[okaya]

On 2018-05-17 17:05, Rajat Jain wrote:
> Hello,
> 
> I have been thinking about adding counters for different kinds of AERs
> and expose them via sysfs. IMHO this would help by giving some sense
> of "link quality" for PCIe links (a lot of correctable AERs may
> indicate system is workable, but may indicate some signal integrity
> issues etc). Currently, on a correctable AER, we do log them, but
> having them in sysfs would allow userspace tools to possibly
> (periodically) poll them and raise an appropriate warning in case of
> too many errors. I know that for my purposes, getting some idea of PCI
> link quality or a way to quantize it, would help.
> 
> Do you think such counters make sense or would be helpful generically?
> Also, please let me know if something like this already exists?

This question came from FB folks last year. They were told to use the 
perf events for counting.

I don't honestly have a strong opinion.

> 
> Thanks,
> 
> Rajat

Re: RFC: Counters for PCI Express AERs

[Rajat Jain]

[+Jes Sorensen]

On Thu, May 17, 2018 at 2:25 PM,  <okaya@codeaurora.org> wrote:
> On 2018-05-17 17:05, Rajat Jain wrote:
>>
>> Hello,
>>
>> I have been thinking about adding counters for different kinds of AERs
>> and expose them via sysfs. IMHO this would help by giving some sense
>> of "link quality" for PCIe links (a lot of correctable AERs may
>> indicate system is workable, but may indicate some signal integrity
>> issues etc). Currently, on a correctable AER, we do log them, but
>> having them in sysfs would allow userspace tools to possibly
>> (periodically) poll them and raise an appropriate warning in case of
>> too many errors. I know that for my purposes, getting some idea of PCI
>> link quality or a way to quantize it, would help.
>>
>> Do you think such counters make sense or would be helpful generically?
>> Also, please let me know if something like this already exists?
>
>
> This question came from FB folks last year. They were told to use the perf
> events for counting.

Thanks for the info. I think you are referring to this:
https://linuxplumbersconf.org/2017/ocw/proposals/4803.html

Jes: did anything come out of the proposal? I'm wondering if you have
any patch that in work-in-progress that I could use may be as a
starting point?

Thanks,

Rajat

>
> I don't honestly have a strong opinion.

Thanks! I'd like to work on this if not already done.

>
>>
>> Thanks,
>>
>> Rajat

Re: RFC: Counters for PCI Express AERs

[Rajat Jain]

[Fixing the new email address for Jes Sorensen now]

On Thu, May 17, 2018 at 2:48 PM, Rajat Jain <rajatja@google.com> wrote:
> [+Jes Sorensen]
>
> On Thu, May 17, 2018 at 2:25 PM,  <okaya@codeaurora.org> wrote:
>> On 2018-05-17 17:05, Rajat Jain wrote:
>>>
>>> Hello,
>>>
>>> I have been thinking about adding counters for different kinds of AERs
>>> and expose them via sysfs. IMHO this would help by giving some sense
>>> of "link quality" for PCIe links (a lot of correctable AERs may
>>> indicate system is workable, but may indicate some signal integrity
>>> issues etc). Currently, on a correctable AER, we do log them, but
>>> having them in sysfs would allow userspace tools to possibly
>>> (periodically) poll them and raise an appropriate warning in case of
>>> too many errors. I know that for my purposes, getting some idea of PCI
>>> link quality or a way to quantize it, would help.
>>>
>>> Do you think such counters make sense or would be helpful generically?
>>> Also, please let me know if something like this already exists?
>>
>>
>> This question came from FB folks last year. They were told to use the perf
>> events for counting.
>
> Thanks for the info. I think you are referring to this:
> https://linuxplumbersconf.org/2017/ocw/proposals/4803.html
>
> Jes: did anything come out of the proposal? I'm wondering if you have
> any patch that in work-in-progress that I could use may be as a
> starting point?
>
> Thanks,
>
> Rajat
>
>>
>> I don't honestly have a strong opinion.
>
> Thanks! I'd like to work on this if not already done.
>
>>
>>>
>>> Thanks,
>>>
>>> Rajat

Re: RFC: Counters for PCI Express AERs

[Jes Sorensen]

On 05/17/2018 05:52 PM, Rajat Jain wrote:
> [Fixing the new email address for Jes Sorensen now]
> 
> On Thu, May 17, 2018 at 2:48 PM, Rajat Jain <rajatja@google.com> wrote:
>> [+Jes Sorensen]
>>
>> On Thu, May 17, 2018 at 2:25 PM,  <okaya@codeaurora.org> wrote:
>>> On 2018-05-17 17:05, Rajat Jain wrote:
>>>>
>>>> Hello,
>>>>
>>>> I have been thinking about adding counters for different kinds of AERs
>>>> and expose them via sysfs. IMHO this would help by giving some sense
>>>> of "link quality" for PCIe links (a lot of correctable AERs may
>>>> indicate system is workable, but may indicate some signal integrity
>>>> issues etc). Currently, on a correctable AER, we do log them, but
>>>> having them in sysfs would allow userspace tools to possibly
>>>> (periodically) poll them and raise an appropriate warning in case of
>>>> too many errors. I know that for my purposes, getting some idea of PCI
>>>> link quality or a way to quantize it, would help.
>>>>
>>>> Do you think such counters make sense or would be helpful generically?
>>>> Also, please let me know if something like this already exists?
>>>
>>>
>>> This question came from FB folks last year. They were told to use the perf
>>> events for counting.
>>
>> Thanks for the info. I think you are referring to this:
>>
>> Jes: did anything come out of the proposal? I'm wondering if you have
>> any patch that in work-in-progress that I could use may be as a
>> starting point?

Kyle McMartin was working on this, I don't know the current status.

Jes

Re: RFC: Counters for PCI Express AERs

[Rajat Jain]

On Fri, May 18, 2018 at 7:24 AM, Jes Sorensen <jsorensen@fb.com> wrote:
> On 05/17/2018 05:52 PM, Rajat Jain wrote:
>> [Fixing the new email address for Jes Sorensen now]
>>
>> On Thu, May 17, 2018 at 2:48 PM, Rajat Jain <rajatja@google.com> wrote:
>>> [+Jes Sorensen]
>>>
>>> On Thu, May 17, 2018 at 2:25 PM,  <okaya@codeaurora.org> wrote:
>>>> On 2018-05-17 17:05, Rajat Jain wrote:
>>>>>
>>>>> Hello,
>>>>>
>>>>> I have been thinking about adding counters for different kinds of AERs
>>>>> and expose them via sysfs. IMHO this would help by giving some sense
>>>>> of "link quality" for PCIe links (a lot of correctable AERs may
>>>>> indicate system is workable, but may indicate some signal integrity
>>>>> issues etc). Currently, on a correctable AER, we do log them, but
>>>>> having them in sysfs would allow userspace tools to possibly
>>>>> (periodically) poll them and raise an appropriate warning in case of
>>>>> too many errors. I know that for my purposes, getting some idea of PCI
>>>>> link quality or a way to quantize it, would help.
>>>>>
>>>>> Do you think such counters make sense or would be helpful generically?
>>>>> Also, please let me know if something like this already exists?
>>>>
>>>>
>>>> This question came from FB folks last year. They were told to use the perf
>>>> events for counting.
>>>
>>> Thanks for the info. I think you are referring to this:
>>>
>>> Jes: did anything come out of the proposal? I'm wondering if you have
>>> any patch that in work-in-progress that I could use may be as a
>>> starting point?
>
> Kyle McMartin was working on this, I don't know the current status.


Never mind, I think I'm more than halfway there and will be sending a
patch in a day or two.

>
> Jes
>
>

Re: RFC: Counters for PCI Express AERs

[Kyle McMartin]

On Fri, May 18, 2018 at 09:31:11AM -0700, Rajat Jain wrote:
> >>> Jes: did anything come out of the proposal? I'm wondering if you have
> >>> any patch that in work-in-progress that I could use may be as a
> >>> starting point?
> >
> > Kyle McMartin was working on this, I don't know the current status.
> 
> 
> Never mind, I think I'm more than halfway there and will be sending a
> patch in a day or two.
> 

Patch set looks good to me, it's pretty analogous to what I came up with
in the fall. I ended up using the tracepoints instead of patching in sysfs
to enable it across all of our kernel versions in production.

Really glad you did this work, hopefully it gets merged.

cheers, Kyle