enabling aspm on ati radeon

enabling aspm on ati radeon

[Phillip Susi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I thought that booting with the pcie_aspm=force option should make
sure this was enabled, but passing this to a 3.8.0 kernel, lspci still
shows:

LnkCap:	Port #0, Speed 5GT/s, Width x16, ASPM L0s L1, Latency L0
<64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot-
LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-

What could be disabling ASPM?  dmesg confirms: [    0.000000] PCIe
ASPM is forcibly enabled

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iQEcBAEBAgAGBQJRlEOfAAoJEJrBOlT6nu75D7gH/1/Nk62FAELr/IJR49pNFNVB
TY0JflxJ7hRMxGUYtcKCgGLA2UrUSf95fFh11NmWMCZFRXHS/9WQXp2gr8KUGXIp
iOlHHME2egFslbr8nUp3aHkmq0/D7lbZdK7eA1PO6hojnxapygbqt6tjpRKT7Vke
djWz+43IiljDD7ZKz1JoUPjJIEV2v/Eg5BpXFEVEiXXFYFfjAU3ZG95/mNuvTX3T
MraQFQOP/4kqpMc75tSsAsfMHg3ciYOt0GIYIk9SUE6mONkAY6o2g4kRgva5CwQC
bBvhDZcQfx8QsIPlTBa74Kjjx9q7yQyOD6lkJTLE5SQWfTiBwB5ET9Dt0kKSJ/I=
=WgMm
-----END PGP SIGNATURE-----

Re: enabling aspm on ati radeon

[Rafael J. Wysocki]

[Add linux-pci to the CC]

On Wednesday, May 15, 2013 10:25:37 PM Phillip Susi wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> I thought that booting with the pcie_aspm=force option should make
> sure this was enabled, but passing this to a 3.8.0 kernel, lspci still
> shows:
> 
> LnkCap:	Port #0, Speed 5GT/s, Width x16, ASPM L0s L1, Latency L0
> <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot-
> LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
> 
> What could be disabling ASPM?  dmesg confirms: [    0.000000] PCIe
> ASPM is forcibly enabled
> 
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.12 (GNU/Linux)
> Comment: Using GnuPG with undefined - http://www.enigmail.net/
> 
> iQEcBAEBAgAGBQJRlEOfAAoJEJrBOlT6nu75D7gH/1/Nk62FAELr/IJR49pNFNVB
> TY0JflxJ7hRMxGUYtcKCgGLA2UrUSf95fFh11NmWMCZFRXHS/9WQXp2gr8KUGXIp
> iOlHHME2egFslbr8nUp3aHkmq0/D7lbZdK7eA1PO6hojnxapygbqt6tjpRKT7Vke
> djWz+43IiljDD7ZKz1JoUPjJIEV2v/Eg5BpXFEVEiXXFYFfjAU3ZG95/mNuvTX3T
> MraQFQOP/4kqpMc75tSsAsfMHg3ciYOt0GIYIk9SUE6mONkAY6o2g4kRgva5CwQC
> bBvhDZcQfx8QsIPlTBa74Kjjx9q7yQyOD6lkJTLE5SQWfTiBwB5ET9Dt0kKSJ/I=
> =WgMm
> -----END PGP SIGNATURE-----
> --
> To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
-- 
I speak only for myself.
Rafael J. Wysocki, Intel Open Source Technology Center.

Re: enabling aspm on ati radeon

[Bjorn Helgaas]

On Thu, May 16, 2013 at 2:58 AM, Rafael J. Wysocki <rjw@sisk.pl> wrote:
> [Add linux-pci to the CC]
>
> On Wednesday, May 15, 2013 10:25:37 PM Phillip Susi wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> I thought that booting with the pcie_aspm=force option should make
>> sure this was enabled, but passing this to a 3.8.0 kernel, lspci still
>> shows:
>>
>> LnkCap:       Port #0, Speed 5GT/s, Width x16, ASPM L0s L1, Latency L0
>> <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot-
>> LnkCtl:       ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
>> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>>
>> What could be disabling ASPM?  dmesg confirms: [    0.000000] PCIe
>> ASPM is forcibly enabled

Can you collect a complete dmesg log and complete "lspci -vv" output, please?

Bjorn

Re: enabling aspm on ati radeon

[Bjorn Helgaas]

On Thu, May 16, 2013 at 4:10 PM, Phillip Susi <psusi@ubuntu.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 05/16/2013 12:42 PM, Bjorn Helgaas wrote:
>> Can you collect a complete dmesg log and complete "lspci -vv" output, please?
> ...

Thanks.  The BIOS gave us control over ASPM via _OSC, and you have
these devices where it looks like we should be enabling ASPM, because
it's supported on both ends of the link:

    00:1c.1 bridge to [bus 03] ASPM L0s L1, ASPM disabled
      03:00.0 ASPM L0s L1, ASPM disabled
    00:1c.3 bridge to [bus 05] ASPM L0s L1, ASPM Disabled
      05:00.0 ASPM L0s L1, ASPM Disabled
    00:1c.4 bridge to [bus 06] ASPM L0s L1, ASPM Disabled
      06:00.0 ASPM L0s L1, ASPM Disabled
    00:1c.7 bridge to [bus 09] ASPM L0s L1, ASPM Disabled
      09:00.0 ASPM L0s L1, ASPM Disabled

You also have this bridge where lspci says "ASPM unknown":

    00:01.0 bridge to [bus 01] ASPM unknown

Just out of curiosity, can you collect the output of "lspci -xxxs
00:01.0"?  There are only two bits in the Link Capabilities ASPM
support field, and all four encodings are defined, so I'm curious
about why lspci says "unknown".

Bjorn

Re: enabling aspm on ati radeon

[Phillip Susi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 5/17/2013 1:31 PM, Bjorn Helgaas wrote:
> You also have this bridge where lspci says "ASPM unknown":
> 
> 00:01.0 bridge to [bus 01] ASPM unknown
> 
> Just out of curiosity, can you collect the output of "lspci -xxxs 
> 00:01.0"?  There are only two bits in the Link Capabilities ASPM 
> support field, and all four encodings are defined, so I'm curious 
> about why lspci says "unknown".

I'll get that as soon as I get home, but why would the ASPM status of
the root complex have anything to do with the rest of the devices?
Enabling ASPM on one link doesn't require it to be enabled on all
upstream links does it?


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJRlnE6AAoJEJrBOlT6nu75nWUIAI7iOcoOklDWhS1FriQJmXTm
X8OJ4VZB8VVPeIuzrW+I6sppUrxqpnnxvrJT7Trnh5SDwwJjwx3m6BVrZHphrlCN
armoZ5jLSuBS9mn4mpeAOfUS7uypAuyL4fEo6DYJa9fPZycL1LukjkuwZMRJ+bfw
sjXQ7soHC52r2wxWhDHZbOl9UEWtYHhBypoI+PpYMO00lQp3gGnJXedm2zHti+a3
2ZtIieydLw++nXDr18WxDbJ5h1rQi626rVip0IcO6VUhhNxcd8hzGFkRAc7vNodw
bjW6wp8f+sddfX5uHkh9zYzwaEblLgBkxqw93xzRdh1ujJHet4R/jL1NTLZ8EZY=
=W4Uc
-----END PGP SIGNATURE-----

Re: enabling aspm on ati radeon

[Bjorn Helgaas]

On Fri, May 17, 2013 at 12:04 PM, Phillip Susi <psusi@ubuntu.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 5/17/2013 1:31 PM, Bjorn Helgaas wrote:
>> You also have this bridge where lspci says "ASPM unknown":
>>
>> 00:01.0 bridge to [bus 01] ASPM unknown
>>
>> Just out of curiosity, can you collect the output of "lspci -xxxs
>> 00:01.0"?  There are only two bits in the Link Capabilities ASPM
>> support field, and all four encodings are defined, so I'm curious
>> about why lspci says "unknown".
>
> I'll get that as soon as I get home, but why would the ASPM status of
> the root complex have anything to do with the rest of the devices?
> Enabling ASPM on one link doesn't require it to be enabled on all
> upstream links does it?

Nope, that one shouldn't affect the other devices.  I was just curious.

I just sent a patch to make lspci decode all possible values there.
In the PCIe r2.1 spec, only 1 ("L0s") and 3 ("L0s and L1") were
defined.  The r3 spec added 0 ("not supported") and 2 ("L1"), but
lspci hadn't quite kept up.

Bjorn

Re: enabling aspm on ati radeon

[Phillip Susi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 5/17/2013 3:52 PM, Bjorn Helgaas wrote:
> I just sent a patch to make lspci decode all possible values
> there. In the PCIe r2.1 spec, only 1 ("L0s") and 3 ("L0s and L1")
> were defined.  The r3 spec added 0 ("not supported") and 2 ("L1"),
> but lspci hadn't quite kept up.

So do you still want the lspci -xxxs output, or was that just to
answer the question of why it said unknown instead of not supported?
Are we then back to no idea?

I notice a number of functions in drivers/pci/pcie/aspm.c that look
like they need used to configure aspm.  The radeon driver does not
appear to call any of them ( grep for aspm came up with nothing ).
Could that be why?  I can't find anything under Documentation/ about
what a driver needs to do to enable aspm.



-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJRlo6eAAoJEJrBOlT6nu75lxQIAMwrt4oDHt5MKiRfpPWIkPO4
DLoFtEax1CGy1Gqmhvw8QpzYw+daqWzYw01t2SDd+owLEHVdp8zzqW21TSld83yb
W9EmZH4NLIO9vsR/UqAHhmbbf9ooAwz+O26Vb7EarPtpNxkRHybDSPYVsv7Cs/R7
54gJOefq7SroiiSr31I0dQT8j57SBRjekJJMGsEh+Et2Cxz+z8CwynCYCoNjQdOQ
g89lyifTEdEy2DuXcRfeCcIH1mTNQazjsRu2/P6I+P/klpyS+b9HqMSUUwNUDwc/
eY1SWrbCt9/2XJ/B6ZxYK3Lmy+25m9fUNkQBsp3BqqXkUICHlE8zTm3DMG1rf6g=
=vIor
-----END PGP SIGNATURE-----

Re: enabling aspm on ati radeon

[Bjorn Helgaas]

On Fri, May 17, 2013 at 2:10 PM, Phillip Susi <psusi@ubuntu.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 5/17/2013 3:52 PM, Bjorn Helgaas wrote:
>> I just sent a patch to make lspci decode all possible values
>> there. In the PCIe r2.1 spec, only 1 ("L0s") and 3 ("L0s and L1")
>> were defined.  The r3 spec added 0 ("not supported") and 2 ("L1"),
>> but lspci hadn't quite kept up.
>
> So do you still want the lspci -xxxs output, or was that just to
> answer the question of why it said unknown instead of not supported?

Don't worry about the lspci -xxx output.  I think the lspci patch
should resolve that, and it's not related to your original issue.

> Are we then back to no idea?
>
> I notice a number of functions in drivers/pci/pcie/aspm.c that look
> like they need used to configure aspm.  The radeon driver does not
> appear to call any of them ( grep for aspm came up with nothing ).
> Could that be why?  I can't find anything under Documentation/ about
> what a driver needs to do to enable aspm.

The driver does not need to do anything to enable ASPM.  I'm sure we
can figure out what's going on here, but I need to review some of the
patches in my queue, so I don't have time to grub around in ASPM at
the moment.

If you want to take a look yourself, I'd start by adding printks in
the pcie_aspm_init_link_state() path.  You can probably figure out
where things go wrong pretty quickly.

Oh, wait a minute...  You mentioned the radeon driver.  The Radeon
device in your system is at 01:00.0, and that is below the 00:01.0
bridge that said "ASPM unknown."  If the bridge leading to the Radeon
device doesn't support ASPM, then of course, we won't be able to turn
on ASPM for it.

That doesn't explain why we don't turn on ASPM for the *other* devices
(03:00.0, 05:00.0, 06:00.0, 09:00.0) though.

Bjorn

Re: enabling aspm on ati radeon

[Phillip Susi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 5/17/2013 4:24 PM, Bjorn Helgaas wrote:
> Oh, wait a minute...  You mentioned the radeon driver.  The Radeon 
> device in your system is at 01:00.0, and that is below the 00:01.0 
> bridge that said "ASPM unknown."  If the bridge leading to the
> Radeon device doesn't support ASPM, then of course, we won't be
> able to turn on ASPM for it.

That's what I was asking about before when I said surely the bridge
higher up doesn't have to have aspm enabled in order to enable it on
the downstream link?

> That doesn't explain why we don't turn on ASPM for the *other*
> devices (03:00.0, 05:00.0, 06:00.0, 09:00.0) though.

00:01.0 says it is the root complex, so wouldn't that make it upstream
of *everything*?  As I understand it, the root complex is built into
the CPU and has 16 lanes going to the video slot, and another 4 lanes
running at gen3 speed that Intel calls the DMI that goes to the PCH,
which has some internal functions, and bridges to another 20 pcie lanes.

Since the root complex is built into the cpu, I wouldn't think ASPM
could possibly make sense on it, so it showing that it doesn't support
ASPM makes sense to me.  That doesn't seem like it should preclude
using ASPM on the link down to the video card.


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJRlp0wAAoJEJrBOlT6nu7543QIAMfU7GnhweyxmunYl8fKq0B9
KCX3PSQaXXmknKxKtSDPtI3BhuTN+5g9KjXX+tDngETRPwq/5ZRXhIAcNjQCpv0d
K5Io/1vh652Aa0eVm1Yae+ejBWB/hWSu/AM/U0L1wyWGqat53QwfSHlMcQOFeuxq
oHoRrE5JtmcSbRbWtXFsOgRx5iXkgMzcwDxZ93Z4djhWjYGX+pGSk0DOTlttG/AP
RyT+oKYs8BmCytk/9/PFE0v/CtaAiNjEL42WAgv3nJMNuezG/CiBeUi7q9+vqKE8
1r+kGTbenKeEkboROWHXN/JCq3IsKCkff6rQC/s+oXBdORpCw46PcwpBno5kK+Q=
=sZdM
-----END PGP SIGNATURE-----

Re: enabling aspm on ati radeon

[Bjorn Helgaas]

On Fri, May 17, 2013 at 3:12 PM, Phillip Susi <psusi@ubuntu.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 5/17/2013 4:24 PM, Bjorn Helgaas wrote:
>> Oh, wait a minute...  You mentioned the radeon driver.  The Radeon
>> device in your system is at 01:00.0, and that is below the 00:01.0
>> bridge that said "ASPM unknown."  If the bridge leading to the
>> Radeon device doesn't support ASPM, then of course, we won't be
>> able to turn on ASPM for it.
>
> That's what I was asking about before when I said surely the bridge
> higher up doesn't have to have aspm enabled in order to enable it on
> the downstream link?

A link has two ends.  Both ends have to support ASPM in order for it
to work.  But ASPM on a particular link has no dependency on things
elsewhere in the hierarchy.

>> That doesn't explain why we don't turn on ASPM for the *other*
>> devices (03:00.0, 05:00.0, 06:00.0, 09:00.0) though.
>
> 00:01.0 says it is the root complex, so wouldn't that make it upstream
> of *everything*?  As I understand it, the root complex is built into
> the CPU and has 16 lanes going to the video slot, and another 4 lanes
> running at gen3 speed that Intel calls the DMI that goes to the PCH,
> which has some internal functions, and bridges to another 20 pcie lanes.
>
> Since the root complex is built into the cpu, I wouldn't think ASPM
> could possibly make sense on it, so it showing that it doesn't support
> ASPM makes sense to me.  That doesn't seem like it should preclude
> using ASPM on the link down to the video card.

The link between 00:01.0 and 01:00.0 apparently only supports ASPM on
one end (the downstream end), so ASPM won't work on it.

Bjorn

Re: enabling aspm on ati radeon

[Phillip Susi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 05/17/2013 05:28 PM, Bjorn Helgaas wrote:> A link has two ends.
Both ends have to support ASPM in order for it
> to work.  But ASPM on a particular link has no dependency on
> things elsewhere in the hierarchy.

Got you... I guess I was still thinking regular PCI and the naming
"root complex" made me think it was the whole root bridge, when it
really is that particular x16 pcie port that goes to the radeon.

> The link between 00:01.0 and 01:00.0 apparently only supports ASPM 
> on one end (the downstream end), so ASPM won't work on it.

I'll check the Intel spec sheet again, but I'm pretty sure it really
does support ASPM.  The CPU probably allows writes to the register,
and my bios must have configured it to say it doesn't support aspm.  I
guess I just need to wrangle setpci into trying to enable it despite
it not advertising support, or maybe see if I can get grub to poke the
capability register to make it say it supports it.

One question I have though is how to identify which port device
corresponds to which downstream device connected to it.  In other
words, how do you know 00:01.0 links to 01:00.0 ( other than seeing
that it's the x16 and knowing the radeon is plugged into the x16 slot ).


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iQEcBAEBAgAGBQJRlr8VAAoJEJrBOlT6nu75zL8IAImEt+ql6u1pqDSFPuDlLwSE
fbFWYYxZAzEGnzbAKGMUhueKHAa4rEnH2ztM8CYxdj3GVfSoTpwgcvz21L/fq/6L
z4OtChaw2cDiCevi95BR2xry2qBLnrHNq1qbD1yb832b5njCSr5SnTvUBUiH/4pN
twEAJ3HZp7Vblts4UMmn5sOT3cGd1Yr6sJp4Y068vPSGyx5Ixm5ROYEedN5tBnBl
zkeC+2vYXk0nIbMgJLWt4ZJRxXmX3/DXt8F1VL12NzlHpotuS5rPRpTR/el3sSty
J6HH/jlgSm+L3zuSPTicVAQBvYGcRhi/NbKDFPnZGGB8E7HsE9SIwNxzjzImyV0=
=X54u
-----END PGP SIGNATURE-----

Re: enabling aspm on ati radeon

[Bjorn Helgaas]

On Fri, May 17, 2013 at 5:36 PM, Phillip Susi <psusi@ubuntu.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 05/17/2013 05:28 PM, Bjorn Helgaas wrote:> A link has two ends.
> Both ends have to support ASPM in order for it
>> to work.  But ASPM on a particular link has no dependency on
>> things elsewhere in the hierarchy.
>
> Got you... I guess I was still thinking regular PCI and the naming
> "root complex" made me think it was the whole root bridge, when it
> really is that particular x16 pcie port that goes to the radeon.
>
>> The link between 00:01.0 and 01:00.0 apparently only supports ASPM
>> on one end (the downstream end), so ASPM won't work on it.
>
> I'll check the Intel spec sheet again, but I'm pretty sure it really
> does support ASPM.  The CPU probably allows writes to the register,
> and my bios must have configured it to say it doesn't support aspm.  I
> guess I just need to wrangle setpci into trying to enable it despite
> it not advertising support, or maybe see if I can get grub to poke the
> capability register to make it say it supports it.

setpci will write whatever you tell it to; it doesn't check any
constraints like "is ASPM supported?"  But the Link Capabilities
register is read-only per spec, so likely you won't be able to change
that field, at least not by writing it directly.

You can always try using setpci to set the ASPM enable bits in the
Link Control register on both the bridge and the device.  When
enabling, I think you're supposed to do the upstream end of the link
first, then the downstream end (and the reverse for disabling).

> One question I have though is how to identify which port device
> corresponds to which downstream device connected to it.  In other
> words, how do you know 00:01.0 links to 01:00.0 ( other than seeing
> that it's the x16 and knowing the radeon is plugged into the x16 slot ).

lspci tells you that:

00:01.0 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core
Processor Family PCI Express Root Port
        Bus: primary=00, secondary=01, subordinate=01, sec-latency=0

The "secondary=01" means the bridge leads to bus 01, which is where
the 01:00.0 device is.

Bjorn

Re: enabling aspm on ati radeon

[Phillip Susi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 05/17/2013 07:52 PM, Bjorn Helgaas wrote:
> setpci will write whatever you tell it to; it doesn't check any 
> constraints like "is ASPM supported?"  But the Link Capabilities 
> register is read-only per spec, so likely you won't be able to
> change that field, at least not by writing it directly.

I assume it is RO according to the PCIe spec ( buggers seem to only
allow members of pcisig to download it ), but I checked the Intel
specs for the chip ( Sandybridge ) and it's RW.  It looks like it
powers up as 00 and it's up to the bios to change it to advertise the
capability, and it seems Asus didn't bother with that.

> You can always try using setpci to set the ASPM enable bits in the 
> Link Control register on both the bridge and the device.  When 
> enabling, I think you're supposed to do the upstream end of the
> link first, then the downstream end (and the reverse for
> disabling).

One thing I don't see anywhere is a status bit showing whether it
actually *is* in L0s or L1 currently, so how to know if it's working?


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iQEcBAEBAgAGBQJRlsgbAAoJEJrBOlT6nu75mVEIAIsgXJjpqx5BTaocW3fJdq/D
6uqBiZxrIiwzCxHcsxyyq2gf2z1cwV/SExb8KCtjycZjXha4Y4awa1ba+1H8QnUe
YjLwRzsVSxlXUArwcUTrGQXsCYL5Q5IYejXAE0ki36Yfp5YvrKDLfuoKet76ROAY
cIGY7w202o9jfCVVoc2nskwXWCOuTgdfffD3Kjhg5eLnB/SxcCfbIWvxw7pNmlW1
bmzBNg3Mbs/UpGSFoSAOKl2/gzniMtjej3iXEuPy/gp/BjrT9ijExlaE5s4gdxom
12GgHMO/+rtvXE0yz9+zD0oL3A8dLUW9cc1+pX6kjiSz6j7V7guRicNP1Zw4TLU=
=VABF
-----END PGP SIGNATURE-----

Re: enabling aspm on ati radeon

[Bjorn Helgaas]

On Fri, May 17, 2013 at 6:15 PM, Phillip Susi <psusi@ubuntu.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 05/17/2013 07:52 PM, Bjorn Helgaas wrote:
>> setpci will write whatever you tell it to; it doesn't check any
>> constraints like "is ASPM supported?"  But the Link Capabilities
>> register is read-only per spec, so likely you won't be able to
>> change that field, at least not by writing it directly.
>
> I assume it is RO according to the PCIe spec ( buggers seem to only
> allow members of pcisig to download it ), but I checked the Intel
> specs for the chip ( Sandybridge ) and it's RW.  It looks like it
> powers up as 00 and it's up to the bios to change it to advertise the
> capability, and it seems Asus didn't bother with that.

Right; I meant RO per the PCIe spec.  If it's RW on your chip, setpci
should be able to write it.

>> You can always try using setpci to set the ASPM enable bits in the
>> Link Control register on both the bridge and the device.  When
>> enabling, I think you're supposed to do the upstream end of the
>> link first, then the downstream end (and the reverse for
>> disabling).
>
> One thing I don't see anywhere is a status bit showing whether it
> actually *is* in L0s or L1 currently, so how to know if it's working?

I don't think software can actually tell -- those transitions are
completely managed by hardware, and whenever the link is active, e.g.,
when you're accessing config space or transferring data, it would be
in L0.

If you can measure the power consumption, you might be able to see a
difference there.  I've seen people report that, but don't know how
they do it.  For example:
https://lkml.kernel.org/r/CANUX_P3F5YhbZX3WGU-j1AGpbXb_T9Bis2ErhvKkFMtDvzatVQ@mail.gmail.com

Bjorn

Re: enabling aspm on ati radeon

[Phillip Susi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 05/17/2013 08:40 PM, Bjorn Helgaas wrote:
> Right; I meant RO per the PCIe spec.  If it's RW on your chip,
> setpci should be able to write it.

I assume it would do no good to change it at this point; it would need
to already be set at the time the kernel reads it and decides whether
or not to enable it, so I'd have to have grub set it before loading
the kernel.

> If you can measure the power consumption, you might be able to see
> a difference there.  I've seen people report that, but don't know
> how they do it.  For example: 
> https://lkml.kernel.org/r/CANUX_P3F5YhbZX3WGU-j1AGpbXb_T9Bis2ErhvKkFMtDvzatVQ@mail.gmail.com

After
> 
enabling it on both ends ( despite the root port still claiming
it is not supported ) I still see ~121 watts according to the
Kill-A-Watt my system is plugged into.  Oh well.

Now that I double check myself, I realize that I misread the Intel
spec.  I was actually looking at the description for the link control
register instead of capabilities.  It uses the word "supported"
instead of "enabled" when describing the enable bits in the control
register, and they seem to skip right over describing the capability
register.  Trying to write to that has no effect, so it seems the CPU
is really telling me it doesn't support ASPM on that link, though it
is supposed to according to the feature description section of the
data sheet.  I am at a loss.


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iQEcBAEBAgAGBQJRltdlAAoJEJrBOlT6nu75O60IAJEEfQfoRgEP24/+5/SDIfnF
lp5rBUc0afHcvV10m/5FAN8nefVfuLn0I6Iq2sHAgKPj7VG032MYXIAddrLK0tDH
owFXqgsUTHemfsrTz+QCKQnJzZVu9iReyXRVovFF/j+jtRs9X45QD8tJHKSUKPMb
uEyy7WeAoxxT6W6cIZSWrRChUnvq3soAcUiPkGdCHJep9jM8vBjidrADENavpzl6
tSHy/jEh5Cehr5oBX8JXB3e2x4AaDj5AE2CPWq1/ttQd0FRcUWZEHOZEcL2gN3Sp
AGm5efe64Gdcl48yqP5WQuGM0K0dHXoXbRkWRNy4LOxP43gynUJWY4ulFqB06SM=
=MOhM
-----END PGP SIGNATURE-----

Re: enabling aspm on ati radeon

[Bjorn Helgaas]

On Fri, May 17, 2013 at 7:20 PM, Phillip Susi <psusi@ubuntu.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 05/17/2013 08:40 PM, Bjorn Helgaas wrote:
>> Right; I meant RO per the PCIe spec.  If it's RW on your chip,
>> setpci should be able to write it.
>
> I assume it would do no good to change it at this point; it would need
> to already be set at the time the kernel reads it and decides whether
> or not to enable it, so I'd have to have grub set it before loading
> the kernel.

Yeah, setting the capability bits after the kernel has booted wouldn't
change anything except what lspci reports.

>> If you can measure the power consumption, you might be able to see
>> a difference there.  I've seen people report that, but don't know
>> how they do it.  For example:
>> https://lkml.kernel.org/r/CANUX_P3F5YhbZX3WGU-j1AGpbXb_T9Bis2ErhvKkFMtDvzatVQ@mail.gmail.com
>
> After
>>
> enabling it on both ends ( despite the root port still claiming
> it is not supported ) I still see ~121 watts according to the
> Kill-A-Watt my system is plugged into.  Oh well.

ASPM only saves power when the PCIe link is idle.  I'm not a video
person, so I don't know how video usage translates into link usage,
and I don't know how much power it saves.  My guess is that the amount
is fairly small and might not be measurable with a Kill-A-Watt.

Bjorn

Re: enabling aspm on ati radeon

[Rafael J. Wysocki]

On Monday, May 20, 2013 07:34:44 AM Bjorn Helgaas wrote:
> On Fri, May 17, 2013 at 7:20 PM, Phillip Susi <psusi@ubuntu.com> wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > On 05/17/2013 08:40 PM, Bjorn Helgaas wrote:
> >> Right; I meant RO per the PCIe spec.  If it's RW on your chip,
> >> setpci should be able to write it.
> >
> > I assume it would do no good to change it at this point; it would need
> > to already be set at the time the kernel reads it and decides whether
> > or not to enable it, so I'd have to have grub set it before loading
> > the kernel.
> 
> Yeah, setting the capability bits after the kernel has booted wouldn't
> change anything except what lspci reports.
> 
> >> If you can measure the power consumption, you might be able to see
> >> a difference there.  I've seen people report that, but don't know
> >> how they do it.  For example:
> >> https://lkml.kernel.org/r/CANUX_P3F5YhbZX3WGU-j1AGpbXb_T9Bis2ErhvKkFMtDvzatVQ@mail.gmail.com
> >
> > After
> >>
> > enabling it on both ends ( despite the root port still claiming
> > it is not supported ) I still see ~121 watts according to the
> > Kill-A-Watt my system is plugged into.  Oh well.
> 
> ASPM only saves power when the PCIe link is idle.  I'm not a video
> person, so I don't know how video usage translates into link usage,
> and I don't know how much power it saves.  My guess is that the amount
> is fairly small and might not be measurable with a Kill-A-Watt.

I'm pretty sure it's measurable, at least in some cases.  Phoronix did quite
some fuss about a commit that caused ASPM to not work any more on some systems
some time ago, so he had to be able to measure it (and it was called a "major
power regression").

Thanks,
Rafael


-- 
I speak only for myself.
Rafael J. Wysocki, Intel Open Source Technology Center.

Re: enabling aspm on ati radeon

[Phillip Susi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 5/20/2013 4:01 PM, Rafael J. Wysocki wrote:
> I'm pretty sure it's measurable, at least in some cases.  Phoronix
> did quite some fuss about a commit that caused ASPM to not work any
> more on some systems some time ago, so he had to be able to measure
> it (and it was called a "major power regression").

That is why I was trying to get it enabled, but the "major regression"
seems to be caused by Radeons using the ASPM as a cue to enter low
power mode.  The energy saved by the bus itself going into the lower
power state I can't imagine is more than a few mW.


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJRm3c5AAoJEJrBOlT6nu75MkUH/3qGUjM3BmDFjCJ5kJUaMOCg
Ps3YFcr37JPumLZNLAwAHeL+eZH1v8G1SkDKPFMfPjIMCY4II4xxthFutV6bslQv
PR5w5jjAg/qk0YXEq8CddhdZkVkJw8nKBZiN6jFFcn/c1UgFN39FHCzUTa3sMG2X
+rEM4RHzrtcIJm46KXkI162Dzcnj7UaGTFYUzwMCxzy3XHn8xKMS1tfsM1TW3sqS
ptcqbsy1rObJewAhdnQZ0pzRgLC7zKWkGlLo00JdFW7wChjsVbM2db++FlpkCT0W
sHSSEEYLFqef+fU9FHptJklMx31nF8QHDGh3S277YqTuIaanMufo+gx4vHJC3tM=
=tcAW
-----END PGP SIGNATURE-----

Re: enabling aspm on ati radeon

[Sebastian]

I read this thread lately and added the force option to my linux-3.9.2.

While most devices enabled ASPM, onboard Ethernet doesnt.
Board is an ASRock 760GM-GS3 and Ethernet chip seems to be RTL8111E.
According to google it looks like this chip isnt able to ASPM, also 
LnkCap shows support, if I interpret LnkCap right.
Back to topic, my Radeon 5450 Evergreen got ASPM enabled.

01:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI 
Manhattan [Mobility Radeon HD 5430 Series] (prog-if 00 [VGA controller])
                 LnkCap: Port #0, Speed 5GT/s, Width x16, ASPM L0s L1, 
Latency L0 <64ns, L1 <1us
                         ClockPM- Surprise- LLActRep- BwNot-
                 LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes Disabled- 
Retrain- CommClk+
                         ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
         Kernel driver in use: radeon

02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. 
RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 06)
                 LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, 
Latency L0 <512ns, L1 <64us
                         ClockPM+ Surprise- LLActRep- BwNot-
                 LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- 
CommClk+
         Kernel driver in use: r8169



Am 21.05.2013 15:31, schrieb Phillip Susi:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 5/20/2013 4:01 PM, Rafael J. Wysocki wrote:
>> I'm pretty sure it's measurable, at least in some cases.  Phoronix
>> did quite some fuss about a commit that caused ASPM to not work any
>> more on some systems some time ago, so he had to be able to measure
>> it (and it was called a "major power regression").
> That is why I was trying to get it enabled, but the "major regression"
> seems to be caused by Radeons using the ASPM as a cue to enter low
> power mode.  The energy saved by the bus itself going into the lower
> power state I can't imagine is more than a few mW.
>
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.17 (MingW32)
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQEcBAEBAgAGBQJRm3c5AAoJEJrBOlT6nu75MkUH/3qGUjM3BmDFjCJ5kJUaMOCg
> Ps3YFcr37JPumLZNLAwAHeL+eZH1v8G1SkDKPFMfPjIMCY4II4xxthFutV6bslQv
> PR5w5jjAg/qk0YXEq8CddhdZkVkJw8nKBZiN6jFFcn/c1UgFN39FHCzUTa3sMG2X
> +rEM4RHzrtcIJm46KXkI162Dzcnj7UaGTFYUzwMCxzy3XHn8xKMS1tfsM1TW3sqS
> ptcqbsy1rObJewAhdnQZ0pzRgLC7zKWkGlLo00JdFW7wChjsVbM2db++FlpkCT0W
> sHSSEEYLFqef+fU9FHptJklMx31nF8QHDGh3S277YqTuIaanMufo+gx4vHJC3tM=
> =tcAW
> -----END PGP SIGNATURE-----
> --
> To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: enabling aspm on ati radeon

[Phillip Susi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

According to the PCI express standard, the Link Capabilities register
lies at offset 0xC within the PCI Express root port capabilities
descriptor.  lspci decoded this and shows Unknown ASPM support,
because its value is zero, and this is why the kernel did not enable
ASPM in the Link Control register.  According to the Intel datasheets,
the offsets AC-AFh ( the structure itself starts at A0h ) are
reserved.  So astonishingly it seems that Intel does not know about
the Link Capabilities register!  On the other hand, the contents of
the register actually do seem to be mostly correct, despite what their
datasheet claims.  The one thing wrong with it is that it claims no
ASPM support, despite Intel claiming that it is supported.

For reference, the Intel datasheet I have been referring to is
document 324642-003: 2nd Generation Intel® CoreTM Processor Family
Desktop, Intel® Pentium® Processor Family Desktop, and Intel® Celeron®
Processor Family Desktop Datasheet, Volume 2, table 2-8 on page 84.


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJRn8Y/AAoJEJrBOlT6nu75P78IAMgL23DGmsBD+TKm3Ootr36E
5iG7NoPcvjYiP/yeHBvhXy5jw4ORfVFLEli82fFOTWAi1YL5mxRKctIiYzu9080N
T1YO34qE/QpXfu6PDYMuNQETteO7MElnM7SHpQQ1oG5pAjPjUB1Bjb54qm4fQt77
11q8W6w3KFegy6cZLqAMHO63gzqUL9G/Akn4BdGvjZyt2ouOZOBU6Siyu0wm+RZY
NUuoy+5wzaej9/T96XCkfHR1e5s32NdvkT9GUxhA8SONEOy892QvOuMWPWv/VHxr
YkVGkaGHXuLLyXNWvy1Tz31HqZlbFaDRw7VzfbW5iE7cMi9EsZxSfjSvSqPZ+/A=
=8hms
-----END PGP SIGNATURE-----
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: enabling aspm on ati radeon

[Phillip Susi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

According to the PCI express standard, the Link Capabilities register
lies at offset 0xC within the PCI Express root port capabilities
descriptor.  lspci decoded this and shows Unknown ASPM support,
because its value is zero, and this is why the kernel did not enable
ASPM in the Link Control register.  According to the Intel datasheets,
the offsets AC-AFh ( the structure itself starts at A0h ) are
reserved.  So astonishingly it seems that Intel does not know about
the Link Capabilities register!  On the other hand, the contents of
the register actually do seem to be mostly correct, despite what their
datasheet claims.  The one thing wrong with it is that it claims no
ASPM support, despite Intel claiming that it is supported.

For reference, the Intel datasheet I have been referring to is
document 324642-003: 2nd Generation Intel® CoreTM Processor Family
Desktop, Intel® Pentium® Processor Family Desktop, and Intel® Celeron®
Processor Family Desktop Datasheet, Volume 2, table 2-8 on page 84.


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJRn8Y/AAoJEJrBOlT6nu75P78IAMgL23DGmsBD+TKm3Ootr36E
5iG7NoPcvjYiP/yeHBvhXy5jw4ORfVFLEli82fFOTWAi1YL5mxRKctIiYzu9080N
T1YO34qE/QpXfu6PDYMuNQETteO7MElnM7SHpQQ1oG5pAjPjUB1Bjb54qm4fQt77
11q8W6w3KFegy6cZLqAMHO63gzqUL9G/Akn4BdGvjZyt2ouOZOBU6Siyu0wm+RZY
NUuoy+5wzaej9/T96XCkfHR1e5s32NdvkT9GUxhA8SONEOy892QvOuMWPWv/VHxr
YkVGkaGHXuLLyXNWvy1Tz31HqZlbFaDRw7VzfbW5iE7cMi9EsZxSfjSvSqPZ+/A=
=8hms
-----END PGP SIGNATURE-----