AMDGPU: RX 6500 XT: System reset when loading module

AMDGPU: RX 6500 XT: System reset when loading module

[Cal Peake]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 881 bytes --]

[ 2nd try after subbing to the ml :) ]

Hi,

I'm trying to get my shiny new Sapphire 6500 XT (PCI 1002:743f rev c1, 
subsys 1da2:e457) up and running, but upon loading the amdgpu driver, the 
system does a hard reset.

If I pass test=1, it will delay the reset -- that is, the framebuffer code 
will kick in and the resolution of the console will change, but after the 
testing is done, it resets.

I've tested kernels 5.15.19, 5.16.9, and 5.17.0-rc3+, all with the latest 
firmware, all with the same result. Trying this on Slackware64 15.0, but 
it's the same story on Fedora Rawhide, openSUSE Tumbleweed, Ubuntu 22.04, 
and EndeavourOS.

Attached is the gzipped kernel output with test=1 passed and DRM debugging 
maxed out.

I've tried a bunch of module options and flipped a bunch of BIOS options 
as well, all to no avail. So, any help is appreciated. Thanks!

-- 
Cal Peake

[-- Attachment #2: kernel.log.gz --]
[-- Type: APPLICATION/octet-stream, Size: 39962 bytes --]

Re: AMDGPU: RX 6500 XT: System reset when loading module

[Christian König]

Hi Cal,

Am 14.02.22 um 00:58 schrieb Cal Peake:
> [ 2nd try after subbing to the ml :) ]
>
> Hi,
>
> I'm trying to get my shiny new Sapphire 6500 XT (PCI 1002:743f rev c1,
> subsys 1da2:e457) up and running, but upon loading the amdgpu driver, the
> system does a hard reset.

well that sounds strongly like some hardware problem. What power supply 
do you have in that system?

 From your dmesg it looks like the kernel driver actually initializes 
fines. So my educated guess is that as soon as the desktop tries to draw 
we suck to much power and go into brown out reset.

>
> If I pass test=1, it will delay the reset -- that is, the framebuffer code
> will kick in and the resolution of the console will change, but after the
> testing is done, it resets.
>
> I've tested kernels 5.15.19, 5.16.9, and 5.17.0-rc3+, all with the latest
> firmware, all with the same result. Trying this on Slackware64 15.0, but
> it's the same story on Fedora Rawhide, openSUSE Tumbleweed, Ubuntu 22.04,
> and EndeavourOS.

Just to make sure that the hardware combination works fine: Does it 
works on windows?

Thanks,
Christian.

>
> Attached is the gzipped kernel output with test=1 passed and DRM debugging
> maxed out.
>
> I've tried a bunch of module options and flipped a bunch of BIOS options
> as well, all to no avail. So, any help is appreciated. Thanks!
>

Re: AMDGPU: RX 6500 XT: System reset when loading module

[Cal Peake]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1248 bytes --]

Hi Christian,

Thanks for the reply.

On Mon, 14 Feb 2022, Christian König wrote:

> well that sounds strongly like some hardware problem. What power supply do you
> have in that system?
> 

Right now, I've got it hooked up to a Corsair VX550W. It's on a test bench 
so the only things hanging off the mobo at the moment are the GPU and a 
pair of NVMe drives.

I'm also testing this system with a RTX 2060 Super and I have no problems 
with that card, even under heavy load.

> From your dmesg it looks like the kernel driver actually initializes fines. So
> my educated guess is that as soon as the desktop tries to draw we suck to much
> power and go into brown out reset.
> 

I'm not even trying for a desktop yet, it's currently booting to runlevel 3 :-)

As mentioned above, the 2060 does just fine and its power needs are 
significantly higher than that of the 6500.

It's possible that it's something with my particular motherboard. Michael 
Larabel over at Phoronix did some performance testing[1] with this same 
model card running on Ubuntu 22.04, seemingly without problem.

Do you have any suggestions for further troubleshooting?

Thanks,

-- 
Cal Peake

[1] https://www.phoronix.com/scan.php?page=article&item=radeon-rx6500xt-linux

Re: AMDGPU: RX 6500 XT: System reset when loading module

[Christian König]

Hi Cal,

Am 14.02.22 um 18:45 schrieb Cal Peake:
> Hi Christian,
>
> Thanks for the reply.
>
> On Mon, 14 Feb 2022, Christian König wrote:
>
>> well that sounds strongly like some hardware problem. What power supply do you
>> have in that system?
>>
> Right now, I've got it hooked up to a Corsair VX550W. It's on a test bench
> so the only things hanging off the mobo at the moment are the GPU and a
> pair of NVMe drives.
>
> I'm also testing this system with a RTX 2060 Super and I have no problems
> with that card, even under heavy load.

Ok, at least it doesn't sounds like it is the power supply.

>>  From your dmesg it looks like the kernel driver actually initializes fines. So
>> my educated guess is that as soon as the desktop tries to draw we suck to much
>> power and go into brown out reset.
>>
> I'm not even trying for a desktop yet, it's currently booting to runlevel 3 :-)
>
> As mentioned above, the 2060 does just fine and its power needs are
> significantly higher than that of the 6500.
>
> It's possible that it's something with my particular motherboard. Michael
> Larabel over at Phoronix did some performance testing[1] with this same
> model card running on Ubuntu 22.04, seemingly without problem.
>
> Do you have any suggestions for further troubleshooting?

Unfortunately not of hand. You could try to play around with the power 
flags, but from your description it sounds like you already did that.

Only other option I see is to try a different motherboard and/or Windows 
with that hardware. If that also doesn't work there must be some 
hardware fault on the RX6500.

Christian.

>
> Thanks,
>

Re: AMDGPU: RX 6500 XT: System reset when loading module

[Cal Peake]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 966 bytes --]

On Thu, 17 Feb 2022, Christian König wrote:

> Ok, at least it doesn't sounds like it is the power supply.

I was able to procure an 800W PSU just to give it a try, and at first it 
seemed to make a difference. The drivers would fully load, but after a 
short amount of time things would become unstable and the system would 
crash or reset. Very bizarre...

> Unfortunately not of hand. You could try to play around with the power flags,
> but from your description it sounds like you already did that.
> 
> Only other option I see is to try a different motherboard and/or Windows with
> that hardware. If that also doesn't work there must be some hardware fault on
> the RX6500.

That was my thinking too. I got the idea to try Windows after I did the 
PSU swap and, like Linux, it was okay at first, but then started crashing 
and resetting. Either the card is defective or it really doesn't like my 
motherboard.

Thanks very much for your help!

-- 
Cal Peake

Re: AMDGPU: RX 6500 XT: System reset when loading module

[Christian König]

Am 17.02.22 um 19:10 schrieb Cal Peake:
> On Thu, 17 Feb 2022, Christian König wrote:
>
>> Ok, at least it doesn't sounds like it is the power supply.
> I was able to procure an 800W PSU just to give it a try, and at first it
> seemed to make a difference. The drivers would fully load, but after a
> short amount of time things would become unstable and the system would
> crash or reset. Very bizarre...

Maybe check if any of the capacitors on the motherboard or RX6500 looks 
damaged. Here are some examples what they normally look like: 
https://en.wikipedia.org/wiki/Capacitor

When capacitors are damaged they usually look like a balloon instead of 
a cylinder because electrolyte liquid inside started to boil because of 
a shortcut.

Capacitors are used to suppress voltage and spikes and valleys. When 
they are damaged the electronics might still work, but becomes unreliable.

>> Unfortunately not of hand. You could try to play around with the power flags,
>> but from your description it sounds like you already did that.
>>
>> Only other option I see is to try a different motherboard and/or Windows with
>> that hardware. If that also doesn't work there must be some hardware fault on
>> the RX6500.
> That was my thinking too. I got the idea to try Windows after I did the
> PSU swap and, like Linux, it was okay at first, but then started crashing
> and resetting. Either the card is defective or it really doesn't like my
> motherboard.

That's a really strong evidence that you have a hardware problem here.

BTW: Where do you got that hardware from? Is it used or a new one? Was 
it maybe used for mining?

Regards,
Christian.

>
> Thanks very much for your help!
>

Re: AMDGPU: RX 6500 XT: System reset when loading module

[Cal Peake]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1318 bytes --]

On Fri, 18 Feb 2022, Christian König wrote:

> That's a really strong evidence that you have a hardware problem here.
> 
> BTW: Where do you got that hardware from? Is it used or a new one? Was it
> maybe used for mining?
> 

Nah, this was a brand new card I bought on launch day from a reputable 
brick-and-mortar.

Since I was getting to the end of my 30-day return window, I decided to 
just take the card back and exchange it for an RX 560 -- the only other 
card comparable in price that was also in stock (thanks miners/scalpers!).

When I got that card home and tried it out, it had the same problem. From 
there, I decided to take a break and try a different approach. First thing 
I wanted to try was stepping through the different BIOS revisions 
available for my motherboard to see if that made any difference.

That's when I saw ASRock had a beta for the next BIOS release available 
for testing. Flashed it, configured the BIOS to my liking, and incredibly, 
no resets or crashes.

I've been under the weather for several days so I'm just now getting back 
to testing things again, but so far, everything seems to be good.

I do wish I could have gotten to this point a little sooner. Exchanging my 
overpriced brand new GPU for an overpriced 5 year old GPU is a little 
stomach turning :-/

-- 
Cal Peake

Re: AMDGPU: RX 6500 XT: System reset when loading module

[Christian König]

Am 23.02.22 um 23:59 schrieb Cal Peake:
> On Fri, 18 Feb 2022, Christian König wrote:
>
>> That's a really strong evidence that you have a hardware problem here.
>>
>> BTW: Where do you got that hardware from? Is it used or a new one? Was it
>> maybe used for mining?
>>
> Nah, this was a brand new card I bought on launch day from a reputable
> brick-and-mortar.
>
> Since I was getting to the end of my 30-day return window, I decided to
> just take the card back and exchange it for an RX 560 -- the only other
> card comparable in price that was also in stock (thanks miners/scalpers!).
>
> When I got that card home and tried it out, it had the same problem. From
> there, I decided to take a break and try a different approach. First thing
> I wanted to try was stepping through the different BIOS revisions
> available for my motherboard to see if that made any difference.
>
> That's when I saw ASRock had a beta for the next BIOS release available
> for testing. Flashed it, configured the BIOS to my liking, and incredibly,
> no resets or crashes.
>
> I've been under the weather for several days so I'm just now getting back
> to testing things again, but so far, everything seems to be good.
>
> I do wish I could have gotten to this point a little sooner. Exchanging my
> overpriced brand new GPU for an overpriced 5 year old GPU is a little
> stomach turning :-/

Well really sorry for that, but there isn't much I can help here.

Interesting to know that it turned out to be the motherboard, going to 
keep that in mind if somebody else is having similar problems.

Regards,
Christian.

Re: AMDGPU: RX 6500 XT: System reset when loading module

[Cal Peake]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 923 bytes --]

On Thu, 24 Feb 2022, Christian König wrote:

> Interesting to know that it turned out to be the motherboard, going to keep
> that in mind if somebody else is having similar problems.

Looks like I might have spoke too soon, getting random resets again. I 
have no idea why my nvidia card runs perfectly fine, but two completely 
different AMD GPUs both have the same problem, or why that problem comes 
and goes at random.

I'm gonna have to do a deeper dive into this until I figure it out (or 
until Intel drops their dGPUs and I can get a modern, OSS-friendly GPU 
that's stable on my system lol).

I'll let you know if I come up with anything more definitive, Christian.

Thanks!

-- 
Cal Peake

P.S. To your earlier point about capacitors: Between the mobo and the GPU, 
the only electrolytic caps are for the audio hardware. Everything else is 
solid polymer, so I'm thinking that that's probably not the culprit :-)

Re: AMDGPU: RX 6500 XT: System reset when loading module

[Alex Deucher]

On Thu, Feb 24, 2022 at 6:12 PM Cal Peake <cp@absolutedigital.net> wrote:
>
> On Thu, 24 Feb 2022, Christian König wrote:
>
> > Interesting to know that it turned out to be the motherboard, going to keep
> > that in mind if somebody else is having similar problems.
>
> Looks like I might have spoke too soon, getting random resets again. I
> have no idea why my nvidia card runs perfectly fine, but two completely
> different AMD GPUs both have the same problem, or why that problem comes
> and goes at random.
>
> I'm gonna have to do a deeper dive into this until I figure it out (or
> until Intel drops their dGPUs and I can get a modern, OSS-friendly GPU
> that's stable on my system lol).
>
> I'll let you know if I come up with anything more definitive, Christian.

Does setting amdgpu.aspm=0 help?

Alex

>
> Thanks!
>
> --
> Cal Peake
>
> P.S. To your earlier point about capacitors: Between the mobo and the GPU,
> the only electrolytic caps are for the audio hardware. Everything else is
> solid polymer, so I'm thinking that that's probably not the culprit :-)

Re: AMDGPU: RX 6500 XT: System reset when loading module

[Cal Peake]

On Fri, 25 Feb 2022, Alex Deucher wrote:

Hi Alex,

> Does setting amdgpu.aspm=0 help?
> 

No, I remember trying it with the 6500XT and I just tried it now with the 
RX560. No help on either.

-- 
Cal Peake

Re: AMDGPU: RX 6500 XT: System reset when loading module

[Cal Peake]

So... with me running out of things to test, I had to give consideration 
to the CPU and memory. Memory was easier to test. Tried a few different 
modules in different combinations to no change.

CPU, sadly, was another story. I was able to swap in a 3700X and it 
yielded positive results. I stressed the RX 560 across Linux and Windows 
yesterday and today and had no instability.

Doing a quick google search for "5950x system crash" returns a lot of 
similar stories. One poster on the AMD community forums[1] said they 
"fixed" things by downgrading from a 5950X to a 3950X (smh).

It's a little funny to me that AMD's own products won't play nice with 
each other. And it's utterly bizarre to me that its flagship desktop CPU 
has so many stability problems.

Just for kicks, I tried the HD 6350 I have in my current rig and it 
crashes the system too. Three different architectures on three different 
vendors' boards, all the same result.

So the RX 560 is going back and I guess I'll have to find a nice potato 
with good open source driver support to be my next GPU lol

-- 
Cal Peake

[1] https://community.amd.com/t5/processors/5950x-random-crashes-pulling-hair-out/td-p/431724

Re: AMDGPU: RX 6500 XT: System reset when loading module [SOLVED]

[Cal Peake]

Hi all,

I wanted to put a capper on this just in case anyone was interested, or in 
case any future people find this thread, because I did find a resolution.

Turns out the way to stop the system from crashing was to disable Global 
C-State Control in the BIOS.

Christian, Alex, you guys seem to have been on the right track in that it 
was something power related.

I haven't yet been able to figure out what Global C-State Control exactly 
does.

My best guess as to what was happening: once the GPU power management code 
was loaded and the GPU dropped into a very low power state, the CPU saw 
this and decided to match it, lowering its own power state to such a point 
that it hard resets the system. (Just a wild theory anyway :-)

If anyone knows what this feature really does, or has any better theories 
as to why it doesn't play nice with AMD GPUs, please do share!

Thanks and kind regards,

-- 
Cal Peake

Re: AMDGPU: RX 6500 XT: System reset when loading module [SOLVED]

[Paul Menzel]

Dear Cal,


Am 12.04.22 um 00:31 schrieb Cal Peake:

> I wanted to put a capper on this just in case anyone was interested, or in
> case any future people find this thread, because I did find a resolution.

Yes, that is very much appreciated.

> Turns out the way to stop the system from crashing was to disable Global
> C-State Control in the BIOS.
> 
> Christian, Alex, you guys seem to have been on the right track in that it
> was something power related.
> 
> I haven't yet been able to figure out what Global C-State Control exactly
> does.
> 
> My best guess as to what was happening: once the GPU power management code
> was loaded and the GPU dropped into a very low power state, the CPU saw
> this and decided to match it, lowering its own power state to such a point
> that it hard resets the system. (Just a wild theory anyway :-)
> 
> If anyone knows what this feature really does, or has any better theories
> as to why it doesn't play nice with AMD GPUs, please do share!

It might be related to bug report *Random Soft Lockup on new Ryzen 
build* [1], and the referenced issues there. It’d be great if you could 
post here or there a summary with your system details (especially the 
system and GPU firmware versions).

Me is still a little upset, how AMD has until now not been able to 
properly analyze and fix this (with the ODMs).


Kind regards,

Paul


[1]: https://bugzilla.kernel.org/show_bug.cgi?id=196683

AMDGPU: RX 6500 XT: System reset when loading module

[Cal Peake]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 841 bytes --]

Hi,

I'm trying to get my shiny new Sapphire 6500 XT (PCI 1002:743f rev c1, 
subsys 1da2:e457) up and running, but upon loading the amdgpu driver, the 
system does a hard reset.

If I pass test=1, it will delay the reset -- that is, the framebuffer code 
will kick in and the resolution of the console will change, but after the 
testing is done, it resets.

I've tested kernels 5.15.19, 5.16.9, and 5.17.0-rc3+, all with the latest 
firmware, all with the same result. Trying this on Slackware64 15.0, but 
it's the same story on Fedora Rawhide, openSUSE Tumbleweed, Ubuntu 22.04, 
and EndeavourOS.

Attached is the gzipped kernel output with test=1 passed and DRM debugging 
maxed out.

I've tried a bunch of module options and flipped a bunch of BIOS options 
as well, all to no avail. So, any help is appreciated. Thanks!

-- 
Cal Peake

[-- Attachment #2: kernel.log.gz --]
[-- Type: APPLICATION/octet-stream, Size: 39962 bytes --]