i915-related and general system freezes with specific kernel config // IOMMU

i915-related and general system freezes with specific kernel config // IOMMU

[Mihai Moldovan]

[-- Attachment #1.1: Type: text/plain, Size: 3648 bytes --]

Hi Daniel, David and everyone else,

I'm experiencing system freezes on a box using the vanilla 3.7.2 (actually down
to 3.2 or something) kernel with a custom configuration.

There are two problems:

  [*] related to i915 with modeset enabled; upon loading the kernel module with
modeset=1, the box will instantly freeze.
  [*] seemingly unrelated to i915; the box will randomly freeze without any
clear indication of why and moreover no apparent trigger.

After months, nay, years of being "locked" into 3.0.2 for the random freezes and
i915 problems, I started playing with the kernel again and out of sheer
desperation installed the current debian testing kernel, based on 3.2.35.

From what I could see, it worked fine... no more crashes, neither when loading
i915, nor randomly after some time (well, at least not for a day.)

This time out of frustration, I ripped the config file used by debian to build
the kernel out of its deb package and rebuilt my (almost[1]) vanilla 3.7.2
kernel with this configuration exactly, updated via the oldconfig target and
changed to include AHCI, RAID and SCSI drivers statically, so that I wouldn't
need some initramfs to boot my system ... and ... with this config, I am not
experiencing any i915 problems nor system freezes?!

I then tried to spot any "obvious" differences between the two config files and
to "approximate" my config file to the debian config.

Comparing the dmesg output from 3.7.2 built with the slightly modified debian
config to my 3.7.2 built with my config, I came across IOMMU entries which
differed. My kernel config enables Intel IOMMU by default, while the debian
config doesn't.

Looking up IOMMU stuff in Documentation/, I found out that IOMMU *may* have bugs
with the internal graphics card and there is an option called
intel_iommu=igfx_off to disable IOMMU remapping for the integrated graphics card...

I tried booting "my" kernel with intel_iommu=igfx_off and lo and behold, no more
crashes when loading i915 with modeset enabled! Yay... but anyway, that's
definitely a kernel bug.

Next, regarding the random freezes... so did the kernel booted with
intel_iommu=igfx_off. It seems the random freeze issue is kind of decoupled of
the graphics issue.

Testing further, I rebooted using iommu=off and intel_iommu=off. So far, I had
no random crashes, but the system uptime of XXXXREPLACEMEXXXX minutes is too
small to draw conclusions yet.

Anyway, booting with both options made my USB ports unusable. Also, my PCIe and
PCI WiFi cards stopped working. Seems like the kernel can't enumerate those
devices due to... guess what, DMA remapping errors!

Note that the debian-config kernel with CONFIG_INTEL_IOMMU=y and
CONFIG_INTEL_IOMMU_DEFAULT_ON=n did not produce such errors. Both my USB and
WiFi cards have been working.

Any idea why is that?

As I'm not sure who to CC exactly, I'm adding both the i915 and Intel IOMMU
maintainers Daniel and David.

I have included several files:

  [*] the "debianish" config file
  [*] my current config file (IOMMU still on by default)
  [*] dmesg for the kernel built with the "debianish" config file
  [*] dmesg for the kernel built with "my" config file, no IOMMU options passed
  [*] dmesg for the kernel built with "my" config file, intel_iommu=igfx_off passed
  [*] dmesg for the kernel built with "my" config file, iommu=off and
intel_iommu=off passed

Hope we can squash those bugs!

Best regards,



Mihai




[1] only one "external" patch applied to ath9k, totally unrelated to the rest of
the system, just changing regulatory stuff.


[-- Attachment #1.2: kernel-config-ionic.bz2 --]
[-- Type: application/x-bzip2, Size: 19502 bytes --]

[-- Attachment #1.3: kernel-config-debianish.bz2 --]
[-- Type: application/x-bzip2, Size: 31937 bytes --]

[-- Attachment #1.4: kern-new.log.bz2 --]
[-- Type: application/x-bzip2, Size: 17231 bytes --]

[-- Attachment #1.5: kern-new-iommu_off.log.bz2 --]
[-- Type: application/x-bzip2, Size: 16365 bytes --]

[-- Attachment #1.6: kern-new-igfx_off.log.bz2 --]
[-- Type: application/x-bzip2, Size: 17252 bytes --]

[-- Attachment #1.7: kern-debian.log.bz2 --]
[-- Type: application/x-bzip2, Size: 15463 bytes --]

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4506 bytes --]

Re: i915-related and general system freezes with specific kernel config // IOMMU

[Mihai Moldovan]

[-- Attachment #1: Type: text/plain, Size: 328 bytes --]

* On 19.01.2013 12:48 AM, Mihai Moldovan wrote:
> Testing further, I rebooted using iommu=off and intel_iommu=off. So far, I had
> no random crashes, but the system uptime of XXXXREPLACEMEXXXX minutes is too
> small to draw conclusions yet.

And by "XXXXREPLACEMEXXXX", I meant 50 (minutes). That's embarrassing, sorry.


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4506 bytes --]

Re: i915-related and general system freezes with specific kernel config // IOMMU

[Daniel Vetter]

Hi Mihai,

You have a gen4.5 chipset which is known to be utterly broken for
IOMMU+intel gpu. Looks like a few distros started enabling IOMMU by
default (fc18 has similar issues) and we've never added the proper
quirks. See https://bugzilla.kernel.org/show_bug.cgi?id=51921 for a
proposed patch to fix this (i.e. automatically set
intel_iommu=igfx_off for affected platfroms). Testing highly welcome.

Cheers, Daniel

On Sat, Jan 19, 2013 at 12:48 AM, Mihai Moldovan <ionic@ionic.de> wrote:
> Hi Daniel, David and everyone else,
>
> I'm experiencing system freezes on a box using the vanilla 3.7.2 (actually down
> to 3.2 or something) kernel with a custom configuration.
>
> There are two problems:
>
>   [*] related to i915 with modeset enabled; upon loading the kernel module with
> modeset=1, the box will instantly freeze.
>   [*] seemingly unrelated to i915; the box will randomly freeze without any
> clear indication of why and moreover no apparent trigger.
>
> After months, nay, years of being "locked" into 3.0.2 for the random freezes and
> i915 problems, I started playing with the kernel again and out of sheer
> desperation installed the current debian testing kernel, based on 3.2.35.
>
> From what I could see, it worked fine... no more crashes, neither when loading
> i915, nor randomly after some time (well, at least not for a day.)
>
> This time out of frustration, I ripped the config file used by debian to build
> the kernel out of its deb package and rebuilt my (almost[1]) vanilla 3.7.2
> kernel with this configuration exactly, updated via the oldconfig target and
> changed to include AHCI, RAID and SCSI drivers statically, so that I wouldn't
> need some initramfs to boot my system ... and ... with this config, I am not
> experiencing any i915 problems nor system freezes?!
>
> I then tried to spot any "obvious" differences between the two config files and
> to "approximate" my config file to the debian config.
>
> Comparing the dmesg output from 3.7.2 built with the slightly modified debian
> config to my 3.7.2 built with my config, I came across IOMMU entries which
> differed. My kernel config enables Intel IOMMU by default, while the debian
> config doesn't.
>
> Looking up IOMMU stuff in Documentation/, I found out that IOMMU *may* have bugs
> with the internal graphics card and there is an option called
> intel_iommu=igfx_off to disable IOMMU remapping for the integrated graphics card...
>
> I tried booting "my" kernel with intel_iommu=igfx_off and lo and behold, no more
> crashes when loading i915 with modeset enabled! Yay... but anyway, that's
> definitely a kernel bug.
>
> Next, regarding the random freezes... so did the kernel booted with
> intel_iommu=igfx_off. It seems the random freeze issue is kind of decoupled of
> the graphics issue.
>
> Testing further, I rebooted using iommu=off and intel_iommu=off. So far, I had
> no random crashes, but the system uptime of XXXXREPLACEMEXXXX minutes is too
> small to draw conclusions yet.
>
> Anyway, booting with both options made my USB ports unusable. Also, my PCIe and
> PCI WiFi cards stopped working. Seems like the kernel can't enumerate those
> devices due to... guess what, DMA remapping errors!
>
> Note that the debian-config kernel with CONFIG_INTEL_IOMMU=y and
> CONFIG_INTEL_IOMMU_DEFAULT_ON=n did not produce such errors. Both my USB and
> WiFi cards have been working.
>
> Any idea why is that?
>
> As I'm not sure who to CC exactly, I'm adding both the i915 and Intel IOMMU
> maintainers Daniel and David.
>
> I have included several files:
>
>   [*] the "debianish" config file
>   [*] my current config file (IOMMU still on by default)
>   [*] dmesg for the kernel built with the "debianish" config file
>   [*] dmesg for the kernel built with "my" config file, no IOMMU options passed
>   [*] dmesg for the kernel built with "my" config file, intel_iommu=igfx_off passed
>   [*] dmesg for the kernel built with "my" config file, iommu=off and
> intel_iommu=off passed
>
> Hope we can squash those bugs!
>
> Best regards,
>
>
>
> Mihai
>
>
>
>
> [1] only one "external" patch applied to ath9k, totally unrelated to the rest of
> the system, just changing regulatory stuff.
>



-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

Re: i915-related and general system freezes with specific kernel config // IOMMU

[Mihai Moldovan]

[-- Attachment #1: Type: text/plain, Size: 1327 bytes --]

* On 19.01.2013 02:27 PM, Daniel Vetter wrote:
> You have a gen4.5 chipset which is known to be utterly broken for
> IOMMU+intel gpu.

Nice description for what I'm seeing. ;)

After some more hours of uptime I'm inclined to say, that "intel_iommu=off
iommu=off" fixes my random freezes as well.
Alas, the USB and PCI(e) problems are still around, but I could test recompiling
3.7.2 with Intel IOMMU turned off completely in the kernel config.
Interestingly, my 3.0.2 kernel which worked fine for so long doesn't even *have*
support for VT-d/Intel IOMMU. This could explain why I wasn't bit by those
problems on all previous versions.


> [...] and we've never added the proper
> quirks. See https://bugzilla.kernel.org/show_bug.cgi?id=51921 for a
> proposed patch to fix this (i.e. automatically set
> intel_iommu=igfx_off for affected platfroms). Testing highly welcome.

From a quick glance, I don't think this patch will work as-is, my PCI ID 2e12 is
missing.
I'll add it to the relevant section.

But even if it worked, I'd still have the "box freezes randomly" issue (mostly
within 5 to 60 minutes of uptime). :(
The only way to get rid of this is disabling Intel IOMMU as a whole via kernel
parameters intel_iommu=off iommu=off.

Anyway, I'll give it a try.

Best regards,



Mihai


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4506 bytes --]

Re: i915-related and general system freezes with specific kernel config // IOMMU

[Daniel Vetter]

On Sat, Jan 19, 2013 at 5:13 PM, Mihai Moldovan <ionic@ionic.de> wrote:
>> [...] and we've never added the proper
>> quirks. See https://bugzilla.kernel.org/show_bug.cgi?id=51921 for a
>> proposed patch to fix this (i.e. automatically set
>> intel_iommu=igfx_off for affected platfroms). Testing highly welcome.
>
> From a quick glance, I don't think this patch will work as-is, my PCI ID 2e12 is
> missing.
> I'll add it to the relevant section.

The quirk matches your pci host bridge, which should have id 2e10, not
the gfx, which has id 2e12.

> But even if it worked, I'd still have the "box freezes randomly" issue (mostly
> within 5 to 60 minutes of uptime). :(
> The only way to get rid of this is disabling Intel IOMMU as a whole via kernel
> parameters intel_iommu=off iommu=off.

Hm, can you try enabling the related iommu quirk:

diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c
index b9d0911..e834395 100644
--- a/drivers/iommu/intel-iommu.c
+++ b/drivers/iommu/intel-iommu.c
@@ -4251,6 +4251,7 @@ static void quirk_iommu_rwbf(struct pci_dev *dev)
 }

 DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x2a40, quirk_iommu_rwbf);
+DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x2e10, quirk_iommu_rwbf);

 #define GGC 0x52
 #define GGC_MEMORY_SIZE_MASK   (0xf << 8)

Cheers, Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

Re: i915-related and general system freezes with specific kernel config // IOMMU

[Mihai Moldovan]

[-- Attachment #1: Type: text/plain, Size: 1470 bytes --]

* On 19.01.2013 05:13 PM, Mihai Moldovan wrote:
> * On 19.01.2013 02:27 PM, Daniel Vetter wrote:
>> You have a gen4.5 chipset which is known to be utterly broken for
>> IOMMU+intel gpu.
> Nice description for what I'm seeing. ;)
>
> After some more hours of uptime I'm inclined to say, that "intel_iommu=off
> iommu=off" fixes my random freezes as well.
> Alas, the USB and PCI(e) problems are still around, but I could test recompiling
> 3.7.2 with Intel IOMMU turned off completely in the kernel config.
> Interestingly, my 3.0.2 kernel which worked fine for so long doesn't even *have*
> support for VT-d/Intel IOMMU. This could explain why I wasn't bit by those
> problems on all previous versions.
>
>
>> [...] and we've never added the proper
>> quirks. See https://bugzilla.kernel.org/show_bug.cgi?id=51921 for a
>> proposed patch to fix this (i.e. automatically set
>> intel_iommu=igfx_off for affected platfroms). Testing highly welcome.
> From a quick glance, I don't think this patch will work as-is, my PCI ID 2e12 is
> missing.
> [...]

Which of course will work, as 2e10 is my DRAM controller as reported by lspci,
sorry.

But, shouldn't the
"DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x2eXX, quirk_iommu_rwbf);"
calls be rather
" DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x2e00, quirk_iommu_g4x_gfx);"
?

The current patch errors out on my while compiling as quirk_iommu_rwbf is not
yet defined at that place.


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4506 bytes --]

Re: i915-related and general system freezes with specific kernel config // IOMMU

[Daniel Vetter]

On Sat, Jan 19, 2013 at 5:26 PM, Mihai Moldovan <ionic@ionic.de> wrote:
> The current patch errors out on my while compiling as quirk_iommu_rwbf is not
> yet defined at that place.

Oops, attached an old patch, updated one should work better.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

Re: i915-related and general system freezes with specific kernel config // IOMMU

[Mihai Moldovan]

[-- Attachment #1: Type: text/plain, Size: 731 bytes --]

Hi Daniel,

the patch does work, i.e., it turns off DMAR for the graphics card and
alleviates the freezes when loading i915/kms.

However, still seeing random machine freezes with it (being consistent with the
behavior I've experienced with intel_iommu=igfx_off).

The patch + forcing RWBF is working, too. Interestingly, this version didn't
randomly freeze yet, after more than 5 hours of uptime! I'll leave the box
running until tomorrow to make sure I did stick around long enough.

All those tested kernels were able to handle USB and PCI(e) devices.

I still have to test turning off IOMMU in general and Intel IOMMU specifically.
Will probably do this tomorrow.

Thank you so far! :)



Mihai






[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4506 bytes --]

Re: i915-related and general system freezes with specific kernel config // IOMMU

[Daniel Vetter]

On Sun, Jan 20, 2013 at 10:52 PM, Mihai Moldovan <ionic@ionic.de> wrote:
> the patch does work, i.e., it turns off DMAR for the graphics card and
> alleviates the freezes when loading i915/kms.
>
> However, still seeing random machine freezes with it (being consistent with the
> behavior I've experienced with intel_iommu=igfx_off).

Thanks for testing, I've just submitted the patch for review. It
should included in a -fixes tree soon and the get backported to stable
kernels.

> The patch + forcing RWBF is working, too. Interestingly, this version didn't
> randomly freeze yet, after more than 5 hours of uptime! I'll leave the box
> running until tomorrow to make sure I did stick around long enough.
>
> All those tested kernels were able to handle USB and PCI(e) devices.
>
> I still have to test turning off IOMMU in general and Intel IOMMU specifically.
> Will probably do this tomorrow.

Please let me know when this works solidly for you, so that I can put
it into a real patch and also submit it for inclusion.

Thanks, Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

Re: i915-related and general system freezes with specific kernel config // IOMMU

[Mihai Moldovan]

[-- Attachment #1: Type: text/plain, Size: 900 bytes --]

* On 20.01.2013 11:49 PM, Daniel Vetter wrote:
> Thanks for testing, I've just submitted the patch for review. It
> should included in a -fixes tree soon and the get backported to
> stable kernels.

Thanks. :)


> Please let me know when this works solidly for you, so that I can
> put it into a real patch and also submit it for inclusion.

No freeze for >24h, I guess we can conclude the quirk does indeed
fix the random freeze issue as well. :)
I'm all for inclusion.


I'm also currently testing a kernel without the Intel IOMMU feature. This seems
to work, too, but also disables Intel TXT and VT-d...
At least not seeing USB and PCI(e) issues. I'll leave the box running for some
more and will afterwards disable IOMMU as a whole to see if I hit USB and PCI(e)
issues again with that combination.

Best regards,



Mihai

[resending to include all previous CC's]


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4506 bytes --]

Re: i915-related and general system freezes with specific kernel config // IOMMU

[Mihai Moldovan]

[-- Attachment #1.1: Type: text/plain, Size: 1178 bytes --]

* On 21.01.2013 07:11 PM, Mihai Moldovan wrote:
> I'm also currently testing a kernel without the Intel IOMMU feature
> [CONFIG_INTEL_IOMMU=n, but CONFIG_IOMMU_SUPPORT=y]. [...] At least
> not seeing USB and PCI(e) issues. I'll leave the box running for some
> more [time] [...]

No freezes for >22h, seems to be fine.


> [...] and will afterwards disable IOMMU as a whole to see if I hit
> USB and PCI(e) issues again with that combination.

The systems seems to run stable with CONFIG_IOMMU_SUPPORT=n set, too. This is
expected.
However: unlike during earlier tests when I disabled IOMMU and Intel IOMMU via
kernel/boot parameters, I am not seeing any DMA mapping errors.

There seems to be a difference between disabling IOMMU/Intel IOMMU statically in
the kernel compared to disabling it via kernel parameter. Is this another bug?

I've attached both kernel ring buffer logs (minus the timings for easier diffing.)

  [*] kern-new-iommu_off.log.bz2 disables IOMMU and Intel IOMMU via boot parameter
  [*] kern-iommu_static_off.log.bz2 has CONFIG_IOMMU_SUPPORT=n set and any IOMMU
support statically disabled (also consequently DMAR)



Mihai


[-- Attachment #1.2: kern-new-iommu_off.log.bz2 --]
[-- Type: application/x-bzip2, Size: 16365 bytes --]

[-- Attachment #1.3: kern-iommu_static_off.log.bz2 --]
[-- Type: application/x-bzip2, Size: 16854 bytes --]

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4506 bytes --]

Re: i915-related and general system freezes with specific kernel config // IOMMU

[Daniel Vetter]

On Tue, Jan 22, 2013 at 7:15 PM, Mihai Moldovan <ionic@ionic.de> wrote:
> * On 21.01.2013 07:11 PM, Mihai Moldovan wrote:
>> I'm also currently testing a kernel without the Intel IOMMU feature
>> [CONFIG_INTEL_IOMMU=n, but CONFIG_IOMMU_SUPPORT=y]. [...] At least
>> not seeing USB and PCI(e) issues. I'll leave the box running for some
>> more [time] [...]
>
> No freezes for >22h, seems to be fine.
>
>
>> [...] and will afterwards disable IOMMU as a whole to see if I hit
>> USB and PCI(e) issues again with that combination.
>
> The systems seems to run stable with CONFIG_IOMMU_SUPPORT=n set, too. This is
> expected.
> However: unlike during earlier tests when I disabled IOMMU and Intel IOMMU via
> kernel/boot parameters, I am not seeing any DMA mapping errors.
>
> There seems to be a difference between disabling IOMMU/Intel IOMMU statically in
> the kernel compared to disabling it via kernel parameter. Is this another bug?

Behaviour should be the same for the actual dma access at the hw
layer, but if you disable things at compile-time at least drm/i915
selects different paths. We've recently killed those though since it's
not worth the complexity at all. But dunno why you still get dma
errors, that shouldn't be possible. Maybe David has an idea.

> I've attached both kernel ring buffer logs (minus the timings for easier diffing.)
>
>   [*] kern-new-iommu_off.log.bz2 disables IOMMU and Intel IOMMU via boot parameter
>   [*] kern-iommu_static_off.log.bz2 has CONFIG_IOMMU_SUPPORT=n set and any IOMMU
> support statically disabled (also consequently DMAR)

In any case I'll ping David about my 2nd quirk patch and whether
that's something which makes sense and could be merged. Thanks a lot
for all the testing you've done.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch