AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Andreas Hartmann]

Hello!

After long and heavy bisecting, I found this commit
"dm crypt: don't allocate pages for a partial request" [1] being the
cause of the ata errors and AMD-Vi IO_PAGE_FAULTs.

That's the bisect I did with Linus' repository
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/


git bisect start
# good: [3466b547e37b988723dc93465b7cb06b4b1f731f] Merge branches 'pnp',
'pm-cpuidle' and 'pm-cpufreq'
git bisect good 3466b547e37b988723dc93465b7cb06b4b1f731f
# bad: [cd50b70ccd5c87794ec28bfb87b7fba9961eb0ae] Merge tag
'pm+acpi-3.20-rc1-3' of
git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm
git bisect bad cd50b70ccd5c87794ec28bfb87b7fba9961eb0ae
# good: [27a22ee4c7d5839fd7e3e441c9d675c8a5c4c22c] Merge branch 'kbuild'
of git://git.kernel.org/pub/scm/linux/kernel/git/mmarek/kbuild
git bisect good 27a22ee4c7d5839fd7e3e441c9d675c8a5c4c22c
# good: [c189cb8ef62832f33b6cf757350a0270532a1ad8] Merge tag
'vfio-v3.20-rc1' of git://github.com/awilliam/linux-vfio
git bisect good c189cb8ef62832f33b6cf757350a0270532a1ad8
# good: [295324556c427d60b41668ab81a43f604533f456] Merge branch
'i2c/for-3.20' of git://git.kernel.org/pub/scm/linux/kernel/git/wsa/linux
git bisect good 295324556c427d60b41668ab81a43f604533f456
# good: [1acd2de5facd7fbea499aea64a3a3d0ec7bb9b51] Merge branch
'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/dtor/input
git bisect good 1acd2de5facd7fbea499aea64a3a3d0ec7bb9b51
# good: [fde9f50f80fe89a9115b4bfa773017272597d85d] target: Add sanity
checks for DPO/FUA bit usage

git bisect good fde9f50f80fe89a9115b4bfa773017272597d85d


# bad: [22aa66a3ee5b61e0f4a0bfeabcaa567861109ec3] dm snapshot: fix a
possible invalid memory access on unload


git bisect bad 22aa66a3ee5b61e0f4a0bfeabcaa567861109ec3


# bad: [7145c241a1bf2841952c3e297c4080b357b3e52d] dm crypt: avoid
deadlock in mempools


git bisect bad 7145c241a1bf2841952c3e297c4080b357b3e52d


# good: [37527b869207ad4c208b1e13967d69b8bba1fbf9] dm io: reject
unsupported DISCARD requests with EOPNOTSUPP


git bisect good 37527b869207ad4c208b1e13967d69b8bba1fbf9


# bad: [cf2f1abfbd0dba701f7f16ef619e4d2485de3366] dm crypt: don't
allocate pages for a partial request


git bisect bad cf2f1abfbd0dba701f7f16ef619e4d2485de3366


# good: [f3396c58fd8442850e759843457d78b6ec3a9589] dm crypt: use unbound
workqueue for request processing

git bisect good f3396c58fd8442850e759843457d78b6ec3a9589


# first bad commit: [cf2f1abfbd0dba701f7f16ef619e4d2485de3366] dm crypt:
don't allocate pages for a partial request


How can I verify (e.g. w/ a patch to Linux 4.0.9) if this patch is
really the culprit?

I'm heavily relying upon encryption:

There are 3 disks:
- One 240 GB SSD (crypted LVM, swap and boot partition)
- Two SATA rotational 3 TB disks (WD ST3000DM001-1CH166, encrypted raid
  /dev/md0, LVM)
- All in all 29 logical volumes with xfs as filesystem (besides swap
  and bootpartition - the latter is ext4).

The system is based on an AMD FX8350 processor (8 core) w/ 24GB RAM.
Motherboard is a Gigabyte GA-990XA-UD3. You can find a complete dmesg
output here [2].



I would be glad to get some assistance!


Thanks in advance,
kind regards,
Andreas Hartmann


[1]
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=cf2f1abfbd0dba701f7f16ef619e4d2485de3366

[2] http://article.gmane.org/gmane.linux.kernel.pci/43539

Re: AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Mike Snitzer]

On Tue, Jul 28 2015 at  1:40pm -0400,
Andreas Hartmann <andihartmann@01019freenet.de> wrote:

> Hello!
> 
> After long and heavy bisecting, I found this commit
> "dm crypt: don't allocate pages for a partial request" [1] being the
> cause of the ata errors and AMD-Vi IO_PAGE_FAULTs.
> 
> That's the bisect I did with Linus' repository
> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/
> 
> 
> git bisect start
> # good: [3466b547e37b988723dc93465b7cb06b4b1f731f] Merge branches 'pnp',
> 'pm-cpuidle' and 'pm-cpufreq'
> git bisect good 3466b547e37b988723dc93465b7cb06b4b1f731f
> # bad: [cd50b70ccd5c87794ec28bfb87b7fba9961eb0ae] Merge tag
> 'pm+acpi-3.20-rc1-3' of
> git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm
> git bisect bad cd50b70ccd5c87794ec28bfb87b7fba9961eb0ae
> # good: [27a22ee4c7d5839fd7e3e441c9d675c8a5c4c22c] Merge branch 'kbuild'
> of git://git.kernel.org/pub/scm/linux/kernel/git/mmarek/kbuild
> git bisect good 27a22ee4c7d5839fd7e3e441c9d675c8a5c4c22c
> # good: [c189cb8ef62832f33b6cf757350a0270532a1ad8] Merge tag
> 'vfio-v3.20-rc1' of git://github.com/awilliam/linux-vfio
> git bisect good c189cb8ef62832f33b6cf757350a0270532a1ad8
> # good: [295324556c427d60b41668ab81a43f604533f456] Merge branch
> 'i2c/for-3.20' of git://git.kernel.org/pub/scm/linux/kernel/git/wsa/linux
> git bisect good 295324556c427d60b41668ab81a43f604533f456
> # good: [1acd2de5facd7fbea499aea64a3a3d0ec7bb9b51] Merge branch
> 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/dtor/input
> git bisect good 1acd2de5facd7fbea499aea64a3a3d0ec7bb9b51
> # good: [fde9f50f80fe89a9115b4bfa773017272597d85d] target: Add sanity
> checks for DPO/FUA bit usage
> 
> git bisect good fde9f50f80fe89a9115b4bfa773017272597d85d
> 
> 
> # bad: [22aa66a3ee5b61e0f4a0bfeabcaa567861109ec3] dm snapshot: fix a
> possible invalid memory access on unload
> 
> 
> git bisect bad 22aa66a3ee5b61e0f4a0bfeabcaa567861109ec3
> 
> 
> # bad: [7145c241a1bf2841952c3e297c4080b357b3e52d] dm crypt: avoid
> deadlock in mempools
> 
> 
> git bisect bad 7145c241a1bf2841952c3e297c4080b357b3e52d
> 
> 
> # good: [37527b869207ad4c208b1e13967d69b8bba1fbf9] dm io: reject
> unsupported DISCARD requests with EOPNOTSUPP
> 
> 
> git bisect good 37527b869207ad4c208b1e13967d69b8bba1fbf9
> 
> 
> # bad: [cf2f1abfbd0dba701f7f16ef619e4d2485de3366] dm crypt: don't
> allocate pages for a partial request
> 
> 
> git bisect bad cf2f1abfbd0dba701f7f16ef619e4d2485de3366
> 
> 
> # good: [f3396c58fd8442850e759843457d78b6ec3a9589] dm crypt: use unbound
> workqueue for request processing
> 
> git bisect good f3396c58fd8442850e759843457d78b6ec3a9589
> 
> 
> # first bad commit: [cf2f1abfbd0dba701f7f16ef619e4d2485de3366] dm crypt:
> don't allocate pages for a partial request
> 
> 
> How can I verify (e.g. w/ a patch to Linux 4.0.9) if this patch is
> really the culprit?
> 
> I'm heavily relying upon encryption:
> 
> There are 3 disks:
> - One 240 GB SSD (crypted LVM, swap and boot partition)
> - Two SATA rotational 3 TB disks (WD ST3000DM001-1CH166, encrypted raid
>   /dev/md0, LVM)
> - All in all 29 logical volumes with xfs as filesystem (besides swap
>   and bootpartition - the latter is ext4).
> 
> The system is based on an AMD FX8350 processor (8 core) w/ 24GB RAM.
> Motherboard is a Gigabyte GA-990XA-UD3. You can find a complete dmesg
> output here [2].
> 
> 
> 
> I would be glad to get some assistance!

Are your SATA devcies using NCQ?

Please see this dm-devl thread (and this post in particular):
https://www.redhat.com/archives/dm-devel/2015-June/msg00005.html

Re: AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Andreas Hartmann]

On 07/28/2015 at 07:50 PM, Mike Snitzer wrote:
[..]
> Are your SATA devcies using NCQ?

Yes. It's enabled:

dmesg| grep -i ncq
ahci 0000:00:11.0: flags: 64bit ncq sntf ilck pm led clo pmp pio slum part
ata2.00: 5860533168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA
ata3.00: 5860533168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA
ata1.00: 468862128 sectors, multi 16: LBA48 NCQ (depth 31/32), AA

As the errors already come up on boot (during mount of partitions or
even before the password for the disk has been provided): How can I
disable NCQ during boot of the kernel? Is there a kernel option?

I'm trying to apply your mentioned patches to 3.19 and report back.

> 
> Please see this dm-devl thread (and this post in particular):
> https://www.redhat.com/archives/dm-devel/2015-June/msg00005.html
> 

Thanks,
Andreas

Re: AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Mike Snitzer]

On Tue, Jul 28 2015 at  2:20pm -0400,
Andreas Hartmann <andihartmann@freenet.de> wrote:

> On 07/28/2015 at 07:50 PM, Mike Snitzer wrote:
> [..]
> > Are your SATA devcies using NCQ?
> 
> Yes. It's enabled:
> 
> dmesg| grep -i ncq
> ahci 0000:00:11.0: flags: 64bit ncq sntf ilck pm led clo pmp pio slum part
> ata2.00: 5860533168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA
> ata3.00: 5860533168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA
> ata1.00: 468862128 sectors, multi 16: LBA48 NCQ (depth 31/32), AA
> 
> As the errors already come up on boot (during mount of partitions or
> even before the password for the disk has been provided): How can I
> disable NCQ during boot of the kernel? Is there a kernel option?

See:
https://ata.wiki.kernel.org/index.php/Libata_FAQ#Enabling.2C_disabling_and_checking_NCQ

alternatively, and likely easier, set this on the kernel commandline:
 libata.force=noncq
 
> I'm trying to apply your mentioned patches to 3.19 and report back.

OK.

Re: AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Andreas Hartmann]

On 07/28/2015 at 08:58 PM, Mike Snitzer wrote:
> On Tue, Jul 28 2015 at  2:20pm -0400,
> Andreas Hartmann <andihartmann@freenet.de> wrote:
> 
>> On 07/28/2015 at 07:50 PM, Mike Snitzer wrote:
>> [..]
>>> Are your SATA devcies using NCQ?
>>
>> Yes. It's enabled:
>>
>> dmesg| grep -i ncq
>> ahci 0000:00:11.0: flags: 64bit ncq sntf ilck pm led clo pmp pio slum part
>> ata2.00: 5860533168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA
>> ata3.00: 5860533168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA
>> ata1.00: 468862128 sectors, multi 16: LBA48 NCQ (depth 31/32), AA
>>
>> As the errors already come up on boot (during mount of partitions or
>> even before the password for the disk has been provided): How can I
>> disable NCQ during boot of the kernel? Is there a kernel option?
> 
> See:
> https://ata.wiki.kernel.org/index.php/Libata_FAQ#Enabling.2C_disabling_and_checking_NCQ
> 
> alternatively, and likely easier, set this on the kernel commandline:
>  libata.force=noncq

ata2.00: FORCE: horkage modified (noncq)
ata2.00: 5860533168 sectors, multi 0: LBA48 NCQ (not used)
ata3.00: FORCE: horkage modified (noncq)
ata3.00: 5860533168 sectors, multi 0: LBA48 NCQ (not used)
ata5.00: FORCE: horkage modified (noncq)
ata1.00: FORCE: horkage modified (noncq)
ata1.00: 468862128 sectors, multi 16: LBA48 NCQ (not used)


Perfectly. Seems to work w/ 3.19.8 and your mentioned patches. But now,
I'm getting another error, which I didn't see before w/ 3.x-kernels:

[drm:btc_dpm_set_power_state [radeon]] *ERROR*
rv770_restrict_performance_levels_before_switch failed

It seams that your patches do have some unwanted side effects :-).

Could you please reexamine your patch "dm crypt: don't allocate
pages for a partial request" - after applying this patch all the
problems are coming up here.



Thanks,
regards,
Andreas

Re: AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Mike Snitzer]

On Tue, Jul 28 2015 at  3:23pm -0400,
Andreas Hartmann <andihartmann@freenet.de> wrote:

> On 07/28/2015 at 08:58 PM, Mike Snitzer wrote:
> > On Tue, Jul 28 2015 at  2:20pm -0400,
> > Andreas Hartmann <andihartmann@freenet.de> wrote:
> > 
> >> On 07/28/2015 at 07:50 PM, Mike Snitzer wrote:
> >> [..]
> >>> Are your SATA devcies using NCQ?
> >>
> >> Yes. It's enabled:
> >>
> >> dmesg| grep -i ncq
> >> ahci 0000:00:11.0: flags: 64bit ncq sntf ilck pm led clo pmp pio slum part
> >> ata2.00: 5860533168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA
> >> ata3.00: 5860533168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA
> >> ata1.00: 468862128 sectors, multi 16: LBA48 NCQ (depth 31/32), AA
> >>
> >> As the errors already come up on boot (during mount of partitions or
> >> even before the password for the disk has been provided): How can I
> >> disable NCQ during boot of the kernel? Is there a kernel option?
> > 
> > See:
> > https://ata.wiki.kernel.org/index.php/Libata_FAQ#Enabling.2C_disabling_and_checking_NCQ
> > 
> > alternatively, and likely easier, set this on the kernel commandline:
> >  libata.force=noncq
> 
> ata2.00: FORCE: horkage modified (noncq)
> ata2.00: 5860533168 sectors, multi 0: LBA48 NCQ (not used)
> ata3.00: FORCE: horkage modified (noncq)
> ata3.00: 5860533168 sectors, multi 0: LBA48 NCQ (not used)
> ata5.00: FORCE: horkage modified (noncq)
> ata1.00: FORCE: horkage modified (noncq)
> ata1.00: 468862128 sectors, multi 16: LBA48 NCQ (not used)
> 
> 
> Perfectly. Seems to work w/ 3.19.8 and your mentioned patches. But now,
> I'm getting another error, which I didn't see before w/ 3.x-kernels:
> 
> [drm:btc_dpm_set_power_state [radeon]] *ERROR*
> rv770_restrict_performance_levels_before_switch failed
> 
> It seams that your patches do have some unwanted side effects :-).

That is a completely different issue.  drm and radeon is a graphics
issue.
 
> Could you please reexamine your patch "dm crypt: don't allocate
> pages for a partial request" - after applying this patch all the
> problems are coming up here.

More likely than not your hardware isn't very good.

Re: AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Andreas Hartmann]

On 07/28/2015 at 21:31 PM, Mike Snitzer wrote:
 > On Tue, Jul 28 2015 at  3:23pm -0400,
> Andreas Hartmann <andihartmann@freenet.de> wrote:
>
>> On 07/28/2015 at 08:58 PM, Mike Snitzer wrote:
>>> On Tue, Jul 28 2015 at  2:20pm -0400,
>>> Andreas Hartmann <andihartmann@freenet.de> wrote:
>>>
>>>> On 07/28/2015 at 07:50 PM, Mike Snitzer wrote:
>>>> [..]
>>>>> Are your SATA devcies using NCQ?
>>>>
>>>> Yes. It's enabled:
>>>>
>>>> dmesg| grep -i ncq
>>>> ahci 0000:00:11.0: flags: 64bit ncq sntf ilck pm led clo pmp pio slum part
>>>> ata2.00: 5860533168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA
>>>> ata3.00: 5860533168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA
>>>> ata1.00: 468862128 sectors, multi 16: LBA48 NCQ (depth 31/32), AA
>>>>
>>>> As the errors already come up on boot (during mount of partitions or
>>>> even before the password for the disk has been provided): How can I
>>>> disable NCQ during boot of the kernel? Is there a kernel option?
>>>
>>> See:
>>> https://ata.wiki.kernel.org/index.php/Libata_FAQ#Enabling.2C_disabling_and_checking_NCQ
>>>
>>> alternatively, and likely easier, set this on the kernel commandline:
>>>   libata.force=noncq
>>
>> ata2.00: FORCE: horkage modified (noncq)
>> ata2.00: 5860533168 sectors, multi 0: LBA48 NCQ (not used)
>> ata3.00: FORCE: horkage modified (noncq)
>> ata3.00: 5860533168 sectors, multi 0: LBA48 NCQ (not used)
>> ata5.00: FORCE: horkage modified (noncq)
>> ata1.00: FORCE: horkage modified (noncq)
>> ata1.00: 468862128 sectors, multi 16: LBA48 NCQ (not used)
>>
>>
>> Perfectly. Seems to work w/ 3.19.8 and your mentioned patches. But now,
>> I'm getting another error, which I didn't see before w/ 3.x-kernels:
>>
>> [drm:btc_dpm_set_power_state [radeon]] *ERROR*
>> rv770_restrict_performance_levels_before_switch failed
>>
>> It seams that your patches do have some unwanted side effects :-).
>
> That is a completely different issue.  drm and radeon is a graphics
> issue.

Nothing changed on radeon code. I just applied your patches. Nothing 
more. Why should radeon been suddenly broken if I apply your patches to 
a stable 3.19.8 code?

These patches trigger tons of AMD-Vi IO_PAGE_FAULTs w/ ncq enabled and 
the IOMMU developers say, that it is not a problem of the iommu code.

>> Could you please reexamine your patch "dm crypt: don't allocate
>> pages for a partial request" - after applying this patch all the
>> problems are coming up here.
>
> More likely than not your hardware isn't very good.

Maybe - maybe not. The only thing I know for sure, is: with these 
patches applied, the machine doesn't work reliably any more. W/ ncq 
disabled, the AMD-Vi IO_PAGE_FAULTs are gone, but a radeon error never 
seen before came instead. Most probably chance. Most probably, it could 
have been risen any other error, too.

I am willing to do tests if you have any idea to be tested - I can 
reproduce it quite easily.


Thanks,
regards,
Andreas

Re: AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Mike Snitzer]

On Tue, Jul 28 2015 at  4:08pm -0400,
Andreas Hartmann <andihartmann@freenet.de> wrote:

> On 07/28/2015 at 21:31 PM, Mike Snitzer wrote:
> > On Tue, Jul 28 2015 at  3:23pm -0400,
> >Andreas Hartmann <andihartmann@freenet.de> wrote:
> >
> >>On 07/28/2015 at 08:58 PM, Mike Snitzer wrote:
> >>>On Tue, Jul 28 2015 at  2:20pm -0400,
> >>>Andreas Hartmann <andihartmann@freenet.de> wrote:
> >>>
> >>>>On 07/28/2015 at 07:50 PM, Mike Snitzer wrote:
> >>>>[..]
> >>>>>Are your SATA devcies using NCQ?
> >>>>
> >>>>Yes. It's enabled:
> >>>>
> >>>>dmesg| grep -i ncq
> >>>>ahci 0000:00:11.0: flags: 64bit ncq sntf ilck pm led clo pmp pio slum part
> >>>>ata2.00: 5860533168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA
> >>>>ata3.00: 5860533168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA
> >>>>ata1.00: 468862128 sectors, multi 16: LBA48 NCQ (depth 31/32), AA
> >>>>
> >>>>As the errors already come up on boot (during mount of partitions or
> >>>>even before the password for the disk has been provided): How can I
> >>>>disable NCQ during boot of the kernel? Is there a kernel option?
> >>>
> >>>See:
> >>>https://ata.wiki.kernel.org/index.php/Libata_FAQ#Enabling.2C_disabling_and_checking_NCQ
> >>>
> >>>alternatively, and likely easier, set this on the kernel commandline:
> >>>  libata.force=noncq
> >>
> >>ata2.00: FORCE: horkage modified (noncq)
> >>ata2.00: 5860533168 sectors, multi 0: LBA48 NCQ (not used)
> >>ata3.00: FORCE: horkage modified (noncq)
> >>ata3.00: 5860533168 sectors, multi 0: LBA48 NCQ (not used)
> >>ata5.00: FORCE: horkage modified (noncq)
> >>ata1.00: FORCE: horkage modified (noncq)
> >>ata1.00: 468862128 sectors, multi 16: LBA48 NCQ (not used)
> >>
> >>
> >>Perfectly. Seems to work w/ 3.19.8 and your mentioned patches. But now,
> >>I'm getting another error, which I didn't see before w/ 3.x-kernels:
> >>
> >>[drm:btc_dpm_set_power_state [radeon]] *ERROR*
> >>rv770_restrict_performance_levels_before_switch failed
> >>
> >>It seams that your patches do have some unwanted side effects :-).
> >
> >That is a completely different issue.  drm and radeon is a graphics
> >issue.
> 
> Nothing changed on radeon code. I just applied your patches. Nothing
> more. Why should radeon been suddenly broken if I apply your patches
> to a stable 3.19.8 code?
> 
> These patches trigger tons of AMD-Vi IO_PAGE_FAULTs w/ ncq enabled
> and the IOMMU developers say, that it is not a problem of the iommu
> code.
> 
> >>Could you please reexamine your patch "dm crypt: don't allocate
> >>pages for a partial request" - after applying this patch all the
> >>problems are coming up here.
> >
> >More likely than not your hardware isn't very good.
> 
> Maybe - maybe not. The only thing I know for sure, is: with these
> patches applied, the machine doesn't work reliably any more. W/ ncq
> disabled, the AMD-Vi IO_PAGE_FAULTs are gone, but a radeon error
> never seen before came instead. Most probably chance. Most probably,
> it could have been risen any other error, too.
> 
> I am willing to do tests if you have any idea to be tested - I can
> reproduce it quite easily.

You can try disabling dm-crypt's parallelization by specifying these 2
features: same_cpu_crypt submit_from_crypt_cpus

It is my understanding that these can be set using the cryptsetup tool.
Milan can you clarify how these features can be set from a high-level
(on an existing crypt device)?

Re: [dm-devel] AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Ondrej Kozina]

On 07/28/2015 11:24 PM, Mike Snitzer wrote:
>>
>> I am willing to do tests if you have any idea to be tested - I can
>> reproduce it quite easily.
>
> You can try disabling dm-crypt's parallelization by specifying these 2
> features: same_cpu_crypt submit_from_crypt_cpus
>

Hi,

please try adding follwoing cryptsetup perf. options while unlocking 
your LUKS containers: --perf-submit_from_crypt_cpus and 
--perf-same_cpu_crypt.

Perhaps focus on SSD disk as previous unresolved report mentioned SSD too

Regards
O.

Re: [dm-devel] AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Milan Broz]

On 07/29/2015 08:17 AM, Ondrej Kozina wrote:
> On 07/28/2015 11:24 PM, Mike Snitzer wrote:
>>>
>>> I am willing to do tests if you have any idea to be tested - I can
>>> reproduce it quite easily.
>>
>> You can try disabling dm-crypt's parallelization by specifying these 2
>> features: same_cpu_crypt submit_from_crypt_cpus
>>
> 
> Hi,
> 
> please try adding follwoing cryptsetup perf. options while unlocking 
> your LUKS containers: --perf-submit_from_crypt_cpus and 
> --perf-same_cpu_crypt.
> 
> Perhaps focus on SSD disk as previous unresolved report mentioned SSD too

Just you need to use cryptsetup 1.6.7 version (older versions do not have this options yet).

Anyway, it seems strange that one patch triggers such a problem (leading to a conclusion
that the patch is wrong) but dm-crypt is know to reveal many problems in other systems
exactly this way...

The patches were tested on so many systems without problem that I really believe
the dmcrypt patch is just trigger for a bug elsewhere.

Anyway, if we can find configuration which fails here, I would like to help to find
the core problem... Papering here just means it will break data later elsewhere.

Thanks for testing it!
Milan

Re: [dm-devel] AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Andreas Hartmann]

On 07/29/2015 at 08:41 AM Milan Broz wrote:
> On 07/29/2015 08:17 AM, Ondrej Kozina wrote:
>> On 07/28/2015 11:24 PM, Mike Snitzer wrote:
>>>>
>>>> I am willing to do tests if you have any idea to be tested - I can
>>>> reproduce it quite easily.
>>>
>>> You can try disabling dm-crypt's parallelization by specifying these 2
>>> features: same_cpu_crypt submit_from_crypt_cpus
>>>
>>
>> Hi,
>>
>> please try adding follwoing cryptsetup perf. options while unlocking 
>> your LUKS containers: --perf-submit_from_crypt_cpus and 
>> --perf-same_cpu_crypt.
>>
>> Perhaps focus on SSD disk as previous unresolved report mentioned SSD too

I'll next try to just boot from the rotational disk. If the problem
doesn't show up here, it could be SSD related, too.

> 
> Just you need to use cryptsetup 1.6.7 version (older versions do not have this options yet).

Thanks for this hint.

I compiled this version and applied it and verified it, if it's really
in initrd. It was in initrd and the additional options where applied, too.

But: nothing changed with enabled ncq - exactly same problem as before
(w/ linux 3.19.8 + additional patches and Linux 4.1 - no difference in
behavior).

> Anyway, it seems strange that one patch triggers such a problem (leading to a conclusion
> that the patch is wrong) but dm-crypt is know to reveal many problems in other systems
> exactly this way...

Maybe - maybe not.

> The patches were tested on so many systems without problem that I really believe
> the dmcrypt patch is just trigger for a bug elsewhere.

I'm having another machine, which doesn't show the problem - but this
machine can't be compared w/ the failing machine, because the other
machine does have a totally different and easy setup (1 ssd crypted lvm
and 3 or 4 logical filesystems - that's it) - and it's much slower :-).

Here I'm  having: 3 disks (1 ssd, 2 x 3 TB rotational), 1 crypted SW
Raid 1 w/ LVM on the rotational disks, another LVM on the ssd and 29 (!)
logical volumes each with xfs and 8 CPU cores, which guarantee high
degree of parallelism. I'm not sure if this is a "usual" setup which is
tested heavily. The error always comes up while mounting the partitions
on boot. Reducing the amount of mounted partitions reduced the amount of
errors (AMD-Vi IO_PAGE_FAULTs).

> 
> Anyway, if we can find configuration which fails here, I would like to help to find
> the core problem... Papering here just means it will break data later elsewhere.
> 
> Thanks for testing it!

I hope we can find the problem.


Regards,
Andreas

Re: [dm-devel] AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Andreas Hartmann]

On 07/29/2015 at 07:23 PM Andreas Hartmann wrote:
> On 07/29/2015 at 08:41 AM Milan Broz wrote:
>> On 07/29/2015 08:17 AM, Ondrej Kozina wrote:
>>> On 07/28/2015 11:24 PM, Mike Snitzer wrote:
>>>>>
>>>>> I am willing to do tests if you have any idea to be tested - I can
>>>>> reproduce it quite easily.
>>>>
>>>> You can try disabling dm-crypt's parallelization by specifying these 2
>>>> features: same_cpu_crypt submit_from_crypt_cpus
>>>>
>>>
>>> Hi,
>>>
>>> please try adding follwoing cryptsetup perf. options while unlocking 
>>> your LUKS containers: --perf-submit_from_crypt_cpus and 
>>> --perf-same_cpu_crypt.
>>>
>>> Perhaps focus on SSD disk as previous unresolved report mentioned SSD too
> 
> I'll next try to just boot from the rotational disk. If the problem
> doesn't show up here, it could be SSD related, too.
> 
>>
>> Just you need to use cryptsetup 1.6.7 version (older versions do not have this options yet).
> 
> Thanks for this hint.
> 
> I compiled this version and applied it and verified it, if it's really
> in initrd. It was in initrd and the additional options where applied, too.
> 
> But: nothing changed with enabled ncq - exactly same problem as before
> (w/ linux 3.19.8 + additional patches and Linux 4.1 - no difference in
> behavior).


Next tests:
I didn't mount any filesystem on bootup located on the ssd (only the
rotational based mounts) -> no problem.

Next: I tried to add the partitions from the ssd to be mounted on boot -
one by one. As there where 4 partitions to be mounted from the ssd, the
errors came up again.

Next: You suspect the ssd as being bad. Therefore, I changed the ssd and
tested with KINGSTON SV300S37A240G. Result: Same as above - errors are
coming up if there are mounted more than 3 partitions at the same time.


Summary:
- --perf-submit_from_crypt_cpus --perf-same_cpu_crypt -> broken anyway
- rotational disks alone: it's ok
- more than 3 partitions on ssd Corsair Force GT -> broken
- more than 3 partitions on ssd KINGSTON SV300S37A240G -> broken



Regards,
Andreas

Re: [dm-devel] AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Milan Broz]

On 07/30/2015 10:30 PM, Andreas Hartmann wrote:
...

> Next tests:
> I didn't mount any filesystem on bootup located on the ssd (only the
> rotational based mounts) -> no problem.
> 
> Next: I tried to add the partitions from the ssd to be mounted on boot -
> one by one. As there where 4 partitions to be mounted from the ssd, the
> errors came up again.

This is strange... Do I understand it correctly that this bug appears only
if you mount more than 3 encrypted partitions? (IOW if you mount only 2 or 3
there is no problem?)

> Next: You suspect the ssd as being bad. Therefore, I changed the ssd and
> tested with KINGSTON SV300S37A240G. Result: Same as above - errors are
> coming up if there are mounted more than 3 partitions at the same time.

ok. I guess it was just image of the same disk on different hw, correct?

> 
> Summary:
> - --perf-submit_from_crypt_cpus --perf-same_cpu_crypt -> broken anyway

That is even more interesting because these should effectively disable
the parralel submit code in dmcrypt.

Could you please verify that these flags are really applied when you used
them? (I am just trying to eliminate possible cryptsetup bug, the code is new).
(Run dmsetup table, you should see submit_from_crypt_cpus/same_cpu_crypt
on the dmcrypt mappping table line according to commandline.)

> - rotational disks alone: it's ok
> - more than 3 partitions on ssd Corsair Force GT -> broken
> - more than 3 partitions on ssd KINGSTON SV300S37A240G -> broken

- if you disable NCQ, the problems goes away?

Could you please post "lsblk" and dmesg output from the situation which fails,
(with mounted >3 partition on boot)?
I think someone should defititely try to reproduce it on some development hw here,
configured storage stack should be visible from the lsblk output.

Thanks for for the info & tests!

Milan

Re: [dm-devel] AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Andreas Hartmann]

[-- Attachment #1: Type: text/plain, Size: 2430 bytes --]

On 07/31/2015 at 09:23 AM, Milan Broz wrote:
> On 07/30/2015 10:30 PM, Andreas Hartmann wrote:
> ...
> 
>> Next tests:
>> I didn't mount any filesystem on bootup located on the ssd (only the
>> rotational based mounts) -> no problem.
>>
>> Next: I tried to add the partitions from the ssd to be mounted on boot -
>> one by one. As there where 4 partitions to be mounted from the ssd, the
>> errors came up again.
> 
> This is strange... Do I understand it correctly that this bug appears only
> if you mount more than 3 encrypted partitions? (IOW if you mount only 2 or 3
> there is no problem?)

Yes. That's correct.

>> Next: You suspect the ssd as being bad. Therefore, I changed the ssd and
>> tested with KINGSTON SV300S37A240G. Result: Same as above - errors are
>> coming up if there are mounted more than 3 partitions at the same time.
> 
> ok. I guess it was just image of the same disk on different hw, correct?

No. It was a different disk, but the filesystem and the LVM was created
newly and the files have been copied from disk a to disk b. So, if there
would have been errors on the original filesystem they wouldn't have
been copied.

>>
>> Summary:
>> - --perf-submit_from_crypt_cpus --perf-same_cpu_crypt -> broken anyway
> 
> That is even more interesting because these should effectively disable
> the parralel submit code in dmcrypt.
> 
> Could you please verify that these flags are really applied when you used
> them? (I am just trying to eliminate possible cryptsetup bug, the code is new).
> (Run dmsetup table, you should see submit_from_crypt_cpus/same_cpu_crypt
> on the dmcrypt mappping table line according to commandline.)

See attached files dmsetup_table and dmesg.broken.

>> - rotational disks alone: it's ok
>> - more than 3 partitions on ssd Corsair Force GT -> broken
>> - more than 3 partitions on ssd KINGSTON SV300S37A240G -> broken
> 
> - if you disable NCQ, the problems goes away?

As I already wrote in another post - than other error came up instead
(radeon). But I will try it especially again in conjunction w/
--perf-submit_from_crypt_cpus --perf-same_cpu_crypt.

> Could you please post "lsblk" and dmesg output from the situation which fails,
> (with mounted >3 partition on boot)?
> I think someone should defititely try to reproduce it on some development hw here,
> configured storage stack should be visible from the lsblk output.

Yes. You'll get them.


Regards,
Andreas


[-- Attachment #2: dmsetup_table.gz --]
[-- Type: application/x-gzip, Size: 738 bytes --]

[-- Attachment #3: dmesg.broken.gz --]
[-- Type: application/x-gzip, Size: 22656 bytes --]

[-- Attachment #4: lsblk.gz --]
[-- Type: application/x-gzip, Size: 774 bytes --]

Re: [dm-devel] AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Andreas Hartmann]

On 07/31/2015 at 09:55 AM Andreas Hartmann wrote:
> On 07/31/2015 at 09:23 AM, Milan Broz wrote:
>> On 07/30/2015 10:30 PM, Andreas Hartmann wrote:
[...]
>>> - rotational disks alone: it's ok
>>> - more than 3 partitions on ssd Corsair Force GT -> broken
>>> - more than 3 partitions on ssd KINGSTON SV300S37A240G -> broken
>>
>> - if you disable NCQ, the problems goes away?
> 
> As I already wrote in another post - than other error came up instead
> (radeon). But I will try it especially again in conjunction w/
> --perf-submit_from_crypt_cpus --perf-same_cpu_crypt.

Combining all three options (noncq, --perf-submit_from_crypt_cpus and
--perf-same_cpu_crypt) seem to "work".


Regards,
Andreas

Re: [dm-devel] AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Milan Broz]

On 07/31/2015 09:55 AM, Andreas Hartmann wrote:
> On 07/31/2015 at 09:23 AM, Milan Broz wrote:

...

>> - if you disable NCQ, the problems goes away?
> 
> As I already wrote in another post - than other error came up instead
> (radeon). But I will try it especially again in conjunction w/
> --perf-submit_from_crypt_cpus --perf-same_cpu_crypt.

Yes, I know, But this is probably just some unfortunate side effect,
this code definitely does not relate to Radeon code.
(Could be some minor compiler change, whatever...)

Thanks,
Milan

Re: AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Milan Broz]

On 07/28/2015 11:24 PM, Mike Snitzer wrote:
> On Tue, Jul 28 2015 at  4:08pm -0400,
> Andreas Hartmann <andihartmann@freenet.de> wrote:
> 
>> On 07/28/2015 at 21:31 PM, Mike Snitzer wrote:
>>> On Tue, Jul 28 2015 at  3:23pm -0400,
>>> Andreas Hartmann <andihartmann@freenet.de> wrote:
>>>
>>>> On 07/28/2015 at 08:58 PM, Mike Snitzer wrote:
>>>>> On Tue, Jul 28 2015 at  2:20pm -0400,
>>>>> Andreas Hartmann <andihartmann@freenet.de> wrote:
>>>>>
>>>>>> On 07/28/2015 at 07:50 PM, Mike Snitzer wrote:
>>>>>> [..]
>>>>>>> Are your SATA devcies using NCQ?
>>>>>>
>>>>>> Yes. It's enabled:
>>>>>>
>>>>>> dmesg| grep -i ncq
>>>>>> ahci 0000:00:11.0: flags: 64bit ncq sntf ilck pm led clo pmp pio slum part
>>>>>> ata2.00: 5860533168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA
>>>>>> ata3.00: 5860533168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA
>>>>>> ata1.00: 468862128 sectors, multi 16: LBA48 NCQ (depth 31/32), AA
>>>>>>
>>>>>> As the errors already come up on boot (during mount of partitions or
>>>>>> even before the password for the disk has been provided): How can I
>>>>>> disable NCQ during boot of the kernel? Is there a kernel option?
>>>>>
>>>>> See:
>>>>> https://ata.wiki.kernel.org/index.php/Libata_FAQ#Enabling.2C_disabling_and_checking_NCQ
>>>>>
>>>>> alternatively, and likely easier, set this on the kernel commandline:
>>>>>  libata.force=noncq
>>>>
>>>> ata2.00: FORCE: horkage modified (noncq)
>>>> ata2.00: 5860533168 sectors, multi 0: LBA48 NCQ (not used)
>>>> ata3.00: FORCE: horkage modified (noncq)
>>>> ata3.00: 5860533168 sectors, multi 0: LBA48 NCQ (not used)
>>>> ata5.00: FORCE: horkage modified (noncq)
>>>> ata1.00: FORCE: horkage modified (noncq)
>>>> ata1.00: 468862128 sectors, multi 16: LBA48 NCQ (not used)
>>>>
>>>>
>>>> Perfectly. Seems to work w/ 3.19.8 and your mentioned patches. But now,
>>>> I'm getting another error, which I didn't see before w/ 3.x-kernels:
>>>>
>>>> [drm:btc_dpm_set_power_state [radeon]] *ERROR*
>>>> rv770_restrict_performance_levels_before_switch failed
>>>>
>>>> It seams that your patches do have some unwanted side effects :-).
>>>
>>> That is a completely different issue.  drm and radeon is a graphics
>>> issue.
>>
>> Nothing changed on radeon code. I just applied your patches. Nothing
>> more. Why should radeon been suddenly broken if I apply your patches
>> to a stable 3.19.8 code?
>>
>> These patches trigger tons of AMD-Vi IO_PAGE_FAULTs w/ ncq enabled
>> and the IOMMU developers say, that it is not a problem of the iommu
>> code.
>>
>>>> Could you please reexamine your patch "dm crypt: don't allocate
>>>> pages for a partial request" - after applying this patch all the
>>>> problems are coming up here.
>>>
>>> More likely than not your hardware isn't very good.
>>
>> Maybe - maybe not. The only thing I know for sure, is: with these
>> patches applied, the machine doesn't work reliably any more. W/ ncq
>> disabled, the AMD-Vi IO_PAGE_FAULTs are gone, but a radeon error
>> never seen before came instead. Most probably chance. Most probably,
>> it could have been risen any other error, too.
>>
>> I am willing to do tests if you have any idea to be tested - I can
>> reproduce it quite easily.
> 
> You can try disabling dm-crypt's parallelization by specifying these 2
> features: same_cpu_crypt submit_from_crypt_cpus
> 
> It is my understanding that these can be set using the cryptsetup tool.
> Milan can you clarify how these features can be set from a high-level
> (on an existing crypt device)?

Just one note - for me it seems that you are hitting an firmware problem
related to NCQ implementation in your SSD see.

Similar page what Mike already mentioned:
https://wiki.archlinux.org/index.php/Solid_State_Drives#Resolving_NCQ_errors

Anyway, I have myself SSD drive with NCQ active and I have never seen this problem
(I am using these dmcrypt patches backported since 3.16 kernel or so).
(And my system is very intensively used with this config.)

Perhaps you could also check if there a new firmware for your SSD?
(From log I see it is Corsair Force GT and there were some known problems.)

Thanks,
Milan

Re: AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Andreas Hartmann]

On 07/28/2015 at 07:50 PM Mike Snitzer wrote:
> On Tue, Jul 28 2015 at  1:40pm -0400,
> Andreas Hartmann <andihartmann@01019freenet.de> wrote:
> 
>> Hello!
>>
>> After long and heavy bisecting, I found this commit
>> "dm crypt: don't allocate pages for a partial request" [1] being the
>> cause of the ata errors and AMD-Vi IO_PAGE_FAULTs.
>>
>> That's the bisect I did with Linus' repository
>> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/
>>
>>
>> git bisect start
>> # good: [3466b547e37b988723dc93465b7cb06b4b1f731f] Merge branches 'pnp',
>> 'pm-cpuidle' and 'pm-cpufreq'
>> git bisect good 3466b547e37b988723dc93465b7cb06b4b1f731f
>> # bad: [cd50b70ccd5c87794ec28bfb87b7fba9961eb0ae] Merge tag
>> 'pm+acpi-3.20-rc1-3' of
>> git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm
>> git bisect bad cd50b70ccd5c87794ec28bfb87b7fba9961eb0ae
>> # good: [27a22ee4c7d5839fd7e3e441c9d675c8a5c4c22c] Merge branch 'kbuild'
>> of git://git.kernel.org/pub/scm/linux/kernel/git/mmarek/kbuild
>> git bisect good 27a22ee4c7d5839fd7e3e441c9d675c8a5c4c22c
>> # good: [c189cb8ef62832f33b6cf757350a0270532a1ad8] Merge tag
>> 'vfio-v3.20-rc1' of git://github.com/awilliam/linux-vfio
>> git bisect good c189cb8ef62832f33b6cf757350a0270532a1ad8
>> # good: [295324556c427d60b41668ab81a43f604533f456] Merge branch
>> 'i2c/for-3.20' of git://git.kernel.org/pub/scm/linux/kernel/git/wsa/linux
>> git bisect good 295324556c427d60b41668ab81a43f604533f456
>> # good: [1acd2de5facd7fbea499aea64a3a3d0ec7bb9b51] Merge branch
>> 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/dtor/input
>> git bisect good 1acd2de5facd7fbea499aea64a3a3d0ec7bb9b51
>> # good: [fde9f50f80fe89a9115b4bfa773017272597d85d] target: Add sanity
>> checks for DPO/FUA bit usage
>>
>> git bisect good fde9f50f80fe89a9115b4bfa773017272597d85d
>>
>>
>> # bad: [22aa66a3ee5b61e0f4a0bfeabcaa567861109ec3] dm snapshot: fix a
>> possible invalid memory access on unload
>>
>>
>> git bisect bad 22aa66a3ee5b61e0f4a0bfeabcaa567861109ec3
>>
>>
>> # bad: [7145c241a1bf2841952c3e297c4080b357b3e52d] dm crypt: avoid
>> deadlock in mempools
>>
>>
>> git bisect bad 7145c241a1bf2841952c3e297c4080b357b3e52d
>>
>>
>> # good: [37527b869207ad4c208b1e13967d69b8bba1fbf9] dm io: reject
>> unsupported DISCARD requests with EOPNOTSUPP
>>
>>
>> git bisect good 37527b869207ad4c208b1e13967d69b8bba1fbf9
>>
>>
>> # bad: [cf2f1abfbd0dba701f7f16ef619e4d2485de3366] dm crypt: don't
>> allocate pages for a partial request
>>
>>
>> git bisect bad cf2f1abfbd0dba701f7f16ef619e4d2485de3366
>>
>>
>> # good: [f3396c58fd8442850e759843457d78b6ec3a9589] dm crypt: use unbound
>> workqueue for request processing
>>
>> git bisect good f3396c58fd8442850e759843457d78b6ec3a9589
>>
>>
>> # first bad commit: [cf2f1abfbd0dba701f7f16ef619e4d2485de3366] dm crypt:
>> don't allocate pages for a partial request
>>
>>
>> How can I verify (e.g. w/ a patch to Linux 4.0.9) if this patch is
>> really the culprit?
>>
>> I'm heavily relying upon encryption:
>>
>> There are 3 disks:
>> - One 240 GB SSD (crypted LVM, swap and boot partition)
>> - Two SATA rotational 3 TB disks (WD ST3000DM001-1CH166, encrypted raid
>>   /dev/md0, LVM)
>> - All in all 29 logical volumes with xfs as filesystem (besides swap
>>   and bootpartition - the latter is ext4).
>>
>> The system is based on an AMD FX8350 processor (8 core) w/ 24GB RAM.
>> Motherboard is a Gigabyte GA-990XA-UD3. You can find a complete dmesg
>> output here [2].
>>
>>
>>
>> I would be glad to get some assistance!
> 
> Are your SATA devcies using NCQ?
> 
> Please see this dm-devl thread (and this post in particular):
> https://www.redhat.com/archives/dm-devel/2015-June/msg00005.html

As suggested, I applied these commits

f3396c58fd8442850e759843457d78b6ec3a9589,
cf2f1abfbd0dba701f7f16ef619e4d2485de3366,
7145c241a1bf2841952c3e297c4080b357b3e52d,
94f5e0243c48aa01441c987743dc468e2d6eaca2,
dc2676210c425ee8e5cb1bec5bc84d004ddf4179,
0f5d8e6ee758f7023e4353cca75d785b2d4f6abe,
b3c5fd3052492f1b8d060799d4f18be5a5438add

to 3.19.8 and the problem is exactly the same as described above.

I don't think that the problem is SSD related, because the ata3 error I
can see belongs to the rotational disk (ata1 would be the SSD).

The git bisect you mentioned is already done: "dm crypt: don't allocate
pages for a partial request" is the culprit.

Besides that: How can I disable ncq? Maybe a kernel patch, which
prevents enabling it because I need it on bootup before the disks are
accessed.



Thanks,
Andreas

Re: AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Mike Snitzer]

On Tue, Jul 28 2015 at  2:56pm -0400,
Andreas Hartmann <andihartmann@freenet.de> wrote:

> On 07/28/2015 at 07:50 PM Mike Snitzer wrote:
> > On Tue, Jul 28 2015 at  1:40pm -0400,
> > Andreas Hartmann <andihartmann@01019freenet.de> wrote:
> > 
> >> Hello!
> >>
> >> After long and heavy bisecting, I found this commit
> >> "dm crypt: don't allocate pages for a partial request" [1] being the
> >> cause of the ata errors and AMD-Vi IO_PAGE_FAULTs.
> >>
> >> That's the bisect I did with Linus' repository
> >> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/
> >>
> >>
> >> git bisect start
> >> # good: [3466b547e37b988723dc93465b7cb06b4b1f731f] Merge branches 'pnp',
> >> 'pm-cpuidle' and 'pm-cpufreq'
> >> git bisect good 3466b547e37b988723dc93465b7cb06b4b1f731f
> >> # bad: [cd50b70ccd5c87794ec28bfb87b7fba9961eb0ae] Merge tag
> >> 'pm+acpi-3.20-rc1-3' of
> >> git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm
> >> git bisect bad cd50b70ccd5c87794ec28bfb87b7fba9961eb0ae
> >> # good: [27a22ee4c7d5839fd7e3e441c9d675c8a5c4c22c] Merge branch 'kbuild'
> >> of git://git.kernel.org/pub/scm/linux/kernel/git/mmarek/kbuild
> >> git bisect good 27a22ee4c7d5839fd7e3e441c9d675c8a5c4c22c
> >> # good: [c189cb8ef62832f33b6cf757350a0270532a1ad8] Merge tag
> >> 'vfio-v3.20-rc1' of git://github.com/awilliam/linux-vfio
> >> git bisect good c189cb8ef62832f33b6cf757350a0270532a1ad8
> >> # good: [295324556c427d60b41668ab81a43f604533f456] Merge branch
> >> 'i2c/for-3.20' of git://git.kernel.org/pub/scm/linux/kernel/git/wsa/linux
> >> git bisect good 295324556c427d60b41668ab81a43f604533f456
> >> # good: [1acd2de5facd7fbea499aea64a3a3d0ec7bb9b51] Merge branch
> >> 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/dtor/input
> >> git bisect good 1acd2de5facd7fbea499aea64a3a3d0ec7bb9b51
> >> # good: [fde9f50f80fe89a9115b4bfa773017272597d85d] target: Add sanity
> >> checks for DPO/FUA bit usage
> >>
> >> git bisect good fde9f50f80fe89a9115b4bfa773017272597d85d
> >>
> >>
> >> # bad: [22aa66a3ee5b61e0f4a0bfeabcaa567861109ec3] dm snapshot: fix a
> >> possible invalid memory access on unload
> >>
> >>
> >> git bisect bad 22aa66a3ee5b61e0f4a0bfeabcaa567861109ec3
> >>
> >>
> >> # bad: [7145c241a1bf2841952c3e297c4080b357b3e52d] dm crypt: avoid
> >> deadlock in mempools
> >>
> >>
> >> git bisect bad 7145c241a1bf2841952c3e297c4080b357b3e52d
> >>
> >>
> >> # good: [37527b869207ad4c208b1e13967d69b8bba1fbf9] dm io: reject
> >> unsupported DISCARD requests with EOPNOTSUPP
> >>
> >>
> >> git bisect good 37527b869207ad4c208b1e13967d69b8bba1fbf9
> >>
> >>
> >> # bad: [cf2f1abfbd0dba701f7f16ef619e4d2485de3366] dm crypt: don't
> >> allocate pages for a partial request
> >>
> >>
> >> git bisect bad cf2f1abfbd0dba701f7f16ef619e4d2485de3366
> >>
> >>
> >> # good: [f3396c58fd8442850e759843457d78b6ec3a9589] dm crypt: use unbound
> >> workqueue for request processing
> >>
> >> git bisect good f3396c58fd8442850e759843457d78b6ec3a9589
> >>
> >>
> >> # first bad commit: [cf2f1abfbd0dba701f7f16ef619e4d2485de3366] dm crypt:
> >> don't allocate pages for a partial request
> >>
> >>
> >> How can I verify (e.g. w/ a patch to Linux 4.0.9) if this patch is
> >> really the culprit?
> >>
> >> I'm heavily relying upon encryption:
> >>
> >> There are 3 disks:
> >> - One 240 GB SSD (crypted LVM, swap and boot partition)
> >> - Two SATA rotational 3 TB disks (WD ST3000DM001-1CH166, encrypted raid
> >>   /dev/md0, LVM)
> >> - All in all 29 logical volumes with xfs as filesystem (besides swap
> >>   and bootpartition - the latter is ext4).
> >>
> >> The system is based on an AMD FX8350 processor (8 core) w/ 24GB RAM.
> >> Motherboard is a Gigabyte GA-990XA-UD3. You can find a complete dmesg
> >> output here [2].
> >>
> >>
> >>
> >> I would be glad to get some assistance!
> > 
> > Are your SATA devcies using NCQ?
> > 
> > Please see this dm-devl thread (and this post in particular):
> > https://www.redhat.com/archives/dm-devel/2015-June/msg00005.html
> 
> As suggested, I applied these commits
> 
> f3396c58fd8442850e759843457d78b6ec3a9589,
> cf2f1abfbd0dba701f7f16ef619e4d2485de3366,
> 7145c241a1bf2841952c3e297c4080b357b3e52d,
> 94f5e0243c48aa01441c987743dc468e2d6eaca2,
> dc2676210c425ee8e5cb1bec5bc84d004ddf4179,
> 0f5d8e6ee758f7023e4353cca75d785b2d4f6abe,
> b3c5fd3052492f1b8d060799d4f18be5a5438add
> 
> to 3.19.8 and the problem is exactly the same as described above.
> 
> I don't think that the problem is SSD related, because the ata3 error I
> can see belongs to the rotational disk (ata1 would be the SSD).
> 
> The git bisect you mentioned is already done: "dm crypt: don't allocate
> pages for a partial request" is the culprit.

Mikulas was saying to biect what is causing ATA to fail.

> Besides that: How can I disable ncq? Maybe a kernel patch, which
> prevents enabling it because I need it on bootup before the disks are
> accessed.

I already answered how, see my previous reply.

Re: [dm-devel] AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Andreas Hartmann]

On 07/28/2015 at 09:29 PM, Mike Snitzer wrote:
[...]
> Mikulas was saying to biect what is causing ATA to fail.

Some good news and some bad news. The good news first:

Your patchset

f3396c58fd8442850e759843457d78b6ec3a9589,
cf2f1abfbd0dba701f7f16ef619e4d2485de3366,
7145c241a1bf2841952c3e297c4080b357b3e52d,
94f5e0243c48aa01441c987743dc468e2d6eaca2,
dc2676210c425ee8e5cb1bec5bc84d004ddf4179,
0f5d8e6ee758f7023e4353cca75d785b2d4f6abe,
b3c5fd3052492f1b8d060799d4f18be5a5438add

seems to works fine w/ 3.18.19 !!

Why did I test it with 3.18.x now? Because I suddenly got two ata errors
(ata1 and ata2) with clean 3.19.8 (w/o the AMD-Vi IO_PAGE_FAULTs) during
normal operation. This means: 3.19 must already be broken, too.

Therefore, I applied your patchset to 3.18.x and it seems to work like a
charme - I don't get any AMD-Vi IO_PAGE_FAULTs on boot and no ata errors
(until now).


Next I did: I tried to bisect between 3.18 and 3.19 with your patchset
applied, because w/ this patchset applied, the problem can be seen
easily and directly on boot. Unfortunately, this does work only a few
git bisect rounds until I got stuck because of interferences with your
extra patches applied:

git bisect start
# good: [b2776bf7149bddd1f4161f14f79520f17fc1d71d] Linux 3.18
git bisect good b2776bf7149bddd1f4161f14f79520f17fc1d71d
# bad: [bfa76d49576599a4b9f9b7a71f23d73d6dcff735] Linux 3.19
git bisect bad bfa76d49576599a4b9f9b7a71f23d73d6dcff735
# good: [54850e73e86e3bc092680d1bdb84eb322f982ab1] zram: change
parameter from vaild_io_request()
git bisect good 54850e73e86e3bc092680d1bdb84eb322f982ab1
# bad: [dab363f938a53ddaee60bfecc1aebdbb3d3af5f0] Merge tag
'staging-3.19-rc1' of
git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging
git bisect bad dab363f938a53ddaee60bfecc1aebdbb3d3af5f0
# bad: [26178ec11ef3c6c814bf16a0a2b9c2f7242e3c64] x86: mm: consolidate
VM_FAULT_RETRY handling
git bisect bad 26178ec11ef3c6c814bf16a0a2b9c2f7242e3c64
# bad: [e7cf773d431a63a2417902696fcc9e0ebdc83bbe] Merge tag
'usb-3.19-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb
git bisect bad e7cf773d431a63a2417902696fcc9e0ebdc83bbe
# good: [9ea18f8cab5f1c36cdd0f09717e35ceb48c36a87] Merge branch
'for-3.19/drivers' of git://git.kernel.dk/linux-block
git bisect good 9ea18f8cab5f1c36cdd0f09717e35ceb48c36a87
Bisecting: 251 revisions left to test after this (roughly 8 steps)
error: Your local changes to the following files would be overwritten by
checkout:
        drivers/md/dm-crypt.c
Please, commit your changes or stash them before you can switch branches.
Aborting


Do you have any idea how to proceed at this point?


Thanks,
Andreas

Re: [dm-devel] AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Andreas Hartmann]

On 08/01/2015 at 04:20 PM Andreas Hartmann wrote:
> On 07/28/2015 at 09:29 PM, Mike Snitzer wrote:
> [...]
>> Mikulas was saying to biect what is causing ATA to fail.
> 
> Some good news and some bad news. The good news first:
> 
> Your patchset
> 
> f3396c58fd8442850e759843457d78b6ec3a9589,
> cf2f1abfbd0dba701f7f16ef619e4d2485de3366,
> 7145c241a1bf2841952c3e297c4080b357b3e52d,
> 94f5e0243c48aa01441c987743dc468e2d6eaca2,
> dc2676210c425ee8e5cb1bec5bc84d004ddf4179,
> 0f5d8e6ee758f7023e4353cca75d785b2d4f6abe,
> b3c5fd3052492f1b8d060799d4f18be5a5438add
> 
> seems to work fine w/ 3.18.19 !!
> 
> Why did I test it with 3.18.x now? Because I suddenly got two ata errors
> (ata1 and ata2) with clean 3.19.8 (w/o the AMD-Vi IO_PAGE_FAULTs) during
> normal operation. This means: 3.19 must already be broken, too.
> 
> Therefore, I applied your patchset to 3.18.x and it seems to work like a
> charme - I don't get any AMD-Vi IO_PAGE_FAULTs on boot and no ata errors
> (until now).
> 
> 
> Next I did: I tried to bisect between 3.18 and 3.19 with your patchset
> applied, because w/ this patchset applied, the problem can be seen
> easily and directly on boot. Unfortunately, this does work only a few
> git bisect rounds until I got stuck because of interferences with your
> extra patches applied:

[Resolved the problems written at the last post.]

Bisecting ended here:

https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=34b48db66e08ca1c1bc07cf305d672ac940268dc

block: remove artifical max_hw_sectors cap


Removing this patch on 3.19 and 4.1 make things working again. Didn't
test 4.0, but I think it's the same. No more AMD-Vi IO_PAGE_FAULTS  with
that patch reverted.


Please check why this patch triggers AMD-Vi IO_PAGE_FAULTS.


Thanks,
kind regards,
Andreas Hartmann

Re: [dm-devel] AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Mikulas Patocka]

On Sun, 2 Aug 2015, Andreas Hartmann wrote:

> On 08/01/2015 at 04:20 PM Andreas Hartmann wrote:
> > On 07/28/2015 at 09:29 PM, Mike Snitzer wrote:
> > [...]
> >> Mikulas was saying to biect what is causing ATA to fail.
> > 
> > Some good news and some bad news. The good news first:
> > 
> > Your patchset
> > 
> > f3396c58fd8442850e759843457d78b6ec3a9589,
> > cf2f1abfbd0dba701f7f16ef619e4d2485de3366,
> > 7145c241a1bf2841952c3e297c4080b357b3e52d,
> > 94f5e0243c48aa01441c987743dc468e2d6eaca2,
> > dc2676210c425ee8e5cb1bec5bc84d004ddf4179,
> > 0f5d8e6ee758f7023e4353cca75d785b2d4f6abe,
> > b3c5fd3052492f1b8d060799d4f18be5a5438add
> > 
> > seems to work fine w/ 3.18.19 !!
> > 
> > Why did I test it with 3.18.x now? Because I suddenly got two ata errors
> > (ata1 and ata2) with clean 3.19.8 (w/o the AMD-Vi IO_PAGE_FAULTs) during
> > normal operation. This means: 3.19 must already be broken, too.
> > 
> > Therefore, I applied your patchset to 3.18.x and it seems to work like a
> > charme - I don't get any AMD-Vi IO_PAGE_FAULTs on boot and no ata errors
> > (until now).
> > 
> > 
> > Next I did: I tried to bisect between 3.18 and 3.19 with your patchset
> > applied, because w/ this patchset applied, the problem can be seen
> > easily and directly on boot. Unfortunately, this does work only a few
> > git bisect rounds until I got stuck because of interferences with your
> > extra patches applied:
> 
> [Resolved the problems written at the last post.]
> 
> Bisecting ended here:
> 
> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=34b48db66e08ca1c1bc07cf305d672ac940268dc
> 
> block: remove artifical max_hw_sectors cap
> 
> 
> Removing this patch on 3.19 and 4.1 make things working again. Didn't
> test 4.0, but I think it's the same. No more AMD-Vi IO_PAGE_FAULTS  with
> that patch reverted.
> 
> 
> Please check why this patch triggers AMD-Vi IO_PAGE_FAULTS.
> 
> 
> Thanks,
> kind regards,
> Andreas Hartmann

I would submit this bug to maintainers of AMD-Vi. They understand the 
hardware, so they should tell why do large I/O requests result in 
IO_PAGE_FAULTs.

It is probably bug either in AMD-Vi driver or in hardware.

Mikulas

Re: [dm-devel] AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Andreas Hartmann]

On 08/01/2015 at 19:57, Mikulas Patocka wrote:
>
>
> On Sun, 2 Aug 2015, Andreas Hartmann wrote:
>
>> On 08/01/2015 at 04:20 PM Andreas Hartmann wrote:
>>> On 07/28/2015 at 09:29 PM, Mike Snitzer wrote:
>>> [...]
>>>> Mikulas was saying to biect what is causing ATA to fail.
>>>
>>> Some good news and some bad news. The good news first:
>>>
>>> Your patchset
>>>
>>> f3396c58fd8442850e759843457d78b6ec3a9589,
>>> cf2f1abfbd0dba701f7f16ef619e4d2485de3366,
>>> 7145c241a1bf2841952c3e297c4080b357b3e52d,
>>> 94f5e0243c48aa01441c987743dc468e2d6eaca2,
>>> dc2676210c425ee8e5cb1bec5bc84d004ddf4179,
>>> 0f5d8e6ee758f7023e4353cca75d785b2d4f6abe,
>>> b3c5fd3052492f1b8d060799d4f18be5a5438add
>>>
>>> seems to work fine w/ 3.18.19 !!
>>>
>>> Why did I test it with 3.18.x now? Because I suddenly got two ata errors
>>> (ata1 and ata2) with clean 3.19.8 (w/o the AMD-Vi IO_PAGE_FAULTs) during
>>> normal operation. This means: 3.19 must already be broken, too.
>>>
>>> Therefore, I applied your patchset to 3.18.x and it seems to work like a
>>> charme - I don't get any AMD-Vi IO_PAGE_FAULTs on boot and no ata errors
>>> (until now).
>>>
>>>
>>> Next I did: I tried to bisect between 3.18 and 3.19 with your patchset
>>> applied, because w/ this patchset applied, the problem can be seen
>>> easily and directly on boot. Unfortunately, this does work only a few
>>> git bisect rounds until I got stuck because of interferences with your
>>> extra patches applied:
>>
>> [Resolved the problems written at the last post.]
>>
>> Bisecting ended here:
>>
>> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=34b48db66e08ca1c1bc07cf305d672ac940268dc
>>
>> block: remove artifical max_hw_sectors cap
>>
>>
>> Removing this patch on 3.19 and 4.1 make things working again. Didn't
>> test 4.0, but I think it's the same. No more AMD-Vi IO_PAGE_FAULTS  with
>> that patch reverted.
>>
>>
>> Please check why this patch triggers AMD-Vi IO_PAGE_FAULTS.
>>
>>
>> Thanks,
>> kind regards,
>> Andreas Hartmann
>
> I would submit this bug to maintainers of AMD-Vi. They understand the
> hardware, so they should tell why do large I/O requests result in
> IO_PAGE_FAULTs.

You forgot the ata errors ... . They are gone, too. I got these ata 
errors on 3.19 w/o your patchset and w/o AMD-Vi IO_PAGES_FAULTs, too.



Regards,
Andreas

Re: [dm-devel] AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Joerg Roedel]

On Sun, Aug 02, 2015 at 08:48:06PM +0200, Andreas Hartmann wrote:
> >>https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=34b48db66e08ca1c1bc07cf305d672ac940268dc
> >>
> >>block: remove artifical max_hw_sectors cap

Looking at the patch, it seems to me that it just uncovered a bug
elsewhere. It looks like an underlying driver doesn't expect the
big io-requests that the patch enables and does not dma-map the whole
target buffer, causing the IO_PAGE_FAULTs later.


	Joerg

Re: AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Mike Snitzer]

On Mon, Aug 03 2015 at  4:12am -0400,
Joerg Roedel <joro@8bytes.org> wrote:

> On Sun, Aug 02, 2015 at 08:48:06PM +0200, Andreas Hartmann wrote:
> > >>https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=34b48db66e08ca1c1bc07cf305d672ac940268dc
> > >>
> > >>block: remove artifical max_hw_sectors cap
> 
> Looking at the patch, it seems to me that it just uncovered a bug
> elsewhere. It looks like an underlying driver doesn't expect the
> big io-requests that the patch enables and does not dma-map the whole
> target buffer, causing the IO_PAGE_FAULTs later.

That patch has caused issues elsewhere too, see this 'Revert "block:
remove artifical max_hw_sectors cap"' thread (if/when lkml.org
cooperates): https://lkml.org/lkml/2015/7/20/572

But it could be that there is a need for a horkage fix for this specific
hardware? something comparable to this?:
http://git.kernel.org/linus/af34d637637eabaf49406eb35c948cd51ba262a6

We are running out of time to fix these whack-a-mole issues in 4.2
though.

Re: AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Jeff Moyer]

Mike Snitzer <snitzer@redhat.com> writes:

> On Mon, Aug 03 2015 at  4:12am -0400,
> Joerg Roedel <joro@8bytes.org> wrote:
>
>> On Sun, Aug 02, 2015 at 08:48:06PM +0200, Andreas Hartmann wrote:
>> > >>https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=34b48db66e08ca1c1bc07cf305d672ac940268dc
>> > >>
>> > >>block: remove artifical max_hw_sectors cap
>> 
>> Looking at the patch, it seems to me that it just uncovered a bug
>> elsewhere. It looks like an underlying driver doesn't expect the
>> big io-requests that the patch enables and does not dma-map the whole
>> target buffer, causing the IO_PAGE_FAULTs later.
>
> That patch has caused issues elsewhere too, see this 'Revert "block:
> remove artifical max_hw_sectors cap"' thread (if/when lkml.org
> cooperates): https://lkml.org/lkml/2015/7/20/572
>
> But it could be that there is a need for a horkage fix for this specific
> hardware? something comparable to this?:
> http://git.kernel.org/linus/af34d637637eabaf49406eb35c948cd51ba262a6
>
> We are running out of time to fix these whack-a-mole issues in 4.2
> though.

CC-ing linux-ide.  Original dm-devel posting:
  https://www.redhat.com/archives/dm-devel/2015-July/msg00178.html

Andreas, I would be curious to know what the value of
/sys/block/sdX/queue/max_hw_sectors_kb is for the affected disks.

Cheers,
Jeff

Re: AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Andreas Hartmann]

On 08/04/2015 at 06:10 PM Jeff Moyer wrote:
> Mike Snitzer <snitzer@redhat.com> writes:
> 
>> On Mon, Aug 03 2015 at  4:12am -0400,
>> Joerg Roedel <joro@8bytes.org> wrote:
>>
>>> On Sun, Aug 02, 2015 at 08:48:06PM +0200, Andreas Hartmann wrote:
>>>>>> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=34b48db66e08ca1c1bc07cf305d672ac940268dc
>>>>>>
>>>>>> block: remove artifical max_hw_sectors cap
>>>
>>> Looking at the patch, it seems to me that it just uncovered a bug
>>> elsewhere. It looks like an underlying driver doesn't expect the
>>> big io-requests that the patch enables and does not dma-map the whole
>>> target buffer, causing the IO_PAGE_FAULTs later.
>>
>> That patch has caused issues elsewhere too, see this 'Revert "block:
>> remove artifical max_hw_sectors cap"' thread (if/when lkml.org
>> cooperates): https://lkml.org/lkml/2015/7/20/572
>>
>> But it could be that there is a need for a horkage fix for this specific
>> hardware? something comparable to this?:
>> http://git.kernel.org/linus/af34d637637eabaf49406eb35c948cd51ba262a6
>>
>> We are running out of time to fix these whack-a-mole issues in 4.2
>> though.
> 
> CC-ing linux-ide.  Original dm-devel posting:
>   https://www.redhat.com/archives/dm-devel/2015-July/msg00178.html
> 
> Andreas, I would be curious to know what the value of
> /sys/block/sdX/queue/max_hw_sectors_kb is for the affected disks.


It's always 32767.

Where does this value come from? Is it empirical or is it calculated (on
base of which parameters)?


Devices are:

1 x Corsair Force GT (SSD)
2 x ST3000DM001-1CH166 (rotational) (WD)


How can I set a higher max_sectors_kb on boot before the partitions are
mounted (-> as kernel option)? Mounting the partitions here (w/ systemd)
is the easiest and best way to trigger the problem!



Thanks,
Andreas

Re: AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Andreas Hartmann]

On 08/04/2015 at 08:11 PM, Andreas Hartmann wrote:
> On 08/04/2015 at 06:10 PM Jeff Moyer wrote:
>> Mike Snitzer <snitzer@redhat.com> writes:
>>
>>> On Mon, Aug 03 2015 at  4:12am -0400,
>>> Joerg Roedel <joro@8bytes.org> wrote:
>>>
>>>> On Sun, Aug 02, 2015 at 08:48:06PM +0200, Andreas Hartmann wrote:
>>>>>>> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=34b48db66e08ca1c1bc07cf305d672ac940268dc
>>>>>>>
>>>>>>> block: remove artifical max_hw_sectors cap
>>>>
>>>> Looking at the patch, it seems to me that it just uncovered a bug
>>>> elsewhere. It looks like an underlying driver doesn't expect the
>>>> big io-requests that the patch enables and does not dma-map the whole
>>>> target buffer, causing the IO_PAGE_FAULTs later.
>>>
>>> That patch has caused issues elsewhere too, see this 'Revert "block:
>>> remove artifical max_hw_sectors cap"' thread (if/when lkml.org
>>> cooperates): https://lkml.org/lkml/2015/7/20/572
>>>
>>> But it could be that there is a need for a horkage fix for this specific
>>> hardware? something comparable to this?:
>>> http://git.kernel.org/linus/af34d637637eabaf49406eb35c948cd51ba262a6
>>>
>>> We are running out of time to fix these whack-a-mole issues in 4.2
>>> though.
>>
>> CC-ing linux-ide.  Original dm-devel posting:
>>   https://www.redhat.com/archives/dm-devel/2015-July/msg00178.html
>>
>> Andreas, I would be curious to know what the value of
>> /sys/block/sdX/queue/max_hw_sectors_kb is for the affected disks.
> 
> 
> It's always 32767.
> 
> Where does this value come from? Is it empirical or is it calculated (on
> base of which parameters)?
> 
> 
> Devices are:
> 
> 1 x Corsair Force GT (SSD)
> 2 x ST3000DM001-1CH166 (rotational) (WD)
> 
> 
> How can I set a higher max_sectors_kb on boot before the partitions are
> mounted (-> as kernel option)? Mounting the partitions here (w/ systemd)
> is the easiest and best way to trigger the problem!

Please - no idea how to set higher max_sectors_kb before mounting of
partitions? Come on! I want to test exactly this situation!


Regards,
Andreas

Re: [dm-devel] AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Andreas Hartmann]

On 08/02/2015 at 07:57 PM, Mikulas Patocka wrote:
> 
> 
> On Sun, 2 Aug 2015, Andreas Hartmann wrote:
> 
>> On 08/01/2015 at 04:20 PM Andreas Hartmann wrote:
>>> On 07/28/2015 at 09:29 PM, Mike Snitzer wrote:
>>> [...]
>>>> Mikulas was saying to biect what is causing ATA to fail.
>>>
>>> Some good news and some bad news. The good news first:
>>>
>>> Your patchset
>>>
>>> f3396c58fd8442850e759843457d78b6ec3a9589,
>>> cf2f1abfbd0dba701f7f16ef619e4d2485de3366,
>>> 7145c241a1bf2841952c3e297c4080b357b3e52d,
>>> 94f5e0243c48aa01441c987743dc468e2d6eaca2,
>>> dc2676210c425ee8e5cb1bec5bc84d004ddf4179,
>>> 0f5d8e6ee758f7023e4353cca75d785b2d4f6abe,
>>> b3c5fd3052492f1b8d060799d4f18be5a5438add
>>>
>>> seems to work fine w/ 3.18.19 !!
>>>
>>> Why did I test it with 3.18.x now? Because I suddenly got two ata errors
>>> (ata1 and ata2) with clean 3.19.8 (w/o the AMD-Vi IO_PAGE_FAULTs) during
>>> normal operation. This means: 3.19 must already be broken, too.
>>>
>>> Therefore, I applied your patchset to 3.18.x and it seems to work like a
>>> charme - I don't get any AMD-Vi IO_PAGE_FAULTs on boot and no ata errors
>>> (until now).
>>>
>>>
>>> Next I did: I tried to bisect between 3.18 and 3.19 with your patchset
>>> applied, because w/ this patchset applied, the problem can be seen
>>> easily and directly on boot. Unfortunately, this does work only a few
>>> git bisect rounds until I got stuck because of interferences with your
>>> extra patches applied:
>>
>> [Resolved the problems written at the last post.]
>>
>> Bisecting ended here:
>>
>> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=34b48db66e08ca1c1bc07cf305d672ac940268dc
>>
>> block: remove artifical max_hw_sectors cap
>>
>>
>> Removing this patch on 3.19 and 4.1 make things working again. Didn't
>> test 4.0, but I think it's the same. No more AMD-Vi IO_PAGE_FAULTS  with
>> that patch reverted.

After long period of testing, I now can say, that max_sectors_kb can be
set to 1024 - higher values produce AMD-Vi IO_PAGE_FAULTS and ata faults.


This patch "sd: Fix maximum I/O size for BLOCK_PC requests"[1] as part
of 4.1.7 produces ata / AMD-Vi IO_PAGE_FAULTS already during boot, too -
no matter if "block: remove artifical max_hw_sectors cap"[2] has been
applied or not.


Next I tested was "dm crypt: constrain crypt device's max_segment_size
to PAGE_SIZE" patch[3] applied to an unchanged 4.1.7 kernel w/o setting
max_sectors_kb to 1024.

Interesting effect was, that booting has been fine, but I could see lots
of ata errors afterwards as soon as there is load on the md raid 1
(during kernel compile e.g.), which is built on *rotational* disks:


[  367.264873] ata2.00: exception Emask 0x0 SAct 0x7fbfffff SErr 0x0
action 0x6 frozen
[  367.264883] ata2.00: failed command: WRITE FPDMA QUEUED
[  367.264893] ata2.00: cmd 61/40:00:b0:7b:d4/05:00:06:00:00/40 tag 0
ncq 688128 out
[  367.264893]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask
0x4 (timeout)
[  367.264899] ata2.00: status: { DRDY }
...
[  367.265332] ata2.00: failed command: WRITE FPDMA QUEUED
[  367.265339] ata2.00: cmd 61/40:f0:30:71:d4/05:00:06:00:00/40 tag 30
ncq 688128 out
[  367.265339]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask
0x4 (timeout)
[  367.265343] ata2.00: status: { DRDY }
[  367.265350] ata2: hard resetting link
[  367.775330] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[  367.776970] ata2.00: configured for UDMA/133
[  367.776997] ata2.00: device reported invalid CHS sector 0
...
[  367.777761] ata2: EH complete


Iow: Using an unpatched kernel >= 3.19 means high risk to break
filesystems if there are given some yet unknown conditions [4].

>>
>>
>> Please check why this patch triggers AMD-Vi IO_PAGE_FAULTS.
> 
> I would submit this bug to maintainers of AMD-Vi. They understand the 
> hardware, so they should tell why do large I/O requests result in 
> IO_PAGE_FAULTs.
> 
> It is probably bug either in AMD-Vi driver or in hardware.

Until now, I didn't hear anything from the maintainers of AMD-Vi.


Regards,
Andreas Hartmann


[1] http://thread.gmane.org/gmane.linux.kernel.commits.head/538464
[2]
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=34b48db66e08ca1c1bc07cf305d672ac940268dc
[3]
http://news.gmane.org/find-root.php?group=gmane.linux.kernel&article=2036495
[4] http://thread.gmane.org/gmane.linux.kernel.pci/43851/focus=44011

Re: [dm-devel] AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Joerg Roedel]

On Sun, Sep 20, 2015 at 08:50:40AM +0200, Andreas Hartmann wrote:
> > I would submit this bug to maintainers of AMD-Vi. They understand the 
> > hardware, so they should tell why do large I/O requests result in 
> > IO_PAGE_FAULTs.
> > 
> > It is probably bug either in AMD-Vi driver or in hardware.
> 
> Until now, I didn't hear anything from the maintainers of AMD-Vi.

What do you mean by this? I've been commenting on this issue in the
past and I thought we agreed that this is no issue of the IOMMU driver.

It it were, bisection should lead to a commit that breaks it, but there
are no commits between v3.18 and v3.19 in the AMD IOMMU driver touching
the DMA-API path.


	Joerg

Re: [dm-devel] AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Mikulas Patocka]

On Tue, 29 Sep 2015, Joerg Roedel wrote:

> On Sun, Sep 20, 2015 at 08:50:40AM +0200, Andreas Hartmann wrote:
> > > I would submit this bug to maintainers of AMD-Vi. They understand the 
> > > hardware, so they should tell why do large I/O requests result in 
> > > IO_PAGE_FAULTs.
> > > 
> > > It is probably bug either in AMD-Vi driver or in hardware.
> > 
> > Until now, I didn't hear anything from the maintainers of AMD-Vi.
> 
> What do you mean by this? I've been commenting on this issue in the
> past and I thought we agreed that this is no issue of the IOMMU driver.
> 
> It it were, bisection should lead to a commit that breaks it, but there
> are no commits between v3.18 and v3.19 in the AMD IOMMU driver touching
> the DMA-API path.
> 
> 	Joerg

I don't know why are you so certain that the bug in not in AMD-Vi IOMMU.

There was a patch (34b48db66e08ca1c1bc07cf305d672ac940268dc) that 
increased default block request size. That patch triggers AMD-Vi page 
faults. The bug may be in ATA driver, in ATA controller on in AMD-Vi 
driver or hardware. I didn't see anything in that thread that proves that 
the bug in not in AMD-Vi IOMMU.

The bug probably existed even before kernel 3.19, but it was masked by the 
fact that I/O request size was artifically capped. Bisecting probably 
won't find it, as it may have existed since ever.

Mikulas

Re: [dm-devel] AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Joerg Roedel]

On Tue, Sep 29, 2015 at 11:58:10AM -0400, Mikulas Patocka wrote:
> On Tue, 29 Sep 2015, Joerg Roedel wrote:
> There was a patch (34b48db66e08ca1c1bc07cf305d672ac940268dc) that 
> increased default block request size. That patch triggers AMD-Vi page 
> faults. The bug may be in ATA driver, in ATA controller on in AMD-Vi 
> driver or hardware. I didn't see anything in that thread that proves that 
> the bug in not in AMD-Vi IOMMU.
> 
> The bug probably existed even before kernel 3.19, but it was masked by the 
> fact that I/O request size was artifically capped. Bisecting probably 
> won't find it, as it may have existed since ever.

Okay, I see. But as long as the request-size is not bigger than 128MB
(the biggest chunk the AMD IOMMU driver can map at once), I don't see
how the IOMMU driver could be at fault.

Which ATA driver is in use when this happens and are there instructions
on how to reproduce the issue?

Alternativly someone who can reproduce it should trace the calls to
__map_single and __unmap_single in the AMD IOMMU driver to find out
whether the addresses which the faults happen on are really mapped, or
at least requested from the AMD IOMMU driver.



	Joerg

Re: [dm-devel] AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Andreas Hartmann]

On 09/29/2015 at 06:20 PM, Joerg Roedel wrote:
> On Tue, Sep 29, 2015 at 11:58:10AM -0400, Mikulas Patocka wrote:
>> On Tue, 29 Sep 2015, Joerg Roedel wrote:
>> There was a patch (34b48db66e08ca1c1bc07cf305d672ac940268dc) that 
>> increased default block request size. That patch triggers AMD-Vi page 
>> faults. The bug may be in ATA driver, in ATA controller on in AMD-Vi 
>> driver or hardware. I didn't see anything in that thread that proves that 
>> the bug in not in AMD-Vi IOMMU.
>>
>> The bug probably existed even before kernel 3.19, but it was masked by the 
>> fact that I/O request size was artifically capped. Bisecting probably 
>> won't find it, as it may have existed since ever.
> 
> Okay, I see. But as long as the request-size is not bigger than 128MB
> (the biggest chunk the AMD IOMMU driver can map at once), I don't see
> how the IOMMU driver could be at fault.
> 
> Which ATA driver is in use when this happens and are there instructions
> on how to reproduce the issue?
> 
> Alternativly someone who can reproduce it should trace the calls to
> __map_single and __unmap_single in the AMD IOMMU driver to find out
> whether the addresses which the faults happen on are really mapped, or
> at least requested from the AMD IOMMU driver.

How can I trace it?


Thanks,
Andreas

Re: [dm-devel] AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Joerg Roedel]

[-- Attachment #1: Type: text/plain, Size: 839 bytes --]

On Wed, Sep 30, 2015 at 04:52:47PM +0200, Andreas Hartmann wrote:
> > Alternativly someone who can reproduce it should trace the calls to
> > __map_single and __unmap_single in the AMD IOMMU driver to find out
> > whether the addresses which the faults happen on are really mapped, or
> > at least requested from the AMD IOMMU driver.
> 
> How can I trace it?

Please apply the attached debug patch on-top of Linux v4.3-rc3 and boot
the machine. After boot you run (as root):


	# cat /sys/kernel/debug/tracing/trace_pipe > trace-data

Please run this in a seperate shell an keep it running.

Then trigger the problem while the above command is running. When you
triggered it, please send me the (compressed) trace-data file, full
dmesg and output of lspci on the box.

Please let me know if you have further questions.


Thanks,

	Joerg


[-- Attachment #2: iommu-debug.patch --]
[-- Type: text/x-diff, Size: 2298 bytes --]

diff --git a/drivers/iommu/amd_iommu.c b/drivers/iommu/amd_iommu.c
index f82060e7..0002e79 100644
--- a/drivers/iommu/amd_iommu.c
+++ b/drivers/iommu/amd_iommu.c
@@ -2465,6 +2465,7 @@ static dma_addr_t __map_single(struct device *dev,
 {
 	dma_addr_t offset = paddr & ~PAGE_MASK;
 	dma_addr_t address, start, ret;
+	phys_addr_t old_paddr = paddr;
 	unsigned int pages;
 	unsigned long align_mask = 0;
 	int i;
@@ -2521,6 +2522,8 @@ retry:
 		domain_flush_pages(&dma_dom->domain, address, size);
 
 out:
+	trace_printk("%s: mapped %llx paddr %llx size %zu\n",
+			dev_name(dev), address, old_paddr, size);
 	return address;
 
 out_unmap:
@@ -2532,6 +2535,9 @@ out_unmap:
 
 	dma_ops_free_addresses(dma_dom, address, pages);
 
+	trace_printk("%s: return DMA_ERROR_CODE paddr %llx size %zu\n",
+			dev_name(dev), old_paddr, size);
+
 	return DMA_ERROR_CODE;
 }
 
@@ -2628,6 +2634,8 @@ static void unmap_page(struct device *dev, dma_addr_t dma_addr, size_t size,
 
 	spin_lock_irqsave(&domain->lock, flags);
 
+	trace_printk("%s: unmap dma_addr %llx size %zu\n",
+			dev_name(dev), dma_addr, size);
 	__unmap_single(domain->priv, dma_addr, size, dir);
 
 	domain_flush_complete(domain);
@@ -2683,9 +2691,13 @@ out:
 	return mapped_elems;
 unmap:
 	for_each_sg(sglist, s, mapped_elems, i) {
-		if (s->dma_address)
+		if (s->dma_address) {
+			trace_printk("%s: unmap dma_addr %llx size %u\n",
+					dev_name(dev), s->dma_address,
+					s->dma_length);
 			__unmap_single(domain->priv, s->dma_address,
 				       s->dma_length, dir);
+		}
 		s->dma_address = s->dma_length = 0;
 	}
 
@@ -2716,6 +2728,9 @@ static void unmap_sg(struct device *dev, struct scatterlist *sglist,
 	spin_lock_irqsave(&domain->lock, flags);
 
 	for_each_sg(sglist, s, nelems, i) {
+	trace_printk("%s: unmap dma_addr %llx size %u\n",
+			dev_name(dev), s->dma_address, s->dma_length);
+
 		__unmap_single(domain->priv, s->dma_address,
 			       s->dma_length, dir);
 		s->dma_address = s->dma_length = 0;
@@ -2813,6 +2828,9 @@ static void free_coherent(struct device *dev, size_t size,
 
 	spin_lock_irqsave(&domain->lock, flags);
 
+	trace_printk("%s: unmap dma_addr %llx size %zu\n",
+			dev_name(dev), dma_addr, size);
+
 	__unmap_single(domain->priv, dma_addr, size, DMA_BIDIRECTIONAL);
 
 	domain_flush_complete(domain);

Re: [dm-devel] AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Andreas Hartmann]

[-- Attachment #1: Type: text/plain, Size: 1345 bytes --]

On 10/06/2015 at 12:13 PM, Joerg Roedel wrote:
> On Wed, Sep 30, 2015 at 04:52:47PM +0200, Andreas Hartmann wrote:
>>> Alternativly someone who can reproduce it should trace the calls to
>>> __map_single and __unmap_single in the AMD IOMMU driver to find out
>>> whether the addresses which the faults happen on are really mapped, or
>>> at least requested from the AMD IOMMU driver.
>>
>> How can I trace it?
> 
> Please apply the attached debug patch on-top of Linux v4.3-rc3 and boot
> the machine. After boot you run (as root):
> 
> 
> 	# cat /sys/kernel/debug/tracing/trace_pipe > trace-data
> 
> Please run this in a seperate shell an keep it running.
> 
> Then trigger the problem while the above command is running. When you
> triggered it, please send me the (compressed) trace-data file, full
> dmesg and output of lspci on the box.

Hmm, *seems* to work fine w/ 4.3-rc2. But I have to do some more tests
to be really sure.


W/ 4.1.10, the problem can be seen most always during boot (systemd) -
but at this point, it is difficult to trace. I have to take a closer
look to find a place to start the trace already during boot process.


But there is another problem w/ 4.3-rc2: Starting a VM w/ PCIe
passthrough doesn't work any more. I'm getting the attached null pointer
dereference and the machine hangs.


Thanks,
regards,
Andreas

[-- Attachment #2: trace --]
[-- Type: text/plain, Size: 21102 bytes --]

Oct  6 20:11:18 localhost kernel: [   32.461794] BUG: unable to handle kernel NULL pointer dereference at 00000000000000b8
Oct  6 20:11:18 localhost kernel: [   32.461853] IP: [<ffffffff8147a8a4>] do_detach+0x24/0xa0
Oct  6 20:11:18 localhost kernel: [   32.461888] PGD 0 
Oct  6 20:11:18 localhost kernel: [   32.461902] Oops: 0002 [#1] PREEMPT SMP 
Oct  6 20:11:18 localhost kernel: [   32.461929] Modules linked in: nf_log_ipv4 nf_log_common xt_LOG ipt_REJECT xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter ip_tables x_tables vfio_iommu_type1 vfio_pci vfio vfio_virqfd drbg ansi_cprng nfsd lockd grace nfs_acl auth_rpcgss sunrpc bridge stp llc tun it87 hwmon_vid snd_hda_codec_hdmi kvm_amd snd_hda_codec_realtek kvm snd_hda_codec_generic fam15h_power usb_storage snd_hda_intel pcspkr serio_raw snd_hda_codec edac_core snd_hda_core edac_mce_amd k10temp snd_hwdep snd_pcm firewire_ohci snd_seq e100 firewire_core crc_itu_t amdkfd sp5100_tco amd_iommu_v2 i2c_piix4 mxm_wmi sr_mod cdrom radeon snd_timer snd_seq_device snd ttm drm_kms_helper xhci_pci drm r8169 xhci_hcd mii fb_sys_fops sysimgblt sysfillrect syscopyarea soundcore i2c_algo_bit shpchp tpm_infineon tpm_tis tpm fjes 8250_fintek wmi button acpi_cpufreq sg thermal xfs libcrc32c linear crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ohci_pci processor scsi_dh_rdac scsi_dh_hp_sw scsi_dh_emc scsi_dh_alua raid456 async_raid6_recov async_pq async_xor xor async_memcpy async_tx raid6_pq raid10 raid1 raid0 md_mod dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_crypt dm_mod aesni_intel ablk_helper cryptd lrw gf128mul glue_helper aes_x86_64 ata_generic pata_atiixp
Oct  6 20:11:18 localhost kernel: [   32.462728] CPU: 0 PID: 9374 Comm: qemu-system-x86 Not tainted 4.3.0-rc2-4-desktop #1
Oct  6 20:11:18 localhost kernel: [   32.462767] Hardware name: Gigabyte Technology Co., Ltd. GA-990XA-UD3/GA-990XA-UD3, BIOS F14b 01/24/2013
Oct  6 20:11:18 localhost kernel: [   32.462814] task: ffff8805f4ecc080 ti: ffff8805e1ec4000 task.ti: ffff8805e1ec4000
Oct  6 20:11:18 localhost kernel: [   32.462851] RIP: 0010:[<ffffffff8147a8a4>]  [<ffffffff8147a8a4>] do_detach+0x24/0xa0
Oct  6 20:11:18 localhost kernel: [   32.462894] RSP: 0018:ffff8805e1ec7ca0  EFLAGS: 00010006
Oct  6 20:11:18 localhost kernel: [   32.462922] RAX: 0000000000000000 RBX: ffff880614bef640 RCX: 00000000000000ff
Oct  6 20:11:18 localhost kernel: [   32.462959] RDX: 0000000000000000 RSI: ffff88062e70c098 RDI: ffff880614bef640
Oct  6 20:11:18 localhost kernel: [   32.462998] RBP: ffff8805e1ec7ca8 R08: ffff880614befc40 R09: 0000000000000000
Oct  6 20:11:18 localhost kernel: [   32.463033] R10: 0000000000000000 R11: ffffffff81a58df8 R12: ffff880614befc40
Oct  6 20:11:18 localhost kernel: [   32.463071] R13: ffff8806144b9858 R14: 0000000000000082 R15: ffff88062e70c098
Oct  6 20:11:18 localhost kernel: [   32.463110] FS:  00007f27cc824b80(0000) GS:ffff88062ec00000(0000) knlGS:0000000000000000
Oct  6 20:11:18 localhost kernel: [   32.463148] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct  6 20:11:18 localhost kernel: [   32.463177] CR2: 00000000000000b8 CR3: 00000000c8011000 CR4: 00000000000406f0
Oct  6 20:11:18 localhost kernel: [   32.463211] Stack:
Oct  6 20:11:18 localhost kernel: [   32.463223]  ffff880614bef640 ffff8805e1ec7cd8 ffffffff8147a96c ffff880614befc40
Oct  6 20:11:18 localhost kernel: [   32.463268]  ffff88062e70c098 0000000000000286 ffff8806144b9800 ffff8805e1ec7d08
Oct  6 20:11:18 localhost kernel: [   32.463310]  ffffffff8147aaf5 ffff880615dd0bc0 ffff8805e8354a00 ffff880614befc40
Oct  6 20:11:18 localhost kernel: [   32.463355] Call Trace:
Oct  6 20:11:18 localhost kernel: [   32.463377]  [<ffffffff8147a96c>] __detach_device+0x4c/0x80
Oct  6 20:11:18 localhost kernel: [   32.463412]  [<ffffffff8147aaf5>] detach_device+0x35/0xa0
Oct  6 20:11:18 localhost kernel: [   32.463444]  [<ffffffff8147b706>] amd_iommu_attach_device+0x66/0x2b0
Oct  6 20:11:18 localhost kernel: [   32.463481]  [<ffffffff81475d8e>] __iommu_attach_device+0x1e/0x80
Oct  6 20:11:18 localhost kernel: [   32.463513]  [<ffffffff81477013>] __iommu_attach_group+0x53/0x80
Oct  6 20:11:18 localhost kernel: [   32.463547]  [<ffffffff8147706b>] iommu_attach_group+0x2b/0x40
Oct  6 20:11:18 localhost kernel: [   32.463583]  [<ffffffffa07e9407>] vfio_iommu_type1_attach_group+0x187/0x4f8 [vfio_iommu_type1]
Oct  6 20:11:18 localhost kernel: [   32.463655]  [<ffffffffa07297e8>] vfio_fops_unl_ioctl+0x1b8/0x290 [vfio]
Oct  6 20:11:18 localhost kernel: [   32.463699]  [<ffffffff811f81cd>] do_vfs_ioctl+0x2cd/0x4c0
Oct  6 20:11:18 localhost kernel: [   32.463740]  [<ffffffff811f8439>] SyS_ioctl+0x79/0x90
Oct  6 20:11:18 localhost kernel: [   32.463775]  [<ffffffff816b3936>] entry_SYSCALL_64_fastpath+0x16/0x75
Oct  6 20:11:18 localhost kernel: [   32.466223] DWARF2 unwinder stuck at entry_SYSCALL_64_fastpath+0x16/0x75
Oct  6 20:11:18 localhost kernel: [   32.466258] 
Oct  6 20:11:18 localhost kernel: [   32.466267] Leftover inexact backtrace:
Oct  6 20:11:18 localhost kernel: [   32.466267] 
Oct  6 20:11:18 localhost kernel: [   32.466298] Code: 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 8b 05 23 c7 d9 00 48 89 e5 53 0f b7 57 40 48 89 fb 48 8b 04 d0 48 63 50 10 48 8b 47 38 <83> ac 90 b8 00 00 00 01 48 8b 47 38 83 a8 b4 00 00 00 01 48 8b 
Oct  6 20:11:18 localhost kernel: [   32.466496] RIP  [<ffffffff8147a8a4>] do_detach+0x24/0xa0
Oct  6 20:11:18 localhost kernel: [   32.466527]  RSP <ffff8805e1ec7ca0>
Oct  6 20:11:18 localhost kernel: [   32.466547] CR2: 00000000000000b8
Oct  6 20:11:18 localhost kernel: [   32.476984] ---[ end trace 09ac28af2000b365 ]---
Oct  6 20:11:18 localhost kernel: [   32.477031] note: qemu-system-x86[9374] exited with preempt_count 2
Oct  6 20:11:19 localhost kernel: [   32.577276] walk_shadow_page_get_mmio_spte: detect reserved bits on spte, addr 0xb8000, dump hierarchy:
Oct  6 20:11:19 localhost kernel: [   32.577279] ------ spte 0x5db618027 level 4.
Oct  6 20:11:19 localhost kernel: [   32.577281] ------ spte 0x5db619027 level 3.
Oct  6 20:11:19 localhost kernel: [   32.577281] ------ spte 0x5db61a027 level 2.
Oct  6 20:11:19 localhost kernel: [   32.577282] ------ spte 0xffff0000000b8f6f level 1.
Oct  6 20:11:19 localhost kernel: [   32.577283] ------------[ cut here ]------------
Oct  6 20:11:19 localhost kernel: [   32.577301] WARNING: CPU: 2 PID: 9389 at ../arch/x86/kvm/mmu.c:3385 handle_mmio_page_fault.isra.85+0x2c/0x31 [kvm]()
Oct  6 20:11:19 localhost kernel: [   32.577302] Modules linked in: nf_log_ipv4 nf_log_common xt_LOG ipt_REJECT xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter ip_tables x_tables vfio_iommu_type1 vfio_pci vfio vfio_virqfd drbg ansi_cprng nfsd lockd grace nfs_acl auth_rpcgss sunrpc bridge stp llc tun it87 hwmon_vid snd_hda_codec_hdmi kvm_amd snd_hda_codec_realtek kvm snd_hda_codec_generic fam15h_power usb_storage snd_hda_intel pcspkr serio_raw snd_hda_codec edac_core snd_hda_core edac_mce_amd k10temp snd_hwdep snd_pcm firewire_ohci snd_seq e100 firewire_core crc_itu_t amdkfd sp5100_tco amd_iommu_v2 i2c_piix4 mxm_wmi sr_mod cdrom radeon snd_timer snd_seq_device snd ttm drm_kms_helper xhci_pci drm r8169 xhci_hcd mii fb_sys_fops sysimgblt sysfillrect syscopyarea soundcore i2c_algo_bit shpchp tpm_infineon tpm_tis tpm fjes 8250_fintek wmi button acpi_cpufreq sg thermal xfs libcrc32c linear crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ohci_pci processor scsi_dh_rdac scsi_dh_hp_sw scsi_dh_emc scsi_dh_alua raid456 async_raid6_recov async_pq async_xor xor async_memcpy async_tx raid6_pq raid10 raid1 raid0 md_mod dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_crypt dm_mod aesni_intel ablk_helper cryptd lrw gf128mul glue_helper aes_x86_64 ata_generic pata_atiixp
Oct  6 20:11:19 localhost kernel: [   32.577357] CPU: 2 PID: 9389 Comm: qemu-system-x86 Tainted: G      D         4.3.0-rc2-4-desktop #1
Oct  6 20:11:19 localhost kernel: [   32.577358] Hardware name: Gigabyte Technology Co., Ltd. GA-990XA-UD3/GA-990XA-UD3, BIOS F14b 01/24/2013
Oct  6 20:11:19 localhost kernel: [   32.577360]  ffffffffa083f3ce ffff8805e0f43ba0 ffffffff81368633 0000000000000000
Oct  6 20:11:19 localhost kernel: [   32.577362]  ffff8805e0f43bd8 ffffffff8106a628 ffff8805e2a70040 00000000000b8000
Oct  6 20:11:19 localhost kernel: [   32.577364]  0000000000000000 000000000000000f 000000000000000f ffff8805e0f43be8
Oct  6 20:11:19 localhost kernel: [   32.577366] Call Trace:
Oct  6 20:11:19 localhost kernel: [   32.577374]  [<ffffffff810085ee>] try_stack_unwind+0x17e/0x190
Oct  6 20:11:19 localhost kernel: [   32.577379]  [<ffffffff8100737f>] dump_trace+0x8f/0x3b0
Oct  6 20:11:19 localhost kernel: [   32.577382]  [<ffffffff8100864d>] show_trace_log_lvl+0x4d/0x60
Oct  6 20:11:19 localhost kernel: [   32.577385]  [<ffffffff810077a1>] show_stack_log_lvl+0x101/0x190
Oct  6 20:11:19 localhost kernel: [   32.577387]  [<ffffffff810086a5>] show_stack+0x25/0x50
Oct  6 20:11:19 localhost kernel: [   32.577390]  [<ffffffff81368633>] dump_stack+0x4b/0x78
Oct  6 20:11:19 localhost kernel: [   32.577394]  [<ffffffff8106a628>] warn_slowpath_common+0x88/0xc0
Oct  6 20:11:19 localhost kernel: [   32.577397]  [<ffffffff8106a71a>] warn_slowpath_null+0x1a/0x20
Oct  6 20:11:19 localhost kernel: [   32.577408]  [<ffffffffa08318eb>] handle_mmio_page_fault.isra.85+0x2c/0x31 [kvm]
Oct  6 20:11:19 localhost kernel: [   32.577427]  [<ffffffffa0816106>] tdp_page_fault+0x246/0x260 [kvm]
Oct  6 20:11:19 localhost kernel: [   32.577441]  [<ffffffffa080fbd4>] kvm_mmu_page_fault+0x24/0x110 [kvm]
Oct  6 20:11:19 localhost kernel: [   32.577446]  [<ffffffffa0cdbfc9>] pf_interception+0xc9/0x150 [kvm_amd]
Oct  6 20:11:19 localhost kernel: [   32.577451]  [<ffffffffa0cdf5a0>] handle_exit+0x180/0x9b0 [kvm_amd]
Oct  6 20:11:19 localhost kernel: [   32.577462]  [<ffffffffa0805dd9>] vcpu_enter_guest+0x769/0xde0 [kvm]
Oct  6 20:11:19 localhost kernel: [   32.577475]  [<ffffffffa080c62a>] kvm_arch_vcpu_ioctl_run+0x2da/0x400 [kvm]
Oct  6 20:11:19 localhost kernel: [   32.577486]  [<ffffffffa07f4d8f>] kvm_vcpu_ioctl+0x30f/0x5c0 [kvm]
Oct  6 20:11:19 localhost kernel: [   32.577490]  [<ffffffff811f81cd>] do_vfs_ioctl+0x2cd/0x4c0
Oct  6 20:11:19 localhost kernel: [   32.577497]  [<ffffffff811f8439>] SyS_ioctl+0x79/0x90
Oct  6 20:11:19 localhost kernel: [   32.577500]  [<ffffffff816b3936>] entry_SYSCALL_64_fastpath+0x16/0x75
Oct  6 20:11:19 localhost kernel: [   32.578805] DWARF2 unwinder stuck at entry_SYSCALL_64_fastpath+0x16/0x75
Oct  6 20:11:19 localhost kernel: [   32.578806] 
Oct  6 20:11:19 localhost kernel: [   32.578806] Leftover inexact backtrace:
Oct  6 20:11:19 localhost kernel: [   32.578806] 
Oct  6 20:11:19 localhost kernel: [   32.578808] ---[ end trace 09ac28af2000b366 ]---
Oct  6 20:11:19 localhost kernel: [   32.655728] walk_shadow_page_get_mmio_spte: detect reserved bits on spte, addr 0xb8000, dump hierarchy:
Oct  6 20:11:19 localhost kernel: [   32.655731] ------ spte 0x5e6703027 level 4.
Oct  6 20:11:19 localhost kernel: [   32.655732] ------ spte 0x5e5fb8027 level 3.
Oct  6 20:11:19 localhost kernel: [   32.655733] ------ spte 0x5e5fb9027 level 2.
Oct  6 20:11:19 localhost kernel: [   32.655734] ------ spte 0xffff0000000b8f67 level 1.
Oct  6 20:11:19 localhost kernel: [   32.655735] ------------[ cut here ]------------
Oct  6 20:11:19 localhost kernel: [   32.655764] WARNING: CPU: 2 PID: 9390 at ../arch/x86/kvm/mmu.c:3385 handle_mmio_page_fault.isra.85+0x2c/0x31 [kvm]()
Oct  6 20:11:19 localhost kernel: [   32.655765] Modules linked in: nf_log_ipv4 nf_log_common xt_LOG ipt_REJECT xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter ip_tables x_tables vfio_iommu_type1 vfio_pci vfio vfio_virqfd drbg ansi_cprng nfsd lockd grace nfs_acl auth_rpcgss sunrpc bridge stp llc tun it87 hwmon_vid snd_hda_codec_hdmi kvm_amd snd_hda_codec_realtek kvm snd_hda_codec_generic fam15h_power usb_storage snd_hda_intel pcspkr serio_raw snd_hda_codec edac_core snd_hda_core edac_mce_amd k10temp snd_hwdep snd_pcm firewire_ohci snd_seq e100 firewire_core crc_itu_t amdkfd sp5100_tco amd_iommu_v2 i2c_piix4 mxm_wmi sr_mod cdrom radeon snd_timer snd_seq_device snd ttm drm_kms_helper xhci_pci drm r8169 xhci_hcd mii fb_sys_fops sysimgblt sysfillrect syscopyarea soundcore i2c_algo_bit shpchp tpm_infineon tpm_tis tpm fjes 8250_fintek wmi button acpi_cpufreq sg thermal xfs libcrc32c linear crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ohci_pci processor scsi_dh_rdac scsi_dh_hp_sw scsi_dh_emc scsi_dh_alua raid456 async_raid6_recov async_pq async_xor xor async_memcpy async_tx raid6_pq raid10 raid1 raid0 md_mod dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_crypt dm_mod aesni_intel ablk_helper cryptd lrw gf128mul glue_helper aes_x86_64 ata_generic pata_atiixp
Oct  6 20:11:19 localhost kernel: [   32.655823] CPU: 2 PID: 9390 Comm: qemu-system-x86 Tainted: G      D W       4.3.0-rc2-4-desktop #1
Oct  6 20:11:19 localhost kernel: [   32.655824] Hardware name: Gigabyte Technology Co., Ltd. GA-990XA-UD3/GA-990XA-UD3, BIOS F14b 01/24/2013
Oct  6 20:11:19 localhost kernel: [   32.655826]  ffffffffa083f3ce ffff8805e221fba0 ffffffff81368633 0000000000000000
Oct  6 20:11:19 localhost kernel: [   32.655828]  ffff8805e221fbd8 ffffffff8106a628 ffff8805e1e30080 00000000000b8000
Oct  6 20:11:19 localhost kernel: [   32.655830]  0000000000000000 000000000000000f 000000000000000f ffff8805e221fbe8
Oct  6 20:11:19 localhost kernel: [   32.655832] Call Trace:
Oct  6 20:11:19 localhost kernel: [   32.655840]  [<ffffffff810085ee>] try_stack_unwind+0x17e/0x190
Oct  6 20:11:19 localhost kernel: [   32.655845]  [<ffffffff8100737f>] dump_trace+0x8f/0x3b0
Oct  6 20:11:19 localhost kernel: [   32.655848]  [<ffffffff8100864d>] show_trace_log_lvl+0x4d/0x60
Oct  6 20:11:19 localhost kernel: [   32.655852]  [<ffffffff810077a1>] show_stack_log_lvl+0x101/0x190
Oct  6 20:11:19 localhost kernel: [   32.655864]  [<ffffffff810086a5>] show_stack+0x25/0x50
Oct  6 20:11:19 localhost kernel: [   32.655869]  [<ffffffff81368633>] dump_stack+0x4b/0x78
Oct  6 20:11:19 localhost kernel: [   32.655877]  [<ffffffff8106a628>] warn_slowpath_common+0x88/0xc0
Oct  6 20:11:19 localhost kernel: [   32.655881]  [<ffffffff8106a71a>] warn_slowpath_null+0x1a/0x20
Oct  6 20:11:19 localhost kernel: [   32.655894]  [<ffffffffa08318eb>] handle_mmio_page_fault.isra.85+0x2c/0x31 [kvm]
Oct  6 20:11:19 localhost kernel: [   32.655917]  [<ffffffffa0816106>] tdp_page_fault+0x246/0x260 [kvm]
Oct  6 20:11:19 localhost kernel: [   32.655940]  [<ffffffffa080fbd4>] kvm_mmu_page_fault+0x24/0x110 [kvm]
Oct  6 20:11:19 localhost kernel: [   32.655956]  [<ffffffffa0cdbfc9>] pf_interception+0xc9/0x150 [kvm_amd]
Oct  6 20:11:19 localhost kernel: [   32.655965]  [<ffffffffa0cdf5a0>] handle_exit+0x180/0x9b0 [kvm_amd]
Oct  6 20:11:19 localhost kernel: [   32.655983]  [<ffffffffa0805dd9>] vcpu_enter_guest+0x769/0xde0 [kvm]
Oct  6 20:11:19 localhost kernel: [   32.656000]  [<ffffffffa080c62a>] kvm_arch_vcpu_ioctl_run+0x2da/0x400 [kvm]
Oct  6 20:11:19 localhost kernel: [   32.656012]  [<ffffffffa07f4d8f>] kvm_vcpu_ioctl+0x30f/0x5c0 [kvm]
Oct  6 20:11:19 localhost kernel: [   32.656017]  [<ffffffff811f81cd>] do_vfs_ioctl+0x2cd/0x4c0
Oct  6 20:11:19 localhost kernel: [   32.656023]  [<ffffffff811f8439>] SyS_ioctl+0x79/0x90
Oct  6 20:11:19 localhost kernel: [   32.656028]  [<ffffffff816b3936>] entry_SYSCALL_64_fastpath+0x16/0x75
Oct  6 20:11:19 localhost kernel: [   32.657322] DWARF2 unwinder stuck at entry_SYSCALL_64_fastpath+0x16/0x75
Oct  6 20:11:19 localhost kernel: [   32.657324] 
Oct  6 20:11:19 localhost kernel: [   32.657324] Leftover inexact backtrace:
Oct  6 20:11:19 localhost kernel: [   32.657324] 
Oct  6 20:11:19 localhost kernel: [   32.657327] ---[ end trace 09ac28af2000b367 ]---
Oct  6 20:11:19 localhost kernel: [   32.715379] walk_shadow_page_get_mmio_spte: detect reserved bits on spte, addr 0xb8000, dump hierarchy:
Oct  6 20:11:19 localhost kernel: [   32.715382] ------ spte 0x5e6707027 level 4.
Oct  6 20:11:19 localhost kernel: [   32.715383] ------ spte 0x5e5781027 level 3.
Oct  6 20:11:19 localhost kernel: [   32.715384] ------ spte 0x5e2291027 level 2.
Oct  6 20:11:19 localhost kernel: [   32.715385] ------ spte 0xffff0000000b8f67 level 1.
Oct  6 20:11:19 localhost kernel: [   32.715386] ------------[ cut here ]------------
Oct  6 20:11:19 localhost kernel: [   32.715405] WARNING: CPU: 2 PID: 9393 at ../arch/x86/kvm/mmu.c:3385 handle_mmio_page_fault.isra.85+0x2c/0x31 [kvm]()
Oct  6 20:11:19 localhost kernel: [   32.715407] Modules linked in: nf_log_ipv4 nf_log_common xt_LOG ipt_REJECT xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter ip_tables x_tables vfio_iommu_type1 vfio_pci vfio vfio_virqfd drbg ansi_cprng nfsd lockd grace nfs_acl auth_rpcgss sunrpc bridge stp llc tun it87 hwmon_vid snd_hda_codec_hdmi kvm_amd snd_hda_codec_realtek kvm snd_hda_codec_generic fam15h_power usb_storage snd_hda_intel pcspkr serio_raw snd_hda_codec edac_core snd_hda_core edac_mce_amd k10temp snd_hwdep snd_pcm firewire_ohci snd_seq e100 firewire_core crc_itu_t amdkfd sp5100_tco amd_iommu_v2 i2c_piix4 mxm_wmi sr_mod cdrom radeon snd_timer snd_seq_device snd ttm drm_kms_helper xhci_pci drm r8169 xhci_hcd mii fb_sys_fops sysimgblt sysfillrect syscopyarea soundcore i2c_algo_bit shpchp tpm_infineon tpm_tis tpm fjes 8250_fintek wmi button acpi_cpufreq sg thermal xfs libcrc32c linear crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ohci_pci processor scsi_dh_rdac scsi_dh_hp_sw scsi_dh_emc scsi_dh_alua raid456 async_raid6_recov async_pq async_xor xor async_memcpy async_tx raid6_pq raid10 raid1 raid0 md_mod dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_crypt dm_mod aesni_intel ablk_helper cryptd lrw gf128mul glue_helper aes_x86_64 ata_generic pata_atiixp
Oct  6 20:11:19 localhost kernel: [   32.715461] CPU: 2 PID: 9393 Comm: qemu-system-x86 Tainted: G      D W       4.3.0-rc2-4-desktop #1
Oct  6 20:11:19 localhost kernel: [   32.715462] Hardware name: Gigabyte Technology Co., Ltd. GA-990XA-UD3/GA-990XA-UD3, BIOS F14b 01/24/2013
Oct  6 20:11:19 localhost kernel: [   32.715464]  ffffffffa083f3ce ffff8805e93afba0 ffffffff81368633 0000000000000000
Oct  6 20:11:19 localhost kernel: [   32.715466]  ffff8805e93afbd8 ffffffff8106a628 ffff8805ec604140 00000000000b8000
Oct  6 20:11:19 localhost kernel: [   32.715468]  0000000000000000 000000000000000f 000000000000000f ffff8805e93afbe8
Oct  6 20:11:19 localhost kernel: [   32.715470] Call Trace:
Oct  6 20:11:19 localhost kernel: [   32.715478]  [<ffffffff810085ee>] try_stack_unwind+0x17e/0x190
Oct  6 20:11:19 localhost kernel: [   32.715483]  [<ffffffff8100737f>] dump_trace+0x8f/0x3b0
Oct  6 20:11:19 localhost kernel: [   32.715486]  [<ffffffff8100864d>] show_trace_log_lvl+0x4d/0x60
Oct  6 20:11:19 localhost kernel: [   32.715488]  [<ffffffff810077a1>] show_stack_log_lvl+0x101/0x190
Oct  6 20:11:19 localhost kernel: [   32.715491]  [<ffffffff810086a5>] show_stack+0x25/0x50
Oct  6 20:11:19 localhost kernel: [   32.715494]  [<ffffffff81368633>] dump_stack+0x4b/0x78
Oct  6 20:11:19 localhost kernel: [   32.715498]  [<ffffffff8106a628>] warn_slowpath_common+0x88/0xc0
Oct  6 20:11:19 localhost kernel: [   32.715501]  [<ffffffff8106a71a>] warn_slowpath_null+0x1a/0x20
Oct  6 20:11:19 localhost kernel: [   32.715512]  [<ffffffffa08318eb>] handle_mmio_page_fault.isra.85+0x2c/0x31 [kvm]
Oct  6 20:11:19 localhost kernel: [   32.715532]  [<ffffffffa0816106>] tdp_page_fault+0x246/0x260 [kvm]
Oct  6 20:11:19 localhost kernel: [   32.715546]  [<ffffffffa080fbd4>] kvm_mmu_page_fault+0x24/0x110 [kvm]
Oct  6 20:11:19 localhost kernel: [   32.715551]  [<ffffffffa0cdbfc9>] pf_interception+0xc9/0x150 [kvm_amd]
Oct  6 20:11:19 localhost kernel: [   32.715556]  [<ffffffffa0cdf5a0>] handle_exit+0x180/0x9b0 [kvm_amd]
Oct  6 20:11:19 localhost kernel: [   32.715567]  [<ffffffffa0805dd9>] vcpu_enter_guest+0x769/0xde0 [kvm]
Oct  6 20:11:19 localhost kernel: [   32.715580]  [<ffffffffa080c62a>] kvm_arch_vcpu_ioctl_run+0x2da/0x400 [kvm]
Oct  6 20:11:19 localhost kernel: [   32.715590]  [<ffffffffa07f4d8f>] kvm_vcpu_ioctl+0x30f/0x5c0 [kvm]
Oct  6 20:11:19 localhost kernel: [   32.715595]  [<ffffffff811f81cd>] do_vfs_ioctl+0x2cd/0x4c0
Oct  6 20:11:19 localhost kernel: [   32.715602]  [<ffffffff811f8439>] SyS_ioctl+0x79/0x90
Oct  6 20:11:19 localhost kernel: [   32.715606]  [<ffffffff816b3936>] entry_SYSCALL_64_fastpath+0x16/0x75
Oct  6 20:11:19 localhost kernel: [   32.716901] DWARF2 unwinder stuck at entry_SYSCALL_64_fastpath+0x16/0x75
Oct  6 20:11:19 localhost kernel: [   32.716902] 
Oct  6 20:11:19 localhost kernel: [   32.716902] Leftover inexact backtrace:
Oct  6 20:11:19 localhost kernel: [   32.716902] 
Oct  6 20:11:19 localhost kernel: [   32.716904] ---[ end trace 09ac28af2000b368 ]---

Re: [dm-devel] AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Joerg Roedel]

On Tue, Oct 06, 2015 at 08:37:59PM +0200, Andreas Hartmann wrote:
> But there is another problem w/ 4.3-rc2: Starting a VM w/ PCIe
> passthrough doesn't work any more. I'm getting the attached null pointer
> dereference and the machine hangs.

Weird, probably a do_detach call for a device that is already detached.
Anyway, I can't reproduce this here on my two AMD IOMMU machines.  Can
you please boot the machine with amd_iommu_dump on the kernel command
line and send me dmesg after boot?

Also, which device are you trying to attach to the guest (pci
bus/device/function)?

Output of lspci might also be helpful.


Thanks,

	Joerg

Re: [dm-devel] AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Andreas Hartmann]

[-- Attachment #1: Type: text/plain, Size: 997 bytes --]

On 10/07/2015 at 05:40 PM Joerg Roedel wrote:
> On Tue, Oct 06, 2015 at 08:37:59PM +0200, Andreas Hartmann wrote:
>> But there is another problem w/ 4.3-rc2: Starting a VM w/ PCIe
>> passthrough doesn't work any more. I'm getting the attached null pointer
>> dereference and the machine hangs.
> 
> Weird, probably a do_detach call for a device that is already detached.
> Anyway, I can't reproduce this here on my two AMD IOMMU machines.  Can
> you please boot the machine with amd_iommu_dump on the kernel command
> line and send me dmesg after boot?

Binding the device to vfio isn't a problem (it's done before the vm is
started). The problem occurs during start of qemu-system-x86_64 (2.3.0).

The attached dmesg.out doesn't show the trace, but the desired iommu dump.

> Also, which device are you trying to attach to the guest (pci
> bus/device/function)?

See attached ath9k.device.

> Output of lspci might also be helpful.

I attached lscpi and dmesg.


Hope that helps,
thanks,
Andreas

[-- Attachment #2: dmesg.out.gz --]
[-- Type: application/x-gzip, Size: 20466 bytes --]

[-- Attachment #3: lspci.gz --]
[-- Type: application/x-gzip, Size: 2528 bytes --]

[-- Attachment #4: ath9k.device.gz --]
[-- Type: application/x-gzip, Size: 1567 bytes --]

Re: [dm-devel] AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Joerg Roedel]

On Wed, Oct 07, 2015 at 07:02:32PM +0200, Andreas Hartmann wrote:
> Binding the device to vfio isn't a problem (it's done before the vm is
> started). The problem occurs during start of qemu-system-x86_64 (2.3.0).
> 
> The attached dmesg.out doesn't show the trace, but the desired iommu dump.
> 
> > Also, which device are you trying to attach to the guest (pci
> > bus/device/function)?
> 
> See attached ath9k.device.

Hmm, can you also test this again with the v4.3-rc4 please? The device
you are attaching has its own group and no aliases, so I really can't
see how the trace could happen, and I can't reproduce it here either.

So I just want to make sure it is not a follow-on bug from the previous
problem.


Thanks,

	Joerg

P.S.: When you build the kernel with debug symbols and the problem
      occurs again, you can find out the source file and line where the
      bug happened with

      	$ objdump -Dlz --start-address=<rip> vmlinux | head

      Replace <rip> with the RIP from the kernel oops and the output
      should show you where in the source the bug comes from. The
      vmlinux file is in the kernels build directory.

Re: [dm-devel] AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Andreas Hartmann]

On 10/08/2015 at 07:30 PM Joerg Roedel wrote:
> On Wed, Oct 07, 2015 at 07:02:32PM +0200, Andreas Hartmann wrote:
>> Binding the device to vfio isn't a problem (it's done before the vm is
>> started). The problem occurs during start of qemu-system-x86_64 (2.3.0).
>>
>> The attached dmesg.out doesn't show the trace, but the desired iommu dump.
>>
>>> Also, which device are you trying to attach to the guest (pci
>>> bus/device/function)?
>>
>> See attached ath9k.device.
> 
> Hmm, can you also test this again with the v4.3-rc4 please? The device
> you are attaching has its own group and no aliases, so I really can't
> see how the trace could happen, and I can't reproduce it here either.

Unchanged - this time hard locked machine and no trace at all because of
data loss after reboot :-(.

Btw: Linux 4.2 doesn't show this problem.

Nevertheless I'll try to get a trace - maybe I'm lucky and the machine
doesn't lock up completely another time :-).


Regards,
Andreas

Re: [dm-devel] AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Andreas Hartmann]

[-- Attachment #1: Type: text/plain, Size: 1347 bytes --]

On 10/08/2015 at 08:59 PM, Andreas Hartmann wrote:
> On 10/08/2015 at 07:30 PM Joerg Roedel wrote:
>> On Wed, Oct 07, 2015 at 07:02:32PM +0200, Andreas Hartmann wrote:
>>> Binding the device to vfio isn't a problem (it's done before the vm is
>>> started). The problem occurs during start of qemu-system-x86_64 (2.3.0).
>>>
>>> The attached dmesg.out doesn't show the trace, but the desired iommu dump.
>>>
>>>> Also, which device are you trying to attach to the guest (pci
>>>> bus/device/function)?
>>>
>>> See attached ath9k.device.
>>
>> Hmm, can you also test this again with the v4.3-rc4 please? The device
>> you are attaching has its own group and no aliases, so I really can't
>> see how the trace could happen, and I can't reproduce it here either.
> 
> Unchanged - this time hard locked machine and no trace at all because of
> data loss after reboot :-(.
> 
> Btw: Linux 4.2 doesn't show this problem.
> 
> Nevertheless I'll try to get a trace - maybe I'm lucky and the machine
> doesn't lock up completely another time :-).

Got it. I attached the complete oops and the output of objdump.

Kernel was linux 4.3-rc4


This time, the oops was caused by the second PCI card I'm passing
through to another VM (the ath9k card worked fine this time - chance?).
I added the lspci output to the attached file, too.


Thanks,
regards,
Andreas

[-- Attachment #2: oops.gz --]
[-- Type: application/x-gzip, Size: 2833 bytes --]

Re: [dm-devel] AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Joerg Roedel]

On Thu, Oct 08, 2015 at 09:47:28PM +0200, Andreas Hartmann wrote:
> Got it. I attached the complete oops and the output of objdump.
> 
> Kernel was linux 4.3-rc4
> 
> 
> This time, the oops was caused by the second PCI card I'm passing
> through to another VM (the ath9k card worked fine this time - chance?).
> I added the lspci output to the attached file, too.

Okay, thanks, this makes more sense to me. It looks like you are
attaching a 32bit PCI device, which has an alias. This is definitly a
bug in the AMD IOMMU driver and I have an idea how to fix it. I'll look
into this after lunch.



	Joerg

[PATCH] iommu/amd: Fix NULL pointer deref on device detach READ FPDMA QUEUED errors since Linux 4.0

[Joerg Roedel]

Hi Andreas,

On Thu, Oct 08, 2015 at 09:47:28PM +0200, Andreas Hartmann wrote:
> This time, the oops was caused by the second PCI card I'm passing
> through to another VM (the ath9k card worked fine this time - chance?).
> I added the lspci output to the attached file, too.

I digged a little bit around here and found a 32bit PCI card and plugged
it into the AMD IOMMU box. I could reproduce the problem and here is
patch which fixes it for me. Can you test it too please? I'd like to
send a pull-req with this fix included to Linus for rc5.

Thanks,

	Joerg

>From d07307c04edffaaa045fb83713f8808e55ffa895 Mon Sep 17 00:00:00 2001
From: Joerg Roedel <jroedel@suse.de>
Date: Fri, 9 Oct 2015 16:23:33 +0200
Subject: [PATCH] iommu/amd: Fix NULL pointer deref on device detach

When a device group is detached from its domain, the iommu
core code calls into the iommu driver to detach each device
individually.

Before this functionality went into the iommu core code, it
was implemented in the drivers, also in the AMD IOMMU
driver as the device alias handling code.

This code is still present, as there might be aliases that
don't exist as real PCI devices (and are therefore invisible
to the iommu core code).

Unfortunatly it might happen now, that a device is unbound
multiple times from its domain, first by the alias handling
code and then by the iommu core code (or vice verca).

This ends up in the do_detach function which dereferences
the dev_data->domain pointer. When the device is already
detached, this pointer is NULL and we get a kernel oops.

Removing the alias code completly is not an option, as that
would also remove the code which handles invisible aliases.
The code could be simplified, but this is too big of a
change outside the merge window.

For now, just check the dev_data->domain pointer in
do_detach and bail out if it is NULL.

Andreas Hartmann <andihartmann@freenet.de>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
 drivers/iommu/amd_iommu.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/drivers/iommu/amd_iommu.c b/drivers/iommu/amd_iommu.c
index f82060e7..08d2775 100644
--- a/drivers/iommu/amd_iommu.c
+++ b/drivers/iommu/amd_iommu.c
@@ -2006,6 +2006,15 @@ static void do_detach(struct iommu_dev_data *dev_data)
 {
 	struct amd_iommu *iommu;
 
+	/*
+	 * First check if the device is still attached. It might already
+	 * be detached from its domain because the generic
+	 * iommu_detach_group code detached it and we try again here in
+	 * our alias handling.
+	 */
+	if (!dev_data->domain)
+		return;
+
 	iommu = amd_iommu_rlookup_table[dev_data->devid];
 
 	/* decrease reference counters */
-- 
2.5.1

Re: [PATCH] iommu/amd: Fix NULL pointer deref on device detach READ FPDMA QUEUED errors since Linux 4.0

[Andreas Hartmann]

Hello Jörg,

On 10/09/2015 at 04:45 PM, Joerg Roedel wrote:
> Hi Andreas,
> 
> On Thu, Oct 08, 2015 at 09:47:28PM +0200, Andreas Hartmann wrote:
>> This time, the oops was caused by the second PCI card I'm passing
>> through to another VM (the ath9k card worked fine this time - chance?).
>> I added the lspci output to the attached file, too.
> 
> I digged a little bit around here and found a 32bit PCI card and plugged
> it into the AMD IOMMU box. I could reproduce the problem and here is
> patch which fixes it for me. Can you test it too please?

Works fine here. Thanks.


But now, I can see the next big problem of v4.3-rc4: My VMs are
connected on the host via tun/tap devices. They themselves are connected
to a bridge device. If there is data sent between the VMs, the (system)
load is more as double (!) of the load seen with 4.1.x e.g. . Having an
allover throughput of about low 35 MBit/s seen by the host over all
tun/tap devices / bridge creates a load of 3 (!!) with 3 VMs being
involved. That's about half of the load produced by kernel compiling
(make -j8).



> I'd like to
> send a pull-req with this fix included to Linus for rc5.
> 
> Thanks,
> 
> 	Joerg
> 
> From d07307c04edffaaa045fb83713f8808e55ffa895 Mon Sep 17 00:00:00 2001
> From: Joerg Roedel <jroedel@suse.de>
> Date: Fri, 9 Oct 2015 16:23:33 +0200
> Subject: [PATCH] iommu/amd: Fix NULL pointer deref on device detach
> 
> When a device group is detached from its domain, the iommu
> core code calls into the iommu driver to detach each device
> individually.
> 
> Before this functionality went into the iommu core code, it
> was implemented in the drivers, also in the AMD IOMMU
> driver as the device alias handling code.
> 
> This code is still present, as there might be aliases that
> don't exist as real PCI devices (and are therefore invisible
> to the iommu core code).
> 
> Unfortunatly it might happen now, that a device is unbound
> multiple times from its domain, first by the alias handling
> code and then by the iommu core code (or vice verca).
> 
> This ends up in the do_detach function which dereferences
> the dev_data->domain pointer. When the device is already
> detached, this pointer is NULL and we get a kernel oops.
> 
> Removing the alias code completly is not an option, as that
> would also remove the code which handles invisible aliases.
> The code could be simplified, but this is too big of a
> change outside the merge window.
> 
> For now, just check the dev_data->domain pointer in
> do_detach and bail out if it is NULL.
> 
> Andreas Hartmann <andihartmann@freenet.de>
> Signed-off-by: Joerg Roedel <jroedel@suse.de>
> ---
>  drivers/iommu/amd_iommu.c | 9 +++++++++
>  1 file changed, 9 insertions(+)
> 
> diff --git a/drivers/iommu/amd_iommu.c b/drivers/iommu/amd_iommu.c
> index f82060e7..08d2775 100644
> --- a/drivers/iommu/amd_iommu.c
> +++ b/drivers/iommu/amd_iommu.c
> @@ -2006,6 +2006,15 @@ static void do_detach(struct iommu_dev_data *dev_data)
>  {
>  	struct amd_iommu *iommu;
>  
> +	/*
> +	 * First check if the device is still attached. It might already
> +	 * be detached from its domain because the generic
> +	 * iommu_detach_group code detached it and we try again here in
> +	 * our alias handling.
> +	 */
> +	if (!dev_data->domain)
> +		return;
> +
>  	iommu = amd_iommu_rlookup_table[dev_data->devid];
>  
>  	/* decrease reference counters */
> 

Re: [dm-devel] AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Joerg Roedel]

On Wed, Oct 07, 2015 at 04:57:31AM +0200, Andreas Hartmann wrote:
> Got it during a single mount (I booted with massively reduced mounts and
> did the mount afterwards manually. During the second manually mount, the
> problem can be seen).
> 
> I attached the requested files. The mount starts at 80 seconds.
> Hope this helps.

Okay, the lowest dma-addr the AMD IOMMU driver returns is 0x1000 and
the highest is 0x7ff4000. All fault addresses are outside of this range,
so the AMD IOMMU driver never returned these addresses.

This doesn't mean that it is not at fault, but it looks still unlikely.
Maybe I can reproduce the problem here. Can you please tell me some
details about the partitions you mounted to trigger this?

I remember something about a xfs->lvm->dm_crypt->md_raid->sata setup, but
having more details may help me to reproduce.



	Joerg

Re: [dm-devel] AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Andreas Hartmann]

On 10/07/2015 at 06:10 PM, Joerg Roedel wrote:
> On Wed, Oct 07, 2015 at 04:57:31AM +0200, Andreas Hartmann wrote:
>> Got it during a single mount (I booted with massively reduced mounts and
>> did the mount afterwards manually. During the second manually mount, the
>> problem can be seen).
>>
>> I attached the requested files. The mount starts at 80 seconds.
>> Hope this helps.
> 
> Okay, the lowest dma-addr the AMD IOMMU driver returns is 0x1000 and
> the highest is 0x7ff4000. All fault addresses are outside of this range,
> so the AMD IOMMU driver never returned these addresses.
> 
> This doesn't mean that it is not at fault, but it looks still unlikely.
> Maybe I can reproduce the problem here. Can you please tell me some
> details about the partitions you mounted to trigger this?
> 
> I remember something about a xfs->lvm->dm_crypt->md_raid->sata setup, but
> having more details may help me to reproduce.

See attachments in http://article.gmane.org/gmane.linux.kernel.pci/43975


To reproduce the error:
First I mounted /daten2, afterwards /raid/mt, which produces the errors.
The ssd mounts have been already active (during boot by fstab).

If I mount all of them already during boot, the system mostly starts to
emergency mode which unfortunately is broken here.


Regards,
Andreas

Re: [dm-devel] AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Joerg Roedel]

On Wed, Oct 07, 2015 at 06:52:58PM +0200, Andreas Hartmann wrote:
> To reproduce the error:
> First I mounted /daten2, afterwards /raid/mt, which produces the errors.
> The ssd mounts have been already active (during boot by fstab).

Okay, I spent the day on that problem, and managed to reproduce it here
on one of my AMD IOMMU boxes. I wasn't an easy journey, as I can only
reproduce it if I setup the crypto partition and everything above that
(like mounting the lvm volumes) _after_ the system has finished booting.
If everything is setup during system boot it works fine and I don't see
any IO_PAGE_FAULTS.

I also tried kernel v4.3-rc4 first, to have it tested with a
self-compiled kernel. It didn't show up there, so I built a 4.1.0, where
it showed up again. Something seems to have fixed the issue in the
latest kernels.

So I looked a little bit around at the commits that were merged into the
respective parts involved here, and found this one:

	586b286 dm crypt: constrain crypt device's max_segment_size to PAGE_SIZE

The problem fixed with this commit looks quite similar to what you have
seen (execpt that there was no IOMMU involved). So I cherry-picked that
commit on 4.1.0 and tested that. The problem was gone.

So it looks like it was a dm-crypt issue, the patch went into v4.3-rc3,
either this kernel of rc4 should fix the problem for you too. Can you
please verify this is fixed for you too with v4.3-rc4?


Thanks,

	Joerg

Re: [dm-devel] AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Andreas Hartmann]

Am 08.10.2015 um 18:39 schrieb Joerg Roedel:
> On Wed, Oct 07, 2015 at 06:52:58PM +0200, Andreas Hartmann wrote:
>> To reproduce the error:
>> First I mounted /daten2, afterwards /raid/mt, which produces the errors.
>> The ssd mounts have been already active (during boot by fstab).
>
> Okay, I spent the day on that problem, and managed to reproduce it here
> on one of my AMD IOMMU boxes. I wasn't an easy journey, as I can only
> reproduce it if I setup the crypto partition and everything above that
> (like mounting the lvm volumes) _after_ the system has finished booting.
> If everything is setup during system boot it works fine and I don't see
> any IO_PAGE_FAULTS.

Thank you very much for spending so much of your time to reproduce the 
problem!

> I also tried kernel v4.3-rc4 first, to have it tested with a
> self-compiled kernel. It didn't show up there, so I built a 4.1.0, where
> it showed up again. Something seems to have fixed the issue in the
> latest kernels.
>
> So I looked a little bit around at the commits that were merged into the
> respective parts involved here, and found this one:
>
> 	586b286 dm crypt: constrain crypt device's max_segment_size to PAGE_SIZE
 >
> The problem fixed with this commit looks quite similar to what you have
> seen (execpt that there was no IOMMU involved). So I cherry-picked that
> commit on 4.1.0 and tested that. The problem was gone.

That's true - I already knew this patch and tested it some weeks ago - 
unfortunately it doesn't fix the problem here.

To be really sure, I just retested it now again. I couldn't see any 
IO_PAGE_FAULTS errors today (unfortunately I can't remember anymore if I 
didn't see them too a few weeks ago) - but the ata errors remain. 
Therefore, this patch isn't a solution for the problem I encounter here.

> So it looks like it was a dm-crypt issue, the patch went into v4.3-rc3,
> either this kernel of rc4 should fix the problem for you too. Can you
> please verify this is fixed for you too with v4.3-rc4?

As I already wrote, I even couldn't see the problem with v4.3-rc2 any 
more (as far as I was able to test because of the other problem). I have 
to do some more tests now with this kernel to be really sure.


Kind regards,
Andreas

Re: [dm-devel] AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Andreas Hartmann]

On 10/08/2015 at 08:21 PM, Andreas Hartmann wrote:
> Am 08.10.2015 um 18:39 schrieb Joerg Roedel:
>> On Wed, Oct 07, 2015 at 06:52:58PM +0200, Andreas Hartmann wrote:
>>> To reproduce the error:
>>> First I mounted /daten2, afterwards /raid/mt, which produces the errors.
>>> The ssd mounts have been already active (during boot by fstab).
>>
>> Okay, I spent the day on that problem, and managed to reproduce it here
>> on one of my AMD IOMMU boxes. I wasn't an easy journey, as I can only
>> reproduce it if I setup the crypto partition and everything above that
>> (like mounting the lvm volumes) _after_ the system has finished booting.
>> If everything is setup during system boot it works fine and I don't see
>> any IO_PAGE_FAULTS.
> 
> Thank you very much for spending so much of your time to reproduce the
> problem!
> 
>> I also tried kernel v4.3-rc4 first, to have it tested with a
>> self-compiled kernel. It didn't show up there, so I built a 4.1.0, where
>> it showed up again. Something seems to have fixed the issue in the
>> latest kernels.
>>
>> So I looked a little bit around at the commits that were merged into the
>> respective parts involved here, and found this one:
>>
>>     586b286 dm crypt: constrain crypt device's max_segment_size to
>> PAGE_SIZE
>>
>> The problem fixed with this commit looks quite similar to what you have
>> seen (execpt that there was no IOMMU involved). So I cherry-picked that
>> commit on 4.1.0 and tested that. The problem was gone.
> 
> That's true - I already knew this patch and tested it some weeks ago -
> unfortunately it doesn't fix the problem here.
> 
> To be really sure, I just retested it now again. I couldn't see any
> IO_PAGE_FAULTS errors today (unfortunately I can't remember anymore if I
> didn't see them too a few weeks ago) - but the ata errors remain.
> Therefore, this patch isn't a solution for the problem I encounter here.
> 
>> So it looks like it was a dm-crypt issue, the patch went into v4.3-rc3,
>> either this kernel of rc4 should fix the problem for you too. Can you
>> please verify this is fixed for you too with v4.3-rc4?
> 
> As I already wrote, I even couldn't see the problem with v4.3-rc2 any
> more (as far as I was able to test because of the other problem). I have
> to do some more tests now with this kernel to be really sure.

I now tested w/ v4.3-rc4. I couldn't see any IO_PAGE_FAULTS but the ata
errors remain. The ata errors can be easily activated by copying a large
file (> 4 GB) from one partition on the raid to another partition on the
raid.



Thanks,
Andreas

Re: [dm-devel] AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Andreas Hartmann]

On 10/08/2015 at 09:52 PM, Andreas Hartmann wrote:
> On 10/08/2015 at 08:21 PM, Andreas Hartmann wrote:
>> Am 08.10.2015 um 18:39 schrieb Joerg Roedel:
>>> On Wed, Oct 07, 2015 at 06:52:58PM +0200, Andreas Hartmann wrote:
>>>> To reproduce the error:
>>>> First I mounted /daten2, afterwards /raid/mt, which produces the errors.
>>>> The ssd mounts have been already active (during boot by fstab).
>>>
>>> Okay, I spent the day on that problem, and managed to reproduce it here
>>> on one of my AMD IOMMU boxes. I wasn't an easy journey, as I can only
>>> reproduce it if I setup the crypto partition and everything above that
>>> (like mounting the lvm volumes) _after_ the system has finished booting.
>>> If everything is setup during system boot it works fine and I don't see
>>> any IO_PAGE_FAULTS.
>>
>> Thank you very much for spending so much of your time to reproduce the
>> problem!
>>
>>> I also tried kernel v4.3-rc4 first, to have it tested with a
>>> self-compiled kernel. It didn't show up there, so I built a 4.1.0, where
>>> it showed up again. Something seems to have fixed the issue in the
>>> latest kernels.
>>>
>>> So I looked a little bit around at the commits that were merged into the
>>> respective parts involved here, and found this one:
>>>
>>>     586b286 dm crypt: constrain crypt device's max_segment_size to
>>> PAGE_SIZE
>>>
>>> The problem fixed with this commit looks quite similar to what you have
>>> seen (execpt that there was no IOMMU involved). So I cherry-picked that
>>> commit on 4.1.0 and tested that. The problem was gone.
>>
>> That's true - I already knew this patch and tested it some weeks ago -
>> unfortunately it doesn't fix the problem here.
>>
>> To be really sure, I just retested it now again. I couldn't see any
>> IO_PAGE_FAULTS errors today (unfortunately I can't remember anymore if I
>> didn't see them too a few weeks ago) - but the ata errors remain.
>> Therefore, this patch isn't a solution for the problem I encounter here.
>>
>>> So it looks like it was a dm-crypt issue, the patch went into v4.3-rc3,
>>> either this kernel of rc4 should fix the problem for you too. Can you
>>> please verify this is fixed for you too with v4.3-rc4?
>>
>> As I already wrote, I even couldn't see the problem with v4.3-rc2 any
>> more (as far as I was able to test because of the other problem). I have
>> to do some more tests now with this kernel to be really sure.
> 
> I now tested w/ v4.3-rc4. I couldn't see any IO_PAGE_FAULTS but the ata
> errors remain. The ata errors can be easily activated by copying a large
> file (> 4 GB) from one partition on the raid to another partition on the
> raid.

Hmmm, I retested this morning w/ v4.3-rc4 and 4.1.10 (with the above
mentioned patch applied) - and now, I didn't get any more ata errors.

I'm confused now. The only difference between yesterday evening and this
morning was, that the machine was over night completely powerless (via
socket outlet switch). Could this really be the reason? Let's wait and
see if this is a persistent state ... .

But the other new error w/ 4.3-rc-2 or 4 while starting a VM with PCI
passthrough remains even this morning :-(. Would have been nice if it
would have gone over night, too ...


Thanks,
regards,
Andreas

Re: [dm-devel] AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Andreas Hartmann]

On 10/09/2015 at 07:20 AM, Andreas Hartmann wrote:
> On 10/08/2015 at 09:52 PM, Andreas Hartmann wrote:
>> On 10/08/2015 at 08:21 PM, Andreas Hartmann wrote:
>>> Am 08.10.2015 um 18:39 schrieb Joerg Roedel:
>>>> On Wed, Oct 07, 2015 at 06:52:58PM +0200, Andreas Hartmann wrote:
>>>>> To reproduce the error:
>>>>> First I mounted /daten2, afterwards /raid/mt, which produces the errors.
>>>>> The ssd mounts have been already active (during boot by fstab).
>>>>
>>>> Okay, I spent the day on that problem, and managed to reproduce it here
>>>> on one of my AMD IOMMU boxes. I wasn't an easy journey, as I can only
>>>> reproduce it if I setup the crypto partition and everything above that
>>>> (like mounting the lvm volumes) _after_ the system has finished booting.
>>>> If everything is setup during system boot it works fine and I don't see
>>>> any IO_PAGE_FAULTS.
>>>
>>> Thank you very much for spending so much of your time to reproduce the
>>> problem!
>>>
>>>> I also tried kernel v4.3-rc4 first, to have it tested with a
>>>> self-compiled kernel. It didn't show up there, so I built a 4.1.0, where
>>>> it showed up again. Something seems to have fixed the issue in the
>>>> latest kernels.
>>>>
>>>> So I looked a little bit around at the commits that were merged into the
>>>> respective parts involved here, and found this one:
>>>>
>>>>     586b286 dm crypt: constrain crypt device's max_segment_size to
>>>> PAGE_SIZE
>>>>
>>>> The problem fixed with this commit looks quite similar to what you have
>>>> seen (execpt that there was no IOMMU involved). So I cherry-picked that
>>>> commit on 4.1.0 and tested that. The problem was gone.
>>>
>>> That's true - I already knew this patch and tested it some weeks ago -
>>> unfortunately it doesn't fix the problem here.
>>>
>>> To be really sure, I just retested it now again. I couldn't see any
>>> IO_PAGE_FAULTS errors today (unfortunately I can't remember anymore if I
>>> didn't see them too a few weeks ago) - but the ata errors remain.
>>> Therefore, this patch isn't a solution for the problem I encounter here.
>>>
>>>> So it looks like it was a dm-crypt issue, the patch went into v4.3-rc3,
>>>> either this kernel of rc4 should fix the problem for you too. Can you
>>>> please verify this is fixed for you too with v4.3-rc4?
>>>
>>> As I already wrote, I even couldn't see the problem with v4.3-rc2 any
>>> more (as far as I was able to test because of the other problem). I have
>>> to do some more tests now with this kernel to be really sure.
>>
>> I now tested w/ v4.3-rc4. I couldn't see any IO_PAGE_FAULTS but the ata
>> errors remain. The ata errors can be easily activated by copying a large
>> file (> 4 GB) from one partition on the raid to another partition on the
>> raid.
> 
> Hmmm, I retested this morning w/ v4.3-rc4 and 4.1.10 (with the above
> mentioned patch applied) - and now, I didn't get any more ata errors.
> 
> I'm confused now. The only difference between yesterday evening and this
> morning was, that the machine was over night completely powerless (via
> socket outlet switch). Could this really be the reason? Let's wait and
> see if this is a persistent state ... .

No - it is not a persistent state. The ata errors are back again (in
3.1.10 w/ the above mentioned patch applied). It just isn't that easy
any more to trigger them. After a short time of intermission w/ power
off / on cycle, the error came up up again doing the first test copy.
This means: there must be something more broken.

If I revert the original culprit of all of the problems (block: remove
artifical max_hw_sectors cap), it is possible to increase max_sectors_kb
to 1024 - any higher value leads to ata or IO_PAGE_FAULTS sooner or later.

v4.3-rc4 isn't usable at all for me as long as is hangs the machine on
the necessary PCI passthrough for VMs (I need them).


Regards,
Andreas

Re: [dm-devel] AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Joerg Roedel]

On Fri, Oct 09, 2015 at 11:15:05AM +0200, Andreas Hartmann wrote:
> v4.3-rc4 isn't usable at all for me as long as is hangs the machine on
> the necessary PCI passthrough for VMs (I need them).

If the fix I just sent you works, could you please test this again with
a (patched) v4.3-rc4 kernel?


Thanks,

	Joerg

Re: [dm-devel] AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Andreas Hartmann]

Hello Jörg,

On 10/09/2015 at 04:59 PM, Joerg Roedel wrote:
> On Fri, Oct 09, 2015 at 11:15:05AM +0200, Andreas Hartmann wrote:
>> v4.3-rc4 isn't usable at all for me as long as is hangs the machine on
>> the necessary PCI passthrough for VMs (I need them).
> 
> If the fix I just sent you works, could you please test this again with
> a (patched) v4.3-rc4 kernel?

Your IOMMU-patch works fine - but the ata-problem can be seen here, too.
Same behavior as with 4.1.10.


Thanks,
regards,
Andreas

Re: [dm-devel] AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Andreas Hartmann]

On 10/09/2015 at 07:46 PM, Andreas Hartmann wrote:
> Hello Jörg,
> 
> On 10/09/2015 at 04:59 PM, Joerg Roedel wrote:
>> On Fri, Oct 09, 2015 at 11:15:05AM +0200, Andreas Hartmann wrote:
>>> v4.3-rc4 isn't usable at all for me as long as is hangs the machine on
>>> the necessary PCI passthrough for VMs (I need them).
>>
>> If the fix I just sent you works, could you please test this again with
>> a (patched) v4.3-rc4 kernel?
> 
> Your IOMMU-patch works fine - but the ata-problem can be seen here, too.
> Same behavior as with 4.1.10.
> 

Ok, this patch seems to fix the ata errors (I did a lot of tests until
now w/ v4.1.10 - but anyway I'm cautious):

http://thread.gmane.org/gmane.linux.scsi/104141/focus=104267

Would be nice to have it in all kernels (as stable patch too in 4.1.x).


Regards,
Andreas

Re: [dm-devel] AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Andreas Hartmann]

On 10/11/2015 at 02:23 PM, Andreas Hartmann wrote:
> On 10/09/2015 at 07:46 PM, Andreas Hartmann wrote:
>> Hello Jörg,
>>
>> On 10/09/2015 at 04:59 PM, Joerg Roedel wrote:
>>> On Fri, Oct 09, 2015 at 11:15:05AM +0200, Andreas Hartmann wrote:
>>>> v4.3-rc4 isn't usable at all for me as long as is hangs the machine on
>>>> the necessary PCI passthrough for VMs (I need them).
>>>
>>> If the fix I just sent you works, could you please test this again with
>>> a (patched) v4.3-rc4 kernel?
>>
>> Your IOMMU-patch works fine - but the ata-problem can be seen here, too.
>> Same behavior as with 4.1.10.
>>
> 
> Ok, this patch seems to fix the ata errors (I did a lot of tests until
> now w/ v4.1.10 - but anyway I'm cautious):
> 
> http://thread.gmane.org/gmane.linux.scsi/104141/focus=104267

-> Forget it - doesn't fix it.


Regards,
Andreas

Re: [dm-devel] AMD-Vi IO_PAGE_FAULTs and ata3.00: failed command: READ FPDMA QUEUED errors since Linux 4.0

[Mikulas Patocka]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 674 bytes --]



On Fri, 9 Oct 2015, Andreas Hartmann wrote:

> Hello Jörg,
> 
> On 10/09/2015 at 04:59 PM, Joerg Roedel wrote:
> > On Fri, Oct 09, 2015 at 11:15:05AM +0200, Andreas Hartmann wrote:
> >> v4.3-rc4 isn't usable at all for me as long as is hangs the machine on
> >> the necessary PCI passthrough for VMs (I need them).
> > 
> > If the fix I just sent you works, could you please test this again with
> > a (patched) v4.3-rc4 kernel?
> 
> Your IOMMU-patch works fine - but the ata-problem can be seen here, too.
> Same behavior as with 4.1.10.

Could you try another ata disk? (copy the whole filesystem to it and run 
the same test)

It may be bug in disk's firmware.

Mikulas