SWIOTLB on 32-bit PAE

SWIOTLB on 32-bit PAE

[Christian Melki]

[-- Attachment #1: Type: text/plain, Size: 1232 bytes --]

Hi,

I discovered that my 32-bit PAE 4.2.0 kernel (no IOMMU code) would hang 
when writing to my USB disk. The kernel spews million(-ish messages per 
sec) to syslog, effectively "hanging" userspace with my kernel.

Oct  2 14:33:06 voodoochild kernel: [  223.287447] nommu_map_sg: 
overflow 25dcac000+1024 of device mask ffffffff
Oct  2 14:33:06 voodoochild kernel: [  223.287448] nommu_map_sg: 
overflow 25dcac000+1024 of device mask ffffffff
Oct  2 14:33:06 voodoochild kernel: [  223.287449] nommu_map_sg: 
overflow 25dcac000+1024 of device mask ffffffff
... etc ...

In my kernel config I noticed that SWIOTLB was not on. It seems SWIOTLB 
is provided for 64-bit and 32-bit with IOMMU/AGPGART code. But if I 
compiled the kernel with PAE and no IOMMU and no other GART code, I 
would not get SWIOTLB. I'd like to think that SWIOTLB should be selected 
for 32-bit PAE as default.

I have attached a oneliner patch which does that.
The patch works for me. The issue where the kernel more or less runs 
endless bashing of (nommu_?)map_sg when failing is another problem I 
guess. I expected the kernel/drivers to have some more graceful 
handling, but I know to little about this area to have a proper opinion.

Regards,
Christian

[-- Attachment #2: swiotlb.patch --]
[-- Type: text/x-patch, Size: 481 bytes --]

diff -urN linux-4.2.orig/arch/x86/Kconfig linux-4.2/arch/x86/Kconfig
--- linux-4.2.orig/arch/x86/Kconfig	2015-10-05 08:56:58.933313678 +0200
+++ linux-4.2/arch/x86/Kconfig	2015-10-05 09:00:00.916306025 +0200
@@ -1282,6 +1282,7 @@
 config X86_PAE
 	bool "PAE (Physical Address Extension) Support"
 	depends on X86_32 && !HIGHMEM4G
+	select SWIOTLB
 	---help---
 	  PAE is required for NX support, and furthermore enables
 	  larger swapspace support for non-overcommit purposes. It

Re: Fwd: SWIOTLB on 32-bit PAE

[Konrad Rzeszutek Wilk]

On Mon, Oct 05, 2015 at 12:44:55PM +0200, Christian Melki wrote:
> Joerg,
> 
> I already sent the patch for Kconfig change to the list.
> But I discovered something even more disturbing which I mailed Konrad about.
> With the change my machine does not hang any more, but instead corrupts data
> which is written. The data however is readable on the machine which
> corrupted it, unreadable on the other machine without the patch. I don't
> understand what is going on. I have never encountered "selective" file
> corruption of fs on different machines. The fs itself checks out cleanly
> with badblock control (ext4). I fail to understand what is happening.

And I believe this ended up being an operator error?

With the patch you sent, you didn't include your SoB 
(Signed-off-By). I would like to include your patch but I cannot
with your SoB. It is matter of reading to Documentation/SubmittingPatches
and just consenting to the Certificate in there.

If you are uncomfortable with that - that is OK, I can make this
patch myself (a one liner :-)) but I figured it owuld be nice
for you to have all the credit.
> 
> Regards,
> Christian
> 
> -------- Forwarded Message --------
> Subject: SWIOTLB on 32-bit PAE
> Date: Mon, 5 Oct 2015 10:31:39 +0200
> From: Christian Melki <christian.melki@t2data.com>
> To: linux-kernel@vger.kernel.org
> CC: konrad.wilk@oracle.com
> 
> Hi,
> 
> I discovered that my 32-bit PAE 4.2.0 kernel (no IOMMU code) would hang
> when writing to my USB disk. The kernel spews million(-ish messages per
> sec) to syslog, effectively "hanging" userspace with my kernel.
> 
> Oct  2 14:33:06 voodoochild kernel: [  223.287447] nommu_map_sg:
> overflow 25dcac000+1024 of device mask ffffffff
> Oct  2 14:33:06 voodoochild kernel: [  223.287448] nommu_map_sg:
> overflow 25dcac000+1024 of device mask ffffffff
> Oct  2 14:33:06 voodoochild kernel: [  223.287449] nommu_map_sg:
> overflow 25dcac000+1024 of device mask ffffffff
> ... etc ...
> 
> In my kernel config I noticed that SWIOTLB was not on. It seems SWIOTLB
> is provided for 64-bit and 32-bit with IOMMU/AGPGART code. But if I
> compiled the kernel with PAE and no IOMMU and no other GART code, I
> would not get SWIOTLB. I'd like to think that SWIOTLB should be selected
> for 32-bit PAE as default.
> 
> I have attached a oneliner patch which does that.
> The patch works for me. The issue where the kernel more or less runs
> endless bashing of (nommu_?)map_sg when failing is another problem I
> guess. I expected the kernel/drivers to have some more graceful
> handling, but I know to little about this area to have a proper opinion.
> 
> Regards,
> Christian
> 
> 
> 

> diff -urN linux-4.2.orig/arch/x86/Kconfig linux-4.2/arch/x86/Kconfig
> --- linux-4.2.orig/arch/x86/Kconfig	2015-10-05 08:56:58.933313678 +0200
> +++ linux-4.2/arch/x86/Kconfig	2015-10-05 09:00:00.916306025 +0200
> @@ -1282,6 +1282,7 @@
>  config X86_PAE
>  	bool "PAE (Physical Address Extension) Support"
>  	depends on X86_32 && !HIGHMEM4G
> +	select SWIOTLB
>  	---help---
>  	  PAE is required for NX support, and furthermore enables
>  	  larger swapspace support for non-overcommit purposes. It
> 

Re: Fwd: SWIOTLB on 32-bit PAE

[Christian Melki]

[-- Attachment #1: Type: text/plain, Size: 3391 bytes --]

Yes. Operator error indeed.

Thanks for asking. Included is the same patch with the SoB.

On 10/05/2015 04:21 PM, Konrad Rzeszutek Wilk wrote:
> On Mon, Oct 05, 2015 at 12:44:55PM +0200, Christian Melki wrote:
>> Joerg,
>>
>> I already sent the patch for Kconfig change to the list.
>> But I discovered something even more disturbing which I mailed Konrad about.
>> With the change my machine does not hang any more, but instead corrupts data
>> which is written. The data however is readable on the machine which
>> corrupted it, unreadable on the other machine without the patch. I don't
>> understand what is going on. I have never encountered "selective" file
>> corruption of fs on different machines. The fs itself checks out cleanly
>> with badblock control (ext4). I fail to understand what is happening.
>
> And I believe this ended up being an operator error?
>
> With the patch you sent, you didn't include your SoB
> (Signed-off-By). I would like to include your patch but I cannot
> with your SoB. It is matter of reading to Documentation/SubmittingPatches
> and just consenting to the Certificate in there.
>
> If you are uncomfortable with that - that is OK, I can make this
> patch myself (a one liner :-)) but I figured it owuld be nice
> for you to have all the credit.
>>
>> Regards,
>> Christian
>>
>> -------- Forwarded Message --------
>> Subject: SWIOTLB on 32-bit PAE
>> Date: Mon, 5 Oct 2015 10:31:39 +0200
>> From: Christian Melki <christian.melki@t2data.com>
>> To: linux-kernel@vger.kernel.org
>> CC: konrad.wilk@oracle.com
>>
>> Hi,
>>
>> I discovered that my 32-bit PAE 4.2.0 kernel (no IOMMU code) would hang
>> when writing to my USB disk. The kernel spews million(-ish messages per
>> sec) to syslog, effectively "hanging" userspace with my kernel.
>>
>> Oct  2 14:33:06 voodoochild kernel: [  223.287447] nommu_map_sg:
>> overflow 25dcac000+1024 of device mask ffffffff
>> Oct  2 14:33:06 voodoochild kernel: [  223.287448] nommu_map_sg:
>> overflow 25dcac000+1024 of device mask ffffffff
>> Oct  2 14:33:06 voodoochild kernel: [  223.287449] nommu_map_sg:
>> overflow 25dcac000+1024 of device mask ffffffff
>> ... etc ...
>>
>> In my kernel config I noticed that SWIOTLB was not on. It seems SWIOTLB
>> is provided for 64-bit and 32-bit with IOMMU/AGPGART code. But if I
>> compiled the kernel with PAE and no IOMMU and no other GART code, I
>> would not get SWIOTLB. I'd like to think that SWIOTLB should be selected
>> for 32-bit PAE as default.
>>
>> I have attached a oneliner patch which does that.
>> The patch works for me. The issue where the kernel more or less runs
>> endless bashing of (nommu_?)map_sg when failing is another problem I
>> guess. I expected the kernel/drivers to have some more graceful
>> handling, but I know to little about this area to have a proper opinion.
>>
>> Regards,
>> Christian
>>
>>
>>
>
>> diff -urN linux-4.2.orig/arch/x86/Kconfig linux-4.2/arch/x86/Kconfig
>> --- linux-4.2.orig/arch/x86/Kconfig	2015-10-05 08:56:58.933313678 +0200
>> +++ linux-4.2/arch/x86/Kconfig	2015-10-05 09:00:00.916306025 +0200
>> @@ -1282,6 +1282,7 @@
>>   config X86_PAE
>>   	bool "PAE (Physical Address Extension) Support"
>>   	depends on X86_32 && !HIGHMEM4G
>> +	select SWIOTLB
>>   	---help---
>>   	  PAE is required for NX support, and furthermore enables
>>   	  larger swapspace support for non-overcommit purposes. It
>>
>


[-- Attachment #2: swiotlb.patch --]
[-- Type: text/plain, Size: 542 bytes --]

diff -urN linux-4.2.orig/arch/x86/Kconfig linux-4.2/arch/x86/Kconfig
--- linux-4.2.orig/arch/x86/Kconfig	2015-10-05 08:56:58.933313678 +0200
+++ linux-4.2/arch/x86/Kconfig	2015-10-05 09:00:00.916306025 +0200
@@ -1282,6 +1282,7 @@
 config X86_PAE
 	bool "PAE (Physical Address Extension) Support"
 	depends on X86_32 && !HIGHMEM4G
+	select SWIOTLB
 	---help---
 	  PAE is required for NX support, and furthermore enables
 	  larger swapspace support for non-overcommit purposes. It

Signed-off-by: Christian Melki <christian.melki@t2data.com>

Re: Fwd: SWIOTLB on 32-bit PAE

[Konrad Rzeszutek Wilk]

On Mon, Oct 05, 2015 at 05:31:33PM +0200, Christian Melki wrote:
> Yes. Operator error indeed.
> 
> Thanks for asking. Included is the same patch with the SoB.

I've stuck it at 
https://git.kernel.org/cgit/linux/kernel/git/konrad/swiotlb.git/ #stable/for-linus-4.3

Will send it out tomorrow if there are no #linux-next issues.
> 
> On 10/05/2015 04:21 PM, Konrad Rzeszutek Wilk wrote:
> >On Mon, Oct 05, 2015 at 12:44:55PM +0200, Christian Melki wrote:
> >>Joerg,
> >>
> >>I already sent the patch for Kconfig change to the list.
> >>But I discovered something even more disturbing which I mailed Konrad about.
> >>With the change my machine does not hang any more, but instead corrupts data
> >>which is written. The data however is readable on the machine which
> >>corrupted it, unreadable on the other machine without the patch. I don't
> >>understand what is going on. I have never encountered "selective" file
> >>corruption of fs on different machines. The fs itself checks out cleanly
> >>with badblock control (ext4). I fail to understand what is happening.
> >
> >And I believe this ended up being an operator error?
> >
> >With the patch you sent, you didn't include your SoB
> >(Signed-off-By). I would like to include your patch but I cannot
> >with your SoB. It is matter of reading to Documentation/SubmittingPatches
> >and just consenting to the Certificate in there.
> >
> >If you are uncomfortable with that - that is OK, I can make this
> >patch myself (a one liner :-)) but I figured it owuld be nice
> >for you to have all the credit.
> >>
> >>Regards,
> >>Christian
> >>
> >>-------- Forwarded Message --------
> >>Subject: SWIOTLB on 32-bit PAE
> >>Date: Mon, 5 Oct 2015 10:31:39 +0200
> >>From: Christian Melki <christian.melki@t2data.com>
> >>To: linux-kernel@vger.kernel.org
> >>CC: konrad.wilk@oracle.com
> >>
> >>Hi,
> >>
> >>I discovered that my 32-bit PAE 4.2.0 kernel (no IOMMU code) would hang
> >>when writing to my USB disk. The kernel spews million(-ish messages per
> >>sec) to syslog, effectively "hanging" userspace with my kernel.
> >>
> >>Oct  2 14:33:06 voodoochild kernel: [  223.287447] nommu_map_sg:
> >>overflow 25dcac000+1024 of device mask ffffffff
> >>Oct  2 14:33:06 voodoochild kernel: [  223.287448] nommu_map_sg:
> >>overflow 25dcac000+1024 of device mask ffffffff
> >>Oct  2 14:33:06 voodoochild kernel: [  223.287449] nommu_map_sg:
> >>overflow 25dcac000+1024 of device mask ffffffff
> >>... etc ...
> >>
> >>In my kernel config I noticed that SWIOTLB was not on. It seems SWIOTLB
> >>is provided for 64-bit and 32-bit with IOMMU/AGPGART code. But if I
> >>compiled the kernel with PAE and no IOMMU and no other GART code, I
> >>would not get SWIOTLB. I'd like to think that SWIOTLB should be selected
> >>for 32-bit PAE as default.
> >>
> >>I have attached a oneliner patch which does that.
> >>The patch works for me. The issue where the kernel more or less runs
> >>endless bashing of (nommu_?)map_sg when failing is another problem I
> >>guess. I expected the kernel/drivers to have some more graceful
> >>handling, but I know to little about this area to have a proper opinion.
> >>
> >>Regards,
> >>Christian
> >>
> >>
> >>
> >
> >>diff -urN linux-4.2.orig/arch/x86/Kconfig linux-4.2/arch/x86/Kconfig
> >>--- linux-4.2.orig/arch/x86/Kconfig	2015-10-05 08:56:58.933313678 +0200
> >>+++ linux-4.2/arch/x86/Kconfig	2015-10-05 09:00:00.916306025 +0200
> >>@@ -1282,6 +1282,7 @@
> >>  config X86_PAE
> >>  	bool "PAE (Physical Address Extension) Support"
> >>  	depends on X86_32 && !HIGHMEM4G
> >>+	select SWIOTLB
> >>  	---help---
> >>  	  PAE is required for NX support, and furthermore enables
> >>  	  larger swapspace support for non-overcommit purposes. It
> >>
> >
> 

> diff -urN linux-4.2.orig/arch/x86/Kconfig linux-4.2/arch/x86/Kconfig
> --- linux-4.2.orig/arch/x86/Kconfig	2015-10-05 08:56:58.933313678 +0200
> +++ linux-4.2/arch/x86/Kconfig	2015-10-05 09:00:00.916306025 +0200
> @@ -1282,6 +1282,7 @@
>  config X86_PAE
>  	bool "PAE (Physical Address Extension) Support"
>  	depends on X86_32 && !HIGHMEM4G
> +	select SWIOTLB
>  	---help---
>  	  PAE is required for NX support, and furthermore enables
>  	  larger swapspace support for non-overcommit purposes. It
> 
> Signed-off-by: Christian Melki <christian.melki@t2data.com>