ext3 directory corruption under Xen

ext3 directory corruption under Xen

[Christopher S. Aker]

We've been seeing a rash of ext3 directory corruption occurring under 
Xen.  All but one of the reports have been with filesystems formatted 
with 1024 blocksize.  We have one report, that's potentialy the same 
bug, occurring on a filesystem with 4096 blocksize (either way, it was 
some type of corruption in that case).  In all cases, the filesystems 
were mounted with ext3's default journaling mode.  No quotas or anything 
else other than the default ext3 mount options.

It's happened on a number of different hosts, all of the same hardware 
and software configuration (Xen 3.2 64bit, 32bit pae dom0, 32bit pae 
domUs.  LVM backend with 3ware hardware RAID-1).  Some of those hosts 
were previously running non-virtaulized Linux and UML, using the 
identical guest images, and under that configuration never experienced 
this problem.

This has occurred under both 2.6.18-xenbits and the more recent pv_ops 
based kernels (2.6.24, 2.6.25), which I presume are all using the same 
blkfront driver code.

The common workloads from the reports seems to be active maildirs and rsync.

The initial errors reported back are all from fs/ext3/dir.c, in 
ext3_check_dir_entry(). Most commonly hit is the "rec_len % 4 != 0" 
check.  We've seen other checks trigger, but my assumption is that those 
happen after more stuff gets whacked out.

Eventually the fs will go read-only.  In extreme cases, the fs is chewed 
through enough that data is lost.

It's tricky to track down the trigger because you can only detect the 
corruption after it's happened.  Our attempts to reproduce this using 
various filesystem thrashing scripts haven't yielded a reliable way to 
trigger it, however we have been successful in triggering it twice -- in 
two weeks :( .

My hope is that this triggers an "a-hah" from someone in LKML or Xen 
land who has experience with this code, or that this is a known issue 
and a fix already lives.

We're scared.  Please help.

Thanks,
-Chris

Re: ext3 directory corruption under Xen

[adam radford]

On Mon, Jun 23, 2008 at 9:15 AM, Christopher S. Aker <caker@theshore.net> wrote:

> It's happened on a number of different hosts, all of the same hardware and
> software configuration (Xen 3.2 64bit, 32bit pae dom0, 32bit pae domUs.  LVM
> backend with 3ware hardware RAID-1).

Do you have Intel EM64T and the older 3w-xxxx driver with 4GB+ RAM?
If so there is a driver patch for older kernels to fix an issue where the driver
was over-riding the default DMA direction with DMA_BIDIRECTIONAL,
and this caused corruption going through the SWIOTLB code (not IOMMU
w/AMD).

A driver patch for older kernels including XenServer-4.1 is available here:

http://www.3ware.com/KB/article.aspx?id=15243&cNode=6I1C6S

This doesn't explain however your failures with 2.6.24 2.6.25, etc, since
this issue is no longer valid after the scsi_dma_map() changes that went
into most scsi drivers since 2.6.23.

-Adam

Re: ext3 directory corruption under Xen

[Christopher S. Aker]

adam radford wrote:
> On Mon, Jun 23, 2008 at 9:15 AM, Christopher S. Aker <caker@theshore.net> wrote:
> 
>> It's happened on a number of different hosts, all of the same hardware and
>> software configuration (Xen 3.2 64bit, 32bit pae dom0, 32bit pae domUs.  LVM
>> backend with 3ware hardware RAID-1).
 >
> A driver patch for older kernels including XenServer-4.1 is available here:
> 
> http://www.3ware.com/KB/article.aspx?id=15243&cNode=6I1C6S

Thanks, but unfortunately, from that link:

"3ware 9000 series controllers are not affected by this issue."

... which is what we're using.  This problem only appeared after 
rebooting these machines into Xen.  Some of affected boxes even ran 
2.6.18 (non Xen) for awhile without any problems.

-Chris

Re: ext3 directory corruption under Xen

[adam radford]

[-- Attachment #1: Type: text/plain, Size: 2880 bytes --]

On Mon, Jun 23, 2008 at 12:53 PM, Christopher S. Aker
<caker@theshore.net> wrote:
> adam radford wrote:
>>
>> On Mon, Jun 23, 2008 at 9:15 AM, Christopher S. Aker <caker@theshore.net>
>> wrote:
>>
>>> It's happened on a number of different hosts, all of the same hardware
>>> and
>>> software configuration (Xen 3.2 64bit, 32bit pae dom0, 32bit pae domUs.
>>>  LVM
>>> backend with 3ware hardware RAID-1).
>
>>
>>
>> A driver patch for older kernels including XenServer-4.1 is available
>> here:
>>
>> http://www.3ware.com/KB/article.aspx?id=15243&cNode=6I1C6S
>
> Thanks, but unfortunately, from that link:
>
> "3ware 9000 series controllers are not affected by this issue."
>
> ... which is what we're using.  This problem only appeared after rebooting
> these machines into Xen.  Some of affected boxes even ran 2.6.18 (non Xen)
> for awhile without any problems.
>
> -Chris
>

I've seen cases where Xen DMA code (pci-dma-xen.c) sends I/O requests
un-necessarily through the SWIOTLB code.

Xen dma: avoid unnecessarily SWIOTLB bounce buffering:

http://lists.xensource.com/archives/html/xen-changelog/2008-04/msg00008.html

so you may need a similar bidirectional fix for the 3w-9xxx driver.

Can you try this patch (in-line below and also attached):

-Adam

diff -Naur linux-2.6.18-xen-3.2.0/drivers/scsi/3w-9xxx.c
linux-2.6.18-xen-3.2.0.new/drivers/scsi/3w-9xxx.c
--- linux-2.6.18-xen-3.2.0/drivers/scsi/3w-9xxx.c	2008-02-12
02:44:33.000000000 -0800
+++ linux-2.6.18-xen-3.2.0.new/drivers/scsi/3w-9xxx.c	2008-06-23
15:42:02.000000000 -0700
@@ -1388,7 +1388,7 @@
 	if (cmd->use_sg == 0)
 		goto out;

-	use_sg = pci_map_sg(pdev, cmd->request_buffer, cmd->use_sg,
DMA_BIDIRECTIONAL);
+	use_sg = pci_map_sg(pdev, cmd->request_buffer, cmd->use_sg,
cmd->sc_data_direction);

 	if (use_sg == 0) {
 		TW_PRINTK(tw_dev->host, TW_DRIVER, 0x1c, "Failed to map scatter
gather list");
@@ -1415,7 +1415,7 @@
 		goto out;
 	}

-	mapping = pci_map_single(pdev, cmd->request_buffer,
cmd->request_bufflen, DMA_BIDIRECTIONAL);
+	mapping = pci_map_single(pdev, cmd->request_buffer,
cmd->request_bufflen, cmd->sc_data_direction);

 	if (mapping == 0) {
 		TW_PRINTK(tw_dev->host, TW_DRIVER, 0x1d, "Failed to map page");
@@ -1423,7 +1423,7 @@
 	}

 	cmd->SCp.phase = TW_PHASE_SINGLE;
-	cmd->SCp.have_data_in = mapping;
+	cmd->SCp.dma_handle = mapping;
 	retval = mapping;
 out:
 	return retval;
@@ -2000,10 +2000,10 @@

 	switch(cmd->SCp.phase) {
 	case TW_PHASE_SINGLE:
-		pci_unmap_single(pdev, cmd->SCp.have_data_in, cmd->request_bufflen,
DMA_BIDIRECTIONAL);
+		pci_unmap_single(pdev, cmd->SCp.dma_handle, cmd->request_bufflen,
cmd->sc_data_direction);
 		break;
 	case TW_PHASE_SGLIST:
-		pci_unmap_sg(pdev, cmd->request_buffer, cmd->use_sg, DMA_BIDIRECTIONAL);
+		pci_unmap_sg(pdev, cmd->request_buffer, cmd->use_sg, cmd->sc_data_direction);
 		break;
 	}
 } /* End twa_unmap_scsi_data() */

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 3w-9xxx_xen3.2.0.patch --]
[-- Type: text/x-patch; name=3w-9xxx_xen3.2.0.patch, Size: 1624 bytes --]

diff -Naur linux-2.6.18-xen-3.2.0/drivers/scsi/3w-9xxx.c linux-2.6.18-xen-3.2.0.new/drivers/scsi/3w-9xxx.c
--- linux-2.6.18-xen-3.2.0/drivers/scsi/3w-9xxx.c	2008-02-12 02:44:33.000000000 -0800
+++ linux-2.6.18-xen-3.2.0.new/drivers/scsi/3w-9xxx.c	2008-06-23 15:42:02.000000000 -0700
@@ -1388,7 +1388,7 @@
 	if (cmd->use_sg == 0)
 		goto out;
 
-	use_sg = pci_map_sg(pdev, cmd->request_buffer, cmd->use_sg, DMA_BIDIRECTIONAL);
+	use_sg = pci_map_sg(pdev, cmd->request_buffer, cmd->use_sg, cmd->sc_data_direction);
 
 	if (use_sg == 0) {
 		TW_PRINTK(tw_dev->host, TW_DRIVER, 0x1c, "Failed to map scatter gather list");
@@ -1415,7 +1415,7 @@
 		goto out;
 	}
 
-	mapping = pci_map_single(pdev, cmd->request_buffer, cmd->request_bufflen, DMA_BIDIRECTIONAL);
+	mapping = pci_map_single(pdev, cmd->request_buffer, cmd->request_bufflen, cmd->sc_data_direction);
 
 	if (mapping == 0) {
 		TW_PRINTK(tw_dev->host, TW_DRIVER, 0x1d, "Failed to map page");
@@ -1423,7 +1423,7 @@
 	}
 
 	cmd->SCp.phase = TW_PHASE_SINGLE;
-	cmd->SCp.have_data_in = mapping;
+	cmd->SCp.dma_handle = mapping;
 	retval = mapping;
 out:
 	return retval;
@@ -2000,10 +2000,10 @@
 
 	switch(cmd->SCp.phase) {
 	case TW_PHASE_SINGLE:
-		pci_unmap_single(pdev, cmd->SCp.have_data_in, cmd->request_bufflen, DMA_BIDIRECTIONAL);
+		pci_unmap_single(pdev, cmd->SCp.dma_handle, cmd->request_bufflen, cmd->sc_data_direction);
 		break;
 	case TW_PHASE_SGLIST:
-		pci_unmap_sg(pdev, cmd->request_buffer, cmd->use_sg, DMA_BIDIRECTIONAL);
+		pci_unmap_sg(pdev, cmd->request_buffer, cmd->use_sg, cmd->sc_data_direction);
 		break;
 	}
 } /* End twa_unmap_scsi_data() */