Ext4 inode & trim errors on Micron eUSB SSD with 2.6.37 kernel

Ext4 inode & trim errors on Micron eUSB SSD with 2.6.37 kernel

[Brian Hutchinson]

Hi,

First post here but have been lurking and searching archives for a while.

I have a custom embedded board that I support which is running Yocto
1.5.  The mkfs.ext4 and all other efs2progs files are fairly
up-to-date ... the kernel is just kind of old (yea, I know, I need to
move up some but that is easier said than done on this thing).

The processor is a TI816x (Cortex A8 armv7 arch).

I've been getting some pretty nasty ext4 errors.  Things like:

EXT4-fs error (device sda1) in ext4_dirty_inode:5716: error -4096

EXT4-fs (sda1): delayed block allocation failed for inode 89 at
logical offset 43008 with max blocks 2016 with error -909004800

EXT4-fs (sda1): This should not happen!! Data will be lost

grow_buffers: requested out-of-range block 14543285590114443400 for device sda1
Aborting journal on device sda1-8.
journal commit I/O error
Apr 17 17:28:38 WCS kernel: jouEXT4-fs error (device sda1):
ext4_journal_start_sb:258: rnal commit I/O Detected aborted
journalerror

EXT4-fs (sda1): Remounting filesystem read-only
Apr 17 17:28:38 WCS kernel: EXT4-fs error (device sda1):
ext4_journal_start_sb:258: Detected aborted journal
Apr 17 17:28:38 WCS kernel: EXT4-fs (sda1): Remounting filesystem read-only


... so I've been educating myself on all things related to Ext4 on SSD
type drives.

The drive I'm using is actually two NAND flash chips on a little
circuit board controller connected to a USB bus that Micron calls an
e230 Real SSD (a 2GB drive).

I discovered that I wasn't mounting the drive with noatime and discard
(for TRIM) so I added those and verified the settings took place:

/dev/sda1 on /media/sda1 type ext4 (rw,noatime,barrier=1,data=ordered,discard)

Doubled checked that the partition was properly aligned by fdisk too:

~# blockdev --getalignoff /dev/sda1
0

I have a script that copies a large file (146M) to four other unique
filenames and once it finishes doing that it does a sync, then erases
them all, does another sync, and the loops doing the same.

This exercises the flash and causes the same errors to appear that the
software team observes while running their application.

After I fixed the mount options (adding noatime and discard) I re-ran
my test and started seeing trim related messages I don't quite know
what to do about:

EXT4-fs error (device sda1) in ext4_dirty_inode:5716: error -4096
EXT4-fs warning (device sda1): ext4_issue_discard:2619: discard not
supported, disabling

In searching this issue, it sounds like trim support isn't in 2.6.37???

Any ideas on what I should do?  I've looked at the commits of the ext4
branch and it isn't obvious to me which of the patches I may need to
fix my issue.

I may just fall back to ext3.

Thanks for any insight.

Regards,

Brian

Re: Ext4 inode & trim errors on Micron eUSB SSD with 2.6.37 kernel

[tytso]

On Thu, Apr 24, 2014 at 04:05:26PM -0400, Brian Hutchinson wrote:
> Hi,
> 
> First post here but have been lurking and searching archives for a while.
> 
> I have a custom embedded board that I support which is running Yocto
> 1.5.  The mkfs.ext4 and all other efs2progs files are fairly
> up-to-date ... the kernel is just kind of old (yea, I know, I need to
> move up some but that is easier said than done on this thing).

Huh?  Yocto 1.5 ships with the 3.10 kernel (and Yocto 1.6 should be
shipping soon with an even newer kernel.)

2.6.37 dates back to 2011, and there have been a *huge* number of bug
fixes since then.

If you can't upgrade to a newer kernel, you might be better off
falling back to ext3, and see if the problem still presents itself
there.  Ext3 will create much more flash wear, but that's the price you
pay for staying on an ancient kernel.  :-)

    		      	      	       - Ted

Re: Ext4 inode & trim errors on Micron eUSB SSD with 2.6.37 kernel

[Brian Hutchinson]

On Thu, Apr 24, 2014 at 7:51 PM,  <tytso@mit.edu> wrote:
> On Thu, Apr 24, 2014 at 04:05:26PM -0400, Brian Hutchinson wrote:
>> Hi,
>>
>> First post here but have been lurking and searching archives for a while.
>>
>> I have a custom embedded board that I support which is running Yocto
>> 1.5.  The mkfs.ext4 and all other efs2progs files are fairly
>> up-to-date ... the kernel is just kind of old (yea, I know, I need to
>> move up some but that is easier said than done on this thing).
>
> Huh?  Yocto 1.5 ships with the 3.10 kernel (and Yocto 1.6 should be
> shipping soon with an even newer kernel.)
>
> 2.6.37 dates back to 2011, and there have been a *huge* number of bug
> fixes since then.
>
> If you can't upgrade to a newer kernel, you might be better off
> falling back to ext3, and see if the problem still presents itself
> there.  Ext3 will create much more flash wear, but that's the price you
> pay for staying on an ancient kernel.  :-)
>
>
                                      - Ted
Hi Ted,

The rootfs only (and the sdk generated toolchain) is from Yocto,
u-boot & kernel are from the TI PSP (which happens to be the latest
release by the way ... newer kernels never were released beyond 2.6.37
... and I don't think they were ever mainlined either).

Yes, 1.6 was announced today but there is no modern oe-core based
support for the evm this board is based on ... only older oe-classic.

I've tried the same tests with ext3 and it did appear to run better
... I didn't get any inode errors and the load on the processor while
running the copy script I mentioned was a bit lower (reported by top)
than what I observed testing with ext4.

I'm attempting to get one of the latest kernels going with this board
but it is a slow process.

Thanks for the response.  I had pretty much concluded going to ext3
was the best option but wanted to bounce it on the list as a sanity
check/confirmation.

In the meantime I'm trying to convince mgmt that effort spent moving
toward a new kernel is better than the effort of trying to back port a
bunch of patches into an old kernel.

Regards,

Brian