Online TRIM/discard performance impact

Online TRIM/discard performance impact

[Martin Rusko]

Hello,

I wanted to ask if following performance drop with online discard
enabled for XFS is normal. I went through mailing list archives and
read information from Lukáš's page
(http://people.redhat.com/lczerner/discard/) so I was prepared for
some performance penalty. But this big?

--> Filesystem is mounted with noatime and discard options ...

/dev/sdb2 on /mnt/test-xfs type xfs (rw,noatime,discard)

--> Deleting freshly unpacked linux kernel sources ...

# time rm -rf linux-3.0.8

real	4m50.155s
user	0m0.012s
sys	0m0.940s

==> It took almost 5 minutes!

--> Deleting same freshly unpacked linux kernel sources this time with
filesystem mounted without 'discard' option ...

/dev/sdb2 on /mnt/test-xfs type xfs (rw,noatime)

# time rm -rf linux-3.0.8

real	0m1.023s
user	0m0.024s
sys	0m0.896s

==> It took a second and something.

SSD drive in question is one of the latest with SF-2281 chipset. I
expected, that TRIM function will just schedule sectors for garbage
collection, which happens some time later (during which drive can be
potentially slower). Trying the same tests with ext4 filesystem, it
got following numbers.

--> ext4 with 'discard' option ...

/dev/sdb3 on /mnt/test-ext4 type ext4 (rw,noatime,discard)

# time rm -rf linux-3.0.8

real	0m0.486s
user	0m0.012s
sys	0m0.468s

--> ext4 without 'discard' option ...

/dev/sdb3 on /mnt/test-ext4 type ext4 (rw,noatime)

root@layla:/mnt/test-ext4# time rm -rf linux-3.0.8

real	0m0.483s
user	0m0.020s
sys	0m0.460s

Tests were repeated several times and it was consistent. Deleting
files on XFS filesystem mounted with 'discard' option was painfully
slow. Kernel version was ...

   Linux layla 3.0.0-12-generic #20-Ubuntu SMP Fri Oct 7 14:56:25 UTC
2011 x86_64 x86_64 x86_64 GNU/Linux

Is this expected behavior?

Best Regards,
Martin

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Online TRIM/discard performance impact

[Christoph Hellwig]

On Sat, Nov 05, 2011 at 09:07:05PM +0100, Martin Rusko wrote:
> SSD drive in question is one of the latest with SF-2281 chipset. I
> expected, that TRIM function will just schedule sectors for garbage
> collection, which happens some time later (during which drive can be
> potentially slower). Trying the same tests with ext4 filesystem, it
> got following numbers.

That's unfortunately not how it works - the TRIM command always drains
the whole queue of outstanding requests, and thus has a fairly severe
impact on performance.  We could reduce it a bit by using vectored
trims like Windows does, but the Linux blocklayer doesn't have the
infrastructure for that yet.

Your workload will remove a lot of inode clusters on XFS, while ext4
never deletes inodes. Using the ikeep mount option for XFS (can't be
set during remount) should speed the performance with discard for
your workload up a lot.  I would still recommand to do a periodic
fstrim run instead, though.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Online TRIM/discard performance impact

[Martin Rusko]

On Mon, Nov 7, 2011 at 11:21 AM, Christoph Hellwig <hch@infradead.org> wrote:
> On Sat, Nov 05, 2011 at 09:07:05PM +0100, Martin Rusko wrote:
>> SSD drive in question is one of the latest with SF-2281 chipset. I
>> expected, that TRIM function will just schedule sectors for garbage
>> collection, which happens some time later (during which drive can be
>> potentially slower). Trying the same tests with ext4 filesystem, it
>> got following numbers.
>
> That's unfortunately not how it works - the TRIM command always drains
> the whole queue of outstanding requests, and thus has a fairly severe
> impact on performance.  We could reduce it a bit by using vectored
> trims like Windows does, but the Linux blocklayer doesn't have the
> infrastructure for that yet.
>
> Your workload will remove a lot of inode clusters on XFS, while ext4
> never deletes inodes. Using the ikeep mount option for XFS (can't be
> set during remount) should speed the performance with discard for
> your workload up a lot.  I would still recommand to do a periodic
> fstrim run instead, though.
>

As per man page ...

ikeep|noikeep
       When inode clusters are emptied of inodes, keep them around on
the disk (ikeep) - this is the traditional XFS behaviour and is still
the
       default for now.  Using the noikeep option, inode clusters are
returned to the free space pool.

... which would imply, that it should be actually rather fast by
default. Even if I had set this explicitly, it didn't really help ...

# mount -o noatime,discard,ikeep /dev/data/test-xfs /mnt/disk

/dev/mapper/data-test--xfs on /mnt/disk type xfs (rw,noatime,discard,ikeep)

# time rm -rf linux-3.0.8

real	4m47.897s
user	0m0.032s
sys	0m0.992s

... normally it would be deleted in a second or even faster.

I'm fine to use periodic fstrim, let say once a day. But this is such
a big difference, that it makes me curious if it is expected behavior
or if I'm doing something terribly wrong. Any idea? Thanks!

Martin

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs