Re: Software RAID and TRIM

[David Brown <david@westcontrol.com>]

On 17/07/2011 23:52, Lutz Vieweg wrote:
> David Brown wrote:
>> However, AFAIUI, you are wrong about TRIM being essential for the
>> continued high performance of SSDs. As long as your SSDs have some
>> over-provisioning (or you only partition something like 90% of the
>> drive), and it's got good garbage collection, then TRIM will have
>> minimal effect.
>
> I beg to differ.
>

Well, I don't have your experience here (I have a couple of 60G SSD's in 
RAID0, without TRIM, but that's hardly in the same class).  So I don't 
expect you to put much weight on my opinions.  But maybe it will give 
you reason for more testing.

> We are using SSDs in very much the way that Tom de Mulder intends,
> and from our extensive performance measurements over many months
> now I can say that (at least if you do have significant amounts
> of write operations) it _does_ make a lot of difference whether you
> periodically discard the unused sectors or not.
> (For us, the write performance measured to be about half as good
> when there are no free erase blocks available anymore.)
>

If there are no free erase blocks, then your SSD's don't have enough 
over-provisioning.  This is, after all, the whole point of having more 
physical flash than the logical disk size would suggest.  Depending on 
the quality of the SSD (more expensive ones have more 
over-provisioning), and the usage patterns (if you have lots of small 
random writes, you'll need more extra space), then you might have to 
"manually" over-provision the disk by only partitioning about 90% of the 
disk.  Of course, you must make sure that the remaining 10% is 
"discarded", or left untouched from new, and that you use the partition 
for your RAID and not the whole disk.

So now you have plenty of erase blocks at any time, and your write 
performance will be good.


TRIM, on the other hand, does not give you any extra free erase blocks. 
  If you think it does, you've misunderstood it.

TRIM exists to make garbage collection a little more efficient - when 
garbage collecting an erase block that contains TRIM'ed blocks, the 
TRIM'ed blocks don't need to be copied.  This saves a small amount of 
time in the copying, and allows slightly denser packing.  It may 
sometimes lead to saving whole erase blocks, but that's seldom the case 
in practice except when erasing large files.

If your disks are reasonably full, then TRIM will not help much because 
the garbage collection will be desperately trying to piece together 
small bits into complete erase blocks, and your performance will drop 
through the floor.  If you have plenty of overprovisioning, then the SSD 
still has lots of completely free erase blocks whenever it needs them.

If your filesystem re-uses (logical) blocks, then TRIM will not help. 
It is /always/ more efficient for the FS to simply write new data to the 
same block, rather than TRIM'ing it first.

TRIM is a very expensive command - it acts a bit like a write, but it is 
not a queued command.  Thus the block layer must wait for /all/ IO 
commands to have completed, then issue the TRIM, then wait for it to 
complete, and then carry on with new commands.  On some SSD's, it will 
(according to something I read) trigger garbage collection, which may 
slow down the SSD.  Even without that, the performance of most meta-data 
operations (such as delete) will drop considerably when they also need 
to do TRIM.

<http://people.redhat.com/jmoyer/discard/ext4_batched_discard/ext4_discard.html>

<http://lwn.net/Articles/347511/>

<http://www.realworldtech.com/beta/forums/index.cfm?action=detail&id=116034&threadid=115697&roomid=2>


On the other hand, your off-line batch TRIM during low use periods could 
well be a win.  The cost of these discards is not going to be an issue, 
and large batched discards are going to be far more useful to the SSD 
than small scattered ones.  I believe that there has been work on a 
similar system in XFS - I don't know what happened to that, or if there 
is any way to make it work in concert with md raid.


What will make a big difference to using SSD's in md raid is the 
sync/no-sync tracking.  This will avoid a lot of unnecessary writes, 
especially with a new array, and leave the SSD with more free blocks (at 
least until the disk is getting full of data).  It is also much higher 
up the things-to-do list, because it will be useful for all uses of md 
raid, and is a perquisite to general discard support.  (Strictly 
speaking it is not needed for SSD's that guarantee a zero return on 
TRIM'ed blocks - but only some SSD's give that guarantee.)


> Of course, you can only benefit from discards if your filesystem
> is not full (because then there is nothing to discard). But any
> kind of "garbage collection" by the SSD itself will not have the
> same effect, since it cannot know which blocks are in use by the
> filesystem.
>

Garbage collection will recycle blocks that have been overwritten.  The 
filesystem knows which logical blocks are in use, and which are free. 
Filesystems already heavily re-use blocks, in the aim of preferring 
faster outer tracks on HD's, and minimizing head movement.  So when a 
file is erased, there's a good chance that those same logical blocks 
will be re-used soon - TRIM is of no benefit in that case.

>> I think other SSD-optimisations, such as those in BTRFS, are much more
>> important.
>
> Actually, (apart from btrfs still being in development, not really
> ready for production use, yet), XFS (-o delaylog,barrier) performs
> better on our SSDs than btrfs - without any SSD-specific options.
>

btrfs is ready for some uses, but is not mature and real-world tested 
enough for serious systems (and its tools are still lacking somewhat). 
But more generally, different filesystems are faster and slower for 
different usage patterns.

One SSD optimisation that many filesystems could implement is to be less 
concerned about fragmentation.  Most modern filesystems go out of their 
way to try to reduce fragmentation, which is great for HD use.  But on 
SSD's, you should be happy to fragment files if it promotes re-use of 
erased blocks, as long as fragments aim to fill complete erase blocks 
(in size and alignment).


> What is really an important factor for SSD performance: The controller.
> The same SSDs perform with significantly lower latency for us when
> connected to SATA controller channels than when connected to SAS
> controllers (and they perform abysmal when used as hardware-RAID
> constituents, in comparison).

That is /very/ interesting to know, and is a data point I haven't read 
elsewhere (though I knew about poor performance of hardware RAID with 
SSD).  Thanks for sharing that.


>
> Regards,
>
> Lutz Vieweg
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>