Automatic SSD trim script

Automatic SSD trim script

[Sten Heinze]

[-- Attachment #1: Type: text/plain, Size: 356 bytes --]

Having experienced the drop in speed when using a SSD with online discard, I wrote a small perl script to run fstrim using batched discard on partitions located on a SSD through cron. Do you think util-linux would be a good place for such a script, given that it is a helper to fstrim? What would be the best way to include one in util-linux?

Thanks,
Sten

[-- Attachment #2: autotrim.pl --]
[-- Type: text/plain, Size: 4151 bytes --]

#!/usr/bin/perl

# This script tries to detect all mounted file systems, which of them are on SSD drives, and run fstrim on them.
# For details see http://wiki.ubuntuusers.de/SSD/TRIM or https://wiki.debian.org/SSDOptimization
# Use cron to run regularly, logger to redirect the output to syslog (essentially the output of fstrim).
# Written by Sten Heinze, 2013.

use strict;
use warnings;
use Cwd qw(realpath);

my $hdparm = '/sbin/hdparm';
my $grep = '/bin/grep';
my $block_dev_path = '/sys/block/';
my $rotational_file = '/queue/rotational';
my $mount = '/bin/mount';
my $discard_option = 'discard';
my $fstrim = '/sbin/fstrim -v'; # verbose needed?

# remove leading and trailing whitespace
sub trim {
  my $str = shift;
  $str =~ s/^\s+|\s+$//g;
  return $str;
}

# read file content
sub read_file {
  my $filename = shift;
  open( my $file_handle, "<", $filename ) || die "$filename: $!";
  my $content = trim( join( '', <$file_handle> ) );
  close( $file_handle );
  return $content;
}

# print and execute the given command
sub e {
  my $cmd = shift;
  print( "$cmd\n" );
  system( "$cmd" );
}

# list all mounted drives; blkid doesn't provide mount points; fstab does ans is another possible source.
# maybe only include fixed/internal drives? /sys/block/sdX/removable doesn't help for deciding if a dev is fixed.
sub get_devs_from_mount {
  my %devs = (); # empty hash

  my $output = `$mount`; 
  my @lines = split( '\n', $output); # split the output into lines
  foreach my $line ( @lines ) {
    if( $line =~ m$(\S+) on (/\S*) type \S+ (.*)$) { # eg. /dev/sda8 on /home type ext4 (rw,relatime,data=ordered)
      my $path = $1;
      my $mount_point = $2;
      my $mount_options = $3;
      next if( index( $mount_options, $discard_option ) != -1 ); # not -1 means found discard, i.e. skip this line
      next if( ! -e $path ); # skip lines that are virtual fs
      $path = realpath( $path ); # get absolute path for those mount points that are uuid symlinks
      $devs{ $path } = $mount_point; # add to hash
    }
  }
  return %devs;
}

# check if device is ssd using hdparm
sub is_ssd_hdparm {
  my $dev = shift;
  $dev = substr( $dev, 5, 3 ); # short to 3 chars: /dev/xxxN to xxx

  return 0 if( ! -X $hdparm || ! -X $grep ); # return no ssd if no hdparm or no grep command available

  `$hdparm -I /dev/$dev 2>&1 | $grep 'TRIM supported' 2>/dev/null`; # perl calls bash, use bash redirect
  if( $? == -1 ) {
    #print "failed to execute: $!\n";
  }
  elsif( $? & 127 ) {
    #printf "child died with signal %d, %s coredump\n", ($? & 127),  ($? & 128) ? 'with' : 'without';
  }
  else {
    my $exit_code = $? >> 8;
    #printf "child exited with value %d\n", $exit_code;
    return 1 if( $exit_code == 0 ); # only if grep found something, TRIM is supported
  }
  return 0;
}

# check if device is ssd using /sys/block/sdX/queue/rotational: 0=SSD, 1=likely HDD, but could be USB memory etc.
# if any error occurs, assume dev is not rotational and return 0.
sub is_ssd_sysfs {
  my $dev = shift;
  $dev = substr( $dev, 5, 3 ); # short to 3 chars: /dev/xxxN to xxx
  my $dev_substr = substr( $dev, 0, 2 );

  return 0 if( $dev_substr ne "sd" ); # if device name is not sdX, assume it is not a ssd
  return 0 if( ! -R $block_dev_path.$dev.$rotational_file ); # return false if file is not readable
  return 0 if( read_file( $block_dev_path.$dev.$rotational_file ) eq "1" ); # if rotational, it's likely not a SSD
  return 1;
}

# check if $block_dev_dir/dev/queue/rotational is 0 -> not reliable, use only as fallback
sub filter_ssd {
  my %devs = @_;
  while( my ( $dev, $mount_point ) = each( %devs ) ) { # loop over hash and remove all non-ssd devs
    delete $devs{ $dev } if( ! is_ssd_hdparm( $dev ) && ! is_ssd_sysfs( $dev ) ); # remove dev if no ssd
  }
  return %devs;
}

# trim the mount points
sub trim_mount_points {
  my %devs = @_;
  while( my ( $dev, $mount_point ) = each( %devs ) ) { # loop over hash and call trim on mount points
    e( "$fstrim $mount_point" );
    #return; # for testing: use only the first
  }
}

# main
my %devs = get_devs_from_mount();
%devs = filter_ssd( %devs );
trim_mount_points( %devs );

Re: Automatic SSD trim script

[Karel Zak]

 Hi Sten,

On Thu, Dec 05, 2013 at 08:20:07PM +0100, Sten Heinze wrote:
> Having experienced the drop in speed when using a SSD with online
> discard, I wrote a small perl script to run fstrim using batched
> discard on partitions located on a SSD through cron. Do you think
> util-linux would be a good place for such a script, given that it is
> a helper to fstrim? What would be the best way to include one in
> util-linux?

I have doubts we want to add a perl script to the package as a regular
util, util-linux is very basic package and dependence on Perl is
unexpected here. Maybe it would be possible to write a simple shell
script for this task and add it to the Documentation/example.files/
directory.


IMHO the best solution would be to improve fstrim to trim all
filesystems where it makes sense.

All we need is to link fstrim with libmount and lib/sysfs.c, add a new
option --all and check non-zero /sys/block/<name>/queue/discard_granularity
(or so).

I guess that implement something like this will be ~30 lines in C :-)  

[CC: to Lukas who is fstrim author]

    Karel


Anyway, a few comments to your script:

> # list all mounted drives; blkid doesn't provide mount points; fstab does ans is another possible source.

 don't use mount(8) to list info about mountpoints, findmnt(8) is better, for example:

    findmnt -clo TARGET -O nodiscard

> # maybe only include fixed/internal drives? /sys/block/sdX/removable doesn't help for deciding if a dev is fixed.
> sub get_devs_from_mount {
>   my %devs = (); # empty hash
> 
>   my $output = `$mount`; 
>   my @lines = split( '\n', $output); # split the output into lines
>   foreach my $line ( @lines ) {
>     if( $line =~ m$(\S+) on (/\S*) type \S+ (.*)$) { # eg. /dev/sda8 on /home type ext4 (rw,relatime,data=ordered)
>       my $path = $1;
>       my $mount_point = $2;
>       my $mount_options = $3;
>       next if( index( $mount_options, $discard_option ) != -1 ); # not -1 means found discard, i.e. skip this line
>       next if( ! -e $path ); # skip lines that are virtual fs
>       $path = realpath( $path ); # get absolute path for those mount points that are uuid symlinks
>       $devs{ $path } = $mount_point; # add to hash
>     }
>   }
>   return %devs;
> }
> 
> # check if device is ssd using hdparm
> sub is_ssd_hdparm {
>   my $dev = shift;
>   $dev = substr( $dev, 5, 3 ); # short to 3 chars: /dev/xxxN to xxx
> 
>   return 0 if( ! -X $hdparm || ! -X $grep ); # return no ssd if no hdparm or no grep command available
> 
>   `$hdparm -I /dev/$dev 2>&1 | $grep 'TRIM supported' 2>/dev/null`; # perl calls bash, use bash redirect


 it would be possible to use lsblk to list devices with non-zero DISC-GRAN column,
 the util also provides mounpoints.

> # check if device is ssd using /sys/block/sdX/queue/rotational: 0=SSD, 1=likely HDD, but could be USB memory etc.

 again, use lsblk

-- 
 Karel Zak  <kzak@redhat.com>
 http://karelzak.blogspot.com

Re: Automatic SSD trim script

[Lukáš Czerner]

On Fri, 6 Dec 2013, Karel Zak wrote:

> Date: Fri, 6 Dec 2013 11:49:44 +0100
> From: Karel Zak <kzak@redhat.com>
> To: Sten Heinze <shze@gmx.de>
> Cc: util-linux@vger.kernel.org, Lukas Czerner <lczerner@redhat.com>
> Subject: Re: Automatic SSD trim script
> 
> 
>  Hi Sten,
> 
> On Thu, Dec 05, 2013 at 08:20:07PM +0100, Sten Heinze wrote:
> > Having experienced the drop in speed when using a SSD with online
> > discard, I wrote a small perl script to run fstrim using batched
> > discard on partitions located on a SSD through cron. Do you think
> > util-linux would be a good place for such a script, given that it is
> > a helper to fstrim? What would be the best way to include one in
> > util-linux?
> 
> I have doubts we want to add a perl script to the package as a regular
> util, util-linux is very basic package and dependence on Perl is
> unexpected here. Maybe it would be possible to write a simple shell
> script for this task and add it to the Documentation/example.files/
> directory.

I very much like the idea of having a simple script to add to cron
to be able to keep SSD's at top of its performance.

> 
> 
> IMHO the best solution would be to improve fstrim to trim all
> filesystems where it makes sense.
> 
> All we need is to link fstrim with libmount and lib/sysfs.c, add a new
> option --all and check non-zero /sys/block/<name>/queue/discard_granularity
> (or so).

Yes and yes. This is the ultimate solution. Simply being able to run
fstrim --all to discard free space on all the mounted supported file
systems is great idea. In that case we do not need any scripts at
all.

Thanks!
-Lukas

> 
> I guess that implement something like this will be ~30 lines in C :-)  
> 
> [CC: to Lukas who is fstrim author]
> 
>     Karel
> 
> 
> Anyway, a few comments to your script:
> 
> > # list all mounted drives; blkid doesn't provide mount points; fstab does ans is another possible source.
> 
>  don't use mount(8) to list info about mountpoints, findmnt(8) is better, for example:
> 
>     findmnt -clo TARGET -O nodiscard
> 
> > # maybe only include fixed/internal drives? /sys/block/sdX/removable doesn't help for deciding if a dev is fixed.
> > sub get_devs_from_mount {
> >   my %devs = (); # empty hash
> > 
> >   my $output = `$mount`; 
> >   my @lines = split( '\n', $output); # split the output into lines
> >   foreach my $line ( @lines ) {
> >     if( $line =~ m$(\S+) on (/\S*) type \S+ (.*)$) { # eg. /dev/sda8 on /home type ext4 (rw,relatime,data=ordered)
> >       my $path = $1;
> >       my $mount_point = $2;
> >       my $mount_options = $3;
> >       next if( index( $mount_options, $discard_option ) != -1 ); # not -1 means found discard, i.e. skip this line
> >       next if( ! -e $path ); # skip lines that are virtual fs
> >       $path = realpath( $path ); # get absolute path for those mount points that are uuid symlinks
> >       $devs{ $path } = $mount_point; # add to hash
> >     }
> >   }
> >   return %devs;
> > }
> > 
> > # check if device is ssd using hdparm
> > sub is_ssd_hdparm {
> >   my $dev = shift;
> >   $dev = substr( $dev, 5, 3 ); # short to 3 chars: /dev/xxxN to xxx
> > 
> >   return 0 if( ! -X $hdparm || ! -X $grep ); # return no ssd if no hdparm or no grep command available
> > 
> >   `$hdparm -I /dev/$dev 2>&1 | $grep 'TRIM supported' 2>/dev/null`; # perl calls bash, use bash redirect
> 
> 
>  it would be possible to use lsblk to list devices with non-zero DISC-GRAN column,
>  the util also provides mounpoints.
> 
> > # check if device is ssd using /sys/block/sdX/queue/rotational: 0=SSD, 1=likely HDD, but could be USB memory etc.
> 
>  again, use lsblk
> 
> 

Re: Automatic SSD trim script

[Karel Zak]

On Mon, Dec 09, 2013 at 02:43:32PM +0100, Lukáš Czerner wrote:
> > All we need is to link fstrim with libmount and lib/sysfs.c, add a new
> > option --all and check non-zero /sys/block/<name>/queue/discard_granularity
> > (or so).
> 
> Yes and yes. This is the ultimate solution. Simply being able to run
> fstrim --all to discard free space on all the mounted supported file
> systems is great idea. In that case we do not need any scripts at
> all.

 Implemented. Sten, maybe you can try fstrim --all from git tree 
 in your crontab.

    Karel

-- 
 Karel Zak  <kzak@redhat.com>
 http://karelzak.blogspot.com

Re: Automatic SSD trim script

[Sten Heinze]

On 12/10/2013 9:44 AM, Karel Zak wrote:
> On Mon, Dec 09, 2013 at 02:43:32PM +0100, Lukáš Czerner wrote:
>>> All we need is to link fstrim with libmount and lib/sysfs.c, add a new
>>> option --all and check non-zero /sys/block/<name>/queue/discard_granularity
>>> (or so).
>>
>> Yes and yes. This is the ultimate solution. Simply being able to run
>> fstrim --all to discard free space on all the mounted supported file
>> systems is great idea. In that case we do not need any scripts at
>> all.
>
>   Implemented. Sten, maybe you can try fstrim --all from git tree
>   in your crontab.
>
>      Karel
>

This was fantastic fast! I will try it out as soon as I have cloned the 
git repo and built it.

Thanks a lot!

Sten

Re: Automatic SSD trim script

[Pádraig Brady]

On 12/10/2013 03:44 PM, Karel Zak wrote:
> On Mon, Dec 09, 2013 at 02:43:32PM +0100, Lukáš Czerner wrote:
>>> All we need is to link fstrim with libmount and lib/sysfs.c, add a new
>>> option --all and check non-zero /sys/block/<name>/queue/discard_granularity
>>> (or so).
>>
>> Yes and yes. This is the ultimate solution. Simply being able to run
>> fstrim --all to discard free space on all the mounted supported file
>> systems is great idea. In that case we do not need any scripts at
>> all.
> 
>  Implemented. Sten, maybe you can try fstrim --all from git tree 
>  in your crontab.

Very nice:
https://git.kernel.org/cgit/utils/util-linux/util-linux.git/commit/?id=36c370c

One foible was it was called multiple times for my /home
Now my /home is mounted a couple of times on my system (why I don't know):

├─/home                         /dev/sdb1           ext4                  rw,relatime,seclabel,user_xattr,barrier=1,data=ordered
│ └─/home                       /dev/sdb1           ext4                  rw,relatime,seclabel,user_xattr,barrier=1,data=ordered

If this was common it might be worth avoiding processing the same entry multiple times.

thanks!
Pádraig

Re: Automatic SSD trim script

[Karel Zak]

On Wed, Dec 11, 2013 at 10:54:07AM +0000, Pádraig Brady wrote:
> Very nice:
> https://git.kernel.org/cgit/utils/util-linux/util-linux.git/commit/?id=36c370c
> 
> One foible was it was called multiple times for my /home
> Now my /home is mounted a couple of times on my system (why I don't know):
> 
> ├─/home                         /dev/sdb1           ext4                  rw,relatime,seclabel,user_xattr,barrier=1,data=ordered
> │ └─/home                       /dev/sdb1           ext4                  rw,relatime,seclabel,user_xattr,barrier=1,data=ordered
> 
> If this was common it might be worth avoiding processing the same entry multiple times.

The common things is something like:

├─/boot                          systemd-1   autofs rw,relatime,fd=41,pgrp=1,timeout=300,minproto=5,maxpr
│ └─/boot                        /dev/sda1   vfat   rw,relatime,fmask=0077,dmask=0077,codepage=437,iochar

but "autofs" is ignored.

My old plan is to add some de-duplication function to libmount... now
I have a real motivation :-)

    Karel

-- 
 Karel Zak  <kzak@redhat.com>
 http://karelzak.blogspot.com

Re: Automatic SSD trim script

[Karel Zak]

On Wed, Dec 11, 2013 at 10:54:07AM +0000, Pádraig Brady wrote:
> If this was common it might be worth avoiding processing the same entry multiple times.

 Implemented. I have also added --uniq to findmnt, so you will see your
 /home only once :-)

    Karel

-- 
 Karel Zak  <kzak@redhat.com>
 http://karelzak.blogspot.com

Re: Automatic SSD trim script

[Marcos Mello]

Karel Zak <kzak@...> writes:

> 
> On Mon, Dec 09, 2013 at 02:43:32PM +0100, Lukáš Czerner wrote:
> > > All we need is to link fstrim with libmount and lib/sysfs.c, add a new
> > > option --all and check non-zero
/sys/block/≤name>/queue/discard_granularity
> > > (or so).
> > 
> > Yes and yes. This is the ultimate solution. Simply being able to run
> > fstrim --all to discard free space on all the mounted supported file
> > systems is great idea. In that case we do not need any scripts at
> > all.
> 
>  Implemented. Sten, maybe you can try fstrim --all from git tree 
>  in your crontab.
> 

Thanks, Karel!

Would it be possible add an option for the "--all" mode to tell fstrim skip
mountpoints witch have "discard" option in fstab?

--
Marcos

Re: Automatic SSD trim script

[Marcos Mello]

Marcos Mello <marcosfrm@...> writes:

> 
[snip]
> 
> Thanks, Karel!
> 
> Would it be possible add an option for the "--all" mode to tell fstrim skip
> mountpoints witch have "discard" option in fstab?
> 

s/witch/which/

Re: Automatic SSD trim script

[Karel Zak]

On Tue, Dec 17, 2013 at 07:46:35PM +0000, Marcos Mello wrote:
> Karel Zak <kzak@...> writes:
> 
> > 
> > On Mon, Dec 09, 2013 at 02:43:32PM +0100, Lukáš Czerner wrote:
> > > > All we need is to link fstrim with libmount and lib/sysfs.c, add a new
> > > > option --all and check non-zero
> /sys/block/≤name>/queue/discard_granularity
> > > > (or so).
> > > 
> > > Yes and yes. This is the ultimate solution. Simply being able to run
> > > fstrim --all to discard free space on all the mounted supported file
> > > systems is great idea. In that case we do not need any scripts at
> > > all.
> > 
> >  Implemented. Sten, maybe you can try fstrim --all from git tree 
> >  in your crontab.
> > 
> 
> Thanks, Karel!
> 
> Would it be possible add an option for the "--all" mode to tell fstrim skip
> mountpoints witch have "discard" option in fstab?

It seems like unnecessary optimization -- the overhead on filesystem with 
active discard is minimal.

(And according to Lukas in some cases FSTRIM ioctl can help you 
 with deleted files smaller than than discard_granularity.)

    Karel

-- 
 Karel Zak  <kzak@redhat.com>
 http://karelzak.blogspot.com