btrfs subvolume snapshot performance problem

btrfs subvolume snapshot performance problem

[Sylvain Alain]

Hi everyone, I'm still have the problem with the snapshot command.

Here what I tested today :

read  writ|files  inodes
   0   408k| 2784   7264
   0   460k| 2784   7264
   0   496k| 2784   7264
   0   424k| 2784   7264
   0   440k| 2784   7264
   0  1280k| 2784   7264
   0   500k| 2784   7264
   0   592k| 2784   7264
   0   600k| 2784   7264
   0   568k| 2784   7264
   0   792k| 2784   7264
   0   756k| 2784   7264
   0   480k| 2784   7264
   0   592k| 2784   7264
   0   432k| 2784   7264
   0   544k| 2784   7264
   0   512k| 2784   7264
   0  2912k| 2784   7264
   0  3332k| 2784   7264
   0    40M| 2784   7264
   0    52M| 2784   7264
   0  1280k| 2784   7264
   0    69M| 2784   7264
-dsk/total- --filesystem-
 read  writ|files  inodes
   0  5584k| 2784   7264
   0   784k| 2784   7264
   0   624k| 2784   7264
   0   616k| 2784   7264
   0   744k| 2784   7264
   0   736k| 2784   7264
   0   652k| 2784   7264
   0   540k| 2784   7264
   0   752k| 2784   7264
   0   780k| 2784   7264
   0   888k| 2784   7264
   0   480k| 2784   7264
   0   504k| 2784   7264
   0   548k| 2784   7264
   0   892k| 2784   7264
   0   580k| 2784   7264
   0   576k| 2784   7264
   0   636k| 2784   7264
   0   544k| 2784   7264
   0   760k| 2784   7264
   0   752k| 2784   7264
   0   648k| 2784   7264
   0   744k| 2784   7264
-dsk/total- --filesystem-
 read  writ|files  inodes
   0   516k| 2784   7264
   0   608k| 2784   7264
   0   672k| 2784   7264
   0   524k| 2784   7264
   0   524k| 2784   7264
   0   520k| 2784   7264
   0   476k| 2784   7264
   0   520k| 2784   7264
   0   568k| 2784   7264
   0   520k| 2784   7264
   0   548k| 2784   7264
   0   616k| 2784   7264
   0   832k| 2784   7264
   0   824k| 2784   7264
   0   700k| 2784   7264
   0   864k| 2784   7264
   0  1208k| 2784   7264
   0  1064k| 2784   7264
   0   588k| 2784   7264
   0   688k| 2784   7264
   0    41M| 2784   7264
 308k   17M| 2784   7269
7488k  456k| 2784   7268
-dsk/total- --filesystem-
 read  writ|files  inodes
8192B  496k| 2784   7268 ^C

When the snapshot run, you see that the write change from K to M.

And now I have a good example for my problem :

gentootux ~ # mount /dev/sda4 -o
noatime,ssd,discard,compress=lzo,noacl,space_cache,subvolid=0
/mnt/disklayout/
gentootux ~ # cd /mnt/disklayout/
gentootux disklayout # ls
@backup  @racine
gentootux disklayout # time btrfs subvolume delete @backup
Delete subvolume '/mnt/disklayout/@backup'

real   0m0.005s
user   0m0.001s
sys   0m0.002s
gentootux disklayout # ls
@racine
gentootux disklayout # time btrfs subvolume snapshot @racine @backup
Create a snapshot of '@racine' in './@backup'

real   0m2.850s
user   0m0.000s
sys   0m0.010s
gentootux disklayout # ls
@backup  @racine
gentootux disklayout # time btrfs subvolume delete @backup
Delete subvolume '/mnt/disklayout/@backup'

real   0m0.001s
user   0m0.000s
sys   0m0.000s
gentootux disklayout # time btrfs subvolume snapshot @racine @backup
Create a snapshot of '@racine' in './@backup'

real   3m53.616s
user   0m0.000s
sys   0m0.299s

Instead of 3 secondes to run the snapshot, it took almost 4 minutes.

Is there any btrfs log that I could look at to detect the bottleneck ?

For the record, when this happens inside my Xfce desktop, if I try to
launch a terminal, nothing happen, it's like my PC freeze until the
snapshot is done.

Thanks !

--
Salut
alp
Sylvain

Re: btrfs subvolume snapshot performance problem

[Liu Bo]

On Sun, Dec 16, 2012 at 06:28:01PM -0500, Sylvain Alain wrote:
> Hi everyone, I'm still have the problem with the snapshot command.
> 
> Here what I tested today :
> 
> read  writ|files  inodes
>    0   408k| 2784   7264
>    0   460k| 2784   7264
>    0   496k| 2784   7264
>    0   424k| 2784   7264
>    0   440k| 2784   7264
>    0  1280k| 2784   7264
>    0   500k| 2784   7264
>    0   592k| 2784   7264
>    0   600k| 2784   7264
>    0   568k| 2784   7264
>    0   792k| 2784   7264
>    0   756k| 2784   7264
>    0   480k| 2784   7264
>    0   592k| 2784   7264
>    0   432k| 2784   7264
>    0   544k| 2784   7264
>    0   512k| 2784   7264
>    0  2912k| 2784   7264
>    0  3332k| 2784   7264
>    0    40M| 2784   7264
>    0    52M| 2784   7264
>    0  1280k| 2784   7264
>    0    69M| 2784   7264
> -dsk/total- --filesystem-
>  read  writ|files  inodes
>    0  5584k| 2784   7264
>    0   784k| 2784   7264
>    0   624k| 2784   7264
>    0   616k| 2784   7264
>    0   744k| 2784   7264
>    0   736k| 2784   7264
>    0   652k| 2784   7264
>    0   540k| 2784   7264
>    0   752k| 2784   7264
>    0   780k| 2784   7264
>    0   888k| 2784   7264
>    0   480k| 2784   7264
>    0   504k| 2784   7264
>    0   548k| 2784   7264
>    0   892k| 2784   7264
>    0   580k| 2784   7264
>    0   576k| 2784   7264
>    0   636k| 2784   7264
>    0   544k| 2784   7264
>    0   760k| 2784   7264
>    0   752k| 2784   7264
>    0   648k| 2784   7264
>    0   744k| 2784   7264
> -dsk/total- --filesystem-
>  read  writ|files  inodes
>    0   516k| 2784   7264
>    0   608k| 2784   7264
>    0   672k| 2784   7264
>    0   524k| 2784   7264
>    0   524k| 2784   7264
>    0   520k| 2784   7264
>    0   476k| 2784   7264
>    0   520k| 2784   7264
>    0   568k| 2784   7264
>    0   520k| 2784   7264
>    0   548k| 2784   7264
>    0   616k| 2784   7264
>    0   832k| 2784   7264
>    0   824k| 2784   7264
>    0   700k| 2784   7264
>    0   864k| 2784   7264
>    0  1208k| 2784   7264
>    0  1064k| 2784   7264
>    0   588k| 2784   7264
>    0   688k| 2784   7264
>    0    41M| 2784   7264
>  308k   17M| 2784   7269
> 7488k  456k| 2784   7268
> -dsk/total- --filesystem-
>  read  writ|files  inodes
> 8192B  496k| 2784   7268 ^C
> 
> When the snapshot run, you see that the write change from K to M.
> 
> And now I have a good example for my problem :
> 
> gentootux ~ # mount /dev/sda4 -o
> noatime,ssd,discard,compress=lzo,noacl,space_cache,subvolid=0
> /mnt/disklayout/
> gentootux ~ # cd /mnt/disklayout/
> gentootux disklayout # ls
> @backup  @racine
> gentootux disklayout # time btrfs subvolume delete @backup
> Delete subvolume '/mnt/disklayout/@backup'
> 
> real   0m0.005s
> user   0m0.001s
> sys   0m0.002s
> gentootux disklayout # ls
> @racine
> gentootux disklayout # time btrfs subvolume snapshot @racine @backup
> Create a snapshot of '@racine' in './@backup'
> 
> real   0m2.850s
> user   0m0.000s
> sys   0m0.010s
> gentootux disklayout # ls
> @backup  @racine
> gentootux disklayout # time btrfs subvolume delete @backup
> Delete subvolume '/mnt/disklayout/@backup'
> 
> real   0m0.001s
> user   0m0.000s
> sys   0m0.000s

Could you please add a 'sync' between them and post the output here?

Something like:

# btrfs subvolume delete @backup &&
# sync &&
# time btrfs subvolume snapshot @racine @backup

thanks,
liubo

> gentootux disklayout # time btrfs subvolume snapshot @racine @backup
> Create a snapshot of '@racine' in './@backup'
> 
> real   3m53.616s
> user   0m0.000s
> sys   0m0.299s
> 
> Instead of 3 secondes to run the snapshot, it took almost 4 minutes.
> 
> Is there any btrfs log that I could look at to detect the bottleneck ?
> 
> For the record, when this happens inside my Xfce desktop, if I try to
> launch a terminal, nothing happen, it's like my PC freeze until the
> snapshot is done.
> 
> Thanks !
> 
> --
> Salut
> alp
> Sylvain
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs subvolume snapshot performance problem

[Sander]

Sylvain Alain wrote (ao):
> gentootux ~ # mount /dev/sda4 -o
> noatime,ssd,discard,compress=lzo,noacl,space_cache,subvolid=0
              ^^^^^^^

> Instead of 3 secondes to run the snapshot, it took almost 4 minutes.

Let me repeat the answer cwillu gave to Russell on this, and Russell's
response:

Russell Coker wrote (ao):
> On Sun, 16 Dec 2012, cwillu <cwillu@cwillu.com> wrote:
> > Don't use discard; it's a non-queuing command, which means your
> > performance will suck unless your device is really terrible at
> > garbage collection (in which case, it's just the lesser of two evils).
> 
> Thanks for the advice. On one of my systems a reinstall of the linux-
> image-3.6-trunk-amd64 package went from almost 4 minutes to only 29
> seconds when I removed the discard option.

Re: btrfs subvolume snapshot performance problem

[Russell Coker]

On Mon, 17 Dec 2012, Sander <sander@humilis.net> wrote:
> Let me repeat the answer cwillu gave to Russell on this, and Russell's
> response:

Also tests did indicate a performance benefit on snapshots when I stopped using 
discard.  It was as much as 15 seconds to create a snapshot and now it's 
consistently less than 1 second.

-- 
My Main Blog         http://etbe.coker.com.au/
My Documents Blog    http://doc.coker.com.au/

Re: btrfs subvolume snapshot performance problem

[Sylvain Alain]

So, if I don't use the discard command, how often do I need to run the
fstrim command ?

I found this thread : https://patrick-nagel.net/blog/archives/337


Sylvain

2012/12/17 Sylvain Alain <d2racing911@gmail.com>:
> So, if I don't use the discard command, how often do I need to run the
> fstrim command ?
>
> I found this thread : https://patrick-nagel.net/blog/archives/337
>
>
> Sylvain
>
> 2012/12/17 Russell Coker <russell@coker.com.au>:
>> On Mon, 17 Dec 2012, Sander <sander@humilis.net> wrote:
>>> Let me repeat the answer cwillu gave to Russell on this, and Russell's
>>> response:
>>
>> Also tests did indicate a performance benefit on snapshots when I stopped using
>> discard.  It was as much as 15 seconds to create a snapshot and now it's
>> consistently less than 1 second.
>>
>> --
>> My Main Blog         http://etbe.coker.com.au/
>> My Documents Blog    http://doc.coker.com.au/
>
>
>
> --
> Salut
> alp
> Sylvain



-- 
Salut
alp
Sylvain

Re: btrfs subvolume snapshot performance problem

[cwillu]

On Tue, Dec 18, 2012 at 7:06 AM, Sylvain Alain <d2racing911@gmail.com> wrote:
> So, if I don't use the discard command, how often do I need to run the
> fstrim command ?

If your ssd isn't a pile of crap, never.  SSD's are always
over-provisioned, and so every time an erase block fills up, the drive
knows that there must be one erase-block worth of garbage which could
be compacted, erased, and added to the pool of empty blocks.  The
crappiest ones only do this as needed (which is why their write speed
plummets with use), and really benefit from people forcing the issue
with -o discard or occasional fstrim.  Everything else should get
along fine without it, although an occasional fstrim certainly won't
hurt: it just shouldn't help much.

> I found this thread : https://patrick-nagel.net/blog/archives/337

It's worth noting that there's a large number of very effective tricks
that an ssd can perform to almost completely negate the caveat
mentioned there.  It really is a solved problem in a modern ssd.

Re: btrfs subvolume snapshot performance problem

[Sylvain Alain]

Hi, I own this SSD :
http://ark.intel.com/products/66250/Intel-SSD-520-Series-240GB-2_5in-SATA-6Gbs-25nm-MLC

I don't think it's a pile of crap :P

So basically, I could remove the discard option inside my /etc/fstab
and I should run fstrim when I have the time.

Maybe someone should update the btrfs wiki and write something about
the discard situation. I tought that I had to use this option to save
some lifespan for my SSD.

2012/12/18 cwillu <cwillu@cwillu.com>:
> On Tue, Dec 18, 2012 at 7:06 AM, Sylvain Alain <d2racing911@gmail.com> wrote:
>> So, if I don't use the discard command, how often do I need to run the
>> fstrim command ?
>
> If your ssd isn't a pile of crap, never.  SSD's are always
> over-provisioned, and so every time an erase block fills up, the drive
> knows that there must be one erase-block worth of garbage which could
> be compacted, erased, and added to the pool of empty blocks.  The
> crappiest ones only do this as needed (which is why their write speed
> plummets with use), and really benefit from people forcing the issue
> with -o discard or occasional fstrim.  Everything else should get
> along fine without it, although an occasional fstrim certainly won't
> hurt: it just shouldn't help much.
>
>> I found this thread : https://patrick-nagel.net/blog/archives/337
>
> It's worth noting that there's a large number of very effective tricks
> that an ssd can perform to almost completely negate the caveat
> mentioned there.  It really is a solved problem in a modern ssd.



-- 
Salut
alp
Sylvain