ssd option for USB flash drive?

ssd option for USB flash drive?

[Stephane Chazelas]

Hiya,

I've not found much detail on what the "ssd" btrfs mount option
did. Would it make sense to enable it to a fs on a USB flash
drive?

I'm using btrfs (over LVM) on a Live Linux USB stick to benefit
from btrfs's compression and am trying to improve the
performance.

Would anybody have any recommendation on how to improve
performance there? Like what would be the best way to
enable/increase writeback buffer or any way to make sure writes
are delayed and asynchronous? Would disabling read-ahead help?
(at which level would it be done?). Any other tip (like
disabling atime, aligning blocks/extents, figure out erase block
sizes if relevant...)?

Many thanks in advance,
Stephane

Re: ssd option for USB flash drive?

[Hubert Kario]

[-- Attachment #1: smime.p7m --]
[-- Type: application/pkcs7-mime, Size: 5451 bytes --]

MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwGggCSABIIG
jUNvbnRlbnQtVHlwZTogVGV4dC9QbGFpbjsNCiAgY2hhcnNldD0iaXNvLTg4NTktMSINCkNvbnRl
bnQtVHJhbnNmZXItRW5jb2Rpbmc6IHF1b3RlZC1wcmludGFibGUNCg0KT24gV2VkbmVzZGF5IDE4
IG9mIE1heSAyMDExIDAwOjAyOjUyIFN0ZXBoYW5lIENoYXplbGFzIHdyb3RlOg0KPiBIaXlhLA0K
Pj0yMA0KPiBJJ3ZlIG5vdCBmb3VuZCBtdWNoIGRldGFpbCBvbiB3aGF0IHRoZSAic3NkIiBidHJm
cyBtb3VudCBvcHRpb24NCj4gZGlkLiBXb3VsZCBpdCBtYWtlIHNlbnNlIHRvIGVuYWJsZSBpdCB0
byBhIGZzIG9uIGEgVVNCIGZsYXNoDQo+IGRyaXZlPw0KDQp5ZXMsIGVuYWJsaW5nIGRpc2NhcmQg
aXMgcG9pbnRsZXNzIHRob3VnaCAobm8gVVNCIHN0b3JhZ2Ugc3VwcG9ydHMgaXQgQUZBSUs9DQop
Lg0KPTIwDQo+IEknbSB1c2luZyBidHJmcyAob3ZlciBMVk0pIG9uIGEgTGl2ZSBMaW51eCBVU0Ig
c3RpY2sgdG8gYmVuZWZpdA0KPiBmcm9tIGJ0cmZzJ3MgY29tcHJlc3Npb24gYW5kIGFtIHRyeWlu
ZyB0byBpbXByb3ZlIHRoZQ0KPiBwZXJmb3JtYW5jZS4NCg0Kc3NkIG1vZGUgd29uJ3QgaW1wcm92
ZSBwZXJmb3JtYW5jZSBieSBtdWNoIChpZiBhbnkpLg0KDQpZb3UgbmVlZCB0byByZW1lbWJlciB0
aGF0IFVTQjIuMCBpcyBsaW1pdGVkIHRvIGFib3V0IDIwLTMwTWlCL3MgKGRlcGVuZGluZyA9DQpv
bj0yMA0KQ1BVKSBzbyBpdCB3aWxsIGJlIHNsb3cgbm8gbWF0dGVyIHdoYXQgeW91IGRvDQo9MjAN
Cj4gV291bGQgYW55Ym9keSBoYXZlIGFueSByZWNvbW1lbmRhdGlvbiBvbiBob3cgdG8gaW1wcm92
ZQ0KPiBwZXJmb3JtYW5jZSB0aGVyZT8gTGlrZSB3aGF0IHdvdWxkIGJlIHRoZSBiZXN0IHdheSB0
bw0KPiBlbmFibGUvaW5jcmVhc2Ugd3JpdGViYWNrIGJ1ZmZlciBvciBhbnkgd2F5IHRvIG1ha2Ug
c3VyZSB3cml0ZXMNCj4gYXJlIGRlbGF5ZWQgYW5kIGFzeW5jaHJvbm91cz8gV291bGQgZGlzYWJs
aW5nIHJlYWQtYWhlYWQgaGVscD8NCj4gKGF0IHdoaWNoIGxldmVsIHdvdWxkIGl0IGJlIGRvbmU/
KS4gQW55IG90aGVyIHRpcCAobGlrZQ0KPiBkaXNhYmxpbmcgYXRpbWUsIGFsaWduaW5nIGJsb2Nr
cy9leHRlbnRzLCBmaWd1cmUgb3V0IGVyYXNlIGJsb2NrDQo+IHNpemVzIGlmIHJlbGV2YW50Li4u
KT8NCg0KYWxpZ25pbmcgbG9naWNhbCBibG9ja3MgdG8gZXJhc2UgYmxvY2tzIGNhbiBnaXZlIHNv
bWUgcGVyZm9ybWFuY2UgYnV0IHRoZSBvPQ0Kbmx5PTIwDQp3YXkgdG8gbWFrZSBpdCByZWFsbHkg
ZmFzdCBpcyBub3QgdG8gdXNlIFVTQg0KPTIwDQo+IE1hbnkgdGhhbmtzIGluIGFkdmFuY2UsDQo+
IFN0ZXBoYW5lDQo+IC0tDQo+IFRvIHVuc3Vic2NyaWJlIGZyb20gdGhpcyBsaXN0OiBzZW5kIHRo
ZSBsaW5lICJ1bnN1YnNjcmliZSBsaW51eC1idHJmcyIgaW4NCj4gdGhlIGJvZHkgb2YgYSBtZXNz
YWdlIHRvIG1ham9yZG9tb0B2Z2VyLmtlcm5lbC5vcmcNCj4gTW9yZSBtYWpvcmRvbW8gaW5mbyBh
dCAgaHR0cDovL3ZnZXIua2VybmVsLm9yZy9tYWpvcmRvbW8taW5mby5odG1sDQoNCj0yRC09MjAN
Ckh1YmVydCBLYXJpbw0KUUJTIC0gUXVhbGl0eSBCdXNpbmVzcyBTb2Z0d2FyZQ0KMDItNjU2IFdh
cnN6YXdhLCB1bC4gS3Nhd2VyPUYzdyAzMC84NQ0KdGVsLiArNDggKDIyKSA2NDYtNjEtNTEsIDY0
Ni03NC0yNA0Kd3d3LnFicy5jb20ucGwNCgAAAAAAAKCCBsowggbGMIIErqADAgECAghgRNYwzIbz
HzANBgkqhkiG9w0BAQsFADBRMSowKAYDVQQLDCFRQlMgQ2xhc3MgMyBDZXJ0aWZpY2F0ZSBBdXRo
b3JpdHkxFjAUBgNVBAoMDVFCUyBKYW4gS3ViYW4xCzAJBgNVBAYTAlBMMB4XDTExMDUxNjEwMDUz
MloXDTEyMDUxNjEwMDUzMlowgaYxHTAbBgkqhkiG9w0BCQEWDmhrYUBxYnMuY29tLnBsMRUwEwYD
VQQDDAxIdWJlcnQgS2FyaW8xDzANBgNVBCoMBkh1YmVydDEOMAwGA1UEBAwFS2FyaW8xFzAVBgNV
BAoMDlFCUyBKYW4gS3ViYcWEMREwDwYDVQQHDAhXYXJzemF3YTEUMBIGA1UECAwLbWF6b3dpZWNr
aWUxCzAJBgNVBAYTAlBMMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtmTUo4Y/Qnf6
MzeQiOYc56cvVa1b3VJaZDGC3ZSj8rPlj68nHL/Tk9FCRI3SYlJ1efskI4j8zVlxKTLetPCEUqE3
PO2UQ0OaFCpdcbZQ9yvVXUbEVNAZ5quE856LmQzQ33/KvsekLEzfYfeLS1yjqrhqIUYblkycZT6w
owsx70NfXtfSb6H9DDjX1u0WpU6jK8lLaphEA9vQB+zEfYQWraCY1E0auEiGbPfPNi3x0if/ckVu
UYCwr7IJaQg4hcYXFhUuKcbbFebUDOl8L1oi/2QwxC83U87pan+2awtbCWD33Q+QwSRoHUuX6pQs
0sPnZehh+PfdmAC9MkWAg8fXRwIDAQABo4ICSjCCAkYwUQYIKwYBBQUHAQEERTBDMEEGCCsGAQUF
BzABhjVodHRwOi8vY2EucWJzLmNvbS5wbDo4MDgwL2VqYmNhL3B1YmxpY3dlYi9zdGF0dXMvb2Nz
cDAdBgNVHQ4EFgQUZ6aWItDHMTdc4y1LLdYvTB/kmhswDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAW
gBRThM4WgbPV3Ji/Qgr6taX1RUTbLDCB/QYDVR0fBIH1MIHyMIHvoIGVoIGShoGPaHR0cDovL2Nh
LnFicy5jb20ucGw6ODA4MC9lamJjYS9wdWJsaWN3ZWIvd2ViZGlzdC9jZXJ0ZGlzdD9jbWQ9Y3Js
Jmlzc3Vlcj1vdT1RQlMlMjBDbGFzcyUyMDMlMjBDZXJ0aWZpY2F0ZSUyMEF1dGhvcml0eSxvPVFC
UyUyMEphbiUyMEt1YmFuLGM9UEyiVaRTMFExKjAoBgNVBAsMIVFCUyBDbGFzcyAzIENlcnRpZmlj
YXRlIEF1dGhvcml0eTEWMBQGA1UECgwNUUJTIEphbiBLdWJhbjELMAkGA1UEBhMCUEwwDgYDVR0P
AQH/BAQDAgSwMGIGA1UdJQRbMFkGCCsGAQUFBwMCBggrBgEFBQcDAwYIKwYBBQUHAwQGCisGAQQB
gjcUAgIGCisGAQQBgjcKAwwGCisGAQQBgjcCARUGCisGAQQBgjcCARYGCSqGSIb3LwEBBTAvBgNV
HREEKDAmgQ5oa2FAcWJzLmNvbS5wbIEUd2VibWFzdGVyQHFicy5jb20ucGwwDQYJKoZIhvcNAQEL
BQADggIBAE2d0JelbaaQJRuxZxndox3YC6JbOU+rG2wtQ0NGrS2xtlqW3NnFrwj0Z9aCAJxFh1kD
6HpeGqhx6QHILCIjjogJ4hJsge66B37oP+xd1SJVGHlpeyA/hS2Fj6r0YnTT6umy0i6tDO9lvbPe
walfklVCwkjqnJsFiMI4qew7HY/Gc/GxHABjE2W9NeXtDDfqzxJ3LUXwuqVHF74EMO7N1mBgiGOM
7yAVuOCGzpX86fgWFpLqRHKymuSj9VgIAa80eX3+LOZ3YYzcSAbY2TJbN1BmU5kfGQMTIceXzB8I
nub1PfUQ7mrLhheq3UIt+3MaUtgfhi2Um1RPtxway6JtUpu+pDXHukM5vgYHRfIgfVE5jClLLeIv
9ktjDpd5cGhhQjtzfeE+kaP8323xj2M+c2RJurcj2TGE48dje3aGLK4a7VSd8fsGNgdOYjua3f29
kRGX/O1hug+bRAd0WYYEjIZL/eB8EAm6k75B9IyGvdyXP5xuSxW4kRRj6cU22M8evhFJ6uM/Rljn
3UESuob2m7hsloJVJbqDPUlvkqT8q26VLdiSoHY0vikw+oxqdgmTm8dicjBi+ajvkonqDP3d2sm+
yqpOld0+2HvEzYrPxCzg85H5WWCmTjR+bgbb04gFWi0e7P6XhGysv5O7TJTFOkZ4WnKM6QqUrpSa
M9PC4eR7MYICHjCCAhoCAQEwXTBRMSowKAYDVQQLDCFRQlMgQ2xhc3MgMyBDZXJ0aWZpY2F0ZSBB
dXRob3JpdHkxFjAUBgNVBAoMDVFCUyBKYW4gS3ViYW4xCzAJBgNVBAYTAlBMAghgRNYwzIbzHzAN
BglghkgBZQMEAgEFAKCBkzAYBgkqhkiG9w0BCQMxCwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEP
Fw0xMTA1MTkxOTA0NTVaMCgGCSqGSIb3DQEJDzEbMBkwCwYJYIZIAWUDBAECMAoGCCqGSIb3DQMH
MC8GCSqGSIb3DQEJBDEiBCBkXfVNESLSWFdkBgMB3fW0zS2M6TWmt3V2nbxUsHEgEjANBgkqhkiG
9w0BAQEFAASCAQAxmupWuvLpLnm8RTEME9hECDjN89WGaIB+SPQ+lDHaxJ7NZQFv5d6wUX9BsWcb
A/Sp4QA/vjz1HUhSdauoKlDVb0QjGudjo2POa4+JH5gV63OzvDNNkIM/KX+g6i+1K796YEk42W6W
bynvboIYDtChcVnRYVFn8vlkTSb94AN+loLHEhCA+uwjSoaJLvpvDUmkwDZg89+jI8HqvrpMvxP5
gEw0vrVVdhXVaYbsX7POUzSQVquwEhO/lvK4V7wkeBwnGg7fOAQ/KIC3Ay9YttZX1E5dD3aGACR5
G7BaviWXAT7RkiK91nHqhVBZNGnlVl7OCerha5GYJMp58OE5EETQAAAAAAAA

Re: ssd option for USB flash drive?

[Hubert Kario]

Sorry, loks like list mailer doesn't like SMIME messages.

On Thursday 19 of May 2011 21:04:54 Hubert Kario wrote:
> On Wednesday 18 of May 2011 00:02:52 Stephane Chazelas wrote:
> > Hiya,
> >=20
> > I've not found much detail on what the "ssd" btrfs mount option
> > did. Would it make sense to enable it to a fs on a USB flash
> > drive?
>=20
> yes, enabling discard is pointless though (no USB storage supports it
> AFAIK).
>=20
> > I'm using btrfs (over LVM) on a Live Linux USB stick to benefit
> > from btrfs's compression and am trying to improve the
> > performance.
>=20
> ssd mode won't improve performance by much (if any).
>=20
> You need to remember that USB2.0 is limited to about 20-30MiB/s (depe=
nding
> on CPU) so it will be slow no matter what you do
>=20
> > Would anybody have any recommendation on how to improve
> > performance there? Like what would be the best way to
> > enable/increase writeback buffer or any way to make sure writes
> > are delayed and asynchronous? Would disabling read-ahead help?
> > (at which level would it be done?). Any other tip (like
> > disabling atime, aligning blocks/extents, figure out erase block
> > sizes if relevant...)?
>=20
> aligning logical blocks to erase blocks can give some performance but=
 the
> only way to make it really fast is not to use USB
>=20
> > Many thanks in advance,
> > Stephane

--=20
Hubert Kario
QBS - Quality Business Software
02-656 Warszawa, ul. Ksawer=F3w 30/85
tel. +48 (22) 646-61-51, 646-74-24
www.qbs.com.pl
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ssd option for USB flash drive?

[Stephane Chazelas]

2011-05-19 21:04:54 +0200, Hubert Kario:
> On Wednesday 18 of May 2011 00:02:52 Stephane Chazelas wrote:
> > Hiya,
> > 
> > I've not found much detail on what the "ssd" btrfs mount option
> > did. Would it make sense to enable it to a fs on a USB flash
> > drive?
> 
> yes, enabling discard is pointless though (no USB storage supports it AFAIK).
>  
> > I'm using btrfs (over LVM) on a Live Linux USB stick to benefit
> > from btrfs's compression and am trying to improve the
> > performance.
> 
> ssd mode won't improve performance by much (if any).
> 
> You need to remember that USB2.0 is limited to about 20-30MiB/s (depending on 
> CPU) so it will be slow no matter what you do

Thanks Hubert for the feedback.

Well, for hard drives over USB, I can get to 40MiB/s read and
write easily. Here, I believe the bottle neck is the flash
memory. With that particular USB flash drive Corsair Voyager GT
16GB, I can get 25MiB/s sequential read and 17MiB/s sequential
write, but that falls down to about 3-5MiB/s random write.

[...]
> aligning logical blocks to erase blocks can give some performance but the only 
> way to make it really fast is not to use USB
[...]

For something that fits in your pocket and is almost
universally bootable, there are not so many other options.

I tried changing the alignment on FAT32 and it didn't make
any difference. Playing with /proc/sys/vm/block_dump, I could see
chunks of 3, 4, 5 data sectors being written at once regardless
of the cluster size being used anyway. Interestingly when a user
process writes to /dev/sdx, block_dump shows 4k writes to
/dev/sdx only regardless of the size of the user writes while if
it goes via the filesystem I can see writes of up to 120k. Also,
I've very little knowledge of what happens at layers below the
block device (scsi interface, usb-storage, and the device
controller itself, for instance, I see
/sys/block/sdi/queue/rotational is 1 for that usb stick, why,
what does that mean in terms of performance and scheduling of
read-writes?)

I wonder now what credit to give to recommendations like in
http://www.patriotmemory.com/forums/showthread.php?3696-HOWTO-Increase-write-speed-by-aligning-FAT32
http://linux-howto-guide.blogspot.com/2009/10/increase-usb-flash-drive-write-speed.html

Doing a apt-get upgrade on that stick takes hours when the same
takes a few minutes on an internal drive.

If I boot a kvm virtual machine on that USB stick with a disk
cache mode of "unsafe", that is writes are hardly every flushed
to underlying storage, then that becomes lightning fast (at the
expense of possibly losing data in case of host failure, but I'm
not too worried about that), and flushing writes to device
upon VM shutdown only takes a couple of minutes.

So I figured that if I could make sure writing to the flash
device is asynchronous (and reads priviledged), that would help.

There's probably some solutions with aufs or some fuse
solutions, but I thought there might be some solution in btrfs
or some standard core layers usually underneath it.

-- 
Stephane

Re: ssd option for USB flash drive?

[cwillu]

> [...]
>> aligning logical blocks to erase blocks can give some performance but the only
>> way to make it really fast is not to use USB
> [...]
>
> For something that fits in your pocket and is almost
> universally bootable, there are not so many other options.

An ssd drive in a USB enclosure is about the size of a cell phone,
just a thought.

> I tried changing the alignment on FAT32 and it didn't make
> any difference. Playing with /proc/sys/vm/block_dump, I could see
> chunks of 3, 4, 5 data sectors being written at once regardless
> of the cluster size being used anyway. Interestingly when a user
> process writes to /dev/sdx, block_dump shows 4k writes to
> /dev/sdx only regardless of the size of the user writes while if
> it goes via the filesystem I can see writes of up to 120k. Also,
> I've very little knowledge of what happens at layers below the
> block device (scsi interface, usb-storage, and the device
> controller itself, for instance, I see
> /sys/block/sdi/queue/rotational is 1 for that usb stick, why,
> what does that mean in terms of performance and scheduling of
> read-writes?)

Try with the "ssd_spread" mount option.

> I wonder now what credit to give to recommendations like in
> http://www.patriotmemory.com/forums/showthread.php?3696-HOWTO-Increase-write-speed-by-aligning-FAT32
> http://linux-howto-guide.blogspot.com/2009/10/increase-usb-flash-drive-write-speed.html
>
> Doing a apt-get upgrade on that stick takes hours when the same
> takes a few minutes on an internal drive.

Also, there's a package "libeatmydata" which will provide an
"eatmydata" command, which you can prefix your apt-get commands with.
This will disable the excessive sync calls that dpkg makes, and should
dramatically decrease the time for those sorts of things to finish.
Flash as found in thumb drives doesn't have much in the way of crash
guarantees anyway, so you're not really giving up much safety.

Re: ssd option for USB flash drive?

[Stephane Chazelas]

2011-05-19 15:54:23 -0600, cwillu:
[...]
> Try with the "ssd_spread" mount option.
[...]

Thanks. I'll try that.

> > I wonder now what credit to give to recommendations like in
> > http://www.patriotmemory.com/forums/showthread.php?3696-HOWTO-Increase-write-speed-by-aligning-FAT32
> > http://linux-howto-guide.blogspot.com/2009/10/increase-usb-flash-drive-write-speed.html
> >
> > Doing a apt-get upgrade on that stick takes hours when the same
> > takes a few minutes on an internal drive.
> 
> Also, there's a package "libeatmydata" which will provide an
> "eatmydata" command, which you can prefix your apt-get commands with.
> This will disable the excessive sync calls that dpkg makes, and should
> dramatically decrease the time for those sorts of things to finish.
> Flash as found in thumb drives doesn't have much in the way of crash
> guarantees anyway, so you're not really giving up much safety.

Thanks. That's very useful indeed.

Note that if you use that on aptitude/apg-get that means that
the daemons started/restarted in the process will be affected,
but it could be all the better in my case.

Now, with that eatmydata, I'm thinking of trying qemu-nbd -c
/dev/nbd0 /dev/mapper/original-device with that and have the
rootfs mounted on that /dev/nbd0.

That eatmydata could be a work around to the problem I was
mentionning at
https://lists.ubuntu.com/archives/ubuntu-server-bugs/2010-June/037846.html

-- 
Stephane

Re: ssd option for USB flash drive?

[cwillu]

On Thu, May 19, 2011 at 4:12 PM, Stephane Chazelas
<stephane_chazelas@yahoo.fr> wrote:
> 2011-05-19 15:54:23 -0600, cwillu:
> [...]
>> Try with the "ssd_spread" mount option.
> [...]
>
> Thanks. I'll try that.
>
>> > I wonder now what credit to give to recommendations like in
>> > http://www.patriotmemory.com/forums/showthread.php?3696-HOWTO-Increase-write-speed-by-aligning-FAT32
>> > http://linux-howto-guide.blogspot.com/2009/10/increase-usb-flash-drive-write-speed.html
>> >
>> > Doing a apt-get upgrade on that stick takes hours when the same
>> > takes a few minutes on an internal drive.
>>
>> Also, there's a package "libeatmydata" which will provide an
>> "eatmydata" command, which you can prefix your apt-get commands with.
>> This will disable the excessive sync calls that dpkg makes, and should
>> dramatically decrease the time for those sorts of things to finish.
>> Flash as found in thumb drives doesn't have much in the way of crash
>> guarantees anyway, so you're not really giving up much safety.
>
> Thanks. That's very useful indeed.
>
> Note that if you use that on aptitude/apg-get that means that
> the daemons started/restarted in the process will be affected,
> but it could be all the better in my case.

Heh, that's a thought I hadn't actually considered :p

That shouldn't affect any services that are managed by message
passing, and so really should be limited to those services from
/etc/init.d/ that don't restart themselves (i.e., where the restart
command is implemented by stop + start rather than telling the already
running process to re-execute), or newly installed services that again
are managed via /etc/init.d/.