Example of BTRFS uglyssima performance : Bitcoin

Example of BTRFS uglyssima performance : Bitcoin

[Swâmi Petaramesh]

Hi folks,

My laptop is a Core i3-2310M with 4 GB RAM, running BTRFS on a 1 TB HD.

I run Ubuntu 12.10 Quantal, kernel 3.5.0-19-generic 64-bit.

I wanted to give a shot at bitcoin so I installed it from the Ubuntu
PPA, and started getting the database from the Internet. (it's now 4.5
GB big on disk). I assume it an SQL DB (SQlite or so ?)

My system currently has been crushing its data for more that 5 days now
(5x24 hours) with the HD busy to the point that the system is completely
unusable and I had to take another laptop to be able to surf the web and
process my email. Disk LED is steadily lit, trying to switch between 2
open windows takes 5 minutes or so...

Entering new blocks into the DB has now slowed down to the point that I
assume that the last 6% that I miss (I'm now at 94.something %) may well
take another couple weeks or so...

Friends running ext4 told me that the initial loading of the DB took
them a couple hours... With BTRFS it's been 5x24 hours and counting... :-(

This filesystem is pure crap as soon as it comes to database processing :-(

Kind regards.

-- 
Swâmi Petaramesh <swami@petaramesh.org> http://petaramesh.org PGP 9076E32E
Ne cherchez pas : Je ne suis pas sur Facebook.

Re: Example of BTRFS uglyssima performance : Bitcoin

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1827 bytes --]

On Mon, Dec 03, 2012 at 12:54:30PM +0100, Swâmi Petaramesh wrote:
> Hi folks,
> 
> My laptop is a Core i3-2310M with 4 GB RAM, running BTRFS on a 1 TB HD.
> 
> I run Ubuntu 12.10 Quantal, kernel 3.5.0-19-generic 64-bit.

   Try 3.7. That's had some significant performance improvements. 3.5
is over 6 months old, which is a long time in btrfs development.

> I wanted to give a shot at bitcoin so I installed it from the Ubuntu
> PPA, and started getting the database from the Internet. (it's now 4.5
> GB big on disk). I assume it an SQL DB (SQlite or so ?)

   Try marking the SQLite database file(s) as nocow, with chattr +C.
That also only works on 3.7, but should deal with your database
problems.

> My system currently has been crushing its data for more that 5 days now
> (5x24 hours) with the HD busy to the point that the system is completely
> unusable and I had to take another laptop to be able to surf the web and
> process my email. Disk LED is steadily lit, trying to switch between 2
> open windows takes 5 minutes or so...
> 
> Entering new blocks into the DB has now slowed down to the point that I
> assume that the last 6% that I miss (I'm now at 94.something %) may well
> take another couple weeks or so...
> 
> Friends running ext4 told me that the initial loading of the DB took
> them a couple hours... With BTRFS it's been 5x24 hours and counting... :-(
> 
> This filesystem is pure crap as soon as it comes to database processing :-(

   3.5 was. 3.7 shouldn't be -- particularly with the use of +C on the
database files.

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
   --- "No!  My collection of rare, incurable diseases! Violated!" ---   

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Example of BTRFS uglyssima performance : Bitcoin

[Swâmi Petaramesh]

Le 03/12/2012 13:09, Hugo Mills a écrit :
> Try 3.7. That's had some significant performance improvements. 3.5 is
> over 6 months old, which is a long time in btrfs development. 

I understand the suggestion from a developper's PoV, but from a user's
that's much too much hassle living ahead of one's distro's kernel, or
using "vanilla" ones. Been there, done that. No more suffering for me
please ;-))

I'll have to stick with current Ubuntu kernel, or at least with a future
backport...

Kind regards.

-- 
Swâmi Petaramesh <swami@petaramesh.org> http://petaramesh.org PGP 9076E32E
Ne cherchez pas : Je ne suis pas sur Facebook.

Re: Example of BTRFS uglyssima performance : Bitcoin

[Chris Samuel]

On 03/12/12 23:24, Swâmi Petaramesh wrote:

> I understand the suggestion from a developper's PoV, but from a
> user's that's much too much hassle living ahead of one's distro's
> kernel, or using "vanilla" ones. Been there, done that. No more
> suffering for me please ;-))
> 
> I'll have to stick with current Ubuntu kernel, or at least with a
> future backport...

In that case maybe using an experimental filesystem that is under rapid
development might not be a good choice, it might be better to stick to
one of the existing stable filesystems instead.

Have you benchmarked your workload on other filesystems?

cheers,
Chris
-- 
 Chris Samuel  :  http://www.csamuel.org/  :  Melbourne, VIC

Re: Example of BTRFS uglyssima performance : Bitcoin

[Swâmi Petaramesh]

Hi Chris,

Le 04/12/2012 03:18, Chris Samuel a écrit :
> In that case maybe using an experimental filesystem that is under rapid
> development might not be a good choice, it might be better to stick to
> one of the existing stable filesystems instead.

I already made the move back and forth ext4 <-> BTRFS twice... Reading
here or there that the new (or the next) kernel is the one that indeed
fixes the bugs (3.3) or gives performance improvements (3.5), or will
make you happy this time, etc...

And each time I'm told that well, the current kernel is not so good
actually, but the next one is so much better ! So you should try the
next beta, or fetch and compile the curent mainline kernel...
...Neverending story...

All this boils down to : I do really need some of the features of BTRFS
(snapshots, checksumming, supposed high reliability...) but I also need
my computers to stay usable on a daily basis without too much hassle and
not spending 3 hours a day beta-testing or upgrading kernels... or
waiting for my mouse click to produce a result.

For getting this result I have a choice between :

- ZFS, that has proven rock-solid, feature-rich and efficient on one of
my machines for 2+ years, but is'nt part of "standard" Linux, will
probably never be, so causes issues at each distro, is tricky installing
as rootfs and which future in Linux is unclear (I don't want to have to
reinstall in 6 months a machine I install now because its filesystem
would end being unsupported...)

- BTRFS, which has already proven that it could kill all my data 3
times, which is badly slow, immature, but part of the standard kernel,
and supposed to improve at a fast pace and have its future ahead.

So basically each time I install a new system, I ask myself : ext4 ?
(but I will lack features I badly need)... ZFS ? (Excellent today, but
will I have to reinstall the machine in 6 months ?)... BTRFS (a slow
unreliable pain in the a** today, but is it this time mature enough that
this bet will make sense ?).

So well... 3.5 is a pain for databases... Now I hope I will be happy in
6 months at next Ubuntu release... If not, maybe in a year... Or a
couple years... Well...

Kind regards.

-- 
Swâmi Petaramesh <swami@petaramesh.org> http://petaramesh.org PGP 9076E32E
Ne cherchez pas : Je ne suis pas sur Facebook.

Re: Example of BTRFS uglyssima performance : Bitcoin

[Wade Cline]

Hi Swâmi,

On 12/03/2012 04:09 AM, Hugo Mills wrote:

> On Mon, Dec 03, 2012 at 12:54:30PM +0100, Swâmi Petaramesh wrote:
>> Hi folks,
>>
>> My laptop is a Core i3-2310M with 4 GB RAM, running BTRFS on a 1 TB HD.
>>
>> I run Ubuntu 12.10 Quantal, kernel 3.5.0-19-generic 64-bit.
>
>     Try 3.7. That's had some significant performance improvements. 3.5
> is over 6 months old, which is a long time in btrfs development.

I'd like to emphasize this point. 3.7 has some _very_ significant performance
improvements for database workloads; you will _need_ to run at least a 3.7
kernel to have an acceptable run speed for that type of workload (at least
in my experience). Seriously.

Also, the nodatacow option Hugo mentioned will also help.

Regards,
Wade

Re: Example of BTRFS uglyssima performance : Bitcoin

[Jan Schmidt]

Hi Swâmi,

On Mon, December 03, 2012 at 12:54 (+0100), Swâmi Petaramesh wrote:
> Hi folks,
> 
> My laptop is a Core i3-2310M with 4 GB RAM, running BTRFS on a 1 TB HD.
> 
> I run Ubuntu 12.10 Quantal, kernel 3.5.0-19-generic 64-bit.
> 
> I wanted to give a shot at bitcoin so I installed it from the Ubuntu
> PPA, and started getting the database from the Internet. (it's now 4.5
> GB big on disk). I assume it an SQL DB (SQlite or so ?)
> 
> My system currently has been crushing its data for more that 5 days now
> (5x24 hours) with the HD busy to the point that the system is completely
> unusable and I had to take another laptop to be able to surf the web and
> process my email. Disk LED is steadily lit, trying to switch between 2
> open windows takes 5 minutes or so...
> 
> Entering new blocks into the DB has now slowed down to the point that I
> assume that the last 6% that I miss (I'm now at 94.something %) may well
> take another couple weeks or so...
> 
> Friends running ext4 told me that the initial loading of the DB took
> them a couple hours... With BTRFS it's been 5x24 hours and counting... :-(

I've experienced exactly the same as you have with btrfs, only with ext4 instead
of btrfs: Use ubuntu (which in the default setup means you're using ecryptfs for
your /home), install the bitcoin package (which puts its files in ~/.bitcoin)
and the computer becomes very much unresponsive once bitcoin is fetching data
blocks.

Unresponsive here means that even the desktop clock misses to count a second
here and then, the mouse pointer stops reacting. Haven't looked further into
this, but I suspect ecryptfs isn't completely innocent in that stack. That said,
I suggest trying to make an ext4 partition on the very same hardware, setup
ecryptfs, mount it as ~/.bitcoin and share your findings :-)

Besides that, as Hugo told you, you can disable btrfs cow on the database files,
but given my experiences I wouldn't put too much hope into that part.

-Jan

Re: Example of BTRFS uglyssima performance : Bitcoin

[Swâmi Petaramesh]

Le 03/12/2012 16:55, Jan Schmidt a écrit :
> Use ubuntu (which in the default setup means you're using ecryptfs for
> your /home),
I actually do not use ecryptfs here, but OTOH I have BTRFS over LUKS/LVM.

But I've been using pretty *anything over LUKS/LVM for years, and I've
never notice it cause any (noticeable to the point of becoming annoying)
system slowdown, whatever tasks I may have processed in such setups
(including servers, big databases, compilations, NAS, etc...)

So I doubt that the encryption is involved here... At least not very
heavily involved. But I may ask my son if he volunteers to test the same
on his unencrypted BTRFS netbook...

> Besides that, as Hugo told you, you can disable btrfs cow on the database files,
> but given my experiences I wouldn't put too much hope into that part.

As far as I understood, this option won't work unless I have a 3.7+
kernel...

BTW, what is the effect of "nocow" with respect to snapshots ? I would
assume that then, snapshots contain the current data ?

Kind regards.

-- 
Swâmi Petaramesh <swami@petaramesh.org> http://petaramesh.org PGP 9076E32E
Ne cherchez pas : Je ne suis pas sur Facebook.

Re: Example of BTRFS uglyssima performance : Bitcoin

[Sander]

Swâmi Petaramesh wrote (ao):
> But I've been using pretty *anything over LUKS/LVM for years, and I've
> never notice it cause any (noticeable to the point of becoming annoying)
> system slowdown, whatever tasks I may have processed in such setups
> (including servers, big databases, compilations, NAS, etc...)

If your system is stable, you could also consider running bitcoin with
eatmydata.

	Sander

Re: Example of BTRFS uglyssima performance : Bitcoin

[Gregory Maxwell]

On Mon, Dec 3, 2012 at 6:54 AM, Swâmi Petaramesh <swami@petaramesh.org> wrote:
> I assume it an SQL DB (SQlite or so ?)

It's BDB with a very pessimal random write workload updating
transaction indexes with stuffed full of sync writes, interspersed
with an equal amount of random reads.

(The next version of the software will change the backend in several
ways, including replacing BDB with leveldb and its orders of magnitude
faster;  but for now it's probably a fine FS abusive test case)