* compression ratio @ 2011-06-20 11:29 Proskurin Kirill 2011-06-20 13:46 ` David Sterba 2011-06-20 14:34 ` Helmut Hullen 0 siblings, 2 replies; 6+ messages in thread From: Proskurin Kirill @ 2011-06-20 11:29 UTC (permalink / raw) To: linux-btrfs Hello all. I`m new to btrfs and do some testing now. What we have: SL6 - kernel 2.6.32-131.2.1.el6.x86_64 mdadm RAID5 with 8 HDD - 27T partition. Mount options is "noatime,noacl,compress-force" I use scribe daemon to copy log files from 200 hosts to that partition for stress testing. But I found what compression ratio is really small. Partition is full of regular log files - plain text. Most of them a big ones(10-20Gb) and they grow in realtime. du -hs /logs/ 1.1T /logs/ And compressed size is(df -h): 908G Regular gzip do a significant more ratio. There is a problem here? File size? Realtime grow of a files? -- Best regards, Proskurin Kirill ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: compression ratio 2011-06-20 11:29 compression ratio Proskurin Kirill @ 2011-06-20 13:46 ` David Sterba 2011-06-20 14:34 ` Helmut Hullen 1 sibling, 0 replies; 6+ messages in thread From: David Sterba @ 2011-06-20 13:46 UTC (permalink / raw) To: linux-btrfs Hi, On Mon, Jun 20, 2011 at 03:29:45PM +0400, Proskurin Kirill wrote: > What we have: > SL6 - kernel 2.6.32-131.2.1.el6.x86_64 > mdadm RAID5 with 8 HDD - 27T partition. btw .32 is very old > Mount options is "noatime,noacl,compress-force" > I use scribe daemon to copy log files from 200 hosts to that > partition for stress testing. > > But I found what compression ratio is really small. It is and it will be due to nature of compression method used and the constraints given: * compression has to be fast, only real-time methods can be considered which have natural speed/ratio tradeoff * you're using zlib (lzo was added recently), which is not that fast and does not have a great compression ratio * there is maximum length of compressed data per round, it's hardcoded to 128KB by now * zlib works in streaming mode, which compresses whole 128KB with dictionary reused * lzo (as implemented in btrfs) however does not reuse dictionary and compresses each page separately; resulting size is even bigger than it would be when full 128KB were compressed > Partition is full of regular log files - plain text. Most of them a > big ones(10-20Gb) and they grow in realtime. Realtime growth should not be a problem. The situation can be slightly improved if we use a method capable of streamed compression (ie. dictionary reuse). This is naturally supported by eg. quicklz and snappy. There are a few other realtime methods (fastlz, lzrw, lzjb) which may be extended to support streaming compression, but with careful evaluation of the result. david ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: compression ratio 2011-06-20 11:29 compression ratio Proskurin Kirill 2011-06-20 13:46 ` David Sterba @ 2011-06-20 14:34 ` Helmut Hullen 2011-06-20 15:17 ` Proskurin Kirill 1 sibling, 1 reply; 6+ messages in thread From: Helmut Hullen @ 2011-06-20 14:34 UTC (permalink / raw) To: linux-btrfs Hallo, Proskurin, Du meintest am 20.06.11: > I`m new to btrfs and do some testing now. > What we have: > SL6 - kernel 2.6.32-131.2.1.el6.x86_64 > mdadm RAID5 with 8 HDD - 27T partition. You should take a newer kernel. On my system I needed 2.6.38.5 and newer for btrfs; older kernels lead to unreadable partitions. Viele Gruesse! Helmut ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: compression ratio 2011-06-20 14:34 ` Helmut Hullen @ 2011-06-20 15:17 ` Proskurin Kirill 2011-06-20 15:22 ` Hugo Mills 0 siblings, 1 reply; 6+ messages in thread From: Proskurin Kirill @ 2011-06-20 15:17 UTC (permalink / raw) To: linux-btrfs On 06/20/2011 06:34 PM, Helmut Hullen wrote: > Hallo, Proskurin, > > Du meintest am 20.06.11: > >> I`m new to btrfs and do some testing now. > >> What we have: >> SL6 - kernel 2.6.32-131.2.1.el6.x86_64 >> mdadm RAID5 with 8 HDD - 27T partition. > > You should take a newer kernel. On my system I needed 2.6.38.5 and newer > for btrfs; older kernels lead to unreadable partitions. What kernel version is most recommended at this time? -- Best regards, Proskurin Kirill ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: compression ratio 2011-06-20 15:17 ` Proskurin Kirill @ 2011-06-20 15:22 ` Hugo Mills 2011-06-20 15:51 ` Mitch Harder 0 siblings, 1 reply; 6+ messages in thread From: Hugo Mills @ 2011-06-20 15:22 UTC (permalink / raw) To: linux-btrfs [-- Attachment #1: Type: text/plain, Size: 783 bytes --] On Mon, Jun 20, 2011 at 07:17:22PM +0400, Proskurin Kirill wrote: > On 06/20/2011 06:34 PM, Helmut Hullen wrote: > >Du meintest am 20.06.11: > >>What we have: > >>SL6 - kernel 2.6.32-131.2.1.el6.x86_64 > >>mdadm RAID5 with 8 HDD - 27T partition. > > > >You should take a newer kernel. On my system I needed 2.6.38.5 and newer > >for btrfs; older kernels lead to unreadable partitions. > > What kernel version is most recommended at this time? It's always "the latest version you can manage to run". So, currently, 2.6.39 or the 3.0-rc series. Hugo. -- === Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk === PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk --- O tempura! O moresushi! --- [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 190 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: compression ratio 2011-06-20 15:22 ` Hugo Mills @ 2011-06-20 15:51 ` Mitch Harder 0 siblings, 0 replies; 6+ messages in thread From: Mitch Harder @ 2011-06-20 15:51 UTC (permalink / raw) To: Hugo Mills, linux-btrfs On Mon, Jun 20, 2011 at 10:22 AM, Hugo Mills <hugo@carfax.org.uk> wrote= : > On Mon, Jun 20, 2011 at 07:17:22PM +0400, Proskurin Kirill wrote: >> On 06/20/2011 06:34 PM, Helmut Hullen wrote: >> >Du meintest am 20.06.11: >> >>What we have: >> >>SL6 - kernel 2.6.32-131.2.1.el6.x86_64 >> >>mdadm RAID5 with 8 HDD - 27T partition. >> > >> >You should take a newer kernel. On my system I needed 2.6.38.5 and = newer >> >for btrfs; older kernels lead to unreadable partitions. >> >> What kernel version is most recommended at this time? > > =A0 It's always "the latest version you can manage to run". So, > currently, 2.6.39 or the 3.0-rc series. > > =A0 Hugo. > have to audit the code yourself > -- > =3D=3D=3D Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.= org.uk =3D=3D=3D > =A0PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org= =2Euk > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 --- O tempura! O moresushi! -= -- > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.10 (GNU/Linux) > > iD8DBQFN/2W9IKyzvlFcI40RAgmuAKCEbTdS/ZjWzNMX3D5t8zNKbULJQACgyl6X > e2SIUGnkDu1BFagOL2e5ymM=3D > =3DSO2f > -----END PGP SIGNATURE----- > > Actually, the btrfs in the newest RHEL 2.6.32 kernels (and by extension, SL kernels) has been heavily backported, so it is fairly current, although probably not up to 2.6.39 level yet. It isn't the most current, but it is more current than most people realize, and should be fairly stable. I think you'll want the RHEL 6.1 (or SL 6.1) 2.6.32 kernel to get the most recent Btrfs backporting. I haven't seen any detailed information posted on the extent of the backporting. So, for right now, you're kind-of on your own to ascertain if the btrfs in your RHEL kernel is suitable for your needs. I believe the original poster is getting confused by results of the 'df' command. Running 'df' will include a large amount of reserved space for metadata. The Btrfs Wiki has some insights on how to evaluate the space consumed by a Btrfs volume. But, in general, running 'df' will not give you a complete picture of disk utilization with Btrfs. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2011-06-20 15:51 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2011-06-20 11:29 compression ratio Proskurin Kirill 2011-06-20 13:46 ` David Sterba 2011-06-20 14:34 ` Helmut Hullen 2011-06-20 15:17 ` Proskurin Kirill 2011-06-20 15:22 ` Hugo Mills 2011-06-20 15:51 ` Mitch Harder
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.