* using fio to test btrfs compression @ 2017-09-18 7:36 shally verma 2017-09-18 8:26 ` Timofey Titovets 2017-09-21 15:26 ` Liu Bo 0 siblings, 2 replies; 16+ messages in thread From: shally verma @ 2017-09-18 7:36 UTC (permalink / raw) To: linux-btrfs Hi I wanted to test btrfs compression using fio command but somehow during fio writes, I don't see code taking route of compression blocks where as If I do a copy to btrfs compression enabled mount point then I can easily see code falling through compression.c. Here's how I do my setup 1. mkfs.btrfs /dev/sdb1 2. mount -t btrfs -o compress=zlib,compress-force /dev/sdb1 /mnt 3. cp <some large test file> /mnt 4. dmesg shows print staments from compression.c and zlib.c confirming compression routine was invoked during write 5. now, copy back from btrfs mount point to home directory also shows decompress call invokation Now, try same with fio commands: fio command fio --directory=/mnt/ --numjobs=1 --direct=0 --buffered=1 --ioengine=libaio --group_reporting --bs=64k --rw=write --iodepth=128 --name=test --size=10G --runtime=180 --time_based But it seems to write uncompressed data. Any help here? what's missing? Thanks Shally ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: using fio to test btrfs compression 2017-09-18 7:36 using fio to test btrfs compression shally verma @ 2017-09-18 8:26 ` Timofey Titovets 2017-09-18 13:28 ` shally verma 2017-09-21 15:26 ` Liu Bo 1 sibling, 1 reply; 16+ messages in thread From: Timofey Titovets @ 2017-09-18 8:26 UTC (permalink / raw) To: shally verma; +Cc: linux-btrfs 2017-09-18 10:36 GMT+03:00 shally verma <shallyvermacavium@gmail.com>: > Hi > > I wanted to test btrfs compression using fio command but somehow > during fio writes, I don't see code taking route of compression blocks > where as If I do a copy to btrfs compression enabled mount point then > I can easily see code falling through compression.c. > > Here's how I do my setup > > 1. mkfs.btrfs /dev/sdb1 > 2. mount -t btrfs -o compress=zlib,compress-force /dev/sdb1 /mnt > 3. cp <some large test file> /mnt > 4. dmesg shows print staments from compression.c and zlib.c confirming > compression routine was invoked during write > 5. now, copy back from btrfs mount point to home directory also shows > decompress call invokation > > Now, try same with fio commands: > > fio command > > fio --directory=/mnt/ --numjobs=1 --direct=0 --buffered=1 > --ioengine=libaio --group_reporting --bs=64k --rw=write --iodepth=128 > --name=test --size=10G --runtime=180 --time_based > > But it seems to write uncompressed data. > > Any help here? what's missing? > > Thanks > Shally > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html 1. mount -t btrfs -o compress=zlib,compress-force -> compress-force=zlib 2. Tune fio to generate compressible data -- Have a nice day, Timofey. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: using fio to test btrfs compression 2017-09-18 8:26 ` Timofey Titovets @ 2017-09-18 13:28 ` shally verma 2017-09-18 13:41 ` Timofey Titovets 0 siblings, 1 reply; 16+ messages in thread From: shally verma @ 2017-09-18 13:28 UTC (permalink / raw) To: Timofey Titovets; +Cc: linux-btrfs, Verma, Shally On Mon, Sep 18, 2017 at 1:56 PM, Timofey Titovets <nefelim4ag@gmail.com> wrote: > 2017-09-18 10:36 GMT+03:00 shally verma <shallyvermacavium@gmail.com>: >> Hi >> >> I wanted to test btrfs compression using fio command but somehow >> during fio writes, I don't see code taking route of compression blocks >> where as If I do a copy to btrfs compression enabled mount point then >> I can easily see code falling through compression.c. >> >> Here's how I do my setup >> >> 1. mkfs.btrfs /dev/sdb1 >> 2. mount -t btrfs -o compress=zlib,compress-force /dev/sdb1 /mnt >> 3. cp <some large test file> /mnt >> 4. dmesg shows print staments from compression.c and zlib.c confirming >> compression routine was invoked during write >> 5. now, copy back from btrfs mount point to home directory also shows >> decompress call invokation >> >> Now, try same with fio commands: >> >> fio command >> >> fio --directory=/mnt/ --numjobs=1 --direct=0 --buffered=1 >> --ioengine=libaio --group_reporting --bs=64k --rw=write --iodepth=128 >> --name=test --size=10G --runtime=180 --time_based >> >> But it seems to write uncompressed data. >> >> Any help here? what's missing? >> >> Thanks >> Shally >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html > > 1. mount -t btrfs -o compress=zlib,compress-force -> compress-force=zlib > 2. Tune fio to generate compressible data > How do I "tune" fio to generate data. I had assumed once compression is enabled on btrfs any system fwrite call will simply compress data into it .Isn't it so? Can you share fio command that I can test? Thanks Shally > > -- > Have a nice day, > Timofey. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: using fio to test btrfs compression 2017-09-18 13:28 ` shally verma @ 2017-09-18 13:41 ` Timofey Titovets 2017-09-20 8:36 ` shally verma 0 siblings, 1 reply; 16+ messages in thread From: Timofey Titovets @ 2017-09-18 13:41 UTC (permalink / raw) To: shally verma; +Cc: linux-btrfs, Verma, Shally 2017-09-18 16:28 GMT+03:00 shally verma <shallyvermacavium@gmail.com>: > On Mon, Sep 18, 2017 at 1:56 PM, Timofey Titovets <nefelim4ag@gmail.com> wrote: >> 2017-09-18 10:36 GMT+03:00 shally verma <shallyvermacavium@gmail.com>: >>> Hi >>> >>> I wanted to test btrfs compression using fio command but somehow >>> during fio writes, I don't see code taking route of compression blocks >>> where as If I do a copy to btrfs compression enabled mount point then >>> I can easily see code falling through compression.c. >>> >>> Here's how I do my setup >>> >>> 1. mkfs.btrfs /dev/sdb1 >>> 2. mount -t btrfs -o compress=zlib,compress-force /dev/sdb1 /mnt >>> 3. cp <some large test file> /mnt >>> 4. dmesg shows print staments from compression.c and zlib.c confirming >>> compression routine was invoked during write >>> 5. now, copy back from btrfs mount point to home directory also shows >>> decompress call invokation >>> >>> Now, try same with fio commands: >>> >>> fio command >>> >>> fio --directory=/mnt/ --numjobs=1 --direct=0 --buffered=1 >>> --ioengine=libaio --group_reporting --bs=64k --rw=write --iodepth=128 >>> --name=test --size=10G --runtime=180 --time_based >>> >>> But it seems to write uncompressed data. >>> >>> Any help here? what's missing? >>> >>> Thanks >>> Shally >>> -- >>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in >>> the body of a message to majordomo@vger.kernel.org >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >> >> 1. mount -t btrfs -o compress=zlib,compress-force -> compress-force=zlib >> 2. Tune fio to generate compressible data >> > How do I "tune" fio to generate data. I had assumed once compression > is enabled on btrfs any system fwrite call will simply compress data > into it .Isn't it so? > Can you share fio command that I can test? > Thanks > Shally >> >> -- >> Have a nice day, >> Timofey. That useless to compress uncompressible data. Also, as you enable compress, not compress-force So after first uncompressible write btrfs just stop trying compress that file. >From man fio: buffer_compress_percentage=int If this is set, then fio will attempt to provide I/O buffer content (on WRITEs) that compresses to the specified level. Fio does this by providing a mix of random data and a fixed pattern. The fixed pattern is either zeros, or the pattern specified by buffer_pattern. If the pattern option is used, it might skew the compression ratio slightly. Note that this is per block size unit, for file/disk wide compression level that matches this setting, you'll also want to set refill_buffers. buffer_compress_chunk=int See buffer_compress_percentage. This setting allows fio to manage how big the ranges of random data and zeroed data is. Without this set, fio will provide buffer_compress_percentage of blocksize random data, followed by the remaining zeroed. With this set to some chunk size smaller than the block size, fio can alternate random and zeroed data throughout the I/O buffer. Good luck :) -- Have a nice day, Timofey. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: using fio to test btrfs compression 2017-09-18 13:41 ` Timofey Titovets @ 2017-09-20 8:36 ` shally verma 2017-09-20 8:44 ` shally verma 0 siblings, 1 reply; 16+ messages in thread From: shally verma @ 2017-09-20 8:36 UTC (permalink / raw) To: Timofey Titovets; +Cc: linux-btrfs, Verma, Shally On Mon, Sep 18, 2017 at 7:11 PM, Timofey Titovets <nefelim4ag@gmail.com> wrote: > 2017-09-18 16:28 GMT+03:00 shally verma <shallyvermacavium@gmail.com>: >> On Mon, Sep 18, 2017 at 1:56 PM, Timofey Titovets <nefelim4ag@gmail.com> wrote: >>> 2017-09-18 10:36 GMT+03:00 shally verma <shallyvermacavium@gmail.com>: >>>> Hi >>>> >>>> I wanted to test btrfs compression using fio command but somehow >>>> during fio writes, I don't see code taking route of compression blocks >>>> where as If I do a copy to btrfs compression enabled mount point then >>>> I can easily see code falling through compression.c. >>>> >>>> Here's how I do my setup >>>> >>>> 1. mkfs.btrfs /dev/sdb1 >>>> 2. mount -t btrfs -o compress=zlib,compress-force /dev/sdb1 /mnt >>>> 3. cp <some large test file> /mnt >>>> 4. dmesg shows print staments from compression.c and zlib.c confirming >>>> compression routine was invoked during write >>>> 5. now, copy back from btrfs mount point to home directory also shows >>>> decompress call invokation >>>> >>>> Now, try same with fio commands: >>>> >>>> fio command >>>> >>>> fio --directory=/mnt/ --numjobs=1 --direct=0 --buffered=1 >>>> --ioengine=libaio --group_reporting --bs=64k --rw=write --iodepth=128 >>>> --name=test --size=10G --runtime=180 --time_based >>>> >>>> But it seems to write uncompressed data. >>>> >>>> Any help here? what's missing? >>>> >>>> Thanks >>>> Shally >>>> -- >>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in >>>> the body of a message to majordomo@vger.kernel.org >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>> >>> 1. mount -t btrfs -o compress=zlib,compress-force -> compress-force=zlib >>> 2. Tune fio to generate compressible data >>> >> How do I "tune" fio to generate data. I had assumed once compression >> is enabled on btrfs any system fwrite call will simply compress data >> into it .Isn't it so? >> Can you share fio command that I can test? >> Thanks >> Shally >>> >>> -- >>> Have a nice day, >>> Timofey. > > That useless to compress uncompressible data. > Also, as you enable compress, not compress-force > So after first uncompressible write btrfs just stop trying compress that file. > > From man fio: > buffer_compress_percentage=int > If this is set, then fio will attempt to provide I/O > buffer content (on WRITEs) that compresses to the specified level. Fio > does this by providing a mix of random data and a fixed pattern. The > fixed pattern is either > zeros, or the pattern specified by buffer_pattern. If > the pattern option is used, it might skew the compression ratio > slightly. Note that this is per block size unit, for file/disk wide > compression level that matches this > setting, you'll also want to set refill_buffers. > > buffer_compress_chunk=int > See buffer_compress_percentage. This setting allows fio > to manage how big the ranges of random data and zeroed data is. > Without this set, fio will provide buffer_compress_percentage of > blocksize random data, followed by > the remaining zeroed. With this set to some chunk size > smaller than the block size, fio can alternate random and zeroed data > throughout the I/O buffer. > > Good luck :) Now. I did following: 1. mount -t btrfs -o compress-force=zlib /dev/sdb1 mnt 2. fio --directory=mnt/ --numjobs=1 --direct=0 --buffered=1 --bs=64k --rw=write --iodepth=128 --name=test --size=1G --buffer_compress_percentage=100 --buffer_pattern=0xFF --refill_buffer --ioengine=libaio 1GN file written uncompressed. Here no compression invoked (though compress-force=zlib) 3. cp mnt/test ./ --> copy back fio generated test file from btrfs mount point to local drive 4. hex dump test file (all FFs) -- confirmed that data is compressible no random data. 5. cp test mnt/ --> now, copy same test again back to mount point (reverse of step 3) . Now, here I see during copying compression is invoked. I am using kernel 4.9 and compress-foce is said to be working for kernel > 2.13 from wiki ... so I wonder what's so special with cp command which is not happening during fio writes??? Thanks Shally > -- > Have a nice day, > Timofey. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: using fio to test btrfs compression 2017-09-20 8:36 ` shally verma @ 2017-09-20 8:44 ` shally verma 2017-09-20 8:55 ` Timofey Titovets 0 siblings, 1 reply; 16+ messages in thread From: shally verma @ 2017-09-20 8:44 UTC (permalink / raw) To: Timofey Titovets; +Cc: linux-btrfs, Verma, Shally On Wed, Sep 20, 2017 at 2:06 PM, shally verma <shallyvermacavium@gmail.com> wrote: > On Mon, Sep 18, 2017 at 7:11 PM, Timofey Titovets <nefelim4ag@gmail.com> wrote: >> 2017-09-18 16:28 GMT+03:00 shally verma <shallyvermacavium@gmail.com>: >>> On Mon, Sep 18, 2017 at 1:56 PM, Timofey Titovets <nefelim4ag@gmail.com> wrote: >>>> 2017-09-18 10:36 GMT+03:00 shally verma <shallyvermacavium@gmail.com>: >>>>> Hi >>>>> >>>>> I wanted to test btrfs compression using fio command but somehow >>>>> during fio writes, I don't see code taking route of compression blocks >>>>> where as If I do a copy to btrfs compression enabled mount point then >>>>> I can easily see code falling through compression.c. >>>>> >>>>> Here's how I do my setup >>>>> >>>>> 1. mkfs.btrfs /dev/sdb1 >>>>> 2. mount -t btrfs -o compress=zlib,compress-force /dev/sdb1 /mnt >>>>> 3. cp <some large test file> /mnt >>>>> 4. dmesg shows print staments from compression.c and zlib.c confirming >>>>> compression routine was invoked during write >>>>> 5. now, copy back from btrfs mount point to home directory also shows >>>>> decompress call invokation >>>>> >>>>> Now, try same with fio commands: >>>>> >>>>> fio command >>>>> >>>>> fio --directory=/mnt/ --numjobs=1 --direct=0 --buffered=1 >>>>> --ioengine=libaio --group_reporting --bs=64k --rw=write --iodepth=128 >>>>> --name=test --size=10G --runtime=180 --time_based >>>>> >>>>> But it seems to write uncompressed data. >>>>> >>>>> Any help here? what's missing? >>>>> >>>>> Thanks >>>>> Shally >>>>> -- >>>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in >>>>> the body of a message to majordomo@vger.kernel.org >>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>> >>>> 1. mount -t btrfs -o compress=zlib,compress-force -> compress-force=zlib >>>> 2. Tune fio to generate compressible data >>>> >>> How do I "tune" fio to generate data. I had assumed once compression >>> is enabled on btrfs any system fwrite call will simply compress data >>> into it .Isn't it so? >>> Can you share fio command that I can test? >>> Thanks >>> Shally >>>> >>>> -- >>>> Have a nice day, >>>> Timofey. >> >> That useless to compress uncompressible data. >> Also, as you enable compress, not compress-force >> So after first uncompressible write btrfs just stop trying compress that file. >> >> From man fio: >> buffer_compress_percentage=int >> If this is set, then fio will attempt to provide I/O >> buffer content (on WRITEs) that compresses to the specified level. Fio >> does this by providing a mix of random data and a fixed pattern. The >> fixed pattern is either >> zeros, or the pattern specified by buffer_pattern. If >> the pattern option is used, it might skew the compression ratio >> slightly. Note that this is per block size unit, for file/disk wide >> compression level that matches this >> setting, you'll also want to set refill_buffers. >> >> buffer_compress_chunk=int >> See buffer_compress_percentage. This setting allows fio >> to manage how big the ranges of random data and zeroed data is. >> Without this set, fio will provide buffer_compress_percentage of >> blocksize random data, followed by >> the remaining zeroed. With this set to some chunk size >> smaller than the block size, fio can alternate random and zeroed data >> throughout the I/O buffer. >> >> Good luck :) > > Now. I did following: > > 1. mount -t btrfs -o compress-force=zlib /dev/sdb1 mnt > > 2. fio --directory=mnt/ --numjobs=1 --direct=0 --buffered=1 --bs=64k > --rw=write --iodepth=128 --name=test --size=1G > --buffer_compress_percentage=100 --buffer_pattern=0xFF --refill_buffer > --ioengine=libaio > > 1GN file written uncompressed. Here no compression invoked (though > compress-force=zlib) > > 3. cp mnt/test ./ --> copy back fio generated test file from btrfs > mount point to local drive > > 4. hex dump test file (all FFs) -- confirmed that data is compressible > no random data. > > 5. cp test mnt/ --> now, copy same test again back to mount point > (reverse of step 3) . Now, here I see during copying compression is > invoked. > > I am using kernel 4.9 and compress-foce is said to be working for > kernel > 2.13 from wiki ... so I wonder what's so special with cp > command which is not happening during fio writes??? > One more catch... I am initiating fio from non-btrfs filesystem i.e. pwd is ext4 based fs where as mount point is btrfs. Could that make difference? > Thanks > Shally > > >> -- >> Have a nice day, >> Timofey. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: using fio to test btrfs compression 2017-09-20 8:44 ` shally verma @ 2017-09-20 8:55 ` Timofey Titovets 2017-09-20 8:59 ` shally verma 0 siblings, 1 reply; 16+ messages in thread From: Timofey Titovets @ 2017-09-20 8:55 UTC (permalink / raw) To: shally verma; +Cc: linux-btrfs, Verma, Shally 2017-09-20 11:44 GMT+03:00 shally verma <shallyvermacavium@gmail.com>: > One more catch... I am initiating fio from non-btrfs filesystem i.e. > pwd is ext4 based fs where as mount point is btrfs. > Could that make difference? > >> Thanks >> Shally Only matter are where you store test file =) If you store test file on btrfs, pwd does nothing. -- Have a nice day, Timofey. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: using fio to test btrfs compression 2017-09-20 8:55 ` Timofey Titovets @ 2017-09-20 8:59 ` shally verma 2017-09-20 9:36 ` Timofey Titovets 0 siblings, 1 reply; 16+ messages in thread From: shally verma @ 2017-09-20 8:59 UTC (permalink / raw) To: Timofey Titovets; +Cc: linux-btrfs, Verma, Shally On Wed, Sep 20, 2017 at 2:25 PM, Timofey Titovets <nefelim4ag@gmail.com> wrote: > 2017-09-20 11:44 GMT+03:00 shally verma <shallyvermacavium@gmail.com>: >> One more catch... I am initiating fio from non-btrfs filesystem i.e. >> pwd is ext4 based fs where as mount point is btrfs. >> Could that make difference? >> >>> Thanks >>> Shally > > Only matter are where you store test file =) > If you store test file on btrfs, pwd does nothing. > Then steps listed in previous mail should work right? Am listing them again here: " ---- > > 1. mount -t btrfs -o compress-force=zlib /dev/sdb1 mnt > > 2. fio --directory=mnt/ --numjobs=1 --direct=0 --buffered=1 --bs=64k > --rw=write --iodepth=128 --name=test --size=1G > --buffer_compress_percentage=100 --buffer_pattern=0xFF --refill_buffer > --ioengine=libaio > > 1GN file written uncompressed. Here no compression invoked (though > compress-force=zlib) > > 3. cp mnt/test ./ --> copy back fio generated test file from btrfs > mount point to local drive > > 4. hex dump test file (all FFs) -- confirmed that data is compressible > no random data. > > 5. cp test mnt/ --> now, copy same test again back to mount point > (reverse of step 3) . Now, here I see during copying compression is > invoked. > > I am using kernel 4.9 and compress-foce is said to be working for > kernel > 2.13 from wiki ... so I wonder what's so special with cp > command which is not happening during fio writes??? "----- Thanks Shally > -- > Have a nice day, > Timofey. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: using fio to test btrfs compression 2017-09-20 8:59 ` shally verma @ 2017-09-20 9:36 ` Timofey Titovets 2017-09-20 10:13 ` shally verma 0 siblings, 1 reply; 16+ messages in thread From: Timofey Titovets @ 2017-09-20 9:36 UTC (permalink / raw) To: shally verma; +Cc: linux-btrfs, Verma, Shally 2017-09-20 11:59 GMT+03:00 shally verma <shallyvermacavium@gmail.com>: > On Wed, Sep 20, 2017 at 2:25 PM, Timofey Titovets <nefelim4ag@gmail.com> wrote: >> 2017-09-20 11:44 GMT+03:00 shally verma <shallyvermacavium@gmail.com>: >>> One more catch... I am initiating fio from non-btrfs filesystem i.e. >>> pwd is ext4 based fs where as mount point is btrfs. >>> Could that make difference? >>> >>>> Thanks >>>> Shally >> >> Only matter are where you store test file =) >> If you store test file on btrfs, pwd does nothing. >> > > Then steps listed in previous mail should work right? Am listing them > again here: > > " ---- >> >> 1. mount -t btrfs -o compress-force=zlib /dev/sdb1 mnt >> >> 2. fio --directory=mnt/ --numjobs=1 --direct=0 --buffered=1 --bs=64k >> --rw=write --iodepth=128 --name=test --size=1G >> --buffer_compress_percentage=100 --buffer_pattern=0xFF --refill_buffer >> --ioengine=libaio >> >> 1GN file written uncompressed. Here no compression invoked (though >> compress-force=zlib) >> >> 3. cp mnt/test ./ --> copy back fio generated test file from btrfs >> mount point to local drive >> >> 4. hex dump test file (all FFs) -- confirmed that data is compressible >> no random data. >> >> 5. cp test mnt/ --> now, copy same test again back to mount point >> (reverse of step 3) . Now, here I see during copying compression is >> invoked. >> >> I am using kernel 4.9 and compress-foce is said to be working for >> kernel > 2.13 from wiki ... so I wonder what's so special with cp >> command which is not happening during fio writes??? > > "----- > > Thanks > Shally > > >> -- >> Have a nice day, >> Timofey. I did try reproduce your problem on 4.14, i'm hard to explain that happens and why file created by FIO will not use compression. Suggest workaround: rm -v /mnt/test.0.0 dd if=/dev/zero of=/mnt/test.0.0 bs=1M count=1024 fio --directory=$HOME/test/ --numjobs=1 --direct=1 --buffered=0 --bs=64k --rw=write --iodepth=128 --name=test --size=1G --buffer_compress_percentage=100 --buffer_pattern=0xFF --refill_buffer --ioengine=libaio You can check if data compressed by filefrag -v /mnt/test.0.0 (you will see encoded extents) -- Have a nice day, Timofey. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: using fio to test btrfs compression 2017-09-20 9:36 ` Timofey Titovets @ 2017-09-20 10:13 ` shally verma 2017-09-20 10:30 ` Timofey Titovets 0 siblings, 1 reply; 16+ messages in thread From: shally verma @ 2017-09-20 10:13 UTC (permalink / raw) To: Timofey Titovets; +Cc: linux-btrfs, Verma, Shally On Wed, Sep 20, 2017 at 3:06 PM, Timofey Titovets <nefelim4ag@gmail.com> wrote: > 2017-09-20 11:59 GMT+03:00 shally verma <shallyvermacavium@gmail.com>: >> On Wed, Sep 20, 2017 at 2:25 PM, Timofey Titovets <nefelim4ag@gmail.com> wrote: >>> 2017-09-20 11:44 GMT+03:00 shally verma <shallyvermacavium@gmail.com>: >>>> One more catch... I am initiating fio from non-btrfs filesystem i.e. >>>> pwd is ext4 based fs where as mount point is btrfs. >>>> Could that make difference? >>>> >>>>> Thanks >>>>> Shally >>> >>> Only matter are where you store test file =) >>> If you store test file on btrfs, pwd does nothing. >>> >> >> Then steps listed in previous mail should work right? Am listing them >> again here: >> >> " ---- >>> >>> 1. mount -t btrfs -o compress-force=zlib /dev/sdb1 mnt >>> >>> 2. fio --directory=mnt/ --numjobs=1 --direct=0 --buffered=1 --bs=64k >>> --rw=write --iodepth=128 --name=test --size=1G >>> --buffer_compress_percentage=100 --buffer_pattern=0xFF --refill_buffer >>> --ioengine=libaio >>> >>> 1GN file written uncompressed. Here no compression invoked (though >>> compress-force=zlib) >>> >>> 3. cp mnt/test ./ --> copy back fio generated test file from btrfs >>> mount point to local drive >>> >>> 4. hex dump test file (all FFs) -- confirmed that data is compressible >>> no random data. >>> >>> 5. cp test mnt/ --> now, copy same test again back to mount point >>> (reverse of step 3) . Now, here I see during copying compression is >>> invoked. >>> >>> I am using kernel 4.9 and compress-foce is said to be working for >>> kernel > 2.13 from wiki ... so I wonder what's so special with cp >>> command which is not happening during fio writes??? >> >> "----- >> >> Thanks >> Shally >> >> >>> -- >>> Have a nice day, >>> Timofey. > > I did try reproduce your problem on 4.14, i'm hard to explain that > happens and why file created by FIO will not use compression. > > Suggest workaround: > rm -v /mnt/test.0.0 > dd if=/dev/zero of=/mnt/test.0.0 bs=1M count=1024 You mean "compression" using "dd" write. > fio --directory=$HOME/test/ --numjobs=1 --direct=1 --buffered=0 > --bs=64k --rw=write --iodepth=128 --name=test --size=1G > --buffer_compress_percentage=100 --buffer_pattern=0xFF --refill_buffer > --ioengine=libaio You mean "no compression" during fio writes and you see similar behavior as I? BTW here --direct=0 and --buffered = 1. > > You can check if data compressed by filefrag -v /mnt/test.0.0 (you > will see encoded extents) > if I do filefrag on mnt/test.0.0, then it shows up like this: ext: logical_offset: physical_offset: length: expected: flags: 0: 0.. 1: 1329922.. 1329923: 2: 1: 2.. 3: 1329923.. 1329924: 2: 1329924: 2: 4.. 5: 1329924.. 1329925: 2: 1329925: An new to this, how does it says extents are encoded here? Thanks Shally > -- > Have a nice day, > Timofey. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: using fio to test btrfs compression 2017-09-20 10:13 ` shally verma @ 2017-09-20 10:30 ` Timofey Titovets 2017-09-20 11:10 ` shally verma 0 siblings, 1 reply; 16+ messages in thread From: Timofey Titovets @ 2017-09-20 10:30 UTC (permalink / raw) To: shally verma; +Cc: linux-btrfs, Verma, Shally 2017-09-20 13:13 GMT+03:00 shally verma <shallyvermacavium@gmail.com>: > On Wed, Sep 20, 2017 at 3:06 PM, Timofey Titovets <nefelim4ag@gmail.com> wrote: >> 2017-09-20 11:59 GMT+03:00 shally verma <shallyvermacavium@gmail.com>: >>> On Wed, Sep 20, 2017 at 2:25 PM, Timofey Titovets <nefelim4ag@gmail.com> wrote: >>>> 2017-09-20 11:44 GMT+03:00 shally verma <shallyvermacavium@gmail.com>: >>>>> One more catch... I am initiating fio from non-btrfs filesystem i.e. >>>>> pwd is ext4 based fs where as mount point is btrfs. >>>>> Could that make difference? >>>>> >>>>>> Thanks >>>>>> Shally >>>> >>>> Only matter are where you store test file =) >>>> If you store test file on btrfs, pwd does nothing. >>>> >>> >>> Then steps listed in previous mail should work right? Am listing them >>> again here: >>> >>> " ---- >>>> >>>> 1. mount -t btrfs -o compress-force=zlib /dev/sdb1 mnt >>>> >>>> 2. fio --directory=mnt/ --numjobs=1 --direct=0 --buffered=1 --bs=64k >>>> --rw=write --iodepth=128 --name=test --size=1G >>>> --buffer_compress_percentage=100 --buffer_pattern=0xFF --refill_buffer >>>> --ioengine=libaio >>>> >>>> 1GN file written uncompressed. Here no compression invoked (though >>>> compress-force=zlib) >>>> >>>> 3. cp mnt/test ./ --> copy back fio generated test file from btrfs >>>> mount point to local drive >>>> >>>> 4. hex dump test file (all FFs) -- confirmed that data is compressible >>>> no random data. >>>> >>>> 5. cp test mnt/ --> now, copy same test again back to mount point >>>> (reverse of step 3) . Now, here I see during copying compression is >>>> invoked. >>>> >>>> I am using kernel 4.9 and compress-foce is said to be working for >>>> kernel > 2.13 from wiki ... so I wonder what's so special with cp >>>> command which is not happening during fio writes??? >>> >>> "----- >>> >>> Thanks >>> Shally >>> >>> >>>> -- >>>> Have a nice day, >>>> Timofey. >> >> I did try reproduce your problem on 4.14, i'm hard to explain that >> happens and why file created by FIO will not use compression. >> >> Suggest workaround: >> rm -v /mnt/test.0.0 >> dd if=/dev/zero of=/mnt/test.0.0 bs=1M count=1024 > > You mean "compression" using "dd" write. > >> fio --directory=$HOME/test/ --numjobs=1 --direct=1 --buffered=0 >> --bs=64k --rw=write --iodepth=128 --name=test --size=1G >> --buffer_compress_percentage=100 --buffer_pattern=0xFF --refill_buffer >> --ioengine=libaio > > You mean "no compression" during fio writes and you see similar behavior as I? > BTW here --direct=0 and --buffered = 1. > >> >> You can check if data compressed by filefrag -v /mnt/test.0.0 (you >> will see encoded extents) >> > if I do filefrag on mnt/test.0.0, then it shows up like this: > ext: logical_offset: physical_offset: length: expected: flags: > 0: 0.. 1: 1329922.. 1329923: 2: > 1: 2.. 3: 1329923.. 1329924: 2: 1329924: > 2: 4.. 5: 1329924.. 1329925: 2: 1329925: > > An new to this, how does it says extents are encoded here? > > Thanks > Shally > >> -- >> Have a nice day, >> Timofey. Yes, i was reproduce your problem, new file created by fio not compressed. So i write you about workaround, because i don't want to deep dig in that problem. Example of encoded: ~ dd if=/dev/zero of=./test.0.0 bs=1M count=1M count=1; sync ~ filefrag -v test.0.0 Filesystem type is: 9123683e File size of test.0.0 is 1048576 (256 blocks of 4096 bytes) ext: logical_offset: physical_offset: length: expected: flags: 0: 0.. 31: 72779979.. 72780010: 32: encoded 1: 32.. 63: 72751509.. 72751540: 32: 72780011: encoded 2: 64.. 95: 72755246.. 72755277: 32: 72751541: encoded 3: 96.. 127: 72755610.. 72755641: 32: 72755278: encoded 4: 128.. 159: 72755631.. 72755662: 32: 72755642: encoded 5: 160.. 191: 72755722.. 72755753: 32: 72755663: encoded 6: 192.. 223: 72757862.. 72757893: 32: 72755754: encoded 7: 224.. 255: 72769455.. 72769486: 32: 72757894: last,encoded,eof test.0.0: 8 extents found -- Have a nice day, Timofey. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: using fio to test btrfs compression 2017-09-20 10:30 ` Timofey Titovets @ 2017-09-20 11:10 ` shally verma 2017-09-21 11:42 ` Duncan [not found] ` <CAGqmi74eviuSf=LL7sGEJnH1YYqu_pHdaGDvq4Qdr4aSpvqxkQ@mail.gmail.com> 0 siblings, 2 replies; 16+ messages in thread From: shally verma @ 2017-09-20 11:10 UTC (permalink / raw) To: Timofey Titovets; +Cc: linux-btrfs, Verma, Shally On Wed, Sep 20, 2017 at 4:00 PM, Timofey Titovets <nefelim4ag@gmail.com> wrote: > 2017-09-20 13:13 GMT+03:00 shally verma <shallyvermacavium@gmail.com>: >> On Wed, Sep 20, 2017 at 3:06 PM, Timofey Titovets <nefelim4ag@gmail.com> wrote: >>> 2017-09-20 11:59 GMT+03:00 shally verma <shallyvermacavium@gmail.com>: >>>> On Wed, Sep 20, 2017 at 2:25 PM, Timofey Titovets <nefelim4ag@gmail.com> wrote: >>>>> 2017-09-20 11:44 GMT+03:00 shally verma <shallyvermacavium@gmail.com>: >>>>>> One more catch... I am initiating fio from non-btrfs filesystem i.e. >>>>>> pwd is ext4 based fs where as mount point is btrfs. >>>>>> Could that make difference? >>>>>> >>>>>>> Thanks >>>>>>> Shally >>>>> >>>>> Only matter are where you store test file =) >>>>> If you store test file on btrfs, pwd does nothing. >>>>> >>>> >>>> Then steps listed in previous mail should work right? Am listing them >>>> again here: >>>> >>>> " ---- >>>>> >>>>> 1. mount -t btrfs -o compress-force=zlib /dev/sdb1 mnt >>>>> >>>>> 2. fio --directory=mnt/ --numjobs=1 --direct=0 --buffered=1 --bs=64k >>>>> --rw=write --iodepth=128 --name=test --size=1G >>>>> --buffer_compress_percentage=100 --buffer_pattern=0xFF --refill_buffer >>>>> --ioengine=libaio >>>>> >>>>> 1GN file written uncompressed. Here no compression invoked (though >>>>> compress-force=zlib) >>>>> >>>>> 3. cp mnt/test ./ --> copy back fio generated test file from btrfs >>>>> mount point to local drive >>>>> >>>>> 4. hex dump test file (all FFs) -- confirmed that data is compressible >>>>> no random data. >>>>> >>>>> 5. cp test mnt/ --> now, copy same test again back to mount point >>>>> (reverse of step 3) . Now, here I see during copying compression is >>>>> invoked. >>>>> >>>>> I am using kernel 4.9 and compress-foce is said to be working for >>>>> kernel > 2.13 from wiki ... so I wonder what's so special with cp >>>>> command which is not happening during fio writes??? >>>> >>>> "----- >>>> >>>> Thanks >>>> Shally >>>> >>>> >>>>> -- >>>>> Have a nice day, >>>>> Timofey. >>> >>> I did try reproduce your problem on 4.14, i'm hard to explain that >>> happens and why file created by FIO will not use compression. >>> >>> Suggest workaround: >>> rm -v /mnt/test.0.0 >>> dd if=/dev/zero of=/mnt/test.0.0 bs=1M count=1024 >> >> You mean "compression" using "dd" write. >> >>> fio --directory=$HOME/test/ --numjobs=1 --direct=1 --buffered=0 >>> --bs=64k --rw=write --iodepth=128 --name=test --size=1G >>> --buffer_compress_percentage=100 --buffer_pattern=0xFF --refill_buffer >>> --ioengine=libaio >> >> You mean "no compression" during fio writes and you see similar behavior as I? >> BTW here --direct=0 and --buffered = 1. >> >>> >>> You can check if data compressed by filefrag -v /mnt/test.0.0 (you >>> will see encoded extents) >>> >> if I do filefrag on mnt/test.0.0, then it shows up like this: >> ext: logical_offset: physical_offset: length: expected: flags: >> 0: 0.. 1: 1329922.. 1329923: 2: >> 1: 2.. 3: 1329923.. 1329924: 2: 1329924: >> 2: 4.. 5: 1329924.. 1329925: 2: 1329925: >> >> An new to this, how does it says extents are encoded here? >> >> Thanks >> Shally >> >>> -- >>> Have a nice day, >>> Timofey. > > Yes, i was reproduce your problem, new file created by fio not compressed. > So i write you about workaround, because i don't want to deep dig in > that problem. > > Example of encoded: > ~ dd if=/dev/zero of=./test.0.0 bs=1M count=1M count=1; sync > ~ filefrag -v test.0.0 > Filesystem type is: 9123683e > File size of test.0.0 is 1048576 (256 blocks of 4096 bytes) > ext: logical_offset: physical_offset: length: expected: flags: > 0: 0.. 31: 72779979.. 72780010: 32: encoded > 1: 32.. 63: 72751509.. 72751540: 32: 72780011: encoded > 2: 64.. 95: 72755246.. 72755277: 32: 72751541: encoded > 3: 96.. 127: 72755610.. 72755641: 32: 72755278: encoded > 4: 128.. 159: 72755631.. 72755662: 32: 72755642: encoded > 5: 160.. 191: 72755722.. 72755753: 32: 72755663: encoded > 6: 192.. 223: 72757862.. 72757893: 32: 72755754: encoded > 7: 224.. 255: 72769455.. 72769486: 32: 72757894: > last,encoded,eof > test.0.0: 8 extents found > Interesting part is I dont see "encoded" under flags. I couldn't understand if flags are retrieved from btrfs file metadata info. As you are running on 4.14 and I am on 4.9 So, am still under doubt, even with dd if files are getting compressed. What is the filesize shown if you run btrfs fi du /mnt/test0.0 .. is it less or actual size? Is there any command that i can run to confirm file has been compressed? So far, I had my prints enabled in kernel/fs/btrfs/compression.c and check in dmesg that code jumped to compress_page() func. Thanks Shally > -- > Have a nice day, > Timofey. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: using fio to test btrfs compression 2017-09-20 11:10 ` shally verma @ 2017-09-21 11:42 ` Duncan [not found] ` <CAGqmi74eviuSf=LL7sGEJnH1YYqu_pHdaGDvq4Qdr4aSpvqxkQ@mail.gmail.com> 1 sibling, 0 replies; 16+ messages in thread From: Duncan @ 2017-09-21 11:42 UTC (permalink / raw) To: linux-btrfs shally verma posted on Wed, 20 Sep 2017 16:40:15 +0530 as excerpted: > Is there any command that i can run to confirm file has been compressed? There is the quite recently posted (and actively updated since then) compsize command. https://github.com/kilobyte/compsize -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman ^ permalink raw reply [flat|nested] 16+ messages in thread
[parent not found: <CAGqmi74eviuSf=LL7sGEJnH1YYqu_pHdaGDvq4Qdr4aSpvqxkQ@mail.gmail.com>]
* Re: using fio to test btrfs compression [not found] ` <CAGqmi74eviuSf=LL7sGEJnH1YYqu_pHdaGDvq4Qdr4aSpvqxkQ@mail.gmail.com> @ 2017-09-22 5:33 ` shally verma 2017-09-22 17:05 ` Timofey Titovets 0 siblings, 1 reply; 16+ messages in thread From: shally verma @ 2017-09-22 5:33 UTC (permalink / raw) To: Timofey Titovets, linux-btrfs; +Cc: vikas.aggarwal, Narayana, Prasad Athreya On Wed, Sep 20, 2017 at 5:26 PM, Timofey Titovets <nefelim4ag@gmail.com> wrote: > 2017-09-20 14:10 GMT+03:00 shally verma <shallyvermacavium@gmail.com>: >> Interesting part is I dont see "encoded" under flags. I couldn't >> understand if flags are retrieved from btrfs file metadata info. As >> you are running on 4.14 and I am on 4.9 >> >> So, am still under doubt, even with dd if files are getting compressed. >> >> What is the filesize shown if you run >> btrfs fi du /mnt/test0.0 .. is it less or actual size? >> >> Is there any command that i can run to confirm file has been compressed? >> >> So far, I had my prints enabled in kernel/fs/btrfs/compression.c and >> check in dmesg that code jumped to compress_page() func. >> >> Thanks >> Shally >> > > Okay, lets play different. > encoded work for last several years for kernel releases, so you must see that. > > Reproduction script: > #!/bin/bash -e > > FILE_NAME=$RANDOM$RANDOM > TMP_DIR=$(mktemp -d) > IMAGE_FILE="$HOME/$FILE_NAME" > > truncate -s 4G $IMAGE_FILE > mkfs.btrfs -m single -L COMPRESS_TEST $IMAGE_FILE > mount -o compress-force $IMAGE_FILE $TMP_DIR > dd if=/dev/zero bs=128K count=2 of=$TMP_DIR/zero > sync > filefrag -v $TMP_DIR/zero > umount $TMP_DIR > rm -v $IMAGE_FILE > > Example output: > ~ sudo ./btrfs_compress_test.sh > btrfs-progs v4.13 > See http://btrfs.wiki.kernel.org for more information. > > Label: COMPRESS_TEST > UUID: abfedc39-dd94-4105-87d6-49eedb13467f > Node size: 16384 > Sector size: 4096 > Filesystem size: 4.00GiB > Block group profiles: > Data: single 8.00MiB > Metadata: single 8.00MiB > System: single 4.00MiB > SSD detected: no > Incompat features: extref, skinny-metadata > Number of devices: 1 > Devices: > ID SIZE PATH > 1 4.00GiB /root/322906281 > > 2+0 records in > 2+0 records out > 262144 bytes (262 kB, 256 KiB) copied, 0.000197746 s, 1.3 GB/s > Filesystem type is: 9123683e > File size of /tmp/tmp.bDyt3EkEG5/zero is 262144 (64 blocks of 4096 bytes) > ext: logical_offset: physical_offset: length: expected: flags: > 0: 0.. 31: 3072.. 3103: 32: encoded > 1: 32.. 63: 3073.. 3104: 32: 3104: > last,encoded,eof > /tmp/tmp.bDyt3EkEG5/zero: 2 extents found > removed '/root/322906281' > > Good luck. Here's my output - Everything is same except: 1. nodesize and sector size = 64K 2. extent length = 2 3. I don't see "encoded" in filefrag here. btrfs-progs v4.13 See http://btrfs.wiki.kernel.org for more information. Label: COMPRESS_TEST UUID: fad6907e-d4eb-4dbb-9014-3918a822c9ce Node size: 65536 Sector size: 65536 Filesystem size: 4.00GiB Block group profiles: Data: single 8.00MiB Metadata: single 8.00MiB System: single 4.00MiB SSD detected: no Incompat features: extref, skinny-metadata Number of devices: 1 Devices: ID SIZE PATH 1 4.00GiB /root/2808626087 2+0 records in 2+0 records out 262144 bytes (262 kB) copied, 0.00028777 s, 911 MB/s Filesystem type is: 9123683e File size of /tmp/tmp.346ESCdOIi/zero is 262144 (4 blocks of 65536 bytes) ext: logical_offset: physical_offset: length: expected: flags: 0: 0.. 1: 192.. 193: 2: 1: 2.. 3: 193.. 194: 2: 194: eof /tmp/tmp.346ESCdOIi/zero: 2 extents found removed '/root/2808626087' And this is my dmesg [170127.417119] BTRFS: device label COMPRESS_TEST devid 1 transid 5 /dev/loop0 [170127.417493] BTRFS info (device loop0): force zlib compression [170127.417496] BTRFS info (device loop0): disk space caching is enabled [170127.417499] BTRFS info (device loop0): has skinny extents [170127.425858] BTRFS info (device loop0): creating UUID tree This is fio --version fio-3.0 What do we doubt here? Thanks Shally > -- > Have a nice day, > Timofey. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: using fio to test btrfs compression 2017-09-22 5:33 ` shally verma @ 2017-09-22 17:05 ` Timofey Titovets 0 siblings, 0 replies; 16+ messages in thread From: Timofey Titovets @ 2017-09-22 17:05 UTC (permalink / raw) To: shally verma; +Cc: linux-btrfs, vikas.aggarwal, Narayana, Prasad Athreya 2017-09-22 8:33 GMT+03:00 shally verma <shallyvermacavium@gmail.com>: > On Wed, Sep 20, 2017 at 5:26 PM, Timofey Titovets <nefelim4ag@gmail.com> wrote: >> 2017-09-20 14:10 GMT+03:00 shally verma <shallyvermacavium@gmail.com>: >>> Interesting part is I dont see "encoded" under flags. I couldn't >>> understand if flags are retrieved from btrfs file metadata info. As >>> you are running on 4.14 and I am on 4.9 >>> >>> So, am still under doubt, even with dd if files are getting compressed. >>> >>> What is the filesize shown if you run >>> btrfs fi du /mnt/test0.0 .. is it less or actual size? >>> >>> Is there any command that i can run to confirm file has been compressed? >>> >>> So far, I had my prints enabled in kernel/fs/btrfs/compression.c and >>> check in dmesg that code jumped to compress_page() func. >>> >>> Thanks >>> Shally >>> >> >> Okay, lets play different. >> encoded work for last several years for kernel releases, so you must see that. >> >> Reproduction script: >> #!/bin/bash -e >> >> FILE_NAME=$RANDOM$RANDOM >> TMP_DIR=$(mktemp -d) >> IMAGE_FILE="$HOME/$FILE_NAME" >> >> truncate -s 4G $IMAGE_FILE >> mkfs.btrfs -m single -L COMPRESS_TEST $IMAGE_FILE >> mount -o compress-force $IMAGE_FILE $TMP_DIR >> dd if=/dev/zero bs=128K count=2 of=$TMP_DIR/zero >> sync >> filefrag -v $TMP_DIR/zero >> umount $TMP_DIR >> rm -v $IMAGE_FILE >> >> Example output: >> ~ sudo ./btrfs_compress_test.sh >> btrfs-progs v4.13 >> See http://btrfs.wiki.kernel.org for more information. >> >> Label: COMPRESS_TEST >> UUID: abfedc39-dd94-4105-87d6-49eedb13467f >> Node size: 16384 >> Sector size: 4096 >> Filesystem size: 4.00GiB >> Block group profiles: >> Data: single 8.00MiB >> Metadata: single 8.00MiB >> System: single 4.00MiB >> SSD detected: no >> Incompat features: extref, skinny-metadata >> Number of devices: 1 >> Devices: >> ID SIZE PATH >> 1 4.00GiB /root/322906281 >> >> 2+0 records in >> 2+0 records out >> 262144 bytes (262 kB, 256 KiB) copied, 0.000197746 s, 1.3 GB/s >> Filesystem type is: 9123683e >> File size of /tmp/tmp.bDyt3EkEG5/zero is 262144 (64 blocks of 4096 bytes) >> ext: logical_offset: physical_offset: length: expected: flags: >> 0: 0.. 31: 3072.. 3103: 32: encoded >> 1: 32.. 63: 3073.. 3104: 32: 3104: >> last,encoded,eof >> /tmp/tmp.bDyt3EkEG5/zero: 2 extents found >> removed '/root/322906281' >> >> Good luck. > Here's my output - Everything is same except: > > 1. nodesize and sector size = 64K > 2. extent length = 2 > 3. I don't see "encoded" in filefrag here. > > btrfs-progs v4.13 > See http://btrfs.wiki.kernel.org for more information. > > Label: COMPRESS_TEST > UUID: fad6907e-d4eb-4dbb-9014-3918a822c9ce > Node size: 65536 > Sector size: 65536 > Filesystem size: 4.00GiB > Block group profiles: > Data: single 8.00MiB > Metadata: single 8.00MiB > System: single 4.00MiB > SSD detected: no > Incompat features: extref, skinny-metadata > Number of devices: 1 > Devices: > ID SIZE PATH > 1 4.00GiB /root/2808626087 > > 2+0 records in > 2+0 records out > 262144 bytes (262 kB) copied, 0.00028777 s, 911 MB/s > Filesystem type is: 9123683e > File size of /tmp/tmp.346ESCdOIi/zero is 262144 (4 blocks of 65536 bytes) > ext: logical_offset: physical_offset: length: expected: flags: > 0: 0.. 1: 192.. 193: 2: > 1: 2.. 3: 193.. 194: 2: 194: eof > /tmp/tmp.346ESCdOIi/zero: 2 extents found > removed '/root/2808626087' > > And this is my dmesg > > [170127.417119] BTRFS: device label COMPRESS_TEST devid 1 transid 5 /dev/loop0 > [170127.417493] BTRFS info (device loop0): force zlib compression > [170127.417496] BTRFS info (device loop0): disk space caching is enabled > [170127.417499] BTRFS info (device loop0): has skinny extents > [170127.425858] BTRFS info (device loop0): creating UUID tree > > This is fio --version > fio-3.0 > > What do we doubt here? > > Thanks > Shally > >> -- >> Have a nice day, >> Timofey. Sorry, no idea. -- Have a nice day, Timofey. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: using fio to test btrfs compression 2017-09-18 7:36 using fio to test btrfs compression shally verma 2017-09-18 8:26 ` Timofey Titovets @ 2017-09-21 15:26 ` Liu Bo 1 sibling, 0 replies; 16+ messages in thread From: Liu Bo @ 2017-09-21 15:26 UTC (permalink / raw) To: shally verma; +Cc: linux-btrfs On Mon, Sep 18, 2017 at 01:06:45PM +0530, shally verma wrote: > Hi > > I wanted to test btrfs compression using fio command but somehow > during fio writes, I don't see code taking route of compression blocks > where as If I do a copy to btrfs compression enabled mount point then > I can easily see code falling through compression.c. > > Here's how I do my setup > > 1. mkfs.btrfs /dev/sdb1 > 2. mount -t btrfs -o compress=zlib,compress-force /dev/sdb1 /mnt > 3. cp <some large test file> /mnt > 4. dmesg shows print staments from compression.c and zlib.c confirming > compression routine was invoked during write > 5. now, copy back from btrfs mount point to home directory also shows > decompress call invokation > > Now, try same with fio commands: > > fio command > > fio --directory=/mnt/ --numjobs=1 --direct=0 --buffered=1 > --ioengine=libaio --group_reporting --bs=64k --rw=write --iodepth=128 > --name=test --size=10G --runtime=180 --time_based fio by default uses fallocate (posix_falloc) to pre-allocate space for the later writes, and PREALLOC path overrides compression path. Like others mentioned, after fio and sync, you'll see 'encoded' in filefrag -v your_file. thanks, -liubo > > But it seems to write uncompressed data. > > Any help here? what's missing? > > Thanks > Shally > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2017-09-22 17:06 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2017-09-18 7:36 using fio to test btrfs compression shally verma
2017-09-18 8:26 ` Timofey Titovets
2017-09-18 13:28 ` shally verma
2017-09-18 13:41 ` Timofey Titovets
2017-09-20 8:36 ` shally verma
2017-09-20 8:44 ` shally verma
2017-09-20 8:55 ` Timofey Titovets
2017-09-20 8:59 ` shally verma
2017-09-20 9:36 ` Timofey Titovets
2017-09-20 10:13 ` shally verma
2017-09-20 10:30 ` Timofey Titovets
2017-09-20 11:10 ` shally verma
2017-09-21 11:42 ` Duncan
[not found] ` <CAGqmi74eviuSf=LL7sGEJnH1YYqu_pHdaGDvq4Qdr4aSpvqxkQ@mail.gmail.com>
2017-09-22 5:33 ` shally verma
2017-09-22 17:05 ` Timofey Titovets
2017-09-21 15:26 ` Liu Bo
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).