From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jagan Reddy Subject: Re: dm-thin vs lvm performance Date: Wed, 18 Jan 2012 11:30:54 -0800 (PST) Message-ID: <1326915054.99335.YahooMailNeo@web36407.mail.mud.yahoo.com> References: <1326416116.56981.YahooMailNeo@web36404.mail.mud.yahoo.com> <20120116124158.GA4426@ubuntu> Reply-To: Jagan Reddy , device-mapper development Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============3417799784578844595==" Return-path: In-Reply-To: <20120116124158.GA4426@ubuntu> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com To: Joe Thornber , device-mapper development List-Id: dm-devel.ids --===============3417799784578844595== Content-Type: multipart/alternative; boundary="-2114655128-221890458-1326915054=:99335" ---2114655128-221890458-1326915054=:99335 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Joe,=0A=A0Thanks for looking into the issue and running the tests and sugge= sting to use "direct" flag. I do see a difference with "direct" flag using = dd. However the difference is significant when using bs=3D64M compared to b= s=3D4k.=A0=0A=0Add-blocksize =A0flags =A0dm-thin =A0 =A0 LVMoutput =A0=A0= =0A64M =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0none =A0 179 MB/s =A0 =A0 =A0 =A0 1GB= /s =A0 =A0 =A0 =A0 =A0=A0=0A64M =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0direct =A0 2= .4GB/s =A0 =A0 =A0 =A0 =A0 3.6GB/s =A0 =A0 =A0 =A0=0A4k =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 none =A0 179 MB/s =A0 =A0 =A0 =A0 =A0965MB/s =A0 =A0 =A0= =0A4k =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 direct =A0 193MB/s =A0 =A0 =A0 = =A0 =A01.7GB/s =A0 =A0 =A0 =A0=0A=0AIn all the tests, dm-thin performance i= s below lvm.=A0I used 8M pages as we are not planning to use snapshots righ= t now. =A0=0A=0AWe are planning to use the dm-thin module in a high end fla= sh device that supports upto 300k IOPS/second. Hence =A0we are doing some p= erformance tests. I wanted to try more parallel async IO instead of dd. A g= ood program I found on web is=A0=0A=0A=0Afsbench.filesystems.org/bench/aio-= stress.c=0A=0A=0A(if you are aware of other tools to test parallel async io= using multiple threads, please let me know. I will give it a try.)=0A=0A= =0AI compiled it with libaio and libpth.=A0=0A=0AThen I ran the test on LVM= lun. Throughput is 9563.56MB/s.=A0Fully allocated TP LUN is=A05399.81MB/s= =0A=0AI used "perf top" to see where most of the time is spent. Lot of time= is spent in=A0_raw_spin_lock. I poked the code little bit and I see that= =A0dm_thin_find_block returns EWOULDBLOCK and hence the bio gets deferred. = It looks like spin locks are used when putting the bio on deferred list and= getting it back from the deferred list. I see that EWOULDBLOCK is returned= from=A0dm_thin_find_block ->=A0dm_btree_lookup->btree_lookup_raw->ro_step-= >bn_read_lock->dm_tm_read_lock->dm_bm_read_try_lock->bl_down_read_nonblock.= =0A=0AI tried to change the code little bit so that=A0dm_thin_find_block us= es array as a cache and looks up btree only if not found in the array.=0AWi= th that change, I see TP performance go way up.=0A=0AFully allocated TP LUN= with array lookup is 9387.17MB/s. With array, no bios are being deferred. = However when I used the array, I see kernel hang after several runs ( yet t= o debug the reason).=0A=0ACode changes are initialize array in pool_ctr() i= n dm-thin.c=0A=0Aextern dm_block_t blockmap[1000];=0A=0Aprintk("blockmap ar= ray inited");=0A=A0=A0 =A0 =A0 =A0for(i=3D0;i<1000;i++)=0A=A0=A0 =A0 =A0 = =A0{=0A=A0=A0 =A0 =A0 =A0 =A0blockmap[i]=3D0xffffffff;=0A=A0=A0 =A0 =A0 =A0= }=0A=0A=0Adm_thin_find_block() in dm-thin-metadata.c=0A=0Aat the begining:= =0A=A0=A0 =A0 =A0 if(blockmap[block]=3D=3D0xffffffff)=0A=A0=A0 =A0 =A0 =A0{= =0A=A0=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0printk("found unassigned block %llu\n"= ,block);=0A=A0=A0 =A0 =A0 =A0}=0A=A0=A0 =A0 =A0 =A0else=0A=A0=A0 =A0 =A0 = =A0{=0A=A0=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0result->block =3D blockmap[block];= =0A=A0=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0result->shared =3D 0;=0A=A0=A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0return 0;=0A=A0=A0 =A0 =A0 =A0}=0A=0Aat the end before = returning:=0A=A0=A0 =A0 =A0 if(r =3D=3D 0)=0A=A0=A0 =A0 =A0 =A0{=0A=A0=A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0blockmap[block] =3D result->block ;=0A=A0=A0 =A0= =A0 =A0}=0A=0A=0ANote that the array changes are only for my testing and n= ot generic enough at all. =A0Command I used for getting the throughput numb= ers are=0A=0Ai=3D0; while [ $i -lt 100 ] ; do ./aio-stress.exe =A0-O -o 1 -= c 16 -t 16 -d 256 /dev/mapper/thin1 2>&1 | grep throughput | cut -f3 -d ' '= | cut -f2 -d '(' | cut -f1 -d ' ' ; let i=3D$i+1; done =A0| awk 'BEGIN{i= =3D0.0} {i+=3D$0} END{print i/100}'=0A=0A=0AThanks,=0AJagan.=0A=0A=0A______= __________________________=0A From: Joe Thornber =0ATo= : Jagan Reddy ; device-mapper development =0ASent: Monday, January 16, 2012 4:42 AM=0ASubject: Re: [dm-deve= l] dm-thin vs lvm performance=0A =0AHi Jagan,=0A=0AOn Thu, Jan 12, 2012 at = 04:55:16PM -0800, Jagan Reddy wrote:=0A> Hi,=0A=0A> I =A0recently started u= sing dm-thin module and first of all, thank you=0A> for the good work. It w= orks great, well documented and the comments=0A> in the code are very usefu= l. =A0I tried to run some performance tests=0A> using a high performance fl= ash device and the performance of dm thin=0A> is about 30% of LVM performan= ce on a full allocated thin provisioned=0A> volume. ( ie after all pages/bl= ocks are allocated using=0A> dd). Performance test does only reads and no w= rites.=A0=0A=0AThanks very much for taking the time to try thinp out proper= ly.=0APeople testing different scenarios is very useful to us.=0A=0AYou're = probably aware that the thinp test suite is available here:=0A=0A=A0 =A0 ht= tps://github.com/jthornber/thinp-test-suite=0A=0AI've added a little set of= tests that recreate your scenario here:=0A=0A=A0 =A0 https://github.com/jt= hornber/thinp-test-suite/blob/master/ramdisk_tests.rb=0A=0AI used a 2G ramd= isk for these tests, and a variety of thinp block=0Asizes and 'dd' options.= =A0 I'll just summarise the main results, also I=0Ashould point out that my= testing was done on a VM hosted on a 4G=0Amachine, so the machine was unde= r a lot of memory pressure and there=0Awas a lot of variance in the benchma= rks.=0A=0Awrites across various volumes=0A-----------------------------=0A= =0Awrite1=0A------=0A=0ATesting write performance.=0A=0Add if=3D/dev/zero o= f=3D/dev/mapper/ oflags=3Ddirect bs=3D64M=0A=0Azeroing new blocks tur= ned on.=0A=0Athinp block size =3D 64k=0A=0A| Linear=A0 =A0 =A0 =A0 =A0 =A0 = =A0 | 2.2 G/s |=0A| Unprovisioned thin=A0 | 1.4 G/s |=0A| Provisioned thin= =A0 =A0 | 1.9=A0 =A0 |=0A| Snap totally shared | 1.5 G/s |=0A| Snap no sha= ring=A0 =A0 | 1.9=A0 =A0 |=0A=0APretty good.=A0 Not showing the drastic d= rop that you were seeing.=A0 The=0Asmall thinp block size means the snaps p= erform nicely (not many=0Acopy-on-writes).=0A=0Awrite2=0A------=0A=0AAs tes= t1, but with 8M thinp block size as in your tests.=0A=0A| Linear=A0 =A0 =A0= =A0 =A0 =A0 =A0 | 2.2 G/s |=0A| Unprovisioned thin=A0 | 1.5 G/s |=0A| Prov= isioned thin=A0 =A0 | 2.2=A0 =A0 |=0A| Snap totally shared | 882 Ms=A0 |= =0A| Snap no sharing=A0 =A0 | 2.2=A0 =A0 | =0A=0AGood results, breaking s= haring performance is down because the large=0Ablock size mean there will b= e more actual copying incurred.=0A=0Awrite3=0A------=0A=0AAs test2, but no = oflags=3Ddirect option to dd.=0A=0A| Linear=A0 =A0 =A0 =A0 =A0 =A0 =A0 | 90= 0 M/s |=0A| Unprovisioned thin=A0 | 579 M/s |=0A| Provisioned thin=A0 =A0 |= 694 M/s |=0A| Snap totally shared | 510 M/s |=0A| Snap no sharing=A0 =A0 = | 654 M/s | =0A=0AAlarming.=A0 Results are similar for thinp block size of = 64k.=0A=0Aread1=0A-----=0A=0ATesting read performance.=0A=0Add if=3D/dev/ma= pper/ of=3D/dev/null iflags=3Ddirect bs=3D64M=0A=0Athinp block size = =3D 64k=0A=0A| Linear=A0 =A0 =A0 =A0 =A0 =A0 =A0 | 3.3 G/s |=0A| Provisione= d thin=A0 =A0 | 2.7 G/s |=0A| Snap no sharing=A0 =A0 | 2.8 G/s | =0A=0A=0A= read2=0A-----=0A=0Aread1 but with 8M thinp block size=0A=0A| Linear=A0 =A0 = =A0 =A0 =A0 =A0 =A0 | 3.3 G/s |=0A| Provisioned thin=A0 =A0 | 3.2 G/s |=0A|= Snap no sharing=A0 =A0 | 3.3 G/s | =0A=0Aread3=0A-----=0A=0AAs read2, but= without the iflags=3Ddirect option to 'dd'.=0A=0A| Linear=A0 =A0 =A0 =A0 = =A0 =A0 =A0 | 1.0 G/s |=0A| Provisioned thin=A0 =A0 | 594 M/s |=0A| Snap no= sharing=A0 =A0 | 605 M/s | =0A=0A=0A=0AI think there are a couple of conc= lusions we can draw from this:=0A=0Ai) dd isn't great for benchmarking bloc= k devices=0Aii) if you are going to use it, then make sure you use O_DIRECT= =0A=0AUsing an instrumented kernel, I've confirmed that these read tests=0A= are taking the fast path through the code.=A0 The mapping tables are all=0A= in memory.=A0 The bio_map function is always returning DM_MAPIO_REMAPPED.= =0A=0AWill you be using a filesystem on these block devices?=0A=0A- Joe ---2114655128-221890458-1326915054=:99335 Content-Type: text/html; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable
Joe,
&= nbsp;Thanks for looking into the issue and running the tests and suggesting= to use "direct" flag. I do see a difference with "direct" flag using dd. H= owever the difference is significant when using bs=3D64M compared to bs=3D4= k. 

dd-blocksize  flags  dm-thin     LVMoutput  = ; 
64M                none   179 MB/= s         1GB/s           = ;
64M                direct &= nbsp; 2.4GB/s           3.6GB/s     &nbs= p;  
4k                 =   none   179 MB/s          965MB/s  = ;    
4k               &= nbsp;   direct   193MB/s          1.7GB/= s        

In all the tests, dm-thin performance is below lvm. I used 8M pages as = we are not planning to use snapshots right now.  

We are planning to use the dm-thin module in a high end flash device th= at supports upto 300k IOPS/second. Hence  we are doing some performanc= e tests. I wanted to try more parallel async IO instead of dd. A good progr= am I found on web is 

fsbench.filesystems.org/bench/aio-stress.c

(if you are aware of other tools to test parallel = async io using multiple threads, please let me know. I will give it a try.)=

I compiled it with libaio and libpth. 
Then I ran the test on LVM lun. Throughput is 9563.56MB/s. Ful= ly allocated TP LUN is 5399.81MB/s

I used "perf top" to= see where most of the time is spent. Lot of time is spent in _raw_spi= n_lock. I poked the code little bit and I see that dm_thin_find_block = returns EWOULDBLOCK and hence the bio gets deferred. It looks like spin loc= ks are used when putting the bio on deferred list and getting it back from = the deferred list. I see that EWOULDBLOCK is returned from dm_thin_fin= d_block -> dm_btree_lookup->btree_lookup_raw->ro_step->bn_= read_lock->dm_tm_read_lock->dm_bm_read_try_lock->bl_down_read_nonb= lock.

I tried to change the code little bit so that dm_thin_find_block uses array as a cache and looks up btree only= if not found in the array.
With that change, I see TP performance go= way up.

Fully allocated TP LUN with array lookup is 9387.= 17MB/s. With array, no bios are being deferred. However when I used the arr= ay, I see kernel hang after several runs ( yet to debug the reason).
<= div style=3D"font-family: arial, helvetica, sans-serif; font-size: 12pt; ">=
Code changes are initialize array in pool_ctr() in dm-thin.c
<= div>        for(i=3D0;i<1000;i++)
&nb= sp;       {
        =  blockmap[i]=3D0xffffffff;
        = ;}


dm_thin_find_block() in d= m-thin-metadata.c

at the begining:
=        if(blockmap[block]=3D=3D0xffffffff)
&n= bsp;       {
        = ;        printk("found unassigned block %llu\n",block);=
        }
    =    else
        {
&= nbsp;               result->bloc= k =3D blockmap[block];
            =    result->shared =3D 0;
      =          return 0;
     =    }

at the end before returning:
       if(r =3D=3D 0)
  =      {
           &= nbsp;    blockmap[block] =3D result->block ;
 &= nbsp;      }


Note th= at the array changes are only for my testing and not generic enough at all.  Command I used for getting the throughput numbers are

i=3D0; while [ $i -lt 100 ] ; do ./aio-stress.exe  -O -o 1 -c 1= 6 -t 16 -d 256 /dev/mapper/thin1 2>&1 | grep throughput | cut -f3 -d= ' ' | cut -f2 -d '(' | cut -f1 -d ' ' ; let i=3D$i+1; done  | awk 'BE= GIN{i=3D0.0} {i+=3D$0} END{print i/100}'

Thanks,
=
Jagan.

=
From= : Joe Thornber <thornber@redhat.com>
To: Jagan Reddy <gjmsreddy@yahoo.com>= ; device-mapper development <dm-devel@redhat.com>
Sent: Monday, January 16, 2012 4:42 AM Subject: Re: [dm-devel]= dm-thin vs lvm performance

=0AHi Jagan,

On T= hu, Jan 12, 2012 at 04:55:16PM -0800, Jagan Reddy wrote:
> Hi,
> I  recently started using dm-thin module and first of all, thank= you
> for the good work. It works great, well documented and the com= ments
> in the code are very useful.  I tried to run some perfor= mance tests
> using a high performance flash device and the performan= ce of dm thin
> is about 30% of LVM performance on a full allocated t= hin provisioned
> volume. ( ie after all pages/blocks are allocated u= sing
> dd). Performance test does only reads and no writes. 
=
Thanks very much for taking the time to try thinp out properly.
Peop= le testing different scenarios is very useful to us.

You're probably= aware that the thinp test suite is available here:

    https://github.com/jthornber/thinp-test-suite
I've added a little set of tests that recreate your scenario here:

=     https://github.com/jthornber/th= inp-test-suite/blob/master/ramdisk_tests.rb

I used a 2G ramdisk = for these tests, and a variety of thinp block
sizes and 'dd' options.&nb= sp; I'll just summarise the main results, also I
should point out that m= y testing was done on a VM hosted on a 4G
machine, so the machine was un= der a lot of memory pressure and there
was a lot of variance in the benc= hmarks.

writes across various volumes
---------------------------= --

write1
------

Testing write performance.

dd if= =3D/dev/zero of=3D/dev/mapper/<thin> oflags=3Ddirect bs=3D64M

= zeroing new blocks turned on.

thinp block size =3D 64k

| Line= ar              | 2.2 G/s |
| Unprovisioned t= hin  | 1.4 G/s |
| Provisioned thin    | 1.9   = |
| Snap totally shared | 1.5 G/s |
| Snap no sharing    = | 1.9    |

Pretty good.  Not showing the drastic dr= op that you were seeing.  The
small thinp block size means the snap= s perform nicely (not many
copy-on-writes).

write2
------
<= br>As test1, but with 8M thinp block size as in your tests.

| Linear=               | 2.2 G/s |
| Unprovisi= oned thin  | 1.5 G/s |
| Provisioned thin    | 2.2  =   |
| Snap totally shared | 882 Ms  |
| Snap no sharing&nb= sp;   | 2.2    |

Good results, breaking sharing pe= rformance is down because the large
block size mean there will be more a= ctual copying incurred.

write3
------

As test2, but no oflags=3Ddirect option to dd.

| Linear       = ;       | 900 M/s |
| Unprovisioned thin  | 579 M/s = |
| Provisioned thin    | 694 M/s |
| Snap totally shared |= 510 M/s |
| Snap no sharing    | 654 M/s |

Alarming.=   Results are similar for thinp block size of 64k.

read1
---= --

Testing read performance.

dd if=3D/dev/mapper/<thin>= of=3D/dev/null iflags=3Ddirect bs=3D64M

thinp block size =3D 64k
| Linear              | 3.3 G/s || Provisioned thin    | 2.7 G/s |
| Snap no sharing  &n= bsp; | 2.8 G/s |


read2
-----

read1 but with 8M thinp= block size

| Linear             = | 3.3 G/s |
| Provisioned thin    | 3.2 G/s |
| Snap no sh= aring    | 3.3 G/s |

read3
-----

As read2, but without the iflags=3Ddirect option to 'dd'.

| Linear    &= nbsp;         | 1.0 G/s |
| Provisioned thin  &= nbsp; | 594 M/s |
| Snap no sharing    | 605 M/s |


I think there are a couple of conclusions we can draw from this:
<= br>i) dd isn't great for benchmarking block devices
ii) if you are going= to use it, then make sure you use O_DIRECT

Using an instrumented ke= rnel, I've confirmed that these read tests
are taking the fast path thro= ugh the code.  The mapping tables are all
in memory.  The bio_= map function is always returning DM_MAPIO_REMAPPED.

Will you be usin= g a filesystem on these block devices?

- Joe


---2114655128-221890458-1326915054=:99335-- --===============3417799784578844595== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline --===============3417799784578844595==--