More random write performance data

More random write performance data

[Steven Pratt]

Given the anomalies we were seeing on random write workloads, I decided 
to simplify the test and do single threaded odirect random write.  This 
should eliminate the locking issue as well as any pdflush bursty 
behavior.  What I got was not quite what I expected.

The most interesting graph is probably #12,  DM write throughput.  We 
see a baseline of ~7MB/sec with spikes every 30 seconds.  I assume the 
spike are meta data related as the io is being done from user space at a 
steady constant rate.  The really odd thing is that for the entire 
almost 2 hour duration, the amplitude of the spike continues to climb, 
meaning the amount of meta data need to be flushed to disk is ever 
increasing.

http://btrfs.boxacle.net/repository/raid/longrun/btrfs-longrun-1thread/btrfs1.ffsb.random_writes__threads_0001.09-04-08_13.05.54/analysis/iostat-processed.001/chart.html

Looking at graph #8 DM IO/sec, we see that there is even a pattern 
within the pattern of spikes.  It # of IOs in each spike appears to 
change at each interval and repeats over a set of 7, 30 second intervals.

Also, we see that we average 12MB/sec of data written out, for 5MB/sec 
of benchmark throughput.

I have queued up a run without checksums and cow to see how much this 
overhead is reduced.

Steve

Re: More random write performance data

[Chris Mason]

On Wed, 2009-04-08 at 16:38 -0500, Steven Pratt wrote:
> Given the anomalies we were seeing on random write workloads, I decided 
> to simplify the test and do single threaded odirect random write.  This 
> should eliminate the locking issue as well as any pdflush bursty 
> behavior.  What I got was not quite what I expected.
> 
> The most interesting graph is probably #12,  DM write throughput.  We 
> see a baseline of ~7MB/sec with spikes every 30 seconds.  I assume the 
> spike are meta data related as the io is being done from user space at a 
> steady constant rate.  The really odd thing is that for the entire 
> almost 2 hour duration, the amplitude of the spike continues to climb, 
> meaning the amount of meta data need to be flushed to disk is ever 
> increasing.
> 
> http://btrfs.boxacle.net/repository/raid/longrun/btrfs-longrun-1thread/btrfs1.ffsb.random_writes__threads_0001.09-04-08_13.05.54/analysis/iostat-processed.001/chart.html
> 
> Looking at graph #8 DM IO/sec, we see that there is even a pattern 
> within the pattern of spikes.  It # of IOs in each spike appears to 
> change at each interval and repeats over a set of 7, 30 second intervals.
> 
> Also, we see that we average 12MB/sec of data written out, for 5MB/sec 
> of benchmark throughput.
> 
> I have queued up a run without checksums and cow to see how much this 
> overhead is reduced.

Really interesting, thanks Steve.

I'll have to run it at home next week, but I think the high metadata
writeback is related to updating backrefs on the extent allocation tree.

Most of the reads during the random write are from the same thing.  So,
we're experimenting with changes on that end as well.

-chris

Re: More random write performance data

[Steven Pratt]

Chris Mason wrote:
> On Wed, 2009-04-08 at 16:38 -0500, Steven Pratt wrote:
>   
>> Given the anomalies we were seeing on random write workloads, I decided 
>> to simplify the test and do single threaded odirect random write.  This 
>> should eliminate the locking issue as well as any pdflush bursty 
>> behavior.  What I got was not quite what I expected.
>>
>> The most interesting graph is probably #12,  DM write throughput.  We 
>> see a baseline of ~7MB/sec with spikes every 30 seconds.  I assume the 
>> spike are meta data related as the io is being done from user space at a 
>> steady constant rate.  The really odd thing is that for the entire 
>> almost 2 hour duration, the amplitude of the spike continues to climb, 
>> meaning the amount of meta data need to be flushed to disk is ever 
>> increasing.
>>
>> http://btrfs.boxacle.net/repository/raid/longrun/btrfs-longrun-1thread/btrfs1.ffsb.random_writes__threads_0001.09-04-08_13.05.54/analysis/iostat-processed.001/chart.html
>>
>> Looking at graph #8 DM IO/sec, we see that there is even a pattern 
>> within the pattern of spikes.  It # of IOs in each spike appears to 
>> change at each interval and repeats over a set of 7, 30 second intervals.
>>
>> Also, we see that we average 12MB/sec of data written out, for 5MB/sec 
>> of benchmark throughput.
>>
>> I have queued up a run without checksums and cow to see how much this 
>> overhead is reduced.
>>     
>
> Really interesting, thanks Steve.
>
> I'll have to run it at home next week, but I think the high metadata
> writeback is related to updating backrefs on the extent allocation tree.
>   
Well, looks like you are correct.  Using nodatacow has virtually 
eliminated the extra writes.  I is also responsible for a whopping 40x 
increase in multi threaded random write performance! (2.5MB/sec -> 
95MB/sec).  See complete details in the new history graphs which I have 
updated with a new baseline, a run with no csums, and a run with no 
csums and no cow.

http://btrfs.boxacle.net/repository/raid/history/History.html

nocow make massive differences on the random write workloads, while no 
csums help the heavily threaded sequential workloads (sequential read 
and create).

Steve

> Most of the reads during the random write are from the same thing.  So,
> we're experimenting with changes on that end as well.
>
> -chris
>
>   

Re: More random write performance data

[Chris Mason]

On Thu, 2009-04-09 at 16:41 -0500, Steven Pratt wrote:
> Chris Mason wrote:
> > On Wed, 2009-04-08 at 16:38 -0500, Steven Pratt wrote:
> >   
> >> Given the anomalies we were seeing on random write workloads, I decided 
> >> to simplify the test and do single threaded odirect random write.  This 
> >> should eliminate the locking issue as well as any pdflush bursty 
> >> behavior.  What I got was not quite what I expected.
> >>
> >> The most interesting graph is probably #12,  DM write throughput.  We 
> >> see a baseline of ~7MB/sec with spikes every 30 seconds.  I assume the 
> >> spike are meta data related as the io is being done from user space at a 
> >> steady constant rate.  The really odd thing is that for the entire 
> >> almost 2 hour duration, the amplitude of the spike continues to climb, 
> >> meaning the amount of meta data need to be flushed to disk is ever 
> >> increasing.
> >>
> >> http://btrfs.boxacle.net/repository/raid/longrun/btrfs-longrun-1thread/btrfs1.ffsb.random_writes__threads_0001.09-04-08_13.05.54/analysis/iostat-processed.001/chart.html
> >>
> >> Looking at graph #8 DM IO/sec, we see that there is even a pattern 
> >> within the pattern of spikes.  It # of IOs in each spike appears to 
> >> change at each interval and repeats over a set of 7, 30 second intervals.
> >>
> >> Also, we see that we average 12MB/sec of data written out, for 5MB/sec 
> >> of benchmark throughput.
> >>
> >> I have queued up a run without checksums and cow to see how much this 
> >> overhead is reduced.
> >>     
> >
> > Really interesting, thanks Steve.
> >
> > I'll have to run it at home next week, but I think the high metadata
> > writeback is related to updating backrefs on the extent allocation tree.
> >   
> Well, looks like you are correct.  Using nodatacow has virtually 
> eliminated the extra writes.  I is also responsible for a whopping 40x 
> increase in multi threaded random write performance! (2.5MB/sec -> 
> 95MB/sec).  See complete details in the new history graphs which I have 
> updated with a new baseline, a run with no csums, and a run with no 
> csums and no cow.
> 
> http://btrfs.boxacle.net/repository/raid/history/History.html
> 

Whoa.  So, we're on the right track, its good to know the btree locking
is scaling well enough for the main btree as well.

> nocow make massive differences on the random write workloads, while no 
> csums help the heavily threaded sequential workloads (sequential read 
> and create).

Ok, thanks again.

-chris