Multi-device update

Multi-device update

[Chris Mason]

Hello everyone,

I've pushed out another set of changes to the unstable trees, and these 
include:

* RAID 1+0 support
    mkfs.btrfs -d raid10 -m raid10 /dev/sd...
    4 or more drives required.

* async work queues for checksumming writes
* Better back references in the multi-device data structs

The async work queues include code to checksum data pages without the FS mutex 
held, greatly increasing streaming write throughput.  On my 4 drive system, I 
was getting around 120MB/s writes with checksumming on.  Now I get 180MB/s, 
which is disk speed.

The rest of the week will be spent doing hot add/remove of devices.  Happy 
testing ;)

-chris

Re: Multi-device update

[Andi Kleen]

Chris Mason <chris.mason@oracle.com> writes:
>
> The async work queues include code to checksum data pages without the FS mutex 

Are they able to distribute work to other cores?

-Andi

Re: Multi-device update

[Chris Mason]

On Wednesday 16 April 2008, Andi Kleen wrote:
> Chris Mason <chris.mason@oracle.com> writes:
> > The async work queues include code to checksum data pages without the FS
> > mutex
>
> Are they able to distribute work to other cores?

Yes, it just uses a workqueue.  The current implemention is pretty simple, it 
surely could be more effective at spreading the work around.

I'm testing a variant that only tosses over to the async queue for pdflush, 
inline reclaim should stay inline.

-chris

Re: Multi-device update

[Andi Kleen]

Chris Mason <chris.mason@oracle.com> writes:

> On Wednesday 16 April 2008, Andi Kleen wrote:
>> Chris Mason <chris.mason@oracle.com> writes:
>> > The async work queues include code to checksum data pages without the FS
>> > mutex
>>
>> Are they able to distribute work to other cores?
>
> Yes, it just uses a workqueue. 

Unfortunately work queues don't do that by default currently. They
tend to process on the current CPU only.

> The current implemention is pretty simple, it 
> surely could be more effective at spreading the work around.
>
> I'm testing a variant that only tosses over to the async queue for pdflush, 
> inline reclaim should stay inline.

Longer term I would hope that write checksum will be basically free by doing
csum-copy at write() time. The only problem is just where to store the
checksum between the write and the final IO? There's no space in 
struct page.

The same could be also done for read() but that might be a little more
tricky because it would require delayed error reporting and it might
be difficult to do this for partial blocks?

-Andi

Re: Multi-device update

[Chris Mason]

On Wednesday 16 April 2008, Andi Kleen wrote:
> Chris Mason <chris.mason@oracle.com> writes:
> > On Wednesday 16 April 2008, Andi Kleen wrote:
> >> Chris Mason <chris.mason@oracle.com> writes:
> >> > The async work queues include code to checksum data pages without the
> >> > FS mutex
> >>
> >> Are they able to distribute work to other cores?
> >
> > Yes, it just uses a workqueue.
>
> Unfortunately work queues don't do that by default currently. They
> tend to process on the current CPU only.

Well, I see multiple work queue threads using CPU time, but I haven't spent 
much time optimizing it.  There's definitely room for improvement.

>
> > The current implemention is pretty simple, it
> > surely could be more effective at spreading the work around.
> >
> > I'm testing a variant that only tosses over to the async queue for
> > pdflush, inline reclaim should stay inline.
>
> Longer term I would hope that write checksum will be basically free by
> doing csum-copy at write() time. The only problem is just where to store
> the checksum between the write and the final IO? There's no space in
> struct page.

At write time is easier (except for mmap) because I can toss the csum directly 
into the btree inside btrfs_file_write.  The current code avoids that 
complexity and does it all at writeout.

One advantage to the current code is that I'm able to optimize tree searches 
away but checksumming a bunch of pages at a time.  Multiple pages worth of 
checksums get stored in a single btree item, so at least for btree operations 
the current code is fairly optimal.

>
> The same could be also done for read() but that might be a little more
> tricky because it would require delayed error reporting and it might
> be difficult to do this for partial blocks?

Yeah, it doesn't quite fit with how the kernel does reads.  For now it is much 
easier if the retry-other-mirror operation happens long before copy_to_user.

-chris

Re: Multi-device update

[Andi Kleen]

Chris Mason wrote:
> On Wednesday 16 April 2008, Andi Kleen wrote:
>> Chris Mason <chris.mason@oracle.com> writes:
>>> On Wednesday 16 April 2008, Andi Kleen wrote:
>>>> Chris Mason <chris.mason@oracle.com> writes:
>>>>> The async work queues include code to checksum data pages without the
>>>>> FS mutex
>>>> Are they able to distribute work to other cores?
>>> Yes, it just uses a workqueue.
>> Unfortunately work queues don't do that by default currently. They
>> tend to process on the current CPU only.
> 
> Well, I see multiple work queue threads using CPU time, but I haven't spent 
> much time optimizing it.  There's definitely room for improvement.

That's likely because you submit from multiple CPUs. But with a single
submitter running on a single CPU there shouldn't be any load balancing
currently.

-Andi

Re: Multi-device update

[Jens Axboe]

On Wed, Apr 16 2008, Andi Kleen wrote:
> Chris Mason wrote:
> > On Wednesday 16 April 2008, Andi Kleen wrote:
> >> Chris Mason <chris.mason@oracle.com> writes:
> >>> On Wednesday 16 April 2008, Andi Kleen wrote:
> >>>> Chris Mason <chris.mason@oracle.com> writes:
> >>>>> The async work queues include code to checksum data pages without the
> >>>>> FS mutex
> >>>> Are they able to distribute work to other cores?
> >>> Yes, it just uses a workqueue.
> >> Unfortunately work queues don't do that by default currently. They
> >> tend to process on the current CPU only.
> > 
> > Well, I see multiple work queue threads using CPU time, but I haven't spent 
> > much time optimizing it.  There's definitely room for improvement.
> 
> That's likely because you submit from multiple CPUs. But with a single
> submitter running on a single CPU there shouldn't be any load balancing
> currently.

There have been various implementations of queue_work_on() posted
through the years, I've had one version that I've used off and on for a
long time:

http://git.kernel.dk/?p=linux-2.6-block.git;a=commit;h=c68c42fd6df96f5b3fb5b8b47c571f233d054c71

then you need some balancing decider on top of that, of course.

-- 
Jens Axboe

Re: Multi-device update

[Andi Kleen]

> There have been various implementations of queue_work_on() posted
> through the years, I've had one version that I've used off and on for a
> long time:

queue_work_on is the wrong interface I think. You rather
want a pool of non pinned threads that are then load balanced by the
scheduler (who knows best what cpus have cycles available)

-Andi

Re: Multi-device update

[Jens Axboe]

On Wed, Apr 16 2008, Andi Kleen wrote:
> 
> > There have been various implementations of queue_work_on() posted
> > through the years, I've had one version that I've used off and on for a
> > long time:
> 
> queue_work_on is the wrong interface I think. You rather
> want a pool of non pinned threads that are then load balanced by the
> scheduler (who knows best what cpus have cycles available)

Yeah, that actually sounds like the best interface. What I described
typically ends up trying to be too clever, you really want to leave any
scheduling decisions to the scheduler.

-- 
Jens Axboe

Re: Multi-device update

[Chris Mason]

On Wednesday 16 April 2008, Andi Kleen wrote:
> > There have been various implementations of queue_work_on() posted
> > through the years, I've had one version that I've used off and on for a
> > long time:
>
> queue_work_on is the wrong interface I think. You rather
> want a pool of non pinned threads that are then load balanced by the
> scheduler (who knows best what cpus have cycles available)

Fair enough, I'll tune things a bit.

-chris