changes to kiobuf support in 2.4.(?)4

changes to kiobuf support in 2.4.(?)4

[Jeremy Higdon]

I'm curious about the changes made to kiobuf support, apparently in
order to improve raw I/O performance.  I was reading through some old
posts circa early April, where I saw reference to a rawio-bench program.
Can someone explain what it does?

I have a driver that uses kiobufs to perform I/O.  Prior to these
changes, the kiobuf allocation and manipulation was very quick and
efficient.  It is now very slow.

The kiobuf part of the I/O request is as follows:

	alloc_kiovec()
	map_user_kiobuf()
	... do I/O, using kiobuf to store mappings ...
	kiobuf_wait_for_io()
	free_kiovec()

Now that the kiobuf is several KB in size, and 1024 buffer heads
are allocated, the alloc_kiovec part goes from a percent or so of
CPU usage to do 13000 requests per second to around 90% CPU usage
to do 3000 per second.

It looks as though the raw driver allocates one kiobuf at open time
(rather than on a per-request basis), but if two or more requests
are issued to a single raw device, it too devolves into the allocate
on every request strategy.

Before I go further, I'd appreciate if someone could confirm my
hypotheses and also explain rawio-bench (and maybe point me to some
source, if available).

thanks

jeremy

Re: changes to kiobuf support in 2.4.(?)4

[Andrea Arcangeli]

On Wed, Aug 01, 2001 at 10:55:00PM -0700, Jeremy Higdon wrote:
> I'm curious about the changes made to kiobuf support, apparently in
> order to improve raw I/O performance.  I was reading through some old
> posts circa early April, where I saw reference to a rawio-bench program.
> Can someone explain what it does?
> 
> I have a driver that uses kiobufs to perform I/O.  Prior to these
> changes, the kiobuf allocation and manipulation was very quick and
> efficient.  It is now very slow.
> 
> The kiobuf part of the I/O request is as follows:
> 
> 	alloc_kiovec()
> 	map_user_kiobuf()
> 	... do I/O, using kiobuf to store mappings ...
> 	kiobuf_wait_for_io()
> 	free_kiovec()
> 
> Now that the kiobuf is several KB in size, and 1024 buffer heads
> are allocated, the alloc_kiovec part goes from a percent or so of
> CPU usage to do 13000 requests per second to around 90% CPU usage
> to do 3000 per second.
> 
> It looks as though the raw driver allocates one kiobuf at open time
> (rather than on a per-request basis), but if two or more requests
> are issued to a single raw device, it too devolves into the allocate
> on every request strategy.
> 
> Before I go further, I'd appreciate if someone could confirm my
> hypotheses and also explain rawio-bench (and maybe point me to some
> source, if available).

That's all right, no one kiobuf should be allocated in any fast path, if
you follow this rule there's no problem.

The kiobuf in 2.4 is mostly used for rawio which can allocate the kiobuf
at open time. With the O_DIRECT patch I added one kiobuf per each filp
(of course allocated only when you open the file with O_DIRECT), the
rawio should simply switch to use the per-file kiobuf so you can open
the raw device multiple times without overhead of allocating the kiobuf
during simultaneous read/writes (if anybody wants to implement this
improvement on top of O_DIRECT please let me know so we don't duplicate
efforts).

The reason of the large allocation and to put the bh inside the kiobuf
is that if we do a small allocation then we end with a zillion of
allocations of the bh and freeing of the bh at every I/O!! (not even at
every read/write syscall, much more frequently) The flood of allocation
and freeing of the 512byte large bh at every I/O was just an obvious
huge amount of
wasted cpu time and that's why allocating them statically in the kiobuf
makes thing much faster (and also it avoids to run out of memory during
I/O making the rawio reliable), but of course now it's proibitive to
allocate any kiobuf in any fast path (even more since I switched to
vmalloc/vfree to allocate the kiobuf since as said above in 2.4 nobody
should ever allocate a kiobuf in any fast path anyways because of its
size). If you want to make kiobuf a remotely light thing you should
make the kiobuf become the I/O entity instead of the bh (which I'm not
saying it's a good thing here). If you don't make the kiobuf the I/O
entity understood at the very low level from the device drivers, then
you'd better keep having the bh in the kiobuf.

OTOH I'm a little biased in the above reasoning since I use the kiobuf
only for doing direct I/O (I always end calling brw_kiovec somehow,
otherwise I wouldn't be using the kiobufs at all). If you are using the
kiobufs for framebuffers and other drivers that never ends calling
brw_kiovec I think you should be using the mmap callback and
remap_page_range instead as most drivers correctly do to avoid the
overhead of the kiobufs. But ok if you really want to use the kiobuf for
non I/O things instead of remap_page_range (dunno why) we could split
off the bh-array allocation from the kiobuf to make it a bit lighter so
you could use it for non-IO operations without the overhead of the bhs,
but still we should adapt rawio to preallocate the bh at open/close time
(separately from the kiovec).

Andrea

Re: changes to kiobuf support in 2.4.(?)4

[Jeremy Higdon]

On Aug 2,  8:43am, Andrea Arcangeli wrote:
> 
> OTOH I'm a little biased in the above reasoning since I use the kiobuf
> only for doing direct I/O (I always end calling brw_kiovec somehow,
> otherwise I wouldn't be using the kiobufs at all). If you are using the
> kiobufs for framebuffers and other drivers that never ends calling
> brw_kiovec I think you should be using the mmap callback and

By "mmap callback", you're referring to the mmap entry in the file_operations
structure?

> remap_page_range instead as most drivers correctly do to avoid the
> overhead of the kiobufs. But ok if you really want to use the kiobuf for
> non I/O things instead of remap_page_range (dunno why) we could split
> off the bh-array allocation from the kiobuf to make it a bit lighter so
> you could use it for non-IO operations without the overhead of the bhs,
> but still we should adapt rawio to preallocate the bh at open/close time
> (separately from the kiovec).
> 
> Andrea

I am doing direct I/O.  I'm using the kiobuf to hold the page addresses
of the user's data buffer, but I'm calling directly into my driver
after doing the map_user_kiobuf() (I have a read/write request, a file
offset, a byte count, and a set of pages to DMA into/out of, and that
gets directly translated into a SCSI command).

It turns out that the old kmem_cache_alloc was very lightweight, so I
could get away with doing it once per I/O request, so I would indeed
profit by going back to a light weight kiobuf, or at least an optional
allocation of the bh and blocks arrays (perhaps turn them into pointers
to arrays?).

thanks

jeremy

Re: changes to kiobuf support in 2.4.(?)4

[Andrea Arcangeli]

On Thu, Aug 02, 2001 at 12:31:52AM -0700, Jeremy Higdon wrote:
> By "mmap callback", you're referring to the mmap entry in the file_operations
> structure?

yes.

> I am doing direct I/O.  I'm using the kiobuf to hold the page addresses
> of the user's data buffer, but I'm calling directly into my driver
> after doing the map_user_kiobuf() (I have a read/write request, a file
> offset, a byte count, and a set of pages to DMA into/out of, and that
> gets directly translated into a SCSI command).
> 
> It turns out that the old kmem_cache_alloc was very lightweight, so I
> could get away with doing it once per I/O request, so I would indeed
> profit by going back to a light weight kiobuf, or at least an optional
> allocation of the bh and blocks arrays (perhaps turn them into pointers
> to arrays?).

I see your problem and it's a valid point indeed. But could you actually
allocate the kiobuf in the file->f_iobuf pointer? I mean: could you
allocate it at open/close too?  That would be the way I prefer since you
would need to allocate the bh anyways later (but with a flood of
alloc/free). So if you could move the kiobufs allocation out of the fast
path you would get a benefit too I believe.

Andrea

Re: changes to kiobuf support in 2.4.(?)4

[Jeremy Higdon]

On Aug 2,  9:45am, Andrea Arcangeli wrote:
> 
> > I am doing direct I/O.  I'm using the kiobuf to hold the page addresses
> > of the user's data buffer, but I'm calling directly into my driver
> > after doing the map_user_kiobuf() (I have a read/write request, a file
> > offset, a byte count, and a set of pages to DMA into/out of, and that
> > gets directly translated into a SCSI command).
> > 
> > It turns out that the old kmem_cache_alloc was very lightweight, so I
> > could get away with doing it once per I/O request, so I would indeed
> > profit by going back to a light weight kiobuf, or at least an optional
> > allocation of the bh and blocks arrays (perhaps turn them into pointers
> > to arrays?).
> 
> I see your problem and it's a valid point indeed. But could you actually
> allocate the kiobuf in the file->f_iobuf pointer? I mean: could you
> allocate it at open/close too?  That would be the way I prefer since you
> would need to allocate the bh anyways later (but with a flood of
> alloc/free). So if you could move the kiobufs allocation out of the fast
> path you would get a benefit too I believe.

I have two answers to this.

The first is that I don't need the bh's or block's in my implementation.
Everything I need is in the old-style kiobuf or is passed as an argument.

The second is I don't see a file->f_iobuf pointer in my source tree, which
is 2.4.8-pre3, I believe.  In fact, the kiobuf pointer is stored in the
raw_devices array in my version of raw.c, and there is only one per raw
device.

Assuming I'm out of date, and there is some way to store a kiobuf pointer
into the file data structure, and I'll never see two requests outstanding
at the same time to the same file, then I could do as you suggest.  I'd
be wasting about 16KB per open file (assuming 512KB and 64 bit) and adding
unneeded CPU overhead at open time, but I could live with that.

> Andrea

thanks

jeremy

Re: changes to kiobuf support in 2.4.(?)4

[Andrea Arcangeli]

On Thu, Aug 02, 2001 at 01:10:44AM -0700, Jeremy Higdon wrote:
> The first is that I don't need the bh's or block's in my implementation.

So the very latest layer of the scsi driver understand the kiobufs
natively for you right? (I assume you ported this functionality only to
a few scsi drivers, right?)

> Everything I need is in the old-style kiobuf or is passed as an argument.

Ok.

> The second is I don't see a file->f_iobuf pointer in my source tree, which
> is 2.4.8-pre3, I believe.  In fact, the kiobuf pointer is stored in the

It's in the O_DIRECT patch.

> raw_devices array in my version of raw.c, and there is only one per raw
> device.

This is why I said also rawio should start using the f_iobuf to have one
kiobuf per-file like with O_DIRECT, infact if you open("/dev/hda",
O_DIRECT) instead of using the rawio API you will just get the kiobuf
per file.

> Assuming I'm out of date, and there is some way to store a kiobuf pointer
> into the file data structure, and I'll never see two requests outstanding
> at the same time to the same file, then I could do as you suggest.  I'd

There's the possibility of two requests outstanding on the same file
still if you share the same file with multiple filedescriptors but I
don't think that's an interesting case to optimize, however I still need
a test_and_set_bit and a slow path allocation of the kiovec to handle
the multiple fd pointing to the same filp case correctly (but I think
that's ok).

> be wasting about 16KB per open file (assuming 512KB and 64 bit) and adding
> unneeded CPU overhead at open time, but I could live with that.

If you don't need the bh and blocks part of the kiobuf and we split it
off (which I would be fine to change if the lowlevel driver would
understand the kiobuf as an I/O entity, that again I'm not saying it's a
good thing or not here) you should still be faster by avoiding
allocating the kiobuf in the fast path and you won't have the 16KB
overhead per open device once/if the kiobuf will shrink in size (so it
should still better than allocating a smaller kiobuf in a fast path).

What do you think?

Andrea

Re: changes to kiobuf support in 2.4.(?)4

[Jeremy Higdon]

On Aug 2, 10:24am, Andrea Arcangeli wrote:
> 
> On Thu, Aug 02, 2001 at 01:10:44AM -0700, Jeremy Higdon wrote:
> > The first is that I don't need the bh's or block's in my implementation.
> 
> So the very latest layer of the scsi driver understand the kiobufs
> natively for you right? (I assume you ported this functionality only to
> a few scsi drivers, right?)

Essentially.  I have a sort of parallel layer.

> > The second is I don't see a file->f_iobuf pointer in my source tree, which
> > is 2.4.8-pre3, I believe.  In fact, the kiobuf pointer is stored in the
> 
> It's in the O_DIRECT patch.
> 
> > raw_devices array in my version of raw.c, and there is only one per raw
> > device.
> 
> This is why I said also rawio should start using the f_iobuf to have one
> kiobuf per-file like with O_DIRECT, infact if you open("/dev/hda",
> O_DIRECT) instead of using the rawio API you will just get the kiobuf
> per file.
> 
> > Assuming I'm out of date, and there is some way to store a kiobuf pointer
> > into the file data structure, and I'll never see two requests outstanding
> > at the same time to the same file, then I could do as you suggest.  I'd
> 
> There's the possibility of two requests outstanding on the same file
> still if you share the same file with multiple filedescriptors but I
> don't think that's an interesting case to optimize, however I still need
> a test_and_set_bit and a slow path allocation of the kiovec to handle
> the multiple fd pointing to the same filp case correctly (but I think
> that's ok).

My understanding is that databases like to have multiple outstanding
requests to the same file, which I believe falls into the the multiple
file descriptors, one file case.  So for us, it is interesting.  Or
do I misunderstand what you wrote?

Actually, I want to be clear on this . . .

	If I do

	dd if=/dev/raw1 . . . &
	dd if=/dev/raw1 . . . &
	wait

with the O_DIRECT patch, do I get some slow path allocations?

> > be wasting about 16KB per open file (assuming 512KB and 64 bit) and adding
> > unneeded CPU overhead at open time, but I could live with that.
> 
> If you don't need the bh and blocks part of the kiobuf and we split it
> off (which I would be fine to change if the lowlevel driver would
> understand the kiobuf as an I/O entity, that again I'm not saying it's a
> good thing or not here) you should still be faster by avoiding
> allocating the kiobuf in the fast path and you won't have the 16KB
> overhead per open device once/if the kiobuf will shrink in size (so it
> should still better than allocating a smaller kiobuf in a fast path).
> 
> What do you think?

At 13000 IOPS, when allocating and freeing on every I/O request,
the allocate/free overhead was approximately .6% on a 2 CPU system,
where the total overhead was about 25%.  So I would theoretically
gain 3% (maybe a little better since there is locking involved) if
I could avoid the alloc/free.

> Andrea
> 
>-- End of excerpt from Andrea Arcangeli

thanks

jeremy

Re: changes to kiobuf support in 2.4.(?)4

[Andrea Arcangeli]

On Thu, Aug 02, 2001 at 01:42:21AM -0700, Jeremy Higdon wrote:
> My understanding is that databases like to have multiple outstanding
> requests to the same file, which I believe falls into the the multiple
> file descriptors, one file case.  So for us, it is interesting.  Or
> do I misunderstand what you wrote?

the databases I know open the file without sharing the fd, so they're
fine even if we share the same iobuf from multiple fd. Infact even with
threads you can open the same file multiple times.

> Actually, I want to be clear on this . . .
> 
> 	If I do
> 
> 	dd if=/dev/raw1 . . . &
> 	dd if=/dev/raw1 . . . &
> 	wait
> 
> with the O_DIRECT patch, do I get some slow path allocations?

That will work fine with the per-file iobuf ala O_DIRECT, no slow path
allocations will ever happen.

Even if you do:

	clone()
	parent:
		fd = open("/dev/raw1")
		write(fd)
		read(fd)
	child:
		fd = open("/dev/raw1")
		write(fd)
		read(fd)

you won't run into slow path allocations.

> At 13000 IOPS, when allocating and freeing on every I/O request,
> the allocate/free overhead was approximately .6% on a 2 CPU system,
> where the total overhead was about 25%.  So I would theoretically
> gain 3% (maybe a little better since there is locking involved) if
> I could avoid the alloc/free.

Ok good.

Andrea

Re: changes to kiobuf support in 2.4.(?)4

[Jeremy Higdon]

On Aug 2, 11:11am, Andrea Arcangeli wrote:
> 
> > At 13000 IOPS, when allocating and freeing on every I/O request,
> > the allocate/free overhead was approximately .6% on a 2 CPU system,
> > where the total overhead was about 25%.  So I would theoretically
> > gain 3% (maybe a little better since there is locking involved) if
> > I could avoid the alloc/free.
> 
> Ok good.
> 
> Andrea


So one more question for now:

Where do I get the O_DIRECT patch?

Oh, and is there a plan to get it into 2.4.X?

(ok, so two questions)

thanks

jeremy

Re: changes to kiobuf support in 2.4.(?)4

[Andrea Arcangeli]

On Thu, Aug 02, 2001 at 02:25:48AM -0700, Jeremy Higdon wrote:
> On Aug 2, 11:11am, Andrea Arcangeli wrote:
> > 
> > > At 13000 IOPS, when allocating and freeing on every I/O request,
> > > the allocate/free overhead was approximately .6% on a 2 CPU system,
> > > where the total overhead was about 25%.  So I would theoretically
> > > gain 3% (maybe a little better since there is locking involved) if
> > > I could avoid the alloc/free.
> > 
> > Ok good.
> > 
> > Andrea
> 
> 
> So one more question for now:
> 
> Where do I get the O_DIRECT patch?

I will upload a new one today in my ftp area on kernel.org against
2.4.8pre3. (the current one has a silly bug spotted by Ken Preslan while
testing O_DIRECT on GFS)

> Oh, and is there a plan to get it into 2.4.X?

The one I will upload today is stable enough to be ok for inclusion and
it generates good numbers. Infact the O_DIRECT way on top of the fs will
be the prefered way for storing DB files compared to rawio on lvm that
people uses today (except when using the shared storage but even for it
I just had feedback from the GFS folks that just verified GFS to work
correctly with the O_DIRECT patch). The only detail left for some DB
that wants to use 2k I/O granularity also on a 4k blocksized-fs is to
relax the granularity of the I/O from the softblocksize to the
hardblocksize but it will be tricky to do that without losing any
performance (however I don't see that as a requirement to make use of
O_DIRECT, infact other people is using it without seeing the
softblocksize constraint as a problem) but we'll have to relax that
constraint eventually to make everybody happy.

Andrea

Re: changes to kiobuf support in 2.4.(?)4

[Gerd Knorr]

>  The reason of the large allocation and to put the bh inside the kiobuf
>  is that if we do a small allocation then we end with a zillion of
>  allocations of the bh and freeing of the bh at every I/O!! (not even at
>  every read/write syscall, much more frequently)

That is true for block device I/O only.  Current bttv versions are using
kiobufs to lock down user pages for DMA.  But I don't need the bh's to
transfer the video frames ...

  Gerd

-- 
Damn lot people confuse usability and eye-candy.

Re: changes to kiobuf support in 2.4.(?)4

[Ingo Oeser]

On Thu, Aug 02, 2001 at 08:23:37AM +0000, Gerd Knorr wrote:
> >  The reason of the large allocation and to put the bh inside the kiobuf
> >  is that if we do a small allocation then we end with a zillion of
> >  allocations of the bh and freeing of the bh at every I/O!! (not even at
> >  every read/write syscall, much more frequently)
> 
> That is true for block device I/O only.  Current bttv versions are using
> kiobufs to lock down user pages for DMA.  But I don't need the bh's to
> transfer the video frames ...

This is another problem. We miss a whole layer of fast streaming
and chunking IO, which is not meant for block devices.

The most practical example would be the sg-driver, but there are
other things which require this kind of semantic:

Transmit Buffer via DMA to device (and programming it for
operations on that buffer) and possible receiving sth. back.

We need no reordering here, but would like to do the DMA directly
from user space buffers.

Examples:

   - GiMP plugin, which renders some complex algorithm on a DSP.

   - Crypto (Co-)processors, which encrypt data streams

   - Screengrabbers, which can grab a specific area.

   - (intelligent, multi) data aquisistion devices (large sensor
     arrays)

   ...

I would like to have a thing like the zero copy IO in the network
layer, but not done with NICs only, but with ANY device.

For the DSP case (which might be the most complex one) I did some
research already, but I don't like to see this work duplicated
over and over again.

I built transfer structs similar to BHs but more simply and have
a queue of that per device and allocate transfers from a slab.

The blocking and unblocking is done in the driver anyway. I only
try to do zero copy IO, if I have complete pages, which
simplifies it a "little" ;-)

Without the coalescing and reordering it's pretty simple.

What do the gurus think here?

Regards

Ingo Oeser
-- 
You mean a kill file only kills email!!!! DAMN!!!! All these years
I thought I was doing the gene pool some good...
                             --- "Clint Wolff" <vaxman@qwest.net>

Re: changes to kiobuf support in 2.4.(?)4

[Andrea Arcangeli]

On Thu, Aug 02, 2001 at 08:23:37AM +0000, Gerd Knorr wrote:
> >  The reason of the large allocation and to put the bh inside the kiobuf
> >  is that if we do a small allocation then we end with a zillion of
> >  allocations of the bh and freeing of the bh at every I/O!! (not even at
> >  every read/write syscall, much more frequently)
> 
> That is true for block device I/O only.  Current bttv versions are using
> kiobufs to lock down user pages for DMA.  But I don't need the bh's to
> transfer the video frames ...

I guess you use map_user_kiobuf to provide zerocopy to read/write too
(not just to user-mapped ram allocated by bttv), right?

If you allocate the kiobuf not in any fast path the vmalloc and big
allocation won't be a real issue even now, however I agree it's ok to
split the bh/block array allocation out of the kiobuf to make it lighter
(but still it won't be a light thing).

Andrea