* Re: raw devices
2001-12-20 12:22 raw devices simon
@ 2001-12-20 17:05 ` Andrea Arcangeli
2001-12-21 7:26 ` simon
1 sibling, 0 replies; 3+ messages in thread
From: Andrea Arcangeli @ 2001-12-20 17:05 UTC (permalink / raw)
To: simon; +Cc: linux-kernel
On Thu, Dec 20, 2001 at 12:22:33PM -0000, simon@baydel.com wrote:
> I have been performing some tests on raw devices, the results of
> which I do not understand. I have 4 FC busses, each with one
> SCSI disk, connect to a Linux system running 2.4.16. I use the raw
> command to bind /dev/raw1 - /dev/raw4 to each of the devices.
> With one process per raw device, running large sequential reads, I
> got a total throughput of 340 Megabytes per second. I also
> observed 85% CPU idle. Following this I performed some more
> tests and then returned to this one. This time the total had gone
> down to 180 and there was no free CPU. I realized that the first
> time I ran the tests, each of the disks that the raw devices were
> mapped to were mounted. I then verified that this data was being
> transferred along the FC bus using an analyzer while the devices
> were mounted. Can anyone explain this to me ? I find it hard to
> believe that the disk should be permitted to be mounted when
> using raw device mappings. If the disks should not be mounted
> why is there such a great performance difference ?
because when mounted rawio uses a blocksize larger than the
softblocksize (for no good reason, but that's another issue).
in short when the disk is mouned a single bh was doing I/O on a page (I
guess you were using 4k blocksize in the fs), while when it was
unmounted it was doing I/O on only 512bytes, so you needed 8 times more
bh and CPU to do the same I/O.
there are many ways to fix this, I guess it would be nice to get the bio
stuff optimized in 2.5 first (bio will be even faster than your rawio
with the fs mounted, because a single metadata entity will be able to do
I/O on more than one page, that's the whole point of the bio design
changes). Then in 2.5 rawio can also be obsoleted and modularized.
O_DIRECT will have to relax its granularity requirements for both fs and
blkdev and so then it will overlap enterely the rawio chardevice
functionality.
btw, if you try to use O_DIRECT on the fs with your current kernel, you
should be just able to read at 340mbyte/sec with most of the CPU idle.
Andrea
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: raw devices
2001-12-20 12:22 raw devices simon
2001-12-20 17:05 ` Andrea Arcangeli
@ 2001-12-21 7:26 ` simon
1 sibling, 0 replies; 3+ messages in thread
From: simon @ 2001-12-21 7:26 UTC (permalink / raw)
To: Andrea Arcangeli; +Cc: linux-kernel
Andrea,
This clears this up, thanks very much. I have tried using the
O_DIRECT and it is almost as fast as the mounted raw devices.
It sounds like 2.5 is set to improve on this. Do you know if there
have been any suggestions regarding using mounting options on a
filesystem so that none of the files on that filesystem are cached ?
Thanks again
Simon.
On 20 Dec 2001, at 18:05, Andrea Arcangeli wrote:
> On Thu, Dec 20, 2001 at 12:22:33PM -0000, simon@baydel.com wrote:
> > I have been performing some tests on raw devices, the results of
> > which I do not understand. I have 4 FC busses, each with one
> > SCSI disk, connect to a Linux system running 2.4.16. I use the raw
> > command to bind /dev/raw1 - /dev/raw4 to each of the devices.
> > With one process per raw device, running large sequential reads, I
> > got a total throughput of 340 Megabytes per second. I also
> > observed 85% CPU idle. Following this I performed some more
> > tests and then returned to this one. This time the total had gone
> > down to 180 and there was no free CPU. I realized that the first
> > time I ran the tests, each of the disks that the raw devices were
> > mapped to were mounted. I then verified that this data was being
> > transferred along the FC bus using an analyzer while the devices
> > were mounted. Can anyone explain this to me ? I find it hard to
> > believe that the disk should be permitted to be mounted when
> > using raw device mappings. If the disks should not be mounted
> > why is there such a great performance difference ?
>
> because when mounted rawio uses a blocksize larger than the
> softblocksize (for no good reason, but that's another issue).
>
> in short when the disk is mouned a single bh was doing I/O on a page (I
> guess you were using 4k blocksize in the fs), while when it was
> unmounted it was doing I/O on only 512bytes, so you needed 8 times more
> bh and CPU to do the same I/O.
>
> there are many ways to fix this, I guess it would be nice to get the bio
> stuff optimized in 2.5 first (bio will be even faster than your rawio
> with the fs mounted, because a single metadata entity will be able to do
> I/O on more than one page, that's the whole point of the bio design
> changes). Then in 2.5 rawio can also be obsoleted and modularized.
> O_DIRECT will have to relax its granularity requirements for both fs and
> blkdev and so then it will overlap enterely the rawio chardevice
> functionality.
>
> btw, if you try to use O_DIRECT on the fs with your current kernel, you
> should be just able to read at 340mbyte/sec with most of the CPU idle.
>
> Andrea
__________________________
Simon Haynes - Baydel
Phone : 44 (0) 1372 378811
Email : simon@baydel.com
__________________________
^ permalink raw reply [flat|nested] 3+ messages in thread