Re: Slow Disk I/O with QPS M3 80GB HD

Re: Slow Disk I/O with QPS M3 80GB HD

[H . J . Lu]

On Mon, Dec 10, 2001 at 08:34:52PM -0800, H . J . Lu wrote:
> I have a very strange problem. The disk I/O of my QPS M3 80GB HD is
> very slow under 2.4.10 and above. I got like 1.77 MB/s from hdparm.
> But under 2.4.9, I got 14 MB/s on the same hardware. A 30GB HD has
> consistent I/O performance under 2.4.9 and above on the same bus. Has
> anyone else seen this? Does anyone have a large (>= 80GB) 1394 HD?
> 

I did a binary search. 2.4.10-pre10 is the last good kernel. I got

# hdparm -t /dev/sda

/dev/sda:
 Timing buffered disk reads:  64 MB in  4.40 seconds = 14.55 MB/sec

Even since 2.4.10-pre11 up to 2.4.16, I got about 1.77 MB/sec on the
same hardware. However, I don't have problems with 80GB IDE HD. Has
anyone seen I/O problems on large (>= 80GB) SCSI HD or HD with SCSI
emulation?


H.J.

Re: Slow Disk I/O with QPS M3 80GB HD

[Ben Collins]

On Mon, Dec 10, 2001 at 11:57:08PM -0800, H . J . Lu wrote:
> On Mon, Dec 10, 2001 at 08:34:52PM -0800, H . J . Lu wrote:
> > I have a very strange problem. The disk I/O of my QPS M3 80GB HD is
> > very slow under 2.4.10 and above. I got like 1.77 MB/s from hdparm.
> > But under 2.4.9, I got 14 MB/s on the same hardware. A 30GB HD has
> > consistent I/O performance under 2.4.9 and above on the same bus. Has
> > anyone else seen this? Does anyone have a large (>= 80GB) 1394 HD?
> > 
> 
> I did a binary search. 2.4.10-pre10 is the last good kernel. I got
> 
> # hdparm -t /dev/sda
> 
> /dev/sda:
>  Timing buffered disk reads:  64 MB in  4.40 seconds = 14.55 MB/sec

Have you checked the way that your ohci and sbp2 devices are detected
under each case? Most notably the max packet size.

Ben

-- 
 .----------=======-=-======-=========-----------=====------------=-=-----.
/                   Ben Collins    --    Debian GNU/Linux                  \
`  bcollins@debian.org  --  bcollins@openldap.org  --  bcollins@linux.com  '
 `---=========------=======-------------=-=-----=-===-======-------=--=---'

Re: Slow Disk I/O with QPS M3 80GB HD

[H . J . Lu]

On Tue, Dec 11, 2001 at 11:05:07AM -0500, Ben Collins wrote:
> On Mon, Dec 10, 2001 at 11:57:08PM -0800, H . J . Lu wrote:
> > On Mon, Dec 10, 2001 at 08:34:52PM -0800, H . J . Lu wrote:
> > > I have a very strange problem. The disk I/O of my QPS M3 80GB HD is
> > > very slow under 2.4.10 and above. I got like 1.77 MB/s from hdparm.
> > > But under 2.4.9, I got 14 MB/s on the same hardware. A 30GB HD has
> > > consistent I/O performance under 2.4.9 and above on the same bus. Has
> > > anyone else seen this? Does anyone have a large (>= 80GB) 1394 HD?
> > > 
> > 
> > I did a binary search. 2.4.10-pre10 is the last good kernel. I got
> > 
> > # hdparm -t /dev/sda
> > 
> > /dev/sda:
> >  Timing buffered disk reads:  64 MB in  4.40 seconds = 14.55 MB/sec
> 
> Have you checked the way that your ohci and sbp2 devices are detected
> under each case? Most notably the max packet size.
> 

They all say

ohci1394_0: OHCI-1394 1.0 (PCI): IRQ=[19]  MMIO=[f8ffd000-f8ffe000]  Max Packet=[1024]
ieee1394: sbp2: SBP-2 device max speed S200 and payload 1KB


H.J.

Re: Slow Disk I/O with QPS M3 80GB HD

[Ben Collins]

On Tue, Dec 11, 2001 at 08:45:52AM -0800, H . J . Lu wrote:
> On Tue, Dec 11, 2001 at 11:05:07AM -0500, Ben Collins wrote:
> > On Mon, Dec 10, 2001 at 11:57:08PM -0800, H . J . Lu wrote:
> > > On Mon, Dec 10, 2001 at 08:34:52PM -0800, H . J . Lu wrote:
> > > > I have a very strange problem. The disk I/O of my QPS M3 80GB HD is
> > > > very slow under 2.4.10 and above. I got like 1.77 MB/s from hdparm.
> > > > But under 2.4.9, I got 14 MB/s on the same hardware. A 30GB HD has
> > > > consistent I/O performance under 2.4.9 and above on the same bus. Has
> > > > anyone else seen this? Does anyone have a large (>= 80GB) 1394 HD?
> > > > 
> > > 
> > > I did a binary search. 2.4.10-pre10 is the last good kernel. I got
> > > 
> > > # hdparm -t /dev/sda
> > > 
> > > /dev/sda:
> > >  Timing buffered disk reads:  64 MB in  4.40 seconds = 14.55 MB/sec
> > 
> > Have you checked the way that your ohci and sbp2 devices are detected
> > under each case? Most notably the max packet size.
> > 
> 
> They all say
> 
> ohci1394_0: OHCI-1394 1.0 (PCI): IRQ=[19]  MMIO=[f8ffd000-f8ffe000]  Max Packet=[1024]
> ieee1394: sbp2: SBP-2 device max speed S200 and payload 1KB

Have you tried linux1394 CVS with a 2.4.10pre10 kernel to narrow down
where the slowdown has occured?

-- 
 .----------=======-=-======-=========-----------=====------------=-=-----.
/                   Ben Collins    --    Debian GNU/Linux                  \
`  bcollins@debian.org  --  bcollins@openldap.org  --  bcollins@linux.com  '
 `---=========------=======-------------=-=-----=-===-======-------=--=---'

Re: Slow Disk I/O with QPS M3 80GB HD

[H . J . Lu]

On Tue, Dec 11, 2001 at 12:05:06PM -0500, Ben Collins wrote:
> On Tue, Dec 11, 2001 at 08:45:52AM -0800, H . J . Lu wrote:
> > On Tue, Dec 11, 2001 at 11:05:07AM -0500, Ben Collins wrote:
> > > On Mon, Dec 10, 2001 at 11:57:08PM -0800, H . J . Lu wrote:
> > > > On Mon, Dec 10, 2001 at 08:34:52PM -0800, H . J . Lu wrote:
> > > > > I have a very strange problem. The disk I/O of my QPS M3 80GB HD is
> > > > > very slow under 2.4.10 and above. I got like 1.77 MB/s from hdparm.
> > > > > But under 2.4.9, I got 14 MB/s on the same hardware. A 30GB HD has
> > > > > consistent I/O performance under 2.4.9 and above on the same bus. Has
> > > > > anyone else seen this? Does anyone have a large (>= 80GB) 1394 HD?
> > > > > 
> > > > 
> > > > I did a binary search. 2.4.10-pre10 is the last good kernel. I got
> > > > 
> > > > # hdparm -t /dev/sda
> > > > 
> > > > /dev/sda:
> > > >  Timing buffered disk reads:  64 MB in  4.40 seconds = 14.55 MB/sec
> > > 
> > > Have you checked the way that your ohci and sbp2 devices are detected
> > > under each case? Most notably the max packet size.
> > > 
> > 
> > They all say
> > 
> > ohci1394_0: OHCI-1394 1.0 (PCI): IRQ=[19]  MMIO=[f8ffd000-f8ffe000]  Max Packet=[1024]
> > ieee1394: sbp2: SBP-2 device max speed S200 and payload 1KB
> 
> Have you tried linux1394 CVS with a 2.4.10pre10 kernel to narrow down
> where the slowdown has occured?

I don't think the 1394 driver is the problem. I have tried the 1394
driver from a good kernel. I got the same result.



H.J.

Re: Slow Disk I/O with QPS M3 80GB HD

[H . J . Lu]

On Mon, Dec 10, 2001 at 11:57:08PM -0800, H . J . Lu wrote:
> On Mon, Dec 10, 2001 at 08:34:52PM -0800, H . J . Lu wrote:
> > I have a very strange problem. The disk I/O of my QPS M3 80GB HD is
> > very slow under 2.4.10 and above. I got like 1.77 MB/s from hdparm.
> > But under 2.4.9, I got 14 MB/s on the same hardware. A 30GB HD has
> > consistent I/O performance under 2.4.9 and above on the same bus. Has
> > anyone else seen this? Does anyone have a large (>= 80GB) 1394 HD?
> > 
> 
> I did a binary search. 2.4.10-pre10 is the last good kernel. I got
> 
> # hdparm -t /dev/sda
> 
> /dev/sda:
>  Timing buffered disk reads:  64 MB in  4.40 seconds = 14.55 MB/sec
> 
> Even since 2.4.10-pre11 up to 2.4.16, I got about 1.77 MB/sec on the
> same hardware. However, I don't have problems with 80GB IDE HD. Has
> anyone seen I/O problems on large (>= 80GB) SCSI HD or HD with SCSI
> emulation?

I tracked own the problem to 40_blkdev-pagecache-17 in the 2.4.10
pre10aa1 patch. When it is applied, the disk I/O on some drives become
very slow. It not only happens to my 80GB 1394 HD, but also the second
IDE drive. Before the patch

# hdparm -t /dev/hdd

/dev/hdd:
 Timing buffered disk reads:  64 MB in  8.02 seconds =  7.98 MB/sec

After the patch

# hdparm -t /dev/hdd

/dev/hdd:
 Timing buffered disk reads:  64 MB in 21.09 seconds =  3.03 MB/sec

The slow down is not as bad as 1394. But it is still very significant.
I couldn't figure out why it only affects certain drives.


H.J.

Re: Slow Disk I/O with QPS M3 80GB HD

[Andrea Arcangeli]

On Tue, Dec 11, 2001 at 03:43:31PM -0800, H . J . Lu wrote:
> On Mon, Dec 10, 2001 at 11:57:08PM -0800, H . J . Lu wrote:
> > On Mon, Dec 10, 2001 at 08:34:52PM -0800, H . J . Lu wrote:
> > > I have a very strange problem. The disk I/O of my QPS M3 80GB HD is
> > > very slow under 2.4.10 and above. I got like 1.77 MB/s from hdparm.
> > > But under 2.4.9, I got 14 MB/s on the same hardware. A 30GB HD has
> > > consistent I/O performance under 2.4.9 and above on the same bus. Has
> > > anyone else seen this? Does anyone have a large (>= 80GB) 1394 HD?
> > > 
> > 
> > I did a binary search. 2.4.10-pre10 is the last good kernel. I got
> > 
> > # hdparm -t /dev/sda
> > 
> > /dev/sda:
> >  Timing buffered disk reads:  64 MB in  4.40 seconds = 14.55 MB/sec
> > 
> > Even since 2.4.10-pre11 up to 2.4.16, I got about 1.77 MB/sec on the
> > same hardware. However, I don't have problems with 80GB IDE HD. Has
> > anyone seen I/O problems on large (>= 80GB) SCSI HD or HD with SCSI
> > emulation?
> 
> I tracked own the problem to 40_blkdev-pagecache-17 in the 2.4.10
> pre10aa1 patch. When it is applied, the disk I/O on some drives become
> very slow. It not only happens to my 80GB 1394 HD, but also the second
> IDE drive. Before the patch
> 
> # hdparm -t /dev/hdd
> 
> /dev/hdd:
>  Timing buffered disk reads:  64 MB in  8.02 seconds =  7.98 MB/sec
> 
> After the patch
> 
> # hdparm -t /dev/hdd
> 
> /dev/hdd:
>  Timing buffered disk reads:  64 MB in 21.09 seconds =  3.03 MB/sec
> 
> The slow down is not as bad as 1394. But it is still very significant.
> I couldn't figure out why it only affects certain drives.

do you have a 4k filesystm mounted on /dev/hdd? if so then you will get
the same performance with latest 2.4 (precisely after 2.4.1).

Andrea

Re: Slow Disk I/O with QPS M3 80GB HD

[H . J . Lu]

On Wed, Dec 12, 2001 at 09:29:54AM +0100, Andrea Arcangeli wrote:
> On Tue, Dec 11, 2001 at 03:43:31PM -0800, H . J . Lu wrote:
> > On Mon, Dec 10, 2001 at 11:57:08PM -0800, H . J . Lu wrote:
> > > On Mon, Dec 10, 2001 at 08:34:52PM -0800, H . J . Lu wrote:
> > > > I have a very strange problem. The disk I/O of my QPS M3 80GB HD is
> > > > very slow under 2.4.10 and above. I got like 1.77 MB/s from hdparm.
> > > > But under 2.4.9, I got 14 MB/s on the same hardware. A 30GB HD has
> > > > consistent I/O performance under 2.4.9 and above on the same bus. Has
> > > > anyone else seen this? Does anyone have a large (>= 80GB) 1394 HD?
> > > > 
> > > 
> > > I did a binary search. 2.4.10-pre10 is the last good kernel. I got
> > > 
> > > # hdparm -t /dev/sda
> > > 
> > > /dev/sda:
> > >  Timing buffered disk reads:  64 MB in  4.40 seconds = 14.55 MB/sec
> > > 
> > > Even since 2.4.10-pre11 up to 2.4.16, I got about 1.77 MB/sec on the
> > > same hardware. However, I don't have problems with 80GB IDE HD. Has
> > > anyone seen I/O problems on large (>= 80GB) SCSI HD or HD with SCSI
> > > emulation?
> > 
> > I tracked own the problem to 40_blkdev-pagecache-17 in the 2.4.10
> > pre10aa1 patch. When it is applied, the disk I/O on some drives become
> > very slow. It not only happens to my 80GB 1394 HD, but also the second
> > IDE drive. Before the patch
> > 
> > # hdparm -t /dev/hdd
> > 
> > /dev/hdd:
> >  Timing buffered disk reads:  64 MB in  8.02 seconds =  7.98 MB/sec
> > 
> > After the patch
> > 
> > # hdparm -t /dev/hdd
> > 
> > /dev/hdd:
> >  Timing buffered disk reads:  64 MB in 21.09 seconds =  3.03 MB/sec
> > 
> > The slow down is not as bad as 1394. But it is still very significant.
> > I couldn't figure out why it only affects certain drives.
> 
> do you have a 4k filesystm mounted on /dev/hdd? if so then you will get
> the same performance with latest 2.4 (precisely after 2.4.1).
> 

Even Sony F707 digital camera is slowed because your change from
2.4.10-pre10 to 2.4.10-pre11. It is a USB memory stick. It is 10 times
slower than before. So far, we have some USB and 1394 storage devices
are slowed down very badly. Some IDE hds also suffer. It is very
strange.


H.J.
----
3 files on USB:

# ls -l
total 5984
-rwxr-xr-x    1 root     root      1991481 Dec  8 21:44 dsc00002.jpg
-rwxr-xr-x    1 root     root      2432127 Dec  8 23:54 dsc00005.jpg
-rwxr-xr-x    1 root     root      1675563 Dec  8 23:59 dsc00009.jpg

1. 2.4.10-pre10

# time /bin/cp * /tmp/
real    0m0.623s
user    0m0.000s
sys     0m0.100s

2. 2.4.10-pre10aa1

# time /bin/cp * /tmp/
real    0m8.952s
user    0m0.000s
sys     0m0.190s


3. 2.4.10-pre11

# time /bin/cp * /tmp/
real    0m8.851s
user    0m0.000s
sys     0m0.170s

Re: Slow Disk I/O with QPS M3 80GB HD

[Andrea Arcangeli]

On Wed, Dec 12, 2001 at 03:38:06PM -0800, H . J . Lu wrote:
> H.J.
> ----
> 3 files on USB:
> 
> # ls -l
> total 5984
> -rwxr-xr-x    1 root     root      1991481 Dec  8 21:44 dsc00002.jpg
> -rwxr-xr-x    1 root     root      2432127 Dec  8 23:54 dsc00005.jpg
> -rwxr-xr-x    1 root     root      1675563 Dec  8 23:59 dsc00009.jpg
> 
> 1. 2.4.10-pre10
> 
> # time /bin/cp * /tmp/
> real    0m0.623s
> user    0m0.000s
> sys     0m0.100s
> 
> 2. 2.4.10-pre10aa1
> 
> # time /bin/cp * /tmp/
> real    0m8.952s
> user    0m0.000s
> sys     0m0.190s
> 
> 
> 3. 2.4.10-pre11
> 
> # time /bin/cp * /tmp/
> real    0m8.851s
> user    0m0.000s
> sys     0m0.170s

but the above have nothing to do with the blkdev-pagecache.
Blkdev-pagecache can matter only with blkdev accesses, blkdev-pagecache
cannot make any difference if the I/O passes through any fs (not via the
block_dev layer) like in the above case.

Can you try to take those cp under strace timestamp and see what's the
syscall that blocks?

Andrea

Re: Slow Disk I/O with QPS M3 80GB HD

[H . J . Lu]

On Thu, Dec 13, 2001 at 12:49:33AM +0100, Andrea Arcangeli wrote:
> On Wed, Dec 12, 2001 at 03:38:06PM -0800, H . J . Lu wrote:
> > H.J.
> > ----
> > 3 files on USB:
> > 
> > # ls -l
> > total 5984
> > -rwxr-xr-x    1 root     root      1991481 Dec  8 21:44 dsc00002.jpg
> > -rwxr-xr-x    1 root     root      2432127 Dec  8 23:54 dsc00005.jpg
> > -rwxr-xr-x    1 root     root      1675563 Dec  8 23:59 dsc00009.jpg
> > 
> > 1. 2.4.10-pre10
> > 
> > # time /bin/cp * /tmp/
> > real    0m0.623s
> > user    0m0.000s
> > sys     0m0.100s
> > 
> > 2. 2.4.10-pre10aa1

I applied all the way to 40_blkdev-pagecache-17 in 2.4.10-pre10aa1.

> > 
> > # time /bin/cp * /tmp/
> > real    0m8.952s
> > user    0m0.000s
> > sys     0m0.190s
> > 
> > 
> > 3. 2.4.10-pre11
> > 
> > # time /bin/cp * /tmp/
> > real    0m8.851s
> > user    0m0.000s
> > sys     0m0.170s
> 
> but the above have nothing to do with the blkdev-pagecache.
> Blkdev-pagecache can matter only with blkdev accesses, blkdev-pagecache
> cannot make any difference if the I/O passes through any fs (not via the
> block_dev layer) like in the above case.

That is why I said it was strange.

> 
> Can you try to take those cp under strace timestamp and see what's the
> syscall that blocks?

It blocks on read.


H.J.