[Bug 41552] New: Performance of writing and reading from multiple drives decreases by 40% when going from Linux Kernel 2.6.36.4 to 2.6.37 (and beyond)

[Bug 41552] New: Performance of writing and reading from multiple drives decreases by 40% when going from Linux Kernel 2.6.36.4 to 2.6.37 (and beyond)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=41552

           Summary: Performance of writing and reading from multiple
                    drives decreases by 40% when going from Linux Kernel
                    2.6.36.4 to 2.6.37 (and beyond)
           Product: IO/Storage
           Version: 2.5
    Kernel Version: 2.6.37
          Platform: All
        OS/Version: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: SCSI
        AssignedTo: linux-scsi@vger.kernel.org
        ReportedBy: mpete_06@hotmail.com
        Regression: No


We have an application that will write and read from every sector on a drive. 
The application can perform these tasks on multiple drives at the same time. 
It is designed to run on top of the Linux Kernel, which we periodically update
so that we can get the latest device drivers.  When performing the last update
from 2.6.33.2 to 2.6.37, we found that the performance of a set of drives
decreased by some 40% (took 3 hours and 11 minutes to write and read from 5
drives on 2.6.37 versus 2 hours and 12 minutes on 2.6.33.3).  I was able to
determine that the issue was in the 2.6.37 Kernel as I was able to run it with
the 2.6.36.4 kernel, and it had the better performance.   After seeing that I/O
throttling was introduced in the 2.6.37 Kernel, I naturally suspected that. 
However, by default, all the throttling was turned off (I attached the actual
.config that was used to build the kernel).  I then tried to turn on the
throttling and set it to a high number to see what would happen.  When I did
that, I was able to reduce the time from 3 hours and 11 minutes to 2 hours and
50 minutes.  There seems to be something there that changed that is impacting
performance on multiple drives.  When we do this same test with only one drive,
the performance is identical between the systems.  This issue still occurs on
Kernel 3.0.2.

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 41552] Performance of writing and reading from multiple drives decreases by 40% when going from Linux Kernel 2.6.36.4 to 2.6.37 (and beyond)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=41552





--- Comment #1 from Mark Petersen <mpete_06@hotmail.com>  2011-08-22 15:21:31 ---
Created an attachment (id=69622)
 --> (https://bugzilla.kernel.org/attachment.cgi?id=69622)
The .config used to build the 2.6.37 Kernel

This is the .config used to build the 2.6.37 Kernel

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 41552] Performance of writing and reading from multiple drives decreases by 40% when going from Linux Kernel 2.6.36.4 to 2.6.37 (and beyond)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=41552





--- Comment #2 from Mark Petersen <mpete_06@hotmail.com>  2011-08-22 15:22:04 ---
Created an attachment (id=69632)
 --> (https://bugzilla.kernel.org/attachment.cgi?id=69632)
The log of system where performance was good

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 41552] Performance of writing and reading from multiple drives decreases by 40% when going from Linux Kernel 2.6.36.4 to 2.6.37 (and beyond)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=41552





--- Comment #3 from Mark Petersen <mpete_06@hotmail.com>  2011-08-22 15:22:47 ---
Created an attachment (id=69642)
 --> (https://bugzilla.kernel.org/attachment.cgi?id=69642)
The log of system where performance was degraded.

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 41552] Performance of writing and reading from multiple drives decreases by 40% when going from Linux Kernel 2.6.36.4 to 2.6.37 (and beyond)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=41552


Andrew Morton <akpm@linux-foundation.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |akpm@linux-foundation.org
         Regression|No                          |Yes




-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 41552] Performance of writing and reading from multiple drives decreases by 40% when going from Linux Kernel 2.6.36.4 to 2.6.37 (and beyond)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=41552





--- Comment #4 from Andrew Morton <akpm@linux-foundation.org>  2011-08-22 19:25:22 ---
(switched to email.  Please respond via emailed reply-to-all, not via the
bugzilla web interface).

On Mon, 22 Aug 2011 15:20:41 GMT
bugzilla-daemon@bugzilla.kernel.org wrote:

> https://bugzilla.kernel.org/show_bug.cgi?id=41552
> 
>            Summary: Performance of writing and reading from multiple
>                     drives decreases by 40% when going from Linux Kernel
>                     2.6.36.4 to 2.6.37 (and beyond)
>            Product: IO/Storage
>            Version: 2.5
>     Kernel Version: 2.6.37
>           Platform: All
>         OS/Version: Linux
>               Tree: Mainline
>             Status: NEW
>           Severity: normal
>           Priority: P1
>          Component: SCSI
>         AssignedTo: linux-scsi@vger.kernel.org
>         ReportedBy: mpete_06@hotmail.com
>         Regression: No
> 
> 
> We have an application that will write and read from every sector on a drive. 
> The application can perform these tasks on multiple drives at the same time. 
> It is designed to run on top of the Linux Kernel, which we periodically update
> so that we can get the latest device drivers.  When performing the last update
> from 2.6.33.2 to 2.6.37, we found that the performance of a set of drives
> decreased by some 40% (took 3 hours and 11 minutes to write and read from 5
> drives on 2.6.37 versus 2 hours and 12 minutes on 2.6.33.3).  I was able to
> determine that the issue was in the 2.6.37 Kernel as I was able to run it with
> the 2.6.36.4 kernel, and it had the better performance.   After seeing that I/O
> throttling was introduced in the 2.6.37 Kernel, I naturally suspected that. 
> However, by default, all the throttling was turned off (I attached the actual
> .config that was used to build the kernel).  I then tried to turn on the
> throttling and set it to a high number to see what would happen.  When I did
> that, I was able to reduce the time from 3 hours and 11 minutes to 2 hours and
> 50 minutes.  There seems to be something there that changed that is impacting
> performance on multiple drives.  When we do this same test with only one drive,
> the performance is identical between the systems.  This issue still occurs on
> Kernel 3.0.2.
> 

Are you able to determine whether this regression is due to slower
reading, to slower writing or to both?

Thanks.

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 41552] Performance of writing and reading from multiple drives decreases by 40% when going from Linux Kernel 2.6.36.4 to 2.6.37 (and beyond)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=41552





--- Comment #5 from Anonymous Emailer <anonymous@kernel-bugs.osdl.org>  2011-08-22 20:21:13 ---
Reply-To: vgoyal@redhat.com

On Mon, Aug 22, 2011 at 02:49:56PM -0500, Mark Petersen wrote:
> 
> The majority of the slowdown we found is coming during the writing as we were doing limited reading for the purpose of the testing.  It may be that it happens in both areas, but we did not do extensive testing with the reading portion of it.

What kind of writes these are? Write slowdown by 40%. Somehow now a days
barriers/flush/fua comes to my mind. Any changes there w.r.t your setup?

Recently Jeff moyer and Mike Snitzer had discovered and fixed a slowdown
in a dm-multipath and disks not having write caches. I guess that's not
your setup. Though mentioning it does not harm.

Thanks
Vivek


> 
> > Date: Mon, 22 Aug 2011 12:24:43 -0700
> > From: akpm@linux-foundation.org
> > To: mpete_06@hotmail.com
> > CC: bugme-daemon@bugzilla.kernel.org; axboe@kernel.dk; vgoyal@redhat.com; linux-mm@kvack.org; linux-scsi@vger.kernel.org
> > Subject: Re: [Bugme-new] [Bug 41552] New: Performance of writing and reading from multiple drives decreases by 40% when going from Linux Kernel 2.6.36.4 to 2.6.37 (and beyond)
> > 
> > 
> > (switched to email.  Please respond via emailed reply-to-all, not via the
> > bugzilla web interface).
> > 
> > On Mon, 22 Aug 2011 15:20:41 GMT
> > bugzilla-daemon@bugzilla.kernel.org wrote:
> > 
> > > https://bugzilla.kernel.org/show_bug.cgi?id=41552
> > > 
> > >            Summary: Performance of writing and reading from multiple
> > >                     drives decreases by 40% when going from Linux Kernel
> > >                     2.6.36.4 to 2.6.37 (and beyond)
> > >            Product: IO/Storage
> > >            Version: 2.5
> > >     Kernel Version: 2.6.37
> > >           Platform: All
> > >         OS/Version: Linux
> > >               Tree: Mainline
> > >             Status: NEW
> > >           Severity: normal
> > >           Priority: P1
> > >          Component: SCSI
> > >         AssignedTo: linux-scsi@vger.kernel.org
> > >         ReportedBy: mpete_06@hotmail.com
> > >         Regression: No
> > > 
> > > 
> > > We have an application that will write and read from every sector on a drive. 
> > > The application can perform these tasks on multiple drives at the same time. 
> > > It is designed to run on top of the Linux Kernel, which we periodically update
> > > so that we can get the latest device drivers.  When performing the last update
> > > from 2.6.33.2 to 2.6.37, we found that the performance of a set of drives
> > > decreased by some 40% (took 3 hours and 11 minutes to write and read from 5
> > > drives on 2.6.37 versus 2 hours and 12 minutes on 2.6.33.3).  I was able to
> > > determine that the issue was in the 2.6.37 Kernel as I was able to run it with
> > > the 2.6.36.4 kernel, and it had the better performance.   After seeing that I/O
> > > throttling was introduced in the 2.6.37 Kernel, I naturally suspected that. 
> > > However, by default, all the throttling was turned off (I attached the actual
> > > .config that was used to build the kernel).  I then tried to turn on the
> > > throttling and set it to a high number to see what would happen.  When I did
> > > that, I was able to reduce the time from 3 hours and 11 minutes to 2 hours and
> > > 50 minutes.  There seems to be something there that changed that is impacting
> > > performance on multiple drives.  When we do this same test with only one drive,
> > > the performance is identical between the systems.  This issue still occurs on
> > > Kernel 3.0.2.
> > > 
> > 
> > Are you able to determine whether this regression is due to slower
> > reading, to slower writing or to both?
> > 
> > Thanks.
>

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 41552] Performance of writing and reading from multiple drives decreases by 40% when going from Linux Kernel 2.6.36.4 to 2.6.37 (and beyond)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=41552





--- Comment #6 from Anonymous Emailer <anonymous@kernel-bugs.osdl.org>  2011-08-22 20:21:35 ---
Reply-To: vgoyal@redhat.com

On Mon, Aug 22, 2011 at 12:24:43PM -0700, Andrew Morton wrote:
> 
> (switched to email.  Please respond via emailed reply-to-all, not via the
> bugzilla web interface).
> 
> On Mon, 22 Aug 2011 15:20:41 GMT
> bugzilla-daemon@bugzilla.kernel.org wrote:
> 
> > https://bugzilla.kernel.org/show_bug.cgi?id=41552
> > 
> >            Summary: Performance of writing and reading from multiple
> >                     drives decreases by 40% when going from Linux Kernel
> >                     2.6.36.4 to 2.6.37 (and beyond)
> >            Product: IO/Storage
> >            Version: 2.5
> >     Kernel Version: 2.6.37
> >           Platform: All
> >         OS/Version: Linux
> >               Tree: Mainline
> >             Status: NEW
> >           Severity: normal
> >           Priority: P1
> >          Component: SCSI
> >         AssignedTo: linux-scsi@vger.kernel.org
> >         ReportedBy: mpete_06@hotmail.com
> >         Regression: No
> > 
> > 
> > We have an application that will write and read from every sector on a drive. 
> > The application can perform these tasks on multiple drives at the same time. 
> > It is designed to run on top of the Linux Kernel, which we periodically update
> > so that we can get the latest device drivers.  When performing the last update
> > from 2.6.33.2 to 2.6.37, we found that the performance of a set of drives
> > decreased by some 40% (took 3 hours and 11 minutes to write and read from 5
> > drives on 2.6.37 versus 2 hours and 12 minutes on 2.6.33.3).  I was able to
> > determine that the issue was in the 2.6.37 Kernel as I was able to run it with
> > the 2.6.36.4 kernel, and it had the better performance.   After seeing that I/O
> > throttling was introduced in the 2.6.37 Kernel, I naturally suspected that. 
> > However, by default, all the throttling was turned off (I attached the actual
> > .config that was used to build the kernel).  I then tried to turn on the
> > throttling and set it to a high number to see what would happen.  When I did
> > that, I was able to reduce the time from 3 hours and 11 minutes to 2 hours and
> > 50 minutes.  There seems to be something there that changed that is impacting
> > performance on multiple drives.  When we do this same test with only one drive,
> > the performance is identical between the systems.  This issue still occurs on
> > Kernel 3.0.2.
> > 
> 
> Are you able to determine whether this regression is due to slower
> reading, to slower writing or to both?

Mark,

As your initial comment says that you see 40% regression even when block
throttling infrastructure is not enabled, I think it is not related to
throttling as blk_throtl_bio() is null when BLK_DEV_THROTTLING=n.

What IO scheduler are you using? Can you try switching IO scheduler to
deadline and see if regression is still there. Trying to figure out if
it has anything to do with IO scheduler.

What file system are you using with what options? Are you using device
mapper to create some special configuration on multiple disks?

Also can you take a trace (blktrace) of any of the disks for 30 seconds
both without regression and after regression and upload it somewhere.
Staring at it might give some clues. 

Thanks
Vivek

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 41552] Performance of writing and reading from multiple drives decreases by 40% when going from Linux Kernel 2.6.36.4 to 2.6.37 (and beyond)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=41552





--- Comment #7 from Mark Petersen <mpete_06@hotmail.com>  2011-08-22 20:24:20 ---
The majority of the slowdown we found is coming during the writing as we were
doing limited reading for the purpose of the testing.  It may be that it
happens in both areas, but we did not do extensive testing with the reading
portion of it.

> Date: Mon, 22 Aug 2011 12:24:43 -0700
> From: akpm@linux-foundation.org
> To: mpete_06@hotmail.com
> CC: bugme-daemon@bugzilla.kernel.org; axboe@kernel.dk; vgoyal@redhat.com; linux-mm@kvack.org; linux-scsi@vger.kernel.org
> Subject: Re: [Bugme-new] [Bug 41552] New: Performance of writing and reading from multiple drives decreases by 40% when going from Linux Kernel 2.6.36.4 to 2.6.37 (and beyond)
> 
> 
> (switched to email.  Please respond via emailed reply-to-all, not via the
> bugzilla web interface).
> 
> On Mon, 22 Aug 2011 15:20:41 GMT
> bugzilla-daemon@bugzilla.kernel.org wrote:
> 
> > https://bugzilla.kernel.org/show_bug.cgi?id=41552
> > 
> >            Summary: Performance of writing and reading from multiple
> >                     drives decreases by 40% when going from Linux Kernel
> >                     2.6.36.4 to 2.6.37 (and beyond)
> >            Product: IO/Storage
> >            Version: 2.5
> >     Kernel Version: 2.6.37
> >           Platform: All
> >         OS/Version: Linux
> >               Tree: Mainline
> >             Status: NEW
> >           Severity: normal
> >           Priority: P1
> >          Component: SCSI
> >         AssignedTo: linux-scsi@vger.kernel.org
> >         ReportedBy: mpete_06@hotmail.com
> >         Regression: No
> > 
> > 
> > We have an application that will write and read from every sector on a drive. 
> > The application can perform these tasks on multiple drives at the same time. 
> > It is designed to run on top of the Linux Kernel, which we periodically update
> > so that we can get the latest device drivers.  When performing the last update
> > from 2.6.33.2 to 2.6.37, we found that the performance of a set of drives
> > decreased by some 40% (took 3 hours and 11 minutes to write and read from 5
> > drives on 2.6.37 versus 2 hours and 12 minutes on 2.6.33.3).  I was able to
> > determine that the issue was in the 2.6.37 Kernel as I was able to run it with
> > the 2.6.36.4 kernel, and it had the better performance.   After seeing that I/O
> > throttling was introduced in the 2.6.37 Kernel, I naturally suspected that. 
> > However, by default, all the throttling was turned off (I attached the actual
> > .config that was used to build the kernel).  I then tried to turn on the
> > throttling and set it to a high number to see what would happen.  When I did
> > that, I was able to reduce the time from 3 hours and 11 minutes to 2 hours and
> > 50 minutes.  There seems to be something there that changed that is impacting
> > performance on multiple drives.  When we do this same test with only one drive,
> > the performance is identical between the systems.  This issue still occurs on
> > Kernel 3.0.2.
> > 
> 
> Are you able to determine whether this regression is due to slower
> reading, to slower writing or to both?
> 
> Thanks.

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 41552] Performance of writing and reading from multiple drives decreases by 40% when going from Linux Kernel 2.6.36.4 to 2.6.37 (and beyond)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=41552





--- Comment #8 from Mark Petersen <mpete_06@hotmail.com>  2011-08-22 21:02:29 ---
The writes we are performing are SCSI commands directly to the device, one
sector at a time.  The only thing we changed between our updates was the Kernel
itself, which we leave everything in there at its default value if it is
enabled (we disable a great many things we don't need).  The latest version I
tried that still showed the issue was v3.0.1.

Thanks,
Mark

> Date: Mon, 22 Aug 2011 15:56:51 -0400
> From: vgoyal@redhat.com
> To: mpete_06@hotmail.com
> CC: akpm@linux-foundation.org; bugme-daemon@bugzilla.kernel.org; axboe@kernel.dk; linux-mm@kvack.org; linux-scsi@vger.kernel.org
> Subject: Re: [Bugme-new] [Bug 41552] New: Performance of writing and reading from multiple drives decreases by 40% when going from Linux Kernel 2.6.36.4 to 2.6.37 (and beyond)
> 
> On Mon, Aug 22, 2011 at 02:49:56PM -0500, Mark Petersen wrote:
> > 
> > The majority of the slowdown we found is coming during the writing as we were doing limited reading for the purpose of the testing.  It may be that it happens in both areas, but we did not do extensive testing with the reading portion of it.
> 
> What kind of writes these are? Write slowdown by 40%. Somehow now a days
> barriers/flush/fua comes to my mind. Any changes there w.r.t your setup?
> 
> Recently Jeff moyer and Mike Snitzer had discovered and fixed a slowdown
> in a dm-multipath and disks not having write caches. I guess that's not
> your setup. Though mentioning it does not harm.
> 
> Thanks
> Vivek
> 
>  
> > 
> > > Date: Mon, 22 Aug 2011 12:24:43 -0700
> > > From: akpm@linux-foundation.org
> > > To: mpete_06@hotmail.com
> > > CC: bugme-daemon@bugzilla.kernel.org; axboe@kernel.dk; vgoyal@redhat.com; linux-mm@kvack.org; linux-scsi@vger.kernel.org
> > > Subject: Re: [Bugme-new] [Bug 41552] New: Performance of writing and reading from multiple drives decreases by 40% when going from Linux Kernel 2.6.36.4 to 2.6.37 (and beyond)
> > > 
> > > 
> > > (switched to email.  Please respond via emailed reply-to-all, not via the
> > > bugzilla web interface).
> > > 
> > > On Mon, 22 Aug 2011 15:20:41 GMT
> > > bugzilla-daemon@bugzilla.kernel.org wrote:
> > > 
> > > > https://bugzilla.kernel.org/show_bug.cgi?id=41552
> > > > 
> > > >            Summary: Performance of writing and reading from multiple
> > > >                     drives decreases by 40% when going from Linux Kernel
> > > >                     2.6.36.4 to 2.6.37 (and beyond)
> > > >            Product: IO/Storage
> > > >            Version: 2.5
> > > >     Kernel Version: 2.6.37
> > > >           Platform: All
> > > >         OS/Version: Linux
> > > >               Tree: Mainline
> > > >             Status: NEW
> > > >           Severity: normal
> > > >           Priority: P1
> > > >          Component: SCSI
> > > >         AssignedTo: linux-scsi@vger.kernel.org
> > > >         ReportedBy: mpete_06@hotmail.com
> > > >         Regression: No
> > > > 
> > > > 
> > > > We have an application that will write and read from every sector on a drive. 
> > > > The application can perform these tasks on multiple drives at the same time. 
> > > > It is designed to run on top of the Linux Kernel, which we periodically update
> > > > so that we can get the latest device drivers.  When performing the last update
> > > > from 2.6.33.2 to 2.6.37, we found that the performance of a set of drives
> > > > decreased by some 40% (took 3 hours and 11 minutes to write and read from 5
> > > > drives on 2.6.37 versus 2 hours and 12 minutes on 2.6.33.3).  I was able to
> > > > determine that the issue was in the 2.6.37 Kernel as I was able to run it with
> > > > the 2.6.36.4 kernel, and it had the better performance.   After seeing that I/O
> > > > throttling was introduced in the 2.6.37 Kernel, I naturally suspected that. 
> > > > However, by default, all the throttling was turned off (I attached the actual
> > > > .config that was used to build the kernel).  I then tried to turn on the
> > > > throttling and set it to a high number to see what would happen.  When I did
> > > > that, I was able to reduce the time from 3 hours and 11 minutes to 2 hours and
> > > > 50 minutes.  There seems to be something there that changed that is impacting
> > > > performance on multiple drives.  When we do this same test with only one drive,
> > > > the performance is identical between the systems.  This issue still occurs on
> > > > Kernel 3.0.2.
> > > > 
> > > 
> > > Are you able to determine whether this regression is due to slower
> > > reading, to slower writing or to both?
> > > 
> > > Thanks.
> >

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 41552] Performance of writing and reading from multiple drives decreases by 40% when going from Linux Kernel 2.6.36.4 to 2.6.37 (and beyond)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=41552





--- Comment #9 from Mark Petersen <mpete_06@hotmail.com>  2011-08-22 21:07:03 ---
We were using the default CFQ scheduler.  I will change it to deadline and see
what happens.  Also, we are not using a file system to perform the writes,
rather we are sending SCSI commands directly to the devices, nor are we doing
anything special to the disks with a device mapper.  We simply write to each
one on a different thread one sector at a time.

I will attempt to get the trace and will add it if I can.

Thanks,
Mark

> Date: Mon, 22 Aug 2011 15:48:54 -0400
> From: vgoyal@redhat.com
> To: mpete_06@hotmail.com
> CC: bugme-daemon@bugzilla.kernel.org; axboe@kernel.dk; linux-mm@kvack.org; linux-scsi@vger.kernel.org; akpm@linux-foundation.org
> Subject: Re: [Bugme-new] [Bug 41552] New: Performance of writing and reading from multiple drives decreases by 40% when going from Linux Kernel 2.6.36.4 to 2.6.37 (and beyond)
> 
> On Mon, Aug 22, 2011 at 12:24:43PM -0700, Andrew Morton wrote:
> > 
> > (switched to email.  Please respond via emailed reply-to-all, not via the
> > bugzilla web interface).
> > 
> > On Mon, 22 Aug 2011 15:20:41 GMT
> > bugzilla-daemon@bugzilla.kernel.org wrote:
> > 
> > > https://bugzilla.kernel.org/show_bug.cgi?id=41552
> > > 
> > >            Summary: Performance of writing and reading from multiple
> > >                     drives decreases by 40% when going from Linux Kernel
> > >                     2.6.36.4 to 2.6.37 (and beyond)
> > >            Product: IO/Storage
> > >            Version: 2.5
> > >     Kernel Version: 2.6.37
> > >           Platform: All
> > >         OS/Version: Linux
> > >               Tree: Mainline
> > >             Status: NEW
> > >           Severity: normal
> > >           Priority: P1
> > >          Component: SCSI
> > >         AssignedTo: linux-scsi@vger.kernel.org
> > >         ReportedBy: mpete_06@hotmail.com
> > >         Regression: No
> > > 
> > > 
> > > We have an application that will write and read from every sector on a drive. 
> > > The application can perform these tasks on multiple drives at the same time. 
> > > It is designed to run on top of the Linux Kernel, which we periodically update
> > > so that we can get the latest device drivers.  When performing the last update
> > > from 2.6.33.2 to 2.6.37, we found that the performance of a set of drives
> > > decreased by some 40% (took 3 hours and 11 minutes to write and read from 5
> > > drives on 2.6.37 versus 2 hours and 12 minutes on 2.6.33.3).  I was able to
> > > determine that the issue was in the 2.6.37 Kernel as I was able to run it with
> > > the 2.6.36.4 kernel, and it had the better performance.   After seeing that I/O
> > > throttling was introduced in the 2.6.37 Kernel, I naturally suspected that. 
> > > However, by default, all the throttling was turned off (I attached the actual
> > > .config that was used to build the kernel).  I then tried to turn on the
> > > throttling and set it to a high number to see what would happen.  When I did
> > > that, I was able to reduce the time from 3 hours and 11 minutes to 2 hours and
> > > 50 minutes.  There seems to be something there that changed that is impacting
> > > performance on multiple drives.  When we do this same test with only one drive,
> > > the performance is identical between the systems.  This issue still occurs on
> > > Kernel 3.0.2.
> > > 
> > 
> > Are you able to determine whether this regression is due to slower
> > reading, to slower writing or to both?
> 
> Mark,
> 
> As your initial comment says that you see 40% regression even when block
> throttling infrastructure is not enabled, I think it is not related to
> throttling as blk_throtl_bio() is null when BLK_DEV_THROTTLING=n.
> 
> What IO scheduler are you using? Can you try switching IO scheduler to
> deadline and see if regression is still there. Trying to figure out if
> it has anything to do with IO scheduler.
> 
> What file system are you using with what options? Are you using device
> mapper to create some special configuration on multiple disks?
> 
> Also can you take a trace (blktrace) of any of the disks for 30 seconds
> both without regression and after regression and upload it somewhere.
> Staring at it might give some clues. 
> 
> Thanks
> Vivek

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

Re: [Bug 41552] Performance of writing and reading from multiple drives decreases by 40% when going from Linux Kernel 2.6.36.4 to 2.6.37 (and beyond)

[James Bottomley]

On Mon, 2011-08-22 at 21:07 +0000, bugzilla-daemon@bugzilla.kernel.org
wrote:
> We were using the default CFQ scheduler.  I will change it to deadline and see
> what happens.  Also, we are not using a file system to perform the writes,
> rather we are sending SCSI commands directly to the devices, nor are we doing
> anything special to the disks with a device mapper.  We simply write to each
> one on a different thread one sector at a time.
> 
> I will attempt to get the trace and will add it if I can.

If you're just using a SG_IO type scsi command, then virtually nothing
we've done in block/scsi should affect this; certainly not the elevators
since we don't do merging on SG_IO, we just use them for queueing.
Since you say "write to each one on a different thread", it's possible
that some scheduler change may have impacted this.

James

[Bug 41552] Performance of writing and reading from multiple drives decreases by 40% when going from Linux Kernel 2.6.36.4 to 2.6.37 (and beyond)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=41552





--- Comment #10 from Anonymous Emailer <anonymous@kernel-bugs.osdl.org>  2011-08-23 14:38:52 ---
Reply-To: James.Bottomley@HansenPartnership.com

On Mon, 2011-08-22 at 21:07 +0000, bugzilla-daemon@bugzilla.kernel.org
wrote:
> We were using the default CFQ scheduler.  I will change it to deadline and see
> what happens.  Also, we are not using a file system to perform the writes,
> rather we are sending SCSI commands directly to the devices, nor are we doing
> anything special to the disks with a device mapper.  We simply write to each
> one on a different thread one sector at a time.
> 
> I will attempt to get the trace and will add it if I can.

If you're just using a SG_IO type scsi command, then virtually nothing
we've done in block/scsi should affect this; certainly not the elevators
since we don't do merging on SG_IO, we just use them for queueing.
Since you say "write to each one on a different thread", it's possible
that some scheduler change may have impacted this.

James

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 41552] Performance of writing and reading from multiple drives decreases by 40% when going from Linux Kernel 2.6.36.4 to 2.6.37 (and beyond)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=41552





--- Comment #11 from Mark Petersen <mpete_06@hotmail.com>  2011-08-24 20:46:38 ---
I was finally able to run it with the deadline scheduler, and got the same
performance.  Unfortunately, I am not able to use the blktrace tool as it
requires a version of libc that we do not have on the system (we have 2.5 and
it requires at least 2.7).  Is there anything else I can use to trace it?

Thanks,
Mark

> Date: Mon, 22 Aug 2011 15:48:54 -0400
> From: vgoyal@redhat.com
> To: mpete_06@hotmail.com
> CC: bugme-daemon@bugzilla.kernel.org; axboe@kernel.dk; linux-mm@kvack.org; linux-scsi@vger.kernel.org; akpm@linux-foundation.org
> Subject: Re: [Bugme-new] [Bug 41552] New: Performance of writing and reading from multiple drives decreases by 40% when going from Linux Kernel 2.6.36.4 to 2.6.37 (and beyond)
> 
> On Mon, Aug 22, 2011 at 12:24:43PM -0700, Andrew Morton wrote:
> > 
> > (switched to email.  Please respond via emailed reply-to-all, not via the
> > bugzilla web interface).
> > 
> > On Mon, 22 Aug 2011 15:20:41 GMT
> > bugzilla-daemon@bugzilla.kernel.org wrote:
> > 
> > > https://bugzilla.kernel.org/show_bug.cgi?id=41552
> > > 
> > >            Summary: Performance of writing and reading from multiple
> > >                     drives decreases by 40% when going from Linux Kernel
> > >                     2.6.36.4 to 2.6.37 (and beyond)
> > >            Product: IO/Storage
> > >            Version: 2.5
> > >     Kernel Version: 2.6.37
> > >           Platform: All
> > >         OS/Version: Linux
> > >               Tree: Mainline
> > >             Status: NEW
> > >           Severity: normal
> > >           Priority: P1
> > >          Component: SCSI
> > >         AssignedTo: linux-scsi@vger.kernel.org
> > >         ReportedBy: mpete_06@hotmail.com
> > >         Regression: No
> > > 
> > > 
> > > We have an application that will write and read from every sector on a drive. 
> > > The application can perform these tasks on multiple drives at the same time. 
> > > It is designed to run on top of the Linux Kernel, which we periodically update
> > > so that we can get the latest device drivers.  When performing the last update
> > > from 2.6.33.2 to 2.6.37, we found that the performance of a set of drives
> > > decreased by some 40% (took 3 hours and 11 minutes to write and read from 5
> > > drives on 2.6.37 versus 2 hours and 12 minutes on 2.6.33.3).  I was able to
> > > determine that the issue was in the 2.6.37 Kernel as I was able to run it with
> > > the 2.6.36.4 kernel, and it had the better performance.   After seeing that I/O
> > > throttling was introduced in the 2.6.37 Kernel, I naturally suspected that. 
> > > However, by default, all the throttling was turned off (I attached the actual
> > > .config that was used to build the kernel).  I then tried to turn on the
> > > throttling and set it to a high number to see what would happen.  When I did
> > > that, I was able to reduce the time from 3 hours and 11 minutes to 2 hours and
> > > 50 minutes.  There seems to be something there that changed that is impacting
> > > performance on multiple drives.  When we do this same test with only one drive,
> > > the performance is identical between the systems.  This issue still occurs on
> > > Kernel 3.0.2.
> > > 
> > 
> > Are you able to determine whether this regression is due to slower
> > reading, to slower writing or to both?
> 
> Mark,
> 
> As your initial comment says that you see 40% regression even when block
> throttling infrastructure is not enabled, I think it is not related to
> throttling as blk_throtl_bio() is null when BLK_DEV_THROTTLING=n.
> 
> What IO scheduler are you using? Can you try switching IO scheduler to
> deadline and see if regression is still there. Trying to figure out if
> it has anything to do with IO scheduler.
> 
> What file system are you using with what options? Are you using device
> mapper to create some special configuration on multiple disks?
> 
> Also can you take a trace (blktrace) of any of the disks for 30 seconds
> both without regression and after regression and upload it somewhere.
> Staring at it might give some clues. 
> 
> Thanks
> Vivek

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 41552] Performance of writing and reading from multiple drives decreases by 40% when going from Linux Kernel 2.6.36.4 to 2.6.37 (and beyond)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=41552





--- Comment #12 from Anonymous Emailer <anonymous@kernel-bugs.osdl.org>  2011-08-25 21:02:55 ---
Reply-To: vgoyal@redhat.com

On Wed, Aug 24, 2011 at 03:11:57PM -0500, Mark Petersen wrote:
> 
> I was finally able to run it with the deadline scheduler, and got the same performance.

You mean you see 40% regression even with deadline? If yes, then it is not a
IO scheduler specific issue.

> Unfortunately, I am not able to use the blktrace tool as it requires a version of libc that we do not have on the system (we have 2.5 and it requires at least 2.7).  Is there anything else I can use to trace it?
> 

You can try using tracing functionality. 

- mount -t debugfs none /sys/kernel/debug
- Enable tracing on the disk you are doing IO to.
  echo 1 > /sys/block/sda/trace/enable
- Enable block traces
  echo blk > /sys/kernel/debug/tracing/current_tracer
- cat /sys/kernel/debug/tracing/trace_pipe > /tmp/trace_output

Let it run for few seconds. Interrupt and kill cat process.
/tmp/trace_output should have useful tracing info.

Thanks
Vivek


> Thanks,
> Mark
> 
> > Date: Mon, 22 Aug 2011 15:48:54 -0400
> > From: vgoyal@redhat.com
> > To: mpete_06@hotmail.com
> > CC: bugme-daemon@bugzilla.kernel.org; axboe@kernel.dk; linux-mm@kvack.org; linux-scsi@vger.kernel.org; akpm@linux-foundation.org
> > Subject: Re: [Bugme-new] [Bug 41552] New: Performance of writing and reading from multiple drives decreases by 40% when going from Linux Kernel 2.6.36.4 to 2.6.37 (and beyond)
> > 
> > On Mon, Aug 22, 2011 at 12:24:43PM -0700, Andrew Morton wrote:
> > > 
> > > (switched to email.  Please respond via emailed reply-to-all, not via the
> > > bugzilla web interface).
> > > 
> > > On Mon, 22 Aug 2011 15:20:41 GMT
> > > bugzilla-daemon@bugzilla.kernel.org wrote:
> > > 
> > > > https://bugzilla.kernel.org/show_bug.cgi?id=41552
> > > > 
> > > >            Summary: Performance of writing and reading from multiple
> > > >                     drives decreases by 40% when going from Linux Kernel
> > > >                     2.6.36.4 to 2.6.37 (and beyond)
> > > >            Product: IO/Storage
> > > >            Version: 2.5
> > > >     Kernel Version: 2.6.37
> > > >           Platform: All
> > > >         OS/Version: Linux
> > > >               Tree: Mainline
> > > >             Status: NEW
> > > >           Severity: normal
> > > >           Priority: P1
> > > >          Component: SCSI
> > > >         AssignedTo: linux-scsi@vger.kernel.org
> > > >         ReportedBy: mpete_06@hotmail.com
> > > >         Regression: No
> > > > 
> > > > 
> > > > We have an application that will write and read from every sector on a drive. 
> > > > The application can perform these tasks on multiple drives at the same time. 
> > > > It is designed to run on top of the Linux Kernel, which we periodically update
> > > > so that we can get the latest device drivers.  When performing the last update
> > > > from 2.6.33.2 to 2.6.37, we found that the performance of a set of drives
> > > > decreased by some 40% (took 3 hours and 11 minutes to write and read from 5
> > > > drives on 2.6.37 versus 2 hours and 12 minutes on 2.6.33.3).  I was able to
> > > > determine that the issue was in the 2.6.37 Kernel as I was able to run it with
> > > > the 2.6.36.4 kernel, and it had the better performance.   After seeing that I/O
> > > > throttling was introduced in the 2.6.37 Kernel, I naturally suspected that. 
> > > > However, by default, all the throttling was turned off (I attached the actual
> > > > .config that was used to build the kernel).  I then tried to turn on the
> > > > throttling and set it to a high number to see what would happen.  When I did
> > > > that, I was able to reduce the time from 3 hours and 11 minutes to 2 hours and
> > > > 50 minutes.  There seems to be something there that changed that is impacting
> > > > performance on multiple drives.  When we do this same test with only one drive,
> > > > the performance is identical between the systems.  This issue still occurs on
> > > > Kernel 3.0.2.
> > > > 
> > > 
> > > Are you able to determine whether this regression is due to slower
> > > reading, to slower writing or to both?
> > 
> > Mark,
> > 
> > As your initial comment says that you see 40% regression even when block
> > throttling infrastructure is not enabled, I think it is not related to
> > throttling as blk_throtl_bio() is null when BLK_DEV_THROTTLING=n.
> > 
> > What IO scheduler are you using? Can you try switching IO scheduler to
> > deadline and see if regression is still there. Trying to figure out if
> > it has anything to do with IO scheduler.
> > 
> > What file system are you using with what options? Are you using device
> > mapper to create some special configuration on multiple disks?
> > 
> > Also can you take a trace (blktrace) of any of the disks for 30 seconds
> > both without regression and after regression and upload it somewhere.
> > Staring at it might give some clues. 
> > 
> > Thanks
> > Vivek
>

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 41552] Performance of writing and reading from multiple drives decreases by 40% when going from Linux Kernel 2.6.36.4 to 2.6.37 (and beyond)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=41552


Alan <alan@lxorguk.ukuu.org.uk> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
                 CC|                            |alan@lxorguk.ukuu.org.uk
         Resolution|                            |OBSOLETE




-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.