Re: Is Read speed faster when 1 disk is failed on raid5 ?

Re: Is Read speed faster when 1 disk is failed on raid5 ?

[raid]

On Tue, 22 Oct 2002, Chris Siebenmann wrote:
>  At a guess: when RAID-5 is intact, the kernel must read the entire
> stripe size worth of data to verify the parity. When a RAID-5 array has
> a dead disk, any intact data block can be handed to the application
> the moment it comes off the disk as there is no point in doing parity
> verification. (The kernel has to reconstruct the missing blocks, but
> most of the blocks are present and good.)

i thought raid5 was doing parity calculation when writing ?
difference between rading from failed an intact raid5 is almost twice the
speed (failed is doing 150MB/sec, intact is doing 80MB/sec)

Re: Is Read speed faster when 1 disk is failed on raid5 ?

[Jakob Oestergaard]

On Tue, Oct 22, 2002 at 11:58:56AM +0200, raid@ddx.a2000.nu wrote:
> On Tue, 22 Oct 2002, Chris Siebenmann wrote:
> >  At a guess: when RAID-5 is intact, the kernel must read the entire
> > stripe size worth of data to verify the parity. When a RAID-5 array has
> > a dead disk, any intact data block can be handed to the application
> > the moment it comes off the disk as there is no point in doing parity
> > verification. (The kernel has to reconstruct the missing blocks, but
> > most of the blocks are present and good.)
> 
> i thought raid5 was doing parity calculation when writing ?

Correct. RAID-5 will not do parity calculation when reading. RAID-5 will
give you availability but it will not give you integrity.

> difference between rading from failed an intact raid5 is almost twice the
> speed (failed is doing 150MB/sec, intact is doing 80MB/sec)

I didn't follow the beginning of this thread, sorry.  Is this SCSI?

-- 
................................................................
:   jakob@unthought.net   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Is Read speed faster when 1 disk is failed on raid5 ?

[raid]

On Tue, 22 Oct 2002, Jakob Oestergaard wrote:

> I didn't follow the beginning of this thread, sorry.  Is this SCSI?

Ide disks on 3ware 7850 using software raid5

Re: Is Read speed faster when 1 disk is failed on raid5 ?

[Jakob Oestergaard]

On Tue, Oct 22, 2002 at 12:49:41PM +0200, raid@ddx.a2000.nu wrote:
> 
> On Tue, 22 Oct 2002, Jakob Oestergaard wrote:
> 
> > I didn't follow the beginning of this thread, sorry.  Is this SCSI?
> 
> Ide disks on 3ware 7850 using software raid5

Ok.  Odd.

My first guess would be that it's some master/slave IDE issue - but you
are probably running all your disks as masters (one disk per channel),
right?

Other than that... Well, if the parity information is intact, the disks
would need to skip the parity blocks when the array is read from
sequentially.  With a degraded array, the reads are all contiguous on
the disks - this could be a difference perhaps??   

What chunk size are you using?  And can you try a chunk size that is an
order of magnitude bigger or smaller?   (might take some time to test
this out).

For example, if you use a 4k chunksize (I don't think you do, if my last
guess holds), then try 128k.  If you are using 64k or above (which,
again, if I am guessing correctly, is probably more like what you're
using), then try 4k.

-- 
................................................................
:   jakob@unthought.net   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Is Read speed faster when 1 disk is failed on raid5 ?

[Yiqiang Ding]

I had the similar result with IDE drives through hpt374 controller. Still
don't understand why a degraded system has better read performance.

YQ
----- Original Message -----
From: "Jakob Oestergaard" <jakob@unthought.net>
To: <raid@ddx.a2000.nu>
Cc: <linux-raid@vger.kernel.org>
Sent: Tuesday, October 22, 2002 3:24 AM
Subject: Re: Is Read speed faster when 1 disk is failed on raid5 ?


> On Tue, Oct 22, 2002 at 12:49:41PM +0200, raid@ddx.a2000.nu wrote:
> >
> > On Tue, 22 Oct 2002, Jakob Oestergaard wrote:
> >
> > > I didn't follow the beginning of this thread, sorry.  Is this SCSI?
> >
> > Ide disks on 3ware 7850 using software raid5
>
> Ok.  Odd.
>
> My first guess would be that it's some master/slave IDE issue - but you
> are probably running all your disks as masters (one disk per channel),
> right?
>
> Other than that... Well, if the parity information is intact, the disks
> would need to skip the parity blocks when the array is read from
> sequentially.  With a degraded array, the reads are all contiguous on
> the disks - this could be a difference perhaps??
>
> What chunk size are you using?  And can you try a chunk size that is an
> order of magnitude bigger or smaller?   (might take some time to test
> this out).
>
> For example, if you use a 4k chunksize (I don't think you do, if my last
> guess holds), then try 128k.  If you are using 64k or above (which,
> again, if I am guessing correctly, is probably more like what you're
> using), then try 4k.
>
> --
> ................................................................
> :   jakob@unthought.net   : And I see the elder races,         :
> :.........................: putrid forms of man                :
> :   Jakob Østergaard      : See him rise and claim the earth,  :
> :        OZ9ABN           : his downfall is at hand.           :
> :.........................:............{Konkhra}...............:
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>


-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Is Read speed faster when 1 disk is failed on raid5 ?

[Jakob Oestergaard]

On Mon, Oct 28, 2002 at 10:27:46AM -0800, Yiqiang Ding wrote:
> I had the similar result with IDE drives through hpt374 controller. Still
> don't understand why a degraded system has better read performance.

Could you comment on the guesses I made below, relating to the chunk
size?

> > Other than that... Well, if the parity information is intact, the disks
> > would need to skip the parity blocks when the array is read from
> > sequentially.  With a degraded array, the reads are all contiguous on
> > the disks - this could be a difference perhaps??
> >
> > What chunk size are you using?  And can you try a chunk size that is an
> > order of magnitude bigger or smaller?   (might take some time to test
> > this out).
> >
> > For example, if you use a 4k chunksize (I don't think you do, if my last
> > guess holds), then try 128k.  If you are using 64k or above (which,
> > again, if I am guessing correctly, is probably more like what you're
> > using), then try 4k.

-- 
................................................................
:   jakob@unthought.net   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Is Read speed faster when 1 disk is failed on raid5 ?

[Yiqiang Ding]

Hi Jakob,

I don't follow your guesses. Why do you think it may be related to chunk
size? Anyway, I'm using 32K.

yq
----- Original Message -----
From: "Jakob Oestergaard" <jakob@unthought.net>
To: "Yiqiang Ding" <yqding@rasilient.com>
Cc: <raid@ddx.a2000.nu>; <linux-raid@vger.kernel.org>
Sent: Monday, October 28, 2002 1:02 PM
Subject: Re: Is Read speed faster when 1 disk is failed on raid5 ?


> On Mon, Oct 28, 2002 at 10:27:46AM -0800, Yiqiang Ding wrote:
> > I had the similar result with IDE drives through hpt374 controller.
Still
> > don't understand why a degraded system has better read performance.
>
> Could you comment on the guesses I made below, relating to the chunk
> size?
>
> > > Other than that... Well, if the parity information is intact, the
disks
> > > would need to skip the parity blocks when the array is read from
> > > sequentially.  With a degraded array, the reads are all contiguous on
> > > the disks - this could be a difference perhaps??
> > >
> > > What chunk size are you using?  And can you try a chunk size that is
an
> > > order of magnitude bigger or smaller?   (might take some time to test
> > > this out).
> > >
> > > For example, if you use a 4k chunksize (I don't think you do, if my
last
> > > guess holds), then try 128k.  If you are using 64k or above (which,
> > > again, if I am guessing correctly, is probably more like what you're
> > > using), then try 4k.
>
> --
> ................................................................
> :   jakob@unthought.net   : And I see the elder races,         :
> :.........................: putrid forms of man                :
> :   Jakob Østergaard      : See him rise and claim the earth,  :
> :        OZ9ABN           : his downfall is at hand.           :
> :.........................:............{Konkhra}...............:
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>


-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Is Read speed faster when 1 disk is failed on raid5 ?

[Jakob Oestergaard]

On Mon, Oct 28, 2002 at 01:37:34PM -0800, Yiqiang Ding wrote:
> Hi Jakob,
> 
> I don't follow your guesses. Why do you think it may be related to chunk
> size? Anyway, I'm using 32K.

Because in RAID-5, each disk will hold blocks like:

Disk 0:  [parity] [data]   [data]   [parity]
Disk 1:  [data]   [parity] [data]   [data]
Disk 2:  [data]   [data]   [parity] [data]

So when reading blocks 0, 1, 2, 3, 4, ... from the array, we will do:

Read disk 1 block 0
Read disk 2 block 0
Read disk 0 block 1
Read disk 2 block 1
Read disk 0 block 2
Read disk 1 block 2
Read disk 1 block 3
Read disk 2 block 3

We can do read-ahead, but the access pattern for disk 0 is:

Block 1, block 2, block 4, ...

For disk 1:

Block 0, block 2, block 3, ...

etc...

So we introduce seeks, because of the parity blocks.

Seeking ruins performance.

In a degraded array, the kernel cannot skip the parity blocks, it must
use them for calculating the lost data.

So my guess is, that this "penalty" actually turns out to be an
optimization (if the chunk size is small - eg. the number of seeks
introduced is large). We will do strictly sequential reads on all disks.

So tell me, have I been smoking something, or does this make sense?  :)

Even better - measure degraded vs. non-degraded read performance on a
RAID-5 array, first with chunk-size 4k, then 32k, then 128k, and post
the results here   ;)

-- 
................................................................
:   jakob@unthought.net   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Is Read speed faster when 1 disk is failed on raid5 ?

[Yiqiang Ding]

Hi Jakob,

Thanks for your kind explanation. Sounds pretty reasonable. I also have done
some tests on raid5 with 4k and 128k chunk size. The results are as follows:
Access Spec     4K(MBps)        4K-deg(MBps)    128K(MBps)
128K-deg(MBps)
2K Seq Read     23.015089       33.293993       25.415035       32.669278
2K Seq Write    27.363041       30.555328       14.185889       16.087862
64K Seq Read    22.952559       44.414774       26.02711        44.036993
64K Seq Write   25.171833       32.67759        13.97861        15.618126

Some conclusions:
1. "Degraded" raid5 has better (sequential) read/write performances. The
biggest difference is in 64k sequential read, almost doubled.
2. Bigger chunk size makes less difference between non-degraded and degraded
RAID5. This is due to less seek penalty for bigger chunksize raid5 according
to Jakob's theory.
3. Bigger chunk size makes worse write performance. Why? Maybe somebody can
explain this.

YQ

----- Original Message -----
From: "Jakob Oestergaard" <jakob@unthought.net>
To: "Yiqiang Ding" <yqding@rasilient.com>
Cc: <raid@ddx.a2000.nu>; <linux-raid@vger.kernel.org>
Sent: Monday, October 28, 2002 4:30 PM
Subject: Re: Is Read speed faster when 1 disk is failed on raid5 ?


> On Mon, Oct 28, 2002 at 01:37:34PM -0800, Yiqiang Ding wrote:
> > Hi Jakob,
> >
> > I don't follow your guesses. Why do you think it may be related to chunk
> > size? Anyway, I'm using 32K.
>
> Because in RAID-5, each disk will hold blocks like:
>
> Disk 0:  [parity] [data]   [data]   [parity]
> Disk 1:  [data]   [parity] [data]   [data]
> Disk 2:  [data]   [data]   [parity] [data]
>
> So when reading blocks 0, 1, 2, 3, 4, ... from the array, we will do:
>
> Read disk 1 block 0
> Read disk 2 block 0
> Read disk 0 block 1
> Read disk 2 block 1
> Read disk 0 block 2
> Read disk 1 block 2
> Read disk 1 block 3
> Read disk 2 block 3
>
> We can do read-ahead, but the access pattern for disk 0 is:
>
> Block 1, block 2, block 4, ...
>
> For disk 1:
>
> Block 0, block 2, block 3, ...
>
> etc...
>
> So we introduce seeks, because of the parity blocks.
>
> Seeking ruins performance.
>
> In a degraded array, the kernel cannot skip the parity blocks, it must
> use them for calculating the lost data.
>
> So my guess is, that this "penalty" actually turns out to be an
> optimization (if the chunk size is small - eg. the number of seeks
> introduced is large). We will do strictly sequential reads on all disks.
>
> So tell me, have I been smoking something, or does this make sense?  :)
>
> Even better - measure degraded vs. non-degraded read performance on a
> RAID-5 array, first with chunk-size 4k, then 32k, then 128k, and post
> the results here   ;)
>
> --
> ................................................................
> :   jakob@unthought.net   : And I see the elder races,         :
> :.........................: putrid forms of man                :
> :   Jakob Østergaard      : See him rise and claim the earth,  :
> :        OZ9ABN           : his downfall is at hand.           :
> :.........................:............{Konkhra}...............:
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>


-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Is Read speed faster when 1 disk is failed on raid5 ?

[Jakob Oestergaard]

On Tue, Oct 29, 2002 at 01:05:59PM -0800, Yiqiang Ding wrote:
> Hi Jakob,
> 

Hello Ding,

> Thanks for your kind explanation. Sounds pretty reasonable. I also have done
> some tests on raid5 with 4k and 128k chunk size. The results are as follows:
> Access Spec     4K(MBps)        4K-deg(MBps)    128K(MBps)
> 128K-deg(MBps)
> 2K Seq Read     23.015089       33.293993       25.415035       32.669278
> 2K Seq Write    27.363041       30.555328       14.185889       16.087862
> 64K Seq Read    22.952559       44.414774       26.02711        44.036993
> 64K Seq Write   25.171833       32.67759        13.97861        15.618126

Very interesting !

> 
> Some conclusions:
> 1. "Degraded" raid5 has better (sequential) read/write performances. The
> biggest difference is in 64k sequential read, almost doubled.
> 2. Bigger chunk size makes less difference between non-degraded and degraded
> RAID5. This is due to less seek penalty for bigger chunksize raid5 according
> to Jakob's theory.
> 3. Bigger chunk size makes worse write performance. Why? Maybe somebody can
> explain this.

I'm going wild-guessing on (3) here...

It could be, that while you are writing your file, a write smaller than
your chunk size is scheduled by the VM (or something - I'm not exactly a
block/VM interaction wizard) - so a 128k parity block is written out.
Some time later, the rest of the parity block is scheduled for writing,
and the same but recalculated 128k parity block is written out once
again.

Neil, or anyone else with more kernel understanding than me, please
comment on that   :)

A work-around for this, as I see it, would be to change the RAID-5
driver so that it - during *writing* only - internally works on 512 byte
"sub-chunks" *no*matter* the actual chunk size on the array.

This does not break compatibility with existing RAIDs as I see it - no
additional information is needed in the superblock either. I think this
optimization could be done completely transparently.

I'd love to come up with a patch, but there's a zero likelihood of that
happening before the weekend.

-- 
................................................................
:   jakob@unthought.net   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 3ware 7500-12, bad write speed

[raid]

Ok did a benchmark between software raid en hw raid
controller 3ware 7850 (7500-8 now?) with 5 * WDC 120GB 7200RPM (2mb cache)
On xeon dual 2GHZ (controller in 64bits pci)

results :

---raid5 5*120gb sw raid :

Version 1.02c       ------Sequential Output------ --Sequential Input-
--Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
storagenew.a2000 1G 18850  99 78959  77 29982  29 24052  99 81547  22
510.5   2
                    ------Sequential Create------ --------Random
Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
/sec %CP
                 16  1933  97 +++++ +++ +++++ +++  1986  98 +++++ +++
5056  98
storagenew.a2000.nu,1G,18850,99,78959,77,29982,29,24052,99,81547,22,510.5,2,16,1933,97,+++++,+++,+++++,+++,1986,98,+++++,+++,5056,98


---raid5 5*120gb hw raid :

Version 1.02c       ------Sequential Output------ --Sequential Input-
--Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
storagenew.a2000 1G 17100  85 32250  23 12053   5 22660  91 49603  10
466.9   1
                    ------Sequential Create------ --------Random
Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
/sec %CP
                 16  1872  93 +++++ +++ +++++ +++  1976  98 +++++ +++
5677 100
storagenew.a2000.nu,1G,17100,85,32250,23,12053,5,22660,91,49603,10,466.9,1,16,1872,93,+++++,+++,+++++,+++,1976,98,+++++,+++,5677,100

Is Read speed faster when 1 disk is failed on raid5 ?

[raid]

1 disk is failed (just for test)
then i did an benchmark
write is the same, but read is almost twice the speed
why is this ?

Version 1.02c       ------Sequential Output------ --Sequential Input-
--Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
storagenew.a2000 1G 18671  99 80063  77 31862  38 23361  99 149929  83
372.6   2
                    ------Sequential Create------ --------Random
Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
/sec %CP
                 16  1898  94 +++++ +++ +++++ +++  1962  99 +++++ +++
5210  97
storagenew.a2000.nu,1G,18671,99,80063,77,31862,38,23361,99,149929,83,372.6,2,16,1898,94,+++++,+++,+++++,+++,1962,99,+++++,+++,5210,97

On Thu, 10 Oct 2002 raid@ddx.a2000.nu wrote:

> Ok did a benchmark between software raid en hw raid
> controller 3ware 7850 (7500-8 now?) with 5 * WDC 120GB 7200RPM (2mb cache)
> On xeon dual 2GHZ (controller in 64bits pci)
>
> results :
>
> ---raid5 5*120gb sw raid :
>
> Version 1.02c       ------Sequential Output------ --Sequential Input-
> --Random-
>                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
> --Seeks--
> Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
> /sec %CP
> storagenew.a2000 1G 18850  99 78959  77 29982  29 24052  99 81547  22
> 510.5   2
>                     ------Sequential Create------ --------Random
> Create--------
>                     -Create-- --Read--- -Delete-- -Create-- --Read---
> -Delete--
>               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
> /sec %CP
>                  16  1933  97 +++++ +++ +++++ +++  1986  98 +++++ +++
> 5056  98
> storagenew.a2000.nu,1G,18850,99,78959,77,29982,29,24052,99,81547,22,510.5,2,16,1933,97,+++++,+++,+++++,+++,1986,98,+++++,+++,5056,98
>
>

Re: Is Read speed faster when 1 disk is failed on raid5 ?

[raid]

did the same test with 4 disks in raid5

4 disks raid5 synced :

Version 1.02c       ------Sequential Output------ --Sequential Input-
--Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
storagenew.a2000 1G 18654  99 64992  63 24127  25 23504  97 57140  17
480.7   2
                    ------Sequential Create------ --------Random
Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
/sec %CP
                 16  1915  96 +++++ +++ +++++ +++  2002  98 +++++ +++
5422  98
storagenew.a2000.nu,1G,18654,99,64992,63,24127,25,23504,97,57140,17,480.7,2,16,1915,96,+++++,+++,+++++,+++,2002,98,+++++,+++,5422,98


raid5 4disks, 1 failed :

Version 1.02c       ------Sequential Output------ --Sequential Input-
--Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
storagenew.a2000 1G 18566  99 58434  57 27743  35 23089  98 119752  56
332.1   1
                    ------Sequential Create------ --------Random
Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
/sec %CP
                 16  1942  97 +++++ +++ +++++ +++  1978  99 +++++ +++
5585  99
storagenew.a2000.nu,1G,18566,99,58434,57,27743,35,23089,98,119752,56,332.1,1,16,1942,97,+++++,+++,+++++,+++,1978,99,+++++,+++,5585,99