[linux-lvm] cache buffer size

[linux-lvm] cache buffer size

[Miwako Nakai]

I'm using jfs on lvm/raid5 with snapshot.

When I copy file, it happens frequently
'raid5: switching cache buffer size XXX -> XXX'.
This message appears to syslog. It caused speed down to access disks.
So, I cut this message. It's seemed ok.

But it's still going to switch cache buffer size. I think it cost
some cpu time. When switching buffer cache size, it always happen
reflushing memories.

I want to fix buffer size. What shall I do?
I asked JFS group. But they do nothing about cache buffer.

kernel 2.4.20-pre10-ac2
lvm    1.0.5
jfs    1.0.23

--
nakai@ezinc.com

Re: [linux-lvm] cache buffer size

[Heinz J . Mauelshagen]

On Fri, Nov 01, 2002 at 12:07:40PM +0900, Miwako Nakai wrote:
> I'm using jfs on lvm/raid5 with snapshot.
> 
> When I copy file, it happens frequently
> 'raid5: switching cache buffer size XXX -> XXX'.
> This message appears to syslog. It caused speed down to access disks.
> So, I cut this message. It's seemed ok.

Could be filesystems with different blocksizes accessing the underlying
raid5 md through LVM constantly forcing the raid5 stripe cache to be flushed.
If that's the case, you want to put filesystems of the _same_ blocksize
onto all of the logical volumes allocated to the raid5 device.

Regards,
Heinz    -- The LVM Guy --

> 
> But it's still going to switch cache buffer size. I think it cost
> some cpu time. When switching buffer cache size, it always happen
> reflushing memories.
> 
> I want to fix buffer size. What shall I do?
> I asked JFS group. But they do nothing about cache buffer.
> 
> kernel 2.4.20-pre10-ac2
> lvm    1.0.5
> jfs    1.0.23
> 
> --
> nakai@ezinc.com
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

*** Software bugs are stupid.
    Nevertheless it needs not so stupid people to solve them ***

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Heinz Mauelshagen                                 Sistina Software Inc.
Senior Consultant/Developer                       Am Sonnenhang 11
                                                  56242 Marienrachdorf
                                                  Germany
Mauelshagen@Sistina.com                           +49 2626 141200
                                                       FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Re: [linux-lvm] cache buffer size

[Miwako Nakai]

Thank you, Mr.Mauelshagen

JFS can not have fixed block size. But I heard from JFS group in IBM
that on 2.5 or later it will be improved. I do not know JFS will have
fixed block size or not. They said 'improved'.

I think LVM(2?) will be able to handle non-consitant block size on
2.5 well.

"Heinz J . Mauelshagen" wrote:
> 
> On Fri, Nov 01, 2002 at 12:07:40PM +0900, Miwako Nakai wrote:
> > I'm using jfs on lvm/raid5 with snapshot.
> >
> > When I copy file, it happens frequently
> > 'raid5: switching cache buffer size XXX -> XXX'.
> > This message appears to syslog. It caused speed down to access disks.
> > So, I cut this message. It's seemed ok.
> 
> Could be filesystems with different blocksizes accessing the underlying
> raid5 md through LVM constantly forcing the raid5 stripe cache to be flushed.
> If that's the case, you want to put filesystems of the _same_ blocksize
> onto all of the logical volumes allocated to the raid5 device.
> 
> Regards,
> Heinz    -- The LVM Guy --
> 
> >
> > But it's still going to switch cache buffer size. I think it cost
> > some cpu time. When switching buffer cache size, it always happen
> > reflushing memories.
> >
> > I want to fix buffer size. What shall I do?
> > I asked JFS group. But they do nothing about cache buffer.
> >
> > kernel 2.4.20-pre10-ac2
> > lvm    1.0.5
> > jfs    1.0.23
> >
> > --
> > nakai@ezinc.com
> 
> *** Software bugs are stupid.
>     Nevertheless it needs not so stupid people to solve them ***
> 
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> 
> Heinz Mauelshagen                                 Sistina Software Inc.
> Senior Consultant/Developer                       Am Sonnenhang 11
>                                                   56242 Marienrachdorf
>                                                   Germany
> Mauelshagen@Sistina.com                           +49 2626 141200
>                                                        FAX 924446
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

--
nakai@ezinc.com

Re: [linux-lvm] cache buffer size

[Heinz J . Mauelshagen]

On Wed, Nov 06, 2002 at 08:28:06AM +0900, Miwako Nakai wrote:
> Thank you, Mr.Mauelshagen
> 
> JFS can not have fixed block size. But I heard from JFS group in IBM
> that on 2.5 or later it will be improved. I do not know JFS will have
> fixed block size or not. They said 'improved'.

Didn't know that as a JFS no-user :)
Fixed block sizes is a suboptimal solution for the various io
needs of a filesystem (think small metedata and large data io).

If they do io with varying blocksizes, that would explain the MD RAID5
behaviour already. No way to avoid it in your combination for now then
I guess.


> 
> I think LVM(2?) will be able to handle non-consitant block size on
> 2.5 well.

Yes.

Regards,
Heinz    -- The LVM Guy --


> 
> "Heinz J . Mauelshagen" wrote:
> > 
> > On Fri, Nov 01, 2002 at 12:07:40PM +0900, Miwako Nakai wrote:
> > > I'm using jfs on lvm/raid5 with snapshot.
> > >
> > > When I copy file, it happens frequently
> > > 'raid5: switching cache buffer size XXX -> XXX'.
> > > This message appears to syslog. It caused speed down to access disks.
> > > So, I cut this message. It's seemed ok.
> > 
> > Could be filesystems with different blocksizes accessing the underlying
> > raid5 md through LVM constantly forcing the raid5 stripe cache to be flushed.
> > If that's the case, you want to put filesystems of the _same_ blocksize
> > onto all of the logical volumes allocated to the raid5 device.
> > 
> > Regards,
> > Heinz    -- The LVM Guy --
> > 
> > >
> > > But it's still going to switch cache buffer size. I think it cost
> > > some cpu time. When switching buffer cache size, it always happen
> > > reflushing memories.
> > >
> > > I want to fix buffer size. What shall I do?
> > > I asked JFS group. But they do nothing about cache buffer.
> > >
> > > kernel 2.4.20-pre10-ac2
> > > lvm    1.0.5
> > > jfs    1.0.23
> > >
> > > --
> > > nakai@ezinc.com
> --
> nakai@ezinc.com
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

*** Software bugs are stupid.
    Nevertheless it needs not so stupid people to solve them ***

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Heinz Mauelshagen                                 Sistina Software Inc.
Senior Consultant/Developer                       Am Sonnenhang 11
                                                  56242 Marienrachdorf
                                                  Germany
Mauelshagen@Sistina.com                           +49 2626 141200
                                                       FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Re: [linux-lvm] cache buffer size

[Steve Best]

Miwako Nakai wrote:
>Thank you, Mr.Mauelshagen
>
>JFS can not have fixed block size. But I heard from JFS group in IBM
>that on 2.5 or later it will be improved. I do not know JFS will have
>fixed block size or not. They said 'improved'.

This isn't right, JFS block size is fixed at 4k. Do you only have this
problem if you snapshot? Could you explain some more about your setup.
Are all of your LVs JFS?

Heinz is it possible that the 1k is coming from the snapshot?

>
>I think LVM(2?) will be able to handle non-consitant block size on
>2.5 well.

>"Heinz J . Mauelshagen" wrote:
>>
>> On Fri, Nov 01, 2002 at 12:07:40PM +0900, Miwako Nakai wrote:
>> > I'm using jfs on lvm/raid5 with snapshot.
>> >
>> > When I copy file, it happens frequently
>> > 'raid5: switching cache buffer size XXX -> XXX'.
>> > This message appears to syslog. It caused speed down to access disks.
>> > So, I cut this message. It's seemed ok.
>>
>> Could be filesystems with different blocksizes accessing the underlying
>> raid5 md through LVM constantly forcing the raid5 stripe cache to be
flushed.
>> If that's the case, you want to put filesystems of the _same_ blocksize
>> onto all of the logical volumes allocated to the raid5 device.
>>
>> Regards,
>> Heinz    -- The LVM Guy --
>>
>> >
>> > But it's still going to switch cache buffer size. I think it cost
>> > some cpu time. When switching buffer cache size, it always happen
>> > reflushing memories.
>> >
>> > I want to fix buffer size. What shall I do?
>> > I asked JFS group. But they do nothing about cache buffer.
>> >
>> > kernel 2.4.20-pre10-ac2
>> > lvm    1.0.5
>> > jfs    1.0.23
>> >
>> > --
>> > nakai@ezinc.com
>>

Steve
JFS for Linux http://oss.software.ibm.com/jfs

Re: [linux-lvm] cache buffer size

[Miwako Nakai]

Thank you Mr.Best

Steve Best wrote:
> 
> Miwako Nakai wrote:
> >Thank you, Mr.Mauelshagen
> >
> >JFS can not have fixed block size. But I heard from JFS group in IBM
> >that on 2.5 or later it will be improved. I do not know JFS will have
> >fixed block size or not. They said 'improved'.
> 
> This isn't right, JFS block size is fixed at 4k. Do you only have this
> problem if you snapshot? Could you explain some more about your setup.
> Are all of your LVs JFS?

YES. It happens only when using snapshot. Does snapshot use 1k block
size?

--
nakai@ezinc.com

Re: [linux-lvm] cache buffer size

[Heinz J . Mauelshagen]

On Wed, Nov 06, 2002 at 11:17:05AM -0600, Steve Best wrote:
> 
> Miwako Nakai wrote:
> >Thank you, Mr.Mauelshagen
> >
> >JFS can not have fixed block size. But I heard from JFS group in IBM
> >that on 2.5 or later it will be improved. I do not know JFS will have
> >fixed block size or not. They said 'improved'.
> 
> This isn't right, JFS block size is fixed at 4k. Do you only have this
> problem if you snapshot? Could you explain some more about your setup.
> Are all of your LVs JFS?
> 
> Heinz is it possible that the 1k is coming from the snapshot?

You pointed to the right spot, Steve :)

Indeed snapshot exception table ios go by blocks of 1k.

Puting the snapshot store on the same device as the original is not
recommended in general for performance reasons.

In case 1k blocks are possible with JFS:

if snapshots are used regularly and there's no seperate PV to put the
snapshot onto, you might want to create a 1k blocked jfs and try if
that performs better than the RAID 5 stripe cache flush penalty with the 4k
blocked filesystem.

Regards,
Heinz    -- The LVM Guy --


> 
> >
> >I think LVM(2?) will be able to handle non-consitant block size on
> >2.5 well.
> 
> >"Heinz J . Mauelshagen" wrote:
> >>
> >> On Fri, Nov 01, 2002 at 12:07:40PM +0900, Miwako Nakai wrote:
> >> > I'm using jfs on lvm/raid5 with snapshot.
> >> >
> >> > When I copy file, it happens frequently
> >> > 'raid5: switching cache buffer size XXX -> XXX'.
> >> > This message appears to syslog. It caused speed down to access disks.
> >> > So, I cut this message. It's seemed ok.
> >>
> >> Could be filesystems with different blocksizes accessing the underlying
> >> raid5 md through LVM constantly forcing the raid5 stripe cache to be
> flushed.
> >> If that's the case, you want to put filesystems of the _same_ blocksize
> >> onto all of the logical volumes allocated to the raid5 device.
> >>
> >> Regards,
> >> Heinz    -- The LVM Guy --
> >>
> >> >
> >> > But it's still going to switch cache buffer size. I think it cost
> >> > some cpu time. When switching buffer cache size, it always happen
> >> > reflushing memories.
> >> >
> >> > I want to fix buffer size. What shall I do?
> >> > I asked JFS group. But they do nothing about cache buffer.
> >> >
> >> > kernel 2.4.20-pre10-ac2
> >> > lvm    1.0.5
> >> > jfs    1.0.23
> >> >
> >> > --
> >> > nakai@ezinc.com
> >>
> 
> Steve
> JFS for Linux http://oss.software.ibm.com/jfs
> 
> 
> 
> 
> 
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

*** Software bugs are stupid.
    Nevertheless it needs not so stupid people to solve them ***

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Heinz Mauelshagen                                 Sistina Software Inc.
Senior Consultant/Developer                       Am Sonnenhang 11
                                                  56242 Marienrachdorf
                                                  Germany
Mauelshagen@Sistina.com                           +49 2626 141200
                                                       FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Re: [linux-lvm] cache buffer size

[Luca Berra]

On Thu, Nov 07, 2002 at 05:58:01PM +0100, Heinz J . Mauelshagen wrote:
>On Wed, Nov 06, 2002 at 11:17:05AM -0600, Steve Best wrote:
>Indeed snapshot exception table ios go by blocks of 1k.
>
>Puting the snapshot store on the same device as the original is not
>recommended in general for performance reasons.
>
>In case 1k blocks are possible with JFS:
>
>if snapshots are used regularly and there's no seperate PV to put the
>snapshot onto, you might want to create a 1k blocked jfs and try if
>that performs better than the RAID 5 stripe cache flush penalty with the 4k
>blocked filesystem.

hello,
is there any chance of having snapshots with 4k blocks for the exception
table (configurable), i want to use snapshot for backing up my root vg
but i also have swap there and it is locked to do page-sized io (4k on
ix86).

L.

-- 
Luca Berra -- bluca@comedia.it
        Communication Media & Services S.r.l.
 /"\
 \ /     ASCII RIBBON CAMPAIGN
  X        AGAINST HTML MAIL
 / \

Re: [linux-lvm] cache buffer size

[Heinz J . Mauelshagen]

On Fri, Nov 08, 2002 at 09:11:42AM +0100, Luca Berra wrote:
> On Thu, Nov 07, 2002 at 05:58:01PM +0100, Heinz J . Mauelshagen wrote:
> >On Wed, Nov 06, 2002 at 11:17:05AM -0600, Steve Best wrote:
> >Indeed snapshot exception table ios go by blocks of 1k.
> >
> >Puting the snapshot store on the same device as the original is not
> >recommended in general for performance reasons.
> >
> >In case 1k blocks are possible with JFS:
> >
> >if snapshots are used regularly and there's no seperate PV to put the
> >snapshot onto, you might want to create a 1k blocked jfs and try if
> >that performs better than the RAID 5 stripe cache flush penalty with the 4k
> >blocked filesystem.
> 
> hello,
> is there any chance of having snapshots with 4k blocks for the exception
> table (configurable), i want to use snapshot for backing up my root vg
> but i also have swap there and it is locked to do page-sized io (4k on
> ix86).

I guess no, because the snapshot io block size is retrieved from
blksize_size[][] for the underlying physical volume.  Because of that we
have 1024 byte for the typical disk drive unless on zSeries.


> 
> L.
> 
> -- 
> Luca Berra -- bluca@comedia.it
>         Communication Media & Services S.r.l.
>  /"\
>  \ /     ASCII RIBBON CAMPAIGN
>   X        AGAINST HTML MAIL
>  / \
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

-- 

Regards,
Heinz    -- The LVM Guy --

*** Software bugs are stupid.
    Nevertheless it needs not so stupid people to solve them ***

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Heinz Mauelshagen                                 Sistina Software Inc.
Senior Consultant/Developer                       Am Sonnenhang 11
                                                  56242 Marienrachdorf
                                                  Germany
Mauelshagen@Sistina.com                           +49 2626 141200
                                                       FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-