RE: Postal 56% waits for flock_lock_file_wait

RE: Postal 56% waits for flock_lock_file_wait

[Ananiev, Leonid I]

> The kernel would appear to be doing exactly what is expected of it.

Each of 16 user threads calls to open() one of 1000 files each 20 sec.
3000 calls per minute in sum.
The open() sleeps.
I'm not sure that users expected just of sleeping.

Leonid

-----Original Message-----
From: Trond Myklebust [mailto:trond.myklebust@fys.uio.no] 
Sent: Sunday, October 01, 2006 8:05 AM
To: Ananiev, Leonid I
Cc: Linux Kernel Mailing List
Subject: RE: Postal 56% waits for flock_lock_file_wait

On Sat, 2006-09-30 at 21:26 +0400, Ananiev, Leonid I wrote:
> > On which filesystem were the above results obtained if it was not
> ext2?
> The default ext3 fs was used.
> 
> > All the above results are telling you is that your test involves
> several
> > processes contending for the same lock, and so all of them barring
the
> > one process that actually holds the lock are idle.
> 
> Yes. It is  flock_lock_file_wait.

That is the function which causes the sleep, yes. So what is your gripe?
The kernel would appear to be doing exactly what is expected of it.

Trond

RE: Postal 56% waits for flock_lock_file_wait

[Trond Myklebust]

On Sun, 2006-10-01 at 20:53 +0400, Ananiev, Leonid I wrote:
> > The kernel would appear to be doing exactly what is expected of it.
> 
> Each of 16 user threads calls to open() one of 1000 files each 20 sec.
> 3000 calls per minute in sum.
> The open() sleeps.

According to your numbers, the call to flock() sleeps. Where do they
show a measurable sleep in open()?

> I'm not sure that users expected just of sleeping.

I still don't get it. The job of the flock() system call is to sleep if
someone already holds the lock, and then grab the lock when it is
released. If that is not what the user expects, then the user has the
option of not calling flock(). This has nothing to do with open().

Trond


> Leonid
> 
> -----Original Message-----
> From: Trond Myklebust [mailto:trond.myklebust@fys.uio.no] 
> Sent: Sunday, October 01, 2006 8:05 AM
> To: Ananiev, Leonid I
> Cc: Linux Kernel Mailing List
> Subject: RE: Postal 56% waits for flock_lock_file_wait
> 
> On Sat, 2006-09-30 at 21:26 +0400, Ananiev, Leonid I wrote:
> > > On which filesystem were the above results obtained if it was not
> > ext2?
> > The default ext3 fs was used.
> > 
> > > All the above results are telling you is that your test involves
> > several
> > > processes contending for the same lock, and so all of them barring
> the
> > > one process that actually holds the lock are idle.
> > 
> > Yes. It is  flock_lock_file_wait.
> 
> That is the function which causes the sleep, yes. So what is your gripe?
> The kernel would appear to be doing exactly what is expected of it.
> 
> Trond

Re: Postal 56% waits for flock_lock_file_wait

[Tim Chen]

On 10/1/06, Trond Myklebust <trond.myklebust@fys.uio.no> wrote:

>
> I still don't get it. The job of the flock() system call is to sleep if
> someone already holds the lock, and then grab the lock when it is
> released. If that is not what the user expects, then the user has the
> option of not calling flock(). This has nothing to do with open().
>
> Trond
>

If I understand Leonid correctly, I think what he is saying is ext3
does not scale very well when you have a large number of processes
acessing file system because of locks in journal.    This is seen in
the excessive idle time.  By comparison, ext2 does not have this
issue.

Tim

Re: Postal 56% waits for flock_lock_file_wait

[Trond Myklebust]

On Mon, 2006-10-02 at 09:57 -0700, Tim Chen wrote:
> On 10/1/06, Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> 
> >
> > I still don't get it. The job of the flock() system call is to sleep if
> > someone already holds the lock, and then grab the lock when it is
> > released. If that is not what the user expects, then the user has the
> > option of not calling flock(). This has nothing to do with open().
> >
> > Trond
> >
> 
> If I understand Leonid correctly, I think what he is saying is ext3
> does not scale very well when you have a large number of processes
> acessing file system because of locks in journal.    This is seen in
> the excessive idle time.  By comparison, ext2 does not have this
> issue.

Ext3 does not use flock() in order to lock its journal. The performance
issues that he is seeing may well be due to the journalling, but that
has nothing to do with flock_lock_file_wait.

Cheers,
  Trond

Re: Postal 56% waits for flock_lock_file_wait

[Alan Cox]

Ar Llu, 2006-10-02 am 13:11 -0400, ysgrifennodd Trond Myklebust:
> Ext3 does not use flock() in order to lock its journal. The performance
> issues that he is seeing may well be due to the journalling, but that
> has nothing to do with flock_lock_file_wait.

The ext3 journal also generally speaking improves many-writer
performance as do the reservations so the claim seems odd on that basis
too. Rerun the test on a gigabyte iRam or similar and you'll see where
the non-media bottlenecks actually are

Re: Postal 56% waits for flock_lock_file_wait

[Dave Jones]

On Mon, Oct 02, 2006 at 06:51:56PM +0100, Alan Cox wrote:
 > Ar Llu, 2006-10-02 am 13:11 -0400, ysgrifennodd Trond Myklebust:
 > > Ext3 does not use flock() in order to lock its journal. The performance
 > > issues that he is seeing may well be due to the journalling, but that
 > > has nothing to do with flock_lock_file_wait.
 > 
 > The ext3 journal also generally speaking improves many-writer
 > performance as do the reservations so the claim seems odd on that basis
 > too. Rerun the test on a gigabyte iRam or similar and you'll see where
 > the non-media bottlenecks actually are

"or similar" maybe. The iRam is pretty much junk in my experience[*].
It rarely survives a mkfs, let alone sustained high throughput I/O.
(And yes, I did try multiple DIMMs, including ones which survive
 memtest86 just fine).

Another "Boots Windows, ship it" QA disaster afaics.

	Dave

[*] And from googling/talking with other owners, my experiences aren't unique.

-- 
http://www.codemonkey.org.uk

Re: Postal 56% waits for flock_lock_file_wait

[Alan Cox]

Ar Llu, 2006-10-02 am 13:40 -0400, ysgrifennodd Dave Jones:
> On Mon, Oct 02, 2006 at 06:51:56PM +0100, Alan Cox wrote:
> "or similar" maybe. The iRam is pretty much junk in my experience[*].
> It rarely survives a mkfs, let alone sustained high throughput I/O.
> (And yes, I did try multiple DIMMs, including ones which survive
>  memtest86 just fine).

That appears to depend on the firmware and featureset used. With vaguely
recent firmware apart from the "failed diagnostics at boot" bug the one
I was loaned seems to be reliable and has fairly recent firmware.

> [*] And from googling/talking with other owners, my experiences aren't unique.

Agreed - even in windows 8)

Alan

Re: Postal 56% waits for flock_lock_file_wait

[Dave Jones]

On Mon, Oct 02, 2006 at 11:00:36PM +0100, Alan Cox wrote:
 > Ar Llu, 2006-10-02 am 13:40 -0400, ysgrifennodd Dave Jones:
 > > On Mon, Oct 02, 2006 at 06:51:56PM +0100, Alan Cox wrote:
 > > "or similar" maybe. The iRam is pretty much junk in my experience[*].
 > > It rarely survives a mkfs, let alone sustained high throughput I/O.
 > > (And yes, I did try multiple DIMMs, including ones which survive
 > >  memtest86 just fine).
 > 
 > That appears to depend on the firmware and featureset used. With vaguely
 > recent firmware apart from the "failed diagnostics at boot" bug the one
 > I was loaned seems to be reliable and has fairly recent firmware.

Mine was latest hardware (rev 1.3), and there wasn't any firmware update
available that I could see. :-/

 > > [*] And from googling/talking with other owners, my experiences aren't unique.
 > Agreed - even in windows 8)

It's amazing crap like this reaches the store shelves.

	Dave

-- 
http://www.codemonkey.org.uk

RE: Postal 56% waits for flock_lock_file_wait

[Ananiev, Leonid I]

> On which filesystem were the above results obtained if it was not
ext2?
The default ext3 fs was used.

> All the above results are telling you is that your test involves
several
> processes contending for the same lock, and so all of them barring the
> one process that actually holds the lock are idle.

Yes. It is  flock_lock_file_wait.

Leonid
-----Original Message-----
From: Trond Myklebust [mailto:trond.myklebust@fys.uio.no] 
Sent: Saturday, September 30, 2006 7:06 PM
To: Ananiev, Leonid I
Cc: Linux Kernel Mailing List
Subject: Re: Postal 56% waits for flock_lock_file_wait

On Sat, 2006-09-30 at 09:25 +0400, Ananiev, Leonid I wrote:
> A benchmark
>              'postal -p 16 localhost list_of_1000_users'
> 56% of run time waits for flock_lock_file_wait;
> Vmstat reports that 66% cpu is idle and  vmstat bi+bo=3600 (far from
> max).
> Postfix server with FD_SETSIZE=2048 was used.
> Similar results got for sendmail. 
> Wchan is counted by
>             while :; do
>                         ps -o user,wchan=WIDE-WCHAN-COLUMN,comm;
> sleep 1;
>            done | awk '/ postfix /{a[$2]++}END{for (i in a) print
> a[i]"\t"i}'
> If ext2 fs is used the Postal throughput is twice more and bi+bo by
50%
> less while  flock_lock_file_wait 60% still.

On which filesystem were the above results obtained if it was not ext2?

> Is flock_lock_file_wait considered as a performance limiting waiting
for
> similar applications in smp?

All the above results are telling you is that your test involves several
processes contending for the same lock, and so all of them barring the
one process that actually holds the lock are idle.

As for the throughput issue, that really depends on the filesystem you
are measuring. For remote filesystems like NFS, locks can _really_ slow
down performance because they are often required to flush all dirty data
to disk prior to releasing the lock (so that it becomes visible to
processes on other clients that might subsequently obtain the lock).

Cheers,
  Trond

RE: Postal 56% waits for flock_lock_file_wait

[Trond Myklebust]

On Sat, 2006-09-30 at 21:26 +0400, Ananiev, Leonid I wrote:
> > On which filesystem were the above results obtained if it was not
> ext2?
> The default ext3 fs was used.
> 
> > All the above results are telling you is that your test involves
> several
> > processes contending for the same lock, and so all of them barring the
> > one process that actually holds the lock are idle.
> 
> Yes. It is  flock_lock_file_wait.

That is the function which causes the sleep, yes. So what is your gripe?
The kernel would appear to be doing exactly what is expected of it.

Trond

Postal 56% waits for flock_lock_file_wait

[Ananiev, Leonid I]

A benchmark
             'postal -p 16 localhost list_of_1000_users'
56% of run time waits for flock_lock_file_wait;
Vmstat reports that 66% cpu is idle and  vmstat bi+bo=3600 (far from
max).
Postfix server with FD_SETSIZE=2048 was used.
Similar results got for sendmail. 
Wchan is counted by
            while :; do
                        ps -o user,wchan=WIDE-WCHAN-COLUMN,comm;
sleep 1;
           done | awk '/ postfix /{a[$2]++}END{for (i in a) print
a[i]"\t"i}'
If ext2 fs is used the Postal throughput is twice more and bi+bo by 50%
less while  flock_lock_file_wait 60% still.

Is flock_lock_file_wait considered as a performance limiting waiting for
similar applications in smp?

Leonid

Re: Postal 56% waits for flock_lock_file_wait

[Trond Myklebust]

On Sat, 2006-09-30 at 09:25 +0400, Ananiev, Leonid I wrote:
> A benchmark
>              'postal -p 16 localhost list_of_1000_users'
> 56% of run time waits for flock_lock_file_wait;
> Vmstat reports that 66% cpu is idle and  vmstat bi+bo=3600 (far from
> max).
> Postfix server with FD_SETSIZE=2048 was used.
> Similar results got for sendmail. 
> Wchan is counted by
>             while :; do
>                         ps -o user,wchan=WIDE-WCHAN-COLUMN,comm;
> sleep 1;
>            done | awk '/ postfix /{a[$2]++}END{for (i in a) print
> a[i]"\t"i}'
> If ext2 fs is used the Postal throughput is twice more and bi+bo by 50%
> less while  flock_lock_file_wait 60% still.

On which filesystem were the above results obtained if it was not ext2?

> Is flock_lock_file_wait considered as a performance limiting waiting for
> similar applications in smp?

All the above results are telling you is that your test involves several
processes contending for the same lock, and so all of them barring the
one process that actually holds the lock are idle.

As for the throughput issue, that really depends on the filesystem you
are measuring. For remote filesystems like NFS, locks can _really_ slow
down performance because they are often required to flush all dirty data
to disk prior to releasing the lock (so that it becomes visible to
processes on other clients that might subsequently obtain the lock).

Cheers,
  Trond

Postal 56% waits for flock_lock_file_wait

[Ananiev, Leonid I]

A benchmark
             'postal -p 16 localhost list_of_1000_users'
56% of run time waits for flock_lock_file_wait;
Vmstat reports that 66% cpu is idle and  vmstat bi+bo=3600 (far from
max).
Postfix server with FD_SETSIZE=2048 was used.
Similar results got for sendmail. 
Wchan is counted by
            while :; do
                        ps -o user,wchan=WIDE-WCHAN-COLUMN,comm;
sleep 1;
           done | awk '/ postfix /{a[$2]++}END{for (i in a) print
a[i]"\t"i}'
If ext2 fs is used the Postal throughput is twice more and bi+bo by 50%
less while  flock_lock_file_wait 60% still.

Is flock_lock_file_wait considered as a performance limiting waiting for
similar applications in smp?

Leonid