Possible performance problem introduced between kernel 2.6.15 and 2.6.17 on xfs code

Possible performance problem introduced between kernel 2.6.15 and 2.6.17 on xfs code

[fdegrassi]

Hi all,
i noticed that changing from kernel 2.6.15 to 2.6.17, gnomevfs-copy 
operations suffered an heavy performance hit (20 times slower).
I checked quickly (i am not a kernel hacker) and i think it could be 
related to libgnomevfs calling fadvice POSIX_FADV_DONTNEED  (in 
gnome_vfs_forget_cache, disabling this behaviour gets back normal 
performance).
I am not able to dig deeper than this, anyway it could be a starting point.
The relevant gnome bugzilla reference is the following: 
http://bugzilla.gnome.org/show_bug.cgi?id=363400

Francesco Degrassi

Re: Possible performance problem introduced between kernel 2.6.15 and 2.6.17 on xfs code

[Eric Sandeen]

fdegrassi wrote:
> Hi all,
> i noticed that changing from kernel 2.6.15 to 2.6.17, gnomevfs-copy 
> operations suffered an heavy performance hit (20 times slower).
> I checked quickly (i am not a kernel hacker) and i think it could be 
> related to libgnomevfs calling fadvice POSIX_FADV_DONTNEED  (in 
> gnome_vfs_forget_cache, disabling this behaviour gets back normal 
> performance).
> I am not able to dig deeper than this, anyway it could be a starting point.
> The relevant gnome bugzilla reference is the following: 
> http://bugzilla.gnome.org/show_bug.cgi?id=363400
> 
> Francesco Degrassi
> 
> 

Is a copy command from the shell affected, or just gnome point-n-click?

-Eric

Re: Possible performance problem introduced between kernel 2.6.15 and 2.6.17 on xfs code

[fdegrassi]

The specific problem is related to gnome point-n-click only.
Anyway, I could be wrong, but i don't think it is a bug in gnome because:
1. it affects specifically local XFS filesystems only
2. the problem is not present on kernel 2.6.15 (I tested 2.6.17 and 
2.6.15, both on ubuntu 6.10)
3. the problem does not manifest if i disable the call to 
gnome_vfs_forget_cache, that, for what i understand, simply calls 
fadvice POSIX_FADV_DONTNEED.

More info can be found on gnome bugzilla entry.
Francesco Degrassi


Eric Sandeen wrote:
> fdegrassi wrote:
>> Hi all,
>> i noticed that changing from kernel 2.6.15 to 2.6.17, gnomevfs-copy 
>> operations suffered an heavy performance hit (20 times slower).
>> I checked quickly (i am not a kernel hacker) and i think it could be 
>> related to libgnomevfs calling fadvice POSIX_FADV_DONTNEED  (in 
>> gnome_vfs_forget_cache, disabling this behaviour gets back normal 
>> performance).
>> I am not able to dig deeper than this, anyway it could be a starting 
>> point.
>> The relevant gnome bugzilla reference is the following: 
>> http://bugzilla.gnome.org/show_bug.cgi?id=363400
>>
>> Francesco Degrassi
>>
>>
>
> Is a copy command from the shell affected, or just gnome point-n-click?
>
> -Eric

Re: Possible performance problem introduced between kernel 2.6.15 and 2.6.17 on xfs code

[Eric Sandeen]

fdegrassi wrote:
> The specific problem is related to gnome point-n-click only.
> Anyway, I could be wrong, but i don't think it is a bug in gnome because:
> 1. it affects specifically local XFS filesystems only
> 2. the problem is not present on kernel 2.6.15 (I tested 2.6.17 and 
> 2.6.15, both on ubuntu 6.10)
> 3. the problem does not manifest if i disable the call to 
> gnome_vfs_forget_cache, that, for what i understand, simply calls 
> fadvice POSIX_FADV_DONTNEED.
> 
> More info can be found on gnome bugzilla entry.
> Francesco Degrassi
> 

Thanks.  I didn't mean to suggest that it was necessarily a gnome bug, 
just wanted to narrow down the problem.  Do you have any idea if other 
filesystems on 2.6.17 also show this slowdown?

-Eric

Re: Possible performance problem introduced between kernel 2.6.15 and 2.6.17 on xfs code

[Francesco Degrassi]

I tried ext2, ext3, jfs, reiser3, vfat and the problem does not show up.
Anyway i discovered that if i create a regular file in an XFS partition,
create an XFS filesystem on it as if it was a regular block device, and
mount it with the loop device, the problem does NOT manifest.
So it seems to affect XFS filesystems on regular block devices only ?
I'm confused.


____________EXAMPLE________________
root@francesco:/tmp$ mount | grep tmp
/dev/sda7 on /tmp type xfs (rw)
root@francesco:/tmp$ dd if=/dev/zero of=bdev bs=1024 count=200000
root@francesco:/tmp$ mkfs.xfs -f bdev
root@francesco:/tmp$ mkdir mpoint
root@francesco:/tmp$ mount -o loop bdev mpoint/
root@francesco:/tmp$ time gnomevfs-copy \
/home/fdegrassi/Desktop/downloads/systemrescuecd-x86-1.2.18.iso mpoint/

real	0m3.998s
user	0m0.080s
sys	0m1.292s

root@francesco:/tmp$ time gnomevfs-copy \
/home/fdegrassi/Desktop/downloads/systemrescuecd-x86-1.2.18.iso /tmp


real	2m55.448s
user	0m0.100s
sys	0m1.984s

________________________END_____________________

Thanks for your time, Eric.

Francesco Degrassi

Eric Sandeen wrote:
> fdegrassi wrote:
>> The specific problem is related to gnome point-n-click only.
>> Anyway, I could be wrong, but i don't think it is a bug in gnome because:
>> 1. it affects specifically local XFS filesystems only
>> 2. the problem is not present on kernel 2.6.15 (I tested 2.6.17 and
>> 2.6.15, both on ubuntu 6.10)
>> 3. the problem does not manifest if i disable the call to
>> gnome_vfs_forget_cache, that, for what i understand, simply calls
>> fadvice POSIX_FADV_DONTNEED.
>>
>> More info can be found on gnome bugzilla entry.
>> Francesco Degrassi
>>
> 
> Thanks.  I didn't mean to suggest that it was necessarily a gnome bug,
> just wanted to narrow down the problem.  Do you have any idea if other
> filesystems on 2.6.17 also show this slowdown?
> 
> -Eric

Re: Possible performance problem introduced between kernel 2.6.15 and 2.6.17 on xfs code

[Iustin Pop]

On Wed, Dec 27, 2006 at 07:03:45PM +0100, Francesco Degrassi wrote:
> I tried ext2, ext3, jfs, reiser3, vfat and the problem does not show up.
> Anyway i discovered that if i create a regular file in an XFS partition,
> create an XFS filesystem on it as if it was a regular block device, and
> mount it with the loop device, the problem does NOT manifest.
> So it seems to affect XFS filesystems on regular block devices only ?
> I'm confused.

This is probably not related, but I remember that barriers were added
sometime between 2.6.15 and 2.6.17 and probably barriers are not enabled
on a loop filesystem. Maybe the gnome vfs thing does some sync which
force flushes or such. I'm no expert, just guessing.

Just to invalidate this (probably wrong) guess, could you try to mount
/tmp with "-o nobarrier" and re-test?

Iustin, who wishes md raid1 would support barriers.

Re: Possible performance problem introduced between kernel 2.6.15 and 2.6.17 on xfs code

[Francesco Degrassi]

Hi Iustin,
i tried and it and indeed the performance problem does not manifest if i
mount the xfs filesystem with the -o nobarrier option. Seems like we are
a little bit closer to the real problem now.
The code in gnomevfs-copy calls fadvice POSIX_FADV_DONTNEED (in
gnome_vfs_forget_cache) and disabling this behaviour gets back normal
performance, so it has something to do with that.

Thanks for your help

Francesco


Iustin Pop wrote:
> On Wed, Dec 27, 2006 at 07:03:45PM +0100, Francesco Degrassi wrote:
>> I tried ext2, ext3, jfs, reiser3, vfat and the problem does not show up.
>> Anyway i discovered that if i create a regular file in an XFS partition,
>> create an XFS filesystem on it as if it was a regular block device, and
>> mount it with the loop device, the problem does NOT manifest.
>> So it seems to affect XFS filesystems on regular block devices only ?
>> I'm confused.
> 
> This is probably not related, but I remember that barriers were added
> sometime between 2.6.15 and 2.6.17 and probably barriers are not enabled
> on a loop filesystem. Maybe the gnome vfs thing does some sync which
> force flushes or such. I'm no expert, just guessing.
> 
> Just to invalidate this (probably wrong) guess, could you try to mount
> /tmp with "-o nobarrier" and re-test?
> 
> Iustin, who wishes md raid1 would support barriers.