XFS & LVM: unexpected cp when issuing mv

XFS & LVM: unexpected cp when issuing mv

[Michael Monnerie]

[-- Attachment #1.1: Type: text/plain, Size: 2619 bytes --]

I have an unexpected behaviour and I hope someone can explain me the 
reasons:

This is an openSUSE 11.2 virtual machine within XENserver. XENserver can 
only create 2TB disks, but I needed more. So I create 2x 2TB disks for 
that VM. These disks have no partitions, but are straight LVM:
# pvscan
  PV /dev/xvdb   VG sharestore   lvm2 [1,95 TB / 0    free]
  PV /dev/xvdc   VG sharestore   lvm2 [1,95 TB / 0    free]
  Total: 2 [3,91 TB] / in use: 2 [3,91 TB] / in no VG: 0 [0   ]

I created one VG, and then one LV:
# vgscan
  Reading all physical volumes.  This may take a while...
  Found volume group "sharestore" using metadata type lvm2
# lvscan
  ACTIVE            '/dev/sharestore/public' [3,91 TB] inherit

On that LV, I created an XFS filesystem, mounted from /etc/fstab:
/dev/sharestore/public /disks/sharestore  xfs        
noatime,nodiratime,logbufs=8,logbsize=256k,attr2,nobarrier,largeio,swalloc,inode64,prjquota

Now when I move from one dir to another, example
mv /disks/sharestore/upload/* /disks/sharestore/download/

within some dirs it's a simple mv where only metadata is moved, but with 
some dirs it's a physical cp+rm of the files. You can easily see that by 
the speed of the mv, plus with iostat:
Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-
sz avgqu-sz   await  svctm  %util
xvdb              0,00     0,00    0,00  647,31     0,00 28424,75    
87,82    18,46   29,71   0,24  15,65
xvdc              0,00     0,40  631,14    2,40 26928,54    76,65    
85,25     5,56    8,69   1,56  98,84

Until now I believed that a mv within one filesystem is always just a 
metadata mv. But it seems I found a case now where even within the same 
filesystem a physical cp+rm is done. Can someone explain me

1) why this happens
2) how I can prevent this?

We have files >5G there, often 20G or more, so a mv should just be a 
metadata mv, everything else is inacceptable.
Could it be the way I created the VG + LV, that there's a cp instead mv?
How could I create all that to get a normal behaviour?

Maybe like this?:
1) create VG only on one disk
2) create LV on that disk
3) create XFS
4) extend VG to 2nd disk
5) extend LV to 2nd disk
6) xfs_growfs to 2nd disk

mfg zmi
-- 
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS & LVM: unexpected cp when issuing mv

[Asdo]

[-- Attachment #1.1: Type: text/plain, Size: 2680 bytes --]

Please paste the output of the following 2 commands:

df  /disks/sharestore/upload/

df  /disks/sharestore/download/


Michael Monnerie wrote:
> I have an unexpected behaviour and I hope someone can explain me the 
> reasons:
>
> This is an openSUSE 11.2 virtual machine within XENserver. XENserver can 
> only create 2TB disks, but I needed more. So I create 2x 2TB disks for 
> that VM. These disks have no partitions, but are straight LVM:
> # pvscan
>   PV /dev/xvdb   VG sharestore   lvm2 [1,95 TB / 0    free]
>   PV /dev/xvdc   VG sharestore   lvm2 [1,95 TB / 0    free]
>   Total: 2 [3,91 TB] / in use: 2 [3,91 TB] / in no VG: 0 [0   ]
>
> I created one VG, and then one LV:
> # vgscan
>   Reading all physical volumes.  This may take a while...
>   Found volume group "sharestore" using metadata type lvm2
> # lvscan
>   ACTIVE            '/dev/sharestore/public' [3,91 TB] inherit
>
> On that LV, I created an XFS filesystem, mounted from /etc/fstab:
> /dev/sharestore/public /disks/sharestore  xfs        
> noatime,nodiratime,logbufs=8,logbsize=256k,attr2,nobarrier,largeio,swalloc,inode64,prjquota
>
> Now when I move from one dir to another, example
> mv /disks/sharestore/upload/* /disks/sharestore/download/
>
> within some dirs it's a simple mv where only metadata is moved, but with 
> some dirs it's a physical cp+rm of the files. You can easily see that by 
> the speed of the mv, plus with iostat:
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-
> sz avgqu-sz   await  svctm  %util
> xvdb              0,00     0,00    0,00  647,31     0,00 28424,75    
> 87,82    18,46   29,71   0,24  15,65
> xvdc              0,00     0,40  631,14    2,40 26928,54    76,65    
> 85,25     5,56    8,69   1,56  98,84
>
> Until now I believed that a mv within one filesystem is always just a 
> metadata mv. But it seems I found a case now where even within the same 
> filesystem a physical cp+rm is done. Can someone explain me
>
> 1) why this happens
> 2) how I can prevent this?
>
> We have files >5G there, often 20G or more, so a mv should just be a 
> metadata mv, everything else is inacceptable.
> Could it be the way I created the VG + LV, that there's a cp instead mv?
> How could I create all that to get a normal behaviour?
>
> Maybe like this?:
> 1) create VG only on one disk
> 2) create LV on that disk
> 3) create XFS
> 4) extend VG to 2nd disk
> 5) extend LV to 2nd disk
> 6) xfs_growfs to 2nd disk
>
> mfg zmi
>   
> ------------------------------------------------------------------------
>
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
>   


[-- Attachment #1.2: Type: text/html, Size: 3056 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS & LVM: unexpected cp when issuing mv

[Michael Monnerie]

[-- Attachment #1.1: Type: Text/Plain, Size: 930 bytes --]

On Sonntag, 29. November 2009 Asdo wrote:
> Please paste the output of the following 2 commands:
 
# df  /disks/sharestore/upload/
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/sharestore-public
                     4192254976 1956672968 2235582008  47% 
/disks/sharestore

# df  /disks/sharestore/download/
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/sharestore-public
                     4192254976 1956675156 2235579820  47% 
/disks/sharestore

Yes, it really *is* the same filesystem.

mfg zmi
-- 
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS & LVM: unexpected cp when issuing mv

[Dave Chinner]

On Sun, Nov 29, 2009 at 02:52:16PM +0100, Michael Monnerie wrote:
> I have an unexpected behaviour and I hope someone can explain me the 
> reasons:
> 
> This is an openSUSE 11.2 virtual machine within XENserver. XENserver can 
> only create 2TB disks, but I needed more. So I create 2x 2TB disks for 
> that VM. These disks have no partitions, but are straight LVM:
> # pvscan
>   PV /dev/xvdb   VG sharestore   lvm2 [1,95 TB / 0    free]
>   PV /dev/xvdc   VG sharestore   lvm2 [1,95 TB / 0    free]
>   Total: 2 [3,91 TB] / in use: 2 [3,91 TB] / in no VG: 0 [0   ]
> 
> I created one VG, and then one LV:
> # vgscan
>   Reading all physical volumes.  This may take a while...
>   Found volume group "sharestore" using metadata type lvm2
> # lvscan
>   ACTIVE            '/dev/sharestore/public' [3,91 TB] inherit
> 
> On that LV, I created an XFS filesystem, mounted from /etc/fstab:
> /dev/sharestore/public /disks/sharestore  xfs        
> noatime,nodiratime,logbufs=8,logbsize=256k,attr2,nobarrier,largeio,swalloc,inode64,prjquota

You are using project quotas.

> Now when I move from one dir to another, example
> mv /disks/sharestore/upload/* /disks/sharestore/download/
> 
> within some dirs it's a simple mv where only metadata is moved, but with 
> some dirs it's a physical cp+rm of the files. You can easily see that by 
> the speed of the mv, plus with iostat:
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-
> sz avgqu-sz   await  svctm  %util
> xvdb              0,00     0,00    0,00  647,31     0,00 28424,75    
> 87,82    18,46   29,71   0,24  15,65
> xvdc              0,00     0,40  631,14    2,40 26928,54    76,65    
> 85,25     5,56    8,69   1,56  98,84
> 
> Until now I believed that a mv within one filesystem is always just a 
> metadata mv. But it seems I found a case now where even within the same 
> filesystem a physical cp+rm is done. Can someone explain me
> 
> 1) why this happens

It happens if you move from one project directory heirarchy to
another - rename is not allowed across project quota boundaries as
the moved data has to be correctly attributed to the new project.
Hence it causes a mv to do a copy/unlink by returning a EXDEV error
to the rename.

> 2) how I can prevent this?

You can't if you are moving from one project to another. If you
move within the project heirarchy, then it will be a rename as per
normal.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS & LVM: unexpected cp when issuing mv

[Michael Monnerie]

[-- Attachment #1.1: Type: Text/Plain, Size: 1319 bytes --]

On Montag, 30. November 2009 Dave Chinner wrote:
> > 1) why this happens
> 
> It happens if you move from one project directory heirarchy to
> another - rename is not allowed across project quota boundaries as
> the moved data has to be correctly attributed to the new project.
> Hence it causes a mv to do a copy/unlink by returning a EXDEV error
> to the rename.
> 
> > 2) how I can prevent this?
> 
> You can't if you are moving from one project to another. If you
> move within the project heirarchy, then it will be a rename as per
> normal.
 
Shit. So I have to turn project quotas off. I can't accept the extra 
load for a simple move, as there are tons of data. Maybe the projet code 
could be redesigned to allow a simple move? Is it that complicated?

If I change from project to user quota - I guess the same would still 
happen? If so, I have to drop quota support, and build a script to check 
quotas manually. It's a pity. :-(

mfg zmi
-- 
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

RE: XFS & LVM: unexpected cp when issuing mv

[Alex Elder]

Michael Monnerie wrote:
> On Montag, 30. November 2009 Dave Chinner wrote:
>>> 1) why this happens
>> 
>> It happens if you move from one project directory heirarchy to
>> another - rename is not allowed across project quota boundaries as
>> the moved data has to be correctly attributed to the new project.
>> Hence it causes a mv to do a copy/unlink by returning a EXDEV error to the rename.
>> 
>>> 2) how I can prevent this?
>> 
>> You can't if you are moving from one project to another. If you
>> move within the project heirarchy, then it will be a rename as per
>> normal.
> 
> Shit. So I have to turn project quotas off. I can't accept the extra
> load for a simple move, as there are tons of data. Maybe the projet code
> could be redesigned to allow a simple move? Is it that complicated?
> 
> If I change from project to user quota - I guess the same would still
> happen? If so, I have to drop quota support, and build a script to check
> quotas manually. It's a pity. :-(

The quota inheritance property Dave described only applies
to project quotas, not group or user quotas.

					-Alex

> mfg zmi

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS & LVM: unexpected cp when issuing mv

[Michael Monnerie]

[-- Attachment #1.1: Type: Text/Plain, Size: 721 bytes --]

On Montag, 30. November 2009 Alex Elder wrote:
> The quota inheritance property Dave described only applies
> to project quotas, not group or user quotas.
> 
Thanks. I thought about it but can't use them anyway, as I would have to 
change the owner of files on upload. But I want to keep that 
information. So turning off quota is our only chance by now :-(

mfg zmi
-- 
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS & LVM: unexpected cp when issuing mv

[Dave Chinner]

On Mon, Nov 30, 2009 at 02:42:20PM +0100, Michael Monnerie wrote:
> On Montag, 30. November 2009 Dave Chinner wrote:
> > > 1) why this happens
> > 
> > It happens if you move from one project directory heirarchy to
> > another - rename is not allowed across project quota boundaries as
> > the moved data has to be correctly attributed to the new project.
> > Hence it causes a mv to do a copy/unlink by returning a EXDEV error
> > to the rename.
> > 
> > > 2) how I can prevent this?
> > 
> > You can't if you are moving from one project to another. If you
> > move within the project heirarchy, then it will be a rename as per
> > normal.
>  
> Shit. So I have to turn project quotas off. I can't accept the extra 
> load for a simple move, as there are tons of data. Maybe the projet code 
> could be redesigned to allow a simple move? Is it that complicated?

IIRC, it was a design decision to make project quotas behave this
way.  Returning EXDEV made the code to handle this very simple -
userspace already handles this error correctly and all the kernel
needs is 3 lines of code to detect differing project IDs on the
inodes and return EXDEV.

To do it without copying is definitely possible - it requires
extending the rename transaction to directly modify the project
quota id of the target inodes and the relevant dquots as well. It
becomes quite complicated because it also has to handle the cases of
source or destination not having quotas at all, the destination
not having enough space (i.e. EDQUOT), etc.

All the nasty corner cases are handled by userspace by returning
EXDEV to force it to copy rather than renaming, so the decision was
made to do the simple (if slow) thing for relatively uncommon
operation of moving data between projects.

> If I change from project to user quota - I guess the same would still 
> happen?

No - the EXDEV error (and therefore the copy) is a feature of project
quotas.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs