Extended attributes limit in Linux

Extended attributes limit in Linux

[Sun_Blood]

[-- Attachment #1.1: Type: text/plain, Size: 811 bytes --]

Hello,

If I understands it correctly XFS don't have a limit to the size of
extended attributes(EA) but Linux impose a limit at 64k.
What I am trying to do is build a backup server that our Apple computers
will use together with rsync to backup files to. The problem I face is that
Apple HFS+ don't have a limit to EA so it has files with more then 64k of
EA in it.

The Linux Kernel has a limit imposed to it in include/linux/limits.h

> #define XATTR_SIZE_MAX 65536    /* size of an extended attribute value
> (64k) */
>
#define XATTR_LIST_MAX 65536    /* size of extended attribute namelist
> (64k) */
>

Changing this values feels unsafe because they will be system wide.

Is the any possibility to combine Linux + XFS to make a backup that works
with EA or is this more a question for kernel.org?

Thanks.

[-- Attachment #1.2: Type: text/html, Size: 1573 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Extended attributes limit in Linux

[Jeff Liu]

Hello,

On 01/31 2014 15:40 PM, Sun_Blood wrote:
> Hello,
> 
> If I understands it correctly XFS don't have a limit to the size of
> extended attributes(EA) but Linux impose a limit at 64k.
> What I am trying to do is build a backup server that our Apple computers
> will use together with rsync to backup files to. The problem I face is
> that Apple HFS+ don't have a limit to EA so it has files with more then
> 64k of EA in it.
> 
> The Linux Kernel has a limit imposed to it in include/linux/limits.h
> 
>     #defineXATTR_SIZE_MAX 65536   /* size of an extended attribute value
>     (64k) */
> 
>     #defineXATTR_LIST_MAX 65536   /* size of extended attribute namelist
>     (64k) */
> 

Yes, 64k is the VFS limit per EA value size.
> Changing this values feels unsafe because they will be system wide.
> 
> Is the any possibility to combine Linux + XFS to make a backup that
> works with EA or is this more a question for kernel.org <http://kernel.org>?

So if an EA value size on HFS+ is larger than 64k, I suspect it would failed to preserve
on Linux+XFS... But if a file has multiple EAs, and each EA's value size is <= 64K, it
would be works IMO.

Thanks,
-Jeff

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Extended attributes limit in Linux

[Vyacheslav Dubeyko]

On Fri, 2014-01-31 at 16:54 +0800, Jeff Liu wrote:
> Hello,
> 
> On 01/31 2014 15:40 PM, Sun_Blood wrote:
> > Hello,
> > 
> > If I understands it correctly XFS don't have a limit to the size of
> > extended attributes(EA) but Linux impose a limit at 64k.
> > What I am trying to do is build a backup server that our Apple computers
> > will use together with rsync to backup files to. The problem I face is
> > that Apple HFS+ don't have a limit to EA so it has files with more then
> > 64k of EA in it.

Technical Note TN1150:
"In an HFS Plus B-tree, the node size is determined by a field
(nodeSize) in the header node. The node size must be a power from 512
through 32,768."

So, as minimum, xattrs unable to be a larger than node size.

But xattrs' size has limitation anyway:

https://github.com/darwin-on-arm/xnu/blob/master/bsd/hfs/hfs.h#L849

/* Maximum extended attribute size supported for all extended attributes except
* resource fork and finder info.
*/
#define HFS_XATTR_MAXSIZE (128 * 1024)

So, I need to check and correct slightly HFS+ xattrs support code.
Because, I used 3082 bytes limitation value. 

Thanks,
Vyacheslav Dubeyko.

> > 
> > The Linux Kernel has a limit imposed to it in include/linux/limits.h
> > 
> >     #defineXATTR_SIZE_MAX 65536   /* size of an extended attribute value
> >     (64k) */
> > 
> >     #defineXATTR_LIST_MAX 65536   /* size of extended attribute namelist
> >     (64k) */
> > 
> 
> Yes, 64k is the VFS limit per EA value size.
> > Changing this values feels unsafe because they will be system wide.
> > 
> > Is the any possibility to combine Linux + XFS to make a backup that
> > works with EA or is this more a question for kernel.org <http://kernel.org>?
> 
> So if an EA value size on HFS+ is larger than 64k, I suspect it would failed to preserve
> on Linux+XFS... But if a file has multiple EAs, and each EA's value size is <= 64K, it
> would be works IMO.
> 
> Thanks,
> -Jeff
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs


_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Extended attributes limit in Linux

[Jeff Liu]

On 01/31 2014 18:44 PM, Vyacheslav Dubeyko wrote:
> On Fri, 2014-01-31 at 16:54 +0800, Jeff Liu wrote:
>> Hello,
>>
>> On 01/31 2014 15:40 PM, Sun_Blood wrote:
>>> Hello,
>>>
>>> If I understands it correctly XFS don't have a limit to the size of
>>> extended attributes(EA) but Linux impose a limit at 64k.
>>> What I am trying to do is build a backup server that our Apple computers
>>> will use together with rsync to backup files to. The problem I face is
>>> that Apple HFS+ don't have a limit to EA so it has files with more then
>>> 64k of EA in it.
> 
> Technical Note TN1150:
> "In an HFS Plus B-tree, the node size is determined by a field
> (nodeSize) in the header node. The node size must be a power from 512
> through 32,768."
> 
> So, as minimum, xattrs unable to be a larger than node size.
> 
> But xattrs' size has limitation anyway:
> 
> https://github.com/darwin-on-arm/xnu/blob/master/bsd/hfs/hfs.h#L849
> 
> /* Maximum extended attribute size supported for all extended attributes except
> * resource fork and finder info.
> */
> #define HFS_XATTR_MAXSIZE (128 * 1024)
> 
> So, I need to check and correct slightly HFS+ xattrs support code.
> Because, I used 3082 bytes limitation value.

I just verified above limits on Mac OS X, that's true.

sh-3.2# a=`perl -e 'print "A"x131072'`
sh-3.2# xattr -w user.comment ${a} xattr_test

sh-3.2# a=`perl -e 'print "A"x131073'`
sh-3.2# xattr -w user.comment ${a} xattr_test
xattr: [Errno 7] Argument list too long: 'xattr'

Hence, that would be a problem to preserve EA with large value size on Linux.

Thanks,
-Jeff

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Extended attributes limit in Linux

[Vyacheslav Dubeyko]

On Fri, 2014-01-31 at 20:24 +0800, Jeff Liu wrote:
> On 01/31 2014 18:44 PM, Vyacheslav Dubeyko wrote:
> > On Fri, 2014-01-31 at 16:54 +0800, Jeff Liu wrote:
> >> Hello,
> >>
> >> On 01/31 2014 15:40 PM, Sun_Blood wrote:
> >>> Hello,
> >>>
> >>> If I understands it correctly XFS don't have a limit to the size of
> >>> extended attributes(EA) but Linux impose a limit at 64k.
> >>> What I am trying to do is build a backup server that our Apple computers
> >>> will use together with rsync to backup files to. The problem I face is
> >>> that Apple HFS+ don't have a limit to EA so it has files with more then
> >>> 64k of EA in it.
> > 
> > Technical Note TN1150:
> > "In an HFS Plus B-tree, the node size is determined by a field
> > (nodeSize) in the header node. The node size must be a power from 512
> > through 32,768."
> > 
> > So, as minimum, xattrs unable to be a larger than node size.
> > 
> > But xattrs' size has limitation anyway:
> > 
> > https://github.com/darwin-on-arm/xnu/blob/master/bsd/hfs/hfs.h#L849
> > 
> > /* Maximum extended attribute size supported for all extended attributes except
> > * resource fork and finder info.
> > */
> > #define HFS_XATTR_MAXSIZE (128 * 1024)
> > 
> > So, I need to check and correct slightly HFS+ xattrs support code.
> > Because, I used 3082 bytes limitation value.
> 
> I just verified above limits on Mac OS X, that's true.
> 
> sh-3.2# a=`perl -e 'print "A"x131072'`
> sh-3.2# xattr -w user.comment ${a} xattr_test
> 
> sh-3.2# a=`perl -e 'print "A"x131073'`
> sh-3.2# xattr -w user.comment ${a} xattr_test
> xattr: [Errno 7] Argument list too long: 'xattr'
> 
> Hence, that would be a problem to preserve EA with large value size on Linux.
> 

I checked the same under Mac OS X 10.6.8 (Snow Leopard). And I have
failed on 3803 bytes size of xattr. So, I suppose that you have Mac OS X
Lion. And EAs is larger under Lion yet.

What version of Mac OS X have you?

Thanks,
Vyacheslav Dubeyko.


_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Extended attributes limit in Linux

[Jeff Liu]

On 01/31 2014 20:52 PM, Vyacheslav Dubeyko wrote:
> On Fri, 2014-01-31 at 20:24 +0800, Jeff Liu wrote:
>> On 01/31 2014 18:44 PM, Vyacheslav Dubeyko wrote:
>>> On Fri, 2014-01-31 at 16:54 +0800, Jeff Liu wrote:
>>>> Hello,
>>>>
>>>> On 01/31 2014 15:40 PM, Sun_Blood wrote:
>>>>> Hello,
>>>>>
>>>>> If I understands it correctly XFS don't have a limit to the size of
>>>>> extended attributes(EA) but Linux impose a limit at 64k.
>>>>> What I am trying to do is build a backup server that our Apple computers
>>>>> will use together with rsync to backup files to. The problem I face is
>>>>> that Apple HFS+ don't have a limit to EA so it has files with more then
>>>>> 64k of EA in it.
>>>
>>> Technical Note TN1150:
>>> "In an HFS Plus B-tree, the node size is determined by a field
>>> (nodeSize) in the header node. The node size must be a power from 512
>>> through 32,768."
>>>
>>> So, as minimum, xattrs unable to be a larger than node size.
>>>
>>> But xattrs' size has limitation anyway:
>>>
>>> https://github.com/darwin-on-arm/xnu/blob/master/bsd/hfs/hfs.h#L849
>>>
>>> /* Maximum extended attribute size supported for all extended attributes except
>>> * resource fork and finder info.
>>> */
>>> #define HFS_XATTR_MAXSIZE (128 * 1024)
>>>
>>> So, I need to check and correct slightly HFS+ xattrs support code.
>>> Because, I used 3082 bytes limitation value.
>>
>> I just verified above limits on Mac OS X, that's true.
>>
>> sh-3.2# a=`perl -e 'print "A"x131072'`
>> sh-3.2# xattr -w user.comment ${a} xattr_test
>>
>> sh-3.2# a=`perl -e 'print "A"x131073'`
>> sh-3.2# xattr -w user.comment ${a} xattr_test
>> xattr: [Errno 7] Argument list too long: 'xattr'
>>
>> Hence, that would be a problem to preserve EA with large value size on Linux.
>>
> 
> I checked the same under Mac OS X 10.6.8 (Snow Leopard). And I have
> failed on 3803 bytes size of xattr. So, I suppose that you have Mac OS X
> Lion. And EAs is larger under Lion yet.
> 
> What version of Mac OS X have you?
> 
Yup, Mountain Lion v10.8.4 :)

FYI, there have a couple of things regarding HFSPlus+xattr+acl on Linux might be
deserved to discuss together.

We once have a discussion about the errno in case of hit the limits of ACLs, which
could be referred to:
http://www.spinics.net/lists/linux-fsdevel/msg71125.html

HFSPlus return ENOMEM in this case, but it should be E2BIG as per Dave's comments.
I worked out a patch series includes HFSPlus, but not yet posted for some reasons.

Also, it seems to me we'd better consolidate the errno for EA as well, that is to
say, it's better to fix the return error to be consistent with VFS interface in case
of the given EA name/value length is larger than the specified limits.

Would you like to take a look at the following two patches?

Thanks,
-Jeff


From: Jie Liu <jeff.liu@oracle.com>
Subject: [PATCH 7/12] HFSPlus: return -E2BIG if hit the maximum size of ACLs

Return -E2BIG rather than -ENOMEM if hit the maximum size of ACLs, because
of the former errno is consistent with the VFS interface.

Signed-off-by: Jie Liu <jeff.liu@oracle.com>
---
 fs/hfsplus/posix_acl.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/fs/hfsplus/posix_acl.c b/fs/hfsplus/posix_acl.c
index b609cc1..2f2788d 100644
--- a/fs/hfsplus/posix_acl.c
+++ b/fs/hfsplus/posix_acl.c
@@ -92,7 +92,7 @@ static int hfsplus_set_posix_acl(struct inode *inode,
 	if (acl) {
 		size = posix_acl_xattr_size(acl->a_count);
 		if (unlikely(size > HFSPLUS_MAX_INLINE_DATA_SIZE))
-			return -ENOMEM;
+			return -E2BIG;
 		value = (char *)hfsplus_alloc_attr_entry();
 		if (unlikely(!value))
 			return -ENOMEM;
-- 
1.8.3.2


From: Jie Liu <jeff.liu@oracle.com>
Subject: [PATCH 8/12] HFSPlus: return -ERANGE if xattr name size is larger than HFSPLUS_ATTR_MAX_STRLEN

Return -ERANGE rather than -EOPNOTSUPP if the length of xattr name is
larger than HFSPLUS_ATTR_MAX_STRLEN, because of the former errno is
consistent with the VFS interface.

Signed-off-by: Jie Liu <jeff.liu@oracle.com>
---
 fs/hfsplus/xattr.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/fs/hfsplus/xattr.c b/fs/hfsplus/xattr.c
index 3c6136f..1ca58be 100644
--- a/fs/hfsplus/xattr.c
+++ b/fs/hfsplus/xattr.c
@@ -919,7 +919,7 @@ static int hfsplus_osx_getxattr(struct dentry *dentry, const char *name,
 		return -EINVAL;
 
 	if (len > HFSPLUS_ATTR_MAX_STRLEN)
-		return -EOPNOTSUPP;
+		return -ERANGE;
 
 	strcpy(xattr_name, XATTR_MAC_OSX_PREFIX);
 	strcpy(xattr_name + XATTR_MAC_OSX_PREFIX_LEN, name);
@@ -938,7 +938,7 @@ static int hfsplus_osx_setxattr(struct dentry *dentry, const char *name,
 		return -EINVAL;
 
 	if (len > HFSPLUS_ATTR_MAX_STRLEN)
-		return -EOPNOTSUPP;
+		return -ERANGE;
 
 	strcpy(xattr_name, XATTR_MAC_OSX_PREFIX);
 	strcpy(xattr_name + XATTR_MAC_OSX_PREFIX_LEN, name);
-- 
1.8.3.2


_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Extended attributes limit in Linux

[Vyacheslav Dubeyko]

On Fri, 2014-01-31 at 21:39 +0800, Jeff Liu wrote:

> > 
> > I checked the same under Mac OS X 10.6.8 (Snow Leopard). And I have
> > failed on 3803 bytes size of xattr. So, I suppose that you have Mac OS X
> > Lion. And EAs is larger under Lion yet.
> > 
> > What version of Mac OS X have you?
> > 
> Yup, Mountain Lion v10.8.4 :)
> 

I suspect that xattrs with significant size is stored in compressed
state on HFS+. I implemented support of compressed xattrs partially but
I don't share this code yet. But, yes, EAs with size greater than 64 KB
can be a problem.

> FYI, there have a couple of things regarding HFSPlus+xattr+acl on Linux might be
> deserved to discuss together.
> 
> We once have a discussion about the errno in case of hit the limits of ACLs, which
> could be referred to:
> http://www.spinics.net/lists/linux-fsdevel/msg71125.html
> 
> HFSPlus return ENOMEM in this case, but it should be E2BIG as per Dave's comments.
> I worked out a patch series includes HFSPlus, but not yet posted for some reasons.
> 
> Also, it seems to me we'd better consolidate the errno for EA as well, that is to
> say, it's better to fix the return error to be consistent with VFS interface in case
> of the given EA name/value length is larger than the specified limits.
> 
> Would you like to take a look at the following two patches?
> 

Yes, sure. Patches looks correct and good for me. But did you take into
account recent significant changes of Christoph Hellwig? If so, then all
looks good.

Thanks,
Vyacheslav Dubeyko.

> Thanks,
> -Jeff
> 
> 
> From: Jie Liu <jeff.liu@oracle.com>
> Subject: [PATCH 7/12] HFSPlus: return -E2BIG if hit the maximum size of ACLs
> 
> Return -E2BIG rather than -ENOMEM if hit the maximum size of ACLs, because
> of the former errno is consistent with the VFS interface.
> 
> Signed-off-by: Jie Liu <jeff.liu@oracle.com>
> ---
>  fs/hfsplus/posix_acl.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/fs/hfsplus/posix_acl.c b/fs/hfsplus/posix_acl.c
> index b609cc1..2f2788d 100644
> --- a/fs/hfsplus/posix_acl.c
> +++ b/fs/hfsplus/posix_acl.c
> @@ -92,7 +92,7 @@ static int hfsplus_set_posix_acl(struct inode *inode,
>  	if (acl) {
>  		size = posix_acl_xattr_size(acl->a_count);
>  		if (unlikely(size > HFSPLUS_MAX_INLINE_DATA_SIZE))
> -			return -ENOMEM;
> +			return -E2BIG;
>  		value = (char *)hfsplus_alloc_attr_entry();
>  		if (unlikely(!value))
>  			return -ENOMEM;
> -- 
> 1.8.3.2
> 
> 
> From: Jie Liu <jeff.liu@oracle.com>
> Subject: [PATCH 8/12] HFSPlus: return -ERANGE if xattr name size is larger than HFSPLUS_ATTR_MAX_STRLEN
> 
> Return -ERANGE rather than -EOPNOTSUPP if the length of xattr name is
> larger than HFSPLUS_ATTR_MAX_STRLEN, because of the former errno is
> consistent with the VFS interface.
> 
> Signed-off-by: Jie Liu <jeff.liu@oracle.com>
> ---
>  fs/hfsplus/xattr.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/fs/hfsplus/xattr.c b/fs/hfsplus/xattr.c
> index 3c6136f..1ca58be 100644
> --- a/fs/hfsplus/xattr.c
> +++ b/fs/hfsplus/xattr.c
> @@ -919,7 +919,7 @@ static int hfsplus_osx_getxattr(struct dentry *dentry, const char *name,
>  		return -EINVAL;
>  
>  	if (len > HFSPLUS_ATTR_MAX_STRLEN)
> -		return -EOPNOTSUPP;
> +		return -ERANGE;
>  
>  	strcpy(xattr_name, XATTR_MAC_OSX_PREFIX);
>  	strcpy(xattr_name + XATTR_MAC_OSX_PREFIX_LEN, name);
> @@ -938,7 +938,7 @@ static int hfsplus_osx_setxattr(struct dentry *dentry, const char *name,
>  		return -EINVAL;
>  
>  	if (len > HFSPLUS_ATTR_MAX_STRLEN)
> -		return -EOPNOTSUPP;
> +		return -ERANGE;
>  
>  	strcpy(xattr_name, XATTR_MAC_OSX_PREFIX);
>  	strcpy(xattr_name + XATTR_MAC_OSX_PREFIX_LEN, name);


_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Extended attributes limit in Linux

[Jeff Liu]

On 01/31 2014 22:21 PM, Vyacheslav Dubeyko wrote:
> On Fri, 2014-01-31 at 21:39 +0800, Jeff Liu wrote:
> 
>>>
>>> I checked the same under Mac OS X 10.6.8 (Snow Leopard). And I have
>>> failed on 3803 bytes size of xattr. So, I suppose that you have Mac OS X
>>> Lion. And EAs is larger under Lion yet.
>>>
>>> What version of Mac OS X have you?
>>>
>> Yup, Mountain Lion v10.8.4 :)
>>
> 
> I suspect that xattrs with significant size is stored in compressed
> state on HFS+. I implemented support of compressed xattrs partially but
> I don't share this code yet. But, yes, EAs with size greater than 64 KB
> can be a problem.
> 
>> FYI, there have a couple of things regarding HFSPlus+xattr+acl on Linux might be
>> deserved to discuss together.
>>
>> We once have a discussion about the errno in case of hit the limits of ACLs, which
>> could be referred to:
>> http://www.spinics.net/lists/linux-fsdevel/msg71125.html
>>
>> HFSPlus return ENOMEM in this case, but it should be E2BIG as per Dave's comments.
>> I worked out a patch series includes HFSPlus, but not yet posted for some reasons.
>>
>> Also, it seems to me we'd better consolidate the errno for EA as well, that is to
>> say, it's better to fix the return error to be consistent with VFS interface in case
>> of the given EA name/value length is larger than the specified limits.
>>
>> Would you like to take a look at the following two patches?
>>
> 
> Yes, sure. Patches looks correct and good for me. But did you take into
> account recent significant changes of Christoph Hellwig? If so, then all
> looks good.

Thanks for the review.  That's one major reason I postpone the patch set submission
as I did it before Christoph's changes got merged, it looks like no obvious conflicts
with that, but I will find time to double check and sent it to you -- still on vacation
for Chinese Spring Festival.

Thanks,
-Jeff

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Extended attributes limit in Linux

[Sun_Blood]

On 31 jan 2014, at 15:33, Jeff Liu <jeff.liu@oracle.com> wrote:

> 
> On 01/31 2014 22:21 PM, Vyacheslav Dubeyko wrote:
>> On Fri, 2014-01-31 at 21:39 +0800, Jeff Liu wrote:
>> 
>>>> 
>>>> I checked the same under Mac OS X 10.6.8 (Snow Leopard). And I have
>>>> failed on 3803 bytes size of xattr. So, I suppose that you have Mac OS X
>>>> Lion. And EAs is larger under Lion yet.
>>>> 
>>>> What version of Mac OS X have you?
>>>> 
>>> Yup, Mountain Lion v10.8.4 :)
>>> 
>> 
>> I suspect that xattrs with significant size is stored in compressed
>> state on HFS+. I implemented support of compressed xattrs partially but
>> I don't share this code yet. But, yes, EAs with size greater than 64 KB
>> can be a problem.


FYI, Example of output from one of the failing files. First from OS X and then same file after failed copy to XFS.

OS X Maverik:
file: "/Users/username/Pictures/iPhoto Library/Database/apdb/BigBlobs.apdb"
type: "\0\0\0\0"
creator: "\0\0\0\0"
attributes: avbstclinmedz
created: 01/25/2014 11:43:17
modified: 01/28/2014 20:02:46


Ubunutu 
getfattr: Removing leading '/' from absolute path names
# file: srv/nas/home/apple_bak_rsync/username/Pictures/iPhoto Library/Database/BigBlobs.apdb
user.com.apple.quarantine="0006;52e39545;iPhoto;”

>> 
>>> FYI, there have a couple of things regarding HFSPlus+xattr+acl on Linux might be
>>> deserved to discuss together.
>>> 
>>> We once have a discussion about the errno in case of hit the limits of ACLs, which
>>> could be referred to:
>>> http://www.spinics.net/lists/linux-fsdevel/msg71125.html
>>> 
>>> HFSPlus return ENOMEM in this case, but it should be E2BIG as per Dave's comments.
>>> I worked out a patch series includes HFSPlus, but not yet posted for some reasons.
>>> 
>>> Also, it seems to me we'd better consolidate the errno for EA as well, that is to
>>> say, it's better to fix the return error to be consistent with VFS interface in case
>>> of the given EA name/value length is larger than the specified limits.
>>> 
>>> Would you like to take a look at the following two patches?
>>> 
>> 
>> Yes, sure. Patches looks correct and good for me. But did you take into
>> account recent significant changes of Christoph Hellwig? If so, then all
>> looks good.
> 
> Thanks for the review.  That's one major reason I postpone the patch set submission
> as I did it before Christoph's changes got merged, it looks like no obvious conflicts
> with that, but I will find time to double check and sent it to you -- still on vacation
> for Chinese Spring Festival.
> 
> Thanks,
> -Jeff

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Extended attributes limit in Linux

[Vyacheslav Dubeyko]

On Jan 31, 2014, at 10:25 PM, Sun_Blood wrote:

> 
> On 31 jan 2014, at 15:33, Jeff Liu <jeff.liu@oracle.com> wrote:
> 
>> 
>> On 01/31 2014 22:21 PM, Vyacheslav Dubeyko wrote:
>>> On Fri, 2014-01-31 at 21:39 +0800, Jeff Liu wrote:
>>> 
>>>>> 
>>>>> I checked the same under Mac OS X 10.6.8 (Snow Leopard). And I have
>>>>> failed on 3803 bytes size of xattr. So, I suppose that you have Mac OS X
>>>>> Lion. And EAs is larger under Lion yet.
>>>>> 
>>>>> What version of Mac OS X have you?
>>>>> 
>>>> Yup, Mountain Lion v10.8.4 :)
>>>> 
>>> 
>>> I suspect that xattrs with significant size is stored in compressed
>>> state on HFS+. I implemented support of compressed xattrs partially but
>>> I don't share this code yet. But, yes, EAs with size greater than 64 KB
>>> can be a problem.
> 
> 
> FYI, Example of output from one of the failing files. First from OS X and then same file after failed copy to XFS.
> 
> OS X Maverik:
> file: "/Users/username/Pictures/iPhoto Library/Database/apdb/BigBlobs.apdb"
> type: "\0\0\0\0"
> creator: "\0\0\0\0"
> attributes: avbstclinmedz
> created: 01/25/2014 11:43:17
> modified: 01/28/2014 20:02:46
> 
> 
> Ubunutu 
> getfattr: Removing leading '/' from absolute path names
> # file: srv/nas/home/apple_bak_rsync/username/Pictures/iPhoto Library/Database/BigBlobs.apdb
> user.com.apple.quarantine="0006;52e39545;iPhoto;”


Sorry, but I don't quite follow your thought. What do you show by this output?
What do you mean? Could you describe in more details?

Thanks,
Vyacheslav Dubeyko.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Extended attributes limit in Linux

[Sun_Blood]

[-- Attachment #1.1: Type: text/plain, Size: 2586 bytes --]


On 1 feb 2014, at 15:08, Vyacheslav Dubeyko <slava@dubeyko.com> wrote:

> 
> On Jan 31, 2014, at 10:25 PM, Sun_Blood wrote:
> 
>> 
>> On 31 jan 2014, at 15:33, Jeff Liu <jeff.liu@oracle.com> wrote:
>> 
>>> 
>>> On 01/31 2014 22:21 PM, Vyacheslav Dubeyko wrote:
>>>> On Fri, 2014-01-31 at 21:39 +0800, Jeff Liu wrote:
>>>> 
>>>>>> 
>>>>>> I checked the same under Mac OS X 10.6.8 (Snow Leopard). And I have
>>>>>> failed on 3803 bytes size of xattr. So, I suppose that you have Mac OS X
>>>>>> Lion. And EAs is larger under Lion yet.
>>>>>> 
>>>>>> What version of Mac OS X have you?
>>>>>> 
>>>>> Yup, Mountain Lion v10.8.4 :)
>>>>> 
>>>> 
>>>> I suspect that xattrs with significant size is stored in compressed
>>>> state on HFS+. I implemented support of compressed xattrs partially but
>>>> I don't share this code yet. But, yes, EAs with size greater than 64 KB
>>>> can be a problem.
>> 
>> 
>> FYI, Example of output from one of the failing files. First from OS X and then same file after failed copy to XFS.
>> 
>> OS X Maverik:
>> file: "/Users/username/Pictures/iPhoto Library/Database/apdb/BigBlobs.apdb"
>> type: "\0\0\0\0"
>> creator: "\0\0\0\0"
>> attributes: avbstclinmedz
>> created: 01/25/2014 11:43:17
>> modified: 01/28/2014 20:02:46
>> 
>> 
>> Ubunutu 
>> getfattr: Removing leading '/' from absolute path names
>> # file: srv/nas/home/apple_bak_rsync/username/Pictures/iPhoto Library/Database/BigBlobs.apdb
>> user.com.apple.quarantine="0006;52e39545;iPhoto;”
> 
> 
> Sorry, but I don't quite follow your thought. What do you show by this output?
> What do you mean? Could you describe in more details?
> 
> Thanks,
> Vyacheslav Dubeyko.

Sorry late reply. The output is just to show what happen after I transfer a file from OS X to XFS that has EA bigger then 64k(I think). When I try for example to rsync this file from OS X to Linux XFS I get this error:
rsync: rsync_xal_set: lsetxattr(""/srv/nas/home/apple_bak_rsync/xxxxxx/Pictures/iPhoto Library/Database/BigBlobs.apdb"","user.com.apple.FinderInfo") failed: Operation not permitted (1)

But also rsync can give this error.
rsync: rsync_xal_set: lsetxattr(""/srv/danne/extern2/1000_EXT/2013/2013-03-05/IMG_6872-Edit.tif"","user.com.apple.ResourceFork") failed: Argument list too long (7)

Is this 2 errors related?

I will make a bug report for rsync also that it should not try to copy files with EA bigger then the destination can handle. But it would be great if XFS could handle this files and be fully compatible with OS X backups.

//Martin

[-- Attachment #1.2: Type: text/html, Size: 3545 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Extended attributes limit in Linux

[Jeff Liu]

On 02/02 2014 22:33 PM, Sun_Blood wrote:
> 
> On 1 feb 2014, at 15:08, Vyacheslav Dubeyko <slava@dubeyko.com
> <mailto:slava@dubeyko.com>> wrote:
> 
>>
>> On Jan 31, 2014, at 10:25 PM, Sun_Blood wrote:
>>
>>>
>>> On 31 jan 2014, at 15:33, Jeff Liu <jeff.liu@oracle.com
>>> <mailto:jeff.liu@oracle.com>> wrote:
>>>
>>>>
>>>> On 01/31 2014 22:21 PM, Vyacheslav Dubeyko wrote:
>>>>> On Fri, 2014-01-31 at 21:39 +0800, Jeff Liu wrote:
>>>>>
>>>>>>>
>>>>>>> I checked the same under Mac OS X 10.6.8 (Snow Leopard). And I have
>>>>>>> failed on 3803 bytes size of xattr. So, I suppose that you have
>>>>>>> Mac OS X
>>>>>>> Lion. And EAs is larger under Lion yet.
>>>>>>>
>>>>>>> What version of Mac OS X have you?
>>>>>>>
>>>>>> Yup, Mountain Lion v10.8.4 :)
>>>>>>
>>>>>
>>>>> I suspect that xattrs with significant size is stored in compressed
>>>>> state on HFS+. I implemented support of compressed xattrs partially but
>>>>> I don't share this code yet. But, yes, EAs with size greater than 64 KB
>>>>> can be a problem.
>>>
>>>
>>> FYI, Example of output from one of the failing files. First from OS X
>>> and then same file after failed copy to XFS.
>>>
>>> OS X Maverik:
>>> file: "/Users/username/Pictures/iPhoto
>>> Library/Database/apdb/BigBlobs.apdb"
>>> type: "\0\0\0\0"
>>> creator: "\0\0\0\0"
>>> attributes: avbstclinmedz
>>> created: 01/25/2014 11:43:17
>>> modified: 01/28/2014 20:02:46
>>>
>>>
>>> Ubunutu 
>>> getfattr: Removing leading '/' from absolute path names
>>> # file: srv/nas/home/apple_bak_rsync/username/Pictures/iPhoto
>>> Library/Database/BigBlobs.apdb
>>> user.com.apple.quarantine="0006;52e39545;iPhoto;”
>>
>>
>> Sorry, but I don't quite follow your thought. What do you show by this
>> output?
>> What do you mean? Could you describe in more details?
>>
>> Thanks,
>> Vyacheslav Dubeyko.
> 
> Sorry late reply. The output is just to show what happen after I
> transfer a file from OS X to XFS that has EA bigger then 64k(I think).
> When I try for example to rsync this file from OS X to Linux XFS I get
> this error:
> rsync: rsync_xal_set:
> lsetxattr(""/srv/nas/home/apple_bak_rsync/xxxxxx/Pictures/iPhoto
> Library/Database/BigBlobs.apdb"","user.com.apple.FinderInfo") failed:
> Operation not permitted (1)
> 
> But also rsync can give this error.
> rsync: rsync_xal_set:
> lsetxattr(""/srv/danne/extern2/1000_EXT/2013/2013-03-05/IMG_6872-Edit.tif"","user.com.apple.ResourceFork")
> failed: Argument list too long (7)
> 
> Is this 2 errors related?

Those errors are unrelated IMO, the first one is due to the permission rules but
I'm not sure the root cause, the second one is occurred as the EA value is larger
than 64K I guess.
 
> 
> I will make a bug report for rsync also that it should not try to copy
> files with EA bigger then the destination can handle. But it would be
> great if XFS could handle this files and be fully compatible with OS X
> backups.

64K size is not limited by XFS directly, it is limited by VFS setxattr syscall.
IOWs, EA set operation is not yet get into XFS when "Argument list too long" error
is returned, so I think you would ran into the same error on other file systems
which are support 64K EA value size as well.

Thanks,
-Jeff

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Extended attributes limit in Linux

[Jeff Liu]

On 02/02 2014 23:12 PM, Jeff Liu wrote:
> 
> On 02/02 2014 22:33 PM, Sun_Blood wrote:
<snip>
>>>> FYI, Example of output from one of the failing files. First from OS X
>>>> and then same file after failed copy to XFS.
>>>>
>>>> OS X Maverik:
>>>> file: "/Users/username/Pictures/iPhoto
>>>> Library/Database/apdb/BigBlobs.apdb"
>>>> type: "\0\0\0\0"
>>>> creator: "\0\0\0\0"
>>>> attributes: avbstclinmedz
>>>> created: 01/25/2014 11:43:17
>>>> modified: 01/28/2014 20:02:46
>>>>
>>>>
>>>> Ubunutu 
>>>> getfattr: Removing leading '/' from absolute path names
>>>> # file: srv/nas/home/apple_bak_rsync/username/Pictures/iPhoto
>>>> Library/Database/BigBlobs.apdb
>>>> user.com.apple.quarantine="0006;52e39545;iPhoto;”
>>>
>>>
>>> Sorry, but I don't quite follow your thought. What do you show by this
>>> output?
>>> What do you mean? Could you describe in more details?
>>>
>>> Thanks,
>>> Vyacheslav Dubeyko.
>>
>> Sorry late reply. The output is just to show what happen after I
>> transfer a file from OS X to XFS that has EA bigger then 64k(I think).
>> When I try for example to rsync this file from OS X to Linux XFS I get
>> this error:
>> rsync: rsync_xal_set:
>> lsetxattr(""/srv/nas/home/apple_bak_rsync/xxxxxx/Pictures/iPhoto
>> Library/Database/BigBlobs.apdb"","user.com.apple.FinderInfo") failed:
>> Operation not permitted (1)
>>
>> But also rsync can give this error.
>> rsync: rsync_xal_set:
>> lsetxattr(""/srv/danne/extern2/1000_EXT/2013/2013-03-05/IMG_6872-Edit.tif"","user.com.apple.ResourceFork")
>> failed: Argument list too long (7)
>>
>> Is this 2 errors related?
> 
> Those errors are unrelated IMO, the first one is due to the permission rules but
> I'm not sure the root cause, the second one is occurred as the EA value is larger
> than 64K I guess.
>  
>>
>> I will make a bug report for rsync also that it should not try to copy
>> files with EA bigger then the destination can handle. But it would be
>> great if XFS could handle this files and be fully compatible with OS X
>> backups.
> 
> 64K size is not limited by XFS directly, it is limited by VFS setxattr syscall.
> IOWs, EA set operation is not yet get into XFS when "Argument list too long" error
> is returned, so I think you would ran into the same error on other file systems
> which are support 64K EA value size as well.

To be more precise, larger EA set operation would not works via setxattr(2) on Linux
regardless of the underlying file systems :).

Thanks,
-Jeff

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Extended attributes limit in Linux

[Sun_Blood]

[-- Attachment #1.1: Type: text/plain, Size: 2906 bytes --]


On 2 feb 2014, at 16:22, Jeff Liu <jeff.liu@oracle.com> wrote:

> 
> On 02/02 2014 23:12 PM, Jeff Liu wrote:
>> 
>> On 02/02 2014 22:33 PM, Sun_Blood wrote:
> <snip>
>>>>> FYI, Example of output from one of the failing files. First from OS X
>>>>> and then same file after failed copy to XFS.
>>>>> 
>>>>> OS X Maverik:
>>>>> file: "/Users/username/Pictures/iPhoto
>>>>> Library/Database/apdb/BigBlobs.apdb"
>>>>> type: "\0\0\0\0"
>>>>> creator: "\0\0\0\0"
>>>>> attributes: avbstclinmedz
>>>>> created: 01/25/2014 11:43:17
>>>>> modified: 01/28/2014 20:02:46
>>>>> 
>>>>> 
>>>>> Ubunutu 
>>>>> getfattr: Removing leading '/' from absolute path names
>>>>> # file: srv/nas/home/apple_bak_rsync/username/Pictures/iPhoto
>>>>> Library/Database/BigBlobs.apdb
>>>>> user.com.apple.quarantine="0006;52e39545;iPhoto;”
>>>> 
>>>> 
>>>> Sorry, but I don't quite follow your thought. What do you show by this
>>>> output?
>>>> What do you mean? Could you describe in more details?
>>>> 
>>>> Thanks,
>>>> Vyacheslav Dubeyko.
>>> 
>>> Sorry late reply. The output is just to show what happen after I
>>> transfer a file from OS X to XFS that has EA bigger then 64k(I think).
>>> When I try for example to rsync this file from OS X to Linux XFS I get
>>> this error:
>>> rsync: rsync_xal_set:
>>> lsetxattr(""/srv/nas/home/apple_bak_rsync/xxxxxx/Pictures/iPhoto
>>> Library/Database/BigBlobs.apdb"","user.com.apple.FinderInfo") failed:
>>> Operation not permitted (1)
>>> 
>>> But also rsync can give this error.
>>> rsync: rsync_xal_set:
>>> lsetxattr(""/srv/danne/extern2/1000_EXT/2013/2013-03-05/IMG_6872-Edit.tif"","user.com.apple.ResourceFork")
>>> failed: Argument list too long (7)
>>> 
>>> Is this 2 errors related?
>> 
>> Those errors are unrelated IMO, the first one is due to the permission rules but
>> I'm not sure the root cause, the second one is occurred as the EA value is larger
>> than 64K I guess.
>> 
>>> 
>>> I will make a bug report for rsync also that it should not try to copy
>>> files with EA bigger then the destination can handle. But it would be
>>> great if XFS could handle this files and be fully compatible with OS X
>>> backups.
>> 
>> 64K size is not limited by XFS directly, it is limited by VFS setxattr syscall.
>> IOWs, EA set operation is not yet get into XFS when "Argument list too long" error
>> is returned, so I think you would ran into the same error on other file systems
>> which are support 64K EA value size as well.
> 
> To be more precise, larger EA set operation would not works via setxattr(2) on Linux
> regardless of the underlying file systems :).

Correct I did a quick test on ext4 and got the same fault.

> 
> Thanks,
> -Jeff

So to be able to support this type of EA that OS X produce we need a change in the Linux kernel? Because XFS does handle it and is not the problem here.

[-- Attachment #1.2: Type: text/html, Size: 4060 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Extended attributes limit in Linux

[Chris Murphy]

On Feb 2, 2014, at 7:33 AM, Sun_Blood <sblood@gmail.com> wrote:
> I will make a bug report for rsync also that it should not try to copy files with EA bigger then the destination can handle. But it would be great if XFS could handle this files and be fully compatible with OS X backups.

What application(s) are creating such files on OS X? Or how are they coming to have such large extended attributes?


Chris Murphy
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Extended attributes limit in Linux

[Sun_Blood]

[-- Attachment #1.1: Type: text/plain, Size: 578 bytes --]

On Mon, Feb 3, 2014 at 2:15 AM, Chris Murphy <lists@colorremedies.com>wrote:

>
> On Feb 2, 2014, at 7:33 AM, Sun_Blood <sblood@gmail.com> wrote:
> > I will make a bug report for rsync also that it should not try to copy
> files with EA bigger then the destination can handle. But it would be great
> if XFS could handle this files and be fully compatible with OS X backups.
>
> What application(s) are creating such files on OS X? Or how are they
> coming to have such large extended attributes?
>
>
> Chris Murphy


They are created by Adobe Photoshop and/or iPhoto.

/Martin

[-- Attachment #1.2: Type: text/html, Size: 1066 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Extended attributes limit in Linux

[Chris Murphy]

[-- Attachment #1.1: Type: text/plain, Size: 1903 bytes --]


On Feb 3, 2014, at 12:14 AM, Sun_Blood <sblood@gmail.com> wrote:

> 
> On Mon, Feb 3, 2014 at 2:15 AM, Chris Murphy <lists@colorremedies.com> wrote:
> 
> On Feb 2, 2014, at 7:33 AM, Sun_Blood <sblood@gmail.com> wrote:
> > I will make a bug report for rsync also that it should not try to copy files with EA bigger then the destination can handle. But it would be great if XFS could handle this files and be fully compatible with OS X backups.
> 
> What application(s) are creating such files on OS X? Or how are they coming to have such large extended attributes?
> 
> 
> Chris Murphy
> 
> They are created by Adobe Photoshop and/or iPhoto.

I'm finding that Photoshop CS6 and older use resource fork to store thumbnail previews, but with API changes occurring since OS X 10.4. such requests for the resource fork actually end up as an extended attribute. This appears with xattr -l as com.apple.ResourceFork. In two example files they are about 50KB each.

Photoshop CC doesn't create thumbnails using resource forks anymore, thus this extended attribute isn't created. Plus the OS X Finder creates thumbnail previews directly from supported file types, even if a thumbnail preview resource isn't available. And last, the Resource Manager is deprecated as of OS X 10.8.x, so I'd expect even com.apple.ResourceFork to eventually go away (i.e. any application wanting to write a resource to the resource fork will fail).
https://developer.apple.com/legacy/library/documentation/Carbon/Reference/Resource_Manager/Reference/reference.html

The way I'd characterize data in com.apple.ResourceFork is that it's disposable data. That is, any application developer who cared about data, wouldn't store it in com.apple.ResourceFork. And this isn't a recent sentiment.

Are you seeing significant extended attribute data other than com.apple.ResourceFork in these files?


Chris Murphy

[-- Attachment #1.2: Type: text/html, Size: 2973 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Extended attributes limit in Linux

[Dave Chinner]

On Fri, Jan 31, 2014 at 04:54:52PM +0800, Jeff Liu wrote:
> Hello,
> 
> On 01/31 2014 15:40 PM, Sun_Blood wrote:
> > Hello,
> > 
> > If I understands it correctly XFS don't have a limit to the size of
> > extended attributes(EA) but Linux impose a limit at 64k.
> > What I am trying to do is build a backup server that our Apple computers
> > will use together with rsync to backup files to. The problem I face is
> > that Apple HFS+ don't have a limit to EA so it has files with more then
> > 64k of EA in it.
> > 
> > The Linux Kernel has a limit imposed to it in include/linux/limits.h
> > 
> >     #defineXATTR_SIZE_MAX 65536   /* size of an extended attribute value
> >     (64k) */
> > 
> >     #defineXATTR_LIST_MAX 65536   /* size of extended attribute namelist
> >     (64k) */
> > 
> 
> Yes, 64k is the VFS limit per EA value size.

XFS has a limit of 64k as well. I suspect that we could handle
larger EAs without too much problem as we already have the concept
of a "remote EA" that is allocated outside of the attribute btree
(but still inside the attribute address space). The problem with
them is that they are written synchronously, and so large attributes
written this way are slow operations.

Still, until the VFS limit is lifted and we do a bunch more work to
avoid XATTR_SIZE_MAX allocations everywhere there's no point looking
at raising the XFS limit....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Extended attributes limit in Linux

[Sun_Blood]

[-- Attachment #1.1: Type: text/plain, Size: 1816 bytes --]

On Sun, Feb 2, 2014 at 10:37 PM, Dave Chinner <david@fromorbit.com> wrote:

> On Fri, Jan 31, 2014 at 04:54:52PM +0800, Jeff Liu wrote:
> > Hello,
> >
> > On 01/31 2014 15:40 PM, Sun_Blood wrote:
> > > Hello,
> > >
> > > If I understands it correctly XFS don't have a limit to the size of
> > > extended attributes(EA) but Linux impose a limit at 64k.
> > > What I am trying to do is build a backup server that our Apple
> computers
> > > will use together with rsync to backup files to. The problem I face is
> > > that Apple HFS+ don't have a limit to EA so it has files with more then
> > > 64k of EA in it.
> > >
> > > The Linux Kernel has a limit imposed to it in include/linux/limits.h
> > >
> > >     #defineXATTR_SIZE_MAX 65536   /* size of an extended attribute
> value
> > >     (64k) */
> > >
> > >     #defineXATTR_LIST_MAX 65536   /* size of extended attribute
> namelist
> > >     (64k) */
> > >
> >
> > Yes, 64k is the VFS limit per EA value size.
>
> XFS has a limit of 64k as well. I suspect that we could handle
> larger EAs without too much problem as we already have the concept
> of a "remote EA" that is allocated outside of the attribute btree
> (but still inside the attribute address space). The problem with
> them is that they are written synchronously, and so large attributes
> written this way are slow operations.
>
> Still, until the VFS limit is lifted and we do a bunch more work to
> avoid XATTR_SIZE_MAX allocations everywhere there's no point looking
> at raising the XFS limit....
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com
>

Ok. Just for fun how could we proceed to get the VFS limit lifted? Or any
other way to handle this problem?
It would be nice to be able to say that XFS is the only file system
available fully compatible with OS X HFS+ =)

/Martin

[-- Attachment #1.2: Type: text/html, Size: 2621 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs