Any better way to interact with xfs?

Any better way to interact with xfs?

[Ryan Lindsay]

[-- Attachment #1.1: Type: text/plain, Size: 1325 bytes --]

Hi

I have a bit of a specific requirement.

I have a server that currently has a 12T worth of data on it on an xfs formatted LV.

currently the file permissions are set via an LDAP, however we are having to move to AD for out authentication mechanism. Which will change the value of the existing UID's and GID's

Since XFS has a lot of mechanisms that allow things to happen in parallel. I was wondering if there was a way of interacting at the file system level to facilitate a change of UID's on files rather than having to just chown recursively down the file system. The faster I can update the permissions the better basically


Ryan Lindsay | Research Infrastructure Lead

Faculty of Science IT
Room G09, Old Geology Bld, University of Melbourne, 3010, VIC
Telephone +61 3 83446255 | Mobile +61 0407537949 | Email rlindsay@unimelb.edu.au

This email and any attachments may contain personal information or information that is otherwise confidential or the subject of copyright. Any use, disclosure or copying of any part of it is prohibited. The University does not warrant that this email or any attachments are free from viruses or defects. Please check any attachments for viruses and defects before opening them. If this email is received in error please delete it and notify us by return email.

[-- Attachment #1.2: Type: text/html, Size: 4283 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Any better way to interact with xfs?

[Eric Sandeen]

On 7/27/16 10:30 PM, Ryan Lindsay wrote:
> Hi
> 
> I have a bit of a specific requirement.
> 
> I have a server that currently has a 12T worth of data on it on an
> xfs formatted LV.
> 
> currently the file permissions are set via an LDAP, however we are
> having to move to AD for out authentication mechanism. Which will
> change the value of the existing UID's and GID's
> 
> Since XFS has a lot of mechanisms that allow things to happen in
> parallel. I was wondering if there was a way of interacting at the
> file system level to facilitate a change of UID's on files rather
> than having to just chown recursively down the file system. The
> faster I can update the permissions the better basically

Hi Ryan - I'm afraid I don't think there is any lower-level mechanism
to replace permissions, just the normal posix syscalls and
standard tools to invoke them.

You can't convince AD to keep the same numbers?  :)

-Eric

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Any better way to interact with xfs?

[Amir Goldstein]

Hi Ryan,

You could use the XFS_IOC_FSINUMBERS/XFS_IOC_FSBULKSTAT API to iterate
over all inodes in the fs.
not having to readdir and recourse the directory tree should safe you
some time (much less i/o).
Also, the interface can be used to make your conversion work parallel
by working on different inode ranges.
You can use xfsdump/xfsrestore code as reference.
Also, not sure if you are using user/group quotas - if you don't,
better make sure that quotas are turned off
to avoid all the accounting caused by your conversion process.

Amir.

On Thu, Jul 28, 2016 at 6:30 AM, Ryan Lindsay <rlindsay@unimelb.edu.au> wrote:
> Hi
>
> I have a bit of a specific requirement.
>
> I have a server that currently has a 12T worth of data on it on an xfs
> formatted LV.
>
> currently the file permissions are set via an LDAP, however we are having to
> move to AD for out authentication mechanism. Which will change the value of
> the existing UID's and GID's
>
> Since XFS has a lot of mechanisms that allow things to happen in parallel. I
> was wondering if there was a way of interacting at the file system level to
> facilitate a change of UID's on files rather than having to just chown
> recursively down the file system. The faster I can update the permissions
> the better basically
>
>
> Ryan Lindsay | Research Infrastructure Lead
>
> Faculty of Science IT
> Room G09, Old Geology Bld, University of Melbourne, 3010, VIC
> Telephone +61 3 83446255 | Mobile +61 0407537949 | Email
> rlindsay@unimelb.edu.au
>
> This email and any attachments may contain personal information or
> information that is otherwise confidential or the subject of copyright. Any
> use, disclosure or copying of any part of it is prohibited. The University
> does not warrant that this email or any attachments are free from viruses or
> defects. Please check any attachments for viruses and defects before opening
> them. If this email is received in error please delete it and notify us by
> return email.
>
>
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
>

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Any better way to interact with xfs?

[Dave Chinner]

On Thu, Jul 28, 2016 at 09:48:07AM +0300, Amir Goldstein wrote:
> Hi Ryan,
> 
> You could use the XFS_IOC_FSINUMBERS/XFS_IOC_FSBULKSTAT API to iterate
> over all inodes in the fs.

That's a read-only API - you can't use it to change the inodes
on disk.

> not having to readdir and recourse the directory tree should safe you
> some time (much less i/o).
> Also, the interface can be used to make your conversion work parallel
> by working on different inode ranges.

It cannot be used to make coherent, atomic changes to the inode
state.

> You can use xfsdump/xfsrestore code as reference.

xfsdump uses bulkstat scan and read inodes, not change anything on
disk.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Any better way to interact with xfs?

[Amir Goldstein]

On Fri, Jul 29, 2016 at 5:25 AM, Dave Chinner <david@fromorbit.com> wrote:
> On Thu, Jul 28, 2016 at 09:48:07AM +0300, Amir Goldstein wrote:
>> Hi Ryan,
>>
>> You could use the XFS_IOC_FSINUMBERS/XFS_IOC_FSBULKSTAT API to iterate
>> over all inodes in the fs.
>
> That's a read-only API - you can't use it to change the inodes
> on disk.
>
>> not having to readdir and recourse the directory tree should safe you
>> some time (much less i/o).
>> Also, the interface can be used to make your conversion work parallel
>> by working on different inode ranges.
>
> It cannot be used to make coherent, atomic changes to the inode
> state.
>

I'm curios: In theory, one can construct a file_handle from bulkstat
info. correct?
and one could use that handle to open_by_handle_at() and fchown(). right?
I realize that both APIs were not intended for this use case, but I wonder:
1. can it be done?
2. what are the possible consequences?
3. what do you mean by "coherent, atomic changes to the inode state"?
does it not play well with the dentry/inode cache?

>> You can use xfsdump/xfsrestore code as reference.
>
> xfsdump uses bulkstat scan and read inodes, not change anything on
> disk.
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Any better way to interact with xfs?

[Dave Chinner]

On Sat, Jul 30, 2016 at 05:26:03PM +0300, Amir Goldstein wrote:
> On Fri, Jul 29, 2016 at 5:25 AM, Dave Chinner <david@fromorbit.com> wrote:
> > On Thu, Jul 28, 2016 at 09:48:07AM +0300, Amir Goldstein wrote:
> >> Hi Ryan,
> >>
> >> You could use the XFS_IOC_FSINUMBERS/XFS_IOC_FSBULKSTAT API to iterate
> >> over all inodes in the fs.
> >
> > That's a read-only API - you can't use it to change the inodes
> > on disk.
> >
> >> not having to readdir and recourse the directory tree should safe you
> >> some time (much less i/o).
> >> Also, the interface can be used to make your conversion work parallel
> >> by working on different inode ranges.
> >
> > It cannot be used to make coherent, atomic changes to the inode
> > state.
> >
> 
> I'm curios: In theory, one can construct a file_handle from bulkstat
> info. correct?
> and one could use that handle to open_by_handle_at() and fchown(). right?
> I realize that both APIs were not intended for this use case, but I wonder:
> 1. can it be done?

Yes. See xfs_fsr.

> 2. what are the possible consequences?

You can modify the data any file in the filesystem without leaving a
trace. Files opened this way do not get their c/mtime changed when
modified by syscalls, and only on CRC enabled filesystem is it
possible to tell if the inode metadata was changed this way. You
still can't tell if file data was changed.

> 3. what do you mean by "coherent, atomic changes to the inode state"?
> does it not play well with the dentry/inode cache?

The inode can change between bulkstat time (i.e. the info the kernel
returned about the inode) and the time you have an open fd and
make the change you want to make. It could have been renamed,
it could have been truncated, it's permissions could have been
changed, etc. None of these sorts of checks are run when opening by
filehandle.

i.e. it's all the same coherency issues that we have with multiple
NFS clients modifying the same file concurrently. They don't know
what each other are doing, and so modifications are going to get
lost or be overwritten incorrectly....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Any better way to interact with xfs?

[Amir Goldstein]

On Sun, Jul 31, 2016 at 3:34 AM, Dave Chinner <david@fromorbit.com> wrote:
> On Sat, Jul 30, 2016 at 05:26:03PM +0300, Amir Goldstein wrote:
>> On Fri, Jul 29, 2016 at 5:25 AM, Dave Chinner <david@fromorbit.com> wrote:
>> > On Thu, Jul 28, 2016 at 09:48:07AM +0300, Amir Goldstein wrote:
>> >> Hi Ryan,
>> >>
>> >> You could use the XFS_IOC_FSINUMBERS/XFS_IOC_FSBULKSTAT API to iterate
>> >> over all inodes in the fs.
>> >
>> > That's a read-only API - you can't use it to change the inodes
>> > on disk.
>> >
>> >> not having to readdir and recourse the directory tree should safe you
>> >> some time (much less i/o).
>> >> Also, the interface can be used to make your conversion work parallel
>> >> by working on different inode ranges.
>> >
>> > It cannot be used to make coherent, atomic changes to the inode
>> > state.
>> >
>>
>> I'm curios: In theory, one can construct a file_handle from bulkstat
>> info. correct?
>> and one could use that handle to open_by_handle_at() and fchown(). right?
>> I realize that both APIs were not intended for this use case, but I wonder:
>> 1. can it be done?
>
> Yes. See xfs_fsr.
>
>> 2. what are the possible consequences?
>
> You can modify the data any file in the filesystem without leaving a
> trace. Files opened this way do not get their c/mtime changed when
> modified by syscalls, and only on CRC enabled filesystem is it
> possible to tell if the inode metadata was changed this way. You
> still can't tell if file data was changed.
>
>> 3. what do you mean by "coherent, atomic changes to the inode state"?
>> does it not play well with the dentry/inode cache?
>
> The inode can change between bulkstat time (i.e. the info the kernel
> returned about the inode) and the time you have an open fd and
> make the change you want to make. It could have been renamed,
> it could have been truncated, it's permissions could have been
> changed, etc. None of these sorts of checks are run when opening by
> filehandle.
>
> i.e. it's all the same coherency issues that we have with multiple
> NFS clients modifying the same file concurrently. They don't know
> what each other are doing, and so modifications are going to get
> lost or be overwritten incorrectly....

All the disclaimers above are very important for the administrator to know.
But I believe that the answer to Ryan's original question is:
Yes, there is a way of interacting at the file system level to
facilitate a change
of UID's on files rather than having to just chown recursively down
the file system.
I'm not sure how much faster it is going to be and whether anyone would want
to invest time in writing this bulkchown tool, but it is definitely
going to reduce
Ryan's system downtime.
Please correct me if I am wrong.

Amir.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Any better way to interact with xfs?

[Dave Chinner]

On Sun, Jul 31, 2016 at 09:12:23AM +0300, Amir Goldstein wrote:
> On Sun, Jul 31, 2016 at 3:34 AM, Dave Chinner <david@fromorbit.com> wrote:
> > i.e. it's all the same coherency issues that we have with multiple
> > NFS clients modifying the same file concurrently. They don't know
> > what each other are doing, and so modifications are going to get
> > lost or be overwritten incorrectly....
> 
> All the disclaimers above are very important for the administrator to know.

Administrator != developer.

These are things a filesystem tool developer would need to know.
They are far, far outside the realm of what a system administrator
needs to know or wants to care about.

> But I believe that the answer to Ryan's original question is:
> Yes, there is a way of interacting at the file system level to
> facilitate a change
> of UID's on files rather than having to just chown recursively down
> the file system.

It's not supported - you'll get to keep all the broken bits to
yourself if you do something like this. You get to handle all the
undocumented wacky corner cases yourself, like unlinked inodes,
inodes that the scan misses because they were created in a region
already scanned before the bulkstat completes, handling partial
completion failures, invalidation of all your backups because they
can't obviously detect that changes have been made, etc.

> I'm not sure how much faster it is going to be and whether anyone
> would want to invest time in writing this bulkchown tool, but it
> is definitely going to reduce Ryan's system downtime.

Is it, though? You say it like it's an unconditional win, but that's
nt necessarily true. i.e. if directory travesal isn't the limiting
factor (e.g. single threaded chown process is the limiting factor),
then simply running concurrent chowns is will be the fastest and
simplest method to scale this out.

> Please correct me if I am wrong.

Bulkstat is not a general purpose API. You can use it if you want
to, but if you don't understand /exactly/ what it is providing, then
the application that uses it wll be buggy and quite possibly
dangerous to the user's data and/or filesystem structure.

You're welcome to play russian roulette with your own filesystems,
but please don't advocate that others should do play along with
you...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Any better way to interact with xfs?

[Dave Chinner]

On Thu, Jul 28, 2016 at 03:30:16AM +0000, Ryan Lindsay wrote:
> Hi
> 
> I have a bit of a specific requirement.
> 
> I have a server that currently has a 12T worth of data on it on an
> xfs formatted LV.
> 
> currently the file permissions are set via an LDAP, however we are
> having to move to AD for out authentication mechanism. Which will
> change the value of the existing UID's and GID's
> 
> Since XFS has a lot of mechanisms that allow things to happen in
> parallel. I was wondering if there was a way of interacting at the
> file system level to facilitate a change of UID's on files rather
> than having to just chown recursively down the file system. The
> faster I can update the permissions the better basically

The directory structure can be modified in parallel, so simply run
multiple recursive chowns down separate branches of the directory
tree on the filesystem. That will run as fast as your IO subsystem
will let it.

e.g. I can remove a 16-subdirectory wide tree with 50 million files
in it using 16 "rm -rf subdir.$i" commands in about 4 minutes. It
runs at 12,000 read IOPS doing it that. i.e. it runs as fast as the
underlying storage can read the directory and inodes into cache. 

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs