[Lustre-devel] Stripe offset default

[Lustre-devel] Stripe offset default

[Eric Barton]

Nick Cardo at NERSC mentioned at the LUG, and again when I chatted
with him at SC10, that setting the default stripe offset is positively
dangerous.  I think he has a point.

Setting file layout precisely can be essential for applications that
know precisely what they're doing and want to control how aggregate
OST load is balanced.  But the default file layout is another matter
entirely.  Setting the default to a fixed layout, or even just a fixed
starting OST has got to be the most direct path to achieving OST load
imbalance.

I believe, now that lfs setstripe uses options rather than positional
parameters, that the risk of unintentional errors is much less.  But
that still leaves the filesystem vulnerable to na?ve or malicious
users.  So I propose that we should continue to check permissions
normally when setting the default stripe _count_, but restrict the
ability to set a specific default stripe _offset_ to the superuser.

Thoughts?

P.S. Do people also agree that setting user/group/ACL permissions on
OST pools would be a Good Thing?

          Cheers,
                   Eric

Eric Barton
CTO Whamcloud, Inc.

[Lustre-devel] Stripe offset default

[Christopher J. Morrone]

On 11/23/2010 09:20 AM, Eric Barton wrote:
>

> I believe, now that lfs setstripe uses options rather than positional
> parameters, that the risk of unintentional errors is much less.  But
> that still leaves the filesystem vulnerable to na?ve or malicious
> users.  So I propose that we should continue to check permissions
> normally when setting the default stripe _count_, but restrict the
> ability to set a specific default stripe _offset_ to the superuser.
>
> Thoughts?

At the very least bug 18547 MUST be completed before doing this. 
Lustre's default layout mechanism is too uneven, and leads to too much 
of a performance problem right now to take away users' only ability to 
lay files out evenly on their own.

How much of a problem, really, are naive/malicious users?

Malicious users can easily fill the OST of their choice with a tiny 
script that creates thousands of empty files, and then fills only the 
files that happened to land on the target OST.

Naive users can fill one or a few OSTs by simply writing really large 
files.  We've had that happen when a user, for instance, decides to 
create a really large tar file without first creating a widely striped 
file to hole the tar date.

But taking away our ability specify file offset where we want them 
definitely hurts testing and debugging efforts.  Testing reproducibility 
is enhanced when we can reproducibly lay out files.  Debugging problems 
on a live system is sometimes made easier by specifying the file offset, 
e.g. when we are trying to target certain network switches or OSS nodes.

And when a problem DOES occur, sysadmins now have lustre quotas to help 
them quickly identify the culprit.

Since the problem user can quickly be identified, and since the users 
can still pretty easily cause the same trouble using other mechanisms, I 
don't think that the benefits necessarily outweigh the disadvantages.

If someone decides that they really, really want this, then I think it 
should be tunable to allow the sysadmin to select whether or not setting 
file offset is restricted to superuser.

Chris

[Lustre-devel] Stripe offset default

[Andreas Dilger]

On 2010-11-23, at 12:49, Christopher J. Morrone wrote:
> On 11/23/2010 09:20 AM, Eric Barton wrote:
>> I believe, now that lfs setstripe uses options rather than positional
>> parameters, that the risk of unintentional errors is much less.  But
>> that still leaves the filesystem vulnerable to na?ve or malicious
>> users.  So I propose that we should continue to check permissions
>> normally when setting the default stripe _count_, but restrict the
>> ability to set a specific default stripe _offset_ to the superuser.
> 
> At the very least bug 18547 MUST be completed before doing this. 
> Lustre's default layout mechanism is too uneven, and leads to too much 
> of a performance problem right now to take away users' only ability to 
> lay files out evenly on their own.

I guess the subtle distinction in Eric's question is for "default striping parameters specified on a directory" and not "striping parameters set on a per-file basis".  In IRC I discussed with Eric that it is like that specifying the starting OST on a directory default layout is _probably_ not what a user wants to do.

I think the majority of the confusion was due to the (archaic and long obsolete) usage of positional "lfs setstripe {size} {offset} {count}" parameters, when in fact most users just want to specify {count} and neither of the others.  The {size} and {offset} values would get various strange "default" values, depending on whether the user remembers their meanings.  This accidentally caused many users to specify OST 0 as the starting offset for their files, instead of using "-1" which means "let the MDS decide which OSTs to use".

Instead, the named options "--size/-s", "--offset/-o" (identical to "--index/-i"), and "--count/-c" to "lfs setstripe" have been available since 1.6.1, but it was only in 1.8.0 that using the old positional parameters would generate a warning, and in 2.1.0 this usage will generate an error.

> How much of a problem, really, are naive/malicious users?

IMHO, limiting the ioctl() to root users is at best obfuscating the issue, and not really solving the problem.  In either case, it doesn't help if the users are running ancient versions of Lustre where the problem has not been fixed.

> Malicious users can easily fill the OST of their choice with a tiny 
> script that creates thousands of empty files, and then fills only the 
> files that happened to land on the target OST.
> 
> Naive users can fill one or a few OSTs by simply writing really large 
> files.  We've had that happen when a user, for instance, decides to 
> create a really large tar file without first creating a widely striped 
> file to hole the tar date.
> 
> But taking away our ability specify file offset where we want them 
> definitely hurts testing and debugging efforts.  Testing reproducibility 
> is enhanced when we can reproducibly lay out files.  Debugging problems 
> on a live system is sometimes made easier by specifying the file offset, 
> e.g. when we are trying to target certain network switches or OSS nodes.
> 
> And when a problem DOES occur, sysadmins now have lustre quotas to help 
> them quickly identify the culprit.
> 
> Since the problem user can quickly be identified, and since the users 
> can still pretty easily cause the same trouble using other mechanisms, I 
> don't think that the benefits necessarily outweigh the disadvantages.
> 
> If someone decides that they really, really want this, then I think it 
> should be tunable to allow the sysadmin to select whether or not setting 
> file offset is restricted to superuser.
> 
> Chris
> _______________________________________________
> Lustre-devel mailing list
> Lustre-devel at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-devel


Cheers, Andreas
--
Andreas Dilger
Lustre Technical Lead
Oracle Corporation Canada Inc.

[Lustre-devel] Stripe offset default

[Christopher J. Morrone]

On 11/23/2010 01:19 PM, Andreas Dilger wrote:
> On 2010-11-23, at 12:49, Christopher J. Morrone wrote:
>> On 11/23/2010 09:20 AM, Eric Barton wrote:
>>> I believe, now that lfs setstripe uses options rather than positional
>>> parameters, that the risk of unintentional errors is much less.  But
>>> that still leaves the filesystem vulnerable to na?ve or malicious
>>> users.  So I propose that we should continue to check permissions
>>> normally when setting the default stripe _count_, but restrict the
>>> ability to set a specific default stripe _offset_ to the superuser.
>>
>> At the very least bug 18547 MUST be completed before doing this.
>> Lustre's default layout mechanism is too uneven, and leads to too much
>> of a performance problem right now to take away users' only ability to
>> lay files out evenly on their own.
>
> I guess the subtle distinction in Eric's question is for "default striping parameters specified on a directory" and not "striping parameters set on a per-file basis".  In IRC I discussed with Eric that it is like that specifying the starting OST on a directory default layout is _probably_ not what a user wants to do.

Ah, ok, I did miss that.  I am less opposed to that, since I can't think 
of a particularly compelling use case for a fixed offset default on 
directories.

[Lustre-devel] Stripe offset default

[Ashley Pittman]

On 23 Nov 2010, at 17:20, Eric Barton wrote:

> Nick Cardo at NERSC mentioned at the LUG, and again when I chatted
> with him at SC10, that setting the default stripe offset is positively
> dangerous.  I think he has a point.
> 
> Setting file layout precisely can be essential for applications that
> know precisely what they're doing and want to control how aggregate
> OST load is balanced.  But the default file layout is another matter
> entirely.  Setting the default to a fixed layout, or even just a fixed
> starting OST has got to be the most direct path to achieving OST load
> imbalance.

Are you talking about the filesystem default (if indeed this can be set) or the setting for directories or for files?  For filesystem default I think you're right, for files it seems like a bad thing to restrict users like this and for directories I'm on the fence but leaning towards "users shouldn't need to do that".

> Thoughts?
> 
> P.S. Do people also agree that setting user/group/ACL permissions on
> OST pools would be a Good Thing?

I think pools could be wonderful once there is the option of moving files between pools dynamically, until then they are of little use IMHO so I'd much rather see that fixed first.

Ashley,

-- 

Ashley Pittman, Bath, UK.

Padb - A parallel job inspection tool for cluster computing
http://padb.pittman.org.uk

[Lustre-devel] Stripe offset default

[Peter Braam]

The  proposal for acl / Group / users violates Posix, and may lead to results users don't expect.  Sgid directories and inheritance on ACLs are safer I think.

Peter

Sent from my iPad

On Nov 23, 2010, at 10:20, "Eric Barton" <eeb@whamcloud.com> wrote:

> 
> Nick Cardo at NERSC mentioned at the LUG, and again when I chatted
> with him at SC10, that setting the default stripe offset is positively
> dangerous.  I think he has a point.
> 
> Setting file layout precisely can be essential for applications that
> know precisely what they're doing and want to control how aggregate
> OST load is balanced.  But the default file layout is another matter
> entirely.  Setting the default to a fixed layout, or even just a fixed
> starting OST has got to be the most direct path to achieving OST load
> imbalance.
> 
> I believe, now that lfs setstripe uses options rather than positional
> parameters, that the risk of unintentional errors is much less.  But
> that still leaves the filesystem vulnerable to na?ve or malicious
> users.  So I propose that we should continue to check permissions
> normally when setting the default stripe _count_, but restrict the
> ability to set a specific default stripe _offset_ to the superuser.
> 
> Thoughts?
> 
> P.S. Do people also agree that setting user/group/ACL permissions on
> OST pools would be a Good Thing?
> 
>          Cheers,
>                   Eric
> 
> Eric Barton
> CTO Whamcloud, Inc.
> 
> 
> _______________________________________________
> Lustre-devel mailing list
> Lustre-devel at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-devel
> ______________________________________________________________________
> This email may contain privileged or confidential information, which should only be used for the purpose for which it was sent by Xyratex. No further rights or licenses are granted to use such information. If you are not the intended recipient of this message, please notify the sender by return and delete it. You may not use, copy, disclose or rely on the information contained in it.
> 
> Internet email is susceptible to data corruption, interception and unauthorised amendment for which Xyratex does not accept liability. While we have taken reasonable precautions to ensure that this email is free of viruses, Xyratex does not accept liability for the presence of any computer viruses in this email, nor for any losses caused as a result of viruses.
> 
> Xyratex Technology Limited (03134912), Registered in England & Wales, Registered Office, Langstone Road, Havant, Hampshire, PO9 1SA.
> 
> The Xyratex group of companies also includes, Xyratex Ltd, registered in Bermuda, Xyratex International Inc, registered in California, Xyratex (Malaysia) Sdn Bhd registered in Malaysia, Xyratex Technology (Wuxi) Co Ltd registered in The People's Republic of China and Xyratex Japan Limited registered in Japan.
> ______________________________________________________________________
> 
> 
______________________________________________________________________
This email may contain privileged or confidential information, which should only be used for the purpose for which it was sent by Xyratex. No further rights or licenses are granted to use such information. If you are not the intended recipient of this message, please notify the sender by return and delete it. You may not use, copy, disclose or rely on the information contained in it.
 
Internet email is susceptible to data corruption, interception and unauthorised amendment for which Xyratex does not accept liability. While we have taken reasonable precautions to ensure that this email is free of viruses, Xyratex does not accept liability for the presence of any computer viruses in this email, nor for any losses caused as a result of viruses.
 
Xyratex Technology Limited (03134912), Registered in England & Wales, Registered Office, Langstone Road, Havant, Hampshire, PO9 1SA.
 
The Xyratex group of companies also includes, Xyratex Ltd, registered in Bermuda, Xyratex International Inc, registered in California, Xyratex (Malaysia) Sdn Bhd registered in Malaysia, Xyratex Technology (Wuxi) Co Ltd registered in The People's Republic of China and Xyratex Japan Limited registered in Japan.
______________________________________________________________________
 

[Lustre-devel] Stripe offset default

[Andreas Dilger]

On 2010-11-23, at 13:50, Peter Braam wrote:
> The  proposal for acl / Group / users violates Posix, and may lead to results users don't expect.  Sgid directories and inheritance on ACLs are safer I think.

I think you misunderstood Eric's comment.  He wasn't suggesting that the use of an OST pool would set the user, group, or ACL of a file created within that pool.  That would indeed be confusing to users.


Rather, Eric was suggesting that there should be a user/group/ACL permission check for the _use_ of a particular pool, so that only allowed users/groups could access the "SSD" pool, for instance.  That has been under discussion for a long time already, and is a natural extension of the base OST pool functionality.  Extending this further to allow and/or select pools based on the originating client NID or the filename also makes sense.

> On Nov 23, 2010, at 10:20, "Eric Barton" <eeb@whamcloud.com> wrote:
>> P.S. Do people also agree that setting user/group/ACL permissions on
>> OST pools would be a Good Thing?

Cheers, Andreas
--
Andreas Dilger
Lustre Technical Lead
Oracle Corporation Canada Inc.

[Lustre-devel] OST pool administration (was RE: Stripe offset default)

[Eric Barton]

Peter,

> The  proposal for acl / Group / users violates Posix, and may lead
> to results users don't expect.  Sgid directories and inheritance
> on ACLs are safer I think.

This should probably be in a separate thread since I was trying to make
a case for improving the administration storage pools (the classic
gold/silver/bronze scenarios) which is really different from the default
file layout issue.

If the default layout in a directory creates files in the "gold" OST
pool but the user has no permissions for that pool, then returning
EPERM on file creation would potentially be inconsistent with the
permissions on the directory itself.  Is this what you meant by being
inconsistent with Posix?  If so, I'm afraid I don't follow the point
about sgid.

I guess using per-pool quotas rather than permissions might be more
natural.  I'd be interested to hear people's opinions...

          Cheers,
                   Eric

[Lustre-devel] OST pool administration (was RE: Stripe offset default)

[Peter Braam]

On 24 November 2010 06:31, Eric Barton <eeb@whamcloud.com> wrote:

> Peter,
>
> > The  proposal for acl / Group / users violates Posix, and may lead
> > to results users don't expect.  Sgid directories and inheritance
> > on ACLs are safer I think.
>
> This should probably be in a separate thread since I was trying to make
> a case for improving the administration storage pools (the classic
> gold/silver/bronze scenarios) which is really different from the default
> file layout issue.
>
> If the default layout in a directory creates files in the "gold" OST
> pool but the user has no permissions for that pool, then returning
> EPERM on file creation would potentially be inconsistent with the
> permissions on the directory itself.  Is this what you meant by being
> inconsistent with Posix?  If so, I'm afraid I don't follow the point
> about sgid.
>
>
Group writeable directories with the sgid mode bit set, exceptionally
inherit permissions from the parent directory upon creation of new
subdirectories (assuming no explicit mode is passed).   So it automatically
descends down the tree, just like the striping attributes do.  I believe
POSIX ACLs have a similar inheritance option.

I have sympathy for the idea btw, but basic permission issues failing
because of it is something that needs some thought.



> I guess using per-pool quotas rather than permissions might be more
> natural.  I'd be interested to hear people's opinions...
>


That makes sense also, and perhaps 0 quota could be interpreted as a denial
to create also, without causing harm.

Peter


>
>          Cheers,
>                   Eric
>
>
>
> _______________________________________________
> Lustre-devel mailing list
> Lustre-devel at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-devel
> ______________________________________________________________________
> This email may contain privileged or confidential information, which should
> only be used for the purpose for which it was sent by Xyratex. No further
> rights or licenses are granted to use such information. If you are not the
> intended recipient of this message, please notify the sender by return and
> delete it. You may not use, copy, disclose or rely on the information
> contained in it.
>
> Internet email is susceptible to data corruption, interception and
> unauthorised amendment for which Xyratex does not accept liability. While we
> have taken reasonable precautions to ensure that this email is free of
> viruses, Xyratex does not accept liability for the presence of any computer
> viruses in this email, nor for any losses caused as a result of viruses.
>
> Xyratex Technology Limited (03134912), Registered in England & Wales,
> Registered Office, Langstone Road, Havant, Hampshire, PO9 1SA.
>
> The Xyratex group of companies also includes, Xyratex Ltd, registered in
> Bermuda, Xyratex International Inc, registered in California, Xyratex
> (Malaysia) Sdn Bhd registered in Malaysia, Xyratex Technology (Wuxi) Co Ltd
> registered in The People's Republic of China and Xyratex Japan Limited
> registered in Japan.
> ______________________________________________________________________
>
>
>

______________________________________________________________________
This email may contain privileged or confidential information, which should only be used for the purpose for which it was sent by Xyratex. No further rights or licenses are granted to use such information. If you are not the intended recipient of this message, please notify the sender by return and delete it. You may not use, copy, disclose or rely on the information contained in it.
 
Internet email is susceptible to data corruption, interception and unauthorised amendment for which Xyratex does not accept liability. While we have taken reasonable precautions to ensure that this email is free of viruses, Xyratex does not accept liability for the presence of any computer viruses in this email, nor for any losses caused as a result of viruses.
 
Xyratex Technology Limited (03134912), Registered in England & Wales, Registered Office, Langstone Road, Havant, Hampshire, PO9 1SA.
 
The Xyratex group of companies also includes, Xyratex Ltd, registered in Bermuda, Xyratex International Inc, registered in California, Xyratex (Malaysia) Sdn Bhd registered in Malaysia, Xyratex Technology (Wuxi) Co Ltd registered in The People's Republic of China and Xyratex Japan Limited registered in Japan.
______________________________________________________________________
 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20101124/1932d6e4/attachment.htm>

[Lustre-devel] Stripe offset default

[Andreas Dilger]

On 2010-11-23, at 10:20, Eric Barton wrote:
> ... setting the default stripe offset is positively dangerous.

Replied to this on Chris Morrone's email, though it seems his email lost Nick from the CC list...  I wouldn't go so far as to call it "dangerous", but the use of old 1.4.x or 1.6.x positional parameters should be eliminated.  That is something that only Nick can manage, since it doesn't matter what we do in newer Lustre releases if the users are only running old versions of the tools.

Starting by applying the "positional parameters return an error" to his 1.6.x lfs would be a good start.  I made this available as attachment 32308 on bug 14836, where that usage was originally deprecated.

> P.S. Do people also agree that setting user/group/ACL permissions on
> OST pools would be a Good Thing?

Sure - being able to limit the use of pools, including the default "use all OSTs" pool, to specific users/groups/nodes/networks would be great.  Similarly, being able to specify certain pools as defaults for particular users/groups/nodes/networks is equally desirable.  Both would need some effort in the LOV QOS code on the MDS, to allow the layout to come from something other than the parent directory.

Cheers, Andreas
--
Andreas Dilger
Lustre Technical Lead
Oracle Corporation Canada Inc.