[Lustre-devel] tunables

[Nathaniel Rutman <Nathan.Rutman@Sun.COM>]

Andreas Dilger wrote:
> On Jan 31, 2008  11:47 -0800, Nathaniel Rutman wrote:
>   
>> 'Andreas Dilger' wrote:
>>     
>>> The patch as it stands now will set any tunables in
>>> /proc/{fs,sys}/{lustre,lnet}/ that match the path regexp supplied.
>>>
>>> This works on both clients and servers.  On 1.6 it just creates a
>>> path regexp to find the various files.  On 1.8 it will eventually
>>> communicate with the uOSS/uMDS process via a socket or something,
>>> but the goal right now is putting an interface in place with the
>>> existing release so that scripts/documentation can be compatible.
>>>
>>> It doesn't necessarily have anything to do with a mountpoint, but it
>>> does require the Lustre devices (that need to be tuned) to be set up.
>>>       
>>   
>> When I originally filed 14471, it was a grand plan of getting rid of /proc/ 
>> and other Linuxisms by putting everything under Lustre control in a special
>> <lustremountpt>/.lustre/param directory.  14471 has now morphed into 
>> "implement lctl {get,set}_param", which, along with 14691, allows a
>> consistent way of accessing the items formerly under proc.  This is fine
>> for a first step, but I still think it doesn't go far enough. We'll
>> eventually need  snapshots: <mntpt>/.lustre/snapshot/ and sooner feeds:
>> <mntpt>/.lustre/feed/
>>     
>
> The "move everything to .../.lustre/param" functionality was moved to
> bug 14687 "Move /proc/{fs,sys}/{lustre,lnet} to /mntpt/.lustre" in
> order to make the separate tasks more manageable.  The work in bug
> 14471 and 14691 are required for userspace servers, and as such for
> the 1.8 release, while 14687 is a "nice to have" and doesn't really
> bear any relationship to snapshots and feeds except that they share
> the ".lustre" virtual directory.
>
> The "lctl {get,set}_param" interfaces are intended to provide a common
> userland interface to tunables and stats regardless of whether the
> servers are running in userspace or the kernel.  I think similar changes
> can easily be made to llstat and collectl to extract stats directly
> from the userspace server processes as well.
>
>   
>> This idea works fine for current clients and servers (which all use 
>> mountpoints), but not for userspace servers.  I don't think we have come
>> up with a good answer for that.
>>     
>
> In fact, the move of the parameters to the lustre mountpoint is
> irrelevant for userspace servers, because they won't have any mountpoint.
> The work in bug 14691 will address the getting and setting of Lustre
> tunables for userspace servers, and access to stats.
>
>   
>> For params that are shared between all mounts, I think its fine to have 
>> access to them from each one; a change anywhere is reflected server-wide.
>>     
>
> This might be considered a security risk, as any malicious client could
> change parameters for the whole system.
>
>   
>> For tunables "relevant without anything at all mounted" say e.g. LNET,
>> or uOSS: I don't know.  Also for tunables that must be set before
>> services are started: I don't know.  FWIW, right now, servers get
>> permanent params out of their config files, but those are read while
>> the services are starting (not before).  Module parameters are used to
>> set pre-startup params. I really don't like the idea of using modules
>> params either (does Solaris use the same API?).  Maybe we need a plain
>> old text config file for portability.
>>     
>
>   
>> Really, I want something contained entirely within Lustre, but I don't see 
>> how to do that unless Lustre is running.
>>
>> I do think the uOSS somehow needs to have a mountpoint.  How is a uOSS 
>> started?
>> Maybe by mounting a stub Lustre client that just provides access to 
>> /.lustre?
>>     
>
> That's just the point - I think you are putting extra requirements in
> place to handle the change to move tunables into .../.lustre.  With
> the work in bug 14691 to communicate directly with the userspace server
> over a socket (a natural process-to-process communication method) there
> is no need to have a mountpoint.  Otherwise, we get into some overly
> complex framework like FUSE to have a userspace process providing services
> to the kernel to provide a mechanism for another userspace process to
> talk to it.  Ugh.
>   
I agree with everything above.
But I was trying to get across a grand unified vision.  If we're going 
to need
snapshots: <mntpt>/.lustre/snapshot/
feeds: <mntpt>/.lustre/feed/
then we may as well have
params: <mntpt>/.lustre/param/

Using lctl {get,set}_param is fine if all we have is params, but what
do we do about snapshots, feeds, etc?  Snapshots in particular
need a real file tree, but I suppose you could argue they only need to
exist on clients.  And I'll have to come up with yet a third interface for
feeds.  I just really like the idea of presenting all of our filesystem
"periphery" stuff in a single, consistent manner.