bind mounting into a generated multi-level directory structure

bind mounting into a generated multi-level directory structure

[Chris Stromsoe]

I have a set of directories that are created and removed programmatically, 
and are hashed three levels deep (/top/hash/hash/name).  The depth is 
constant, the hashes and names are not.  There are around 35k at any time. 
I would like to bind mount a common directory into each top-level 
(/top/hash/hash/name/data) on demand.

Can I use autofs to do that without having to explicitly list all of the 
directory paths in auto.master?

-Chris

Re: bind mounting into a generated multi-level directory structure

[Stef Bon]

Chris Stromsoe wrote:
> I have a set of directories that are created and removed programmatically, 
> and are hashed three levels deep (/top/hash/hash/name).  The depth is 
> constant, the hashes and names are not.  There are around 35k at any time. 
> I would like to bind mount a common directory into each top-level 
> (/top/hash/hash/name/data) on demand.
>
> Can I use autofs to do that without having to explicitly list all of the 
> directory paths in auto.master?
>   
Why not use symlinks to this shared common directory, and mount that one?

Stef Bon

Re: bind mounting into a generated multi-level directory structure

[Ian Kent]

On Sun, 2008-02-10 at 00:29 -0800, Chris Stromsoe wrote:
> I have a set of directories that are created and removed programmatically, 
> and are hashed three levels deep (/top/hash/hash/name).  The depth is 
> constant, the hashes and names are not.  There are around 35k at any time. 
> I would like to bind mount a common directory into each top-level 
> (/top/hash/hash/name/data) on demand.

I'm not clear on what you mean by "top"?

> 
> Can I use autofs to do that without having to explicitly list all of the 
> directory paths in auto.master?

What version of autofs are you using?

Ian

Re: bind mounting into a generated multi-level directory structure

[Chris Stromsoe]

On Mon, 11 Feb 2008, Ian Kent wrote:
> On Sun, 2008-02-10 at 00:29 -0800, Chris Stromsoe wrote:
>
>> I have a set of directories that are created and removed 
>> programmatically, and are hashed three levels deep 
>> (/top/hash/hash/name).  The depth is constant, the hashes and names are 
>> not.  There are around 35k at any time. I would like to bind mount a 
>> common directory into each top-level (/top/hash/hash/name/data) on 
>> demand.
>
> I'm not clear on what you mean by "top"?

"/top" is the root of my directory tree.  I have roughly 35k directories 
that are stored as /top/hash1/hash2/name, where hash1 is one hash of 
"name" and hash2 is a different hash of "name".  Beneath "name" there are 
a variety of other files and directories.  New "name" directories are 
hashed, created, and populated on the fly.

I would like to bind mount a common directory onto 
/top/hash1/hash2/name/data, so that I can chroot into .../name/ and run 
common code, minimizing exposure to the rest of the system.  I would like 
to use autofs if possible, so that I don't have to copy 20Mb+ of data 35k 
times or maintain 35k bind mounts or do something else that continously 
uses resources.

I also don't want to have to edit auto.master every time I add or remove a 
"name" directory, or have to maintain 35k entries in a flat file.

>> Can I use autofs to do that without having to explicitly list all of 
>> the directory paths in auto.master?
>
> What version of autofs are you using?

I've been playing with 4.1.4, from Debian stable, but am more than willing 
to use 5.0.3 if it will do what I need.

Playing around so far, the best I've been able to come up with is doing a 
bind mount of /top/hash1/hash2/name to a simpler /mnt/name (using a 
program map to compute the hashes), then hopfully submounting the "/data" 
directory, but I haven't been able to figure out how to get /mnt/name/data 
mounted.  Can you do submounts of a program map?

My auto.master is:

/mnt	/etc/auto.mnt

And /etc/auto.mnt is (I'm using bogus "hash" values for test purposes):

#!/usr/bin/perl
my $f = lc shift @ARGV;
my $d = sprintf "/top/a/a/%s", $f;
exit 1 if ! -d $d;
printf ":%s\n", $d;



-Chris

Re: bind mounting into a generated multi-level directory structure

[Ian Kent]

On Mon, 2008-02-11 at 01:53 -0800, Chris Stromsoe wrote:
> On Mon, 11 Feb 2008, Ian Kent wrote:
> > On Sun, 2008-02-10 at 00:29 -0800, Chris Stromsoe wrote:
> >
> >> I have a set of directories that are created and removed 
> >> programmatically, and are hashed three levels deep 
> >> (/top/hash/hash/name).  The depth is constant, the hashes and names are 
> >> not.  There are around 35k at any time. I would like to bind mount a 
> >> common directory into each top-level (/top/hash/hash/name/data) on 
> >> demand.
> >
> > I'm not clear on what you mean by "top"?
> 
> "/top" is the root of my directory tree.  I have roughly 35k directories 
> that are stored as /top/hash1/hash2/name, where hash1 is one hash of 
> "name" and hash2 is a different hash of "name".  Beneath "name" there are 
> a variety of other files and directories.  New "name" directories are 
> hashed, created, and populated on the fly.
> 
> I would like to bind mount a common directory onto 
> /top/hash1/hash2/name/data, so that I can chroot into .../name/ and run 
> common code, minimizing exposure to the rest of the system.  I would like 
> to use autofs if possible, so that I don't have to copy 20Mb+ of data 35k 
> times or maintain 35k bind mounts or do something else that continously 
> uses resources.
> 
> I also don't want to have to edit auto.master every time I add or remove a 
> "name" directory, or have to maintain 35k entries in a flat file.
> 
> >> Can I use autofs to do that without having to explicitly list all of 
> >> the directory paths in auto.master?
> >
> > What version of autofs are you using?
> 
> I've been playing with 4.1.4, from Debian stable, but am more than willing 
> to use 5.0.3 if it will do what I need.
> 
> Playing around so far, the best I've been able to come up with is doing a 
> bind mount of /top/hash1/hash2/name to a simpler /mnt/name (using a 
> program map to compute the hashes), then hopfully submounting the "/data" 
> directory, but I haven't been able to figure out how to get /mnt/name/data 
> mounted.  Can you do submounts of a program map?

So let me see if I've got this yet.

You're happy to use a program map to generate the hash
directory, /top/hash1/hash2/name on something like /mnt/name.

You need to be able to submount path /top/hash1/hash2/name/data onto
something like /mnt/name/data.

> 
> My auto.master is:
> 
> /mnt	/etc/auto.mnt
> 
> And /etc/auto.mnt is (I'm using bogus "hash" values for test purposes):
> 
> #!/usr/bin/perl
> my $f = lc shift @ARGV;
> my $d = sprintf "/top/a/a/%s", $f;
> exit 1 if ! -d $d;
> printf ":%s\n", $d;

So if my reading is correct then you probably need to generate a mount
string from the program map that looks something like (I haven't had to
work with macro defines for a while so the syntax might not be quite
right):

-fstype=autofs -Dhash1=$h1 -Dhash2=$h2 -Dname=$f /etc/auto.submount

In the above the $h1, $h2 and $f need to be substituted with the values
computed by the script, probably using something like you're printf
above.

The /etc/auto.submount would be something like:

*	:/top/$(hash1}/$hash2}/${name}/data

or if "data" was actually the key

*	:/top/$(hash1}/$hash2}/${name}/&

Other thoughts.

I can't be sure this will work so you'll need to try it out and see how
you go.

The $f above in -Dname=$f might be able to be substituted with &, the
value of the key, but that might confuse the autofs macro substitution
code.

I can't remember whether we have to ":" escape the leading "/" of a
submount map, I'm sure you'll work that out fairly quickly though.

There are clearly a number of variations on this but you will need to
work out what is best.

Ian

Re: bind mounting into a generated multi-level directory structure

[Stef Bon]

Chris Stromsoe wrote:
> I have a set of directories that are created and removed programmatically, 
> and are hashed three levels deep (/top/hash/hash/name).  The depth is 
> constant, the hashes and names are not.  There are around 35k at any time. 
> I would like to bind mount a common directory into each top-level 
> (/top/hash/hash/name/data) on demand.
>
> Can I use autofs to do that without having to explicitly list all of the 
> directory paths in auto.master?
>   
Why not use symlinks to this shared common directory, and mount that one?

Stef Bon

Re: bind mounting into a generated multi-level directory structure

[Chris Stromsoe]

On Mon, 11 Feb 2008, Ian Kent wrote:
> On Mon, 2008-02-11 at 01:53 -0800, Chris Stromsoe wrote:
>
>> Playing around so far, the best I've been able to come up with is doing 
>> a bind mount of /top/hash1/hash2/name to a simpler /mnt/name (using a 
>> program map to compute the hashes), then hopfully submounting the 
>> "/data" directory, but I haven't been able to figure out how to get 
>> /mnt/name/data mounted.  Can you do submounts of a program map?
>
> So let me see if I've got this yet.
>
> You're happy to use a program map to generate the hash directory, 
> /top/hash1/hash2/name on something like /mnt/name.
> You need to be able to submount path /top/hash1/hash2/name/data onto 
> something like /mnt/name/data.
> 
> So if my reading is correct then you probably need to generate a mount 
> string from the program map that looks something like (I haven't had to 
> work with macro defines for a while so the syntax might not be quite 
> right):

Thanks for the pointers.  They got me thinking in a different direction. 
Instead of mounting the data directory into the raw hash directories, I'm 
generating a bind mount on demand, mounting the shared static directory 
onto /opt/chroot/key, then bind mounting the hash directory onto 
/opt/chroot/key/user using multi-mount.

For the archives, in the event anybody else comes looking to do something 
similar, my auto.master is

   /opt/chroot	/etc/auto.chroot


My (cut down) auto.chroot script is:

   #!/usr/bin/perl
   my $f = lc shift @ARGV;
   exit 1 unless &validate($f);

   my ($h1,$h2) = &hash($f);
   my $d = sprintf "/opt/u/%s/%s/%s", $h1, $h2, $f;
   exit 1 unless &check($d);

   printf "-fstype=bind,ro :/opt/dataroot \\\n";
   printf "/user -fstype=bind $dirname\n";

   exit 0;



-Chris

Re: bind mounting into a generated multi-level directory structure

[chris barry]

On Sun, 2008-02-10 at 15:18 +0100, Stef Bon wrote:
> Chris Stromsoe wrote:
> > I have a set of directories that are created and removed programmatically, 
> > and are hashed three levels deep (/top/hash/hash/name).  The depth is 
> > constant, the hashes and names are not.  There are around 35k at any time. 
> > I would like to bind mount a common directory into each top-level 
> > (/top/hash/hash/name/data) on demand.
> >
> > Can I use autofs to do that without having to explicitly list all of the 
> > directory paths in auto.master?
> >   
> Why not use symlinks to this shared common directory, and mount that one?
> 
> Stef Bon

His goal is to create a chroot'd environment, and the symlink cannot
function there. man chroot to help you understand why.

-C

Re: bind mounting into a generated multi-level directory structure

[Stef Bon]

chris barry wrote:
> On Sun, 2008-02-10 at 15:18 +0100, Stef Bon wrote:
>   
>> Chris Stromsoe wrote:
>>     
>>> I have a set of directories that are created and removed programmatically, 
>>> and are hashed three levels deep (/top/hash/hash/name).  The depth is 
>>> constant, the hashes and names are not.  There are around 35k at any time. 
>>> I would like to bind mount a common directory into each top-level 
>>> (/top/hash/hash/name/data) on demand.
>>>
>>> Can I use autofs to do that without having to explicitly list all of the 
>>> directory paths in auto.master?
>>>   
>>>       
>> Why not use symlinks to this shared common directory, and mount that one?
>>
>> Stef Bon
>>     
>
> His goal is to create a chroot'd environment, and the symlink cannot
> function there.
>   
Good to mention. I did not understand it's about a chrooted environment. 
The earlier emails do not say anything at all about that.

Stef

Re: bind mounting into a generated multi-level directory structure

[Chris Stromsoe]

On Fri, 15 Feb 2008, Stef Bon wrote:
> chris barry wrote:
>> On Sun, 2008-02-10 at 15:18 +0100, Stef Bon wrote:
>>
>>> Chris Stromsoe wrote:
>>>
>>>> I have a set of directories that are created and removed 
>>>> programmatically, and are hashed three levels deep 
>>>> (/top/hash/hash/name).  The depth is constant, the hashes and names 
>>>> are not.  There are around 35k at any time. I would like to bind 
>>>> mount a common directory into each top-level 
>>>> (/top/hash/hash/name/data) on demand.
>>>>
>>>> Can I use autofs to do that without having to explicitly list all of 
>>>> the directory paths in auto.master?
>>>>
>>>>
>>> Why not use symlinks to this shared common directory, and mount that 
>>> one?
>>>
>>> Stef Bon
>>>
>>
>> His goal is to create a chroot'd environment, and the symlink cannot 
>> function there.
>
> Good to mention. I did not understand it's about a chrooted environment. 
> The earlier emails do not say anything at all about that.

Sorry if I wasn't clear in the original message.  I was trying to simplify 
what I wanted as much as possible, but I guess I oversimplified.

My eventual solution was to use multi-mount to bind mount a shared 
directory with libraries and other common data to a "simple" generated 
path as a container, and then bind mount the hashed directory inside of 
that volume.  I'm using the container volume as a chroot environment to 
run untrusted code (php) on a web server, and wanted to minimize exposure 
to the rest of the machine as much as possible.



-Chris

Re: bind mounting into a generated multi-level directory structure

[Stef Bon]

Chris Stromsoe wrote:
> My eventual solution was to use multi-mount to bind mount a shared 
> directory with libraries and other common data to a "simple" generated 
> path as a container, and then bind mount the hashed directory inside 
> of that volume.  I'm using the container volume as a chroot 
> environment to run untrusted code (php) on a web server, and wanted to 
> minimize exposure to the rest of the machine as much as possible.
Ok, it's more clear to me now. I'm wondering, isn't it possible to 
create a custom "chroot" command, which will of course do the chroot, 
but also do the necessary binds. I've been working on a login shell 
(chroot_union) which is the standard shell of a user. When this user 
logs in (starts a session) a copy of the system is created with unionfs. 
Then a chroot is done, and a perfect environment where the user can do 
anything is there. Isn't that possible?

Stef Bon

Re: bind mounting into a generated multi-level directory structure

[Chris Stromsoe]

On Sat, 16 Feb 2008, Stef Bon wrote:
> Chris Stromsoe wrote:
> 
>> My eventual solution was to use multi-mount to bind mount a shared 
>> directory with libraries and other common data to a "simple" generated 
>> path as a container, and then bind mount the hashed directory inside of 
>> that volume.  I'm using the container volume as a chroot environment to 
>> run untrusted code (php) on a web server, and wanted to minimize 
>> exposure to the rest of the machine as much as possible.
> 
> Ok, it's more clear to me now. I'm wondering, isn't it possible to 
> create a custom "chroot" command, which will of course do the chroot, 
> but also do the necessary binds.

Probably.  But I have over 35k virtual hosts.  I need to have the mounts 
expire after some period of non-use.  Having the chroot process handle the 
binds means that I would need to have some other process sitting around to 
unmount everything after a period of inactivity.  Which might as well be 
autofs at that point.

> I've been working on a login shell (chroot_union) which is the standard 
> shell of a user. When this user logs in (starts a session) a copy of the 
> system is created with unionfs. Then a chroot is done, and a perfect 
> environment where the user can do anything is there. Isn't that 
> possible?

The server is only accessible via ftp and http.  I'm trying to limit the 
environment only to what is necessary to run and nothing more.  If I were 
trying to provide a more complete environment with shell access it might 
make sense to use unionfs, but I'm not doing anything that complex. Autofs 
managing the bind mounts (together with a chroot cgi wrapper to limit 
access) covers everything I need.


-Chris

OT: was Re: bind mounting into a generated multi-level directory structure

[chris barry]

On Fri, 2008-02-15 at 13:44 -0800, Chris Stromsoe wrote:
> My eventual solution was to use multi-mount to bind mount a shared 
> directory with libraries and other common data to a "simple" generated 
> path as a container, and then bind mount the hashed directory inside of 
> that volume.  I'm using the container volume as a chroot environment to 
> run untrusted code (php) on a web server, and wanted to minimize exposure 
> to the rest of the machine as much as possible.
> 
> -Chris

This sounds like a very cool solution. Any chance of you whipping up a
more detailed recipe?


-C