handling rdma apps using chroot

handling rdma apps using chroot

[Sean Hefty]

I'm looking for ideas on the best way for handling applications which use
chroot, which results in failures using the rdma libraries.  Specifically,
and not surprisingly, I see failures reading /sys/class/... and
/dev/infiniband/...

For background, I'm trying to run vsftpd over rsockets using the preload
library with fork support.  vsftpd will run only if I comment out its use of
chroot.  I can reproduce the issue by inserting a chroot call into rstream.

Does anyone have any suggestions?

- Sean

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: handling rdma apps using chroot

[Jason Gunthorpe]

On Wed, Aug 15, 2012 at 12:07:25PM -0700, Sean Hefty wrote:
> I'm looking for ideas on the best way for handling applications which use
> chroot, which results in failures using the rdma libraries.  Specifically,
> and not surprisingly, I see failures reading /sys/class/... and
> /dev/infiniband/...
> 
> For background, I'm trying to run vsftpd over rsockets using the preload
> library with fork support.  vsftpd will run only if I comment out its use of
> chroot.  I can reproduce the issue by inserting a chroot call into rstream.
> 
> Does anyone have any suggestions?

Somehow you need to open the verbs device before doing the choot.. I
think once verbs is open there is no further need for sysfs and dev..

Sounds like you are chrooting before creating the listening socket?

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: handling rdma apps using chroot

[Hefty, Sean]

> Somehow you need to open the verbs device before doing the choot.. I
> think once verbs is open there is no further need for sysfs and dev..

I could hook chroot to force this.   That should fix the /sys issue.  However, the librdmacm accesses /dev when creating an event channel, which occurs after chroot.  (Maybe that call is just hopelessly broken trying to support chroot.)

> Sounds like you are chrooting before creating the listening socket?

Correct - this is one of the results of supporting fork.  The ftp daemon listens on a normal socket, then switches to an rsocket on the first data transfer, after any fork presumably has occurred.

Btw, I've tried using symlinks and running from '/'.  The symlinks don't work, but running from '/' does for rstream.  Neither work for vsftpd, likely from other security calls that it makes.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: handling rdma apps using chroot

[Jason Gunthorpe]

On Wed, Aug 15, 2012 at 07:44:38PM +0000, Hefty, Sean wrote:
> > Somehow you need to open the verbs device before doing the choot.. I
> > think once verbs is open there is no further need for sysfs and dev..
> 
> I could hook chroot to force this.  That should fix the /sys issue.
> However, the librdmacm accesses /dev when creating an event channel,
> which occurs after chroot.  (Maybe that call is just hopelessly
> broken trying to support chroot.)

There are lots of issues with using a dev/sysfs interface instead of
system calls and trying to support chroot... Not sure how rdmacm
works, but verbs returns event channel FDs directly 'in-band' which
avoids further use of dev.. 

Even better would be to use eventfd() to have user space create the
event channel and pass that FD up into the kernel for this sort of
stuff, but I'm not sure if eventfd() can replace all the use cases of
event channels, though they are very similar.

> Correct - this is one of the results of supporting fork.  The ftp
> daemon listens on a normal socket, then switches to an rsocket on
> the first data transfer, after any fork presumably has occurred.

Not sure how that switch works, but you could open the relevent ib
devices on a listen?
 
> Btw, I've tried using symlinks and running from '/'.  The symlinks
> don't work, but running from '/' does for rstream.  Neither work for
> vsftpd, likely from other security calls that it makes.

Symlinks? You can bind mount sysfs and create a mini /dev/ in the
chroot and things should work OK, but obviously exposing those items
to the chroot significantly defeats the point of the chroot in the
first place.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: handling rdma apps using chroot

[Hefty, Sean]

> There are lots of issues with using a dev/sysfs interface instead of
> system calls and trying to support chroot... Not sure how rdmacm
> works, but verbs returns event channel FDs directly 'in-band' which
> avoids further use of dev..

The rdmacm calls open() in rdma_create_event_channel()...

> Even better would be to use eventfd() to have user space create the
> event channel and pass that FD up into the kernel for this sort of
> stuff, but I'm not sure if eventfd() can replace all the use cases of
> event channels, though they are very similar.

This looks promising as a way to merge together all these different RDMA related FDs.  It's a significant change and more of a long term solution though.  On the surface, it looks like a better approach than trying to fix rdma_create_event_channel directly.

> Symlinks? You can bind mount sysfs and create a mini /dev/ in the
> chroot and things should work OK, but obviously exposing those items
> to the chroot significantly defeats the point of the chroot in the
> first place.

softlinks using 'ln -s'.  bind --mount didn't work either.  I'll have to look deeper into the code.

This is really more about having fork support with rsockets and uncovering/fixing the problems than seriously trying to get vsftp to run.  I'm already forced to use 'one process' mode.  If I can get vsftp to run, that would be great, but I hesitant to intercept and discard chroot calls just to make that happen.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html