Getting a lot of fs-generated information to user space

Getting a lot of fs-generated information to user space

[Jan Schmidt]

Hi,

I'm looking for a way to get a lot of information generated by the
kernel to user space.

Here comes why: I'm working on btrfs send, which should generate a full
or incremental stream of a file system to be received elsewhere. I don't
want to send data and metadata from the internal trees for various
reasons. But to efficiently detect hardlinks and clones (reflinks),
kernel help is required. It would be most efficient (ram and runtime) to
generate a list of file names in kernel in a certain order and annotate
each such record with hardlink- and clone-specific information.

I'd like to leave it at that for the moment, please just think of it as
a bunch of really useful information that can only be generated by the
kernel and is needed by user space. However, I can write a lot more on
the background and why I think this should really be the way to go, if
anyone wishes.

Now I'm a bit at a loss, though. I have two (and a half) ideas how to
push that bunch of data to user space, and both feel ugly a different
way. And for each of them, I could need a hint for the right direction:

(1) Normally, requests to the file system go through ioctls (on the fd
of the mountpoint) and the result is small enough to be returned when
the ioctl finishes. That said, I thought of passing a user land fd along
with this ioctl to the kernel and make it dump the generated bits there.
Only, I don't see how to turn a fd into a struct file pointer. And I
don't know if that would be considered really ugly by a lot of people.

(2) In btrfs, we have an additional control device, /dev/btrfs-control.
The read(2) call seems a good interface to use. Only, noone talks to
/dev/btrfs-control about file systems right now. The only call it
implements is BTRFS_IOC_SCAN_DEV to scan a device whether it contains a
btrfs file system. I'm uncertain whether a completely different
mechanism of triggering some action on the file system is a good
solution. And, if it really was, I'm not sure how this would be best
implemented.

(3) The last idea is a combination of the other two: Use the good part
from (1), which is doing the ioctl on the fd of the mountpoint, telling
the fs what we want. Then (this is the part from (2)) open
/dev/btrfs-control for reading and grab the output. Only, I have no plan
how to tie those two distinct kernel requests together in a way that
call (1) triggers output on device (2)... And would that really make it
any less ugly?

Any hints on /how/ to implement /which/ of those (or even something
completely different, of course) would be really helpful. I'm trying to
avoid botching up a solution that no one considers appropriate :-)

Thanks,
-Jan

Re: Getting a lot of fs-generated information to user space

[Matthew Wilcox]

On Fri, Aug 12, 2011 at 02:14:15PM +0200, Jan Schmidt wrote:
> (1) Normally, requests to the file system go through ioctls (on the fd
> of the mountpoint) and the result is small enough to be returned when
> the ioctl finishes. That said, I thought of passing a user land fd along
> with this ioctl to the kernel and make it dump the generated bits there.
> Only, I don't see how to turn a fd into a struct file pointer. And I
> don't know if that would be considered really ugly by a lot of people.

struct file *filp = fget(fd);
...
fput(filp);

That said, why not have the ioctl mutate the existing fd?
ie in userspace:

int fd = open("/mnt/btrfs");
ioctl(fd, BTRFS_IOC_STREAM);
while (...) {
	read(fd, buf, 4096);
	...
}
close(fd);

-- 
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

Re: Getting a lot of fs-generated information to user space

[Jan Schmidt]

On 12.08.2011 14:31, Matthew Wilcox wrote:
> On Fri, Aug 12, 2011 at 02:14:15PM +0200, Jan Schmidt wrote:
>> (1) Normally, requests to the file system go through ioctls (on the fd
>> of the mountpoint) and the result is small enough to be returned when
>> the ioctl finishes. That said, I thought of passing a user land fd along
>> with this ioctl to the kernel and make it dump the generated bits there.
>> Only, I don't see how to turn a fd into a struct file pointer. And I
>> don't know if that would be considered really ugly by a lot of people.
> 
> struct file *filp = fget(fd);
> ...
> fput(filp);

Thanks.

> That said, why not have the ioctl mutate the existing fd?
> ie in userspace:
> 
> int fd = open("/mnt/btrfs");
> ioctl(fd, BTRFS_IOC_STREAM);
> while (...) {
> 	read(fd, buf, 4096);
> 	...
> }
> close(fd);

Woo :-) That looks good. I assume from your suggestion at least you do
not consider this too ugly.

One must keep in mind that user space would also be able to open
"/mnt/btrfs/any/file" to achieve the same. An ioctl changing read on a
file (apparently) might me less intuitive. But still, I really like that
approach.

Thanks!
-Jan

Re: Getting a lot of fs-generated information to user space

[Jan Schmidt]

On 12.08.2011 14:31, Matthew Wilcox wrote:
> On Fri, Aug 12, 2011 at 02:14:15PM +0200, Jan Schmidt wrote:
>> (1) Normally, requests to the file system go through ioctls (on the fd
>> of the mountpoint) and the result is small enough to be returned when
>> the ioctl finishes. That said, I thought of passing a user land fd along
>> with this ioctl to the kernel and make it dump the generated bits there.
>> Only, I don't see how to turn a fd into a struct file pointer. And I
>> don't know if that would be considered really ugly by a lot of people.
> 
> struct file *filp = fget(fd);
> ...
> fput(filp);
>
> That said, why not have the ioctl mutate the existing fd?
> ie in userspace:
> 
> int fd = open("/mnt/btrfs");
> ioctl(fd, BTRFS_IOC_STREAM);
> while (...) {
> 	read(fd, buf, 4096);
> 	...
> }
> close(fd);

After thinking twice, this has a drawback: I'd have to track state
between two read(2) calls, waiting for userspace to "pull" more data.

In contrast, I'd rather use a "push" like approach, having the buffering
done for me. Getting back to my suggestion (1, giving a fd to the kernel
where the output is expected), that should look like this in userspace:

-
int fd;
int pipefd[2];
struct io_args io_args;

fd = open("/mnt/btrfs");
pipe(pipefd);
io_agrs.dest = pipefd[0];

/* thread 1 */
ioctl(fd, BTRFS_IOC_STREAM, &io_args);
/* thread 2 */
while (...) {
	read(pipefd[1], buf, 4096);
	...
}
-

Any suggestions and opinions appreciated. Thanks,
-Jan

Re: Getting a lot of fs-generated information to user space

[Matthew Wilcox]

On Fri, Aug 12, 2011 at 04:41:41PM +0200, Jan Schmidt wrote:
> After thinking twice, this has a drawback: I'd have to track state
> between two read(2) calls, waiting for userspace to "pull" more data.

That should be OK.  You've got filp->private_data to store your state in,
though you'll have to coordinate with btrfs' other use of ->private_data.
Not hard, just a little bit of work.

-- 
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

Re: Getting a lot of fs-generated information to user space

[Andi Kleen]

An alternative to cooking up something custom would be to use relayfs.

-Andi

-- 
ak@linux.intel.com -- Speaking for myself only