fine grain block allocation API request

fine grain block allocation API request

[Dmitry Monakhov]

Currently we have not public API for block allocation from some
specific region (group/flex_group/blk_range).  The only existing
hack is available in migrate.c where we use goal_inode for
ext4_new_inode().

In fact this tend make some specific task very complicated.
For example my e4defrag2 utility try to compact small files from
one group together. In order to allocate blocks from given group
I have to perform number of crazy stuff, such as:
1) Prepare cache of directories according to it's group (where dir_inode
 is placed)

2) Try to allocate local donor file with local blocks (from given group)
   Pseudo code of donor creation procedure :
   int find_donor(int grp, ...) {
       for (i = 0; i = dir_cache[grp].num;i++) {
           fd = openat(dir_cache[grp].fd[i], "donor", O_CREAT|flags,... )
           fstat(fd, &stat);
           if (inode_group(stat->st_ino) != grp)
              goto next_candidate;
           do_falloc(fd, size);
           /* Check that blocks we allocated are belongs to group we want*/
           if (!is_block_local(fd, group))
              goto next_candidate;
           goto found_donor;
          .....
       }
      .....
   }
All this machinery is very ugly, unstable and increase code-base about
500-700 lines of code. And I want to ask you why we do not have such
API? AFAIR this API was requested several times, but it was rejected.
I do understand that this is not good idea to allow unprivileged user to
manipulate block allocator, but IMHO we can hide it under CAP_RESOURCES
or CAP_ADMIN.
Please tell be your opinion. I'll happy to implement that ioctl.

Re: fine grain block allocation API request

[Andreas Dilger]

One question to ask is whether a goal inode allocation API would be enough
or if you need specific block allocation, or both? If you could start
allocation at a specific inode, would that be enough to avoid the messy
code below?

Cheers, Andreas

> On Aug 13, 2014, at 15:45, Dmitry Monakhov <dmonakhov@openvz.org> wrote:
> 
> 
> Currently we have not public API for block allocation from some
> specific region (group/flex_group/blk_range).  The only existing
> hack is available in migrate.c where we use goal_inode for
> ext4_new_inode().
> 
> In fact this tend make some specific task very complicated.
> For example my e4defrag2 utility try to compact small files from
> one group together. In order to allocate blocks from given group
> I have to perform number of crazy stuff, such as:
> 1) Prepare cache of directories according to it's group (where dir_inode
> is placed)
> 
> 2) Try to allocate local donor file with local blocks (from given group)
>   Pseudo code of donor creation procedure :
>   int find_donor(int grp, ...) {
>       for (i = 0; i = dir_cache[grp].num;i++) {
>           fd = openat(dir_cache[grp].fd[i], "donor", O_CREAT|flags,... )
>           fstat(fd, &stat);
>           if (inode_group(stat->st_ino) != grp)
>              goto next_candidate;
>           do_falloc(fd, size);
>           /* Check that blocks we allocated are belongs to group we want*/
>           if (!is_block_local(fd, group))
>              goto next_candidate;
>           goto found_donor;
>          .....
>       }
>      .....
>   }
> All this machinery is very ugly, unstable and increase code-base about
> 500-700 lines of code. And I want to ask you why we do not have such
> API? AFAIR this API was requested several times, but it was rejected.
> I do understand that this is not good idea to allow unprivileged user to
> manipulate block allocator, but IMHO we can hide it under CAP_RESOURCES
> or CAP_ADMIN.
> Please tell be your opinion. I'll happy to implement that ioctl.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: fine grain block allocation API request

[Dmitry Monakhov]

On Wed, 13 Aug 2014 22:38:06 +0200, Andreas Dilger <adilger@dilger.ca> wrote:
> One question to ask is whether a goal inode allocation API would be enough
> or if you need specific block allocation, or both? If you could start
> allocation at a specific inode, would that be enough to avoid the messy
> code below?
In order to pack small files I need goal block allocator API.
Goal inode allocation API is almost useless for my task. But other tasks may
gain from that feature. For example this allow to implement online filesystem shrink.
 
> 
> Cheers, Andreas
> 
> > On Aug 13, 2014, at 15:45, Dmitry Monakhov <dmonakhov@openvz.org> wrote:
> > 
> > 
> > Currently we have not public API for block allocation from some
> > specific region (group/flex_group/blk_range).  The only existing
> > hack is available in migrate.c where we use goal_inode for
> > ext4_new_inode().
> > 
> > In fact this tend make some specific task very complicated.
> > For example my e4defrag2 utility try to compact small files from
> > one group together. In order to allocate blocks from given group
> > I have to perform number of crazy stuff, such as:
> > 1) Prepare cache of directories according to it's group (where dir_inode
> > is placed)
> > 
> > 2) Try to allocate local donor file with local blocks (from given group)
> >   Pseudo code of donor creation procedure :
> >   int find_donor(int grp, ...) {
> >       for (i = 0; i = dir_cache[grp].num;i++) {
> >           fd = openat(dir_cache[grp].fd[i], "donor", O_CREAT|flags,... )
> >           fstat(fd, &stat);
> >           if (inode_group(stat->st_ino) != grp)
> >              goto next_candidate;
> >           do_falloc(fd, size);
> >           /* Check that blocks we allocated are belongs to group we want*/
> >           if (!is_block_local(fd, group))
> >              goto next_candidate;
> >           goto found_donor;
> >          .....
> >       }
> >      .....
> >   }
> > All this machinery is very ugly, unstable and increase code-base about
> > 500-700 lines of code. And I want to ask you why we do not have such
> > API? AFAIR this API was requested several times, but it was rejected.
> > I do understand that this is not good idea to allow unprivileged user to
> > manipulate block allocator, but IMHO we can hide it under CAP_RESOURCES
> > or CAP_ADMIN.
> > Please tell be your opinion. I'll happy to implement that ioctl.
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html