Cross-subvolume rename behavior

Cross-subvolume rename behavior

[Sean Greenslade]

Hello, all. I'm currently tracking down the source of some strange
behavior in my setup. I recognize that this isn't strictly a btrfs
issue, but I figured I'd start at the bottom of the stack and work my
way up.

I have a server with a btrfs filesystem on it that I remotely access on
several systems via an sshfs mount. For the most part this works
perfectly, but I just discovered that moving files between subvolumes on
that mount fails with a confusing "Operation not permitted" error.

After doing some digging, it turns out it's not actually a permissions
error. If I do the same operation locally on the server, it succeeds,
but an strace of the mv reveals that the rename() syscall returns EXDEV.
The mv util takes this as a sign to fall back on the copy-and-delete
routine, so the move succeeds. Unfortunately, it seems that somewhere in
sshfs, sftp, or fuse, the EXDEV is getting turned into a generic
failure, which mv apparently interprets as "permission denied".

So my question for the btrfs devs: is rename()-ing across subvolumes
not feasible, or is this simply a case of no one has implemented that
yet? 

Thanks for any insights,

--Sean

Re: Cross-subvolume rename behavior

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1999 bytes --]

On Wed, Mar 22, 2017 at 10:37:23PM -0700, Sean Greenslade wrote:
> Hello, all. I'm currently tracking down the source of some strange
> behavior in my setup. I recognize that this isn't strictly a btrfs
> issue, but I figured I'd start at the bottom of the stack and work my
> way up.
> 
> I have a server with a btrfs filesystem on it that I remotely access on
> several systems via an sshfs mount. For the most part this works
> perfectly, but I just discovered that moving files between subvolumes on
> that mount fails with a confusing "Operation not permitted" error.
> 
> After doing some digging, it turns out it's not actually a permissions
> error. If I do the same operation locally on the server, it succeeds,
> but an strace of the mv reveals that the rename() syscall returns EXDEV.
> The mv util takes this as a sign to fall back on the copy-and-delete
> routine, so the move succeeds. Unfortunately, it seems that somewhere in
> sshfs, sftp, or fuse, the EXDEV is getting turned into a generic
> failure, which mv apparently interprets as "permission denied".
> 
> So my question for the btrfs devs: is rename()-ing across subvolumes
> not feasible, or is this simply a case of no one has implemented that
> yet? 

   Direct rename (using rename(2)) isn't possible across subvols,
which is what the EXDEV result indicates. The solution is exactly what
mv does, which is reflink-and-delete (which is cheaper than
copy-and-delete, because no data is moved). In theory, you probably
could implement rename across subvolumes in the FS, but it would just
be moving the exact same operations from userspace to kernel space.

   I think that the solution here is for the sshfs stack to be fixed
so that it passes the EXDEV up to the mv command properly, and passes
the subsequent server-side copy (reflink) back down correctly.

   Hugo.

-- 
Hugo Mills             | I'm on a 30-day diet. So far I've lost 18 days.
hugo@... carfax.org.uk |
http://carfax.org.uk/  |
PGP: E2AB1DE4          |

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Cross-subvolume rename behavior

[Austin S. Hemmelgarn]

On 2017-03-23 06:09, Hugo Mills wrote:
> On Wed, Mar 22, 2017 at 10:37:23PM -0700, Sean Greenslade wrote:
>> Hello, all. I'm currently tracking down the source of some strange
>> behavior in my setup. I recognize that this isn't strictly a btrfs
>> issue, but I figured I'd start at the bottom of the stack and work my
>> way up.
>>
>> I have a server with a btrfs filesystem on it that I remotely access on
>> several systems via an sshfs mount. For the most part this works
>> perfectly, but I just discovered that moving files between subvolumes on
>> that mount fails with a confusing "Operation not permitted" error.
>>
>> After doing some digging, it turns out it's not actually a permissions
>> error. If I do the same operation locally on the server, it succeeds,
>> but an strace of the mv reveals that the rename() syscall returns EXDEV.
>> The mv util takes this as a sign to fall back on the copy-and-delete
>> routine, so the move succeeds. Unfortunately, it seems that somewhere in
>> sshfs, sftp, or fuse, the EXDEV is getting turned into a generic
>> failure, which mv apparently interprets as "permission denied".
>>
>> So my question for the btrfs devs: is rename()-ing across subvolumes
>> not feasible, or is this simply a case of no one has implemented that
>> yet?
>
>    Direct rename (using rename(2)) isn't possible across subvols,
> which is what the EXDEV result indicates. The solution is exactly what
> mv does, which is reflink-and-delete (which is cheaper than
> copy-and-delete, because no data is moved). In theory, you probably
> could implement rename across subvolumes in the FS, but it would just
> be moving the exact same operations from userspace to kernel space.
Doing so though would have the advantage that it theoretically could be 
made (almost) atomic like a normal rename is, whereas the fallback in mv 
is absolutely not atomic.
>
>    I think that the solution here is for the sshfs stack to be fixed
> so that it passes the EXDEV up to the mv command properly, and passes
> the subsequent server-side copy (reflink) back down correctly.
This would be wonderful in theory, but it can't pass down the reflink, 
because the SFTP protocol (which is what sshfs uses) doesn't even have 
the concept of reflinks, so implementing this completely would require a 
revision to the SFTP protocol, which I don't see as likely to happen.

Re: Cross-subvolume rename behavior

[Sean Greenslade]

On Thu, Mar 23, 2017 at 07:23:40AM -0400, Austin S. Hemmelgarn wrote:
> On 2017-03-23 06:09, Hugo Mills wrote:
> >    Direct rename (using rename(2)) isn't possible across subvols,
> > which is what the EXDEV result indicates. The solution is exactly what
> > mv does, which is reflink-and-delete (which is cheaper than
> > copy-and-delete, because no data is moved). In theory, you probably
> > could implement rename across subvolumes in the FS, but it would just
> > be moving the exact same operations from userspace to kernel space.
> >
> Doing so though would have the advantage that it theoretically could be made
> (almost) atomic like a normal rename is, whereas the fallback in mv is
> absolutely not atomic.
> > 
> >    I think that the solution here is for the sshfs stack to be fixed
> > so that it passes the EXDEV up to the mv command properly, and passes
> > the subsequent server-side copy (reflink) back down correctly.
> 
> This would be wonderful in theory, but it can't pass down the reflink,
> because the SFTP protocol (which is what sshfs uses) doesn't even have the
> concept of reflinks, so implementing this completely would require a
> revision to the SFTP protocol, which I don't see as likely to happen.

Thanks for the insights, Hugo and Austin. That's more or less what I was
expecting, so I guess my next stop will be with the openssh folks.
Though I was looking over the SFTP protocol, and none of the SSH_FX_*
errors seem suitable for this purpose. This might require a workaround
in sshfs...

--Sean