"btrfs send" fails with having too many open fd's

"btrfs send" fails with having too many open fd's

[Robert Buhren]

Hi there,

i'm trying to send/receive a snapshot of my home dir for some time now, 
but in the end it always fails because the maximum number of open fd's 
is exceeded.

Kern log says:

"6,7226,15012711248,-;VFS: file-max limit 799969 reached"


My setup is as follows:

Kernel: 3.9-rc5
btrfs-progs: compiled from git commit: commit 
7854c8b667654502f69e05584729146a06827bc6

btrfs fi show:

Label: none  uuid: ccedbef9-417b-454c-b1d4-8b4220111e6a
     Total devices 1 FS bytes used 177.79GB
     devid    1 size 210.85GB used 210.85GB path /dev/dm-0

Label: 'BACKUP'  uuid: 019e75e4-aa2e-495d-9711-28ef7150b1f3
     Total devices 2 FS bytes used 137.35GB
     devid    2 size 298.09GB used 142.01GB path /dev/sdc1
     devid    1 size 298.09GB used 142.03GB path /dev/sdb1


The send fs is on a dm-crypt device and the receive on a btrfs raid1
A scrub for both fs ran fine without errors.I can however send/receive 
my root fs which is much smaller (~17GB).

The error occurs also if i just use "btrfs send" and pipe the output to 
a file.
It takes a very long time before my system crashes (several hours) so i 
wasn't able to monitor when exactly the fd's increase.
In the beginning "btrfs send" just opens less than 10 fd's, but at some 
point "btrfs send" must open much more.

If further information is needed, i'd be happy to help. I'm also in 
#btrfs on freenode.

Regards,

Robert


PS: Please respond to my email, i'm not subscribed to the list.

Re: "btrfs send" fails with having too many open fd's

[David Sterba]

On Sun, Apr 07, 2013 at 10:58:41AM +0200, Robert Buhren wrote:
> "6,7226,15012711248,-;VFS: file-max limit 799969 reached"

> Label: 'BACKUP'  uuid: 019e75e4-aa2e-495d-9711-28ef7150b1f3
>     Total devices 2 FS bytes used 137.35GB

> It takes a very long time before my system crashes (several hours) so i
> wasn't able to monitor when exactly the fd's increase.
> In the beginning "btrfs send" just opens less than 10 fd's, but at some
> point "btrfs send" must open much more.

It could be a case of

1) an endless loop, the file limit you've set is quite high (799969), I
doubt that so many open fd's are needed at any point.

2) missing file close after in some error path

The file's are open via
send_write
  open_cur_inode_file
    dentry_open

and closed from callbacks in close_cur_inode_file.

Case 2 looks more realistic, but I don't have an idea what could go
wrong.

david

Re: "btrfs send" fails with having too many open fd's

[Robert Buhren]

On 08.04.2013 15:11, David Sterba wrote:
> On Sun, Apr 07, 2013 at 10:58:41AM +0200, Robert Buhren wrote:
>> "6,7226,15012711248,-;VFS: file-max limit 799969 reached"
>> Label: 'BACKUP'  uuid: 019e75e4-aa2e-495d-9711-28ef7150b1f3
>>      Total devices 2 FS bytes used 137.35GB
>> It takes a very long time before my system crashes (several hours) so i
>> wasn't able to monitor when exactly the fd's increase.
>> In the beginning "btrfs send" just opens less than 10 fd's, but at some
>> point "btrfs send" must open much more.
> It could be a case of
>
> 1) an endless loop, the file limit you've set is quite high (799969), I
> doubt that so many open fd's are needed at any point.
>
> 2) missing file close after in some error path
>
> The file's are open via
> send_write
>    open_cur_inode_file
>      dentry_open
>
> and closed from callbacks in close_cur_inode_file.
>
> Case 2 looks more realistic, but I don't have an idea what could go
> wrong.
>
> david
Do you have a suggestion on how to debug it further?

Regards,

Robert