number of hardlinks for directory in ls -lid always 1?

number of hardlinks for directory in ls -lid always 1?

[Martin Steigerwald]

Hi!

On BTRFS I see

martin@merkaba:~> ls -lid /usr/local
27138 drwxrwsr-x 1 root staff 62 Aug 15  2014 /usr/local
martin@merkaba:~> ls -lid /usr/local/.
27138 drwxrwsr-x 1 root staff 62 Aug 15  2014 /usr/local/.
martin@merkaba:~> ls -lid /usr/local/bin/..
27138 drwxrwsr-x 1 root staff 62 Aug 15  2014 /usr/local/bin/..

On other filesystems like Ext4 I see the actual number of hardlinks to the 
directory.

Is this intended behaviour of BTRFS or a bug?



With files it works:

martin@merkaba:~> cd Zeit
martin@merkaba:~/Zeit> mkdir links
martin@merkaba:~/Zeit> cd links
martin@merkaba:~/Zeit/links> LANG=C df -hT .
Filesystem             Type   Size  Used Avail Use% Mounted on
/dev/mapper/msata-home btrfs  170G  120G   48G  72% /home
martin@merkaba:~/Zeit/links> echo "hello" > file
martin@merkaba:~/Zeit/links> ls -li
insgesamt 4
12158633 -rw-r--r-- 1 martin martin 6 Mär 17 14:32 file
martin@merkaba:~/Zeit/links> ln file hardlink1
martin@merkaba:~/Zeit/links> ls -li           
insgesamt 8
12158633 -rw-r--r-- 2 martin martin 6 Mär 17 14:32 file
12158633 -rw-r--r-- 2 martin martin 6 Mär 17 14:32 hardlink1

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: number of hardlinks for directory in ls -lid always 1?

[David Sterba]

On Tue, Mar 17, 2015 at 02:33:30PM +0100, Martin Steigerwald wrote:
> On BTRFS I see
> 
> martin@merkaba:~> ls -lid /usr/local
> 27138 drwxrwsr-x 1 root staff 62 Aug 15  2014 /usr/local
> martin@merkaba:~> ls -lid /usr/local/.
> 27138 drwxrwsr-x 1 root staff 62 Aug 15  2014 /usr/local/.
> martin@merkaba:~> ls -lid /usr/local/bin/..
> 27138 drwxrwsr-x 1 root staff 62 Aug 15  2014 /usr/local/bin/..
> 
> On other filesystems like Ext4 I see the actual number of hardlinks to the 
> directory.
> 
> Is this intended behaviour of BTRFS or a bug?

Intended behaviour, this has been asked in the past, I don't have the
link sorry, try searching the mailinglist archives.

Re: number of hardlinks for directory in ls -lid always 1?

[Martin Steigerwald]

Am Dienstag, 17. März 2015, 17:07:17 schrieb David Sterba:
> On Tue, Mar 17, 2015 at 02:33:30PM +0100, Martin Steigerwald wrote:
> > On BTRFS I see
> > 
> > martin@merkaba:~> ls -lid /usr/local
> > 27138 drwxrwsr-x 1 root staff 62 Aug 15  2014 /usr/local
> > martin@merkaba:~> ls -lid /usr/local/.
> > 27138 drwxrwsr-x 1 root staff 62 Aug 15  2014 /usr/local/.
> > martin@merkaba:~> ls -lid /usr/local/bin/..
> > 27138 drwxrwsr-x 1 root staff 62 Aug 15  2014 /usr/local/bin/..
> > 
> > On other filesystems like Ext4 I see the actual number of hardlinks to
> > the directory.
> > 
> > Is this intended behaviour of BTRFS or a bug?
> 
> Intended behaviour, this has been asked in the past, I don't have the
> link sorry, try searching the mailinglist archives.

I did, yet I didn´t find anything.

Any idea on what words except "hardlinks", "hard links", "number", "ls -l" 
I can throw at Baloo fulltext search on BTRFS mailing list folder in 
KMail? 

I also tried various search terms with Startpage.

Thanks,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: number of hardlinks for directory in ls -lid always 1?

[David Sterba]

On Wed, Mar 18, 2015 at 02:31:43PM +0100, Martin Steigerwald wrote:
> Am Dienstag, 17. März 2015, 17:07:17 schrieb David Sterba:
> > On Tue, Mar 17, 2015 at 02:33:30PM +0100, Martin Steigerwald wrote:
> > > On BTRFS I see
> > > 
> > > martin@merkaba:~> ls -lid /usr/local
> > > 27138 drwxrwsr-x 1 root staff 62 Aug 15  2014 /usr/local
> > > martin@merkaba:~> ls -lid /usr/local/.
> > > 27138 drwxrwsr-x 1 root staff 62 Aug 15  2014 /usr/local/.
> > > martin@merkaba:~> ls -lid /usr/local/bin/..
> > > 27138 drwxrwsr-x 1 root staff 62 Aug 15  2014 /usr/local/bin/..
> > > 
> > > On other filesystems like Ext4 I see the actual number of hardlinks to
> > > the directory.
> > > 
> > > Is this intended behaviour of BTRFS or a bug?
> > 
> > Intended behaviour, this has been asked in the past, I don't have the
> > link sorry, try searching the mailinglist archives.

http://thread.gmane.org/gmane.comp.file-systems.btrfs/14634

"Directories always have a link count of 1 in btrfs.  This tells find
not to use the link count as the count of subdirectories in the
directory."

http://thread.gmane.org/gmane.comp.file-systems.btrfs/29906

"As I understand it, inferring the number of directory entries from
st_nlink is an optimization that isn't universally valid. If that
count is 1, it must be considered invalid, and programs that don't
handle this correctly are broken.  Coreutils handle this, at least..."

Re: number of hardlinks for directory in ls -lid always 1?

[Martin Steigerwald]

Am Mittwoch, 18. März 2015, 14:52:30 schrieb David Sterba:
> On Wed, Mar 18, 2015 at 02:31:43PM +0100, Martin Steigerwald wrote:
> > Am Dienstag, 17. März 2015, 17:07:17 schrieb David Sterba:
> > > On Tue, Mar 17, 2015 at 02:33:30PM +0100, Martin Steigerwald wrote:
> > > > On BTRFS I see
> > > > 
> > > > martin@merkaba:~> ls -lid /usr/local
> > > > 27138 drwxrwsr-x 1 root staff 62 Aug 15  2014 /usr/local
> > > > martin@merkaba:~> ls -lid /usr/local/.
> > > > 27138 drwxrwsr-x 1 root staff 62 Aug 15  2014 /usr/local/.
> > > > martin@merkaba:~> ls -lid /usr/local/bin/..
> > > > 27138 drwxrwsr-x 1 root staff 62 Aug 15  2014 /usr/local/bin/..
> > > > 
> > > > On other filesystems like Ext4 I see the actual number of
> > > > hardlinks to
> > > > the directory.
> > > > 
> > > > Is this intended behaviour of BTRFS or a bug?
> > > 
> > > Intended behaviour, this has been asked in the past, I don't have
> > > the
> > > link sorry, try searching the mailinglist archives.
> 
> http://thread.gmane.org/gmane.comp.file-systems.btrfs/14634
> 
> "Directories always have a link count of 1 in btrfs.  This tells find
> not to use the link count as the count of subdirectories in the
> directory."
> 
> http://thread.gmane.org/gmane.comp.file-systems.btrfs/29906
> 
> "As I understand it, inferring the number of directory entries from
> st_nlink is an optimization that isn't universally valid. If that
> count is 1, it must be considered invalid, and programs that don't
> handle this correctly are broken.  Coreutils handle this, at least..."

Okay, thanks. That was before I subscribed to BTRFS mailinglist so not in 
local mail archive.

It explains that having a correct hardlink number for directory is not 
mandatory, but it doesn´t explain why BTRFS always has 1 in there instead 
of the actual count of hardlinks. Is this an performance optimization for 
BTRFS or are there any other reasons why BTRFS does it this way?

Thanks,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: number of hardlinks for directory in ls -lid always 1?

[David Sterba]

On Wed, Mar 18, 2015 at 03:23:50PM +0100, Martin Steigerwald wrote:
> It explains that having a correct hardlink number for directory is not 
> mandatory, but it doesn´t explain why BTRFS always has 1 in there instead 
> of the actual count of hardlinks. Is this an performance optimization for 
> BTRFS or are there any other reasons why BTRFS does it this way?

I believe it's for performance reasons. New inodes do not update the
parent directory metadata wrt link counts, compared to other filesystems
that do that.

The real performance hit could be noticeable. The directory inode is
cached in memory, so first update would be a bit slower, but the
metadata block needs to be cow-ed on each new file. It's stress on
b-tree locking and allocating new buffers for the metadata blocks.

Re: number of hardlinks for directory in ls -lid always 1?

[Filipe David Manana]

On Thu, Mar 19, 2015 at 1:21 PM, David Sterba <dsterba@suse.cz> wrote:
> On Wed, Mar 18, 2015 at 03:23:50PM +0100, Martin Steigerwald wrote:
>> It explains that having a correct hardlink number for directory is not
>> mandatory, but it doesn´t explain why BTRFS always has 1 in there instead
>> of the actual count of hardlinks. Is this an performance optimization for
>> BTRFS or are there any other reasons why BTRFS does it this way?
>
> I believe it's for performance reasons. New inodes do not update the
> parent directory metadata wrt link counts, compared to other filesystems
> that do that.

Weird. Because creating a new inode implies adding the dentry to the
parent directory, which implies updating the directory's i_size.

>
> The real performance hit could be noticeable. The directory inode is
> cached in memory, so first update would be a bit slower, but the
> metadata block needs to be cow-ed on each new file. It's stress on
> b-tree locking and allocating new buffers for the metadata blocks.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Filipe David Manana,

"Reasonable men adapt themselves to the world.
 Unreasonable men adapt the world to themselves.
 That's why all progress depends on unreasonable men."

Re: number of hardlinks for directory in ls -lid always 1?

[Kai Krakow]

Filipe David Manana <fdmanana@gmail.com> schrieb:

> On Thu, Mar 19, 2015 at 1:21 PM, David Sterba <dsterba@suse.cz> wrote:
>> On Wed, Mar 18, 2015 at 03:23:50PM +0100, Martin Steigerwald wrote:
>>> It explains that having a correct hardlink number for directory is not
>>> mandatory, but it doesn´t explain why BTRFS always has 1 in there
>>> instead of the actual count of hardlinks. Is this an performance
>>> optimization for BTRFS or are there any other reasons why BTRFS does it
>>> this way?
>>
>> I believe it's for performance reasons. New inodes do not update the
>> parent directory metadata wrt link counts, compared to other filesystems
>> that do that.
> 
> Weird. Because creating a new inode implies adding the dentry to the
> parent directory, which implies updating the directory's i_size.

Maybe related to snapshots? Which brings us back to COW and performance:

>> The real performance hit could be noticeable. The directory inode is
>> cached in memory, so first update would be a bit slower, but the
>> metadata block needs to be cow-ed on each new file. It's stress on
>> b-tree locking and allocating new buffers for the metadata blocks.
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 
> 
-- 
Replies to list only preferred.

Re: number of hardlinks for directory in ls -lid always 1?

[David Sterba]

On Thu, Mar 19, 2015 at 09:47:15PM +0000, Filipe David Manana wrote:
> On Thu, Mar 19, 2015 at 1:21 PM, David Sterba <dsterba@suse.cz> wrote:
> > On Wed, Mar 18, 2015 at 03:23:50PM +0100, Martin Steigerwald wrote:
> >> It explains that having a correct hardlink number for directory is not
> >> mandatory, but it doesn´t explain why BTRFS always has 1 in there instead
> >> of the actual count of hardlinks. Is this an performance optimization for
> >> BTRFS or are there any other reasons why BTRFS does it this way?
> >
> > I believe it's for performance reasons. New inodes do not update the
> > parent directory metadata wrt link counts, compared to other filesystems
> > that do that.
> 
> Weird. Because creating a new inode implies adding the dentry to the
> parent directory, which implies updating the directory's i_size.

Ah right, sorry, I've missed that.

Re: number of hardlinks for directory in ls -lid always 1?

[David Sterba]

On Thu, Mar 19, 2015 at 09:47:15PM +0000, Filipe David Manana wrote:
> On Thu, Mar 19, 2015 at 1:21 PM, David Sterba <dsterba@suse.cz> wrote:
> > On Wed, Mar 18, 2015 at 03:23:50PM +0100, Martin Steigerwald wrote:
> >> It explains that having a correct hardlink number for directory is not
> >> mandatory, but it doesn´t explain why BTRFS always has 1 in there instead
> >> of the actual count of hardlinks. Is this an performance optimization for
> >> BTRFS or are there any other reasons why BTRFS does it this way?
> >
> > I believe it's for performance reasons. New inodes do not update the
> > parent directory metadata wrt link counts, compared to other filesystems
> > that do that.
> 
> Weird. Because creating a new inode implies adding the dentry to the
> parent directory, which implies updating the directory's i_size.

I wonder why the link count is not maintained then. The directory inode
is modified anyway, add a few ifs here and there will not make things
worse and we could get rid of this special behaviour, plus find could
use the link count reliably.

It can be turned on in a fully backward compatible way:

* new directories/subvolumes will get dir nlink set to 2
* if mkdir/rmdir/move sees a parent link count of 1, no change
* otherwise inc/dec the link count for new subdirectoris/subvolumes

Jeff Liu reported a problem, without further details
http://article.gmane.org/gmane.comp.file-systems.btrfs/14628

so I still might be missing something.

Re: number of hardlinks for directory in ls -lid always 1?

[Filipe David Manana]

On Fri, Mar 20, 2015 at 12:39 PM, David Sterba <dsterba@suse.cz> wrote:
> On Thu, Mar 19, 2015 at 09:47:15PM +0000, Filipe David Manana wrote:
>> On Thu, Mar 19, 2015 at 1:21 PM, David Sterba <dsterba@suse.cz> wrote:
>> > On Wed, Mar 18, 2015 at 03:23:50PM +0100, Martin Steigerwald wrote:
>> >> It explains that having a correct hardlink number for directory is not
>> >> mandatory, but it doesn´t explain why BTRFS always has 1 in there instead
>> >> of the actual count of hardlinks. Is this an performance optimization for
>> >> BTRFS or are there any other reasons why BTRFS does it this way?
>> >
>> > I believe it's for performance reasons. New inodes do not update the
>> > parent directory metadata wrt link counts, compared to other filesystems
>> > that do that.
>>
>> Weird. Because creating a new inode implies adding the dentry to the
>> parent directory, which implies updating the directory's i_size.
>
> I wonder why the link count is not maintained then. The directory inode
> is modified anyway, add a few ifs here and there will not make things
> worse and we could get rid of this special behaviour, plus find could
> use the link count reliably.
>
> It can be turned on in a fully backward compatible way:
>
> * new directories/subvolumes will get dir nlink set to 2
> * if mkdir/rmdir/move sees a parent link count of 1, no change
> * otherwise inc/dec the link count for new subdirectoris/subvolumes
>
> Jeff Liu reported a problem, without further details
> http://article.gmane.org/gmane.comp.file-systems.btrfs/14628
>
> so I still might be missing something.

Maybe this was all before delayed inodes were introduced, where
updating inode items always implied touching the btrees.



-- 
Filipe David Manana,

"Reasonable men adapt themselves to the world.
 Unreasonable men adapt the world to themselves.
 That's why all progress depends on unreasonable men."

Re: number of hardlinks for directory in ls -lid always 1?

[Martin Steigerwald]

Am Freitag, 20. März 2015, 12:59:14 schrieb Filipe David Manana:
> On Fri, Mar 20, 2015 at 12:39 PM, David Sterba <dsterba@suse.cz> wrote:
> > On Thu, Mar 19, 2015 at 09:47:15PM +0000, Filipe David Manana wrote:
> >> On Thu, Mar 19, 2015 at 1:21 PM, David Sterba <dsterba@suse.cz> 
wrote:
> >> > On Wed, Mar 18, 2015 at 03:23:50PM +0100, Martin Steigerwald wrote:
> >> >> It explains that having a correct hardlink number for directory is
> >> >> not
> >> >> mandatory, but it doesn´t explain why BTRFS always has 1 in there
> >> >> instead of the actual count of hardlinks. Is this an performance
> >> >> optimization for BTRFS or are there any other reasons why BTRFS
> >> >> does it this way?>> > 
> >> > I believe it's for performance reasons. New inodes do not update
> >> > the
> >> > parent directory metadata wrt link counts, compared to other
> >> > filesystems that do that.
> >> 
> >> Weird. Because creating a new inode implies adding the dentry to the
> >> parent directory, which implies updating the directory's i_size.
> > 
> > I wonder why the link count is not maintained then. The directory
> > inode
> > is modified anyway, add a few ifs here and there will not make things
> > worse and we could get rid of this special behaviour, plus find could
> > use the link count reliably.
> > 
> > It can be turned on in a fully backward compatible way:
> > 
> > * new directories/subvolumes will get dir nlink set to 2
> > * if mkdir/rmdir/move sees a parent link count of 1, no change
> > * otherwise inc/dec the link count for new subdirectoris/subvolumes
> > 
> > Jeff Liu reported a problem, without further details
> > http://article.gmane.org/gmane.comp.file-systems.btrfs/14628
> > 
> > so I still might be missing something.
> 
> Maybe this was all before delayed inodes were introduced, where
> updating inode items always implied touching the btrees.

Interesting.

Well this came up due to an exercise to find all hardlinks to /usr/local 
in a Linux training. And one participant of the training did this on a 
SLES 11 SP 3 VM with BTRFS.

I did some search and indeed see nothing about link count for directories 
related to POSIX standard. Incrementing link count only seems to be 
mentioned for files:

http://pubs.opengroup.org/onlinepubs/009695399/functions/link.html

Even when people tend to rely on this behavior for forensics as in:

http://digital-forensics.sans.org/blog/2009/06/19/directory-link-counts-and-hidden-directories/

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7