Apparent metadata corruption (file that simultaneously does/does not exist) on kernel 3.17.3

Apparent metadata corruption (file that simultaneously does/does not exist) on kernel 3.17.3

[Daniel Miranda]

Hello,

After I had some brief stability issues with my computer, it seems
some form of metadata corruption took place in my BTRFS filesystem,
and now a particular file seems to exist, but I cannot access any
details on it or delete it.

If I try to `ls` in the directory it is in, that's what I get:

ls: cannot access string.h: No such file or directory
total 0
drwxr-xr-x. 1 danielkza mock 16 Nov 21 14:18 ./
drwxr-xr-x. 1 danielkza mock  6 Nov 21 14:18 ../
-?????????? ? ?         ?     ?            ? string.h

If I try to delete it I get:

rm: cannot remove ‘string.h’: No such file or directory

I'm using kernel 3.17.3 from Fedora 21. I got no messages on dmesg or
anything of the sort. I know the btrfs fsck situation is complicated,
but is there any utility I should use to try and repair this? Losing
this file is not a problem, it's just one header from the kernel I was
building.

Regards,
Daniel Miranda

Re: Apparent metadata corruption (file that simultaneously does/does not exist) on kernel 3.17.3

[Qu Wenruo]

Hi,

What's the btrfsck output? Without --repair option.

Also, if it is OK for you, would you please dump the btrfs with 
'btrfs-image' command?
'-c 9' option is highly recommended considering the size of it.
This will helps a lot for developers to test the btrfsck repair function.

Thanks,
Qu

-------- Original Message --------
Subject: Apparent metadata corruption (file that simultaneously 
does/does not exist) on kernel 3.17.3
From: Daniel Miranda <danielkza2@gmail.com>
To: <linux-btrfs@vger.kernel.org>
Date: 2014年11月25日 13:04
> Hello,
>
> After I had some brief stability issues with my computer, it seems
> some form of metadata corruption took place in my BTRFS filesystem,
> and now a particular file seems to exist, but I cannot access any
> details on it or delete it.
>
> If I try to `ls` in the directory it is in, that's what I get:
>
> ls: cannot access string.h: No such file or directory
> total 0
> drwxr-xr-x. 1 danielkza mock 16 Nov 21 14:18 ./
> drwxr-xr-x. 1 danielkza mock  6 Nov 21 14:18 ../
> -?????????? ? ?         ?     ?            ? string.h
>
> If I try to delete it I get:
>
> rm: cannot remove ‘string.h’: No such file or directory
>
> I'm using kernel 3.17.3 from Fedora 21. I got no messages on dmesg or
> anything of the sort. I know the btrfs fsck situation is complicated,
> but is there any utility I should use to try and repair this? Losing
> this file is not a problem, it's just one header from the kernel I was
> building.
>
> Regards,
> Daniel Miranda
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Apparent metadata corruption (file that simultaneously does/does not exist) on kernel 3.17.3

[Daniel Miranda]

I'll go run that and get you the output.

I can do the image dump, sure. I don't know how long it might take to
upload it somewhere though. Right now `btrfs fi df` shows about 2GiB
of metadata (it's a 120GiB volume). I'll see how large it ends up
after compression.

Thanks for the quick response,
Daniel

On Tue, Nov 25, 2014 at 3:10 AM, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:
> Hi,
>
> What's the btrfsck output? Without --repair option.
>
> Also, if it is OK for you, would you please dump the btrfs with
> 'btrfs-image' command?
> '-c 9' option is highly recommended considering the size of it.
> This will helps a lot for developers to test the btrfsck repair function.
>
> Thanks,
> Qu
>
>
> -------- Original Message --------
> Subject: Apparent metadata corruption (file that simultaneously does/does
> not exist) on kernel 3.17.3
> From: Daniel Miranda <danielkza2@gmail.com>
> To: <linux-btrfs@vger.kernel.org>
> Date: 2014年11月25日 13:04
>>
>> Hello,
>>
>> After I had some brief stability issues with my computer, it seems
>> some form of metadata corruption took place in my BTRFS filesystem,
>> and now a particular file seems to exist, but I cannot access any
>> details on it or delete it.
>>
>> If I try to `ls` in the directory it is in, that's what I get:
>>
>> ls: cannot access string.h: No such file or directory
>> total 0
>> drwxr-xr-x. 1 danielkza mock 16 Nov 21 14:18 ./
>> drwxr-xr-x. 1 danielkza mock  6 Nov 21 14:18 ../
>> -?????????? ? ?         ?     ?            ? string.h
>>
>> If I try to delete it I get:
>>
>> rm: cannot remove ‘string.h’: No such file or directory
>>
>> I'm using kernel 3.17.3 from Fedora 21. I got no messages on dmesg or
>> anything of the sort. I know the btrfs fsck situation is complicated,
>> but is there any utility I should use to try and repair this? Losing
>> this file is not a problem, it's just one header from the kernel I was
>> building.
>>
>> Regards,
>> Daniel Miranda
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>

Re: Apparent metadata corruption (file that simultaneously does/does not exist) on kernel 3.17.3

[Qu Wenruo]

-------- Original Message --------
Subject: Re: Apparent metadata corruption (file that simultaneously 
does/does not exist) on kernel 3.17.3
From: Daniel Miranda <danielkza2@gmail.com>
To: Qu Wenruo <quwenruo@cn.fujitsu.com>
Date: 2014年11月25日 13:14
> I'll go run that and get you the output.
Thanks.
>
> I can do the image dump, sure. I don't know how long it might take to
> upload it somewhere though. Right now `btrfs fi df` shows about 2GiB
> of metadata (it's a 120GiB volume). I'll see how large it ends up
> after compression.
120G volume seems quite small, compared the images I received recently 
(1T x2 RAID1 and 4T single).
With '-c 9' it shouldn't be too huge I think(The 1T raid1 is about 1G 
metadata with -c9).

BTW, btrfs-image dump will have all the filenames and hierarchy, even 
without its data,
it is still better considering your privacy twice before uploading.

Thanks,
Qu
>
> Thanks for the quick response,
> Daniel
>
> On Tue, Nov 25, 2014 at 3:10 AM, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:
>> Hi,
>>
>> What's the btrfsck output? Without --repair option.
>>
>> Also, if it is OK for you, would you please dump the btrfs with
>> 'btrfs-image' command?
>> '-c 9' option is highly recommended considering the size of it.
>> This will helps a lot for developers to test the btrfsck repair function.
>>
>> Thanks,
>> Qu
>>
>>
>> -------- Original Message --------
>> Subject: Apparent metadata corruption (file that simultaneously does/does
>> not exist) on kernel 3.17.3
>> From: Daniel Miranda <danielkza2@gmail.com>
>> To: <linux-btrfs@vger.kernel.org>
>> Date: 2014年11月25日 13:04
>>> Hello,
>>>
>>> After I had some brief stability issues with my computer, it seems
>>> some form of metadata corruption took place in my BTRFS filesystem,
>>> and now a particular file seems to exist, but I cannot access any
>>> details on it or delete it.
>>>
>>> If I try to `ls` in the directory it is in, that's what I get:
>>>
>>> ls: cannot access string.h: No such file or directory
>>> total 0
>>> drwxr-xr-x. 1 danielkza mock 16 Nov 21 14:18 ./
>>> drwxr-xr-x. 1 danielkza mock  6 Nov 21 14:18 ../
>>> -?????????? ? ?         ?     ?            ? string.h
>>>
>>> If I try to delete it I get:
>>>
>>> rm: cannot remove ‘string.h’: No such file or directory
>>>
>>> I'm using kernel 3.17.3 from Fedora 21. I got no messages on dmesg or
>>> anything of the sort. I know the btrfs fsck situation is complicated,
>>> but is there any utility I should use to try and repair this? Losing
>>> this file is not a problem, it's just one header from the kernel I was
>>> building.
>>>
>>> Regards,
>>> Daniel Miranda
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>

Re: Apparent metadata corruption (file that simultaneously does/does not exist) on kernel 3.17.3

[Daniel Miranda]

Here are the logs. I'll send you a link to my dump directly after I
finish uploading it. Please notify me when you have downloaded it so I
can delete it.

checking extents
checking free space cache
checking fs roots
root 5 inode 17149868 errors 2000, link count wrong
    unresolved ref dir 17182377 index 245 namelen 8 name string.h
filetype 1 errors 1, no dir item
root 5 inode 17182377 errors 200, dir isize wrong
Checking filesystem on /dev/mapper/fedora_daniel--pc-root
UUID: fef8f718-0622-4cb1-9597-749650d366a4
found 55108022156 bytes used err is 1
total csum bytes: 89787396
total tree bytes: 2303455232
total fs tree bytes: 2024841216
total extent tree bytes: 145272832
btree space waste bytes: 529672422
file data blocks allocated: 253414481920
 referenced 94127726592
Btrfs v3.17


Regards,
Daniel

On Tue, Nov 25, 2014 at 3:20 AM, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:
>
> -------- Original Message --------
> Subject: Re: Apparent metadata corruption (file that simultaneously
> does/does not exist) on kernel 3.17.3
> From: Daniel Miranda <danielkza2@gmail.com>
> To: Qu Wenruo <quwenruo@cn.fujitsu.com>
> Date: 2014年11月25日 13:14
>>
>> I'll go run that and get you the output.
>
> Thanks.
>>
>>
>> I can do the image dump, sure. I don't know how long it might take to
>> upload it somewhere though. Right now `btrfs fi df` shows about 2GiB
>> of metadata (it's a 120GiB volume). I'll see how large it ends up
>> after compression.
>
> 120G volume seems quite small, compared the images I received recently (1T
> x2 RAID1 and 4T single).
> With '-c 9' it shouldn't be too huge I think(The 1T raid1 is about 1G
> metadata with -c9).
>
> BTW, btrfs-image dump will have all the filenames and hierarchy, even
> without its data,
> it is still better considering your privacy twice before uploading.
>
> Thanks,
> Qu
>
>>
>> Thanks for the quick response,
>> Daniel
>>
>> On Tue, Nov 25, 2014 at 3:10 AM, Qu Wenruo <quwenruo@cn.fujitsu.com>
>> wrote:
>>>
>>> Hi,
>>>
>>> What's the btrfsck output? Without --repair option.
>>>
>>> Also, if it is OK for you, would you please dump the btrfs with
>>> 'btrfs-image' command?
>>> '-c 9' option is highly recommended considering the size of it.
>>> This will helps a lot for developers to test the btrfsck repair function.
>>>
>>> Thanks,
>>> Qu
>>>
>>>
>>> -------- Original Message --------
>>> Subject: Apparent metadata corruption (file that simultaneously does/does
>>> not exist) on kernel 3.17.3
>>> From: Daniel Miranda <danielkza2@gmail.com>
>>> To: <linux-btrfs@vger.kernel.org>
>>> Date: 2014年11月25日 13:04
>>>>
>>>> Hello,
>>>>
>>>> After I had some brief stability issues with my computer, it seems
>>>> some form of metadata corruption took place in my BTRFS filesystem,
>>>> and now a particular file seems to exist, but I cannot access any
>>>> details on it or delete it.
>>>>
>>>> If I try to `ls` in the directory it is in, that's what I get:
>>>>
>>>> ls: cannot access string.h: No such file or directory
>>>> total 0
>>>> drwxr-xr-x. 1 danielkza mock 16 Nov 21 14:18 ./
>>>> drwxr-xr-x. 1 danielkza mock  6 Nov 21 14:18 ../
>>>> -?????????? ? ?         ?     ?            ? string.h
>>>>
>>>> If I try to delete it I get:
>>>>
>>>> rm: cannot remove ‘string.h’: No such file or directory
>>>>
>>>> I'm using kernel 3.17.3 from Fedora 21. I got no messages on dmesg or
>>>> anything of the sort. I know the btrfs fsck situation is complicated,
>>>> but is there any utility I should use to try and repair this? Losing
>>>> this file is not a problem, it's just one header from the kernel I was
>>>> building.
>>>>
>>>> Regards,
>>>> Daniel Miranda
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
>>>> in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>>
>

Re: Apparent metadata corruption (file that simultaneously does/does not exist) on kernel 3.17.3

[Qu Wenruo]

-------- Original Message --------
Subject: Re: Apparent metadata corruption (file that simultaneously 
does/does not exist) on kernel 3.17.3
From: Daniel Miranda <danielkza2@gmail.com>
To: Qu Wenruo <quwenruo@cn.fujitsu.com>
Date: 2014年11月25日 15:20
> Here are the logs. I'll send you a link to my dump directly after I
> finish uploading it. Please notify me when you have downloaded it so I
> can delete it.
>
> checking extents
> checking free space cache
> checking fs roots
> root 5 inode 17149868 errors 2000, link count wrong
>      unresolved ref dir 17182377 index 245 namelen 8 name string.h
> filetype 1 errors 1, no dir item
link count error seems resolved by Josef's patch commit already in 3.17.2.
If using 3.17.2, josef's commit will rebuild the dir item and dir index.
> root 5 inode 17182377 errors 200, dir isize wrong
This isize error seems caused by previous line.
If 3.17.2 can repair above problem, it should not be a problem and will 
disappear.

According to the above output, btrfsck --repair with btrfs-progs 3.17.2 
has a good chance repairing it.
Just have a try.

Thanks,
Qu
> Checking filesystem on /dev/mapper/fedora_daniel--pc-root
> UUID: fef8f718-0622-4cb1-9597-749650d366a4
> found 55108022156 bytes used err is 1
> total csum bytes: 89787396
> total tree bytes: 2303455232
> total fs tree bytes: 2024841216
> total extent tree bytes: 145272832
> btree space waste bytes: 529672422
> file data blocks allocated: 253414481920
>   referenced 94127726592
> Btrfs v3.17
>
>
> Regards,
> Daniel
>
> On Tue, Nov 25, 2014 at 3:20 AM, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:
>> -------- Original Message --------
>> Subject: Re: Apparent metadata corruption (file that simultaneously
>> does/does not exist) on kernel 3.17.3
>> From: Daniel Miranda <danielkza2@gmail.com>
>> To: Qu Wenruo <quwenruo@cn.fujitsu.com>
>> Date: 2014年11月25日 13:14
>>> I'll go run that and get you the output.
>> Thanks.
>>>
>>> I can do the image dump, sure. I don't know how long it might take to
>>> upload it somewhere though. Right now `btrfs fi df` shows about 2GiB
>>> of metadata (it's a 120GiB volume). I'll see how large it ends up
>>> after compression.
>> 120G volume seems quite small, compared the images I received recently (1T
>> x2 RAID1 and 4T single).
>> With '-c 9' it shouldn't be too huge I think(The 1T raid1 is about 1G
>> metadata with -c9).
>>
>> BTW, btrfs-image dump will have all the filenames and hierarchy, even
>> without its data,
>> it is still better considering your privacy twice before uploading.
>>
>> Thanks,
>> Qu
>>
>>> Thanks for the quick response,
>>> Daniel
>>>
>>> On Tue, Nov 25, 2014 at 3:10 AM, Qu Wenruo <quwenruo@cn.fujitsu.com>
>>> wrote:
>>>> Hi,
>>>>
>>>> What's the btrfsck output? Without --repair option.
>>>>
>>>> Also, if it is OK for you, would you please dump the btrfs with
>>>> 'btrfs-image' command?
>>>> '-c 9' option is highly recommended considering the size of it.
>>>> This will helps a lot for developers to test the btrfsck repair function.
>>>>
>>>> Thanks,
>>>> Qu
>>>>
>>>>
>>>> -------- Original Message --------
>>>> Subject: Apparent metadata corruption (file that simultaneously does/does
>>>> not exist) on kernel 3.17.3
>>>> From: Daniel Miranda <danielkza2@gmail.com>
>>>> To: <linux-btrfs@vger.kernel.org>
>>>> Date: 2014年11月25日 13:04
>>>>> Hello,
>>>>>
>>>>> After I had some brief stability issues with my computer, it seems
>>>>> some form of metadata corruption took place in my BTRFS filesystem,
>>>>> and now a particular file seems to exist, but I cannot access any
>>>>> details on it or delete it.
>>>>>
>>>>> If I try to `ls` in the directory it is in, that's what I get:
>>>>>
>>>>> ls: cannot access string.h: No such file or directory
>>>>> total 0
>>>>> drwxr-xr-x. 1 danielkza mock 16 Nov 21 14:18 ./
>>>>> drwxr-xr-x. 1 danielkza mock  6 Nov 21 14:18 ../
>>>>> -?????????? ? ?         ?     ?            ? string.h
>>>>>
>>>>> If I try to delete it I get:
>>>>>
>>>>> rm: cannot remove ‘string.h’: No such file or directory
>>>>>
>>>>> I'm using kernel 3.17.3 from Fedora 21. I got no messages on dmesg or
>>>>> anything of the sort. I know the btrfs fsck situation is complicated,
>>>>> but is there any utility I should use to try and repair this? Losing
>>>>> this file is not a problem, it's just one header from the kernel I was
>>>>> building.
>>>>>
>>>>> Regards,
>>>>> Daniel Miranda
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
>>>>> in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>

Re: Apparent metadata corruption (file that simultaneously does/does not exist) on kernel 3.17.3

[Daniel Miranda]

I just ran the repair but the ghost file has not disappeared, unfortunately.

On Tue, Nov 25, 2014 at 5:26 AM, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:
>
> -------- Original Message --------
> Subject: Re: Apparent metadata corruption (file that simultaneously
> does/does not exist) on kernel 3.17.3
> From: Daniel Miranda <danielkza2@gmail.com>
> To: Qu Wenruo <quwenruo@cn.fujitsu.com>
> Date: 2014年11月25日 15:20
>>
>> Here are the logs. I'll send you a link to my dump directly after I
>> finish uploading it. Please notify me when you have downloaded it so I
>> can delete it.
>>
>> checking extents
>> checking free space cache
>> checking fs roots
>> root 5 inode 17149868 errors 2000, link count wrong
>>      unresolved ref dir 17182377 index 245 namelen 8 name string.h
>> filetype 1 errors 1, no dir item
>
> link count error seems resolved by Josef's patch commit already in 3.17.2.
> If using 3.17.2, josef's commit will rebuild the dir item and dir index.
>>
>> root 5 inode 17182377 errors 200, dir isize wrong
>
> This isize error seems caused by previous line.
> If 3.17.2 can repair above problem, it should not be a problem and will
> disappear.
>
> According to the above output, btrfsck --repair with btrfs-progs 3.17.2 has
> a good chance repairing it.
> Just have a try.
>
> Thanks,
> Qu
>
>> Checking filesystem on /dev/mapper/fedora_daniel--pc-root
>> UUID: fef8f718-0622-4cb1-9597-749650d366a4
>> found 55108022156 bytes used err is 1
>> total csum bytes: 89787396
>> total tree bytes: 2303455232
>> total fs tree bytes: 2024841216
>> total extent tree bytes: 145272832
>> btree space waste bytes: 529672422
>> file data blocks allocated: 253414481920
>>   referenced 94127726592
>> Btrfs v3.17
>>
>>
>> Regards,
>> Daniel
>>
>> On Tue, Nov 25, 2014 at 3:20 AM, Qu Wenruo <quwenruo@cn.fujitsu.com>
>> wrote:
>>>
>>> -------- Original Message --------
>>> Subject: Re: Apparent metadata corruption (file that simultaneously
>>> does/does not exist) on kernel 3.17.3
>>> From: Daniel Miranda <danielkza2@gmail.com>
>>> To: Qu Wenruo <quwenruo@cn.fujitsu.com>
>>> Date: 2014年11月25日 13:14
>>>>
>>>> I'll go run that and get you the output.
>>>
>>> Thanks.
>>>>
>>>>
>>>> I can do the image dump, sure. I don't know how long it might take to
>>>> upload it somewhere though. Right now `btrfs fi df` shows about 2GiB
>>>> of metadata (it's a 120GiB volume). I'll see how large it ends up
>>>> after compression.
>>>
>>> 120G volume seems quite small, compared the images I received recently
>>> (1T
>>> x2 RAID1 and 4T single).
>>> With '-c 9' it shouldn't be too huge I think(The 1T raid1 is about 1G
>>> metadata with -c9).
>>>
>>> BTW, btrfs-image dump will have all the filenames and hierarchy, even
>>> without its data,
>>> it is still better considering your privacy twice before uploading.
>>>
>>> Thanks,
>>> Qu
>>>
>>>> Thanks for the quick response,
>>>> Daniel
>>>>
>>>> On Tue, Nov 25, 2014 at 3:10 AM, Qu Wenruo <quwenruo@cn.fujitsu.com>
>>>> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> What's the btrfsck output? Without --repair option.
>>>>>
>>>>> Also, if it is OK for you, would you please dump the btrfs with
>>>>> 'btrfs-image' command?
>>>>> '-c 9' option is highly recommended considering the size of it.
>>>>> This will helps a lot for developers to test the btrfsck repair
>>>>> function.
>>>>>
>>>>> Thanks,
>>>>> Qu
>>>>>
>>>>>
>>>>> -------- Original Message --------
>>>>> Subject: Apparent metadata corruption (file that simultaneously
>>>>> does/does
>>>>> not exist) on kernel 3.17.3
>>>>> From: Daniel Miranda <danielkza2@gmail.com>
>>>>> To: <linux-btrfs@vger.kernel.org>
>>>>> Date: 2014年11月25日 13:04
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> After I had some brief stability issues with my computer, it seems
>>>>>> some form of metadata corruption took place in my BTRFS filesystem,
>>>>>> and now a particular file seems to exist, but I cannot access any
>>>>>> details on it or delete it.
>>>>>>
>>>>>> If I try to `ls` in the directory it is in, that's what I get:
>>>>>>
>>>>>> ls: cannot access string.h: No such file or directory
>>>>>> total 0
>>>>>> drwxr-xr-x. 1 danielkza mock 16 Nov 21 14:18 ./
>>>>>> drwxr-xr-x. 1 danielkza mock  6 Nov 21 14:18 ../
>>>>>> -?????????? ? ?         ?     ?            ? string.h
>>>>>>
>>>>>> If I try to delete it I get:
>>>>>>
>>>>>> rm: cannot remove ‘string.h’: No such file or directory
>>>>>>
>>>>>> I'm using kernel 3.17.3 from Fedora 21. I got no messages on dmesg or
>>>>>> anything of the sort. I know the btrfs fsck situation is complicated,
>>>>>> but is there any utility I should use to try and repair this? Losing
>>>>>> this file is not a problem, it's just one header from the kernel I was
>>>>>> building.
>>>>>>
>>>>>> Regards,
>>>>>> Daniel Miranda
>>>>>> --
>>>>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
>>>>>> in
>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>
>>>>>
>

Re: Apparent metadata corruption (file that simultaneously does/does not exist) on kernel 3.17.3

[Qu Wenruo]

Hi Daniel,

With your btrfs-image dump, I tested with my patchset sent to maillist, 
my patchset succeeds fixing the image.

You can get the patchset and then apply it on 3.17.2, and --repair 
should fix it.
The file with nlink error will be moved to 'lost+found' dir.

Although the best fixing should be just adding the missing dir_index,
but currently the patchset does quite well and does not need to do any 
modify.

The patchset can be extracted using patchwork:
0001: https://patchwork.kernel.org/patch/5364131/mbox/
0002: https://patchwork.kernel.org/patch/5364141/mbox/
0003: https://patchwork.kernel.org/patch/5364101/mbox/
0004 v2: https://patchwork.kernel.org/patch/5383611/mbox/
0005 v2: https://patchwork.kernel.org/patch/5383601/mbox/
0006: https://patchwork.kernel.org/patch/5364151/mbox

Any feedback is welcomed to improve the patches.

Thanks,
Qu


-------- Original Message --------
Subject: Re: Apparent metadata corruption (file that simultaneously 
does/does not exist) on kernel 3.17.3
From: Daniel Miranda <danielkza2@gmail.com>
To: Qu Wenruo <quwenruo@cn.fujitsu.com>
Date: 2014年11月25日 15:42
> I just ran the repair but the ghost file has not disappeared, unfortunately.
>
> On Tue, Nov 25, 2014 at 5:26 AM, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:
>> -------- Original Message --------
>> Subject: Re: Apparent metadata corruption (file that simultaneously
>> does/does not exist) on kernel 3.17.3
>> From: Daniel Miranda <danielkza2@gmail.com>
>> To: Qu Wenruo <quwenruo@cn.fujitsu.com>
>> Date: 2014年11月25日 15:20
>>> Here are the logs. I'll send you a link to my dump directly after I
>>> finish uploading it. Please notify me when you have downloaded it so I
>>> can delete it.
>>>
>>> checking extents
>>> checking free space cache
>>> checking fs roots
>>> root 5 inode 17149868 errors 2000, link count wrong
>>>       unresolved ref dir 17182377 index 245 namelen 8 name string.h
>>> filetype 1 errors 1, no dir item
>> link count error seems resolved by Josef's patch commit already in 3.17.2.
>> If using 3.17.2, josef's commit will rebuild the dir item and dir index.
>>> root 5 inode 17182377 errors 200, dir isize wrong
>> This isize error seems caused by previous line.
>> If 3.17.2 can repair above problem, it should not be a problem and will
>> disappear.
>>
>> According to the above output, btrfsck --repair with btrfs-progs 3.17.2 has
>> a good chance repairing it.
>> Just have a try.
>>
>> Thanks,
>> Qu
>>
>>> Checking filesystem on /dev/mapper/fedora_daniel--pc-root
>>> UUID: fef8f718-0622-4cb1-9597-749650d366a4
>>> found 55108022156 bytes used err is 1
>>> total csum bytes: 89787396
>>> total tree bytes: 2303455232
>>> total fs tree bytes: 2024841216
>>> total extent tree bytes: 145272832
>>> btree space waste bytes: 529672422
>>> file data blocks allocated: 253414481920
>>>    referenced 94127726592
>>> Btrfs v3.17
>>>
>>>
>>> Regards,
>>> Daniel
>>>
>>> On Tue, Nov 25, 2014 at 3:20 AM, Qu Wenruo <quwenruo@cn.fujitsu.com>
>>> wrote:
>>>> -------- Original Message --------
>>>> Subject: Re: Apparent metadata corruption (file that simultaneously
>>>> does/does not exist) on kernel 3.17.3
>>>> From: Daniel Miranda <danielkza2@gmail.com>
>>>> To: Qu Wenruo <quwenruo@cn.fujitsu.com>
>>>> Date: 2014年11月25日 13:14
>>>>> I'll go run that and get you the output.
>>>> Thanks.
>>>>>
>>>>> I can do the image dump, sure. I don't know how long it might take to
>>>>> upload it somewhere though. Right now `btrfs fi df` shows about 2GiB
>>>>> of metadata (it's a 120GiB volume). I'll see how large it ends up
>>>>> after compression.
>>>> 120G volume seems quite small, compared the images I received recently
>>>> (1T
>>>> x2 RAID1 and 4T single).
>>>> With '-c 9' it shouldn't be too huge I think(The 1T raid1 is about 1G
>>>> metadata with -c9).
>>>>
>>>> BTW, btrfs-image dump will have all the filenames and hierarchy, even
>>>> without its data,
>>>> it is still better considering your privacy twice before uploading.
>>>>
>>>> Thanks,
>>>> Qu
>>>>
>>>>> Thanks for the quick response,
>>>>> Daniel
>>>>>
>>>>> On Tue, Nov 25, 2014 at 3:10 AM, Qu Wenruo <quwenruo@cn.fujitsu.com>
>>>>> wrote:
>>>>>> Hi,
>>>>>>
>>>>>> What's the btrfsck output? Without --repair option.
>>>>>>
>>>>>> Also, if it is OK for you, would you please dump the btrfs with
>>>>>> 'btrfs-image' command?
>>>>>> '-c 9' option is highly recommended considering the size of it.
>>>>>> This will helps a lot for developers to test the btrfsck repair
>>>>>> function.
>>>>>>
>>>>>> Thanks,
>>>>>> Qu
>>>>>>
>>>>>>
>>>>>> -------- Original Message --------
>>>>>> Subject: Apparent metadata corruption (file that simultaneously
>>>>>> does/does
>>>>>> not exist) on kernel 3.17.3
>>>>>> From: Daniel Miranda <danielkza2@gmail.com>
>>>>>> To: <linux-btrfs@vger.kernel.org>
>>>>>> Date: 2014年11月25日 13:04
>>>>>>> Hello,
>>>>>>>
>>>>>>> After I had some brief stability issues with my computer, it seems
>>>>>>> some form of metadata corruption took place in my BTRFS filesystem,
>>>>>>> and now a particular file seems to exist, but I cannot access any
>>>>>>> details on it or delete it.
>>>>>>>
>>>>>>> If I try to `ls` in the directory it is in, that's what I get:
>>>>>>>
>>>>>>> ls: cannot access string.h: No such file or directory
>>>>>>> total 0
>>>>>>> drwxr-xr-x. 1 danielkza mock 16 Nov 21 14:18 ./
>>>>>>> drwxr-xr-x. 1 danielkza mock  6 Nov 21 14:18 ../
>>>>>>> -?????????? ? ?         ?     ?            ? string.h
>>>>>>>
>>>>>>> If I try to delete it I get:
>>>>>>>
>>>>>>> rm: cannot remove ‘string.h’: No such file or directory
>>>>>>>
>>>>>>> I'm using kernel 3.17.3 from Fedora 21. I got no messages on dmesg or
>>>>>>> anything of the sort. I know the btrfs fsck situation is complicated,
>>>>>>> but is there any utility I should use to try and repair this? Losing
>>>>>>> this file is not a problem, it's just one header from the kernel I was
>>>>>>> building.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Daniel Miranda
>>>>>>> --
>>>>>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
>>>>>>> in
>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>>

Re: Apparent metadata corruption (file that simultaneously does/does not exist) on kernel 3.17.3

[Daniel Miranda]

Alright, I'll just have to understand how to build btrfs-progs now,
since I'm currently just using the packages from the Fedora repo.

Thanks for all the help and time spent so far,
Daniel


On Wed, Nov 26, 2014 at 12:41 AM, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:
> Hi Daniel,
>
> With your btrfs-image dump, I tested with my patchset sent to maillist, my
> patchset succeeds fixing the image.
>
> You can get the patchset and then apply it on 3.17.2, and --repair should
> fix it.
> The file with nlink error will be moved to 'lost+found' dir.
>
> Although the best fixing should be just adding the missing dir_index,
> but currently the patchset does quite well and does not need to do any
> modify.
>
> The patchset can be extracted using patchwork:
> 0001: https://patchwork.kernel.org/patch/5364131/mbox/
> 0002: https://patchwork.kernel.org/patch/5364141/mbox/
> 0003: https://patchwork.kernel.org/patch/5364101/mbox/
> 0004 v2: https://patchwork.kernel.org/patch/5383611/mbox/
> 0005 v2: https://patchwork.kernel.org/patch/5383601/mbox/
> 0006: https://patchwork.kernel.org/patch/5364151/mbox
>
> Any feedback is welcomed to improve the patches.
>
> Thanks,
> Qu
>
>
> -------- Original Message --------
> Subject: Re: Apparent metadata corruption (file that simultaneously
> does/does not exist) on kernel 3.17.3
> From: Daniel Miranda <danielkza2@gmail.com>
> To: Qu Wenruo <quwenruo@cn.fujitsu.com>
> Date: 2014年11月25日 15:42
>>
>> I just ran the repair but the ghost file has not disappeared,
>> unfortunately.
>>
>> On Tue, Nov 25, 2014 at 5:26 AM, Qu Wenruo <quwenruo@cn.fujitsu.com>
>> wrote:
>>>
>>> -------- Original Message --------
>>> Subject: Re: Apparent metadata corruption (file that simultaneously
>>> does/does not exist) on kernel 3.17.3
>>> From: Daniel Miranda <danielkza2@gmail.com>
>>> To: Qu Wenruo <quwenruo@cn.fujitsu.com>
>>> Date: 2014年11月25日 15:20
>>>>
>>>> Here are the logs. I'll send you a link to my dump directly after I
>>>> finish uploading it. Please notify me when you have downloaded it so I
>>>> can delete it.
>>>>
>>>> checking extents
>>>> checking free space cache
>>>> checking fs roots
>>>> root 5 inode 17149868 errors 2000, link count wrong
>>>>       unresolved ref dir 17182377 index 245 namelen 8 name string.h
>>>> filetype 1 errors 1, no dir item
>>>
>>> link count error seems resolved by Josef's patch commit already in
>>> 3.17.2.
>>> If using 3.17.2, josef's commit will rebuild the dir item and dir index.
>>>>
>>>> root 5 inode 17182377 errors 200, dir isize wrong
>>>
>>> This isize error seems caused by previous line.
>>> If 3.17.2 can repair above problem, it should not be a problem and will
>>> disappear.
>>>
>>> According to the above output, btrfsck --repair with btrfs-progs 3.17.2
>>> has
>>> a good chance repairing it.
>>> Just have a try.
>>>
>>> Thanks,
>>> Qu
>>>
>>>> Checking filesystem on /dev/mapper/fedora_daniel--pc-root
>>>> UUID: fef8f718-0622-4cb1-9597-749650d366a4
>>>> found 55108022156 bytes used err is 1
>>>> total csum bytes: 89787396
>>>> total tree bytes: 2303455232
>>>> total fs tree bytes: 2024841216
>>>> total extent tree bytes: 145272832
>>>> btree space waste bytes: 529672422
>>>> file data blocks allocated: 253414481920
>>>>    referenced 94127726592
>>>> Btrfs v3.17
>>>>
>>>>
>>>> Regards,
>>>> Daniel
>>>>
>>>> On Tue, Nov 25, 2014 at 3:20 AM, Qu Wenruo <quwenruo@cn.fujitsu.com>
>>>> wrote:
>>>>>
>>>>> -------- Original Message --------
>>>>> Subject: Re: Apparent metadata corruption (file that simultaneously
>>>>> does/does not exist) on kernel 3.17.3
>>>>> From: Daniel Miranda <danielkza2@gmail.com>
>>>>> To: Qu Wenruo <quwenruo@cn.fujitsu.com>
>>>>> Date: 2014年11月25日 13:14
>>>>>>
>>>>>> I'll go run that and get you the output.
>>>>>
>>>>> Thanks.
>>>>>>
>>>>>>
>>>>>> I can do the image dump, sure. I don't know how long it might take to
>>>>>> upload it somewhere though. Right now `btrfs fi df` shows about 2GiB
>>>>>> of metadata (it's a 120GiB volume). I'll see how large it ends up
>>>>>> after compression.
>>>>>
>>>>> 120G volume seems quite small, compared the images I received recently
>>>>> (1T
>>>>> x2 RAID1 and 4T single).
>>>>> With '-c 9' it shouldn't be too huge I think(The 1T raid1 is about 1G
>>>>> metadata with -c9).
>>>>>
>>>>> BTW, btrfs-image dump will have all the filenames and hierarchy, even
>>>>> without its data,
>>>>> it is still better considering your privacy twice before uploading.
>>>>>
>>>>> Thanks,
>>>>> Qu
>>>>>
>>>>>> Thanks for the quick response,
>>>>>> Daniel
>>>>>>
>>>>>> On Tue, Nov 25, 2014 at 3:10 AM, Qu Wenruo <quwenruo@cn.fujitsu.com>
>>>>>> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> What's the btrfsck output? Without --repair option.
>>>>>>>
>>>>>>> Also, if it is OK for you, would you please dump the btrfs with
>>>>>>> 'btrfs-image' command?
>>>>>>> '-c 9' option is highly recommended considering the size of it.
>>>>>>> This will helps a lot for developers to test the btrfsck repair
>>>>>>> function.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Qu
>>>>>>>
>>>>>>>
>>>>>>> -------- Original Message --------
>>>>>>> Subject: Apparent metadata corruption (file that simultaneously
>>>>>>> does/does
>>>>>>> not exist) on kernel 3.17.3
>>>>>>> From: Daniel Miranda <danielkza2@gmail.com>
>>>>>>> To: <linux-btrfs@vger.kernel.org>
>>>>>>> Date: 2014年11月25日 13:04
>>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> After I had some brief stability issues with my computer, it seems
>>>>>>>> some form of metadata corruption took place in my BTRFS filesystem,
>>>>>>>> and now a particular file seems to exist, but I cannot access any
>>>>>>>> details on it or delete it.
>>>>>>>>
>>>>>>>> If I try to `ls` in the directory it is in, that's what I get:
>>>>>>>>
>>>>>>>> ls: cannot access string.h: No such file or directory
>>>>>>>> total 0
>>>>>>>> drwxr-xr-x. 1 danielkza mock 16 Nov 21 14:18 ./
>>>>>>>> drwxr-xr-x. 1 danielkza mock  6 Nov 21 14:18 ../
>>>>>>>> -?????????? ? ?         ?     ?            ? string.h
>>>>>>>>
>>>>>>>> If I try to delete it I get:
>>>>>>>>
>>>>>>>> rm: cannot remove ‘string.h’: No such file or directory
>>>>>>>>
>>>>>>>> I'm using kernel 3.17.3 from Fedora 21. I got no messages on dmesg
>>>>>>>> or
>>>>>>>> anything of the sort. I know the btrfs fsck situation is
>>>>>>>> complicated,
>>>>>>>> but is there any utility I should use to try and repair this? Losing
>>>>>>>> this file is not a problem, it's just one header from the kernel I
>>>>>>>> was
>>>>>>>> building.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Daniel Miranda
>>>>>>>> --
>>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>>> linux-btrfs"
>>>>>>>> in
>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>>>
>>>>>>>
>

Re: Apparent metadata corruption (file that simultaneously does/does not exist) on kernel 3.17.3

[Daniel Miranda]

Hello again,

Sorry for the delay, I had some things to do this past week, including
figuring out the stability problems that I was having, but everything
is good now. I rebuilt the Fedora package for btrfs-progs 3.17.2 with
your patches, and btrfsck successfully removed the orphan file! The
contents seem to be intact in /lost+found. Thank you very much Qu,
you've been immensily helpful.

Regards,
Daniel


On Wed, Nov 26, 2014 at 1:07 AM, Daniel Miranda <danielkza2@gmail.com> wrote:
> Alright, I'll just have to understand how to build btrfs-progs now,
> since I'm currently just using the packages from the Fedora repo.
>
> Thanks for all the help and time spent so far,
> Daniel
>
>
> On Wed, Nov 26, 2014 at 12:41 AM, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:
>> Hi Daniel,
>>
>> With your btrfs-image dump, I tested with my patchset sent to maillist, my
>> patchset succeeds fixing the image.
>>
>> You can get the patchset and then apply it on 3.17.2, and --repair should
>> fix it.
>> The file with nlink error will be moved to 'lost+found' dir.
>>
>> Although the best fixing should be just adding the missing dir_index,
>> but currently the patchset does quite well and does not need to do any
>> modify.
>>
>> The patchset can be extracted using patchwork:
>> 0001: https://patchwork.kernel.org/patch/5364131/mbox/
>> 0002: https://patchwork.kernel.org/patch/5364141/mbox/
>> 0003: https://patchwork.kernel.org/patch/5364101/mbox/
>> 0004 v2: https://patchwork.kernel.org/patch/5383611/mbox/
>> 0005 v2: https://patchwork.kernel.org/patch/5383601/mbox/
>> 0006: https://patchwork.kernel.org/patch/5364151/mbox
>>
>> Any feedback is welcomed to improve the patches.
>>
>> Thanks,
>> Qu
>>
>>
>> -------- Original Message --------
>> Subject: Re: Apparent metadata corruption (file that simultaneously
>> does/does not exist) on kernel 3.17.3
>> From: Daniel Miranda <danielkza2@gmail.com>
>> To: Qu Wenruo <quwenruo@cn.fujitsu.com>
>> Date: 2014年11月25日 15:42
>>>
>>> I just ran the repair but the ghost file has not disappeared,
>>> unfortunately.
>>>
>>> On Tue, Nov 25, 2014 at 5:26 AM, Qu Wenruo <quwenruo@cn.fujitsu.com>
>>> wrote:
>>>>
>>>> -------- Original Message --------
>>>> Subject: Re: Apparent metadata corruption (file that simultaneously
>>>> does/does not exist) on kernel 3.17.3
>>>> From: Daniel Miranda <danielkza2@gmail.com>
>>>> To: Qu Wenruo <quwenruo@cn.fujitsu.com>
>>>> Date: 2014年11月25日 15:20
>>>>>
>>>>> Here are the logs. I'll send you a link to my dump directly after I
>>>>> finish uploading it. Please notify me when you have downloaded it so I
>>>>> can delete it.
>>>>>
>>>>> checking extents
>>>>> checking free space cache
>>>>> checking fs roots
>>>>> root 5 inode 17149868 errors 2000, link count wrong
>>>>>       unresolved ref dir 17182377 index 245 namelen 8 name string.h
>>>>> filetype 1 errors 1, no dir item
>>>>
>>>> link count error seems resolved by Josef's patch commit already in
>>>> 3.17.2.
>>>> If using 3.17.2, josef's commit will rebuild the dir item and dir index.
>>>>>
>>>>> root 5 inode 17182377 errors 200, dir isize wrong
>>>>
>>>> This isize error seems caused by previous line.
>>>> If 3.17.2 can repair above problem, it should not be a problem and will
>>>> disappear.
>>>>
>>>> According to the above output, btrfsck --repair with btrfs-progs 3.17.2
>>>> has
>>>> a good chance repairing it.
>>>> Just have a try.
>>>>
>>>> Thanks,
>>>> Qu
>>>>
>>>>> Checking filesystem on /dev/mapper/fedora_daniel--pc-root
>>>>> UUID: fef8f718-0622-4cb1-9597-749650d366a4
>>>>> found 55108022156 bytes used err is 1
>>>>> total csum bytes: 89787396
>>>>> total tree bytes: 2303455232
>>>>> total fs tree bytes: 2024841216
>>>>> total extent tree bytes: 145272832
>>>>> btree space waste bytes: 529672422
>>>>> file data blocks allocated: 253414481920
>>>>>    referenced 94127726592
>>>>> Btrfs v3.17
>>>>>
>>>>>
>>>>> Regards,
>>>>> Daniel
>>>>>
>>>>> On Tue, Nov 25, 2014 at 3:20 AM, Qu Wenruo <quwenruo@cn.fujitsu.com>
>>>>> wrote:
>>>>>>
>>>>>> -------- Original Message --------
>>>>>> Subject: Re: Apparent metadata corruption (file that simultaneously
>>>>>> does/does not exist) on kernel 3.17.3
>>>>>> From: Daniel Miranda <danielkza2@gmail.com>
>>>>>> To: Qu Wenruo <quwenruo@cn.fujitsu.com>
>>>>>> Date: 2014年11月25日 13:14
>>>>>>>
>>>>>>> I'll go run that and get you the output.
>>>>>>
>>>>>> Thanks.
>>>>>>>
>>>>>>>
>>>>>>> I can do the image dump, sure. I don't know how long it might take to
>>>>>>> upload it somewhere though. Right now `btrfs fi df` shows about 2GiB
>>>>>>> of metadata (it's a 120GiB volume). I'll see how large it ends up
>>>>>>> after compression.
>>>>>>
>>>>>> 120G volume seems quite small, compared the images I received recently
>>>>>> (1T
>>>>>> x2 RAID1 and 4T single).
>>>>>> With '-c 9' it shouldn't be too huge I think(The 1T raid1 is about 1G
>>>>>> metadata with -c9).
>>>>>>
>>>>>> BTW, btrfs-image dump will have all the filenames and hierarchy, even
>>>>>> without its data,
>>>>>> it is still better considering your privacy twice before uploading.
>>>>>>
>>>>>> Thanks,
>>>>>> Qu
>>>>>>
>>>>>>> Thanks for the quick response,
>>>>>>> Daniel
>>>>>>>
>>>>>>> On Tue, Nov 25, 2014 at 3:10 AM, Qu Wenruo <quwenruo@cn.fujitsu.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> What's the btrfsck output? Without --repair option.
>>>>>>>>
>>>>>>>> Also, if it is OK for you, would you please dump the btrfs with
>>>>>>>> 'btrfs-image' command?
>>>>>>>> '-c 9' option is highly recommended considering the size of it.
>>>>>>>> This will helps a lot for developers to test the btrfsck repair
>>>>>>>> function.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Qu
>>>>>>>>
>>>>>>>>
>>>>>>>> -------- Original Message --------
>>>>>>>> Subject: Apparent metadata corruption (file that simultaneously
>>>>>>>> does/does
>>>>>>>> not exist) on kernel 3.17.3
>>>>>>>> From: Daniel Miranda <danielkza2@gmail.com>
>>>>>>>> To: <linux-btrfs@vger.kernel.org>
>>>>>>>> Date: 2014年11月25日 13:04
>>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> After I had some brief stability issues with my computer, it seems
>>>>>>>>> some form of metadata corruption took place in my BTRFS filesystem,
>>>>>>>>> and now a particular file seems to exist, but I cannot access any
>>>>>>>>> details on it or delete it.
>>>>>>>>>
>>>>>>>>> If I try to `ls` in the directory it is in, that's what I get:
>>>>>>>>>
>>>>>>>>> ls: cannot access string.h: No such file or directory
>>>>>>>>> total 0
>>>>>>>>> drwxr-xr-x. 1 danielkza mock 16 Nov 21 14:18 ./
>>>>>>>>> drwxr-xr-x. 1 danielkza mock  6 Nov 21 14:18 ../
>>>>>>>>> -?????????? ? ?         ?     ?            ? string.h
>>>>>>>>>
>>>>>>>>> If I try to delete it I get:
>>>>>>>>>
>>>>>>>>> rm: cannot remove ‘string.h’: No such file or directory
>>>>>>>>>
>>>>>>>>> I'm using kernel 3.17.3 from Fedora 21. I got no messages on dmesg
>>>>>>>>> or
>>>>>>>>> anything of the sort. I know the btrfs fsck situation is
>>>>>>>>> complicated,
>>>>>>>>> but is there any utility I should use to try and repair this? Losing
>>>>>>>>> this file is not a problem, it's just one header from the kernel I
>>>>>>>>> was
>>>>>>>>> building.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Daniel Miranda
>>>>>>>>> --
>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>>>> linux-btrfs"
>>>>>>>>> in
>>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>>>>
>>>>>>>>
>>