machine gets unresponsive during btrfs balance

machine gets unresponsive during btrfs balance

[Andreas Philipp]

Hi,

I am using a btrfs filesystem created with raid0 for data and metadata
for (temporary) storage of tv recordings from my vdr. The filesystem was
created under kernel version 2.6.34. An initial btrfs balance command
succeeded. Since I upgraded to 2.6.35-rcX and 2.6.35 btrfs balance no
longer finishes but puts the machine in some unresponsive state.
Unfortunately, I do not see any kernel oops or other debug information
because even the display freezes. The last thing that happens are that
those two lines are written to /var/log/messages:
Aug 11 21:42:23 thor kernel: btrfs: found 62911 extents
Aug 11 21:42:24 thor kernel: btrfs: relocating block group 1723913469952
flags 9
After that the machine becomes immediately unresponsive.

As I did not see anything that might be related to my problem in the
changelog for 2.6.35.1 I did not try again with this version.

Thanks,
Andreas

Re: machine gets unresponsive during btrfs balance

[Yan, Zheng]

On Thu, Aug 12, 2010 at 3:14 PM, Andreas Philipp
<philipp.andreas@gmail.com> wrote:
> Hi,
>
> I am using a btrfs filesystem created with raid0 for data and metadata
> for (temporary) storage of tv recordings from my vdr. The filesystem was
> created under kernel version 2.6.34. An initial btrfs balance command
> succeeded. Since I upgraded to 2.6.35-rcX and 2.6.35 btrfs balance no
> longer finishes but puts the machine in some unresponsive state.
> Unfortunately, I do not see any kernel oops or other debug information
> because even the display freezes. The last thing that happens are that
> those two lines are written to /var/log/messages:
> Aug 11 21:42:23 thor kernel: btrfs: found 62911 extents
> Aug 11 21:42:24 thor kernel: btrfs: relocating block group 1723913469952
> flags 9
> After that the machine becomes immediately unresponsive.
>
> As I did not see anything that might be related to my problem in the
> changelog for 2.6.35.1 I did not try again with this version.
>

Do you have more than one machines? would you please setup netconsole
to see what happen.

Thanks
Yan, Zheng

Re: machine gets unresponsive during btrfs balance

[Andreas Philipp]

On 12.08.2010 10:04, Yan, Zheng wrote:
> On Thu, Aug 12, 2010 at 3:14 PM, Andreas Philipp
> <philipp.andreas@gmail.com> wrote:
>   
>> Hi,
>>
>> I am using a btrfs filesystem created with raid0 for data and metadata
>> for (temporary) storage of tv recordings from my vdr. The filesystem was
>> created under kernel version 2.6.34. An initial btrfs balance command
>> succeeded. Since I upgraded to 2.6.35-rcX and 2.6.35 btrfs balance no
>> longer finishes but puts the machine in some unresponsive state.
>> Unfortunately, I do not see any kernel oops or other debug information
>> because even the display freezes. The last thing that happens are that
>> those two lines are written to /var/log/messages:
>> Aug 11 21:42:23 thor kernel: btrfs: found 62911 extents
>> Aug 11 21:42:24 thor kernel: btrfs: relocating block group 1723913469952
>> flags 9
>> After that the machine becomes immediately unresponsive.
>>
>> As I did not see anything that might be related to my problem in the
>> changelog for 2.6.35.1 I did not try again with this version.
>>
>>     
> Do you have more than one machines? would you please setup netconsole
> to see what happen.
>   
Sorry for not responding earlier but I did not find time before this
morning. After I have set up netconsole I am running btrfs balance at
the moment - now on 2.6.35.1.

Yours,
Andreas

Re: machine gets unresponsive during btrfs balance

[Andreas Philipp]

[-- Attachment #1: Type: text/plain, Size: 1435 bytes --]

On 12.08.2010 10:04, Yan, Zheng wrote:
> On Thu, Aug 12, 2010 at 3:14 PM, Andreas Philipp
> <philipp.andreas@gmail.com> wrote:
>   
>> Hi,
>>
>> I am using a btrfs filesystem created with raid0 for data and metadata
>> for (temporary) storage of tv recordings from my vdr. The filesystem was
>> created under kernel version 2.6.34. An initial btrfs balance command
>> succeeded. Since I upgraded to 2.6.35-rcX and 2.6.35 btrfs balance no
>> longer finishes but puts the machine in some unresponsive state.
>> Unfortunately, I do not see any kernel oops or other debug information
>> because even the display freezes. The last thing that happens are that
>> those two lines are written to /var/log/messages:
>> Aug 11 21:42:23 thor kernel: btrfs: found 62911 extents
>> Aug 11 21:42:24 thor kernel: btrfs: relocating block group 1723913469952
>> flags 9
>> After that the machine becomes immediately unresponsive.
>>
>> As I did not see anything that might be related to my problem in the
>> changelog for 2.6.35.1 I did not try again with this version.
>>
>>     
> Do you have more than one machines? would you please setup netconsole
> to see what happen.
>   
I have reproduced the error on v2.6.35.1 and recorded all kernel output
with netconsole. The interesting point is that this time the machine did
not crash but the btrfs balance segfaulted at exact the same position
where the previous crashes had happened.

Yours,
Andreas

[-- Attachment #2: netconsole.log.gz --]
[-- Type: application/gzip, Size: 7403 bytes --]

Re: machine gets unresponsive during btrfs balance

[Johannes Hirte]

On Saturday 14 August 2010 00:11:55 Andreas Philipp wrote:
> On 12.08.2010 10:04, Yan, Zheng wrote:
> > On Thu, Aug 12, 2010 at 3:14 PM, Andreas Philipp
> > <philipp.andreas@gmail.com> wrote:
> >   
> >> Hi,
> >>
> >> I am using a btrfs filesystem created with raid0 for data and metadata
> >> for (temporary) storage of tv recordings from my vdr. The filesystem was
> >> created under kernel version 2.6.34. An initial btrfs balance command
> >> succeeded. Since I upgraded to 2.6.35-rcX and 2.6.35 btrfs balance no
> >> longer finishes but puts the machine in some unresponsive state.
> >> Unfortunately, I do not see any kernel oops or other debug information
> >> because even the display freezes. The last thing that happens are that
> >> those two lines are written to /var/log/messages:
> >> Aug 11 21:42:23 thor kernel: btrfs: found 62911 extents
> >> Aug 11 21:42:24 thor kernel: btrfs: relocating block group 1723913469952
> >> flags 9
> >> After that the machine becomes immediately unresponsive.
> >>
> >> As I did not see anything that might be related to my problem in the
> >> changelog for 2.6.35.1 I did not try again with this version.
> >>
> >>     
> > Do you have more than one machines? would you please setup netconsole
> > to see what happen.
> >   
> I have reproduced the error on v2.6.35.1 and recorded all kernel output
> with netconsole. The interesting point is that this time the machine did
> not crash but the btrfs balance segfaulted at exact the same position
> where the previous crashes had happened.

Looks like another manifestation of the csum bug. Are you able to read all 
files from the affected volume? Did you tried a balance with an 2.6.34 kernel 
after the test with 2.6.35?

regards,
  Johannes

Re: machine gets unresponsive during btrfs balance

[Andreas Philipp]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 26.08.2010 15:27, Johannes Hirte wrote:
> On Saturday 14 August 2010 00:11:55 Andreas Philipp wrote:
>> On 12.08.2010 10:04, Yan, Zheng wrote:
>>> On Thu, Aug 12, 2010 at 3:14 PM, Andreas Philipp
>>> <philipp.andreas@gmail.com> wrote:
>>>   
>>>> Hi,
>>>>
>>>> I am using a btrfs filesystem created with raid0 for data and metadata
>>>> for (temporary) storage of tv recordings from my vdr. The filesystem was
>>>> created under kernel version 2.6.34. An initial btrfs balance command
>>>> succeeded. Since I upgraded to 2.6.35-rcX and 2.6.35 btrfs balance no
>>>> longer finishes but puts the machine in some unresponsive state.
>>>> Unfortunately, I do not see any kernel oops or other debug information
>>>> because even the display freezes. The last thing that happens are that
>>>> those two lines are written to /var/log/messages:
>>>> Aug 11 21:42:23 thor kernel: btrfs: found 62911 extents
>>>> Aug 11 21:42:24 thor kernel: btrfs: relocating block group 1723913469952
>>>> flags 9
>>>> After that the machine becomes immediately unresponsive.
>>>>
>>>> As I did not see anything that might be related to my problem in the
>>>> changelog for 2.6.35.1 I did not try again with this version.
>>>>
>>>>     
>>> Do you have more than one machines? would you please setup netconsole
>>> to see what happen.
>>>   
>> I have reproduced the error on v2.6.35.1 and recorded all kernel output
>> with netconsole. The interesting point is that this time the machine did
>> not crash but the btrfs balance segfaulted at exact the same position
>> where the previous crashes had happened.
> 
> Looks like another manifestation of the csum bug. Are you able to read all 
> files from the affected volume? Did you tried a balance with an 2.6.34 kernel 
> after the test with 2.6.35?
> 
Till now I did not see any unreadable files but I did not do a
complete test. No, I did not try to balance with an 2.6.34 kernel. If
it helps I can switch back and try.

Yours,
Andreas
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJMdm6MAAoJEJIcBJ3+Xkgi9vUQALX7V6fOs+DJR9NGRr21uY1a
/tFj5e1r71Mryn6uFcnb2Iukf6oNirxc3n4XcSgfel/GppTPAt+s8a3jS048SNa1
XpPhpuA5j9vtPns46ZR5Bg9rTtAIa7oDO8Ko2lewnHcrZN9qoGyroTz+eIzqv/U/
N8BmGTKTE1TwKfETE8jVXXHvsQuV6vYeOJqtWzPkETw9gldHThTSvr3pl7+46yFT
iCv5n/IHsdAyeZpSJm1Jp3xxAGZJsrkhoKiyNcHHt7+UshcKpFYky4lOCAlDgpPO
quZhEHiYKUeHGYb8vueaebCgy2panrYcnEgwoGkI7XLPvTlWY/5uUgV/54rlUdwi
jqo+zgyNaBnEwUkqTAPzqZQBYb7XA5uJS7UchFhf5rpgFZeEX+gUnNYtEtUkkYLk
9vAAeXl8OyNJnBAH97/FBpRw0nVYpXeuE8/dvc1TfbHDjkOQLlgEYg3T4PW+NgsV
IwVOChoFPEmyFndAvONphTmUQjzGMJu2Y+3p9D7ZDRQOo8AHjGrErWPY+zGkZkd1
MtFCIKeTyOoR+13U4xAxi+2alGf7UE0jdxoGMQPnh7COjOYpUgMt1zFnksxn6yY+
RcFEWJSoBkOPMePxzJUTOCVb/Qr4V6HCJNEylBxF1VsECAOkjTSxl4XTAXrLKYVu
pllFS2PYozhZMoKrDKQE
=ldPn
-----END PGP SIGNATURE-----

Re: machine gets unresponsive during btrfs balance

[Johannes Hirte]

On Thursday 26 August 2010 15:39:25 Andreas Philipp wrote:
> On 26.08.2010 15:27, Johannes Hirte wrote:
> > Looks like another manifestation of the csum bug. Are you able to read all
> > files from the affected volume? Did you tried a balance with an 2.6.34 kernel
> > after the test with 2.6.35?
> >
> Till now I did not see any unreadable files but I did not do a
> complete test. No, I did not try to balance with an 2.6.34 kernel. If
> it helps I can switch back and try.

I hope it helps to localize the error. It's still not clear where this starts 
an what kernels are affected.

regards,
  Johannes

Re: machine gets unresponsive during btrfs balance

[Lubos Kolouch]

Johannes Hirte, Thu, 26 Aug 2010 18:38:30 +0200:

> On Thursday 26 August 2010 15:39:25 Andreas Philipp wrote:
>> On 26.08.2010 15:27, Johannes Hirte wrote:
>> > Looks like another manifestation of the csum bug. Are you able to
>> > read all files from the affected volume? Did you tried a balance with
>> > an 2.6.34 kernel after the test with 2.6.35?
>> >
>> Till now I did not see any unreadable files but I did not do a complete
>> test. No, I did not try to balance with an 2.6.34 kernel. If it helps I
>> can switch back and try.
> 
> I hope it helps to localize the error. It's still not clear where this
> starts an what kernels are affected.
> 
> regards,
>   Johannes

I have now as well a volume that shows the csum bug and no, I cannot read 
all files. Please let me know if I can test something. Kernel 2.6.35-
gentoo-r4

Regards

Lubos

Re: machine gets unresponsive during btrfs balance

[Andreas Philipp]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
 
On 26.08.2010 18:38, Johannes Hirte wrote:
> On Thursday 26 August 2010 15:39:25 Andreas Philipp wrote:
>> On 26.08.2010 15:27, Johannes Hirte wrote:
>>> Looks like another manifestation of the csum bug. Are you able
>>> to read all files from the affected volume? Did you tried a
>>> balance with an 2.6.34 kernel after the test with 2.6.35?
>>>
>> Till now I did not see any unreadable files but I did not do a
>> complete test. No, I did not try to balance with an 2.6.34
>> kernel. If it helps I can switch back and try.
>
> I hope it helps to localize the error. It's still not clear where
> this starts an what kernels are affected.
>
Unfortunately, I cannot reboot at until next Monday. But in the
meantime, I do a "test read" of all files on the volume to see whether
something cannot be read, too.

Regards,
Andreas
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
 
iQIcBAEBAgAGBQJMemYyAAoJEJIcBJ3+XkgiEOgQALeGqTtd8SxZr+GgqLvWusMu
sdwszFOi/4azNZ848DAt8Xs47X/6c0K+I2ZfzQqXTGJFlGCviKa4JDxQeQf9Fy6A
q9aayLMOKqBMVTPSjIIR/y/GO6YT3M62jhnehT1wGO27JJWZzrL8xsXVpJxzJflI
V67pMbPT1FJLGKmFpJbB2AIPUuNNaSwaG9170rTlD5fCpwSE4mzzeHJK21ihG5p6
4YdGGW+ak6iB86RkYPZU69XlKOgzb1opHDcGE6rSGRncj6fdyNXhmEUiiM/RNzs8
9yFQ8TspGGnmiszrpqZ6K6b2ajEpqxtGPbOj87Xc6dRfo5/ByrXhpdHpd70gfTRZ
oMcqcyt/wfIMXonOKrkJi+Zqb2K2SzellNQ2rMpDzdJK48J7tNTw/wfoZ+l44zNm
vtCWQa04KBfhkIgqEf8GHpq4RwIaDJ4twW2TjWZNLFCTa7i1HDgmyY5K34pYmAXT
ToOgOq4b6WnnEBC0+5Fm5AYm8eknwUtRxSvH1MZyGAHQndw+38EoXgP5fe8JOsqL
VVHP1eaXBS06eSrhSYmj10bvmnn6rOBG7LxJLYlsFvrHdx/TgxMxcMDMBtWM5vsk
Lmlo0RizVlMjlsGINSKXSouv/y9h6Pyp5CVc4Xw9ByGGGkBX3bH8mI8TKdKey5y7
jUsZn1LtF3rYTBIgET5N
=8PvR
-----END PGP SIGNATURE-----