Re: bug: corrupt filesystem, cannot delete tmp files created just before crash.

Re: bug: corrupt filesystem, cannot delete tmp files created just before crash.

[Duncan]

Sandy McArthur posted on Wed, 17 Jul 2013 10:06:05 -0400 as excerpted:

> have a btrfs filesystem that is corrupt so that I cannot remove a few
> files. Attempting to delete these temp files from before a crash leaves
> the filesystem read-only and sends a trace to the syslog. Assistance
> correcting this issue is most appreciated.
> 
> I have two disks /dev/sd{b,c}1 to make up this filesystem.
> I've updated to the latest btrfs-tools and kernel packages available.
> The crash happened with kernel 3.8.13-gentoo.
> 
> # btrfs version Btrfs v0.20-rc1
> # [uname -r] 3.10.1-gentoo

Hi fellow gentooer. =:^)  I'm just a user too so won't attempt a 
technical answer.  However...

I see you've tried the latest packages for both kernel and btrfs-tools.  
That's good as otherwise that's one of the the first suggestion you'd 
get.  However you don't mention the btrfs wiki, nor do you mention trying 
what it suggests in such cases, so I'll assume you're not familiar with 
it.  (Additionally, being a wiki and btrfs still being under development, 
it's worth checking back every couple months or so, and possibly using 
the wiki history function to see what has changed on your pages of 
interest since your last visit.)

Main page (for bookmarking):

https://btrfs.wiki.kernel.org/index.php/Main_Page

Of course right at the top there it mentions (in bold) that btrfs is 
under heavy development and to run the latest, which you're basically 
doing, altho you apparently haven't tried the latest kernel rc or the 
btrfs-tools git build (which being a gentooer, I know are available, but 
masked by default).  If the following suggestion doesn't help, you might 
try them, as fixes really are going in all the time.

Meanwhile, I recommend reading up on the documentation section of the 
wiki.  In particular, altho with the corruption it may not help in your 
case, pay attention to the no-space sections of the FAQ and Problem FAQ 
pages.  Even more in particular, when there's space problems it 
recommends attempting to clobber/truncate a file in place, the idea being 
to free up space without having to allocate additional metadata space to 
do it.  Again, with corruption it may not help, but it's worth a try.

echo > /path/to/file

Of course, even more with a development filesystem than ordinarily, you 
should have good backups, so you shouldn't need to worry too much about 
finding clobber candidates since you can recover most files from backup 
in any case.

If the echo/clobber doesn't work, try again after mounting with the 
nodatacow option.  But read the wiki for the details.

There's also mount options such as recovery (and skip-balance if you have 
an aborted balance it'd otherwise be trying to restart), that you can 
try.  Of course if you have the space to do so, it might be worth dd-ing 
the filesystem elsewhere as a backup image, in case you screw things up 
worse while experimenting.

Meanwhile, based on my interested-admin most definitely NOT kernel-dev 
technical level following of this list at least, there /have/ been recent 
no-space, extents maintenance and other cleanups in the really-latest 
code (3.11-rc1+ kernel and live-git btrfs-tools, and there may be further 
patches posted to the list that haven't actually been committed yet), 
that may well help in your case.  I'm not technically qualified to match 
backtraces against commits/patches and identify a solid match, but it's 
definitely worth a try.

Finally, as background once you're out of the tight spot, since you're 
running a multi-device filesystem, you're likely to find the discussion 
of that on the multiple devices, sysadmin guide, and use cases pages 
useful.  FWIW, here I'm running most of my btrfs filesystems in dual-
device raid1 (both data/metadata) mode, to take advantage of the 
checksumming and extra copy to lookup in case of checksum error, that 
btrfs offers, in addition to the device-loss scenario that raid1 helps 
protect against.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: bug: corrupt filesystem, cannot delete tmp files created just before crash.

[Sandy McArthur]

I was able to recover the filesystem using the btrfsck from git (Btrfs
v0.20-rc1-358-g194aa4a) .

I encourage btrfsck to output a line similar to "Errors found. Run
again with --repair to attempt repairs." when errors are found.

>From using other fsck tools, I expected repairs to be attempted unless
I specified for no changes to be attempted. This would have saved me
time and the emotional grief of my perceiving that my problems were
persisting a btrfsck.


On Wed, Jul 17, 2013 at 7:38 PM, Duncan <1i5t5.duncan@cox.net> wrote:
> Sandy McArthur posted on Wed, 17 Jul 2013 10:06:05 -0400 as excerpted:
>
>> have a btrfs filesystem that is corrupt so that I cannot remove a few
>> files. Attempting to delete these temp files from before a crash leaves
>> the filesystem read-only and sends a trace to the syslog. Assistance
>> correcting this issue is most appreciated.
>>
>> I have two disks /dev/sd{b,c}1 to make up this filesystem.
>> I've updated to the latest btrfs-tools and kernel packages available.
>> The crash happened with kernel 3.8.13-gentoo.
>>
>> # btrfs version Btrfs v0.20-rc1
>> # [uname -r] 3.10.1-gentoo
>
> Hi fellow gentooer. =:^)  I'm just a user too so won't attempt a
> technical answer.  However...
>
> I see you've tried the latest packages for both kernel and btrfs-tools.
> That's good as otherwise that's one of the the first suggestion you'd
> get.  However you don't mention the btrfs wiki, nor do you mention trying
> what it suggests in such cases, so I'll assume you're not familiar with
> it.  (Additionally, being a wiki and btrfs still being under development,
> it's worth checking back every couple months or so, and possibly using
> the wiki history function to see what has changed on your pages of
> interest since your last visit.)
>
> Main page (for bookmarking):
>
> https://btrfs.wiki.kernel.org/index.php/Main_Page
>
> Of course right at the top there it mentions (in bold) that btrfs is
> under heavy development and to run the latest, which you're basically
> doing, altho you apparently haven't tried the latest kernel rc or the
> btrfs-tools git build (which being a gentooer, I know are available, but
> masked by default).  If the following suggestion doesn't help, you might
> try them, as fixes really are going in all the time.
>
> Meanwhile, I recommend reading up on the documentation section of the
> wiki.  In particular, altho with the corruption it may not help in your
> case, pay attention to the no-space sections of the FAQ and Problem FAQ
> pages.  Even more in particular, when there's space problems it
> recommends attempting to clobber/truncate a file in place, the idea being
> to free up space without having to allocate additional metadata space to
> do it.  Again, with corruption it may not help, but it's worth a try.
>
> echo > /path/to/file
>
> Of course, even more with a development filesystem than ordinarily, you
> should have good backups, so you shouldn't need to worry too much about
> finding clobber candidates since you can recover most files from backup
> in any case.
>
> If the echo/clobber doesn't work, try again after mounting with the
> nodatacow option.  But read the wiki for the details.
>
> There's also mount options such as recovery (and skip-balance if you have
> an aborted balance it'd otherwise be trying to restart), that you can
> try.  Of course if you have the space to do so, it might be worth dd-ing
> the filesystem elsewhere as a backup image, in case you screw things up
> worse while experimenting.
>
> Meanwhile, based on my interested-admin most definitely NOT kernel-dev
> technical level following of this list at least, there /have/ been recent
> no-space, extents maintenance and other cleanups in the really-latest
> code (3.11-rc1+ kernel and live-git btrfs-tools, and there may be further
> patches posted to the list that haven't actually been committed yet),
> that may well help in your case.  I'm not technically qualified to match
> backtraces against commits/patches and identify a solid match, but it's
> definitely worth a try.
>
> Finally, as background once you're out of the tight spot, since you're
> running a multi-device filesystem, you're likely to find the discussion
> of that on the multiple devices, sysadmin guide, and use cases pages
> useful.  FWIW, here I'm running most of my btrfs filesystems in dual-
> device raid1 (both data/metadata) mode, to take advantage of the
> checksumming and extra copy to lookup in case of checksum error, that
> btrfs offers, in addition to the device-loss scenario that raid1 helps
> protect against.
>
> --
> Duncan - List replies preferred.   No HTML msgs.
> "Every nonfree program has a lord, a master --
> and if you use the program, he is your master."  Richard Stallman
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Sandy McArthur

"He who dares not offend cannot be honest."
- Thomas Paine

Re: bug: corrupt filesystem, cannot delete tmp files created just before crash.

[Sandy McArthur]

Should I interpret the different used amounts (902.01GB vs 902.03GB)
on my recovered RAID1 filesystem as that not all data is actually
mirrored and so I should run a balance? The devices in the filesystem
below are the same make/model drives.

# btrfs fi show
Label: 'mcmedia'  uuid: 92b3345e-2589-423c-a228-d569bf94ab58
Total devices 2 FS bytes used 905.33GB
devid    2 size 2.73TB used 902.01GB path /dev/sdc1
devid    1 size 2.73TB used 902.03GB path /dev/sdb1

Btrfs v0.20-rc1-358-g194aa4a

# btrfs fi df /mnt/media/
Data, RAID1: total=894.00GB, used=892.99GB
Data: total=12.01GB, used=11.56GB
System, RAID1: total=8.00MB, used=132.00KB
System: total=4.00MB, used=0.00
Metadata, RAID1: total=2.00GB, used=1.13GB
Metadata: total=8.00MB, used=0.00

On Thu, Jul 18, 2013 at 10:19 AM, Sandy McArthur <sandymac@gmail.com> wrote:
> I was able to recover the filesystem using the btrfsck from git (Btrfs
> v0.20-rc1-358-g194aa4a) .
>
> I encourage btrfsck to output a line similar to "Errors found. Run
> again with --repair to attempt repairs." when errors are found.
>
> From using other fsck tools, I expected repairs to be attempted unless
> I specified for no changes to be attempted. This would have saved me
> time and the emotional grief of my perceiving that my problems were
> persisting a btrfsck.
>
>
> On Wed, Jul 17, 2013 at 7:38 PM, Duncan <1i5t5.duncan@cox.net> wrote:
>> Sandy McArthur posted on Wed, 17 Jul 2013 10:06:05 -0400 as excerpted:
>>
>>> have a btrfs filesystem that is corrupt so that I cannot remove a few
>>> files. Attempting to delete these temp files from before a crash leaves
>>> the filesystem read-only and sends a trace to the syslog. Assistance
>>> correcting this issue is most appreciated.
>>>
>>> I have two disks /dev/sd{b,c}1 to make up this filesystem.
>>> I've updated to the latest btrfs-tools and kernel packages available.
>>> The crash happened with kernel 3.8.13-gentoo.
>>>
>>> # btrfs version Btrfs v0.20-rc1
>>> # [uname -r] 3.10.1-gentoo
>>
>> Hi fellow gentooer. =:^)  I'm just a user too so won't attempt a
>> technical answer.  However...
>>
>> I see you've tried the latest packages for both kernel and btrfs-tools.
>> That's good as otherwise that's one of the the first suggestion you'd
>> get.  However you don't mention the btrfs wiki, nor do you mention trying
>> what it suggests in such cases, so I'll assume you're not familiar with
>> it.  (Additionally, being a wiki and btrfs still being under development,
>> it's worth checking back every couple months or so, and possibly using
>> the wiki history function to see what has changed on your pages of
>> interest since your last visit.)
>>
>> Main page (for bookmarking):
>>
>> https://btrfs.wiki.kernel.org/index.php/Main_Page
>>
>> Of course right at the top there it mentions (in bold) that btrfs is
>> under heavy development and to run the latest, which you're basically
>> doing, altho you apparently haven't tried the latest kernel rc or the
>> btrfs-tools git build (which being a gentooer, I know are available, but
>> masked by default).  If the following suggestion doesn't help, you might
>> try them, as fixes really are going in all the time.
>>
>> Meanwhile, I recommend reading up on the documentation section of the
>> wiki.  In particular, altho with the corruption it may not help in your
>> case, pay attention to the no-space sections of the FAQ and Problem FAQ
>> pages.  Even more in particular, when there's space problems it
>> recommends attempting to clobber/truncate a file in place, the idea being
>> to free up space without having to allocate additional metadata space to
>> do it.  Again, with corruption it may not help, but it's worth a try.
>>
>> echo > /path/to/file
>>
>> Of course, even more with a development filesystem than ordinarily, you
>> should have good backups, so you shouldn't need to worry too much about
>> finding clobber candidates since you can recover most files from backup
>> in any case.
>>
>> If the echo/clobber doesn't work, try again after mounting with the
>> nodatacow option.  But read the wiki for the details.
>>
>> There's also mount options such as recovery (and skip-balance if you have
>> an aborted balance it'd otherwise be trying to restart), that you can
>> try.  Of course if you have the space to do so, it might be worth dd-ing
>> the filesystem elsewhere as a backup image, in case you screw things up
>> worse while experimenting.
>>
>> Meanwhile, based on my interested-admin most definitely NOT kernel-dev
>> technical level following of this list at least, there /have/ been recent
>> no-space, extents maintenance and other cleanups in the really-latest
>> code (3.11-rc1+ kernel and live-git btrfs-tools, and there may be further
>> patches posted to the list that haven't actually been committed yet),
>> that may well help in your case.  I'm not technically qualified to match
>> backtraces against commits/patches and identify a solid match, but it's
>> definitely worth a try.
>>
>> Finally, as background once you're out of the tight spot, since you're
>> running a multi-device filesystem, you're likely to find the discussion
>> of that on the multiple devices, sysadmin guide, and use cases pages
>> useful.  FWIW, here I'm running most of my btrfs filesystems in dual-
>> device raid1 (both data/metadata) mode, to take advantage of the
>> checksumming and extra copy to lookup in case of checksum error, that
>> btrfs offers, in addition to the device-loss scenario that raid1 helps
>> protect against.
>>
>> --
>> Duncan - List replies preferred.   No HTML msgs.
>> "Every nonfree program has a lord, a master --
>> and if you use the program, he is your master."  Richard Stallman
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>
> --
> Sandy McArthur
>
> "He who dares not offend cannot be honest."
> - Thomas Paine



-- 
Sandy McArthur

"He who dares not offend cannot be honest."
- Thomas Paine

Re: bug: corrupt filesystem, cannot delete tmp files created just before crash.

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 6265 bytes --]

On Thu, Jul 18, 2013 at 11:11:03AM -0400, Sandy McArthur wrote:
> Should I interpret the different used amounts (902.01GB vs 902.03GB)
> on my recovered RAID1 filesystem as that not all data is actually
> mirrored and so I should run a balance? The devices in the filesystem
> below are the same make/model drives.
> 
> # btrfs fi show
> Label: 'mcmedia'  uuid: 92b3345e-2589-423c-a228-d569bf94ab58
> Total devices 2 FS bytes used 905.33GB
> devid    2 size 2.73TB used 902.01GB path /dev/sdc1
> devid    1 size 2.73TB used 902.03GB path /dev/sdb1
> 
> Btrfs v0.20-rc1-358-g194aa4a
> 
> # btrfs fi df /mnt/media/
> Data, RAID1: total=894.00GB, used=892.99GB
> Data: total=12.01GB, used=11.56GB

              ^^^^^^^ This is unmirrored data.

# btrfs balance start -dconvert=raid1,soft /mountpoint

is the incantation you need: convert your data to RAID-1, and ignore
anything which has already been converted.

   Hugo.

> System, RAID1: total=8.00MB, used=132.00KB
> System: total=4.00MB, used=0.00
> Metadata, RAID1: total=2.00GB, used=1.13GB
> Metadata: total=8.00MB, used=0.00
> 
> On Thu, Jul 18, 2013 at 10:19 AM, Sandy McArthur <sandymac@gmail.com> wrote:
> > I was able to recover the filesystem using the btrfsck from git (Btrfs
> > v0.20-rc1-358-g194aa4a) .
> >
> > I encourage btrfsck to output a line similar to "Errors found. Run
> > again with --repair to attempt repairs." when errors are found.
> >
> > From using other fsck tools, I expected repairs to be attempted unless
> > I specified for no changes to be attempted. This would have saved me
> > time and the emotional grief of my perceiving that my problems were
> > persisting a btrfsck.
> >
> >
> > On Wed, Jul 17, 2013 at 7:38 PM, Duncan <1i5t5.duncan@cox.net> wrote:
> >> Sandy McArthur posted on Wed, 17 Jul 2013 10:06:05 -0400 as excerpted:
> >>
> >>> have a btrfs filesystem that is corrupt so that I cannot remove a few
> >>> files. Attempting to delete these temp files from before a crash leaves
> >>> the filesystem read-only and sends a trace to the syslog. Assistance
> >>> correcting this issue is most appreciated.
> >>>
> >>> I have two disks /dev/sd{b,c}1 to make up this filesystem.
> >>> I've updated to the latest btrfs-tools and kernel packages available.
> >>> The crash happened with kernel 3.8.13-gentoo.
> >>>
> >>> # btrfs version Btrfs v0.20-rc1
> >>> # [uname -r] 3.10.1-gentoo
> >>
> >> Hi fellow gentooer. =:^)  I'm just a user too so won't attempt a
> >> technical answer.  However...
> >>
> >> I see you've tried the latest packages for both kernel and btrfs-tools.
> >> That's good as otherwise that's one of the the first suggestion you'd
> >> get.  However you don't mention the btrfs wiki, nor do you mention trying
> >> what it suggests in such cases, so I'll assume you're not familiar with
> >> it.  (Additionally, being a wiki and btrfs still being under development,
> >> it's worth checking back every couple months or so, and possibly using
> >> the wiki history function to see what has changed on your pages of
> >> interest since your last visit.)
> >>
> >> Main page (for bookmarking):
> >>
> >> https://btrfs.wiki.kernel.org/index.php/Main_Page
> >>
> >> Of course right at the top there it mentions (in bold) that btrfs is
> >> under heavy development and to run the latest, which you're basically
> >> doing, altho you apparently haven't tried the latest kernel rc or the
> >> btrfs-tools git build (which being a gentooer, I know are available, but
> >> masked by default).  If the following suggestion doesn't help, you might
> >> try them, as fixes really are going in all the time.
> >>
> >> Meanwhile, I recommend reading up on the documentation section of the
> >> wiki.  In particular, altho with the corruption it may not help in your
> >> case, pay attention to the no-space sections of the FAQ and Problem FAQ
> >> pages.  Even more in particular, when there's space problems it
> >> recommends attempting to clobber/truncate a file in place, the idea being
> >> to free up space without having to allocate additional metadata space to
> >> do it.  Again, with corruption it may not help, but it's worth a try.
> >>
> >> echo > /path/to/file
> >>
> >> Of course, even more with a development filesystem than ordinarily, you
> >> should have good backups, so you shouldn't need to worry too much about
> >> finding clobber candidates since you can recover most files from backup
> >> in any case.
> >>
> >> If the echo/clobber doesn't work, try again after mounting with the
> >> nodatacow option.  But read the wiki for the details.
> >>
> >> There's also mount options such as recovery (and skip-balance if you have
> >> an aborted balance it'd otherwise be trying to restart), that you can
> >> try.  Of course if you have the space to do so, it might be worth dd-ing
> >> the filesystem elsewhere as a backup image, in case you screw things up
> >> worse while experimenting.
> >>
> >> Meanwhile, based on my interested-admin most definitely NOT kernel-dev
> >> technical level following of this list at least, there /have/ been recent
> >> no-space, extents maintenance and other cleanups in the really-latest
> >> code (3.11-rc1+ kernel and live-git btrfs-tools, and there may be further
> >> patches posted to the list that haven't actually been committed yet),
> >> that may well help in your case.  I'm not technically qualified to match
> >> backtraces against commits/patches and identify a solid match, but it's
> >> definitely worth a try.
> >>
> >> Finally, as background once you're out of the tight spot, since you're
> >> running a multi-device filesystem, you're likely to find the discussion
> >> of that on the multiple devices, sysadmin guide, and use cases pages
> >> useful.  FWIW, here I'm running most of my btrfs filesystems in dual-
> >> device raid1 (both data/metadata) mode, to take advantage of the
> >> checksumming and extra copy to lookup in case of checksum error, that
> >> btrfs offers, in addition to the device-loss scenario that raid1 helps
> >> protect against.
> >>

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
    --- Reintarnation:  Coming back from the dead as a hillbilly. ---    

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: bug: corrupt filesystem, cannot delete tmp files created just before crash.

[Josef Bacik]

On Thu, Jul 18, 2013 at 04:21:28PM +0100, Hugo Mills wrote:
> On Thu, Jul 18, 2013 at 11:11:03AM -0400, Sandy McArthur wrote:
> > Should I interpret the different used amounts (902.01GB vs 902.03GB)
> > on my recovered RAID1 filesystem as that not all data is actually
> > mirrored and so I should run a balance? The devices in the filesystem
> > below are the same make/model drives.
> > 
> > # btrfs fi show
> > Label: 'mcmedia'  uuid: 92b3345e-2589-423c-a228-d569bf94ab58
> > Total devices 2 FS bytes used 905.33GB
> > devid    2 size 2.73TB used 902.01GB path /dev/sdc1
> > devid    1 size 2.73TB used 902.03GB path /dev/sdb1
> > 
> > Btrfs v0.20-rc1-358-g194aa4a
> > 
> > # btrfs fi df /mnt/media/
> > Data, RAID1: total=894.00GB, used=892.99GB
> > Data: total=12.01GB, used=11.56GB
> 
>               ^^^^^^^ This is unmirrored data.
> 
> # btrfs balance start -dconvert=raid1,soft /mountpoint
> 
> is the incantation you need: convert your data to RAID-1, and ignore
> anything which has already been converted.
> 

Build btrfs-next and boot into that first if you can before you do this as there
is a slight bug with balance that will corrupt data if you crash.  Thanks,

Josef

bug: corrupt filesystem, cannot delete tmp files created just before crash.

[Sandy McArthur]

[-- Attachment #1: Type: text/plain, Size: 1210 bytes --]

I have a btrfs filesystem that is corrupt so that I cannot remove a
few files. Attempting to delete these temp files from before a crash
leaves the filesystem read-only and sends a trace to the syslog.
Assistance correcting this issue is most appreciated.

I have two disks /dev/sd{b,c}1 to make up this filesystem. Running
btrfsck fails with:

# btrfsck /dev/sdb1
failed to read /dev/sr0
failed to read /dev/sr0
checking extents
warning, start mismatch 1508629082112 1508601819136
btrfsck: btrfsck.c:2700: run_next_block: Assertion `!(ret)' failed.
Aborted

I've updated to the latest btrfs-tools and kernel packages available.
The crash happened with kernel 3.8.13-gentoo.

# btrfs version
Btrfs v0.20-rc1
# uname -a
Linux mcplex 3.10.1-gentoo #1 SMP Wed Jul 17 00:36:11 EDT 2013 x86_64
Intel(R) Core(TM) i7-2600S CPU @ 2.80GHz GenuineIntel GNU/Linux

Attached are the syslog lines caused by a `rm
/path/to/busted/tmp-file` from different kernel versions as they look
different enough to me to possibly be helpful.

I'm happy to provide whatever other info is needed. An image of the
filesystem isn't so practical as it's large.

--
Sandy McArthur

"He who dares not offend cannot be honest."
- Thomas Paine

[-- Attachment #2: btrfs-trace-kernel3.8.13-gentoo.txt --]
[-- Type: text/plain, Size: 5549 bytes --]

Jul 17 00:09:03 mcplex kernel: block group 1507562881024 has an wrong amount of free space
Jul 17 00:09:03 mcplex kernel: btrfs: failed to load free space cache for block group 1507562881024
Jul 17 00:09:07 mcplex kernel: ------------[ cut here ]------------
Jul 17 00:09:07 mcplex kernel: kernel BUG at fs/btrfs/extent-tree.c:5269!
Jul 17 00:09:07 mcplex kernel: invalid opcode: 0000 [#1] SMP 
Jul 17 00:09:07 mcplex kernel: Modules linked in: ipv6 coretemp kvm_intel kvm r8169 mii iTCO_wdt acpi_c
pufreq thermal mperf processor rtc_cmos snd_hda_codec_hdmi lpc_ich fan button video mfd_core i2c_i801 i
2c_core snd_hda_codec_realtek microcode pcspkr thermal_sys backlight snd_hda_intel snd_hda_codec snd_hw
dep snd_pcm snd_page_alloc snd_timer snd xts gf128mul aes_x86_64 cbc sha256_generic libiscsi scsi_trans
port_iscsi tg3 libphy ptp pps_core e1000 fuse nfs lockd sunrpc jfs multipath linear raid10 raid456 asyn
c_pq async_xor xor async_memcpy async_raid6_recov raid6_pq async_tx raid1 raid0 dm_snapshot dm_crypt dm
_mirror dm_region_hash dm_log dm_mod hid_sunplus hid_sony hid_samsung hid_pl hid_petalynx hid_monterey 
hid_microsoft hid_logitech hid_gyration hid_ezkey hid_cypress hid_chicony hid_cherry hid_belkin hid_app
le hid_a4tech sl811_hcd usbhid xhci_hcd ohci_hcd uhci_hcd usb_storage ehci_pci ehci_hcd usbcore usb_com
mon aic94xx libsas lpfc qla2xxx megaraid_sas megaraid_mbox megaraid_mm megaraid aacraid sx8 DAC960 ccis
s 3w_9xxx 3w_xxxx mptsas scsi_transport_sas mptfc scsi_transport_fc scsi_tgt mptspi mptscsih mptbase at
p870u dc395x qla1280 imm parport dmx3191d sym53c8xx gdth advansys initio BusLogic arcmsr aic7xxx aic79x
x scsi_transport_spi sg pdc_adma sata_inic162x sata_mv ata_piix ahci libahci sata_qstor sata_vsc sata_u
li sata_sis sata_sx4 sata_nv sata_via sata_svw sata_sil24 sata_sil sata_promise pata_sl82c105 pata_cs55
30 pata_cs5520 pata_via pata_jmicron pata_marvell pata_sis pata_netcell pata_sc1200 pata_pdc202xx_old p
ata_triflex pata_atiixp pata_opti pata_amd pata_ali pata_it8213 pata_pcmcia pcmcia pcmcia_core pata_ns8
7415 pata_ns87410 pata_serverworks pata_artop pata_it821x pata_optidma pata_hpt3x2n pata_hpt3x3 pata_hp
t37x pata_hpt366 pata_cmd64x pata_efar pata_rz1000 pata_sil680 pata_radisys pata_pdc2027x pata_mpiix li
bata
Jul 17 00:09:07 mcplex kernel: CPU 2 
Jul 17 00:09:07 mcplex kernel: Pid: 18174, comm: rm Not tainted 3.8.13-gentoo #1 Shuttle Inc. SH67H/FH67H
Jul 17 00:09:07 mcplex kernel: RIP: 0010:[<ffffffff812ac988>]  [<ffffffff812ac988>] __btrfs_free_extent+0x47e/0x634
Jul 17 00:09:07 mcplex kernel: RSP: 0018:ffff88040ab7bb18  EFLAGS: 00010293
Jul 17 00:09:07 mcplex kernel: RAX: 0000000000000001 RBX: ffff880409882250 RCX: ffff88040c24d000
Jul 17 00:09:07 mcplex kernel: RDX: 000000000000000d RSI: 0000000000000f96 RDI: ffff880409a7f8b0
Jul 17 00:09:07 mcplex kernel: RBP: ffff88040ab7bbe8 R08: 0000000000001000 R09: ffff88040ab7bac8
Jul 17 00:09:07 mcplex kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Jul 17 00:09:07 mcplex kernel: R13: ffff88040bcfc000 R14: 0000000000000001 R15: ffff8804098a4f18
Jul 17 00:09:07 mcplex kernel: FS:  00007f673dfdd700(0000) GS:ffff88041f280000(0000) knlGS:0000000000000000
Jul 17 00:09:07 mcplex kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 17 00:09:07 mcplex kernel: CR2: 00007fc3217fd7f0 CR3: 000000040ac54000 CR4: 00000000000407e0
Jul 17 00:09:07 mcplex kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jul 17 00:09:07 mcplex kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jul 17 00:09:07 mcplex kernel: Process rm (pid: 18174, threadinfo ffff88040ab7a000, task ffff88040e3852e0)
Jul 17 00:09:07 mcplex kernel: Stack:
Jul 17 00:09:07 mcplex kernel: 0000000000000000 0000000000000127 0000000000000b42 0000000020000000
Jul 17 00:09:07 mcplex kernel: 0000000000000000 0000000000000850 ffff88040ab7bb68 ffffffff00000035
Jul 17 00:09:07 mcplex kernel: ffff88040b4b6800 0000000000000f96 000000000e000000 0000000100000001
Jul 17 00:09:07 mcplex kernel: Call Trace:
Jul 17 00:09:07 mcplex kernel: [<ffffffff812afa8d>] run_clustered_refs+0x7fb/0x8d6
Jul 17 00:09:07 mcplex kernel: [<ffffffff812f5700>] ? btrfs_check_delayed_seq+0x2a/0x43
Jul 17 00:09:07 mcplex kernel: [<ffffffff812b2959>] btrfs_run_delayed_refs+0x105/0x28f
Jul 17 00:09:07 mcplex kernel: [<ffffffff812a0da4>] ? btrfs_put_tree_mod_seq+0xe5/0x105
Jul 17 00:09:07 mcplex kernel: [<ffffffff812c0284>] __btrfs_end_transaction+0xe2/0x2cd
Jul 17 00:09:07 mcplex kernel: [<ffffffff812c049a>] btrfs_end_transaction+0xb/0xd
Jul 17 00:09:07 mcplex kernel: [<ffffffff812c91a6>] btrfs_evict_inode+0x289/0x2a8
Jul 17 00:09:07 mcplex kernel: [<ffffffff810d1c64>] evict+0xa3/0x159
Jul 17 00:09:07 mcplex kernel: [<ffffffff810d2542>] iput+0x127/0x130
Jul 17 00:09:07 mcplex kernel: [<ffffffff810c8989>] do_unlinkat+0x12d/0x1c7
Jul 17 00:09:07 mcplex kernel: [<ffffffff810c1e7c>] ? sys_newfstatat+0x25/0x2e
Jul 17 00:09:07 mcplex kernel: [<ffffffff810ca6f6>] sys_unlinkat+0x24/0x26
Jul 17 00:09:07 mcplex kernel: [<ffffffff814f06d2>] system_call_fastpath+0x16/0x1b
Jul 17 00:09:07 mcplex kernel: Code: 00 00 48 c7 c7 74 f1 5f 81 e8 f7 f9 d7 ff 48 8b b5 78 ff ff ff 48 8b 7d 98 31 d2 31 c9 e8 1b 4c 02 00 48 63 55 20 48 39 d0 73 02 <0f> 0b 48 89 c1 48 29 d1 0f 84 92 00 00 00 48 83 7d 28 00 74 22 
Jul 17 00:09:07 mcplex kernel: RIP  [<ffffffff812ac988>] __btrfs_free_extent+0x47e/0x634
Jul 17 00:09:07 mcplex kernel: RSP <ffff88040ab7bb18>
Jul 17 00:09:07 mcplex kernel: ---[ end trace 89f67bfd3df65052 ]---

[-- Attachment #3: btrfs-trace-kernel3.10.1-gentoo.txt --]
[-- Type: text/plain, Size: 4766 bytes --]

# Logs from attempt to rm a leftover temp file after a crash.
Jul 17 09:41:53 mcplex kernel: BTRFS error (device sdc1): trying to drop 13 refs but we only have 1 for bytenr 1545143844864
Jul 17 09:41:53 mcplex kernel: 
Jul 17 09:41:53 mcplex kernel: ------------[ cut here ]------------
Jul 17 09:41:53 mcplex kernel: WARNING: at fs/btrfs/super.c:254 __btrfs_abort_transaction+0x51/0xc2()
Jul 17 09:41:53 mcplex kernel: btrfs: Transaction aborted (error -22)
Jul 17 09:41:53 mcplex kernel: Modules linked in: ipv6 coretemp kvm_intel acpi_cpufreq mperf r8169 kvm rtc_cmos thermal iTCO_wdt i2c_i801 video fan lpc_ich mfd_core processor button backlight i2c_core pcspkr snd_hda_codec_hdmi mii snd_hda_codec_realtek thermal_sys snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_page_alloc snd_timer snd microcode xts gf128mul aes_x86_64 cbc sha256_generic libiscsi scsi_transport_iscsi tg3 libphy ptp pps_core e1000 fuse nfs lockd sunrpc jfs multipath linear raid10 raid456 async_pq async_xor async_memcpy async_raid6_recov async_tx raid1 raid0 dm_snapshot dm_crypt dm_mirror dm_region_hash dm_log dm_mod hid_sunplus hid_sony hid_samsung hid_pl hid_petalynx hid_gyration sl811_hcd usbhid xhci_hcd ohci_hcd uhci_hcd usb_storage ehci_pci ehci_hcd usbcore usb_common aic94xx libsas lpfc crc_t10dif qla2xxx megaraid_sas megaraid_mbox megaraid_mm megaraid aacraid sx8 DAC960 cciss 3w_9xxx 3w_xxxx mptsas scsi_transport_sas mptfc scsi_transport_fc scsi_tgt mptspi mptscsih mptbase atp870u dc395x qla1280 imm parport dmx3191d sym53c8xx gdth advansys initio BusLogic arcmsr aic7xxx aic79xx scsi_transport_spi sg pdc_adma sata_inic162x sata_mv ata_piix ahci libahci sata_qstor sata_vsc sata_uli sata_sis sata_sx4 sata_nv sata_via sata_svw sata_sil24 sata_sil sata_promise pata_sl82c105 pata_cs5530 pata_cs5520 pata_via pata_jmicron pata_marvell pata_sis pata_netcell pata_sc1200 pata_pdc202xx_old pata_triflex pata_atiixp pata_opti pata_amd pata_ali pata_it8213 pata_pcmcia pcmcia pcmcia_core pata_ns87415 pata_ns87410 pata_serverworks pata_artop pata_it821x pata_optidma pata_hpt3x2n pata_hpt3x3 pata_hpt37x pata_hpt366 pata_cmd64x pata_efar pata_rz1000 pata_sil680 pata_radisys pata_pdc2027x pata_mpiix libata
Jul 17 09:41:53 mcplex kernel: CPU: 0 PID: 16344 Comm: rm Not tainted 3.10.1-gentoo #1
Jul 17 09:41:53 mcplex kernel: Hardware name: Shuttle Inc. SH67H/FH67H, BIOS 2.03 02/19/2013
Jul 17 09:41:53 mcplex kernel: ffffffff8163f455 ffff88040b0119d8 ffffffff81512703 ffff88040b011a18
Jul 17 09:41:53 mcplex kernel: ffffffff8102d72e ffff88040b011a48 ffff88040b011a28 00000000ffffffea
Jul 17 09:41:53 mcplex kernel: ffff88040b1a8800 ffff880409fa39f0 00000000000015be ffff88040b011a78
Jul 17 09:41:53 mcplex kernel: Call Trace:
Jul 17 09:41:53 mcplex kernel: [<ffffffff81512703>] dump_stack+0x19/0x1b
Jul 17 09:41:53 mcplex kernel: [<ffffffff8102d72e>] warn_slowpath_common+0x62/0x7b
Jul 17 09:41:53 mcplex kernel: [<ffffffff8102d7db>] warn_slowpath_fmt+0x41/0x43
Jul 17 09:41:53 mcplex kernel: [<ffffffff812aaa31>] ? btrfs_printk+0xb2/0xb4
Jul 17 09:41:53 mcplex kernel: [<ffffffff812aab84>] __btrfs_abort_transaction+0x51/0xc2
Jul 17 09:41:53 mcplex kernel: [<ffffffff812b969b>] __btrfs_free_extent+0x5fe/0x7da
Jul 17 09:41:53 mcplex kernel: [<ffffffff8150f021>] ? cache_flusharray+0x8a/0x93
Jul 17 09:41:53 mcplex kernel: [<ffffffff812bc9cc>] run_clustered_refs+0x8d6/0xa9f
Jul 17 09:41:53 mcplex kernel: [<ffffffff813045b2>] ? find_ref_head+0x92/0xbe
Jul 17 09:41:53 mcplex kernel: [<ffffffff813052ee>] ? btrfs_find_ref_cluster+0xd7/0x111
Jul 17 09:41:53 mcplex kernel: [<ffffffff812bfc3f>] btrfs_run_delayed_refs+0x22f/0x462
Jul 17 09:41:53 mcplex kernel: [<ffffffff812cdc37>] __btrfs_end_transaction+0xcc/0x283
Jul 17 09:41:53 mcplex kernel: [<ffffffff812cde19>] btrfs_end_transaction+0xb/0xd
Jul 17 09:41:53 mcplex kernel: [<ffffffff812d7193>] btrfs_evict_inode+0x284/0x2ab
Jul 17 09:41:53 mcplex kernel: [<ffffffff810d61c7>] evict+0xa3/0x159
Jul 17 09:41:53 mcplex kernel: [<ffffffff810d6add>] iput+0x127/0x130
Jul 17 09:41:53 mcplex kernel: [<ffffffff810cd236>] do_unlinkat+0x12d/0x1c7
Jul 17 09:41:53 mcplex kernel: [<ffffffff810c6666>] ? SyS_newfstatat+0x25/0x2e
Jul 17 09:41:53 mcplex kernel: [<ffffffff810cefb8>] SyS_unlinkat+0x24/0x26
Jul 17 09:41:53 mcplex kernel: [<ffffffff81516c52>] system_call_fastpath+0x16/0x1b
Jul 17 09:41:53 mcplex kernel: ---[ end trace 7a2cc0cecc3baaaa ]---
Jul 17 09:41:53 mcplex kernel: BTRFS error (device sdc1) in __btrfs_free_extent:5566: errno=-22 unknown
Jul 17 09:41:53 mcplex kernel: BTRFS info (device sdc1): forced readonly
Jul 17 09:41:53 mcplex kernel: BTRFS debug (device sdc1): run_one_delayed_ref returned -22
Jul 17 09:41:53 mcplex kernel: BTRFS error (device sdc1) in btrfs_run_delayed_refs:2630: errno=-22 unknown