fs was hung and sort of full

fs was hung and sort of full

[Rich Rauenzahn]

I have a mount point that became ... hung for lack of a better word.

I was doing a LARGE sort of a file in it, using temporary files in
that directory, and the
system hung and probably the watchdog kicked in, reset the system.

Upon reboot, the filesystem hung if you touched it.  Processes were
unkillable if they were using it.

Kernel logs showed nothing ... nothing in dmesg.

$ btrfs filesystem df /.MEDIA/
Data, RAID1: total=1.51TiB, used=1.51TiB
System, RAID1: total=32.00MiB, used=304.00KiB
Metadata, RAID1: total=4.00GiB, used=2.17GiB
GlobalReserve, single: total=512.00MiB, used=2.23MiB

.....is it out of space?  Data might be?  It's 2 x 2TB drives in RAID1,
and I don't think I'm quite up to that above.   I'm using a single
partition:

Disk /dev/sde: 3907029168 sectors, 1.8 TiB
Logical sector size: 512 bytes
Disk identifier (GUID): 5214ED9D-769A-4DF8-886F-8EEC3FDD4D0D
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 3907029134
Partitions will be aligned on 8-sector boundaries
Total free space is 6 sectors (3.0 KiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1              40      3907029134   1.8 TiB     8300  BTRFS MEDIA

I'm not using any advanced features like snapshots or subvolumes here.

I did try a btrfs balance, but it seemed to suffer the same hanging fate...

btfs check was fine:

$ sudo btrfs check /dev/sde1
Checking filesystem on /dev/sde1
UUID: 6a69975b-20f9-408f-9120-c457d23d0e55
checking extents
checking free space cache
checking fs roots
checking csums
checking root refs
found 1661050089472 bytes used err is 0
total csum bytes: 1612603964
total tree bytes: 2325479424
total fs tree bytes: 280821760
total extent tree bytes: 182353920
btree space waste bytes: 317572754
file data blocks allocated: 2495485378560
 referenced 1674323156992

$ sudo btrfs fi show /dev/sde1
Label: 'MEDIA'  uuid: 6a69975b-20f9-408f-9120-c457d23d0e55
        Total devices 2 FS bytes used 1.51TiB
        devid    1 size 1.82TiB used 1.51TiB path /dev/sde1
        devid    2 size 1.82TiB used 1.51TiB path /dev/sdd1

$ sudo mount -oro /.MEDIA

worked... and could interact with it

$ sudo mount -orw,remount /.MEDIA

worked ... and I removed the temporary files, and it seem to be ok now.

But why did it hang before?

Re: fs was hung and sort of full

[Chris Murphy]

If there's no hung task listed in dmesg you could try to do sysrq+t to
find out what everything's up to, although then you have to learn how
to parse the result.


Chris Murphy

Re: fs was hung and sort of full

[Rich Rauenzahn]

It is still hanging afterall.  Currently, it hangs on mount:

[154422.778624] mount           D    0 27894  27742 0x00000084
[154422.778625] Call Trace:
[154422.779018]  __schedule+0x28d/0x880
[154422.779494]  schedule+0x36/0x80
[154422.779886]  io_schedule+0x16/0x40
[154422.780288]  do_read_cache_page+0x3da/0x5c0
[154422.780669]  ? blkdev_writepages+0x10/0x10
[154422.781052]  ? mntput+0x24/0x40
[154422.781457]  ? page_cache_tree_insert+0xd0/0xd0
[154422.781849]  read_cache_page_gfp+0x1b/0x20
[154422.782272]  btrfs_scan_one_device+0x71/0x260 [btrfs]
[154422.782658]  ? __free_pages+0x25/0x30
[154422.783052]  ? free_pages.part.94+0x40/0x50
[154422.783521]  ? free_pages+0x13/0x20
[154422.783940]  btrfs_mount+0x2cd/0xfa0 [btrfs]
[154422.784353]  ? find_next_bit+0xb/0x10
[154422.784736]  ? pcpu_next_unpop+0x3c/0x50
[154422.785124]  ? find_next_bit+0xb/0x10
[154422.785565]  mount_fs+0x39/0x150
[154422.785949]  ? __alloc_percpu+0x15/0x20
[154422.786349]  vfs_kern_mount+0x67/0x130
[154422.786732]  btrfs_mount+0x19d/0xfa0 [btrfs]
[154422.787120]  ? find_next_bit+0xb/0x10
[154422.787586]  ? pcpu_next_unpop+0x3c/0x50
[154422.787966]  mount_fs+0x39/0x150
[154422.788360]  ? __alloc_percpu+0x15/0x20
[154422.788732]  vfs_kern_mount+0x67/0x130
[154422.789109]  do_mount+0x1f5/0xca0
[154422.789525]  SyS_mount+0x83/0xd0
[154422.789900]  do_syscall_64+0x74/0x1b0
[154422.790313]  entry_SYSCALL_64_after_hwframe+0x21/0x86

I do have a disk 100% busy (only one of the pair ...) ... so maybe
it's "working"?

Running 4.15.4-1.el7.elrepo.x86_64


On Thu, Feb 22, 2018 at 3:52 PM, Chris Murphy <lists@colorremedies.com> wrote:
> If there's no hung task listed in dmesg you could try to do sysrq+t to
> find out what everything's up to, although then you have to learn how
> to parse the result.
>
>
> Chris Murphy