Re: Hang with xfs/285 on 2026-03-02 kernel

[Matthew Wilcox <willy@infradead.org>]

On Sat, Apr 04, 2026 at 10:42:59PM +1100, Dave Chinner wrote:
> On Fri, Apr 03, 2026 at 04:35:46PM +0100, Matthew Wilcox wrote:
> > This is with commit 5619b098e2fb so after 7.0-rc6
> > INFO: task fsstress:3762792 blocked on a semaphore likely last held by task fsstress:3762793
> > task:fsstress        state:D stack:0     pid:3762793 tgid:3762793 ppid:3762783 task_flags:0x440140 flags:0x00080800
> > Call Trace:
> >  <TASK>
> >  __schedule+0x560/0xfc0
> >  schedule+0x3e/0x140
> >  schedule_timeout+0x84/0x110
> >  ? __pfx_process_timeout+0x10/0x10
> >  io_schedule_timeout+0x5b/0x80
> >  xfs_buf_alloc+0x793/0x7d0
> 
> -ENOMEM.
> 
> It'll be looping here:
> 
> fallback:
>         for (;;) {
>                 bp->b_addr = __vmalloc(size, gfp_mask);
>                 if (bp->b_addr)
>                         break;
>                 if (flags & XBF_READ_AHEAD)
>                         return -ENOMEM;
>                 XFS_STATS_INC(bp->b_mount, xb_page_retries);
>                 memalloc_retry_wait(gfp_mask);
>         }
> 
> If it is looping here long enough to trigger the hang check timer,
> then the MM subsystem is not making progress reclaiming memory. This
> is probably a 16kB allocation (it's an inode cluster buffer), and
> the allocation context is NOFAIL because it is within a transaction
> (this loop pre-dates __vmalloc() supporting __GFP_NOFAIL)....

There may be something else going on.  I reproduced it again and ssh'd
into the VM.

# free
               total        used        free      shared  buff/cache   available
Mem:         3988260     1197132      240080         144     3147496     2791128
Swap:        2097148      258128     1839020

There are five instances of fsstress running.  Very slowly, but they are
accumulating seconds of CPU time:

root@deadly-kvm:~# ps -aux |grep fsstress
root     3745227  0.0  0.0   2664  1476 ?        S    06:48   0:00 ./ltp/fsstress -p 4 -d /mnt/scratch -n 2000000
root     3745236  7.5  1.6 127928 65256 ?        D    06:48  42:54 ./ltp/fsstress -p 4 -d /mnt/scratch -n 2000000
root     3745237  7.6  1.5 124644 61308 ?        D    06:48  42:55 ./ltp/fsstress -p 4 -d /mnt/scratch -n 2000000
root     3745238  7.6  1.6 130844 65584 ?        D    06:48  43:01 ./ltp/fsstress -p 4 -d /mnt/scratch -n 2000000
root     3745239  7.6  1.6 126524 66536 ?        D    06:48  42:58 ./ltp/fsstress -p 4 -d /mnt/scratch -n 2000000
root@deadly-kvm:~# ps -aux |grep fsstress
root     3745227  0.0  0.0   2664  1476 ?        S    06:48   0:00 ./ltp/fsstress -p 4 -d /mnt/scratch -n 2000000
root     3745236  5.5  1.6 133116 66708 ?        R    06:48  45:44 ./ltp/fsstress -p 4 -d /mnt/scratch -n 2000000
root     3745237  5.5  1.5 130136 62516 ?        R    06:48  45:45 ./ltp/fsstress -p 4 -d /mnt/scratch -n 2000000
root     3745238  5.5  1.6 136520 65944 ?        R    06:48  45:52 ./ltp/fsstress -p 4 -d /mnt/scratch -n 2000000
root     3745239  5.5  1.7 131988 67884 ?        R    06:48  45:50 ./ltp/fsstress -p 4 -d /mnt/scratch -n 2000000

# cat /proc/3745239/stack
[<0>] xfs_buf_lock+0x4b/0x170
[<0>] xfs_buf_find_lock+0x69/0x140
[<0>] xfs_buf_get_map+0x265/0xbd0
[<0>] xfs_buf_read_map+0x59/0x2e0
[<0>] xfs_trans_read_buf_map+0x1bb/0x560
[<0>] xfs_read_agi+0xab/0x1a0
(...)

# cat /proc/3745238/stack
[<0>] xfs_buf_alloc+0x793/0x7d0
[<0>] xfs_buf_get_map+0x651/0xbd0
[<0>] xfs_buf_readahead_map+0x3b/0x1b0
[<0>] xfs_iwalk_ichunk_ra+0xe9/0x130
[<0>] xfs_iwalk_ag+0x185/0x2d0
(...)

It doesn't _seem_ like the system is struggling for memory.

# cat /proc/meminfo
MemTotal:        3988260 kB
MemFree:          241956 kB
MemAvailable:    2781960 kB
Buffers:            5184 kB
Cached:          2503020 kB
SwapCached:         4860 kB
Active:          2062948 kB
Inactive:         713828 kB
Active(anon):      85800 kB
Inactive(anon):   182968 kB
Active(file):    1977148 kB
Inactive(file):   530860 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       2097148 kB
SwapFree:        1823052 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:        267836 kB
Mapped:            16280 kB
Shmem:               144 kB
KReclaimable:     628212 kB
Slab:             783840 kB
SReclaimable:     628212 kB
SUnreclaim:       155628 kB
KernelStack:        3536 kB
PageTables:         3680 kB
SecPageTables:         0 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     4091276 kB
Committed_AS:     560852 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       13004 kB
VmallocChunk:          0 kB
Percpu:             7360 kB
AnonHugePages:     12288 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
Balloon:               0 kB
DirectMap4k:      153396 kB
DirectMap2M:     4040704 kB
DirectMap1G:     2097152 kB

and an excerpt of zoneinfo:

Node 0, zone   Normal
  pages free     27350
        boost    18939
        min      27357
        low      29461
        high     31565
        promo    33669
        spanned  524288
        present  524288
        managed  496128
        cma      0
        protection: (0, 0, 0, 0)
      nr_free_pages 27350
      nr_free_pages_blocks 0
      nr_zone_inactive_anon 21269
      nr_zone_active_anon 9703
      nr_zone_inactive_file 62769
      nr_zone_active_file 228878
      nr_zone_unevictable 0
      nr_zone_write_pending 0
      nr_mlock     0
      nr_free_cma  0