raid5-cache: Cannot get enough stripes due to memory pressure. Recovery failed.

raid5-cache: Cannot get enough stripes due to memory pressure. Recovery failed.

[Larkin Lowrey]

I have a 12 disk raid6 array that I can no longer assemble. When I 
booted I got dumped into the dracut recovery shell and manually 
assembling the array yielded...

# mdadm --assemble /dev/md2 /dev/dm-3 /dev/sd[e-lu-x]1

md/raid:md2: device sde1 operational as raid disk 0
md/raid:md2: device sdx1 operational as raid disk 11
md/raid:md2: device sdw1 operational as raid disk 10
md/raid:md2: device sdv1 operational as raid disk 9
md/raid:md2: device sdu1 operational as raid disk 8
md/raid:md2: device sdl1 operational as raid disk 7
md/raid:md2: device sdk1 operational as raid disk 6
md/raid:md2: device sdj1 operational as raid disk 5
md/raid:md2: device sdi1 operational as raid disk 4
md/raid:md2: device sdh1 operational as raid disk 3
  md/raid:md2: device sdg1 operational as raid disk 2
md/raid:md2: device sdf1 operational as raid disk 1
md/raid:md2: raid level 6 active with 12 out of 12 devices, algorithm 2
md/raid:md2: Cannot get enough stripes due to memory pressure. Recovery 
failed.
md/raid:md2: failed to run raid set.
md: pers->run() failed ...
md: md2 stopped.
RUN_ARRAY /dev/md2: Input/output error

The journal is on LVM volume /dev/dm-3 and is 2GB. It's sitting on a 
raid10 of 4 SSDs.

The system has 16GB of RAM and 8GB of swap. Memory looks good:

# free
               total        used        free      shared buff/cache   
available
Mem:       16327416      242296    15930344        1216 154776    15821820
Swap:       8388604           0     8388604

The prior shutdown could have been unclean for the array. The shutdown 
process got stuck and I eventually had to use the reset button.

I have a second (8 drive) raid6 array on this host that has a 2GB 
journal and it won't assemble either.

Kernel: 4.13.10-100.fc25.x86_64

What can I do? Do I need more RAM?

--Larkin

Re: raid5-cache: Cannot get enough stripes due to memory pressure. Recovery failed.

[Wols Lists]

On 04/11/17 16:13, Larkin Lowrey wrote:
> The prior shutdown could have been unclean for the array. The shutdown
> process got stuck and I eventually had to use the reset button.
> 
> I have a second (8 drive) raid6 array on this host that has a 2GB
> journal and it won't assemble either.
> 
> Kernel: 4.13.10-100.fc25.x86_64
> 
> What can I do? Do I need more RAM?

You're in the recovery shell ... does/can that use all available ram?
(Like DOS needed special drivers to use ram above 1MB). So I doubt you
need more ram.

Get a rescue disk of some sort, boot into that, so you've got a full
working linux off DVD/CD, and see if you can assemble the arrays from there.

Cheers,
Wol

Re: raid5-cache: Cannot get enough stripes due to memory pressure. Recovery failed.

[Larkin Lowrey]

The rescue shell had the full kernel loaded and all 16GB was available 
so that's not the issue.

I booted off a USB drive and was able to get both arrays to assemble. 
The kernel was much older, 4.11.8-300.fc26.x86_64.

After a clean shutdown and reboot to a 4.13.X kernel I no longer get 
dumped into the recovery shell but the boot process hangs. The 8 drive 
array gets assembled but the hang happens immediately after all of the 
drives for the 12 drive array are detected so I suspect the hang happens 
during the assembly of the 12 drive array.

When I go back to boot off the USB drive (4.11.8 kernel) both assemble 
just fine. Could there be regression between 4.11.8 and 4.13.X?

I no longer have full boot images with older kernels than 4.13 so I 
can't easily test to identify the point of regression.

--Larkin

On 11/4/2017 1:54 PM, Wols Lists wrote:
> On 04/11/17 16:13, Larkin Lowrey wrote:
>> The prior shutdown could have been unclean for the array. The shutdown
>> process got stuck and I eventually had to use the reset button.
>>
>> I have a second (8 drive) raid6 array on this host that has a 2GB
>> journal and it won't assemble either.
>>
>> Kernel: 4.13.10-100.fc25.x86_64
>>
>> What can I do? Do I need more RAM?
> You're in the recovery shell ... does/can that use all available ram?
> (Like DOS needed special drivers to use ram above 1MB). So I doubt you
> need more ram.
>
> Get a rescue disk of some sort, boot into that, so you've got a full
> working linux off DVD/CD, and see if you can assemble the arrays from there.
>
> Cheers,
> Wol
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html