Question about forcing 'disable-memdev'

Question about forcing 'disable-memdev'

[Cao, Quanquan/曹 全全]

Hi, Dave

On the basis of this patch, I conducted some tests and encountered 
unexpected errors. I would like to inquire whether the design here is 
reasonable? Below are the steps of my testing:

Link: 
https://lore.kernel.org/linux-cxl/170138109724.2882696.123294980050048623.stgit@djiang5-mobl3/


Problem description: after creating a region, directly forcing 
'disable-memdev' and then consuming memory leads to a kernel panic.


Test environment:
KERNEL	6.8.0-rc1
QEMU	8.2.0-rc4

Test steps：
       step1: set memory auto_online to movable zones.
            echo online_movable > 
/sys/devices/system/memory/auto_online_blocks
       step2: create region
            cxl create-region -t ram -d decoder0.0 -m mem0
       step3: disable memdev
            cxl disable-memdev mem0 -f
       step4: consum CXL memory
            ./consumemem   <------kernel panic

numactl node status:
       step1: numactl -H

	available: 2 nodes (0-1)
	node 0 cpus: 0 1
	node 0 size: 968 MB
	node 0 free: 664 MB
	node 1 cpus: 2 3
	node 1 size: 683 MB
	node 1 free: 333 MB
	node distances:
	node   0   1
	  0:  10  20

     step2: numactl -H

	available: 3 nodes (0-2)
	node 0 cpus: 0 1
	node 0 size: 968 MB
	node 0 free: 677 MB
	node 1 cpus: 2 3
	node 1 size: 683 MB
	node 1 free: 333 MB
	node 2 cpus:
	node 2 size: 256 MB
	node 2 free: 256 MB
	node distances:
	node   0   1   2
	  0:  10  20  20
	  1:  20  10  20
	  2:  20  20  10

     step3: numactl -H

	available: 3 nodes (0-2)
	node 0 cpus: 0 1
	node 0 size: 968 MB
	node 0 free: 686 MB
	node 1 cpus: 2 3
	node 1 size: 683 MB
	node 1 free: 336 MB
	node 2 cpus:
	node 2 size: 256 MB
	node 2 free: 256 MB
	node distances:
	node   0   1   2
	  0:  10  20  20
	  1:  20  10  20
	  2:  20  20  10

Re: Question about forcing 'disable-memdev'

[Dave Jiang]

On 2/26/24 10:32 PM, Cao, Quanquan/曹 全全 wrote:
> Hi, Dave
> 
> On the basis of this patch, I conducted some tests and encountered unexpected errors. I would like to inquire whether the design here is reasonable? Below are the steps of my testing:
> 
> Link: https://lore.kernel.org/linux-cxl/170138109724.2882696.123294980050048623.stgit@djiang5-mobl3/
> 
> 
> Problem description: after creating a region, directly forcing 'disable-memdev' and then consuming memory leads to a kernel panic.

If you are forcing memory disable when the memory cannot be offlined, then this behavior is expected. You are ripping the memory away from underneath kernel mm. The reason the check was added is to prevent the users from doing exactly that.


> 
> 
> Test environment:
> KERNEL    6.8.0-rc1
> QEMU    8.2.0-rc4
> 
> Test steps：
>       step1: set memory auto_online to movable zones.
>            echo online_movable > /sys/devices/system/memory/auto_online_blocks
>       step2: create region
>            cxl create-region -t ram -d decoder0.0 -m mem0
>       step3: disable memdev
>            cxl disable-memdev mem0 -f
>       step4: consum CXL memory
>            ./consumemem   <------kernel panic
> 
> numactl node status:
>       step1: numactl -H
> 
>     available: 2 nodes (0-1)
>     node 0 cpus: 0 1
>     node 0 size: 968 MB
>     node 0 free: 664 MB
>     node 1 cpus: 2 3
>     node 1 size: 683 MB
>     node 1 free: 333 MB
>     node distances:
>     node   0   1
>       0:  10  20
> 
>     step2: numactl -H
> 
>     available: 3 nodes (0-2)
>     node 0 cpus: 0 1
>     node 0 size: 968 MB
>     node 0 free: 677 MB
>     node 1 cpus: 2 3
>     node 1 size: 683 MB
>     node 1 free: 333 MB
>     node 2 cpus:
>     node 2 size: 256 MB
>     node 2 free: 256 MB
>     node distances:
>     node   0   1   2
>       0:  10  20  20
>       1:  20  10  20
>       2:  20  20  10
> 
>     step3: numactl -H
> 
>     available: 3 nodes (0-2)
>     node 0 cpus: 0 1
>     node 0 size: 968 MB
>     node 0 free: 686 MB
>     node 1 cpus: 2 3
>     node 1 size: 683 MB
>     node 1 free: 336 MB
>     node 2 cpus:
>     node 2 size: 256 MB
>     node 2 free: 256 MB
>     node distances:
>     node   0   1   2
>       0:  10  20  20
>       1:  20  10  20
>       2:  20  20  10

Re: Question about forcing 'disable-memdev'

[Jane Chu]

On 2/27/2024 8:40 AM, Dave Jiang wrote:

>
> On 2/26/24 10:32 PM, Cao, Quanquan/曹 全全 wrote:
>> Hi, Dave
>>
>> On the basis of this patch, I conducted some tests and encountered unexpected errors. I would like to inquire whether the design here is reasonable? Below are the steps of my testing:
>>
>> Link: https://lore.kernel.org/linux-cxl/170138109724.2882696.123294980050048623.stgit@djiang5-mobl3/
>>
>>
>> Problem description: after creating a region, directly forcing 'disable-memdev' and then consuming memory leads to a kernel panic.
> If you are forcing memory disable when the memory cannot be offlined, then this behavior is expected. You are ripping the memory away from underneath kernel mm. The reason the check was added is to prevent the users from doing exactly that.

Since user is doing the illegal thing, shouldn't that lead to SIGBUS or 
SIGKILL ?

thanks,

-jane

>
>>
>> Test environment:
>> KERNEL    6.8.0-rc1
>> QEMU    8.2.0-rc4
>>
>> Test steps：
>>        step1: set memory auto_online to movable zones.
>>             echo online_movable > /sys/devices/system/memory/auto_online_blocks
>>        step2: create region
>>             cxl create-region -t ram -d decoder0.0 -m mem0
>>        step3: disable memdev
>>             cxl disable-memdev mem0 -f
>>        step4: consum CXL memory
>>             ./consumemem   <------kernel panic
>>
>> numactl node status:
>>        step1: numactl -H
>>
>>      available: 2 nodes (0-1)
>>      node 0 cpus: 0 1
>>      node 0 size: 968 MB
>>      node 0 free: 664 MB
>>      node 1 cpus: 2 3
>>      node 1 size: 683 MB
>>      node 1 free: 333 MB
>>      node distances:
>>      node   0   1
>>        0:  10  20
>>
>>      step2: numactl -H
>>
>>      available: 3 nodes (0-2)
>>      node 0 cpus: 0 1
>>      node 0 size: 968 MB
>>      node 0 free: 677 MB
>>      node 1 cpus: 2 3
>>      node 1 size: 683 MB
>>      node 1 free: 333 MB
>>      node 2 cpus:
>>      node 2 size: 256 MB
>>      node 2 free: 256 MB
>>      node distances:
>>      node   0   1   2
>>        0:  10  20  20
>>        1:  20  10  20
>>        2:  20  20  10
>>
>>      step3: numactl -H
>>
>>      available: 3 nodes (0-2)
>>      node 0 cpus: 0 1
>>      node 0 size: 968 MB
>>      node 0 free: 686 MB
>>      node 1 cpus: 2 3
>>      node 1 size: 683 MB
>>      node 1 free: 336 MB
>>      node 2 cpus:
>>      node 2 size: 256 MB
>>      node 2 free: 256 MB
>>      node distances:
>>      node   0   1   2
>>        0:  10  20  20
>>        1:  20  10  20
>>        2:  20  20  10

Re: Question about forcing 'disable-memdev'

[Dave Jiang]

On 2/27/24 1:24 PM, Jane Chu wrote:
> On 2/27/2024 8:40 AM, Dave Jiang wrote:
> 
>>
>> On 2/26/24 10:32 PM, Cao, Quanquan/曹 全全 wrote:
>>> Hi, Dave
>>>
>>> On the basis of this patch, I conducted some tests and encountered unexpected errors. I would like to inquire whether the design here is reasonable? Below are the steps of my testing:
>>>
>>> Link: https://lore.kernel.org/linux-cxl/170138109724.2882696.123294980050048623.stgit@djiang5-mobl3/
>>>
>>>
>>> Problem description: after creating a region, directly forcing 'disable-memdev' and then consuming memory leads to a kernel panic.
>> If you are forcing memory disable when the memory cannot be offlined, then this behavior is expected. You are ripping the memory away from underneath kernel mm. The reason the check was added is to prevent the users from doing exactly that.
> 
> Since user is doing the illegal thing, shouldn't that lead to SIGBUS or SIGKILL ?

The behavior is unpredictable once the CXL memory is ripped away. If the memory only backed user memory then you may see SIGBUS. But if the memory backed kernel data then kernel OOPs is not out of question. 

> 
> thanks,
> 
> -jane
> 
>>
>>>
>>> Test environment:
>>> KERNEL    6.8.0-rc1
>>> QEMU    8.2.0-rc4
>>>
>>> Test steps：
>>>        step1: set memory auto_online to movable zones.
>>>             echo online_movable > /sys/devices/system/memory/auto_online_blocks
>>>        step2: create region
>>>             cxl create-region -t ram -d decoder0.0 -m mem0
>>>        step3: disable memdev
>>>             cxl disable-memdev mem0 -f
>>>        step4: consum CXL memory
>>>             ./consumemem   <------kernel panic
>>>
>>> numactl node status:
>>>        step1: numactl -H
>>>
>>>      available: 2 nodes (0-1)
>>>      node 0 cpus: 0 1
>>>      node 0 size: 968 MB
>>>      node 0 free: 664 MB
>>>      node 1 cpus: 2 3
>>>      node 1 size: 683 MB
>>>      node 1 free: 333 MB
>>>      node distances:
>>>      node   0   1
>>>        0:  10  20
>>>
>>>      step2: numactl -H
>>>
>>>      available: 3 nodes (0-2)
>>>      node 0 cpus: 0 1
>>>      node 0 size: 968 MB
>>>      node 0 free: 677 MB
>>>      node 1 cpus: 2 3
>>>      node 1 size: 683 MB
>>>      node 1 free: 333 MB
>>>      node 2 cpus:
>>>      node 2 size: 256 MB
>>>      node 2 free: 256 MB
>>>      node distances:
>>>      node   0   1   2
>>>        0:  10  20  20
>>>        1:  20  10  20
>>>        2:  20  20  10
>>>
>>>      step3: numactl -H
>>>
>>>      available: 3 nodes (0-2)
>>>      node 0 cpus: 0 1
>>>      node 0 size: 968 MB
>>>      node 0 free: 686 MB
>>>      node 1 cpus: 2 3
>>>      node 1 size: 683 MB
>>>      node 1 free: 336 MB
>>>      node 2 cpus:
>>>      node 2 size: 256 MB
>>>      node 2 free: 256 MB
>>>      node distances:
>>>      node   0   1   2
>>>        0:  10  20  20
>>>        1:  20  10  20
>>>        2:  20  20  10

Re: Question about forcing 'disable-memdev'

[Jane Chu]

On 2/27/2024 12:28 PM, Dave Jiang wrote:

>
> On 2/27/24 1:24 PM, Jane Chu wrote:
>> On 2/27/2024 8:40 AM, Dave Jiang wrote:
>>
>>> On 2/26/24 10:32 PM, Cao, Quanquan/曹 全全 wrote:
>>>> Hi, Dave
>>>>
>>>> On the basis of this patch, I conducted some tests and encountered unexpected errors. I would like to inquire whether the design here is reasonable? Below are the steps of my testing:
>>>>
>>>> Link: https://lore.kernel.org/linux-cxl/170138109724.2882696.123294980050048623.stgit@djiang5-mobl3/
>>>>
>>>>
>>>> Problem description: after creating a region, directly forcing 'disable-memdev' and then consuming memory leads to a kernel panic.
>>> If you are forcing memory disable when the memory cannot be offlined, then this behavior is expected. You are ripping the memory away from underneath kernel mm. The reason the check was added is to prevent the users from doing exactly that.
>> Since user is doing the illegal thing, shouldn't that lead to SIGBUS or SIGKILL ?
> The behavior is unpredictable once the CXL memory is ripped away. If the memory only backed user memory then you may see SIGBUS. But if the memory backed kernel data then kernel OOPs is not out of question.

Make sense, thanks for the clarification.

-jane

>
>> thanks,
>>
>> -jane
>>
>>>> Test environment:
>>>> KERNEL    6.8.0-rc1
>>>> QEMU    8.2.0-rc4
>>>>
>>>> Test steps：
>>>>         step1: set memory auto_online to movable zones.
>>>>              echo online_movable > /sys/devices/system/memory/auto_online_blocks
>>>>         step2: create region
>>>>              cxl create-region -t ram -d decoder0.0 -m mem0
>>>>         step3: disable memdev
>>>>              cxl disable-memdev mem0 -f
>>>>         step4: consum CXL memory
>>>>              ./consumemem   <------kernel panic
>>>>
>>>> numactl node status:
>>>>         step1: numactl -H
>>>>
>>>>       available: 2 nodes (0-1)
>>>>       node 0 cpus: 0 1
>>>>       node 0 size: 968 MB
>>>>       node 0 free: 664 MB
>>>>       node 1 cpus: 2 3
>>>>       node 1 size: 683 MB
>>>>       node 1 free: 333 MB
>>>>       node distances:
>>>>       node   0   1
>>>>         0:  10  20
>>>>
>>>>       step2: numactl -H
>>>>
>>>>       available: 3 nodes (0-2)
>>>>       node 0 cpus: 0 1
>>>>       node 0 size: 968 MB
>>>>       node 0 free: 677 MB
>>>>       node 1 cpus: 2 3
>>>>       node 1 size: 683 MB
>>>>       node 1 free: 333 MB
>>>>       node 2 cpus:
>>>>       node 2 size: 256 MB
>>>>       node 2 free: 256 MB
>>>>       node distances:
>>>>       node   0   1   2
>>>>         0:  10  20  20
>>>>         1:  20  10  20
>>>>         2:  20  20  10
>>>>
>>>>       step3: numactl -H
>>>>
>>>>       available: 3 nodes (0-2)
>>>>       node 0 cpus: 0 1
>>>>       node 0 size: 968 MB
>>>>       node 0 free: 686 MB
>>>>       node 1 cpus: 2 3
>>>>       node 1 size: 683 MB
>>>>       node 1 free: 336 MB
>>>>       node 2 cpus:
>>>>       node 2 size: 256 MB
>>>>       node 2 free: 256 MB
>>>>       node distances:
>>>>       node   0   1   2
>>>>         0:  10  20  20
>>>>         1:  20  10  20
>>>>         2:  20  20  10

Re: Question about forcing 'disable-memdev'

[Dan Williams]

Jane Chu wrote:
> On 2/27/2024 12:28 PM, Dave Jiang wrote:
> 
> >
> > On 2/27/24 1:24 PM, Jane Chu wrote:
> >> On 2/27/2024 8:40 AM, Dave Jiang wrote:
> >>
> >>> On 2/26/24 10:32 PM, Cao, Quanquan/曹 全全 wrote:
> >>>> Hi, Dave
> >>>>
> >>>> On the basis of this patch, I conducted some tests and encountered unexpected errors. I would like to inquire whether the design here is reasonable? Below are the steps of my testing:
> >>>>
> >>>> Link: https://lore.kernel.org/linux-cxl/170138109724.2882696.123294980050048623.stgit@djiang5-mobl3/
> >>>>
> >>>>
> >>>> Problem description: after creating a region, directly forcing 'disable-memdev' and then consuming memory leads to a kernel panic.
> >>> If you are forcing memory disable when the memory cannot be
> >>> offlined, then this behavior is expected. You are ripping the
> >>> memory away from underneath kernel mm. The reason the check was
> >>> added is to prevent the users from doing exactly that.
> >> Since user is doing the illegal thing, shouldn't that lead to
> >> SIGBUS or SIGKILL ?
> > The behavior is unpredictable once the CXL memory is ripped away. If
> > the memory only backed user memory then you may see SIGBUS. But if
> > the memory backed kernel data then kernel OOPs is not out of
> > question.
> 
> Make sense, thanks for the clarification.

I will just add consider the case of a technician physically removing a
card without shutting down the kernel's usage of it. That event is
indistinguishable from "cxl disable-memdev --force" at the driver level
since the driver just gets the same ->remove() callback with no
opportunity to return an error.

So this is a case of trusting the system administrator to know best, and
is why --force is documented as:

       -f, --force
	   DANGEROUS: Override the safety measure that blocks attempts
	   to disable a device if the tool determines the memdev is in active
	   usage. Recall that CXL memory ranges might have been established by
	   platform firmware and disabling an active device is akin to force
	   removing memory from a running system.