* Question about forcing 'disable-memdev'
@ 2024-02-27 5:32 Cao, Quanquan/曹 全全
2024-02-27 16:40 ` Dave Jiang
0 siblings, 1 reply; 6+ messages in thread
From: Cao, Quanquan/曹 全全 @ 2024-02-27 5:32 UTC (permalink / raw)
To: Dave Jiang, linux-cxl, nvdimm; +Cc: vishal.l.verma
Hi, Dave
On the basis of this patch, I conducted some tests and encountered
unexpected errors. I would like to inquire whether the design here is
reasonable? Below are the steps of my testing:
Link:
https://lore.kernel.org/linux-cxl/170138109724.2882696.123294980050048623.stgit@djiang5-mobl3/
Problem description: after creating a region, directly forcing
'disable-memdev' and then consuming memory leads to a kernel panic.
Test environment:
KERNEL 6.8.0-rc1
QEMU 8.2.0-rc4
Test steps：
step1: set memory auto_online to movable zones.
echo online_movable >
/sys/devices/system/memory/auto_online_blocks
step2: create region
cxl create-region -t ram -d decoder0.0 -m mem0
step3: disable memdev
cxl disable-memdev mem0 -f
step4: consum CXL memory
./consumemem <------kernel panic
numactl node status:
step1: numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 1
node 0 size: 968 MB
node 0 free: 664 MB
node 1 cpus: 2 3
node 1 size: 683 MB
node 1 free: 333 MB
node distances:
node 0 1
0: 10 20
step2: numactl -H
available: 3 nodes (0-2)
node 0 cpus: 0 1
node 0 size: 968 MB
node 0 free: 677 MB
node 1 cpus: 2 3
node 1 size: 683 MB
node 1 free: 333 MB
node 2 cpus:
node 2 size: 256 MB
node 2 free: 256 MB
node distances:
node 0 1 2
0: 10 20 20
1: 20 10 20
2: 20 20 10
step3: numactl -H
available: 3 nodes (0-2)
node 0 cpus: 0 1
node 0 size: 968 MB
node 0 free: 686 MB
node 1 cpus: 2 3
node 1 size: 683 MB
node 1 free: 336 MB
node 2 cpus:
node 2 size: 256 MB
node 2 free: 256 MB
node distances:
node 0 1 2
0: 10 20 20
1: 20 10 20
2: 20 20 10
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: Question about forcing 'disable-memdev'
2024-02-27 5:32 Question about forcing 'disable-memdev' Cao, Quanquan/曹 全全
@ 2024-02-27 16:40 ` Dave Jiang
2024-02-27 20:24 ` Jane Chu
0 siblings, 1 reply; 6+ messages in thread
From: Dave Jiang @ 2024-02-27 16:40 UTC (permalink / raw)
To: Cao, Quanquan/曹 全全, linux-cxl, nvdimm; +Cc: vishal.l.verma
On 2/26/24 10:32 PM, Cao, Quanquan/曹 全全 wrote:
> Hi, Dave
>
> On the basis of this patch, I conducted some tests and encountered unexpected errors. I would like to inquire whether the design here is reasonable? Below are the steps of my testing:
>
> Link: https://lore.kernel.org/linux-cxl/170138109724.2882696.123294980050048623.stgit@djiang5-mobl3/
>
>
> Problem description: after creating a region, directly forcing 'disable-memdev' and then consuming memory leads to a kernel panic.
If you are forcing memory disable when the memory cannot be offlined, then this behavior is expected. You are ripping the memory away from underneath kernel mm. The reason the check was added is to prevent the users from doing exactly that.
>
>
> Test environment:
> KERNEL 6.8.0-rc1
> QEMU 8.2.0-rc4
>
> Test steps：
> step1: set memory auto_online to movable zones.
> echo online_movable > /sys/devices/system/memory/auto_online_blocks
> step2: create region
> cxl create-region -t ram -d decoder0.0 -m mem0
> step3: disable memdev
> cxl disable-memdev mem0 -f
> step4: consum CXL memory
> ./consumemem <------kernel panic
>
> numactl node status:
> step1: numactl -H
>
> available: 2 nodes (0-1)
> node 0 cpus: 0 1
> node 0 size: 968 MB
> node 0 free: 664 MB
> node 1 cpus: 2 3
> node 1 size: 683 MB
> node 1 free: 333 MB
> node distances:
> node 0 1
> 0: 10 20
>
> step2: numactl -H
>
> available: 3 nodes (0-2)
> node 0 cpus: 0 1
> node 0 size: 968 MB
> node 0 free: 677 MB
> node 1 cpus: 2 3
> node 1 size: 683 MB
> node 1 free: 333 MB
> node 2 cpus:
> node 2 size: 256 MB
> node 2 free: 256 MB
> node distances:
> node 0 1 2
> 0: 10 20 20
> 1: 20 10 20
> 2: 20 20 10
>
> step3: numactl -H
>
> available: 3 nodes (0-2)
> node 0 cpus: 0 1
> node 0 size: 968 MB
> node 0 free: 686 MB
> node 1 cpus: 2 3
> node 1 size: 683 MB
> node 1 free: 336 MB
> node 2 cpus:
> node 2 size: 256 MB
> node 2 free: 256 MB
> node distances:
> node 0 1 2
> 0: 10 20 20
> 1: 20 10 20
> 2: 20 20 10
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Question about forcing 'disable-memdev'
2024-02-27 16:40 ` Dave Jiang
@ 2024-02-27 20:24 ` Jane Chu
2024-02-27 20:28 ` Dave Jiang
0 siblings, 1 reply; 6+ messages in thread
From: Jane Chu @ 2024-02-27 20:24 UTC (permalink / raw)
To: Dave Jiang, Cao, Quanquan/曹 全全, linux-cxl,
nvdimm
Cc: vishal.l.verma
On 2/27/2024 8:40 AM, Dave Jiang wrote:
>
> On 2/26/24 10:32 PM, Cao, Quanquan/曹 全全 wrote:
>> Hi, Dave
>>
>> On the basis of this patch, I conducted some tests and encountered unexpected errors. I would like to inquire whether the design here is reasonable? Below are the steps of my testing:
>>
>> Link: https://lore.kernel.org/linux-cxl/170138109724.2882696.123294980050048623.stgit@djiang5-mobl3/
>>
>>
>> Problem description: after creating a region, directly forcing 'disable-memdev' and then consuming memory leads to a kernel panic.
> If you are forcing memory disable when the memory cannot be offlined, then this behavior is expected. You are ripping the memory away from underneath kernel mm. The reason the check was added is to prevent the users from doing exactly that.
Since user is doing the illegal thing, shouldn't that lead to SIGBUS or
SIGKILL ?
thanks,
-jane
>
>>
>> Test environment:
>> KERNEL 6.8.0-rc1
>> QEMU 8.2.0-rc4
>>
>> Test steps：
>> step1: set memory auto_online to movable zones.
>> echo online_movable > /sys/devices/system/memory/auto_online_blocks
>> step2: create region
>> cxl create-region -t ram -d decoder0.0 -m mem0
>> step3: disable memdev
>> cxl disable-memdev mem0 -f
>> step4: consum CXL memory
>> ./consumemem <------kernel panic
>>
>> numactl node status:
>> step1: numactl -H
>>
>> available: 2 nodes (0-1)
>> node 0 cpus: 0 1
>> node 0 size: 968 MB
>> node 0 free: 664 MB
>> node 1 cpus: 2 3
>> node 1 size: 683 MB
>> node 1 free: 333 MB
>> node distances:
>> node 0 1
>> 0: 10 20
>>
>> step2: numactl -H
>>
>> available: 3 nodes (0-2)
>> node 0 cpus: 0 1
>> node 0 size: 968 MB
>> node 0 free: 677 MB
>> node 1 cpus: 2 3
>> node 1 size: 683 MB
>> node 1 free: 333 MB
>> node 2 cpus:
>> node 2 size: 256 MB
>> node 2 free: 256 MB
>> node distances:
>> node 0 1 2
>> 0: 10 20 20
>> 1: 20 10 20
>> 2: 20 20 10
>>
>> step3: numactl -H
>>
>> available: 3 nodes (0-2)
>> node 0 cpus: 0 1
>> node 0 size: 968 MB
>> node 0 free: 686 MB
>> node 1 cpus: 2 3
>> node 1 size: 683 MB
>> node 1 free: 336 MB
>> node 2 cpus:
>> node 2 size: 256 MB
>> node 2 free: 256 MB
>> node distances:
>> node 0 1 2
>> 0: 10 20 20
>> 1: 20 10 20
>> 2: 20 20 10
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Question about forcing 'disable-memdev'
2024-02-27 20:24 ` Jane Chu
@ 2024-02-27 20:28 ` Dave Jiang
2024-02-28 20:17 ` Jane Chu
0 siblings, 1 reply; 6+ messages in thread
From: Dave Jiang @ 2024-02-27 20:28 UTC (permalink / raw)
To: Jane Chu, Cao, Quanquan/曹 全全, linux-cxl,
nvdimm
Cc: vishal.l.verma
On 2/27/24 1:24 PM, Jane Chu wrote:
> On 2/27/2024 8:40 AM, Dave Jiang wrote:
>
>>
>> On 2/26/24 10:32 PM, Cao, Quanquan/曹 全全 wrote:
>>> Hi, Dave
>>>
>>> On the basis of this patch, I conducted some tests and encountered unexpected errors. I would like to inquire whether the design here is reasonable? Below are the steps of my testing:
>>>
>>> Link: https://lore.kernel.org/linux-cxl/170138109724.2882696.123294980050048623.stgit@djiang5-mobl3/
>>>
>>>
>>> Problem description: after creating a region, directly forcing 'disable-memdev' and then consuming memory leads to a kernel panic.
>> If you are forcing memory disable when the memory cannot be offlined, then this behavior is expected. You are ripping the memory away from underneath kernel mm. The reason the check was added is to prevent the users from doing exactly that.
>
> Since user is doing the illegal thing, shouldn't that lead to SIGBUS or SIGKILL ?
The behavior is unpredictable once the CXL memory is ripped away. If the memory only backed user memory then you may see SIGBUS. But if the memory backed kernel data then kernel OOPs is not out of question.
>
> thanks,
>
> -jane
>
>>
>>>
>>> Test environment:
>>> KERNEL 6.8.0-rc1
>>> QEMU 8.2.0-rc4
>>>
>>> Test steps：
>>> step1: set memory auto_online to movable zones.
>>> echo online_movable > /sys/devices/system/memory/auto_online_blocks
>>> step2: create region
>>> cxl create-region -t ram -d decoder0.0 -m mem0
>>> step3: disable memdev
>>> cxl disable-memdev mem0 -f
>>> step4: consum CXL memory
>>> ./consumemem <------kernel panic
>>>
>>> numactl node status:
>>> step1: numactl -H
>>>
>>> available: 2 nodes (0-1)
>>> node 0 cpus: 0 1
>>> node 0 size: 968 MB
>>> node 0 free: 664 MB
>>> node 1 cpus: 2 3
>>> node 1 size: 683 MB
>>> node 1 free: 333 MB
>>> node distances:
>>> node 0 1
>>> 0: 10 20
>>>
>>> step2: numactl -H
>>>
>>> available: 3 nodes (0-2)
>>> node 0 cpus: 0 1
>>> node 0 size: 968 MB
>>> node 0 free: 677 MB
>>> node 1 cpus: 2 3
>>> node 1 size: 683 MB
>>> node 1 free: 333 MB
>>> node 2 cpus:
>>> node 2 size: 256 MB
>>> node 2 free: 256 MB
>>> node distances:
>>> node 0 1 2
>>> 0: 10 20 20
>>> 1: 20 10 20
>>> 2: 20 20 10
>>>
>>> step3: numactl -H
>>>
>>> available: 3 nodes (0-2)
>>> node 0 cpus: 0 1
>>> node 0 size: 968 MB
>>> node 0 free: 686 MB
>>> node 1 cpus: 2 3
>>> node 1 size: 683 MB
>>> node 1 free: 336 MB
>>> node 2 cpus:
>>> node 2 size: 256 MB
>>> node 2 free: 256 MB
>>> node distances:
>>> node 0 1 2
>>> 0: 10 20 20
>>> 1: 20 10 20
>>> 2: 20 20 10
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Question about forcing 'disable-memdev'
2024-02-27 20:28 ` Dave Jiang
@ 2024-02-28 20:17 ` Jane Chu
2024-03-07 20:55 ` Dan Williams
0 siblings, 1 reply; 6+ messages in thread
From: Jane Chu @ 2024-02-28 20:17 UTC (permalink / raw)
To: Dave Jiang, Cao, Quanquan/曹 全全, linux-cxl,
nvdimm
Cc: vishal.l.verma
On 2/27/2024 12:28 PM, Dave Jiang wrote:
>
> On 2/27/24 1:24 PM, Jane Chu wrote:
>> On 2/27/2024 8:40 AM, Dave Jiang wrote:
>>
>>> On 2/26/24 10:32 PM, Cao, Quanquan/曹 全全 wrote:
>>>> Hi, Dave
>>>>
>>>> On the basis of this patch, I conducted some tests and encountered unexpected errors. I would like to inquire whether the design here is reasonable? Below are the steps of my testing:
>>>>
>>>> Link: https://lore.kernel.org/linux-cxl/170138109724.2882696.123294980050048623.stgit@djiang5-mobl3/
>>>>
>>>>
>>>> Problem description: after creating a region, directly forcing 'disable-memdev' and then consuming memory leads to a kernel panic.
>>> If you are forcing memory disable when the memory cannot be offlined, then this behavior is expected. You are ripping the memory away from underneath kernel mm. The reason the check was added is to prevent the users from doing exactly that.
>> Since user is doing the illegal thing, shouldn't that lead to SIGBUS or SIGKILL ?
> The behavior is unpredictable once the CXL memory is ripped away. If the memory only backed user memory then you may see SIGBUS. But if the memory backed kernel data then kernel OOPs is not out of question.
Make sense, thanks for the clarification.
-jane
>
>> thanks,
>>
>> -jane
>>
>>>> Test environment:
>>>> KERNEL 6.8.0-rc1
>>>> QEMU 8.2.0-rc4
>>>>
>>>> Test steps：
>>>> step1: set memory auto_online to movable zones.
>>>> echo online_movable > /sys/devices/system/memory/auto_online_blocks
>>>> step2: create region
>>>> cxl create-region -t ram -d decoder0.0 -m mem0
>>>> step3: disable memdev
>>>> cxl disable-memdev mem0 -f
>>>> step4: consum CXL memory
>>>> ./consumemem <------kernel panic
>>>>
>>>> numactl node status:
>>>> step1: numactl -H
>>>>
>>>> available: 2 nodes (0-1)
>>>> node 0 cpus: 0 1
>>>> node 0 size: 968 MB
>>>> node 0 free: 664 MB
>>>> node 1 cpus: 2 3
>>>> node 1 size: 683 MB
>>>> node 1 free: 333 MB
>>>> node distances:
>>>> node 0 1
>>>> 0: 10 20
>>>>
>>>> step2: numactl -H
>>>>
>>>> available: 3 nodes (0-2)
>>>> node 0 cpus: 0 1
>>>> node 0 size: 968 MB
>>>> node 0 free: 677 MB
>>>> node 1 cpus: 2 3
>>>> node 1 size: 683 MB
>>>> node 1 free: 333 MB
>>>> node 2 cpus:
>>>> node 2 size: 256 MB
>>>> node 2 free: 256 MB
>>>> node distances:
>>>> node 0 1 2
>>>> 0: 10 20 20
>>>> 1: 20 10 20
>>>> 2: 20 20 10
>>>>
>>>> step3: numactl -H
>>>>
>>>> available: 3 nodes (0-2)
>>>> node 0 cpus: 0 1
>>>> node 0 size: 968 MB
>>>> node 0 free: 686 MB
>>>> node 1 cpus: 2 3
>>>> node 1 size: 683 MB
>>>> node 1 free: 336 MB
>>>> node 2 cpus:
>>>> node 2 size: 256 MB
>>>> node 2 free: 256 MB
>>>> node distances:
>>>> node 0 1 2
>>>> 0: 10 20 20
>>>> 1: 20 10 20
>>>> 2: 20 20 10
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Question about forcing 'disable-memdev'
2024-02-28 20:17 ` Jane Chu
@ 2024-03-07 20:55 ` Dan Williams
0 siblings, 0 replies; 6+ messages in thread
From: Dan Williams @ 2024-03-07 20:55 UTC (permalink / raw)
To: Jane Chu, Dave Jiang, Cao, Quanquan/曹 全全,
linux-cxl, nvdimm
Cc: vishal.l.verma
Jane Chu wrote:
> On 2/27/2024 12:28 PM, Dave Jiang wrote:
>
> >
> > On 2/27/24 1:24 PM, Jane Chu wrote:
> >> On 2/27/2024 8:40 AM, Dave Jiang wrote:
> >>
> >>> On 2/26/24 10:32 PM, Cao, Quanquan/曹 全全 wrote:
> >>>> Hi, Dave
> >>>>
> >>>> On the basis of this patch, I conducted some tests and encountered unexpected errors. I would like to inquire whether the design here is reasonable? Below are the steps of my testing:
> >>>>
> >>>> Link: https://lore.kernel.org/linux-cxl/170138109724.2882696.123294980050048623.stgit@djiang5-mobl3/
> >>>>
> >>>>
> >>>> Problem description: after creating a region, directly forcing 'disable-memdev' and then consuming memory leads to a kernel panic.
> >>> If you are forcing memory disable when the memory cannot be
> >>> offlined, then this behavior is expected. You are ripping the
> >>> memory away from underneath kernel mm. The reason the check was
> >>> added is to prevent the users from doing exactly that.
> >> Since user is doing the illegal thing, shouldn't that lead to
> >> SIGBUS or SIGKILL ?
> > The behavior is unpredictable once the CXL memory is ripped away. If
> > the memory only backed user memory then you may see SIGBUS. But if
> > the memory backed kernel data then kernel OOPs is not out of
> > question.
>
> Make sense, thanks for the clarification.
I will just add consider the case of a technician physically removing a
card without shutting down the kernel's usage of it. That event is
indistinguishable from "cxl disable-memdev --force" at the driver level
since the driver just gets the same ->remove() callback with no
opportunity to return an error.
So this is a case of trusting the system administrator to know best, and
is why --force is documented as:
-f, --force
DANGEROUS: Override the safety measure that blocks attempts
to disable a device if the tool determines the memdev is in active
usage. Recall that CXL memory ranges might have been established by
platform firmware and disabling an active device is akin to force
removing memory from a running system.
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2024-03-07 20:55 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-02-27 5:32 Question about forcing 'disable-memdev' Cao, Quanquan/曹 全全
2024-02-27 16:40 ` Dave Jiang
2024-02-27 20:24 ` Jane Chu
2024-02-27 20:28 ` Dave Jiang
2024-02-28 20:17 ` Jane Chu
2024-03-07 20:55 ` Dan Williams
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox