From: Jonathan Cameron <Jonathan.Cameron@Huawei.com>
To: "Cao, Quanquan/曹 全全" <caoqq@fujitsu.com>
Cc: <dan.j.williams@intel.com>, <dave.jiang@intel.com>,
<vishal.l.verma@intel.com>, <linux-cxl@vger.kernel.org>
Subject: Re: Question about a segmentation fault issue when accessing memory in a region
Date: Tue, 6 Feb 2024 10:17:19 +0000 [thread overview]
Message-ID: <20240206101719.000065bd@Huawei.com> (raw)
In-Reply-To: <3e84b919-7631-d1db-3e1d-33000f3f3868@fujitsu.com>
On Tue, 6 Feb 2024 15:38:31 +0800
"Cao, Quanquan/曹 全全" <caoqq@fujitsu.com> wrote:
> Hi, guys:
>
> I tried to create a region on the 2hb+6mem environment.
> Fortunately, the region was created successfully,
> and the system memory can be seen in the system that
> the system memory has increased.
>
> When I tried to use the memory on the region, a segfault occurred.
> anyone has idea?
There are some issues with TCG (QEMU emulator) and other parts
of QEMU and using CXL emulated memory as 'normal memory'.
The way I did the interleaving causes problems for parts of QEMU that
assume they can access memory within a page without doing lookups.
There are other problems with the page table walker emulation and
some locking issues.
Short term I'll try to get a solution out that works ok without
any interleave, and then we can look at whether we can enable
some forms of interleave (greater than page granularity).
Trying to solve this in the general case would require some
tricky caching layer to be introduced. Can probably be done but
may not be worth the effort.
Jonathan
>
> Environment info:
> os version: fedora 39
> linux version: 6.7-rc5
> qemu version: v8.2.0-rc4
> kvm: off
>
> The detailed operation commands are as follows.
> ========> Before region creation
> # numactl -H
> available: 2 nodes (0-1)
> node 0 cpus: 0 1
> node 0 size: 1006 MB
> node 0 free: 624 MB
> node 1 cpus: 2 3
> node 1 size: 899 MB
> node 1 free: 694 MB
> node distances:
> node 0 1
> 0: 10 20
> 1: 20 10
>
> # free -h
> total used free shared buff/cache
> available
> Mem: 1.9Gi 448Mi 1.3Gi 1.1Mi 330Mi
> 1.4Gi
> Swap: 1.9Gi 0B 1.9Gi
>
> ========> create a new region
> # echo online_movable > /sys/devices/system/memory/auto_online_blocks
> # cxl list -M -P -D -T -E -u
> [
> {
> "ports":[
> {
> "port":"port1",
> "host":"pci0000:20",
> "depth":1,
> "nr_dports":2,
> "dports":[
> {
> "dport":"0000:20:00.0",
> "id":"0"
> },
> {
> "dport":"0000:20:01.0",
> "id":"0x1"
> }
> ],
> "endpoints:port1":[
> {
> "endpoint":"endpoint5",
> "host":"mem1",
> "parent_dport":"0000:20:01.0",
> "depth":2,
> "memdev":{
> "memdev":"mem1",
> "ram_size":"256.00 MiB (268.44 MB)",
> "serial":"0",
> "host":"0000:22:00.0"
> }
> },
> {
> "endpoint":"endpoint4",
> "host":"mem0",
> "parent_dport":"0000:20:00.0",
> "depth":2,
> "memdev":{
> "memdev":"mem0",
> "ram_size":"256.00 MiB (268.44 MB)",
> "serial":"0",
> "host":"0000:21:00.0"
> }
> }
> ]
> },
> {
> "port":"port3",
> "host":"pci0000:0c",
> "depth":1,
> "nr_dports":2,
> "dports":[
> {
> "dport":"0000:0c:00.0",
> "id":"0"
> },
> {
> "dport":"0000:0c:01.0",
> "id":"0x1"
> }
> ],
> "endpoints:port3":[
> {
> "endpoint":"endpoint7",
> "host":"mem3",
> "parent_dport":"0000:0c:00.0",
> "depth":2,
> "memdev":{
> "memdev":"mem3",
> "ram_size":"256.00 MiB (268.44 MB)",
> "serial":"0",
> "host":"0000:0d:00.0"
> }
> },
> {
> "endpoint":"endpoint9",
> "host":"mem5",
> "parent_dport":"0000:0c:01.0",
> "depth":2,
> "memdev":{
> "memdev":"mem5",
> "ram_size":"256.00 MiB (268.44 MB)",
> "serial":"0",
> "host":"0000:0e:00.0"
> }
> }
> ]
> },
> {
> "port":"port2",
> "host":"pci0000:16",
> "depth":1,
> "nr_dports":2,
> "dports":[
> {
> "dport":"0000:16:00.0",
> "id":"0"
> },
> {
> "dport":"0000:16:01.0",
> "id":"0x1"
> }
> ],
> "endpoints:port2":[
> {
> "endpoint":"endpoint6",
> "host":"mem2",
> "parent_dport":"0000:16:01.0",
> "depth":2,
> "memdev":{
> "memdev":"mem2",
> "ram_size":"256.00 MiB (268.44 MB)",
> "serial":"0",
> "host":"0000:18:00.0"
> }
> },
> {
> "endpoint":"endpoint8",
> "host":"mem4",
> "parent_dport":"0000:16:00.0",
> "depth":2,
> "memdev":{
> "memdev":"mem4",
> "ram_size":"256.00 MiB (268.44 MB)",
> "serial":"0",
> "host":"0000:17:00.0"
> }
> }
> ]
> }
> ]
> },
> {
> "root decoders":[
> {
> "decoder":"decoder0.0",
> "resource":"0x750000000",
> "size":"4.00 GiB (4.29 GB)",
> "interleave_ways":3,
> "interleave_granularity":4096,
> "max_available_extent":"4.00 GiB (4.29 GB)",
> "pmem_capable":true,
> "volatile_capable":true,
> "accelmem_capable":true,
> "nr_targets":3,
> "targets":[
> {
> "target":"pci0000:20",
> "alias":"ACPI0016:00",
> "position":2,
> "id":"0x20"
> },
> {
> "target":"pci0000:16",
> "alias":"ACPI0016:01",
> "position":1,
> "id":"0x16"
> },
> {
> "target":"pci0000:0c",
> "alias":"ACPI0016:02",
> "position":0,
> "id":"0xc"
> }
> ]
> }
> ]
> }
> ]
> # cxl create-region -t ram -d decoder0.0 -m mem3 mem2 mem1 mem5 mem4 mem0
> {
> "region":"region0",
> "resource":"0x750000000",
> "size":"1536.00 MiB (1610.61 MB)",
> "type":"ram",
> "interleave_ways":6,
> "interleave_granularity":4096,
> "decode_state":"commit",
> "mappings":[
> {
> "position":5,
> "memdev":"mem0",
> "decoder":"decoder4.0"
> },
> {
> "position":4,
> "memdev":"mem4",
> "decoder":"decoder8.0"
> },
> {
> "position":3,
> "memdev":"mem5",
> "decoder":"decoder9.0"
> },
> {
> "position":2,
> "memdev":"mem1",
> "decoder":"decoder5.0"
> },
> {
> "position":1,
> "memdev":"mem2",
> "decoder":"decoder6.0"
> },
> {
> "position":0,
> "memdev":"mem3",
> "decoder":"decoder7.0"
> }
> ]
> }
> cxl region: cmd_create_region: created 1 region
>
> ===> After region creation
> # numactl -H
> available: 3 nodes (0-2)
> node 0 cpus: 0 1
> node 0 size: 1006 MB
> node 0 free: 736 MB
> node 1 cpus: 2 3
> node 1 size: 899 MB
> node 1 free: 548 MB
> node 2 cpus:
> node 2 size: 1536 MB <-- new numa node was create
> node 2 free: 1536 MB
> node distances:
> node 0 1 2
> 0: 10 20 20
> 1: 20 10 20
> 2: 20 20 10
>
> # free -h <-- System memory increased
> total used free shared buff/cache
> available
> Mem: 3.4Gi 504Mi 2.8Gi 1.1Mi 338Mi
> 2.9Gi
> Swap: 1.9Gi 0B 1.9Gi
>
> # numactl --membind=2 ls
> Segmentation fault (core dumped)
>
> # dmesg
> [ 310.119439] show_signal_msg: 20 callbacks suppressed
> [ 310.119499] ls[1153]: segfault at 1c454 ip 00007fb4cf771523 sp
> 00007fff57753fe0 error 4 in ld-linux-x86-64.so.2[7fb4cf753000+27000]
> likely on CPU 2 (core 0, socket 2)
> [ 310.350803] Code: 01 42 08 48 8b bd 98 fd ff ff 48 8b 57 70 48 85 d2
> 74 04 48 01 42 08 48 8b bd 98 fd ff ff 48 8b 97 60 01 00 00 48 85 d2 74
> 04 <48> 01 42 08 48 8b bd 98 fd ff ff 48 8b 8f f8 00 00 00 48 85 c9 74
>
prev parent reply other threads:[~2024-02-06 10:17 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-02-06 7:38 Question about a segmentation fault issue when accessing memory in a region Cao, Quanquan/曹 全全
2024-02-06 8:00 ` Cao, Quanquan/曹 全全
2024-02-06 10:17 ` Jonathan Cameron [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20240206101719.000065bd@Huawei.com \
--to=jonathan.cameron@huawei.com \
--cc=caoqq@fujitsu.com \
--cc=dan.j.williams@intel.com \
--cc=dave.jiang@intel.com \
--cc=linux-cxl@vger.kernel.org \
--cc=vishal.l.verma@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox