[SPDK] SEGFAULT when calling spdk_nvme_qpair_process_completions

[SPDK] SEGFAULT when calling spdk_nvme_qpair_process_completions

[@ 2019-01-08 12:27 UTC (permalink / raw)]

[-- Attachment #1: Type: text/plain, Size: 1716 bytes --]

Hi. I'm currently developing a layer on top of the spdk nvme driver. I'm 
getting a  segfault when calling spdk_nvme_qpair_process_completions, 
but only on some specific drive reads. gdb shows that at 
nvme_qpair.c:395, when trying to access qpair->ctrlr this triggers a 
segmentation fault, so i guess the controller state is not valid. This 
error seems to be consistent between executions of the program, but 
appears in different times in different access patterns. What could 
cause this to happen?

I'm developing in the vagrant image given by spdk, so not on real hardware.

gdb Backtrace

Thread 2 "taspdk_merge_so" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff5301700 (LWP 6036)]
spdk_nvme_qpair_process_completions (qpair=0x2000009ffef8, 
max_completions=0) at nvme_qpair.c:395
395             if (qpair->ctrlr->is_failed) {
(gdb) bt
#0  spdk_nvme_qpair_process_completions (qpair=0x2000009ffef8, 
max_completions=0) at nvme_qpair.c:395
#1  0x00007ffff73b210d in taspdk_poll_queues (data=0x0) at taspdk.c:9
#2  0x00007ffff64d6822 in PollingAPI::handleServices() () from 
/home/vagrant/ompss-2/nanos6/installation//lib/libnanos6-optimized.so.0.0.0
#3  0x00007ffff64daf44 in LeaderThread::body() () from 
/home/vagrant/ompss-2/nanos6/installation//lib/libnanos6-optimized.so.0.0.0
#4  0x00007ffff653eb8f in kernel_level_thread_body_wrapper(void*) [clone 
.lto_priv.221] () from 
/home/vagrant/ompss-2/nanos6/installation//lib/libnanos6-optimized.so.0.0.0
#5  0x00007ffff6f6d6db in start_thread (arg=0x7ffff5301700) at 
pthread_create.c:463
#6  0x00007ffff6c9688f in clone () at 
../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Re: [SPDK] SEGFAULT when calling spdk_nvme_qpair_process_completions

[Luse, Paul E]

[-- Attachment #1: Type: text/plain, Size: 2535 bytes --]

Hi Aleix,

Thanks, nothing comes to mind off the top of my head - would you mind filling an issue at https://github.com/spdk/spdk/issues for this?  That will both make sure we get all the info needed to reproduce and get it reviewed in our weekly issue scrub forum.  Especially if you're using the spdk vagrant scripts this should be reproducible.   Well, unless someone else has a quick suggestion or idea here on the mail list... 

Thx
Paul 


-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of aleix sanchis ramírez
Sent: Tuesday, January 8, 2019 5:28 AM
To: spdk(a)lists.01.org
Subject: [SPDK] SEGFAULT when calling spdk_nvme_qpair_process_completions

Hi. I'm currently developing a layer on top of the spdk nvme driver. I'm getting a  segfault when calling spdk_nvme_qpair_process_completions,
but only on some specific drive reads. gdb shows that at nvme_qpair.c:395, when trying to access qpair->ctrlr this triggers a segmentation fault, so i guess the controller state is not valid. This error seems to be consistent between executions of the program, but appears in different times in different access patterns. What could cause this to happen?

I'm developing in the vagrant image given by spdk, so not on real hardware.

gdb Backtrace

Thread 2 "taspdk_merge_so" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff5301700 (LWP 6036)] spdk_nvme_qpair_process_completions (qpair=0x2000009ffef8,
max_completions=0) at nvme_qpair.c:395
395             if (qpair->ctrlr->is_failed) {
(gdb) bt
#0  spdk_nvme_qpair_process_completions (qpair=0x2000009ffef8,
max_completions=0) at nvme_qpair.c:395
#1  0x00007ffff73b210d in taspdk_poll_queues (data=0x0) at taspdk.c:9
#2  0x00007ffff64d6822 in PollingAPI::handleServices() () from
/home/vagrant/ompss-2/nanos6/installation//lib/libnanos6-optimized.so.0.0.0
#3  0x00007ffff64daf44 in LeaderThread::body() () from
/home/vagrant/ompss-2/nanos6/installation//lib/libnanos6-optimized.so.0.0.0
#4  0x00007ffff653eb8f in kernel_level_thread_body_wrapper(void*) [clone .lto_priv.221] () from
/home/vagrant/ompss-2/nanos6/installation//lib/libnanos6-optimized.so.0.0.0
#5  0x00007ffff6f6d6db in start_thread (arg=0x7ffff5301700) at
pthread_create.c:463
#6  0x00007ffff6c9688f in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95

_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] SEGFAULT when calling spdk_nvme_qpair_process_completions

[Harris, James R]

[-- Attachment #1: Type: text/plain, Size: 3512 bytes --]

Hi Aleix,

We will need reproduction steps if an issue is filed in GitHub.  Do you have a smaller reproduction vehicle that you can share?  Without seeing the code for the layer you have on top of the NVMe driver, it will be difficult to debug.

A few questions:

1) Can you describe the threading model for your program?
2) How are you allocating (and freeing) queues in your program?
3) What does qpair->ctrlr equal when this crash occurs?  Meaning is it NULL, some garbage value, something else?

Thanks,

-Jim


On 1/8/19, 2:51 PM, "SPDK on behalf of Luse, Paul E" <spdk-bounces(a)lists.01.org on behalf of paul.e.luse(a)intel.com> wrote:

    Hi Aleix,
    
    Thanks, nothing comes to mind off the top of my head - would you mind filling an issue at https://github.com/spdk/spdk/issues for this?  That will both make sure we get all the info needed to reproduce and get it reviewed in our weekly issue scrub forum.  Especially if you're using the spdk vagrant scripts this should be reproducible.   Well, unless someone else has a quick suggestion or idea here on the mail list... 
    
    Thx
    Paul 
    
    
    -----Original Message-----
    From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of aleix sanchis ramírez
    Sent: Tuesday, January 8, 2019 5:28 AM
    To: spdk(a)lists.01.org
    Subject: [SPDK] SEGFAULT when calling spdk_nvme_qpair_process_completions
    
    Hi. I'm currently developing a layer on top of the spdk nvme driver. I'm getting a  segfault when calling spdk_nvme_qpair_process_completions,
    but only on some specific drive reads. gdb shows that at nvme_qpair.c:395, when trying to access qpair->ctrlr this triggers a segmentation fault, so i guess the controller state is not valid. This error seems to be consistent between executions of the program, but appears in different times in different access patterns. What could cause this to happen?
    
    I'm developing in the vagrant image given by spdk, so not on real hardware.
    
    gdb Backtrace
    
    Thread 2 "taspdk_merge_so" received signal SIGSEGV, Segmentation fault.
    [Switching to Thread 0x7ffff5301700 (LWP 6036)] spdk_nvme_qpair_process_completions (qpair=0x2000009ffef8,
    max_completions=0) at nvme_qpair.c:395
    395             if (qpair->ctrlr->is_failed) {
    (gdb) bt
    #0  spdk_nvme_qpair_process_completions (qpair=0x2000009ffef8,
    max_completions=0) at nvme_qpair.c:395
    #1  0x00007ffff73b210d in taspdk_poll_queues (data=0x0) at taspdk.c:9
    #2  0x00007ffff64d6822 in PollingAPI::handleServices() () from
    /home/vagrant/ompss-2/nanos6/installation//lib/libnanos6-optimized.so.0.0.0
    #3  0x00007ffff64daf44 in LeaderThread::body() () from
    /home/vagrant/ompss-2/nanos6/installation//lib/libnanos6-optimized.so.0.0.0
    #4  0x00007ffff653eb8f in kernel_level_thread_body_wrapper(void*) [clone .lto_priv.221] () from
    /home/vagrant/ompss-2/nanos6/installation//lib/libnanos6-optimized.so.0.0.0
    #5  0x00007ffff6f6d6db in start_thread (arg=0x7ffff5301700) at
    pthread_create.c:463
    #6  0x00007ffff6c9688f in clone () at
    ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
    
    _______________________________________________
    SPDK mailing list
    SPDK(a)lists.01.org
    https://lists.01.org/mailman/listinfo/spdk
    _______________________________________________
    SPDK mailing list
    SPDK(a)lists.01.org
    https://lists.01.org/mailman/listinfo/spdk