Re: [SPDK] invalid usermode virtual address 0xffff8f600000

Re: [SPDK] invalid usermode virtual address 0xffff8f600000

[Oza Oza]

[-- Attachment #1: Type: text/plain, Size: 1460 bytes --]

Latest SPDK stopped working on armv8 port for us.

When I checked in back to this commit it starts working.



commit effea17dadbc28d201a4ec793e1d7c169e2faac6

Author: Daniel Verkamp <daniel.verkamp(a)intel.com>

Date:   Wed Feb 22 15:06:31 2017 -0700



    nvmf: make invalid rdma_req case into an assert



    The wr_id should never be NULL - it will always correspond to a request

    we previously posted.  Convert the check to an assert() so we notice if

    this ever happens (which would indicate a programming error somewhere

    else).



    While we're here, add a more robust check to make sure the request is

    actually in the correct array of requests for the connection being

    polled (also in an assert, since this should never fail in normal

    execution).



    Change-Id: I855763d7d827fb8cf00a775c7bc2ccb579db8d0f

    Signed-off-by: Daniel Verkamp <daniel.verkamp(a)intel.com>



*From:* Oza Oza [mailto:oza.oza(a)broadcom.com]
*Sent:* Thursday, April 27, 2017 12:03 PM
*To:* 'Storage Performance Development Kit'
*Subject:* invalid usermode virtual address 0xffff8f600000



Hi,



Why do I get invalid user mode virtual address when I enabled DEBUG in
SPDK/DPDK ?



invalid usermode virtual address 0xffff8f600000

invalid usermode virtual address 0xffff0f800000

invalid usermode virtual address 0xffff0f400000



this is armv8.



Regards,

Oza.

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 4681 bytes --]

Re: [SPDK] invalid usermode virtual address 0xffff8f600000

[Oza Oza]

[-- Attachment #1: Type: text/plain, Size: 3039 bytes --]

Yes. armv8 has 48 bit physical address and give 256 TB of memory map to user
space. so with that it is fine now.

Once our development/porting is done, we will consider up-streaming the
patches, though it might take little time.
Sorry about that.

Regards,
Oza.
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, James R
Sent: Thursday, April 27, 2017 11:23 PM
To: Storage Performance Development Kit
Subject: Re: [SPDK] invalid usermode virtual address 0xffff8f600000

Hi Oza,

> On Apr 27, 2017, at 12:04 AM, Oza Oza <oza.oza(a)broadcom.com> wrote:
>
> Latest SPDK stopped working on armv8 port for us.

I would suggest working to upstream your armv8 patches.  Without
understanding the changes made to enable SPDK on armv8, it is difficult for
the community to keep from breaking a set of out-of-tree patches.

More details around “stopped working” would also help understanding the
issues.  For example, are you testing NVMe PCI devices, NVMe-oF, etc.

> When I checked in back to this commit it starts working.

Could you double check your git bisect?  I do not see how this commit (or
the one before or after it) could affect anything related to a difference in
CPU architecture.

>
> commit effea17dadbc28d201a4ec793e1d7c169e2faac6
> Author: Daniel Verkamp <daniel.verkamp(a)intel.com>
> Date:   Wed Feb 22 15:06:31 2017 -0700
>
>     nvmf: make invalid rdma_req case into an assert
>
>     The wr_id should never be NULL - it will always correspond to a
> request
>     we previously posted.  Convert the check to an assert() so we notice
> if
>     this ever happens (which would indicate a programming error somewhere
>     else).
>
>     While we're here, add a more robust check to make sure the request is
>     actually in the correct array of requests for the connection being
>     polled (also in an assert, since this should never fail in normal
>     execution).
>
>     Change-Id: I855763d7d827fb8cf00a775c7bc2ccb579db8d0f
>     Signed-off-by: Daniel Verkamp <daniel.verkamp(a)intel.com>
>
> From: Oza Oza [mailto:oza.oza(a)broadcom.com]
> Sent: Thursday, April 27, 2017 12:03 PM
> To: 'Storage Performance Development Kit'
> Subject: invalid usermode virtual address 0xffff8f600000
>
> Hi,
>
> Why do I get invalid user mode virtual address when I enabled DEBUG in
> SPDK/DPDK ?
>
> invalid usermode virtual address 0xffff8f600000 invalid usermode
> virtual address 0xffff0f800000 invalid usermode virtual address
> 0xffff0f400000

vtophys.c fails any translation operation on a virtual address > (1ULL <<
47).  Looks like on armv8 the limit should be at least (1ULL << 48).

>
> this is armv8.
>
> Regards,
> Oza.
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] invalid usermode virtual address 0xffff8f600000

[Verkamp, Daniel]

[-- Attachment #1: Type: text/plain, Size: 1601 bytes --]

> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, James R
> Sent: Thursday, April 27, 2017 10:53 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] invalid usermode virtual address 0xffff8f600000
> 
> Hi Oza,
> 
> > On Apr 27, 2017, at 12:04 AM, Oza Oza <oza.oza(a)broadcom.com> wrote:
> >
> > Latest SPDK stopped working on armv8 port for us.
> 
> I would suggest working to upstream your armv8 patches.  Without
> understanding the changes made to enable SPDK on armv8, it is difficult for the
> community to keep from breaking a set of out-of-tree patches.
> 
> More details around “stopped working” would also help understanding the
> issues.  For example, are you testing NVMe PCI devices, NVMe-oF, etc.
> 
> > When I checked in back to this commit it starts working.
> 
> Could you double check your git bisect?  I do not see how this commit (or the
> one before or after it) could affect anything related to a difference in CPU
> architecture.
> 

Hi Oza,

In addition to what Jim mentioned, the vtophys.c address translation code (which is where the "invalid usermode virtual address" message is coming from) assumes the Linux x86-64 userspace memory layout; this means that all valid usermode addresses must be in the low 47 bits of the address space, and they must be canonical (high bits are all the same and must be 0 for userspace addresses).

If the memory layout on your platform is different, you will need to adapt vtophys for this case.
 
Thanks,
-- Daniel

Re: [SPDK] invalid usermode virtual address 0xffff8f600000

[Harris, James R]

[-- Attachment #1: Type: text/plain, Size: 2385 bytes --]

Hi Oza,

> On Apr 27, 2017, at 12:04 AM, Oza Oza <oza.oza(a)broadcom.com> wrote:
> 
> Latest SPDK stopped working on armv8 port for us.

I would suggest working to upstream your armv8 patches.  Without understanding the changes made to enable SPDK on armv8, it is difficult for the community to keep from breaking a set of out-of-tree patches.

More details around “stopped working” would also help understanding the issues.  For example, are you testing NVMe PCI devices, NVMe-oF, etc.

> When I checked in back to this commit it starts working.

Could you double check your git bisect?  I do not see how this commit (or the one before or after it) could affect anything related to a difference in CPU architecture.

>  
> commit effea17dadbc28d201a4ec793e1d7c169e2faac6
> Author: Daniel Verkamp <daniel.verkamp(a)intel.com>
> Date:   Wed Feb 22 15:06:31 2017 -0700
>  
>     nvmf: make invalid rdma_req case into an assert
>  
>     The wr_id should never be NULL - it will always correspond to a request
>     we previously posted.  Convert the check to an assert() so we notice if
>     this ever happens (which would indicate a programming error somewhere
>     else).
>  
>     While we're here, add a more robust check to make sure the request is
>     actually in the correct array of requests for the connection being
>     polled (also in an assert, since this should never fail in normal
>     execution).
>  
>     Change-Id: I855763d7d827fb8cf00a775c7bc2ccb579db8d0f
>     Signed-off-by: Daniel Verkamp <daniel.verkamp(a)intel.com>
>  
> From: Oza Oza [mailto:oza.oza(a)broadcom.com] 
> Sent: Thursday, April 27, 2017 12:03 PM
> To: 'Storage Performance Development Kit'
> Subject: invalid usermode virtual address 0xffff8f600000
>  
> Hi,
>  
> Why do I get invalid user mode virtual address when I enabled DEBUG in SPDK/DPDK ?
>  
> invalid usermode virtual address 0xffff8f600000
> invalid usermode virtual address 0xffff0f800000
> invalid usermode virtual address 0xffff0f400000

vtophys.c fails any translation operation on a virtual address > (1ULL << 47).  Looks like on armv8 the limit should be at least (1ULL << 48).

>  
> this is armv8.
>  
> Regards,
> Oza.
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

[SPDK] invalid usermode virtual address 0xffff8f600000

[Oza Oza]

[-- Attachment #1: Type: text/plain, Size: 299 bytes --]

Hi,



Why do I get invalid user mode virtual address when I enabled DEBUG in
SPDK/DPDK ?



invalid usermode virtual address 0xffff8f600000

invalid usermode virtual address 0xffff0f800000

invalid usermode virtual address 0xffff0f400000



this is armv8.



Regards,

Oza.

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 1589 bytes --]