Re: [SPDK] Netlist storage card

[Walker, Benjamin <benjamin.walker at intel.com>]

[-- Attachment #1: Type: text/plain, Size: 18399 bytes --]

On Mon, 2017-01-30 at 10:20 +1300, Dale Corlett wrote:
> Hi,
> 
> I have managed to get the Netlist card initialised with SPDK and map
> the registers, I now need to read and write to it.
> The Netlist card uses DMA to read/write so I think I will need to
> allocate DMA memory. Is there a way to do this through SPDK? Are
> there SPDK functions equivalent to the DMA API functions
> like dma_alloc_coherent()?

See spdk_malloc, spdk_zmalloc, spdk_realloc, and spdk_free.

> 
> Thanks,
> Dale
> 
> 
> On Wed, Jan 11, 2017 at 1:38 PM, Dale Corlett <dale.corlett(a)nyriad.co
> m> wrote:
> > Hi,
> > 
> > For the Netlist SPDK PCI driver that I am making, does it need to
> > use VFIO, or UIO? If so, how are these configured in a userspace
> > driver?
> > 
> > I also have not got the DPDK debug messages to show yet.
> > 
> > Thanks,
> > Dale
> > 
> > On Tue, Jan 10, 2017 at 8:37 AM, Dale Corlett <dale.corlett(a)nyriad.
> > com> wrote:
> > > Hi Daniel,
> > > 
> > > Thanks for your reply.
> > > 
> > > I have set the log level using rte_set_log_level(RTE_LOG_DEBUG)
> > > but the debug messages still do not display. I
> > > used rte_get_log_level() to check that the log level was set to
> > > DEBUG, it returned 8 which corresponds to the RTE_LOG_DEBUG
> > > macro. So it seems like the log level is set to debug, but the
> > > debug messages do not show in the terminal.
> > > 
> > > Do the EAL messages get written anywhere else? Could I have
> > > broken something when I set the EXTRA_CFLAGS environment
> > > variable?
> > > 
> > > 
> > > Thanks,
> > > Dale
> > > 
> > > On Tue, Jan 10, 2017 at 7:38 AM, Verkamp, Daniel <daniel.verkamp@
> > > intel.com> wrote:
> > > > Hi Dale,
> > > >  
> > > > This is controlled by the log level; the SPDK example programs
> > > > don’t currently expose this as a command-line parameter, but
> > > > you can add a call to rte_set_log_level() early in the program
> > > > to change it from the default: http://dpdk.org/doc/api/rte__log
> > > > _8h.html
> > > >  
> > > > -- Daniel
> > > >  
> > > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Dale
> > > > Corlett
> > > > Sent: Saturday, January 7, 2017 3:49 PM
> > > > 
> > > > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > > > Subject: Re: [SPDK] Netlist storage card
> > > > 
> > > > 
> > > >  
> > > > Hi,
> > > >  
> > > > For an SPDK application I am trying to enable the EAL debug
> > > > message such as: "RTE_LOG(DEBUG, EAL, "  Not managed by a
> > > > supported kernel driver, skipped\n");".
> > > > I have tried compiling DPDK for debug by setting the
> > > > environment variable "EXTRA_CFLAGS" to "-O0 -g" as described
> > > > here: http://dpdk.readthedocs.io/en/v16.07/prog_guide/dev_kit_r
> > > > oot_make_help.html#compiling-for-debug, but this does not work.
> > > >  
> > > > Is there any other way to enable these EAL debug messages?
> > > >  
> > > > Thanks,
> > > > Dale
> > > >  
> > > > On Mon, Dec 19, 2016 at 5:59 PM, Dileep Sharma <dsharma(a)cloudsi
> > > > mple.com> wrote:
> > > > Hi Dale,
> > > >  
> > > > SPDK uses the DPDK's PCI generic to get the information about
> > > > the PCI devices. And DPDK in turn uses the uio driver to get
> > > > the information about the underlying PCI devices. I'll suggest
> > > > you to follow the code flow of rte_eal_pci_probe() routine in
> > > > DPDK to get the feel how DPDK works for PCI devices. During
> > > > code path flow, you'll came to know about various data
> > > > structures that are similar to kernel code and then you can use
> > > > them effectively in your code.
> > > >  
> > > > Thanks,
> > > > Dileep
> > > >  
> > > > On Sun, Dec 18, 2016 at 9:27 AM, Dale Corlett <dale.corlett(a)nyr
> > > > iad.com> wrote:
> > > > Hi,
> > > >  
> > > > I have now moved on to creating a user space driver for the
> > > > netlist storage card. I have written a simple kernel driver
> > > > that uses the PCI driver that just gets the pcie register
> > > > information from the netlist card. I want to port this kernel
> > > > driver to user space to work with SPDK, but I am finding it
> > > > difficult to workout the steps required to do this. I have
> > > > noticed that some structures like spdk_pci_id,
> > > > and spdk_pci_device which are defined in SPDK are similar to
> > > > structures that I have used in my kernel driver: pci_device_id,
> > > > and pci_dev. However, SPDK does not seem to have structures
> > > > equivalent to pci_driver which I have used in my kernel
> > > > driver. 
> > > > How does SPDK link the PCI driver to the user space driver?
> > > > Any assistance would be greatly appreciated.
> > > >  
> > > > My simple kernel driver for the Netlist card:
> > > > #include <linux/pci.h>
> > > > #include <linux/init.h>
> > > > #include <linux/module.h>
> > > >  
> > > >  
> > > > #define EV_VENDOR_ID 0x1C1B    // Netlist
> > > > #define EV_DEVICE_ID_BAR32_WINDOW_32M_16GB 0x0006 //
> > > > EXPRESSvault NVDIMM/DDR3 PCIe GEN3 x8 lane -        For debug
> > > > use only
> > > >  
> > > > #define PCI_DRIVER_NAME "Netlistev3"
> > > >  
> > > > // Function prototypes:
> > > > static int netlist_pci_probe(struct pci_dev *dev, const struct
> > > > pci_device_id *id);
> > > >  
> > > > //PCI_DEVICE(EV_VENDOR_ID, EV_DEVICE_ID_BAR32_WINDOW_32M_16GB);
> > > > static const struct pci_device_id netlist_pci_ids [] = 
> > > > {
> > > > {
> > > > .vendor = EV_VENDOR_ID,
> > > > .device = EV_DEVICE_ID_BAR32_WINDOW_32M_16GB,
> > > > .subvendor = PCI_ANY_ID,
> > > > .subdevice = PCI_ANY_ID,
> > > > },
> > > > { 
> > > > // end: all zeroes
> > > > .vendor = 0,
> > > > .device = 0,
> > > > .subvendor = 0,
> > > > .subdevice = 0,
> > > > }
> > > > };
> > > >  
> > > > static struct pci_driver netlist_pci_driver = 
> > > > {
> > > > .name           = PCI_DRIVER_NAME,
> > > > .id_table       = netlist_pci_ids,
> > > > .probe          = netlist_pci_probe,
> > > > // .remove         = netlist_pci_remove,
> > > > // .err_handler    = &pci_err_handler, // This is a structure
> > > > };
> > > >  
> > > > static int netlist_pci_probe(struct pci_dev *dev, const struct
> > > > pci_device_id *id)
> > > > {
> > > > printk("pci_enable_device\n");
> > > > if (pci_enable_device(dev) < 0)
> > > > {
> > > > printk("pci_enable_device FAILED\n");
> > > > return -ENODEV;
> > > > }
> > > > unsigned int venid;
> > > > int offset;
> > > > for(offset = 0x00; offset <= 0x03C; offset += 4)
> > > > {
> > > > int retVal = pci_read_config_dword(dev, offset, &venid);
> > > > printk ("byte offset 0x%X = 0x%X\n", offset, venid);
> > > > }
> > > > return 0;
> > > > }
> > > >  
> > > >  
> > > > static int netlist_init(void)
> > > > {
> > > > int err = pci_register_driver(&netlist_pci_driver);
> > > >     
> > > >     if(err < 0)
> > > >     {
> > > >     printk("error registering netlist_pci_driver\n");
> > > >     }
> > > >     else if(err == 0)
> > > >     {
> > > >     printk("netlist_pci_driver registered successfully\n");
> > > >     }
> > > >     return err;
> > > > }
> > > >  
> > > > static void netlist_cleanup(void)
> > > > {
> > > > pci_unregister_driver(&netlist_pci_driver);
> > > > printk("netlist_pci_driver unregistered\n");
> > > > }
> > > >  
> > > > module_init(netlist_init);
> > > > module_exit(netlist_cleanup);
> > > >  
> > > > Thanks,
> > > > Dale
> > > >  
> > > > On Fri, Dec 2, 2016 at 10:55 AM, Dale Corlett <dale.corlett(a)nyr
> > > > iad.com> wrote:
> > > > Hi,
> > > >  
> > > > I have the iscsi_tgt program compiled but I do not know how to
> > > > setup the iscsi.conf file. I have removed all of what I think
> > > > is unnecessary for my application from the iscsi.conf.in
> > > > example so I have the following:
> > > >  
> > > > [iSCSI]
> > > >   # node name (not include optional part)
> > > >   # Users can optionally change this to fit their environment.
> > > >   NodeBase "iqn.2016-06.io.spdk"
> > > >  
> > > >   AuthFile /usr/local/etc/spdk/auth.conf
> > > >  
> > > >   MinConnectionsPerCore 4
> > > >   # Power saving related variable, this parameter defines how
> > > > long an iSCSI
> > > >   # connection must be idle before moving it to a state where
> > > > it will consume
> > > >   # less power. This variable is defined in terms of
> > > > microseconds. We set default
> > > >   # value as 5ms.
> > > >   MinConnectionIdleInterval 5000
> > > >  
> > > >   # Socket I/O timeout sec. (0 is infinite)
> > > >   Timeout 30
> > > >  
> > > >   # authentication information for discovery session
> > > >   DiscoveryAuthMethod Auto
> > > >  
> > > >   #MaxSessions 128
> > > >   #MaxConnectionsPerSession 2
> > > >  
> > > >   # iSCSI initial parameters negotiate with initiators
> > > >   # NOTE: incorrect values might crash
> > > >   DefaultTime2Wait 2
> > > >   DefaultTime2Retain 60
> > > >  
> > > >   ImmediateData Yes
> > > >   ErrorRecoveryLevel 0
> > > >  
> > > > [AIO]
> > > > AIO /dev/ev3mema
> > > >  
> > > > [TargetNode1]
> > > >  TargetName disk1
> > > >   TargetAlias "Data Disk1"
> > > >   Mapping PortalGroup1 InitiatorGroup1
> > > >   AuthMethod Auto
> > > >   AuthGroup AuthGroup1
> > > >   # Enable header and data digest
> > > >   # UseDigest Header Data
> > > >   UseDigest Auto
> > > > # Using the first AIO target
> > > >   LUN0 AIO0
> > > >  
> > > > But when I run the iscsi_tgt program like this: sudo
> > > > ./iscsi_tgt -c ../../etc/spdk/iscsi.conf I get the following
> > > > output:
> > > >  
> > > > Starting Intel(R) DPDK initialization ... 
> > > > [ DPDK EAL parameters: iscsi -c 1 -n 4 -m 2048 --master-lcore=0 
> > > > --file-prefix=rte0 --proc-type=auto ]
> > > > EAL: Detected 4 lcore(s)
> > > > EAL: Auto-detected process type: PRIMARY
> > > > EAL: No free hugepages reported in hugepages-1048576kB
> > > > EAL: Probing VFIO support...
> > > > done.
> > > > Occupied cpu core mask is 0x1
> > > > Occupied cpu socket mask is 0x1
> > > > Ioat Copy Engine Offload Enabled
> > > > tgt_node.c: 590:spdk_iscsi_tgt_node_add_map: ***ERROR***
> > > > iqn.2016-06.io.spdk:disk1: PortalGroup1 not found
> > > > tgt_node.c: 727:spdk_iscsi_tgt_node_construct: ***ERROR***
> > > > could not add map to target
> > > > tgt_node.c: 982:spdk_cf_add_iscsi_tgt_node: ***ERROR***
> > > > tgt_node1: add_iscsi_target_node error
> > > > tgt_node.c:1006:spdk_iscsi_init_tgt_nodes: ***ERROR***
> > > > spdk_cf_add_iscsi_tgt_node() failed
> > > > iscsi_subsystem.c: 964:spdk_iscsi_subsystem_init: ***ERROR***
> > > > spdk_iscsi_init_tgt_nodes() failed
> > > > app.c: 404:spdk_app_init: ***ERROR*** spdk_subsystem_init()
> > > > failed
> > > >  
> > > > I just want the most basic configuration so that I can just see
> > > > if the Netlist card works with SPDK.
> > > >  
> > > > Thanks,
> > > > Dale
> > > >  
> > > > On Thu, Dec 1, 2016 at 5:39 PM, Dale Corlett <dale.corlett(a)nyri
> > > > ad.com> wrote:
> > > > Hi Param,
> > > >  
> > > > Awesome it worked!
> > > >  
> > > > Thanks,
> > > > Dale
> > > >  
> > > > On Thu, Dec 1, 2016 at 5:06 PM, Kumaraparameshwaran Rathnavel <
> > > > krath(a)cloudsimple.com> wrote:
> > > > Hi Dale,
> > > >  
> > > > When you compile your SPDK application try giving
> > > > DPDK_DIR=/path/to/dpdk/x86_build folder. This should solve your
> > > > problem. From SPDK directory give make and the DPDK_DIR.
> > > >  
> > > > Regards,
> > > > Param.
> > > >  
> > > > On 01-Dec-2016, at 9:23 AM, Dale Corlett <dale.corlett(a)nyriad.c
> > > > om> wrote:
> > > >  
> > > > Hello again,
> > > >  
> > > > When I was trying to run iscsi_tgt (as instructed in http://www
> > > > .spdk.io/spdk/doc/iscsi_getting_started.html) I found that the
> > > > app/iscsi_tgt.c was not compiled, and when I tried to compile
> > > > it I got the following error:
> > > > iscsi_tgt.c:38:24: fatal error: rte_config.h: No such file or
> > > > directory
> > > > From an online search I have seen that many other people get a
> > > > similar error to do with missing the rte_config.h file, but
> > > > there do not seem to be any resolutions.
> > > > I have found a rte_config.h file in one of the dpdk
> > > > directories. I tried putting this in the /usr/include
> > > > directory, but this did not fix it.
> > > >  
> > > > Does anyone have any ideas of how to fix this? Has anyone
> > > > encountered this problem before?
> > > >  
> > > > Thanks,
> > > > Dale
> > > >  
> > > >  
> > > >  
> > > > On Thu, Dec 1, 2016 at 5:46 AM, Walker, Benjamin <benjamin.walk
> > > > er(a)intel.com> wrote:
> > > > Note also that running the iSCSI target will not create a block
> > > > device on your local system by itself. It’s just a C executable
> > > > that’s opening up some listening sockets. To see a block device
> > > > on a system, you have to connect to the target using an iSCSI
> > > > initiator. The one we use is iscsiadm, which is not part of
> > > > SPDK but is in wide use and is packaged on all mainstream Linux
> > > > distributions. This is the same iSCSI initiator people use with
> > > > the Linux kernel iSCSI target.
> > > >  
> > > > All of the above is covered in that guide you linked.
> > > >  
> > > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of
> > > > Harris, James R
> > > > Sent: Tuesday, November 29, 2016 6:07 PM
> > > > 
> > > > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > > > Subject: Re: [SPDK] Netlist storage card
> > > >  
> > > >  
> > > > Hi Dale,
> > > >  
> > > > You can pass the name of your config file to the iscsi_tgt app
> > > > using the –c option.
> > > >  
> > > > -Jim
> > > >  
> > > >  
> > > > From: SPDK <spdk-bounces(a)lists.01.org> on behalf of Dale
> > > > Corlett <dale.corlett(a)nyriad.com>
> > > > Reply-To: Storage Performance Development Kit <spdk(a)lists.01.or
> > > > g>
> > > > Date: Tuesday, November 29, 2016 at 3:38 PM
> > > > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > > > Subject: Re: [SPDK] Netlist storage card
> > > >  
> > > > Hi,
> > > >  
> > > > I have added the [AIO] section to the config
> > > > file: /home/dale/spdk/etc/spdk/iscsi.conf.in but I cannot
> > > > figure out how to "run" SPDK to create the SPDK block device
> > > > called AIO0. 
> > > >  
> > > > I have tried following the guid for the iscsi block device: htt
> > > > p://www.spdk.io/spdk/doc/iscsi_getting_started.html but I am
> > > > not sure if I have used the correct config file.
> > > >  
> > > > Thanks,
> > > > Dale
> > > >  
> > > > On Wed, Nov 30, 2016 at 10:41 AM, Daniel Verkamp <daniel.verkam
> > > > p(a)intel.com> wrote:
> > > > On 11/29/2016 02:15 PM, Dale Corlett wrote:
> > > > > Hi,
> > > > >
> > > > > Thank you all for your replies.
> > > > >
> > > > > Jim, I think that I will first try the AIO first, and then
> > > > try
> > > > > writing the userspace driver and bdev module. I am not sure
> > > > of what I
> > > > > should do with the config files, or how to get the .ko file
> > > > for the
> > > > > AIO module. Is there a makefile that gets the .ko module file
> > > > or do I
> > > > > have to create one?
> > > > >
> > > > > Also, when trying to use ./scripts/setup.sh I found that it
> > > > did not
> > > > > work because I got then message: "logname: no login name". I
> > > > think
> > > > > that this is a problem with the 16.04 version of Ubuntu:
> > > > > https://bugs.launchpad.net/ubuntu/+source/gnome-terminal/+bug
> > > > /1537645
> > > > > so I changed line 177 to: username= `echo whoami` and this
> > > > seems to
> > > > > work.
> > > > >
> > > > > Thanks, Dale
> > > > 
> > > > Hi Dale,
> > > > 
> > > > You do not need any special kernel modules to use the SPDK AIO
> > > > bdev -
> > > > it uses the libaio userspace library provided by glibc and the
> > > > block
> > > > devices that are exposed by the usual kernel drivers for any
> > > > block
> > > > device that is supported by Linux.
> > > > 
> > > > Thanks for the report about logname - I'll prepare a patch to
> > > > work
> > > > around that issue. The only time the username provided by
> > > > logname is
> > > > used is to provide access to an unprivileged user to the VFIO
> > > > device,
> > > > so if you are running the programs using SPDK as root, there
> > > > should have
> > > > no negative effect.
> > > > 
> > > > Thanks,
> > > > -- Daniel
> > > > _______________________________________________
> > > > SPDK mailing list
> > > > SPDK(a)lists.01.org
> > > > https://lists.01.org/mailman/listinfo/spdk
> > > >  
> > > > 
> > > > _______________________________________________
> > > > SPDK mailing list
> > > > SPDK(a)lists.01.org
> > > > https://lists.01.org/mailman/listinfo/spdk
> > > > 
> > > >  
> > > > _______________________________________________
> > > > SPDK mailing list
> > > > SPDK(a)lists.01.org
> > > > https://lists.01.org/mailman/listinfo/spdk
> > > >  
> > > > 
> > > > _______________________________________________
> > > > SPDK mailing list
> > > > SPDK(a)lists.01.org
> > > > https://lists.01.org/mailman/listinfo/spdk
> > > > 
> > > >  
> > > >  
> > > >  
> > > > 
> > > > _______________________________________________
> > > > SPDK mailing list
> > > > SPDK(a)lists.01.org
> > > > https://lists.01.org/mailman/listinfo/spdk
> > > > 
> > > > 
> > > > 
> > > >  
> > > > --
> > > > Thanks,
> > > > Dileep Sharma
> > > > 
> > > > _______________________________________________
> > > > SPDK mailing list
> > > > SPDK(a)lists.01.org
> > > > https://lists.01.org/mailman/listinfo/spdk
> > > > 
> > > >  
> > > > 
> > > > _______________________________________________
> > > > SPDK mailing list
> > > > SPDK(a)lists.01.org
> > > > https://lists.01.org/mailman/listinfo/spdk
> > > > 
> > > 
> > > 
> > 
> > 
> 
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
> 

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 3274 bytes --]