On Mon, 2017-01-30 at 10:20 +1300, Dale Corlett wrote: > Hi, > > I have managed to get the Netlist card initialised with SPDK and map > the registers, I now need to read and write to it. > The Netlist card uses DMA to read/write so I think I will need to > allocate DMA memory. Is there a way to do this through SPDK? Are > there SPDK functions equivalent to the DMA API functions > like dma_alloc_coherent()? See spdk_malloc, spdk_zmalloc, spdk_realloc, and spdk_free. > > Thanks, > Dale > > > On Wed, Jan 11, 2017 at 1:38 PM, Dale Corlett m> wrote: > > Hi, > > > > For the Netlist SPDK PCI driver that I am making, does it need to > > use VFIO, or UIO? If so, how are these configured in a userspace > > driver? > > > > I also have not got the DPDK debug messages to show yet. > > > > Thanks, > > Dale > > > > On Tue, Jan 10, 2017 at 8:37 AM, Dale Corlett > com> wrote: > > > Hi Daniel, > > > > > > Thanks for your reply. > > > > > > I have set the log level using rte_set_log_level(RTE_LOG_DEBUG) > > > but the debug messages still do not display. I > > > used rte_get_log_level() to check that the log level was set to > > > DEBUG, it returned 8 which corresponds to the RTE_LOG_DEBUG > > > macro. So it seems like the log level is set to debug, but the > > > debug messages do not show in the terminal. > > > > > > Do the EAL messages get written anywhere else? Could I have > > > broken something when I set the EXTRA_CFLAGS environment > > > variable? > > > > > > > > > Thanks, > > > Dale > > > > > > On Tue, Jan 10, 2017 at 7:38 AM, Verkamp, Daniel > > intel.com> wrote: > > > > Hi Dale, > > > >   > > > > This is controlled by the log level; the SPDK example programs > > > > don’t currently expose this as a command-line parameter, but > > > > you can add a call to rte_set_log_level() early in the program > > > > to change it from the default: http://dpdk.org/doc/api/rte__log > > > > _8h.html > > > >   > > > > -- Daniel > > > >   > > > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Dale > > > > Corlett > > > > Sent: Saturday, January 7, 2017 3:49 PM > > > > > > > > To: Storage Performance Development Kit > > > > Subject: Re: [SPDK] Netlist storage card > > > > > > > > > > > >   > > > > Hi, > > > >   > > > > For an SPDK application I am trying to enable the EAL debug > > > > message such as: "RTE_LOG(DEBUG, EAL, "  Not managed by a > > > > supported kernel driver, skipped\n");". > > > > I have tried compiling DPDK for debug by setting the > > > > environment variable "EXTRA_CFLAGS" to "-O0 -g" as described > > > > here: http://dpdk.readthedocs.io/en/v16.07/prog_guide/dev_kit_r > > > > oot_make_help.html#compiling-for-debug, but this does not work. > > > >   > > > > Is there any other way to enable these EAL debug messages? > > > >   > > > > Thanks, > > > > Dale > > > >   > > > > On Mon, Dec 19, 2016 at 5:59 PM, Dileep Sharma > > > mple.com> wrote: > > > > Hi Dale, > > > >   > > > > SPDK uses the DPDK's PCI generic to get the information about > > > > the PCI devices. And DPDK in turn uses the uio driver to get > > > > the information about the underlying PCI devices. I'll suggest > > > > you to follow the code flow of rte_eal_pci_probe() routine in > > > > DPDK to get the feel how DPDK works for PCI devices. During > > > > code path flow, you'll came to know about various data > > > > structures that are similar to kernel code and then you can use > > > > them effectively in your code. > > > >   > > > > Thanks, > > > > Dileep > > > >   > > > > On Sun, Dec 18, 2016 at 9:27 AM, Dale Corlett > > > iad.com> wrote: > > > > Hi, > > > >   > > > > I have now moved on to creating a user space driver for the > > > > netlist storage card. I have written a simple kernel driver > > > > that uses the PCI driver that just gets the pcie register > > > > information from the netlist card. I want to port this kernel > > > > driver to user space to work with SPDK, but I am finding it > > > > difficult to workout the steps required to do this. I have > > > > noticed that some structures like spdk_pci_id, > > > > and spdk_pci_device which are defined in SPDK are similar to > > > > structures that I have used in my kernel driver: pci_device_id, > > > > and pci_dev. However, SPDK does not seem to have structures > > > > equivalent to pci_driver which I have used in my kernel > > > > driver.  > > > > How does SPDK link the PCI driver to the user space driver? > > > > Any assistance would be greatly appreciated. > > > >   > > > > My simple kernel driver for the Netlist card: > > > > #include > > > > #include > > > > #include > > > >   > > > >   > > > > #define EV_VENDOR_ID 0x1C1B    // Netlist > > > > #define EV_DEVICE_ID_BAR32_WINDOW_32M_16GB 0x0006 // > > > > EXPRESSvault NVDIMM/DDR3 PCIe GEN3 x8 lane -        For debug > > > > use only > > > >   > > > > #define PCI_DRIVER_NAME "Netlistev3" > > > >   > > > > // Function prototypes: > > > > static int netlist_pci_probe(struct pci_dev *dev, const struct > > > > pci_device_id *id); > > > >   > > > > //PCI_DEVICE(EV_VENDOR_ID, EV_DEVICE_ID_BAR32_WINDOW_32M_16GB); > > > > static const struct pci_device_id netlist_pci_ids [] =  > > > > { > > > > { > > > > .vendor = EV_VENDOR_ID, > > > > .device = EV_DEVICE_ID_BAR32_WINDOW_32M_16GB, > > > > .subvendor = PCI_ANY_ID, > > > > .subdevice = PCI_ANY_ID, > > > > }, > > > > {  > > > > // end: all zeroes > > > > .vendor = 0, > > > > .device = 0, > > > > .subvendor = 0, > > > > .subdevice = 0, > > > > } > > > > }; > > > >   > > > > static struct pci_driver netlist_pci_driver =  > > > > { > > > > .name           = PCI_DRIVER_NAME, > > > > .id_table       = netlist_pci_ids, > > > > .probe          = netlist_pci_probe, > > > > // .remove         = netlist_pci_remove, > > > > // .err_handler    = &pci_err_handler, // This is a structure > > > > }; > > > >   > > > > static int netlist_pci_probe(struct pci_dev *dev, const struct > > > > pci_device_id *id) > > > > { > > > > printk("pci_enable_device\n"); > > > > if (pci_enable_device(dev) < 0) > > > > { > > > > printk("pci_enable_device FAILED\n"); > > > > return -ENODEV; > > > > } > > > > unsigned int venid; > > > > int offset; > > > > for(offset = 0x00; offset <= 0x03C; offset += 4) > > > > { > > > > int retVal = pci_read_config_dword(dev, offset, &venid); > > > > printk ("byte offset 0x%X = 0x%X\n", offset, venid); > > > > } > > > > return 0; > > > > } > > > >   > > > >   > > > > static int netlist_init(void) > > > > { > > > > int err = pci_register_driver(&netlist_pci_driver); > > > >      > > > >     if(err < 0) > > > >     { > > > >     printk("error registering netlist_pci_driver\n"); > > > >     } > > > >     else if(err == 0) > > > >     { > > > >     printk("netlist_pci_driver registered successfully\n"); > > > >     } > > > >     return err; > > > > } > > > >   > > > > static void netlist_cleanup(void) > > > > { > > > > pci_unregister_driver(&netlist_pci_driver); > > > > printk("netlist_pci_driver unregistered\n"); > > > > } > > > >   > > > > module_init(netlist_init); > > > > module_exit(netlist_cleanup); > > > >   > > > > Thanks, > > > > Dale > > > >   > > > > On Fri, Dec 2, 2016 at 10:55 AM, Dale Corlett > > > iad.com> wrote: > > > > Hi, > > > >   > > > > I have the iscsi_tgt program compiled but I do not know how to > > > > setup the iscsi.conf file. I have removed all of what I think > > > > is unnecessary for my application from the iscsi.conf.in > > > > example so I have the following: > > > >   > > > > [iSCSI] > > > >   # node name (not include optional part) > > > >   # Users can optionally change this to fit their environment. > > > >   NodeBase "iqn.2016-06.io.spdk" > > > >   > > > >   AuthFile /usr/local/etc/spdk/auth.conf > > > >   > > > >   MinConnectionsPerCore 4 > > > >   # Power saving related variable, this parameter defines how > > > > long an iSCSI > > > >   # connection must be idle before moving it to a state where > > > > it will consume > > > >   # less power. This variable is defined in terms of > > > > microseconds. We set default > > > >   # value as 5ms. > > > >   MinConnectionIdleInterval 5000 > > > >   > > > >   # Socket I/O timeout sec. (0 is infinite) > > > >   Timeout 30 > > > >   > > > >   # authentication information for discovery session > > > >   DiscoveryAuthMethod Auto > > > >   > > > >   #MaxSessions 128 > > > >   #MaxConnectionsPerSession 2 > > > >   > > > >   # iSCSI initial parameters negotiate with initiators > > > >   # NOTE: incorrect values might crash > > > >   DefaultTime2Wait 2 > > > >   DefaultTime2Retain 60 > > > >   > > > >   ImmediateData Yes > > > >   ErrorRecoveryLevel 0 > > > >   > > > > [AIO] > > > > AIO /dev/ev3mema > > > >   > > > > [TargetNode1] > > > >  TargetName disk1 > > > >   TargetAlias "Data Disk1" > > > >   Mapping PortalGroup1 InitiatorGroup1 > > > >   AuthMethod Auto > > > >   AuthGroup AuthGroup1 > > > >   # Enable header and data digest > > > >   # UseDigest Header Data > > > >   UseDigest Auto > > > > # Using the first AIO target > > > >   LUN0 AIO0 > > > >   > > > > But when I run the iscsi_tgt program like this: sudo > > > > ./iscsi_tgt -c ../../etc/spdk/iscsi.conf I get the following > > > > output: > > > >   > > > > Starting Intel(R) DPDK initialization ...  > > > > [ DPDK EAL parameters: iscsi -c 1 -n 4 -m 2048 --master-lcore=0 > > > > --file-prefix=rte0 --proc-type=auto ] > > > > EAL: Detected 4 lcore(s) > > > > EAL: Auto-detected process type: PRIMARY > > > > EAL: No free hugepages reported in hugepages-1048576kB > > > > EAL: Probing VFIO support... > > > > done. > > > > Occupied cpu core mask is 0x1 > > > > Occupied cpu socket mask is 0x1 > > > > Ioat Copy Engine Offload Enabled > > > > tgt_node.c: 590:spdk_iscsi_tgt_node_add_map: ***ERROR*** > > > > iqn.2016-06.io.spdk:disk1: PortalGroup1 not found > > > > tgt_node.c: 727:spdk_iscsi_tgt_node_construct: ***ERROR*** > > > > could not add map to target > > > > tgt_node.c: 982:spdk_cf_add_iscsi_tgt_node: ***ERROR*** > > > > tgt_node1: add_iscsi_target_node error > > > > tgt_node.c:1006:spdk_iscsi_init_tgt_nodes: ***ERROR*** > > > > spdk_cf_add_iscsi_tgt_node() failed > > > > iscsi_subsystem.c: 964:spdk_iscsi_subsystem_init: ***ERROR*** > > > > spdk_iscsi_init_tgt_nodes() failed > > > > app.c: 404:spdk_app_init: ***ERROR*** spdk_subsystem_init() > > > > failed > > > >   > > > > I just want the most basic configuration so that I can just see > > > > if the Netlist card works with SPDK. > > > >   > > > > Thanks, > > > > Dale > > > >   > > > > On Thu, Dec 1, 2016 at 5:39 PM, Dale Corlett > > > ad.com> wrote: > > > > Hi Param, > > > >   > > > > Awesome it worked! > > > >   > > > > Thanks, > > > > Dale > > > >   > > > > On Thu, Dec 1, 2016 at 5:06 PM, Kumaraparameshwaran Rathnavel < > > > > krath(a)cloudsimple.com> wrote: > > > > Hi Dale, > > > >   > > > > When you compile your SPDK application try giving > > > > DPDK_DIR=/path/to/dpdk/x86_build folder. This should solve your > > > > problem. From SPDK directory give make and the DPDK_DIR. > > > >   > > > > Regards, > > > > Param. > > > >   > > > > On 01-Dec-2016, at 9:23 AM, Dale Corlett > > > om> wrote: > > > >   > > > > Hello again, > > > >   > > > > When I was trying to run iscsi_tgt (as instructed in http://www > > > > .spdk.io/spdk/doc/iscsi_getting_started.html) I found that the > > > > app/iscsi_tgt.c was not compiled, and when I tried to compile > > > > it I got the following error: > > > > iscsi_tgt.c:38:24: fatal error: rte_config.h: No such file or > > > > directory > > > > From an online search I have seen that many other people get a > > > > similar error to do with missing the rte_config.h file, but > > > > there do not seem to be any resolutions. > > > > I have found a rte_config.h file in one of the dpdk > > > > directories. I tried putting this in the /usr/include > > > > directory, but this did not fix it. > > > >   > > > > Does anyone have any ideas of how to fix this? Has anyone > > > > encountered this problem before? > > > >   > > > > Thanks, > > > > Dale > > > >   > > > >   > > > >   > > > > On Thu, Dec 1, 2016 at 5:46 AM, Walker, Benjamin > > > er(a)intel.com> wrote: > > > > Note also that running the iSCSI target will not create a block > > > > device on your local system by itself. It’s just a C executable > > > > that’s opening up some listening sockets. To see a block device > > > > on a system, you have to connect to the target using an iSCSI > > > > initiator. The one we use is iscsiadm, which is not part of > > > > SPDK but is in wide use and is packaged on all mainstream Linux > > > > distributions. This is the same iSCSI initiator people use with > > > > the Linux kernel iSCSI target. > > > >   > > > > All of the above is covered in that guide you linked. > > > >   > > > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of > > > > Harris, James R > > > > Sent: Tuesday, November 29, 2016 6:07 PM > > > > > > > > To: Storage Performance Development Kit > > > > Subject: Re: [SPDK] Netlist storage card > > > >   > > > >   > > > > Hi Dale, > > > >   > > > > You can pass the name of your config file to the iscsi_tgt app > > > > using the –c option. > > > >   > > > > -Jim > > > >   > > > >   > > > > From: SPDK on behalf of Dale > > > > Corlett > > > > Reply-To: Storage Performance Development Kit > > > g> > > > > Date: Tuesday, November 29, 2016 at 3:38 PM > > > > To: Storage Performance Development Kit > > > > Subject: Re: [SPDK] Netlist storage card > > > >   > > > > Hi, > > > >   > > > > I have added the [AIO] section to the config > > > > file: /home/dale/spdk/etc/spdk/iscsi.conf.in but I cannot > > > > figure out how to "run" SPDK to create the SPDK block device > > > > called AIO0.  > > > >   > > > > I have tried following the guid for the iscsi block device: htt > > > > p://www.spdk.io/spdk/doc/iscsi_getting_started.html but I am > > > > not sure if I have used the correct config file. > > > >   > > > > Thanks, > > > > Dale > > > >   > > > > On Wed, Nov 30, 2016 at 10:41 AM, Daniel Verkamp > > > p(a)intel.com> wrote: > > > > On 11/29/2016 02:15 PM, Dale Corlett wrote: > > > > > Hi, > > > > > > > > > > Thank you all for your replies. > > > > > > > > > > Jim, I think that I will first try the AIO first, and then > > > > try > > > > > writing the userspace driver and bdev module. I am not sure > > > > of what I > > > > > should do with the config files, or how to get the .ko file > > > > for the > > > > > AIO module. Is there a makefile that gets the .ko module file > > > > or do I > > > > > have to create one? > > > > > > > > > > Also, when trying to use ./scripts/setup.sh I found that it > > > > did not > > > > > work because I got then message: "logname: no login name". I > > > > think > > > > > that this is a problem with the 16.04 version of Ubuntu: > > > > > https://bugs.launchpad.net/ubuntu/+source/gnome-terminal/+bug > > > > /1537645 > > > > > so I changed line 177 to: username= `echo whoami` and this > > > > seems to > > > > > work. > > > > > > > > > > Thanks, Dale > > > > > > > > Hi Dale, > > > > > > > > You do not need any special kernel modules to use the SPDK AIO > > > > bdev - > > > > it uses the libaio userspace library provided by glibc and the > > > > block > > > > devices that are exposed by the usual kernel drivers for any > > > > block > > > > device that is supported by Linux. > > > > > > > > Thanks for the report about logname - I'll prepare a patch to > > > > work > > > > around that issue. The only time the username provided by > > > > logname is > > > > used is to provide access to an unprivileged user to the VFIO > > > > device, > > > > so if you are running the programs using SPDK as root, there > > > > should have > > > > no negative effect. > > > > > > > > Thanks, > > > > -- Daniel > > > > _______________________________________________ > > > > SPDK mailing list > > > > SPDK(a)lists.01.org > > > > https://lists.01.org/mailman/listinfo/spdk > > > >   > > > > > > > > _______________________________________________ > > > > SPDK mailing list > > > > SPDK(a)lists.01.org > > > > https://lists.01.org/mailman/listinfo/spdk > > > > > > > >   > > > > _______________________________________________ > > > > SPDK mailing list > > > > SPDK(a)lists.01.org > > > > https://lists.01.org/mailman/listinfo/spdk > > > >   > > > > > > > > _______________________________________________ > > > > SPDK mailing list > > > > SPDK(a)lists.01.org > > > > https://lists.01.org/mailman/listinfo/spdk > > > > > > > >   > > > >   > > > >   > > > > > > > > _______________________________________________ > > > > SPDK mailing list > > > > SPDK(a)lists.01.org > > > > https://lists.01.org/mailman/listinfo/spdk > > > > > > > > > > > > > > > >   > > > > -- > > > > Thanks, > > > > Dileep Sharma > > > > > > > > _______________________________________________ > > > > SPDK mailing list > > > > SPDK(a)lists.01.org > > > > https://lists.01.org/mailman/listinfo/spdk > > > > > > > >   > > > > > > > > _______________________________________________ > > > > SPDK mailing list > > > > SPDK(a)lists.01.org > > > > https://lists.01.org/mailman/listinfo/spdk > > > > > > > > > > > > > > > > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk >