FW: About the patch ”https://lore.kernel.org/linux-iommu/20240412082121.33382-1-yi.l.liu@intel.com/ “ for help

FW: About the patch ”https://lore.kernel.org/linux-iommu/20240412082121.33382-1-yi.l.liu@intel.com/ “ for help

[XueMei Yue]

[-- Attachment #1.1.1: Type: text/plain, Size: 1434 bytes --]

DEAR ALL ,

No  I  have know the root cause ,  the issue  ouccured when below code run my AMD test PC.

So  could you guys  give  some suggestion ?  Will very much appreciate if you feel free to reply！！

[cid:image001.png@01DADCEE.6F2E3F80]

From: XueMei Yue
Sent: 2024年7月22日 9:50
To: yi.l.liu@intel.com
Subject: About the patch ”https://lore.kernel.org/linux-iommu/20240412082121.33382-1-yi.l.liu@intel.com/ “ for help


hi,

I am an firmware engineer from china . I want to test the function ATS that need the support of “iommufd+pasid”

Now I will test some new function ，will use pasid +  iommufd，

i used your patch from  https://lore.kernel.org/linux-iommu/20240412082121.33382-1-yi.l.liu@intel.com/

my test step is

1  use spdk ，  run   ./scripts/setup,  the nvme device drive is changed from nvme to vfio-pci

2  run my example ，see the attchment ，using g++ to compile

when run my  example . the below error occurs ，could you give some suggestion ？
[cid:image004.png@01DADCEE.6F6C59F0][[ root@hippo ./iommufd.o  Program ./iommufd.o  the map va is : 7fe866435OOO, the iova is O, the size is 1048576  VFIO DEVICE PASID ATTACH IOMMUFD PT failed! error is operation not supported  [rooQ@hippo xuemei]#]
Will very much appreciate if you feel free to reply！！

Best regards
XueMei


[-- Attachment #1.1.2: Type: text/html, Size: 12337 bytes --]

[-- Attachment #1.2: image002.png --]
[-- Type: image/png, Size: 8499 bytes --]

[-- Attachment #1.3: image001.png --]
[-- Type: image/png, Size: 37770 bytes --]

[-- Attachment #1.4: image004.png --]
[-- Type: image/png, Size: 177 bytes --]

[-- Attachment #2: iommufd0716.cpp --]
[-- Type: text/plain, Size: 5775 bytes --]

#include <stdio.h>
#include <unistd.h>
#include <stdint.h>
#include <stdlib.h>
#include <linux/vfio.h>
#include <inttypes.h>
#include <string.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/ioctl.h>
#include <sys/mman.h>
#include <string>
#include <iostream>
#include <cstring>
#include <linux/iommufd.h>
 
struct args
{
	__u64	iova;				/* IO virtual address */
	__u64	size;
	std::string vfioX;
	std::string bdf;

};
int main(int argc, char *argv[]) 
{
    int ret = 0;
	int cdev_fd, iommufd, i;
	
    std::cout << "Program name: " << argv[0] << std::endl;
		struct args para={0};
    for (int i = 1; i < argc; ++i) 
	{
	
        std::cout << "Argument " << i << ": " << argv[i] << std::endl;
		if(!std::strcmp(argv[i], "-help"))
		{
		    printf("-iova:         input the map address iova\n");
			printf("-size:         input the Number of bytes to copy and map\n");
			printf("-device:       input the bdf\n");
			printf("-vfio  :       input the vfioX\n");
			return 0;	
		}
		if(!std::strcmp(argv[i], "-iova"))
		{
			i++;
			char *address = argv[i];
			sscanf(address, "%" SCNx64, &para.iova);
			printf("The address is: %" PRIx64 "\n", para.iova);
		}
		if(!std::strcmp(argv[i], "-size"))
		{
		    i++;
			para.size = atoi(argv[i]);
            std::cout << "the size is "<< para.size << std::endl;
		}
		if(!std::strcmp(argv[i], "-vfio"))
		{
		  	i++;
			para.vfioX = argv[i];
            std::cout << "vfioX is "<< para.vfioX << std::endl;

		}
        if(!std::strcmp(argv[i] ,  "-device"))
		{
		    i++;
			char *input_string = argv[i];
			para.bdf = input_string ;
		}
			
    }
	
	
	struct vfio_device_bind_iommufd bind = {
        .argsz = sizeof(bind),
        .flags = 0,
    };
     struct iommu_ioas_alloc alloc_data  = {
        .size = sizeof(alloc_data),
        .flags = 0,
     };
     struct vfio_device_attach_iommufd_pt attach_data = {
        .argsz = sizeof(attach_data),
        .flags = 0,
     };
    struct iommu_ioas_map map = {
        .size = sizeof(map),
        .flags = IOMMU_IOAS_MAP_READABLE |
                 IOMMU_IOAS_MAP_WRITEABLE |
                 IOMMU_IOAS_MAP_FIXED_IOVA,
        .__reserved = 0,
    };
	struct iommu_ioas_unmap unmap = {
		.size = sizeof(unmap),
	};
	struct vfio_device_pasid_attach_iommufd_pt pasid = {
		.argsz = sizeof(pasid),
        .flags = 0,
	};
	
	struct vfio_device_pasid_detach_iommufd_pt detach_pasid = {
		.argsz = sizeof(detach_pasid),
        .flags = 0,
	};


    /* Open the group */
    std::string vfio;
    if(!para.vfioX.empty())
    {
	    vfio = "/dev/vfio/devices/" + para.vfioX;
    }
    else 
    {
        vfio = "/dev/vfio/devices/vfio0";
    }
    cdev_fd = open(vfio.c_str(), O_RDWR);
    iommufd = open("/dev/iommu", O_RDWR);

    if (cdev_fd < 0 || iommufd <0) 
    {
	    /* if file not found, it's not an error */
	    if (errno != ENOENT) 
		{
	        std::cout << "Cannot open "<< iommufd  << " or  "<< cdev_fd <<" , error is : " <<  strerror(errno);
	        return -1;
	    } 
    }
    bind.iommufd = iommufd;

    if(ioctl(cdev_fd, VFIO_DEVICE_BIND_IOMMUFD, &bind))
    {
	std::cout << "VFIO_DEVICE_BIND_IOMMUFD failed! "<<strerror(errno)<<std::endl;
        return 1;
    }

    if(ioctl(iommufd, IOMMU_IOAS_ALLOC, &alloc_data))
    {
        std::cout << "IOMMU_IOAS_ALLOC failed! "<< strerror(errno) << std::endl ;
        return 1;
    }
	
    attach_data.pt_id = alloc_data.out_ioas_id;
    if(ioctl(cdev_fd, VFIO_DEVICE_ATTACH_IOMMUFD_PT, &attach_data))
    {
	std::cout << "VFIO_DEVICE_ATTACH_IOMMUFD_PT failed! "<< std::endl;
        return 1;
    }

    /* Allocate some space and setup a DMA mapping */
	
	map.length = 1024 * 1024;
    map.iova = 0; /* 1MB starting at 0x0 from device view */
	if(para.size)
    {
	    map.length = para.size; 
    }
    if(para.iova!= map.iova)
    {
	    map.iova = para.iova;  /* 1MB starting at 0x0 from device view */
    }
    map.user_va = (int64_t)mmap(0, map.length, PROT_READ | PROT_WRITE,
                            MAP_PRIVATE | MAP_ANONYMOUS, 0, 0);

    map.ioas_id = alloc_data.out_ioas_id;

    printf("the map va is : %llx,the iova is % llx,the size is %d\n",map.user_va,map.iova,map.length);
    if(ioctl(iommufd, IOMMU_IOAS_MAP, &map))
    {
	std::cout << "IOMMU_IOAS_MAP failed! "<< std::endl;
        return 1;
    }

    /*attachAssociate a pasid (of a cdev device) with an address space within the bound iommufd*/
    pasid.pasid = 0;
    pasid.pt_id = alloc_data.out_ioas_id;
    if(ioctl(cdev_fd, VFIO_DEVICE_PASID_ATTACH_IOMMUFD_PT, &pasid))
    {
	std::cout << "VFIO_DEVICE_PASID_ATTACH_IOMMUFD_PT failed! error is "<<strerror(errno)<< std::endl;
        return 1;
    }
     /*attachAssociate a pasid (of a cdev device) with an address space within the bound iommufd*/
    pasid.pasid = 1;
    pasid.pt_id = alloc_data.out_ioas_id;
    if(ioctl(cdev_fd, VFIO_DEVICE_PASID_ATTACH_IOMMUFD_PT, &pasid))
    {
        std::cout << "VFIO_DEVICE_PASID_ATTACH_IOMMUFD_PT failed! "<< std::endl;
        return 1;
    }
    
	
    while(1)
    {
        char input[5];
	sleep(150);
	printf("could you want EXIT ?enter [yes|no]? : ");
        scanf("%s", input);
	if(!std::strcmp(input, "yes"))
	{
	     break;
	}
    }
	
    detach_pasid.pasid = alloc_data.out_ioas_id;
    if(ioctl(cdev_fd, VFIO_DEVICE_PASID_DETACH_IOMMUFD_PT, &detach_pasid))
    {
	std::cout << "VFIO_DEVICE_PASID_DETACH_IOMMUFD_PT failed! "<< std::endl;
        return 1;
    }
	
	/*DMA unmapping */
	unmap.ioas_id = map.ioas_id;
	unmap.iova = map.iova;
	unmap.length = map.length;
	if(ioctl(iommufd, IOMMU_IOAS_UNMAP, &unmap))
    {
	    std::cout << "IOMMU_IOAS_UNMAP failed! "<< std::endl;
        return 1;
    }

    sleep(150);
	
    return 0;

/* Other device operations as stated in "VFIO Usage Example" */
}

Re: FW: About the patch ”https://lore.kernel.org/linux-iommu/20240412082121.33382-1-yi.l.liu@intel.com/ “ for help

[Yi Liu]

On 2024/7/23 10:52, XueMei Yue wrote:
> DEAR ALL ,
> 
> No  I  have know the root cause ,  the issue  ouccured when below code run my AMD test PC.
> 
> So  could you guys  give  some suggestion ?  Will very much appreciate if you feel free to reply！！
> 
> [cid:image001.png@01DADCEE.6F2E3F80]

For the people that cannot see the picture. It shows the pasid attach
path failed as the domain->ops->set_dev_pasid is null, hence failed with
EOPNOTSUPP errno.

I don't think the AMD iommu driver has supported the set_dev_pasid callback
for the non-SVA domains. Suravee can keep me honest. And you may also need
to check if your test PC has the PASID capability.

-- 
Regards,
Yi Liu

RE: FW: About the patch ”https://lore.kernel.org/linux-iommu/20240412082121.33382-1-yi.l.liu@intel.com/ “ for help

[XueMei Yue]

[-- Attachment #1: Type: text/plain, Size: 1749 bytes --]

Thank you for your reply！
My pc has the PASID capability, See the attachment.

 " I don't think the AMD iommu driver has supported the set_dev_pasid callback for the non-SVA domains."
   ------ xuemei :  So if I want to use the PASID to test PCIE ATS request messages,could you give some suggestions ? usr SVA domain can solve thie issue ? 


-----Original Message-----
From: Yi Liu <yi.l.liu@intel.com> 
Sent: 2024年7月23日 11:04
To: XueMei Yue <xuemeiyue@petaio.com>; iommu@lists.linux.dev; alex.williamson@redhat.com; robin.murphy@arm.com; eric.auger@redhat.com; nicolinc@nvidia.com; kvm@vger.kernel.org; chao.p.peng@linux.intel.com; baolu.lu@linux.intel.com; joro@8bytes.org; Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
Subject: Re: FW: About the patch ”https://lore.kernel.org/linux-iommu/20240412082121.33382-1-yi.l.liu@intel.com/ “ for help

CAUTION: This email originated from outside of PetaIO. Do not click on links or open attachments unless you recognize the sender and know that the content is safe.


On 2024/7/23 10:52, XueMei Yue wrote:
> DEAR ALL ,
>
> No  I  have know the root cause ,  the issue  ouccured when below code run my AMD test PC.
>
> So  could you guys  give  some suggestion ?  Will very much appreciate 
> if you feel free to reply！！
>
> [cid:image001.png@01DADCEE.6F2E3F80]

For the people that cannot see the picture. It shows the pasid attach path failed as the domain->ops->set_dev_pasid is null, hence failed with EOPNOTSUPP errno.

I don't think the AMD iommu driver has supported the set_dev_pasid callback for the non-SVA domains. Suravee can keep me honest. And you may also need to check if your test PC has the PASID capability.

--
Regards,
Yi Liu

[-- Attachment #2: pasid.png --]
[-- Type: image/png, Size: 4104 bytes --]

Re: FW: About the patch ”https://lore.kernel.org/linux-iommu/20240412082121.33382-1-yi.l.liu@intel.com/ “ for help

[Yi Liu]

On 2024/7/23 11:22, XueMei Yue wrote:
> Thank you for your reply！
> My pc has the PASID capability, See the attachment.

ok.

BTW. A heads up: you are looping the mailing list, so you'd better use
the plain text format and avoid including pictures if it can be expressed
by text.

>   " I don't think the AMD iommu driver has supported the set_dev_pasid callback for the non-SVA domains."
>     ------ xuemei :  So if I want to use the PASID to test PCIE ATS request messages,could you give some suggestions ? usr SVA domain can solve thie issue ?

You should not mix ATS with PASID, ATS does not rely on PASID. You
should be able to test ATS without PASID. ATS is a performance feature,
so if you want to test it in system level, you need a benchmark to do
it. Or you can test it in pci transaction level, you would need tools
for it then. It's up to you.

-- 
Regards,
Yi Liu

RE: FW: About the patch ”https://lore.kernel.org/linux-iommu/20240412082121.33382-1-yi.l.liu@intel.com/ “ for help

[XueMei Yue]

Hi Yi Liu, 
Thanks for your suggestion! 
we have tested ATS without PASID successfully.
Now I want use PASID to verify other function.maybe not related to ATS. 
Could you give some suggestion about my example "iommufd0716.cpp", How to make it run successfully via linux user API ? 
Thanks very much ! 

-----Original Message-----
From: Yi Liu <yi.l.liu@intel.com> 
Sent: 2024年7月23日 12:06
To: XueMei Yue <xuemeiyue@petaio.com>; iommu@lists.linux.dev; alex.williamson@redhat.com; robin.murphy@arm.com; eric.auger@redhat.com; nicolinc@nvidia.com; kvm@vger.kernel.org; chao.p.peng@linux.intel.com; baolu.lu@linux.intel.com; joro@8bytes.org; Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
Subject: Re: FW: About the patch ”https://lore.kernel.org/linux-iommu/20240412082121.33382-1-yi.l.liu@intel.com/ “ for help

CAUTION: This email originated from outside of PetaIO. Do not click on links or open attachments unless you recognize the sender and know that the content is safe.


On 2024/7/23 11:22, XueMei Yue wrote:
> Thank you for your reply！
> My pc has the PASID capability, See the attachment.

ok.

BTW. A heads up: you are looping the mailing list, so you'd better use the plain text format and avoid including pictures if it can be expressed by text.

>   " I don't think the AMD iommu driver has supported the set_dev_pasid callback for the non-SVA domains."
>     ------ xuemei :  So if I want to use the PASID to test PCIE ATS request messages,could you give some suggestions ? usr SVA domain can solve thie issue ?

You should not mix ATS with PASID, ATS does not rely on PASID. You should be able to test ATS without PASID. ATS is a performance feature, so if you want to test it in system level, you need a benchmark to do it. Or you can test it in pci transaction level, you would need tools for it then. It's up to you.

--
Regards,
Yi Liu

Re: FW: About the patch ”https://lore.kernel.org/linux-iommu/20240412082121.33382-1-yi.l.liu@intel.com/ “ for help

[Yi Liu]

On 2024/7/23 14:52, XueMei Yue wrote:
> Hi Yi Liu,
> Thanks for your suggestion!
> we have tested ATS without PASID successfully.
> Now I want use PASID to verify other function.maybe not related to ATS.
> Could you give some suggestion about my example "iommufd0716.cpp", How to make it run successfully via linux user API ?
> Thanks very much !

you need to make the pasid attach path work first. As I mentioned, the AMD
driver does not support it yet.

> -----Original Message-----
> From: Yi Liu <yi.l.liu@intel.com>
> Sent: 2024年7月23日 12:06
> To: XueMei Yue <xuemeiyue@petaio.com>; iommu@lists.linux.dev; alex.williamson@redhat.com; robin.murphy@arm.com; eric.auger@redhat.com; nicolinc@nvidia.com; kvm@vger.kernel.org; chao.p.peng@linux.intel.com; baolu.lu@linux.intel.com; joro@8bytes.org; Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
> Subject: Re: FW: About the patch ”https://lore.kernel.org/linux-iommu/20240412082121.33382-1-yi.l.liu@intel.com/ “ for help
> 
> CAUTION: This email originated from outside of PetaIO. Do not click on links or open attachments unless you recognize the sender and know that the content is safe.
> 
> 
> On 2024/7/23 11:22, XueMei Yue wrote:
>> Thank you for your reply！
>> My pc has the PASID capability, See the attachment.
> 
> ok.
> 
> BTW. A heads up: you are looping the mailing list, so you'd better use the plain text format and avoid including pictures if it can be expressed by text.
> 
>>    " I don't think the AMD iommu driver has supported the set_dev_pasid callback for the non-SVA domains."
>>      ------ xuemei :  So if I want to use the PASID to test PCIE ATS request messages,could you give some suggestions ? usr SVA domain can solve thie issue ?
> 
> You should not mix ATS with PASID, ATS does not rely on PASID. You should be able to test ATS without PASID. ATS is a performance feature, so if you want to test it in system level, you need a benchmark to do it. Or you can test it in pci transaction level, you would need tools for it then. It's up to you.
> 
> --
> Regards,
> Yi Liu

-- 
Regards,
Yi Liu

Re: FW: About the patch âhttps://lore.kernel.org/linux-iommu/20240412082121.33382-1-yi.l.liu@intel.com/ â for help

[Vasant Hegde]

On 7/24/2024 8:21 AM, Yi Liu wrote:
> On 2024/7/23 14:52, XueMei Yue wrote:
>> Hi Yi Liu,
>> Thanks for your suggestion!
>> we have tested ATS without PASID successfully.
>> Now I want use PASID to verify other function.maybe not related to ATS.
>> Could you give some suggestion about my example "iommufd0716.cpp", How to make
>> it run successfully via linux user API ?
>> Thanks very much !
> 
> you need to make the pasid attach path work first. As I mentioned, the AMD
> driver does not support it yet.

Thanks Yi for responding. I missed this thread.

You are right. Currently AMD driver doesn't support attaching PASID outside SVA
domain.

Regarding ATS, AMD driver enables it by default if both IOMMU and device has ATS
support. So you should be able to use/test this path.

If you want to test PASID/PRI combinations then you can use SVA interfaces.

Regarding exposing PASID to user space via iommufd, we haven't looked into it in
detail. Its in our list. but it will take some time.

-Vasant

RE: FW: About the patch âhttps://lore.kernel.org/linux-iommu/20240412082121.33382-1-yi.l.liu@intel.com/ â for help

[XueMei Yue]

Hi Vasant & Yi,
 
Thanks for your reply, now I am trying to test my new feature on Intel platform.

Best regards
Xuemei

-----Original Message-----
From: Vasant Hegde <vasant.hegde@amd.com> 
Sent: 2024年7月26日 17:19
To: Yi Liu <yi.l.liu@intel.com>; XueMei Yue <xuemeiyue@petaio.com>; iommu@lists.linux.dev; alex.williamson@redhat.com; robin.murphy@arm.com; eric.auger@redhat.com; nicolinc@nvidia.com; kvm@vger.kernel.org; chao.p.peng@linux.intel.com; baolu.lu@linux.intel.com; joro@8bytes.org; Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
Subject: Re: FW: About the patch âhttps://lore.kernel.org/linux-iommu/20240412082121.33382-1-yi.l.liu@intel.com/ â for help

CAUTION: This email originated from outside of PetaIO. Do not click on links or open attachments unless you recognize the sender and know that the content is safe.


On 7/24/2024 8:21 AM, Yi Liu wrote:
> On 2024/7/23 14:52, XueMei Yue wrote:
>> Hi Yi Liu,
>> Thanks for your suggestion!
>> we have tested ATS without PASID successfully.
>> Now I want use PASID to verify other function.maybe not related to ATS.
>> Could you give some suggestion about my example "iommufd0716.cpp", 
>> How to make it run successfully via linux user API ?
>> Thanks very much !
>
> you need to make the pasid attach path work first. As I mentioned, the 
> AMD driver does not support it yet.

Thanks Yi for responding. I missed this thread.

You are right. Currently AMD driver doesn't support attaching PASID outside SVA domain.

Regarding ATS, AMD driver enables it by default if both IOMMU and device has ATS support. So you should be able to use/test this path.

If you want to test PASID/PRI combinations then you can use SVA interfaces.

Regarding exposing PASID to user space via iommufd, we haven't looked into it in detail. Its in our list. but it will take some time.

-Vasant