Re: Vfio-on-arm: the device tree does not contain the DMA device

[Alvise Rigo <a.rigo-lrHrjnjw1UfHK3s98zE1ajGjJy/sRE9J@public.gmane.org>]

Linaro has a great set of pre-built images here http://releases.linaro.org/.

For example these should work fine:
http://releases.linaro.org/13.12/ubuntu/saucy-images

alvise

Il 20/06/2014 10:23, Shannon Zhao ha scritto:
> Hi Alvise,
> 
> I'm so sorry that I can't use qemu-debootstrap because my environment can't access some links using proxy.
> Are there other ways to get a proper fs?
> 
> Thanks
> Shannon
> 
> On 2014/6/20 15:15, Alvise Rigo wrote:
>> Hi Shannon,
>>
>> If you follow the steps in the section "Creating the File System" of the
>> guide you should be able to obtain a working fs.
>>
>> Regards,
>> alvise
>>
>> Il 20/06/2014 04:57, Shannon Zhao ha scritto:
>>> Hi Bharat,
>>>
>>> Thanks for your advice .
>>>
>>> I'm using the rootfs getting from the url "http://www.virtualopensystems.com/downloads/guides/kvm_on_arm/fs-alip-armel.cramfs".
>>>
>>> Can you give me another link to get a newer rootfs?
>>>
>>> Thank you
>>> Shannon
>>>
>>> On 2014/6/19 20:20, Bharat.Bhushan-KZfg59tc24xl57MIdRCFDg@public.gmane.org wrote:
>>>>
>>>>
>>>>> -----Original Message-----
>>>>> From: Shannon Zhao [mailto:zhaoshenglong-hv44wF8Li93QT0dZR+AlfA@public.gmane.org]
>>>>> Sent: Thursday, June 19, 2014 5:47 PM
>>>>> To: Bhushan Bharat-R65777; Alvise Rigo; a.motakis-lrHrjnjw1UfHK3s98zE1ajGjJy/sRE9J@public.gmane.org;
>>>>> iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org; kvmarm-FPEHb7Xf0XXUo1n7N8X6UoWGPAHP3yOg@public.gmane.org
>>>>> Subject: Re: Vfio-on-arm: the device tree does not contain the DMA device
>>>>>
>>>>>
>>>>>
>>>>> On 2014/6/19 20:12, Bharat.Bhushan-KZfg59tc24xl57MIdRCFDg@public.gmane.org wrote:
>>>>>>
>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: Shannon Zhao [mailto:zhaoshenglong-hv44wF8Li93QT0dZR+AlfA@public.gmane.org]
>>>>>>> Sent: Thursday, June 19, 2014 5:38 PM
>>>>>>> To: Alvise Rigo; Bhushan Bharat-R65777;
>>>>>>> a.motakis-lrHrjnjw1UfHK3s98zE1ajGjJy/sRE9J@public.gmane.org; iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org;
>>>>>>> kvmarm-FPEHb7Xf0XXUo1n7N8X6UoWGPAHP3yOg@public.gmane.org
>>>>>>> Subject: Re: Vfio-on-arm: the device tree does not contain the DMA
>>>>>>> device
>>>>>>>
>>>>>>> Hi Alvise,
>>>>>>>
>>>>>>> I run following cmds :
>>>>>>> 	echo 2c0a0000.dma >
>>>>>>> /sys/bus/platform/drivers/vfio-platform/vfio_bind
>>>>>>
>>>>>> run "ls /dev/vfio/" before and after the above command
>>>>> Hi Bharat,
>>>>>
>>>>> Nothing appeared either before or after the cmd "echo 2c0a0000.dma >
>>>>> /sys/bus/platform/drivers/vfio-platform/vfio_bind".
>>>>> There is only /dev/vfio.
>>>>
>>>> I am not sure, but may be you are using pretty old rootfs with broken udev rules. Try to fetch a newer rootfs form somewhere.
>>>>
>>>> Thanks
>>>> -Bharat
>>>>
>>>>
>>>>>>
>>>>>> Thanks
>>>>>> -Bharat
>>>>>>
>>>>>>
>>>>>>> 	./qemu-system-arm -enable-kvm -M vexpress-a15 -cpu cortex-a15 \
>>>>>>>         	-device
>>>>>>>
>>>>> pl330,vfio_group="/dev/vfio/0",vfio_device="2c0a0000.dma",vfio_container="/dev/v
>>>>>>> fio/vfio" \
>>>>>>>         	-append "earlyprintk ip=dhcp console=ttyAMA0 rootwait" \
>>>>>>>         	-kernel zImage -dtb guest-a15.dtb -initrd filesystem.cpio.gz  -
>>>>>>> nographic But the error still exists.
>>>>>>> 	qemu-system-arm: -device
>>>>>>>
>>>>> pl330,vfio_group=/dev/vfio/0,vfio_device=2c0a0000.dma,vfio_container=/dev/vfio/v
>>>>>>> fio: Parameter 'driver' expects pluggable device type
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Shannon
>>>>>>>
>>>>>>> On 2014/6/19 19:26, Alvise Rigo wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> Il 19/06/2014 13:11, Shannon Zhao ha scritto:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 2014/6/19 18:47, Alvise Rigo wrote:
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> Il 19/06/2014 11:38, Shannon Zhao ha scritto:
>>>>>>>>>>> Hi Bharat,
>>>>>>>>>>>
>>>>>>>>>>> On 2014/6/19 17:19, Bharat.Bhushan-KZfg59tc24xl57MIdRCFDg@public.gmane.org wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>>> From: kvmarm-bounces-FPEHb7Xf0XXUo1n7N8X6UoWGPAHP3yOg@public.gmane.org [mailto:kvmarm-
>>>>>>>>>>>>> bounces-FPEHb7Xf0XXUo1n7N8X6UoWGPAHP3yOg@public.gmane.org] On Behalf Of Shannon Zhao
>>>>>>>>>>>>> Sent: Thursday, June 19, 2014 2:45 PM
>>>>>>>>>>>>> To: Alvise Rigo; a.motakis-lrHrjnjw1UfHK3s98zE1ajGjJy/sRE9J@public.gmane.org;
>>>>>>>>>>>>> iommu-cunTk1MwBs/ROKNJybVBZg@public.gmane.org foundation.org; kvmarm-FPEHb7Xf0XXUo1n7N8X6UoWGPAHP3yOg@public.gmane.org
>>>>>>>>>>>>> Subject: Re: Vfio-on-arm: the device tree does not contain the
>>>>>>>>>>>>> DMA device
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Alvise,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for your reply.
>>>>>>>>>>>>> I have apply the patch and could see the DMA devices.
>>>>>>>>>>>>> But there is no vosys_dmatest int the directory
>>>>>>>>>>>>> "/sys/kernel/debug/" after mounting debugfs.
>>>>>>>>>>>>> I found that the branch vfio-platform-v4 doesn't contain the
>>>>>>>>>>>>> module vosys_dmatest.
>>>>>>>>>>>>> Then I port it from the branch pl330_dmatest and run vosys_dmatest
>>>>>>> successfully.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 	echo 1 > /sys/kernel/debug/vosys_dmatest/start
>>>>>>>>>>>>>
>>>>>>>>>>>>> 	[ 6145.343419] dmaengine: __dma_request_channel: success
>>>>>>> (dma0chan0)
>>>>>>>>>>>>> 	[ 6145.344000] vosys dmatest: callback: transfer completed
>>>>>>>>>>>>> 	[ 6145.344147] vosys dmatest: tx completed
>>>>>>>>>>>>> 	[ 6145.344589] vosys dmatest: source and destination match.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Download the qemu and use following command to start a guest:
>>>>>>>>>>>>> 	./qemu-system-arm -enable-kvm -M vexpress-a15 -cpu cortex-a15 \
>>>>>>>>>>>>>         	-device
>>>>>>> pl330,vfio_group="/dev/vfio/0",vfio_device="2c0a0000.dma",\
>>>>>>>>>>>>>         	vfio_container="/dev/vfio/vfio" \
>>>>>>>>>>>>>         	-append "earlyprintk ip=dhcp console=ttyAMA0 rootwait" \
>>>>>>>>>>>>>         	-kernel zImage -dtb guest.dtb -initrd filesystem.cpio.gz
>>>>>>>>>>>>> - nographic An error reported:
>>>>>>>>>>>>> 	qemu-system-arm: -device
>>>>>>>>>>>>> pl330,vfio_group=/dev/vfio/0,vfio_device=2c0a0000.dma,:
>>>>>>>>>>>>
>>>>>>>>>>>> Why do we need to give "vfio_group=/dev/vfio/0" ?
>>>>>>>>>>> I refer to the url
>>>>>>> "http://www.virtualopensystems.com/en/solutions/guides/vfio-on-arm/".
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> 	could not open disk image vfio_container=/dev/vfio:
>>>>>>>>>>>>> 	Could not open 'vfio_container=/dev/vfio': No such file or
>>>>>>>>>>>>> directory
>>>>>>>>>>>>
>>>>>>>>>>>> Are you unbinding "2c0a0000.dma" from its driver and binding with "vfio-
>>>>>>> playform" driver ?
>>>>>>>>>>> How to unbind "2c0a0000.dma" from its driver?
>>>>>>>>>>> I have tried following cmd to bind "2c0a0000.dma" with "vfio-playform"
>>>>>>> driver.
>>>>>>>>>>> 	echo 2c0a0000.dma >
>>>>>>>>>>> /sys/bus/platform/drivers/vfio-platform/vfio_bind
>>>>>>>>>>> But there is no /dev/vfio/0 or /dev/vfio/vfio .
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Can you please verify that while configuring the host you enabled:
>>>>>>>>>> - Device Drivers -> IOMMU Hardware Support -> ARM Ltd. System MMU
>>>>>>>>>> (SMMU) Support and
>>>>>>>>>> - Device Drivers -> VFIO Non-Privileged userspace driver framework
>>>>>>>>>> -> VFIO support for platform devices
>>>>>>>>>>
>>>>>>>>> Hi Alvise,
>>>>>>>>>
>>>>>>>>> I have checked the two configuration are enabled.
>>>>>>>>>
>>>>>>>>> Then try the following cmd, but another error reported:
>>>>>>>>> 	qemu-system-arm: -device
>>>>>>>>> pl330,vfio_group=/dev/vfio/0,vfio_device=2c0a0000.dma,vfio_container=
>>>>>>>>> /dev/vfio/vfio: Parameter 'driver' expects pluggable device type
>>>>>>>>>
>>>>>>>>> Do I need to unbind the "2c0a0000.dma" from its driver and binding with
>>>>>>> "vfio-platform" driver as Bharat mentioned ?
>>>>>>>>
>>>>>>>> Yes, the 2c0a0000.dma should not be bound to any driver at boot, so
>>>>>>>> you just need to:
>>>>>>>> echo 2c0a0000.dma > /sys/bus/platform/drivers/vfio-platform/vfio_bind
>>>>>>>> to bind the device to the VFIO driver.
>>>>>>>
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> alvise
>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Then try to update the pl330-vfio-dev branch and run QEMU with the
>>>>> following
>>>>>>> command:
>>>>>>>>>>
>>>>>>>>>> ./qemu-system-arm -enable-kvm -M vexpress-a15 -cpu cortex-a15 \
>>>>>>>>>>     -device
>>>>>>>
>>>>> pl330,vfio_group="/dev/vfio/0",vfio_device="2c0a0000.dma",vfio_container="/dev/v
>>>>>>> fio/vfio" \
>>>>>>>>>>     -append "earlyprintk ip=dhcp console=ttyAMA0 rootwait" \
>>>>>>>>>>     -kernel zImage -dtb guest.dtb -initrd filesystem.cpio.gz
>>>>>>>>>>
>>>>>>>>>> that means without breaking the "-device pl330" options.
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> alvise
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks
>>>>>>>>>>>> -Bharat
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Is there something I'm wrong or do I miss some steps?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>> Shannon
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 2014/6/18 21:24, Alvise Rigo wrote:
>>>>>>>>>>>>>> Hi Shannon,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thank you for reporting this issue, the device tree file was indeed
>>>>>>> missing.
>>>>>>>>>>>>>> To solve the problem, you can apply the following patch to the
>>>>>>>>>>>>>> repository you mentioned
>>>>>>>>>>>>>> (git://github.com/virtualopensystems/arm-dts.git).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> For your information, soon we will update the guide with a newer
>>>>>>>>>>>>>> version of VFIO with this issue fixed.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>>> alvise
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>  fast_models/rtsm_ve-cortex_a15x4_a7x4.dts | 100
>>>>>>>>>>>>>> +++++++++++++++++++++++++++++-
>>>>>>>>>>>>>>  1 file changed, 97 insertions(+), 3 deletions(-)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> diff --git a/fast_models/rtsm_ve-cortex_a15x4_a7x4.dts
>>>>>>>>>>>>>> b/fast_models/rtsm_ve-cortex_a15x4_a7x4.dts
>>>>>>>>>>>>>> index ef1275b..79c6af2 100644
>>>>>>>>>>>>>> --- a/fast_models/rtsm_ve-cortex_a15x4_a7x4.dts
>>>>>>>>>>>>>> +++ b/fast_models/rtsm_ve-cortex_a15x4_a7x4.dts
>>>>>>>>>>>>>> @@ -142,7 +142,7 @@
>>>>>>>>>>>>>>  		cpu7: cpu@7 {
>>>>>>>>>>>>>>  			device_type = "cpu";
>>>>>>>>>>>>>>  			compatible = "arm,cortex-a7";
>>>>>>>>>>>>>> -			reg = <0x104>;
>>>>>>>>>>>>>> +			reg = <0x103>;
>>>>>>>>>>>>>>  			cluster = <&cluster1>;
>>>>>>>>>>>>>>  			core = <&core7>;
>>>>>>>>>>>>>>  		};
>>>>>>>>>>>>>> @@ -159,7 +159,7 @@
>>>>>>>>>>>>>>  		#address-cells = <0>;
>>>>>>>>>>>>>>  		interrupt-controller;
>>>>>>>>>>>>>>  		reg = <0 0x2c001000 0 0x1000>,
>>>>>>>>>>>>>> -		      <0 0x2c002000 0 0x2000>,
>>>>>>>>>>>>>> +		      <0 0x2c002000 0 0x1000>,
>>>>>>>>>>>>>>  		      <0 0x2c004000 0 0x2000>,
>>>>>>>>>>>>>>  		      <0 0x2c006000 0 0x2000>;
>>>>>>>>>>>>>>  		interrupts = <1 9 0xf04>;
>>>>>>>>>>>>>> @@ -183,6 +183,101 @@
>>>>>>>>>>>>>>  			     <1 10 0xf08>;
>>>>>>>>>>>>>>  	};
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> +	smmu@2c0b0000 {
>>>>>>>>>>>>>> +                compatible = "arm,smmu-v2";
>>>>>>>>>>>>>> +                reg = <0x0 0x2c0b0000 0x0 0x10000>;
>>>>>>>>>>>>>> +                #global-interrupts = <2>;
>>>>>>>>>>>>>> +                interrupts = <0 58 4>,
>>>>>>>>>>>>>> +                             <0 59 4>,
>>>>>>>>>>>>>> +                             <0 60 4>,
>>>>>>>>>>>>>> +                             <0 60 4>,
>>>>>>>>>>>>>> +                             <0 60 4>,
>>>>>>>>>>>>>> +                             <0 60 4>,
>>>>>>>>>>>>>> +                             <0 60 4>,
>>>>>>>>>>>>>> +                             <0 60 4>,
>>>>>>>>>>>>>> +                             <0 60 4>;
>>>>>>>>>>>>>> +                mmu-masters = <&dma0 0x0 0x8>,
>>>>>>>>>>>>>> +                              <&dma1 0x1 0x9>,
>>>>>>>>>>>>>> +                              <&dma2 0x2 0xa>,
>>>>>>>>>>>>>> +                              <&dma3 0x3 0xb>;
>>>>>>>>>>>>>> +        };
>>>>>>>>>>>>>> +
>>>>>>>>>>>>>> +	dma0: dma@2c0a0000 {
>>>>>>>>>>>>>> +		compatible = "arm,pl330";
>>>>>>>>>>>>>> +		reg = <0 0x2c0a0000 0 0x1000>;
>>>>>>>>>>>>>> +		interrupts = <0 64 4>,
>>>>>>>>>>>>>> +			     <0 65 4>,
>>>>>>>>>>>>>> +			     <0 66 4>,
>>>>>>>>>>>>>> +			     <0 67 4>,
>>>>>>>>>>>>>> +			     <0 68 4>,
>>>>>>>>>>>>>> +			     <0 69 4>,
>>>>>>>>>>>>>> +			     <0 70 4>,
>>>>>>>>>>>>>> +			     <0 71 4>;
>>>>>>>>>>>>>> +		#dma-cells = <1>;
>>>>>>>>>>>>>> +		#stream-id-cells = <2>;
>>>>>>>>>>>>>> +		clocks = <&oscclk6a>;
>>>>>>>>>>>>>> +		clock-names = "apb_pclk";
>>>>>>>>>>>>>> +	};
>>>>>>>>>>>>>> +
>>>>>>>>>>>>>> +	dma1: dma@2c0a1000 {
>>>>>>>>>>>>>> +		compatible = "arm,pl330", "arm,primecell";
>>>>>>>>>>>>>> +		reg = <0 0x2c0a1000 0 0x1000>;
>>>>>>>>>>>>>> +		interrupts = <0 96 4>,
>>>>>>>>>>>>>> +			     <0 97 4>,
>>>>>>>>>>>>>> +			     <0 98 4>,
>>>>>>>>>>>>>> +			     <0 99 4>,
>>>>>>>>>>>>>> +			     <0 100 4>,
>>>>>>>>>>>>>> +			     <0 101 4>,
>>>>>>>>>>>>>> +			     <0 102 4>,
>>>>>>>>>>>>>> +			     <0 103 4>;
>>>>>>>>>>>>>> +		#dma-cells = <1>;
>>>>>>>>>>>>>> +		#stream-id-cells = <2>;
>>>>>>>>>>>>>> +		clocks = <&oscclk6a>;
>>>>>>>>>>>>>> +		clock-names = "apb_pclk";
>>>>>>>>>>>>>> +	};
>>>>>>>>>>>>>> +
>>>>>>>>>>>>>> +	dma2: dma@2c0a2000 {
>>>>>>>>>>>>>> +		compatible = "arm,pl330", "arm,primecell";
>>>>>>>>>>>>>> +		reg = <0 0x2c0a2000 0 0x1000>;
>>>>>>>>>>>>>> +		interrupts = <0 128 4>,
>>>>>>>>>>>>>> +			     <0 129 4>,
>>>>>>>>>>>>>> +			     <0 130 4>,
>>>>>>>>>>>>>> +			     <0 131 4>,
>>>>>>>>>>>>>> +			     <0 132 4>,
>>>>>>>>>>>>>> +			     <0 133 4>,
>>>>>>>>>>>>>> +			     <0 134 4>,
>>>>>>>>>>>>>> +			     <0 135 4>;
>>>>>>>>>>>>>> +		#dma-cells = <1>;
>>>>>>>>>>>>>> +		#stream-id-cells = <2>;
>>>>>>>>>>>>>> +		clocks = <&oscclk6a>;
>>>>>>>>>>>>>> +		clock-names = "apb_pclk";
>>>>>>>>>>>>>> +	};
>>>>>>>>>>>>>> +
>>>>>>>>>>>>>> +	dma3: dma@2c0a3000 {
>>>>>>>>>>>>>> +		compatible = "arm,pl330", "arm,primecell";
>>>>>>>>>>>>>> +		reg = <0 0x2c0a3000 0 0x1000>;
>>>>>>>>>>>>>> +		interrupts = <0 160 4>,
>>>>>>>>>>>>>> +			     <0 161 4>,
>>>>>>>>>>>>>> +			     <0 162 4>,
>>>>>>>>>>>>>> +			     <0 163 4>,
>>>>>>>>>>>>>> +			     <0 164 4>,
>>>>>>>>>>>>>> +			     <0 165 4>,
>>>>>>>>>>>>>> +			     <0 166 4>,
>>>>>>>>>>>>>> +			     <0 167 4>;
>>>>>>>>>>>>>> +		#dma-cells = <1>;
>>>>>>>>>>>>>> +		#stream-id-cells = <2>;
>>>>>>>>>>>>>> +		clocks = <&oscclk6a>;
>>>>>>>>>>>>>> +		clock-names = "apb_pclk";
>>>>>>>>>>>>>> +	};
>>>>>>>>>>>>>> +
>>>>>>>>>>>>>> +	oscclk6a: oscclk6a {
>>>>>>>>>>>>>> +		/* Reference 24MHz clock */
>>>>>>>>>>>>>> +		compatible = "fixed-clock";
>>>>>>>>>>>>>> +		#clock-cells = <0>;
>>>>>>>>>>>>>> +		clock-frequency = <24000000>;
>>>>>>>>>>>>>> +		clock-output-names = "oscclk6a";
>>>>>>>>>>>>>> +	};
>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>  	smb {
>>>>>>>>>>>>>>  		compatible = "simple-bus";
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> @@ -244,4 +339,3 @@
>>>>>>>>>>>>>>  		/include/ "rtsm_ve-motherboard.dtsi"
>>>>>>>>>>>>>>  	};
>>>>>>>>>>>>>>  };
>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Shannon
>>>>>>>>>>>>>
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> kvmarm mailing list
>>>>>>>>>>>>> kvmarm-FPEHb7Xf0XXUo1n7N8X6UoWGPAHP3yOg@public.gmane.org
>>>>>>>>>>>>> https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
>>>>>>>>>>>>
>>>>>>>>>>>> .
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> .
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> .
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Shannon
>>>>>>
>>>>>>
>>>>>> .
>>>>>>
>>>>>
>>>>> --
>>>>> Shannon
>>>>
>>>>
>>>> .
>>>>
>>>
>>
>> .
>>
>