exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks support for IOMMU

exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks support for IOMMU

[Mario Marietto]

[-- Attachment #1: Type: text/plain, Size: 4975 bytes --]

Hello,

We are a team of linux enthusiasts who are trying to boot Xen on a
Samsung XE303C12 Chromebook aka "snow"
following the suggestions in the slide show presentation here:
https://www.slideshare.net/xen_com_mgr/xpds16-porting-xen-on-arm-to-a-new-soc-julien-grall-arm

This device uses an exynos5250 SOC dual core 1.7 GHz with 2 MB RAM, it is
a Samsung armv7 chip with virtualization extensions.

In particular, we have it working fairly well both on the bare metal with
a recent 6.1.59 Linux LTS kernel and also with a recent 5.4.257 LTS
kernel with KVM, the older LTS kernel version is used to test KVM because
support for KVM on arm v7 was removed from Linux around kernel version
5.7. So we know we have the hypervisor mode enabled because we were able
to use it with KVM.

For Xen, we are using the latest Debian build of Xen 4.17 for the Debian
armhf architecture:

(XEN) Xen version 4.17.2-pre (Debian 4.17.1+2-gb773c48e36-1)
(pkg-xen-devel@xxxxxxxxxxxxxxxxxxxxxxx) (arm-linux-gnueabihf-gcc (Debian
12.2.0-14) 12.2.0) debug=n Thu May 18 19:26:30 UTC 2023

The Linux kernel is a custom build that adds the Xen config kernel
options (CONFIG_XEN_DOM0, etc) on top of a kernel that works well on the
same Chromebook model on the bare metal. I can provide the config options
of the kernel that was used if that is helpful.

Our method of booting is to have u-boot boot the Xen hypervisor and load
the device tree after adding the dom0 to the otherwise unaltered device
tree from the Linux kernel using u-boot fdt commands to add a /chosen
node, as described on the Xen wiki and in the pages linked from there. We
have also tried adding and loading an initrd.img using the device tree
/chosen node but that made no difference in our tests.

We actually have the Linux LTS kernel version 6.1.59 working as dom0 with
Xen using the same version of u-boot that we used for KVM, but with a big
problem.

The problem we see is that when booting the 6.1.59 kernel version as dom0
with Xen, the screen is totally dark and the only way to access the
system is remotely through ssh. Logs indicate most everything else is
working, such as the wifi card so we can access it remotely via ssh and a
USB optical mouse lights up when connected so USB is also working.
Obviously, the disk is also working. The Chromebook is configured to boot
from the device's SD card slot by turning on Chrome OS developer mode
options to enable booting from the SD card slot.

The mystery is that when booting the exact same 6.1.59 kernel on the bare
metal instead of booting it as dom0 on Xen, it boots up with full access
to the screen and we can interact with the system using the X.org windows
system. But booting as dom0 with Xen, the screen is totally dark and the
only access we have to the system is through the network via ssh. Also,
when booting the 5.4.257 kernel with KVM in hypervisor mode, the screen
works and we can interact with the system through the X.org windows
system.
Exploring the log file,we have seen the errors below :


With Xen (or in bare metal):

devuan-bunsen kernel: [drm] Exynos DRM: using 14400000.fimd device for DMA
mapping operations
devuan-bunsen kernel: exynos-drm exynos-drm: bound 14400000.fimd (ops
0xc0d96354)
devuan-bunsen kernel: exynos-drm exynos-drm: bound 14450000.mixer (ops
0xc0d97554)
devuan-bunsen kernel: exynos-drm exynos-drm: bound 145b0000.dp-controller
(ops 0xc0d97278)
devuan-bunsen kernel: exynos-drm exynos-drm: bound 14530000.hdmi (ops
0xc0d97bd0)
...
devuan-bunsen kernel: Console: switching to colour frame buffer device
170x48
devuan-bunsen kernel: exynos-drm exynos-drm: [drm] fb0: exynosdrmfb frame
buffer device
devuan-bunsen kernel: [drm] Initialized exynos 1.1.0 20180330 for
exynos-drm on minor 0

In this case,the kernel is able to use the exynos-drm kernel to start the
fb0 device. But with Xen we get this error with exynos-drm:

devuan-bunsen kernel: [drm] Exynos DRM: using 14400000.fimd device for DMA
mapping operations
devuan-bunsen kernel: exynos-drm exynos-drm: bound 14400000.fimd (ops
0xc0d96354)
devuan-bunsen kernel: exynos-mixer 14450000.mixer:
[drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks support
for IOMMU
devuan-bunsen kernel: exynos-drm exynos-drm: failed to bind 14450000.mixer
(ops 0xc0d97554): -22
devuan-bunsen kernel: exynos-drm exynos-drm: adev bind failed: -22
devuan-bunsen kernel: exynos-dp: probe of 145b0000.dp-controller failed
with error -22

I'm trying to find for a solution and I've googled a little bit and I
found this web site :

https://lore.kernel.org/linux-arm-kernel/20220208171823.226211-8-krzysztof.kozlowski@canonical.com/

with your email address and I tried to ask for some help for fixing the bug.

Any ideas why booting the same Linux kernel that results in a working
X.org display on the bare metal instead as dom0 on Xen would cause the
display to remain dark, but most other basic functions would work, such
as network, disk, and USB ? thanks.


-- 
Mario.

[-- Attachment #2: Type: text/html, Size: 6470 bytes --]

exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks support for IOMMU

[Mario Marietto]

Hello,

We are a team of linux enthusiasts who are trying to boot Xen on a
Samsung XE303C12 Chromebook aka "snow"
following the suggestions in the slide show presentation here:

https://www.slideshare.net/xen_com_mgr/xpds16-porting-xen-on-arm-to-a-new-soc-julien-grall-arm

This device uses an exynos5250 SOC dual core 1.7 GHz with 2 MB RAM, it is
a Samsung armv7 chip with virtualization extensions.

In particular, we have it working fairly well both on the bare metal with
a recent 6.1.59 Linux LTS kernel and also with a recent 5.4.257 LTS
kernel with KVM, the older LTS kernel version is used to test KVM because
support for KVM on arm v7 was removed from Linux around kernel version
5.7. So we know we have the hypervisor mode enabled because we were able
to use it with KVM.

For Xen, we are using the latest Debian build of Xen 4.17 for the Debian
armhf architecture:

(XEN) Xen version 4.17.2-pre (Debian 4.17.1+2-gb773c48e36-1)
(pkg-xen-devel@xxxxxxxxxxxxxxxxxxxxxxx) (arm-linux-gnueabihf-gcc (Debian
12.2.0-14) 12.2.0) debug=n Thu May 18 19:26:30 UTC 2023

The Linux kernel is a custom build that adds the Xen config kernel
options (CONFIG_XEN_DOM0, etc) on top of a kernel that works well on the
same Chromebook model on the bare metal. I can provide the config options
of the kernel that was used if that is helpful.

Our method of booting is to have u-boot boot the Xen hypervisor and load
the device tree after adding the dom0 to the otherwise unaltered device
tree from the Linux kernel using u-boot fdt commands to add a /chosen
node, as described on the Xen wiki and in the pages linked from there. We
have also tried adding and loading an initrd.img using the device tree
/chosen node but that made no difference in our tests.

We actually have the Linux LTS kernel version 6.1.59 working as dom0 with
Xen using the same version of u-boot that we used for KVM, but with a big
problem.

The problem we see is that when booting the 6.1.59 kernel version as dom0
with Xen, the screen is totally dark and the only way to access the
system is remotely through ssh. Logs indicate most everything else is
working, such as the wifi card so we can access it remotely via ssh and a
USB optical mouse lights up when connected so USB is also working.
Obviously, the disk is also working. The Chromebook is configured to boot
from the device's SD card slot by turning on Chrome OS developer mode
options to enable booting from the SD card slot.

The mystery is that when booting the exact same 6.1.59 kernel on the bare
metal instead of booting it as dom0 on Xen, it boots up with full access
to the screen and we can interact with the system using the X.org windows
system. But booting as dom0 with Xen, the screen is totally dark and the
only access we have to the system is through the network via ssh. Also,
when booting the 5.4.257 kernel with KVM in hypervisor mode, the screen
works and we can interact with the system through the X.org windows
system.

Exploring the log file,we have seen the errors below :


With Xen (or in bare metal):

devuan-bunsen kernel: [drm] Exynos DRM: using 14400000.fimd device for
DMA mapping operations
devuan-bunsen kernel: exynos-drm exynos-drm: bound 14400000.fimd (ops
0xc0d96354)
devuan-bunsen kernel: exynos-drm exynos-drm: bound 14450000.mixer (ops
0xc0d97554)
devuan-bunsen kernel: exynos-drm exynos-drm: bound
145b0000.dp-controller (ops 0xc0d97278)
devuan-bunsen kernel: exynos-drm exynos-drm: bound 14530000.hdmi (ops
0xc0d97bd0)
...
devuan-bunsen kernel: Console: switching to colour frame buffer device 170x48
devuan-bunsen kernel: exynos-drm exynos-drm: [drm] fb0: exynosdrmfb
frame buffer device
devuan-bunsen kernel: [drm] Initialized exynos 1.1.0 20180330 for
exynos-drm on minor 0

In this case,the kernel is able to use the exynos-drm kernel to start
the fb0 device. But with Xen we get this error with exynos-drm:

devuan-bunsen kernel: [drm] Exynos DRM: using 14400000.fimd device for
DMA mapping operations
devuan-bunsen kernel: exynos-drm exynos-drm: bound 14400000.fimd (ops
0xc0d96354)
devuan-bunsen kernel: exynos-mixer 14450000.mixer:
[drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks
support for IOMMU
devuan-bunsen kernel: exynos-drm exynos-drm: failed to bind
14450000.mixer (ops 0xc0d97554): -22
devuan-bunsen kernel: exynos-drm exynos-drm: adev bind failed: -22
devuan-bunsen kernel: exynos-dp: probe of 145b0000.dp-controller
failed with error -22


Any ideas why booting the same Linux kernel that results in a working
X.org display on the bare metal instead as dom0 on Xen would cause the
display to remain dark, but most other basic functions would work, such
as network, disk, and USB ? thanks.

-- 
Mario.

Re: exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks support for IOMMU

[Marek Szyprowski]

Hi,

On 31.10.2023 00:03, Mario Marietto wrote:
> We are a team of linux enthusiasts who are trying to boot Xen on a 
> Samsung XE303C12 Chromebook aka "snow" following the suggestions in 
> the slide show presentation here: 
> https://www.slideshare.net/xen_com_mgr/xpds16-porting-xen-on-arm-to-a-new-soc-julien-grall-arm 
> This device uses an exynos5250 SOC dual core 1.7 GHz with 2 MB RAM, it 
> is a Samsung armv7 chip with virtualization extensions. In particular, 
> we have it working fairly well both on the bare metal with a recent 
> 6.1.59 Linux LTS kernel and also with a recent 5.4.257 LTS kernel with 
> KVM, the older LTS kernel version is used to test KVM because support 
> for KVM on arm v7 was removed from Linux around kernel version 5.7. So 
> we know we have the hypervisor mode enabled because we were able to 
> use it with KVM. For Xen, we are using the latest Debian build of Xen 
> 4.17 for the Debian armhf architecture: (XEN) Xen version 4.17.2-pre 
> (Debian 4.17.1+2-gb773c48e36-1) 
> (pkg-xen-devel@xxxxxxxxxxxxxxxxxxxxxxx) (arm-linux-gnueabihf-gcc 
> (Debian 12.2.0-14) 12.2.0) debug=n Thu May 18 19:26:30 UTC 2023 The 
> Linux kernel is a custom build that adds the Xen config kernel options 
> (CONFIG_XEN_DOM0, etc) on top of a kernel that works well on the same 
> Chromebook model on the bare metal. I can provide the config options 
> of the kernel that was used if that is helpful. Our method of booting 
> is to have u-boot boot the Xen hypervisor and load the device tree 
> after adding the dom0 to the otherwise unaltered device tree from the 
> Linux kernel using u-boot fdt commands to add a /chosen node, as 
> described on the Xen wiki and in the pages linked from there. We have 
> also tried adding and loading an initrd.img using the device tree 
> /chosen node but that made no difference in our tests. We actually 
> have the Linux LTS kernel version 6.1.59 working as dom0 with Xen 
> using the same version of u-boot that we used for KVM, but with a big 
> problem. The problem we see is that when booting the 6.1.59 kernel 
> version as dom0 with Xen, the screen is totally dark and the only way 
> to access the system is remotely through ssh. Logs indicate most 
> everything else is working, such as the wifi card so we can access it 
> remotely via ssh and a USB optical mouse lights up when connected so 
> USB is also working. Obviously, the disk is also working. The 
> Chromebook is configured to boot from the device's SD card slot by 
> turning on Chrome OS developer mode options to enable booting from the 
> SD card slot. The mystery is that when booting the exact same 6.1.59 
> kernel on the bare metal instead of booting it as dom0 on Xen, it 
> boots up with full access to the screen and we can interact with the 
> system using the X.org windows system. But booting as dom0 with Xen, 
> the screen is totally dark and the only access we have to the system 
> is through the network via ssh. Also, when booting the 5.4.257 kernel 
> with KVM in hypervisor mode, the screen works and we can interact with 
> the system through the X.org windows system. Exploring the log file,we 
> have seen the errors below :
>
> With Xen (or in bare metal):
>
> devuan-bunsen kernel: [drm] Exynos DRM: using 14400000.fimd device for 
> DMA mapping operations
> devuan-bunsen kernel: exynos-drm exynos-drm: bound 14400000.fimd (ops 
> 0xc0d96354)
> devuan-bunsen kernel: exynos-drm exynos-drm: bound 14450000.mixer (ops 
> 0xc0d97554)
> devuan-bunsen kernel: exynos-drm exynos-drm: bound 
> 145b0000.dp-controller (ops 0xc0d97278)
> devuan-bunsen kernel: exynos-drm exynos-drm: bound 14530000.hdmi (ops 
> 0xc0d97bd0)
> ...
> devuan-bunsen kernel: Console: switching to colour frame buffer device 
> 170x48
> devuan-bunsen kernel: exynos-drm exynos-drm: [drm] fb0: exynosdrmfb 
> frame buffer device
> devuan-bunsen kernel: [drm] Initialized exynos 1.1.0 20180330 for 
> exynos-drm on minor 0
>
> In this case,the kernel is able to use the exynos-drm kernel to start 
> the fb0 device. But with Xen we get this error with exynos-drm:
>
> devuan-bunsen kernel: [drm] Exynos DRM: using 14400000.fimd device for 
> DMA mapping operations
> devuan-bunsen kernel: exynos-drm exynos-drm: bound 14400000.fimd (ops 
> 0xc0d96354)
> devuan-bunsen kernel: exynos-mixer 14450000.mixer: 
> [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks 
> support for IOMMU
> devuan-bunsen kernel: exynos-drm exynos-drm: failed to bind 
> 14450000.mixer (ops 0xc0d97554): -22
> devuan-bunsen kernel: exynos-drm exynos-drm: adev bind failed: -22
> devuan-bunsen kernel: exynos-dp: probe of 145b0000.dp-controller 
> failed with error -22
>
> I'm trying to find for a solution and I've googled a little bit and I 
> found this web site : 
> https://lore.kernel.org/linux-arm-kernel/20220208171823.226211-8-krzysztof.kozlowski@canonical.com/ 
> with your email address and I tried to ask for some help for fixing 
> the bug. Any ideas why booting the same Linux kernel that results in a 
> working X.org display on the bare metal instead as dom0 on Xen would 
> cause the display to remain dark, but most other basic functions would 
> work, such as network, disk, and USB ? thanks.


Thanks for the detailed description! Good to hear that those boards are 
still being used for various projects. I also have Snow Chromebook and 
use it for daily tests of linux-next branch.

Frankly speaking I have no idea what might happen wrong. There have been 
some changes recently in the Exynos IOMMU driver related to 
initialization, maybe your changes related to Xen enabling changed 
somehow the order of device initialization during boot. I assume that 
the device-tree you use for the bare metal run and Xen enabled run 
doesn't differ in the areas describing the hardware blocks.

Please check if cherry-picking the commit 
https://github.com/torvalds/linux/commit/bbc4d205d93f52ee18dfa7858d51489c0506547f 
to your v6.1.59 based kernel helps anyhow.

If not, then as a temporary workaround please disable 
CONFIG_DRM_EXYNOS_MIXER and CONFIG_DRM_EXYNOS_HDMI in your kernel config 
and check what will happen (You will lose the HDMI output, but maybe 
this won't a big issue).


Best regards
-- 
Marek Szyprowski, PhD
Samsung R&D Institute Poland

Re: exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks support for IOMMU

[Marek Szyprowski]

Hi,

On 31.10.2023 00:03, Mario Marietto wrote:
> We are a team of linux enthusiasts who are trying to boot Xen on a 
> Samsung XE303C12 Chromebook aka "snow" following the suggestions in 
> the slide show presentation here: 
> https://www.slideshare.net/xen_com_mgr/xpds16-porting-xen-on-arm-to-a-new-soc-julien-grall-arm 
> This device uses an exynos5250 SOC dual core 1.7 GHz with 2 MB RAM, it 
> is a Samsung armv7 chip with virtualization extensions. In particular, 
> we have it working fairly well both on the bare metal with a recent 
> 6.1.59 Linux LTS kernel and also with a recent 5.4.257 LTS kernel with 
> KVM, the older LTS kernel version is used to test KVM because support 
> for KVM on arm v7 was removed from Linux around kernel version 5.7. So 
> we know we have the hypervisor mode enabled because we were able to 
> use it with KVM. For Xen, we are using the latest Debian build of Xen 
> 4.17 for the Debian armhf architecture: (XEN) Xen version 4.17.2-pre 
> (Debian 4.17.1+2-gb773c48e36-1) 
> (pkg-xen-devel@xxxxxxxxxxxxxxxxxxxxxxx) (arm-linux-gnueabihf-gcc 
> (Debian 12.2.0-14) 12.2.0) debug=n Thu May 18 19:26:30 UTC 2023 The 
> Linux kernel is a custom build that adds the Xen config kernel options 
> (CONFIG_XEN_DOM0, etc) on top of a kernel that works well on the same 
> Chromebook model on the bare metal. I can provide the config options 
> of the kernel that was used if that is helpful. Our method of booting 
> is to have u-boot boot the Xen hypervisor and load the device tree 
> after adding the dom0 to the otherwise unaltered device tree from the 
> Linux kernel using u-boot fdt commands to add a /chosen node, as 
> described on the Xen wiki and in the pages linked from there. We have 
> also tried adding and loading an initrd.img using the device tree 
> /chosen node but that made no difference in our tests. We actually 
> have the Linux LTS kernel version 6.1.59 working as dom0 with Xen 
> using the same version of u-boot that we used for KVM, but with a big 
> problem. The problem we see is that when booting the 6.1.59 kernel 
> version as dom0 with Xen, the screen is totally dark and the only way 
> to access the system is remotely through ssh. Logs indicate most 
> everything else is working, such as the wifi card so we can access it 
> remotely via ssh and a USB optical mouse lights up when connected so 
> USB is also working. Obviously, the disk is also working. The 
> Chromebook is configured to boot from the device's SD card slot by 
> turning on Chrome OS developer mode options to enable booting from the 
> SD card slot. The mystery is that when booting the exact same 6.1.59 
> kernel on the bare metal instead of booting it as dom0 on Xen, it 
> boots up with full access to the screen and we can interact with the 
> system using the X.org windows system. But booting as dom0 with Xen, 
> the screen is totally dark and the only access we have to the system 
> is through the network via ssh. Also, when booting the 5.4.257 kernel 
> with KVM in hypervisor mode, the screen works and we can interact with 
> the system through the X.org windows system. Exploring the log file,we 
> have seen the errors below :
>
> With Xen (or in bare metal):
>
> devuan-bunsen kernel: [drm] Exynos DRM: using 14400000.fimd device for 
> DMA mapping operations
> devuan-bunsen kernel: exynos-drm exynos-drm: bound 14400000.fimd (ops 
> 0xc0d96354)
> devuan-bunsen kernel: exynos-drm exynos-drm: bound 14450000.mixer (ops 
> 0xc0d97554)
> devuan-bunsen kernel: exynos-drm exynos-drm: bound 
> 145b0000.dp-controller (ops 0xc0d97278)
> devuan-bunsen kernel: exynos-drm exynos-drm: bound 14530000.hdmi (ops 
> 0xc0d97bd0)
> ...
> devuan-bunsen kernel: Console: switching to colour frame buffer device 
> 170x48
> devuan-bunsen kernel: exynos-drm exynos-drm: [drm] fb0: exynosdrmfb 
> frame buffer device
> devuan-bunsen kernel: [drm] Initialized exynos 1.1.0 20180330 for 
> exynos-drm on minor 0
>
> In this case,the kernel is able to use the exynos-drm kernel to start 
> the fb0 device. But with Xen we get this error with exynos-drm:
>
> devuan-bunsen kernel: [drm] Exynos DRM: using 14400000.fimd device for 
> DMA mapping operations
> devuan-bunsen kernel: exynos-drm exynos-drm: bound 14400000.fimd (ops 
> 0xc0d96354)
> devuan-bunsen kernel: exynos-mixer 14450000.mixer: 
> [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks 
> support for IOMMU
> devuan-bunsen kernel: exynos-drm exynos-drm: failed to bind 
> 14450000.mixer (ops 0xc0d97554): -22
> devuan-bunsen kernel: exynos-drm exynos-drm: adev bind failed: -22
> devuan-bunsen kernel: exynos-dp: probe of 145b0000.dp-controller 
> failed with error -22
>
> I'm trying to find for a solution and I've googled a little bit and I 
> found this web site : 
> https://lore.kernel.org/linux-arm-kernel/20220208171823.226211-8-krzysztof.kozlowski@canonical.com/ 
> with your email address and I tried to ask for some help for fixing 
> the bug. Any ideas why booting the same Linux kernel that results in a 
> working X.org display on the bare metal instead as dom0 on Xen would 
> cause the display to remain dark, but most other basic functions would 
> work, such as network, disk, and USB ? thanks.


Thanks for the detailed description! Good to hear that those boards are 
still being used for various projects. I also have Snow Chromebook and 
use it for daily tests of linux-next branch.

Frankly speaking I have no idea what might happen wrong. There have been 
some changes recently in the Exynos IOMMU driver related to 
initialization, maybe your changes related to Xen enabling changed 
somehow the order of device initialization during boot. I assume that 
the device-tree you use for the bare metal run and Xen enabled run 
doesn't differ in the areas describing the hardware blocks.

Please check if cherry-picking the commit 
https://github.com/torvalds/linux/commit/bbc4d205d93f52ee18dfa7858d51489c0506547f 
to your v6.1.59 based kernel helps anyhow.

If not, then as a temporary workaround please disable 
CONFIG_DRM_EXYNOS_MIXER and CONFIG_DRM_EXYNOS_HDMI in your kernel config 
and check what will happen (You will lose the HDMI output, but maybe 
this won't a big issue).


Best regards
-- 
Marek Szyprowski, PhD
Samsung R&D Institute Poland


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks support for IOMMU

[Marek Szyprowski]

Hi,

On 31.10.2023 00:03, Mario Marietto wrote:
> We are a team of linux enthusiasts who are trying to boot Xen on a 
> Samsung XE303C12 Chromebook aka "snow" following the suggestions in 
> the slide show presentation here: 
> https://www.slideshare.net/xen_com_mgr/xpds16-porting-xen-on-arm-to-a-new-soc-julien-grall-arm 
> This device uses an exynos5250 SOC dual core 1.7 GHz with 2 MB RAM, it 
> is a Samsung armv7 chip with virtualization extensions. In particular, 
> we have it working fairly well both on the bare metal with a recent 
> 6.1.59 Linux LTS kernel and also with a recent 5.4.257 LTS kernel with 
> KVM, the older LTS kernel version is used to test KVM because support 
> for KVM on arm v7 was removed from Linux around kernel version 5.7. So 
> we know we have the hypervisor mode enabled because we were able to 
> use it with KVM. For Xen, we are using the latest Debian build of Xen 
> 4.17 for the Debian armhf architecture: (XEN) Xen version 4.17.2-pre 
> (Debian 4.17.1+2-gb773c48e36-1) 
> (pkg-xen-devel@xxxxxxxxxxxxxxxxxxxxxxx) (arm-linux-gnueabihf-gcc 
> (Debian 12.2.0-14) 12.2.0) debug=n Thu May 18 19:26:30 UTC 2023 The 
> Linux kernel is a custom build that adds the Xen config kernel options 
> (CONFIG_XEN_DOM0, etc) on top of a kernel that works well on the same 
> Chromebook model on the bare metal. I can provide the config options 
> of the kernel that was used if that is helpful. Our method of booting 
> is to have u-boot boot the Xen hypervisor and load the device tree 
> after adding the dom0 to the otherwise unaltered device tree from the 
> Linux kernel using u-boot fdt commands to add a /chosen node, as 
> described on the Xen wiki and in the pages linked from there. We have 
> also tried adding and loading an initrd.img using the device tree 
> /chosen node but that made no difference in our tests. We actually 
> have the Linux LTS kernel version 6.1.59 working as dom0 with Xen 
> using the same version of u-boot that we used for KVM, but with a big 
> problem. The problem we see is that when booting the 6.1.59 kernel 
> version as dom0 with Xen, the screen is totally dark and the only way 
> to access the system is remotely through ssh. Logs indicate most 
> everything else is working, such as the wifi card so we can access it 
> remotely via ssh and a USB optical mouse lights up when connected so 
> USB is also working. Obviously, the disk is also working. The 
> Chromebook is configured to boot from the device's SD card slot by 
> turning on Chrome OS developer mode options to enable booting from the 
> SD card slot. The mystery is that when booting the exact same 6.1.59 
> kernel on the bare metal instead of booting it as dom0 on Xen, it 
> boots up with full access to the screen and we can interact with the 
> system using the X.org windows system. But booting as dom0 with Xen, 
> the screen is totally dark and the only access we have to the system 
> is through the network via ssh. Also, when booting the 5.4.257 kernel 
> with KVM in hypervisor mode, the screen works and we can interact with 
> the system through the X.org windows system. Exploring the log file,we 
> have seen the errors below :
>
> With Xen (or in bare metal):
>
> devuan-bunsen kernel: [drm] Exynos DRM: using 14400000.fimd device for 
> DMA mapping operations
> devuan-bunsen kernel: exynos-drm exynos-drm: bound 14400000.fimd (ops 
> 0xc0d96354)
> devuan-bunsen kernel: exynos-drm exynos-drm: bound 14450000.mixer (ops 
> 0xc0d97554)
> devuan-bunsen kernel: exynos-drm exynos-drm: bound 
> 145b0000.dp-controller (ops 0xc0d97278)
> devuan-bunsen kernel: exynos-drm exynos-drm: bound 14530000.hdmi (ops 
> 0xc0d97bd0)
> ...
> devuan-bunsen kernel: Console: switching to colour frame buffer device 
> 170x48
> devuan-bunsen kernel: exynos-drm exynos-drm: [drm] fb0: exynosdrmfb 
> frame buffer device
> devuan-bunsen kernel: [drm] Initialized exynos 1.1.0 20180330 for 
> exynos-drm on minor 0
>
> In this case,the kernel is able to use the exynos-drm kernel to start 
> the fb0 device. But with Xen we get this error with exynos-drm:
>
> devuan-bunsen kernel: [drm] Exynos DRM: using 14400000.fimd device for 
> DMA mapping operations
> devuan-bunsen kernel: exynos-drm exynos-drm: bound 14400000.fimd (ops 
> 0xc0d96354)
> devuan-bunsen kernel: exynos-mixer 14450000.mixer: 
> [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks 
> support for IOMMU
> devuan-bunsen kernel: exynos-drm exynos-drm: failed to bind 
> 14450000.mixer (ops 0xc0d97554): -22
> devuan-bunsen kernel: exynos-drm exynos-drm: adev bind failed: -22
> devuan-bunsen kernel: exynos-dp: probe of 145b0000.dp-controller 
> failed with error -22
>
> I'm trying to find for a solution and I've googled a little bit and I 
> found this web site : 
> https://lore.kernel.org/linux-arm-kernel/20220208171823.226211-8-krzysztof.kozlowski@canonical.com/ 
> with your email address and I tried to ask for some help for fixing 
> the bug. Any ideas why booting the same Linux kernel that results in a 
> working X.org display on the bare metal instead as dom0 on Xen would 
> cause the display to remain dark, but most other basic functions would 
> work, such as network, disk, and USB ? thanks.


Thanks for the detailed description! Good to hear that those boards are 
still being used for various projects. I also have Snow Chromebook and 
use it for daily tests of linux-next branch.

Frankly speaking I have no idea what might happen wrong. There have been 
some changes recently in the Exynos IOMMU driver related to 
initialization, maybe your changes related to Xen enabling changed 
somehow the order of device initialization during boot. I assume that 
the device-tree you use for the bare metal run and Xen enabled run 
doesn't differ in the areas describing the hardware blocks.

Please check if cherry-picking the commit 
https://github.com/torvalds/linux/commit/bbc4d205d93f52ee18dfa7858d51489c0506547f 
to your v6.1.59 based kernel helps anyhow.

If not, then as a temporary workaround please disable 
CONFIG_DRM_EXYNOS_MIXER and CONFIG_DRM_EXYNOS_HDMI in your kernel config 
and check what will happen (You will lose the HDMI output, but maybe 
this won't a big issue).


Best regards
-- 
Marek Szyprowski, PhD
Samsung R&D Institute Poland

Re: exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks support for IOMMU

[Krzysztof Kozlowski]

On 31/10/2023 01:04, Mario Marietto wrote:
> Hello,
> 
> We are a team of linux enthusiasts who are trying to boot Xen on a
> Samsung XE303C12 Chromebook aka "snow"
> following the suggestions in the slide show presentation here:
> 
> https://www.slideshare.net/xen_com_mgr/xpds16-porting-xen-on-arm-to-a-new-soc-julien-grall-arm
> 
> This device uses an exynos5250 SOC dual core 1.7 GHz with 2 MB RAM, it is
> a Samsung armv7 chip with virtualization extensions.
> 
> In particular, we have it working fairly well both on the bare metal with
> a recent 6.1.59 Linux LTS kernel and also with a recent 5.4.257 LTS

Oh, these are old... Although there should be no noticeable changes
against v6.1.

v5.4 is not recent. It is four years old!

> kernel with KVM, the older LTS kernel version is used to test KVM because
> support for KVM on arm v7 was removed from Linux around kernel version
> 5.7. So we know we have the hypervisor mode enabled because we were able
> to use it with KVM.
> 
> For Xen, we are using the latest Debian build of Xen 4.17 for the Debian
> armhf architecture:

...

> 
> In this case,the kernel is able to use the exynos-drm kernel to start
> the fb0 device. But with Xen we get this error with exynos-drm:
> 
> devuan-bunsen kernel: [drm] Exynos DRM: using 14400000.fimd device for
> DMA mapping operations
> devuan-bunsen kernel: exynos-drm exynos-drm: bound 14400000.fimd (ops
> 0xc0d96354)
> devuan-bunsen kernel: exynos-mixer 14450000.mixer:
> [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks
> support for IOMMU

Hi Mario,

I don't test DRM on my Exynos boards, so my knowledge is limited, but
isn't this the answer?

> devuan-bunsen kernel: exynos-drm exynos-drm: failed to bind
> 14450000.mixer (ops 0xc0d97554): -22
> devuan-bunsen kernel: exynos-drm exynos-drm: adev bind failed: -22
> devuan-bunsen kernel: exynos-dp: probe of 145b0000.dp-controller
> failed with error -22

And that's the final confirmation that display did not probe successfully.

> 
> 
> Any ideas why booting the same Linux kernel that results in a working
> X.org display on the bare metal instead as dom0 on Xen would cause the
> display to remain dark, but most other basic functions would work, such
> as network, disk, and USB ? thanks.

BTW, it's usually good to Cc the maintainer as well, not only the
mailing list :). scripts/get_maintainer.pl will tell you the maintainers
of Exynos and Exynos DRM.

Best regards,
Krzysztof

Re: exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks support for IOMMU

[Stefano Stabellini]

Hi Mario,

I am adding xen-devel and a couple of other Xen maintainers that might
know how to help make progress on this issues.

Replies inline below.


On Tue, 31 Oct 2023, Mario Marietto wrote:
> Hello,
> 
> We are a team of linux enthusiasts who are trying to boot Xen on a
> Samsung XE303C12 Chromebook aka "snow"
> following the suggestions in the slide show presentation here:
> 
> https://www.slideshare.net/xen_com_mgr/xpds16-porting-xen-on-arm-to-a-new-soc-julien-grall-arm
> 
> This device uses an exynos5250 SOC dual core 1.7 GHz with 2 MB RAM, it is
> a Samsung armv7 chip with virtualization extensions.
> 
> In particular, we have it working fairly well both on the bare metal with
> a recent 6.1.59 Linux LTS kernel and also with a recent 5.4.257 LTS
> kernel with KVM, the older LTS kernel version is used to test KVM because
> support for KVM on arm v7 was removed from Linux around kernel version
> 5.7. So we know we have the hypervisor mode enabled because we were able
> to use it with KVM.
> 
> For Xen, we are using the latest Debian build of Xen 4.17 for the Debian
> armhf architecture:
> 
> (XEN) Xen version 4.17.2-pre (Debian 4.17.1+2-gb773c48e36-1)
> (pkg-xen-devel@xxxxxxxxxxxxxxx
> xxxxxxxx) (arm-linux-gnueabihf-gcc (Debian
> 12.2.0-14) 12.2.0) debug=n Thu May 18 19:26:30 UTC 2023
> 
> The Linux kernel is a custom build that adds the Xen config kernel
> options (CONFIG_XEN_DOM0, etc) on top of a kernel that works well on the
> same Chromebook model on the bare metal. I can provide the config options
> of the kernel that was used if that is helpful.
> 
> Our method of booting is to have u-boot boot the Xen hypervisor and load
> the device tree after adding the dom0 to the otherwise unaltered device
> tree from the Linux kernel using u-boot fdt commands to add a /chosen
> node, as described on the Xen wiki and in the pages linked from there. We
> have also tried adding and loading an initrd.img using the device tree
> /chosen node but that made no difference in our tests.
> 
> We actually have the Linux LTS kernel version 6.1.59 working as dom0 with
> Xen using the same version of u-boot that we used for KVM, but with a big
> problem.
> 
> The problem we see is that when booting the 6.1.59 kernel version as dom0
> with Xen, the screen is totally dark and the only way to access the
> system is remotely through ssh. Logs indicate most everything else is
> working, such as the wifi card so we can access it remotely via ssh and a
> USB optical mouse lights up when connected so USB is also working.
> Obviously, the disk is also working. The Chromebook is configured to boot
> from the device's SD card slot by turning on Chrome OS developer mode
> options to enable booting from the SD card slot.
> 
> The mystery is that when booting the exact same 6.1.59 kernel on the bare
> metal instead of booting it as dom0 on Xen, it boots up with full access
> to the screen and we can interact with the system using the X.org windows
> system. But booting as dom0 with Xen, the screen is totally dark and the
> only access we have to the system is through the network via ssh. Also,
> when booting the 5.4.257 kernel with KVM in hypervisor mode, the screen
> works and we can interact with the system through the X.org windows
> system.
> 
> Exploring the log file,we have seen the errors below :
> 
> 
> With Xen (or in bare metal):
> 
> devuan-bunsen kernel: [drm] Exynos DRM: using 14400000.fimd device for
> DMA mapping operations
> devuan-bunsen kernel: exynos-drm exynos-drm: bound 14400000.fimd (ops
> 0xc0d96354)
> devuan-bunsen kernel: exynos-drm exynos-drm: bound 14450000.mixer (ops
> 0xc0d97554)
> devuan-bunsen kernel: exynos-drm exynos-drm: bound
> 145b0000.dp-controller (ops 0xc0d97278)
> devuan-bunsen kernel: exynos-drm exynos-drm: bound 14530000.hdmi (ops
> 0xc0d97bd0)
> ...
> devuan-bunsen kernel: Console: switching to colour frame buffer device 170x48
> devuan-bunsen kernel: exynos-drm exynos-drm: [drm] fb0: exynosdrmfb
> frame buffer device
> devuan-bunsen kernel: [drm] Initialized exynos 1.1.0 20180330 for
> exynos-drm on minor 0
> 
> In this case,the kernel is able to use the exynos-drm kernel to start
> the fb0 device. But with Xen we get this error with exynos-drm:
> 
> devuan-bunsen kernel: [drm] Exynos DRM: using 14400000.fimd device for
> DMA mapping operations
> devuan-bunsen kernel: exynos-drm exynos-drm: bound 14400000.fimd (ops
> 0xc0d96354)
> devuan-bunsen kernel: exynos-mixer 14450000.mixer:
> [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks
> support for IOMMU
> devuan-bunsen kernel: exynos-drm exynos-drm: failed to bind
> 14450000.mixer (ops 0xc0d97554): -22
> devuan-bunsen kernel: exynos-drm exynos-drm: adev bind failed: -22
> devuan-bunsen kernel: exynos-dp: probe of 145b0000.dp-controller
> failed with error -22
> 
> 
> Any ideas why booting the same Linux kernel that results in a working
> X.org display on the bare metal instead as dom0 on Xen would cause the
> display to remain dark, but most other basic functions would work, such
> as network, disk, and USB ? thanks.

I think the issue is that Linux 6.1.59 is using the IOMMU to get the GPU
to work. When running on top of Xen, the IOMMU is not available to
Linux. That's because the IOMMU is used by Xen to enforce protection
between VMs.

It might help to rebuild Linux without IOMMU support (remove the driver
from the kconfig) so that Linux cannot attempt to use the IOMMU.

Also see
https://marc.info/?l=xen-users&m=169844593525492

Re: exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks support for IOMMU

[Mario Marietto]

I'm aware of the presence of that post. I'm working on the same
project with the guy who explained the problem. Unfortunately,the
solution proposed does not work well. Xen is working,but the screen is
still black.

On Wed, Nov 1, 2023 at 12:04 AM Stefano Stabellini
<sstabellini@kernel.org> wrote:
>
> Hi Mario,
>
> I am adding xen-devel and a couple of other Xen maintainers that might
> know how to help make progress on this issues.
>
> Replies inline below.
>
>
> On Tue, 31 Oct 2023, Mario Marietto wrote:
> > Hello,
> >
> > We are a team of linux enthusiasts who are trying to boot Xen on a
> > Samsung XE303C12 Chromebook aka "snow"
> > following the suggestions in the slide show presentation here:
> >
> > https://www.slideshare.net/xen_com_mgr/xpds16-porting-xen-on-arm-to-a-new-soc-julien-grall-arm
> >
> > This device uses an exynos5250 SOC dual core 1.7 GHz with 2 MB RAM, it is
> > a Samsung armv7 chip with virtualization extensions.
> >
> > In particular, we have it working fairly well both on the bare metal with
> > a recent 6.1.59 Linux LTS kernel and also with a recent 5.4.257 LTS
> > kernel with KVM, the older LTS kernel version is used to test KVM because
> > support for KVM on arm v7 was removed from Linux around kernel version
> > 5.7. So we know we have the hypervisor mode enabled because we were able
> > to use it with KVM.
> >
> > For Xen, we are using the latest Debian build of Xen 4.17 for the Debian
> > armhf architecture:
> >
> > (XEN) Xen version 4.17.2-pre (Debian 4.17.1+2-gb773c48e36-1)
> > (pkg-xen-devel@xxxxxxxxxxxxxxx
> > xxxxxxxx) (arm-linux-gnueabihf-gcc (Debian
> > 12.2.0-14) 12.2.0) debug=n Thu May 18 19:26:30 UTC 2023
> >
> > The Linux kernel is a custom build that adds the Xen config kernel
> > options (CONFIG_XEN_DOM0, etc) on top of a kernel that works well on the
> > same Chromebook model on the bare metal. I can provide the config options
> > of the kernel that was used if that is helpful.
> >
> > Our method of booting is to have u-boot boot the Xen hypervisor and load
> > the device tree after adding the dom0 to the otherwise unaltered device
> > tree from the Linux kernel using u-boot fdt commands to add a /chosen
> > node, as described on the Xen wiki and in the pages linked from there. We
> > have also tried adding and loading an initrd.img using the device tree
> > /chosen node but that made no difference in our tests.
> >
> > We actually have the Linux LTS kernel version 6.1.59 working as dom0 with
> > Xen using the same version of u-boot that we used for KVM, but with a big
> > problem.
> >
> > The problem we see is that when booting the 6.1.59 kernel version as dom0
> > with Xen, the screen is totally dark and the only way to access the
> > system is remotely through ssh. Logs indicate most everything else is
> > working, such as the wifi card so we can access it remotely via ssh and a
> > USB optical mouse lights up when connected so USB is also working.
> > Obviously, the disk is also working. The Chromebook is configured to boot
> > from the device's SD card slot by turning on Chrome OS developer mode
> > options to enable booting from the SD card slot.
> >
> > The mystery is that when booting the exact same 6.1.59 kernel on the bare
> > metal instead of booting it as dom0 on Xen, it boots up with full access
> > to the screen and we can interact with the system using the X.org windows
> > system. But booting as dom0 with Xen, the screen is totally dark and the
> > only access we have to the system is through the network via ssh. Also,
> > when booting the 5.4.257 kernel with KVM in hypervisor mode, the screen
> > works and we can interact with the system through the X.org windows
> > system.
> >
> > Exploring the log file,we have seen the errors below :
> >
> >
> > With Xen (or in bare metal):
> >
> > devuan-bunsen kernel: [drm] Exynos DRM: using 14400000.fimd device for
> > DMA mapping operations
> > devuan-bunsen kernel: exynos-drm exynos-drm: bound 14400000.fimd (ops
> > 0xc0d96354)
> > devuan-bunsen kernel: exynos-drm exynos-drm: bound 14450000.mixer (ops
> > 0xc0d97554)
> > devuan-bunsen kernel: exynos-drm exynos-drm: bound
> > 145b0000.dp-controller (ops 0xc0d97278)
> > devuan-bunsen kernel: exynos-drm exynos-drm: bound 14530000.hdmi (ops
> > 0xc0d97bd0)
> > ...
> > devuan-bunsen kernel: Console: switching to colour frame buffer device 170x48
> > devuan-bunsen kernel: exynos-drm exynos-drm: [drm] fb0: exynosdrmfb
> > frame buffer device
> > devuan-bunsen kernel: [drm] Initialized exynos 1.1.0 20180330 for
> > exynos-drm on minor 0
> >
> > In this case,the kernel is able to use the exynos-drm kernel to start
> > the fb0 device. But with Xen we get this error with exynos-drm:
> >
> > devuan-bunsen kernel: [drm] Exynos DRM: using 14400000.fimd device for
> > DMA mapping operations
> > devuan-bunsen kernel: exynos-drm exynos-drm: bound 14400000.fimd (ops
> > 0xc0d96354)
> > devuan-bunsen kernel: exynos-mixer 14450000.mixer:
> > [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks
> > support for IOMMU
> > devuan-bunsen kernel: exynos-drm exynos-drm: failed to bind
> > 14450000.mixer (ops 0xc0d97554): -22
> > devuan-bunsen kernel: exynos-drm exynos-drm: adev bind failed: -22
> > devuan-bunsen kernel: exynos-dp: probe of 145b0000.dp-controller
> > failed with error -22
> >
> >
> > Any ideas why booting the same Linux kernel that results in a working
> > X.org display on the bare metal instead as dom0 on Xen would cause the
> > display to remain dark, but most other basic functions would work, such
> > as network, disk, and USB ? thanks.
>
> I think the issue is that Linux 6.1.59 is using the IOMMU to get the GPU
> to work. When running on top of Xen, the IOMMU is not available to
> Linux. That's because the IOMMU is used by Xen to enforce protection
> between VMs.
>
> It might help to rebuild Linux without IOMMU support (remove the driver
> from the kconfig) so that Linux cannot attempt to use the IOMMU.
>
> Also see
> https://marc.info/?l=xen-users&m=169844593525492



--
Mario.

Re: exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks support for IOMMU

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 7892 bytes --]

Unfortunately there is no easy solution.

Do you know the version of the SMMU available on the platform? If it is
a SMMUv3 you can try to use the nested SMMU patch series to enable a
virtual SMMU in Dom0: https://marc.info/?l=xen-devel&m=166991020831005
That way, Xen can use the SMMU to protect VMs, and Dom0 can also use the
SMMU for its own purposes at the same time.

Alternatively, you can dig into the details of the exynos-drm driver to
see what exactly is the dependency on the IOMMU framework in Linux and
remove the dependency. Unfortunately none of us in this thread are
expert on exynos-drm so it would be difficult to advise on how to do
that. For example, I don't know how you could debug the x11 problem you
described because I don't typically work with x11 or with the exynos. If
there is an open source mailing list for exynos-drm development they
might be able to advise on how to remove the IOMMU dependency there.

The final option, which is a gross hack, would be to let Dom0 use the
IOMMU for its own gain. Xen doesn't use the IOMMU. If you do that you
lose freedom from interference between the VMs and you cannot run driver
domains or directly assign devices to DomUs. But if you are running a
research project you might be OK with that. To get it to work, you need
to hack Xen so that it remaps the IOMMU to Dom0 to let Dom0 program it
directly. The attached patch (untested) would be a place to start. You
also need to pass iommu=false to the Xen command line to prevent Xen
from using the iommu itself.

Cheers,

Stefano


On Wed, 1 Nov 2023, Mario Marietto wrote:
> I'm aware of the presence of that post. I'm working on the same
> project with the guy who explained the problem. Unfortunately,the
> solution proposed does not work well. Xen is working,but the screen is
> still black.
> 
> On Wed, Nov 1, 2023 at 12:04 AM Stefano Stabellini
> <sstabellini@kernel.org> wrote:
> >
> > Hi Mario,
> >
> > I am adding xen-devel and a couple of other Xen maintainers that might
> > know how to help make progress on this issues.
> >
> > Replies inline below.
> >
> >
> > On Tue, 31 Oct 2023, Mario Marietto wrote:
> > > Hello,
> > >
> > > We are a team of linux enthusiasts who are trying to boot Xen on a
> > > Samsung XE303C12 Chromebook aka "snow"
> > > following the suggestions in the slide show presentation here:
> > >
> > > https://www.slideshare.net/xen_com_mgr/xpds16-porting-xen-on-arm-to-a-new-soc-julien-grall-arm
> > >
> > > This device uses an exynos5250 SOC dual core 1.7 GHz with 2 MB RAM, it is
> > > a Samsung armv7 chip with virtualization extensions.
> > >
> > > In particular, we have it working fairly well both on the bare metal with
> > > a recent 6.1.59 Linux LTS kernel and also with a recent 5.4.257 LTS
> > > kernel with KVM, the older LTS kernel version is used to test KVM because
> > > support for KVM on arm v7 was removed from Linux around kernel version
> > > 5.7. So we know we have the hypervisor mode enabled because we were able
> > > to use it with KVM.
> > >
> > > For Xen, we are using the latest Debian build of Xen 4.17 for the Debian
> > > armhf architecture:
> > >
> > > (XEN) Xen version 4.17.2-pre (Debian 4.17.1+2-gb773c48e36-1)
> > > (pkg-xen-devel@xxxxxxxxxxxxxxx
> > > xxxxxxxx) (arm-linux-gnueabihf-gcc (Debian
> > > 12.2.0-14) 12.2.0) debug=n Thu May 18 19:26:30 UTC 2023
> > >
> > > The Linux kernel is a custom build that adds the Xen config kernel
> > > options (CONFIG_XEN_DOM0, etc) on top of a kernel that works well on the
> > > same Chromebook model on the bare metal. I can provide the config options
> > > of the kernel that was used if that is helpful.
> > >
> > > Our method of booting is to have u-boot boot the Xen hypervisor and load
> > > the device tree after adding the dom0 to the otherwise unaltered device
> > > tree from the Linux kernel using u-boot fdt commands to add a /chosen
> > > node, as described on the Xen wiki and in the pages linked from there. We
> > > have also tried adding and loading an initrd.img using the device tree
> > > /chosen node but that made no difference in our tests.
> > >
> > > We actually have the Linux LTS kernel version 6.1.59 working as dom0 with
> > > Xen using the same version of u-boot that we used for KVM, but with a big
> > > problem.
> > >
> > > The problem we see is that when booting the 6.1.59 kernel version as dom0
> > > with Xen, the screen is totally dark and the only way to access the
> > > system is remotely through ssh. Logs indicate most everything else is
> > > working, such as the wifi card so we can access it remotely via ssh and a
> > > USB optical mouse lights up when connected so USB is also working.
> > > Obviously, the disk is also working. The Chromebook is configured to boot
> > > from the device's SD card slot by turning on Chrome OS developer mode
> > > options to enable booting from the SD card slot.
> > >
> > > The mystery is that when booting the exact same 6.1.59 kernel on the bare
> > > metal instead of booting it as dom0 on Xen, it boots up with full access
> > > to the screen and we can interact with the system using the X.org windows
> > > system. But booting as dom0 with Xen, the screen is totally dark and the
> > > only access we have to the system is through the network via ssh. Also,
> > > when booting the 5.4.257 kernel with KVM in hypervisor mode, the screen
> > > works and we can interact with the system through the X.org windows
> > > system.
> > >
> > > Exploring the log file,we have seen the errors below :
> > >
> > >
> > > With Xen (or in bare metal):
> > >
> > > devuan-bunsen kernel: [drm] Exynos DRM: using 14400000.fimd device for
> > > DMA mapping operations
> > > devuan-bunsen kernel: exynos-drm exynos-drm: bound 14400000.fimd (ops
> > > 0xc0d96354)
> > > devuan-bunsen kernel: exynos-drm exynos-drm: bound 14450000.mixer (ops
> > > 0xc0d97554)
> > > devuan-bunsen kernel: exynos-drm exynos-drm: bound
> > > 145b0000.dp-controller (ops 0xc0d97278)
> > > devuan-bunsen kernel: exynos-drm exynos-drm: bound 14530000.hdmi (ops
> > > 0xc0d97bd0)
> > > ...
> > > devuan-bunsen kernel: Console: switching to colour frame buffer device 170x48
> > > devuan-bunsen kernel: exynos-drm exynos-drm: [drm] fb0: exynosdrmfb
> > > frame buffer device
> > > devuan-bunsen kernel: [drm] Initialized exynos 1.1.0 20180330 for
> > > exynos-drm on minor 0
> > >
> > > In this case,the kernel is able to use the exynos-drm kernel to start
> > > the fb0 device. But with Xen we get this error with exynos-drm:
> > >
> > > devuan-bunsen kernel: [drm] Exynos DRM: using 14400000.fimd device for
> > > DMA mapping operations
> > > devuan-bunsen kernel: exynos-drm exynos-drm: bound 14400000.fimd (ops
> > > 0xc0d96354)
> > > devuan-bunsen kernel: exynos-mixer 14450000.mixer:
> > > [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks
> > > support for IOMMU
> > > devuan-bunsen kernel: exynos-drm exynos-drm: failed to bind
> > > 14450000.mixer (ops 0xc0d97554): -22
> > > devuan-bunsen kernel: exynos-drm exynos-drm: adev bind failed: -22
> > > devuan-bunsen kernel: exynos-dp: probe of 145b0000.dp-controller
> > > failed with error -22
> > >
> > >
> > > Any ideas why booting the same Linux kernel that results in a working
> > > X.org display on the bare metal instead as dom0 on Xen would cause the
> > > display to remain dark, but most other basic functions would work, such
> > > as network, disk, and USB ? thanks.
> >
> > I think the issue is that Linux 6.1.59 is using the IOMMU to get the GPU
> > to work. When running on top of Xen, the IOMMU is not available to
> > Linux. That's because the IOMMU is used by Xen to enforce protection
> > between VMs.
> >
> > It might help to rebuild Linux without IOMMU support (remove the driver
> > from the kconfig) so that Linux cannot attempt to use the IOMMU.
> >
> > Also see
> > https://marc.info/?l=xen-users&m=169844593525492

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-diff; name=dom0-iommu.patch, Size: 2449 bytes --]

diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 49792dd590..74fc554364 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -1119,25 +1119,10 @@ static int __init write_properties(struct domain *d, struct kernel_info *kinfo,
     const struct dt_property *prop, *status = NULL;
     int res = 0;
     int had_dom0_bootargs = 0;
-    struct dt_device_node *iommu_node;
 
     if ( kinfo->cmdline && kinfo->cmdline[0] )
         bootargs = &kinfo->cmdline[0];
 
-    /*
-     * We always skip the IOMMU device when creating DT for hwdom if there is
-     * an appropriate driver for it in Xen (device_get_class(iommu_node)
-     * returns DEVICE_IOMMU).
-     * We should also skip the IOMMU specific properties of the master device
-     * behind that IOMMU in order to avoid exposing an half complete IOMMU
-     * bindings to hwdom.
-     * Use "iommu_node" as an indicator of the master device which properties
-     * should be skipped.
-     */
-    iommu_node = dt_parse_phandle(node, "iommus", 0);
-    if ( iommu_node && device_get_class(iommu_node) != DEVICE_IOMMU )
-        iommu_node = NULL;
-
     dt_for_each_property_node (node, prop)
     {
         const void *prop_data = prop->value;
@@ -1195,19 +1180,6 @@ static int __init write_properties(struct domain *d, struct kernel_info *kinfo,
             continue;
         }
 
-        if ( iommu_node )
-        {
-            /* Don't expose IOMMU specific properties to hwdom */
-            if ( dt_property_name_is_equal(prop, "iommus") )
-                continue;
-
-            if ( dt_property_name_is_equal(prop, "iommu-map") )
-                continue;
-
-            if ( dt_property_name_is_equal(prop, "iommu-map-mask") )
-                continue;
-        }
-
         res = fdt_property(kinfo->fdt, prop->name, prop_data, prop_len);
 
         if ( res )
@@ -2372,16 +2344,6 @@ static int __init handle_node(struct domain *d, struct kernel_info *kinfo,
         return 0;
     }
 
-    /*
-     * Even if the IOMMU device is not used by Xen, it should not be
-     * passthrough to DOM0
-     */
-    if ( device_get_class(node) == DEVICE_IOMMU )
-    {
-        dt_dprintk(" IOMMU, skip it\n");
-        return 0;
-    }
-
     /*
      * The vGIC does not support routing hardware PPIs to guest. So
      * we need to skip any node using PPIs.

Re: exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks support for IOMMU

[Chuck Zmudzinski]

On 10/31/2023 7:45 PM, Stefano Stabellini wrote:
> Unfortunately there is no easy solution.
> 
> Do you know the version of the SMMU available on the platform? 

I am trying to discern, but I doubt we have v3 because we are
working on a very old chromebook from 2012, and I am finding
patches for smmv3 in Linux not starting until 2015. It is good to
know about this option, though, for future work we might do on newer
devices.

> If it is a SMMUv3 you can try to use the nested SMMU patch series to
> enable a virtual SMMU in Dom0: https://marc.info/?l=xen-devel&m=166991020831005
> That way, Xen can use the SMMU to protect VMs, and Dom0 can also use the
> SMMU for its own purposes at the same time.
> 
> Alternatively, you can dig into the details of the exynos-drm driver to
> see what exactly is the dependency on the IOMMU framework in Linux and
> remove the dependency. Unfortunately none of us in this thread are
> expert on exynos-drm so it would be difficult to advise on how to do
> that. For example, I don't know how you could debug the x11 problem you
> described because I don't typically work with x11 or with the exynos. If
> there is an open source mailing list for exynos-drm development they
> might be able to advise on how to remove the IOMMU dependency there.

We have received this message from Marek Szyprowski of Samsung:

https://lore.kernel.org/lkml/7a71e348-f892-4fd4-8857-b72f35ab5134@samsung.com/

Marek recommends this patch to possibly help with this issue:

https://github.com/torvalds/linux/commit/bbc4d205d93f52ee18dfa7858d51489c0506547f

and also these kernel config settings:

On 10/31/2023 8:08 AM, Marek Szyprowski wrote:
> If not, then as a temporary workaround please disable 
> CONFIG_DRM_EXYNOS_MIXER and CONFIG_DRM_EXYNOS_HDMI in your kernel config 
> and check what will happen (You will lose the HDMI output, but maybe 
> this won't a big issue).

Mario and I have preliminary evidence that applying both of Marek's
recommendations to the 6.1.59 kernel have improved the situation to
the point where now the Chromebook can run X.org on Xen. We are doing
further tests to see how Marek's patch and/or the kernel config settings
to disable the mixer and the HDMI affect the behavior of the GPU in dom0
on Xen.

> 
> The final option, which is a gross hack, would be to let Dom0 use the
> IOMMU for its own gain. Xen doesn't use the IOMMU. If you do that you
> lose freedom from interference between the VMs and you cannot run driver
> domains or directly assign devices to DomUs. But if you are running a
> research project you might be OK with that. To get it to work, you need
> to hack Xen so that it remaps the IOMMU to Dom0 to let Dom0 program it
> directly. The attached patch (untested) would be a place to start. You
> also need to pass iommu=false to the Xen command line to prevent Xen
> from using the iommu itself.

I am interested to investigate if only the mixer and the HDMI is causing
the trouble. Based on what you are telling me about Xen not exposing the
IOMMU to dom0, I don't fully understand the original log messages I was
getting when I followed Julien's suggestion to find the symbols associated
with each address in the original message, and those seemed to indicate that
the exynos_drm device was using the IOMMU in dom0, but the mixer was not,
and the fact that they both were not using the IOMMU is what caused the
test to fail and Linux refuse to initialize the GPU on dom0, whereas on
bare metal, the logs indicated both the exynos mixer, which I think is a
sub device of the exynos_drm, and the exynos_drm, use the IOMMU on bare
metal.

I also found this patch which suggests if we can get the devices to work
we will be compromising the security and isolation between guests:

https://patchwork.kernel.org/project/linux-arm-kernel/patch/20190301192017.39770-1-dianders@chromium.org/

There are plenty of unanswered questions here in my mind,

Cheers,

Chuck

> 
> Cheers,
> 
> Stefano
> 
> 
> On Wed, 1 Nov 2023, Mario Marietto wrote:
>> I'm aware of the presence of that post. I'm working on the same
>> project with the guy who explained the problem. Unfortunately,the
>> solution proposed does not work well. Xen is working,but the screen is
>> still black.
>> 
>> On Wed, Nov 1, 2023 at 12:04 AM Stefano Stabellini
>> <sstabellini@kernel.org> wrote:
>> >
>> > Hi Mario,
>> >
>> > I am adding xen-devel and a couple of other Xen maintainers that might
>> > know how to help make progress on this issues.
>> >
>> > Replies inline below.
>> >
>> >
>> > On Tue, 31 Oct 2023, Mario Marietto wrote:
>> > > Hello,
>> > >
>> > > We are a team of linux enthusiasts who are trying to boot Xen on a
>> > > Samsung XE303C12 Chromebook aka "snow"
>> > > following the suggestions in the slide show presentation here:
>> > >
>> > > https://www.slideshare.net/xen_com_mgr/xpds16-porting-xen-on-arm-to-a-new-soc-julien-grall-arm
>> > >
>> > > This device uses an exynos5250 SOC dual core 1.7 GHz with 2 MB RAM, it is
>> > > a Samsung armv7 chip with virtualization extensions.
>> > >
>> > > In particular, we have it working fairly well both on the bare metal with
>> > > a recent 6.1.59 Linux LTS kernel and also with a recent 5.4.257 LTS
>> > > kernel with KVM, the older LTS kernel version is used to test KVM because
>> > > support for KVM on arm v7 was removed from Linux around kernel version
>> > > 5.7. So we know we have the hypervisor mode enabled because we were able
>> > > to use it with KVM.
>> > >
>> > > For Xen, we are using the latest Debian build of Xen 4.17 for the Debian
>> > > armhf architecture:
>> > >
>> > > (XEN) Xen version 4.17.2-pre (Debian 4.17.1+2-gb773c48e36-1)
>> > > (pkg-xen-devel@xxxxxxxxxxxxxxx
>> > > xxxxxxxx) (arm-linux-gnueabihf-gcc (Debian
>> > > 12.2.0-14) 12.2.0) debug=n Thu May 18 19:26:30 UTC 2023
>> > >
>> > > The Linux kernel is a custom build that adds the Xen config kernel
>> > > options (CONFIG_XEN_DOM0, etc) on top of a kernel that works well on the
>> > > same Chromebook model on the bare metal. I can provide the config options
>> > > of the kernel that was used if that is helpful.
>> > >
>> > > Our method of booting is to have u-boot boot the Xen hypervisor and load
>> > > the device tree after adding the dom0 to the otherwise unaltered device
>> > > tree from the Linux kernel using u-boot fdt commands to add a /chosen
>> > > node, as described on the Xen wiki and in the pages linked from there. We
>> > > have also tried adding and loading an initrd.img using the device tree
>> > > /chosen node but that made no difference in our tests.
>> > >
>> > > We actually have the Linux LTS kernel version 6.1.59 working as dom0 with
>> > > Xen using the same version of u-boot that we used for KVM, but with a big
>> > > problem.
>> > >
>> > > The problem we see is that when booting the 6.1.59 kernel version as dom0
>> > > with Xen, the screen is totally dark and the only way to access the
>> > > system is remotely through ssh. Logs indicate most everything else is
>> > > working, such as the wifi card so we can access it remotely via ssh and a
>> > > USB optical mouse lights up when connected so USB is also working.
>> > > Obviously, the disk is also working. The Chromebook is configured to boot
>> > > from the device's SD card slot by turning on Chrome OS developer mode
>> > > options to enable booting from the SD card slot.
>> > >
>> > > The mystery is that when booting the exact same 6.1.59 kernel on the bare
>> > > metal instead of booting it as dom0 on Xen, it boots up with full access
>> > > to the screen and we can interact with the system using the X.org windows
>> > > system. But booting as dom0 with Xen, the screen is totally dark and the
>> > > only access we have to the system is through the network via ssh. Also,
>> > > when booting the 5.4.257 kernel with KVM in hypervisor mode, the screen
>> > > works and we can interact with the system through the X.org windows
>> > > system.
>> > >
>> > > Exploring the log file,we have seen the errors below :
>> > >
>> > >
>> > > With Xen (or in bare metal):
>> > >
>> > > devuan-bunsen kernel: [drm] Exynos DRM: using 14400000.fimd device for
>> > > DMA mapping operations
>> > > devuan-bunsen kernel: exynos-drm exynos-drm: bound 14400000.fimd (ops
>> > > 0xc0d96354)
>> > > devuan-bunsen kernel: exynos-drm exynos-drm: bound 14450000.mixer (ops
>> > > 0xc0d97554)
>> > > devuan-bunsen kernel: exynos-drm exynos-drm: bound
>> > > 145b0000.dp-controller (ops 0xc0d97278)
>> > > devuan-bunsen kernel: exynos-drm exynos-drm: bound 14530000.hdmi (ops
>> > > 0xc0d97bd0)
>> > > ...
>> > > devuan-bunsen kernel: Console: switching to colour frame buffer device 170x48
>> > > devuan-bunsen kernel: exynos-drm exynos-drm: [drm] fb0: exynosdrmfb
>> > > frame buffer device
>> > > devuan-bunsen kernel: [drm] Initialized exynos 1.1.0 20180330 for
>> > > exynos-drm on minor 0
>> > >
>> > > In this case,the kernel is able to use the exynos-drm kernel to start
>> > > the fb0 device. But with Xen we get this error with exynos-drm:
>> > >
>> > > devuan-bunsen kernel: [drm] Exynos DRM: using 14400000.fimd device for
>> > > DMA mapping operations
>> > > devuan-bunsen kernel: exynos-drm exynos-drm: bound 14400000.fimd (ops
>> > > 0xc0d96354)
>> > > devuan-bunsen kernel: exynos-mixer 14450000.mixer:
>> > > [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks
>> > > support for IOMMU
>> > > devuan-bunsen kernel: exynos-drm exynos-drm: failed to bind
>> > > 14450000.mixer (ops 0xc0d97554): -22
>> > > devuan-bunsen kernel: exynos-drm exynos-drm: adev bind failed: -22
>> > > devuan-bunsen kernel: exynos-dp: probe of 145b0000.dp-controller
>> > > failed with error -22
>> > >
>> > >
>> > > Any ideas why booting the same Linux kernel that results in a working
>> > > X.org display on the bare metal instead as dom0 on Xen would cause the
>> > > display to remain dark, but most other basic functions would work, such
>> > > as network, disk, and USB ? thanks.
>> >
>> > I think the issue is that Linux 6.1.59 is using the IOMMU to get the GPU
>> > to work. When running on top of Xen, the IOMMU is not available to
>> > Linux. That's because the IOMMU is used by Xen to enforce protection
>> > between VMs.
>> >
>> > It might help to rebuild Linux without IOMMU support (remove the driver
>> > from the kconfig) so that Linux cannot attempt to use the IOMMU.
>> >
>> > Also see
>> > https://marc.info/?l=xen-users&m=169844593525492

Re: exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks support for IOMMU

[Chuck Zmudzinski]

On 10/31/2023 8:08 AM, Marek Szyprowski wrote:
> Hi,
> 
> On 31.10.2023 00:03, Mario Marietto wrote:
>> We are a team of linux enthusiasts who are trying to boot Xen on a 
>> Samsung XE303C12 Chromebook aka "snow" following the suggestions in 
>> the slide show presentation here: 
>> https://www.slideshare.net/xen_com_mgr/xpds16-porting-xen-on-arm-to-a-new-soc-julien-grall-arm 
>> This device uses an exynos5250 SOC dual core 1.7 GHz with 2 MB RAM, it 
>> is a Samsung armv7 chip with virtualization extensions. In particular, 
>> we have it working fairly well both on the bare metal with a recent 
>> 6.1.59 Linux LTS kernel and also with a recent 5.4.257 LTS kernel with 
>> KVM, the older LTS kernel version is used to test KVM because support 
>> for KVM on arm v7 was removed from Linux around kernel version 5.7. So 
>> we know we have the hypervisor mode enabled because we were able to 
>> use it with KVM. For Xen, we are using the latest Debian build of Xen 
>> 4.17 for the Debian armhf architecture: (XEN) Xen version 4.17.2-pre 
>> (Debian 4.17.1+2-gb773c48e36-1) 
>> (pkg-xen-devel@xxxxxxxxxxxxxxxxxxxxxxx) (arm-linux-gnueabihf-gcc 
>> (Debian 12.2.0-14) 12.2.0) debug=n Thu May 18 19:26:30 UTC 2023 The 
>> Linux kernel is a custom build that adds the Xen config kernel options 
>> (CONFIG_XEN_DOM0, etc) on top of a kernel that works well on the same 
>> Chromebook model on the bare metal. I can provide the config options 
>> of the kernel that was used if that is helpful. Our method of booting 
>> is to have u-boot boot the Xen hypervisor and load the device tree 
>> after adding the dom0 to the otherwise unaltered device tree from the 
>> Linux kernel using u-boot fdt commands to add a /chosen node, as 
>> described on the Xen wiki and in the pages linked from there. We have 
>> also tried adding and loading an initrd.img using the device tree 
>> /chosen node but that made no difference in our tests. We actually 
>> have the Linux LTS kernel version 6.1.59 working as dom0 with Xen 
>> using the same version of u-boot that we used for KVM, but with a big 
>> problem. The problem we see is that when booting the 6.1.59 kernel 
>> version as dom0 with Xen, the screen is totally dark and the only way 
>> to access the system is remotely through ssh. Logs indicate most 
>> everything else is working, such as the wifi card so we can access it 
>> remotely via ssh and a USB optical mouse lights up when connected so 
>> USB is also working. Obviously, the disk is also working. The 
>> Chromebook is configured to boot from the device's SD card slot by 
>> turning on Chrome OS developer mode options to enable booting from the 
>> SD card slot. The mystery is that when booting the exact same 6.1.59 
>> kernel on the bare metal instead of booting it as dom0 on Xen, it 
>> boots up with full access to the screen and we can interact with the 
>> system using the X.org windows system. But booting as dom0 with Xen, 
>> the screen is totally dark and the only access we have to the system 
>> is through the network via ssh. Also, when booting the 5.4.257 kernel 
>> with KVM in hypervisor mode, the screen works and we can interact with 
>> the system through the X.org windows system. Exploring the log file,we 
>> have seen the errors below :
>>
>> Without Xen (or in bare metal):
>>
>> devuan-bunsen kernel: [drm] Exynos DRM: using 14400000.fimd device for 
>> DMA mapping operations
>> devuan-bunsen kernel: exynos-drm exynos-drm: bound 14400000.fimd (ops 
>> 0xc0d96354)
>> devuan-bunsen kernel: exynos-drm exynos-drm: bound 14450000.mixer (ops 
>> 0xc0d97554)
>> devuan-bunsen kernel: exynos-drm exynos-drm: bound 
>> 145b0000.dp-controller (ops 0xc0d97278)
>> devuan-bunsen kernel: exynos-drm exynos-drm: bound 14530000.hdmi (ops 
>> 0xc0d97bd0)
>> ...
>> devuan-bunsen kernel: Console: switching to colour frame buffer device 
>> 170x48
>> devuan-bunsen kernel: exynos-drm exynos-drm: [drm] fb0: exynosdrmfb 
>> frame buffer device
>> devuan-bunsen kernel: [drm] Initialized exynos 1.1.0 20180330 for 
>> exynos-drm on minor 0
>>
>> In this case,the kernel is able to use the exynos-drm kernel to start 
>> the fb0 device. But with Xen we get this error with exynos-drm:
>>
>> devuan-bunsen kernel: [drm] Exynos DRM: using 14400000.fimd device for 
>> DMA mapping operations
>> devuan-bunsen kernel: exynos-drm exynos-drm: bound 14400000.fimd (ops 
>> 0xc0d96354)
>> devuan-bunsen kernel: exynos-mixer 14450000.mixer: 
>> [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks 
>> support for IOMMU
>> devuan-bunsen kernel: exynos-drm exynos-drm: failed to bind 
>> 14450000.mixer (ops 0xc0d97554): -22
>> devuan-bunsen kernel: exynos-drm exynos-drm: adev bind failed: -22
>> devuan-bunsen kernel: exynos-dp: probe of 145b0000.dp-controller 
>> failed with error -22
>>
>> I'm trying to find for a solution and I've googled a little bit and I 
>> found this web site : 
>> https://lore.kernel.org/linux-arm-kernel/20220208171823.226211-8-krzysztof.kozlowski@canonical.com/ 
>> with your email address and I tried to ask for some help for fixing 
>> the bug. Any ideas why booting the same Linux kernel that results in a 
>> working X.org display on the bare metal instead as dom0 on Xen would 
>> cause the display to remain dark, but most other basic functions would 
>> work, such as network, disk, and USB ? thanks.
> 
> 
> Thanks for the detailed description! Good to hear that those boards are 
> still being used for various projects. I also have Snow Chromebook and 
> use it for daily tests of linux-next branch.

Adding Julien Grall and Stefano Stabellini

Hi Marek,

Thanks for responding to Mario's question. I also have been doing these
experiments with a Chromebook Snow, and I am the one who reported this
problem on the xen-users ML here:

https://lists.xenproject.org/archives/html/xen-users/2023-10/msg00021.html

You might find that thread interesting, especially here with some additional
log messages from the exynos_drm driver (exynos_drm_dma.c, I believe):

https://lists.xenproject.org/archives/html/xen-users/2023-10/msg00032.html

This issue is also discussed some on the xen-devel ML here:

https://lists.xenproject.org/archives/html/xen-devel/2023-11/msg00003.html

> 
> Frankly speaking I have no idea what might happen wrong. There have been 
> some changes recently in the Exynos IOMMU driver related to 
> initialization, maybe your changes related to Xen enabling changed 
> somehow the order of device initialization during boot. I assume that 
> the device-tree you use for the bare metal run and Xen enabled run 
> doesn't differ in the areas describing the hardware blocks.
> 
> Please check if cherry-picking the commit 
> https://github.com/torvalds/linux/commit/bbc4d205d93f52ee18dfa7858d51489c0506547f 
> to your v6.1.59 based kernel helps anyhow.

I tried adding that fix of the exynos IOMMU initialization from
Linux > 6.2 on top of the 6.1.59 kernel I used for the original report,
but that made no difference on Xen - it still failed with the mixer lacks
support for IOMMU message and the screen is totally dark.

> 
> If not, then as a temporary workaround please disable 
> CONFIG_DRM_EXYNOS_MIXER and CONFIG_DRM_EXYNOS_HDMI in your kernel config 
> and check what will happen (You will lose the HDMI output, but maybe 
> this won't a big issue).

This change causes the GPU to work fairly well AFAICS. Removing the mixer
and HDMI allowed the GPU to initialize, and the display manager started
normally and enabled logging into an ordinary X11 session. Based on the log
messages I was seeing, this was an obvious thing to try. Thanks for
suggesting it.

But I have a question:

How are the mixer and HDMI devices related to the main drm device? The problem
in the exynos_drm_dma driver was that on Xen, the main drm device wanted to
use IOMMU version of dma_ops, but the mixer (and probably also the HDMI if
it wouldn't have exited first) wanted to use the Xen swiotlb version of dma_ops,
but on bare metal all three devices want to use the IOMMU version of dma_ops.

The problem obviously has something to do with the fact that Xen does not
expose the same IOMMU capability to Linux as is available when running on
the bare metal.

Cheers,

> 
> 
> Best regards

Re: exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks support for IOMMU

[Chuck Zmudzinski]

On 10/31/2023 8:08 AM, Marek Szyprowski wrote:
> Hi,
> 
> On 31.10.2023 00:03, Mario Marietto wrote:
>> We are a team of linux enthusiasts who are trying to boot Xen on a 
>> Samsung XE303C12 Chromebook aka "snow" following the suggestions in 
>> the slide show presentation here: 
>> https://www.slideshare.net/xen_com_mgr/xpds16-porting-xen-on-arm-to-a-new-soc-julien-grall-arm 
>> This device uses an exynos5250 SOC dual core 1.7 GHz with 2 MB RAM, it 
>> is a Samsung armv7 chip with virtualization extensions. In particular, 
>> we have it working fairly well both on the bare metal with a recent 
>> 6.1.59 Linux LTS kernel and also with a recent 5.4.257 LTS kernel with 
>> KVM, the older LTS kernel version is used to test KVM because support 
>> for KVM on arm v7 was removed from Linux around kernel version 5.7. So 
>> we know we have the hypervisor mode enabled because we were able to 
>> use it with KVM. For Xen, we are using the latest Debian build of Xen 
>> 4.17 for the Debian armhf architecture: (XEN) Xen version 4.17.2-pre 
>> (Debian 4.17.1+2-gb773c48e36-1) 
>> (pkg-xen-devel@xxxxxxxxxxxxxxxxxxxxxxx) (arm-linux-gnueabihf-gcc 
>> (Debian 12.2.0-14) 12.2.0) debug=n Thu May 18 19:26:30 UTC 2023 The 
>> Linux kernel is a custom build that adds the Xen config kernel options 
>> (CONFIG_XEN_DOM0, etc) on top of a kernel that works well on the same 
>> Chromebook model on the bare metal. I can provide the config options 
>> of the kernel that was used if that is helpful. Our method of booting 
>> is to have u-boot boot the Xen hypervisor and load the device tree 
>> after adding the dom0 to the otherwise unaltered device tree from the 
>> Linux kernel using u-boot fdt commands to add a /chosen node, as 
>> described on the Xen wiki and in the pages linked from there. We have 
>> also tried adding and loading an initrd.img using the device tree 
>> /chosen node but that made no difference in our tests. We actually 
>> have the Linux LTS kernel version 6.1.59 working as dom0 with Xen 
>> using the same version of u-boot that we used for KVM, but with a big 
>> problem. The problem we see is that when booting the 6.1.59 kernel 
>> version as dom0 with Xen, the screen is totally dark and the only way 
>> to access the system is remotely through ssh. Logs indicate most 
>> everything else is working, such as the wifi card so we can access it 
>> remotely via ssh and a USB optical mouse lights up when connected so 
>> USB is also working. Obviously, the disk is also working. The 
>> Chromebook is configured to boot from the device's SD card slot by 
>> turning on Chrome OS developer mode options to enable booting from the 
>> SD card slot. The mystery is that when booting the exact same 6.1.59 
>> kernel on the bare metal instead of booting it as dom0 on Xen, it 
>> boots up with full access to the screen and we can interact with the 
>> system using the X.org windows system. But booting as dom0 with Xen, 
>> the screen is totally dark and the only access we have to the system 
>> is through the network via ssh. Also, when booting the 5.4.257 kernel 
>> with KVM in hypervisor mode, the screen works and we can interact with 
>> the system through the X.org windows system. Exploring the log file,we 
>> have seen the errors below :
>>
>> Without Xen (or in bare metal):
>>
>> devuan-bunsen kernel: [drm] Exynos DRM: using 14400000.fimd device for 
>> DMA mapping operations
>> devuan-bunsen kernel: exynos-drm exynos-drm: bound 14400000.fimd (ops 
>> 0xc0d96354)
>> devuan-bunsen kernel: exynos-drm exynos-drm: bound 14450000.mixer (ops 
>> 0xc0d97554)
>> devuan-bunsen kernel: exynos-drm exynos-drm: bound 
>> 145b0000.dp-controller (ops 0xc0d97278)
>> devuan-bunsen kernel: exynos-drm exynos-drm: bound 14530000.hdmi (ops 
>> 0xc0d97bd0)
>> ...
>> devuan-bunsen kernel: Console: switching to colour frame buffer device 
>> 170x48
>> devuan-bunsen kernel: exynos-drm exynos-drm: [drm] fb0: exynosdrmfb 
>> frame buffer device
>> devuan-bunsen kernel: [drm] Initialized exynos 1.1.0 20180330 for 
>> exynos-drm on minor 0
>>
>> In this case,the kernel is able to use the exynos-drm kernel to start 
>> the fb0 device. But with Xen we get this error with exynos-drm:
>>
>> devuan-bunsen kernel: [drm] Exynos DRM: using 14400000.fimd device for 
>> DMA mapping operations
>> devuan-bunsen kernel: exynos-drm exynos-drm: bound 14400000.fimd (ops 
>> 0xc0d96354)
>> devuan-bunsen kernel: exynos-mixer 14450000.mixer: 
>> [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks 
>> support for IOMMU
>> devuan-bunsen kernel: exynos-drm exynos-drm: failed to bind 
>> 14450000.mixer (ops 0xc0d97554): -22
>> devuan-bunsen kernel: exynos-drm exynos-drm: adev bind failed: -22
>> devuan-bunsen kernel: exynos-dp: probe of 145b0000.dp-controller 
>> failed with error -22
>>
>> I'm trying to find for a solution and I've googled a little bit and I 
>> found this web site : 
>> https://lore.kernel.org/linux-arm-kernel/20220208171823.226211-8-krzysztof.kozlowski@canonical.com/ 
>> with your email address and I tried to ask for some help for fixing 
>> the bug. Any ideas why booting the same Linux kernel that results in a 
>> working X.org display on the bare metal instead as dom0 on Xen would 
>> cause the display to remain dark, but most other basic functions would 
>> work, such as network, disk, and USB ? thanks.
> 
> 
> Thanks for the detailed description! Good to hear that those boards are 
> still being used for various projects. I also have Snow Chromebook and 
> use it for daily tests of linux-next branch.

Adding Julien Grall and Stefano Stabellini

Hi Marek,

Thanks for responding to Mario's question. I also have been doing these
experiments with a Chromebook Snow, and I am the one who reported this
problem on the xen-users ML here:

https://lists.xenproject.org/archives/html/xen-users/2023-10/msg00021.html

You might find that thread interesting, especially here with some additional
log messages from the exynos_drm driver (exynos_drm_dma.c, I believe):

https://lists.xenproject.org/archives/html/xen-users/2023-10/msg00032.html

This issue is also discussed some on the xen-devel ML here:

https://lists.xenproject.org/archives/html/xen-devel/2023-11/msg00003.html

> 
> Frankly speaking I have no idea what might happen wrong. There have been 
> some changes recently in the Exynos IOMMU driver related to 
> initialization, maybe your changes related to Xen enabling changed 
> somehow the order of device initialization during boot. I assume that 
> the device-tree you use for the bare metal run and Xen enabled run 
> doesn't differ in the areas describing the hardware blocks.
> 
> Please check if cherry-picking the commit 
> https://github.com/torvalds/linux/commit/bbc4d205d93f52ee18dfa7858d51489c0506547f 
> to your v6.1.59 based kernel helps anyhow.

I tried adding that fix of the exynos IOMMU initialization from
Linux > 6.2 on top of the 6.1.59 kernel I used for the original report,
but that made no difference on Xen - it still failed with the mixer lacks
support for IOMMU message and the screen is totally dark.

> 
> If not, then as a temporary workaround please disable 
> CONFIG_DRM_EXYNOS_MIXER and CONFIG_DRM_EXYNOS_HDMI in your kernel config 
> and check what will happen (You will lose the HDMI output, but maybe 
> this won't a big issue).

This change causes the GPU to work fairly well AFAICS. Removing the mixer
and HDMI allowed the GPU to initialize, and the display manager started
normally and enabled logging into an ordinary X11 session. Based on the log
messages I was seeing, this was an obvious thing to try. Thanks for
suggesting it.

But I have a question:

How are the mixer and HDMI devices related to the main drm device? The problem
in the exynos_drm_dma driver was that on Xen, the main drm device wanted to
use IOMMU version of dma_ops, but the mixer (and probably also the HDMI if
it wouldn't have exited first) wanted to use the Xen swiotlb version of dma_ops,
but on bare metal all three devices want to use the IOMMU version of dma_ops.

The problem obviously has something to do with the fact that Xen does not
expose the same IOMMU capability to Linux as is available when running on
the bare metal.

Cheers,

> 
> 
> Best regards


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks support for IOMMU

[Chuck Zmudzinski]

On 10/31/2023 8:08 AM, Marek Szyprowski wrote:
> Hi,
> 
> On 31.10.2023 00:03, Mario Marietto wrote:
>> We are a team of linux enthusiasts who are trying to boot Xen on a 
>> Samsung XE303C12 Chromebook aka "snow" following the suggestions in 
>> the slide show presentation here: 
>> https://www.slideshare.net/xen_com_mgr/xpds16-porting-xen-on-arm-to-a-new-soc-julien-grall-arm 
>> This device uses an exynos5250 SOC dual core 1.7 GHz with 2 MB RAM, it 
>> is a Samsung armv7 chip with virtualization extensions. In particular, 
>> we have it working fairly well both on the bare metal with a recent 
>> 6.1.59 Linux LTS kernel and also with a recent 5.4.257 LTS kernel with 
>> KVM, the older LTS kernel version is used to test KVM because support 
>> for KVM on arm v7 was removed from Linux around kernel version 5.7. So 
>> we know we have the hypervisor mode enabled because we were able to 
>> use it with KVM. For Xen, we are using the latest Debian build of Xen 
>> 4.17 for the Debian armhf architecture: (XEN) Xen version 4.17.2-pre 
>> (Debian 4.17.1+2-gb773c48e36-1) 
>> (pkg-xen-devel@xxxxxxxxxxxxxxxxxxxxxxx) (arm-linux-gnueabihf-gcc 
>> (Debian 12.2.0-14) 12.2.0) debug=n Thu May 18 19:26:30 UTC 2023 The 
>> Linux kernel is a custom build that adds the Xen config kernel options 
>> (CONFIG_XEN_DOM0, etc) on top of a kernel that works well on the same 
>> Chromebook model on the bare metal. I can provide the config options 
>> of the kernel that was used if that is helpful. Our method of booting 
>> is to have u-boot boot the Xen hypervisor and load the device tree 
>> after adding the dom0 to the otherwise unaltered device tree from the 
>> Linux kernel using u-boot fdt commands to add a /chosen node, as 
>> described on the Xen wiki and in the pages linked from there. We have 
>> also tried adding and loading an initrd.img using the device tree 
>> /chosen node but that made no difference in our tests. We actually 
>> have the Linux LTS kernel version 6.1.59 working as dom0 with Xen 
>> using the same version of u-boot that we used for KVM, but with a big 
>> problem. The problem we see is that when booting the 6.1.59 kernel 
>> version as dom0 with Xen, the screen is totally dark and the only way 
>> to access the system is remotely through ssh. Logs indicate most 
>> everything else is working, such as the wifi card so we can access it 
>> remotely via ssh and a USB optical mouse lights up when connected so 
>> USB is also working. Obviously, the disk is also working. The 
>> Chromebook is configured to boot from the device's SD card slot by 
>> turning on Chrome OS developer mode options to enable booting from the 
>> SD card slot. The mystery is that when booting the exact same 6.1.59 
>> kernel on the bare metal instead of booting it as dom0 on Xen, it 
>> boots up with full access to the screen and we can interact with the 
>> system using the X.org windows system. But booting as dom0 with Xen, 
>> the screen is totally dark and the only access we have to the system 
>> is through the network via ssh. Also, when booting the 5.4.257 kernel 
>> with KVM in hypervisor mode, the screen works and we can interact with 
>> the system through the X.org windows system. Exploring the log file,we 
>> have seen the errors below :
>>
>> Without Xen (or in bare metal):
>>
>> devuan-bunsen kernel: [drm] Exynos DRM: using 14400000.fimd device for 
>> DMA mapping operations
>> devuan-bunsen kernel: exynos-drm exynos-drm: bound 14400000.fimd (ops 
>> 0xc0d96354)
>> devuan-bunsen kernel: exynos-drm exynos-drm: bound 14450000.mixer (ops 
>> 0xc0d97554)
>> devuan-bunsen kernel: exynos-drm exynos-drm: bound 
>> 145b0000.dp-controller (ops 0xc0d97278)
>> devuan-bunsen kernel: exynos-drm exynos-drm: bound 14530000.hdmi (ops 
>> 0xc0d97bd0)
>> ...
>> devuan-bunsen kernel: Console: switching to colour frame buffer device 
>> 170x48
>> devuan-bunsen kernel: exynos-drm exynos-drm: [drm] fb0: exynosdrmfb 
>> frame buffer device
>> devuan-bunsen kernel: [drm] Initialized exynos 1.1.0 20180330 for 
>> exynos-drm on minor 0
>>
>> In this case,the kernel is able to use the exynos-drm kernel to start 
>> the fb0 device. But with Xen we get this error with exynos-drm:
>>
>> devuan-bunsen kernel: [drm] Exynos DRM: using 14400000.fimd device for 
>> DMA mapping operations
>> devuan-bunsen kernel: exynos-drm exynos-drm: bound 14400000.fimd (ops 
>> 0xc0d96354)
>> devuan-bunsen kernel: exynos-mixer 14450000.mixer: 
>> [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks 
>> support for IOMMU
>> devuan-bunsen kernel: exynos-drm exynos-drm: failed to bind 
>> 14450000.mixer (ops 0xc0d97554): -22
>> devuan-bunsen kernel: exynos-drm exynos-drm: adev bind failed: -22
>> devuan-bunsen kernel: exynos-dp: probe of 145b0000.dp-controller 
>> failed with error -22
>>
>> I'm trying to find for a solution and I've googled a little bit and I 
>> found this web site : 
>> https://lore.kernel.org/linux-arm-kernel/20220208171823.226211-8-krzysztof.kozlowski@canonical.com/ 
>> with your email address and I tried to ask for some help for fixing 
>> the bug. Any ideas why booting the same Linux kernel that results in a 
>> working X.org display on the bare metal instead as dom0 on Xen would 
>> cause the display to remain dark, but most other basic functions would 
>> work, such as network, disk, and USB ? thanks.
> 
> 
> Thanks for the detailed description! Good to hear that those boards are 
> still being used for various projects. I also have Snow Chromebook and 
> use it for daily tests of linux-next branch.

Adding Julien Grall and Stefano Stabellini

Hi Marek,

Thanks for responding to Mario's question. I also have been doing these
experiments with a Chromebook Snow, and I am the one who reported this
problem on the xen-users ML here:

https://lists.xenproject.org/archives/html/xen-users/2023-10/msg00021.html

You might find that thread interesting, especially here with some additional
log messages from the exynos_drm driver (exynos_drm_dma.c, I believe):

https://lists.xenproject.org/archives/html/xen-users/2023-10/msg00032.html

This issue is also discussed some on the xen-devel ML here:

https://lists.xenproject.org/archives/html/xen-devel/2023-11/msg00003.html

> 
> Frankly speaking I have no idea what might happen wrong. There have been 
> some changes recently in the Exynos IOMMU driver related to 
> initialization, maybe your changes related to Xen enabling changed 
> somehow the order of device initialization during boot. I assume that 
> the device-tree you use for the bare metal run and Xen enabled run 
> doesn't differ in the areas describing the hardware blocks.
> 
> Please check if cherry-picking the commit 
> https://github.com/torvalds/linux/commit/bbc4d205d93f52ee18dfa7858d51489c0506547f 
> to your v6.1.59 based kernel helps anyhow.

I tried adding that fix of the exynos IOMMU initialization from
Linux > 6.2 on top of the 6.1.59 kernel I used for the original report,
but that made no difference on Xen - it still failed with the mixer lacks
support for IOMMU message and the screen is totally dark.

> 
> If not, then as a temporary workaround please disable 
> CONFIG_DRM_EXYNOS_MIXER and CONFIG_DRM_EXYNOS_HDMI in your kernel config 
> and check what will happen (You will lose the HDMI output, but maybe 
> this won't a big issue).

This change causes the GPU to work fairly well AFAICS. Removing the mixer
and HDMI allowed the GPU to initialize, and the display manager started
normally and enabled logging into an ordinary X11 session. Based on the log
messages I was seeing, this was an obvious thing to try. Thanks for
suggesting it.

But I have a question:

How are the mixer and HDMI devices related to the main drm device? The problem
in the exynos_drm_dma driver was that on Xen, the main drm device wanted to
use IOMMU version of dma_ops, but the mixer (and probably also the HDMI if
it wouldn't have exited first) wanted to use the Xen swiotlb version of dma_ops,
but on bare metal all three devices want to use the IOMMU version of dma_ops.

The problem obviously has something to do with the fact that Xen does not
expose the same IOMMU capability to Linux as is available when running on
the bare metal.

Cheers,

> 
> 
> Best regards

Re: exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks support for IOMMU

[Julien Grall]

Hi,

@Stefano, as you pointed out, there is already a thread on xen-users for 
this discussion. Could we use this thread for any discussion? This would 
make easier to follow.

Some high level comment below.

On 01/11/2023 02:50, Chuck Zmudzinski wrote:
> On 10/31/2023 7:45 PM, Stefano Stabellini wrote:
>> Unfortunately there is no easy solution.
>>
>> Do you know the version of the SMMU available on the platform?
> 
> I am trying to discern, but I doubt we have v3 because we are
> working on a very old chromebook from 2012, and I am finding
> patches for smmv3 in Linux not starting until 2015. It is good to
> know about this option, though, for future work we might do on newer
> devices.

The chromebook is using the Exynos Sysmmu. So there is no SMMU.

> 
>> If it is a SMMUv3 you can try to use the nested SMMU patch series to
>> enable a virtual SMMU in Dom0: https://marc.info/?l=xen-devel&m=166991020831005
>> That way, Xen can use the SMMU to protect VMs, and Dom0 can also use the
>> SMMU for its own purposes at the same time.
>>
>> Alternatively, you can dig into the details of the exynos-drm driver to
>> see what exactly is the dependency on the IOMMU framework in Linux and
>> remove the dependency. Unfortunately none of us in this thread are
>> expert on exynos-drm so it would be difficult to advise on how to do
>> that. For example, I don't know how you could debug the x11 problem you
>> described because I don't typically work with x11 or with the exynos. If
>> there is an open source mailing list for exynos-drm development they
>> might be able to advise on how to remove the IOMMU dependency there.
> 
> We have received this message from Marek Szyprowski of Samsung:
> 
> https://lore.kernel.org/lkml/7a71e348-f892-4fd4-8857-b72f35ab5134@samsung.com/
> 
> Marek recommends this patch to possibly help with this issue:
> 
> https://github.com/torvalds/linux/commit/bbc4d205d93f52ee18dfa7858d51489c0506547f
> 
> and also these kernel config settings:
> 
> On 10/31/2023 8:08 AM, Marek Szyprowski wrote:
>> If not, then as a temporary workaround please disable
>> CONFIG_DRM_EXYNOS_MIXER and CONFIG_DRM_EXYNOS_HDMI in your kernel config
>> and check what will happen (You will lose the HDMI output, but maybe
>> this won't a big issue).
> 
> Mario and I have preliminary evidence that applying both of Marek's
> recommendations to the 6.1.59 kernel have improved the situation to
> the point where now the Chromebook can run X.org on Xen. We are doing
> further tests to see how Marek's patch and/or the kernel config settings
> to disable the mixer and the HDMI affect the behavior of the GPU in dom0
> on Xen.
> 
>>
>> The final option, which is a gross hack, would be to let Dom0 use the
>> IOMMU for its own gain. Xen doesn't use the IOMMU. If you do that you
>> lose freedom from interference between the VMs and you cannot run driver
>> domains or directly assign devices to DomUs. But if you are running a
>> research project you might be OK with that. To get it to work, you need
>> to hack Xen so that it remaps the IOMMU to Dom0 to let Dom0 program it
>> directly. The attached patch (untested) would be a place to start. You
>> also need to pass iommu=false to the Xen command line to prevent Xen
>> from using the iommu itself.

This is actually one of the reason why I am suggesting to do all the 
investigation in one thread. There, we already discovered that the IOMMU 
was assigned to dom0 because Xen doesn't have a driver and we don't hide 
them by default.

> 
> I am interested to investigate if only the mixer and the HDMI is causing
> the trouble. Based on what you are telling me about Xen not exposing the
> IOMMU to dom0, I don't fully understand the original log messages I was
> getting when I followed Julien's suggestion to find the symbols associated
> with each address in the original message, and those seemed to indicate that
> the exynos_drm device was using the IOMMU in dom0, but the mixer was not,
> and the fact that they both were not using the IOMMU is what caused the
> test to fail and Linux refuse to initialize the GPU on dom0, whereas on
> bare metal, the logs indicated both the exynos mixer, which I think is a
> sub device of the exynos_drm, and the exynos_drm, use the IOMMU on bare
> metal.
> 
> I also found this patch which suggests if we can get the devices to work
> we will be compromising the security and isolation between guests:

If you don't assign any device to the guests, then you will not break 
any isolation between guests because dom0 will own all of them.

But for passthrough, you would want to the IOMMU owned by Xen rather 
than dom0. Unless the Exynos sysmmu support 2-stages page-tables, then 
dom0 will not be able to use the IOMMU.

Cheers,

-- 
Julien Grall

Re: exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks support for IOMMU

[Chuck Zmudzinski]

On 11/1/2023 4:27 AM, Julien Grall wrote:
> Hi,
> 
> @Stefano, as you pointed out, there is already a thread on xen-users for 
> this discussion. Could we use this thread for any discussion? This would 
> make easier to follow.
> 
> Some high level comment below.

I agree to keep the discussion here and not at other places.

I just want to add that the best results for Xen dom0 so far are
by implementing Marek's suggestion to disable these two settings
in the 6.1.59 kernel config, but leaving everything else the same,
including keeping the EXYNOS_IOMMU support enabled:

# CONFIG_DRM_EXYNOS_MIXER is not set

Disabling the mixer also makes this unavailable:

# CONFIG_HDMI

With this change, the GPU is working well enough to allow the display
manager and an X11 session to run normally on the built-in display of the
Chromebook. The Wifi also works well.

The patch from Linux 6.2 and above that fixes the exynos IOMMU initialization
did not help at all, and the same error is reported that the mixer lacks
support for IOMMU.


> 
> On 01/11/2023 02:50, Chuck Zmudzinski wrote:
>> On 10/31/2023 7:45 PM, Stefano Stabellini wrote:
>>> Unfortunately there is no easy solution.
>>>
>>> Do you know the version of the SMMU available on the platform?
>> 
>> I am trying to discern, but I doubt we have v3 because we are
>> working on a very old chromebook from 2012, and I am finding
>> patches for smmv3 in Linux not starting until 2015. It is good to
>> know about this option, though, for future work we might do on newer
>> devices.
> 
> The chromebook is using the Exynos Sysmmu. So there is no SMMU.
> 
>> 
>>> If it is a SMMUv3 you can try to use the nested SMMU patch series to
>>> enable a virtual SMMU in Dom0: https://marc.info/?l=xen-devel&m=166991020831005
>>> That way, Xen can use the SMMU to protect VMs, and Dom0 can also use the
>>> SMMU for its own purposes at the same time.
>>>
>>> Alternatively, you can dig into the details of the exynos-drm driver to
>>> see what exactly is the dependency on the IOMMU framework in Linux and
>>> remove the dependency. Unfortunately none of us in this thread are
>>> expert on exynos-drm so it would be difficult to advise on how to do
>>> that. For example, I don't know how you could debug the x11 problem you
>>> described because I don't typically work with x11 or with the exynos. If
>>> there is an open source mailing list for exynos-drm development they
>>> might be able to advise on how to remove the IOMMU dependency there.
>> 
>> We have received this message from Marek Szyprowski of Samsung:
>> 
>> https://lore.kernel.org/lkml/7a71e348-f892-4fd4-8857-b72f35ab5134@samsung.com/
>> 
>> Marek recommends this patch to possibly help with this issue:
>> 
>> https://github.com/torvalds/linux/commit/bbc4d205d93f52ee18dfa7858d51489c0506547f
>> 
>> and also these kernel config settings:
>> 
>> On 10/31/2023 8:08 AM, Marek Szyprowski wrote:
>>> If not, then as a temporary workaround please disable
>>> CONFIG_DRM_EXYNOS_MIXER and CONFIG_DRM_EXYNOS_HDMI in your kernel config
>>> and check what will happen (You will lose the HDMI output, but maybe
>>> this won't a big issue).
>> 
>> Mario and I have preliminary evidence that applying both of Marek's
>> recommendations to the 6.1.59 kernel have improved the situation to
>> the point where now the Chromebook can run X.org on Xen. We are doing
>> further tests to see how Marek's patch and/or the kernel config settings
>> to disable the mixer and the HDMI affect the behavior of the GPU in dom0
>> on Xen.
>> 
>>>
>>> The final option, which is a gross hack, would be to let Dom0 use the
>>> IOMMU for its own gain. Xen doesn't use the IOMMU. If you do that you
>>> lose freedom from interference between the VMs and you cannot run driver
>>> domains or directly assign devices to DomUs. But if you are running a
>>> research project you might be OK with that. To get it to work, you need
>>> to hack Xen so that it remaps the IOMMU to Dom0 to let Dom0 program it
>>> directly. The attached patch (untested) would be a place to start. You
>>> also need to pass iommu=false to the Xen command line to prevent Xen
>>> from using the iommu itself.
> 
> This is actually one of the reason why I am suggesting to do all the 
> investigation in one thread. There, we already discovered that the IOMMU 
> was assigned to dom0 because Xen doesn't have a driver and we don't hide 
> them by default.
> 
>> 
>> I am interested to investigate if only the mixer and the HDMI is causing
>> the trouble. Based on what you are telling me about Xen not exposing the
>> IOMMU to dom0, I don't fully understand the original log messages I was
>> getting when I followed Julien's suggestion to find the symbols associated
>> with each address in the original message, and those seemed to indicate that
>> the exynos_drm device was using the IOMMU in dom0, but the mixer was not,
>> and the fact that they both were not using the IOMMU is what caused the
>> test to fail and Linux refuse to initialize the GPU on dom0, whereas on
>> bare metal, the logs indicated both the exynos mixer, which I think is a
>> sub device of the exynos_drm, and the exynos_drm, use the IOMMU on bare
>> metal.
>> 
>> I also found this patch which suggests if we can get the devices to work
>> we will be compromising the security and isolation between guests:
> 
> If you don't assign any device to the guests, then you will not break 
> any isolation between guests because dom0 will own all of them.
> 
> But for passthrough, you would want to the IOMMU owned by Xen rather 
> than dom0. Unless the Exynos sysmmu support 2-stages page-tables, then 
> dom0 will not be able to use the IOMMU.
> 
> Cheers,
> 

Re: exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks support for IOMMU

[Julien Grall]

On 01/11/2023 08:45, Chuck Zmudzinski wrote:
> On 11/1/2023 4:27 AM, Julien Grall wrote:
>> Hi,
>>
>> @Stefano, as you pointed out, there is already a thread on xen-users for
>> this discussion. Could we use this thread for any discussion? This would
>> make easier to follow.
>>
>> Some high level comment below.
> 
> I agree to keep the discussion here and not at other places.

I was meant to suggest the other thread :). But either is fine with me. 
I just want to avoid avoid multiple seperate threads for the discussion.

> 
> I just want to add that the best results for Xen dom0 so far are
> by implementing Marek's suggestion to disable these two settings
> in the 6.1.59 kernel config, but leaving everything else the same,
> including keeping the EXYNOS_IOMMU support enabled:
That's good news! I would be interested to hear how this works once you 
start to have PV backend in dom0 (I expect that the IOMMU will get 
confused with grant mapping).

Also, do you plan to passthrough any of the devices protected by IOMMU?

> 
> # CONFIG_DRM_EXYNOS_MIXER is not set
> 
> Disabling the mixer also makes this unavailable:
> 
> # CONFIG_HDMI
> 
> With this change, the GPU is working well enough to allow the display
> manager and an X11 session to run normally on the built-in display of the
> Chromebook. The Wifi also works well.

I saw your other answer about the Wifi not working when the IOMMU is not 
used. I was about to reply there, but instead I will do it here.

TBH, I am quite surprised this is the case. The only difference with 
baremetal would be the RAM regions. Do you know if the Wifi dongle only 
accept certain physical address?

Cheers,

-- 
Julien Grall

Re: exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks support for IOMMU

[Mario Marietto]

Hi Marek,

I would like to recompile and install the kernel 6.6 on my ARM
Chromebook. I would like to know if your patch has been accepted and
included there. Thanks.

On Wed, Nov 1, 2023 at 8:48 AM Chuck Zmudzinski <brchuckz@netscape.net> wrote:
>
> On 10/31/2023 8:08 AM, Marek Szyprowski wrote:
> > Hi,
> >
> > On 31.10.2023 00:03, Mario Marietto wrote:
> >> We are a team of linux enthusiasts who are trying to boot Xen on a
> >> Samsung XE303C12 Chromebook aka "snow" following the suggestions in
> >> the slide show presentation here:
> >> https://www.slideshare.net/xen_com_mgr/xpds16-porting-xen-on-arm-to-a-new-soc-julien-grall-arm
> >> This device uses an exynos5250 SOC dual core 1.7 GHz with 2 MB RAM, it
> >> is a Samsung armv7 chip with virtualization extensions. In particular,
> >> we have it working fairly well both on the bare metal with a recent
> >> 6.1.59 Linux LTS kernel and also with a recent 5.4.257 LTS kernel with
> >> KVM, the older LTS kernel version is used to test KVM because support
> >> for KVM on arm v7 was removed from Linux around kernel version 5.7. So
> >> we know we have the hypervisor mode enabled because we were able to
> >> use it with KVM. For Xen, we are using the latest Debian build of Xen
> >> 4.17 for the Debian armhf architecture: (XEN) Xen version 4.17.2-pre
> >> (Debian 4.17.1+2-gb773c48e36-1)
> >> (pkg-xen-devel@xxxxxxxxxxxxxxxxxxxxxxx) (arm-linux-gnueabihf-gcc
> >> (Debian 12.2.0-14) 12.2.0) debug=n Thu May 18 19:26:30 UTC 2023 The
> >> Linux kernel is a custom build that adds the Xen config kernel options
> >> (CONFIG_XEN_DOM0, etc) on top of a kernel that works well on the same
> >> Chromebook model on the bare metal. I can provide the config options
> >> of the kernel that was used if that is helpful. Our method of booting
> >> is to have u-boot boot the Xen hypervisor and load the device tree
> >> after adding the dom0 to the otherwise unaltered device tree from the
> >> Linux kernel using u-boot fdt commands to add a /chosen node, as
> >> described on the Xen wiki and in the pages linked from there. We have
> >> also tried adding and loading an initrd.img using the device tree
> >> /chosen node but that made no difference in our tests. We actually
> >> have the Linux LTS kernel version 6.1.59 working as dom0 with Xen
> >> using the same version of u-boot that we used for KVM, but with a big
> >> problem. The problem we see is that when booting the 6.1.59 kernel
> >> version as dom0 with Xen, the screen is totally dark and the only way
> >> to access the system is remotely through ssh. Logs indicate most
> >> everything else is working, such as the wifi card so we can access it
> >> remotely via ssh and a USB optical mouse lights up when connected so
> >> USB is also working. Obviously, the disk is also working. The
> >> Chromebook is configured to boot from the device's SD card slot by
> >> turning on Chrome OS developer mode options to enable booting from the
> >> SD card slot. The mystery is that when booting the exact same 6.1.59
> >> kernel on the bare metal instead of booting it as dom0 on Xen, it
> >> boots up with full access to the screen and we can interact with the
> >> system using the X.org windows system. But booting as dom0 with Xen,
> >> the screen is totally dark and the only access we have to the system
> >> is through the network via ssh. Also, when booting the 5.4.257 kernel
> >> with KVM in hypervisor mode, the screen works and we can interact with
> >> the system through the X.org windows system. Exploring the log file,we
> >> have seen the errors below :
> >>
> >> Without Xen (or in bare metal):
> >>
> >> devuan-bunsen kernel: [drm] Exynos DRM: using 14400000.fimd device for
> >> DMA mapping operations
> >> devuan-bunsen kernel: exynos-drm exynos-drm: bound 14400000.fimd (ops
> >> 0xc0d96354)
> >> devuan-bunsen kernel: exynos-drm exynos-drm: bound 14450000.mixer (ops
> >> 0xc0d97554)
> >> devuan-bunsen kernel: exynos-drm exynos-drm: bound
> >> 145b0000.dp-controller (ops 0xc0d97278)
> >> devuan-bunsen kernel: exynos-drm exynos-drm: bound 14530000.hdmi (ops
> >> 0xc0d97bd0)
> >> ...
> >> devuan-bunsen kernel: Console: switching to colour frame buffer device
> >> 170x48
> >> devuan-bunsen kernel: exynos-drm exynos-drm: [drm] fb0: exynosdrmfb
> >> frame buffer device
> >> devuan-bunsen kernel: [drm] Initialized exynos 1.1.0 20180330 for
> >> exynos-drm on minor 0
> >>
> >> In this case,the kernel is able to use the exynos-drm kernel to start
> >> the fb0 device. But with Xen we get this error with exynos-drm:
> >>
> >> devuan-bunsen kernel: [drm] Exynos DRM: using 14400000.fimd device for
> >> DMA mapping operations
> >> devuan-bunsen kernel: exynos-drm exynos-drm: bound 14400000.fimd (ops
> >> 0xc0d96354)
> >> devuan-bunsen kernel: exynos-mixer 14450000.mixer:
> >> [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks
> >> support for IOMMU
> >> devuan-bunsen kernel: exynos-drm exynos-drm: failed to bind
> >> 14450000.mixer (ops 0xc0d97554): -22
> >> devuan-bunsen kernel: exynos-drm exynos-drm: adev bind failed: -22
> >> devuan-bunsen kernel: exynos-dp: probe of 145b0000.dp-controller
> >> failed with error -22
> >>
> >> I'm trying to find for a solution and I've googled a little bit and I
> >> found this web site :
> >> https://lore.kernel.org/linux-arm-kernel/20220208171823.226211-8-krzysztof.kozlowski@canonical.com/
> >> with your email address and I tried to ask for some help for fixing
> >> the bug. Any ideas why booting the same Linux kernel that results in a
> >> working X.org display on the bare metal instead as dom0 on Xen would
> >> cause the display to remain dark, but most other basic functions would
> >> work, such as network, disk, and USB ? thanks.
> >
> >
> > Thanks for the detailed description! Good to hear that those boards are
> > still being used for various projects. I also have Snow Chromebook and
> > use it for daily tests of linux-next branch.
>
> Adding Julien Grall and Stefano Stabellini
>
> Hi Marek,
>
> Thanks for responding to Mario's question. I also have been doing these
> experiments with a Chromebook Snow, and I am the one who reported this
> problem on the xen-users ML here:
>
> https://lists.xenproject.org/archives/html/xen-users/2023-10/msg00021.html
>
> You might find that thread interesting, especially here with some additional
> log messages from the exynos_drm driver (exynos_drm_dma.c, I believe):
>
> https://lists.xenproject.org/archives/html/xen-users/2023-10/msg00032.html
>
> This issue is also discussed some on the xen-devel ML here:
>
> https://lists.xenproject.org/archives/html/xen-devel/2023-11/msg00003.html
>
> >
> > Frankly speaking I have no idea what might happen wrong. There have been
> > some changes recently in the Exynos IOMMU driver related to
> > initialization, maybe your changes related to Xen enabling changed
> > somehow the order of device initialization during boot. I assume that
> > the device-tree you use for the bare metal run and Xen enabled run
> > doesn't differ in the areas describing the hardware blocks.
> >
> > Please check if cherry-picking the commit
> > https://github.com/torvalds/linux/commit/bbc4d205d93f52ee18dfa7858d51489c0506547f
> > to your v6.1.59 based kernel helps anyhow.
>
> I tried adding that fix of the exynos IOMMU initialization from
> Linux > 6.2 on top of the 6.1.59 kernel I used for the original report,
> but that made no difference on Xen - it still failed with the mixer lacks
> support for IOMMU message and the screen is totally dark.
>
> >
> > If not, then as a temporary workaround please disable
> > CONFIG_DRM_EXYNOS_MIXER and CONFIG_DRM_EXYNOS_HDMI in your kernel config
> > and check what will happen (You will lose the HDMI output, but maybe
> > this won't a big issue).
>
> This change causes the GPU to work fairly well AFAICS. Removing the mixer
> and HDMI allowed the GPU to initialize, and the display manager started
> normally and enabled logging into an ordinary X11 session. Based on the log
> messages I was seeing, this was an obvious thing to try. Thanks for
> suggesting it.
>
> But I have a question:
>
> How are the mixer and HDMI devices related to the main drm device? The problem
> in the exynos_drm_dma driver was that on Xen, the main drm device wanted to
> use IOMMU version of dma_ops, but the mixer (and probably also the HDMI if
> it wouldn't have exited first) wanted to use the Xen swiotlb version of dma_ops,
> but on bare metal all three devices want to use the IOMMU version of dma_ops.
>
> The problem obviously has something to do with the fact that Xen does not
> expose the same IOMMU capability to Linux as is available when running on
> the bare metal.
>
> Cheers,
>
> >
> >
> > Best regards
>


-- 
Mario.

Re: exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks support for IOMMU

[Mario Marietto]

Hi Marek,

I would like to recompile and install the kernel 6.6 on my ARM
Chromebook. I would like to know if your patch has been accepted and
included there. Thanks.

On Wed, Nov 1, 2023 at 8:48 AM Chuck Zmudzinski <brchuckz@netscape.net> wrote:
>
> On 10/31/2023 8:08 AM, Marek Szyprowski wrote:
> > Hi,
> >
> > On 31.10.2023 00:03, Mario Marietto wrote:
> >> We are a team of linux enthusiasts who are trying to boot Xen on a
> >> Samsung XE303C12 Chromebook aka "snow" following the suggestions in
> >> the slide show presentation here:
> >> https://www.slideshare.net/xen_com_mgr/xpds16-porting-xen-on-arm-to-a-new-soc-julien-grall-arm
> >> This device uses an exynos5250 SOC dual core 1.7 GHz with 2 MB RAM, it
> >> is a Samsung armv7 chip with virtualization extensions. In particular,
> >> we have it working fairly well both on the bare metal with a recent
> >> 6.1.59 Linux LTS kernel and also with a recent 5.4.257 LTS kernel with
> >> KVM, the older LTS kernel version is used to test KVM because support
> >> for KVM on arm v7 was removed from Linux around kernel version 5.7. So
> >> we know we have the hypervisor mode enabled because we were able to
> >> use it with KVM. For Xen, we are using the latest Debian build of Xen
> >> 4.17 for the Debian armhf architecture: (XEN) Xen version 4.17.2-pre
> >> (Debian 4.17.1+2-gb773c48e36-1)
> >> (pkg-xen-devel@xxxxxxxxxxxxxxxxxxxxxxx) (arm-linux-gnueabihf-gcc
> >> (Debian 12.2.0-14) 12.2.0) debug=n Thu May 18 19:26:30 UTC 2023 The
> >> Linux kernel is a custom build that adds the Xen config kernel options
> >> (CONFIG_XEN_DOM0, etc) on top of a kernel that works well on the same
> >> Chromebook model on the bare metal. I can provide the config options
> >> of the kernel that was used if that is helpful. Our method of booting
> >> is to have u-boot boot the Xen hypervisor and load the device tree
> >> after adding the dom0 to the otherwise unaltered device tree from the
> >> Linux kernel using u-boot fdt commands to add a /chosen node, as
> >> described on the Xen wiki and in the pages linked from there. We have
> >> also tried adding and loading an initrd.img using the device tree
> >> /chosen node but that made no difference in our tests. We actually
> >> have the Linux LTS kernel version 6.1.59 working as dom0 with Xen
> >> using the same version of u-boot that we used for KVM, but with a big
> >> problem. The problem we see is that when booting the 6.1.59 kernel
> >> version as dom0 with Xen, the screen is totally dark and the only way
> >> to access the system is remotely through ssh. Logs indicate most
> >> everything else is working, such as the wifi card so we can access it
> >> remotely via ssh and a USB optical mouse lights up when connected so
> >> USB is also working. Obviously, the disk is also working. The
> >> Chromebook is configured to boot from the device's SD card slot by
> >> turning on Chrome OS developer mode options to enable booting from the
> >> SD card slot. The mystery is that when booting the exact same 6.1.59
> >> kernel on the bare metal instead of booting it as dom0 on Xen, it
> >> boots up with full access to the screen and we can interact with the
> >> system using the X.org windows system. But booting as dom0 with Xen,
> >> the screen is totally dark and the only access we have to the system
> >> is through the network via ssh. Also, when booting the 5.4.257 kernel
> >> with KVM in hypervisor mode, the screen works and we can interact with
> >> the system through the X.org windows system. Exploring the log file,we
> >> have seen the errors below :
> >>
> >> Without Xen (or in bare metal):
> >>
> >> devuan-bunsen kernel: [drm] Exynos DRM: using 14400000.fimd device for
> >> DMA mapping operations
> >> devuan-bunsen kernel: exynos-drm exynos-drm: bound 14400000.fimd (ops
> >> 0xc0d96354)
> >> devuan-bunsen kernel: exynos-drm exynos-drm: bound 14450000.mixer (ops
> >> 0xc0d97554)
> >> devuan-bunsen kernel: exynos-drm exynos-drm: bound
> >> 145b0000.dp-controller (ops 0xc0d97278)
> >> devuan-bunsen kernel: exynos-drm exynos-drm: bound 14530000.hdmi (ops
> >> 0xc0d97bd0)
> >> ...
> >> devuan-bunsen kernel: Console: switching to colour frame buffer device
> >> 170x48
> >> devuan-bunsen kernel: exynos-drm exynos-drm: [drm] fb0: exynosdrmfb
> >> frame buffer device
> >> devuan-bunsen kernel: [drm] Initialized exynos 1.1.0 20180330 for
> >> exynos-drm on minor 0
> >>
> >> In this case,the kernel is able to use the exynos-drm kernel to start
> >> the fb0 device. But with Xen we get this error with exynos-drm:
> >>
> >> devuan-bunsen kernel: [drm] Exynos DRM: using 14400000.fimd device for
> >> DMA mapping operations
> >> devuan-bunsen kernel: exynos-drm exynos-drm: bound 14400000.fimd (ops
> >> 0xc0d96354)
> >> devuan-bunsen kernel: exynos-mixer 14450000.mixer:
> >> [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks
> >> support for IOMMU
> >> devuan-bunsen kernel: exynos-drm exynos-drm: failed to bind
> >> 14450000.mixer (ops 0xc0d97554): -22
> >> devuan-bunsen kernel: exynos-drm exynos-drm: adev bind failed: -22
> >> devuan-bunsen kernel: exynos-dp: probe of 145b0000.dp-controller
> >> failed with error -22
> >>
> >> I'm trying to find for a solution and I've googled a little bit and I
> >> found this web site :
> >> https://lore.kernel.org/linux-arm-kernel/20220208171823.226211-8-krzysztof.kozlowski@canonical.com/
> >> with your email address and I tried to ask for some help for fixing
> >> the bug. Any ideas why booting the same Linux kernel that results in a
> >> working X.org display on the bare metal instead as dom0 on Xen would
> >> cause the display to remain dark, but most other basic functions would
> >> work, such as network, disk, and USB ? thanks.
> >
> >
> > Thanks for the detailed description! Good to hear that those boards are
> > still being used for various projects. I also have Snow Chromebook and
> > use it for daily tests of linux-next branch.
>
> Adding Julien Grall and Stefano Stabellini
>
> Hi Marek,
>
> Thanks for responding to Mario's question. I also have been doing these
> experiments with a Chromebook Snow, and I am the one who reported this
> problem on the xen-users ML here:
>
> https://lists.xenproject.org/archives/html/xen-users/2023-10/msg00021.html
>
> You might find that thread interesting, especially here with some additional
> log messages from the exynos_drm driver (exynos_drm_dma.c, I believe):
>
> https://lists.xenproject.org/archives/html/xen-users/2023-10/msg00032.html
>
> This issue is also discussed some on the xen-devel ML here:
>
> https://lists.xenproject.org/archives/html/xen-devel/2023-11/msg00003.html
>
> >
> > Frankly speaking I have no idea what might happen wrong. There have been
> > some changes recently in the Exynos IOMMU driver related to
> > initialization, maybe your changes related to Xen enabling changed
> > somehow the order of device initialization during boot. I assume that
> > the device-tree you use for the bare metal run and Xen enabled run
> > doesn't differ in the areas describing the hardware blocks.
> >
> > Please check if cherry-picking the commit
> > https://github.com/torvalds/linux/commit/bbc4d205d93f52ee18dfa7858d51489c0506547f
> > to your v6.1.59 based kernel helps anyhow.
>
> I tried adding that fix of the exynos IOMMU initialization from
> Linux > 6.2 on top of the 6.1.59 kernel I used for the original report,
> but that made no difference on Xen - it still failed with the mixer lacks
> support for IOMMU message and the screen is totally dark.
>
> >
> > If not, then as a temporary workaround please disable
> > CONFIG_DRM_EXYNOS_MIXER and CONFIG_DRM_EXYNOS_HDMI in your kernel config
> > and check what will happen (You will lose the HDMI output, but maybe
> > this won't a big issue).
>
> This change causes the GPU to work fairly well AFAICS. Removing the mixer
> and HDMI allowed the GPU to initialize, and the display manager started
> normally and enabled logging into an ordinary X11 session. Based on the log
> messages I was seeing, this was an obvious thing to try. Thanks for
> suggesting it.
>
> But I have a question:
>
> How are the mixer and HDMI devices related to the main drm device? The problem
> in the exynos_drm_dma driver was that on Xen, the main drm device wanted to
> use IOMMU version of dma_ops, but the mixer (and probably also the HDMI if
> it wouldn't have exited first) wanted to use the Xen swiotlb version of dma_ops,
> but on bare metal all three devices want to use the IOMMU version of dma_ops.
>
> The problem obviously has something to do with the fact that Xen does not
> expose the same IOMMU capability to Linux as is available when running on
> the bare metal.
>
> Cheers,
>
> >
> >
> > Best regards
>


-- 
Mario.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks support for IOMMU

[Mario Marietto]

Hi Marek,

I would like to recompile and install the kernel 6.6 on my ARM
Chromebook. I would like to know if your patch has been accepted and
included there. Thanks.

On Wed, Nov 1, 2023 at 8:48 AM Chuck Zmudzinski <brchuckz@netscape.net> wrote:
>
> On 10/31/2023 8:08 AM, Marek Szyprowski wrote:
> > Hi,
> >
> > On 31.10.2023 00:03, Mario Marietto wrote:
> >> We are a team of linux enthusiasts who are trying to boot Xen on a
> >> Samsung XE303C12 Chromebook aka "snow" following the suggestions in
> >> the slide show presentation here:
> >> https://www.slideshare.net/xen_com_mgr/xpds16-porting-xen-on-arm-to-a-new-soc-julien-grall-arm
> >> This device uses an exynos5250 SOC dual core 1.7 GHz with 2 MB RAM, it
> >> is a Samsung armv7 chip with virtualization extensions. In particular,
> >> we have it working fairly well both on the bare metal with a recent
> >> 6.1.59 Linux LTS kernel and also with a recent 5.4.257 LTS kernel with
> >> KVM, the older LTS kernel version is used to test KVM because support
> >> for KVM on arm v7 was removed from Linux around kernel version 5.7. So
> >> we know we have the hypervisor mode enabled because we were able to
> >> use it with KVM. For Xen, we are using the latest Debian build of Xen
> >> 4.17 for the Debian armhf architecture: (XEN) Xen version 4.17.2-pre
> >> (Debian 4.17.1+2-gb773c48e36-1)
> >> (pkg-xen-devel@xxxxxxxxxxxxxxxxxxxxxxx) (arm-linux-gnueabihf-gcc
> >> (Debian 12.2.0-14) 12.2.0) debug=n Thu May 18 19:26:30 UTC 2023 The
> >> Linux kernel is a custom build that adds the Xen config kernel options
> >> (CONFIG_XEN_DOM0, etc) on top of a kernel that works well on the same
> >> Chromebook model on the bare metal. I can provide the config options
> >> of the kernel that was used if that is helpful. Our method of booting
> >> is to have u-boot boot the Xen hypervisor and load the device tree
> >> after adding the dom0 to the otherwise unaltered device tree from the
> >> Linux kernel using u-boot fdt commands to add a /chosen node, as
> >> described on the Xen wiki and in the pages linked from there. We have
> >> also tried adding and loading an initrd.img using the device tree
> >> /chosen node but that made no difference in our tests. We actually
> >> have the Linux LTS kernel version 6.1.59 working as dom0 with Xen
> >> using the same version of u-boot that we used for KVM, but with a big
> >> problem. The problem we see is that when booting the 6.1.59 kernel
> >> version as dom0 with Xen, the screen is totally dark and the only way
> >> to access the system is remotely through ssh. Logs indicate most
> >> everything else is working, such as the wifi card so we can access it
> >> remotely via ssh and a USB optical mouse lights up when connected so
> >> USB is also working. Obviously, the disk is also working. The
> >> Chromebook is configured to boot from the device's SD card slot by
> >> turning on Chrome OS developer mode options to enable booting from the
> >> SD card slot. The mystery is that when booting the exact same 6.1.59
> >> kernel on the bare metal instead of booting it as dom0 on Xen, it
> >> boots up with full access to the screen and we can interact with the
> >> system using the X.org windows system. But booting as dom0 with Xen,
> >> the screen is totally dark and the only access we have to the system
> >> is through the network via ssh. Also, when booting the 5.4.257 kernel
> >> with KVM in hypervisor mode, the screen works and we can interact with
> >> the system through the X.org windows system. Exploring the log file,we
> >> have seen the errors below :
> >>
> >> Without Xen (or in bare metal):
> >>
> >> devuan-bunsen kernel: [drm] Exynos DRM: using 14400000.fimd device for
> >> DMA mapping operations
> >> devuan-bunsen kernel: exynos-drm exynos-drm: bound 14400000.fimd (ops
> >> 0xc0d96354)
> >> devuan-bunsen kernel: exynos-drm exynos-drm: bound 14450000.mixer (ops
> >> 0xc0d97554)
> >> devuan-bunsen kernel: exynos-drm exynos-drm: bound
> >> 145b0000.dp-controller (ops 0xc0d97278)
> >> devuan-bunsen kernel: exynos-drm exynos-drm: bound 14530000.hdmi (ops
> >> 0xc0d97bd0)
> >> ...
> >> devuan-bunsen kernel: Console: switching to colour frame buffer device
> >> 170x48
> >> devuan-bunsen kernel: exynos-drm exynos-drm: [drm] fb0: exynosdrmfb
> >> frame buffer device
> >> devuan-bunsen kernel: [drm] Initialized exynos 1.1.0 20180330 for
> >> exynos-drm on minor 0
> >>
> >> In this case,the kernel is able to use the exynos-drm kernel to start
> >> the fb0 device. But with Xen we get this error with exynos-drm:
> >>
> >> devuan-bunsen kernel: [drm] Exynos DRM: using 14400000.fimd device for
> >> DMA mapping operations
> >> devuan-bunsen kernel: exynos-drm exynos-drm: bound 14400000.fimd (ops
> >> 0xc0d96354)
> >> devuan-bunsen kernel: exynos-mixer 14450000.mixer:
> >> [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks
> >> support for IOMMU
> >> devuan-bunsen kernel: exynos-drm exynos-drm: failed to bind
> >> 14450000.mixer (ops 0xc0d97554): -22
> >> devuan-bunsen kernel: exynos-drm exynos-drm: adev bind failed: -22
> >> devuan-bunsen kernel: exynos-dp: probe of 145b0000.dp-controller
> >> failed with error -22
> >>
> >> I'm trying to find for a solution and I've googled a little bit and I
> >> found this web site :
> >> https://lore.kernel.org/linux-arm-kernel/20220208171823.226211-8-krzysztof.kozlowski@canonical.com/
> >> with your email address and I tried to ask for some help for fixing
> >> the bug. Any ideas why booting the same Linux kernel that results in a
> >> working X.org display on the bare metal instead as dom0 on Xen would
> >> cause the display to remain dark, but most other basic functions would
> >> work, such as network, disk, and USB ? thanks.
> >
> >
> > Thanks for the detailed description! Good to hear that those boards are
> > still being used for various projects. I also have Snow Chromebook and
> > use it for daily tests of linux-next branch.
>
> Adding Julien Grall and Stefano Stabellini
>
> Hi Marek,
>
> Thanks for responding to Mario's question. I also have been doing these
> experiments with a Chromebook Snow, and I am the one who reported this
> problem on the xen-users ML here:
>
> https://lists.xenproject.org/archives/html/xen-users/2023-10/msg00021.html
>
> You might find that thread interesting, especially here with some additional
> log messages from the exynos_drm driver (exynos_drm_dma.c, I believe):
>
> https://lists.xenproject.org/archives/html/xen-users/2023-10/msg00032.html
>
> This issue is also discussed some on the xen-devel ML here:
>
> https://lists.xenproject.org/archives/html/xen-devel/2023-11/msg00003.html
>
> >
> > Frankly speaking I have no idea what might happen wrong. There have been
> > some changes recently in the Exynos IOMMU driver related to
> > initialization, maybe your changes related to Xen enabling changed
> > somehow the order of device initialization during boot. I assume that
> > the device-tree you use for the bare metal run and Xen enabled run
> > doesn't differ in the areas describing the hardware blocks.
> >
> > Please check if cherry-picking the commit
> > https://github.com/torvalds/linux/commit/bbc4d205d93f52ee18dfa7858d51489c0506547f
> > to your v6.1.59 based kernel helps anyhow.
>
> I tried adding that fix of the exynos IOMMU initialization from
> Linux > 6.2 on top of the 6.1.59 kernel I used for the original report,
> but that made no difference on Xen - it still failed with the mixer lacks
> support for IOMMU message and the screen is totally dark.
>
> >
> > If not, then as a temporary workaround please disable
> > CONFIG_DRM_EXYNOS_MIXER and CONFIG_DRM_EXYNOS_HDMI in your kernel config
> > and check what will happen (You will lose the HDMI output, but maybe
> > this won't a big issue).
>
> This change causes the GPU to work fairly well AFAICS. Removing the mixer
> and HDMI allowed the GPU to initialize, and the display manager started
> normally and enabled logging into an ordinary X11 session. Based on the log
> messages I was seeing, this was an obvious thing to try. Thanks for
> suggesting it.
>
> But I have a question:
>
> How are the mixer and HDMI devices related to the main drm device? The problem
> in the exynos_drm_dma driver was that on Xen, the main drm device wanted to
> use IOMMU version of dma_ops, but the mixer (and probably also the HDMI if
> it wouldn't have exited first) wanted to use the Xen swiotlb version of dma_ops,
> but on bare metal all three devices want to use the IOMMU version of dma_ops.
>
> The problem obviously has something to do with the fact that Xen does not
> expose the same IOMMU capability to Linux as is available when running on
> the bare metal.
>
> Cheers,
>
> >
> >
> > Best regards
>


-- 
Mario.

Re: exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks support for IOMMU

[Marek Szyprowski]

[-- Attachment #1: Type: text/plain, Size: 704 bytes --]


On 01.11.2023 03:50, Chuck Zmudzinski wrote:
> On 10/31/2023 7:45 PM, Stefano Stabellini wrote:
>> Unfortunately there is no easy solution.
>>
>> Do you know the version of the SMMU available on the platform?
> I am trying to discern, but I doubt we have v3 because we are
> working on a very old chromebook from 2012, and I am finding
> patches for smmv3 in Linux not starting until 2015. It is good to
> know about this option, though, for future work we might do on newer
> devices.

Just to clarify. Exynos SMMU is a custom hardware designed by Samsung, 
it is not based on ARM's SMMU. Linux has a separate driver for it.

 > ...

Best regards
-- 
Marek Szyprowski, PhD
Samsung R&D Institute Poland

[-- Attachment #2.1: Type: text/html, Size: 2065 bytes --]

[-- Attachment #2.2: Type: image/png, Size: 22957 bytes --]

Re: exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks support for IOMMU

[Mario Marietto]

[-- Attachment #1.1: Type: text/plain, Size: 1141 bytes --]

@Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> : I've
recompiled the kernel 6.6 on my ARM Chromebook to test if your patch helped
to fix the "black screen" issue. And not,it does not work. Is this expected
or is there something wrong with your patch ?

On Thu, Nov 2, 2023 at 11:00 AM Marek Szyprowski <m.szyprowski@samsung.com>
wrote:

>
> On 01.11.2023 03:50, Chuck Zmudzinski wrote:
>
> On 10/31/2023 7:45 PM, Stefano Stabellini wrote:
>
> Unfortunately there is no easy solution.
>
> Do you know the version of the SMMU available on the platform?
>
> I am trying to discern, but I doubt we have v3 because we are
> working on a very old chromebook from 2012, and I am finding
> patches for smmv3 in Linux not starting until 2015. It is good to
> know about this option, though, for future work we might do on newer
> devices.
>
> Just to clarify. Exynos SMMU is a custom hardware designed by Samsung, it
> is not based on ARM's SMMU. Linux has a separate driver for it.
>
> > ...
>
> Best regards
> --
> Marek Szyprowski, PhD
> Samsung R&D Institute Poland
>
>
>
>
>

-- 
Mario.

[-- Attachment #1.2: Type: text/html, Size: 2627 bytes --]

[-- Attachment #2: noname --]
[-- Type: image/png, Size: 22957 bytes --]

Re: exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks support for IOMMU

[Chuck Zmudzinski]

Hi everyone,

This reply is intended to clarify the latest test results and bring the
clarifications and other relevant discussion to the xen-devel mailing list.

On 11/1/2023 5:14 AM, Julien Grall wrote:
> 
> 
> On 01/11/2023 08:45, Chuck Zmudzinski wrote:
>> On 11/1/2023 4:27 AM, Julien Grall wrote:
>>> Hi,
>>>
>>> @Stefano, as you pointed out, there is already a thread on xen-users for
>>> this discussion. Could we use this thread for any discussion? This would
>>> make easier to follow.
>>>
>>> Some high level comment below.
>> 
>> I agree to keep the discussion here and not at other places.
> 
> I was meant to suggest the other thread :). But either is fine with me. 
> I just want to avoid avoid multiple seperate threads for the discussion.
> 
>> 
>> I just want to add that the best results for Xen dom0 so far are
>> by implementing Marek's suggestion to disable these two settings
>> in the 6.1.59 kernel config, but leaving everything else the same,
>> including keeping the EXYNOS_IOMMU support enabled:

I got even better results with a small patch in arch/arm/mm to disable
the overwriting of dma_ops with xen_swiotlb_dma_ops for a device when
the dma_ops are already set to use the iommu_ops, otherwise, the
current behavior of overwriting or setting dma_ops for the first time
with the xen_swiotlb_dma_ops is done. This totally fixes the error,
and also allows the HDMI port to work with Linux dom0 on Xen!

> That's good news! I would be interested to hear how this works once you 
> start to have PV backend in dom0 (I expect that the IOMMU will get 
> confused with grant mapping).

I did lots of tests such as building a kernel in a domU with PV block
and network frontend drivers connected to dom0 on the backend while
also building the qemu device model in dom0 using a Linux kernel in dom0
with the aforementioned patch to not overwrite dma_ops if dma_ops is
already set to iommu_ops, and on this Chromebook IOMMU had no confusion
and the feared DMA errors and memory corruption did not materialize!

So I am preparing to submit a patch to lkml to fix the exynos mixer.
on Xen. I just finished testing a version of the patch implemented as
a new config option that is set when support for the device causing
the trouble, the exynos mixer, is present in Linux and EXYNOS_IOMMU
config option is also enabled. I think this is a conservative approach
to add a new config option that can be set for cases like this
Chromebook when the devices that need to use IOMMU are behaving nicely
and do not cause any trouble on Xen. The default will continue to be
that Linux will overwrite IOMMU dma_ops with xen_swiotlb_dma_ops on
Xen unless the new config option is set.

> 
> Also, do you plan to passthrough any of the devices protected by IOMMU?

No. On this Chromebook, the only two devices using IOMMMU in the system on
dom0 with the soon-to-be proposed patch are the exynos-fimd and the
exynos-mixer, which support video for dom0. All other devices in the system
are using the xen_swiotlb_dma_ops. These facts, I think, explain why the
feared DMA errors and IOMMU confusion with the PV drivers for other guests
on the system did not materialize in this case. 

> 
>> 
>> # CONFIG_DRM_EXYNOS_MIXER is not set
>> 
>> Disabling the mixer also makes this unavailable:
>> 
>> # CONFIG_HDMI
>> 
>> With this change, the GPU is working well enough to allow the display
>> manager and an X11 session to run normally on the built-in display of the
>> Chromebook. The Wifi also works well.

As mentioned earlier, these settings worked also, but with the disadvantage
of disabling support for the HDMI port on the Chromebook. My latest tests
indicate we can fix this on Xen without giving up support for the HDMI!

> 
> I saw your other answer about the Wifi not working when the IOMMU is not 
> used. I was about to reply there, but instead I will do it here.
> 
> TBH, I am quite surprised this is the case. The only difference with 
> baremetal would be the RAM regions. Do you know if the Wifi dongle only 
> accept certain physical address?

The Wifi device worked well enough to associate with a Wifi access point
without EXYNOS_IOMMU enabled, it just failed to get IP addresses from
DHCP. I don't know if the exynos Wifi device requires a certain physical
address for the function of acquiring IP addresses from DHCP to work. Marek
might be able to answer that question.

In any case, since the Chromebook works fine on Xen, including Wifi, when
the EXYNOS_IOMMU is used by Linux dom0 as long as Linux does not overwrite
the dma_ops with xen_swiotlb_dma_ops when they had previously been set to
iommu_ops in arch/arm/mm/dma-mapping.c, finding the answer to the problem of
Wifi not working when the IOMMU is not used is not essential because the
default for both exynos systems and multi_v7 arm systems in Linux is to use
the exynos IOMMU when it is available, both on bare metal and on Xen / dom0.

Cheers,

> 
> Cheers,
>