Kubernetes & Docker on Xilinx Versal Yocto

Kubernetes & Docker on Xilinx Versal Yocto

[Ramon Fried]

Hi all,
I'm working on the bring-up of Kubernetes worker node on Xilinx
Versal. (ARM A72)
The storage media (NOR flash) is too small, and I'm running all the
filesystems as tmpfs, which I load at boot time.

I'm experiencing a lot of issues bringing up Docker and Kubernetes
together this way, it seems that running these from tmpfs is not that
trivial.
The overlayfs causes real weird issues where suddenly the proc fs is
unmounted, symbolic links that stop working, etc.

Did anyone succeed running Kubernetes with Docker using the overlayfs2 plugin ?
I saw on the Docker manual, that overlayfs2 only supports ext4, I'm
wondering if I'm on the right track.

Would love to share notes.
Thanks,
Ramon.

Re: [meta-virtualization] Kubernetes & Docker on Xilinx Versal Yocto

[Robert Berger]

Hi,

My comments are inline

On 10/08/2021 18:06, Ramon Fried wrote:
> Hi all,
> I'm working on the bring-up of Kubernetes worker node on Xilinx
> Versal. (ARM A72)
> The storage media (NOR flash) is too small, and I'm running all the
> filesystems as tmpfs, which I load at boot time.
> 
> I'm experiencing a lot of issues bringing up Docker and Kubernetes
> together this way, it seems that running these from tmpfs is not that
> trivial.

I am not sure you can run an overlayfs over a tmpfs - never tried that.

> The overlayfs causes real weird issues where suddenly the proc fs is
> unmounted, symbolic links that stop working, etc.
> 
> Did anyone succeed running Kubernetes with Docker using the overlayfs2 plugin ?
> I saw on the Docker manual, that overlayfs2 only supports ext4, I'm
> wondering if I'm on the right track.

I typically use an SD card (big enough) and dedicate a btrfs partition 
for containers and/or kubernetes.

e.g. /dev/mmcblk0p3 on /var/lib/rancher type btrfs 
(rw,noatime,ssd,space_cache,subvolid=5,subvol=/)

This seems to work fine and as a nice side effect I can use btrfs 
subvolumes for snapshots.

kubernetes writes to a subvolume /var/lib/rancher/k3s and I can do 
snapshots e.g. to /var/lib/rancher/snapshot/k3s/xxx

root@k3s-node-1:/var/lib/rancher# btrfs subvolume list /var/lib/rancher
ID 257 gen 16 top level 5 path k3s
ID 258 gen 22 top level 5 path snapshot
ID 260 gen 21 top level 258 path snapshot/k3s/2021-08-10-11:25:17
ID 261 gen 21 top level 258 path snapshot/k3s/2021-08-10-11:28:00

> 
> Would love to share notes.
> Thanks,
> Ramon.

Regards,

Robert

> 
> 
> 
> 
> 

Re: [meta-virtualization] Kubernetes & Docker on Xilinx Versal Yocto

[Ramon Fried]

On Fri, Aug 13, 2021 at 11:28 AM Robert Berger@yocto.user
<robert.berger.yocto.user@gmail.com> wrote:
>
> Hi,
>
> My comments are inline
>
> On 10/08/2021 18:06, Ramon Fried wrote:
> > Hi all,
> > I'm working on the bring-up of Kubernetes worker node on Xilinx
> > Versal. (ARM A72)
> > The storage media (NOR flash) is too small, and I'm running all the
> > filesystems as tmpfs, which I load at boot time.
> >
> > I'm experiencing a lot of issues bringing up Docker and Kubernetes
> > together this way, it seems that running these from tmpfs is not that
> > trivial.
>
> I am not sure you can run an overlayfs over a tmpfs - never tried that.
>
> > The overlayfs causes real weird issues where suddenly the proc fs is
> > unmounted, symbolic links that stop working, etc.
> >
> > Did anyone succeed running Kubernetes with Docker using the overlayfs2 plugin ?
> > I saw on the Docker manual, that overlayfs2 only supports ext4, I'm
> > wondering if I'm on the right track.
>
> I typically use an SD card (big enough) and dedicate a btrfs partition
> for containers and/or kubernetes.
>
> e.g. /dev/mmcblk0p3 on /var/lib/rancher type btrfs
> (rw,noatime,ssd,space_cache,subvolid=5,subvol=/)
>
> This seems to work fine and as a nice side effect I can use btrfs
> subvolumes for snapshots.
>
> kubernetes writes to a subvolume /var/lib/rancher/k3s and I can do
> snapshots e.g. to /var/lib/rancher/snapshot/k3s/xxx
>
> root@k3s-node-1:/var/lib/rancher# btrfs subvolume list /var/lib/rancher
> ID 257 gen 16 top level 5 path k3s
> ID 258 gen 22 top level 5 path snapshot
> ID 260 gen 21 top level 258 path snapshot/k3s/2021-08-10-11:25:17
> ID 261 gen 21 top level 258 path snapshot/k3s/2021-08-10-11:28:00
>
> >
> > Would love to share notes.
> > Thanks,
> > Ramon.
>
> Regards,
>
> Robert
>
> >
> >
> >
> > 
> >
>
Thanks Robert,
I finally managed to run Kubernetes by setting the storage driver to
"vfs", although Kubernetes complains in pre-flight, it actually works.