[Virtio-fs] Issues running fio tests against a network mount routed from the host

[Virtio-fs] Issues running fio tests against a network mount routed from the host

[Bharat Kunwar]

[-- Attachment #1: Type: text/plain, Size: 1236 bytes --]

Hello

I am here to describe a problem that I’ve run into when trying to run `fio` against a network mount routed from the host using virtiofs. My wider context is that I am doing this in a kata-containerised Kubernetes environment using the new `kata-qemu` introduced in version 1.7. I have previously run the same tests with `kata-qemu` runtime with 9p filesystem and the same tests have run to completion.

Just to clarify, I can run the same fio tests to a local filesystem on the host and things run happily to completion. I have attached perf flame graphs corresponding to the failure cases, BeeGFS and CephFS and the successful case with host filesystem. The Fhgfs* process are related to BeeGFS.

I enabled debug on the kata side the the resulting systemd journal entries are captured in debug.log attached. Let me know if I can provide any more diagnostic info. I am currently attempting to build an image to boot up with symbols enabled so that I can capture some backtraces but that is all taking longer than expected (having never built a kernel tarball before). If there is a prebuilt image that I can use to boot up to use with virtiofs, let me know as that’d save me a ton of time.

Best

Bharat






[-- Attachment #2: kernel-beegfs-fio.svg --]
[-- Type: image/svg+xml, Size: 1302896 bytes --]

[-- Attachment #3: kernel-cephfs-fio.svg --]
[-- Type: image/svg+xml, Size: 793530 bytes --]

[-- Attachment #4: kernel-host-fio.svg --]
[-- Type: image/svg+xml, Size: 1017349 bytes --]

Re: [Virtio-fs] Issues running fio tests against a network mount routed from the host

[Eric Ren]

On Fri, Jun 14, 2019 at 09:45:27PM +0100, Bharat Kunwar wrote:
Hi,

> I am here to describe a problem that I’ve run into when trying to run `fio` against a network mount routed from the host using virtiofs. My wider context is that I am doing this in a kata-containerised Kubernetes environment using the new `kata-qemu` introduced in version 1.7. I have previously run the same tests with `kata-qemu` runtime with 9p filesystem and the same tests have run to completion.
> 

Could you please eleborate how you setup the test env and perform your
testing so that others can reproduce your problem?

Thanks,
Eric

> Just to clarify, I can run the same fio tests to a local filesystem on the host and things run happily to completion. I have attached perf flame graphs corresponding to the failure cases, BeeGFS and CephFS and the successful case with host filesystem. The Fhgfs* process are related to BeeGFS.
> 
> I enabled debug on the kata side the the resulting systemd journal entries are captured in debug.log attached. Let me know if I can provide any more diagnostic info. I am currently attempting to build an image to boot up with symbols enabled so that I can capture some backtraces but that is all taking longer than expected (having never built a kernel tarball before). If there is a prebuilt image that I can use to boot up to use with virtiofs, let me know as that’d save me a ton of time.
> 
> Best
> 
> Bharat
> 
> 
> 
> 
> 




> _______________________________________________
> Virtio-fs mailing list
> Virtio-fs@redhat.com
> https://www.redhat.com/mailman/listinfo/virtio-fs

Re: [Virtio-fs] Issues running fio tests against a network mount routed from the host

[Dr. David Alan Gilbert]

* Bharat Kunwar (bharat@stackhpc.com) wrote:
> Hello

Hi,

> I am here to describe a problem that I’ve run into when trying to run `fio` against a network mount routed from the host using virtiofs. My wider context is that I am doing this in a kata-containerised Kubernetes environment using the new `kata-qemu` introduced in version 1.7. I have previously run the same tests with `kata-qemu` runtime with 9p filesystem and the same tests have run to completion.
> 
> Just to clarify, I can run the same fio tests to a local filesystem on the host and things run happily to completion. I have attached perf flame graphs corresponding to the failure cases, BeeGFS and CephFS and the successful case with host filesystem. The Fhgfs* process are related to BeeGFS.
> 
> I enabled debug on the kata side the the resulting systemd journal entries are captured in debug.log attached. Let me know if I can provide any more diagnostic info. I am currently attempting to build an image to boot up with symbols enabled so that I can capture some backtraces but that is all taking longer than expected (having never built a kernel tarball before). If there is a prebuilt image that I can use to boot up to use with virtiofs, let me know as that’d save me a ton of time.

For some reason the SVGs don't seem to want to open for me; but they're
probably the wrong thing anyway; and I don't see the log you say you
attacked - those sound potentially useful.

Which fio test fails exactly, and with what error?

Dave

> Best
> 
> Bharat
> 
> 
> 
> 
> 




> _______________________________________________
> Virtio-fs mailing list
> Virtio-fs@redhat.com
> https://www.redhat.com/mailman/listinfo/virtio-fs

--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK