Intention to work on GSoC project

Intention to work on GSoC project

[Sahil]

Hi,

My name is Sahil and I go by the pseudonym 'valdaarhun' on Github. I have
never contributed to QEMU before but I have used it a few times as an end
user. I developed an interest in virtualization during my internship at
VMware and would like to dive deeper in this subfield.

My current full-time job does not allow me to take part in external programs
that are paid. I would like to work on one of the proposed projects outside
of GSoC. I have gone through QEMU's list of GSoC '24 projects [1] and am
interested in two of them:

1. Add packed virtqueue to Shadow Virtqueue
2. vhost-user memory isolation

Based on what I have understood, they are somewhat related and are part
of the migration subsystem. I feel the learning curve of the first project
will be less steep and will make me better prepared to tackle the second
project as well.

I have read the "Getting Started for Developers" [2] wiki page. I have also
built QEMU from source.

I think my next step should be to read the documentation on the migration
subsystem [3], the blog posts attached in the first project's description
and virtqueue's implementation. Would you also recommend that I work on a
QEMU issue that is open on Gitlab and related to virtqueues/virtio to 
familiarize
myself with the codebase? I went through the issues tagged as "device:virtio" 
[4]
but can't really tell if any of them are good for beginners. One of them has 
the
"bite-size" tag [5]. It also has a patch attached but hasn't been merged. 
Shall I
work on getting that merged?

I have worked on a few smaller systems programming issues in other
organizations (eg: strace [6], htop [7]) in the past.

I look forward to hearing from you.

Regards,
Sahil
https://github.com/valdaarhun

[1] https://wiki.qemu.org/Google_Summer_of_Code_2024
[2] https://wiki.qemu.org/Documentation/GettingStartedDevelopers
[3] https://qemu-project.gitlab.io/qemu/devel/migration/index.html
[4] https://gitlab.com/qemu-project/qemu/-/issues/?
label_name%5B%5D=device%3Avirtio
[5] https://gitlab.com/qemu-project/qemu/-/issues/230
[6] https://github.com/strace/strace/commits?author=valdaarhun
[7] https://github.com/htop-dev/htop/commits?author=valdaarhun

Re: Intention to work on GSoC project

[Stefano Garzarella]

Hi Sahil,

On Sun, Feb 25, 2024 at 10:38 PM Sahil <icegambit91@gmail.com> wrote:
>
> Hi,
>
> My name is Sahil and I go by the pseudonym 'valdaarhun' on Github. I have
> never contributed to QEMU before but I have used it a few times as an end
> user. I developed an interest in virtualization during my internship at
> VMware and would like to dive deeper in this subfield.
>
> My current full-time job does not allow me to take part in external programs
> that are paid. I would like to work on one of the proposed projects outside
> of GSoC.

Sure, not a problem at all, also because for this year QEMU was not
accepted in GSoC, so anybody can work on those projects if they have
time

> I have gone through QEMU's list of GSoC '24 projects [1] and am
> interested in two of them:
>
> 1. Add packed virtqueue to Shadow Virtqueue
> 2. vhost-user memory isolation
>
> Based on what I have understood, they are somewhat related and are part
> of the migration subsystem. I feel the learning curve of the first project
> will be less steep and will make me better prepared to tackle the second
> project as well.

The first project is for sure related with migration. While vhost-user
memory isolation is not really related to migration, but both are
related to virtio devices.
Anyway, your plan looks good to me!

>
> I have read the "Getting Started for Developers" [2] wiki page. I have also
> built QEMU from source.

Great!

>
> I think my next step should be to read the documentation on the migration
> subsystem [3], the blog posts attached in the first project's description
> and virtqueue's implementation. Would you also recommend that I work on a
> QEMU issue that is open on Gitlab and related to virtqueues/virtio to
> familiarize
> myself with the codebase? I went through the issues tagged as "device:virtio"
> [4]
> but can't really tell if any of them are good for beginners. One of them has
> the
> "bite-size" tag [5]. It also has a patch attached but hasn't been merged.
> Shall I
> work on getting that merged?

Yeah, "bite-size" issues should be better to understand how to
contribute to QEMU.
Feel free to work on any issue, doing the work or helping to complete
old patches.

>
> I have worked on a few smaller systems programming issues in other
> organizations (eg: strace [6], htop [7]) in the past.
>
> I look forward to hearing from you.

Feel free to reach us if you have more questions on the projects.

Thanks,
Stefano

Re: Intention to work on GSoC project

[Sahil]

Hi,

Thank you for your reply.

On Thursday, February 29, 2024 2:02:25 PM IST you wrote:
> [...]
> The first project is for sure related with migration. While vhost-user
> memory isolation is not really related to migration, but both are
> related to virtio devices.
> [...]
> Yeah, "bite-size" issues should be better to understand how to
> contribute to QEMU. Feel free to work on any issue, doing the work
> or helping to complete old patches.

Got it. In that case it'll be best to complete the bite-sized task related to
virtio [1]. A patch [2] had been long ago but it hasn't been completed yet.
I'll work on getting it merged.

I have already started reading the documentation on the migration subsystem
and the hands-on blogs in the project description. That will help me get started
with the first project.

> Feel free to reach us if you have more questions on the projects.

Yes, I'll definitely do that.

Thanks,
Sahil

[1] https://gitlab.com/qemu-project/qemu/-/issues/230
[2] https://patchwork.kernel.org/project/qemu-devel/patch/20220414112902.41390-1-codeguy.moteen@gmail.com/

Re: Intention to work on GSoC project

[Eugenio Perez Martin]

On Thu, Feb 29, 2024 at 9:32 AM Stefano Garzarella <sgarzare@redhat.com> wrote:
>
> Hi Sahil,
>
> On Sun, Feb 25, 2024 at 10:38 PM Sahil <icegambit91@gmail.com> wrote:
> >
> > Hi,
> >
> > My name is Sahil and I go by the pseudonym 'valdaarhun' on Github. I have
> > never contributed to QEMU before but I have used it a few times as an end
> > user. I developed an interest in virtualization during my internship at
> > VMware and would like to dive deeper in this subfield.
> >
> > My current full-time job does not allow me to take part in external programs
> > that are paid. I would like to work on one of the proposed projects outside
> > of GSoC.
>
> Sure, not a problem at all, also because for this year QEMU was not
> accepted in GSoC, so anybody can work on those projects if they have
> time
>
> > I have gone through QEMU's list of GSoC '24 projects [1] and am
> > interested in two of them:
> >
> > 1. Add packed virtqueue to Shadow Virtqueue
> > 2. vhost-user memory isolation
> >
> > Based on what I have understood, they are somewhat related and are part
> > of the migration subsystem. I feel the learning curve of the first project
> > will be less steep and will make me better prepared to tackle the second
> > project as well.
>
> The first project is for sure related with migration. While vhost-user
> memory isolation is not really related to migration, but both are
> related to virtio devices.
> Anyway, your plan looks good to me!
>

I agree with all Stefano's answers above.

> >
> > I have read the "Getting Started for Developers" [2] wiki page. I have also
> > built QEMU from source.
>
> Great!
>
> >
> > I think my next step should be to read the documentation on the migration
> > subsystem [3], the blog posts attached in the first project's description
> > and virtqueue's implementation.

You can add the recently published
"virtio-live-migration-technical-deep-dive" :) [1].

[1] https://developers.redhat.com/articles/2024/02/21/virtio-live-migration-technical-deep-dive

> > Would you also recommend that I work on a
> > QEMU issue that is open on Gitlab and related to virtqueues/virtio to
> > familiarize
> > myself with the codebase? I went through the issues tagged as "device:virtio"
> > [4]
> > but can't really tell if any of them are good for beginners. One of them has
> > the
> > "bite-size" tag [5]. It also has a patch attached but hasn't been merged.
> > Shall I
> > work on getting that merged?
>
> Yeah, "bite-size" issues should be better to understand how to
> contribute to QEMU.
> Feel free to work on any issue, doing the work or helping to complete
> old patches.
>

Right,

There are few other tasks pending for QEMU in general and SVQ in
particular. I'll add them as gitlab issues by today.

Thanks!

> >
> > I have worked on a few smaller systems programming issues in other
> > organizations (eg: strace [6], htop [7]) in the past.
> >
> > I look forward to hearing from you.
>
> Feel free to reach us if you have more questions on the projects.
>
> Thanks,
> Stefano
>

Re: Intention to work on GSoC project

[Sahil]

Hi,

On Friday, March 1, 2024 1:10:49 PM IST you wrote:
> [...]
> You can add the recently published
> "virtio-live-migration-technical-deep-dive" :) [1].
> 
> [1]
> https://developers.redhat.com/articles/2024/02/21/virtio-live-migration-technical-deep-dive

Thank you. This resource will help me cover a lot of ground.

> [...]
> There are few other tasks pending for QEMU in general and SVQ in
> particular. I'll add them as gitlab issues by today.
> 

That would be nice. I'll keep an eye out for the new issues.

Thanks,
Sahil

Re: Intention to work on GSoC project

[Eugenio Perez Martin]

On Fri, Mar 1, 2024 at 7:29 PM Sahil <icegambit91@gmail.com> wrote:
>
> Hi,
>
> On Friday, March 1, 2024 1:10:49 PM IST you wrote:
> > [...]
> > You can add the recently published
> > "virtio-live-migration-technical-deep-dive" :) [1].
> >
> > [1]
> > https://developers.redhat.com/articles/2024/02/21/virtio-live-migration-technical-deep-dive
>
> Thank you. This resource will help me cover a lot of ground.
>
> > [...]
> > There are few other tasks pending for QEMU in general and SVQ in
> > particular. I'll add them as gitlab issues by today.
> >
>
> That would be nice. I'll keep an eye out for the new issues.
>

Hi Sahil,

It's being hard to find a good self-contained small task related to
the project to be honest. As it would be out of SVQ, would it be ok
for you if we start straight to the task of adding the packed vq
format to SVQ?

Thanks!

Re: Intention to work on GSoC project

[Sahil]

Hi,

Thank you for your email.

On Thursday, March 14, 2024 8:39:45 PM IST Eugenio Perez Martin wrote:
> Hi Sahil,
> 
> It's being hard to find a good self-contained small task related to
> the project to be honest. As it would be out of SVQ, would it be ok
> for you if we start straight to the task of adding the packed vq
> format to SVQ?
> 
> Thanks!

Sure, this works too! I would love to get started with the project.

I have a small update as well. I have read through a few docs and
articles to familiarize myself with the relevant terminology and
technicalities.

1. "About", "system emulation" and "user mode emulation" sections of
    the user documentation [1]
2. The migration subsystem [2]

Some sections in the above docs were difficult to grasp. For the time
being, I have focused on those parts that I thought were relevant
to the project.

I have also read through the following articles:

1. Introduction to virtio-networking and vhost-net [3]
2. Deep dive into Virtio-networking and vhost-net [4]
3. Virtualized Hardware Devices [5]
4. VFIO - "Virtual Function I/O" (Just the introduction) [6]
5. Virtio-net failover: An introduction [7]

I hope I haven't gone off on a tangent. I was planning to finish reading
up on the following articles as well:

1. Virtqueues and virtio ring: How the data travels [8]
2. Packed virtqueue: How to reduce overhead with virtio [9]
3. Virtio live migration technical deep dive [10]
4. Hands on vDPA: what do you do when you ain't got the hardware v2 (Part 1) [11]

I believe the hands-on vPDA article will have me set up a development
environment for the project as well.

Please let me know if I should amend my roadmap. I am
excited to get started :)

Thanks,
Sahil

[1] https://www.qemu.org/docs/master/index.html
[2] https://www.qemu.org/docs/master/devel/migration/index.html
[3] https://www.redhat.com/en/blog/introduction-virtio-networking-and-vhost-net
[4] https://www.redhat.com/en/blog/deep-dive-virtio-networking-and-vhost-net
[5] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_getting_started_guide/sec-virtualization_getting_started-products-virtualized-hardware-devices
[6] https://www.kernel.org/doc/html/latest/driver-api/vfio.html
[7] https://www.redhat.com/en/blog/virtio-net-failover-introduction
[8] https://www.redhat.com/en/blog/virtqueues-and-virtio-ring-how-data-travels
[9] https://developers.redhat.com/articles/2024/02/21/virtio-live-migration-technical-deep-dive
[10] https://www.redhat.com/en/blog/packed-virtqueue-how-reduce-overhead-virtio
[11] https://www.redhat.com/en/blog/hands-vdpa-what-do-you-do-when-you-aint-got-hardware-part-1

Re: Intention to work on GSoC project

[Eugenio Perez Martin]

On Fri, Mar 15, 2024 at 8:15 AM Sahil <icegambit91@gmail.com> wrote:
>
> Hi,
>
> Thank you for your email.
>
> On Thursday, March 14, 2024 8:39:45 PM IST Eugenio Perez Martin wrote:
> > Hi Sahil,
> >
> > It's being hard to find a good self-contained small task related to
> > the project to be honest. As it would be out of SVQ, would it be ok
> > for you if we start straight to the task of adding the packed vq
> > format to SVQ?
> >
> > Thanks!
>
> Sure, this works too! I would love to get started with the project.
>
> I have a small update as well. I have read through a few docs and
> articles to familiarize myself with the relevant terminology and
> technicalities.
>
> 1. "About", "system emulation" and "user mode emulation" sections of
>     the user documentation [1]
> 2. The migration subsystem [2]
>
> Some sections in the above docs were difficult to grasp. For the time
> being, I have focused on those parts that I thought were relevant
> to the project.
>

Please feel free to ask any questions, maybe we can improve the doc :).

> I have also read through the following articles:
>
> 1. Introduction to virtio-networking and vhost-net [3]
> 2. Deep dive into Virtio-networking and vhost-net [4]
> 3. Virtualized Hardware Devices [5]
> 4. VFIO - "Virtual Function I/O" (Just the introduction) [6]
> 5. Virtio-net failover: An introduction [7]
>
> I hope I haven't gone off on a tangent. I was planning to finish reading
> up on the following articles as well:
>

There is a post before the first in the series:
https://www.redhat.com/en/blog/virtio-devices-and-drivers-overview-headjack-and-phone

> 1. Virtqueues and virtio ring: How the data travels [8]
> 2. Packed virtqueue: How to reduce overhead with virtio [9]
> 3. Virtio live migration technical deep dive [10]
> 4. Hands on vDPA: what do you do when you ain't got the hardware v2 (Part 1) [11]
>

I think it's a good plan!

If you feel like you're reading a lot of theory and want to get your
hands dirty already, you can also start messing with the code with the
blogs you already read. Or, maybe, after reading the Packed virtqueue
one, your call.

In a very brute-forced description, you can start trying to copy all
the *packed* stuff of kernel's drivers/virtio/virtio_ring.c into
vhost_shadow_virtqueue.c. There is a lot more in the task, and I can
get into more detail if you want either here or in a meeting.

If you prefer to continue with the theory it is ok too.

> I believe the hands-on vPDA article will have me set up a development
> environment for the project as well.
>

Yes, that's right.

> Please let me know if I should amend my roadmap. I am
> excited to get started :)
>

I think it is a great plan!

Thanks!

> Thanks,
> Sahil
>
> [1] https://www.qemu.org/docs/master/index.html
> [2] https://www.qemu.org/docs/master/devel/migration/index.html
> [3] https://www.redhat.com/en/blog/introduction-virtio-networking-and-vhost-net
> [4] https://www.redhat.com/en/blog/deep-dive-virtio-networking-and-vhost-net
> [5] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_getting_started_guide/sec-virtualization_getting_started-products-virtualized-hardware-devices
> [6] https://www.kernel.org/doc/html/latest/driver-api/vfio.html
> [7] https://www.redhat.com/en/blog/virtio-net-failover-introduction
> [8] https://www.redhat.com/en/blog/virtqueues-and-virtio-ring-how-data-travels
> [9] https://developers.redhat.com/articles/2024/02/21/virtio-live-migration-technical-deep-dive
> [10] https://www.redhat.com/en/blog/packed-virtqueue-how-reduce-overhead-virtio
> [11] https://www.redhat.com/en/blog/hands-vdpa-what-do-you-do-when-you-aint-got-hardware-part-1
>
>

Re: Intention to work on GSoC project

[Sahil]

Hi,

Thank you for your reply.

On Friday, March 15, 2024 4:57:39 PM IST Eugenio Perez Martin wrote:
> [...]
> > Some sections in the above docs were difficult to grasp. For the time
> > being, I have focused on those parts that I thought were relevant
> > to the project.
> 
> Please feel free to ask any questions, maybe we can improve the doc :).

I understood the introductory sections of the documentation such as the
"About QEMU" section and the first half of the "system emulation". Sections
and subsections that went into greater detail were a little overwhelming
such as the "QEMU virtio-net standby" subsection [1] or the "migration
features" [2] subsection. But the red hat blogs and deep-dive articles helped
cover a lot of ground conceptually.

I feel once I start getting my hands dirty, I'll be able to absorb these concepts
much better.

I did have two questions that I would like to ask.

Q1.
Regarding the "Deep dive into Virtio-networking and vhost-net" article [3],
the "Introduction" subsection of the "Vhost protocol" section mentions that
sending the available buffer notification involves a vCPU interrupt (4th bullet
point). But in figure 2, the arrow for the "available buffer notification" indicates
a PCI interrupt. Initially I thought they were two different interrupts but I am
a little confused about this now.

Q2.
In the "Virtio-net failover operation" section of the "Virtio-net failover: An
introduction" article [4], there are five bullet points under the first figure.
The second point states that the guest kernel needs the ability to switch
between the VFIO device and the vfio-net device. I was wondering if
"vfio-net" is a typo and if it should be "virtio-net" instead.

> [...]
> There is a post before the first in the series:
> https://www.redhat.com/en/blog/virtio-devices-and-drivers-overview-headjack-
> and-phone

Got it. I didn't know this was the first in the series. I have now covered this as
well, so I can move on to "Virtqueues and virtio ring: How the data travels" [3] :)

> > 1. Virtqueues and virtio ring: How the data travels [8]
> > 2. Packed virtqueue: How to reduce overhead with virtio [9]
> > 3. Virtio live migration technical deep dive [10]
> > 4. Hands on vDPA: what do you do when you ain't got the hardware v2 (Part
> > 1) [11]
> I think it's a good plan!
> 
> If you feel like you're reading a lot of theory and want to get your
> hands dirty already, you can also start messing with the code with the
> blogs you already read. Or, maybe, after reading the Packed virtqueue
> one, your call.
> 
> In a very brute-forced description, you can start trying to copy all
> the *packed* stuff of kernel's drivers/virtio/virtio_ring.c into
> vhost_shadow_virtqueue.c.

I would love to start with some hands-on tasks. I'll take a look at
the kernel's "drivers/virtio/virtio_ring.c". I think I should also start
going through the "vhost_shadow_virtqueue.c" [4] source code.

> There is a lot more in the task, and I can get into more detail
> if you want either here or in a meeting.

Thank you. Either means of communication works for me although
the latter will require some coordination.

> If you prefer to continue with the theory it is ok too.

A good balance of theory and practice would be nice at this stage.
It'll keep my brains from getting too muddled up.

Thanks,
Sahil

[1] https://www.qemu.org/docs/master/system/virtio-net-failover.html
[2] https://www.qemu.org/docs/master/devel/migration/features.html
[3] https://www.redhat.com/en/blog/deep-dive-virtio-networking-and-vhost-net
[4] https://www.redhat.com/en/blog/virtio-net-failover-introduction
[5] https://www.redhat.com/en/blog/virtqueues-and-virtio-ring-how-data-travels

Re: Intention to work on GSoC project

[Sahil]

Hi,

I was reading the "Virtqueues and virtio ring: How the data travels"
article [1]. There are a few things that I have not understood in the
"avail rings" section.

Q1.
Step 2 in the "Process to make a buffer available" diagram depicts
how the virtio driver writes the descriptor index in the avail ring.
In the example, the descriptor index #0 is written in the first entry.
But in figure 2, the number 0 is in the 4th position in the avail ring.
Is the avail ring queue an array of "struct virtq_avail" which maintains
metadata such as the number of descriptor indexes in the header?

Also, in the second position, the number changes from 0 (figure 1) to
1 (figure 2). I haven't understood what idx, 0 (later 1) and ring[] represent
in the figures. Does this number represent the number of descriptors
that are currently in the avail ring?

Q2.

There's this paragraph in the article right below the above mentioned
diagram:

> The avail ring must be able to hold the same number of descriptors
> as the descriptor area, and the descriptor area must have a size power
> of two, so idx wraps naturally at some point. For example, if the ring
> size is 256 entries, idx 1 references the same descriptor as idx 257, 513...
> And it will wrap at a 16 bit boundary. This way, neither side needs to
> worry about processing an invalid idx: They are all valid.

I haven't really understood this. I have understood that idx is calculated
as idx mod queue_length. But I haven't understood the "16 bit boundary"
part.

I am also not very clear on how a queue length that is not a power of 2
might cause trouble. Could you please expand on this?

Q3.
I have started going through the source code in "drivers/virtio/virtio_ring.c".
I have understood that the virtio driver runs in the guest's kernel. Does that
mean the drivers in "drivers/virtio/*" are enabled when linux is being run in
a guest VM?

Thanks,
Sahil

[1] https://www.redhat.com/en/blog/virtqueues-and-virtio-ring-how-data-travels

Re: Intention to work on GSoC project

[Eugenio Perez Martin]

On Mon, Mar 18, 2024 at 8:47 PM Sahil <icegambit91@gmail.com> wrote:
>
> Hi,
>
> I was reading the "Virtqueues and virtio ring: How the data travels"
> article [1]. There are a few things that I have not understood in the
> "avail rings" section.
>
> Q1.
> Step 2 in the "Process to make a buffer available" diagram depicts
> how the virtio driver writes the descriptor index in the avail ring.
> In the example, the descriptor index #0 is written in the first entry.
> But in figure 2, the number 0 is in the 4th position in the avail ring.
> Is the avail ring queue an array of "struct virtq_avail" which maintains
> metadata such as the number of descriptor indexes in the header?
>

struct virtq_avail has two members: uint16_t idx and ring[]. To be in
the first position of the avail ring means to be in ring[0] there.

Idx and ring[] are just headers in the figure, not actual positions.
Same as Avail. Now that you mention maybe there is a better way to
represent that, yes.

Let me know if I didn't explain it well.

> Also, in the second position, the number changes from 0 (figure 1) to
> 1 (figure 2). I haven't understood what idx, 0 (later 1) and ring[] represent
> in the figures. Does this number represent the number of descriptors
> that are currently in the avail ring?
>

It is the position in ring[] where the device needs to stop looking
for descriptors. It starts at 0, and when the device sees 1 it means
ring[0] has a descriptor to process.

Now you need to apply a "modulo virtqueue size" to that index. So if
the virtqueue is 256, avail_idx 257 means the last valid descriptor is
at 0. This happens naturally when the driver keeps adding descriptors
and wraps the queue.

The authoritative source of this is the VirtQueues section of the
virtio standard [1], feel free to check it in case it clarifies
something better.

> Q2.
>
> There's this paragraph in the article right below the above mentioned
> diagram:
>
> > The avail ring must be able to hold the same number of descriptors
> > as the descriptor area, and the descriptor area must have a size power
> > of two, so idx wraps naturally at some point. For example, if the ring
> > size is 256 entries, idx 1 references the same descriptor as idx 257, 513...
> > And it will wrap at a 16 bit boundary. This way, neither side needs to
> > worry about processing an invalid idx: They are all valid.
>
> I haven't really understood this. I have understood that idx is calculated
> as idx mod queue_length. But I haven't understood the "16 bit boundary"
> part.
>

avail_idx is an uin16_t, so ((uint16_t)-1) + 1 == 0.

> I am also not very clear on how a queue length that is not a power of 2
> might cause trouble. Could you please expand on this?
>

That's a limitation in the standard, but I'm not sure where it comes
from beyond being computationally easier to calculate ring position
with a mask than with a remainder of a random non-power-of-two number.
Packed virtqueue removes that limitation.

> Q3.
> I have started going through the source code in "drivers/virtio/virtio_ring.c".
> I have understood that the virtio driver runs in the guest's kernel. Does that
> mean the drivers in "drivers/virtio/*" are enabled when linux is being run in
> a guest VM?
>

For PCI devices, as long as it detects a device with vendor == Red
Hat, Inc. (0x1AF4) and device ID 0x1000 through 0x107F inclusive, yes.
You can also load and unload manually with modprobe as other drivers.

Let me know if you have more doubts. Thanks!

[1] https://docs.oasis-open.org/virtio/virtio/v1.3/virtio-v1.3.html


> Thanks,
> Sahil
>
> [1] https://www.redhat.com/en/blog/virtqueues-and-virtio-ring-how-data-travels
>
>
>
>

Re: Intention to work on GSoC project

[Sahil]

Hi,

Thank you for your reply.

On Wednesday, March 20, 2024 9:27:00 PM IST Eugenio Perez Martin wrote:
> [...]
> > Q1.
> > Regarding the "Deep dive into Virtio-networking and vhost-net"
> > article [3], the "Introduction" subsection of the "Vhost protocol"
> > section mentions that sending the available buffer notification
> > involves a vCPU interrupt (4th bullet point).
> 
> Now I realize we used a very misleading term there :). Without
> ioeventfd, when the guest writes to the PCI notification area the
> guest vCPU is totally paused there, and the control is handed to
> host's KVM first and QEMU after it. The same physical CPU of the
> machine needs to switch context because of that.
> 
> Is an interruption of the execution and a context switch. Maybe
> "paused" is a better term.
> 
> > But in figure 2, the arrow for the "available buffer notification"
> > indicates a PCI interrupt. Initially I thought they were two different
> > interrupts but I am a little confused about this now.
> 
> They are different, but at that part of the blog is just the direction
> of who interrupts / notifies who :).

Thank you for the clarification. This makes sense now.

On Wednesday, March 20, 2024 9:59:30 PM IST Eugenio Perez Martin wrote:
> [...]
> > Q1.
> > Step 2 in the "Process to make a buffer available" diagram depicts
> > how the virtio driver writes the descriptor index in the avail ring.
> > In the example, the descriptor index #0 is written in the first entry.
> > But in figure 2, the number 0 is in the 4th position in the avail ring.
> > Is the avail ring queue an array of "struct virtq_avail" which maintains
> > metadata such as the number of descriptor indexes in the header?
> 
> struct virtq_avail has two members: uint16_t idx and ring[]. To be in
> the first position of the avail ring means to be in ring[0] there.
> 
> Idx and ring[] are just headers in the figure, not actual positions.
> Same as Avail. Now that you mention maybe there is a better way to
> represent that, yes.
> 
> Let me know if I didn't explain it well.
> 
> > Also, in the second position, the number changes from 0 (figure 1) to
> > 1 (figure 2). I haven't understood what idx, 0 (later 1) and ring[]
> > represent in the figures. Does this number represent the number of
> > descriptors that are currently in the avail ring?
> 
> It is the position in ring[] where the device needs to stop looking
> for descriptors. It starts at 0, and when the device sees 1 it means
> ring[0] has a descriptor to process.
> 
> Now you need to apply a "modulo virtqueue size" to that index. So if
> the virtqueue is 256, avail_idx 257 means the last valid descriptor is
> at 0. This happens naturally when the driver keeps adding descriptors
> and wraps the queue.
> 
> The authoritative source of this is the VirtQueues section of the
> virtio standard [1], feel free to check it in case it clarifies
> something better.

I have understood this as well. Reading the virtio specification was
definitely helpful.

> > Q2.
> > 
> > There's this paragraph in the article right below the above mentioned
> > diagram:
> > > The avail ring must be able to hold the same number of descriptors
> > > as the descriptor area, and the descriptor area must have a size power
> > > of two, so idx wraps naturally at some point. For example, if the ring
> > > size is 256 entries, idx 1 references the same descriptor as idx 257,
> > > 513... And it will wrap at a 16 bit boundary. This way, neither side
> > > needs to worry about processing an invalid idx: They are all valid.
> > 
> > I haven't really understood this. I have understood that idx is calculated
> > as idx mod queue_length. But I haven't understood the "16 bit boundary"
> > part.
> 
> avail_idx is an uin16_t, so ((uint16_t)-1) + 1 == 0.
> > I am also not very clear on how a queue length that is not a power of 2
> > might cause trouble. Could you please expand on this?
> 
> That's a limitation in the standard, but I'm not sure where it comes
> from beyond being computationally easier to calculate ring position
> with a mask than with a remainder of a random non-power-of-two number.

Understood.

> Packed virtqueue removes that limitation.

Right, I noticed that "queue_size" does not have to be a power of 2 for
packed virtqueues.

> > Q3.
> > I have started going through the source code in
> > "drivers/virtio/virtio_ring.c". I have understood that the virtio driver
> > runs in the guest's kernel. Does that mean the drivers in
> > "drivers/virtio/*" are enabled when linux is being run in a guest VM?
> 
> For PCI devices, as long as it detects a device with vendor == Red
> Hat, Inc. (0x1AF4) and device ID 0x1000 through 0x107F inclusive, yes.
> You can also load and unload manually with modprobe as other drivers.

Got it. The linux documentation in "documentation/driver-api/virtio/virtio.rst"
covers this as well. I must have missed this before sending the email.

> Let me know if you have more doubts. Thanks!

I have also read through the "Packed virtqueue: How to reduce overhead
with virtio" article [1] and the relevant section from the virtio specification [2].
This has been slightly difficult to grasp. Going through the source in
"drivers/virtio/virtio_ring.c" is helping a bit but I am still confused about
a few things.

Q1.
Section 2.7.4 of the virtio spec [3] states that in an available descriptor, the
"Element Length" stores the length of the buffer element. In the next few lines,
it also states that the "Element Length" is reserved for used descriptors and is
ignored by drivers. This sounds a little contradictory given that drivers write
available desciptors in the descriptor ring.

Q2.
In the Red Hat article, just below the first listing ("Memory layout of a packed
virtqueue descriptor"), there's the following line referring to the buffer id in
"virtq_desc":

> This time, the id field is not an index for the device to look for the buffer: it is
> an opaque value for it, only has meaning for the driver.

But the device returns the buffer id when it writes the used descriptor to the
descriptor ring. The "only has meaning for the driver" part has got me a little
confused. Which buffer id is this that the device returns? Is it related to the
buffer id in the available descriptor?

Thanks,
Sahil

[1] https://www.redhat.com/en/blog/packed-virtqueue-how-reduce-overhead-virtio
[2] https://docs.oasis-open.org/virtio/virtio/v1.1/csprd01/virtio-v1.1-csprd01.html#x1-610007
[3] https://docs.oasis-open.org/virtio/virtio/v1.1/csprd01/virtio-v1.1-csprd01.html#x1-650004

Re: Intention to work on GSoC project

[daleyoung4242]

Hi,

On Monday, March 25, 2024 21:20:32 CST Sahil wrote:
> Q1.
> Section 2.7.4 of the virtio spec [3] states that in an available descriptor,
> the "Element Length" stores the length of the buffer element. In the next
> few lines, it also states that the "Element Length" is reserved for used
> descriptors and is ignored by drivers. This sounds a little contradictory
> given that drivers write available desciptors in the descriptor ring.

When VIRTQ_DESC_F_WRITE is set, the device will use "Element Length" to 
specify the length it writes. When VIRTQ_DESC_F_WRITE is not set, which means 
the buffer is read-only for the device, "Element Length" will not be changed by 
the device, so drivers just ignore it.

> Q2.
> In the Red Hat article, just below the first listing ("Memory layout of a
> packed virtqueue descriptor"), there's the following line referring to the
> buffer id in
> "virtq_desc":
> > This time, the id field is not an index for the device to look for the
> > buffer: it is an opaque value for it, only has meaning for the driver.
> 
> But the device returns the buffer id when it writes the used descriptor to
> the descriptor ring. The "only has meaning for the driver" part has got me
> a little confused. Which buffer id is this that the device returns? Is it
> related to the buffer id in the available descriptor?

In my understanding, buffer id is the element that avail descriptor marks to 
identify when adding descriptors to table. Device will returns the buffer id in 
the processed descriptor or the last descriptor in a chain, and write it to 
the descriptor that used idx refers to (first one in the chain). Then used idx 
increments.

The Packed Virtqueue blog [1] is helpful, but some details in the examples are 
making me confused. 

Q1.
In the last step of the two-entries descriptor table example, it says both 
buffers #0 and #1 are available for the device. I understand descriptor[0] is 
available and descriptor[1] is not, but there is no ID #0 now. So does the 
device got buffer #0 by notification beforehand? If so, does it mean buffer #0 
will be lost when notifications are disabled?

Q2.
If I'm right, the buffer id of the first descriptor in chain will be overwritten 
with the last id. But in Figure: Using another descriptor chain in the last 
step of the chain example, the buffer id of the first descriptor (4th in the 
table) in second chain doesn't change to 1. Is this a typo or I'm wrong?

Thanks,
Dale Young

[1] https://www.redhat.com/en/blog/packed-virtqueue-how-reduce-overhead-virtio

Re: Intention to work on GSoC project

[Sahil]

Hi,

On Monday, April 1, 2024 11:53:11 PM IST daleyoung4242@gmail.com wrote:
> Hi,
> 
> On Monday, March 25, 2024 21:20:32 CST Sahil wrote:
> > Q1.
> > Section 2.7.4 of the virtio spec [3] states that in an available
> > descriptor, the "Element Length" stores the length of the buffer element.
> > In the next few lines, it also states that the "Element Length" is
> > reserved for used descriptors and is ignored by drivers. This sounds a
> > little contradictory given that drivers write available desciptors in the
> > descriptor ring.
> When VIRTQ_DESC_F_WRITE is set, the device will use "Element Length" to
> specify the length it writes. When VIRTQ_DESC_F_WRITE is not set, which
> means the buffer is read-only for the device, "Element Length" will not be
> changed by the device, so drivers just ignore it.


Thank you for the clarification. I think I misunderstood what I had read
in the virtio spec. What I have understood now is that "Element Length"
has different meanings for available and used descriptors.

Correct me if I am wrong - for available descriptors, it represents the
length of the buffer. For used descriptors, it represents the length of
the buffer that is written to by the device if it's write-only, otherwise it
has no meaning and hence can be ignored by drivers.

> > Q2.
> > In the Red Hat article, just below the first listing ("Memory layout of a
> > packed virtqueue descriptor"), there's the following line referring to the
> > buffer id in
> > 
> > "virtq_desc":
> > > This time, the id field is not an index for the device to look for the
> > > buffer: it is an opaque value for it, only has meaning for the driver.
> > 
> > But the device returns the buffer id when it writes the used descriptor to
> > the descriptor ring. The "only has meaning for the driver" part has got me
> > a little confused. Which buffer id is this that the device returns? Is it
> > related to the buffer id in the available descriptor?
> 
> In my understanding, buffer id is the element that avail descriptor marks to
> identify when adding descriptors to table. Device will returns the buffer
> id in the processed descriptor or the last descriptor in a chain, and write
> it to the descriptor that used idx refers to (first one in the chain). Then
> used idx increments.
> 
> The Packed Virtqueue blog [1] is helpful, but some details in the examples
> are making me confused.
> 
> Q1.
> In the last step of the two-entries descriptor table example, it says both
> buffers #0 and #1 are available for the device. I understand descriptor[0]
> is available and descriptor[1] is not, but there is no ID #0 now. So does
> the device got buffer #0 by notification beforehand? If so, does it mean
> buffer #0 will be lost when notifications are disabled?
> 

I too have a similar question and understanding the relation between buffer
ids in the used and available descriptors might give more insight into this. For
available descriptors, the buffer id is used to associate descriptors with a
particular buffer. I am still not very sure about ids in used descriptors.

Regarding Q1, both buffers #0 and #1 are available. In the mentioned figure,
both descriptor[0] and descriptor[1] are available. This figure follows the figure
with the caption "Using first buffer out of order". So in the first figure the device
reads buffer #1 and writes the used descriptor but it still has buffer #0 to read.
That still belongs to the device while buffer #1 can now be handled by the driver
once again. So in the next figure, the driver makes buffer #1 available again. The
device can still read buffer #0 from the previous batch of available descriptors.

Based on what I have understood, the driver can't touch the descriptor
corresponding to buffer #0 until the device acknowledges it. I did find the
figure a little confusing as well. I think once the meaning of buffer id is clear
from the driver's and device's perspective, it'll be easier to understand the
figure.

I am also not very sure about what happens when notifications are disabled.
I'll have to read up on that again. But I believe the driver still won't be able to
touch #0 until the device uses it.

I think going through the source should better explain these concepts.

Thanks,
Sahil

Re: Intention to work on GSoC project

[Eugenio Perez Martin]

On Tue, Apr 2, 2024 at 6:58 AM Sahil <icegambit91@gmail.com> wrote:
>
> Hi,
>
> On Monday, April 1, 2024 11:53:11 PM IST daleyoung4242@gmail.com wrote:
> > Hi,
> >
> > On Monday, March 25, 2024 21:20:32 CST Sahil wrote:
> > > Q1.
> > > Section 2.7.4 of the virtio spec [3] states that in an available
> > > descriptor, the "Element Length" stores the length of the buffer element.
> > > In the next few lines, it also states that the "Element Length" is
> > > reserved for used descriptors and is ignored by drivers. This sounds a
> > > little contradictory given that drivers write available desciptors in the
> > > descriptor ring.
> > When VIRTQ_DESC_F_WRITE is set, the device will use "Element Length" to
> > specify the length it writes. When VIRTQ_DESC_F_WRITE is not set, which
> > means the buffer is read-only for the device, "Element Length" will not be
> > changed by the device, so drivers just ignore it.
>
>
> Thank you for the clarification. I think I misunderstood what I had read
> in the virtio spec. What I have understood now is that "Element Length"
> has different meanings for available and used descriptors.
>
> Correct me if I am wrong - for available descriptors, it represents the
> length of the buffer. For used descriptors, it represents the length of
> the buffer that is written to by the device if it's write-only, otherwise it
> has no meaning and hence can be ignored by drivers.
>

Both answers are correct.

> > > Q2.
> > > In the Red Hat article, just below the first listing ("Memory layout of a
> > > packed virtqueue descriptor"), there's the following line referring to the
> > > buffer id in
> > >
> > > "virtq_desc":
> > > > This time, the id field is not an index for the device to look for the
> > > > buffer: it is an opaque value for it, only has meaning for the driver.
> > >
> > > But the device returns the buffer id when it writes the used descriptor to
> > > the descriptor ring. The "only has meaning for the driver" part has got me
> > > a little confused. Which buffer id is this that the device returns? Is it
> > > related to the buffer id in the available descriptor?
> >
> > In my understanding, buffer id is the element that avail descriptor marks to
> > identify when adding descriptors to table. Device will returns the buffer
> > id in the processed descriptor or the last descriptor in a chain, and write
> > it to the descriptor that used idx refers to (first one in the chain). Then
> > used idx increments.
> >
> > The Packed Virtqueue blog [1] is helpful, but some details in the examples
> > are making me confused.
> >
> > Q1.
> > In the last step of the two-entries descriptor table example, it says both
> > buffers #0 and #1 are available for the device. I understand descriptor[0]
> > is available and descriptor[1] is not, but there is no ID #0 now. So does
> > the device got buffer #0 by notification beforehand? If so, does it mean
> > buffer #0 will be lost when notifications are disabled?
> >
>

I guess you mean the table labeled "Figure: Full two-entries descriptor table".

Take into account that the descriptor table is not the state of all
the descriptors. That information must be maintained by the device and
the driver internally.

The descriptor table is used as a circular buffer, where one part is
writable by the driver and the other part is writable by the device.
For the device to override the descriptor table entry where descriptor
id 0 used to be does not mean that the descriptor id 0 is used. It
just means that the device communicates to the driver that descriptor
1 is used, and both sides need to keep the descriptor state
coherently.

> I too have a similar question and understanding the relation between buffer
> ids in the used and available descriptors might give more insight into this. For
> available descriptors, the buffer id is used to associate descriptors with a
> particular buffer. I am still not very sure about ids in used descriptors.
>
> Regarding Q1, both buffers #0 and #1 are available. In the mentioned figure,
> both descriptor[0] and descriptor[1] are available. This figure follows the figure
> with the caption "Using first buffer out of order". So in the first figure the device
> reads buffer #1 and writes the used descriptor but it still has buffer #0 to read.
> That still belongs to the device while buffer #1 can now be handled by the driver
> once again. So in the next figure, the driver makes buffer #1 available again. The
> device can still read buffer #0 from the previous batch of available descriptors.
>
> Based on what I have understood, the driver can't touch the descriptor
> corresponding to buffer #0 until the device acknowledges it. I did find the
> figure a little confusing as well. I think once the meaning of buffer id is clear
> from the driver's and device's perspective, it'll be easier to understand the
> figure.
>

I think you got it right. Please let me know if you have further questions.

> I am also not very sure about what happens when notifications are disabled.
> I'll have to read up on that again. But I believe the driver still won't be able to
> touch #0 until the device uses it.
>

If one side disables notification it needs to check the indexes or the
flags by its own means: Timers, read the memory in a busy loop, etc.

Sorry for the late reply!

Thanks!

> I think going through the source should better explain these concepts.
>
> Thanks,
> Sahil
>
>
>

Re: Intention to work on GSoC project

[Sahil]

Hi,

Thank you for the reply.

On Tuesday, April 2, 2024 5:08:24 PM IST Eugenio Perez Martin wrote:
> [...]
> > > > Q2.
> > > > In the Red Hat article, just below the first listing ("Memory layout of a
> > > > packed virtqueue descriptor"), there's the following line referring to the 
> > > > buffer id in "virtq_desc":
> > > > > This time, the id field is not an index for the device to look for the
> > > > > buffer: it is an opaque value for it, only has meaning for the driver.
> > > > 
> > > > But the device returns the buffer id when it writes the used descriptor to
> > > > the descriptor ring. The "only has meaning for the driver" part has got me
> > > > a little confused. Which buffer id is this that the device returns? Is it related
> > > > to the buffer id in the available descriptor?
> > > 
> > > In my understanding, buffer id is the element that avail descriptor
> > > marks to identify when adding descriptors to table. Device will returns
> > > the buffer id in the processed descriptor or the last descriptor in a
> > > chain, and write it to the descriptor that used idx refers to (first
> > > one in the chain). Then used idx increments.
> > > 
> > > The Packed Virtqueue blog [1] is helpful, but some details in the
> > > examples
> > > are making me confused.
> > > 
> > > Q1.
> > > In the last step of the two-entries descriptor table example, it says
> > > both buffers #0 and #1 are available for the device. I understand
> > > descriptor[0] is available and descriptor[1] is not, but there is no ID #0
> > > now. So does the device got buffer #0 by notification beforehand? If so,
> > > does it mean buffer #0 will be lost when notifications are disabled?
> 
> I guess you mean the table labeled "Figure: Full two-entries descriptor
> table".
> 
> Take into account that the descriptor table is not the state of all
> the descriptors. That information must be maintained by the device and
> the driver internally.
> 
> The descriptor table is used as a circular buffer, where one part is
> writable by the driver and the other part is writable by the device.
> For the device to override the descriptor table entry where descriptor
> id 0 used to be does not mean that the descriptor id 0 is used. It
> just means that the device communicates to the driver that descriptor
> 1 is used, and both sides need to keep the descriptor state
> coherently.
> 
> > I too have a similar question and understanding the relation between
> > buffer
> > ids in the used and available descriptors might give more insight into
> > this. For available descriptors, the buffer id is used to associate
> > descriptors with a particular buffer. I am still not very sure about ids
> > in used descriptors.
> > 
> > Regarding Q1, both buffers #0 and #1 are available. In the mentioned
> > figure, both descriptor[0] and descriptor[1] are available. This figure
> > follows the figure with the caption "Using first buffer out of order". So
> > in the first figure the device reads buffer #1 and writes the used
> > descriptor but it still has buffer #0 to read. That still belongs to the
> > device while buffer #1 can now be handled by the driver once again. So in
> > the next figure, the driver makes buffer #1 available again. The device
> > can still read buffer #0 from the previous batch of available
> > descriptors.
> > 
> > Based on what I have understood, the driver can't touch the descriptor
> > corresponding to buffer #0 until the device acknowledges it. I did find
> > the
> > figure a little confusing as well. I think once the meaning of buffer id
> > is clear from the driver's and device's perspective, it'll be easier to
> > understand the figure.
> 
> I think you got it right. Please let me know if you have further questions.

I would like to clarify one thing in the figure "Full two-entries descriptor
table". The driver can only overwrite a used descriptor in the descriptor
ring, right? And likewise for the device? So in the figure, the driver will
have to wait until descriptor[1] is used before it can overwrite it?

Suppose the device marks descriptor[0] as used. I think the driver will
not be able to overwrite that descriptor entry because it has to go in
order and is at descriptor[1]. Is that correct? Is it possible for the driver
to go "backwards" in the descriptor ring?

> > I am also not very sure about what happens when notifications are
> > disabled.
> > I'll have to read up on that again. But I believe the driver still won't
> > be able to touch #0 until the device uses it.
> 
> If one side disables notification it needs to check the indexes or the
> flags by its own means: Timers, read the memory in a busy loop, etc.

Understood. Thank you for the clarification.

I have some questions from the "Virtio live migration technical deep
dive" article [1].

Q1.
In the paragraph just above Figure 6, there is the following line:
> the vhost kernel thread and QEMU may run in different CPU threads,
> so these writes must be synchronized with QEMU cleaning of the dirty
> bitmap, and this write must be seen strictly after the modifications of
> the guest memory by the QEMU thread.

I am not clear on the last part of the statement. The modification of guest
memory is being done by the vhost device and not by the QEMU thread, right?

Q2.
In the first point of the "Dynamic device state: virtqueue state" section:
>The guest makes available N descriptors at the source of the migration,
>so its avail idx member in the avail idx is N.

I think there's a typo here: "...avail idx member in the avail ring is N"
instead of "...avail idx is N".

Regarding the implementation of this project, can the project be broken
down into two parts:
1. implementing packed virtqueues in QEMU, and
2. providing mechanisms for (live) migration to work with packed
    virtqueues.

I am ready to start working on the implementation. In one of your
previous emails you had talked about moving packed virtqueue
related implementation from the kernel's drivers/virtio/virtio_ring.c
into vhost_shadow_virtqueue.c.

My plan is to also understand how split virtqueue has been implemented
in QEMU. I think that'll be helpful when moving the kernel's implementation
to QEMU.

Please let me know if I should change my approach.

Thanks,
Sahil

Re: Intention to work on GSoC project

[Eugenio Perez Martin]

On Wed, Apr 3, 2024 at 4:36 PM Sahil <icegambit91@gmail.com> wrote:
>
> Hi,
>
> Thank you for the reply.
>
> On Tuesday, April 2, 2024 5:08:24 PM IST Eugenio Perez Martin wrote:
> > [...]
> > > > > Q2.
> > > > > In the Red Hat article, just below the first listing ("Memory layout of a
> > > > > packed virtqueue descriptor"), there's the following line referring to the
> > > > > buffer id in "virtq_desc":
> > > > > > This time, the id field is not an index for the device to look for the
> > > > > > buffer: it is an opaque value for it, only has meaning for the driver.
> > > > >
> > > > > But the device returns the buffer id when it writes the used descriptor to
> > > > > the descriptor ring. The "only has meaning for the driver" part has got me
> > > > > a little confused. Which buffer id is this that the device returns? Is it related
> > > > > to the buffer id in the available descriptor?
> > > >
> > > > In my understanding, buffer id is the element that avail descriptor
> > > > marks to identify when adding descriptors to table. Device will returns
> > > > the buffer id in the processed descriptor or the last descriptor in a
> > > > chain, and write it to the descriptor that used idx refers to (first
> > > > one in the chain). Then used idx increments.
> > > >
> > > > The Packed Virtqueue blog [1] is helpful, but some details in the
> > > > examples
> > > > are making me confused.
> > > >
> > > > Q1.
> > > > In the last step of the two-entries descriptor table example, it says
> > > > both buffers #0 and #1 are available for the device. I understand
> > > > descriptor[0] is available and descriptor[1] is not, but there is no ID #0
> > > > now. So does the device got buffer #0 by notification beforehand? If so,
> > > > does it mean buffer #0 will be lost when notifications are disabled?
> >
> > I guess you mean the table labeled "Figure: Full two-entries descriptor
> > table".
> >
> > Take into account that the descriptor table is not the state of all
> > the descriptors. That information must be maintained by the device and
> > the driver internally.
> >
> > The descriptor table is used as a circular buffer, where one part is
> > writable by the driver and the other part is writable by the device.
> > For the device to override the descriptor table entry where descriptor
> > id 0 used to be does not mean that the descriptor id 0 is used. It
> > just means that the device communicates to the driver that descriptor
> > 1 is used, and both sides need to keep the descriptor state
> > coherently.
> >
> > > I too have a similar question and understanding the relation between
> > > buffer
> > > ids in the used and available descriptors might give more insight into
> > > this. For available descriptors, the buffer id is used to associate
> > > descriptors with a particular buffer. I am still not very sure about ids
> > > in used descriptors.
> > >
> > > Regarding Q1, both buffers #0 and #1 are available. In the mentioned
> > > figure, both descriptor[0] and descriptor[1] are available. This figure
> > > follows the figure with the caption "Using first buffer out of order". So
> > > in the first figure the device reads buffer #1 and writes the used
> > > descriptor but it still has buffer #0 to read. That still belongs to the
> > > device while buffer #1 can now be handled by the driver once again. So in
> > > the next figure, the driver makes buffer #1 available again. The device
> > > can still read buffer #0 from the previous batch of available
> > > descriptors.
> > >
> > > Based on what I have understood, the driver can't touch the descriptor
> > > corresponding to buffer #0 until the device acknowledges it. I did find
> > > the
> > > figure a little confusing as well. I think once the meaning of buffer id
> > > is clear from the driver's and device's perspective, it'll be easier to
> > > understand the figure.
> >
> > I think you got it right. Please let me know if you have further questions.
>
> I would like to clarify one thing in the figure "Full two-entries descriptor
> table". The driver can only overwrite a used descriptor in the descriptor
> ring, right?

Except for the first round, the driver can only write to used entries
in the descriptor table. In other words, their avail and used flags
must be equal.

> And likewise for the device?

Yes, but with avail descs. I think you got this already, but I want to
be as complete as possible here.

> So in the figure, the driver will
> have to wait until descriptor[1] is used before it can overwrite it?
>

Yes, but I think it is easier to think that both descriptor id 0 and 1
are available already. The descriptor id must be less than virtqueue
size.

An entry with a valid buffer and length must be invalid because of the
descriptor id in that situation, either because it is a number > vq
length or because it is a descriptor already available.

> Suppose the device marks descriptor[0] as used. I think the driver will
> not be able to overwrite that descriptor entry because it has to go in
> order and is at descriptor[1]. Is that correct?

The device must write one descriptor as used, either 0 or 1, at
descriptors[0] as all the descriptors are available.

Now, it does not matter if the device marks as used one or the two
descriptors: the driver must write its next available descriptor at
descriptor[1]. This is not because descriptor[1] contains a special
field or data, but because the driver must write the avail descriptors
sequentially, so the device knows the address to poll or check after a
notification.

In other words, descriptor[1] is just a buffer space from the driver
to communicate an available descriptor to the device. It does not
matter what it contained before the writing, as the driver must
process that information before writing the new available descriptor.

> Is it possible for the driver
> to go "backwards" in the descriptor ring?
>

Nope, under any circumstance.

> > > I am also not very sure about what happens when notifications are
> > > disabled.
> > > I'll have to read up on that again. But I believe the driver still won't
> > > be able to touch #0 until the device uses it.
> >
> > If one side disables notification it needs to check the indexes or the
> > flags by its own means: Timers, read the memory in a busy loop, etc.
>
> Understood. Thank you for the clarification.
>
> I have some questions from the "Virtio live migration technical deep
> dive" article [1].
>
> Q1.
> In the paragraph just above Figure 6, there is the following line:
> > the vhost kernel thread and QEMU may run in different CPU threads,
> > so these writes must be synchronized with QEMU cleaning of the dirty
> > bitmap, and this write must be seen strictly after the modifications of
> > the guest memory by the QEMU thread.
>
> I am not clear on the last part of the statement. The modification of guest
> memory is being done by the vhost device and not by the QEMU thread, right?
>

QEMU also writes to the bitmap cleaning it, so it knows the memory
does not need to be resent.

Feel free to ask questions about this, but you don't need to interact
with the dirty bitmap in the project.

> Q2.
> In the first point of the "Dynamic device state: virtqueue state" section:
> >The guest makes available N descriptors at the source of the migration,
> >so its avail idx member in the avail idx is N.
>
> I think there's a typo here: "...avail idx member in the avail ring is N"
> instead of "...avail idx is N".
>

Good catch! I'll fix it, thanks!

> Regarding the implementation of this project, can the project be broken
> down into two parts:
> 1. implementing packed virtqueues in QEMU, and

Right, but let me expand on this: QEMU already supports packed
virtqueue in an emulated device (hw/virtio/virtio.c). The missing part
is the "driver" one, to be able to communicate with a vDPA device, at
hw/virtio/vhost-shadow-virtqueue.c.

> 2. providing mechanisms for (live) migration to work with packed
>     virtqueues.
>

The first part implements packed vq without live migration and then
test for live migration on top, yes. But if everything is right, this
step should require no changes from 1 actually. It is totally ok if we
need to dedicate time to this of course.

> I am ready to start working on the implementation. In one of your
> previous emails you had talked about moving packed virtqueue
> related implementation from the kernel's drivers/virtio/virtio_ring.c
> into vhost_shadow_virtqueue.c.
>

Right.

> My plan is to also understand how split virtqueue has been implemented
> in QEMU. I think that'll be helpful when moving the kernel's implementation
> to QEMU.
>

Sure, the split virtqueue is implemented in the same file
vhost_shadow_virtqueue.c. If you deploy vhost_vdpa +vdpa_sim or
vp_vdpa [1][2], you can:
* Run QEMU with -netdev type=vhost-vdpa,x-svq=on
* Set GDB breakpoint in interesting functions like
vhost_handle_guest_kick and vhost_svq_flush.

Thanks!

> Please let me know if I should change my approach.
>
> Thanks,
> Sahil
>
>

Re: Intention to work on GSoC project

[Sahil]

Hi,

On Thursday, April 4, 2024 12:07:49 AM IST Eugenio Perez Martin wrote:
> On Wed, Apr 3, 2024 at 4:36 PM Sahil <icegambit91@gmail.com> wrote:
> [...]
> > I would like to clarify one thing in the figure "Full two-entries
> > descriptor table". The driver can only overwrite a used descriptor in the
> > descriptor ring, right?
> 
> Except for the first round, the driver can only write to used entries
> in the descriptor table. In other words, their avail and used flags
> must be equal.
> 
> > And likewise for the device?
> 
> Yes, but with avail descs. I think you got this already, but I want to
> be as complete as possible here.
> 
> > So in the figure, the driver will have to wait until descriptor[1] is
> > used before it can overwrite it?
> 
> Yes, but I think it is easier to think that both descriptor id 0 and 1
> are available already. The descriptor id must be less than virtqueue
> size.
> 
> An entry with a valid buffer and length must be invalid because of the
> descriptor id in that situation, either because it is a number > vq
> length or because it is a descriptor already available.

I didn't think of it in that way. This makes sense now.

> > Suppose the device marks descriptor[0] as used. I think the driver will
> > not be able to overwrite that descriptor entry because it has to go in
> > order and is at descriptor[1]. Is that correct?
> 
> The device must write one descriptor as used, either 0 or 1, at
> descriptors[0] as all the descriptors are available.
> 
> Now, it does not matter if the device marks as used one or the two
> descriptors: the driver must write its next available descriptor at
> descriptor[1]. This is not because descriptor[1] contains a special
> field or data, but because the driver must write the avail descriptors
> sequentially, so the device knows the address to poll or check after a
> notification.
> 
> In other words, descriptor[1] is just a buffer space from the driver
> to communicate an available descriptor to the device. It does not
> matter what it contained before the writing, as the driver must
> process that information before writing the new available descriptor.
> 
> > Is it possible for the driver
> > to go "backwards" in the descriptor ring?
> 
> Nope, under any circumstance.

Understood. Thank you for the clarification.

> [...]
> > Q1.
> > In the paragraph just above Figure 6, there is the following line:
> > > the vhost kernel thread and QEMU may run in different CPU threads,
> > > so these writes must be synchronized with QEMU cleaning of the dirty
> > > bitmap, and this write must be seen strictly after the modifications of
> > > the guest memory by the QEMU thread.
> > 
> > I am not clear on the last part of the statement. The modification of
> > guest memory is being done by the vhost device and not by the QEMU
> > thread, right?
> 
> QEMU also writes to the bitmap cleaning it, so it knows the memory
> does not need to be resent.

Oh, I thought, from figure 6, the bitmap is a part of QEMU's memory but is
separate from the guest's memory.

> Feel free to ask questions about this, but you don't need to interact
> with the dirty bitmap in the project.

Understood, I won't go off on a tangent in that case.

> [...]
> > Regarding the implementation of this project, can the project be broken
> > down into two parts:
> > 1. implementing packed virtqueues in QEMU, and
> 
> Right, but let me expand on this: QEMU already supports packed
> virtqueue in an emulated device (hw/virtio/virtio.c). The missing part
> is the "driver" one, to be able to communicate with a vDPA device, at
> hw/virtio/vhost-shadow-virtqueue.c.

Got it. I'll take a look at "hw/virtio/virtio.c".

> [...]
> > My plan is to also understand how split virtqueue has been implemented
> > in QEMU. I think that'll be helpful when moving the kernel's implementation
> > to QEMU.
> 
> Sure, the split virtqueue is implemented in the same file
> vhost_shadow_virtqueue.c. If you deploy vhost_vdpa +vdpa_sim or
> vp_vdpa [1][2], you can:
> * Run QEMU with -netdev type=vhost-vdpa,x-svq=on
> * Set GDB breakpoint in interesting functions like
> vhost_handle_guest_kick and vhost_svq_flush.

I'll set up this environment as well.

Thanks,
Sahil

Re: Intention to work on GSoC project

[Sahil]

Hi,

On Friday, April 5, 2024 12:36:02 AM IST Sahil wrote:
> [...] 
> I'll set up this environment as well.

I would like to post an update here. I spent the last week 
trying to set up the environment as described in the blog [1].
I initially tried to get the L1 VM running on my host machine
(Arch Linux). However,  I was unable to use virt-sysprep or
virt-cutomize to install packages in the qcow2 image. It wasn't
able to resolve the hosts while downloading the packages.

According to the logs, /etc/resolv.conf was a dangling symlink.
I tried to use "virt-rescue" to configure DNS resolution. I tried
following these sections [2], [3] in the Arch wiki but that didn't
work either. I tried using qemu-nbd as well following this section
[4] to access the image. While I managed gain access to the
image, I wasn't able to install packages after performing a
chroot.

One workaround was to set this environment up in a VM. I
decided to set up the environment with a Fedora image in
virtualbox acting as L0. I have managed to set up an L1 VM
in this environment and I can load it using qemu-kvm.

I have one question though. One of the options (use case 1 in [1])
given to the "qemu-kvm" command is:
> -device virtio-net-pci,netdev=vhost-vdpa0,bus=pcie.0,addr=0x7\
> ,disable-modern=off,page-per-vq=on

This gives an error:
> Bus "pcie.0" not found

Does pcie refer to PCI Express? Changing this to pci.0 works.
I read through the "device buses" section in QEMU's user
documentation [5], but I have still not understood this. 

"ls /sys/bus/pci/devices/* | grep vdpa" does not give any results.
Replacing pci with pci_express doesn't give any results either. How
does one know which pci bus the vdpa device is connected to?
I have gone through the "vDPA bus drivers" section of the "vDPA
kernel framework" article [6] but I haven't managed to find an
answer yet. Am I missing something here?

There's one more thing. In "use case 1" of "Running traffic with
vhost_vdpa in Guest" [1], running "modprobe pktgen" in the L1 VM
gives an error:
> module pktgen couldn't be found in /lib/modules/6.5.6-300.fc39.x86_64.

The kernel version is 6.5.6-300.fc39.x86_64. I haven't tried building
pktgen manually in L1. I'll try that and will check if vdpa_sim works
as expected after that.

[1] https://www.redhat.com/en/blog/hands-vdpa-what-do-you-do-when-you-aint-got-hardware-part-1
[2] https://wiki.archlinux.org/title/QEMU#User-mode_networking
[3] https://wiki.archlinux.org/title/Systemd-networkd#Required_services_and_setup
[4] https://wiki.archlinux.org/title/QEMU#Mounting_a_partition_from_a_qcow2_image
[5] https://qemu-project.gitlab.io/qemu/system/device-emulation.html
[6] https://www.redhat.com/en/blog/vdpa-kernel-framework-part-1-vdpa-bus-abstracting-hardware

Thanks,
Sahil

Re: Intention to work on GSoC project

[Eugenio Perez Martin]

On Sun, Apr 14, 2024 at 8:52 PM Sahil <icegambit91@gmail.com> wrote:
>
> Hi,
>
> On Friday, April 5, 2024 12:36:02 AM IST Sahil wrote:
> > [...]
> > I'll set up this environment as well.
>
> I would like to post an update here. I spent the last week
> trying to set up the environment as described in the blog [1].
> I initially tried to get the L1 VM running on my host machine
> (Arch Linux). However,  I was unable to use virt-sysprep or
> virt-cutomize to install packages in the qcow2 image. It wasn't
> able to resolve the hosts while downloading the packages.
>
> According to the logs, /etc/resolv.conf was a dangling symlink.
> I tried to use "virt-rescue" to configure DNS resolution. I tried
> following these sections [2], [3] in the Arch wiki but that didn't
> work either. I tried using qemu-nbd as well following this section
> [4] to access the image. While I managed gain access to the
> image, I wasn't able to install packages after performing a
> chroot.
>
> One workaround was to set this environment up in a VM. I
> decided to set up the environment with a Fedora image in
> virtualbox acting as L0. I have managed to set up an L1 VM
> in this environment and I can load it using qemu-kvm.
>

I'm not clear if the complaint of the dangling pointer comes from the
host or from the guest env, but I think it is ok to continue if you've
been able to build the env.

> I have one question though. One of the options (use case 1 in [1])
> given to the "qemu-kvm" command is:
> > -device virtio-net-pci,netdev=vhost-vdpa0,bus=pcie.0,addr=0x7\
> > ,disable-modern=off,page-per-vq=on
>
> This gives an error:
> > Bus "pcie.0" not found
>
> Does pcie refer to PCI Express? Changing this to pci.0 works.

Yes, you don't need to mess with pcie stuff so this solution is
totally valid. I think we need to change that part in the tutorial.

> I read through the "device buses" section in QEMU's user
> documentation [5], but I have still not understood this.
>
> "ls /sys/bus/pci/devices/* | grep vdpa" does not give any results.
> Replacing pci with pci_express doesn't give any results either. How
> does one know which pci bus the vdpa device is connected to?
> I have gone through the "vDPA bus drivers" section of the "vDPA
> kernel framework" article [6] but I haven't managed to find an
> answer yet. Am I missing something here?
>

You cannot see the vDPA device from the guest. From the guest POV is a
regular virtio over PCI bus.

From the host, vdpa_sim is not a PCI device either, so you cannot see
under /sys/bus. Do you have a vdpa* entry under
/sys/bus/vdpa/devices/?

> There's one more thing. In "use case 1" of "Running traffic with
> vhost_vdpa in Guest" [1], running "modprobe pktgen" in the L1 VM
> gives an error:
> > module pktgen couldn't be found in /lib/modules/6.5.6-300.fc39.x86_64.
>
> The kernel version is 6.5.6-300.fc39.x86_64. I haven't tried building
> pktgen manually in L1. I'll try that and will check if vdpa_sim works
> as expected after that.
>

Did you install kernel-modules-internal?

Thanks!

> [1] https://www.redhat.com/en/blog/hands-vdpa-what-do-you-do-when-you-aint-got-hardware-part-1
> [2] https://wiki.archlinux.org/title/QEMU#User-mode_networking
> [3] https://wiki.archlinux.org/title/Systemd-networkd#Required_services_and_setup
> [4] https://wiki.archlinux.org/title/QEMU#Mounting_a_partition_from_a_qcow2_image
> [5] https://qemu-project.gitlab.io/qemu/system/device-emulation.html
> [6] https://www.redhat.com/en/blog/vdpa-kernel-framework-part-1-vdpa-bus-abstracting-hardware
>
> Thanks,
> Sahil
>
>

Re: Intention to work on GSoC project

[Sahil]

Hi,

Thank you for your reply.

On Monday, April 15, 2024 2:27:36 PM IST Eugenio Perez Martin wrote:
> [...]
> > I have one question though. One of the options (use case 1 in [1])
> > 
> > given to the "qemu-kvm" command is:
> > > -device virtio-net-pci,netdev=vhost-vdpa0,bus=pcie.0,addr=0x7\
> > > ,disable-modern=off,page-per-vq=on
> > 
> > This gives an error:
> > > Bus "pcie.0" not found
> > 
> > Does pcie refer to PCI Express? Changing this to pci.0 works.
> 
> Yes, you don't need to mess with pcie stuff so this solution is
> totally valid. I think we need to change that part in the tutorial.
> 

Understood.

> > I read through the "device buses" section in QEMU's user
> > documentation [5], but I have still not understood this.
> > 
> > "ls /sys/bus/pci/devices/* | grep vdpa" does not give any results.
> > Replacing pci with pci_express doesn't give any results either. How
> > does one know which pci bus the vdpa device is connected to?
> > I have gone through the "vDPA bus drivers" section of the "vDPA
> > kernel framework" article [6] but I haven't managed to find an
> > answer yet. Am I missing something here?
> 
> You cannot see the vDPA device from the guest. From the guest POV is a
> regular virtio over PCI bus.
>
> From the host, vdpa_sim is not a PCI device either, so you cannot see
> under /sys/bus. Do you have a vdpa* entry under
> /sys/bus/vdpa/devices/?
>

After re-reading the linked articles, I think I have got some more
clarity. One confusion was related to the difference between vdpa
and vhost-vdpa.

So far what I have understood is that L0 acts as the host and L1
acts as the guest in this setup. I understand that the guest can't
see the vDPA device.

I now also understand that vdpa_sim is not a PCI device. I am also
under the impression that vdpa refers to the vdpa bus while
vhost-vdpa is the device. Is my understanding correct?

After running the commands in the blog [1], I see that there's a
vhost-vdpa-0 device under /dev.

I also have an entry "vdpa0" under /sys/bus/vdpa/devices/ which
is a symlink to /sys/devices/vdpa0. There's a dir "vhost-vdpa-0"
under "/sys/devices/vdpa0". Hypothetically, if vhost-vdpa-0 had
been a PCI device, then it would have been present under
/sys/bus/pci/devices, right?

Another source of confusion was the pci.0 option passed to the
qemu-kvm command. But I have understood this as well now:
"-device virtio-net-pci" is a pci device.
 
> > There's one more thing. In "use case 1" of "Running traffic with
> > vhost_vdpa in Guest" [1], running "modprobe pktgen" in the L1 VM
> > 
> > gives an error:
> > > module pktgen couldn't be found in /lib/modules/6.5.6-300.fc39.x86_64.
> > 
> > The kernel version is 6.5.6-300.fc39.x86_64. I haven't tried building
> > pktgen manually in L1. I'll try that and will check if vdpa_sim works
> > as expected after that.
> 
> Did you install kernel-modules-internal?

I just realized I had the wrong version of kernel-modules-internal
installed. It works after installing the right version.

Thanks,
Sahil

[1] https://www.redhat.com/en/blog/hands-vdpa-what-do-you-do-when-you-aint-got-hardware-part-1

Re: Intention to work on GSoC project

[Eugenio Perez Martin]

On Mon, Apr 15, 2024 at 9:42 PM Sahil <icegambit91@gmail.com> wrote:
>
> Hi,
>
> Thank you for your reply.
>
> On Monday, April 15, 2024 2:27:36 PM IST Eugenio Perez Martin wrote:
> > [...]
> > > I have one question though. One of the options (use case 1 in [1])
> > >
> > > given to the "qemu-kvm" command is:
> > > > -device virtio-net-pci,netdev=vhost-vdpa0,bus=pcie.0,addr=0x7\
> > > > ,disable-modern=off,page-per-vq=on
> > >
> > > This gives an error:
> > > > Bus "pcie.0" not found
> > >
> > > Does pcie refer to PCI Express? Changing this to pci.0 works.
> >
> > Yes, you don't need to mess with pcie stuff so this solution is
> > totally valid. I think we need to change that part in the tutorial.
> >
>
> Understood.
>
> > > I read through the "device buses" section in QEMU's user
> > > documentation [5], but I have still not understood this.
> > >
> > > "ls /sys/bus/pci/devices/* | grep vdpa" does not give any results.
> > > Replacing pci with pci_express doesn't give any results either. How
> > > does one know which pci bus the vdpa device is connected to?
> > > I have gone through the "vDPA bus drivers" section of the "vDPA
> > > kernel framework" article [6] but I haven't managed to find an
> > > answer yet. Am I missing something here?
> >
> > You cannot see the vDPA device from the guest. From the guest POV is a
> > regular virtio over PCI bus.
> >
> > From the host, vdpa_sim is not a PCI device either, so you cannot see
> > under /sys/bus. Do you have a vdpa* entry under
> > /sys/bus/vdpa/devices/?
> >
>
> After re-reading the linked articles, I think I have got some more
> clarity. One confusion was related to the difference between vdpa
> and vhost-vdpa.
>
> So far what I have understood is that L0 acts as the host and L1
> acts as the guest in this setup. I understand that the guest can't
> see the vDPA device.
>
> I now also understand that vdpa_sim is not a PCI device. I am also
> under the impression that vdpa refers to the vdpa bus while
> vhost-vdpa is the device. Is my understanding correct?
>
> After running the commands in the blog [1], I see that there's a
> vhost-vdpa-0 device under /dev.
>
> I also have an entry "vdpa0" under /sys/bus/vdpa/devices/ which
> is a symlink to /sys/devices/vdpa0. There's a dir "vhost-vdpa-0"
> under "/sys/devices/vdpa0". Hypothetically, if vhost-vdpa-0 had
> been a PCI device, then it would have been present under
> /sys/bus/pci/devices, right?
>

Right. You'll check that scenario with the vp_vdpa one.

> Another source of confusion was the pci.0 option passed to the
> qemu-kvm command. But I have understood this as well now:
> "-device virtio-net-pci" is a pci device.
>
> > > There's one more thing. In "use case 1" of "Running traffic with
> > > vhost_vdpa in Guest" [1], running "modprobe pktgen" in the L1 VM
> > >
> > > gives an error:
> > > > module pktgen couldn't be found in /lib/modules/6.5.6-300.fc39.x86_64.
> > >
> > > The kernel version is 6.5.6-300.fc39.x86_64. I haven't tried building
> > > pktgen manually in L1. I'll try that and will check if vdpa_sim works
> > > as expected after that.
> >
> > Did you install kernel-modules-internal?
>
> I just realized I had the wrong version of kernel-modules-internal
> installed. It works after installing the right version.
>

Good! So you can move to vp_vdpa, or do you have more doubts about vdpa_sim?

Thanks!

Re: Intention to work on GSoC project

[Sahil]

Hi,

Thank you for your reply.

On Tuesday, April 16, 2024 2:11:16 PM IST Eugenio Perez Martin wrote:
> [...]
> > After re-reading the linked articles, I think I have got some more
> > clarity. One confusion was related to the difference between vdpa
> > and vhost-vdpa.
> > 
> > So far what I have understood is that L0 acts as the host and L1
> > acts as the guest in this setup. I understand that the guest can't
> > see the vDPA device.
> > 
> > I now also understand that vdpa_sim is not a PCI device. I am also
> > under the impression that vdpa refers to the vdpa bus while
> > vhost-vdpa is the device. Is my understanding correct?
> > 
> > After running the commands in the blog [1], I see that there's a
> > vhost-vdpa-0 device under /dev.
> > 
> > I also have an entry "vdpa0" under /sys/bus/vdpa/devices/ which
> > is a symlink to /sys/devices/vdpa0. There's a dir "vhost-vdpa-0"
> > under "/sys/devices/vdpa0". Hypothetically, if vhost-vdpa-0 had
> > been a PCI device, then it would have been present under
> > /sys/bus/pci/devices, right?
> 
> Right. You'll check that scenario with the vp_vdpa one.
> 
> > Another source of confusion was the pci.0 option passed to the
> > qemu-kvm command. But I have understood this as well now:
> > "-device virtio-net-pci" is a pci device.
> > 
> > > > There's one more thing. In "use case 1" of "Running traffic with
> > > > vhost_vdpa in Guest" [1], running "modprobe pktgen" in the L1 VM
> > > > 
> > > > gives an error:
> > > > > module pktgen couldn't be found in
> > > > > /lib/modules/6.5.6-300.fc39.x86_64.
> > > > 
> > > > The kernel version is 6.5.6-300.fc39.x86_64. I haven't tried building
> > > > pktgen manually in L1. I'll try that and will check if vdpa_sim works
> > > > as expected after that.
> > > 
> > > Did you install kernel-modules-internal?
> > 
> > I just realized I had the wrong version of kernel-modules-internal
> > installed. It works after installing the right version.
> 
> Good! So you can move to vp_vdpa, or do you have more doubts
> about vdpa_sim?

I am ready to move on to vp_vdpa. I don't have any more doubts about
vdpa_sim as of now.

Thanks,
Sahil

Re: Intention to work on GSoC project

[Sahil]

Hi,

It's been a while since I last gave an update. Sorry about that. I am ready
to get my hands dirty and start with the implementation.

I have gone through the source of linux's drivers/virtio/virtio_ring.c [1], and
QEMU's hw/virtio/virtio.c [2] and hw/virtio/vhost-shadow-virtqueue.c [3].

Before actually starting I would like to make sure I am on the right track. In
vhost-shadow-virtqueue.c, there's a function "vhost_svq_add" which in turn
calls "vhost_svq_add_split".

Shall I start by implementing a mechanism to check if the feature bit
"VIRTIO_F_RING_PACKED" is set (using "virtio_vdev_has_feature")? And
if it's supported, "vhost_svq_add" should call "vhost_svq_add_packed".
Following this, I can then start implementing "vhost_svq_add_packed"
and progress from there.

What are your thoughts on this?

Thanks,
Sahil

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/virtio/virtio.c
[2] https://gitlab.com/qemu-project/qemu/-/blob/master/hw/virtio/virtio.c
[3] https://gitlab.com/qemu-project/qemu/-/blob/master/hw/virtio/vhost-shadow-virtqueue.c

Re: Intention to work on GSoC project

[Eugenio Perez Martin]

On Mon, May 6, 2024 at 9:00 PM Sahil <icegambit91@gmail.com> wrote:
>
> Hi,
>
> It's been a while since I last gave an update. Sorry about that. I am ready
> to get my hands dirty and start with the implementation.
>

No worries!

> I have gone through the source of linux's drivers/virtio/virtio_ring.c [1], and
> QEMU's hw/virtio/virtio.c [2] and hw/virtio/vhost-shadow-virtqueue.c [3].
>
> Before actually starting I would like to make sure I am on the right track. In
> vhost-shadow-virtqueue.c, there's a function "vhost_svq_add" which in turn
> calls "vhost_svq_add_split".
>
> Shall I start by implementing a mechanism to check if the feature bit
> "VIRTIO_F_RING_PACKED" is set (using "virtio_vdev_has_feature")? And
> if it's supported, "vhost_svq_add" should call "vhost_svq_add_packed".
> Following this, I can then start implementing "vhost_svq_add_packed"
> and progress from there.
>
> What are your thoughts on this?
>

Yes, that's totally right.

I recommend you to also disable _F_EVENT_IDX to start, so the first
version is easier.

Also, you can send as many incomplete RFCs as you want. For example,
you can send a first version that only implements reading of the guest
avail ring, so we know we're aligned on that. Then, we can send
subsequents RFCs adding features on top.

Does that make sense to you?

Thanks!

> Thanks,
> Sahil
>
> [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/virtio/virtio.c
> [2] https://gitlab.com/qemu-project/qemu/-/blob/master/hw/virtio/virtio.c
> [3] https://gitlab.com/qemu-project/qemu/-/blob/master/hw/virtio/vhost-shadow-virtqueue.c
>
>

Re: Intention to work on GSoC project

[Sahil]

Hi,

On Tuesday, May 7, 2024 12:44:33 PM IST Eugenio Perez Martin wrote:
> [...]
> > Shall I start by implementing a mechanism to check if the feature bit
> > "VIRTIO_F_RING_PACKED" is set (using "virtio_vdev_has_feature")? And
> > if it's supported, "vhost_svq_add" should call "vhost_svq_add_packed".
> > Following this, I can then start implementing "vhost_svq_add_packed"
> > and progress from there.
> > 
> > What are your thoughts on this?
> 
> Yes, that's totally right.
> 
> I recommend you to also disable _F_EVENT_IDX to start, so the first
> version is easier.
> 
> Also, you can send as many incomplete RFCs as you want. For example,
> you can send a first version that only implements reading of the guest
> avail ring, so we know we're aligned on that. Then, we can send
> subsequents RFCs adding features on top.
> 
> Does that make sense to you?
> 
> Thanks!
>

Thank you for your reply. This is perfect. I'll start with this and will
send the first RFC as soon as possible.

Thanks,
Sahil

Re: Intention to work on GSoC project

[Sahil]

Hi,

On Wednesday, May 8, 2024 8:53:12 AM GMT+5:30 Sahil wrote:
> Hi,
> 
> On Tuesday, May 7, 2024 12:44:33 PM IST Eugenio Perez Martin wrote:
> > [...]
> > 
> > > Shall I start by implementing a mechanism to check if the feature bit
> > > "VIRTIO_F_RING_PACKED" is set (using "virtio_vdev_has_feature")? And
> > > if it's supported, "vhost_svq_add" should call "vhost_svq_add_packed".
> > > Following this, I can then start implementing "vhost_svq_add_packed"
> > > and progress from there.
> > > 
> > > What are your thoughts on this?
> > 
> > Yes, that's totally right.
> > 
> > I recommend you to also disable _F_EVENT_IDX to start, so the first
> > version is easier.
> > 
> > Also, you can send as many incomplete RFCs as you want. For example,
> > you can send a first version that only implements reading of the guest
> > avail ring, so we know we're aligned on that. Then, we can send
> > subsequents RFCs adding features on top.
>

I have started working on implementing packed virtqueue support in
vhost-shadow-virtqueue.c. The changes I have made so far are very
minimal. I have one confusion as well.

In "vhost_svq_add()" [1], a structure of type "VhostShadowVirtqueue"
is being used. My initial idea was to create a whole new structure (eg:
VhostShadowVirtqueuePacked). But I realized that "VhostShadowVirtqueue"
is being used in a lot of other places such as in "struct vhost_vdpa" [2]
(in "vhost-vdpa.h"). So maybe this isn't a good idea.

The problem is that "VhostShadowVirtqueue" has a member of type "struct
vring" [3] which represents a split virtqueue [4]. My idea now is to instead
wrap this member in a union so that the struct would look something like
this.

struct VhostShadowVirtqueue {
        union {
                struct vring vring;
                struct packed_vring vring;
        }
        ...
}

I am not entirely sure if this is a good idea. It is similar to what's been done
in linux's "drivers/virtio/virtio_ring.c" ("struct vring_virtqueue" [5]).

I thought I would ask this first before continuing further.

Thanks,
Sahil

[1] https://gitlab.com/qemu-project/qemu/-/blob/master/hw/virtio/vhost-shadow-virtqueue.c#L249
[2] https://gitlab.com/qemu-project/qemu/-/blob/master/include/hw/virtio/vhost-vdpa.h#L69
[3] https://gitlab.com/qemu-project/qemu/-/blob/master/hw/virtio/vhost-shadow-virtqueue.h#L52
[4] https://gitlab.com/qemu-project/qemu/-/blob/master/include/standard-headers/linux/virtio_ring.h#L156
[5] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/virtio/virtio_ring.c#n199

Re: Intention to work on GSoC project

[Eugenio Perez Martin]

On Mon, May 13, 2024 at 3:49 PM Sahil <icegambit91@gmail.com> wrote:
>
> Hi,
>
> On Wednesday, May 8, 2024 8:53:12 AM GMT+5:30 Sahil wrote:
> > Hi,
> >
> > On Tuesday, May 7, 2024 12:44:33 PM IST Eugenio Perez Martin wrote:
> > > [...]
> > >
> > > > Shall I start by implementing a mechanism to check if the feature bit
> > > > "VIRTIO_F_RING_PACKED" is set (using "virtio_vdev_has_feature")? And
> > > > if it's supported, "vhost_svq_add" should call "vhost_svq_add_packed".
> > > > Following this, I can then start implementing "vhost_svq_add_packed"
> > > > and progress from there.
> > > >
> > > > What are your thoughts on this?
> > >
> > > Yes, that's totally right.
> > >
> > > I recommend you to also disable _F_EVENT_IDX to start, so the first
> > > version is easier.
> > >
> > > Also, you can send as many incomplete RFCs as you want. For example,
> > > you can send a first version that only implements reading of the guest
> > > avail ring, so we know we're aligned on that. Then, we can send
> > > subsequents RFCs adding features on top.
> >
>
> I have started working on implementing packed virtqueue support in
> vhost-shadow-virtqueue.c. The changes I have made so far are very
> minimal. I have one confusion as well.
>
> In "vhost_svq_add()" [1], a structure of type "VhostShadowVirtqueue"
> is being used. My initial idea was to create a whole new structure (eg:
> VhostShadowVirtqueuePacked). But I realized that "VhostShadowVirtqueue"
> is being used in a lot of other places such as in "struct vhost_vdpa" [2]
> (in "vhost-vdpa.h"). So maybe this isn't a good idea.
>
> The problem is that "VhostShadowVirtqueue" has a member of type "struct
> vring" [3] which represents a split virtqueue [4]. My idea now is to instead
> wrap this member in a union so that the struct would look something like
> this.
>
> struct VhostShadowVirtqueue {
>         union {
>                 struct vring vring;
>                 struct packed_vring vring;
>         }
>         ...
> }
>
> I am not entirely sure if this is a good idea. It is similar to what's been done
> in linux's "drivers/virtio/virtio_ring.c" ("struct vring_virtqueue" [5]).
>
> I thought I would ask this first before continuing further.
>

That's right, this second option makes perfect sense.

VhostShadowVirtqueue should abstract both split and packed. You'll see
that some members are reused, while others are only used in one
version so they are placed after a union. They should follow the same
pattern, although it is not a problem if we need to divert a little
bit from the kernel's code.

Thanks!

> Thanks,
> Sahil
>
> [1] https://gitlab.com/qemu-project/qemu/-/blob/master/hw/virtio/vhost-shadow-virtqueue.c#L249
> [2] https://gitlab.com/qemu-project/qemu/-/blob/master/include/hw/virtio/vhost-vdpa.h#L69
> [3] https://gitlab.com/qemu-project/qemu/-/blob/master/hw/virtio/vhost-shadow-virtqueue.h#L52
> [4] https://gitlab.com/qemu-project/qemu/-/blob/master/include/standard-headers/linux/virtio_ring.h#L156
> [5] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/virtio/virtio_ring.c#n199
>
>

Re: Intention to work on GSoC project

[Sahil]

Hi,

On Monday, May 13, 2024 7:53:40 PM GMT+5:30 Eugenio Perez Martin wrote:
> [...]
> > I have started working on implementing packed virtqueue support in
> > vhost-shadow-virtqueue.c. The changes I have made so far are very
> > minimal. I have one confusion as well.
> > 
> > In "vhost_svq_add()" [1], a structure of type "VhostShadowVirtqueue"
> > is being used. My initial idea was to create a whole new structure (eg:
> > VhostShadowVirtqueuePacked). But I realized that "VhostShadowVirtqueue"
> > is being used in a lot of other places such as in "struct vhost_vdpa" [2]
> > (in "vhost-vdpa.h"). So maybe this isn't a good idea.
> > 
> > The problem is that "VhostShadowVirtqueue" has a member of type "struct
> > vring" [3] which represents a split virtqueue [4]. My idea now is to
> > instead wrap this member in a union so that the struct would look
> > something like this.
> > 
> > struct VhostShadowVirtqueue {
> >         union {
> >                 struct vring vring;
> >                 struct packed_vring vring;
> >         }
> >         ...
> > }
> > 
> > I am not entirely sure if this is a good idea. It is similar to what's
> > been done in linux's "drivers/virtio/virtio_ring.c" ("struct
> > vring_virtqueue" [5]).
> > 
> > I thought I would ask this first before continuing further.
> 
> That's right, this second option makes perfect sense.
> 
> VhostShadowVirtqueue should abstract both split and packed. You'll see
> that some members are reused, while others are only used in one
> version so they are placed after a union. They should follow the same
> pattern, although it is not a problem if we need to divert a little
> bit from the kernel's code.
> 

Understood, thank you for the reply.

Thanks,
Sahil

Re: Intention to work on GSoC project

[Eugenio Perez Martin]

On Sat, Mar 16, 2024 at 9:27 PM Sahil <icegambit91@gmail.com> wrote:
>
> Hi,
>
> Thank you for your reply.
>
> On Friday, March 15, 2024 4:57:39 PM IST Eugenio Perez Martin wrote:
> > [...]
> > > Some sections in the above docs were difficult to grasp. For the time
> > > being, I have focused on those parts that I thought were relevant
> > > to the project.
> >
> > Please feel free to ask any questions, maybe we can improve the doc :).
>
> I understood the introductory sections of the documentation such as the
> "About QEMU" section and the first half of the "system emulation". Sections
> and subsections that went into greater detail were a little overwhelming
> such as the "QEMU virtio-net standby" subsection [1] or the "migration
> features" [2] subsection. But the red hat blogs and deep-dive articles helped
> cover a lot of ground conceptually.
>
> I feel once I start getting my hands dirty, I'll be able to absorb these concepts
> much better.
>
> I did have two questions that I would like to ask.
>
> Q1.
> Regarding the "Deep dive into Virtio-networking and vhost-net" article [3],
> the "Introduction" subsection of the "Vhost protocol" section mentions that
> sending the available buffer notification involves a vCPU interrupt (4th bullet
> point).

Now I realize we used a very misleading term there :). Without
ioeventfd, when the guest writes to the PCI notification area the
guest vCPU is totally paused there, and the control is handed to
host's KVM first and QEMU after it. The same physical CPU of the
machine needs to switch context because of that.

Is an interruption of the execution and a context switch. Maybe
"paused" is a better term.

> But in figure 2, the arrow for the "available buffer notification" indicates
> a PCI interrupt. Initially I thought they were two different interrupts but I am
> a little confused about this now.
>

They are different, but at that part of the blog is just the direction
of who interrupts / notifies who :).

> Q2.
> In the "Virtio-net failover operation" section of the "Virtio-net failover: An
> introduction" article [4], there are five bullet points under the first figure.
> The second point states that the guest kernel needs the ability to switch
> between the VFIO device and the vfio-net device. I was wondering if
> "vfio-net" is a typo and if it should be "virtio-net" instead.
>

Good catch :). CCing Laurent, the author of the blog, in case he can
modify the text.

> > [...]
> > There is a post before the first in the series:
> > https://www.redhat.com/en/blog/virtio-devices-and-drivers-overview-headjack-
> > and-phone
>
> Got it. I didn't know this was the first in the series. I have now covered this as
> well, so I can move on to "Virtqueues and virtio ring: How the data travels" [3] :)
>
> > > 1. Virtqueues and virtio ring: How the data travels [8]
> > > 2. Packed virtqueue: How to reduce overhead with virtio [9]
> > > 3. Virtio live migration technical deep dive [10]
> > > 4. Hands on vDPA: what do you do when you ain't got the hardware v2 (Part
> > > 1) [11]
> > I think it's a good plan!
> >
> > If you feel like you're reading a lot of theory and want to get your
> > hands dirty already, you can also start messing with the code with the
> > blogs you already read. Or, maybe, after reading the Packed virtqueue
> > one, your call.
> >
> > In a very brute-forced description, you can start trying to copy all
> > the *packed* stuff of kernel's drivers/virtio/virtio_ring.c into
> > vhost_shadow_virtqueue.c.
>
> I would love to start with some hands-on tasks. I'll take a look at
> the kernel's "drivers/virtio/virtio_ring.c". I think I should also start
> going through the "vhost_shadow_virtqueue.c" [4] source code.
>
> > There is a lot more in the task, and I can get into more detail
> > if you want either here or in a meeting.
>
> Thank you. Either means of communication works for me although
> the latter will require some coordination.
>
> > If you prefer to continue with the theory it is ok too.
>
> A good balance of theory and practice would be nice at this stage.
> It'll keep my brains from getting too muddled up.
>
> Thanks,
> Sahil
>
> [1] https://www.qemu.org/docs/master/system/virtio-net-failover.html
> [2] https://www.qemu.org/docs/master/devel/migration/features.html
> [3] https://www.redhat.com/en/blog/deep-dive-virtio-networking-and-vhost-net
> [4] https://www.redhat.com/en/blog/virtio-net-failover-introduction
> [5] https://www.redhat.com/en/blog/virtqueues-and-virtio-ring-how-data-travels
>
>