Building QEMU as a Shared Library

Building QEMU as a Shared Library

[Saanjh Sengupta]

[-- Attachment #1: Type: text/plain, Size: 329 bytes --]

Hi,

I referred to your mailing chains on suggesting QEMU to be built as a shared library.

Change meson.build to build QEMU as a shared library (with PIC enabled for static libraries)

Could you please suggest what exactly has to be enabled in the meson.build?

I am confused on that front.

Regards
Saanjh Sengupta

[-- Attachment #2: Type: text/html, Size: 843 bytes --]

Re: Building QEMU as a Shared Library

[Philippe Mathieu-Daudé]

Cc'ing our meson experts

On 22/2/25 14:36, Saanjh Sengupta wrote:
> Hi,
> 
> I referred to your mailing chains on suggesting QEMU to be built as a 
> shared library.
> 
> *Change meson.build to build QEMU as a shared library (with PIC enabled 
> for static libraries)*
> *
> *
> Could you please suggest what exactly has to be enabled in the meson.build?
> 
> I am confused on that front.
> 
> Regards
> Saanjh Sengupta

Re: Building QEMU as a Shared Library

[Pierrick Bouvier]

Hi Saanjh,

here is a minimal patch that builds one shared library per target (arch, 
mode) where arch is cpu arch, and mode is system or user, and launch 
system-aarch64 through a simple driver:

https://github.com/pbo-linaro/qemu/commit/fbb39cc64f77d4bf1e5e50795c75b62735bf5c5f

With this, it could be possible to create a driver that can execute any 
existing target. It's a sort of single binary for QEMU, but shared 
objects are mandatory, and duplicates all the QEMU state. So there is no 
real benefit compared to having different processes.

In more, to be able to do concurrent emulations, there are much more 
problems to be solved. QEMU state is correctly kept per target, but all 
other libraries states are shared. There are various issues if you 
launch two emulations at the same time in two threads:
- glib global context
- qemu calls exit in many places, which stops the whole process
- probably other things I didn't explore

At this point, even though qemu targets can be built as shared objects, 
I would recommend to use different processes, and implement some form on 
IPC to synchronize all this.
Another possibility is to try to build machines without using the 
existing main, but I'm not sure it's worth all the hassle.

What are you trying to achieve?

Regards,
Pierrick

On 2/24/25 01:10, Philippe Mathieu-Daudé wrote:
> Cc'ing our meson experts
> 
> On 22/2/25 14:36, Saanjh Sengupta wrote:
>> Hi,
>>
>> I referred to your mailing chains on suggesting QEMU to be built as a
>> shared library.
>>
>> *Change meson.build to build QEMU as a shared library (with PIC enabled
>> for static libraries)*
>> *
>> *
>> Could you please suggest what exactly has to be enabled in the meson.build?
>>
>> I am confused on that front.
>>
>> Regards
>> Saanjh Sengupta
> 

Re: Building QEMU as a Shared Library

[Saanjh Sengupta]

[-- Attachment #1: Type: text/plain, Size: 2790 bytes --]



Hi,

Thank you so much for your inputs. I was able to create the .so file of QEMU.

Actually, what we are trying is to understand and explore possibilities of Virtual Time Control in QEMU. In short, what I mean to say is an approach via which I can tell QEMU to emulate for XYZ time when the I give a trigger and then pause the emulation by itself after the XYZ time is completed.

On that front itself, do you have any inputs/ideas regarding the same?


Regards
Saanjh Sengupta
________________________________
From: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Sent: Tuesday, February 25, 2025 6:29:44 AM
To: Philippe Mathieu-Daudé <philmd@linaro.org>; Paolo Bonzini <pbonzini@redhat.com>; Marc-André Lureau <marcandre.lureau@redhat.com>
Cc: amir.gonnen@neuroblade.ai <amir.gonnen@neuroblade.ai>; qemu-devel@nongnu.org <qemu-devel@nongnu.org>; Saanjh Sengupta <saanjhsengupta@outlook.com>
Subject: Re: Building QEMU as a Shared Library

Hi Saanjh,

here is a minimal patch that builds one shared library per target (arch,
mode) where arch is cpu arch, and mode is system or user, and launch
system-aarch64 through a simple driver:

https://github.com/pbo-linaro/qemu/commit/fbb39cc64f77d4bf1e5e50795c75b62735bf5c5f

With this, it could be possible to create a driver that can execute any
existing target. It's a sort of single binary for QEMU, but shared
objects are mandatory, and duplicates all the QEMU state. So there is no
real benefit compared to having different processes.

In more, to be able to do concurrent emulations, there are much more
problems to be solved. QEMU state is correctly kept per target, but all
other libraries states are shared. There are various issues if you
launch two emulations at the same time in two threads:
- glib global context
- qemu calls exit in many places, which stops the whole process
- probably other things I didn't explore

At this point, even though qemu targets can be built as shared objects,
I would recommend to use different processes, and implement some form on
IPC to synchronize all this.
Another possibility is to try to build machines without using the
existing main, but I'm not sure it's worth all the hassle.

What are you trying to achieve?

Regards,
Pierrick

On 2/24/25 01:10, Philippe Mathieu-Daudé wrote:
> Cc'ing our meson experts
>
> On 22/2/25 14:36, Saanjh Sengupta wrote:
>> Hi,
>>
>> I referred to your mailing chains on suggesting QEMU to be built as a
>> shared library.
>>
>> *Change meson.build to build QEMU as a shared library (with PIC enabled
>> for static libraries)*
>> *
>> *
>> Could you please suggest what exactly has to be enabled in the meson.build?
>>
>> I am confused on that front.
>>
>> Regards
>> Saanjh Sengupta
>


[-- Attachment #2: Type: text/html, Size: 4112 bytes --]

Re: Building QEMU as a Shared Library

[Pierrick Bouvier]

Hi Saanjh,

depending what you are trying to achieve exactly, plugins can provide a 
solution. It's convenient and you can stay on top of QEMU upstream, 
without having to create a downstream fork.

We already have plugins for stopping after a given number of 
instructions, or slow down execution of a VM:

# stop after executing 1'000'000 instructions:
$ ./build/qemu-system-x86_64 -plugin 
./build/contrib/plugins/libstoptrigger,icount=1000000 -d plugin

# execute no more than 1'000'000 instructions per second:
$ ./build/qemu-system-x86_64 -plugin 
./build/contrib/plugins/libips.so,ips=1000000 -d plugin

You can see source code associated (./contrib/plugins/stoptrigger.c and 
./contrib/plugins/ips.c), to implement something similar to what you 
want, but based on time.
Would that satisfy your need?

Regards,
Pierrick

On 3/3/25 21:53, Saanjh Sengupta wrote:
> 
> 
> Hi,
> 
> Thank you so much for your inputs. I was able to create the .so file of 
> QEMU.
> 
> Actually, what we are trying is to understand and explore possibilities 
> of Virtual Time Control in QEMU. In short, what I mean to say is an 
> approach via which I can tell QEMU to emulate for XYZ time when the I 
> give a trigger and then pause the emulation by itself after the XYZ time 
> is completed.
> 
> On that front itself, do you have any inputs/ideas regarding the same?
> 
> 
> Regards
> Saanjh Sengupta
> ------------------------------------------------------------------------
> *From:* Pierrick Bouvier <pierrick.bouvier@linaro.org>
> *Sent:* Tuesday, February 25, 2025 6:29:44 AM
> *To:* Philippe Mathieu-Daudé <philmd@linaro.org>; Paolo Bonzini 
> <pbonzini@redhat.com>; Marc-André Lureau <marcandre.lureau@redhat.com>
> *Cc:* amir.gonnen@neuroblade.ai <amir.gonnen@neuroblade.ai>; qemu- 
> devel@nongnu.org <qemu-devel@nongnu.org>; Saanjh Sengupta 
> <saanjhsengupta@outlook.com>
> *Subject:* Re: Building QEMU as a Shared Library
> Hi Saanjh,
> 
> here is a minimal patch that builds one shared library per target (arch,
> mode) where arch is cpu arch, and mode is system or user, and launch
> system-aarch64 through a simple driver:
> 
> https://github.com/pbo-linaro/qemu/commit/ 
> fbb39cc64f77d4bf1e5e50795c75b62735bf5c5f <https://github.com/pbo-linaro/ 
> qemu/commit/fbb39cc64f77d4bf1e5e50795c75b62735bf5c5f>
> 
> With this, it could be possible to create a driver that can execute any
> existing target. It's a sort of single binary for QEMU, but shared
> objects are mandatory, and duplicates all the QEMU state. So there is no
> real benefit compared to having different processes.
> 
> In more, to be able to do concurrent emulations, there are much more
> problems to be solved. QEMU state is correctly kept per target, but all
> other libraries states are shared. There are various issues if you
> launch two emulations at the same time in two threads:
> - glib global context
> - qemu calls exit in many places, which stops the whole process
> - probably other things I didn't explore
> 
> At this point, even though qemu targets can be built as shared objects,
> I would recommend to use different processes, and implement some form on
> IPC to synchronize all this.
> Another possibility is to try to build machines without using the
> existing main, but I'm not sure it's worth all the hassle.
> 
> What are you trying to achieve?
> 
> Regards,
> Pierrick
> 
> On 2/24/25 01:10, Philippe Mathieu-Daudé wrote:
>> Cc'ing our meson experts
>> 
>> On 22/2/25 14:36, Saanjh Sengupta wrote:
>>> Hi,
>>>
>>> I referred to your mailing chains on suggesting QEMU to be built as a
>>> shared library.
>>>
>>> *Change meson.build to build QEMU as a shared library (with PIC enabled
>>> for static libraries)*
>>> *
>>> *
>>> Could you please suggest what exactly has to be enabled in the meson.build?
>>>
>>> I am confused on that front.
>>>
>>> Regards
>>> Saanjh Sengupta
>> 
> 

Re: Building QEMU as a Shared Library

[Saanjh Sengupta]

[-- Attachment #1: Type: text/plain, Size: 4621 bytes --]

Hi,

Thank you for your inputs. Let me check with the information you provided. Will let you know if I am stuck with something.


Regards
Saanjh Sengupta
________________________________
From: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Sent: Wednesday, March 5, 2025 5:20:38 AM
To: Saanjh Sengupta <saanjhsengupta@outlook.com>; Philippe Mathieu-Daudé <philmd@linaro.org>; Paolo Bonzini <pbonzini@redhat.com>; Marc-André Lureau <marcandre.lureau@redhat.com>
Cc: amir.gonnen@neuroblade.ai <amir.gonnen@neuroblade.ai>; qemu-devel@nongnu.org <qemu-devel@nongnu.org>; Alex Bennée <alex.bennee@linaro.org>
Subject: Re: Building QEMU as a Shared Library

Hi Saanjh,

depending what you are trying to achieve exactly, plugins can provide a
solution. It's convenient and you can stay on top of QEMU upstream,
without having to create a downstream fork.

We already have plugins for stopping after a given number of
instructions, or slow down execution of a VM:

# stop after executing 1'000'000 instructions:
$ ./build/qemu-system-x86_64 -plugin
./build/contrib/plugins/libstoptrigger,icount=1000000 -d plugin

# execute no more than 1'000'000 instructions per second:
$ ./build/qemu-system-x86_64 -plugin
./build/contrib/plugins/libips.so,ips=1000000 -d plugin

You can see source code associated (./contrib/plugins/stoptrigger.c and
./contrib/plugins/ips.c), to implement something similar to what you
want, but based on time.
Would that satisfy your need?

Regards,
Pierrick

On 3/3/25 21:53, Saanjh Sengupta wrote:
>
>
> Hi,
>
> Thank you so much for your inputs. I was able to create the .so file of
> QEMU.
>
> Actually, what we are trying is to understand and explore possibilities
> of Virtual Time Control in QEMU. In short, what I mean to say is an
> approach via which I can tell QEMU to emulate for XYZ time when the I
> give a trigger and then pause the emulation by itself after the XYZ time
> is completed.
>
> On that front itself, do you have any inputs/ideas regarding the same?
>
>
> Regards
> Saanjh Sengupta
> ------------------------------------------------------------------------
> *From:* Pierrick Bouvier <pierrick.bouvier@linaro.org>
> *Sent:* Tuesday, February 25, 2025 6:29:44 AM
> *To:* Philippe Mathieu-Daudé <philmd@linaro.org>; Paolo Bonzini
> <pbonzini@redhat.com>; Marc-André Lureau <marcandre.lureau@redhat.com>
> *Cc:* amir.gonnen@neuroblade.ai <amir.gonnen@neuroblade.ai>; qemu-
> devel@nongnu.org <qemu-devel@nongnu.org>; Saanjh Sengupta
> <saanjhsengupta@outlook.com>
> *Subject:* Re: Building QEMU as a Shared Library
> Hi Saanjh,
>
> here is a minimal patch that builds one shared library per target (arch,
> mode) where arch is cpu arch, and mode is system or user, and launch
> system-aarch64 through a simple driver:
>
> https://github.com/pbo-linaro/qemu/commit/
> fbb39cc64f77d4bf1e5e50795c75b62735bf5c5f <https://github.com/pbo-linaro/
> qemu/commit/fbb39cc64f77d4bf1e5e50795c75b62735bf5c5f>
>
> With this, it could be possible to create a driver that can execute any
> existing target. It's a sort of single binary for QEMU, but shared
> objects are mandatory, and duplicates all the QEMU state. So there is no
> real benefit compared to having different processes.
>
> In more, to be able to do concurrent emulations, there are much more
> problems to be solved. QEMU state is correctly kept per target, but all
> other libraries states are shared. There are various issues if you
> launch two emulations at the same time in two threads:
> - glib global context
> - qemu calls exit in many places, which stops the whole process
> - probably other things I didn't explore
>
> At this point, even though qemu targets can be built as shared objects,
> I would recommend to use different processes, and implement some form on
> IPC to synchronize all this.
> Another possibility is to try to build machines without using the
> existing main, but I'm not sure it's worth all the hassle.
>
> What are you trying to achieve?
>
> Regards,
> Pierrick
>
> On 2/24/25 01:10, Philippe Mathieu-Daudé wrote:
>> Cc'ing our meson experts
>>
>> On 22/2/25 14:36, Saanjh Sengupta wrote:
>>> Hi,
>>>
>>> I referred to your mailing chains on suggesting QEMU to be built as a
>>> shared library.
>>>
>>> *Change meson.build to build QEMU as a shared library (with PIC enabled
>>> for static libraries)*
>>> *
>>> *
>>> Could you please suggest what exactly has to be enabled in the meson.build?
>>>
>>> I am confused on that front.
>>>
>>> Regards
>>> Saanjh Sengupta
>>
>


[-- Attachment #2: Type: text/html, Size: 6391 bytes --]

Re: Building QEMU as a Shared Library

[Saanjh Sengupta]

[-- Attachment #1: Type: text/plain, Size: 5337 bytes --]

Hi,

I have a couple of questions:

  1.
When I use the libstoptrigger.so: in that case the QEMU 's emulation stops after executing the defined number of instructions. Post this, the whole QEMU terminates. And while using the libips.so I am assuming that the QEMU doesn't execute no more than the defined instructions. Please correct me if I am wrong.
  2.
In my case, I want the QEMU to start emulation for some time and PAUSE it's emulation for some time; after it is Paused (it's virtual time is also to be paused) and then let's say for after 'x' time period it should resume it's virtual time.

[image]

I have added this segment inside the update_system_time function inside the ipsPlugin.c. but once the instructions reach to the defined limit the virtual time does not seem to stop.
Do you have any suggestions on that front?


Regards
Saanjh Sengupta
________________________________
From: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Sent: Wednesday, March 5, 2025 5:20:38 AM
To: Saanjh Sengupta <saanjhsengupta@outlook.com>; Philippe Mathieu-Daudé <philmd@linaro.org>; Paolo Bonzini <pbonzini@redhat.com>; Marc-André Lureau <marcandre.lureau@redhat.com>
Cc: amir.gonnen@neuroblade.ai <amir.gonnen@neuroblade.ai>; qemu-devel@nongnu.org <qemu-devel@nongnu.org>; Alex Bennée <alex.bennee@linaro.org>
Subject: Re: Building QEMU as a Shared Library

Hi Saanjh,

depending what you are trying to achieve exactly, plugins can provide a
solution. It's convenient and you can stay on top of QEMU upstream,
without having to create a downstream fork.

We already have plugins for stopping after a given number of
instructions, or slow down execution of a VM:

# stop after executing 1'000'000 instructions:
$ ./build/qemu-system-x86_64 -plugin
./build/contrib/plugins/libstoptrigger,icount=1000000 -d plugin

# execute no more than 1'000'000 instructions per second:
$ ./build/qemu-system-x86_64 -plugin
./build/contrib/plugins/libips.so,ips=1000000 -d plugin

You can see source code associated (./contrib/plugins/stoptrigger.c and
./contrib/plugins/ips.c), to implement something similar to what you
want, but based on time.
Would that satisfy your need?

Regards,
Pierrick

On 3/3/25 21:53, Saanjh Sengupta wrote:
>
>
> Hi,
>
> Thank you so much for your inputs. I was able to create the .so file of
> QEMU.
>
> Actually, what we are trying is to understand and explore possibilities
> of Virtual Time Control in QEMU. In short, what I mean to say is an
> approach via which I can tell QEMU to emulate for XYZ time when the I
> give a trigger and then pause the emulation by itself after the XYZ time
> is completed.
>
> On that front itself, do you have any inputs/ideas regarding the same?
>
>
> Regards
> Saanjh Sengupta
> ------------------------------------------------------------------------
> *From:* Pierrick Bouvier <pierrick.bouvier@linaro.org>
> *Sent:* Tuesday, February 25, 2025 6:29:44 AM
> *To:* Philippe Mathieu-Daudé <philmd@linaro.org>; Paolo Bonzini
> <pbonzini@redhat.com>; Marc-André Lureau <marcandre.lureau@redhat.com>
> *Cc:* amir.gonnen@neuroblade.ai <amir.gonnen@neuroblade.ai>; qemu-
> devel@nongnu.org <qemu-devel@nongnu.org>; Saanjh Sengupta
> <saanjhsengupta@outlook.com>
> *Subject:* Re: Building QEMU as a Shared Library
> Hi Saanjh,
>
> here is a minimal patch that builds one shared library per target (arch,
> mode) where arch is cpu arch, and mode is system or user, and launch
> system-aarch64 through a simple driver:
>
> https://github.com/pbo-linaro/qemu/commit/
> fbb39cc64f77d4bf1e5e50795c75b62735bf5c5f <https://github.com/pbo-linaro/
> qemu/commit/fbb39cc64f77d4bf1e5e50795c75b62735bf5c5f>
>
> With this, it could be possible to create a driver that can execute any
> existing target. It's a sort of single binary for QEMU, but shared
> objects are mandatory, and duplicates all the QEMU state. So there is no
> real benefit compared to having different processes.
>
> In more, to be able to do concurrent emulations, there are much more
> problems to be solved. QEMU state is correctly kept per target, but all
> other libraries states are shared. There are various issues if you
> launch two emulations at the same time in two threads:
> - glib global context
> - qemu calls exit in many places, which stops the whole process
> - probably other things I didn't explore
>
> At this point, even though qemu targets can be built as shared objects,
> I would recommend to use different processes, and implement some form on
> IPC to synchronize all this.
> Another possibility is to try to build machines without using the
> existing main, but I'm not sure it's worth all the hassle.
>
> What are you trying to achieve?
>
> Regards,
> Pierrick
>
> On 2/24/25 01:10, Philippe Mathieu-Daudé wrote:
>> Cc'ing our meson experts
>>
>> On 22/2/25 14:36, Saanjh Sengupta wrote:
>>> Hi,
>>>
>>> I referred to your mailing chains on suggesting QEMU to be built as a
>>> shared library.
>>>
>>> *Change meson.build to build QEMU as a shared library (with PIC enabled
>>> for static libraries)*
>>> *
>>> *
>>> Could you please suggest what exactly has to be enabled in the meson.build?
>>>
>>> I am confused on that front.
>>>
>>> Regards
>>> Saanjh Sengupta
>>
>


[-- Attachment #2: Type: text/html, Size: 7776 bytes --]

Re: Building QEMU as a Shared Library

[Saanjh Sengupta]

[-- Attachment #1.1: Type: text/plain, Size: 5914 bytes --]



Hi,

You can find the image attached.

Regards
Saanjh Sengupta
________________________________
From: Saanjh Sengupta <saanjhsengupta@outlook.com>
Sent: Tuesday, March 11, 2025 3:20:48 PM
To: Pierrick Bouvier <pierrick.bouvier@linaro.org>; Philippe Mathieu-Daudé <philmd@linaro.org>; Paolo Bonzini <pbonzini@redhat.com>; Marc-André Lureau <marcandre.lureau@redhat.com>
Cc: amir.gonnen@neuroblade.ai <amir.gonnen@neuroblade.ai>; qemu-devel@nongnu.org <qemu-devel@nongnu.org>; Alex Bennée <alex.bennee@linaro.org>
Subject: Re: Building QEMU as a Shared Library

Hi,

I have a couple of questions:

  1.
When I use the libstoptrigger.so: in that case the QEMU 's emulation stops after executing the defined number of instructions. Post this, the whole QEMU terminates. And while using the libips.so I am assuming that the QEMU doesn't execute no more than the defined instructions. Please correct me if I am wrong.
  2.
In my case, I want the QEMU to start emulation for some time and PAUSE it's emulation for some time; after it is Paused (it's virtual time is also to be paused) and then let's say for after 'x' time period it should resume it's virtual time.

[image]

I have added this segment inside the update_system_time function inside the ipsPlugin.c. but once the instructions reach to the defined limit the virtual time does not seem to stop.
Do you have any suggestions on that front?


Regards
Saanjh Sengupta
________________________________
From: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Sent: Wednesday, March 5, 2025 5:20:38 AM
To: Saanjh Sengupta <saanjhsengupta@outlook.com>; Philippe Mathieu-Daudé <philmd@linaro.org>; Paolo Bonzini <pbonzini@redhat.com>; Marc-André Lureau <marcandre.lureau@redhat.com>
Cc: amir.gonnen@neuroblade.ai <amir.gonnen@neuroblade.ai>; qemu-devel@nongnu.org <qemu-devel@nongnu.org>; Alex Bennée <alex.bennee@linaro.org>
Subject: Re: Building QEMU as a Shared Library

Hi Saanjh,

depending what you are trying to achieve exactly, plugins can provide a
solution. It's convenient and you can stay on top of QEMU upstream,
without having to create a downstream fork.

We already have plugins for stopping after a given number of
instructions, or slow down execution of a VM:

# stop after executing 1'000'000 instructions:
$ ./build/qemu-system-x86_64 -plugin
./build/contrib/plugins/libstoptrigger,icount=1000000 -d plugin

# execute no more than 1'000'000 instructions per second:
$ ./build/qemu-system-x86_64 -plugin
./build/contrib/plugins/libips.so,ips=1000000 -d plugin

You can see source code associated (./contrib/plugins/stoptrigger.c and
./contrib/plugins/ips.c), to implement something similar to what you
want, but based on time.
Would that satisfy your need?

Regards,
Pierrick

On 3/3/25 21:53, Saanjh Sengupta wrote:
>
>
> Hi,
>
> Thank you so much for your inputs. I was able to create the .so file of
> QEMU.
>
> Actually, what we are trying is to understand and explore possibilities
> of Virtual Time Control in QEMU. In short, what I mean to say is an
> approach via which I can tell QEMU to emulate for XYZ time when the I
> give a trigger and then pause the emulation by itself after the XYZ time
> is completed.
>
> On that front itself, do you have any inputs/ideas regarding the same?
>
>
> Regards
> Saanjh Sengupta
> ------------------------------------------------------------------------
> *From:* Pierrick Bouvier <pierrick.bouvier@linaro.org>
> *Sent:* Tuesday, February 25, 2025 6:29:44 AM
> *To:* Philippe Mathieu-Daudé <philmd@linaro.org>; Paolo Bonzini
> <pbonzini@redhat.com>; Marc-André Lureau <marcandre.lureau@redhat.com>
> *Cc:* amir.gonnen@neuroblade.ai <amir.gonnen@neuroblade.ai>; qemu-
> devel@nongnu.org <qemu-devel@nongnu.org>; Saanjh Sengupta
> <saanjhsengupta@outlook.com>
> *Subject:* Re: Building QEMU as a Shared Library
> Hi Saanjh,
>
> here is a minimal patch that builds one shared library per target (arch,
> mode) where arch is cpu arch, and mode is system or user, and launch
> system-aarch64 through a simple driver:
>
> https://github.com/pbo-linaro/qemu/commit/
> fbb39cc64f77d4bf1e5e50795c75b62735bf5c5f <https://github.com/pbo-linaro/
> qemu/commit/fbb39cc64f77d4bf1e5e50795c75b62735bf5c5f>
>
> With this, it could be possible to create a driver that can execute any
> existing target. It's a sort of single binary for QEMU, but shared
> objects are mandatory, and duplicates all the QEMU state. So there is no
> real benefit compared to having different processes.
>
> In more, to be able to do concurrent emulations, there are much more
> problems to be solved. QEMU state is correctly kept per target, but all
> other libraries states are shared. There are various issues if you
> launch two emulations at the same time in two threads:
> - glib global context
> - qemu calls exit in many places, which stops the whole process
> - probably other things I didn't explore
>
> At this point, even though qemu targets can be built as shared objects,
> I would recommend to use different processes, and implement some form on
> IPC to synchronize all this.
> Another possibility is to try to build machines without using the
> existing main, but I'm not sure it's worth all the hassle.
>
> What are you trying to achieve?
>
> Regards,
> Pierrick
>
> On 2/24/25 01:10, Philippe Mathieu-Daudé wrote:
>> Cc'ing our meson experts
>>
>> On 22/2/25 14:36, Saanjh Sengupta wrote:
>>> Hi,
>>>
>>> I referred to your mailing chains on suggesting QEMU to be built as a
>>> shared library.
>>>
>>> *Change meson.build to build QEMU as a shared library (with PIC enabled
>>> for static libraries)*
>>> *
>>> *
>>> Could you please suggest what exactly has to be enabled in the meson.build?
>>>
>>> I am confused on that front.
>>>
>>> Regards
>>> Saanjh Sengupta
>>
>


[-- Attachment #1.2: Type: text/html, Size: 8942 bytes --]

[-- Attachment #2: tmp_5e620eff-26a6-46e0-b25b-a1c257cf28eb.png --]
[-- Type: image/png, Size: 21530 bytes --]

Re: Building QEMU as a Shared Library

[Pierrick Bouvier]

On 3/11/25 02:50, Saanjh Sengupta wrote:
> Hi,
> 
> I have a couple of questions:
> 
>  1.
>     When I use the libstoptrigger.so: in that case the QEMU 's emulation
>     stops after executing the defined number of instructions. Post this,
>     the whole QEMU terminates. And while using the libips.so I am
>     assuming that the QEMU doesn't execute no more than the defined
>     instructions. Please correct me if I am wrong.

That's correct for both plugins, with the additional note that libips 
does this per second only.

>  2.
>     In my case, I want the QEMU to start emulation for some time and
>     PAUSE it's emulation for some time; after it is Paused (it's virtual
>     time is also to be paused) and then let's say for after 'x' time
>     period it should resume it's virtual time.
> 

The virtual time variable in ips plugin is only related to this plugin, 
and based on how many instructions have been executed, which is 
different from what you want to achieve.

Stoptrigger might be a better fit for what you want to do, and instead 
of exiting, you want to resume emulation after N insn.
The function qemu_clock_advance_virtual_time() can only be used to move 
the time forward, and you can not stop the "virtual time" by design.

> image
> 
> 
> I have added this segment inside the update_system_time function inside 
> the ipsPlugin.c. but once the instructions reach to the defined limit 
> the virtual time does not seem to stop.
> Do you have any suggestions on that front?
> 
> 
> Regards
> Saanjh Sengupta
> ------------------------------------------------------------------------
> *From:* Pierrick Bouvier <pierrick.bouvier@linaro.org>
> *Sent:* Wednesday, March 5, 2025 5:20:38 AM
> *To:* Saanjh Sengupta <saanjhsengupta@outlook.com>; Philippe Mathieu- 
> Daudé <philmd@linaro.org>; Paolo Bonzini <pbonzini@redhat.com>; Marc- 
> André Lureau <marcandre.lureau@redhat.com>
> *Cc:* amir.gonnen@neuroblade.ai <amir.gonnen@neuroblade.ai>; qemu- 
> devel@nongnu.org <qemu-devel@nongnu.org>; Alex Bennée 
> <alex.bennee@linaro.org>
> *Subject:* Re: Building QEMU as a Shared Library
> Hi Saanjh,
> 
> depending what you are trying to achieve exactly, plugins can provide a
> solution. It's convenient and you can stay on top of QEMU upstream,
> without having to create a downstream fork.
> 
> We already have plugins for stopping after a given number of
> instructions, or slow down execution of a VM:
> 
> # stop after executing 1'000'000 instructions:
> $ ./build/qemu-system-x86_64 -plugin
> ./build/contrib/plugins/libstoptrigger,icount=1000000 -d plugin
> 
> # execute no more than 1'000'000 instructions per second:
> $ ./build/qemu-system-x86_64 -plugin
> ./build/contrib/plugins/libips.so,ips=1000000 -d plugin
> 
> You can see source code associated (./contrib/plugins/stoptrigger.c and
> ./contrib/plugins/ips.c), to implement something similar to what you
> want, but based on time.
> Would that satisfy your need?
> 
> Regards,
> Pierrick
> 
> On 3/3/25 21:53, Saanjh Sengupta wrote:
>> 
>> 
>> Hi,
>> 
>> Thank you so much for your inputs. I was able to create the .so file of 
>> QEMU.
>> 
>> Actually, what we are trying is to understand and explore possibilities 
>> of Virtual Time Control in QEMU. In short, what I mean to say is an 
>> approach via which I can tell QEMU to emulate for XYZ time when the I 
>> give a trigger and then pause the emulation by itself after the XYZ time 
>> is completed.
>> 
>> On that front itself, do you have any inputs/ideas regarding the same?
>> 
>> 
>> Regards
>> Saanjh Sengupta
>> ------------------------------------------------------------------------
>> *From:* Pierrick Bouvier <pierrick.bouvier@linaro.org>
>> *Sent:* Tuesday, February 25, 2025 6:29:44 AM
>> *To:* Philippe Mathieu-Daudé <philmd@linaro.org>; Paolo Bonzini 
>> <pbonzini@redhat.com>; Marc-André Lureau <marcandre.lureau@redhat.com>
>> *Cc:* amir.gonnen@neuroblade.ai <amir.gonnen@neuroblade.ai>; qemu- 
>> devel@nongnu.org <qemu-devel@nongnu.org>; Saanjh Sengupta 
>> <saanjhsengupta@outlook.com>
>> *Subject:* Re: Building QEMU as a Shared Library
>> Hi Saanjh,
>> 
>> here is a minimal patch that builds one shared library per target (arch,
>> mode) where arch is cpu arch, and mode is system or user, and launch
>> system-aarch64 through a simple driver:
>> 
>> https://github.com/pbo-linaro/qemu/commit/ <https://github.com/pbo- 
> linaro/qemu/commit/>
>> fbb39cc64f77d4bf1e5e50795c75b62735bf5c5f <https://github.com/pbo-linaro/
>> qemu/commit/fbb39cc64f77d4bf1e5e50795c75b62735bf5c5f>
>> 
>> With this, it could be possible to create a driver that can execute any
>> existing target. It's a sort of single binary for QEMU, but shared
>> objects are mandatory, and duplicates all the QEMU state. So there is no
>> real benefit compared to having different processes.
>> 
>> In more, to be able to do concurrent emulations, there are much more
>> problems to be solved. QEMU state is correctly kept per target, but all
>> other libraries states are shared. There are various issues if you
>> launch two emulations at the same time in two threads:
>> - glib global context
>> - qemu calls exit in many places, which stops the whole process
>> - probably other things I didn't explore
>> 
>> At this point, even though qemu targets can be built as shared objects,
>> I would recommend to use different processes, and implement some form on
>> IPC to synchronize all this.
>> Another possibility is to try to build machines without using the
>> existing main, but I'm not sure it's worth all the hassle.
>> 
>> What are you trying to achieve?
>> 
>> Regards,
>> Pierrick
>> 
>> On 2/24/25 01:10, Philippe Mathieu-Daudé wrote:
>>> Cc'ing our meson experts
>>> 
>>> On 22/2/25 14:36, Saanjh Sengupta wrote:
>>>> Hi,
>>>>
>>>> I referred to your mailing chains on suggesting QEMU to be built as a
>>>> shared library.
>>>>
>>>> *Change meson.build to build QEMU as a shared library (with PIC enabled
>>>> for static libraries)*
>>>> *
>>>> *
>>>> Could you please suggest what exactly has to be enabled in the meson.build?
>>>>
>>>> I am confused on that front.
>>>>
>>>> Regards
>>>> Saanjh Sengupta
>>> 
>> 
> 

Re: Building QEMU as a Shared Library

[Saanjh Sengupta]

[-- Attachment #1: Type: text/plain, Size: 7942 bytes --]



Hi,

Thank you for the clarification. Regarding the last time
"Stoptrigger might be a better fit for what you want to do, and instead of exiting, you want to resume emulation after N insn. The function qemu_clock_advance_virtual_time() can only be used to move the time forward, and you can not stop the "virtual time" by design."

I did not quite understand this. Even if I have to modify the stopTrigger plugin, I would want it to pause rather than exiting.
For example: It gets 10000 instructions executed after that it should pause and after some time it should then resume again execute till 20000 instructions (because previously it executed till 10000 and then it must execute till 20000). How do I do this? How do I state the code to pause the qemu's emulation after 10000 instructions?

Moreover, I tried an activity where I was utilising the QMP protocol to control the virtual time (with respect to the IPS plugin). In that context when the QMP stop is triggered, my virtual time does got freezed until the resume is triggered. Does this mean I am able to manipulate the virtual time of the QEMU?



Regards
Saanjh Sengupta
________________________________
From: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Sent: Wednesday, March 12, 2025 2:14:47 AM
To: Saanjh Sengupta <saanjhsengupta@outlook.com>; Philippe Mathieu-Daudé <philmd@linaro.org>; Paolo Bonzini <pbonzini@redhat.com>; Marc-André Lureau <marcandre.lureau@redhat.com>
Cc: amir.gonnen@neuroblade.ai <amir.gonnen@neuroblade.ai>; qemu-devel@nongnu.org <qemu-devel@nongnu.org>; Alex Bennée <alex.bennee@linaro.org>
Subject: Re: Building QEMU as a Shared Library

On 3/11/25 02:50, Saanjh Sengupta wrote:
> Hi,
>
> I have a couple of questions:
>
>  1.
>     When I use the libstoptrigger.so: in that case the QEMU 's emulation
>     stops after executing the defined number of instructions. Post this,
>     the whole QEMU terminates. And while using the libips.so I am
>     assuming that the QEMU doesn't execute no more than the defined
>     instructions. Please correct me if I am wrong.

That's correct for both plugins, with the additional note that libips
does this per second only.

>  2.
>     In my case, I want the QEMU to start emulation for some time and
>     PAUSE it's emulation for some time; after it is Paused (it's virtual
>     time is also to be paused) and then let's say for after 'x' time
>     period it should resume it's virtual time.
>

The virtual time variable in ips plugin is only related to this plugin,
and based on how many instructions have been executed, which is
different from what you want to achieve.

Stoptrigger might be a better fit for what you want to do, and instead
of exiting, you want to resume emulation after N insn.
The function qemu_clock_advance_virtual_time() can only be used to move
the time forward, and you can not stop the "virtual time" by design.

> image
>
>
> I have added this segment inside the update_system_time function inside
> the ipsPlugin.c. but once the instructions reach to the defined limit
> the virtual time does not seem to stop.
> Do you have any suggestions on that front?
>
>
> Regards
> Saanjh Sengupta
> ------------------------------------------------------------------------
> *From:* Pierrick Bouvier <pierrick.bouvier@linaro.org>
> *Sent:* Wednesday, March 5, 2025 5:20:38 AM
> *To:* Saanjh Sengupta <saanjhsengupta@outlook.com>; Philippe Mathieu-
> Daudé <philmd@linaro.org>; Paolo Bonzini <pbonzini@redhat.com>; Marc-
> André Lureau <marcandre.lureau@redhat.com>
> *Cc:* amir.gonnen@neuroblade.ai <amir.gonnen@neuroblade.ai>; qemu-
> devel@nongnu.org <qemu-devel@nongnu.org>; Alex Bennée
> <alex.bennee@linaro.org>
> *Subject:* Re: Building QEMU as a Shared Library
> Hi Saanjh,
>
> depending what you are trying to achieve exactly, plugins can provide a
> solution. It's convenient and you can stay on top of QEMU upstream,
> without having to create a downstream fork.
>
> We already have plugins for stopping after a given number of
> instructions, or slow down execution of a VM:
>
> # stop after executing 1'000'000 instructions:
> $ ./build/qemu-system-x86_64 -plugin
> ./build/contrib/plugins/libstoptrigger,icount=1000000 -d plugin
>
> # execute no more than 1'000'000 instructions per second:
> $ ./build/qemu-system-x86_64 -plugin
> ./build/contrib/plugins/libips.so,ips=1000000 -d plugin
>
> You can see source code associated (./contrib/plugins/stoptrigger.c and
> ./contrib/plugins/ips.c), to implement something similar to what you
> want, but based on time.
> Would that satisfy your need?
>
> Regards,
> Pierrick
>
> On 3/3/25 21:53, Saanjh Sengupta wrote:
>>
>>
>> Hi,
>>
>> Thank you so much for your inputs. I was able to create the .so file of
>> QEMU.
>>
>> Actually, what we are trying is to understand and explore possibilities
>> of Virtual Time Control in QEMU. In short, what I mean to say is an
>> approach via which I can tell QEMU to emulate for XYZ time when the I
>> give a trigger and then pause the emulation by itself after the XYZ time
>> is completed.
>>
>> On that front itself, do you have any inputs/ideas regarding the same?
>>
>>
>> Regards
>> Saanjh Sengupta
>> ------------------------------------------------------------------------
>> *From:* Pierrick Bouvier <pierrick.bouvier@linaro.org>
>> *Sent:* Tuesday, February 25, 2025 6:29:44 AM
>> *To:* Philippe Mathieu-Daudé <philmd@linaro.org>; Paolo Bonzini
>> <pbonzini@redhat.com>; Marc-André Lureau <marcandre.lureau@redhat.com>
>> *Cc:* amir.gonnen@neuroblade.ai <amir.gonnen@neuroblade.ai>; qemu-
>> devel@nongnu.org <qemu-devel@nongnu.org>; Saanjh Sengupta
>> <saanjhsengupta@outlook.com>
>> *Subject:* Re: Building QEMU as a Shared Library
>> Hi Saanjh,
>>
>> here is a minimal patch that builds one shared library per target (arch,
>> mode) where arch is cpu arch, and mode is system or user, and launch
>> system-aarch64 through a simple driver:
>>
>> https://github.com/pbo-linaro/qemu/commit/ <https://github.com/pbo-
> linaro/qemu/commit/>
>> fbb39cc64f77d4bf1e5e50795c75b62735bf5c5f <https://github.com/pbo-linaro/
>> qemu/commit/fbb39cc64f77d4bf1e5e50795c75b62735bf5c5f>
>>
>> With this, it could be possible to create a driver that can execute any
>> existing target. It's a sort of single binary for QEMU, but shared
>> objects are mandatory, and duplicates all the QEMU state. So there is no
>> real benefit compared to having different processes.
>>
>> In more, to be able to do concurrent emulations, there are much more
>> problems to be solved. QEMU state is correctly kept per target, but all
>> other libraries states are shared. There are various issues if you
>> launch two emulations at the same time in two threads:
>> - glib global context
>> - qemu calls exit in many places, which stops the whole process
>> - probably other things I didn't explore
>>
>> At this point, even though qemu targets can be built as shared objects,
>> I would recommend to use different processes, and implement some form on
>> IPC to synchronize all this.
>> Another possibility is to try to build machines without using the
>> existing main, but I'm not sure it's worth all the hassle.
>>
>> What are you trying to achieve?
>>
>> Regards,
>> Pierrick
>>
>> On 2/24/25 01:10, Philippe Mathieu-Daudé wrote:
>>> Cc'ing our meson experts
>>>
>>> On 22/2/25 14:36, Saanjh Sengupta wrote:
>>>> Hi,
>>>>
>>>> I referred to your mailing chains on suggesting QEMU to be built as a
>>>> shared library.
>>>>
>>>> *Change meson.build to build QEMU as a shared library (with PIC enabled
>>>> for static libraries)*
>>>> *
>>>> *
>>>> Could you please suggest what exactly has to be enabled in the meson.build?
>>>>
>>>> I am confused on that front.
>>>>
>>>> Regards
>>>> Saanjh Sengupta
>>>
>>
>


[-- Attachment #2: Type: text/html, Size: 10905 bytes --]

Re: Building QEMU as a Shared Library

[Pierrick Bouvier]

On 3/11/25 21:31, Saanjh Sengupta wrote:
> 
> 
> Hi,
> 
> Thank you for the clarification. Regarding the last time
> /"Stoptrigger might be a better fit for what you want to do, and instead 
> of exiting, you want to resume emulation after N insn. The function 
> qemu_clock_advance_virtual_time() can only be used to move the time 
> forward, and you can not stop the "virtual time" by design."/
> /
> /
> I did not quite understand this. Even if I have to modify the 
> stopTrigger plugin, I would want it to pause rather than exiting.
> For example: It gets 10000 instructions executed after that it should 
> pause and after some time it should then resume again execute till 20000 
> instructions (because previously it executed till 10000 and then it must 
> execute till 20000). How do I do this? How do I state the code to pause 
> the qemu's emulation after 10000 instructions?
> 

By using g_usleep to pause the current cpu.
As well, it's needed to reset insn_count to 0 to count instructions again.

With this command line:
./build/qemu-system-x86_64 -plugin 
./build/contrib/plugins/libstoptrigger.so,icount=1000 -d plugin

And with those changes to stoptrigger:

diff --git a/contrib/plugins/stoptrigger.c b/contrib/plugins/stoptrigger.c
index b3a6ed66a7b..77fd413cef1 100644
--- a/contrib/plugins/stoptrigger.c
+++ b/contrib/plugins/stoptrigger.c
@@ -41,11 +41,12 @@ typedef struct {
      int exit_code;
  } ExitInfo;

-static void exit_emulation(int return_code, char *message)
+static void pause_emulation(int return_code, char *message)
  {
      qemu_plugin_outs(message);
      g_free(message);
-    exit(return_code);
+    /* exit(return_code); */
+    g_usleep(1 * G_USEC_PER_SEC);
  }

  static void exit_icount_reached(unsigned int cpu_index, void *udata)
@@ -53,7 +54,9 @@ static void exit_icount_reached(unsigned int 
cpu_index, void *udata)
      uint64_t insn_vaddr = qemu_plugin_u64_get(current_pc, cpu_index);
      char *msg = g_strdup_printf("icount reached at 0x%" PRIx64 ", 
exiting\n",
                                  insn_vaddr);
-    exit_emulation(icount_exit_code, msg);
+    pause_emulation(icount_exit_code, msg);
+    /* reset instruction counter */
+    qemu_plugin_u64_set(insn_count, cpu_index, 0);
  }

  static void exit_address_reached(unsigned int cpu_index, void *udata)
@@ -61,7 +64,7 @@ static void exit_address_reached(unsigned int 
cpu_index, void *udata)
      ExitInfo *ei = udata;
      g_assert(ei);
      char *msg = g_strdup_printf("0x%" PRIx64 " reached, exiting\n", 
ei->exit_addr);
-    exit_emulation(ei->exit_code, msg);
+    pause_emulation(ei->exit_code, msg);
  }


> Moreover, I tried an activity where I was utilising the QMP protocol to 
> control the virtual time (with respect to the IPS plugin). In that 
> context when the QMP stop is triggered, my virtual time does got freezed 
> until the resume is triggered. Does this mean I am able to manipulate 
> the virtual time of the QEMU?
> 

I am not sure of how it works, but the plugin interface only allows to 
move time forward.

> 
> 
> Regards
> Saanjh Sengupta
> ------------------------------------------------------------------------
> *From:* Pierrick Bouvier <pierrick.bouvier@linaro.org>
> *Sent:* Wednesday, March 12, 2025 2:14:47 AM
> *To:* Saanjh Sengupta <saanjhsengupta@outlook.com>; Philippe Mathieu- 
> Daudé <philmd@linaro.org>; Paolo Bonzini <pbonzini@redhat.com>; Marc- 
> André Lureau <marcandre.lureau@redhat.com>
> *Cc:* amir.gonnen@neuroblade.ai <amir.gonnen@neuroblade.ai>; qemu- 
> devel@nongnu.org <qemu-devel@nongnu.org>; Alex Bennée 
> <alex.bennee@linaro.org>
> *Subject:* Re: Building QEMU as a Shared Library
> On 3/11/25 02:50, Saanjh Sengupta wrote:
>> Hi,
>> 
>> I have a couple of questions:
>> 
>>  1.
>>     When I use the libstoptrigger.so: in that case the QEMU 's emulation
>>     stops after executing the defined number of instructions. Post this,
>>     the whole QEMU terminates. And while using the libips.so I am
>>     assuming that the QEMU doesn't execute no more than the defined
>>     instructions. Please correct me if I am wrong.
> 
> That's correct for both plugins, with the additional note that libips
> does this per second only.
> 
>>  2.
>>     In my case, I want the QEMU to start emulation for some time and
>>     PAUSE it's emulation for some time; after it is Paused (it's virtual
>>     time is also to be paused) and then let's say for after 'x' time
>>     period it should resume it's virtual time.
>> 
> 
> The virtual time variable in ips plugin is only related to this plugin,
> and based on how many instructions have been executed, which is
> different from what you want to achieve.
> 
> Stoptrigger might be a better fit for what you want to do, and instead
> of exiting, you want to resume emulation after N insn.
> The function qemu_clock_advance_virtual_time() can only be used to move
> the time forward, and you can not stop the "virtual time" by design.
> 
>> image
>> 
>> 
>> I have added this segment inside the update_system_time function inside 
>> the ipsPlugin.c. but once the instructions reach to the defined limit 
>> the virtual time does not seem to stop.
>> Do you have any suggestions on that front?
>> 
>> 
>> Regards
>> Saanjh Sengupta
>> ------------------------------------------------------------------------
>> *From:* Pierrick Bouvier <pierrick.bouvier@linaro.org>
>> *Sent:* Wednesday, March 5, 2025 5:20:38 AM
>> *To:* Saanjh Sengupta <saanjhsengupta@outlook.com>; Philippe Mathieu- 
>> Daudé <philmd@linaro.org>; Paolo Bonzini <pbonzini@redhat.com>; Marc- 
>> André Lureau <marcandre.lureau@redhat.com>
>> *Cc:* amir.gonnen@neuroblade.ai <amir.gonnen@neuroblade.ai>; qemu- 
>> devel@nongnu.org <qemu-devel@nongnu.org>; Alex Bennée 
>> <alex.bennee@linaro.org>
>> *Subject:* Re: Building QEMU as a Shared Library
>> Hi Saanjh,
>> 
>> depending what you are trying to achieve exactly, plugins can provide a
>> solution. It's convenient and you can stay on top of QEMU upstream,
>> without having to create a downstream fork.
>> 
>> We already have plugins for stopping after a given number of
>> instructions, or slow down execution of a VM:
>> 
>> # stop after executing 1'000'000 instructions:
>> $ ./build/qemu-system-x86_64 -plugin
>> ./build/contrib/plugins/libstoptrigger,icount=1000000 -d plugin
>> 
>> # execute no more than 1'000'000 instructions per second:
>> $ ./build/qemu-system-x86_64 -plugin
>> ./build/contrib/plugins/libips.so,ips=1000000 -d plugin
>> 
>> You can see source code associated (./contrib/plugins/stoptrigger.c and
>> ./contrib/plugins/ips.c), to implement something similar to what you
>> want, but based on time.
>> Would that satisfy your need?
>> 
>> Regards,
>> Pierrick
>> 
>> On 3/3/25 21:53, Saanjh Sengupta wrote:
>>> 
>>> 
>>> Hi,
>>> 
>>> Thank you so much for your inputs. I was able to create the .so file of 
>>> QEMU.
>>> 
>>> Actually, what we are trying is to understand and explore possibilities 
>>> of Virtual Time Control in QEMU. In short, what I mean to say is an 
>>> approach via which I can tell QEMU to emulate for XYZ time when the I 
>>> give a trigger and then pause the emulation by itself after the XYZ time 
>>> is completed.
>>> 
>>> On that front itself, do you have any inputs/ideas regarding the same?
>>> 
>>> 
>>> Regards
>>> Saanjh Sengupta
>>> ------------------------------------------------------------------------
>>> *From:* Pierrick Bouvier <pierrick.bouvier@linaro.org>
>>> *Sent:* Tuesday, February 25, 2025 6:29:44 AM
>>> *To:* Philippe Mathieu-Daudé <philmd@linaro.org>; Paolo Bonzini 
>>> <pbonzini@redhat.com>; Marc-André Lureau <marcandre.lureau@redhat.com>
>>> *Cc:* amir.gonnen@neuroblade.ai <amir.gonnen@neuroblade.ai>; qemu- 
>>> devel@nongnu.org <qemu-devel@nongnu.org>; Saanjh Sengupta 
>>> <saanjhsengupta@outlook.com>
>>> *Subject:* Re: Building QEMU as a Shared Library
>>> Hi Saanjh,
>>> 
>>> here is a minimal patch that builds one shared library per target (arch,
>>> mode) where arch is cpu arch, and mode is system or user, and launch
>>> system-aarch64 through a simple driver:
>>> 
>>> https://github.com/pbo-linaro/qemu/commit/ <https://github.com/pbo- 
> linaro/qemu/commit/> <https://github.com/pbo-
>> linaro/qemu/commit/>
>>> fbb39cc64f77d4bf1e5e50795c75b62735bf5c5f <https://github.com/pbo-linaro/
>>> qemu/commit/fbb39cc64f77d4bf1e5e50795c75b62735bf5c5f>
>>> 
>>> With this, it could be possible to create a driver that can execute any
>>> existing target. It's a sort of single binary for QEMU, but shared
>>> objects are mandatory, and duplicates all the QEMU state. So there is no
>>> real benefit compared to having different processes.
>>> 
>>> In more, to be able to do concurrent emulations, there are much more
>>> problems to be solved. QEMU state is correctly kept per target, but all
>>> other libraries states are shared. There are various issues if you
>>> launch two emulations at the same time in two threads:
>>> - glib global context
>>> - qemu calls exit in many places, which stops the whole process
>>> - probably other things I didn't explore
>>> 
>>> At this point, even though qemu targets can be built as shared objects,
>>> I would recommend to use different processes, and implement some form on
>>> IPC to synchronize all this.
>>> Another possibility is to try to build machines without using the
>>> existing main, but I'm not sure it's worth all the hassle.
>>> 
>>> What are you trying to achieve?
>>> 
>>> Regards,
>>> Pierrick
>>> 
>>> On 2/24/25 01:10, Philippe Mathieu-Daudé wrote:
>>>> Cc'ing our meson experts
>>>> 
>>>> On 22/2/25 14:36, Saanjh Sengupta wrote:
>>>>> Hi,
>>>>>
>>>>> I referred to your mailing chains on suggesting QEMU to be built as a
>>>>> shared library.
>>>>>
>>>>> *Change meson.build to build QEMU as a shared library (with PIC enabled
>>>>> for static libraries)*
>>>>> *
>>>>> *
>>>>> Could you please suggest what exactly has to be enabled in the meson.build?
>>>>>
>>>>> I am confused on that front.
>>>>>
>>>>> Regards
>>>>> Saanjh Sengupta
>>>> 
>>> 
>> 
> 

Re: Building QEMU as a Shared Library

[Saanjh Sengupta]

[-- Attachment #1: Type: text/plain, Size: 11385 bytes --]

Hi,

What we are trying to achieve is that the QEMU should run for a particular number of instructions, let's say for example 10000 instructions and then pause it's emulation. After a resume trigger is received to the QEMU it must resume it's emulation and start the instruction count from 10001 (which basically means that the context should be saved).

In the previous mail when you mentioned g_usleep, I believe this shall not work (as per our use-case) since it will reset the instruction count to 0 (as per what you mentioned).

To achieve the use-case, do you have any leads/suggestions ?


Regards
Saanjh Sengupta

________________________________
From: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Sent: Wednesday, March 12, 2025 11:50:23 am
To: Saanjh Sengupta <saanjhsengupta@outlook.com>; Philippe Mathieu-Daudé <philmd@linaro.org>; Paolo Bonzini <pbonzini@redhat.com>; Marc-André Lureau <marcandre.lureau@redhat.com>
Cc: amir.gonnen@neuroblade.ai <amir.gonnen@neuroblade.ai>; qemu-devel@nongnu.org <qemu-devel@nongnu.org>; Alex Bennée <alex.bennee@linaro.org>
Subject: Re: Building QEMU as a Shared Library

On 3/11/25 21:31, Saanjh Sengupta wrote:
>
>
> Hi,
>
> Thank you for the clarification. Regarding the last time
> /"Stoptrigger might be a better fit for what you want to do, and instead
> of exiting, you want to resume emulation after N insn. The function
> qemu_clock_advance_virtual_time() can only be used to move the time
> forward, and you can not stop the "virtual time" by design."/
> /
> /
> I did not quite understand this. Even if I have to modify the
> stopTrigger plugin, I would want it to pause rather than exiting.
> For example: It gets 10000 instructions executed after that it should
> pause and after some time it should then resume again execute till 20000
> instructions (because previously it executed till 10000 and then it must
> execute till 20000). How do I do this? How do I state the code to pause
> the qemu's emulation after 10000 instructions?
>

By using g_usleep to pause the current cpu.
As well, it's needed to reset insn_count to 0 to count instructions again.

With this command line:
./build/qemu-system-x86_64 -plugin
./build/contrib/plugins/libstoptrigger.so,icount=1000 -d plugin

And with those changes to stoptrigger:

diff --git a/contrib/plugins/stoptrigger.c b/contrib/plugins/stoptrigger.c
index b3a6ed66a7b..77fd413cef1 100644
--- a/contrib/plugins/stoptrigger.c
+++ b/contrib/plugins/stoptrigger.c
@@ -41,11 +41,12 @@ typedef struct {
      int exit_code;
  } ExitInfo;

-static void exit_emulation(int return_code, char *message)
+static void pause_emulation(int return_code, char *message)
  {
      qemu_plugin_outs(message);
      g_free(message);
-    exit(return_code);
+    /* exit(return_code); */
+    g_usleep(1 * G_USEC_PER_SEC);
  }

  static void exit_icount_reached(unsigned int cpu_index, void *udata)
@@ -53,7 +54,9 @@ static void exit_icount_reached(unsigned int
cpu_index, void *udata)
      uint64_t insn_vaddr = qemu_plugin_u64_get(current_pc, cpu_index);
      char *msg = g_strdup_printf("icount reached at 0x%" PRIx64 ",
exiting\n",
                                  insn_vaddr);
-    exit_emulation(icount_exit_code, msg);
+    pause_emulation(icount_exit_code, msg);
+    /* reset instruction counter */
+    qemu_plugin_u64_set(insn_count, cpu_index, 0);
  }

  static void exit_address_reached(unsigned int cpu_index, void *udata)
@@ -61,7 +64,7 @@ static void exit_address_reached(unsigned int
cpu_index, void *udata)
      ExitInfo *ei = udata;
      g_assert(ei);
      char *msg = g_strdup_printf("0x%" PRIx64 " reached, exiting\n",
ei->exit_addr);
-    exit_emulation(ei->exit_code, msg);
+    pause_emulation(ei->exit_code, msg);
  }


> Moreover, I tried an activity where I was utilising the QMP protocol to
> control the virtual time (with respect to the IPS plugin). In that
> context when the QMP stop is triggered, my virtual time does got freezed
> until the resume is triggered. Does this mean I am able to manipulate
> the virtual time of the QEMU?
>

I am not sure of how it works, but the plugin interface only allows to
move time forward.

>
>
> Regards
> Saanjh Sengupta
> ------------------------------------------------------------------------
> *From:* Pierrick Bouvier <pierrick.bouvier@linaro.org>
> *Sent:* Wednesday, March 12, 2025 2:14:47 AM
> *To:* Saanjh Sengupta <saanjhsengupta@outlook.com>; Philippe Mathieu-
> Daudé <philmd@linaro.org>; Paolo Bonzini <pbonzini@redhat.com>; Marc-
> André Lureau <marcandre.lureau@redhat.com>
> *Cc:* amir.gonnen@neuroblade.ai <amir.gonnen@neuroblade.ai>; qemu-
> devel@nongnu.org <qemu-devel@nongnu.org>; Alex Bennée
> <alex.bennee@linaro.org>
> *Subject:* Re: Building QEMU as a Shared Library
> On 3/11/25 02:50, Saanjh Sengupta wrote:
>> Hi,
>>
>> I have a couple of questions:
>>
>>  1.
>>     When I use the libstoptrigger.so: in that case the QEMU 's emulation
>>     stops after executing the defined number of instructions. Post this,
>>     the whole QEMU terminates. And while using the libips.so I am
>>     assuming that the QEMU doesn't execute no more than the defined
>>     instructions. Please correct me if I am wrong.
>
> That's correct for both plugins, with the additional note that libips
> does this per second only.
>
>>  2.
>>     In my case, I want the QEMU to start emulation for some time and
>>     PAUSE it's emulation for some time; after it is Paused (it's virtual
>>     time is also to be paused) and then let's say for after 'x' time
>>     period it should resume it's virtual time.
>>
>
> The virtual time variable in ips plugin is only related to this plugin,
> and based on how many instructions have been executed, which is
> different from what you want to achieve.
>
> Stoptrigger might be a better fit for what you want to do, and instead
> of exiting, you want to resume emulation after N insn.
> The function qemu_clock_advance_virtual_time() can only be used to move
> the time forward, and you can not stop the "virtual time" by design.
>
>> image
>>
>>
>> I have added this segment inside the update_system_time function inside
>> the ipsPlugin.c. but once the instructions reach to the defined limit
>> the virtual time does not seem to stop.
>> Do you have any suggestions on that front?
>>
>>
>> Regards
>> Saanjh Sengupta
>> ------------------------------------------------------------------------
>> *From:* Pierrick Bouvier <pierrick.bouvier@linaro.org>
>> *Sent:* Wednesday, March 5, 2025 5:20:38 AM
>> *To:* Saanjh Sengupta <saanjhsengupta@outlook.com>; Philippe Mathieu-
>> Daudé <philmd@linaro.org>; Paolo Bonzini <pbonzini@redhat.com>; Marc-
>> André Lureau <marcandre.lureau@redhat.com>
>> *Cc:* amir.gonnen@neuroblade.ai <amir.gonnen@neuroblade.ai>; qemu-
>> devel@nongnu.org <qemu-devel@nongnu.org>; Alex Bennée
>> <alex.bennee@linaro.org>
>> *Subject:* Re: Building QEMU as a Shared Library
>> Hi Saanjh,
>>
>> depending what you are trying to achieve exactly, plugins can provide a
>> solution. It's convenient and you can stay on top of QEMU upstream,
>> without having to create a downstream fork.
>>
>> We already have plugins for stopping after a given number of
>> instructions, or slow down execution of a VM:
>>
>> # stop after executing 1'000'000 instructions:
>> $ ./build/qemu-system-x86_64 -plugin
>> ./build/contrib/plugins/libstoptrigger,icount=1000000 -d plugin
>>
>> # execute no more than 1'000'000 instructions per second:
>> $ ./build/qemu-system-x86_64 -plugin
>> ./build/contrib/plugins/libips.so,ips=1000000 -d plugin
>>
>> You can see source code associated (./contrib/plugins/stoptrigger.c and
>> ./contrib/plugins/ips.c), to implement something similar to what you
>> want, but based on time.
>> Would that satisfy your need?
>>
>> Regards,
>> Pierrick
>>
>> On 3/3/25 21:53, Saanjh Sengupta wrote:
>>>
>>>
>>> Hi,
>>>
>>> Thank you so much for your inputs. I was able to create the .so file of
>>> QEMU.
>>>
>>> Actually, what we are trying is to understand and explore possibilities
>>> of Virtual Time Control in QEMU. In short, what I mean to say is an
>>> approach via which I can tell QEMU to emulate for XYZ time when the I
>>> give a trigger and then pause the emulation by itself after the XYZ time
>>> is completed.
>>>
>>> On that front itself, do you have any inputs/ideas regarding the same?
>>>
>>>
>>> Regards
>>> Saanjh Sengupta
>>> ------------------------------------------------------------------------
>>> *From:* Pierrick Bouvier <pierrick.bouvier@linaro.org>
>>> *Sent:* Tuesday, February 25, 2025 6:29:44 AM
>>> *To:* Philippe Mathieu-Daudé <philmd@linaro.org>; Paolo Bonzini
>>> <pbonzini@redhat.com>; Marc-André Lureau <marcandre.lureau@redhat.com>
>>> *Cc:* amir.gonnen@neuroblade.ai <amir.gonnen@neuroblade.ai>; qemu-
>>> devel@nongnu.org <qemu-devel@nongnu.org>; Saanjh Sengupta
>>> <saanjhsengupta@outlook.com>
>>> *Subject:* Re: Building QEMU as a Shared Library
>>> Hi Saanjh,
>>>
>>> here is a minimal patch that builds one shared library per target (arch,
>>> mode) where arch is cpu arch, and mode is system or user, and launch
>>> system-aarch64 through a simple driver:
>>>
>>> https://github.com/pbo-linaro/qemu/commit/ <https://github.com/pbo-
> linaro/qemu/commit/> <https://github.com/pbo-
>> linaro/qemu/commit/>
>>> fbb39cc64f77d4bf1e5e50795c75b62735bf5c5f <https://github.com/pbo-linaro/
>>> qemu/commit/fbb39cc64f77d4bf1e5e50795c75b62735bf5c5f>
>>>
>>> With this, it could be possible to create a driver that can execute any
>>> existing target. It's a sort of single binary for QEMU, but shared
>>> objects are mandatory, and duplicates all the QEMU state. So there is no
>>> real benefit compared to having different processes.
>>>
>>> In more, to be able to do concurrent emulations, there are much more
>>> problems to be solved. QEMU state is correctly kept per target, but all
>>> other libraries states are shared. There are various issues if you
>>> launch two emulations at the same time in two threads:
>>> - glib global context
>>> - qemu calls exit in many places, which stops the whole process
>>> - probably other things I didn't explore
>>>
>>> At this point, even though qemu targets can be built as shared objects,
>>> I would recommend to use different processes, and implement some form on
>>> IPC to synchronize all this.
>>> Another possibility is to try to build machines without using the
>>> existing main, but I'm not sure it's worth all the hassle.
>>>
>>> What are you trying to achieve?
>>>
>>> Regards,
>>> Pierrick
>>>
>>> On 2/24/25 01:10, Philippe Mathieu-Daudé wrote:
>>>> Cc'ing our meson experts
>>>>
>>>> On 22/2/25 14:36, Saanjh Sengupta wrote:
>>>>> Hi,
>>>>>
>>>>> I referred to your mailing chains on suggesting QEMU to be built as a
>>>>> shared library.
>>>>>
>>>>> *Change meson.build to build QEMU as a shared library (with PIC enabled
>>>>> for static libraries)*
>>>>> *
>>>>> *
>>>>> Could you please suggest what exactly has to be enabled in the meson.build?
>>>>>
>>>>> I am confused on that front.
>>>>>
>>>>> Regards
>>>>> Saanjh Sengupta
>>>>
>>>
>>
>



[-- Attachment #2: Type: text/html, Size: 16018 bytes --]

Re: Building QEMU as a Shared Library

[Pierrick Bouvier]

On 3/13/25 04:34, Saanjh Sengupta wrote:
> Hi,
> 
> What we are trying to achieve is that the QEMU should run for a 
> particular number of instructions, let's say for example 10000 
> instructions and then pause it's emulation. After a resume trigger is 
> received to the QEMU it must resume it's emulation and start the 
> instruction count from 10001 (which basically means that the context 
> should be saved).
> 
> In the previous mail when you mentioned g_usleep, I believe this shall 
> not work (as per our use-case) since it will reset the instruction count 
> to 0 (as per what you mentioned).
> 

By resetting the instruction counter, I just meant the reset the 
variable used to stop every N instructions. It does not affect the state 
of QEMU itself.

I invite you to check our plugins documentation [1], which explains the 
usage of conditional callbacks, and associated functions.

[1] https://www.qemu.org/docs/master/devel/tcg-plugins.html#writing-plugins

Regards,
Pierrick

> To achieve the use-case, do you have any leads/suggestions ?
> 
> 
> Regards
> Saanjh Sengupta
> 
> ------------------------------------------------------------------------
> *From:* Pierrick Bouvier <pierrick.bouvier@linaro.org>
> *Sent:* Wednesday, March 12, 2025 11:50:23 am
> *To:* Saanjh Sengupta <saanjhsengupta@outlook.com>; Philippe Mathieu- 
> Daudé <philmd@linaro.org>; Paolo Bonzini <pbonzini@redhat.com>; Marc- 
> André Lureau <marcandre.lureau@redhat.com>
> *Cc:* amir.gonnen@neuroblade.ai <amir.gonnen@neuroblade.ai>; qemu- 
> devel@nongnu.org <qemu-devel@nongnu.org>; Alex Bennée 
> <alex.bennee@linaro.org>
> *Subject:* Re: Building QEMU as a Shared Library
> 
> On 3/11/25 21:31, Saanjh Sengupta wrote:
>> 
>> 
>> Hi,
>> 
>> Thank you for the clarification. Regarding the last time
>> /"Stoptrigger might be a better fit for what you want to do, and instead 
>> of exiting, you want to resume emulation after N insn. The function 
>> qemu_clock_advance_virtual_time() can only be used to move the time 
>> forward, and you can not stop the "virtual time" by design."/
>> /
>> /
>> I did not quite understand this. Even if I have to modify the 
>> stopTrigger plugin, I would want it to pause rather than exiting.
>> For example: It gets 10000 instructions executed after that it should 
>> pause and after some time it should then resume again execute till 20000 
>> instructions (because previously it executed till 10000 and then it must 
>> execute till 20000). How do I do this? How do I state the code to pause 
>> the qemu's emulation after 10000 instructions?
>> 
> 
> By using g_usleep to pause the current cpu.
> As well, it's needed to reset insn_count to 0 to count instructions again.
> 
> With this command line:
> ./build/qemu-system-x86_64 -plugin
> ./build/contrib/plugins/libstoptrigger.so,icount=1000 -d plugin
> 
> And with those changes to stoptrigger:
> 
> diff --git a/contrib/plugins/stoptrigger.c b/contrib/plugins/stoptrigger.c
> index b3a6ed66a7b..77fd413cef1 100644
> --- a/contrib/plugins/stoptrigger.c
> +++ b/contrib/plugins/stoptrigger.c
> @@ -41,11 +41,12 @@ typedef struct {
>        int exit_code;
>    } ExitInfo;
> 
> -static void exit_emulation(int return_code, char *message)
> +static void pause_emulation(int return_code, char *message)
>    {
>        qemu_plugin_outs(message);
>        g_free(message);
> -    exit(return_code);
> +    /* exit(return_code); */
> +    g_usleep(1 * G_USEC_PER_SEC);
>    }
> 
>    static void exit_icount_reached(unsigned int cpu_index, void *udata)
> @@ -53,7 +54,9 @@ static void exit_icount_reached(unsigned int
> cpu_index, void *udata)
>        uint64_t insn_vaddr = qemu_plugin_u64_get(current_pc, cpu_index);
>        char *msg = g_strdup_printf("icount reached at 0x%" PRIx64 ",
> exiting\n",
>                                    insn_vaddr);
> -    exit_emulation(icount_exit_code, msg);
> +    pause_emulation(icount_exit_code, msg);
> +    /* reset instruction counter */
> +    qemu_plugin_u64_set(insn_count, cpu_index, 0);
>    }
> 
>    static void exit_address_reached(unsigned int cpu_index, void *udata)
> @@ -61,7 +64,7 @@ static void exit_address_reached(unsigned int
> cpu_index, void *udata)
>        ExitInfo *ei = udata;
>        g_assert(ei);
>        char *msg = g_strdup_printf("0x%" PRIx64 " reached, exiting\n",
> ei->exit_addr);
> -    exit_emulation(ei->exit_code, msg);
> +    pause_emulation(ei->exit_code, msg);
>    }
> 
> 
>> Moreover, I tried an activity where I was utilising the QMP protocol to 
>> control the virtual time (with respect to the IPS plugin). In that 
>> context when the QMP stop is triggered, my virtual time does got freezed 
>> until the resume is triggered. Does this mean I am able to manipulate 
>> the virtual time of the QEMU?
>> 
> 
> I am not sure of how it works, but the plugin interface only allows to
> move time forward.
> 
>> 
>> 
>> Regards
>> Saanjh Sengupta
>> ------------------------------------------------------------------------
>> *From:* Pierrick Bouvier <pierrick.bouvier@linaro.org>
>> *Sent:* Wednesday, March 12, 2025 2:14:47 AM
>> *To:* Saanjh Sengupta <saanjhsengupta@outlook.com>; Philippe Mathieu- 
>> Daudé <philmd@linaro.org>; Paolo Bonzini <pbonzini@redhat.com>; Marc- 
>> André Lureau <marcandre.lureau@redhat.com>
>> *Cc:* amir.gonnen@neuroblade.ai <amir.gonnen@neuroblade.ai>; qemu- 
>> devel@nongnu.org <qemu-devel@nongnu.org>; Alex Bennée 
>> <alex.bennee@linaro.org>
>> *Subject:* Re: Building QEMU as a Shared Library
>> On 3/11/25 02:50, Saanjh Sengupta wrote:
>>> Hi,
>>> 
>>> I have a couple of questions:
>>> 
>>>  1.
>>>     When I use the libstoptrigger.so: in that case the QEMU 's emulation
>>>     stops after executing the defined number of instructions. Post this,
>>>     the whole QEMU terminates. And while using the libips.so I am
>>>     assuming that the QEMU doesn't execute no more than the defined
>>>     instructions. Please correct me if I am wrong.
>> 
>> That's correct for both plugins, with the additional note that libips
>> does this per second only.
>> 
>>>  2.
>>>     In my case, I want the QEMU to start emulation for some time and
>>>     PAUSE it's emulation for some time; after it is Paused (it's virtual
>>>     time is also to be paused) and then let's say for after 'x' time
>>>     period it should resume it's virtual time.
>>> 
>> 
>> The virtual time variable in ips plugin is only related to this plugin,
>> and based on how many instructions have been executed, which is
>> different from what you want to achieve.
>> 
>> Stoptrigger might be a better fit for what you want to do, and instead
>> of exiting, you want to resume emulation after N insn.
>> The function qemu_clock_advance_virtual_time() can only be used to move
>> the time forward, and you can not stop the "virtual time" by design.
>> 
>>> image
>>> 
>>> 
>>> I have added this segment inside the update_system_time function inside 
>>> the ipsPlugin.c. but once the instructions reach to the defined limit 
>>> the virtual time does not seem to stop.
>>> Do you have any suggestions on that front?
>>> 
>>> 
>>> Regards
>>> Saanjh Sengupta
>>> ------------------------------------------------------------------------
>>> *From:* Pierrick Bouvier <pierrick.bouvier@linaro.org>
>>> *Sent:* Wednesday, March 5, 2025 5:20:38 AM
>>> *To:* Saanjh Sengupta <saanjhsengupta@outlook.com>; Philippe Mathieu- 
>>> Daudé <philmd@linaro.org>; Paolo Bonzini <pbonzini@redhat.com>; Marc- 
>>> André Lureau <marcandre.lureau@redhat.com>
>>> *Cc:* amir.gonnen@neuroblade.ai <amir.gonnen@neuroblade.ai>; qemu- 
>>> devel@nongnu.org <qemu-devel@nongnu.org>; Alex Bennée 
>>> <alex.bennee@linaro.org>
>>> *Subject:* Re: Building QEMU as a Shared Library
>>> Hi Saanjh,
>>> 
>>> depending what you are trying to achieve exactly, plugins can provide a
>>> solution. It's convenient and you can stay on top of QEMU upstream,
>>> without having to create a downstream fork.
>>> 
>>> We already have plugins for stopping after a given number of
>>> instructions, or slow down execution of a VM:
>>> 
>>> # stop after executing 1'000'000 instructions:
>>> $ ./build/qemu-system-x86_64 -plugin
>>> ./build/contrib/plugins/libstoptrigger,icount=1000000 -d plugin
>>> 
>>> # execute no more than 1'000'000 instructions per second:
>>> $ ./build/qemu-system-x86_64 -plugin
>>> ./build/contrib/plugins/libips.so,ips=1000000 -d plugin
>>> 
>>> You can see source code associated (./contrib/plugins/stoptrigger.c and
>>> ./contrib/plugins/ips.c), to implement something similar to what you
>>> want, but based on time.
>>> Would that satisfy your need?
>>> 
>>> Regards,
>>> Pierrick
>>> 
>>> On 3/3/25 21:53, Saanjh Sengupta wrote:
>>>> 
>>>> 
>>>> Hi,
>>>> 
>>>> Thank you so much for your inputs. I was able to create the .so file of 
>>>> QEMU.
>>>> 
>>>> Actually, what we are trying is to understand and explore possibilities 
>>>> of Virtual Time Control in QEMU. In short, what I mean to say is an 
>>>> approach via which I can tell QEMU to emulate for XYZ time when the I 
>>>> give a trigger and then pause the emulation by itself after the XYZ time 
>>>> is completed.
>>>> 
>>>> On that front itself, do you have any inputs/ideas regarding the same?
>>>> 
>>>> 
>>>> Regards
>>>> Saanjh Sengupta
>>>> ------------------------------------------------------------------------
>>>> *From:* Pierrick Bouvier <pierrick.bouvier@linaro.org>
>>>> *Sent:* Tuesday, February 25, 2025 6:29:44 AM
>>>> *To:* Philippe Mathieu-Daudé <philmd@linaro.org>; Paolo Bonzini 
>>>> <pbonzini@redhat.com>; Marc-André Lureau <marcandre.lureau@redhat.com>
>>>> *Cc:* amir.gonnen@neuroblade.ai <amir.gonnen@neuroblade.ai>; qemu- 
>>>> devel@nongnu.org <qemu-devel@nongnu.org>; Saanjh Sengupta 
>>>> <saanjhsengupta@outlook.com>
>>>> *Subject:* Re: Building QEMU as a Shared Library
>>>> Hi Saanjh,
>>>> 
>>>> here is a minimal patch that builds one shared library per target (arch,
>>>> mode) where arch is cpu arch, and mode is system or user, and launch
>>>> system-aarch64 through a simple driver:
>>>> 
>>>> https://github.com/pbo-linaro/qemu/commit/ <https://github.com/pbo- 
> linaro/qemu/commit/> <https://github.com/pbo-
>> linaro/qemu/commit/> <https://github.com/pbo-
>>> linaro/qemu/commit/>
>>>> fbb39cc64f77d4bf1e5e50795c75b62735bf5c5f <https://github.com/pbo-linaro/
>>>> qemu/commit/fbb39cc64f77d4bf1e5e50795c75b62735bf5c5f>
>>>> 
>>>> With this, it could be possible to create a driver that can execute any
>>>> existing target. It's a sort of single binary for QEMU, but shared
>>>> objects are mandatory, and duplicates all the QEMU state. So there is no
>>>> real benefit compared to having different processes.
>>>> 
>>>> In more, to be able to do concurrent emulations, there are much more
>>>> problems to be solved. QEMU state is correctly kept per target, but all
>>>> other libraries states are shared. There are various issues if you
>>>> launch two emulations at the same time in two threads:
>>>> - glib global context
>>>> - qemu calls exit in many places, which stops the whole process
>>>> - probably other things I didn't explore
>>>> 
>>>> At this point, even though qemu targets can be built as shared objects,
>>>> I would recommend to use different processes, and implement some form on
>>>> IPC to synchronize all this.
>>>> Another possibility is to try to build machines without using the
>>>> existing main, but I'm not sure it's worth all the hassle.
>>>> 
>>>> What are you trying to achieve?
>>>> 
>>>> Regards,
>>>> Pierrick
>>>> 
>>>> On 2/24/25 01:10, Philippe Mathieu-Daudé wrote:
>>>>> Cc'ing our meson experts
>>>>> 
>>>>> On 22/2/25 14:36, Saanjh Sengupta wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I referred to your mailing chains on suggesting QEMU to be built as a
>>>>>> shared library.
>>>>>>
>>>>>> *Change meson.build to build QEMU as a shared library (with PIC enabled
>>>>>> for static libraries)*
>>>>>> *
>>>>>> *
>>>>>> Could you please suggest what exactly has to be enabled in the meson.build?
>>>>>>
>>>>>> I am confused on that front.
>>>>>>
>>>>>> Regards
>>>>>> Saanjh Sengupta
>>>>> 
>>>> 
>>> 
>> 
> 
> 

Re: Building QEMU as a Shared Library

[Alex Bennée]

Saanjh Sengupta <saanjhsengupta@outlook.com> writes:

> Hi,
>
> What we are trying to achieve is that the QEMU should run for a particular number of instructions, let's say for example
> 10000 instructions and then pause it's emulation. After a resume trigger is received to the QEMU it must resume it's
> emulation and start the instruction count from 10001 (which basically
> means that the context should be saved).

I think you want to run under icount and you will need to modify the
trigger plugin. Under icount we run each vCPU in turn, so if the plugin
pauses the vCPU will de-facto be paused.

You would have to implement some sort of control interface in the
plugin. Or you could add an API to trigger the gdbstub. I think I had
that on a patch series at one point.

>
> In the previous mail when you mentioned g_usleep, I believe this shall not work (as per our use-case) since it will reset the
> instruction count to 0 (as per what you mentioned). 
>
> To achieve the use-case, do you have any leads/suggestions ? 
>
> Regards
> Saanjh Sengupta 
>
> -------------------------------------------------------------------------------------------------------------------------
> From: Pierrick Bouvier <pierrick.bouvier@linaro.org>
> Sent: Wednesday, March 12, 2025 11:50:23 am
> To: Saanjh Sengupta <saanjhsengupta@outlook.com>; Philippe Mathieu-Daudé <philmd@linaro.org>; Paolo Bonzini
> <pbonzini@redhat.com>; Marc-André Lureau <marcandre.lureau@redhat.com>
> Cc: amir.gonnen@neuroblade.ai <amir.gonnen@neuroblade.ai>; qemu-devel@nongnu.org <qemu-devel@nongnu.org>; Alex
> Bennée <alex.bennee@linaro.org>
> Subject: Re: Building QEMU as a Shared Library
>
> On 3/11/25 21:31, Saanjh Sengupta wrote:
>> 
>> 
>> Hi,
>> 
>> Thank you for the clarification. Regarding the last time
>> /"Stoptrigger might be a better fit for what you want to do, and instead 
>> of exiting, you want to resume emulation after N insn. The function 
>> qemu_clock_advance_virtual_time() can only be used to move the time 
>> forward, and you can not stop the "virtual time" by design."/
>> /
>> /
>> I did not quite understand this. Even if I have to modify the 
>> stopTrigger plugin, I would want it to pause rather than exiting.
>> For example: It gets 10000 instructions executed after that it should 
>> pause and after some time it should then resume again execute till 20000 
>> instructions (because previously it executed till 10000 and then it must 
>> execute till 20000). How do I do this? How do I state the code to pause 
>> the qemu's emulation after 10000 instructions?
>> 
>
> By using g_usleep to pause the current cpu.
> As well, it's needed to reset insn_count to 0 to count instructions again.
>
> With this command line:
> ./build/qemu-system-x86_64 -plugin 
> ./build/contrib/plugins/libstoptrigger.so,icount=1000 -d plugin
>
> And with those changes to stoptrigger:
>
> diff --git a/contrib/plugins/stoptrigger.c b/contrib/plugins/stoptrigger.c
> index b3a6ed66a7b..77fd413cef1 100644
> --- a/contrib/plugins/stoptrigger.c
> +++ b/contrib/plugins/stoptrigger.c
> @@ -41,11 +41,12 @@ typedef struct {
>       int exit_code;
>   } ExitInfo;
>
> -static void exit_emulation(int return_code, char *message)
> +static void pause_emulation(int return_code, char *message)
>   {
>       qemu_plugin_outs(message);
>       g_free(message);
> -    exit(return_code);
> +    /* exit(return_code); */
> +    g_usleep(1 * G_USEC_PER_SEC);
>   }
>
>   static void exit_icount_reached(unsigned int cpu_index, void *udata)
> @@ -53,7 +54,9 @@ static void exit_icount_reached(unsigned int 
> cpu_index, void *udata)
>       uint64_t insn_vaddr = qemu_plugin_u64_get(current_pc, cpu_index);
>       char *msg = g_strdup_printf("icount reached at 0x%" PRIx64 ", 
> exiting\n",
>                                   insn_vaddr);
> -    exit_emulation(icount_exit_code, msg);
> +    pause_emulation(icount_exit_code, msg);
> +    /* reset instruction counter */
> +    qemu_plugin_u64_set(insn_count, cpu_index, 0);
>   }
>
>   static void exit_address_reached(unsigned int cpu_index, void *udata)
> @@ -61,7 +64,7 @@ static void exit_address_reached(unsigned int 
> cpu_index, void *udata)
>       ExitInfo *ei = udata;
>       g_assert(ei);
>       char *msg = g_strdup_printf("0x%" PRIx64 " reached, exiting\n", 
> ei->exit_addr);
> -    exit_emulation(ei->exit_code, msg);
> +    pause_emulation(ei->exit_code, msg);
>   }
>
>> Moreover, I tried an activity where I was utilising the QMP protocol to 
>> control the virtual time (with respect to the IPS plugin). In that 
>> context when the QMP stop is triggered, my virtual time does got freezed 
>> until the resume is triggered. Does this mean I am able to manipulate 
>> the virtual time of the QEMU?
>> 
>
> I am not sure of how it works, but the plugin interface only allows to 
> move time forward.
>
>> 
>> 
>> Regards
>> Saanjh Sengupta
>> ------------------------------------------------------------------------
>> *From:* Pierrick Bouvier <pierrick.bouvier@linaro.org>
>> *Sent:* Wednesday, March 12, 2025 2:14:47 AM
>> *To:* Saanjh Sengupta <saanjhsengupta@outlook.com>; Philippe Mathieu- 
>> Daudé <philmd@linaro.org>; Paolo Bonzini <pbonzini@redhat.com>; Marc- 
>> André Lureau <marcandre.lureau@redhat.com>
>> *Cc:* amir.gonnen@neuroblade.ai <amir.gonnen@neuroblade.ai>; qemu- 
>> devel@nongnu.org <qemu-devel@nongnu.org>; Alex Bennée 
>> <alex.bennee@linaro.org>
>> *Subject:* Re: Building QEMU as a Shared Library
>> On 3/11/25 02:50, Saanjh Sengupta wrote:
>>> Hi,
>>> 
>>> I have a couple of questions:
>>> 
>>>  1.
>>>     When I use the libstoptrigger.so: in that case the QEMU 's emulation
>>>     stops after executing the defined number of instructions. Post this,
>>>     the whole QEMU terminates. And while using the libips.so I am
>>>     assuming that the QEMU doesn't execute no more than the defined
>>>     instructions. Please correct me if I am wrong.
>> 
>> That's correct for both plugins, with the additional note that libips
>> does this per second only.
>> 
>>>  2.
>>>     In my case, I want the QEMU to start emulation for some time and
>>>     PAUSE it's emulation for some time; after it is Paused (it's virtual
>>>     time is also to be paused) and then let's say for after 'x' time
>>>     period it should resume it's virtual time.
>>> 
>> 
>> The virtual time variable in ips plugin is only related to this plugin,
>> and based on how many instructions have been executed, which is
>> different from what you want to achieve.
>> 
>> Stoptrigger might be a better fit for what you want to do, and instead
>> of exiting, you want to resume emulation after N insn.
>> The function qemu_clock_advance_virtual_time() can only be used to move
>> the time forward, and you can not stop the "virtual time" by design.
>> 
>>> image
>>> 
>>> 
>>> I have added this segment inside the update_system_time function inside 
>>> the ipsPlugin.c. but once the instructions reach to the defined limit 
>>> the virtual time does not seem to stop.
>>> Do you have any suggestions on that front?
>>> 
>>> 
>>> Regards
>>> Saanjh Sengupta
>>> ------------------------------------------------------------------------
>>> *From:* Pierrick Bouvier <pierrick.bouvier@linaro.org>
>>> *Sent:* Wednesday, March 5, 2025 5:20:38 AM
>>> *To:* Saanjh Sengupta <saanjhsengupta@outlook.com>; Philippe Mathieu- 
>>> Daudé <philmd@linaro.org>; Paolo Bonzini <pbonzini@redhat.com>; Marc- 
>>> André Lureau <marcandre.lureau@redhat.com>
>>> *Cc:* amir.gonnen@neuroblade.ai <amir.gonnen@neuroblade.ai>; qemu- 
>>> devel@nongnu.org <qemu-devel@nongnu.org>; Alex Bennée 
>>> <alex.bennee@linaro.org>
>>> *Subject:* Re: Building QEMU as a Shared Library
>>> Hi Saanjh,
>>> 
>>> depending what you are trying to achieve exactly, plugins can provide a
>>> solution. It's convenient and you can stay on top of QEMU upstream,
>>> without having to create a downstream fork.
>>> 
>>> We already have plugins for stopping after a given number of
>>> instructions, or slow down execution of a VM:
>>> 
>>> # stop after executing 1'000'000 instructions:
>>> $ ./build/qemu-system-x86_64 -plugin
>>> ./build/contrib/plugins/libstoptrigger,icount=1000000 -d plugin
>>> 
>>> # execute no more than 1'000'000 instructions per second:
>>> $ ./build/qemu-system-x86_64 -plugin
>>> ./build/contrib/plugins/libips.so,ips=1000000 -d plugin
>>> 
>>> You can see source code associated (./contrib/plugins/stoptrigger.c and
>>> ./contrib/plugins/ips.c), to implement something similar to what you
>>> want, but based on time.
>>> Would that satisfy your need?
>>> 
>>> Regards,
>>> Pierrick
>>> 
>>> On 3/3/25 21:53, Saanjh Sengupta wrote:
>>>> 
>>>> 
>>>> Hi,
>>>> 
>>>> Thank you so much for your inputs. I was able to create the .so file of 
>>>> QEMU.
>>>> 
>>>> Actually, what we are trying is to understand and explore possibilities 
>>>> of Virtual Time Control in QEMU. In short, what I mean to say is an 
>>>> approach via which I can tell QEMU to emulate for XYZ time when the I 
>>>> give a trigger and then pause the emulation by itself after the XYZ time 
>>>> is completed.
>>>> 
>>>> On that front itself, do you have any inputs/ideas regarding the same?
>>>> 
>>>> 
>>>> Regards
>>>> Saanjh Sengupta
>>>> ------------------------------------------------------------------------
>>>> *From:* Pierrick Bouvier <pierrick.bouvier@linaro.org>
>>>> *Sent:* Tuesday, February 25, 2025 6:29:44 AM
>>>> *To:* Philippe Mathieu-Daudé <philmd@linaro.org>; Paolo Bonzini 
>>>> <pbonzini@redhat.com>; Marc-André Lureau <marcandre.lureau@redhat.com>
>>>> *Cc:* amir.gonnen@neuroblade.ai <amir.gonnen@neuroblade.ai>; qemu- 
>>>> devel@nongnu.org <qemu-devel@nongnu.org>; Saanjh Sengupta 
>>>> <saanjhsengupta@outlook.com>
>>>> *Subject:* Re: Building QEMU as a Shared Library
>>>> Hi Saanjh,
>>>> 
>>>> here is a minimal patch that builds one shared library per target (arch,
>>>> mode) where arch is cpu arch, and mode is system or user, and launch
>>>> system-aarch64 through a simple driver:
>>>> 
>>>> https://github.com/pbo-linaro/qemu/commit/ <https://github.com/pbo- 
>> linaro/qemu/commit/> <https://github.com/pbo-
>>> linaro/qemu/commit/>
>>>> fbb39cc64f77d4bf1e5e50795c75b62735bf5c5f <https://github.com/pbo-linaro/
>>>> qemu/commit/fbb39cc64f77d4bf1e5e50795c75b62735bf5c5f>
>>>> 
>>>> With this, it could be possible to create a driver that can execute any
>>>> existing target. It's a sort of single binary for QEMU, but shared
>>>> objects are mandatory, and duplicates all the QEMU state. So there is no
>>>> real benefit compared to having different processes.
>>>> 
>>>> In more, to be able to do concurrent emulations, there are much more
>>>> problems to be solved. QEMU state is correctly kept per target, but all
>>>> other libraries states are shared. There are various issues if you
>>>> launch two emulations at the same time in two threads:
>>>> - glib global context
>>>> - qemu calls exit in many places, which stops the whole process
>>>> - probably other things I didn't explore
>>>> 
>>>> At this point, even though qemu targets can be built as shared objects,
>>>> I would recommend to use different processes, and implement some form on
>>>> IPC to synchronize all this.
>>>> Another possibility is to try to build machines without using the
>>>> existing main, but I'm not sure it's worth all the hassle.
>>>> 
>>>> What are you trying to achieve?
>>>> 
>>>> Regards,
>>>> Pierrick
>>>> 
>>>> On 2/24/25 01:10, Philippe Mathieu-Daudé wrote:
>>>>> Cc'ing our meson experts
>>>>> 
>>>>> On 22/2/25 14:36, Saanjh Sengupta wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I referred to your mailing chains on suggesting QEMU to be built as a
>>>>>> shared library.
>>>>>>
>>>>>> *Change meson.build to build QEMU as a shared library (with PIC enabled
>>>>>> for static libraries)*
>>>>>> *
>>>>>> *
>>>>>> Could you please suggest what exactly has to be enabled in the meson.build?
>>>>>>
>>>>>> I am confused on that front.
>>>>>>
>>>>>> Regards
>>>>>> Saanjh Sengupta
>>>>> 
>>>> 
>>> 
>> 

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: Building QEMU as a Shared Library

[Saanjh Sengupta]

[-- Attachment #1: Type: text/plain, Size: 11565 bytes --]

Hi Alex,

You mentioned about a patch series; do you have it handy with you?

If so, could you please direct me to the same ?

On 14 Mar 2025, at 12:11 AM, Alex Bennée <alex.bennee@linaro.org> wrote:

Saanjh Sengupta <saanjhsengupta@outlook.com<mailto:saanjhsengupta@outlook.com>> writes:

Hi,

What we are trying to achieve is that the QEMU should run for a particular number of instructions, let's say for example
10000 instructions and then pause it's emulation. After a resume trigger is received to the QEMU it must resume it's
emulation and start the instruction count from 10001 (which basically
means that the context should be saved).

I think you want to run under icount and you will need to modify the
trigger plugin. Under icount we run each vCPU in turn, so if the plugin
pauses the vCPU will de-facto be paused.

You would have to implement some sort of control interface in the
plugin. Or you could add an API to trigger the gdbstub. I think I had
that on a patch series at one point.


In the previous mail when you mentioned g_usleep, I believe this shall not work (as per our use-case) since it will reset the
instruction count to 0 (as per what you mentioned).

To achieve the use-case, do you have any leads/suggestions ?

Regards
Saanjh Sengupta

-------------------------------------------------------------------------------------------------------------------------
From: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Sent: Wednesday, March 12, 2025 11:50:23 am
To: Saanjh Sengupta <saanjhsengupta@outlook.com>; Philippe Mathieu-Daudé <philmd@linaro.org>; Paolo Bonzini
<pbonzini@redhat.com>; Marc-André Lureau <marcandre.lureau@redhat.com>
Cc: amir.gonnen@neuroblade.ai <amir.gonnen@neuroblade.ai>; qemu-devel@nongnu.org <qemu-devel@nongnu.org>; Alex
Bennée <alex.bennee@linaro.org>
Subject: Re: Building QEMU as a Shared Library

On 3/11/25 21:31, Saanjh Sengupta wrote:


Hi,

Thank you for the clarification. Regarding the last time
/"Stoptrigger might be a better fit for what you want to do, and instead
of exiting, you want to resume emulation after N insn. The function
qemu_clock_advance_virtual_time() can only be used to move the time
forward, and you can not stop the "virtual time" by design."/
/
/
I did not quite understand this. Even if I have to modify the
stopTrigger plugin, I would want it to pause rather than exiting.
For example: It gets 10000 instructions executed after that it should
pause and after some time it should then resume again execute till 20000
instructions (because previously it executed till 10000 and then it must
execute till 20000). How do I do this? How do I state the code to pause
the qemu's emulation after 10000 instructions?


By using g_usleep to pause the current cpu.
As well, it's needed to reset insn_count to 0 to count instructions again.

With this command line:
./build/qemu-system-x86_64 -plugin
./build/contrib/plugins/libstoptrigger.so,icount=1000 -d plugin

And with those changes to stoptrigger:

diff --git a/contrib/plugins/stoptrigger.c b/contrib/plugins/stoptrigger.c
index b3a6ed66a7b..77fd413cef1 100644
--- a/contrib/plugins/stoptrigger.c
+++ b/contrib/plugins/stoptrigger.c
@@ -41,11 +41,12 @@ typedef struct {
     int exit_code;
 } ExitInfo;

-static void exit_emulation(int return_code, char *message)
+static void pause_emulation(int return_code, char *message)
 {
     qemu_plugin_outs(message);
     g_free(message);
-    exit(return_code);
+    /* exit(return_code); */
+    g_usleep(1 * G_USEC_PER_SEC);
 }

 static void exit_icount_reached(unsigned int cpu_index, void *udata)
@@ -53,7 +54,9 @@ static void exit_icount_reached(unsigned int
cpu_index, void *udata)
     uint64_t insn_vaddr = qemu_plugin_u64_get(current_pc, cpu_index);
     char *msg = g_strdup_printf("icount reached at 0x%" PRIx64 ",
exiting\n",
                                 insn_vaddr);
-    exit_emulation(icount_exit_code, msg);
+    pause_emulation(icount_exit_code, msg);
+    /* reset instruction counter */
+    qemu_plugin_u64_set(insn_count, cpu_index, 0);
 }

 static void exit_address_reached(unsigned int cpu_index, void *udata)
@@ -61,7 +64,7 @@ static void exit_address_reached(unsigned int
cpu_index, void *udata)
     ExitInfo *ei = udata;
     g_assert(ei);
     char *msg = g_strdup_printf("0x%" PRIx64 " reached, exiting\n",
ei->exit_addr);
-    exit_emulation(ei->exit_code, msg);
+    pause_emulation(ei->exit_code, msg);
 }

Moreover, I tried an activity where I was utilising the QMP protocol to
control the virtual time (with respect to the IPS plugin). In that
context when the QMP stop is triggered, my virtual time does got freezed
until the resume is triggered. Does this mean I am able to manipulate
the virtual time of the QEMU?


I am not sure of how it works, but the plugin interface only allows to
move time forward.



Regards
Saanjh Sengupta
------------------------------------------------------------------------
*From:* Pierrick Bouvier <pierrick.bouvier@linaro.org>
*Sent:* Wednesday, March 12, 2025 2:14:47 AM
*To:* Saanjh Sengupta <saanjhsengupta@outlook.com>; Philippe Mathieu-
Daudé <philmd@linaro.org>; Paolo Bonzini <pbonzini@redhat.com>; Marc-
André Lureau <marcandre.lureau@redhat.com>
*Cc:* amir.gonnen@neuroblade.ai <amir.gonnen@neuroblade.ai>; qemu-
devel@nongnu.org <qemu-devel@nongnu.org>; Alex Bennée
<alex.bennee@linaro.org>
*Subject:* Re: Building QEMU as a Shared Library
On 3/11/25 02:50, Saanjh Sengupta wrote:
Hi,

I have a couple of questions:

1.
   When I use the libstoptrigger.so: in that case the QEMU 's emulation
   stops after executing the defined number of instructions. Post this,
   the whole QEMU terminates. And while using the libips.so I am
   assuming that the QEMU doesn't execute no more than the defined
   instructions. Please correct me if I am wrong.

That's correct for both plugins, with the additional note that libips
does this per second only.

2.
   In my case, I want the QEMU to start emulation for some time and
   PAUSE it's emulation for some time; after it is Paused (it's virtual
   time is also to be paused) and then let's say for after 'x' time
   period it should resume it's virtual time.


The virtual time variable in ips plugin is only related to this plugin,
and based on how many instructions have been executed, which is
different from what you want to achieve.

Stoptrigger might be a better fit for what you want to do, and instead
of exiting, you want to resume emulation after N insn.
The function qemu_clock_advance_virtual_time() can only be used to move
the time forward, and you can not stop the "virtual time" by design.

image


I have added this segment inside the update_system_time function inside
the ipsPlugin.c. but once the instructions reach to the defined limit
the virtual time does not seem to stop.
Do you have any suggestions on that front?


Regards
Saanjh Sengupta
------------------------------------------------------------------------
*From:* Pierrick Bouvier <pierrick.bouvier@linaro.org>
*Sent:* Wednesday, March 5, 2025 5:20:38 AM
*To:* Saanjh Sengupta <saanjhsengupta@outlook.com>; Philippe Mathieu-
Daudé <philmd@linaro.org>; Paolo Bonzini <pbonzini@redhat.com>; Marc-
André Lureau <marcandre.lureau@redhat.com>
*Cc:* amir.gonnen@neuroblade.ai <amir.gonnen@neuroblade.ai>; qemu-
devel@nongnu.org <qemu-devel@nongnu.org>; Alex Bennée
<alex.bennee@linaro.org>
*Subject:* Re: Building QEMU as a Shared Library
Hi Saanjh,

depending what you are trying to achieve exactly, plugins can provide a
solution. It's convenient and you can stay on top of QEMU upstream,
without having to create a downstream fork.

We already have plugins for stopping after a given number of
instructions, or slow down execution of a VM:

# stop after executing 1'000'000 instructions:
$ ./build/qemu-system-x86_64 -plugin
./build/contrib/plugins/libstoptrigger,icount=1000000 -d plugin

# execute no more than 1'000'000 instructions per second:
$ ./build/qemu-system-x86_64 -plugin
./build/contrib/plugins/libips.so,ips=1000000 -d plugin

You can see source code associated (./contrib/plugins/stoptrigger.c and
./contrib/plugins/ips.c), to implement something similar to what you
want, but based on time.
Would that satisfy your need?

Regards,
Pierrick

On 3/3/25 21:53, Saanjh Sengupta wrote:


Hi,

Thank you so much for your inputs. I was able to create the .so file of
QEMU.

Actually, what we are trying is to understand and explore possibilities
of Virtual Time Control in QEMU. In short, what I mean to say is an
approach via which I can tell QEMU to emulate for XYZ time when the I
give a trigger and then pause the emulation by itself after the XYZ time
is completed.

On that front itself, do you have any inputs/ideas regarding the same?


Regards
Saanjh Sengupta
------------------------------------------------------------------------
*From:* Pierrick Bouvier <pierrick.bouvier@linaro.org>
*Sent:* Tuesday, February 25, 2025 6:29:44 AM
*To:* Philippe Mathieu-Daudé <philmd@linaro.org>; Paolo Bonzini
<pbonzini@redhat.com>; Marc-André Lureau <marcandre.lureau@redhat.com>
*Cc:* amir.gonnen@neuroblade.ai <amir.gonnen@neuroblade.ai>; qemu-
devel@nongnu.org <qemu-devel@nongnu.org>; Saanjh Sengupta
<saanjhsengupta@outlook.com>
*Subject:* Re: Building QEMU as a Shared Library
Hi Saanjh,

here is a minimal patch that builds one shared library per target (arch,
mode) where arch is cpu arch, and mode is system or user, and launch
system-aarch64 through a simple driver:

https://github.com/pbo-linaro/qemu/commit/ <https://github.com/pbo-
linaro/qemu/commit/> <https://github.com/pbo-
linaro/qemu/commit/>
fbb39cc64f77d4bf1e5e50795c75b62735bf5c5f <https://github.com/pbo-linaro/
qemu/commit/fbb39cc64f77d4bf1e5e50795c75b62735bf5c5f>

With this, it could be possible to create a driver that can execute any
existing target. It's a sort of single binary for QEMU, but shared
objects are mandatory, and duplicates all the QEMU state. So there is no
real benefit compared to having different processes.

In more, to be able to do concurrent emulations, there are much more
problems to be solved. QEMU state is correctly kept per target, but all
other libraries states are shared. There are various issues if you
launch two emulations at the same time in two threads:
- glib global context
- qemu calls exit in many places, which stops the whole process
- probably other things I didn't explore

At this point, even though qemu targets can be built as shared objects,
I would recommend to use different processes, and implement some form on
IPC to synchronize all this.
Another possibility is to try to build machines without using the
existing main, but I'm not sure it's worth all the hassle.

What are you trying to achieve?

Regards,
Pierrick

On 2/24/25 01:10, Philippe Mathieu-Daudé wrote:
Cc'ing our meson experts

On 22/2/25 14:36, Saanjh Sengupta wrote:
Hi,

I referred to your mailing chains on suggesting QEMU to be built as a
shared library.

*Change meson.build to build QEMU as a shared library (with PIC enabled
for static libraries)*
*
*
Could you please suggest what exactly has to be enabled in the meson.build?

I am confused on that front.

Regards
Saanjh Sengupta





--
Alex Bennée
Virtualisation Tech Lead @ Linaro


[-- Attachment #2: Type: text/html, Size: 26221 bytes --]

Re: Building QEMU as a Shared Library

[Alex Bennée]

Saanjh Sengupta <saanjhsengupta@outlook.com> writes:

> Hi Alex,  
>
> You mentioned about a patch series; do you have it handy with you?

  Message-Id: <20241022105614.839199-18-alex.bennee@linaro.org>
  Date: Tue, 22 Oct 2024 11:56:11 +0100
  Subject: [PATCH v2 17/20] plugins: add ability to register a GDB triggered callback
  From: =?UTF-8?q?Alex=20Benn=C3=A9e?= <alex.bennee@linaro.org>

>
> If so, could you please direct me to the same ?
>
>  On 14 Mar 2025, at 12:11 AM, Alex Bennée <alex.bennee@linaro.org> wrote:
>
>  Saanjh Sengupta <saanjhsengupta@outlook.com> writes:
>
>  Hi,
>
>  What we are trying to achieve is that the QEMU should run for a particular number of instructions, let's say for
>  example
>  10000 instructions and then pause it's emulation. After a resume trigger is received to the QEMU it must resume it's
>  emulation and start the instruction count from 10001 (which basically
>  means that the context should be saved).
>
>  I think you want to run under icount and you will need to modify the
>  trigger plugin. Under icount we run each vCPU in turn, so if the plugin
>  pauses the vCPU will de-facto be paused.
>
>  You would have to implement some sort of control interface in the
>  plugin. Or you could add an API to trigger the gdbstub. I think I had
>  that on a patch series at one point.
>
>  In the previous mail when you mentioned g_usleep, I believe this shall not work (as per our use-case) since it will
>  reset the
>  instruction count to 0 (as per what you mentioned). 
>
>  To achieve the use-case, do you have any leads/suggestions ? 
>
>  Regards
>  Saanjh Sengupta 
>
>  -------------------------------------------------------------------------------------------------------------------------
>  From: Pierrick Bouvier <pierrick.bouvier@linaro.org>
>  Sent: Wednesday, March 12, 2025 11:50:23 am
>  To: Saanjh Sengupta <saanjhsengupta@outlook.com>; Philippe Mathieu-Daudé <philmd@linaro.org>; Paolo Bonzini
>  <pbonzini@redhat.com>; Marc-André Lureau <marcandre.lureau@redhat.com>
>  Cc: amir.gonnen@neuroblade.ai <amir.gonnen@neuroblade.ai>; qemu-devel@nongnu.org
>  <qemu-devel@nongnu.org>; Alex
>  Bennée <alex.bennee@linaro.org>
>  Subject: Re: Building QEMU as a Shared Library
>
>  On 3/11/25 21:31, Saanjh Sengupta wrote:
>
>  Hi,
>
>  Thank you for the clarification. Regarding the last time
>  /"Stoptrigger might be a better fit for what you want to do, and instead 
>  of exiting, you want to resume emulation after N insn. The function 
>  qemu_clock_advance_virtual_time() can only be used to move the time 
>  forward, and you can not stop the "virtual time" by design."/
>  /
>  /
>  I did not quite understand this. Even if I have to modify the 
>  stopTrigger plugin, I would want it to pause rather than exiting.
>  For example: It gets 10000 instructions executed after that it should 
>  pause and after some time it should then resume again execute till 20000 
>  instructions (because previously it executed till 10000 and then it must 
>  execute till 20000). How do I do this? How do I state the code to pause 
>  the qemu's emulation after 10000 instructions?
>
>  By using g_usleep to pause the current cpu.
>  As well, it's needed to reset insn_count to 0 to count instructions again.
>
>  With this command line:
>  ./build/qemu-system-x86_64 -plugin 
>  ./build/contrib/plugins/libstoptrigger.so,icount=1000 -d plugin
>
>  And with those changes to stoptrigger:
>
>  diff --git a/contrib/plugins/stoptrigger.c b/contrib/plugins/stoptrigger.c
>  index b3a6ed66a7b..77fd413cef1 100644
>  --- a/contrib/plugins/stoptrigger.c
>  +++ b/contrib/plugins/stoptrigger.c
>  @@ -41,11 +41,12 @@ typedef struct {
>       int exit_code;
>   } ExitInfo;
>
>  -static void exit_emulation(int return_code, char *message)
>  +static void pause_emulation(int return_code, char *message)
>   {
>       qemu_plugin_outs(message);
>       g_free(message);
>  -    exit(return_code);
>  +    /* exit(return_code); */
>  +    g_usleep(1 * G_USEC_PER_SEC);
>   }
>
>   static void exit_icount_reached(unsigned int cpu_index, void *udata)
>  @@ -53,7 +54,9 @@ static void exit_icount_reached(unsigned int 
>  cpu_index, void *udata)
>       uint64_t insn_vaddr = qemu_plugin_u64_get(current_pc, cpu_index);
>       char *msg = g_strdup_printf("icount reached at 0x%" PRIx64 ", 
>  exiting\n",
>                                   insn_vaddr);
>  -    exit_emulation(icount_exit_code, msg);
>  +    pause_emulation(icount_exit_code, msg);
>  +    /* reset instruction counter */
>  +    qemu_plugin_u64_set(insn_count, cpu_index, 0);
>   }
>
>   static void exit_address_reached(unsigned int cpu_index, void *udata)
>  @@ -61,7 +64,7 @@ static void exit_address_reached(unsigned int 
>  cpu_index, void *udata)
>       ExitInfo *ei = udata;
>       g_assert(ei);
>       char *msg = g_strdup_printf("0x%" PRIx64 " reached, exiting\n", 
>  ei->exit_addr);
>  -    exit_emulation(ei->exit_code, msg);
>  +    pause_emulation(ei->exit_code, msg);
>   }
>
>  Moreover, I tried an activity where I was utilising the QMP protocol to 
>  control the virtual time (with respect to the IPS plugin). In that 
>  context when the QMP stop is triggered, my virtual time does got freezed 
>  until the resume is triggered. Does this mean I am able to manipulate 
>  the virtual time of the QEMU?
>
>  I am not sure of how it works, but the plugin interface only allows to 
>  move time forward.
>
>  Regards
>  Saanjh Sengupta
>  ------------------------------------------------------------------------
>  *From:* Pierrick Bouvier <pierrick.bouvier@linaro.org>
>  *Sent:* Wednesday, March 12, 2025 2:14:47 AM
>  *To:* Saanjh Sengupta <saanjhsengupta@outlook.com>; Philippe Mathieu- 
>  Daudé <philmd@linaro.org>; Paolo Bonzini <pbonzini@redhat.com>; Marc- 
>  André Lureau <marcandre.lureau@redhat.com>
>  *Cc:* amir.gonnen@neuroblade.ai <amir.gonnen@neuroblade.ai>; qemu- 
>  devel@nongnu.org <qemu-devel@nongnu.org>; Alex Bennée 
>  <alex.bennee@linaro.org>
>  *Subject:* Re: Building QEMU as a Shared Library
>  On 3/11/25 02:50, Saanjh Sengupta wrote:
>
>  Hi,
>
>  I have a couple of questions:
>
>  1.
>     When I use the libstoptrigger.so: in that case the QEMU 's emulation
>     stops after executing the defined number of instructions. Post this,
>     the whole QEMU terminates. And while using the libips.so I am
>     assuming that the QEMU doesn't execute no more than the defined
>     instructions. Please correct me if I am wrong.
>
>  That's correct for both plugins, with the additional note that libips
>  does this per second only.
>
>  2.
>     In my case, I want the QEMU to start emulation for some time and
>     PAUSE it's emulation for some time; after it is Paused (it's virtual
>     time is also to be paused) and then let's say for after 'x' time
>     period it should resume it's virtual time.
>
>  The virtual time variable in ips plugin is only related to this plugin,
>  and based on how many instructions have been executed, which is
>  different from what you want to achieve.
>
>  Stoptrigger might be a better fit for what you want to do, and instead
>  of exiting, you want to resume emulation after N insn.
>  The function qemu_clock_advance_virtual_time() can only be used to move
>  the time forward, and you can not stop the "virtual time" by design.
>
>  image
>
>  I have added this segment inside the update_system_time function inside 
>  the ipsPlugin.c. but once the instructions reach to the defined limit 
>  the virtual time does not seem to stop.
>  Do you have any suggestions on that front?
>
>  Regards
>  Saanjh Sengupta
>  ------------------------------------------------------------------------
>  *From:* Pierrick Bouvier <pierrick.bouvier@linaro.org>
>  *Sent:* Wednesday, March 5, 2025 5:20:38 AM
>  *To:* Saanjh Sengupta <saanjhsengupta@outlook.com>; Philippe Mathieu- 
>  Daudé <philmd@linaro.org>; Paolo Bonzini <pbonzini@redhat.com>; Marc- 
>  André Lureau <marcandre.lureau@redhat.com>
>  *Cc:* amir.gonnen@neuroblade.ai <amir.gonnen@neuroblade.ai>; qemu- 
>  devel@nongnu.org <qemu-devel@nongnu.org>; Alex Bennée 
>  <alex.bennee@linaro.org>
>  *Subject:* Re: Building QEMU as a Shared Library
>  Hi Saanjh,
>
>  depending what you are trying to achieve exactly, plugins can provide a
>  solution. It's convenient and you can stay on top of QEMU upstream,
>  without having to create a downstream fork.
>
>  We already have plugins for stopping after a given number of
>  instructions, or slow down execution of a VM:
>
>  # stop after executing 1'000'000 instructions:
>  $ ./build/qemu-system-x86_64 -plugin
>  ./build/contrib/plugins/libstoptrigger,icount=1000000 -d plugin
>
>  # execute no more than 1'000'000 instructions per second:
>  $ ./build/qemu-system-x86_64 -plugin
>  ./build/contrib/plugins/libips.so,ips=1000000 -d plugin
>
>  You can see source code associated (./contrib/plugins/stoptrigger.c and
>  ./contrib/plugins/ips.c), to implement something similar to what you
>  want, but based on time.
>  Would that satisfy your need?
>
>  Regards,
>  Pierrick
>
>  On 3/3/25 21:53, Saanjh Sengupta wrote:
>
>  Hi,
>
>  Thank you so much for your inputs. I was able to create the .so file of 
>  QEMU.
>
>  Actually, what we are trying is to understand and explore possibilities 
>  of Virtual Time Control in QEMU. In short, what I mean to say is an 
>  approach via which I can tell QEMU to emulate for XYZ time when the I 
>  give a trigger and then pause the emulation by itself after the XYZ time 
>  is completed.
>
>  On that front itself, do you have any inputs/ideas regarding the same?
>
>  Regards
>  Saanjh Sengupta
>  ------------------------------------------------------------------------
>  *From:* Pierrick Bouvier <pierrick.bouvier@linaro.org>
>  *Sent:* Tuesday, February 25, 2025 6:29:44 AM
>  *To:* Philippe Mathieu-Daudé <philmd@linaro.org>; Paolo Bonzini 
>  <pbonzini@redhat.com>; Marc-André Lureau <marcandre.lureau@redhat.com>
>  *Cc:* amir.gonnen@neuroblade.ai <amir.gonnen@neuroblade.ai>; qemu- 
>  devel@nongnu.org <qemu-devel@nongnu.org>; Saanjh Sengupta 
>  <saanjhsengupta@outlook.com>
>  *Subject:* Re: Building QEMU as a Shared Library
>  Hi Saanjh,
>
>  here is a minimal patch that builds one shared library per target (arch,
>  mode) where arch is cpu arch, and mode is system or user, and launch
>  system-aarch64 through a simple driver:
>
>  https://github.com/pbo-linaro/qemu/commit/ <https://github.com/pbo- 
>
>  linaro/qemu/commit/> <https://github.com/pbo-
>
>  linaro/qemu/commit/>
>
>  fbb39cc64f77d4bf1e5e50795c75b62735bf5c5f <https://github.com/pbo-linaro/
>  qemu/commit/fbb39cc64f77d4bf1e5e50795c75b62735bf5c5f>
>
>  With this, it could be possible to create a driver that can execute any
>  existing target. It's a sort of single binary for QEMU, but shared
>  objects are mandatory, and duplicates all the QEMU state. So there is no
>  real benefit compared to having different processes.
>
>  In more, to be able to do concurrent emulations, there are much more
>  problems to be solved. QEMU state is correctly kept per target, but all
>  other libraries states are shared. There are various issues if you
>  launch two emulations at the same time in two threads:
>  - glib global context
>  - qemu calls exit in many places, which stops the whole process
>  - probably other things I didn't explore
>
>  At this point, even though qemu targets can be built as shared objects,
>  I would recommend to use different processes, and implement some form on
>  IPC to synchronize all this.
>  Another possibility is to try to build machines without using the
>  existing main, but I'm not sure it's worth all the hassle.
>
>  What are you trying to achieve?
>
>  Regards,
>  Pierrick
>
>  On 2/24/25 01:10, Philippe Mathieu-Daudé wrote:
>
>  Cc'ing our meson experts
>
>  On 22/2/25 14:36, Saanjh Sengupta wrote:
>
>  Hi,
>
>  I referred to your mailing chains on suggesting QEMU to be built as a
>  shared library.
>
>  *Change meson.build to build QEMU as a shared library (with PIC enabled
>  for static libraries)*
>  *
>  *
>  Could you please suggest what exactly has to be enabled in the meson.build?
>
>  I am confused on that front.
>
>  Regards
>  Saanjh Sengupta
>
>  -- 
>  Alex Bennée
>  Virtualisation Tech Lead @ Linaro

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: Building QEMU as a Shared Library

[Saanjh Sengupta]

[-- Attachment #1: Type: text/plain, Size: 13281 bytes --]



Hi,

Unable to view this. Any GitHub repository diff page I could view for the same?

Regards
Saanjh Sengupta
________________________________
From: Alex Bennée <alex.bennee@linaro.org>
Sent: Tuesday, March 18, 2025 1:48:01 PM
To: Saanjh Sengupta <saanjhsengupta@outlook.com>
Cc: Pierrick Bouvier <pierrick.bouvier@linaro.org>; Philippe Mathieu-Daudé <philmd@linaro.org>; Paolo Bonzini <pbonzini@redhat.com>; Marc-André Lureau <marcandre.lureau@redhat.com>; amir.gonnen@neuroblade.ai <amir.gonnen@neuroblade.ai>; qemu-devel@nongnu.org <qemu-devel@nongnu.org>
Subject: Re: Building QEMU as a Shared Library

Saanjh Sengupta <saanjhsengupta@outlook.com> writes:

> Hi Alex,
>
> You mentioned about a patch series; do you have it handy with you?

  Message-Id: <20241022105614.839199-18-alex.bennee@linaro.org>
  Date: Tue, 22 Oct 2024 11:56:11 +0100
  Subject: [PATCH v2 17/20] plugins: add ability to register a GDB triggered callback
  From: =?UTF-8?q?Alex=20Benn=C3=A9e?= <alex.bennee@linaro.org>

>
> If so, could you please direct me to the same ?
>
>  On 14 Mar 2025, at 12:11 AM, Alex Bennée <alex.bennee@linaro.org> wrote:
>
>  Saanjh Sengupta <saanjhsengupta@outlook.com> writes:
>
>  Hi,
>
>  What we are trying to achieve is that the QEMU should run for a particular number of instructions, let's say for
>  example
>  10000 instructions and then pause it's emulation. After a resume trigger is received to the QEMU it must resume it's
>  emulation and start the instruction count from 10001 (which basically
>  means that the context should be saved).
>
>  I think you want to run under icount and you will need to modify the
>  trigger plugin. Under icount we run each vCPU in turn, so if the plugin
>  pauses the vCPU will de-facto be paused.
>
>  You would have to implement some sort of control interface in the
>  plugin. Or you could add an API to trigger the gdbstub. I think I had
>  that on a patch series at one point.
>
>  In the previous mail when you mentioned g_usleep, I believe this shall not work (as per our use-case) since it will
>  reset the
>  instruction count to 0 (as per what you mentioned).
>
>  To achieve the use-case, do you have any leads/suggestions ?
>
>  Regards
>  Saanjh Sengupta
>
>  -------------------------------------------------------------------------------------------------------------------------
>  From: Pierrick Bouvier <pierrick.bouvier@linaro.org>
>  Sent: Wednesday, March 12, 2025 11:50:23 am
>  To: Saanjh Sengupta <saanjhsengupta@outlook.com>; Philippe Mathieu-Daudé <philmd@linaro.org>; Paolo Bonzini
>  <pbonzini@redhat.com>; Marc-André Lureau <marcandre.lureau@redhat.com>
>  Cc: amir.gonnen@neuroblade.ai <amir.gonnen@neuroblade.ai>; qemu-devel@nongnu.org
>  <qemu-devel@nongnu.org>; Alex
>  Bennée <alex.bennee@linaro.org>
>  Subject: Re: Building QEMU as a Shared Library
>
>  On 3/11/25 21:31, Saanjh Sengupta wrote:
>
>  Hi,
>
>  Thank you for the clarification. Regarding the last time
>  /"Stoptrigger might be a better fit for what you want to do, and instead
>  of exiting, you want to resume emulation after N insn. The function
>  qemu_clock_advance_virtual_time() can only be used to move the time
>  forward, and you can not stop the "virtual time" by design."/
>  /
>  /
>  I did not quite understand this. Even if I have to modify the
>  stopTrigger plugin, I would want it to pause rather than exiting.
>  For example: It gets 10000 instructions executed after that it should
>  pause and after some time it should then resume again execute till 20000
>  instructions (because previously it executed till 10000 and then it must
>  execute till 20000). How do I do this? How do I state the code to pause
>  the qemu's emulation after 10000 instructions?
>
>  By using g_usleep to pause the current cpu.
>  As well, it's needed to reset insn_count to 0 to count instructions again.
>
>  With this command line:
>  ./build/qemu-system-x86_64 -plugin
>  ./build/contrib/plugins/libstoptrigger.so,icount=1000 -d plugin
>
>  And with those changes to stoptrigger:
>
>  diff --git a/contrib/plugins/stoptrigger.c b/contrib/plugins/stoptrigger.c
>  index b3a6ed66a7b..77fd413cef1 100644
>  --- a/contrib/plugins/stoptrigger.c
>  +++ b/contrib/plugins/stoptrigger.c
>  @@ -41,11 +41,12 @@ typedef struct {
>       int exit_code;
>   } ExitInfo;
>
>  -static void exit_emulation(int return_code, char *message)
>  +static void pause_emulation(int return_code, char *message)
>   {
>       qemu_plugin_outs(message);
>       g_free(message);
>  -    exit(return_code);
>  +    /* exit(return_code); */
>  +    g_usleep(1 * G_USEC_PER_SEC);
>   }
>
>   static void exit_icount_reached(unsigned int cpu_index, void *udata)
>  @@ -53,7 +54,9 @@ static void exit_icount_reached(unsigned int
>  cpu_index, void *udata)
>       uint64_t insn_vaddr = qemu_plugin_u64_get(current_pc, cpu_index);
>       char *msg = g_strdup_printf("icount reached at 0x%" PRIx64 ",
>  exiting\n",
>                                   insn_vaddr);
>  -    exit_emulation(icount_exit_code, msg);
>  +    pause_emulation(icount_exit_code, msg);
>  +    /* reset instruction counter */
>  +    qemu_plugin_u64_set(insn_count, cpu_index, 0);
>   }
>
>   static void exit_address_reached(unsigned int cpu_index, void *udata)
>  @@ -61,7 +64,7 @@ static void exit_address_reached(unsigned int
>  cpu_index, void *udata)
>       ExitInfo *ei = udata;
>       g_assert(ei);
>       char *msg = g_strdup_printf("0x%" PRIx64 " reached, exiting\n",
>  ei->exit_addr);
>  -    exit_emulation(ei->exit_code, msg);
>  +    pause_emulation(ei->exit_code, msg);
>   }
>
>  Moreover, I tried an activity where I was utilising the QMP protocol to
>  control the virtual time (with respect to the IPS plugin). In that
>  context when the QMP stop is triggered, my virtual time does got freezed
>  until the resume is triggered. Does this mean I am able to manipulate
>  the virtual time of the QEMU?
>
>  I am not sure of how it works, but the plugin interface only allows to
>  move time forward.
>
>  Regards
>  Saanjh Sengupta
>  ------------------------------------------------------------------------
>  *From:* Pierrick Bouvier <pierrick.bouvier@linaro.org>
>  *Sent:* Wednesday, March 12, 2025 2:14:47 AM
>  *To:* Saanjh Sengupta <saanjhsengupta@outlook.com>; Philippe Mathieu-
>  Daudé <philmd@linaro.org>; Paolo Bonzini <pbonzini@redhat.com>; Marc-
>  André Lureau <marcandre.lureau@redhat.com>
>  *Cc:* amir.gonnen@neuroblade.ai <amir.gonnen@neuroblade.ai>; qemu-
>  devel@nongnu.org <qemu-devel@nongnu.org>; Alex Bennée
>  <alex.bennee@linaro.org>
>  *Subject:* Re: Building QEMU as a Shared Library
>  On 3/11/25 02:50, Saanjh Sengupta wrote:
>
>  Hi,
>
>  I have a couple of questions:
>
>  1.
>     When I use the libstoptrigger.so: in that case the QEMU 's emulation
>     stops after executing the defined number of instructions. Post this,
>     the whole QEMU terminates. And while using the libips.so I am
>     assuming that the QEMU doesn't execute no more than the defined
>     instructions. Please correct me if I am wrong.
>
>  That's correct for both plugins, with the additional note that libips
>  does this per second only.
>
>  2.
>     In my case, I want the QEMU to start emulation for some time and
>     PAUSE it's emulation for some time; after it is Paused (it's virtual
>     time is also to be paused) and then let's say for after 'x' time
>     period it should resume it's virtual time.
>
>  The virtual time variable in ips plugin is only related to this plugin,
>  and based on how many instructions have been executed, which is
>  different from what you want to achieve.
>
>  Stoptrigger might be a better fit for what you want to do, and instead
>  of exiting, you want to resume emulation after N insn.
>  The function qemu_clock_advance_virtual_time() can only be used to move
>  the time forward, and you can not stop the "virtual time" by design.
>
>  image
>
>  I have added this segment inside the update_system_time function inside
>  the ipsPlugin.c. but once the instructions reach to the defined limit
>  the virtual time does not seem to stop.
>  Do you have any suggestions on that front?
>
>  Regards
>  Saanjh Sengupta
>  ------------------------------------------------------------------------
>  *From:* Pierrick Bouvier <pierrick.bouvier@linaro.org>
>  *Sent:* Wednesday, March 5, 2025 5:20:38 AM
>  *To:* Saanjh Sengupta <saanjhsengupta@outlook.com>; Philippe Mathieu-
>  Daudé <philmd@linaro.org>; Paolo Bonzini <pbonzini@redhat.com>; Marc-
>  André Lureau <marcandre.lureau@redhat.com>
>  *Cc:* amir.gonnen@neuroblade.ai <amir.gonnen@neuroblade.ai>; qemu-
>  devel@nongnu.org <qemu-devel@nongnu.org>; Alex Bennée
>  <alex.bennee@linaro.org>
>  *Subject:* Re: Building QEMU as a Shared Library
>  Hi Saanjh,
>
>  depending what you are trying to achieve exactly, plugins can provide a
>  solution. It's convenient and you can stay on top of QEMU upstream,
>  without having to create a downstream fork.
>
>  We already have plugins for stopping after a given number of
>  instructions, or slow down execution of a VM:
>
>  # stop after executing 1'000'000 instructions:
>  $ ./build/qemu-system-x86_64 -plugin
>  ./build/contrib/plugins/libstoptrigger,icount=1000000 -d plugin
>
>  # execute no more than 1'000'000 instructions per second:
>  $ ./build/qemu-system-x86_64 -plugin
>  ./build/contrib/plugins/libips.so,ips=1000000 -d plugin
>
>  You can see source code associated (./contrib/plugins/stoptrigger.c and
>  ./contrib/plugins/ips.c), to implement something similar to what you
>  want, but based on time.
>  Would that satisfy your need?
>
>  Regards,
>  Pierrick
>
>  On 3/3/25 21:53, Saanjh Sengupta wrote:
>
>  Hi,
>
>  Thank you so much for your inputs. I was able to create the .so file of
>  QEMU.
>
>  Actually, what we are trying is to understand and explore possibilities
>  of Virtual Time Control in QEMU. In short, what I mean to say is an
>  approach via which I can tell QEMU to emulate for XYZ time when the I
>  give a trigger and then pause the emulation by itself after the XYZ time
>  is completed.
>
>  On that front itself, do you have any inputs/ideas regarding the same?
>
>  Regards
>  Saanjh Sengupta
>  ------------------------------------------------------------------------
>  *From:* Pierrick Bouvier <pierrick.bouvier@linaro.org>
>  *Sent:* Tuesday, February 25, 2025 6:29:44 AM
>  *To:* Philippe Mathieu-Daudé <philmd@linaro.org>; Paolo Bonzini
>  <pbonzini@redhat.com>; Marc-André Lureau <marcandre.lureau@redhat.com>
>  *Cc:* amir.gonnen@neuroblade.ai <amir.gonnen@neuroblade.ai>; qemu-
>  devel@nongnu.org <qemu-devel@nongnu.org>; Saanjh Sengupta
>  <saanjhsengupta@outlook.com>
>  *Subject:* Re: Building QEMU as a Shared Library
>  Hi Saanjh,
>
>  here is a minimal patch that builds one shared library per target (arch,
>  mode) where arch is cpu arch, and mode is system or user, and launch
>  system-aarch64 through a simple driver:
>
>  https://github.com/pbo-linaro/qemu/commit/ <https://github.com/pbo-
>
>  linaro/qemu/commit/> <https://github.com/pbo-
>
>  linaro/qemu/commit/>
>
>  fbb39cc64f77d4bf1e5e50795c75b62735bf5c5f <https://github.com/pbo-linaro/
>  qemu/commit/fbb39cc64f77d4bf1e5e50795c75b62735bf5c5f>
>
>  With this, it could be possible to create a driver that can execute any
>  existing target. It's a sort of single binary for QEMU, but shared
>  objects are mandatory, and duplicates all the QEMU state. So there is no
>  real benefit compared to having different processes.
>
>  In more, to be able to do concurrent emulations, there are much more
>  problems to be solved. QEMU state is correctly kept per target, but all
>  other libraries states are shared. There are various issues if you
>  launch two emulations at the same time in two threads:
>  - glib global context
>  - qemu calls exit in many places, which stops the whole process
>  - probably other things I didn't explore
>
>  At this point, even though qemu targets can be built as shared objects,
>  I would recommend to use different processes, and implement some form on
>  IPC to synchronize all this.
>  Another possibility is to try to build machines without using the
>  existing main, but I'm not sure it's worth all the hassle.
>
>  What are you trying to achieve?
>
>  Regards,
>  Pierrick
>
>  On 2/24/25 01:10, Philippe Mathieu-Daudé wrote:
>
>  Cc'ing our meson experts
>
>  On 22/2/25 14:36, Saanjh Sengupta wrote:
>
>  Hi,
>
>  I referred to your mailing chains on suggesting QEMU to be built as a
>  shared library.
>
>  *Change meson.build to build QEMU as a shared library (with PIC enabled
>  for static libraries)*
>  *
>  *
>  Could you please suggest what exactly has to be enabled in the meson.build?
>
>  I am confused on that front.
>
>  Regards
>  Saanjh Sengupta
>
>  --
>  Alex Bennée
>  Virtualisation Tech Lead @ Linaro

--
Alex Bennée
Virtualisation Tech Lead @ Linaro

[-- Attachment #2: Type: text/html, Size: 18575 bytes --]

Re: Building QEMU as a Shared Library

[Alex Bennée]

Saanjh Sengupta <saanjhsengupta@outlook.com> writes:

> Hi,
>
> Unable to view this. Any GitHub repository diff page I could view for
> the same?

It was part of this series:

  https://patchew.org/QEMU/20241022105614.839199-1-alex.bennee@linaro.org/

Specifically this patch:

  https://patchew.org/QEMU/20241022105614.839199-1-alex.bennee@linaro.org/20241022105614.839199-18-alex.bennee@linaro.org/

>
> Regards
> Saanjh Sengupta 
> -------------------------------------------------------------------------------------------------------------------------
> From: Alex Bennée <alex.bennee@linaro.org>
> Sent: Tuesday, March 18, 2025 1:48:01 PM
> To: Saanjh Sengupta <saanjhsengupta@outlook.com>
> Cc: Pierrick Bouvier <pierrick.bouvier@linaro.org>; Philippe Mathieu-Daudé <philmd@linaro.org>; Paolo Bonzini
> <pbonzini@redhat.com>; Marc-André Lureau <marcandre.lureau@redhat.com>; amir.gonnen@neuroblade.ai
> <amir.gonnen@neuroblade.ai>; qemu-devel@nongnu.org <qemu-devel@nongnu.org>
> Subject: Re: Building QEMU as a Shared Library 
>  
> Saanjh Sengupta <saanjhsengupta@outlook.com> writes:
>
>> Hi Alex,  
>>
>> You mentioned about a patch series; do you have it handy with you?
>
>   Message-Id: <20241022105614.839199-18-alex.bennee@linaro.org>
>   Date: Tue, 22 Oct 2024 11:56:11 +0100
>   Subject: [PATCH v2 17/20] plugins: add ability to register a GDB triggered callback
>   From: =?UTF-8?q?Alex=20Benn=C3=A9e?= <alex.bennee@linaro.org>
>
>>
>> If so, could you please direct me to the same ?
>>
>>  On 14 Mar 2025, at 12:11 AM, Alex Bennée <alex.bennee@linaro.org> wrote:
>>
>>  Saanjh Sengupta <saanjhsengupta@outlook.com> writes:
>>
>>  Hi,
>>
>>  What we are trying to achieve is that the QEMU should run for a particular number of instructions, let's say for
>>  example
>>  10000 instructions and then pause it's emulation. After a resume trigger is received to the QEMU it must resume it's
>>  emulation and start the instruction count from 10001 (which basically
>>  means that the context should be saved).
>>
>>  I think you want to run under icount and you will need to modify the
>>  trigger plugin. Under icount we run each vCPU in turn, so if the plugin
>>  pauses the vCPU will de-facto be paused.
>>
>>  You would have to implement some sort of control interface in the
>>  plugin. Or you could add an API to trigger the gdbstub. I think I had
>>  that on a patch series at one point.
>>
>>  In the previous mail when you mentioned g_usleep, I believe this shall not work (as per our use-case) since it will
>>  reset the
>>  instruction count to 0 (as per what you mentioned). 
>>
>>  To achieve the use-case, do you have any leads/suggestions ? 
>>
>>  Regards
>>  Saanjh Sengupta 
>>
>>  -------------------------------------------------------------------------------------------------------------------------
>>  From: Pierrick Bouvier <pierrick.bouvier@linaro.org>
>>  Sent: Wednesday, March 12, 2025 11:50:23 am
>>  To: Saanjh Sengupta <saanjhsengupta@outlook.com>; Philippe Mathieu-Daudé <philmd@linaro.org>; Paolo Bonzini
>>  <pbonzini@redhat.com>; Marc-André Lureau <marcandre.lureau@redhat.com>
>>  Cc: amir.gonnen@neuroblade.ai <amir.gonnen@neuroblade.ai>; qemu-devel@nongnu.org
>>  <qemu-devel@nongnu.org>; Alex
>>  Bennée <alex.bennee@linaro.org>
>>  Subject: Re: Building QEMU as a Shared Library
>>
>>  On 3/11/25 21:31, Saanjh Sengupta wrote:
>>
>>  Hi,
>>
>>  Thank you for the clarification. Regarding the last time
>>  /"Stoptrigger might be a better fit for what you want to do, and instead 
>>  of exiting, you want to resume emulation after N insn. The function 
>>  qemu_clock_advance_virtual_time() can only be used to move the time 
>>  forward, and you can not stop the "virtual time" by design."/
>>  /
>>  /
>>  I did not quite understand this. Even if I have to modify the 
>>  stopTrigger plugin, I would want it to pause rather than exiting.
>>  For example: It gets 10000 instructions executed after that it should 
>>  pause and after some time it should then resume again execute till 20000 
>>  instructions (because previously it executed till 10000 and then it must 
>>  execute till 20000). How do I do this? How do I state the code to pause 
>>  the qemu's emulation after 10000 instructions?
>>
>>  By using g_usleep to pause the current cpu.
>>  As well, it's needed to reset insn_count to 0 to count instructions again.
>>
>>  With this command line:
>>  ./build/qemu-system-x86_64 -plugin 
>>  ./build/contrib/plugins/libstoptrigger.so,icount=1000 -d plugin
>>
>>  And with those changes to stoptrigger:
>>
>>  diff --git a/contrib/plugins/stoptrigger.c b/contrib/plugins/stoptrigger.c
>>  index b3a6ed66a7b..77fd413cef1 100644
>>  --- a/contrib/plugins/stoptrigger.c
>>  +++ b/contrib/plugins/stoptrigger.c
>>  @@ -41,11 +41,12 @@ typedef struct {
>>       int exit_code;
>>   } ExitInfo;
>>
>>  -static void exit_emulation(int return_code, char *message)
>>  +static void pause_emulation(int return_code, char *message)
>>   {
>>       qemu_plugin_outs(message);
>>       g_free(message);
>>  -    exit(return_code);
>>  +    /* exit(return_code); */
>>  +    g_usleep(1 * G_USEC_PER_SEC);
>>   }
>>
>>   static void exit_icount_reached(unsigned int cpu_index, void *udata)
>>  @@ -53,7 +54,9 @@ static void exit_icount_reached(unsigned int 
>>  cpu_index, void *udata)
>>       uint64_t insn_vaddr = qemu_plugin_u64_get(current_pc, cpu_index);
>>       char *msg = g_strdup_printf("icount reached at 0x%" PRIx64 ", 
>>  exiting\n",
>>                                   insn_vaddr);
>>  -    exit_emulation(icount_exit_code, msg);
>>  +    pause_emulation(icount_exit_code, msg);
>>  +    /* reset instruction counter */
>>  +    qemu_plugin_u64_set(insn_count, cpu_index, 0);
>>   }
>>
>>   static void exit_address_reached(unsigned int cpu_index, void *udata)
>>  @@ -61,7 +64,7 @@ static void exit_address_reached(unsigned int 
>>  cpu_index, void *udata)
>>       ExitInfo *ei = udata;
>>       g_assert(ei);
>>       char *msg = g_strdup_printf("0x%" PRIx64 " reached, exiting\n", 
>>  ei->exit_addr);
>>  -    exit_emulation(ei->exit_code, msg);
>>  +    pause_emulation(ei->exit_code, msg);
>>   }
>>
>>  Moreover, I tried an activity where I was utilising the QMP protocol to 
>>  control the virtual time (with respect to the IPS plugin). In that 
>>  context when the QMP stop is triggered, my virtual time does got freezed 
>>  until the resume is triggered. Does this mean I am able to manipulate 
>>  the virtual time of the QEMU?
>>
>>  I am not sure of how it works, but the plugin interface only allows to 
>>  move time forward.
>>
>>  Regards
>>  Saanjh Sengupta
>>  ------------------------------------------------------------------------
>>  *From:* Pierrick Bouvier <pierrick.bouvier@linaro.org>
>>  *Sent:* Wednesday, March 12, 2025 2:14:47 AM
>>  *To:* Saanjh Sengupta <saanjhsengupta@outlook.com>; Philippe Mathieu- 
>>  Daudé <philmd@linaro.org>; Paolo Bonzini <pbonzini@redhat.com>; Marc- 
>>  André Lureau <marcandre.lureau@redhat.com>
>>  *Cc:* amir.gonnen@neuroblade.ai <amir.gonnen@neuroblade.ai>; qemu- 
>>  devel@nongnu.org <qemu-devel@nongnu.org>; Alex Bennée 
>>  <alex.bennee@linaro.org>
>>  *Subject:* Re: Building QEMU as a Shared Library
>>  On 3/11/25 02:50, Saanjh Sengupta wrote:
>>
>>  Hi,
>>
>>  I have a couple of questions:
>>
>>  1.
>>     When I use the libstoptrigger.so: in that case the QEMU 's emulation
>>     stops after executing the defined number of instructions. Post this,
>>     the whole QEMU terminates. And while using the libips.so I am
>>     assuming that the QEMU doesn't execute no more than the defined
>>     instructions. Please correct me if I am wrong.
>>
>>  That's correct for both plugins, with the additional note that libips
>>  does this per second only.
>>
>>  2.
>>     In my case, I want the QEMU to start emulation for some time and
>>     PAUSE it's emulation for some time; after it is Paused (it's virtual
>>     time is also to be paused) and then let's say for after 'x' time
>>     period it should resume it's virtual time.
>>
>>  The virtual time variable in ips plugin is only related to this plugin,
>>  and based on how many instructions have been executed, which is
>>  different from what you want to achieve.
>>
>>  Stoptrigger might be a better fit for what you want to do, and instead
>>  of exiting, you want to resume emulation after N insn.
>>  The function qemu_clock_advance_virtual_time() can only be used to move
>>  the time forward, and you can not stop the "virtual time" by design.
>>
>>  image
>>
>>  I have added this segment inside the update_system_time function inside 
>>  the ipsPlugin.c. but once the instructions reach to the defined limit 
>>  the virtual time does not seem to stop.
>>  Do you have any suggestions on that front?
>>
>>  Regards
>>  Saanjh Sengupta
>>  ------------------------------------------------------------------------
>>  *From:* Pierrick Bouvier <pierrick.bouvier@linaro.org>
>>  *Sent:* Wednesday, March 5, 2025 5:20:38 AM
>>  *To:* Saanjh Sengupta <saanjhsengupta@outlook.com>; Philippe Mathieu- 
>>  Daudé <philmd@linaro.org>; Paolo Bonzini <pbonzini@redhat.com>; Marc- 
>>  André Lureau <marcandre.lureau@redhat.com>
>>  *Cc:* amir.gonnen@neuroblade.ai <amir.gonnen@neuroblade.ai>; qemu- 
>>  devel@nongnu.org <qemu-devel@nongnu.org>; Alex Bennée 
>>  <alex.bennee@linaro.org>
>>  *Subject:* Re: Building QEMU as a Shared Library
>>  Hi Saanjh,
>>
>>  depending what you are trying to achieve exactly, plugins can provide a
>>  solution. It's convenient and you can stay on top of QEMU upstream,
>>  without having to create a downstream fork.
>>
>>  We already have plugins for stopping after a given number of
>>  instructions, or slow down execution of a VM:
>>
>>  # stop after executing 1'000'000 instructions:
>>  $ ./build/qemu-system-x86_64 -plugin
>>  ./build/contrib/plugins/libstoptrigger,icount=1000000 -d plugin
>>
>>  # execute no more than 1'000'000 instructions per second:
>>  $ ./build/qemu-system-x86_64 -plugin
>>  ./build/contrib/plugins/libips.so,ips=1000000 -d plugin
>>
>>  You can see source code associated (./contrib/plugins/stoptrigger.c and
>>  ./contrib/plugins/ips.c), to implement something similar to what you
>>  want, but based on time.
>>  Would that satisfy your need?
>>
>>  Regards,
>>  Pierrick
>>
>>  On 3/3/25 21:53, Saanjh Sengupta wrote:
>>
>>  Hi,
>>
>>  Thank you so much for your inputs. I was able to create the .so file of 
>>  QEMU.
>>
>>  Actually, what we are trying is to understand and explore possibilities 
>>  of Virtual Time Control in QEMU. In short, what I mean to say is an 
>>  approach via which I can tell QEMU to emulate for XYZ time when the I 
>>  give a trigger and then pause the emulation by itself after the XYZ time 
>>  is completed.
>>
>>  On that front itself, do you have any inputs/ideas regarding the same?
>>
>>  Regards
>>  Saanjh Sengupta
>>  ------------------------------------------------------------------------
>>  *From:* Pierrick Bouvier <pierrick.bouvier@linaro.org>
>>  *Sent:* Tuesday, February 25, 2025 6:29:44 AM
>>  *To:* Philippe Mathieu-Daudé <philmd@linaro.org>; Paolo Bonzini 
>>  <pbonzini@redhat.com>; Marc-André Lureau <marcandre.lureau@redhat.com>
>>  *Cc:* amir.gonnen@neuroblade.ai <amir.gonnen@neuroblade.ai>; qemu- 
>>  devel@nongnu.org <qemu-devel@nongnu.org>; Saanjh Sengupta 
>>  <saanjhsengupta@outlook.com>
>>  *Subject:* Re: Building QEMU as a Shared Library
>>  Hi Saanjh,
>>
>>  here is a minimal patch that builds one shared library per target (arch,
>>  mode) where arch is cpu arch, and mode is system or user, and launch
>>  system-aarch64 through a simple driver:
>>
>>  https://github.com/pbo-linaro/qemu/commit/ <https://github.com/pbo- 
>>
>>  linaro/qemu/commit/> <https://github.com/pbo-
>>
>>  linaro/qemu/commit/>
>>
>>  fbb39cc64f77d4bf1e5e50795c75b62735bf5c5f <https://github.com/pbo-linaro/
>>  qemu/commit/fbb39cc64f77d4bf1e5e50795c75b62735bf5c5f>
>>
>>  With this, it could be possible to create a driver that can execute any
>>  existing target. It's a sort of single binary for QEMU, but shared
>>  objects are mandatory, and duplicates all the QEMU state. So there is no
>>  real benefit compared to having different processes.
>>
>>  In more, to be able to do concurrent emulations, there are much more
>>  problems to be solved. QEMU state is correctly kept per target, but all
>>  other libraries states are shared. There are various issues if you
>>  launch two emulations at the same time in two threads:
>>  - glib global context
>>  - qemu calls exit in many places, which stops the whole process
>>  - probably other things I didn't explore
>>
>>  At this point, even though qemu targets can be built as shared objects,
>>  I would recommend to use different processes, and implement some form on
>>  IPC to synchronize all this.
>>  Another possibility is to try to build machines without using the
>>  existing main, but I'm not sure it's worth all the hassle.
>>
>>  What are you trying to achieve?
>>
>>  Regards,
>>  Pierrick
>>
>>  On 2/24/25 01:10, Philippe Mathieu-Daudé wrote:
>>
>>  Cc'ing our meson experts
>>
>>  On 22/2/25 14:36, Saanjh Sengupta wrote:
>>
>>  Hi,
>>
>>  I referred to your mailing chains on suggesting QEMU to be built as a
>>  shared library.
>>
>>  *Change meson.build to build QEMU as a shared library (with PIC enabled
>>  for static libraries)*
>>  *
>>  *
>>  Could you please suggest what exactly has to be enabled in the meson.build?
>>
>>  I am confused on that front.
>>
>>  Regards
>>  Saanjh Sengupta
>>
>>  -- 
>>  Alex Bennée
>>  Virtualisation Tech Lead @ Linaro

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Building QEMU as a shared library

[Amir Gonnen]

[-- Attachment #1: Type: text/plain, Size: 2574 bytes --]

Hi,

Before sending a patch, I would like to check if it's of interest to the community.

My goal is to simulate a mixed architecture system.
Today QEMU strongly assumes that the simulated system is a *single architecture*.
Changing this assumption and supporting mixed architecture in QEMU proved to be
non-trivial and may require significant development effort. Common code such as
TCG and others explicitly include architecture specific header files, for example.

A possible solution, discussed on https://stackoverflow.com/q/63229262/619493 is to
separate the simulation to multiple processes (as done by Xilinx) and to use some form
of Interprocess Communication channel between them.
Such solution has several disadvantages:

- Harder to synchronize simulation between processes
- Performance impact of Interprocess Communication
- Harder to debug, profile and maintain

Instead, I would like to suggest a new approach we use at Neuroblade to achieve this:
Build QEMU as a shared library that can be loaded and used directly in a larger simulation.
Today we build qemu-system-nios2 shared library and load it from qemu-system-x86_64 in order
to simulate an x86_64 system that also consists of multiple nios2 cores.
In our simulation, two independent "main" functions are running on different threads, and
simulation synchronization is reduced to synchronizing threads.

To achieve this, I needed to do the following changes in QEMU:

1. Avoid Glib global context. Use a different context (g_main_context_new) for each QEMU instance.
2. Change meson.build to build QEMU as a shared library (with PIC enabled for static libraries)
3. Define a C API for the library and export it (with a -Wl,--version-script)

These changes seem enough for simulating mixed architecture system on a single process.

If this approach sounds useful, I'll be happy to send patches.
I'd appreciate if you could provide your feedback!

Thanks,
Amir


The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential and/or privileged information and may be legally protected from disclosure. If you are not the intended recipient of this message or their agent, or if this message has been addressed to you in error, please immediately alert the sender by reply email and then delete this message and any attachments. If you are not the intended recipient, you are hereby notified that any use, dissemination, copying, or storage of this message or its attachments is strictly prohibited.

[-- Attachment #2: Type: text/html, Size: 5725 bytes --]

Re: Building QEMU as a shared library

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2658 bytes --]

On Wed, Dec 15, 2021 at 08:18:53AM +0000, Amir Gonnen wrote:
> Before sending a patch, I would like to check if it's of interest to the community.
> 
> My goal is to simulate a mixed architecture system.
> Today QEMU strongly assumes that the simulated system is a *single architecture*.
> Changing this assumption and supporting mixed architecture in QEMU proved to be
> non-trivial and may require significant development effort. Common code such as
> TCG and others explicitly include architecture specific header files, for example.

Hi Amir,
Simulating heterogenous machines comes up from periodically. So far no
one has upstreamed a solution but there is definitely interest.

I suggest going ahead and posting the code even if it's not cleaned up.

> A possible solution, discussed on https://stackoverflow.com/q/63229262/619493 is to
> separate the simulation to multiple processes (as done by Xilinx) and to use some form
> of Interprocess Communication channel between them.
> Such solution has several disadvantages:
> 
> - Harder to synchronize simulation between processes
> - Performance impact of Interprocess Communication
> - Harder to debug, profile and maintain
> 
> Instead, I would like to suggest a new approach we use at Neuroblade to achieve this:
> Build QEMU as a shared library that can be loaded and used directly in a larger simulation.
> Today we build qemu-system-nios2 shared library and load it from qemu-system-x86_64 in order
> to simulate an x86_64 system that also consists of multiple nios2 cores.
> In our simulation, two independent "main" functions are running on different threads, and
> simulation synchronization is reduced to synchronizing threads.
> 
> To achieve this, I needed to do the following changes in QEMU:
> 
> 1. Avoid Glib global context. Use a different context (g_main_context_new) for each QEMU instance.
> 2. Change meson.build to build QEMU as a shared library (with PIC enabled for static libraries)
> 3. Define a C API for the library and export it (with a -Wl,--version-script)
> 
> These changes seem enough for simulating mixed architecture system on a single process.
> 
> If this approach sounds useful, I'll be happy to send patches.
> I'd appreciate if you could provide your feedback!

I'm curious how much synchronization and IPC there is between the QEMU
shared libraries? I would have guessed that the pain of making
communication work efficiently between processes would be less than the
pain of solving global state bugs related to shared libraries within a
single process.

Were there issues with POSIX signal handlers?

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Building QEMU as a shared library

[Daniel P. Berrangé]

On Wed, Dec 15, 2021 at 09:45:56AM +0000, Stefan Hajnoczi wrote:
> On Wed, Dec 15, 2021 at 08:18:53AM +0000, Amir Gonnen wrote:
> > Before sending a patch, I would like to check if it's of interest to the community.
> > 
> > My goal is to simulate a mixed architecture system.
> > Today QEMU strongly assumes that the simulated system is a *single architecture*.
> > Changing this assumption and supporting mixed architecture in QEMU proved to be
> > non-trivial and may require significant development effort. Common code such as
> > TCG and others explicitly include architecture specific header files, for example.
> 
> Hi Amir,
> Simulating heterogenous machines comes up from periodically. So far no
> one has upstreamed a solution but there is definitely interest.
> 
> I suggest going ahead and posting the code even if it's not cleaned up.
> 
> > A possible solution, discussed on https://stackoverflow.com/q/63229262/619493 is to
> > separate the simulation to multiple processes (as done by Xilinx) and to use some form
> > of Interprocess Communication channel between them.
> > Such solution has several disadvantages:
> > 
> > - Harder to synchronize simulation between processes
> > - Performance impact of Interprocess Communication
> > - Harder to debug, profile and maintain
> > 
> > Instead, I would like to suggest a new approach we use at Neuroblade to achieve this:
> > Build QEMU as a shared library that can be loaded and used directly in a larger simulation.
> > Today we build qemu-system-nios2 shared library and load it from qemu-system-x86_64 in order
> > to simulate an x86_64 system that also consists of multiple nios2 cores.
> > In our simulation, two independent "main" functions are running on different threads, and
> > simulation synchronization is reduced to synchronizing threads.
> > 
> > To achieve this, I needed to do the following changes in QEMU:
> > 
> > 1. Avoid Glib global context. Use a different context (g_main_context_new) for each QEMU instance.
> > 2. Change meson.build to build QEMU as a shared library (with PIC enabled for static libraries)
> > 3. Define a C API for the library and export it (with a -Wl,--version-script)

I'm curious what kind of scope the C API has and whether it is likely to
conflict with any general plans we have for future evolution to QEMU's
design, especially around configuration / CLI.

In your example where the client loading the QEMU shared library is
QEMU itself, the licensing doesn't become an issue, as everything
is within the same project codebase.

If the shared library were exposed to consumers outside of the
QEMU project, then licensing is a significant stumbling block,
as any consumer has to be GPL-v2-only compatible. It is pretty
easy to get into a license incompatibility situation with this
requirement. I fear alot of people would just ignore this as an
"inconvenience" and knowingly use incompatible code.

Both of these issues could be avoided if we make any shuared
library a QEMU-internal thing, by not installing any headers
and doing something to ensure its ABI changes on every build.

> > These changes seem enough for simulating mixed architecture system on a single process.
> > 
> > If this approach sounds useful, I'll be happy to send patches.
> > I'd appreciate if you could provide your feedback!
> 
> I'm curious how much synchronization and IPC there is between the QEMU
> shared libraries? I would have guessed that the pain of making
> communication work efficiently between processes would be less than the
> pain of solving global state bugs related to shared libraries within a
> single process.

We've already got ability to have device backends run in separate
processes with vhost-user, and have various other efforts underway
to support more generalized out of tree device implementations.
With this direction in mind, it feels conceptually desirable  if
we are able to support heterogenous CPU emulation using co-operating
processes too, as opposed to a monolithic process architecture.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

RE: Building QEMU as a shared library

[Amir Gonnen]

Hi Stefan,

Easier/faster synchronization is just one side of the issue.
It's much easier to debug a single process or profile it, for example. It's also easier to deploy and maintain a single process.

For now, the only "global state" problem I had to fix was Glib global context.
I didn't see an issue with posix signals. Any other global state you think I should take care of?
Of course, I only tested it for my own simple use case (qemu-system-nios2 + qemu-system-x86_64 with no GUI)

As a first step , I intend to send a patch that removes the Glib context global state.

Thanks,
Amir

-----Original Message-----
From: Stefan Hajnoczi <stefanha@redhat.com>
Sent: Wednesday, December 15, 2021 11:46 AM
To: Amir Gonnen <amir.gonnen@neuroblade.ai>
Cc: qemu-devel@nongnu.org; peter.maydell@linaro.org; f4bug@amsat.org
Subject: Re: Building QEMU as a shared library

On Wed, Dec 15, 2021 at 08:18:53AM +0000, Amir Gonnen wrote:
> Before sending a patch, I would like to check if it's of interest to the community.
>
> My goal is to simulate a mixed architecture system.
> Today QEMU strongly assumes that the simulated system is a *single architecture*.
> Changing this assumption and supporting mixed architecture in QEMU
> proved to be non-trivial and may require significant development
> effort. Common code such as TCG and others explicitly include architecture specific header files, for example.

Hi Amir,
Simulating heterogenous machines comes up from periodically. So far no one has upstreamed a solution but there is definitely interest.

I suggest going ahead and posting the code even if it's not cleaned up.

> A possible solution, discussed on
> https://stackoverflow.com/q/63229262/619493 is to separate the
> simulation to multiple processes (as done by Xilinx) and to use some form of Interprocess Communication channel between them.
> Such solution has several disadvantages:
>
> - Harder to synchronize simulation between processes
> - Performance impact of Interprocess Communication
> - Harder to debug, profile and maintain
>
> Instead, I would like to suggest a new approach we use at Neuroblade to achieve this:
> Build QEMU as a shared library that can be loaded and used directly in a larger simulation.
> Today we build qemu-system-nios2 shared library and load it from
> qemu-system-x86_64 in order to simulate an x86_64 system that also consists of multiple nios2 cores.
> In our simulation, two independent "main" functions are running on
> different threads, and simulation synchronization is reduced to synchronizing threads.
>
> To achieve this, I needed to do the following changes in QEMU:
>
> 1. Avoid Glib global context. Use a different context (g_main_context_new) for each QEMU instance.
> 2. Change meson.build to build QEMU as a shared library (with PIC
> enabled for static libraries) 3. Define a C API for the library and
> export it (with a -Wl,--version-script)
>
> These changes seem enough for simulating mixed architecture system on a single process.
>
> If this approach sounds useful, I'll be happy to send patches.
> I'd appreciate if you could provide your feedback!

I'm curious how much synchronization and IPC there is between the QEMU shared libraries? I would have guessed that the pain of making communication work efficiently between processes would be less than the pain of solving global state bugs related to shared libraries within a single process.

Were there issues with POSIX signal handlers?

Stefan

The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential and/or privileged information and may be legally protected from disclosure. If you are not the intended recipient of this message or their agent, or if this message has been addressed to you in error, please immediately alert the sender by reply email and then delete this message and any attachments. If you are not the intended recipient, you are hereby notified that any use, dissemination, copying, or storage of this message or its attachments is strictly prohibited.

Re: Building QEMU as a shared library

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1784 bytes --]

On Wed, Dec 15, 2021 at 12:18:16PM +0000, Amir Gonnen wrote:
> Easier/faster synchronization is just one side of the issue.
> It's much easier to debug a single process or profile it, for example. It's also easier to deploy and maintain a single process.
> 
> For now, the only "global state" problem I had to fix was Glib global context.
> I didn't see an issue with posix signals. Any other global state you think I should take care of?

Signal handlers are interesting because they are per-process, not
per-thread. Luckily QEMU doesn't use signals that much. Accelerators
like KVM use SIGUSR1 while TCG doesn't rely on signals.

One example is what happens when you send SIGINT (Ctrl+C) to the
process. Only one of the shared libraries will handle the signal. The
other one will not be aware that SIGINT happened.

The signal handlers that execute in specific threads (e.g. vCPU threads)
are likely to crash or behave in weird ways. For example,
softmmu/cpus.c:

  static void sigbus_handler(int n, siginfo_t *siginfo, void *ctx)
  {
      if (siginfo->si_code != BUS_MCEERR_AO && siginfo->si_code != BUS_MCEERR_AR) {
          sigbus_reraise();
      }
  
      if (current_cpu) {
          ^^^^^^^^^^^

current_cpu is a thread-local variable. The problem is that the
current_cpu symbol belongs to a specific shared library (nios2 or
x86_64).

If nios2 installs this signal handler then x86_64 vCPUs will not be able
to handle SIGBUS because the current_cpu variable is trying to fetch the
thread-local storage allocated to the other library.

Global variables are also broken in the same way. If the other shared
library's version of the signal handler function is executed we'll
access the other global variable and not our own.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Building QEMU as a shared library

[Peter Maydell]

On Wed, 15 Dec 2021 at 13:23, Stefan Hajnoczi <stefanha@redhat.com> wrote:
>
> On Wed, Dec 15, 2021 at 12:18:16PM +0000, Amir Gonnen wrote:
> > Easier/faster synchronization is just one side of the issue.
> > It's much easier to debug a single process or profile it, for example. It's also easier to deploy and maintain a single process.
> >
> > For now, the only "global state" problem I had to fix was Glib global context.
> > I didn't see an issue with posix signals. Any other global state you think I should take care of?
>
> Signal handlers are interesting because they are per-process, not
> per-thread. Luckily QEMU doesn't use signals that much. Accelerators
> like KVM use SIGUSR1 while TCG doesn't rely on signals.

I think TCG also uses SIGUSR1 for SIG_IPI.

-- PMM

Re: Building QEMU as a shared library

[Peter Maydell]

On Wed, 15 Dec 2021 at 08:18, Amir Gonnen <amir.gonnen@neuroblade.ai> wrote:
> My goal is to simulate a mixed architecture system.
>
> Today QEMU strongly assumes that the simulated system is a *single architecture*.
> Changing this assumption and supporting mixed architecture in QEMU proved to be
> non-trivial and may require significant development effort. Common code such as
> TCG and others explicitly include architecture specific header files, for example.

Yeah. This is definitely something we'd like to fix some day. It's
the approach I would prefer for getting multi-architecture machines.

> Instead, I would like to suggest a new approach we use at Neuroblade to achieve this:
> Build QEMU as a shared library that can be loaded and used directly in a larger simulation.
> Today we build qemu-system-nios2 shared library and load it from qemu-system-x86_64 in order
> to simulate an x86_64 system that also consists of multiple nios2 cores.
> In our simulation, two independent "main" functions are running on different threads, and
> simulation synchronization is reduced to synchronizing threads.

I agree with Stefan that you should go ahead and send the code as
an RFC patchset, but I feel like there is a lot of work required
to really get the codebase into a state where it is a clean
shared library...

-- PMM

Re: Building QEMU as a shared library

[Daniel P. Berrangé]

On Wed, Dec 15, 2021 at 10:10:35AM +0000, Peter Maydell wrote:
> On Wed, 15 Dec 2021 at 08:18, Amir Gonnen <amir.gonnen@neuroblade.ai> wrote:
> > My goal is to simulate a mixed architecture system.
> >
> > Today QEMU strongly assumes that the simulated system is a *single architecture*.
> > Changing this assumption and supporting mixed architecture in QEMU proved to be
> > non-trivial and may require significant development effort. Common code such as
> > TCG and others explicitly include architecture specific header files, for example.
> 
> Yeah. This is definitely something we'd like to fix some day. It's
> the approach I would prefer for getting multi-architecture machines.
> 
> > Instead, I would like to suggest a new approach we use at Neuroblade to achieve this:
> > Build QEMU as a shared library that can be loaded and used directly in a larger simulation.
> > Today we build qemu-system-nios2 shared library and load it from qemu-system-x86_64 in order
> > to simulate an x86_64 system that also consists of multiple nios2 cores.
> > In our simulation, two independent "main" functions are running on different threads, and
> > simulation synchronization is reduced to synchronizing threads.
> 
> I agree with Stefan that you should go ahead and send the code as
> an RFC patchset, but I feel like there is a lot of work required
> to really get the codebase into a state where it is a clean
> shared library...

I expect there could end up being a big difference between

   "clean for use with QEMU CLI config X"

vs

   "clean for use with all possible QEMU CLI configs"


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Building QEMU as a shared library

[Philippe Mathieu-Daudé]

Hi Peter,

On 12/15/21 11:10, Peter Maydell wrote:
> On Wed, 15 Dec 2021 at 08:18, Amir Gonnen <amir.gonnen@neuroblade.ai> wrote:
>> My goal is to simulate a mixed architecture system.
>>
>> Today QEMU strongly assumes that the simulated system is a *single architecture*.
>> Changing this assumption and supporting mixed architecture in QEMU proved to be
>> non-trivial and may require significant development effort. Common code such as
>> TCG and others explicitly include architecture specific header files, for example.
> 
> Yeah. This is definitely something we'd like to fix some day. It's
> the approach I would prefer for getting multi-architecture machines.

Am I understanding correctly your preference would be *not* using shared
libraries, but having a monolithic process able to use any configuration
of heterogeneous architectures?

What are your thoughts on Daniel idea to where (IIUC) cores can are
external processes wired via vhost-user. One problem is not all
operating systems supported provide this possibility.

>> Instead, I would like to suggest a new approach we use at Neuroblade to achieve this:
>> Build QEMU as a shared library that can be loaded and used directly in a larger simulation.
>> Today we build qemu-system-nios2 shared library and load it from qemu-system-x86_64 in order
>> to simulate an x86_64 system that also consists of multiple nios2 cores.
>> In our simulation, two independent "main" functions are running on different threads, and
>> simulation synchronization is reduced to synchronizing threads.
> 
> I agree with Stefan that you should go ahead and send the code as
> an RFC patchset, but I feel like there is a lot of work required
> to really get the codebase into a state where it is a clean
> shared library...
> 
> -- PMM
> 

Re: Building QEMU as a shared library

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 857 bytes --]

On Thu, Dec 23, 2021 at 10:49:46AM +0100, Philippe Mathieu-Daudé wrote:
> What are your thoughts on Daniel idea to where (IIUC) cores can are
> external processes wired via vhost-user. One problem is not all
> operating systems supported provide this possibility.

There is no fundamental limitation that prevents vhost-user from being
used on non-Linux OSes. Eventfds can be replaced with pipes or other
file descriptors. Shared memory fd passing can be replaced with an
OS-specific shared memory API.

If the OS does not support these things:

- Johannes Berg added in-band notifications ("docs: vhost-user: add
  in-band kick/call messages"), so eventfds aren't strictly necessary
  anymore.

- David Gilbert is working on in-band DMA, reducing the need for shared
  memory (at the cost of extra inter-process communication).

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Building QEMU as a shared library

[Peter Maydell]

On Thu, 23 Dec 2021 at 09:49, Philippe Mathieu-Daudé <f4bug@amsat.org> wrote:
>
> Hi Peter,
>
> On 12/15/21 11:10, Peter Maydell wrote:
> > On Wed, 15 Dec 2021 at 08:18, Amir Gonnen <amir.gonnen@neuroblade.ai> wrote:
> >> My goal is to simulate a mixed architecture system.
> >>
> >> Today QEMU strongly assumes that the simulated system is a *single architecture*.
> >> Changing this assumption and supporting mixed architecture in QEMU proved to be
> >> non-trivial and may require significant development effort. Common code such as
> >> TCG and others explicitly include architecture specific header files, for example.
> >
> > Yeah. This is definitely something we'd like to fix some day. It's
> > the approach I would prefer for getting multi-architecture machines.
>
> Am I understanding correctly your preference would be *not* using shared
> libraries, but having a monolithic process able to use any configuration
> of heterogeneous architectures?

That would be my preference, yes. On the other hand I know there's
a lot of work in trying to get there, so I don't want to rule out the
idea that maybe pragmatically we do something else instead.

> What are your thoughts on Daniel idea to where (IIUC) cores can are
> external processes wired via vhost-user.

It sounds a bit awkward to me -- you end up with a system where
QEMU's scheduler between vCPUs is no longer the only thing deciding
what gets to run. (eg, how do you emulate atomic accesses? currently
we do those by "stop all other CPUs, do the thing, restart" for the
cases where we can't rely on the host's atomic insns.)
The multiple-host-processes model can certainly be made to work,
though -- AIUI this is how Xilinx's out-of-tree stuff works.

-- PMM