RFC: QEMU rv64/rv32 CPU, 'max' CPU and profile support

RFC: QEMU rv64/rv32 CPU, 'max' CPU and profile support

[Daniel Henrique Barboza]

Hi,

We don't have a 'max' cpu type in riscv32 and riscv64. This is a special,
fictional CPU type that has all possible extensions enabled. It can be
either an alias for a real CPU or a new CPU type. It's used by tooling
such as libvirt for capabilities probing.

Well, in our case it can't be an alias because we don't have any CPU type
that enables all extensions. Which brings me to my first question:
what's the criteria for marking a certain extension default true/false
in rv32/rv64/rv128?

E.g. this is the riscv,isa DT for rv64:

rv64imafdch_zicbom_zicboz_zicsr_zifencei_zihintpause_zawrs_zfa_zca_zcd_zba_zbb_zbc_zbs_sstc_svadu

It has a lot of extensions but it's missing some common ones like svinval
and smstateen.

We can add/remove some extensions from these generic CPUs, document it
and be done with it. I want to propose a different design though:

1 - introduce the 'max' CPU. I guess we'll need some prep work in how
the extensions are being declared/stored in the code. I'll work on
it;

2 - introduce rva23U64/rva23S64. I would like to add a new 'profile' option
for generic CPUs, with '-cpu rv64,profile=rva23U64' enabling a whole set
of extensions. We would also support enabling optional profile extensions.
Yes, we don't have all the profile extensions available yet (neither does
the kernel AFAIK) but nothing is stopping us from adding the base
framework;

3 - reduce the generic CPUs up to the minimum required to boot Linux.
Yes, a rather arbitrary criteria, but Linux is the most common OS used
by RISC-V QEMU currently so we'll go with what most people are using.
This would reduce rv64 to "rv64ima".

The end result is that we would have a max extensions CPU, a barebones
generic CPU that can be fully customizable from the ground up, and
a profile option to allow users to boost the generic CPUs with a
set of extensions. We will avoid having to deal with generic CPUs having
an (apparently) arbitrary set of extensions.

I don't think we'll have too much opposition on adding (1) so I'll get
on it. I really would like to hear about (2) and (3). (3) will break
users that are using rv64 while counting with that whole set of extensions
that are being enabled by default, but this is why we're going to add
(2) first.


Please comment. Andrea, feel free to give us libvirt/tooling insights.


Thanks,

Daniel

Re: RFC: QEMU rv64/rv32 CPU, 'max' CPU and profile support

[Palmer Dabbelt]

On Mon, 10 Jul 2023 10:53:12 PDT (-0700), dbarboza@ventanamicro.com wrote:
> Hi,
>
> We don't have a 'max' cpu type in riscv32 and riscv64. This is a special,
> fictional CPU type that has all possible extensions enabled. It can be
> either an alias for a real CPU or a new CPU type. It's used by tooling
> such as libvirt for capabilities probing.
>
> Well, in our case it can't be an alias because we don't have any CPU type
> that enables all extensions. Which brings me to my first question:
> what's the criteria for marking a certain extension default true/false
> in rv32/rv64/rv128?

Pretty sure it's just been a somewhat arbitrary decision, with stuff 
that's likely to be used being enabled when it's added.

> E.g. this is the riscv,isa DT for rv64:
>
> rv64imafdch_zicbom_zicboz_zicsr_zifencei_zihintpause_zawrs_zfa_zca_zcd_zba_zbb_zbc_zbs_sstc_svadu
>
> It has a lot of extensions but it's missing some common ones like svinval
> and smstateen.
>
> We can add/remove some extensions from these generic CPUs, document it
> and be done with it. I want to propose a different design though:
>
> 1 - introduce the 'max' CPU. I guess we'll need some prep work in how
> the extensions are being declared/stored in the code. I'll work on
> it;

IMO it's the right idea in spirit, but there's a handful of conflicting 
extensions -- mostly vendor stuff, but even some of the standard ones 
have conflicting encodings.  So I think there's likely to be some 
decisions to make there, but when there's conflicts there's usually also 
one called out as being aimed at microcontroller-class systems so there 
should be a natural choice.

> 2 - introduce rva23U64/rva23S64. I would like to add a new 'profile' option
> for generic CPUs, with '-cpu rv64,profile=rva23U64' enabling a whole set
> of extensions. We would also support enabling optional profile extensions.
> Yes, we don't have all the profile extensions available yet (neither does
> the kernel AFAIK) but nothing is stopping us from adding the base
> framework;

Would we also make incompatible extensions forbidden in the profile 
CPUs?

If we add the 23 profiles, we should probably also add some other bases.  
We've got a defacto base in Linux that's rv{32,64}ima (plus some cruft 
as things have drifted around), I think there's also some interest in 
the 22 profiles.

> 3 - reduce the generic CPUs up to the minimum required to boot Linux.
> Yes, a rather arbitrary criteria, but Linux is the most common OS used
> by RISC-V QEMU currently so we'll go with what most people are using.
> This would reduce rv64 to "rv64ima".

Have a "minimum CPU required to boot Linux" option seems reasonable.  
Most users will probably want at least FD and C, though, so maybe we 
want something like that as well?

> The end result is that we would have a max extensions CPU, a barebones
> generic CPU that can be fully customizable from the ground up, and
> a profile option to allow users to boost the generic CPUs with a
> set of extensions. We will avoid having to deal with generic CPUs having
> an (apparently) arbitrary set of extensions.

IMO we still want to keep the ability to turn on/off extensions, the 
profiles are a long way from being concrete and we'll still have a lot 
of systems floating around that don't line up with them.  Being able to 
test those configurations in QEMU is a huge benefit for users, even if 
it's a headache to keep around all the options.

> I don't think we'll have too much opposition on adding (1) so I'll get
> on it. I really would like to hear about (2) and (3). (3) will break
> users that are using rv64 while counting with that whole set of extensions
> that are being enabled by default, but this is why we're going to add
> (2) first.
>
>
> Please comment. Andrea, feel free to give us libvirt/tooling insights.
>
>
> Thanks,
>
> Daniel

Re: RFC: QEMU rv64/rv32 CPU, 'max' CPU and profile support

[Daniel Henrique Barboza]

On 7/10/23 15:13, Palmer Dabbelt wrote:
> On Mon, 10 Jul 2023 10:53:12 PDT (-0700), dbarboza@ventanamicro.com wrote:
>> Hi,
>>
>> We don't have a 'max' cpu type in riscv32 and riscv64. This is a special,
>> fictional CPU type that has all possible extensions enabled. It can be
>> either an alias for a real CPU or a new CPU type. It's used by tooling
>> such as libvirt for capabilities probing.
>>
>> Well, in our case it can't be an alias because we don't have any CPU type
>> that enables all extensions. Which brings me to my first question:
>> what's the criteria for marking a certain extension default true/false
>> in rv32/rv64/rv128?
> 
> Pretty sure it's just been a somewhat arbitrary decision, with stuff that's likely to be used being enabled when it's added.
> 
>> E.g. this is the riscv,isa DT for rv64:
>>
>> rv64imafdch_zicbom_zicboz_zicsr_zifencei_zihintpause_zawrs_zfa_zca_zcd_zba_zbb_zbc_zbs_sstc_svadu
>>
>> It has a lot of extensions but it's missing some common ones like svinval
>> and smstateen.
>>
>> We can add/remove some extensions from these generic CPUs, document it
>> and be done with it. I want to propose a different design though:
>>
>> 1 - introduce the 'max' CPU. I guess we'll need some prep work in how
>> the extensions are being declared/stored in the code. I'll work on
>> it;
> 
> IMO it's the right idea in spirit, but there's a handful of conflicting extensions -- mostly vendor stuff, but even some of the standard ones have conflicting encodings.  So I think there's likely to be some decisions to make there, but when there's conflicts there's usually also one called out as being aimed at microcontroller-class systems so there should be a natural choice.
> 
>> 2 - introduce rva23U64/rva23S64. I would like to add a new 'profile' option
>> for generic CPUs, with '-cpu rv64,profile=rva23U64' enabling a whole set
>> of extensions. We would also support enabling optional profile extensions.
>> Yes, we don't have all the profile extensions available yet (neither does
>> the kernel AFAIK) but nothing is stopping us from adding the base
>> framework;
> 
> Would we also make incompatible extensions forbidden in the profile CPUs?

I think we can allow those as long as they're not in conflict with mandatory
profile extensions (not sure if there's a case of that).

> 
> If we add the 23 profiles, we should probably also add some other bases. We've got a defacto base in Linux that's rv{32,64}ima (plus some cruft as things have drifted around), I think there's also some interest in the 22 profiles.

rv32/64ima would be the base of the minimal generic CPUs I proposed below.

> 
>> 3 - reduce the generic CPUs up to the minimum required to boot Linux.
>> Yes, a rather arbitrary criteria, but Linux is the most common OS used
>> by RISC-V QEMU currently so we'll go with what most people are using.
>> This would reduce rv64 to "rv64ima".
> 
> Have a "minimum CPU required to boot Linux" option seems reasonable. Most users will probably want at least FD and C, though, so maybe we want something like that as well?

I don't mind making the minimal CPU be rv32/64imafdc.

> 
>> The end result is that we would have a max extensions CPU, a barebones
>> generic CPU that can be fully customizable from the ground up, and
>> a profile option to allow users to boost the generic CPUs with a
>> set of extensions. We will avoid having to deal with generic CPUs having
>> an (apparently) arbitrary set of extensions.
> 
> IMO we still want to keep the ability to turn on/off extensions, the profiles are a long way from being concrete and we'll still have a lot of systems floating around that don't line up with them.  Being able to test those configurations in QEMU is a huge benefit for users, even if it's a headache to keep around all the options.

Absolutely. What I meant is that, today, rv64 has a default set of extensions
that seems arbitrary. It would be good to have more predictability, i.e. have
a fixed set of extensions (like rv64imafdc) that users can use as a base to
enable/disable extensions.

The same thing can be done with profiles. I don't see a reason to forbid users
to do this:

'-cpu rv64,profile=rva23S64,icbom=false'

Meaning "I want the rva23S64 profile without Zicbom". The user should be aware that
Zicbom is mandatory for the profile, but since it's being explicitly disabled in the
command line, we can comply. This can be used for debug/development purposes or
the user is doing something silly. As long as it's explicit in the command line
it's fine.


Thanks,

Daniel


> 
>> I don't think we'll have too much opposition on adding (1) so I'll get
>> on it. I really would like to hear about (2) and (3). (3) will break
>> users that are using rv64 while counting with that whole set of extensions
>> that are being enabled by default, but this is why we're going to add
>> (2) first.
>>
>>
>> Please comment. Andrea, feel free to give us libvirt/tooling insights.
>>
>>
>> Thanks,
>>
>> Daniel

Re: RFC: QEMU rv64/rv32 CPU, 'max' CPU and profile support

[Palmer Dabbelt]

On Mon, 10 Jul 2023 11:46:34 PDT (-0700), dbarboza@ventanamicro.com wrote:
>
>
> On 7/10/23 15:13, Palmer Dabbelt wrote:
>> On Mon, 10 Jul 2023 10:53:12 PDT (-0700), dbarboza@ventanamicro.com wrote:
>>> Hi,
>>>
>>> We don't have a 'max' cpu type in riscv32 and riscv64. This is a special,
>>> fictional CPU type that has all possible extensions enabled. It can be
>>> either an alias for a real CPU or a new CPU type. It's used by tooling
>>> such as libvirt for capabilities probing.
>>>
>>> Well, in our case it can't be an alias because we don't have any CPU type
>>> that enables all extensions. Which brings me to my first question:
>>> what's the criteria for marking a certain extension default true/false
>>> in rv32/rv64/rv128?
>>
>> Pretty sure it's just been a somewhat arbitrary decision, with stuff that's likely to be used being enabled when it's added.
>>
>>> E.g. this is the riscv,isa DT for rv64:
>>>
>>> rv64imafdch_zicbom_zicboz_zicsr_zifencei_zihintpause_zawrs_zfa_zca_zcd_zba_zbb_zbc_zbs_sstc_svadu
>>>
>>> It has a lot of extensions but it's missing some common ones like svinval
>>> and smstateen.
>>>
>>> We can add/remove some extensions from these generic CPUs, document it
>>> and be done with it. I want to propose a different design though:
>>>
>>> 1 - introduce the 'max' CPU. I guess we'll need some prep work in how
>>> the extensions are being declared/stored in the code. I'll work on
>>> it;
>>
>> IMO it's the right idea in spirit, but there's a handful of conflicting extensions -- mostly vendor stuff, but even some of the standard ones have conflicting encodings.  So I think there's likely to be some decisions to make there, but when there's conflicts there's usually also one called out as being aimed at microcontroller-class systems so there should be a natural choice.
>>
>>> 2 - introduce rva23U64/rva23S64. I would like to add a new 'profile' option
>>> for generic CPUs, with '-cpu rv64,profile=rva23U64' enabling a whole set
>>> of extensions. We would also support enabling optional profile extensions.
>>> Yes, we don't have all the profile extensions available yet (neither does
>>> the kernel AFAIK) but nothing is stopping us from adding the base
>>> framework;
>>
>> Would we also make incompatible extensions forbidden in the profile CPUs?
>
> I think we can allow those as long as they're not in conflict with mandatory
> profile extensions (not sure if there's a case of that).

That seems reasonable to me too.

>> If we add the 23 profiles, we should probably also add some other bases. We've got a defacto base in Linux that's rv{32,64}ima (plus some cruft as things have drifted around), I think there's also some interest in the 22 profiles.
>
> rv32/64ima would be the base of the minimal generic CPUs I proposed below.

The Linux base has a few other things like Zicntr, Zifencei, Zicsr, etc 
-- essentially IMA from back before that stuff was all removed.

>>> 3 - reduce the generic CPUs up to the minimum required to boot Linux.
>>> Yes, a rather arbitrary criteria, but Linux is the most common OS used
>>> by RISC-V QEMU currently so we'll go with what most people are using.
>>> This would reduce rv64 to "rv64ima".
>>
>> Have a "minimum CPU required to boot Linux" option seems reasonable. Most users will probably want at least FD and C, though, so maybe we want something like that as well?
>
> I don't mind making the minimal CPU be rv32/64imafdc.

Ya, IDK if there's a right answer.  Maybe someone else has a stronger 
opinion?  IMO it's kind of just a double-edged sword as to which we pick 
for the minimum.

>>> The end result is that we would have a max extensions CPU, a barebones
>>> generic CPU that can be fully customizable from the ground up, and
>>> a profile option to allow users to boost the generic CPUs with a
>>> set of extensions. We will avoid having to deal with generic CPUs having
>>> an (apparently) arbitrary set of extensions.
>>
>> IMO we still want to keep the ability to turn on/off extensions, the profiles are a long way from being concrete and we'll still have a lot of systems floating around that don't line up with them.  Being able to test those configurations in QEMU is a huge benefit for users, even if it's a headache to keep around all the options.
>
> Absolutely. What I meant is that, today, rv64 has a default set of extensions
> that seems arbitrary. It would be good to have more predictability, i.e. have
> a fixed set of extensions (like rv64imafdc) that users can use as a base to
> enable/disable extensions.
>
> The same thing can be done with profiles. I don't see a reason to forbid users
> to do this:
>
> '-cpu rv64,profile=rva23S64,icbom=false'
>
> Meaning "I want the rva23S64 profile without Zicbom". The user should be aware that
> Zicbom is mandatory for the profile, but since it's being explicitly disabled in the
> command line, we can comply. This can be used for debug/development purposes or
> the user is doing something silly. As long as it's explicit in the command line
> it's fine.

Cool, I think I just misunderstood the original post.  I thought you 
were trying to say we should drop the per-extension properties.  I 
definitely agree that it's worth taking a look at the base CPUs, what 
we've got now is kind of just some arbitrary stuff that's left over from 
when the ISA was a lot simpler.

That'd also make the "should we allow incompatible extensions in 
profiles" questions easier, as we'd essentially just be saying the 
profile is shorthand for some set of base/extensions.

I guess that also brings up what to do with the U/S profiles split.  My 
guess is that most users just want both to match, so rva23s64 would 
include all the mandatory rva23u64 base/extensions?

> Thanks,
>
> Daniel
>
>
>>
>>> I don't think we'll have too much opposition on adding (1) so I'll get
>>> on it. I really would like to hear about (2) and (3). (3) will break
>>> users that are using rv64 while counting with that whole set of extensions
>>> that are being enabled by default, but this is why we're going to add
>>> (2) first.
>>>
>>>
>>> Please comment. Andrea, feel free to give us libvirt/tooling insights.
>>>
>>>
>>> Thanks,
>>>
>>> Daniel

Re: RFC: QEMU rv64/rv32 CPU, 'max' CPU and profile support

[Daniel Henrique Barboza]

On 7/10/23 16:06, Palmer Dabbelt wrote:
> On Mon, 10 Jul 2023 11:46:34 PDT (-0700), dbarboza@ventanamicro.com wrote:
>>
>>
>> On 7/10/23 15:13, Palmer Dabbelt wrote:
>>> On Mon, 10 Jul 2023 10:53:12 PDT (-0700), dbarboza@ventanamicro.com wrote:
>>>> Hi,
>>>>
>>>> We don't have a 'max' cpu type in riscv32 and riscv64. This is a special,
>>>> fictional CPU type that has all possible extensions enabled. It can be
>>>> either an alias for a real CPU or a new CPU type. It's used by tooling
>>>> such as libvirt for capabilities probing.
>>>>
>>>> Well, in our case it can't be an alias because we don't have any CPU type
>>>> that enables all extensions. Which brings me to my first question:
>>>> what's the criteria for marking a certain extension default true/false
>>>> in rv32/rv64/rv128?
>>>
>>> Pretty sure it's just been a somewhat arbitrary decision, with stuff that's likely to be used being enabled when it's added.
>>>
>>>> E.g. this is the riscv,isa DT for rv64:
>>>>
>>>> rv64imafdch_zicbom_zicboz_zicsr_zifencei_zihintpause_zawrs_zfa_zca_zcd_zba_zbb_zbc_zbs_sstc_svadu
>>>>
>>>> It has a lot of extensions but it's missing some common ones like svinval
>>>> and smstateen.
>>>>
>>>> We can add/remove some extensions from these generic CPUs, document it
>>>> and be done with it. I want to propose a different design though:
>>>>
>>>> 1 - introduce the 'max' CPU. I guess we'll need some prep work in how
>>>> the extensions are being declared/stored in the code. I'll work on
>>>> it;
>>>
>>> IMO it's the right idea in spirit, but there's a handful of conflicting extensions -- mostly vendor stuff, but even some of the standard ones have conflicting encodings.  So I think there's likely to be some decisions to make there, but when there's conflicts there's usually also one called out as being aimed at microcontroller-class systems so there should be a natural choice.
>>>
>>>> 2 - introduce rva23U64/rva23S64. I would like to add a new 'profile' option
>>>> for generic CPUs, with '-cpu rv64,profile=rva23U64' enabling a whole set
>>>> of extensions. We would also support enabling optional profile extensions.
>>>> Yes, we don't have all the profile extensions available yet (neither does
>>>> the kernel AFAIK) but nothing is stopping us from adding the base
>>>> framework;
>>>
>>> Would we also make incompatible extensions forbidden in the profile CPUs?
>>
>> I think we can allow those as long as they're not in conflict with mandatory
>> profile extensions (not sure if there's a case of that).
> 
> That seems reasonable to me too.
> 
>>> If we add the 23 profiles, we should probably also add some other bases. We've got a defacto base in Linux that's rv{32,64}ima (plus some cruft as things have drifted around), I think there's also some interest in the 22 profiles.
>>
>> rv32/64ima would be the base of the minimal generic CPUs I proposed below.
> 
> The Linux base has a few other things like Zicntr, Zifencei, Zicsr, etc -- essentially IMA from back before that stuff was all removed.


Hmm I guess we'll need some more thought in how we define a minimal set for
Linux then.

> 
>>>> 3 - reduce the generic CPUs up to the minimum required to boot Linux.
>>>> Yes, a rather arbitrary criteria, but Linux is the most common OS used
>>>> by RISC-V QEMU currently so we'll go with what most people are using.
>>>> This would reduce rv64 to "rv64ima".
>>>
>>> Have a "minimum CPU required to boot Linux" option seems reasonable. Most users will probably want at least FD and C, though, so maybe we want something like that as well?
>>
>> I don't mind making the minimal CPU be rv32/64imafdc.
> 
> Ya, IDK if there's a right answer.  Maybe someone else has a stronger opinion?  IMO it's kind of just a double-edged sword as to which we pick for the minimum.

As long as we change it and document it it should be ok. Users will still be able
to enable/disable extensions from the base rv64 CPU as well.


> 
>>>> The end result is that we would have a max extensions CPU, a barebones
>>>> generic CPU that can be fully customizable from the ground up, and
>>>> a profile option to allow users to boost the generic CPUs with a
>>>> set of extensions. We will avoid having to deal with generic CPUs having
>>>> an (apparently) arbitrary set of extensions.
>>>
>>> IMO we still want to keep the ability to turn on/off extensions, the profiles are a long way from being concrete and we'll still have a lot of systems floating around that don't line up with them.  Being able to test those configurations in QEMU is a huge benefit for users, even if it's a headache to keep around all the options.
>>
>> Absolutely. What I meant is that, today, rv64 has a default set of extensions
>> that seems arbitrary. It would be good to have more predictability, i.e. have
>> a fixed set of extensions (like rv64imafdc) that users can use as a base to
>> enable/disable extensions.
>>
>> The same thing can be done with profiles. I don't see a reason to forbid users
>> to do this:
>>
>> '-cpu rv64,profile=rva23S64,icbom=false'
>>
>> Meaning "I want the rva23S64 profile without Zicbom". The user should be aware that
>> Zicbom is mandatory for the profile, but since it's being explicitly disabled in the
>> command line, we can comply. This can be used for debug/development purposes or
>> the user is doing something silly. As long as it's explicit in the command line
>> it's fine.
> 
> Cool, I think I just misunderstood the original post.  I thought you were trying to say we should drop the per-extension properties.  I definitely agree that it's worth taking a look at the base CPUs, what we've got now is kind of just some arbitrary stuff that's left over from when the ISA was a lot simpler.
> 
> That'd also make the "should we allow incompatible extensions in profiles" questions easier, as we'd essentially just be saying the profile is shorthand for some set of base/extensions.
> 
> I guess that also brings up what to do with the U/S profiles split.  My guess is that most users just want both to match, so rva23s64 would include all the mandatory rva23u64 base/extensions?

I think so. The profile doc states:

RVA23S64 Mandatory Extensions

  The following unprivileged extensions are mandatory:

   The RVA23S64 mandatory unprivileged extensions include all the mandatory
   unprivileged extensions in RVA23U64.


So '-cpu rv64,profile=rva32S64' would include all the mandatory U64 extensions
plus all the mandatory S64. We should add both, so S64 will be build on top of
U64.


Thanks,

Daniel



> 
>> Thanks,
>>
>> Daniel
>>
>>
>>>
>>>> I don't think we'll have too much opposition on adding (1) so I'll get
>>>> on it. I really would like to hear about (2) and (3). (3) will break
>>>> users that are using rv64 while counting with that whole set of extensions
>>>> that are being enabled by default, but this is why we're going to add
>>>> (2) first.
>>>>
>>>>
>>>> Please comment. Andrea, feel free to give us libvirt/tooling insights.
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Daniel

Re: RFC: QEMU rv64/rv32 CPU, 'max' CPU and profile support

[Palmer Dabbelt]

On Mon, 10 Jul 2023 12:26:21 PDT (-0700), dbarboza@ventanamicro.com wrote:
>
>
> On 7/10/23 16:06, Palmer Dabbelt wrote:
>> On Mon, 10 Jul 2023 11:46:34 PDT (-0700), dbarboza@ventanamicro.com wrote:
>>>
>>>
>>> On 7/10/23 15:13, Palmer Dabbelt wrote:
>>>> On Mon, 10 Jul 2023 10:53:12 PDT (-0700), dbarboza@ventanamicro.com wrote:
>>>>> Hi,
>>>>>
>>>>> We don't have a 'max' cpu type in riscv32 and riscv64. This is a special,
>>>>> fictional CPU type that has all possible extensions enabled. It can be
>>>>> either an alias for a real CPU or a new CPU type. It's used by tooling
>>>>> such as libvirt for capabilities probing.
>>>>>
>>>>> Well, in our case it can't be an alias because we don't have any CPU type
>>>>> that enables all extensions. Which brings me to my first question:
>>>>> what's the criteria for marking a certain extension default true/false
>>>>> in rv32/rv64/rv128?
>>>>
>>>> Pretty sure it's just been a somewhat arbitrary decision, with stuff that's likely to be used being enabled when it's added.
>>>>
>>>>> E.g. this is the riscv,isa DT for rv64:
>>>>>
>>>>> rv64imafdch_zicbom_zicboz_zicsr_zifencei_zihintpause_zawrs_zfa_zca_zcd_zba_zbb_zbc_zbs_sstc_svadu
>>>>>
>>>>> It has a lot of extensions but it's missing some common ones like svinval
>>>>> and smstateen.
>>>>>
>>>>> We can add/remove some extensions from these generic CPUs, document it
>>>>> and be done with it. I want to propose a different design though:
>>>>>
>>>>> 1 - introduce the 'max' CPU. I guess we'll need some prep work in how
>>>>> the extensions are being declared/stored in the code. I'll work on
>>>>> it;
>>>>
>>>> IMO it's the right idea in spirit, but there's a handful of conflicting extensions -- mostly vendor stuff, but even some of the standard ones have conflicting encodings.  So I think there's likely to be some decisions to make there, but when there's conflicts there's usually also one called out as being aimed at microcontroller-class systems so there should be a natural choice.
>>>>
>>>>> 2 - introduce rva23U64/rva23S64. I would like to add a new 'profile' option
>>>>> for generic CPUs, with '-cpu rv64,profile=rva23U64' enabling a whole set
>>>>> of extensions. We would also support enabling optional profile extensions.
>>>>> Yes, we don't have all the profile extensions available yet (neither does
>>>>> the kernel AFAIK) but nothing is stopping us from adding the base
>>>>> framework;
>>>>
>>>> Would we also make incompatible extensions forbidden in the profile CPUs?
>>>
>>> I think we can allow those as long as they're not in conflict with mandatory
>>> profile extensions (not sure if there's a case of that).
>>
>> That seems reasonable to me too.
>>
>>>> If we add the 23 profiles, we should probably also add some other bases. We've got a defacto base in Linux that's rv{32,64}ima (plus some cruft as things have drifted around), I think there's also some interest in the 22 profiles.
>>>
>>> rv32/64ima would be the base of the minimal generic CPUs I proposed below.
>>
>> The Linux base has a few other things like Zicntr, Zifencei, Zicsr, etc -- essentially IMA from back before that stuff was all removed.
>
>
> Hmm I guess we'll need some more thought in how we define a minimal set for
> Linux then.

There's a few bits in the kernel docs about it, but they're all kind of 
ad-hoc because we tend to find out about the ISA changes after they 
happen.  Probably still the best starting point, though.

The best one I could find is the RISCV_HWPROBE_BASE_BEHAVIOR_IMA 
description in <https://docs.kernel.org/riscv/hwprobe.html> (though 
looks like we've got some formatting bugs...)

>>>>> 3 - reduce the generic CPUs up to the minimum required to boot Linux.
>>>>> Yes, a rather arbitrary criteria, but Linux is the most common OS used
>>>>> by RISC-V QEMU currently so we'll go with what most people are using.
>>>>> This would reduce rv64 to "rv64ima".
>>>>
>>>> Have a "minimum CPU required to boot Linux" option seems reasonable. Most users will probably want at least FD and C, though, so maybe we want something like that as well?
>>>
>>> I don't mind making the minimal CPU be rv32/64imafdc.
>>
>> Ya, IDK if there's a right answer.  Maybe someone else has a stronger opinion?  IMO it's kind of just a double-edged sword as to which we pick for the minimum.
>
> As long as we change it and document it it should be ok. Users will still be able
> to enable/disable extensions from the base rv64 CPU as well.
>
>
>>
>>>>> The end result is that we would have a max extensions CPU, a barebones
>>>>> generic CPU that can be fully customizable from the ground up, and
>>>>> a profile option to allow users to boost the generic CPUs with a
>>>>> set of extensions. We will avoid having to deal with generic CPUs having
>>>>> an (apparently) arbitrary set of extensions.
>>>>
>>>> IMO we still want to keep the ability to turn on/off extensions, the profiles are a long way from being concrete and we'll still have a lot of systems floating around that don't line up with them.  Being able to test those configurations in QEMU is a huge benefit for users, even if it's a headache to keep around all the options.
>>>
>>> Absolutely. What I meant is that, today, rv64 has a default set of extensions
>>> that seems arbitrary. It would be good to have more predictability, i.e. have
>>> a fixed set of extensions (like rv64imafdc) that users can use as a base to
>>> enable/disable extensions.
>>>
>>> The same thing can be done with profiles. I don't see a reason to forbid users
>>> to do this:
>>>
>>> '-cpu rv64,profile=rva23S64,icbom=false'
>>>
>>> Meaning "I want the rva23S64 profile without Zicbom". The user should be aware that
>>> Zicbom is mandatory for the profile, but since it's being explicitly disabled in the
>>> command line, we can comply. This can be used for debug/development purposes or
>>> the user is doing something silly. As long as it's explicit in the command line
>>> it's fine.
>>
>> Cool, I think I just misunderstood the original post.  I thought you were trying to say we should drop the per-extension properties.  I definitely agree that it's worth taking a look at the base CPUs, what we've got now is kind of just some arbitrary stuff that's left over from when the ISA was a lot simpler.
>>
>> That'd also make the "should we allow incompatible extensions in profiles" questions easier, as we'd essentially just be saying the profile is shorthand for some set of base/extensions.
>>
>> I guess that also brings up what to do with the U/S profiles split.  My guess is that most users just want both to match, so rva23s64 would include all the mandatory rva23u64 base/extensions?
>
> I think so. The profile doc states:
>
> RVA23S64 Mandatory Extensions
>
>   The following unprivileged extensions are mandatory:
>
>    The RVA23S64 mandatory unprivileged extensions include all the mandatory
>    unprivileged extensions in RVA23U64.
>
>
> So '-cpu rv64,profile=rva32S64' would include all the mandatory U64 extensions
> plus all the mandatory S64. We should add both, so S64 will be build on top of
> U64.

Thanks, I'd missed that.  So I think that's an easy one.

>
>
> Thanks,
>
> Daniel
>
>
>
>>
>>> Thanks,
>>>
>>> Daniel
>>>
>>>
>>>>
>>>>> I don't think we'll have too much opposition on adding (1) so I'll get
>>>>> on it. I really would like to hear about (2) and (3). (3) will break
>>>>> users that are using rv64 while counting with that whole set of extensions
>>>>> that are being enabled by default, but this is why we're going to add
>>>>> (2) first.
>>>>>
>>>>>
>>>>> Please comment. Andrea, feel free to give us libvirt/tooling insights.
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Daniel

Re: RFC: QEMU rv64/rv32 CPU, 'max' CPU and profile support

[Andrew Jones]

On Mon, Jul 10, 2023 at 02:53:12PM -0300, Daniel Henrique Barboza wrote:
> Hi,
> 
> We don't have a 'max' cpu type in riscv32 and riscv64. This is a special,
> fictional CPU type that has all possible extensions enabled. It can be
> either an alias for a real CPU or a new CPU type. It's used by tooling
> such as libvirt for capabilities probing.
> 
> Well, in our case it can't be an alias because we don't have any CPU type
> that enables all extensions. Which brings me to my first question:
> what's the criteria for marking a certain extension default true/false
> in rv32/rv64/rv128?
> 
> E.g. this is the riscv,isa DT for rv64:
> 
> rv64imafdch_zicbom_zicboz_zicsr_zifencei_zihintpause_zawrs_zfa_zca_zcd_zba_zbb_zbc_zbs_sstc_svadu
> 
> It has a lot of extensions but it's missing some common ones like svinval
> and smstateen.
> 
> We can add/remove some extensions from these generic CPUs, document it
> and be done with it. I want to propose a different design though:
> 
> 1 - introduce the 'max' CPU. I guess we'll need some prep work in how
> the extensions are being declared/stored in the code. I'll work on
> it;
> 
> 2 - introduce rva23U64/rva23S64. I would like to add a new 'profile' option
> for generic CPUs, with '-cpu rv64,profile=rva23U64' enabling a whole set
> of extensions. We would also support enabling optional profile extensions.
> Yes, we don't have all the profile extensions available yet (neither does
> the kernel AFAIK) but nothing is stopping us from adding the base
> framework;
> 
> 3 - reduce the generic CPUs up to the minimum required to boot Linux.
> Yes, a rather arbitrary criteria, but Linux is the most common OS used
> by RISC-V QEMU currently so we'll go with what most people are using.
> This would reduce rv64 to "rv64ima".
> 
> The end result is that we would have a max extensions CPU, a barebones
> generic CPU that can be fully customizable from the ground up, and
> a profile option to allow users to boost the generic CPUs with a
> set of extensions. We will avoid having to deal with generic CPUs having
> an (apparently) arbitrary set of extensions.
> 
> I don't think we'll have too much opposition on adding (1) so I'll get
> on it. I really would like to hear about (2) and (3). (3) will break
> users that are using rv64 while counting with that whole set of extensions
> that are being enabled by default, but this is why we're going to add
> (2) first.

Everything sounds good to me, once we've agreed on a proper "minimal
Linux supporting" base to map rv64 to. One thing I would change, though,
is rather than have an enum property 'profile', I'd create booleans
rva22S64, rva23S64, ... Keeping cpu properties boolean simplifies
introspection. And, when we have mutually exclusive profiles, QEMU can
error out at cpu feature finalize time, but, when they're not mutually
exclusive, then why not allow more than one to be selected.

Thanks,
drew

Re: RFC: QEMU rv64/rv32 CPU, 'max' CPU and profile support

[Daniel Henrique Barboza]

On 7/11/23 05:30, Andrew Jones wrote:
> On Mon, Jul 10, 2023 at 02:53:12PM -0300, Daniel Henrique Barboza wrote:
>> Hi,
>>
>> We don't have a 'max' cpu type in riscv32 and riscv64. This is a special,
>> fictional CPU type that has all possible extensions enabled. It can be
>> either an alias for a real CPU or a new CPU type. It's used by tooling
>> such as libvirt for capabilities probing.
>>
>> Well, in our case it can't be an alias because we don't have any CPU type
>> that enables all extensions. Which brings me to my first question:
>> what's the criteria for marking a certain extension default true/false
>> in rv32/rv64/rv128?
>>
>> E.g. this is the riscv,isa DT for rv64:
>>
>> rv64imafdch_zicbom_zicboz_zicsr_zifencei_zihintpause_zawrs_zfa_zca_zcd_zba_zbb_zbc_zbs_sstc_svadu
>>
>> It has a lot of extensions but it's missing some common ones like svinval
>> and smstateen.
>>
>> We can add/remove some extensions from these generic CPUs, document it
>> and be done with it. I want to propose a different design though:
>>
>> 1 - introduce the 'max' CPU. I guess we'll need some prep work in how
>> the extensions are being declared/stored in the code. I'll work on
>> it;
>>
>> 2 - introduce rva23U64/rva23S64. I would like to add a new 'profile' option
>> for generic CPUs, with '-cpu rv64,profile=rva23U64' enabling a whole set
>> of extensions. We would also support enabling optional profile extensions.
>> Yes, we don't have all the profile extensions available yet (neither does
>> the kernel AFAIK) but nothing is stopping us from adding the base
>> framework;
>>
>> 3 - reduce the generic CPUs up to the minimum required to boot Linux.
>> Yes, a rather arbitrary criteria, but Linux is the most common OS used
>> by RISC-V QEMU currently so we'll go with what most people are using.
>> This would reduce rv64 to "rv64ima".
>>
>> The end result is that we would have a max extensions CPU, a barebones
>> generic CPU that can be fully customizable from the ground up, and
>> a profile option to allow users to boost the generic CPUs with a
>> set of extensions. We will avoid having to deal with generic CPUs having
>> an (apparently) arbitrary set of extensions.
>>
>> I don't think we'll have too much opposition on adding (1) so I'll get
>> on it. I really would like to hear about (2) and (3). (3) will break
>> users that are using rv64 while counting with that whole set of extensions
>> that are being enabled by default, but this is why we're going to add
>> (2) first.
> 
> Everything sounds good to me, once we've agreed on a proper "minimal
> Linux supporting" base to map rv64 to. One thing I would change, though,
> is rather than have an enum property 'profile', I'd create booleans
> rva22S64, rva23S64, ... Keeping cpu properties boolean simplifies
> introspection. And, when we have mutually exclusive profiles, QEMU can
> error out at cpu feature finalize time, but, when they're not mutually
> exclusive, then why not allow more than one to be selected.

So instead of

'-cpu rv64,profile=rva23s64'

we would have

'-cpu rv64,rva23s64=true'

With the possibility of stacking profiles one after the other, assuming they're
compatible. Looks good to me.


Daniel

> 
> Thanks,
> drew

Re: RFC: QEMU rv64/rv32 CPU, 'max' CPU and profile support

[Andrea Bolognani]

Sorry for being late to the party, and I see that you're already a
few versions in with the actual patches. Hopefully nothing I write
here is going to be too disruptive of the work you've already done :)

On Mon, Jul 10, 2023 at 02:53:12PM -0300, Daniel Henrique Barboza wrote:
> 1 - introduce the 'max' CPU. I guess we'll need some prep work in how
> the extensions are being declared/stored in the code. I'll work on
> it;

Yes please. At least for TCG, this is pretty much a no-brainer and
what most people will probably want to use.

> 2 - introduce rva23U64/rva23S64. I would like to add a new 'profile' option
> for generic CPUs, with '-cpu rv64,profile=rva23U64' enabling a whole set
> of extensions. We would also support enabling optional profile extensions.
> Yes, we don't have all the profile extensions available yet (neither does
> the kernel AFAIK) but nothing is stopping us from adding the base
> framework;

IIUC profiles are basically shorthands for well-defined sets of
extensions. The idea definitely sounds valuable, but I wonder how
we'd expose it from the libvirt XML.

My question would be, why should this be -cpu rv64,profile=rva23U64
instead of just -cpu rva23U64? "Base ISA plus a number of extensions"
pretty much sounds like a named CPU model to me, but I'm sure there
is more to it :)

> 3 - reduce the generic CPUs up to the minimum required to boot Linux.
> Yes, a rather arbitrary criteria, but Linux is the most common OS used
> by RISC-V QEMU currently so we'll go with what most people are using.
> This would reduce rv64 to "rv64ima".

Can't we make the generic CPUs really bare-bones instead?

Not sure what exactly that would look like, but as long as you can
enable specific extensions explicitly I don't see the advantage in
targeting a specific use case (boot Linux), no matter how popular,
instead of providing useful building blocks for custom behavior.

The use case of booting Linux, and really most RISC-V operating
systems, should already be well served by -cpu max for TCG and -cpu
host for KVM.

-- 
Andrea Bolognani / Red Hat / Virtualization

Re: RFC: QEMU rv64/rv32 CPU, 'max' CPU and profile support

[Daniel Henrique Barboza]

On 7/27/23 10:28, Andrea Bolognani wrote:
> Sorry for being late to the party, and I see that you're already a
> few versions in with the actual patches. Hopefully nothing I write
> here is going to be too disruptive of the work you've already done :)

We just finished warming up. National anthem starting in a few seconds.
Whole match ahead of us yet :P

> 
> On Mon, Jul 10, 2023 at 02:53:12PM -0300, Daniel Henrique Barboza wrote:
>> 1 - introduce the 'max' CPU. I guess we'll need some prep work in how
>> the extensions are being declared/stored in the code. I'll work on
>> it;
> 
> Yes please. At least for TCG, this is pretty much a no-brainer and
> what most people will probably want to use.
> 
>> 2 - introduce rva23U64/rva23S64. I would like to add a new 'profile' option
>> for generic CPUs, with '-cpu rv64,profile=rva23U64' enabling a whole set
>> of extensions. We would also support enabling optional profile extensions.
>> Yes, we don't have all the profile extensions available yet (neither does
>> the kernel AFAIK) but nothing is stopping us from adding the base
>> framework;
> 
> IIUC profiles are basically shorthands for well-defined sets of
> extensions. The idea definitely sounds valuable, but I wonder how
> we'd expose it from the libvirt XML.

I was thinking about adding them like CPU features, maybe? Like it is
already done for hyperv, gic, sev and so on. So we would have, e.g.:

<features>
   <profile>rva23U64</profile>
</features>


> 
> My question would be, why should this be -cpu rv64,profile=rva23U64
> instead of just -cpu rva23U64? "Base ISA plus a number of extensions"
> pretty much sounds like a named CPU model to me, but I'm sure there
> is more to it :)

We can do what makes sense since it's a virtual CPU model anyways.

The advantage of implementing profiles using a new 'profile' flag instead
of using a new CPU model is that we can pile up profiles on top of
each other using the existing generic CPU (rv64) which is also the default
CPU for some time. Implementing a new profile would then be a matter of
adding a new flag instead of a new CPU model.


> 
>> 3 - reduce the generic CPUs up to the minimum required to boot Linux.
>> Yes, a rather arbitrary criteria, but Linux is the most common OS used
>> by RISC-V QEMU currently so we'll go with what most people are using.
>> This would reduce rv64 to "rv64ima".
> 
> Can't we make the generic CPUs really bare-bones instead?
> 
> Not sure what exactly that would look like, but as long as you can
> enable specific extensions explicitly I don't see the advantage in
> targeting a specific use case (boot Linux), no matter how popular,
> instead of providing useful building blocks for custom behavior.

I agree that it's kind of lame to play favorites here and assume that Linux
is the only case that we care about. There's a high chance that other OSes
like FreeBSD might require even less extensions than Linux, so why can't
we use FreeBSD minimal extension set then?

The idea here is less about 'make rv64 a minimal CPU' and more about 'the
generic CPUs should have a predictable extension set'. Having a criteria
like 'It needs to boot Linux' grant that we will not break (too many)
existing users of these generics default CPUs that became used to a random
set of extensions and, all of a sudden, their setup stop working.

> 
> The use case of booting Linux, and really most RISC-V operating
> systems, should already be well served by -cpu max for TCG and -cpu
> host for KVM.

That's a fair point.

What if we:

- leave the rv64 CPU as is. We'll document the default enabled extensions the
CPU already has and leave it at that. We'll have to refrain from enabling
more default extensions in it though;

- if required, add a new 'barebones' CPU. Call it 'rv64min' or any other
suggestive name. Since it's a new CPU we can really strip it out without
worrying about breaking existing setups. Can be a good CPU for debugging;

- all future profiles are implemented as flags, similar to what we already
do with regular extensions. That will enable us to do things like '-cpu
rv64,rva22U64=true,rva23S64=true' and let user combo profiles that might
be compatible with each other.

And, reproducing here a question LIU Zhiwei asked me in the QEMU SIG meeting,
I believe enabling a profile should do just that - enable a certain set of
extensions. If the user wants to enable a profile and then disable some
mandatory extensions, have at it.


Let me know your thoughts. I have one more prep series to send before
starting the profile implementation, so we have time.


Thanks,

Daniel




> 

Re: RFC: QEMU rv64/rv32 CPU, 'max' CPU and profile support

[Andrea Bolognani]

On Thu, Jul 27, 2023 at 08:04:07PM -0300, Daniel Henrique Barboza wrote:
> On 7/27/23 10:28, Andrea Bolognani wrote:
> > Sorry for being late to the party, and I see that you're already a
> > few versions in with the actual patches. Hopefully nothing I write
> > here is going to be too disruptive of the work you've already done :)
>
> We just finished warming up. National anthem starting in a few seconds.
> Whole match ahead of us yet :P

Good, because as you can see I'm clearly taking my sweet time in
between replies :/ Once again, sorry about that.

> > My question would be, why should this be -cpu rv64,profile=rva23U64
> > instead of just -cpu rva23U64? "Base ISA plus a number of extensions"
> > pretty much sounds like a named CPU model to me, but I'm sure there
> > is more to it :)
>
> We can do what makes sense since it's a virtual CPU model anyways.
>
> The advantage of implementing profiles using a new 'profile' flag instead
> of using a new CPU model is that we can pile up profiles on top of
> each other using the existing generic CPU (rv64) which is also the default
> CPU for some time. Implementing a new profile would then be a matter of
> adding a new flag instead of a new CPU model.

Is there a concrete use case for layering profiles like that, though?

I've given a quick look to the specification for RVA(20|22|23)U64 and
it looks like subsequent version of each profile are completely
additive: extensions that were optional are now mandatory, new
extensions are introduced as either mandatory or optional. This is in
line with what I was expecting going in.

So, why would I want to do something like

  -cpu rv64,profile=rva20u64,profile=rva23u64

when the latter is a superset of the former? It seems that it would
achieve exactly the same as

  -cpu rv64,profile=rva23u64

but with extra steps.

> > > 3 - reduce the generic CPUs up to the minimum required to boot Linux.
> > > Yes, a rather arbitrary criteria, but Linux is the most common OS used
> > > by RISC-V QEMU currently so we'll go with what most people are using.
> > > This would reduce rv64 to "rv64ima".
> >
> > Can't we make the generic CPUs really bare-bones instead?
> >
> > Not sure what exactly that would look like, but as long as you can
> > enable specific extensions explicitly I don't see the advantage in
> > targeting a specific use case (boot Linux), no matter how popular,
> > instead of providing useful building blocks for custom behavior.
>
> I agree that it's kind of lame to play favorites here and assume that Linux
> is the only case that we care about. There's a high chance that other OSes
> like FreeBSD might require even less extensions than Linux, so why can't
> we use FreeBSD minimal extension set then?
>
> The idea here is less about 'make rv64 a minimal CPU' and more about 'the
> generic CPUs should have a predictable extension set'. Having a criteria
> like 'It needs to boot Linux' grant that we will not break (too many)
> existing users of these generics default CPUs that became used to a random
> set of extensions and, all of a sudden, their setup stop working.

IMO using "can boot Linux" as the criteria is directly at odds with
having "predictable set of extensions" as the goal.

As Linux changes, so will its requirements in terms of extensions. I
don't think that you can resonably work off the assumption that the
set of extensions needed to boot Linux today will be sufficient to
boot Linux 5-10 years from now.

> What if we:
>
> - leave the rv64 CPU as is. We'll document the default enabled extensions the
> CPU already has and leave it at that. We'll have to refrain from enabling
> more default extensions in it though;
>
> - if required, add a new 'barebones' CPU. Call it 'rv64min' or any other
> suggestive name. Since it's a new CPU we can really strip it out without
> worrying about breaking existing setups. Can be a good CPU for debugging;
>
> - all future profiles are implemented as flags, similar to what we already
> do with regular extensions. That will enable us to do things like '-cpu
> rv64,rva22U64=true,rva23S64=true' and let user combo profiles that might
> be compatible with each other.
>
> And, reproducing here a question LIU Zhiwei asked me in the QEMU SIG meeting,
> I believe enabling a profile should do just that - enable a certain set of
> extensions. If the user wants to enable a profile and then disable some
> mandatory extensions, have at it.

Yeah, I agree that it should be possible for users to go out of their
way and create whatever FrankenCPU they like by picking a profile and
then enabling/disabling arbitrary extensions.

I expect that users for this feature are initially going to be
primarily developers; at some point, libvirt's CPU driver will likely
need to use it to define synthetic CPU models that are migratable
between machines with different hardware revisions or coming from
heterogeneous vendors. But that's probably a few more years down the
line :)

> Let me know your thoughts. I have one more prep series to send before
> starting the profile implementation, so we have time.

My opinion so far remains that profiles should be exposed as named
CPU models.

Regarding the fate of rv64, it would make sense to me to turn it into
the Minimum Viable CPU, to use as base for completely custom CPU
models. However, due to backwards compatibility concerns that would
probably have to happen quite some time after the max CPU model has
been introduced, so freezing it in its current state feels like a
sensible idea.

-- 
Andrea Bolognani / Red Hat / Virtualization

Re: RFC: QEMU rv64/rv32 CPU, 'max' CPU and profile support

[Andrew Jones]

On Thu, Sep 07, 2023 at 10:51:33AM -0500, Andrea Bolognani wrote:
> On Thu, Jul 27, 2023 at 08:04:07PM -0300, Daniel Henrique Barboza wrote:
> > On 7/27/23 10:28, Andrea Bolognani wrote:
> > > Sorry for being late to the party, and I see that you're already a
> > > few versions in with the actual patches. Hopefully nothing I write
> > > here is going to be too disruptive of the work you've already done :)
> >
> > We just finished warming up. National anthem starting in a few seconds.
> > Whole match ahead of us yet :P
> 
> Good, because as you can see I'm clearly taking my sweet time in
> between replies :/ Once again, sorry about that.
> 
> > > My question would be, why should this be -cpu rv64,profile=rva23U64
> > > instead of just -cpu rva23U64? "Base ISA plus a number of extensions"
> > > pretty much sounds like a named CPU model to me, but I'm sure there
> > > is more to it :)
> >
> > We can do what makes sense since it's a virtual CPU model anyways.
> >
> > The advantage of implementing profiles using a new 'profile' flag instead
> > of using a new CPU model is that we can pile up profiles on top of
> > each other using the existing generic CPU (rv64) which is also the default
> > CPU for some time. Implementing a new profile would then be a matter of
> > adding a new flag instead of a new CPU model.
> 
> Is there a concrete use case for layering profiles like that, though?
> 
> I've given a quick look to the specification for RVA(20|22|23)U64 and
> it looks like subsequent version of each profile are completely
> additive: extensions that were optional are now mandatory, new
> extensions are introduced as either mandatory or optional. This is in
> line with what I was expecting going in.
> 
> So, why would I want to do something like
> 
>   -cpu rv64,profile=rva20u64,profile=rva23u64
> 
> when the latter is a superset of the former? It seems that it would
> achieve exactly the same as
> 
>   -cpu rv64,profile=rva23u64
> 
> but with extra steps.

I don't see anything in the profile spec that states we can count on later
profiles being supersets of older profiles, which means we cannot. Also,
I'd guess vendors will want to implement as broad a set as possible of
profiles and then list them all when advertising. So, IMO, something like
'-cpu rv64,rva20u64=on,rva23u64=on' would be a more natural way to model a
cpu which advertises being compatible with those two profiles.

(I swapped the 'profile=' to a '=on' syntax, because I think each profile
should be a boolean "virtual" extension.)

I'm also not sure of the benefits of using a cpu model vs. a base model
and a bunch of virtual profile extensions. If it's because it's a hassle
to collect extensions that should be enabled, then I think we'll have to
live with that for unnamed riscv cpu models. Profiles will always lag new
extensions which users will want to enable, so users will be adding plenty
of cpu properties anyway. Of course users that don't care about being
explicit have 'max' to avoid all that.

> 
> > > > 3 - reduce the generic CPUs up to the minimum required to boot Linux.
> > > > Yes, a rather arbitrary criteria, but Linux is the most common OS used
> > > > by RISC-V QEMU currently so we'll go with what most people are using.
> > > > This would reduce rv64 to "rv64ima".
> > >
> > > Can't we make the generic CPUs really bare-bones instead?
> > >
> > > Not sure what exactly that would look like, but as long as you can
> > > enable specific extensions explicitly I don't see the advantage in
> > > targeting a specific use case (boot Linux), no matter how popular,
> > > instead of providing useful building blocks for custom behavior.
> >
> > I agree that it's kind of lame to play favorites here and assume that Linux
> > is the only case that we care about. There's a high chance that other OSes
> > like FreeBSD might require even less extensions than Linux, so why can't
> > we use FreeBSD minimal extension set then?
> >
> > The idea here is less about 'make rv64 a minimal CPU' and more about 'the
> > generic CPUs should have a predictable extension set'. Having a criteria
> > like 'It needs to boot Linux' grant that we will not break (too many)
> > existing users of these generics default CPUs that became used to a random
> > set of extensions and, all of a sudden, their setup stop working.
> 
> IMO using "can boot Linux" as the criteria is directly at odds with
> having "predictable set of extensions" as the goal.
> 
> As Linux changes, so will its requirements in terms of extensions. I
> don't think that you can resonably work off the assumption that the
> set of extensions needed to boot Linux today will be sufficient to
> boot Linux 5-10 years from now.

That's a good point. Thinking about this some more I think we should
freeze and deprecate 'rv32' and 'rv64' (which is too bad, since those
are good names). Then, we should introduce new 32 and 64 cpu models
which implement specific, released, unpriv and priv, ISA specs and
do not have any extensions enabled. The cpu model names would capture
the spec version somehow, hopefully someway that isn't too verbose.

(To avoid being verbose, we'll likely have to create our own versioning
scheme which maps to the spec versioning, since it's a date, or maybe
better would be to use a spec git commit hash. So, something like
base64v1, base64v2, ... where base64v* are documented to be specific
commits of the spec and 'base64' would always alias the latest base64v*
version. (Maybe 'base' is a bad name, though, since 'base64' is also
a core utility. I'm already missing 'rv64'...))

Thanks,
drew

Re: RFC: QEMU rv64/rv32 CPU, 'max' CPU and profile support

[Andrew Jones]

On Fri, Sep 08, 2023 at 10:39:45AM +0200, Andrew Jones wrote:
> On Thu, Sep 07, 2023 at 10:51:33AM -0500, Andrea Bolognani wrote:
> > On Thu, Jul 27, 2023 at 08:04:07PM -0300, Daniel Henrique Barboza wrote:
> > > On 7/27/23 10:28, Andrea Bolognani wrote:
> > > > Sorry for being late to the party, and I see that you're already a
> > > > few versions in with the actual patches. Hopefully nothing I write
> > > > here is going to be too disruptive of the work you've already done :)
> > >
> > > We just finished warming up. National anthem starting in a few seconds.
> > > Whole match ahead of us yet :P
> > 
> > Good, because as you can see I'm clearly taking my sweet time in
> > between replies :/ Once again, sorry about that.
> > 
> > > > My question would be, why should this be -cpu rv64,profile=rva23U64
> > > > instead of just -cpu rva23U64? "Base ISA plus a number of extensions"
> > > > pretty much sounds like a named CPU model to me, but I'm sure there
> > > > is more to it :)
> > >
> > > We can do what makes sense since it's a virtual CPU model anyways.
> > >
> > > The advantage of implementing profiles using a new 'profile' flag instead
> > > of using a new CPU model is that we can pile up profiles on top of
> > > each other using the existing generic CPU (rv64) which is also the default
> > > CPU for some time. Implementing a new profile would then be a matter of
> > > adding a new flag instead of a new CPU model.
> > 
> > Is there a concrete use case for layering profiles like that, though?
> > 
> > I've given a quick look to the specification for RVA(20|22|23)U64 and
> > it looks like subsequent version of each profile are completely
> > additive: extensions that were optional are now mandatory, new
> > extensions are introduced as either mandatory or optional. This is in
> > line with what I was expecting going in.
> > 
> > So, why would I want to do something like
> > 
> >   -cpu rv64,profile=rva20u64,profile=rva23u64
> > 
> > when the latter is a superset of the former? It seems that it would
> > achieve exactly the same as
> > 
> >   -cpu rv64,profile=rva23u64
> > 
> > but with extra steps.
> 
> I don't see anything in the profile spec that states we can count on later
> profiles being supersets of older profiles, which means we cannot. Also,

I just found a potential example of why we cannot. A recent mail thread[1]
stated some vendors are considering dropping support for the C extension
on "higher end" platforms. If this were to happen, then a profile may be
proposed where C is optional. In that case, platforms that still have C
could state they are compatible with the older profiles where it's
mandated and also the new profile where it's optional, whereas the
platforms that don't have C will only state the new profile.

[1] https://groups.google.com/a/riscv.org/g/fw-exchange/c/qxnKKS4jhLk

Thanks,
drew

Re: RFC: QEMU rv64/rv32 CPU, 'max' CPU and profile support

[Andrea Bolognani]

On Fri, Sep 08, 2023 at 11:13:54AM +0200, Andrew Jones wrote:
> On Fri, Sep 08, 2023 at 10:39:45AM +0200, Andrew Jones wrote:
> > On Thu, Sep 07, 2023 at 10:51:33AM -0500, Andrea Bolognani wrote:
> > > So, why would I want to do something like
> > >
> > >   -cpu rv64,profile=rva20u64,profile=rva23u64
> > >
> > > when the latter is a superset of the former? It seems that it would
> > > achieve exactly the same as
> > >
> > >   -cpu rv64,profile=rva23u64
> > >
> > > but with extra steps.
> >
> > I don't see anything in the profile spec that states we can count on later
> > profiles being supersets of older profiles, which means we cannot.
>
> I just found a potential example of why we cannot. A recent mail thread[1]
> stated some vendors are considering dropping support for the C extension
> on "higher end" platforms. If this were to happen, then a profile may be
> proposed where C is optional. In that case, platforms that still have C
> could state they are compatible with the older profiles where it's
> mandated and also the new profile where it's optional, whereas the
> platforms that don't have C will only state the new profile.
>
> [1] https://groups.google.com/a/riscv.org/g/fw-exchange/c/qxnKKS4jhLk

I can't access the URL, but I get the gist. Point taken about
profiles not necessarily being purely additive, though I don't think
this invalidates my criticism of the approach.

> > Also,
> > I'd guess vendors will want to implement as broad a set as possible of
> > profiles and then list them all when advertising. So, IMO, something like
> > '-cpu rv64,rva20u64=on,rva23u64=on' would be a more natural way to model a
> > cpu which advertises being compatible with those two profiles.

Wouldn't that be exposed as a named vendor CPU rather than something
that users have to assemble themselves by layering profiles?

Note that I have zero issues with the implementation of said named
vendor CPU being, internally, handled effectively the same way as
what you've just described. I'm simply unconvinced that we should
expose this mechanism to users.

> > (I swapped the 'profile=' to a '=on' syntax, because I think each profile
> > should be a boolean "virtual" extension.)

On one hand, I like this because it avoids having to invent a new
mechanism. On the other, I dislike it because it overloads an
existing mechanism O:-)

> > I'm also not sure of the benefits of using a cpu model vs. a base model
> > and a bunch of virtual profile extensions. If it's because it's a hassle
> > to collect extensions that should be enabled, then I think we'll have to
> > live with that for unnamed riscv cpu models. Profiles will always lag new
> > extensions which users will want to enable, so users will be adding plenty
> > of cpu properties anyway. Of course users that don't care about being
> > explicit have 'max' to avoid all that.

I feel that this validates my stance on the matter.

If you're working on defining a new CPU model and need fine-grained
control over the exact set of extensions, then you can already do
that by explicitly flipping each one of them on or off, and adding
the ability to layer profiles doesn't add much value except possibly
removing a bit of verbosity. Not particularly compelling IMO.

If you're testing an application and want to ensure it works fine on
CPUs that implement a specific profile, e.g. rva20u64 because even
older hardware should be able to run it, then you're not going to
need or indeed want to layer additional profiles on top of that. So a
regular named CPU model will work just fine.

  (This also gets really interesting when you start thinking about
  extensions that are optional for the profile... Maybe we need
  something like '-cpu rva23u64,optional-extensions=off' so that
  testing against the baseline for a profile doesn't require combing
  through the spec and disabling extensions manually?)

If you're someone who just needs a RISC-V VM, then you're either
going to be using 'max' or a named vendor CPU and profiles will not
enter the picture at all.

-- 
Andrea Bolognani / Red Hat / Virtualization

Re: RFC: QEMU rv64/rv32 CPU, 'max' CPU and profile support

[Andrew Jones]

On Fri, Sep 08, 2023 at 03:28:12AM -0700, Andrea Bolognani wrote:
> On Fri, Sep 08, 2023 at 11:13:54AM +0200, Andrew Jones wrote:
> > On Fri, Sep 08, 2023 at 10:39:45AM +0200, Andrew Jones wrote:
> > > On Thu, Sep 07, 2023 at 10:51:33AM -0500, Andrea Bolognani wrote:
> > > > So, why would I want to do something like
> > > >
> > > >   -cpu rv64,profile=rva20u64,profile=rva23u64
> > > >
> > > > when the latter is a superset of the former? It seems that it would
> > > > achieve exactly the same as
> > > >
> > > >   -cpu rv64,profile=rva23u64
> > > >
> > > > but with extra steps.
> > >
> > > I don't see anything in the profile spec that states we can count on later
> > > profiles being supersets of older profiles, which means we cannot.
> >
> > I just found a potential example of why we cannot. A recent mail thread[1]
> > stated some vendors are considering dropping support for the C extension
> > on "higher end" platforms. If this were to happen, then a profile may be
> > proposed where C is optional. In that case, platforms that still have C
> > could state they are compatible with the older profiles where it's
> > mandated and also the new profile where it's optional, whereas the
> > platforms that don't have C will only state the new profile.
> >
> > [1] https://groups.google.com/a/riscv.org/g/fw-exchange/c/qxnKKS4jhLk
> 
> I can't access the URL, but I get the gist. Point taken about
> profiles not necessarily being purely additive, though I don't think
> this invalidates my criticism of the approach.
> 
> > > Also,
> > > I'd guess vendors will want to implement as broad a set as possible of
> > > profiles and then list them all when advertising. So, IMO, something like
> > > '-cpu rv64,rva20u64=on,rva23u64=on' would be a more natural way to model a
> > > cpu which advertises being compatible with those two profiles.
> 
> Wouldn't that be exposed as a named vendor CPU rather than something

Probably eventually, but I think it'd be nice to translate a CPU
description to a QEMU command line without having to do anything other
than specify a base CPU model and a list of extensions, where most of the
extensions could get enabled through the selection of profiles without
even having to know which ones.

> that users have to assemble themselves by layering profiles?

Combing multiple profiles should be considered as creating a union of the
mandatory extensions each profile represents, rather than as layering.

Let's take the hypothetical case of C being dropped from rva24u64 as an
example. When specifying both rva22u64 and rva24u64
(-cpu min64,rva22u64=on,rva24u64=on) the user will get C enabled without
even having to know about it, whereas if the user could only select the
latest profile (-cpu rva24u64), C would not get enabled unless the user
was aware that it needed to be explicitly enabled (-cpu rva24u64,c=on).

(I changed the name of the base CPU type in the example above to 'min64'.
More on that below.)

> 
> Note that I have zero issues with the implementation of said named
> vendor CPU being, internally, handled effectively the same way as
> what you've just described. I'm simply unconvinced that we should
> expose this mechanism to users.
> 
> > > (I swapped the 'profile=' to a '=on' syntax, because I think each profile
> > > should be a boolean "virtual" extension.)
> 
> On one hand, I like this because it avoids having to invent a new
> mechanism. On the other, I dislike it because it overloads an
> existing mechanism O:-)
> 
> > > I'm also not sure of the benefits of using a cpu model vs. a base model
> > > and a bunch of virtual profile extensions. If it's because it's a hassle
> > > to collect extensions that should be enabled, then I think we'll have to
> > > live with that for unnamed riscv cpu models. Profiles will always lag new
> > > extensions which users will want to enable, so users will be adding plenty
> > > of cpu properties anyway. Of course users that don't care about being
> > > explicit have 'max' to avoid all that.
> 
> I feel that this validates my stance on the matter.
> 
> If you're working on defining a new CPU model and need fine-grained
> control over the exact set of extensions, then you can already do
> that by explicitly flipping each one of them on or off, and adding
> the ability to layer profiles doesn't add much value except possibly
> removing a bit of verbosity. Not particularly compelling IMO.

I think it's compelling, because the extension lists that profiles provide
are long and mostly uninteresting. For example, how often do we want to
type and think about imafd_Zicsr_Zifencei? I think we'd mostly rather take
those for granted, and we can, because we just specify 'g' instead. Indeed
the profile spec even points out that using profiles as a way to deal with
"unwieldy" ISA strings is another motivation for them. A RISC-V QEMU CPU
command line is effectively an ISA string, so I think it's appropriate to
apply profiles to it as well.

> 
> If you're testing an application and want to ensure it works fine on
> CPUs that implement a specific profile, e.g. rva20u64 because even
> older hardware should be able to run it, then you're not going to
> need or indeed want to layer additional profiles on top of that. So a
> regular named CPU model will work just fine.

This line of reasoning makes sense if profiles only layer, but, as I tried
to illustrate above, we should consider profile combining to be the
forming of extension unions, where even the older profiles can contribute
to the set.

> 
>   (This also gets really interesting when you start thinking about
>   extensions that are optional for the profile... Maybe we need
>   something like '-cpu rva23u64,optional-extensions=off' so that
>   testing against the baseline for a profile doesn't require combing
>   through the spec and disabling extensions manually?)

I don't think the profiles (whether they are CPU virtual extensions or
CPU models) should enable the optional extensions by default. When
software targets a profile (or set of profiles with the virtual
extension approach) then it should never assume the optional extensions
will be present. Developers should need to consciously include those
extensions when creating the QEMU platform used for testing.

> 
> If you're someone who just needs a RISC-V VM, then you're either
> going to be using 'max' or a named vendor CPU and profiles will not
> enter the picture at all.

I agree, but I think we need to ensure developers can easily build more
specific platforms too, including those that combine profiles.


Now somewhat switching topics...

Thinking about this more today, I think we should have two non-vendor CPU
models ('min' and 'max'). We also need to handle xlen, though, so I
suggest we have min32, min64, max32 and max64 (and someday min128 and
max128), but those would just be convenience CPU model names. The same
CPU models could be created with 'min,xlen32=on', 'min,xlen64=on',
'max,xlen32=on', and 'max,xlen64=on' instead (and one could replace xlen*
with both an mxlen* and an sxlen* when they don't match).

Besides the *xlen* boolean properties, 'min' and 'max', would both also
get 'version1', 'version2', ... boolean properties which map to the ISA
spec version(s) supported. (min32, min64, max32, and max64 would always
implement the latest version in order to remain useful shorthands.)

In the end users could choose between
 - vendor named models
 - min/max,xlen*,version* + profiles/extensions...
 - min32/max32/min64/max64 + profiles/extensions...
 - host (for KVM, and max/max32/max64 could all alias 'host')

'any', 'rv32', and 'rv64' get deprecated.

Thanks,
drew

Re: RFC: QEMU rv64/rv32 CPU, 'max' CPU and profile support

[Daniel Henrique Barboza]

On 9/8/23 11:10, Andrew Jones wrote:
> On Fri, Sep 08, 2023 at 03:28:12AM -0700, Andrea Bolognani wrote:
>> On Fri, Sep 08, 2023 at 11:13:54AM +0200, Andrew Jones wrote:
>>> On Fri, Sep 08, 2023 at 10:39:45AM +0200, Andrew Jones wrote:
>>>> On Thu, Sep 07, 2023 at 10:51:33AM -0500, Andrea Bolognani wrote:
>>>>> So, why would I want to do something like
>>>>>
>>>>>    -cpu rv64,profile=rva20u64,profile=rva23u64
>>>>>
>>>>> when the latter is a superset of the former? It seems that it would
>>>>> achieve exactly the same as
>>>>>
>>>>>    -cpu rv64,profile=rva23u64
>>>>>
>>>>> but with extra steps.
>>>>
>>>> I don't see anything in the profile spec that states we can count on later
>>>> profiles being supersets of older profiles, which means we cannot.
>>>
>>> I just found a potential example of why we cannot. A recent mail thread[1]
>>> stated some vendors are considering dropping support for the C extension
>>> on "higher end" platforms. If this were to happen, then a profile may be
>>> proposed where C is optional. In that case, platforms that still have C
>>> could state they are compatible with the older profiles where it's
>>> mandated and also the new profile where it's optional, whereas the
>>> platforms that don't have C will only state the new profile.
>>>
>>> [1] https://groups.google.com/a/riscv.org/g/fw-exchange/c/qxnKKS4jhLk
>>
>> I can't access the URL, but I get the gist. Point taken about
>> profiles not necessarily being purely additive, though I don't think
>> this invalidates my criticism of the approach.
>>
>>>> Also,
>>>> I'd guess vendors will want to implement as broad a set as possible of
>>>> profiles and then list them all when advertising. So, IMO, something like
>>>> '-cpu rv64,rva20u64=on,rva23u64=on' would be a more natural way to model a
>>>> cpu which advertises being compatible with those two profiles.
>>
>> Wouldn't that be exposed as a named vendor CPU rather than something
> 
> Probably eventually, but I think it'd be nice to translate a CPU
> description to a QEMU command line without having to do anything other
> than specify a base CPU model and a list of extensions, where most of the
> extensions could get enabled through the selection of profiles without
> even having to know which ones.
> 
>> that users have to assemble themselves by layering profiles?
> 
> Combing multiple profiles should be considered as creating a union of the
> mandatory extensions each profile represents, rather than as layering.
> 
> Let's take the hypothetical case of C being dropped from rva24u64 as an
> example. When specifying both rva22u64 and rva24u64
> (-cpu min64,rva22u64=on,rva24u64=on) the user will get C enabled without
> even having to know about it, whereas if the user could only select the
> latest profile (-cpu rva24u64), C would not get enabled unless the user
> was aware that it needed to be explicitly enabled (-cpu rva24u64,c=on).
> 
> (I changed the name of the base CPU type in the example above to 'min64'.
> More on that below.)
> 
>>
>> Note that I have zero issues with the implementation of said named
>> vendor CPU being, internally, handled effectively the same way as
>> what you've just described. I'm simply unconvinced that we should
>> expose this mechanism to users.
>>
>>>> (I swapped the 'profile=' to a '=on' syntax, because I think each profile
>>>> should be a boolean "virtual" extension.)
>>
>> On one hand, I like this because it avoids having to invent a new
>> mechanism. On the other, I dislike it because it overloads an
>> existing mechanism O:-)
>>
>>>> I'm also not sure of the benefits of using a cpu model vs. a base model
>>>> and a bunch of virtual profile extensions. If it's because it's a hassle
>>>> to collect extensions that should be enabled, then I think we'll have to
>>>> live with that for unnamed riscv cpu models. Profiles will always lag new
>>>> extensions which users will want to enable, so users will be adding plenty
>>>> of cpu properties anyway. Of course users that don't care about being
>>>> explicit have 'max' to avoid all that.
>>
>> I feel that this validates my stance on the matter.
>>
>> If you're working on defining a new CPU model and need fine-grained
>> control over the exact set of extensions, then you can already do
>> that by explicitly flipping each one of them on or off, and adding
>> the ability to layer profiles doesn't add much value except possibly
>> removing a bit of verbosity. Not particularly compelling IMO.
> 
> I think it's compelling, because the extension lists that profiles provide
> are long and mostly uninteresting. For example, how often do we want to
> type and think about imafd_Zicsr_Zifencei? I think we'd mostly rather take
> those for granted, and we can, because we just specify 'g' instead. Indeed
> the profile spec even points out that using profiles as a way to deal with
> "unwieldy" ISA strings is another motivation for them. A RISC-V QEMU CPU
> command line is effectively an ISA string, so I think it's appropriate to
> apply profiles to it as well.
> 
>>
>> If you're testing an application and want to ensure it works fine on
>> CPUs that implement a specific profile, e.g. rva20u64 because even
>> older hardware should be able to run it, then you're not going to
>> need or indeed want to layer additional profiles on top of that. So a
>> regular named CPU model will work just fine.
> 
> This line of reasoning makes sense if profiles only layer, but, as I tried
> to illustrate above, we should consider profile combining to be the
> forming of extension unions, where even the older profiles can contribute
> to the set.
> 
>>
>>    (This also gets really interesting when you start thinking about
>>    extensions that are optional for the profile... Maybe we need
>>    something like '-cpu rva23u64,optional-extensions=off' so that
>>    testing against the baseline for a profile doesn't require combing
>>    through the spec and disabling extensions manually?)
> 
> I don't think the profiles (whether they are CPU virtual extensions or
> CPU models) should enable the optional extensions by default. When
> software targets a profile (or set of profiles with the virtual
> extension approach) then it should never assume the optional extensions
> will be present. Developers should need to consciously include those
> extensions when creating the QEMU platform used for testing.
> 
>>
>> If you're someone who just needs a RISC-V VM, then you're either
>> going to be using 'max' or a named vendor CPU and profiles will not
>> enter the picture at all.
> 
> I agree, but I think we need to ensure developers can easily build more
> specific platforms too, including those that combine profiles.
> 
> 
> Now somewhat switching topics...
> 
> Thinking about this more today, I think we should have two non-vendor CPU
> models ('min' and 'max'). We also need to handle xlen, though, so I
> suggest we have min32, min64, max32 and max64 (and someday min128 and
> max128), but those would just be convenience CPU model names. The same
> CPU models could be created with 'min,xlen32=on', 'min,xlen64=on',
> 'max,xlen32=on', and 'max,xlen64=on' instead (and one could replace xlen*
> with both an mxlen* and an sxlen* when they don't match).

This makes sense when/if we unify the 32 and 64 bit binaries (CCing Alistair for
context). Today we have 32 and 64 bit CPUs in their own binaries (riscv32 and
riscv64) and it's not possible to run a 32 bit CPU with qemu-system-riscv64 because
we do not expose xlen or mxl to be set via command line.


As for the 'max' CPU I'm having second guesses about its name. We have 'rv32' and
'rv64', but 'max' can be either 32 or 64 bit depending on the binary used. This can
be considered an advantage now but it can be confusing if/when we decide to merge
all CPUs in a single binary. I believe a better name would be rv32max/rv64max or
max32/max64. Any thoughts?


Thanks,


Daniel


> 
> Besides the *xlen* boolean properties, 'min' and 'max', would both also
> get 'version1', 'version2', ... boolean properties which map to the ISA
> spec version(s) supported. (min32, min64, max32, and max64 would always
> implement the latest version in order to remain useful shorthands.)
> 
> In the end users could choose between
>   - vendor named models
>   - min/max,xlen*,version* + profiles/extensions...
>   - min32/max32/min64/max64 + profiles/extensions...
>   - host (for KVM, and max/max32/max64 could all alias 'host')
> 
> 'any', 'rv32', and 'rv64' get deprecated.
> 
> Thanks,
> drew

Re: RFC: QEMU rv64/rv32 CPU, 'max' CPU and profile support

[Daniel Henrique Barboza]

On 9/11/23 16:46, Daniel Henrique Barboza wrote:
> 
> 
> On 9/8/23 11:10, Andrew Jones wrote:
>> On Fri, Sep 08, 2023 at 03:28:12AM -0700, Andrea Bolognani wrote:
>>> On Fri, Sep 08, 2023 at 11:13:54AM +0200, Andrew Jones wrote:
>>>> On Fri, Sep 08, 2023 at 10:39:45AM +0200, Andrew Jones wrote:
>>>>> On Thu, Sep 07, 2023 at 10:51:33AM -0500, Andrea Bolognani wrote:
>>>>>> So, why would I want to do something like
>>>>>>
>>>>>>    -cpu rv64,profile=rva20u64,profile=rva23u64
>>>>>>
>>>>>> when the latter is a superset of the former? It seems that it would
>>>>>> achieve exactly the same as
>>>>>>
>>>>>>    -cpu rv64,profile=rva23u64
>>>>>>
>>>>>> but with extra steps.
>>>>>
>>>>> I don't see anything in the profile spec that states we can count on later
>>>>> profiles being supersets of older profiles, which means we cannot.
>>>>
>>>> I just found a potential example of why we cannot. A recent mail thread[1]
>>>> stated some vendors are considering dropping support for the C extension
>>>> on "higher end" platforms. If this were to happen, then a profile may be
>>>> proposed where C is optional. In that case, platforms that still have C
>>>> could state they are compatible with the older profiles where it's
>>>> mandated and also the new profile where it's optional, whereas the
>>>> platforms that don't have C will only state the new profile.
>>>>
>>>> [1] https://groups.google.com/a/riscv.org/g/fw-exchange/c/qxnKKS4jhLk
>>>
>>> I can't access the URL, but I get the gist. Point taken about
>>> profiles not necessarily being purely additive, though I don't think
>>> this invalidates my criticism of the approach.
>>>
>>>>> Also,
>>>>> I'd guess vendors will want to implement as broad a set as possible of
>>>>> profiles and then list them all when advertising. So, IMO, something like
>>>>> '-cpu rv64,rva20u64=on,rva23u64=on' would be a more natural way to model a
>>>>> cpu which advertises being compatible with those two profiles.
>>>
>>> Wouldn't that be exposed as a named vendor CPU rather than something
>>
>> Probably eventually, but I think it'd be nice to translate a CPU
>> description to a QEMU command line without having to do anything other
>> than specify a base CPU model and a list of extensions, where most of the
>> extensions could get enabled through the selection of profiles without
>> even having to know which ones.
>>
>>> that users have to assemble themselves by layering profiles?
>>
>> Combing multiple profiles should be considered as creating a union of the
>> mandatory extensions each profile represents, rather than as layering.
>>
>> Let's take the hypothetical case of C being dropped from rva24u64 as an
>> example. When specifying both rva22u64 and rva24u64
>> (-cpu min64,rva22u64=on,rva24u64=on) the user will get C enabled without
>> even having to know about it, whereas if the user could only select the
>> latest profile (-cpu rva24u64), C would not get enabled unless the user
>> was aware that it needed to be explicitly enabled (-cpu rva24u64,c=on).
>>
>> (I changed the name of the base CPU type in the example above to 'min64'.
>> More on that below.)
>>
>>>
>>> Note that I have zero issues with the implementation of said named
>>> vendor CPU being, internally, handled effectively the same way as
>>> what you've just described. I'm simply unconvinced that we should
>>> expose this mechanism to users.
>>>
>>>>> (I swapped the 'profile=' to a '=on' syntax, because I think each profile
>>>>> should be a boolean "virtual" extension.)
>>>
>>> On one hand, I like this because it avoids having to invent a new
>>> mechanism. On the other, I dislike it because it overloads an
>>> existing mechanism O:-)
>>>
>>>>> I'm also not sure of the benefits of using a cpu model vs. a base model
>>>>> and a bunch of virtual profile extensions. If it's because it's a hassle
>>>>> to collect extensions that should be enabled, then I think we'll have to
>>>>> live with that for unnamed riscv cpu models. Profiles will always lag new
>>>>> extensions which users will want to enable, so users will be adding plenty
>>>>> of cpu properties anyway. Of course users that don't care about being
>>>>> explicit have 'max' to avoid all that.
>>>
>>> I feel that this validates my stance on the matter.
>>>
>>> If you're working on defining a new CPU model and need fine-grained
>>> control over the exact set of extensions, then you can already do
>>> that by explicitly flipping each one of them on or off, and adding
>>> the ability to layer profiles doesn't add much value except possibly
>>> removing a bit of verbosity. Not particularly compelling IMO.
>>
>> I think it's compelling, because the extension lists that profiles provide
>> are long and mostly uninteresting. For example, how often do we want to
>> type and think about imafd_Zicsr_Zifencei? I think we'd mostly rather take
>> those for granted, and we can, because we just specify 'g' instead. Indeed
>> the profile spec even points out that using profiles as a way to deal with
>> "unwieldy" ISA strings is another motivation for them. A RISC-V QEMU CPU
>> command line is effectively an ISA string, so I think it's appropriate to
>> apply profiles to it as well.
>>
>>>
>>> If you're testing an application and want to ensure it works fine on
>>> CPUs that implement a specific profile, e.g. rva20u64 because even
>>> older hardware should be able to run it, then you're not going to
>>> need or indeed want to layer additional profiles on top of that. So a
>>> regular named CPU model will work just fine.
>>
>> This line of reasoning makes sense if profiles only layer, but, as I tried
>> to illustrate above, we should consider profile combining to be the
>> forming of extension unions, where even the older profiles can contribute
>> to the set.
>>
>>>
>>>    (This also gets really interesting when you start thinking about
>>>    extensions that are optional for the profile... Maybe we need
>>>    something like '-cpu rva23u64,optional-extensions=off' so that
>>>    testing against the baseline for a profile doesn't require combing
>>>    through the spec and disabling extensions manually?)
>>
>> I don't think the profiles (whether they are CPU virtual extensions or
>> CPU models) should enable the optional extensions by default. When
>> software targets a profile (or set of profiles with the virtual
>> extension approach) then it should never assume the optional extensions
>> will be present. Developers should need to consciously include those
>> extensions when creating the QEMU platform used for testing.
>>
>>>
>>> If you're someone who just needs a RISC-V VM, then you're either
>>> going to be using 'max' or a named vendor CPU and profiles will not
>>> enter the picture at all.
>>
>> I agree, but I think we need to ensure developers can easily build more
>> specific platforms too, including those that combine profiles.
>>
>>
>> Now somewhat switching topics...
>>
>> Thinking about this more today, I think we should have two non-vendor CPU
>> models ('min' and 'max'). We also need to handle xlen, though, so I
>> suggest we have min32, min64, max32 and max64 (and someday min128 and
>> max128), but those would just be convenience CPU model names. The same
>> CPU models could be created with 'min,xlen32=on', 'min,xlen64=on',
>> 'max,xlen32=on', and 'max,xlen64=on' instead (and one could replace xlen*
>> with both an mxlen* and an sxlen* when they don't match).
> 
> This makes sense when/if we unify the 32 and 64 bit binaries (CCing Alistair for
> context). Today we have 32 and 64 bit CPUs in their own binaries (riscv32 and
> riscv64) and it's not possible to run a 32 bit CPU with qemu-system-riscv64 because
> we do not expose xlen or mxl to be set via command line.
> 
> 
> As for the 'max' CPU I'm having second guesses about its name. We have 'rv32' and
> 'rv64', but 'max' can be either 32 or 64 bit depending on the binary used. This can
> be considered an advantage now but it can be confusing if/when we decide to merge
> all CPUs in a single binary. I believe a better name would be rv32max/rv64max or
> max32/max64. Any thoughts?

Guess I'll answer my own question. Another thing with the 'max' CPU name is that all
other archs uses the same CPU name for a similar concept. Use the same name will spare
libvirt and other tooling from having to deal with every other arch having a 'max' CPU
and risc-v using another name.

We might eventually have to deal with an unified binary with both 32 and 64 bits CPUs
and then 'max' will need to be disambiguated via max32/max64 aliases or xlen properties.
For now we should keep 'max' as is.


Thanks,

Daniel


> 
> 
> Thanks,
> 
> 
> Daniel
> 
> 
>>
>> Besides the *xlen* boolean properties, 'min' and 'max', would both also
>> get 'version1', 'version2', ... boolean properties which map to the ISA
>> spec version(s) supported. (min32, min64, max32, and max64 would always
>> implement the latest version in order to remain useful shorthands.)
>>
>> In the end users could choose between
>>   - vendor named models
>>   - min/max,xlen*,version* + profiles/extensions...
>>   - min32/max32/min64/max64 + profiles/extensions...
>>   - host (for KVM, and max/max32/max64 could all alias 'host')
>>
>> 'any', 'rv32', and 'rv64' get deprecated.
>>
>> Thanks,
>> drew

Re: RFC: QEMU rv64/rv32 CPU, 'max' CPU and profile support

[Andrea Bolognani]

On Mon, Sep 11, 2023 at 07:01:38PM -0300, Daniel Henrique Barboza wrote:
> On 9/11/23 16:46, Daniel Henrique Barboza wrote:
> > On 9/8/23 11:10, Andrew Jones wrote:
> > > On Fri, Sep 08, 2023 at 03:28:12AM -0700, Andrea Bolognani wrote:
> > > > Wouldn't that be exposed as a named vendor CPU rather than something
> > > > that users have to assemble themselves by layering profiles?
> > >
> > > Combing multiple profiles should be considered as creating a union of the
> > > mandatory extensions each profile represents, rather than as layering.
> > >
> > > Let's take the hypothetical case of C being dropped from rva24u64 as an
> > > example. When specifying both rva22u64 and rva24u64
> > > (-cpu min64,rva22u64=on,rva24u64=on) the user will get C enabled without
> > > even having to know about it, whereas if the user could only select the
> > > latest profile (-cpu rva24u64), C would not get enabled unless the user
> > > was aware that it needed to be explicitly enabled (-cpu rva24u64,c=on).

Right, that's what I had in mind when I was talking about layering.
Using the term union is probably more appropriate though :)

> > > > If you're working on defining a new CPU model and need fine-grained
> > > > control over the exact set of extensions, then you can already do
> > > > that by explicitly flipping each one of them on or off, and adding
> > > > the ability to layer profiles doesn't add much value except possibly
> > > > removing a bit of verbosity. Not particularly compelling IMO.
> > >
> > > I think it's compelling, because the extension lists that profiles provide
> > > are long and mostly uninteresting. For example, how often do we want to
> > > type and think about imafd_Zicsr_Zifencei? I think we'd mostly rather take
> > > those for granted, and we can, because we just specify 'g' instead. Indeed
> > > the profile spec even points out that using profiles as a way to deal with
> > > "unwieldy" ISA strings is another motivation for them. A RISC-V QEMU CPU
> > > command line is effectively an ISA string, so I think it's appropriate to
> > > apply profiles to it as well.

I'm not sure this will save a lot of typing unless you're enabling a
lot of profiles which are possibly far apart from each other in terms
of release year, but I also don't have a very strong opposition to
the approach. As you say, we are already effectively implementing it
with virtual extensions like "g", so extending the applicability
further to cover profiles is not too much of a stretch.

I think libvirt will probably need to learn what set of extensions
each virtual extension/profile maps to, but since the definition will
come directly from actual specifications that shouldn't be a big
problem.

> > > >    (This also gets really interesting when you start thinking about
> > > >    extensions that are optional for the profile... Maybe we need
> > > >    something like '-cpu rva23u64,optional-extensions=off' so that
> > > >    testing against the baseline for a profile doesn't require combing
> > > >    through the spec and disabling extensions manually?)
> > >
> > > I don't think the profiles (whether they are CPU virtual extensions or
> > > CPU models) should enable the optional extensions by default. When
> > > software targets a profile (or set of profiles with the virtual
> > > extension approach) then it should never assume the optional extensions
> > > will be present. Developers should need to consciously include those
> > > extensions when creating the QEMU platform used for testing.

Fair enough.

Should there be a way to explicitly enable all possible optional
extensions, then? To allow the developer to verify that the
application works correctly on a minimal implementation of the
profile, but also takes advantage of optional extensions if they are
present? Or would that be achieved by enabling the relevant
extensions manually? Or perhaps by using the max CPU model instead?

> > > Thinking about this more today, I think we should have two non-vendor CPU
> > > models ('min' and 'max'). We also need to handle xlen, though, so I
> > > suggest we have min32, min64, max32 and max64 (and someday min128 and
> > > max128), but those would just be convenience CPU model names. The same
> > > CPU models could be created with 'min,xlen32=on', 'min,xlen64=on',
> > > 'max,xlen32=on', and 'max,xlen64=on' instead (and one could replace xlen*
> > > with both an mxlen* and an sxlen* when they don't match).
> >
> > This makes sense when/if we unify the 32 and 64 bit binaries (CCing Alistair for
> > context). Today we have 32 and 64 bit CPUs in their own binaries (riscv32 and
> > riscv64) and it's not possible to run a 32 bit CPU with qemu-system-riscv64 because
> > we do not expose xlen or mxl to be set via command line.
> >
> >
> > As for the 'max' CPU I'm having second guesses about its name. We have 'rv32' and
> > 'rv64', but 'max' can be either 32 or 64 bit depending on the binary used. This can
> > be considered an advantage now but it can be confusing if/when we decide to merge
> > all CPUs in a single binary. I believe a better name would be rv32max/rv64max or
> > max32/max64. Any thoughts?
>
> Guess I'll answer my own question. Another thing with the 'max' CPU name is that all
> other archs uses the same CPU name for a similar concept. Use the same name will spare
> libvirt and other tooling from having to deal with every other arch having a 'max' CPU
> and risc-v using another name.
>
> We might eventually have to deal with an unified binary with both 32 and 64 bits CPUs
> and then 'max' will need to be disambiguated via max32/max64 aliases or xlen properties.
> For now we should keep 'max' as is.

Agreed.

As you note, when the time comes this is something that all
architectures, not just RISC-V, will have to deal with.

-- 
Andrea Bolognani / Red Hat / Virtualization

Re: RFC: QEMU rv64/rv32 CPU, 'max' CPU and profile support

[Andrew Jones]

On Thu, Sep 14, 2023 at 07:52:10AM -0700, Andrea Bolognani wrote:
> On Mon, Sep 11, 2023 at 07:01:38PM -0300, Daniel Henrique Barboza wrote:
> > On 9/11/23 16:46, Daniel Henrique Barboza wrote:
> > > On 9/8/23 11:10, Andrew Jones wrote:
> > > > On Fri, Sep 08, 2023 at 03:28:12AM -0700, Andrea Bolognani wrote:
> > > > > Wouldn't that be exposed as a named vendor CPU rather than something
> > > > > that users have to assemble themselves by layering profiles?
> > > >
> > > > Combing multiple profiles should be considered as creating a union of the
> > > > mandatory extensions each profile represents, rather than as layering.
> > > >
> > > > Let's take the hypothetical case of C being dropped from rva24u64 as an
> > > > example. When specifying both rva22u64 and rva24u64
> > > > (-cpu min64,rva22u64=on,rva24u64=on) the user will get C enabled without
> > > > even having to know about it, whereas if the user could only select the
> > > > latest profile (-cpu rva24u64), C would not get enabled unless the user
> > > > was aware that it needed to be explicitly enabled (-cpu rva24u64,c=on).
> 
> Right, that's what I had in mind when I was talking about layering.
> Using the term union is probably more appropriate though :)
> 
> > > > > If you're working on defining a new CPU model and need fine-grained
> > > > > control over the exact set of extensions, then you can already do
> > > > > that by explicitly flipping each one of them on or off, and adding
> > > > > the ability to layer profiles doesn't add much value except possibly
> > > > > removing a bit of verbosity. Not particularly compelling IMO.
> > > >
> > > > I think it's compelling, because the extension lists that profiles provide
> > > > are long and mostly uninteresting. For example, how often do we want to
> > > > type and think about imafd_Zicsr_Zifencei? I think we'd mostly rather take
> > > > those for granted, and we can, because we just specify 'g' instead. Indeed
> > > > the profile spec even points out that using profiles as a way to deal with
> > > > "unwieldy" ISA strings is another motivation for them. A RISC-V QEMU CPU
> > > > command line is effectively an ISA string, so I think it's appropriate to
> > > > apply profiles to it as well.
> 
> I'm not sure this will save a lot of typing unless you're enabling a
> lot of profiles which are possibly far apart from each other in terms
> of release year, but I also don't have a very strong opposition to
> the approach. As you say, we are already effectively implementing it
> with virtual extensions like "g", so extending the applicability
> further to cover profiles is not too much of a stretch.
> 
> I think libvirt will probably need to learn what set of extensions
> each virtual extension/profile maps to, but since the definition will
> come directly from actual specifications that shouldn't be a big
> problem.
> 
> > > > >    (This also gets really interesting when you start thinking about
> > > > >    extensions that are optional for the profile... Maybe we need
> > > > >    something like '-cpu rva23u64,optional-extensions=off' so that
> > > > >    testing against the baseline for a profile doesn't require combing
> > > > >    through the spec and disabling extensions manually?)
> > > >
> > > > I don't think the profiles (whether they are CPU virtual extensions or
> > > > CPU models) should enable the optional extensions by default. When
> > > > software targets a profile (or set of profiles with the virtual
> > > > extension approach) then it should never assume the optional extensions
> > > > will be present. Developers should need to consciously include those
> > > > extensions when creating the QEMU platform used for testing.
> 
> Fair enough.
> 
> Should there be a way to explicitly enable all possible optional
> extensions, then? To allow the developer to verify that the
> application works correctly on a minimal implementation of the
> profile, but also takes advantage of optional extensions if they are
> present? Or would that be achieved by enabling the relevant
> extensions manually? Or perhaps by using the max CPU model instead?

We could add a variant of each profile which represents everything the
profile mentions. That could maybe be held off until the need arises,
though, since 'max' may satisfy most users which aren't concerned about
which extensions are mandatory vs. optional.

Thanks,
drew

Re: RFC: QEMU rv64/rv32 CPU, 'max' CPU and profile support

[Palmer Dabbelt]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 4910 bytes --]

On Thu, 14 Sep 2023 08:08:40 PDT (-0700), ajones@ventanamicro.com wrote:
> On Thu, Sep 14, 2023 at 07:52:10AM -0700, Andrea Bolognani wrote:
>> On Mon, Sep 11, 2023 at 07:01:38PM -0300, Daniel Henrique Barboza wrote:
>> > On 9/11/23 16:46, Daniel Henrique Barboza wrote:
>> > > On 9/8/23 11:10, Andrew Jones wrote:
>> > > > On Fri, Sep 08, 2023 at 03:28:12AM -0700, Andrea Bolognani wrote:
>> > > > > Wouldn't that be exposed as a named vendor CPU rather than something
>> > > > > that users have to assemble themselves by layering profiles?
>> > > >
>> > > > Combing multiple profiles should be considered as creating a union of the
>> > > > mandatory extensions each profile represents, rather than as layering.
>> > > >
>> > > > Let's take the hypothetical case of C being dropped from rva24u64 as an
>> > > > example. When specifying both rva22u64 and rva24u64
>> > > > (-cpu min64,rva22u64=on,rva24u64=on) the user will get C enabled without
>> > > > even having to know about it, whereas if the user could only select the
>> > > > latest profile (-cpu rva24u64), C would not get enabled unless the user
>> > > > was aware that it needed to be explicitly enabled (-cpu rva24u64,c=on).
>>
>> Right, that's what I had in mind when I was talking about layering.
>> Using the term union is probably more appropriate though :)
>>
>> > > > > If you're working on defining a new CPU model and need fine-grained
>> > > > > control over the exact set of extensions, then you can already do
>> > > > > that by explicitly flipping each one of them on or off, and adding
>> > > > > the ability to layer profiles doesn't add much value except possibly
>> > > > > removing a bit of verbosity. Not particularly compelling IMO.
>> > > >
>> > > > I think it's compelling, because the extension lists that profiles provide
>> > > > are long and mostly uninteresting. For example, how often do we want to
>> > > > type and think about imafd_Zicsr_Zifencei? I think we'd mostly rather take
>> > > > those for granted, and we can, because we just specify 'g' instead. Indeed
>> > > > the profile spec even points out that using profiles as a way to deal with
>> > > > "unwieldy" ISA strings is another motivation for them. A RISC-V QEMU CPU
>> > > > command line is effectively an ISA string, so I think it's appropriate to
>> > > > apply profiles to it as well.
>>
>> I'm not sure this will save a lot of typing unless you're enabling a
>> lot of profiles which are possibly far apart from each other in terms
>> of release year, but I also don't have a very strong opposition to
>> the approach. As you say, we are already effectively implementing it
>> with virtual extensions like "g", so extending the applicability
>> further to cover profiles is not too much of a stretch.
>>
>> I think libvirt will probably need to learn what set of extensions
>> each virtual extension/profile maps to, but since the definition will
>> come directly from actual specifications that shouldn't be a big
>> problem.
>>
>> > > > >    (This also gets really interesting when you start thinking about
>> > > > >    extensions that are optional for the profile... Maybe we need
>> > > > >    something like '-cpu rva23u64,optional-extensions=off' so that
>> > > > >    testing against the baseline for a profile doesn't require combing
>> > > > >    through the spec and disabling extensions manually?)
>> > > >
>> > > > I don't think the profiles (whether they are CPU virtual extensions or
>> > > > CPU models) should enable the optional extensions by default. When
>> > > > software targets a profile (or set of profiles with the virtual
>> > > > extension approach) then it should never assume the optional extensions
>> > > > will be present. Developers should need to consciously include those
>> > > > extensions when creating the QEMU platform used for testing.
>>
>> Fair enough.
>>
>> Should there be a way to explicitly enable all possible optional
>> extensions, then? To allow the developer to verify that the
>> application works correctly on a minimal implementation of the
>> profile, but also takes advantage of optional extensions if they are
>> present? Or would that be achieved by enabling the relevant
>> extensions manually? Or perhaps by using the max CPU model instead?
>
> We could add a variant of each profile which represents everything the
> profile mentions. That could maybe be held off until the need arises,
> though, since 'max' may satisfy most users which aren't concerned about
> which extensions are mandatory vs. optional.

At the risk of adding to the bikeshedding here:  IMO we should start 
with a small set of CPU types that we know are useful.  We're not going 
to be able to cover everything, hence the options to turn on/off 
extensions/features.  Anything we add needs to go through the 
deprecation process if we remove it, though, so it's a bit of a 
headache.

> Thanks,
> drew

Re: RFC: QEMU rv64/rv32 CPU, 'max' CPU and profile support

[Daniel Henrique Barboza]

On 9/14/23 12:08, Andrew Jones wrote:
> On Thu, Sep 14, 2023 at 07:52:10AM -0700, Andrea Bolognani wrote:
>> On Mon, Sep 11, 2023 at 07:01:38PM -0300, Daniel Henrique Barboza wrote:
>>> On 9/11/23 16:46, Daniel Henrique Barboza wrote:
>>>> On 9/8/23 11:10, Andrew Jones wrote:
>>>>> On Fri, Sep 08, 2023 at 03:28:12AM -0700, Andrea Bolognani wrote:
>>>>>> Wouldn't that be exposed as a named vendor CPU rather than something
>>>>>> that users have to assemble themselves by layering profiles?
>>>>>
>>>>> Combing multiple profiles should be considered as creating a union of the
>>>>> mandatory extensions each profile represents, rather than as layering.
>>>>>
>>>>> Let's take the hypothetical case of C being dropped from rva24u64 as an
>>>>> example. When specifying both rva22u64 and rva24u64
>>>>> (-cpu min64,rva22u64=on,rva24u64=on) the user will get C enabled without
>>>>> even having to know about it, whereas if the user could only select the
>>>>> latest profile (-cpu rva24u64), C would not get enabled unless the user
>>>>> was aware that it needed to be explicitly enabled (-cpu rva24u64,c=on).
>>
>> Right, that's what I had in mind when I was talking about layering.
>> Using the term union is probably more appropriate though :)
>>
>>>>>> If you're working on defining a new CPU model and need fine-grained
>>>>>> control over the exact set of extensions, then you can already do
>>>>>> that by explicitly flipping each one of them on or off, and adding
>>>>>> the ability to layer profiles doesn't add much value except possibly
>>>>>> removing a bit of verbosity. Not particularly compelling IMO.
>>>>>
>>>>> I think it's compelling, because the extension lists that profiles provide
>>>>> are long and mostly uninteresting. For example, how often do we want to
>>>>> type and think about imafd_Zicsr_Zifencei? I think we'd mostly rather take
>>>>> those for granted, and we can, because we just specify 'g' instead. Indeed
>>>>> the profile spec even points out that using profiles as a way to deal with
>>>>> "unwieldy" ISA strings is another motivation for them. A RISC-V QEMU CPU
>>>>> command line is effectively an ISA string, so I think it's appropriate to
>>>>> apply profiles to it as well.
>>
>> I'm not sure this will save a lot of typing unless you're enabling a
>> lot of profiles which are possibly far apart from each other in terms
>> of release year, but I also don't have a very strong opposition to
>> the approach. As you say, we are already effectively implementing it
>> with virtual extensions like "g", so extending the applicability
>> further to cover profiles is not too much of a stretch.
>>
>> I think libvirt will probably need to learn what set of extensions
>> each virtual extension/profile maps to, but since the definition will
>> come directly from actual specifications that shouldn't be a big
>> problem.
>>
>>>>>>     (This also gets really interesting when you start thinking about
>>>>>>     extensions that are optional for the profile... Maybe we need
>>>>>>     something like '-cpu rva23u64,optional-extensions=off' so that
>>>>>>     testing against the baseline for a profile doesn't require combing
>>>>>>     through the spec and disabling extensions manually?)
>>>>>
>>>>> I don't think the profiles (whether they are CPU virtual extensions or
>>>>> CPU models) should enable the optional extensions by default. When
>>>>> software targets a profile (or set of profiles with the virtual
>>>>> extension approach) then it should never assume the optional extensions
>>>>> will be present. Developers should need to consciously include those
>>>>> extensions when creating the QEMU platform used for testing.
>>
>> Fair enough.
>>
>> Should there be a way to explicitly enable all possible optional
>> extensions, then? To allow the developer to verify that the
>> application works correctly on a minimal implementation of the
>> profile, but also takes advantage of optional extensions if they are
>> present? Or would that be achieved by enabling the relevant
>> extensions manually? Or perhaps by using the max CPU model instead?
> 
> We could add a variant of each profile which represents everything the
> profile mentions. That could maybe be held off until the need arises,
> though, since 'max' may satisfy most users which aren't concerned about
> which extensions are mandatory vs. optional.


In my current prototype (not yet sent upstream, hopefully this month) I'm enabling
all possible extensions, both mandatory and optional, with 'rva22u64=true'. Perhaps
we should limit that to enable mandatory exts only, and then add a 'rva22u64-max'
or similar to enable all possible extensions QEMU supports for that profile.


Thanks,


Daniel

> 
> Thanks,
> drew

Re: RFC: QEMU rv64/rv32 CPU, 'max' CPU and profile support

[Andrew Jones]

On Thu, Sep 14, 2023 at 06:11:49PM -0300, Daniel Henrique Barboza wrote:
> 
> 
> On 9/14/23 12:08, Andrew Jones wrote:
> > On Thu, Sep 14, 2023 at 07:52:10AM -0700, Andrea Bolognani wrote:
> > > On Mon, Sep 11, 2023 at 07:01:38PM -0300, Daniel Henrique Barboza wrote:
> > > > On 9/11/23 16:46, Daniel Henrique Barboza wrote:
> > > > > On 9/8/23 11:10, Andrew Jones wrote:
> > > > > > On Fri, Sep 08, 2023 at 03:28:12AM -0700, Andrea Bolognani wrote:
> > > > > > > Wouldn't that be exposed as a named vendor CPU rather than something
> > > > > > > that users have to assemble themselves by layering profiles?
> > > > > > 
> > > > > > Combing multiple profiles should be considered as creating a union of the
> > > > > > mandatory extensions each profile represents, rather than as layering.
> > > > > > 
> > > > > > Let's take the hypothetical case of C being dropped from rva24u64 as an
> > > > > > example. When specifying both rva22u64 and rva24u64
> > > > > > (-cpu min64,rva22u64=on,rva24u64=on) the user will get C enabled without
> > > > > > even having to know about it, whereas if the user could only select the
> > > > > > latest profile (-cpu rva24u64), C would not get enabled unless the user
> > > > > > was aware that it needed to be explicitly enabled (-cpu rva24u64,c=on).
> > > 
> > > Right, that's what I had in mind when I was talking about layering.
> > > Using the term union is probably more appropriate though :)
> > > 
> > > > > > > If you're working on defining a new CPU model and need fine-grained
> > > > > > > control over the exact set of extensions, then you can already do
> > > > > > > that by explicitly flipping each one of them on or off, and adding
> > > > > > > the ability to layer profiles doesn't add much value except possibly
> > > > > > > removing a bit of verbosity. Not particularly compelling IMO.
> > > > > > 
> > > > > > I think it's compelling, because the extension lists that profiles provide
> > > > > > are long and mostly uninteresting. For example, how often do we want to
> > > > > > type and think about imafd_Zicsr_Zifencei? I think we'd mostly rather take
> > > > > > those for granted, and we can, because we just specify 'g' instead. Indeed
> > > > > > the profile spec even points out that using profiles as a way to deal with
> > > > > > "unwieldy" ISA strings is another motivation for them. A RISC-V QEMU CPU
> > > > > > command line is effectively an ISA string, so I think it's appropriate to
> > > > > > apply profiles to it as well.
> > > 
> > > I'm not sure this will save a lot of typing unless you're enabling a
> > > lot of profiles which are possibly far apart from each other in terms
> > > of release year, but I also don't have a very strong opposition to
> > > the approach. As you say, we are already effectively implementing it
> > > with virtual extensions like "g", so extending the applicability
> > > further to cover profiles is not too much of a stretch.
> > > 
> > > I think libvirt will probably need to learn what set of extensions
> > > each virtual extension/profile maps to, but since the definition will
> > > come directly from actual specifications that shouldn't be a big
> > > problem.
> > > 
> > > > > > >     (This also gets really interesting when you start thinking about
> > > > > > >     extensions that are optional for the profile... Maybe we need
> > > > > > >     something like '-cpu rva23u64,optional-extensions=off' so that
> > > > > > >     testing against the baseline for a profile doesn't require combing
> > > > > > >     through the spec and disabling extensions manually?)
> > > > > > 
> > > > > > I don't think the profiles (whether they are CPU virtual extensions or
> > > > > > CPU models) should enable the optional extensions by default. When
> > > > > > software targets a profile (or set of profiles with the virtual
> > > > > > extension approach) then it should never assume the optional extensions
> > > > > > will be present. Developers should need to consciously include those
> > > > > > extensions when creating the QEMU platform used for testing.
> > > 
> > > Fair enough.
> > > 
> > > Should there be a way to explicitly enable all possible optional
> > > extensions, then? To allow the developer to verify that the
> > > application works correctly on a minimal implementation of the
> > > profile, but also takes advantage of optional extensions if they are
> > > present? Or would that be achieved by enabling the relevant
> > > extensions manually? Or perhaps by using the max CPU model instead?
> > 
> > We could add a variant of each profile which represents everything the
> > profile mentions. That could maybe be held off until the need arises,
> > though, since 'max' may satisfy most users which aren't concerned about
> > which extensions are mandatory vs. optional.
> 
> 
> In my current prototype (not yet sent upstream, hopefully this month) I'm enabling
> all possible extensions, both mandatory and optional, with 'rva22u64=true'. Perhaps
> we should limit that to enable mandatory exts only, and then add a 'rva22u64-max'
> or similar to enable all possible extensions QEMU supports for that profile.

Yes, I would only enable mandatory extensions, otherwise the concept of
the profile becomes less useful. Most users will want to use a base cpu
type and profiles to remove extensions which they would get with the 'max'
cpu type (i.e. they would use profiles to create a platform with a known
set of extensions). Optional extensions are not generally known to be on a
platform. Users should enable those extensions themselves when they know
the specific platform they're modeling has them.

Whether or not we also offer a profile variant that includes the optional
extensions is debatable. I'd probably defer their creation for now. We
can always define them later.

Thanks,
drew

Re: RFC: QEMU rv64/rv32 CPU, 'max' CPU and profile support

[Andrew Jones]

On Thu, Sep 14, 2023 at 11:09:57AM -0700, Palmer Dabbelt wrote:
> On Thu, 14 Sep 2023 08:08:40 PDT (-0700), ajones@ventanamicro.com wrote:
> > On Thu, Sep 14, 2023 at 07:52:10AM -0700, Andrea Bolognani wrote:
> >> On Mon, Sep 11, 2023 at 07:01:38PM -0300, Daniel Henrique Barboza wrote:
> >> > On 9/11/23 16:46, Daniel Henrique Barboza wrote:
> >> > > On 9/8/23 11:10, Andrew Jones wrote:
> >> > > > On Fri, Sep 08, 2023 at 03:28:12AM -0700, Andrea Bolognani wrote:
> >> > > > > Wouldn't that be exposed as a named vendor CPU rather than something
> >> > > > > that users have to assemble themselves by layering profiles?
> >> > > >
> >> > > > Combing multiple profiles should be considered as creating a union of the
> >> > > > mandatory extensions each profile represents, rather than as layering.
> >> > > >
> >> > > > Let's take the hypothetical case of C being dropped from rva24u64 as an
> >> > > > example. When specifying both rva22u64 and rva24u64
> >> > > > (-cpu min64,rva22u64=on,rva24u64=on) the user will get C enabled without
> >> > > > even having to know about it, whereas if the user could only select the
> >> > > > latest profile (-cpu rva24u64), C would not get enabled unless the user
> >> > > > was aware that it needed to be explicitly enabled (-cpu rva24u64,c=on).
> >>
> >> Right, that's what I had in mind when I was talking about layering.
> >> Using the term union is probably more appropriate though :)
> >>
> >> > > > > If you're working on defining a new CPU model and need fine-grained
> >> > > > > control over the exact set of extensions, then you can already do
> >> > > > > that by explicitly flipping each one of them on or off, and adding
> >> > > > > the ability to layer profiles doesn't add much value except possibly
> >> > > > > removing a bit of verbosity. Not particularly compelling IMO.
> >> > > >
> >> > > > I think it's compelling, because the extension lists that profiles provide
> >> > > > are long and mostly uninteresting. For example, how often do we want to
> >> > > > type and think about imafd_Zicsr_Zifencei? I think we'd mostly rather take
> >> > > > those for granted, and we can, because we just specify 'g' instead. Indeed
> >> > > > the profile spec even points out that using profiles as a way to deal with
> >> > > > "unwieldy" ISA strings is another motivation for them. A RISC-V QEMU CPU
> >> > > > command line is effectively an ISA string, so I think it's appropriate to
> >> > > > apply profiles to it as well.
> >>
> >> I'm not sure this will save a lot of typing unless you're enabling a
> >> lot of profiles which are possibly far apart from each other in terms
> >> of release year, but I also don't have a very strong opposition to
> >> the approach. As you say, we are already effectively implementing it
> >> with virtual extensions like "g", so extending the applicability
> >> further to cover profiles is not too much of a stretch.
> >>
> >> I think libvirt will probably need to learn what set of extensions
> >> each virtual extension/profile maps to, but since the definition will
> >> come directly from actual specifications that shouldn't be a big
> >> problem.
> >>
> >> > > > >    (This also gets really interesting when you start thinking about
> >> > > > >    extensions that are optional for the profile... Maybe we need
> >> > > > >    something like '-cpu rva23u64,optional-extensions=off' so that
> >> > > > >    testing against the baseline for a profile doesn't require combing
> >> > > > >    through the spec and disabling extensions manually?)
> >> > > >
> >> > > > I don't think the profiles (whether they are CPU virtual extensions or
> >> > > > CPU models) should enable the optional extensions by default. When
> >> > > > software targets a profile (or set of profiles with the virtual
> >> > > > extension approach) then it should never assume the optional extensions
> >> > > > will be present. Developers should need to consciously include those
> >> > > > extensions when creating the QEMU platform used for testing.
> >>
> >> Fair enough.
> >>
> >> Should there be a way to explicitly enable all possible optional
> >> extensions, then? To allow the developer to verify that the
> >> application works correctly on a minimal implementation of the
> >> profile, but also takes advantage of optional extensions if they are
> >> present? Or would that be achieved by enabling the relevant
> >> extensions manually? Or perhaps by using the max CPU model instead?
> >
> > We could add a variant of each profile which represents everything the
> > profile mentions. That could maybe be held off until the need arises,
> > though, since 'max' may satisfy most users which aren't concerned about
> > which extensions are mandatory vs. optional.
> 
> At the risk of adding to the bikeshedding here:  IMO we should start 
> with a small set of CPU types that we know are useful.  We're not going 
> to be able to cover everything, hence the options to turn on/off 
> extensions/features.  Anything we add needs to go through the 
> deprecation process if we remove it, though, so it's a bit of a 
> headache.
>

I agree. I think we need a 'max' since it'll be the one primarily used by
developers and CI. Then, we also need a way to build a specific cpu which
may not have a vendor name yet or ever be assigned one. That means we need
a "base", which we've tried out different names for (rv32/rv64,
min, rv32i/rv64i, ...), but they're all the same thing. The "base" cpu
type and the "max" cpu type should never need to be depreciated. We
may find it useful to introduce alias cpu types for particular uses of
them, e.g. max64 == 'max,xlen64=on', but we can wait and see before
introducing those.

Defining profiles as "virtual extensions", which represent collections of
extensions, avoids creating new cpu types for each of them. We just
create new properties. I wouldn't expect to ever need to deprecate those
properties, since they'd only be a list of extensions and a cpu finalize
function which validates that all the mandatory extensions remain enabled
and are compatible with everything else that got enabled.

Thanks,
drew