Exposing host debug capabilities to userspace

Exposing host debug capabilities to userspace

[Alex Bennée]

Hi,

I've almost finished the ARMv8 guest debug support but I have one
problem left to solve. userspace needs to know how many hardware debug
registers are available for GDB to use. This information is available
from the ID_AA64DFR0_EL1 register. Currently I abuse GET_ONE_REG to
fetch it's value however semantically this is poor as it's API is for
getting guest state not host state and they could theoretically have
different values.

So far the options I've examined are:

* KVM ioctl GET_ONE_REG(ID_AA64DFR0_EL1)

As explained above, abusing a guest state API for host configuration.

* ptrace(PTRACE_GETREGSET, NT_ARM_HW_WATCH)

This is used by GDB to access the host details in debug-monitors.
However the ptrace API really wants you to attach to a process before
calling PTRACE_GETREGSET. Currently I've tried attaching to the
thread_id of the vCPU but this fails with EPERM, I suspect because
attaching to your own threads likely upsets the kernel.

* KVM ioctl KVM_GET_DEBUGREGS

This is currently x86 only and looks like it's more aimed at debug
registers than capability stuff. Also I'm not sure what the state of
this ioctl is compared to KVM_SET_GUEST_DEBUG. Do these APIs overlap or
is one an older deprecated x86 only API?

* Export the information via sysfs

I suppose the correct canonical non-subsystem specific way to make this
information available it to expose the data in some sort of sysfs node?
However I don't see any existing sysfs structure for the CPU.

* Expand /proc/cpuinfo

I suspect adding extra text to be badly parsed by userspace is just
horrid and unacceptable behaviour ;-)

* Add another KVM ioctl?

This would have the downside of being specific to KVM and of course
proliferating the API space again.

I'm open to any suggestions and look forward to your valued feedback ;-)

-- 
Alex Bennée

Re: Exposing host debug capabilities to userspace

[Christoffer Dall]

On Thu, Nov 20, 2014 at 04:55:14PM +0000, Alex Bennée wrote:
> Hi,
> 
> I've almost finished the ARMv8 guest debug support but I have one
> problem left to solve. userspace needs to know how many hardware debug
> registers are available for GDB to use. This information is available
> from the ID_AA64DFR0_EL1 register. Currently I abuse GET_ONE_REG to
> fetch it's value however semantically this is poor as it's API is for
> getting guest state not host state and they could theoretically have
> different values.
> 
> So far the options I've examined are:
> 
> * KVM ioctl GET_ONE_REG(ID_AA64DFR0_EL1)
> 
> As explained above, abusing a guest state API for host configuration.

It's just wrong, and we should only do this if there's absolutely no
other way to do this.

> 
> * ptrace(PTRACE_GETREGSET, NT_ARM_HW_WATCH)
> 
> This is used by GDB to access the host details in debug-monitors.
> However the ptrace API really wants you to attach to a process before
> calling PTRACE_GETREGSET. Currently I've tried attaching to the
> thread_id of the vCPU but this fails with EPERM, I suspect because
> attaching to your own threads likely upsets the kernel.

Can you confirm your suspicion?  This seems like a rather good approach
so we should really investigate why this doesn't work and explore ways
to get it working.

> 
> * KVM ioctl KVM_GET_DEBUGREGS
> 
> This is currently x86 only and looks like it's more aimed at debug
> registers than capability stuff. Also I'm not sure what the state of
> this ioctl is compared to KVM_SET_GUEST_DEBUG. Do these APIs overlap or
> is one an older deprecated x86 only API?

The API text and a brief glance of the x86 code seems to indicate that
this is also the vcpu state...

> 
> * Export the information via sysfs
> 
> I suppose the correct canonical non-subsystem specific way to make this
> information available it to expose the data in some sort of sysfs node?
> However I don't see any existing sysfs structure for the CPU.
> 
> * Expand /proc/cpuinfo
> 
> I suspect adding extra text to be badly parsed by userspace is just
> horrid and unacceptable behaviour ;-)
> 
> * Add another KVM ioctl?
> 
> This would have the downside of being specific to KVM and of course
> proliferating the API space again.
> 
This may not be that bad, for example, could we ever imaging that we'd
only want to export a few of the debug registers for host gdbstub usage?

-Christoffer

Re: Exposing host debug capabilities to userspace

[Alex Bennée]

Christoffer Dall <christoffer.dall@linaro.org> writes:

> On Thu, Nov 20, 2014 at 04:55:14PM +0000, Alex Bennée wrote:
>> Hi,
>> 
>> I've almost finished the ARMv8 guest debug support but I have one
>> problem left to solve. userspace needs to know how many hardware debug
>> registers are available for GDB to use.
>> * KVM ioctl KVM_GET_DEBUGREGS
>> 
>> This is currently x86 only and looks like it's more aimed at debug
>> registers than capability stuff. Also I'm not sure what the state of
>> this ioctl is compared to KVM_SET_GUEST_DEBUG. Do these APIs overlap or
>> is one an older deprecated x86 only API?
>
> The API text and a brief glance of the x86 code seems to indicate that
> this is also the vcpu state...

Yeah I was getting confused as to the difference between the two API
calls. Is this just an x86 version of what GET/SET_ONE_REG replaced?

>> * Add another KVM ioctl?
>> 
>> This would have the downside of being specific to KVM and of course
>> proliferating the API space again.
>> 
> This may not be that bad, for example, could we ever imaging that we'd
> only want to export a few of the debug registers for host gdbstub
> usage?

However it is general information which might be useful to the whole
system (although I suspect KVM and PTRACE are the only two). It would be
a shame to have an informational API wrapped up in the extra
boiler-plate of a specific API.

-- 
Alex Bennée

Re: Exposing host debug capabilities to userspace

[Alex Bennée]

Christoffer Dall <christoffer.dall@linaro.org> writes:

> On Thu, Nov 20, 2014 at 04:55:14PM +0000, Alex Bennée wrote:
<snip>
>> 
>> * ptrace(PTRACE_GETREGSET, NT_ARM_HW_WATCH)
>> 
>> This is used by GDB to access the host details in debug-monitors.
>> However the ptrace API really wants you to attach to a process before
>> calling PTRACE_GETREGSET. Currently I've tried attaching to the
>> thread_id of the vCPU but this fails with EPERM, I suspect because
>> attaching to your own threads likely upsets the kernel.
>
> Can you confirm your suspicion?  This seems like a rather good approach
> so we should really investigate why this doesn't work and explore ways
> to get it working.

From ptrace_attach:

	retval = -EPERM;
	if (unlikely(task->flags & PF_KTHREAD))
		goto out;
	if (same_thread_group(task, current))
		goto out;

I think this is what is triggering my EPERM. I'm going to dig into the
history of code around that bit. While I can see it might be undesirable
I'm not sure if it has to be verbotten...

-- 
Alex Bennée

Re: Exposing host debug capabilities to userspace

[Peter Maydell]

On 24 November 2014 at 11:16, Alex Bennée
<alex@zen.linaro.local.i-did-not-set--mail-host-address--so-tickle-me>
wrote:

^^^ :-)

> Alex Bennée <alex.bennee@linaro.org> writes:
>> * KVM ioctl KVM_GET_DEBUGREGS
>>
>> This is currently x86 only and looks like it's more aimed at debug
>> registers than capability stuff. Also I'm not sure what the state of
>> this ioctl is compared to KVM_SET_GUEST_DEBUG. Do these APIs overlap or
>> is one an older deprecated x86 only API?
>
> I'm minded to re-use this ioctl and define it for ARM as reading the
> host debug architecture state ID_AA64DFR0/1_EL1. Currently for x86 it's
> used for getting vcpu debug registers which on ARM is handled via the
> GET/SET one reg interface.

This seems a bit odd. Either the x86 use of this ioctl is
for accessing guest state, in which case using it on ARM
for host state is a bit weird, or else why is x86 doing
its debug via host state and ARM using guest state?

It may well still be the best choice, but it just feels
like maybe something isn't lined up right...

-- PMM

Re: Exposing host debug capabilities to userspace

[Christoffer Dall]

On Mon, Nov 24, 2014 at 12:21 PM, Peter Maydell
<peter.maydell@linaro.org> wrote:
> On 24 November 2014 at 11:16, Alex Bennée
> <alex@zen.linaro.local.i-did-not-set--mail-host-address--so-tickle-me>
> wrote:
>
> ^^^ :-)
>
>> Alex Bennée <alex.bennee@linaro.org> writes:
>>> * KVM ioctl KVM_GET_DEBUGREGS
>>>
>>> This is currently x86 only and looks like it's more aimed at debug
>>> registers than capability stuff. Also I'm not sure what the state of
>>> this ioctl is compared to KVM_SET_GUEST_DEBUG. Do these APIs overlap or
>>> is one an older deprecated x86 only API?
>>
>> I'm minded to re-use this ioctl and define it for ARM as reading the
>> host debug architecture state ID_AA64DFR0/1_EL1. Currently for x86 it's
>> used for getting vcpu debug registers which on ARM is handled via the
>> GET/SET one reg interface.
>
> This seems a bit odd. Either the x86 use of this ioctl is
> for accessing guest state, in which case using it on ARM
> for host state is a bit weird, or else why is x86 doing
> its debug via host state and ARM using guest state?
>
> It may well still be the best choice, but it just feels
> like maybe something isn't lined up right...
>
It seems weird, agreed, but somehow what we're left with except for
adding a new ioctl.  I think part of the explanation may simply be
that x86 solves this problem in an inherently different way.

-Christoffer

Re: Exposing host debug capabilities to userspace

[Alex Bennée]

Fixed CC:kvmarm, Added: Alexander Graf, Fixed: my From:

Replying to myself with additional information on each option

Alex Bennée <alex.bennee@linaro.org> writes:

> Hi,
>
> I've almost finished the ARMv8 guest debug support but I have one
> problem left to solve. userspace needs to know how many hardware debug
> registers are available for GDB to use. This information is available
> from the ID_AA64DFR0_EL1 register. Currently I abuse GET_ONE_REG to
> fetch it's value however semantically this is poor as it's API is for
> getting guest state not host state and they could theoretically have
> different values.
>
> So far the options I've examined are:
>
> * KVM ioctl GET_ONE_REG(ID_AA64DFR0_EL1)
Nope, guest state API
> * ptrace(PTRACE_GETREGSET, NT_ARM_HW_WATCH)
Nope, ptrace requires attachment and you can't attach to your own thread
group.
> * KVM ioctl KVM_GET_DEBUGREGS
>
> This is currently x86 only and looks like it's more aimed at debug
> registers than capability stuff. Also I'm not sure what the state of
> this ioctl is compared to KVM_SET_GUEST_DEBUG. Do these APIs overlap or
> is one an older deprecated x86 only API?

I'm minded to re-use this ioctl and define it for ARM as reading the
host debug architecture state ID_AA64DFR0/1_EL1. Currently for x86 it's
used for getting vcpu debug registers which on ARM is handled via the
GET/SET one reg interface.

> * Export the information via sysfs
>
> I suppose the correct canonical non-subsystem specific way to make this
> information available it to expose the data in some sort of sysfs node?
> However I don't see any existing sysfs structure for the CPU.

I suspect this would get complicated depending on the architecture.

> * Expand /proc/cpuinfo
>
> I suspect adding extra text to be badly parsed by userspace is just
> horrid and unacceptable behaviour ;-)
>
> * Add another KVM ioctl?
>
> This would have the downside of being specific to KVM and of course
> proliferating the API space again.

So unless there are any objections my intention to re-use the existing
API calls for ARM architectures.

-- 
Alex Bennée

Re: Exposing host debug capabilities to userspace

[Alexander Graf]

On 24.11.14 12:35, Alex Bennée wrote:
> 
> Fixed CC:kvmarm, Added: Alexander Graf, Fixed: my From:
> 
> Replying to myself with additional information on each option
> 
> Alex Bennée <alex.bennee@linaro.org> writes:
> 
>> Hi,
>>
>> I've almost finished the ARMv8 guest debug support but I have one
>> problem left to solve. userspace needs to know how many hardware debug
>> registers are available for GDB to use. This information is available
>> from the ID_AA64DFR0_EL1 register. Currently I abuse GET_ONE_REG to
>> fetch it's value however semantically this is poor as it's API is for
>> getting guest state not host state and they could theoretically have
>> different values.
>>
>> So far the options I've examined are:
>>
>> * KVM ioctl GET_ONE_REG(ID_AA64DFR0_EL1)
> Nope, guest state API

What's the problem with using ONE_REG for this? After all, the total
number of available guest debug register is a guest vcpu property of
some sort.

Btw, looking through the mess that we have today with asymmetric
SET_GUEST_DEBUG and GET_DEBUGREGS ioctls I can't shake off the feeling
that we're best off just doing all of the debug register sync via
ONE_REGs. That way we at least have a guaranteed symmetric interface
that can get and set values, so we won't trip over it on live migration.


Alex

Re: Exposing host debug capabilities to userspace

[Peter Maydell]

On 24 November 2014 at 12:26, Alexander Graf <agraf@suse.de> wrote:
> On 24.11.14 12:35, Alex Bennée wrote:
>>> * KVM ioctl GET_ONE_REG(ID_AA64DFR0_EL1)
>> Nope, guest state API
>
> What's the problem with using ONE_REG for this? After all, the total
> number of available guest debug register is a guest vcpu property of
> some sort.

Yes, but we don't want to know about properties of the guest
vCPU. In an ideal world QEMU could reserve say half the debug
registers for debugging the VM on startup and have KVM expose
ID registers indicating to the guest that it only had the
other half...

-- PMM

Re: Exposing host debug capabilities to userspace

[Alexander Graf]

> Am 24.11.2014 um 13:32 schrieb Peter Maydell <peter.maydell@linaro.org>:
> 
>> On 24 November 2014 at 12:26, Alexander Graf <agraf@suse.de> wrote:
>> On 24.11.14 12:35, Alex Bennée wrote:
>>>> * KVM ioctl GET_ONE_REG(ID_AA64DFR0_EL1)
>>> Nope, guest state API
>> 
>> What's the problem with using ONE_REG for this? After all, the total
>> number of available guest debug register is a guest vcpu property of
>> some sort.
> 
> Yes, but we don't want to know about properties of the guest
> vCPU. In an ideal world QEMU could reserve say half the debug
> registers for debugging the VM on startup and have KVM expose
> ID registers indicating to the guest that it only had the
> other half...

Yup, so create another (read-only) ONE_REG that exposes the number of actual guest debug registers.

Alex

> 
> -- PMM

Re: Exposing host debug capabilities to userspace

[Peter Maydell]

On 24 November 2014 at 12:41, Alexander Graf <agraf@suse.de> wrote:
>> Am 24.11.2014 um 13:32 schrieb Peter Maydell <peter.maydell@linaro.org>:
>> Yes, but we don't want to know about properties of the guest
>> vCPU. In an ideal world QEMU could reserve say half the debug
>> registers for debugging the VM on startup and have KVM expose
>> ID registers indicating to the guest that it only had the
>> other half...
>
> Yup, so create another (read-only) ONE_REG that exposes the number
> of actual guest debug registers.

I'm confused. ONE_REG is for guest state, and the ID register
by definition is how we tell the guest how many debug registers
it has. What we want to know (and perhaps even control) for
debugging the VM is how many debug registers the host has.

-- PMM

Re: Exposing host debug capabilities to userspace

[Alexander Graf]

On 24.11.14 13:44, Peter Maydell wrote:
> On 24 November 2014 at 12:41, Alexander Graf <agraf@suse.de> wrote:
>>> Am 24.11.2014 um 13:32 schrieb Peter Maydell <peter.maydell@linaro.org>:
>>> Yes, but we don't want to know about properties of the guest
>>> vCPU. In an ideal world QEMU could reserve say half the debug
>>> registers for debugging the VM on startup and have KVM expose
>>> ID registers indicating to the guest that it only had the
>>> other half...
>>
>> Yup, so create another (read-only) ONE_REG that exposes the number
>> of actual guest debug registers.
> 
> I'm confused. ONE_REG is for guest state, and the ID register
> by definition is how we tell the guest how many debug registers
> it has. What we want to know (and perhaps even control) for
> debugging the VM is how many debug registers the host has.

No, we don't want to know how many debug registers the host has. We want
to know how many debug registers the guest has.

Imagine you're running on A57 today with 8 debug registers (no idea if
that's true, but assume it is). Tomorrow there will be a new core -
let's call it A67 - with 16 debug registers.

To make sure your legacy, badly written guest behaves exactly the same -
especially after live migration - you want to spawn a VM with -cpu A57.
That implies you want to expose 8 debug registers into the guest. So
debug register synchronization should only be aware of those 8 registers.

So what we really care about is the number of debug registers available
to a guest vcpu. That in turn means it's guest state and as such can
easily go into ONE_REG.


Alex

Re: Exposing host debug capabilities to userspace

[Christoffer Dall]

On Mon, Nov 24, 2014 at 1:51 PM, Alexander Graf <agraf@suse.de> wrote:
>
>
> On 24.11.14 13:44, Peter Maydell wrote:
>> On 24 November 2014 at 12:41, Alexander Graf <agraf@suse.de> wrote:
>>>> Am 24.11.2014 um 13:32 schrieb Peter Maydell <peter.maydell@linaro.org>:
>>>> Yes, but we don't want to know about properties of the guest
>>>> vCPU. In an ideal world QEMU could reserve say half the debug
>>>> registers for debugging the VM on startup and have KVM expose
>>>> ID registers indicating to the guest that it only had the
>>>> other half...
>>>
>>> Yup, so create another (read-only) ONE_REG that exposes the number
>>> of actual guest debug registers.
>>
>> I'm confused. ONE_REG is for guest state, and the ID register
>> by definition is how we tell the guest how many debug registers
>> it has. What we want to know (and perhaps even control) for
>> debugging the VM is how many debug registers the host has.
>
> No, we don't want to know how many debug registers the host has. We want
> to know how many debug registers the guest has.
>
> Imagine you're running on A57 today with 8 debug registers (no idea if
> that's true, but assume it is). Tomorrow there will be a new core -
> let's call it A67 - with 16 debug registers.
>
> To make sure your legacy, badly written guest behaves exactly the same -
> especially after live migration - you want to spawn a VM with -cpu A57.
> That implies you want to expose 8 debug registers into the guest. So
> debug register synchronization should only be aware of those 8 registers.
>
> So what we really care about is the number of debug registers available
> to a guest vcpu. That in turn means it's guest state and as such can
> easily go into ONE_REG.
>
We already export this for the guest via ONE_REG.

What we want to do is support gdbstubs in QEMU to debug the guest, and
to do this, QEMU needs to know how many hardware registers on the host
there is; the guest will never see this information.

So this is really about the host, the guest side is trivially handled
through ONE_REG.

-Christoffer

Re: Exposing host debug capabilities to userspace

[Alexander Graf]

On 24.11.14 13:53, Christoffer Dall wrote:
> On Mon, Nov 24, 2014 at 1:51 PM, Alexander Graf <agraf@suse.de> wrote:
>>
>>
>> On 24.11.14 13:44, Peter Maydell wrote:
>>> On 24 November 2014 at 12:41, Alexander Graf <agraf@suse.de> wrote:
>>>>> Am 24.11.2014 um 13:32 schrieb Peter Maydell <peter.maydell@linaro.org>:
>>>>> Yes, but we don't want to know about properties of the guest
>>>>> vCPU. In an ideal world QEMU could reserve say half the debug
>>>>> registers for debugging the VM on startup and have KVM expose
>>>>> ID registers indicating to the guest that it only had the
>>>>> other half...
>>>>
>>>> Yup, so create another (read-only) ONE_REG that exposes the number
>>>> of actual guest debug registers.
>>>
>>> I'm confused. ONE_REG is for guest state, and the ID register
>>> by definition is how we tell the guest how many debug registers
>>> it has. What we want to know (and perhaps even control) for
>>> debugging the VM is how many debug registers the host has.
>>
>> No, we don't want to know how many debug registers the host has. We want
>> to know how many debug registers the guest has.
>>
>> Imagine you're running on A57 today with 8 debug registers (no idea if
>> that's true, but assume it is). Tomorrow there will be a new core -
>> let's call it A67 - with 16 debug registers.
>>
>> To make sure your legacy, badly written guest behaves exactly the same -
>> especially after live migration - you want to spawn a VM with -cpu A57.
>> That implies you want to expose 8 debug registers into the guest. So
>> debug register synchronization should only be aware of those 8 registers.
>>
>> So what we really care about is the number of debug registers available
>> to a guest vcpu. That in turn means it's guest state and as such can
>> easily go into ONE_REG.
>>
> We already export this for the guest via ONE_REG.
> 
> What we want to do is support gdbstubs in QEMU to debug the guest, and
> to do this, QEMU needs to know how many hardware registers on the host
> there is; the guest will never see this information.
> 
> So this is really about the host, the guest side is trivially handled
> through ONE_REG.

That's the cp15 register that happens to get exposed to the guest. You
can just add another ONE_REG that does not have a cp15 equivalent to
expose the number of the vcpu's actually available debug registers.


Alex

Re: Exposing host debug capabilities to userspace

[Christoffer Dall]

On Mon, Nov 24, 2014 at 1:56 PM, Alexander Graf <agraf@suse.de> wrote:
>
>
> On 24.11.14 13:53, Christoffer Dall wrote:
>> On Mon, Nov 24, 2014 at 1:51 PM, Alexander Graf <agraf@suse.de> wrote:
>>>
>>>
>>> On 24.11.14 13:44, Peter Maydell wrote:
>>>> On 24 November 2014 at 12:41, Alexander Graf <agraf@suse.de> wrote:
>>>>>> Am 24.11.2014 um 13:32 schrieb Peter Maydell <peter.maydell@linaro.org>:
>>>>>> Yes, but we don't want to know about properties of the guest
>>>>>> vCPU. In an ideal world QEMU could reserve say half the debug
>>>>>> registers for debugging the VM on startup and have KVM expose
>>>>>> ID registers indicating to the guest that it only had the
>>>>>> other half...
>>>>>
>>>>> Yup, so create another (read-only) ONE_REG that exposes the number
>>>>> of actual guest debug registers.
>>>>
>>>> I'm confused. ONE_REG is for guest state, and the ID register
>>>> by definition is how we tell the guest how many debug registers
>>>> it has. What we want to know (and perhaps even control) for
>>>> debugging the VM is how many debug registers the host has.
>>>
>>> No, we don't want to know how many debug registers the host has. We want
>>> to know how many debug registers the guest has.
>>>
>>> Imagine you're running on A57 today with 8 debug registers (no idea if
>>> that's true, but assume it is). Tomorrow there will be a new core -
>>> let's call it A67 - with 16 debug registers.
>>>
>>> To make sure your legacy, badly written guest behaves exactly the same -
>>> especially after live migration - you want to spawn a VM with -cpu A57.
>>> That implies you want to expose 8 debug registers into the guest. So
>>> debug register synchronization should only be aware of those 8 registers.
>>>
>>> So what we really care about is the number of debug registers available
>>> to a guest vcpu. That in turn means it's guest state and as such can
>>> easily go into ONE_REG.
>>>
>> We already export this for the guest via ONE_REG.
>>
>> What we want to do is support gdbstubs in QEMU to debug the guest, and
>> to do this, QEMU needs to know how many hardware registers on the host
>> there is; the guest will never see this information.
>>
>> So this is really about the host, the guest side is trivially handled
>> through ONE_REG.
>
> That's the cp15 register that happens to get exposed to the guest. You
> can just add another ONE_REG that does not have a cp15 equivalent to
> expose the number of the vcpu's actually available debug registers.
>
>
The fact that we currently map the guest vcpu registers to that of the
host doesn't mean that will always be the case (which you argued for
above).

If you migrate a VM from a CPU with 16 debug registers (or emulate a
CPU with 16 debug registers) on a physical CPU with only 8 debug
registers, you cannot tell QEMU that it has 16 debug registers on the
hardware to configure to debug the guest, which is what ONE_REG would
give you.

You could add another set of registers to ONE_REG which says "these
are the host versions", or you could say that you don't ever support a
setup where the guest number of debug registers is not the same as the
host number, but both would be wrong.

-Christoffer

Re: Exposing host debug capabilities to userspace

[Alexander Graf]

On 24.11.14 14:10, Christoffer Dall wrote:
> On Mon, Nov 24, 2014 at 1:56 PM, Alexander Graf <agraf@suse.de> wrote:
>>
>>
>> On 24.11.14 13:53, Christoffer Dall wrote:
>>> On Mon, Nov 24, 2014 at 1:51 PM, Alexander Graf <agraf@suse.de> wrote:
>>>>
>>>>
>>>> On 24.11.14 13:44, Peter Maydell wrote:
>>>>> On 24 November 2014 at 12:41, Alexander Graf <agraf@suse.de> wrote:
>>>>>>> Am 24.11.2014 um 13:32 schrieb Peter Maydell <peter.maydell@linaro.org>:
>>>>>>> Yes, but we don't want to know about properties of the guest
>>>>>>> vCPU. In an ideal world QEMU could reserve say half the debug
>>>>>>> registers for debugging the VM on startup and have KVM expose
>>>>>>> ID registers indicating to the guest that it only had the
>>>>>>> other half...
>>>>>>
>>>>>> Yup, so create another (read-only) ONE_REG that exposes the number
>>>>>> of actual guest debug registers.
>>>>>
>>>>> I'm confused. ONE_REG is for guest state, and the ID register
>>>>> by definition is how we tell the guest how many debug registers
>>>>> it has. What we want to know (and perhaps even control) for
>>>>> debugging the VM is how many debug registers the host has.
>>>>
>>>> No, we don't want to know how many debug registers the host has. We want
>>>> to know how many debug registers the guest has.
>>>>
>>>> Imagine you're running on A57 today with 8 debug registers (no idea if
>>>> that's true, but assume it is). Tomorrow there will be a new core -
>>>> let's call it A67 - with 16 debug registers.
>>>>
>>>> To make sure your legacy, badly written guest behaves exactly the same -
>>>> especially after live migration - you want to spawn a VM with -cpu A57.
>>>> That implies you want to expose 8 debug registers into the guest. So
>>>> debug register synchronization should only be aware of those 8 registers.
>>>>
>>>> So what we really care about is the number of debug registers available
>>>> to a guest vcpu. That in turn means it's guest state and as such can
>>>> easily go into ONE_REG.
>>>>
>>> We already export this for the guest via ONE_REG.
>>>
>>> What we want to do is support gdbstubs in QEMU to debug the guest, and
>>> to do this, QEMU needs to know how many hardware registers on the host
>>> there is; the guest will never see this information.
>>>
>>> So this is really about the host, the guest side is trivially handled
>>> through ONE_REG.
>>
>> That's the cp15 register that happens to get exposed to the guest. You
>> can just add another ONE_REG that does not have a cp15 equivalent to
>> expose the number of the vcpu's actually available debug registers.
>>
>>
> The fact that we currently map the guest vcpu registers to that of the
> host doesn't mean that will always be the case (which you argued for
> above).
> 
> If you migrate a VM from a CPU with 16 debug registers (or emulate a
> CPU with 16 debug registers) on a physical CPU with only 8 debug
> registers, you cannot tell QEMU that it has 16 debug registers on the
> hardware to configure to debug the guest, which is what ONE_REG would
> give you.
> 
> You could add another set of registers to ONE_REG which says "these
> are the host versions", or you could say that you don't ever support a
> setup where the guest number of debug registers is not the same as the
> host number, but both would be wrong.

Why would QEMU ever want to access debug registers that it didn't ask
for in the first place? If we simply limit ourselves to the register set
the vcpu has we don't have any problems and the complexity matrix
shrinks noticably, no?


Alex

Re: Exposing host debug capabilities to userspace

[Christoffer Dall]

On Mon, Nov 24, 2014 at 03:07:35PM +0100, Alexander Graf wrote:
> 
> 
> On 24.11.14 14:10, Christoffer Dall wrote:
> > On Mon, Nov 24, 2014 at 1:56 PM, Alexander Graf <agraf@suse.de> wrote:
> >>
> >>
> >> On 24.11.14 13:53, Christoffer Dall wrote:
> >>> On Mon, Nov 24, 2014 at 1:51 PM, Alexander Graf <agraf@suse.de> wrote:
> >>>>
> >>>>
> >>>> On 24.11.14 13:44, Peter Maydell wrote:
> >>>>> On 24 November 2014 at 12:41, Alexander Graf <agraf@suse.de> wrote:
> >>>>>>> Am 24.11.2014 um 13:32 schrieb Peter Maydell <peter.maydell@linaro.org>:
> >>>>>>> Yes, but we don't want to know about properties of the guest
> >>>>>>> vCPU. In an ideal world QEMU could reserve say half the debug
> >>>>>>> registers for debugging the VM on startup and have KVM expose
> >>>>>>> ID registers indicating to the guest that it only had the
> >>>>>>> other half...
> >>>>>>
> >>>>>> Yup, so create another (read-only) ONE_REG that exposes the number
> >>>>>> of actual guest debug registers.
> >>>>>
> >>>>> I'm confused. ONE_REG is for guest state, and the ID register
> >>>>> by definition is how we tell the guest how many debug registers
> >>>>> it has. What we want to know (and perhaps even control) for
> >>>>> debugging the VM is how many debug registers the host has.
> >>>>
> >>>> No, we don't want to know how many debug registers the host has. We want
> >>>> to know how many debug registers the guest has.
> >>>>
> >>>> Imagine you're running on A57 today with 8 debug registers (no idea if
> >>>> that's true, but assume it is). Tomorrow there will be a new core -
> >>>> let's call it A67 - with 16 debug registers.
> >>>>
> >>>> To make sure your legacy, badly written guest behaves exactly the same -
> >>>> especially after live migration - you want to spawn a VM with -cpu A57.
> >>>> That implies you want to expose 8 debug registers into the guest. So
> >>>> debug register synchronization should only be aware of those 8 registers.
> >>>>
> >>>> So what we really care about is the number of debug registers available
> >>>> to a guest vcpu. That in turn means it's guest state and as such can
> >>>> easily go into ONE_REG.
> >>>>
> >>> We already export this for the guest via ONE_REG.
> >>>
> >>> What we want to do is support gdbstubs in QEMU to debug the guest, and
> >>> to do this, QEMU needs to know how many hardware registers on the host
> >>> there is; the guest will never see this information.
> >>>
> >>> So this is really about the host, the guest side is trivially handled
> >>> through ONE_REG.
> >>
> >> That's the cp15 register that happens to get exposed to the guest. You
> >> can just add another ONE_REG that does not have a cp15 equivalent to
> >> expose the number of the vcpu's actually available debug registers.
> >>
> >>
> > The fact that we currently map the guest vcpu registers to that of the
> > host doesn't mean that will always be the case (which you argued for
> > above).
> > 
> > If you migrate a VM from a CPU with 16 debug registers (or emulate a
> > CPU with 16 debug registers) on a physical CPU with only 8 debug
> > registers, you cannot tell QEMU that it has 16 debug registers on the
> > hardware to configure to debug the guest, which is what ONE_REG would
> > give you.
> > 
> > You could add another set of registers to ONE_REG which says "these
> > are the host versions", or you could say that you don't ever support a
> > setup where the guest number of debug registers is not the same as the
> > host number, but both would be wrong.
> 
> Why would QEMU ever want to access debug registers that it didn't ask
> for in the first place? If we simply limit ourselves to the register set
> the vcpu has we don't have any problems and the complexity matrix
> shrinks noticably, no?
> 

(For others following this, not hanging out on #kvm-arm, I provide a
summary).

We had a lenghty IRC discussion on this, for the curious, go read it
here:
http://irclogs.linaro.org/2014/11/24/%23kvm-arm.html

The point of confusion is that other KVM architectures use the ONE_REG
interface to set the debug registers, for both guest and host state, and
in fact this can be overlayed.  The propose idea of using SET_DEBUGREGS
was flawed, because this also pertains to guest state and is a
deprecated ABI.

So that left us with two choices:

(1) Implement a new ABI to retrieve and set host debugging independently
of the guest state.

(2) Overlay the host debugging of a guest with the guest's internal
debugging state (a process inside the guest is debugging a process).

Option (1) has the usual drawback of having to design an ABI, consumes
the ioctl number space etc.  But it also has (the more severe) drawback
that debugging the guest using QEMU gdbstubs breaks guest debugging,
because we would keep completely distinct register sets.

Option (2) feels semantically slightly weird in the cases where the
number of guest debug regs differs from the number of host debug regs,
but as Alex Graf pointed out, we probably wouldn't dream of supporting
more guest debug registers than the host has (refuse migration, don't
emulate that VCPU on the PCPU etc.), and we don't care deeply about
exposing fewer host debug regs than actually available.  Migrating a VM
that is being debugged by the host is simply not supported, so any state
obtained via ONE_REG for migration is for the guest, not the host.

Option (2) has the additional benefit that we can intermix guest and
host registers; debug exceptions are in this case always routed to user
space and QEMU's magic determines if the exception is for itself or if
it should be injected back into the guest.  Guest accesses to debug regs
are always trapped when the host uses any debug regs, thus not exposing
any sensitive state to the guest.

Given this information, I personally lean towards approach (2) since it
is similar to what is done for other architectures and caters to the
most probable use cases that we would care about.

Feel free to point out omissions or errors in my reasoning.

-Christoffer

Re: Exposing host debug capabilities to userspace

[Paolo Bonzini]

On 24/11/2014 15:52, Christoffer Dall wrote:
> 
> We had a lenghty IRC discussion on this, for the curious, go read it
> here:
> http://irclogs.linaro.org/2014/11/24/%23kvm-arm.html

Some notes...

> chazy	but saying that you want to design an ABI that potentially exposes fewer debug registers to gdbstubs than your hardware supports and breaks gdbstub support if you happen to support emulation of a cpu with more debug registers in the future, because you want to re-use ONE_REG is not a great approach either
> 
> agraf	chazy: but you wouldn't even remotely think of doing it, no?
> 
> agraf	chazy: I'm saying that QEMU shouldn't have access to the excess registers
> 
> chazy	agraf: I wouldn’t even remotely think of doing nested virtualization, but people have
> 
> agraf	chazy: QEMU spawns an A57, it gets 8 debug registers
> 
> chazy	agraf: I understand, but I don’t agree with your rationale
> 
> agraf	chazy: not "QEMU spawns an A57 on an A67, so it gets 8 real and 8 hidden debug registers that it also needs to be aware of mappings of"

I disagree with Alex.  QEMU can use the 16 debug registers as it wishes,
since it sets them with KVM_SET_GUEST_DEBUG.  The guest's eight can be
mapped to the last 8 if those are the required semantics for the
architecture.  Just exit to QEMU if you do a debug register write while
gdbstub debugging is active; QEMU gets the contents of the 8
VCPU-visible registers with ONE_REG (equivalent of x86 GET_DEBUGREGS);
calls KVM_SET_GUEST_DEBUG to reflect them in the 16-register state; and
restarts.

It works as long as you can trap both reads and writes.

> chazy ajb-linaro: don’t migrate a guest that is getting debugged is
> the answer to that one I believe
> 
> ajb-linaro	chazy: at least with the overlay approach that happens automatically
> 
> ajb-linaro	the overlay isn't migrated and the new host isn't calling SET_GUEST_DEBUG so whatever the debug registers where before are restorerd

Right.  The destination will lose the hardware breakpoints/watchpoints
because it starts the guest with KVM_SET_GUEST_DEBUG.  Apart from losing
them, everything should work fine.  The dest sets the 8 VCPU-visible
registers with ONE_REG, the hypervisor reflects them in a subset of the
16 physical registers, and "that's it".

Paolo

Re: Exposing host debug capabilities to userspace

[Alex Bennée]

Alexander Graf <agraf@suse.de> writes:

>> Am 24.11.2014 um 13:32 schrieb Peter Maydell <peter.maydell@linaro.org>:
>> 
>>> On 24 November 2014 at 12:26, Alexander Graf <agraf@suse.de> wrote:
>>> On 24.11.14 12:35, Alex Bennée wrote:
>>>>> * KVM ioctl GET_ONE_REG(ID_AA64DFR0_EL1)
>>>> Nope, guest state API
>>> 
>>> What's the problem with using ONE_REG for this? After all, the total
>>> number of available guest debug register is a guest vcpu property of
>>> some sort.
>> 
>> Yes, but we don't want to know about properties of the guest
>> vCPU. In an ideal world QEMU could reserve say half the debug
>> registers for debugging the VM on startup and have KVM expose
>> ID registers indicating to the guest that it only had the
>> other half...
>
> Yup, so create another (read-only) ONE_REG that exposes the number of
> actual guest debug registers.

Does the GET/SET_ONE_REG have the concept of a read-only register? I'd
be worried by code just blindly iterating over the lists getting thrown
when one doesn't work.

>
> Alex
>
>> 
>> -- PMM

-- 
Alex Bennée

Re: Exposing host debug capabilities to userspace

[Christoffer Dall]

On Mon, Nov 24, 2014 at 1:53 PM, Alex Bennée <alex.bennee@linaro.org> wrote:
>
> Alexander Graf <agraf@suse.de> writes:
>
>>> Am 24.11.2014 um 13:32 schrieb Peter Maydell <peter.maydell@linaro.org>:
>>>
>>>> On 24 November 2014 at 12:26, Alexander Graf <agraf@suse.de> wrote:
>>>> On 24.11.14 12:35, Alex Bennée wrote:
>>>>>> * KVM ioctl GET_ONE_REG(ID_AA64DFR0_EL1)
>>>>> Nope, guest state API
>>>>
>>>> What's the problem with using ONE_REG for this? After all, the total
>>>> number of available guest debug register is a guest vcpu property of
>>>> some sort.
>>>
>>> Yes, but we don't want to know about properties of the guest
>>> vCPU. In an ideal world QEMU could reserve say half the debug
>>> registers for debugging the VM on startup and have KVM expose
>>> ID registers indicating to the guest that it only had the
>>> other half...
>>
>> Yup, so create another (read-only) ONE_REG that exposes the number of
>> actual guest debug registers.
>
> Does the GET/SET_ONE_REG have the concept of a read-only register? I'd
> be worried by code just blindly iterating over the lists getting thrown
> when one doesn't work.
>

yes, we have invariant cp15 registers.

-Christoffer

Re: Exposing host debug capabilities to userspace

[Alex Bennée]

Alex Bennée <alex.bennee@linaro.org> writes:

<snip>
> Alex Bennée <alex.bennee@linaro.org> writes:
>
>> Hi,
>>
>> I've almost finished the ARMv8 guest debug support but I have one
>> problem left to solve. userspace needs to know how many hardware debug
>> registers are available for GDB to use. This information is available
>> from the ID_AA64DFR0_EL1 register.
<Snip>
>> So far the options I've examined are:
>>
>> * KVM ioctl GET_ONE_REG(ID_AA64DFR0_EL1)
>> * ptrace(PTRACE_GETREGSET, NT_ARM_HW_WATCH)
>> * KVM ioctl KVM_GET_DEBUGREGS
>> * Export the information via sysfs
>> * Expand /proc/cpuinfo
>> * Add another KVM ioctl?

Alexander Graf pointed out that KVM_CHECK_EXTENSION can return any
positive number for success. How about using:

max_hw_bps = kvm_check_extension(kvm_state, KVM_CAP_GUEST_DEBUG_HW_BPS);
max_hw_wps = kvm_check_extension(kvm_state, KVM_CAP_GUEST_DEBUG_HW_WPS);

Seems pretty sane, doesn't change the semantics of an API and is
architecture agnostic if others need the number?

-- 
Alex Bennée

Re: Exposing host debug capabilities to userspace

[Paolo Bonzini]

On 24/11/2014 14:59, Alex Bennée wrote:
> Alexander Graf pointed out that KVM_CHECK_EXTENSION can return any
> positive number for success. How about using:
> 
> max_hw_bps = kvm_check_extension(kvm_state, KVM_CAP_GUEST_DEBUG_HW_BPS);
> max_hw_wps = kvm_check_extension(kvm_state, KVM_CAP_GUEST_DEBUG_HW_WPS);
> 
> Seems pretty sane, doesn't change the semantics of an API and is
> architecture agnostic if others need the number?

Yes, this was going to be my suggestion as well.  Just I would use a
bitmask in case some register can act as both breakpoint and watchpoint.

On x86, each of the four bp/wp registers (each register can act as both)
can be used for either guest or gdbstub debugging.  If the
KVM_GUESTDBG_USE_HW_BP feature is enabled, the guest is entered with
"made up" debug register contents, that we pass via
KVM_SET_GUEST_DEBUG's struct kvm_guest_debug_arch.  Otherwise, the guest
is entered with real debug register contents passed via
KVM_SET_DEBUGREGS.  Reads/writes of the debug registers trap to KVM
(which helps the guest see the expected values of the debug registers in
the former case).  There is no KVM_GET_GUEST_DEBUG because the
corresponding info is passed via struct kvm_debug_exit_arch.

If gdbstub hardware breakpoints are enabled, all hardware breakpoints
exit to userspace.  QEMU then decides whether the breakpoint came from
guest debugging (and then injects an exception), or from gdbstub
debugging (and then suspends execution).  Same for software breakpoints.
 If the total request is >4 hardware breakpoints, someone will have to
lose and some gdbstub breakpoints will be missed.

Paolo

Re: Exposing host debug capabilities to userspace

[Alexander Graf]

On 25.11.14 17:21, Paolo Bonzini wrote:
> 
> 
> On 24/11/2014 14:59, Alex Bennée wrote:
>> Alexander Graf pointed out that KVM_CHECK_EXTENSION can return any
>> positive number for success. How about using:
>>
>> max_hw_bps = kvm_check_extension(kvm_state, KVM_CAP_GUEST_DEBUG_HW_BPS);
>> max_hw_wps = kvm_check_extension(kvm_state, KVM_CAP_GUEST_DEBUG_HW_WPS);
>>
>> Seems pretty sane, doesn't change the semantics of an API and is
>> architecture agnostic if others need the number?
> 
> Yes, this was going to be my suggestion as well.  Just I would use a
> bitmask in case some register can act as both breakpoint and watchpoint.
> 
> On x86, each of the four bp/wp registers (each register can act as both)
> can be used for either guest or gdbstub debugging.  If the
> KVM_GUESTDBG_USE_HW_BP feature is enabled, the guest is entered with
> "made up" debug register contents, that we pass via
> KVM_SET_GUEST_DEBUG's struct kvm_guest_debug_arch.  Otherwise, the guest
> is entered with real debug register contents passed via
> KVM_SET_DEBUGREGS.  Reads/writes of the debug registers trap to KVM
> (which helps the guest see the expected values of the debug registers in
> the former case).  There is no KVM_GET_GUEST_DEBUG because the
> corresponding info is passed via struct kvm_debug_exit_arch.
> 
> If gdbstub hardware breakpoints are enabled, all hardware breakpoints
> exit to userspace.  QEMU then decides whether the breakpoint came from
> guest debugging (and then injects an exception), or from gdbstub
> debugging (and then suspends execution).  Same for software breakpoints.
>  If the total request is >4 hardware breakpoints, someone will have to
> lose and some gdbstub breakpoints will be missed.

Unfortunately on ARM you also have a few other constraints - the debug
register space is partitioned into magic super debug registers at the
top (with an implementation specific amount) and normal debug registers
in the lower end of the space.

The main pain I have with exposing host information is that it's going
to be interesting and challenging enough to get all of this right merely
for the guest debug register space. Exposing the host debug register
space as well means there is even more space for breakage.

I would just treat gdbstub debugging as the ugly step child that it is
and not introduce anything special for it (except for debug event
deflection to user space). Then only ever work on guest debug registers
and call it a day. Chances are just too high that we get the interfaces
wrong and shoot ourselves in the foot.


Alex

Re: Exposing host debug capabilities to userspace

[Paolo Bonzini]

On 25/11/2014 17:35, Alexander Graf wrote:
> Unfortunately on ARM you also have a few other constraints - the debug
> register space is partitioned into magic super debug registers at the
> top (with an implementation specific amount) and normal debug registers
> in the lower end of the space.

Does the gdbstub ever need to use the magic super debug registers?  Even
if it does, the logic is the same as x86.  On x86 you might run out of
breakpoints, on ARM you might run out of breakpoints in general or magic
super breakpoints in particular.

> I would just treat gdbstub debugging as the ugly step child that it is
> and not introduce anything special for it (except for debug event
> deflection to user space). Then only ever work on guest debug registers
> and call it a day. Chances are just too high that we get the interfaces
> wrong and shoot ourselves in the foot.

I agree.  But we do need a way to retrieve the number of valid fields in
struct kvm_guest_debug_arch, if that is not architecturally defined
(e.g. on x86 it's just always 4).  A "hidden" ONE_REG, whose existence
is determined by (ARM || ARM64) &&
kvm_check_extension(KVM_CAP_SET_GUEST_DEBUG), is certainly fine and I
like it because it doesn't pollute the global KVM_CAP_* namespace.

The alternative is a capability; however, if you start with two
capabilities you'll end up needing four, and then realize that returning
a struct via ONE_REG would have been a better idea.  :)  We already have
how many breakpoints, how many watchpoints, how many magic super debug
registers,...

Paolo

Re: Exposing host debug capabilities to userspace

[Alex Bennée]

Paolo Bonzini <pbonzini@redhat.com> writes:

> On 25/11/2014 17:35, Alexander Graf wrote:
>> Unfortunately on ARM you also have a few other constraints - the debug
>> register space is partitioned into magic super debug registers at the
>> top (with an implementation specific amount) and normal debug registers
>> in the lower end of the space.
>
> Does the gdbstub ever need to use the magic super debug registers?  Even
> if it does, the logic is the same as x86.  On x86 you might run out of
> breakpoints, on ARM you might run out of breakpoints in general or magic
> super breakpoints in particular.

By default gdbstub uses normal software breakpoints unless a) the user
asks for a HW one or b) the memory is RO (i.e a ROM). For watchpoints
the situation is reversed and they are used by default if available.

>> I would just treat gdbstub debugging as the ugly step child that it is
>> and not introduce anything special for it (except for debug event
>> deflection to user space). Then only ever work on guest debug registers
>> and call it a day. Chances are just too high that we get the interfaces
>> wrong and shoot ourselves in the foot.
>
> I agree.  But we do need a way to retrieve the number of valid fields in
> struct kvm_guest_debug_arch, if that is not architecturally defined
> (e.g. on x86 it's just always 4).

It's IMPDEF (implementation defined) up to a maximum of 16.

> A "hidden" ONE_REG, whose existence
> is determined by (ARM || ARM64) &&
> kvm_check_extension(KVM_CAP_SET_GUEST_DEBUG), is certainly fine and I
> like it because it doesn't pollute the global KVM_CAP_* namespace.

My original implementation more or less did that but my main worry is
migration code tends to iterate over the whole ONE_REG list and
introducing a value that doesn't happen to be strictly the guests is
semantically impure. Of course if you mean by "hidden" don't export it
via GET_REG_LIST then I guess that would work. Does seem a little ugly
as the internal KVM code now needs to learn about hidden and non-hidden
registers.

> The alternative is a capability; however, if you start with two
> capabilities you'll end up needing four, and then realize that returning
> a struct via ONE_REG would have been a better idea.  :)  We already have
> how many breakpoints, how many watchpoints, how many magic super debug
> registers,...

Only two, not sure what the super debug registers are?

Certainly on the QEMU side the capability based approach is beautifully
simple:

    max_hw_wp = kvm_check_extension(cs->kvm_state, KVM_CAP_GUEST_DEBUG_HW_WPS);
    max_hw_bp = kvm_check_extension(cs->kvm_state, KVM_CAP_GUEST_DEBUG_HW_BPS);

thanks to kvm_check_extension zeroing failure modes.

And in it's defence it's a generic enough capability to be used across
any other architectures that need to expose this information compared to
the many PPC specific capabilities. Perhaps a KVM_ARCH_EXTENSION ioctl
would be more useful in reducing the growth of the space?

>
> Paolo

-- 
Alex Bennée

Re: Exposing host debug capabilities to userspace

[Peter Maydell]

On 25 November 2014 at 16:50, Paolo Bonzini <pbonzini@redhat.com> wrote:
>
>
> On 25/11/2014 17:35, Alexander Graf wrote:
>> Unfortunately on ARM you also have a few other constraints - the debug
>> register space is partitioned into magic super debug registers at the
>> top (with an implementation specific amount) and normal debug registers
>> in the lower end of the space.
>
> Does the gdbstub ever need to use the magic super debug registers?

No. The extra features of the trailing registers in the range
are "context matching", which means you can set them up as
"breakpoint on this context ID" or "breakpoint on this VMID"
[read: breakpoints specific to a particular process ID or
particular VM], generally as linked breakpoints adding a
constraint to a plain address-match breakpoint register.
The gdbstub doesn't use context bps and I don't think Linux
does either (though of course it might in future).

-- PMM