[Qemu-devel] Emulation of TCG OPAL self-encrypting drive

[Qemu-devel] Emulation of TCG OPAL self-encrypting drive

[David Kozub]

Hi,

Can QEMU emulate an OPAL disk? The only relevant thing I found is a post 
from 2017 about TPM that mentions OPAL:
https://lists.gnu.org/archive/html/qemu-devel/2017-07/msg04586.html

specifically this bit:

> Well, at some point somebody's going to want us to implement this,
> but... they can do that when they do that.

So I assume it is not implemented. (?)

I agree with the sentiment expressed in the mail linked above w.r.t. OPAL 
security. I'm interested in this from SW development/debugging/fiddling 
perspective. A sufficient solution for me would not add any real 
encryption but would respond to the various OPAL commands send via ATA 
TRUSTED SEND/RECEIVE commands.

In fact, a more generic solution would work for me: If it was possible to 
send ATA commands from QEMU to a separate process which could then handle 
them as it liked and reply back to QEMU. This could be useful for other 
fiddling/debugging situations too.

Or, just a pass-through to a block device in the host - but a pass-through 
that would allow OPAL commands.

I'm grateful for any hints/ideas. Perhaps something like this is already 
possible with QEMU?

Best regards,
David

Re: [Qemu-devel] Emulation of TCG OPAL self-encrypting drive

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1740 bytes --]

On Sat, Jan 05, 2019 at 07:27:03PM +0100, David Kozub wrote:
> Hi,
> 
> Can QEMU emulate an OPAL disk? The only relevant thing I found is a post
> from 2017 about TPM that mentions OPAL:
> https://lists.gnu.org/archive/html/qemu-devel/2017-07/msg04586.html

CCing John Snow (IDE/ATA) and Kevin Wolf (QEMU block layer).

> specifically this bit:
> 
> > Well, at some point somebody's going to want us to implement this,
> > but... they can do that when they do that.
> 
> So I assume it is not implemented. (?)

Right.

> I agree with the sentiment expressed in the mail linked above w.r.t. OPAL
> security. I'm interested in this from SW development/debugging/fiddling
> perspective. A sufficient solution for me would not add any real encryption

QEMU supports LUKS encrypted disk images so no new code is needed for
the actual encryption.

> but would respond to the various OPAL commands send via ATA TRUSTED
> SEND/RECEIVE commands.
> 
> In fact, a more generic solution would work for me: If it was possible to
> send ATA commands from QEMU to a separate process which could then handle
> them as it liked and reply back to QEMU. This could be useful for other
> fiddling/debugging situations too.

Might as well implement it in QEMU so users can easily take advantage of
it without setting up external software.

> Or, just a pass-through to a block device in the host - but a pass-through
> that would allow OPAL commands.

You can pass through a storage controller using PCI passthrough or you
can pass through a SCSI LUN, but there is no ATA passthrough.

> I'm grateful for any hints/ideas. Perhaps something like this is already
> possible with QEMU?
> 
> Best regards,
> David
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] Emulation of TCG OPAL self-encrypting drive

[David Kozub]

On Mon, 7 Jan 2019, Stefan Hajnoczi wrote:

> QEMU supports LUKS encrypted disk images so no new code is needed for
> the actual encryption.

Thanks for the feedback, Stefan. I know very little about qemu internals 
(I looked around a bit). One issue is: OPAL needs some persistent data 
outside of the actual user-visible data. How does that fit in with storage 
in QEMU? Perhaps the implementation could just occupy a fixed size of the 
associated storage for the OPAL state.

>> Or, just a pass-through to a block device in the host - but a pass-through
>> that would allow OPAL commands.
>
> You can pass through a storage controller using PCI passthrough or you
> can pass through a SCSI LUN, but there is no ATA passthrough.

I currently don't have a usable box for PCI passthrough. I'm thinking 
that ATA passthrough could be generally usable for any fiddling and 
perhaps not too difficult to implement.

If I understand QEMU sources correctly, this needs to touch hw/ide/core.c 
(ide_exec_cmd), either adding a layer for OPAL, or just forwarding ATA 
commands for pass-through. Right?

Best regards,
David

Re: [Qemu-devel] Emulation of TCG OPAL self-encrypting drive

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1351 bytes --]

On Thu, Jan 10, 2019 at 12:05:32AM +0100, David Kozub wrote:
> On Mon, 7 Jan 2019, Stefan Hajnoczi wrote:
> 
> > QEMU supports LUKS encrypted disk images so no new code is needed for
> > the actual encryption.
> 
> Thanks for the feedback, Stefan. I know very little about qemu internals (I
> looked around a bit). One issue is: OPAL needs some persistent data outside
> of the actual user-visible data. How does that fit in with storage in QEMU?
> Perhaps the implementation could just occupy a fixed size of the associated
> storage for the OPAL state.

See block/crypto.c for the LUKS block driver.  Perhaps OPAL needs to
something similar (OPAL state + LUKS).

> > > Or, just a pass-through to a block device in the host - but a pass-through
> > > that would allow OPAL commands.
> > 
> > You can pass through a storage controller using PCI passthrough or you
> > can pass through a SCSI LUN, but there is no ATA passthrough.
> 
> I currently don't have a usable box for PCI passthrough. I'm thinking that
> ATA passthrough could be generally usable for any fiddling and perhaps not
> too difficult to implement.
> 
> If I understand QEMU sources correctly, this needs to touch hw/ide/core.c
> (ide_exec_cmd), either adding a layer for OPAL, or just forwarding ATA
> commands for pass-through. Right?

Yes.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] Emulation of TCG OPAL self-encrypting drive

[John Snow]

On 1/7/19 4:16 AM, Stefan Hajnoczi wrote:
> On Sat, Jan 05, 2019 at 07:27:03PM +0100, David Kozub wrote:
>> Hi,
>>
>> Can QEMU emulate an OPAL disk? The only relevant thing I found is a post
>> from 2017 about TPM that mentions OPAL:
>> https://lists.gnu.org/archive/html/qemu-devel/2017-07/msg04586.html
> 
> CCing John Snow (IDE/ATA) and Kevin Wolf (QEMU block layer).
> 
>> specifically this bit:
>>
>>> Well, at some point somebody's going to want us to implement this,
>>> but... they can do that when they do that.
>>
>> So I assume it is not implemented. (?)
> 
> Right.
> 
>> I agree with the sentiment expressed in the mail linked above w.r.t. OPAL
>> security. I'm interested in this from SW development/debugging/fiddling
>> perspective. A sufficient solution for me would not add any real encryption
> 
> QEMU supports LUKS encrypted disk images so no new code is needed for
> the actual encryption.
> 
>> but would respond to the various OPAL commands send via ATA TRUSTED
>> SEND/RECEIVE commands.
>>
>> In fact, a more generic solution would work for me: If it was possible to
>> send ATA commands from QEMU to a separate process which could then handle
>> them as it liked and reply back to QEMU. This could be useful for other
>> fiddling/debugging situations too.
> 
> Might as well implement it in QEMU so users can easily take advantage of
> it without setting up external software.
> 
>> Or, just a pass-through to a block device in the host - but a pass-through
>> that would allow OPAL commands.
> 
> You can pass through a storage controller using PCI passthrough or you
> can pass through a SCSI LUN, but there is no ATA passthrough.
> 
>> I'm grateful for any hints/ideas. Perhaps something like this is already
>> possible with QEMU?
>>
>> Best regards,
>> David
>>

I can answer some questions about the ATA layer, but I'm not well read
on OPAL or the interrelationship between the two.

We don't have an ATA-style passthrough in QEMU right now and nobody has
ever asked! Would you mind elaborating for me what kind of setup you're
looking to accomplish and maybe I can give you some better hints?

--js

Re: [Qemu-devel] Emulation of TCG OPAL self-encrypting drive

[David Kozub]

On Wed, 16 Jan 2019, John Snow wrote:

> I can answer some questions about the ATA layer, but I'm not well read
> on OPAL or the interrelationship between the two.
>
> We don't have an ATA-style passthrough in QEMU right now and nobody has
> ever asked! Would you mind elaborating for me what kind of setup you're
> looking to accomplish and maybe I can give you some better hints?
>
> --js

Hi John,

I'm thinking about adding OPAL support into grub. And I thought being able 
to do the development with qemu would make the development much easier.

While I read the OPAL spec, I can't say I understand it. But as far as I 
know, the relationship between OPAL and ATA is not too complicated. The 
OPAL spec itself is written in interface-agnostic way. It just defines 
there should be two interface commands ("Trusted Commands"), described as 
"IF-SEND" and "IF-RECV". (See [1], 1.4.1 Global Terminology and 2.3.1 Host 
<-> TPer Communication Infrastructure) and the OPAL spec essentially 
defines the paylods for these.

The actual definition of what "Trused Commands" are for ATA comes from 
"TCG Storage Interface Interactions Specification (SIIS)"[3], specifically 
"4.2 Mapping of IF-SEND and IF-RECV". (Well, it's not that simple, as 
there are other things like error handling.)

Of course, OPAL affects other commands, like reads and writes, based on 
the OPAL state (see [1], 5.7 Locking Template).

Furthermore, there is the "shadow MBR" feature, which, when enabled (and 
not marked as done), makes the disk present a different content at the 
beginning. (This is typically used to store a program that can be booted, 
asks for password, unlocks the disk, sets the "mbr done" flag to hide the 
shadow MBR and reboots to the real OS. - see e.g. 5.7.2.5.3 Done in [1].)

Here are the specs relevant for OPAL v2:
[1] https://trustedcomputinggroup.org/wp-content/uploads/TCG_Storage_Architecture_Core_Spec_v2.01_r1.00.pdf
[2] https://trustedcomputinggroup.org/wp-content/uploads/TCG_Storage-Opal_SSC_v2.01_rev1.00.pdf
[3] https://trustedcomputinggroup.org/wp-content/uploads/TCG_SWG_SIIS_v1.08_r1.00.pdf

Implementing the spec (for what an "OPAL" disk need to support, I think 
[2] is the important document) seems like a big task.

So I'm thinking ATA passthrough might be a more attainable goal. This way 
I could use a real OPAL drive w/o needing a spare controller for PCI 
passthrough. And with some more fiddling, I could even connect this to 
userspace (I was thinking of ATA over Ethernet). This could be useful for 
other development/fiddling scenarios too.

But I don't know much about ATA. Would it be sufficient to just do an 
SG_IO IOCTL on the target block device for each ide_exec_cmd? How do 
things like NCQ fit into this?

Best regards,
David

Re: [Qemu-devel] Emulation of TCG OPAL self-encrypting drive

[John Snow]

On 1/17/19 6:04 PM, David Kozub wrote:
> On Wed, 16 Jan 2019, John Snow wrote:
> 
>> I can answer some questions about the ATA layer, but I'm not well read
>> on OPAL or the interrelationship between the two.
>>
>> We don't have an ATA-style passthrough in QEMU right now and nobody has
>> ever asked! Would you mind elaborating for me what kind of setup you're
>> looking to accomplish and maybe I can give you some better hints?
>>
>> --js
> 
> Hi John,
> 
> I'm thinking about adding OPAL support into grub. And I thought being
> able to do the development with qemu would make the development much
> easier.
> 
> While I read the OPAL spec, I can't say I understand it. But as far as I
> know, the relationship between OPAL and ATA is not too complicated. The
> OPAL spec itself is written in interface-agnostic way. It just defines
> there should be two interface commands ("Trusted Commands"), described
> as "IF-SEND" and "IF-RECV". (See [1], 1.4.1 Global Terminology and 2.3.1
> Host <-> TPer Communication Infrastructure) and the OPAL spec
> essentially defines the paylods for these.
> 
> The actual definition of what "Trused Commands" are for ATA comes from
> "TCG Storage Interface Interactions Specification (SIIS)"[3],
> specifically "4.2 Mapping of IF-SEND and IF-RECV". (Well, it's not that
> simple, as there are other things like error handling.)
> 
> Of course, OPAL affects other commands, like reads and writes, based on
> the OPAL state (see [1], 5.7 Locking Template).
> 
> Furthermore, there is the "shadow MBR" feature, which, when enabled (and
> not marked as done), makes the disk present a different content at the
> beginning. (This is typically used to store a program that can be
> booted, asks for password, unlocks the disk, sets the "mbr done" flag to
> hide the shadow MBR and reboots to the real OS. - see e.g. 5.7.2.5.3
> Done in [1].)
> 
> Here are the specs relevant for OPAL v2:
> [1]
> https://trustedcomputinggroup.org/wp-content/uploads/TCG_Storage_Architecture_Core_Spec_v2.01_r1.00.pdf
> 
> [2]
> https://trustedcomputinggroup.org/wp-content/uploads/TCG_Storage-Opal_SSC_v2.01_rev1.00.pdf
> 
> [3]
> https://trustedcomputinggroup.org/wp-content/uploads/TCG_SWG_SIIS_v1.08_r1.00.pdf
> 
> 
> Implementing the spec (for what an "OPAL" disk need to support, I think
> [2] is the important document) seems like a big task.
> 
> So I'm thinking ATA passthrough might be a more attainable goal. This
> way I could use a real OPAL drive w/o needing a spare controller for PCI
> passthrough. And with some more fiddling, I could even connect this to
> userspace (I was thinking of ATA over Ethernet). This could be useful
> for other development/fiddling scenarios too.
> 
> But I don't know much about ATA. Would it be sufficient to just do an
> SG_IO IOCTL on the target block device for each ide_exec_cmd? How do
> things like NCQ fit into this?
> 
> Best regards,
> David

Admittedly I'm not too sure of how the ATA support in Linux works to
know what the passthrough would actually look like, bit-wise. I know
there's some SCSI abstraction layer that can drive ATA devices, but I'm
not completely clear on the actual plumbing. How you want to attempt
this might depend on what the linux ATA drivers look like and at which
interface layer you could conceivably drive them. I don't know the
answer to this.

I can give you maybe a brief overview of some of the obviously useful
choke points in QEMU, though...

the ATA support in QEMU comes in a few different levels:

(1) IDE/ATA/PATA disks use a register set and PIO to directly read and
write values to individual registers. You can see this interface in
hw/ide.core.c for ide_ioport_write, ide_status_read, and ide_cmd_write.
When the drive is in a PIO data loop, you can read or write data to a
buffer by repeatedly writing to a certain register address, implemented
with ide_data_[read|write][w|l].


ide_exec_cmd serves as the "start processing" signal in QEMU, and uses
the various registers manipulated in the above calls stored in `IDEState
*s` to know which command to emulate. The arguments to ide_exec_cmd
aren't sufficient instruction alone. ide_exec_cmd is triggered whenever
the guest updates the command register.

CDROM emulation actually does use SCSI packets. Generally the guest
sends the 0xA0 PACKET command to the drive and then the drive waits for
a SCSI CDB to arrive via PIO. When the packet has arrived in full,
ide_atapi_cmd() processes it. However, there are a few places in this
code where we dip into the ATA registers to formulate a reply, so the
logical split isn't perfect.

(2) PCI IDE utilizes additional BMDMA features outside of those core
registers and are driven separately. It does not fully wrap the register
interface present.

(3) SATA devices begin using FIS packets. They're a message format that
lets you send commands, update registers, read values, etc. They're the
basic interface unit at this level. Both NCQ and traditional ATA
commands are delivered using FIS Register Host-to-Device update packets.
(The command, as always, is activated when the ATA device itself
receives an update to its command register.)

QEMU doesn't have a clean separation for ATA and SATA emulation, so the
SATA device emulation actually happens at the interface layer in QEMU
instead, as a hack. See hw/ide/ahci.c and look for this blurb:

```
    /* Check for NCQ command */
    if (is_ncq(cmd_fis[2])) {
	process_ncq_command(s, port, cmd_fis, slot);
        return;
    }
```

This feels like maybe high level and useful enough to be able to
intercept for passthrough purposes, but I'm not sure how to handle
things like DMA routing or PIO access, which is still... technically
allowed at this layer, and might be used by early bootup routines.

A problem is that QEMU does not really disentangle the concept of a
"SATA Device" and the "AHCI controller", so a lot of the FIS responses
in QEMU go straight into the controller's buffer, and we'd have to split
all of that out.

(4) the AHCI controller manages sending and receiving the FIS packets.
You fill a buffer with the FIS packet to send and manipulate AHCI PCI
registers to send it off. FIS responses are buffered from the SATA
drives informing the controller of the new register values.

the AHCI command buffers include space for guests to pre-write their
SCSI CDBs, and the controller handles sending both the outer ATA command
and the inner ATAPI packet to the device.

Again, in QEMU, we cheat a little and layers (3) and (4) are pretty well
smooshed together. In general, the AHCI layer sends FIS packets back and
forth from the SATA layer, which decomposes the FIS packets into
constituent register updates, which are sent into layer (1) for
processing. None of these layers are really truly strictly separated,
unfortunately.




...I'd have to read up a bit on what interfaces Linux offers where we
might be able to do an ATA passthrough and I can make better
recommendations on where to take the surgeon's scalpel.

Hope this helps even 1% instead of just being a useless info dump.

--js

Re: [Qemu-devel] Emulation of TCG OPAL self-encrypting drive

[David Kozub]

On Thu, 17 Jan 2019, John Snow wrote:

> Admittedly I'm not too sure of how the ATA support in Linux works to
> know what the passthrough would actually look like, bit-wise. I know
> there's some SCSI abstraction layer that can drive ATA devices, but I'm
> not completely clear on the actual plumbing. How you want to attempt
> this might depend on what the linux ATA drivers look like and at which
> interface layer you could conceivably drive them. I don't know the
> answer to this.

Hi John,

I looked into how to send ATA commands in Linux from userspace. It seems 
to be a relatively obscure topic (it's not every day that one needs this, 
after all).

All googling points me to SG_IO ioctl. Searching for info on that IOCTL 
leads to some documentation.[1] But not only does that date to 2008, it 
also talks only about SCSI, so I was wondering where/how does ATA come 
into picture. Yet tools like hdparm and sedutil use SG_IO to send ATA 
commands.

I dived into Linux kernel sources. There are other relevant IOCTLs, like 
HDIO_DRIVE_CMD, but they seem quite constrained. SG_IO, at least when 
issued on a disk that uses the "sd" driver, reaches kernel's libata and 
libata is then able to unwrap the command and pass it on to an ATA disk. 
(I believe this is the abstraction layer you mention.)

Specifically, ata_scsi_queuecmd is invoked. It then calls either 
ata_scsi_translate or ata_scsi_simulate. The pass-through translation 
(from a SCSI cmd to ATA taskfile) is implemented in ata_scsi_pass_thru.[2]

I think SG_IO IOCTL allows one to send a single ATA command and the call 
blocks until the request is processed.

I don't know if there are other mechanisms how to pass ATA commands from 
userspace.

If anybody reading this has more ideas/info on this topic, please let me 
know.

[1] http://sg.danny.cz/sg/sg_io.html
[2] https://github.com/torvalds/linux/blob/30bac164aca750892b93eef350439a0562a68647/drivers/ata/libata-scsi.c#L3138

> I can give you maybe a brief overview of some of the obviously useful
> choke points in QEMU, though...
>
> the ATA support in QEMU comes in a few different levels:
>
> (1) IDE/ATA/PATA disks use a register set and PIO to directly read and
> write values to individual registers. You can see this interface in
> hw/ide.core.c for ide_ioport_write, ide_status_read, and ide_cmd_write.
> When the drive is in a PIO data loop, you can read or write data to a
> buffer by repeatedly writing to a certain register address, implemented
> with ide_data_[read|write][w|l].
>
>
> ide_exec_cmd serves as the "start processing" signal in QEMU, and uses
> the various registers manipulated in the above calls stored in `IDEState
> *s` to know which command to emulate. The arguments to ide_exec_cmd
> aren't sufficient instruction alone. ide_exec_cmd is triggered whenever
> the guest updates the command register.
>
> CDROM emulation actually does use SCSI packets. Generally the guest
> sends the 0xA0 PACKET command to the drive and then the drive waits for
> a SCSI CDB to arrive via PIO. When the packet has arrived in full,
> ide_atapi_cmd() processes it. However, there are a few places in this
> code where we dip into the ATA registers to formulate a reply, so the
> logical split isn't perfect.
>
> (2) PCI IDE utilizes additional BMDMA features outside of those core
> registers and are driven separately. It does not fully wrap the register
> interface present.
>
> (3) SATA devices begin using FIS packets. They're a message format that
> lets you send commands, update registers, read values, etc. They're the
> basic interface unit at this level. Both NCQ and traditional ATA
> commands are delivered using FIS Register Host-to-Device update packets.
> (The command, as always, is activated when the ATA device itself
> receives an update to its command register.)
>
> QEMU doesn't have a clean separation for ATA and SATA emulation, so the
> SATA device emulation actually happens at the interface layer in QEMU
> instead, as a hack. See hw/ide/ahci.c and look for this blurb:
>
> ```
>    /* Check for NCQ command */
>    if (is_ncq(cmd_fis[2])) {
> 	process_ncq_command(s, port, cmd_fis, slot);
>        return;
>    }
> ```
>
> This feels like maybe high level and useful enough to be able to
> intercept for passthrough purposes, but I'm not sure how to handle
> things like DMA routing or PIO access, which is still... technically
> allowed at this layer, and might be used by early bootup routines.
>
> A problem is that QEMU does not really disentangle the concept of a
> "SATA Device" and the "AHCI controller", so a lot of the FIS responses
> in QEMU go straight into the controller's buffer, and we'd have to split
> all of that out.
>
> (4) the AHCI controller manages sending and receiving the FIS packets.
> You fill a buffer with the FIS packet to send and manipulate AHCI PCI
> registers to send it off. FIS responses are buffered from the SATA
> drives informing the controller of the new register values.
>
> the AHCI command buffers include space for guests to pre-write their
> SCSI CDBs, and the controller handles sending both the outer ATA command
> and the inner ATAPI packet to the device.
>
> Again, in QEMU, we cheat a little and layers (3) and (4) are pretty well
> smooshed together. In general, the AHCI layer sends FIS packets back and
> forth from the SATA layer, which decomposes the FIS packets into
> constituent register updates, which are sent into layer (1) for
> processing. None of these layers are really truly strictly separated,
> unfortunately.

A naive question... which of these "details" (as in... was the command 
dispatched via PIO, or in a FIS packet) are needed for ATA pass-through? 
The only mechanism I'm currently aware of, in Linux, lets us pass a single 
command, synchronously, and it doesn't let us tune much more. This sounds 
like a horrible thing for performance, but this might in itself be OK for 
the fiddling/debugging scenarios I'm after. But even ignoring performance, 
is this sufficient? I.e. to extract - through whatever means - a sequence 
of ATA commands (no matter how they were delivered from the guest), then 
sequentially execute them, and then deliver them to the guest in the 
appripriate way (depending on how they were received)?

> Hope this helps even 1% instead of just being a useless info dump.

Thank you for this "executive summary". It helps me find the relevant code 
in QEMU and the terms to google for, as I don't know much about this.

Best regards,
David

Re: [Qemu-devel] Emulation of TCG OPAL self-encrypting drive

[John Snow]

On 1/23/19 5:39 PM, David Kozub wrote:
> On Thu, 17 Jan 2019, John Snow wrote:
> 
>> Admittedly I'm not too sure of how the ATA support in Linux works to
>> know what the passthrough would actually look like, bit-wise. I know
>> there's some SCSI abstraction layer that can drive ATA devices, but I'm
>> not completely clear on the actual plumbing. How you want to attempt
>> this might depend on what the linux ATA drivers look like and at which
>> interface layer you could conceivably drive them. I don't know the
>> answer to this.
> 
> Hi John,
> 
> I looked into how to send ATA commands in Linux from userspace. It seems
> to be a relatively obscure topic (it's not every day that one needs
> this, after all).
> 
> All googling points me to SG_IO ioctl. Searching for info on that IOCTL
> leads to some documentation.[1] But not only does that date to 2008, it
> also talks only about SCSI, so I was wondering where/how does ATA come
> into picture. Yet tools like hdparm and sedutil use SG_IO to send ATA
> commands.
> 
> I dived into Linux kernel sources. There are other relevant IOCTLs, like
> HDIO_DRIVE_CMD, but they seem quite constrained. SG_IO, at least when
> issued on a disk that uses the "sd" driver, reaches kernel's libata and
> libata is then able to unwrap the command and pass it on to an ATA disk.
> (I believe this is the abstraction layer you mention.)
> 
> Specifically, ata_scsi_queuecmd is invoked. It then calls either
> ata_scsi_translate or ata_scsi_simulate. The pass-through translation
> (from a SCSI cmd to ATA taskfile) is implemented in ata_scsi_pass_thru.[2]
> 

That sounds about right to me from what I know of the modern ATA driver
in Linux.

> I think SG_IO IOCTL allows one to send a single ATA command and the call
> blocks until the request is processed.
> 

The problem(?) with this is that you don't get to choose the ATA
command, exactly ... the way the SG_IO interface works for ATA is that
you choose a SCSI command and the linux driver translates it into ATA
for you, then issues that command.

In this way, I don't think* you could do ATA passthrough exactly -- not
in the sense that you can have an emulated ATA device passing through
ATA commands to a real device.

...but you could create an emulated SCSI disk and then pass those SCSI
commands to an ATA device -- achieving a *kind* of pass through, but I
don't know if that's helpful to your project. If so, I'd start looking
at the scsi disk sources instead of the ATA sources.

> I don't know if there are other mechanisms how to pass ATA commands from
> userspace.
> 
> If anybody reading this has more ideas/info on this topic, please let me
> know.
> 
> [1] http://sg.danny.cz/sg/sg_io.html
> [2]
> https://github.com/torvalds/linux/blob/30bac164aca750892b93eef350439a0562a68647/drivers/ata/libata-scsi.c#L3138
> 
> 
>> I can give you maybe a brief overview of some of the obviously useful
>> choke points in QEMU, though...
>>
>> the ATA support in QEMU comes in a few different levels:
>>
>> (1) IDE/ATA/PATA disks use a register set and PIO to directly read and
>> write values to individual registers. You can see this interface in
>> hw/ide.core.c for ide_ioport_write, ide_status_read, and ide_cmd_write.
>> When the drive is in a PIO data loop, you can read or write data to a
>> buffer by repeatedly writing to a certain register address, implemented
>> with ide_data_[read|write][w|l].
>>
>>
>> ide_exec_cmd serves as the "start processing" signal in QEMU, and uses
>> the various registers manipulated in the above calls stored in `IDEState
>> *s` to know which command to emulate. The arguments to ide_exec_cmd
>> aren't sufficient instruction alone. ide_exec_cmd is triggered whenever
>> the guest updates the command register.
>>
>> CDROM emulation actually does use SCSI packets. Generally the guest
>> sends the 0xA0 PACKET command to the drive and then the drive waits for
>> a SCSI CDB to arrive via PIO. When the packet has arrived in full,
>> ide_atapi_cmd() processes it. However, there are a few places in this
>> code where we dip into the ATA registers to formulate a reply, so the
>> logical split isn't perfect.
>>
>> (2) PCI IDE utilizes additional BMDMA features outside of those core
>> registers and are driven separately. It does not fully wrap the register
>> interface present.
>>
>> (3) SATA devices begin using FIS packets. They're a message format that
>> lets you send commands, update registers, read values, etc. They're the
>> basic interface unit at this level. Both NCQ and traditional ATA
>> commands are delivered using FIS Register Host-to-Device update packets.
>> (The command, as always, is activated when the ATA device itself
>> receives an update to its command register.)
>>
>> QEMU doesn't have a clean separation for ATA and SATA emulation, so the
>> SATA device emulation actually happens at the interface layer in QEMU
>> instead, as a hack. See hw/ide/ahci.c and look for this blurb:
>>
>> ```
>>    /* Check for NCQ command */
>>    if (is_ncq(cmd_fis[2])) {
>>     process_ncq_command(s, port, cmd_fis, slot);
>>        return;
>>    }
>> ```
>>
>> This feels like maybe high level and useful enough to be able to
>> intercept for passthrough purposes, but I'm not sure how to handle
>> things like DMA routing or PIO access, which is still... technically
>> allowed at this layer, and might be used by early bootup routines.
>>
>> A problem is that QEMU does not really disentangle the concept of a
>> "SATA Device" and the "AHCI controller", so a lot of the FIS responses
>> in QEMU go straight into the controller's buffer, and we'd have to split
>> all of that out.
>>
>> (4) the AHCI controller manages sending and receiving the FIS packets.
>> You fill a buffer with the FIS packet to send and manipulate AHCI PCI
>> registers to send it off. FIS responses are buffered from the SATA
>> drives informing the controller of the new register values.
>>
>> the AHCI command buffers include space for guests to pre-write their
>> SCSI CDBs, and the controller handles sending both the outer ATA command
>> and the inner ATAPI packet to the device.
>>
>> Again, in QEMU, we cheat a little and layers (3) and (4) are pretty well
>> smooshed together. In general, the AHCI layer sends FIS packets back and
>> forth from the SATA layer, which decomposes the FIS packets into
>> constituent register updates, which are sent into layer (1) for
>> processing. None of these layers are really truly strictly separated,
>> unfortunately.
> 
> A naive question... which of these "details" (as in... was the command
> dispatched via PIO, or in a FIS packet) are needed for ATA pass-through?
> The only mechanism I'm currently aware of, in Linux, lets us pass a
> single command, synchronously, and it doesn't let us tune much more.
> This sounds like a horrible thing for performance, but this might in
> itself be OK for the fiddling/debugging scenarios I'm after. But even
> ignoring performance, is this sufficient? I.e. to extract - through
> whatever means - a sequence of ATA commands (no matter how they were
> delivered from the guest), then sequentially execute them, and then
> deliver them to the guest in the appripriate way (depending on how they
> were received)?
> 

It depends at which level you want to intercept the command, because at
each phase the way the command looks is different.

At the PIO level, the command is actually a series of writes to device
registers instead of looking like any kind of unified "command" that you
could point to.

At the FIS level, it actually starts to resemble a coherent packet that
can be issued to the device. FIS packets are kind of like the SATA
analog to SCSI CDB packets.

None of these are too useful for SG_IO, I think.

>> Hope this helps even 1% instead of just being a useless info dump.
> 
> Thank you for this "executive summary". It helps me find the relevant
> code in QEMU and the terms to google for, as I don't know much about this.
> 
> Best regards,
> David

Re: [Qemu-devel] Emulation of TCG OPAL self-encrypting drive

[David Kozub]

On Wed, 23 Jan 2019, John Snow wrote:
> On 1/23/19 5:39 PM, David Kozub wrote:
>> On Thu, 17 Jan 2019, John Snow wrote:
>
>> I think SG_IO IOCTL allows one to send a single ATA command and the call
>> blocks until the request is processed.
>>
>
> The problem(?) with this is that you don't get to choose the ATA
> command, exactly ... the way the SG_IO interface works for ATA is that
> you choose a SCSI command and the linux driver translates it into ATA
> for you, then issues that command.
>
> In this way, I don't think* you could do ATA passthrough exactly -- not
> in the sense that you can have an emulated ATA device passing through
> ATA commands to a real device.

libata seems to support SCSI / ATA Translation, including ATA PASS THROUGH 
(12) and ATA PASS THROUGH (16). Is this not sufficient? (The 
implementation can be seen in ata_scsi_pass_thru.)

> ...but you could create an emulated SCSI disk and then pass those SCSI
> commands to an ATA device -- achieving a *kind* of pass through, but I
> don't know if that's helpful to your project. If so, I'd start looking
> at the scsi disk sources instead of the ATA sources.

Perhaps. Or maybe fogetting about pass-through and really just 
implementing OPAL in QEMU.

>> I don't know if there are other mechanisms how to pass ATA commands from
>> userspace.
>>
>> If anybody reading this has more ideas/info on this topic, please let me
>> know.
>>
>> [1] http://sg.danny.cz/sg/sg_io.html
>> [2]
>> https://github.com/torvalds/linux/blob/30bac164aca750892b93eef350439a0562a68647/drivers/ata/libata-scsi.c#L3138
>>
>>
>>> I can give you maybe a brief overview of some of the obviously useful
>>> choke points in QEMU, though...
>>>
>>> the ATA support in QEMU comes in a few different levels:
>>>
>>> (1) IDE/ATA/PATA disks use a register set and PIO to directly read and
>>> write values to individual registers. You can see this interface in
>>> hw/ide.core.c for ide_ioport_write, ide_status_read, and ide_cmd_write.
>>> When the drive is in a PIO data loop, you can read or write data to a
>>> buffer by repeatedly writing to a certain register address, implemented
>>> with ide_data_[read|write][w|l].
>>>
>>>
>>> ide_exec_cmd serves as the "start processing" signal in QEMU, and uses
>>> the various registers manipulated in the above calls stored in `IDEState
>>> *s` to know which command to emulate. The arguments to ide_exec_cmd
>>> aren't sufficient instruction alone. ide_exec_cmd is triggered whenever
>>> the guest updates the command register.
>>>
>>> CDROM emulation actually does use SCSI packets. Generally the guest
>>> sends the 0xA0 PACKET command to the drive and then the drive waits for
>>> a SCSI CDB to arrive via PIO. When the packet has arrived in full,
>>> ide_atapi_cmd() processes it. However, there are a few places in this
>>> code where we dip into the ATA registers to formulate a reply, so the
>>> logical split isn't perfect.
>>>
>>> (2) PCI IDE utilizes additional BMDMA features outside of those core
>>> registers and are driven separately. It does not fully wrap the register
>>> interface present.
>>>
>>> (3) SATA devices begin using FIS packets. They're a message format that
>>> lets you send commands, update registers, read values, etc. They're the
>>> basic interface unit at this level. Both NCQ and traditional ATA
>>> commands are delivered using FIS Register Host-to-Device update packets.
>>> (The command, as always, is activated when the ATA device itself
>>> receives an update to its command register.)
>>>
>>> QEMU doesn't have a clean separation for ATA and SATA emulation, so the
>>> SATA device emulation actually happens at the interface layer in QEMU
>>> instead, as a hack. See hw/ide/ahci.c and look for this blurb:
>>>
>>> ```
>>>    /* Check for NCQ command */
>>>    if (is_ncq(cmd_fis[2])) {
>>>     process_ncq_command(s, port, cmd_fis, slot);
>>>        return;
>>>    }
>>> ```
>>>
>>> This feels like maybe high level and useful enough to be able to
>>> intercept for passthrough purposes, but I'm not sure how to handle
>>> things like DMA routing or PIO access, which is still... technically
>>> allowed at this layer, and might be used by early bootup routines.
>>>
>>> A problem is that QEMU does not really disentangle the concept of a
>>> "SATA Device" and the "AHCI controller", so a lot of the FIS responses
>>> in QEMU go straight into the controller's buffer, and we'd have to split
>>> all of that out.
>>>
>>> (4) the AHCI controller manages sending and receiving the FIS packets.
>>> You fill a buffer with the FIS packet to send and manipulate AHCI PCI
>>> registers to send it off. FIS responses are buffered from the SATA
>>> drives informing the controller of the new register values.
>>>
>>> the AHCI command buffers include space for guests to pre-write their
>>> SCSI CDBs, and the controller handles sending both the outer ATA command
>>> and the inner ATAPI packet to the device.
>>>
>>> Again, in QEMU, we cheat a little and layers (3) and (4) are pretty well
>>> smooshed together. In general, the AHCI layer sends FIS packets back and
>>> forth from the SATA layer, which decomposes the FIS packets into
>>> constituent register updates, which are sent into layer (1) for
>>> processing. None of these layers are really truly strictly separated,
>>> unfortunately.
>>
>> A naive question... which of these "details" (as in... was the command
>> dispatched via PIO, or in a FIS packet) are needed for ATA pass-through?
>> The only mechanism I'm currently aware of, in Linux, lets us pass a
>> single command, synchronously, and it doesn't let us tune much more.
>> This sounds like a horrible thing for performance, but this might in
>> itself be OK for the fiddling/debugging scenarios I'm after. But even
>> ignoring performance, is this sufficient? I.e. to extract - through
>> whatever means - a sequence of ATA commands (no matter how they were
>> delivered from the guest), then sequentially execute them, and then
>> deliver them to the guest in the appripriate way (depending on how they
>> were received)?
>>
>
> It depends at which level you want to intercept the command, because at
> each phase the way the command looks is different.
>
> At the PIO level, the command is actually a series of writes to device
> registers instead of looking like any kind of unified "command" that you
> could point to.

I really need to read up on this. But at some point with PIO, QEMU figures 
out what to do with the disk (i.e. what to read/write). So QEMU could 
either create an ATA command for this, or maybe even end up doing ordinary 
read/write. But combining read/write via a block device with IOCTLs sounds 
a bit dangerous.

>
> At the FIS level, it actually starts to resemble a coherent packet that
> can be issued to the device. FIS packets are kind of like the SATA
> analog to SCSI CDB packets.
>
> None of these are too useful for SG_IO, I think.
>
>>> Hope this helps even 1% instead of just being a useless info dump.
>>
>> Thank you for this "executive summary". It helps me find the relevant
>> code in QEMU and the terms to google for, as I don't know much about this.
>>
>> Best regards,
>> David

Re: [Qemu-devel] Emulation of TCG OPAL self-encrypting drive

[John Snow]

On 1/24/19 5:24 AM, David Kozub wrote:
>>
> 
> libata seems to support SCSI / ATA Translation, including ATA PASS
> THROUGH (12) and ATA PASS THROUGH (16). Is this not sufficient? (The
> implementation can be seen in ata_scsi_pass_thru.)
> 

Oh, I missed this! Thanks for pointing it out. I'll take a look and see
if I can recommend where to start making incisions in QEMU.

>> ...but you could create an emulated SCSI disk and then pass those SCSI
>> commands to an ATA device -- achieving a *kind* of pass through, but I
>> don't know if that's helpful to your project. If so, I'd start looking
>> at the scsi disk sources instead of the ATA sources.
> 
> Perhaps. Or maybe fogetting about pass-through and really just
> implementing OPAL in QEMU.