Re: “Backend has not unmapped grant” errors

[Juergen Gross <jgross@suse.com>]

[-- Attachment #1.1.1: Type: text/plain, Size: 1704 bytes --]

On 23.08.22 09:40, Demi Marie Obenour wrote:
> I recently had a VM’s /dev/xvdb stop working with a “backend has not
> unmapped grant” error.  Since /dev/xvdb was the VM’s private volume,
> that rendered the VM effectively useless.  I had to kill it with
> qvm-kill.
> 
> The backend of /dev/xvdb is dom0, so a malicious backend is clearly not
> the cause of this.  I believe the actual cause is a race condition, such
> as the following:
> 
> 1. GUI agent in VM allocates grant X.
> 2. GUI agent tells GUI daemon in dom0 to map X.
> 3. GUI agent frees grant X.
> 4. blkfront allocates grant X and passes it to dom0.
> 5. dom0’s blkback maps grant X.
> 6. blkback unmaps grant X.
> 7. GUI daemon maps grant X.
> 8. blkfront tries to revoke access to grant X and fails.  Disaster
>     ensues.
> 
> What could be done to prevent this race?  Right now all of the
> approaches I can think of are horribly backwards-incompatible.  They
> require replacing grant IDs with some sort of handle, and requiring
> userspace to pass these handles to ioctls.  It is also possible that
> netfront and blkfront could race against each other in a way that causes
> this, though I suspect that race would be much harder to trigger.
> 
> This has happened more than once so it is not a fluke due to e.g. cosmic
> rays or other random bit-flips.
> 
> Marek, do you have any suggestions?

To me that sounds like the interface of the GUI is the culprit.

The GUI agent in the guest should only free a grant, if it got a message
from the backend that it can do so. Just assuming to be able to free it
because it isn't in use currently is the broken assumption here.


Juergen

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3149 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]