Re: nvme-host: disk corruptions when issuing IDENTIFY commands via ioctl()

Re: nvme-host: disk corruptions when issuing IDENTIFY commands via ioctl()

[Ming Lei]

On Tue, Mar 08, 2022 at 04:39:04PM -0800, Keith Busch wrote:
> On Wed, Mar 09, 2022 at 08:18:47AM +0800, Ming Lei wrote:
> > Given NVMe spec states that data length of IDENTIFY command should be
> > 4096bytes, and PRP list can't be used. 
> > 
> > So looks nvme driver need to validate the command before submitting to
> > hardware, otherwise any buggy application can break FS or memory easily.
> 
> No way. The driver does not police the user passthrough interface for
> these kinds of things.

So you trust application to provide correct data always?

From user viewpoint, this defect provides one easy hole to break FS or
memory, it is one serious issue, IMO. The FS/memory corruption can
be reproduced easily even in VM.

> It couldn't ever be complete or future proof if
> it did.

But the spec states clearly the data length of IDENTIFY command is 4096
and PRP list can't be used, so why do you think it isn't complete or
future proof to validate data length of IDENTIFY in nvme driver?



Thanks, 
Ming

Re: nvme-host: disk corruptions when issuing IDENTIFY commands via ioctl()

[Keith Busch]

On Wed, Mar 09, 2022 at 09:02:42AM +0800, Ming Lei wrote:
> On Tue, Mar 08, 2022 at 04:39:04PM -0800, Keith Busch wrote:
> > On Wed, Mar 09, 2022 at 08:18:47AM +0800, Ming Lei wrote:
> > > Given NVMe spec states that data length of IDENTIFY command should be
> > > 4096bytes, and PRP list can't be used. 
> > > 
> > > So looks nvme driver need to validate the command before submitting to
> > > hardware, otherwise any buggy application can break FS or memory easily.
> > 
> > No way. The driver does not police the user passthrough interface for
> > these kinds of things.
> 
> So you trust application to provide correct data always?
>
> From user viewpoint, this defect provides one easy hole to break FS or
> memory, it is one serious issue, IMO. The FS/memory corruption can
> be reproduced easily even in VM.

It doesn't seem so serious considering it's been this way for 10 years,
and we already knew about this. It's even been reported before:

  http://lists.infradead.org/pipermail/linux-nvme/2013-August/000365.html

> > It couldn't ever be complete or future proof if
> > it did.
> 
> But the spec states clearly the data length of IDENTIFY command is 4096
> and PRP list can't be used, so why do you think it isn't complete or
> future proof to validate data length of IDENTIFY in nvme driver?

The current spec says that opcode uses 4k today. What about some time in
the future? And why are you focusing on Identify anyway? The same
potential for abuse exists with any of the other numerous opcodes that
don't have a fixed transfer size, most of which the driver couldn't
possibly ever know what the transfer length is supposed to be. This is a
priviledged operation; the applications get to own the fallout if they
misuse it.

Re: nvme-host: disk corruptions when issuing IDENTIFY commands via ioctl()

[Ming Lei]

On Tue, Mar 08, 2022 at 05:14:29PM -0800, Keith Busch wrote:
> On Wed, Mar 09, 2022 at 09:02:42AM +0800, Ming Lei wrote:
> > On Tue, Mar 08, 2022 at 04:39:04PM -0800, Keith Busch wrote:
> > > On Wed, Mar 09, 2022 at 08:18:47AM +0800, Ming Lei wrote:
> > > > Given NVMe spec states that data length of IDENTIFY command should be
> > > > 4096bytes, and PRP list can't be used. 
> > > > 
> > > > So looks nvme driver need to validate the command before submitting to
> > > > hardware, otherwise any buggy application can break FS or memory easily.
> > > 
> > > No way. The driver does not police the user passthrough interface for
> > > these kinds of things.
> > 
> > So you trust application to provide correct data always?
> >
> > From user viewpoint, this defect provides one easy hole to break FS or
> > memory, it is one serious issue, IMO. The FS/memory corruption can
> > be reproduced easily even in VM.
> 
> It doesn't seem so serious considering it's been this way for 10 years,
> and we already knew about this. It's even been reported before:
> 
>   http://lists.infradead.org/pipermail/linux-nvme/2013-August/000365.html

BTW, this issue is actually one real report from one Red Hat Customer.

> 
> > > It couldn't ever be complete or future proof if
> > > it did.
> > 
> > But the spec states clearly the data length of IDENTIFY command is 4096
> > and PRP list can't be used, so why do you think it isn't complete or
> > future proof to validate data length of IDENTIFY in nvme driver?
> 
> The current spec says that opcode uses 4k today. What about some time in
> the future?

spec change should only be applied on future hardware, which can not break
current in-market hardware.

nvme target has validated the Identify's transfer length already.

> And why are you focusing on Identify anyway?

Nvme spec states explicitly that the following 4 commands can't use PRP list:

- Identify command
- Namespace Attachment command
- Namespace Management command
- Set Features command

So it should be enough to just validate these commands.

Thanks, 
Ming

Re: nvme-host: disk corruptions when issuing IDENTIFY commands via ioctl()

[Keith Busch]

On Wed, Mar 09, 2022 at 10:48:35AM +0800, Ming Lei wrote:
> > > On Tue, Mar 08, 2022 at 04:39:04PM -0800, Keith Busch wrote:
> 
> BTW, this issue is actually one real report from one Red Hat Customer.

And the correct fix has always been to fix the application.

> > > 
> > > But the spec states clearly the data length of IDENTIFY command is 4096
> > > and PRP list can't be used, so why do you think it isn't complete or
> > > future proof to validate data length of IDENTIFY in nvme driver?
> > 
> > The current spec says that opcode uses 4k today. What about some time in
> > the future?
> 
> spec change should only be applied on future hardware, which can not break
> current in-market hardware.

If a new Identify CNS were invented by committee that uses 8k, then the
older driver enforcing only 4k mappings will create more corruption, and
then it would be the driver's fault.

> nvme target has validated the Identify's transfer length already.

nvmet provides a fabrics targets, which uses SGL, not PRP. The SGL
encodes the length, so it's possible to validate it.
 
> > And why are you focusing on Identify anyway?
> 
> Nvme spec states explicitly that the following 4 commands can't use PRP list:
> 
> - Identify command
> - Namespace Attachment command
> - Namespace Management command
> - Set Features command
> 
> So it should be enough to just validate these commands.

Why are these 4 opcodes so special that the driver should provide
training wheels for broken apps, yet it must trust the same app with the
hundreds of other possible opcodes through the same interface?