[PATCH crypto-2.6] crypto: ccp: add timeout support in the SEV command

[PATCH crypto-2.6] crypto: ccp: add timeout support in the SEV command

[Brijesh Singh]

Currently, the CCP driver assumes that the SEV command issued to the PSP
will always return (i.e. it will never hang).  But recently, firmware bugs
have shown that a command can hang.  Since of the SEV commands are used
in probe routines, this can cause boot hangs and/or loss of virtualization
capabilities.

To protect against firmware bugs, add a timeout in the SEV command
execution flow.  If a command does not complete within the specified
timeout then return -ETIMEOUT and stop the driver from executing any
further commands since the state of the SEV firmware is unknown.

Cc: Tom Lendacky <thomas.lendacky@amd.com>
Cc: Gary Hook <Gary.Hook@amd.com>
Cc: Herbert Xu <herbert@gondor.apana.org.au>
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Brijesh Singh <brijesh.singh@amd.com>
---
 drivers/crypto/ccp/psp-dev.c | 46 +++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 41 insertions(+), 5 deletions(-)

diff --git a/drivers/crypto/ccp/psp-dev.c b/drivers/crypto/ccp/psp-dev.c
index 218739b..72790d8 100644
--- a/drivers/crypto/ccp/psp-dev.c
+++ b/drivers/crypto/ccp/psp-dev.c
@@ -38,6 +38,17 @@ static DEFINE_MUTEX(sev_cmd_mutex);
 static struct sev_misc_dev *misc_dev;
 static struct psp_device *psp_master;
 
+static int psp_cmd_timeout = 100;
+module_param(psp_cmd_timeout, int, 0644);
+MODULE_PARM_DESC(psp_cmd_timeout, " default timeout value, in seconds, for PSP commands");
+
+static int psp_probe_timeout = 5;
+module_param(psp_probe_timeout, int, 0644);
+MODULE_PARM_DESC(psp_probe_timeout, " default timeout value, in seconds, during PSP device probe");
+
+static bool psp_dead;
+static int psp_timeout;
+
 static struct psp_device *psp_alloc_struct(struct sp_device *sp)
 {
 	struct device *dev = sp->dev;
@@ -82,10 +93,19 @@ static irqreturn_t psp_irq_handler(int irq, void *data)
 	return IRQ_HANDLED;
 }
 
-static void sev_wait_cmd_ioc(struct psp_device *psp, unsigned int *reg)
+static int sev_wait_cmd_ioc(struct psp_device *psp,
+			    unsigned int *reg, unsigned int timeout)
 {
-	wait_event(psp->sev_int_queue, psp->sev_int_rcvd);
+	int ret;
+
+	ret = wait_event_timeout(psp->sev_int_queue,
+			psp->sev_int_rcvd, timeout * HZ);
+	if (!ret)
+		return -ETIMEDOUT;
+
 	*reg = ioread32(psp->io_regs + psp->vdata->cmdresp_reg);
+
+	return 0;
 }
 
 static int sev_cmd_buffer_len(int cmd)
@@ -133,12 +153,15 @@ static int __sev_do_cmd_locked(int cmd, void *data, int *psp_ret)
 	if (!psp)
 		return -ENODEV;
 
+	if (psp_dead)
+		return -EBUSY;
+
 	/* Get the physical address of the command buffer */
 	phys_lsb = data ? lower_32_bits(__psp_pa(data)) : 0;
 	phys_msb = data ? upper_32_bits(__psp_pa(data)) : 0;
 
-	dev_dbg(psp->dev, "sev command id %#x buffer 0x%08x%08x\n",
-		cmd, phys_msb, phys_lsb);
+	dev_dbg(psp->dev, "sev command id %#x buffer 0x%08x%08x timeout %us\n",
+		cmd, phys_msb, phys_lsb, psp_timeout);
 
 	print_hex_dump_debug("(in):  ", DUMP_PREFIX_OFFSET, 16, 2, data,
 			     sev_cmd_buffer_len(cmd), false);
@@ -154,7 +177,18 @@ static int __sev_do_cmd_locked(int cmd, void *data, int *psp_ret)
 	iowrite32(reg, psp->io_regs + psp->vdata->cmdresp_reg);
 
 	/* wait for command completion */
-	sev_wait_cmd_ioc(psp, &reg);
+	ret = sev_wait_cmd_ioc(psp, &reg, psp_timeout);
+	if (ret) {
+		if (psp_ret)
+			*psp_ret = 0;
+
+		dev_err(psp->dev, "sev command %#x timed out, disabling PSP \n", cmd);
+		psp_dead = true;
+
+		return ret;
+	}
+
+	psp_timeout = psp_cmd_timeout;
 
 	if (psp_ret)
 		*psp_ret = reg & PSP_CMDRESP_ERR_MASK;
@@ -888,6 +922,8 @@ void psp_pci_init(void)
 
 	psp_master = sp->psp_data;
 
+	psp_timeout = psp_probe_timeout;
+
 	if (sev_get_api_version())
 		goto err;
 
-- 
2.7.4

Re: [PATCH crypto-2.6] crypto: ccp: add timeout support in the SEV command

[Herbert Xu]

On Wed, Aug 15, 2018 at 04:11:25PM -0500, Brijesh Singh wrote:
> Currently, the CCP driver assumes that the SEV command issued to the PSP
> will always return (i.e. it will never hang).  But recently, firmware bugs
> have shown that a command can hang.  Since of the SEV commands are used
> in probe routines, this can cause boot hangs and/or loss of virtualization
> capabilities.
> 
> To protect against firmware bugs, add a timeout in the SEV command
> execution flow.  If a command does not complete within the specified
> timeout then return -ETIMEOUT and stop the driver from executing any
> further commands since the state of the SEV firmware is unknown.
> 
> Cc: Tom Lendacky <thomas.lendacky@amd.com>
> Cc: Gary Hook <Gary.Hook@amd.com>
> Cc: Herbert Xu <herbert@gondor.apana.org.au>
> Cc: linux-kernel@vger.kernel.org
> Signed-off-by: Brijesh Singh <brijesh.singh@amd.com>
> ---
>  drivers/crypto/ccp/psp-dev.c | 46 +++++++++++++++++++++++++++++++++++++++-----
>  1 file changed, 41 insertions(+), 5 deletions(-)

Patch applied.  Thanks.
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH crypto-2.6] crypto: ccp: add timeout support in the SEV command

[Brijesh Singh]

On 9/4/18 12:19 AM, Herbert Xu wrote:
> On Wed, Aug 15, 2018 at 04:11:25PM -0500, Brijesh Singh wrote:
>> Currently, the CCP driver assumes that the SEV command issued to the PSP
>> will always return (i.e. it will never hang).  But recently, firmware bugs
>> have shown that a command can hang.  Since of the SEV commands are used
>> in probe routines, this can cause boot hangs and/or loss of virtualization
>> capabilities.
>>
>> To protect against firmware bugs, add a timeout in the SEV command
>> execution flow.  If a command does not complete within the specified
>> timeout then return -ETIMEOUT and stop the driver from executing any
>> further commands since the state of the SEV firmware is unknown.
>>
>> Cc: Tom Lendacky <thomas.lendacky@amd.com>
>> Cc: Gary Hook <Gary.Hook@amd.com>
>> Cc: Herbert Xu <herbert@gondor.apana.org.au>
>> Cc: linux-kernel@vger.kernel.org
>> Signed-off-by: Brijesh Singh <brijesh.singh@amd.com>
>> ---
>>  drivers/crypto/ccp/psp-dev.c | 46 +++++++++++++++++++++++++++++++++++++++-----
>>  1 file changed, 41 insertions(+), 5 deletions(-)
> Patch applied.  Thanks.
Thanks Herbert.

Can you please include this patch in your next 4.19-rcX pull request?
Multiple folks are encountering this firmware bug on Ryzen systems. I
will submit the modified version of this patch for stable release (s) so
that we fix issues in 4.18,.17 and .16. thank you.

-Brijesh

Re: [PATCH crypto-2.6] crypto: ccp: add timeout support in the SEV command

[Herbert Xu]

On Tue, Sep 11, 2018 at 07:11:05PM -0500, Brijesh Singh wrote:
>
> Can you please include this patch in your next 4.19-rcX pull request?
> Multiple folks are encountering this firmware bug on Ryzen systems. I
> will submit the modified version of this patch for stable release (s) so
> that we fix issues in 4.18,.17 and .16. thank you.

OK, I will add it to the crypto tree.

Cheers,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH crypto-2.6] crypto: ccp: add timeout support in the SEV command

[Borislav Petkov]

On Wed, Aug 15, 2018 at 04:11:25PM -0500, Brijesh Singh wrote:
> Currently, the CCP driver assumes that the SEV command issued to the PSP
> will always return (i.e. it will never hang).  But recently, firmware bugs
> have shown that a command can hang.  Since of the SEV commands are used
> in probe routines, this can cause boot hangs and/or loss of virtualization
> capabilities.
> 
> To protect against firmware bugs, add a timeout in the SEV command
> execution flow.  If a command does not complete within the specified
> timeout then return -ETIMEOUT and stop the driver from executing any
> further commands since the state of the SEV firmware is unknown.
> 
> Cc: Tom Lendacky <thomas.lendacky@amd.com>
> Cc: Gary Hook <Gary.Hook@amd.com>
> Cc: Herbert Xu <herbert@gondor.apana.org.au>
> Cc: linux-kernel@vger.kernel.org
> Signed-off-by: Brijesh Singh <brijesh.singh@amd.com>
> ---
>  drivers/crypto/ccp/psp-dev.c | 46 +++++++++++++++++++++++++++++++++++++++-----
>  1 file changed, 41 insertions(+), 5 deletions(-)
> 
> diff --git a/drivers/crypto/ccp/psp-dev.c b/drivers/crypto/ccp/psp-dev.c
> index 218739b..72790d8 100644
> --- a/drivers/crypto/ccp/psp-dev.c
> +++ b/drivers/crypto/ccp/psp-dev.c
> @@ -38,6 +38,17 @@ static DEFINE_MUTEX(sev_cmd_mutex);
>  static struct sev_misc_dev *misc_dev;
>  static struct psp_device *psp_master;
>  
> +static int psp_cmd_timeout = 100;
> +module_param(psp_cmd_timeout, int, 0644);
> +MODULE_PARM_DESC(psp_cmd_timeout, " default timeout value, in seconds, for PSP commands");
> +
> +static int psp_probe_timeout = 5;
> +module_param(psp_probe_timeout, int, 0644);
> +MODULE_PARM_DESC(psp_probe_timeout, " default timeout value, in seconds, during PSP device probe");

Just a question: what prevents the user from supplying non-sensical
values here?

I think we should clamp them to only allowed values because I don't want
to be debugging some strange bugs due to that.

> +
> +static bool psp_dead;
> +static int psp_timeout;
> +
>  static struct psp_device *psp_alloc_struct(struct sp_device *sp)
>  {
>  	struct device *dev = sp->dev;
> @@ -82,10 +93,19 @@ static irqreturn_t psp_irq_handler(int irq, void *data)
>  	return IRQ_HANDLED;
>  }
>  
> -static void sev_wait_cmd_ioc(struct psp_device *psp, unsigned int *reg)
> +static int sev_wait_cmd_ioc(struct psp_device *psp,
> +			    unsigned int *reg, unsigned int timeout)
>  {
> -	wait_event(psp->sev_int_queue, psp->sev_int_rcvd);
> +	int ret;
> +
> +	ret = wait_event_timeout(psp->sev_int_queue,
> +			psp->sev_int_rcvd, timeout * HZ);

Align args at opening brace.

> +	if (!ret)
> +		return -ETIMEDOUT;
> +
>  	*reg = ioread32(psp->io_regs + psp->vdata->cmdresp_reg);
> +
> +	return 0;
>  }
>  
>  static int sev_cmd_buffer_len(int cmd)
> @@ -133,12 +153,15 @@ static int __sev_do_cmd_locked(int cmd, void *data, int *psp_ret)
>  	if (!psp)
>  		return -ENODEV;
>  
> +	if (psp_dead)
> +		return -EBUSY;
> +
>  	/* Get the physical address of the command buffer */
>  	phys_lsb = data ? lower_32_bits(__psp_pa(data)) : 0;
>  	phys_msb = data ? upper_32_bits(__psp_pa(data)) : 0;
>  
> -	dev_dbg(psp->dev, "sev command id %#x buffer 0x%08x%08x\n",
> -		cmd, phys_msb, phys_lsb);
> +	dev_dbg(psp->dev, "sev command id %#x buffer 0x%08x%08x timeout %us\n",
> +		cmd, phys_msb, phys_lsb, psp_timeout);
>  
>  	print_hex_dump_debug("(in):  ", DUMP_PREFIX_OFFSET, 16, 2, data,
>  			     sev_cmd_buffer_len(cmd), false);
> @@ -154,7 +177,18 @@ static int __sev_do_cmd_locked(int cmd, void *data, int *psp_ret)
>  	iowrite32(reg, psp->io_regs + psp->vdata->cmdresp_reg);
>  
>  	/* wait for command completion */
> -	sev_wait_cmd_ioc(psp, &reg);
> +	ret = sev_wait_cmd_ioc(psp, &reg, psp_timeout);
> +	if (ret) {
> +		if (psp_ret)
> +			*psp_ret = 0;
> +
> +		dev_err(psp->dev, "sev command %#x timed out, disabling PSP \n", cmd);
									   ^

Trailing space.

-- 
Regards/Gruss,
    Boris.

Good mailing practices for 400: avoid top-posting and trim the reply.

Re: [PATCH crypto-2.6] crypto: ccp: add timeout support in the SEV command

[Brijesh Singh]

Hi Boris,


On 09/04/2018 03:11 AM, Borislav Petkov wrote:
...

>> +
>> +static int psp_probe_timeout = 5;
>> +module_param(psp_probe_timeout, int, 0644);
>> +MODULE_PARM_DESC(psp_probe_timeout, " default timeout value, in seconds, during PSP device probe");
> 
> Just a question: what prevents the user from supplying non-sensical
> values here?
> 
> I think we should clamp them to only allowed values because I don't want
> to be debugging some strange bugs due to that.
> 

Nothing prevent user from supplying a bogus number. The main question
is, clamp with what number ?

IMO, if user is overriding the default timeout number then its possible
that user is dealing with a buggy firmware which does not work with
default timeout and silently clamping the value will not help them.


- Brijesh

Re: [PATCH crypto-2.6] crypto: ccp: add timeout support in the SEV command

[Borislav Petkov]

On Mon, Sep 10, 2018 at 02:06:57PM -0500, Brijesh Singh wrote:
> Nothing prevent user from supplying a bogus number. The main question
> is, clamp with what number ?

So you definitely want to forbid too large timeouts - that wouldn't make
any sense anyway. And too small either, because a too small timeout
would make a potentially functioning fw broken.

> IMO, if user is overriding the default timeout number then its possible
> that user is dealing with a buggy firmware which does not work with
> default timeout and silently clamping the value will not help them.

No one said "silently" - you simply say:

	"Correcting PSP	"Correcting PSP probe timeout to X seconds."

when loading the driver so that the user is aware that the value she
entered might not be an optimal one.

-- 
Regards/Gruss,
    Boris.

Good mailing practices for 400: avoid top-posting and trim the reply.