[PATCH v2 1/1] remoteproc: core: probe subdevices before booting coprocessor

[PATCH v2 1/1] remoteproc: core: probe subdevices before booting coprocessor

[Loic Pallardy]

With subdevice support introduction, coprocessor boot sequence has
changed. Related coprocessor subdevices are now starting after firmware
and resource table loading and coprocessor boot.

But some subdevices can resources to allocate before coprocessor start,
like rpmsg buffers allocation for example.

This patch probes subdevices just before loading resource table,
to keep backward compatibility with existing firmwares.

Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
---
v2: remove unused stop_rproc label

 drivers/remoteproc/remoteproc_core.c | 18 ++++++++----------
 1 file changed, 8 insertions(+), 10 deletions(-)

diff --git a/drivers/remoteproc/remoteproc_core.c b/drivers/remoteproc/remoteproc_core.c
index f0f6ec1..61f06a7 100644
--- a/drivers/remoteproc/remoteproc_core.c
+++ b/drivers/remoteproc/remoteproc_core.c
@@ -913,6 +913,14 @@ static int rproc_fw_boot(struct rproc *rproc, const struct firmware *fw)
 		goto clean_up_resources;
 	}
 
+	/* probe any subdevices for the remote processor */
+	ret = rproc_probe_subdevices(rproc);
+	if (ret) {
+		dev_err(dev, "failed to probe subdevices for %s: %d\n",
+			rproc->name, ret);
+		goto clean_up_resources;
+	}
+
 	/*
 	 * The starting device has been given the rproc->table_ptr as the
 	 * resource table. The address of the vring along with the other
@@ -932,22 +940,12 @@ static int rproc_fw_boot(struct rproc *rproc, const struct firmware *fw)
 		goto clean_up_resources;
 	}
 
-	/* probe any subdevices for the remote processor */
-	ret = rproc_probe_subdevices(rproc);
-	if (ret) {
-		dev_err(dev, "failed to probe subdevices for %s: %d\n",
-			rproc->name, ret);
-		goto stop_rproc;
-	}
-
 	rproc->state = RPROC_RUNNING;
 
 	dev_info(dev, "remote processor %s is now up\n", rproc->name);
 
 	return 0;
 
-stop_rproc:
-	rproc->ops->stop(rproc);
 clean_up_resources:
 	rproc_resource_cleanup(rproc);
 clean_up:
-- 
1.9.1

Re: [PATCH v2 1/1] remoteproc: core: probe subdevices before booting coprocessor

[Bjorn Andersson]

On Wed 07 Dec 12:49 PST 2016, Loic Pallardy wrote:

> With subdevice support introduction, coprocessor boot sequence has
> changed. Related coprocessor subdevices are now starting after firmware
> and resource table loading and coprocessor boot.
> 

...just as they always have been.

> But some subdevices can resources to allocate before coprocessor start,
> like rpmsg buffers allocation for example.
> 

"for example"?

I can think of numerous examples where rpmsg devices would need this
mechanism as well, but there's no guarantees that just because we probe
subdevices before we boot that these devices has had their chance to
probe and register their resources.

How would this work if some of the virtio drivers used by the firmware
are compiled as modules and not yet loaded or anything hitting an
EPROBE_DEFER?

> This patch probes subdevices just before loading resource table,
> to keep backward compatibility with existing firmwares.
> 

Backward compatibility with firmware written specifically for the
single, yet unreleased, version of the Linux kernel with this behavior?



I still believe that resources that are so strictly tied to the state of
the firmware should be declared as part of the remoteproc, not
registered by other devices that might or might not finish their
registration in time - unless those resources can be associated
to the remote dynamically.

Regards,
Bjorn

> Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
> ---
> v2: remove unused stop_rproc label
> 
>  drivers/remoteproc/remoteproc_core.c | 18 ++++++++----------
>  1 file changed, 8 insertions(+), 10 deletions(-)
> 
> diff --git a/drivers/remoteproc/remoteproc_core.c b/drivers/remoteproc/remoteproc_core.c
> index f0f6ec1..61f06a7 100644
> --- a/drivers/remoteproc/remoteproc_core.c
> +++ b/drivers/remoteproc/remoteproc_core.c
> @@ -913,6 +913,14 @@ static int rproc_fw_boot(struct rproc *rproc, const struct firmware *fw)
>  		goto clean_up_resources;
>  	}
>  
> +	/* probe any subdevices for the remote processor */
> +	ret = rproc_probe_subdevices(rproc);
> +	if (ret) {
> +		dev_err(dev, "failed to probe subdevices for %s: %d\n",
> +			rproc->name, ret);
> +		goto clean_up_resources;
> +	}
> +
>  	/*
>  	 * The starting device has been given the rproc->table_ptr as the
>  	 * resource table. The address of the vring along with the other
> @@ -932,22 +940,12 @@ static int rproc_fw_boot(struct rproc *rproc, const struct firmware *fw)
>  		goto clean_up_resources;
>  	}
>  
> -	/* probe any subdevices for the remote processor */
> -	ret = rproc_probe_subdevices(rproc);
> -	if (ret) {
> -		dev_err(dev, "failed to probe subdevices for %s: %d\n",
> -			rproc->name, ret);
> -		goto stop_rproc;
> -	}
> -
>  	rproc->state = RPROC_RUNNING;
>  
>  	dev_info(dev, "remote processor %s is now up\n", rproc->name);
>  
>  	return 0;
>  
> -stop_rproc:
> -	rproc->ops->stop(rproc);
>  clean_up_resources:
>  	rproc_resource_cleanup(rproc);
>  clean_up:
> -- 
> 1.9.1
> 

Re: [PATCH v2 1/1] remoteproc: core: probe subdevices before booting coprocessor

[loic pallardy]

On 12/08/2016 12:23 AM, Bjorn Andersson wrote:
> On Wed 07 Dec 12:49 PST 2016, Loic Pallardy wrote:
>
>> With subdevice support introduction, coprocessor boot sequence has
>> changed. Related coprocessor subdevices are now starting after firmware
>> and resource table loading and coprocessor boot.
>>
>
> ...just as they always have been.
>
>> But some subdevices can resources to allocate before coprocessor start,
>> like rpmsg buffers allocation for example.
>>
>
> "for example"?
>
> I can think of numerous examples where rpmsg devices would need this
> mechanism as well, but there's no guarantees that just because we probe
> subdevices before we boot that these devices has had their chance to
> probe and register their resources.
>
> How would this work if some of the virtio drivers used by the firmware
> are compiled as modules and not yet loaded or anything hitting an
> EPROBE_DEFER?
OK You're true, need to be compliant with module mode too.
>
>> This patch probes subdevices just before loading resource table,
>> to keep backward compatibility with existing firmwares.
>>
>
> Backward compatibility with firmware written specifically for the
> single, yet unreleased, version of the Linux kernel with this behavior?
No backward compatibility in view. We are working to have firmwares 
released for 96board B2260 asap. Just need to cop with SoC HW and ST 
RTOS constraints.
>
>
>
> I still believe that resources that are so strictly tied to the state of
> the firmware should be declared as part of the remoteproc, not
> registered by other devices that might or might not finish their
> registration in time - unless those resources can be associated
> to the remote dynamically.
Yes please drop out this patch. I'll work on firmware side to be more 
flexible for initialization sequence.

Regards,
Loic
>
> Regards,
> Bjorn
>
>> Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
>> ---
>> v2: remove unused stop_rproc label
>>
>>  drivers/remoteproc/remoteproc_core.c | 18 ++++++++----------
>>  1 file changed, 8 insertions(+), 10 deletions(-)
>>
>> diff --git a/drivers/remoteproc/remoteproc_core.c b/drivers/remoteproc/remoteproc_core.c
>> index f0f6ec1..61f06a7 100644
>> --- a/drivers/remoteproc/remoteproc_core.c
>> +++ b/drivers/remoteproc/remoteproc_core.c
>> @@ -913,6 +913,14 @@ static int rproc_fw_boot(struct rproc *rproc, const struct firmware *fw)
>>  		goto clean_up_resources;
>>  	}
>>
>> +	/* probe any subdevices for the remote processor */
>> +	ret = rproc_probe_subdevices(rproc);
>> +	if (ret) {
>> +		dev_err(dev, "failed to probe subdevices for %s: %d\n",
>> +			rproc->name, ret);
>> +		goto clean_up_resources;
>> +	}
>> +
>>  	/*
>>  	 * The starting device has been given the rproc->table_ptr as the
>>  	 * resource table. The address of the vring along with the other
>> @@ -932,22 +940,12 @@ static int rproc_fw_boot(struct rproc *rproc, const struct firmware *fw)
>>  		goto clean_up_resources;
>>  	}
>>
>> -	/* probe any subdevices for the remote processor */
>> -	ret = rproc_probe_subdevices(rproc);
>> -	if (ret) {
>> -		dev_err(dev, "failed to probe subdevices for %s: %d\n",
>> -			rproc->name, ret);
>> -		goto stop_rproc;
>> -	}
>> -
>>  	rproc->state = RPROC_RUNNING;
>>
>>  	dev_info(dev, "remote processor %s is now up\n", rproc->name);
>>
>>  	return 0;
>>
>> -stop_rproc:
>> -	rproc->ops->stop(rproc);
>>  clean_up_resources:
>>  	rproc_resource_cleanup(rproc);
>>  clean_up:
>> --
>> 1.9.1
>>

Re: [PATCH v2 1/1] remoteproc: core: probe subdevices before booting coprocessor

[loic pallardy]

On 12/11/2016 09:30 PM, loic pallardy wrote:
>
>
> On 12/08/2016 12:23 AM, Bjorn Andersson wrote:
>> On Wed 07 Dec 12:49 PST 2016, Loic Pallardy wrote:
>>
>>> With subdevice support introduction, coprocessor boot sequence has
>>> changed. Related coprocessor subdevices are now starting after firmware
>>> and resource table loading and coprocessor boot.
>>>
[..]
>>
>>
>> I still believe that resources that are so strictly tied to the state of
>> the firmware should be declared as part of the remoteproc, not
>> registered by other devices that might or might not finish their
>> registration in time - unless those resources can be associated
>> to the remote dynamically.
> Yes please drop out this patch. I'll work on firmware side to be more
> flexible for initialization sequence.
>
Hi Bjorn,

Reworking coprocessor boot sequence to guarantee struct fw_rsc_vdev of 
resource table is accessed only after first kick generated by 
virtio_rpmsg, I found the issue in coming from a patch present in 
remoteproc pull request for v4.10 [1]: remoteproc: Merge table_ptr and 
cached_table pointers [2]

This patch disables the capability of rproc subdev to modifiy 
dynamically the resource table after coprocessor boot. rproc_virtio_set 
function has no more effect after rproc_boot.

Patch above was not fixing the right issue but just masking it...
Reverting patch [2] fix it.

As subdevices are booted after rproc (and I agree with your arguments 
mentioned previously), dynamic resource table modifications should be 
possible (of course HW sync between host and rproc is required for rsc 
access).

Moreover reading patch [2] commit header, I think that security memory 
lock down should be an option and not the default. If memory region has 
to be locked down, I agree that all resource table content will be fixed 
and no dynamic modification possible. But that's not valid all today.
May be a DT properties will help to manage both configurations? And 
adding a warning in rproc_virtio_set function when rsc table in more 
accessible will be nice too.

Regards,
Loic

[1] https://lkml.org/lkml/2016/12/12/661 : remoteproc: Merge table_ptr 
and cached_table pointers
[2] http://www.spinics.net/lists/linux-remoteproc/msg00717.html


> Regards,
> Loic
>>
>> Regards,
>> Bjorn
>>
>>> Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
>>> ---
>>> v2: remove unused stop_rproc label
>>>
>>>  drivers/remoteproc/remoteproc_core.c | 18 ++++++++----------
>>>  1 file changed, 8 insertions(+), 10 deletions(-)
>>>
>>> diff --git a/drivers/remoteproc/remoteproc_core.c
>>> b/drivers/remoteproc/remoteproc_core.c
>>> index f0f6ec1..61f06a7 100644
>>> --- a/drivers/remoteproc/remoteproc_core.c
>>> +++ b/drivers/remoteproc/remoteproc_core.c
>>> @@ -913,6 +913,14 @@ static int rproc_fw_boot(struct rproc *rproc,
>>> const struct firmware *fw)
>>>          goto clean_up_resources;
>>>      }
>>>
>>> +    /* probe any subdevices for the remote processor */
>>> +    ret = rproc_probe_subdevices(rproc);
>>> +    if (ret) {
>>> +        dev_err(dev, "failed to probe subdevices for %s: %d\n",
>>> +            rproc->name, ret);
>>> +        goto clean_up_resources;
>>> +    }
>>> +
>>>      /*
>>>       * The starting device has been given the rproc->table_ptr as the
>>>       * resource table. The address of the vring along with the other
>>> @@ -932,22 +940,12 @@ static int rproc_fw_boot(struct rproc *rproc,
>>> const struct firmware *fw)
>>>          goto clean_up_resources;
>>>      }
>>>
>>> -    /* probe any subdevices for the remote processor */
>>> -    ret = rproc_probe_subdevices(rproc);
>>> -    if (ret) {
>>> -        dev_err(dev, "failed to probe subdevices for %s: %d\n",
>>> -            rproc->name, ret);
>>> -        goto stop_rproc;
>>> -    }
>>> -
>>>      rproc->state = RPROC_RUNNING;
>>>
>>>      dev_info(dev, "remote processor %s is now up\n", rproc->name);
>>>
>>>      return 0;
>>>
>>> -stop_rproc:
>>> -    rproc->ops->stop(rproc);
>>>  clean_up_resources:
>>>      rproc_resource_cleanup(rproc);
>>>  clean_up:
>>> --
>>> 1.9.1
>>>

Re: [PATCH v2 1/1] remoteproc: core: probe subdevices before booting coprocessor

[Bjorn Andersson]

On Tue 13 Dec 09:22 PST 2016, loic pallardy wrote:

> 
> 
> On 12/11/2016 09:30 PM, loic pallardy wrote:
> >
> >
> >On 12/08/2016 12:23 AM, Bjorn Andersson wrote:
> >>On Wed 07 Dec 12:49 PST 2016, Loic Pallardy wrote:
> >>
> >>>With subdevice support introduction, coprocessor boot sequence has
> >>>changed. Related coprocessor subdevices are now starting after firmware
> >>>and resource table loading and coprocessor boot.
> >>>
> [..]
> >>
> >>
> >>I still believe that resources that are so strictly tied to the state of
> >>the firmware should be declared as part of the remoteproc, not
> >>registered by other devices that might or might not finish their
> >>registration in time - unless those resources can be associated
> >>to the remote dynamically.
> >Yes please drop out this patch. I'll work on firmware side to be more
> >flexible for initialization sequence.
> >
> Hi Bjorn,
> 
> Reworking coprocessor boot sequence to guarantee struct fw_rsc_vdev of
> resource table is accessed only after first kick generated by virtio_rpmsg,
> I found the issue in coming from a patch present in remoteproc pull request
> for v4.10 [1]: remoteproc: Merge table_ptr and cached_table pointers [2]
> 
> This patch disables the capability of rproc subdev to modifiy dynamically
> the resource table after coprocessor boot. rproc_virtio_set function has no
> more effect after rproc_boot.
> 

You're right, the resource table can't be read-only.

> Patch above was not fixing the right issue but just masking it...
> Reverting patch [2] fix it.
> 

Thanks for testing and reporting this, I will prepare a fix/revert of
it.

> As subdevices are booted after rproc (and I agree with your arguments
> mentioned previously), dynamic resource table modifications should be
> possible (of course HW sync between host and rproc is required for rsc
> access).
>
> Moreover reading patch [2] commit header, I think that security memory lock
> down should be an option and not the default. If memory region has to be
> locked down, I agree that all resource table content will be fixed and no
> dynamic modification possible. But that's not valid all today.

The commit message captures your requirement from our discussions at
Linaro Connect, I'm sorry for missing the fact that it doesn't work.

Regards,
Bjorn

Re: [PATCH v2 1/1] remoteproc: core: probe subdevices before booting coprocessor

[Patrice Chotard]

On 12/07/2016 09:49 PM, Loic Pallardy wrote:
> With subdevice support introduction, coprocessor boot sequence has
> changed. Related coprocessor subdevices are now starting after firmware
> and resource table loading and coprocessor boot.
> 
> But some subdevices can resources to allocate before coprocessor start,
> like rpmsg buffers allocation for example.
> 
> This patch probes subdevices just before loading resource table,
> to keep backward compatibility with existing firmwares.
> 
> Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
> ---
> v2: remove unused stop_rproc label
> 
>  drivers/remoteproc/remoteproc_core.c | 18 ++++++++----------
>  1 file changed, 8 insertions(+), 10 deletions(-)
> 
> diff --git a/drivers/remoteproc/remoteproc_core.c b/drivers/remoteproc/remoteproc_core.c
> index f0f6ec1..61f06a7 100644
> --- a/drivers/remoteproc/remoteproc_core.c
> +++ b/drivers/remoteproc/remoteproc_core.c
> @@ -913,6 +913,14 @@ static int rproc_fw_boot(struct rproc *rproc, const struct firmware *fw)
>  		goto clean_up_resources;
>  	}
>  
> +	/* probe any subdevices for the remote processor */
> +	ret = rproc_probe_subdevices(rproc);
> +	if (ret) {
> +		dev_err(dev, "failed to probe subdevices for %s: %d\n",
> +			rproc->name, ret);
> +		goto clean_up_resources;
> +	}
> +
>  	/*
>  	 * The starting device has been given the rproc->table_ptr as the
>  	 * resource table. The address of the vring along with the other
> @@ -932,22 +940,12 @@ static int rproc_fw_boot(struct rproc *rproc, const struct firmware *fw)
>  		goto clean_up_resources;
>  	}
>  
> -	/* probe any subdevices for the remote processor */
> -	ret = rproc_probe_subdevices(rproc);
> -	if (ret) {
> -		dev_err(dev, "failed to probe subdevices for %s: %d\n",
> -			rproc->name, ret);
> -		goto stop_rproc;
> -	}
> -
>  	rproc->state = RPROC_RUNNING;
>  
>  	dev_info(dev, "remote processor %s is now up\n", rproc->name);
>  
>  	return 0;
>  
> -stop_rproc:
> -	rproc->ops->stop(rproc);
>  clean_up_resources:
>  	rproc_resource_cleanup(rproc);
>  clean_up:
> 
Hi Loic

Acked-by: Patrice Chotard <patrice.chotard@st.com>

Patrice

Re: [PATCH v2 1/1] remoteproc: core: probe subdevices before booting coprocessor

[Patrice Chotard]

On 01/04/2017 08:40 AM, Patrice Chotard wrote:
> On 12/07/2016 09:49 PM, Loic Pallardy wrote:
>> With subdevice support introduction, coprocessor boot sequence has
>> changed. Related coprocessor subdevices are now starting after firmware
>> and resource table loading and coprocessor boot.
>>
>> But some subdevices can resources to allocate before coprocessor start,
>> like rpmsg buffers allocation for example.
>>
>> This patch probes subdevices just before loading resource table,
>> to keep backward compatibility with existing firmwares.
>>
>> Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
>> ---
>> v2: remove unused stop_rproc label
>>
>>  drivers/remoteproc/remoteproc_core.c | 18 ++++++++----------
>>  1 file changed, 8 insertions(+), 10 deletions(-)
>>
>> diff --git a/drivers/remoteproc/remoteproc_core.c b/drivers/remoteproc/remoteproc_core.c
>> index f0f6ec1..61f06a7 100644
>> --- a/drivers/remoteproc/remoteproc_core.c
>> +++ b/drivers/remoteproc/remoteproc_core.c
>> @@ -913,6 +913,14 @@ static int rproc_fw_boot(struct rproc *rproc, const struct firmware *fw)
>>  		goto clean_up_resources;
>>  	}
>>  
>> +	/* probe any subdevices for the remote processor */
>> +	ret = rproc_probe_subdevices(rproc);
>> +	if (ret) {
>> +		dev_err(dev, "failed to probe subdevices for %s: %d\n",
>> +			rproc->name, ret);
>> +		goto clean_up_resources;
>> +	}
>> +
>>  	/*
>>  	 * The starting device has been given the rproc->table_ptr as the
>>  	 * resource table. The address of the vring along with the other
>> @@ -932,22 +940,12 @@ static int rproc_fw_boot(struct rproc *rproc, const struct firmware *fw)
>>  		goto clean_up_resources;
>>  	}
>>  
>> -	/* probe any subdevices for the remote processor */
>> -	ret = rproc_probe_subdevices(rproc);
>> -	if (ret) {
>> -		dev_err(dev, "failed to probe subdevices for %s: %d\n",
>> -			rproc->name, ret);
>> -		goto stop_rproc;
>> -	}
>> -
>>  	rproc->state = RPROC_RUNNING;
>>  
>>  	dev_info(dev, "remote processor %s is now up\n", rproc->name);
>>  
>>  	return 0;
>>  
>> -stop_rproc:
>> -	rproc->ops->stop(rproc);
>>  clean_up_resources:
>>  	rproc_resource_cleanup(rproc);
>>  clean_up:
>>
> Hi Loic
> 
> Acked-by: Patrice Chotard <patrice.chotard@st.com>
> 
> Patrice
> 

Sorry i selected the wrong patch, remove my Acked-by from this patch.

Patrice