vio.c:__vio_register_driver && LPAR Migration issue

vio.c:__vio_register_driver && LPAR Migration issue

[Michael Bringmann]

Hello:
    I have been observing an anomaly during LPAR migrations between
a couple of P8 systems.

This is the problem.  After migrating an LPAR, the PPC mobility code
receives RTAS requests to delete nodes with platform-/hardware-specific
attributes when restarting the kernel after a migration.  My example is
for migration between a P8 Alpine and a P8 Brazos.  Among the nodes
that I see being deleted are 'ibm,random-v1', 'ibm,compression-v1',
'ibm,platform-facilities', and 'ibm,sym-encryption-v1'.  Of these
nodes, the following are created during initial boot by calls to
vio_register_driver:

drivers/char/hw_random/pseries-rng.c
        ibm,random-v1

drivers/crypto/nx/nx-842-pseries.c
        ibm,compression-v1

drivers/crypto/nx/nx.c
        ibm,sym-encryption-v1

After the migration, these nodes are deleted, but nothing recreates
them.  If I boot the LPAR on the target system, the nodes are added
again.

My question is how do we recreate these nodes after migration?

Thanks.

-- 
Michael W. Bringmann
Linux Technology Center
IBM Corporation
Tie-Line  363-5196
External: (512) 286-5196
Cell:       (512) 466-0650
mwb@linux.vnet.ibm.com

Re: vio.c:__vio_register_driver && LPAR Migration issue

[Tyrel Datwyler]

On 08/02/2018 11:15 AM, Michael Bringmann wrote:
> Hello:
>     I have been observing an anomaly during LPAR migrations between
> a couple of P8 systems.
> 
> This is the problem.  After migrating an LPAR, the PPC mobility code
> receives RTAS requests to delete nodes with platform-/hardware-specific
> attributes when restarting the kernel after a migration.  My example is
> for migration between a P8 Alpine and a P8 Brazos.  Among the nodes
> that I see being deleted are 'ibm,random-v1', 'ibm,compression-v1',
> 'ibm,platform-facilities', and 'ibm,sym-encryption-v1'.  Of these
> nodes, the following are created during initial boot by calls to
> vio_register_driver:
> 
> drivers/char/hw_random/pseries-rng.c
>         ibm,random-v1
> 
> drivers/crypto/nx/nx-842-pseries.c
>         ibm,compression-v1
> 
> drivers/crypto/nx/nx.c
>         ibm,sym-encryption-v1
> 
> After the migration, these nodes are deleted, but nothing recreates
> them.  If I boot the LPAR on the target system, the nodes are added
> again.
> 
> My question is how do we recreate these nodes after migration?

Hmm, I'd have to see the scenario in action, but these should be added back by ibm,update-nodes RTAS call. There is some debug code in driver/of/dynamic.c that can be enabled that will log node/property dynamic reconfiguration events.

-Tyrel

> 
> Thanks.
> 

Re: vio.c:__vio_register_driver && LPAR Migration issue

[Tyrel Datwyler]

On 08/03/2018 02:23 PM, Tyrel Datwyler wrote:
> On 08/02/2018 11:15 AM, Michael Bringmann wrote:
>> Hello:
>>     I have been observing an anomaly during LPAR migrations between
>> a couple of P8 systems.
>>
>> This is the problem.  After migrating an LPAR, the PPC mobility code
>> receives RTAS requests to delete nodes with platform-/hardware-specific
>> attributes when restarting the kernel after a migration.  My example is
>> for migration between a P8 Alpine and a P8 Brazos.  Among the nodes
>> that I see being deleted are 'ibm,random-v1', 'ibm,compression-v1',
>> 'ibm,platform-facilities', and 'ibm,sym-encryption-v1'.  Of these
>> nodes, the following are created during initial boot by calls to
>> vio_register_driver:
>>
>> drivers/char/hw_random/pseries-rng.c
>>         ibm,random-v1
>>
>> drivers/crypto/nx/nx-842-pseries.c
>>         ibm,compression-v1
>>
>> drivers/crypto/nx/nx.c
>>         ibm,sym-encryption-v1
>>
>> After the migration, these nodes are deleted, but nothing recreates
>> them.  If I boot the LPAR on the target system, the nodes are added
>> again.
>>
>> My question is how do we recreate these nodes after migration?
> 
> Hmm, I'd have to see the scenario in action, but these should be added back by ibm,update-nodes RTAS call. There is some debug code in driver/of/dynamic.c that can be enabled that will log node/property dynamic reconfiguration events.
> 

I took a quick look by turning of phandle caching and turning on the OF reconfig debug on one of my lpars. I noticed that it looks like this could be a firmware issue. Looks to me like we aren't being notified to re-add those child nodes of ibm,platform-facilities.

[ 1671.638041] OF: notify DETACH_NODE     /cpus/l2-cache@2022
[ 1671.638094] OF: notify DETACH_NODE     /cpus/l3-cache@3122
[ 1671.638119] OF: notify DETACH_NODE     /cpus/l2-cache@2023
[ 1671.638136] OF: notify DETACH_NODE     /cpus/l3-cache@3123
[ 1671.638164] OF: notify DETACH_NODE     /ibm,platform-facilities/ibm,random-v1
[ 1671.638182] OF: notify DETACH_NODE     /ibm,platform-facilities/ibm,compression-v1
[ 1671.638198] OF: notify DETACH_NODE     /ibm,platform-facilities/ibm,sym-encryption-v1
[ 1671.638219] OF: notify DETACH_NODE     /ibm,platform-facilities
[ 1671.776419] OF: notify UPDATE_PROPERTY /:ibm,model-class

... snipped property updates

[ 1672.129941] OF: notify UPDATE_PROPERTY /cpus/PowerPC,POWER8@8:ibm,dfp
[ 1672.147551] OF: notify ATTACH_NODE     /cpus/l2-cache@202e
[ 1672.166321] OF: notify ATTACH_NODE     /cpus/l3-cache@312e
[ 1672.183971] OF: notify ATTACH_NODE     /cpus/l2-cache@202f
[ 1672.202752] OF: notify ATTACH_NODE     /cpus/l3-cache@312f
[ 1672.230760] OF: notify ATTACH_NODE     /ibm,platform-facilities

Need to verify this by tracing the RTAS calls to ibm,update-nodes. I'll try and look at that tomorrow. The loop that processes the nodes to update in pseries_devicetree_update() blindly ignores the return codes from delete_dt_node(), update_dt_node(), and add_dt_node(). So, it is also possible that we are notified, but are silently failing the add.

-Tyrel

Re: vio.c:__vio_register_driver && LPAR Migration issue

[Michael Bringmann]

On 08/04/2018 06:01 PM, Tyrel Datwyler wrote:
> On 08/03/2018 02:23 PM, Tyrel Datwyler wrote:
>> On 08/02/2018 11:15 AM, Michael Bringmann wrote:
>>> Hello:
>>>     I have been observing an anomaly during LPAR migrations between
>>> a couple of P8 systems.
>>>
>>> This is the problem.  After migrating an LPAR, the PPC mobility code
>>> receives RTAS requests to delete nodes with platform-/hardware-specific
>>> attributes when restarting the kernel after a migration.  My example is
>>> for migration between a P8 Alpine and a P8 Brazos.  Among the nodes
>>> that I see being deleted are 'ibm,random-v1', 'ibm,compression-v1',
>>> 'ibm,platform-facilities', and 'ibm,sym-encryption-v1'.  Of these
>>> nodes, the following are created during initial boot by calls to
>>> vio_register_driver:
>>>
>>> drivers/char/hw_random/pseries-rng.c
>>>         ibm,random-v1
>>>
>>> drivers/crypto/nx/nx-842-pseries.c
>>>         ibm,compression-v1
>>>
>>> drivers/crypto/nx/nx.c
>>>         ibm,sym-encryption-v1
>>>
>>> After the migration, these nodes are deleted, but nothing recreates
>>> them.  If I boot the LPAR on the target system, the nodes are added
>>> again.
>>>
>>> My question is how do we recreate these nodes after migration?
>>
>> Hmm, I'd have to see the scenario in action, but these should be added back by ibm,update-nodes RTAS call. There is some debug code in driver/of/dynamic.c that can be enabled that will log node/property dynamic reconfiguration events.
>>
> 
> I took a quick look by turning of phandle caching and turning on the OF reconfig debug on one of my lpars. I noticed that it looks like this could be a firmware issue. Looks to me like we aren't being notified to re-add those child nodes of ibm,platform-facilities.
> 
> [ 1671.638041] OF: notify DETACH_NODE     /cpus/l2-cache@2022
> [ 1671.638094] OF: notify DETACH_NODE     /cpus/l3-cache@3122
> [ 1671.638119] OF: notify DETACH_NODE     /cpus/l2-cache@2023
> [ 1671.638136] OF: notify DETACH_NODE     /cpus/l3-cache@3123
> [ 1671.638164] OF: notify DETACH_NODE     /ibm,platform-facilities/ibm,random-v1
> [ 1671.638182] OF: notify DETACH_NODE     /ibm,platform-facilities/ibm,compression-v1
> [ 1671.638198] OF: notify DETACH_NODE     /ibm,platform-facilities/ibm,sym-encryption-v1
> [ 1671.638219] OF: notify DETACH_NODE     /ibm,platform-facilities
> [ 1671.776419] OF: notify UPDATE_PROPERTY /:ibm,model-class
> 
> ... snipped property updates
> 
> [ 1672.129941] OF: notify UPDATE_PROPERTY /cpus/PowerPC,POWER8@8:ibm,dfp
> [ 1672.147551] OF: notify ATTACH_NODE     /cpus/l2-cache@202e
> [ 1672.166321] OF: notify ATTACH_NODE     /cpus/l3-cache@312e
> [ 1672.183971] OF: notify ATTACH_NODE     /cpus/l2-cache@202f
> [ 1672.202752] OF: notify ATTACH_NODE     /cpus/l3-cache@312f
> [ 1672.230760] OF: notify ATTACH_NODE     /ibm,platform-facilities
> 
> Need to verify this by tracing the RTAS calls to ibm,update-nodes. I'll try and look at that tomorrow. The loop that processes the nodes to update in pseries_devicetree_update() blindly ignores the return codes from delete_dt_node(), update_dt_node(), and add_dt_node(). So, it is also possible that we are notified, but are silently failing the add.

I am seeing the same set of detach/attach events for migration between an Alpine and Brazos.
I did the logging up in delete_dt_node, update_dt_node, and add_dt_node, and did not see any
notification of additional nodes.

> 
> -Tyrel
> 

Michael

-- 
Michael W. Bringmann
Linux Technology Center
IBM Corporation
Tie-Line  363-5196
External: (512) 286-5196
Cell:       (512) 466-0650
mwb@linux.vnet.ibm.com