RapidIO - general questions

RapidIO - general questions

[Jan Neskudla]

Hallo 

we'd likes to use a RapidIO as a general communication bus on our new
product, and so I have some questions about general design of Linux RIO
subsystem. I did not find any better mailing list for RapidIO
discussion. 

[1] - we'd like to implement following features 
    * Hot-plug (hot-insert/hot-remove) of devices
    * Error handling (port-write packets - configuration, handling of
them)
    * Static ID configuration based on port numbers 
    * Aux driver - basic driver, for sending messages over different
mboxes, handling ranges of doorbells	

    Is it here anyone who is working on any improvement, or anyone who
knows the development plans for RapidIO subsystem? 

[2] - I have a following problem with a current implementation of
loading drivers. The driver probe-function call is based on comparison
of VendorID (VID) and DeviceID (DID) only. Thus if I have 3 devices with
same DID and VID connected to the same network (bus), the driver is
loaded 3times, instead only once for the actual device Master port. 

Rionet driver solved this by enabling to call initialization function
just once, and it expect that this is the Master port. 

Is it this correct behavior  ? It looks to me that RapidIO is handled
like a local bus (like PCI) 
 
                                   Jan 

Re: RapidIO - general questions

[Li Yang]

cc'ed LKML

On Tue, May 12, 2009 at 5:17 PM, Jan Neskudla <jan.neskudla.ext@nsn.com> wr=
ote:
> Hallo
>
> we'd likes to use a RapidIO as a general communication bus on our new
> product, and so I have some questions about general design of Linux RIO
> subsystem. I did not find any better mailing list for RapidIO
> discussion.
>
> [1] - we'd like to implement following features
> =C2=A0 =C2=A0* Hot-plug (hot-insert/hot-remove) of devices
> =C2=A0 =C2=A0* Error handling (port-write packets - configuration, handli=
ng of
> them)
> =C2=A0 =C2=A0* Static ID configuration based on port numbers
> =C2=A0 =C2=A0* Aux driver - basic driver, for sending messages over diffe=
rent
> mboxes, handling ranges of doorbells
>
> =C2=A0 =C2=A0Is it here anyone who is working on any improvement, or anyo=
ne who
> knows the development plans for RapidIO subsystem?
>

AFAIK, there is no one currently working on these features for Linux.
It will be good if you can add these useful features.

> [2] - I have a following problem with a current implementation of
> loading drivers. The driver probe-function call is based on comparison
> of VendorID (VID) and DeviceID (DID) only. Thus if I have 3 devices with
> same DID and VID connected to the same network (bus), the driver is
> loaded 3times, instead only once for the actual device Master port.

This should be the correct way as you actually have 3 instances of the devi=
ce.

>
> Rionet driver solved this by enabling to call initialization function
> just once, and it expect that this is the Master port.

Rionet is kind of special.  It's not working like a simple device
driver, but more like a customized protocol stack to support multiple
ethernet over rio links.

>
> Is it this correct behavior =C2=A0? It looks to me that RapidIO is handle=
d
> like a local bus (like PCI)

This is correct behavior.  All of them are using Linux device/driver
infrastructure, but rionet is a special device.

- Leo

Re: RapidIO - general questions

[Jan Neskudla]

On Wed, 2009-05-13 at 18:57 +0800, ext Li Yang wrote:
> cc'ed LKML
> 
> On Tue, May 12, 2009 at 5:17 PM, Jan Neskudla <jan.neskudla.ext@nsn.com> wrote:
> > Hallo
> >
> > we'd likes to use a RapidIO as a general communication bus on our new
> > product, and so I have some questions about general design of Linux RIO
> > subsystem. I did not find any better mailing list for RapidIO
> > discussion.
> >
> > [1] - we'd like to implement following features
> >    * Hot-plug (hot-insert/hot-remove) of devices
> >    * Error handling (port-write packets - configuration, handling of
> > them)
> >    * Static ID configuration based on port numbers
> >    * Aux driver - basic driver, for sending messages over different
> > mboxes, handling ranges of doorbells
> >
> >    Is it here anyone who is working on any improvement, or anyone who
> > knows the development plans for RapidIO subsystem?
> >
> 
> AFAIK, there is no one currently working on these features for Linux.
> It will be good if you can add these useful features.
Yes it looks like that, currently we are analyzing current rapidIO
system, and how we can add these features. 

> 
> > [2] - I have a following problem with a current implementation of
> > loading drivers. The driver probe-function call is based on comparison
> > of VendorID (VID) and DeviceID (DID) only. Thus if I have 3 devices with
> > same DID and VID connected to the same network (bus), the driver is
> > loaded 3times, instead only once for the actual device Master port.
> 
> This should be the correct way as you actually have 3 instances of the device.
> 
> >
> > Rionet driver solved this by enabling to call initialization function
> > just once, and it expect that this is the Master port.
> 
> Rionet is kind of special.  It's not working like a simple device
> driver, but more like a customized protocol stack to support multiple
> ethernet over rio links.
> 
> >
> > Is it this correct behavior  ? It looks to me that RapidIO is handled
> > like a local bus (like PCI)
> 
> This is correct behavior.  All of them are using Linux device/driver
> infrastructure, but rionet is a special device.

But I do not have a 3 devices on one silicon. I am talking about 3
devices (3 x EP8548 boards + IDT switch) connected over rapidIO through
the switch. And in this case I'd like to have only one driver siting on
the top of Linux RapidIO subsystem. I don't see the advantage of loading
a driver locally for remote device. Am I missing something  ?

And one more think, I am getting so much Bus errors OOPSes. Whenever
there is a problem with a comunication over Rio I get such a kernel OPS.
I had to add some delays into some function to be able to finish the
enum+discovery process. Did you have some experience with some bigger
rio network running under linux ? 


                Jan 

Re: RapidIO - general questions

[Li Yang]

On Fri, May 15, 2009 at 3:33 PM, Jan Neskudla <jan.neskudla.ext@nsn.com> wr=
ote:
> On Wed, 2009-05-13 at 18:57 +0800, ext Li Yang wrote:
>> cc'ed LKML
>>
>> On Tue, May 12, 2009 at 5:17 PM, Jan Neskudla <jan.neskudla.ext@nsn.com>=
 wrote:
>> > Hallo
>> >
>> > we'd likes to use a RapidIO as a general communication bus on our new
>> > product, and so I have some questions about general design of Linux RI=
O
>> > subsystem. I did not find any better mailing list for RapidIO
>> > discussion.
>> >
>> > [1] - we'd like to implement following features
>> > =C2=A0 =C2=A0* Hot-plug (hot-insert/hot-remove) of devices
>> > =C2=A0 =C2=A0* Error handling (port-write packets - configuration, han=
dling of
>> > them)
>> > =C2=A0 =C2=A0* Static ID configuration based on port numbers
>> > =C2=A0 =C2=A0* Aux driver - basic driver, for sending messages over di=
fferent
>> > mboxes, handling ranges of doorbells
>> >
>> > =C2=A0 =C2=A0Is it here anyone who is working on any improvement, or a=
nyone who
>> > knows the development plans for RapidIO subsystem?
>> >
>>
>> AFAIK, there is no one currently working on these features for Linux.
>> It will be good if you can add these useful features.
> Yes it looks like that, currently we are analyzing current rapidIO
> system, and how we can add these features.
>
>>
>> > [2] - I have a following problem with a current implementation of
>> > loading drivers. The driver probe-function call is based on comparison
>> > of VendorID (VID) and DeviceID (DID) only. Thus if I have 3 devices wi=
th
>> > same DID and VID connected to the same network (bus), the driver is
>> > loaded 3times, instead only once for the actual device Master port.
>>
>> This should be the correct way as you actually have 3 instances of the d=
evice.
>>
>> >
>> > Rionet driver solved this by enabling to call initialization function
>> > just once, and it expect that this is the Master port.
>>
>> Rionet is kind of special. =C2=A0It's not working like a simple device
>> driver, but more like a customized protocol stack to support multiple
>> ethernet over rio links.
>>
>> >
>> > Is it this correct behavior =C2=A0? It looks to me that RapidIO is han=
dled
>> > like a local bus (like PCI)
>>
>> This is correct behavior. =C2=A0All of them are using Linux device/drive=
r
>> infrastructure, but rionet is a special device.
>
> But I do not have a 3 devices on one silicon. I am talking about 3
> devices (3 x EP8548 boards + IDT switch) connected over rapidIO through
> the switch. And in this case I'd like to have only one driver siting on
> the top of Linux RapidIO subsystem. I don't see the advantage of loading

You are having one driver, but it probes 3 times for each device using
the driver.

> a driver locally for remote device. Am I missing something =C2=A0?

If you want to interact with the remote device, you need the driver to
do the work locally.

>
> And one more think, I am getting so much Bus errors OOPSes. Whenever
> there is a problem with a comunication over Rio I get such a kernel OPS.
> I had to add some delays into some function to be able to finish the
> enum+discovery process. Did you have some experience with some bigger
> rio network running under linux ?

It looks like an known issue for switched rio network, but I don't
have the correct equipment to reproduce the problem here.  Could you
do some basic debugging and share your findings?  Thanks.

- Leo

Re: RapidIO - general questions

[Jan Neskudla]

n Fri, 2009-05-15 at 15:56 +0800, ext Li Yang wrote:
> On Fri, May 15, 2009 at 3:33 PM, Jan Neskudla <jan.neskudla.ext@nsn.com> wrote:
> > On Wed, 2009-05-13 at 18:57 +0800, ext Li Yang wrote:
> >> cc'ed LKML
> >>
> >> On Tue, May 12, 2009 at 5:17 PM, Jan Neskudla <jan.neskudla.ext@nsn.com> wrote:
> >> > Hallo
> >> >
> >> > we'd likes to use a RapidIO as a general communication bus on our new
> >> > product, and so I have some questions about general design of Linux RIO
> >> > subsystem. I did not find any better mailing list for RapidIO
> >> > discussion.
> >> >
> >> > [1] - we'd like to implement following features
> >> >    * Hot-plug (hot-insert/hot-remove) of devices
> >> >    * Error handling (port-write packets - configuration, handling of
> >> > them)
> >> >    * Static ID configuration based on port numbers
> >> >    * Aux driver - basic driver, for sending messages over different
> >> > mboxes, handling ranges of doorbells
> >> >
> >> >    Is it here anyone who is working on any improvement, or anyone who
> >> > knows the development plans for RapidIO subsystem?
> >> >
> >>
> >> AFAIK, there is no one currently working on these features for Linux.
> >> It will be good if you can add these useful features.
> > Yes it looks like that, currently we are analyzing current rapidIO
> > system, and how we can add these features.
> >
> >>
> >> > [2] - I have a following problem with a current implementation of
> >> > loading drivers. The driver probe-function call is based on comparison
> >> > of VendorID (VID) and DeviceID (DID) only. Thus if I have 3 devices with
> >> > same DID and VID connected to the same network (bus), the driver is
> >> > loaded 3times, instead only once for the actual device Master port.
> >>
> >> This should be the correct way as you actually have 3 instances of the device.
> >>
> >> >
> >> > Rionet driver solved this by enabling to call initialization function
> >> > just once, and it expect that this is the Master port.
> >>
> >> Rionet is kind of special.  It's not working like a simple device
> >> driver, but more like a customized protocol stack to support multiple
> >> ethernet over rio links.
> >>
> >> >
> >> > Is it this correct behavior  ? It looks to me that RapidIO is handled
> >> > like a local bus (like PCI)
> >>
> >> This is correct behavior.  All of them are using Linux device/driver
> >> infrastructure, but rionet is a special device.
> >
> > But I do not have a 3 devices on one silicon. I am talking about 3
> > devices (3 x EP8548 boards + IDT switch) connected over rapidIO through
> > the switch. And in this case I'd like to have only one driver siting on
> > the top of Linux RapidIO subsystem. I don't see the advantage of loading
> 
> You are having one driver, but it probes 3 times for each device using
> the driver.
> 
> > a driver locally for remote device. Am I missing something  ?
> 
> If you want to interact with the remote device, you need the driver to
> do the work locally.

We are going to use a RapidIO as a bigger network of active devices, and
each will have each own driver (sitting on its own), and all the
settings will be done over maintenance packets. 

May be it will be solved by the fact, that we are going to use a
staticIDs, so there will be no discovery as it is now. And thus there
will be only one device visible in the internal structures of the
subsystem, and thus only one drive will be loaded. 

> 
> >
> > And one more think, I am getting so much Bus errors OOPSes. Whenever
> > there is a problem with a comunication over Rio I get such a kernel OPS.
> > I had to add some delays into some function to be able to finish the
> > enum+discovery process. Did you have some experience with some bigger
> > rio network running under linux ?
> 
> It looks like an known issue for switched rio network, but I don't
> have the correct equipment to reproduce the problem here.  Could you
> do some basic debugging and share your findings?  Thanks.

I tried to acquired some info about the problem, I found that the OOPS
always occur when there is no respond from the device or the respond is
too slow. I always got that error during function call
rio_get_host_deviceid_lock when it tries to access a remote device or
switch. This function is the first call of the rio_mport_read_config_32
so is also first try of remote access to any device. 

It is a timing issue, and after placing a printk into the
rio_get_host_deviceid_lock the OOPSing almost disappeared. 

                                 Jan 

> 
> - Leo

RE: RapidIO - general questions

[Anderson, Trevor]

With=20regards=20to=20your=20Oops:=20we=20sometimes=20find=20that,=20altho=
ugh=20a=20switch=20may
report=20a=20port=20being=20active,=20whenever
we=20try=20to=20discover=20what=20lies=20behind=20it,=20transfer=20errors=20=
occur=20that=20are
non-recoverable.

As=20a=20solution,=20on=20Freescale=20MPC8641D,=20we=20use=20a=20DMA=20tra=
nsfer=20to=20perform=20a
simple=20MAINT=20read=20on=20a=20new=20device,=20just=20to=20see=20if=20it=
=20responds.=20Transfer
errors=20may=20still=20happen,=20but=20they=20are=20recoverable=20in=20tha=
t=20case,=20and=20you
can=20train=20your=20code=20to=20avoid=20that=20port=20or=20try=20it=20lat=
er.=20(The=20DMA=20engine
will=20report=20the=20error,=20not=20the=20processor.)



>=20-----Original=20Message-----
>=20From:=20linuxppc-dev-bounces+tanderson=3Dcurtisswright.com@ozlabs.org
[mailto:linuxppc-dev-
>=20bounces+tanderson=3Dcurtisswright.com@ozlabs.org]=20On=20Behalf=20Of=20=
Jan
Neskudla
>=20Sent:=20Wednesday,=20May=2020,=202009=2012:00=20AM
>=20To:=20ext=20Li=20Yang
>=20Cc:=20linuxppc-dev;=20linux-kernel@vger.kernel.org
>=20Subject:=20Re:=20RapidIO=20-=20general=20questions
>=20
>=20n=20Fri,=202009-05-15=20at=2015:56=20+0800,=20ext=20Li=20Yang=20wrote:=

>=20>=20On=20Fri,=20May=2015,=202009=20at=203:33=20PM,=20Jan=20Neskudla
<jan.neskudla.ext@nsn.com>=20wrote:
>=20>=20>=20On=20Wed,=202009-05-13=20at=2018:57=20+0800,=20ext=20Li=20Yang=
=20wrote:
>=20>=20>>=20cc'ed=20LKML
>=20>=20>>
>=20>=20>>=20On=20Tue,=20May=2012,=202009=20at=205:17=20PM,=20Jan=20Neskud=
la
<jan.neskudla.ext@nsn.com>=20wrote:
>=20>=20>>=20>=20Hallo
>=20>=20>>=20>
>=20>=20>>=20>=20we'd=20likes=20to=20use=20a=20RapidIO=20as=20a=20general=20=
communication=20bus=20on
our=20new
>=20>=20>>=20>=20product,=20and=20so=20I=20have=20some=20questions=20about=
=20general=20design=20of
Linux=20RIO
>=20>=20>>=20>=20subsystem.=20I=20did=20not=20find=20any=20better=20mailin=
g=20list=20for=20RapidIO
>=20>=20>>=20>=20discussion.
>=20>=20>>=20>
>=20>=20>>=20>=20[1]=20-=20we'd=20like=20to=20implement=20following=20feat=
ures
>=20>=20>>=20>=20=20=20=20*=20Hot-plug=20(hot-insert/hot-remove)=20of=20de=
vices
>=20>=20>>=20>=20=20=20=20*=20Error=20handling=20(port-write=20packets=20-=
=20configuration,
handling=20of
>=20>=20>>=20>=20them)
>=20>=20>>=20>=20=20=20=20*=20Static=20ID=20configuration=20based=20on=20p=
ort=20numbers
>=20>=20>>=20>=20=20=20=20*=20Aux=20driver=20-=20basic=20driver,=20for=20s=
ending=20messages=20over
different
>=20>=20>>=20>=20mboxes,=20handling=20ranges=20of=20doorbells
>=20>=20>>=20>
>=20>=20>>=20>=20=20=20=20Is=20it=20here=20anyone=20who=20is=20working=20o=
n=20any=20improvement,=20or
anyone=20who
>=20>=20>>=20>=20knows=20the=20development=20plans=20for=20RapidIO=20subsy=
stem?
>=20>=20>>=20>
>=20>=20>>
>=20>=20>>=20AFAIK,=20there=20is=20no=20one=20currently=20working=20on=20t=
hese=20features=20for
Linux.
>=20>=20>>=20It=20will=20be=20good=20if=20you=20can=20add=20these=20useful=
=20features.
>=20>=20>=20Yes=20it=20looks=20like=20that,=20currently=20we=20are=20analy=
zing=20current=20rapidIO
>=20>=20>=20system,=20and=20how=20we=20can=20add=20these=20features.
>=20>=20>
>=20>=20>>
>=20>=20>>=20>=20[2]=20-=20I=20have=20a=20following=20problem=20with=20a=20=
current=20implementation
of
>=20>=20>>=20>=20loading=20drivers.=20The=20driver=20probe-function=20call=
=20is=20based=20on
comparison
>=20>=20>>=20>=20of=20VendorID=20(VID)=20and=20DeviceID=20(DID)=20only.=20=
Thus=20if=20I=20have=203
devices=20with
>=20>=20>>=20>=20same=20DID=20and=20VID=20connected=20to=20the=20same=20ne=
twork=20(bus),=20the
driver=20is
>=20>=20>>=20>=20loaded=203times,=20instead=20only=20once=20for=20the=20ac=
tual=20device=20Master
port.
>=20>=20>>
>=20>=20>>=20This=20should=20be=20the=20correct=20way=20as=20you=20actuall=
y=20have=203=20instances
of=20the=20device.
>=20>=20>>
>=20>=20>>=20>
>=20>=20>>=20>=20Rionet=20driver=20solved=20this=20by=20enabling=20to=20ca=
ll=20initialization
function
>=20>=20>>=20>=20just=20once,=20and=20it=20expect=20that=20this=20is=20the=
=20Master=20port.
>=20>=20>>
>=20>=20>>=20Rionet=20is=20kind=20of=20special.=20=20It's=20not=20working=20=
like=20a=20simple=20device
>=20>=20>>=20driver,=20but=20more=20like=20a=20customized=20protocol=20sta=
ck=20to=20support
multiple
>=20>=20>>=20ethernet=20over=20rio=20links.
>=20>=20>>
>=20>=20>>=20>
>=20>=20>>=20>=20Is=20it=20this=20correct=20behavior=20=20?=20It=20looks=20=
to=20me=20that=20RapidIO=20is
handled
>=20>=20>>=20>=20like=20a=20local=20bus=20(like=20PCI)
>=20>=20>>
>=20>=20>>=20This=20is=20correct=20behavior.=20=20All=20of=20them=20are=20=
using=20Linux
device/driver
>=20>=20>>=20infrastructure,=20but=20rionet=20is=20a=20special=20device.
>=20>=20>
>=20>=20>=20But=20I=20do=20not=20have=20a=203=20devices=20on=20one=20silic=
on.=20I=20am=20talking=20about=203
>=20>=20>=20devices=20(3=20x=20EP8548=20boards=20+=20IDT=20switch)=20conne=
cted=20over=20rapidIO
through
>=20>=20>=20the=20switch.=20And=20in=20this=20case=20I'd=20like=20to=20hav=
e=20only=20one=20driver
siting=20on
>=20>=20>=20the=20top=20of=20Linux=20RapidIO=20subsystem.=20I=20don't=20se=
e=20the=20advantage=20of
loading
>=20>
>=20>=20You=20are=20having=20one=20driver,=20but=20it=20probes=203=20times=
=20for=20each=20device
using
>=20>=20the=20driver.
>=20>
>=20>=20>=20a=20driver=20locally=20for=20remote=20device.=20Am=20I=20missi=
ng=20something=20=20?
>=20>
>=20>=20If=20you=20want=20to=20interact=20with=20the=20remote=20device,=20=
you=20need=20the=20driver
to
>=20>=20do=20the=20work=20locally.
>=20
>=20We=20are=20going=20to=20use=20a=20RapidIO=20as=20a=20bigger=20network=20=
of=20active=20devices,
and
>=20each=20will=20have=20each=20own=20driver=20(sitting=20on=20its=20own),=
=20and=20all=20the
>=20settings=20will=20be=20done=20over=20maintenance=20packets.
>=20
>=20May=20be=20it=20will=20be=20solved=20by=20the=20fact,=20that=20we=20ar=
e=20going=20to=20use=20a
>=20staticIDs,=20so=20there=20will=20be=20no=20discovery=20as=20it=20is=20=
now.=20And=20thus=20there
>=20will=20be=20only=20one=20device=20visible=20in=20the=20internal=20stru=
ctures=20of=20the
>=20subsystem,=20and=20thus=20only=20one=20drive=20will=20be=20loaded.
>=20
>=20>
>=20>=20>
>=20>=20>=20And=20one=20more=20think,=20I=20am=20getting=20so=20much=20Bus=
=20errors=20OOPSes.
Whenever
>=20>=20>=20there=20is=20a=20problem=20with=20a=20comunication=20over=20Ri=
o=20I=20get=20such=20a
kernel=20OPS.
>=20>=20>=20I=20had=20to=20add=20some=20delays=20into=20some=20function=20=
to=20be=20able=20to=20finish
the
>=20>=20>=20enum+discovery=20process.=20Did=20you=20have=20some=20experien=
ce=20with=20some
bigger
>=20>=20>=20rio=20network=20running=20under=20linux=20?
>=20>
>=20>=20It=20looks=20like=20an=20known=20issue=20for=20switched=20rio=20ne=
twork,=20but=20I=20don't
>=20>=20have=20the=20correct=20equipment=20to=20reproduce=20the=20problem=20=
here.=20=20Could=20you
>=20>=20do=20some=20basic=20debugging=20and=20share=20your=20findings?=20=20=
Thanks.
>=20
>=20I=20tried=20to=20acquired=20some=20info=20about=20the=20problem,=20I=20=
found=20that=20the=20OOPS
>=20always=20occur=20when=20there=20is=20no=20respond=20from=20the=20devic=
e=20or=20the=20respond
is
>=20too=20slow.=20I=20always=20got=20that=20error=20during=20function=20ca=
ll
>=20rio_get_host_deviceid_lock=20when=20it=20tries=20to=20access=20a=20rem=
ote=20device=20or
>=20switch.=20This=20function=20is=20the=20first=20call=20of=20the
rio_mport_read_config_32
>=20so=20is=20also=20first=20try=20of=20remote=20access=20to=20any=20devic=
e.
>=20
>=20It=20is=20a=20timing=20issue,=20and=20after=20placing=20a=20printk=20i=
nto=20the
>=20rio_get_host_deviceid_lock=20the=20OOPSing=20almost=20disappeared.
>=20
>=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20Jan
>=20
>=20>
>=20>=20-=20Leo
>=20
>=20_______________________________________________
>=20Linuxppc-dev=20mailing=20list
>=20Linuxppc-dev@ozlabs.org
>=20https://ozlabs.org/mailman/listinfo/linuxppc-dev

_______________________________________________________________________
This=20e-mail=20and=20any=20files=20transmitted=20with=20it=20are=20propri=
etary=20and=20intended=20solely=20for=20the=20use=20of=20the=20individual=20=
or=20entity=20to=20whom=20they=20are=20addressed.=20If=20you=20have=20reas=
on=20to=20believe=20that=20you=20have=20received=20this=20e-mail=20in=20er=
ror,=20please=20notify=20the=20sender=20and=20destroy=20this=20email=20and=
=20any=20attached=20files.=20Please=20note=20that=20any=20views=20or=20opi=
nions=20presented=20in=20this=20e-mail=20are=20solely=20those=20of=20the=20=
author=20and=20do=20not=20necessarily=20represent=20those=20of=20the=20Cur=
tiss-Wright=20Corporation=20or=20any=20of=20its=20subsidiaries.=20=20Docum=
ents=20attached=20hereto=20may=20contain=20technology=20subject=20to=20gov=
ernment=20export=20regulations.=20Recipient=20is=20solely=20responsible=20=
for=20ensuring=20that=20any=20re-export,=20transfer=20or=20disclosure=20of=
=20this=20information=20is=20in=20accordance=20with=20applicable=20governm=
ent=20export=20regulations.=20=20The=20recipient=20should=20check=20this=20=
e-mail=20and=20any=20attachments=20for=20the=20presence=20of=20viruses.=20=
Curtiss-Wright=20Corporation=20and=20its=20subsidiaries=20accept=20no=20li=
ability=20for=20any=20damage=20caused=20by=20any=20virus=20transmitted=20b=
y=20this=20e-mail.

Re: RapidIO - general questions

[Jan Neskudla]

Hi as I already informed you, 
we'd like to contribute to the Linux RapidIO subsystem, with several
features, Here are few general information about the design of
implementation of such features.

naming: 
domain - a several boards connected together via rio, with only one host
host -  a MCU with host bit set, only one per domain
domain master - a host which have also domain_master_bit set ( boot
param), only one in overall system. 

* Domains configuration - traverse the whole rio network by domain
master to find all the hosts, providing a domain IDs to them and finally
programing domain routing tables to the switches. Since we are
cooperating with a IDT as a switch supplier, the IDT will public their
private API for setting such a domain routing tables under GPL. 
- there is an issue how to proceed with a locking of the switches,
during enumeration of domains and later on during enumeration of
endpoint by hosts. All the time, of running system, there  will be two
MCUs trying to configure the same switch in certain situation,
especially during hot-plug of domain. So this is not clear yet.

* Static ID configuration based on port numbers  
  tree sysfs files for providing necessary info : 
      host_id - this is the same like riohdid boot parameter now
      switch_ids - this file is for providing source ids for switches to
be able to report problems via port-write packets 
      endpoint_ids - to provide list of all endpoint in one domain.  
  I read somewhere that passing of structures via sysFS is not
acceptable, but how to pass some more information to the kernel. 
I expect to pass those structures via bin sysFS files, analyze the input
not only cast it, and than use it. Is it this OK ? 

* User triggered enumeration/discovery
This is necessary because of the static IDs. They have to be know before
enumeration can start, and this is most general way of doing this. 
So once the static Ids are provided over sysFS files, then via another
sysFS file a enumeration is triggered. I am talking about enumeration
only because the endpoints that wait for the enumeration are going to
preform discovery as usual, after enumeration. 
This needs some changes. Whole enumeration process have to be put into
the kthread and discovery as well. Then kernel can boots up to user
space and enumeration can be triggered. 
Is there any standard way in the kernel how to postpone and than trigger
a configuration of a bus from the user space ? 

* User space library for configuring switches 
IDT is going to provide user space GPL library that will covers an rio
switch configuration based on rio spec 2.0. 

* Error handling (port-write packets - configuration, handling of them)
This should be as general as possible, and IDT is designing this part.
So a port-write packet (have to be written) driver will receive a
packet, analyze this and perform an action: - two scenarios can happen.
1. The port-write info is part of the rio spec 2.0 and the packet is
processed by kernel directly 
2. The port-write info is vendor specific and it is passed to the user
space, where is processed and proper action is than taken via current
sysFS config files. 

* Hot-plug (hot-insert/hot-remove) of devices
This is case of error handling. 
In case of any error covered by rio spec 2.0 (bad CRC, bad
character, .. ) the ports of the switch that generates this port-write
info are scanned for PORT_OK status or PORT_UNINITIALIZED so we are able
to catch the hot-plug/hot-extract of any device. 
Hotplug should be functional in the time of enumeration, because
enumeration process traverse the system port by port, and if some
endpoint is powered on after enumeration process testes its port but
before the end of the standard enumeration process, this device can be
missed. 

* Aux driver - basic driver, for sending messages over different
mboxes, 
right now we implemented this as a character device. If any one is
interested let me know, I will send this to you. 

On the end we'd like to support already existing scenario of dynamic
assignment of the devID as well as static ID and user space triggered
enumeration. 

The question is if there is a static table of IDs, do we still needs an
discovery process on every endpoint ? Propagating any hot-plug/hot-
extract event to every endpoint to reflect this in local sysfs structure
would be quite hard. 

Any comment to this topics is highly appreciated, as well as forwarding
this to anyone who can be interested. 

                           Jan 


 

On Wed, 2009-05-20 at 09:00 +0200, ext Jan Neskudla wrote:
> n Fri, 2009-05-15 at 15:56 +0800, ext Li Yang wrote:
> > On Fri, May 15, 2009 at 3:33 PM, Jan Neskudla
> <jan.neskudla.ext@nsn.com> wrote:
> > > On Wed, 2009-05-13 at 18:57 +0800, ext Li Yang wrote:
> > >> cc'ed LKML
> > >>
> > >> On Tue, May 12, 2009 at 5:17 PM, Jan Neskudla
> <jan.neskudla.ext@nsn.com> wrote:
> > >> > Hallo
> > >> >
> > >> > we'd likes to use a RapidIO as a general communication bus on
> our new
> > >> > product, and so I have some questions about general design of
> Linux RIO
> > >> > subsystem. I did not find any better mailing list for RapidIO
> > >> > discussion.
> > >> >
> > >> > [1] - we'd like to implement following features
> > >> >    * Hot-plug (hot-insert/hot-remove) of devices
> > >> >    * Error handling (port-write packets - configuration,
> handling of
> > >> > them)
> > >> >    * Static ID configuration based on port numbers
> > >> >    * Aux driver - basic driver, for sending messages over
> different
> > >> > mboxes, handling ranges of doorbells
> > >> >
> > >> >    Is it here anyone who is working on any improvement, or
> anyone who
> > >> > knows the development plans for RapidIO subsystem?
> > >> >
> > >>
> > >> AFAIK, there is no one currently working on these features for
> Linux.
> > >> It will be good if you can add these useful features.
> > > Yes it looks like that, currently we are analyzing current rapidIO
> > > system, and how we can add these features.
> > >
> > >>
> > >> > [2] - I have a following problem with a current implementation
> of
> > >> > loading drivers. The driver probe-function call is based on
> comparison
> > >> > of VendorID (VID) and DeviceID (DID) only. Thus if I have 3
> devices with
> > >> > same DID and VID connected to the same network (bus), the
> driver is
> > >> > loaded 3times, instead only once for the actual device Master
> port.
> > >>
> > >> This should be the correct way as you actually have 3 instances
> of the device.
> > >>
> > >> >
> > >> > Rionet driver solved this by enabling to call initialization
> function
> > >> > just once, and it expect that this is the Master port.
> > >>
> > >> Rionet is kind of special.  It's not working like a simple device
> > >> driver, but more like a customized protocol stack to support
> multiple
> > >> ethernet over rio links.
> > >>
> > >> >
> > >> > Is it this correct behavior  ? It looks to me that RapidIO is
> handled
> > >> > like a local bus (like PCI)
> > >>
> > >> This is correct behavior.  All of them are using Linux
> device/driver
> > >> infrastructure, but rionet is a special device.
> > >
> > > But I do not have a 3 devices on one silicon. I am talking about 3
> > > devices (3 x EP8548 boards + IDT switch) connected over rapidIO
> through
> > > the switch. And in this case I'd like to have only one driver
> siting on
> > > the top of Linux RapidIO subsystem. I don't see the advantage of
> loading
> >
> > You are having one driver, but it probes 3 times for each device
> using
> > the driver.
> >
> > > a driver locally for remote device. Am I missing something  ?
> >
> > If you want to interact with the remote device, you need the driver
> to
> > do the work locally.
> 
> We are going to use a RapidIO as a bigger network of active devices,
> and
> each will have each own driver (sitting on its own), and all the
> settings will be done over maintenance packets.
> 
> May be it will be solved by the fact, that we are going to use a
> staticIDs, so there will be no discovery as it is now. And thus there
> will be only one device visible in the internal structures of the
> subsystem, and thus only one drive will be loaded.
> 
> >
> > >
> > > And one more think, I am getting so much Bus errors OOPSes.
> Whenever
> > > there is a problem with a comunication over Rio I get such a
> kernel OPS.
> > > I had to add some delays into some function to be able to finish
> the
> > > enum+discovery process. Did you have some experience with some
> bigger
> > > rio network running under linux ?
> >
> > It looks like an known issue for switched rio network, but I don't
> > have the correct equipment to reproduce the problem here.  Could you
> > do some basic debugging and share your findings?  Thanks.
> 
> I tried to acquired some info about the problem, I found that the OOPS
> always occur when there is no respond from the device or the respond
> is
> too slow. I always got that error during function call
> rio_get_host_deviceid_lock when it tries to access a remote device or
> switch. This function is the first call of the
> rio_mport_read_config_32
> so is also first try of remote access to any device.
> 
> It is a timing issue, and after placing a printk into the
> rio_get_host_deviceid_lock the OOPSing almost disappeared.
> 
>                                  Jan
> 
> >
> > - Leo
> 
> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@ozlabs.org
> https://ozlabs.org/mailman/listinfo/linuxppc-dev
> 
> 

Re: RapidIO - general questions

[david.hagood]

Do you know (and if you know, can you comment) if IDT is planning on
offering RIO (and more importantly to me sRIO) chipsets that can be used
on other architectures besides the various PPC chips with embedded sRIO
controllers?

Re: RapidIO - general questions

[Jan Neskudla]

On Mon, 2009-06-29 at 10:44 -0500, ext david.hagood@gmail.com wrote:
> Do you know (and if you know, can you comment) if IDT is planning on
> offering RIO (and more importantly to me sRIO) chipsets that can be used
> on other architectures besides the various PPC chips with embedded sRIO
> controllers?
I am using only switches, and they are not fixed to any architecture.
You can use them independently to any MCU. 
I do not know if they are planning to provides something more. 
                     Jan 
> 
> 
> 
> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/linuxppc-dev