[Fwd: IPoIB]

[Fwd: IPoIB]

[Joseph Golio]

Joseph Golio wrote:

To whom it may concern,


I originally sent this note to Alan Cox, but he has suggested that I send it to
you folks. I'm
new at this process so please bare with me...

So, here goes.

> Alan,
>
>     I have a question. I am working on developing a Linux driver for IP over
> Infiniband (IPoIB) and
>     have run into an issue that I need your advice. The draft standard from the
> IETF on IPoIB
>     encapsulation and address resolution over Infiniband networks (see the link
> below - section 6.1.1)
>     defines the hardware address as being 20 bytes in length. It appears that
> the "netdevice.h" file in
>     Linux has MAX_ADDR_LEN set to 7 (at least in my version which is SuSe 7.3 -
> kernel 2.4.4).
>
> http://www.ietf.org/internet-drafts/draft-ietf-ipoib-ip-over-infiniband-01.txt
>
>     Obviously, this will not.
>
>     What do you suggest as a good solution to this problem ?
>
>     Thanks in advance for a quick response.
>
> Thanks for your time,
>
> Joe Golio
> golio@vieo.com
> 651-698-9350 (ext. 111)
> VIEO Inc.
>
> VIEO: The Adaptive Application Infrastructure Management (AAIM) Company

Re: [Fwd: IPoIB]

[Thomas 'Dent' Mirlacher]

--snip/snip

> >     I have a question. I am working on developing a Linux driver for IP over
> > Infiniband (IPoIB) and
> >     have run into an issue that I need your advice. The draft standard from the
> > IETF on IPoIB
> >     encapsulation and address resolution over Infiniband networks (see the link
> > below - section 6.1.1)
> >     defines the hardware address as being 20 bytes in length. It appears that
> > the "netdevice.h" file in
> >     Linux has MAX_ADDR_LEN set to 7 (at least in my version which is SuSe 7.3 -


for 2.5.18 at least it's set to 8, but there is no reason to not change it to
20 beside wasting some memory

n time for
	a) broadcast address
	b) device address
	int netdevice
sum(m[n]) times for the multicast list

where n == number of network devices, m == number of MC entries per device
as i can see it.

and this "overhead" should be really acceptable :)

... probably you will break some "external" stuff like freeswan, but this
shouldn't be your problem.

	tm

-- 
in some way i do, and in some way i don't.

Re: [Fwd: IPoIB]

[Joseph Golio]

Thomas 'Dent' Mirlacher wrote:

That's what I thought. Now, I can make the change to my local systems for
development and we could even give instructions to our customers to make
the change to their systems also. However, how would I get this change
propogated to the next release of the Linux kernel (2.5.18 plus...) ?

Thanks,

Joe

> --snip/snip
>
> > >     I have a question. I am working on developing a Linux driver for IP over
> > > Infiniband (IPoIB) and
> > >     have run into an issue that I need your advice. The draft standard from the
> > > IETF on IPoIB
> > >     encapsulation and address resolution over Infiniband networks (see the link
> > > below - section 6.1.1)
> > >     defines the hardware address as being 20 bytes in length. It appears that
> > > the "netdevice.h" file in
> > >     Linux has MAX_ADDR_LEN set to 7 (at least in my version which is SuSe 7.3 -
>
> for 2.5.18 at least it's set to 8, but there is no reason to not change it to
> 20 beside wasting some memory
>
> n time for
>         a) broadcast address
>         b) device address
>         int netdevice
> sum(m[n]) times for the multicast list
>
> where n == number of network devices, m == number of MC entries per device
> as i can see it.
>
> and this "overhead" should be really acceptable :)
>
> ... probably you will break some "external" stuff like freeswan, but this
> shouldn't be your problem.
>
>         tm
>
> --
> in some way i do, and in some way i don't.

Re: [Fwd: IPoIB]

[Joseph Golio]

Thomas 'Dent' Mirlacher wrote:

Thomas,

    Thanks for the reply. The more I dig into this, the more I figure out that the
problem is larger than I though. Not only do I need to increase the size of a
hardware address, I also need to provide a new interface service routine to allow
the device driver to lookup/translate a portion of the hardware address  and return
more information that needs to go into the ARP cache.

Specifically, the IETF IPoIB group has defined an Infiniband hardware address to
have 1 byte of reserved, 3 bytes of Queue Pair Number (QPN) and 16 bytes of
Global Identifier (GID). Unfortunately, in order to build the hardware header
in packets to send, another piece of information is needed -- that being the Local
Identifier (LID). If the LID is kept in the ARP cache, then it will be readily
available
from "arp_find()" at the time of transmission. If we don't do that, we would have to
have a separate table and mechanism in the driver to map from QPN/GID ===> LID.
I would rather not do that if I didn't have to. It seems fundamental (at least to me)
that
everything one needs to build the hardware header  (at least everything that is
related
specifically to the destination) should be stored in the ARP cache.

Thoughts  anyone ?

Thanks,

Joe

> --snip/snip
>
> > >     I have a question. I am working on developing a Linux driver for IP over
> > > Infiniband (IPoIB) and
> > >     have run into an issue that I need your advice. The draft standard from the
> > > IETF on IPoIB
> > >     encapsulation and address resolution over Infiniband networks (see the link
> > > below - section 6.1.1)
> > >     defines the hardware address as being 20 bytes in length. It appears that
> > > the "netdevice.h" file in
> > >     Linux has MAX_ADDR_LEN set to 7 (at least in my version which is SuSe 7.3 -
>
> for 2.5.18 at least it's set to 8, but there is no reason to not change it to
> 20 beside wasting some memory
>
> n time for
>         a) broadcast address
>         b) device address
>         int netdevice
> sum(m[n]) times for the multicast list
>
> where n == number of network devices, m == number of MC entries per device
> as i can see it.
>
> and this "overhead" should be really acceptable :)
>
> ... probably you will break some "external" stuff like freeswan, but this
> shouldn't be your problem.
>
>         tm
>
> --
> in some way i do, and in some way i don't.

Re: [Fwd: IPoIB]

[Thomas 'Dent' Mirlacher]

--snip/snip

> The more I dig into this, the more I figure out that the
> problem is larger than I though. Not only do I need to increase the size of a
> hardware address, I also need to provide a new interface service routine to
> allow the device driver to lookup/translate a portion of the hardware address
> and return more information that needs to go into the ARP cache.
> 
> Specifically, the IETF IPoIB group has defined an Infiniband hardware
> address to have 1 byte of reserved, 3 bytes of Queue Pair Number (QPN) and
> 16 bytes of Global Identifier (GID). Unfortunately, in order to build the
> hardware header in packets to send, another piece of information is needed
> -- that being the Local
> Identifier (LID). If the LID is kept in the ARP cache, then it will be readily
> available from "arp_find()" at the time of transmission.

> If we don't do that, we would have to
> have a separate table and mechanism in the driver to map from
> QPN/GID ===> LID.
> I would rather not do that if I didn't have to. It seems fundamental
> (at least to me) that everything one needs to build the hardware header
> (at least everything that is related
> specifically to the destination) should be stored in the ARP cache.

you can already do that, since you will save everything in the arp
cache which is the size of dev->addr_len. - did i miss the point?

cheers,

	tm

-- 
in some way i do, and in some way i don't.

Re: [Fwd: IPoIB]

[Joseph Golio]

Thomas 'Dent' Mirlacher wrote:

> --snip/snip
>
> > The more I dig into this, the more I figure out that the
> > problem is larger than I though. Not only do I need to increase the size of a
> > hardware address, I also need to provide a new interface service routine to
> > allow the device driver to lookup/translate a portion of the hardware address
> > and return more information that needs to go into the ARP cache.
> >
> > Specifically, the IETF IPoIB group has defined an Infiniband hardware
> > address to have 1 byte of reserved, 3 bytes of Queue Pair Number (QPN) and
> > 16 bytes of Global Identifier (GID). Unfortunately, in order to build the
> > hardware header in packets to send, another piece of information is needed
> > -- that being the Local
> > Identifier (LID). If the LID is kept in the ARP cache, then it will be readily
> > available from "arp_find()" at the time of transmission.
>
> > If we don't do that, we would have to
> > have a separate table and mechanism in the driver to map from
> > QPN/GID ===> LID.
> > I would rather not do that if I didn't have to. It seems fundamental
> > (at least to me) that everything one needs to build the hardware header
> > (at least everything that is related
> > specifically to the destination) should be stored in the ARP cache.
>
> you can already do that, since you will save everything in the arp
> cache which is the size of dev->addr_len. - did i miss the point?

Either you missed the point, I didn't make the point, or I'm missing something...
:-)

Let me try this again. The IETF has defined the hardware address as having 2
fields.
The QPN and GID. Unfortunately, we need a 3rd item to build the headers. The LID.
The LID can be gotten by IBTA methods once we know the GID. So, implementations
that follow the IETF spec will be setting their local hardware address to their own

GID and some QPN. So, when I send out an ARP request and get back an ARP reply,
it will have QPN and GID. So, what get's stored in my local ARP cache, if I don't
do
anything to the ARP code, is simply the QPN/GID value. When I go to send a packet,
I will call arp_find() and get back the QPN/GID. What I really need is the LID. SO,

either I would have to translate QPN/GID to LID at that time (NOT GOOD), or, back
when the ARP reply came in, the ARP code could look at the arp_hrd and see if it's
 (32) ARPHRD_INFINIBAND and if it is, call an interface service routine to
suplement
the ARP info before storing it in the cache. I know, it's not pretty, but I'm
really struggling
here. If I were to do this in the driver, I would have to have some way of knowing
that the
ARP had completed so that I could then go out and translate the GID to LID and
stuff it
into a private driver lookup table (hashed probably). Not real crazy about that
idea...

Joe

>
>
> cheers,
>
>         tm
>
> --
> in some way i do, and in some way i don't.