Networking status

Networking status

[Marco Gerards]

Hi,

This weekend I worked on networking support for GRUB 2.  It is still
far from done.  A lot of people are interested in this functionality,
so I hope this email answers all questions.  At first I said I would
do this in February, but things went more smoothly than I expected.

First of all, what have I been doing:

Because I like working in userspace because it makes things easier to
debug, I am using grub-emu to implement networking.  In order to be
able to do this I am using TAP (a virtual Ethernet device).

I have developed Ethernet support.  At the moment only the TAP device
is supported.  But I plan to add support for IEEE 1275 and perhaps
UNDI support.  Vincent was working on support for PCI cards, I hope he
will find the time to finish this some day...

Except Ethernet support, I've also written ARP support and made it
possible to send IP+UDP packets.  The support to receive IP+UDP is not
finished yet.

How everything is implemented will be described when I send in the
patch, but feel free to ask questions about the implementation and
design.

Things I planned to do:

- Finish IPv4+UDP receive.
- Implement DHCP/BOOTP.
- Clean up the code a bit and document everything properly.
- Add IEEE 1275 support (so networking will work on the PPC and
  SPARC).
- Implement TFTP.

Things I will perhaps work on:

- UNDI
- Starting a GRUB hackers manual with clear examples of networking and
  how it all works.

Things I (most likely) will not do:

- PCI card support.

Thanks,
Marco

Re: Networking status

[NIIBE Yutaka]

Marco Gerards wrote:
> Things I planned to do:
>
> - Finish IPv4+UDP receive.
> - Implement DHCP/BOOTP.
> - Clean up the code a bit and document everything properly.
> - Add IEEE 1275 support (so networking will work on the PPC and
>   SPARC).
> - Implement TFTP.

FWIW, I developed a boot loader for M32R architecture which supports
HTTP download: http://www.gniibe.org/software/m32r-g00ff-20060107.tar.gz

It is single threaded, I mean, with polling (or busy loop).  It's only
for boot loader.

It supports both of IPv4 and IPv6.  ARP, Neighbour discovery of IPv6,
UDP, DHCP, DHCPv6, DNS, and HTTP/TCP is supported.

I know that TFTP is most popular for booting through the net.  But,
for our purpose (development of embedded system), HTTP is better.
People don't run TFTP server normally.  HTTP server is common, and put
a kernel on the web is easy thing to do.

If my code will be considered useful, I could transfer the copyright
to FSF (currently it's copyrighted by FSIJ).
-- 

Re: Networking status

[Jun OKAJIMA]

>
>FWIW, I developed a boot loader for M32R architecture which supports
>HTTP download: http://www.gniibe.org/software/m32r-g00ff-20060107.tar.gz
>


Dont get me wrong if my question is wrong   --- I dont know about
grub developmet so much. But, I have one question.

Why you dont use kexec() but grub?
I mean, first you boot a tiny linux kernel with kexec() patch.
A candidate is like linux-tiny. Then you wget to a desired kernel,
then kexec() it. What is wrong with this approach?

I believe implementing much function to boot loader is not so good idea.

                 --- Okajima, Jun. Tokyo, Japan.

URLs:
http://www.selenic.com/linux-tiny/

Re: Networking status

[NIIBE Yutaka]

Jun OKAJIMA wrote:
> Why you dont use kexec() but grub?

Just a comment from my point of view.

Perhaps, you live in a happy world, Linux on IA-32, or something where
all the functionalities are available *already*.

To run kexec successfully, we need working Linux beforehand.  FWIW,
we are developing or porting Linux itself.

Linux or other kernel requires many feature to porters.  Say, context
switch, semaphore, interrupt handling and others.  While my code runs
in single thread, no requirement on such things.
-- 

Re: Networking status

[Marco Gerards]

NIIBE Yutaka <gniibe@m17n.org> writes:

> FWIW, I developed a boot loader for M32R architecture which supports
> HTTP download: http://www.gniibe.org/software/m32r-g00ff-20060107.tar.gz
>
> It is single threaded, I mean, with polling (or busy loop).  It's only
> for boot loader.
>
> It supports both of IPv4 and IPv6.  ARP, Neighbour discovery of IPv6,
> UDP, DHCP, DHCPv6, DNS, and HTTP/TCP is supported.

Perhaps we can have a look how to integrate this with my code, do you
think it is possible?

Some advantages and disadvantages of both of our works.  Please
correct me if I am wrong:

My code:

* Advantages
- Already well integrated with GRUB 2.
- Already designed to be modular and support more than Ethernet.
* Disadvantages
- Not complete yet (no DHCP, BOOTP and TFTP, although they are all
  quite trivial I think).
- Not tested a lot yet.
- Support for testing by using TAP.

Your code:

* Advantages
- I assume it was tested a lot?
- TCP+HTTP
- DHCP, BOOTP and IPv6

From the looks the structure looks similar so I assume it can
integrate well.  Besides that my focus is on Ethernet, portability,
IPv4, UDP and TFTP.  All this is mostly implemented and I think it is
better I commit these things first.  I'd like IP+UDP to be in one
module and to implement BOOTP/DHCP, TFTP, etc. on top of that in
separate modules.

After that we can have a look at your code, see how well TCP
integrates.  What I am wondering is how complete it is, does it do
retransmissions, slow start, congestion control, etc?  Are you willing
to work with me on this?

> I know that TFTP is most popular for booting through the net.  But,
> for our purpose (development of embedded system), HTTP is better.
> People don't run TFTP server normally.  HTTP server is common, and put
> a kernel on the web is easy thing to do.

We should at least support TFTP because many people have a TFTP server
anyways.  For example people use PXE to load GRUB from the network
(TFTP), so it would be convenient to use TFTP as well for loading the
kernel.

But if you have a small TCP implementations that works well, be could
have a look at how to create a module out of it.  It would also be
very nice if NFS can be supported by GRUB 2 in that case, NFS is often
available in the environments network booting is used and is capable
of listing directory contents.

But at first I want to have the core functionality so people can use
it as the base of their code, for example for driver support.  At the
moment TCP and HTTP, if possible, are not that important.

> If my code will be considered useful, I could transfer the copyright
> to FSF (currently it's copyrighted by FSIJ).

Is FSIJ the company you work for and are they willing to assign their
copyrights to the FSF?  Perhaps it might be better to discuss this
off-list?

Thanks,
Marco

Re: Networking status

[NIIBE Yutaka]

Thanks for your attention.

Marco Gerards wrote about my code:
> * Advantages
> - I assume it was tested a lot?
> - TCP+HTTP
> - DHCP, BOOTP and IPv6

* Disadvantages
  - non-modular implementation, assume ethernet
  - implementation which takes advantage of layer-violation

> After that we can have a look at your code, see how well TCP
> integrates.  What I am wondering is how complete it is, does it do
> retransmissions, slow start, congestion control, etc?

Don't expect much. :-)  It's a kind of hack.

TCP is not a separate module, but it's actually HTTP+TCP.  Client side
is assumed.  You know, retransmissions, slow start, congestion
control, or path MTU discovery, all the things are only (moslty)
required by server side.  I mean, my code doesn't implement this
because of no requirement.

Actually, before I have implemented it, I also had thought we need
such functionalities for TCP.  While coding it, I found that the
requirement for TCP is quite limited to download a kernel.

All that client side is needed is to send SYN, and

    GET /somewhere/kernel/image HTTP 1.x

and after that, it receive packets.  Receiving a packet, it sends ACK.
It is server side which need retransmission, or other things.

I say, it's a 'hack', because of it.  It's not full featured software,
but it works well (for the purpose).

> We should at least support TFTP because many people have a TFTP server
> anyways.  For example people use PXE to load GRUB from the network
> (TFTP), so it would be convenient to use TFTP as well for loading the
> kernel.

Yup, for GRUB, I see.

> But if you have a small TCP implementations that works well, be could
> have a look at how to create a module out of it.  It would also be
> very nice if NFS can be supported by GRUB 2 in that case, NFS is often
> available in the environments network booting is used and is capable
> of listing directory contents.

I have an experience of implementing NFS for bootloader, it was called
sh-boot (for SuperH architecture).  At that time, I took code from
etherboot, and it was not so difficult.

> But at first I want to have the core functionality so people can use
> it as the base of their code, for example for driver support.  At the
> moment TCP and HTTP, if possible, are not that important.

I see.  For GRUB, it's true.  HTTP boot has been (and *is*) important
for our project.

> Is FSIJ the company you work for and are they willing to assign their
> copyrights to the FSF?  Perhaps it might be better to discuss this
> off-list?

FSIJ, the Free Software Initiative of Japan, is non-profit
organization under Tokyo Metropolitan Government.  It's a sister
organization of FSFE, Free Software Foundation Europe.

So, please feel free to take any portion of code if you think it is
useful.

Currently, I'm not sure if you like my code, or my code is useful for
GRUB development.  I just want to support your hacking.

If you will carefully select features, you don't need to be afraid of
the Dragon, {TCP,UDP} / IPv4 or IPv6 protocol stack.  Because we don't
need full featured software, layer violation is your weapon, I'd say. :-)
-- 

Re: Networking status

[Marco Gerards]

NIIBE Yutaka <gniibe@m17n.org> writes:

>> After that we can have a look at your code, see how well TCP
>> integrates.  What I am wondering is how complete it is, does it do
>> retransmissions, slow start, congestion control, etc?
>
> Don't expect much. :-)  It's a kind of hack.
>
> TCP is not a separate module, but it's actually HTTP+TCP.  Client side
> is assumed.  You know, retransmissions, slow start, congestion
> control, or path MTU discovery, all the things are only (moslty)
> required by server side.  I mean, my code doesn't implement this
> because of no requirement.
>
> Actually, before I have implemented it, I also had thought we need
> such functionalities for TCP.  While coding it, I found that the
> requirement for TCP is quite limited to download a kernel.
>
> All that client side is needed is to send SYN, and
>
>     GET /somewhere/kernel/image HTTP 1.x
>
> and after that, it receive packets.  Receiving a packet, it sends ACK.
> It is server side which need retransmission, or other things.

What if the request does not fit well into a single packet?  I agree
that we just basic functionality, but with the requirement that it
still works properly according to the rules.

> I say, it's a 'hack', because of it.  It's not full featured software,
> but it works well (for the purpose).

Right.

>> But if you have a small TCP implementations that works well, be could
>> have a look at how to create a module out of it.  It would also be
>> very nice if NFS can be supported by GRUB 2 in that case, NFS is often
>> available in the environments network booting is used and is capable
>> of listing directory contents.
>
> I have an experience of implementing NFS for bootloader, it was called
> sh-boot (for SuperH architecture).  At that time, I took code from
> etherboot, and it was not so difficult.

Oh, cool.

>> But at first I want to have the core functionality so people can use
>> it as the base of their code, for example for driver support.  At the
>> moment TCP and HTTP, if possible, are not that important.
>
> I see.  For GRUB, it's true.  HTTP boot has been (and *is*) important
> for our project.

I can perfectly understand that.  It sucks to install a TFTP server at
a client. :-)

>> Is FSIJ the company you work for and are they willing to assign their
>> copyrights to the FSF?  Perhaps it might be better to discuss this
>> off-list?
>
> FSIJ, the Free Software Initiative of Japan, is non-profit
> organization under Tokyo Metropolitan Government.  It's a sister
> organization of FSFE, Free Software Foundation Europe.

Oh, I see.  I didn't know that; great to hear about this initiative.

> So, please feel free to take any portion of code if you think it is
> useful.
>
> Currently, I'm not sure if you like my code, or my code is useful for
> GRUB development.  I just want to support your hacking.

Sure.  We can look at it at a later time, if you'd like.  My first
priority is getting GRUB at a state it is working and can replace and
compete with GRUB Legacy.

> If you will carefully select features, you don't need to be afraid of
> the Dragon, {TCP,UDP} / IPv4 or IPv6 protocol stack.  Because we don't
> need full featured software, layer violation is your weapon, I'd say. :-)

heh :)

A bootloader has other requirements.  For example, in my IPv4
implementation, I just set the don't fragment bit so I don't have to
implement fragmentation.  Surely there are some other things we can do
like this.  Too bad I could not find an RFC specially for firmware and
bootloaders.

--
Marco