Tips for local testing guestfwd

Tips for local testing guestfwd

[Felix Wu]

[-- Attachment #1: Type: text/plain, Size: 1318 bytes --]

Hi all,

TL,DR: I am working on QEMU ipv6 guestfwd feature and finished coding, and
would like to learn the best practice to test it.
Context: in slirp side this task is tracking by [1].
Currently, I have done following:
i. made char parse + guestfwd functions happy with ipv6 address.
ii. enabled debug print and made sure the ip and port are inserted into the
forward list in libslirp.
To sufficiently verify it's working, I do have three questions:
1. I want to forward a port in the guest (OS in QEMU) to a port 22 on the
host OS, and ssh from guest back to host,
does this sound reasonable? If this is not a good idea, what method is
recommended?
2. I want to understand what ip I should use. Currently I have following
formats for the QEMU invocation in ipv6:
```
guestfwd=tcp:[::1]:1234-tcp:[my:host:ip:from:ifconfig]:22
```
I know the general form is `guestfwd=tcp:server:port-dev`, where
server:port is for guest, dev is for host.
Adding [] in my implementation will let QEMU know it's ipv6.
Is the aforementioned invocation correct? Or in this case [::1] is the
local host address and I should put qemu address
for it instead?
3. Is there a default ipv6 address for QEMU instance? I think I need it in
the invocation.

Thanks in advance! Felix

[1]. https://gitlab.freedesktop.org/slirp/libslirp/-/issues/67

[-- Attachment #2: Type: text/html, Size: 1788 bytes --]

Re: Tips for local testing guestfwd

[Lukas Straub]

[-- Attachment #1: Type: text/plain, Size: 1522 bytes --]

CC'ing SLIRP and net maintainer.

On Sun, 25 Jun 2023 22:58:36 -0700
Felix Wu <flwu@google.com> wrote:

> Hi all,
> 
> TL,DR: I am working on QEMU ipv6 guestfwd feature and finished coding, and
> would like to learn the best practice to test it.
> Context: in slirp side this task is tracking by [1].
> Currently, I have done following:
> i. made char parse + guestfwd functions happy with ipv6 address.
> ii. enabled debug print and made sure the ip and port are inserted into the
> forward list in libslirp.
> To sufficiently verify it's working, I do have three questions:
> 1. I want to forward a port in the guest (OS in QEMU) to a port 22 on the
> host OS, and ssh from guest back to host,
> does this sound reasonable? If this is not a good idea, what method is
> recommended?
> 2. I want to understand what ip I should use. Currently I have following
> formats for the QEMU invocation in ipv6:
> ```
> guestfwd=tcp:[::1]:1234-tcp:[my:host:ip:from:ifconfig]:22
> ```
> I know the general form is `guestfwd=tcp:server:port-dev`, where
> server:port is for guest, dev is for host.
> Adding [] in my implementation will let QEMU know it's ipv6.
> Is the aforementioned invocation correct? Or in this case [::1] is the
> local host address and I should put qemu address
> for it instead?
> 3. Is there a default ipv6 address for QEMU instance? I think I need it in
> the invocation.
> 
> Thanks in advance! Felix
> 
> [1]. https://gitlab.freedesktop.org/slirp/libslirp/-/issues/67


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Tips for local testing guestfwd

[Samuel Thibault]

Hello,

Felix Wu <flwu@google.com> wrote:
> 2. I want to understand what ip I should use. Currently I have following
> formats for the QEMU invocation in ipv6:
> ```
> guestfwd=tcp:[::1]:1234-tcp:[my:host:ip:from:ifconfig]:22
> ```
> I know the general form is `guestfwd=tcp:server:port-dev`, where
> server:port is for guest,

Yes, the address to be used within the guest network. So it needs to be
within the guest network.

> Is the aforementioned invocation correct?

No, because ::1 isn't in the guest network.

> Or in this case [::1] is the local host address and I should put qemu
> address for it instead?

You can use whatever IP you want, as long as it's in the guest network.
e.g. [fec0::1234] if you're with the default fec0::/64 network.

> 3. Is there a default ipv6 address for QEMU instance? I think I need it in
> the invocation.

By default it's address 2 within the prefix, i.e. fec0::2 with the
default fec0::/64 network.

Samuel

Re: Tips for local testing guestfwd

[Felix Wu]

[-- Attachment #1: Type: text/plain, Size: 3422 bytes --]

Hi all,

I am continuing debugging the ipv6 guestfwd feature, and I would like to
understand the behavior of slirp better.

Progress I've made:
Let QEMU take parameter like following:
guestfwd=tcp:[fec0::105]:54322-tcp:[::1]:6655
For slirp side, I basically searched for the appearance of gfwd_list and
made all code traverse the fwd list compatible with ipv6.
With these change, now I can see the packets coming out of the guest OS to
the assigned guest server port via tcpdump:
```
00:38:18.831831 IP6 fdb5:481:10ce:0:8c41:aaff:fea9:f674.52190 >
fec0::105.54322: tcp 0
        0x0000:  600a 1f94 0028 0640 fdb5 0481 10ce 0000  `....(.@........
        0x0010:  8c41 aaff fea9 f674 fec0 0000 0000 0000  .A.....t........
        0x0020:  0000 0000 0000 0105 cbde d432 df6d 8332  ...........2.m.2
        0x0030:  0000 0000 a0*02* fd20 535f 0000 0204 05a0  ........S_......
        0x0040:  0402 080a b87b fd3b 0000 0000 0103 0307  .....{.;........
```
02 == SYN so it looks good. But both tcpdump and wireshark (looking into
packet dump provided by QEMU invocation) didn't see any response and this
packet never reached the host.
I added multiple prints inside slirp and confirmed the ipv6 version of [1]
was reached.
in tcp_output function [2], I got following print:
qemu-system-aarch64: info: Slirp: AF_INET6 out dst ip =
fdb5:481:10ce:0:8c41:aaff:fea9:f674, port = 52190
qemu-system-aarch64: info: Slirp: AF_INET6 out src ip = fec0::105, port =
54322
It looks like there should be something being sent back to the guest,
unless my understanding of tcp_output is wrong.

To understand the datapath of guestfwd better, I have the following
questions:
1. What's the meaning of tcp_input and tcp_output? My guess is the
following graph, but I would like to confirm.
           tcp_input     tcp_output
QEMU --------> slirp ----------> host
            <--------       <----------
         tcp_output   tcp_input

2. I don't see port 6655 in the above process. How does slirp know 6655 is
the port that needs to be visited on the host side?

Thanks in advance, Felix
[1].
https://gitlab.freedesktop.org/slirp/libslirp/-/blob/master/src/tcp_input.c#L630
[2].
https://gitlab.freedesktop.org/slirp/libslirp/-/blob/master/src/tcp_output.c#L477


On Mon, Jun 26, 2023 at 3:08 AM Samuel Thibault <samuel.thibault@gnu.org>
wrote:

> Hello,
>
> Felix Wu <flwu@google.com> wrote:
> > 2. I want to understand what ip I should use. Currently I have following
> > formats for the QEMU invocation in ipv6:
> > ```
> > guestfwd=tcp:[::1]:1234-tcp:[my:host:ip:from:ifconfig]:22
> > ```
> > I know the general form is `guestfwd=tcp:server:port-dev`, where
> > server:port is for guest,
>
> Yes, the address to be used within the guest network. So it needs to be
> within the guest network.
>
> > Is the aforementioned invocation correct?
>
> No, because ::1 isn't in the guest network.
>
> > Or in this case [::1] is the local host address and I should put qemu
> > address for it instead?
>
> You can use whatever IP you want, as long as it's in the guest network.
> e.g. [fec0::1234] if you're with the default fec0::/64 network.
>
> > 3. Is there a default ipv6 address for QEMU instance? I think I need it
> in
> > the invocation.
>
> By default it's address 2 within the prefix, i.e. fec0::2 with the
> default fec0::/64 network.
>
> Samuel
>

[-- Attachment #2: Type: text/html, Size: 7496 bytes --]

Re: Tips for local testing guestfwd

[Samuel Thibault]

Hello,

Felix Wu, le mar. 18 juil. 2023 18:12:16 -0700, a ecrit:
> 02 == SYN so it looks good. But both tcpdump and wireshark (looking into packet
> dump provided by QEMU invocation)

Which packet dump?

> I added multiple prints inside slirp and confirmed the ipv6 version of [1] was
> reached.
> in tcp_output function [2], I got following print:
> qemu-system-aarch64: info: Slirp: AF_INET6 out dst ip =
> fdb5:481:10ce:0:8c41:aaff:fea9:f674, port = 52190
> qemu-system-aarch64: info: Slirp: AF_INET6 out src ip = fec0::105, port = 54322
> It looks like there should be something being sent back to the guest,

That's what it is.

> unless my understanding of tcp_output is wrong.

It looks so.

> To understand the datapath of guestfwd better, I have the following questions:
> 1. What's the meaning of tcp_input and tcp_output? My guess is the following
> graph, but I would like to confirm.

No, tcp_input is for packets that come from the guest, and tcp_output is
for packets that are send to the guest. So it's like that:

>         tcp_input    write_cb          host send()
> QEMU --------> slirp -----------> QEMU --------------------> host
>     <--------        <---------         <-----------------
>          tcp_output  slirp_socket_recv    host recv()

> 2. I don't see port 6655 in the above process. How does slirp know 6655 is the
> port that needs to be visited on the host side?

Slirp itself *doesn't* know that port. The guestfwd piece just calls the
SlirpWriteCb when it has data coming from the guest. See the
documentation:

/* Set up port forwarding between a port in the guest network and a
 * callback that will receive the data coming from the port */
SLIRP_EXPORT
int slirp_add_guestfwd(Slirp *slirp, SlirpWriteCb write_cb, void *opaque,
                       struct in_addr *guest_addr, int guest_port);

and 

/* This is called by the application for a guestfwd, to provide the data to be
 * sent on the forwarded port */
SLIRP_EXPORT
void slirp_socket_recv(Slirp *slirp, struct in_addr guest_addr, int guest_port,
                       const uint8_t *buf, int size);

Samuel

Re: Tips for local testing guestfwd

[Felix Wu]

[-- Attachment #1: Type: text/plain, Size: 2940 bytes --]

Hi Samuel,

Thanks for the clarification! I missed the email so didn't reply in time,
but was able to figure it out.

Hi everyone,
IPv6 guestfwd works in my local test but it has a weird bug: if you send
two requests, the first one gets the correct response, but the second one
gets stuck.
I am using a simple http server for this test, and just noticed this bug
also exists in IPv4 guestfwd. I've documented it in
https://gitlab.com/qemu-project/qemu/-/issues/1835.

Just want to check if anyone has seen the same issue before.

Thanks! Felix

On Thu, Jul 20, 2023 at 7:54 AM Samuel Thibault <samuel.thibault@gnu.org>
wrote:

> Hello,
>
> Felix Wu, le mar. 18 juil. 2023 18:12:16 -0700, a ecrit:
> > 02 == SYN so it looks good. But both tcpdump and wireshark (looking into
> packet
> > dump provided by QEMU invocation)
>
> Which packet dump?
>
> > I added multiple prints inside slirp and confirmed the ipv6 version of
> [1] was
> > reached.
> > in tcp_output function [2], I got following print:
> > qemu-system-aarch64: info: Slirp: AF_INET6 out dst ip =
> > fdb5:481:10ce:0:8c41:aaff:fea9:f674, port = 52190
> > qemu-system-aarch64: info: Slirp: AF_INET6 out src ip = fec0::105, port
> = 54322
> > It looks like there should be something being sent back to the guest,
>
> That's what it is.
>
> > unless my understanding of tcp_output is wrong.
>
> It looks so.
>
> > To understand the datapath of guestfwd better, I have the following
> questions:
> > 1. What's the meaning of tcp_input and tcp_output? My guess is the
> following
> > graph, but I would like to confirm.
>
> No, tcp_input is for packets that come from the guest, and tcp_output is
> for packets that are send to the guest. So it's like that:
>
> >         tcp_input    write_cb          host send()
> > QEMU --------> slirp -----------> QEMU --------------------> host
> >     <--------        <---------         <-----------------
> >          tcp_output  slirp_socket_recv    host recv()
>
> > 2. I don't see port 6655 in the above process. How does slirp know 6655
> is the
> > port that needs to be visited on the host side?
>
> Slirp itself *doesn't* know that port. The guestfwd piece just calls the
> SlirpWriteCb when it has data coming from the guest. See the
> documentation:
>
> /* Set up port forwarding between a port in the guest network and a
>  * callback that will receive the data coming from the port */
> SLIRP_EXPORT
> int slirp_add_guestfwd(Slirp *slirp, SlirpWriteCb write_cb, void *opaque,
>                        struct in_addr *guest_addr, int guest_port);
>
> and
>
> /* This is called by the application for a guestfwd, to provide the data
> to be
>  * sent on the forwarded port */
> SLIRP_EXPORT
> void slirp_socket_recv(Slirp *slirp, struct in_addr guest_addr, int
> guest_port,
>                        const uint8_t *buf, int size);
>
> Samuel
>

[-- Attachment #2: Type: text/html, Size: 3701 bytes --]

Re: Tips for local testing guestfwd

[Felix Wu]

[-- Attachment #1: Type: text/plain, Size: 3969 bytes --]

Update on debugging this thing (already updated
https://gitlab.com/qemu-project/qemu/-/issues/1835):
I saw that `tcp_chr_free_connection` was called after the first response
being successfully sent:
```

slirp_guestfwd_write guestfwd_write: size 80tcp_chr_write
tcp_chr_write: s->state:2tcp_chr_write tcp_chr_write:
len:80qemu_chr_write_parameter len: 80 // tracking
qemu_chr_writeqemu_chr_write_res len: 80 // same
thingtcp_chr_free_connection tcp_chr_free_connection: state: 2,
changing it to disconnecttcp_chr_change_state tcp_chr_change_state:
state: 2, next state: 0 // state 2==connected, 0==disconnected.

```
And after that, the state of `SocketChardev` remained disconnected, and
when the 2nd request came in, the `tcp_chr_write` dropped it directly.
Maybe this state machine should be reset after every connection? Not sure.

On Thu, Aug 17, 2023 at 11:58 AM Felix Wu <flwu@google.com> wrote:

> Hi Samuel,
>
> Thanks for the clarification! I missed the email so didn't reply in time,
> but was able to figure it out.
>
> Hi everyone,
> IPv6 guestfwd works in my local test but it has a weird bug: if you send
> two requests, the first one gets the correct response, but the second one
> gets stuck.
> I am using a simple http server for this test, and just noticed this bug
> also exists in IPv4 guestfwd. I've documented it in
> https://gitlab.com/qemu-project/qemu/-/issues/1835.
>
> Just want to check if anyone has seen the same issue before.
>
> Thanks! Felix
>
> On Thu, Jul 20, 2023 at 7:54 AM Samuel Thibault <samuel.thibault@gnu.org>
> wrote:
>
>> Hello,
>>
>> Felix Wu, le mar. 18 juil. 2023 18:12:16 -0700, a ecrit:
>> > 02 == SYN so it looks good. But both tcpdump and wireshark (looking
>> into packet
>> > dump provided by QEMU invocation)
>>
>> Which packet dump?
>>
>> > I added multiple prints inside slirp and confirmed the ipv6 version of
>> [1] was
>> > reached.
>> > in tcp_output function [2], I got following print:
>> > qemu-system-aarch64: info: Slirp: AF_INET6 out dst ip =
>> > fdb5:481:10ce:0:8c41:aaff:fea9:f674, port = 52190
>> > qemu-system-aarch64: info: Slirp: AF_INET6 out src ip = fec0::105, port
>> = 54322
>> > It looks like there should be something being sent back to the guest,
>>
>> That's what it is.
>>
>> > unless my understanding of tcp_output is wrong.
>>
>> It looks so.
>>
>> > To understand the datapath of guestfwd better, I have the following
>> questions:
>> > 1. What's the meaning of tcp_input and tcp_output? My guess is the
>> following
>> > graph, but I would like to confirm.
>>
>> No, tcp_input is for packets that come from the guest, and tcp_output is
>> for packets that are send to the guest. So it's like that:
>>
>> >         tcp_input    write_cb          host send()
>> > QEMU --------> slirp -----------> QEMU --------------------> host
>> >     <--------        <---------         <-----------------
>> >          tcp_output  slirp_socket_recv    host recv()
>>
>> > 2. I don't see port 6655 in the above process. How does slirp know 6655
>> is the
>> > port that needs to be visited on the host side?
>>
>> Slirp itself *doesn't* know that port. The guestfwd piece just calls the
>> SlirpWriteCb when it has data coming from the guest. See the
>> documentation:
>>
>> /* Set up port forwarding between a port in the guest network and a
>>  * callback that will receive the data coming from the port */
>> SLIRP_EXPORT
>> int slirp_add_guestfwd(Slirp *slirp, SlirpWriteCb write_cb, void *opaque,
>>                        struct in_addr *guest_addr, int guest_port);
>>
>> and
>>
>> /* This is called by the application for a guestfwd, to provide the data
>> to be
>>  * sent on the forwarded port */
>> SLIRP_EXPORT
>> void slirp_socket_recv(Slirp *slirp, struct in_addr guest_addr, int
>> guest_port,
>>                        const uint8_t *buf, int size);
>>
>> Samuel
>>
>

[-- Attachment #2: Type: text/html, Size: 5602 bytes --]

Re: Tips for local testing guestfwd

[Felix Wu]

[-- Attachment #1: Type: text/plain, Size: 7138 bytes --]

Hi,
I noticed why the chardev socket backend disconnects, and I would like to
make this a RFC to see how I should fix it.
Current scenario after boot-up:

   1. tcp_chr_read_poll keeps polling the slirp_socket_can_recv, and
   slirp_socket_can_recv returns 0 since slirp_find_ctl_socket couldn't
   find the guestfwd socket.
   2. The returned 0 in step 1 was assigned to the s->max_size (s is
SocketChardev
   *), and the socket chardev handler won't read since readable size is 0.
   3. When the 1st request is sent, the guestfwd socket is added into the
   slirp's socket list, instead of 0, tcp_chr_read_poll will return the
   result of sopreprbuf > 0.
   4. tcp_chr_read reads the thing.
   5. tcp_chr_read_poll still returns things > 0, which is the output of
   sopreprbuf.
   6. tcp_chr_read reads the thing again, but there's nothing in the
   buffer, so it's unhappy, and closes the connection.
   7. any follow-up requests won't be handled.

These tcp_chr* functions are in fle [1], and slirp_* are in fle [2].

My questions:
1. Since this thing doesn't work on 2nd and later requests, I want to know
how this thing is supposed to work, and to avoid asking people vaguely, I
will provide my though following and please correct me if I am wrong:
a. The state machine in chardev socket should maintain a connected
state (s->state
== TCP_CHARDEV_STATE_CONNECTED), this means no change in [1].
b. slirp_socket_can_recv should return 0 once all data is read instead of
outcome from sopreprbuf. This means I need to remove the socket or change
its state to no file descriptor [3], namely somehow reset it.
c. When a new request comes in, it will need to add the socket back to this
slirp instance's socket list, populate its file descriptor, and establish
the connection.

b and c sounds convoluted so I want to check.

2. What is the outcome of sopreprbuf function [3]?
Since it's returned to the tcp_chr_read_poll function, I thought it's the
readable bytes in the socket, but in my test I noticed following thing:

tcp_chr_read_poll_size : s->max_size: 132480
tcp_chr_read : size: 2076
tcp_chr_read_poll_size : s->max_size: 129600
tcp_chr_read : size: 0

Even there's not remaining things in the buffer (read size 0), it's still
non-zero, and thus the read function keeps reading it until it becomes
unhappy.
Also, 132480-129600 = 2880 vs 2076, the read byte doesn't match.

Either I need to go with the way in question 1, b.c. steps, or I don't need
to delete the socket, but the sopreprbuf wasn't proper to be used there and
I need to correct it.
Also updated https://gitlab.com/qemu-project/qemu/-/issues/1835.

Any feedback will be appreciated, thanks!
Felix

[1].
https://gitlab.com/qemu-project/qemu/-/blob/master/chardev/char-socket.c#L141
[2].
https://gitlab.freedesktop.org/slirp/libslirp/-/blob/master/src/slirp.c#L1582
[3].
https://gitlab.freedesktop.org/slirp/libslirp/-/blob/master/src/socket.h#L221

On Wed, Aug 23, 2023 at 10:27 AM Felix Wu <flwu@google.com> wrote:

> Update on debugging this thing (already updated
> https://gitlab.com/qemu-project/qemu/-/issues/1835):
> I saw that `tcp_chr_free_connection` was called after the first response
> being successfully sent:
> ```
>
> slirp_guestfwd_write guestfwd_write: size 80tcp_chr_write tcp_chr_write: s->state:2tcp_chr_write tcp_chr_write: len:80qemu_chr_write_parameter len: 80 // tracking qemu_chr_writeqemu_chr_write_res len: 80 // same thingtcp_chr_free_connection tcp_chr_free_connection: state: 2, changing it to disconnecttcp_chr_change_state tcp_chr_change_state: state: 2, next state: 0 // state 2==connected, 0==disconnected.
>
> ```
> And after that, the state of `SocketChardev` remained disconnected, and
> when the 2nd request came in, the `tcp_chr_write` dropped it directly.
> Maybe this state machine should be reset after every connection? Not sure.
>
> On Thu, Aug 17, 2023 at 11:58 AM Felix Wu <flwu@google.com> wrote:
>
>> Hi Samuel,
>>
>> Thanks for the clarification! I missed the email so didn't reply in time,
>> but was able to figure it out.
>>
>> Hi everyone,
>> IPv6 guestfwd works in my local test but it has a weird bug: if you send
>> two requests, the first one gets the correct response, but the second one
>> gets stuck.
>> I am using a simple http server for this test, and just noticed this bug
>> also exists in IPv4 guestfwd. I've documented it in
>> https://gitlab.com/qemu-project/qemu/-/issues/1835.
>>
>> Just want to check if anyone has seen the same issue before.
>>
>> Thanks! Felix
>>
>> On Thu, Jul 20, 2023 at 7:54 AM Samuel Thibault <samuel.thibault@gnu.org>
>> wrote:
>>
>>> Hello,
>>>
>>> Felix Wu, le mar. 18 juil. 2023 18:12:16 -0700, a ecrit:
>>> > 02 == SYN so it looks good. But both tcpdump and wireshark (looking
>>> into packet
>>> > dump provided by QEMU invocation)
>>>
>>> Which packet dump?
>>>
>>> > I added multiple prints inside slirp and confirmed the ipv6 version of
>>> [1] was
>>> > reached.
>>> > in tcp_output function [2], I got following print:
>>> > qemu-system-aarch64: info: Slirp: AF_INET6 out dst ip =
>>> > fdb5:481:10ce:0:8c41:aaff:fea9:f674, port = 52190
>>> > qemu-system-aarch64: info: Slirp: AF_INET6 out src ip = fec0::105,
>>> port = 54322
>>> > It looks like there should be something being sent back to the guest,
>>>
>>> That's what it is.
>>>
>>> > unless my understanding of tcp_output is wrong.
>>>
>>> It looks so.
>>>
>>> > To understand the datapath of guestfwd better, I have the following
>>> questions:
>>> > 1. What's the meaning of tcp_input and tcp_output? My guess is the
>>> following
>>> > graph, but I would like to confirm.
>>>
>>> No, tcp_input is for packets that come from the guest, and tcp_output is
>>> for packets that are send to the guest. So it's like that:
>>>
>>> >         tcp_input    write_cb          host send()
>>> > QEMU --------> slirp -----------> QEMU --------------------> host
>>> >     <--------        <---------         <-----------------
>>> >          tcp_output  slirp_socket_recv    host recv()
>>>
>>> > 2. I don't see port 6655 in the above process. How does slirp know
>>> 6655 is the
>>> > port that needs to be visited on the host side?
>>>
>>> Slirp itself *doesn't* know that port. The guestfwd piece just calls the
>>> SlirpWriteCb when it has data coming from the guest. See the
>>> documentation:
>>>
>>> /* Set up port forwarding between a port in the guest network and a
>>>  * callback that will receive the data coming from the port */
>>> SLIRP_EXPORT
>>> int slirp_add_guestfwd(Slirp *slirp, SlirpWriteCb write_cb, void *opaque,
>>>                        struct in_addr *guest_addr, int guest_port);
>>>
>>> and
>>>
>>> /* This is called by the application for a guestfwd, to provide the data
>>> to be
>>>  * sent on the forwarded port */
>>> SLIRP_EXPORT
>>> void slirp_socket_recv(Slirp *slirp, struct in_addr guest_addr, int
>>> guest_port,
>>>                        const uint8_t *buf, int size);
>>>
>>> Samuel
>>>
>>

[-- Attachment #2: Type: text/html, Size: 10931 bytes --]