From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9A394EE14D3 for ; Wed, 6 Sep 2023 20:54:26 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qdzWr-0007f8-6t; Wed, 06 Sep 2023 16:53:37 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qdzWp-0007er-Fx for qemu-devel@nongnu.org; Wed, 06 Sep 2023 16:53:35 -0400 Received: from mail-wm1-x329.google.com ([2a00:1450:4864:20::329]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qdzWl-0004Yx-7p for qemu-devel@nongnu.org; Wed, 06 Sep 2023 16:53:34 -0400 Received: by mail-wm1-x329.google.com with SMTP id 5b1f17b1804b1-40061928e5aso2988975e9.3 for ; Wed, 06 Sep 2023 13:53:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1694033608; x=1694638408; darn=nongnu.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=ljeWiOBK1YRJqpbJLOZI5OE5Eu/gk5L+xw9UUP2Stp4=; b=NnZXxX8mD53QPf0TYNBQt2pZd4Ch4/5sxDIkfAl6o49A19R5F1MTSzHSlBZeMHRI1Z EbIxGEBmydN+P9fNluPNj9dWmgwqxaWYVxc22fxWpPi+nUGcienyrJZPZf7ES3eGPGt9 N6OXSkUDOlrrtFhjMQGIvFlvk7+iXwqBvIxHb66bjEKEhig+NsF9Ngp0KSCKE9mLeIPz VfZayt7BX9rVZBXLTUiUV38gmTpL05eU24L2bQkbIs87CFvgS5BUIa+SnYlVek3LAk6N viFFgzAWntqKYg+zcXoVS6bJ2cD2xhaDK0S3hQQUX4MNKKBkPJ8JWsvw6B4N4GtWmyHI t7BQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1694033608; x=1694638408; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=ljeWiOBK1YRJqpbJLOZI5OE5Eu/gk5L+xw9UUP2Stp4=; b=TcJ9nwmB3kbuFWiqFKn7c0geFRNu/c9vXYe9/E9W6u7p5cgS9/Z2AU8qx4j5ZFXVrc kyOYTYWCjdEf3QWpinakJxd7m7fxyr3xV+ju1NTIP4J0vUy6+37EbDktPc9tJg8bhP25 J55dtaFyPmtr7MUhyFBOAK2YlXHyYDqfFqDvrzrlM2PCjeSF/w1BLZ2G0w04GxnHHz9U mf4pRjoZgaE1hwXRBvWSA4asHAbKYiHwF1srbZQlOByEk0lZeCay9KEM4NIUPoszBEZu ae4TPBufK0w2/vgE40+qVOUwYrj562s/GM2FKi+fZfgYAUpaadocsdUz5oJlOYElVmfo uu/Q== X-Gm-Message-State: AOJu0Yx1ux+ezmrRCN0WrcwwPBMxk4IZBdAC2x87mu4ON6da4MVmPSRM IYFBpqX51BAFa64L07g4Q5vW7/jwXVSaf2uwwI3bEAXCP+I0IyiGGfsKqw== X-Google-Smtp-Source: AGHT+IHuWOmsSdQeVdS2BkFK1xjvYkGdPyPhBwwwm9Qzh2jk2oZNmBnDSgTceK8zXeK1dWQRnsUyU1YUZYYZy/ZbAiY= X-Received: by 2002:adf:ef83:0:b0:31d:1833:4130 with SMTP id d3-20020adfef83000000b0031d18334130mr3200290wro.71.1694033608091; Wed, 06 Sep 2023 13:53:28 -0700 (PDT) MIME-Version: 1.0 References: <20230626114916.45355529@mobian.usb.local> <20230626100819.vtkuuvzg376hktk2@begin> <20230720145415.w7s3ystkrf5gc66y@begin> In-Reply-To: From: Felix Wu Date: Wed, 6 Sep 2023 13:53:16 -0700 Message-ID: Subject: Re: Tips for local testing guestfwd To: Samuel Thibault Cc: Lukas Straub , qemu-devel@nongnu.org, Jason Wang Content-Type: multipart/alternative; boundary="0000000000007360490604b6ef55" Received-SPF: pass client-ip=2a00:1450:4864:20::329; envelope-from=flwu@google.com; helo=mail-wm1-x329.google.com X-Spam_score_int: -175 X-Spam_score: -17.6 X-Spam_bar: ----------------- X-Spam_report: (-17.6 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, ENV_AND_HDR_SPF_MATCH=-0.5, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, USER_IN_DEF_DKIM_WL=-7.5, USER_IN_DEF_SPF_WL=-7.5 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org --0000000000007360490604b6ef55 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi, I noticed why the chardev socket backend disconnects, and I would like to make this a RFC to see how I should fix it. Current scenario after boot-up: 1. tcp_chr_read_poll keeps polling the slirp_socket_can_recv, and slirp_socket_can_recv returns 0 since slirp_find_ctl_socket couldn't find the guestfwd socket. 2. The returned 0 in step 1 was assigned to the s->max_size (s is SocketChardev *), and the socket chardev handler won't read since readable size is 0. 3. When the 1st request is sent, the guestfwd socket is added into the slirp's socket list, instead of 0, tcp_chr_read_poll will return the result of sopreprbuf > 0. 4. tcp_chr_read reads the thing. 5. tcp_chr_read_poll still returns things > 0, which is the output of sopreprbuf. 6. tcp_chr_read reads the thing again, but there's nothing in the buffer, so it's unhappy, and closes the connection. 7. any follow-up requests won't be handled. These tcp_chr* functions are in fle [1], and slirp_* are in fle [2]. My questions: 1. Since this thing doesn't work on 2nd and later requests, I want to know how this thing is supposed to work, and to avoid asking people vaguely, I will provide my though following and please correct me if I am wrong: a. The state machine in chardev socket should maintain a connected state (s->state =3D=3D TCP_CHARDEV_STATE_CONNECTED), this means no change in [1]. b. slirp_socket_can_recv should return 0 once all data is read instead of outcome from sopreprbuf. This means I need to remove the socket or change its state to no file descriptor [3], namely somehow reset it. c. When a new request comes in, it will need to add the socket back to this slirp instance's socket list, populate its file descriptor, and establish the connection. b and c sounds convoluted so I want to check. 2. What is the outcome of sopreprbuf function [3]? Since it's returned to the tcp_chr_read_poll function, I thought it's the readable bytes in the socket, but in my test I noticed following thing: tcp_chr_read_poll_size : s->max_size: 132480 tcp_chr_read : size: 2076 tcp_chr_read_poll_size : s->max_size: 129600 tcp_chr_read : size: 0 Even there's not remaining things in the buffer (read size 0), it's still non-zero, and thus the read function keeps reading it until it becomes unhappy. Also, 132480-129600 =3D 2880 vs 2076, the read byte doesn't match. Either I need to go with the way in question 1, b.c. steps, or I don't need to delete the socket, but the sopreprbuf wasn't proper to be used there and I need to correct it. Also updated https://gitlab.com/qemu-project/qemu/-/issues/1835. Any feedback will be appreciated, thanks! Felix [1]. https://gitlab.com/qemu-project/qemu/-/blob/master/chardev/char-socket.c#L1= 41 [2]. https://gitlab.freedesktop.org/slirp/libslirp/-/blob/master/src/slirp.c#L15= 82 [3]. https://gitlab.freedesktop.org/slirp/libslirp/-/blob/master/src/socket.h#L2= 21 On Wed, Aug 23, 2023 at 10:27=E2=80=AFAM Felix Wu wrote: > Update on debugging this thing (already updated > https://gitlab.com/qemu-project/qemu/-/issues/1835): > I saw that `tcp_chr_free_connection` was called after the first response > being successfully sent: > ``` > > slirp_guestfwd_write guestfwd_write: size 80tcp_chr_write tcp_chr_write: = s->state:2tcp_chr_write tcp_chr_write: len:80qemu_chr_write_parameter len: = 80 // tracking qemu_chr_writeqemu_chr_write_res len: 80 // same thingtcp_ch= r_free_connection tcp_chr_free_connection: state: 2, changing it to disconn= ecttcp_chr_change_state tcp_chr_change_state: state: 2, next state: 0 // st= ate 2=3D=3Dconnected, 0=3D=3Ddisconnected. > > ``` > And after that, the state of `SocketChardev` remained disconnected, and > when the 2nd request came in, the `tcp_chr_write` dropped it directly. > Maybe this state machine should be reset after every connection? Not sure= . > > On Thu, Aug 17, 2023 at 11:58=E2=80=AFAM Felix Wu wrote= : > >> Hi Samuel, >> >> Thanks for the clarification! I missed the email so didn't reply in time= , >> but was able to figure it out. >> >> Hi everyone, >> IPv6 guestfwd works in my local test but it has a weird bug: if you send >> two requests, the first one gets the correct response, but the second on= e >> gets stuck. >> I am using a simple http server for this test, and just noticed this bug >> also exists in IPv4 guestfwd. I've documented it in >> https://gitlab.com/qemu-project/qemu/-/issues/1835. >> >> Just want to check if anyone has seen the same issue before. >> >> Thanks! Felix >> >> On Thu, Jul 20, 2023 at 7:54=E2=80=AFAM Samuel Thibault >> wrote: >> >>> Hello, >>> >>> Felix Wu, le mar. 18 juil. 2023 18:12:16 -0700, a ecrit: >>> > 02 =3D=3D SYN so it looks good. But both tcpdump and wireshark (looki= ng >>> into packet >>> > dump provided by QEMU invocation) >>> >>> Which packet dump? >>> >>> > I added multiple prints inside slirp and confirmed the ipv6 version o= f >>> [1] was >>> > reached. >>> > in tcp_output function [2], I got following print: >>> > qemu-system-aarch64: info: Slirp: AF_INET6 out dst ip =3D >>> > fdb5:481:10ce:0:8c41:aaff:fea9:f674, port =3D 52190 >>> > qemu-system-aarch64: info: Slirp: AF_INET6 out src ip =3D fec0::105, >>> port =3D 54322 >>> > It looks like there should be something being sent back to the guest, >>> >>> That's what it is. >>> >>> > unless my understanding of tcp_output is wrong. >>> >>> It looks so. >>> >>> > To understand the datapath of guestfwd better, I have the following >>> questions: >>> > 1. What's the meaning of tcp_input and tcp_output? My guess is the >>> following >>> > graph, but I would like to confirm. >>> >>> No, tcp_input is for packets that come from the guest, and tcp_output i= s >>> for packets that are send to the guest. So it's like that: >>> >>> > tcp_input write_cb host send() >>> > QEMU --------> slirp -----------> QEMU --------------------> host >>> > <-------- <--------- <----------------- >>> > tcp_output slirp_socket_recv host recv() >>> >>> > 2. I don't see port 6655 in the above process. How does slirp know >>> 6655 is the >>> > port that needs to be visited on the host side? >>> >>> Slirp itself *doesn't* know that port. The guestfwd piece just calls th= e >>> SlirpWriteCb when it has data coming from the guest. See the >>> documentation: >>> >>> /* Set up port forwarding between a port in the guest network and a >>> * callback that will receive the data coming from the port */ >>> SLIRP_EXPORT >>> int slirp_add_guestfwd(Slirp *slirp, SlirpWriteCb write_cb, void *opaqu= e, >>> struct in_addr *guest_addr, int guest_port); >>> >>> and >>> >>> /* This is called by the application for a guestfwd, to provide the dat= a >>> to be >>> * sent on the forwarded port */ >>> SLIRP_EXPORT >>> void slirp_socket_recv(Slirp *slirp, struct in_addr guest_addr, int >>> guest_port, >>> const uint8_t *buf, int size); >>> >>> Samuel >>> >> --0000000000007360490604b6ef55 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi,
I noticed why the chardev socket backend disconnec= ts, and I would like to make this a RFC to see how I should fix it.
Current scenario=C2=A0after boot-up:
  1. tcp_chr_rea= d_poll=C2=A0keeps polling the slirp_socket_can_recv, an= d slirp_socket_can_recv returns 0 since slirp_find_ctl_s= ocket couldn't find the guestfwd socket.
  2. The returned 0 in step 1 was assigned to the s->max_size (s is SocketChardev *), and the socket chardev handler won't read sin= ce readable size is 0.
  3. When the 1st request is sent, the guestfwd socket is added into the sli= rp's socket list, instead of 0, tcp_chr_read_poll will ret= urn the result of sopreprbuf > 0.
  4. tcp_chr_read reads the thing.
  5. tcp_chr_read_poll still returns things > 0, which is the ou= tput of sopreprbuf.
  6. tcp_chr_read reads the thing again, but there's nothing in= the buffer, so it's unhappy, and closes the connection.
  7. any follow-up requests won't be handled.
These=C2=A0<= span style=3D"font-family:monospace">tcp_chr*=C2=A0functions are in = fle [1], and=C2=A0slirp_*=C2=A0are in fle [2].

My questions:
1. = Since this thing doesn't=C2=A0work on 2nd and later requests, I want to= know how this thing is supposed to work, and to avoid asking people vaguel= y, I will provide my though following and please correct me if I am wrong:<= /div>
a. The state machine in chardev socket should maintain a connecte= d state (s->state =3D=3D TCP_CHARDEV_STATE_CONN= ECTED), this means no change in [1].
b.=C2=A0slirp_socket_can_recv should return 0 once all data is read instead of outcome from= =C2=A0sopreprbuf. This means I need = to remove the socket or change its state to no file descriptor [3], namely = somehow reset it.
c. When a new request comes in, it will need to= add the socket back to this slirp instance's socket list, populate its= file descriptor, and establish the connection.

b = and c sounds convoluted so I want to check.

2. Wha= t is the outcome of sopreprbuf=C2=A0functio= n [3]?
Since it's returned to the=C2=A0tcp_chr_read_poll f= unction, I thought it's the readable bytes in the socket, but in my tes= t I noticed following thing:

tcp_chr_read_poll_size : s-&g= t;max_size: 132480
tcp_chr_read : size: 2076
tcp_chr_read_poll_size : s->max_size: 129600
<= span style=3D"font-family:monospace">tcp_chr_read : size: 0

Even there's not remaini= ng things in the buffer (read size 0), it's still non-zero, and thus th= e read function keeps reading it until it becomes unhappy.
Also,= =C2=A0132480-129600 =3D 2880 vs 2076,= the read byte doesn't match.

E= ither I need to go with the way in question 1, b.c. steps, or I don't n= eed to delete the socket, but the=C2=A0sopreprbuf wasn't prop= er to be used there and I need to correct it.

On Wed, Aug 23, 2023 at 10:27=E2=80=AFAM Felix Wu <flwu@google.com> wrote:
Update on deb= ugging=C2=A0this thing (already updated=C2=A0https://gitlab.com/qemu-project/qemu/-/issues/1835):
I= saw that `tcp_chr_free_connection` was called after the first response bei= ng successfully sent:
```
slirp_guestfwd_w=
rite guestfwd_write: size 80
tcp_chr_wri=
te tcp_chr_write: s->state:2
tcp_chr_wri=
te tcp_chr_write: len:80
qemu_chr_wr=
ite_parameter len: 80 // tracking qemu_chr_write
qemu_chr_wr=
ite_res len: 80 // same thing
tcp_chr_fre=
e_connection tcp_chr_free_connection: state: 2, changing it to disconnect
tcp_chr_cha=
nge_state tcp_chr_change_state: state: 2, next state: 0 // state 2=3D=3Dcon=
nected, 0=3D=3Ddisconnected.
```
And after that, the state of `SocketChardev`= remained disconnected, and when the 2nd request came in, the `tcp_chr_writ= e` dropped it directly.
Maybe this state machine should be reset = after every connection? Not sure.

On Thu, Aug 17, 2023 at 11:58=E2=80= =AFAM Felix Wu <flw= u@google.com> wrote:
Hi Samuel,

Thanks for the c= larification! I missed the email so didn't reply in time, but was able = to figure it out.

Hi everyone,
IPv6 gues= tfwd works in my local test but it has a weird bug: if you send two request= s, the first one gets the correct response, but the second one gets stuck.<= /div>
I am using a simple http server for this test, and just noticed t= his bug also=C2=A0exists in=C2=A0IPv4 guestfwd. I've documented it in= =C2=A0https://gitlab.com/qemu-pr= oject/qemu/-/issues/1835.

Just want to check i= f anyone has seen the same issue before.

Thanks! F= elix

On Thu, Jul 20, 2023 at 7:54=E2=80=AFAM Samuel Thibault <samuel.thibault@gnu= .org> wrote:
Hello,

Felix Wu, le mar. 18 juil. 2023 18:12:16 -0700, a ecrit:
> 02 =3D=3D SYN so it looks good. But both tcpdump and wireshark (lookin= g into packet
> dump provided by QEMU invocation)

Which packet dump?

> I added multiple prints inside slirp and confirmed the ipv6 version of= [1] was
> reached.
> in tcp_output function [2], I got following print:
> qemu-system-aarch64: info: Slirp: AF_INET6 out dst ip =3D
> fdb5:481:10ce:0:8c41:aaff:fea9:f674, port =3D 52190
> qemu-system-aarch64: info: Slirp: AF_INET6 out src ip =3D fec0::105, p= ort =3D 54322
> It looks like there should be something being sent back to the guest,<= br>
That's what it is.

> unless my understanding of tcp_output is wrong.

It looks so.

> To understand the datapath of guestfwd better, I have the following qu= estions:
> 1. What's the meaning of tcp_input and tcp_output? My guess is the= following
> graph, but I would like to confirm.

No, tcp_input is for packets that come from the guest, and tcp_output is for packets that are send to the guest. So it's like that:

> =C2=A0 =C2=A0 =C2=A0=C2=A0 =C2=A0tcp_input=C2=A0 =C2=A0 write_cb=C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 host send()
> QEMU --------> slirp -----------> QEMU --------------------> = host
> =C2=A0 =C2=A0 <-------- =C2=A0=C2=A0 =C2=A0 =C2=A0 <---------=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<-----------------
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0tcp_output =C2=A0slirp_socket_recv= =C2=A0 =C2=A0 host recv()

> 2.=C2=A0I don't see port 6655 in the above=C2=A0process. How does = slirp know 6655 is the
> port that needs to be visited on the host side?

Slirp itself *doesn't* know that port. The guestfwd piece just calls th= e
SlirpWriteCb when it has data coming from the guest. See the
documentation:

/* Set up port forwarding between a port in the guest network and a
=C2=A0* callback that will receive the data coming from the port */
SLIRP_EXPORT
int slirp_add_guestfwd(Slirp *slirp, SlirpWriteCb write_cb, void *opaque, =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0struct in_addr *guest_addr, int guest_port);

and

/* This is called by the application for a guestfwd, to provide the data to= be
=C2=A0* sent on the forwarded port */
SLIRP_EXPORT
void slirp_socket_recv(Slirp *slirp, struct in_addr guest_addr, int guest_p= ort,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0const uint8_t *buf, int size);

Samuel
--0000000000007360490604b6ef55--