From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DA163C3271E for ; Mon, 8 Jul 2024 05:18:08 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sQgkp-0000iD-So; Mon, 08 Jul 2024 01:17:35 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sQgkn-0000hb-Hg for qemu-devel@nongnu.org; Mon, 08 Jul 2024 01:17:33 -0400 Received: from mail-oa1-x29.google.com ([2001:4860:4864:20::29]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1sQgkk-0001QI-1x for qemu-devel@nongnu.org; Mon, 08 Jul 2024 01:17:33 -0400 Received: by mail-oa1-x29.google.com with SMTP id 586e51a60fabf-25e24900a82so1977151fac.3 for ; Sun, 07 Jul 2024 22:17:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=smartx-com.20230601.gappssmtp.com; s=20230601; t=1720415844; x=1721020644; darn=nongnu.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=hYU6Ezi+RQGxoXLwHHQ+rOxM2scJghg6+JA4TjSTHF4=; b=ZYunbyd1f9CoK1K43dlNS/jQ0RMLrszqgJ1kfrWDk1ENOhoxKdXrUblQiwK0rb2oSI AsjQ68cdXdjS7ZZdYHhCEOnA8zWq1JmE5UwSb8rr+Jzjgk8txDqRFmKmpqdoxG1uubO6 /3l8nk4++nwQbLJy328LAMS0SGg828rqdBj94Un22rzYTL+mxWBbW4lVQKo67ap7gyD+ 5UqZUaEmRlGaS8YYZNYWBd/7isCGQlxb8aUe8LleOSJzq1/gbbhSbjaZiGy0csCNdgwx zQ/H6UYGTslUjxC/r48TD/u4N/xRcbzjUmdLGWtWQZtQtdtD8aob/aLnajWykJ0ACkBh i+cw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720415844; x=1721020644; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=hYU6Ezi+RQGxoXLwHHQ+rOxM2scJghg6+JA4TjSTHF4=; b=ThpFSvk7v5c0mriJKoCYGA1RTbln73GGi46gQbw7AZqM2nUH+hzVGL7HHpAQaBxUvn qQc50Fo7r2IAH2XMAy+yGpvNTv2OTgppBhC+N8FBUlMVwtE1F+x2edTCWNORRfRz/Qh7 Li4vZg8TSD8jb1z1QWRl23IA2RImaPRB8QTakzy3Slg2H4xlgGgF4IUZmwUjIHRFRzWo g5EEikNgvYisw7xvKGjjqOatcrojIDvs0gQgVwOa8R593Y6vHN8my69hBzxPy15OyAvG DUoPkhUvGiu6oOLj4WwLqCwjS9ozpHaot191XEwHuggkq+n8ATkJ/jWSOCj1TMvh8dwe MAMQ== X-Gm-Message-State: AOJu0YyZM67OFN7cwhO+1o2m26REJHsGoEBT+H0Nm5o0TZt6vMgatFSh Fxrd1nR5ru2qodZlEVtS5QqidMbP5a8Sdyuh9CLi7KT2WhE92MZM9vkuZI0YOrWcs83Z99W2mxM eOltc9jN2CTVoom+G/jOWO9O5m9vK56EVzs7s3A== X-Google-Smtp-Source: AGHT+IGAHqvWLrSXy4rlSpHVUb2pnILjE/TZvAskqYVQpH3hM5A7unsa3Z51ijw/k3pI+ZEph9v+Bt7/YWWNNb/BQz8= X-Received: by 2002:a05:6871:24d5:b0:25e:65f:a530 with SMTP id 586e51a60fabf-25e2bea6836mr8839090fac.41.1720415843267; Sun, 07 Jul 2024 22:17:23 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Yong Huang Date: Mon, 8 Jul 2024 13:17:07 +0800 Message-ID: Subject: Re: [PATCH] e1000: Fix the unexpected assumption that the receive buffer is full To: Jason Wang Cc: qemu-devel@nongnu.org Content-Type: multipart/alternative; boundary="000000000000345123061cb58738" Received-SPF: pass client-ip=2001:4860:4864:20::29; envelope-from=yong.huang@smartx.com; helo=mail-oa1-x29.google.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org --000000000000345123061cb58738 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Mon, Jul 8, 2024 at 11:21=E2=80=AFAM Jason Wang wr= ote: > On Sat, Jul 6, 2024 at 4:30=E2=80=AFAM Hyman Huang wrote: > > > > Unexpected work by certain Windows guests equipped with the e1000 > > interface can cause the network to go down and never come back up > > again unless the guest's interface is reset. > > > > To reproduce the failure: > > 1. Set up two guests with a Windows 2016 or 2019 server operating > > system. > > I vaguely remember e1000 support for Windows has been deprecated for > several years... > > That's why e1000e or igb is implemented in Qemu. > > > 2. Set up the e1000 interface for the guests. > > 3. Pressurize the network slightly between two guests using the iPerf > tool. > > > > The network goes down after a few days (2-5days), and the issue > > is the result of not adhering to the e1000 specification. Refer > > to the details of the specification at the following link: > > > https://www.intel.com/content/dam/doc/manual/pci-pci-x-family-gbe-control= lers-software-dev-manual.pdf > > > > Chapter 3.2.6 describe the Receive Descriptor Tail register(RDT) > > as following: > > This register holds a value that is an offset from the base, and > > identifies the location beyond the last descriptor hardware can > > process. Note that tail should still point to an area in the > > descriptor ring (somewhere between RDBA and RDBA + RDLEN). > > This is because tail points to the location where software writes > > the first new descriptor. > > > > This means that if the provider=E2=80=94in this case, QEMU=E2=80=94has = not yet > > loaded the packet, > > What do you mean by "load" here? > Sorry for failing to describe the details. The guest driver retrieves the packet from the receive ring buffer after QEMU forwards it from the tun/tap interface in the e1000 emulation. I used "load" to express "putting packets into the receive ring buffer." > > > RDT should never point to that place. > > And "that place"? > If a descriptor in the receive ring buffer has not been filled with a packet address by QEMU, the descriptor therefore doesn't have any available packets. The location of the descriptor should not be referred to by RDT because the location is in the range that "hardware" handles. "that place" means the location of the descriptor in the ring buffer that QEMU hasn't set any available packets related to. > > > When > > implementing the emulation of the e1000 interface, QEMU evaluates > > if the receive ring buffer is full once the RDT equals the RDH, > > based on the assumption that guest drivers adhere to this > > criterion strictly. > > > > We applied the following log patch to assist in analyzing the > > issue and eventually obtained the unexpected information. > > > > Log patch: > > ----------------------------------------------------------------- > > |--- a/hw/net/e1000.c > > |+++ b/hw/net/e1000.c > > |@@ -836,6 +836,9 @@ e1000_set_link_status(NetClientState *nc) > > | static bool e1000_has_rxbufs(E1000State *s, size_t total_size) > > | { > > | int bufs; > > |+ DBGOUT(RX, "rxbuf_size =3D %u, s->mac_reg[RDLEN] =3D %u, > s->mac_reg[RDH] =3D %u, s->mac_reg[RDT] =3D %u\n", > > |+ s->rxbuf_size, s->mac_reg[RDLEN], s->mac_reg[RDH], > s->mac_reg[RDT]); > > |+ > > | /* Fast-path short packets */ > > | if (total_size <=3D s->rxbuf_size) { > > | if (s->mac_reg[RDH] =3D=3D s->mac_reg[RDT] && s->last_overrun= ) > > |@@ -1022,6 +1025,9 @@ e1000_receive_iov(NetClientState *nc, const > struct iovec *iov, int iovcnt) > > | s->rxbuf_min_shift) > > | n |=3D E1000_ICS_RXDMT0; > > | > > |+ DBGOUT(RX, "rxbuf_size =3D %u, s->mac_reg[RDLEN] =3D %u, > s->mac_reg[RDH] =3D %u, s->mac_reg[RDT] =3D %u\n", > > |+ s->rxbuf_size, s->mac_reg[RDLEN], s->mac_reg[RDH], > s->mac_reg[RDT]); > > |+ > > ----------------------------------------------------------------- > > > > The last few logs of information when the network is down: > > > > e1000: total_size =3D 1, rxbuf_size =3D 2048, s->mac_reg[RDLEN] =3D 163= 84, > s->mac_reg[RDH] =3D 897, s->mac_reg[RDT] =3D 885 > > <- the receive ring buffer is checked for fullness in the > > e1000_has_rxbufs function, not full. > > > > e1000: total_size =3D 64, rxbuf_size =3D 2048, s->mac_reg[RDLEN] =3D 16= 384, > s->mac_reg[RDH] =3D 898, s->mac_reg[RDT] =3D 885 > > <- RDT stays the same, RDH updates to 898, and 1 descriptor > > utilized after putting the packet to ring buffer. > > > > e1000: total_size =3D 1, rxbuf_size =3D 2048, s->mac_reg[RDLEN] =3D 163= 84, > s->mac_reg[RDH] =3D 898, s->mac_reg[RDT] =3D 885 > > <- the receive ring buffer is checked for fullness in the > > e1000_has_rxbufs function, not full. > > > > e1000: total_size =3D 64, rxbuf_size =3D 2048, s->mac_reg[RDLEN] =3D 16= 384, > s->mac_reg[RDH] =3D 899, s->mac_reg[RDT] =3D 885 > > <- RDT stays the same, RDH updates to 899, and 1 descriptor > > utilized after putting the packet to ring buffer. > > > > e1000: total_size =3D 1, rxbuf_size =3D 2048, s->mac_reg[RDLEN] =3D 163= 84, > s->mac_reg[RDH] =3D 899, s->mac_reg[RDT] =3D 885 > > <- the receive ring buffer is checked for fullness in the > > e1000_has_rxbufs function, not full. > > > > e1000: total_size =3D 64, rxbuf_size =3D 2048, s->mac_reg[RDLEN] =3D 16= 384, > s->mac_reg[RDH] =3D 900, s->mac_reg[RDT] =3D 885 > > <- RDT stays the same, RDH updates to 900 , and 1 descriptor > > utilized after putting the packet to ring buffer. > > > > e1000: total_size =3D 1, rxbuf_size =3D 2048, s->mac_reg[RDLEN] =3D 163= 84, > s->mac_reg[RDH] =3D 900, s->mac_reg[RDT] =3D 900 > > <- The ring is full, according to e1000_has_rxbufs, because > > of the RDT update to 900 and equals RDH ! > > Just to make sure I understand this, RDT=3D=3DRDH means the ring is empty= I > think? > See commit: > > commit e5b8b0d4ba29fe1268ba049519a1b0cf8552a21a > Author: Dmitry Fleytman > Date: Fri Oct 19 07:56:55 2012 +0200 > > e1000: drop check_rxov, always treat RX ring with RDH =3D=3D RDT as e= mpty > > Real HW always treats RX ring with RDH =3D=3D RDT as empty. > Emulation is supposed to behave the same. > Indeed, I'm confused :(, the description in the comment claims that RX rings with RDH =3D=3D RDT as empty, but in implementation, it treats that a= s overrun. See the following 2 contexts: 1. e1000_can_receive: static bool e1000_can_receive(NetClientState *nc) { E1000State *s =3D qemu_get_nic_opaque(nc); // e1000_has_rxbufs return true means ring buffer has // available descriptors to use for QEMU. // false means ring buffer overrun and QEMU should queue the packet // and wait for the RDT update and available descriptors can be used. return e1000x_rx_ready(&s->parent_obj, s->mac_reg) && e1000_has_rxbufs(s, 1) && !timer_pending(s->flush_queue_timer); } 2. e1000_receive_iov: if (!e1000_has_rxbufs(s, total_size)) { // e1000_has_rxbufs return false means overrun and QEMU should // inject RXO interrupt to guest e1000_receiver_overrun(s, total_size); return -1; } > > Reported-by: Chris Webb > Reported-by: Richard Davies > Signed-off-by: Dmitry Fleytman > Signed-off-by: Stefan Hajnoczi > > Or did you mean we need to revert the above commit in fact? > Yes, this commit was overlooked by me but it is almost what I want to do in this patch. > > Thanks > > Thanks for the comment, Yong --=20 Best regards --000000000000345123061cb58738 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Mon, Jul 8, 202= 4 at 11:21=E2=80=AFAM Jason Wang <jasowang@redhat.com> wrote:
On Sat, Jul 6= , 2024 at 4:30=E2=80=AFAM Hyman Huang <yong.huang@smartx.com> wrote:
>
> Unexpected work by certain Windows guests equipped with the e1000
> interface can cause the network to go down and never come back up
> again unless the guest's interface is reset.
>
> To reproduce the failure:
> 1. Set up two guests with a Windows 2016 or 2019 server operating
>=C2=A0 =C2=A0 system.

I vaguely remember e1000 support for Windows has been deprecated for
several years...

That's why e1000e or igb is implemented in Qemu.

> 2. Set up the e1000 interface for the guests.
> 3. Pressurize the network slightly between two guests using the iPerf = tool.
>
> The network goes down after a few days (2-5days), and the issue
> is the result of not adhering to the e1000 specification. Refer
> to the details of the specification at the following link:
> https://www.intel.com/content/dam/doc/manual/pci-pci-x-family-gbe-con= trollers-software-dev-manual.pdf
>
> Chapter 3.2.6 describe the Receive Descriptor Tail register(RDT)
> as following:
> This register holds a value that is an offset from the base, and
> identifies the location beyond the last descriptor hardware can
> process. Note that tail should still point to an area in the
> descriptor ring (somewhere between RDBA and RDBA + RDLEN).
> This is because tail points to the location where software writes
> the first new descriptor.
>
> This means that if the provider=E2=80=94in this case, QEMU=E2=80=94has= not yet
> loaded the packet,

What do you mean by "load" here?

<= div>
Sorry for failing to describe the details.

The guest driver retrieves the packet from the = receive ring buffer
after QEMU forwards it from the tun/t= ap interface in the e1000
emulation.

I used "load" to express= "putting packets into the receive ring buffer."=C2=A0
=C2=A0

> RDT should never point to that place.

And "that place"?
<= div class=3D"gmail_default">If a descriptor in the receive ring buffer has n= ot been filled with a
packet= address by QEMU, the descriptor therefore doesn't have any
available packets. The location of the = descriptor should not be referred
to by RDT because the location is in the range that "hardware&qu= ot; handles.=C2=A0

"that place" means the location of the descriptor = in the ring buffer
that QEMU= hasn't set any available packets related to.
=
=C2=A0

> When
> implementing the emulation of the e1000 interface, QEMU evaluates
> if the receive ring buffer is full once the RDT equals the RDH,
> based on the assumption that guest drivers adhere to this
> criterion strictly.
>
> We applied the following log patch to assist in analyzing the
> issue and eventually obtained the unexpected information.
>
> Log patch:
> -----------------------------------------------------------------
> |--- a/hw/net/e1000.c
> |+++ b/hw/net/e1000.c
> |@@ -836,6 +836,9 @@ e1000_set_link_status(NetClientState *nc)
> | static bool e1000_has_rxbufs(E1000State *s, size_t total_size)
> | {
> |=C2=A0 =C2=A0 =C2=A0int bufs;
> |+=C2=A0 =C2=A0 DBGOUT(RX, "rxbuf_size =3D %u, s->mac_reg[RDLE= N] =3D %u, s->mac_reg[RDH] =3D %u, s->mac_reg[RDT] =3D %u\n", > |+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0s->rxbuf_size, s->mac= _reg[RDLEN], s->mac_reg[RDH], s->mac_reg[RDT]);
> |+
> |=C2=A0 =C2=A0 =C2=A0/* Fast-path short packets */
> |=C2=A0 =C2=A0 =C2=A0if (total_size <=3D s->rxbuf_size) {
> |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0if (s->mac_reg[RDH] =3D=3D s->= ;mac_reg[RDT] && s->last_overrun)
> |@@ -1022,6 +1025,9 @@ e1000_receive_iov(NetClientState *nc, const str= uct iovec *iov, int iovcnt)
> |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0s->rxbuf_min_shift)
> |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0n |=3D E1000_ICS_RXDMT0;
> |
> |+=C2=A0 =C2=A0 DBGOUT(RX, "rxbuf_size =3D %u, s->mac_reg[RDLE= N] =3D %u, s->mac_reg[RDH] =3D %u, s->mac_reg[RDT] =3D %u\n", > |+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0s->rxbuf_size, s->mac= _reg[RDLEN], s->mac_reg[RDH], s->mac_reg[RDT]);
> |+
> -----------------------------------------------------------------
>
> The last few logs of information when the network is down:
>
> e1000: total_size =3D 1, rxbuf_size =3D 2048, s->mac_reg[RDLEN] =3D= 16384, s->mac_reg[RDH] =3D 897, s->mac_reg[RDT] =3D 885
> <- the receive ring buffer is checked for fullness in the
> e1000_has_rxbufs function, not full.
>
> e1000: total_size =3D 64, rxbuf_size =3D 2048, s->mac_reg[RDLEN] = =3D 16384, s->mac_reg[RDH] =3D 898, s->mac_reg[RDT] =3D 885
> <- RDT stays the same, RDH updates to 898, and 1 descriptor
> utilized after putting the packet to ring buffer.
>
> e1000: total_size =3D 1, rxbuf_size =3D 2048, s->mac_reg[RDLEN] =3D= 16384, s->mac_reg[RDH] =3D 898, s->mac_reg[RDT] =3D 885
> <- the receive ring buffer is checked for fullness in the
> e1000_has_rxbufs function, not full.
>
> e1000: total_size =3D 64, rxbuf_size =3D 2048, s->mac_reg[RDLEN] = =3D 16384, s->mac_reg[RDH] =3D 899, s->mac_reg[RDT] =3D 885
> <- RDT stays the same, RDH updates to 899, and 1 descriptor
> utilized after putting the packet to ring buffer.
>
> e1000: total_size =3D 1, rxbuf_size =3D 2048, s->mac_reg[RDLEN] =3D= 16384, s->mac_reg[RDH] =3D 899, s->mac_reg[RDT] =3D 885
> <- the receive ring buffer is checked for fullness in the
> e1000_has_rxbufs function, not full.
>
> e1000: total_size =3D 64, rxbuf_size =3D 2048, s->mac_reg[RDLEN] = =3D 16384, s->mac_reg[RDH] =3D 900, s->mac_reg[RDT] =3D 885
> <- RDT stays the same, RDH updates to 900 , and 1 descriptor
> utilized after putting the packet to ring buffer.
>
> e1000: total_size =3D 1, rxbuf_size =3D 2048, s->mac_reg[RDLEN] =3D= 16384, s->mac_reg[RDH] =3D 900, s->mac_reg[RDT] =3D 900
> <- The ring is full, according to e1000_has_rxbufs, because
> of the RDT update to 900 and equals RDH !

Just to make sure I understand this, RDT=3D=3DRDH means the ring is empty I= think?=C2=A0

See commit:

commit e5b8b0d4ba29fe1268ba049519a1b0cf8552a21a
Author: Dmitry Fleytman <dmitry@daynix.com>
Date:=C2=A0 =C2=A0Fri Oct 19 07:56:55 2012 +0200

=C2=A0 =C2=A0 e1000: drop check_rxov, always treat RX ring with RDH =3D=3D = RDT as empty

=C2=A0 =C2=A0 Real HW always treats RX ring with RDH =3D=3D RDT as empty. =C2=A0 =C2=A0 Emulation is supposed to behave the same.

Indeed, I'm confused :(, =C2=A0the = description in the comment= =C2=A0claims that RX
rings with RDH =3D=3D RDT as=C2=A0empty, but in im= plementation, it treats=C2=A0that as
overrun= .

See th= e following 2 contexts:

1.=C2=A0= e1000_can_receive:
static bool e1000_can_receive(NetClientState *nc)=
{
=C2=A0 =C2=A0 E1000State *s =3D qemu_get_nic_opaque(nc);
=C2=A0= =C2=A0 //=C2=A0e1000_has_rxbufs return true means ring buffer has
=C2= =A0 =C2=A0 // available descriptors to use for QEMU.
=C2=A0 =C2=A0 /= / false means ring buffer overrun and QEMU should queue the packet
= =C2=A0 =C2=A0 // and wait for the RDT update and available=C2=A0descriptors= can be used.

=C2=A0 =C2=A0 return e1000x_rx_ready(&s->pa= rent_obj, s->mac_reg) &&
=C2=A0 =C2=A0 =C2=A0 =C2=A0 e1000_ha= s_rxbufs(s, 1) && !timer_pending(s->flush_queue_timer);
}

= 2.=C2=A0e1000_receive_iov:<= /div>
= =C2=A0 =C2=A0 if (!e1000_has_rxbufs(s, total_size)) {
=C2=A0 =C2=A0 = =C2=A0 =C2=A0 //=C2=A0e1000_has_rxbufs return false means overrun and QEMU sho= uld
=C2=A0 =C2=A0 =C2=A0 =C2=A0 // inject RXO interrupt to guest
=C2=A0 =C2=A0 =C2=A0 =C2=A0= e1000_receiver_overrun(s, total_size);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 retu= rn -1;
=C2=A0 =C2=A0 }
=C2=A0

=C2=A0 =C2=A0 Reported-by: Chris Webb <chris.webb@elastichosts.com>
=C2=A0 =C2=A0 Reported-by: Richard Davies <richard.davies@elastichosts.com= >
=C2=A0 =C2=A0 Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
=C2=A0 =C2=A0 Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>

Or did you mean we need to revert the above commit in fact?

Yes, this commit was overlooked by me but= it is almost what I want to do
in this patch.
=C2=A0

Thanks


Thanks fo= r the comment,

Yong

--
Best regards
--000000000000345123061cb58738--