From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 56C6ECD1296 for ; Wed, 10 Apr 2024 07:57:48 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ruSpO-0003QL-Ln; Wed, 10 Apr 2024 03:57:06 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ruSpN-0003P7-Ic for qemu-devel@nongnu.org; Wed, 10 Apr 2024 03:57:05 -0400 Received: from mail-dm6nam12on2107.outbound.protection.outlook.com ([40.107.243.107] helo=NAM12-DM6-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ruSpK-0001q7-PL for qemu-devel@nongnu.org; Wed, 10 Apr 2024 03:57:05 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=kfSjQ2BAFc9ujtBdWCvb6PP6zOf60PxWkDInrQooXdR+/LqSbd5Da9Iy2/YD190tCiYJau43NoWEn3z1Yt6saz9EoiixDKrME2LPa/w1YswUjHacUBkn17IIOZYqIg40LT3xpCdNei7urQDSSZu4hRQ3Z/9yQzwTku6vhfaBS8FzX4IFp6mRdC/7/k+lxjzdyfR2rF+UbMMFQU52D64D+6VIv+OKc88/1Z/86eNN4AS3ScKwEVbCwZPFYq6dmqM7zZ2sCb43RFbvCGBHBgqhLM4zWEepqbSHkNCipEyk2wxrnmD1eTy8V45fDqSx3D4TfOOo/I41a9Dp+crH4LWYjw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=rxk4EHMvk8injGQD6IS4GsGUxpqPuTlEhj9/5b94KSU=; b=GEkl6BzqkAEG9degZgX5G+OnzHkChzpI/O0ZZT9SGN0igMm8kO8F+U9AEQpQW9Y5InxIlvHIVYTvdcOYtSvQigBBiTKwfswECiPfLgYIKJUwnAhCn/ETjukimG0toHX8GwYwEmNJ9rEECmvs52RAcLVicIgdKvRRHkMpE/i3w8Qyx+qgUu6jPV0jc22h9mm8Sl+e1q1EIJqW8yT5NR+8EvNyfC7GNbP7DUw9bPS7oo+uP4c49rpzK1hMgdj1P/aM8fE/zfa8BOa0E4QcMH3+5kMdpACYgMWYlgWckQo4oUHQaXHwZPWs9OYB/Rerd1B44DcZrP6pG+RItUUiZRIOkA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rxk4EHMvk8injGQD6IS4GsGUxpqPuTlEhj9/5b94KSU=; b=TJlE8IsQa8lH64wNM8ekL7sxUUQfLIZg1mdYNMDx0HS26VBGdWDQ8uwE/GE57L10pjGBE5tnjwMk4cie4xzaA4bfy78Pf/AKzWxbKvfj1JXaOHiwcwm5vWrRW78T4mXHSJmEK6F37AVV63cLvjFmvZOuV4x78sTPyBsg8lEZlUVsKAfnB6plcHEwwJ1mqR/i5c2s2Sd9rAUilw5QYixbr90HaB1f0Z8fYJiVPSCF06s7ln4DX/JhI8hne77w1PyE2f8hCu4sK1P3ZkvZ8ZwU/avTqmtX+fGqycLktVtNBmhNzAWIFUAsn0ZY+tje2Sc4kTZJIt68WuAP6loZh15hnw== Received: from DM4PR12MB5168.namprd12.prod.outlook.com (2603:10b6:5:397::8) by CH3PR12MB8257.namprd12.prod.outlook.com (2603:10b6:610:121::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.46; Wed, 10 Apr 2024 07:51:55 +0000 Received: from DM4PR12MB5168.namprd12.prod.outlook.com ([fe80::a69:16d9:139b:efb5]) by DM4PR12MB5168.namprd12.prod.outlook.com ([fe80::a69:16d9:139b:efb5%4]) with mapi id 15.20.7409.053; Wed, 10 Apr 2024 07:51:51 +0000 Content-Type: multipart/alternative; boundary="------------gMyzBj7MIBsoPaz9aNFQ75ot" Message-ID: Date: Wed, 10 Apr 2024 15:51:41 +0800 User-Agent: Mozilla Thunderbird From: Yajun Wu Subject: Re: vhost-user-blk reconnect issue To: Li Feng , Stefano Garzarella , "raphael.norwitz@nutanix.com" Cc: =?UTF-8?Q?Alex_Benn=C3=A9_e?= , "qemu-devel@nongnu.org" , "mst@redhat.com" , Parav Pandit References: <70EC669B-52F8-408B-866B-9AFFB7EBAE96@smartx.com> Content-Language: en-US In-Reply-To: X-ClientProxiedBy: SI2PR04CA0013.apcprd04.prod.outlook.com (2603:1096:4:197::6) To DM4PR12MB5168.namprd12.prod.outlook.com (2603:10b6:5:397::8) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM4PR12MB5168:EE_|CH3PR12MB8257:EE_ X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: raBP9uCRA5vgHduGbmhPB1yzAKyW0T741IamurM5TPRYE7n1bKnt8YRNkBOYbQ+Vcaiz8tY9Klna4vgqu8b3NWNkZbKh6UQEu1QhoNCsVyRLk86Ba7RaVWy1BCzYmtun8drVlVsauD8r/JdPl6/KXNUmYLVDftd/BkeL+G3TsehZJeXli5hKjshw9cTwF3xBMYoLyP+8t02QYFSgqO4XA82C4DibpXKFhp7s/CiZinVFrbd+OCyOtZyaa2OAmr7bdyoLzm3eQU4KZO+VJpMvJyTju1a31vOLHgDh5laX1NP4cibuWeovQm8czK8oWEPqEqnYNxLFvq8SjSn/eoGsuKsUapVInbeHFIVLZ/B/DsczCFZaf7BJGYU3Pd8iLSVaPs/4ey4H7dpgYjx1nYV/Sjmmw+/UKzBycyZz6RtYyB1MLtk0DtNnNL2sZxRBU7N+qUbKvMCrCPh/gyP/saOiSBoOYEEkdE5k1W1ZwuFLKoP3ch2oSrUdFxSt3Gmqg0Ea0KSqt3imRXmuo4ToyJ7TlEjnoPG47HvTdQXwv8qHxZnuFiKftH4kp8cNuSaDQqyVcoboUlJZKX5Oiu5Rof3jEY3CjkuUuxvufAg889RPvM2U0MqlYmc3HvTwrimsGjZG3VKfMEzOs9yG3r1NGyxnihnNPo8Yw2xVGU0Oj4BxC6ZHq+ni8XgJ5n9/45Hj3RuMhzL/uvVEjlMmrP9chPyn/w== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM4PR12MB5168.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(366007)(1800799015)(376005)(4143199003); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?UzZVd0hGTnpQWGR0clAwTmJkSUlyU0lZeWUwQnl1bDZEZnFjbmJCWGhjcWgz?= =?utf-8?B?bVdCQnNRRkFYRHF3UXNzVUtmUktNRjR4a0pGQ01UcVpuSWJHVXBpMzVmamdr?= =?utf-8?B?Sk1pZFZqdVNDNHlveGhydDF6Y1NiTGo3aUNkU2twQldGWWxUWU1ZV0RuU3Aw?= =?utf-8?B?ZTRVUm9kVHorbVlTL0lwRG5CUUpmSkoxVkhMckZjL2lsb2ZMUWZVM1JsV0s3?= =?utf-8?B?eEMwSUFXUVdaZDJZR0o5azhBaHJodkRMUkh1RXJTMy92endlN3dsdWY4R1RG?= =?utf-8?B?aXdjbEZuKzVhVTVBakwxQjd5UFdYcFVqbmxMRjNXRE9GSUFwUVRGR09KT2Js?= =?utf-8?B?R2ZEL3R0Q1kyZXZwbk8zVkV5d0hPY2MrNGFqMFN0UjlxNlZFanllcXgvbzF3?= =?utf-8?B?L1liK0hvOVVQR2QzZWpDMHdzQWZIUkg0eFJMQ205VjFsVzRQK1NnWk1VSGNj?= =?utf-8?B?eTVab3hTTFR6YWk0dDdBd0NjeDhTVmxjd0d6TS9aaExHNERudDVvNmFzM2c0?= =?utf-8?B?Vm14cUNlbU92WmFZTEpjTkRrUkdQTzJJazlSZnZyUEFnL0dpSm9vZU9jQU05?= =?utf-8?B?NU5LdE42UGRiVWRtdHRaSG5vcDRhaWJnWjFZdHNDT05BOFFiMEpuQ09CQVFI?= =?utf-8?B?V21PTWtEaG41bmM5WUhlOWdyNitLM2RRb2xGMEVRYzVkaC9lVDE5dUUvLzVl?= =?utf-8?B?YnhoMS9YcU1MelJRVTYrMXdVcDVPSms3ZUpzQS9sY2xBZlhjQWhPczJ0MFAv?= =?utf-8?B?N0MwbE5MNEt2eitveVpMKzR1b2VYWWhoK1BvVEh0NEhXMS84ZXJ3OWVVTjBa?= =?utf-8?B?US9FeVRQdjhEall0aGlNVlBMR3FhMytQMDRzdHpPVWVINktZRTNGRkRWQXpS?= =?utf-8?B?VXQyNVlqeStuV0tSMFFZSlZKWDBMUFNUc1h6a2U3NVBSUXF5cjdWMUxraGIz?= =?utf-8?B?eENWSTM2K2JSVnhqdmJIRzlQRU13T2pEYS8xdWpKVXVwbUhjUEFMYTAxU21B?= =?utf-8?B?VkpmalV1MXhIY2tzL0IwdE5WdHAzT0h5NUNWRTJBc2ZMb3hTMHRuZXN4c0FL?= =?utf-8?B?ajBKTlFVSUNNK1dSd2RubEUwbFdLZEl1MU9ic2wzaG5IMnBybTVNak1NRjlW?= =?utf-8?B?VC9pODdNN1RldGlGTnRRdDNTc21ab29lZUZsQk05RThLWUJmUXR0cmxHMC9G?= =?utf-8?B?NWMyTTFEUUZpSkNFZjYyajRSbHJ6YWdjK0hTZnZoRHlnNkNFaGxnRi9DbzN6?= =?utf-8?B?STQwTjUyQ0RVUnZsMzYvanVXMjhRRktJYis4Nks5WFBYNHI5RXFVWGNoMllk?= =?utf-8?B?UnBGS1NDRGZYbGpldHR4VVNCRXNWdjdsRTNKRXZ5OUpxOHJ2bjZCZHpQeVNo?= =?utf-8?B?SEN3em5tdi9FVDczMVRRQ2labURYVzV0VzRaaHNjeTNlOFZsNkl6cVByN0tW?= =?utf-8?B?OHQ5TW9wNXcwT1Y1TVZUYlQ0OHlaZ2tKZlZsU0tYS3VIL0c5cFoyMGpOeE5h?= =?utf-8?B?aWNjbTI4eDB0YkVoZW1MRlVqZDQ4RmxoTGltZGNkQUtFM2tsN2gzcUR4MDlW?= =?utf-8?B?N3VwM3lMNDVZY042bnVhRFFYdVNpcUZjY0ZqeWFRTHU1ZEtaZXNIK3VVZXB0?= =?utf-8?B?bDF1ZDgrbW1kZ1VXei9URlQ0SUxNMjB1Zk5Fb2tKN1BGbUxTWGVEYUVTZ1lq?= =?utf-8?B?ZUx6aU5hWFg5am5yTjZldlY3dU1Hb0ppMW1zOUlNditFOHozeldISzBxS0VN?= =?utf-8?B?VkxsWmdlMDFzK0tsT2R1NUsyNGxqbGhMV3pZUmZBZGVpUkFkdUR1QjBCZGpl?= =?utf-8?B?TVdxMVU2aFNPUE1LbFdiWE9sQzQxRXdDVno2ejRtUHpKSzlyZFh4YUVYT1g4?= =?utf-8?B?UHdTdXBxOEtBQ3RscjJ2OXNISEdEcWU1V1lPNHpmYVlEQ0V2YmJIRG9PaXNs?= =?utf-8?B?M2xiSVhsOVhJRXIvaWZqMEkvcHNVV3YxVlRsRkpiS042d1FtWmlDZGVWZDZo?= =?utf-8?B?Q1JZdWhjdGtZNkNoSGdSYk55RzJVN2d0cG94aTlVeHJpdFcyd1NPRVdzZUNv?= =?utf-8?B?Q0FpeDBXUDVHNUNGVWJrMU9GOTFNTHdYN1U0OXdIUVlYV0E0Q1lhSEduOFk5?= =?utf-8?Q?5A5ZnZO+Ig4NuBnENZTt3EC5c?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 35e64634-d988-488e-dbcc-08dc593314cc X-MS-Exchange-CrossTenant-AuthSource: DM4PR12MB5168.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Apr 2024 07:51:51.1245 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: nFZyf/WSXSQwkm92ueyBDmL/WBizY/EJ46+iE3Jvnfz8Nno58Rs6FIKdHfNrB2QmukPmHXrBJE0WB2wOJcP63w== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB8257 Received-SPF: softfail client-ip=40.107.243.107; envelope-from=yajunw@nvidia.com; helo=NAM12-DM6-obe.outbound.protection.outlook.com X-Spam_score_int: -23 X-Spam_score: -2.4 X-Spam_bar: -- X-Spam_report: (-2.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.701, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, RCVD_IN_BL_SPAMCOP_NET=1.347, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_REMOTE_IMAGE=0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org --------------gMyzBj7MIBsoPaz9aNFQ75ot Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 4/2/2024 4:44 PM, Li Feng wrote: > *External email: Use caution opening links or attachments* > > > > Hi, > > I tested it today and there is indeed a problem in this scenario. > It seems that the first version of the patch is the best and can > handle all scenarios. > With this patch, the previously merged patches are no longer useful. > > > I will revert this patch and submit a new fix. Do you have any comments? > > Revert: > https://lore.kernel.org/all/a68c0148e9bf105f9e83ff5e763b8fcb6f7ba9be.1697644299.git.mst@redhat.com/ > > New: > https://lore.kernel.org/all/20230804052954.2918915-2-fengli@smartx.com/ > Looks good to me. Thanks, Yajun > > Thanks, > Li > >> 2024年4月1日 16:43，Yajun Wu 写道： >> >> >> On 4/1/2024 4:34 PM, Li Feng wrote: >>> *External email: Use caution opening links or attachments* >>> >>> >>> Hi yajun, >>> >>> I have submitted a patch to fix this problem a few months ago, but >>> in the end this solution was not accepted and other solutions >>> were adopted to fix it. >>> >>> [PATCH 1/2] vhost-user: fix lost reconnect - Li Feng >>> >>> lore.kernel.org >>> >>> >>> >>> >>> >> I think this fix is valid. >> >>> This is the merged fix: >>> >>> >>> [PULL 76/83] vhost-user: fix lost reconnect - Michael S. Tsirkin >>> >>> lore.kernel.org >>> >>> >>> >>> >> >> My tests are with this fix, failed in the two scenarios I mentioned. >> >>> >>> Thanks, >>> Li >>> >>>> 2024年4月1日 10:08，Yajun Wu 写道： >>>> >>>> >>>> On 3/27/2024 6:47 PM, Stefano Garzarella wrote: >>>>> External email: Use caution opening links or attachments >>>>> >>>>> >>>>> Hi Yajun, >>>>> >>>>> On Mon, Mar 25, 2024 at 10:54:13AM +0000, Yajun Wu wrote: >>>>>> Hi experts, >>>>>> >>>>>> With latest QEMU (8.2.90), we find two vhost-user-blk backend >>>>>> reconnect >>>>>> failure scenarios: >>>>> Do you know if has it ever worked and so it's a regression, or have we >>>>> always had this problem? >>>> >>>> I am afraid this commit: "71e076a07d (2022-12-01 02:30:13 -0500) >>>> hw/virtio: generalise CHR_EVENT_CLOSED handling"  caused both >>>> failures. Previous hash is good. >>>> >>>> I suspect the "if (vhost->vdev)" in vhost_user_async_close_bh is >>>> the cause, previous code doesn't have this check? >>>> >>>>> >>>>> Thanks, >>>>> Stefano >>>>> >>>>>> 1. Disconnect vhost-user-blk backend before guest driver probe >>>>>> vblk device, then reconnect backend after guest driver probe >>>>>> device. QEMU won't send out any vhost messages to restore backend. >>>>>> This is because vhost->vdev is NULL before guest driver probe >>>>>> vblk device, so vhost_user_blk_disconnect won't be called, >>>>>> s->connected is still true. Next vhost_user_blk_connect will >>>>>> simply return without doing anything. >>>>>> >>>>>> 2. modprobe -r virtio-blk inside VM, then disconnect backend, >>>>>> then reconnect backend, then modprobe virtio-blk. QEMU won't send >>>>>> messages in vhost_dev_init. >>>>>> This is because rmmod will let qemu call vhost_user_blk_stop, >>>>>> vhost->vdev also become NULL(in vhost_dev_stop), >>>>>> vhost_user_blk_disconnect won't be called. Again s->connected is >>>>>> still true, even chr connect is closed. >>>>>> >>>>>> I think even vhost->vdev is NULL, vhost_user_blk_disconnect >>>>>> should be called when chr connect close? >>>>>> Hope we can have a fix soon. >>>>>> >>>>>> >>>>>> Thanks, >>>>>> Yajun >>>>>> >>> > --------------gMyzBj7MIBsoPaz9aNFQ75ot Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable


On 4/2/2024 4:44 PM, Li Feng wrote:
=20
Extern= al email: Use caution opening links or attachments


Hi,

I tested it today and there is indeed a problem in this scenario.
It seems that the first version of the patch is the best and can handle all scenarios.
With this patch, the previously merged patches are no longer useful.


I will revert this patch and submit a new fix. Do you have any comments?

Looks good to me.

Thanks,

Yajun


Thanks,
Li

2024=E5=B9=B44=E6=9C=881=E6=97=A5 16:43=EF=BC=8CYajun Wu <= a class=3D"moz-txt-link-rfc2396E" href=3D"mailto:yajunw@nvidia.com"><yaj= unw@nvidia.com> =E5=86=99=E9=81=93=EF=BC=9A


On 4/1/2024 4:34 PM, Li Feng wrote:
External email: Use caution opening links or attachments

I think this fix is valid.

My tests are with this fix, failed in the two scenarios I mentioned.


Thanks,
Li

2024=E5=B9=B44=E6=9C=881=E6=97=A5 10:0= 8=EF=BC=8CYajun Wu <yajunw@nvidia.com> =E5=86=99= =E9=81=93=EF=BC=9A


On 3/27/2024 6:47 PM, Stefano Garzarella wrote:
External email: Use caution opening links or attachments


Hi Yajun,

On Mon, Mar 25, 2024 at 10:54:13AM +0000, Yajun Wu wrote:
Hi experts,

With latest QEMU (8.2.90), we find two vhost-user-blk backend reconnect
failure scenarios:
Do you know if has it ever worked and so it's a regression, or have we
always had this problem?

I am afraid this commit: "71e076a0= 7d (2022-12-01 02:30:13 -0500) hw/virtio: generalise CHR_EVENT_CLOSED handling"  c= aused both failures. Previous hash is good.

I suspect the "if (vhost->vdev)= " in vhost_user_async_close_bh is the cause, previous code doesn't have this check?


Thanks,
Stefano

1. Disconnect vhost-user-blk backend before guest driver probe vblk device, then reconnect backend after guest driver probe device. QEMU won't send out any vhost messages to restore backend.
This is because vhost->vdev is NULL before guest driver probe vblk device, so vhost_user_blk_disconnect won't be called, s->connected is still true. Next vhost_user_blk_connect will simply return without doing anything.

2. modprobe -r virtio-blk inside VM, then disconnect backend, then reconnect backend, then modprobe virtio-blk. QEMU won't send messages in vhost_dev_init. This is because rmmod will let qemu call vhost_user_blk_stop, vhost->vdev also become NULL(in vhost_dev_stop), vhost_user_blk_disconnect won't be called. Again s->connected is still true, even chr connect is closed.

I think even vhost->vdev is NULL, vhost_user_blk_disconnect should be called when chr connect close?
Hope we can have a fix soon.


Thanks,
Yajun



--------------gMyzBj7MIBsoPaz9aNFQ75ot--