From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 080A2C001B0 for ; Sun, 13 Aug 2023 09:50:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231151AbjHMJuo (ORCPT ); Sun, 13 Aug 2023 05:50:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52132 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229484AbjHMJun (ORCPT ); Sun, 13 Aug 2023 05:50:43 -0400 Received: from out28-44.mail.aliyun.com (out28-44.mail.aliyun.com [115.124.28.44]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AE270E7B for ; Sun, 13 Aug 2023 02:50:43 -0700 (PDT) X-Alimail-AntiSpam: AC=CONTINUE;BC=0.04969759|-1;CH=green;DM=|CONTINUE|false|;DS=CONTINUE|ham_regular_dialog|0.344122-0.0200471-0.635831;FP=0|0|0|0|0|0|0|0;HT=ay29a033018047206;MF=wangyugui@e16-tech.com;NM=1;PH=DS;RN=4;RT=4;SR=0;TI=SMTPD_---.UFCvudK_1691920232; Received: from 192.168.2.112(mailfrom:wangyugui@e16-tech.com fp:SMTPD_---.UFCvudK_1691920232) by smtp.aliyun-inc.com; Sun, 13 Aug 2023 17:50:38 +0800 Date: Sun, 13 Aug 2023 17:50:38 +0800 From: Wang Yugui To: Chris Mason Subject: Re: btrfs write-bandwidth performance regression of 6.5-rc4/rc3 Cc: Christoph Hellwig , linux-btrfs@vger.kernel.org, Chris Mason In-Reply-To: <20ab0be0-e7d0-632b-b94c-89d76911f1ed@meta.com> References: <20230811222321.2AD2.409509F4@e16-tech.com> <20ab0be0-e7d0-632b-b94c-89d76911f1ed@meta.com> Message-Id: <20230813175032.AA17.409509F4@e16-tech.com> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="------_64D8A6F600000000AA12_MULTIPART_MIXED_" Content-Transfer-Encoding: 7bit X-Mailer: Becky! ver. 2.81.04 [en] Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org --------_64D8A6F600000000AA12_MULTIPART_MIXED_ Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit Hi, > On 8/11/23 10:23 AM, Wang Yugui wrote: > > Hi, > > > > > >> On Wed, Aug 02, 2023 at 08:04:57AM +0800, Wang Yugui wrote: > >>>> And with only a revert of > >>>> > >>>> "btrfs: submit IO synchronously for fast checksum implementations"? > >>> > >>> GOOD performance when only (Revert "btrfs: submit IO synchronously for fast > >>> checksum implementations") > >> > >> Ok, so you have a case where the offload for the checksumming generation > >> actually helps (by a lot). Adding Chris to the Cc list as he was > >> involved with this. > >> > >>>>> - if (test_bit(BTRFS_FS_CSUM_IMPL_FAST, &bbio->fs_info->flags)) > >>>>> + if ((bbio->bio.bi_opf & REQ_META) && test_bit(BTRFS_FS_CSUM_IMPL_FAST, &bbio->fs_info->flags)) > >>>>> return false; > >>>> > >>>> This disables synchronous checksum calculation entirely for data I/O. > >>> > >>> without this fix, data I/O checksum is always synchronous? > >>> this is a feature change of "btrfs: submit IO synchronously for fast checksum implementations"? > >> > >> It is never with the above patch. > >> > >>> > >>>> Also I'm curious if you see any differents for a non-RAID0 (i.e. > >>>> single profile) workload. > >>> > >>> '-m single -d single' is about 10% slow that '-m raid1 -d raid0' in this test > >>> case. > >> > >> How does it compare with and without the revert? Can you add the numbers? > > > > Looking through the thread, you're comparing -m single -d single, but > btrfs is still doing the raid. > > Sorry to keep asking for more runs, but these numbers are a surprise, > and I probably won't have time today to reproduce before vacation next > week (sadly, Christoph and I aren't going together). > > Can you please do a run where lvm or MD raid are providing the raid0? no LVM/MD used here. > It doesn't look like you're using compression, but I wanted to double check. Yes. '-m xx -d yy' with other default mkfs.btrfs option, so no compression. > How much ram do you have? 192G ECC memory. two CPU numa nodes, but all PCIe3 NVMe SSD are connected to one NVMe HBA/ one numa node. > Your fio run has 4 jobs going, can I please see the full fio output for > a fast run and a slow run? fio results are saved into attachment files (fast.text & slow.txt) Best Regards Wang Yugui (wangyugui@e16-tech.com) 2023/08/13 --------_64D8A6F600000000AA12_MULTIPART_MIXED_ Content-Type: application/octet-stream; name="fast.txt" Content-Disposition: attachment; filename="fast.txt" Content-Transfer-Encoding: base64 KyBmaW8gLW5hbWUgd3JpdGUtYmFuZHdpZHRoIC1ydz13cml0ZSAtYnM9MTAyNEtpIC1zaXplPTMy R2kgLXJ1bnRpbWU9MzAgLWlvZGVwdGggMSAtaW9lbmdpbmUgc3luYyAtemVyb19idWZmZXJzPTEg LWRpcmVjdD0wIC1lbmRfZnN5bmM9MSAtbnVtam9icz00IC1kaXJlY3Rvcnk9L21udC90ZXN0DQp3 cml0ZS1iYW5kd2lkdGg6IChnPTApOiBydz13cml0ZSwgYnM9KFIpIDEwMjRLaUItMTAyNEtpQiwg KFcpIDEwMjRLaUItMTAyNEtpQiwgKFQpIDEwMjRLaUItMTAyNEtpQiwgaW9lbmdpbmU9c3luYywg aW9kZXB0aD0xDQouLi4NCmZpby0zLjE5DQpTdGFydGluZyA0IHByb2Nlc3Nlcw0Kd3JpdGUtYmFu ZHdpZHRoOiBMYXlpbmcgb3V0IElPIGZpbGUgKDEgZmlsZSAvIDMyNzY4TWlCKQ0Kd3JpdGUtYmFu ZHdpZHRoOiBMYXlpbmcgb3V0IElPIGZpbGUgKDEgZmlsZSAvIDMyNzY4TWlCKQ0Kd3JpdGUtYmFu ZHdpZHRoOiBMYXlpbmcgb3V0IElPIGZpbGUgKDEgZmlsZSAvIDMyNzY4TWlCKQ0Kd3JpdGUtYmFu ZHdpZHRoOiBMYXlpbmcgb3V0IElPIGZpbGUgKDEgZmlsZSAvIDMyNzY4TWlCKQ0KSm9iczogMSAo Zj0xKTogW18oMyksRigxKV1bMTAwLjAlXVtldGEgMDBtOjAwc10NCndyaXRlLWJhbmR3aWR0aDog KGdyb3VwaWQ9MCwgam9icz0xKTogZXJyPSAwOiBwaWQ9MjU3MDogU3VuIEF1ZyAxMyAxNzo0Nzow MiAyMDIzDQogIHdyaXRlOiBJT1BTPTY5NywgQlc9Njk3TWlCL3MgKDczMU1CL3MpKDIxLjdHaUIv MzE4ODNtc2VjKTsgMCB6b25lIHJlc2V0cw0KICAgIGNsYXQgKHVzZWMpOiBtaW49MzI1LCBtYXg9 MjMwMDMsIGF2Zz0xMzQ1LjEzLCBzdGRldj0zMTU4LjE2DQogICAgIGxhdCAodXNlYyk6IG1pbj0z MjYsIG1heD0yMzAwMywgYXZnPTEzNDUuNTcsIHN0ZGV2PTMxNTguMTYNCiAgICBjbGF0IHBlcmNl bnRpbGVzICh1c2VjKToNCiAgICAgfCAgMS4wMHRoPVsgIDM1NV0sICA1LjAwdGg9WyAgMzg4XSwg MTAuMDB0aD1bICA0MDhdLCAyMC4wMHRoPVsgIDU1M10sDQogICAgIHwgMzAuMDB0aD1bICA3MzRd LCA0MC4wMHRoPVsgIDgzMl0sIDUwLjAwdGg9WyAgODY1XSwgNjAuMDB0aD1bICA4ODldLA0KICAg ICB8IDcwLjAwdGg9WyAgOTE0XSwgODAuMDB0aD1bIDEwMjBdLCA5MC4wMHRoPVsgMTQxOF0sIDk1 LjAwdGg9WyAxNjk2XSwNCiAgICAgfCA5OS4wMHRoPVsyMTYyN10sIDk5LjUwdGg9WzIxODkwXSwg OTkuOTB0aD1bMjIxNTJdLCA5OS45NXRoPVsyMjE1Ml0sDQogICAgIHwgOTkuOTl0aD1bMjI2NzZd DQogICBidyAoICBLaUIvcyk6IG1pbj01MDU4NTYsIG1heD0xNTI3MDE3LCBwZXI9MjYuMTMlLCBh dmc9NzU1MzEzLjQ2LCBzdGRldj0xODExMjIuNzQsIHNhbXBsZXM9NTkNCiAgIGlvcHMgICAgICAg IDogbWluPSAgNDk0LCBtYXg9IDE0OTEsIGF2Zz03MzcuNTksIHN0ZGV2PTE3Ni44OCwgc2FtcGxl cz01OQ0KICBsYXQgKHVzZWMpICAgOiA1MDA9MTcuODQlLCA3NTA9MTIuNzglLCAxMDAwPTQ4LjQ4 JQ0KICBsYXQgKG1zZWMpICAgOiAyPTE3LjA3JSwgND0xLjQ0JSwgNTA9Mi4zOSUNCiAgY3B1ICAg ICAgICAgIDogdXNyPTAuNDIlLCBzeXM9NTkuMTclLCBjdHg9NzQxMiwgbWFqZj0wLCBtaW5mPTI4 Mg0KICBJTyBkZXB0aHMgICAgOiAxPTEwMC4wJSwgMj0wLjAlLCA0PTAuMCUsIDg9MC4wJSwgMTY9 MC4wJSwgMzI9MC4wJSwgPj02ND0wLjAlDQogICAgIHN1Ym1pdCAgICA6IDA9MC4wJSwgND0xMDAu MCUsIDg9MC4wJSwgMTY9MC4wJSwgMzI9MC4wJSwgNjQ9MC4wJSwgPj02ND0wLjAlDQogICAgIGNv bXBsZXRlICA6IDA9MC4wJSwgND0xMDAuMCUsIDg9MC4wJSwgMTY9MC4wJSwgMzI9MC4wJSwgNjQ9 MC4wJSwgPj02ND0wLjAlDQogICAgIGlzc3VlZCByd3RzOiB0b3RhbD0wLDIyMjM4LDAsMCBzaG9y dD0wLDAsMCwwIGRyb3BwZWQ9MCwwLDAsMA0KICAgICBsYXRlbmN5ICAgOiB0YXJnZXQ9MCwgd2lu ZG93PTAsIHBlcmNlbnRpbGU9MTAwLjAwJSwgZGVwdGg9MQ0Kd3JpdGUtYmFuZHdpZHRoOiAoZ3Jv dXBpZD0wLCBqb2JzPTEpOiBlcnI9IDA6IHBpZD0yNTcxOiBTdW4gQXVnIDEzIDE3OjQ3OjAyIDIw MjMNCiAgd3JpdGU6IElPUFM9NzE4LCBCVz03MTlNaUIvcyAoNzU0TUIvcykoMjEuOEdpQi8zMTA5 OW1zZWMpOyAwIHpvbmUgcmVzZXRzDQogICAgY2xhdCAodXNlYyk6IG1pbj0zMjYsIG1heD0yMzEx NSwgYXZnPTEzMzguMzIsIHN0ZGV2PTMxNjAuNjkNCiAgICAgbGF0ICh1c2VjKTogbWluPTMyNiwg bWF4PTIzMTE1LCBhdmc9MTMzOC43NSwgc3RkZXY9MzE2MC42OQ0KICAgIGNsYXQgcGVyY2VudGls ZXMgKHVzZWMpOg0KICAgICB8ICAxLjAwdGg9WyAgMzUxXSwgIDUuMDB0aD1bICAzODNdLCAxMC4w MHRoPVsgIDQwNF0sIDIwLjAwdGg9WyAgNDc4XSwNCiAgICAgfCAzMC4wMHRoPVsgIDYyN10sIDQw LjAwdGg9WyAgODI0XSwgNTAuMDB0aD1bICA4NjVdLCA2MC4wMHRoPVsgIDg4OV0sDQogICAgIHwg NzAuMDB0aD1bICA5MTRdLCA4MC4wMHRoPVsgMTA3NF0sIDkwLjAwdGg9WyAxNDE4XSwgOTUuMDB0 aD1bIDE2OTZdLA0KICAgICB8IDk5LjAwdGg9WzIxNjI3XSwgOTkuNTB0aD1bMjE4OTBdLCA5OS45 MHRoPVsyMjE1Ml0sIDk5Ljk1dGg9WzIyNjc2XSwNCiAgICAgfCA5OS45OXRoPVsyMjkzOF0NCiAg IGJ3ICggIEtpQi9zKTogbWluPTUwMTc2MCwgbWF4PTE4ODE0MTYsIHBlcj0yNi4yMiUsIGF2Zz03 NTc3NDguMjAsIHN0ZGV2PTIwNjQwNy44OSwgc2FtcGxlcz01OQ0KICAgaW9wcyAgICAgICAgOiBt aW49ICA0OTAsIG1heD0gMTgzNywgYXZnPTczOS45OCwgc3RkZXY9MjAxLjU0LCBzYW1wbGVzPTU5 DQogIGxhdCAodXNlYykgICA6IDUwMD0yMS41NiUsIDc1MD0xMi42MSUsIDEwMDA9NDQuMjAlDQog IGxhdCAobXNlYykgICA6IDI9MTcuNzIlLCA0PTEuNTMlLCAyMD0wLjAxJSwgNTA9Mi4zOCUNCiAg Y3B1ICAgICAgICAgIDogdXNyPTAuMzglLCBzeXM9NjEuODglLCBjdHg9MTczNjQsIG1hamY9MCwg bWluZj0yODQNCiAgSU8gZGVwdGhzICAgIDogMT0xMDAuMCUsIDI9MC4wJSwgND0wLjAlLCA4PTAu MCUsIDE2PTAuMCUsIDMyPTAuMCUsID49NjQ9MC4wJQ0KICAgICBzdWJtaXQgICAgOiAwPTAuMCUs IDQ9MTAwLjAlLCA4PTAuMCUsIDE2PTAuMCUsIDMyPTAuMCUsIDY0PTAuMCUsID49NjQ9MC4wJQ0K ICAgICBjb21wbGV0ZSAgOiAwPTAuMCUsIDQ9MTAwLjAlLCA4PTAuMCUsIDE2PTAuMCUsIDMyPTAu MCUsIDY0PTAuMCUsID49NjQ9MC4wJQ0KICAgICBpc3N1ZWQgcnd0czogdG90YWw9MCwyMjM1Myww LDAgc2hvcnQ9MCwwLDAsMCBkcm9wcGVkPTAsMCwwLDANCiAgICAgbGF0ZW5jeSAgIDogdGFyZ2V0 PTAsIHdpbmRvdz0wLCBwZXJjZW50aWxlPTEwMC4wMCUsIGRlcHRoPTENCndyaXRlLWJhbmR3aWR0 aDogKGdyb3VwaWQ9MCwgam9icz0xKTogZXJyPSAwOiBwaWQ9MjU3MjogU3VuIEF1ZyAxMyAxNzo0 NzowMiAyMDIzDQogIHdyaXRlOiBJT1BTPTcyMCwgQlc9NzIxTWlCL3MgKDc1Nk1CL3MpKDIyLjRH aUIvMzE3NjBtc2VjKTsgMCB6b25lIHJlc2V0cw0KICAgIGNsYXQgKHVzZWMpOiBtaW49MzQxLCBt YXg9MjI5MzIsIGF2Zz0xMzA3LjExLCBzdGRldj0zMjI2LjQwDQogICAgIGxhdCAodXNlYyk6IG1p bj0zNDEsIG1heD0yMjkzMiwgYXZnPTEzMDcuNTIsIHN0ZGV2PTMyMjYuNDENCiAgICBjbGF0IHBl cmNlbnRpbGVzICh1c2VjKToNCiAgICAgfCAgMS4wMHRoPVsgIDM2N10sICA1LjAwdGg9WyAgMzky XSwgMTAuMDB0aD1bICA0MDRdLCAyMC4wMHRoPVsgIDQzN10sDQogICAgIHwgMzAuMDB0aD1bICA1 NTNdLCA0MC4wMHRoPVsgIDcwOV0sIDUwLjAwdGg9WyAgODMyXSwgNjAuMDB0aD1bICA4NzNdLA0K ICAgICB8IDcwLjAwdGg9WyAgOTE0XSwgODAuMDB0aD1bIDEwMzddLCA5MC4wMHRoPVsgMTI3MF0s IDk1LjAwdGg9WyAxNTY1XSwNCiAgICAgfCA5OS4wMHRoPVsyMTM2NV0sIDk5LjUwdGg9WzIxODkw XSwgOTkuOTB0aD1bMjIxNTJdLCA5OS45NXRoPVsyMjE1Ml0sDQogICAgIHwgOTkuOTl0aD1bMjI2 NzZdDQogICBidyAoICBLaUIvcyk6IG1pbj01MTYwNDcsIG1heD0yMjc4Mjc3LCBwZXI9MjYuODEl LCBhdmc9Nzc0NzgzLjI1LCBzdGRldj0yNTc5OTMuMzksIHNhbXBsZXM9NTkNCiAgIGlvcHMgICAg ICAgIDogbWluPSAgNTAzLCBtYXg9IDIyMjQsIGF2Zz03NTYuNTksIHN0ZGV2PTI1MS44Nywgc2Ft cGxlcz01OQ0KICBsYXQgKHVzZWMpICAgOiA1MDA9MjUuOTElLCA3NTA9MTYuNjAlLCAxMDAwPTMz LjI5JQ0KICBsYXQgKG1zZWMpICAgOiAyPTIwLjE0JSwgND0xLjU1JSwgNTA9Mi41MCUNCiAgY3B1 ICAgICAgICAgIDogdXNyPTAuNDUlLCBzeXM9NTYuNzclLCBjdHg9MTMyMDQsIG1hamY9MCwgbWlu Zj0xMg0KICBJTyBkZXB0aHMgICAgOiAxPTEwMC4wJSwgMj0wLjAlLCA0PTAuMCUsIDg9MC4wJSwg MTY9MC4wJSwgMzI9MC4wJSwgPj02ND0wLjAlDQogICAgIHN1Ym1pdCAgICA6IDA9MC4wJSwgND0x MDAuMCUsIDg9MC4wJSwgMTY9MC4wJSwgMzI9MC4wJSwgNjQ9MC4wJSwgPj02ND0wLjAlDQogICAg IGNvbXBsZXRlICA6IDA9MC4wJSwgND0xMDAuMCUsIDg9MC4wJSwgMTY9MC4wJSwgMzI9MC4wJSwg NjQ9MC4wJSwgPj02ND0wLjAlDQogICAgIGlzc3VlZCByd3RzOiB0b3RhbD0wLDIyODk0LDAsMCBz aG9ydD0wLDAsMCwwIGRyb3BwZWQ9MCwwLDAsMA0KICAgICBsYXRlbmN5ICAgOiB0YXJnZXQ9MCwg d2luZG93PTAsIHBlcmNlbnRpbGU9MTAwLjAwJSwgZGVwdGg9MQ0Kd3JpdGUtYmFuZHdpZHRoOiAo Z3JvdXBpZD0wLCBqb2JzPTEpOiBlcnI9IDA6IHBpZD0yNTczOiBTdW4gQXVnIDEzIDE3OjQ3OjAy IDIwMjMNCiAgd3JpdGU6IElPUFM9NzEzLCBCVz03MTNNaUIvcyAoNzQ4TUIvcykoMjIuM0dpQi8z MTk5OG1zZWMpOyAwIHpvbmUgcmVzZXRzDQogICAgY2xhdCAodXNlYyk6IG1pbj0zNDIsIG1heD0y MzA1MCwgYXZnPTEzMTAuNTEsIHN0ZGV2PTMzMDYuNDcNCiAgICAgbGF0ICh1c2VjKTogbWluPTM0 MywgbWF4PTIzMDUwLCBhdmc9MTMxMC45NSwgc3RkZXY9MzMwNi40OA0KICAgIGNsYXQgcGVyY2Vu dGlsZXMgKHVzZWMpOg0KICAgICB8ICAxLjAwdGg9WyAgMzcxXSwgIDUuMDB0aD1bICAzODhdLCAx MC4wMHRoPVsgIDQwNF0sIDIwLjAwdGg9WyAgNDMzXSwNCiAgICAgfCAzMC4wMHRoPVsgIDUzN10s IDQwLjAwdGg9WyAgNjg1XSwgNTAuMDB0aD1bICA4NDhdLCA2MC4wMHRoPVsgIDg4MV0sDQogICAg IHwgNzAuMDB0aD1bICA5MTRdLCA4MC4wMHRoPVsgMTAyOV0sIDkwLjAwdGg9WyAxMTg4XSwgOTUu MDB0aD1bIDE0MDFdLA0KICAgICB8IDk5LjAwdGg9WzIxMzY1XSwgOTkuNTB0aD1bMjE4OTBdLCA5 OS45MHRoPVsyMjE1Ml0sIDk5Ljk1dGg9WzIyMTUyXSwNCiAgICAgfCA5OS45OXRoPVsyMjQxNF0N CiAgIGJ3ICggIEtpQi9zKTogbWluPTUwOTk1MiwgbWF4PTIyOTEzNDIsIHBlcj0yNi43MyUsIGF2 Zz03NzI1MzkuMzIsIHN0ZGV2PTI1ODEwNS45OSwgc2FtcGxlcz01OQ0KICAgaW9wcyAgICAgICAg OiBtaW49ICA0OTgsIG1heD0gMjIzNywgYXZnPTc1NC40MSwgc3RkZXY9MjUyLjAxLCBzYW1wbGVz PTU5DQogIGxhdCAodXNlYykgICA6IDUwMD0yNy44NyUsIDc1MD0xNC41MSUsIDEwMDA9MzQuNjgl DQogIGxhdCAobXNlYykgICA6IDI9MjAuMzAlLCA1MD0yLjY0JQ0KICBjcHUgICAgICAgICAgOiB1 c3I9MC40MCUsIHN5cz01NC43MyUsIGN0eD0xNjUxOSwgbWFqZj0wLCBtaW5mPTEyDQogIElPIGRl cHRocyAgICA6IDE9MTAwLjAlLCAyPTAuMCUsIDQ9MC4wJSwgOD0wLjAlLCAxNj0wLjAlLCAzMj0w LjAlLCA+PTY0PTAuMCUNCiAgICAgc3VibWl0ICAgIDogMD0wLjAlLCA0PTEwMC4wJSwgOD0wLjAl LCAxNj0wLjAlLCAzMj0wLjAlLCA2ND0wLjAlLCA+PTY0PTAuMCUNCiAgICAgY29tcGxldGUgIDog MD0wLjAlLCA0PTEwMC4wJSwgOD0wLjAlLCAxNj0wLjAlLCAzMj0wLjAlLCA2ND0wLjAlLCA+PTY0 PTAuMCUNCiAgICAgaXNzdWVkIHJ3dHM6IHRvdGFsPTAsMjI4MjYsMCwwIHNob3J0PTAsMCwwLDAg ZHJvcHBlZD0wLDAsMCwwDQogICAgIGxhdGVuY3kgICA6IHRhcmdldD0wLCB3aW5kb3c9MCwgcGVy Y2VudGlsZT0xMDAuMDAlLCBkZXB0aD0xDQoNClJ1biBzdGF0dXMgZ3JvdXAgMCAoYWxsIGpvYnMp Og0KICBXUklURTogYnc9MjgyMk1pQi9zICgyOTU5TUIvcyksIDY5N01pQi9zLTcyMU1pQi9zICg3 MzFNQi9zLTc1Nk1CL3MpLCBpbz04OC4yR2lCICg5NC43R0IpLCBydW49MzEwOTktMzE5OThtc2Vj DQo= --------_64D8A6F600000000AA12_MULTIPART_MIXED_ Content-Type: application/octet-stream; name="slow.txt" Content-Disposition: attachment; filename="slow.txt" Content-Transfer-Encoding: base64 KyBmaW8gLW5hbWUgd3JpdGUtYmFuZHdpZHRoIC1ydz13cml0ZSAtYnM9MTAyNEtpIC1zaXplPTMy R2kgLXJ1bnRpbWU9MzAgLWlvZGVwdGggMSAtaW9lbmdpbmUgc3luYyAtemVyb19idWZmZXJzPTEg LWRpcmVjdD0wIC1lbmRfZnN5bmM9MSAtbnVtam9icz00IC1kaXJlY3Rvcnk9L21udC90ZXN0DQp3 cml0ZS1iYW5kd2lkdGg6IChnPTApOiBydz13cml0ZSwgYnM9KFIpIDEwMjRLaUItMTAyNEtpQiwg KFcpIDEwMjRLaUItMTAyNEtpQiwgKFQpIDEwMjRLaUItMTAyNEtpQiwgaW9lbmdpbmU9c3luYywg aW9kZXB0aD0xDQouLi4NCmZpby0zLjE5DQpTdGFydGluZyA0IHByb2Nlc3Nlcw0Kd3JpdGUtYmFu ZHdpZHRoOiBMYXlpbmcgb3V0IElPIGZpbGUgKDEgZmlsZSAvIDMyNzY4TWlCKQ0Kd3JpdGUtYmFu ZHdpZHRoOiBMYXlpbmcgb3V0IElPIGZpbGUgKDEgZmlsZSAvIDMyNzY4TWlCKQ0Kd3JpdGUtYmFu ZHdpZHRoOiBMYXlpbmcgb3V0IElPIGZpbGUgKDEgZmlsZSAvIDMyNzY4TWlCKQ0Kd3JpdGUtYmFu ZHdpZHRoOiBMYXlpbmcgb3V0IElPIGZpbGUgKDEgZmlsZSAvIDMyNzY4TWlCKQ0KSm9iczogNCAo Zj00KTogW0YoNCldWzEwMC4wJV1bdz04OS45TWlCL3NdW3c9ODkgSU9QU11bZXRhIDAwbTowMHNd DQp3cml0ZS1iYW5kd2lkdGg6IChncm91cGlkPTAsIGpvYnM9MSk6IGVycj0gMDogcGlkPTE3OTU6 IFN1biBBdWcgMTMgMTc6Mzc6MjIgMjAyMw0KICB3cml0ZTogSU9QUz0zMjUsIEJXPTMyNk1pQi9z ICgzNDJNQi9zKSgxMC4xR2lCLzMxNzU4bXNlYyk7IDAgem9uZSByZXNldHMNCiAgICBjbGF0ICh1 c2VjKTogbWluPTMzNSwgbWF4PTI4Nzk5LCBhdmc9Mjg5NC4zNCwgc3RkZXY9NTM3My44MQ0KICAg ICBsYXQgKHVzZWMpOiBtaW49MzM1LCBtYXg9Mjg3OTksIGF2Zz0yODk0LjkyLCBzdGRldj01Mzcz LjgxDQogICAgY2xhdCBwZXJjZW50aWxlcyAodXNlYyk6DQogICAgIHwgIDEuMDB0aD1bICAzNTld LCAgNS4wMHRoPVsgIDM3OV0sIDEwLjAwdGg9WyAgNDA0XSwgMjAuMDB0aD1bICA4MzJdLA0KICAg ICB8IDMwLjAwdGg9WyAgODU3XSwgNDAuMDB0aD1bICA4NzNdLCA1MC4wMHRoPVsgIDg4MV0sIDYw LjAwdGg9WyAgODk4XSwNCiAgICAgfCA3MC4wMHRoPVsgIDk2M10sIDgwLjAwdGg9WyAxMzM2XSwg OTAuMDB0aD1bMTYzMTldLCA5NS4wMHRoPVsxNjkwOV0sDQogICAgIHwgOTkuMDB0aD1bMTc5NTdd LCA5OS41MHRoPVsxODc0NF0sIDk5LjkwdGg9WzIxNjI3XSwgOTkuOTV0aD1bMjE2MjddLA0KICAg ICB8IDk5Ljk5dGg9WzIxODkwXQ0KICAgYncgKCAgS2lCL3MpOiBtaW49MjU2MDAwLCBtYXg9MTkw OTE4MCwgcGVyPTI2LjI2JSwgYXZnPTM0NzkyNC41NCwgc3RkZXY9MjIzNTUzLjU2LCBzYW1wbGVz PTU5DQogICBpb3BzICAgICAgICA6IG1pbj0gIDI1MCwgbWF4PSAxODY0LCBhdmc9MzM5Ljc2LCBz dGRldj0yMTguMjYsIHNhbXBsZXM9NTkNCiAgbGF0ICh1c2VjKSAgIDogNTAwPTEyLjExJSwgNzUw PTQuNjYlLCAxMDAwPTU3LjIyJQ0KICBsYXQgKG1zZWMpICAgOiAyPTEzLjMxJSwgND0wLjE2JSwg MjA9MTIuMTklLCA1MD0wLjM2JQ0KICBjcHUgICAgICAgICAgOiB1c3I9MC4xOCUsIHN5cz0zMS42 MCUsIGN0eD0xNDM1LCBtYWpmPTAsIG1pbmY9MTENCiAgSU8gZGVwdGhzICAgIDogMT0xMDAuMCUs IDI9MC4wJSwgND0wLjAlLCA4PTAuMCUsIDE2PTAuMCUsIDMyPTAuMCUsID49NjQ9MC4wJQ0KICAg ICBzdWJtaXQgICAgOiAwPTAuMCUsIDQ9MTAwLjAlLCA4PTAuMCUsIDE2PTAuMCUsIDMyPTAuMCUs IDY0PTAuMCUsID49NjQ9MC4wJQ0KICAgICBjb21wbGV0ZSAgOiAwPTAuMCUsIDQ9MTAwLjAlLCA4 PTAuMCUsIDE2PTAuMCUsIDMyPTAuMCUsIDY0PTAuMCUsID49NjQ9MC4wJQ0KICAgICBpc3N1ZWQg cnd0czogdG90YWw9MCwxMDM0OCwwLDAgc2hvcnQ9MCwwLDAsMCBkcm9wcGVkPTAsMCwwLDANCiAg ICAgbGF0ZW5jeSAgIDogdGFyZ2V0PTAsIHdpbmRvdz0wLCBwZXJjZW50aWxlPTEwMC4wMCUsIGRl cHRoPTENCndyaXRlLWJhbmR3aWR0aDogKGdyb3VwaWQ9MCwgam9icz0xKTogZXJyPSAwOiBwaWQ9 MTc5NjogU3VuIEF1ZyAxMyAxNzozNzoyMiAyMDIzDQogIHdyaXRlOiBJT1BTPTMyNywgQlc9MzI4 TWlCL3MgKDM0NE1CL3MpKDEwLjFHaUIvMzE1NzZtc2VjKTsgMCB6b25lIHJlc2V0cw0KICAgIGNs YXQgKHVzZWMpOiBtaW49MzM3LCBtYXg9Mjg4ODcsIGF2Zz0yODk0Ljk3LCBzdGRldj01NDUzLjc3 DQogICAgIGxhdCAodXNlYyk6IG1pbj0zMzcsIG1heD0yODg4OCwgYXZnPTI4OTUuNTcsIHN0ZGV2 PTU0NTMuNzYNCiAgICBjbGF0IHBlcmNlbnRpbGVzICh1c2VjKToNCiAgICAgfCAgMS4wMHRoPVsg IDM1OV0sICA1LjAwdGg9WyAgMzgzXSwgMTAuMDB0aD1bICA0MDRdLCAyMC4wMHRoPVsgIDc3NV0s DQogICAgIHwgMzAuMDB0aD1bICA4MzJdLCA0MC4wMHRoPVsgIDg0MF0sIDUwLjAwdGg9WyAgODU3 XSwgNjAuMDB0aD1bICA4NjVdLA0KICAgICB8IDcwLjAwdGg9WyAgOTM4XSwgODAuMDB0aD1bIDEw MjBdLCA5MC4wMHRoPVsxNjQ1MF0sIDk1LjAwdGg9WzE2OTA5XSwNCiAgICAgfCA5OS4wMHRoPVsx ODIyMF0sIDk5LjUwdGg9WzE5MDA2XSwgOTkuOTB0aD1bMjE2MjddLCA5OS45NXRoPVsyMTg5MF0s DQogICAgIHwgOTkuOTl0aD1bMjIxNTJdDQogICBidyAoICBLaUIvcyk6IG1pbj0yNjAwOTYsIG1h eD0xOTAxMDE0LCBwZXI9MjYuMjUlLCBhdmc9MzQ3Nzg2LjE0LCBzdGRldj0yMjIwNjEuMjYsIHNh bXBsZXM9NTkNCiAgIGlvcHMgICAgICAgIDogbWluPSAgMjU0LCBtYXg9IDE4NTYsIGF2Zz0zMzku NjMsIHN0ZGV2PTIxNi44MCwgc2FtcGxlcz01OQ0KICBsYXQgKHVzZWMpICAgOiA1MDA9MTIuNDAl LCA3NTA9Ni40OSUsIDEwMDA9NTguMTAlDQogIGxhdCAobXNlYykgICA6IDI9MTAuMTUlLCAyMD0x Mi41NyUsIDUwPTAuMzAlDQogIGNwdSAgICAgICAgICA6IHVzcj0wLjIwJSwgc3lzPTI5LjQwJSwg Y3R4PTE0NDEsIG1hamY9MCwgbWluZj0xMg0KICBJTyBkZXB0aHMgICAgOiAxPTEwMC4wJSwgMj0w LjAlLCA0PTAuMCUsIDg9MC4wJSwgMTY9MC4wJSwgMzI9MC4wJSwgPj02ND0wLjAlDQogICAgIHN1 Ym1pdCAgICA6IDA9MC4wJSwgND0xMDAuMCUsIDg9MC4wJSwgMTY9MC4wJSwgMzI9MC4wJSwgNjQ9 MC4wJSwgPj02ND0wLjAlDQogICAgIGNvbXBsZXRlICA6IDA9MC4wJSwgND0xMDAuMCUsIDg9MC4w JSwgMTY9MC4wJSwgMzI9MC4wJSwgNjQ9MC4wJSwgPj02ND0wLjAlDQogICAgIGlzc3VlZCByd3Rz OiB0b3RhbD0wLDEwMzQ2LDAsMCBzaG9ydD0wLDAsMCwwIGRyb3BwZWQ9MCwwLDAsMA0KICAgICBs YXRlbmN5ICAgOiB0YXJnZXQ9MCwgd2luZG93PTAsIHBlcmNlbnRpbGU9MTAwLjAwJSwgZGVwdGg9 MQ0Kd3JpdGUtYmFuZHdpZHRoOiAoZ3JvdXBpZD0wLCBqb2JzPTEpOiBlcnI9IDA6IHBpZD0xNzk3 OiBTdW4gQXVnIDEzIDE3OjM3OjIyIDIwMjMNCiAgd3JpdGU6IElPUFM9MzE3LCBCVz0zMTdNaUIv cyAoMzMyTUIvcykoOS44MkdpQi8zMTcxNW1zZWMpOyAwIHpvbmUgcmVzZXRzDQogICAgY2xhdCAo dXNlYyk6IG1pbj0zMjcsIG1heD0yMjU1NywgYXZnPTI5NzguNjcsIHN0ZGV2PTU0MzQuNzQNCiAg ICAgbGF0ICh1c2VjKTogbWluPTMyNywgbWF4PTIyNTU3LCBhdmc9Mjk3OS4yNywgc3RkZXY9NTQz NC43Mw0KICAgIGNsYXQgcGVyY2VudGlsZXMgKHVzZWMpOg0KICAgICB8ICAxLjAwdGg9WyAgMzQ3 XSwgIDUuMDB0aD1bICAzOTJdLCAxMC4wMHRoPVsgIDg0OF0sIDIwLjAwdGg9WyAgODY1XSwNCiAg ICAgfCAzMC4wMHRoPVsgIDg2NV0sIDQwLjAwdGg9WyAgODczXSwgNTAuMDB0aD1bICA4ODFdLCA2 MC4wMHRoPVsgIDg5OF0sDQogICAgIHwgNzAuMDB0aD1bICA5MTRdLCA4MC4wMHRoPVsgMTQwMV0s IDkwLjAwdGg9WzE2NDUwXSwgOTUuMDB0aD1bMTc2OTVdLA0KICAgICB8IDk5LjAwdGg9WzE3OTU3 XSwgOTkuNTB0aD1bMTg0ODJdLCA5OS45MHRoPVsyMTg5MF0sIDk5Ljk1dGg9WzIyMTUyXSwNCiAg ICAgfCA5OS45OXRoPVsyMjQxNF0NCiAgIGJ3ICggIEtpQi9zKTogbWluPTI1ODA0OCwgbWF4PTE0 MTkyMjgsIHBlcj0yNS42NCUsIGF2Zz0zMzk3NTkuMTIsIHN0ZGV2PTE2NjA0My4zMSwgc2FtcGxl cz01OQ0KICAgaW9wcyAgICAgICAgOiBtaW49ICAyNTIsIG1heD0gMTM4NSwgYXZnPTMzMS43OCwg c3RkZXY9MTYyLjA0LCBzYW1wbGVzPTU5DQogIGxhdCAodXNlYykgICA6IDUwMD03LjIzJSwgNzUw PTAuNTElLCAxMDAwPTY3LjI4JQ0KICBsYXQgKG1zZWMpICAgOiAyPTEyLjI0JSwgND0wLjAzJSwg MjA9MTIuNDElLCA1MD0wLjMwJQ0KICBjcHUgICAgICAgICAgOiB1c3I9MC4xMCUsIHN5cz0zMC44 MiUsIGN0eD0xMzM0LCBtYWpmPTAsIG1pbmY9Mjg1DQogIElPIGRlcHRocyAgICA6IDE9MTAwLjAl LCAyPTAuMCUsIDQ9MC4wJSwgOD0wLjAlLCAxNj0wLjAlLCAzMj0wLjAlLCA+PTY0PTAuMCUNCiAg ICAgc3VibWl0ICAgIDogMD0wLjAlLCA0PTEwMC4wJSwgOD0wLjAlLCAxNj0wLjAlLCAzMj0wLjAl LCA2ND0wLjAlLCA+PTY0PTAuMCUNCiAgICAgY29tcGxldGUgIDogMD0wLjAlLCA0PTEwMC4wJSwg OD0wLjAlLCAxNj0wLjAlLCAzMj0wLjAlLCA2ND0wLjAlLCA+PTY0PTAuMCUNCiAgICAgaXNzdWVk IHJ3dHM6IHRvdGFsPTAsMTAwNTUsMCwwIHNob3J0PTAsMCwwLDAgZHJvcHBlZD0wLDAsMCwwDQog ICAgIGxhdGVuY3kgICA6IHRhcmdldD0wLCB3aW5kb3c9MCwgcGVyY2VudGlsZT0xMDAuMDAlLCBk ZXB0aD0xDQp3cml0ZS1iYW5kd2lkdGg6IChncm91cGlkPTAsIGpvYnM9MSk6IGVycj0gMDogcGlk PTE3OTg6IFN1biBBdWcgMTMgMTc6Mzc6MjIgMjAyMw0KICB3cml0ZTogSU9QUz0zMjUsIEJXPTMy Nk1pQi9zICgzNDJNQi9zKSgxMC4xR2lCLzMxNzUxbXNlYyk7IDAgem9uZSByZXNldHMNCiAgICBj bGF0ICh1c2VjKTogbWluPTMyNywgbWF4PTIyMjU5LCBhdmc9Mjg5NS45NSwgc3RkZXY9NTM5NS44 OA0KICAgICBsYXQgKHVzZWMpOiBtaW49MzI3LCBtYXg9MjIyNTksIGF2Zz0yODk2LjUzLCBzdGRl dj01Mzk1Ljg3DQogICAgY2xhdCBwZXJjZW50aWxlcyAodXNlYyk6DQogICAgIHwgIDEuMDB0aD1b ICAzNTVdLCAgNS4wMHRoPVsgIDM4OF0sIDEwLjAwdGg9WyAgNTA2XSwgMjAuMDB0aD1bICA4NTdd LA0KICAgICB8IDMwLjAwdGg9WyAgODY1XSwgNDAuMDB0aD1bICA4NzNdLCA1MC4wMHRoPVsgIDg4 MV0sIDYwLjAwdGg9WyAgODg5XSwNCiAgICAgfCA3MC4wMHRoPVsgIDkwNl0sIDgwLjAwdGg9WyAx MjU0XSwgOTAuMDB0aD1bMTY0NTBdLCA5NS4wMHRoPVsxNzY5NV0sDQogICAgIHwgOTkuMDB0aD1b MTc5NTddLCA5OS41MHRoPVsxODQ4Ml0sIDk5LjkwdGg9WzIxNjI3XSwgOTkuOTV0aD1bMjE4OTBd LA0KICAgICB8IDk5Ljk5dGg9WzIyMTUyXQ0KICAgYncgKCAgS2lCL3MpOiBtaW49MjYwMDk2LCBt YXg9MTgyNTg4OCwgcGVyPTI2LjI3JSwgYXZnPTM0ODA3NC44NSwgc3RkZXY9MjE1ODk2LjQ3LCBz YW1wbGVzPTU5DQogICBpb3BzICAgICAgICA6IG1pbj0gIDI1NCwgbWF4PSAxNzgzLCBhdmc9MzM5 LjkyLCBzdGRldj0yMTAuODMsIHNhbXBsZXM9NTkNCiAgbGF0ICh1c2VjKSAgIDogNTAwPTkuNzkl LCA3NTA9MS43NiUsIDEwMDA9NjQuMzglDQogIGxhdCAobXNlYykgICA6IDI9MTEuNjAlLCA0PTAu MDclLCAyMD0xMi4wOSUsIDUwPTAuMzIlDQogIGNwdSAgICAgICAgICA6IHVzcj0wLjE1JSwgc3lz PTMxLjMxJSwgY3R4PTIzNjcsIG1hamY9MCwgbWluZj0yODQNCiAgSU8gZGVwdGhzICAgIDogMT0x MDAuMCUsIDI9MC4wJSwgND0wLjAlLCA4PTAuMCUsIDE2PTAuMCUsIDMyPTAuMCUsID49NjQ9MC4w JQ0KICAgICBzdWJtaXQgICAgOiAwPTAuMCUsIDQ9MTAwLjAlLCA4PTAuMCUsIDE2PTAuMCUsIDMy PTAuMCUsIDY0PTAuMCUsID49NjQ9MC4wJQ0KICAgICBjb21wbGV0ZSAgOiAwPTAuMCUsIDQ9MTAw LjAlLCA4PTAuMCUsIDE2PTAuMCUsIDMyPTAuMCUsIDY0PTAuMCUsID49NjQ9MC4wJQ0KICAgICBp c3N1ZWQgcnd0czogdG90YWw9MCwxMDM0MiwwLDAgc2hvcnQ9MCwwLDAsMCBkcm9wcGVkPTAsMCww LDANCiAgICAgbGF0ZW5jeSAgIDogdGFyZ2V0PTAsIHdpbmRvdz0wLCBwZXJjZW50aWxlPTEwMC4w MCUsIGRlcHRoPTENCg0KUnVuIHN0YXR1cyBncm91cCAwIChhbGwgam9icyk6DQogIFdSSVRFOiBi dz0xMjk0TWlCL3MgKDEzNTdNQi9zKSwgMzE3TWlCL3MtMzI4TWlCL3MgKDMzMk1CL3MtMzQ0TUIv cyksIGlvPTQwLjFHaUIgKDQzLjFHQiksIHJ1bj0zMTU3Ni0zMTc1OG1zZWMNCg== --------_64D8A6F600000000AA12_MULTIPART_MIXED_--