From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751890AbbIAXao (ORCPT ); Tue, 1 Sep 2015 19:30:44 -0400 Received: from mail-by2on0146.outbound.protection.outlook.com ([207.46.100.146]:45826 "EHLO na01-by2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1750856AbbIAXam (ORCPT ); Tue, 1 Sep 2015 19:30:42 -0400 X-Greylist: delayed 1195 seconds by postgrey-1.27 at vger.kernel.org; Tue, 01 Sep 2015 19:30:42 EDT Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=uwe.koziolek@redknee.com; Subject: Re: [PATCH] net/bonding: send arp in interval if no active slave To: Jarod Wilson , Jay Vosburgh References: <1439828583-27325-1-git-send-email-jarod@redhat.com> <20150817165500.GA21512@vps.falico.eu> <55D215F7.3080905@redhat.com> <55D22E64.6020807@redknee.com> <2649.1439838866@famine> <55D2494F.3020800@redknee.com> <55E4D35B.4090502@redhat.com> CC: Veaceslav Falico , , Andy Gospodarek , From: Uwe Koziolek Message-ID: <55E63195.400@redknee.com> Date: Wed, 2 Sep 2015 01:15:33 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: <55E4D35B.4090502@redhat.com> Content-Type: text/plain; charset="iso-8859-15"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [84.132.46.233] X-ClientProxiedBy: VI1PR06CA0013.eurprd06.prod.outlook.com (25.162.116.151) To CO2PR0501MB901.namprd05.prod.outlook.com (10.141.247.16) X-Microsoft-Exchange-Diagnostics: 1;CO2PR0501MB901;2:NoFAnjESBVuNPOjADxycexQjcdGdrFDtg0fRUGGRzHgZcZlcZkjOSMOq8318QT1ET4pNKTMH9WwHe7FBJ4GzZrMmSY3NCkRDl91l7UfFFX4BJxxJHNy84eoaqUYfItVx1jB5h1YsyfraRXRvCDxD1TA3iYd1yPFeSrf/GVLGWX4=;3:WRwWohPXgSGcOyjVEHL7yV5KZ+VONmghh7levXXV6YQN0YzQ64DPFxr0C0YQst0i8dS1kcq6OQwVRLLqAthJ9c3DOKPWtzjY0eZQnfJBWzihj0Sd5eAUrlVab0Qd5DxjMhwCj1AKX34cRyvS7I/Zdg==;25:BV8JM3OCdE8otwn3ypWj7aEQB+0syQ+Pe5LYc0b285/LJOrtTzrb/7ErHDWzpdX3Rskp/IR5OzGIV5HD44Uxs4iRrJFhp1Oz6DTsmbe+9CRAH3P5U2ScDiVqgaUTmY1rG7SjhPYadeAQog3XhKTX3x9ArOPjteAGzLJjqobCNOfjV76HLtHUYxGwYXYaCQPRjXIvzS6NvcLPNL6fy8FsusAJ2tU7p8BedtUlVIG3hFSeJZBPc3xKqqQE0u6lBNALyqpZtLx+AezorTmNg3cRXQ== X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:CO2PR0501MB901; X-Microsoft-Exchange-Diagnostics: 1;CO2PR0501MB901;20:WkGxhBKTHIxvAFI2YP+VRvEoTSLMeis6gms+h//R5v4VvcKqjpkw11l6mp+m2P66aD1c5MJW2dW4Bw7QPoB2h50ruJ2o1QmXZgIKx2t165Isp5qoJhe9jy1Jxcrm6iSHlqbCNDBmLzxQ3TUQxD1Jp5pxPuAfWi1d53lP8cZaitbcFpOL9/PeAujp81T0qISdCv5Mp3nPzw7nr/9X/rSL0w8EaNGL770YJSc/dmET94BemNvWjWjn4LRxZvJ6tCrwzBVg06bW4iCpQUB3+ju2UzaDeUeiw5Uysifh/S8DfAgVj3+UtdckUxfG7jZ6LQzK6cekjFoTmGWevXMTL6xmLrjIqm/kOedtRnnoiQ0OpkrvO9S/HMWSJvVZsxPFyQ9hSvBj7haSZWYZTbZoJGWlZ+N/M1KSQy3iA4x9cWCTmL74fnnTjxZxv04xxWe4jhvHtbw4FMOcPIfi8dfmkQLNZ7XCEsG4mwGg3wN4IJcPvisOueWwtfuIB5579IepPWws;4:AFykhKDLcyAWdmjRTPiGvD38MJJ/6vfd3dy8/OMNaLSUCHXGNGDZoOYX3sixtcCdtX5gkApCOsfcJsZWKfcsJlAD+/D6kLoUnNh5VCPiH+dGhnAkRhCuQz8tcEsU4iIlUdeVlRYDDDIWuG35zR2ntPAJR+SIB0Jew/PivhuRuCq1vpttfoTM8otgcy9AS5UdH9WG3haH+mRXJsT0UoVV14AoZ21zDpcZiv5Rp3HAaBTIXScw7P94ARdLx7Jy6OXNkTZcPWAy55a5XPxwANaf38FA//R+2WaZwQfa4p74H/6C1ZC5f8lKmRamJt22OwJZawQU4RVG8n4CrLHjDdu18A== X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(601004)(5005006)(8121501046)(3002001);SRVR:CO2PR0501MB901;BCL:0;PCL:0;RULEID:;SRVR:CO2PR0501MB901; X-Forefront-PRVS: 06860EDC7B X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10019020)(6049001)(6009001)(377454003)(377424004)(189002)(199003)(24454002)(86362001)(92566002)(68736005)(46102003)(101416001)(2950100001)(54356999)(77156002)(76176999)(42186005)(93886004)(50466002)(77096005)(5001830100001)(65816999)(4001350100001)(36756003)(5001770100001)(19580405001)(83506001)(5001860100001)(19580395003)(65956001)(50986999)(64126003)(81156007)(5001960100002)(87976001)(4001540100001)(62966003)(64706001)(23756003)(40100003)(33656002)(65806001)(66066001)(122386002)(5007970100001)(97736004)(105586002)(117156001)(189998001)(47776003)(106356001)(5001920100001)(5004730100002);DIR:OUT;SFP:1102;SCL:1;SRVR:CO2PR0501MB901;H:[192.168.222.3];FPR:;SPF:None;PTR:InfoNoRecords;MX:1;A:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?iso-8859-15?Q?1;CO2PR0501MB901;23:csKj6/d3crEVzSR9mW3xR0MuhJMSe+BoGeuBI?= =?iso-8859-15?Q?sj/S6Gex7569PybXKm7JZkC87mymziUFoDYiPuOdO9wg+KAypbOpT/2OE?= =?iso-8859-15?Q?tVtk0VYwNQOyg4O+OY2qiANOXm0w3FD8tbwBK7zmtEM5e2a5ewogSUllo?= =?iso-8859-15?Q?/gr9JrMcJpZ80vVTzTQakjR9q9g01ocX2glyaDUzq+Z0l0Qya0tQaJBdO?= =?iso-8859-15?Q?1H1yhFaC3gM4C0WBrOssQnLiQGiL+rYIxvN3DuKkmOX31D++6yt6AqieJ?= =?iso-8859-15?Q?0ysG79kQUT1OSzOisJbUfMwjIwKxsqsNx5wQe4goKhW8ln6877UB2vpan?= =?iso-8859-15?Q?DVsJ90rLnSFU0QQ/vu+Zyl3qBMXZt190WS5HYsz1Xv3I8hnhFGSYT/VSz?= =?iso-8859-15?Q?y3NmKjkBhbxi+rQnJj+KC8r75LR0GO7pJUS7vIcKWeGKDbSHLitM1m/a0?= =?iso-8859-15?Q?Ct6+MLxZ94Os8cX0+dQ+Si7FIijAyFi/9N3ZmETQgQL/hNhTbUSlVGfa7?= =?iso-8859-15?Q?xXI2/gBsbBWag2ko8tBeNxx2r1A2LN5K+uUxsdGrnjfkujc5ArUYbp8VT?= =?iso-8859-15?Q?imOZVeCHkEuNpFp0E1KUPnY2H1+78ledfR8d6tBhzjiqc5QRIsiJv2aTr?= =?iso-8859-15?Q?HA8zGwCe0y2B0Mo4S78RoSSyZf4UpdOmJ1AkPsqnrtvvYZhLN/T2AW3l5?= =?iso-8859-15?Q?SAobH+nNbXYScwFXYbZpy3q/U+qS/Z/t+JYrzFhCdKfRe8zHMwi5HAhCd?= =?iso-8859-15?Q?CV6D+xiXdWRyRyl6xUqsCkk2CadoCE5qY2nhF1Q0n0UsNA5SMELjhTVZ3?= =?iso-8859-15?Q?fse4U0tB9AF/sYxUWwdy+xVtDUX6Py4PVyznyXMS9MVElxK4KO2Fv6seX?= =?iso-8859-15?Q?cOtJeqks1st2U4w+PJSEf9Ppw0XL9DPpfSxiVZB6GJcfcqgTc3yjiMpkY?= =?iso-8859-15?Q?dLQZCNi2twSP4FB0M5rkgOGfUo0LTGxUi7QfJZcPh3zh18x9zw5MG7Odu?= =?iso-8859-15?Q?6e7MG/35otls70QDtgmDYfjPGWsS+TTXe9YMfjv0Qd2a4syvZqx6iXIQM?= =?iso-8859-15?Q?O4jmuKt8JIjOUzqlMggRRk9tL8IIT8sqT2M7/I+kYPewjTQCFjjKenxjd?= =?iso-8859-15?Q?hS4z5w71wOQXMwq1tAcezuV7hra/t+FsklrB5OaKI9uCqhgdaz+KXUkkp?= =?iso-8859-15?Q?gJadoMAoqf/24QOG+M7IiMa5DqhlRjvvbgLyUoiwP+cczuvtT+QswiMOH?= =?iso-8859-15?Q?PecknX6OsnpHavS1Mg1f1EgC03OKs7qGOIPH1MGxwUALUye7i2w3Sicu6?= =?iso-8859-15?Q?qanjbpFd/nXoaBegwS1TFP7/UCFMEWFuGnkD3R8PT4V7m954dD4oHz19A?= =?iso-8859-15?Q?4AatS+3j6c7K1D5EINNKj9XnU48phCi1WQpWiLXuLVexydVb9STdepG21?= =?iso-8859-15?Q?4sCrolZcwt4kaxBytzFCWNOh2gdM3q0a/m2FsQzLzSMNSp6+aqTtHf/Sk?= =?iso-8859-15?Q?+yHKIZnLO5Q8iPGYLNyZEEfTQ=3D=3D?= X-Microsoft-Exchange-Diagnostics: 1;CO2PR0501MB901;5:opBVTqx65Y1TBjMsWJPa2LAVC7RQ9kSe88Q/xxlWbVcZuPTZKWj7LgGR8SAkmi2keRuGU8/YwaQ7U2HjTPfl2+TLtMpcMI814ti3h5ppyI0+NkXcOYaNaD6rFZzG1Le17wZuNNR55dqxYHz1rz5Mqw==;24:UqubloeWG2JAHKY5v5exPP8EO+4mKi0CeMKiPpRGqlIYfpmYTKVdiVz0H+rfqTXgsO4/mmFUXZxploU+A7vnXb+YdnoQhz0Bx9m+i4xoSpA=;20:RuNIRAJanCbod5SaH1IgTwsmsCjW+zB3N56jl0L3iEeMcKEn/v8aak4cU1X5ZAKZNEoaaNBVhsgc/Ge6sGgRig== SpamDiagnosticOutput: 1:23 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: redknee.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Sep 2015 23:15:48.7961 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO2PR0501MB901 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, 01.09.2015 at 00:21 +0200 Jarod Wilson wrote: > On 2015-08-17 4:51 PM, Uwe Koziolek wrote: >> On Mon, Aug 17, 2015 at 09:14PM +0200, Jay Vosburgh wrote: >>> Uwe Koziolek wrote: >>> >>>> On2015-08-17 07:12 PM,Jarod Wilson wrote: > ... >>>>> Uwe, can you perhaps further enlighten us as to what num_grat_arp >>>>> settings were tried that didn't help? I'm still of the mind that if >>>>> num_grat_arp *didn't* help, we probably need to do something keyed >>>>> off >>>>> num_grat_arp. >>>> The bonding slaves are connected to high available switches, each >>>> of the >>>> slaves is connected to a different switch. If the bond is starting, >>>> only >>>> the selected slave sends one arp-request. If a matching >>>> arp_response was >>>> received, this slave and the bond is going into state up, sending the >>>> gratitious arps... >>>> But if you got no arp reply the next slave was selected. >>>> With most of the newer switches, not overloaded, or with other >>>> software >>>> bugs, or with a single switch configuration, you would get a arp >>>> response >>>> on the first arp request. >>>> But in case of high availability configuration with non perfect >>>> switches >>>> like HP ProCurve 54xx, also with some Cisco models, you may not get a >>>> response on the first arp request. >>>> >>>> I have seen network snoops, there the switches are not responding >>>> to the >>>> first arp request on slave 1, the second arp request was sent on >>>> slave 2 >>>> but the response was received on slave one, and all following arp >>>> requests are anwsered on the wrong slave for a longer time. >>> Could you elaborate on the exact "high availability >>> configuration" here, including the model(s) of switch(es) involved? >>> >>> Is this some kind of race between the switch or switches >>> updating the forwarding tables and the bond flip flopping between the >>> slaves? E.g., source MAC from ARP sent on slave 1 is used to populate >>> the forwarding table, but (for whatever reason) there is no reply. ARP >>> on slave 2 is sent (using the same source MAC, unless you set >>> fail_over_mac), but forwarding tables still send that MAC to slave >>> 1, so >>> reply is sent there. >> High availability: >> 2 managed switches with routing capabilities have an interconnect. >> One slave of a bonding interface is connected to the first switch, the >> second slave is connected to the other switch. >> The switch models are HP ProCurve 5406 and HP ProCurve 5412. As far as i >> remember also HP E 3500 and E 3800 are also >> affected, for the affected Cisco models I can't answer today. >> Affected single switch configurations was not seen. >> >> Yes, race conditions with delayed upgrades of the forwarding tables is a >> well matching explanation for the problem. >> >>>> The proposed change sents up to 3 arp requests on a down bond using >>>> the >>>> same slave, delayed by arp_interval. >>>> Using problematic switches i have seen the the arp response on the >>>> right >>>> slave at latest on the second arp request. So the bond is going into >>>> state >>>> up. >>>> >>>> How does it works: >>>> The bonds in up state are handled on the beginning of >>>> bond_ab_arp_probe >>>> procedure, the other part of this procedure is handling the slave >>>> change. >>>> The proposed change is bypassing the slave change for 2 additional >>>> calls >>>> of bond_ab_arp_probe. >>>> Now the retries are not only for an up bond available, they are also >>>> implemented for a down bond. >>> Does this delay failover or bringup on switches that are not >>> "problematic"? I.e., if arp_interval is, say, 1000 (1 second), will >>> this impact failover / recovery times? >>> >>> -J >> It depends. >> failover times are not impacted, this is handled different. >> Only the transition from a down bonding interface (bond and all slaves >> are down) to the state up can be increased by up to 2 times >> arp_interval, >> If the selected interface did not came up .If well working switches are >> used, and everything other is also ok, there are no impacts. > > Jay, any further thoughts on this given Uwe's reply? Uwe, did you have > a chance to get affected Cisco model numbers too? > The affected Cisco model was a C3750.