* How to verify multichannel [not found] <1371710722.29134084.1589816497911.JavaMail.zimbra@redhat.com> @ 2020-05-18 16:01 ` Xiaoli Feng 2020-05-18 17:17 ` Aurélien Aptel 0 siblings, 1 reply; 5+ messages in thread From: Xiaoli Feng @ 2020-05-18 16:01 UTC (permalink / raw) To: linux-cifs Hello, When I test multichannel. I don't know how to verify if it works. I just catch network packages to see if server and client communicate with multiple IP. But from my test, it doesn't. Does any one know how to verify multichannel? Setup: server is samba server in linux upstream. client is linux upstream. smb.conf is: [global] interfaces = eth1, eth2, team0 server multi channel support = yes vfs objects = recycle aio_pthread aio read size = 1 aio write size = 1 [cifs] path=/mnt/cifs writeable=yes Create network team in server and client. eth1 and eth2 work as team slave. Mount with multichannel option(use one of team ip to mount). Then use "dd" to create 1G file. The used time are the same for multichannel and no multichannel. Compare the time for copying 1G file. The result are also the same. By the way, network team running mode is round-robin. Thanks. -- Best regards! XiaoLi Feng 冯小丽 ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: How to verify multichannel 2020-05-18 16:01 ` How to verify multichannel Xiaoli Feng @ 2020-05-18 17:17 ` Aurélien Aptel 2020-05-19 16:15 ` Xiaoli Feng 0 siblings, 1 reply; 5+ messages in thread From: Aurélien Aptel @ 2020-05-18 17:17 UTC (permalink / raw) To: Xiaoli Feng, linux-cifs Hi Xiaoli, Xiaoli Feng <xifeng@redhat.com> writes: > Hello, > > When I test multichannel. I don't know how to verify if it works. I just catch network > packages to see if server and client communicate with multiple IP. But from my test, it > doesn't. Does any one know how to verify multichannel? Testing if it works vs checking speed improvement are 2 different things unfortunately. Source code =========== First you need the latest patchset which is available on my github remote: https://github.com/aaptel/linux.git You can use the "multichan-page" branch or you can extract the last patches of it (with "multichannel:" in the title). These patches add the multichannel support to the main read/write codepaths from memory pages. Client setup ============ Then you need to mount with "vers=3.11,multichannel,max_channels=N" where N will be the total number of channels the client will try to open. Do a network capture of the mount and run couple of commands in the mount point (ls, cat, stat, ...). If the client also has multiple interface, make sure you capture all interfaces traffic (otherwise you might ignore the other channels). Interface list check ==================== Look at the trace in wireshark and use "smb2" filter. Look for "Ioctl Response FSCTL_QUERY_NETWORK_INTERFACE_INFO", if the server has multiple interfaces you should see a list. You can also look at /proc/fs/cifs/DebugDate to see the list of interfaces the client saw and has connected to. Multichannel check ================== Still in wireshark: - right-click on the first "Negotiate protocol" packet - "Colorize conversation" > "F5 TCP" You can repeat this for each channel to see clearly different colors for each channels. I have made some screenshots, maybe this can help: https://imgur.com/a/j3IBewF You can see the alternating colors in my traces that shows different channels being used. Speedup check ============= Now this is the hard part. If you test with VMs and the interface are "virtual", when you make 2 channels you are actually just dividing by 2 the bandwidth of the virtual bus, so you won't see much speedup. To see a speedup you need to limit the bandwidth of the server interface, for example 1MB/s to each, so that together with 2 channel you might see 2MB/s. You can look into iperf to check raw bandwidth and tc to limit it: - https://iperf.fr/ - https://netbeez.net/blog/how-to-use-the-linux-traffic-control/ Also similarly if the VM only has 1 CPU you might not see much improvement as it will switch from one NIC to the other sequentially instead of in parallel. This gets tricky very quick... > Setup: > server is samba server in linux upstream. > client is linux upstream. > smb.conf is: > [global] > interfaces = eth1, eth2, team0 > server multi channel support = yes > vfs objects = recycle aio_pthread > aio read size = 1 > aio write size = 1 > [cifs] > path=/mnt/cifs > writeable=yes This looks ok but I've mostly used Windows Server 2019 in my test. Let me know how it goes. Cheers, -- Aurélien Aptel / SUSE Labs Samba Team GPG: 1839 CB5F 9F5B FB9B AA97 8C99 03C8 A49B 521B D5D3 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg, DE GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 247165 (AG München) ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: How to verify multichannel 2020-05-18 17:17 ` Aurélien Aptel @ 2020-05-19 16:15 ` Xiaoli Feng 2020-05-19 16:37 ` Steve French 2020-05-19 16:54 ` Aurélien Aptel 0 siblings, 2 replies; 5+ messages in thread From: Xiaoli Feng @ 2020-05-19 16:15 UTC (permalink / raw) To: Aurélien Aptel; +Cc: linux-cifs Thanks Aurélien for your professional information. It's very detailed and useful. Now I can see the multiple channel info in network package when mount with option "max_channel=2". If doesn't specify it. Client will only open one channel. And my smb.conf setup below can work for multichannel. Seems server and client don't require multiple network interfaces. And also don't need network team. But When test seedup, it isn't change. I use two vm in the same host. And each have 4 cpu and 1G memory. Maybe it's the problem. Thanks so much. ----- Original Message ----- > From: "Aurélien Aptel" <aaptel@suse.com> > To: "Xiaoli Feng" <xifeng@redhat.com>, linux-cifs@vger.kernel.org > Sent: Tuesday, May 19, 2020 1:17:11 AM > Subject: Re: How to verify multichannel > > Hi Xiaoli, > > Xiaoli Feng <xifeng@redhat.com> writes: > > Hello, > > > > When I test multichannel. I don't know how to verify if it works. I just > > catch network > > packages to see if server and client communicate with multiple IP. But from > > my test, it > > doesn't. Does any one know how to verify multichannel? > > Testing if it works vs checking speed improvement are 2 different > things unfortunately. > > Source code > =========== > > First you need the latest patchset which is available on my github remote: > > https://github.com/aaptel/linux.git > > You can use the "multichan-page" branch or you can extract the last > patches of it (with "multichannel:" in the title). These patches add the > multichannel support to the main read/write codepaths from memory pages. > > Client setup > ============ > > Then you need to mount with "vers=3.11,multichannel,max_channels=N" where N > will be the total number of channels the client will try to open. > > Do a network capture of the mount and run couple of commands in the > mount point (ls, cat, stat, ...). > > If the client also has multiple interface, make sure you capture all > interfaces traffic (otherwise you might ignore the other channels). > > Interface list check > ==================== > > Look at the trace in wireshark and use "smb2" filter. > > Look for "Ioctl Response FSCTL_QUERY_NETWORK_INTERFACE_INFO", if the > server has multiple interfaces you should see a list. > > You can also look at /proc/fs/cifs/DebugDate to see the list of > interfaces the client saw and has connected to. > > Multichannel check > ================== > > Still in wireshark: > - right-click on the first "Negotiate protocol" packet > - "Colorize conversation" > "F5 TCP" > > You can repeat this for each channel to see clearly different colors for > each channels. > > I have made some screenshots, maybe this can help: > > https://imgur.com/a/j3IBewF > > You can see the alternating colors in my traces that shows different > channels being used. > > Speedup check > ============= > > Now this is the hard part. > > If you test with VMs and the interface are "virtual", when you make 2 > channels you are actually just dividing by 2 the bandwidth of the > virtual bus, so you won't see much speedup. > > To see a speedup you need to limit the bandwidth of the server > interface, for example 1MB/s to each, so that together with 2 channel > you might see 2MB/s. > > You can look into iperf to check raw bandwidth and tc to limit it: > - https://iperf.fr/ > - https://netbeez.net/blog/how-to-use-the-linux-traffic-control/ > > Also similarly if the VM only has 1 CPU you might not see much > improvement as it will switch from one NIC to the other sequentially > instead of in parallel. > > This gets tricky very quick... > > > > Setup: > > server is samba server in linux upstream. > > client is linux upstream. > > smb.conf is: > > [global] > > interfaces = eth1, eth2, team0 > > server multi channel support = yes > > vfs objects = recycle aio_pthread > > aio read size = 1 > > aio write size = 1 > > [cifs] > > path=/mnt/cifs > > writeable=yes > > This looks ok but I've mostly used Windows Server 2019 in my test. > > Let me know how it goes. > > Cheers, > -- > Aurélien Aptel / SUSE Labs Samba Team > GPG: 1839 CB5F 9F5B FB9B AA97 8C99 03C8 A49B 521B D5D3 > SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg, DE > GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 247165 (AG München) > > ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: How to verify multichannel 2020-05-19 16:15 ` Xiaoli Feng @ 2020-05-19 16:37 ` Steve French 2020-05-19 16:54 ` Aurélien Aptel 1 sibling, 0 replies; 5+ messages in thread From: Steve French @ 2020-05-19 16:37 UTC (permalink / raw) To: Xiaoli Feng; +Cc: Aurélien Aptel, CIFS You can also "cat /proc/fs/cifs/Stats" and see the requests sent per channel which can be helpful. For example it enabled us to see that prior to Aurelien's recent patch series (which is targeted for 5.8 kernel) we only received a slight benefit in large i/o workloads from multichannel since reads and writes would be sent on the same channel (other request types would be spread out across the channels). With Aurelien's patch series, even with only one network adapter, I am seeing significant performance gains for large i/o workloads especially for encrypted or signed workloads where the extra parallelism on cifsd threads helps. With multiple network adapters on the client and the server, we can probably improve this scenario quite a bit though - choosing which adapters to use (and eventually adding integration with a userspace Witness Protocol helper to notify about network interface changes) On Tue, May 19, 2020 at 11:15 AM Xiaoli Feng <xifeng@redhat.com> wrote: > > Thanks Aurélien for your professional information. It's very detailed and > useful. > > Now I can see the multiple channel info in network package when mount with > option "max_channel=2". If doesn't specify it. Client will only open one > channel. And my smb.conf setup below can work for multichannel. Seems server > and client don't require multiple network interfaces. And also don't need > network team. > > But When test seedup, it isn't change. I use two vm in the same host. And > each have 4 cpu and 1G memory. Maybe it's the problem. > > Thanks so much. > > ----- Original Message ----- > > From: "Aurélien Aptel" <aaptel@suse.com> > > To: "Xiaoli Feng" <xifeng@redhat.com>, linux-cifs@vger.kernel.org > > Sent: Tuesday, May 19, 2020 1:17:11 AM > > Subject: Re: How to verify multichannel > > > > Hi Xiaoli, > > > > Xiaoli Feng <xifeng@redhat.com> writes: > > > Hello, > > > > > > When I test multichannel. I don't know how to verify if it works. I just > > > catch network > > > packages to see if server and client communicate with multiple IP. But from > > > my test, it > > > doesn't. Does any one know how to verify multichannel? > > > > Testing if it works vs checking speed improvement are 2 different > > things unfortunately. > > > > Source code > > =========== > > > > First you need the latest patchset which is available on my github remote: > > > > https://github.com/aaptel/linux.git > > > > You can use the "multichan-page" branch or you can extract the last > > patches of it (with "multichannel:" in the title). These patches add the > > multichannel support to the main read/write codepaths from memory pages. > > > > Client setup > > ============ > > > > Then you need to mount with "vers=3.11,multichannel,max_channels=N" where N > > will be the total number of channels the client will try to open. > > > > Do a network capture of the mount and run couple of commands in the > > mount point (ls, cat, stat, ...). > > > > If the client also has multiple interface, make sure you capture all > > interfaces traffic (otherwise you might ignore the other channels). > > > > Interface list check > > ==================== > > > > Look at the trace in wireshark and use "smb2" filter. > > > > Look for "Ioctl Response FSCTL_QUERY_NETWORK_INTERFACE_INFO", if the > > server has multiple interfaces you should see a list. > > > > You can also look at /proc/fs/cifs/DebugDate to see the list of > > interfaces the client saw and has connected to. > > > > Multichannel check > > ================== > > > > Still in wireshark: > > - right-click on the first "Negotiate protocol" packet > > - "Colorize conversation" > "F5 TCP" > > > > You can repeat this for each channel to see clearly different colors for > > each channels. > > > > I have made some screenshots, maybe this can help: > > > > https://imgur.com/a/j3IBewF > > > > You can see the alternating colors in my traces that shows different > > channels being used. > > > > Speedup check > > ============= > > > > Now this is the hard part. > > > > If you test with VMs and the interface are "virtual", when you make 2 > > channels you are actually just dividing by 2 the bandwidth of the > > virtual bus, so you won't see much speedup. > > > > To see a speedup you need to limit the bandwidth of the server > > interface, for example 1MB/s to each, so that together with 2 channel > > you might see 2MB/s. > > > > You can look into iperf to check raw bandwidth and tc to limit it: > > - https://iperf.fr/ > > - https://netbeez.net/blog/how-to-use-the-linux-traffic-control/ > > > > Also similarly if the VM only has 1 CPU you might not see much > > improvement as it will switch from one NIC to the other sequentially > > instead of in parallel. > > > > This gets tricky very quick... > > > > > > > Setup: > > > server is samba server in linux upstream. > > > client is linux upstream. > > > smb.conf is: > > > [global] > > > interfaces = eth1, eth2, team0 > > > server multi channel support = yes > > > vfs objects = recycle aio_pthread > > > aio read size = 1 > > > aio write size = 1 > > > [cifs] > > > path=/mnt/cifs > > > writeable=yes > > > > This looks ok but I've mostly used Windows Server 2019 in my test. > > > > Let me know how it goes. > > > > Cheers, > > -- > > Aurélien Aptel / SUSE Labs Samba Team > > GPG: 1839 CB5F 9F5B FB9B AA97 8C99 03C8 A49B 521B D5D3 > > SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg, DE > > GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 247165 (AG München) > > > > > -- Thanks, Steve ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: How to verify multichannel 2020-05-19 16:15 ` Xiaoli Feng 2020-05-19 16:37 ` Steve French @ 2020-05-19 16:54 ` Aurélien Aptel 1 sibling, 0 replies; 5+ messages in thread From: Aurélien Aptel @ 2020-05-19 16:54 UTC (permalink / raw) To: Xiaoli Feng; +Cc: linux-cifs Xiaoli Feng <xifeng@redhat.com> writes: > Now I can see the multiple channel info in network package when mount with > option "max_channel=2". If doesn't specify it. Client will only open one > channel. That's correct. > And my smb.conf setup below can work for multichannel. Seems server > and client don't require multiple network interfaces. And also don't need > network team. The client will try to reuse the same server interface if it has the RSS capability flag set. https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-smb2/fcd862d1-1b85-42df-92b1-e103199f531f RSS flag means that this server interface can handle things in parallel efficiently. > But When test seedup, it isn't change. I use two vm in the same host. And > each have 4 cpu and 1G memory. Maybe it's the problem. Maybe the problem is not the number of CPU but the virtual network bus bandwidth. Let me explain: forget SMB and measure network speed (with iperf) between 2 VMs. Let's say you get 1GB/s. Now you add a virtual interface to the server VM and measure again on the 2 interfaces *at the same time*. You will see at most 500MB/s on each. If you want to measure speedup, you need to limit the bandwidth of the server interfaces with tc so that when you max out the server interface bandwidth, you don't also max out the total virtual bus bandwidth. If you limit 1MB/s on each, 1+1 = 2MB/s which is still << 1GB/s but you will get speedup of 2. > Thanks so much. No problem :) Cheers, -- Aurélien Aptel / SUSE Labs Samba Team GPG: 1839 CB5F 9F5B FB9B AA97 8C99 03C8 A49B 521B D5D3 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg, DE GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 247165 (AG München) ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2020-05-19 16:54 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <1371710722.29134084.1589816497911.JavaMail.zimbra@redhat.com>
2020-05-18 16:01 ` How to verify multichannel Xiaoli Feng
2020-05-18 17:17 ` Aurélien Aptel
2020-05-19 16:15 ` Xiaoli Feng
2020-05-19 16:37 ` Steve French
2020-05-19 16:54 ` Aurélien Aptel
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.