diff for duplicates of <20200923181126.223766-1-nitesh@redhat.com> diff --git a/a/1.txt b/N1/1.txt index e621855..c6ba635 100644 --- a/a/1.txt +++ b/N1/1.txt @@ -4,15 +4,15 @@ based on housekeeping CPUs". Issue ===== -With the current implementation device drivers while creating their MSIX ? ? ? ? -vectors only take num_online_cpus() into consideration which works quite well ? -for a non-RT environment, but in an RT environment that has a large number of ? -isolated CPUs and very few housekeeping CPUs this could lead to a problem. ? ? -The problem will be triggered when something like tuned will try to move all ? ? -the IRQs from isolated CPUs to the limited number of housekeeping CPUs to ? ? ? -prevent interruptions for a latency-sensitive workload that will be running on ? -the isolated CPUs. This failure is caused because of the per CPU vector ? ? ? ? -limitation. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? +With the current implementation device drivers while creating their MSIX +vectors only take num_online_cpus() into consideration which works quite well +for a non-RT environment, but in an RT environment that has a large number of +isolated CPUs and very few housekeeping CPUs this could lead to a problem. +The problem will be triggered when something like tuned will try to move all +the IRQs from isolated CPUs to the limited number of housekeeping CPUs to +prevent interruptions for a latency-sensitive workload that will be running on +the isolated CPUs. This failure is caused because of the per CPU vector +limitation. Proposed Fix @@ -21,12 +21,12 @@ In this patch-set, the following changes are proposed: - A generic API hk_num_online_cpus() which is meant to return the online housekeeping CPUs that are meant to handle managed IRQ jobs. - i40e: Specifically for the i40e driver the num_online_cpus() used in -? i40e_init_msix() to calculate numbers msix vectors is replaced with the above -? defined API. This is done to restrict the number of msix vectors for i40e in + i40e_init_msix() to calculate numbers msix vectors is replaced with the above + defined API. This is done to restrict the number of msix vectors for i40e in RT environments. - pci_alloc_irq_vector(): With the help of hk_num_online_cpus() the max_vecs -? passed in pci_alloc_irq_vector() is restricted only to the online -? housekeeping CPUs only in an RT environment. However, if the min_vecs exceeds + passed in pci_alloc_irq_vector() is restricted only to the online + housekeeping CPUs only in an RT environment. However, if the min_vecs exceeds the online housekeeping CPUs, max_vecs is limited based on the min_vecs instead. @@ -45,43 +45,43 @@ Testing ======= Functionality: - To test that the issue is resolved with i40e change I added a tracepoint -? in i40e_init_msix() to find the number of CPUs derived for vector creation -? with and without tuned's realtime-virtual-host profile. As per expectation -? with the profile applied I was only getting the number of housekeeping CPUs -? and all available CPUs without it. + in i40e_init_msix() to find the number of CPUs derived for vector creation + with and without tuned's realtime-virtual-host profile. As per expectation + with the profile applied I was only getting the number of housekeeping CPUs + and all available CPUs without it. Similarly did a few more tests with different modes eg with only nohz_full, isolcpus etc. Performance: - To analyze the performance impact I have targetted the change introduced in -? pci_alloc_irq_vectors() and compared the results against a vanilla kernel -? (5.9.0-rc3) results. - -? Setup Information: -? + I had a couple of 24-core machines connected back to back via a couple of -? ? mlx5 NICs and I analyzed the average bitrate for server-client TCP and UDP -? ? transmission via iperf. -? + To minimize the Bitrate variation of iperf TCP and UDP stream test I have -? ? applied the tuned's network-throughput profile and disabled HT. -?Test Information: -? + For the environment that had no isolated CPUs: -? ? I have tested with single stream and 24 streams (same as that of online -? ? CPUs). -? + For the environment that had 20 isolated CPUs: -? ? I have tested with single stream, 4 streams (same as that the number of -? ? housekeeping) and 24 streams (same as that of online CPUs). - -?Results: -? # UDP Stream Test: -? + There was no degradation observed in UDP stream tests in both + pci_alloc_irq_vectors() and compared the results against a vanilla kernel + (5.9.0-rc3) results. + + Setup Information: + + I had a couple of 24-core machines connected back to back via a couple of + mlx5 NICs and I analyzed the average bitrate for server-client TCP and UDP + transmission via iperf. + + To minimize the Bitrate variation of iperf TCP and UDP stream test I have + applied the tuned's network-throughput profile and disabled HT. + Test Information: + + For the environment that had no isolated CPUs: + I have tested with single stream and 24 streams (same as that of online + CPUs). + + For the environment that had 20 isolated CPUs: + I have tested with single stream, 4 streams (same as that the number of + housekeeping) and 24 streams (same as that of online CPUs). + + Results: + # UDP Stream Test: + + There was no degradation observed in UDP stream tests in both environments. (With isolated CPUs and without isolated CPUs after the introduction of the patches). -? # TCP Stream Test - No isolated CPUs: -? + No noticeable degradation was observed. -? # TCP Stream Test - With isolated CPUs: -? + Multiple Stream (4) ?- Average degradation of around 5-6% -? + Multiple Stream (24) - Average degradation of around 2-3% -? + Single Stream ? ? ? ?- Even on a vanilla kernel the Bitrate observed for + # TCP Stream Test - No isolated CPUs: + + No noticeable degradation was observed. + # TCP Stream Test - With isolated CPUs: + + Multiple Stream (4) - Average degradation of around 5-6% + + Multiple Stream (24) - Average degradation of around 2-3% + + Single Stream - Even on a vanilla kernel the Bitrate observed for a TCP single stream test seem to vary significantly across different runs (eg. the % variation between the best and the worst case on @@ -95,22 +95,22 @@ be happy to discuss/perform them. Changes from v1[2]: ================== -Patch1: ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? +Patch1: - Replaced num_houskeeeping_cpus() with hk_num_online_cpus() and started using the cpumask corresponding to HK_FLAG_MANAGED_IRQ to derive the number of - online housekeeping CPUs. This is based on Frederic Weisbecker's suggestion. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? -- Since the hk_num_online_cpus() is self-explanatory, got rid of ? ? ? ? ? ? -? the comment that was added previously. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? -Patch2: ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? + online housekeeping CPUs. This is based on Frederic Weisbecker's suggestion. +- Since the hk_num_online_cpus() is self-explanatory, got rid of + the comment that was added previously. +Patch2: - Added a new patch that is meant to enable managed IRQ isolation for nohz_full - CPUs. This is based on Frederic Weisbecker's suggestion.? ? ? ? ? ? ? -Patch4 (PCI): ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? + CPUs. This is based on Frederic Weisbecker's suggestion. +Patch4 (PCI): - For cases where the min_vecs exceeds the online housekeeping CPUs, instead of skipping modification to max_vecs, started restricting it based on the - min_vecs. This is based on a suggestion from Marcelo Tosatti.? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? + min_vecs. This is based on a suggestion from Marcelo Tosatti. -[1] https://lore.kernel.org/lkml/20200922095440.GA5217 at lenoir/ -[2] https://lore.kernel.org/lkml/20200909150818.313699-1-nitesh at redhat.com/ +[1] https://lore.kernel.org/lkml/20200922095440.GA5217@lenoir/ +[2] https://lore.kernel.org/lkml/20200909150818.313699-1-nitesh@redhat.com/ Nitesh Narayan Lal (4): sched/isolation: API to get housekeeping online CPUs diff --git a/a/content_digest b/N1/content_digest index 9daa35c..4732186 100644 --- a/a/content_digest +++ b/N1/content_digest @@ -1,7 +1,32 @@ "From\0Nitesh Narayan Lal <nitesh@redhat.com>\0" - "Subject\0[Intel-wired-lan] [PATCH v2 0/4] isolation: limit msix vectors based on housekeeping CPUs\0" + "Subject\0[PATCH v2 0/4] isolation: limit msix vectors based on housekeeping CPUs\0" "Date\0Wed, 23 Sep 2020 14:11:22 -0400\0" - "To\0intel-wired-lan@osuosl.org\0" + "To\0linux-kernel@vger.kernel.org" + netdev@vger.kernel.org + linux-pci@vger.kernel.org + intel-wired-lan@lists.osuosl.org + frederic@kernel.org + mtosatti@redhat.com + sassmann@redhat.com + jesse.brandeburg@intel.com + lihong.yang@intel.com + helgaas@kernel.org + nitesh@redhat.com + jeffrey.t.kirsher@intel.com + jacob.e.keller@intel.com + jlelli@redhat.com + hch@infradead.org + bhelgaas@google.com + mike.marciniszyn@intel.com + dennis.dalessandro@intel.com + thomas.lendacky@amd.com + jerinj@marvell.com + mathias.nyman@intel.com + jiri@nvidia.com + mingo@redhat.com + peterz@infradead.org + juri.lelli@redhat.com + " vincent.guittot@linaro.org\0" "\00:1\0" "b\0" "This is a follow-up posting for \"[RFC v1 0/3] isolation: limit msix vectors\n" @@ -10,15 +35,15 @@ "\n" "Issue\n" "=====\n" - "With the current implementation device drivers while creating their MSIX ? ? ? ?\n" - "vectors only take num_online_cpus() into consideration which works quite well ?\n" - "for a non-RT environment, but in an RT environment that has a large number of ? \n" - "isolated CPUs and very few housekeeping CPUs this could lead to a problem. ? ?\n" - "The problem will be triggered when something like tuned will try to move all ? ?\n" - "the IRQs from isolated CPUs to the limited number of housekeeping CPUs to ? ? ? \n" - "prevent interruptions for a latency-sensitive workload that will be running on ? \n" - "the isolated CPUs. This failure is caused because of the per CPU vector ? ? ? ? \n" - "limitation. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? \n" + "With the current implementation device drivers while creating their MSIX \302\240 \302\240 \302\240 \302\240\n" + "vectors only take num_online_cpus() into consideration which works quite well \302\240\n" + "for a non-RT environment, but in an RT environment that has a large number of \302\240 \n" + "isolated CPUs and very few housekeeping CPUs this could lead to a problem. \302\240 \302\240\n" + "The problem will be triggered when something like tuned will try to move all \302\240 \302\240\n" + "the IRQs from isolated CPUs to the limited number of housekeeping CPUs to \302\240 \302\240 \302\240 \n" + "prevent interruptions for a latency-sensitive workload that will be running on \302\240 \n" + "the isolated CPUs. This failure is caused because of the per CPU vector \302\240 \302\240 \302\240 \302\240 \n" + "limitation. \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \n" "\n" "\n" "Proposed Fix\n" @@ -27,12 +52,12 @@ "- A generic API hk_num_online_cpus() which is meant to return the online\n" " housekeeping CPUs that are meant to handle managed IRQ jobs.\n" "- i40e: Specifically for the i40e driver the num_online_cpus() used in \n" - "? i40e_init_msix() to calculate numbers msix vectors is replaced with the above\n" - "? defined API. This is done to restrict the number of msix vectors for i40e in\n" + "\302\240 i40e_init_msix() to calculate numbers msix vectors is replaced with the above\n" + "\302\240 defined API. This is done to restrict the number of msix vectors for i40e in\n" " RT environments.\n" "- pci_alloc_irq_vector(): With the help of hk_num_online_cpus() the max_vecs\n" - "? passed in pci_alloc_irq_vector() is restricted only to the online\n" - "? housekeeping CPUs only in an RT environment. However, if the min_vecs exceeds\n" + "\302\240 passed in pci_alloc_irq_vector() is restricted only to the online\n" + "\302\240 housekeeping CPUs only in an RT environment. However, if the min_vecs exceeds\n" " the online housekeeping CPUs, max_vecs is limited based on the min_vecs\n" " instead.\n" "\n" @@ -51,43 +76,43 @@ "=======\n" "Functionality:\n" "- To test that the issue is resolved with i40e change I added a tracepoint\n" - "? in i40e_init_msix() to find the number of CPUs derived for vector creation\n" - "? with and without tuned's realtime-virtual-host profile. As per expectation\n" - "? with the profile applied I was only getting the number of housekeeping CPUs\n" - "? and all available CPUs without it.\n" + "\302\240 in i40e_init_msix() to find the number of CPUs derived for vector creation\n" + "\302\240 with and without tuned's realtime-virtual-host profile. As per expectation\n" + "\302\240 with the profile applied I was only getting the number of housekeeping CPUs\n" + "\302\240 and all available CPUs without it.\n" " Similarly did a few more tests with different modes eg with only\n" " nohz_full, isolcpus etc.\n" "\n" "Performance:\n" "- To analyze the performance impact I have targetted the change introduced in \n" - "? pci_alloc_irq_vectors() and compared the results against a vanilla kernel\n" - "? (5.9.0-rc3) results.\n" - "\n" - "? Setup Information:\n" - "? + I had a couple of 24-core machines connected back to back via a couple of\n" - "? ? mlx5 NICs and I analyzed the average bitrate for server-client TCP and UDP\n" - "? ? transmission via iperf. \n" - "? + To minimize the Bitrate variation of iperf TCP and UDP stream test I have\n" - "? ? applied the tuned's network-throughput profile and disabled HT.\n" - "?Test Information:\n" - "? + For the environment that had no isolated CPUs:\n" - "? ? I have tested with single stream and 24 streams (same as that of online\n" - "? ? CPUs).\n" - "? + For the environment that had 20 isolated CPUs:\n" - "? ? I have tested with single stream, 4 streams (same as that the number of\n" - "? ? housekeeping) and 24 streams (same as that of online CPUs).\n" - "\n" - "?Results:\n" - "? # UDP Stream Test:\n" - "? + There was no degradation observed in UDP stream tests in both\n" + "\302\240 pci_alloc_irq_vectors() and compared the results against a vanilla kernel\n" + "\302\240 (5.9.0-rc3) results.\n" + "\n" + "\302\240 Setup Information:\n" + "\302\240 + I had a couple of 24-core machines connected back to back via a couple of\n" + "\302\240 \302\240 mlx5 NICs and I analyzed the average bitrate for server-client TCP and UDP\n" + "\302\240 \302\240 transmission via iperf. \n" + "\302\240 + To minimize the Bitrate variation of iperf TCP and UDP stream test I have\n" + "\302\240 \302\240 applied the tuned's network-throughput profile and disabled HT.\n" + "\302\240Test Information:\n" + "\302\240 + For the environment that had no isolated CPUs:\n" + "\302\240 \302\240 I have tested with single stream and 24 streams (same as that of online\n" + "\302\240 \302\240 CPUs).\n" + "\302\240 + For the environment that had 20 isolated CPUs:\n" + "\302\240 \302\240 I have tested with single stream, 4 streams (same as that the number of\n" + "\302\240 \302\240 housekeeping) and 24 streams (same as that of online CPUs).\n" + "\n" + "\302\240Results:\n" + "\302\240 # UDP Stream Test:\n" + "\302\240 + There was no degradation observed in UDP stream tests in both\n" " environments. (With isolated CPUs and without isolated CPUs after the\n" " introduction of the patches).\n" - "? # TCP Stream Test - No isolated CPUs:\n" - "? + No noticeable degradation was observed.\n" - "? # TCP Stream Test - With isolated CPUs:\n" - "? + Multiple Stream (4) ?- Average degradation of around 5-6%\n" - "? + Multiple Stream (24) - Average degradation of around 2-3%\n" - "? + Single Stream ? ? ? ?- Even on a vanilla kernel the Bitrate observed for\n" + "\302\240 # TCP Stream Test - No isolated CPUs:\n" + "\302\240 + No noticeable degradation was observed.\n" + "\302\240 # TCP Stream Test - With isolated CPUs:\n" + "\302\240 + Multiple Stream (4) \302\240- Average degradation of around 5-6%\n" + "\302\240 + Multiple Stream (24) - Average degradation of around 2-3%\n" + "\302\240 + Single Stream \302\240 \302\240 \302\240 \302\240- Even on a vanilla kernel the Bitrate observed for\n" " a TCP single stream test seem to vary\n" " significantly across different runs (eg. the %\n" " variation between the best and the worst case on\n" @@ -101,22 +126,22 @@ "\n" "Changes from v1[2]:\n" "==================\n" - "Patch1: ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? \n" + "Patch1: \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \n" "- Replaced num_houskeeeping_cpus() with hk_num_online_cpus() and started using\n" " the cpumask corresponding to HK_FLAG_MANAGED_IRQ to derive the number of\n" - " online housekeeping CPUs. This is based on Frederic Weisbecker's suggestion. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? \n" - "- Since the hk_num_online_cpus() is self-explanatory, got rid of ? ? ? ? ? ? \n" - "? the comment that was added previously. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? \n" - "Patch2: ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? \n" + " online housekeeping CPUs. This is based on Frederic Weisbecker's suggestion. \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \n" + "- Since the hk_num_online_cpus() is self-explanatory, got rid of \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \n" + "\302\240 the comment that was added previously. \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \n" + "Patch2: \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \n" "- Added a new patch that is meant to enable managed IRQ isolation for nohz_full\n" - " CPUs. This is based on Frederic Weisbecker's suggestion.? ? ? ? ? ? ? \n" - "Patch4 (PCI): ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? \n" + " CPUs. This is based on Frederic Weisbecker's suggestion.\302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \n" + "Patch4 (PCI): \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \n" "- For cases where the min_vecs exceeds the online housekeeping CPUs, instead\n" " of skipping modification to max_vecs, started restricting it based on the\n" - " min_vecs. This is based on a suggestion from Marcelo Tosatti.? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??\n" + " min_vecs. This is based on a suggestion from Marcelo Tosatti.\302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240 \302\240\302\240\n" "\n" - "[1] https://lore.kernel.org/lkml/20200922095440.GA5217 at lenoir/\n" - "[2] https://lore.kernel.org/lkml/20200909150818.313699-1-nitesh at redhat.com/\n" + "[1] https://lore.kernel.org/lkml/20200922095440.GA5217@lenoir/\n" + "[2] https://lore.kernel.org/lkml/20200909150818.313699-1-nitesh@redhat.com/\n" "\n" "Nitesh Narayan Lal (4):\n" " sched/isolation: API to get housekeeping online CPUs\n" @@ -132,4 +157,4 @@ "\n" -- -b4643d6d30c58e58e50d0775727efd69b4c5ba553979580e1f7c70dd12692ffa +a42b2e258e8224a0ecbef03e3a18eeaea437149663ed451e7a6b4276c141c61f
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.