* Misleading throughput number using ntb_perf with memcpy_toio()
@ 2017-11-02 18:33 Karl Kao
2017-11-02 18:39 ` Dave Jiang
0 siblings, 1 reply; 6+ messages in thread
From: Karl Kao @ 2017-11-02 18:33 UTC (permalink / raw)
To: linux-ntb
[-- Attachment #1.1: Type: text/plain, Size: 1414 bytes --]
Hi Folks,
We have one system with PLX NTB enabled with Gen3 speed on x8 lanes width.
Technically, the NTB bandwidth is 64 Gbit/s (8192 MBytes/s) in outbound
direction to its peer on this system.
The ntb_perf comes with the throughput number of 9147 MBytes/s which is
misleading in comparison to hardware bandwidth of 8192 MBytes/s.
We have a few questions with regard to the misleading throughput number.
The overall is when memcpy() returns, data may have not yet been put in
ingress buffer inside PLX NTB which is constrained by flow control credits,
limitation of 8192 Mbytes/s.
- The ntb_perf moves memory block to IO bus, using memcpy_toio(), a
macro of memcpy(). Once cpu core is done with mov instructions, the
memcpy() returns. Is this statement accurate?
- At the point when memcpy() is done with mov instructions, would the
data had been moved to internal buffer inside processor, instead of going
through IO bus into PLX chip?
- Where is the buffer, IIO buffer? What's buffer size that can
accommodate gigabytes data?
- Once we are clearer about the memcpy_tpio, would there be any API that
can accurately measure the IO write throughput?
[11:02][root@BRYCE2-DEV][/sys/kernel/debug/ntb_perf/0000:06:00.0]\> cat run
0: copied 4294967296 bytes in 469542 usecs, 9147 MBytes/s
[11:03][root@BRYCE2-DEV][/sys/kernel/debug/ntb_perf/0000:06:00.0]\>
Thanks,
Karl
[-- Attachment #1.2: Type: text/html, Size: 1609 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Misleading throughput number using ntb_perf with memcpy_toio()
2017-11-02 18:33 Misleading throughput number using ntb_perf with memcpy_toio() Karl Kao
@ 2017-11-02 18:39 ` Dave Jiang
2017-11-02 18:46 ` Allen Hubbe
2017-11-02 18:48 ` Karl Kao
0 siblings, 2 replies; 6+ messages in thread
From: Dave Jiang @ 2017-11-02 18:39 UTC (permalink / raw)
To: Karl Kao, linux-ntb
On 11/02/2017 11:33 AM, Karl Kao wrote:
> Hi Folks,
>
> We have one system with PLX NTB enabled with Gen3 speed on x8 lanes
> width. Technically, the NTB bandwidth is 64 Gbit/s (8192 MBytes/s) in
> outbound direction to its peer on this system.
> The ntb_perf comes with the throughput number of 9147 MBytes/s which is
> misleading in comparison to hardware bandwidth of 8192 MBytes/s.
>
> We have a few questions with regard to the misleading throughput number.
> The overall is when memcpy() returns, data may have not yet been put in
> ingress buffer inside PLX NTB which is constrained by flow control
> credits, limitation of 8192 Mbytes/s.
>
> * The ntb_perf moves memory block to IO bus, using memcpy_toio(), a
> macro of memcpy(). Once cpu core is done with mov instructions, the
> memcpy() returns. Is this statement accurate?
> * At the point when memcpy() is done with mov instructions, would the
> data had been moved to internal buffer inside processor, instead of
> going through IO bus into PLX chip?
> * Where is the buffer, IIO buffer? What's buffer size that can
> accommodate gigabytes data?
> * Once we are clearer about the memcpy_tpio, would there be any API
> that can accurately measure the IO write throughput?
>
>
> [11:02][root@BRYCE2-DEV][/sys/kernel/debug/ntb_perf/0000:06:00.0]\> cat run
> 0: copied 4294967296 bytes in 469542 usecs, 9147 MBytes/s
> [11:03][root@BRYCE2-DEV][/sys/kernel/debug/ntb_perf/0000:06:00.0]\>
Hi Karl. Yes the issue with ntb_perf is that it blindly copies into the
memory window with no acknowledgement from the peer node. So the
performance can be somewhat misleading because you don't know if it has
made it completely to the other side. The only way to know for certain
everything has made it over is to do a small read at the end of the
write I think?
^ permalink raw reply [flat|nested] 6+ messages in thread
* RE: Misleading throughput number using ntb_perf with memcpy_toio()
2017-11-02 18:39 ` Dave Jiang
@ 2017-11-02 18:46 ` Allen Hubbe
2017-11-02 18:48 ` Karl Kao
1 sibling, 0 replies; 6+ messages in thread
From: Allen Hubbe @ 2017-11-02 18:46 UTC (permalink / raw)
To: 'Dave Jiang', 'Karl Kao', 'linux-ntb'
From: Dave Jiang
> On 11/02/2017 11:33 AM, Karl Kao wrote:
> > Hi Folks,
> >
> > We have one system with PLX NTB enabled with Gen3 speed on x8 lanes
> > width. Technically, the NTB bandwidth is 64 Gbit/s (8192 MBytes/s) in
> > outbound direction to its peer on this system.
> > The ntb_perf comes with the throughput number of 9147 MBytes/s which is
> > misleading in comparison to hardware bandwidth of 8192 MBytes/s.
> >
> > We have a few questions with regard to the misleading throughput number.
> > The overall is when memcpy() returns, data may have not yet been put in
> > ingress buffer inside PLX NTB which is constrained by flow control
> > credits, limitation of 8192 Mbytes/s.
> >
> > * The ntb_perf moves memory block to IO bus, using memcpy_toio(), a
> > macro of memcpy(). Once cpu core is done with mov instructions, the
> > memcpy() returns. Is this statement accurate?
> > * At the point when memcpy() is done with mov instructions, would the
> > data had been moved to internal buffer inside processor, instead of
> > going through IO bus into PLX chip?
> > * Where is the buffer, IIO buffer? What's buffer size that can
> > accommodate gigabytes data?
> > * Once we are clearer about the memcpy_tpio, would there be any API
> > that can accurately measure the IO write throughput?
> >
> >
> > [11:02][root@BRYCE2-DEV][/sys/kernel/debug/ntb_perf/0000:06:00.0]\> cat run
> > 0: copied 4294967296 bytes in 469542 usecs, 9147 MBytes/s
> > [11:03][root@BRYCE2-DEV][/sys/kernel/debug/ntb_perf/0000:06:00.0]\>
>
>
> Hi Karl. Yes the issue with ntb_perf is that it blindly copies into the
> memory window with no acknowledgement from the peer node. So the
> performance can be somewhat misleading because you don't know if it has
> made it completely to the other side. The only way to know for certain
> everything has made it over is to do a small read at the end of the
> write I think?
Or, a warm-up phase to fill up write queues, before the measured phase of the throughput test?
We don't need to know that it made it to the other side. That would be needed for latency, but for throughput we only care about the rate it was transmitted even if it takes a long time to arrive.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Misleading throughput number using ntb_perf with memcpy_toio()
2017-11-02 18:39 ` Dave Jiang
2017-11-02 18:46 ` Allen Hubbe
@ 2017-11-02 18:48 ` Karl Kao
2017-11-02 19:34 ` Dave Jiang
1 sibling, 1 reply; 6+ messages in thread
From: Karl Kao @ 2017-11-02 18:48 UTC (permalink / raw)
To: Dave Jiang, linux-ntb
[-- Attachment #1: Type: text/plain, Size: 2313 bytes --]
Hi Dave, the no acknowledge from peer is justified by doing posted pcie transactions. I don’t understand why memcpy can return and claim it’s done the job with assumption that internal buffer is sufficient to take gigabytes data.
From: Dave Jiang <dave.jiang@intel.com>
Date: Thursday, November 2, 2017 at 11:39
To: Karl Kao <ykao@tintri.com>, linux-ntb <linux-ntb@googlegroups.com>
Subject: Re: Misleading throughput number using ntb_perf with memcpy_toio()
On 11/02/2017 11:33 AM, Karl Kao wrote:
Hi Folks,
We have one system with PLX NTB enabled with Gen3 speed on x8 lanes
width. Technically, the NTB bandwidth is 64 Gbit/s (8192 MBytes/s) in
outbound direction to its peer on this system.
The ntb_perf comes with the throughput number of 9147 MBytes/s which is
misleading in comparison to hardware bandwidth of 8192 MBytes/s.
We have a few questions with regard to the misleading throughput number.
The overall is when memcpy() returns, data may have not yet been put in
ingress buffer inside PLX NTB which is constrained by flow control
credits, limitation of 8192 Mbytes/s.
* The ntb_perf moves memory block to IO bus, using memcpy_toio(), a
macro of memcpy(). Once cpu core is done with mov instructions, the
memcpy() returns. Is this statement accurate?
* At the point when memcpy() is done with mov instructions, would the
data had been moved to internal buffer inside processor, instead of
going through IO bus into PLX chip?
* Where is the buffer, IIO buffer? What's buffer size that can
accommodate gigabytes data?
* Once we are clearer about the memcpy_tpio, would there be any API
that can accurately measure the IO write throughput?
[11:02][root@BRYCE2-DEV][/sys/kernel/debug/ntb_perf/0000:06:00.0]\> cat run
0: copied 4294967296 bytes in 469542 usecs, 9147 MBytes/s
[11:03][root@BRYCE2-DEV][/sys/kernel/debug/ntb_perf/0000:06:00.0]\>
Hi Karl. Yes the issue with ntb_perf is that it blindly copies into the
memory window with no acknowledgement from the peer node. So the
performance can be somewhat misleading because you don't know if it has
made it completely to the other side. The only way to know for certain
everything has made it over is to do a small read at the end of the
write I think?
[-- Attachment #2: Type: text/html, Size: 6777 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: Misleading throughput number using ntb_perf with memcpy_toio()
2017-11-02 18:48 ` Karl Kao
@ 2017-11-02 19:34 ` Dave Jiang
2017-11-02 20:27 ` Karl Kao
0 siblings, 1 reply; 6+ messages in thread
From: Dave Jiang @ 2017-11-02 19:34 UTC (permalink / raw)
To: Karl Kao, linux-ntb
On 11/02/2017 11:48 AM, Karl Kao wrote:
> Hi Dave, the no acknowledge from peer is justified by doing posted pcie
> transactions. I don’t understand why memcpy can return and claim it’s
> done the job with assumption that internal buffer is sufficient to take
> gigabytes data.
memcpy is just CPU instructions right? It returns as soon as CPU is done
copy. I don't think it has any ideas with regards to I/O buffers or
anything like that.
>
>
>
>
>
>
>
> *From: *Dave Jiang <dave.jiang@intel.com>
> *Date: *Thursday, November 2, 2017 at 11:39
> *To: *Karl Kao <ykao@tintri.com>, linux-ntb <linux-ntb@googlegroups.com>
> *Subject: *Re: Misleading throughput number using ntb_perf with
> memcpy_toio()
>
>
>
>
>
>
>
> On 11/02/2017 11:33 AM, Karl Kao wrote:
>
> Hi Folks,
>
> We have one system with PLX NTB enabled with Gen3 speed on x8 lanes
>
> width. Technically, the NTB bandwidth is 64 Gbit/s (8192 MBytes/s) in
>
> outbound direction to its peer on this system.
>
> The ntb_perf comes with the throughput number of 9147 MBytes/s which is
>
> misleading in comparison to hardware bandwidth of 8192 MBytes/s.
>
> We have a few questions with regard to the misleading throughput number.
>
> The overall is when memcpy() returns, data may have not yet been put in
>
> ingress buffer inside PLX NTB which is constrained by flow control
>
> credits, limitation of 8192 Mbytes/s.
>
> * The ntb_perf moves memory block to IO bus, using memcpy_toio(), a
>
> macro of memcpy(). Once cpu core is done with mov instructions, the
>
> memcpy() returns. Is this statement accurate?
>
> * At the point when memcpy() is done with mov instructions, would the
>
> data had been moved to internal buffer inside processor, instead of
>
> going through IO bus into PLX chip?
>
> * Where is the buffer, IIO buffer? What's buffer size that can
>
> accommodate gigabytes data?
>
> * Once we are clearer about the memcpy_tpio, would there be any API
>
> that can accurately measure the IO write throughput?
>
> [11:02][root@BRYCE2-DEV][/sys/kernel/debug/ntb_perf/0000:06:00.0]\>
> cat run
>
> 0: copied 4294967296 bytes in 469542 usecs, 9147 MBytes/s
>
> [11:03][root@BRYCE2-DEV][/sys/kernel/debug/ntb_perf/0000:06:00.0]\>
>
>
>
>
>
> Hi Karl. Yes the issue with ntb_perf is that it blindly copies into the
>
> memory window with no acknowledgement from the peer node. So the
>
> performance can be somewhat misleading because you don't know if it has
>
> made it completely to the other side. The only way to know for certain
>
> everything has made it over is to do a small read at the end of the
>
> write I think?
>
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "linux-ntb" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to linux-ntb+unsubscribe@googlegroups.com
> <mailto:linux-ntb+unsubscribe@googlegroups.com>.
> To post to this group, send email to linux-ntb@googlegroups.com
> <mailto:linux-ntb@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/linux-ntb/56307FCF-28EE-4857-BA37-86D84CD617F7%40tintri.com
> <https://groups.google.com/d/msgid/linux-ntb/56307FCF-28EE-4857-BA37-86D84CD617F7%40tintri.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Misleading throughput number using ntb_perf with memcpy_toio()
2017-11-02 19:34 ` Dave Jiang
@ 2017-11-02 20:27 ` Karl Kao
0 siblings, 0 replies; 6+ messages in thread
From: Karl Kao @ 2017-11-02 20:27 UTC (permalink / raw)
To: Dave Jiang, linux-ntb
[-- Attachment #1: Type: text/plain, Size: 3997 bytes --]
Yes, the memcpy is a set of CPU instructions. However, CPU is moving data from regular memory to destination address targeted at I/O. Wouldn’t the CPU move data to certain I/O buffer before the I/O bus is free to take?
From: Dave Jiang <dave.jiang@intel.com>
Date: Thursday, November 2, 2017 at 12:34
To: Karl Kao <ykao@tintri.com>, linux-ntb <linux-ntb@googlegroups.com>
Subject: Re: Misleading throughput number using ntb_perf with memcpy_toio()
On 11/02/2017 11:48 AM, Karl Kao wrote:
Hi Dave, the no acknowledge from peer is justified by doing posted pcie
transactions. I don’t understand why memcpy can return and claim it’s
done the job with assumption that internal buffer is sufficient to take
gigabytes data.
memcpy is just CPU instructions right? It returns as soon as CPU is done
copy. I don't think it has any ideas with regards to I/O buffers or
anything like that.
*From: *Dave Jiang <dave.jiang@intel.com<mailto:dave.jiang@intel.com>>
*Date: *Thursday, November 2, 2017 at 11:39
*To: *Karl Kao <ykao@tintri.com<mailto:ykao@tintri.com>>, linux-ntb <linux-ntb@googlegroups.com<mailto:linux-ntb@googlegroups.com>>
*Subject: *Re: Misleading throughput number using ntb_perf with
memcpy_toio()
On 11/02/2017 11:33 AM, Karl Kao wrote:
Hi Folks,
We have one system with PLX NTB enabled with Gen3 speed on x8 lanes
width. Technically, the NTB bandwidth is 64 Gbit/s (8192 MBytes/s) in
outbound direction to its peer on this system.
The ntb_perf comes with the throughput number of 9147 MBytes/s which is
misleading in comparison to hardware bandwidth of 8192 MBytes/s.
We have a few questions with regard to the misleading throughput number.
The overall is when memcpy() returns, data may have not yet been put in
ingress buffer inside PLX NTB which is constrained by flow control
credits, limitation of 8192 Mbytes/s.
* The ntb_perf moves memory block to IO bus, using memcpy_toio(), a
macro of memcpy(). Once cpu core is done with mov instructions, the
memcpy() returns. Is this statement accurate?
* At the point when memcpy() is done with mov instructions, would the
data had been moved to internal buffer inside processor, instead of
going through IO bus into PLX chip?
* Where is the buffer, IIO buffer? What's buffer size that can
accommodate gigabytes data?
* Once we are clearer about the memcpy_tpio, would there be any API
that can accurately measure the IO write throughput?
[11:02][root@BRYCE2-DEV][/sys/kernel/debug/ntb_perf/0000:06:00.0]\>
cat run
0: copied 4294967296 bytes in 469542 usecs, 9147 MBytes/s
[11:03][root@BRYCE2-DEV][/sys/kernel/debug/ntb_perf/0000:06:00.0]\>
Hi Karl. Yes the issue with ntb_perf is that it blindly copies into the
memory window with no acknowledgement from the peer node. So the
performance can be somewhat misleading because you don't know if it has
made it completely to the other side. The only way to know for certain
everything has made it over is to do a small read at the end of the
write I think?
--
You received this message because you are subscribed to the Google
Groups "linux-ntb" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to linux-ntb+unsubscribe@googlegroups.com<mailto:linux-ntb+unsubscribe@googlegroups.com>
<mailto:linux-ntb+unsubscribe@googlegroups.com>.
To post to this group, send email to linux-ntb@googlegroups.com<mailto:linux-ntb@googlegroups.com>
<mailto:linux-ntb@googlegroups.com>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/linux-ntb/56307FCF-28EE-4857-BA37-86D84CD617F7%40tintri.com
<https://groups.google.com/d/msgid/linux-ntb/56307FCF-28EE-4857-BA37-86D84CD617F7%40tintri.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.
[-- Attachment #2: Type: text/html, Size: 11305 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2017-11-02 20:28 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2017-11-02 18:33 Misleading throughput number using ntb_perf with memcpy_toio() Karl Kao
2017-11-02 18:39 ` Dave Jiang
2017-11-02 18:46 ` Allen Hubbe
2017-11-02 18:48 ` Karl Kao
2017-11-02 19:34 ` Dave Jiang
2017-11-02 20:27 ` Karl Kao
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.