Re: [V9fs-developer] :[RFC] [PATCH 6/7] [net/9p] Read and Write side zerocopy changes for 9P2000.L protocol.

["Venkateswararao Jujjuri (JV)" <jvrao@linux.vnet.ibm.com>]

On 2/9/2011 1:09 PM, Venkateswararao Jujjuri (JV) wrote:
> WRITE

dd if=/dev/zero of=/pmnt/file1 bs=4096 count=1MB (variable bs = IO SIZE)

> 
> IO SIZE      TOTAL SIZE       No ZC                ZC
> 1                   1MB                22.4 kb/s         19.8 kb/s
> 32                 32MB              711 kb/s          633 kb/s
> 64                 64MB              1.4 mb/s          1.3  mb/s
> 128               128MB             2.8 mb/s          2.6 mb/s
> 256               256MB             5.6 mb/s          5.1 mb/s
> 512               512MB            10.4 mb/s        10.2 mb/s
> 1024              1GB               19.7 mb/s         20.4 mb/s
> 2048              2GB               40.1 mb/s          43.7 mb/s
> 4096              4GB               71.4 mb/s          73.1 mb/s
> 
> 
> 
> READ
dd of=/dev/null if=/pmnt/file1 bs=4096 count=1MB(variable bs = IO SIZE)

> IO SIZE      TOTAL SIZE       No ZC                ZC
> 1                   1MB                26.6 kb/s         23.1 kb/s
> 32                 32MB              783 kb/s           734 kb/s
> 64                 64MB              1.7 mb/s          1.5 mb/s
> 128               128MB             3.4 mb/s          3.0 mb/s
> 256               256MB             4.2 mb/s           5.9 mb/s
> 512               512MB            6.9 mb/s            11.6 mb/s
> 1024              1GB               23.3 mb/s          23.4 mb/s
> 2048              2GB               42.5 mb/s          45.4 mb/s
> 4096              4GB               67.4 mb/s          73.9 mb/s
> 
> As you can see, the difference is marginal..but zc improves as the IO size
> increases.
> In the past we have seen tremendous improvements with different msizes.
> It  is mostly because of shipping bigger chunks of data which is possible with
> zero copy.
> Also it could be my setup/box even on the host I am getting same/similar numbers.
> 
> - JV
> 
> 
> 
> 
> On 2/8/2011 1:16 PM, Eric Van Hensbergen wrote:
>> oh and, for reference, while a different environment, my request is
>> based on a little bit more than idle fancy.  Check out the graph on
>> page 6 in: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.108.8182&rep=rep1&type=pdf
>>
>>         -eric
>>
>>
>> On Tue, Feb 8, 2011 at 3:09 PM, Eric Van Hensbergen <ericvh@gmail.com> wrote:
>>> One thing I wonder is if we always want to zero copy for payload.  In
>>> the extreme, do we want to take the overhead of pinning an extra page
>>> if we are only reading/writing a byte?  memcpy is expensive for large
>>> packets, but may actually be more efficient for small packets.
>>>
>>> Have we done any performance measurements of this code with various
>>> payload sizes versus non-zero-copy?  Of course that may not really
>>> show the impact of pinning the extra pages....
>>>
>>> In any case, if such a tradeoff did exist, we might choose to not do
>>> zero copy for requests smaller than some size -- and that might
>>> alleviate some of the problems with the legacy protocols (such as the
>>> Rerror issue) -- killing two birds with one stone.  In any case, given
>>> the implementations it should be really easy to shut me up with
>>> comparison data of zc and non-zc for 1, 64, 128, 256, 512, 1024, 2048,
>>> 4192 byte payloads (without caches enabled of course).
>>>
>>>       -eric
>>>
>>>
>>> On Mon, Feb 7, 2011 at 12:57 AM, Venkateswararao Jujjuri (JV)
>>> <jvrao@linux.vnet.ibm.com> wrote:
>>>> On 2/6/2011 11:21 PM, Venkateswararao Jujjuri (JV) wrote:
>>>>> Signed-off-by: Venkateswararao Jujjuri <jvrao@linux.vnet.ibm.com>
>>>>> ---
>>>>>  net/9p/client.c   |   45 +++++++++++++++++++++++++++++++--------------
>>>>>  net/9p/protocol.c |   38 ++++++++++++++++++++++++++++++++++++++
>>>>>  2 files changed, 69 insertions(+), 14 deletions(-)
>>>>>
>>>>> diff --git a/net/9p/client.c b/net/9p/client.c
>>>>> index a848bca..f939edf 100644
>>>>> --- a/net/9p/client.c
>>>>> +++ b/net/9p/client.c
>>>>> @@ -1270,7 +1270,14 @@ p9_client_read(struct p9_fid *fid, char *data, char __user *udata, u64 offset,
>>>>>       if (count < rsize)
>>>>>               rsize = count;
>>>>>
>>>>> -     req = p9_client_rpc(clnt, P9_TREAD, "dqd", fid->fid, offset, rsize);
>>>>> +     if ((clnt->trans_mod->pref & P9_TRANS_PREF_PAYLOAD_MASK) ==
>>>>> +                     P9_TRANS_PREF_PAYLOAD_SEP) {
>>>>> +             req = p9_client_rpc(clnt, P9_TREAD, "dqE", fid->fid, offset,
>>>>> +                             rsize, data ? data : udata);
>>>>> +     } else {
>>>>> +             req = p9_client_rpc(clnt, P9_TREAD, "dqd", fid->fid, offset,
>>>>> +                             rsize);
>>>>> +     }
>>>>>       if (IS_ERR(req)) {
>>>>>               err = PTR_ERR(req);
>>>>>               goto error;
>>>>> @@ -1284,13 +1291,15 @@ p9_client_read(struct p9_fid *fid, char *data, char __user *udata, u64 offset,
>>>>>
>>>>>       P9_DPRINTK(P9_DEBUG_9P, "<<< RREAD count %d\n", count);
>>>>>
>>>>> -     if (data) {
>>>>> -             memmove(data, dataptr, count);
>>>>> -     } else {
>>>>> -             err = copy_to_user(udata, dataptr, count);
>>>>> -             if (err) {
>>>>> -                     err = -EFAULT;
>>>>> -                     goto free_and_error;
>>>>> +     if (!req->tc->pbuf_size) {
>>>>> +             if (data) {
>>>>> +                     memmove(data, dataptr, count);
>>>>> +             } else {
>>>>> +                     err = copy_to_user(udata, dataptr, count);
>>>>> +                     if (err) {
>>>>> +                             err = -EFAULT;
>>>>> +                             goto free_and_error;
>>>>> +                     }
>>>>>               }
>>>>>       }
>>>>>       p9_free_req(clnt, req);
>>>>> @@ -1323,12 +1332,20 @@ p9_client_write(struct p9_fid *fid, char *data, const char __user *udata,
>>>>>
>>>>>       if (count < rsize)
>>>>>               rsize = count;
>>>>> -     if (data)
>>>>> -             req = p9_client_rpc(clnt, P9_TWRITE, "dqD", fid->fid, offset,
>>>>> -                                                             rsize, data);
>>>>> -     else
>>>>> -             req = p9_client_rpc(clnt, P9_TWRITE, "dqU", fid->fid, offset,
>>>>> -                                                             rsize, udata);
>>>>> +
>>>>> +     if ((clnt->trans_mod->pref & P9_TRANS_PREF_PAYLOAD_MASK) ==
>>>>> +                     P9_TRANS_PREF_PAYLOAD_SEP) {
>>>>> +             req = p9_client_rpc(clnt, P9_TWRITE, "dqF", fid->fid, offset,
>>>>> +                             rsize, data ? data : udata);
>>>>> +     } else {
>>>>> +             if (data)
>>>>> +                     req = p9_client_rpc(clnt, P9_TWRITE, "dqD", fid->fid,
>>>>> +                                     offset, rsize, data);
>>>>> +             else
>>>>> +                     req = p9_client_rpc(clnt, P9_TWRITE, "dqU", fid->fid,
>>>>> +                                     offset, rsize, udata);
>>>>> +     }
>>>>> +
>>>>>       if (IS_ERR(req)) {
>>>>>               err = PTR_ERR(req);
>>>>>               goto error;
>>>>> diff --git a/net/9p/protocol.c b/net/9p/protocol.c
>>>>> index dfc358f..ea778dd 100644
>>>>> --- a/net/9p/protocol.c
>>>>> +++ b/net/9p/protocol.c
>>>>> @@ -114,6 +114,24 @@ pdu_write_u(struct p9_fcall *pdu, const char __user *udata, size_t size)
>>>>>       return size - len;
>>>>>  }
>>>>>
>>>>> +static size_t
>>>>> +pdu_write_uw(struct p9_fcall *pdu, const char *udata, size_t size)
>>>>> +{
>>>>> +     size_t len = min(pdu->capacity - pdu->size, size);
>>>>> +     pdu->pbuf = udata;
>>>>> +     pdu->pbuf_size = len;
>>>>> +     return size - len;
>>>>> +}
>>>>> +
>>>>> +static size_t
>>>>> +pdu_write_ur(struct p9_fcall *pdu, const char *udata, size_t size)
>>>>> +{
>>>>> +     size_t len = min(pdu->capacity - pdu->size, size);
>>>>> +     pdu->pbuf = udata;
>>>>> +     pdu->pbuf_size = len;
>>>>> +     return size - len;
>>>>> +}
>>>>> +
>>>>>  /*
>>>>>       b - int8_t
>>>>>       w - int16_t
>>>>> @@ -445,6 +463,26 @@ p9pdu_vwritef(struct p9_fcall *pdu, int proto_version, const char *fmt,
>>>>>                                       errcode = -EFAULT;
>>>>>                       }
>>>>>                       break;
>>>>> +             case 'E':{
>>>>> +                              int32_t count = va_arg(ap, int32_t);
>>>>> +                              const char *udata = va_arg(ap, const void *);
>>>>> +                              errcode = p9pdu_writef(pdu, proto_version, "d",
>>>>> +                                              count);
>>>>> +                              if (!errcode && pdu_write_ur(pdu, udata,
>>>>> +                                                      count))
>>>>> +                                      errcode = -EFAULT;
>>>>> +                      }
>>>>> +                      break;
>>>>> +             case 'F':{
>>>>> +                              int32_t count = va_arg(ap, int32_t);
>>>>> +                              const char *udata = va_arg(ap, const void *);
>>>>> +                              errcode = p9pdu_writef(pdu, proto_version, "d",
>>>>> +                                              count);
>>>>> +                              if (!errcode && pdu_write_uw(pdu, udata,
>>>>> +                                                      count))
>>>>> +                                      errcode = -EFAULT;
>>>>> +                      }
>>>>> +                      break;
>>>>>               case 'U':{
>>>>>                               int32_t count = va_arg(ap, int32_t);
>>>>>                               const char __user *udata =
>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> The modern datacenter depends on network connectivity to access resources
>>>> and provide services. The best practices for maximizing a physical server's
>>>> connectivity to a physical network are well understood - see how these
>>>> rules translate into the virtual world?
>>>> http://p.sf.net/sfu/oracle-sfdevnlfb
>>>> _______________________________________________
>>>> V9fs-developer mailing list
>>>> V9fs-developer@lists.sourceforge.net
>>>> https://lists.sourceforge.net/lists/listinfo/v9fs-developer
>>>>
>>>
> 
> 
> 
> ------------------------------------------------------------------------------
> The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE:
> Pinpoint memory and threading errors before they happen.
> Find and fix more than 250 security defects in the development cycle.
> Locate bottlenecks in serial and parallel code that limit performance.
> http://p.sf.net/sfu/intel-dev2devfeb
> _______________________________________________
> V9fs-developer mailing list
> V9fs-developer@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/v9fs-developer