Resending: fio testing on S3 object storage

Resending: fio testing on S3 object storage

[eliezer]

Greetings,

First of all, my apologies for re-sending this email.  I've now sent it to
the correct maillist this time and in the proper mail format.

How does one run fio against an S3 object storage device?

Here�s the fio parameters I�m using but I�m getting the below error message:

# fio --ioengine=http --iodepth=1 --http_mode=s3 --http_s3_region=us-geo
--http_s3_key=123mYkEyInfo --http_s3_keyid=123mYkEyiDinfO --bs=16k
--size=100MB --http_host=s3-api.us-geo.objectstorage.inthecloud.net
--https=on --http_verbose=2
fio: engine https not loadable
fio: failed to load engine

Am I missing a kernel module required by fio �or am I missing a few more fio
parameters from the command line?�� If so, what do I need to do to enable
that module?

Here�s my server info:

# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.5 (Maipo)

# fio --version
fio-3.1

# uname -a
Linux mytestserver.testfio.com 3.10.0-862.11.6.el7.x86_64 #1 SMP Fri Aug 10
16:55:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Many thanks in advance for your attention and your response(s).

Regards,
Eliezer

RE: Resending: fio testing on S3 object storage

[eliezer]

It looks like I don�t have the "http" ioengine on my server:

# fio --enghelp
Available IO engines:
        cpuio
        mmap
        sync
        psync
        vsync
        pvsync
        pvsync2
        null
        net
        netsplice
        ftruncate
        libaio
        rdma
        posixaio
        falloc
        e4defrag
        splice
        rbd
        mtd
        pmemblk
        dev-dax
        sg
        binject

How would I go about adding the "http" ioengine on my server?

Thank you in advance.

Regards,
Eliezer

-----Original Message-----
From: Eliezer Nebab (eliezer@integritech.solutions)
[mailto:eliezer@integritech.solutions] 
Sent: Wednesday, September 26, 2018 10:52 AM
To: 'fio@vger.kernel.org'
Subject: Resending: fio testing on S3 object storage

Greetings,

First of all, my apologies for re-sending this email.  I've now sent it to
the correct maillist this time and in the proper mail format.

How does one run fio against an S3 object storage device?

Here�s the fio parameters I�m using but I�m getting the below error message:

# fio --ioengine=http --iodepth=1 --http_mode=s3 --http_s3_region=us-geo
--http_s3_key=123mYkEyInfo --http_s3_keyid=123mYkEyiDinfO --bs=16k
--size=100MB --http_host=s3-api.us-geo.objectstorage.inthecloud.net
--https=on --http_verbose=2
fio: engine https not loadable
fio: failed to load engine

Am I missing a kernel module required by fio �or am I missing a few more fio
parameters from the command line?�� If so, what do I need to do to enable
that module?

Here�s my server info:

# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.5 (Maipo)

# fio --version
fio-3.1

# uname -a
Linux mytestserver.testfio.com 3.10.0-862.11.6.el7.x86_64 #1 SMP Fri Aug 10
16:55:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Many thanks in advance for your attention and your response(s).

Regards,
Eliezer

Re: Resending: fio testing on S3 object storage

[Sitsofe Wheeler]

Hi,

You
On Wed, 26 Sep 2018 at 17:54, <eliezer@integritech.solutions> wrote:
>
> It looks like I don’t have the "http" ioengine on my server:
>
> # fio --enghelp
> Available IO engines:
>         cpuio
>         mmap
>         sync
>         psync
>         vsync
>         pvsync
>         pvsync2
>         null
>         net
>         netsplice
>         ftruncate
>         libaio
>         rdma
>         posixaio
>         falloc
>         e4defrag
>         splice
>         rbd
>         mtd
>         pmemblk
>         dev-dax
>         sg
>         binject

You need to have the appropriate dependencies available when you go to
compile fio (you'll find that ./configure says that http is enabled).
I believe for the http engine you need the development libraries for
curl and openssl installed...

-- 
Sitsofe | http://sucs.org/~sits/

RE: Resending: fio testing on S3 object storage

[eliezer]

Thank you Sitsofe and Samuel for the response.

Per your suggestion, I installed the following packages on my RH 7.5:

openssl
openssl-devel
curl-devel

Then I downloaded the latest fio source and compiled it.

Now, I'm seeing the "http" engine listed:

# fio --enghelp
Available IO engines:
        cpuio
        mmap
        sync
        psync
        vsync
        pvsync
        pvsync2
        null
        net
        netsplice
        ftruncate
        filecreate
        posixaio
        falloc
        e4defrag
        splice
>>    http  <<
        mtd
        sg


Many thanks!

Do you have some examples of fio benchmark params for object storage that you could share?


Regards,
Eliezer


-----Original Message-----
From: fio-owner@vger.kernel.org [mailto:fio-owner@vger.kernel.org] On Behalf Of Sitsofe Wheeler
Sent: Thursday, September 27, 2018 8:18 AM
To: eliezer@integritech.solutions
Cc: fio
Subject: Re: Resending: fio testing on S3 object storage

Hi,

You
On Wed, 26 Sep 2018 at 17:54, <eliezer@integritech.solutions> wrote:
>
> It looks like I don’t have the "http" ioengine on my server:
>
> # fio --enghelp
> Available IO engines:
>         cpuio
>         mmap
>         sync
>         psync
>         vsync
>         pvsync
>         pvsync2
>         null
>         net
>         netsplice
>         ftruncate
>         libaio
>         rdma
>         posixaio
>         falloc
>         e4defrag
>         splice
>         rbd
>         mtd
>         pmemblk
>         dev-dax
>         sg
>         binject

You need to have the appropriate dependencies available when you go to
compile fio (you'll find that ./configure says that http is enabled).
I believe for the http engine you need the development libraries for
curl and openssl installed...

-- 
Sitsofe | http://sucs.org/~sits/

Re: Resending: fio testing on S3 object storage

[Samuel Shapiro]

Well, I'm just investigating this engine myself as well, but you can
start playing with that workload and modify it accordingly:
https://github.com/axboe/fio/blob/master/examples/http-s3.fio

On Thu, Sep 27, 2018 at 7:43 PM <eliezer@integritech.solutions> wrote:
>
> Thank you Sitsofe and Samuel for the response.
>
> Per your suggestion, I installed the following packages on my RH 7.5:
>
> openssl
> openssl-devel
> curl-devel
>
> Then I downloaded the latest fio source and compiled it.
>
> Now, I'm seeing the "http" engine listed:
>
> # fio --enghelp
> Available IO engines:
>         cpuio
>         mmap
>         sync
>         psync
>         vsync
>         pvsync
>         pvsync2
>         null
>         net
>         netsplice
>         ftruncate
>         filecreate
>         posixaio
>         falloc
>         e4defrag
>         splice
> >>    http  <<
>         mtd
>         sg
>
>
> Many thanks!
>
> Do you have some examples of fio benchmark params for object storage that you could share?
>
>
> Regards,
> Eliezer
>
>
> -----Original Message-----
> From: fio-owner@vger.kernel.org [mailto:fio-owner@vger.kernel.org] On Behalf Of Sitsofe Wheeler
> Sent: Thursday, September 27, 2018 8:18 AM
> To: eliezer@integritech.solutions
> Cc: fio
> Subject: Re: Resending: fio testing on S3 object storage
>
> Hi,
>
> You
> On Wed, 26 Sep 2018 at 17:54, <eliezer@integritech.solutions> wrote:
> >
> > It looks like I don’t have the "http" ioengine on my server:
> >
> > # fio --enghelp
> > Available IO engines:
> >         cpuio
> >         mmap
> >         sync
> >         psync
> >         vsync
> >         pvsync
> >         pvsync2
> >         null
> >         net
> >         netsplice
> >         ftruncate
> >         libaio
> >         rdma
> >         posixaio
> >         falloc
> >         e4defrag
> >         splice
> >         rbd
> >         mtd
> >         pmemblk
> >         dev-dax
> >         sg
> >         binject
>
> You need to have the appropriate dependencies available when you go to
> compile fio (you'll find that ./configure says that http is enabled).
> I believe for the http engine you need the development libraries for
> curl and openssl installed...
>
> --
> Sitsofe | http://sucs.org/~sits/
>