Re: [Qemu-devel] [PATCH v20 5/5] block/gluster: add support for multiple gluster servers

[Eric Blake <eblake@redhat.com>]

[-- Attachment #1: Type: text/plain, Size: 10704 bytes --]

On 07/19/2016 10:57 AM, Prasanna Kumar Kalever wrote:
> This patch adds a way to specify multiple volfile servers to the gluster
> block backend of QEMU with tcp|rdma transport types and their port numbers.

If rdma is deprecated, we don't need to mention it here.

> 
> Problem:
> 
> Currently VM Image on gluster volume is specified like this:
> 
> file=gluster[+tcp]://host[:port]/testvol/a.img
> 
> Say we have three hosts in a trusted pool with replica 3 volume in action.
> When the host mentioned in the command above goes down for some reason,
> the other two hosts are still available. But there's currently no way
> to tell QEMU about them.
> 
> Solution:
> 
> New way of specifying VM Image on gluster volume with volfile servers:
> (We still support old syntax to maintain backward compatibility)
> 
> Basic command line syntax looks like:
> 
> Pattern I:
>  -drive driver=gluster,
>         volume=testvol,path=/path/a.raw,[debug=N,]
>         server.0.type=tcp,
>         server.0.host=1.2.3.4,
>         server.0.port=24007,
>         server.1.type=unix,
>         server.1.socket=/path/socketfile

So, I haven't checked this yet, but if I'm correct, the old syntax was:

-drive driver=gluster,
       volume=testvol,path=/path/a.raw,
       server.type=tcp,
       server.host=1.2.3.4,
       server.port=24007

Is that syntax still supported?  That is, if I only specify one server,
does 'server.FOO' mean the same as 'server.0.FOO'?  Or am I completely
wrong?  Maybe listing the old syntax for comparison is in order,
especially since you claim above that the old syntax is still supported.


> 
> Pattern II:
>  'json:{"driver":"qcow2","file":{"driver":"gluster",
>        "volume":"testvol","path":"/path/a.qcow2",["debug":N,]
>        "server":[{hostinfo_1}, ...{hostinfo_N}]}}'
> 
>    driver      => 'gluster' (protocol name)
>    volume      => name of gluster volume where our VM image resides
>    path        => absolute path of image in gluster volume
>   [debug]      => libgfapi loglevel [(0 - 9) default 4 -> Error]
> 
>   {hostinfo}   => {{type:"tcp",host:"1.2.3.4"[,port=24007]},
>                    {type:"unix",socket:"/path/sockfile"}}
> 
>    type        => transport type used to connect to gluster management daemon,
>                   it can be tcp|unix
>    host        => host address (hostname/ipv4/ipv6 addresses/socket path)

You dropped ipv6 in the last patch, if you want to update this comment.

>    port        => port number on which glusterd is listening.
>    socket      => path to socket file
> 
> Examples:
> 1.
>  -drive driver=qcow2,file.driver=gluster,
>         file.volume=testvol,file.path=/path/a.qcow2,file.debug=9,
>         file.server.0.type=tcp,
>         file.server.0.host=1.2.3.4,
>         file.server.0.port=24007,
>         file.server.1.type=tcp,
>         file.server.1.socket=/var/run/glusterd.socket
> 2.
>  'json:{"driver":"qcow2","file":{"driver":"gluster","volume":"testvol",
>          "path":"/path/a.qcow2","debug":9,"server":
>          [{type:"tcp",host:"1.2.3.4",port=24007},
>           {type:"unix",socket:"/var/run/glusterd.socket"}] } }'
> 
> This patch gives a mechanism to provide all the server addresses, which are in
> replica set, so in case host1 is down VM can still boot from any of the
> active hosts.
> 
> This is equivalent to the backup-volfile-servers option supported by
> mount.glusterfs (FUSE way of mounting gluster volume)
> 
> credits: sincere thanks to all the supporters
> 
> Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
> ---
>  block/gluster.c      | 397 +++++++++++++++++++++++++++++++++++++++++++++------
>  qapi/block-core.json |   2 +-
>  2 files changed, 358 insertions(+), 41 deletions(-)
> 

> +static QemuOptsList runtime_tcp_opts = {
> +    .name = "gluster_tcp",
> +    .head = QTAILQ_HEAD_INITIALIZER(runtime_tcp_opts.head),
> +    .desc = {
> +        {
> +            .name = GLUSTER_OPT_TYPE,
> +            .type = QEMU_OPT_STRING,
> +            .help = "tcp|unix",
> +        },
> +        {
> +            .name = GLUSTER_OPT_HOST,
> +            .type = QEMU_OPT_STRING,
> +            .help = "host address (hostname/ipv4/ipv6 addresses)",
> +        },

Awkward to state ipv6 here,

> +        {
> +            .name = GLUSTER_OPT_PORT,
> +            .type = QEMU_OPT_NUMBER,
> +            .help = "port number on which glusterd is listening (default 24007)",
> +        },
> +        {
> +            .name = "to",
> +            .type = QEMU_OPT_NUMBER,
> +            .help = "max port number, not supported by gluster",
> +        },
> +        {
> +            .name = "ipv4",
> +            .type = QEMU_OPT_BOOL,
> +            .help = "ipv4 bool value, not supported by gluster",
> +        },
> +        {
> +            .name = "ipv6",
> +            .type = QEMU_OPT_BOOL,
> +            .help = "ipv6 bool value, not supported by gluster",

but then to state it is not supported here.

Do we actually have to provide QemuOpt listings for the options that we
don't actually support?  That is, must QemuOpt precisely match the
InetAddressSocket struct we are parsing into, or can it be merely a
subset where we omit the portions we won't use?


> @@ -284,6 +381,226 @@ out:
>      return NULL;
>  }
>  
> +static int qapi_enum_parse(const char *opt)
> +{
> +    int i;
> +
> +    if (!opt) {
> +        return GLUSTER_TRANSPORT__MAX;
> +    }
> +
> +    for (i = 0; i < GLUSTER_TRANSPORT__MAX; i++) {
> +        if (!strcmp(opt, GlusterTransport_lookup[i])) {
> +            return i;
> +        }
> +    }
> +
> +    return i;

Is this duplicating any functionality that we already have?

> +}
> +
> +/*
> + * Convert the json formatted command line into qapi.
> +*/
> +static int qemu_gluster_parse_json(BlockdevOptionsGluster *gconf,
> +                                  QDict *options, Error **errp)

Indentation is off.

> +{
> +    QemuOpts *opts;
> +    GlusterServer *gsconf;
> +    GlusterServerList *curr = NULL;
> +    QDict *backing_options = NULL;
> +    Error *local_err = NULL;
> +    char *str = NULL;
> +    const char *ptr;
> +    size_t num_servers;
> +    int i;
> +
> +    /* create opts info from runtime_json_opts list */
> +    opts = qemu_opts_create(&runtime_json_opts, NULL, 0, &error_abort);
> +    qemu_opts_absorb_qdict(opts, options, &local_err);

In fact, if we were to use opts_visitor_new(), could we let the
already-existing generated QAPI code parse from QemuOpts into QAPI form
without having to do it by hand here?  Hmm, maybe not...

> +    if (local_err) {
> +        goto out;
> +    }
> +
> +    num_servers = qdict_array_entries(options, GLUSTER_OPT_SERVER_PATTERN);
> +    if (num_servers < 1) {
> +        error_setg(&local_err, QERR_MISSING_PARAMETER, "server");
> +        goto out;
> +    }
> +
> +    ptr = qemu_opt_get(opts, GLUSTER_OPT_VOLUME);
> +    if (!ptr) {
> +        error_setg(&local_err, QERR_MISSING_PARAMETER, GLUSTER_OPT_VOLUME);
> +        goto out;
> +    }
> +    gconf->volume = g_strdup(ptr);
> +
> +    ptr = qemu_opt_get(opts, GLUSTER_OPT_PATH);
> +    if (!ptr) {
> +        error_setg(&local_err, QERR_MISSING_PARAMETER, GLUSTER_OPT_PATH);
> +        goto out;
> +    }
> +    gconf->path = g_strdup(ptr);
> +    qemu_opts_del(opts);
> +
> +    for (i = 0; i < num_servers; i++) {
> +        str = g_strdup_printf(GLUSTER_OPT_SERVER_PATTERN"%d.", i);
> +        qdict_extract_subqdict(options, &backing_options, str);

...since the OptsVisitor doesn't handle structs, but we are definitely
parsing a sub-struct.  I also wonder if Dan's work on a new string-based
qmp-input-visitor, and/or his qdict_crumple() work, could reduce the
amount of effort needed here.

It's not to say that your patch is wrong, only that we may have
follow-up patches that can improve it.

> +
> +        /* create opts info from runtime_type_opts list */
> +        opts = qemu_opts_create(&runtime_type_opts, NULL, 0, &error_abort);
> +        qemu_opts_absorb_qdict(opts, backing_options, &local_err);
> +        if (local_err) {
> +            goto out;
> +        }
> +
> +        ptr = qemu_opt_get(opts, GLUSTER_OPT_TYPE);
> +        gsconf = g_new0(GlusterServer, 1);
> +        gsconf->type = qapi_enum_parse(ptr);
> +        if (!ptr) {
> +            error_setg(&local_err, QERR_MISSING_PARAMETER, GLUSTER_OPT_TYPE);
> +            error_append_hint(&local_err, GERR_INDEX_HINT, i);

Nothing else in our code base uses GERR_INDEX_HINT. I'd MUCH prefer
open-coding the string instead of relying on some unknown string literal
from glib, since we have less control over whether glib will always keep
the format markers lined up with what we are using.


> +            /* defend for unsupported fields in InetSocketAddress,
> +             * i.e. @ipv4, @ipv6  and @to
> +             */
> +            ptr = qemu_opt_get(opts, GLUSTER_OPT_TO);
> +            if (ptr) {
> +                gsconf->u.tcp.has_to = true;
> +            }
> +            ptr = qemu_opt_get(opts, GLUSTER_OPT_IPV4);
> +            if (ptr) {
> +                gsconf->u.tcp.has_ipv4 = true;
> +            }
> +            ptr = qemu_opt_get(opts, GLUSTER_OPT_IPV6);
> +            if (ptr) {
> +                gsconf->u.tcp.has_ipv6 = true;
> +            }
> +            if (gsconf->u.tcp.has_to) {
> +                error_setg(&local_err, "Parameter 'to' not supported");
> +                goto out;
> +            }
> +            if (gsconf->u.tcp.has_ipv4 || gsconf->u.tcp.has_ipv6) {
> +                error_setg(&local_err, "Parameters 'ipv4/ipv6' not supported");

I know we are rejecting ipv6 until gluster supports it, but do we really
have to reject ipv4?  On the other hand, it's always
backwards-compatible to relax this restriction later on, but harder to
add it in after a release where it was not present.


> +++ b/qapi/block-core.json
> @@ -2111,7 +2111,7 @@
>  { 'struct': 'BlockdevOptionsGluster',
>    'data': { 'volume': 'str',
>              'path': 'str',
> -            'server': 'GlusterServer',
> +            'server': ['GlusterServer'],

Documentation should probably be tweaked to mention that 'server' is now
a list of servers, and should not be empty.  For that matter, did you
test that the error message is sane when there are no servers listed?

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]