Re: [PATCH v2 3/3] block/curl: add support for S3 presigned URLs

[Markus Armbruster <armbru@redhat.com>]

Antoine Damhet <adamhet@scaleway.com> writes:

> S3 presigned URLs are signed for a specific HTTP method (typically GET
> for our use cases). The curl block driver currently issues a HEAD
> request to discover the web server features and the file size, which
> fails with 'HTTP 403' (forbidden).
>
> Add a 'force-range' option that skips the HEAD request and instead
> issues a minimal GET request (querying 1 byte from the server) to
> extract the file size from the 'Content-Range' response header. To
> achieve this the 'curl_header_cb' is redesigned to generically parse
> HTTP headers.
>
> $ $QEMU -drive driver=https,\
>              'url=https://s3.example.com/some.img?X-Amz-Security-Token=XXX',
>              force-range=true
>
> Enabling the 'force-range' option without the web server specified with
> @url supporting it might cause the server to respond successfully with
> 'HTTP 200' and attempt to send the whole file body. With the
> 'CURLOPT_NOBODY' option set the libcurl will skip reading after the
> headers and close the connection. QEMU still gracefully detects the
> missing feature. This might waste a small number of TCP packets but is
> otherwise transparent to the user.
>
> Signed-off-by: Antoine Damhet <adamhet@scaleway.com>

[...]

> diff --git a/docs/system/device-url-syntax.rst.inc b/docs/system/device-url-syntax.rst.inc
> index aae65d138c00..445e2a0a4157 100644
> --- a/docs/system/device-url-syntax.rst.inc
> +++ b/docs/system/device-url-syntax.rst.inc
> @@ -179,6 +179,12 @@ These are specified using a special URL syntax.
>        get the size of the image to be downloaded. If not set, the
>        default timeout of 5 seconds is used.
>  
> +   ``force-range``
> +      Assume the HTTP backend supports range requests and avoid doing
> +      an HTTP HEAD request to discover the feature. Typically S3
> +      presigned URLs will only support one method and refuse other
> +      request types.
> +

Similar to the description in qapi/block-core.json.  I find the latter
clearer.  Perhaps you'd like to use it here.  Entirely up to you.

>     Note that when passing options to qemu explicitly, ``driver`` is the
>     value of <protocol>.
>  
> diff --git a/qapi/block-core.json b/qapi/block-core.json
> index a7871705fa69..50e7078cbec0 100644
> --- a/qapi/block-core.json
> +++ b/qapi/block-core.json
> @@ -4582,12 +4582,19 @@
>  # @cookie-secret: ID of a QCryptoSecret object providing the cookie
>  #     data in a secure way.  See @cookie for the format.  (since 2.10)
>  #
> +# @force-range: Don't issue a HEAD HTTP request to discover if the
> +#     http server supports range requests and rely only on GET
> +#     requests.  This is especially useful for S3 presigned URLs where
> +#     HEAD requests are unauthorized.  Defaults to false.
> +#     (default: false; since 11.0)
> +#
>  # Since: 2.9
>  ##
>  { 'struct': 'BlockdevOptionsCurlHttp',
>    'base': 'BlockdevOptionsCurlBase',
>    'data': { '*cookie': 'str',
> -            '*cookie-secret': 'str'} }
> +            '*cookie-secret': 'str',
> +            '*force-range': 'bool'} }
>  
>  ##
>  # @BlockdevOptionsCurlHttps:

QAPI schema
Acked-by: Markus Armbruster <armbru@redhat.com>