Recent changes (master)

[Jens Axboe <axboe@kernel.dk>]

The following changes since commit 8ac0eec77e9c25a054ff1fbff243ffdffcd6e858:

  nvme/streams: avoid allocating too large a buffer (2024-08-05 18:15:24 +0000)

are available in the Git repository at:

  git://git.kernel.dk/fio.git master

for you to fetch changes up to 55b10ccca6a2ba623360a38c3fdf7332461646bb:

  Merge branch 'master' of https://github.com/scaleoutsean/fio (2024-08-06 09:17:03 -0600)

----------------------------------------------------------------
Jens Axboe (1):
      Merge branch 'master' of https://github.com/scaleoutsean/fio

Sean Lee (1):
      Improve http_host, filename in docs and example/http-s3.fio

 HOWTO.rst            | 14 ++++++++++++--
 examples/http-s3.fio | 16 +++++++++++++---
 fio.1                | 11 +++++++++--
 3 files changed, 34 insertions(+), 7 deletions(-)

---

Diff of recent changes:

diff --git a/HOWTO.rst b/HOWTO.rst
index 9eeb203e..11b1c669 100644
--- a/HOWTO.rst
+++ b/HOWTO.rst
@@ -805,6 +805,11 @@ Target file/device
 	Note: Windows and FreeBSD (refer to geom(4)) prevent write access to areas
 	of the disk containing in-use data (e.g. filesystems).
 
+	For HTTP and S3 access, specify a valid URL path or S3 key, respectively.
+	A filename for path-style S3 includes a bucket name (:file:`/bucket/k/e.y`)
+	while a virtual-hosted-style S3 filename :file:`/k/e.y` does not because 
+	its bucket name is specified in :option:`http_host`.
+
 	The filename "`-`" is a reserved name, meaning *stdin* or *stdout*.  Which
 	of the two depends on the read/write direction set.
 
@@ -2934,8 +2939,13 @@ with the caveat that when used on the command line, they must come after the
 
 .. option:: http_host=str : [http]
 
-	Hostname to connect to. For S3, this could be the bucket hostname.
-	Default is **localhost**
+	Hostname to connect to. HTTP port 80 is used automatically when the value of 
+	the https parameter is *off*, and HTTPS port 443 if it is *on*. A 
+	virtual-hosted-style S3 hostname starts with a bucket name, while a 
+	path-style S3 hostname does not. See 
+	https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html for 
+	detailed examples.
+	Default is **localhost** (path-style S3 hostname)
 
 .. option:: http_user=str : [http]
 
diff --git a/examples/http-s3.fio b/examples/http-s3.fio
index 2dcae364..c16e89b1 100644
--- a/examples/http-s3.fio
+++ b/examples/http-s3.fio
@@ -1,19 +1,29 @@
 # Example test for the HTTP engine's S3 support against Amazon AWS.
 # Obviously, you have to adjust the S3 credentials; for this example,
 # they're passed in via the environment.
-#
+# For non-AWS S3 implementations, refer to your S3 vendor's region 
+# settings. Note that the region value appears twice, in http_host and 
+# http_s3_region.
+# This example uses virtual-hosted-style requests:
+# https://bucket-name.s3.region-code.amazonaws.com/k/e.y
+# For path-style, prefix the key with the bucket name in the filename
+# so that filename=/bucket-name/k/e.y:
+# https://s3.region-code.amazonaws.com/bucket-name/k/e.y
+
+# Reference for Virtual-hosted-style vs. Path-style URLs:
+# https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html
 
 [global]
 ioengine=http
 name=test
 direct=1
-filename=/larsmb-fio-test/object
+filename=/k/e.y
 http_verbose=0
 https=on
 http_mode=s3
 http_s3_key=${S3_KEY}
 http_s3_keyid=${S3_ID}
-http_host=s3.eu-central-1.amazonaws.com
+http_host=bucket-name.s3.eu-central-1.amazonaws.com
 http_s3_region=eu-central-1
 group_reporting
 
diff --git a/fio.1 b/fio.1
index 15bafbf9..6db76b2c 100644
--- a/fio.1
+++ b/fio.1
@@ -585,6 +585,11 @@ the first device, `\\\\.\\PhysicalDrive1' for the second etc.
 Note: Windows and FreeBSD prevent write access to areas
 of the disk containing in-use data (e.g. filesystems).
 .P
+For HTTP and S3 access, specify a valid URL path or S3 key, respectively. 
+A filename for path-style S3 includes a bucket name (`/bucket/k/e.y') 
+while a virtual-hosted-style S3 filename (`/k/e.y') does not because its 
+bucket name is specified in \fBhttp_host\fR.
+.P
 The filename `\-' is a reserved name, meaning *stdin* or *stdout*. Which
 of the two depends on the read/write direction set.
 .RE
@@ -2554,8 +2559,10 @@ Touching all objects affects ceph caches and likely impacts test results.
 Enabled by default.
 .TP
 .BI (http)http_host \fR=\fPstr
-Hostname to connect to. For S3, this could be the bucket name. Default
-is \fBlocalhost\fR
+Hostname to connect to.  HTTP port 80 is used automatically when the value 
+of the \fBhttps\fP parameter is \fRoff\fP, and HTTPS port 443 if it is \Ron\fP.  
+A virtual-hosted-style S3 hostname starts with a bucket name, while a 
+path-style S3 hostname does not.  Default is \fBlocalhost\fR.
 .TP
 .BI (http)http_user \fR=\fPstr
 Username for HTTP authentication.