* [PATCH v1] erofs-utils: refactor OCI code for better modularity
@ 2025-09-01 5:10 ChengyuZhu6
2025-09-01 7:01 ` [PATCH v1-changed] " ChengyuZhu6
` (5 more replies)
0 siblings, 6 replies; 19+ messages in thread
From: ChengyuZhu6 @ 2025-09-01 5:10 UTC (permalink / raw)
To: linux-erofs; +Cc: xiang, hsiangkao, Chengyu Zhu
From: Chengyu Zhu <hudsonzhu@tencent.com>
Refactor OCI code to improve code organization and maintainability:
- Add `struct ocierofs_layer_info` to encapsulate layer metadata
- Extract authentication logic into `ocierofs_prepare_auth()`
- Split layer processing into `ocierofs_prepare_layers()`
- Move OCI parsing functions from `mkfs/main.c` to `lib/remotes/oci.c`
- Add `ocierofs_process_tar_stream()` for separate tar processing
- Improve error handling with `ocierofs_free_layers_info()`
- Refactor `ocierofs_extract_layer()` to return file descriptor
Signed-off-by: Chengyu Zhu <hudsonzhu@tencent.com>
---
lib/liberofs_oci.h | 89 ++++++++
lib/remotes/oci.c | 536 ++++++++++++++++++++++++++++++++++-----------
mkfs/main.c | 200 +----------------
3 files changed, 503 insertions(+), 322 deletions(-)
diff --git a/lib/liberofs_oci.h b/lib/liberofs_oci.h
index 3a8108b..bd83966 100644
--- a/lib/liberofs_oci.h
+++ b/lib/liberofs_oci.h
@@ -19,6 +19,23 @@ struct erofs_inode;
struct CURL;
struct erofs_importer;
+/**
+ * struct ocierofs_layer_info
+ * @digest: OCI content-addressable digest (e.g. "sha256:...")
+ * @media_type: mediaType string from the manifest
+ * @size: layer size in bytes from the manifest (0 if not available)
+ *
+ * This structure is exposed to callers so they can enumerate image layers,
+ * decide which ones to fetch, and pass the digest back to download APIs.
+ * Fields are heap-allocated NUL-terminated strings owned by the caller
+ * once returned from public APIs; the caller must free them.
+ */
+struct ocierofs_layer_info {
+ char *digest;
+ char *media_type;
+ u64 size;
+};
+
/**
* struct erofs_oci_params - OCI configuration parameters
* @registry: registry hostname (e.g., "registry-1.docker.io")
@@ -88,6 +105,78 @@ int erofs_oci_params_set_string(char **field, const char *value);
*/
int ocierofs_build_trees(struct erofs_importer *importer, struct erofs_oci *oci);
+/*
+ * ocierofs_parse_options - Parse comma-separated OCI options string
+ * @oci: OCI client structure to update
+ * @options_str: comma-separated options string
+ *
+ * Parse OCI options string containing comma-separated key=value pairs.
+ * Supported options include platform, layer, username, and password.
+ *
+ * Return: 0 on success, negative errno on failure
+ */
+int ocierofs_parse_options(struct erofs_oci *oci, char *options_str);
+
+/*
+ * ocierofs_parse_ref - Parse OCI image reference string
+ * @oci: OCI client structure to update
+ * @ref_str: OCI image reference in various formats
+ *
+ * Parse OCI image reference which can be in formats:
+ * - registry.example.com/namespace/repo:tag
+ * - namespace/repo:tag (uses default registry)
+ * - repo:tag (adds library/ prefix for Docker Hub)
+ * - repo (uses default tag "latest")
+ *
+ * Return: 0 on success, negative errno on failure
+ */
+int ocierofs_parse_ref(struct erofs_oci *oci, const char *ref_str);
+
+/*
+ * ocierofs_prepare_layers - Prepare OCI layers for processing
+ * @oci: OCI client structure with configured parameters
+ * @auth_header: Pointer to store authentication header
+ * @using_basic: Pointer to store basic auth flag
+ * @manifest_digest: Pointer to store manifest digest
+ * @layers: Pointer to store layers information
+ * @layer_count: Pointer to store number of layers
+ * @start_index: Pointer to store starting layer index
+ *
+ * Prepare authentication, get manifest digest and layers information
+ * for OCI image processing. This function handles all the preparation
+ * work needed before processing OCI layers.
+ *
+ * Return: 0 on success, negative errno on failure
+ */
+int ocierofs_prepare_layers(struct erofs_oci *oci, char **auth_header,
+ bool *using_basic, char **manifest_digest,
+ struct ocierofs_layer_info ***layers,
+ int *layer_count, int *start_index);
+
+/**
+ * ocierofs_free_layers_info - Free layer information array
+ * @layers: array of layer information structures
+ * @count: number of layers in the array
+ *
+ * Free all layer information structures and the array itself.
+ * This function handles NULL pointers safely.
+ */
+void ocierofs_free_layers_info(struct ocierofs_layer_info **layers, int count);
+
+/**
+ * ocierofs_prepare_auth - Prepare authentication for OCI requests
+ * @oci: OCI client structure
+ * @auth_header_out: pointer to store authentication header
+ * @using_basic_auth: pointer to store basic auth flag
+ *
+ * Prepare authentication header for OCI registry requests.
+ * This function handles both token-based and basic authentication.
+ *
+ * Return: 0 on success, negative errno on failure
+ */
+int ocierofs_prepare_auth(struct erofs_oci *oci, char **auth_header_out,
+ bool *using_basic_auth);
+
#ifdef __cplusplus
}
#endif
diff --git a/lib/remotes/oci.c b/lib/remotes/oci.c
index 0fb8c1f..803175a 100644
--- a/lib/remotes/oci.c
+++ b/lib/remotes/oci.c
@@ -42,7 +42,6 @@ struct erofs_oci_response {
};
struct erofs_oci_stream {
- struct erofs_tarfile tarfile;
const char *digest;
int blobfd;
};
@@ -181,7 +180,7 @@ static int ocierofs_request_perform(struct erofs_oci *oci,
ret = ocierofs_curl_setup_rq(oci->curl, req->url,
OCIEROFS_HTTP_GET, req->headers,
- ocierofs_write_callback, resp,
+ ocierofs_write_callback, resp,
NULL, NULL);
if (ret)
return ret;
@@ -568,7 +567,7 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci,
const char *api_registry;
int ret = 0, len, i;
- if (!registry || !repository || !tag || !platform)
+ if (!registry || !repository || !tag)
return ERR_PTR(-EINVAL);
api_registry = (!strcmp(registry, DOCKER_REGISTRY)) ? DOCKER_API_REGISTRY : registry;
@@ -581,8 +580,8 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci,
req.headers = curl_slist_append(req.headers,
"Accept: " DOCKER_MEDIATYPE_MANIFEST_LIST ","
- OCI_MEDIATYPE_INDEX "," DOCKER_MEDIATYPE_MANIFEST_V1 ","
- DOCKER_MEDIATYPE_MANIFEST_V2);
+ OCI_MEDIATYPE_INDEX "," OCI_MEDIATYPE_MANIFEST ","
+ DOCKER_MEDIATYPE_MANIFEST_V1 "," DOCKER_MEDIATYPE_MANIFEST_V2);
ret = ocierofs_request_perform(oci, &req, &resp);
if (ret)
@@ -663,7 +662,24 @@ out:
return ret ? ERR_PTR(ret) : digest;
}
-static char **ocierofs_get_layers_info(struct erofs_oci *oci,
+void ocierofs_free_layers_info(struct ocierofs_layer_info **layers, int count)
+{
+ int i;
+
+ if (!layers)
+ return;
+
+ for (i = 0; i < count; i++) {
+ if (layers[i]) {
+ free(layers[i]->digest);
+ free(layers[i]->media_type);
+ free(layers[i]);
+ }
+ }
+ free(layers);
+}
+
+static struct ocierofs_layer_info **ocierofs_fetch_layers_info(struct erofs_oci *oci,
const char *registry,
const char *repository,
const char *digest,
@@ -672,10 +688,10 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci,
{
struct erofs_oci_request req = {};
struct erofs_oci_response resp = {};
- json_object *root, *layers, *layer, *digest_obj;
- char **layers_info = NULL;
+ json_object *root, *layers, *layer, *digest_obj, *media_type_obj, *size_obj;
+ struct ocierofs_layer_info **layers_info = NULL;
const char *api_registry;
- int ret, len, i, j;
+ int ret, len, i;
if (!registry || !repository || !digest || !layer_count)
return ERR_PTR(-EINVAL);
@@ -725,7 +741,7 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci,
goto out_json;
}
- layers_info = calloc(len, sizeof(char *));
+ layers_info = calloc(len, sizeof(*layers_info));
if (!layers_info) {
ret = -ENOMEM;
goto out_json;
@@ -740,11 +756,25 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci,
goto out_free;
}
- layers_info[i] = strdup(json_object_get_string(digest_obj));
+ layers_info[i] = calloc(1, sizeof(**layers_info));
if (!layers_info[i]) {
ret = -ENOMEM;
goto out_free;
}
+ layers_info[i]->digest = strdup(json_object_get_string(digest_obj));
+ if (!layers_info[i]->digest) {
+ ret = -ENOMEM;
+ goto out_free;
+ }
+ if (json_object_object_get_ex(layer, "mediaType", &media_type_obj))
+ layers_info[i]->media_type = strdup(json_object_get_string(media_type_obj));
+ else
+ layers_info[i]->media_type = NULL;
+
+ if (json_object_object_get_ex(layer, "size", &size_obj))
+ layers_info[i]->size = json_object_get_int64(size_obj);
+ else
+ layers_info[i]->size = 0;
}
*layer_count = len;
@@ -756,11 +786,7 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci,
return layers_info;
out_free:
- if (layers_info) {
- for (j = 0; j < i; j++)
- free(layers_info[j]);
- }
- free(layers_info);
+ ocierofs_free_layers_info(layers_info, i);
out_json:
json_object_put(root);
out:
@@ -771,8 +797,93 @@ out:
return ERR_PTR(ret);
}
-static int ocierofs_extract_layer(struct erofs_oci *oci, struct erofs_importer *importer,
- const char *digest, const char *auth_header)
+/**
+ * ocierofs_process_tar_stream - Process tar stream from file descriptor
+ * @importer: EROFS importer structure
+ * @fd: File descriptor containing tar data
+ *
+ * Initialize tar stream, parse all entries, and clean up resources.
+ *
+ * Return: 0 on success, negative errno on failure
+ */
+static int ocierofs_process_tar_stream(struct erofs_importer *importer, int fd)
+{
+ struct erofs_tarfile tarfile = {};
+ int ret;
+
+ memset(&tarfile, 0, sizeof(tarfile));
+ init_list_head(&tarfile.global.xattrs);
+
+ ret = erofs_iostream_open(&tarfile.ios, fd, EROFS_IOS_DECODER_GZIP);
+ if (ret) {
+ erofs_err("failed to initialize tar stream: %s",
+ erofs_strerror(ret));
+ return ret;
+ }
+
+ do {
+ ret = tarerofs_parse_tar(importer, &tarfile);
+ /* Continue parsing until end of archive */
+ } while (!ret);
+ erofs_iostream_close(&tarfile.ios);
+
+ if (ret < 0 && ret != -ENODATA) {
+ erofs_err("failed to process tar stream: %s",
+ erofs_strerror(ret));
+ return ret;
+ }
+
+ return 0;
+}
+
+int ocierofs_prepare_auth(struct erofs_oci *oci, char **auth_header_out,
+ bool *using_basic_auth)
+{
+ char *auth_header = NULL;
+ int ret = 0;
+
+ if (using_basic_auth)
+ *using_basic_auth = false;
+ if (auth_header_out)
+ *auth_header_out = NULL;
+
+ if (oci->params.username && oci->params.password &&
+ oci->params.username[0] && oci->params.password[0]) {
+ auth_header = ocierofs_get_auth_token(oci,
+ oci->params.registry,
+ oci->params.repository,
+ oci->params.username,
+ oci->params.password);
+ if (IS_ERR(auth_header)) {
+ auth_header = NULL;
+ ret = ocierofs_curl_setup_basic_auth(oci->curl,
+ oci->params.username,
+ oci->params.password);
+ if (ret)
+ return ret;
+ if (using_basic_auth)
+ *using_basic_auth = true;
+ }
+ } else {
+ auth_header = ocierofs_get_auth_token(oci,
+ oci->params.registry,
+ oci->params.repository,
+ NULL, NULL);
+ if (IS_ERR(auth_header))
+ auth_header = NULL;
+ }
+
+ if (auth_header_out)
+ *auth_header_out = auth_header;
+ else
+ free(auth_header);
+ return 0;
+}
+
+static int ocierofs_download_blob_to_fd(struct erofs_oci *oci,
+ const char *digest,
+ const char *auth_header,
+ int outfd)
{
struct erofs_oci_request req = {};
struct erofs_oci_stream stream = {};
@@ -782,20 +893,14 @@ static int ocierofs_extract_layer(struct erofs_oci *oci, struct erofs_importer *
stream = (struct erofs_oci_stream) {
.digest = digest,
- .blobfd = erofs_tmpfile(),
+ .blobfd = outfd,
};
- if (stream.blobfd < 0) {
- erofs_err("failed to create temporary file for %s", digest);
- return -errno;
- }
api_registry = (!strcmp(oci->params.registry, DOCKER_REGISTRY)) ?
DOCKER_API_REGISTRY : oci->params.registry;
if (asprintf(&req.url, "https://%s/v2/%s/blobs/%s",
- api_registry, oci->params.repository, digest) == -1) {
- ret = -ENOMEM;
- goto out;
- }
+ api_registry, oci->params.repository, digest) == -1)
+ return -ENOMEM;
if (auth_header && strstr(auth_header, "Bearer"))
req.headers = curl_slist_append(req.headers, auth_header);
@@ -822,6 +927,32 @@ static int ocierofs_extract_layer(struct erofs_oci *oci, struct erofs_importer *
ret = -EIO;
goto out;
}
+ ret = 0;
+out:
+ if (req.headers)
+ curl_slist_free_all(req.headers);
+ free(req.url);
+ return ret;
+}
+
+static int ocierofs_extract_layer(struct erofs_oci *oci,
+ const char *digest, const char *auth_header)
+{
+ struct erofs_oci_stream stream = {};
+ int ret;
+
+ stream = (struct erofs_oci_stream) {
+ .digest = digest,
+ .blobfd = erofs_tmpfile(),
+ };
+ if (stream.blobfd < 0) {
+ erofs_err("failed to create temporary file for %s", digest);
+ return -errno;
+ }
+
+ ret = ocierofs_download_blob_to_fd(oci, digest, auth_header, stream.blobfd);
+ if (ret)
+ goto out;
if (lseek(stream.blobfd, 0, SEEK_SET) < 0) {
erofs_err("failed to seek to beginning of temp file: %s",
@@ -830,36 +961,73 @@ static int ocierofs_extract_layer(struct erofs_oci *oci, struct erofs_importer *
goto out;
}
- memset(&stream.tarfile, 0, sizeof(stream.tarfile));
- init_list_head(&stream.tarfile.global.xattrs);
+ return stream.blobfd;
- ret = erofs_iostream_open(&stream.tarfile.ios, stream.blobfd,
- EROFS_IOS_DECODER_GZIP);
- if (ret) {
- erofs_err("failed to initialize tar stream: %s",
+out:
+ if (stream.blobfd >= 0)
+ close(stream.blobfd);
+ return ret;
+}
+
+int ocierofs_prepare_layers(struct erofs_oci *oci, char **auth_header,
+ bool *using_basic, char **manifest_digest,
+ struct ocierofs_layer_info ***layers,
+ int *layer_count, int *start_index)
+{
+ int ret;
+
+ ret = ocierofs_prepare_auth(oci, auth_header, using_basic);
+ if (ret)
+ return ret;
+
+ *manifest_digest = ocierofs_get_manifest_digest(oci, oci->params.registry,
+ oci->params.repository,
+ oci->params.tag,
+ oci->params.platform,
+ *auth_header);
+ if (IS_ERR(*manifest_digest)) {
+ ret = PTR_ERR(*manifest_digest);
+ erofs_err("failed to get manifest digest: %s",
erofs_strerror(ret));
- goto out;
+ goto out_auth;
}
- do {
- ret = tarerofs_parse_tar(importer, &stream.tarfile);
- /* Continue parsing until end of archive */
- } while (!ret);
- erofs_iostream_close(&stream.tarfile.ios);
+ *layers = ocierofs_fetch_layers_info(oci, oci->params.registry,
+ oci->params.repository,
+ *manifest_digest, *auth_header,
+ layer_count);
+ if (IS_ERR(*layers)) {
+ ret = PTR_ERR(*layers);
+ erofs_err("failed to get image layers: %s", erofs_strerror(ret));
+ goto out_manifest;
+ }
- if (ret < 0 && ret != -ENODATA) {
- erofs_err("failed to process tar stream: %s",
- erofs_strerror(ret));
- goto out;
+ if (oci->params.layer_index >= 0) {
+ if (oci->params.layer_index >= *layer_count) {
+ erofs_err("layer index %d exceeds available layers (%d)",
+ oci->params.layer_index, *layer_count);
+ ret = -EINVAL;
+ goto out_layers;
+ }
+ *layer_count = 1;
+ *start_index = oci->params.layer_index;
+ } else {
+ *start_index = 0;
}
- ret = 0;
-out:
- if (stream.blobfd >= 0)
- close(stream.blobfd);
- if (req.headers)
- curl_slist_free_all(req.headers);
- free(req.url);
+ return 0;
+
+out_layers:
+ free(*layers);
+ *layers = NULL;
+out_manifest:
+ free(*manifest_digest);
+ *manifest_digest = NULL;
+out_auth:
+ free(*auth_header);
+ *auth_header = NULL;
+ if (*using_basic)
+ ocierofs_curl_clear_auth(oci->curl);
return ret;
}
@@ -877,102 +1045,48 @@ int ocierofs_build_trees(struct erofs_importer *importer, struct erofs_oci *oci)
{
char *auth_header = NULL;
char *manifest_digest = NULL;
- char **layers = NULL;
+ struct ocierofs_layer_info **layers = NULL;
int layer_count = 0;
int ret, i;
+ bool using_basic = false;
if (!importer || !oci)
return -EINVAL;
- if (oci->params.username && oci->params.password &&
- oci->params.username[0] && oci->params.password[0]) {
- auth_header = ocierofs_get_auth_token(oci,
- oci->params.registry,
- oci->params.repository,
- oci->params.username,
- oci->params.password);
- if (IS_ERR(auth_header)) {
- auth_header = NULL;
- ret = ocierofs_curl_setup_basic_auth(oci->curl,
- oci->params.username,
- oci->params.password);
- if (ret)
- goto out;
- }
- } else {
- auth_header = ocierofs_get_auth_token(oci,
- oci->params.registry,
- oci->params.repository,
- NULL, NULL);
- if (IS_ERR(auth_header))
- auth_header = NULL;
- }
-
- manifest_digest = ocierofs_get_manifest_digest(oci, oci->params.registry,
- oci->params.repository,
- oci->params.tag,
- oci->params.platform,
- auth_header);
- if (IS_ERR(manifest_digest)) {
- ret = PTR_ERR(manifest_digest);
- erofs_err("failed to get manifest digest: %s",
- erofs_strerror(ret));
- goto out_auth;
- }
-
- layers = ocierofs_get_layers_info(oci, oci->params.registry,
- oci->params.repository,
- manifest_digest, auth_header,
- &layer_count);
- if (IS_ERR(layers)) {
- ret = PTR_ERR(layers);
- erofs_err("failed to get image layers: %s", erofs_strerror(ret));
- goto out_manifest;
- }
-
- if (oci->params.layer_index >= 0) {
- if (oci->params.layer_index >= layer_count) {
- erofs_err("layer index %d exceeds available layers (%d)",
- oci->params.layer_index, layer_count);
- ret = -EINVAL;
- goto out_layers;
- }
- layer_count = 1;
- i = oci->params.layer_index;
- } else {
- i = 0;
- }
+ ret = ocierofs_prepare_layers(oci, &auth_header, &using_basic,
+ &manifest_digest, &layers, &layer_count, &i);
+ if (ret)
+ return ret;
while (i < layer_count) {
- char *trimmed = erofs_trim_for_progressinfo(layers[i],
+ char *trimmed = erofs_trim_for_progressinfo(layers[i]->digest,
sizeof("Extracting layer ...") - 1);
erofs_update_progressinfo("Extracting layer %d: %s ...", i,
trimmed);
free(trimmed);
- ret = ocierofs_extract_layer(oci, importer, layers[i],
+ int fd = ocierofs_extract_layer(oci, layers[i]->digest,
auth_header);
- if (ret) {
+ if (fd < 0) {
erofs_err("failed to extract layer %d: %s", i,
+ erofs_strerror(fd));
+ break;
+ }
+ ret = ocierofs_process_tar_stream(importer, fd);
+ close(fd);
+ if (ret) {
+ erofs_err("failed to process tar stream for layer %d: %s", i,
erofs_strerror(ret));
break;
}
i++;
}
-out_layers:
- for (i = 0; i < layer_count; i++)
- free(layers[i]);
- free(layers);
-out_manifest:
+
+ ocierofs_free_layers_info(layers, layer_count);
free(manifest_digest);
-out_auth:
free(auth_header);
-
- if (oci->params.username && oci->params.password &&
- oci->params.username[0] && oci->params.password[0] &&
- !auth_header) {
+ if (using_basic)
ocierofs_curl_clear_auth(oci->curl);
- }
-out:
+
return ret;
}
@@ -1066,3 +1180,177 @@ int erofs_oci_params_set_string(char **field, const char *value)
*field = new_value;
return 0;
}
+
+int ocierofs_parse_options(struct erofs_oci *oci, char *options_str)
+{
+ char *opt, *q, *p;
+ int ret;
+
+ if (!options_str)
+ return 0;
+
+ opt = options_str;
+ while (opt) {
+ q = strchr(opt, ',');
+ if (q)
+ *q = '\0';
+
+ p = strstr(opt, "platform=");
+ if (p) {
+ p += strlen("platform=");
+ ret = erofs_oci_params_set_string(&oci->params.platform, p);
+ if (ret) {
+ erofs_err("failed to set platform");
+ return ret;
+ }
+ } else {
+ p = strstr(opt, "layer=");
+ if (p) {
+ p += strlen("layer=");
+ oci->params.layer_index = atoi(p);
+ if (oci->params.layer_index < 0) {
+ erofs_err("invalid layer index %d",
+ oci->params.layer_index);
+ return -EINVAL;
+ }
+ } else {
+ p = strstr(opt, "username=");
+ if (p) {
+ p += strlen("username=");
+ ret = erofs_oci_params_set_string(&oci->params.username, p);
+ if (ret) {
+ erofs_err("failed to set username");
+ return ret;
+ }
+ } else {
+ p = strstr(opt, "password=");
+ if (p) {
+ p += strlen("password=");
+ ret = erofs_oci_params_set_string(&oci->params.password, p);
+ if (ret) {
+ erofs_err("failed to set password");
+ return ret;
+ }
+ } else {
+ erofs_err("invalid --oci value %s", opt);
+ return -EINVAL;
+ }
+ }
+ }
+ }
+
+ opt = q ? q + 1 : NULL;
+ }
+
+ return 0;
+}
+
+int ocierofs_parse_ref(struct erofs_oci *oci, const char *ref_str)
+{
+ char *slash, *colon, *dot;
+ const char *repo_part;
+ size_t len;
+
+ if (!oci || !ref_str)
+ return -EINVAL;
+
+ slash = strchr(ref_str, '/');
+ if (slash) {
+ dot = strchr(ref_str, '.');
+ if (dot && dot < slash) {
+ char *registry_str;
+
+ len = slash - ref_str;
+ registry_str = strndup(ref_str, len);
+
+ if (!registry_str) {
+ erofs_err("failed to allocate memory for registry");
+ return -ENOMEM;
+ }
+ if (erofs_oci_params_set_string(&oci->params.registry,
+ registry_str)) {
+ free(registry_str);
+ erofs_err("failed to set registry");
+ return -ENOMEM;
+ }
+ free(registry_str);
+ repo_part = slash + 1;
+ } else {
+ repo_part = ref_str;
+ }
+ } else {
+ repo_part = ref_str;
+ }
+
+ colon = strchr(repo_part, ':');
+ if (colon) {
+ char *repo_str;
+
+ len = colon - repo_part;
+ repo_str = strndup(repo_part, len);
+
+ if (!repo_str) {
+ erofs_err("failed to allocate memory for repository");
+ return -ENOMEM;
+ }
+
+ if (!strchr(repo_str, '/') &&
+ (!strcmp(oci->params.registry, DOCKER_API_REGISTRY) ||
+ !strcmp(oci->params.registry, DOCKER_REGISTRY))) {
+ char *full_repo;
+
+ if (asprintf(&full_repo, "library/%s", repo_str) == -1) {
+ free(repo_str);
+ erofs_err("failed to allocate memory for full repository name");
+ return -ENOMEM;
+ }
+ free(repo_str);
+ repo_str = full_repo;
+ }
+
+ if (erofs_oci_params_set_string(&oci->params.repository,
+ repo_str)) {
+ free(repo_str);
+ erofs_err("failed to set repository");
+ return -ENOMEM;
+ }
+ free(repo_str);
+
+ if (erofs_oci_params_set_string(&oci->params.tag,
+ colon + 1)) {
+ erofs_err("failed to set tag");
+ return -ENOMEM;
+ }
+ } else {
+ char *repo_str = strdup(repo_part);
+
+ if (!repo_str) {
+ erofs_err("failed to allocate memory for repository");
+ return -ENOMEM;
+ }
+
+ if (!strchr(repo_str, '/') &&
+ (!strcmp(oci->params.registry, DOCKER_API_REGISTRY) ||
+ !strcmp(oci->params.registry, DOCKER_REGISTRY))) {
+ char *full_repo;
+
+ if (asprintf(&full_repo, "library/%s", repo_str) == -1) {
+ free(repo_str);
+ erofs_err("failed to allocate memory for full repository name");
+ return -ENOMEM;
+ }
+ free(repo_str);
+ repo_str = full_repo;
+ }
+
+ if (erofs_oci_params_set_string(&oci->params.repository,
+ repo_str)) {
+ free(repo_str);
+ erofs_err("failed to set repository");
+ return -ENOMEM;
+ }
+ free(repo_str);
+ }
+
+ return 0;
+}
diff --git a/mkfs/main.c b/mkfs/main.c
index bc895f1..825cff3 100644
--- a/mkfs/main.c
+++ b/mkfs/main.c
@@ -688,202 +688,6 @@ static int mkfs_parse_s3_cfg(char *cfg_str)
}
#endif
-#ifdef OCIEROFS_ENABLED
-
-
-/**
- * mkfs_parse_oci_options - Parse comma-separated OCI options string
- * @options_str: comma-separated options string
- *
- * Parse OCI options string containing comma-separated key=value pairs.
- * Supported options include platform, layer, username, and password.
- *
- * Return: 0 on success, negative errno on failure
- */
-static int mkfs_parse_oci_options(char *options_str)
-{
- char *opt, *q, *p;
- int ret;
-
- if (!options_str)
- return 0;
-
- opt = options_str;
- while (opt) {
- q = strchr(opt, ',');
- if (q)
- *q = '\0';
-
- p = strstr(opt, "platform=");
- if (p) {
- p += strlen("platform=");
- ret = erofs_oci_params_set_string(&ocicfg.params.platform, p);
- if (ret) {
- erofs_err("failed to set platform");
- return ret;
- }
- } else {
- p = strstr(opt, "layer=");
- if (p) {
- p += strlen("layer=");
- ocicfg.params.layer_index = atoi(p);
- if (ocicfg.params.layer_index < 0) {
- erofs_err("invalid layer index %d",
- ocicfg.params.layer_index);
- return -EINVAL;
- }
- } else {
- p = strstr(opt, "username=");
- if (p) {
- p += strlen("username=");
- ret = erofs_oci_params_set_string(&ocicfg.params.username, p);
- if (ret) {
- erofs_err("failed to set username");
- return ret;
- }
- } else {
- p = strstr(opt, "password=");
- if (p) {
- p += strlen("password=");
- ret = erofs_oci_params_set_string(&ocicfg.params.password, p);
- if (ret) {
- erofs_err("failed to set password");
- return ret;
- }
- } else {
- erofs_err("invalid --oci value %s", opt);
- return -EINVAL;
- }
- }
- }
- }
-
- opt = q ? q + 1 : NULL;
- }
-
- return 0;
-}
-
-/**
- * mkfs_parse_oci_ref - Parse OCI image reference string
- * @ref_str: OCI image reference in various formats
- *
- * Parse OCI image reference which can be in formats:
- * - registry.example.com/namespace/repo:tag
- * - namespace/repo:tag (uses default registry)
- * - repo:tag (adds library/ prefix for Docker Hub)
- * - repo (uses default tag "latest")
- *
- * Return: 0 on success, negative errno on failure
- */
-static int mkfs_parse_oci_ref(const char *ref_str)
-{
- char *slash, *colon, *dot;
- const char *repo_part;
- size_t len;
-
- slash = strchr(ref_str, '/');
- if (slash) {
- dot = strchr(ref_str, '.');
- if (dot && dot < slash) {
- char *registry_str;
-
- len = slash - ref_str;
- registry_str = strndup(ref_str, len);
-
- if (!registry_str) {
- erofs_err("failed to allocate memory for registry");
- return -ENOMEM;
- }
- if (erofs_oci_params_set_string(&ocicfg.params.registry,
- registry_str)) {
- free(registry_str);
- erofs_err("failed to set registry");
- return -ENOMEM;
- }
- free(registry_str);
- repo_part = slash + 1;
- } else {
- repo_part = ref_str;
- }
- } else {
- repo_part = ref_str;
- }
-
- colon = strchr(repo_part, ':');
- if (colon) {
- char *repo_str;
-
- len = colon - repo_part;
- repo_str = strndup(repo_part, len);
-
- if (!repo_str) {
- erofs_err("failed to allocate memory for repository");
- return -ENOMEM;
- }
-
- if (!strchr(repo_str, '/') &&
- (!strcmp(ocicfg.params.registry, DOCKER_API_REGISTRY) ||
- !strcmp(ocicfg.params.registry, DOCKER_REGISTRY))) {
- char *full_repo;
-
- if (asprintf(&full_repo, "library/%s", repo_str) == -1) {
- free(repo_str);
- erofs_err("failed to allocate memory for full repository name");
- return -ENOMEM;
- }
- free(repo_str);
- repo_str = full_repo;
- }
-
- if (erofs_oci_params_set_string(&ocicfg.params.repository,
- repo_str)) {
- free(repo_str);
- erofs_err("failed to set repository");
- return -ENOMEM;
- }
- free(repo_str);
-
- if (erofs_oci_params_set_string(&ocicfg.params.tag,
- colon + 1)) {
- erofs_err("failed to set tag");
- return -ENOMEM;
- }
- } else {
- char *repo_str = strdup(repo_part);
-
- if (!repo_str) {
- erofs_err("failed to allocate memory for repository");
- return -ENOMEM;
- }
-
- if (!strchr(repo_str, '/') &&
- (!strcmp(ocicfg.params.registry, DOCKER_API_REGISTRY) ||
- !strcmp(ocicfg.params.registry, DOCKER_REGISTRY))) {
- char *full_repo;
-
- if (asprintf(&full_repo, "library/%s", repo_str) == -1) {
- free(repo_str);
- erofs_err("failed to allocate memory for full repository name");
- return -ENOMEM;
- }
- free(repo_str);
- repo_str = full_repo;
- }
-
- if (erofs_oci_params_set_string(&ocicfg.params.repository,
- repo_str)) {
- free(repo_str);
- erofs_err("failed to set repository");
- return -ENOMEM;
- }
- free(repo_str);
- }
-
- return 0;
-}
-#endif
-
static int mkfs_parse_one_compress_alg(char *alg,
struct erofs_compr_opts *copts)
{
@@ -1955,10 +1759,10 @@ int main(int argc, char **argv)
if (err)
goto exit;
- err = mkfs_parse_oci_options(mkfs_oci_options);
+ err = ocierofs_parse_options(&ocicfg, mkfs_oci_options);
if (err)
goto exit;
- err = mkfs_parse_oci_ref(cfg.c_src_path);
+ err = ocierofs_parse_ref(&ocicfg, cfg.c_src_path);
if (err)
goto exit;
--
2.51.0
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH v1-changed] erofs-utils: refactor OCI code for better modularity
2025-09-01 5:10 [PATCH v1] erofs-utils: refactor OCI code for better modularity ChengyuZhu6
@ 2025-09-01 7:01 ` ChengyuZhu6
2025-09-02 1:52 ` Gao Xiang
2025-09-02 3:17 ` [PATCH v2] " ChengyuZhu6
` (4 subsequent siblings)
5 siblings, 1 reply; 19+ messages in thread
From: ChengyuZhu6 @ 2025-09-01 7:01 UTC (permalink / raw)
To: linux-erofs; +Cc: xiang, hsiangkao, Chengyu Zhu
From: Chengyu Zhu <hudsonzhu@tencent.com>
Refactor OCI code to improve code organization and maintainability:
- Add `struct ocierofs_layer_info` to encapsulate layer metadata
- Extract authentication logic into `ocierofs_prepare_auth()`
- Split layer processing into `ocierofs_prepare_layers()`
- Move OCI parsing functions from `mkfs/main.c` to `lib/remotes/oci.c`
- Add `ocierofs_process_tar_stream()` for separate tar processing
- Improve error handling with `ocierofs_free_layers_info()`
- Refactor `ocierofs_extract_layer()` to return file descriptor
Signed-off-by: Chengyu Zhu <hudsonzhu@tencent.com>
---
lib/liberofs_oci.h | 100 +++++++++
lib/remotes/oci.c | 540 +++++++++++++++++++++++++++++++++------------
mkfs/main.c | 200 +----------------
3 files changed, 506 insertions(+), 334 deletions(-)
diff --git a/lib/liberofs_oci.h b/lib/liberofs_oci.h
index 3a8108b..698fe07 100644
--- a/lib/liberofs_oci.h
+++ b/lib/liberofs_oci.h
@@ -19,6 +19,23 @@ struct erofs_inode;
struct CURL;
struct erofs_importer;
+/**
+ * struct ocierofs_layer_info
+ * @digest: OCI content-addressable digest (e.g. "sha256:...")
+ * @media_type: mediaType string from the manifest
+ * @size: layer size in bytes from the manifest (0 if not available)
+ *
+ * This structure is exposed to callers so they can enumerate image layers,
+ * decide which ones to fetch, and pass the digest back to download APIs.
+ * Fields are heap-allocated NUL-terminated strings owned by the caller
+ * once returned from public APIs; the caller must free them.
+ */
+struct ocierofs_layer_info {
+ char *digest;
+ char *media_type;
+ u64 size;
+};
+
/**
* struct erofs_oci_params - OCI configuration parameters
* @registry: registry hostname (e.g., "registry-1.docker.io")
@@ -88,6 +105,89 @@ int erofs_oci_params_set_string(char **field, const char *value);
*/
int ocierofs_build_trees(struct erofs_importer *importer, struct erofs_oci *oci);
+/*
+ * ocierofs_parse_options - Parse comma-separated OCI options string
+ * @oci: OCI client structure to update
+ * @options_str: comma-separated options string
+ *
+ * Parse OCI options string containing comma-separated key=value pairs.
+ * Supported options include platform, layer, username, and password.
+ *
+ * Return: 0 on success, negative errno on failure
+ */
+int ocierofs_parse_options(struct erofs_oci *oci, char *options_str);
+
+/*
+ * ocierofs_parse_ref - Parse OCI image reference string
+ * @oci: OCI client structure to update
+ * @ref_str: OCI image reference in various formats
+ *
+ * Parse OCI image reference which can be in formats:
+ * - registry.example.com/namespace/repo:tag
+ * - namespace/repo:tag (uses default registry)
+ * - repo:tag (adds library/ prefix for Docker Hub)
+ * - repo (uses default tag "latest")
+ *
+ * Return: 0 on success, negative errno on failure
+ */
+int ocierofs_parse_ref(struct erofs_oci *oci, const char *ref_str);
+
+/*
+ * ocierofs_prepare_layers - Prepare OCI layers for processing
+ * @oci: OCI client structure with configured parameters
+ * @auth_header: Pointer to store authentication header
+ * @using_basic: Pointer to store basic auth flag
+ * @manifest_digest: Pointer to store manifest digest
+ * @layers: Pointer to store layers information
+ * @layer_count: Pointer to store number of layers
+ * @start_index: Pointer to store starting layer index
+ *
+ * Prepare authentication, get manifest digest and layers information
+ * for OCI image processing. This function handles all the preparation
+ * work needed before processing OCI layers.
+ *
+ * Return: 0 on success, negative errno on failure
+ */
+int ocierofs_prepare_layers(struct erofs_oci *oci, char **auth_header,
+ bool *using_basic, char **manifest_digest,
+ struct ocierofs_layer_info ***layers,
+ int *layer_count, int *start_index);
+
+/**
+ * ocierofs_free_layers_info - Free layer information array
+ * @layers: array of layer information structures
+ * @count: number of layers in the array
+ *
+ * Free all layer information structures and the array itself.
+ * This function handles NULL pointers safely.
+ */
+void ocierofs_free_layers_info(struct ocierofs_layer_info **layers, int count);
+
+/**
+ * ocierofs_prepare_auth - Prepare authentication for OCI requests
+ * @oci: OCI client structure
+ * @auth_header_out: pointer to store authentication header
+ * @using_basic_auth: pointer to store basic auth flag
+ *
+ * Prepare authentication header for OCI registry requests.
+ * This function handles both token-based and basic authentication.
+ *
+ * Return: 0 on success, negative errno on failure
+ */
+int ocierofs_prepare_auth(struct erofs_oci *oci, char **auth_header_out,
+ bool *using_basic_auth);
+
+/**
+ * ocierofs_curl_clear_auth - Clear basic authentication from CURL handle
+ * @curl: CURL handle to clear authentication from
+ *
+ * Clear basic authentication credentials from a CURL handle.
+ * This should be called after using basic authentication to clean up.
+ *
+ * Return: 0 on success, negative errno on failure
+ */
+int ocierofs_curl_clear_auth(struct CURL *curl);
+
#ifdef __cplusplus
}
#endif
diff --git a/lib/remotes/oci.c b/lib/remotes/oci.c
index 0fb8c1f..9774d8d 100644
--- a/lib/remotes/oci.c
+++ b/lib/remotes/oci.c
@@ -42,7 +42,6 @@ struct erofs_oci_response {
};
struct erofs_oci_stream {
- struct erofs_tarfile tarfile;
const char *digest;
int blobfd;
};
@@ -111,7 +110,7 @@ static int ocierofs_curl_setup_basic_auth(struct CURL *curl, const char *usernam
return 0;
}
-static int ocierofs_curl_clear_auth(struct CURL *curl)
+int ocierofs_curl_clear_auth(struct CURL *curl)
{
curl_easy_setopt(curl, CURLOPT_USERPWD, NULL);
curl_easy_setopt(curl, CURLOPT_HTTPAUTH, CURLAUTH_NONE);
@@ -181,7 +180,7 @@ static int ocierofs_request_perform(struct erofs_oci *oci,
ret = ocierofs_curl_setup_rq(oci->curl, req->url,
OCIEROFS_HTTP_GET, req->headers,
- ocierofs_write_callback, resp,
+ ocierofs_write_callback, resp,
NULL, NULL);
if (ret)
return ret;
@@ -568,7 +567,7 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci,
const char *api_registry;
int ret = 0, len, i;
- if (!registry || !repository || !tag || !platform)
+ if (!registry || !repository || !tag)
return ERR_PTR(-EINVAL);
api_registry = (!strcmp(registry, DOCKER_REGISTRY)) ? DOCKER_API_REGISTRY : registry;
@@ -581,8 +580,8 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci,
req.headers = curl_slist_append(req.headers,
"Accept: " DOCKER_MEDIATYPE_MANIFEST_LIST ","
- OCI_MEDIATYPE_INDEX "," DOCKER_MEDIATYPE_MANIFEST_V1 ","
- DOCKER_MEDIATYPE_MANIFEST_V2);
+ OCI_MEDIATYPE_INDEX "," OCI_MEDIATYPE_MANIFEST ","
+ DOCKER_MEDIATYPE_MANIFEST_V1 "," DOCKER_MEDIATYPE_MANIFEST_V2);
ret = ocierofs_request_perform(oci, &req, &resp);
if (ret)
@@ -663,7 +662,24 @@ out:
return ret ? ERR_PTR(ret) : digest;
}
-static char **ocierofs_get_layers_info(struct erofs_oci *oci,
+void ocierofs_free_layers_info(struct ocierofs_layer_info **layers, int count)
+{
+ int i;
+
+ if (!layers)
+ return;
+
+ for (i = 0; i < count; i++) {
+ if (layers[i]) {
+ free(layers[i]->digest);
+ free(layers[i]->media_type);
+ free(layers[i]);
+ }
+ }
+ free(layers);
+}
+
+static struct ocierofs_layer_info **ocierofs_fetch_layers_info(struct erofs_oci *oci,
const char *registry,
const char *repository,
const char *digest,
@@ -672,10 +688,10 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci,
{
struct erofs_oci_request req = {};
struct erofs_oci_response resp = {};
- json_object *root, *layers, *layer, *digest_obj;
- char **layers_info = NULL;
+ json_object *root, *layers, *layer, *digest_obj, *media_type_obj, *size_obj;
+ struct ocierofs_layer_info **layers_info = NULL;
const char *api_registry;
- int ret, len, i, j;
+ int ret, len, i;
if (!registry || !repository || !digest || !layer_count)
return ERR_PTR(-EINVAL);
@@ -725,7 +741,7 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci,
goto out_json;
}
- layers_info = calloc(len, sizeof(char *));
+ layers_info = calloc(len, sizeof(*layers_info));
if (!layers_info) {
ret = -ENOMEM;
goto out_json;
@@ -740,11 +756,25 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci,
goto out_free;
}
- layers_info[i] = strdup(json_object_get_string(digest_obj));
+ layers_info[i] = calloc(1, sizeof(**layers_info));
if (!layers_info[i]) {
ret = -ENOMEM;
goto out_free;
}
+ layers_info[i]->digest = strdup(json_object_get_string(digest_obj));
+ if (!layers_info[i]->digest) {
+ ret = -ENOMEM;
+ goto out_free;
+ }
+ if (json_object_object_get_ex(layer, "mediaType", &media_type_obj))
+ layers_info[i]->media_type = strdup(json_object_get_string(media_type_obj));
+ else
+ layers_info[i]->media_type = NULL;
+
+ if (json_object_object_get_ex(layer, "size", &size_obj))
+ layers_info[i]->size = json_object_get_int64(size_obj);
+ else
+ layers_info[i]->size = 0;
}
*layer_count = len;
@@ -756,11 +786,7 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci,
return layers_info;
out_free:
- if (layers_info) {
- for (j = 0; j < i; j++)
- free(layers_info[j]);
- }
- free(layers_info);
+ ocierofs_free_layers_info(layers_info, i);
out_json:
json_object_put(root);
out:
@@ -771,8 +797,93 @@ out:
return ERR_PTR(ret);
}
-static int ocierofs_extract_layer(struct erofs_oci *oci, struct erofs_importer *importer,
- const char *digest, const char *auth_header)
+/**
+ * ocierofs_process_tar_stream - Process tar stream from file descriptor
+ * @importer: EROFS importer structure
+ * @fd: File descriptor containing tar data
+ *
+ * Initialize tar stream, parse all entries, and clean up resources.
+ *
+ * Return: 0 on success, negative errno on failure
+ */
+static int ocierofs_process_tar_stream(struct erofs_importer *importer, int fd)
+{
+ struct erofs_tarfile tarfile = {};
+ int ret;
+
+ memset(&tarfile, 0, sizeof(tarfile));
+ init_list_head(&tarfile.global.xattrs);
+
+ ret = erofs_iostream_open(&tarfile.ios, fd, EROFS_IOS_DECODER_GZIP);
+ if (ret) {
+ erofs_err("failed to initialize tar stream: %s",
+ erofs_strerror(ret));
+ return ret;
+ }
+
+ do {
+ ret = tarerofs_parse_tar(importer, &tarfile);
+ /* Continue parsing until end of archive */
+ } while (!ret);
+ erofs_iostream_close(&tarfile.ios);
+
+ if (ret < 0 && ret != -ENODATA) {
+ erofs_err("failed to process tar stream: %s",
+ erofs_strerror(ret));
+ return ret;
+ }
+
+ return 0;
+}
+
+int ocierofs_prepare_auth(struct erofs_oci *oci, char **auth_header_out,
+ bool *using_basic_auth)
+{
+ char *auth_header = NULL;
+ int ret = 0;
+
+ if (using_basic_auth)
+ *using_basic_auth = false;
+ if (auth_header_out)
+ *auth_header_out = NULL;
+
+ if (oci->params.username && oci->params.password &&
+ oci->params.username[0] && oci->params.password[0]) {
+ auth_header = ocierofs_get_auth_token(oci,
+ oci->params.registry,
+ oci->params.repository,
+ oci->params.username,
+ oci->params.password);
+ if (IS_ERR(auth_header)) {
+ auth_header = NULL;
+ ret = ocierofs_curl_setup_basic_auth(oci->curl,
+ oci->params.username,
+ oci->params.password);
+ if (ret)
+ return ret;
+ if (using_basic_auth)
+ *using_basic_auth = true;
+ }
+ } else {
+ auth_header = ocierofs_get_auth_token(oci,
+ oci->params.registry,
+ oci->params.repository,
+ NULL, NULL);
+ if (IS_ERR(auth_header))
+ auth_header = NULL;
+ }
+
+ if (auth_header_out)
+ *auth_header_out = auth_header;
+ else
+ free(auth_header);
+ return 0;
+}
+
+static int ocierofs_download_blob_to_fd(struct erofs_oci *oci,
+ const char *digest,
+ const char *auth_header,
+ int outfd)
{
struct erofs_oci_request req = {};
struct erofs_oci_stream stream = {};
@@ -782,20 +893,14 @@ static int ocierofs_extract_layer(struct erofs_oci *oci, struct erofs_importer *
stream = (struct erofs_oci_stream) {
.digest = digest,
- .blobfd = erofs_tmpfile(),
+ .blobfd = outfd,
};
- if (stream.blobfd < 0) {
- erofs_err("failed to create temporary file for %s", digest);
- return -errno;
- }
api_registry = (!strcmp(oci->params.registry, DOCKER_REGISTRY)) ?
DOCKER_API_REGISTRY : oci->params.registry;
if (asprintf(&req.url, "https://%s/v2/%s/blobs/%s",
- api_registry, oci->params.repository, digest) == -1) {
- ret = -ENOMEM;
- goto out;
- }
+ api_registry, oci->params.repository, digest) == -1)
+ return -ENOMEM;
if (auth_header && strstr(auth_header, "Bearer"))
req.headers = curl_slist_append(req.headers, auth_header);
@@ -822,6 +927,32 @@ static int ocierofs_extract_layer(struct erofs_oci *oci, struct erofs_importer *
ret = -EIO;
goto out;
}
+ ret = 0;
+out:
+ if (req.headers)
+ curl_slist_free_all(req.headers);
+ free(req.url);
+ return ret;
+}
+
+static int ocierofs_extract_layer(struct erofs_oci *oci,
+ const char *digest, const char *auth_header)
+{
+ struct erofs_oci_stream stream = {};
+ int ret;
+
+ stream = (struct erofs_oci_stream) {
+ .digest = digest,
+ .blobfd = erofs_tmpfile(),
+ };
+ if (stream.blobfd < 0) {
+ erofs_err("failed to create temporary file for %s", digest);
+ return -errno;
+ }
+
+ ret = ocierofs_download_blob_to_fd(oci, digest, auth_header, stream.blobfd);
+ if (ret)
+ goto out;
if (lseek(stream.blobfd, 0, SEEK_SET) < 0) {
erofs_err("failed to seek to beginning of temp file: %s",
@@ -830,162 +961,125 @@ static int ocierofs_extract_layer(struct erofs_oci *oci, struct erofs_importer *
goto out;
}
- memset(&stream.tarfile, 0, sizeof(stream.tarfile));
- init_list_head(&stream.tarfile.global.xattrs);
-
- ret = erofs_iostream_open(&stream.tarfile.ios, stream.blobfd,
- EROFS_IOS_DECODER_GZIP);
- if (ret) {
- erofs_err("failed to initialize tar stream: %s",
- erofs_strerror(ret));
- goto out;
- }
-
- do {
- ret = tarerofs_parse_tar(importer, &stream.tarfile);
- /* Continue parsing until end of archive */
- } while (!ret);
- erofs_iostream_close(&stream.tarfile.ios);
-
- if (ret < 0 && ret != -ENODATA) {
- erofs_err("failed to process tar stream: %s",
- erofs_strerror(ret));
- goto out;
- }
- ret = 0;
+ return stream.blobfd;
out:
if (stream.blobfd >= 0)
close(stream.blobfd);
- if (req.headers)
- curl_slist_free_all(req.headers);
- free(req.url);
return ret;
}
-/**
- * ocierofs_build_trees - Build file trees from OCI container image layers
- * @importer: EROFS importer structure
- * @oci: OCI client structure with configured parameters
- *
- * Extract and build file system trees from all layers of an OCI container
- * image.
- *
- * Return: 0 on success, negative errno on failure
- */
-int ocierofs_build_trees(struct erofs_importer *importer, struct erofs_oci *oci)
+int ocierofs_prepare_layers(struct erofs_oci *oci, char **auth_header,
+ bool *using_basic, char **manifest_digest,
+ struct ocierofs_layer_info ***layers,
+ int *layer_count, int *start_index)
{
- char *auth_header = NULL;
- char *manifest_digest = NULL;
- char **layers = NULL;
- int layer_count = 0;
- int ret, i;
-
- if (!importer || !oci)
- return -EINVAL;
+ int ret;
- if (oci->params.username && oci->params.password &&
- oci->params.username[0] && oci->params.password[0]) {
- auth_header = ocierofs_get_auth_token(oci,
- oci->params.registry,
- oci->params.repository,
- oci->params.username,
- oci->params.password);
- if (IS_ERR(auth_header)) {
- auth_header = NULL;
- ret = ocierofs_curl_setup_basic_auth(oci->curl,
- oci->params.username,
- oci->params.password);
- if (ret)
- goto out;
- }
- } else {
- auth_header = ocierofs_get_auth_token(oci,
- oci->params.registry,
- oci->params.repository,
- NULL, NULL);
- if (IS_ERR(auth_header))
- auth_header = NULL;
- }
+ ret = ocierofs_prepare_auth(oci, auth_header, using_basic);
+ if (ret)
+ return ret;
- manifest_digest = ocierofs_get_manifest_digest(oci, oci->params.registry,
+ *manifest_digest = ocierofs_get_manifest_digest(oci, oci->params.registry,
oci->params.repository,
oci->params.tag,
oci->params.platform,
- auth_header);
- if (IS_ERR(manifest_digest)) {
- ret = PTR_ERR(manifest_digest);
+ *auth_header);
+ if (IS_ERR(*manifest_digest)) {
+ ret = PTR_ERR(*manifest_digest);
erofs_err("failed to get manifest digest: %s",
erofs_strerror(ret));
goto out_auth;
}
- layers = ocierofs_get_layers_info(oci, oci->params.registry,
- oci->params.repository,
- manifest_digest, auth_header,
- &layer_count);
- if (IS_ERR(layers)) {
- ret = PTR_ERR(layers);
+ *layers = ocierofs_fetch_layers_info(oci, oci->params.registry,
+ oci->params.repository,
+ *manifest_digest, *auth_header,
+ layer_count);
+ if (IS_ERR(*layers)) {
+ ret = PTR_ERR(*layers);
erofs_err("failed to get image layers: %s", erofs_strerror(ret));
goto out_manifest;
}
if (oci->params.layer_index >= 0) {
- if (oci->params.layer_index >= layer_count) {
+ if (oci->params.layer_index >= *layer_count) {
erofs_err("layer index %d exceeds available layers (%d)",
- oci->params.layer_index, layer_count);
+ oci->params.layer_index, *layer_count);
ret = -EINVAL;
goto out_layers;
}
- layer_count = 1;
- i = oci->params.layer_index;
+ *layer_count = 1;
+ *start_index = oci->params.layer_index;
} else {
- i = 0;
+ *start_index = 0;
}
+ return 0;
+
+out_layers:
+ free(*layers);
+ *layers = NULL;
+out_manifest:
+ free(*manifest_digest);
+ *manifest_digest = NULL;
+out_auth:
+ free(*auth_header);
+ *auth_header = NULL;
+ if (*using_basic)
+ ocierofs_curl_clear_auth(oci->curl);
+ return ret;
+}
+
+int ocierofs_build_trees(struct erofs_importer *importer, struct erofs_oci *oci)
+{
+ char *auth_header = NULL;
+ char *manifest_digest = NULL;
+ struct ocierofs_layer_info **layers = NULL;
+ int layer_count = 0;
+ int ret, i;
+ bool using_basic = false;
+
+ if (!importer || !oci)
+ return -EINVAL;
+
+ ret = ocierofs_prepare_layers(oci, &auth_header, &using_basic,
+ &manifest_digest, &layers, &layer_count, &i);
+ if (ret)
+ return ret;
+
while (i < layer_count) {
- char *trimmed = erofs_trim_for_progressinfo(layers[i],
+ char *trimmed = erofs_trim_for_progressinfo(layers[i]->digest,
sizeof("Extracting layer ...") - 1);
erofs_update_progressinfo("Extracting layer %d: %s ...", i,
trimmed);
free(trimmed);
- ret = ocierofs_extract_layer(oci, importer, layers[i],
+ int fd = ocierofs_extract_layer(oci, layers[i]->digest,
auth_header);
- if (ret) {
+ if (fd < 0) {
erofs_err("failed to extract layer %d: %s", i,
+ erofs_strerror(fd));
+ break;
+ }
+ ret = ocierofs_process_tar_stream(importer, fd);
+ close(fd);
+ if (ret) {
+ erofs_err("failed to process tar stream for layer %d: %s", i,
erofs_strerror(ret));
break;
}
i++;
}
-out_layers:
- for (i = 0; i < layer_count; i++)
- free(layers[i]);
- free(layers);
-out_manifest:
+
+ ocierofs_free_layers_info(layers, layer_count);
free(manifest_digest);
-out_auth:
free(auth_header);
-
- if (oci->params.username && oci->params.password &&
- oci->params.username[0] && oci->params.password[0] &&
- !auth_header) {
+ if (using_basic)
ocierofs_curl_clear_auth(oci->curl);
- }
-out:
+
return ret;
}
-/**
- * ocierofs_init - Initialize OCI client with default parameters
- * @oci: OCI client structure to initialize
- *
- * Initialize OCI client structure, set up CURL handle, and configure
- * default parameters including platform (linux/amd64), registry
- * (registry-1.docker.io), and tag (latest).
- *
- * Return: 0 on success, negative errno on failure
- */
int ocierofs_init(struct erofs_oci *oci)
{
if (!oci)
@@ -1066,3 +1160,177 @@ int erofs_oci_params_set_string(char **field, const char *value)
*field = new_value;
return 0;
}
+
+int ocierofs_parse_options(struct erofs_oci *oci, char *options_str)
+{
+ char *opt, *q, *p;
+ int ret;
+
+ if (!options_str)
+ return 0;
+
+ opt = options_str;
+ while (opt) {
+ q = strchr(opt, ',');
+ if (q)
+ *q = '\0';
+
+ p = strstr(opt, "platform=");
+ if (p) {
+ p += strlen("platform=");
+ ret = erofs_oci_params_set_string(&oci->params.platform, p);
+ if (ret) {
+ erofs_err("failed to set platform");
+ return ret;
+ }
+ } else {
+ p = strstr(opt, "layer=");
+ if (p) {
+ p += strlen("layer=");
+ oci->params.layer_index = atoi(p);
+ if (oci->params.layer_index < 0) {
+ erofs_err("invalid layer index %d",
+ oci->params.layer_index);
+ return -EINVAL;
+ }
+ } else {
+ p = strstr(opt, "username=");
+ if (p) {
+ p += strlen("username=");
+ ret = erofs_oci_params_set_string(&oci->params.username, p);
+ if (ret) {
+ erofs_err("failed to set username");
+ return ret;
+ }
+ } else {
+ p = strstr(opt, "password=");
+ if (p) {
+ p += strlen("password=");
+ ret = erofs_oci_params_set_string(&oci->params.password, p);
+ if (ret) {
+ erofs_err("failed to set password");
+ return ret;
+ }
+ } else {
+ erofs_err("invalid --oci value %s", opt);
+ return -EINVAL;
+ }
+ }
+ }
+ }
+
+ opt = q ? q + 1 : NULL;
+ }
+
+ return 0;
+}
+
+int ocierofs_parse_ref(struct erofs_oci *oci, const char *ref_str)
+{
+ char *slash, *colon, *dot;
+ const char *repo_part;
+ size_t len;
+
+ if (!oci || !ref_str)
+ return -EINVAL;
+
+ slash = strchr(ref_str, '/');
+ if (slash) {
+ dot = strchr(ref_str, '.');
+ if (dot && dot < slash) {
+ char *registry_str;
+
+ len = slash - ref_str;
+ registry_str = strndup(ref_str, len);
+
+ if (!registry_str) {
+ erofs_err("failed to allocate memory for registry");
+ return -ENOMEM;
+ }
+ if (erofs_oci_params_set_string(&oci->params.registry,
+ registry_str)) {
+ free(registry_str);
+ erofs_err("failed to set registry");
+ return -ENOMEM;
+ }
+ free(registry_str);
+ repo_part = slash + 1;
+ } else {
+ repo_part = ref_str;
+ }
+ } else {
+ repo_part = ref_str;
+ }
+
+ colon = strchr(repo_part, ':');
+ if (colon) {
+ char *repo_str;
+
+ len = colon - repo_part;
+ repo_str = strndup(repo_part, len);
+
+ if (!repo_str) {
+ erofs_err("failed to allocate memory for repository");
+ return -ENOMEM;
+ }
+
+ if (!strchr(repo_str, '/') &&
+ (!strcmp(oci->params.registry, DOCKER_API_REGISTRY) ||
+ !strcmp(oci->params.registry, DOCKER_REGISTRY))) {
+ char *full_repo;
+
+ if (asprintf(&full_repo, "library/%s", repo_str) == -1) {
+ free(repo_str);
+ erofs_err("failed to allocate memory for full repository name");
+ return -ENOMEM;
+ }
+ free(repo_str);
+ repo_str = full_repo;
+ }
+
+ if (erofs_oci_params_set_string(&oci->params.repository,
+ repo_str)) {
+ free(repo_str);
+ erofs_err("failed to set repository");
+ return -ENOMEM;
+ }
+ free(repo_str);
+
+ if (erofs_oci_params_set_string(&oci->params.tag,
+ colon + 1)) {
+ erofs_err("failed to set tag");
+ return -ENOMEM;
+ }
+ } else {
+ char *repo_str = strdup(repo_part);
+
+ if (!repo_str) {
+ erofs_err("failed to allocate memory for repository");
+ return -ENOMEM;
+ }
+
+ if (!strchr(repo_str, '/') &&
+ (!strcmp(oci->params.registry, DOCKER_API_REGISTRY) ||
+ !strcmp(oci->params.registry, DOCKER_REGISTRY))) {
+ char *full_repo;
+
+ if (asprintf(&full_repo, "library/%s", repo_str) == -1) {
+ free(repo_str);
+ erofs_err("failed to allocate memory for full repository name");
+ return -ENOMEM;
+ }
+ free(repo_str);
+ repo_str = full_repo;
+ }
+
+ if (erofs_oci_params_set_string(&oci->params.repository,
+ repo_str)) {
+ free(repo_str);
+ erofs_err("failed to set repository");
+ return -ENOMEM;
+ }
+ free(repo_str);
+ }
+
+ return 0;
+}
diff --git a/mkfs/main.c b/mkfs/main.c
index bc895f1..825cff3 100644
--- a/mkfs/main.c
+++ b/mkfs/main.c
@@ -688,202 +688,6 @@ static int mkfs_parse_s3_cfg(char *cfg_str)
}
#endif
-#ifdef OCIEROFS_ENABLED
-
-
-/**
- * mkfs_parse_oci_options - Parse comma-separated OCI options string
- * @options_str: comma-separated options string
- *
- * Parse OCI options string containing comma-separated key=value pairs.
- * Supported options include platform, layer, username, and password.
- *
- * Return: 0 on success, negative errno on failure
- */
-static int mkfs_parse_oci_options(char *options_str)
-{
- char *opt, *q, *p;
- int ret;
-
- if (!options_str)
- return 0;
-
- opt = options_str;
- while (opt) {
- q = strchr(opt, ',');
- if (q)
- *q = '\0';
-
- p = strstr(opt, "platform=");
- if (p) {
- p += strlen("platform=");
- ret = erofs_oci_params_set_string(&ocicfg.params.platform, p);
- if (ret) {
- erofs_err("failed to set platform");
- return ret;
- }
- } else {
- p = strstr(opt, "layer=");
- if (p) {
- p += strlen("layer=");
- ocicfg.params.layer_index = atoi(p);
- if (ocicfg.params.layer_index < 0) {
- erofs_err("invalid layer index %d",
- ocicfg.params.layer_index);
- return -EINVAL;
- }
- } else {
- p = strstr(opt, "username=");
- if (p) {
- p += strlen("username=");
- ret = erofs_oci_params_set_string(&ocicfg.params.username, p);
- if (ret) {
- erofs_err("failed to set username");
- return ret;
- }
- } else {
- p = strstr(opt, "password=");
- if (p) {
- p += strlen("password=");
- ret = erofs_oci_params_set_string(&ocicfg.params.password, p);
- if (ret) {
- erofs_err("failed to set password");
- return ret;
- }
- } else {
- erofs_err("invalid --oci value %s", opt);
- return -EINVAL;
- }
- }
- }
- }
-
- opt = q ? q + 1 : NULL;
- }
-
- return 0;
-}
-
-/**
- * mkfs_parse_oci_ref - Parse OCI image reference string
- * @ref_str: OCI image reference in various formats
- *
- * Parse OCI image reference which can be in formats:
- * - registry.example.com/namespace/repo:tag
- * - namespace/repo:tag (uses default registry)
- * - repo:tag (adds library/ prefix for Docker Hub)
- * - repo (uses default tag "latest")
- *
- * Return: 0 on success, negative errno on failure
- */
-static int mkfs_parse_oci_ref(const char *ref_str)
-{
- char *slash, *colon, *dot;
- const char *repo_part;
- size_t len;
-
- slash = strchr(ref_str, '/');
- if (slash) {
- dot = strchr(ref_str, '.');
- if (dot && dot < slash) {
- char *registry_str;
-
- len = slash - ref_str;
- registry_str = strndup(ref_str, len);
-
- if (!registry_str) {
- erofs_err("failed to allocate memory for registry");
- return -ENOMEM;
- }
- if (erofs_oci_params_set_string(&ocicfg.params.registry,
- registry_str)) {
- free(registry_str);
- erofs_err("failed to set registry");
- return -ENOMEM;
- }
- free(registry_str);
- repo_part = slash + 1;
- } else {
- repo_part = ref_str;
- }
- } else {
- repo_part = ref_str;
- }
-
- colon = strchr(repo_part, ':');
- if (colon) {
- char *repo_str;
-
- len = colon - repo_part;
- repo_str = strndup(repo_part, len);
-
- if (!repo_str) {
- erofs_err("failed to allocate memory for repository");
- return -ENOMEM;
- }
-
- if (!strchr(repo_str, '/') &&
- (!strcmp(ocicfg.params.registry, DOCKER_API_REGISTRY) ||
- !strcmp(ocicfg.params.registry, DOCKER_REGISTRY))) {
- char *full_repo;
-
- if (asprintf(&full_repo, "library/%s", repo_str) == -1) {
- free(repo_str);
- erofs_err("failed to allocate memory for full repository name");
- return -ENOMEM;
- }
- free(repo_str);
- repo_str = full_repo;
- }
-
- if (erofs_oci_params_set_string(&ocicfg.params.repository,
- repo_str)) {
- free(repo_str);
- erofs_err("failed to set repository");
- return -ENOMEM;
- }
- free(repo_str);
-
- if (erofs_oci_params_set_string(&ocicfg.params.tag,
- colon + 1)) {
- erofs_err("failed to set tag");
- return -ENOMEM;
- }
- } else {
- char *repo_str = strdup(repo_part);
-
- if (!repo_str) {
- erofs_err("failed to allocate memory for repository");
- return -ENOMEM;
- }
-
- if (!strchr(repo_str, '/') &&
- (!strcmp(ocicfg.params.registry, DOCKER_API_REGISTRY) ||
- !strcmp(ocicfg.params.registry, DOCKER_REGISTRY))) {
- char *full_repo;
-
- if (asprintf(&full_repo, "library/%s", repo_str) == -1) {
- free(repo_str);
- erofs_err("failed to allocate memory for full repository name");
- return -ENOMEM;
- }
- free(repo_str);
- repo_str = full_repo;
- }
-
- if (erofs_oci_params_set_string(&ocicfg.params.repository,
- repo_str)) {
- free(repo_str);
- erofs_err("failed to set repository");
- return -ENOMEM;
- }
- free(repo_str);
- }
-
- return 0;
-}
-#endif
-
static int mkfs_parse_one_compress_alg(char *alg,
struct erofs_compr_opts *copts)
{
@@ -1955,10 +1759,10 @@ int main(int argc, char **argv)
if (err)
goto exit;
- err = mkfs_parse_oci_options(mkfs_oci_options);
+ err = ocierofs_parse_options(&ocicfg, mkfs_oci_options);
if (err)
goto exit;
- err = mkfs_parse_oci_ref(cfg.c_src_path);
+ err = ocierofs_parse_ref(&ocicfg, cfg.c_src_path);
if (err)
goto exit;
--
2.51.0
^ permalink raw reply related [flat|nested] 19+ messages in thread
* Re: [PATCH v1-changed] erofs-utils: refactor OCI code for better modularity
2025-09-01 7:01 ` [PATCH v1-changed] " ChengyuZhu6
@ 2025-09-02 1:52 ` Gao Xiang
0 siblings, 0 replies; 19+ messages in thread
From: Gao Xiang @ 2025-09-02 1:52 UTC (permalink / raw)
To: ChengyuZhu6, linux-erofs; +Cc: xiang, Chengyu Zhu
On 2025/9/1 15:01, ChengyuZhu6 wrote:
> From: Chengyu Zhu <hudsonzhu@tencent.com>
>
> Refactor OCI code to improve code organization and maintainability:
>
> - Add `struct ocierofs_layer_info` to encapsulate layer metadata
> - Extract authentication logic into `ocierofs_prepare_auth()`
> - Split layer processing into `ocierofs_prepare_layers()`
> - Move OCI parsing functions from `mkfs/main.c` to `lib/remotes/oci.c`
> - Add `ocierofs_process_tar_stream()` for separate tar processing
> - Improve error handling with `ocierofs_free_layers_info()`
> - Refactor `ocierofs_extract_layer()` to return file descriptor
>
> Signed-off-by: Chengyu Zhu <hudsonzhu@tencent.com>
> ---
> lib/liberofs_oci.h | 100 +++++++++
> lib/remotes/oci.c | 540 +++++++++++++++++++++++++++++++++------------
> mkfs/main.c | 200 +----------------
> 3 files changed, 506 insertions(+), 334 deletions(-)
>
> diff --git a/lib/liberofs_oci.h b/lib/liberofs_oci.h
> index 3a8108b..698fe07 100644
> --- a/lib/liberofs_oci.h
> +++ b/lib/liberofs_oci.h
> @@ -19,6 +19,23 @@ struct erofs_inode;
> struct CURL;
> struct erofs_importer;
>
> +/**
> + * struct ocierofs_layer_info
> + * @digest: OCI content-addressable digest (e.g. "sha256:...")
> + * @media_type: mediaType string from the manifest
> + * @size: layer size in bytes from the manifest (0 if not available)
> + *
> + * This structure is exposed to callers so they can enumerate image layers,
> + * decide which ones to fetch, and pass the digest back to download APIs.
> + * Fields are heap-allocated NUL-terminated strings owned by the caller
> + * once returned from public APIs; the caller must free them.
> + */
> +struct ocierofs_layer_info {
> + char *digest;
> + char *media_type;
> + u64 size;
> +};
> +
> /**
> * struct erofs_oci_params - OCI configuration parameters
> * @registry: registry hostname (e.g., "registry-1.docker.io")
> @@ -88,6 +105,89 @@ int erofs_oci_params_set_string(char **field, const char *value);
> */
> int ocierofs_build_trees(struct erofs_importer *importer, struct erofs_oci *oci);
>
> +/*
> + * ocierofs_parse_options - Parse comma-separated OCI options string
> + * @oci: OCI client structure to update
> + * @options_str: comma-separated options string
> + *
> + * Parse OCI options string containing comma-separated key=value pairs.
> + * Supported options include platform, layer, username, and password.
> + *
> + * Return: 0 on success, negative errno on failure
> + */
> +int ocierofs_parse_options(struct erofs_oci *oci, char *options_str);
Can we leave this functionality in `mkfs/main.c`.
liberofs is not the place to keep option parser.
> +
> +/*
> + * ocierofs_parse_ref - Parse OCI image reference string
> + * @oci: OCI client structure to update
> + * @ref_str: OCI image reference in various formats
> + *
> + * Parse OCI image reference which can be in formats:
> + * - registry.example.com/namespace/repo:tag
> + * - namespace/repo:tag (uses default registry)
> + * - repo:tag (adds library/ prefix for Docker Hub)
Is there some reference for this rule?
> + * - repo (uses default tag "latest")
> + *
> + * Return: 0 on success, negative errno on failure
> + */
> +int ocierofs_parse_ref(struct erofs_oci *oci, const char *ref_str);
> +
> +/*
> + * ocierofs_prepare_layers - Prepare OCI layers for processing
> + * @oci: OCI client structure with configured parameters
> + * @auth_header: Pointer to store authentication header
> + * @using_basic: Pointer to store basic auth flag
> + * @manifest_digest: Pointer to store manifest digest
> + * @layers: Pointer to store layers information
> + * @layer_count: Pointer to store number of layers
> + * @start_index: Pointer to store starting layer index
> + *
> + * Prepare authentication, get manifest digest and layers information
> + * for OCI image processing. This function handles all the preparation
> + * work needed before processing OCI layers.
> + *
> + * Return: 0 on success, negative errno on failure
> + */
> +int ocierofs_prepare_layers(struct erofs_oci *oci, char **auth_header,
> + bool *using_basic, char **manifest_digest,
> + struct ocierofs_layer_info ***layers,
> + int *layer_count, int *start_index);
Could we have a way to wrap these arguments into
a structure too?
> +
> +/**
> + * ocierofs_free_layers_info - Free layer information array
> + * @layers: array of layer information structures
> + * @count: number of layers in the array
> + *
> + * Free all layer information structures and the array itself.
> + * This function handles NULL pointers safely.
> + */
> +void ocierofs_free_layers_info(struct ocierofs_layer_info **layers, int count);
> +
> +/**
> + * ocierofs_prepare_auth - Prepare authentication for OCI requests
> + * @oci: OCI client structure
> + * @auth_header_out: pointer to store authentication header
> + * @using_basic_auth: pointer to store basic auth flag
> + *
> + * Prepare authentication header for OCI registry requests.
> + * This function handles both token-based and basic authentication.
> + *
> + * Return: 0 on success, negative errno on failure
> + */
> +int ocierofs_prepare_auth(struct erofs_oci *oci, char **auth_header_out,
> + bool *using_basic_auth);
> +
> +/**
> + * ocierofs_curl_clear_auth - Clear basic authentication from CURL handle
> + * @curl: CURL handle to clear authentication from
> + *
> + * Clear basic authentication credentials from a CURL handle.
> + * This should be called after using basic authentication to clean up.
> + *
> + * Return: 0 on success, negative errno on failure
> + */
> +int ocierofs_curl_clear_auth(struct CURL *curl);
pass in `erofs_oci` instead?
> +
> #ifdef __cplusplus
> }
> #endif
> diff --git a/lib/remotes/oci.c b/lib/remotes/oci.c
> index 0fb8c1f..9774d8d 100644
> --- a/lib/remotes/oci.c
> +++ b/lib/remotes/oci.c
> @@ -42,7 +42,6 @@ struct erofs_oci_response {
> };
>
> struct erofs_oci_stream {
> - struct erofs_tarfile tarfile;
> const char *digest;
> int blobfd;
> };
> @@ -111,7 +110,7 @@ static int ocierofs_curl_setup_basic_auth(struct CURL *curl, const char *usernam
> return 0;
> }
>
> -static int ocierofs_curl_clear_auth(struct CURL *curl)
> +int ocierofs_curl_clear_auth(struct CURL *curl)
> {
> curl_easy_setopt(curl, CURLOPT_USERPWD, NULL);
> curl_easy_setopt(curl, CURLOPT_HTTPAUTH, CURLAUTH_NONE);
> @@ -181,7 +180,7 @@ static int ocierofs_request_perform(struct erofs_oci *oci,
>
> ret = ocierofs_curl_setup_rq(oci->curl, req->url,
> OCIEROFS_HTTP_GET, req->headers,
> - ocierofs_write_callback, resp,
> + ocierofs_write_callback, resp,
> NULL, NULL);
> if (ret)
> return ret;
> @@ -568,7 +567,7 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci,
> const char *api_registry;
> int ret = 0, len, i;
>
> - if (!registry || !repository || !tag || !platform)
> + if (!registry || !repository || !tag)
> return ERR_PTR(-EINVAL);
>
> api_registry = (!strcmp(registry, DOCKER_REGISTRY)) ? DOCKER_API_REGISTRY : registry;
> @@ -581,8 +580,8 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci,
>
> req.headers = curl_slist_append(req.headers,
> "Accept: " DOCKER_MEDIATYPE_MANIFEST_LIST ","
> - OCI_MEDIATYPE_INDEX "," DOCKER_MEDIATYPE_MANIFEST_V1 ","
> - DOCKER_MEDIATYPE_MANIFEST_V2);
> + OCI_MEDIATYPE_INDEX "," OCI_MEDIATYPE_MANIFEST ","
> + DOCKER_MEDIATYPE_MANIFEST_V1 "," DOCKER_MEDIATYPE_MANIFEST_V2);
>
> ret = ocierofs_request_perform(oci, &req, &resp);
> if (ret)
> @@ -663,7 +662,24 @@ out:
> return ret ? ERR_PTR(ret) : digest;
> }
>
> -static char **ocierofs_get_layers_info(struct erofs_oci *oci,
> +void ocierofs_free_layers_info(struct ocierofs_layer_info **layers, int count)
> +{
> + int i;
> +
> + if (!layers)
> + return;
> +
> + for (i = 0; i < count; i++) {
> + if (layers[i]) {
> + free(layers[i]->digest);
> + free(layers[i]->media_type);
> + free(layers[i]);
> + }
> + }
> + free(layers);
> +}
> +
> +static struct ocierofs_layer_info **ocierofs_fetch_layers_info(struct erofs_oci *oci,
> const char *registry,
> const char *repository,
> const char *digest,
> @@ -672,10 +688,10 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci,
> {
> struct erofs_oci_request req = {};
> struct erofs_oci_response resp = {};
> - json_object *root, *layers, *layer, *digest_obj;
> - char **layers_info = NULL;
> + json_object *root, *layers, *layer, *digest_obj, *media_type_obj, *size_obj;
> + struct ocierofs_layer_info **layers_info = NULL;
> const char *api_registry;
> - int ret, len, i, j;
> + int ret, len, i;
>
> if (!registry || !repository || !digest || !layer_count)
> return ERR_PTR(-EINVAL);
> @@ -725,7 +741,7 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci,
> goto out_json;
> }
>
> - layers_info = calloc(len, sizeof(char *));
> + layers_info = calloc(len, sizeof(*layers_info));
> if (!layers_info) {
> ret = -ENOMEM;
> goto out_json;
> @@ -740,11 +756,25 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci,
> goto out_free;
> }
>
> - layers_info[i] = strdup(json_object_get_string(digest_obj));
> + layers_info[i] = calloc(1, sizeof(**layers_info));
> if (!layers_info[i]) {
> ret = -ENOMEM;
> goto out_free;
> }
> + layers_info[i]->digest = strdup(json_object_get_string(digest_obj));
> + if (!layers_info[i]->digest) {
> + ret = -ENOMEM;
> + goto out_free;
> + }
> + if (json_object_object_get_ex(layer, "mediaType", &media_type_obj))
> + layers_info[i]->media_type = strdup(json_object_get_string(media_type_obj));
> + else
> + layers_info[i]->media_type = NULL;
> +
> + if (json_object_object_get_ex(layer, "size", &size_obj))
> + layers_info[i]->size = json_object_get_int64(size_obj);
> + else
> + layers_info[i]->size = 0;
> }
>
> *layer_count = len;
> @@ -756,11 +786,7 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci,
> return layers_info;
>
> out_free:
> - if (layers_info) {
> - for (j = 0; j < i; j++)
> - free(layers_info[j]);
> - }
> - free(layers_info);
> + ocierofs_free_layers_info(layers_info, i);
> out_json:
> json_object_put(root);
> out:
> @@ -771,8 +797,93 @@ out:
> return ERR_PTR(ret);
> }
>
> -static int ocierofs_extract_layer(struct erofs_oci *oci, struct erofs_importer *importer,
> - const char *digest, const char *auth_header)
> +/**
> + * ocierofs_process_tar_stream - Process tar stream from file descriptor
> + * @importer: EROFS importer structure
> + * @fd: File descriptor containing tar data
> + *
> + * Initialize tar stream, parse all entries, and clean up resources.
> + *
> + * Return: 0 on success, negative errno on failure
> + */
> +static int ocierofs_process_tar_stream(struct erofs_importer *importer, int fd)
> +{
> + struct erofs_tarfile tarfile = {};
> + int ret;
> +
> + memset(&tarfile, 0, sizeof(tarfile));
struct erofs_tarfile tarfile = {};
already indicates
memset(&tarfile, 0, sizeof(tarfile));
Thanks,
Gao Xiang
^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH v2] erofs-utils: refactor OCI code for better modularity
2025-09-01 5:10 [PATCH v1] erofs-utils: refactor OCI code for better modularity ChengyuZhu6
2025-09-01 7:01 ` [PATCH v1-changed] " ChengyuZhu6
@ 2025-09-02 3:17 ` ChengyuZhu6
2025-09-02 3:29 ` [PATCH v2-changed] " ChengyuZhu6
` (3 subsequent siblings)
5 siblings, 0 replies; 19+ messages in thread
From: ChengyuZhu6 @ 2025-09-02 3:17 UTC (permalink / raw)
To: linux-erofs; +Cc: xiang, hsiangkao, Chengyu Zhu
From: Chengyu Zhu <hudsonzhu@tencent.com>
Refactor OCI code to improve code organization and maintainability:
- Add `struct ocierofs_layer_info` to encapsulate layer metadata
- Extract authentication logic into `ocierofs_prepare_auth()`
- Split layer processing into `ocierofs_prepare_layers()`
- Move OCI parsing functions from `mkfs/main.c` to `lib/remotes/oci.c`
- Add `ocierofs_process_tar_stream()` for separate tar processing
- Improve error handling with `ocierofs_free_layers_info()`
- Refactor `ocierofs_extract_layer()` to return file descriptor
Signed-off-by: Chengyu Zhu <hudsonzhu@tencent.com>
---
lib/liberofs_oci.h | 136 ++++++++++-
lib/remotes/oci.c | 558 +++++++++++++++++++++++++++++++++------------
mkfs/main.c | 155 ++-----------
3 files changed, 572 insertions(+), 277 deletions(-)
diff --git a/lib/liberofs_oci.h b/lib/liberofs_oci.h
index 3a8108b..360725b 100644
--- a/lib/liberofs_oci.h
+++ b/lib/liberofs_oci.h
@@ -19,7 +19,24 @@ struct erofs_inode;
struct CURL;
struct erofs_importer;
-/**
+/*
+ * struct ocierofs_layer_info
+ * @digest: OCI content-addressable digest (e.g. "sha256:...")
+ * @media_type: mediaType string from the manifest
+ * @size: layer size in bytes from the manifest (0 if not available)
+ *
+ * This structure is exposed to callers so they can enumerate image layers,
+ * decide which ones to fetch, and pass the digest back to download APIs.
+ * Fields are heap-allocated NUL-terminated strings owned by the caller
+ * once returned from public APIs; the caller must free them.
+ */
+struct ocierofs_layer_info {
+ char *digest;
+ char *media_type;
+ u64 size;
+};
+
+/*
* struct erofs_oci_params - OCI configuration parameters
* @registry: registry hostname (e.g., "registry-1.docker.io")
* @repository: image repository (e.g., "library/ubuntu")
@@ -43,7 +60,28 @@ struct erofs_oci_params {
int layer_index;
};
-/**
+/*
+ * struct ocierofs_image_context - OCI image preparation context
+ * @auth_header: Authentication header for OCI requests
+ * @using_basic: Flag indicating if basic auth is being used
+ * @manifest_digest: Manifest digest string
+ * @layers: Array of layer information structures
+ * @layer_count: Number of layers in the array
+ * @start_index: Starting layer index for processing
+ *
+ * This structure consolidates all the parameters needed for OCI image
+ * preparation, making the API cleaner and more maintainable.
+ */
+struct ocierofs_image_context {
+ char *auth_header;
+ bool using_basic;
+ char *manifest_digest;
+ struct ocierofs_layer_info **layers;
+ int layer_count;
+ int start_index;
+};
+
+/*
* struct erofs_oci - Combined OCI client structure
* @curl: CURL handle for HTTP requests
* @params: OCI configuration parameters
@@ -88,6 +126,100 @@ int erofs_oci_params_set_string(char **field, const char *value);
*/
int ocierofs_build_trees(struct erofs_importer *importer, struct erofs_oci *oci);
+/*
+ * ocierofs_parse_ref - Parse OCI image reference string
+ * @oci: OCI client structure to update
+ * @ref_str: OCI image reference in various formats
+ *
+ * Parse OCI image reference which can be in formats:
+ * - registry.example.com/namespace/repo:tag
+ * - namespace/repo:tag (uses default registry)
+ * - repo:tag (adds library/ prefix for Docker Hub)
+ * - repo (uses default tag "latest")
+ * See: https://github.com/distribution/reference/blob/main/reference.go
+ *
+ * Return: 0 on success, negative errno on failure
+ */
+int ocierofs_parse_ref(struct erofs_oci *oci, const char *ref_str);
+
+/*
+ * ocierofs_prepare_layers - Prepare OCI layers for processing
+ * @oci: OCI client structure with configured parameters
+ * @auth_header: Pointer to store authentication header
+ * @using_basic: Pointer to store basic auth flag
+ * @manifest_digest: Pointer to store manifest digest
+ * @layers: Pointer to store layers information
+ * @layer_count: Pointer to store number of layers
+ * @start_index: Pointer to store starting layer index
+ *
+ * Prepare authentication, get manifest digest and layers information
+ * for OCI image processing. This function handles all the preparation
+ * work needed before processing OCI layers.
+ *
+ * Return: 0 on success, negative errno on failure
+ */
+int ocierofs_prepare_layers(struct erofs_oci *oci, char **auth_header,
+ bool *using_basic, char **manifest_digest,
+ struct ocierofs_layer_info ***layers,
+ int *layer_count, int *start_index);
+
+/*
+ * ocierofs_prepare_image_ctx - Prepare OCI image using context structure
+ * @oci: OCI client structure with configured parameters
+ * @ctx: Image context structure to populate
+ *
+ * Prepare authentication, get manifest digest and layers information
+ * for OCI image processing using a consolidated context structure.
+ * This is the preferred interface for new code.
+ *
+ * Return: 0 on success, negative errno on failure
+ */
+int ocierofs_prepare_image_ctx(struct erofs_oci *oci, struct ocierofs_image_context *ctx);
+
+/*
+ * ocierofs_free_layers_info - Free layer information array
+ * @layers: array of layer information structures
+ * @count: number of layers in the array
+ *
+ * Free all layer information structures and the array itself.
+ * This function handles NULL pointers safely.
+ */
+void ocierofs_free_layers_info(struct ocierofs_layer_info **layers, int count);
+
+/*
+ * ocierofs_free_image_ctx - Free image context and all associated resources
+ * @ctx: image context structure to free
+ *
+ * Free all resources associated with the image context, including
+ * layers info, manifest digest, and auth header.
+ */
+void ocierofs_free_image_ctx(struct ocierofs_image_context *ctx);
+
+/*
+ * ocierofs_prepare_auth - Prepare authentication for OCI requests
+ * @oci: OCI client structure
+ * @auth_header_out: pointer to store authentication header
+ * @using_basic_auth: pointer to store basic auth flag
+ *
+ * Prepare authentication header for OCI registry requests.
+ * This function handles both token-based and basic authentication.
+ *
+ * Return: 0 on success, negative errno on failure
+ */
+int ocierofs_prepare_auth(struct erofs_oci *oci, char **auth_header_out,
+ bool *using_basic_auth);
+
+/*
+ * ocierofs_curl_clear_auth - Clear basic authentication from OCI client
+ * @oci: OCI client structure to clear authentication from
+ *
+ * Clear basic authentication credentials from the OCI client's CURL handle.
+ * This should be called after using basic authentication to clean up.
+ *
+ * Return: 0 on success, negative errno on failure
+ */
+int ocierofs_curl_clear_auth(struct erofs_oci *oci);
+
#ifdef __cplusplus
}
#endif
diff --git a/lib/remotes/oci.c b/lib/remotes/oci.c
index 0fb8c1f..f78ad6b 100644
--- a/lib/remotes/oci.c
+++ b/lib/remotes/oci.c
@@ -42,7 +42,6 @@ struct erofs_oci_response {
};
struct erofs_oci_stream {
- struct erofs_tarfile tarfile;
const char *digest;
int blobfd;
};
@@ -111,10 +110,10 @@ static int ocierofs_curl_setup_basic_auth(struct CURL *curl, const char *usernam
return 0;
}
-static int ocierofs_curl_clear_auth(struct CURL *curl)
+int ocierofs_curl_clear_auth(struct erofs_oci *oci)
{
- curl_easy_setopt(curl, CURLOPT_USERPWD, NULL);
- curl_easy_setopt(curl, CURLOPT_HTTPAUTH, CURLAUTH_NONE);
+ curl_easy_setopt(oci->curl, CURLOPT_USERPWD, NULL);
+ curl_easy_setopt(oci->curl, CURLOPT_HTTPAUTH, CURLAUTH_NONE);
return 0;
}
@@ -181,7 +180,7 @@ static int ocierofs_request_perform(struct erofs_oci *oci,
ret = ocierofs_curl_setup_rq(oci->curl, req->url,
OCIEROFS_HTTP_GET, req->headers,
- ocierofs_write_callback, resp,
+ ocierofs_write_callback, resp,
NULL, NULL);
if (ret)
return ret;
@@ -377,7 +376,7 @@ static char *ocierofs_get_auth_token_with_url(struct erofs_oci *oci,
}
ret = ocierofs_request_perform(oci, &req, &resp);
- ocierofs_curl_clear_auth(oci->curl);
+ ocierofs_curl_clear_auth(oci);
if (ret)
goto out_url;
@@ -568,7 +567,7 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci,
const char *api_registry;
int ret = 0, len, i;
- if (!registry || !repository || !tag || !platform)
+ if (!registry || !repository || !tag)
return ERR_PTR(-EINVAL);
api_registry = (!strcmp(registry, DOCKER_REGISTRY)) ? DOCKER_API_REGISTRY : registry;
@@ -581,8 +580,8 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci,
req.headers = curl_slist_append(req.headers,
"Accept: " DOCKER_MEDIATYPE_MANIFEST_LIST ","
- OCI_MEDIATYPE_INDEX "," DOCKER_MEDIATYPE_MANIFEST_V1 ","
- DOCKER_MEDIATYPE_MANIFEST_V2);
+ OCI_MEDIATYPE_INDEX "," OCI_MEDIATYPE_MANIFEST ","
+ DOCKER_MEDIATYPE_MANIFEST_V1 "," DOCKER_MEDIATYPE_MANIFEST_V2);
ret = ocierofs_request_perform(oci, &req, &resp);
if (ret)
@@ -663,7 +662,24 @@ out:
return ret ? ERR_PTR(ret) : digest;
}
-static char **ocierofs_get_layers_info(struct erofs_oci *oci,
+void ocierofs_free_layers_info(struct ocierofs_layer_info **layers, int count)
+{
+ int i;
+
+ if (!layers)
+ return;
+
+ for (i = 0; i < count; i++) {
+ if (layers[i]) {
+ free(layers[i]->digest);
+ free(layers[i]->media_type);
+ free(layers[i]);
+ }
+ }
+ free(layers);
+}
+
+static struct ocierofs_layer_info **ocierofs_fetch_layers_info(struct erofs_oci *oci,
const char *registry,
const char *repository,
const char *digest,
@@ -672,10 +688,10 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci,
{
struct erofs_oci_request req = {};
struct erofs_oci_response resp = {};
- json_object *root, *layers, *layer, *digest_obj;
- char **layers_info = NULL;
+ json_object *root, *layers, *layer, *digest_obj, *media_type_obj, *size_obj;
+ struct ocierofs_layer_info **layers_info = NULL;
const char *api_registry;
- int ret, len, i, j;
+ int ret, len, i;
if (!registry || !repository || !digest || !layer_count)
return ERR_PTR(-EINVAL);
@@ -725,7 +741,7 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci,
goto out_json;
}
- layers_info = calloc(len, sizeof(char *));
+ layers_info = calloc(len, sizeof(*layers_info));
if (!layers_info) {
ret = -ENOMEM;
goto out_json;
@@ -740,11 +756,25 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci,
goto out_free;
}
- layers_info[i] = strdup(json_object_get_string(digest_obj));
+ layers_info[i] = calloc(1, sizeof(**layers_info));
if (!layers_info[i]) {
ret = -ENOMEM;
goto out_free;
}
+ layers_info[i]->digest = strdup(json_object_get_string(digest_obj));
+ if (!layers_info[i]->digest) {
+ ret = -ENOMEM;
+ goto out_free;
+ }
+ if (json_object_object_get_ex(layer, "mediaType", &media_type_obj))
+ layers_info[i]->media_type = strdup(json_object_get_string(media_type_obj));
+ else
+ layers_info[i]->media_type = NULL;
+
+ if (json_object_object_get_ex(layer, "size", &size_obj))
+ layers_info[i]->size = json_object_get_int64(size_obj);
+ else
+ layers_info[i]->size = 0;
}
*layer_count = len;
@@ -756,11 +786,7 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci,
return layers_info;
out_free:
- if (layers_info) {
- for (j = 0; j < i; j++)
- free(layers_info[j]);
- }
- free(layers_info);
+ ocierofs_free_layers_info(layers_info, i);
out_json:
json_object_put(root);
out:
@@ -771,8 +797,92 @@ out:
return ERR_PTR(ret);
}
-static int ocierofs_extract_layer(struct erofs_oci *oci, struct erofs_importer *importer,
- const char *digest, const char *auth_header)
+/**
+ * ocierofs_process_tar_stream - Process tar stream from file descriptor
+ * @importer: EROFS importer structure
+ * @fd: File descriptor containing tar data
+ *
+ * Initialize tar stream, parse all entries, and clean up resources.
+ *
+ * Return: 0 on success, negative errno on failure
+ */
+static int ocierofs_process_tar_stream(struct erofs_importer *importer, int fd)
+{
+ struct erofs_tarfile tarfile = {};
+ int ret;
+
+ init_list_head(&tarfile.global.xattrs);
+
+ ret = erofs_iostream_open(&tarfile.ios, fd, EROFS_IOS_DECODER_GZIP);
+ if (ret) {
+ erofs_err("failed to initialize tar stream: %s",
+ erofs_strerror(ret));
+ return ret;
+ }
+
+ do {
+ ret = tarerofs_parse_tar(importer, &tarfile);
+ /* Continue parsing until end of archive */
+ } while (!ret);
+ erofs_iostream_close(&tarfile.ios);
+
+ if (ret < 0 && ret != -ENODATA) {
+ erofs_err("failed to process tar stream: %s",
+ erofs_strerror(ret));
+ return ret;
+ }
+
+ return 0;
+}
+
+int ocierofs_prepare_auth(struct erofs_oci *oci, char **auth_header_out,
+ bool *using_basic_auth)
+{
+ char *auth_header = NULL;
+ int ret = 0;
+
+ if (using_basic_auth)
+ *using_basic_auth = false;
+ if (auth_header_out)
+ *auth_header_out = NULL;
+
+ if (oci->params.username && oci->params.password &&
+ oci->params.username[0] && oci->params.password[0]) {
+ auth_header = ocierofs_get_auth_token(oci,
+ oci->params.registry,
+ oci->params.repository,
+ oci->params.username,
+ oci->params.password);
+ if (IS_ERR(auth_header)) {
+ auth_header = NULL;
+ ret = ocierofs_curl_setup_basic_auth(oci->curl,
+ oci->params.username,
+ oci->params.password);
+ if (ret)
+ return ret;
+ if (using_basic_auth)
+ *using_basic_auth = true;
+ }
+ } else {
+ auth_header = ocierofs_get_auth_token(oci,
+ oci->params.registry,
+ oci->params.repository,
+ NULL, NULL);
+ if (IS_ERR(auth_header))
+ auth_header = NULL;
+ }
+
+ if (auth_header_out)
+ *auth_header_out = auth_header;
+ else
+ free(auth_header);
+ return 0;
+}
+
+static int ocierofs_download_blob_to_fd(struct erofs_oci *oci,
+ const char *digest,
+ const char *auth_header,
+ int outfd)
{
struct erofs_oci_request req = {};
struct erofs_oci_stream stream = {};
@@ -782,20 +892,14 @@ static int ocierofs_extract_layer(struct erofs_oci *oci, struct erofs_importer *
stream = (struct erofs_oci_stream) {
.digest = digest,
- .blobfd = erofs_tmpfile(),
+ .blobfd = outfd,
};
- if (stream.blobfd < 0) {
- erofs_err("failed to create temporary file for %s", digest);
- return -errno;
- }
api_registry = (!strcmp(oci->params.registry, DOCKER_REGISTRY)) ?
DOCKER_API_REGISTRY : oci->params.registry;
if (asprintf(&req.url, "https://%s/v2/%s/blobs/%s",
- api_registry, oci->params.repository, digest) == -1) {
- ret = -ENOMEM;
- goto out;
- }
+ api_registry, oci->params.repository, digest) == -1)
+ return -ENOMEM;
if (auth_header && strstr(auth_header, "Bearer"))
req.headers = curl_slist_append(req.headers, auth_header);
@@ -822,6 +926,32 @@ static int ocierofs_extract_layer(struct erofs_oci *oci, struct erofs_importer *
ret = -EIO;
goto out;
}
+ ret = 0;
+out:
+ if (req.headers)
+ curl_slist_free_all(req.headers);
+ free(req.url);
+ return ret;
+}
+
+static int ocierofs_extract_layer(struct erofs_oci *oci,
+ const char *digest, const char *auth_header)
+{
+ struct erofs_oci_stream stream = {};
+ int ret;
+
+ stream = (struct erofs_oci_stream) {
+ .digest = digest,
+ .blobfd = erofs_tmpfile(),
+ };
+ if (stream.blobfd < 0) {
+ erofs_err("failed to create temporary file for %s", digest);
+ return -errno;
+ }
+
+ ret = ocierofs_download_blob_to_fd(oci, digest, auth_header, stream.blobfd);
+ if (ret)
+ goto out;
if (lseek(stream.blobfd, 0, SEEK_SET) < 0) {
erofs_err("failed to seek to beginning of temp file: %s",
@@ -830,162 +960,181 @@ static int ocierofs_extract_layer(struct erofs_oci *oci, struct erofs_importer *
goto out;
}
- memset(&stream.tarfile, 0, sizeof(stream.tarfile));
- init_list_head(&stream.tarfile.global.xattrs);
+ return stream.blobfd;
- ret = erofs_iostream_open(&stream.tarfile.ios, stream.blobfd,
- EROFS_IOS_DECODER_GZIP);
- if (ret) {
- erofs_err("failed to initialize tar stream: %s",
+out:
+ if (stream.blobfd >= 0)
+ close(stream.blobfd);
+ return ret;
+}
+
+int ocierofs_prepare_layers(struct erofs_oci *oci, char **auth_header,
+ bool *using_basic, char **manifest_digest,
+ struct ocierofs_layer_info ***layers,
+ int *layer_count, int *start_index)
+{
+ int ret;
+
+ ret = ocierofs_prepare_auth(oci, auth_header, using_basic);
+ if (ret)
+ return ret;
+
+ *manifest_digest = ocierofs_get_manifest_digest(oci, oci->params.registry,
+ oci->params.repository,
+ oci->params.tag,
+ oci->params.platform,
+ *auth_header);
+ if (IS_ERR(*manifest_digest)) {
+ ret = PTR_ERR(*manifest_digest);
+ erofs_err("failed to get manifest digest: %s",
erofs_strerror(ret));
- goto out;
+ goto out_auth;
}
- do {
- ret = tarerofs_parse_tar(importer, &stream.tarfile);
- /* Continue parsing until end of archive */
- } while (!ret);
- erofs_iostream_close(&stream.tarfile.ios);
+ *layers = ocierofs_fetch_layers_info(oci, oci->params.registry,
+ oci->params.repository,
+ *manifest_digest, *auth_header,
+ layer_count);
+ if (IS_ERR(*layers)) {
+ ret = PTR_ERR(*layers);
+ erofs_err("failed to get image layers: %s", erofs_strerror(ret));
+ goto out_manifest;
+ }
- if (ret < 0 && ret != -ENODATA) {
- erofs_err("failed to process tar stream: %s",
- erofs_strerror(ret));
- goto out;
+ if (oci->params.layer_index >= 0) {
+ if (oci->params.layer_index >= *layer_count) {
+ erofs_err("layer index %d exceeds available layers (%d)",
+ oci->params.layer_index, *layer_count);
+ ret = -EINVAL;
+ goto out_layers;
+ }
+ *layer_count = 1;
+ *start_index = oci->params.layer_index;
+ } else {
+ *start_index = 0;
}
- ret = 0;
-out:
- if (stream.blobfd >= 0)
- close(stream.blobfd);
- if (req.headers)
- curl_slist_free_all(req.headers);
- free(req.url);
+ return 0;
+
+out_layers:
+ free(*layers);
+ *layers = NULL;
+out_manifest:
+ free(*manifest_digest);
+ *manifest_digest = NULL;
+out_auth:
+ free(*auth_header);
+ *auth_header = NULL;
+ if (*using_basic)
+ ocierofs_curl_clear_auth(oci);
return ret;
}
-/**
- * ocierofs_build_trees - Build file trees from OCI container image layers
- * @importer: EROFS importer structure
- * @oci: OCI client structure with configured parameters
- *
- * Extract and build file system trees from all layers of an OCI container
- * image.
- *
- * Return: 0 on success, negative errno on failure
- */
-int ocierofs_build_trees(struct erofs_importer *importer, struct erofs_oci *oci)
+int ocierofs_prepare_image_ctx(struct erofs_oci *oci, struct ocierofs_image_context *ctx)
{
- char *auth_header = NULL;
- char *manifest_digest = NULL;
- char **layers = NULL;
- int layer_count = 0;
- int ret, i;
+ int ret;
- if (!importer || !oci)
+ if (!oci || !ctx)
return -EINVAL;
- if (oci->params.username && oci->params.password &&
- oci->params.username[0] && oci->params.password[0]) {
- auth_header = ocierofs_get_auth_token(oci,
- oci->params.registry,
- oci->params.repository,
- oci->params.username,
- oci->params.password);
- if (IS_ERR(auth_header)) {
- auth_header = NULL;
- ret = ocierofs_curl_setup_basic_auth(oci->curl,
- oci->params.username,
- oci->params.password);
- if (ret)
- goto out;
- }
- } else {
- auth_header = ocierofs_get_auth_token(oci,
- oci->params.registry,
- oci->params.repository,
- NULL, NULL);
- if (IS_ERR(auth_header))
- auth_header = NULL;
- }
+ *ctx = (struct ocierofs_image_context){};
- manifest_digest = ocierofs_get_manifest_digest(oci, oci->params.registry,
- oci->params.repository,
- oci->params.tag,
- oci->params.platform,
- auth_header);
- if (IS_ERR(manifest_digest)) {
- ret = PTR_ERR(manifest_digest);
+ ret = ocierofs_prepare_auth(oci, &ctx->auth_header, &ctx->using_basic);
+ if (ret)
+ return ret;
+
+ ctx->manifest_digest = ocierofs_get_manifest_digest(oci, oci->params.registry,
+ oci->params.repository,
+ oci->params.tag,
+ oci->params.platform,
+ ctx->auth_header);
+ if (IS_ERR(ctx->manifest_digest)) {
+ ret = PTR_ERR(ctx->manifest_digest);
erofs_err("failed to get manifest digest: %s",
erofs_strerror(ret));
goto out_auth;
}
- layers = ocierofs_get_layers_info(oci, oci->params.registry,
- oci->params.repository,
- manifest_digest, auth_header,
- &layer_count);
- if (IS_ERR(layers)) {
- ret = PTR_ERR(layers);
+ ctx->layers = ocierofs_fetch_layers_info(oci, oci->params.registry,
+ oci->params.repository,
+ ctx->manifest_digest, ctx->auth_header,
+ &ctx->layer_count);
+ if (IS_ERR(ctx->layers)) {
+ ret = PTR_ERR(ctx->layers);
erofs_err("failed to get image layers: %s", erofs_strerror(ret));
goto out_manifest;
}
if (oci->params.layer_index >= 0) {
- if (oci->params.layer_index >= layer_count) {
+ if (oci->params.layer_index >= ctx->layer_count) {
erofs_err("layer index %d exceeds available layers (%d)",
- oci->params.layer_index, layer_count);
+ oci->params.layer_index, ctx->layer_count);
ret = -EINVAL;
goto out_layers;
}
- layer_count = 1;
- i = oci->params.layer_index;
+ ctx->layer_count = 1;
+ ctx->start_index = oci->params.layer_index;
} else {
- i = 0;
+ ctx->start_index = 0;
}
- while (i < layer_count) {
- char *trimmed = erofs_trim_for_progressinfo(layers[i],
+ return 0;
+
+out_layers:
+ free(ctx->layers);
+ ctx->layers = NULL;
+out_manifest:
+ free(ctx->manifest_digest);
+ ctx->manifest_digest = NULL;
+out_auth:
+ free(ctx->auth_header);
+ ctx->auth_header = NULL;
+ if (ctx->using_basic)
+ ocierofs_curl_clear_auth(oci);
+ return ret;
+}
+
+int ocierofs_build_trees(struct erofs_importer *importer, struct erofs_oci *oci)
+{
+ struct ocierofs_image_context image_ctx = {};
+ int ret, i;
+
+ if (!importer || !oci)
+ return -EINVAL;
+
+ ret = ocierofs_prepare_image_ctx(oci, &image_ctx);
+ if (ret)
+ return ret;
+
+ i = image_ctx.start_index;
+ while (i < image_ctx.layer_count) {
+ char *trimmed = erofs_trim_for_progressinfo(image_ctx.layers[i]->digest,
sizeof("Extracting layer ...") - 1);
erofs_update_progressinfo("Extracting layer %d: %s ...", i,
trimmed);
free(trimmed);
- ret = ocierofs_extract_layer(oci, importer, layers[i],
- auth_header);
- if (ret) {
+ int fd = ocierofs_extract_layer(oci, image_ctx.layers[i]->digest,
+ image_ctx.auth_header);
+ if (fd < 0) {
erofs_err("failed to extract layer %d: %s", i,
+ erofs_strerror(fd));
+ break;
+ }
+ ret = ocierofs_process_tar_stream(importer, fd);
+ close(fd);
+ if (ret) {
+ erofs_err("failed to process tar stream for layer %d: %s", i,
erofs_strerror(ret));
break;
}
i++;
}
-out_layers:
- for (i = 0; i < layer_count; i++)
- free(layers[i]);
- free(layers);
-out_manifest:
- free(manifest_digest);
-out_auth:
- free(auth_header);
- if (oci->params.username && oci->params.password &&
- oci->params.username[0] && oci->params.password[0] &&
- !auth_header) {
- ocierofs_curl_clear_auth(oci->curl);
- }
-out:
+ ocierofs_free_image_ctx(&image_ctx);
+
return ret;
}
-/**
- * ocierofs_init - Initialize OCI client with default parameters
- * @oci: OCI client structure to initialize
- *
- * Initialize OCI client structure, set up CURL handle, and configure
- * default parameters including platform (linux/amd64), registry
- * (registry-1.docker.io), and tag (latest).
- *
- * Return: 0 on success, negative errno on failure
- */
int ocierofs_init(struct erofs_oci *oci)
{
if (!oci)
@@ -1066,3 +1215,130 @@ int erofs_oci_params_set_string(char **field, const char *value)
*field = new_value;
return 0;
}
+
+int ocierofs_parse_ref(struct erofs_oci *oci, const char *ref_str)
+{
+ char *slash, *colon, *dot;
+ const char *repo_part;
+ size_t len;
+
+ if (!oci || !ref_str)
+ return -EINVAL;
+
+ slash = strchr(ref_str, '/');
+ if (slash) {
+ dot = strchr(ref_str, '.');
+ if (dot && dot < slash) {
+ char *registry_str;
+
+ len = slash - ref_str;
+ registry_str = strndup(ref_str, len);
+
+ if (!registry_str) {
+ erofs_err("failed to allocate memory for registry");
+ return -ENOMEM;
+ }
+ if (erofs_oci_params_set_string(&oci->params.registry,
+ registry_str)) {
+ free(registry_str);
+ erofs_err("failed to set registry");
+ return -ENOMEM;
+ }
+ free(registry_str);
+ repo_part = slash + 1;
+ } else {
+ repo_part = ref_str;
+ }
+ } else {
+ repo_part = ref_str;
+ }
+
+ colon = strchr(repo_part, ':');
+ if (colon) {
+ char *repo_str;
+
+ len = colon - repo_part;
+ repo_str = strndup(repo_part, len);
+
+ if (!repo_str) {
+ erofs_err("failed to allocate memory for repository");
+ return -ENOMEM;
+ }
+
+ if (!strchr(repo_str, '/') &&
+ (!strcmp(oci->params.registry, DOCKER_API_REGISTRY) ||
+ !strcmp(oci->params.registry, DOCKER_REGISTRY))) {
+ char *full_repo;
+
+ if (asprintf(&full_repo, "library/%s", repo_str) == -1) {
+ free(repo_str);
+ erofs_err("failed to allocate memory for full repository name");
+ return -ENOMEM;
+ }
+ free(repo_str);
+ repo_str = full_repo;
+ }
+
+ if (erofs_oci_params_set_string(&oci->params.repository,
+ repo_str)) {
+ free(repo_str);
+ erofs_err("failed to set repository");
+ return -ENOMEM;
+ }
+ free(repo_str);
+
+ if (erofs_oci_params_set_string(&oci->params.tag,
+ colon + 1)) {
+ erofs_err("failed to set tag");
+ return -ENOMEM;
+ }
+ } else {
+ char *repo_str = strdup(repo_part);
+
+ if (!repo_str) {
+ erofs_err("failed to allocate memory for repository");
+ return -ENOMEM;
+ }
+
+ if (!strchr(repo_str, '/') &&
+ (!strcmp(oci->params.registry, DOCKER_API_REGISTRY) ||
+ !strcmp(oci->params.registry, DOCKER_REGISTRY))) {
+ char *full_repo;
+
+ if (asprintf(&full_repo, "library/%s", repo_str) == -1) {
+ free(repo_str);
+ erofs_err("failed to allocate memory for full repository name");
+ return -ENOMEM;
+ }
+ free(repo_str);
+ repo_str = full_repo;
+ }
+
+ if (erofs_oci_params_set_string(&oci->params.repository,
+ repo_str)) {
+ free(repo_str);
+ erofs_err("failed to set repository");
+ return -ENOMEM;
+ }
+ free(repo_str);
+ }
+
+ return 0;
+}
+
+void ocierofs_free_image_ctx(struct ocierofs_image_context *ctx)
+{
+ if (!ctx)
+ return;
+
+ ocierofs_free_layers_info(ctx->layers, ctx->layer_count);
+ free(ctx->manifest_digest);
+ free(ctx->auth_header);
+
+ ctx->layers = NULL;
+ ctx->manifest_digest = NULL;
+ ctx->auth_header = NULL;
+ ctx->layer_count = 0;
+ ctx->start_index = 0;
+ ctx->using_basic = false;
+}
diff --git a/mkfs/main.c b/mkfs/main.c
index bc895f1..24d384d 100644
--- a/mkfs/main.c
+++ b/mkfs/main.c
@@ -689,18 +689,24 @@ static int mkfs_parse_s3_cfg(char *cfg_str)
#endif
#ifdef OCIEROFS_ENABLED
-
-
-/**
- * mkfs_parse_oci_options - Parse comma-separated OCI options string
+/*
+ * mkfs_parse_oci_options - Parse OCI options for mkfs tool
+ * @oci: OCI client structure to update
* @options_str: comma-separated options string
*
* Parse OCI options string containing comma-separated key=value pairs.
- * Supported options include platform, layer, username, and password.
+ * This function is specific to the mkfs tool and can be enhanced
+ * independently of other tools.
+ *
+ * Supported options include:
+ * - platform=<os/arch>: target platform (e.g., "linux/amd64")
+ * - layer=<index>: specific layer to extract (0-based index)
+ * - username=<user>: authentication username
+ * - password=<pass>: authentication password
*
* Return: 0 on success, negative errno on failure
*/
-static int mkfs_parse_oci_options(char *options_str)
+static int mkfs_parse_oci_options(struct erofs_oci *oci, char *options_str)
{
char *opt, *q, *p;
int ret;
@@ -717,7 +723,7 @@ static int mkfs_parse_oci_options(char *options_str)
p = strstr(opt, "platform=");
if (p) {
p += strlen("platform=");
- ret = erofs_oci_params_set_string(&ocicfg.params.platform, p);
+ ret = erofs_oci_params_set_string(&oci->params.platform, p);
if (ret) {
erofs_err("failed to set platform");
return ret;
@@ -726,17 +732,17 @@ static int mkfs_parse_oci_options(char *options_str)
p = strstr(opt, "layer=");
if (p) {
p += strlen("layer=");
- ocicfg.params.layer_index = atoi(p);
- if (ocicfg.params.layer_index < 0) {
+ oci->params.layer_index = atoi(p);
+ if (oci->params.layer_index < 0) {
erofs_err("invalid layer index %d",
- ocicfg.params.layer_index);
+ oci->params.layer_index);
return -EINVAL;
}
} else {
p = strstr(opt, "username=");
if (p) {
p += strlen("username=");
- ret = erofs_oci_params_set_string(&ocicfg.params.username, p);
+ ret = erofs_oci_params_set_string(&oci->params.username, p);
if (ret) {
erofs_err("failed to set username");
return ret;
@@ -745,13 +751,13 @@ static int mkfs_parse_oci_options(char *options_str)
p = strstr(opt, "password=");
if (p) {
p += strlen("password=");
- ret = erofs_oci_params_set_string(&ocicfg.params.password, p);
+ ret = erofs_oci_params_set_string(&oci->params.password, p);
if (ret) {
erofs_err("failed to set password");
return ret;
}
} else {
- erofs_err("invalid --oci value %s", opt);
+ erofs_err("mkfs: invalid --oci value %s", opt);
return -EINVAL;
}
}
@@ -763,125 +769,6 @@ static int mkfs_parse_oci_options(char *options_str)
return 0;
}
-
-/**
- * mkfs_parse_oci_ref - Parse OCI image reference string
- * @ref_str: OCI image reference in various formats
- *
- * Parse OCI image reference which can be in formats:
- * - registry.example.com/namespace/repo:tag
- * - namespace/repo:tag (uses default registry)
- * - repo:tag (adds library/ prefix for Docker Hub)
- * - repo (uses default tag "latest")
- *
- * Return: 0 on success, negative errno on failure
- */
-static int mkfs_parse_oci_ref(const char *ref_str)
-{
- char *slash, *colon, *dot;
- const char *repo_part;
- size_t len;
-
- slash = strchr(ref_str, '/');
- if (slash) {
- dot = strchr(ref_str, '.');
- if (dot && dot < slash) {
- char *registry_str;
-
- len = slash - ref_str;
- registry_str = strndup(ref_str, len);
-
- if (!registry_str) {
- erofs_err("failed to allocate memory for registry");
- return -ENOMEM;
- }
- if (erofs_oci_params_set_string(&ocicfg.params.registry,
- registry_str)) {
- free(registry_str);
- erofs_err("failed to set registry");
- return -ENOMEM;
- }
- free(registry_str);
- repo_part = slash + 1;
- } else {
- repo_part = ref_str;
- }
- } else {
- repo_part = ref_str;
- }
-
- colon = strchr(repo_part, ':');
- if (colon) {
- char *repo_str;
-
- len = colon - repo_part;
- repo_str = strndup(repo_part, len);
-
- if (!repo_str) {
- erofs_err("failed to allocate memory for repository");
- return -ENOMEM;
- }
-
- if (!strchr(repo_str, '/') &&
- (!strcmp(ocicfg.params.registry, DOCKER_API_REGISTRY) ||
- !strcmp(ocicfg.params.registry, DOCKER_REGISTRY))) {
- char *full_repo;
-
- if (asprintf(&full_repo, "library/%s", repo_str) == -1) {
- free(repo_str);
- erofs_err("failed to allocate memory for full repository name");
- return -ENOMEM;
- }
- free(repo_str);
- repo_str = full_repo;
- }
-
- if (erofs_oci_params_set_string(&ocicfg.params.repository,
- repo_str)) {
- free(repo_str);
- erofs_err("failed to set repository");
- return -ENOMEM;
- }
- free(repo_str);
-
- if (erofs_oci_params_set_string(&ocicfg.params.tag,
- colon + 1)) {
- erofs_err("failed to set tag");
- return -ENOMEM;
- }
- } else {
- char *repo_str = strdup(repo_part);
-
- if (!repo_str) {
- erofs_err("failed to allocate memory for repository");
- return -ENOMEM;
- }
-
- if (!strchr(repo_str, '/') &&
- (!strcmp(ocicfg.params.registry, DOCKER_API_REGISTRY) ||
- !strcmp(ocicfg.params.registry, DOCKER_REGISTRY))) {
- char *full_repo;
-
- if (asprintf(&full_repo, "library/%s", repo_str) == -1) {
- free(repo_str);
- erofs_err("failed to allocate memory for full repository name");
- return -ENOMEM;
- }
- free(repo_str);
- repo_str = full_repo;
- }
-
- if (erofs_oci_params_set_string(&ocicfg.params.repository,
- repo_str)) {
- free(repo_str);
- erofs_err("failed to set repository");
- return -ENOMEM;
- }
- free(repo_str);
- }
-
- return 0;
-}
#endif
static int mkfs_parse_one_compress_alg(char *alg,
@@ -1955,10 +1842,10 @@ int main(int argc, char **argv)
if (err)
goto exit;
- err = mkfs_parse_oci_options(mkfs_oci_options);
+ err = mkfs_parse_oci_options(&ocicfg, mkfs_oci_options);
if (err)
goto exit;
- err = mkfs_parse_oci_ref(cfg.c_src_path);
+ err = ocierofs_parse_ref(&ocicfg, cfg.c_src_path);
if (err)
goto exit;
--
2.51.0
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH v2-changed] erofs-utils: refactor OCI code for better modularity
2025-09-01 5:10 [PATCH v1] erofs-utils: refactor OCI code for better modularity ChengyuZhu6
2025-09-01 7:01 ` [PATCH v1-changed] " ChengyuZhu6
2025-09-02 3:17 ` [PATCH v2] " ChengyuZhu6
@ 2025-09-02 3:29 ` ChengyuZhu6
2025-09-03 8:29 ` [PATCH v3 0/2] erofs-utils: refactor OCI and add NBD-backed OCI image mounting ChengyuZhu6
` (2 subsequent siblings)
5 siblings, 0 replies; 19+ messages in thread
From: ChengyuZhu6 @ 2025-09-02 3:29 UTC (permalink / raw)
To: linux-erofs; +Cc: xiang, hsiangkao, Chengyu Zhu
From: Chengyu Zhu <hudsonzhu@tencent.com>
Refactor OCI code to improve code organization and maintainability:
- Add `struct ocierofs_layer_info` to encapsulate layer metadata
- Extract authentication logic into `ocierofs_prepare_auth()`
- Split layer processing into `ocierofs_prepare_layers()`
- Move OCI parsing functions from `mkfs/main.c` to `lib/remotes/oci.c`
- Add `ocierofs_process_tar_stream()` for separate tar processing
- Improve error handling with `ocierofs_free_layers_info()`
- Refactor `ocierofs_extract_layer()` to return file descriptor
Signed-off-by: Chengyu Zhu <hudsonzhu@tencent.com>
---
lib/liberofs_oci.h | 115 ++++++++++-
lib/remotes/oci.c | 501 ++++++++++++++++++++++++++++++++-------------
mkfs/main.c | 155 ++------------
3 files changed, 489 insertions(+), 282 deletions(-)
diff --git a/lib/liberofs_oci.h b/lib/liberofs_oci.h
index 3a8108b..34d531c 100644
--- a/lib/liberofs_oci.h
+++ b/lib/liberofs_oci.h
@@ -19,7 +19,24 @@ struct erofs_inode;
struct CURL;
struct erofs_importer;
-/**
+/*
+ * struct ocierofs_layer_info
+ * @digest: OCI content-addressable digest (e.g. "sha256:...")
+ * @media_type: mediaType string from the manifest
+ * @size: layer size in bytes from the manifest (0 if not available)
+ *
+ * This structure is exposed to callers so they can enumerate image layers,
+ * decide which ones to fetch, and pass the digest back to download APIs.
+ * Fields are heap-allocated NUL-terminated strings owned by the caller
+ * once returned from public APIs; the caller must free them.
+ */
+struct ocierofs_layer_info {
+ char *digest;
+ char *media_type;
+ u64 size;
+};
+
+/*
* struct erofs_oci_params - OCI configuration parameters
* @registry: registry hostname (e.g., "registry-1.docker.io")
* @repository: image repository (e.g., "library/ubuntu")
@@ -43,7 +60,28 @@ struct erofs_oci_params {
int layer_index;
};
-/**
+/*
+ * struct ocierofs_image_context - OCI image preparation context
+ * @auth_header: Authentication header for OCI requests
+ * @using_basic: Flag indicating if basic auth is being used
+ * @manifest_digest: Manifest digest string
+ * @layers: Array of layer information structures
+ * @layer_count: Number of layers in the array
+ * @start_index: Starting layer index for processing
+ *
+ * This structure consolidates all the parameters needed for OCI image
+ * preparation, making the API cleaner and more maintainable.
+ */
+struct ocierofs_image_context {
+ char *auth_header;
+ bool using_basic;
+ char *manifest_digest;
+ struct ocierofs_layer_info **layers;
+ int layer_count;
+ int start_index;
+};
+
+/*
* struct erofs_oci - Combined OCI client structure
* @curl: CURL handle for HTTP requests
* @params: OCI configuration parameters
@@ -88,6 +126,79 @@ int erofs_oci_params_set_string(char **field, const char *value);
*/
int ocierofs_build_trees(struct erofs_importer *importer, struct erofs_oci *oci);
+/*
+ * ocierofs_parse_ref - Parse OCI image reference string
+ * @oci: OCI client structure to update
+ * @ref_str: OCI image reference in various formats
+ *
+ * Parse OCI image reference which can be in formats:
+ * - registry.example.com/namespace/repo:tag
+ * - namespace/repo:tag (uses default registry)
+ * - repo:tag (adds library/ prefix for Docker Hub)
+ * - repo (uses default tag "latest")
+ * See: https://github.com/distribution/reference/blob/main/reference.go
+ *
+ * Return: 0 on success, negative errno on failure
+ */
+int ocierofs_parse_ref(struct erofs_oci *oci, const char *ref_str);
+
+/*
+ * ocierofs_prepare_layers - Prepare OCI image
+ * @oci: OCI client structure with configured parameters
+ * @ctx: Image context structure to populate
+ *
+ * Prepare authentication, get manifest digest and layers information
+ * for OCI image processing using a consolidated context structure.
+ * This is the preferred interface for new code.
+ *
+ * Return: 0 on success, negative errno on failure
+ */
+int ocierofs_prepare_layers(struct erofs_oci *oci, struct ocierofs_image_context *ctx);
+
+/*
+ * ocierofs_free_layers_info - Free layer information array
+ * @layers: array of layer information structures
+ * @count: number of layers in the array
+ *
+ * Free all layer information structures and the array itself.
+ * This function handles NULL pointers safely.
+ */
+void ocierofs_free_layers_info(struct ocierofs_layer_info **layers, int count);
+
+/*
+ * ocierofs_free_image_ctx - Free image context and all associated resources
+ * @ctx: image context structure to free
+ *
+ * Free all resources associated with the image context, including
+ * layers info, manifest digest, and auth header.
+ */
+void ocierofs_free_image_ctx(struct ocierofs_image_context *ctx);
+
+/*
+ * ocierofs_prepare_auth - Prepare authentication for OCI requests
+ * @oci: OCI client structure
+ * @auth_header_out: pointer to store authentication header
+ * @using_basic_auth: pointer to store basic auth flag
+ *
+ * Prepare authentication header for OCI registry requests.
+ * This function handles both token-based and basic authentication.
+ *
+ * Return: 0 on success, negative errno on failure
+ */
+int ocierofs_prepare_auth(struct erofs_oci *oci, char **auth_header_out,
+ bool *using_basic_auth);
+
+/*
+ * ocierofs_curl_clear_auth - Clear basic authentication from OCI client
+ * @oci: OCI client structure to clear authentication from
+ *
+ * Clear basic authentication credentials from the OCI client's CURL handle.
+ * This should be called after using basic authentication to clean up.
+ *
+ * Return: 0 on success, negative errno on failure
+ */
+int ocierofs_curl_clear_auth(struct erofs_oci *oci);
+
#ifdef __cplusplus
}
#endif
diff --git a/lib/remotes/oci.c b/lib/remotes/oci.c
index 0fb8c1f..9fc6bf2 100644
--- a/lib/remotes/oci.c
+++ b/lib/remotes/oci.c
@@ -42,7 +42,6 @@ struct erofs_oci_response {
};
struct erofs_oci_stream {
- struct erofs_tarfile tarfile;
const char *digest;
int blobfd;
};
@@ -111,10 +110,10 @@ static int ocierofs_curl_setup_basic_auth(struct CURL *curl, const char *usernam
return 0;
}
-static int ocierofs_curl_clear_auth(struct CURL *curl)
+int ocierofs_curl_clear_auth(struct erofs_oci *oci)
{
- curl_easy_setopt(curl, CURLOPT_USERPWD, NULL);
- curl_easy_setopt(curl, CURLOPT_HTTPAUTH, CURLAUTH_NONE);
+ curl_easy_setopt(oci->curl, CURLOPT_USERPWD, NULL);
+ curl_easy_setopt(oci->curl, CURLOPT_HTTPAUTH, CURLAUTH_NONE);
return 0;
}
@@ -181,7 +180,7 @@ static int ocierofs_request_perform(struct erofs_oci *oci,
ret = ocierofs_curl_setup_rq(oci->curl, req->url,
OCIEROFS_HTTP_GET, req->headers,
- ocierofs_write_callback, resp,
+ ocierofs_write_callback, resp,
NULL, NULL);
if (ret)
return ret;
@@ -377,7 +376,7 @@ static char *ocierofs_get_auth_token_with_url(struct erofs_oci *oci,
}
ret = ocierofs_request_perform(oci, &req, &resp);
- ocierofs_curl_clear_auth(oci->curl);
+ ocierofs_curl_clear_auth(oci);
if (ret)
goto out_url;
@@ -568,7 +567,7 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci,
const char *api_registry;
int ret = 0, len, i;
- if (!registry || !repository || !tag || !platform)
+ if (!registry || !repository || !tag)
return ERR_PTR(-EINVAL);
api_registry = (!strcmp(registry, DOCKER_REGISTRY)) ? DOCKER_API_REGISTRY : registry;
@@ -581,8 +580,8 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci,
req.headers = curl_slist_append(req.headers,
"Accept: " DOCKER_MEDIATYPE_MANIFEST_LIST ","
- OCI_MEDIATYPE_INDEX "," DOCKER_MEDIATYPE_MANIFEST_V1 ","
- DOCKER_MEDIATYPE_MANIFEST_V2);
+ OCI_MEDIATYPE_INDEX "," OCI_MEDIATYPE_MANIFEST ","
+ DOCKER_MEDIATYPE_MANIFEST_V1 "," DOCKER_MEDIATYPE_MANIFEST_V2);
ret = ocierofs_request_perform(oci, &req, &resp);
if (ret)
@@ -663,7 +662,24 @@ out:
return ret ? ERR_PTR(ret) : digest;
}
-static char **ocierofs_get_layers_info(struct erofs_oci *oci,
+void ocierofs_free_layers_info(struct ocierofs_layer_info **layers, int count)
+{
+ int i;
+
+ if (!layers)
+ return;
+
+ for (i = 0; i < count; i++) {
+ if (layers[i]) {
+ free(layers[i]->digest);
+ free(layers[i]->media_type);
+ free(layers[i]);
+ }
+ }
+ free(layers);
+}
+
+static struct ocierofs_layer_info **ocierofs_fetch_layers_info(struct erofs_oci *oci,
const char *registry,
const char *repository,
const char *digest,
@@ -672,10 +688,10 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci,
{
struct erofs_oci_request req = {};
struct erofs_oci_response resp = {};
- json_object *root, *layers, *layer, *digest_obj;
- char **layers_info = NULL;
+ json_object *root, *layers, *layer, *digest_obj, *media_type_obj, *size_obj;
+ struct ocierofs_layer_info **layers_info = NULL;
const char *api_registry;
- int ret, len, i, j;
+ int ret, len, i;
if (!registry || !repository || !digest || !layer_count)
return ERR_PTR(-EINVAL);
@@ -725,7 +741,7 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci,
goto out_json;
}
- layers_info = calloc(len, sizeof(char *));
+ layers_info = calloc(len, sizeof(*layers_info));
if (!layers_info) {
ret = -ENOMEM;
goto out_json;
@@ -740,11 +756,25 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci,
goto out_free;
}
- layers_info[i] = strdup(json_object_get_string(digest_obj));
+ layers_info[i] = calloc(1, sizeof(**layers_info));
if (!layers_info[i]) {
ret = -ENOMEM;
goto out_free;
}
+ layers_info[i]->digest = strdup(json_object_get_string(digest_obj));
+ if (!layers_info[i]->digest) {
+ ret = -ENOMEM;
+ goto out_free;
+ }
+ if (json_object_object_get_ex(layer, "mediaType", &media_type_obj))
+ layers_info[i]->media_type = strdup(json_object_get_string(media_type_obj));
+ else
+ layers_info[i]->media_type = NULL;
+
+ if (json_object_object_get_ex(layer, "size", &size_obj))
+ layers_info[i]->size = json_object_get_int64(size_obj);
+ else
+ layers_info[i]->size = 0;
}
*layer_count = len;
@@ -756,11 +786,7 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci,
return layers_info;
out_free:
- if (layers_info) {
- for (j = 0; j < i; j++)
- free(layers_info[j]);
- }
- free(layers_info);
+ ocierofs_free_layers_info(layers_info, i);
out_json:
json_object_put(root);
out:
@@ -771,8 +797,92 @@ out:
return ERR_PTR(ret);
}
-static int ocierofs_extract_layer(struct erofs_oci *oci, struct erofs_importer *importer,
- const char *digest, const char *auth_header)
+/**
+ * ocierofs_process_tar_stream - Process tar stream from file descriptor
+ * @importer: EROFS importer structure
+ * @fd: File descriptor containing tar data
+ *
+ * Initialize tar stream, parse all entries, and clean up resources.
+ *
+ * Return: 0 on success, negative errno on failure
+ */
+static int ocierofs_process_tar_stream(struct erofs_importer *importer, int fd)
+{
+ struct erofs_tarfile tarfile = {};
+ int ret;
+
+ init_list_head(&tarfile.global.xattrs);
+
+ ret = erofs_iostream_open(&tarfile.ios, fd, EROFS_IOS_DECODER_GZIP);
+ if (ret) {
+ erofs_err("failed to initialize tar stream: %s",
+ erofs_strerror(ret));
+ return ret;
+ }
+
+ do {
+ ret = tarerofs_parse_tar(importer, &tarfile);
+ /* Continue parsing until end of archive */
+ } while (!ret);
+ erofs_iostream_close(&tarfile.ios);
+
+ if (ret < 0 && ret != -ENODATA) {
+ erofs_err("failed to process tar stream: %s",
+ erofs_strerror(ret));
+ return ret;
+ }
+
+ return 0;
+}
+
+int ocierofs_prepare_auth(struct erofs_oci *oci, char **auth_header_out,
+ bool *using_basic_auth)
+{
+ char *auth_header = NULL;
+ int ret = 0;
+
+ if (using_basic_auth)
+ *using_basic_auth = false;
+ if (auth_header_out)
+ *auth_header_out = NULL;
+
+ if (oci->params.username && oci->params.password &&
+ oci->params.username[0] && oci->params.password[0]) {
+ auth_header = ocierofs_get_auth_token(oci,
+ oci->params.registry,
+ oci->params.repository,
+ oci->params.username,
+ oci->params.password);
+ if (IS_ERR(auth_header)) {
+ auth_header = NULL;
+ ret = ocierofs_curl_setup_basic_auth(oci->curl,
+ oci->params.username,
+ oci->params.password);
+ if (ret)
+ return ret;
+ if (using_basic_auth)
+ *using_basic_auth = true;
+ }
+ } else {
+ auth_header = ocierofs_get_auth_token(oci,
+ oci->params.registry,
+ oci->params.repository,
+ NULL, NULL);
+ if (IS_ERR(auth_header))
+ auth_header = NULL;
+ }
+
+ if (auth_header_out)
+ *auth_header_out = auth_header;
+ else
+ free(auth_header);
+ return 0;
+}
+
+static int ocierofs_download_blob_to_fd(struct erofs_oci *oci,
+ const char *digest,
+ const char *auth_header,
+ int outfd)
{
struct erofs_oci_request req = {};
struct erofs_oci_stream stream = {};
@@ -782,20 +892,14 @@ static int ocierofs_extract_layer(struct erofs_oci *oci, struct erofs_importer *
stream = (struct erofs_oci_stream) {
.digest = digest,
- .blobfd = erofs_tmpfile(),
+ .blobfd = outfd,
};
- if (stream.blobfd < 0) {
- erofs_err("failed to create temporary file for %s", digest);
- return -errno;
- }
api_registry = (!strcmp(oci->params.registry, DOCKER_REGISTRY)) ?
DOCKER_API_REGISTRY : oci->params.registry;
if (asprintf(&req.url, "https://%s/v2/%s/blobs/%s",
- api_registry, oci->params.repository, digest) == -1) {
- ret = -ENOMEM;
- goto out;
- }
+ api_registry, oci->params.repository, digest) == -1)
+ return -ENOMEM;
if (auth_header && strstr(auth_header, "Bearer"))
req.headers = curl_slist_append(req.headers, auth_header);
@@ -822,6 +926,32 @@ static int ocierofs_extract_layer(struct erofs_oci *oci, struct erofs_importer *
ret = -EIO;
goto out;
}
+ ret = 0;
+out:
+ if (req.headers)
+ curl_slist_free_all(req.headers);
+ free(req.url);
+ return ret;
+}
+
+static int ocierofs_extract_layer(struct erofs_oci *oci,
+ const char *digest, const char *auth_header)
+{
+ struct erofs_oci_stream stream = {};
+ int ret;
+
+ stream = (struct erofs_oci_stream) {
+ .digest = digest,
+ .blobfd = erofs_tmpfile(),
+ };
+ if (stream.blobfd < 0) {
+ erofs_err("failed to create temporary file for %s", digest);
+ return -errno;
+ }
+
+ ret = ocierofs_download_blob_to_fd(oci, digest, auth_header, stream.blobfd);
+ if (ret)
+ goto out;
if (lseek(stream.blobfd, 0, SEEK_SET) < 0) {
erofs_err("failed to seek to beginning of temp file: %s",
@@ -830,162 +960,114 @@ static int ocierofs_extract_layer(struct erofs_oci *oci, struct erofs_importer *
goto out;
}
- memset(&stream.tarfile, 0, sizeof(stream.tarfile));
- init_list_head(&stream.tarfile.global.xattrs);
-
- ret = erofs_iostream_open(&stream.tarfile.ios, stream.blobfd,
- EROFS_IOS_DECODER_GZIP);
- if (ret) {
- erofs_err("failed to initialize tar stream: %s",
- erofs_strerror(ret));
- goto out;
- }
-
- do {
- ret = tarerofs_parse_tar(importer, &stream.tarfile);
- /* Continue parsing until end of archive */
- } while (!ret);
- erofs_iostream_close(&stream.tarfile.ios);
-
- if (ret < 0 && ret != -ENODATA) {
- erofs_err("failed to process tar stream: %s",
- erofs_strerror(ret));
- goto out;
- }
- ret = 0;
+ return stream.blobfd;
out:
if (stream.blobfd >= 0)
close(stream.blobfd);
- if (req.headers)
- curl_slist_free_all(req.headers);
- free(req.url);
return ret;
}
-/**
- * ocierofs_build_trees - Build file trees from OCI container image layers
- * @importer: EROFS importer structure
- * @oci: OCI client structure with configured parameters
- *
- * Extract and build file system trees from all layers of an OCI container
- * image.
- *
- * Return: 0 on success, negative errno on failure
- */
-int ocierofs_build_trees(struct erofs_importer *importer, struct erofs_oci *oci)
+int ocierofs_prepare_layers(struct erofs_oci *oci, struct ocierofs_image_context *ctx)
{
- char *auth_header = NULL;
- char *manifest_digest = NULL;
- char **layers = NULL;
- int layer_count = 0;
- int ret, i;
-
- if (!importer || !oci)
- return -EINVAL;
+ int ret;
- if (oci->params.username && oci->params.password &&
- oci->params.username[0] && oci->params.password[0]) {
- auth_header = ocierofs_get_auth_token(oci,
- oci->params.registry,
- oci->params.repository,
- oci->params.username,
- oci->params.password);
- if (IS_ERR(auth_header)) {
- auth_header = NULL;
- ret = ocierofs_curl_setup_basic_auth(oci->curl,
- oci->params.username,
- oci->params.password);
- if (ret)
- goto out;
- }
- } else {
- auth_header = ocierofs_get_auth_token(oci,
- oci->params.registry,
- oci->params.repository,
- NULL, NULL);
- if (IS_ERR(auth_header))
- auth_header = NULL;
- }
+ ret = ocierofs_prepare_auth(oci, &ctx->auth_header, &ctx->using_basic);
+ if (ret)
+ return ret;
- manifest_digest = ocierofs_get_manifest_digest(oci, oci->params.registry,
- oci->params.repository,
- oci->params.tag,
- oci->params.platform,
- auth_header);
- if (IS_ERR(manifest_digest)) {
- ret = PTR_ERR(manifest_digest);
+ ctx->manifest_digest = ocierofs_get_manifest_digest(oci, oci->params.registry,
+ oci->params.repository,
+ oci->params.tag,
+ oci->params.platform,
+ ctx->auth_header);
+ if (IS_ERR(ctx->manifest_digest)) {
+ ret = PTR_ERR(ctx->manifest_digest);
erofs_err("failed to get manifest digest: %s",
erofs_strerror(ret));
goto out_auth;
}
- layers = ocierofs_get_layers_info(oci, oci->params.registry,
- oci->params.repository,
- manifest_digest, auth_header,
- &layer_count);
- if (IS_ERR(layers)) {
- ret = PTR_ERR(layers);
+ ctx->layers = ocierofs_fetch_layers_info(oci, oci->params.registry,
+ oci->params.repository,
+ ctx->manifest_digest, ctx->auth_header,
+ &ctx->layer_count);
+ if (IS_ERR(ctx->layers)) {
+ ret = PTR_ERR(ctx->layers);
erofs_err("failed to get image layers: %s", erofs_strerror(ret));
goto out_manifest;
}
if (oci->params.layer_index >= 0) {
- if (oci->params.layer_index >= layer_count) {
+ if (oci->params.layer_index >= ctx->layer_count) {
erofs_err("layer index %d exceeds available layers (%d)",
- oci->params.layer_index, layer_count);
+ oci->params.layer_index, ctx->layer_count);
ret = -EINVAL;
goto out_layers;
}
- layer_count = 1;
- i = oci->params.layer_index;
+ ctx->layer_count = 1;
+ ctx->start_index = oci->params.layer_index;
} else {
- i = 0;
+ ctx->start_index = 0;
}
- while (i < layer_count) {
- char *trimmed = erofs_trim_for_progressinfo(layers[i],
+ return 0;
+
+out_layers:
+ free(ctx->layers);
+ ctx->layers = NULL;
+out_manifest:
+ free(ctx->manifest_digest);
+ ctx->manifest_digest = NULL;
+out_auth:
+ free(ctx->auth_header);
+ ctx->auth_header = NULL;
+ if (ctx->using_basic)
+ ocierofs_curl_clear_auth(oci);
+ return ret;
+}
+
+int ocierofs_build_trees(struct erofs_importer *importer, struct erofs_oci *oci)
+{
+ struct ocierofs_image_context image_ctx = {};
+ int ret, i;
+
+ if (!importer || !oci)
+ return -EINVAL;
+
+ ret = ocierofs_prepare_layers(oci, &image_ctx);
+ if (ret)
+ return ret;
+
+ i = image_ctx.start_index;
+ while (i < image_ctx.layer_count) {
+ char *trimmed = erofs_trim_for_progressinfo(image_ctx.layers[i]->digest,
sizeof("Extracting layer ...") - 1);
erofs_update_progressinfo("Extracting layer %d: %s ...", i,
trimmed);
free(trimmed);
- ret = ocierofs_extract_layer(oci, importer, layers[i],
- auth_header);
- if (ret) {
+ int fd = ocierofs_extract_layer(oci, image_ctx.layers[i]->digest,
+ image_ctx.auth_header);
+ if (fd < 0) {
erofs_err("failed to extract layer %d: %s", i,
+ erofs_strerror(fd));
+ break;
+ }
+ ret = ocierofs_process_tar_stream(importer, fd);
+ close(fd);
+ if (ret) {
+ erofs_err("failed to process tar stream for layer %d: %s", i,
erofs_strerror(ret));
break;
}
i++;
}
-out_layers:
- for (i = 0; i < layer_count; i++)
- free(layers[i]);
- free(layers);
-out_manifest:
- free(manifest_digest);
-out_auth:
- free(auth_header);
- if (oci->params.username && oci->params.password &&
- oci->params.username[0] && oci->params.password[0] &&
- !auth_header) {
- ocierofs_curl_clear_auth(oci->curl);
- }
-out:
+ ocierofs_free_image_ctx(&image_ctx);
+
return ret;
}
-/**
- * ocierofs_init - Initialize OCI client with default parameters
- * @oci: OCI client structure to initialize
- *
- * Initialize OCI client structure, set up CURL handle, and configure
- * default parameters including platform (linux/amd64), registry
- * (registry-1.docker.io), and tag (latest).
- *
- * Return: 0 on success, negative errno on failure
- */
int ocierofs_init(struct erofs_oci *oci)
{
if (!oci)
@@ -1066,3 +1148,130 @@ int erofs_oci_params_set_string(char **field, const char *value)
*field = new_value;
return 0;
}
+
+int ocierofs_parse_ref(struct erofs_oci *oci, const char *ref_str)
+{
+ char *slash, *colon, *dot;
+ const char *repo_part;
+ size_t len;
+
+ if (!oci || !ref_str)
+ return -EINVAL;
+
+ slash = strchr(ref_str, '/');
+ if (slash) {
+ dot = strchr(ref_str, '.');
+ if (dot && dot < slash) {
+ char *registry_str;
+
+ len = slash - ref_str;
+ registry_str = strndup(ref_str, len);
+
+ if (!registry_str) {
+ erofs_err("failed to allocate memory for registry");
+ return -ENOMEM;
+ }
+ if (erofs_oci_params_set_string(&oci->params.registry,
+ registry_str)) {
+ free(registry_str);
+ erofs_err("failed to set registry");
+ return -ENOMEM;
+ }
+ free(registry_str);
+ repo_part = slash + 1;
+ } else {
+ repo_part = ref_str;
+ }
+ } else {
+ repo_part = ref_str;
+ }
+
+ colon = strchr(repo_part, ':');
+ if (colon) {
+ char *repo_str;
+
+ len = colon - repo_part;
+ repo_str = strndup(repo_part, len);
+
+ if (!repo_str) {
+ erofs_err("failed to allocate memory for repository");
+ return -ENOMEM;
+ }
+
+ if (!strchr(repo_str, '/') &&
+ (!strcmp(oci->params.registry, DOCKER_API_REGISTRY) ||
+ !strcmp(oci->params.registry, DOCKER_REGISTRY))) {
+ char *full_repo;
+
+ if (asprintf(&full_repo, "library/%s", repo_str) == -1) {
+ free(repo_str);
+ erofs_err("failed to allocate memory for full repository name");
+ return -ENOMEM;
+ }
+ free(repo_str);
+ repo_str = full_repo;
+ }
+
+ if (erofs_oci_params_set_string(&oci->params.repository,
+ repo_str)) {
+ free(repo_str);
+ erofs_err("failed to set repository");
+ return -ENOMEM;
+ }
+ free(repo_str);
+
+ if (erofs_oci_params_set_string(&oci->params.tag,
+ colon + 1)) {
+ erofs_err("failed to set tag");
+ return -ENOMEM;
+ }
+ } else {
+ char *repo_str = strdup(repo_part);
+
+ if (!repo_str) {
+ erofs_err("failed to allocate memory for repository");
+ return -ENOMEM;
+ }
+
+ if (!strchr(repo_str, '/') &&
+ (!strcmp(oci->params.registry, DOCKER_API_REGISTRY) ||
+ !strcmp(oci->params.registry, DOCKER_REGISTRY))) {
+ char *full_repo;
+
+ if (asprintf(&full_repo, "library/%s", repo_str) == -1) {
+ free(repo_str);
+ erofs_err("failed to allocate memory for full repository name");
+ return -ENOMEM;
+ }
+ free(repo_str);
+ repo_str = full_repo;
+ }
+
+ if (erofs_oci_params_set_string(&oci->params.repository,
+ repo_str)) {
+ free(repo_str);
+ erofs_err("failed to set repository");
+ return -ENOMEM;
+ }
+ free(repo_str);
+ }
+
+ return 0;
+}
+
+void ocierofs_free_image_ctx(struct ocierofs_image_context *ctx)
+{
+ if (!ctx)
+ return;
+
+ ocierofs_free_layers_info(ctx->layers, ctx->layer_count);
+ free(ctx->manifest_digest);
+ free(ctx->auth_header);
+
+ ctx->layers = NULL;
+ ctx->manifest_digest = NULL;
+ ctx->auth_header = NULL;
+ ctx->layer_count = 0;
+ ctx->start_index = 0;
+ ctx->using_basic = false;
+}
diff --git a/mkfs/main.c b/mkfs/main.c
index bc895f1..24d384d 100644
--- a/mkfs/main.c
+++ b/mkfs/main.c
@@ -689,18 +689,24 @@ static int mkfs_parse_s3_cfg(char *cfg_str)
#endif
#ifdef OCIEROFS_ENABLED
-
-
-/**
- * mkfs_parse_oci_options - Parse comma-separated OCI options string
+/*
+ * mkfs_parse_oci_options - Parse OCI options for mkfs tool
+ * @oci: OCI client structure to update
* @options_str: comma-separated options string
*
* Parse OCI options string containing comma-separated key=value pairs.
- * Supported options include platform, layer, username, and password.
+ * This function is specific to the mkfs tool and can be enhanced
+ * independently of other tools.
+ *
+ * Supported options include:
+ * - platform=<os/arch>: target platform (e.g., "linux/amd64")
+ * - layer=<index>: specific layer to extract (0-based index)
+ * - username=<user>: authentication username
+ * - password=<pass>: authentication password
*
* Return: 0 on success, negative errno on failure
*/
-static int mkfs_parse_oci_options(char *options_str)
+static int mkfs_parse_oci_options(struct erofs_oci *oci, char *options_str)
{
char *opt, *q, *p;
int ret;
@@ -717,7 +723,7 @@ static int mkfs_parse_oci_options(char *options_str)
p = strstr(opt, "platform=");
if (p) {
p += strlen("platform=");
- ret = erofs_oci_params_set_string(&ocicfg.params.platform, p);
+ ret = erofs_oci_params_set_string(&oci->params.platform, p);
if (ret) {
erofs_err("failed to set platform");
return ret;
@@ -726,17 +732,17 @@ static int mkfs_parse_oci_options(char *options_str)
p = strstr(opt, "layer=");
if (p) {
p += strlen("layer=");
- ocicfg.params.layer_index = atoi(p);
- if (ocicfg.params.layer_index < 0) {
+ oci->params.layer_index = atoi(p);
+ if (oci->params.layer_index < 0) {
erofs_err("invalid layer index %d",
- ocicfg.params.layer_index);
+ oci->params.layer_index);
return -EINVAL;
}
} else {
p = strstr(opt, "username=");
if (p) {
p += strlen("username=");
- ret = erofs_oci_params_set_string(&ocicfg.params.username, p);
+ ret = erofs_oci_params_set_string(&oci->params.username, p);
if (ret) {
erofs_err("failed to set username");
return ret;
@@ -745,13 +751,13 @@ static int mkfs_parse_oci_options(char *options_str)
p = strstr(opt, "password=");
if (p) {
p += strlen("password=");
- ret = erofs_oci_params_set_string(&ocicfg.params.password, p);
+ ret = erofs_oci_params_set_string(&oci->params.password, p);
if (ret) {
erofs_err("failed to set password");
return ret;
}
} else {
- erofs_err("invalid --oci value %s", opt);
+ erofs_err("mkfs: invalid --oci value %s", opt);
return -EINVAL;
}
}
@@ -763,125 +769,6 @@ static int mkfs_parse_oci_options(char *options_str)
return 0;
}
-
-/**
- * mkfs_parse_oci_ref - Parse OCI image reference string
- * @ref_str: OCI image reference in various formats
- *
- * Parse OCI image reference which can be in formats:
- * - registry.example.com/namespace/repo:tag
- * - namespace/repo:tag (uses default registry)
- * - repo:tag (adds library/ prefix for Docker Hub)
- * - repo (uses default tag "latest")
- *
- * Return: 0 on success, negative errno on failure
- */
-static int mkfs_parse_oci_ref(const char *ref_str)
-{
- char *slash, *colon, *dot;
- const char *repo_part;
- size_t len;
-
- slash = strchr(ref_str, '/');
- if (slash) {
- dot = strchr(ref_str, '.');
- if (dot && dot < slash) {
- char *registry_str;
-
- len = slash - ref_str;
- registry_str = strndup(ref_str, len);
-
- if (!registry_str) {
- erofs_err("failed to allocate memory for registry");
- return -ENOMEM;
- }
- if (erofs_oci_params_set_string(&ocicfg.params.registry,
- registry_str)) {
- free(registry_str);
- erofs_err("failed to set registry");
- return -ENOMEM;
- }
- free(registry_str);
- repo_part = slash + 1;
- } else {
- repo_part = ref_str;
- }
- } else {
- repo_part = ref_str;
- }
-
- colon = strchr(repo_part, ':');
- if (colon) {
- char *repo_str;
-
- len = colon - repo_part;
- repo_str = strndup(repo_part, len);
-
- if (!repo_str) {
- erofs_err("failed to allocate memory for repository");
- return -ENOMEM;
- }
-
- if (!strchr(repo_str, '/') &&
- (!strcmp(ocicfg.params.registry, DOCKER_API_REGISTRY) ||
- !strcmp(ocicfg.params.registry, DOCKER_REGISTRY))) {
- char *full_repo;
-
- if (asprintf(&full_repo, "library/%s", repo_str) == -1) {
- free(repo_str);
- erofs_err("failed to allocate memory for full repository name");
- return -ENOMEM;
- }
- free(repo_str);
- repo_str = full_repo;
- }
-
- if (erofs_oci_params_set_string(&ocicfg.params.repository,
- repo_str)) {
- free(repo_str);
- erofs_err("failed to set repository");
- return -ENOMEM;
- }
- free(repo_str);
-
- if (erofs_oci_params_set_string(&ocicfg.params.tag,
- colon + 1)) {
- erofs_err("failed to set tag");
- return -ENOMEM;
- }
- } else {
- char *repo_str = strdup(repo_part);
-
- if (!repo_str) {
- erofs_err("failed to allocate memory for repository");
- return -ENOMEM;
- }
-
- if (!strchr(repo_str, '/') &&
- (!strcmp(ocicfg.params.registry, DOCKER_API_REGISTRY) ||
- !strcmp(ocicfg.params.registry, DOCKER_REGISTRY))) {
- char *full_repo;
-
- if (asprintf(&full_repo, "library/%s", repo_str) == -1) {
- free(repo_str);
- erofs_err("failed to allocate memory for full repository name");
- return -ENOMEM;
- }
- free(repo_str);
- repo_str = full_repo;
- }
-
- if (erofs_oci_params_set_string(&ocicfg.params.repository,
- repo_str)) {
- free(repo_str);
- erofs_err("failed to set repository");
- return -ENOMEM;
- }
- free(repo_str);
- }
-
- return 0;
-}
#endif
static int mkfs_parse_one_compress_alg(char *alg,
@@ -1955,10 +1842,10 @@ int main(int argc, char **argv)
if (err)
goto exit;
- err = mkfs_parse_oci_options(mkfs_oci_options);
+ err = mkfs_parse_oci_options(&ocicfg, mkfs_oci_options);
if (err)
goto exit;
- err = mkfs_parse_oci_ref(cfg.c_src_path);
+ err = ocierofs_parse_ref(&ocicfg, cfg.c_src_path);
if (err)
goto exit;
--
2.51.0
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH v3 0/2] erofs-utils: refactor OCI and add NBD-backed OCI image mounting
2025-09-01 5:10 [PATCH v1] erofs-utils: refactor OCI code for better modularity ChengyuZhu6
` (2 preceding siblings ...)
2025-09-02 3:29 ` [PATCH v2-changed] " ChengyuZhu6
@ 2025-09-03 8:29 ` ChengyuZhu6
2025-09-03 8:29 ` [PATCH v3 1/2] erofs-utils: refactor OCI code for better modularity ChengyuZhu6
2025-09-03 8:29 ` [PATCH v3 2/2] erofs-utils: add NBD-backed OCI image mounting ChengyuZhu6
2025-09-04 5:33 ` [PATCH v4 0/3] erofs-utils: refactor OCI and " ChengyuZhu6
2025-09-04 6:36 ` [PATCH v5 0/3] erofs-utils: refactor OCI and " ChengyuZhu6
5 siblings, 2 replies; 19+ messages in thread
From: ChengyuZhu6 @ 2025-09-03 8:29 UTC (permalink / raw)
To: linux-erofs; +Cc: xiang, hsiangkao, Chengyu Zhu
From: Chengyu Zhu <hudsonzhu@tencent.com>
This series refactors the OCI handling in erofs-utils and adds NBD-backed
mounting of OCI images. It enables mounting EROFS container images directly
from registries without pre-downloading
Chengyu Zhu (2):
erofs-utils: refactor OCI code for better modularity
erofs-utils: add NBD-backed OCI image mounting
lib/liberofs_oci.h | 97 ++--
lib/remotes/oci.c | 1087 +++++++++++++++++++++++++++++---------------
mkfs/main.c | 192 ++------
mount/Makefile.am | 3 +-
mount/main.c | 236 ++++++++++
5 files changed, 1036 insertions(+), 579 deletions(-)
--
2.51.0
^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH v3 1/2] erofs-utils: refactor OCI code for better modularity
2025-09-03 8:29 ` [PATCH v3 0/2] erofs-utils: refactor OCI and add NBD-backed OCI image mounting ChengyuZhu6
@ 2025-09-03 8:29 ` ChengyuZhu6
2025-09-03 8:47 ` Gao Xiang
2025-09-03 8:29 ` [PATCH v3 2/2] erofs-utils: add NBD-backed OCI image mounting ChengyuZhu6
1 sibling, 1 reply; 19+ messages in thread
From: ChengyuZhu6 @ 2025-09-03 8:29 UTC (permalink / raw)
To: linux-erofs; +Cc: xiang, hsiangkao, Chengyu Zhu
From: Chengyu Zhu <hudsonzhu@tencent.com>
Refactor OCI code to improve code organization and maintainability:
Key changes:
- Add `ocierofs_get_api_registry()` and unify API endpoint selection.
- Implement Bearer token discovery with Basic fallback; cache auth header.
- Parse layer metadata (digest, mediaType, size) and add a proper free helper.
- Split blob download from tar processing; process tar via a temp fd.
- Rework init/teardown into `ocierofs_init()` and `ocierofs_ctx_cleanup()`.
- Update mkfs to use `struct ocierofs_config` and new `--oci` parsing.
Signed-off-by: Chengyu Zhu <hudsonzhu@tencent.com>
---
lib/liberofs_oci.h | 84 ++---
lib/remotes/oci.c | 885 ++++++++++++++++++++++++++-------------------
mkfs/main.c | 192 ++--------
3 files changed, 566 insertions(+), 595 deletions(-)
diff --git a/lib/liberofs_oci.h b/lib/liberofs_oci.h
index 3a8108b..d119a2b 100644
--- a/lib/liberofs_oci.h
+++ b/lib/liberofs_oci.h
@@ -8,85 +8,65 @@
#include <stdbool.h>
-#define DOCKER_REGISTRY "docker.io"
-#define DOCKER_API_REGISTRY "registry-1.docker.io"
-
#ifdef __cplusplus
extern "C" {
#endif
-struct erofs_inode;
-struct CURL;
struct erofs_importer;
-/**
- * struct erofs_oci_params - OCI configuration parameters
- * @registry: registry hostname (e.g., "registry-1.docker.io")
- * @repository: image repository (e.g., "library/ubuntu")
- * @tag: image tag or digest (e.g., "latest" or sha256:...)
+/*
+ * struct ocierofs_config - OCI configuration structure
+ * @image_ref: OCI image reference (e.g., "ubuntu:latest", "myregistry.com/app:v1.0")
* @platform: target platform in "os/arch" format (e.g., "linux/amd64")
* @username: username for authentication (optional)
* @password: password for authentication (optional)
* @layer_index: specific layer to extract (-1 for all layers)
*
- * Configuration structure for OCI image parameters including registry
- * location, image identification, platform specification, and authentication
- * credentials.
*/
-struct erofs_oci_params {
- char *registry;
- char *repository;
- char *tag;
+struct ocierofs_config {
+ char *image_ref;
char *platform;
char *username;
char *password;
int layer_index;
};
-/**
- * struct erofs_oci - Combined OCI client structure
- * @curl: CURL handle for HTTP requests
- * @params: OCI configuration parameters
- *
- * Main OCI client structure combining CURL HTTP client with
- * OCI-specific configuration parameters.
- */
-struct erofs_oci {
- struct CURL *curl;
- struct erofs_oci_params params;
+struct ocierofs_layer_info {
+ char *digest;
+ char *media_type;
+ u64 size;
};
-/*
- * ocierofs_init - Initialize OCI client with default parameters
- * @oci: OCI client structure to initialize
- *
- * Return: 0 on success, negative errno on failure
- */
-int ocierofs_init(struct erofs_oci *oci);
+struct ocierofs_ctx {
+ struct {
+ struct CURL *curl;
+ char *auth_header;
+ bool using_basic;
+ } net;
-/*
- * ocierofs_cleanup - Clean up OCI client and free allocated resources
- * @oci: OCI client structure to clean up
- */
-void ocierofs_cleanup(struct erofs_oci *oci);
+ struct {
+ char *registry;
+ char *repository;
+ char *platform;
+ int layer_index;
+ char *tag;
+ char *manifest_digest;
+ struct ocierofs_layer_info **layers;
+ int layer_count;
+ } img;
+};
-/*
- * erofs_oci_params_set_string - Set a string field with dynamic allocation
- * @field: pointer to the string field to set
- * @value: string value to set
- *
- * Return: 0 on success, negative errno on failure
- */
-int erofs_oci_params_set_string(char **field, const char *value);
+int ocierofs_init(struct ocierofs_ctx *ctx, const struct ocierofs_config *config);
/*
- * ocierofs_build_trees - Build file trees from OCI container image layers
- * @root: root inode to build the file tree under
- * @oci: OCI client structure with configured parameters
+ * ocierofs_build_trees - Build file trees from an OCI container image
+ * @importer: erofs importer to populate
+ * @cfg: oci configuration
*
* Return: 0 on success, negative errno on failure
*/
-int ocierofs_build_trees(struct erofs_importer *importer, struct erofs_oci *oci);
+int ocierofs_build_trees(struct erofs_importer *importer,
+ const struct ocierofs_config *cfg);
#ifdef __cplusplus
}
diff --git a/lib/remotes/oci.c b/lib/remotes/oci.c
index 0fb8c1f..01f1e24 100644
--- a/lib/remotes/oci.c
+++ b/lib/remotes/oci.c
@@ -21,6 +21,9 @@
#include "liberofs_oci.h"
#include "liberofs_private.h"
+#define DOCKER_REGISTRY "docker.io"
+#define DOCKER_API_REGISTRY "registry-1.docker.io"
+
#define DOCKER_MEDIATYPE_MANIFEST_V2 \
"application/vnd.docker.distribution.manifest.v2+json"
#define DOCKER_MEDIATYPE_MANIFEST_V1 \
@@ -30,28 +33,67 @@
#define OCI_MEDIATYPE_MANIFEST "application/vnd.oci.image.manifest.v1+json"
#define OCI_MEDIATYPE_INDEX "application/vnd.oci.image.index.v1+json"
-struct erofs_oci_request {
+struct ocierofs_request {
char *url;
struct curl_slist *headers;
};
-struct erofs_oci_response {
+struct ocierofs_response {
char *data;
size_t size;
long http_code;
};
-struct erofs_oci_stream {
- struct erofs_tarfile tarfile;
+struct ocierofs_stream {
const char *digest;
int blobfd;
};
+static inline const char *ocierofs_get_api_registry(const char *registry)
+{
+ if (!registry)
+ return DOCKER_API_REGISTRY;
+ return !strcmp(registry, DOCKER_REGISTRY) ? DOCKER_API_REGISTRY : registry;
+}
+
+static inline bool ocierofs_is_manifest_list(const char *media_type)
+{
+ return media_type && (!strcmp(media_type, DOCKER_MEDIATYPE_MANIFEST_LIST) ||
+ !strcmp(media_type, OCI_MEDIATYPE_INDEX));
+}
+
+static inline bool ocierofs_is_manifest(const char *media_type)
+{
+ return media_type && (!strcmp(media_type, DOCKER_MEDIATYPE_MANIFEST_V2) ||
+ !strcmp(media_type, OCI_MEDIATYPE_MANIFEST));
+}
+
+static inline void ocierofs_request_cleanup(struct ocierofs_request *req)
+{
+ if (!req)
+ return;
+ if (req->headers)
+ curl_slist_free_all(req->headers);
+ free(req->url);
+ req->url = NULL;
+ req->headers = NULL;
+}
+
+static inline void ocierofs_response_cleanup(struct ocierofs_response *resp)
+{
+ if (!resp)
+ return;
+ free(resp->data);
+ resp->data = NULL;
+ resp->size = 0;
+ resp->http_code = 0;
+}
+
static size_t ocierofs_write_callback(void *contents, size_t size,
size_t nmemb, void *userp)
{
size_t realsize = size * nmemb;
- struct erofs_oci_response *resp = userp;
+ struct ocierofs_response *resp = userp;
char *ptr;
if (!resp->data)
@@ -72,16 +114,23 @@ static size_t ocierofs_write_callback(void *contents, size_t size,
static size_t ocierofs_layer_write_callback(void *contents, size_t size,
size_t nmemb, void *userp)
{
- struct erofs_oci_stream *stream = userp;
+ struct ocierofs_stream *stream = userp;
size_t realsize = size * nmemb;
+ const char *buf = contents;
+ size_t written = 0;
if (stream->blobfd < 0)
return 0;
- if (write(stream->blobfd, contents, realsize) != realsize) {
- erofs_err("failed to write layer data for layer %s",
- stream->digest);
- return 0;
+ while (written < realsize) {
+ ssize_t n = write(stream->blobfd, buf + written, realsize - written);
+
+ if (n < 0) {
+ erofs_err("failed to write layer data for layer %s",
+ stream->digest);
+ return 0;
+ }
+ written += n;
}
return realsize;
}
@@ -93,6 +142,14 @@ static int ocierofs_curl_setup_common_options(struct CURL *curl)
curl_easy_setopt(curl, CURLOPT_TIMEOUT, 120L);
curl_easy_setopt(curl, CURLOPT_NOSIGNAL, 1L);
curl_easy_setopt(curl, CURLOPT_USERAGENT, "ocierofs/" PACKAGE_VERSION);
+ curl_easy_setopt(curl, CURLOPT_ACCEPT_ENCODING, "");
+ curl_easy_setopt(curl, CURLOPT_TCP_KEEPALIVE, 1L);
+#if defined(CURLOPT_TCP_KEEPIDLE)
+ curl_easy_setopt(curl, CURLOPT_TCP_KEEPIDLE, 30L);
+#endif
+#if defined(CURLOPT_TCP_KEEPINTVL)
+ curl_easy_setopt(curl, CURLOPT_TCP_KEEPINTVL, 15L);
+#endif
return 0;
}
@@ -111,10 +168,10 @@ static int ocierofs_curl_setup_basic_auth(struct CURL *curl, const char *usernam
return 0;
}
-static int ocierofs_curl_clear_auth(struct CURL *curl)
+static int ocierofs_curl_clear_auth(struct ocierofs_ctx *ctx)
{
- curl_easy_setopt(curl, CURLOPT_USERPWD, NULL);
- curl_easy_setopt(curl, CURLOPT_HTTPAUTH, CURLAUTH_NONE);
+ curl_easy_setopt(ctx->net.curl, CURLOPT_USERPWD, NULL);
+ curl_easy_setopt(ctx->net.curl, CURLOPT_HTTPAUTH, CURLAUTH_NONE);
return 0;
}
@@ -173,20 +230,20 @@ static int ocierofs_curl_perform(struct CURL *curl, long *http_code_out)
return 0;
}
-static int ocierofs_request_perform(struct erofs_oci *oci,
- struct erofs_oci_request *req,
- struct erofs_oci_response *resp)
+static int ocierofs_request_perform(struct ocierofs_ctx *ctx,
+ struct ocierofs_request *req,
+ struct ocierofs_response *resp)
{
int ret;
- ret = ocierofs_curl_setup_rq(oci->curl, req->url,
+ ret = ocierofs_curl_setup_rq(ctx->net.curl, req->url,
OCIEROFS_HTTP_GET, req->headers,
- ocierofs_write_callback, resp,
+ ocierofs_write_callback, resp,
NULL, NULL);
if (ret)
return ret;
- ret = ocierofs_curl_perform(oci->curl, &resp->http_code);
+ ret = ocierofs_curl_perform(ctx->net.curl, &resp->http_code);
if (ret)
return ret;
@@ -201,15 +258,10 @@ static int ocierofs_request_perform(struct erofs_oci *oci,
* @realm_out: pointer to store realm value
* @service_out: pointer to store service value
* @scope_out: pointer to store scope value
- *
- * Parse Bearer authentication header and extract realm, service, and scope
- * parameters for subsequent token requests.
- *
- * Return: 0 on success, negative errno on failure
*/
static int ocierofs_parse_auth_header(const char *auth_header,
- char **realm_out, char **service_out,
- char **scope_out)
+ char **realm_out, char **service_out,
+ char **scope_out)
{
char *realm = NULL, *service = NULL, *scope = NULL;
static const char * const param_names[] = {"realm=", "service=", "scope="};
@@ -218,7 +270,6 @@ static int ocierofs_parse_auth_header(const char *auth_header,
const char *p;
int i, ret = 0;
- // https://datatracker.ietf.org/doc/html/rfc6750#section-3
if (strncmp(auth_header, "Bearer ", strlen("Bearer ")))
return -EINVAL;
@@ -226,7 +277,6 @@ static int ocierofs_parse_auth_header(const char *auth_header,
if (!header_copy)
return -ENOMEM;
- /* Clean up header: replace newlines with spaces and remove double spaces */
for (char *q = header_copy; *q; q++) {
if (*q == '\n' || *q == '\r')
*q = ' ';
@@ -274,22 +324,9 @@ out:
return ret;
}
-/**
- * ocierofs_extract_www_auth_info - Extract WWW-Authenticate header information
- * @resp_data: HTTP response data containing headers
- * @realm_out: pointer to store realm value (optional)
- * @service_out: pointer to store service value (optional)
- * @scope_out: pointer to store scope value (optional)
- *
- * Extract realm, service, and scope from WWW-Authenticate header in HTTP response.
- * This function handles the common pattern of parsing WWW-Authenticate headers
- * that appears in multiple places in the OCI authentication flow.
- *
- * Return: 0 on success, negative errno on failure
- */
static int ocierofs_extract_www_auth_info(const char *resp_data,
- char **realm_out, char **service_out,
- char **scope_out)
+ char **realm_out, char **service_out,
+ char **scope_out)
{
char *www_auth;
char *line_end;
@@ -333,29 +370,15 @@ static int ocierofs_extract_www_auth_info(const char *resp_data,
return ret;
}
-/**
- * ocierofs_get_auth_token_with_url - Get authentication token from auth server
- * @oci: OCI client structure
- * @auth_url: authentication server URL
- * @service: service name for authentication
- * @repository: repository name
- * @username: username for basic auth (optional)
- * @password: password for basic auth (optional)
- *
- * Request authentication token from the specified auth server URL using
- * basic authentication if credentials are provided.
- *
- * Return: authentication header string on success, ERR_PTR on failure
- */
-static char *ocierofs_get_auth_token_with_url(struct erofs_oci *oci,
- const char *auth_url,
- const char *service,
- const char *repository,
- const char *username,
- const char *password)
+static char *ocierofs_get_auth_token_with_url(struct ocierofs_ctx *ctx,
+ const char *auth_url,
+ const char *service,
+ const char *repository,
+ const char *username,
+ const char *password)
{
- struct erofs_oci_request req = {};
- struct erofs_oci_response resp = {};
+ struct ocierofs_request req = {};
+ struct ocierofs_response resp = {};
json_object *root, *token_obj, *access_token_obj;
const char *token;
char *auth_header = NULL;
@@ -370,14 +393,14 @@ static char *ocierofs_get_auth_token_with_url(struct erofs_oci *oci,
}
if (username && password && *username) {
- ret = ocierofs_curl_setup_basic_auth(oci->curl, username,
- password);
+ ret = ocierofs_curl_setup_basic_auth(ctx->net.curl, username,
+ password);
if (ret)
goto out_url;
}
- ret = ocierofs_request_perform(oci, &req, &resp);
- ocierofs_curl_clear_auth(oci->curl);
+ ret = ocierofs_request_perform(ctx, &req, &resp);
+ ocierofs_curl_clear_auth(ctx);
if (ret)
goto out_url;
@@ -391,7 +414,7 @@ static char *ocierofs_get_auth_token_with_url(struct erofs_oci *oci,
if (!root) {
erofs_err("failed to parse auth response");
ret = -EINVAL;
- goto out_url;
+ goto out_json;
}
if (!json_object_object_get_ex(root, "token", &token_obj) &&
@@ -416,16 +439,16 @@ static char *ocierofs_get_auth_token_with_url(struct erofs_oci *oci,
out_json:
json_object_put(root);
out_url:
- free(req.url);
- free(resp.data);
+ ocierofs_response_cleanup(&resp);
+ ocierofs_request_cleanup(&req);
return ret ? ERR_PTR(ret) : auth_header;
}
-static char *ocierofs_discover_auth_endpoint(struct erofs_oci *oci,
- const char *registry,
- const char *repository)
+static char *ocierofs_discover_auth_endpoint(struct ocierofs_ctx *ctx,
+ const char *registry,
+ const char *repository)
{
- struct erofs_oci_response resp = {};
+ struct ocierofs_response resp = {};
char *realm = NULL;
char *service = NULL;
char *result = NULL;
@@ -434,20 +457,20 @@ static char *ocierofs_discover_auth_endpoint(struct erofs_oci *oci,
CURLcode res;
long http_code;
- api_registry = (!strcmp(registry, DOCKER_REGISTRY)) ? DOCKER_API_REGISTRY : registry;
+ api_registry = ocierofs_get_api_registry(registry);
if (asprintf(&test_url, "https://%s/v2/%s/manifests/nonexistent",
api_registry, repository) < 0)
return NULL;
- curl_easy_reset(oci->curl);
- ocierofs_curl_setup_common_options(oci->curl);
+ curl_easy_reset(ctx->net.curl);
+ ocierofs_curl_setup_common_options(ctx->net.curl);
- ocierofs_curl_setup_rq(oci->curl, test_url, OCIEROFS_HTTP_HEAD, NULL,
+ ocierofs_curl_setup_rq(ctx->net.curl, test_url, OCIEROFS_HTTP_HEAD, NULL,
NULL, NULL, ocierofs_write_callback, &resp);
- res = curl_easy_perform(oci->curl);
- curl_easy_getinfo(oci->curl, CURLINFO_RESPONSE_CODE, &http_code);
+ res = curl_easy_perform(ctx->net.curl);
+ curl_easy_getinfo(ctx->net.curl, CURLINFO_RESPONSE_CODE, &http_code);
if (res == CURLE_OK && (http_code == 401 || http_code == 403 ||
http_code == 404) && resp.data) {
@@ -458,14 +481,14 @@ static char *ocierofs_discover_auth_endpoint(struct erofs_oci *oci,
}
free(realm);
free(service);
- free(resp.data);
+ ocierofs_response_cleanup(&resp);
free(test_url);
return result;
}
-static char *ocierofs_get_auth_token(struct erofs_oci *oci, const char *registry,
- const char *repository, const char *username,
- const char *password)
+static char *ocierofs_get_auth_token(struct ocierofs_ctx *ctx, const char *registry,
+ const char *repository, const char *username,
+ const char *password)
{
static const char * const auth_patterns[] = {
"https://%s/v2/auth",
@@ -484,35 +507,35 @@ static char *ocierofs_get_auth_token(struct erofs_oci *oci, const char *registry
!strcmp(registry, DOCKER_REGISTRY);
if (docker_reg) {
service = "registry.docker.io";
- auth_header = ocierofs_get_auth_token_with_url(oci,
+ auth_header = ocierofs_get_auth_token_with_url(ctx,
"https://auth.docker.io/token", service, repository,
username, password);
if (!IS_ERR(auth_header))
return auth_header;
}
- discovered_auth_url = ocierofs_discover_auth_endpoint(oci, registry, repository);
+ discovered_auth_url = ocierofs_discover_auth_endpoint(ctx, registry, repository);
if (discovered_auth_url) {
const char *api_registry, *auth_service;
- struct erofs_oci_response resp = {};
+ struct ocierofs_response resp = {};
char *test_url;
CURLcode res;
long http_code;
- api_registry = (!strcmp(registry, DOCKER_REGISTRY)) ? DOCKER_API_REGISTRY : registry;
+ api_registry = ocierofs_get_api_registry(registry);
if (asprintf(&test_url, "https://%s/v2/%s/manifests/nonexistent",
api_registry, repository) >= 0) {
- curl_easy_reset(oci->curl);
- ocierofs_curl_setup_common_options(oci->curl);
+ curl_easy_reset(ctx->net.curl);
+ ocierofs_curl_setup_common_options(ctx->net.curl);
- ocierofs_curl_setup_rq(oci->curl, test_url,
- OCIEROFS_HTTP_HEAD, NULL,
- NULL, NULL,
- ocierofs_write_callback, &resp);
+ ocierofs_curl_setup_rq(ctx->net.curl, test_url,
+ OCIEROFS_HTTP_HEAD, NULL,
+ NULL, NULL,
+ ocierofs_write_callback, &resp);
- res = curl_easy_perform(oci->curl);
- curl_easy_getinfo(oci->curl, CURLINFO_RESPONSE_CODE, &http_code);
+ res = curl_easy_perform(ctx->net.curl);
+ curl_easy_getinfo(ctx->net.curl, CURLINFO_RESPONSE_CODE, &http_code);
if (res == CURLE_OK && (http_code == 401 || http_code == 403 ||
http_code == 404) && resp.data) {
@@ -521,14 +544,14 @@ static char *ocierofs_get_auth_token(struct erofs_oci *oci, const char *registry
ocierofs_extract_www_auth_info(resp.data, &realm, &discovered_service, NULL);
free(realm);
}
- free(resp.data);
+ ocierofs_response_cleanup(&resp);
free(test_url);
}
auth_service = discovered_service ? discovered_service : service;
- auth_header = ocierofs_get_auth_token_with_url(oci, discovered_auth_url,
- auth_service, repository,
- username, password);
+ auth_header = ocierofs_get_auth_token_with_url(ctx, discovered_auth_url,
+ auth_service, repository,
+ username, password);
free(discovered_auth_url);
free(discovered_service);
if (!IS_ERR(auth_header))
@@ -541,9 +564,9 @@ static char *ocierofs_get_auth_token(struct erofs_oci *oci, const char *registry
if (asprintf(&auth_url, auth_patterns[i], registry) < 0)
continue;
- auth_header = ocierofs_get_auth_token_with_url(oci, auth_url,
- service, repository,
- username, password);
+ auth_header = ocierofs_get_auth_token_with_url(ctx, auth_url,
+ service, repository,
+ username, password);
free(auth_url);
if (!IS_ERR(auth_header))
@@ -554,24 +577,24 @@ static char *ocierofs_get_auth_token(struct erofs_oci *oci, const char *registry
return ERR_PTR(-ENOENT);
}
-static char *ocierofs_get_manifest_digest(struct erofs_oci *oci,
- const char *registry,
- const char *repository, const char *tag,
- const char *platform,
- const char *auth_header)
+static char *ocierofs_get_manifest_digest(struct ocierofs_ctx *ctx,
+ const char *registry,
+ const char *repository, const char *tag,
+ const char *platform,
+ const char *auth_header)
{
- struct erofs_oci_request req = {};
- struct erofs_oci_response resp = {};
+ struct ocierofs_request req = {};
+ struct ocierofs_response resp = {};
json_object *root, *manifests, *manifest, *platform_obj, *arch_obj;
json_object *os_obj, *digest_obj, *schema_obj, *media_type_obj;
char *digest = NULL;
const char *api_registry;
int ret = 0, len, i;
- if (!registry || !repository || !tag || !platform)
+ if (!registry || !repository || !tag)
return ERR_PTR(-EINVAL);
- api_registry = (!strcmp(registry, DOCKER_REGISTRY)) ? DOCKER_API_REGISTRY : registry;
+ api_registry = ocierofs_get_api_registry(registry);
if (asprintf(&req.url, "https://%s/v2/%s/manifests/%s",
api_registry, repository, tag) < 0)
return ERR_PTR(-ENOMEM);
@@ -581,10 +604,10 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci,
req.headers = curl_slist_append(req.headers,
"Accept: " DOCKER_MEDIATYPE_MANIFEST_LIST ","
- OCI_MEDIATYPE_INDEX "," DOCKER_MEDIATYPE_MANIFEST_V1 ","
- DOCKER_MEDIATYPE_MANIFEST_V2);
+ OCI_MEDIATYPE_INDEX "," OCI_MEDIATYPE_MANIFEST ","
+ DOCKER_MEDIATYPE_MANIFEST_V1 "," DOCKER_MEDIATYPE_MANIFEST_V2);
- ret = ocierofs_request_perform(oci, &req, &resp);
+ ret = ocierofs_request_perform(ctx, &req, &resp);
if (ret)
goto out;
@@ -612,8 +635,7 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci,
if (json_object_object_get_ex(root, "mediaType", &media_type_obj)) {
const char *media_type = json_object_get_string(media_type_obj);
- if (!strcmp(media_type, DOCKER_MEDIATYPE_MANIFEST_V2) ||
- !strcmp(media_type, OCI_MEDIATYPE_MANIFEST)) {
+ if (ocierofs_is_manifest(media_type)) {
digest = strdup(tag);
ret = 0;
goto out_json;
@@ -631,9 +653,9 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci,
manifest = json_object_array_get_idx(manifests, i);
if (json_object_object_get_ex(manifest, "platform",
- &platform_obj) &&
+ &platform_obj) &&
json_object_object_get_ex(platform_obj, "architecture",
- &arch_obj) &&
+ &arch_obj) &&
json_object_object_get_ex(platform_obj, "os", &os_obj) &&
json_object_object_get_ex(manifest, "digest", &digest_obj)) {
const char *arch = json_object_get_string(arch_obj);
@@ -655,34 +677,47 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci,
out_json:
json_object_put(root);
out:
- free(resp.data);
- if (req.headers)
- curl_slist_free_all(req.headers);
- free(req.url);
-
+ ocierofs_response_cleanup(&resp);
+ ocierofs_request_cleanup(&req);
return ret ? ERR_PTR(ret) : digest;
}
-static char **ocierofs_get_layers_info(struct erofs_oci *oci,
- const char *registry,
- const char *repository,
- const char *digest,
- const char *auth_header,
- int *layer_count)
+static void ocierofs_free_layers_info(struct ocierofs_layer_info **layers, int count)
+{
+ int i;
+
+ if (!layers)
+ return;
+
+ for (i = 0; i < count; i++) {
+ if (layers[i]) {
+ free(layers[i]->digest);
+ free(layers[i]->media_type);
+ free(layers[i]);
+ }
+ }
+ free(layers);
+}
+
+static struct ocierofs_layer_info **ocierofs_fetch_layers_info(struct ocierofs_ctx *ctx,
+ const char *registry,
+ const char *repository,
+ const char *digest,
+ const char *auth_header,
+ int *layer_count)
{
- struct erofs_oci_request req = {};
- struct erofs_oci_response resp = {};
- json_object *root, *layers, *layer, *digest_obj;
- char **layers_info = NULL;
+ struct ocierofs_request req = {};
+ struct ocierofs_response resp = {};
+ json_object *root, *layers, *layer, *digest_obj, *media_type_obj, *size_obj;
+ struct ocierofs_layer_info **layers_info = NULL;
const char *api_registry;
- int ret, len, i, j;
+ int ret, len, i;
if (!registry || !repository || !digest || !layer_count)
return ERR_PTR(-EINVAL);
*layer_count = 0;
- api_registry = (!strcmp(registry, DOCKER_REGISTRY) ?
- DOCKER_API_REGISTRY : registry);
+ api_registry = ocierofs_get_api_registry(registry);
if (asprintf(&req.url, "https://%s/v2/%s/manifests/%s",
api_registry, repository, digest) < 0)
@@ -694,7 +729,7 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci,
req.headers = curl_slist_append(req.headers,
"Accept: " OCI_MEDIATYPE_MANIFEST "," DOCKER_MEDIATYPE_MANIFEST_V2);
- ret = ocierofs_request_perform(oci, &req, &resp);
+ ret = ocierofs_request_perform(ctx, &req, &resp);
if (ret)
goto out;
@@ -725,7 +760,7 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci,
goto out_json;
}
- layers_info = calloc(len, sizeof(char *));
+ layers_info = calloc(len, sizeof(*layers_info));
if (!layers_info) {
ret = -ENOMEM;
goto out_json;
@@ -740,329 +775,421 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci,
goto out_free;
}
- layers_info[i] = strdup(json_object_get_string(digest_obj));
+ layers_info[i] = calloc(1, sizeof(**layers_info));
if (!layers_info[i]) {
ret = -ENOMEM;
goto out_free;
}
+ layers_info[i]->digest = strdup(json_object_get_string(digest_obj));
+ if (!layers_info[i]->digest) {
+ ret = -ENOMEM;
+ goto out_free;
+ }
+ if (json_object_object_get_ex(layer, "mediaType", &media_type_obj))
+ layers_info[i]->media_type = strdup(json_object_get_string(media_type_obj));
+ else
+ layers_info[i]->media_type = NULL;
+
+ if (json_object_object_get_ex(layer, "size", &size_obj))
+ layers_info[i]->size = json_object_get_int64(size_obj);
+ else
+ layers_info[i]->size = 0;
}
*layer_count = len;
json_object_put(root);
- free(resp.data);
- if (req.headers)
- curl_slist_free_all(req.headers);
- free(req.url);
+ ocierofs_response_cleanup(&resp);
+ ocierofs_request_cleanup(&req);
return layers_info;
out_free:
- if (layers_info) {
- for (j = 0; j < i; j++)
- free(layers_info[j]);
- }
- free(layers_info);
+ ocierofs_free_layers_info(layers_info, i);
out_json:
json_object_put(root);
out:
- free(resp.data);
- if (req.headers)
- curl_slist_free_all(req.headers);
- free(req.url);
+ ocierofs_response_cleanup(&resp);
+ ocierofs_request_cleanup(&req);
return ERR_PTR(ret);
}
-static int ocierofs_extract_layer(struct erofs_oci *oci, struct erofs_importer *importer,
- const char *digest, const char *auth_header)
+static int ocierofs_process_tar_stream(struct erofs_importer *importer, int fd)
{
- struct erofs_oci_request req = {};
- struct erofs_oci_stream stream = {};
- const char *api_registry;
- long http_code;
+ struct erofs_tarfile tarfile = {};
int ret;
- stream = (struct erofs_oci_stream) {
- .digest = digest,
- .blobfd = erofs_tmpfile(),
- };
- if (stream.blobfd < 0) {
- erofs_err("failed to create temporary file for %s", digest);
- return -errno;
- }
-
- api_registry = (!strcmp(oci->params.registry, DOCKER_REGISTRY)) ?
- DOCKER_API_REGISTRY : oci->params.registry;
- if (asprintf(&req.url, "https://%s/v2/%s/blobs/%s",
- api_registry, oci->params.repository, digest) == -1) {
- ret = -ENOMEM;
- goto out;
- }
-
- if (auth_header && strstr(auth_header, "Bearer"))
- req.headers = curl_slist_append(req.headers, auth_header);
-
- curl_easy_reset(oci->curl);
-
- ret = ocierofs_curl_setup_common_options(oci->curl);
- if (ret)
- goto out;
-
- ret = ocierofs_curl_setup_rq(oci->curl, req.url, OCIEROFS_HTTP_GET,
- req.headers,
- ocierofs_layer_write_callback,
- &stream, NULL, NULL);
- if (ret)
- goto out;
-
- ret = ocierofs_curl_perform(oci->curl, &http_code);
- if (ret)
- goto out;
-
- if (http_code < 200 || http_code >= 300) {
- erofs_err("HTTP request failed with code %ld", http_code);
- ret = -EIO;
- goto out;
- }
+ init_list_head(&tarfile.global.xattrs);
- if (lseek(stream.blobfd, 0, SEEK_SET) < 0) {
- erofs_err("failed to seek to beginning of temp file: %s",
- strerror(errno));
- ret = -errno;
- goto out;
- }
-
- memset(&stream.tarfile, 0, sizeof(stream.tarfile));
- init_list_head(&stream.tarfile.global.xattrs);
-
- ret = erofs_iostream_open(&stream.tarfile.ios, stream.blobfd,
- EROFS_IOS_DECODER_GZIP);
+ ret = erofs_iostream_open(&tarfile.ios, fd, EROFS_IOS_DECODER_GZIP);
if (ret) {
erofs_err("failed to initialize tar stream: %s",
erofs_strerror(ret));
- goto out;
+ return ret;
}
do {
- ret = tarerofs_parse_tar(importer, &stream.tarfile);
+ ret = tarerofs_parse_tar(importer, &tarfile);
/* Continue parsing until end of archive */
} while (!ret);
- erofs_iostream_close(&stream.tarfile.ios);
+ erofs_iostream_close(&tarfile.ios);
if (ret < 0 && ret != -ENODATA) {
erofs_err("failed to process tar stream: %s",
erofs_strerror(ret));
- goto out;
+ return ret;
}
- ret = 0;
-out:
- if (stream.blobfd >= 0)
- close(stream.blobfd);
- if (req.headers)
- curl_slist_free_all(req.headers);
- free(req.url);
- return ret;
+ return 0;
}
-/**
- * ocierofs_build_trees - Build file trees from OCI container image layers
- * @importer: EROFS importer structure
- * @oci: OCI client structure with configured parameters
- *
- * Extract and build file system trees from all layers of an OCI container
- * image.
- *
- * Return: 0 on success, negative errno on failure
- */
-int ocierofs_build_trees(struct erofs_importer *importer, struct erofs_oci *oci)
+static int ocierofs_prepare_auth(struct ocierofs_ctx *ctx,
+ const char *username,
+ const char *password)
{
char *auth_header = NULL;
- char *manifest_digest = NULL;
- char **layers = NULL;
- int layer_count = 0;
- int ret, i;
-
- if (!importer || !oci)
- return -EINVAL;
+ int ret = 0;
+
+ ctx->net.using_basic = false;
+ free(ctx->net.auth_header);
+ ctx->net.auth_header = NULL;
+
+ /* Try to get a Bearer token first (with credentials if provided) */
+ auth_header = ocierofs_get_auth_token(ctx,
+ ctx->img.registry,
+ ctx->img.repository,
+ username, password);
+ if (!IS_ERR(auth_header)) {
+ ctx->net.auth_header = auth_header;
+ return 0;
+ }
- if (oci->params.username && oci->params.password &&
- oci->params.username[0] && oci->params.password[0]) {
- auth_header = ocierofs_get_auth_token(oci,
- oci->params.registry,
- oci->params.repository,
- oci->params.username,
- oci->params.password);
- if (IS_ERR(auth_header)) {
- auth_header = NULL;
- ret = ocierofs_curl_setup_basic_auth(oci->curl,
- oci->params.username,
- oci->params.password);
- if (ret)
- goto out;
- }
- } else {
- auth_header = ocierofs_get_auth_token(oci,
- oci->params.registry,
- oci->params.repository,
- NULL, NULL);
- if (IS_ERR(auth_header))
- auth_header = NULL;
+ /* If token failed but credentials are provided, fall back to Basic */
+ if (username && password && *username && *password) {
+ ret = ocierofs_curl_setup_basic_auth(ctx->net.curl,
+ username, password);
+ if (ret)
+ return ret;
+ ctx->net.using_basic = true;
}
+ return 0;
+}
- manifest_digest = ocierofs_get_manifest_digest(oci, oci->params.registry,
- oci->params.repository,
- oci->params.tag,
- oci->params.platform,
- auth_header);
- if (IS_ERR(manifest_digest)) {
- ret = PTR_ERR(manifest_digest);
+static int ocierofs_prepare_layers(struct ocierofs_ctx *ctx,
+ const struct ocierofs_config *config)
+{
+ int ret;
+
+ ret = ocierofs_prepare_auth(ctx, config ? config->username : NULL,
+ config ? config->password : NULL);
+ if (ret)
+ return ret;
+
+ ctx->img.manifest_digest = ocierofs_get_manifest_digest(ctx, ctx->img.registry,
+ ctx->img.repository,
+ ctx->img.tag,
+ ctx->img.platform,
+ ctx->net.auth_header);
+ if (IS_ERR(ctx->img.manifest_digest)) {
+ ret = PTR_ERR(ctx->img.manifest_digest);
erofs_err("failed to get manifest digest: %s",
erofs_strerror(ret));
+ ctx->img.manifest_digest = NULL;
goto out_auth;
}
- layers = ocierofs_get_layers_info(oci, oci->params.registry,
- oci->params.repository,
- manifest_digest, auth_header,
- &layer_count);
- if (IS_ERR(layers)) {
- ret = PTR_ERR(layers);
+ ctx->img.layers = ocierofs_fetch_layers_info(ctx, ctx->img.registry,
+ ctx->img.repository,
+ ctx->img.manifest_digest, ctx->net.auth_header,
+ &ctx->img.layer_count);
+ if (IS_ERR(ctx->img.layers)) {
+ ret = PTR_ERR(ctx->img.layers);
erofs_err("failed to get image layers: %s", erofs_strerror(ret));
+ ctx->img.layers = NULL;
goto out_manifest;
}
- if (oci->params.layer_index >= 0) {
- if (oci->params.layer_index >= layer_count) {
+ if (ctx->img.layer_index >= 0) {
+ if (ctx->img.layer_index >= ctx->img.layer_count) {
erofs_err("layer index %d exceeds available layers (%d)",
- oci->params.layer_index, layer_count);
+ ctx->img.layer_index, ctx->img.layer_count);
ret = -EINVAL;
goto out_layers;
}
- layer_count = 1;
- i = oci->params.layer_index;
+ ctx->img.layer_count = 1;
} else {
- i = 0;
+ ctx->img.layer_index = 0;
}
- while (i < layer_count) {
- char *trimmed = erofs_trim_for_progressinfo(layers[i],
- sizeof("Extracting layer ...") - 1);
- erofs_update_progressinfo("Extracting layer %d: %s ...", i,
- trimmed);
- free(trimmed);
- ret = ocierofs_extract_layer(oci, importer, layers[i],
- auth_header);
- if (ret) {
- erofs_err("failed to extract layer %d: %s", i,
- erofs_strerror(ret));
- break;
- }
- i++;
- }
+ return 0;
+
out_layers:
- for (i = 0; i < layer_count; i++)
- free(layers[i]);
- free(layers);
+ free(ctx->img.layers);
+ ctx->img.layers = NULL;
out_manifest:
- free(manifest_digest);
+ free(ctx->img.manifest_digest);
+ ctx->img.manifest_digest = NULL;
out_auth:
- free(auth_header);
-
- if (oci->params.username && oci->params.password &&
- oci->params.username[0] && oci->params.password[0] &&
- !auth_header) {
- ocierofs_curl_clear_auth(oci->curl);
- }
-out:
+ free(ctx->net.auth_header);
+ ctx->net.auth_header = NULL;
+ if (ctx->net.using_basic)
+ ocierofs_curl_clear_auth(ctx);
return ret;
}
-/**
- * ocierofs_init - Initialize OCI client with default parameters
- * @oci: OCI client structure to initialize
- *
- * Initialize OCI client structure, set up CURL handle, and configure
- * default parameters including platform (linux/amd64), registry
- * (registry-1.docker.io), and tag (latest).
- *
- * Return: 0 on success, negative errno on failure
- */
-int ocierofs_init(struct erofs_oci *oci)
+static int ocierofs_parse_ref(struct ocierofs_ctx *ctx, const char *ref_str)
{
- if (!oci)
+ char *slash, *colon, *dot;
+ const char *repo_part;
+ size_t len;
+ char *tmp;
+
+ if (!ctx || !ref_str)
return -EINVAL;
- *oci = (struct erofs_oci){};
- oci->curl = curl_easy_init();
- if (!oci->curl)
+ slash = strchr(ref_str, '/');
+ if (slash) {
+ dot = strchr(ref_str, '.');
+ if (dot && dot < slash) {
+ len = slash - ref_str;
+ tmp = strndup(ref_str, len);
+ if (!tmp)
+ return -ENOMEM;
+ free(ctx->img.registry);
+ ctx->img.registry = tmp;
+ repo_part = slash + 1;
+ } else {
+ repo_part = ref_str;
+ }
+ } else {
+ repo_part = ref_str;
+ }
+
+ colon = strchr(repo_part, ':');
+ if (colon) {
+ len = colon - repo_part;
+ tmp = strndup(repo_part, len);
+ if (!tmp)
+ return -ENOMEM;
+
+ if (!strchr(tmp, '/') &&
+ (!strcmp(ctx->img.registry, DOCKER_API_REGISTRY) ||
+ !strcmp(ctx->img.registry, DOCKER_REGISTRY))) {
+ char *full_repo;
+
+ if (asprintf(&full_repo, "library/%s", tmp) == -1) {
+ free(tmp);
+ return -ENOMEM;
+ }
+ free(tmp);
+ tmp = full_repo;
+ }
+ free(ctx->img.repository);
+ ctx->img.repository = tmp;
+
+ free(ctx->img.tag);
+ ctx->img.tag = strdup(colon + 1);
+ if (!ctx->img.tag)
+ return -ENOMEM;
+ } else {
+ tmp = strdup(repo_part);
+ if (!tmp)
+ return -ENOMEM;
+
+ if (!strchr(tmp, '/') &&
+ (!strcmp(ctx->img.registry, DOCKER_API_REGISTRY) ||
+ !strcmp(ctx->img.registry, DOCKER_REGISTRY))) {
+
+ char *full_repo;
+ if (asprintf(&full_repo, "library/%s", tmp) == -1) {
+ free(tmp);
+ return -ENOMEM;
+ }
+ free(tmp);
+ tmp = full_repo;
+ }
+ free(ctx->img.repository);
+ ctx->img.repository = tmp;
+ }
+ return 0;
+}
+
+int ocierofs_init(struct ocierofs_ctx *ctx, const struct ocierofs_config *config)
+{
+ int ret;
+
+ ctx->net.curl = curl_easy_init();
+ if (!ctx->net.curl)
return -EIO;
- if (ocierofs_curl_setup_common_options(oci->curl)) {
- ocierofs_cleanup(oci);
+ if (ocierofs_curl_setup_common_options(ctx->net.curl))
return -EIO;
- }
- if (erofs_oci_params_set_string(&oci->params.platform,
- "linux/amd64") ||
- erofs_oci_params_set_string(&oci->params.registry,
- DOCKER_API_REGISTRY) ||
- erofs_oci_params_set_string(&oci->params.tag, "latest")) {
- ocierofs_cleanup(oci);
+ ctx->img.layer_index = -1;
+ ctx->img.registry = strdup("registry-1.docker.io");
+ ctx->img.tag = strdup("latest");
+ if (config && config->platform)
+ ctx->img.platform = strdup(config->platform);
+ else
+ ctx->img.platform = strdup("linux/amd64");
+ if (!ctx->img.registry || !ctx->img.tag || !ctx->img.platform)
return -ENOMEM;
- }
- oci->params.layer_index = -1; /* -1 means extract all layers */
+
+ if (config && config->layer_index >= 0)
+ ctx->img.layer_index = config->layer_index;
+
+ ret = ocierofs_parse_ref(ctx, config->image_ref);
+ if (ret)
+ return ret;
+
+ ret = ocierofs_prepare_layers(ctx, config);
+ if (ret)
+ return ret;
+
return 0;
}
-/**
- * ocierofs_cleanup - Clean up OCI client and free allocated resources
- * @oci: OCI client structure to clean up
- *
- * Clean up CURL handle, free all allocated string parameters, and
- * reset the OCI client structure to a clean state.
- */
-void ocierofs_cleanup(struct erofs_oci *oci)
+static int ocierofs_download_blob_to_fd(struct ocierofs_ctx *ctx,
+ const char *digest,
+ const char *auth_header,
+ int outfd)
{
- if (!oci)
- return;
+ struct ocierofs_request req = {};
+ struct ocierofs_stream stream = {};
+ const char *api_registry;
+ long http_code;
+ int ret;
+
+ stream = (struct ocierofs_stream) {
+ .digest = digest,
+ .blobfd = outfd,
+ };
- if (oci->curl) {
- curl_easy_cleanup(oci->curl);
- oci->curl = NULL;
+ api_registry = ocierofs_get_api_registry(ctx->img.registry);
+ if (asprintf(&req.url, "https://%s/v2/%s/blobs/%s",
+ api_registry, ctx->img.repository, digest) == -1)
+ return -ENOMEM;
+
+ if (auth_header && strstr(auth_header, "Bearer"))
+ req.headers = curl_slist_append(req.headers, auth_header);
+
+ curl_easy_reset(ctx->net.curl);
+
+ ret = ocierofs_curl_setup_common_options(ctx->net.curl);
+ if (ret)
+ goto out;
+
+ ret = ocierofs_curl_setup_rq(ctx->net.curl, req.url, OCIEROFS_HTTP_GET,
+ req.headers,
+ ocierofs_layer_write_callback,
+ &stream, NULL, NULL);
+ if (ret)
+ goto out;
+
+ ret = ocierofs_curl_perform(ctx->net.curl, &http_code);
+ if (ret)
+ goto out;
+
+ if (http_code < 200 || http_code >= 300) {
+ erofs_err("HTTP request failed with code %ld", http_code);
+ ret = -EIO;
+ goto out;
+ }
+ ret = 0;
+out:
+ ocierofs_request_cleanup(&req);
+ return ret;
+}
+
+static int ocierofs_extract_layer(struct ocierofs_ctx *ctx,
+ const char *digest, const char *auth_header)
+{
+ struct ocierofs_stream stream = {};
+ int ret;
+
+ stream = (struct ocierofs_stream) {
+ .digest = digest,
+ .blobfd = erofs_tmpfile(),
+ };
+ if (stream.blobfd < 0) {
+ erofs_err("failed to create temporary file for %s", digest);
+ return -errno;
+ }
+
+ ret = ocierofs_download_blob_to_fd(ctx, digest, auth_header, stream.blobfd);
+ if (ret)
+ goto out;
+
+ if (lseek(stream.blobfd, 0, SEEK_SET) < 0) {
+ erofs_err("failed to seek to beginning of temp file: %s",
+ strerror(errno));
+ ret = -errno;
+ goto out;
}
- free(oci->params.registry);
- free(oci->params.repository);
- free(oci->params.tag);
- free(oci->params.platform);
- free(oci->params.username);
- free(oci->params.password);
-
- oci->params.registry = NULL;
- oci->params.repository = NULL;
- oci->params.tag = NULL;
- oci->params.platform = NULL;
- oci->params.username = NULL;
- oci->params.password = NULL;
+ return stream.blobfd;
+
+out:
+ if (stream.blobfd >= 0)
+ close(stream.blobfd);
+ return ret;
}
-int erofs_oci_params_set_string(char **field, const char *value)
+static void ocierofs_ctx_cleanup(struct ocierofs_ctx *ctx)
{
- char *new_value;
+ if (!ctx)
+ return;
- if (!field)
- return -EINVAL;
+ if (ctx->net.curl) {
+ curl_easy_cleanup(ctx->net.curl);
+ ctx->net.curl = NULL;
+ }
+ free(ctx->net.auth_header);
+ ctx->net.auth_header = NULL;
+
+ ocierofs_free_layers_info(ctx->img.layers, ctx->img.layer_count);
+ free(ctx->img.registry);
+ free(ctx->img.repository);
+ free(ctx->img.tag);
+ free(ctx->img.platform);
+ free(ctx->img.manifest_digest);
+ memset(&ctx->img, 0, sizeof(ctx->img));
+}
- if (!value) {
- free(*field);
- *field = NULL;
- return 0;
+int ocierofs_build_trees(struct erofs_importer *importer,
+ const struct ocierofs_config *config)
+{
+ struct ocierofs_ctx ctx = {};
+ int ret, i;
+
+ ret = ocierofs_init(&ctx, config);
+ if (ret) {
+ ocierofs_ctx_cleanup(&ctx);
+ return ret;
}
- new_value = strdup(value);
- if (!new_value)
- return -ENOMEM;
+ i = ctx.img.layer_index;
+ while (i < ctx.img.layer_count) {
+ char *trimmed = erofs_trim_for_progressinfo(ctx.img.layers[i]->digest,
+ sizeof("Extracting layer ...") - 1);
+ erofs_update_progressinfo("Extracting layer %d: %s ...", i,
+ trimmed);
+ free(trimmed);
+ int fd = ocierofs_extract_layer(&ctx, ctx.img.layers[i]->digest,
+ ctx.net.auth_header);
+ if (fd < 0) {
+ erofs_err("failed to extract layer %d: %s", i,
+ erofs_strerror(fd));
+ break;
+ }
+ ret = ocierofs_process_tar_stream(importer, fd);
+ close(fd);
+ if (ret) {
+ erofs_err("failed to process tar stream for layer %d: %s", i,
+ erofs_strerror(ret));
+ break;
+ }
+ i++;
+ }
- free(*field);
- *field = new_value;
- return 0;
+ ocierofs_ctx_cleanup(&ctx);
+ return ret;
}
diff --git a/mkfs/main.c b/mkfs/main.c
index bc895f1..5e8da7a 100644
--- a/mkfs/main.c
+++ b/mkfs/main.c
@@ -150,18 +150,18 @@ static void usage(int argc, char **argv)
* "0-9,100-109" instead of a continuous "0-109", and to
* state what those two subranges respectively mean. */
printf("%s [,level=<0-9,100-109>]\t0-9=normal, 100-109=extreme (default=%i)\n",
- spaces, s->c->default_level);
+ spaces, s->c->default_level);
else
printf("%s [,level=<0-%i>]\t\t(default=%i)\n",
- spaces, s->c->best_level, s->c->default_level);
+ spaces, s->c->best_level, s->c->default_level);
}
if (s->c->setdictsize) {
if (s->c->default_dictsize)
printf("%s [,dictsize=<dictsize>]\t(default=%u, max=%u)\n",
- spaces, s->c->default_dictsize, s->c->max_dictsize);
+ spaces, s->c->default_dictsize, s->c->max_dictsize);
else
printf("%s [,dictsize=<dictsize>]\t(default=<auto>, max=%u)\n",
- spaces, s->c->max_dictsize);
+ spaces, s->c->max_dictsize);
}
}
printf(
@@ -272,7 +272,7 @@ static struct erofs_s3 s3cfg;
#endif
#ifdef OCIEROFS_ENABLED
-static struct erofs_oci ocicfg;
+static struct ocierofs_config ocicfg;
static char *mkfs_oci_options;
#endif
@@ -689,21 +689,14 @@ static int mkfs_parse_s3_cfg(char *cfg_str)
#endif
#ifdef OCIEROFS_ENABLED
-
-
-/**
- * mkfs_parse_oci_options - Parse comma-separated OCI options string
+/*
+ * mkfs_parse_oci_options - Parse OCI options for mkfs tool
+ * @cfg: OCI configuration structure to update
* @options_str: comma-separated options string
- *
- * Parse OCI options string containing comma-separated key=value pairs.
- * Supported options include platform, layer, username, and password.
- *
- * Return: 0 on success, negative errno on failure
*/
-static int mkfs_parse_oci_options(char *options_str)
+static int mkfs_parse_oci_options(struct ocierofs_config *oci_cfg, char *options_str)
{
char *opt, *q, *p;
- int ret;
if (!options_str)
return 0;
@@ -717,41 +710,38 @@ static int mkfs_parse_oci_options(char *options_str)
p = strstr(opt, "platform=");
if (p) {
p += strlen("platform=");
- ret = erofs_oci_params_set_string(&ocicfg.params.platform, p);
- if (ret) {
- erofs_err("failed to set platform");
- return ret;
- }
+ free(oci_cfg->platform);
+ oci_cfg->platform = strdup(p);
+ if (!oci_cfg->platform)
+ return -ENOMEM;
} else {
p = strstr(opt, "layer=");
if (p) {
p += strlen("layer=");
- ocicfg.params.layer_index = atoi(p);
- if (ocicfg.params.layer_index < 0) {
+ oci_cfg->layer_index = atoi(p);
+ if (oci_cfg->layer_index < 0) {
erofs_err("invalid layer index %d",
- ocicfg.params.layer_index);
+ oci_cfg->layer_index);
return -EINVAL;
}
} else {
p = strstr(opt, "username=");
if (p) {
p += strlen("username=");
- ret = erofs_oci_params_set_string(&ocicfg.params.username, p);
- if (ret) {
- erofs_err("failed to set username");
- return ret;
- }
+ free(oci_cfg->username);
+ oci_cfg->username = strdup(p);
+ if (!oci_cfg->username)
+ return -ENOMEM;
} else {
p = strstr(opt, "password=");
if (p) {
p += strlen("password=");
- ret = erofs_oci_params_set_string(&ocicfg.params.password, p);
- if (ret) {
- erofs_err("failed to set password");
- return ret;
- }
+ free(oci_cfg->password);
+ oci_cfg->password = strdup(p);
+ if (!oci_cfg->password)
+ return -ENOMEM;
} else {
- erofs_err("invalid --oci value %s", opt);
+ erofs_err("mkfs: invalid --oci value %s", opt);
return -EINVAL;
}
}
@@ -763,125 +753,6 @@ static int mkfs_parse_oci_options(char *options_str)
return 0;
}
-
-/**
- * mkfs_parse_oci_ref - Parse OCI image reference string
- * @ref_str: OCI image reference in various formats
- *
- * Parse OCI image reference which can be in formats:
- * - registry.example.com/namespace/repo:tag
- * - namespace/repo:tag (uses default registry)
- * - repo:tag (adds library/ prefix for Docker Hub)
- * - repo (uses default tag "latest")
- *
- * Return: 0 on success, negative errno on failure
- */
-static int mkfs_parse_oci_ref(const char *ref_str)
-{
- char *slash, *colon, *dot;
- const char *repo_part;
- size_t len;
-
- slash = strchr(ref_str, '/');
- if (slash) {
- dot = strchr(ref_str, '.');
- if (dot && dot < slash) {
- char *registry_str;
-
- len = slash - ref_str;
- registry_str = strndup(ref_str, len);
-
- if (!registry_str) {
- erofs_err("failed to allocate memory for registry");
- return -ENOMEM;
- }
- if (erofs_oci_params_set_string(&ocicfg.params.registry,
- registry_str)) {
- free(registry_str);
- erofs_err("failed to set registry");
- return -ENOMEM;
- }
- free(registry_str);
- repo_part = slash + 1;
- } else {
- repo_part = ref_str;
- }
- } else {
- repo_part = ref_str;
- }
-
- colon = strchr(repo_part, ':');
- if (colon) {
- char *repo_str;
-
- len = colon - repo_part;
- repo_str = strndup(repo_part, len);
-
- if (!repo_str) {
- erofs_err("failed to allocate memory for repository");
- return -ENOMEM;
- }
-
- if (!strchr(repo_str, '/') &&
- (!strcmp(ocicfg.params.registry, DOCKER_API_REGISTRY) ||
- !strcmp(ocicfg.params.registry, DOCKER_REGISTRY))) {
- char *full_repo;
-
- if (asprintf(&full_repo, "library/%s", repo_str) == -1) {
- free(repo_str);
- erofs_err("failed to allocate memory for full repository name");
- return -ENOMEM;
- }
- free(repo_str);
- repo_str = full_repo;
- }
-
- if (erofs_oci_params_set_string(&ocicfg.params.repository,
- repo_str)) {
- free(repo_str);
- erofs_err("failed to set repository");
- return -ENOMEM;
- }
- free(repo_str);
-
- if (erofs_oci_params_set_string(&ocicfg.params.tag,
- colon + 1)) {
- erofs_err("failed to set tag");
- return -ENOMEM;
- }
- } else {
- char *repo_str = strdup(repo_part);
-
- if (!repo_str) {
- erofs_err("failed to allocate memory for repository");
- return -ENOMEM;
- }
-
- if (!strchr(repo_str, '/') &&
- (!strcmp(ocicfg.params.registry, DOCKER_API_REGISTRY) ||
- !strcmp(ocicfg.params.registry, DOCKER_REGISTRY))) {
- char *full_repo;
-
- if (asprintf(&full_repo, "library/%s", repo_str) == -1) {
- free(repo_str);
- erofs_err("failed to allocate memory for full repository name");
- return -ENOMEM;
- }
- free(repo_str);
- repo_str = full_repo;
- }
-
- if (erofs_oci_params_set_string(&ocicfg.params.repository,
- repo_str)) {
- free(repo_str);
- erofs_err("failed to set repository");
- return -ENOMEM;
- }
- free(repo_str);
- }
-
- return 0;
-}
#endif
static int mkfs_parse_one_compress_alg(char *alg,
@@ -1951,16 +1822,12 @@ int main(int argc, char **argv)
#endif
#ifdef OCIEROFS_ENABLED
} else if (source_mode == EROFS_MKFS_SOURCE_OCI) {
- err = ocierofs_init(&ocicfg);
- if (err)
- goto exit;
+ ocicfg.layer_index = -1;
- err = mkfs_parse_oci_options(mkfs_oci_options);
- if (err)
- goto exit;
- err = mkfs_parse_oci_ref(cfg.c_src_path);
+ err = mkfs_parse_oci_options(&ocicfg, mkfs_oci_options);
if (err)
goto exit;
+ ocicfg.image_ref = cfg.c_src_path;
if (incremental_mode ||
dataimport_mode == EROFS_MKFS_DATA_IMPORT_RVSP ||
@@ -2035,9 +1902,6 @@ exit:
erofs_blob_exit();
erofs_xattr_cleanup_name_prefixes();
erofs_rebuild_cleanup();
-#ifdef OCIEROFS_ENABLED
- ocierofs_cleanup(&ocicfg);
-#endif
erofs_diskbuf_exit();
if (source_mode == EROFS_MKFS_SOURCE_TAR) {
erofs_iostream_close(&erofstar.ios);
--
2.51.0
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH v3 2/2] erofs-utils: add NBD-backed OCI image mounting
2025-09-03 8:29 ` [PATCH v3 0/2] erofs-utils: refactor OCI and add NBD-backed OCI image mounting ChengyuZhu6
2025-09-03 8:29 ` [PATCH v3 1/2] erofs-utils: refactor OCI code for better modularity ChengyuZhu6
@ 2025-09-03 8:29 ` ChengyuZhu6
1 sibling, 0 replies; 19+ messages in thread
From: ChengyuZhu6 @ 2025-09-03 8:29 UTC (permalink / raw)
To: linux-erofs; +Cc: xiang, hsiangkao, Chengyu Zhu
From: Chengyu Zhu <hudsonzhu@tencent.com>
- Add HTTP range downloads for OCI blobs
- Introduce ocierofs_iostream for virtual file I/O
- Add --oci option for OCI image mounting with NBD backend
New mount.erofs -t erofs.nbd option: --oci=[option] source-image mountpoint
Supported options:
- platform=os/arch (default: linux/amd64)
- layer=N (extract specific layer, default: all layers)
- username/password (basic authentication)
e.g.:
./mount/mount.erofs -t erofs.nbd --oci=platform=linux/amd64 \
quay.io/chengyuzhu6/golang:1.22.8-erofs /tmp/test/
Signed-off-by: Chengyu Zhu <hudsonzhu@tencent.com>
---
lib/liberofs_oci.h | 19 +++-
lib/remotes/oci.c | 242 ++++++++++++++++++++++++++++++++++++++++++++-
mount/Makefile.am | 3 +-
mount/main.c | 236 +++++++++++++++++++++++++++++++++++++++++++
4 files changed, 493 insertions(+), 7 deletions(-)
diff --git a/lib/liberofs_oci.h b/lib/liberofs_oci.h
index d119a2b..f35228c 100644
--- a/lib/liberofs_oci.h
+++ b/lib/liberofs_oci.h
@@ -7,7 +7,6 @@
#define __EROFS_OCI_H
#include <stdbool.h>
-
#ifdef __cplusplus
extern "C" {
#endif
@@ -56,7 +55,11 @@ struct ocierofs_ctx {
} img;
};
-int ocierofs_init(struct ocierofs_ctx *ctx, const struct ocierofs_config *config);
+struct ocierofs_iostream {
+ struct ocierofs_ctx *ctx;
+ struct erofs_vfile vf;
+ u64 offset;
+};
/*
* ocierofs_build_trees - Build file trees from an OCI container image
@@ -65,8 +68,16 @@ int ocierofs_init(struct ocierofs_ctx *ctx, const struct ocierofs_config *config
*
* Return: 0 on success, negative errno on failure
*/
-int ocierofs_build_trees(struct erofs_importer *importer,
- const struct ocierofs_config *cfg);
+int ocierofs_build_trees(struct erofs_importer *importer, const struct ocierofs_config *cfg);
+
+int ocierofs_is_erofs_native_image(struct ocierofs_ctx *ctx);
+
+int ocierofs_init(struct ocierofs_ctx *ctx, const struct ocierofs_config *config);
+
+void ocierofs_ctx_cleanup(struct ocierofs_ctx *ctx);
+
+int ocierofs_iostream_open(struct ocierofs_iostream *oci_iostream, struct ocierofs_ctx *oci_ctx);
+void ocierofs_iostream_close(struct ocierofs_iostream *oci_iostream);
#ifdef __cplusplus
}
diff --git a/lib/remotes/oci.c b/lib/remotes/oci.c
index 01f1e24..8750772 100644
--- a/lib/remotes/oci.c
+++ b/lib/remotes/oci.c
@@ -33,6 +33,9 @@
#define OCI_MEDIATYPE_MANIFEST "application/vnd.oci.image.manifest.v1+json"
#define OCI_MEDIATYPE_INDEX "application/vnd.oci.image.index.v1+json"
+/* Erofs Native Layer Media Type */
+#define EROFS_MEDIATYPE "application/vnd.erofs"
+
struct ocierofs_request {
char *url;
struct curl_slist *headers;
@@ -1133,7 +1136,7 @@ out:
return ret;
}
-static void ocierofs_ctx_cleanup(struct ocierofs_ctx *ctx)
+void ocierofs_ctx_cleanup(struct ocierofs_ctx *ctx)
{
if (!ctx)
return;
@@ -1193,3 +1196,240 @@ int ocierofs_build_trees(struct erofs_importer *importer,
ocierofs_ctx_cleanup(&ctx);
return ret;
}
+
+static int ocierofs_download_blob_range(struct ocierofs_ctx *ctx, off_t offset, size_t length,
+ void **out_buf, size_t *out_size)
+{
+ struct ocierofs_request req = {};
+ struct ocierofs_response resp = {};
+ const char *api_registry;
+ char rangehdr[64];
+ long http_code = 0;
+ int ret;
+ int index = ctx->img.layer_index;
+
+ if (offset < 0)
+ return -EINVAL;
+
+ api_registry = ocierofs_get_api_registry(ctx->img.registry);
+ if (asprintf(&req.url, "https://%s/v2/%s/blobs/%s",
+ api_registry, ctx->img.repository, ctx->img.layers[index]->digest) == -1)
+ return -ENOMEM;
+
+ if (length)
+ snprintf(rangehdr, sizeof(rangehdr), "Range: bytes=%lld-%lld",
+ (long long)offset, (long long)(offset + (off_t)length - 1));
+ else
+ snprintf(rangehdr, sizeof(rangehdr), "Range: bytes=%lld-",
+ (long long)offset);
+
+ if (ctx->net.auth_header && strstr(ctx->net.auth_header, "Bearer"))
+ req.headers = curl_slist_append(req.headers, ctx->net.auth_header);
+ req.headers = curl_slist_append(req.headers, rangehdr);
+
+ curl_easy_reset(ctx->net.curl);
+
+ ret = ocierofs_curl_setup_common_options(ctx->net.curl);
+ if (ret)
+ goto out;
+
+ ret = ocierofs_curl_setup_rq(ctx->net.curl, req.url, OCIEROFS_HTTP_GET,
+ req.headers,
+ ocierofs_write_callback,
+ &resp, NULL, NULL);
+ if (ret)
+ goto out;
+
+ ret = ocierofs_curl_perform(ctx->net.curl, &http_code);
+ if (ret)
+ goto out;
+
+ if (http_code == 206) {
+ *out_buf = resp.data;
+ *out_size = resp.size;
+ resp.data = NULL;
+ ret = 0;
+ } else if (http_code == 200) {
+ if (offset == 0) {
+ *out_buf = resp.data;
+ *out_size = resp.size;
+ resp.data = NULL;
+ ret = 0;
+ } else {
+ if (offset < resp.size) {
+ size_t available = resp.size - offset;
+ size_t copy_size = length ? min_t(size_t, length, available) : available;
+
+ *out_buf = malloc(copy_size);
+ if (!*out_buf) {
+ ret = -ENOMEM;
+ goto out;
+ }
+ memcpy(*out_buf, resp.data + offset, copy_size);
+ *out_size = copy_size;
+ ret = 0;
+ } else {
+ *out_buf = NULL;
+ *out_size = 0;
+ ret = 0;
+ }
+ }
+ } else {
+ erofs_err("HTTP range request failed with code %ld", http_code);
+ ret = -EIO;
+ }
+
+out:
+ if (req.headers)
+ curl_slist_free_all(req.headers);
+ free(req.url);
+ free(resp.data);
+ return ret;
+}
+
+static ssize_t ocierofs_io_pread(struct erofs_vfile *vf, void *buf, size_t len, u64 offset)
+{
+ struct ocierofs_iostream *oci_iostream = *(struct ocierofs_iostream **)vf->payload;
+ void *download_buf = NULL;
+ size_t download_size = 0;
+ ssize_t ret;
+
+ ret = ocierofs_download_blob_range(oci_iostream->ctx, offset, len,
+ &download_buf, &download_size);
+ if (ret < 0) {
+ memset(buf, 0, len);
+ return len;
+ }
+
+ if (download_buf && download_size > 0) {
+ memcpy(buf, download_buf, download_size);
+ free(download_buf);
+ return download_size;
+ }
+
+ return 0;
+}
+
+static ssize_t ocierofs_io_read(struct erofs_vfile *vf, void *buf, size_t len)
+{
+ struct ocierofs_iostream *oci_iostream = *(struct ocierofs_iostream **)vf->payload;
+ ssize_t ret;
+
+ ret = ocierofs_io_pread(vf, buf, len, oci_iostream->offset);
+ if (ret > 0)
+ oci_iostream->offset += ret;
+
+ return ret;
+}
+
+static off_t ocierofs_io_lseek(struct erofs_vfile *vf, u64 offset, int whence)
+{
+ struct ocierofs_iostream *oci_iostream = *(struct ocierofs_iostream **)vf->payload;
+ off_t new_offset;
+ int layer_index = oci_iostream->ctx->img.layer_index;
+
+ switch (whence) {
+ case SEEK_SET:
+ new_offset = offset;
+ break;
+ case SEEK_CUR:
+ new_offset = oci_iostream->offset + offset;
+ break;
+ case SEEK_END:
+ new_offset = oci_iostream->ctx->img.layers[layer_index]->size + offset;
+ break;
+ default:
+ return -1;
+ }
+
+ if (new_offset < 0 || new_offset > oci_iostream->ctx->img.layers[layer_index]->size)
+ return -1;
+
+ oci_iostream->offset = new_offset;
+ return new_offset;
+}
+
+static ssize_t ocierofs_io_sendfile(struct erofs_vfile *vout, struct erofs_vfile *vin,
+ off_t *pos, size_t count)
+{
+ struct ocierofs_iostream *oci_iostream = *(struct ocierofs_iostream **)vin->payload;
+ char *buf = NULL;
+ ssize_t total_written = 0;
+ ssize_t ret = 0;
+
+ buf = malloc(min_t(size_t, count, 32768));
+ if (!buf)
+ return -ENOMEM;
+
+ while (count > 0) {
+ size_t to_read = min_t(size_t, count, 32768);
+ u64 read_offset = pos ? *pos : oci_iostream->offset;
+
+ ret = ocierofs_io_pread(vin, buf, to_read, read_offset);
+ if (ret <= 0) {
+ erofs_err("OCI I/O sendfile: failed to read from OCI: %s",
+ erofs_strerror(ret));
+ memset(buf, 0, to_read);
+ ret = to_read;
+ }
+
+ ssize_t written = write(vout->fd, buf, ret);
+
+ if (written < 0) {
+ erofs_err("OCI I/O sendfile: failed to write to output: %s",
+ strerror(errno));
+ ret = -errno;
+ break;
+ }
+
+ if (written != ret) {
+ erofs_err("OCI I/O sendfile: partial write: %zd != %zd", written, ret);
+ ret = written;
+ }
+
+ total_written += ret;
+ count -= ret;
+ if (pos)
+ *pos += ret;
+ else
+ oci_iostream->offset += ret;
+ }
+
+ free(buf);
+ return count;
+}
+
+static struct erofs_vfops ocierofs_io_vfops = {
+ .pread = ocierofs_io_pread,
+ .read = ocierofs_io_read,
+ .lseek = ocierofs_io_lseek,
+ .sendfile = ocierofs_io_sendfile,
+};
+
+int ocierofs_iostream_open(struct ocierofs_iostream *oci_iostream, struct ocierofs_ctx *oci_ctx)
+{
+
+ memset(oci_iostream, 0, sizeof(*oci_iostream));
+ oci_iostream->ctx = oci_ctx;
+ oci_iostream->vf.ops = &ocierofs_io_vfops;
+ oci_iostream->vf.fd = -1;
+ *(struct ocierofs_iostream **)oci_iostream->vf.payload = oci_iostream;
+
+ return 0;
+}
+
+void ocierofs_iostream_close(struct ocierofs_iostream *oci_iostream)
+{
+ close(oci_iostream->vf.fd);
+}
+
+int ocierofs_is_erofs_native_image(struct ocierofs_ctx *ctx)
+{
+ if (ctx->img.layer_count > 0 && ctx->img.layers[0] &&
+ ctx->img.layers[0]->media_type) {
+ if (strcmp(ctx->img.layers[0]->media_type, EROFS_MEDIATYPE) != 0)
+ return -ENOENT;
+ return 0;
+ }
+ return -ENOENT;
+}
diff --git a/mount/Makefile.am b/mount/Makefile.am
index d93f3f4..7b971f5 100644
--- a/mount/Makefile.am
+++ b/mount/Makefile.am
@@ -9,5 +9,4 @@ mount_erofs_SOURCES = main.c
mount_erofs_CFLAGS = -Wall -I$(top_srcdir)/include
mount_erofs_LDADD = $(top_builddir)/lib/liberofs.la ${libselinux_LIBS} \
${liblz4_LIBS} ${liblzma_LIBS} ${zlib_LIBS} ${libdeflate_LIBS} \
- ${libzstd_LIBS} ${libqpl_LIBS} ${libxxhash_LIBS} ${libnl3_LIBS}
-endif
+ ${libzstd_LIBS} ${libqpl_LIBS} ${libxxhash_LIBS} ${libnl3_LIBS} ${openssl_LIBS}
diff --git a/mount/main.c b/mount/main.c
index 139b532..77c16f5 100644
--- a/mount/main.c
+++ b/mount/main.c
@@ -15,6 +15,7 @@
#include "erofs/err.h"
#include "erofs/io.h"
#include "../lib/liberofs_nbd.h"
+#include "../lib/liberofs_oci.h"
#ifdef HAVE_LINUX_LOOP_H
#include <linux/loop.h>
#else
@@ -34,6 +35,10 @@ struct loop_info {
#include <sys/sysmacros.h>
#endif
+#ifdef OCIEROFS_ENABLED
+static struct ocierofs_config ocicfg;
+#endif
+
enum erofs_backend_drv {
EROFSAUTO,
EROFSLOCAL,
@@ -56,6 +61,11 @@ static struct erofsmount_cfg {
long flags;
enum erofs_backend_drv backend;
enum erofsmount_mode mountmode;
+ bool umount;
+#ifdef OCIEROFS_ENABLED
+ char *oci_options;
+ bool use_oci;
+#endif
} mountcfg = {
.full_options = "ro",
.flags = MS_RDONLY, /* default mountflags */
@@ -128,6 +138,9 @@ static int erofsmount_parse_options(int argc, char **argv)
static const struct option long_options[] = {
{"help", no_argument, 0, 'h'},
{"reattach", no_argument, 0, 512},
+#ifdef OCIEROFS_ENABLED
+ {"oci", optional_argument, 0, 513},
+#endif
{0, 0, 0, 0},
};
char *dot;
@@ -165,6 +178,12 @@ static int erofsmount_parse_options(int argc, char **argv)
case 512:
mountcfg.mountmode = EROFSMOUNT_MODE_REATTACH;
break;
+#ifdef OCIEROFS_ENABLED
+ case 513:
+ mountcfg.oci_options = optarg;
+ mountcfg.use_oci = true;
+ break;
+#endif
default:
return -EINVAL;
}
@@ -198,6 +217,74 @@ static int erofsmount_parse_options(int argc, char **argv)
return 0;
}
+static int mount_parse_oci_options(struct ocierofs_config *oci_cfg, char *options_str)
+{
+ char *opt, *q, *p;
+
+ if (!options_str)
+ return 0;
+
+ opt = options_str;
+ while (opt) {
+ q = strchr(opt, ',');
+ if (q)
+ *q = '\0';
+
+ p = strstr(opt, "platform=");
+ if (p) {
+ p += strlen("platform=");
+ free(oci_cfg->platform);
+ oci_cfg->platform = strdup(p);
+ if (!oci_cfg->platform)
+ return -ENOMEM;
+ opt = q ? q + 1 : NULL;
+ continue;
+ }
+
+ p = strstr(opt, "layer=");
+ if (p) {
+ p += strlen("layer=");
+ oci_cfg->layer_index = atoi(p);
+ if (oci_cfg->layer_index < 0) {
+ erofs_err("invalid layer index %d",
+ oci_cfg->layer_index);
+ return -EINVAL;
+ }
+ opt = q ? q + 1 : NULL;
+ continue;
+ }
+
+ p = strstr(opt, "username=");
+ if (p) {
+ p += strlen("username=");
+ free(oci_cfg->username);
+ oci_cfg->username = strdup(p);
+ if (!oci_cfg->username)
+ return -ENOMEM;
+ opt = q ? q + 1 : NULL;
+ continue;
+ }
+
+ p = strstr(opt, "password=");
+ if (p) {
+ p += strlen("password=");
+ free(oci_cfg->password);
+ oci_cfg->password = strdup(p);
+ if (!oci_cfg->password)
+ return -ENOMEM;
+ opt = q ? q + 1 : NULL;
+ continue;
+ }
+
+ erofs_err("mkfs: invalid --oci value %s", opt);
+ return -EINVAL;
+
+ opt = q ? q + 1 : NULL;
+ }
+
+ return 0;
+}
+
static int erofsmount_fuse(const char *source, const char *mountpoint,
const char *fstype, const char *options)
{
@@ -750,6 +837,122 @@ err_out:
return err < 0 ? err : 0;
}
+/**
+ * erofsmount_startnbd_oci - Start NBD server for OCI image
+ * @nbdfd: NBD device file descriptor
+ * @oci_ctx: OCI client structure (pre-authenticated)
+ * @auth_header: pre-authenticated auth header
+ *
+ * Start an NBD server that serves data from an OCI image layer.
+ * This function reuses the existing erofsmount_nbd_loopfn logic
+ * but uses erofsoci_iostream as the virtual device instead of a local file.
+ * The OCI client should be pre-authenticated to avoid concurrent auth issues.
+ *
+ * Return: 0 on success, negative errno on failure
+ */
+static int erofsmount_startnbd_oci(int nbdfd, struct ocierofs_ctx *oci_ctx)
+{
+ struct erofsmount_nbd_ctx ctx = {};
+ struct ocierofs_iostream *oci_iostream = NULL;
+ uintptr_t retcode;
+ pthread_t th;
+ int err, err2;
+ int blkbits = 12;
+ int index = oci_ctx->img.layer_index;
+ u64 blocks;
+
+ blocks = (oci_ctx->img.layers[index]->size + (1ULL << blkbits) - 1) >> blkbits;
+
+ oci_iostream = malloc(sizeof(struct ocierofs_iostream));
+ if (!oci_iostream)
+ return -ENOMEM;
+
+ err = ocierofs_iostream_open(oci_iostream, oci_ctx);
+ if (err) {
+ free(oci_iostream);
+ return err;
+ }
+
+ ctx.vd = oci_iostream->vf;
+
+ err = erofs_nbd_connect(nbdfd, blkbits, blocks);
+ if (err < 0) {
+ ocierofs_iostream_close(oci_iostream);
+ free(oci_iostream);
+ return err;
+ }
+ ctx.sk.fd = err;
+
+ err = -pthread_create(&th, NULL, erofsmount_nbd_loopfn, &ctx);
+ if (err) {
+ ocierofs_iostream_close(oci_iostream);
+ free(oci_iostream);
+ close(ctx.sk.fd);
+ return err;
+ }
+
+ err = erofs_nbd_do_it(nbdfd);
+ err2 = -pthread_join(th, (void **)&retcode);
+ if (!err2 && retcode) {
+ erofs_err("NBD worker failed with %s",
+ erofs_strerror(retcode));
+ err2 = retcode;
+ }
+
+ ocierofs_iostream_close(oci_iostream);
+ free(oci_iostream);
+
+ return err ?: err2;
+}
+
+static int erofsmount_nbd_oci(struct ocierofs_ctx *ctx, const char *mountpoint,
+ const char *fstype, int flags, const char *options)
+{
+ char nbdpath[32];
+ int num, nbdfd;
+ pid_t pid;
+ long err;
+
+ if (strcmp(fstype, "erofs")) {
+ fprintf(stderr, "unsupported filesystem type `%s`\n", fstype);
+ return -ENODEV;
+ }
+
+ flags |= O_RDONLY;
+
+ num = erofs_nbd_devscan();
+ if (num < 0)
+ return num;
+
+ (void)snprintf(nbdpath, sizeof(nbdpath), "/dev/nbd%d", num);
+ nbdfd = open(nbdpath, O_RDWR);
+ if (nbdfd < 0)
+ return -errno;
+
+ if ((pid = fork()) == 0) {
+ return erofsmount_startnbd_oci(nbdfd, ctx) ?
+ EXIT_FAILURE : EXIT_SUCCESS;
+ }
+ close(nbdfd);
+
+ while (1) {
+ err = erofs_nbd_in_service(num);
+ if (err == -ENOENT || err == -ENOTCONN) {
+ usleep(50000);
+ continue;
+ }
+ if (err >= 0)
+ err = (err != pid ? -EBUSY : 0);
+ break;
+ }
+ if (!err) {
+ err = mount(nbdpath, mountpoint, fstype, flags, options);
+ if (err < 0)
+ err = -errno;
+ }
+ return err;
+}
+
int main(int argc, char *argv[])
{
int err;
@@ -785,9 +988,42 @@ int main(int argc, char *argv[])
}
if (mountcfg.backend == EROFSNBD) {
+#ifdef OCIEROFS_ENABLED
+ if (mountcfg.use_oci) {
+ struct ocierofs_ctx ctx = {};
+
+ ocicfg.image_ref = mountcfg.device;
+ err = mount_parse_oci_options(&ocicfg, mountcfg.oci_options);
+ if (err)
+ goto exit;
+ err = ocierofs_init(&ctx, &ocicfg);
+ if (err) {
+ ocierofs_ctx_cleanup(&ctx);
+ goto exit;
+ }
+
+ err = ocierofs_is_erofs_native_image(&ctx);
+ if (err) {
+ ocierofs_ctx_cleanup(&ctx);
+ goto exit;
+ }
+
+ err = erofsmount_nbd_oci(&ctx, mountcfg.target,
+ mountcfg.fstype, mountcfg.flags, mountcfg.options);
+ if (err) {
+ ocierofs_ctx_cleanup(&ctx);
+ goto exit;
+ }
+ } else {
+ err = erofsmount_nbd(mountcfg.device, mountcfg.target,
+ mountcfg.fstype, mountcfg.flags,
+ mountcfg.options);
+ }
+#else
err = erofsmount_nbd(mountcfg.device, mountcfg.target,
mountcfg.fstype, mountcfg.flags,
mountcfg.options);
+#endif
goto exit;
}
--
2.51.0
^ permalink raw reply related [flat|nested] 19+ messages in thread
* Re: [PATCH v3 1/2] erofs-utils: refactor OCI code for better modularity
2025-09-03 8:29 ` [PATCH v3 1/2] erofs-utils: refactor OCI code for better modularity ChengyuZhu6
@ 2025-09-03 8:47 ` Gao Xiang
0 siblings, 0 replies; 19+ messages in thread
From: Gao Xiang @ 2025-09-03 8:47 UTC (permalink / raw)
To: ChengyuZhu6, linux-erofs; +Cc: xiang, Chengyu Zhu
On 2025/9/3 16:29, ChengyuZhu6 wrote:
> From: Chengyu Zhu <hudsonzhu@tencent.com>
>
> Refactor OCI code to improve code organization and maintainability:
>
> Key changes:
> - Add `ocierofs_get_api_registry()` and unify API endpoint selection.
> - Implement Bearer token discovery with Basic fallback; cache auth header.
> - Parse layer metadata (digest, mediaType, size) and add a proper free helper.
> - Split blob download from tar processing; process tar via a temp fd.
> - Rework init/teardown into `ocierofs_init()` and `ocierofs_ctx_cleanup()`.
> - Update mkfs to use `struct ocierofs_config` and new `--oci` parsing.
>
> Signed-off-by: Chengyu Zhu <hudsonzhu@tencent.com>
> ---
> lib/liberofs_oci.h | 84 ++---
> lib/remotes/oci.c | 885 ++++++++++++++++++++++++++-------------------
> mkfs/main.c | 192 ++--------
> 3 files changed, 566 insertions(+), 595 deletions(-)
>
> diff --git a/lib/liberofs_oci.h b/lib/liberofs_oci.h
> index 3a8108b..d119a2b 100644
> --- a/lib/liberofs_oci.h
> +++ b/lib/liberofs_oci.h
> @@ -8,85 +8,65 @@
>
> #include <stdbool.h>
>
> -#define DOCKER_REGISTRY "docker.io"
> -#define DOCKER_API_REGISTRY "registry-1.docker.io"
> -
> #ifdef __cplusplus
> extern "C" {
> #endif
>
> -struct erofs_inode;
> -struct CURL;
> struct erofs_importer;
>
> -/**
> - * struct erofs_oci_params - OCI configuration parameters
> - * @registry: registry hostname (e.g., "registry-1.docker.io")
> - * @repository: image repository (e.g., "library/ubuntu")
> - * @tag: image tag or digest (e.g., "latest" or sha256:...)
> +/*
> + * struct ocierofs_config - OCI configuration structure
> + * @image_ref: OCI image reference (e.g., "ubuntu:latest", "myregistry.com/app:v1.0")
> * @platform: target platform in "os/arch" format (e.g., "linux/amd64")
> * @username: username for authentication (optional)
> * @password: password for authentication (optional)
> * @layer_index: specific layer to extract (-1 for all layers)
> *
> - * Configuration structure for OCI image parameters including registry
> - * location, image identification, platform specification, and authentication
> - * credentials.
> */
> -struct erofs_oci_params {
> - char *registry;
> - char *repository;
> - char *tag;
> +struct ocierofs_config {
> + char *image_ref;
> char *platform;
> char *username;
> char *password;
> int layer_index;
> };
>
> -/**
> - * struct erofs_oci - Combined OCI client structure
> - * @curl: CURL handle for HTTP requests
> - * @params: OCI configuration parameters
> - *
> - * Main OCI client structure combining CURL HTTP client with
> - * OCI-specific configuration parameters.
> - */
> -struct erofs_oci {
> - struct CURL *curl;
> - struct erofs_oci_params params;
> +struct ocierofs_layer_info {
> + char *digest;
> + char *media_type;
> + u64 size;
> };
>
> -/*
> - * ocierofs_init - Initialize OCI client with default parameters
> - * @oci: OCI client structure to initialize
> - *
> - * Return: 0 on success, negative errno on failure
> - */
> -int ocierofs_init(struct erofs_oci *oci);
> +struct ocierofs_ctx {
> + struct {
> + struct CURL *curl;
> + char *auth_header;
> + bool using_basic;
> + } net;
Is it necessary to use two anonymous structures here?
Could we just unfold those fields?
...
> diff --git a/mkfs/main.c b/mkfs/main.c
> index bc895f1..5e8da7a 100644
> --- a/mkfs/main.c
> +++ b/mkfs/main.c
> @@ -150,18 +150,18 @@ static void usage(int argc, char **argv)
> * "0-9,100-109" instead of a continuous "0-109", and to
> * state what those two subranges respectively mean. */
> printf("%s [,level=<0-9,100-109>]\t0-9=normal, 100-109=extreme (default=%i)\n",
> - spaces, s->c->default_level);
> + spaces, s->c->default_level);
> else
> printf("%s [,level=<0-%i>]\t\t(default=%i)\n",
> - spaces, s->c->best_level, s->c->default_level);
> + spaces, s->c->best_level, s->c->default_level);
> }
> if (s->c->setdictsize) {
> if (s->c->default_dictsize)
> printf("%s [,dictsize=<dictsize>]\t(default=%u, max=%u)\n",
> - spaces, s->c->default_dictsize, s->c->max_dictsize);
> + spaces, s->c->default_dictsize, s->c->max_dictsize);
> else
> printf("%s [,dictsize=<dictsize>]\t(default=<auto>, max=%u)\n",
> - spaces, s->c->max_dictsize);
> + spaces, s->c->max_dictsize);
Are those changes unnecessary?
> }
> }
> printf(
> @@ -272,7 +272,7 @@ static struct erofs_s3 s3cfg;
> #endif
>
> #ifdef OCIEROFS_ENABLED
> -static struct erofs_oci ocicfg;
> +static struct ocierofs_config ocicfg;
> static char *mkfs_oci_options;
> #endif
>
> @@ -689,21 +689,14 @@ static int mkfs_parse_s3_cfg(char *cfg_str)
> #endif
>
> #ifdef OCIEROFS_ENABLED
> -
> -
> -/**
> - * mkfs_parse_oci_options - Parse comma-separated OCI options string
> +/*
> + * mkfs_parse_oci_options - Parse OCI options for mkfs tool
> + * @cfg: OCI configuration structure to update
> * @options_str: comma-separated options string
> - *
> - * Parse OCI options string containing comma-separated key=value pairs.
> - * Supported options include platform, layer, username, and password.
> - *
> - * Return: 0 on success, negative errno on failure
> */
> -static int mkfs_parse_oci_options(char *options_str)
> +static int mkfs_parse_oci_options(struct ocierofs_config *oci_cfg, char *options_str)
> {
> char *opt, *q, *p;
> - int ret;
>
> if (!options_str)
> return 0;
> @@ -717,41 +710,38 @@ static int mkfs_parse_oci_options(char *options_str)
> p = strstr(opt, "platform=");
> if (p) {
> p += strlen("platform=");
> - ret = erofs_oci_params_set_string(&ocicfg.params.platform, p);
> - if (ret) {
> - erofs_err("failed to set platform");
> - return ret;
> - }
> + free(oci_cfg->platform);
> + oci_cfg->platform = strdup(p);
> + if (!oci_cfg->platform)
> + return -ENOMEM;
> } else {
> p = strstr(opt, "layer=");
> if (p) {
> p += strlen("layer=");
> - ocicfg.params.layer_index = atoi(p);
> - if (ocicfg.params.layer_index < 0) {
> + oci_cfg->layer_index = atoi(p);
> + if (oci_cfg->layer_index < 0) {
Could you just use `strtoul` since it can check invalid
arguments better?
> erofs_err("invalid layer index %d",
> - ocicfg.params.layer_index);
> + oci_cfg->layer_index);
> return -EINVAL;
> }
> } else {
> p = strstr(opt, "username=");
> if (p) {
> p += strlen("username=");
> - ret = erofs_oci_params_set_string(&ocicfg.params.username, p);
> - if (ret) {
> - erofs_err("failed to set username");
> - return ret;
> - }
> + free(oci_cfg->username);
> + oci_cfg->username = strdup(p);
> + if (!oci_cfg->username)
> + return -ENOMEM;
> } else {
> p = strstr(opt, "password=");
> if (p) {
> p += strlen("password=");
> - ret = erofs_oci_params_set_string(&ocicfg.params.password, p);
> - if (ret) {
> - erofs_err("failed to set password");
> - return ret;
> - }
> + free(oci_cfg->password);
> + oci_cfg->password = strdup(p);
> + if (!oci_cfg->password)
> + return -ENOMEM;
> } else {
> - erofs_err("invalid --oci value %s", opt);
> + erofs_err("mkfs: invalid --oci value %s", opt);
`mkfs:` prefix is unneeded.
Thanks,
Gao Xiang
^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH v4 0/3] erofs-utils: refactor OCI and add NBD-backed OCI image mounting
2025-09-01 5:10 [PATCH v1] erofs-utils: refactor OCI code for better modularity ChengyuZhu6
` (3 preceding siblings ...)
2025-09-03 8:29 ` [PATCH v3 0/2] erofs-utils: refactor OCI and add NBD-backed OCI image mounting ChengyuZhu6
@ 2025-09-04 5:33 ` ChengyuZhu6
2025-09-04 5:33 ` [PATCH v4 1/3] erofs-utils:mkfs: move parse_oci_ref to lib ChengyuZhu6
` (2 more replies)
2025-09-04 6:36 ` [PATCH v5 0/3] erofs-utils: refactor OCI and " ChengyuZhu6
5 siblings, 3 replies; 19+ messages in thread
From: ChengyuZhu6 @ 2025-09-04 5:33 UTC (permalink / raw)
To: linux-erofs; +Cc: xiang, hsiangkao, Chengyu Zhu
From: Chengyu Zhu <hudsonzhu@tencent.com>
This series refactors the OCI handling in erofs-utils and adds NBD-backed
mounting of OCI images. It enables mounting EROFS container images directly
from registries without pre-downloading.
Chengyu Zhu (3):
erofs-utils:mkfs: move parse_oci_ref to lib
erofs-utils: refactor OCI code for better modularity
erofs-utils: add NBD-backed OCI image mounting
lib/liberofs_oci.h | 91 ++--
lib/remotes/oci.c | 1093 +++++++++++++++++++++++++++++---------------
mkfs/main.c | 201 ++------
mount/Makefile.am | 2 +-
mount/main.c | 257 +++++++++--
5 files changed, 1014 insertions(+), 630 deletions(-)
--
2.51.0
^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH v4 1/3] erofs-utils:mkfs: move parse_oci_ref to lib
2025-09-04 5:33 ` [PATCH v4 0/3] erofs-utils: refactor OCI and " ChengyuZhu6
@ 2025-09-04 5:33 ` ChengyuZhu6
2025-09-04 5:33 ` [PATCH v4 2/3] erofs-utils: refactor OCI code for better modularity ChengyuZhu6
2025-09-04 5:33 ` [PATCH v4 3/3] erofs-utils: add NBD-backed OCI image mounting ChengyuZhu6
2 siblings, 0 replies; 19+ messages in thread
From: ChengyuZhu6 @ 2025-09-04 5:33 UTC (permalink / raw)
To: linux-erofs; +Cc: xiang, hsiangkao, Chengyu Zhu
From: Chengyu Zhu <hudsonzhu@tencent.com>
move parse_oci_ref to oci.c.
Signed-off-by: Chengyu Zhu <hudsonzhu@tencent.com>
---
lib/liberofs_oci.h | 12 +++--
lib/remotes/oci.c | 110 +++++++++++++++++++++++++++++++++++++++++
mkfs/main.c | 121 +--------------------------------------------
3 files changed, 120 insertions(+), 123 deletions(-)
diff --git a/lib/liberofs_oci.h b/lib/liberofs_oci.h
index 3a8108b..bce63ef 100644
--- a/lib/liberofs_oci.h
+++ b/lib/liberofs_oci.h
@@ -8,9 +8,6 @@
#include <stdbool.h>
-#define DOCKER_REGISTRY "docker.io"
-#define DOCKER_API_REGISTRY "registry-1.docker.io"
-
#ifdef __cplusplus
extern "C" {
#endif
@@ -79,6 +76,15 @@ void ocierofs_cleanup(struct erofs_oci *oci);
*/
int erofs_oci_params_set_string(char **field, const char *value);
+/*
+ * ocierofs_parse_ref - Parse OCI image reference string
+ * @oci: OCI client structure
+ * @ref_str: OCI image reference string
+ *
+ * Return: 0 on success, negative errno on failure
+ */
+int ocierofs_parse_ref(struct erofs_oci *oci, const char *ref_str);
+
/*
* ocierofs_build_trees - Build file trees from OCI container image layers
* @root: root inode to build the file tree under
diff --git a/lib/remotes/oci.c b/lib/remotes/oci.c
index 0fb8c1f..e6f0c23 100644
--- a/lib/remotes/oci.c
+++ b/lib/remotes/oci.c
@@ -21,6 +21,9 @@
#include "liberofs_oci.h"
#include "liberofs_private.h"
+#define DOCKER_REGISTRY "docker.io"
+#define DOCKER_API_REGISTRY "registry-1.docker.io"
+
#define DOCKER_MEDIATYPE_MANIFEST_V2 \
"application/vnd.docker.distribution.manifest.v2+json"
#define DOCKER_MEDIATYPE_MANIFEST_V1 \
@@ -1066,3 +1069,110 @@ int erofs_oci_params_set_string(char **field, const char *value)
*field = new_value;
return 0;
}
+
+int ocierofs_parse_ref(struct erofs_oci *oci, const char *ref_str)
+{
+ char *slash, *colon, *dot;
+ const char *repo_part;
+ size_t len;
+
+ slash = strchr(ref_str, '/');
+ if (slash) {
+ dot = strchr(ref_str, '.');
+ if (dot && dot < slash) {
+ char *registry_str;
+
+ len = slash - ref_str;
+ registry_str = strndup(ref_str, len);
+
+ if (!registry_str) {
+ erofs_err("failed to allocate memory for registry");
+ return -ENOMEM;
+ }
+ if (erofs_oci_params_set_string(&oci->params.registry,
+ registry_str)) {
+ free(registry_str);
+ erofs_err("failed to set registry");
+ return -ENOMEM;
+ }
+ free(registry_str);
+ repo_part = slash + 1;
+ } else {
+ repo_part = ref_str;
+ }
+ } else {
+ repo_part = ref_str;
+ }
+
+ colon = strchr(repo_part, ':');
+ if (colon) {
+ char *repo_str;
+
+ len = colon - repo_part;
+ repo_str = strndup(repo_part, len);
+
+ if (!repo_str) {
+ erofs_err("failed to allocate memory for repository");
+ return -ENOMEM;
+ }
+
+ if (!strchr(repo_str, '/') &&
+ (!strcmp(oci->params.registry, DOCKER_API_REGISTRY) ||
+ !strcmp(oci->params.registry, DOCKER_REGISTRY))) {
+ char *full_repo;
+
+ if (asprintf(&full_repo, "library/%s", repo_str) == -1) {
+ free(repo_str);
+ erofs_err("failed to allocate memory for full repository name");
+ return -ENOMEM;
+ }
+ free(repo_str);
+ repo_str = full_repo;
+ }
+
+ if (erofs_oci_params_set_string(&oci->params.repository,
+ repo_str)) {
+ free(repo_str);
+ erofs_err("failed to set repository");
+ return -ENOMEM;
+ }
+ free(repo_str);
+
+ if (erofs_oci_params_set_string(&oci->params.tag,
+ colon + 1)) {
+ erofs_err("failed to set tag");
+ return -ENOMEM;
+ }
+ } else {
+ char *repo_str = strdup(repo_part);
+
+ if (!repo_str) {
+ erofs_err("failed to allocate memory for repository");
+ return -ENOMEM;
+ }
+
+ if (!strchr(repo_str, '/') &&
+ (!strcmp(oci->params.registry, DOCKER_API_REGISTRY) ||
+ !strcmp(oci->params.registry, DOCKER_REGISTRY))) {
+ char *full_repo;
+
+ if (asprintf(&full_repo, "library/%s", repo_str) == -1) {
+ free(repo_str);
+ erofs_err("failed to allocate memory for full repository name");
+ return -ENOMEM;
+ }
+ free(repo_str);
+ repo_str = full_repo;
+ }
+
+ if (erofs_oci_params_set_string(&oci->params.repository,
+ repo_str)) {
+ free(repo_str);
+ erofs_err("failed to set repository");
+ return -ENOMEM;
+ }
+ free(repo_str);
+ }
+
+ return 0;
+}
diff --git a/mkfs/main.c b/mkfs/main.c
index bc895f1..064392d 100644
--- a/mkfs/main.c
+++ b/mkfs/main.c
@@ -763,125 +763,6 @@ static int mkfs_parse_oci_options(char *options_str)
return 0;
}
-
-/**
- * mkfs_parse_oci_ref - Parse OCI image reference string
- * @ref_str: OCI image reference in various formats
- *
- * Parse OCI image reference which can be in formats:
- * - registry.example.com/namespace/repo:tag
- * - namespace/repo:tag (uses default registry)
- * - repo:tag (adds library/ prefix for Docker Hub)
- * - repo (uses default tag "latest")
- *
- * Return: 0 on success, negative errno on failure
- */
-static int mkfs_parse_oci_ref(const char *ref_str)
-{
- char *slash, *colon, *dot;
- const char *repo_part;
- size_t len;
-
- slash = strchr(ref_str, '/');
- if (slash) {
- dot = strchr(ref_str, '.');
- if (dot && dot < slash) {
- char *registry_str;
-
- len = slash - ref_str;
- registry_str = strndup(ref_str, len);
-
- if (!registry_str) {
- erofs_err("failed to allocate memory for registry");
- return -ENOMEM;
- }
- if (erofs_oci_params_set_string(&ocicfg.params.registry,
- registry_str)) {
- free(registry_str);
- erofs_err("failed to set registry");
- return -ENOMEM;
- }
- free(registry_str);
- repo_part = slash + 1;
- } else {
- repo_part = ref_str;
- }
- } else {
- repo_part = ref_str;
- }
-
- colon = strchr(repo_part, ':');
- if (colon) {
- char *repo_str;
-
- len = colon - repo_part;
- repo_str = strndup(repo_part, len);
-
- if (!repo_str) {
- erofs_err("failed to allocate memory for repository");
- return -ENOMEM;
- }
-
- if (!strchr(repo_str, '/') &&
- (!strcmp(ocicfg.params.registry, DOCKER_API_REGISTRY) ||
- !strcmp(ocicfg.params.registry, DOCKER_REGISTRY))) {
- char *full_repo;
-
- if (asprintf(&full_repo, "library/%s", repo_str) == -1) {
- free(repo_str);
- erofs_err("failed to allocate memory for full repository name");
- return -ENOMEM;
- }
- free(repo_str);
- repo_str = full_repo;
- }
-
- if (erofs_oci_params_set_string(&ocicfg.params.repository,
- repo_str)) {
- free(repo_str);
- erofs_err("failed to set repository");
- return -ENOMEM;
- }
- free(repo_str);
-
- if (erofs_oci_params_set_string(&ocicfg.params.tag,
- colon + 1)) {
- erofs_err("failed to set tag");
- return -ENOMEM;
- }
- } else {
- char *repo_str = strdup(repo_part);
-
- if (!repo_str) {
- erofs_err("failed to allocate memory for repository");
- return -ENOMEM;
- }
-
- if (!strchr(repo_str, '/') &&
- (!strcmp(ocicfg.params.registry, DOCKER_API_REGISTRY) ||
- !strcmp(ocicfg.params.registry, DOCKER_REGISTRY))) {
- char *full_repo;
-
- if (asprintf(&full_repo, "library/%s", repo_str) == -1) {
- free(repo_str);
- erofs_err("failed to allocate memory for full repository name");
- return -ENOMEM;
- }
- free(repo_str);
- repo_str = full_repo;
- }
-
- if (erofs_oci_params_set_string(&ocicfg.params.repository,
- repo_str)) {
- free(repo_str);
- erofs_err("failed to set repository");
- return -ENOMEM;
- }
- free(repo_str);
- }
-
- return 0;
-}
#endif
static int mkfs_parse_one_compress_alg(char *alg,
@@ -1958,7 +1839,7 @@ int main(int argc, char **argv)
err = mkfs_parse_oci_options(mkfs_oci_options);
if (err)
goto exit;
- err = mkfs_parse_oci_ref(cfg.c_src_path);
+ err = ocierofs_parse_ref(&ocicfg, cfg.c_src_path);
if (err)
goto exit;
--
2.51.0
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH v4 2/3] erofs-utils: refactor OCI code for better modularity
2025-09-04 5:33 ` [PATCH v4 0/3] erofs-utils: refactor OCI and " ChengyuZhu6
2025-09-04 5:33 ` [PATCH v4 1/3] erofs-utils:mkfs: move parse_oci_ref to lib ChengyuZhu6
@ 2025-09-04 5:33 ` ChengyuZhu6
2025-09-04 5:53 ` Gao Xiang
2025-09-04 5:33 ` [PATCH v4 3/3] erofs-utils: add NBD-backed OCI image mounting ChengyuZhu6
2 siblings, 1 reply; 19+ messages in thread
From: ChengyuZhu6 @ 2025-09-04 5:33 UTC (permalink / raw)
To: linux-erofs; +Cc: xiang, hsiangkao, Chengyu Zhu
From: Chengyu Zhu <hudsonzhu@tencent.com>
Refactor OCI code to improve code organization and maintainability:
Key changes:
- Add `ocierofs_get_api_registry()` and unify API endpoint selection.
- Implement Bearer token discovery with Basic fallback; cache auth header.
- Parse layer metadata (digest, mediaType, size) and add a proper free helper.
- Split blob download from tar processing; process tar via a temp fd.
- Rework init/teardown into `ocierofs_init()` and `ocierofs_ctx_cleanup()`.
- Update mkfs to use `struct ocierofs_config` and new `--oci` parsing.
Signed-off-by: Chengyu Zhu <hudsonzhu@tencent.com>
---
lib/liberofs_oci.h | 87 ++---
lib/remotes/oci.c | 956 ++++++++++++++++++++++-----------------------
mkfs/main.c | 82 ++--
3 files changed, 537 insertions(+), 588 deletions(-)
diff --git a/lib/liberofs_oci.h b/lib/liberofs_oci.h
index bce63ef..8622464 100644
--- a/lib/liberofs_oci.h
+++ b/lib/liberofs_oci.h
@@ -12,87 +12,56 @@
extern "C" {
#endif
-struct erofs_inode;
-struct CURL;
struct erofs_importer;
-/**
- * struct erofs_oci_params - OCI configuration parameters
- * @registry: registry hostname (e.g., "registry-1.docker.io")
- * @repository: image repository (e.g., "library/ubuntu")
- * @tag: image tag or digest (e.g., "latest" or sha256:...)
+/*
+ * struct ocierofs_config - OCI configuration structure
+ * @image_ref: OCI image reference (e.g., "ubuntu:latest", "myregistry.com/app:v1.0")
* @platform: target platform in "os/arch" format (e.g., "linux/amd64")
* @username: username for authentication (optional)
* @password: password for authentication (optional)
* @layer_index: specific layer to extract (-1 for all layers)
*
- * Configuration structure for OCI image parameters including registry
- * location, image identification, platform specification, and authentication
- * credentials.
*/
-struct erofs_oci_params {
- char *registry;
- char *repository;
- char *tag;
+struct ocierofs_config {
+ char *image_ref;
char *platform;
char *username;
char *password;
int layer_index;
};
-/**
- * struct erofs_oci - Combined OCI client structure
- * @curl: CURL handle for HTTP requests
- * @params: OCI configuration parameters
- *
- * Main OCI client structure combining CURL HTTP client with
- * OCI-specific configuration parameters.
- */
-struct erofs_oci {
- struct CURL *curl;
- struct erofs_oci_params params;
+struct ocierofs_layer_info {
+ char *digest;
+ char *media_type;
+ u64 size;
};
-/*
- * ocierofs_init - Initialize OCI client with default parameters
- * @oci: OCI client structure to initialize
- *
- * Return: 0 on success, negative errno on failure
- */
-int ocierofs_init(struct erofs_oci *oci);
-
-/*
- * ocierofs_cleanup - Clean up OCI client and free allocated resources
- * @oci: OCI client structure to clean up
- */
-void ocierofs_cleanup(struct erofs_oci *oci);
-
-/*
- * erofs_oci_params_set_string - Set a string field with dynamic allocation
- * @field: pointer to the string field to set
- * @value: string value to set
- *
- * Return: 0 on success, negative errno on failure
- */
-int erofs_oci_params_set_string(char **field, const char *value);
+struct ocierofs_ctx {
+ struct CURL *curl;
+ char *auth_header;
+ bool using_basic;
+ char *registry;
+ char *repository;
+ char *platform;
+ char *tag;
+ char *manifest_digest;
+ struct ocierofs_layer_info **layers;
+ int layer_index;
+ int layer_count;
+};
-/*
- * ocierofs_parse_ref - Parse OCI image reference string
- * @oci: OCI client structure
- * @ref_str: OCI image reference string
- *
- * Return: 0 on success, negative errno on failure
- */
-int ocierofs_parse_ref(struct erofs_oci *oci, const char *ref_str);
+int ocierofs_init(struct ocierofs_ctx *ctx, const struct ocierofs_config *config);
/*
- * ocierofs_build_trees - Build file trees from OCI container image layers
- * @root: root inode to build the file tree under
- * @oci: OCI client structure with configured parameters
+ * ocierofs_build_trees - Build file trees from an OCI container image
+ * @importer: erofs importer to populate
+ * @cfg: oci configuration
*
* Return: 0 on success, negative errno on failure
*/
-int ocierofs_build_trees(struct erofs_importer *importer, struct erofs_oci *oci);
+int ocierofs_build_trees(struct erofs_importer *importer,
+ const struct ocierofs_config *cfg);
#ifdef __cplusplus
}
diff --git a/lib/remotes/oci.c b/lib/remotes/oci.c
index e6f0c23..4291dc6 100644
--- a/lib/remotes/oci.c
+++ b/lib/remotes/oci.c
@@ -33,28 +33,67 @@
#define OCI_MEDIATYPE_MANIFEST "application/vnd.oci.image.manifest.v1+json"
#define OCI_MEDIATYPE_INDEX "application/vnd.oci.image.index.v1+json"
-struct erofs_oci_request {
+struct ocierofs_request {
char *url;
struct curl_slist *headers;
};
-struct erofs_oci_response {
+struct ocierofs_response {
char *data;
size_t size;
long http_code;
};
-struct erofs_oci_stream {
- struct erofs_tarfile tarfile;
+struct ocierofs_stream {
const char *digest;
int blobfd;
};
+static inline const char *ocierofs_get_api_registry(const char *registry)
+{
+ if (!registry)
+ return DOCKER_API_REGISTRY;
+ return !strcmp(registry, DOCKER_REGISTRY) ? DOCKER_API_REGISTRY : registry;
+}
+
+static inline bool ocierofs_is_manifest_list(const char *media_type)
+{
+ return media_type && (!strcmp(media_type, DOCKER_MEDIATYPE_MANIFEST_LIST) ||
+ !strcmp(media_type, OCI_MEDIATYPE_INDEX));
+}
+
+static inline bool ocierofs_is_manifest(const char *media_type)
+{
+ return media_type && (!strcmp(media_type, DOCKER_MEDIATYPE_MANIFEST_V2) ||
+ !strcmp(media_type, OCI_MEDIATYPE_MANIFEST));
+}
+
+static inline void ocierofs_request_cleanup(struct ocierofs_request *req)
+{
+ if (!req)
+ return;
+ if (req->headers)
+ curl_slist_free_all(req->headers);
+ free(req->url);
+ req->url = NULL;
+ req->headers = NULL;
+}
+
+static inline void ocierofs_response_cleanup(struct ocierofs_response *resp)
+{
+ if (!resp)
+ return;
+ free(resp->data);
+ resp->data = NULL;
+ resp->size = 0;
+ resp->http_code = 0;
+}
+
static size_t ocierofs_write_callback(void *contents, size_t size,
size_t nmemb, void *userp)
{
size_t realsize = size * nmemb;
- struct erofs_oci_response *resp = userp;
+ struct ocierofs_response *resp = userp;
char *ptr;
if (!resp->data)
@@ -75,16 +114,23 @@ static size_t ocierofs_write_callback(void *contents, size_t size,
static size_t ocierofs_layer_write_callback(void *contents, size_t size,
size_t nmemb, void *userp)
{
- struct erofs_oci_stream *stream = userp;
+ struct ocierofs_stream *stream = userp;
size_t realsize = size * nmemb;
+ const char *buf = contents;
+ size_t written = 0;
if (stream->blobfd < 0)
return 0;
- if (write(stream->blobfd, contents, realsize) != realsize) {
- erofs_err("failed to write layer data for layer %s",
- stream->digest);
- return 0;
+ while (written < realsize) {
+ ssize_t n = write(stream->blobfd, buf + written, realsize - written);
+
+ if (n < 0) {
+ erofs_err("failed to write layer data for layer %s",
+ stream->digest);
+ return 0;
+ }
+ written += n;
}
return realsize;
}
@@ -96,6 +142,14 @@ static int ocierofs_curl_setup_common_options(struct CURL *curl)
curl_easy_setopt(curl, CURLOPT_TIMEOUT, 120L);
curl_easy_setopt(curl, CURLOPT_NOSIGNAL, 1L);
curl_easy_setopt(curl, CURLOPT_USERAGENT, "ocierofs/" PACKAGE_VERSION);
+ curl_easy_setopt(curl, CURLOPT_ACCEPT_ENCODING, "");
+ curl_easy_setopt(curl, CURLOPT_TCP_KEEPALIVE, 1L);
+#if defined(CURLOPT_TCP_KEEPIDLE)
+ curl_easy_setopt(curl, CURLOPT_TCP_KEEPIDLE, 30L);
+#endif
+#if defined(CURLOPT_TCP_KEEPINTVL)
+ curl_easy_setopt(curl, CURLOPT_TCP_KEEPINTVL, 15L);
+#endif
return 0;
}
@@ -114,10 +168,10 @@ static int ocierofs_curl_setup_basic_auth(struct CURL *curl, const char *usernam
return 0;
}
-static int ocierofs_curl_clear_auth(struct CURL *curl)
+static int ocierofs_curl_clear_auth(struct ocierofs_ctx *ctx)
{
- curl_easy_setopt(curl, CURLOPT_USERPWD, NULL);
- curl_easy_setopt(curl, CURLOPT_HTTPAUTH, CURLAUTH_NONE);
+ curl_easy_setopt(ctx->curl, CURLOPT_USERPWD, NULL);
+ curl_easy_setopt(ctx->curl, CURLOPT_HTTPAUTH, CURLAUTH_NONE);
return 0;
}
@@ -176,20 +230,20 @@ static int ocierofs_curl_perform(struct CURL *curl, long *http_code_out)
return 0;
}
-static int ocierofs_request_perform(struct erofs_oci *oci,
- struct erofs_oci_request *req,
- struct erofs_oci_response *resp)
+static int ocierofs_request_perform(struct ocierofs_ctx *ctx,
+ struct ocierofs_request *req,
+ struct ocierofs_response *resp)
{
int ret;
- ret = ocierofs_curl_setup_rq(oci->curl, req->url,
+ ret = ocierofs_curl_setup_rq(ctx->curl, req->url,
OCIEROFS_HTTP_GET, req->headers,
- ocierofs_write_callback, resp,
+ ocierofs_write_callback, resp,
NULL, NULL);
if (ret)
return ret;
- ret = ocierofs_curl_perform(oci->curl, &resp->http_code);
+ ret = ocierofs_curl_perform(ctx->curl, &resp->http_code);
if (ret)
return ret;
@@ -204,15 +258,10 @@ static int ocierofs_request_perform(struct erofs_oci *oci,
* @realm_out: pointer to store realm value
* @service_out: pointer to store service value
* @scope_out: pointer to store scope value
- *
- * Parse Bearer authentication header and extract realm, service, and scope
- * parameters for subsequent token requests.
- *
- * Return: 0 on success, negative errno on failure
*/
static int ocierofs_parse_auth_header(const char *auth_header,
- char **realm_out, char **service_out,
- char **scope_out)
+ char **realm_out, char **service_out,
+ char **scope_out)
{
char *realm = NULL, *service = NULL, *scope = NULL;
static const char * const param_names[] = {"realm=", "service=", "scope="};
@@ -221,7 +270,6 @@ static int ocierofs_parse_auth_header(const char *auth_header,
const char *p;
int i, ret = 0;
- // https://datatracker.ietf.org/doc/html/rfc6750#section-3
if (strncmp(auth_header, "Bearer ", strlen("Bearer ")))
return -EINVAL;
@@ -229,7 +277,6 @@ static int ocierofs_parse_auth_header(const char *auth_header,
if (!header_copy)
return -ENOMEM;
- /* Clean up header: replace newlines with spaces and remove double spaces */
for (char *q = header_copy; *q; q++) {
if (*q == '\n' || *q == '\r')
*q = ' ';
@@ -277,22 +324,9 @@ out:
return ret;
}
-/**
- * ocierofs_extract_www_auth_info - Extract WWW-Authenticate header information
- * @resp_data: HTTP response data containing headers
- * @realm_out: pointer to store realm value (optional)
- * @service_out: pointer to store service value (optional)
- * @scope_out: pointer to store scope value (optional)
- *
- * Extract realm, service, and scope from WWW-Authenticate header in HTTP response.
- * This function handles the common pattern of parsing WWW-Authenticate headers
- * that appears in multiple places in the OCI authentication flow.
- *
- * Return: 0 on success, negative errno on failure
- */
static int ocierofs_extract_www_auth_info(const char *resp_data,
- char **realm_out, char **service_out,
- char **scope_out)
+ char **realm_out, char **service_out,
+ char **scope_out)
{
char *www_auth;
char *line_end;
@@ -336,29 +370,12 @@ static int ocierofs_extract_www_auth_info(const char *resp_data,
return ret;
}
-/**
- * ocierofs_get_auth_token_with_url - Get authentication token from auth server
- * @oci: OCI client structure
- * @auth_url: authentication server URL
- * @service: service name for authentication
- * @repository: repository name
- * @username: username for basic auth (optional)
- * @password: password for basic auth (optional)
- *
- * Request authentication token from the specified auth server URL using
- * basic authentication if credentials are provided.
- *
- * Return: authentication header string on success, ERR_PTR on failure
- */
-static char *ocierofs_get_auth_token_with_url(struct erofs_oci *oci,
- const char *auth_url,
- const char *service,
- const char *repository,
- const char *username,
- const char *password)
+static char *ocierofs_get_auth_token_with_url(struct ocierofs_ctx *ctx, const char *auth_url,
+ const char *service, const char *repository,
+ const char *username, const char *password)
{
- struct erofs_oci_request req = {};
- struct erofs_oci_response resp = {};
+ struct ocierofs_request req = {};
+ struct ocierofs_response resp = {};
json_object *root, *token_obj, *access_token_obj;
const char *token;
char *auth_header = NULL;
@@ -373,40 +390,36 @@ static char *ocierofs_get_auth_token_with_url(struct erofs_oci *oci,
}
if (username && password && *username) {
- ret = ocierofs_curl_setup_basic_auth(oci->curl, username,
- password);
+ ret = ocierofs_curl_setup_basic_auth(ctx->curl, username,
+ password);
if (ret)
goto out_url;
}
- ret = ocierofs_request_perform(oci, &req, &resp);
- ocierofs_curl_clear_auth(oci->curl);
+ ret = ocierofs_request_perform(ctx, &req, &resp);
+ ocierofs_curl_clear_auth(ctx);
if (ret)
goto out_url;
if (!resp.data) {
- erofs_err("empty response from auth server");
ret = -EINVAL;
goto out_url;
}
root = json_tokener_parse(resp.data);
if (!root) {
- erofs_err("failed to parse auth response");
ret = -EINVAL;
- goto out_url;
+ goto out_json;
}
if (!json_object_object_get_ex(root, "token", &token_obj) &&
!json_object_object_get_ex(root, "access_token", &access_token_obj)) {
- erofs_err("no token found in auth response");
ret = -EINVAL;
goto out_json;
}
token = json_object_get_string(token_obj ? token_obj : access_token_obj);
if (!token) {
- erofs_err("invalid token in auth response");
ret = -EINVAL;
goto out_json;
}
@@ -419,16 +432,15 @@ static char *ocierofs_get_auth_token_with_url(struct erofs_oci *oci,
out_json:
json_object_put(root);
out_url:
- free(req.url);
- free(resp.data);
+ ocierofs_response_cleanup(&resp);
+ ocierofs_request_cleanup(&req);
return ret ? ERR_PTR(ret) : auth_header;
}
-static char *ocierofs_discover_auth_endpoint(struct erofs_oci *oci,
- const char *registry,
+static char *ocierofs_discover_auth_endpoint(struct ocierofs_ctx *ctx, const char *registry,
const char *repository)
{
- struct erofs_oci_response resp = {};
+ struct ocierofs_response resp = {};
char *realm = NULL;
char *service = NULL;
char *result = NULL;
@@ -437,20 +449,20 @@ static char *ocierofs_discover_auth_endpoint(struct erofs_oci *oci,
CURLcode res;
long http_code;
- api_registry = (!strcmp(registry, DOCKER_REGISTRY)) ? DOCKER_API_REGISTRY : registry;
+ api_registry = ocierofs_get_api_registry(registry);
if (asprintf(&test_url, "https://%s/v2/%s/manifests/nonexistent",
api_registry, repository) < 0)
return NULL;
- curl_easy_reset(oci->curl);
- ocierofs_curl_setup_common_options(oci->curl);
+ curl_easy_reset(ctx->curl);
+ ocierofs_curl_setup_common_options(ctx->curl);
- ocierofs_curl_setup_rq(oci->curl, test_url, OCIEROFS_HTTP_HEAD, NULL,
+ ocierofs_curl_setup_rq(ctx->curl, test_url, OCIEROFS_HTTP_HEAD, NULL,
NULL, NULL, ocierofs_write_callback, &resp);
- res = curl_easy_perform(oci->curl);
- curl_easy_getinfo(oci->curl, CURLINFO_RESPONSE_CODE, &http_code);
+ res = curl_easy_perform(ctx->curl);
+ curl_easy_getinfo(ctx->curl, CURLINFO_RESPONSE_CODE, &http_code);
if (res == CURLE_OK && (http_code == 401 || http_code == 403 ||
http_code == 404) && resp.data) {
@@ -461,14 +473,13 @@ static char *ocierofs_discover_auth_endpoint(struct erofs_oci *oci,
}
free(realm);
free(service);
- free(resp.data);
+ ocierofs_response_cleanup(&resp);
free(test_url);
return result;
}
-static char *ocierofs_get_auth_token(struct erofs_oci *oci, const char *registry,
- const char *repository, const char *username,
- const char *password)
+static char *ocierofs_get_auth_token(struct ocierofs_ctx *ctx, const char *registry,
+ const char *repository, const char *username, const char *password)
{
static const char * const auth_patterns[] = {
"https://%s/v2/auth",
@@ -487,35 +498,35 @@ static char *ocierofs_get_auth_token(struct erofs_oci *oci, const char *registry
!strcmp(registry, DOCKER_REGISTRY);
if (docker_reg) {
service = "registry.docker.io";
- auth_header = ocierofs_get_auth_token_with_url(oci,
+ auth_header = ocierofs_get_auth_token_with_url(ctx,
"https://auth.docker.io/token", service, repository,
username, password);
if (!IS_ERR(auth_header))
return auth_header;
}
- discovered_auth_url = ocierofs_discover_auth_endpoint(oci, registry, repository);
+ discovered_auth_url = ocierofs_discover_auth_endpoint(ctx, registry, repository);
if (discovered_auth_url) {
const char *api_registry, *auth_service;
- struct erofs_oci_response resp = {};
+ struct ocierofs_response resp = {};
char *test_url;
CURLcode res;
long http_code;
- api_registry = (!strcmp(registry, DOCKER_REGISTRY)) ? DOCKER_API_REGISTRY : registry;
+ api_registry = ocierofs_get_api_registry(registry);
if (asprintf(&test_url, "https://%s/v2/%s/manifests/nonexistent",
api_registry, repository) >= 0) {
- curl_easy_reset(oci->curl);
- ocierofs_curl_setup_common_options(oci->curl);
+ curl_easy_reset(ctx->curl);
+ ocierofs_curl_setup_common_options(ctx->curl);
- ocierofs_curl_setup_rq(oci->curl, test_url,
- OCIEROFS_HTTP_HEAD, NULL,
- NULL, NULL,
- ocierofs_write_callback, &resp);
+ ocierofs_curl_setup_rq(ctx->curl, test_url,
+ OCIEROFS_HTTP_HEAD, NULL,
+ NULL, NULL,
+ ocierofs_write_callback, &resp);
- res = curl_easy_perform(oci->curl);
- curl_easy_getinfo(oci->curl, CURLINFO_RESPONSE_CODE, &http_code);
+ res = curl_easy_perform(ctx->curl);
+ curl_easy_getinfo(ctx->curl, CURLINFO_RESPONSE_CODE, &http_code);
if (res == CURLE_OK && (http_code == 401 || http_code == 403 ||
http_code == 404) && resp.data) {
@@ -524,14 +535,14 @@ static char *ocierofs_get_auth_token(struct erofs_oci *oci, const char *registry
ocierofs_extract_www_auth_info(resp.data, &realm, &discovered_service, NULL);
free(realm);
}
- free(resp.data);
+ ocierofs_response_cleanup(&resp);
free(test_url);
}
auth_service = discovered_service ? discovered_service : service;
- auth_header = ocierofs_get_auth_token_with_url(oci, discovered_auth_url,
- auth_service, repository,
- username, password);
+ auth_header = ocierofs_get_auth_token_with_url(ctx, discovered_auth_url,
+ auth_service, repository,
+ username, password);
free(discovered_auth_url);
free(discovered_service);
if (!IS_ERR(auth_header))
@@ -544,9 +555,9 @@ static char *ocierofs_get_auth_token(struct erofs_oci *oci, const char *registry
if (asprintf(&auth_url, auth_patterns[i], registry) < 0)
continue;
- auth_header = ocierofs_get_auth_token_with_url(oci, auth_url,
- service, repository,
- username, password);
+ auth_header = ocierofs_get_auth_token_with_url(ctx, auth_url,
+ service, repository,
+ username, password);
free(auth_url);
if (!IS_ERR(auth_header))
@@ -557,24 +568,21 @@ static char *ocierofs_get_auth_token(struct erofs_oci *oci, const char *registry
return ERR_PTR(-ENOENT);
}
-static char *ocierofs_get_manifest_digest(struct erofs_oci *oci,
- const char *registry,
- const char *repository, const char *tag,
- const char *platform,
- const char *auth_header)
+static char *ocierofs_get_manifest_digest(struct ocierofs_ctx *ctx,
+ const char *registry,
+ const char *repository, const char *tag,
+ const char *platform,
+ const char *auth_header)
{
- struct erofs_oci_request req = {};
- struct erofs_oci_response resp = {};
+ struct ocierofs_request req = {};
+ struct ocierofs_response resp = {};
json_object *root, *manifests, *manifest, *platform_obj, *arch_obj;
json_object *os_obj, *digest_obj, *schema_obj, *media_type_obj;
char *digest = NULL;
const char *api_registry;
int ret = 0, len, i;
- if (!registry || !repository || !tag || !platform)
- return ERR_PTR(-EINVAL);
-
- api_registry = (!strcmp(registry, DOCKER_REGISTRY)) ? DOCKER_API_REGISTRY : registry;
+ api_registry = ocierofs_get_api_registry(registry);
if (asprintf(&req.url, "https://%s/v2/%s/manifests/%s",
api_registry, repository, tag) < 0)
return ERR_PTR(-ENOMEM);
@@ -584,22 +592,20 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci,
req.headers = curl_slist_append(req.headers,
"Accept: " DOCKER_MEDIATYPE_MANIFEST_LIST ","
- OCI_MEDIATYPE_INDEX "," DOCKER_MEDIATYPE_MANIFEST_V1 ","
- DOCKER_MEDIATYPE_MANIFEST_V2);
+ OCI_MEDIATYPE_INDEX "," OCI_MEDIATYPE_MANIFEST ","
+ DOCKER_MEDIATYPE_MANIFEST_V1 "," DOCKER_MEDIATYPE_MANIFEST_V2);
- ret = ocierofs_request_perform(oci, &req, &resp);
+ ret = ocierofs_request_perform(ctx, &req, &resp);
if (ret)
goto out;
if (!resp.data) {
- erofs_err("empty response from manifest request");
ret = -EINVAL;
goto out;
}
root = json_tokener_parse(resp.data);
if (!root) {
- erofs_err("failed to parse manifest JSON");
ret = -EINVAL;
goto out;
}
@@ -615,8 +621,7 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci,
if (json_object_object_get_ex(root, "mediaType", &media_type_obj)) {
const char *media_type = json_object_get_string(media_type_obj);
- if (!strcmp(media_type, DOCKER_MEDIATYPE_MANIFEST_V2) ||
- !strcmp(media_type, OCI_MEDIATYPE_MANIFEST)) {
+ if (ocierofs_is_manifest(media_type)) {
digest = strdup(tag);
ret = 0;
goto out_json;
@@ -624,7 +629,6 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci,
}
if (!json_object_object_get_ex(root, "manifests", &manifests)) {
- erofs_err("no manifests found in manifest list");
ret = -EINVAL;
goto out_json;
}
@@ -634,9 +638,9 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci,
manifest = json_object_array_get_idx(manifests, i);
if (json_object_object_get_ex(manifest, "platform",
- &platform_obj) &&
+ &platform_obj) &&
json_object_object_get_ex(platform_obj, "architecture",
- &arch_obj) &&
+ &arch_obj) &&
json_object_object_get_ex(platform_obj, "os", &os_obj) &&
json_object_object_get_ex(manifest, "digest", &digest_obj)) {
const char *arch = json_object_get_string(arch_obj);
@@ -658,34 +662,44 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci,
out_json:
json_object_put(root);
out:
- free(resp.data);
- if (req.headers)
- curl_slist_free_all(req.headers);
- free(req.url);
-
+ ocierofs_response_cleanup(&resp);
+ ocierofs_request_cleanup(&req);
return ret ? ERR_PTR(ret) : digest;
}
-static char **ocierofs_get_layers_info(struct erofs_oci *oci,
- const char *registry,
- const char *repository,
- const char *digest,
- const char *auth_header,
- int *layer_count)
+static void ocierofs_free_layers_info(struct ocierofs_layer_info **layers, int count)
{
- struct erofs_oci_request req = {};
- struct erofs_oci_response resp = {};
- json_object *root, *layers, *layer, *digest_obj;
- char **layers_info = NULL;
- const char *api_registry;
- int ret, len, i, j;
+ int i;
- if (!registry || !repository || !digest || !layer_count)
- return ERR_PTR(-EINVAL);
+ if (!layers)
+ return;
+
+ for (i = 0; i < count; i++) {
+ if (layers[i]) {
+ free(layers[i]->digest);
+ free(layers[i]->media_type);
+ free(layers[i]);
+ }
+ }
+ free(layers);
+}
+
+static struct ocierofs_layer_info **ocierofs_fetch_layers_info(struct ocierofs_ctx *ctx,
+ const char *registry,
+ const char *repository,
+ const char *digest,
+ const char *auth_header,
+ int *layer_count)
+{
+ struct ocierofs_request req = {};
+ struct ocierofs_response resp = {};
+ json_object *root, *layers, *layer, *digest_obj, *media_type_obj, *size_obj;
+ struct ocierofs_layer_info **layers_info = NULL;
+ const char *api_registry;
+ int ret, len, i;
*layer_count = 0;
- api_registry = (!strcmp(registry, DOCKER_REGISTRY) ?
- DOCKER_API_REGISTRY : registry);
+ api_registry = ocierofs_get_api_registry(registry);
if (asprintf(&req.url, "https://%s/v2/%s/manifests/%s",
api_registry, repository, digest) < 0)
@@ -697,38 +711,34 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci,
req.headers = curl_slist_append(req.headers,
"Accept: " OCI_MEDIATYPE_MANIFEST "," DOCKER_MEDIATYPE_MANIFEST_V2);
- ret = ocierofs_request_perform(oci, &req, &resp);
+ ret = ocierofs_request_perform(ctx, &req, &resp);
if (ret)
goto out;
if (!resp.data) {
- erofs_err("empty response from layers request");
ret = -EINVAL;
goto out;
}
root = json_tokener_parse(resp.data);
if (!root) {
- erofs_err("failed to parse manifest JSON");
ret = -EINVAL;
goto out;
}
if (!json_object_object_get_ex(root, "layers", &layers) ||
json_object_get_type(layers) != json_type_array) {
- erofs_err("no layers found in manifest");
ret = -EINVAL;
goto out_json;
}
len = json_object_array_length(layers);
if (!len) {
- erofs_err("empty layer list in manifest");
ret = -EINVAL;
goto out_json;
}
- layers_info = calloc(len, sizeof(char *));
+ layers_info = calloc(len, sizeof(*layers_info));
if (!layers_info) {
ret = -ENOMEM;
goto out_json;
@@ -738,364 +748,188 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci,
layer = json_object_array_get_idx(layers, i);
if (!json_object_object_get_ex(layer, "digest", &digest_obj)) {
- erofs_err("failed to parse layer %d", i);
ret = -EINVAL;
goto out_free;
}
- layers_info[i] = strdup(json_object_get_string(digest_obj));
+ layers_info[i] = calloc(1, sizeof(**layers_info));
if (!layers_info[i]) {
ret = -ENOMEM;
goto out_free;
}
+ layers_info[i]->digest = strdup(json_object_get_string(digest_obj));
+ if (!layers_info[i]->digest) {
+ ret = -ENOMEM;
+ goto out_free;
+ }
+ if (json_object_object_get_ex(layer, "mediaType", &media_type_obj))
+ layers_info[i]->media_type = strdup(json_object_get_string(media_type_obj));
+ else
+ layers_info[i]->media_type = NULL;
+
+ if (json_object_object_get_ex(layer, "size", &size_obj))
+ layers_info[i]->size = json_object_get_int64(size_obj);
+ else
+ layers_info[i]->size = 0;
}
*layer_count = len;
json_object_put(root);
- free(resp.data);
- if (req.headers)
- curl_slist_free_all(req.headers);
- free(req.url);
+ ocierofs_response_cleanup(&resp);
+ ocierofs_request_cleanup(&req);
return layers_info;
out_free:
- if (layers_info) {
- for (j = 0; j < i; j++)
- free(layers_info[j]);
- }
- free(layers_info);
+ ocierofs_free_layers_info(layers_info, i);
out_json:
json_object_put(root);
out:
- free(resp.data);
- if (req.headers)
- curl_slist_free_all(req.headers);
- free(req.url);
+ ocierofs_response_cleanup(&resp);
+ ocierofs_request_cleanup(&req);
return ERR_PTR(ret);
}
-static int ocierofs_extract_layer(struct erofs_oci *oci, struct erofs_importer *importer,
- const char *digest, const char *auth_header)
+static int ocierofs_process_tar_stream(struct erofs_importer *importer, int fd)
{
- struct erofs_oci_request req = {};
- struct erofs_oci_stream stream = {};
- const char *api_registry;
- long http_code;
+ struct erofs_tarfile tarfile = {};
int ret;
- stream = (struct erofs_oci_stream) {
- .digest = digest,
- .blobfd = erofs_tmpfile(),
- };
- if (stream.blobfd < 0) {
- erofs_err("failed to create temporary file for %s", digest);
- return -errno;
- }
-
- api_registry = (!strcmp(oci->params.registry, DOCKER_REGISTRY)) ?
- DOCKER_API_REGISTRY : oci->params.registry;
- if (asprintf(&req.url, "https://%s/v2/%s/blobs/%s",
- api_registry, oci->params.repository, digest) == -1) {
- ret = -ENOMEM;
- goto out;
- }
-
- if (auth_header && strstr(auth_header, "Bearer"))
- req.headers = curl_slist_append(req.headers, auth_header);
-
- curl_easy_reset(oci->curl);
-
- ret = ocierofs_curl_setup_common_options(oci->curl);
- if (ret)
- goto out;
-
- ret = ocierofs_curl_setup_rq(oci->curl, req.url, OCIEROFS_HTTP_GET,
- req.headers,
- ocierofs_layer_write_callback,
- &stream, NULL, NULL);
- if (ret)
- goto out;
-
- ret = ocierofs_curl_perform(oci->curl, &http_code);
- if (ret)
- goto out;
-
- if (http_code < 200 || http_code >= 300) {
- erofs_err("HTTP request failed with code %ld", http_code);
- ret = -EIO;
- goto out;
- }
+ init_list_head(&tarfile.global.xattrs);
- if (lseek(stream.blobfd, 0, SEEK_SET) < 0) {
- erofs_err("failed to seek to beginning of temp file: %s",
- strerror(errno));
- ret = -errno;
- goto out;
- }
-
- memset(&stream.tarfile, 0, sizeof(stream.tarfile));
- init_list_head(&stream.tarfile.global.xattrs);
-
- ret = erofs_iostream_open(&stream.tarfile.ios, stream.blobfd,
- EROFS_IOS_DECODER_GZIP);
+ ret = erofs_iostream_open(&tarfile.ios, fd, EROFS_IOS_DECODER_GZIP);
if (ret) {
erofs_err("failed to initialize tar stream: %s",
erofs_strerror(ret));
- goto out;
+ return ret;
}
do {
- ret = tarerofs_parse_tar(importer, &stream.tarfile);
+ ret = tarerofs_parse_tar(importer, &tarfile);
/* Continue parsing until end of archive */
} while (!ret);
- erofs_iostream_close(&stream.tarfile.ios);
+ erofs_iostream_close(&tarfile.ios);
if (ret < 0 && ret != -ENODATA) {
erofs_err("failed to process tar stream: %s",
erofs_strerror(ret));
- goto out;
+ return ret;
}
- ret = 0;
-out:
- if (stream.blobfd >= 0)
- close(stream.blobfd);
- if (req.headers)
- curl_slist_free_all(req.headers);
- free(req.url);
- return ret;
+ return 0;
}
-/**
- * ocierofs_build_trees - Build file trees from OCI container image layers
- * @importer: EROFS importer structure
- * @oci: OCI client structure with configured parameters
- *
- * Extract and build file system trees from all layers of an OCI container
- * image.
- *
- * Return: 0 on success, negative errno on failure
- */
-int ocierofs_build_trees(struct erofs_importer *importer, struct erofs_oci *oci)
+static int ocierofs_prepare_auth(struct ocierofs_ctx *ctx,
+ const char *username,
+ const char *password)
{
char *auth_header = NULL;
- char *manifest_digest = NULL;
- char **layers = NULL;
- int layer_count = 0;
- int ret, i;
-
- if (!importer || !oci)
- return -EINVAL;
+ int ret = 0;
+
+ ctx->using_basic = false;
+ free(ctx->auth_header);
+ ctx->auth_header = NULL;
+
+ auth_header = ocierofs_get_auth_token(ctx,
+ ctx->registry,
+ ctx->repository,
+ username, password);
+ if (!IS_ERR(auth_header)) {
+ ctx->auth_header = auth_header;
+ return 0;
+ }
- if (oci->params.username && oci->params.password &&
- oci->params.username[0] && oci->params.password[0]) {
- auth_header = ocierofs_get_auth_token(oci,
- oci->params.registry,
- oci->params.repository,
- oci->params.username,
- oci->params.password);
- if (IS_ERR(auth_header)) {
- auth_header = NULL;
- ret = ocierofs_curl_setup_basic_auth(oci->curl,
- oci->params.username,
- oci->params.password);
- if (ret)
- goto out;
- }
- } else {
- auth_header = ocierofs_get_auth_token(oci,
- oci->params.registry,
- oci->params.repository,
- NULL, NULL);
- if (IS_ERR(auth_header))
- auth_header = NULL;
+ if (username && password && *username && *password) {
+ ret = ocierofs_curl_setup_basic_auth(ctx->curl,
+ username, password);
+ if (ret)
+ return ret;
+ ctx->using_basic = true;
}
+ return 0;
+}
- manifest_digest = ocierofs_get_manifest_digest(oci, oci->params.registry,
- oci->params.repository,
- oci->params.tag,
- oci->params.platform,
- auth_header);
- if (IS_ERR(manifest_digest)) {
- ret = PTR_ERR(manifest_digest);
+static int ocierofs_prepare_layers(struct ocierofs_ctx *ctx,
+ const struct ocierofs_config *config)
+{
+ int ret;
+
+ ret = ocierofs_prepare_auth(ctx, config ? config->username : NULL,
+ config ? config->password : NULL);
+ if (ret)
+ return ret;
+
+ ctx->manifest_digest = ocierofs_get_manifest_digest(ctx, ctx->registry,
+ ctx->repository,
+ ctx->tag,
+ ctx->platform,
+ ctx->auth_header);
+ if (IS_ERR(ctx->manifest_digest)) {
+ ret = PTR_ERR(ctx->manifest_digest);
erofs_err("failed to get manifest digest: %s",
erofs_strerror(ret));
+ ctx->manifest_digest = NULL;
goto out_auth;
}
- layers = ocierofs_get_layers_info(oci, oci->params.registry,
- oci->params.repository,
- manifest_digest, auth_header,
- &layer_count);
- if (IS_ERR(layers)) {
- ret = PTR_ERR(layers);
+ ctx->layers = ocierofs_fetch_layers_info(ctx, ctx->registry,
+ ctx->repository,
+ ctx->manifest_digest, ctx->auth_header,
+ &ctx->layer_count);
+ if (IS_ERR(ctx->layers)) {
+ ret = PTR_ERR(ctx->layers);
erofs_err("failed to get image layers: %s", erofs_strerror(ret));
+ ctx->layers = NULL;
goto out_manifest;
}
- if (oci->params.layer_index >= 0) {
- if (oci->params.layer_index >= layer_count) {
+ if (ctx->layer_index >= 0) {
+ if (ctx->layer_index >= ctx->layer_count) {
erofs_err("layer index %d exceeds available layers (%d)",
- oci->params.layer_index, layer_count);
+ ctx->layer_index, ctx->layer_count);
ret = -EINVAL;
goto out_layers;
}
- layer_count = 1;
- i = oci->params.layer_index;
+ ctx->layer_count = 1;
} else {
- i = 0;
+ ctx->layer_index = 0;
}
- while (i < layer_count) {
- char *trimmed = erofs_trim_for_progressinfo(layers[i],
- sizeof("Extracting layer ...") - 1);
- erofs_update_progressinfo("Extracting layer %d: %s ...", i,
- trimmed);
- free(trimmed);
- ret = ocierofs_extract_layer(oci, importer, layers[i],
- auth_header);
- if (ret) {
- erofs_err("failed to extract layer %d: %s", i,
- erofs_strerror(ret));
- break;
- }
- i++;
- }
+ return 0;
+
out_layers:
- for (i = 0; i < layer_count; i++)
- free(layers[i]);
- free(layers);
+ free(ctx->layers);
+ ctx->layers = NULL;
out_manifest:
- free(manifest_digest);
+ free(ctx->manifest_digest);
+ ctx->manifest_digest = NULL;
out_auth:
- free(auth_header);
-
- if (oci->params.username && oci->params.password &&
- oci->params.username[0] && oci->params.password[0] &&
- !auth_header) {
- ocierofs_curl_clear_auth(oci->curl);
- }
-out:
+ free(ctx->auth_header);
+ ctx->auth_header = NULL;
+ if (ctx->using_basic)
+ ocierofs_curl_clear_auth(ctx);
return ret;
}
-/**
- * ocierofs_init - Initialize OCI client with default parameters
- * @oci: OCI client structure to initialize
- *
- * Initialize OCI client structure, set up CURL handle, and configure
- * default parameters including platform (linux/amd64), registry
- * (registry-1.docker.io), and tag (latest).
- *
- * Return: 0 on success, negative errno on failure
- */
-int ocierofs_init(struct erofs_oci *oci)
-{
- if (!oci)
- return -EINVAL;
-
- *oci = (struct erofs_oci){};
- oci->curl = curl_easy_init();
- if (!oci->curl)
- return -EIO;
-
- if (ocierofs_curl_setup_common_options(oci->curl)) {
- ocierofs_cleanup(oci);
- return -EIO;
- }
-
- if (erofs_oci_params_set_string(&oci->params.platform,
- "linux/amd64") ||
- erofs_oci_params_set_string(&oci->params.registry,
- DOCKER_API_REGISTRY) ||
- erofs_oci_params_set_string(&oci->params.tag, "latest")) {
- ocierofs_cleanup(oci);
- return -ENOMEM;
- }
- oci->params.layer_index = -1; /* -1 means extract all layers */
- return 0;
-}
-
-/**
- * ocierofs_cleanup - Clean up OCI client and free allocated resources
- * @oci: OCI client structure to clean up
- *
- * Clean up CURL handle, free all allocated string parameters, and
- * reset the OCI client structure to a clean state.
- */
-void ocierofs_cleanup(struct erofs_oci *oci)
-{
- if (!oci)
- return;
-
- if (oci->curl) {
- curl_easy_cleanup(oci->curl);
- oci->curl = NULL;
- }
-
- free(oci->params.registry);
- free(oci->params.repository);
- free(oci->params.tag);
- free(oci->params.platform);
- free(oci->params.username);
- free(oci->params.password);
-
- oci->params.registry = NULL;
- oci->params.repository = NULL;
- oci->params.tag = NULL;
- oci->params.platform = NULL;
- oci->params.username = NULL;
- oci->params.password = NULL;
-}
-
-int erofs_oci_params_set_string(char **field, const char *value)
-{
- char *new_value;
-
- if (!field)
- return -EINVAL;
-
- if (!value) {
- free(*field);
- *field = NULL;
- return 0;
- }
-
- new_value = strdup(value);
- if (!new_value)
- return -ENOMEM;
-
- free(*field);
- *field = new_value;
- return 0;
-}
-
-int ocierofs_parse_ref(struct erofs_oci *oci, const char *ref_str)
+static int ocierofs_parse_ref(struct ocierofs_ctx *ctx, const char *ref_str)
{
char *slash, *colon, *dot;
const char *repo_part;
size_t len;
+ char *tmp;
+
+ if (!ctx || !ref_str)
+ return -EINVAL;
slash = strchr(ref_str, '/');
if (slash) {
dot = strchr(ref_str, '.');
if (dot && dot < slash) {
- char *registry_str;
-
len = slash - ref_str;
- registry_str = strndup(ref_str, len);
-
- if (!registry_str) {
- erofs_err("failed to allocate memory for registry");
- return -ENOMEM;
- }
- if (erofs_oci_params_set_string(&oci->params.registry,
- registry_str)) {
- free(registry_str);
- erofs_err("failed to set registry");
+ tmp = strndup(ref_str, len);
+ if (!tmp)
return -ENOMEM;
- }
- free(registry_str);
+ free(ctx->registry);
+ ctx->registry = tmp;
repo_part = slash + 1;
} else {
repo_part = ref_str;
@@ -1106,73 +940,231 @@ int ocierofs_parse_ref(struct erofs_oci *oci, const char *ref_str)
colon = strchr(repo_part, ':');
if (colon) {
- char *repo_str;
-
len = colon - repo_part;
- repo_str = strndup(repo_part, len);
-
- if (!repo_str) {
- erofs_err("failed to allocate memory for repository");
+ tmp = strndup(repo_part, len);
+ if (!tmp)
return -ENOMEM;
- }
- if (!strchr(repo_str, '/') &&
- (!strcmp(oci->params.registry, DOCKER_API_REGISTRY) ||
- !strcmp(oci->params.registry, DOCKER_REGISTRY))) {
+ if (!strchr(tmp, '/') &&
+ (!strcmp(ctx->registry, DOCKER_API_REGISTRY) ||
+ !strcmp(ctx->registry, DOCKER_REGISTRY))) {
char *full_repo;
- if (asprintf(&full_repo, "library/%s", repo_str) == -1) {
- free(repo_str);
- erofs_err("failed to allocate memory for full repository name");
+ if (asprintf(&full_repo, "library/%s", tmp) == -1) {
+ free(tmp);
return -ENOMEM;
}
- free(repo_str);
- repo_str = full_repo;
+ free(tmp);
+ tmp = full_repo;
}
+ free(ctx->repository);
+ ctx->repository = tmp;
- if (erofs_oci_params_set_string(&oci->params.repository,
- repo_str)) {
- free(repo_str);
- erofs_err("failed to set repository");
+ free(ctx->tag);
+ ctx->tag = strdup(colon + 1);
+ if (!ctx->tag)
return -ENOMEM;
- }
- free(repo_str);
-
- if (erofs_oci_params_set_string(&oci->params.tag,
- colon + 1)) {
- erofs_err("failed to set tag");
- return -ENOMEM;
- }
} else {
- char *repo_str = strdup(repo_part);
-
- if (!repo_str) {
- erofs_err("failed to allocate memory for repository");
+ tmp = strdup(repo_part);
+ if (!tmp)
return -ENOMEM;
- }
- if (!strchr(repo_str, '/') &&
- (!strcmp(oci->params.registry, DOCKER_API_REGISTRY) ||
- !strcmp(oci->params.registry, DOCKER_REGISTRY))) {
+ if (!strchr(tmp, '/') &&
+ (!strcmp(ctx->registry, DOCKER_API_REGISTRY) ||
+ !strcmp(ctx->registry, DOCKER_REGISTRY))) {
+
char *full_repo;
- if (asprintf(&full_repo, "library/%s", repo_str) == -1) {
- free(repo_str);
- erofs_err("failed to allocate memory for full repository name");
+ if (asprintf(&full_repo, "library/%s", tmp) == -1) {
+ free(tmp);
return -ENOMEM;
}
- free(repo_str);
- repo_str = full_repo;
+ free(tmp);
+ tmp = full_repo;
}
+ free(ctx->repository);
+ ctx->repository = tmp;
+ }
+ return 0;
+}
- if (erofs_oci_params_set_string(&oci->params.repository,
- repo_str)) {
- free(repo_str);
- erofs_err("failed to set repository");
- return -ENOMEM;
+int ocierofs_init(struct ocierofs_ctx *ctx, const struct ocierofs_config *config)
+{
+ int ret;
+
+ ctx->curl = curl_easy_init();
+ if (!ctx->curl)
+ return -EIO;
+
+ if (ocierofs_curl_setup_common_options(ctx->curl))
+ return -EIO;
+
+ ctx->layer_index = -1;
+ ctx->registry = strdup("registry-1.docker.io");
+ ctx->tag = strdup("latest");
+ if (config && config->platform)
+ ctx->platform = strdup(config->platform);
+ else
+ ctx->platform = strdup("linux/amd64");
+ if (!ctx->registry || !ctx->tag || !ctx->platform)
+ return -ENOMEM;
+
+ if (config && config->layer_index >= 0)
+ ctx->layer_index = config->layer_index;
+
+ ret = ocierofs_parse_ref(ctx, config->image_ref);
+ if (ret)
+ return ret;
+
+ ret = ocierofs_prepare_layers(ctx, config);
+ if (ret)
+ return ret;
+
+ return 0;
+}
+
+static int ocierofs_download_blob_to_fd(struct ocierofs_ctx *ctx,
+ const char *digest,
+ const char *auth_header,
+ int outfd)
+{
+ struct ocierofs_request req = {};
+ struct ocierofs_stream stream = {};
+ const char *api_registry;
+ long http_code;
+ int ret;
+
+ stream = (struct ocierofs_stream) {
+ .digest = digest,
+ .blobfd = outfd,
+ };
+
+ api_registry = ocierofs_get_api_registry(ctx->registry);
+ if (asprintf(&req.url, "https://%s/v2/%s/blobs/%s",
+ api_registry, ctx->repository, digest) == -1)
+ return -ENOMEM;
+
+ if (auth_header && strstr(auth_header, "Bearer"))
+ req.headers = curl_slist_append(req.headers, auth_header);
+
+ curl_easy_reset(ctx->curl);
+
+ ret = ocierofs_curl_setup_common_options(ctx->curl);
+ if (ret)
+ goto out;
+
+ ret = ocierofs_curl_setup_rq(ctx->curl, req.url, OCIEROFS_HTTP_GET,
+ req.headers,
+ ocierofs_layer_write_callback,
+ &stream, NULL, NULL);
+ if (ret)
+ goto out;
+
+ ret = ocierofs_curl_perform(ctx->curl, &http_code);
+ if (ret)
+ goto out;
+
+ if (http_code < 200 || http_code >= 300) {
+ erofs_err("HTTP request failed with code %ld", http_code);
+ ret = -EIO;
+ goto out;
+ }
+ ret = 0;
+out:
+ ocierofs_request_cleanup(&req);
+ return ret;
+}
+
+static int ocierofs_extract_layer(struct ocierofs_ctx *ctx,
+ const char *digest, const char *auth_header)
+{
+ struct ocierofs_stream stream = {};
+ int ret;
+
+ stream = (struct ocierofs_stream) {
+ .digest = digest,
+ .blobfd = erofs_tmpfile(),
+ };
+ if (stream.blobfd < 0) {
+ erofs_err("failed to create temporary file for %s", digest);
+ return -errno;
+ }
+
+ ret = ocierofs_download_blob_to_fd(ctx, digest, auth_header, stream.blobfd);
+ if (ret)
+ goto out;
+
+ if (lseek(stream.blobfd, 0, SEEK_SET) < 0) {
+ erofs_err("failed to seek to beginning of temp file: %s",
+ strerror(errno));
+ ret = -errno;
+ goto out;
+ }
+
+ return stream.blobfd;
+
+out:
+ if (stream.blobfd >= 0)
+ close(stream.blobfd);
+ return ret;
+}
+
+static void ocierofs_ctx_cleanup(struct ocierofs_ctx *ctx)
+{
+ if (!ctx)
+ return;
+
+ if (ctx->curl) {
+ curl_easy_cleanup(ctx->curl);
+ ctx->curl = NULL;
+ }
+ free(ctx->auth_header);
+ ctx->auth_header = NULL;
+
+ ocierofs_free_layers_info(ctx->layers, ctx->layer_count);
+ free(ctx->registry);
+ free(ctx->repository);
+ free(ctx->tag);
+ free(ctx->platform);
+ free(ctx->manifest_digest);
+}
+
+int ocierofs_build_trees(struct erofs_importer *importer,
+ const struct ocierofs_config *config)
+{
+ struct ocierofs_ctx ctx = {};
+ int ret, i;
+
+ ret = ocierofs_init(&ctx, config);
+ if (ret) {
+ ocierofs_ctx_cleanup(&ctx);
+ return ret;
+ }
+
+ i = ctx.layer_index;
+ while (i < ctx.layer_count) {
+ char *trimmed = erofs_trim_for_progressinfo(ctx.layers[i]->digest,
+ sizeof("Extracting layer ...") - 1);
+ erofs_update_progressinfo("Extracting layer %d: %s ...", i,
+ trimmed);
+ free(trimmed);
+ int fd = ocierofs_extract_layer(&ctx, ctx.layers[i]->digest,
+ ctx.auth_header);
+ if (fd < 0) {
+ erofs_err("failed to extract layer %d: %s", i,
+ erofs_strerror(fd));
+ break;
}
- free(repo_str);
+ ret = ocierofs_process_tar_stream(importer, fd);
+ close(fd);
+ if (ret) {
+ erofs_err("failed to process tar stream for layer %d: %s", i,
+ erofs_strerror(ret));
+ break;
+ }
+ i++;
}
- return 0;
+ ocierofs_ctx_cleanup(&ctx);
+ return ret;
}
diff --git a/mkfs/main.c b/mkfs/main.c
index 064392d..721c0f0 100644
--- a/mkfs/main.c
+++ b/mkfs/main.c
@@ -150,18 +150,18 @@ static void usage(int argc, char **argv)
* "0-9,100-109" instead of a continuous "0-109", and to
* state what those two subranges respectively mean. */
printf("%s [,level=<0-9,100-109>]\t0-9=normal, 100-109=extreme (default=%i)\n",
- spaces, s->c->default_level);
+ spaces, s->c->default_level);
else
printf("%s [,level=<0-%i>]\t\t(default=%i)\n",
- spaces, s->c->best_level, s->c->default_level);
+ spaces, s->c->best_level, s->c->default_level);
}
if (s->c->setdictsize) {
if (s->c->default_dictsize)
printf("%s [,dictsize=<dictsize>]\t(default=%u, max=%u)\n",
- spaces, s->c->default_dictsize, s->c->max_dictsize);
+ spaces, s->c->default_dictsize, s->c->max_dictsize);
else
printf("%s [,dictsize=<dictsize>]\t(default=<auto>, max=%u)\n",
- spaces, s->c->max_dictsize);
+ spaces, s->c->max_dictsize);
}
}
printf(
@@ -272,7 +272,7 @@ static struct erofs_s3 s3cfg;
#endif
#ifdef OCIEROFS_ENABLED
-static struct erofs_oci ocicfg;
+static struct ocierofs_config ocicfg;
static char *mkfs_oci_options;
#endif
@@ -689,21 +689,14 @@ static int mkfs_parse_s3_cfg(char *cfg_str)
#endif
#ifdef OCIEROFS_ENABLED
-
-
-/**
- * mkfs_parse_oci_options - Parse comma-separated OCI options string
+/*
+ * mkfs_parse_oci_options - Parse OCI options for mkfs tool
+ * @cfg: OCI configuration structure to update
* @options_str: comma-separated options string
- *
- * Parse OCI options string containing comma-separated key=value pairs.
- * Supported options include platform, layer, username, and password.
- *
- * Return: 0 on success, negative errno on failure
*/
-static int mkfs_parse_oci_options(char *options_str)
+static int mkfs_parse_oci_options(struct ocierofs_config *oci_cfg, char *options_str)
{
char *opt, *q, *p;
- int ret;
if (!options_str)
return 0;
@@ -717,41 +710,43 @@ static int mkfs_parse_oci_options(char *options_str)
p = strstr(opt, "platform=");
if (p) {
p += strlen("platform=");
- ret = erofs_oci_params_set_string(&ocicfg.params.platform, p);
- if (ret) {
- erofs_err("failed to set platform");
- return ret;
- }
+ free(oci_cfg->platform);
+ oci_cfg->platform = strdup(p);
+ if (!oci_cfg->platform)
+ return -ENOMEM;
} else {
p = strstr(opt, "layer=");
if (p) {
p += strlen("layer=");
- ocicfg.params.layer_index = atoi(p);
- if (ocicfg.params.layer_index < 0) {
- erofs_err("invalid layer index %d",
- ocicfg.params.layer_index);
- return -EINVAL;
+ {
+ char *endptr;
+ unsigned long v = strtoul(p, &endptr, 10);
+
+ if (endptr == p || *endptr != '\0') {
+ erofs_err("invalid layer index %s",
+ p);
+ return -EINVAL;
+ }
+ oci_cfg->layer_index = (int)v;
}
} else {
p = strstr(opt, "username=");
if (p) {
p += strlen("username=");
- ret = erofs_oci_params_set_string(&ocicfg.params.username, p);
- if (ret) {
- erofs_err("failed to set username");
- return ret;
- }
+ free(oci_cfg->username);
+ oci_cfg->username = strdup(p);
+ if (!oci_cfg->username)
+ return -ENOMEM;
} else {
p = strstr(opt, "password=");
if (p) {
p += strlen("password=");
- ret = erofs_oci_params_set_string(&ocicfg.params.password, p);
- if (ret) {
- erofs_err("failed to set password");
- return ret;
- }
+ free(oci_cfg->password);
+ oci_cfg->password = strdup(p);
+ if (!oci_cfg->password)
+ return -ENOMEM;
} else {
- erofs_err("invalid --oci value %s", opt);
+ erofs_err("mkfs: invalid --oci value %s", opt);
return -EINVAL;
}
}
@@ -1832,16 +1827,12 @@ int main(int argc, char **argv)
#endif
#ifdef OCIEROFS_ENABLED
} else if (source_mode == EROFS_MKFS_SOURCE_OCI) {
- err = ocierofs_init(&ocicfg);
- if (err)
- goto exit;
+ ocicfg.layer_index = -1;
- err = mkfs_parse_oci_options(mkfs_oci_options);
- if (err)
- goto exit;
- err = ocierofs_parse_ref(&ocicfg, cfg.c_src_path);
+ err = mkfs_parse_oci_options(&ocicfg, mkfs_oci_options);
if (err)
goto exit;
+ ocicfg.image_ref = cfg.c_src_path;
if (incremental_mode ||
dataimport_mode == EROFS_MKFS_DATA_IMPORT_RVSP ||
@@ -1916,9 +1907,6 @@ exit:
erofs_blob_exit();
erofs_xattr_cleanup_name_prefixes();
erofs_rebuild_cleanup();
-#ifdef OCIEROFS_ENABLED
- ocierofs_cleanup(&ocicfg);
-#endif
erofs_diskbuf_exit();
if (source_mode == EROFS_MKFS_SOURCE_TAR) {
erofs_iostream_close(&erofstar.ios);
--
2.51.0
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH v4 3/3] erofs-utils: add NBD-backed OCI image mounting
2025-09-04 5:33 ` [PATCH v4 0/3] erofs-utils: refactor OCI and " ChengyuZhu6
2025-09-04 5:33 ` [PATCH v4 1/3] erofs-utils:mkfs: move parse_oci_ref to lib ChengyuZhu6
2025-09-04 5:33 ` [PATCH v4 2/3] erofs-utils: refactor OCI code for better modularity ChengyuZhu6
@ 2025-09-04 5:33 ` ChengyuZhu6
2 siblings, 0 replies; 19+ messages in thread
From: ChengyuZhu6 @ 2025-09-04 5:33 UTC (permalink / raw)
To: linux-erofs; +Cc: xiang, hsiangkao, Chengyu Zhu
From: Chengyu Zhu <hudsonzhu@tencent.com>
- Add HTTP range downloads for OCI blobs
- Introduce ocierofs_iostream for virtual file I/O
- Add --oci option for OCI image mounting with NBD backend
New mount.erofs -t erofs.nbd option: -o=[options] source-image mountpoint
Supported oci options:
- oci.platform=os/arch (default: linux/amd64)
- oci=N (extract specific layer, default: all layers)
- oci.username/oci.password (basic authentication)
e.g.:
./mount/mount.erofs -t erofs.nbd -o 'oci=1,oci.platform=linux/amd64' \
quay.io/chengyuzhu6/golang:1.22.8-erofs /tmp/test/
Signed-off-by: Chengyu Zhu <hudsonzhu@tencent.com>
---
lib/liberofs_oci.h | 14 +++
lib/remotes/oci.c | 241 +++++++++++++++++++++++++++++++++++++++++-
mount/Makefile.am | 2 +-
mount/main.c | 257 ++++++++++++++++++++++++++++++++++++++-------
4 files changed, 476 insertions(+), 38 deletions(-)
diff --git a/lib/liberofs_oci.h b/lib/liberofs_oci.h
index 8622464..71ea333 100644
--- a/lib/liberofs_oci.h
+++ b/lib/liberofs_oci.h
@@ -51,6 +51,12 @@ struct ocierofs_ctx {
int layer_count;
};
+struct ocierofs_iostream {
+ struct ocierofs_ctx *ctx;
+ struct erofs_vfile vf;
+ u64 offset;
+};
+
int ocierofs_init(struct ocierofs_ctx *ctx, const struct ocierofs_config *config);
/*
@@ -63,6 +69,14 @@ int ocierofs_init(struct ocierofs_ctx *ctx, const struct ocierofs_config *config
int ocierofs_build_trees(struct erofs_importer *importer,
const struct ocierofs_config *cfg);
+int ocierofs_is_erofs_native_image(struct ocierofs_ctx *ctx);
+
+void ocierofs_ctx_cleanup(struct ocierofs_ctx *ctx);
+
+int ocierofs_iostream_open(struct ocierofs_iostream *oci_iostream, struct ocierofs_ctx *oci_ctx);
+
+void ocierofs_iostream_close(struct ocierofs_iostream *oci_iostream);
+
#ifdef __cplusplus
}
#endif
diff --git a/lib/remotes/oci.c b/lib/remotes/oci.c
index 4291dc6..0fdff1f 100644
--- a/lib/remotes/oci.c
+++ b/lib/remotes/oci.c
@@ -33,6 +33,9 @@
#define OCI_MEDIATYPE_MANIFEST "application/vnd.oci.image.manifest.v1+json"
#define OCI_MEDIATYPE_INDEX "application/vnd.oci.image.index.v1+json"
+/* Erofs Native Layer Media Type */
+#define EROFS_MEDIATYPE "application/vnd.erofs"
+
struct ocierofs_request {
char *url;
struct curl_slist *headers;
@@ -1109,7 +1112,7 @@ out:
return ret;
}
-static void ocierofs_ctx_cleanup(struct ocierofs_ctx *ctx)
+void ocierofs_ctx_cleanup(struct ocierofs_ctx *ctx)
{
if (!ctx)
return;
@@ -1168,3 +1171,239 @@ int ocierofs_build_trees(struct erofs_importer *importer,
ocierofs_ctx_cleanup(&ctx);
return ret;
}
+
+static int ocierofs_download_blob_range(struct ocierofs_ctx *ctx, off_t offset, size_t length,
+ void **out_buf, size_t *out_size)
+{
+ struct ocierofs_request req = {};
+ struct ocierofs_response resp = {};
+ const char *api_registry;
+ char rangehdr[64];
+ long http_code = 0;
+ int ret;
+ int index = ctx->layer_index;
+
+ if (offset < 0)
+ return -EINVAL;
+
+ api_registry = ocierofs_get_api_registry(ctx->registry);
+ if (asprintf(&req.url, "https://%s/v2/%s/blobs/%s",
+ api_registry, ctx->repository, ctx->layers[index]->digest) == -1)
+ return -ENOMEM;
+
+ if (length)
+ snprintf(rangehdr, sizeof(rangehdr), "Range: bytes=%lld-%lld",
+ (long long)offset, (long long)(offset + (off_t)length - 1));
+ else
+ snprintf(rangehdr, sizeof(rangehdr), "Range: bytes=%lld-",
+ (long long)offset);
+
+ if (ctx->auth_header && strstr(ctx->auth_header, "Bearer"))
+ req.headers = curl_slist_append(req.headers, ctx->auth_header);
+ req.headers = curl_slist_append(req.headers, rangehdr);
+
+ curl_easy_reset(ctx->curl);
+
+ ret = ocierofs_curl_setup_common_options(ctx->curl);
+ if (ret)
+ goto out;
+
+ ret = ocierofs_curl_setup_rq(ctx->curl, req.url, OCIEROFS_HTTP_GET,
+ req.headers,
+ ocierofs_write_callback,
+ &resp, NULL, NULL);
+ if (ret)
+ goto out;
+
+ ret = ocierofs_curl_perform(ctx->curl, &http_code);
+ if (ret)
+ goto out;
+
+ if (http_code == 206) {
+ *out_buf = resp.data;
+ *out_size = resp.size;
+ resp.data = NULL;
+ ret = 0;
+ } else if (http_code == 200) {
+ if (offset == 0) {
+ *out_buf = resp.data;
+ *out_size = resp.size;
+ resp.data = NULL;
+ ret = 0;
+ } else {
+ if (offset < resp.size) {
+ size_t available = resp.size - offset;
+ size_t copy_size = length ? min_t(size_t, length, available) : available;
+
+ *out_buf = malloc(copy_size);
+ if (!*out_buf) {
+ ret = -ENOMEM;
+ goto out;
+ }
+ memcpy(*out_buf, resp.data + offset, copy_size);
+ *out_size = copy_size;
+ ret = 0;
+ } else {
+ *out_buf = NULL;
+ *out_size = 0;
+ ret = 0;
+ }
+ }
+ } else {
+ erofs_err("HTTP range request failed with code %ld", http_code);
+ ret = -EIO;
+ }
+
+out:
+ if (req.headers)
+ curl_slist_free_all(req.headers);
+ free(req.url);
+ free(resp.data);
+ return ret;
+}
+
+static ssize_t ocierofs_io_pread(struct erofs_vfile *vf, void *buf, size_t len, u64 offset)
+{
+ struct ocierofs_iostream *oci_iostream = *(struct ocierofs_iostream **)vf->payload;
+ void *download_buf = NULL;
+ size_t download_size = 0;
+ ssize_t ret;
+
+ ret = ocierofs_download_blob_range(oci_iostream->ctx, offset, len,
+ &download_buf, &download_size);
+ if (ret < 0) {
+ memset(buf, 0, len);
+ return len;
+ }
+
+ if (download_buf && download_size > 0) {
+ memcpy(buf, download_buf, download_size);
+ free(download_buf);
+ return download_size;
+ }
+
+ return 0;
+}
+
+static ssize_t ocierofs_io_read(struct erofs_vfile *vf, void *buf, size_t len)
+{
+ struct ocierofs_iostream *oci_iostream = *(struct ocierofs_iostream **)vf->payload;
+ ssize_t ret;
+
+ ret = ocierofs_io_pread(vf, buf, len, oci_iostream->offset);
+ if (ret > 0)
+ oci_iostream->offset += ret;
+
+ return ret;
+}
+
+static off_t ocierofs_io_lseek(struct erofs_vfile *vf, u64 offset, int whence)
+{
+ struct ocierofs_iostream *oci_iostream = *(struct ocierofs_iostream **)vf->payload;
+ off_t new_offset;
+ int layer_index = oci_iostream->ctx->layer_index;
+
+ switch (whence) {
+ case SEEK_SET:
+ new_offset = offset;
+ break;
+ case SEEK_CUR:
+ new_offset = oci_iostream->offset + offset;
+ break;
+ case SEEK_END:
+ new_offset = oci_iostream->ctx->layers[layer_index]->size + offset;
+ break;
+ default:
+ return -1;
+ }
+
+ if (new_offset < 0 || new_offset > oci_iostream->ctx->layers[layer_index]->size)
+ return -1;
+
+ oci_iostream->offset = new_offset;
+ return new_offset;
+}
+
+static ssize_t ocierofs_io_sendfile(struct erofs_vfile *vout, struct erofs_vfile *vin,
+ off_t *pos, size_t count)
+{
+ struct ocierofs_iostream *oci_iostream = *(struct ocierofs_iostream **)vin->payload;
+ char *buf = NULL;
+ ssize_t total_written = 0;
+ ssize_t ret = 0;
+
+ buf = malloc(min_t(size_t, count, 32768));
+ if (!buf)
+ return -ENOMEM;
+
+ while (count > 0) {
+ size_t to_read = min_t(size_t, count, 32768);
+ u64 read_offset = pos ? *pos : oci_iostream->offset;
+
+ ret = ocierofs_io_pread(vin, buf, to_read, read_offset);
+ if (ret <= 0) {
+ erofs_err("OCI I/O sendfile: failed to read from OCI: %s",
+ erofs_strerror(ret));
+ memset(buf, 0, to_read);
+ ret = to_read;
+ }
+
+ ssize_t written = write(vout->fd, buf, ret);
+
+ if (written < 0) {
+ erofs_err("OCI I/O sendfile: failed to write to output: %s",
+ strerror(errno));
+ ret = -errno;
+ break;
+ }
+
+ if (written != ret) {
+ erofs_err("OCI I/O sendfile: partial write: %zd != %zd", written, ret);
+ ret = written;
+ }
+
+ total_written += ret;
+ count -= ret;
+ if (pos)
+ *pos += ret;
+ else
+ oci_iostream->offset += ret;
+ }
+
+ free(buf);
+ return count;
+}
+
+static struct erofs_vfops ocierofs_io_vfops = {
+ .pread = ocierofs_io_pread,
+ .read = ocierofs_io_read,
+ .lseek = ocierofs_io_lseek,
+ .sendfile = ocierofs_io_sendfile,
+};
+
+int ocierofs_iostream_open(struct ocierofs_iostream *oci_iostream, struct ocierofs_ctx *oci_ctx)
+{
+ memset(oci_iostream, 0, sizeof(*oci_iostream));
+ oci_iostream->ctx = oci_ctx;
+ oci_iostream->vf.ops = &ocierofs_io_vfops;
+ oci_iostream->vf.fd = -1;
+ *(struct ocierofs_iostream **)oci_iostream->vf.payload = oci_iostream;
+
+ return 0;
+}
+
+void ocierofs_iostream_close(struct ocierofs_iostream *oci_iostream)
+{
+ close(oci_iostream->vf.fd);
+}
+
+int ocierofs_is_erofs_native_image(struct ocierofs_ctx *ctx)
+{
+ if (ctx->layer_count > 0 && ctx->layers[0] &&
+ ctx->layers[0]->media_type) {
+ if (strcmp(ctx->layers[0]->media_type, EROFS_MEDIATYPE) != 0)
+ return -ENOENT;
+ return 0;
+ }
+ return -ENOENT;
+}
diff --git a/mount/Makefile.am b/mount/Makefile.am
index d93f3f4..0b4447f 100644
--- a/mount/Makefile.am
+++ b/mount/Makefile.am
@@ -9,5 +9,5 @@ mount_erofs_SOURCES = main.c
mount_erofs_CFLAGS = -Wall -I$(top_srcdir)/include
mount_erofs_LDADD = $(top_builddir)/lib/liberofs.la ${libselinux_LIBS} \
${liblz4_LIBS} ${liblzma_LIBS} ${zlib_LIBS} ${libdeflate_LIBS} \
- ${libzstd_LIBS} ${libqpl_LIBS} ${libxxhash_LIBS} ${libnl3_LIBS}
+ ${libzstd_LIBS} ${libqpl_LIBS} ${libxxhash_LIBS} ${libnl3_LIBS} ${openssl_LIBS}
endif
diff --git a/mount/main.c b/mount/main.c
index a270f0a..177bca3 100644
--- a/mount/main.c
+++ b/mount/main.c
@@ -15,6 +15,7 @@
#include "erofs/err.h"
#include "erofs/io.h"
#include "../lib/liberofs_nbd.h"
+#include "../lib/liberofs_oci.h"
#ifdef HAVE_LINUX_LOOP_H
#include <linux/loop.h>
#else
@@ -34,6 +35,10 @@ struct loop_info {
#include <sys/sysmacros.h>
#endif
+#ifdef OCIEROFS_ENABLED
+static struct ocierofs_config ocicfg;
+#endif
+
enum erofs_backend_drv {
EROFSAUTO,
EROFSLOCAL,
@@ -56,12 +61,76 @@ static struct erofsmount_cfg {
long flags;
enum erofs_backend_drv backend;
enum erofsmount_mode mountmode;
+#ifdef OCIEROFS_ENABLED
+ bool use_oci;
+#endif
} mountcfg = {
.full_options = "ro",
.flags = MS_RDONLY, /* default mountflags */
.fstype = "erofs",
};
+enum erofs_nbd_source_type {
+ EROFSNBD_SOURCE_LOCAL,
+ EROFSNBD_SOURCE_OCI,
+};
+
+union erofs_nbd_source {
+ const char *device_path;
+ struct ocierofs_ctx *oci_ctx;
+};
+
+union erofs_nbd_source src;
+
+static int parse_oci_option(struct ocierofs_config *oci_cfg, const char *option)
+{
+ char *p;
+
+ p = strstr(option, "oci=");
+ if (p != NULL) {
+ p += strlen("oci=");
+ {
+ char *endptr;
+ unsigned long v = strtoul(p, &endptr, 10);
+
+ if (endptr == p || *endptr != '\0')
+ return -EINVAL;
+ oci_cfg->layer_index = (int)v;
+ }
+ } else {
+ p = strstr(option, "oci.platform=");
+ if (p != NULL) {
+ p += strlen("oci.platform=");
+ free(oci_cfg->platform);
+ oci_cfg->platform = strdup(p);
+ if (!oci_cfg->platform)
+ return -ENOMEM;
+ } else {
+ p = strstr(option, "oci.username=");
+ if (p != NULL) {
+ p += strlen("oci.username=");
+ free(oci_cfg->username);
+ oci_cfg->username = strdup(p);
+ if (!oci_cfg->username)
+ return -ENOMEM;
+ } else {
+ p = strstr(option, "oci.password=");
+ if (p != NULL) {
+ p += strlen("oci.password=");
+ free(oci_cfg->password);
+ oci_cfg->password = strdup(p);
+ if (!oci_cfg->password)
+ return -ENOMEM;
+ } else {
+ return -EINVAL;
+ }
+ }
+ }
+ }
+
+ return 0;
+}
+
static long erofsmount_parse_flagopts(char *s, long flags, char **more)
{
static const struct {
@@ -90,29 +159,41 @@ static long erofsmount_parse_flagopts(char *s, long flags, char **more)
comma = strchr(s, ',');
if (comma)
*comma = '\0';
- for (i = 0; i < ARRAY_SIZE(opts); ++i) {
- if (!strcasecmp(s, opts[i].name)) {
- if (opts[i].flags < 0)
- flags &= opts[i].flags;
- else
- flags |= opts[i].flags;
- break;
- }
- }
- if (more && i >= ARRAY_SIZE(opts)) {
- int sl = strlen(s);
- char *new = *more;
+ if (strncmp(s, "oci", 3) == 0) {
+#ifdef OCIEROFS_ENABLED
+ int err = parse_oci_option(&ocicfg, s);
- i = new ? strlen(new) : 0;
- new = realloc(new, i + strlen(s) + 2);
- if (!new)
- return -ENOMEM;
- if (i)
- new[i++] = ',';
- memcpy(new + i, s, sl);
- new[i + sl] = '\0';
- *more = new;
+ if (err < 0)
+ return err;
+#else
+ return -EINVAL;
+#endif
+ } else {
+ for (i = 0; i < ARRAY_SIZE(opts); ++i) {
+ if (!strcasecmp(s, opts[i].name)) {
+ if (opts[i].flags < 0)
+ flags &= opts[i].flags;
+ else
+ flags |= opts[i].flags;
+ break;
+ }
+ }
+
+ if (more && i >= ARRAY_SIZE(opts)) {
+ int sl = strlen(s);
+ char *new = *more;
+
+ i = new ? strlen(new) : 0;
+ new = realloc(new, i + strlen(s) + 2);
+ if (!new)
+ return -ENOMEM;
+ if (i)
+ new[i++] = ',';
+ memcpy(new + i, s, sl);
+ new[i + sl] = '\0';
+ *more = new;
+ }
}
if (!comma)
@@ -120,6 +201,11 @@ static long erofsmount_parse_flagopts(char *s, long flags, char **more)
*comma = ',';
s = comma + 1;
}
+
+#ifdef OCIEROFS_ENABLED
+ if (ocicfg.platform || ocicfg.username || ocicfg.password || ocicfg.layer_index != 0)
+ mountcfg.use_oci = true;
+#endif
return flags;
}
@@ -540,9 +626,64 @@ err_identifier:
return err;
}
-static int erofsmount_nbd(const char *source, const char *mountpoint,
- const char *fstype, int flags,
- const char *options)
+static int erofsmount_startnbd_oci(int nbdfd, struct ocierofs_ctx *oci_ctx)
+{
+ struct erofsmount_nbd_ctx ctx = {};
+ struct ocierofs_iostream *oci_iostream = NULL;
+ uintptr_t retcode;
+ pthread_t th;
+ int err, err2;
+ int blkbits = 12;
+ int index = oci_ctx->layer_index;
+ u64 blocks;
+
+ blocks = (oci_ctx->layers[index]->size + (1ULL << blkbits) - 1) >> blkbits;
+
+ oci_iostream = malloc(sizeof(struct ocierofs_iostream));
+ if (!oci_iostream)
+ return -ENOMEM;
+
+ err = ocierofs_iostream_open(oci_iostream, oci_ctx);
+ if (err) {
+ free(oci_iostream);
+ return err;
+ }
+
+ ctx.vd = oci_iostream->vf;
+
+ err = erofs_nbd_connect(nbdfd, blkbits, blocks);
+ if (err < 0) {
+ ocierofs_iostream_close(oci_iostream);
+ free(oci_iostream);
+ return err;
+ }
+ ctx.sk.fd = err;
+
+ err = -pthread_create(&th, NULL, erofsmount_nbd_loopfn, &ctx);
+ if (err) {
+ ocierofs_iostream_close(oci_iostream);
+ free(oci_iostream);
+ close(ctx.sk.fd);
+ return err;
+ }
+
+ err = erofs_nbd_do_it(nbdfd);
+ err2 = -pthread_join(th, (void **)&retcode);
+ if (!err2 && retcode) {
+ erofs_err("NBD worker failed with %s",
+ erofs_strerror(retcode));
+ err2 = retcode;
+ }
+
+ ocierofs_iostream_close(oci_iostream);
+ free(oci_iostream);
+
+ return err ?: err2;
+}
+
+static int erofsmount_nbd(union erofs_nbd_source source, enum erofs_nbd_source_type source_type,
+ const char *mountpoint, const char *fstype,
+ int flags, const char *options)
{
bool is_netlink = false;
char nbdpath[32], *id;
@@ -555,11 +696,19 @@ static int erofsmount_nbd(const char *source, const char *mountpoint,
mountcfg.fstype);
return -ENODEV;
}
+
flags |= MS_RDONLY;
- err = erofsmount_startnbd_nl(&pid, source);
- if (err < 0) {
- erofs_info("Fall back to ioctl-based NBD; failover is unsupported");
+ if (source_type == EROFSNBD_SOURCE_LOCAL) {
+ err = erofsmount_startnbd_nl(&pid, source.device_path);
+ if (err >= 0) {
+ num = err;
+ (void)snprintf(nbdpath, sizeof(nbdpath), "/dev/nbd%d", num);
+ is_netlink = true;
+ }
+ }
+
+ if (!is_netlink) {
num = erofs_nbd_devscan();
if (num < 0)
return num;
@@ -569,14 +718,16 @@ static int erofsmount_nbd(const char *source, const char *mountpoint,
if (nbdfd < 0)
return -errno;
- if ((pid = fork()) == 0)
- return erofsmount_startnbd(nbdfd, source) ?
- EXIT_FAILURE : EXIT_SUCCESS;
+ if ((pid = fork()) == 0) {
+ if (source_type == EROFSNBD_SOURCE_OCI) {
+ return erofsmount_startnbd_oci(nbdfd, source.oci_ctx) ?
+ EXIT_FAILURE : EXIT_SUCCESS;
+ } else {
+ return erofsmount_startnbd(nbdfd, source.device_path) ?
+ EXIT_FAILURE : EXIT_SUCCESS;
+ }
+ }
close(nbdfd);
- } else {
- num = err;
- (void)snprintf(nbdpath, sizeof(nbdpath), "/dev/nbd%d", num);
- is_netlink = true;
}
while (1) {
@@ -594,7 +745,7 @@ static int erofsmount_nbd(const char *source, const char *mountpoint,
if (err < 0)
err = -errno;
- if (!err && is_netlink) {
+ if (!err && is_netlink && source_type == EROFSNBD_SOURCE_LOCAL) {
id = erofs_nbd_get_identifier(num);
if (id == ERR_PTR(-ENOENT))
id = NULL;
@@ -799,9 +950,43 @@ int main(int argc, char *argv[])
}
if (mountcfg.backend == EROFSNBD) {
- err = erofsmount_nbd(mountcfg.device, mountcfg.target,
+#ifdef OCIEROFS_ENABLED
+ if (mountcfg.use_oci) {
+ struct ocierofs_ctx ctx = {};
+
+ ocicfg.image_ref = mountcfg.device;
+ err = ocierofs_init(&ctx, &ocicfg);
+ if (err) {
+ ocierofs_ctx_cleanup(&ctx);
+ goto exit;
+ }
+
+ err = ocierofs_is_erofs_native_image(&ctx);
+ if (err) {
+ ocierofs_ctx_cleanup(&ctx);
+ goto exit;
+ }
+
+ src.oci_ctx = &ctx;
+
+ err = erofsmount_nbd(src, EROFSNBD_SOURCE_OCI, mountcfg.target,
+ mountcfg.fstype, mountcfg.flags, mountcfg.options);
+ if (err) {
+ ocierofs_ctx_cleanup(&ctx);
+ goto exit;
+ }
+ } else {
+ src.device_path = mountcfg.device;
+ err = erofsmount_nbd(src, EROFSNBD_SOURCE_LOCAL, mountcfg.target,
+ mountcfg.fstype, mountcfg.flags,
+ mountcfg.options);
+ }
+#else
+ src.device_path = mountcfg.device;
+ err = erofsmount_nbd(src, EROFSNBD_SOURCE_LOCAL, mountcfg.target,
mountcfg.fstype, mountcfg.flags,
mountcfg.options);
+#endif
goto exit;
}
--
2.51.0
^ permalink raw reply related [flat|nested] 19+ messages in thread
* Re: [PATCH v4 2/3] erofs-utils: refactor OCI code for better modularity
2025-09-04 5:33 ` [PATCH v4 2/3] erofs-utils: refactor OCI code for better modularity ChengyuZhu6
@ 2025-09-04 5:53 ` Gao Xiang
0 siblings, 0 replies; 19+ messages in thread
From: Gao Xiang @ 2025-09-04 5:53 UTC (permalink / raw)
To: ChengyuZhu6, linux-erofs; +Cc: xiang, Chengyu Zhu
Hi Chengyu,
On 2025/9/4 13:33, ChengyuZhu6 wrote:
> From: Chengyu Zhu <hudsonzhu@tencent.com>
..
> @@ -96,6 +142,14 @@ static int ocierofs_curl_setup_common_options(struct CURL *curl)
> curl_easy_setopt(curl, CURLOPT_TIMEOUT, 120L);
> curl_easy_setopt(curl, CURLOPT_NOSIGNAL, 1L);
> curl_easy_setopt(curl, CURLOPT_USERAGENT, "ocierofs/" PACKAGE_VERSION);
> + curl_easy_setopt(curl, CURLOPT_ACCEPT_ENCODING, "");
> + curl_easy_setopt(curl, CURLOPT_TCP_KEEPALIVE, 1L);
> +#if defined(CURLOPT_TCP_KEEPIDLE)
> + curl_easy_setopt(curl, CURLOPT_TCP_KEEPIDLE, 30L);
> +#endif
> +#if defined(CURLOPT_TCP_KEEPINTVL)
> + curl_easy_setopt(curl, CURLOPT_TCP_KEEPINTVL, 15L);
> +#endif
> return 0;
> }
>
> @@ -114,10 +168,10 @@ static int ocierofs_curl_setup_basic_auth(struct CURL *curl, const char *usernam
> return 0;
> }
>
> -static int ocierofs_curl_clear_auth(struct CURL *curl)
> +static int ocierofs_curl_clear_auth(struct ocierofs_ctx *ctx)
> {
> - curl_easy_setopt(curl, CURLOPT_USERPWD, NULL);
> - curl_easy_setopt(curl, CURLOPT_HTTPAUTH, CURLAUTH_NONE);
> + curl_easy_setopt(ctx->curl, CURLOPT_USERPWD, NULL);
> + curl_easy_setopt(ctx->curl, CURLOPT_HTTPAUTH, CURLAUTH_NONE);
> return 0;
> }
>
> @@ -176,20 +230,20 @@ static int ocierofs_curl_perform(struct CURL *curl, long *http_code_out)
> return 0;
> }
>
> -static int ocierofs_request_perform(struct erofs_oci *oci,
> - struct erofs_oci_request *req,
> - struct erofs_oci_response *resp)
> +static int ocierofs_request_perform(struct ocierofs_ctx *ctx,
> +
...
>
> @@ -204,15 +258,10 @@ static int ocierofs_request_perform(struct erofs_oci *oci,
> * @realm_out: pointer to store realm value
> * @service_out: pointer to store service value
> * @scope_out: pointer to store scope value
> - *
> - * Parse Bearer authentication header and extract realm, service, and scope
> - * parameters for subsequent token requests.
> - *
> - * Return: 0 on success, negative errno on failure
Why do we drop this comment?
> */
> static int ocierofs_parse_auth_header(const char *auth_header,
> - char **realm_out, char **service_out,
> - char **scope_out)
> + char **realm_out, char **service_out,
> + char **scope_out)
Let's not adjust lines which are unrelated to the commit.
> {
> char *realm = NULL, *service = NULL, *scope = NULL;
> static const char * const param_names[] = {"realm=", "service=", "scope="};
> @@ -221,7 +270,6 @@ static int ocierofs_parse_auth_header(const char *auth_header,
> const char *p;
> int i, ret = 0;
>
> - // https://datatracker.ietf.org/doc/html/rfc6750#section-3
Same here.
> if (strncmp(auth_header, "Bearer ", strlen("Bearer ")))
> return -EINVAL;
>
> @@ -229,7 +277,6 @@ static int ocierofs_parse_auth_header(const char *auth_header,
> if (!header_copy)
> return -ENOMEM;
>
> - /* Clean up header: replace newlines with spaces and remove double spaces */
Same here.
If they are unrelated to the new feature, let's refine
ocierofs_parse_auth_header() later.
> for (char *q = header_copy; *q; q++) {
> if (*q == '\n' || *q == '\r')
> *q = ' ';
> @@ -277,22 +324,9 @@ out:
> return ret;
> }
>
> -/**
> - * ocierofs_extract_www_auth_info - Extract WWW-Authenticate header information
> - * @resp_data: HTTP response data containing headers
> - * @realm_out: pointer to store realm value (optional)
> - * @service_out: pointer to store service value (optional)
> - * @scope_out: pointer to store scope value (optional)
> - *
> - * Extract realm, service, and scope from WWW-Authenticate header in HTTP response.
> - * This function handles the common pattern of parsing WWW-Authenticate headers
> - * that appears in multiple places in the OCI authentication flow.
> - *
> - * Return: 0 on success, negative errno on failure
> - */
Same here.
> static int ocierofs_extract_www_auth_info(const char *resp_data,
> - char **realm_out, char **service_out,
> - char **scope_out)
> + char **realm_out, char **service_out,
> + char **scope_out)
Same here.
> {> char *www_auth;
> char *line_end;
> @@ -336,29 +370,12 @@ static int ocierofs_extract_www_auth_info(const char *resp_data,
> return ret;
> }
>
> -/**
> - * ocierofs_get_auth_token_with_url - Get authentication token from auth server
> - * @oci: OCI client structure
> - * @auth_url: authentication server URL
> - * @service: service name for authentication
> - * @repository: repository name
> - * @username: username for basic auth (optional)
> - * @password: password for basic auth (optional)
> - *
> - * Request authentication token from the specified auth server URL using
> - * basic authentication if credentials are provided.
> - *
> - * Return: authentication header string on success, ERR_PTR on failure
> - */
> -static char *ocierofs_get_auth_token_with_url(struct erofs_oci *oci,
> - const char *auth_url,
> - const char *service,
> - const char *repository,
> - const char *username,
> - const char *password)
> +static char *ocierofs_get_auth_token_with_url(struct ocierofs_ctx *ctx, const char *auth_url,
> + const char *service, const char *repository,
> + const char *username, const char *password)
Since the arguments are changed, I'm fine with this change.
> {
> - struct erofs_oci_request req = {};
> - struct erofs_oci_response resp = {};
> + struct ocierofs_request req = {};
> + struct ocierofs_response resp = {};
> json_object *root, *token_obj, *access_token_obj;
> const char *token;
> char *auth_header = NULL;
> @@ -373,40 +390,36 @@ static char *ocierofs_get_auth_token_with_url(struct erofs_oci *oci,
> }
>
> if (username && password && *username) {
> - ret = ocierofs_curl_setup_basic_auth(oci->curl, username,
> - password);
> + ret = ocierofs_curl_setup_basic_auth(ctx->curl, username,
> + password);
Password can be aligned with `(` in the previous line?
> if (ret)
> goto out_url;
> }
>
> - ret = ocierofs_request_perform(oci, &req, &resp);
> - ocierofs_curl_clear_auth(oci->curl);
> + ret = ocierofs_request_perform(ctx, &req, &resp);
> + ocierofs_curl_clear_auth(ctx);
> if (ret)
> goto out_url;
>
> if (!resp.data) {
> - erofs_err("empty response from auth server");
Why drop this?
> ret = -EINVAL;
> goto out_url;
> }
>
> root = json_tokener_parse(resp.data);
> if (!root) {
> - erofs_err("failed to parse auth response");
Why drop this?
> ret = -EINVAL;
> - goto out_url;
> + goto out_json;
> }
>
> if (!json_object_object_get_ex(root, "token", &token_obj) &&
> !json_object_object_get_ex(root, "access_token", &access_token_obj)) {
> - erofs_err("no token found in auth response");
Why drop this?
> ret = -EINVAL;
> goto out_json;
> }
>
> token = json_object_get_string(token_obj ? token_obj : access_token_obj);
> if (!token) {
> - erofs_err("invalid token in auth response");
Why drop this?
...
> auth_service = discovered_service ? discovered_service : service;
> - auth_header = ocierofs_get_auth_token_with_url(oci, discovered_auth_url,
> - auth_service, repository,
> - username, password);
> + auth_header = ocierofs_get_auth_token_with_url(ctx, discovered_auth_url,
> + auth_service, repository,
> + username, password);
Same here.
> free(discovered_auth_url);
> free(discovered_service);
> if (!IS_ERR(auth_header))
> @@ -544,9 +555,9 @@ static char *ocierofs_get_auth_token(struct erofs_oci *oci, const char *registry
> if (asprintf(&auth_url, auth_patterns[i], registry) < 0)
> continue;
>
> - auth_header = ocierofs_get_auth_token_with_url(oci, auth_url,
> - service, repository,
> - username, password);
> + auth_header = ocierofs_get_auth_token_with_url(ctx, auth_url,
> + service, repository,
> + username, password);
Same here.
> free(auth_url);
>
> if (!IS_ERR(auth_header))
> @@ -557,24 +568,21 @@ static char *ocierofs_get_auth_token(struct erofs_oci *oci, const char *registry
> return ERR_PTR(-ENOENT);
> }
>
> -static char *ocierofs_get_manifest_digest(struct erofs_oci *oci,
> - const char *registry,
> - const char *repository, const char *tag,
> - const char *platform,
> - const char *auth_header)
> +static char *ocierofs_get_manifest_digest(struct ocierofs_ctx *ctx,
> + const char *registry,
> + const char *repository, const char *tag,
> + const char *platform,
> + const char *auth_header)
> {
> - struct erofs_oci_request req = {};
> - struct erofs_oci_response resp = {};
> + struct ocierofs_request req = {};
> + struct ocierofs_response resp = {};
> json_object *root, *manifests, *manifest, *platform_obj, *arch_obj;
> json_object *os_obj, *digest_obj, *schema_obj, *media_type_obj;
> char *digest = NULL;
> const char *api_registry;
> int ret = 0, len, i;
>
> - if (!registry || !repository || !tag || !platform)
> - return ERR_PTR(-EINVAL);
> -
> - api_registry = (!strcmp(registry, DOCKER_REGISTRY)) ? DOCKER_API_REGISTRY : registry;
> + api_registry = ocierofs_get_api_registry(registry);
> if (asprintf(&req.url, "https://%s/v2/%s/manifests/%s",
> api_registry, repository, tag) < 0)
> return ERR_PTR(-ENOMEM);
> @@ -584,22 +592,20 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci,
>
> req.headers = curl_slist_append(req.headers,
> "Accept: " DOCKER_MEDIATYPE_MANIFEST_LIST ","
> - OCI_MEDIATYPE_INDEX "," DOCKER_MEDIATYPE_MANIFEST_V1 ","
> - DOCKER_MEDIATYPE_MANIFEST_V2);
> + OCI_MEDIATYPE_INDEX "," OCI_MEDIATYPE_MANIFEST ","
> + DOCKER_MEDIATYPE_MANIFEST_V1 "," DOCKER_MEDIATYPE_MANIFEST_V2);
>
> - ret = ocierofs_request_perform(oci, &req, &resp);
> + ret = ocierofs_request_perform(ctx, &req, &resp);
> if (ret)
> goto out;
>
> if (!resp.data) {
> - erofs_err("empty response from manifest request");
Same here.
> ret = -EINVAL;
> goto out;
> }
>
> root = json_tokener_parse(resp.data);
> if (!root) {
> - erofs_err("failed to parse manifest JSON");
Same here.
> ret = -EINVAL;
> goto out;
> }
> @@ -615,8 +621,7 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci,
> if (json_object_object_get_ex(root, "mediaType", &media_type_obj)) {
> const char *media_type = json_object_get_string(media_type_obj);
>
> - if (!strcmp(media_type, DOCKER_MEDIATYPE_MANIFEST_V2) ||
> - !strcmp(media_type, OCI_MEDIATYPE_MANIFEST)) {
> + if (ocierofs_is_manifest(media_type)) {
> digest = strdup(tag);
> ret = 0;
> goto out_json;
> @@ -624,7 +629,6 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci,
> }
>
> if (!json_object_object_get_ex(root, "manifests", &manifests)) {
> - erofs_err("no manifests found in manifest list");
Same here.
> ret = -EINVAL;
> goto out_json;
> }
> @@ -634,9 +638,9 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci,
> manifest = json_object_array_get_idx(manifests, i);
>
> if (json_object_object_get_ex(manifest, "platform",
> - &platform_obj) &&
> + &platform_obj) &&
No need to change.
> json_object_object_get_ex(platform_obj, "architecture",
> - &arch_obj) &&
> + &arch_obj) &&
No need to change.
...
> - }
> + free(oci_cfg->password);
> + oci_cfg->password = strdup(p);
> + if (!oci_cfg->password)
> + return -ENOMEM;
> } else {
> - erofs_err("invalid --oci value %s", opt);
> + erofs_err("mkfs: invalid --oci value %s", opt);
Unneeded `mkfs:`.
Thanks,
Gao Xiang
^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH v5 0/3] erofs-utils: refactor OCI and add NBD-backed OCI image mounting
2025-09-01 5:10 [PATCH v1] erofs-utils: refactor OCI code for better modularity ChengyuZhu6
` (4 preceding siblings ...)
2025-09-04 5:33 ` [PATCH v4 0/3] erofs-utils: refactor OCI and " ChengyuZhu6
@ 2025-09-04 6:36 ` ChengyuZhu6
2025-09-04 6:36 ` [PATCH v5 1/3] erofs-utils:mkfs: move parse_oci_ref to lib ChengyuZhu6
` (3 more replies)
5 siblings, 4 replies; 19+ messages in thread
From: ChengyuZhu6 @ 2025-09-04 6:36 UTC (permalink / raw)
To: linux-erofs; +Cc: xiang, hsiangkao, Chengyu Zhu
From: Chengyu Zhu <hudsonzhu@tencent.com>
This series refactors the OCI handling in erofs-utils and adds NBD-backed
mounting of OCI images. It enables mounting EROFS container images directly
from registries without pre-downloading.
Chengyu Zhu (3):
erofs-utils:mkfs: move parse_oci_ref to lib
erofs-utils: refactor OCI code for better modularity
erofs-utils: add NBD-backed OCI image mounting
lib/liberofs_oci.h | 88 ++--
lib/remotes/oci.c | 1029 +++++++++++++++++++++++++++++++-------------
mkfs/main.c | 186 ++------
mount/Makefile.am | 2 +-
mount/main.c | 257 +++++++++--
5 files changed, 1013 insertions(+), 549 deletions(-)
--
2.51.0
^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH v5 1/3] erofs-utils:mkfs: move parse_oci_ref to lib
2025-09-04 6:36 ` [PATCH v5 0/3] erofs-utils: refactor OCI and " ChengyuZhu6
@ 2025-09-04 6:36 ` ChengyuZhu6
2025-09-04 6:36 ` [PATCH v5 2/3] erofs-utils: refactor OCI code for better modularity ChengyuZhu6
` (2 subsequent siblings)
3 siblings, 0 replies; 19+ messages in thread
From: ChengyuZhu6 @ 2025-09-04 6:36 UTC (permalink / raw)
To: linux-erofs; +Cc: xiang, hsiangkao, Chengyu Zhu
From: Chengyu Zhu <hudsonzhu@tencent.com>
move parse_oci_ref to oci.c.
Signed-off-by: Chengyu Zhu <hudsonzhu@tencent.com>
---
lib/liberofs_oci.h | 12 +++--
lib/remotes/oci.c | 110 +++++++++++++++++++++++++++++++++++++++++
mkfs/main.c | 121 +--------------------------------------------
3 files changed, 120 insertions(+), 123 deletions(-)
diff --git a/lib/liberofs_oci.h b/lib/liberofs_oci.h
index 3a8108b..bce63ef 100644
--- a/lib/liberofs_oci.h
+++ b/lib/liberofs_oci.h
@@ -8,9 +8,6 @@
#include <stdbool.h>
-#define DOCKER_REGISTRY "docker.io"
-#define DOCKER_API_REGISTRY "registry-1.docker.io"
-
#ifdef __cplusplus
extern "C" {
#endif
@@ -79,6 +76,15 @@ void ocierofs_cleanup(struct erofs_oci *oci);
*/
int erofs_oci_params_set_string(char **field, const char *value);
+/*
+ * ocierofs_parse_ref - Parse OCI image reference string
+ * @oci: OCI client structure
+ * @ref_str: OCI image reference string
+ *
+ * Return: 0 on success, negative errno on failure
+ */
+int ocierofs_parse_ref(struct erofs_oci *oci, const char *ref_str);
+
/*
* ocierofs_build_trees - Build file trees from OCI container image layers
* @root: root inode to build the file tree under
diff --git a/lib/remotes/oci.c b/lib/remotes/oci.c
index 0fb8c1f..e6f0c23 100644
--- a/lib/remotes/oci.c
+++ b/lib/remotes/oci.c
@@ -21,6 +21,9 @@
#include "liberofs_oci.h"
#include "liberofs_private.h"
+#define DOCKER_REGISTRY "docker.io"
+#define DOCKER_API_REGISTRY "registry-1.docker.io"
+
#define DOCKER_MEDIATYPE_MANIFEST_V2 \
"application/vnd.docker.distribution.manifest.v2+json"
#define DOCKER_MEDIATYPE_MANIFEST_V1 \
@@ -1066,3 +1069,110 @@ int erofs_oci_params_set_string(char **field, const char *value)
*field = new_value;
return 0;
}
+
+int ocierofs_parse_ref(struct erofs_oci *oci, const char *ref_str)
+{
+ char *slash, *colon, *dot;
+ const char *repo_part;
+ size_t len;
+
+ slash = strchr(ref_str, '/');
+ if (slash) {
+ dot = strchr(ref_str, '.');
+ if (dot && dot < slash) {
+ char *registry_str;
+
+ len = slash - ref_str;
+ registry_str = strndup(ref_str, len);
+
+ if (!registry_str) {
+ erofs_err("failed to allocate memory for registry");
+ return -ENOMEM;
+ }
+ if (erofs_oci_params_set_string(&oci->params.registry,
+ registry_str)) {
+ free(registry_str);
+ erofs_err("failed to set registry");
+ return -ENOMEM;
+ }
+ free(registry_str);
+ repo_part = slash + 1;
+ } else {
+ repo_part = ref_str;
+ }
+ } else {
+ repo_part = ref_str;
+ }
+
+ colon = strchr(repo_part, ':');
+ if (colon) {
+ char *repo_str;
+
+ len = colon - repo_part;
+ repo_str = strndup(repo_part, len);
+
+ if (!repo_str) {
+ erofs_err("failed to allocate memory for repository");
+ return -ENOMEM;
+ }
+
+ if (!strchr(repo_str, '/') &&
+ (!strcmp(oci->params.registry, DOCKER_API_REGISTRY) ||
+ !strcmp(oci->params.registry, DOCKER_REGISTRY))) {
+ char *full_repo;
+
+ if (asprintf(&full_repo, "library/%s", repo_str) == -1) {
+ free(repo_str);
+ erofs_err("failed to allocate memory for full repository name");
+ return -ENOMEM;
+ }
+ free(repo_str);
+ repo_str = full_repo;
+ }
+
+ if (erofs_oci_params_set_string(&oci->params.repository,
+ repo_str)) {
+ free(repo_str);
+ erofs_err("failed to set repository");
+ return -ENOMEM;
+ }
+ free(repo_str);
+
+ if (erofs_oci_params_set_string(&oci->params.tag,
+ colon + 1)) {
+ erofs_err("failed to set tag");
+ return -ENOMEM;
+ }
+ } else {
+ char *repo_str = strdup(repo_part);
+
+ if (!repo_str) {
+ erofs_err("failed to allocate memory for repository");
+ return -ENOMEM;
+ }
+
+ if (!strchr(repo_str, '/') &&
+ (!strcmp(oci->params.registry, DOCKER_API_REGISTRY) ||
+ !strcmp(oci->params.registry, DOCKER_REGISTRY))) {
+ char *full_repo;
+
+ if (asprintf(&full_repo, "library/%s", repo_str) == -1) {
+ free(repo_str);
+ erofs_err("failed to allocate memory for full repository name");
+ return -ENOMEM;
+ }
+ free(repo_str);
+ repo_str = full_repo;
+ }
+
+ if (erofs_oci_params_set_string(&oci->params.repository,
+ repo_str)) {
+ free(repo_str);
+ erofs_err("failed to set repository");
+ return -ENOMEM;
+ }
+ free(repo_str);
+ }
+
+ return 0;
+}
diff --git a/mkfs/main.c b/mkfs/main.c
index bc895f1..064392d 100644
--- a/mkfs/main.c
+++ b/mkfs/main.c
@@ -763,125 +763,6 @@ static int mkfs_parse_oci_options(char *options_str)
return 0;
}
-
-/**
- * mkfs_parse_oci_ref - Parse OCI image reference string
- * @ref_str: OCI image reference in various formats
- *
- * Parse OCI image reference which can be in formats:
- * - registry.example.com/namespace/repo:tag
- * - namespace/repo:tag (uses default registry)
- * - repo:tag (adds library/ prefix for Docker Hub)
- * - repo (uses default tag "latest")
- *
- * Return: 0 on success, negative errno on failure
- */
-static int mkfs_parse_oci_ref(const char *ref_str)
-{
- char *slash, *colon, *dot;
- const char *repo_part;
- size_t len;
-
- slash = strchr(ref_str, '/');
- if (slash) {
- dot = strchr(ref_str, '.');
- if (dot && dot < slash) {
- char *registry_str;
-
- len = slash - ref_str;
- registry_str = strndup(ref_str, len);
-
- if (!registry_str) {
- erofs_err("failed to allocate memory for registry");
- return -ENOMEM;
- }
- if (erofs_oci_params_set_string(&ocicfg.params.registry,
- registry_str)) {
- free(registry_str);
- erofs_err("failed to set registry");
- return -ENOMEM;
- }
- free(registry_str);
- repo_part = slash + 1;
- } else {
- repo_part = ref_str;
- }
- } else {
- repo_part = ref_str;
- }
-
- colon = strchr(repo_part, ':');
- if (colon) {
- char *repo_str;
-
- len = colon - repo_part;
- repo_str = strndup(repo_part, len);
-
- if (!repo_str) {
- erofs_err("failed to allocate memory for repository");
- return -ENOMEM;
- }
-
- if (!strchr(repo_str, '/') &&
- (!strcmp(ocicfg.params.registry, DOCKER_API_REGISTRY) ||
- !strcmp(ocicfg.params.registry, DOCKER_REGISTRY))) {
- char *full_repo;
-
- if (asprintf(&full_repo, "library/%s", repo_str) == -1) {
- free(repo_str);
- erofs_err("failed to allocate memory for full repository name");
- return -ENOMEM;
- }
- free(repo_str);
- repo_str = full_repo;
- }
-
- if (erofs_oci_params_set_string(&ocicfg.params.repository,
- repo_str)) {
- free(repo_str);
- erofs_err("failed to set repository");
- return -ENOMEM;
- }
- free(repo_str);
-
- if (erofs_oci_params_set_string(&ocicfg.params.tag,
- colon + 1)) {
- erofs_err("failed to set tag");
- return -ENOMEM;
- }
- } else {
- char *repo_str = strdup(repo_part);
-
- if (!repo_str) {
- erofs_err("failed to allocate memory for repository");
- return -ENOMEM;
- }
-
- if (!strchr(repo_str, '/') &&
- (!strcmp(ocicfg.params.registry, DOCKER_API_REGISTRY) ||
- !strcmp(ocicfg.params.registry, DOCKER_REGISTRY))) {
- char *full_repo;
-
- if (asprintf(&full_repo, "library/%s", repo_str) == -1) {
- free(repo_str);
- erofs_err("failed to allocate memory for full repository name");
- return -ENOMEM;
- }
- free(repo_str);
- repo_str = full_repo;
- }
-
- if (erofs_oci_params_set_string(&ocicfg.params.repository,
- repo_str)) {
- free(repo_str);
- erofs_err("failed to set repository");
- return -ENOMEM;
- }
- free(repo_str);
- }
-
- return 0;
-}
#endif
static int mkfs_parse_one_compress_alg(char *alg,
@@ -1958,7 +1839,7 @@ int main(int argc, char **argv)
err = mkfs_parse_oci_options(mkfs_oci_options);
if (err)
goto exit;
- err = mkfs_parse_oci_ref(cfg.c_src_path);
+ err = ocierofs_parse_ref(&ocicfg, cfg.c_src_path);
if (err)
goto exit;
--
2.51.0
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH v5 2/3] erofs-utils: refactor OCI code for better modularity
2025-09-04 6:36 ` [PATCH v5 0/3] erofs-utils: refactor OCI and " ChengyuZhu6
2025-09-04 6:36 ` [PATCH v5 1/3] erofs-utils:mkfs: move parse_oci_ref to lib ChengyuZhu6
@ 2025-09-04 6:36 ` ChengyuZhu6
2025-09-04 6:36 ` [PATCH v5 3/3] erofs-utils: add NBD-backed OCI image mounting ChengyuZhu6
2025-09-04 7:19 ` [PATCH v5 0/3] erofs-utils: refactor OCI and " Gao Xiang
3 siblings, 0 replies; 19+ messages in thread
From: ChengyuZhu6 @ 2025-09-04 6:36 UTC (permalink / raw)
To: linux-erofs; +Cc: xiang, hsiangkao, Chengyu Zhu
From: Chengyu Zhu <hudsonzhu@tencent.com>
Refactor OCI code to improve code organization and maintainability:
Key changes:
- Add `ocierofs_get_api_registry()` and unify API endpoint selection.
- Implement Bearer token discovery with Basic fallback; cache auth header.
- Parse layer metadata (digest, mediaType, size) and add a proper free helper.
- Split blob download from tar processing; process tar via a temp fd.
- Rework init/teardown into `ocierofs_init()` and `ocierofs_ctx_cleanup()`.
- Update mkfs to use `struct ocierofs_config` and new `--oci` parsing.
Signed-off-by: Chengyu Zhu <hudsonzhu@tencent.com>
---
lib/liberofs_oci.h | 82 ++---
lib/remotes/oci.c | 891 ++++++++++++++++++++++++---------------------
mkfs/main.c | 67 ++--
3 files changed, 535 insertions(+), 505 deletions(-)
diff --git a/lib/liberofs_oci.h b/lib/liberofs_oci.h
index bce63ef..ebb427a 100644
--- a/lib/liberofs_oci.h
+++ b/lib/liberofs_oci.h
@@ -12,15 +12,11 @@
extern "C" {
#endif
-struct erofs_inode;
-struct CURL;
struct erofs_importer;
-/**
- * struct erofs_oci_params - OCI configuration parameters
- * @registry: registry hostname (e.g., "registry-1.docker.io")
- * @repository: image repository (e.g., "library/ubuntu")
- * @tag: image tag or digest (e.g., "latest" or sha256:...)
+/*
+ * struct ocierofs_config - OCI configuration structure
+ * @image_ref: OCI image reference (e.g., "ubuntu:latest", "myregistry.com/app:v1.0")
* @platform: target platform in "os/arch" format (e.g., "linux/amd64")
* @username: username for authentication (optional)
* @password: password for authentication (optional)
@@ -30,69 +26,45 @@ struct erofs_importer;
* location, image identification, platform specification, and authentication
* credentials.
*/
-struct erofs_oci_params {
- char *registry;
- char *repository;
- char *tag;
+struct ocierofs_config {
+ char *image_ref;
char *platform;
char *username;
char *password;
int layer_index;
};
-/**
- * struct erofs_oci - Combined OCI client structure
- * @curl: CURL handle for HTTP requests
- * @params: OCI configuration parameters
- *
- * Main OCI client structure combining CURL HTTP client with
- * OCI-specific configuration parameters.
- */
-struct erofs_oci {
- struct CURL *curl;
- struct erofs_oci_params params;
+struct ocierofs_layer_info {
+ char *digest;
+ char *media_type;
+ u64 size;
};
-/*
- * ocierofs_init - Initialize OCI client with default parameters
- * @oci: OCI client structure to initialize
- *
- * Return: 0 on success, negative errno on failure
- */
-int ocierofs_init(struct erofs_oci *oci);
-
-/*
- * ocierofs_cleanup - Clean up OCI client and free allocated resources
- * @oci: OCI client structure to clean up
- */
-void ocierofs_cleanup(struct erofs_oci *oci);
-
-/*
- * erofs_oci_params_set_string - Set a string field with dynamic allocation
- * @field: pointer to the string field to set
- * @value: string value to set
- *
- * Return: 0 on success, negative errno on failure
- */
-int erofs_oci_params_set_string(char **field, const char *value);
+struct ocierofs_ctx {
+ struct CURL *curl;
+ char *auth_header;
+ bool using_basic;
+ char *registry;
+ char *repository;
+ char *platform;
+ char *tag;
+ char *manifest_digest;
+ struct ocierofs_layer_info **layers;
+ int layer_index;
+ int layer_count;
+};
-/*
- * ocierofs_parse_ref - Parse OCI image reference string
- * @oci: OCI client structure
- * @ref_str: OCI image reference string
- *
- * Return: 0 on success, negative errno on failure
- */
-int ocierofs_parse_ref(struct erofs_oci *oci, const char *ref_str);
+int ocierofs_init(struct ocierofs_ctx *ctx, const struct ocierofs_config *config);
/*
* ocierofs_build_trees - Build file trees from OCI container image layers
- * @root: root inode to build the file tree under
- * @oci: OCI client structure with configured parameters
+ * @importer: erofs importer to populate
+ * @cfg: oci configuration
*
* Return: 0 on success, negative errno on failure
*/
-int ocierofs_build_trees(struct erofs_importer *importer, struct erofs_oci *oci);
+int ocierofs_build_trees(struct erofs_importer *importer,
+ const struct ocierofs_config *cfg);
#ifdef __cplusplus
}
diff --git a/lib/remotes/oci.c b/lib/remotes/oci.c
index e6f0c23..26e83d7 100644
--- a/lib/remotes/oci.c
+++ b/lib/remotes/oci.c
@@ -33,28 +33,67 @@
#define OCI_MEDIATYPE_MANIFEST "application/vnd.oci.image.manifest.v1+json"
#define OCI_MEDIATYPE_INDEX "application/vnd.oci.image.index.v1+json"
-struct erofs_oci_request {
+struct ocierofs_request {
char *url;
struct curl_slist *headers;
};
-struct erofs_oci_response {
+struct ocierofs_response {
char *data;
size_t size;
long http_code;
};
-struct erofs_oci_stream {
- struct erofs_tarfile tarfile;
+struct ocierofs_stream {
const char *digest;
int blobfd;
};
+static inline const char *ocierofs_get_api_registry(const char *registry)
+{
+ if (!registry)
+ return DOCKER_API_REGISTRY;
+ return !strcmp(registry, DOCKER_REGISTRY) ? DOCKER_API_REGISTRY : registry;
+}
+
+static inline bool ocierofs_is_manifest_list(const char *media_type)
+{
+ return media_type && (!strcmp(media_type, DOCKER_MEDIATYPE_MANIFEST_LIST) ||
+ !strcmp(media_type, OCI_MEDIATYPE_INDEX));
+}
+
+static inline bool ocierofs_is_manifest(const char *media_type)
+{
+ return media_type && (!strcmp(media_type, DOCKER_MEDIATYPE_MANIFEST_V2) ||
+ !strcmp(media_type, OCI_MEDIATYPE_MANIFEST));
+}
+
+static inline void ocierofs_request_cleanup(struct ocierofs_request *req)
+{
+ if (!req)
+ return;
+ if (req->headers)
+ curl_slist_free_all(req->headers);
+ free(req->url);
+ req->url = NULL;
+ req->headers = NULL;
+}
+
+static inline void ocierofs_response_cleanup(struct ocierofs_response *resp)
+{
+ if (!resp)
+ return;
+ free(resp->data);
+ resp->data = NULL;
+ resp->size = 0;
+ resp->http_code = 0;
+}
+
static size_t ocierofs_write_callback(void *contents, size_t size,
size_t nmemb, void *userp)
{
size_t realsize = size * nmemb;
- struct erofs_oci_response *resp = userp;
+ struct ocierofs_response *resp = userp;
char *ptr;
if (!resp->data)
@@ -75,16 +114,23 @@ static size_t ocierofs_write_callback(void *contents, size_t size,
static size_t ocierofs_layer_write_callback(void *contents, size_t size,
size_t nmemb, void *userp)
{
- struct erofs_oci_stream *stream = userp;
+ struct ocierofs_stream *stream = userp;
size_t realsize = size * nmemb;
+ const char *buf = contents;
+ size_t written = 0;
if (stream->blobfd < 0)
return 0;
- if (write(stream->blobfd, contents, realsize) != realsize) {
- erofs_err("failed to write layer data for layer %s",
- stream->digest);
- return 0;
+ while (written < realsize) {
+ ssize_t n = write(stream->blobfd, buf + written, realsize - written);
+
+ if (n < 0) {
+ erofs_err("failed to write layer data for layer %s",
+ stream->digest);
+ return 0;
+ }
+ written += n;
}
return realsize;
}
@@ -96,6 +142,14 @@ static int ocierofs_curl_setup_common_options(struct CURL *curl)
curl_easy_setopt(curl, CURLOPT_TIMEOUT, 120L);
curl_easy_setopt(curl, CURLOPT_NOSIGNAL, 1L);
curl_easy_setopt(curl, CURLOPT_USERAGENT, "ocierofs/" PACKAGE_VERSION);
+ curl_easy_setopt(curl, CURLOPT_ACCEPT_ENCODING, "");
+ curl_easy_setopt(curl, CURLOPT_TCP_KEEPALIVE, 1L);
+#if defined(CURLOPT_TCP_KEEPIDLE)
+ curl_easy_setopt(curl, CURLOPT_TCP_KEEPIDLE, 30L);
+#endif
+#if defined(CURLOPT_TCP_KEEPINTVL)
+ curl_easy_setopt(curl, CURLOPT_TCP_KEEPINTVL, 15L);
+#endif
return 0;
}
@@ -114,10 +168,10 @@ static int ocierofs_curl_setup_basic_auth(struct CURL *curl, const char *usernam
return 0;
}
-static int ocierofs_curl_clear_auth(struct CURL *curl)
+static int ocierofs_curl_clear_auth(struct ocierofs_ctx *ctx)
{
- curl_easy_setopt(curl, CURLOPT_USERPWD, NULL);
- curl_easy_setopt(curl, CURLOPT_HTTPAUTH, CURLAUTH_NONE);
+ curl_easy_setopt(ctx->curl, CURLOPT_USERPWD, NULL);
+ curl_easy_setopt(ctx->curl, CURLOPT_HTTPAUTH, CURLAUTH_NONE);
return 0;
}
@@ -176,20 +230,20 @@ static int ocierofs_curl_perform(struct CURL *curl, long *http_code_out)
return 0;
}
-static int ocierofs_request_perform(struct erofs_oci *oci,
- struct erofs_oci_request *req,
- struct erofs_oci_response *resp)
+static int ocierofs_request_perform(struct ocierofs_ctx *ctx,
+ struct ocierofs_request *req,
+ struct ocierofs_response *resp)
{
int ret;
- ret = ocierofs_curl_setup_rq(oci->curl, req->url,
+ ret = ocierofs_curl_setup_rq(ctx->curl, req->url,
OCIEROFS_HTTP_GET, req->headers,
ocierofs_write_callback, resp,
NULL, NULL);
if (ret)
return ret;
- ret = ocierofs_curl_perform(oci->curl, &resp->http_code);
+ ret = ocierofs_curl_perform(ctx->curl, &resp->http_code);
if (ret)
return ret;
@@ -338,7 +392,7 @@ static int ocierofs_extract_www_auth_info(const char *resp_data,
/**
* ocierofs_get_auth_token_with_url - Get authentication token from auth server
- * @oci: OCI client structure
+ * @ctx: OCI context structure
* @auth_url: authentication server URL
* @service: service name for authentication
* @repository: repository name
@@ -350,15 +404,12 @@ static int ocierofs_extract_www_auth_info(const char *resp_data,
*
* Return: authentication header string on success, ERR_PTR on failure
*/
-static char *ocierofs_get_auth_token_with_url(struct erofs_oci *oci,
- const char *auth_url,
- const char *service,
- const char *repository,
- const char *username,
- const char *password)
+static char *ocierofs_get_auth_token_with_url(struct ocierofs_ctx *ctx, const char *auth_url,
+ const char *service, const char *repository,
+ const char *username, const char *password)
{
- struct erofs_oci_request req = {};
- struct erofs_oci_response resp = {};
+ struct ocierofs_request req = {};
+ struct ocierofs_response resp = {};
json_object *root, *token_obj, *access_token_obj;
const char *token;
char *auth_header = NULL;
@@ -373,14 +424,14 @@ static char *ocierofs_get_auth_token_with_url(struct erofs_oci *oci,
}
if (username && password && *username) {
- ret = ocierofs_curl_setup_basic_auth(oci->curl, username,
+ ret = ocierofs_curl_setup_basic_auth(ctx->curl, username,
password);
if (ret)
goto out_url;
}
- ret = ocierofs_request_perform(oci, &req, &resp);
- ocierofs_curl_clear_auth(oci->curl);
+ ret = ocierofs_request_perform(ctx, &req, &resp);
+ ocierofs_curl_clear_auth(ctx);
if (ret)
goto out_url;
@@ -394,7 +445,7 @@ static char *ocierofs_get_auth_token_with_url(struct erofs_oci *oci,
if (!root) {
erofs_err("failed to parse auth response");
ret = -EINVAL;
- goto out_url;
+ goto out_json;
}
if (!json_object_object_get_ex(root, "token", &token_obj) &&
@@ -419,16 +470,16 @@ static char *ocierofs_get_auth_token_with_url(struct erofs_oci *oci,
out_json:
json_object_put(root);
out_url:
- free(req.url);
- free(resp.data);
+ ocierofs_response_cleanup(&resp);
+ ocierofs_request_cleanup(&req);
return ret ? ERR_PTR(ret) : auth_header;
}
-static char *ocierofs_discover_auth_endpoint(struct erofs_oci *oci,
+static char *ocierofs_discover_auth_endpoint(struct ocierofs_ctx *ctx,
const char *registry,
const char *repository)
{
- struct erofs_oci_response resp = {};
+ struct ocierofs_response resp = {};
char *realm = NULL;
char *service = NULL;
char *result = NULL;
@@ -437,20 +488,20 @@ static char *ocierofs_discover_auth_endpoint(struct erofs_oci *oci,
CURLcode res;
long http_code;
- api_registry = (!strcmp(registry, DOCKER_REGISTRY)) ? DOCKER_API_REGISTRY : registry;
+ api_registry = ocierofs_get_api_registry(registry);
if (asprintf(&test_url, "https://%s/v2/%s/manifests/nonexistent",
api_registry, repository) < 0)
return NULL;
- curl_easy_reset(oci->curl);
- ocierofs_curl_setup_common_options(oci->curl);
+ curl_easy_reset(ctx->curl);
+ ocierofs_curl_setup_common_options(ctx->curl);
- ocierofs_curl_setup_rq(oci->curl, test_url, OCIEROFS_HTTP_HEAD, NULL,
+ ocierofs_curl_setup_rq(ctx->curl, test_url, OCIEROFS_HTTP_HEAD, NULL,
NULL, NULL, ocierofs_write_callback, &resp);
- res = curl_easy_perform(oci->curl);
- curl_easy_getinfo(oci->curl, CURLINFO_RESPONSE_CODE, &http_code);
+ res = curl_easy_perform(ctx->curl);
+ curl_easy_getinfo(ctx->curl, CURLINFO_RESPONSE_CODE, &http_code);
if (res == CURLE_OK && (http_code == 401 || http_code == 403 ||
http_code == 404) && resp.data) {
@@ -461,12 +512,12 @@ static char *ocierofs_discover_auth_endpoint(struct erofs_oci *oci,
}
free(realm);
free(service);
- free(resp.data);
+ ocierofs_response_cleanup(&resp);
free(test_url);
return result;
}
-static char *ocierofs_get_auth_token(struct erofs_oci *oci, const char *registry,
+static char *ocierofs_get_auth_token(struct ocierofs_ctx *ctx, const char *registry,
const char *repository, const char *username,
const char *password)
{
@@ -487,35 +538,35 @@ static char *ocierofs_get_auth_token(struct erofs_oci *oci, const char *registry
!strcmp(registry, DOCKER_REGISTRY);
if (docker_reg) {
service = "registry.docker.io";
- auth_header = ocierofs_get_auth_token_with_url(oci,
+ auth_header = ocierofs_get_auth_token_with_url(ctx,
"https://auth.docker.io/token", service, repository,
username, password);
if (!IS_ERR(auth_header))
return auth_header;
}
- discovered_auth_url = ocierofs_discover_auth_endpoint(oci, registry, repository);
+ discovered_auth_url = ocierofs_discover_auth_endpoint(ctx, registry, repository);
if (discovered_auth_url) {
const char *api_registry, *auth_service;
- struct erofs_oci_response resp = {};
+ struct ocierofs_response resp = {};
char *test_url;
CURLcode res;
long http_code;
- api_registry = (!strcmp(registry, DOCKER_REGISTRY)) ? DOCKER_API_REGISTRY : registry;
+ api_registry = ocierofs_get_api_registry(registry);
if (asprintf(&test_url, "https://%s/v2/%s/manifests/nonexistent",
api_registry, repository) >= 0) {
- curl_easy_reset(oci->curl);
- ocierofs_curl_setup_common_options(oci->curl);
+ curl_easy_reset(ctx->curl);
+ ocierofs_curl_setup_common_options(ctx->curl);
- ocierofs_curl_setup_rq(oci->curl, test_url,
+ ocierofs_curl_setup_rq(ctx->curl, test_url,
OCIEROFS_HTTP_HEAD, NULL,
NULL, NULL,
ocierofs_write_callback, &resp);
- res = curl_easy_perform(oci->curl);
- curl_easy_getinfo(oci->curl, CURLINFO_RESPONSE_CODE, &http_code);
+ res = curl_easy_perform(ctx->curl);
+ curl_easy_getinfo(ctx->curl, CURLINFO_RESPONSE_CODE, &http_code);
if (res == CURLE_OK && (http_code == 401 || http_code == 403 ||
http_code == 404) && resp.data) {
@@ -524,12 +575,12 @@ static char *ocierofs_get_auth_token(struct erofs_oci *oci, const char *registry
ocierofs_extract_www_auth_info(resp.data, &realm, &discovered_service, NULL);
free(realm);
}
- free(resp.data);
+ ocierofs_response_cleanup(&resp);
free(test_url);
}
auth_service = discovered_service ? discovered_service : service;
- auth_header = ocierofs_get_auth_token_with_url(oci, discovered_auth_url,
+ auth_header = ocierofs_get_auth_token_with_url(ctx, discovered_auth_url,
auth_service, repository,
username, password);
free(discovered_auth_url);
@@ -544,7 +595,7 @@ static char *ocierofs_get_auth_token(struct erofs_oci *oci, const char *registry
if (asprintf(&auth_url, auth_patterns[i], registry) < 0)
continue;
- auth_header = ocierofs_get_auth_token_with_url(oci, auth_url,
+ auth_header = ocierofs_get_auth_token_with_url(ctx, auth_url,
service, repository,
username, password);
free(auth_url);
@@ -557,24 +608,21 @@ static char *ocierofs_get_auth_token(struct erofs_oci *oci, const char *registry
return ERR_PTR(-ENOENT);
}
-static char *ocierofs_get_manifest_digest(struct erofs_oci *oci,
+static char *ocierofs_get_manifest_digest(struct ocierofs_ctx *ctx,
const char *registry,
const char *repository, const char *tag,
const char *platform,
const char *auth_header)
{
- struct erofs_oci_request req = {};
- struct erofs_oci_response resp = {};
+ struct ocierofs_request req = {};
+ struct ocierofs_response resp = {};
json_object *root, *manifests, *manifest, *platform_obj, *arch_obj;
json_object *os_obj, *digest_obj, *schema_obj, *media_type_obj;
char *digest = NULL;
const char *api_registry;
int ret = 0, len, i;
- if (!registry || !repository || !tag || !platform)
- return ERR_PTR(-EINVAL);
-
- api_registry = (!strcmp(registry, DOCKER_REGISTRY)) ? DOCKER_API_REGISTRY : registry;
+ api_registry = ocierofs_get_api_registry(registry);
if (asprintf(&req.url, "https://%s/v2/%s/manifests/%s",
api_registry, repository, tag) < 0)
return ERR_PTR(-ENOMEM);
@@ -584,10 +632,10 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci,
req.headers = curl_slist_append(req.headers,
"Accept: " DOCKER_MEDIATYPE_MANIFEST_LIST ","
- OCI_MEDIATYPE_INDEX "," DOCKER_MEDIATYPE_MANIFEST_V1 ","
- DOCKER_MEDIATYPE_MANIFEST_V2);
+ OCI_MEDIATYPE_INDEX "," OCI_MEDIATYPE_MANIFEST ","
+ DOCKER_MEDIATYPE_MANIFEST_V1 "," DOCKER_MEDIATYPE_MANIFEST_V2);
- ret = ocierofs_request_perform(oci, &req, &resp);
+ ret = ocierofs_request_perform(ctx, &req, &resp);
if (ret)
goto out;
@@ -615,8 +663,7 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci,
if (json_object_object_get_ex(root, "mediaType", &media_type_obj)) {
const char *media_type = json_object_get_string(media_type_obj);
- if (!strcmp(media_type, DOCKER_MEDIATYPE_MANIFEST_V2) ||
- !strcmp(media_type, OCI_MEDIATYPE_MANIFEST)) {
+ if (ocierofs_is_manifest(media_type)) {
digest = strdup(tag);
ret = 0;
goto out_json;
@@ -658,34 +705,44 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci,
out_json:
json_object_put(root);
out:
- free(resp.data);
- if (req.headers)
- curl_slist_free_all(req.headers);
- free(req.url);
-
+ ocierofs_response_cleanup(&resp);
+ ocierofs_request_cleanup(&req);
return ret ? ERR_PTR(ret) : digest;
}
-static char **ocierofs_get_layers_info(struct erofs_oci *oci,
- const char *registry,
- const char *repository,
- const char *digest,
- const char *auth_header,
- int *layer_count)
+static void ocierofs_free_layers_info(struct ocierofs_layer_info **layers, int count)
{
- struct erofs_oci_request req = {};
- struct erofs_oci_response resp = {};
- json_object *root, *layers, *layer, *digest_obj;
- char **layers_info = NULL;
- const char *api_registry;
- int ret, len, i, j;
+ int i;
- if (!registry || !repository || !digest || !layer_count)
- return ERR_PTR(-EINVAL);
+ if (!layers)
+ return;
+
+ for (i = 0; i < count; i++) {
+ if (layers[i]) {
+ free(layers[i]->digest);
+ free(layers[i]->media_type);
+ free(layers[i]);
+ }
+ }
+ free(layers);
+}
+
+static struct ocierofs_layer_info **ocierofs_fetch_layers_info(struct ocierofs_ctx *ctx,
+ const char *registry,
+ const char *repository,
+ const char *digest,
+ const char *auth_header,
+ int *layer_count)
+{
+ struct ocierofs_request req = {};
+ struct ocierofs_response resp = {};
+ json_object *root, *layers, *layer, *digest_obj, *media_type_obj, *size_obj;
+ struct ocierofs_layer_info **layers_info = NULL;
+ const char *api_registry;
+ int ret, len, i;
*layer_count = 0;
- api_registry = (!strcmp(registry, DOCKER_REGISTRY) ?
- DOCKER_API_REGISTRY : registry);
+ api_registry = ocierofs_get_api_registry(registry);
if (asprintf(&req.url, "https://%s/v2/%s/manifests/%s",
api_registry, repository, digest) < 0)
@@ -697,7 +754,7 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci,
req.headers = curl_slist_append(req.headers,
"Accept: " OCI_MEDIATYPE_MANIFEST "," DOCKER_MEDIATYPE_MANIFEST_V2);
- ret = ocierofs_request_perform(oci, &req, &resp);
+ ret = ocierofs_request_perform(ctx, &req, &resp);
if (ret)
goto out;
@@ -723,12 +780,11 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci,
len = json_object_array_length(layers);
if (!len) {
- erofs_err("empty layer list in manifest");
ret = -EINVAL;
goto out_json;
}
- layers_info = calloc(len, sizeof(char *));
+ layers_info = calloc(len, sizeof(*layers_info));
if (!layers_info) {
ret = -ENOMEM;
goto out_json;
@@ -738,364 +794,196 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci,
layer = json_object_array_get_idx(layers, i);
if (!json_object_object_get_ex(layer, "digest", &digest_obj)) {
- erofs_err("failed to parse layer %d", i);
ret = -EINVAL;
goto out_free;
}
- layers_info[i] = strdup(json_object_get_string(digest_obj));
+ layers_info[i] = calloc(1, sizeof(**layers_info));
if (!layers_info[i]) {
ret = -ENOMEM;
goto out_free;
}
+ layers_info[i]->digest = strdup(json_object_get_string(digest_obj));
+ if (!layers_info[i]->digest) {
+ ret = -ENOMEM;
+ goto out_free;
+ }
+ if (json_object_object_get_ex(layer, "mediaType", &media_type_obj))
+ layers_info[i]->media_type = strdup(json_object_get_string(media_type_obj));
+ else
+ layers_info[i]->media_type = NULL;
+
+ if (json_object_object_get_ex(layer, "size", &size_obj))
+ layers_info[i]->size = json_object_get_int64(size_obj);
+ else
+ layers_info[i]->size = 0;
}
*layer_count = len;
json_object_put(root);
- free(resp.data);
- if (req.headers)
- curl_slist_free_all(req.headers);
- free(req.url);
+ ocierofs_response_cleanup(&resp);
+ ocierofs_request_cleanup(&req);
return layers_info;
out_free:
- if (layers_info) {
- for (j = 0; j < i; j++)
- free(layers_info[j]);
- }
- free(layers_info);
+ ocierofs_free_layers_info(layers_info, i);
out_json:
json_object_put(root);
out:
- free(resp.data);
- if (req.headers)
- curl_slist_free_all(req.headers);
- free(req.url);
+ ocierofs_response_cleanup(&resp);
+ ocierofs_request_cleanup(&req);
return ERR_PTR(ret);
}
-static int ocierofs_extract_layer(struct erofs_oci *oci, struct erofs_importer *importer,
- const char *digest, const char *auth_header)
+static int ocierofs_process_tar_stream(struct erofs_importer *importer, int fd)
{
- struct erofs_oci_request req = {};
- struct erofs_oci_stream stream = {};
- const char *api_registry;
- long http_code;
+ struct erofs_tarfile tarfile = {};
int ret;
- stream = (struct erofs_oci_stream) {
- .digest = digest,
- .blobfd = erofs_tmpfile(),
- };
- if (stream.blobfd < 0) {
- erofs_err("failed to create temporary file for %s", digest);
- return -errno;
- }
-
- api_registry = (!strcmp(oci->params.registry, DOCKER_REGISTRY)) ?
- DOCKER_API_REGISTRY : oci->params.registry;
- if (asprintf(&req.url, "https://%s/v2/%s/blobs/%s",
- api_registry, oci->params.repository, digest) == -1) {
- ret = -ENOMEM;
- goto out;
- }
-
- if (auth_header && strstr(auth_header, "Bearer"))
- req.headers = curl_slist_append(req.headers, auth_header);
+ init_list_head(&tarfile.global.xattrs);
- curl_easy_reset(oci->curl);
-
- ret = ocierofs_curl_setup_common_options(oci->curl);
- if (ret)
- goto out;
-
- ret = ocierofs_curl_setup_rq(oci->curl, req.url, OCIEROFS_HTTP_GET,
- req.headers,
- ocierofs_layer_write_callback,
- &stream, NULL, NULL);
- if (ret)
- goto out;
-
- ret = ocierofs_curl_perform(oci->curl, &http_code);
- if (ret)
- goto out;
-
- if (http_code < 200 || http_code >= 300) {
- erofs_err("HTTP request failed with code %ld", http_code);
- ret = -EIO;
- goto out;
- }
-
- if (lseek(stream.blobfd, 0, SEEK_SET) < 0) {
- erofs_err("failed to seek to beginning of temp file: %s",
- strerror(errno));
- ret = -errno;
- goto out;
- }
-
- memset(&stream.tarfile, 0, sizeof(stream.tarfile));
- init_list_head(&stream.tarfile.global.xattrs);
-
- ret = erofs_iostream_open(&stream.tarfile.ios, stream.blobfd,
- EROFS_IOS_DECODER_GZIP);
+ ret = erofs_iostream_open(&tarfile.ios, fd, EROFS_IOS_DECODER_GZIP);
if (ret) {
erofs_err("failed to initialize tar stream: %s",
erofs_strerror(ret));
- goto out;
+ return ret;
}
do {
- ret = tarerofs_parse_tar(importer, &stream.tarfile);
+ ret = tarerofs_parse_tar(importer, &tarfile);
/* Continue parsing until end of archive */
} while (!ret);
- erofs_iostream_close(&stream.tarfile.ios);
+ erofs_iostream_close(&tarfile.ios);
if (ret < 0 && ret != -ENODATA) {
erofs_err("failed to process tar stream: %s",
erofs_strerror(ret));
- goto out;
+ return ret;
}
- ret = 0;
-out:
- if (stream.blobfd >= 0)
- close(stream.blobfd);
- if (req.headers)
- curl_slist_free_all(req.headers);
- free(req.url);
- return ret;
+ return 0;
}
-/**
- * ocierofs_build_trees - Build file trees from OCI container image layers
- * @importer: EROFS importer structure
- * @oci: OCI client structure with configured parameters
- *
- * Extract and build file system trees from all layers of an OCI container
- * image.
- *
- * Return: 0 on success, negative errno on failure
- */
-int ocierofs_build_trees(struct erofs_importer *importer, struct erofs_oci *oci)
+static int ocierofs_prepare_auth(struct ocierofs_ctx *ctx,
+ const char *username,
+ const char *password)
{
char *auth_header = NULL;
- char *manifest_digest = NULL;
- char **layers = NULL;
- int layer_count = 0;
- int ret, i;
-
- if (!importer || !oci)
- return -EINVAL;
+ int ret = 0;
+
+ ctx->using_basic = false;
+ free(ctx->auth_header);
+ ctx->auth_header = NULL;
+
+ auth_header = ocierofs_get_auth_token(ctx,
+ ctx->registry,
+ ctx->repository,
+ username, password);
+ if (!IS_ERR(auth_header)) {
+ ctx->auth_header = auth_header;
+ return 0;
+ }
- if (oci->params.username && oci->params.password &&
- oci->params.username[0] && oci->params.password[0]) {
- auth_header = ocierofs_get_auth_token(oci,
- oci->params.registry,
- oci->params.repository,
- oci->params.username,
- oci->params.password);
- if (IS_ERR(auth_header)) {
- auth_header = NULL;
- ret = ocierofs_curl_setup_basic_auth(oci->curl,
- oci->params.username,
- oci->params.password);
- if (ret)
- goto out;
- }
- } else {
- auth_header = ocierofs_get_auth_token(oci,
- oci->params.registry,
- oci->params.repository,
- NULL, NULL);
- if (IS_ERR(auth_header))
- auth_header = NULL;
+ if (username && password && *username && *password) {
+ ret = ocierofs_curl_setup_basic_auth(ctx->curl,
+ username, password);
+ if (ret)
+ return ret;
+ ctx->using_basic = true;
}
+ return 0;
+}
- manifest_digest = ocierofs_get_manifest_digest(oci, oci->params.registry,
- oci->params.repository,
- oci->params.tag,
- oci->params.platform,
- auth_header);
- if (IS_ERR(manifest_digest)) {
- ret = PTR_ERR(manifest_digest);
+static int ocierofs_prepare_layers(struct ocierofs_ctx *ctx,
+ const struct ocierofs_config *config)
+{
+ int ret;
+
+ ret = ocierofs_prepare_auth(ctx, config ? config->username : NULL,
+ config ? config->password : NULL);
+ if (ret)
+ return ret;
+
+ ctx->manifest_digest = ocierofs_get_manifest_digest(ctx, ctx->registry,
+ ctx->repository,
+ ctx->tag,
+ ctx->platform,
+ ctx->auth_header);
+ if (IS_ERR(ctx->manifest_digest)) {
+ ret = PTR_ERR(ctx->manifest_digest);
erofs_err("failed to get manifest digest: %s",
erofs_strerror(ret));
+ ctx->manifest_digest = NULL;
goto out_auth;
}
- layers = ocierofs_get_layers_info(oci, oci->params.registry,
- oci->params.repository,
- manifest_digest, auth_header,
- &layer_count);
- if (IS_ERR(layers)) {
- ret = PTR_ERR(layers);
+ ctx->layers = ocierofs_fetch_layers_info(ctx, ctx->registry,
+ ctx->repository,
+ ctx->manifest_digest, ctx->auth_header,
+ &ctx->layer_count);
+ if (IS_ERR(ctx->layers)) {
+ ret = PTR_ERR(ctx->layers);
erofs_err("failed to get image layers: %s", erofs_strerror(ret));
+ ctx->layers = NULL;
goto out_manifest;
}
- if (oci->params.layer_index >= 0) {
- if (oci->params.layer_index >= layer_count) {
+ if (ctx->layer_index >= 0) {
+ if (ctx->layer_index >= ctx->layer_count) {
erofs_err("layer index %d exceeds available layers (%d)",
- oci->params.layer_index, layer_count);
+ ctx->layer_index, ctx->layer_count);
ret = -EINVAL;
goto out_layers;
}
- layer_count = 1;
- i = oci->params.layer_index;
+ ctx->layer_count = 1;
} else {
- i = 0;
+ ctx->layer_index = 0;
}
- while (i < layer_count) {
- char *trimmed = erofs_trim_for_progressinfo(layers[i],
- sizeof("Extracting layer ...") - 1);
- erofs_update_progressinfo("Extracting layer %d: %s ...", i,
- trimmed);
- free(trimmed);
- ret = ocierofs_extract_layer(oci, importer, layers[i],
- auth_header);
- if (ret) {
- erofs_err("failed to extract layer %d: %s", i,
- erofs_strerror(ret));
- break;
- }
- i++;
- }
+ return 0;
+
out_layers:
- for (i = 0; i < layer_count; i++)
- free(layers[i]);
- free(layers);
+ free(ctx->layers);
+ ctx->layers = NULL;
out_manifest:
- free(manifest_digest);
+ free(ctx->manifest_digest);
+ ctx->manifest_digest = NULL;
out_auth:
- free(auth_header);
-
- if (oci->params.username && oci->params.password &&
- oci->params.username[0] && oci->params.password[0] &&
- !auth_header) {
- ocierofs_curl_clear_auth(oci->curl);
- }
-out:
+ free(ctx->auth_header);
+ ctx->auth_header = NULL;
+ if (ctx->using_basic)
+ ocierofs_curl_clear_auth(ctx);
return ret;
}
-/**
- * ocierofs_init - Initialize OCI client with default parameters
- * @oci: OCI client structure to initialize
- *
- * Initialize OCI client structure, set up CURL handle, and configure
- * default parameters including platform (linux/amd64), registry
- * (registry-1.docker.io), and tag (latest).
+/*
+ * ocierofs_parse_ref - Parse OCI image reference string
+ * @ctx: OCI context structure
+ * @ref_str: OCI image reference string
*
* Return: 0 on success, negative errno on failure
*/
-int ocierofs_init(struct erofs_oci *oci)
-{
- if (!oci)
- return -EINVAL;
-
- *oci = (struct erofs_oci){};
- oci->curl = curl_easy_init();
- if (!oci->curl)
- return -EIO;
-
- if (ocierofs_curl_setup_common_options(oci->curl)) {
- ocierofs_cleanup(oci);
- return -EIO;
- }
-
- if (erofs_oci_params_set_string(&oci->params.platform,
- "linux/amd64") ||
- erofs_oci_params_set_string(&oci->params.registry,
- DOCKER_API_REGISTRY) ||
- erofs_oci_params_set_string(&oci->params.tag, "latest")) {
- ocierofs_cleanup(oci);
- return -ENOMEM;
- }
- oci->params.layer_index = -1; /* -1 means extract all layers */
- return 0;
-}
-
-/**
- * ocierofs_cleanup - Clean up OCI client and free allocated resources
- * @oci: OCI client structure to clean up
- *
- * Clean up CURL handle, free all allocated string parameters, and
- * reset the OCI client structure to a clean state.
- */
-void ocierofs_cleanup(struct erofs_oci *oci)
-{
- if (!oci)
- return;
-
- if (oci->curl) {
- curl_easy_cleanup(oci->curl);
- oci->curl = NULL;
- }
-
- free(oci->params.registry);
- free(oci->params.repository);
- free(oci->params.tag);
- free(oci->params.platform);
- free(oci->params.username);
- free(oci->params.password);
-
- oci->params.registry = NULL;
- oci->params.repository = NULL;
- oci->params.tag = NULL;
- oci->params.platform = NULL;
- oci->params.username = NULL;
- oci->params.password = NULL;
-}
-
-int erofs_oci_params_set_string(char **field, const char *value)
-{
- char *new_value;
-
- if (!field)
- return -EINVAL;
-
- if (!value) {
- free(*field);
- *field = NULL;
- return 0;
- }
- new_value = strdup(value);
- if (!new_value)
- return -ENOMEM;
-
- free(*field);
- *field = new_value;
- return 0;
-}
-
-int ocierofs_parse_ref(struct erofs_oci *oci, const char *ref_str)
+static int ocierofs_parse_ref(struct ocierofs_ctx *ctx, const char *ref_str)
{
char *slash, *colon, *dot;
const char *repo_part;
size_t len;
+ char *tmp;
+
+ if (!ctx || !ref_str)
+ return -EINVAL;
slash = strchr(ref_str, '/');
if (slash) {
dot = strchr(ref_str, '.');
if (dot && dot < slash) {
- char *registry_str;
-
len = slash - ref_str;
- registry_str = strndup(ref_str, len);
-
- if (!registry_str) {
- erofs_err("failed to allocate memory for registry");
- return -ENOMEM;
- }
- if (erofs_oci_params_set_string(&oci->params.registry,
- registry_str)) {
- free(registry_str);
- erofs_err("failed to set registry");
+ tmp = strndup(ref_str, len);
+ if (!tmp)
return -ENOMEM;
- }
- free(registry_str);
+ free(ctx->registry);
+ ctx->registry = tmp;
repo_part = slash + 1;
} else {
repo_part = ref_str;
@@ -1106,73 +994,250 @@ int ocierofs_parse_ref(struct erofs_oci *oci, const char *ref_str)
colon = strchr(repo_part, ':');
if (colon) {
- char *repo_str;
-
len = colon - repo_part;
- repo_str = strndup(repo_part, len);
-
- if (!repo_str) {
- erofs_err("failed to allocate memory for repository");
+ tmp = strndup(repo_part, len);
+ if (!tmp)
return -ENOMEM;
- }
- if (!strchr(repo_str, '/') &&
- (!strcmp(oci->params.registry, DOCKER_API_REGISTRY) ||
- !strcmp(oci->params.registry, DOCKER_REGISTRY))) {
+ if (!strchr(tmp, '/') &&
+ (!strcmp(ctx->registry, DOCKER_API_REGISTRY) ||
+ !strcmp(ctx->registry, DOCKER_REGISTRY))) {
char *full_repo;
- if (asprintf(&full_repo, "library/%s", repo_str) == -1) {
- free(repo_str);
- erofs_err("failed to allocate memory for full repository name");
+ if (asprintf(&full_repo, "library/%s", tmp) == -1) {
+ free(tmp);
return -ENOMEM;
}
- free(repo_str);
- repo_str = full_repo;
+ free(tmp);
+ tmp = full_repo;
}
+ free(ctx->repository);
+ ctx->repository = tmp;
- if (erofs_oci_params_set_string(&oci->params.repository,
- repo_str)) {
- free(repo_str);
- erofs_err("failed to set repository");
+ free(ctx->tag);
+ ctx->tag = strdup(colon + 1);
+ if (!ctx->tag)
return -ENOMEM;
- }
- free(repo_str);
-
- if (erofs_oci_params_set_string(&oci->params.tag,
- colon + 1)) {
- erofs_err("failed to set tag");
- return -ENOMEM;
- }
} else {
- char *repo_str = strdup(repo_part);
-
- if (!repo_str) {
- erofs_err("failed to allocate memory for repository");
+ tmp = strdup(repo_part);
+ if (!tmp)
return -ENOMEM;
- }
- if (!strchr(repo_str, '/') &&
- (!strcmp(oci->params.registry, DOCKER_API_REGISTRY) ||
- !strcmp(oci->params.registry, DOCKER_REGISTRY))) {
+ if (!strchr(tmp, '/') &&
+ (!strcmp(ctx->registry, DOCKER_API_REGISTRY) ||
+ !strcmp(ctx->registry, DOCKER_REGISTRY))) {
+
char *full_repo;
- if (asprintf(&full_repo, "library/%s", repo_str) == -1) {
- free(repo_str);
- erofs_err("failed to allocate memory for full repository name");
+ if (asprintf(&full_repo, "library/%s", tmp) == -1) {
+ free(tmp);
return -ENOMEM;
}
- free(repo_str);
- repo_str = full_repo;
+ free(tmp);
+ tmp = full_repo;
}
+ free(ctx->repository);
+ ctx->repository = tmp;
+ }
+ return 0;
+}
- if (erofs_oci_params_set_string(&oci->params.repository,
- repo_str)) {
- free(repo_str);
- erofs_err("failed to set repository");
- return -ENOMEM;
+/**
+ * ocierofs_init - Initialize OCI context
+ * @ctx: OCI context structure to initialize
+ * @config: OCI configuration
+ *
+ * Initialize OCI context structure, set up CURL handle, and configure
+ * default parameters including platform (linux/amd64), registry
+ * (registry-1.docker.io), and tag (latest).
+ *
+ * Return: 0 on success, negative errno on failure
+ */
+int ocierofs_init(struct ocierofs_ctx *ctx, const struct ocierofs_config *config)
+{
+ int ret;
+
+ ctx->curl = curl_easy_init();
+ if (!ctx->curl)
+ return -EIO;
+
+ if (ocierofs_curl_setup_common_options(ctx->curl))
+ return -EIO;
+
+ ctx->layer_index = -1;
+ ctx->registry = strdup("registry-1.docker.io");
+ ctx->tag = strdup("latest");
+ if (config && config->platform)
+ ctx->platform = strdup(config->platform);
+ else
+ ctx->platform = strdup("linux/amd64");
+ if (!ctx->registry || !ctx->tag || !ctx->platform)
+ return -ENOMEM;
+
+ if (config && config->layer_index >= 0)
+ ctx->layer_index = config->layer_index;
+
+ ret = ocierofs_parse_ref(ctx, config->image_ref);
+ if (ret)
+ return ret;
+
+ ret = ocierofs_prepare_layers(ctx, config);
+ if (ret)
+ return ret;
+
+ return 0;
+}
+
+static int ocierofs_download_blob_to_fd(struct ocierofs_ctx *ctx,
+ const char *digest,
+ const char *auth_header,
+ int outfd)
+{
+ struct ocierofs_request req = {};
+ struct ocierofs_stream stream = {};
+ const char *api_registry;
+ long http_code;
+ int ret;
+
+ stream = (struct ocierofs_stream) {
+ .digest = digest,
+ .blobfd = outfd,
+ };
+
+ api_registry = ocierofs_get_api_registry(ctx->registry);
+ if (asprintf(&req.url, "https://%s/v2/%s/blobs/%s",
+ api_registry, ctx->repository, digest) == -1)
+ return -ENOMEM;
+
+ if (auth_header && strstr(auth_header, "Bearer"))
+ req.headers = curl_slist_append(req.headers, auth_header);
+
+ curl_easy_reset(ctx->curl);
+
+ ret = ocierofs_curl_setup_common_options(ctx->curl);
+ if (ret)
+ goto out;
+
+ ret = ocierofs_curl_setup_rq(ctx->curl, req.url, OCIEROFS_HTTP_GET,
+ req.headers,
+ ocierofs_layer_write_callback,
+ &stream, NULL, NULL);
+ if (ret)
+ goto out;
+
+ ret = ocierofs_curl_perform(ctx->curl, &http_code);
+ if (ret)
+ goto out;
+
+ if (http_code < 200 || http_code >= 300) {
+ erofs_err("HTTP request failed with code %ld", http_code);
+ ret = -EIO;
+ goto out;
+ }
+ ret = 0;
+out:
+ ocierofs_request_cleanup(&req);
+ return ret;
+}
+
+static int ocierofs_extract_layer(struct ocierofs_ctx *ctx,
+ const char *digest, const char *auth_header)
+{
+ struct ocierofs_stream stream = {};
+ int ret;
+
+ stream = (struct ocierofs_stream) {
+ .digest = digest,
+ .blobfd = erofs_tmpfile(),
+ };
+ if (stream.blobfd < 0) {
+ erofs_err("failed to create temporary file for %s", digest);
+ return -errno;
+ }
+
+ ret = ocierofs_download_blob_to_fd(ctx, digest, auth_header, stream.blobfd);
+ if (ret)
+ goto out;
+
+ if (lseek(stream.blobfd, 0, SEEK_SET) < 0) {
+ erofs_err("failed to seek to beginning of temp file: %s",
+ strerror(errno));
+ ret = -errno;
+ goto out;
+ }
+
+ return stream.blobfd;
+
+out:
+ if (stream.blobfd >= 0)
+ close(stream.blobfd);
+ return ret;
+}
+
+
+/**
+ * ocierofs_ctx_cleanup - Clean up OCI context and free allocated resources
+ * @ctx: OCI context structure to clean up
+ *
+ * Clean up CURL handle, free all allocated string parameters, and
+ * reset the OCI context structure to a clean state.
+ */
+static void ocierofs_ctx_cleanup(struct ocierofs_ctx *ctx)
+{
+ if (!ctx)
+ return;
+
+ if (ctx->curl) {
+ curl_easy_cleanup(ctx->curl);
+ ctx->curl = NULL;
+ }
+ free(ctx->auth_header);
+ ctx->auth_header = NULL;
+
+ ocierofs_free_layers_info(ctx->layers, ctx->layer_count);
+ free(ctx->registry);
+ free(ctx->repository);
+ free(ctx->tag);
+ free(ctx->platform);
+ free(ctx->manifest_digest);
+}
+
+int ocierofs_build_trees(struct erofs_importer *importer,
+ const struct ocierofs_config *config)
+{
+ struct ocierofs_ctx ctx = {};
+ int ret, i;
+
+ ret = ocierofs_init(&ctx, config);
+ if (ret) {
+ ocierofs_ctx_cleanup(&ctx);
+ return ret;
+ }
+
+ i = ctx.layer_index;
+ while (i < ctx.layer_count) {
+ char *trimmed = erofs_trim_for_progressinfo(ctx.layers[i]->digest,
+ sizeof("Extracting layer ...") - 1);
+ erofs_update_progressinfo("Extracting layer %d: %s ...", i,
+ trimmed);
+ free(trimmed);
+ int fd = ocierofs_extract_layer(&ctx, ctx.layers[i]->digest,
+ ctx.auth_header);
+ if (fd < 0) {
+ erofs_err("failed to extract layer %d: %s", i,
+ erofs_strerror(fd));
+ break;
}
- free(repo_str);
+ ret = ocierofs_process_tar_stream(importer, fd);
+ close(fd);
+ if (ret) {
+ erofs_err("failed to process tar stream for layer %d: %s", i,
+ erofs_strerror(ret));
+ break;
+ }
+ i++;
}
- return 0;
+ ocierofs_ctx_cleanup(&ctx);
+ return ret;
}
diff --git a/mkfs/main.c b/mkfs/main.c
index 064392d..28588db 100644
--- a/mkfs/main.c
+++ b/mkfs/main.c
@@ -272,7 +272,7 @@ static struct erofs_s3 s3cfg;
#endif
#ifdef OCIEROFS_ENABLED
-static struct erofs_oci ocicfg;
+static struct ocierofs_config ocicfg;
static char *mkfs_oci_options;
#endif
@@ -689,10 +689,9 @@ static int mkfs_parse_s3_cfg(char *cfg_str)
#endif
#ifdef OCIEROFS_ENABLED
-
-
-/**
+/*
* mkfs_parse_oci_options - Parse comma-separated OCI options string
+ * @cfg: OCI configuration structure to update
* @options_str: comma-separated options string
*
* Parse OCI options string containing comma-separated key=value pairs.
@@ -700,10 +699,9 @@ static int mkfs_parse_s3_cfg(char *cfg_str)
*
* Return: 0 on success, negative errno on failure
*/
-static int mkfs_parse_oci_options(char *options_str)
+static int mkfs_parse_oci_options(struct ocierofs_config *oci_cfg, char *options_str)
{
char *opt, *q, *p;
- int ret;
if (!options_str)
return 0;
@@ -717,41 +715,43 @@ static int mkfs_parse_oci_options(char *options_str)
p = strstr(opt, "platform=");
if (p) {
p += strlen("platform=");
- ret = erofs_oci_params_set_string(&ocicfg.params.platform, p);
- if (ret) {
- erofs_err("failed to set platform");
- return ret;
- }
+ free(oci_cfg->platform);
+ oci_cfg->platform = strdup(p);
+ if (!oci_cfg->platform)
+ return -ENOMEM;
} else {
p = strstr(opt, "layer=");
if (p) {
p += strlen("layer=");
- ocicfg.params.layer_index = atoi(p);
- if (ocicfg.params.layer_index < 0) {
- erofs_err("invalid layer index %d",
- ocicfg.params.layer_index);
- return -EINVAL;
+ {
+ char *endptr;
+ unsigned long v = strtoul(p, &endptr, 10);
+
+ if (endptr == p || *endptr != '\0') {
+ erofs_err("invalid layer index %s",
+ p);
+ return -EINVAL;
+ }
+ oci_cfg->layer_index = (int)v;
}
} else {
p = strstr(opt, "username=");
if (p) {
p += strlen("username=");
- ret = erofs_oci_params_set_string(&ocicfg.params.username, p);
- if (ret) {
- erofs_err("failed to set username");
- return ret;
- }
+ free(oci_cfg->username);
+ oci_cfg->username = strdup(p);
+ if (!oci_cfg->username)
+ return -ENOMEM;
} else {
p = strstr(opt, "password=");
if (p) {
p += strlen("password=");
- ret = erofs_oci_params_set_string(&ocicfg.params.password, p);
- if (ret) {
- erofs_err("failed to set password");
- return ret;
- }
+ free(oci_cfg->password);
+ oci_cfg->password = strdup(p);
+ if (!oci_cfg->password)
+ return -ENOMEM;
} else {
- erofs_err("invalid --oci value %s", opt);
+ erofs_err("mkfs: invalid --oci value %s", opt);
return -EINVAL;
}
}
@@ -1832,16 +1832,12 @@ int main(int argc, char **argv)
#endif
#ifdef OCIEROFS_ENABLED
} else if (source_mode == EROFS_MKFS_SOURCE_OCI) {
- err = ocierofs_init(&ocicfg);
- if (err)
- goto exit;
+ ocicfg.layer_index = -1;
- err = mkfs_parse_oci_options(mkfs_oci_options);
- if (err)
- goto exit;
- err = ocierofs_parse_ref(&ocicfg, cfg.c_src_path);
+ err = mkfs_parse_oci_options(&ocicfg, mkfs_oci_options);
if (err)
goto exit;
+ ocicfg.image_ref = cfg.c_src_path;
if (incremental_mode ||
dataimport_mode == EROFS_MKFS_DATA_IMPORT_RVSP ||
@@ -1916,9 +1912,6 @@ exit:
erofs_blob_exit();
erofs_xattr_cleanup_name_prefixes();
erofs_rebuild_cleanup();
-#ifdef OCIEROFS_ENABLED
- ocierofs_cleanup(&ocicfg);
-#endif
erofs_diskbuf_exit();
if (source_mode == EROFS_MKFS_SOURCE_TAR) {
erofs_iostream_close(&erofstar.ios);
--
2.51.0
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH v5 3/3] erofs-utils: add NBD-backed OCI image mounting
2025-09-04 6:36 ` [PATCH v5 0/3] erofs-utils: refactor OCI and " ChengyuZhu6
2025-09-04 6:36 ` [PATCH v5 1/3] erofs-utils:mkfs: move parse_oci_ref to lib ChengyuZhu6
2025-09-04 6:36 ` [PATCH v5 2/3] erofs-utils: refactor OCI code for better modularity ChengyuZhu6
@ 2025-09-04 6:36 ` ChengyuZhu6
2025-09-04 7:19 ` [PATCH v5 0/3] erofs-utils: refactor OCI and " Gao Xiang
3 siblings, 0 replies; 19+ messages in thread
From: ChengyuZhu6 @ 2025-09-04 6:36 UTC (permalink / raw)
To: linux-erofs; +Cc: xiang, hsiangkao, Chengyu Zhu
From: Chengyu Zhu <hudsonzhu@tencent.com>
- Add HTTP range downloads for OCI blobs
- Introduce ocierofs_iostream for virtual file I/O
- Add --oci option for OCI image mounting with NBD backend
New mount.erofs -t erofs.nbd option: -o=[options] source-image mountpoint
Supported oci options:
- oci.platform=os/arch (default: linux/amd64)
- oci=N (extract specific layer, default: all layers)
- oci.username/oci.password (basic authentication)
e.g.:
./mount/mount.erofs -t erofs.nbd -o 'oci=1,oci.platform=linux/amd64' \
quay.io/chengyuzhu6/golang:1.22.8-erofs /tmp/test/
Signed-off-by: Chengyu Zhu <hudsonzhu@tencent.com>
---
lib/liberofs_oci.h | 14 +++
lib/remotes/oci.c | 242 +++++++++++++++++++++++++++++++++++++++++-
mount/Makefile.am | 2 +-
mount/main.c | 257 ++++++++++++++++++++++++++++++++++++++-------
4 files changed, 476 insertions(+), 39 deletions(-)
diff --git a/lib/liberofs_oci.h b/lib/liberofs_oci.h
index ebb427a..61cfdc7 100644
--- a/lib/liberofs_oci.h
+++ b/lib/liberofs_oci.h
@@ -54,6 +54,12 @@ struct ocierofs_ctx {
int layer_count;
};
+struct ocierofs_iostream {
+ struct ocierofs_ctx *ctx;
+ struct erofs_vfile vf;
+ u64 offset;
+};
+
int ocierofs_init(struct ocierofs_ctx *ctx, const struct ocierofs_config *config);
/*
@@ -66,6 +72,14 @@ int ocierofs_init(struct ocierofs_ctx *ctx, const struct ocierofs_config *config
int ocierofs_build_trees(struct erofs_importer *importer,
const struct ocierofs_config *cfg);
+int ocierofs_is_erofs_native_image(struct ocierofs_ctx *ctx);
+
+void ocierofs_ctx_cleanup(struct ocierofs_ctx *ctx);
+
+int ocierofs_iostream_open(struct ocierofs_iostream *oci_iostream, struct ocierofs_ctx *oci_ctx);
+
+void ocierofs_iostream_close(struct ocierofs_iostream *oci_iostream);
+
#ifdef __cplusplus
}
#endif
diff --git a/lib/remotes/oci.c b/lib/remotes/oci.c
index 26e83d7..cfe6547 100644
--- a/lib/remotes/oci.c
+++ b/lib/remotes/oci.c
@@ -33,6 +33,9 @@
#define OCI_MEDIATYPE_MANIFEST "application/vnd.oci.image.manifest.v1+json"
#define OCI_MEDIATYPE_INDEX "application/vnd.oci.image.index.v1+json"
+/* Erofs Native Layer Media Type */
+#define EROFS_MEDIATYPE "application/vnd.erofs"
+
struct ocierofs_request {
char *url;
struct curl_slist *headers;
@@ -1174,7 +1177,6 @@ out:
return ret;
}
-
/**
* ocierofs_ctx_cleanup - Clean up OCI context and free allocated resources
* @ctx: OCI context structure to clean up
@@ -1182,7 +1184,7 @@ out:
* Clean up CURL handle, free all allocated string parameters, and
* reset the OCI context structure to a clean state.
*/
-static void ocierofs_ctx_cleanup(struct ocierofs_ctx *ctx)
+void ocierofs_ctx_cleanup(struct ocierofs_ctx *ctx)
{
if (!ctx)
return;
@@ -1241,3 +1243,239 @@ int ocierofs_build_trees(struct erofs_importer *importer,
ocierofs_ctx_cleanup(&ctx);
return ret;
}
+
+static int ocierofs_download_blob_range(struct ocierofs_ctx *ctx, off_t offset, size_t length,
+ void **out_buf, size_t *out_size)
+{
+ struct ocierofs_request req = {};
+ struct ocierofs_response resp = {};
+ const char *api_registry;
+ char rangehdr[64];
+ long http_code = 0;
+ int ret;
+ int index = ctx->layer_index;
+
+ if (offset < 0)
+ return -EINVAL;
+
+ api_registry = ocierofs_get_api_registry(ctx->registry);
+ if (asprintf(&req.url, "https://%s/v2/%s/blobs/%s",
+ api_registry, ctx->repository, ctx->layers[index]->digest) == -1)
+ return -ENOMEM;
+
+ if (length)
+ snprintf(rangehdr, sizeof(rangehdr), "Range: bytes=%lld-%lld",
+ (long long)offset, (long long)(offset + (off_t)length - 1));
+ else
+ snprintf(rangehdr, sizeof(rangehdr), "Range: bytes=%lld-",
+ (long long)offset);
+
+ if (ctx->auth_header && strstr(ctx->auth_header, "Bearer"))
+ req.headers = curl_slist_append(req.headers, ctx->auth_header);
+ req.headers = curl_slist_append(req.headers, rangehdr);
+
+ curl_easy_reset(ctx->curl);
+
+ ret = ocierofs_curl_setup_common_options(ctx->curl);
+ if (ret)
+ goto out;
+
+ ret = ocierofs_curl_setup_rq(ctx->curl, req.url, OCIEROFS_HTTP_GET,
+ req.headers,
+ ocierofs_write_callback,
+ &resp, NULL, NULL);
+ if (ret)
+ goto out;
+
+ ret = ocierofs_curl_perform(ctx->curl, &http_code);
+ if (ret)
+ goto out;
+
+ if (http_code == 206) {
+ *out_buf = resp.data;
+ *out_size = resp.size;
+ resp.data = NULL;
+ ret = 0;
+ } else if (http_code == 200) {
+ if (offset == 0) {
+ *out_buf = resp.data;
+ *out_size = resp.size;
+ resp.data = NULL;
+ ret = 0;
+ } else {
+ if (offset < resp.size) {
+ size_t available = resp.size - offset;
+ size_t copy_size = length ? min_t(size_t, length, available) : available;
+
+ *out_buf = malloc(copy_size);
+ if (!*out_buf) {
+ ret = -ENOMEM;
+ goto out;
+ }
+ memcpy(*out_buf, resp.data + offset, copy_size);
+ *out_size = copy_size;
+ ret = 0;
+ } else {
+ *out_buf = NULL;
+ *out_size = 0;
+ ret = 0;
+ }
+ }
+ } else {
+ erofs_err("HTTP range request failed with code %ld", http_code);
+ ret = -EIO;
+ }
+
+out:
+ if (req.headers)
+ curl_slist_free_all(req.headers);
+ free(req.url);
+ free(resp.data);
+ return ret;
+}
+
+static ssize_t ocierofs_io_pread(struct erofs_vfile *vf, void *buf, size_t len, u64 offset)
+{
+ struct ocierofs_iostream *oci_iostream = *(struct ocierofs_iostream **)vf->payload;
+ void *download_buf = NULL;
+ size_t download_size = 0;
+ ssize_t ret;
+
+ ret = ocierofs_download_blob_range(oci_iostream->ctx, offset, len,
+ &download_buf, &download_size);
+ if (ret < 0) {
+ memset(buf, 0, len);
+ return len;
+ }
+
+ if (download_buf && download_size > 0) {
+ memcpy(buf, download_buf, download_size);
+ free(download_buf);
+ return download_size;
+ }
+
+ return 0;
+}
+
+static ssize_t ocierofs_io_read(struct erofs_vfile *vf, void *buf, size_t len)
+{
+ struct ocierofs_iostream *oci_iostream = *(struct ocierofs_iostream **)vf->payload;
+ ssize_t ret;
+
+ ret = ocierofs_io_pread(vf, buf, len, oci_iostream->offset);
+ if (ret > 0)
+ oci_iostream->offset += ret;
+
+ return ret;
+}
+
+static off_t ocierofs_io_lseek(struct erofs_vfile *vf, u64 offset, int whence)
+{
+ struct ocierofs_iostream *oci_iostream = *(struct ocierofs_iostream **)vf->payload;
+ off_t new_offset;
+ int layer_index = oci_iostream->ctx->layer_index;
+
+ switch (whence) {
+ case SEEK_SET:
+ new_offset = offset;
+ break;
+ case SEEK_CUR:
+ new_offset = oci_iostream->offset + offset;
+ break;
+ case SEEK_END:
+ new_offset = oci_iostream->ctx->layers[layer_index]->size + offset;
+ break;
+ default:
+ return -1;
+ }
+
+ if (new_offset < 0 || new_offset > oci_iostream->ctx->layers[layer_index]->size)
+ return -1;
+
+ oci_iostream->offset = new_offset;
+ return new_offset;
+}
+
+static ssize_t ocierofs_io_sendfile(struct erofs_vfile *vout, struct erofs_vfile *vin,
+ off_t *pos, size_t count)
+{
+ struct ocierofs_iostream *oci_iostream = *(struct ocierofs_iostream **)vin->payload;
+ char *buf = NULL;
+ ssize_t total_written = 0;
+ ssize_t ret = 0;
+
+ buf = malloc(min_t(size_t, count, 32768));
+ if (!buf)
+ return -ENOMEM;
+
+ while (count > 0) {
+ size_t to_read = min_t(size_t, count, 32768);
+ u64 read_offset = pos ? *pos : oci_iostream->offset;
+
+ ret = ocierofs_io_pread(vin, buf, to_read, read_offset);
+ if (ret <= 0) {
+ erofs_err("OCI I/O sendfile: failed to read from OCI: %s",
+ erofs_strerror(ret));
+ memset(buf, 0, to_read);
+ ret = to_read;
+ }
+
+ ssize_t written = write(vout->fd, buf, ret);
+
+ if (written < 0) {
+ erofs_err("OCI I/O sendfile: failed to write to output: %s",
+ strerror(errno));
+ ret = -errno;
+ break;
+ }
+
+ if (written != ret) {
+ erofs_err("OCI I/O sendfile: partial write: %zd != %zd", written, ret);
+ ret = written;
+ }
+
+ total_written += ret;
+ count -= ret;
+ if (pos)
+ *pos += ret;
+ else
+ oci_iostream->offset += ret;
+ }
+
+ free(buf);
+ return count;
+}
+
+static struct erofs_vfops ocierofs_io_vfops = {
+ .pread = ocierofs_io_pread,
+ .read = ocierofs_io_read,
+ .lseek = ocierofs_io_lseek,
+ .sendfile = ocierofs_io_sendfile,
+};
+
+int ocierofs_iostream_open(struct ocierofs_iostream *oci_iostream, struct ocierofs_ctx *oci_ctx)
+{
+ memset(oci_iostream, 0, sizeof(*oci_iostream));
+ oci_iostream->ctx = oci_ctx;
+ oci_iostream->vf.ops = &ocierofs_io_vfops;
+ oci_iostream->vf.fd = -1;
+ *(struct ocierofs_iostream **)oci_iostream->vf.payload = oci_iostream;
+
+ return 0;
+}
+
+void ocierofs_iostream_close(struct ocierofs_iostream *oci_iostream)
+{
+ close(oci_iostream->vf.fd);
+}
+
+int ocierofs_is_erofs_native_image(struct ocierofs_ctx *ctx)
+{
+ if (ctx->layer_count > 0 && ctx->layers[0] &&
+ ctx->layers[0]->media_type) {
+ if (strcmp(ctx->layers[0]->media_type, EROFS_MEDIATYPE) != 0)
+ return -ENOENT;
+ return 0;
+ }
+ return -ENOENT;
+}
diff --git a/mount/Makefile.am b/mount/Makefile.am
index d93f3f4..0b4447f 100644
--- a/mount/Makefile.am
+++ b/mount/Makefile.am
@@ -9,5 +9,5 @@ mount_erofs_SOURCES = main.c
mount_erofs_CFLAGS = -Wall -I$(top_srcdir)/include
mount_erofs_LDADD = $(top_builddir)/lib/liberofs.la ${libselinux_LIBS} \
${liblz4_LIBS} ${liblzma_LIBS} ${zlib_LIBS} ${libdeflate_LIBS} \
- ${libzstd_LIBS} ${libqpl_LIBS} ${libxxhash_LIBS} ${libnl3_LIBS}
+ ${libzstd_LIBS} ${libqpl_LIBS} ${libxxhash_LIBS} ${libnl3_LIBS} ${openssl_LIBS}
endif
diff --git a/mount/main.c b/mount/main.c
index a270f0a..177bca3 100644
--- a/mount/main.c
+++ b/mount/main.c
@@ -15,6 +15,7 @@
#include "erofs/err.h"
#include "erofs/io.h"
#include "../lib/liberofs_nbd.h"
+#include "../lib/liberofs_oci.h"
#ifdef HAVE_LINUX_LOOP_H
#include <linux/loop.h>
#else
@@ -34,6 +35,10 @@ struct loop_info {
#include <sys/sysmacros.h>
#endif
+#ifdef OCIEROFS_ENABLED
+static struct ocierofs_config ocicfg;
+#endif
+
enum erofs_backend_drv {
EROFSAUTO,
EROFSLOCAL,
@@ -56,12 +61,76 @@ static struct erofsmount_cfg {
long flags;
enum erofs_backend_drv backend;
enum erofsmount_mode mountmode;
+#ifdef OCIEROFS_ENABLED
+ bool use_oci;
+#endif
} mountcfg = {
.full_options = "ro",
.flags = MS_RDONLY, /* default mountflags */
.fstype = "erofs",
};
+enum erofs_nbd_source_type {
+ EROFSNBD_SOURCE_LOCAL,
+ EROFSNBD_SOURCE_OCI,
+};
+
+union erofs_nbd_source {
+ const char *device_path;
+ struct ocierofs_ctx *oci_ctx;
+};
+
+union erofs_nbd_source src;
+
+static int parse_oci_option(struct ocierofs_config *oci_cfg, const char *option)
+{
+ char *p;
+
+ p = strstr(option, "oci=");
+ if (p != NULL) {
+ p += strlen("oci=");
+ {
+ char *endptr;
+ unsigned long v = strtoul(p, &endptr, 10);
+
+ if (endptr == p || *endptr != '\0')
+ return -EINVAL;
+ oci_cfg->layer_index = (int)v;
+ }
+ } else {
+ p = strstr(option, "oci.platform=");
+ if (p != NULL) {
+ p += strlen("oci.platform=");
+ free(oci_cfg->platform);
+ oci_cfg->platform = strdup(p);
+ if (!oci_cfg->platform)
+ return -ENOMEM;
+ } else {
+ p = strstr(option, "oci.username=");
+ if (p != NULL) {
+ p += strlen("oci.username=");
+ free(oci_cfg->username);
+ oci_cfg->username = strdup(p);
+ if (!oci_cfg->username)
+ return -ENOMEM;
+ } else {
+ p = strstr(option, "oci.password=");
+ if (p != NULL) {
+ p += strlen("oci.password=");
+ free(oci_cfg->password);
+ oci_cfg->password = strdup(p);
+ if (!oci_cfg->password)
+ return -ENOMEM;
+ } else {
+ return -EINVAL;
+ }
+ }
+ }
+ }
+
+ return 0;
+}
+
static long erofsmount_parse_flagopts(char *s, long flags, char **more)
{
static const struct {
@@ -90,29 +159,41 @@ static long erofsmount_parse_flagopts(char *s, long flags, char **more)
comma = strchr(s, ',');
if (comma)
*comma = '\0';
- for (i = 0; i < ARRAY_SIZE(opts); ++i) {
- if (!strcasecmp(s, opts[i].name)) {
- if (opts[i].flags < 0)
- flags &= opts[i].flags;
- else
- flags |= opts[i].flags;
- break;
- }
- }
- if (more && i >= ARRAY_SIZE(opts)) {
- int sl = strlen(s);
- char *new = *more;
+ if (strncmp(s, "oci", 3) == 0) {
+#ifdef OCIEROFS_ENABLED
+ int err = parse_oci_option(&ocicfg, s);
- i = new ? strlen(new) : 0;
- new = realloc(new, i + strlen(s) + 2);
- if (!new)
- return -ENOMEM;
- if (i)
- new[i++] = ',';
- memcpy(new + i, s, sl);
- new[i + sl] = '\0';
- *more = new;
+ if (err < 0)
+ return err;
+#else
+ return -EINVAL;
+#endif
+ } else {
+ for (i = 0; i < ARRAY_SIZE(opts); ++i) {
+ if (!strcasecmp(s, opts[i].name)) {
+ if (opts[i].flags < 0)
+ flags &= opts[i].flags;
+ else
+ flags |= opts[i].flags;
+ break;
+ }
+ }
+
+ if (more && i >= ARRAY_SIZE(opts)) {
+ int sl = strlen(s);
+ char *new = *more;
+
+ i = new ? strlen(new) : 0;
+ new = realloc(new, i + strlen(s) + 2);
+ if (!new)
+ return -ENOMEM;
+ if (i)
+ new[i++] = ',';
+ memcpy(new + i, s, sl);
+ new[i + sl] = '\0';
+ *more = new;
+ }
}
if (!comma)
@@ -120,6 +201,11 @@ static long erofsmount_parse_flagopts(char *s, long flags, char **more)
*comma = ',';
s = comma + 1;
}
+
+#ifdef OCIEROFS_ENABLED
+ if (ocicfg.platform || ocicfg.username || ocicfg.password || ocicfg.layer_index != 0)
+ mountcfg.use_oci = true;
+#endif
return flags;
}
@@ -540,9 +626,64 @@ err_identifier:
return err;
}
-static int erofsmount_nbd(const char *source, const char *mountpoint,
- const char *fstype, int flags,
- const char *options)
+static int erofsmount_startnbd_oci(int nbdfd, struct ocierofs_ctx *oci_ctx)
+{
+ struct erofsmount_nbd_ctx ctx = {};
+ struct ocierofs_iostream *oci_iostream = NULL;
+ uintptr_t retcode;
+ pthread_t th;
+ int err, err2;
+ int blkbits = 12;
+ int index = oci_ctx->layer_index;
+ u64 blocks;
+
+ blocks = (oci_ctx->layers[index]->size + (1ULL << blkbits) - 1) >> blkbits;
+
+ oci_iostream = malloc(sizeof(struct ocierofs_iostream));
+ if (!oci_iostream)
+ return -ENOMEM;
+
+ err = ocierofs_iostream_open(oci_iostream, oci_ctx);
+ if (err) {
+ free(oci_iostream);
+ return err;
+ }
+
+ ctx.vd = oci_iostream->vf;
+
+ err = erofs_nbd_connect(nbdfd, blkbits, blocks);
+ if (err < 0) {
+ ocierofs_iostream_close(oci_iostream);
+ free(oci_iostream);
+ return err;
+ }
+ ctx.sk.fd = err;
+
+ err = -pthread_create(&th, NULL, erofsmount_nbd_loopfn, &ctx);
+ if (err) {
+ ocierofs_iostream_close(oci_iostream);
+ free(oci_iostream);
+ close(ctx.sk.fd);
+ return err;
+ }
+
+ err = erofs_nbd_do_it(nbdfd);
+ err2 = -pthread_join(th, (void **)&retcode);
+ if (!err2 && retcode) {
+ erofs_err("NBD worker failed with %s",
+ erofs_strerror(retcode));
+ err2 = retcode;
+ }
+
+ ocierofs_iostream_close(oci_iostream);
+ free(oci_iostream);
+
+ return err ?: err2;
+}
+
+static int erofsmount_nbd(union erofs_nbd_source source, enum erofs_nbd_source_type source_type,
+ const char *mountpoint, const char *fstype,
+ int flags, const char *options)
{
bool is_netlink = false;
char nbdpath[32], *id;
@@ -555,11 +696,19 @@ static int erofsmount_nbd(const char *source, const char *mountpoint,
mountcfg.fstype);
return -ENODEV;
}
+
flags |= MS_RDONLY;
- err = erofsmount_startnbd_nl(&pid, source);
- if (err < 0) {
- erofs_info("Fall back to ioctl-based NBD; failover is unsupported");
+ if (source_type == EROFSNBD_SOURCE_LOCAL) {
+ err = erofsmount_startnbd_nl(&pid, source.device_path);
+ if (err >= 0) {
+ num = err;
+ (void)snprintf(nbdpath, sizeof(nbdpath), "/dev/nbd%d", num);
+ is_netlink = true;
+ }
+ }
+
+ if (!is_netlink) {
num = erofs_nbd_devscan();
if (num < 0)
return num;
@@ -569,14 +718,16 @@ static int erofsmount_nbd(const char *source, const char *mountpoint,
if (nbdfd < 0)
return -errno;
- if ((pid = fork()) == 0)
- return erofsmount_startnbd(nbdfd, source) ?
- EXIT_FAILURE : EXIT_SUCCESS;
+ if ((pid = fork()) == 0) {
+ if (source_type == EROFSNBD_SOURCE_OCI) {
+ return erofsmount_startnbd_oci(nbdfd, source.oci_ctx) ?
+ EXIT_FAILURE : EXIT_SUCCESS;
+ } else {
+ return erofsmount_startnbd(nbdfd, source.device_path) ?
+ EXIT_FAILURE : EXIT_SUCCESS;
+ }
+ }
close(nbdfd);
- } else {
- num = err;
- (void)snprintf(nbdpath, sizeof(nbdpath), "/dev/nbd%d", num);
- is_netlink = true;
}
while (1) {
@@ -594,7 +745,7 @@ static int erofsmount_nbd(const char *source, const char *mountpoint,
if (err < 0)
err = -errno;
- if (!err && is_netlink) {
+ if (!err && is_netlink && source_type == EROFSNBD_SOURCE_LOCAL) {
id = erofs_nbd_get_identifier(num);
if (id == ERR_PTR(-ENOENT))
id = NULL;
@@ -799,9 +950,43 @@ int main(int argc, char *argv[])
}
if (mountcfg.backend == EROFSNBD) {
- err = erofsmount_nbd(mountcfg.device, mountcfg.target,
+#ifdef OCIEROFS_ENABLED
+ if (mountcfg.use_oci) {
+ struct ocierofs_ctx ctx = {};
+
+ ocicfg.image_ref = mountcfg.device;
+ err = ocierofs_init(&ctx, &ocicfg);
+ if (err) {
+ ocierofs_ctx_cleanup(&ctx);
+ goto exit;
+ }
+
+ err = ocierofs_is_erofs_native_image(&ctx);
+ if (err) {
+ ocierofs_ctx_cleanup(&ctx);
+ goto exit;
+ }
+
+ src.oci_ctx = &ctx;
+
+ err = erofsmount_nbd(src, EROFSNBD_SOURCE_OCI, mountcfg.target,
+ mountcfg.fstype, mountcfg.flags, mountcfg.options);
+ if (err) {
+ ocierofs_ctx_cleanup(&ctx);
+ goto exit;
+ }
+ } else {
+ src.device_path = mountcfg.device;
+ err = erofsmount_nbd(src, EROFSNBD_SOURCE_LOCAL, mountcfg.target,
+ mountcfg.fstype, mountcfg.flags,
+ mountcfg.options);
+ }
+#else
+ src.device_path = mountcfg.device;
+ err = erofsmount_nbd(src, EROFSNBD_SOURCE_LOCAL, mountcfg.target,
mountcfg.fstype, mountcfg.flags,
mountcfg.options);
+#endif
goto exit;
}
--
2.51.0
^ permalink raw reply related [flat|nested] 19+ messages in thread
* Re: [PATCH v5 0/3] erofs-utils: refactor OCI and add NBD-backed OCI image mounting
2025-09-04 6:36 ` [PATCH v5 0/3] erofs-utils: refactor OCI and " ChengyuZhu6
` (2 preceding siblings ...)
2025-09-04 6:36 ` [PATCH v5 3/3] erofs-utils: add NBD-backed OCI image mounting ChengyuZhu6
@ 2025-09-04 7:19 ` Gao Xiang
3 siblings, 0 replies; 19+ messages in thread
From: Gao Xiang @ 2025-09-04 7:19 UTC (permalink / raw)
To: ChengyuZhu6, linux-erofs; +Cc: xiang, Chengyu Zhu
On 2025/9/4 14:36, ChengyuZhu6 wrote:
> From: Chengyu Zhu <hudsonzhu@tencent.com>
>
> This series refactors the OCI handling in erofs-utils and adds NBD-backed
> mounting of OCI images. It enables mounting EROFS container images directly
> from registries without pre-downloading.
>
> Chengyu Zhu (3):
> erofs-utils:mkfs: move parse_oci_ref to lib
> erofs-utils: refactor OCI code for better modularity
I've applied the first two patches to -experimental.
Thanks,
Gao Xiang
^ permalink raw reply [flat|nested] 19+ messages in thread
end of thread, other threads:[~2025-09-04 7:19 UTC | newest]
Thread overview: 19+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-09-01 5:10 [PATCH v1] erofs-utils: refactor OCI code for better modularity ChengyuZhu6
2025-09-01 7:01 ` [PATCH v1-changed] " ChengyuZhu6
2025-09-02 1:52 ` Gao Xiang
2025-09-02 3:17 ` [PATCH v2] " ChengyuZhu6
2025-09-02 3:29 ` [PATCH v2-changed] " ChengyuZhu6
2025-09-03 8:29 ` [PATCH v3 0/2] erofs-utils: refactor OCI and add NBD-backed OCI image mounting ChengyuZhu6
2025-09-03 8:29 ` [PATCH v3 1/2] erofs-utils: refactor OCI code for better modularity ChengyuZhu6
2025-09-03 8:47 ` Gao Xiang
2025-09-03 8:29 ` [PATCH v3 2/2] erofs-utils: add NBD-backed OCI image mounting ChengyuZhu6
2025-09-04 5:33 ` [PATCH v4 0/3] erofs-utils: refactor OCI and " ChengyuZhu6
2025-09-04 5:33 ` [PATCH v4 1/3] erofs-utils:mkfs: move parse_oci_ref to lib ChengyuZhu6
2025-09-04 5:33 ` [PATCH v4 2/3] erofs-utils: refactor OCI code for better modularity ChengyuZhu6
2025-09-04 5:53 ` Gao Xiang
2025-09-04 5:33 ` [PATCH v4 3/3] erofs-utils: add NBD-backed OCI image mounting ChengyuZhu6
2025-09-04 6:36 ` [PATCH v5 0/3] erofs-utils: refactor OCI and " ChengyuZhu6
2025-09-04 6:36 ` [PATCH v5 1/3] erofs-utils:mkfs: move parse_oci_ref to lib ChengyuZhu6
2025-09-04 6:36 ` [PATCH v5 2/3] erofs-utils: refactor OCI code for better modularity ChengyuZhu6
2025-09-04 6:36 ` [PATCH v5 3/3] erofs-utils: add NBD-backed OCI image mounting ChengyuZhu6
2025-09-04 7:19 ` [PATCH v5 0/3] erofs-utils: refactor OCI and " Gao Xiang
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.