* [PATCH v1] erofs-utils: refactor OCI code for better modularity
@ 2025-09-01 5:10 ChengyuZhu6
2025-09-01 7:01 ` [PATCH v1-changed] " ChengyuZhu6
` (5 more replies)
0 siblings, 6 replies; 19+ messages in thread
From: ChengyuZhu6 @ 2025-09-01 5:10 UTC (permalink / raw)
To: linux-erofs; +Cc: xiang, hsiangkao, Chengyu Zhu
From: Chengyu Zhu <hudsonzhu@tencent.com>
Refactor OCI code to improve code organization and maintainability:
- Add `struct ocierofs_layer_info` to encapsulate layer metadata
- Extract authentication logic into `ocierofs_prepare_auth()`
- Split layer processing into `ocierofs_prepare_layers()`
- Move OCI parsing functions from `mkfs/main.c` to `lib/remotes/oci.c`
- Add `ocierofs_process_tar_stream()` for separate tar processing
- Improve error handling with `ocierofs_free_layers_info()`
- Refactor `ocierofs_extract_layer()` to return file descriptor
Signed-off-by: Chengyu Zhu <hudsonzhu@tencent.com>
---
lib/liberofs_oci.h | 89 ++++++++
lib/remotes/oci.c | 536 ++++++++++++++++++++++++++++++++++-----------
mkfs/main.c | 200 +----------------
3 files changed, 503 insertions(+), 322 deletions(-)
diff --git a/lib/liberofs_oci.h b/lib/liberofs_oci.h
index 3a8108b..bd83966 100644
--- a/lib/liberofs_oci.h
+++ b/lib/liberofs_oci.h
@@ -19,6 +19,23 @@ struct erofs_inode;
struct CURL;
struct erofs_importer;
+/**
+ * struct ocierofs_layer_info
+ * @digest: OCI content-addressable digest (e.g. "sha256:...")
+ * @media_type: mediaType string from the manifest
+ * @size: layer size in bytes from the manifest (0 if not available)
+ *
+ * This structure is exposed to callers so they can enumerate image layers,
+ * decide which ones to fetch, and pass the digest back to download APIs.
+ * Fields are heap-allocated NUL-terminated strings owned by the caller
+ * once returned from public APIs; the caller must free them.
+ */
+struct ocierofs_layer_info {
+ char *digest;
+ char *media_type;
+ u64 size;
+};
+
/**
* struct erofs_oci_params - OCI configuration parameters
* @registry: registry hostname (e.g., "registry-1.docker.io")
@@ -88,6 +105,78 @@ int erofs_oci_params_set_string(char **field, const char *value);
*/
int ocierofs_build_trees(struct erofs_importer *importer, struct erofs_oci *oci);
+/*
+ * ocierofs_parse_options - Parse comma-separated OCI options string
+ * @oci: OCI client structure to update
+ * @options_str: comma-separated options string
+ *
+ * Parse OCI options string containing comma-separated key=value pairs.
+ * Supported options include platform, layer, username, and password.
+ *
+ * Return: 0 on success, negative errno on failure
+ */
+int ocierofs_parse_options(struct erofs_oci *oci, char *options_str);
+
+/*
+ * ocierofs_parse_ref - Parse OCI image reference string
+ * @oci: OCI client structure to update
+ * @ref_str: OCI image reference in various formats
+ *
+ * Parse OCI image reference which can be in formats:
+ * - registry.example.com/namespace/repo:tag
+ * - namespace/repo:tag (uses default registry)
+ * - repo:tag (adds library/ prefix for Docker Hub)
+ * - repo (uses default tag "latest")
+ *
+ * Return: 0 on success, negative errno on failure
+ */
+int ocierofs_parse_ref(struct erofs_oci *oci, const char *ref_str);
+
+/*
+ * ocierofs_prepare_layers - Prepare OCI layers for processing
+ * @oci: OCI client structure with configured parameters
+ * @auth_header: Pointer to store authentication header
+ * @using_basic: Pointer to store basic auth flag
+ * @manifest_digest: Pointer to store manifest digest
+ * @layers: Pointer to store layers information
+ * @layer_count: Pointer to store number of layers
+ * @start_index: Pointer to store starting layer index
+ *
+ * Prepare authentication, get manifest digest and layers information
+ * for OCI image processing. This function handles all the preparation
+ * work needed before processing OCI layers.
+ *
+ * Return: 0 on success, negative errno on failure
+ */
+int ocierofs_prepare_layers(struct erofs_oci *oci, char **auth_header,
+ bool *using_basic, char **manifest_digest,
+ struct ocierofs_layer_info ***layers,
+ int *layer_count, int *start_index);
+
+/**
+ * ocierofs_free_layers_info - Free layer information array
+ * @layers: array of layer information structures
+ * @count: number of layers in the array
+ *
+ * Free all layer information structures and the array itself.
+ * This function handles NULL pointers safely.
+ */
+void ocierofs_free_layers_info(struct ocierofs_layer_info **layers, int count);
+
+/**
+ * ocierofs_prepare_auth - Prepare authentication for OCI requests
+ * @oci: OCI client structure
+ * @auth_header_out: pointer to store authentication header
+ * @using_basic_auth: pointer to store basic auth flag
+ *
+ * Prepare authentication header for OCI registry requests.
+ * This function handles both token-based and basic authentication.
+ *
+ * Return: 0 on success, negative errno on failure
+ */
+int ocierofs_prepare_auth(struct erofs_oci *oci, char **auth_header_out,
+ bool *using_basic_auth);
+
#ifdef __cplusplus
}
#endif
diff --git a/lib/remotes/oci.c b/lib/remotes/oci.c
index 0fb8c1f..803175a 100644
--- a/lib/remotes/oci.c
+++ b/lib/remotes/oci.c
@@ -42,7 +42,6 @@ struct erofs_oci_response {
};
struct erofs_oci_stream {
- struct erofs_tarfile tarfile;
const char *digest;
int blobfd;
};
@@ -181,7 +180,7 @@ static int ocierofs_request_perform(struct erofs_oci *oci,
ret = ocierofs_curl_setup_rq(oci->curl, req->url,
OCIEROFS_HTTP_GET, req->headers,
- ocierofs_write_callback, resp,
+ ocierofs_write_callback, resp,
NULL, NULL);
if (ret)
return ret;
@@ -568,7 +567,7 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci,
const char *api_registry;
int ret = 0, len, i;
- if (!registry || !repository || !tag || !platform)
+ if (!registry || !repository || !tag)
return ERR_PTR(-EINVAL);
api_registry = (!strcmp(registry, DOCKER_REGISTRY)) ? DOCKER_API_REGISTRY : registry;
@@ -581,8 +580,8 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci,
req.headers = curl_slist_append(req.headers,
"Accept: " DOCKER_MEDIATYPE_MANIFEST_LIST ","
- OCI_MEDIATYPE_INDEX "," DOCKER_MEDIATYPE_MANIFEST_V1 ","
- DOCKER_MEDIATYPE_MANIFEST_V2);
+ OCI_MEDIATYPE_INDEX "," OCI_MEDIATYPE_MANIFEST ","
+ DOCKER_MEDIATYPE_MANIFEST_V1 "," DOCKER_MEDIATYPE_MANIFEST_V2);
ret = ocierofs_request_perform(oci, &req, &resp);
if (ret)
@@ -663,7 +662,24 @@ out:
return ret ? ERR_PTR(ret) : digest;
}
-static char **ocierofs_get_layers_info(struct erofs_oci *oci,
+void ocierofs_free_layers_info(struct ocierofs_layer_info **layers, int count)
+{
+ int i;
+
+ if (!layers)
+ return;
+
+ for (i = 0; i < count; i++) {
+ if (layers[i]) {
+ free(layers[i]->digest);
+ free(layers[i]->media_type);
+ free(layers[i]);
+ }
+ }
+ free(layers);
+}
+
+static struct ocierofs_layer_info **ocierofs_fetch_layers_info(struct erofs_oci *oci,
const char *registry,
const char *repository,
const char *digest,
@@ -672,10 +688,10 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci,
{
struct erofs_oci_request req = {};
struct erofs_oci_response resp = {};
- json_object *root, *layers, *layer, *digest_obj;
- char **layers_info = NULL;
+ json_object *root, *layers, *layer, *digest_obj, *media_type_obj, *size_obj;
+ struct ocierofs_layer_info **layers_info = NULL;
const char *api_registry;
- int ret, len, i, j;
+ int ret, len, i;
if (!registry || !repository || !digest || !layer_count)
return ERR_PTR(-EINVAL);
@@ -725,7 +741,7 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci,
goto out_json;
}
- layers_info = calloc(len, sizeof(char *));
+ layers_info = calloc(len, sizeof(*layers_info));
if (!layers_info) {
ret = -ENOMEM;
goto out_json;
@@ -740,11 +756,25 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci,
goto out_free;
}
- layers_info[i] = strdup(json_object_get_string(digest_obj));
+ layers_info[i] = calloc(1, sizeof(**layers_info));
if (!layers_info[i]) {
ret = -ENOMEM;
goto out_free;
}
+ layers_info[i]->digest = strdup(json_object_get_string(digest_obj));
+ if (!layers_info[i]->digest) {
+ ret = -ENOMEM;
+ goto out_free;
+ }
+ if (json_object_object_get_ex(layer, "mediaType", &media_type_obj))
+ layers_info[i]->media_type = strdup(json_object_get_string(media_type_obj));
+ else
+ layers_info[i]->media_type = NULL;
+
+ if (json_object_object_get_ex(layer, "size", &size_obj))
+ layers_info[i]->size = json_object_get_int64(size_obj);
+ else
+ layers_info[i]->size = 0;
}
*layer_count = len;
@@ -756,11 +786,7 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci,
return layers_info;
out_free:
- if (layers_info) {
- for (j = 0; j < i; j++)
- free(layers_info[j]);
- }
- free(layers_info);
+ ocierofs_free_layers_info(layers_info, i);
out_json:
json_object_put(root);
out:
@@ -771,8 +797,93 @@ out:
return ERR_PTR(ret);
}
-static int ocierofs_extract_layer(struct erofs_oci *oci, struct erofs_importer *importer,
- const char *digest, const char *auth_header)
+/**
+ * ocierofs_process_tar_stream - Process tar stream from file descriptor
+ * @importer: EROFS importer structure
+ * @fd: File descriptor containing tar data
+ *
+ * Initialize tar stream, parse all entries, and clean up resources.
+ *
+ * Return: 0 on success, negative errno on failure
+ */
+static int ocierofs_process_tar_stream(struct erofs_importer *importer, int fd)
+{
+ struct erofs_tarfile tarfile = {};
+ int ret;
+
+ memset(&tarfile, 0, sizeof(tarfile));
+ init_list_head(&tarfile.global.xattrs);
+
+ ret = erofs_iostream_open(&tarfile.ios, fd, EROFS_IOS_DECODER_GZIP);
+ if (ret) {
+ erofs_err("failed to initialize tar stream: %s",
+ erofs_strerror(ret));
+ return ret;
+ }
+
+ do {
+ ret = tarerofs_parse_tar(importer, &tarfile);
+ /* Continue parsing until end of archive */
+ } while (!ret);
+ erofs_iostream_close(&tarfile.ios);
+
+ if (ret < 0 && ret != -ENODATA) {
+ erofs_err("failed to process tar stream: %s",
+ erofs_strerror(ret));
+ return ret;
+ }
+
+ return 0;
+}
+
+int ocierofs_prepare_auth(struct erofs_oci *oci, char **auth_header_out,
+ bool *using_basic_auth)
+{
+ char *auth_header = NULL;
+ int ret = 0;
+
+ if (using_basic_auth)
+ *using_basic_auth = false;
+ if (auth_header_out)
+ *auth_header_out = NULL;
+
+ if (oci->params.username && oci->params.password &&
+ oci->params.username[0] && oci->params.password[0]) {
+ auth_header = ocierofs_get_auth_token(oci,
+ oci->params.registry,
+ oci->params.repository,
+ oci->params.username,
+ oci->params.password);
+ if (IS_ERR(auth_header)) {
+ auth_header = NULL;
+ ret = ocierofs_curl_setup_basic_auth(oci->curl,
+ oci->params.username,
+ oci->params.password);
+ if (ret)
+ return ret;
+ if (using_basic_auth)
+ *using_basic_auth = true;
+ }
+ } else {
+ auth_header = ocierofs_get_auth_token(oci,
+ oci->params.registry,
+ oci->params.repository,
+ NULL, NULL);
+ if (IS_ERR(auth_header))
+ auth_header = NULL;
+ }
+
+ if (auth_header_out)
+ *auth_header_out = auth_header;
+ else
+ free(auth_header);
+ return 0;
+}
+
+static int ocierofs_download_blob_to_fd(struct erofs_oci *oci,
+ const char *digest,
+ const char *auth_header,
+ int outfd)
{
struct erofs_oci_request req = {};
struct erofs_oci_stream stream = {};
@@ -782,20 +893,14 @@ static int ocierofs_extract_layer(struct erofs_oci *oci, struct erofs_importer *
stream = (struct erofs_oci_stream) {
.digest = digest,
- .blobfd = erofs_tmpfile(),
+ .blobfd = outfd,
};
- if (stream.blobfd < 0) {
- erofs_err("failed to create temporary file for %s", digest);
- return -errno;
- }
api_registry = (!strcmp(oci->params.registry, DOCKER_REGISTRY)) ?
DOCKER_API_REGISTRY : oci->params.registry;
if (asprintf(&req.url, "https://%s/v2/%s/blobs/%s",
- api_registry, oci->params.repository, digest) == -1) {
- ret = -ENOMEM;
- goto out;
- }
+ api_registry, oci->params.repository, digest) == -1)
+ return -ENOMEM;
if (auth_header && strstr(auth_header, "Bearer"))
req.headers = curl_slist_append(req.headers, auth_header);
@@ -822,6 +927,32 @@ static int ocierofs_extract_layer(struct erofs_oci *oci, struct erofs_importer *
ret = -EIO;
goto out;
}
+ ret = 0;
+out:
+ if (req.headers)
+ curl_slist_free_all(req.headers);
+ free(req.url);
+ return ret;
+}
+
+static int ocierofs_extract_layer(struct erofs_oci *oci,
+ const char *digest, const char *auth_header)
+{
+ struct erofs_oci_stream stream = {};
+ int ret;
+
+ stream = (struct erofs_oci_stream) {
+ .digest = digest,
+ .blobfd = erofs_tmpfile(),
+ };
+ if (stream.blobfd < 0) {
+ erofs_err("failed to create temporary file for %s", digest);
+ return -errno;
+ }
+
+ ret = ocierofs_download_blob_to_fd(oci, digest, auth_header, stream.blobfd);
+ if (ret)
+ goto out;
if (lseek(stream.blobfd, 0, SEEK_SET) < 0) {
erofs_err("failed to seek to beginning of temp file: %s",
@@ -830,36 +961,73 @@ static int ocierofs_extract_layer(struct erofs_oci *oci, struct erofs_importer *
goto out;
}
- memset(&stream.tarfile, 0, sizeof(stream.tarfile));
- init_list_head(&stream.tarfile.global.xattrs);
+ return stream.blobfd;
- ret = erofs_iostream_open(&stream.tarfile.ios, stream.blobfd,
- EROFS_IOS_DECODER_GZIP);
- if (ret) {
- erofs_err("failed to initialize tar stream: %s",
+out:
+ if (stream.blobfd >= 0)
+ close(stream.blobfd);
+ return ret;
+}
+
+int ocierofs_prepare_layers(struct erofs_oci *oci, char **auth_header,
+ bool *using_basic, char **manifest_digest,
+ struct ocierofs_layer_info ***layers,
+ int *layer_count, int *start_index)
+{
+ int ret;
+
+ ret = ocierofs_prepare_auth(oci, auth_header, using_basic);
+ if (ret)
+ return ret;
+
+ *manifest_digest = ocierofs_get_manifest_digest(oci, oci->params.registry,
+ oci->params.repository,
+ oci->params.tag,
+ oci->params.platform,
+ *auth_header);
+ if (IS_ERR(*manifest_digest)) {
+ ret = PTR_ERR(*manifest_digest);
+ erofs_err("failed to get manifest digest: %s",
erofs_strerror(ret));
- goto out;
+ goto out_auth;
}
- do {
- ret = tarerofs_parse_tar(importer, &stream.tarfile);
- /* Continue parsing until end of archive */
- } while (!ret);
- erofs_iostream_close(&stream.tarfile.ios);
+ *layers = ocierofs_fetch_layers_info(oci, oci->params.registry,
+ oci->params.repository,
+ *manifest_digest, *auth_header,
+ layer_count);
+ if (IS_ERR(*layers)) {
+ ret = PTR_ERR(*layers);
+ erofs_err("failed to get image layers: %s", erofs_strerror(ret));
+ goto out_manifest;
+ }
- if (ret < 0 && ret != -ENODATA) {
- erofs_err("failed to process tar stream: %s",
- erofs_strerror(ret));
- goto out;
+ if (oci->params.layer_index >= 0) {
+ if (oci->params.layer_index >= *layer_count) {
+ erofs_err("layer index %d exceeds available layers (%d)",
+ oci->params.layer_index, *layer_count);
+ ret = -EINVAL;
+ goto out_layers;
+ }
+ *layer_count = 1;
+ *start_index = oci->params.layer_index;
+ } else {
+ *start_index = 0;
}
- ret = 0;
-out:
- if (stream.blobfd >= 0)
- close(stream.blobfd);
- if (req.headers)
- curl_slist_free_all(req.headers);
- free(req.url);
+ return 0;
+
+out_layers:
+ free(*layers);
+ *layers = NULL;
+out_manifest:
+ free(*manifest_digest);
+ *manifest_digest = NULL;
+out_auth:
+ free(*auth_header);
+ *auth_header = NULL;
+ if (*using_basic)
+ ocierofs_curl_clear_auth(oci->curl);
return ret;
}
@@ -877,102 +1045,48 @@ int ocierofs_build_trees(struct erofs_importer *importer, struct erofs_oci *oci)
{
char *auth_header = NULL;
char *manifest_digest = NULL;
- char **layers = NULL;
+ struct ocierofs_layer_info **layers = NULL;
int layer_count = 0;
int ret, i;
+ bool using_basic = false;
if (!importer || !oci)
return -EINVAL;
- if (oci->params.username && oci->params.password &&
- oci->params.username[0] && oci->params.password[0]) {
- auth_header = ocierofs_get_auth_token(oci,
- oci->params.registry,
- oci->params.repository,
- oci->params.username,
- oci->params.password);
- if (IS_ERR(auth_header)) {
- auth_header = NULL;
- ret = ocierofs_curl_setup_basic_auth(oci->curl,
- oci->params.username,
- oci->params.password);
- if (ret)
- goto out;
- }
- } else {
- auth_header = ocierofs_get_auth_token(oci,
- oci->params.registry,
- oci->params.repository,
- NULL, NULL);
- if (IS_ERR(auth_header))
- auth_header = NULL;
- }
-
- manifest_digest = ocierofs_get_manifest_digest(oci, oci->params.registry,
- oci->params.repository,
- oci->params.tag,
- oci->params.platform,
- auth_header);
- if (IS_ERR(manifest_digest)) {
- ret = PTR_ERR(manifest_digest);
- erofs_err("failed to get manifest digest: %s",
- erofs_strerror(ret));
- goto out_auth;
- }
-
- layers = ocierofs_get_layers_info(oci, oci->params.registry,
- oci->params.repository,
- manifest_digest, auth_header,
- &layer_count);
- if (IS_ERR(layers)) {
- ret = PTR_ERR(layers);
- erofs_err("failed to get image layers: %s", erofs_strerror(ret));
- goto out_manifest;
- }
-
- if (oci->params.layer_index >= 0) {
- if (oci->params.layer_index >= layer_count) {
- erofs_err("layer index %d exceeds available layers (%d)",
- oci->params.layer_index, layer_count);
- ret = -EINVAL;
- goto out_layers;
- }
- layer_count = 1;
- i = oci->params.layer_index;
- } else {
- i = 0;
- }
+ ret = ocierofs_prepare_layers(oci, &auth_header, &using_basic,
+ &manifest_digest, &layers, &layer_count, &i);
+ if (ret)
+ return ret;
while (i < layer_count) {
- char *trimmed = erofs_trim_for_progressinfo(layers[i],
+ char *trimmed = erofs_trim_for_progressinfo(layers[i]->digest,
sizeof("Extracting layer ...") - 1);
erofs_update_progressinfo("Extracting layer %d: %s ...", i,
trimmed);
free(trimmed);
- ret = ocierofs_extract_layer(oci, importer, layers[i],
+ int fd = ocierofs_extract_layer(oci, layers[i]->digest,
auth_header);
- if (ret) {
+ if (fd < 0) {
erofs_err("failed to extract layer %d: %s", i,
+ erofs_strerror(fd));
+ break;
+ }
+ ret = ocierofs_process_tar_stream(importer, fd);
+ close(fd);
+ if (ret) {
+ erofs_err("failed to process tar stream for layer %d: %s", i,
erofs_strerror(ret));
break;
}
i++;
}
-out_layers:
- for (i = 0; i < layer_count; i++)
- free(layers[i]);
- free(layers);
-out_manifest:
+
+ ocierofs_free_layers_info(layers, layer_count);
free(manifest_digest);
-out_auth:
free(auth_header);
-
- if (oci->params.username && oci->params.password &&
- oci->params.username[0] && oci->params.password[0] &&
- !auth_header) {
+ if (using_basic)
ocierofs_curl_clear_auth(oci->curl);
- }
-out:
+
return ret;
}
@@ -1066,3 +1180,177 @@ int erofs_oci_params_set_string(char **field, const char *value)
*field = new_value;
return 0;
}
+
+int ocierofs_parse_options(struct erofs_oci *oci, char *options_str)
+{
+ char *opt, *q, *p;
+ int ret;
+
+ if (!options_str)
+ return 0;
+
+ opt = options_str;
+ while (opt) {
+ q = strchr(opt, ',');
+ if (q)
+ *q = '\0';
+
+ p = strstr(opt, "platform=");
+ if (p) {
+ p += strlen("platform=");
+ ret = erofs_oci_params_set_string(&oci->params.platform, p);
+ if (ret) {
+ erofs_err("failed to set platform");
+ return ret;
+ }
+ } else {
+ p = strstr(opt, "layer=");
+ if (p) {
+ p += strlen("layer=");
+ oci->params.layer_index = atoi(p);
+ if (oci->params.layer_index < 0) {
+ erofs_err("invalid layer index %d",
+ oci->params.layer_index);
+ return -EINVAL;
+ }
+ } else {
+ p = strstr(opt, "username=");
+ if (p) {
+ p += strlen("username=");
+ ret = erofs_oci_params_set_string(&oci->params.username, p);
+ if (ret) {
+ erofs_err("failed to set username");
+ return ret;
+ }
+ } else {
+ p = strstr(opt, "password=");
+ if (p) {
+ p += strlen("password=");
+ ret = erofs_oci_params_set_string(&oci->params.password, p);
+ if (ret) {
+ erofs_err("failed to set password");
+ return ret;
+ }
+ } else {
+ erofs_err("invalid --oci value %s", opt);
+ return -EINVAL;
+ }
+ }
+ }
+ }
+
+ opt = q ? q + 1 : NULL;
+ }
+
+ return 0;
+}
+
+int ocierofs_parse_ref(struct erofs_oci *oci, const char *ref_str)
+{
+ char *slash, *colon, *dot;
+ const char *repo_part;
+ size_t len;
+
+ if (!oci || !ref_str)
+ return -EINVAL;
+
+ slash = strchr(ref_str, '/');
+ if (slash) {
+ dot = strchr(ref_str, '.');
+ if (dot && dot < slash) {
+ char *registry_str;
+
+ len = slash - ref_str;
+ registry_str = strndup(ref_str, len);
+
+ if (!registry_str) {
+ erofs_err("failed to allocate memory for registry");
+ return -ENOMEM;
+ }
+ if (erofs_oci_params_set_string(&oci->params.registry,
+ registry_str)) {
+ free(registry_str);
+ erofs_err("failed to set registry");
+ return -ENOMEM;
+ }
+ free(registry_str);
+ repo_part = slash + 1;
+ } else {
+ repo_part = ref_str;
+ }
+ } else {
+ repo_part = ref_str;
+ }
+
+ colon = strchr(repo_part, ':');
+ if (colon) {
+ char *repo_str;
+
+ len = colon - repo_part;
+ repo_str = strndup(repo_part, len);
+
+ if (!repo_str) {
+ erofs_err("failed to allocate memory for repository");
+ return -ENOMEM;
+ }
+
+ if (!strchr(repo_str, '/') &&
+ (!strcmp(oci->params.registry, DOCKER_API_REGISTRY) ||
+ !strcmp(oci->params.registry, DOCKER_REGISTRY))) {
+ char *full_repo;
+
+ if (asprintf(&full_repo, "library/%s", repo_str) == -1) {
+ free(repo_str);
+ erofs_err("failed to allocate memory for full repository name");
+ return -ENOMEM;
+ }
+ free(repo_str);
+ repo_str = full_repo;
+ }
+
+ if (erofs_oci_params_set_string(&oci->params.repository,
+ repo_str)) {
+ free(repo_str);
+ erofs_err("failed to set repository");
+ return -ENOMEM;
+ }
+ free(repo_str);
+
+ if (erofs_oci_params_set_string(&oci->params.tag,
+ colon + 1)) {
+ erofs_err("failed to set tag");
+ return -ENOMEM;
+ }
+ } else {
+ char *repo_str = strdup(repo_part);
+
+ if (!repo_str) {
+ erofs_err("failed to allocate memory for repository");
+ return -ENOMEM;
+ }
+
+ if (!strchr(repo_str, '/') &&
+ (!strcmp(oci->params.registry, DOCKER_API_REGISTRY) ||
+ !strcmp(oci->params.registry, DOCKER_REGISTRY))) {
+ char *full_repo;
+
+ if (asprintf(&full_repo, "library/%s", repo_str) == -1) {
+ free(repo_str);
+ erofs_err("failed to allocate memory for full repository name");
+ return -ENOMEM;
+ }
+ free(repo_str);
+ repo_str = full_repo;
+ }
+
+ if (erofs_oci_params_set_string(&oci->params.repository,
+ repo_str)) {
+ free(repo_str);
+ erofs_err("failed to set repository");
+ return -ENOMEM;
+ }
+ free(repo_str);
+ }
+
+ return 0;
+}
diff --git a/mkfs/main.c b/mkfs/main.c
index bc895f1..825cff3 100644
--- a/mkfs/main.c
+++ b/mkfs/main.c
@@ -688,202 +688,6 @@ static int mkfs_parse_s3_cfg(char *cfg_str)
}
#endif
-#ifdef OCIEROFS_ENABLED
-
-
-/**
- * mkfs_parse_oci_options - Parse comma-separated OCI options string
- * @options_str: comma-separated options string
- *
- * Parse OCI options string containing comma-separated key=value pairs.
- * Supported options include platform, layer, username, and password.
- *
- * Return: 0 on success, negative errno on failure
- */
-static int mkfs_parse_oci_options(char *options_str)
-{
- char *opt, *q, *p;
- int ret;
-
- if (!options_str)
- return 0;
-
- opt = options_str;
- while (opt) {
- q = strchr(opt, ',');
- if (q)
- *q = '\0';
-
- p = strstr(opt, "platform=");
- if (p) {
- p += strlen("platform=");
- ret = erofs_oci_params_set_string(&ocicfg.params.platform, p);
- if (ret) {
- erofs_err("failed to set platform");
- return ret;
- }
- } else {
- p = strstr(opt, "layer=");
- if (p) {
- p += strlen("layer=");
- ocicfg.params.layer_index = atoi(p);
- if (ocicfg.params.layer_index < 0) {
- erofs_err("invalid layer index %d",
- ocicfg.params.layer_index);
- return -EINVAL;
- }
- } else {
- p = strstr(opt, "username=");
- if (p) {
- p += strlen("username=");
- ret = erofs_oci_params_set_string(&ocicfg.params.username, p);
- if (ret) {
- erofs_err("failed to set username");
- return ret;
- }
- } else {
- p = strstr(opt, "password=");
- if (p) {
- p += strlen("password=");
- ret = erofs_oci_params_set_string(&ocicfg.params.password, p);
- if (ret) {
- erofs_err("failed to set password");
- return ret;
- }
- } else {
- erofs_err("invalid --oci value %s", opt);
- return -EINVAL;
- }
- }
- }
- }
-
- opt = q ? q + 1 : NULL;
- }
-
- return 0;
-}
-
-/**
- * mkfs_parse_oci_ref - Parse OCI image reference string
- * @ref_str: OCI image reference in various formats
- *
- * Parse OCI image reference which can be in formats:
- * - registry.example.com/namespace/repo:tag
- * - namespace/repo:tag (uses default registry)
- * - repo:tag (adds library/ prefix for Docker Hub)
- * - repo (uses default tag "latest")
- *
- * Return: 0 on success, negative errno on failure
- */
-static int mkfs_parse_oci_ref(const char *ref_str)
-{
- char *slash, *colon, *dot;
- const char *repo_part;
- size_t len;
-
- slash = strchr(ref_str, '/');
- if (slash) {
- dot = strchr(ref_str, '.');
- if (dot && dot < slash) {
- char *registry_str;
-
- len = slash - ref_str;
- registry_str = strndup(ref_str, len);
-
- if (!registry_str) {
- erofs_err("failed to allocate memory for registry");
- return -ENOMEM;
- }
- if (erofs_oci_params_set_string(&ocicfg.params.registry,
- registry_str)) {
- free(registry_str);
- erofs_err("failed to set registry");
- return -ENOMEM;
- }
- free(registry_str);
- repo_part = slash + 1;
- } else {
- repo_part = ref_str;
- }
- } else {
- repo_part = ref_str;
- }
-
- colon = strchr(repo_part, ':');
- if (colon) {
- char *repo_str;
-
- len = colon - repo_part;
- repo_str = strndup(repo_part, len);
-
- if (!repo_str) {
- erofs_err("failed to allocate memory for repository");
- return -ENOMEM;
- }
-
- if (!strchr(repo_str, '/') &&
- (!strcmp(ocicfg.params.registry, DOCKER_API_REGISTRY) ||
- !strcmp(ocicfg.params.registry, DOCKER_REGISTRY))) {
- char *full_repo;
-
- if (asprintf(&full_repo, "library/%s", repo_str) == -1) {
- free(repo_str);
- erofs_err("failed to allocate memory for full repository name");
- return -ENOMEM;
- }
- free(repo_str);
- repo_str = full_repo;
- }
-
- if (erofs_oci_params_set_string(&ocicfg.params.repository,
- repo_str)) {
- free(repo_str);
- erofs_err("failed to set repository");
- return -ENOMEM;
- }
- free(repo_str);
-
- if (erofs_oci_params_set_string(&ocicfg.params.tag,
- colon + 1)) {
- erofs_err("failed to set tag");
- return -ENOMEM;
- }
- } else {
- char *repo_str = strdup(repo_part);
-
- if (!repo_str) {
- erofs_err("failed to allocate memory for repository");
- return -ENOMEM;
- }
-
- if (!strchr(repo_str, '/') &&
- (!strcmp(ocicfg.params.registry, DOCKER_API_REGISTRY) ||
- !strcmp(ocicfg.params.registry, DOCKER_REGISTRY))) {
- char *full_repo;
-
- if (asprintf(&full_repo, "library/%s", repo_str) == -1) {
- free(repo_str);
- erofs_err("failed to allocate memory for full repository name");
- return -ENOMEM;
- }
- free(repo_str);
- repo_str = full_repo;
- }
-
- if (erofs_oci_params_set_string(&ocicfg.params.repository,
- repo_str)) {
- free(repo_str);
- erofs_err("failed to set repository");
- return -ENOMEM;
- }
- free(repo_str);
- }
-
- return 0;
-}
-#endif
-
static int mkfs_parse_one_compress_alg(char *alg,
struct erofs_compr_opts *copts)
{
@@ -1955,10 +1759,10 @@ int main(int argc, char **argv)
if (err)
goto exit;
- err = mkfs_parse_oci_options(mkfs_oci_options);
+ err = ocierofs_parse_options(&ocicfg, mkfs_oci_options);
if (err)
goto exit;
- err = mkfs_parse_oci_ref(cfg.c_src_path);
+ err = ocierofs_parse_ref(&ocicfg, cfg.c_src_path);
if (err)
goto exit;
--
2.51.0
^ permalink raw reply related [flat|nested] 19+ messages in thread* [PATCH v1-changed] erofs-utils: refactor OCI code for better modularity 2025-09-01 5:10 [PATCH v1] erofs-utils: refactor OCI code for better modularity ChengyuZhu6 @ 2025-09-01 7:01 ` ChengyuZhu6 2025-09-02 1:52 ` Gao Xiang 2025-09-02 3:17 ` [PATCH v2] " ChengyuZhu6 ` (4 subsequent siblings) 5 siblings, 1 reply; 19+ messages in thread From: ChengyuZhu6 @ 2025-09-01 7:01 UTC (permalink / raw) To: linux-erofs; +Cc: xiang, hsiangkao, Chengyu Zhu From: Chengyu Zhu <hudsonzhu@tencent.com> Refactor OCI code to improve code organization and maintainability: - Add `struct ocierofs_layer_info` to encapsulate layer metadata - Extract authentication logic into `ocierofs_prepare_auth()` - Split layer processing into `ocierofs_prepare_layers()` - Move OCI parsing functions from `mkfs/main.c` to `lib/remotes/oci.c` - Add `ocierofs_process_tar_stream()` for separate tar processing - Improve error handling with `ocierofs_free_layers_info()` - Refactor `ocierofs_extract_layer()` to return file descriptor Signed-off-by: Chengyu Zhu <hudsonzhu@tencent.com> --- lib/liberofs_oci.h | 100 +++++++++ lib/remotes/oci.c | 540 +++++++++++++++++++++++++++++++++------------ mkfs/main.c | 200 +---------------- 3 files changed, 506 insertions(+), 334 deletions(-) diff --git a/lib/liberofs_oci.h b/lib/liberofs_oci.h index 3a8108b..698fe07 100644 --- a/lib/liberofs_oci.h +++ b/lib/liberofs_oci.h @@ -19,6 +19,23 @@ struct erofs_inode; struct CURL; struct erofs_importer; +/** + * struct ocierofs_layer_info + * @digest: OCI content-addressable digest (e.g. "sha256:...") + * @media_type: mediaType string from the manifest + * @size: layer size in bytes from the manifest (0 if not available) + * + * This structure is exposed to callers so they can enumerate image layers, + * decide which ones to fetch, and pass the digest back to download APIs. + * Fields are heap-allocated NUL-terminated strings owned by the caller + * once returned from public APIs; the caller must free them. + */ +struct ocierofs_layer_info { + char *digest; + char *media_type; + u64 size; +}; + /** * struct erofs_oci_params - OCI configuration parameters * @registry: registry hostname (e.g., "registry-1.docker.io") @@ -88,6 +105,89 @@ int erofs_oci_params_set_string(char **field, const char *value); */ int ocierofs_build_trees(struct erofs_importer *importer, struct erofs_oci *oci); +/* + * ocierofs_parse_options - Parse comma-separated OCI options string + * @oci: OCI client structure to update + * @options_str: comma-separated options string + * + * Parse OCI options string containing comma-separated key=value pairs. + * Supported options include platform, layer, username, and password. + * + * Return: 0 on success, negative errno on failure + */ +int ocierofs_parse_options(struct erofs_oci *oci, char *options_str); + +/* + * ocierofs_parse_ref - Parse OCI image reference string + * @oci: OCI client structure to update + * @ref_str: OCI image reference in various formats + * + * Parse OCI image reference which can be in formats: + * - registry.example.com/namespace/repo:tag + * - namespace/repo:tag (uses default registry) + * - repo:tag (adds library/ prefix for Docker Hub) + * - repo (uses default tag "latest") + * + * Return: 0 on success, negative errno on failure + */ +int ocierofs_parse_ref(struct erofs_oci *oci, const char *ref_str); + +/* + * ocierofs_prepare_layers - Prepare OCI layers for processing + * @oci: OCI client structure with configured parameters + * @auth_header: Pointer to store authentication header + * @using_basic: Pointer to store basic auth flag + * @manifest_digest: Pointer to store manifest digest + * @layers: Pointer to store layers information + * @layer_count: Pointer to store number of layers + * @start_index: Pointer to store starting layer index + * + * Prepare authentication, get manifest digest and layers information + * for OCI image processing. This function handles all the preparation + * work needed before processing OCI layers. + * + * Return: 0 on success, negative errno on failure + */ +int ocierofs_prepare_layers(struct erofs_oci *oci, char **auth_header, + bool *using_basic, char **manifest_digest, + struct ocierofs_layer_info ***layers, + int *layer_count, int *start_index); + +/** + * ocierofs_free_layers_info - Free layer information array + * @layers: array of layer information structures + * @count: number of layers in the array + * + * Free all layer information structures and the array itself. + * This function handles NULL pointers safely. + */ +void ocierofs_free_layers_info(struct ocierofs_layer_info **layers, int count); + +/** + * ocierofs_prepare_auth - Prepare authentication for OCI requests + * @oci: OCI client structure + * @auth_header_out: pointer to store authentication header + * @using_basic_auth: pointer to store basic auth flag + * + * Prepare authentication header for OCI registry requests. + * This function handles both token-based and basic authentication. + * + * Return: 0 on success, negative errno on failure + */ +int ocierofs_prepare_auth(struct erofs_oci *oci, char **auth_header_out, + bool *using_basic_auth); + +/** + * ocierofs_curl_clear_auth - Clear basic authentication from CURL handle + * @curl: CURL handle to clear authentication from + * + * Clear basic authentication credentials from a CURL handle. + * This should be called after using basic authentication to clean up. + * + * Return: 0 on success, negative errno on failure + */ +int ocierofs_curl_clear_auth(struct CURL *curl); + #ifdef __cplusplus } #endif diff --git a/lib/remotes/oci.c b/lib/remotes/oci.c index 0fb8c1f..9774d8d 100644 --- a/lib/remotes/oci.c +++ b/lib/remotes/oci.c @@ -42,7 +42,6 @@ struct erofs_oci_response { }; struct erofs_oci_stream { - struct erofs_tarfile tarfile; const char *digest; int blobfd; }; @@ -111,7 +110,7 @@ static int ocierofs_curl_setup_basic_auth(struct CURL *curl, const char *usernam return 0; } -static int ocierofs_curl_clear_auth(struct CURL *curl) +int ocierofs_curl_clear_auth(struct CURL *curl) { curl_easy_setopt(curl, CURLOPT_USERPWD, NULL); curl_easy_setopt(curl, CURLOPT_HTTPAUTH, CURLAUTH_NONE); @@ -181,7 +180,7 @@ static int ocierofs_request_perform(struct erofs_oci *oci, ret = ocierofs_curl_setup_rq(oci->curl, req->url, OCIEROFS_HTTP_GET, req->headers, - ocierofs_write_callback, resp, + ocierofs_write_callback, resp, NULL, NULL); if (ret) return ret; @@ -568,7 +567,7 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci, const char *api_registry; int ret = 0, len, i; - if (!registry || !repository || !tag || !platform) + if (!registry || !repository || !tag) return ERR_PTR(-EINVAL); api_registry = (!strcmp(registry, DOCKER_REGISTRY)) ? DOCKER_API_REGISTRY : registry; @@ -581,8 +580,8 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci, req.headers = curl_slist_append(req.headers, "Accept: " DOCKER_MEDIATYPE_MANIFEST_LIST "," - OCI_MEDIATYPE_INDEX "," DOCKER_MEDIATYPE_MANIFEST_V1 "," - DOCKER_MEDIATYPE_MANIFEST_V2); + OCI_MEDIATYPE_INDEX "," OCI_MEDIATYPE_MANIFEST "," + DOCKER_MEDIATYPE_MANIFEST_V1 "," DOCKER_MEDIATYPE_MANIFEST_V2); ret = ocierofs_request_perform(oci, &req, &resp); if (ret) @@ -663,7 +662,24 @@ out: return ret ? ERR_PTR(ret) : digest; } -static char **ocierofs_get_layers_info(struct erofs_oci *oci, +void ocierofs_free_layers_info(struct ocierofs_layer_info **layers, int count) +{ + int i; + + if (!layers) + return; + + for (i = 0; i < count; i++) { + if (layers[i]) { + free(layers[i]->digest); + free(layers[i]->media_type); + free(layers[i]); + } + } + free(layers); +} + +static struct ocierofs_layer_info **ocierofs_fetch_layers_info(struct erofs_oci *oci, const char *registry, const char *repository, const char *digest, @@ -672,10 +688,10 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci, { struct erofs_oci_request req = {}; struct erofs_oci_response resp = {}; - json_object *root, *layers, *layer, *digest_obj; - char **layers_info = NULL; + json_object *root, *layers, *layer, *digest_obj, *media_type_obj, *size_obj; + struct ocierofs_layer_info **layers_info = NULL; const char *api_registry; - int ret, len, i, j; + int ret, len, i; if (!registry || !repository || !digest || !layer_count) return ERR_PTR(-EINVAL); @@ -725,7 +741,7 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci, goto out_json; } - layers_info = calloc(len, sizeof(char *)); + layers_info = calloc(len, sizeof(*layers_info)); if (!layers_info) { ret = -ENOMEM; goto out_json; @@ -740,11 +756,25 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci, goto out_free; } - layers_info[i] = strdup(json_object_get_string(digest_obj)); + layers_info[i] = calloc(1, sizeof(**layers_info)); if (!layers_info[i]) { ret = -ENOMEM; goto out_free; } + layers_info[i]->digest = strdup(json_object_get_string(digest_obj)); + if (!layers_info[i]->digest) { + ret = -ENOMEM; + goto out_free; + } + if (json_object_object_get_ex(layer, "mediaType", &media_type_obj)) + layers_info[i]->media_type = strdup(json_object_get_string(media_type_obj)); + else + layers_info[i]->media_type = NULL; + + if (json_object_object_get_ex(layer, "size", &size_obj)) + layers_info[i]->size = json_object_get_int64(size_obj); + else + layers_info[i]->size = 0; } *layer_count = len; @@ -756,11 +786,7 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci, return layers_info; out_free: - if (layers_info) { - for (j = 0; j < i; j++) - free(layers_info[j]); - } - free(layers_info); + ocierofs_free_layers_info(layers_info, i); out_json: json_object_put(root); out: @@ -771,8 +797,93 @@ out: return ERR_PTR(ret); } -static int ocierofs_extract_layer(struct erofs_oci *oci, struct erofs_importer *importer, - const char *digest, const char *auth_header) +/** + * ocierofs_process_tar_stream - Process tar stream from file descriptor + * @importer: EROFS importer structure + * @fd: File descriptor containing tar data + * + * Initialize tar stream, parse all entries, and clean up resources. + * + * Return: 0 on success, negative errno on failure + */ +static int ocierofs_process_tar_stream(struct erofs_importer *importer, int fd) +{ + struct erofs_tarfile tarfile = {}; + int ret; + + memset(&tarfile, 0, sizeof(tarfile)); + init_list_head(&tarfile.global.xattrs); + + ret = erofs_iostream_open(&tarfile.ios, fd, EROFS_IOS_DECODER_GZIP); + if (ret) { + erofs_err("failed to initialize tar stream: %s", + erofs_strerror(ret)); + return ret; + } + + do { + ret = tarerofs_parse_tar(importer, &tarfile); + /* Continue parsing until end of archive */ + } while (!ret); + erofs_iostream_close(&tarfile.ios); + + if (ret < 0 && ret != -ENODATA) { + erofs_err("failed to process tar stream: %s", + erofs_strerror(ret)); + return ret; + } + + return 0; +} + +int ocierofs_prepare_auth(struct erofs_oci *oci, char **auth_header_out, + bool *using_basic_auth) +{ + char *auth_header = NULL; + int ret = 0; + + if (using_basic_auth) + *using_basic_auth = false; + if (auth_header_out) + *auth_header_out = NULL; + + if (oci->params.username && oci->params.password && + oci->params.username[0] && oci->params.password[0]) { + auth_header = ocierofs_get_auth_token(oci, + oci->params.registry, + oci->params.repository, + oci->params.username, + oci->params.password); + if (IS_ERR(auth_header)) { + auth_header = NULL; + ret = ocierofs_curl_setup_basic_auth(oci->curl, + oci->params.username, + oci->params.password); + if (ret) + return ret; + if (using_basic_auth) + *using_basic_auth = true; + } + } else { + auth_header = ocierofs_get_auth_token(oci, + oci->params.registry, + oci->params.repository, + NULL, NULL); + if (IS_ERR(auth_header)) + auth_header = NULL; + } + + if (auth_header_out) + *auth_header_out = auth_header; + else + free(auth_header); + return 0; +} + +static int ocierofs_download_blob_to_fd(struct erofs_oci *oci, + const char *digest, + const char *auth_header, + int outfd) { struct erofs_oci_request req = {}; struct erofs_oci_stream stream = {}; @@ -782,20 +893,14 @@ static int ocierofs_extract_layer(struct erofs_oci *oci, struct erofs_importer * stream = (struct erofs_oci_stream) { .digest = digest, - .blobfd = erofs_tmpfile(), + .blobfd = outfd, }; - if (stream.blobfd < 0) { - erofs_err("failed to create temporary file for %s", digest); - return -errno; - } api_registry = (!strcmp(oci->params.registry, DOCKER_REGISTRY)) ? DOCKER_API_REGISTRY : oci->params.registry; if (asprintf(&req.url, "https://%s/v2/%s/blobs/%s", - api_registry, oci->params.repository, digest) == -1) { - ret = -ENOMEM; - goto out; - } + api_registry, oci->params.repository, digest) == -1) + return -ENOMEM; if (auth_header && strstr(auth_header, "Bearer")) req.headers = curl_slist_append(req.headers, auth_header); @@ -822,6 +927,32 @@ static int ocierofs_extract_layer(struct erofs_oci *oci, struct erofs_importer * ret = -EIO; goto out; } + ret = 0; +out: + if (req.headers) + curl_slist_free_all(req.headers); + free(req.url); + return ret; +} + +static int ocierofs_extract_layer(struct erofs_oci *oci, + const char *digest, const char *auth_header) +{ + struct erofs_oci_stream stream = {}; + int ret; + + stream = (struct erofs_oci_stream) { + .digest = digest, + .blobfd = erofs_tmpfile(), + }; + if (stream.blobfd < 0) { + erofs_err("failed to create temporary file for %s", digest); + return -errno; + } + + ret = ocierofs_download_blob_to_fd(oci, digest, auth_header, stream.blobfd); + if (ret) + goto out; if (lseek(stream.blobfd, 0, SEEK_SET) < 0) { erofs_err("failed to seek to beginning of temp file: %s", @@ -830,162 +961,125 @@ static int ocierofs_extract_layer(struct erofs_oci *oci, struct erofs_importer * goto out; } - memset(&stream.tarfile, 0, sizeof(stream.tarfile)); - init_list_head(&stream.tarfile.global.xattrs); - - ret = erofs_iostream_open(&stream.tarfile.ios, stream.blobfd, - EROFS_IOS_DECODER_GZIP); - if (ret) { - erofs_err("failed to initialize tar stream: %s", - erofs_strerror(ret)); - goto out; - } - - do { - ret = tarerofs_parse_tar(importer, &stream.tarfile); - /* Continue parsing until end of archive */ - } while (!ret); - erofs_iostream_close(&stream.tarfile.ios); - - if (ret < 0 && ret != -ENODATA) { - erofs_err("failed to process tar stream: %s", - erofs_strerror(ret)); - goto out; - } - ret = 0; + return stream.blobfd; out: if (stream.blobfd >= 0) close(stream.blobfd); - if (req.headers) - curl_slist_free_all(req.headers); - free(req.url); return ret; } -/** - * ocierofs_build_trees - Build file trees from OCI container image layers - * @importer: EROFS importer structure - * @oci: OCI client structure with configured parameters - * - * Extract and build file system trees from all layers of an OCI container - * image. - * - * Return: 0 on success, negative errno on failure - */ -int ocierofs_build_trees(struct erofs_importer *importer, struct erofs_oci *oci) +int ocierofs_prepare_layers(struct erofs_oci *oci, char **auth_header, + bool *using_basic, char **manifest_digest, + struct ocierofs_layer_info ***layers, + int *layer_count, int *start_index) { - char *auth_header = NULL; - char *manifest_digest = NULL; - char **layers = NULL; - int layer_count = 0; - int ret, i; - - if (!importer || !oci) - return -EINVAL; + int ret; - if (oci->params.username && oci->params.password && - oci->params.username[0] && oci->params.password[0]) { - auth_header = ocierofs_get_auth_token(oci, - oci->params.registry, - oci->params.repository, - oci->params.username, - oci->params.password); - if (IS_ERR(auth_header)) { - auth_header = NULL; - ret = ocierofs_curl_setup_basic_auth(oci->curl, - oci->params.username, - oci->params.password); - if (ret) - goto out; - } - } else { - auth_header = ocierofs_get_auth_token(oci, - oci->params.registry, - oci->params.repository, - NULL, NULL); - if (IS_ERR(auth_header)) - auth_header = NULL; - } + ret = ocierofs_prepare_auth(oci, auth_header, using_basic); + if (ret) + return ret; - manifest_digest = ocierofs_get_manifest_digest(oci, oci->params.registry, + *manifest_digest = ocierofs_get_manifest_digest(oci, oci->params.registry, oci->params.repository, oci->params.tag, oci->params.platform, - auth_header); - if (IS_ERR(manifest_digest)) { - ret = PTR_ERR(manifest_digest); + *auth_header); + if (IS_ERR(*manifest_digest)) { + ret = PTR_ERR(*manifest_digest); erofs_err("failed to get manifest digest: %s", erofs_strerror(ret)); goto out_auth; } - layers = ocierofs_get_layers_info(oci, oci->params.registry, - oci->params.repository, - manifest_digest, auth_header, - &layer_count); - if (IS_ERR(layers)) { - ret = PTR_ERR(layers); + *layers = ocierofs_fetch_layers_info(oci, oci->params.registry, + oci->params.repository, + *manifest_digest, *auth_header, + layer_count); + if (IS_ERR(*layers)) { + ret = PTR_ERR(*layers); erofs_err("failed to get image layers: %s", erofs_strerror(ret)); goto out_manifest; } if (oci->params.layer_index >= 0) { - if (oci->params.layer_index >= layer_count) { + if (oci->params.layer_index >= *layer_count) { erofs_err("layer index %d exceeds available layers (%d)", - oci->params.layer_index, layer_count); + oci->params.layer_index, *layer_count); ret = -EINVAL; goto out_layers; } - layer_count = 1; - i = oci->params.layer_index; + *layer_count = 1; + *start_index = oci->params.layer_index; } else { - i = 0; + *start_index = 0; } + return 0; + +out_layers: + free(*layers); + *layers = NULL; +out_manifest: + free(*manifest_digest); + *manifest_digest = NULL; +out_auth: + free(*auth_header); + *auth_header = NULL; + if (*using_basic) + ocierofs_curl_clear_auth(oci->curl); + return ret; +} + +int ocierofs_build_trees(struct erofs_importer *importer, struct erofs_oci *oci) +{ + char *auth_header = NULL; + char *manifest_digest = NULL; + struct ocierofs_layer_info **layers = NULL; + int layer_count = 0; + int ret, i; + bool using_basic = false; + + if (!importer || !oci) + return -EINVAL; + + ret = ocierofs_prepare_layers(oci, &auth_header, &using_basic, + &manifest_digest, &layers, &layer_count, &i); + if (ret) + return ret; + while (i < layer_count) { - char *trimmed = erofs_trim_for_progressinfo(layers[i], + char *trimmed = erofs_trim_for_progressinfo(layers[i]->digest, sizeof("Extracting layer ...") - 1); erofs_update_progressinfo("Extracting layer %d: %s ...", i, trimmed); free(trimmed); - ret = ocierofs_extract_layer(oci, importer, layers[i], + int fd = ocierofs_extract_layer(oci, layers[i]->digest, auth_header); - if (ret) { + if (fd < 0) { erofs_err("failed to extract layer %d: %s", i, + erofs_strerror(fd)); + break; + } + ret = ocierofs_process_tar_stream(importer, fd); + close(fd); + if (ret) { + erofs_err("failed to process tar stream for layer %d: %s", i, erofs_strerror(ret)); break; } i++; } -out_layers: - for (i = 0; i < layer_count; i++) - free(layers[i]); - free(layers); -out_manifest: + + ocierofs_free_layers_info(layers, layer_count); free(manifest_digest); -out_auth: free(auth_header); - - if (oci->params.username && oci->params.password && - oci->params.username[0] && oci->params.password[0] && - !auth_header) { + if (using_basic) ocierofs_curl_clear_auth(oci->curl); - } -out: + return ret; } -/** - * ocierofs_init - Initialize OCI client with default parameters - * @oci: OCI client structure to initialize - * - * Initialize OCI client structure, set up CURL handle, and configure - * default parameters including platform (linux/amd64), registry - * (registry-1.docker.io), and tag (latest). - * - * Return: 0 on success, negative errno on failure - */ int ocierofs_init(struct erofs_oci *oci) { if (!oci) @@ -1066,3 +1160,177 @@ int erofs_oci_params_set_string(char **field, const char *value) *field = new_value; return 0; } + +int ocierofs_parse_options(struct erofs_oci *oci, char *options_str) +{ + char *opt, *q, *p; + int ret; + + if (!options_str) + return 0; + + opt = options_str; + while (opt) { + q = strchr(opt, ','); + if (q) + *q = '\0'; + + p = strstr(opt, "platform="); + if (p) { + p += strlen("platform="); + ret = erofs_oci_params_set_string(&oci->params.platform, p); + if (ret) { + erofs_err("failed to set platform"); + return ret; + } + } else { + p = strstr(opt, "layer="); + if (p) { + p += strlen("layer="); + oci->params.layer_index = atoi(p); + if (oci->params.layer_index < 0) { + erofs_err("invalid layer index %d", + oci->params.layer_index); + return -EINVAL; + } + } else { + p = strstr(opt, "username="); + if (p) { + p += strlen("username="); + ret = erofs_oci_params_set_string(&oci->params.username, p); + if (ret) { + erofs_err("failed to set username"); + return ret; + } + } else { + p = strstr(opt, "password="); + if (p) { + p += strlen("password="); + ret = erofs_oci_params_set_string(&oci->params.password, p); + if (ret) { + erofs_err("failed to set password"); + return ret; + } + } else { + erofs_err("invalid --oci value %s", opt); + return -EINVAL; + } + } + } + } + + opt = q ? q + 1 : NULL; + } + + return 0; +} + +int ocierofs_parse_ref(struct erofs_oci *oci, const char *ref_str) +{ + char *slash, *colon, *dot; + const char *repo_part; + size_t len; + + if (!oci || !ref_str) + return -EINVAL; + + slash = strchr(ref_str, '/'); + if (slash) { + dot = strchr(ref_str, '.'); + if (dot && dot < slash) { + char *registry_str; + + len = slash - ref_str; + registry_str = strndup(ref_str, len); + + if (!registry_str) { + erofs_err("failed to allocate memory for registry"); + return -ENOMEM; + } + if (erofs_oci_params_set_string(&oci->params.registry, + registry_str)) { + free(registry_str); + erofs_err("failed to set registry"); + return -ENOMEM; + } + free(registry_str); + repo_part = slash + 1; + } else { + repo_part = ref_str; + } + } else { + repo_part = ref_str; + } + + colon = strchr(repo_part, ':'); + if (colon) { + char *repo_str; + + len = colon - repo_part; + repo_str = strndup(repo_part, len); + + if (!repo_str) { + erofs_err("failed to allocate memory for repository"); + return -ENOMEM; + } + + if (!strchr(repo_str, '/') && + (!strcmp(oci->params.registry, DOCKER_API_REGISTRY) || + !strcmp(oci->params.registry, DOCKER_REGISTRY))) { + char *full_repo; + + if (asprintf(&full_repo, "library/%s", repo_str) == -1) { + free(repo_str); + erofs_err("failed to allocate memory for full repository name"); + return -ENOMEM; + } + free(repo_str); + repo_str = full_repo; + } + + if (erofs_oci_params_set_string(&oci->params.repository, + repo_str)) { + free(repo_str); + erofs_err("failed to set repository"); + return -ENOMEM; + } + free(repo_str); + + if (erofs_oci_params_set_string(&oci->params.tag, + colon + 1)) { + erofs_err("failed to set tag"); + return -ENOMEM; + } + } else { + char *repo_str = strdup(repo_part); + + if (!repo_str) { + erofs_err("failed to allocate memory for repository"); + return -ENOMEM; + } + + if (!strchr(repo_str, '/') && + (!strcmp(oci->params.registry, DOCKER_API_REGISTRY) || + !strcmp(oci->params.registry, DOCKER_REGISTRY))) { + char *full_repo; + + if (asprintf(&full_repo, "library/%s", repo_str) == -1) { + free(repo_str); + erofs_err("failed to allocate memory for full repository name"); + return -ENOMEM; + } + free(repo_str); + repo_str = full_repo; + } + + if (erofs_oci_params_set_string(&oci->params.repository, + repo_str)) { + free(repo_str); + erofs_err("failed to set repository"); + return -ENOMEM; + } + free(repo_str); + } + + return 0; +} diff --git a/mkfs/main.c b/mkfs/main.c index bc895f1..825cff3 100644 --- a/mkfs/main.c +++ b/mkfs/main.c @@ -688,202 +688,6 @@ static int mkfs_parse_s3_cfg(char *cfg_str) } #endif -#ifdef OCIEROFS_ENABLED - - -/** - * mkfs_parse_oci_options - Parse comma-separated OCI options string - * @options_str: comma-separated options string - * - * Parse OCI options string containing comma-separated key=value pairs. - * Supported options include platform, layer, username, and password. - * - * Return: 0 on success, negative errno on failure - */ -static int mkfs_parse_oci_options(char *options_str) -{ - char *opt, *q, *p; - int ret; - - if (!options_str) - return 0; - - opt = options_str; - while (opt) { - q = strchr(opt, ','); - if (q) - *q = '\0'; - - p = strstr(opt, "platform="); - if (p) { - p += strlen("platform="); - ret = erofs_oci_params_set_string(&ocicfg.params.platform, p); - if (ret) { - erofs_err("failed to set platform"); - return ret; - } - } else { - p = strstr(opt, "layer="); - if (p) { - p += strlen("layer="); - ocicfg.params.layer_index = atoi(p); - if (ocicfg.params.layer_index < 0) { - erofs_err("invalid layer index %d", - ocicfg.params.layer_index); - return -EINVAL; - } - } else { - p = strstr(opt, "username="); - if (p) { - p += strlen("username="); - ret = erofs_oci_params_set_string(&ocicfg.params.username, p); - if (ret) { - erofs_err("failed to set username"); - return ret; - } - } else { - p = strstr(opt, "password="); - if (p) { - p += strlen("password="); - ret = erofs_oci_params_set_string(&ocicfg.params.password, p); - if (ret) { - erofs_err("failed to set password"); - return ret; - } - } else { - erofs_err("invalid --oci value %s", opt); - return -EINVAL; - } - } - } - } - - opt = q ? q + 1 : NULL; - } - - return 0; -} - -/** - * mkfs_parse_oci_ref - Parse OCI image reference string - * @ref_str: OCI image reference in various formats - * - * Parse OCI image reference which can be in formats: - * - registry.example.com/namespace/repo:tag - * - namespace/repo:tag (uses default registry) - * - repo:tag (adds library/ prefix for Docker Hub) - * - repo (uses default tag "latest") - * - * Return: 0 on success, negative errno on failure - */ -static int mkfs_parse_oci_ref(const char *ref_str) -{ - char *slash, *colon, *dot; - const char *repo_part; - size_t len; - - slash = strchr(ref_str, '/'); - if (slash) { - dot = strchr(ref_str, '.'); - if (dot && dot < slash) { - char *registry_str; - - len = slash - ref_str; - registry_str = strndup(ref_str, len); - - if (!registry_str) { - erofs_err("failed to allocate memory for registry"); - return -ENOMEM; - } - if (erofs_oci_params_set_string(&ocicfg.params.registry, - registry_str)) { - free(registry_str); - erofs_err("failed to set registry"); - return -ENOMEM; - } - free(registry_str); - repo_part = slash + 1; - } else { - repo_part = ref_str; - } - } else { - repo_part = ref_str; - } - - colon = strchr(repo_part, ':'); - if (colon) { - char *repo_str; - - len = colon - repo_part; - repo_str = strndup(repo_part, len); - - if (!repo_str) { - erofs_err("failed to allocate memory for repository"); - return -ENOMEM; - } - - if (!strchr(repo_str, '/') && - (!strcmp(ocicfg.params.registry, DOCKER_API_REGISTRY) || - !strcmp(ocicfg.params.registry, DOCKER_REGISTRY))) { - char *full_repo; - - if (asprintf(&full_repo, "library/%s", repo_str) == -1) { - free(repo_str); - erofs_err("failed to allocate memory for full repository name"); - return -ENOMEM; - } - free(repo_str); - repo_str = full_repo; - } - - if (erofs_oci_params_set_string(&ocicfg.params.repository, - repo_str)) { - free(repo_str); - erofs_err("failed to set repository"); - return -ENOMEM; - } - free(repo_str); - - if (erofs_oci_params_set_string(&ocicfg.params.tag, - colon + 1)) { - erofs_err("failed to set tag"); - return -ENOMEM; - } - } else { - char *repo_str = strdup(repo_part); - - if (!repo_str) { - erofs_err("failed to allocate memory for repository"); - return -ENOMEM; - } - - if (!strchr(repo_str, '/') && - (!strcmp(ocicfg.params.registry, DOCKER_API_REGISTRY) || - !strcmp(ocicfg.params.registry, DOCKER_REGISTRY))) { - char *full_repo; - - if (asprintf(&full_repo, "library/%s", repo_str) == -1) { - free(repo_str); - erofs_err("failed to allocate memory for full repository name"); - return -ENOMEM; - } - free(repo_str); - repo_str = full_repo; - } - - if (erofs_oci_params_set_string(&ocicfg.params.repository, - repo_str)) { - free(repo_str); - erofs_err("failed to set repository"); - return -ENOMEM; - } - free(repo_str); - } - - return 0; -} -#endif - static int mkfs_parse_one_compress_alg(char *alg, struct erofs_compr_opts *copts) { @@ -1955,10 +1759,10 @@ int main(int argc, char **argv) if (err) goto exit; - err = mkfs_parse_oci_options(mkfs_oci_options); + err = ocierofs_parse_options(&ocicfg, mkfs_oci_options); if (err) goto exit; - err = mkfs_parse_oci_ref(cfg.c_src_path); + err = ocierofs_parse_ref(&ocicfg, cfg.c_src_path); if (err) goto exit; -- 2.51.0 ^ permalink raw reply related [flat|nested] 19+ messages in thread
* Re: [PATCH v1-changed] erofs-utils: refactor OCI code for better modularity 2025-09-01 7:01 ` [PATCH v1-changed] " ChengyuZhu6 @ 2025-09-02 1:52 ` Gao Xiang 0 siblings, 0 replies; 19+ messages in thread From: Gao Xiang @ 2025-09-02 1:52 UTC (permalink / raw) To: ChengyuZhu6, linux-erofs; +Cc: xiang, Chengyu Zhu On 2025/9/1 15:01, ChengyuZhu6 wrote: > From: Chengyu Zhu <hudsonzhu@tencent.com> > > Refactor OCI code to improve code organization and maintainability: > > - Add `struct ocierofs_layer_info` to encapsulate layer metadata > - Extract authentication logic into `ocierofs_prepare_auth()` > - Split layer processing into `ocierofs_prepare_layers()` > - Move OCI parsing functions from `mkfs/main.c` to `lib/remotes/oci.c` > - Add `ocierofs_process_tar_stream()` for separate tar processing > - Improve error handling with `ocierofs_free_layers_info()` > - Refactor `ocierofs_extract_layer()` to return file descriptor > > Signed-off-by: Chengyu Zhu <hudsonzhu@tencent.com> > --- > lib/liberofs_oci.h | 100 +++++++++ > lib/remotes/oci.c | 540 +++++++++++++++++++++++++++++++++------------ > mkfs/main.c | 200 +---------------- > 3 files changed, 506 insertions(+), 334 deletions(-) > > diff --git a/lib/liberofs_oci.h b/lib/liberofs_oci.h > index 3a8108b..698fe07 100644 > --- a/lib/liberofs_oci.h > +++ b/lib/liberofs_oci.h > @@ -19,6 +19,23 @@ struct erofs_inode; > struct CURL; > struct erofs_importer; > > +/** > + * struct ocierofs_layer_info > + * @digest: OCI content-addressable digest (e.g. "sha256:...") > + * @media_type: mediaType string from the manifest > + * @size: layer size in bytes from the manifest (0 if not available) > + * > + * This structure is exposed to callers so they can enumerate image layers, > + * decide which ones to fetch, and pass the digest back to download APIs. > + * Fields are heap-allocated NUL-terminated strings owned by the caller > + * once returned from public APIs; the caller must free them. > + */ > +struct ocierofs_layer_info { > + char *digest; > + char *media_type; > + u64 size; > +}; > + > /** > * struct erofs_oci_params - OCI configuration parameters > * @registry: registry hostname (e.g., "registry-1.docker.io") > @@ -88,6 +105,89 @@ int erofs_oci_params_set_string(char **field, const char *value); > */ > int ocierofs_build_trees(struct erofs_importer *importer, struct erofs_oci *oci); > > +/* > + * ocierofs_parse_options - Parse comma-separated OCI options string > + * @oci: OCI client structure to update > + * @options_str: comma-separated options string > + * > + * Parse OCI options string containing comma-separated key=value pairs. > + * Supported options include platform, layer, username, and password. > + * > + * Return: 0 on success, negative errno on failure > + */ > +int ocierofs_parse_options(struct erofs_oci *oci, char *options_str); Can we leave this functionality in `mkfs/main.c`. liberofs is not the place to keep option parser. > + > +/* > + * ocierofs_parse_ref - Parse OCI image reference string > + * @oci: OCI client structure to update > + * @ref_str: OCI image reference in various formats > + * > + * Parse OCI image reference which can be in formats: > + * - registry.example.com/namespace/repo:tag > + * - namespace/repo:tag (uses default registry) > + * - repo:tag (adds library/ prefix for Docker Hub) Is there some reference for this rule? > + * - repo (uses default tag "latest") > + * > + * Return: 0 on success, negative errno on failure > + */ > +int ocierofs_parse_ref(struct erofs_oci *oci, const char *ref_str); > + > +/* > + * ocierofs_prepare_layers - Prepare OCI layers for processing > + * @oci: OCI client structure with configured parameters > + * @auth_header: Pointer to store authentication header > + * @using_basic: Pointer to store basic auth flag > + * @manifest_digest: Pointer to store manifest digest > + * @layers: Pointer to store layers information > + * @layer_count: Pointer to store number of layers > + * @start_index: Pointer to store starting layer index > + * > + * Prepare authentication, get manifest digest and layers information > + * for OCI image processing. This function handles all the preparation > + * work needed before processing OCI layers. > + * > + * Return: 0 on success, negative errno on failure > + */ > +int ocierofs_prepare_layers(struct erofs_oci *oci, char **auth_header, > + bool *using_basic, char **manifest_digest, > + struct ocierofs_layer_info ***layers, > + int *layer_count, int *start_index); Could we have a way to wrap these arguments into a structure too? > + > +/** > + * ocierofs_free_layers_info - Free layer information array > + * @layers: array of layer information structures > + * @count: number of layers in the array > + * > + * Free all layer information structures and the array itself. > + * This function handles NULL pointers safely. > + */ > +void ocierofs_free_layers_info(struct ocierofs_layer_info **layers, int count); > + > +/** > + * ocierofs_prepare_auth - Prepare authentication for OCI requests > + * @oci: OCI client structure > + * @auth_header_out: pointer to store authentication header > + * @using_basic_auth: pointer to store basic auth flag > + * > + * Prepare authentication header for OCI registry requests. > + * This function handles both token-based and basic authentication. > + * > + * Return: 0 on success, negative errno on failure > + */ > +int ocierofs_prepare_auth(struct erofs_oci *oci, char **auth_header_out, > + bool *using_basic_auth); > + > +/** > + * ocierofs_curl_clear_auth - Clear basic authentication from CURL handle > + * @curl: CURL handle to clear authentication from > + * > + * Clear basic authentication credentials from a CURL handle. > + * This should be called after using basic authentication to clean up. > + * > + * Return: 0 on success, negative errno on failure > + */ > +int ocierofs_curl_clear_auth(struct CURL *curl); pass in `erofs_oci` instead? > + > #ifdef __cplusplus > } > #endif > diff --git a/lib/remotes/oci.c b/lib/remotes/oci.c > index 0fb8c1f..9774d8d 100644 > --- a/lib/remotes/oci.c > +++ b/lib/remotes/oci.c > @@ -42,7 +42,6 @@ struct erofs_oci_response { > }; > > struct erofs_oci_stream { > - struct erofs_tarfile tarfile; > const char *digest; > int blobfd; > }; > @@ -111,7 +110,7 @@ static int ocierofs_curl_setup_basic_auth(struct CURL *curl, const char *usernam > return 0; > } > > -static int ocierofs_curl_clear_auth(struct CURL *curl) > +int ocierofs_curl_clear_auth(struct CURL *curl) > { > curl_easy_setopt(curl, CURLOPT_USERPWD, NULL); > curl_easy_setopt(curl, CURLOPT_HTTPAUTH, CURLAUTH_NONE); > @@ -181,7 +180,7 @@ static int ocierofs_request_perform(struct erofs_oci *oci, > > ret = ocierofs_curl_setup_rq(oci->curl, req->url, > OCIEROFS_HTTP_GET, req->headers, > - ocierofs_write_callback, resp, > + ocierofs_write_callback, resp, > NULL, NULL); > if (ret) > return ret; > @@ -568,7 +567,7 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci, > const char *api_registry; > int ret = 0, len, i; > > - if (!registry || !repository || !tag || !platform) > + if (!registry || !repository || !tag) > return ERR_PTR(-EINVAL); > > api_registry = (!strcmp(registry, DOCKER_REGISTRY)) ? DOCKER_API_REGISTRY : registry; > @@ -581,8 +580,8 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci, > > req.headers = curl_slist_append(req.headers, > "Accept: " DOCKER_MEDIATYPE_MANIFEST_LIST "," > - OCI_MEDIATYPE_INDEX "," DOCKER_MEDIATYPE_MANIFEST_V1 "," > - DOCKER_MEDIATYPE_MANIFEST_V2); > + OCI_MEDIATYPE_INDEX "," OCI_MEDIATYPE_MANIFEST "," > + DOCKER_MEDIATYPE_MANIFEST_V1 "," DOCKER_MEDIATYPE_MANIFEST_V2); > > ret = ocierofs_request_perform(oci, &req, &resp); > if (ret) > @@ -663,7 +662,24 @@ out: > return ret ? ERR_PTR(ret) : digest; > } > > -static char **ocierofs_get_layers_info(struct erofs_oci *oci, > +void ocierofs_free_layers_info(struct ocierofs_layer_info **layers, int count) > +{ > + int i; > + > + if (!layers) > + return; > + > + for (i = 0; i < count; i++) { > + if (layers[i]) { > + free(layers[i]->digest); > + free(layers[i]->media_type); > + free(layers[i]); > + } > + } > + free(layers); > +} > + > +static struct ocierofs_layer_info **ocierofs_fetch_layers_info(struct erofs_oci *oci, > const char *registry, > const char *repository, > const char *digest, > @@ -672,10 +688,10 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci, > { > struct erofs_oci_request req = {}; > struct erofs_oci_response resp = {}; > - json_object *root, *layers, *layer, *digest_obj; > - char **layers_info = NULL; > + json_object *root, *layers, *layer, *digest_obj, *media_type_obj, *size_obj; > + struct ocierofs_layer_info **layers_info = NULL; > const char *api_registry; > - int ret, len, i, j; > + int ret, len, i; > > if (!registry || !repository || !digest || !layer_count) > return ERR_PTR(-EINVAL); > @@ -725,7 +741,7 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci, > goto out_json; > } > > - layers_info = calloc(len, sizeof(char *)); > + layers_info = calloc(len, sizeof(*layers_info)); > if (!layers_info) { > ret = -ENOMEM; > goto out_json; > @@ -740,11 +756,25 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci, > goto out_free; > } > > - layers_info[i] = strdup(json_object_get_string(digest_obj)); > + layers_info[i] = calloc(1, sizeof(**layers_info)); > if (!layers_info[i]) { > ret = -ENOMEM; > goto out_free; > } > + layers_info[i]->digest = strdup(json_object_get_string(digest_obj)); > + if (!layers_info[i]->digest) { > + ret = -ENOMEM; > + goto out_free; > + } > + if (json_object_object_get_ex(layer, "mediaType", &media_type_obj)) > + layers_info[i]->media_type = strdup(json_object_get_string(media_type_obj)); > + else > + layers_info[i]->media_type = NULL; > + > + if (json_object_object_get_ex(layer, "size", &size_obj)) > + layers_info[i]->size = json_object_get_int64(size_obj); > + else > + layers_info[i]->size = 0; > } > > *layer_count = len; > @@ -756,11 +786,7 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci, > return layers_info; > > out_free: > - if (layers_info) { > - for (j = 0; j < i; j++) > - free(layers_info[j]); > - } > - free(layers_info); > + ocierofs_free_layers_info(layers_info, i); > out_json: > json_object_put(root); > out: > @@ -771,8 +797,93 @@ out: > return ERR_PTR(ret); > } > > -static int ocierofs_extract_layer(struct erofs_oci *oci, struct erofs_importer *importer, > - const char *digest, const char *auth_header) > +/** > + * ocierofs_process_tar_stream - Process tar stream from file descriptor > + * @importer: EROFS importer structure > + * @fd: File descriptor containing tar data > + * > + * Initialize tar stream, parse all entries, and clean up resources. > + * > + * Return: 0 on success, negative errno on failure > + */ > +static int ocierofs_process_tar_stream(struct erofs_importer *importer, int fd) > +{ > + struct erofs_tarfile tarfile = {}; > + int ret; > + > + memset(&tarfile, 0, sizeof(tarfile)); struct erofs_tarfile tarfile = {}; already indicates memset(&tarfile, 0, sizeof(tarfile)); Thanks, Gao Xiang ^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH v2] erofs-utils: refactor OCI code for better modularity 2025-09-01 5:10 [PATCH v1] erofs-utils: refactor OCI code for better modularity ChengyuZhu6 2025-09-01 7:01 ` [PATCH v1-changed] " ChengyuZhu6 @ 2025-09-02 3:17 ` ChengyuZhu6 2025-09-02 3:29 ` [PATCH v2-changed] " ChengyuZhu6 ` (3 subsequent siblings) 5 siblings, 0 replies; 19+ messages in thread From: ChengyuZhu6 @ 2025-09-02 3:17 UTC (permalink / raw) To: linux-erofs; +Cc: xiang, hsiangkao, Chengyu Zhu From: Chengyu Zhu <hudsonzhu@tencent.com> Refactor OCI code to improve code organization and maintainability: - Add `struct ocierofs_layer_info` to encapsulate layer metadata - Extract authentication logic into `ocierofs_prepare_auth()` - Split layer processing into `ocierofs_prepare_layers()` - Move OCI parsing functions from `mkfs/main.c` to `lib/remotes/oci.c` - Add `ocierofs_process_tar_stream()` for separate tar processing - Improve error handling with `ocierofs_free_layers_info()` - Refactor `ocierofs_extract_layer()` to return file descriptor Signed-off-by: Chengyu Zhu <hudsonzhu@tencent.com> --- lib/liberofs_oci.h | 136 ++++++++++- lib/remotes/oci.c | 558 +++++++++++++++++++++++++++++++++------------ mkfs/main.c | 155 ++----------- 3 files changed, 572 insertions(+), 277 deletions(-) diff --git a/lib/liberofs_oci.h b/lib/liberofs_oci.h index 3a8108b..360725b 100644 --- a/lib/liberofs_oci.h +++ b/lib/liberofs_oci.h @@ -19,7 +19,24 @@ struct erofs_inode; struct CURL; struct erofs_importer; -/** +/* + * struct ocierofs_layer_info + * @digest: OCI content-addressable digest (e.g. "sha256:...") + * @media_type: mediaType string from the manifest + * @size: layer size in bytes from the manifest (0 if not available) + * + * This structure is exposed to callers so they can enumerate image layers, + * decide which ones to fetch, and pass the digest back to download APIs. + * Fields are heap-allocated NUL-terminated strings owned by the caller + * once returned from public APIs; the caller must free them. + */ +struct ocierofs_layer_info { + char *digest; + char *media_type; + u64 size; +}; + +/* * struct erofs_oci_params - OCI configuration parameters * @registry: registry hostname (e.g., "registry-1.docker.io") * @repository: image repository (e.g., "library/ubuntu") @@ -43,7 +60,28 @@ struct erofs_oci_params { int layer_index; }; -/** +/* + * struct ocierofs_image_context - OCI image preparation context + * @auth_header: Authentication header for OCI requests + * @using_basic: Flag indicating if basic auth is being used + * @manifest_digest: Manifest digest string + * @layers: Array of layer information structures + * @layer_count: Number of layers in the array + * @start_index: Starting layer index for processing + * + * This structure consolidates all the parameters needed for OCI image + * preparation, making the API cleaner and more maintainable. + */ +struct ocierofs_image_context { + char *auth_header; + bool using_basic; + char *manifest_digest; + struct ocierofs_layer_info **layers; + int layer_count; + int start_index; +}; + +/* * struct erofs_oci - Combined OCI client structure * @curl: CURL handle for HTTP requests * @params: OCI configuration parameters @@ -88,6 +126,100 @@ int erofs_oci_params_set_string(char **field, const char *value); */ int ocierofs_build_trees(struct erofs_importer *importer, struct erofs_oci *oci); +/* + * ocierofs_parse_ref - Parse OCI image reference string + * @oci: OCI client structure to update + * @ref_str: OCI image reference in various formats + * + * Parse OCI image reference which can be in formats: + * - registry.example.com/namespace/repo:tag + * - namespace/repo:tag (uses default registry) + * - repo:tag (adds library/ prefix for Docker Hub) + * - repo (uses default tag "latest") + * See: https://github.com/distribution/reference/blob/main/reference.go + * + * Return: 0 on success, negative errno on failure + */ +int ocierofs_parse_ref(struct erofs_oci *oci, const char *ref_str); + +/* + * ocierofs_prepare_layers - Prepare OCI layers for processing + * @oci: OCI client structure with configured parameters + * @auth_header: Pointer to store authentication header + * @using_basic: Pointer to store basic auth flag + * @manifest_digest: Pointer to store manifest digest + * @layers: Pointer to store layers information + * @layer_count: Pointer to store number of layers + * @start_index: Pointer to store starting layer index + * + * Prepare authentication, get manifest digest and layers information + * for OCI image processing. This function handles all the preparation + * work needed before processing OCI layers. + * + * Return: 0 on success, negative errno on failure + */ +int ocierofs_prepare_layers(struct erofs_oci *oci, char **auth_header, + bool *using_basic, char **manifest_digest, + struct ocierofs_layer_info ***layers, + int *layer_count, int *start_index); + +/* + * ocierofs_prepare_image_ctx - Prepare OCI image using context structure + * @oci: OCI client structure with configured parameters + * @ctx: Image context structure to populate + * + * Prepare authentication, get manifest digest and layers information + * for OCI image processing using a consolidated context structure. + * This is the preferred interface for new code. + * + * Return: 0 on success, negative errno on failure + */ +int ocierofs_prepare_image_ctx(struct erofs_oci *oci, struct ocierofs_image_context *ctx); + +/* + * ocierofs_free_layers_info - Free layer information array + * @layers: array of layer information structures + * @count: number of layers in the array + * + * Free all layer information structures and the array itself. + * This function handles NULL pointers safely. + */ +void ocierofs_free_layers_info(struct ocierofs_layer_info **layers, int count); + +/* + * ocierofs_free_image_ctx - Free image context and all associated resources + * @ctx: image context structure to free + * + * Free all resources associated with the image context, including + * layers info, manifest digest, and auth header. + */ +void ocierofs_free_image_ctx(struct ocierofs_image_context *ctx); + +/* + * ocierofs_prepare_auth - Prepare authentication for OCI requests + * @oci: OCI client structure + * @auth_header_out: pointer to store authentication header + * @using_basic_auth: pointer to store basic auth flag + * + * Prepare authentication header for OCI registry requests. + * This function handles both token-based and basic authentication. + * + * Return: 0 on success, negative errno on failure + */ +int ocierofs_prepare_auth(struct erofs_oci *oci, char **auth_header_out, + bool *using_basic_auth); + +/* + * ocierofs_curl_clear_auth - Clear basic authentication from OCI client + * @oci: OCI client structure to clear authentication from + * + * Clear basic authentication credentials from the OCI client's CURL handle. + * This should be called after using basic authentication to clean up. + * + * Return: 0 on success, negative errno on failure + */ +int ocierofs_curl_clear_auth(struct erofs_oci *oci); + #ifdef __cplusplus } #endif diff --git a/lib/remotes/oci.c b/lib/remotes/oci.c index 0fb8c1f..f78ad6b 100644 --- a/lib/remotes/oci.c +++ b/lib/remotes/oci.c @@ -42,7 +42,6 @@ struct erofs_oci_response { }; struct erofs_oci_stream { - struct erofs_tarfile tarfile; const char *digest; int blobfd; }; @@ -111,10 +110,10 @@ static int ocierofs_curl_setup_basic_auth(struct CURL *curl, const char *usernam return 0; } -static int ocierofs_curl_clear_auth(struct CURL *curl) +int ocierofs_curl_clear_auth(struct erofs_oci *oci) { - curl_easy_setopt(curl, CURLOPT_USERPWD, NULL); - curl_easy_setopt(curl, CURLOPT_HTTPAUTH, CURLAUTH_NONE); + curl_easy_setopt(oci->curl, CURLOPT_USERPWD, NULL); + curl_easy_setopt(oci->curl, CURLOPT_HTTPAUTH, CURLAUTH_NONE); return 0; } @@ -181,7 +180,7 @@ static int ocierofs_request_perform(struct erofs_oci *oci, ret = ocierofs_curl_setup_rq(oci->curl, req->url, OCIEROFS_HTTP_GET, req->headers, - ocierofs_write_callback, resp, + ocierofs_write_callback, resp, NULL, NULL); if (ret) return ret; @@ -377,7 +376,7 @@ static char *ocierofs_get_auth_token_with_url(struct erofs_oci *oci, } ret = ocierofs_request_perform(oci, &req, &resp); - ocierofs_curl_clear_auth(oci->curl); + ocierofs_curl_clear_auth(oci); if (ret) goto out_url; @@ -568,7 +567,7 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci, const char *api_registry; int ret = 0, len, i; - if (!registry || !repository || !tag || !platform) + if (!registry || !repository || !tag) return ERR_PTR(-EINVAL); api_registry = (!strcmp(registry, DOCKER_REGISTRY)) ? DOCKER_API_REGISTRY : registry; @@ -581,8 +580,8 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci, req.headers = curl_slist_append(req.headers, "Accept: " DOCKER_MEDIATYPE_MANIFEST_LIST "," - OCI_MEDIATYPE_INDEX "," DOCKER_MEDIATYPE_MANIFEST_V1 "," - DOCKER_MEDIATYPE_MANIFEST_V2); + OCI_MEDIATYPE_INDEX "," OCI_MEDIATYPE_MANIFEST "," + DOCKER_MEDIATYPE_MANIFEST_V1 "," DOCKER_MEDIATYPE_MANIFEST_V2); ret = ocierofs_request_perform(oci, &req, &resp); if (ret) @@ -663,7 +662,24 @@ out: return ret ? ERR_PTR(ret) : digest; } -static char **ocierofs_get_layers_info(struct erofs_oci *oci, +void ocierofs_free_layers_info(struct ocierofs_layer_info **layers, int count) +{ + int i; + + if (!layers) + return; + + for (i = 0; i < count; i++) { + if (layers[i]) { + free(layers[i]->digest); + free(layers[i]->media_type); + free(layers[i]); + } + } + free(layers); +} + +static struct ocierofs_layer_info **ocierofs_fetch_layers_info(struct erofs_oci *oci, const char *registry, const char *repository, const char *digest, @@ -672,10 +688,10 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci, { struct erofs_oci_request req = {}; struct erofs_oci_response resp = {}; - json_object *root, *layers, *layer, *digest_obj; - char **layers_info = NULL; + json_object *root, *layers, *layer, *digest_obj, *media_type_obj, *size_obj; + struct ocierofs_layer_info **layers_info = NULL; const char *api_registry; - int ret, len, i, j; + int ret, len, i; if (!registry || !repository || !digest || !layer_count) return ERR_PTR(-EINVAL); @@ -725,7 +741,7 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci, goto out_json; } - layers_info = calloc(len, sizeof(char *)); + layers_info = calloc(len, sizeof(*layers_info)); if (!layers_info) { ret = -ENOMEM; goto out_json; @@ -740,11 +756,25 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci, goto out_free; } - layers_info[i] = strdup(json_object_get_string(digest_obj)); + layers_info[i] = calloc(1, sizeof(**layers_info)); if (!layers_info[i]) { ret = -ENOMEM; goto out_free; } + layers_info[i]->digest = strdup(json_object_get_string(digest_obj)); + if (!layers_info[i]->digest) { + ret = -ENOMEM; + goto out_free; + } + if (json_object_object_get_ex(layer, "mediaType", &media_type_obj)) + layers_info[i]->media_type = strdup(json_object_get_string(media_type_obj)); + else + layers_info[i]->media_type = NULL; + + if (json_object_object_get_ex(layer, "size", &size_obj)) + layers_info[i]->size = json_object_get_int64(size_obj); + else + layers_info[i]->size = 0; } *layer_count = len; @@ -756,11 +786,7 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci, return layers_info; out_free: - if (layers_info) { - for (j = 0; j < i; j++) - free(layers_info[j]); - } - free(layers_info); + ocierofs_free_layers_info(layers_info, i); out_json: json_object_put(root); out: @@ -771,8 +797,92 @@ out: return ERR_PTR(ret); } -static int ocierofs_extract_layer(struct erofs_oci *oci, struct erofs_importer *importer, - const char *digest, const char *auth_header) +/** + * ocierofs_process_tar_stream - Process tar stream from file descriptor + * @importer: EROFS importer structure + * @fd: File descriptor containing tar data + * + * Initialize tar stream, parse all entries, and clean up resources. + * + * Return: 0 on success, negative errno on failure + */ +static int ocierofs_process_tar_stream(struct erofs_importer *importer, int fd) +{ + struct erofs_tarfile tarfile = {}; + int ret; + + init_list_head(&tarfile.global.xattrs); + + ret = erofs_iostream_open(&tarfile.ios, fd, EROFS_IOS_DECODER_GZIP); + if (ret) { + erofs_err("failed to initialize tar stream: %s", + erofs_strerror(ret)); + return ret; + } + + do { + ret = tarerofs_parse_tar(importer, &tarfile); + /* Continue parsing until end of archive */ + } while (!ret); + erofs_iostream_close(&tarfile.ios); + + if (ret < 0 && ret != -ENODATA) { + erofs_err("failed to process tar stream: %s", + erofs_strerror(ret)); + return ret; + } + + return 0; +} + +int ocierofs_prepare_auth(struct erofs_oci *oci, char **auth_header_out, + bool *using_basic_auth) +{ + char *auth_header = NULL; + int ret = 0; + + if (using_basic_auth) + *using_basic_auth = false; + if (auth_header_out) + *auth_header_out = NULL; + + if (oci->params.username && oci->params.password && + oci->params.username[0] && oci->params.password[0]) { + auth_header = ocierofs_get_auth_token(oci, + oci->params.registry, + oci->params.repository, + oci->params.username, + oci->params.password); + if (IS_ERR(auth_header)) { + auth_header = NULL; + ret = ocierofs_curl_setup_basic_auth(oci->curl, + oci->params.username, + oci->params.password); + if (ret) + return ret; + if (using_basic_auth) + *using_basic_auth = true; + } + } else { + auth_header = ocierofs_get_auth_token(oci, + oci->params.registry, + oci->params.repository, + NULL, NULL); + if (IS_ERR(auth_header)) + auth_header = NULL; + } + + if (auth_header_out) + *auth_header_out = auth_header; + else + free(auth_header); + return 0; +} + +static int ocierofs_download_blob_to_fd(struct erofs_oci *oci, + const char *digest, + const char *auth_header, + int outfd) { struct erofs_oci_request req = {}; struct erofs_oci_stream stream = {}; @@ -782,20 +892,14 @@ static int ocierofs_extract_layer(struct erofs_oci *oci, struct erofs_importer * stream = (struct erofs_oci_stream) { .digest = digest, - .blobfd = erofs_tmpfile(), + .blobfd = outfd, }; - if (stream.blobfd < 0) { - erofs_err("failed to create temporary file for %s", digest); - return -errno; - } api_registry = (!strcmp(oci->params.registry, DOCKER_REGISTRY)) ? DOCKER_API_REGISTRY : oci->params.registry; if (asprintf(&req.url, "https://%s/v2/%s/blobs/%s", - api_registry, oci->params.repository, digest) == -1) { - ret = -ENOMEM; - goto out; - } + api_registry, oci->params.repository, digest) == -1) + return -ENOMEM; if (auth_header && strstr(auth_header, "Bearer")) req.headers = curl_slist_append(req.headers, auth_header); @@ -822,6 +926,32 @@ static int ocierofs_extract_layer(struct erofs_oci *oci, struct erofs_importer * ret = -EIO; goto out; } + ret = 0; +out: + if (req.headers) + curl_slist_free_all(req.headers); + free(req.url); + return ret; +} + +static int ocierofs_extract_layer(struct erofs_oci *oci, + const char *digest, const char *auth_header) +{ + struct erofs_oci_stream stream = {}; + int ret; + + stream = (struct erofs_oci_stream) { + .digest = digest, + .blobfd = erofs_tmpfile(), + }; + if (stream.blobfd < 0) { + erofs_err("failed to create temporary file for %s", digest); + return -errno; + } + + ret = ocierofs_download_blob_to_fd(oci, digest, auth_header, stream.blobfd); + if (ret) + goto out; if (lseek(stream.blobfd, 0, SEEK_SET) < 0) { erofs_err("failed to seek to beginning of temp file: %s", @@ -830,162 +960,181 @@ static int ocierofs_extract_layer(struct erofs_oci *oci, struct erofs_importer * goto out; } - memset(&stream.tarfile, 0, sizeof(stream.tarfile)); - init_list_head(&stream.tarfile.global.xattrs); + return stream.blobfd; - ret = erofs_iostream_open(&stream.tarfile.ios, stream.blobfd, - EROFS_IOS_DECODER_GZIP); - if (ret) { - erofs_err("failed to initialize tar stream: %s", +out: + if (stream.blobfd >= 0) + close(stream.blobfd); + return ret; +} + +int ocierofs_prepare_layers(struct erofs_oci *oci, char **auth_header, + bool *using_basic, char **manifest_digest, + struct ocierofs_layer_info ***layers, + int *layer_count, int *start_index) +{ + int ret; + + ret = ocierofs_prepare_auth(oci, auth_header, using_basic); + if (ret) + return ret; + + *manifest_digest = ocierofs_get_manifest_digest(oci, oci->params.registry, + oci->params.repository, + oci->params.tag, + oci->params.platform, + *auth_header); + if (IS_ERR(*manifest_digest)) { + ret = PTR_ERR(*manifest_digest); + erofs_err("failed to get manifest digest: %s", erofs_strerror(ret)); - goto out; + goto out_auth; } - do { - ret = tarerofs_parse_tar(importer, &stream.tarfile); - /* Continue parsing until end of archive */ - } while (!ret); - erofs_iostream_close(&stream.tarfile.ios); + *layers = ocierofs_fetch_layers_info(oci, oci->params.registry, + oci->params.repository, + *manifest_digest, *auth_header, + layer_count); + if (IS_ERR(*layers)) { + ret = PTR_ERR(*layers); + erofs_err("failed to get image layers: %s", erofs_strerror(ret)); + goto out_manifest; + } - if (ret < 0 && ret != -ENODATA) { - erofs_err("failed to process tar stream: %s", - erofs_strerror(ret)); - goto out; + if (oci->params.layer_index >= 0) { + if (oci->params.layer_index >= *layer_count) { + erofs_err("layer index %d exceeds available layers (%d)", + oci->params.layer_index, *layer_count); + ret = -EINVAL; + goto out_layers; + } + *layer_count = 1; + *start_index = oci->params.layer_index; + } else { + *start_index = 0; } - ret = 0; -out: - if (stream.blobfd >= 0) - close(stream.blobfd); - if (req.headers) - curl_slist_free_all(req.headers); - free(req.url); + return 0; + +out_layers: + free(*layers); + *layers = NULL; +out_manifest: + free(*manifest_digest); + *manifest_digest = NULL; +out_auth: + free(*auth_header); + *auth_header = NULL; + if (*using_basic) + ocierofs_curl_clear_auth(oci); return ret; } -/** - * ocierofs_build_trees - Build file trees from OCI container image layers - * @importer: EROFS importer structure - * @oci: OCI client structure with configured parameters - * - * Extract and build file system trees from all layers of an OCI container - * image. - * - * Return: 0 on success, negative errno on failure - */ -int ocierofs_build_trees(struct erofs_importer *importer, struct erofs_oci *oci) +int ocierofs_prepare_image_ctx(struct erofs_oci *oci, struct ocierofs_image_context *ctx) { - char *auth_header = NULL; - char *manifest_digest = NULL; - char **layers = NULL; - int layer_count = 0; - int ret, i; + int ret; - if (!importer || !oci) + if (!oci || !ctx) return -EINVAL; - if (oci->params.username && oci->params.password && - oci->params.username[0] && oci->params.password[0]) { - auth_header = ocierofs_get_auth_token(oci, - oci->params.registry, - oci->params.repository, - oci->params.username, - oci->params.password); - if (IS_ERR(auth_header)) { - auth_header = NULL; - ret = ocierofs_curl_setup_basic_auth(oci->curl, - oci->params.username, - oci->params.password); - if (ret) - goto out; - } - } else { - auth_header = ocierofs_get_auth_token(oci, - oci->params.registry, - oci->params.repository, - NULL, NULL); - if (IS_ERR(auth_header)) - auth_header = NULL; - } + *ctx = (struct ocierofs_image_context){}; - manifest_digest = ocierofs_get_manifest_digest(oci, oci->params.registry, - oci->params.repository, - oci->params.tag, - oci->params.platform, - auth_header); - if (IS_ERR(manifest_digest)) { - ret = PTR_ERR(manifest_digest); + ret = ocierofs_prepare_auth(oci, &ctx->auth_header, &ctx->using_basic); + if (ret) + return ret; + + ctx->manifest_digest = ocierofs_get_manifest_digest(oci, oci->params.registry, + oci->params.repository, + oci->params.tag, + oci->params.platform, + ctx->auth_header); + if (IS_ERR(ctx->manifest_digest)) { + ret = PTR_ERR(ctx->manifest_digest); erofs_err("failed to get manifest digest: %s", erofs_strerror(ret)); goto out_auth; } - layers = ocierofs_get_layers_info(oci, oci->params.registry, - oci->params.repository, - manifest_digest, auth_header, - &layer_count); - if (IS_ERR(layers)) { - ret = PTR_ERR(layers); + ctx->layers = ocierofs_fetch_layers_info(oci, oci->params.registry, + oci->params.repository, + ctx->manifest_digest, ctx->auth_header, + &ctx->layer_count); + if (IS_ERR(ctx->layers)) { + ret = PTR_ERR(ctx->layers); erofs_err("failed to get image layers: %s", erofs_strerror(ret)); goto out_manifest; } if (oci->params.layer_index >= 0) { - if (oci->params.layer_index >= layer_count) { + if (oci->params.layer_index >= ctx->layer_count) { erofs_err("layer index %d exceeds available layers (%d)", - oci->params.layer_index, layer_count); + oci->params.layer_index, ctx->layer_count); ret = -EINVAL; goto out_layers; } - layer_count = 1; - i = oci->params.layer_index; + ctx->layer_count = 1; + ctx->start_index = oci->params.layer_index; } else { - i = 0; + ctx->start_index = 0; } - while (i < layer_count) { - char *trimmed = erofs_trim_for_progressinfo(layers[i], + return 0; + +out_layers: + free(ctx->layers); + ctx->layers = NULL; +out_manifest: + free(ctx->manifest_digest); + ctx->manifest_digest = NULL; +out_auth: + free(ctx->auth_header); + ctx->auth_header = NULL; + if (ctx->using_basic) + ocierofs_curl_clear_auth(oci); + return ret; +} + +int ocierofs_build_trees(struct erofs_importer *importer, struct erofs_oci *oci) +{ + struct ocierofs_image_context image_ctx = {}; + int ret, i; + + if (!importer || !oci) + return -EINVAL; + + ret = ocierofs_prepare_image_ctx(oci, &image_ctx); + if (ret) + return ret; + + i = image_ctx.start_index; + while (i < image_ctx.layer_count) { + char *trimmed = erofs_trim_for_progressinfo(image_ctx.layers[i]->digest, sizeof("Extracting layer ...") - 1); erofs_update_progressinfo("Extracting layer %d: %s ...", i, trimmed); free(trimmed); - ret = ocierofs_extract_layer(oci, importer, layers[i], - auth_header); - if (ret) { + int fd = ocierofs_extract_layer(oci, image_ctx.layers[i]->digest, + image_ctx.auth_header); + if (fd < 0) { erofs_err("failed to extract layer %d: %s", i, + erofs_strerror(fd)); + break; + } + ret = ocierofs_process_tar_stream(importer, fd); + close(fd); + if (ret) { + erofs_err("failed to process tar stream for layer %d: %s", i, erofs_strerror(ret)); break; } i++; } -out_layers: - for (i = 0; i < layer_count; i++) - free(layers[i]); - free(layers); -out_manifest: - free(manifest_digest); -out_auth: - free(auth_header); - if (oci->params.username && oci->params.password && - oci->params.username[0] && oci->params.password[0] && - !auth_header) { - ocierofs_curl_clear_auth(oci->curl); - } -out: + ocierofs_free_image_ctx(&image_ctx); + return ret; } -/** - * ocierofs_init - Initialize OCI client with default parameters - * @oci: OCI client structure to initialize - * - * Initialize OCI client structure, set up CURL handle, and configure - * default parameters including platform (linux/amd64), registry - * (registry-1.docker.io), and tag (latest). - * - * Return: 0 on success, negative errno on failure - */ int ocierofs_init(struct erofs_oci *oci) { if (!oci) @@ -1066,3 +1215,130 @@ int erofs_oci_params_set_string(char **field, const char *value) *field = new_value; return 0; } + +int ocierofs_parse_ref(struct erofs_oci *oci, const char *ref_str) +{ + char *slash, *colon, *dot; + const char *repo_part; + size_t len; + + if (!oci || !ref_str) + return -EINVAL; + + slash = strchr(ref_str, '/'); + if (slash) { + dot = strchr(ref_str, '.'); + if (dot && dot < slash) { + char *registry_str; + + len = slash - ref_str; + registry_str = strndup(ref_str, len); + + if (!registry_str) { + erofs_err("failed to allocate memory for registry"); + return -ENOMEM; + } + if (erofs_oci_params_set_string(&oci->params.registry, + registry_str)) { + free(registry_str); + erofs_err("failed to set registry"); + return -ENOMEM; + } + free(registry_str); + repo_part = slash + 1; + } else { + repo_part = ref_str; + } + } else { + repo_part = ref_str; + } + + colon = strchr(repo_part, ':'); + if (colon) { + char *repo_str; + + len = colon - repo_part; + repo_str = strndup(repo_part, len); + + if (!repo_str) { + erofs_err("failed to allocate memory for repository"); + return -ENOMEM; + } + + if (!strchr(repo_str, '/') && + (!strcmp(oci->params.registry, DOCKER_API_REGISTRY) || + !strcmp(oci->params.registry, DOCKER_REGISTRY))) { + char *full_repo; + + if (asprintf(&full_repo, "library/%s", repo_str) == -1) { + free(repo_str); + erofs_err("failed to allocate memory for full repository name"); + return -ENOMEM; + } + free(repo_str); + repo_str = full_repo; + } + + if (erofs_oci_params_set_string(&oci->params.repository, + repo_str)) { + free(repo_str); + erofs_err("failed to set repository"); + return -ENOMEM; + } + free(repo_str); + + if (erofs_oci_params_set_string(&oci->params.tag, + colon + 1)) { + erofs_err("failed to set tag"); + return -ENOMEM; + } + } else { + char *repo_str = strdup(repo_part); + + if (!repo_str) { + erofs_err("failed to allocate memory for repository"); + return -ENOMEM; + } + + if (!strchr(repo_str, '/') && + (!strcmp(oci->params.registry, DOCKER_API_REGISTRY) || + !strcmp(oci->params.registry, DOCKER_REGISTRY))) { + char *full_repo; + + if (asprintf(&full_repo, "library/%s", repo_str) == -1) { + free(repo_str); + erofs_err("failed to allocate memory for full repository name"); + return -ENOMEM; + } + free(repo_str); + repo_str = full_repo; + } + + if (erofs_oci_params_set_string(&oci->params.repository, + repo_str)) { + free(repo_str); + erofs_err("failed to set repository"); + return -ENOMEM; + } + free(repo_str); + } + + return 0; +} + +void ocierofs_free_image_ctx(struct ocierofs_image_context *ctx) +{ + if (!ctx) + return; + + ocierofs_free_layers_info(ctx->layers, ctx->layer_count); + free(ctx->manifest_digest); + free(ctx->auth_header); + + ctx->layers = NULL; + ctx->manifest_digest = NULL; + ctx->auth_header = NULL; + ctx->layer_count = 0; + ctx->start_index = 0; + ctx->using_basic = false; +} diff --git a/mkfs/main.c b/mkfs/main.c index bc895f1..24d384d 100644 --- a/mkfs/main.c +++ b/mkfs/main.c @@ -689,18 +689,24 @@ static int mkfs_parse_s3_cfg(char *cfg_str) #endif #ifdef OCIEROFS_ENABLED - - -/** - * mkfs_parse_oci_options - Parse comma-separated OCI options string +/* + * mkfs_parse_oci_options - Parse OCI options for mkfs tool + * @oci: OCI client structure to update * @options_str: comma-separated options string * * Parse OCI options string containing comma-separated key=value pairs. - * Supported options include platform, layer, username, and password. + * This function is specific to the mkfs tool and can be enhanced + * independently of other tools. + * + * Supported options include: + * - platform=<os/arch>: target platform (e.g., "linux/amd64") + * - layer=<index>: specific layer to extract (0-based index) + * - username=<user>: authentication username + * - password=<pass>: authentication password * * Return: 0 on success, negative errno on failure */ -static int mkfs_parse_oci_options(char *options_str) +static int mkfs_parse_oci_options(struct erofs_oci *oci, char *options_str) { char *opt, *q, *p; int ret; @@ -717,7 +723,7 @@ static int mkfs_parse_oci_options(char *options_str) p = strstr(opt, "platform="); if (p) { p += strlen("platform="); - ret = erofs_oci_params_set_string(&ocicfg.params.platform, p); + ret = erofs_oci_params_set_string(&oci->params.platform, p); if (ret) { erofs_err("failed to set platform"); return ret; @@ -726,17 +732,17 @@ static int mkfs_parse_oci_options(char *options_str) p = strstr(opt, "layer="); if (p) { p += strlen("layer="); - ocicfg.params.layer_index = atoi(p); - if (ocicfg.params.layer_index < 0) { + oci->params.layer_index = atoi(p); + if (oci->params.layer_index < 0) { erofs_err("invalid layer index %d", - ocicfg.params.layer_index); + oci->params.layer_index); return -EINVAL; } } else { p = strstr(opt, "username="); if (p) { p += strlen("username="); - ret = erofs_oci_params_set_string(&ocicfg.params.username, p); + ret = erofs_oci_params_set_string(&oci->params.username, p); if (ret) { erofs_err("failed to set username"); return ret; @@ -745,13 +751,13 @@ static int mkfs_parse_oci_options(char *options_str) p = strstr(opt, "password="); if (p) { p += strlen("password="); - ret = erofs_oci_params_set_string(&ocicfg.params.password, p); + ret = erofs_oci_params_set_string(&oci->params.password, p); if (ret) { erofs_err("failed to set password"); return ret; } } else { - erofs_err("invalid --oci value %s", opt); + erofs_err("mkfs: invalid --oci value %s", opt); return -EINVAL; } } @@ -763,125 +769,6 @@ static int mkfs_parse_oci_options(char *options_str) return 0; } - -/** - * mkfs_parse_oci_ref - Parse OCI image reference string - * @ref_str: OCI image reference in various formats - * - * Parse OCI image reference which can be in formats: - * - registry.example.com/namespace/repo:tag - * - namespace/repo:tag (uses default registry) - * - repo:tag (adds library/ prefix for Docker Hub) - * - repo (uses default tag "latest") - * - * Return: 0 on success, negative errno on failure - */ -static int mkfs_parse_oci_ref(const char *ref_str) -{ - char *slash, *colon, *dot; - const char *repo_part; - size_t len; - - slash = strchr(ref_str, '/'); - if (slash) { - dot = strchr(ref_str, '.'); - if (dot && dot < slash) { - char *registry_str; - - len = slash - ref_str; - registry_str = strndup(ref_str, len); - - if (!registry_str) { - erofs_err("failed to allocate memory for registry"); - return -ENOMEM; - } - if (erofs_oci_params_set_string(&ocicfg.params.registry, - registry_str)) { - free(registry_str); - erofs_err("failed to set registry"); - return -ENOMEM; - } - free(registry_str); - repo_part = slash + 1; - } else { - repo_part = ref_str; - } - } else { - repo_part = ref_str; - } - - colon = strchr(repo_part, ':'); - if (colon) { - char *repo_str; - - len = colon - repo_part; - repo_str = strndup(repo_part, len); - - if (!repo_str) { - erofs_err("failed to allocate memory for repository"); - return -ENOMEM; - } - - if (!strchr(repo_str, '/') && - (!strcmp(ocicfg.params.registry, DOCKER_API_REGISTRY) || - !strcmp(ocicfg.params.registry, DOCKER_REGISTRY))) { - char *full_repo; - - if (asprintf(&full_repo, "library/%s", repo_str) == -1) { - free(repo_str); - erofs_err("failed to allocate memory for full repository name"); - return -ENOMEM; - } - free(repo_str); - repo_str = full_repo; - } - - if (erofs_oci_params_set_string(&ocicfg.params.repository, - repo_str)) { - free(repo_str); - erofs_err("failed to set repository"); - return -ENOMEM; - } - free(repo_str); - - if (erofs_oci_params_set_string(&ocicfg.params.tag, - colon + 1)) { - erofs_err("failed to set tag"); - return -ENOMEM; - } - } else { - char *repo_str = strdup(repo_part); - - if (!repo_str) { - erofs_err("failed to allocate memory for repository"); - return -ENOMEM; - } - - if (!strchr(repo_str, '/') && - (!strcmp(ocicfg.params.registry, DOCKER_API_REGISTRY) || - !strcmp(ocicfg.params.registry, DOCKER_REGISTRY))) { - char *full_repo; - - if (asprintf(&full_repo, "library/%s", repo_str) == -1) { - free(repo_str); - erofs_err("failed to allocate memory for full repository name"); - return -ENOMEM; - } - free(repo_str); - repo_str = full_repo; - } - - if (erofs_oci_params_set_string(&ocicfg.params.repository, - repo_str)) { - free(repo_str); - erofs_err("failed to set repository"); - return -ENOMEM; - } - free(repo_str); - } - - return 0; -} #endif static int mkfs_parse_one_compress_alg(char *alg, @@ -1955,10 +1842,10 @@ int main(int argc, char **argv) if (err) goto exit; - err = mkfs_parse_oci_options(mkfs_oci_options); + err = mkfs_parse_oci_options(&ocicfg, mkfs_oci_options); if (err) goto exit; - err = mkfs_parse_oci_ref(cfg.c_src_path); + err = ocierofs_parse_ref(&ocicfg, cfg.c_src_path); if (err) goto exit; -- 2.51.0 ^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH v2-changed] erofs-utils: refactor OCI code for better modularity 2025-09-01 5:10 [PATCH v1] erofs-utils: refactor OCI code for better modularity ChengyuZhu6 2025-09-01 7:01 ` [PATCH v1-changed] " ChengyuZhu6 2025-09-02 3:17 ` [PATCH v2] " ChengyuZhu6 @ 2025-09-02 3:29 ` ChengyuZhu6 2025-09-03 8:29 ` [PATCH v3 0/2] erofs-utils: refactor OCI and add NBD-backed OCI image mounting ChengyuZhu6 ` (2 subsequent siblings) 5 siblings, 0 replies; 19+ messages in thread From: ChengyuZhu6 @ 2025-09-02 3:29 UTC (permalink / raw) To: linux-erofs; +Cc: xiang, hsiangkao, Chengyu Zhu From: Chengyu Zhu <hudsonzhu@tencent.com> Refactor OCI code to improve code organization and maintainability: - Add `struct ocierofs_layer_info` to encapsulate layer metadata - Extract authentication logic into `ocierofs_prepare_auth()` - Split layer processing into `ocierofs_prepare_layers()` - Move OCI parsing functions from `mkfs/main.c` to `lib/remotes/oci.c` - Add `ocierofs_process_tar_stream()` for separate tar processing - Improve error handling with `ocierofs_free_layers_info()` - Refactor `ocierofs_extract_layer()` to return file descriptor Signed-off-by: Chengyu Zhu <hudsonzhu@tencent.com> --- lib/liberofs_oci.h | 115 ++++++++++- lib/remotes/oci.c | 501 ++++++++++++++++++++++++++++++++------------- mkfs/main.c | 155 ++------------ 3 files changed, 489 insertions(+), 282 deletions(-) diff --git a/lib/liberofs_oci.h b/lib/liberofs_oci.h index 3a8108b..34d531c 100644 --- a/lib/liberofs_oci.h +++ b/lib/liberofs_oci.h @@ -19,7 +19,24 @@ struct erofs_inode; struct CURL; struct erofs_importer; -/** +/* + * struct ocierofs_layer_info + * @digest: OCI content-addressable digest (e.g. "sha256:...") + * @media_type: mediaType string from the manifest + * @size: layer size in bytes from the manifest (0 if not available) + * + * This structure is exposed to callers so they can enumerate image layers, + * decide which ones to fetch, and pass the digest back to download APIs. + * Fields are heap-allocated NUL-terminated strings owned by the caller + * once returned from public APIs; the caller must free them. + */ +struct ocierofs_layer_info { + char *digest; + char *media_type; + u64 size; +}; + +/* * struct erofs_oci_params - OCI configuration parameters * @registry: registry hostname (e.g., "registry-1.docker.io") * @repository: image repository (e.g., "library/ubuntu") @@ -43,7 +60,28 @@ struct erofs_oci_params { int layer_index; }; -/** +/* + * struct ocierofs_image_context - OCI image preparation context + * @auth_header: Authentication header for OCI requests + * @using_basic: Flag indicating if basic auth is being used + * @manifest_digest: Manifest digest string + * @layers: Array of layer information structures + * @layer_count: Number of layers in the array + * @start_index: Starting layer index for processing + * + * This structure consolidates all the parameters needed for OCI image + * preparation, making the API cleaner and more maintainable. + */ +struct ocierofs_image_context { + char *auth_header; + bool using_basic; + char *manifest_digest; + struct ocierofs_layer_info **layers; + int layer_count; + int start_index; +}; + +/* * struct erofs_oci - Combined OCI client structure * @curl: CURL handle for HTTP requests * @params: OCI configuration parameters @@ -88,6 +126,79 @@ int erofs_oci_params_set_string(char **field, const char *value); */ int ocierofs_build_trees(struct erofs_importer *importer, struct erofs_oci *oci); +/* + * ocierofs_parse_ref - Parse OCI image reference string + * @oci: OCI client structure to update + * @ref_str: OCI image reference in various formats + * + * Parse OCI image reference which can be in formats: + * - registry.example.com/namespace/repo:tag + * - namespace/repo:tag (uses default registry) + * - repo:tag (adds library/ prefix for Docker Hub) + * - repo (uses default tag "latest") + * See: https://github.com/distribution/reference/blob/main/reference.go + * + * Return: 0 on success, negative errno on failure + */ +int ocierofs_parse_ref(struct erofs_oci *oci, const char *ref_str); + +/* + * ocierofs_prepare_layers - Prepare OCI image + * @oci: OCI client structure with configured parameters + * @ctx: Image context structure to populate + * + * Prepare authentication, get manifest digest and layers information + * for OCI image processing using a consolidated context structure. + * This is the preferred interface for new code. + * + * Return: 0 on success, negative errno on failure + */ +int ocierofs_prepare_layers(struct erofs_oci *oci, struct ocierofs_image_context *ctx); + +/* + * ocierofs_free_layers_info - Free layer information array + * @layers: array of layer information structures + * @count: number of layers in the array + * + * Free all layer information structures and the array itself. + * This function handles NULL pointers safely. + */ +void ocierofs_free_layers_info(struct ocierofs_layer_info **layers, int count); + +/* + * ocierofs_free_image_ctx - Free image context and all associated resources + * @ctx: image context structure to free + * + * Free all resources associated with the image context, including + * layers info, manifest digest, and auth header. + */ +void ocierofs_free_image_ctx(struct ocierofs_image_context *ctx); + +/* + * ocierofs_prepare_auth - Prepare authentication for OCI requests + * @oci: OCI client structure + * @auth_header_out: pointer to store authentication header + * @using_basic_auth: pointer to store basic auth flag + * + * Prepare authentication header for OCI registry requests. + * This function handles both token-based and basic authentication. + * + * Return: 0 on success, negative errno on failure + */ +int ocierofs_prepare_auth(struct erofs_oci *oci, char **auth_header_out, + bool *using_basic_auth); + +/* + * ocierofs_curl_clear_auth - Clear basic authentication from OCI client + * @oci: OCI client structure to clear authentication from + * + * Clear basic authentication credentials from the OCI client's CURL handle. + * This should be called after using basic authentication to clean up. + * + * Return: 0 on success, negative errno on failure + */ +int ocierofs_curl_clear_auth(struct erofs_oci *oci); + #ifdef __cplusplus } #endif diff --git a/lib/remotes/oci.c b/lib/remotes/oci.c index 0fb8c1f..9fc6bf2 100644 --- a/lib/remotes/oci.c +++ b/lib/remotes/oci.c @@ -42,7 +42,6 @@ struct erofs_oci_response { }; struct erofs_oci_stream { - struct erofs_tarfile tarfile; const char *digest; int blobfd; }; @@ -111,10 +110,10 @@ static int ocierofs_curl_setup_basic_auth(struct CURL *curl, const char *usernam return 0; } -static int ocierofs_curl_clear_auth(struct CURL *curl) +int ocierofs_curl_clear_auth(struct erofs_oci *oci) { - curl_easy_setopt(curl, CURLOPT_USERPWD, NULL); - curl_easy_setopt(curl, CURLOPT_HTTPAUTH, CURLAUTH_NONE); + curl_easy_setopt(oci->curl, CURLOPT_USERPWD, NULL); + curl_easy_setopt(oci->curl, CURLOPT_HTTPAUTH, CURLAUTH_NONE); return 0; } @@ -181,7 +180,7 @@ static int ocierofs_request_perform(struct erofs_oci *oci, ret = ocierofs_curl_setup_rq(oci->curl, req->url, OCIEROFS_HTTP_GET, req->headers, - ocierofs_write_callback, resp, + ocierofs_write_callback, resp, NULL, NULL); if (ret) return ret; @@ -377,7 +376,7 @@ static char *ocierofs_get_auth_token_with_url(struct erofs_oci *oci, } ret = ocierofs_request_perform(oci, &req, &resp); - ocierofs_curl_clear_auth(oci->curl); + ocierofs_curl_clear_auth(oci); if (ret) goto out_url; @@ -568,7 +567,7 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci, const char *api_registry; int ret = 0, len, i; - if (!registry || !repository || !tag || !platform) + if (!registry || !repository || !tag) return ERR_PTR(-EINVAL); api_registry = (!strcmp(registry, DOCKER_REGISTRY)) ? DOCKER_API_REGISTRY : registry; @@ -581,8 +580,8 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci, req.headers = curl_slist_append(req.headers, "Accept: " DOCKER_MEDIATYPE_MANIFEST_LIST "," - OCI_MEDIATYPE_INDEX "," DOCKER_MEDIATYPE_MANIFEST_V1 "," - DOCKER_MEDIATYPE_MANIFEST_V2); + OCI_MEDIATYPE_INDEX "," OCI_MEDIATYPE_MANIFEST "," + DOCKER_MEDIATYPE_MANIFEST_V1 "," DOCKER_MEDIATYPE_MANIFEST_V2); ret = ocierofs_request_perform(oci, &req, &resp); if (ret) @@ -663,7 +662,24 @@ out: return ret ? ERR_PTR(ret) : digest; } -static char **ocierofs_get_layers_info(struct erofs_oci *oci, +void ocierofs_free_layers_info(struct ocierofs_layer_info **layers, int count) +{ + int i; + + if (!layers) + return; + + for (i = 0; i < count; i++) { + if (layers[i]) { + free(layers[i]->digest); + free(layers[i]->media_type); + free(layers[i]); + } + } + free(layers); +} + +static struct ocierofs_layer_info **ocierofs_fetch_layers_info(struct erofs_oci *oci, const char *registry, const char *repository, const char *digest, @@ -672,10 +688,10 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci, { struct erofs_oci_request req = {}; struct erofs_oci_response resp = {}; - json_object *root, *layers, *layer, *digest_obj; - char **layers_info = NULL; + json_object *root, *layers, *layer, *digest_obj, *media_type_obj, *size_obj; + struct ocierofs_layer_info **layers_info = NULL; const char *api_registry; - int ret, len, i, j; + int ret, len, i; if (!registry || !repository || !digest || !layer_count) return ERR_PTR(-EINVAL); @@ -725,7 +741,7 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci, goto out_json; } - layers_info = calloc(len, sizeof(char *)); + layers_info = calloc(len, sizeof(*layers_info)); if (!layers_info) { ret = -ENOMEM; goto out_json; @@ -740,11 +756,25 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci, goto out_free; } - layers_info[i] = strdup(json_object_get_string(digest_obj)); + layers_info[i] = calloc(1, sizeof(**layers_info)); if (!layers_info[i]) { ret = -ENOMEM; goto out_free; } + layers_info[i]->digest = strdup(json_object_get_string(digest_obj)); + if (!layers_info[i]->digest) { + ret = -ENOMEM; + goto out_free; + } + if (json_object_object_get_ex(layer, "mediaType", &media_type_obj)) + layers_info[i]->media_type = strdup(json_object_get_string(media_type_obj)); + else + layers_info[i]->media_type = NULL; + + if (json_object_object_get_ex(layer, "size", &size_obj)) + layers_info[i]->size = json_object_get_int64(size_obj); + else + layers_info[i]->size = 0; } *layer_count = len; @@ -756,11 +786,7 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci, return layers_info; out_free: - if (layers_info) { - for (j = 0; j < i; j++) - free(layers_info[j]); - } - free(layers_info); + ocierofs_free_layers_info(layers_info, i); out_json: json_object_put(root); out: @@ -771,8 +797,92 @@ out: return ERR_PTR(ret); } -static int ocierofs_extract_layer(struct erofs_oci *oci, struct erofs_importer *importer, - const char *digest, const char *auth_header) +/** + * ocierofs_process_tar_stream - Process tar stream from file descriptor + * @importer: EROFS importer structure + * @fd: File descriptor containing tar data + * + * Initialize tar stream, parse all entries, and clean up resources. + * + * Return: 0 on success, negative errno on failure + */ +static int ocierofs_process_tar_stream(struct erofs_importer *importer, int fd) +{ + struct erofs_tarfile tarfile = {}; + int ret; + + init_list_head(&tarfile.global.xattrs); + + ret = erofs_iostream_open(&tarfile.ios, fd, EROFS_IOS_DECODER_GZIP); + if (ret) { + erofs_err("failed to initialize tar stream: %s", + erofs_strerror(ret)); + return ret; + } + + do { + ret = tarerofs_parse_tar(importer, &tarfile); + /* Continue parsing until end of archive */ + } while (!ret); + erofs_iostream_close(&tarfile.ios); + + if (ret < 0 && ret != -ENODATA) { + erofs_err("failed to process tar stream: %s", + erofs_strerror(ret)); + return ret; + } + + return 0; +} + +int ocierofs_prepare_auth(struct erofs_oci *oci, char **auth_header_out, + bool *using_basic_auth) +{ + char *auth_header = NULL; + int ret = 0; + + if (using_basic_auth) + *using_basic_auth = false; + if (auth_header_out) + *auth_header_out = NULL; + + if (oci->params.username && oci->params.password && + oci->params.username[0] && oci->params.password[0]) { + auth_header = ocierofs_get_auth_token(oci, + oci->params.registry, + oci->params.repository, + oci->params.username, + oci->params.password); + if (IS_ERR(auth_header)) { + auth_header = NULL; + ret = ocierofs_curl_setup_basic_auth(oci->curl, + oci->params.username, + oci->params.password); + if (ret) + return ret; + if (using_basic_auth) + *using_basic_auth = true; + } + } else { + auth_header = ocierofs_get_auth_token(oci, + oci->params.registry, + oci->params.repository, + NULL, NULL); + if (IS_ERR(auth_header)) + auth_header = NULL; + } + + if (auth_header_out) + *auth_header_out = auth_header; + else + free(auth_header); + return 0; +} + +static int ocierofs_download_blob_to_fd(struct erofs_oci *oci, + const char *digest, + const char *auth_header, + int outfd) { struct erofs_oci_request req = {}; struct erofs_oci_stream stream = {}; @@ -782,20 +892,14 @@ static int ocierofs_extract_layer(struct erofs_oci *oci, struct erofs_importer * stream = (struct erofs_oci_stream) { .digest = digest, - .blobfd = erofs_tmpfile(), + .blobfd = outfd, }; - if (stream.blobfd < 0) { - erofs_err("failed to create temporary file for %s", digest); - return -errno; - } api_registry = (!strcmp(oci->params.registry, DOCKER_REGISTRY)) ? DOCKER_API_REGISTRY : oci->params.registry; if (asprintf(&req.url, "https://%s/v2/%s/blobs/%s", - api_registry, oci->params.repository, digest) == -1) { - ret = -ENOMEM; - goto out; - } + api_registry, oci->params.repository, digest) == -1) + return -ENOMEM; if (auth_header && strstr(auth_header, "Bearer")) req.headers = curl_slist_append(req.headers, auth_header); @@ -822,6 +926,32 @@ static int ocierofs_extract_layer(struct erofs_oci *oci, struct erofs_importer * ret = -EIO; goto out; } + ret = 0; +out: + if (req.headers) + curl_slist_free_all(req.headers); + free(req.url); + return ret; +} + +static int ocierofs_extract_layer(struct erofs_oci *oci, + const char *digest, const char *auth_header) +{ + struct erofs_oci_stream stream = {}; + int ret; + + stream = (struct erofs_oci_stream) { + .digest = digest, + .blobfd = erofs_tmpfile(), + }; + if (stream.blobfd < 0) { + erofs_err("failed to create temporary file for %s", digest); + return -errno; + } + + ret = ocierofs_download_blob_to_fd(oci, digest, auth_header, stream.blobfd); + if (ret) + goto out; if (lseek(stream.blobfd, 0, SEEK_SET) < 0) { erofs_err("failed to seek to beginning of temp file: %s", @@ -830,162 +960,114 @@ static int ocierofs_extract_layer(struct erofs_oci *oci, struct erofs_importer * goto out; } - memset(&stream.tarfile, 0, sizeof(stream.tarfile)); - init_list_head(&stream.tarfile.global.xattrs); - - ret = erofs_iostream_open(&stream.tarfile.ios, stream.blobfd, - EROFS_IOS_DECODER_GZIP); - if (ret) { - erofs_err("failed to initialize tar stream: %s", - erofs_strerror(ret)); - goto out; - } - - do { - ret = tarerofs_parse_tar(importer, &stream.tarfile); - /* Continue parsing until end of archive */ - } while (!ret); - erofs_iostream_close(&stream.tarfile.ios); - - if (ret < 0 && ret != -ENODATA) { - erofs_err("failed to process tar stream: %s", - erofs_strerror(ret)); - goto out; - } - ret = 0; + return stream.blobfd; out: if (stream.blobfd >= 0) close(stream.blobfd); - if (req.headers) - curl_slist_free_all(req.headers); - free(req.url); return ret; } -/** - * ocierofs_build_trees - Build file trees from OCI container image layers - * @importer: EROFS importer structure - * @oci: OCI client structure with configured parameters - * - * Extract and build file system trees from all layers of an OCI container - * image. - * - * Return: 0 on success, negative errno on failure - */ -int ocierofs_build_trees(struct erofs_importer *importer, struct erofs_oci *oci) +int ocierofs_prepare_layers(struct erofs_oci *oci, struct ocierofs_image_context *ctx) { - char *auth_header = NULL; - char *manifest_digest = NULL; - char **layers = NULL; - int layer_count = 0; - int ret, i; - - if (!importer || !oci) - return -EINVAL; + int ret; - if (oci->params.username && oci->params.password && - oci->params.username[0] && oci->params.password[0]) { - auth_header = ocierofs_get_auth_token(oci, - oci->params.registry, - oci->params.repository, - oci->params.username, - oci->params.password); - if (IS_ERR(auth_header)) { - auth_header = NULL; - ret = ocierofs_curl_setup_basic_auth(oci->curl, - oci->params.username, - oci->params.password); - if (ret) - goto out; - } - } else { - auth_header = ocierofs_get_auth_token(oci, - oci->params.registry, - oci->params.repository, - NULL, NULL); - if (IS_ERR(auth_header)) - auth_header = NULL; - } + ret = ocierofs_prepare_auth(oci, &ctx->auth_header, &ctx->using_basic); + if (ret) + return ret; - manifest_digest = ocierofs_get_manifest_digest(oci, oci->params.registry, - oci->params.repository, - oci->params.tag, - oci->params.platform, - auth_header); - if (IS_ERR(manifest_digest)) { - ret = PTR_ERR(manifest_digest); + ctx->manifest_digest = ocierofs_get_manifest_digest(oci, oci->params.registry, + oci->params.repository, + oci->params.tag, + oci->params.platform, + ctx->auth_header); + if (IS_ERR(ctx->manifest_digest)) { + ret = PTR_ERR(ctx->manifest_digest); erofs_err("failed to get manifest digest: %s", erofs_strerror(ret)); goto out_auth; } - layers = ocierofs_get_layers_info(oci, oci->params.registry, - oci->params.repository, - manifest_digest, auth_header, - &layer_count); - if (IS_ERR(layers)) { - ret = PTR_ERR(layers); + ctx->layers = ocierofs_fetch_layers_info(oci, oci->params.registry, + oci->params.repository, + ctx->manifest_digest, ctx->auth_header, + &ctx->layer_count); + if (IS_ERR(ctx->layers)) { + ret = PTR_ERR(ctx->layers); erofs_err("failed to get image layers: %s", erofs_strerror(ret)); goto out_manifest; } if (oci->params.layer_index >= 0) { - if (oci->params.layer_index >= layer_count) { + if (oci->params.layer_index >= ctx->layer_count) { erofs_err("layer index %d exceeds available layers (%d)", - oci->params.layer_index, layer_count); + oci->params.layer_index, ctx->layer_count); ret = -EINVAL; goto out_layers; } - layer_count = 1; - i = oci->params.layer_index; + ctx->layer_count = 1; + ctx->start_index = oci->params.layer_index; } else { - i = 0; + ctx->start_index = 0; } - while (i < layer_count) { - char *trimmed = erofs_trim_for_progressinfo(layers[i], + return 0; + +out_layers: + free(ctx->layers); + ctx->layers = NULL; +out_manifest: + free(ctx->manifest_digest); + ctx->manifest_digest = NULL; +out_auth: + free(ctx->auth_header); + ctx->auth_header = NULL; + if (ctx->using_basic) + ocierofs_curl_clear_auth(oci); + return ret; +} + +int ocierofs_build_trees(struct erofs_importer *importer, struct erofs_oci *oci) +{ + struct ocierofs_image_context image_ctx = {}; + int ret, i; + + if (!importer || !oci) + return -EINVAL; + + ret = ocierofs_prepare_layers(oci, &image_ctx); + if (ret) + return ret; + + i = image_ctx.start_index; + while (i < image_ctx.layer_count) { + char *trimmed = erofs_trim_for_progressinfo(image_ctx.layers[i]->digest, sizeof("Extracting layer ...") - 1); erofs_update_progressinfo("Extracting layer %d: %s ...", i, trimmed); free(trimmed); - ret = ocierofs_extract_layer(oci, importer, layers[i], - auth_header); - if (ret) { + int fd = ocierofs_extract_layer(oci, image_ctx.layers[i]->digest, + image_ctx.auth_header); + if (fd < 0) { erofs_err("failed to extract layer %d: %s", i, + erofs_strerror(fd)); + break; + } + ret = ocierofs_process_tar_stream(importer, fd); + close(fd); + if (ret) { + erofs_err("failed to process tar stream for layer %d: %s", i, erofs_strerror(ret)); break; } i++; } -out_layers: - for (i = 0; i < layer_count; i++) - free(layers[i]); - free(layers); -out_manifest: - free(manifest_digest); -out_auth: - free(auth_header); - if (oci->params.username && oci->params.password && - oci->params.username[0] && oci->params.password[0] && - !auth_header) { - ocierofs_curl_clear_auth(oci->curl); - } -out: + ocierofs_free_image_ctx(&image_ctx); + return ret; } -/** - * ocierofs_init - Initialize OCI client with default parameters - * @oci: OCI client structure to initialize - * - * Initialize OCI client structure, set up CURL handle, and configure - * default parameters including platform (linux/amd64), registry - * (registry-1.docker.io), and tag (latest). - * - * Return: 0 on success, negative errno on failure - */ int ocierofs_init(struct erofs_oci *oci) { if (!oci) @@ -1066,3 +1148,130 @@ int erofs_oci_params_set_string(char **field, const char *value) *field = new_value; return 0; } + +int ocierofs_parse_ref(struct erofs_oci *oci, const char *ref_str) +{ + char *slash, *colon, *dot; + const char *repo_part; + size_t len; + + if (!oci || !ref_str) + return -EINVAL; + + slash = strchr(ref_str, '/'); + if (slash) { + dot = strchr(ref_str, '.'); + if (dot && dot < slash) { + char *registry_str; + + len = slash - ref_str; + registry_str = strndup(ref_str, len); + + if (!registry_str) { + erofs_err("failed to allocate memory for registry"); + return -ENOMEM; + } + if (erofs_oci_params_set_string(&oci->params.registry, + registry_str)) { + free(registry_str); + erofs_err("failed to set registry"); + return -ENOMEM; + } + free(registry_str); + repo_part = slash + 1; + } else { + repo_part = ref_str; + } + } else { + repo_part = ref_str; + } + + colon = strchr(repo_part, ':'); + if (colon) { + char *repo_str; + + len = colon - repo_part; + repo_str = strndup(repo_part, len); + + if (!repo_str) { + erofs_err("failed to allocate memory for repository"); + return -ENOMEM; + } + + if (!strchr(repo_str, '/') && + (!strcmp(oci->params.registry, DOCKER_API_REGISTRY) || + !strcmp(oci->params.registry, DOCKER_REGISTRY))) { + char *full_repo; + + if (asprintf(&full_repo, "library/%s", repo_str) == -1) { + free(repo_str); + erofs_err("failed to allocate memory for full repository name"); + return -ENOMEM; + } + free(repo_str); + repo_str = full_repo; + } + + if (erofs_oci_params_set_string(&oci->params.repository, + repo_str)) { + free(repo_str); + erofs_err("failed to set repository"); + return -ENOMEM; + } + free(repo_str); + + if (erofs_oci_params_set_string(&oci->params.tag, + colon + 1)) { + erofs_err("failed to set tag"); + return -ENOMEM; + } + } else { + char *repo_str = strdup(repo_part); + + if (!repo_str) { + erofs_err("failed to allocate memory for repository"); + return -ENOMEM; + } + + if (!strchr(repo_str, '/') && + (!strcmp(oci->params.registry, DOCKER_API_REGISTRY) || + !strcmp(oci->params.registry, DOCKER_REGISTRY))) { + char *full_repo; + + if (asprintf(&full_repo, "library/%s", repo_str) == -1) { + free(repo_str); + erofs_err("failed to allocate memory for full repository name"); + return -ENOMEM; + } + free(repo_str); + repo_str = full_repo; + } + + if (erofs_oci_params_set_string(&oci->params.repository, + repo_str)) { + free(repo_str); + erofs_err("failed to set repository"); + return -ENOMEM; + } + free(repo_str); + } + + return 0; +} + +void ocierofs_free_image_ctx(struct ocierofs_image_context *ctx) +{ + if (!ctx) + return; + + ocierofs_free_layers_info(ctx->layers, ctx->layer_count); + free(ctx->manifest_digest); + free(ctx->auth_header); + + ctx->layers = NULL; + ctx->manifest_digest = NULL; + ctx->auth_header = NULL; + ctx->layer_count = 0; + ctx->start_index = 0; + ctx->using_basic = false; +} diff --git a/mkfs/main.c b/mkfs/main.c index bc895f1..24d384d 100644 --- a/mkfs/main.c +++ b/mkfs/main.c @@ -689,18 +689,24 @@ static int mkfs_parse_s3_cfg(char *cfg_str) #endif #ifdef OCIEROFS_ENABLED - - -/** - * mkfs_parse_oci_options - Parse comma-separated OCI options string +/* + * mkfs_parse_oci_options - Parse OCI options for mkfs tool + * @oci: OCI client structure to update * @options_str: comma-separated options string * * Parse OCI options string containing comma-separated key=value pairs. - * Supported options include platform, layer, username, and password. + * This function is specific to the mkfs tool and can be enhanced + * independently of other tools. + * + * Supported options include: + * - platform=<os/arch>: target platform (e.g., "linux/amd64") + * - layer=<index>: specific layer to extract (0-based index) + * - username=<user>: authentication username + * - password=<pass>: authentication password * * Return: 0 on success, negative errno on failure */ -static int mkfs_parse_oci_options(char *options_str) +static int mkfs_parse_oci_options(struct erofs_oci *oci, char *options_str) { char *opt, *q, *p; int ret; @@ -717,7 +723,7 @@ static int mkfs_parse_oci_options(char *options_str) p = strstr(opt, "platform="); if (p) { p += strlen("platform="); - ret = erofs_oci_params_set_string(&ocicfg.params.platform, p); + ret = erofs_oci_params_set_string(&oci->params.platform, p); if (ret) { erofs_err("failed to set platform"); return ret; @@ -726,17 +732,17 @@ static int mkfs_parse_oci_options(char *options_str) p = strstr(opt, "layer="); if (p) { p += strlen("layer="); - ocicfg.params.layer_index = atoi(p); - if (ocicfg.params.layer_index < 0) { + oci->params.layer_index = atoi(p); + if (oci->params.layer_index < 0) { erofs_err("invalid layer index %d", - ocicfg.params.layer_index); + oci->params.layer_index); return -EINVAL; } } else { p = strstr(opt, "username="); if (p) { p += strlen("username="); - ret = erofs_oci_params_set_string(&ocicfg.params.username, p); + ret = erofs_oci_params_set_string(&oci->params.username, p); if (ret) { erofs_err("failed to set username"); return ret; @@ -745,13 +751,13 @@ static int mkfs_parse_oci_options(char *options_str) p = strstr(opt, "password="); if (p) { p += strlen("password="); - ret = erofs_oci_params_set_string(&ocicfg.params.password, p); + ret = erofs_oci_params_set_string(&oci->params.password, p); if (ret) { erofs_err("failed to set password"); return ret; } } else { - erofs_err("invalid --oci value %s", opt); + erofs_err("mkfs: invalid --oci value %s", opt); return -EINVAL; } } @@ -763,125 +769,6 @@ static int mkfs_parse_oci_options(char *options_str) return 0; } - -/** - * mkfs_parse_oci_ref - Parse OCI image reference string - * @ref_str: OCI image reference in various formats - * - * Parse OCI image reference which can be in formats: - * - registry.example.com/namespace/repo:tag - * - namespace/repo:tag (uses default registry) - * - repo:tag (adds library/ prefix for Docker Hub) - * - repo (uses default tag "latest") - * - * Return: 0 on success, negative errno on failure - */ -static int mkfs_parse_oci_ref(const char *ref_str) -{ - char *slash, *colon, *dot; - const char *repo_part; - size_t len; - - slash = strchr(ref_str, '/'); - if (slash) { - dot = strchr(ref_str, '.'); - if (dot && dot < slash) { - char *registry_str; - - len = slash - ref_str; - registry_str = strndup(ref_str, len); - - if (!registry_str) { - erofs_err("failed to allocate memory for registry"); - return -ENOMEM; - } - if (erofs_oci_params_set_string(&ocicfg.params.registry, - registry_str)) { - free(registry_str); - erofs_err("failed to set registry"); - return -ENOMEM; - } - free(registry_str); - repo_part = slash + 1; - } else { - repo_part = ref_str; - } - } else { - repo_part = ref_str; - } - - colon = strchr(repo_part, ':'); - if (colon) { - char *repo_str; - - len = colon - repo_part; - repo_str = strndup(repo_part, len); - - if (!repo_str) { - erofs_err("failed to allocate memory for repository"); - return -ENOMEM; - } - - if (!strchr(repo_str, '/') && - (!strcmp(ocicfg.params.registry, DOCKER_API_REGISTRY) || - !strcmp(ocicfg.params.registry, DOCKER_REGISTRY))) { - char *full_repo; - - if (asprintf(&full_repo, "library/%s", repo_str) == -1) { - free(repo_str); - erofs_err("failed to allocate memory for full repository name"); - return -ENOMEM; - } - free(repo_str); - repo_str = full_repo; - } - - if (erofs_oci_params_set_string(&ocicfg.params.repository, - repo_str)) { - free(repo_str); - erofs_err("failed to set repository"); - return -ENOMEM; - } - free(repo_str); - - if (erofs_oci_params_set_string(&ocicfg.params.tag, - colon + 1)) { - erofs_err("failed to set tag"); - return -ENOMEM; - } - } else { - char *repo_str = strdup(repo_part); - - if (!repo_str) { - erofs_err("failed to allocate memory for repository"); - return -ENOMEM; - } - - if (!strchr(repo_str, '/') && - (!strcmp(ocicfg.params.registry, DOCKER_API_REGISTRY) || - !strcmp(ocicfg.params.registry, DOCKER_REGISTRY))) { - char *full_repo; - - if (asprintf(&full_repo, "library/%s", repo_str) == -1) { - free(repo_str); - erofs_err("failed to allocate memory for full repository name"); - return -ENOMEM; - } - free(repo_str); - repo_str = full_repo; - } - - if (erofs_oci_params_set_string(&ocicfg.params.repository, - repo_str)) { - free(repo_str); - erofs_err("failed to set repository"); - return -ENOMEM; - } - free(repo_str); - } - - return 0; -} #endif static int mkfs_parse_one_compress_alg(char *alg, @@ -1955,10 +1842,10 @@ int main(int argc, char **argv) if (err) goto exit; - err = mkfs_parse_oci_options(mkfs_oci_options); + err = mkfs_parse_oci_options(&ocicfg, mkfs_oci_options); if (err) goto exit; - err = mkfs_parse_oci_ref(cfg.c_src_path); + err = ocierofs_parse_ref(&ocicfg, cfg.c_src_path); if (err) goto exit; -- 2.51.0 ^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH v3 0/2] erofs-utils: refactor OCI and add NBD-backed OCI image mounting 2025-09-01 5:10 [PATCH v1] erofs-utils: refactor OCI code for better modularity ChengyuZhu6 ` (2 preceding siblings ...) 2025-09-02 3:29 ` [PATCH v2-changed] " ChengyuZhu6 @ 2025-09-03 8:29 ` ChengyuZhu6 2025-09-03 8:29 ` [PATCH v3 1/2] erofs-utils: refactor OCI code for better modularity ChengyuZhu6 2025-09-03 8:29 ` [PATCH v3 2/2] erofs-utils: add NBD-backed OCI image mounting ChengyuZhu6 2025-09-04 5:33 ` [PATCH v4 0/3] erofs-utils: refactor OCI and " ChengyuZhu6 2025-09-04 6:36 ` [PATCH v5 0/3] erofs-utils: refactor OCI and " ChengyuZhu6 5 siblings, 2 replies; 19+ messages in thread From: ChengyuZhu6 @ 2025-09-03 8:29 UTC (permalink / raw) To: linux-erofs; +Cc: xiang, hsiangkao, Chengyu Zhu From: Chengyu Zhu <hudsonzhu@tencent.com> This series refactors the OCI handling in erofs-utils and adds NBD-backed mounting of OCI images. It enables mounting EROFS container images directly from registries without pre-downloading Chengyu Zhu (2): erofs-utils: refactor OCI code for better modularity erofs-utils: add NBD-backed OCI image mounting lib/liberofs_oci.h | 97 ++-- lib/remotes/oci.c | 1087 +++++++++++++++++++++++++++++--------------- mkfs/main.c | 192 ++------ mount/Makefile.am | 3 +- mount/main.c | 236 ++++++++++ 5 files changed, 1036 insertions(+), 579 deletions(-) -- 2.51.0 ^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH v3 1/2] erofs-utils: refactor OCI code for better modularity 2025-09-03 8:29 ` [PATCH v3 0/2] erofs-utils: refactor OCI and add NBD-backed OCI image mounting ChengyuZhu6 @ 2025-09-03 8:29 ` ChengyuZhu6 2025-09-03 8:47 ` Gao Xiang 2025-09-03 8:29 ` [PATCH v3 2/2] erofs-utils: add NBD-backed OCI image mounting ChengyuZhu6 1 sibling, 1 reply; 19+ messages in thread From: ChengyuZhu6 @ 2025-09-03 8:29 UTC (permalink / raw) To: linux-erofs; +Cc: xiang, hsiangkao, Chengyu Zhu From: Chengyu Zhu <hudsonzhu@tencent.com> Refactor OCI code to improve code organization and maintainability: Key changes: - Add `ocierofs_get_api_registry()` and unify API endpoint selection. - Implement Bearer token discovery with Basic fallback; cache auth header. - Parse layer metadata (digest, mediaType, size) and add a proper free helper. - Split blob download from tar processing; process tar via a temp fd. - Rework init/teardown into `ocierofs_init()` and `ocierofs_ctx_cleanup()`. - Update mkfs to use `struct ocierofs_config` and new `--oci` parsing. Signed-off-by: Chengyu Zhu <hudsonzhu@tencent.com> --- lib/liberofs_oci.h | 84 ++--- lib/remotes/oci.c | 885 ++++++++++++++++++++++++++------------------- mkfs/main.c | 192 ++-------- 3 files changed, 566 insertions(+), 595 deletions(-) diff --git a/lib/liberofs_oci.h b/lib/liberofs_oci.h index 3a8108b..d119a2b 100644 --- a/lib/liberofs_oci.h +++ b/lib/liberofs_oci.h @@ -8,85 +8,65 @@ #include <stdbool.h> -#define DOCKER_REGISTRY "docker.io" -#define DOCKER_API_REGISTRY "registry-1.docker.io" - #ifdef __cplusplus extern "C" { #endif -struct erofs_inode; -struct CURL; struct erofs_importer; -/** - * struct erofs_oci_params - OCI configuration parameters - * @registry: registry hostname (e.g., "registry-1.docker.io") - * @repository: image repository (e.g., "library/ubuntu") - * @tag: image tag or digest (e.g., "latest" or sha256:...) +/* + * struct ocierofs_config - OCI configuration structure + * @image_ref: OCI image reference (e.g., "ubuntu:latest", "myregistry.com/app:v1.0") * @platform: target platform in "os/arch" format (e.g., "linux/amd64") * @username: username for authentication (optional) * @password: password for authentication (optional) * @layer_index: specific layer to extract (-1 for all layers) * - * Configuration structure for OCI image parameters including registry - * location, image identification, platform specification, and authentication - * credentials. */ -struct erofs_oci_params { - char *registry; - char *repository; - char *tag; +struct ocierofs_config { + char *image_ref; char *platform; char *username; char *password; int layer_index; }; -/** - * struct erofs_oci - Combined OCI client structure - * @curl: CURL handle for HTTP requests - * @params: OCI configuration parameters - * - * Main OCI client structure combining CURL HTTP client with - * OCI-specific configuration parameters. - */ -struct erofs_oci { - struct CURL *curl; - struct erofs_oci_params params; +struct ocierofs_layer_info { + char *digest; + char *media_type; + u64 size; }; -/* - * ocierofs_init - Initialize OCI client with default parameters - * @oci: OCI client structure to initialize - * - * Return: 0 on success, negative errno on failure - */ -int ocierofs_init(struct erofs_oci *oci); +struct ocierofs_ctx { + struct { + struct CURL *curl; + char *auth_header; + bool using_basic; + } net; -/* - * ocierofs_cleanup - Clean up OCI client and free allocated resources - * @oci: OCI client structure to clean up - */ -void ocierofs_cleanup(struct erofs_oci *oci); + struct { + char *registry; + char *repository; + char *platform; + int layer_index; + char *tag; + char *manifest_digest; + struct ocierofs_layer_info **layers; + int layer_count; + } img; +}; -/* - * erofs_oci_params_set_string - Set a string field with dynamic allocation - * @field: pointer to the string field to set - * @value: string value to set - * - * Return: 0 on success, negative errno on failure - */ -int erofs_oci_params_set_string(char **field, const char *value); +int ocierofs_init(struct ocierofs_ctx *ctx, const struct ocierofs_config *config); /* - * ocierofs_build_trees - Build file trees from OCI container image layers - * @root: root inode to build the file tree under - * @oci: OCI client structure with configured parameters + * ocierofs_build_trees - Build file trees from an OCI container image + * @importer: erofs importer to populate + * @cfg: oci configuration * * Return: 0 on success, negative errno on failure */ -int ocierofs_build_trees(struct erofs_importer *importer, struct erofs_oci *oci); +int ocierofs_build_trees(struct erofs_importer *importer, + const struct ocierofs_config *cfg); #ifdef __cplusplus } diff --git a/lib/remotes/oci.c b/lib/remotes/oci.c index 0fb8c1f..01f1e24 100644 --- a/lib/remotes/oci.c +++ b/lib/remotes/oci.c @@ -21,6 +21,9 @@ #include "liberofs_oci.h" #include "liberofs_private.h" +#define DOCKER_REGISTRY "docker.io" +#define DOCKER_API_REGISTRY "registry-1.docker.io" + #define DOCKER_MEDIATYPE_MANIFEST_V2 \ "application/vnd.docker.distribution.manifest.v2+json" #define DOCKER_MEDIATYPE_MANIFEST_V1 \ @@ -30,28 +33,67 @@ #define OCI_MEDIATYPE_MANIFEST "application/vnd.oci.image.manifest.v1+json" #define OCI_MEDIATYPE_INDEX "application/vnd.oci.image.index.v1+json" -struct erofs_oci_request { +struct ocierofs_request { char *url; struct curl_slist *headers; }; -struct erofs_oci_response { +struct ocierofs_response { char *data; size_t size; long http_code; }; -struct erofs_oci_stream { - struct erofs_tarfile tarfile; +struct ocierofs_stream { const char *digest; int blobfd; }; +static inline const char *ocierofs_get_api_registry(const char *registry) +{ + if (!registry) + return DOCKER_API_REGISTRY; + return !strcmp(registry, DOCKER_REGISTRY) ? DOCKER_API_REGISTRY : registry; +} + +static inline bool ocierofs_is_manifest_list(const char *media_type) +{ + return media_type && (!strcmp(media_type, DOCKER_MEDIATYPE_MANIFEST_LIST) || + !strcmp(media_type, OCI_MEDIATYPE_INDEX)); +} + +static inline bool ocierofs_is_manifest(const char *media_type) +{ + return media_type && (!strcmp(media_type, DOCKER_MEDIATYPE_MANIFEST_V2) || + !strcmp(media_type, OCI_MEDIATYPE_MANIFEST)); +} + +static inline void ocierofs_request_cleanup(struct ocierofs_request *req) +{ + if (!req) + return; + if (req->headers) + curl_slist_free_all(req->headers); + free(req->url); + req->url = NULL; + req->headers = NULL; +} + +static inline void ocierofs_response_cleanup(struct ocierofs_response *resp) +{ + if (!resp) + return; + free(resp->data); + resp->data = NULL; + resp->size = 0; + resp->http_code = 0; +} + static size_t ocierofs_write_callback(void *contents, size_t size, size_t nmemb, void *userp) { size_t realsize = size * nmemb; - struct erofs_oci_response *resp = userp; + struct ocierofs_response *resp = userp; char *ptr; if (!resp->data) @@ -72,16 +114,23 @@ static size_t ocierofs_write_callback(void *contents, size_t size, static size_t ocierofs_layer_write_callback(void *contents, size_t size, size_t nmemb, void *userp) { - struct erofs_oci_stream *stream = userp; + struct ocierofs_stream *stream = userp; size_t realsize = size * nmemb; + const char *buf = contents; + size_t written = 0; if (stream->blobfd < 0) return 0; - if (write(stream->blobfd, contents, realsize) != realsize) { - erofs_err("failed to write layer data for layer %s", - stream->digest); - return 0; + while (written < realsize) { + ssize_t n = write(stream->blobfd, buf + written, realsize - written); + + if (n < 0) { + erofs_err("failed to write layer data for layer %s", + stream->digest); + return 0; + } + written += n; } return realsize; } @@ -93,6 +142,14 @@ static int ocierofs_curl_setup_common_options(struct CURL *curl) curl_easy_setopt(curl, CURLOPT_TIMEOUT, 120L); curl_easy_setopt(curl, CURLOPT_NOSIGNAL, 1L); curl_easy_setopt(curl, CURLOPT_USERAGENT, "ocierofs/" PACKAGE_VERSION); + curl_easy_setopt(curl, CURLOPT_ACCEPT_ENCODING, ""); + curl_easy_setopt(curl, CURLOPT_TCP_KEEPALIVE, 1L); +#if defined(CURLOPT_TCP_KEEPIDLE) + curl_easy_setopt(curl, CURLOPT_TCP_KEEPIDLE, 30L); +#endif +#if defined(CURLOPT_TCP_KEEPINTVL) + curl_easy_setopt(curl, CURLOPT_TCP_KEEPINTVL, 15L); +#endif return 0; } @@ -111,10 +168,10 @@ static int ocierofs_curl_setup_basic_auth(struct CURL *curl, const char *usernam return 0; } -static int ocierofs_curl_clear_auth(struct CURL *curl) +static int ocierofs_curl_clear_auth(struct ocierofs_ctx *ctx) { - curl_easy_setopt(curl, CURLOPT_USERPWD, NULL); - curl_easy_setopt(curl, CURLOPT_HTTPAUTH, CURLAUTH_NONE); + curl_easy_setopt(ctx->net.curl, CURLOPT_USERPWD, NULL); + curl_easy_setopt(ctx->net.curl, CURLOPT_HTTPAUTH, CURLAUTH_NONE); return 0; } @@ -173,20 +230,20 @@ static int ocierofs_curl_perform(struct CURL *curl, long *http_code_out) return 0; } -static int ocierofs_request_perform(struct erofs_oci *oci, - struct erofs_oci_request *req, - struct erofs_oci_response *resp) +static int ocierofs_request_perform(struct ocierofs_ctx *ctx, + struct ocierofs_request *req, + struct ocierofs_response *resp) { int ret; - ret = ocierofs_curl_setup_rq(oci->curl, req->url, + ret = ocierofs_curl_setup_rq(ctx->net.curl, req->url, OCIEROFS_HTTP_GET, req->headers, - ocierofs_write_callback, resp, + ocierofs_write_callback, resp, NULL, NULL); if (ret) return ret; - ret = ocierofs_curl_perform(oci->curl, &resp->http_code); + ret = ocierofs_curl_perform(ctx->net.curl, &resp->http_code); if (ret) return ret; @@ -201,15 +258,10 @@ static int ocierofs_request_perform(struct erofs_oci *oci, * @realm_out: pointer to store realm value * @service_out: pointer to store service value * @scope_out: pointer to store scope value - * - * Parse Bearer authentication header and extract realm, service, and scope - * parameters for subsequent token requests. - * - * Return: 0 on success, negative errno on failure */ static int ocierofs_parse_auth_header(const char *auth_header, - char **realm_out, char **service_out, - char **scope_out) + char **realm_out, char **service_out, + char **scope_out) { char *realm = NULL, *service = NULL, *scope = NULL; static const char * const param_names[] = {"realm=", "service=", "scope="}; @@ -218,7 +270,6 @@ static int ocierofs_parse_auth_header(const char *auth_header, const char *p; int i, ret = 0; - // https://datatracker.ietf.org/doc/html/rfc6750#section-3 if (strncmp(auth_header, "Bearer ", strlen("Bearer "))) return -EINVAL; @@ -226,7 +277,6 @@ static int ocierofs_parse_auth_header(const char *auth_header, if (!header_copy) return -ENOMEM; - /* Clean up header: replace newlines with spaces and remove double spaces */ for (char *q = header_copy; *q; q++) { if (*q == '\n' || *q == '\r') *q = ' '; @@ -274,22 +324,9 @@ out: return ret; } -/** - * ocierofs_extract_www_auth_info - Extract WWW-Authenticate header information - * @resp_data: HTTP response data containing headers - * @realm_out: pointer to store realm value (optional) - * @service_out: pointer to store service value (optional) - * @scope_out: pointer to store scope value (optional) - * - * Extract realm, service, and scope from WWW-Authenticate header in HTTP response. - * This function handles the common pattern of parsing WWW-Authenticate headers - * that appears in multiple places in the OCI authentication flow. - * - * Return: 0 on success, negative errno on failure - */ static int ocierofs_extract_www_auth_info(const char *resp_data, - char **realm_out, char **service_out, - char **scope_out) + char **realm_out, char **service_out, + char **scope_out) { char *www_auth; char *line_end; @@ -333,29 +370,15 @@ static int ocierofs_extract_www_auth_info(const char *resp_data, return ret; } -/** - * ocierofs_get_auth_token_with_url - Get authentication token from auth server - * @oci: OCI client structure - * @auth_url: authentication server URL - * @service: service name for authentication - * @repository: repository name - * @username: username for basic auth (optional) - * @password: password for basic auth (optional) - * - * Request authentication token from the specified auth server URL using - * basic authentication if credentials are provided. - * - * Return: authentication header string on success, ERR_PTR on failure - */ -static char *ocierofs_get_auth_token_with_url(struct erofs_oci *oci, - const char *auth_url, - const char *service, - const char *repository, - const char *username, - const char *password) +static char *ocierofs_get_auth_token_with_url(struct ocierofs_ctx *ctx, + const char *auth_url, + const char *service, + const char *repository, + const char *username, + const char *password) { - struct erofs_oci_request req = {}; - struct erofs_oci_response resp = {}; + struct ocierofs_request req = {}; + struct ocierofs_response resp = {}; json_object *root, *token_obj, *access_token_obj; const char *token; char *auth_header = NULL; @@ -370,14 +393,14 @@ static char *ocierofs_get_auth_token_with_url(struct erofs_oci *oci, } if (username && password && *username) { - ret = ocierofs_curl_setup_basic_auth(oci->curl, username, - password); + ret = ocierofs_curl_setup_basic_auth(ctx->net.curl, username, + password); if (ret) goto out_url; } - ret = ocierofs_request_perform(oci, &req, &resp); - ocierofs_curl_clear_auth(oci->curl); + ret = ocierofs_request_perform(ctx, &req, &resp); + ocierofs_curl_clear_auth(ctx); if (ret) goto out_url; @@ -391,7 +414,7 @@ static char *ocierofs_get_auth_token_with_url(struct erofs_oci *oci, if (!root) { erofs_err("failed to parse auth response"); ret = -EINVAL; - goto out_url; + goto out_json; } if (!json_object_object_get_ex(root, "token", &token_obj) && @@ -416,16 +439,16 @@ static char *ocierofs_get_auth_token_with_url(struct erofs_oci *oci, out_json: json_object_put(root); out_url: - free(req.url); - free(resp.data); + ocierofs_response_cleanup(&resp); + ocierofs_request_cleanup(&req); return ret ? ERR_PTR(ret) : auth_header; } -static char *ocierofs_discover_auth_endpoint(struct erofs_oci *oci, - const char *registry, - const char *repository) +static char *ocierofs_discover_auth_endpoint(struct ocierofs_ctx *ctx, + const char *registry, + const char *repository) { - struct erofs_oci_response resp = {}; + struct ocierofs_response resp = {}; char *realm = NULL; char *service = NULL; char *result = NULL; @@ -434,20 +457,20 @@ static char *ocierofs_discover_auth_endpoint(struct erofs_oci *oci, CURLcode res; long http_code; - api_registry = (!strcmp(registry, DOCKER_REGISTRY)) ? DOCKER_API_REGISTRY : registry; + api_registry = ocierofs_get_api_registry(registry); if (asprintf(&test_url, "https://%s/v2/%s/manifests/nonexistent", api_registry, repository) < 0) return NULL; - curl_easy_reset(oci->curl); - ocierofs_curl_setup_common_options(oci->curl); + curl_easy_reset(ctx->net.curl); + ocierofs_curl_setup_common_options(ctx->net.curl); - ocierofs_curl_setup_rq(oci->curl, test_url, OCIEROFS_HTTP_HEAD, NULL, + ocierofs_curl_setup_rq(ctx->net.curl, test_url, OCIEROFS_HTTP_HEAD, NULL, NULL, NULL, ocierofs_write_callback, &resp); - res = curl_easy_perform(oci->curl); - curl_easy_getinfo(oci->curl, CURLINFO_RESPONSE_CODE, &http_code); + res = curl_easy_perform(ctx->net.curl); + curl_easy_getinfo(ctx->net.curl, CURLINFO_RESPONSE_CODE, &http_code); if (res == CURLE_OK && (http_code == 401 || http_code == 403 || http_code == 404) && resp.data) { @@ -458,14 +481,14 @@ static char *ocierofs_discover_auth_endpoint(struct erofs_oci *oci, } free(realm); free(service); - free(resp.data); + ocierofs_response_cleanup(&resp); free(test_url); return result; } -static char *ocierofs_get_auth_token(struct erofs_oci *oci, const char *registry, - const char *repository, const char *username, - const char *password) +static char *ocierofs_get_auth_token(struct ocierofs_ctx *ctx, const char *registry, + const char *repository, const char *username, + const char *password) { static const char * const auth_patterns[] = { "https://%s/v2/auth", @@ -484,35 +507,35 @@ static char *ocierofs_get_auth_token(struct erofs_oci *oci, const char *registry !strcmp(registry, DOCKER_REGISTRY); if (docker_reg) { service = "registry.docker.io"; - auth_header = ocierofs_get_auth_token_with_url(oci, + auth_header = ocierofs_get_auth_token_with_url(ctx, "https://auth.docker.io/token", service, repository, username, password); if (!IS_ERR(auth_header)) return auth_header; } - discovered_auth_url = ocierofs_discover_auth_endpoint(oci, registry, repository); + discovered_auth_url = ocierofs_discover_auth_endpoint(ctx, registry, repository); if (discovered_auth_url) { const char *api_registry, *auth_service; - struct erofs_oci_response resp = {}; + struct ocierofs_response resp = {}; char *test_url; CURLcode res; long http_code; - api_registry = (!strcmp(registry, DOCKER_REGISTRY)) ? DOCKER_API_REGISTRY : registry; + api_registry = ocierofs_get_api_registry(registry); if (asprintf(&test_url, "https://%s/v2/%s/manifests/nonexistent", api_registry, repository) >= 0) { - curl_easy_reset(oci->curl); - ocierofs_curl_setup_common_options(oci->curl); + curl_easy_reset(ctx->net.curl); + ocierofs_curl_setup_common_options(ctx->net.curl); - ocierofs_curl_setup_rq(oci->curl, test_url, - OCIEROFS_HTTP_HEAD, NULL, - NULL, NULL, - ocierofs_write_callback, &resp); + ocierofs_curl_setup_rq(ctx->net.curl, test_url, + OCIEROFS_HTTP_HEAD, NULL, + NULL, NULL, + ocierofs_write_callback, &resp); - res = curl_easy_perform(oci->curl); - curl_easy_getinfo(oci->curl, CURLINFO_RESPONSE_CODE, &http_code); + res = curl_easy_perform(ctx->net.curl); + curl_easy_getinfo(ctx->net.curl, CURLINFO_RESPONSE_CODE, &http_code); if (res == CURLE_OK && (http_code == 401 || http_code == 403 || http_code == 404) && resp.data) { @@ -521,14 +544,14 @@ static char *ocierofs_get_auth_token(struct erofs_oci *oci, const char *registry ocierofs_extract_www_auth_info(resp.data, &realm, &discovered_service, NULL); free(realm); } - free(resp.data); + ocierofs_response_cleanup(&resp); free(test_url); } auth_service = discovered_service ? discovered_service : service; - auth_header = ocierofs_get_auth_token_with_url(oci, discovered_auth_url, - auth_service, repository, - username, password); + auth_header = ocierofs_get_auth_token_with_url(ctx, discovered_auth_url, + auth_service, repository, + username, password); free(discovered_auth_url); free(discovered_service); if (!IS_ERR(auth_header)) @@ -541,9 +564,9 @@ static char *ocierofs_get_auth_token(struct erofs_oci *oci, const char *registry if (asprintf(&auth_url, auth_patterns[i], registry) < 0) continue; - auth_header = ocierofs_get_auth_token_with_url(oci, auth_url, - service, repository, - username, password); + auth_header = ocierofs_get_auth_token_with_url(ctx, auth_url, + service, repository, + username, password); free(auth_url); if (!IS_ERR(auth_header)) @@ -554,24 +577,24 @@ static char *ocierofs_get_auth_token(struct erofs_oci *oci, const char *registry return ERR_PTR(-ENOENT); } -static char *ocierofs_get_manifest_digest(struct erofs_oci *oci, - const char *registry, - const char *repository, const char *tag, - const char *platform, - const char *auth_header) +static char *ocierofs_get_manifest_digest(struct ocierofs_ctx *ctx, + const char *registry, + const char *repository, const char *tag, + const char *platform, + const char *auth_header) { - struct erofs_oci_request req = {}; - struct erofs_oci_response resp = {}; + struct ocierofs_request req = {}; + struct ocierofs_response resp = {}; json_object *root, *manifests, *manifest, *platform_obj, *arch_obj; json_object *os_obj, *digest_obj, *schema_obj, *media_type_obj; char *digest = NULL; const char *api_registry; int ret = 0, len, i; - if (!registry || !repository || !tag || !platform) + if (!registry || !repository || !tag) return ERR_PTR(-EINVAL); - api_registry = (!strcmp(registry, DOCKER_REGISTRY)) ? DOCKER_API_REGISTRY : registry; + api_registry = ocierofs_get_api_registry(registry); if (asprintf(&req.url, "https://%s/v2/%s/manifests/%s", api_registry, repository, tag) < 0) return ERR_PTR(-ENOMEM); @@ -581,10 +604,10 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci, req.headers = curl_slist_append(req.headers, "Accept: " DOCKER_MEDIATYPE_MANIFEST_LIST "," - OCI_MEDIATYPE_INDEX "," DOCKER_MEDIATYPE_MANIFEST_V1 "," - DOCKER_MEDIATYPE_MANIFEST_V2); + OCI_MEDIATYPE_INDEX "," OCI_MEDIATYPE_MANIFEST "," + DOCKER_MEDIATYPE_MANIFEST_V1 "," DOCKER_MEDIATYPE_MANIFEST_V2); - ret = ocierofs_request_perform(oci, &req, &resp); + ret = ocierofs_request_perform(ctx, &req, &resp); if (ret) goto out; @@ -612,8 +635,7 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci, if (json_object_object_get_ex(root, "mediaType", &media_type_obj)) { const char *media_type = json_object_get_string(media_type_obj); - if (!strcmp(media_type, DOCKER_MEDIATYPE_MANIFEST_V2) || - !strcmp(media_type, OCI_MEDIATYPE_MANIFEST)) { + if (ocierofs_is_manifest(media_type)) { digest = strdup(tag); ret = 0; goto out_json; @@ -631,9 +653,9 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci, manifest = json_object_array_get_idx(manifests, i); if (json_object_object_get_ex(manifest, "platform", - &platform_obj) && + &platform_obj) && json_object_object_get_ex(platform_obj, "architecture", - &arch_obj) && + &arch_obj) && json_object_object_get_ex(platform_obj, "os", &os_obj) && json_object_object_get_ex(manifest, "digest", &digest_obj)) { const char *arch = json_object_get_string(arch_obj); @@ -655,34 +677,47 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci, out_json: json_object_put(root); out: - free(resp.data); - if (req.headers) - curl_slist_free_all(req.headers); - free(req.url); - + ocierofs_response_cleanup(&resp); + ocierofs_request_cleanup(&req); return ret ? ERR_PTR(ret) : digest; } -static char **ocierofs_get_layers_info(struct erofs_oci *oci, - const char *registry, - const char *repository, - const char *digest, - const char *auth_header, - int *layer_count) +static void ocierofs_free_layers_info(struct ocierofs_layer_info **layers, int count) +{ + int i; + + if (!layers) + return; + + for (i = 0; i < count; i++) { + if (layers[i]) { + free(layers[i]->digest); + free(layers[i]->media_type); + free(layers[i]); + } + } + free(layers); +} + +static struct ocierofs_layer_info **ocierofs_fetch_layers_info(struct ocierofs_ctx *ctx, + const char *registry, + const char *repository, + const char *digest, + const char *auth_header, + int *layer_count) { - struct erofs_oci_request req = {}; - struct erofs_oci_response resp = {}; - json_object *root, *layers, *layer, *digest_obj; - char **layers_info = NULL; + struct ocierofs_request req = {}; + struct ocierofs_response resp = {}; + json_object *root, *layers, *layer, *digest_obj, *media_type_obj, *size_obj; + struct ocierofs_layer_info **layers_info = NULL; const char *api_registry; - int ret, len, i, j; + int ret, len, i; if (!registry || !repository || !digest || !layer_count) return ERR_PTR(-EINVAL); *layer_count = 0; - api_registry = (!strcmp(registry, DOCKER_REGISTRY) ? - DOCKER_API_REGISTRY : registry); + api_registry = ocierofs_get_api_registry(registry); if (asprintf(&req.url, "https://%s/v2/%s/manifests/%s", api_registry, repository, digest) < 0) @@ -694,7 +729,7 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci, req.headers = curl_slist_append(req.headers, "Accept: " OCI_MEDIATYPE_MANIFEST "," DOCKER_MEDIATYPE_MANIFEST_V2); - ret = ocierofs_request_perform(oci, &req, &resp); + ret = ocierofs_request_perform(ctx, &req, &resp); if (ret) goto out; @@ -725,7 +760,7 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci, goto out_json; } - layers_info = calloc(len, sizeof(char *)); + layers_info = calloc(len, sizeof(*layers_info)); if (!layers_info) { ret = -ENOMEM; goto out_json; @@ -740,329 +775,421 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci, goto out_free; } - layers_info[i] = strdup(json_object_get_string(digest_obj)); + layers_info[i] = calloc(1, sizeof(**layers_info)); if (!layers_info[i]) { ret = -ENOMEM; goto out_free; } + layers_info[i]->digest = strdup(json_object_get_string(digest_obj)); + if (!layers_info[i]->digest) { + ret = -ENOMEM; + goto out_free; + } + if (json_object_object_get_ex(layer, "mediaType", &media_type_obj)) + layers_info[i]->media_type = strdup(json_object_get_string(media_type_obj)); + else + layers_info[i]->media_type = NULL; + + if (json_object_object_get_ex(layer, "size", &size_obj)) + layers_info[i]->size = json_object_get_int64(size_obj); + else + layers_info[i]->size = 0; } *layer_count = len; json_object_put(root); - free(resp.data); - if (req.headers) - curl_slist_free_all(req.headers); - free(req.url); + ocierofs_response_cleanup(&resp); + ocierofs_request_cleanup(&req); return layers_info; out_free: - if (layers_info) { - for (j = 0; j < i; j++) - free(layers_info[j]); - } - free(layers_info); + ocierofs_free_layers_info(layers_info, i); out_json: json_object_put(root); out: - free(resp.data); - if (req.headers) - curl_slist_free_all(req.headers); - free(req.url); + ocierofs_response_cleanup(&resp); + ocierofs_request_cleanup(&req); return ERR_PTR(ret); } -static int ocierofs_extract_layer(struct erofs_oci *oci, struct erofs_importer *importer, - const char *digest, const char *auth_header) +static int ocierofs_process_tar_stream(struct erofs_importer *importer, int fd) { - struct erofs_oci_request req = {}; - struct erofs_oci_stream stream = {}; - const char *api_registry; - long http_code; + struct erofs_tarfile tarfile = {}; int ret; - stream = (struct erofs_oci_stream) { - .digest = digest, - .blobfd = erofs_tmpfile(), - }; - if (stream.blobfd < 0) { - erofs_err("failed to create temporary file for %s", digest); - return -errno; - } - - api_registry = (!strcmp(oci->params.registry, DOCKER_REGISTRY)) ? - DOCKER_API_REGISTRY : oci->params.registry; - if (asprintf(&req.url, "https://%s/v2/%s/blobs/%s", - api_registry, oci->params.repository, digest) == -1) { - ret = -ENOMEM; - goto out; - } - - if (auth_header && strstr(auth_header, "Bearer")) - req.headers = curl_slist_append(req.headers, auth_header); - - curl_easy_reset(oci->curl); - - ret = ocierofs_curl_setup_common_options(oci->curl); - if (ret) - goto out; - - ret = ocierofs_curl_setup_rq(oci->curl, req.url, OCIEROFS_HTTP_GET, - req.headers, - ocierofs_layer_write_callback, - &stream, NULL, NULL); - if (ret) - goto out; - - ret = ocierofs_curl_perform(oci->curl, &http_code); - if (ret) - goto out; - - if (http_code < 200 || http_code >= 300) { - erofs_err("HTTP request failed with code %ld", http_code); - ret = -EIO; - goto out; - } + init_list_head(&tarfile.global.xattrs); - if (lseek(stream.blobfd, 0, SEEK_SET) < 0) { - erofs_err("failed to seek to beginning of temp file: %s", - strerror(errno)); - ret = -errno; - goto out; - } - - memset(&stream.tarfile, 0, sizeof(stream.tarfile)); - init_list_head(&stream.tarfile.global.xattrs); - - ret = erofs_iostream_open(&stream.tarfile.ios, stream.blobfd, - EROFS_IOS_DECODER_GZIP); + ret = erofs_iostream_open(&tarfile.ios, fd, EROFS_IOS_DECODER_GZIP); if (ret) { erofs_err("failed to initialize tar stream: %s", erofs_strerror(ret)); - goto out; + return ret; } do { - ret = tarerofs_parse_tar(importer, &stream.tarfile); + ret = tarerofs_parse_tar(importer, &tarfile); /* Continue parsing until end of archive */ } while (!ret); - erofs_iostream_close(&stream.tarfile.ios); + erofs_iostream_close(&tarfile.ios); if (ret < 0 && ret != -ENODATA) { erofs_err("failed to process tar stream: %s", erofs_strerror(ret)); - goto out; + return ret; } - ret = 0; -out: - if (stream.blobfd >= 0) - close(stream.blobfd); - if (req.headers) - curl_slist_free_all(req.headers); - free(req.url); - return ret; + return 0; } -/** - * ocierofs_build_trees - Build file trees from OCI container image layers - * @importer: EROFS importer structure - * @oci: OCI client structure with configured parameters - * - * Extract and build file system trees from all layers of an OCI container - * image. - * - * Return: 0 on success, negative errno on failure - */ -int ocierofs_build_trees(struct erofs_importer *importer, struct erofs_oci *oci) +static int ocierofs_prepare_auth(struct ocierofs_ctx *ctx, + const char *username, + const char *password) { char *auth_header = NULL; - char *manifest_digest = NULL; - char **layers = NULL; - int layer_count = 0; - int ret, i; - - if (!importer || !oci) - return -EINVAL; + int ret = 0; + + ctx->net.using_basic = false; + free(ctx->net.auth_header); + ctx->net.auth_header = NULL; + + /* Try to get a Bearer token first (with credentials if provided) */ + auth_header = ocierofs_get_auth_token(ctx, + ctx->img.registry, + ctx->img.repository, + username, password); + if (!IS_ERR(auth_header)) { + ctx->net.auth_header = auth_header; + return 0; + } - if (oci->params.username && oci->params.password && - oci->params.username[0] && oci->params.password[0]) { - auth_header = ocierofs_get_auth_token(oci, - oci->params.registry, - oci->params.repository, - oci->params.username, - oci->params.password); - if (IS_ERR(auth_header)) { - auth_header = NULL; - ret = ocierofs_curl_setup_basic_auth(oci->curl, - oci->params.username, - oci->params.password); - if (ret) - goto out; - } - } else { - auth_header = ocierofs_get_auth_token(oci, - oci->params.registry, - oci->params.repository, - NULL, NULL); - if (IS_ERR(auth_header)) - auth_header = NULL; + /* If token failed but credentials are provided, fall back to Basic */ + if (username && password && *username && *password) { + ret = ocierofs_curl_setup_basic_auth(ctx->net.curl, + username, password); + if (ret) + return ret; + ctx->net.using_basic = true; } + return 0; +} - manifest_digest = ocierofs_get_manifest_digest(oci, oci->params.registry, - oci->params.repository, - oci->params.tag, - oci->params.platform, - auth_header); - if (IS_ERR(manifest_digest)) { - ret = PTR_ERR(manifest_digest); +static int ocierofs_prepare_layers(struct ocierofs_ctx *ctx, + const struct ocierofs_config *config) +{ + int ret; + + ret = ocierofs_prepare_auth(ctx, config ? config->username : NULL, + config ? config->password : NULL); + if (ret) + return ret; + + ctx->img.manifest_digest = ocierofs_get_manifest_digest(ctx, ctx->img.registry, + ctx->img.repository, + ctx->img.tag, + ctx->img.platform, + ctx->net.auth_header); + if (IS_ERR(ctx->img.manifest_digest)) { + ret = PTR_ERR(ctx->img.manifest_digest); erofs_err("failed to get manifest digest: %s", erofs_strerror(ret)); + ctx->img.manifest_digest = NULL; goto out_auth; } - layers = ocierofs_get_layers_info(oci, oci->params.registry, - oci->params.repository, - manifest_digest, auth_header, - &layer_count); - if (IS_ERR(layers)) { - ret = PTR_ERR(layers); + ctx->img.layers = ocierofs_fetch_layers_info(ctx, ctx->img.registry, + ctx->img.repository, + ctx->img.manifest_digest, ctx->net.auth_header, + &ctx->img.layer_count); + if (IS_ERR(ctx->img.layers)) { + ret = PTR_ERR(ctx->img.layers); erofs_err("failed to get image layers: %s", erofs_strerror(ret)); + ctx->img.layers = NULL; goto out_manifest; } - if (oci->params.layer_index >= 0) { - if (oci->params.layer_index >= layer_count) { + if (ctx->img.layer_index >= 0) { + if (ctx->img.layer_index >= ctx->img.layer_count) { erofs_err("layer index %d exceeds available layers (%d)", - oci->params.layer_index, layer_count); + ctx->img.layer_index, ctx->img.layer_count); ret = -EINVAL; goto out_layers; } - layer_count = 1; - i = oci->params.layer_index; + ctx->img.layer_count = 1; } else { - i = 0; + ctx->img.layer_index = 0; } - while (i < layer_count) { - char *trimmed = erofs_trim_for_progressinfo(layers[i], - sizeof("Extracting layer ...") - 1); - erofs_update_progressinfo("Extracting layer %d: %s ...", i, - trimmed); - free(trimmed); - ret = ocierofs_extract_layer(oci, importer, layers[i], - auth_header); - if (ret) { - erofs_err("failed to extract layer %d: %s", i, - erofs_strerror(ret)); - break; - } - i++; - } + return 0; + out_layers: - for (i = 0; i < layer_count; i++) - free(layers[i]); - free(layers); + free(ctx->img.layers); + ctx->img.layers = NULL; out_manifest: - free(manifest_digest); + free(ctx->img.manifest_digest); + ctx->img.manifest_digest = NULL; out_auth: - free(auth_header); - - if (oci->params.username && oci->params.password && - oci->params.username[0] && oci->params.password[0] && - !auth_header) { - ocierofs_curl_clear_auth(oci->curl); - } -out: + free(ctx->net.auth_header); + ctx->net.auth_header = NULL; + if (ctx->net.using_basic) + ocierofs_curl_clear_auth(ctx); return ret; } -/** - * ocierofs_init - Initialize OCI client with default parameters - * @oci: OCI client structure to initialize - * - * Initialize OCI client structure, set up CURL handle, and configure - * default parameters including platform (linux/amd64), registry - * (registry-1.docker.io), and tag (latest). - * - * Return: 0 on success, negative errno on failure - */ -int ocierofs_init(struct erofs_oci *oci) +static int ocierofs_parse_ref(struct ocierofs_ctx *ctx, const char *ref_str) { - if (!oci) + char *slash, *colon, *dot; + const char *repo_part; + size_t len; + char *tmp; + + if (!ctx || !ref_str) return -EINVAL; - *oci = (struct erofs_oci){}; - oci->curl = curl_easy_init(); - if (!oci->curl) + slash = strchr(ref_str, '/'); + if (slash) { + dot = strchr(ref_str, '.'); + if (dot && dot < slash) { + len = slash - ref_str; + tmp = strndup(ref_str, len); + if (!tmp) + return -ENOMEM; + free(ctx->img.registry); + ctx->img.registry = tmp; + repo_part = slash + 1; + } else { + repo_part = ref_str; + } + } else { + repo_part = ref_str; + } + + colon = strchr(repo_part, ':'); + if (colon) { + len = colon - repo_part; + tmp = strndup(repo_part, len); + if (!tmp) + return -ENOMEM; + + if (!strchr(tmp, '/') && + (!strcmp(ctx->img.registry, DOCKER_API_REGISTRY) || + !strcmp(ctx->img.registry, DOCKER_REGISTRY))) { + char *full_repo; + + if (asprintf(&full_repo, "library/%s", tmp) == -1) { + free(tmp); + return -ENOMEM; + } + free(tmp); + tmp = full_repo; + } + free(ctx->img.repository); + ctx->img.repository = tmp; + + free(ctx->img.tag); + ctx->img.tag = strdup(colon + 1); + if (!ctx->img.tag) + return -ENOMEM; + } else { + tmp = strdup(repo_part); + if (!tmp) + return -ENOMEM; + + if (!strchr(tmp, '/') && + (!strcmp(ctx->img.registry, DOCKER_API_REGISTRY) || + !strcmp(ctx->img.registry, DOCKER_REGISTRY))) { + + char *full_repo; + if (asprintf(&full_repo, "library/%s", tmp) == -1) { + free(tmp); + return -ENOMEM; + } + free(tmp); + tmp = full_repo; + } + free(ctx->img.repository); + ctx->img.repository = tmp; + } + return 0; +} + +int ocierofs_init(struct ocierofs_ctx *ctx, const struct ocierofs_config *config) +{ + int ret; + + ctx->net.curl = curl_easy_init(); + if (!ctx->net.curl) return -EIO; - if (ocierofs_curl_setup_common_options(oci->curl)) { - ocierofs_cleanup(oci); + if (ocierofs_curl_setup_common_options(ctx->net.curl)) return -EIO; - } - if (erofs_oci_params_set_string(&oci->params.platform, - "linux/amd64") || - erofs_oci_params_set_string(&oci->params.registry, - DOCKER_API_REGISTRY) || - erofs_oci_params_set_string(&oci->params.tag, "latest")) { - ocierofs_cleanup(oci); + ctx->img.layer_index = -1; + ctx->img.registry = strdup("registry-1.docker.io"); + ctx->img.tag = strdup("latest"); + if (config && config->platform) + ctx->img.platform = strdup(config->platform); + else + ctx->img.platform = strdup("linux/amd64"); + if (!ctx->img.registry || !ctx->img.tag || !ctx->img.platform) return -ENOMEM; - } - oci->params.layer_index = -1; /* -1 means extract all layers */ + + if (config && config->layer_index >= 0) + ctx->img.layer_index = config->layer_index; + + ret = ocierofs_parse_ref(ctx, config->image_ref); + if (ret) + return ret; + + ret = ocierofs_prepare_layers(ctx, config); + if (ret) + return ret; + return 0; } -/** - * ocierofs_cleanup - Clean up OCI client and free allocated resources - * @oci: OCI client structure to clean up - * - * Clean up CURL handle, free all allocated string parameters, and - * reset the OCI client structure to a clean state. - */ -void ocierofs_cleanup(struct erofs_oci *oci) +static int ocierofs_download_blob_to_fd(struct ocierofs_ctx *ctx, + const char *digest, + const char *auth_header, + int outfd) { - if (!oci) - return; + struct ocierofs_request req = {}; + struct ocierofs_stream stream = {}; + const char *api_registry; + long http_code; + int ret; + + stream = (struct ocierofs_stream) { + .digest = digest, + .blobfd = outfd, + }; - if (oci->curl) { - curl_easy_cleanup(oci->curl); - oci->curl = NULL; + api_registry = ocierofs_get_api_registry(ctx->img.registry); + if (asprintf(&req.url, "https://%s/v2/%s/blobs/%s", + api_registry, ctx->img.repository, digest) == -1) + return -ENOMEM; + + if (auth_header && strstr(auth_header, "Bearer")) + req.headers = curl_slist_append(req.headers, auth_header); + + curl_easy_reset(ctx->net.curl); + + ret = ocierofs_curl_setup_common_options(ctx->net.curl); + if (ret) + goto out; + + ret = ocierofs_curl_setup_rq(ctx->net.curl, req.url, OCIEROFS_HTTP_GET, + req.headers, + ocierofs_layer_write_callback, + &stream, NULL, NULL); + if (ret) + goto out; + + ret = ocierofs_curl_perform(ctx->net.curl, &http_code); + if (ret) + goto out; + + if (http_code < 200 || http_code >= 300) { + erofs_err("HTTP request failed with code %ld", http_code); + ret = -EIO; + goto out; + } + ret = 0; +out: + ocierofs_request_cleanup(&req); + return ret; +} + +static int ocierofs_extract_layer(struct ocierofs_ctx *ctx, + const char *digest, const char *auth_header) +{ + struct ocierofs_stream stream = {}; + int ret; + + stream = (struct ocierofs_stream) { + .digest = digest, + .blobfd = erofs_tmpfile(), + }; + if (stream.blobfd < 0) { + erofs_err("failed to create temporary file for %s", digest); + return -errno; + } + + ret = ocierofs_download_blob_to_fd(ctx, digest, auth_header, stream.blobfd); + if (ret) + goto out; + + if (lseek(stream.blobfd, 0, SEEK_SET) < 0) { + erofs_err("failed to seek to beginning of temp file: %s", + strerror(errno)); + ret = -errno; + goto out; } - free(oci->params.registry); - free(oci->params.repository); - free(oci->params.tag); - free(oci->params.platform); - free(oci->params.username); - free(oci->params.password); - - oci->params.registry = NULL; - oci->params.repository = NULL; - oci->params.tag = NULL; - oci->params.platform = NULL; - oci->params.username = NULL; - oci->params.password = NULL; + return stream.blobfd; + +out: + if (stream.blobfd >= 0) + close(stream.blobfd); + return ret; } -int erofs_oci_params_set_string(char **field, const char *value) +static void ocierofs_ctx_cleanup(struct ocierofs_ctx *ctx) { - char *new_value; + if (!ctx) + return; - if (!field) - return -EINVAL; + if (ctx->net.curl) { + curl_easy_cleanup(ctx->net.curl); + ctx->net.curl = NULL; + } + free(ctx->net.auth_header); + ctx->net.auth_header = NULL; + + ocierofs_free_layers_info(ctx->img.layers, ctx->img.layer_count); + free(ctx->img.registry); + free(ctx->img.repository); + free(ctx->img.tag); + free(ctx->img.platform); + free(ctx->img.manifest_digest); + memset(&ctx->img, 0, sizeof(ctx->img)); +} - if (!value) { - free(*field); - *field = NULL; - return 0; +int ocierofs_build_trees(struct erofs_importer *importer, + const struct ocierofs_config *config) +{ + struct ocierofs_ctx ctx = {}; + int ret, i; + + ret = ocierofs_init(&ctx, config); + if (ret) { + ocierofs_ctx_cleanup(&ctx); + return ret; } - new_value = strdup(value); - if (!new_value) - return -ENOMEM; + i = ctx.img.layer_index; + while (i < ctx.img.layer_count) { + char *trimmed = erofs_trim_for_progressinfo(ctx.img.layers[i]->digest, + sizeof("Extracting layer ...") - 1); + erofs_update_progressinfo("Extracting layer %d: %s ...", i, + trimmed); + free(trimmed); + int fd = ocierofs_extract_layer(&ctx, ctx.img.layers[i]->digest, + ctx.net.auth_header); + if (fd < 0) { + erofs_err("failed to extract layer %d: %s", i, + erofs_strerror(fd)); + break; + } + ret = ocierofs_process_tar_stream(importer, fd); + close(fd); + if (ret) { + erofs_err("failed to process tar stream for layer %d: %s", i, + erofs_strerror(ret)); + break; + } + i++; + } - free(*field); - *field = new_value; - return 0; + ocierofs_ctx_cleanup(&ctx); + return ret; } diff --git a/mkfs/main.c b/mkfs/main.c index bc895f1..5e8da7a 100644 --- a/mkfs/main.c +++ b/mkfs/main.c @@ -150,18 +150,18 @@ static void usage(int argc, char **argv) * "0-9,100-109" instead of a continuous "0-109", and to * state what those two subranges respectively mean. */ printf("%s [,level=<0-9,100-109>]\t0-9=normal, 100-109=extreme (default=%i)\n", - spaces, s->c->default_level); + spaces, s->c->default_level); else printf("%s [,level=<0-%i>]\t\t(default=%i)\n", - spaces, s->c->best_level, s->c->default_level); + spaces, s->c->best_level, s->c->default_level); } if (s->c->setdictsize) { if (s->c->default_dictsize) printf("%s [,dictsize=<dictsize>]\t(default=%u, max=%u)\n", - spaces, s->c->default_dictsize, s->c->max_dictsize); + spaces, s->c->default_dictsize, s->c->max_dictsize); else printf("%s [,dictsize=<dictsize>]\t(default=<auto>, max=%u)\n", - spaces, s->c->max_dictsize); + spaces, s->c->max_dictsize); } } printf( @@ -272,7 +272,7 @@ static struct erofs_s3 s3cfg; #endif #ifdef OCIEROFS_ENABLED -static struct erofs_oci ocicfg; +static struct ocierofs_config ocicfg; static char *mkfs_oci_options; #endif @@ -689,21 +689,14 @@ static int mkfs_parse_s3_cfg(char *cfg_str) #endif #ifdef OCIEROFS_ENABLED - - -/** - * mkfs_parse_oci_options - Parse comma-separated OCI options string +/* + * mkfs_parse_oci_options - Parse OCI options for mkfs tool + * @cfg: OCI configuration structure to update * @options_str: comma-separated options string - * - * Parse OCI options string containing comma-separated key=value pairs. - * Supported options include platform, layer, username, and password. - * - * Return: 0 on success, negative errno on failure */ -static int mkfs_parse_oci_options(char *options_str) +static int mkfs_parse_oci_options(struct ocierofs_config *oci_cfg, char *options_str) { char *opt, *q, *p; - int ret; if (!options_str) return 0; @@ -717,41 +710,38 @@ static int mkfs_parse_oci_options(char *options_str) p = strstr(opt, "platform="); if (p) { p += strlen("platform="); - ret = erofs_oci_params_set_string(&ocicfg.params.platform, p); - if (ret) { - erofs_err("failed to set platform"); - return ret; - } + free(oci_cfg->platform); + oci_cfg->platform = strdup(p); + if (!oci_cfg->platform) + return -ENOMEM; } else { p = strstr(opt, "layer="); if (p) { p += strlen("layer="); - ocicfg.params.layer_index = atoi(p); - if (ocicfg.params.layer_index < 0) { + oci_cfg->layer_index = atoi(p); + if (oci_cfg->layer_index < 0) { erofs_err("invalid layer index %d", - ocicfg.params.layer_index); + oci_cfg->layer_index); return -EINVAL; } } else { p = strstr(opt, "username="); if (p) { p += strlen("username="); - ret = erofs_oci_params_set_string(&ocicfg.params.username, p); - if (ret) { - erofs_err("failed to set username"); - return ret; - } + free(oci_cfg->username); + oci_cfg->username = strdup(p); + if (!oci_cfg->username) + return -ENOMEM; } else { p = strstr(opt, "password="); if (p) { p += strlen("password="); - ret = erofs_oci_params_set_string(&ocicfg.params.password, p); - if (ret) { - erofs_err("failed to set password"); - return ret; - } + free(oci_cfg->password); + oci_cfg->password = strdup(p); + if (!oci_cfg->password) + return -ENOMEM; } else { - erofs_err("invalid --oci value %s", opt); + erofs_err("mkfs: invalid --oci value %s", opt); return -EINVAL; } } @@ -763,125 +753,6 @@ static int mkfs_parse_oci_options(char *options_str) return 0; } - -/** - * mkfs_parse_oci_ref - Parse OCI image reference string - * @ref_str: OCI image reference in various formats - * - * Parse OCI image reference which can be in formats: - * - registry.example.com/namespace/repo:tag - * - namespace/repo:tag (uses default registry) - * - repo:tag (adds library/ prefix for Docker Hub) - * - repo (uses default tag "latest") - * - * Return: 0 on success, negative errno on failure - */ -static int mkfs_parse_oci_ref(const char *ref_str) -{ - char *slash, *colon, *dot; - const char *repo_part; - size_t len; - - slash = strchr(ref_str, '/'); - if (slash) { - dot = strchr(ref_str, '.'); - if (dot && dot < slash) { - char *registry_str; - - len = slash - ref_str; - registry_str = strndup(ref_str, len); - - if (!registry_str) { - erofs_err("failed to allocate memory for registry"); - return -ENOMEM; - } - if (erofs_oci_params_set_string(&ocicfg.params.registry, - registry_str)) { - free(registry_str); - erofs_err("failed to set registry"); - return -ENOMEM; - } - free(registry_str); - repo_part = slash + 1; - } else { - repo_part = ref_str; - } - } else { - repo_part = ref_str; - } - - colon = strchr(repo_part, ':'); - if (colon) { - char *repo_str; - - len = colon - repo_part; - repo_str = strndup(repo_part, len); - - if (!repo_str) { - erofs_err("failed to allocate memory for repository"); - return -ENOMEM; - } - - if (!strchr(repo_str, '/') && - (!strcmp(ocicfg.params.registry, DOCKER_API_REGISTRY) || - !strcmp(ocicfg.params.registry, DOCKER_REGISTRY))) { - char *full_repo; - - if (asprintf(&full_repo, "library/%s", repo_str) == -1) { - free(repo_str); - erofs_err("failed to allocate memory for full repository name"); - return -ENOMEM; - } - free(repo_str); - repo_str = full_repo; - } - - if (erofs_oci_params_set_string(&ocicfg.params.repository, - repo_str)) { - free(repo_str); - erofs_err("failed to set repository"); - return -ENOMEM; - } - free(repo_str); - - if (erofs_oci_params_set_string(&ocicfg.params.tag, - colon + 1)) { - erofs_err("failed to set tag"); - return -ENOMEM; - } - } else { - char *repo_str = strdup(repo_part); - - if (!repo_str) { - erofs_err("failed to allocate memory for repository"); - return -ENOMEM; - } - - if (!strchr(repo_str, '/') && - (!strcmp(ocicfg.params.registry, DOCKER_API_REGISTRY) || - !strcmp(ocicfg.params.registry, DOCKER_REGISTRY))) { - char *full_repo; - - if (asprintf(&full_repo, "library/%s", repo_str) == -1) { - free(repo_str); - erofs_err("failed to allocate memory for full repository name"); - return -ENOMEM; - } - free(repo_str); - repo_str = full_repo; - } - - if (erofs_oci_params_set_string(&ocicfg.params.repository, - repo_str)) { - free(repo_str); - erofs_err("failed to set repository"); - return -ENOMEM; - } - free(repo_str); - } - - return 0; -} #endif static int mkfs_parse_one_compress_alg(char *alg, @@ -1951,16 +1822,12 @@ int main(int argc, char **argv) #endif #ifdef OCIEROFS_ENABLED } else if (source_mode == EROFS_MKFS_SOURCE_OCI) { - err = ocierofs_init(&ocicfg); - if (err) - goto exit; + ocicfg.layer_index = -1; - err = mkfs_parse_oci_options(mkfs_oci_options); - if (err) - goto exit; - err = mkfs_parse_oci_ref(cfg.c_src_path); + err = mkfs_parse_oci_options(&ocicfg, mkfs_oci_options); if (err) goto exit; + ocicfg.image_ref = cfg.c_src_path; if (incremental_mode || dataimport_mode == EROFS_MKFS_DATA_IMPORT_RVSP || @@ -2035,9 +1902,6 @@ exit: erofs_blob_exit(); erofs_xattr_cleanup_name_prefixes(); erofs_rebuild_cleanup(); -#ifdef OCIEROFS_ENABLED - ocierofs_cleanup(&ocicfg); -#endif erofs_diskbuf_exit(); if (source_mode == EROFS_MKFS_SOURCE_TAR) { erofs_iostream_close(&erofstar.ios); -- 2.51.0 ^ permalink raw reply related [flat|nested] 19+ messages in thread
* Re: [PATCH v3 1/2] erofs-utils: refactor OCI code for better modularity 2025-09-03 8:29 ` [PATCH v3 1/2] erofs-utils: refactor OCI code for better modularity ChengyuZhu6 @ 2025-09-03 8:47 ` Gao Xiang 0 siblings, 0 replies; 19+ messages in thread From: Gao Xiang @ 2025-09-03 8:47 UTC (permalink / raw) To: ChengyuZhu6, linux-erofs; +Cc: xiang, Chengyu Zhu On 2025/9/3 16:29, ChengyuZhu6 wrote: > From: Chengyu Zhu <hudsonzhu@tencent.com> > > Refactor OCI code to improve code organization and maintainability: > > Key changes: > - Add `ocierofs_get_api_registry()` and unify API endpoint selection. > - Implement Bearer token discovery with Basic fallback; cache auth header. > - Parse layer metadata (digest, mediaType, size) and add a proper free helper. > - Split blob download from tar processing; process tar via a temp fd. > - Rework init/teardown into `ocierofs_init()` and `ocierofs_ctx_cleanup()`. > - Update mkfs to use `struct ocierofs_config` and new `--oci` parsing. > > Signed-off-by: Chengyu Zhu <hudsonzhu@tencent.com> > --- > lib/liberofs_oci.h | 84 ++--- > lib/remotes/oci.c | 885 ++++++++++++++++++++++++++------------------- > mkfs/main.c | 192 ++-------- > 3 files changed, 566 insertions(+), 595 deletions(-) > > diff --git a/lib/liberofs_oci.h b/lib/liberofs_oci.h > index 3a8108b..d119a2b 100644 > --- a/lib/liberofs_oci.h > +++ b/lib/liberofs_oci.h > @@ -8,85 +8,65 @@ > > #include <stdbool.h> > > -#define DOCKER_REGISTRY "docker.io" > -#define DOCKER_API_REGISTRY "registry-1.docker.io" > - > #ifdef __cplusplus > extern "C" { > #endif > > -struct erofs_inode; > -struct CURL; > struct erofs_importer; > > -/** > - * struct erofs_oci_params - OCI configuration parameters > - * @registry: registry hostname (e.g., "registry-1.docker.io") > - * @repository: image repository (e.g., "library/ubuntu") > - * @tag: image tag or digest (e.g., "latest" or sha256:...) > +/* > + * struct ocierofs_config - OCI configuration structure > + * @image_ref: OCI image reference (e.g., "ubuntu:latest", "myregistry.com/app:v1.0") > * @platform: target platform in "os/arch" format (e.g., "linux/amd64") > * @username: username for authentication (optional) > * @password: password for authentication (optional) > * @layer_index: specific layer to extract (-1 for all layers) > * > - * Configuration structure for OCI image parameters including registry > - * location, image identification, platform specification, and authentication > - * credentials. > */ > -struct erofs_oci_params { > - char *registry; > - char *repository; > - char *tag; > +struct ocierofs_config { > + char *image_ref; > char *platform; > char *username; > char *password; > int layer_index; > }; > > -/** > - * struct erofs_oci - Combined OCI client structure > - * @curl: CURL handle for HTTP requests > - * @params: OCI configuration parameters > - * > - * Main OCI client structure combining CURL HTTP client with > - * OCI-specific configuration parameters. > - */ > -struct erofs_oci { > - struct CURL *curl; > - struct erofs_oci_params params; > +struct ocierofs_layer_info { > + char *digest; > + char *media_type; > + u64 size; > }; > > -/* > - * ocierofs_init - Initialize OCI client with default parameters > - * @oci: OCI client structure to initialize > - * > - * Return: 0 on success, negative errno on failure > - */ > -int ocierofs_init(struct erofs_oci *oci); > +struct ocierofs_ctx { > + struct { > + struct CURL *curl; > + char *auth_header; > + bool using_basic; > + } net; Is it necessary to use two anonymous structures here? Could we just unfold those fields? ... > diff --git a/mkfs/main.c b/mkfs/main.c > index bc895f1..5e8da7a 100644 > --- a/mkfs/main.c > +++ b/mkfs/main.c > @@ -150,18 +150,18 @@ static void usage(int argc, char **argv) > * "0-9,100-109" instead of a continuous "0-109", and to > * state what those two subranges respectively mean. */ > printf("%s [,level=<0-9,100-109>]\t0-9=normal, 100-109=extreme (default=%i)\n", > - spaces, s->c->default_level); > + spaces, s->c->default_level); > else > printf("%s [,level=<0-%i>]\t\t(default=%i)\n", > - spaces, s->c->best_level, s->c->default_level); > + spaces, s->c->best_level, s->c->default_level); > } > if (s->c->setdictsize) { > if (s->c->default_dictsize) > printf("%s [,dictsize=<dictsize>]\t(default=%u, max=%u)\n", > - spaces, s->c->default_dictsize, s->c->max_dictsize); > + spaces, s->c->default_dictsize, s->c->max_dictsize); > else > printf("%s [,dictsize=<dictsize>]\t(default=<auto>, max=%u)\n", > - spaces, s->c->max_dictsize); > + spaces, s->c->max_dictsize); Are those changes unnecessary? > } > } > printf( > @@ -272,7 +272,7 @@ static struct erofs_s3 s3cfg; > #endif > > #ifdef OCIEROFS_ENABLED > -static struct erofs_oci ocicfg; > +static struct ocierofs_config ocicfg; > static char *mkfs_oci_options; > #endif > > @@ -689,21 +689,14 @@ static int mkfs_parse_s3_cfg(char *cfg_str) > #endif > > #ifdef OCIEROFS_ENABLED > - > - > -/** > - * mkfs_parse_oci_options - Parse comma-separated OCI options string > +/* > + * mkfs_parse_oci_options - Parse OCI options for mkfs tool > + * @cfg: OCI configuration structure to update > * @options_str: comma-separated options string > - * > - * Parse OCI options string containing comma-separated key=value pairs. > - * Supported options include platform, layer, username, and password. > - * > - * Return: 0 on success, negative errno on failure > */ > -static int mkfs_parse_oci_options(char *options_str) > +static int mkfs_parse_oci_options(struct ocierofs_config *oci_cfg, char *options_str) > { > char *opt, *q, *p; > - int ret; > > if (!options_str) > return 0; > @@ -717,41 +710,38 @@ static int mkfs_parse_oci_options(char *options_str) > p = strstr(opt, "platform="); > if (p) { > p += strlen("platform="); > - ret = erofs_oci_params_set_string(&ocicfg.params.platform, p); > - if (ret) { > - erofs_err("failed to set platform"); > - return ret; > - } > + free(oci_cfg->platform); > + oci_cfg->platform = strdup(p); > + if (!oci_cfg->platform) > + return -ENOMEM; > } else { > p = strstr(opt, "layer="); > if (p) { > p += strlen("layer="); > - ocicfg.params.layer_index = atoi(p); > - if (ocicfg.params.layer_index < 0) { > + oci_cfg->layer_index = atoi(p); > + if (oci_cfg->layer_index < 0) { Could you just use `strtoul` since it can check invalid arguments better? > erofs_err("invalid layer index %d", > - ocicfg.params.layer_index); > + oci_cfg->layer_index); > return -EINVAL; > } > } else { > p = strstr(opt, "username="); > if (p) { > p += strlen("username="); > - ret = erofs_oci_params_set_string(&ocicfg.params.username, p); > - if (ret) { > - erofs_err("failed to set username"); > - return ret; > - } > + free(oci_cfg->username); > + oci_cfg->username = strdup(p); > + if (!oci_cfg->username) > + return -ENOMEM; > } else { > p = strstr(opt, "password="); > if (p) { > p += strlen("password="); > - ret = erofs_oci_params_set_string(&ocicfg.params.password, p); > - if (ret) { > - erofs_err("failed to set password"); > - return ret; > - } > + free(oci_cfg->password); > + oci_cfg->password = strdup(p); > + if (!oci_cfg->password) > + return -ENOMEM; > } else { > - erofs_err("invalid --oci value %s", opt); > + erofs_err("mkfs: invalid --oci value %s", opt); `mkfs:` prefix is unneeded. Thanks, Gao Xiang ^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH v3 2/2] erofs-utils: add NBD-backed OCI image mounting 2025-09-03 8:29 ` [PATCH v3 0/2] erofs-utils: refactor OCI and add NBD-backed OCI image mounting ChengyuZhu6 2025-09-03 8:29 ` [PATCH v3 1/2] erofs-utils: refactor OCI code for better modularity ChengyuZhu6 @ 2025-09-03 8:29 ` ChengyuZhu6 1 sibling, 0 replies; 19+ messages in thread From: ChengyuZhu6 @ 2025-09-03 8:29 UTC (permalink / raw) To: linux-erofs; +Cc: xiang, hsiangkao, Chengyu Zhu From: Chengyu Zhu <hudsonzhu@tencent.com> - Add HTTP range downloads for OCI blobs - Introduce ocierofs_iostream for virtual file I/O - Add --oci option for OCI image mounting with NBD backend New mount.erofs -t erofs.nbd option: --oci=[option] source-image mountpoint Supported options: - platform=os/arch (default: linux/amd64) - layer=N (extract specific layer, default: all layers) - username/password (basic authentication) e.g.: ./mount/mount.erofs -t erofs.nbd --oci=platform=linux/amd64 \ quay.io/chengyuzhu6/golang:1.22.8-erofs /tmp/test/ Signed-off-by: Chengyu Zhu <hudsonzhu@tencent.com> --- lib/liberofs_oci.h | 19 +++- lib/remotes/oci.c | 242 ++++++++++++++++++++++++++++++++++++++++++++- mount/Makefile.am | 3 +- mount/main.c | 236 +++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 493 insertions(+), 7 deletions(-) diff --git a/lib/liberofs_oci.h b/lib/liberofs_oci.h index d119a2b..f35228c 100644 --- a/lib/liberofs_oci.h +++ b/lib/liberofs_oci.h @@ -7,7 +7,6 @@ #define __EROFS_OCI_H #include <stdbool.h> - #ifdef __cplusplus extern "C" { #endif @@ -56,7 +55,11 @@ struct ocierofs_ctx { } img; }; -int ocierofs_init(struct ocierofs_ctx *ctx, const struct ocierofs_config *config); +struct ocierofs_iostream { + struct ocierofs_ctx *ctx; + struct erofs_vfile vf; + u64 offset; +}; /* * ocierofs_build_trees - Build file trees from an OCI container image @@ -65,8 +68,16 @@ int ocierofs_init(struct ocierofs_ctx *ctx, const struct ocierofs_config *config * * Return: 0 on success, negative errno on failure */ -int ocierofs_build_trees(struct erofs_importer *importer, - const struct ocierofs_config *cfg); +int ocierofs_build_trees(struct erofs_importer *importer, const struct ocierofs_config *cfg); + +int ocierofs_is_erofs_native_image(struct ocierofs_ctx *ctx); + +int ocierofs_init(struct ocierofs_ctx *ctx, const struct ocierofs_config *config); + +void ocierofs_ctx_cleanup(struct ocierofs_ctx *ctx); + +int ocierofs_iostream_open(struct ocierofs_iostream *oci_iostream, struct ocierofs_ctx *oci_ctx); +void ocierofs_iostream_close(struct ocierofs_iostream *oci_iostream); #ifdef __cplusplus } diff --git a/lib/remotes/oci.c b/lib/remotes/oci.c index 01f1e24..8750772 100644 --- a/lib/remotes/oci.c +++ b/lib/remotes/oci.c @@ -33,6 +33,9 @@ #define OCI_MEDIATYPE_MANIFEST "application/vnd.oci.image.manifest.v1+json" #define OCI_MEDIATYPE_INDEX "application/vnd.oci.image.index.v1+json" +/* Erofs Native Layer Media Type */ +#define EROFS_MEDIATYPE "application/vnd.erofs" + struct ocierofs_request { char *url; struct curl_slist *headers; @@ -1133,7 +1136,7 @@ out: return ret; } -static void ocierofs_ctx_cleanup(struct ocierofs_ctx *ctx) +void ocierofs_ctx_cleanup(struct ocierofs_ctx *ctx) { if (!ctx) return; @@ -1193,3 +1196,240 @@ int ocierofs_build_trees(struct erofs_importer *importer, ocierofs_ctx_cleanup(&ctx); return ret; } + +static int ocierofs_download_blob_range(struct ocierofs_ctx *ctx, off_t offset, size_t length, + void **out_buf, size_t *out_size) +{ + struct ocierofs_request req = {}; + struct ocierofs_response resp = {}; + const char *api_registry; + char rangehdr[64]; + long http_code = 0; + int ret; + int index = ctx->img.layer_index; + + if (offset < 0) + return -EINVAL; + + api_registry = ocierofs_get_api_registry(ctx->img.registry); + if (asprintf(&req.url, "https://%s/v2/%s/blobs/%s", + api_registry, ctx->img.repository, ctx->img.layers[index]->digest) == -1) + return -ENOMEM; + + if (length) + snprintf(rangehdr, sizeof(rangehdr), "Range: bytes=%lld-%lld", + (long long)offset, (long long)(offset + (off_t)length - 1)); + else + snprintf(rangehdr, sizeof(rangehdr), "Range: bytes=%lld-", + (long long)offset); + + if (ctx->net.auth_header && strstr(ctx->net.auth_header, "Bearer")) + req.headers = curl_slist_append(req.headers, ctx->net.auth_header); + req.headers = curl_slist_append(req.headers, rangehdr); + + curl_easy_reset(ctx->net.curl); + + ret = ocierofs_curl_setup_common_options(ctx->net.curl); + if (ret) + goto out; + + ret = ocierofs_curl_setup_rq(ctx->net.curl, req.url, OCIEROFS_HTTP_GET, + req.headers, + ocierofs_write_callback, + &resp, NULL, NULL); + if (ret) + goto out; + + ret = ocierofs_curl_perform(ctx->net.curl, &http_code); + if (ret) + goto out; + + if (http_code == 206) { + *out_buf = resp.data; + *out_size = resp.size; + resp.data = NULL; + ret = 0; + } else if (http_code == 200) { + if (offset == 0) { + *out_buf = resp.data; + *out_size = resp.size; + resp.data = NULL; + ret = 0; + } else { + if (offset < resp.size) { + size_t available = resp.size - offset; + size_t copy_size = length ? min_t(size_t, length, available) : available; + + *out_buf = malloc(copy_size); + if (!*out_buf) { + ret = -ENOMEM; + goto out; + } + memcpy(*out_buf, resp.data + offset, copy_size); + *out_size = copy_size; + ret = 0; + } else { + *out_buf = NULL; + *out_size = 0; + ret = 0; + } + } + } else { + erofs_err("HTTP range request failed with code %ld", http_code); + ret = -EIO; + } + +out: + if (req.headers) + curl_slist_free_all(req.headers); + free(req.url); + free(resp.data); + return ret; +} + +static ssize_t ocierofs_io_pread(struct erofs_vfile *vf, void *buf, size_t len, u64 offset) +{ + struct ocierofs_iostream *oci_iostream = *(struct ocierofs_iostream **)vf->payload; + void *download_buf = NULL; + size_t download_size = 0; + ssize_t ret; + + ret = ocierofs_download_blob_range(oci_iostream->ctx, offset, len, + &download_buf, &download_size); + if (ret < 0) { + memset(buf, 0, len); + return len; + } + + if (download_buf && download_size > 0) { + memcpy(buf, download_buf, download_size); + free(download_buf); + return download_size; + } + + return 0; +} + +static ssize_t ocierofs_io_read(struct erofs_vfile *vf, void *buf, size_t len) +{ + struct ocierofs_iostream *oci_iostream = *(struct ocierofs_iostream **)vf->payload; + ssize_t ret; + + ret = ocierofs_io_pread(vf, buf, len, oci_iostream->offset); + if (ret > 0) + oci_iostream->offset += ret; + + return ret; +} + +static off_t ocierofs_io_lseek(struct erofs_vfile *vf, u64 offset, int whence) +{ + struct ocierofs_iostream *oci_iostream = *(struct ocierofs_iostream **)vf->payload; + off_t new_offset; + int layer_index = oci_iostream->ctx->img.layer_index; + + switch (whence) { + case SEEK_SET: + new_offset = offset; + break; + case SEEK_CUR: + new_offset = oci_iostream->offset + offset; + break; + case SEEK_END: + new_offset = oci_iostream->ctx->img.layers[layer_index]->size + offset; + break; + default: + return -1; + } + + if (new_offset < 0 || new_offset > oci_iostream->ctx->img.layers[layer_index]->size) + return -1; + + oci_iostream->offset = new_offset; + return new_offset; +} + +static ssize_t ocierofs_io_sendfile(struct erofs_vfile *vout, struct erofs_vfile *vin, + off_t *pos, size_t count) +{ + struct ocierofs_iostream *oci_iostream = *(struct ocierofs_iostream **)vin->payload; + char *buf = NULL; + ssize_t total_written = 0; + ssize_t ret = 0; + + buf = malloc(min_t(size_t, count, 32768)); + if (!buf) + return -ENOMEM; + + while (count > 0) { + size_t to_read = min_t(size_t, count, 32768); + u64 read_offset = pos ? *pos : oci_iostream->offset; + + ret = ocierofs_io_pread(vin, buf, to_read, read_offset); + if (ret <= 0) { + erofs_err("OCI I/O sendfile: failed to read from OCI: %s", + erofs_strerror(ret)); + memset(buf, 0, to_read); + ret = to_read; + } + + ssize_t written = write(vout->fd, buf, ret); + + if (written < 0) { + erofs_err("OCI I/O sendfile: failed to write to output: %s", + strerror(errno)); + ret = -errno; + break; + } + + if (written != ret) { + erofs_err("OCI I/O sendfile: partial write: %zd != %zd", written, ret); + ret = written; + } + + total_written += ret; + count -= ret; + if (pos) + *pos += ret; + else + oci_iostream->offset += ret; + } + + free(buf); + return count; +} + +static struct erofs_vfops ocierofs_io_vfops = { + .pread = ocierofs_io_pread, + .read = ocierofs_io_read, + .lseek = ocierofs_io_lseek, + .sendfile = ocierofs_io_sendfile, +}; + +int ocierofs_iostream_open(struct ocierofs_iostream *oci_iostream, struct ocierofs_ctx *oci_ctx) +{ + + memset(oci_iostream, 0, sizeof(*oci_iostream)); + oci_iostream->ctx = oci_ctx; + oci_iostream->vf.ops = &ocierofs_io_vfops; + oci_iostream->vf.fd = -1; + *(struct ocierofs_iostream **)oci_iostream->vf.payload = oci_iostream; + + return 0; +} + +void ocierofs_iostream_close(struct ocierofs_iostream *oci_iostream) +{ + close(oci_iostream->vf.fd); +} + +int ocierofs_is_erofs_native_image(struct ocierofs_ctx *ctx) +{ + if (ctx->img.layer_count > 0 && ctx->img.layers[0] && + ctx->img.layers[0]->media_type) { + if (strcmp(ctx->img.layers[0]->media_type, EROFS_MEDIATYPE) != 0) + return -ENOENT; + return 0; + } + return -ENOENT; +} diff --git a/mount/Makefile.am b/mount/Makefile.am index d93f3f4..7b971f5 100644 --- a/mount/Makefile.am +++ b/mount/Makefile.am @@ -9,5 +9,4 @@ mount_erofs_SOURCES = main.c mount_erofs_CFLAGS = -Wall -I$(top_srcdir)/include mount_erofs_LDADD = $(top_builddir)/lib/liberofs.la ${libselinux_LIBS} \ ${liblz4_LIBS} ${liblzma_LIBS} ${zlib_LIBS} ${libdeflate_LIBS} \ - ${libzstd_LIBS} ${libqpl_LIBS} ${libxxhash_LIBS} ${libnl3_LIBS} -endif + ${libzstd_LIBS} ${libqpl_LIBS} ${libxxhash_LIBS} ${libnl3_LIBS} ${openssl_LIBS} diff --git a/mount/main.c b/mount/main.c index 139b532..77c16f5 100644 --- a/mount/main.c +++ b/mount/main.c @@ -15,6 +15,7 @@ #include "erofs/err.h" #include "erofs/io.h" #include "../lib/liberofs_nbd.h" +#include "../lib/liberofs_oci.h" #ifdef HAVE_LINUX_LOOP_H #include <linux/loop.h> #else @@ -34,6 +35,10 @@ struct loop_info { #include <sys/sysmacros.h> #endif +#ifdef OCIEROFS_ENABLED +static struct ocierofs_config ocicfg; +#endif + enum erofs_backend_drv { EROFSAUTO, EROFSLOCAL, @@ -56,6 +61,11 @@ static struct erofsmount_cfg { long flags; enum erofs_backend_drv backend; enum erofsmount_mode mountmode; + bool umount; +#ifdef OCIEROFS_ENABLED + char *oci_options; + bool use_oci; +#endif } mountcfg = { .full_options = "ro", .flags = MS_RDONLY, /* default mountflags */ @@ -128,6 +138,9 @@ static int erofsmount_parse_options(int argc, char **argv) static const struct option long_options[] = { {"help", no_argument, 0, 'h'}, {"reattach", no_argument, 0, 512}, +#ifdef OCIEROFS_ENABLED + {"oci", optional_argument, 0, 513}, +#endif {0, 0, 0, 0}, }; char *dot; @@ -165,6 +178,12 @@ static int erofsmount_parse_options(int argc, char **argv) case 512: mountcfg.mountmode = EROFSMOUNT_MODE_REATTACH; break; +#ifdef OCIEROFS_ENABLED + case 513: + mountcfg.oci_options = optarg; + mountcfg.use_oci = true; + break; +#endif default: return -EINVAL; } @@ -198,6 +217,74 @@ static int erofsmount_parse_options(int argc, char **argv) return 0; } +static int mount_parse_oci_options(struct ocierofs_config *oci_cfg, char *options_str) +{ + char *opt, *q, *p; + + if (!options_str) + return 0; + + opt = options_str; + while (opt) { + q = strchr(opt, ','); + if (q) + *q = '\0'; + + p = strstr(opt, "platform="); + if (p) { + p += strlen("platform="); + free(oci_cfg->platform); + oci_cfg->platform = strdup(p); + if (!oci_cfg->platform) + return -ENOMEM; + opt = q ? q + 1 : NULL; + continue; + } + + p = strstr(opt, "layer="); + if (p) { + p += strlen("layer="); + oci_cfg->layer_index = atoi(p); + if (oci_cfg->layer_index < 0) { + erofs_err("invalid layer index %d", + oci_cfg->layer_index); + return -EINVAL; + } + opt = q ? q + 1 : NULL; + continue; + } + + p = strstr(opt, "username="); + if (p) { + p += strlen("username="); + free(oci_cfg->username); + oci_cfg->username = strdup(p); + if (!oci_cfg->username) + return -ENOMEM; + opt = q ? q + 1 : NULL; + continue; + } + + p = strstr(opt, "password="); + if (p) { + p += strlen("password="); + free(oci_cfg->password); + oci_cfg->password = strdup(p); + if (!oci_cfg->password) + return -ENOMEM; + opt = q ? q + 1 : NULL; + continue; + } + + erofs_err("mkfs: invalid --oci value %s", opt); + return -EINVAL; + + opt = q ? q + 1 : NULL; + } + + return 0; +} + static int erofsmount_fuse(const char *source, const char *mountpoint, const char *fstype, const char *options) { @@ -750,6 +837,122 @@ err_out: return err < 0 ? err : 0; } +/** + * erofsmount_startnbd_oci - Start NBD server for OCI image + * @nbdfd: NBD device file descriptor + * @oci_ctx: OCI client structure (pre-authenticated) + * @auth_header: pre-authenticated auth header + * + * Start an NBD server that serves data from an OCI image layer. + * This function reuses the existing erofsmount_nbd_loopfn logic + * but uses erofsoci_iostream as the virtual device instead of a local file. + * The OCI client should be pre-authenticated to avoid concurrent auth issues. + * + * Return: 0 on success, negative errno on failure + */ +static int erofsmount_startnbd_oci(int nbdfd, struct ocierofs_ctx *oci_ctx) +{ + struct erofsmount_nbd_ctx ctx = {}; + struct ocierofs_iostream *oci_iostream = NULL; + uintptr_t retcode; + pthread_t th; + int err, err2; + int blkbits = 12; + int index = oci_ctx->img.layer_index; + u64 blocks; + + blocks = (oci_ctx->img.layers[index]->size + (1ULL << blkbits) - 1) >> blkbits; + + oci_iostream = malloc(sizeof(struct ocierofs_iostream)); + if (!oci_iostream) + return -ENOMEM; + + err = ocierofs_iostream_open(oci_iostream, oci_ctx); + if (err) { + free(oci_iostream); + return err; + } + + ctx.vd = oci_iostream->vf; + + err = erofs_nbd_connect(nbdfd, blkbits, blocks); + if (err < 0) { + ocierofs_iostream_close(oci_iostream); + free(oci_iostream); + return err; + } + ctx.sk.fd = err; + + err = -pthread_create(&th, NULL, erofsmount_nbd_loopfn, &ctx); + if (err) { + ocierofs_iostream_close(oci_iostream); + free(oci_iostream); + close(ctx.sk.fd); + return err; + } + + err = erofs_nbd_do_it(nbdfd); + err2 = -pthread_join(th, (void **)&retcode); + if (!err2 && retcode) { + erofs_err("NBD worker failed with %s", + erofs_strerror(retcode)); + err2 = retcode; + } + + ocierofs_iostream_close(oci_iostream); + free(oci_iostream); + + return err ?: err2; +} + +static int erofsmount_nbd_oci(struct ocierofs_ctx *ctx, const char *mountpoint, + const char *fstype, int flags, const char *options) +{ + char nbdpath[32]; + int num, nbdfd; + pid_t pid; + long err; + + if (strcmp(fstype, "erofs")) { + fprintf(stderr, "unsupported filesystem type `%s`\n", fstype); + return -ENODEV; + } + + flags |= O_RDONLY; + + num = erofs_nbd_devscan(); + if (num < 0) + return num; + + (void)snprintf(nbdpath, sizeof(nbdpath), "/dev/nbd%d", num); + nbdfd = open(nbdpath, O_RDWR); + if (nbdfd < 0) + return -errno; + + if ((pid = fork()) == 0) { + return erofsmount_startnbd_oci(nbdfd, ctx) ? + EXIT_FAILURE : EXIT_SUCCESS; + } + close(nbdfd); + + while (1) { + err = erofs_nbd_in_service(num); + if (err == -ENOENT || err == -ENOTCONN) { + usleep(50000); + continue; + } + if (err >= 0) + err = (err != pid ? -EBUSY : 0); + break; + } + if (!err) { + err = mount(nbdpath, mountpoint, fstype, flags, options); + if (err < 0) + err = -errno; + } + return err; +} + int main(int argc, char *argv[]) { int err; @@ -785,9 +988,42 @@ int main(int argc, char *argv[]) } if (mountcfg.backend == EROFSNBD) { +#ifdef OCIEROFS_ENABLED + if (mountcfg.use_oci) { + struct ocierofs_ctx ctx = {}; + + ocicfg.image_ref = mountcfg.device; + err = mount_parse_oci_options(&ocicfg, mountcfg.oci_options); + if (err) + goto exit; + err = ocierofs_init(&ctx, &ocicfg); + if (err) { + ocierofs_ctx_cleanup(&ctx); + goto exit; + } + + err = ocierofs_is_erofs_native_image(&ctx); + if (err) { + ocierofs_ctx_cleanup(&ctx); + goto exit; + } + + err = erofsmount_nbd_oci(&ctx, mountcfg.target, + mountcfg.fstype, mountcfg.flags, mountcfg.options); + if (err) { + ocierofs_ctx_cleanup(&ctx); + goto exit; + } + } else { + err = erofsmount_nbd(mountcfg.device, mountcfg.target, + mountcfg.fstype, mountcfg.flags, + mountcfg.options); + } +#else err = erofsmount_nbd(mountcfg.device, mountcfg.target, mountcfg.fstype, mountcfg.flags, mountcfg.options); +#endif goto exit; } -- 2.51.0 ^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH v4 0/3] erofs-utils: refactor OCI and add NBD-backed OCI image mounting 2025-09-01 5:10 [PATCH v1] erofs-utils: refactor OCI code for better modularity ChengyuZhu6 ` (3 preceding siblings ...) 2025-09-03 8:29 ` [PATCH v3 0/2] erofs-utils: refactor OCI and add NBD-backed OCI image mounting ChengyuZhu6 @ 2025-09-04 5:33 ` ChengyuZhu6 2025-09-04 5:33 ` [PATCH v4 1/3] erofs-utils:mkfs: move parse_oci_ref to lib ChengyuZhu6 ` (2 more replies) 2025-09-04 6:36 ` [PATCH v5 0/3] erofs-utils: refactor OCI and " ChengyuZhu6 5 siblings, 3 replies; 19+ messages in thread From: ChengyuZhu6 @ 2025-09-04 5:33 UTC (permalink / raw) To: linux-erofs; +Cc: xiang, hsiangkao, Chengyu Zhu From: Chengyu Zhu <hudsonzhu@tencent.com> This series refactors the OCI handling in erofs-utils and adds NBD-backed mounting of OCI images. It enables mounting EROFS container images directly from registries without pre-downloading. Chengyu Zhu (3): erofs-utils:mkfs: move parse_oci_ref to lib erofs-utils: refactor OCI code for better modularity erofs-utils: add NBD-backed OCI image mounting lib/liberofs_oci.h | 91 ++-- lib/remotes/oci.c | 1093 +++++++++++++++++++++++++++++--------------- mkfs/main.c | 201 ++------ mount/Makefile.am | 2 +- mount/main.c | 257 +++++++++-- 5 files changed, 1014 insertions(+), 630 deletions(-) -- 2.51.0 ^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH v4 1/3] erofs-utils:mkfs: move parse_oci_ref to lib 2025-09-04 5:33 ` [PATCH v4 0/3] erofs-utils: refactor OCI and " ChengyuZhu6 @ 2025-09-04 5:33 ` ChengyuZhu6 2025-09-04 5:33 ` [PATCH v4 2/3] erofs-utils: refactor OCI code for better modularity ChengyuZhu6 2025-09-04 5:33 ` [PATCH v4 3/3] erofs-utils: add NBD-backed OCI image mounting ChengyuZhu6 2 siblings, 0 replies; 19+ messages in thread From: ChengyuZhu6 @ 2025-09-04 5:33 UTC (permalink / raw) To: linux-erofs; +Cc: xiang, hsiangkao, Chengyu Zhu From: Chengyu Zhu <hudsonzhu@tencent.com> move parse_oci_ref to oci.c. Signed-off-by: Chengyu Zhu <hudsonzhu@tencent.com> --- lib/liberofs_oci.h | 12 +++-- lib/remotes/oci.c | 110 +++++++++++++++++++++++++++++++++++++++++ mkfs/main.c | 121 +-------------------------------------------- 3 files changed, 120 insertions(+), 123 deletions(-) diff --git a/lib/liberofs_oci.h b/lib/liberofs_oci.h index 3a8108b..bce63ef 100644 --- a/lib/liberofs_oci.h +++ b/lib/liberofs_oci.h @@ -8,9 +8,6 @@ #include <stdbool.h> -#define DOCKER_REGISTRY "docker.io" -#define DOCKER_API_REGISTRY "registry-1.docker.io" - #ifdef __cplusplus extern "C" { #endif @@ -79,6 +76,15 @@ void ocierofs_cleanup(struct erofs_oci *oci); */ int erofs_oci_params_set_string(char **field, const char *value); +/* + * ocierofs_parse_ref - Parse OCI image reference string + * @oci: OCI client structure + * @ref_str: OCI image reference string + * + * Return: 0 on success, negative errno on failure + */ +int ocierofs_parse_ref(struct erofs_oci *oci, const char *ref_str); + /* * ocierofs_build_trees - Build file trees from OCI container image layers * @root: root inode to build the file tree under diff --git a/lib/remotes/oci.c b/lib/remotes/oci.c index 0fb8c1f..e6f0c23 100644 --- a/lib/remotes/oci.c +++ b/lib/remotes/oci.c @@ -21,6 +21,9 @@ #include "liberofs_oci.h" #include "liberofs_private.h" +#define DOCKER_REGISTRY "docker.io" +#define DOCKER_API_REGISTRY "registry-1.docker.io" + #define DOCKER_MEDIATYPE_MANIFEST_V2 \ "application/vnd.docker.distribution.manifest.v2+json" #define DOCKER_MEDIATYPE_MANIFEST_V1 \ @@ -1066,3 +1069,110 @@ int erofs_oci_params_set_string(char **field, const char *value) *field = new_value; return 0; } + +int ocierofs_parse_ref(struct erofs_oci *oci, const char *ref_str) +{ + char *slash, *colon, *dot; + const char *repo_part; + size_t len; + + slash = strchr(ref_str, '/'); + if (slash) { + dot = strchr(ref_str, '.'); + if (dot && dot < slash) { + char *registry_str; + + len = slash - ref_str; + registry_str = strndup(ref_str, len); + + if (!registry_str) { + erofs_err("failed to allocate memory for registry"); + return -ENOMEM; + } + if (erofs_oci_params_set_string(&oci->params.registry, + registry_str)) { + free(registry_str); + erofs_err("failed to set registry"); + return -ENOMEM; + } + free(registry_str); + repo_part = slash + 1; + } else { + repo_part = ref_str; + } + } else { + repo_part = ref_str; + } + + colon = strchr(repo_part, ':'); + if (colon) { + char *repo_str; + + len = colon - repo_part; + repo_str = strndup(repo_part, len); + + if (!repo_str) { + erofs_err("failed to allocate memory for repository"); + return -ENOMEM; + } + + if (!strchr(repo_str, '/') && + (!strcmp(oci->params.registry, DOCKER_API_REGISTRY) || + !strcmp(oci->params.registry, DOCKER_REGISTRY))) { + char *full_repo; + + if (asprintf(&full_repo, "library/%s", repo_str) == -1) { + free(repo_str); + erofs_err("failed to allocate memory for full repository name"); + return -ENOMEM; + } + free(repo_str); + repo_str = full_repo; + } + + if (erofs_oci_params_set_string(&oci->params.repository, + repo_str)) { + free(repo_str); + erofs_err("failed to set repository"); + return -ENOMEM; + } + free(repo_str); + + if (erofs_oci_params_set_string(&oci->params.tag, + colon + 1)) { + erofs_err("failed to set tag"); + return -ENOMEM; + } + } else { + char *repo_str = strdup(repo_part); + + if (!repo_str) { + erofs_err("failed to allocate memory for repository"); + return -ENOMEM; + } + + if (!strchr(repo_str, '/') && + (!strcmp(oci->params.registry, DOCKER_API_REGISTRY) || + !strcmp(oci->params.registry, DOCKER_REGISTRY))) { + char *full_repo; + + if (asprintf(&full_repo, "library/%s", repo_str) == -1) { + free(repo_str); + erofs_err("failed to allocate memory for full repository name"); + return -ENOMEM; + } + free(repo_str); + repo_str = full_repo; + } + + if (erofs_oci_params_set_string(&oci->params.repository, + repo_str)) { + free(repo_str); + erofs_err("failed to set repository"); + return -ENOMEM; + } + free(repo_str); + } + + return 0; +} diff --git a/mkfs/main.c b/mkfs/main.c index bc895f1..064392d 100644 --- a/mkfs/main.c +++ b/mkfs/main.c @@ -763,125 +763,6 @@ static int mkfs_parse_oci_options(char *options_str) return 0; } - -/** - * mkfs_parse_oci_ref - Parse OCI image reference string - * @ref_str: OCI image reference in various formats - * - * Parse OCI image reference which can be in formats: - * - registry.example.com/namespace/repo:tag - * - namespace/repo:tag (uses default registry) - * - repo:tag (adds library/ prefix for Docker Hub) - * - repo (uses default tag "latest") - * - * Return: 0 on success, negative errno on failure - */ -static int mkfs_parse_oci_ref(const char *ref_str) -{ - char *slash, *colon, *dot; - const char *repo_part; - size_t len; - - slash = strchr(ref_str, '/'); - if (slash) { - dot = strchr(ref_str, '.'); - if (dot && dot < slash) { - char *registry_str; - - len = slash - ref_str; - registry_str = strndup(ref_str, len); - - if (!registry_str) { - erofs_err("failed to allocate memory for registry"); - return -ENOMEM; - } - if (erofs_oci_params_set_string(&ocicfg.params.registry, - registry_str)) { - free(registry_str); - erofs_err("failed to set registry"); - return -ENOMEM; - } - free(registry_str); - repo_part = slash + 1; - } else { - repo_part = ref_str; - } - } else { - repo_part = ref_str; - } - - colon = strchr(repo_part, ':'); - if (colon) { - char *repo_str; - - len = colon - repo_part; - repo_str = strndup(repo_part, len); - - if (!repo_str) { - erofs_err("failed to allocate memory for repository"); - return -ENOMEM; - } - - if (!strchr(repo_str, '/') && - (!strcmp(ocicfg.params.registry, DOCKER_API_REGISTRY) || - !strcmp(ocicfg.params.registry, DOCKER_REGISTRY))) { - char *full_repo; - - if (asprintf(&full_repo, "library/%s", repo_str) == -1) { - free(repo_str); - erofs_err("failed to allocate memory for full repository name"); - return -ENOMEM; - } - free(repo_str); - repo_str = full_repo; - } - - if (erofs_oci_params_set_string(&ocicfg.params.repository, - repo_str)) { - free(repo_str); - erofs_err("failed to set repository"); - return -ENOMEM; - } - free(repo_str); - - if (erofs_oci_params_set_string(&ocicfg.params.tag, - colon + 1)) { - erofs_err("failed to set tag"); - return -ENOMEM; - } - } else { - char *repo_str = strdup(repo_part); - - if (!repo_str) { - erofs_err("failed to allocate memory for repository"); - return -ENOMEM; - } - - if (!strchr(repo_str, '/') && - (!strcmp(ocicfg.params.registry, DOCKER_API_REGISTRY) || - !strcmp(ocicfg.params.registry, DOCKER_REGISTRY))) { - char *full_repo; - - if (asprintf(&full_repo, "library/%s", repo_str) == -1) { - free(repo_str); - erofs_err("failed to allocate memory for full repository name"); - return -ENOMEM; - } - free(repo_str); - repo_str = full_repo; - } - - if (erofs_oci_params_set_string(&ocicfg.params.repository, - repo_str)) { - free(repo_str); - erofs_err("failed to set repository"); - return -ENOMEM; - } - free(repo_str); - } - - return 0; -} #endif static int mkfs_parse_one_compress_alg(char *alg, @@ -1958,7 +1839,7 @@ int main(int argc, char **argv) err = mkfs_parse_oci_options(mkfs_oci_options); if (err) goto exit; - err = mkfs_parse_oci_ref(cfg.c_src_path); + err = ocierofs_parse_ref(&ocicfg, cfg.c_src_path); if (err) goto exit; -- 2.51.0 ^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH v4 2/3] erofs-utils: refactor OCI code for better modularity 2025-09-04 5:33 ` [PATCH v4 0/3] erofs-utils: refactor OCI and " ChengyuZhu6 2025-09-04 5:33 ` [PATCH v4 1/3] erofs-utils:mkfs: move parse_oci_ref to lib ChengyuZhu6 @ 2025-09-04 5:33 ` ChengyuZhu6 2025-09-04 5:53 ` Gao Xiang 2025-09-04 5:33 ` [PATCH v4 3/3] erofs-utils: add NBD-backed OCI image mounting ChengyuZhu6 2 siblings, 1 reply; 19+ messages in thread From: ChengyuZhu6 @ 2025-09-04 5:33 UTC (permalink / raw) To: linux-erofs; +Cc: xiang, hsiangkao, Chengyu Zhu From: Chengyu Zhu <hudsonzhu@tencent.com> Refactor OCI code to improve code organization and maintainability: Key changes: - Add `ocierofs_get_api_registry()` and unify API endpoint selection. - Implement Bearer token discovery with Basic fallback; cache auth header. - Parse layer metadata (digest, mediaType, size) and add a proper free helper. - Split blob download from tar processing; process tar via a temp fd. - Rework init/teardown into `ocierofs_init()` and `ocierofs_ctx_cleanup()`. - Update mkfs to use `struct ocierofs_config` and new `--oci` parsing. Signed-off-by: Chengyu Zhu <hudsonzhu@tencent.com> --- lib/liberofs_oci.h | 87 ++--- lib/remotes/oci.c | 956 ++++++++++++++++++++++----------------------- mkfs/main.c | 82 ++-- 3 files changed, 537 insertions(+), 588 deletions(-) diff --git a/lib/liberofs_oci.h b/lib/liberofs_oci.h index bce63ef..8622464 100644 --- a/lib/liberofs_oci.h +++ b/lib/liberofs_oci.h @@ -12,87 +12,56 @@ extern "C" { #endif -struct erofs_inode; -struct CURL; struct erofs_importer; -/** - * struct erofs_oci_params - OCI configuration parameters - * @registry: registry hostname (e.g., "registry-1.docker.io") - * @repository: image repository (e.g., "library/ubuntu") - * @tag: image tag or digest (e.g., "latest" or sha256:...) +/* + * struct ocierofs_config - OCI configuration structure + * @image_ref: OCI image reference (e.g., "ubuntu:latest", "myregistry.com/app:v1.0") * @platform: target platform in "os/arch" format (e.g., "linux/amd64") * @username: username for authentication (optional) * @password: password for authentication (optional) * @layer_index: specific layer to extract (-1 for all layers) * - * Configuration structure for OCI image parameters including registry - * location, image identification, platform specification, and authentication - * credentials. */ -struct erofs_oci_params { - char *registry; - char *repository; - char *tag; +struct ocierofs_config { + char *image_ref; char *platform; char *username; char *password; int layer_index; }; -/** - * struct erofs_oci - Combined OCI client structure - * @curl: CURL handle for HTTP requests - * @params: OCI configuration parameters - * - * Main OCI client structure combining CURL HTTP client with - * OCI-specific configuration parameters. - */ -struct erofs_oci { - struct CURL *curl; - struct erofs_oci_params params; +struct ocierofs_layer_info { + char *digest; + char *media_type; + u64 size; }; -/* - * ocierofs_init - Initialize OCI client with default parameters - * @oci: OCI client structure to initialize - * - * Return: 0 on success, negative errno on failure - */ -int ocierofs_init(struct erofs_oci *oci); - -/* - * ocierofs_cleanup - Clean up OCI client and free allocated resources - * @oci: OCI client structure to clean up - */ -void ocierofs_cleanup(struct erofs_oci *oci); - -/* - * erofs_oci_params_set_string - Set a string field with dynamic allocation - * @field: pointer to the string field to set - * @value: string value to set - * - * Return: 0 on success, negative errno on failure - */ -int erofs_oci_params_set_string(char **field, const char *value); +struct ocierofs_ctx { + struct CURL *curl; + char *auth_header; + bool using_basic; + char *registry; + char *repository; + char *platform; + char *tag; + char *manifest_digest; + struct ocierofs_layer_info **layers; + int layer_index; + int layer_count; +}; -/* - * ocierofs_parse_ref - Parse OCI image reference string - * @oci: OCI client structure - * @ref_str: OCI image reference string - * - * Return: 0 on success, negative errno on failure - */ -int ocierofs_parse_ref(struct erofs_oci *oci, const char *ref_str); +int ocierofs_init(struct ocierofs_ctx *ctx, const struct ocierofs_config *config); /* - * ocierofs_build_trees - Build file trees from OCI container image layers - * @root: root inode to build the file tree under - * @oci: OCI client structure with configured parameters + * ocierofs_build_trees - Build file trees from an OCI container image + * @importer: erofs importer to populate + * @cfg: oci configuration * * Return: 0 on success, negative errno on failure */ -int ocierofs_build_trees(struct erofs_importer *importer, struct erofs_oci *oci); +int ocierofs_build_trees(struct erofs_importer *importer, + const struct ocierofs_config *cfg); #ifdef __cplusplus } diff --git a/lib/remotes/oci.c b/lib/remotes/oci.c index e6f0c23..4291dc6 100644 --- a/lib/remotes/oci.c +++ b/lib/remotes/oci.c @@ -33,28 +33,67 @@ #define OCI_MEDIATYPE_MANIFEST "application/vnd.oci.image.manifest.v1+json" #define OCI_MEDIATYPE_INDEX "application/vnd.oci.image.index.v1+json" -struct erofs_oci_request { +struct ocierofs_request { char *url; struct curl_slist *headers; }; -struct erofs_oci_response { +struct ocierofs_response { char *data; size_t size; long http_code; }; -struct erofs_oci_stream { - struct erofs_tarfile tarfile; +struct ocierofs_stream { const char *digest; int blobfd; }; +static inline const char *ocierofs_get_api_registry(const char *registry) +{ + if (!registry) + return DOCKER_API_REGISTRY; + return !strcmp(registry, DOCKER_REGISTRY) ? DOCKER_API_REGISTRY : registry; +} + +static inline bool ocierofs_is_manifest_list(const char *media_type) +{ + return media_type && (!strcmp(media_type, DOCKER_MEDIATYPE_MANIFEST_LIST) || + !strcmp(media_type, OCI_MEDIATYPE_INDEX)); +} + +static inline bool ocierofs_is_manifest(const char *media_type) +{ + return media_type && (!strcmp(media_type, DOCKER_MEDIATYPE_MANIFEST_V2) || + !strcmp(media_type, OCI_MEDIATYPE_MANIFEST)); +} + +static inline void ocierofs_request_cleanup(struct ocierofs_request *req) +{ + if (!req) + return; + if (req->headers) + curl_slist_free_all(req->headers); + free(req->url); + req->url = NULL; + req->headers = NULL; +} + +static inline void ocierofs_response_cleanup(struct ocierofs_response *resp) +{ + if (!resp) + return; + free(resp->data); + resp->data = NULL; + resp->size = 0; + resp->http_code = 0; +} + static size_t ocierofs_write_callback(void *contents, size_t size, size_t nmemb, void *userp) { size_t realsize = size * nmemb; - struct erofs_oci_response *resp = userp; + struct ocierofs_response *resp = userp; char *ptr; if (!resp->data) @@ -75,16 +114,23 @@ static size_t ocierofs_write_callback(void *contents, size_t size, static size_t ocierofs_layer_write_callback(void *contents, size_t size, size_t nmemb, void *userp) { - struct erofs_oci_stream *stream = userp; + struct ocierofs_stream *stream = userp; size_t realsize = size * nmemb; + const char *buf = contents; + size_t written = 0; if (stream->blobfd < 0) return 0; - if (write(stream->blobfd, contents, realsize) != realsize) { - erofs_err("failed to write layer data for layer %s", - stream->digest); - return 0; + while (written < realsize) { + ssize_t n = write(stream->blobfd, buf + written, realsize - written); + + if (n < 0) { + erofs_err("failed to write layer data for layer %s", + stream->digest); + return 0; + } + written += n; } return realsize; } @@ -96,6 +142,14 @@ static int ocierofs_curl_setup_common_options(struct CURL *curl) curl_easy_setopt(curl, CURLOPT_TIMEOUT, 120L); curl_easy_setopt(curl, CURLOPT_NOSIGNAL, 1L); curl_easy_setopt(curl, CURLOPT_USERAGENT, "ocierofs/" PACKAGE_VERSION); + curl_easy_setopt(curl, CURLOPT_ACCEPT_ENCODING, ""); + curl_easy_setopt(curl, CURLOPT_TCP_KEEPALIVE, 1L); +#if defined(CURLOPT_TCP_KEEPIDLE) + curl_easy_setopt(curl, CURLOPT_TCP_KEEPIDLE, 30L); +#endif +#if defined(CURLOPT_TCP_KEEPINTVL) + curl_easy_setopt(curl, CURLOPT_TCP_KEEPINTVL, 15L); +#endif return 0; } @@ -114,10 +168,10 @@ static int ocierofs_curl_setup_basic_auth(struct CURL *curl, const char *usernam return 0; } -static int ocierofs_curl_clear_auth(struct CURL *curl) +static int ocierofs_curl_clear_auth(struct ocierofs_ctx *ctx) { - curl_easy_setopt(curl, CURLOPT_USERPWD, NULL); - curl_easy_setopt(curl, CURLOPT_HTTPAUTH, CURLAUTH_NONE); + curl_easy_setopt(ctx->curl, CURLOPT_USERPWD, NULL); + curl_easy_setopt(ctx->curl, CURLOPT_HTTPAUTH, CURLAUTH_NONE); return 0; } @@ -176,20 +230,20 @@ static int ocierofs_curl_perform(struct CURL *curl, long *http_code_out) return 0; } -static int ocierofs_request_perform(struct erofs_oci *oci, - struct erofs_oci_request *req, - struct erofs_oci_response *resp) +static int ocierofs_request_perform(struct ocierofs_ctx *ctx, + struct ocierofs_request *req, + struct ocierofs_response *resp) { int ret; - ret = ocierofs_curl_setup_rq(oci->curl, req->url, + ret = ocierofs_curl_setup_rq(ctx->curl, req->url, OCIEROFS_HTTP_GET, req->headers, - ocierofs_write_callback, resp, + ocierofs_write_callback, resp, NULL, NULL); if (ret) return ret; - ret = ocierofs_curl_perform(oci->curl, &resp->http_code); + ret = ocierofs_curl_perform(ctx->curl, &resp->http_code); if (ret) return ret; @@ -204,15 +258,10 @@ static int ocierofs_request_perform(struct erofs_oci *oci, * @realm_out: pointer to store realm value * @service_out: pointer to store service value * @scope_out: pointer to store scope value - * - * Parse Bearer authentication header and extract realm, service, and scope - * parameters for subsequent token requests. - * - * Return: 0 on success, negative errno on failure */ static int ocierofs_parse_auth_header(const char *auth_header, - char **realm_out, char **service_out, - char **scope_out) + char **realm_out, char **service_out, + char **scope_out) { char *realm = NULL, *service = NULL, *scope = NULL; static const char * const param_names[] = {"realm=", "service=", "scope="}; @@ -221,7 +270,6 @@ static int ocierofs_parse_auth_header(const char *auth_header, const char *p; int i, ret = 0; - // https://datatracker.ietf.org/doc/html/rfc6750#section-3 if (strncmp(auth_header, "Bearer ", strlen("Bearer "))) return -EINVAL; @@ -229,7 +277,6 @@ static int ocierofs_parse_auth_header(const char *auth_header, if (!header_copy) return -ENOMEM; - /* Clean up header: replace newlines with spaces and remove double spaces */ for (char *q = header_copy; *q; q++) { if (*q == '\n' || *q == '\r') *q = ' '; @@ -277,22 +324,9 @@ out: return ret; } -/** - * ocierofs_extract_www_auth_info - Extract WWW-Authenticate header information - * @resp_data: HTTP response data containing headers - * @realm_out: pointer to store realm value (optional) - * @service_out: pointer to store service value (optional) - * @scope_out: pointer to store scope value (optional) - * - * Extract realm, service, and scope from WWW-Authenticate header in HTTP response. - * This function handles the common pattern of parsing WWW-Authenticate headers - * that appears in multiple places in the OCI authentication flow. - * - * Return: 0 on success, negative errno on failure - */ static int ocierofs_extract_www_auth_info(const char *resp_data, - char **realm_out, char **service_out, - char **scope_out) + char **realm_out, char **service_out, + char **scope_out) { char *www_auth; char *line_end; @@ -336,29 +370,12 @@ static int ocierofs_extract_www_auth_info(const char *resp_data, return ret; } -/** - * ocierofs_get_auth_token_with_url - Get authentication token from auth server - * @oci: OCI client structure - * @auth_url: authentication server URL - * @service: service name for authentication - * @repository: repository name - * @username: username for basic auth (optional) - * @password: password for basic auth (optional) - * - * Request authentication token from the specified auth server URL using - * basic authentication if credentials are provided. - * - * Return: authentication header string on success, ERR_PTR on failure - */ -static char *ocierofs_get_auth_token_with_url(struct erofs_oci *oci, - const char *auth_url, - const char *service, - const char *repository, - const char *username, - const char *password) +static char *ocierofs_get_auth_token_with_url(struct ocierofs_ctx *ctx, const char *auth_url, + const char *service, const char *repository, + const char *username, const char *password) { - struct erofs_oci_request req = {}; - struct erofs_oci_response resp = {}; + struct ocierofs_request req = {}; + struct ocierofs_response resp = {}; json_object *root, *token_obj, *access_token_obj; const char *token; char *auth_header = NULL; @@ -373,40 +390,36 @@ static char *ocierofs_get_auth_token_with_url(struct erofs_oci *oci, } if (username && password && *username) { - ret = ocierofs_curl_setup_basic_auth(oci->curl, username, - password); + ret = ocierofs_curl_setup_basic_auth(ctx->curl, username, + password); if (ret) goto out_url; } - ret = ocierofs_request_perform(oci, &req, &resp); - ocierofs_curl_clear_auth(oci->curl); + ret = ocierofs_request_perform(ctx, &req, &resp); + ocierofs_curl_clear_auth(ctx); if (ret) goto out_url; if (!resp.data) { - erofs_err("empty response from auth server"); ret = -EINVAL; goto out_url; } root = json_tokener_parse(resp.data); if (!root) { - erofs_err("failed to parse auth response"); ret = -EINVAL; - goto out_url; + goto out_json; } if (!json_object_object_get_ex(root, "token", &token_obj) && !json_object_object_get_ex(root, "access_token", &access_token_obj)) { - erofs_err("no token found in auth response"); ret = -EINVAL; goto out_json; } token = json_object_get_string(token_obj ? token_obj : access_token_obj); if (!token) { - erofs_err("invalid token in auth response"); ret = -EINVAL; goto out_json; } @@ -419,16 +432,15 @@ static char *ocierofs_get_auth_token_with_url(struct erofs_oci *oci, out_json: json_object_put(root); out_url: - free(req.url); - free(resp.data); + ocierofs_response_cleanup(&resp); + ocierofs_request_cleanup(&req); return ret ? ERR_PTR(ret) : auth_header; } -static char *ocierofs_discover_auth_endpoint(struct erofs_oci *oci, - const char *registry, +static char *ocierofs_discover_auth_endpoint(struct ocierofs_ctx *ctx, const char *registry, const char *repository) { - struct erofs_oci_response resp = {}; + struct ocierofs_response resp = {}; char *realm = NULL; char *service = NULL; char *result = NULL; @@ -437,20 +449,20 @@ static char *ocierofs_discover_auth_endpoint(struct erofs_oci *oci, CURLcode res; long http_code; - api_registry = (!strcmp(registry, DOCKER_REGISTRY)) ? DOCKER_API_REGISTRY : registry; + api_registry = ocierofs_get_api_registry(registry); if (asprintf(&test_url, "https://%s/v2/%s/manifests/nonexistent", api_registry, repository) < 0) return NULL; - curl_easy_reset(oci->curl); - ocierofs_curl_setup_common_options(oci->curl); + curl_easy_reset(ctx->curl); + ocierofs_curl_setup_common_options(ctx->curl); - ocierofs_curl_setup_rq(oci->curl, test_url, OCIEROFS_HTTP_HEAD, NULL, + ocierofs_curl_setup_rq(ctx->curl, test_url, OCIEROFS_HTTP_HEAD, NULL, NULL, NULL, ocierofs_write_callback, &resp); - res = curl_easy_perform(oci->curl); - curl_easy_getinfo(oci->curl, CURLINFO_RESPONSE_CODE, &http_code); + res = curl_easy_perform(ctx->curl); + curl_easy_getinfo(ctx->curl, CURLINFO_RESPONSE_CODE, &http_code); if (res == CURLE_OK && (http_code == 401 || http_code == 403 || http_code == 404) && resp.data) { @@ -461,14 +473,13 @@ static char *ocierofs_discover_auth_endpoint(struct erofs_oci *oci, } free(realm); free(service); - free(resp.data); + ocierofs_response_cleanup(&resp); free(test_url); return result; } -static char *ocierofs_get_auth_token(struct erofs_oci *oci, const char *registry, - const char *repository, const char *username, - const char *password) +static char *ocierofs_get_auth_token(struct ocierofs_ctx *ctx, const char *registry, + const char *repository, const char *username, const char *password) { static const char * const auth_patterns[] = { "https://%s/v2/auth", @@ -487,35 +498,35 @@ static char *ocierofs_get_auth_token(struct erofs_oci *oci, const char *registry !strcmp(registry, DOCKER_REGISTRY); if (docker_reg) { service = "registry.docker.io"; - auth_header = ocierofs_get_auth_token_with_url(oci, + auth_header = ocierofs_get_auth_token_with_url(ctx, "https://auth.docker.io/token", service, repository, username, password); if (!IS_ERR(auth_header)) return auth_header; } - discovered_auth_url = ocierofs_discover_auth_endpoint(oci, registry, repository); + discovered_auth_url = ocierofs_discover_auth_endpoint(ctx, registry, repository); if (discovered_auth_url) { const char *api_registry, *auth_service; - struct erofs_oci_response resp = {}; + struct ocierofs_response resp = {}; char *test_url; CURLcode res; long http_code; - api_registry = (!strcmp(registry, DOCKER_REGISTRY)) ? DOCKER_API_REGISTRY : registry; + api_registry = ocierofs_get_api_registry(registry); if (asprintf(&test_url, "https://%s/v2/%s/manifests/nonexistent", api_registry, repository) >= 0) { - curl_easy_reset(oci->curl); - ocierofs_curl_setup_common_options(oci->curl); + curl_easy_reset(ctx->curl); + ocierofs_curl_setup_common_options(ctx->curl); - ocierofs_curl_setup_rq(oci->curl, test_url, - OCIEROFS_HTTP_HEAD, NULL, - NULL, NULL, - ocierofs_write_callback, &resp); + ocierofs_curl_setup_rq(ctx->curl, test_url, + OCIEROFS_HTTP_HEAD, NULL, + NULL, NULL, + ocierofs_write_callback, &resp); - res = curl_easy_perform(oci->curl); - curl_easy_getinfo(oci->curl, CURLINFO_RESPONSE_CODE, &http_code); + res = curl_easy_perform(ctx->curl); + curl_easy_getinfo(ctx->curl, CURLINFO_RESPONSE_CODE, &http_code); if (res == CURLE_OK && (http_code == 401 || http_code == 403 || http_code == 404) && resp.data) { @@ -524,14 +535,14 @@ static char *ocierofs_get_auth_token(struct erofs_oci *oci, const char *registry ocierofs_extract_www_auth_info(resp.data, &realm, &discovered_service, NULL); free(realm); } - free(resp.data); + ocierofs_response_cleanup(&resp); free(test_url); } auth_service = discovered_service ? discovered_service : service; - auth_header = ocierofs_get_auth_token_with_url(oci, discovered_auth_url, - auth_service, repository, - username, password); + auth_header = ocierofs_get_auth_token_with_url(ctx, discovered_auth_url, + auth_service, repository, + username, password); free(discovered_auth_url); free(discovered_service); if (!IS_ERR(auth_header)) @@ -544,9 +555,9 @@ static char *ocierofs_get_auth_token(struct erofs_oci *oci, const char *registry if (asprintf(&auth_url, auth_patterns[i], registry) < 0) continue; - auth_header = ocierofs_get_auth_token_with_url(oci, auth_url, - service, repository, - username, password); + auth_header = ocierofs_get_auth_token_with_url(ctx, auth_url, + service, repository, + username, password); free(auth_url); if (!IS_ERR(auth_header)) @@ -557,24 +568,21 @@ static char *ocierofs_get_auth_token(struct erofs_oci *oci, const char *registry return ERR_PTR(-ENOENT); } -static char *ocierofs_get_manifest_digest(struct erofs_oci *oci, - const char *registry, - const char *repository, const char *tag, - const char *platform, - const char *auth_header) +static char *ocierofs_get_manifest_digest(struct ocierofs_ctx *ctx, + const char *registry, + const char *repository, const char *tag, + const char *platform, + const char *auth_header) { - struct erofs_oci_request req = {}; - struct erofs_oci_response resp = {}; + struct ocierofs_request req = {}; + struct ocierofs_response resp = {}; json_object *root, *manifests, *manifest, *platform_obj, *arch_obj; json_object *os_obj, *digest_obj, *schema_obj, *media_type_obj; char *digest = NULL; const char *api_registry; int ret = 0, len, i; - if (!registry || !repository || !tag || !platform) - return ERR_PTR(-EINVAL); - - api_registry = (!strcmp(registry, DOCKER_REGISTRY)) ? DOCKER_API_REGISTRY : registry; + api_registry = ocierofs_get_api_registry(registry); if (asprintf(&req.url, "https://%s/v2/%s/manifests/%s", api_registry, repository, tag) < 0) return ERR_PTR(-ENOMEM); @@ -584,22 +592,20 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci, req.headers = curl_slist_append(req.headers, "Accept: " DOCKER_MEDIATYPE_MANIFEST_LIST "," - OCI_MEDIATYPE_INDEX "," DOCKER_MEDIATYPE_MANIFEST_V1 "," - DOCKER_MEDIATYPE_MANIFEST_V2); + OCI_MEDIATYPE_INDEX "," OCI_MEDIATYPE_MANIFEST "," + DOCKER_MEDIATYPE_MANIFEST_V1 "," DOCKER_MEDIATYPE_MANIFEST_V2); - ret = ocierofs_request_perform(oci, &req, &resp); + ret = ocierofs_request_perform(ctx, &req, &resp); if (ret) goto out; if (!resp.data) { - erofs_err("empty response from manifest request"); ret = -EINVAL; goto out; } root = json_tokener_parse(resp.data); if (!root) { - erofs_err("failed to parse manifest JSON"); ret = -EINVAL; goto out; } @@ -615,8 +621,7 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci, if (json_object_object_get_ex(root, "mediaType", &media_type_obj)) { const char *media_type = json_object_get_string(media_type_obj); - if (!strcmp(media_type, DOCKER_MEDIATYPE_MANIFEST_V2) || - !strcmp(media_type, OCI_MEDIATYPE_MANIFEST)) { + if (ocierofs_is_manifest(media_type)) { digest = strdup(tag); ret = 0; goto out_json; @@ -624,7 +629,6 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci, } if (!json_object_object_get_ex(root, "manifests", &manifests)) { - erofs_err("no manifests found in manifest list"); ret = -EINVAL; goto out_json; } @@ -634,9 +638,9 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci, manifest = json_object_array_get_idx(manifests, i); if (json_object_object_get_ex(manifest, "platform", - &platform_obj) && + &platform_obj) && json_object_object_get_ex(platform_obj, "architecture", - &arch_obj) && + &arch_obj) && json_object_object_get_ex(platform_obj, "os", &os_obj) && json_object_object_get_ex(manifest, "digest", &digest_obj)) { const char *arch = json_object_get_string(arch_obj); @@ -658,34 +662,44 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci, out_json: json_object_put(root); out: - free(resp.data); - if (req.headers) - curl_slist_free_all(req.headers); - free(req.url); - + ocierofs_response_cleanup(&resp); + ocierofs_request_cleanup(&req); return ret ? ERR_PTR(ret) : digest; } -static char **ocierofs_get_layers_info(struct erofs_oci *oci, - const char *registry, - const char *repository, - const char *digest, - const char *auth_header, - int *layer_count) +static void ocierofs_free_layers_info(struct ocierofs_layer_info **layers, int count) { - struct erofs_oci_request req = {}; - struct erofs_oci_response resp = {}; - json_object *root, *layers, *layer, *digest_obj; - char **layers_info = NULL; - const char *api_registry; - int ret, len, i, j; + int i; - if (!registry || !repository || !digest || !layer_count) - return ERR_PTR(-EINVAL); + if (!layers) + return; + + for (i = 0; i < count; i++) { + if (layers[i]) { + free(layers[i]->digest); + free(layers[i]->media_type); + free(layers[i]); + } + } + free(layers); +} + +static struct ocierofs_layer_info **ocierofs_fetch_layers_info(struct ocierofs_ctx *ctx, + const char *registry, + const char *repository, + const char *digest, + const char *auth_header, + int *layer_count) +{ + struct ocierofs_request req = {}; + struct ocierofs_response resp = {}; + json_object *root, *layers, *layer, *digest_obj, *media_type_obj, *size_obj; + struct ocierofs_layer_info **layers_info = NULL; + const char *api_registry; + int ret, len, i; *layer_count = 0; - api_registry = (!strcmp(registry, DOCKER_REGISTRY) ? - DOCKER_API_REGISTRY : registry); + api_registry = ocierofs_get_api_registry(registry); if (asprintf(&req.url, "https://%s/v2/%s/manifests/%s", api_registry, repository, digest) < 0) @@ -697,38 +711,34 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci, req.headers = curl_slist_append(req.headers, "Accept: " OCI_MEDIATYPE_MANIFEST "," DOCKER_MEDIATYPE_MANIFEST_V2); - ret = ocierofs_request_perform(oci, &req, &resp); + ret = ocierofs_request_perform(ctx, &req, &resp); if (ret) goto out; if (!resp.data) { - erofs_err("empty response from layers request"); ret = -EINVAL; goto out; } root = json_tokener_parse(resp.data); if (!root) { - erofs_err("failed to parse manifest JSON"); ret = -EINVAL; goto out; } if (!json_object_object_get_ex(root, "layers", &layers) || json_object_get_type(layers) != json_type_array) { - erofs_err("no layers found in manifest"); ret = -EINVAL; goto out_json; } len = json_object_array_length(layers); if (!len) { - erofs_err("empty layer list in manifest"); ret = -EINVAL; goto out_json; } - layers_info = calloc(len, sizeof(char *)); + layers_info = calloc(len, sizeof(*layers_info)); if (!layers_info) { ret = -ENOMEM; goto out_json; @@ -738,364 +748,188 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci, layer = json_object_array_get_idx(layers, i); if (!json_object_object_get_ex(layer, "digest", &digest_obj)) { - erofs_err("failed to parse layer %d", i); ret = -EINVAL; goto out_free; } - layers_info[i] = strdup(json_object_get_string(digest_obj)); + layers_info[i] = calloc(1, sizeof(**layers_info)); if (!layers_info[i]) { ret = -ENOMEM; goto out_free; } + layers_info[i]->digest = strdup(json_object_get_string(digest_obj)); + if (!layers_info[i]->digest) { + ret = -ENOMEM; + goto out_free; + } + if (json_object_object_get_ex(layer, "mediaType", &media_type_obj)) + layers_info[i]->media_type = strdup(json_object_get_string(media_type_obj)); + else + layers_info[i]->media_type = NULL; + + if (json_object_object_get_ex(layer, "size", &size_obj)) + layers_info[i]->size = json_object_get_int64(size_obj); + else + layers_info[i]->size = 0; } *layer_count = len; json_object_put(root); - free(resp.data); - if (req.headers) - curl_slist_free_all(req.headers); - free(req.url); + ocierofs_response_cleanup(&resp); + ocierofs_request_cleanup(&req); return layers_info; out_free: - if (layers_info) { - for (j = 0; j < i; j++) - free(layers_info[j]); - } - free(layers_info); + ocierofs_free_layers_info(layers_info, i); out_json: json_object_put(root); out: - free(resp.data); - if (req.headers) - curl_slist_free_all(req.headers); - free(req.url); + ocierofs_response_cleanup(&resp); + ocierofs_request_cleanup(&req); return ERR_PTR(ret); } -static int ocierofs_extract_layer(struct erofs_oci *oci, struct erofs_importer *importer, - const char *digest, const char *auth_header) +static int ocierofs_process_tar_stream(struct erofs_importer *importer, int fd) { - struct erofs_oci_request req = {}; - struct erofs_oci_stream stream = {}; - const char *api_registry; - long http_code; + struct erofs_tarfile tarfile = {}; int ret; - stream = (struct erofs_oci_stream) { - .digest = digest, - .blobfd = erofs_tmpfile(), - }; - if (stream.blobfd < 0) { - erofs_err("failed to create temporary file for %s", digest); - return -errno; - } - - api_registry = (!strcmp(oci->params.registry, DOCKER_REGISTRY)) ? - DOCKER_API_REGISTRY : oci->params.registry; - if (asprintf(&req.url, "https://%s/v2/%s/blobs/%s", - api_registry, oci->params.repository, digest) == -1) { - ret = -ENOMEM; - goto out; - } - - if (auth_header && strstr(auth_header, "Bearer")) - req.headers = curl_slist_append(req.headers, auth_header); - - curl_easy_reset(oci->curl); - - ret = ocierofs_curl_setup_common_options(oci->curl); - if (ret) - goto out; - - ret = ocierofs_curl_setup_rq(oci->curl, req.url, OCIEROFS_HTTP_GET, - req.headers, - ocierofs_layer_write_callback, - &stream, NULL, NULL); - if (ret) - goto out; - - ret = ocierofs_curl_perform(oci->curl, &http_code); - if (ret) - goto out; - - if (http_code < 200 || http_code >= 300) { - erofs_err("HTTP request failed with code %ld", http_code); - ret = -EIO; - goto out; - } + init_list_head(&tarfile.global.xattrs); - if (lseek(stream.blobfd, 0, SEEK_SET) < 0) { - erofs_err("failed to seek to beginning of temp file: %s", - strerror(errno)); - ret = -errno; - goto out; - } - - memset(&stream.tarfile, 0, sizeof(stream.tarfile)); - init_list_head(&stream.tarfile.global.xattrs); - - ret = erofs_iostream_open(&stream.tarfile.ios, stream.blobfd, - EROFS_IOS_DECODER_GZIP); + ret = erofs_iostream_open(&tarfile.ios, fd, EROFS_IOS_DECODER_GZIP); if (ret) { erofs_err("failed to initialize tar stream: %s", erofs_strerror(ret)); - goto out; + return ret; } do { - ret = tarerofs_parse_tar(importer, &stream.tarfile); + ret = tarerofs_parse_tar(importer, &tarfile); /* Continue parsing until end of archive */ } while (!ret); - erofs_iostream_close(&stream.tarfile.ios); + erofs_iostream_close(&tarfile.ios); if (ret < 0 && ret != -ENODATA) { erofs_err("failed to process tar stream: %s", erofs_strerror(ret)); - goto out; + return ret; } - ret = 0; -out: - if (stream.blobfd >= 0) - close(stream.blobfd); - if (req.headers) - curl_slist_free_all(req.headers); - free(req.url); - return ret; + return 0; } -/** - * ocierofs_build_trees - Build file trees from OCI container image layers - * @importer: EROFS importer structure - * @oci: OCI client structure with configured parameters - * - * Extract and build file system trees from all layers of an OCI container - * image. - * - * Return: 0 on success, negative errno on failure - */ -int ocierofs_build_trees(struct erofs_importer *importer, struct erofs_oci *oci) +static int ocierofs_prepare_auth(struct ocierofs_ctx *ctx, + const char *username, + const char *password) { char *auth_header = NULL; - char *manifest_digest = NULL; - char **layers = NULL; - int layer_count = 0; - int ret, i; - - if (!importer || !oci) - return -EINVAL; + int ret = 0; + + ctx->using_basic = false; + free(ctx->auth_header); + ctx->auth_header = NULL; + + auth_header = ocierofs_get_auth_token(ctx, + ctx->registry, + ctx->repository, + username, password); + if (!IS_ERR(auth_header)) { + ctx->auth_header = auth_header; + return 0; + } - if (oci->params.username && oci->params.password && - oci->params.username[0] && oci->params.password[0]) { - auth_header = ocierofs_get_auth_token(oci, - oci->params.registry, - oci->params.repository, - oci->params.username, - oci->params.password); - if (IS_ERR(auth_header)) { - auth_header = NULL; - ret = ocierofs_curl_setup_basic_auth(oci->curl, - oci->params.username, - oci->params.password); - if (ret) - goto out; - } - } else { - auth_header = ocierofs_get_auth_token(oci, - oci->params.registry, - oci->params.repository, - NULL, NULL); - if (IS_ERR(auth_header)) - auth_header = NULL; + if (username && password && *username && *password) { + ret = ocierofs_curl_setup_basic_auth(ctx->curl, + username, password); + if (ret) + return ret; + ctx->using_basic = true; } + return 0; +} - manifest_digest = ocierofs_get_manifest_digest(oci, oci->params.registry, - oci->params.repository, - oci->params.tag, - oci->params.platform, - auth_header); - if (IS_ERR(manifest_digest)) { - ret = PTR_ERR(manifest_digest); +static int ocierofs_prepare_layers(struct ocierofs_ctx *ctx, + const struct ocierofs_config *config) +{ + int ret; + + ret = ocierofs_prepare_auth(ctx, config ? config->username : NULL, + config ? config->password : NULL); + if (ret) + return ret; + + ctx->manifest_digest = ocierofs_get_manifest_digest(ctx, ctx->registry, + ctx->repository, + ctx->tag, + ctx->platform, + ctx->auth_header); + if (IS_ERR(ctx->manifest_digest)) { + ret = PTR_ERR(ctx->manifest_digest); erofs_err("failed to get manifest digest: %s", erofs_strerror(ret)); + ctx->manifest_digest = NULL; goto out_auth; } - layers = ocierofs_get_layers_info(oci, oci->params.registry, - oci->params.repository, - manifest_digest, auth_header, - &layer_count); - if (IS_ERR(layers)) { - ret = PTR_ERR(layers); + ctx->layers = ocierofs_fetch_layers_info(ctx, ctx->registry, + ctx->repository, + ctx->manifest_digest, ctx->auth_header, + &ctx->layer_count); + if (IS_ERR(ctx->layers)) { + ret = PTR_ERR(ctx->layers); erofs_err("failed to get image layers: %s", erofs_strerror(ret)); + ctx->layers = NULL; goto out_manifest; } - if (oci->params.layer_index >= 0) { - if (oci->params.layer_index >= layer_count) { + if (ctx->layer_index >= 0) { + if (ctx->layer_index >= ctx->layer_count) { erofs_err("layer index %d exceeds available layers (%d)", - oci->params.layer_index, layer_count); + ctx->layer_index, ctx->layer_count); ret = -EINVAL; goto out_layers; } - layer_count = 1; - i = oci->params.layer_index; + ctx->layer_count = 1; } else { - i = 0; + ctx->layer_index = 0; } - while (i < layer_count) { - char *trimmed = erofs_trim_for_progressinfo(layers[i], - sizeof("Extracting layer ...") - 1); - erofs_update_progressinfo("Extracting layer %d: %s ...", i, - trimmed); - free(trimmed); - ret = ocierofs_extract_layer(oci, importer, layers[i], - auth_header); - if (ret) { - erofs_err("failed to extract layer %d: %s", i, - erofs_strerror(ret)); - break; - } - i++; - } + return 0; + out_layers: - for (i = 0; i < layer_count; i++) - free(layers[i]); - free(layers); + free(ctx->layers); + ctx->layers = NULL; out_manifest: - free(manifest_digest); + free(ctx->manifest_digest); + ctx->manifest_digest = NULL; out_auth: - free(auth_header); - - if (oci->params.username && oci->params.password && - oci->params.username[0] && oci->params.password[0] && - !auth_header) { - ocierofs_curl_clear_auth(oci->curl); - } -out: + free(ctx->auth_header); + ctx->auth_header = NULL; + if (ctx->using_basic) + ocierofs_curl_clear_auth(ctx); return ret; } -/** - * ocierofs_init - Initialize OCI client with default parameters - * @oci: OCI client structure to initialize - * - * Initialize OCI client structure, set up CURL handle, and configure - * default parameters including platform (linux/amd64), registry - * (registry-1.docker.io), and tag (latest). - * - * Return: 0 on success, negative errno on failure - */ -int ocierofs_init(struct erofs_oci *oci) -{ - if (!oci) - return -EINVAL; - - *oci = (struct erofs_oci){}; - oci->curl = curl_easy_init(); - if (!oci->curl) - return -EIO; - - if (ocierofs_curl_setup_common_options(oci->curl)) { - ocierofs_cleanup(oci); - return -EIO; - } - - if (erofs_oci_params_set_string(&oci->params.platform, - "linux/amd64") || - erofs_oci_params_set_string(&oci->params.registry, - DOCKER_API_REGISTRY) || - erofs_oci_params_set_string(&oci->params.tag, "latest")) { - ocierofs_cleanup(oci); - return -ENOMEM; - } - oci->params.layer_index = -1; /* -1 means extract all layers */ - return 0; -} - -/** - * ocierofs_cleanup - Clean up OCI client and free allocated resources - * @oci: OCI client structure to clean up - * - * Clean up CURL handle, free all allocated string parameters, and - * reset the OCI client structure to a clean state. - */ -void ocierofs_cleanup(struct erofs_oci *oci) -{ - if (!oci) - return; - - if (oci->curl) { - curl_easy_cleanup(oci->curl); - oci->curl = NULL; - } - - free(oci->params.registry); - free(oci->params.repository); - free(oci->params.tag); - free(oci->params.platform); - free(oci->params.username); - free(oci->params.password); - - oci->params.registry = NULL; - oci->params.repository = NULL; - oci->params.tag = NULL; - oci->params.platform = NULL; - oci->params.username = NULL; - oci->params.password = NULL; -} - -int erofs_oci_params_set_string(char **field, const char *value) -{ - char *new_value; - - if (!field) - return -EINVAL; - - if (!value) { - free(*field); - *field = NULL; - return 0; - } - - new_value = strdup(value); - if (!new_value) - return -ENOMEM; - - free(*field); - *field = new_value; - return 0; -} - -int ocierofs_parse_ref(struct erofs_oci *oci, const char *ref_str) +static int ocierofs_parse_ref(struct ocierofs_ctx *ctx, const char *ref_str) { char *slash, *colon, *dot; const char *repo_part; size_t len; + char *tmp; + + if (!ctx || !ref_str) + return -EINVAL; slash = strchr(ref_str, '/'); if (slash) { dot = strchr(ref_str, '.'); if (dot && dot < slash) { - char *registry_str; - len = slash - ref_str; - registry_str = strndup(ref_str, len); - - if (!registry_str) { - erofs_err("failed to allocate memory for registry"); - return -ENOMEM; - } - if (erofs_oci_params_set_string(&oci->params.registry, - registry_str)) { - free(registry_str); - erofs_err("failed to set registry"); + tmp = strndup(ref_str, len); + if (!tmp) return -ENOMEM; - } - free(registry_str); + free(ctx->registry); + ctx->registry = tmp; repo_part = slash + 1; } else { repo_part = ref_str; @@ -1106,73 +940,231 @@ int ocierofs_parse_ref(struct erofs_oci *oci, const char *ref_str) colon = strchr(repo_part, ':'); if (colon) { - char *repo_str; - len = colon - repo_part; - repo_str = strndup(repo_part, len); - - if (!repo_str) { - erofs_err("failed to allocate memory for repository"); + tmp = strndup(repo_part, len); + if (!tmp) return -ENOMEM; - } - if (!strchr(repo_str, '/') && - (!strcmp(oci->params.registry, DOCKER_API_REGISTRY) || - !strcmp(oci->params.registry, DOCKER_REGISTRY))) { + if (!strchr(tmp, '/') && + (!strcmp(ctx->registry, DOCKER_API_REGISTRY) || + !strcmp(ctx->registry, DOCKER_REGISTRY))) { char *full_repo; - if (asprintf(&full_repo, "library/%s", repo_str) == -1) { - free(repo_str); - erofs_err("failed to allocate memory for full repository name"); + if (asprintf(&full_repo, "library/%s", tmp) == -1) { + free(tmp); return -ENOMEM; } - free(repo_str); - repo_str = full_repo; + free(tmp); + tmp = full_repo; } + free(ctx->repository); + ctx->repository = tmp; - if (erofs_oci_params_set_string(&oci->params.repository, - repo_str)) { - free(repo_str); - erofs_err("failed to set repository"); + free(ctx->tag); + ctx->tag = strdup(colon + 1); + if (!ctx->tag) return -ENOMEM; - } - free(repo_str); - - if (erofs_oci_params_set_string(&oci->params.tag, - colon + 1)) { - erofs_err("failed to set tag"); - return -ENOMEM; - } } else { - char *repo_str = strdup(repo_part); - - if (!repo_str) { - erofs_err("failed to allocate memory for repository"); + tmp = strdup(repo_part); + if (!tmp) return -ENOMEM; - } - if (!strchr(repo_str, '/') && - (!strcmp(oci->params.registry, DOCKER_API_REGISTRY) || - !strcmp(oci->params.registry, DOCKER_REGISTRY))) { + if (!strchr(tmp, '/') && + (!strcmp(ctx->registry, DOCKER_API_REGISTRY) || + !strcmp(ctx->registry, DOCKER_REGISTRY))) { + char *full_repo; - if (asprintf(&full_repo, "library/%s", repo_str) == -1) { - free(repo_str); - erofs_err("failed to allocate memory for full repository name"); + if (asprintf(&full_repo, "library/%s", tmp) == -1) { + free(tmp); return -ENOMEM; } - free(repo_str); - repo_str = full_repo; + free(tmp); + tmp = full_repo; } + free(ctx->repository); + ctx->repository = tmp; + } + return 0; +} - if (erofs_oci_params_set_string(&oci->params.repository, - repo_str)) { - free(repo_str); - erofs_err("failed to set repository"); - return -ENOMEM; +int ocierofs_init(struct ocierofs_ctx *ctx, const struct ocierofs_config *config) +{ + int ret; + + ctx->curl = curl_easy_init(); + if (!ctx->curl) + return -EIO; + + if (ocierofs_curl_setup_common_options(ctx->curl)) + return -EIO; + + ctx->layer_index = -1; + ctx->registry = strdup("registry-1.docker.io"); + ctx->tag = strdup("latest"); + if (config && config->platform) + ctx->platform = strdup(config->platform); + else + ctx->platform = strdup("linux/amd64"); + if (!ctx->registry || !ctx->tag || !ctx->platform) + return -ENOMEM; + + if (config && config->layer_index >= 0) + ctx->layer_index = config->layer_index; + + ret = ocierofs_parse_ref(ctx, config->image_ref); + if (ret) + return ret; + + ret = ocierofs_prepare_layers(ctx, config); + if (ret) + return ret; + + return 0; +} + +static int ocierofs_download_blob_to_fd(struct ocierofs_ctx *ctx, + const char *digest, + const char *auth_header, + int outfd) +{ + struct ocierofs_request req = {}; + struct ocierofs_stream stream = {}; + const char *api_registry; + long http_code; + int ret; + + stream = (struct ocierofs_stream) { + .digest = digest, + .blobfd = outfd, + }; + + api_registry = ocierofs_get_api_registry(ctx->registry); + if (asprintf(&req.url, "https://%s/v2/%s/blobs/%s", + api_registry, ctx->repository, digest) == -1) + return -ENOMEM; + + if (auth_header && strstr(auth_header, "Bearer")) + req.headers = curl_slist_append(req.headers, auth_header); + + curl_easy_reset(ctx->curl); + + ret = ocierofs_curl_setup_common_options(ctx->curl); + if (ret) + goto out; + + ret = ocierofs_curl_setup_rq(ctx->curl, req.url, OCIEROFS_HTTP_GET, + req.headers, + ocierofs_layer_write_callback, + &stream, NULL, NULL); + if (ret) + goto out; + + ret = ocierofs_curl_perform(ctx->curl, &http_code); + if (ret) + goto out; + + if (http_code < 200 || http_code >= 300) { + erofs_err("HTTP request failed with code %ld", http_code); + ret = -EIO; + goto out; + } + ret = 0; +out: + ocierofs_request_cleanup(&req); + return ret; +} + +static int ocierofs_extract_layer(struct ocierofs_ctx *ctx, + const char *digest, const char *auth_header) +{ + struct ocierofs_stream stream = {}; + int ret; + + stream = (struct ocierofs_stream) { + .digest = digest, + .blobfd = erofs_tmpfile(), + }; + if (stream.blobfd < 0) { + erofs_err("failed to create temporary file for %s", digest); + return -errno; + } + + ret = ocierofs_download_blob_to_fd(ctx, digest, auth_header, stream.blobfd); + if (ret) + goto out; + + if (lseek(stream.blobfd, 0, SEEK_SET) < 0) { + erofs_err("failed to seek to beginning of temp file: %s", + strerror(errno)); + ret = -errno; + goto out; + } + + return stream.blobfd; + +out: + if (stream.blobfd >= 0) + close(stream.blobfd); + return ret; +} + +static void ocierofs_ctx_cleanup(struct ocierofs_ctx *ctx) +{ + if (!ctx) + return; + + if (ctx->curl) { + curl_easy_cleanup(ctx->curl); + ctx->curl = NULL; + } + free(ctx->auth_header); + ctx->auth_header = NULL; + + ocierofs_free_layers_info(ctx->layers, ctx->layer_count); + free(ctx->registry); + free(ctx->repository); + free(ctx->tag); + free(ctx->platform); + free(ctx->manifest_digest); +} + +int ocierofs_build_trees(struct erofs_importer *importer, + const struct ocierofs_config *config) +{ + struct ocierofs_ctx ctx = {}; + int ret, i; + + ret = ocierofs_init(&ctx, config); + if (ret) { + ocierofs_ctx_cleanup(&ctx); + return ret; + } + + i = ctx.layer_index; + while (i < ctx.layer_count) { + char *trimmed = erofs_trim_for_progressinfo(ctx.layers[i]->digest, + sizeof("Extracting layer ...") - 1); + erofs_update_progressinfo("Extracting layer %d: %s ...", i, + trimmed); + free(trimmed); + int fd = ocierofs_extract_layer(&ctx, ctx.layers[i]->digest, + ctx.auth_header); + if (fd < 0) { + erofs_err("failed to extract layer %d: %s", i, + erofs_strerror(fd)); + break; } - free(repo_str); + ret = ocierofs_process_tar_stream(importer, fd); + close(fd); + if (ret) { + erofs_err("failed to process tar stream for layer %d: %s", i, + erofs_strerror(ret)); + break; + } + i++; } - return 0; + ocierofs_ctx_cleanup(&ctx); + return ret; } diff --git a/mkfs/main.c b/mkfs/main.c index 064392d..721c0f0 100644 --- a/mkfs/main.c +++ b/mkfs/main.c @@ -150,18 +150,18 @@ static void usage(int argc, char **argv) * "0-9,100-109" instead of a continuous "0-109", and to * state what those two subranges respectively mean. */ printf("%s [,level=<0-9,100-109>]\t0-9=normal, 100-109=extreme (default=%i)\n", - spaces, s->c->default_level); + spaces, s->c->default_level); else printf("%s [,level=<0-%i>]\t\t(default=%i)\n", - spaces, s->c->best_level, s->c->default_level); + spaces, s->c->best_level, s->c->default_level); } if (s->c->setdictsize) { if (s->c->default_dictsize) printf("%s [,dictsize=<dictsize>]\t(default=%u, max=%u)\n", - spaces, s->c->default_dictsize, s->c->max_dictsize); + spaces, s->c->default_dictsize, s->c->max_dictsize); else printf("%s [,dictsize=<dictsize>]\t(default=<auto>, max=%u)\n", - spaces, s->c->max_dictsize); + spaces, s->c->max_dictsize); } } printf( @@ -272,7 +272,7 @@ static struct erofs_s3 s3cfg; #endif #ifdef OCIEROFS_ENABLED -static struct erofs_oci ocicfg; +static struct ocierofs_config ocicfg; static char *mkfs_oci_options; #endif @@ -689,21 +689,14 @@ static int mkfs_parse_s3_cfg(char *cfg_str) #endif #ifdef OCIEROFS_ENABLED - - -/** - * mkfs_parse_oci_options - Parse comma-separated OCI options string +/* + * mkfs_parse_oci_options - Parse OCI options for mkfs tool + * @cfg: OCI configuration structure to update * @options_str: comma-separated options string - * - * Parse OCI options string containing comma-separated key=value pairs. - * Supported options include platform, layer, username, and password. - * - * Return: 0 on success, negative errno on failure */ -static int mkfs_parse_oci_options(char *options_str) +static int mkfs_parse_oci_options(struct ocierofs_config *oci_cfg, char *options_str) { char *opt, *q, *p; - int ret; if (!options_str) return 0; @@ -717,41 +710,43 @@ static int mkfs_parse_oci_options(char *options_str) p = strstr(opt, "platform="); if (p) { p += strlen("platform="); - ret = erofs_oci_params_set_string(&ocicfg.params.platform, p); - if (ret) { - erofs_err("failed to set platform"); - return ret; - } + free(oci_cfg->platform); + oci_cfg->platform = strdup(p); + if (!oci_cfg->platform) + return -ENOMEM; } else { p = strstr(opt, "layer="); if (p) { p += strlen("layer="); - ocicfg.params.layer_index = atoi(p); - if (ocicfg.params.layer_index < 0) { - erofs_err("invalid layer index %d", - ocicfg.params.layer_index); - return -EINVAL; + { + char *endptr; + unsigned long v = strtoul(p, &endptr, 10); + + if (endptr == p || *endptr != '\0') { + erofs_err("invalid layer index %s", + p); + return -EINVAL; + } + oci_cfg->layer_index = (int)v; } } else { p = strstr(opt, "username="); if (p) { p += strlen("username="); - ret = erofs_oci_params_set_string(&ocicfg.params.username, p); - if (ret) { - erofs_err("failed to set username"); - return ret; - } + free(oci_cfg->username); + oci_cfg->username = strdup(p); + if (!oci_cfg->username) + return -ENOMEM; } else { p = strstr(opt, "password="); if (p) { p += strlen("password="); - ret = erofs_oci_params_set_string(&ocicfg.params.password, p); - if (ret) { - erofs_err("failed to set password"); - return ret; - } + free(oci_cfg->password); + oci_cfg->password = strdup(p); + if (!oci_cfg->password) + return -ENOMEM; } else { - erofs_err("invalid --oci value %s", opt); + erofs_err("mkfs: invalid --oci value %s", opt); return -EINVAL; } } @@ -1832,16 +1827,12 @@ int main(int argc, char **argv) #endif #ifdef OCIEROFS_ENABLED } else if (source_mode == EROFS_MKFS_SOURCE_OCI) { - err = ocierofs_init(&ocicfg); - if (err) - goto exit; + ocicfg.layer_index = -1; - err = mkfs_parse_oci_options(mkfs_oci_options); - if (err) - goto exit; - err = ocierofs_parse_ref(&ocicfg, cfg.c_src_path); + err = mkfs_parse_oci_options(&ocicfg, mkfs_oci_options); if (err) goto exit; + ocicfg.image_ref = cfg.c_src_path; if (incremental_mode || dataimport_mode == EROFS_MKFS_DATA_IMPORT_RVSP || @@ -1916,9 +1907,6 @@ exit: erofs_blob_exit(); erofs_xattr_cleanup_name_prefixes(); erofs_rebuild_cleanup(); -#ifdef OCIEROFS_ENABLED - ocierofs_cleanup(&ocicfg); -#endif erofs_diskbuf_exit(); if (source_mode == EROFS_MKFS_SOURCE_TAR) { erofs_iostream_close(&erofstar.ios); -- 2.51.0 ^ permalink raw reply related [flat|nested] 19+ messages in thread
* Re: [PATCH v4 2/3] erofs-utils: refactor OCI code for better modularity 2025-09-04 5:33 ` [PATCH v4 2/3] erofs-utils: refactor OCI code for better modularity ChengyuZhu6 @ 2025-09-04 5:53 ` Gao Xiang 0 siblings, 0 replies; 19+ messages in thread From: Gao Xiang @ 2025-09-04 5:53 UTC (permalink / raw) To: ChengyuZhu6, linux-erofs; +Cc: xiang, Chengyu Zhu Hi Chengyu, On 2025/9/4 13:33, ChengyuZhu6 wrote: > From: Chengyu Zhu <hudsonzhu@tencent.com> .. > @@ -96,6 +142,14 @@ static int ocierofs_curl_setup_common_options(struct CURL *curl) > curl_easy_setopt(curl, CURLOPT_TIMEOUT, 120L); > curl_easy_setopt(curl, CURLOPT_NOSIGNAL, 1L); > curl_easy_setopt(curl, CURLOPT_USERAGENT, "ocierofs/" PACKAGE_VERSION); > + curl_easy_setopt(curl, CURLOPT_ACCEPT_ENCODING, ""); > + curl_easy_setopt(curl, CURLOPT_TCP_KEEPALIVE, 1L); > +#if defined(CURLOPT_TCP_KEEPIDLE) > + curl_easy_setopt(curl, CURLOPT_TCP_KEEPIDLE, 30L); > +#endif > +#if defined(CURLOPT_TCP_KEEPINTVL) > + curl_easy_setopt(curl, CURLOPT_TCP_KEEPINTVL, 15L); > +#endif > return 0; > } > > @@ -114,10 +168,10 @@ static int ocierofs_curl_setup_basic_auth(struct CURL *curl, const char *usernam > return 0; > } > > -static int ocierofs_curl_clear_auth(struct CURL *curl) > +static int ocierofs_curl_clear_auth(struct ocierofs_ctx *ctx) > { > - curl_easy_setopt(curl, CURLOPT_USERPWD, NULL); > - curl_easy_setopt(curl, CURLOPT_HTTPAUTH, CURLAUTH_NONE); > + curl_easy_setopt(ctx->curl, CURLOPT_USERPWD, NULL); > + curl_easy_setopt(ctx->curl, CURLOPT_HTTPAUTH, CURLAUTH_NONE); > return 0; > } > > @@ -176,20 +230,20 @@ static int ocierofs_curl_perform(struct CURL *curl, long *http_code_out) > return 0; > } > > -static int ocierofs_request_perform(struct erofs_oci *oci, > - struct erofs_oci_request *req, > - struct erofs_oci_response *resp) > +static int ocierofs_request_perform(struct ocierofs_ctx *ctx, > + ... > > @@ -204,15 +258,10 @@ static int ocierofs_request_perform(struct erofs_oci *oci, > * @realm_out: pointer to store realm value > * @service_out: pointer to store service value > * @scope_out: pointer to store scope value > - * > - * Parse Bearer authentication header and extract realm, service, and scope > - * parameters for subsequent token requests. > - * > - * Return: 0 on success, negative errno on failure Why do we drop this comment? > */ > static int ocierofs_parse_auth_header(const char *auth_header, > - char **realm_out, char **service_out, > - char **scope_out) > + char **realm_out, char **service_out, > + char **scope_out) Let's not adjust lines which are unrelated to the commit. > { > char *realm = NULL, *service = NULL, *scope = NULL; > static const char * const param_names[] = {"realm=", "service=", "scope="}; > @@ -221,7 +270,6 @@ static int ocierofs_parse_auth_header(const char *auth_header, > const char *p; > int i, ret = 0; > > - // https://datatracker.ietf.org/doc/html/rfc6750#section-3 Same here. > if (strncmp(auth_header, "Bearer ", strlen("Bearer "))) > return -EINVAL; > > @@ -229,7 +277,6 @@ static int ocierofs_parse_auth_header(const char *auth_header, > if (!header_copy) > return -ENOMEM; > > - /* Clean up header: replace newlines with spaces and remove double spaces */ Same here. If they are unrelated to the new feature, let's refine ocierofs_parse_auth_header() later. > for (char *q = header_copy; *q; q++) { > if (*q == '\n' || *q == '\r') > *q = ' '; > @@ -277,22 +324,9 @@ out: > return ret; > } > > -/** > - * ocierofs_extract_www_auth_info - Extract WWW-Authenticate header information > - * @resp_data: HTTP response data containing headers > - * @realm_out: pointer to store realm value (optional) > - * @service_out: pointer to store service value (optional) > - * @scope_out: pointer to store scope value (optional) > - * > - * Extract realm, service, and scope from WWW-Authenticate header in HTTP response. > - * This function handles the common pattern of parsing WWW-Authenticate headers > - * that appears in multiple places in the OCI authentication flow. > - * > - * Return: 0 on success, negative errno on failure > - */ Same here. > static int ocierofs_extract_www_auth_info(const char *resp_data, > - char **realm_out, char **service_out, > - char **scope_out) > + char **realm_out, char **service_out, > + char **scope_out) Same here. > {> char *www_auth; > char *line_end; > @@ -336,29 +370,12 @@ static int ocierofs_extract_www_auth_info(const char *resp_data, > return ret; > } > > -/** > - * ocierofs_get_auth_token_with_url - Get authentication token from auth server > - * @oci: OCI client structure > - * @auth_url: authentication server URL > - * @service: service name for authentication > - * @repository: repository name > - * @username: username for basic auth (optional) > - * @password: password for basic auth (optional) > - * > - * Request authentication token from the specified auth server URL using > - * basic authentication if credentials are provided. > - * > - * Return: authentication header string on success, ERR_PTR on failure > - */ > -static char *ocierofs_get_auth_token_with_url(struct erofs_oci *oci, > - const char *auth_url, > - const char *service, > - const char *repository, > - const char *username, > - const char *password) > +static char *ocierofs_get_auth_token_with_url(struct ocierofs_ctx *ctx, const char *auth_url, > + const char *service, const char *repository, > + const char *username, const char *password) Since the arguments are changed, I'm fine with this change. > { > - struct erofs_oci_request req = {}; > - struct erofs_oci_response resp = {}; > + struct ocierofs_request req = {}; > + struct ocierofs_response resp = {}; > json_object *root, *token_obj, *access_token_obj; > const char *token; > char *auth_header = NULL; > @@ -373,40 +390,36 @@ static char *ocierofs_get_auth_token_with_url(struct erofs_oci *oci, > } > > if (username && password && *username) { > - ret = ocierofs_curl_setup_basic_auth(oci->curl, username, > - password); > + ret = ocierofs_curl_setup_basic_auth(ctx->curl, username, > + password); Password can be aligned with `(` in the previous line? > if (ret) > goto out_url; > } > > - ret = ocierofs_request_perform(oci, &req, &resp); > - ocierofs_curl_clear_auth(oci->curl); > + ret = ocierofs_request_perform(ctx, &req, &resp); > + ocierofs_curl_clear_auth(ctx); > if (ret) > goto out_url; > > if (!resp.data) { > - erofs_err("empty response from auth server"); Why drop this? > ret = -EINVAL; > goto out_url; > } > > root = json_tokener_parse(resp.data); > if (!root) { > - erofs_err("failed to parse auth response"); Why drop this? > ret = -EINVAL; > - goto out_url; > + goto out_json; > } > > if (!json_object_object_get_ex(root, "token", &token_obj) && > !json_object_object_get_ex(root, "access_token", &access_token_obj)) { > - erofs_err("no token found in auth response"); Why drop this? > ret = -EINVAL; > goto out_json; > } > > token = json_object_get_string(token_obj ? token_obj : access_token_obj); > if (!token) { > - erofs_err("invalid token in auth response"); Why drop this? ... > auth_service = discovered_service ? discovered_service : service; > - auth_header = ocierofs_get_auth_token_with_url(oci, discovered_auth_url, > - auth_service, repository, > - username, password); > + auth_header = ocierofs_get_auth_token_with_url(ctx, discovered_auth_url, > + auth_service, repository, > + username, password); Same here. > free(discovered_auth_url); > free(discovered_service); > if (!IS_ERR(auth_header)) > @@ -544,9 +555,9 @@ static char *ocierofs_get_auth_token(struct erofs_oci *oci, const char *registry > if (asprintf(&auth_url, auth_patterns[i], registry) < 0) > continue; > > - auth_header = ocierofs_get_auth_token_with_url(oci, auth_url, > - service, repository, > - username, password); > + auth_header = ocierofs_get_auth_token_with_url(ctx, auth_url, > + service, repository, > + username, password); Same here. > free(auth_url); > > if (!IS_ERR(auth_header)) > @@ -557,24 +568,21 @@ static char *ocierofs_get_auth_token(struct erofs_oci *oci, const char *registry > return ERR_PTR(-ENOENT); > } > > -static char *ocierofs_get_manifest_digest(struct erofs_oci *oci, > - const char *registry, > - const char *repository, const char *tag, > - const char *platform, > - const char *auth_header) > +static char *ocierofs_get_manifest_digest(struct ocierofs_ctx *ctx, > + const char *registry, > + const char *repository, const char *tag, > + const char *platform, > + const char *auth_header) > { > - struct erofs_oci_request req = {}; > - struct erofs_oci_response resp = {}; > + struct ocierofs_request req = {}; > + struct ocierofs_response resp = {}; > json_object *root, *manifests, *manifest, *platform_obj, *arch_obj; > json_object *os_obj, *digest_obj, *schema_obj, *media_type_obj; > char *digest = NULL; > const char *api_registry; > int ret = 0, len, i; > > - if (!registry || !repository || !tag || !platform) > - return ERR_PTR(-EINVAL); > - > - api_registry = (!strcmp(registry, DOCKER_REGISTRY)) ? DOCKER_API_REGISTRY : registry; > + api_registry = ocierofs_get_api_registry(registry); > if (asprintf(&req.url, "https://%s/v2/%s/manifests/%s", > api_registry, repository, tag) < 0) > return ERR_PTR(-ENOMEM); > @@ -584,22 +592,20 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci, > > req.headers = curl_slist_append(req.headers, > "Accept: " DOCKER_MEDIATYPE_MANIFEST_LIST "," > - OCI_MEDIATYPE_INDEX "," DOCKER_MEDIATYPE_MANIFEST_V1 "," > - DOCKER_MEDIATYPE_MANIFEST_V2); > + OCI_MEDIATYPE_INDEX "," OCI_MEDIATYPE_MANIFEST "," > + DOCKER_MEDIATYPE_MANIFEST_V1 "," DOCKER_MEDIATYPE_MANIFEST_V2); > > - ret = ocierofs_request_perform(oci, &req, &resp); > + ret = ocierofs_request_perform(ctx, &req, &resp); > if (ret) > goto out; > > if (!resp.data) { > - erofs_err("empty response from manifest request"); Same here. > ret = -EINVAL; > goto out; > } > > root = json_tokener_parse(resp.data); > if (!root) { > - erofs_err("failed to parse manifest JSON"); Same here. > ret = -EINVAL; > goto out; > } > @@ -615,8 +621,7 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci, > if (json_object_object_get_ex(root, "mediaType", &media_type_obj)) { > const char *media_type = json_object_get_string(media_type_obj); > > - if (!strcmp(media_type, DOCKER_MEDIATYPE_MANIFEST_V2) || > - !strcmp(media_type, OCI_MEDIATYPE_MANIFEST)) { > + if (ocierofs_is_manifest(media_type)) { > digest = strdup(tag); > ret = 0; > goto out_json; > @@ -624,7 +629,6 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci, > } > > if (!json_object_object_get_ex(root, "manifests", &manifests)) { > - erofs_err("no manifests found in manifest list"); Same here. > ret = -EINVAL; > goto out_json; > } > @@ -634,9 +638,9 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci, > manifest = json_object_array_get_idx(manifests, i); > > if (json_object_object_get_ex(manifest, "platform", > - &platform_obj) && > + &platform_obj) && No need to change. > json_object_object_get_ex(platform_obj, "architecture", > - &arch_obj) && > + &arch_obj) && No need to change. ... > - } > + free(oci_cfg->password); > + oci_cfg->password = strdup(p); > + if (!oci_cfg->password) > + return -ENOMEM; > } else { > - erofs_err("invalid --oci value %s", opt); > + erofs_err("mkfs: invalid --oci value %s", opt); Unneeded `mkfs:`. Thanks, Gao Xiang ^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH v4 3/3] erofs-utils: add NBD-backed OCI image mounting 2025-09-04 5:33 ` [PATCH v4 0/3] erofs-utils: refactor OCI and " ChengyuZhu6 2025-09-04 5:33 ` [PATCH v4 1/3] erofs-utils:mkfs: move parse_oci_ref to lib ChengyuZhu6 2025-09-04 5:33 ` [PATCH v4 2/3] erofs-utils: refactor OCI code for better modularity ChengyuZhu6 @ 2025-09-04 5:33 ` ChengyuZhu6 2 siblings, 0 replies; 19+ messages in thread From: ChengyuZhu6 @ 2025-09-04 5:33 UTC (permalink / raw) To: linux-erofs; +Cc: xiang, hsiangkao, Chengyu Zhu From: Chengyu Zhu <hudsonzhu@tencent.com> - Add HTTP range downloads for OCI blobs - Introduce ocierofs_iostream for virtual file I/O - Add --oci option for OCI image mounting with NBD backend New mount.erofs -t erofs.nbd option: -o=[options] source-image mountpoint Supported oci options: - oci.platform=os/arch (default: linux/amd64) - oci=N (extract specific layer, default: all layers) - oci.username/oci.password (basic authentication) e.g.: ./mount/mount.erofs -t erofs.nbd -o 'oci=1,oci.platform=linux/amd64' \ quay.io/chengyuzhu6/golang:1.22.8-erofs /tmp/test/ Signed-off-by: Chengyu Zhu <hudsonzhu@tencent.com> --- lib/liberofs_oci.h | 14 +++ lib/remotes/oci.c | 241 +++++++++++++++++++++++++++++++++++++++++- mount/Makefile.am | 2 +- mount/main.c | 257 ++++++++++++++++++++++++++++++++++++++------- 4 files changed, 476 insertions(+), 38 deletions(-) diff --git a/lib/liberofs_oci.h b/lib/liberofs_oci.h index 8622464..71ea333 100644 --- a/lib/liberofs_oci.h +++ b/lib/liberofs_oci.h @@ -51,6 +51,12 @@ struct ocierofs_ctx { int layer_count; }; +struct ocierofs_iostream { + struct ocierofs_ctx *ctx; + struct erofs_vfile vf; + u64 offset; +}; + int ocierofs_init(struct ocierofs_ctx *ctx, const struct ocierofs_config *config); /* @@ -63,6 +69,14 @@ int ocierofs_init(struct ocierofs_ctx *ctx, const struct ocierofs_config *config int ocierofs_build_trees(struct erofs_importer *importer, const struct ocierofs_config *cfg); +int ocierofs_is_erofs_native_image(struct ocierofs_ctx *ctx); + +void ocierofs_ctx_cleanup(struct ocierofs_ctx *ctx); + +int ocierofs_iostream_open(struct ocierofs_iostream *oci_iostream, struct ocierofs_ctx *oci_ctx); + +void ocierofs_iostream_close(struct ocierofs_iostream *oci_iostream); + #ifdef __cplusplus } #endif diff --git a/lib/remotes/oci.c b/lib/remotes/oci.c index 4291dc6..0fdff1f 100644 --- a/lib/remotes/oci.c +++ b/lib/remotes/oci.c @@ -33,6 +33,9 @@ #define OCI_MEDIATYPE_MANIFEST "application/vnd.oci.image.manifest.v1+json" #define OCI_MEDIATYPE_INDEX "application/vnd.oci.image.index.v1+json" +/* Erofs Native Layer Media Type */ +#define EROFS_MEDIATYPE "application/vnd.erofs" + struct ocierofs_request { char *url; struct curl_slist *headers; @@ -1109,7 +1112,7 @@ out: return ret; } -static void ocierofs_ctx_cleanup(struct ocierofs_ctx *ctx) +void ocierofs_ctx_cleanup(struct ocierofs_ctx *ctx) { if (!ctx) return; @@ -1168,3 +1171,239 @@ int ocierofs_build_trees(struct erofs_importer *importer, ocierofs_ctx_cleanup(&ctx); return ret; } + +static int ocierofs_download_blob_range(struct ocierofs_ctx *ctx, off_t offset, size_t length, + void **out_buf, size_t *out_size) +{ + struct ocierofs_request req = {}; + struct ocierofs_response resp = {}; + const char *api_registry; + char rangehdr[64]; + long http_code = 0; + int ret; + int index = ctx->layer_index; + + if (offset < 0) + return -EINVAL; + + api_registry = ocierofs_get_api_registry(ctx->registry); + if (asprintf(&req.url, "https://%s/v2/%s/blobs/%s", + api_registry, ctx->repository, ctx->layers[index]->digest) == -1) + return -ENOMEM; + + if (length) + snprintf(rangehdr, sizeof(rangehdr), "Range: bytes=%lld-%lld", + (long long)offset, (long long)(offset + (off_t)length - 1)); + else + snprintf(rangehdr, sizeof(rangehdr), "Range: bytes=%lld-", + (long long)offset); + + if (ctx->auth_header && strstr(ctx->auth_header, "Bearer")) + req.headers = curl_slist_append(req.headers, ctx->auth_header); + req.headers = curl_slist_append(req.headers, rangehdr); + + curl_easy_reset(ctx->curl); + + ret = ocierofs_curl_setup_common_options(ctx->curl); + if (ret) + goto out; + + ret = ocierofs_curl_setup_rq(ctx->curl, req.url, OCIEROFS_HTTP_GET, + req.headers, + ocierofs_write_callback, + &resp, NULL, NULL); + if (ret) + goto out; + + ret = ocierofs_curl_perform(ctx->curl, &http_code); + if (ret) + goto out; + + if (http_code == 206) { + *out_buf = resp.data; + *out_size = resp.size; + resp.data = NULL; + ret = 0; + } else if (http_code == 200) { + if (offset == 0) { + *out_buf = resp.data; + *out_size = resp.size; + resp.data = NULL; + ret = 0; + } else { + if (offset < resp.size) { + size_t available = resp.size - offset; + size_t copy_size = length ? min_t(size_t, length, available) : available; + + *out_buf = malloc(copy_size); + if (!*out_buf) { + ret = -ENOMEM; + goto out; + } + memcpy(*out_buf, resp.data + offset, copy_size); + *out_size = copy_size; + ret = 0; + } else { + *out_buf = NULL; + *out_size = 0; + ret = 0; + } + } + } else { + erofs_err("HTTP range request failed with code %ld", http_code); + ret = -EIO; + } + +out: + if (req.headers) + curl_slist_free_all(req.headers); + free(req.url); + free(resp.data); + return ret; +} + +static ssize_t ocierofs_io_pread(struct erofs_vfile *vf, void *buf, size_t len, u64 offset) +{ + struct ocierofs_iostream *oci_iostream = *(struct ocierofs_iostream **)vf->payload; + void *download_buf = NULL; + size_t download_size = 0; + ssize_t ret; + + ret = ocierofs_download_blob_range(oci_iostream->ctx, offset, len, + &download_buf, &download_size); + if (ret < 0) { + memset(buf, 0, len); + return len; + } + + if (download_buf && download_size > 0) { + memcpy(buf, download_buf, download_size); + free(download_buf); + return download_size; + } + + return 0; +} + +static ssize_t ocierofs_io_read(struct erofs_vfile *vf, void *buf, size_t len) +{ + struct ocierofs_iostream *oci_iostream = *(struct ocierofs_iostream **)vf->payload; + ssize_t ret; + + ret = ocierofs_io_pread(vf, buf, len, oci_iostream->offset); + if (ret > 0) + oci_iostream->offset += ret; + + return ret; +} + +static off_t ocierofs_io_lseek(struct erofs_vfile *vf, u64 offset, int whence) +{ + struct ocierofs_iostream *oci_iostream = *(struct ocierofs_iostream **)vf->payload; + off_t new_offset; + int layer_index = oci_iostream->ctx->layer_index; + + switch (whence) { + case SEEK_SET: + new_offset = offset; + break; + case SEEK_CUR: + new_offset = oci_iostream->offset + offset; + break; + case SEEK_END: + new_offset = oci_iostream->ctx->layers[layer_index]->size + offset; + break; + default: + return -1; + } + + if (new_offset < 0 || new_offset > oci_iostream->ctx->layers[layer_index]->size) + return -1; + + oci_iostream->offset = new_offset; + return new_offset; +} + +static ssize_t ocierofs_io_sendfile(struct erofs_vfile *vout, struct erofs_vfile *vin, + off_t *pos, size_t count) +{ + struct ocierofs_iostream *oci_iostream = *(struct ocierofs_iostream **)vin->payload; + char *buf = NULL; + ssize_t total_written = 0; + ssize_t ret = 0; + + buf = malloc(min_t(size_t, count, 32768)); + if (!buf) + return -ENOMEM; + + while (count > 0) { + size_t to_read = min_t(size_t, count, 32768); + u64 read_offset = pos ? *pos : oci_iostream->offset; + + ret = ocierofs_io_pread(vin, buf, to_read, read_offset); + if (ret <= 0) { + erofs_err("OCI I/O sendfile: failed to read from OCI: %s", + erofs_strerror(ret)); + memset(buf, 0, to_read); + ret = to_read; + } + + ssize_t written = write(vout->fd, buf, ret); + + if (written < 0) { + erofs_err("OCI I/O sendfile: failed to write to output: %s", + strerror(errno)); + ret = -errno; + break; + } + + if (written != ret) { + erofs_err("OCI I/O sendfile: partial write: %zd != %zd", written, ret); + ret = written; + } + + total_written += ret; + count -= ret; + if (pos) + *pos += ret; + else + oci_iostream->offset += ret; + } + + free(buf); + return count; +} + +static struct erofs_vfops ocierofs_io_vfops = { + .pread = ocierofs_io_pread, + .read = ocierofs_io_read, + .lseek = ocierofs_io_lseek, + .sendfile = ocierofs_io_sendfile, +}; + +int ocierofs_iostream_open(struct ocierofs_iostream *oci_iostream, struct ocierofs_ctx *oci_ctx) +{ + memset(oci_iostream, 0, sizeof(*oci_iostream)); + oci_iostream->ctx = oci_ctx; + oci_iostream->vf.ops = &ocierofs_io_vfops; + oci_iostream->vf.fd = -1; + *(struct ocierofs_iostream **)oci_iostream->vf.payload = oci_iostream; + + return 0; +} + +void ocierofs_iostream_close(struct ocierofs_iostream *oci_iostream) +{ + close(oci_iostream->vf.fd); +} + +int ocierofs_is_erofs_native_image(struct ocierofs_ctx *ctx) +{ + if (ctx->layer_count > 0 && ctx->layers[0] && + ctx->layers[0]->media_type) { + if (strcmp(ctx->layers[0]->media_type, EROFS_MEDIATYPE) != 0) + return -ENOENT; + return 0; + } + return -ENOENT; +} diff --git a/mount/Makefile.am b/mount/Makefile.am index d93f3f4..0b4447f 100644 --- a/mount/Makefile.am +++ b/mount/Makefile.am @@ -9,5 +9,5 @@ mount_erofs_SOURCES = main.c mount_erofs_CFLAGS = -Wall -I$(top_srcdir)/include mount_erofs_LDADD = $(top_builddir)/lib/liberofs.la ${libselinux_LIBS} \ ${liblz4_LIBS} ${liblzma_LIBS} ${zlib_LIBS} ${libdeflate_LIBS} \ - ${libzstd_LIBS} ${libqpl_LIBS} ${libxxhash_LIBS} ${libnl3_LIBS} + ${libzstd_LIBS} ${libqpl_LIBS} ${libxxhash_LIBS} ${libnl3_LIBS} ${openssl_LIBS} endif diff --git a/mount/main.c b/mount/main.c index a270f0a..177bca3 100644 --- a/mount/main.c +++ b/mount/main.c @@ -15,6 +15,7 @@ #include "erofs/err.h" #include "erofs/io.h" #include "../lib/liberofs_nbd.h" +#include "../lib/liberofs_oci.h" #ifdef HAVE_LINUX_LOOP_H #include <linux/loop.h> #else @@ -34,6 +35,10 @@ struct loop_info { #include <sys/sysmacros.h> #endif +#ifdef OCIEROFS_ENABLED +static struct ocierofs_config ocicfg; +#endif + enum erofs_backend_drv { EROFSAUTO, EROFSLOCAL, @@ -56,12 +61,76 @@ static struct erofsmount_cfg { long flags; enum erofs_backend_drv backend; enum erofsmount_mode mountmode; +#ifdef OCIEROFS_ENABLED + bool use_oci; +#endif } mountcfg = { .full_options = "ro", .flags = MS_RDONLY, /* default mountflags */ .fstype = "erofs", }; +enum erofs_nbd_source_type { + EROFSNBD_SOURCE_LOCAL, + EROFSNBD_SOURCE_OCI, +}; + +union erofs_nbd_source { + const char *device_path; + struct ocierofs_ctx *oci_ctx; +}; + +union erofs_nbd_source src; + +static int parse_oci_option(struct ocierofs_config *oci_cfg, const char *option) +{ + char *p; + + p = strstr(option, "oci="); + if (p != NULL) { + p += strlen("oci="); + { + char *endptr; + unsigned long v = strtoul(p, &endptr, 10); + + if (endptr == p || *endptr != '\0') + return -EINVAL; + oci_cfg->layer_index = (int)v; + } + } else { + p = strstr(option, "oci.platform="); + if (p != NULL) { + p += strlen("oci.platform="); + free(oci_cfg->platform); + oci_cfg->platform = strdup(p); + if (!oci_cfg->platform) + return -ENOMEM; + } else { + p = strstr(option, "oci.username="); + if (p != NULL) { + p += strlen("oci.username="); + free(oci_cfg->username); + oci_cfg->username = strdup(p); + if (!oci_cfg->username) + return -ENOMEM; + } else { + p = strstr(option, "oci.password="); + if (p != NULL) { + p += strlen("oci.password="); + free(oci_cfg->password); + oci_cfg->password = strdup(p); + if (!oci_cfg->password) + return -ENOMEM; + } else { + return -EINVAL; + } + } + } + } + + return 0; +} + static long erofsmount_parse_flagopts(char *s, long flags, char **more) { static const struct { @@ -90,29 +159,41 @@ static long erofsmount_parse_flagopts(char *s, long flags, char **more) comma = strchr(s, ','); if (comma) *comma = '\0'; - for (i = 0; i < ARRAY_SIZE(opts); ++i) { - if (!strcasecmp(s, opts[i].name)) { - if (opts[i].flags < 0) - flags &= opts[i].flags; - else - flags |= opts[i].flags; - break; - } - } - if (more && i >= ARRAY_SIZE(opts)) { - int sl = strlen(s); - char *new = *more; + if (strncmp(s, "oci", 3) == 0) { +#ifdef OCIEROFS_ENABLED + int err = parse_oci_option(&ocicfg, s); - i = new ? strlen(new) : 0; - new = realloc(new, i + strlen(s) + 2); - if (!new) - return -ENOMEM; - if (i) - new[i++] = ','; - memcpy(new + i, s, sl); - new[i + sl] = '\0'; - *more = new; + if (err < 0) + return err; +#else + return -EINVAL; +#endif + } else { + for (i = 0; i < ARRAY_SIZE(opts); ++i) { + if (!strcasecmp(s, opts[i].name)) { + if (opts[i].flags < 0) + flags &= opts[i].flags; + else + flags |= opts[i].flags; + break; + } + } + + if (more && i >= ARRAY_SIZE(opts)) { + int sl = strlen(s); + char *new = *more; + + i = new ? strlen(new) : 0; + new = realloc(new, i + strlen(s) + 2); + if (!new) + return -ENOMEM; + if (i) + new[i++] = ','; + memcpy(new + i, s, sl); + new[i + sl] = '\0'; + *more = new; + } } if (!comma) @@ -120,6 +201,11 @@ static long erofsmount_parse_flagopts(char *s, long flags, char **more) *comma = ','; s = comma + 1; } + +#ifdef OCIEROFS_ENABLED + if (ocicfg.platform || ocicfg.username || ocicfg.password || ocicfg.layer_index != 0) + mountcfg.use_oci = true; +#endif return flags; } @@ -540,9 +626,64 @@ err_identifier: return err; } -static int erofsmount_nbd(const char *source, const char *mountpoint, - const char *fstype, int flags, - const char *options) +static int erofsmount_startnbd_oci(int nbdfd, struct ocierofs_ctx *oci_ctx) +{ + struct erofsmount_nbd_ctx ctx = {}; + struct ocierofs_iostream *oci_iostream = NULL; + uintptr_t retcode; + pthread_t th; + int err, err2; + int blkbits = 12; + int index = oci_ctx->layer_index; + u64 blocks; + + blocks = (oci_ctx->layers[index]->size + (1ULL << blkbits) - 1) >> blkbits; + + oci_iostream = malloc(sizeof(struct ocierofs_iostream)); + if (!oci_iostream) + return -ENOMEM; + + err = ocierofs_iostream_open(oci_iostream, oci_ctx); + if (err) { + free(oci_iostream); + return err; + } + + ctx.vd = oci_iostream->vf; + + err = erofs_nbd_connect(nbdfd, blkbits, blocks); + if (err < 0) { + ocierofs_iostream_close(oci_iostream); + free(oci_iostream); + return err; + } + ctx.sk.fd = err; + + err = -pthread_create(&th, NULL, erofsmount_nbd_loopfn, &ctx); + if (err) { + ocierofs_iostream_close(oci_iostream); + free(oci_iostream); + close(ctx.sk.fd); + return err; + } + + err = erofs_nbd_do_it(nbdfd); + err2 = -pthread_join(th, (void **)&retcode); + if (!err2 && retcode) { + erofs_err("NBD worker failed with %s", + erofs_strerror(retcode)); + err2 = retcode; + } + + ocierofs_iostream_close(oci_iostream); + free(oci_iostream); + + return err ?: err2; +} + +static int erofsmount_nbd(union erofs_nbd_source source, enum erofs_nbd_source_type source_type, + const char *mountpoint, const char *fstype, + int flags, const char *options) { bool is_netlink = false; char nbdpath[32], *id; @@ -555,11 +696,19 @@ static int erofsmount_nbd(const char *source, const char *mountpoint, mountcfg.fstype); return -ENODEV; } + flags |= MS_RDONLY; - err = erofsmount_startnbd_nl(&pid, source); - if (err < 0) { - erofs_info("Fall back to ioctl-based NBD; failover is unsupported"); + if (source_type == EROFSNBD_SOURCE_LOCAL) { + err = erofsmount_startnbd_nl(&pid, source.device_path); + if (err >= 0) { + num = err; + (void)snprintf(nbdpath, sizeof(nbdpath), "/dev/nbd%d", num); + is_netlink = true; + } + } + + if (!is_netlink) { num = erofs_nbd_devscan(); if (num < 0) return num; @@ -569,14 +718,16 @@ static int erofsmount_nbd(const char *source, const char *mountpoint, if (nbdfd < 0) return -errno; - if ((pid = fork()) == 0) - return erofsmount_startnbd(nbdfd, source) ? - EXIT_FAILURE : EXIT_SUCCESS; + if ((pid = fork()) == 0) { + if (source_type == EROFSNBD_SOURCE_OCI) { + return erofsmount_startnbd_oci(nbdfd, source.oci_ctx) ? + EXIT_FAILURE : EXIT_SUCCESS; + } else { + return erofsmount_startnbd(nbdfd, source.device_path) ? + EXIT_FAILURE : EXIT_SUCCESS; + } + } close(nbdfd); - } else { - num = err; - (void)snprintf(nbdpath, sizeof(nbdpath), "/dev/nbd%d", num); - is_netlink = true; } while (1) { @@ -594,7 +745,7 @@ static int erofsmount_nbd(const char *source, const char *mountpoint, if (err < 0) err = -errno; - if (!err && is_netlink) { + if (!err && is_netlink && source_type == EROFSNBD_SOURCE_LOCAL) { id = erofs_nbd_get_identifier(num); if (id == ERR_PTR(-ENOENT)) id = NULL; @@ -799,9 +950,43 @@ int main(int argc, char *argv[]) } if (mountcfg.backend == EROFSNBD) { - err = erofsmount_nbd(mountcfg.device, mountcfg.target, +#ifdef OCIEROFS_ENABLED + if (mountcfg.use_oci) { + struct ocierofs_ctx ctx = {}; + + ocicfg.image_ref = mountcfg.device; + err = ocierofs_init(&ctx, &ocicfg); + if (err) { + ocierofs_ctx_cleanup(&ctx); + goto exit; + } + + err = ocierofs_is_erofs_native_image(&ctx); + if (err) { + ocierofs_ctx_cleanup(&ctx); + goto exit; + } + + src.oci_ctx = &ctx; + + err = erofsmount_nbd(src, EROFSNBD_SOURCE_OCI, mountcfg.target, + mountcfg.fstype, mountcfg.flags, mountcfg.options); + if (err) { + ocierofs_ctx_cleanup(&ctx); + goto exit; + } + } else { + src.device_path = mountcfg.device; + err = erofsmount_nbd(src, EROFSNBD_SOURCE_LOCAL, mountcfg.target, + mountcfg.fstype, mountcfg.flags, + mountcfg.options); + } +#else + src.device_path = mountcfg.device; + err = erofsmount_nbd(src, EROFSNBD_SOURCE_LOCAL, mountcfg.target, mountcfg.fstype, mountcfg.flags, mountcfg.options); +#endif goto exit; } -- 2.51.0 ^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH v5 0/3] erofs-utils: refactor OCI and add NBD-backed OCI image mounting 2025-09-01 5:10 [PATCH v1] erofs-utils: refactor OCI code for better modularity ChengyuZhu6 ` (4 preceding siblings ...) 2025-09-04 5:33 ` [PATCH v4 0/3] erofs-utils: refactor OCI and " ChengyuZhu6 @ 2025-09-04 6:36 ` ChengyuZhu6 2025-09-04 6:36 ` [PATCH v5 1/3] erofs-utils:mkfs: move parse_oci_ref to lib ChengyuZhu6 ` (3 more replies) 5 siblings, 4 replies; 19+ messages in thread From: ChengyuZhu6 @ 2025-09-04 6:36 UTC (permalink / raw) To: linux-erofs; +Cc: xiang, hsiangkao, Chengyu Zhu From: Chengyu Zhu <hudsonzhu@tencent.com> This series refactors the OCI handling in erofs-utils and adds NBD-backed mounting of OCI images. It enables mounting EROFS container images directly from registries without pre-downloading. Chengyu Zhu (3): erofs-utils:mkfs: move parse_oci_ref to lib erofs-utils: refactor OCI code for better modularity erofs-utils: add NBD-backed OCI image mounting lib/liberofs_oci.h | 88 ++-- lib/remotes/oci.c | 1029 +++++++++++++++++++++++++++++++------------- mkfs/main.c | 186 ++------ mount/Makefile.am | 2 +- mount/main.c | 257 +++++++++-- 5 files changed, 1013 insertions(+), 549 deletions(-) -- 2.51.0 ^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH v5 1/3] erofs-utils:mkfs: move parse_oci_ref to lib 2025-09-04 6:36 ` [PATCH v5 0/3] erofs-utils: refactor OCI and " ChengyuZhu6 @ 2025-09-04 6:36 ` ChengyuZhu6 2025-09-04 6:36 ` [PATCH v5 2/3] erofs-utils: refactor OCI code for better modularity ChengyuZhu6 ` (2 subsequent siblings) 3 siblings, 0 replies; 19+ messages in thread From: ChengyuZhu6 @ 2025-09-04 6:36 UTC (permalink / raw) To: linux-erofs; +Cc: xiang, hsiangkao, Chengyu Zhu From: Chengyu Zhu <hudsonzhu@tencent.com> move parse_oci_ref to oci.c. Signed-off-by: Chengyu Zhu <hudsonzhu@tencent.com> --- lib/liberofs_oci.h | 12 +++-- lib/remotes/oci.c | 110 +++++++++++++++++++++++++++++++++++++++++ mkfs/main.c | 121 +-------------------------------------------- 3 files changed, 120 insertions(+), 123 deletions(-) diff --git a/lib/liberofs_oci.h b/lib/liberofs_oci.h index 3a8108b..bce63ef 100644 --- a/lib/liberofs_oci.h +++ b/lib/liberofs_oci.h @@ -8,9 +8,6 @@ #include <stdbool.h> -#define DOCKER_REGISTRY "docker.io" -#define DOCKER_API_REGISTRY "registry-1.docker.io" - #ifdef __cplusplus extern "C" { #endif @@ -79,6 +76,15 @@ void ocierofs_cleanup(struct erofs_oci *oci); */ int erofs_oci_params_set_string(char **field, const char *value); +/* + * ocierofs_parse_ref - Parse OCI image reference string + * @oci: OCI client structure + * @ref_str: OCI image reference string + * + * Return: 0 on success, negative errno on failure + */ +int ocierofs_parse_ref(struct erofs_oci *oci, const char *ref_str); + /* * ocierofs_build_trees - Build file trees from OCI container image layers * @root: root inode to build the file tree under diff --git a/lib/remotes/oci.c b/lib/remotes/oci.c index 0fb8c1f..e6f0c23 100644 --- a/lib/remotes/oci.c +++ b/lib/remotes/oci.c @@ -21,6 +21,9 @@ #include "liberofs_oci.h" #include "liberofs_private.h" +#define DOCKER_REGISTRY "docker.io" +#define DOCKER_API_REGISTRY "registry-1.docker.io" + #define DOCKER_MEDIATYPE_MANIFEST_V2 \ "application/vnd.docker.distribution.manifest.v2+json" #define DOCKER_MEDIATYPE_MANIFEST_V1 \ @@ -1066,3 +1069,110 @@ int erofs_oci_params_set_string(char **field, const char *value) *field = new_value; return 0; } + +int ocierofs_parse_ref(struct erofs_oci *oci, const char *ref_str) +{ + char *slash, *colon, *dot; + const char *repo_part; + size_t len; + + slash = strchr(ref_str, '/'); + if (slash) { + dot = strchr(ref_str, '.'); + if (dot && dot < slash) { + char *registry_str; + + len = slash - ref_str; + registry_str = strndup(ref_str, len); + + if (!registry_str) { + erofs_err("failed to allocate memory for registry"); + return -ENOMEM; + } + if (erofs_oci_params_set_string(&oci->params.registry, + registry_str)) { + free(registry_str); + erofs_err("failed to set registry"); + return -ENOMEM; + } + free(registry_str); + repo_part = slash + 1; + } else { + repo_part = ref_str; + } + } else { + repo_part = ref_str; + } + + colon = strchr(repo_part, ':'); + if (colon) { + char *repo_str; + + len = colon - repo_part; + repo_str = strndup(repo_part, len); + + if (!repo_str) { + erofs_err("failed to allocate memory for repository"); + return -ENOMEM; + } + + if (!strchr(repo_str, '/') && + (!strcmp(oci->params.registry, DOCKER_API_REGISTRY) || + !strcmp(oci->params.registry, DOCKER_REGISTRY))) { + char *full_repo; + + if (asprintf(&full_repo, "library/%s", repo_str) == -1) { + free(repo_str); + erofs_err("failed to allocate memory for full repository name"); + return -ENOMEM; + } + free(repo_str); + repo_str = full_repo; + } + + if (erofs_oci_params_set_string(&oci->params.repository, + repo_str)) { + free(repo_str); + erofs_err("failed to set repository"); + return -ENOMEM; + } + free(repo_str); + + if (erofs_oci_params_set_string(&oci->params.tag, + colon + 1)) { + erofs_err("failed to set tag"); + return -ENOMEM; + } + } else { + char *repo_str = strdup(repo_part); + + if (!repo_str) { + erofs_err("failed to allocate memory for repository"); + return -ENOMEM; + } + + if (!strchr(repo_str, '/') && + (!strcmp(oci->params.registry, DOCKER_API_REGISTRY) || + !strcmp(oci->params.registry, DOCKER_REGISTRY))) { + char *full_repo; + + if (asprintf(&full_repo, "library/%s", repo_str) == -1) { + free(repo_str); + erofs_err("failed to allocate memory for full repository name"); + return -ENOMEM; + } + free(repo_str); + repo_str = full_repo; + } + + if (erofs_oci_params_set_string(&oci->params.repository, + repo_str)) { + free(repo_str); + erofs_err("failed to set repository"); + return -ENOMEM; + } + free(repo_str); + } + + return 0; +} diff --git a/mkfs/main.c b/mkfs/main.c index bc895f1..064392d 100644 --- a/mkfs/main.c +++ b/mkfs/main.c @@ -763,125 +763,6 @@ static int mkfs_parse_oci_options(char *options_str) return 0; } - -/** - * mkfs_parse_oci_ref - Parse OCI image reference string - * @ref_str: OCI image reference in various formats - * - * Parse OCI image reference which can be in formats: - * - registry.example.com/namespace/repo:tag - * - namespace/repo:tag (uses default registry) - * - repo:tag (adds library/ prefix for Docker Hub) - * - repo (uses default tag "latest") - * - * Return: 0 on success, negative errno on failure - */ -static int mkfs_parse_oci_ref(const char *ref_str) -{ - char *slash, *colon, *dot; - const char *repo_part; - size_t len; - - slash = strchr(ref_str, '/'); - if (slash) { - dot = strchr(ref_str, '.'); - if (dot && dot < slash) { - char *registry_str; - - len = slash - ref_str; - registry_str = strndup(ref_str, len); - - if (!registry_str) { - erofs_err("failed to allocate memory for registry"); - return -ENOMEM; - } - if (erofs_oci_params_set_string(&ocicfg.params.registry, - registry_str)) { - free(registry_str); - erofs_err("failed to set registry"); - return -ENOMEM; - } - free(registry_str); - repo_part = slash + 1; - } else { - repo_part = ref_str; - } - } else { - repo_part = ref_str; - } - - colon = strchr(repo_part, ':'); - if (colon) { - char *repo_str; - - len = colon - repo_part; - repo_str = strndup(repo_part, len); - - if (!repo_str) { - erofs_err("failed to allocate memory for repository"); - return -ENOMEM; - } - - if (!strchr(repo_str, '/') && - (!strcmp(ocicfg.params.registry, DOCKER_API_REGISTRY) || - !strcmp(ocicfg.params.registry, DOCKER_REGISTRY))) { - char *full_repo; - - if (asprintf(&full_repo, "library/%s", repo_str) == -1) { - free(repo_str); - erofs_err("failed to allocate memory for full repository name"); - return -ENOMEM; - } - free(repo_str); - repo_str = full_repo; - } - - if (erofs_oci_params_set_string(&ocicfg.params.repository, - repo_str)) { - free(repo_str); - erofs_err("failed to set repository"); - return -ENOMEM; - } - free(repo_str); - - if (erofs_oci_params_set_string(&ocicfg.params.tag, - colon + 1)) { - erofs_err("failed to set tag"); - return -ENOMEM; - } - } else { - char *repo_str = strdup(repo_part); - - if (!repo_str) { - erofs_err("failed to allocate memory for repository"); - return -ENOMEM; - } - - if (!strchr(repo_str, '/') && - (!strcmp(ocicfg.params.registry, DOCKER_API_REGISTRY) || - !strcmp(ocicfg.params.registry, DOCKER_REGISTRY))) { - char *full_repo; - - if (asprintf(&full_repo, "library/%s", repo_str) == -1) { - free(repo_str); - erofs_err("failed to allocate memory for full repository name"); - return -ENOMEM; - } - free(repo_str); - repo_str = full_repo; - } - - if (erofs_oci_params_set_string(&ocicfg.params.repository, - repo_str)) { - free(repo_str); - erofs_err("failed to set repository"); - return -ENOMEM; - } - free(repo_str); - } - - return 0; -} #endif static int mkfs_parse_one_compress_alg(char *alg, @@ -1958,7 +1839,7 @@ int main(int argc, char **argv) err = mkfs_parse_oci_options(mkfs_oci_options); if (err) goto exit; - err = mkfs_parse_oci_ref(cfg.c_src_path); + err = ocierofs_parse_ref(&ocicfg, cfg.c_src_path); if (err) goto exit; -- 2.51.0 ^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH v5 2/3] erofs-utils: refactor OCI code for better modularity 2025-09-04 6:36 ` [PATCH v5 0/3] erofs-utils: refactor OCI and " ChengyuZhu6 2025-09-04 6:36 ` [PATCH v5 1/3] erofs-utils:mkfs: move parse_oci_ref to lib ChengyuZhu6 @ 2025-09-04 6:36 ` ChengyuZhu6 2025-09-04 6:36 ` [PATCH v5 3/3] erofs-utils: add NBD-backed OCI image mounting ChengyuZhu6 2025-09-04 7:19 ` [PATCH v5 0/3] erofs-utils: refactor OCI and " Gao Xiang 3 siblings, 0 replies; 19+ messages in thread From: ChengyuZhu6 @ 2025-09-04 6:36 UTC (permalink / raw) To: linux-erofs; +Cc: xiang, hsiangkao, Chengyu Zhu From: Chengyu Zhu <hudsonzhu@tencent.com> Refactor OCI code to improve code organization and maintainability: Key changes: - Add `ocierofs_get_api_registry()` and unify API endpoint selection. - Implement Bearer token discovery with Basic fallback; cache auth header. - Parse layer metadata (digest, mediaType, size) and add a proper free helper. - Split blob download from tar processing; process tar via a temp fd. - Rework init/teardown into `ocierofs_init()` and `ocierofs_ctx_cleanup()`. - Update mkfs to use `struct ocierofs_config` and new `--oci` parsing. Signed-off-by: Chengyu Zhu <hudsonzhu@tencent.com> --- lib/liberofs_oci.h | 82 ++--- lib/remotes/oci.c | 891 ++++++++++++++++++++++++--------------------- mkfs/main.c | 67 ++-- 3 files changed, 535 insertions(+), 505 deletions(-) diff --git a/lib/liberofs_oci.h b/lib/liberofs_oci.h index bce63ef..ebb427a 100644 --- a/lib/liberofs_oci.h +++ b/lib/liberofs_oci.h @@ -12,15 +12,11 @@ extern "C" { #endif -struct erofs_inode; -struct CURL; struct erofs_importer; -/** - * struct erofs_oci_params - OCI configuration parameters - * @registry: registry hostname (e.g., "registry-1.docker.io") - * @repository: image repository (e.g., "library/ubuntu") - * @tag: image tag or digest (e.g., "latest" or sha256:...) +/* + * struct ocierofs_config - OCI configuration structure + * @image_ref: OCI image reference (e.g., "ubuntu:latest", "myregistry.com/app:v1.0") * @platform: target platform in "os/arch" format (e.g., "linux/amd64") * @username: username for authentication (optional) * @password: password for authentication (optional) @@ -30,69 +26,45 @@ struct erofs_importer; * location, image identification, platform specification, and authentication * credentials. */ -struct erofs_oci_params { - char *registry; - char *repository; - char *tag; +struct ocierofs_config { + char *image_ref; char *platform; char *username; char *password; int layer_index; }; -/** - * struct erofs_oci - Combined OCI client structure - * @curl: CURL handle for HTTP requests - * @params: OCI configuration parameters - * - * Main OCI client structure combining CURL HTTP client with - * OCI-specific configuration parameters. - */ -struct erofs_oci { - struct CURL *curl; - struct erofs_oci_params params; +struct ocierofs_layer_info { + char *digest; + char *media_type; + u64 size; }; -/* - * ocierofs_init - Initialize OCI client with default parameters - * @oci: OCI client structure to initialize - * - * Return: 0 on success, negative errno on failure - */ -int ocierofs_init(struct erofs_oci *oci); - -/* - * ocierofs_cleanup - Clean up OCI client and free allocated resources - * @oci: OCI client structure to clean up - */ -void ocierofs_cleanup(struct erofs_oci *oci); - -/* - * erofs_oci_params_set_string - Set a string field with dynamic allocation - * @field: pointer to the string field to set - * @value: string value to set - * - * Return: 0 on success, negative errno on failure - */ -int erofs_oci_params_set_string(char **field, const char *value); +struct ocierofs_ctx { + struct CURL *curl; + char *auth_header; + bool using_basic; + char *registry; + char *repository; + char *platform; + char *tag; + char *manifest_digest; + struct ocierofs_layer_info **layers; + int layer_index; + int layer_count; +}; -/* - * ocierofs_parse_ref - Parse OCI image reference string - * @oci: OCI client structure - * @ref_str: OCI image reference string - * - * Return: 0 on success, negative errno on failure - */ -int ocierofs_parse_ref(struct erofs_oci *oci, const char *ref_str); +int ocierofs_init(struct ocierofs_ctx *ctx, const struct ocierofs_config *config); /* * ocierofs_build_trees - Build file trees from OCI container image layers - * @root: root inode to build the file tree under - * @oci: OCI client structure with configured parameters + * @importer: erofs importer to populate + * @cfg: oci configuration * * Return: 0 on success, negative errno on failure */ -int ocierofs_build_trees(struct erofs_importer *importer, struct erofs_oci *oci); +int ocierofs_build_trees(struct erofs_importer *importer, + const struct ocierofs_config *cfg); #ifdef __cplusplus } diff --git a/lib/remotes/oci.c b/lib/remotes/oci.c index e6f0c23..26e83d7 100644 --- a/lib/remotes/oci.c +++ b/lib/remotes/oci.c @@ -33,28 +33,67 @@ #define OCI_MEDIATYPE_MANIFEST "application/vnd.oci.image.manifest.v1+json" #define OCI_MEDIATYPE_INDEX "application/vnd.oci.image.index.v1+json" -struct erofs_oci_request { +struct ocierofs_request { char *url; struct curl_slist *headers; }; -struct erofs_oci_response { +struct ocierofs_response { char *data; size_t size; long http_code; }; -struct erofs_oci_stream { - struct erofs_tarfile tarfile; +struct ocierofs_stream { const char *digest; int blobfd; }; +static inline const char *ocierofs_get_api_registry(const char *registry) +{ + if (!registry) + return DOCKER_API_REGISTRY; + return !strcmp(registry, DOCKER_REGISTRY) ? DOCKER_API_REGISTRY : registry; +} + +static inline bool ocierofs_is_manifest_list(const char *media_type) +{ + return media_type && (!strcmp(media_type, DOCKER_MEDIATYPE_MANIFEST_LIST) || + !strcmp(media_type, OCI_MEDIATYPE_INDEX)); +} + +static inline bool ocierofs_is_manifest(const char *media_type) +{ + return media_type && (!strcmp(media_type, DOCKER_MEDIATYPE_MANIFEST_V2) || + !strcmp(media_type, OCI_MEDIATYPE_MANIFEST)); +} + +static inline void ocierofs_request_cleanup(struct ocierofs_request *req) +{ + if (!req) + return; + if (req->headers) + curl_slist_free_all(req->headers); + free(req->url); + req->url = NULL; + req->headers = NULL; +} + +static inline void ocierofs_response_cleanup(struct ocierofs_response *resp) +{ + if (!resp) + return; + free(resp->data); + resp->data = NULL; + resp->size = 0; + resp->http_code = 0; +} + static size_t ocierofs_write_callback(void *contents, size_t size, size_t nmemb, void *userp) { size_t realsize = size * nmemb; - struct erofs_oci_response *resp = userp; + struct ocierofs_response *resp = userp; char *ptr; if (!resp->data) @@ -75,16 +114,23 @@ static size_t ocierofs_write_callback(void *contents, size_t size, static size_t ocierofs_layer_write_callback(void *contents, size_t size, size_t nmemb, void *userp) { - struct erofs_oci_stream *stream = userp; + struct ocierofs_stream *stream = userp; size_t realsize = size * nmemb; + const char *buf = contents; + size_t written = 0; if (stream->blobfd < 0) return 0; - if (write(stream->blobfd, contents, realsize) != realsize) { - erofs_err("failed to write layer data for layer %s", - stream->digest); - return 0; + while (written < realsize) { + ssize_t n = write(stream->blobfd, buf + written, realsize - written); + + if (n < 0) { + erofs_err("failed to write layer data for layer %s", + stream->digest); + return 0; + } + written += n; } return realsize; } @@ -96,6 +142,14 @@ static int ocierofs_curl_setup_common_options(struct CURL *curl) curl_easy_setopt(curl, CURLOPT_TIMEOUT, 120L); curl_easy_setopt(curl, CURLOPT_NOSIGNAL, 1L); curl_easy_setopt(curl, CURLOPT_USERAGENT, "ocierofs/" PACKAGE_VERSION); + curl_easy_setopt(curl, CURLOPT_ACCEPT_ENCODING, ""); + curl_easy_setopt(curl, CURLOPT_TCP_KEEPALIVE, 1L); +#if defined(CURLOPT_TCP_KEEPIDLE) + curl_easy_setopt(curl, CURLOPT_TCP_KEEPIDLE, 30L); +#endif +#if defined(CURLOPT_TCP_KEEPINTVL) + curl_easy_setopt(curl, CURLOPT_TCP_KEEPINTVL, 15L); +#endif return 0; } @@ -114,10 +168,10 @@ static int ocierofs_curl_setup_basic_auth(struct CURL *curl, const char *usernam return 0; } -static int ocierofs_curl_clear_auth(struct CURL *curl) +static int ocierofs_curl_clear_auth(struct ocierofs_ctx *ctx) { - curl_easy_setopt(curl, CURLOPT_USERPWD, NULL); - curl_easy_setopt(curl, CURLOPT_HTTPAUTH, CURLAUTH_NONE); + curl_easy_setopt(ctx->curl, CURLOPT_USERPWD, NULL); + curl_easy_setopt(ctx->curl, CURLOPT_HTTPAUTH, CURLAUTH_NONE); return 0; } @@ -176,20 +230,20 @@ static int ocierofs_curl_perform(struct CURL *curl, long *http_code_out) return 0; } -static int ocierofs_request_perform(struct erofs_oci *oci, - struct erofs_oci_request *req, - struct erofs_oci_response *resp) +static int ocierofs_request_perform(struct ocierofs_ctx *ctx, + struct ocierofs_request *req, + struct ocierofs_response *resp) { int ret; - ret = ocierofs_curl_setup_rq(oci->curl, req->url, + ret = ocierofs_curl_setup_rq(ctx->curl, req->url, OCIEROFS_HTTP_GET, req->headers, ocierofs_write_callback, resp, NULL, NULL); if (ret) return ret; - ret = ocierofs_curl_perform(oci->curl, &resp->http_code); + ret = ocierofs_curl_perform(ctx->curl, &resp->http_code); if (ret) return ret; @@ -338,7 +392,7 @@ static int ocierofs_extract_www_auth_info(const char *resp_data, /** * ocierofs_get_auth_token_with_url - Get authentication token from auth server - * @oci: OCI client structure + * @ctx: OCI context structure * @auth_url: authentication server URL * @service: service name for authentication * @repository: repository name @@ -350,15 +404,12 @@ static int ocierofs_extract_www_auth_info(const char *resp_data, * * Return: authentication header string on success, ERR_PTR on failure */ -static char *ocierofs_get_auth_token_with_url(struct erofs_oci *oci, - const char *auth_url, - const char *service, - const char *repository, - const char *username, - const char *password) +static char *ocierofs_get_auth_token_with_url(struct ocierofs_ctx *ctx, const char *auth_url, + const char *service, const char *repository, + const char *username, const char *password) { - struct erofs_oci_request req = {}; - struct erofs_oci_response resp = {}; + struct ocierofs_request req = {}; + struct ocierofs_response resp = {}; json_object *root, *token_obj, *access_token_obj; const char *token; char *auth_header = NULL; @@ -373,14 +424,14 @@ static char *ocierofs_get_auth_token_with_url(struct erofs_oci *oci, } if (username && password && *username) { - ret = ocierofs_curl_setup_basic_auth(oci->curl, username, + ret = ocierofs_curl_setup_basic_auth(ctx->curl, username, password); if (ret) goto out_url; } - ret = ocierofs_request_perform(oci, &req, &resp); - ocierofs_curl_clear_auth(oci->curl); + ret = ocierofs_request_perform(ctx, &req, &resp); + ocierofs_curl_clear_auth(ctx); if (ret) goto out_url; @@ -394,7 +445,7 @@ static char *ocierofs_get_auth_token_with_url(struct erofs_oci *oci, if (!root) { erofs_err("failed to parse auth response"); ret = -EINVAL; - goto out_url; + goto out_json; } if (!json_object_object_get_ex(root, "token", &token_obj) && @@ -419,16 +470,16 @@ static char *ocierofs_get_auth_token_with_url(struct erofs_oci *oci, out_json: json_object_put(root); out_url: - free(req.url); - free(resp.data); + ocierofs_response_cleanup(&resp); + ocierofs_request_cleanup(&req); return ret ? ERR_PTR(ret) : auth_header; } -static char *ocierofs_discover_auth_endpoint(struct erofs_oci *oci, +static char *ocierofs_discover_auth_endpoint(struct ocierofs_ctx *ctx, const char *registry, const char *repository) { - struct erofs_oci_response resp = {}; + struct ocierofs_response resp = {}; char *realm = NULL; char *service = NULL; char *result = NULL; @@ -437,20 +488,20 @@ static char *ocierofs_discover_auth_endpoint(struct erofs_oci *oci, CURLcode res; long http_code; - api_registry = (!strcmp(registry, DOCKER_REGISTRY)) ? DOCKER_API_REGISTRY : registry; + api_registry = ocierofs_get_api_registry(registry); if (asprintf(&test_url, "https://%s/v2/%s/manifests/nonexistent", api_registry, repository) < 0) return NULL; - curl_easy_reset(oci->curl); - ocierofs_curl_setup_common_options(oci->curl); + curl_easy_reset(ctx->curl); + ocierofs_curl_setup_common_options(ctx->curl); - ocierofs_curl_setup_rq(oci->curl, test_url, OCIEROFS_HTTP_HEAD, NULL, + ocierofs_curl_setup_rq(ctx->curl, test_url, OCIEROFS_HTTP_HEAD, NULL, NULL, NULL, ocierofs_write_callback, &resp); - res = curl_easy_perform(oci->curl); - curl_easy_getinfo(oci->curl, CURLINFO_RESPONSE_CODE, &http_code); + res = curl_easy_perform(ctx->curl); + curl_easy_getinfo(ctx->curl, CURLINFO_RESPONSE_CODE, &http_code); if (res == CURLE_OK && (http_code == 401 || http_code == 403 || http_code == 404) && resp.data) { @@ -461,12 +512,12 @@ static char *ocierofs_discover_auth_endpoint(struct erofs_oci *oci, } free(realm); free(service); - free(resp.data); + ocierofs_response_cleanup(&resp); free(test_url); return result; } -static char *ocierofs_get_auth_token(struct erofs_oci *oci, const char *registry, +static char *ocierofs_get_auth_token(struct ocierofs_ctx *ctx, const char *registry, const char *repository, const char *username, const char *password) { @@ -487,35 +538,35 @@ static char *ocierofs_get_auth_token(struct erofs_oci *oci, const char *registry !strcmp(registry, DOCKER_REGISTRY); if (docker_reg) { service = "registry.docker.io"; - auth_header = ocierofs_get_auth_token_with_url(oci, + auth_header = ocierofs_get_auth_token_with_url(ctx, "https://auth.docker.io/token", service, repository, username, password); if (!IS_ERR(auth_header)) return auth_header; } - discovered_auth_url = ocierofs_discover_auth_endpoint(oci, registry, repository); + discovered_auth_url = ocierofs_discover_auth_endpoint(ctx, registry, repository); if (discovered_auth_url) { const char *api_registry, *auth_service; - struct erofs_oci_response resp = {}; + struct ocierofs_response resp = {}; char *test_url; CURLcode res; long http_code; - api_registry = (!strcmp(registry, DOCKER_REGISTRY)) ? DOCKER_API_REGISTRY : registry; + api_registry = ocierofs_get_api_registry(registry); if (asprintf(&test_url, "https://%s/v2/%s/manifests/nonexistent", api_registry, repository) >= 0) { - curl_easy_reset(oci->curl); - ocierofs_curl_setup_common_options(oci->curl); + curl_easy_reset(ctx->curl); + ocierofs_curl_setup_common_options(ctx->curl); - ocierofs_curl_setup_rq(oci->curl, test_url, + ocierofs_curl_setup_rq(ctx->curl, test_url, OCIEROFS_HTTP_HEAD, NULL, NULL, NULL, ocierofs_write_callback, &resp); - res = curl_easy_perform(oci->curl); - curl_easy_getinfo(oci->curl, CURLINFO_RESPONSE_CODE, &http_code); + res = curl_easy_perform(ctx->curl); + curl_easy_getinfo(ctx->curl, CURLINFO_RESPONSE_CODE, &http_code); if (res == CURLE_OK && (http_code == 401 || http_code == 403 || http_code == 404) && resp.data) { @@ -524,12 +575,12 @@ static char *ocierofs_get_auth_token(struct erofs_oci *oci, const char *registry ocierofs_extract_www_auth_info(resp.data, &realm, &discovered_service, NULL); free(realm); } - free(resp.data); + ocierofs_response_cleanup(&resp); free(test_url); } auth_service = discovered_service ? discovered_service : service; - auth_header = ocierofs_get_auth_token_with_url(oci, discovered_auth_url, + auth_header = ocierofs_get_auth_token_with_url(ctx, discovered_auth_url, auth_service, repository, username, password); free(discovered_auth_url); @@ -544,7 +595,7 @@ static char *ocierofs_get_auth_token(struct erofs_oci *oci, const char *registry if (asprintf(&auth_url, auth_patterns[i], registry) < 0) continue; - auth_header = ocierofs_get_auth_token_with_url(oci, auth_url, + auth_header = ocierofs_get_auth_token_with_url(ctx, auth_url, service, repository, username, password); free(auth_url); @@ -557,24 +608,21 @@ static char *ocierofs_get_auth_token(struct erofs_oci *oci, const char *registry return ERR_PTR(-ENOENT); } -static char *ocierofs_get_manifest_digest(struct erofs_oci *oci, +static char *ocierofs_get_manifest_digest(struct ocierofs_ctx *ctx, const char *registry, const char *repository, const char *tag, const char *platform, const char *auth_header) { - struct erofs_oci_request req = {}; - struct erofs_oci_response resp = {}; + struct ocierofs_request req = {}; + struct ocierofs_response resp = {}; json_object *root, *manifests, *manifest, *platform_obj, *arch_obj; json_object *os_obj, *digest_obj, *schema_obj, *media_type_obj; char *digest = NULL; const char *api_registry; int ret = 0, len, i; - if (!registry || !repository || !tag || !platform) - return ERR_PTR(-EINVAL); - - api_registry = (!strcmp(registry, DOCKER_REGISTRY)) ? DOCKER_API_REGISTRY : registry; + api_registry = ocierofs_get_api_registry(registry); if (asprintf(&req.url, "https://%s/v2/%s/manifests/%s", api_registry, repository, tag) < 0) return ERR_PTR(-ENOMEM); @@ -584,10 +632,10 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci, req.headers = curl_slist_append(req.headers, "Accept: " DOCKER_MEDIATYPE_MANIFEST_LIST "," - OCI_MEDIATYPE_INDEX "," DOCKER_MEDIATYPE_MANIFEST_V1 "," - DOCKER_MEDIATYPE_MANIFEST_V2); + OCI_MEDIATYPE_INDEX "," OCI_MEDIATYPE_MANIFEST "," + DOCKER_MEDIATYPE_MANIFEST_V1 "," DOCKER_MEDIATYPE_MANIFEST_V2); - ret = ocierofs_request_perform(oci, &req, &resp); + ret = ocierofs_request_perform(ctx, &req, &resp); if (ret) goto out; @@ -615,8 +663,7 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci, if (json_object_object_get_ex(root, "mediaType", &media_type_obj)) { const char *media_type = json_object_get_string(media_type_obj); - if (!strcmp(media_type, DOCKER_MEDIATYPE_MANIFEST_V2) || - !strcmp(media_type, OCI_MEDIATYPE_MANIFEST)) { + if (ocierofs_is_manifest(media_type)) { digest = strdup(tag); ret = 0; goto out_json; @@ -658,34 +705,44 @@ static char *ocierofs_get_manifest_digest(struct erofs_oci *oci, out_json: json_object_put(root); out: - free(resp.data); - if (req.headers) - curl_slist_free_all(req.headers); - free(req.url); - + ocierofs_response_cleanup(&resp); + ocierofs_request_cleanup(&req); return ret ? ERR_PTR(ret) : digest; } -static char **ocierofs_get_layers_info(struct erofs_oci *oci, - const char *registry, - const char *repository, - const char *digest, - const char *auth_header, - int *layer_count) +static void ocierofs_free_layers_info(struct ocierofs_layer_info **layers, int count) { - struct erofs_oci_request req = {}; - struct erofs_oci_response resp = {}; - json_object *root, *layers, *layer, *digest_obj; - char **layers_info = NULL; - const char *api_registry; - int ret, len, i, j; + int i; - if (!registry || !repository || !digest || !layer_count) - return ERR_PTR(-EINVAL); + if (!layers) + return; + + for (i = 0; i < count; i++) { + if (layers[i]) { + free(layers[i]->digest); + free(layers[i]->media_type); + free(layers[i]); + } + } + free(layers); +} + +static struct ocierofs_layer_info **ocierofs_fetch_layers_info(struct ocierofs_ctx *ctx, + const char *registry, + const char *repository, + const char *digest, + const char *auth_header, + int *layer_count) +{ + struct ocierofs_request req = {}; + struct ocierofs_response resp = {}; + json_object *root, *layers, *layer, *digest_obj, *media_type_obj, *size_obj; + struct ocierofs_layer_info **layers_info = NULL; + const char *api_registry; + int ret, len, i; *layer_count = 0; - api_registry = (!strcmp(registry, DOCKER_REGISTRY) ? - DOCKER_API_REGISTRY : registry); + api_registry = ocierofs_get_api_registry(registry); if (asprintf(&req.url, "https://%s/v2/%s/manifests/%s", api_registry, repository, digest) < 0) @@ -697,7 +754,7 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci, req.headers = curl_slist_append(req.headers, "Accept: " OCI_MEDIATYPE_MANIFEST "," DOCKER_MEDIATYPE_MANIFEST_V2); - ret = ocierofs_request_perform(oci, &req, &resp); + ret = ocierofs_request_perform(ctx, &req, &resp); if (ret) goto out; @@ -723,12 +780,11 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci, len = json_object_array_length(layers); if (!len) { - erofs_err("empty layer list in manifest"); ret = -EINVAL; goto out_json; } - layers_info = calloc(len, sizeof(char *)); + layers_info = calloc(len, sizeof(*layers_info)); if (!layers_info) { ret = -ENOMEM; goto out_json; @@ -738,364 +794,196 @@ static char **ocierofs_get_layers_info(struct erofs_oci *oci, layer = json_object_array_get_idx(layers, i); if (!json_object_object_get_ex(layer, "digest", &digest_obj)) { - erofs_err("failed to parse layer %d", i); ret = -EINVAL; goto out_free; } - layers_info[i] = strdup(json_object_get_string(digest_obj)); + layers_info[i] = calloc(1, sizeof(**layers_info)); if (!layers_info[i]) { ret = -ENOMEM; goto out_free; } + layers_info[i]->digest = strdup(json_object_get_string(digest_obj)); + if (!layers_info[i]->digest) { + ret = -ENOMEM; + goto out_free; + } + if (json_object_object_get_ex(layer, "mediaType", &media_type_obj)) + layers_info[i]->media_type = strdup(json_object_get_string(media_type_obj)); + else + layers_info[i]->media_type = NULL; + + if (json_object_object_get_ex(layer, "size", &size_obj)) + layers_info[i]->size = json_object_get_int64(size_obj); + else + layers_info[i]->size = 0; } *layer_count = len; json_object_put(root); - free(resp.data); - if (req.headers) - curl_slist_free_all(req.headers); - free(req.url); + ocierofs_response_cleanup(&resp); + ocierofs_request_cleanup(&req); return layers_info; out_free: - if (layers_info) { - for (j = 0; j < i; j++) - free(layers_info[j]); - } - free(layers_info); + ocierofs_free_layers_info(layers_info, i); out_json: json_object_put(root); out: - free(resp.data); - if (req.headers) - curl_slist_free_all(req.headers); - free(req.url); + ocierofs_response_cleanup(&resp); + ocierofs_request_cleanup(&req); return ERR_PTR(ret); } -static int ocierofs_extract_layer(struct erofs_oci *oci, struct erofs_importer *importer, - const char *digest, const char *auth_header) +static int ocierofs_process_tar_stream(struct erofs_importer *importer, int fd) { - struct erofs_oci_request req = {}; - struct erofs_oci_stream stream = {}; - const char *api_registry; - long http_code; + struct erofs_tarfile tarfile = {}; int ret; - stream = (struct erofs_oci_stream) { - .digest = digest, - .blobfd = erofs_tmpfile(), - }; - if (stream.blobfd < 0) { - erofs_err("failed to create temporary file for %s", digest); - return -errno; - } - - api_registry = (!strcmp(oci->params.registry, DOCKER_REGISTRY)) ? - DOCKER_API_REGISTRY : oci->params.registry; - if (asprintf(&req.url, "https://%s/v2/%s/blobs/%s", - api_registry, oci->params.repository, digest) == -1) { - ret = -ENOMEM; - goto out; - } - - if (auth_header && strstr(auth_header, "Bearer")) - req.headers = curl_slist_append(req.headers, auth_header); + init_list_head(&tarfile.global.xattrs); - curl_easy_reset(oci->curl); - - ret = ocierofs_curl_setup_common_options(oci->curl); - if (ret) - goto out; - - ret = ocierofs_curl_setup_rq(oci->curl, req.url, OCIEROFS_HTTP_GET, - req.headers, - ocierofs_layer_write_callback, - &stream, NULL, NULL); - if (ret) - goto out; - - ret = ocierofs_curl_perform(oci->curl, &http_code); - if (ret) - goto out; - - if (http_code < 200 || http_code >= 300) { - erofs_err("HTTP request failed with code %ld", http_code); - ret = -EIO; - goto out; - } - - if (lseek(stream.blobfd, 0, SEEK_SET) < 0) { - erofs_err("failed to seek to beginning of temp file: %s", - strerror(errno)); - ret = -errno; - goto out; - } - - memset(&stream.tarfile, 0, sizeof(stream.tarfile)); - init_list_head(&stream.tarfile.global.xattrs); - - ret = erofs_iostream_open(&stream.tarfile.ios, stream.blobfd, - EROFS_IOS_DECODER_GZIP); + ret = erofs_iostream_open(&tarfile.ios, fd, EROFS_IOS_DECODER_GZIP); if (ret) { erofs_err("failed to initialize tar stream: %s", erofs_strerror(ret)); - goto out; + return ret; } do { - ret = tarerofs_parse_tar(importer, &stream.tarfile); + ret = tarerofs_parse_tar(importer, &tarfile); /* Continue parsing until end of archive */ } while (!ret); - erofs_iostream_close(&stream.tarfile.ios); + erofs_iostream_close(&tarfile.ios); if (ret < 0 && ret != -ENODATA) { erofs_err("failed to process tar stream: %s", erofs_strerror(ret)); - goto out; + return ret; } - ret = 0; -out: - if (stream.blobfd >= 0) - close(stream.blobfd); - if (req.headers) - curl_slist_free_all(req.headers); - free(req.url); - return ret; + return 0; } -/** - * ocierofs_build_trees - Build file trees from OCI container image layers - * @importer: EROFS importer structure - * @oci: OCI client structure with configured parameters - * - * Extract and build file system trees from all layers of an OCI container - * image. - * - * Return: 0 on success, negative errno on failure - */ -int ocierofs_build_trees(struct erofs_importer *importer, struct erofs_oci *oci) +static int ocierofs_prepare_auth(struct ocierofs_ctx *ctx, + const char *username, + const char *password) { char *auth_header = NULL; - char *manifest_digest = NULL; - char **layers = NULL; - int layer_count = 0; - int ret, i; - - if (!importer || !oci) - return -EINVAL; + int ret = 0; + + ctx->using_basic = false; + free(ctx->auth_header); + ctx->auth_header = NULL; + + auth_header = ocierofs_get_auth_token(ctx, + ctx->registry, + ctx->repository, + username, password); + if (!IS_ERR(auth_header)) { + ctx->auth_header = auth_header; + return 0; + } - if (oci->params.username && oci->params.password && - oci->params.username[0] && oci->params.password[0]) { - auth_header = ocierofs_get_auth_token(oci, - oci->params.registry, - oci->params.repository, - oci->params.username, - oci->params.password); - if (IS_ERR(auth_header)) { - auth_header = NULL; - ret = ocierofs_curl_setup_basic_auth(oci->curl, - oci->params.username, - oci->params.password); - if (ret) - goto out; - } - } else { - auth_header = ocierofs_get_auth_token(oci, - oci->params.registry, - oci->params.repository, - NULL, NULL); - if (IS_ERR(auth_header)) - auth_header = NULL; + if (username && password && *username && *password) { + ret = ocierofs_curl_setup_basic_auth(ctx->curl, + username, password); + if (ret) + return ret; + ctx->using_basic = true; } + return 0; +} - manifest_digest = ocierofs_get_manifest_digest(oci, oci->params.registry, - oci->params.repository, - oci->params.tag, - oci->params.platform, - auth_header); - if (IS_ERR(manifest_digest)) { - ret = PTR_ERR(manifest_digest); +static int ocierofs_prepare_layers(struct ocierofs_ctx *ctx, + const struct ocierofs_config *config) +{ + int ret; + + ret = ocierofs_prepare_auth(ctx, config ? config->username : NULL, + config ? config->password : NULL); + if (ret) + return ret; + + ctx->manifest_digest = ocierofs_get_manifest_digest(ctx, ctx->registry, + ctx->repository, + ctx->tag, + ctx->platform, + ctx->auth_header); + if (IS_ERR(ctx->manifest_digest)) { + ret = PTR_ERR(ctx->manifest_digest); erofs_err("failed to get manifest digest: %s", erofs_strerror(ret)); + ctx->manifest_digest = NULL; goto out_auth; } - layers = ocierofs_get_layers_info(oci, oci->params.registry, - oci->params.repository, - manifest_digest, auth_header, - &layer_count); - if (IS_ERR(layers)) { - ret = PTR_ERR(layers); + ctx->layers = ocierofs_fetch_layers_info(ctx, ctx->registry, + ctx->repository, + ctx->manifest_digest, ctx->auth_header, + &ctx->layer_count); + if (IS_ERR(ctx->layers)) { + ret = PTR_ERR(ctx->layers); erofs_err("failed to get image layers: %s", erofs_strerror(ret)); + ctx->layers = NULL; goto out_manifest; } - if (oci->params.layer_index >= 0) { - if (oci->params.layer_index >= layer_count) { + if (ctx->layer_index >= 0) { + if (ctx->layer_index >= ctx->layer_count) { erofs_err("layer index %d exceeds available layers (%d)", - oci->params.layer_index, layer_count); + ctx->layer_index, ctx->layer_count); ret = -EINVAL; goto out_layers; } - layer_count = 1; - i = oci->params.layer_index; + ctx->layer_count = 1; } else { - i = 0; + ctx->layer_index = 0; } - while (i < layer_count) { - char *trimmed = erofs_trim_for_progressinfo(layers[i], - sizeof("Extracting layer ...") - 1); - erofs_update_progressinfo("Extracting layer %d: %s ...", i, - trimmed); - free(trimmed); - ret = ocierofs_extract_layer(oci, importer, layers[i], - auth_header); - if (ret) { - erofs_err("failed to extract layer %d: %s", i, - erofs_strerror(ret)); - break; - } - i++; - } + return 0; + out_layers: - for (i = 0; i < layer_count; i++) - free(layers[i]); - free(layers); + free(ctx->layers); + ctx->layers = NULL; out_manifest: - free(manifest_digest); + free(ctx->manifest_digest); + ctx->manifest_digest = NULL; out_auth: - free(auth_header); - - if (oci->params.username && oci->params.password && - oci->params.username[0] && oci->params.password[0] && - !auth_header) { - ocierofs_curl_clear_auth(oci->curl); - } -out: + free(ctx->auth_header); + ctx->auth_header = NULL; + if (ctx->using_basic) + ocierofs_curl_clear_auth(ctx); return ret; } -/** - * ocierofs_init - Initialize OCI client with default parameters - * @oci: OCI client structure to initialize - * - * Initialize OCI client structure, set up CURL handle, and configure - * default parameters including platform (linux/amd64), registry - * (registry-1.docker.io), and tag (latest). +/* + * ocierofs_parse_ref - Parse OCI image reference string + * @ctx: OCI context structure + * @ref_str: OCI image reference string * * Return: 0 on success, negative errno on failure */ -int ocierofs_init(struct erofs_oci *oci) -{ - if (!oci) - return -EINVAL; - - *oci = (struct erofs_oci){}; - oci->curl = curl_easy_init(); - if (!oci->curl) - return -EIO; - - if (ocierofs_curl_setup_common_options(oci->curl)) { - ocierofs_cleanup(oci); - return -EIO; - } - - if (erofs_oci_params_set_string(&oci->params.platform, - "linux/amd64") || - erofs_oci_params_set_string(&oci->params.registry, - DOCKER_API_REGISTRY) || - erofs_oci_params_set_string(&oci->params.tag, "latest")) { - ocierofs_cleanup(oci); - return -ENOMEM; - } - oci->params.layer_index = -1; /* -1 means extract all layers */ - return 0; -} - -/** - * ocierofs_cleanup - Clean up OCI client and free allocated resources - * @oci: OCI client structure to clean up - * - * Clean up CURL handle, free all allocated string parameters, and - * reset the OCI client structure to a clean state. - */ -void ocierofs_cleanup(struct erofs_oci *oci) -{ - if (!oci) - return; - - if (oci->curl) { - curl_easy_cleanup(oci->curl); - oci->curl = NULL; - } - - free(oci->params.registry); - free(oci->params.repository); - free(oci->params.tag); - free(oci->params.platform); - free(oci->params.username); - free(oci->params.password); - - oci->params.registry = NULL; - oci->params.repository = NULL; - oci->params.tag = NULL; - oci->params.platform = NULL; - oci->params.username = NULL; - oci->params.password = NULL; -} - -int erofs_oci_params_set_string(char **field, const char *value) -{ - char *new_value; - - if (!field) - return -EINVAL; - - if (!value) { - free(*field); - *field = NULL; - return 0; - } - new_value = strdup(value); - if (!new_value) - return -ENOMEM; - - free(*field); - *field = new_value; - return 0; -} - -int ocierofs_parse_ref(struct erofs_oci *oci, const char *ref_str) +static int ocierofs_parse_ref(struct ocierofs_ctx *ctx, const char *ref_str) { char *slash, *colon, *dot; const char *repo_part; size_t len; + char *tmp; + + if (!ctx || !ref_str) + return -EINVAL; slash = strchr(ref_str, '/'); if (slash) { dot = strchr(ref_str, '.'); if (dot && dot < slash) { - char *registry_str; - len = slash - ref_str; - registry_str = strndup(ref_str, len); - - if (!registry_str) { - erofs_err("failed to allocate memory for registry"); - return -ENOMEM; - } - if (erofs_oci_params_set_string(&oci->params.registry, - registry_str)) { - free(registry_str); - erofs_err("failed to set registry"); + tmp = strndup(ref_str, len); + if (!tmp) return -ENOMEM; - } - free(registry_str); + free(ctx->registry); + ctx->registry = tmp; repo_part = slash + 1; } else { repo_part = ref_str; @@ -1106,73 +994,250 @@ int ocierofs_parse_ref(struct erofs_oci *oci, const char *ref_str) colon = strchr(repo_part, ':'); if (colon) { - char *repo_str; - len = colon - repo_part; - repo_str = strndup(repo_part, len); - - if (!repo_str) { - erofs_err("failed to allocate memory for repository"); + tmp = strndup(repo_part, len); + if (!tmp) return -ENOMEM; - } - if (!strchr(repo_str, '/') && - (!strcmp(oci->params.registry, DOCKER_API_REGISTRY) || - !strcmp(oci->params.registry, DOCKER_REGISTRY))) { + if (!strchr(tmp, '/') && + (!strcmp(ctx->registry, DOCKER_API_REGISTRY) || + !strcmp(ctx->registry, DOCKER_REGISTRY))) { char *full_repo; - if (asprintf(&full_repo, "library/%s", repo_str) == -1) { - free(repo_str); - erofs_err("failed to allocate memory for full repository name"); + if (asprintf(&full_repo, "library/%s", tmp) == -1) { + free(tmp); return -ENOMEM; } - free(repo_str); - repo_str = full_repo; + free(tmp); + tmp = full_repo; } + free(ctx->repository); + ctx->repository = tmp; - if (erofs_oci_params_set_string(&oci->params.repository, - repo_str)) { - free(repo_str); - erofs_err("failed to set repository"); + free(ctx->tag); + ctx->tag = strdup(colon + 1); + if (!ctx->tag) return -ENOMEM; - } - free(repo_str); - - if (erofs_oci_params_set_string(&oci->params.tag, - colon + 1)) { - erofs_err("failed to set tag"); - return -ENOMEM; - } } else { - char *repo_str = strdup(repo_part); - - if (!repo_str) { - erofs_err("failed to allocate memory for repository"); + tmp = strdup(repo_part); + if (!tmp) return -ENOMEM; - } - if (!strchr(repo_str, '/') && - (!strcmp(oci->params.registry, DOCKER_API_REGISTRY) || - !strcmp(oci->params.registry, DOCKER_REGISTRY))) { + if (!strchr(tmp, '/') && + (!strcmp(ctx->registry, DOCKER_API_REGISTRY) || + !strcmp(ctx->registry, DOCKER_REGISTRY))) { + char *full_repo; - if (asprintf(&full_repo, "library/%s", repo_str) == -1) { - free(repo_str); - erofs_err("failed to allocate memory for full repository name"); + if (asprintf(&full_repo, "library/%s", tmp) == -1) { + free(tmp); return -ENOMEM; } - free(repo_str); - repo_str = full_repo; + free(tmp); + tmp = full_repo; } + free(ctx->repository); + ctx->repository = tmp; + } + return 0; +} - if (erofs_oci_params_set_string(&oci->params.repository, - repo_str)) { - free(repo_str); - erofs_err("failed to set repository"); - return -ENOMEM; +/** + * ocierofs_init - Initialize OCI context + * @ctx: OCI context structure to initialize + * @config: OCI configuration + * + * Initialize OCI context structure, set up CURL handle, and configure + * default parameters including platform (linux/amd64), registry + * (registry-1.docker.io), and tag (latest). + * + * Return: 0 on success, negative errno on failure + */ +int ocierofs_init(struct ocierofs_ctx *ctx, const struct ocierofs_config *config) +{ + int ret; + + ctx->curl = curl_easy_init(); + if (!ctx->curl) + return -EIO; + + if (ocierofs_curl_setup_common_options(ctx->curl)) + return -EIO; + + ctx->layer_index = -1; + ctx->registry = strdup("registry-1.docker.io"); + ctx->tag = strdup("latest"); + if (config && config->platform) + ctx->platform = strdup(config->platform); + else + ctx->platform = strdup("linux/amd64"); + if (!ctx->registry || !ctx->tag || !ctx->platform) + return -ENOMEM; + + if (config && config->layer_index >= 0) + ctx->layer_index = config->layer_index; + + ret = ocierofs_parse_ref(ctx, config->image_ref); + if (ret) + return ret; + + ret = ocierofs_prepare_layers(ctx, config); + if (ret) + return ret; + + return 0; +} + +static int ocierofs_download_blob_to_fd(struct ocierofs_ctx *ctx, + const char *digest, + const char *auth_header, + int outfd) +{ + struct ocierofs_request req = {}; + struct ocierofs_stream stream = {}; + const char *api_registry; + long http_code; + int ret; + + stream = (struct ocierofs_stream) { + .digest = digest, + .blobfd = outfd, + }; + + api_registry = ocierofs_get_api_registry(ctx->registry); + if (asprintf(&req.url, "https://%s/v2/%s/blobs/%s", + api_registry, ctx->repository, digest) == -1) + return -ENOMEM; + + if (auth_header && strstr(auth_header, "Bearer")) + req.headers = curl_slist_append(req.headers, auth_header); + + curl_easy_reset(ctx->curl); + + ret = ocierofs_curl_setup_common_options(ctx->curl); + if (ret) + goto out; + + ret = ocierofs_curl_setup_rq(ctx->curl, req.url, OCIEROFS_HTTP_GET, + req.headers, + ocierofs_layer_write_callback, + &stream, NULL, NULL); + if (ret) + goto out; + + ret = ocierofs_curl_perform(ctx->curl, &http_code); + if (ret) + goto out; + + if (http_code < 200 || http_code >= 300) { + erofs_err("HTTP request failed with code %ld", http_code); + ret = -EIO; + goto out; + } + ret = 0; +out: + ocierofs_request_cleanup(&req); + return ret; +} + +static int ocierofs_extract_layer(struct ocierofs_ctx *ctx, + const char *digest, const char *auth_header) +{ + struct ocierofs_stream stream = {}; + int ret; + + stream = (struct ocierofs_stream) { + .digest = digest, + .blobfd = erofs_tmpfile(), + }; + if (stream.blobfd < 0) { + erofs_err("failed to create temporary file for %s", digest); + return -errno; + } + + ret = ocierofs_download_blob_to_fd(ctx, digest, auth_header, stream.blobfd); + if (ret) + goto out; + + if (lseek(stream.blobfd, 0, SEEK_SET) < 0) { + erofs_err("failed to seek to beginning of temp file: %s", + strerror(errno)); + ret = -errno; + goto out; + } + + return stream.blobfd; + +out: + if (stream.blobfd >= 0) + close(stream.blobfd); + return ret; +} + + +/** + * ocierofs_ctx_cleanup - Clean up OCI context and free allocated resources + * @ctx: OCI context structure to clean up + * + * Clean up CURL handle, free all allocated string parameters, and + * reset the OCI context structure to a clean state. + */ +static void ocierofs_ctx_cleanup(struct ocierofs_ctx *ctx) +{ + if (!ctx) + return; + + if (ctx->curl) { + curl_easy_cleanup(ctx->curl); + ctx->curl = NULL; + } + free(ctx->auth_header); + ctx->auth_header = NULL; + + ocierofs_free_layers_info(ctx->layers, ctx->layer_count); + free(ctx->registry); + free(ctx->repository); + free(ctx->tag); + free(ctx->platform); + free(ctx->manifest_digest); +} + +int ocierofs_build_trees(struct erofs_importer *importer, + const struct ocierofs_config *config) +{ + struct ocierofs_ctx ctx = {}; + int ret, i; + + ret = ocierofs_init(&ctx, config); + if (ret) { + ocierofs_ctx_cleanup(&ctx); + return ret; + } + + i = ctx.layer_index; + while (i < ctx.layer_count) { + char *trimmed = erofs_trim_for_progressinfo(ctx.layers[i]->digest, + sizeof("Extracting layer ...") - 1); + erofs_update_progressinfo("Extracting layer %d: %s ...", i, + trimmed); + free(trimmed); + int fd = ocierofs_extract_layer(&ctx, ctx.layers[i]->digest, + ctx.auth_header); + if (fd < 0) { + erofs_err("failed to extract layer %d: %s", i, + erofs_strerror(fd)); + break; } - free(repo_str); + ret = ocierofs_process_tar_stream(importer, fd); + close(fd); + if (ret) { + erofs_err("failed to process tar stream for layer %d: %s", i, + erofs_strerror(ret)); + break; + } + i++; } - return 0; + ocierofs_ctx_cleanup(&ctx); + return ret; } diff --git a/mkfs/main.c b/mkfs/main.c index 064392d..28588db 100644 --- a/mkfs/main.c +++ b/mkfs/main.c @@ -272,7 +272,7 @@ static struct erofs_s3 s3cfg; #endif #ifdef OCIEROFS_ENABLED -static struct erofs_oci ocicfg; +static struct ocierofs_config ocicfg; static char *mkfs_oci_options; #endif @@ -689,10 +689,9 @@ static int mkfs_parse_s3_cfg(char *cfg_str) #endif #ifdef OCIEROFS_ENABLED - - -/** +/* * mkfs_parse_oci_options - Parse comma-separated OCI options string + * @cfg: OCI configuration structure to update * @options_str: comma-separated options string * * Parse OCI options string containing comma-separated key=value pairs. @@ -700,10 +699,9 @@ static int mkfs_parse_s3_cfg(char *cfg_str) * * Return: 0 on success, negative errno on failure */ -static int mkfs_parse_oci_options(char *options_str) +static int mkfs_parse_oci_options(struct ocierofs_config *oci_cfg, char *options_str) { char *opt, *q, *p; - int ret; if (!options_str) return 0; @@ -717,41 +715,43 @@ static int mkfs_parse_oci_options(char *options_str) p = strstr(opt, "platform="); if (p) { p += strlen("platform="); - ret = erofs_oci_params_set_string(&ocicfg.params.platform, p); - if (ret) { - erofs_err("failed to set platform"); - return ret; - } + free(oci_cfg->platform); + oci_cfg->platform = strdup(p); + if (!oci_cfg->platform) + return -ENOMEM; } else { p = strstr(opt, "layer="); if (p) { p += strlen("layer="); - ocicfg.params.layer_index = atoi(p); - if (ocicfg.params.layer_index < 0) { - erofs_err("invalid layer index %d", - ocicfg.params.layer_index); - return -EINVAL; + { + char *endptr; + unsigned long v = strtoul(p, &endptr, 10); + + if (endptr == p || *endptr != '\0') { + erofs_err("invalid layer index %s", + p); + return -EINVAL; + } + oci_cfg->layer_index = (int)v; } } else { p = strstr(opt, "username="); if (p) { p += strlen("username="); - ret = erofs_oci_params_set_string(&ocicfg.params.username, p); - if (ret) { - erofs_err("failed to set username"); - return ret; - } + free(oci_cfg->username); + oci_cfg->username = strdup(p); + if (!oci_cfg->username) + return -ENOMEM; } else { p = strstr(opt, "password="); if (p) { p += strlen("password="); - ret = erofs_oci_params_set_string(&ocicfg.params.password, p); - if (ret) { - erofs_err("failed to set password"); - return ret; - } + free(oci_cfg->password); + oci_cfg->password = strdup(p); + if (!oci_cfg->password) + return -ENOMEM; } else { - erofs_err("invalid --oci value %s", opt); + erofs_err("mkfs: invalid --oci value %s", opt); return -EINVAL; } } @@ -1832,16 +1832,12 @@ int main(int argc, char **argv) #endif #ifdef OCIEROFS_ENABLED } else if (source_mode == EROFS_MKFS_SOURCE_OCI) { - err = ocierofs_init(&ocicfg); - if (err) - goto exit; + ocicfg.layer_index = -1; - err = mkfs_parse_oci_options(mkfs_oci_options); - if (err) - goto exit; - err = ocierofs_parse_ref(&ocicfg, cfg.c_src_path); + err = mkfs_parse_oci_options(&ocicfg, mkfs_oci_options); if (err) goto exit; + ocicfg.image_ref = cfg.c_src_path; if (incremental_mode || dataimport_mode == EROFS_MKFS_DATA_IMPORT_RVSP || @@ -1916,9 +1912,6 @@ exit: erofs_blob_exit(); erofs_xattr_cleanup_name_prefixes(); erofs_rebuild_cleanup(); -#ifdef OCIEROFS_ENABLED - ocierofs_cleanup(&ocicfg); -#endif erofs_diskbuf_exit(); if (source_mode == EROFS_MKFS_SOURCE_TAR) { erofs_iostream_close(&erofstar.ios); -- 2.51.0 ^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH v5 3/3] erofs-utils: add NBD-backed OCI image mounting 2025-09-04 6:36 ` [PATCH v5 0/3] erofs-utils: refactor OCI and " ChengyuZhu6 2025-09-04 6:36 ` [PATCH v5 1/3] erofs-utils:mkfs: move parse_oci_ref to lib ChengyuZhu6 2025-09-04 6:36 ` [PATCH v5 2/3] erofs-utils: refactor OCI code for better modularity ChengyuZhu6 @ 2025-09-04 6:36 ` ChengyuZhu6 2025-09-04 7:19 ` [PATCH v5 0/3] erofs-utils: refactor OCI and " Gao Xiang 3 siblings, 0 replies; 19+ messages in thread From: ChengyuZhu6 @ 2025-09-04 6:36 UTC (permalink / raw) To: linux-erofs; +Cc: xiang, hsiangkao, Chengyu Zhu From: Chengyu Zhu <hudsonzhu@tencent.com> - Add HTTP range downloads for OCI blobs - Introduce ocierofs_iostream for virtual file I/O - Add --oci option for OCI image mounting with NBD backend New mount.erofs -t erofs.nbd option: -o=[options] source-image mountpoint Supported oci options: - oci.platform=os/arch (default: linux/amd64) - oci=N (extract specific layer, default: all layers) - oci.username/oci.password (basic authentication) e.g.: ./mount/mount.erofs -t erofs.nbd -o 'oci=1,oci.platform=linux/amd64' \ quay.io/chengyuzhu6/golang:1.22.8-erofs /tmp/test/ Signed-off-by: Chengyu Zhu <hudsonzhu@tencent.com> --- lib/liberofs_oci.h | 14 +++ lib/remotes/oci.c | 242 +++++++++++++++++++++++++++++++++++++++++- mount/Makefile.am | 2 +- mount/main.c | 257 ++++++++++++++++++++++++++++++++++++++------- 4 files changed, 476 insertions(+), 39 deletions(-) diff --git a/lib/liberofs_oci.h b/lib/liberofs_oci.h index ebb427a..61cfdc7 100644 --- a/lib/liberofs_oci.h +++ b/lib/liberofs_oci.h @@ -54,6 +54,12 @@ struct ocierofs_ctx { int layer_count; }; +struct ocierofs_iostream { + struct ocierofs_ctx *ctx; + struct erofs_vfile vf; + u64 offset; +}; + int ocierofs_init(struct ocierofs_ctx *ctx, const struct ocierofs_config *config); /* @@ -66,6 +72,14 @@ int ocierofs_init(struct ocierofs_ctx *ctx, const struct ocierofs_config *config int ocierofs_build_trees(struct erofs_importer *importer, const struct ocierofs_config *cfg); +int ocierofs_is_erofs_native_image(struct ocierofs_ctx *ctx); + +void ocierofs_ctx_cleanup(struct ocierofs_ctx *ctx); + +int ocierofs_iostream_open(struct ocierofs_iostream *oci_iostream, struct ocierofs_ctx *oci_ctx); + +void ocierofs_iostream_close(struct ocierofs_iostream *oci_iostream); + #ifdef __cplusplus } #endif diff --git a/lib/remotes/oci.c b/lib/remotes/oci.c index 26e83d7..cfe6547 100644 --- a/lib/remotes/oci.c +++ b/lib/remotes/oci.c @@ -33,6 +33,9 @@ #define OCI_MEDIATYPE_MANIFEST "application/vnd.oci.image.manifest.v1+json" #define OCI_MEDIATYPE_INDEX "application/vnd.oci.image.index.v1+json" +/* Erofs Native Layer Media Type */ +#define EROFS_MEDIATYPE "application/vnd.erofs" + struct ocierofs_request { char *url; struct curl_slist *headers; @@ -1174,7 +1177,6 @@ out: return ret; } - /** * ocierofs_ctx_cleanup - Clean up OCI context and free allocated resources * @ctx: OCI context structure to clean up @@ -1182,7 +1184,7 @@ out: * Clean up CURL handle, free all allocated string parameters, and * reset the OCI context structure to a clean state. */ -static void ocierofs_ctx_cleanup(struct ocierofs_ctx *ctx) +void ocierofs_ctx_cleanup(struct ocierofs_ctx *ctx) { if (!ctx) return; @@ -1241,3 +1243,239 @@ int ocierofs_build_trees(struct erofs_importer *importer, ocierofs_ctx_cleanup(&ctx); return ret; } + +static int ocierofs_download_blob_range(struct ocierofs_ctx *ctx, off_t offset, size_t length, + void **out_buf, size_t *out_size) +{ + struct ocierofs_request req = {}; + struct ocierofs_response resp = {}; + const char *api_registry; + char rangehdr[64]; + long http_code = 0; + int ret; + int index = ctx->layer_index; + + if (offset < 0) + return -EINVAL; + + api_registry = ocierofs_get_api_registry(ctx->registry); + if (asprintf(&req.url, "https://%s/v2/%s/blobs/%s", + api_registry, ctx->repository, ctx->layers[index]->digest) == -1) + return -ENOMEM; + + if (length) + snprintf(rangehdr, sizeof(rangehdr), "Range: bytes=%lld-%lld", + (long long)offset, (long long)(offset + (off_t)length - 1)); + else + snprintf(rangehdr, sizeof(rangehdr), "Range: bytes=%lld-", + (long long)offset); + + if (ctx->auth_header && strstr(ctx->auth_header, "Bearer")) + req.headers = curl_slist_append(req.headers, ctx->auth_header); + req.headers = curl_slist_append(req.headers, rangehdr); + + curl_easy_reset(ctx->curl); + + ret = ocierofs_curl_setup_common_options(ctx->curl); + if (ret) + goto out; + + ret = ocierofs_curl_setup_rq(ctx->curl, req.url, OCIEROFS_HTTP_GET, + req.headers, + ocierofs_write_callback, + &resp, NULL, NULL); + if (ret) + goto out; + + ret = ocierofs_curl_perform(ctx->curl, &http_code); + if (ret) + goto out; + + if (http_code == 206) { + *out_buf = resp.data; + *out_size = resp.size; + resp.data = NULL; + ret = 0; + } else if (http_code == 200) { + if (offset == 0) { + *out_buf = resp.data; + *out_size = resp.size; + resp.data = NULL; + ret = 0; + } else { + if (offset < resp.size) { + size_t available = resp.size - offset; + size_t copy_size = length ? min_t(size_t, length, available) : available; + + *out_buf = malloc(copy_size); + if (!*out_buf) { + ret = -ENOMEM; + goto out; + } + memcpy(*out_buf, resp.data + offset, copy_size); + *out_size = copy_size; + ret = 0; + } else { + *out_buf = NULL; + *out_size = 0; + ret = 0; + } + } + } else { + erofs_err("HTTP range request failed with code %ld", http_code); + ret = -EIO; + } + +out: + if (req.headers) + curl_slist_free_all(req.headers); + free(req.url); + free(resp.data); + return ret; +} + +static ssize_t ocierofs_io_pread(struct erofs_vfile *vf, void *buf, size_t len, u64 offset) +{ + struct ocierofs_iostream *oci_iostream = *(struct ocierofs_iostream **)vf->payload; + void *download_buf = NULL; + size_t download_size = 0; + ssize_t ret; + + ret = ocierofs_download_blob_range(oci_iostream->ctx, offset, len, + &download_buf, &download_size); + if (ret < 0) { + memset(buf, 0, len); + return len; + } + + if (download_buf && download_size > 0) { + memcpy(buf, download_buf, download_size); + free(download_buf); + return download_size; + } + + return 0; +} + +static ssize_t ocierofs_io_read(struct erofs_vfile *vf, void *buf, size_t len) +{ + struct ocierofs_iostream *oci_iostream = *(struct ocierofs_iostream **)vf->payload; + ssize_t ret; + + ret = ocierofs_io_pread(vf, buf, len, oci_iostream->offset); + if (ret > 0) + oci_iostream->offset += ret; + + return ret; +} + +static off_t ocierofs_io_lseek(struct erofs_vfile *vf, u64 offset, int whence) +{ + struct ocierofs_iostream *oci_iostream = *(struct ocierofs_iostream **)vf->payload; + off_t new_offset; + int layer_index = oci_iostream->ctx->layer_index; + + switch (whence) { + case SEEK_SET: + new_offset = offset; + break; + case SEEK_CUR: + new_offset = oci_iostream->offset + offset; + break; + case SEEK_END: + new_offset = oci_iostream->ctx->layers[layer_index]->size + offset; + break; + default: + return -1; + } + + if (new_offset < 0 || new_offset > oci_iostream->ctx->layers[layer_index]->size) + return -1; + + oci_iostream->offset = new_offset; + return new_offset; +} + +static ssize_t ocierofs_io_sendfile(struct erofs_vfile *vout, struct erofs_vfile *vin, + off_t *pos, size_t count) +{ + struct ocierofs_iostream *oci_iostream = *(struct ocierofs_iostream **)vin->payload; + char *buf = NULL; + ssize_t total_written = 0; + ssize_t ret = 0; + + buf = malloc(min_t(size_t, count, 32768)); + if (!buf) + return -ENOMEM; + + while (count > 0) { + size_t to_read = min_t(size_t, count, 32768); + u64 read_offset = pos ? *pos : oci_iostream->offset; + + ret = ocierofs_io_pread(vin, buf, to_read, read_offset); + if (ret <= 0) { + erofs_err("OCI I/O sendfile: failed to read from OCI: %s", + erofs_strerror(ret)); + memset(buf, 0, to_read); + ret = to_read; + } + + ssize_t written = write(vout->fd, buf, ret); + + if (written < 0) { + erofs_err("OCI I/O sendfile: failed to write to output: %s", + strerror(errno)); + ret = -errno; + break; + } + + if (written != ret) { + erofs_err("OCI I/O sendfile: partial write: %zd != %zd", written, ret); + ret = written; + } + + total_written += ret; + count -= ret; + if (pos) + *pos += ret; + else + oci_iostream->offset += ret; + } + + free(buf); + return count; +} + +static struct erofs_vfops ocierofs_io_vfops = { + .pread = ocierofs_io_pread, + .read = ocierofs_io_read, + .lseek = ocierofs_io_lseek, + .sendfile = ocierofs_io_sendfile, +}; + +int ocierofs_iostream_open(struct ocierofs_iostream *oci_iostream, struct ocierofs_ctx *oci_ctx) +{ + memset(oci_iostream, 0, sizeof(*oci_iostream)); + oci_iostream->ctx = oci_ctx; + oci_iostream->vf.ops = &ocierofs_io_vfops; + oci_iostream->vf.fd = -1; + *(struct ocierofs_iostream **)oci_iostream->vf.payload = oci_iostream; + + return 0; +} + +void ocierofs_iostream_close(struct ocierofs_iostream *oci_iostream) +{ + close(oci_iostream->vf.fd); +} + +int ocierofs_is_erofs_native_image(struct ocierofs_ctx *ctx) +{ + if (ctx->layer_count > 0 && ctx->layers[0] && + ctx->layers[0]->media_type) { + if (strcmp(ctx->layers[0]->media_type, EROFS_MEDIATYPE) != 0) + return -ENOENT; + return 0; + } + return -ENOENT; +} diff --git a/mount/Makefile.am b/mount/Makefile.am index d93f3f4..0b4447f 100644 --- a/mount/Makefile.am +++ b/mount/Makefile.am @@ -9,5 +9,5 @@ mount_erofs_SOURCES = main.c mount_erofs_CFLAGS = -Wall -I$(top_srcdir)/include mount_erofs_LDADD = $(top_builddir)/lib/liberofs.la ${libselinux_LIBS} \ ${liblz4_LIBS} ${liblzma_LIBS} ${zlib_LIBS} ${libdeflate_LIBS} \ - ${libzstd_LIBS} ${libqpl_LIBS} ${libxxhash_LIBS} ${libnl3_LIBS} + ${libzstd_LIBS} ${libqpl_LIBS} ${libxxhash_LIBS} ${libnl3_LIBS} ${openssl_LIBS} endif diff --git a/mount/main.c b/mount/main.c index a270f0a..177bca3 100644 --- a/mount/main.c +++ b/mount/main.c @@ -15,6 +15,7 @@ #include "erofs/err.h" #include "erofs/io.h" #include "../lib/liberofs_nbd.h" +#include "../lib/liberofs_oci.h" #ifdef HAVE_LINUX_LOOP_H #include <linux/loop.h> #else @@ -34,6 +35,10 @@ struct loop_info { #include <sys/sysmacros.h> #endif +#ifdef OCIEROFS_ENABLED +static struct ocierofs_config ocicfg; +#endif + enum erofs_backend_drv { EROFSAUTO, EROFSLOCAL, @@ -56,12 +61,76 @@ static struct erofsmount_cfg { long flags; enum erofs_backend_drv backend; enum erofsmount_mode mountmode; +#ifdef OCIEROFS_ENABLED + bool use_oci; +#endif } mountcfg = { .full_options = "ro", .flags = MS_RDONLY, /* default mountflags */ .fstype = "erofs", }; +enum erofs_nbd_source_type { + EROFSNBD_SOURCE_LOCAL, + EROFSNBD_SOURCE_OCI, +}; + +union erofs_nbd_source { + const char *device_path; + struct ocierofs_ctx *oci_ctx; +}; + +union erofs_nbd_source src; + +static int parse_oci_option(struct ocierofs_config *oci_cfg, const char *option) +{ + char *p; + + p = strstr(option, "oci="); + if (p != NULL) { + p += strlen("oci="); + { + char *endptr; + unsigned long v = strtoul(p, &endptr, 10); + + if (endptr == p || *endptr != '\0') + return -EINVAL; + oci_cfg->layer_index = (int)v; + } + } else { + p = strstr(option, "oci.platform="); + if (p != NULL) { + p += strlen("oci.platform="); + free(oci_cfg->platform); + oci_cfg->platform = strdup(p); + if (!oci_cfg->platform) + return -ENOMEM; + } else { + p = strstr(option, "oci.username="); + if (p != NULL) { + p += strlen("oci.username="); + free(oci_cfg->username); + oci_cfg->username = strdup(p); + if (!oci_cfg->username) + return -ENOMEM; + } else { + p = strstr(option, "oci.password="); + if (p != NULL) { + p += strlen("oci.password="); + free(oci_cfg->password); + oci_cfg->password = strdup(p); + if (!oci_cfg->password) + return -ENOMEM; + } else { + return -EINVAL; + } + } + } + } + + return 0; +} + static long erofsmount_parse_flagopts(char *s, long flags, char **more) { static const struct { @@ -90,29 +159,41 @@ static long erofsmount_parse_flagopts(char *s, long flags, char **more) comma = strchr(s, ','); if (comma) *comma = '\0'; - for (i = 0; i < ARRAY_SIZE(opts); ++i) { - if (!strcasecmp(s, opts[i].name)) { - if (opts[i].flags < 0) - flags &= opts[i].flags; - else - flags |= opts[i].flags; - break; - } - } - if (more && i >= ARRAY_SIZE(opts)) { - int sl = strlen(s); - char *new = *more; + if (strncmp(s, "oci", 3) == 0) { +#ifdef OCIEROFS_ENABLED + int err = parse_oci_option(&ocicfg, s); - i = new ? strlen(new) : 0; - new = realloc(new, i + strlen(s) + 2); - if (!new) - return -ENOMEM; - if (i) - new[i++] = ','; - memcpy(new + i, s, sl); - new[i + sl] = '\0'; - *more = new; + if (err < 0) + return err; +#else + return -EINVAL; +#endif + } else { + for (i = 0; i < ARRAY_SIZE(opts); ++i) { + if (!strcasecmp(s, opts[i].name)) { + if (opts[i].flags < 0) + flags &= opts[i].flags; + else + flags |= opts[i].flags; + break; + } + } + + if (more && i >= ARRAY_SIZE(opts)) { + int sl = strlen(s); + char *new = *more; + + i = new ? strlen(new) : 0; + new = realloc(new, i + strlen(s) + 2); + if (!new) + return -ENOMEM; + if (i) + new[i++] = ','; + memcpy(new + i, s, sl); + new[i + sl] = '\0'; + *more = new; + } } if (!comma) @@ -120,6 +201,11 @@ static long erofsmount_parse_flagopts(char *s, long flags, char **more) *comma = ','; s = comma + 1; } + +#ifdef OCIEROFS_ENABLED + if (ocicfg.platform || ocicfg.username || ocicfg.password || ocicfg.layer_index != 0) + mountcfg.use_oci = true; +#endif return flags; } @@ -540,9 +626,64 @@ err_identifier: return err; } -static int erofsmount_nbd(const char *source, const char *mountpoint, - const char *fstype, int flags, - const char *options) +static int erofsmount_startnbd_oci(int nbdfd, struct ocierofs_ctx *oci_ctx) +{ + struct erofsmount_nbd_ctx ctx = {}; + struct ocierofs_iostream *oci_iostream = NULL; + uintptr_t retcode; + pthread_t th; + int err, err2; + int blkbits = 12; + int index = oci_ctx->layer_index; + u64 blocks; + + blocks = (oci_ctx->layers[index]->size + (1ULL << blkbits) - 1) >> blkbits; + + oci_iostream = malloc(sizeof(struct ocierofs_iostream)); + if (!oci_iostream) + return -ENOMEM; + + err = ocierofs_iostream_open(oci_iostream, oci_ctx); + if (err) { + free(oci_iostream); + return err; + } + + ctx.vd = oci_iostream->vf; + + err = erofs_nbd_connect(nbdfd, blkbits, blocks); + if (err < 0) { + ocierofs_iostream_close(oci_iostream); + free(oci_iostream); + return err; + } + ctx.sk.fd = err; + + err = -pthread_create(&th, NULL, erofsmount_nbd_loopfn, &ctx); + if (err) { + ocierofs_iostream_close(oci_iostream); + free(oci_iostream); + close(ctx.sk.fd); + return err; + } + + err = erofs_nbd_do_it(nbdfd); + err2 = -pthread_join(th, (void **)&retcode); + if (!err2 && retcode) { + erofs_err("NBD worker failed with %s", + erofs_strerror(retcode)); + err2 = retcode; + } + + ocierofs_iostream_close(oci_iostream); + free(oci_iostream); + + return err ?: err2; +} + +static int erofsmount_nbd(union erofs_nbd_source source, enum erofs_nbd_source_type source_type, + const char *mountpoint, const char *fstype, + int flags, const char *options) { bool is_netlink = false; char nbdpath[32], *id; @@ -555,11 +696,19 @@ static int erofsmount_nbd(const char *source, const char *mountpoint, mountcfg.fstype); return -ENODEV; } + flags |= MS_RDONLY; - err = erofsmount_startnbd_nl(&pid, source); - if (err < 0) { - erofs_info("Fall back to ioctl-based NBD; failover is unsupported"); + if (source_type == EROFSNBD_SOURCE_LOCAL) { + err = erofsmount_startnbd_nl(&pid, source.device_path); + if (err >= 0) { + num = err; + (void)snprintf(nbdpath, sizeof(nbdpath), "/dev/nbd%d", num); + is_netlink = true; + } + } + + if (!is_netlink) { num = erofs_nbd_devscan(); if (num < 0) return num; @@ -569,14 +718,16 @@ static int erofsmount_nbd(const char *source, const char *mountpoint, if (nbdfd < 0) return -errno; - if ((pid = fork()) == 0) - return erofsmount_startnbd(nbdfd, source) ? - EXIT_FAILURE : EXIT_SUCCESS; + if ((pid = fork()) == 0) { + if (source_type == EROFSNBD_SOURCE_OCI) { + return erofsmount_startnbd_oci(nbdfd, source.oci_ctx) ? + EXIT_FAILURE : EXIT_SUCCESS; + } else { + return erofsmount_startnbd(nbdfd, source.device_path) ? + EXIT_FAILURE : EXIT_SUCCESS; + } + } close(nbdfd); - } else { - num = err; - (void)snprintf(nbdpath, sizeof(nbdpath), "/dev/nbd%d", num); - is_netlink = true; } while (1) { @@ -594,7 +745,7 @@ static int erofsmount_nbd(const char *source, const char *mountpoint, if (err < 0) err = -errno; - if (!err && is_netlink) { + if (!err && is_netlink && source_type == EROFSNBD_SOURCE_LOCAL) { id = erofs_nbd_get_identifier(num); if (id == ERR_PTR(-ENOENT)) id = NULL; @@ -799,9 +950,43 @@ int main(int argc, char *argv[]) } if (mountcfg.backend == EROFSNBD) { - err = erofsmount_nbd(mountcfg.device, mountcfg.target, +#ifdef OCIEROFS_ENABLED + if (mountcfg.use_oci) { + struct ocierofs_ctx ctx = {}; + + ocicfg.image_ref = mountcfg.device; + err = ocierofs_init(&ctx, &ocicfg); + if (err) { + ocierofs_ctx_cleanup(&ctx); + goto exit; + } + + err = ocierofs_is_erofs_native_image(&ctx); + if (err) { + ocierofs_ctx_cleanup(&ctx); + goto exit; + } + + src.oci_ctx = &ctx; + + err = erofsmount_nbd(src, EROFSNBD_SOURCE_OCI, mountcfg.target, + mountcfg.fstype, mountcfg.flags, mountcfg.options); + if (err) { + ocierofs_ctx_cleanup(&ctx); + goto exit; + } + } else { + src.device_path = mountcfg.device; + err = erofsmount_nbd(src, EROFSNBD_SOURCE_LOCAL, mountcfg.target, + mountcfg.fstype, mountcfg.flags, + mountcfg.options); + } +#else + src.device_path = mountcfg.device; + err = erofsmount_nbd(src, EROFSNBD_SOURCE_LOCAL, mountcfg.target, mountcfg.fstype, mountcfg.flags, mountcfg.options); +#endif goto exit; } -- 2.51.0 ^ permalink raw reply related [flat|nested] 19+ messages in thread
* Re: [PATCH v5 0/3] erofs-utils: refactor OCI and add NBD-backed OCI image mounting 2025-09-04 6:36 ` [PATCH v5 0/3] erofs-utils: refactor OCI and " ChengyuZhu6 ` (2 preceding siblings ...) 2025-09-04 6:36 ` [PATCH v5 3/3] erofs-utils: add NBD-backed OCI image mounting ChengyuZhu6 @ 2025-09-04 7:19 ` Gao Xiang 3 siblings, 0 replies; 19+ messages in thread From: Gao Xiang @ 2025-09-04 7:19 UTC (permalink / raw) To: ChengyuZhu6, linux-erofs; +Cc: xiang, Chengyu Zhu On 2025/9/4 14:36, ChengyuZhu6 wrote: > From: Chengyu Zhu <hudsonzhu@tencent.com> > > This series refactors the OCI handling in erofs-utils and adds NBD-backed > mounting of OCI images. It enables mounting EROFS container images directly > from registries without pre-downloading. > > Chengyu Zhu (3): > erofs-utils:mkfs: move parse_oci_ref to lib > erofs-utils: refactor OCI code for better modularity I've applied the first two patches to -experimental. Thanks, Gao Xiang ^ permalink raw reply [flat|nested] 19+ messages in thread
end of thread, other threads:[~2025-09-04 7:19 UTC | newest] Thread overview: 19+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2025-09-01 5:10 [PATCH v1] erofs-utils: refactor OCI code for better modularity ChengyuZhu6 2025-09-01 7:01 ` [PATCH v1-changed] " ChengyuZhu6 2025-09-02 1:52 ` Gao Xiang 2025-09-02 3:17 ` [PATCH v2] " ChengyuZhu6 2025-09-02 3:29 ` [PATCH v2-changed] " ChengyuZhu6 2025-09-03 8:29 ` [PATCH v3 0/2] erofs-utils: refactor OCI and add NBD-backed OCI image mounting ChengyuZhu6 2025-09-03 8:29 ` [PATCH v3 1/2] erofs-utils: refactor OCI code for better modularity ChengyuZhu6 2025-09-03 8:47 ` Gao Xiang 2025-09-03 8:29 ` [PATCH v3 2/2] erofs-utils: add NBD-backed OCI image mounting ChengyuZhu6 2025-09-04 5:33 ` [PATCH v4 0/3] erofs-utils: refactor OCI and " ChengyuZhu6 2025-09-04 5:33 ` [PATCH v4 1/3] erofs-utils:mkfs: move parse_oci_ref to lib ChengyuZhu6 2025-09-04 5:33 ` [PATCH v4 2/3] erofs-utils: refactor OCI code for better modularity ChengyuZhu6 2025-09-04 5:53 ` Gao Xiang 2025-09-04 5:33 ` [PATCH v4 3/3] erofs-utils: add NBD-backed OCI image mounting ChengyuZhu6 2025-09-04 6:36 ` [PATCH v5 0/3] erofs-utils: refactor OCI and " ChengyuZhu6 2025-09-04 6:36 ` [PATCH v5 1/3] erofs-utils:mkfs: move parse_oci_ref to lib ChengyuZhu6 2025-09-04 6:36 ` [PATCH v5 2/3] erofs-utils: refactor OCI code for better modularity ChengyuZhu6 2025-09-04 6:36 ` [PATCH v5 3/3] erofs-utils: add NBD-backed OCI image mounting ChengyuZhu6 2025-09-04 7:19 ` [PATCH v5 0/3] erofs-utils: refactor OCI and " Gao Xiang
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.