[PATCH 1/2 v2] rbd: allow importing from stdin

[PATCH 1/2 v2] rbd: allow importing from stdin

[Christian Brunner]

This patch allows importing images from stdin with the following command:

   rbd import --size=<size in MB> - [dest-image]

v1 -> v2: stat stdin as well

Signed-off-by: Christian Brunner <chb@muc.de>
---
 src/rbd.cc |   26 +++++++++++++++++++-------
 1 files changed, 19 insertions(+), 7 deletions(-)

diff --git a/src/rbd.cc b/src/rbd.cc
index 655d9a2..1a0c510 100644
--- a/src/rbd.cc
+++ b/src/rbd.cc
@@ -402,16 +402,21 @@ done_img:
 }
 
 static int do_import(librbd::RBD &rbd, librados::IoCtx& io_ctx,
-		     const char *imgname, int *order, const char *path)
+		     const char *imgname, int *order, const char *path,
+		     int64_t size)
 {
-  int fd = open(path, O_RDONLY);
-  int r;
-  int64_t size = 0;
+  int fd, r;
   struct stat stat_buf;
   string md_oid;
   struct fiemap *fiemap;
   MyProgressContext pc("Importing image");
 
+  if (! strcmp(path, "-")) {
+    fd = 0;
+  } else {
+    fd = open(path, O_RDONLY);
+  }
+
   if (fd < 0) {
     r = -errno;
     cerr << "error opening " << path << std::endl;
@@ -424,7 +429,9 @@ static int do_import(librbd::RBD &rbd, librados::IoCtx& io_ctx,
     cerr << "stat error " << path << std::endl;
     return r;
   }
-  size = (uint64_t)stat_buf.st_size;
+  if (stat_buf.st_size)
+    size = (uint64_t)stat_buf.st_size;
+
   if (!size) {
     r = get_block_device_size(fd, &size);
     if (r < 0) {
@@ -505,7 +512,12 @@ static int do_import(librbd::RBD &rbd, librados::IoCtx& io_ctx,
       while (cur_seg) {
         bufferptr p(cur_seg);
         //cerr << "reading " << cur_seg << " bytes at offset " << file_pos << std::endl;
-        ssize_t rval = TEMP_FAILURE_RETRY(::pread(fd, p.c_str(), cur_seg, file_pos));
+	ssize_t rval;
+	if(extent == 0 && fiemap->fm_extents[extent].fe_logical == 0) {
+	  rval = TEMP_FAILURE_RETRY(::read(fd, p.c_str(), cur_seg));
+	} else {
+	  rval = TEMP_FAILURE_RETRY(::pread(fd, p.c_str(), cur_seg, file_pos));
+	}
         if (rval < 0) {
           r = -errno;
           cerr << "error reading file: " << cpp_strerror(r) << std::endl;
@@ -1303,7 +1315,7 @@ int main(int argc, const char **argv)
       cerr << "pathname should be specified" << std::endl;
       exit(1);
     }
-    r = do_import(rbd, dest_io_ctx, destname, &order, path);
+    r = do_import(rbd, dest_io_ctx, destname, &order, path, size);
     if (r < 0) {
       cerr << "import failed: " << cpp_strerror(-r) << std::endl;
       exit(1);
-- 
1.7.1

Re: [PATCH 1/2 v2] rbd: allow importing from stdin

[Yehuda Sadeh Weinraub]

On Wed, May 16, 2012 at 1:32 PM, Christian Brunner <chb@muc.de> wrote:
> This patch allows importing images from stdin with the following command:
>
>   rbd import --size=<size in MB> - [dest-image]
>
> v1 -> v2: stat stdin as well
>
> Signed-off-by: Christian Brunner <chb@muc.de>
> ---
>  src/rbd.cc |   26 +++++++++++++++++++-------
>  1 files changed, 19 insertions(+), 7 deletions(-)
>
> diff --git a/src/rbd.cc b/src/rbd.cc
> index 655d9a2..1a0c510 100644
> --- a/src/rbd.cc
> +++ b/src/rbd.cc
> @@ -402,16 +402,21 @@ done_img:
>  }
>
>  static int do_import(librbd::RBD &rbd, librados::IoCtx& io_ctx,
> -                    const char *imgname, int *order, const char *path)
> +                    const char *imgname, int *order, const char *path,
> +                    int64_t size)
>  {
> -  int fd = open(path, O_RDONLY);
> -  int r;
> -  int64_t size = 0;
> +  int fd, r;
>   struct stat stat_buf;
>   string md_oid;
>   struct fiemap *fiemap;
>   MyProgressContext pc("Importing image");
>
> +  if (! strcmp(path, "-")) {
> +    fd = 0;

fd = STDIN_FILENO;

> +  } else {
> +    fd = open(path, O_RDONLY);
> +  }
> +
>   if (fd < 0) {
>     r = -errno;
>     cerr << "error opening " << path << std::endl;
> @@ -424,7 +429,9 @@ static int do_import(librbd::RBD &rbd, librados::IoCtx& io_ctx,
>     cerr << "stat error " << path << std::endl;
>     return r;
>   }
> -  size = (uint64_t)stat_buf.st_size;
> +  if (stat_buf.st_size)
> +    size = (uint64_t)stat_buf.st_size;
> +
>   if (!size) {
>     r = get_block_device_size(fd, &size);

I assume you're going to fail here with stdin, and when size wasn't
passed? or when we're trying to import empty image?

>     if (r < 0) {
> @@ -505,7 +512,12 @@ static int do_import(librbd::RBD &rbd, librados::IoCtx& io_ctx,
>       while (cur_seg) {
>         bufferptr p(cur_seg);
>         //cerr << "reading " << cur_seg << " bytes at offset " << file_pos << std::endl;
> -        ssize_t rval = TEMP_FAILURE_RETRY(::pread(fd, p.c_str(), cur_seg, file_pos));
> +       ssize_t rval;
> +       if(extent == 0 && fiemap->fm_extents[extent].fe_logical == 0) {
> +         rval = TEMP_FAILURE_RETRY(::read(fd, p.c_str(), cur_seg));
> +       } else {
> +         rval = TEMP_FAILURE_RETRY(::pread(fd, p.c_str(), cur_seg, file_pos));
> +       }
>         if (rval < 0) {
>           r = -errno;
>           cerr << "error reading file: " << cpp_strerror(r) << std::endl;
> @@ -1303,7 +1315,7 @@ int main(int argc, const char **argv)
>       cerr << "pathname should be specified" << std::endl;
>       exit(1);
>     }
> -    r = do_import(rbd, dest_io_ctx, destname, &order, path);
> +    r = do_import(rbd, dest_io_ctx, destname, &order, path, size);

I'd rather not pass size for stdin. We should instead make it so that
we don't read size at all and just importing while !eof (or do that
only when size is 0). We'd need to create the image with size=0 and
update the header in the end.


Thanks,
Yehuda
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 1/2 v2] rbd: allow importing from stdin

[Tommi Virtanen]

On Wed, May 16, 2012 at 3:25 PM, Yehuda Sadeh Weinraub
<yehudasa@gmail.com> wrote:
> I'd rather not pass size for stdin. We should instead make it so that
> we don't read size at all and just importing while !eof (or do that
> only when size is 0). We'd need to create the image with size=0 and
> update the header in the end.

Hopefully something there would prevent us from leaking objects if the
rbd import crashed half way through. Perhaps the header needs to
change before each object gets created?

Re: [PATCH 1/2 v2] rbd: allow importing from stdin

[Josh Durgin]

On 05/16/2012 04:51 PM, Tommi Virtanen wrote:
> On Wed, May 16, 2012 at 3:25 PM, Yehuda Sadeh Weinraub
> <yehudasa@gmail.com>  wrote:
>> I'd rather not pass size for stdin. We should instead make it so that
>> we don't read size at all and just importing while !eof (or do that
>> only when size is 0). We'd need to create the image with size=0 and
>> update the header in the end.
>
> Hopefully something there would prevent us from leaking objects if the
> rbd import crashed half way through. Perhaps the header needs to
> change before each object gets created?

librbd prevents you from writing beyond the bounds of an image.
If we keep that restriction, you'd need to resize the image to
have enough space for the import, and resize down to the final
size at the end.

Re: [PATCH 1/2 v2] rbd: allow importing from stdin

[Tommi Virtanen]

On Wed, May 16, 2012 at 4:58 PM, Josh Durgin <josh.durgin@inktank.com> wrote:
> librbd prevents you from writing beyond the bounds of an image.
> If we keep that restriction, you'd need to resize the image to
> have enough space for the import, and resize down to the final
> size at the end.

That's actually good, so it doesn't leak objects.

The import could allocate in nicely sized chunks, and size down at the end.