rados.py: add tmap_to_omap method

rados.py: add tmap_to_omap method

[Alexandre Oliva]

[-- Attachment #1: Type: text/plain, Size: 960 bytes --]

I wanted to script a tmap_to_omap conversion of a few million dirs, but
running the rados program for each inode would have made it take too
long, so I wrote the attached python script, that I'm now running in
multiple copies in parallel with GNU parallel.

Before it would run, however, I found out python bindings of rados's
tmap_to_omap hadn't been implemented.  It might have been because it
takes a bool argument, and I couldn't find out how to pass it as such.

I ended up forcing the bool argument to False and passing it as an int
0.  This could fail on machines whose ABIs pass bools differently from
ints, but IIRC bools are just aliases to char in C, at least within GCC,
and since char is promoted to int for argument passing, passing an int
should work, especially if it's zero.  For nonzero values, endianness
could matter if the argument is passed on the stack, so I settled for
forcing it to zero.

If anyone knows better, please let me know.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: tmap-to-omap.py --]
[-- Type: text/x-python, Size: 741 bytes --]

#! /usr/bin/python

import sys
import rados
import time

r = rados.Rados(conffile='/etc/ceph/ceph.conf')
r.connect()
i = r.open_ioctx('metadata')
c = 0
err = 0
tbase = time.time()
if len(sys.argv) > 1:
    seq = sys.argv[1:]
    verbose = False
else:
    seq = sys.stdin.readlines()
    verbose = True
for s in seq:
    if verbose:
        sys.stderr.write('\r%i %4.2f/s ' % (c, c / (time.time() - tbase)))
        sys.stderr.flush()
    c = c + 1
    key = '%x.00000000' % int(s.split('\t')[0])
    try:
      i.tmap_to_omap(key)
      print(s)
    except:
      try:
        if i.stat(key)[0] == 0:
	  print(s)
      except:
        sys.stderr.write('\n%s failed\n' % s)
        sys.stderr.flush()
        err = -1
        break
exit(err)

[-- Attachment #3: Type: text/plain, Size: 1568 bytes --]


---

Add tmap_to_omap method to rados python bindings, without the bool
argument: it is forced to false.

Signed-off-by: Alexandre Oliva <oliva@gnu.org>
---
 src/pybind/rados.py |   18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/src/pybind/rados.py b/src/pybind/rados.py
index 93e5040..a2394aa 100644
--- a/src/pybind/rados.py
+++ b/src/pybind/rados.py
@@ -1438,6 +1438,24 @@ returned %d, but should return zero on success." % (self.name, ret))
             raise make_ex(ret, "Failed to stat %r" % key)
         return psize.value, time.localtime(pmtime.value)
 
+    def tmap_to_omap(self, key):
+        """
+        Convert a dir from tmap to omap.
+
+        :param key: the name of the object to convert
+        :type key: str
+
+        :raises: :class:`TypeError`
+        :raises: :class:`Error`
+        :returns: int - 0 on success, otherwise raises an error
+        """
+        self.require_ioctx_open()
+        ret = run_in_thread(self.librados.rados_tmap_to_omap,
+                            (self.io, c_char_p(key), c_int(0)))
+        if ret < 0:
+            raise make_ex(ret, "Failed to convert %s from tmap to omap" % key)
+        return 0
+
     def get_xattr(self, key, xattr_name):
         """
         Get the value of an extended attribute on an object.


-- 
Alexandre Oliva, freedom fighter    http://FSFLA.org/~lxoliva/
You must be the change you wish to see in the world. -- Gandhi
Be Free! -- http://FSFLA.org/   FSF Latin America board member
Free Software Evangelist|Red Hat Brasil GNU Toolchain Engineer

Re: rados.py: add tmap_to_omap method

[Loic Dachary]

[-- Attachment #1: Type: text/plain, Size: 2754 bytes --]

Hi,

This patch is improving

http://tracker.ceph.com/issues/9532

which is part of

http://tracker.ceph.com/issues/6114

But I do not know the answer to the bool question ;-)

Cheers

On 10/10/2014 07:57, Alexandre Oliva wrote:
> I wanted to script a tmap_to_omap conversion of a few million dirs, but
> running the rados program for each inode would have made it take too
> long, so I wrote the attached python script, that I'm now running in
> multiple copies in parallel with GNU parallel.
> 
> Before it would run, however, I found out python bindings of rados's
> tmap_to_omap hadn't been implemented.  It might have been because it
> takes a bool argument, and I couldn't find out how to pass it as such.
> 
> I ended up forcing the bool argument to False and passing it as an int
> 0.  This could fail on machines whose ABIs pass bools differently from
> ints, but IIRC bools are just aliases to char in C, at least within GCC,
> and since char is promoted to int for argument passing, passing an int
> should work, especially if it's zero.  For nonzero values, endianness
> could matter if the argument is passed on the stack, so I settled for
> forcing it to zero.
> 
> If anyone knows better, please let me know.
> 
> 
> 
> 
> ---
> 
> Add tmap_to_omap method to rados python bindings, without the bool
> argument: it is forced to false.
> 
> Signed-off-by: Alexandre Oliva <oliva@gnu.org>
> ---
>  src/pybind/rados.py |   18 ++++++++++++++++++
>  1 file changed, 18 insertions(+)
> 
> diff --git a/src/pybind/rados.py b/src/pybind/rados.py
> index 93e5040..a2394aa 100644
> --- a/src/pybind/rados.py
> +++ b/src/pybind/rados.py
> @@ -1438,6 +1438,24 @@ returned %d, but should return zero on success." % (self.name, ret))
>              raise make_ex(ret, "Failed to stat %r" % key)
>          return psize.value, time.localtime(pmtime.value)
>  
> +    def tmap_to_omap(self, key):
> +        """
> +        Convert a dir from tmap to omap.
> +
> +        :param key: the name of the object to convert
> +        :type key: str
> +
> +        :raises: :class:`TypeError`
> +        :raises: :class:`Error`
> +        :returns: int - 0 on success, otherwise raises an error
> +        """
> +        self.require_ioctx_open()
> +        ret = run_in_thread(self.librados.rados_tmap_to_omap,
> +                            (self.io, c_char_p(key), c_int(0)))
> +        if ret < 0:
> +            raise make_ex(ret, "Failed to convert %s from tmap to omap" % key)
> +        return 0
> +
>      def get_xattr(self, key, xattr_name):
>          """
>          Get the value of an extended attribute on an object.
> 
> 

-- 
Loïc Dachary, Artisan Logiciel Libre


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 263 bytes --]