[Qemu-devel] [Sheepdog Announcement] erasure coding is fully functional, please try test

[Qemu-devel] [Sheepdog Announcement] erasure coding is fully functional, please try test

[Liu Yuan]

Hello all,

Sheepdog is a distributed storage system that QEMU VM and RESTful services (in progress).

Erasure coding is now seamlessly functional with all other features such as
snapshot/clone/cluster-wide/multi-disk/auto-healing e.c with following characteristics:

1 Data is erasure coded automatically while being written to sheepdog storage, no extra operations.
2 Support random read/write, in-place-update, misaligned read/write
3 Support to run any type VM images or attach as a vdisk of a VM
4 User-defined coding scheme on a VDI basis
5 Better read/write performance compared to fully replication scheme
6 A single cluster can both support replicated VDI or erasure-coded VDI

You can get more info at

https://github.com/sheepdog/sheepdog/wiki for a general wiki
https://github.com/sheepdog/sheepdog/wiki/Erasure-Code-Support for erasure coding

Anyone who is interested can

git clone https://github.com/sheepdog/sheepdog.git

For a 10 minutes quick start with a single machine, you can try:
(Assume you are debian based system)

# Compile from the source

 $ sudo apt-get install liburcu-dev
 $ git clone git://github.com/sheepdog/sheepdog.git
 $ cd sheepdog
 $ ./autogen.sh; ./configure --disable-corosync
 $ make; sudo make install

# Create a 6 node cluster with local driver
 $ for i in `seq 0 5`; do sheep /tmp/store$i -n -c local -z $i -p 700$i;done
 $ dog cluster format

# Create a replicated thin-provisioned 20G volume with 3 copies
 $ dog vdi create -c 3 replica 20G
 $ dog vdi list # show vdi list
 $ dog node info # show node information
 $ dog cluster info # show cluster infomation

# Create a erasure-coded (4 data strips and 2 parity strips) 20G volume
 $ dog vdi create -c 4:2 erasure 20G

# Now you should have 2 vdi created
 $ dog vdi list

# You can install OS on these volumes with upstream QEMU
 $ qemu-system-x86_64 -m 1024 --enable-kvm \
   -drive file=sheepdog:erasure,if=virtio -cdrom path_to_your_iso

# or attach the volumes with existant VM
 $ qemu-system-x86_64 -m 1024 --enable-kvm \
   -drive file=your_image,if=virtio -drive file=sheepdog:erasure,if=virtio

# Take a live disk-only snapshot of running VM
 $ dog vdi snapshot -s tag erasure

# Mount the volume(vdi) to local storage file system
 $ sheepfs dir
 $ echo erasure > dir/vdi/mount
# then you can do whatever with the mounted file at dir/vdi/volume/erasure

Have fun

And feedbacks are always welcome.

Thanks
Yuan