System Storage Manager

System Storage Manager

[Lukas Czerner]

Hello everyone,

I would like to introduce to you a new tool called System Storage
Manager (ssm). It is supposed to provide easy to use command line
interface to manage your storage using various technologies like
lvm, btrfs, encrypted volumes and possibly more.

Background
----------

In more sophisticated enterprise storage environments, management with
Device Mapper (dm), Logical Volume Manager (LVM), or Multiple Devices
(md) is becoming increasingly more difficult. With file systems added
to the mix, the number of tools needed to configure and manage storage
has grown so large that it is simply not user friendly. With so many
options for a system administrator to consider, the opportunity for
errors and problems is large.

The btrfs administration tools have shown us that storage management can
be simplified, and we are working to bring that ease of use to Linux
filesystems in general.


You can also find some more information in my presentation from LinuxCon
Prague this year:

http://people.redhat.com/lczerner/files/lczerner_fsm.pdf


The code is still under development and no release has been made yet, but
I would like to share with you what I have done so far, since the
progress has been a bit slower than I have previously expected. The
project lives on the sf.net :

https://sourceforge.net/projects/storagemanager/

and you can grab source files from git repository here:

https://sourceforge.net/p/storagemanager/code/ci/a1a5fd616d06030f94b9d2e80ee6ebcad09ad35f/tree/

More information can be found on the projects home page, or in the
README file.

https://sourceforge.net/p/storagemanager/home/Home/


Notes
-----

- It is written in python
- So far it supports commands : check, resize, create, list, add, remove
- More commands to come : mirror, snapshot(!)
- It does not support raid yet, except raid 0 from lvm (striped volume)
- It has been tested with python 2.7, but would like to make it work on
  python 2.6 as well.
- It comes with some doctests, unittests and regression tests written in
  bash (although it is for lvm only so far)
- Its "modular" design should make it relatively simple to add support
  for more back-ends other than lvm, or btrfs


Things to be done before the actual release
-------------------------------------------

- Create btrfs bash tests
- Create btrfs unittests
- Extend python unittests to other backends
- Use wipefs -a before using the device in add()
- Consider using wipefs -a after removing the device
- Remove the physical volume after it is removed from the group
- Figure out how to create better pool names so it is unique in the system and
  between the systems.
- Add mirror support
- Add snapshots support
- Add raid support
- use lsblk and blkid to get information
- Better table alignment when the output spans multiple lines
- Better error handling - not just plain Exception, but rather named exception
  and handle it as main() in ssm module
- Update readme
- Add more documentation into the code


Or course any comment or ideas would be highly appreciated. Please
report bugs directly to me.

Thanks!
-Lukas

[linux-lvm] System Storage Manager

[Lukas Czerner]

Hello everyone,

I would like to introduce to you a new tool called System Storage
Manager (ssm). It is supposed to provide easy to use command line
interface to manage your storage using various technologies like
lvm, btrfs, encrypted volumes and possibly more.

Background
----------

In more sophisticated enterprise storage environments, management with
Device Mapper (dm), Logical Volume Manager (LVM), or Multiple Devices
(md) is becoming increasingly more difficult. With file systems added
to the mix, the number of tools needed to configure and manage storage
has grown so large that it is simply not user friendly. With so many
options for a system administrator to consider, the opportunity for
errors and problems is large.

The btrfs administration tools have shown us that storage management can
be simplified, and we are working to bring that ease of use to Linux
filesystems in general.


You can also find some more information in my presentation from LinuxCon
Prague this year:

http://people.redhat.com/lczerner/files/lczerner_fsm.pdf


The code is still under development and no release has been made yet, but
I would like to share with you what I have done so far, since the
progress has been a bit slower than I have previously expected. The
project lives on the sf.net :

https://sourceforge.net/projects/storagemanager/

and you can grab source files from git repository here:

https://sourceforge.net/p/storagemanager/code/ci/a1a5fd616d06030f94b9d2e80ee6ebcad09ad35f/tree/

More information can be found on the projects home page, or in the
README file.

https://sourceforge.net/p/storagemanager/home/Home/


Notes
-----

- It is written in python
- So far it supports commands : check, resize, create, list, add, remove
- More commands to come : mirror, snapshot(!)
- It does not support raid yet, except raid 0 from lvm (striped volume)
- It has been tested with python 2.7, but would like to make it work on
  python 2.6 as well.
- It comes with some doctests, unittests and regression tests written in
  bash (although it is for lvm only so far)
- Its "modular" design should make it relatively simple to add support
  for more back-ends other than lvm, or btrfs


Things to be done before the actual release
-------------------------------------------

- Create btrfs bash tests
- Create btrfs unittests
- Extend python unittests to other backends
- Use wipefs -a before using the device in add()
- Consider using wipefs -a after removing the device
- Remove the physical volume after it is removed from the group
- Figure out how to create better pool names so it is unique in the system and
  between the systems.
- Add mirror support
- Add snapshots support
- Add raid support
- use lsblk and blkid to get information
- Better table alignment when the output spans multiple lines
- Better error handling - not just plain Exception, but rather named exception
  and handle it as main() in ssm module
- Update readme
- Add more documentation into the code


Or course any comment or ideas would be highly appreciated. Please
report bugs directly to me.

Thanks!
-Lukas

Re: System Storage Manager

[Pádraig Brady]

On 12/07/2011 10:20 AM, Lukas Czerner wrote:
> Hello everyone,
>=20
> I would like to introduce to you a new tool called System Storage
> Manager (ssm). It is supposed to provide easy to use command line
> interface to manage your storage using various technologies like
> lvm, btrfs, encrypted volumes and possibly more.
>=20
> Background
> ----------
>=20
> In more sophisticated enterprise storage environments, management wit=
h
> Device Mapper (dm), Logical Volume Manager (LVM), or Multiple Devices
> (md) is becoming increasingly more difficult. With file systems added
> to the mix, the number of tools needed to configure and manage storag=
e
> has grown so large that it is simply not user friendly. With so many
> options for a system administrator to consider, the opportunity for
> errors and problems is large.

This seems like a worthwhile project given the overlap
in functionality from the various commands.
It echoes some of the functionality provided by libguestfs
for VM images I think.

> The btrfs administration tools have shown us that storage management =
can
> be simplified, and we are working to bring that ease of use to Linux
> filesystems in general.
>=20
>=20
> You can also find some more information in my presentation from Linux=
Con
> Prague this year:
>=20
> http://people.redhat.com/lczerner/files/lczerner_fsm.pdf

This had the name 'fsm'.
I do think 'ssm' is better, however I would drop the '.py' from
the command name. This is incidental and restrictive going forward.
Note I checked the debian and freebsd man pages and 'ssm' is available.

> The code is still under development and no release has been made yet,=
 but
> I would like to share with you what I have done so far, since the
> progress has been a bit slower than I have previously expected. The
> project lives on the sf.net :
>=20
> https://sourceforge.net/projects/storagemanager/

This is a much cleaner landing page:
http://sourceforge.net/p/storagemanager/home/Home/

cheers,
P=E1draig.

Re: System Storage Manager

[Pádraig Brady]

On 12/07/2011 10:20 AM, Lukas Czerner wrote:
> Hello everyone,
> 
> I would like to introduce to you a new tool called System Storage
> Manager (ssm). It is supposed to provide easy to use command line
> interface to manage your storage using various technologies like
> lvm, btrfs, encrypted volumes and possibly more.
> 
> Background
> ----------
> 
> In more sophisticated enterprise storage environments, management with
> Device Mapper (dm), Logical Volume Manager (LVM), or Multiple Devices
> (md) is becoming increasingly more difficult. With file systems added
> to the mix, the number of tools needed to configure and manage storage
> has grown so large that it is simply not user friendly. With so many
> options for a system administrator to consider, the opportunity for
> errors and problems is large.

This seems like a worthwhile project given the overlap
in functionality from the various commands.
It echoes some of the functionality provided by libguestfs
for VM images I think.

> The btrfs administration tools have shown us that storage management can
> be simplified, and we are working to bring that ease of use to Linux
> filesystems in general.
> 
> 
> You can also find some more information in my presentation from LinuxCon
> Prague this year:
> 
> http://people.redhat.com/lczerner/files/lczerner_fsm.pdf

This had the name 'fsm'.
I do think 'ssm' is better, however I would drop the '.py' from
the command name. This is incidental and restrictive going forward.
Note I checked the debian and freebsd man pages and 'ssm' is available.

> The code is still under development and no release has been made yet, but
> I would like to share with you what I have done so far, since the
> progress has been a bit slower than I have previously expected. The
> project lives on the sf.net :
> 
> https://sourceforge.net/projects/storagemanager/

This is a much cleaner landing page:
http://sourceforge.net/p/storagemanager/home/Home/

cheers,
Pádraig.

Re: [linux-lvm] System Storage Manager

[Pádraig Brady]

On 12/07/2011 10:20 AM, Lukas Czerner wrote:
> Hello everyone,
> 
> I would like to introduce to you a new tool called System Storage
> Manager (ssm). It is supposed to provide easy to use command line
> interface to manage your storage using various technologies like
> lvm, btrfs, encrypted volumes and possibly more.
> 
> Background
> ----------
> 
> In more sophisticated enterprise storage environments, management with
> Device Mapper (dm), Logical Volume Manager (LVM), or Multiple Devices
> (md) is becoming increasingly more difficult. With file systems added
> to the mix, the number of tools needed to configure and manage storage
> has grown so large that it is simply not user friendly. With so many
> options for a system administrator to consider, the opportunity for
> errors and problems is large.

This seems like a worthwhile project given the overlap
in functionality from the various commands.
It echoes some of the functionality provided by libguestfs
for VM images I think.

> The btrfs administration tools have shown us that storage management can
> be simplified, and we are working to bring that ease of use to Linux
> filesystems in general.
> 
> 
> You can also find some more information in my presentation from LinuxCon
> Prague this year:
> 
> http://people.redhat.com/lczerner/files/lczerner_fsm.pdf

This had the name 'fsm'.
I do think 'ssm' is better, however I would drop the '.py' from
the command name. This is incidental and restrictive going forward.
Note I checked the debian and freebsd man pages and 'ssm' is available.

> The code is still under development and no release has been made yet, but
> I would like to share with you what I have done so far, since the
> progress has been a bit slower than I have previously expected. The
> project lives on the sf.net :
> 
> https://sourceforge.net/projects/storagemanager/

This is a much cleaner landing page:
http://sourceforge.net/p/storagemanager/home/Home/

cheers,
P�draig.

Re: System Storage Manager

[Lukas Czerner]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2690 bytes --]

On Wed, 7 Dec 2011, Pádraig Brady wrote:

> On 12/07/2011 10:20 AM, Lukas Czerner wrote:
> > Hello everyone,
> > 
> > I would like to introduce to you a new tool called System Storage
> > Manager (ssm). It is supposed to provide easy to use command line
> > interface to manage your storage using various technologies like
> > lvm, btrfs, encrypted volumes and possibly more.
> > 
> > Background
> > ----------
> > 
> > In more sophisticated enterprise storage environments, management with
> > Device Mapper (dm), Logical Volume Manager (LVM), or Multiple Devices
> > (md) is becoming increasingly more difficult. With file systems added
> > to the mix, the number of tools needed to configure and manage storage
> > has grown so large that it is simply not user friendly. With so many
> > options for a system administrator to consider, the opportunity for
> > errors and problems is large.
> 
> This seems like a worthwhile project given the overlap
> in functionality from the various commands.
> It echoes some of the functionality provided by libguestfs
> for VM images I think.
> 
> > The btrfs administration tools have shown us that storage management can
> > be simplified, and we are working to bring that ease of use to Linux
> > filesystems in general.
> > 
> > 
> > You can also find some more information in my presentation from LinuxCon
> > Prague this year:
> > 
> > http://people.redhat.com/lczerner/files/lczerner_fsm.pdf
> 
> This had the name 'fsm'.

Yes I had to change it, since fsm implies file system, but it is not
only about file systems.

> I do think 'ssm' is better, however I would drop the '.py' from

Already done. But I can see that it is still present in the README file.
Will fix it, tanks.

> the command name. This is incidental and restrictive going forward.
> Note I checked the debian and freebsd man pages and 'ssm' is available.

I could not find it on http://manpages.debian.net/cgi-bin/man.cgi but I
know there is a project called ssm:

ssm.i686 : Macromolecular coordinate superposition library

but my project is called System Storage Manager and ssm is just the
executable. I could not find any binary with the same name (within
Fedora), not sure if that will be a problem...

> 
> > The code is still under development and no release has been made yet, but
> > I would like to share with you what I have done so far, since the
> > progress has been a bit slower than I have previously expected. The
> > project lives on the sf.net :
> > 
> > https://sourceforge.net/projects/storagemanager/
> 
> This is a much cleaner landing page:
> http://sourceforge.net/p/storagemanager/home/Home/
> 
> cheers,
> Pádraig.
> 

Thanks for comments!
-Lukas

Re: [linux-lvm] System Storage Manager

[Lukas Czerner]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2768 bytes --]

On Wed, 7 Dec 2011, P�draig Brady wrote:

> On 12/07/2011 10:20 AM, Lukas Czerner wrote:
> > Hello everyone,
> > 
> > I would like to introduce to you a new tool called System Storage
> > Manager (ssm). It is supposed to provide easy to use command line
> > interface to manage your storage using various technologies like
> > lvm, btrfs, encrypted volumes and possibly more.
> > 
> > Background
> > ----------
> > 
> > In more sophisticated enterprise storage environments, management with
> > Device Mapper (dm), Logical Volume Manager (LVM), or Multiple Devices
> > (md) is becoming increasingly more difficult. With file systems added
> > to the mix, the number of tools needed to configure and manage storage
> > has grown so large that it is simply not user friendly. With so many
> > options for a system administrator to consider, the opportunity for
> > errors and problems is large.
> 
> This seems like a worthwhile project given the overlap
> in functionality from the various commands.
> It echoes some of the functionality provided by libguestfs
> for VM images I think.
> 
> > The btrfs administration tools have shown us that storage management can
> > be simplified, and we are working to bring that ease of use to Linux
> > filesystems in general.
> > 
> > 
> > You can also find some more information in my presentation from LinuxCon
> > Prague this year:
> > 
> > http://people.redhat.com/lczerner/files/lczerner_fsm.pdf
> 
> This had the name 'fsm'.

Yes I had to change it, since fsm implies file system, but it is not
only about file systems.

> I do think 'ssm' is better, however I would drop the '.py' from

Already done. But I can see that it is still present in the README file.
Will fix it, tanks.

> the command name. This is incidental and restrictive going forward.
> Note I checked the debian and freebsd man pages and 'ssm' is available.

I could not find it on http://manpages.debian.net/cgi-bin/man.cgi but I
know there is a project called ssm:

ssm.i686 : Macromolecular coordinate superposition library

but my project is called System Storage Manager and ssm is just the
executable. I could not find any binary with the same name (within
Fedora), not sure if that will be a problem...

> 
> > The code is still under development and no release has been made yet, but
> > I would like to share with you what I have done so far, since the
> > progress has been a bit slower than I have previously expected. The
> > project lives on the sf.net :
> > 
> > https://sourceforge.net/projects/storagemanager/
> 
> This is a much cleaner landing page:
> http://sourceforge.net/p/storagemanager/home/Home/
> 
> cheers,
> P�draig.
> 

Thanks for comments!
-Lukas