[ANNOUNCE] xen ocaml tools

[ANNOUNCE] xen ocaml tools

[Vincent Hanquez]

Hi xen-devel,

I'm pleased to announce the availability of the Xen ocaml tools
repository. This contains for now, a fully rewritten version of xenstored,
which should be better than the C version available in the xen repository.

= License

This is part of an effort to try to opensource some Citrix internals
things that the open-source community could use and/or find useful. As such
all the code in this repository are licensed under the LGPLv2 with ocaml
static exception as common in the ocaml world. Please see the LICENSE file.

= Improvements (compared to C xenstored)

* scalability: transactions have a much better behavior in this version
  leading to significant improvements during aggressive use of xenstored
  and preventing possible guest xenstored DoS.
* no disk database: everything is cheaply stored in memory.
* small code base in a functional language.

= Getting the source

as easy as: hg clone http://xenbits.xensource.com/ext/xen-ocaml-tools.hg

= Building

You need to have a built xen, and a working ocaml compiler (native and
bytecode). if you don't have xen installed in /, you need to set
your env XEN_DIST_ROOT to the xen dist path (on bourne shell
export XEN_DIST_ROOT=/path/to/my/built/xen/dist/)

once you're there, typing make will build the binaries.

= Installing

ocaml xenstored is a drop in replacement of C xenstored, replacing
/usr/sbin/xenstored by the ocaml one will just works.
You also need to copy xenstored.conf in the xenstored directory on the 
destination at /etc/xen/.

Once you did that, you need to restart your host
(just like if you would want to restart C xenstored).

= Any problems

well, let us know, Thomas (CCed) and I, we'll be happy to help you solve 
problems that may arise.

Cheers,
-- 
Vincent Hanquez

Re: [ANNOUNCE] xen ocaml tools

[Patrick Colp]

I'm really excited to see somebody else working on an OCaml XenStore! I 
was wondering if you could tell me what the difference are between this 
implementation and the one I recently released to the community?

I think it would be great if we could work together to merge our work so 
we can get the best of both worlds.


Patrick


Vincent Hanquez wrote:
> Hi xen-devel,
> 
> I'm pleased to announce the availability of the Xen ocaml tools
> repository. This contains for now, a fully rewritten version of xenstored,
> which should be better than the C version available in the xen repository.
> 
> = License
> 
> This is part of an effort to try to opensource some Citrix internals
> things that the open-source community could use and/or find useful. As such
> all the code in this repository are licensed under the LGPLv2 with ocaml
> static exception as common in the ocaml world. Please see the LICENSE file.
> 
> = Improvements (compared to C xenstored)
> 
> * scalability: transactions have a much better behavior in this version
>  leading to significant improvements during aggressive use of xenstored
>  and preventing possible guest xenstored DoS.
> * no disk database: everything is cheaply stored in memory.
> * small code base in a functional language.
> 
> = Getting the source
> 
> as easy as: hg clone http://xenbits.xensource.com/ext/xen-ocaml-tools.hg
> 
> = Building
> 
> You need to have a built xen, and a working ocaml compiler (native and
> bytecode). if you don't have xen installed in /, you need to set
> your env XEN_DIST_ROOT to the xen dist path (on bourne shell
> export XEN_DIST_ROOT=/path/to/my/built/xen/dist/)
> 
> once you're there, typing make will build the binaries.
> 
> = Installing
> 
> ocaml xenstored is a drop in replacement of C xenstored, replacing
> /usr/sbin/xenstored by the ocaml one will just works.
> You also need to copy xenstored.conf in the xenstored directory on the 
> destination at /etc/xen/.
> 
> Once you did that, you need to restart your host
> (just like if you would want to restart C xenstored).
> 
> = Any problems
> 
> well, let us know, Thomas (CCed) and I, we'll be happy to help you solve 
> problems that may arise.
> 
> Cheers,

Re: [ANNOUNCE] xen ocaml tools

[Vincent Hanquez]

Patrick Colp wrote:
> I'm really excited to see somebody else working on an OCaml XenStore! I 
> was wondering if you could tell me what the difference are between this 
> implementation and the one I recently released to the community?
>   
this is a bit hard to tell without testing your version.

but i think the main difference is the way we handle transactions, which 
should provide a stable average time to commit transactions when having 
lots of xenstore traffic from guests.
> I think it would be great if we could work together to merge our work so 
> we can get the best of both worlds.
>   
yes, it would be nice to not have 2 needlessly "competing" versions.
hopefully we can make that happens at some point.

-- 
Vincent

Re: [ANNOUNCE] xen ocaml tools

[Patrick Colp]

[-- Attachment #1: Type: text/plain, Size: 2107 bytes --]

Vincent Hanquez wrote:
> Patrick Colp wrote:
>> I'm really excited to see somebody else working on an OCaml XenStore! 
>> I was wondering if you could tell me what the difference are between 
>> this implementation and the one I recently released to the community?
>>   
> this is a bit hard to tell without testing your version.
> 
> but i think the main difference is the way we handle transactions, which 
> should provide a stable average time to commit transactions when having 
> lots of xenstore traffic from guests.

I think you're thinking of my initial release last year. The version I released 
a few months ago also has an in-memory store and greatly improved transactions. 
It was motivated by the need to survive things like DoS attacks.

I wrote a little attack program (in OCaml) which runs from any DomU and brought 
the original xenstored to its knees. With the attack going, it's impossible to 
bring a new domain up -- it just hangs forever attempting to bring it up. 
Basically, the attack just hammers xenstored with micro-transactions. With the 
original transaction system, which allows the first committing transaction in a 
generation to win, long transactions could never complete. I implemented 
transactions that would enable all concurrent but non-conflicting transactions 
to commit. This made my version of xenstored resilient to the attack.

I played around with this with your version too, but found that, while it would 
not hang forever while attempting to load a domain, it would instead die after a 
few seconds with the following error:

Error: (2, 'No such file or directory')

I tried with with the eagain mode thing (random dropping of 1/3 of all 
transactions) both enabled and disabled, but it had the same effect (except that 
with the mode enabled, 1/3 of all transactions would fail regardless of if they 
should or not).

I've been reading over your code and noticed that you seem to have a 
mini-implementation of libxc. I was wondering why you chose to do this over 
using the pre-existing libxenctrl? Does this make the final executable smaller?


Patrick

[-- Attachment #2: attack.tar.gz --]
[-- Type: application/x-gzip, Size: 7655 bytes --]

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [ANNOUNCE] xen ocaml tools

[Patrick Colp]

[-- Attachment #1: Type: text/plain, Size: 2598 bytes --]

I've cleaned up the attack code (and added copy right info), Makefile, and 
README a bit for the attack. Please find the revised version attached.


Patrick


Patrick Colp wrote:
> Vincent Hanquez wrote:
>> Patrick Colp wrote:
>>> I'm really excited to see somebody else working on an OCaml XenStore! 
>>> I was wondering if you could tell me what the difference are between 
>>> this implementation and the one I recently released to the community?
>>>   
>> this is a bit hard to tell without testing your version.
>>
>> but i think the main difference is the way we handle transactions, 
>> which should provide a stable average time to commit transactions when 
>> having lots of xenstore traffic from guests.
> 
> I think you're thinking of my initial release last year. The version I 
> released a few months ago also has an in-memory store and greatly 
> improved transactions. It was motivated by the need to survive things 
> like DoS attacks.
> 
> I wrote a little attack program (in OCaml) which runs from any DomU and 
> brought the original xenstored to its knees. With the attack going, it's 
> impossible to bring a new domain up -- it just hangs forever attempting 
> to bring it up. Basically, the attack just hammers xenstored with 
> micro-transactions. With the original transaction system, which allows 
> the first committing transaction in a generation to win, long 
> transactions could never complete. I implemented transactions that would 
> enable all concurrent but non-conflicting transactions to commit. This 
> made my version of xenstored resilient to the attack.
> 
> I played around with this with your version too, but found that, while 
> it would not hang forever while attempting to load a domain, it would 
> instead die after a few seconds with the following error:
> 
> Error: (2, 'No such file or directory')
> 
> I tried with with the eagain mode thing (random dropping of 1/3 of all 
> transactions) both enabled and disabled, but it had the same effect 
> (except that with the mode enabled, 1/3 of all transactions would fail 
> regardless of if they should or not).
> 
> I've been reading over your code and noticed that you seem to have a 
> mini-implementation of libxc. I was wondering why you chose to do this 
> over using the pre-existing libxenctrl? Does this make the final 
> executable smaller?
> 
> 
> Patrick
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

[-- Attachment #2: attack.tar.gz --]
[-- Type: application/x-gzip, Size: 7542 bytes --]

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [ANNOUNCE] xen ocaml tools

[Vincent Hanquez]

Patrick Colp wrote:
>
> I think you're thinking of my initial release last year. The version I released 
> a few months ago also has an in-memory store and greatly improved transactions. 
> It was motivated by the need to survive things like DoS attacks.
>   
Is that possible to find your version of xenstored in a tarball somewhere ?
attack.tar.gz seems to contains lots of things related to xenstored, but 
yet seems to missing watches and permissions.
> I wrote a little attack program (in OCaml) which runs from any DomU and brought 
> the original xenstored to its knees. With the attack going, it's impossible to 
> bring a new domain up -- it just hangs forever attempting to bring it up. 
> Basically, the attack just hammers xenstored with micro-transactions. With the 
> original transaction system, which allows the first committing transaction in a 
> generation to win, long transactions could never complete. I implemented 
> transactions that would enable all concurrent but non-conflicting transactions 
> to commit. This made my version of xenstored resilient to the attack.
>
> I played around with this with your version too, but found that, while it would 
> not hang forever while attempting to load a domain, it would instead die after a 
> few seconds with the following error:
>
> Error: (2, 'No such file or directory')
>   
i haven't really had time to look yet (i've been swamped with others 
things), but will try to run your program.
but what is dying in the scenario you described ? ocaml xenstored or the 
attack program ?
> I tried with with the eagain mode thing (random dropping of 1/3 of all 
> transactions) both enabled and disabled, but it had the same effect (except that 
> with the mode enabled, 1/3 of all transactions would fail regardless of if they 
> should or not).
>
> I've been reading over your code and noticed that you seem to have a 
> mini-implementation of libxc. I was wondering why you chose to do this over 
> using the pre-existing libxenctrl? Does this make the final executable smaller?
>   
libxenctrl is using the GPL license, whereas we chose the LGPL license 
with ocaml static exception everywhere.

-- 
Vincent

Re: [ANNOUNCE] xen ocaml tools

[Patrick Colp]

[-- Attachment #1: Type: text/plain, Size: 2840 bytes --]

Vincent Hanquez wrote:
> Patrick Colp wrote:
>>
>> I think you're thinking of my initial release last year. The version I 
>> released a few months ago also has an in-memory store and greatly 
>> improved transactions. It was motivated by the need to survive things 
>> like DoS attacks.
>>   
> Is that possible to find your version of xenstored in a tarball somewhere ?
> attack.tar.gz seems to contains lots of things related to xenstored, but 
> yet seems to missing watches and permissions.

The attack code utilises some of the code from the version of XenStore I 
wrote. I released a patch for it to the list as well as a link to my 
website where the code available as a gzip and bzip2:

website: http://cs.ubc.ca/~pjcolp/

bzip2 direct link: http://cs.ubc.ca/~pjcolp/xenstore-ocaml.tar.bz2
gzip direct link: http://cs.ubc.ca/~pjcolp/xenstore-ocaml.tar.gz

I'm attaching the gzip version to this e-mail as well. The code is 
designed to be compiled against Xen as a replacement to the C version. I 
put it in the tools/xenstore directory in the xen-unstable tree to 
compile it.

>> I wrote a little attack program (in OCaml) which runs from any DomU 
>> and brought the original xenstored to its knees. With the attack 
>> going, it's impossible to bring a new domain up -- it just hangs 
>> forever attempting to bring it up. Basically, the attack just hammers 
>> xenstored with micro-transactions. With the original transaction 
>> system, which allows the first committing transaction in a generation 
>> to win, long transactions could never complete. I implemented 
>> transactions that would enable all concurrent but non-conflicting 
>> transactions to commit. This made my version of xenstored resilient to 
>> the attack.
>>
>> I played around with this with your version too, but found that, while 
>> it would not hang forever while attempting to load a domain, it would 
>> instead die after a few seconds with the following error:
>>
>> Error: (2, 'No such file or directory')
>>   
> i haven't really had time to look yet (i've been swamped with others 
> things), but will try to run your program.
> but what is dying in the scenario you described ? ocaml xenstored or the 
> attack program ?

When the attack is run, neither the attack nor xenstored die, but while 
the attack is running it is impossible to start a domain. In the C 
version, it would hang forever attempting to start a domain. When I ran 
it against your OCaml version, it wouldn't hang but instead after a few 
seconds the domain start would quit with the error:

Error: (2, 'No such file or directory')


If you're interested, I think it would be great to trying to merge the 
two XenStores together to get the best of both worlds. Are you going to 
be at Xen Summit? If so, would you be interested in chatting about this 
in person?


Patrick

[-- Attachment #2: xenstore-ocaml.tar.gz --]
[-- Type: application/x-gzip, Size: 94611 bytes --]

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [ANNOUNCE] xen ocaml tools

[Jun Koi]

On Tue, Feb 17, 2009 at 12:01 AM, Vincent Hanquez
<vincent.hanquez@eu.citrix.com> wrote:
> Patrick Colp wrote:
>>
>> I think you're thinking of my initial release last year. The version I
>> released a few months ago also has an in-memory store and greatly improved
>> transactions. It was motivated by the need to survive things like DoS
>> attacks.
>>
>
> Is that possible to find your version of xenstored in a tarball somewhere ?
> attack.tar.gz seems to contains lots of things related to xenstored, but yet
> seems to missing watches and permissions.
>>
>> I wrote a little attack program (in OCaml) which runs from any DomU and
>> brought the original xenstored to its knees. With the attack going, it's
>> impossible to bring a new domain up -- it just hangs forever attempting to
>> bring it up. Basically, the attack just hammers xenstored with
>> micro-transactions. With the original transaction system, which allows the
>> first committing transaction in a generation to win, long transactions could
>> never complete. I implemented transactions that would enable all concurrent
>> but non-conflicting transactions to commit. This made my version of
>> xenstored resilient to the attack.
>>
>> I played around with this with your version too, but found that, while it
>> would not hang forever while attempting to load a domain, it would instead
>> die after a few seconds with the following error:
>>
>> Error: (2, 'No such file or directory')
>>
>
> i haven't really had time to look yet (i've been swamped with others
> things), but will try to run your program.
> but what is dying in the scenario you described ? ocaml xenstored or the
> attack program ?
>>
>> I tried with with the eagain mode thing (random dropping of 1/3 of all
>> transactions) both enabled and disabled, but it had the same effect (except
>> that with the mode enabled, 1/3 of all transactions would fail regardless of
>> if they should or not).
>>
>> I've been reading over your code and noticed that you seem to have a
>> mini-implementation of libxc. I was wondering why you chose to do this over
>> using the pre-existing libxenctrl? Does this make the final executable
>> smaller?
>>
>
> libxenctrl is using the GPL license, whereas we chose the LGPL license with
> ocaml static exception everywhere.

This part is interesting!

I have some doubts here: for ex, now I want to reimplement some parts
of libxenstore in BSD or LGPL. But I cannot avoid to include
"/usr/include/{xs,xenctrl|.h", because they have some must-have data
definition. But it seems all these headers are in GPL, so my code
violates the copyright! Or not??

Thanks,
Jun

Re: [ANNOUNCE] xen ocaml tools

[Keir Fraser]

On 17/02/2009 01:19, "Jun Koi" <junkoi2004@gmail.com> wrote:

>> libxenctrl is using the GPL license, whereas we chose the LGPL license with
>> ocaml static exception everywhere.
> 
> This part is interesting!
> 
> I have some doubts here: for ex, now I want to reimplement some parts
> of libxenstore in BSD or LGPL. But I cannot avoid to include
> "/usr/include/{xs,xenctrl|.h", because they have some must-have data
> definition. But it seems all these headers are in GPL, so my code
> violates the copyright! Or not??

I think as long as the boilerplate and formatting is your own, the actual
meat of the struct definitions you could take and relicense without really
any issue. Since as you say there is really no other way those definitions
could be written.

 -- Keir