Any plans to support JTA and XA in jgit?

Any plans to support JTA and XA in jgit?

[Farrukh Najmi]

Does the gjit team have any plans to implement support for JTA in gjit 
so as to allow distributed transactions using 2 phase commit? This would 
be very powerful when git is being used in conjunction with other 
transaction resource managers such as databases.

-- 
Regards,
Farrukh Najmi

Web: http://www.wellfleetsoftware.com

Re: Any plans to support JTA and XA in jgit?

[Robin Rosenberg]

torsdag 13 november 2008 20:20:49 skrev Farrukh Najmi:
> 
> Does the gjit team have any plans to implement support for JTA in gjit 
> so as to allow distributed transactions using 2 phase commit? This would 
> be very powerful when git is being used in conjunction with other 
> transaction resource managers such as databases.

No such plans exist. We do not even have a J2EE Resource Manager yet. I
did some toying implementing one. As for XA support, I guess that would
not be very hard per se, but my thoughts on JEE support was more in the direction
of gitweb-like stuff, i.e. reading.

Trying to involve git in distributed database transactions might be cool, but seriously: Do you
need it? As for JEE my ideas are: A nice JSP tag library and a resource manager. When is
an entirely different question, as is who. Did you look at  my experiment in a reply of mine
in another recent jgit thread?

The term "distributed" in XA is not quite the same as in distributed verison control. If it would,
then we'd send SQL commands over e-mail (now, /that/ would be cool :).

-- robin

Re: Any plans to support JTA and XA in jgit?

[Robin Rosenberg]

torsdag 13 november 2008 20:57:31 skrev Robin Rosenberg:
> As for JEE my ideas are: A nice JSP tag library and a resource manager. When is
s/resource manager/resource connector/

-- robin

Re: Any plans to support JTA and XA in jgit?

[Farrukh Najmi]

Robin Rosenberg wrote:
> torsdag 13 november 2008 20:20:49 skrev Farrukh Najmi:
>   
>> Does the gjit team have any plans to implement support for JTA in gjit 
>> so as to allow distributed transactions using 2 phase commit? This would 
>> be very powerful when git is being used in conjunction with other 
>> transaction resource managers such as databases.
>>     
>
> No such plans exist. We do not even have a J2EE Resource Manager yet. I
> did some toying implementing one. As for XA support, I guess that would
> not be very hard per se, but my thoughts on JEE support was more in the direction
> of gitweb-like stuff, i.e. reading.
>   

I am not an expert on this but what I thought could be done is to 
enhance jgit so it can serve the role of a JTA compliant transactional 
resource manager similar to that in JDBC, JMS etc. As part of that it 
could serve resources as JTA XAResource.
Not sure how hard all this could be though.
> Trying to involve git in distributed database transactions might be cool, but seriously: Do you
> need it? 

The problem I am trying to solve is this. In my service I need to store 
metadata in a relational db and content in git such that both either 
commit or not in a single transaction. If one commits and the other does 
not that is a serious integrity issue. Seems to me, two phase commit 
would be the right solution for that in the long run. This what JDBC + 
JMS topologies do.

A totally separate issue I have to sort out is how to handle multiple 
unrelated transactions that are modifying the same git repo. If a 
transaction needs to be rolled back how do roll back exactly some 
changes in some files in git that were impacted by the transaction. This 
is not easy because git (and most VCS) do not have transaction isolation 
like databases do. Any suggestions?

> As for JEE my ideas are: A nice JSP tag library and a resource manager. When is
> an entirely different question, as is who. Did you look at  my experiment in a reply of mine
> in another recent jgit thread?
>   

I am not very well versed in tag libraries myself. My situation is one 
where everything happens inside a SOAP service endpoint and so I suspect 
JSP tag libraries are not likely to be useful in that situation. Let me 
know if I am wrong in this assessment.

> The term "distributed" in XA is not quite the same as in distributed verison control. 
That is correct.

> If it would,
> then we'd send SQL commands over e-mail (now, /that/ would be cool :)

+1

Thanks very much for your help and that of other colleagues on the list.

-- 
Regards,
Farrukh Najmi

Web: http://www.wellfleetsoftware.com

Re: Any plans to support JTA and XA in jgit?

[Robin Rosenberg]

torsdag 13 november 2008 21:27:42 skrev Farrukh Najmi:
> A totally separate issue I have to sort out is how to handle multiple 
> unrelated transactions that are modifying the same git repo. If a 
> transaction needs to be rolled back how do roll back exactly some 
> changes in some files in git that were impacted by the transaction. This 
> is not easy because git (and most VCS) do not have transaction isolation 
> like databases do. Any suggestions?
Updating the head ref is the hard part, but we have locking already there
and it is atomic on any sane file system. There is no waiting implemented
so one gets an error if two threads try to update the same head simultaneously.
Seems doable I'd say. For server use one could wrap the repo with a real database,
and just let the disk based refs mirror the database view,  though that is quite a bit
off focus for my own needs.

> I am not very well versed in tag libraries myself. My situation is one 
> where everything happens inside a SOAP service endpoint and so I suspect 
> JSP tag libraries are not likely to be useful in that situation. Let me 
> know if I am wrong in this assessment.
not at all.

-- robin

Re: Any plans to support JTA and XA in jgit?

[Kyle Moffett]

On Thu, Nov 13, 2008 at 3:27 PM, Farrukh Najmi
<farrukh@wellfleetsoftware.com> wrote:
> The problem I am trying to solve is this. In my service I need to store
> metadata in a relational db and content in git such that both either commit
> or not in a single transaction. If one commits and the other does not that
> is a serious integrity issue. Seems to me, two phase commit would be the
> right solution for that in the long run. This what JDBC + JMS topologies do.

That's really easy!  First tweak jgit so that *instead* of using
.git/refs, it uses your database to store references and exports them
on a routine basis to .git/refs for debugging purposes.  Then, for
each database update:

(1)  Start transaction
(2)  Commit the change to GIT (adds ref update to the transaction)
(3)  Make other metadata updates
(4)  Commit transaction

Then set up periodic garbage collection and you're done!  If the
transaction is aborted, there will simply be a bunch of random loose
objects in the git repository, which will be cleaned up the next time
you garbage collect.  The ref update will be atomic and conditional
with the rest of the transaction, and in git the *only* part that
really matters for atomicity is the ref.

Cheers,
Kyle Moffett

Re: Any plans to support JTA and XA in jgit?

[Farrukh Najmi]

Kyle Moffett wrote:
> On Thu, Nov 13, 2008 at 3:27 PM, Farrukh Najmi
> <farrukh@wellfleetsoftware.com> wrote:
>   
>> The problem I am trying to solve is this. In my service I need to store
>> metadata in a relational db and content in git such that both either commit
>> or not in a single transaction. If one commits and the other does not that
>> is a serious integrity issue. Seems to me, two phase commit would be the
>> right solution for that in the long run. This what JDBC + JMS topologies do.
>>     
>
> That's really easy!  First tweak jgit so that *instead* of using
> .git/refs, it uses your database to store references and exports them
> on a routine basis to .git/refs for debugging purposes.  
Wow! This does sound simple in theory. Of course I have much to learn 
about jgit code base first.
Any pointers on what parts of the code are involved that I should start 
looking at? I would hate to
keep a fork of gjit around. Would this work be of interest to the 
project if it could be done in a way
that a few properties control whether to use database or not?

Thanks Kyle!

> Then, for
> each database update:
>
> (1)  Start transaction
> (2)  Commit the change to GIT (adds ref update to the transaction)
> (3)  Make other metadata updates
> (4)  Commit transaction
>
> Then set up periodic garbage collection and you're done!  If the
> transaction is aborted, there will simply be a bunch of random loose
> objects in the git repository, which will be cleaned up the next time
> you garbage collect.  The ref update will be atomic and conditional
> with the rest of the transaction, and in git the *only* part that
> really matters for atomicity is the ref.
>
> Cheers,
> Kyle Moffett
>
>   


-- 
Regards,
Farrukh Najmi

Web: http://www.wellfleetsoftware.com