Re: Any plans to support JTA and XA in jgit?

Re: Any plans to support JTA and XA in jgit?

[Robin Rosenberg]

torsdag 13 november 2008 20:20:49 skrev Farrukh Najmi:
> 
> Does the gjit team have any plans to implement support for JTA in gjit 
> so as to allow distributed transactions using 2 phase commit? This would 
> be very powerful when git is being used in conjunction with other 
> transaction resource managers such as databases.

No such plans exist. We do not even have a J2EE Resource Manager yet. I
did some toying implementing one. As for XA support, I guess that would
not be very hard per se, but my thoughts on JEE support was more in the direction
of gitweb-like stuff, i.e. reading.

Trying to involve git in distributed database transactions might be cool, but seriously: Do you
need it? As for JEE my ideas are: A nice JSP tag library and a resource manager. When is
an entirely different question, as is who. Did you look at  my experiment in a reply of mine
in another recent jgit thread?

The term "distributed" in XA is not quite the same as in distributed verison control. If it would,
then we'd send SQL commands over e-mail (now, /that/ would be cool :).

-- robin

Re: Any plans to support JTA and XA in jgit?

[Farrukh Najmi]

Robin Rosenberg wrote:
> torsdag 13 november 2008 20:20:49 skrev Farrukh Najmi:
>   
>> Does the gjit team have any plans to implement support for JTA in gjit 
>> so as to allow distributed transactions using 2 phase commit? This would 
>> be very powerful when git is being used in conjunction with other 
>> transaction resource managers such as databases.
>>     
>
> No such plans exist. We do not even have a J2EE Resource Manager yet. I
> did some toying implementing one. As for XA support, I guess that would
> not be very hard per se, but my thoughts on JEE support was more in the direction
> of gitweb-like stuff, i.e. reading.
>   

I am not an expert on this but what I thought could be done is to 
enhance jgit so it can serve the role of a JTA compliant transactional 
resource manager similar to that in JDBC, JMS etc. As part of that it 
could serve resources as JTA XAResource.
Not sure how hard all this could be though.
> Trying to involve git in distributed database transactions might be cool, but seriously: Do you
> need it? 

The problem I am trying to solve is this. In my service I need to store 
metadata in a relational db and content in git such that both either 
commit or not in a single transaction. If one commits and the other does 
not that is a serious integrity issue. Seems to me, two phase commit 
would be the right solution for that in the long run. This what JDBC + 
JMS topologies do.

A totally separate issue I have to sort out is how to handle multiple 
unrelated transactions that are modifying the same git repo. If a 
transaction needs to be rolled back how do roll back exactly some 
changes in some files in git that were impacted by the transaction. This 
is not easy because git (and most VCS) do not have transaction isolation 
like databases do. Any suggestions?

> As for JEE my ideas are: A nice JSP tag library and a resource manager. When is
> an entirely different question, as is who. Did you look at  my experiment in a reply of mine
> in another recent jgit thread?
>   

I am not very well versed in tag libraries myself. My situation is one 
where everything happens inside a SOAP service endpoint and so I suspect 
JSP tag libraries are not likely to be useful in that situation. Let me 
know if I am wrong in this assessment.

> The term "distributed" in XA is not quite the same as in distributed verison control. 
That is correct.

> If it would,
> then we'd send SQL commands over e-mail (now, /that/ would be cool :)

+1

Thanks very much for your help and that of other colleagues on the list.

-- 
Regards,
Farrukh Najmi

Web: http://www.wellfleetsoftware.com

Re: Any plans to support JTA and XA in jgit?

[Robin Rosenberg]

torsdag 13 november 2008 21:27:42 skrev Farrukh Najmi:
> A totally separate issue I have to sort out is how to handle multiple 
> unrelated transactions that are modifying the same git repo. If a 
> transaction needs to be rolled back how do roll back exactly some 
> changes in some files in git that were impacted by the transaction. This 
> is not easy because git (and most VCS) do not have transaction isolation 
> like databases do. Any suggestions?
Updating the head ref is the hard part, but we have locking already there
and it is atomic on any sane file system. There is no waiting implemented
so one gets an error if two threads try to update the same head simultaneously.
Seems doable I'd say. For server use one could wrap the repo with a real database,
and just let the disk based refs mirror the database view,  though that is quite a bit
off focus for my own needs.

> I am not very well versed in tag libraries myself. My situation is one 
> where everything happens inside a SOAP service endpoint and so I suspect 
> JSP tag libraries are not likely to be useful in that situation. Let me 
> know if I am wrong in this assessment.
not at all.

-- robin

Re: Any plans to support JTA and XA in jgit?

[Kyle Moffett]

On Thu, Nov 13, 2008 at 3:27 PM, Farrukh Najmi
<farrukh@wellfleetsoftware.com> wrote:
> The problem I am trying to solve is this. In my service I need to store
> metadata in a relational db and content in git such that both either commit
> or not in a single transaction. If one commits and the other does not that
> is a serious integrity issue. Seems to me, two phase commit would be the
> right solution for that in the long run. This what JDBC + JMS topologies do.

That's really easy!  First tweak jgit so that *instead* of using
.git/refs, it uses your database to store references and exports them
on a routine basis to .git/refs for debugging purposes.  Then, for
each database update:

(1)  Start transaction
(2)  Commit the change to GIT (adds ref update to the transaction)
(3)  Make other metadata updates
(4)  Commit transaction

Then set up periodic garbage collection and you're done!  If the
transaction is aborted, there will simply be a bunch of random loose
objects in the git repository, which will be cleaned up the next time
you garbage collect.  The ref update will be atomic and conditional
with the rest of the transaction, and in git the *only* part that
really matters for atomicity is the ref.

Cheers,
Kyle Moffett

Re: Any plans to support JTA and XA in jgit?

[Farrukh Najmi]

Kyle Moffett wrote:
> On Thu, Nov 13, 2008 at 3:27 PM, Farrukh Najmi
> <farrukh@wellfleetsoftware.com> wrote:
>   
>> The problem I am trying to solve is this. In my service I need to store
>> metadata in a relational db and content in git such that both either commit
>> or not in a single transaction. If one commits and the other does not that
>> is a serious integrity issue. Seems to me, two phase commit would be the
>> right solution for that in the long run. This what JDBC + JMS topologies do.
>>     
>
> That's really easy!  First tweak jgit so that *instead* of using
> .git/refs, it uses your database to store references and exports them
> on a routine basis to .git/refs for debugging purposes.  
Wow! This does sound simple in theory. Of course I have much to learn 
about jgit code base first.
Any pointers on what parts of the code are involved that I should start 
looking at? I would hate to
keep a fork of gjit around. Would this work be of interest to the 
project if it could be done in a way
that a few properties control whether to use database or not?

Thanks Kyle!

> Then, for
> each database update:
>
> (1)  Start transaction
> (2)  Commit the change to GIT (adds ref update to the transaction)
> (3)  Make other metadata updates
> (4)  Commit transaction
>
> Then set up periodic garbage collection and you're done!  If the
> transaction is aborted, there will simply be a bunch of random loose
> objects in the git repository, which will be cleaned up the next time
> you garbage collect.  The ref update will be atomic and conditional
> with the rest of the transaction, and in git the *only* part that
> really matters for atomicity is the ref.
>
> Cheers,
> Kyle Moffett
>
>   


-- 
Regards,
Farrukh Najmi

Web: http://www.wellfleetsoftware.com

Any overview available on jgit codebase?

[Farrukh Najmi]

I am looking for any available docs that will help me learn the layout 
of the jgit code base.

In the ideal world here is what would help newbies get productive with 
the project quickly:

1. Have a clear one line description available in the javadocs for each 
package.
2. Have a clear one line description available in javadocs for each 
class in each package.
3. Have a clear one line description available in javadocs for each 
method in each class.

I would gladly do this and contribute the changes back but its a chicken 
and egg situation since I do not know the code base.

I realize its a lot to ask but would anyone familiar with code base 
volunteer to do at least (1) above?

At minimum could anyone tell me what packages and classes to look at in 
order to make changes to jgit so it uses a database instead of 
filesystem to references.

Lastly, where can I find information on the structure of the .git directory?

Thanks for your help.

Kyle Moffett wrote:
> On Thu, Nov 13, 2008 at 3:27 PM, Farrukh Najmi
> <farrukh@wellfleetsoftware.com> wrote:
>   
>> The problem I am trying to solve is this. In my service I need to store
>> metadata in a relational db and content in git such that both either commit
>> or not in a single transaction. If one commits and the other does not that
>> is a serious integrity issue. Seems to me, two phase commit would be the
>> right solution for that in the long run. This what JDBC + JMS topologies do.
>>     
>
> That's really easy!  First tweak jgit so that *instead* of using
> .git/refs, it uses your database to store references and exports them
> on a routine basis to .git/refs for debugging purposes.  Then, for
> each database update:
>
> (1)  Start transaction
> (2)  Commit the change to GIT (adds ref update to the transaction)
> (3)  Make other metadata updates
> (4)  Commit transaction
>
> Then set up periodic garbage collection and you're done!  If the
> transaction is aborted, there will simply be a bunch of random loose
> objects in the git repository, which will be cleaned up the next time
> you garbage collect.  The ref update will be atomic and conditional
> with the rest of the transaction, and in git the *only* part that
> really matters for atomicity is the ref.
>
> Cheers,
> Kyle Moffett
>
>   


-- 
Regards,
Farrukh Najmi

Web: http://www.wellfleetsoftware.com

Re: Any overview available on jgit codebase?

[Shawn O. Pearce]

Farrukh Najmi <farrukh@wellfleetsoftware.com> wrote:
>
> At minimum could anyone tell me what packages and classes to look at in  
> order to make changes to jgit so it uses a database instead of  
> filesystem to references.

See RefDatabase, created by Repository.  You would want to replace
RefDatabase with your own implementation, which means abstracting
that class to be an abstract base class or an interface.

RefUpdate and RefLogWriter are connected to RefDatabase, so you
would likely need to modify those too in order to make the change,
especially since RefUpdate is the "INSERT OR UPDATE OR DELETE"
logic.  :)

BTW, I want to do that RefDatabase abstraction anyway, because I'd
like to get an in-memory only version of RefDatabase, to make unit
testing easier.  So abstracting this code may be useful beyond just
your desire to back Git with a JDBC-style database.

-- 
Shawn.

Re: Any overview available on jgit codebase?

[Farrukh Najmi]

Shawn O. Pearce wrote:
> Farrukh Najmi <farrukh@wellfleetsoftware.com> wrote:
>   
>> At minimum could anyone tell me what packages and classes to look at in  
>> order to make changes to jgit so it uses a database instead of  
>> filesystem to references.
>>     
>
> See RefDatabase, created by Repository.  You would want to replace
> RefDatabase with your own implementation, which means abstracting
> that class to be an abstract base class or an interface.
>
> RefUpdate and RefLogWriter are connected to RefDatabase, so you
> would likely need to modify those too in order to make the change,
> especially since RefUpdate is the "INSERT OR UPDATE OR DELETE"
> logic.  :)
>
> BTW, I want to do that RefDatabase abstraction anyway, because I'd
> like to get an in-memory only version of RefDatabase, to make unit
> testing easier.  So abstracting this code may be useful beyond just
> your desire to back Git with a JDBC-style database.
>   

This is just what I was hoping for. Thanks Shawn. I will try and do this 
next week and ask for help as needed. Thank you.

-- 
Regards,
Farrukh Najmi

Web: http://www.wellfleetsoftware.com