Question about scm security holes

Question about scm security holes

[walt]

I just saw this article about the "google hackers" exploiting weaknesses in scms,
Perforce in particular:

http://www.wired.com/threatlevel/2010/03/source-code-hacks/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+wired%2Findex+%28Wired%3A+Index+3+%28Top+Stories+2%29%29

I guess google didn't take Linus's advice to dump Perforce :)

I can't tell from the article if Perforce is any worse than any other scm for
security holes, in fact it seems to imply that others haven't been tested in
the same way.

Just curious if anyone here has any thoughts about how the article may or may
not have any relevance for git (git being the scm I use most, by far, which is
the reason I'm interested).

Thanks

Re: Question about scm security holes

[Avery Pennarun]

On Thu, Mar 4, 2010 at 3:09 PM, walt <w41ter@gmail.com> wrote:
> I can't tell from the article if Perforce is any worse than any other scm
> for security holes, in fact it seems to imply that others haven't been tested in
> the same way.
>
> Just curious if anyone here has any thoughts about how the article may or
> may not have any relevance for git (git being the scm I use most, by far, which
> is the reason I'm interested).

The attack was uninteresting.  The paper seems to go on and on about
different ways that an attacker can steal source code by accessing a
poorly-secured SCM server.  This discussion is kind of moot for git,
where every single developer workstation has a complete copy of the
entire project history anyway.

An attack in which someone untraceably modified the repo to contain
modified code would be a little more interesting.  git makes this sort
of thing pretty much impossible to do without it being *noticeable* at
least.  Traceable, not so much, because you can create a commit with
whatever committer/author names you want and then push them in.
Commits aren't GPG-signed, only tags are, so there are lots of ways to
forge a commit from someone else and mess up the audit log.  At least
you can't edit old commits without people noticing, though.

Have fun,

Avery

Re: Question about scm security holes

[John Tapsell]

On 5 March 2010 02:03, Avery Pennarun <apenwarr@gmail.com> wrote:
> modified code would be a little more interesting.  git makes this sort
> of thing pretty much impossible to do without it being *noticeable* at
> least.  Traceable, not so much, because you can create a commit with
> whatever committer/author names you want and then push them in.

Which is why you simply record the username of whoever pushed them in.
 This is what gitorious.org does etc.

John

Re: Question about scm security holes

[Avery Pennarun]

On Thu, Mar 4, 2010 at 10:00 PM, John Tapsell <johnflux@gmail.com> wrote:
> On 5 March 2010 02:03, Avery Pennarun <apenwarr@gmail.com> wrote:
>> modified code would be a little more interesting.  git makes this sort
>> of thing pretty much impossible to do without it being *noticeable* at
>> least.  Traceable, not so much, because you can create a commit with
>> whatever committer/author names you want and then push them in.
>
> Which is why you simply record the username of whoever pushed them in.
>  This is what gitorious.org does etc.

Not bad, but it's still very hard to trace properly.  Imagine I pull
from a peer, then push my combined branch into the central repo.
It'll say I'm pushing in patches from me *and* my friend.  Did I forge
them or are they real?

Avery

Re: Question about scm security holes

[John Tapsell]

On 5 March 2010 03:19, Avery Pennarun <apenwarr@gmail.com> wrote:
> On Thu, Mar 4, 2010 at 10:00 PM, John Tapsell <johnflux@gmail.com> wrote:
>> On 5 March 2010 02:03, Avery Pennarun <apenwarr@gmail.com> wrote:
>>> modified code would be a little more interesting.  git makes this sort
>>> of thing pretty much impossible to do without it being *noticeable* at
>>> least.  Traceable, not so much, because you can create a commit with
>>> whatever committer/author names you want and then push them in.
>>
>> Which is why you simply record the username of whoever pushed them in.
>>  This is what gitorious.org does etc.
>
> Not bad, but it's still very hard to trace properly.  Imagine I pull
> from a peer, then push my combined branch into the central repo.
> It'll say I'm pushing in patches from me *and* my friend.  Did I forge
> them or are they real?

While true, it's still traceable back to you.  You did the push, so
you are responsible for that code.  It wouldn't be any different to
just pushing a bad commit yourself.

Re: Question about scm security holes

[walt]

On 03/04/2010 06:03 PM, Avery Pennarun wrote:

> ...you can create a commit with
> whatever committer/author names you want and then push them in.
> Commits aren't GPG-signed, only tags are, so there are lots of ways to
> forge a commit from someone else and mess up the audit log...

Thanks, that's the kind of reply I was hoping for.  Do you think there
should be a way to sign the commits themselves, at least as an option?

I certainly wouldn't bother, but OTOH nobody wants to steal my code :-/

Do you suppose the devs at Adobe carry the complete source repository
home on their laptops every night?  (Not if they use Perforce, of course,
but they might if they adopted git as their scm.)

Re: Question about scm security holes

[Avery Pennarun]

On Thu, Mar 4, 2010 at 10:20 PM, walt <w41ter@gmail.com> wrote:
> On 03/04/2010 06:03 PM, Avery Pennarun wrote:
>> ...you can create a commit with
>> whatever committer/author names you want and then push them in.
>> Commits aren't GPG-signed, only tags are, so there are lots of ways to
>> forge a commit from someone else and mess up the audit log...
>
> Thanks, that's the kind of reply I was hoping for.  Do you think there
> should be a way to sign the commits themselves, at least as an option?
>
> I certainly wouldn't bother, but OTOH nobody wants to steal my code :-/

The whole thing is a bit overblown.  One of my friends once took me on
a tour of Microsoft on a weekend.  The place was mostly deserted, but
tons of developers left their workstations unlocked overnight, and
everyone had a private office.  And with tens of thousands of
developers on the campus, nobody would know if you're supposed to be
there or not.

It would have been easy to walk off with the source code to Windows
from one of those workstations.  The fact is, nobody really *wants*
the source code to Windows, except probably to look at it and be
horrified.

What would you do if you stole the source code to Adobe's flash
player?  It's illegal (in the U.S. anyway) to reverse engineer it and
it's illegal to steal it, so you're on the wrong side of the law no
matter how you pretend you managed to figure out a way around their
DRM or whatever.

People describe source code as a company's "crown jewels," but that's
a bit of a joke.  I can barely get our interns to figure out how to
compile and understand our code.  Expecting a thief to do it, with
nothing but a raw repo and hundreds of gigabytes of crap, is pure
paranoia.

Sneaking in patches?  Yeah, watch out for that.  But you should be
reviewing patch changelogs anyway.  At least git prevents people from
*retroactively* changing stuff; they can only add patches on top, so
it's easy to review after a break-in.

Have fun,

Avery

Re: Question about scm security holes

[Andreas Krey]

On Thu, 04 Mar 2010 21:03:08 +0000, Avery Pennarun wrote:
...
> where every single developer workstation has a complete copy of the
> entire project history anyway.

It's the point of a dev workstation to have access to the code,
so McAfees whining about SCMs letting that happen is moot.

What would be helping here is a separation between internet-facing
and local work into separate machines.

> least.  Traceable, not so much, because you can create a commit with
> whatever committer/author names you want and then push them in.

You can still log who pushed what into your blessed repo,
and hold that person accountable.

Andreas

Re: Question about scm security holes

[Johannes Schindelin]

Hi,

On Thu, 4 Mar 2010, Avery Pennarun wrote:

> An attack in which someone untraceably modified the repo to contain 
> modified code would be a little more interesting.

I disagree that stealing the code in this particular case is 
uninteresting. You know that there are billions in the business how to 
manipulate Google's search results. If you can see how Google rates 
websites, you can prepare your xxx sites for it, and nobody would be able 
to know, let alone prove, that you "reverse-engineered" the system.

> git makes this sort of thing pretty much impossible to do without it 
> being *noticeable* at least.

That is not true in all cases.

If you're talking about a workflow as git.git has it, you're right, there 
is a maintainer, and a refused push would ring all kinds of alarm bells 
there.

Except, of course, when the maintainer happens to work on different 
machines, and is likely to pull from her main repository quite often. 
Think "get something compiling on an obscure platform while developing 
something different on your main computer, then do a criss-cross merge at 
the end".

It gets even much, much worse in the common setup of companies: a central 
repository. (The two main reasons why a central repository is used are: 
tradition (we did it with Subversion, too), and bottleneck problems: a 
single maintainer reviewing all changes is often deemed too expensive 
and slow.)

So in the regular case, it is _very_ easy to sneak in a code-change 
unnoticedly.

The trick now is to craft the commit in such a manner that it will not be 
noticed retro-actively. This is a simple case of social engineering: you 
have to imitate the style of the committer/author you are impersonating. 
The commit message must look like the usual ones (typos, preferred words, 
grammar, length of paragraphs, comprehensibility, etc)

Likewise, the code has to be analyzed for style, and obviously for most 
likely targets of a backdoor (both in terms of "it is a perfect spot for 
a backdoor" and "it is not uncommon for the author to touch that 
part of the code").

Crafting the commit message and the backdoor needs some time, and it needs 
to be done _after_ succeeding with the break-in, as you can only then 
start analyzing style (and most likely workflow -- whether there is a 
single maintainer or whether everybody pushes to a single repository).

The most likely route, therefore is to have _two_ break-ins. One for 
reconaissance, the second for the actual change.

Conclusion: there are no technical reasons why Git should be better than 
Perforce when it comes to a break-in.

Short version: it's a social problem, so it needs a social solution.

Ciao,
Dscho

P.S.: Sorry for the overly long mail. I did not have time to make it 
short.

Re: Question about scm security holes

[Jakub Narebski]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
> On Thu, 4 Mar 2010, Avery Pennarun wrote:
> 
> > An attack in which someone untraceably modified the repo to contain 
> > modified code would be a little more interesting.

> > git makes this sort of thing pretty much impossible to do without it 
> > being *noticeable* at least.
> 
> That is not true in all cases.
> 
> If you're talking about a workflow as git.git has it, you're right, there 
> is a maintainer, and a refused push would ring all kinds of alarm bells 
> there.

[...]
> It gets even much, much worse in the common setup of companies: a central 
> repository. (The two main reasons why a central repository is used are: 
> tradition (we did it with Subversion, too), and bottleneck problems: a 
> single maintainer reviewing all changes is often deemed too expensive 
> and slow.)

About "bottleneck problem".  Frederick Brooks wrote in his seminal
book "The Mythical Man-Month" that recommended way of organizing teams
is *with a maintainer*.  But this is less known that his most famous
statement: "Adding manpower to a late software project makes it
later." (The Brooks's Law)... and I guess companies do not know about
this one either :-)

-- 
Jakub Narebski
Poland
ShadeHawk on #git

Re: Question about scm security holes

[Avery Pennarun]

On Fri, Mar 5, 2010 at 4:25 AM, Johannes Schindelin
<Johannes.Schindelin@gmx.de> wrote:
> The trick now is to craft the commit in such a manner that it will not be
> noticed retro-actively. This is a simple case of social engineering: you
> have to imitate the style of the committer/author you are impersonating.
> The commit message must look like the usual ones (typos, preferred words,
> grammar, length of paragraphs, comprehensibility, etc)
>
> Likewise, the code has to be analyzed for style, and obviously for most
> likely targets of a backdoor (both in terms of "it is a perfect spot for
> a backdoor" and "it is not uncommon for the author to touch that
> part of the code").

There is still one major advantage to preventing modification of past
commits: once you find out there's been a breach, you can just go back
through the commits *since* the breach and double-check them.  Without
that guarantee, you have to recheck *every* commit, which is much more
work.

Not to say that a sneaky commit would be easy to detect, though.  I
often add bugs to my own code without even trying to hide them, and
they're still pretty hard to find afterward.

Have fun,

Avery

Re: Question about scm security holes

[Johannes Schindelin]

Hi,

On Fri, 5 Mar 2010, Avery Pennarun wrote:

> On Fri, Mar 5, 2010 at 4:25 AM, Johannes Schindelin
> <Johannes.Schindelin@gmx.de> wrote:
> > The trick now is to craft the commit in such a manner that it will not be
> > noticed retro-actively. This is a simple case of social engineering: you
> > have to imitate the style of the committer/author you are impersonating.
> > The commit message must look like the usual ones (typos, preferred words,
> > grammar, length of paragraphs, comprehensibility, etc)
> >
> > Likewise, the code has to be analyzed for style, and obviously for most
> > likely targets of a backdoor (both in terms of "it is a perfect spot for
> > a backdoor" and "it is not uncommon for the author to touch that
> > part of the code").
> 
> There is still one major advantage to preventing modification of past
> commits: once you find out there's been a breach, you can just go back
> through the commits *since* the breach and double-check them.

If you find out which commit it was in the past, you can always revert it. 
It does not take Git to do it.

I am all in favor of Git, yes, but let's be honest: Git does not prevent 
an intelligent break-in.

To repeat, as I seem to not have made the point before: a break-in is a 
social problem, so it requires a social solution.

Ciao,
Dscho

Re: Question about scm security holes

[Daniel Barkalow]

On Thu, 4 Mar 2010, walt wrote:

> I just saw this article about the "google hackers" exploiting weaknesses in
> scms,
> Perforce in particular:
> 
> http://www.wired.com/threatlevel/2010/03/source-code-hacks/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+wired%2Findex+%28Wired%3A+Index+3+%28Top+Stories+2%29%29
> 
> I guess google didn't take Linus's advice to dump Perforce :)
> 
> I can't tell from the article if Perforce is any worse than any other scm for
> security holes, in fact it seems to imply that others haven't been tested in
> the same way.
> 
> Just curious if anyone here has any thoughts about how the article may or may
> not have any relevance for git (git being the scm I use most, by far, which is
> the reason I'm interested).

I took a look at the white paper the article links to. I had to ignore a 
lot of the introductory sections (yes, the most secure system would be to
prevent people from doing any work that might be stolen or released after 
it was corrupted), but I assume that the "findings" are the actually 
relevant part. Comparing git and Perforce here:

 - The Perforce server software for Windows installs to run as root. I'm 
   not sure what the norm is for git central repositories on Windows, but 
   it's probably better. I don't know if people actually run Perforce 
   servers on Windows in practice, either.

 - Perforce has built-in authorization and authentication. By default, it 
   allows unauthenticated people to create users without any specific 
   authorization. It transmits passwords in cleartext in some cases. It 
   discloses a lot of information about the authorization and 
   authentication in force to arbitrary people, including users of the 
   internal web site who do not have protocol access at all. It issues 
   login tickets that last a long time. The authorization controls are not 
   applied reliably to operations that modify the authorization and 
   authentication information in some of the server software. The initial 
   configuration with respect to access control is completely 
   unrestrictive. Git does not have built-in authorization or 
   authentication, so avoiding or making these mistakes is outside git's 
   scope.

 - Perforce sends all of content over the network in cleartext. This is 
   essentially true of git as well, but in order to get any sort of access 
   control with git, you need to use some wrapping method, which will 
   generally provide encryption as well.

 - Perforce stores, on the server, the location of the working directory 
   on the client, and this is used by the client to place files. Git does 
   not store this information at all.

In general, they seem to have found numerous flaws due to the fact that 
Perforce includes security-related code while not being designed by 
security specialists. Git is designed not to include security-related 
code, and to have properly developed security code control access to it. 
It is possible to run Perforce in a configuration where access control is 
external to Perforce, but it's not easy or standard.

On the other hand, I don't see any indication that the attack they were 
investigating used any of the problems they found, or any problems of a 
similar class. The actual attack seemed to involve a successful attack on 
the workstation of someone with legitimate priviledges, which the 
attackers then used. It's hard to say if any security measures on the part 
of the SCM could have any effect other than limiting the choice of the 
user to target.

	-Daniel
*This .sig left intentionally blank*