Feature request - Subtree checkouts

Feature request - Subtree checkouts

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 964 bytes --]

Heya,

I'm trying to dig at various issues that are potential holdups for
migrating the Gentoo CVS tree into Git.

Since shallow checkouts are now available, there's just one more thing
that's missing: subtree checkouts. Not to be confused with sub-projects.

If the master tree has this as some example contents:
/foo
/abc/...
/bar/example
/bar/baz/some-content

We need to be able to check out arbitrary subtrees. So I might want to
check out everything (as the CVS administrator), while one of the more
focused developers just wants to check out /bar/baz/.

P.S. Does this list do some weird spam-blocking? I've tried 3 times now
to use git-send-email to send an unrelated minor patch set
(--subject-prefix for git-format-patch), and it has never shown up on
the list :-(.

-- 
Robin Hugh Johnson
Gentoo Linux Developer & Council Member
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 321 bytes --]

Re: Feature request - Subtree checkouts

[Shawn O. Pearce]

"Robin H. Johnson" <robbat2@gentoo.org> wrote:
> P.S. Does this list do some weird spam-blocking? I've tried 3 times now
> to use git-send-email to send an unrelated minor patch set
> (--subject-prefix for git-format-patch), and it has never shown up on
> the list :-(.

Yes.  It has cut down on our spam, but it has also caused some
things to be blocked, almost without good reason.  ;-)

I send a fair number of patches to this list (at times anyway) and
am also unable to use git-send-email.  If I have the email also CC
back to me it does make it through a number of SMTP server hops,
including my own spam filters, but it never makes it through the
Git mailing list.  So I dump the patches to an mbox with --stdout,
open them up in mutt and resend them there.  I've bound shift-G
to resend-message for that purpose...

-- 
Shawn.

Re: Feature request - Subtree checkouts

[Junio C Hamano]

"Shawn O. Pearce" <spearce@spearce.org> writes:

> "Robin H. Johnson" <robbat2@gentoo.org> wrote:
>> P.S. Does this list do some weird spam-blocking? I've tried 3 times now
>> to use git-send-email to send an unrelated minor patch set
>> (--subject-prefix for git-format-patch), and it has never shown up on
>> the list :-(.
>
> Yes.  It has cut down on our spam, but it has also caused some
> things to be blocked, almost without good reason.  ;-)
>
> I send a fair number of patches to this list (at times anyway) and
> am also unable to use git-send-email.  If I have the email also CC
> back to me it does make it through a number of SMTP server hops,
> including my own spam filters, but it never makes it through the
> Git mailing list.  So I dump the patches to an mbox with --stdout,
> open them up in mutt and resend them there.  I've bound shift-G
> to resend-message for that purpose...

Well, we need to do something about this.  I haven't seen
Robin's patch neither on the list nor in my mailbox (if they
were CC'ed to me).

One thing that people need to be careful about is which SMTP
server they use.  I had an impression (I do not use send-email
myself) that it defaulted to local MTA, so the mail trail would
look like your local MTA receives from the MUA (which is
send-email), which forwards it to whereever destination (or
intermediaries).  On the other hand, I suspect many people use
their ISP's SMTP server when using their usual MUA, so the mail
trail would look different.  I do not know what filtering vger
does, but if it is filtering based on the MTA address
(dul.dnsbl.sorbs filtering comes to mind), that would make a
difference.

Re: Feature request - Subtree checkouts

[Linus Torvalds]

On Tue, 10 Apr 2007, Junio C Hamano wrote:
> 
> One thing that people need to be careful about is which SMTP
> server they use.

ABSOLUTELY!

There are a ton of spam blockers that simply *refuse* to accept email from 
people who just randomly send to port 25. 

For example, I will personally never see email that comes directly to my 
email address though an open mail relay *or* from something that appears 
to be just a random botnet PC (I forget the exact rule, since I'm hapily 
ignorant of MIS, but I think it boils down to requiring a good reverse DNS 
lookup).

That's getting much more common. Most spam is done through botnets, and 
they still try to do the direct-to-port-25 thing, exactly because if you 
go through a *real* SMTP host, your ISP will generally shut you down 
pretty quickly if you're spamming.

So special spammers with their own machines will work with ISP's that 
don't care (they make money off the spammers), but botnets depend on 
cracked Windows machines, and those often have ISP's that *do* care, 
because they get complaints and it costs them money if they don't.

> I had an impression (I do not use send-email myself) that it defaulted 
> to local MTA, so the mail trail would look like your local MTA receives 
> from the MUA (which is send-email), which forwards it to whereever 
> destination (or intermediaries).

A lot of people configure their MUA to send specially, and never even 
configure their MTA at all apart from whatever default configuration it 
has.

For example, I may have sendmail installed on my machine, but its only 
purpose in life is to do *local* email delivery (using fetchmail -> 
sendmail). It wouldn't know how to send an email outside of the machine.

Instead, my MUA is configured to do this:

	# List of SMTP servers for sending mail. If blank: Unix Pine uses sendmail.
	smtp-server=localhost:10025

where localhost port 10025 is just a ssh tunnel to inside the osdl 
network. If I tried to send email any other way, people would often not 
accept it, because various SMTP servers will simply *refuse* to forward 
emails that claim to be coming from a point that isn't somethign that they 
actually *receive* email for.

And I don't think my setup is at all unusual. I may be unusual in my 
choice of MUA, but most MUA's will have configuration for where to send 
the mail, and I think very few people actually configure sendmail to do it 
right.

So no, I don't think people should assume that "sendmail" magically knows 
how to send emails other than to the local machine (ie use it to send 
problem reports to "root", or let fetchmail deliver to it for local 
emails, but that's about it)

			Linus

[OT] Re: Feature request - Subtree checkouts

[alan]

On Tue, 10 Apr 2007, Linus Torvalds wrote:

>
>
> On Tue, 10 Apr 2007, Junio C Hamano wrote:
>>
>> One thing that people need to be careful about is which SMTP
>> server they use.
>
> ABSOLUTELY!
>
> There are a ton of spam blockers that simply *refuse* to accept email from
> people who just randomly send to port 25.
>
> For example, I will personally never see email that comes directly to my
> email address though an open mail relay *or* from something that appears
> to be just a random botnet PC (I forget the exact rule, since I'm hapily
> ignorant of MIS, but I think it boils down to requiring a good reverse DNS
> lookup).

Depending on your definition of "good".

I run my mail server off my DSL line.  I prefer having control over my 
mail server instead of being chained to what my ISP provides.  (The 
problems of having been a sysadmin for way too many years.) I don't have 
control over the reverse ip address, but I do over my DNS resolution. 
(Well, most of it. A couple domains are sitting on really old dns servers 
from years past.)

> That's getting much more common. Most spam is done through botnets, and
> they still try to do the direct-to-port-25 thing, exactly because if you
> go through a *real* SMTP host, your ISP will generally shut you down
> pretty quickly if you're spamming.

Which makes Greylisting a useful tool.  However, some people define a 
"real SMTP host" as being the one your ISP provides and no other.  No 
matter how good your OS or how stringent your rulesets for sending mail 
are.

> So special spammers with their own machines will work with ISP's that
> don't care (they make money off the spammers), but botnets depend on
> cracked Windows machines, and those often have ISP's that *do* care,
> because they get complaints and it costs them money if they don't.

And the various RBL lists will just nuke everyone in the same class c. (I 
have seen it happen for people at colos.  Someone on the class c get 
compromised, sends spam and the whole address space gets blocked. 
Sometimes I think the anti-spam methods are as obnoxious as the spammers. 
Almost.)

>> I had an impression (I do not use send-email myself) that it defaulted
>> to local MTA, so the mail trail would look like your local MTA receives
>> from the MUA (which is send-email), which forwards it to whereever
>> destination (or intermediaries).
>
> A lot of people configure their MUA to send specially, and never even
> configure their MTA at all apart from whatever default configuration it
> has.

But the people who use git are probably the exceptions.  (At least I hope 
so...)

A number of programs assume that the local MTA can send mail.  (The bug 
handling software for Gnome is an example of that.)

This sounds more like a distro problem though.  The MTA configuration 
should be a proper part of an install, not something that you have to 
uncover and piece together later.

Of course I also believe that Sendmail configuration was designed by an 
organic chemistry major as part of an "experiment" in ergot derivitives.

-- 
"Invoking the supernatural can explain anything, and hence explains nothing."
                   - University of Utah bioengineering professor Gregory Clark

Re: [OT] Re: Feature request - Subtree checkouts

[Linus Torvalds]

On Tue, 10 Apr 2007, alan wrote:
> > 
> > For example, I will personally never see email that comes directly to my
> > email address though an open mail relay *or* from something that appears
> > to be just a random botnet PC (I forget the exact rule, since I'm hapily
> > ignorant of MIS, but I think it boils down to requiring a good reverse DNS
> > lookup).
> 
> Depending on your definition of "good".

Well, the most common case (and the thing I *think* our spam software does 
here) is to just confirm that the reverse DNS lookup (that you want to do 
*anyway* for the "Received" headers for the email) will resolve back to 
the same IP (aka "FCrDNS").

It's also possible to just not accept mail if the reverse lookup indicates 
that the sending IP address is a dynamic address, which you can sometimes 
see from the hostname. I would suggest you *not* name your hosts to 
contain a lot of numbers and the string "dhcp", for example ;)

> Sometimes I think the anti-spam methods are as obnoxious as the 
> spammers. Almost.)

I'll take strict anti-spam methods any day. I get about 10 pieces of spam 
a day, that I can handle easily without worrying about it. I shudder to 
even just think about what it used to be like before aggressive spam 
filtering.

So I'm personally *solidly* in the camp that says "if you want to send me 
email, it's worth making a conscious effort to not look like spam".

		Linus

Re: [OT] Re: Feature request - Subtree checkouts

[alan]

On Tue, 10 Apr 2007, Linus Torvalds wrote:

>
>
> On Tue, 10 Apr 2007, alan wrote:
>>>
>>> For example, I will personally never see email that comes directly to my
>>> email address though an open mail relay *or* from something that appears
>>> to be just a random botnet PC (I forget the exact rule, since I'm hapily
>>> ignorant of MIS, but I think it boils down to requiring a good reverse DNS
>>> lookup).
>>
>> Depending on your definition of "good".
>
> Well, the most common case (and the thing I *think* our spam software does
> here) is to just confirm that the reverse DNS lookup (that you want to do
> *anyway* for the "Received" headers for the email) will resolve back to
> the same IP (aka "FCrDNS").
>
> It's also possible to just not accept mail if the reverse lookup indicates
> that the sending IP address is a dynamic address, which you can sometimes
> see from the hostname. I would suggest you *not* name your hosts to
> contain a lot of numbers and the string "dhcp", for example ;)

I actually have a fixed IP address.

>> Sometimes I think the anti-spam methods are as obnoxious as the
>> spammers. Almost.)
>
> I'll take strict anti-spam methods any day. I get about 10 pieces of spam
> a day, that I can handle easily without worrying about it. I shudder to
> even just think about what it used to be like before aggressive spam
> filtering.

Greylisting dropped my spam level by at least 90%.  RBLs have, for the 
most part, had far too many false positives to be useful.  (If it was just 
me, it would not be so bad, but my wife gets mail on this server as well. 
She is not so forgiving.)

> So I'm personally *solidly* in the camp that says "if you want to send me
> email, it's worth making a conscious effort to not look like spam".

I try pretty hard. However, some anti-spam methods share some of the same 
methods with dowsing and other witchcraft. What "looks like spam" seems 
pretty subjective at times.

I do have a reason for being a bit negative about it.  I once ran a very 
large development list back in the last century.  A piece of spam got past 
my spam filters and onto the list.  Some idiot got offended and reported 
me as a spammer (because my ip was the first thing he saw) and reported me 
to my isp and to my isp's upstream feed.  Took me a couple days to get 
everyone calmed down.  Excessive anti-spammers still get on my nerves 
after that incident.

-- 
"Invoking the supernatural can explain anything, and hence explains nothing."
                   - University of Utah bioengineering professor Gregory Clark

Re: [OT] Re: Feature request - Subtree checkouts

[Christer Weinigel]

Linus Torvalds <torvalds@linux-foundation.org> writes:

> It's also possible to just not accept mail if the reverse lookup indicates 
> that the sending IP address is a dynamic address, which you can sometimes 
> see from the hostname. I would suggest you *not* name your hosts to 
> contain a lot of numbers and the string "dhcp", for example ;)

That would be a very bad idea I think.  Doing that would lose quite a
lot of small companies and individuals such as me that run a mail
server but are unable to get the ISP to change the reverse DNS.  For
example I do have a fixed IP, but have an reverse DNS pointer which
looks like 1-2-3-4-5a.foo.bar.bostream.se.  

Forcing everybody to send mail through their ISP (and I'm not even
sure if my ADSL subscription includes such a service) would be a big
loss.  First of all its a philosophical thing, I think that it's very
important that small shops or individuals should be able to control
the services they need, the internet is supposed to be peer to peer.
Second because the ISP's mess up a lot more often than I do, for
example Telia, one of the largest ISPs in Sweden have been having
massive mail server problems during the last week which I'm happily
unaffected by.

    /Christer

-- 
"Just how much can I get away with and still go to heaven?"

Christer Weinigel <christer@weinigel.se>  http://www.weinigel.se

Re: [OT] Re: Feature request - Subtree checkouts

[alan]

On Tue, 11 Apr 2007, Christer Weinigel wrote:

> Linus Torvalds <torvalds@linux-foundation.org> writes:
>
>> It's also possible to just not accept mail if the reverse lookup indicates
>> that the sending IP address is a dynamic address, which you can sometimes
>> see from the hostname. I would suggest you *not* name your hosts to
>> contain a lot of numbers and the string "dhcp", for example ;)
>
> That would be a very bad idea I think.  Doing that would lose quite a
> lot of small companies and individuals such as me that run a mail
> server but are unable to get the ISP to change the reverse DNS.  For
> example I do have a fixed IP, but have an reverse DNS pointer which
> looks like 1-2-3-4-5a.foo.bar.bostream.se.

I am in the same situation.  I also have three domains.  Which one do I 
pick?  I can't afford to get an individual ip address for each.  Virtual 
servers on a single ip also will have similar problems.

> Forcing everybody to send mail through their ISP (and I'm not even
> sure if my ADSL subscription includes such a service) would be a big
> loss.  First of all its a philosophical thing, I think that it's very
> important that small shops or individuals should be able to control
> the services they need, the internet is supposed to be peer to peer.
> Second because the ISP's mess up a lot more often than I do, for
> example Telia, one of the largest ISPs in Sweden have been having
> massive mail server problems during the last week which I'm happily
> unaffected by.

My ISP only reciently started to think about using greylisting.  They 
route all their mail through a filter service that I do not trust.  (Too 
many false positives.  I also expect that all this talk about "forking 
children" would get me on some list somewhere.)

I have more experience running mail servers than the people at my ISP. 
(Not their fault.  They are young.)

-- 
"Invoking the supernatural can explain anything, and hence explains nothing."
                   - University of Utah bioengineering professor Gregory Clark

Re: [OT] Re: Feature request - Subtree checkouts

[David Lang]

On Tue, 10 Apr 2007, alan wrote:

> 
> On Tue, 11 Apr 2007, Christer Weinigel wrote:
>
>> Linus Torvalds <torvalds@linux-foundation.org> writes:
>> 
>>> It's also possible to just not accept mail if the reverse lookup indicates
>>> that the sending IP address is a dynamic address, which you can sometimes
>>> see from the hostname. I would suggest you *not* name your hosts to
>>> contain a lot of numbers and the string "dhcp", for example ;)
>> 
>> That would be a very bad idea I think.  Doing that would lose quite a
>> lot of small companies and individuals such as me that run a mail
>> server but are unable to get the ISP to change the reverse DNS.  For
>> example I do have a fixed IP, but have an reverse DNS pointer which
>> looks like 1-2-3-4-5a.foo.bar.bostream.se.
>
> I am in the same situation.  I also have three domains.  Which one do I pick? 
> I can't afford to get an individual ip address for each.  Virtual servers on 
> a single ip also will have similar problems.

this isn't a problem. as long as you can lookup 1-2-3-4-5a.foo.bar.bostream.se. 
and get your IP address you pass this test.

David Lang

Re: [OT] Re: Feature request - Subtree checkouts

[Jimmy Tang]

On Tue, Apr 10, 2007 at 02:33:54PM -0700, alan wrote:
> >
> >For example, I will personally never see email that comes directly to my
> >email address though an open mail relay *or* from something that appears
> >to be just a random botnet PC (I forget the exact rule, since I'm hapily
> >ignorant of MIS, but I think it boils down to requiring a good reverse DNS
> >lookup).
> 
> Depending on your definition of "good".
> 
> I run my mail server off my DSL line.  I prefer having control over my 
> mail server instead of being chained to what my ISP provides.  (The 
> problems of having been a sysadmin for way too many years.) I don't have 
> control over the reverse ip address, but I do over my DNS resolution. 
> (Well, most of it. A couple domains are sitting on really old dns servers 
> from years past.)
> 
> >That's getting much more common. Most spam is done through botnets, and
> >they still try to do the direct-to-port-25 thing, exactly because if you
> >go through a *real* SMTP host, your ISP will generally shut you down
> >pretty quickly if you're spamming.
> 
> Which makes Greylisting a useful tool.  However, some people define a 
> "real SMTP host" as being the one your ISP provides and no other.  No 
> matter how good your OS or how stringent your rulesets for sending mail 
> are.
> 

greylisting unfortunately requires "some maintenance" to keep it
going well, and it also breaks some mail appliances and probably some
MTA's that are completely compliant in retrying to send mails. 

another tactic which is probably just as good if not better than
greylisting is "nolisting", that is to have your primary mx point to a
non-existant machine with a real ip-address and dns entries (or even
a machine with a firewall that runs iptables that blackholes or does
funky stuff to anything coming in on port 25 and then just drops the
connections). 

if the remote end is a compliant MTA it will failover to the secondary
mx which is a real machine that receives mail. but it probably suffers
from the same problem of mail appliances not being completely compliant
to the specs on how MTA's should work.

nolisting offers almost as good effects as greylisting without the
hassle of maintaining a list.


Jimmy.

-- 
Jimmy Tang
Trinity Centre for High Performance Computing,
Lloyd Building, Trinity College Dublin, Dublin 2, Ireland.
http://www.tchpc.tcd.ie/ | http://www.tchpc.tcd.ie/~jtang

Re: Feature request - Subtree checkouts

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 1732 bytes --]

On Tue, Apr 10, 2007 at 01:28:17PM -0700, Junio C Hamano wrote:
> Well, we need to do something about this.  I haven't seen
> Robin's patch neither on the list nor in my mailbox (if they
> were CC'ed to me).
They definitely were CC'd to you Junio.

> One thing that people need to be careful about is which SMTP
> server they use.  I had an impression (I do not use send-email
> myself) that it defaulted to local MTA, so the mail trail would
> look like your local MTA receives from the MUA (which is
> send-email), which forwards it to whereever destination (or
> intermediaries).  On the other hand, I suspect many people use
> their ISP's SMTP server when using their usual MUA, so the mail
> trail would look different.  I do not know what filtering vger
> does, but if it is filtering based on the MTA address
> (dul.dnsbl.sorbs filtering comes to mind), that would make a
> difference.
No, it's not SMTP path differences.

I _know_ that my mail path is identical for git-send-email as well as my
MUA, because I had problems with GIT and whitespace in email addresses
initially ;-).

Since I have access to all the mail servers in my path (home -> AUTH
SMTP @ work -> wherever), I dug at the logs, and found that vger did
indeed accept my email to the list, but the messages never turned up on
the list.

The only weirdness I saw in that, is that the envelope sender did not
appear to be set correct from git-send-email. Give me a moment to
explore down that path.

-- 
Robin Hugh Johnson
E-Mail     : robbat2@orbis-terrarum.net
Home Page  : http://www.orbis-terrarum.net/?l=people.robbat2
ICQ#       : 30269588 or 41961639
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 321 bytes --]

Re: Feature request - Subtree checkouts

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 548 bytes --]

On Tue, Apr 10, 2007 at 02:10:41PM -0700, Robin H. Johnson wrote:
> The only weirdness I saw in that, is that the envelope sender did not
> appear to be set correct from git-send-email. Give me a moment to
> explore down that path.
Ok, see my patch series for envelope sender, that is the source of the
problem.

-- 
Robin Hugh Johnson
E-Mail     : robbat2@orbis-terrarum.net
Home Page  : http://www.orbis-terrarum.net/?l=people.robbat2
ICQ#       : 30269588 or 41961639
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 321 bytes --]

Re: Feature request - Subtree checkouts

[Shawn O. Pearce]

Junio C Hamano <junkio@cox.net> wrote:
> One thing that people need to be careful about is which SMTP
> server they use.  I had an impression (I do not use send-email
> myself) that it defaulted to local MTA, so the mail trail would
> look like your local MTA receives from the MUA (which is
> send-email), which forwards it to whereever destination (or
> intermediaries).  On the other hand, I suspect many people use
> their ISP's SMTP server when using their usual MUA, so the mail
> trail would look different.  I do not know what filtering vger
> does, but if it is filtering based on the MTA address
> (dul.dnsbl.sorbs filtering comes to mind), that would make a
> difference.

I'm pretty sure the last time I tried git-send-email I had the
MTA path exactly the same.  My mutt sends to `localhost`, which
forwards over an SSL channel to my colo'd spearce.org mail server,
and that relays to the final destination.  Hence spearce.org mail
always originates from spearce.org.

Now I ran git-send-email on a different system, but had it connect
over SMTP to port 25 of the same system mutt runs on, so the
initial Received line was different, but othewrise the mail path
was the same.

I also have a few other addresses that I can send to that will go
out the spearce.org colo'd box to another SMTP system, then bounce
back, and the round-trip was the same for anything from mutt and
from git-send-email.

I marked the whole mess up to some mail header difference that
git-send-email wasn't sending the same way mutt was, and that vger
cared about. I didn't see the difference easily.  I just gave up.
;-)

-- 
Shawn.

Re: Feature request - Subtree checkouts

[Johannes Schindelin]

Hi,

On Tue, 10 Apr 2007, Robin H. Johnson wrote:

> Since shallow checkouts are now available, there's just one more thing 
> that's missing: subtree checkouts. Not to be confused with sub-projects.

It should not be _that_ difficult. If I'm correct, it should touch exactly 
the same code as the commit v1.3.0~263^2~6 ("Assume unchanged" git).

ATM I have no time to do anything about it, so feel free to give it a try.

Ciao,
Dscho