[PATCH v4 0/3] send-email: add oauth2 support and fix outlook breaking threads

[PATCH v4 0/3] send-email: add oauth2 support and fix outlook breaking threads

[Aditya Garg]

Hi all!

This patch series includes three changes:

1. It adds support for Oauth2 authentication, which is now compulsory by Microsoft.
   This patch has been rebased to the latest version from the original version
   at https://lore.kernel.org/git/20250125190131.48717-1-julian@swagemakers.org/

2. The second patch makes the script reply to the message id set by the outlook,
   since outlook has its own proprietary way to handle message ids,
   and does not allow user to set their own. As a result, threads were breaking.

3. The final patch adds a new option to generate passwords like OAuth2 tokens.
   This is useful for users who want to use a script which generates tokens for
   OAuth2 authentication.

Detailed description of each patch has been done in the respective patches

BTW, I am sending this series using the patched send-email by these patches from
Outlook!

v2:
- Fix errors flagged by the CI

v3:
- Add third patch to generate passwords like OAuth2 tokens

v4:
- Make log message of the second patch more clear.
- Change "Outlook: Retrieved Message-ID:" to "Outlook reassigned Message-ID to:"
- Update documentation for smtp-passeval.

Aditya Garg (2):
  send-email: retrieve Message-ID from outlook SMTP server
  send-email: add option to generate passswords like OAuth2 tokens

Julian Swagemakers (1):
  send-email: implement SMTP bearer authentication

 Documentation/git-send-email.adoc | 13 +++-
 git-send-email.perl               | 99 ++++++++++++++++++++++++++++++-
 2 files changed, 110 insertions(+), 2 deletions(-)

-- 
2.49.0

[PATCH v4 1/3] send-email: implement SMTP bearer authentication

[Aditya Garg]

From: Julian Swagemakers <julian@swagemakers.org>

Manually send SMTP AUTH command for auth type OAUTHBEARER and XOAUTH2.
This is necessary since they are currently not supported by the Perls
Authen::SASL module.

The bearer token needs to be passed in as the password. This can be done
with git-credential-oauth[0] after minor modifications[1]. Which will
allow using git send-email with Gmail and oauth2 authentication:

    [credential]
        helper = cache --timeout 7200    # two hours
        helper = oauth
    [sendemail]
        smtpEncryption = tls
        smtpServer = smtp.gmail.com
        smtpUser = example@gmail.com
        smtpServerPort = 587
        smtpauth = OAUTHBEARER

As well as Office 365 accounts:

    [credential]
        helper = cache --timeout 7200   # two hours
        helper = oauth
    [sendemail]
        smtpEncryption = tls
        smtpServer = smtp.office365.com
        smtpUser = example@example.com
        smtpServerPort = 587
        smtpauth = XOAUTH2

[0] https://github.com/hickford/git-credential-oauth
[1] https://github.com/hickford/git-credential-oauth/issues/48

Tested-by: M Hickford <mirth.hickford@gmail.com>
Signed-off-by: Julian Swagemakers <julian@swagemakers.org>
Signed-off-by: Aditya Garg <gargaditya08@live.com>
---
 Documentation/git-send-email.adoc |  5 ++-
 git-send-email.perl               | 64 ++++++++++++++++++++++++++++++-
 2 files changed, 67 insertions(+), 2 deletions(-)

diff --git a/Documentation/git-send-email.adoc b/Documentation/git-send-email.adoc
index 7f223db42d..1bf75c060d 100644
--- a/Documentation/git-send-email.adoc
+++ b/Documentation/git-send-email.adoc
@@ -213,7 +213,10 @@ SMTP server and if it is supported by the utilized SASL library, the mechanism
 is used for authentication. If neither 'sendemail.smtpAuth' nor `--smtp-auth`
 is specified, all mechanisms supported by the SASL library can be used. The
 special value 'none' maybe specified to completely disable authentication
-independently of `--smtp-user`
+independently of `--smtp-user`. Specifying `OAUTHBEARER` or `XOAUTH2` will
+bypass SASL negotiation and force bearer authentication. In this case the
+bearer token must be provided with `--smtp-pass` or using a credential helper
+and `--smtp-encryption=tls` must be set.
 
 --smtp-pass[=<password>]::
 	Password for SMTP-AUTH. The argument is optional: If no
diff --git a/git-send-email.perl b/git-send-email.perl
index 1f613fa979..a6cafda29c 100755
--- a/git-send-email.perl
+++ b/git-send-email.perl
@@ -1398,6 +1398,63 @@ sub smtp_host_string {
 	}
 }
 
+sub generate_oauthbearer_string {
+	# This will generate the oauthbearer string used for authentication.
+	#
+	# "n,a=" {User} ",^Ahost=" {Host} "^Aport=" {Port} "^Aauth=Bearer " {Access Token} "^A^A
+	#
+	# The first part `n,a=" {User} ",` is the gs2 header described in RFC5801.
+	# * gs2-cb-flag `n` -> client does not support CB
+	# * gs2-authzid `a=" {User} "`
+	#
+	# The second part are key value pairs containing host, port and auth as
+	# described in RFC7628.
+	#
+	# https://datatracker.ietf.org/doc/html/rfc5801
+	# https://datatracker.ietf.org/doc/html/rfc7628
+	my $username = shift;
+	my $token = shift;
+	return "n,a=$username,\001port=$smtp_server_port\001auth=Bearer $token\001\001";
+}
+
+sub generate_xoauth2_string {
+	# "user=" {User} "^Aauth=Bearer " {Access Token} "^A^A"
+	# https://developers.google.com/gmail/imap/xoauth2-protocol#initial_client_response
+	my $username = shift;
+	my $token = shift;
+	return "user=$username\001auth=Bearer $token\001\001";
+}
+
+sub smtp_bearer_auth {
+	my $username = shift;
+	my $token = shift;
+	my $auth_string;
+	if ($smtp_encryption ne "tls") {
+		# As described in RFC7628 TLS is required and will be enforced
+		# at this point.
+		#
+		# https://datatracker.ietf.org/doc/html/rfc7628#section-3
+		die sprintf(__("For %s TLS is required."), $smtp_auth);
+	}
+	if ($smtp_auth eq "OAUTHBEARER") {
+		$auth_string = generate_oauthbearer_string($username, $token);
+	} elsif ($smtp_auth eq "XOAUTH2") {
+		$auth_string = generate_xoauth2_string($username, $token);
+	}
+	my $encoded_auth_string = MIME::Base64::encode($auth_string, "");
+	$smtp->command("AUTH $smtp_auth $encoded_auth_string\r\n");
+	use Net::Cmd qw(CMD_OK);
+	if ($smtp->response() == CMD_OK){
+		return 1;
+	} else {
+		# Send dummy request on authentication failure according to rfc7628.
+		# https://datatracker.ietf.org/doc/html/rfc7628#section-3.2.3
+		$smtp->command(MIME::Base64::encode("\001"));
+		$smtp->response();
+		return 0;
+	}
+}
+
 # Returns 1 if authentication succeeded or was not necessary
 # (smtp_user was not specified), and 0 otherwise.
 
@@ -1436,7 +1493,12 @@ sub smtp_auth_maybe {
 
 		# catch all SMTP auth error in a unified eval block
 		eval {
-			if ($smtp_auth) {
+			if (defined $smtp_auth && ($smtp_auth eq "OAUTHBEARER" || $smtp_auth eq "XOAUTH2")) {
+				# Since Authen:SASL does not support XOAUTH2 nor OAUTHBEARER we will
+				# manually authenticate for these types. The password field should
+				# contain the auth token at this point.
+				$result = smtp_bearer_auth($cred->{'username'}, $cred->{'password'});
+			} elsif ($smtp_auth) {
 				my $sasl = Authen::SASL->new(
 					mechanism => $smtp_auth,
 					callback => {
-- 
2.49.0

Re: [PATCH v4 1/3] send-email: implement SMTP bearer authentication

[Junio C Hamano]

Aditya Garg <gargaditya08@live.com> writes:

> From: Julian Swagemakers <julian@swagemakers.org>
>
> Manually send SMTP AUTH command for auth type OAUTHBEARER and XOAUTH2.
> This is necessary since they are currently not supported by the Perls
> Authen::SASL module.
>
> The bearer token needs to be passed in as the password. This can be done
> with git-credential-oauth[0] after minor modifications[1]. Which will
> allow using git send-email with Gmail and oauth2 authentication:

I am not familiar with this area, especially with Authen::SASL, so
I'd appreciate a second eye from other reviewers.

Having said that, the documentation is very clearly written, so is
the above log message.

Please fold overly long lines the patch adds.  We officially aim for
80-column soft limit, and we allow going over it when folding lines
to stay under it makes the result less readable. But lines added to
the credential callback to call smtp_bearer_auth() are a way too
wide, for example [*].

Footnote [*] The text themselves are not overly wide, but the long
lines there are primarily due to them deeply indented.  I have to
wonder if it is a sign that the part of the code needs to be a bit
better refactored, e.g., by defining the callback sub defined
elsewhere and passed to Git::credential() call as a variable that
holds a reference to it, instead of defining an anonymous sub in
place there, for example.

Re: [PATCH v4 1/3] send-email: implement SMTP bearer authentication

[Aditya Garg]

> On 23 Apr 2025, at 11:34 PM, Junio C Hamano <gitster@pobox.com> wrote:
> 
> Aditya Garg <gargaditya08@live.com> writes:
> 
>> From: Julian Swagemakers <julian@swagemakers.org>
>> 
>> Manually send SMTP AUTH command for auth type OAUTHBEARER and XOAUTH2.
>> This is necessary since they are currently not supported by the Perls
>> Authen::SASL module.
>> 
>> The bearer token needs to be passed in as the password. This can be done
>> with git-credential-oauth[0] after minor modifications[1]. Which will
>> allow using git send-email with Gmail and oauth2 authentication:
> 
> I am not familiar with this area, especially with Authen::SASL, so
> I'd appreciate a second eye from other reviewers.

What I've noticed is that other reviewers didn't seem to have reviewed this
patch for more than a year when the original author proposed this patch.
Also, oauth2 is something that is significant in today's world and is definitely
more secure as well.

Nevertheless, your concern is quite valid, but I would also appreciate atleast
pinging the other reviewers who might have the knowledge. 

I'm Ccing Greg, who is credited for this script and the maintainer of the perl module
as well, with a hope to get a positive response.
> 
> Having said that, the documentation is very clearly written, so is
> the above log message.
> 
> Please fold overly long lines the patch adds.  We officially aim for
> 80-column soft limit, and we allow going over it when folding lines
> to stay under it makes the result less readable. But lines added to
> the credential callback to call smtp_bearer_auth() are a way too
> wide, for example [*].
> 
> Footnote [*] The text themselves are not overly wide, but the long
> lines there are primarily due to them deeply indented.  I have to
> wonder if it is a sign that the part of the code needs to be a bit
> better refactored, e.g., by defining the callback sub defined
> elsewhere and passed to Git::credential() call as a variable that
> holds a reference to it, instead of defining an anonymous sub in
> place there, for example.

Re: [PATCH v4 1/3] send-email: implement SMTP bearer authentication

[Greg Kroah-Hartman]

On Wed, Apr 23, 2025 at 06:33:52PM +0000, Aditya Garg wrote:
> 
> 
> > On 23 Apr 2025, at 11:34 PM, Junio C Hamano <gitster@pobox.com> wrote:
> > 
> > Aditya Garg <gargaditya08@live.com> writes:
> > 
> >> From: Julian Swagemakers <julian@swagemakers.org>
> >> 
> >> Manually send SMTP AUTH command for auth type OAUTHBEARER and XOAUTH2.
> >> This is necessary since they are currently not supported by the Perls
> >> Authen::SASL module.
> >> 
> >> The bearer token needs to be passed in as the password. This can be done
> >> with git-credential-oauth[0] after minor modifications[1]. Which will
> >> allow using git send-email with Gmail and oauth2 authentication:
> > 
> > I am not familiar with this area, especially with Authen::SASL, so
> > I'd appreciate a second eye from other reviewers.
> 
> What I've noticed is that other reviewers didn't seem to have reviewed this
> patch for more than a year when the original author proposed this patch.
> Also, oauth2 is something that is significant in today's world and is definitely
> more secure as well.
> 
> Nevertheless, your concern is quite valid, but I would also appreciate atleast
> pinging the other reviewers who might have the knowledge. 
> 
> I'm Ccing Greg, who is credited for this script and the maintainer of the perl module
> as well, with a hope to get a positive response.

Please when you resend the series, feel free to cc: me.  Dragging me in
like this doesn't help much without any context.

thanks,

greg k-h

Re: [PATCH v4 1/3] send-email: implement SMTP bearer authentication

[Aditya Garg]

Hi Greg

> 
> Please when you resend the series, feel free to cc: me.  Dragging me in
> like this doesn't help much without any context.
> 

I've Cced you in the v5 of this series.

For the context:

Junio is not familiar with Authen::SASL and thus, I added you in Cc for
your review regarding implementation of SMTP bearer authentication done
in the first patch.

Thanks
Aditya

[PATCH v4 2/3] send-email: retrieve Message-ID from outlook SMTP server

[Aditya Garg]

The script generates a Message-ID alongwith the other headers when
gen_header is called, and is sent alongwith the email. For most email
providers, including gmail, the Message-ID goes unchanged to the
recipient.

But, this does not seem to be a case with Outlook. In Outlook, when we
send our own Message-ID as a part of the headers, it discards it. Rather
it generates a new random Message-ID and that is was the recipient gets.
The Message-ID we specified get stored as a part of Outlook's
proprietary X-Microsoft-Original-Message-ID header.

This is a problem because the Message-ID is crucial when we are sending
multiple emails in a thread. The current implementation for threads in
the script replies to the Message-ID it generated, but due to Outlook's
behavior, it is not the same as the one that the recipient got, thus
breaking threads. So a need arises to retrieve the Message-ID from the
server response and set it in the In-Reply-To and References email
headers instead of using the self generated one for the purpose of
replies.

The $smtp->message variable in this script for outlook is something like
this:

2.0.0 OK <Message-ID> [Hostname=Some-hostname]

The Message-ID here is the one the receipient gets, rather than the one
the script generated.

This patch uses the fact above and retrieves the Message-ID from the
server response. It then changes the value of the $message_id variable
to the one received from the server. This value will be used when next
and subsequent messages are sent as replies to the message, thus
preserving the threading of the messages.

Signed-off-by: Aditya Garg <gargaditya08@live.com>
---
 git-send-email.perl | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/git-send-email.perl b/git-send-email.perl
index a6cafda29c..a18e978e22 100755
--- a/git-send-email.perl
+++ b/git-send-email.perl
@@ -1636,6 +1636,11 @@ sub gen_header {
 	return ($recipients_ref, $to, $date, $gitversion, $cc, $ccline, $header);
 }
 
+sub is_outlook {
+	my ($host) = @_;
+	return ($host eq 'smtp.office365.com' || $host eq 'smtp-mail.outlook.com');
+}
+
 # Prepares the email, then asks the user what to do.
 #
 # If the user chooses to send the email, it's sent and 1 is returned.
@@ -1799,6 +1804,21 @@ sub send_message {
 			$smtp->datasend("$line") or die $smtp->message;
 		}
 		$smtp->dataend() or die $smtp->message;
+
+		# Outlook discards the Message-ID header we set while sending the email.
+		# It instead saves it in its proprietary X-Microsoft-Original-Message-ID
+		# header and assigns a new random Message-ID to the email. So in order to
+		# avoid breaking threads, we simply retrieve the Message-ID from the server
+		# response and assign it to $message_id.
+		if (is_outlook($smtp_server)) {
+			if ($smtp->message =~ /<([^>]+)>/) {
+				$message_id = "<$1>";
+				printf __("Outlook reassigned Message-ID to: %s\n"), $message_id;
+			} else {
+				warn __("Warning: Could not retrieve Message-ID from server response.\n");
+			}
+		}
+
 		$smtp->code =~ /250|200/ or die sprintf(__("Failed to send %s\n"), $subject).$smtp->message;
 	}
 	if ($quiet) {
-- 
2.49.0

Re: [PATCH v4 2/3] send-email: retrieve Message-ID from outlook SMTP server

[Junio C Hamano]

Aditya Garg <gargaditya08@live.com> writes:

> The script generates a Message-ID alongwith the other headers when
> gen_header is called, and is sent alongwith the email. For most email
> providers, including gmail, the Message-ID goes unchanged to the
> recipient.
>
> But, this does not seem to be a case with Outlook. In Outlook, when we
> send our own Message-ID as a part of the headers, it discards it. Rather
> it generates a new random Message-ID and that is was the recipient gets.

"Rather" -> "Then".
"that is was the" -> "that is what the".

probably.

> The Message-ID we specified get stored as a part of Outlook's
> proprietary X-Microsoft-Original-Message-ID header.

For our purpose, X-MS-Original stuff is an extra noise that can be
omitted, as there is no way we or recipients can make good use of
the value on that field.

> This is a problem because the Message-ID is crucial when we are sending
> multiple emails in a thread. The current implementation for threads in
> the script replies to the Message-ID it generated, but due to Outlook's
> behavior, it is not the same as the one that the recipient got, thus
> breaking threads. So a need arises to retrieve the Message-ID from the
> server response and set it in the In-Reply-To and References email
> headers instead of using the self generated one for the purpose of
> replies.
>
> The $smtp->message variable in this script for outlook is something like
> this:
>
> 2.0.0 OK <Message-ID> [Hostname=Some-hostname]
>
> The Message-ID here is the one the receipient gets, rather than the one
> the script generated.
>
> This patch uses the fact above and retrieves the Message-ID from the
> server response. It then changes the value of the $message_id variable
> to the one received from the server. This value will be used when next
> and subsequent messages are sent as replies to the message, thus
> preserving the threading of the messages.
>
> Signed-off-by: Aditya Garg <gargaditya08@live.com>
> ---

Thanks for a thorough description.  It reads very well.

>  git-send-email.perl | 20 ++++++++++++++++++++
>  1 file changed, 20 insertions(+)
>
> diff --git a/git-send-email.perl b/git-send-email.perl
> index a6cafda29c..a18e978e22 100755
> --- a/git-send-email.perl
> +++ b/git-send-email.perl
> @@ -1636,6 +1636,11 @@ sub gen_header {
>  	return ($recipients_ref, $to, $date, $gitversion, $cc, $ccline, $header);
>  }
>  
> +sub is_outlook {
> +	my ($host) = @_;
> +	return ($host eq 'smtp.office365.com' || $host eq 'smtp-mail.outlook.com');
> +}
> +
>  # Prepares the email, then asks the user what to do.
>  #
>  # If the user chooses to send the email, it's sent and 1 is returned.
> @@ -1799,6 +1804,21 @@ sub send_message {
>  			$smtp->datasend("$line") or die $smtp->message;
>  		}
>  		$smtp->dataend() or die $smtp->message;
> +
> +		# Outlook discards the Message-ID header we set while sending the email.
> +		# It instead saves it in its proprietary X-Microsoft-Original-Message-ID
> +		# header and assigns a new random Message-ID to the email. So in order to

Again, "It instead ... header and" is probably better left unsaid.

> +		# avoid breaking threads, we simply retrieve the Message-ID from the server
> +		# response and assign it to $message_id.

Perhaps add ", which will then be assigned to $in_reply_to by the
caller when the next message is sent as a response to this message"
at the end?

Other than that, looks superb.  Thanks.

> +		if (is_outlook($smtp_server)) {
> +			if ($smtp->message =~ /<([^>]+)>/) {
> +				$message_id = "<$1>";
> +				printf __("Outlook reassigned Message-ID to: %s\n"), $message_id;
> +			} else {
> +				warn __("Warning: Could not retrieve Message-ID from server response.\n");
> +			}
> +		}
> +
>  		$smtp->code =~ /250|200/ or die sprintf(__("Failed to send %s\n"), $subject).$smtp->message;
>  	}
>  	if ($quiet) {

Re: [PATCH v4 2/3] send-email: retrieve Message-ID from outlook SMTP server

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 2348 bytes --]

On 2025-04-23 at 12:19:46, Aditya Garg wrote:
> This is a problem because the Message-ID is crucial when we are sending
> multiple emails in a thread. The current implementation for threads in
> the script replies to the Message-ID it generated, but due to Outlook's
> behavior, it is not the same as the one that the recipient got, thus
> breaking threads. So a need arises to retrieve the Message-ID from the
> server response and set it in the In-Reply-To and References email
> headers instead of using the self generated one for the purpose of
> replies.

This behaviour is allowed by the standard.  It's not uncommon for
smarthosts to replace the Message-ID header because they are responsible
for making it unique.

I certainly don't love it and it has the possibility to break lots of
things, as this patch demonstrates, but it is technically allowed.

> The $smtp->message variable in this script for outlook is something like
> this:
> 
> 2.0.0 OK <Message-ID> [Hostname=Some-hostname]
> 
> The Message-ID here is the one the receipient gets, rather than the one
> the script generated.

"recipient"

> diff --git a/git-send-email.perl b/git-send-email.perl
> index a6cafda29c..a18e978e22 100755
> --- a/git-send-email.perl
> +++ b/git-send-email.perl
> @@ -1636,6 +1636,11 @@ sub gen_header {
>  	return ($recipients_ref, $to, $date, $gitversion, $cc, $ccline, $header);
>  }
>  
> +sub is_outlook {
> +	my ($host) = @_;
> +	return ($host eq 'smtp.office365.com' || $host eq 'smtp-mail.outlook.com');
> +}

Are we certain that these are the only two possible values for this?  My
worry is that we'll have some other host (or the same host with some
other hostname) that does the same thing and then they'll have the same
problem.  For instance, if I set my domain `smtp-outlook.example.com` to
be a CNAME for `smtp.office365.com`, then this would fail and I'm
concerned that we'll have corporate environments with that
configuration.

What I would recommend here is that instead we set an option that
controls the message ID generation.  We might have "as-is" for the
default behaviour, "auto" to use the `is_outlook` function above to
guess, and something like "data-response" to always use the approach
you've written below.
-- 
brian m. carlson (they/them)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 325 bytes --]

Re: [PATCH v4 2/3] send-email: retrieve Message-ID from outlook SMTP server

[Aditya Garg]

> On 24 Apr 2025, at 4:23 AM, brian m. carlson <sandals@crustytoothpaste.net> wrote:
> 
> On 2025-04-23 at 12:19:46, Aditya Garg wrote:
>> This is a problem because the Message-ID is crucial when we are sending
>> multiple emails in a thread. The current implementation for threads in
>> the script replies to the Message-ID it generated, but due to Outlook's
>> behavior, it is not the same as the one that the recipient got, thus
>> breaking threads. So a need arises to retrieve the Message-ID from the
>> server response and set it in the In-Reply-To and References email
>> headers instead of using the self generated one for the purpose of
>> replies.
> 
> This behaviour is allowed by the standard.  It's not uncommon for
> smarthosts to replace the Message-ID header because they are responsible
> for making it unique.
> 
> I certainly don't love it and it has the possibility to break lots of
> things, as this patch demonstrates, but it is technically allowed.
> 
>> The $smtp->message variable in this script for outlook is something like
>> this:
>> 
>> 2.0.0 OK <Message-ID> [Hostname=Some-hostname]
>> 
>> The Message-ID here is the one the receipient gets, rather than the one
>> the script generated.
> 
> "recipient"
> 
>> diff --git a/git-send-email.perl b/git-send-email.perl
>> index a6cafda29c..a18e978e22 100755
>> --- a/git-send-email.perl
>> +++ b/git-send-email.perl
>> @@ -1636,6 +1636,11 @@ sub gen_header {
>>    return ($recipients_ref, $to, $date, $gitversion, $cc, $ccline, $header);
>> }
>> 
>> +sub is_outlook {
>> +    my ($host) = @_;
>> +    return ($host eq 'smtp.office365.com' || $host eq 'smtp-mail.outlook.com');
>> +}
> 
> Are we certain that these are the only two possible values for this?  My
> worry is that we'll have some other host (or the same host with some
> other hostname) that does the same thing and then they'll have the same
> problem.  For instance, if I set my domain `smtp-outlook.example.com` to
> be a CNAME for `smtp.office365.com`, then this would fail and I'm
> concerned that we'll have corporate environments with that
> configuration.
> 
> What I would recommend here is that instead we set an option that
> controls the message ID generation.  We might have "as-is" for the
> default behaviour, "auto" to use the `is_outlook` function above to
> guess, and something like "data-response" to always use the approach
> you've written below.

Tbh I'm against using additional options. They are an unecessary complication
for users. I would say the patch is a good start for such problems, and can be
expanded further as per needs. Also, I don't know how the server response is
for other non outlook email providers modifying the message ID, so data-response
isn't really ideal.

Talking about corporate environments, they have their own ways to send emails
and I doubt they really use git send-email.

> --
> brian m. carlson (they/them)
> Toronto, Ontario, CA
> <signature.asc>

[PATCH v4 3/3] send-email: add option to generate passswords like OAuth2 tokens

[Aditya Garg]

Some email providers like outlook allow only OAuth2 tokens to be used
for authentication. This commit adds an option to generate OAuth2 tokens
using scripts like M365-IMAP[1]. This option is similar to passwordeval
in msmtp.

Example usage:

[sendemail]
    smtpEncryption = tls
    smtpServer = smtp.office365.com
    smtpUser = someone@outlook.com
    smtpServerPort = 587
    smtpauth = XOAUTH2
    smtpPassEval = cd /workspaces/codespaces-blank/M365-IMAP && python3 ./refresh_token.py

Signed-off-by: Aditya Garg <gargaditya08@live.com>
---
 Documentation/git-send-email.adoc |  8 ++++++++
 git-send-email.perl               | 15 +++++++++++++++
 2 files changed, 23 insertions(+)

diff --git a/Documentation/git-send-email.adoc b/Documentation/git-send-email.adoc
index 1bf75c060d..f478559582 100644
--- a/Documentation/git-send-email.adoc
+++ b/Documentation/git-send-email.adoc
@@ -230,6 +230,14 @@ or on the command line. If a username has been specified (with
 specified (with `--smtp-pass` or `sendemail.smtpPass`), then
 a password is obtained using 'git-credential'.
 
+--smtp-passeval[=<command>]::
+	Generate password like OAuth2 token for SMTP AUTH. If specified,
+	it will use the output of the command specified as a password for
+	authentication.
++
+Note that it will override any existing password specified using
+`--smtp-pass` or a `sendemail.smtpPass`.
+
 --no-smtp-auth::
 	Disable SMTP authentication. Short hand for `--smtp-auth=none`
 
diff --git a/git-send-email.perl b/git-send-email.perl
index a18e978e22..cafb9aa43b 100755
--- a/git-send-email.perl
+++ b/git-send-email.perl
@@ -59,6 +59,8 @@ sub usage {
     --smtp-server-port      <int>  * Outgoing SMTP server port.
     --smtp-user             <str>  * Username for SMTP-AUTH.
     --smtp-pass             <str>  * Password for SMTP-AUTH; not necessary.
+    --smtp-passeval         <str>  * Path to script or a command to generate
+                                     password like OAuth2 token for SMTP-AUTH.
     --smtp-encryption       <str>  * tls or ssl; anything else disables.
     --smtp-ssl                     * Deprecated. Use '--smtp-encryption ssl'.
     --smtp-ssl-cert-path    <str>  * Path to ca-certificates (either directory or file).
@@ -280,6 +282,7 @@ sub do_edit {
 my ($auto_8bit_encoding);
 my ($compose_encoding);
 my ($sendmail_cmd);
+my ($smtp_authpasseval);
 my ($mailmap_file, $mailmap_blob);
 # Variables with corresponding config settings & hardcoded defaults
 my ($debug_net_smtp) = 0;		# Net::SMTP, see send_message()
@@ -316,6 +319,7 @@ sub do_edit {
     "smtppass" => \$smtp_authpass,
     "smtpdomain" => \$smtp_domain,
     "smtpauth" => \$smtp_auth,
+    "smtppasseval" => \$smtp_authpasseval,
     "smtpbatchsize" => \$batch_size,
     "smtprelogindelay" => \$relogin_delay,
     "to" => \@config_to,
@@ -516,6 +520,7 @@ sub config_regexp {
 		    "smtp-server-port=s" => \$smtp_server_port,
 		    "smtp-user=s" => \$smtp_authuser,
 		    "smtp-pass:s" => \$smtp_authpass,
+		    "smtp-passeval=s" => \$smtp_authpasseval,
 		    "smtp-ssl" => sub { $smtp_encryption = 'ssl' },
 		    "smtp-encryption=s" => \$smtp_encryption,
 		    "smtp-ssl-cert-path=s" => \$smtp_ssl_cert_path,
@@ -1463,6 +1468,16 @@ sub smtp_auth_maybe {
 		return 1;
 	}
 
+	# If smtpPassEval is set, run the user specified command to get the password
+	if (defined $smtp_authpasseval) {
+		printf __("Executing token generating script: %s\n"), $smtp_authpasseval;
+		chomp(my $generated_password = `$smtp_authpasseval 2>&1`);
+		if ($? != 0) {
+			die sprintf(__("Failed to execute token generating script: %s\n"), $smtp_authpasseval);
+		}
+		$smtp_authpass = $generated_password;
+	}
+
 	# Workaround AUTH PLAIN/LOGIN interaction defect
 	# with Authen::SASL::Cyrus
 	eval {
-- 
2.49.0

Re: [PATCH v4 3/3] send-email: add option to generate passswords like OAuth2 tokens

[Junio C Hamano]

Aditya Garg <gargaditya08@live.com> writes:

> @@ -230,6 +230,14 @@ or on the command line. If a username has been specified (with
>  specified (with `--smtp-pass` or `sendemail.smtpPass`), then
>  a password is obtained using 'git-credential'.
>  
> +--smtp-passeval[=<command>]::

Lose the pair of [] that marks the value optional.  Compare it with,
say, --smtp-user that is described as:

    --smtp-user=<user>::
            Username for SMTP-AUTH. Default is ...

because they are defined in %options (below) in a similar way, like
so:

>  		    "smtp-user=s" => \$smtp_authuser,
>  		    "smtp-pass:s" => \$smtp_authpass,
> +		    "smtp-passeval=s" => \$smtp_authpasseval,
>  		    "smtp-ssl" => sub { $smtp_encryption = 'ssl' },

taking a string value =s that is not optional.

> +	Generate password like OAuth2 token for SMTP AUTH. If specified,
> +	it will use the output of the command specified as a password for
> +	authentication.
> ++

> diff --git a/git-send-email.perl b/git-send-email.perl
> index a18e978e22..cafb9aa43b 100755
> --- a/git-send-email.perl
> +++ b/git-send-email.perl
> @@ -59,6 +59,8 @@ sub usage {
>      --smtp-server-port      <int>  * Outgoing SMTP server port.
>      --smtp-user             <str>  * Username for SMTP-AUTH.
>      --smtp-pass             <str>  * Password for SMTP-AUTH; not necessary.
> +    --smtp-passeval         <str>  * Path to script or a command to generate
> +                                     password like OAuth2 token for SMTP-AUTH.
>      --smtp-encryption       <str>  * tls or ssl; anything else disables.
>      --smtp-ssl                     * Deprecated. Use '--smtp-encryption ssl'.
>      --smtp-ssl-cert-path    <str>  * Path to ca-certificates (either directory or file).

Looking good.

> +	# If smtpPassEval is set, run the user specified command to get the password
> +	if (defined $smtp_authpasseval) {
> +		printf __("Executing token generating script: %s\n"), $smtp_authpasseval;
> +		chomp(my $generated_password = `$smtp_authpasseval 2>&1`);

How careful do we need to protect ourselves against a bad value in
this variable (like "rm -rf $HOME; password-command") ?  Are we OK
with trusting that the command line and the configuration file are
not under control of an attacker?  I am assuming it is OK, but you
folks have thought about this code path much longer than I have, so
I thought I should ask just to make sure.

Thanks.

Re: [PATCH v4 3/3] send-email: add option to generate passswords like OAuth2 tokens

[Aditya Garg]

> On 24 Apr 2025, at 12:33 AM, Junio C Hamano <gitster@pobox.com> wrote:
> 
> Aditya Garg <gargaditya08@live.com> writes:
> 
>> @@ -230,6 +230,14 @@ or on the command line. If a username has been specified (with
>> specified (with `--smtp-pass` or `sendemail.smtpPass`), then
>> a password is obtained using 'git-credential'.
>> 
>> +--smtp-passeval[=<command>]::
> 
> Lose the pair of [] that marks the value optional.  Compare it with,
> say, --smtp-user that is described as:

The value is not optional. It doesn't make sense to leave this empty right?
> 
>    --smtp-user=<user>::
>            Username for SMTP-AUTH. Default is ...
> 
> because they are defined in %options (below) in a similar way, like
> so:
> 
>>            "smtp-user=s" => \$smtp_authuser,
>>            "smtp-pass:s" => \$smtp_authpass,
>> +            "smtp-passeval=s" => \$smtp_authpasseval,
>>            "smtp-ssl" => sub { $smtp_encryption = 'ssl' },
> 
> taking a string value =s that is not optional.
> 
>> +    Generate password like OAuth2 token for SMTP AUTH. If specified,
>> +    it will use the output of the command specified as a password for
>> +    authentication.
>> ++
> 
>> diff --git a/git-send-email.perl b/git-send-email.perl
>> index a18e978e22..cafb9aa43b 100755
>> --- a/git-send-email.perl
>> +++ b/git-send-email.perl
>> @@ -59,6 +59,8 @@ sub usage {
>>     --smtp-server-port      <int>  * Outgoing SMTP server port.
>>     --smtp-user             <str>  * Username for SMTP-AUTH.
>>     --smtp-pass             <str>  * Password for SMTP-AUTH; not necessary.
>> +    --smtp-passeval         <str>  * Path to script or a command to generate
>> +                                     password like OAuth2 token for SMTP-AUTH.
>>     --smtp-encryption       <str>  * tls or ssl; anything else disables.
>>     --smtp-ssl                     * Deprecated. Use '--smtp-encryption ssl'.
>>     --smtp-ssl-cert-path    <str>  * Path to ca-certificates (either directory or file).
> 
> Looking good.
> 
>> +    # If smtpPassEval is set, run the user specified command to get the password
>> +    if (defined $smtp_authpasseval) {
>> +        printf __("Executing token generating script: %s\n"), $smtp_authpasseval;
>> +        chomp(my $generated_password = `$smtp_authpasseval 2>&1`);
> 
> How careful do we need to protect ourselves against a bad value in
> this variable (like "rm -rf $HOME; password-command") ?  Are we OK
> with trusting that the command line and the configuration file are
> not under control of an attacker?

I would ask the same question for the sendmailcmd option which also
invokes the shell. Ideally we should be able to use git credential manager,
but Microsoft has other plans. There are not very good methods to get OAuth2
tokens for free Microsoft accounts, such relatively unsafe ways are some easier
options.

Attackers these days can also get your access token, refresh token etc for OAuth2.
The user should be atleast aware of ways to keep their system secure. Maybe use
LUKS? And commands like rn -rf $HOME, I would say the risk is equal to typing 
sudo rm -rf $HOME on the terminal, but it doesn't mean we remove the terminal.

>  I am assuming it is OK, but your
> folks have thought about this code path much longer than I have, so
> I thought I should ask just to make sure.
> 
> Thanks.

Re: [PATCH v4 3/3] send-email: add option to generate passswords like OAuth2 tokens

[Junio C Hamano]

Aditya Garg <gargaditya08@live.com> writes:

>> On 24 Apr 2025, at 12:33 AM, Junio C Hamano <gitster@pobox.com> wrote:
>> 
>> Aditya Garg <gargaditya08@live.com> writes:
>> 
>>> @@ -230,6 +230,14 @@ or on the command line. If a username has been specified (with
>>> specified (with `--smtp-pass` or `sendemail.smtpPass`), then
>>> a password is obtained using 'git-credential'.
>>> 
>>> +--smtp-passeval[=<command>]::
>> 
>> Lose the pair of [] that marks the value optional.  Compare it with,
>> say, --smtp-user that is described as:
>
> The value is not optional. It doesn't make sense to leave this empty right?

That is why I said "Lose the pair of []".  IOW what we see above is
wrong and it should be something like

    --smtp-passeval=<command>::

without the [].  Just like --smtp-user takes a mandatory <user>,
like this:

>>    --smtp-user=<user>::
>>            Username for SMTP-AUTH. Default is ...

your "passeval" takes a mandatory <command>, hence =<command> should
NOT be enclosed in a pair of [].

Re: [PATCH v4 0/3] send-email: add oauth2 support and fix outlook breaking threads

[M Hickford]

On Wed, 23 Apr 2025 at 13:29, Aditya Garg <gargaditya08@live.com> wrote:
> This patch series includes three changes:
>
> 1. It adds support for Oauth2 authentication, which is now compulsory by Microsoft.
>    This patch has been rebased to the latest version from the original version
>    at https://lore.kernel.org/git/20250125190131.48717-1-julian@swagemakers.org/

Fantastic!

> BTW, I am sending this series using the patched send-email by these patches from
> Outlook!

Nice. I notice that patch v4 is a separate thread from patch v3. Any idea why?

Ideally all updates would belong to the same thread, as at
https://lore.kernel.org/git/20250420-505-wire-up-sparse-via-meson-v4-0-66e14134e822@gmail.com/#related

Re: [PATCH v4 0/3] send-email: add oauth2 support and fix outlook breaking threads

[Aditya Garg]

> On 24 Apr 2025, at 2:20 AM, M Hickford <mirth.hickford@gmail.com> wrote:
> 
> On Wed, 23 Apr 2025 at 13:29, Aditya Garg <gargaditya08@live.com> wrote:
>> This patch series includes three changes:
>> 
>> 1. It adds support for Oauth2 authentication, which is now compulsory by Microsoft.
>>   This patch has been rebased to the latest version from the original version
>>   at https://lore.kernel.org/git/20250125190131.48717-1-julian@swagemakers.org/
> 
> Fantastic!
> 
>> BTW, I am sending this series using the patched send-email by these patches from
>> Outlook!
> 
> Nice. I notice that patch v4 is a separate thread from patch v3. Any idea why?

I just didn't know I had to make them as a part of the same thread. It's my first
time contributing to git. I've only contributed to the Linux kernel before, where
there isn't any hard and fast rule to keep newer versions in the same thread.

> Ideally all updates would belong to the same thread, as at
> https://lore.kernel.org/git/20250420-505-wire-up-sparse-via-meson-v4-0-66e14134e822@gmail.com/#related

[PATCH v5 0/3] send-email: add oauth2 support and fix outlook breaking threads

[Aditya Garg]

Hi all!

This patch series includes three changes:

1. It adds support for Oauth2 authentication, which is now compulsory by Microsoft.
   This patch has been rebased to the latest version from the original version
   at https://lore.kernel.org/git/20250125190131.48717-1-julian@swagemakers.org/

2. The second patch makes the script reply to the message id set by the outlook,
   since outlook has its own proprietary way to handle message ids,
   and does not allow user to set their own. As a result, threads were breaking.

3. The final patch adds a new option to generate passwords like OAuth2 tokens.
   This is useful for users who want to use a script which generates tokens for
   OAuth2 authentication.

Detailed description of each patch has been done in the respective patches

BTW, I am sending this series using the patched send-email by these patches from
Outlook!

v2:
- Fix errors flagged by the CI

v3:
- Add third patch to generate passwords like OAuth2 tokens

v4:
- Make log message of the second patch more clear.
- Change "Outlook: Retrieved Message-ID:" to "Outlook reassigned Message-ID to:"
- Update documentation for smtp-passeval.

v5:
- Fix minor grammar issues in the commit messages.
- Attempt to wrap code in 80 characters in 1st patch.
- Create additional sub to check whether we are using OAuth2 authentication

Aditya Garg (2):
  send-email: retrieve Message-ID from outlook SMTP server
  send-email: add option to generate passswords like OAuth2 tokens

Julian Swagemakers (1):
  send-email: implement SMTP bearer authentication

 Documentation/git-send-email.adoc |  13 +++-
 git-send-email.perl               | 107 +++++++++++++++++++++++++++++-
 2 files changed, 118 insertions(+), 2 deletions(-)

-- 
2.49.0

[PATCH v5 1/3] send-email: implement SMTP bearer authentication

[Aditya Garg]

From: Julian Swagemakers <julian@swagemakers.org>

Manually send SMTP AUTH command for auth type OAUTHBEARER and XOAUTH2.
This is necessary since they are currently not supported by the Perls
Authen::SASL module.

The bearer token needs to be passed in as the password. This can be done
with git-credential-oauth[0] after minor modifications[1]. Which will
allow using git send-email with Gmail and oauth2 authentication:

    [credential]
        helper = cache --timeout 7200    # two hours
        helper = oauth
    [sendemail]
        smtpEncryption = tls
        smtpServer = smtp.gmail.com
        smtpUser = example@gmail.com
        smtpServerPort = 587
        smtpauth = OAUTHBEARER

As well as Office 365 accounts:

    [credential]
        helper = cache --timeout 7200   # two hours
        helper = oauth
    [sendemail]
        smtpEncryption = tls
        smtpServer = smtp.office365.com
        smtpUser = example@example.com
        smtpServerPort = 587
        smtpauth = XOAUTH2

[0] https://github.com/hickford/git-credential-oauth
[1] https://github.com/hickford/git-credential-oauth/issues/48

Tested-by: M Hickford <mirth.hickford@gmail.com>
Signed-off-by: Julian Swagemakers <julian@swagemakers.org>
Signed-off-by: Aditya Garg <gargaditya08@live.com>
---
 Documentation/git-send-email.adoc |  5 ++-
 git-send-email.perl               | 71 ++++++++++++++++++++++++++++++-
 2 files changed, 74 insertions(+), 2 deletions(-)

diff --git a/Documentation/git-send-email.adoc b/Documentation/git-send-email.adoc
index 7f223db42d..1bf75c060d 100644
--- a/Documentation/git-send-email.adoc
+++ b/Documentation/git-send-email.adoc
@@ -213,7 +213,10 @@ SMTP server and if it is supported by the utilized SASL library, the mechanism
 is used for authentication. If neither 'sendemail.smtpAuth' nor `--smtp-auth`
 is specified, all mechanisms supported by the SASL library can be used. The
 special value 'none' maybe specified to completely disable authentication
-independently of `--smtp-user`
+independently of `--smtp-user`. Specifying `OAUTHBEARER` or `XOAUTH2` will
+bypass SASL negotiation and force bearer authentication. In this case the
+bearer token must be provided with `--smtp-pass` or using a credential helper
+and `--smtp-encryption=tls` must be set.
 
 --smtp-pass[=<password>]::
 	Password for SMTP-AUTH. The argument is optional: If no
diff --git a/git-send-email.perl b/git-send-email.perl
index 1f613fa979..9ba47a6f38 100755
--- a/git-send-email.perl
+++ b/git-send-email.perl
@@ -1398,6 +1398,70 @@ sub smtp_host_string {
 	}
 }
 
+sub generate_oauthbearer_string {
+	# This will generate the oauthbearer string used for authentication.
+	#
+	# "n,a=" {User} ",^Ahost=" {Host} "^Aport=" {Port} "^Aauth=Bearer " {Access Token} "^A^A
+	#
+	# The first part `n,a=" {User} ",` is the gs2 header described in RFC5801.
+	# * gs2-cb-flag `n` -> client does not support CB
+	# * gs2-authzid `a=" {User} "`
+	#
+	# The second part are key value pairs containing host, port and auth as
+	# described in RFC7628.
+	#
+	# https://datatracker.ietf.org/doc/html/rfc5801
+	# https://datatracker.ietf.org/doc/html/rfc7628
+	my $username = shift;
+	my $token = shift;
+	return "n,a=$username,\001port=$smtp_server_port\001auth=Bearer $token\001\001";
+}
+
+sub generate_xoauth2_string {
+	# "user=" {User} "^Aauth=Bearer " {Access Token} "^A^A"
+	# https://developers.google.com/gmail/imap/xoauth2-protocol#initial_client_response
+	my $username = shift;
+	my $token = shift;
+	return "user=$username\001auth=Bearer $token\001\001";
+}
+
+sub smtp_bearer_auth {
+	my $username = shift;
+	my $token = shift;
+	my $auth_string;
+	if ($smtp_encryption ne "tls") {
+		# As described in RFC7628 TLS is required and will be enforced
+		# at this point.
+		#
+		# https://datatracker.ietf.org/doc/html/rfc7628#section-3
+		die sprintf(__("For %s TLS is required."), $smtp_auth);
+	}
+	if ($smtp_auth eq "OAUTHBEARER") {
+		$auth_string = generate_oauthbearer_string($username, $token);
+	} elsif ($smtp_auth eq "XOAUTH2") {
+		$auth_string = generate_xoauth2_string($username, $token);
+	}
+	my $encoded_auth_string = MIME::Base64::encode($auth_string, "");
+	$smtp->command("AUTH $smtp_auth $encoded_auth_string\r\n");
+	use Net::Cmd qw(CMD_OK);
+	if ($smtp->response() == CMD_OK){
+		return 1;
+	} else {
+		# Send dummy request on authentication failure according to rfc7628.
+		# https://datatracker.ietf.org/doc/html/rfc7628#section-3.2.3
+		$smtp->command(MIME::Base64::encode("\001"));
+		$smtp->response();
+		return 0;
+	}
+}
+
+# Check if we are using OAuth2.0 tokens
+
+sub is_smtp_bearer_auth {
+	my ($auth_method) = @_;
+	return ($auth_method eq "OAUTHBEARER" || $auth_method eq "XOAUTH2");
+}
+
 # Returns 1 if authentication succeeded or was not necessary
 # (smtp_user was not specified), and 0 otherwise.
 
@@ -1436,7 +1500,12 @@ sub smtp_auth_maybe {
 
 		# catch all SMTP auth error in a unified eval block
 		eval {
-			if ($smtp_auth) {
+			if (defined $smtp_auth && (is_smtp_bearer_auth($smtp_auth))) {
+				# Since Authen:SASL does not support XOAUTH2 nor OAUTHBEARER we
+				# will manually authenticate for these types. The password field
+				# should contain the auth token at this point.
+				$result = smtp_bearer_auth($cred->{'username'}, $cred->{'password'});
+			} elsif ($smtp_auth) {
 				my $sasl = Authen::SASL->new(
 					mechanism => $smtp_auth,
 					callback => {
-- 
2.49.0

Re: [PATCH v5 1/3] send-email: implement SMTP bearer authentication

[Julian Swagemakers]

Nice to see this gaining attention, if desired I'm available for
questions or changes. I'm unsure how etiquette is when someone resends
a patch and who is responsible for followups.

Regards Julian

Re: [PATCH v5 1/3] send-email: implement SMTP bearer authentication

[Aditya Garg]

Hi Eric

> As I said in https://github.com/gbarr/perl-authen-sasl/issues/18#issuecomment-2453040190 <https://github.com/gbarr/perl-authen-sasl/issues/18#issuecomment-2453040190> : I'd love to implement XOAUTH2 and/or OAUTHBEARER, but I don't have a setup available to test with, so I need someone to provide a patch *and* a description of how to test, preferably against a publicly available service so I don't need to go through complex setup myself.

I saw the code of perl-authen-sasl and I miserably failed in porting the logic used here to the repo.

Perl is not something I am very strong at, so not really feasible for me to work ahead. I do was able to return the formatted base64 encoded string as per the logic, but still I got authentication issues. Currently I aim to have git-send-email working, and a review from an experienced person is needed.

I would be happy to be able to test any proposed patch though.

Julian, you might be interested here?

If you are interested, you can checkout my XOAUTH2.pm file, which doesn't authenticate for some reason.

-->8--

package Authen::SASL::Perl::XOAUTH2;

use strict;
use warnings;
use MIME::Base64;
use vars qw($VERSION @ISA);

$VERSION = "1.00";
@ISA     = qw(Authen::SASL::Perl);

my %secflags = (
    noanonymous => 1,
);

sub _order { 1 }

sub _secflags {
    shift;
    scalar grep { $secflags{$_} } @_;
}

sub mechanism { 'XOAUTH2' }

sub client_start {
    my $self = shift;
    $self->{stage} = 0;
    '';
}

sub client_step {
    my ($self, $challenge) = @_;

    my $stage = ++$self->{stage};
    if ($stage == 1) {
        # Generate the XOAUTH2 authentication string
        my $username = $self->_call('user');
        my $token    = $self->_call('pass'); # OAuth 2.0 access token
        my $auth_string = "user=$username\001auth=Bearer $token\001\001";
        my $encoded_auth_string = encode_base64($auth_string, '');

        # Send the encoded authentication string
        return $encoded_auth_string;
    } else {
        # Handle authentication failure by sending a dummy request
        my $dummy_request = encode_base64("\001", '');
        return $dummy_request;
    }
}
}

1;

__END__

=head1 NAME

Authen::SASL::Perl::XOAUTH2 - XOAUTH2 Authentication class

=head1 SYNOPSIS

  use Authen::SASL qw(Perl);

  $sasl = Authen::SASL->new(
    mechanism => 'XOAUTH2',
    callback  => {
      user => $user,
      pass => $access_token,
    },
  );

=head1 DESCRIPTION

This module implements the client side of the XOAUTH2 SASL mechanism, which is used for OAuth 2.0-based authentication.

=head2 CALLBACK

The callbacks used are:

=head3 Client

=over 4

=item user

The username to be used for authentication.

=item pass

The OAuth 2.0 access token to be used for authentication.

=back

=head1 SEE ALSO

L<Authen::SASL>,
L<MIME::Base64>

=head1 AUTHORS

Written by [Your Name].

=head1 COPYRIGHT

This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

=cut

----
> 
> Please let me know who to talk to to have it added!
> 
> 
> -- 
> Bye,
> 
> Erik.
> 
> http://efficito.com <http://efficito.com/> -- Hosted accounting and ERP.
> Robust and Flexible. No vendor lock-in.

Re: [PATCH v5 1/3] send-email: implement SMTP bearer authentication

[Erik Huelsmann]

Hi Aditya,


On Thu, Apr 24, 2025 at 8:23 PM Aditya Garg <gargaditya08@live.com> wrote:
>
> Hi Eric
>
> > As I said in https://github.com/gbarr/perl-authen-sasl/issues/18#issuecomment-2453040190 <https://github.com/gbarr/perl-authen-sasl/issues/18#issuecomment-2453040190> : I'd love to implement XOAUTH2 and/or OAUTHBEARER, but I don't have a setup available to test with, so I need someone to provide a patch *and* a description of how to test, preferably against a publicly available service so I don't need to go through complex setup myself.
>
> I saw the code of perl-authen-sasl and I miserably failed in porting the logic used here to the repo.
>
> Perl is not something I am very strong at, so not really feasible for me to work ahead. I do was able to return the formatted base64 encoded string as per the logic, but still I got authentication issues.

I took a look at the code you sent XOAUTH2.pm. After a bit of
puzzling, I found what's going wrong: I think you should return the
base64 encoded string from "client_start()" instead of from the first
step. The step function should only be called in case the server
returns an error.

> Currently I aim to have git-send-email working, and a review from an experienced person is needed.
>
> I would be happy to be able to test any proposed patch though.



> Julian, you might be interested here?
>
> If you are interested, you can checkout my XOAUTH2.pm file, which doesn't authenticate for some reason.
>
> -->8--
>
> package Authen::SASL::Perl::XOAUTH2;
>
> use strict;
> use warnings;
> use MIME::Base64;
> use vars qw($VERSION @ISA);
>
> $VERSION = "1.00";
> @ISA     = qw(Authen::SASL::Perl);
>
> my %secflags = (
>     noanonymous => 1,
> );
>
> sub _order { 1 }
>
> sub _secflags {
>     shift;
>     scalar grep { $secflags{$_} } @_;
> }
>
> sub mechanism { 'XOAUTH2' }
>
> sub client_start {
>     my $self = shift;
>     $self->{stage} = 0;
>     '';
> }
>
> sub client_step {
>     my ($self, $challenge) = @_;
>
>     my $stage = ++$self->{stage};
>     if ($stage == 1) {

This bit should be in "client_start{}":
>         # Generate the XOAUTH2 authentication string
>         my $username = $self->_call('user');
>         my $token    = $self->_call('pass'); # OAuth 2.0 access token
>         my $auth_string = "user=$username\001auth=Bearer $token\001\001";
>         my $encoded_auth_string = encode_base64($auth_string, '');
>
>         # Send the encoded authentication string
>         return $encoded_auth_string;
Up until here.

>     } else {
>         # Handle authentication failure by sending a dummy request
>         my $dummy_request = encode_base64("\001", '');
>         return $dummy_request;
>     }
> }
> }
>
> 1;
>
> __END__
>
> =head1 NAME
>
> Authen::SASL::Perl::XOAUTH2 - XOAUTH2 Authentication class
>
> =head1 SYNOPSIS
>
>   use Authen::SASL qw(Perl);
>
>   $sasl = Authen::SASL->new(
>     mechanism => 'XOAUTH2',
>     callback  => {
>       user => $user,
>       pass => $access_token,
>     },
>   );
>
> =head1 DESCRIPTION
>
> This module implements the client side of the XOAUTH2 SASL mechanism, which is used for OAuth 2.0-based authentication.
>
> =head2 CALLBACK
>
> The callbacks used are:
>
> =head3 Client
>
> =over 4
>
> =item user
>
> The username to be used for authentication.
>
> =item pass
>
> The OAuth 2.0 access token to be used for authentication.
>
> =back
>
> =head1 SEE ALSO
>
> L<Authen::SASL>,
> L<MIME::Base64>
>
> =head1 AUTHORS
>
> Written by [Your Name].
>
> =head1 COPYRIGHT
>
> This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
>
> =cut
>
> ----
> >
> > Please let me know who to talk to to have it added!
> >
> >
> > --
> > Bye,
> >
> > Erik.
> >
> > http://efficito.com <http://efficito.com/> -- Hosted accounting and ERP.
> > Robust and Flexible. No vendor lock-in.
>


-- 
Bye,

Erik.

http://efficito.com -- Hosted accounting and ERP.
Robust and Flexible. No vendor lock-in.

Re: [PATCH v5 1/3] send-email: implement SMTP bearer authentication

[Julian Swagemakers]

On Thu Apr 24, 2025 at 8:22 PM CEST, Aditya Garg wrote:
> I saw the code of perl-authen-sasl and I miserably failed in porting
> the logic used here to the repo.
>
> Perl is not something I am very strong at, so not really feasible for
> me to work ahead. I do was able to return the formatted base64 encoded
> string as per the logic, but still I got authentication issues.
> Currently I aim to have git-send-email working, and a review from an
> experienced person is needed.
>
> I would be happy to be able to test any proposed patch though.
>
> Julian, you might be interested here?

I fear I'm also a rookie when it comes to perl, but with your code and
the comment from Erik I managed to get it working. client_start and
client_step have to look like this:

sub client_start {
    my $self = shift;
    $self->{stage} = 0;
    # Generate the XOAUTH2 authentication string
    my $username = $self->_call('user');
    my $token    = $self->_call('pass'); # OAuth 2.0 access token
    my $auth_string = "user=$username\001auth=Bearer $token\001\001";
    return $auth_string
}

sub client_step {
    my ($self, $challenge) = @_;

    my $stage = ++$self->{stage};

    if ($stage == 1) {
        # Handle authentication failure by sending a dummy request
        return "\001"
    } else {
        return $self->set_error("Invalid sequence");
    }
}

Aditya, do you want to create a pull request on perl-authen-sasl, or
should I?

Regards Julian

Re: [PATCH v5 1/3] send-email: implement SMTP bearer authentication

[Aditya Garg]

Hi Julian

> On 25 Apr 2025, at 11:49 AM, Julian Swagemakers <julian@swagemakers.org> wrote:
> 
> On Thu Apr 24, 2025 at 8:22 PM CEST, Aditya Garg wrote:
>> I saw the code of perl-authen-sasl and I miserably failed in porting
>> the logic used here to the repo.
>> 
>> Perl is not something I am very strong at, so not really feasible for
>> me to work ahead. I do was able to return the formatted base64 encoded
>> string as per the logic, but still I got authentication issues.
>> Currently I aim to have git-send-email working, and a review from an
>> experienced person is needed.
>> 
>> I would be happy to be able to test any proposed patch though.
>> 
>> Julian, you might be interested here?
> 
> I fear I'm also a rookie when it comes to perl, but with your code and
> the comment from Erik I managed to get it working. client_start and
> client_step have to look like this:
> 
> sub client_start {
>    my $self = shift;
>    $self->{stage} = 0;
>    # Generate the XOAUTH2 authentication string
>    my $username = $self->_call('user');
>    my $token    = $self->_call('pass'); # OAuth 2.0 access token
>    my $auth_string = "user=$username\001auth=Bearer $token\001\001";
>    return $auth_string
> }
> 
> sub client_step {
>    my ($self, $challenge) = @_;
> 
>    my $stage = ++$self->{stage};
> 
>    if ($stage == 1) {
>        # Handle authentication failure by sending a dummy request
>        return "\001"
>    } else {
>        return $self->set_error("Invalid sequence");
>    }
> }
> 
> Aditya, do you want to create a pull request on perl-authen-sasl, or
> should I?

I'll do some tests on my end then I can open a PR. If I face issues, I'll ping you again!

Thanks
Aditya

Re: [PATCH v5 1/3] send-email: implement SMTP bearer authentication

[Aditya Garg]

Hi Erik, Julian

> Aditya, do you want to create a pull request on perl-authen-sasl, or
> should I?

I've sent a PR here: https://github.com/gbarr/perl-authen-sasl/pull/19

Julian, I have added your and my name to the Copyright here. If you or someone has any objection in any one of this, please let me know.

Cheers!
Aditya

Re: [PATCH v5 1/3] send-email: implement SMTP bearer authentication

[Erik Hulsmann]

Hi Aditya, Julian,


On 25-04-2025 11:45, Aditya Garg wrote:
>> Aditya, do you want to create a pull request on perl-authen-sasl, or
>> should I?
> I've sent a PR here: https://github.com/gbarr/perl-authen-sasl/pull/19
Thanks for the contribution!

I've reviewed the PR and added some improvement suggestions as well as 
some questions. Good to see XOAUTH2 move forward!


Regards,


Erik.

Re: [PATCH v5 1/3] send-email: implement SMTP bearer authentication

[Aditya Garg]

Hi Eric

> As I said in https://github.com/gbarr/perl-authen-sasl/issues/18#issuecomment-2453040190 <https://github.com/gbarr/perl-authen-sasl/issues/18#issuecomment-2453040190> : I'd love to implement XOAUTH2 and/or OAUTHBEARER, but I don't have a setup available to test with, so I need someone to provide a patch *and* a description of how to test, preferably against a publicly available service so I don't need to go through complex setup myself.

I saw the code of perl-authen-sasl and I miserably failed in porting the logic used here to the repo.

Perl is not something I am very strong at, so not really feasible for me to work ahead. I do was able to return the formatted base64 encoded string as per the logic, but still I got authentication issues. Currently I aim to have git-send-email working, and a review from an experienced person is needed.

I would be happy to be able to test any proposed patch though.

Julian, you might be interested here?

If you are interested, you can checkout my XOAUTH2.pm file, which doesn't authenticate for some reason.

-->8--

package Authen::SASL::Perl::XOAUTH2;

use strict;
use warnings;
use MIME::Base64;
use vars qw($VERSION @ISA);

$VERSION = "1.00";
@ISA     = qw(Authen::SASL::Perl);

my %secflags = (
    noanonymous => 1,
);

sub _order { 1 }

sub _secflags {
    shift;
    scalar grep { $secflags{$_} } @_;
}

sub mechanism { 'XOAUTH2' }

sub client_start {
    my $self = shift;
    $self->{stage} = 0;
    '';
}

sub client_step {
    my ($self, $challenge) = @_;

    my $stage = ++$self->{stage};
    if ($stage == 1) {
        # Generate the XOAUTH2 authentication string
        my $username = $self->_call('user');
        my $token    = $self->_call('pass'); # OAuth 2.0 access token
        my $auth_string = "user=$username\001auth=Bearer $token\001\001";
        my $encoded_auth_string = encode_base64($auth_string, '');

        # Send the encoded authentication string
        return $encoded_auth_string;
    } else {
        # Handle authentication failure by sending a dummy request
        my $dummy_request = encode_base64("\001", '');
        return $dummy_request;
    }
}
}

1;

__END__

=head1 NAME

Authen::SASL::Perl::XOAUTH2 - XOAUTH2 Authentication class

=head1 SYNOPSIS

  use Authen::SASL qw(Perl);

  $sasl = Authen::SASL->new(
    mechanism => 'XOAUTH2',
    callback  => {
      user => $user,
      pass => $access_token,
    },
  );

=head1 DESCRIPTION

This module implements the client side of the XOAUTH2 SASL mechanism, which is used for OAuth 2.0-based authentication.

=head2 CALLBACK

The callbacks used are:

=head3 Client

=over 4

=item user

The username to be used for authentication.

=item pass

The OAuth 2.0 access token to be used for authentication.

=back

=head1 SEE ALSO

L<Authen::SASL>,
L<MIME::Base64>

=head1 AUTHORS

Written by [Your Name].

=head1 COPYRIGHT

This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

=cut

----
> 
> Please let me know who to talk to to have it added!
> 
> 
> -- 
> Bye,
> 
> Erik.
> 
> http://efficito.com <http://efficito.com/> -- Hosted accounting and ERP.
> Robust and Flexible. No vendor lock-in.

[PATCH v5 2/3] send-email: retrieve Message-ID from outlook SMTP server

[Aditya Garg]

The script generates a Message-ID alongwith the other headers when
gen_header is called, and is sent alongwith the email. For most email
providers, including gmail, the Message-ID goes unchanged to the
recipient.

But, this does not seem to be a case with Outlook. In Outlook, when we
send our own Message-ID as a part of the headers, it discards it. Then
it generates a new random Message-ID and that is what the recipient
gets.

This is a problem because the Message-ID is crucial when we are sending
multiple emails in a thread. The current implementation for threads in
the script replies to the Message-ID it generated, but due to Outlook's
behavior, it is not the same as the one that the recipient got, thus
breaking threads. So a need arises to retrieve the Message-ID from the
server response and set it in the In-Reply-To and References email
headers instead of using the self generated one for the purpose of
replies.

The $smtp->message variable in this script for outlook is something like
this:

2.0.0 OK <Message-ID> [Hostname=Some-hostname]

The Message-ID here is the one the recipient gets, rather than the one
the script generated.

This patch uses the fact above and retrieves the Message-ID from the
server response. It then changes the value of the $message_id variable
to the one received from the server. This value will be used when next
and subsequent messages are sent as replies to the message, thus
preserving the threading of the messages.

Signed-off-by: Aditya Garg <gargaditya08@live.com>
---
 git-send-email.perl | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/git-send-email.perl b/git-send-email.perl
index 9ba47a6f38..8c8544f120 100755
--- a/git-send-email.perl
+++ b/git-send-email.perl
@@ -1643,6 +1643,11 @@ sub gen_header {
 	return ($recipients_ref, $to, $date, $gitversion, $cc, $ccline, $header);
 }
 
+sub is_outlook {
+	my ($host) = @_;
+	return ($host eq 'smtp.office365.com' || $host eq 'smtp-mail.outlook.com');
+}
+
 # Prepares the email, then asks the user what to do.
 #
 # If the user chooses to send the email, it's sent and 1 is returned.
@@ -1806,6 +1811,22 @@ sub send_message {
 			$smtp->datasend("$line") or die $smtp->message;
 		}
 		$smtp->dataend() or die $smtp->message;
+
+		# Outlook discards the Message-ID header we set while sending the email
+		# and generates a new random Message-ID. So in order to avoid breaking
+		# threads, we simply retrieve the Message-ID from the server response
+		# and assign it to the $message_id variable, which will then be
+		# assigned to $in_reply_to by the caller when the next message is sent
+		# as a response to this message.
+		if (is_outlook($smtp_server)) {
+			if ($smtp->message =~ /<([^>]+)>/) {
+				$message_id = "<$1>";
+				printf __("Outlook reassigned Message-ID to: %s\n"), $message_id;
+			} else {
+				warn __("Warning: Could not retrieve Message-ID from server response.\n");
+			}
+		}
+
 		$smtp->code =~ /250|200/ or die sprintf(__("Failed to send %s\n"), $subject).$smtp->message;
 	}
 	if ($quiet) {
-- 
2.49.0

Re: [PATCH v5 2/3] send-email: retrieve Message-ID from outlook SMTP server

[Greg Kroah-Hartman]

On Thu, Apr 24, 2025 at 07:53:54AM +0000, Aditya Garg wrote:
> The script generates a Message-ID alongwith the other headers when
> gen_header is called, and is sent alongwith the email. For most email
> providers, including gmail, the Message-ID goes unchanged to the
> recipient.
> 
> But, this does not seem to be a case with Outlook. In Outlook, when we
> send our own Message-ID as a part of the headers, it discards it. Then
> it generates a new random Message-ID and that is what the recipient
> gets.
> 
> This is a problem because the Message-ID is crucial when we are sending
> multiple emails in a thread. The current implementation for threads in
> the script replies to the Message-ID it generated, but due to Outlook's
> behavior, it is not the same as the one that the recipient got, thus
> breaking threads. So a need arises to retrieve the Message-ID from the
> server response and set it in the In-Reply-To and References email
> headers instead of using the self generated one for the purpose of
> replies.
> 
> The $smtp->message variable in this script for outlook is something like
> this:
> 
> 2.0.0 OK <Message-ID> [Hostname=Some-hostname]
> 
> The Message-ID here is the one the recipient gets, rather than the one
> the script generated.
> 
> This patch uses the fact above and retrieves the Message-ID from the
> server response. It then changes the value of the $message_id variable
> to the one received from the server. This value will be used when next
> and subsequent messages are sent as replies to the message, thus
> preserving the threading of the messages.
> 
> Signed-off-by: Aditya Garg <gargaditya08@live.com>
> ---
>  git-send-email.perl | 21 +++++++++++++++++++++
>  1 file changed, 21 insertions(+)
> 
> diff --git a/git-send-email.perl b/git-send-email.perl
> index 9ba47a6f38..8c8544f120 100755
> --- a/git-send-email.perl
> +++ b/git-send-email.perl
> @@ -1643,6 +1643,11 @@ sub gen_header {
>  	return ($recipients_ref, $to, $date, $gitversion, $cc, $ccline, $header);
>  }
>  
> +sub is_outlook {
> +	my ($host) = @_;
> +	return ($host eq 'smtp.office365.com' || $host eq 'smtp-mail.outlook.com');
> +}

No real objection here, but what about all of the company-hosted outlook
server systems out there?  Do they need this same type of "flag"?  And
if so, why not make it a config variable?

thanks,

greg k-h

Re: [PATCH v5 2/3] send-email: retrieve Message-ID from outlook SMTP server

[Yao Zi]

On Thu, Apr 24, 2025 at 03:09:20PM +0200, Greg Kroah-Hartman wrote:
> On Thu, Apr 24, 2025 at 07:53:54AM +0000, Aditya Garg wrote:
> > The script generates a Message-ID alongwith the other headers when
> > gen_header is called, and is sent alongwith the email. For most email
> > providers, including gmail, the Message-ID goes unchanged to the
> > recipient.
> > 
> > But, this does not seem to be a case with Outlook. In Outlook, when we
> > send our own Message-ID as a part of the headers, it discards it. Then
> > it generates a new random Message-ID and that is what the recipient
> > gets.
> > 
> > This is a problem because the Message-ID is crucial when we are sending
> > multiple emails in a thread. The current implementation for threads in
> > the script replies to the Message-ID it generated, but due to Outlook's
> > behavior, it is not the same as the one that the recipient got, thus
> > breaking threads. So a need arises to retrieve the Message-ID from the
> > server response and set it in the In-Reply-To and References email
> > headers instead of using the self generated one for the purpose of
> > replies.
> > 
> > The $smtp->message variable in this script for outlook is something like
> > this:
> > 
> > 2.0.0 OK <Message-ID> [Hostname=Some-hostname]
> > 
> > The Message-ID here is the one the recipient gets, rather than the one
> > the script generated.
> > 
> > This patch uses the fact above and retrieves the Message-ID from the
> > server response. It then changes the value of the $message_id variable
> > to the one received from the server. This value will be used when next
> > and subsequent messages are sent as replies to the message, thus
> > preserving the threading of the messages.
> > 
> > Signed-off-by: Aditya Garg <gargaditya08@live.com>
> > ---
> >  git-send-email.perl | 21 +++++++++++++++++++++
> >  1 file changed, 21 insertions(+)
> > 
> > diff --git a/git-send-email.perl b/git-send-email.perl
> > index 9ba47a6f38..8c8544f120 100755
> > --- a/git-send-email.perl
> > +++ b/git-send-email.perl
> > @@ -1643,6 +1643,11 @@ sub gen_header {
> >  	return ($recipients_ref, $to, $date, $gitversion, $cc, $ccline, $header);
> >  }
> >  
> > +sub is_outlook {
> > +	my ($host) = @_;
> > +	return ($host eq 'smtp.office365.com' || $host eq 'smtp-mail.outlook.com');
> > +}
> 
> No real objection here, but what about all of the company-hosted outlook
> server systems out there?  Do they need this same type of "flag"?  And
> if so, why not make it a config variable?

Not only Outlook comes with such quirk, AFAIK the mail service that
Tencent provides for personal usage does as well. I don't think it's a
good idea to hardcode the problematic providers.

Not sure whether similar ideas have been proposed earlier: since this
quirk affects only following e-mails but not the coverletter which
doesn't have a In-reply-to field, is it possible to detect the quirk
with the response of sending the coverletter by comparing the desired
Message-ID and the one in response? We could throw a warning and
automatically fixes following mails if the bad case really happens.

This could avoid a broken thread for newcomers and should play well with
an option introduced together for specifying dedicated behaviour.

> thanks,
> 
> greg k-h

Please Cc me on future updates of the series, thank you Aditya.

Best regards,
Yao Zi

Re: [PATCH v5 2/3] send-email: retrieve Message-ID from outlook SMTP server

[Aditya Garg]

> Not only Outlook comes with such quirk, AFAIK the mail service that
> Tencent provides for personal usage does as well. I don't think it's a
> good idea to hardcode the problematic providers.
> 
> Not sure whether similar ideas have been proposed earlier: since this
> quirk affects only following e-mails but not the coverletter which
> doesn't have a In-reply-to field, is it possible to detect the quirk
> with the response of sending the coverletter by comparing the desired
> Message-ID and the one in response? We could throw a warning and
> automatically fixes following mails if the bad case really happens.

From what I understand, what you want here is, that

1. Irrespective of the email provider, I use the logic I am using for each mail.
2. I extract the message-id, and compare it with the intended one.
3. Give a warning and fix it.

But, no. Outlook luckily shows the message ID in its response, and is there in
$smtp->message. In fact you can see the whole server response here:

https://github.com/marlam/msmtp/issues/190#issuecomment-2794784869

Now say I use this logic with gmail. The $smtp->message in gmail is similar to
outlook, but has 2 main differences:

1. The angular brackets <> are missing.
2. There is no message id! Rather it has a random string of numbers and letters
   that I also am not aware of what they mean.

So, different providers have different ways, to respond.

What does Tencent do? Have you tried to log the SMTP messages?

In any case I don't find any way to automatically determine this. And I certainly
don't think we can poke into receiver's email to see what message id they got.

I also don't know how corporates work. Is the SMTP server for them the same ultimately?
Or are they using some Azure server?

Tbh Microsoft does not like following standards with Outlook. It doesn't even support
OAUTHBEARER which is supposed to be the standard, rather uses Google's XOAUTH2. So why
not hardcode? Do you have any other solution?> 
> This could avoid a broken thread for newcomers and should play well with
> an option introduced together for specifying dedicated behaviour.
> 
>> thanks,
>>
>> greg k-h
> 
> Please Cc me on future updates of the series, thank you Aditya.
> 
> Best regards,
> Yao Zi

Re: [PATCH v5 2/3] send-email: retrieve Message-ID from outlook SMTP server

[Yao Zi]

On Mon, Apr 28, 2025 at 01:35:32AM +0530, Aditya Garg wrote:
> > Not only Outlook comes with such quirk, AFAIK the mail service that
> > Tencent provides for personal usage does as well. I don't think it's a
> > good idea to hardcode the problematic providers.
> > 
> > Not sure whether similar ideas have been proposed earlier: since this
> > quirk affects only following e-mails but not the coverletter which
> > doesn't have a In-reply-to field, is it possible to detect the quirk
> > with the response of sending the coverletter by comparing the desired
> > Message-ID and the one in response? We could throw a warning and
> > automatically fixes following mails if the bad case really happens.
> 
> From what I understand, what you want here is, that
> 
> 1. Irrespective of the email provider, I use the logic I am using for each mail.
> 2. I extract the message-id, and compare it with the intended one.
> 3. Give a warning and fix it.

Yes.

> But, no. Outlook luckily shows the message ID in its response, and is there in
> $smtp->message. In fact you can see the whole server response here:
> 
> https://github.com/marlam/msmtp/issues/190#issuecomment-2794784869
> 
> Now say I use this logic with gmail. The $smtp->message in gmail is similar to
> outlook, but has 2 main differences:
> 
> 1. The angular brackets <> are missing.
> 2. There is no message id! Rather it has a random string of numbers and letters
>    that I also am not aware of what they mean.
> 
> So, different providers have different ways, to respond.

Thanks for explaining this, it's really not that simple as my first
glance. With the explanation, hardcoding the provider seems pretty
reasonable to me.

> What does Tencent do? Have you tried to log the SMTP messages?

Sadly no. I don't really use their services but just have seen guys hit
by the same problem.

> In any case I don't find any way to automatically determine this. And I certainly
> don't think we can poke into receiver's email to see what message id they got.
> 
> I also don't know how corporates work. Is the SMTP server for them the same ultimately?
> Or are they using some Azure server?
> 
> Tbh Microsoft does not like following standards with Outlook. It doesn't even support
> OAUTHBEARER which is supposed to be the standard, rather uses Google's XOAUTH2. So why
> not hardcode? Do you have any other solution?> 
> > This could avoid a broken thread for newcomers and should play well with
> > an option introduced together for specifying dedicated behaviour.
> > 
> >> thanks,
> >>
> >> greg k-h
> > 
> > Please Cc me on future updates of the series, thank you Aditya.
> > 
> > Best regards,
> > Yao Zi
> 

Thanks for the fix,
Yao Zi

Re: [PATCH v5 2/3] send-email: retrieve Message-ID from outlook SMTP server

[Aditya Garg]

Hi Greg

Sorry for the late reply. For some reason this mail didn't even reach my inbox :(. 
> 
> No real objection here, but what about all of the company-hosted outlook
> server systems out there?  Do they need this same type of "flag"?  And
> if so, why not make it a config variable?

I have proposed a patch for the same here:

https://lore.kernel.org/git/PN3PR01MB95973B932F4961FFFA9786CBB8872@PN3PR01MB9597.INDPRD01.PROD.OUTLOOK.COM/

Thanks
Aditya

[PATCH v5 3/3] send-email: add option to generate passswords like OAuth2 tokens

[Aditya Garg]

Some email providers like outlook allow only OAuth2 tokens to be used
for authentication. This commit adds an option to generate OAuth2 tokens
using scripts like M365-IMAP[1]. This option is similar to passwordeval
in msmtp.

Example usage:

[sendemail]
    smtpEncryption = tls
    smtpServer = smtp.office365.com
    smtpUser = someone@outlook.com
    smtpServerPort = 587
    smtpauth = XOAUTH2
    smtpPassEval = cd /workspaces/codespaces-blank/M365-IMAP && python3 ./refresh_token.py

Signed-off-by: Aditya Garg <gargaditya08@live.com>
---
 Documentation/git-send-email.adoc |  8 ++++++++
 git-send-email.perl               | 15 +++++++++++++++
 2 files changed, 23 insertions(+)

diff --git a/Documentation/git-send-email.adoc b/Documentation/git-send-email.adoc
index 1bf75c060d..f478559582 100644
--- a/Documentation/git-send-email.adoc
+++ b/Documentation/git-send-email.adoc
@@ -230,6 +230,14 @@ or on the command line. If a username has been specified (with
 specified (with `--smtp-pass` or `sendemail.smtpPass`), then
 a password is obtained using 'git-credential'.
 
+--smtp-passeval[=<command>]::
+	Generate password like OAuth2 token for SMTP AUTH. If specified,
+	it will use the output of the command specified as a password for
+	authentication.
++
+Note that it will override any existing password specified using
+`--smtp-pass` or a `sendemail.smtpPass`.
+
 --no-smtp-auth::
 	Disable SMTP authentication. Short hand for `--smtp-auth=none`
 
diff --git a/git-send-email.perl b/git-send-email.perl
index 8c8544f120..d34797f852 100755
--- a/git-send-email.perl
+++ b/git-send-email.perl
@@ -59,6 +59,8 @@ sub usage {
     --smtp-server-port      <int>  * Outgoing SMTP server port.
     --smtp-user             <str>  * Username for SMTP-AUTH.
     --smtp-pass             <str>  * Password for SMTP-AUTH; not necessary.
+    --smtp-passeval         <str>  * Path to script or a command to generate
+                                     password like OAuth2 token for SMTP-AUTH.
     --smtp-encryption       <str>  * tls or ssl; anything else disables.
     --smtp-ssl                     * Deprecated. Use '--smtp-encryption ssl'.
     --smtp-ssl-cert-path    <str>  * Path to ca-certificates (either directory or file).
@@ -280,6 +282,7 @@ sub do_edit {
 my ($auto_8bit_encoding);
 my ($compose_encoding);
 my ($sendmail_cmd);
+my ($smtp_authpasseval);
 my ($mailmap_file, $mailmap_blob);
 # Variables with corresponding config settings & hardcoded defaults
 my ($debug_net_smtp) = 0;		# Net::SMTP, see send_message()
@@ -316,6 +319,7 @@ sub do_edit {
     "smtppass" => \$smtp_authpass,
     "smtpdomain" => \$smtp_domain,
     "smtpauth" => \$smtp_auth,
+    "smtppasseval" => \$smtp_authpasseval,
     "smtpbatchsize" => \$batch_size,
     "smtprelogindelay" => \$relogin_delay,
     "to" => \@config_to,
@@ -516,6 +520,7 @@ sub config_regexp {
 		    "smtp-server-port=s" => \$smtp_server_port,
 		    "smtp-user=s" => \$smtp_authuser,
 		    "smtp-pass:s" => \$smtp_authpass,
+		    "smtp-passeval=s" => \$smtp_authpasseval,
 		    "smtp-ssl" => sub { $smtp_encryption = 'ssl' },
 		    "smtp-encryption=s" => \$smtp_encryption,
 		    "smtp-ssl-cert-path=s" => \$smtp_ssl_cert_path,
@@ -1470,6 +1475,16 @@ sub smtp_auth_maybe {
 		return 1;
 	}
 
+	# If smtpPassEval is set, run the user specified command to get the password
+	if (defined $smtp_authpasseval) {
+		printf __("Executing token generating script: %s\n"), $smtp_authpasseval;
+		chomp(my $generated_password = `$smtp_authpasseval 2>&1`);
+		if ($? != 0) {
+			die sprintf(__("Failed to execute token generating script: %s\n"), $smtp_authpasseval);
+		}
+		$smtp_authpass = $generated_password;
+	}
+
 	# Workaround AUTH PLAIN/LOGIN interaction defect
 	# with Authen::SASL::Cyrus
 	eval {
-- 
2.49.0

Re: [PATCH v5 3/3] send-email: add option to generate passswords like OAuth2 tokens

[Julian Swagemakers]

On Thu Apr 24, 2025 at 9:53 AM CEST, Aditya Garg wrote:
> Some email providers like outlook allow only OAuth2 tokens to be used
> for authentication. This commit adds an option to generate OAuth2 tokens
> using scripts like M365-IMAP[1]. This option is similar to passwordeval
> in msmtp.
>
> Example usage:
>
> [sendemail]
>     smtpEncryption = tls
>     smtpServer = smtp.office365.com
>     smtpUser = someone@outlook.com
>     smtpServerPort = 587
>     smtpauth = XOAUTH2
>     smtpPassEval = cd /workspaces/codespaces-blank/M365-IMAP && python3 ./refresh_token.py
>

Having the option `smtpPassEval` is a little more intuitive, but can't
this also be achieved using a custom credential helper[0]?

Something like:

    [credential "smtp://smtp.office365.com:587"]
        username = someone@outlook.com
        helper = "!f() { test \"$1\" = get && echo \"password=$(cd /workspaces/codespaces-blank/M365-IMAP && python3 ./refresh_token.py)\"; }; f"

Regards Julian

[0] https://git-scm.com/docs/gitcredentials.html#_custom_helpers

Re: [PATCH v5 3/3] send-email: add option to generate passswords like OAuth2 tokens

[Aditya Garg]

> Something like:
> 
>     [credential "smtp://smtp.office365.com:587"]
>         username = someone@outlook.com
>         helper = "!f() { test \"$1\" = get && echo \"password=$(cd /workspaces/codespaces-blank/M365-IMAP && python3 ./refresh_token.py)\"; }; f"

Interesting, and this works too!. I wasn't aware of this.

Junio, I can drop the third patch if you want.

Re: [PATCH v5 3/3] send-email: add option to generate passswords like OAuth2 tokens

[Junio C Hamano]

Aditya Garg <gargaditya08@live.com> writes:

>> Something like:
>> 
>>     [credential "smtp://smtp.office365.com:587"]
>>         username = someone@outlook.com
>>         helper = "!f() { test \"$1\" = get && echo \"password=$(cd /workspaces/codespaces-blank/M365-IMAP && python3 ./refresh_token.py)\"; }; f"
>
> Interesting, and this works too!. I wasn't aware of this.
>
> Junio, I can drop the third patch if you want.

What I want does not matter in this case, no? ;-)

It does look like an additional mechanism only for this use case is
unneeded, but it would be nice to address a related but different
problem, which is

    What made you write the extra mechanism in the first place?

In other words, was the current documentation insufficient for you
to realize that credential helper is an existing good solution for
your problem and there was no need to add a new mechanism?  If so,
there will be numerous next "you" who will also want to use OAuth2
token from a program output and get frustrated because they cannot
find how to do so in our documentation, no?  Would a new paragraph
with an example like Julian gave above be something we want to add,
or do we already have enough information there and the only thing
users need to do is to look a bit more carefully?

Thanks.

Re: [PATCH v5 3/3] send-email: add option to generate passswords like OAuth2 tokens

[Aditya Garg]

On 24-04-2025 08:50 pm, Junio C Hamano wrote:
> Aditya Garg <gargaditya08@live.com> writes:
> 
>>> Something like:
>>>
>>>     [credential "smtp://smtp.office365.com:587"]
>>>         username = someone@outlook.com
>>>         helper = "!f() { test \"$1\" = get && echo \"password=$(cd /workspaces/codespaces-blank/M365-IMAP && python3 ./refresh_token.py)\"; }; f"
>>
>> Interesting, and this works too!. I wasn't aware of this.
>>
>> Junio, I can drop the third patch if you want.
> 
> What I want does not matter in this case, no? ;-)
> 
> It does look like an additional mechanism only for this use case is
> unneeded, but it would be nice to address a related but different
> problem, which is
> 
>     What made you write the extra mechanism in the first place?
> 
> In other words, was the current documentation insufficient for you
> to realize that credential helper is an existing good solution for
> your problem and there was no need to add a new mechanism?  If so,
> there will be numerous next "you" who will also want to use OAuth2
> token from a program output and get frustrated because they cannot
> find how to do so in our documentation, no?  Would a new paragraph
> with an example like Julian gave above be something we want to add,
> or do we already have enough information there and the only thing
> users need to do is to look a bit more carefully?

That's actually a good question. In my case, I already was frustrated
with no good option available for Outlook users to send patches reliably
to the Linux kernel. I had been banging my head and Googling a lot to
get a way. I finally reached msmtp, but then the threading broke, giving
rise to another issue. After chatting with the msmtp dev, I came to know
that its the server side issue with outlook. At this point, I was not left
with any more power to read more docs, and decided to modify the script myself.

The first hurdle was OAuth2, which I searched the mailing list with hopes, and
got the first patch, which worked beautifully. Now the only issues left were
threads and a reliable way to get OAuth2 tokens. I already had a helper script
to get the token, but I gave https://github.com/git-ecosystem/git-credential-manager
a shot. Unfortunately, it didn't work with free outlook accounts, so I was left
with my helper script. With this failure with git credential manager, I was in
an impression that its not possible to use [credential]. My bad here for not
checking the docs for this.

So I decided to take things in my own hands and patch the script. Logged a bit
and fixed the threads. Then I was inspired by the passwordeval function I used
in msmtp, and just implemented it in the third patch.



TL;DR: I should have looked at the docs of credentials as well. I kept stuck
on docs on send-mail.



My suggestion: Maybe add a small link to relevant docs in other docs as well.
Eg, in the description of SmtpPass, you can add a suggestion to use [credentials]
if you want to generate a token or something and link its doc.

Re: [PATCH v5 3/3] send-email: add option to generate passswords like OAuth2 tokens

[Junio C Hamano]

Aditya Garg <gargaditya08@live.com> writes:

> TL;DR: I should have looked at the docs of credentials as well. I kept stuck
> on docs on send-mail.

Meaning the documentation on send-email is lacking, which is a good
conclusion to draw from this episode, and have somebody (it does not
have to be you) to look into.

> My suggestion: Maybe add a small link to relevant docs in other docs as well.
> Eg, in the description of SmtpPass, you can add a suggestion to use [credentials]
> if you want to generate a token or something and link its doc.

Yes, wonderful.

Thanks for your suggestion.

[PATCH v6 0/1] send-email: add oauth2 support and fix outlook breaking threads

[Aditya Garg]

Hi all!

This patch series includes three changes:

1. It adds support for Oauth2 authentication, which is now compulsory by Microsoft.
   This patch has been rebased to the latest version from the original version
   at https://lore.kernel.org/git/20250125190131.48717-1-julian@swagemakers.org/

2. The second patch makes the script reply to the message id set by the outlook,
   since outlook has its own proprietary way to handle message ids,
   and does not allow user to set their own. As a result, threads were breaking.

3. The final patch adds a new option to generate passwords like OAuth2 tokens.
   This is useful for users who want to use a script which generates tokens for
   OAuth2 authentication.

Detailed description of each patch has been done in the respective patches

BTW, I am sending this series using the patched send-email by these patches from
Outlook!

v2:
- Fix errors flagged by the CI

v3:
- Add third patch to generate passwords like OAuth2 tokens

v4:
- Make log message of the second patch more clear.
- Change "Outlook: Retrieved Message-ID:" to "Outlook reassigned Message-ID to:"
- Update documentation for smtp-passeval.

v5:
- Fix minor grammar issues in the commit messages.
- Attempt to wrap code in 80 characters in 1st patch.
- Create additional sub to check whether we are using OAuth2 authentication

v6:
- Remove smtp bearer patch since Authen::SASL module can be patched
- Remove SmtpPassEval patch since a similar mechanism exists in [credential]

Aditya Garg (1):
  send-email: retrieve Message-ID from outlook SMTP server

 git-send-email.perl | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

-- 
2.49.0

[PATCH v6 1/1] send-email: retrieve Message-ID from outlook SMTP server

[Aditya Garg]

The script generates a Message-ID alongwith the other headers when
gen_header is called, and is sent alongwith the email. For most email
providers, including gmail, the Message-ID goes unchanged to the
recipient.

But, this does not seem to be a case with Outlook. In Outlook, when we
send our own Message-ID as a part of the headers, it discards it. Then
it generates a new random Message-ID and that is what the recipient
gets.

This is a problem because the Message-ID is crucial when we are sending
multiple emails in a thread. The current implementation for threads in
the script replies to the Message-ID it generated, but due to Outlook's
behavior, it is not the same as the one that the recipient got, thus
breaking threads. So a need arises to retrieve the Message-ID from the
server response and set it in the In-Reply-To and References email
headers instead of using the self generated one for the purpose of
replies.

The $smtp->message variable in this script for outlook is something like
this:

2.0.0 OK <Message-ID> [Hostname=Some-hostname]

The Message-ID here is the one the recipient gets, rather than the one
the script generated.

This patch uses the fact above and retrieves the Message-ID from the
server response. It then changes the value of the $message_id variable
to the one received from the server. This value will be used when next
and subsequent messages are sent as replies to the message, thus
preserving the threading of the messages.

Signed-off-by: Aditya Garg <gargaditya08@live.com>
---
 git-send-email.perl | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/git-send-email.perl b/git-send-email.perl
index 1f613fa979..618474916e 100755
--- a/git-send-email.perl
+++ b/git-send-email.perl
@@ -1574,6 +1574,11 @@ sub gen_header {
 	return ($recipients_ref, $to, $date, $gitversion, $cc, $ccline, $header);
 }
 
+sub is_outlook {
+	my ($host) = @_;
+	return ($host eq 'smtp.office365.com' || $host eq 'smtp-mail.outlook.com');
+}
+
 # Prepares the email, then asks the user what to do.
 #
 # If the user chooses to send the email, it's sent and 1 is returned.
@@ -1737,6 +1742,22 @@ sub send_message {
 			$smtp->datasend("$line") or die $smtp->message;
 		}
 		$smtp->dataend() or die $smtp->message;
+
+		# Outlook discards the Message-ID header we set while sending the email
+		# and generates a new random Message-ID. So in order to avoid breaking
+		# threads, we simply retrieve the Message-ID from the server response
+		# and assign it to the $message_id variable, which will then be
+		# assigned to $in_reply_to by the caller when the next message is sent
+		# as a response to this message.
+		if (is_outlook($smtp_server)) {
+			if ($smtp->message =~ /<([^>]+)>/) {
+				$message_id = "<$1>";
+				printf __("Outlook reassigned Message-ID to: %s\n"), $message_id;
+			} else {
+				warn __("Warning: Could not retrieve Message-ID from server response.\n");
+			}
+		}
+
 		$smtp->code =~ /250|200/ or die sprintf(__("Failed to send %s\n"), $subject).$smtp->message;
 	}
 	if ($quiet) {
-- 
2.49.0

Re: [PATCH v6 1/1] send-email: retrieve Message-ID from outlook SMTP server

[Aditya Garg]

Hi Junio

> The script generates a Message-ID alongwith the other headers when
> gen_header is called, and is sent alongwith the email. For most email
> providers, including gmail, the Message-ID goes unchanged to the
> recipient.
> 
> But, this does not seem to be a case with Outlook. In Outlook, when we
> send our own Message-ID as a part of the headers, it discards it. Then
> it generates a new random Message-ID and that is what the recipient
> gets.
> 
> This is a problem because the Message-ID is crucial when we are sending
> multiple emails in a thread. The current implementation for threads in
> the script replies to the Message-ID it generated, but due to Outlook's
> behavior, it is not the same as the one that the recipient got, thus
> breaking threads. So a need arises to retrieve the Message-ID from the
> server response and set it in the In-Reply-To and References email
> headers instead of using the self generated one for the purpose of
> replies.
> 
> The $smtp->message variable in this script for outlook is something like
> this:
> 
> 2.0.0 OK <Message-ID> [Hostname=Some-hostname]
> 
> The Message-ID here is the one the recipient gets, rather than the one
> the script generated.
> 
> This patch uses the fact above and retrieves the Message-ID from the
> server response. It then changes the value of the $message_id variable
> to the one received from the server. This value will be used when next
> and subsequent messages are sent as replies to the message, thus
> preserving the threading of the messages.
> 
> Signed-off-by: Aditya Garg <gargaditya08@live.com>
> ---

Authen::SASL now finally supports XOAUTH2 and OAUTHBEARER thanks to Erik
and Julian! (Link: https://github.com/gbarr/perl-authen-sasl/commit/958a3aa165d30cf4e3cbb36dc45306de627aa13f)

Now the only really needed patch is this, i.e., the v6 with a single patch!

Thanks
Aditya

Re: [PATCH v6 1/1] send-email: retrieve Message-ID from outlook SMTP server

[Erik Huelsmann]

Hi Junio, Aditya,

>
> Authen::SASL now finally supports XOAUTH2 and OAUTHBEARER thanks to Erik
> and Julian! (Link: https://github.com/gbarr/perl-authen-sasl/commit/958a3aa165d30cf4e3cbb36dc45306de627aa13f)

And it's official: https://metacpan.org/release/EHUELS/Authen-SASL-2.1800


-- 
Bye,

Erik.

http://efficito.com -- Hosted accounting and ERP.
Robust and Flexible. No vendor lock-in.

Re: [PATCH v6 1/1] send-email: retrieve Message-ID from outlook SMTP server

[Junio C Hamano]

Erik Huelsmann <ehuels@gmail.com> writes:

> Hi Junio, Aditya,
>
>>
>> Authen::SASL now finally supports XOAUTH2 and OAUTHBEARER thanks to Erik
>> and Julian! (Link: https://github.com/gbarr/perl-authen-sasl/commit/958a3aa165d30cf4e3cbb36dc45306de627aa13f)
>
> And it's official: https://metacpan.org/release/EHUELS/Authen-SASL-2.1800

Wonderful.

We tend to, however, try to cater to those whose distros are slow to
adjust to upstream changes.  What's the ETA for the updated module
to major distros?

Thanks.

Re: [PATCH v6 1/1] send-email: retrieve Message-ID from outlook SMTP server

[Erik Huelsmann]

On Fri, Apr 25, 2025 at 7:08 PM Junio C Hamano <gitster@pobox.com> wrote:

> >> Authen::SASL now finally supports XOAUTH2 and OAUTHBEARER thanks to Erik
> >> and Julian! (Link: https://github.com/gbarr/perl-authen-sasl/commit/958a3aa165d30cf4e3cbb36dc45306de627aa13f)
> >
> > And it's official: https://metacpan.org/release/EHUELS/Authen-SASL-2.1800
>
> Wonderful.
>
> We tend to, however, try to cater to those whose distros are slow to
> adjust to upstream changes.  What's the ETA for the updated module
> to major distros?

To be honest, I have *no* idea. I think Debian is stabilizing Trixie
now, so maybe it's in the one that will be after that (in 2 years?).

You could however decide to support XOAUTH2 and OAUTHBEARER only when
they are available? Then you don't need to increase the minimum
library requirement: there's no API difference between 2.1700 (the
current version until today) and 2.1800 (the newly released version).
So if you were to probe existence of Authen::SASL::Perl::XOAUTH2
and/or Authen::SASL::Perl::OAUTHBEARER, you could conditionally
disable the feature if the probe fails. (Using "eval { require
Authen::SASL::Perl::XOAUTH2; 1 }" should do what you need: return
false if the probe fails; true if it succeeds.)

-- 
Bye,

Erik.

http://efficito.com -- Hosted accounting and ERP.
Robust and Flexible. No vendor lock-in.

Re: [PATCH v6 1/1] send-email: retrieve Message-ID from outlook SMTP server

[Aditya Garg]

On 26-04-2025 12:35 am, Erik Huelsmann wrote:
> On Fri, Apr 25, 2025 at 7:08 PM Junio C Hamano <gitster@pobox.com> wrote:
> 
>>>> Authen::SASL now finally supports XOAUTH2 and OAUTHBEARER thanks to Erik
>>>> and Julian! (Link: https://github.com/gbarr/perl-authen-sasl/commit/958a3aa165d30cf4e3cbb36dc45306de627aa13f)
>>>
>>> And it's official: https://metacpan.org/release/EHUELS/Authen-SASL-2.1800
>>
>> Wonderful.
>>
>> We tend to, however, try to cater to those whose distros are slow to
>> adjust to upstream changes.  What's the ETA for the updated module
>> to major distros?
> 
> To be honest, I have *no* idea. I think Debian is stabilizing Trixie
> now, so maybe it's in the one that will be after that (in 2 years?).
> 
> You could however decide to support XOAUTH2 and OAUTHBEARER only when
> they are available? Then you don't need to increase the minimum
> library requirement: there's no API difference between 2.1700 (the
> current version until today) and 2.1800 (the newly released version).
> So if you were to probe existence of Authen::SASL::Perl::XOAUTH2
> and/or Authen::SASL::Perl::OAUTHBEARER, you could conditionally
> disable the feature if the probe fails. (Using "eval { require
> Authen::SASL::Perl::XOAUTH2; 1 }" should do what you need: return
> false if the probe fails; true if it succeeds.)

Even if we modify the send-email script, distros slow to adjust
will also not update this so soon :). Its more of a wait and watch thing tbh.

Re: [PATCH v6 1/1] send-email: retrieve Message-ID from outlook SMTP server

[Junio C Hamano]

Aditya Garg <gargaditya08@live.com> writes:

> +sub is_outlook {
> +	my ($host) = @_;
> +	return ($host eq 'smtp.office365.com' || $host eq 'smtp-mail.outlook.com');
> +}

There were a few messages that raised concerns with respect to
on-prem installations of Outlook based servers, where the hostname
cannot be used to tell if we need this message-id tweaking.

The following is a completely untested patch, but it should be
sufficient to illustrate how simple it would be to support an
option to do so, if we cared about the issue enough.

Will queue your patch _without_ this tweak, at least for now.

Thanks.

 git-send-email.perl | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git c/git-send-email.perl w/git-send-email.perl
index 618474916e..dff3d861e4 100755
--- c/git-send-email.perl
+++ w/git-send-email.perl
@@ -60,6 +60,7 @@ sub usage {
     --smtp-user             <str>  * Username for SMTP-AUTH.
     --smtp-pass             <str>  * Password for SMTP-AUTH; not necessary.
     --smtp-encryption       <str>  * tls or ssl; anything else disables.
+    --smtp-outlook-id-tweak <0|1>  * The server munges Message-ID.
     --smtp-ssl                     * Deprecated. Use '--smtp-encryption ssl'.
     --smtp-ssl-cert-path    <str>  * Path to ca-certificates (either directory or file).
                                      Pass an empty string to disable certificate
@@ -290,6 +291,7 @@ sub do_edit {
 my $mailmap = 0;
 my $target_xfer_encoding = 'auto';
 my $forbid_sendmail_variables = 1;
+my $outlook_id_tweak = -1; # we need to tell --no-opt and lack of it
 
 my %config_bool_settings = (
     "thread" => \$thread,
@@ -305,6 +307,7 @@ sub do_edit {
     "xmailer" => \$use_xmailer,
     "forbidsendmailvariables" => \$forbid_sendmail_variables,
     "mailmap" => \$mailmap,
+    "outlookidtweak" => \$outlook_id_tweak,
 );
 
 my %config_settings = (
@@ -518,6 +521,7 @@ sub config_regexp {
 		    "smtp-pass:s" => \$smtp_authpass,
 		    "smtp-ssl" => sub { $smtp_encryption = 'ssl' },
 		    "smtp-encryption=s" => \$smtp_encryption,
+		    "smtp-outlook-id-tweak!" => \$outlook_id_tweak,
 		    "smtp-ssl-cert-path=s" => \$smtp_ssl_cert_path,
 		    "smtp-debug:i" => \$debug_net_smtp,
 		    "smtp-domain:s" => \$smtp_domain,
@@ -1576,7 +1580,13 @@ sub gen_header {
 
 sub is_outlook {
 	my ($host) = @_;
-	return ($host eq 'smtp.office365.com' || $host eq 'smtp-mail.outlook.com');
+
+	if ($outlook_id_tweak < 0) {
+		$outlook_id_tweak = 
+		    ($host eq 'smtp.office365.com' ||
+		     $host eq 'smtp-mail.outlook.com') ? 1 : 0;
+	}
+	return $outlook_id_tweak;
 }
 
 # Prepares the email, then asks the user what to do.

Re: [PATCH v6 1/1] send-email: retrieve Message-ID from outlook SMTP server

[Aditya Garg]

On 25-04-2025 10:53 pm, Junio C Hamano wrote:
> Aditya Garg <gargaditya08@live.com> writes:
> 
>> +sub is_outlook {
>> +	my ($host) = @_;
>> +	return ($host eq 'smtp.office365.com' || $host eq 'smtp-mail.outlook.com');
>> +}
> 
> There were a few messages that raised concerns with respect to
> on-prem installations of Outlook based servers, where the hostname
> cannot be used to tell if we need this message-id tweaking.
> 
> The following is a completely untested patch, but it should be
> sufficient to illustrate how simple it would be to support an
> option to do so, if we cared about the issue enough.
> 
> Will queue your patch _without_ this tweak, at least for now.

If you really want to make it configurable, then an auto mode should also be there.
In that mode, it uses my logic. I just prefer users from having more out of the
box experience. By the auto mode, we can both configure it, as well as have an
out of the box experience.

Re: [PATCH v6 1/1] send-email: retrieve Message-ID from outlook SMTP server

[Aditya Garg]

Hi Junio

> There were a few messages that raised concerns with respect to
> on-prem installations of Outlook based servers, where the hostname
> cannot be used to tell if we need this message-id tweaking.
> 
> The following is a completely untested patch, but it should be
> sufficient to illustrate how simple it would be to support an
> option to do so, if we cared about the issue enough.

I tested this patch and it works well. I've added some docs as well and it now looks like the one below.

--->8---
From e06ccabb5a0cef100e50e2b9d6d3c0a1769bda59 Mon Sep 17 00:00:00 2001
From: Aditya Garg <gargaditya08@live.com>
Date: Sat, 26 Apr 2025 08:25:25 +0000
Subject: [PATCH] send-email: add --[no-]smtp-outlook-id-tweak option

From: Junio C Hamano <gitster@pobox.com>

Add an option to allow users to specifically enable or disable
retrieving the Message-ID from the Outlook SMTP server. This can be
for other hosts mimicking the behaviour of Outlook, or for users who set
a custom domain to be a CNAME for the Outlook SMTP server.

Co-authored-by: Aditya Garg <gargaditya08@live.com>
Signed-off-by: Aditya Garg <gargaditya08@live.com>
---
 Documentation/git-send-email.adoc | 15 +++++++++++++++
 git-send-email.perl               | 11 ++++++++++-
 2 files changed, 25 insertions(+), 1 deletion(-)

diff --git a/Documentation/git-send-email.adoc b/Documentation/git-send-email.adoc
index 7f223db42d..8a84fd4342 100644
--- a/Documentation/git-send-email.adoc
+++ b/Documentation/git-send-email.adoc
@@ -421,6 +421,21 @@ recipient's MUA.
 	`sendemail.mailmap.file` or `sendemail.mailmap.blob` configuration
 	values. Defaults to `sendemail.mailmap`.
 
+--[no-]smtp-outlook-id-tweak::
+	Outlook servers discard the Message-ID sent via email and assign a
+	new random Message-ID, thus breaking threads.
++
+--
+- '--smtp-outlook-id-tweak' will attempt to retrieve the ID from the server
+  irrespective of the SMTP server being used. Use only if Microsoft is your
+  email provider.
+- '--no-smtp-outlook-id-tweak' will disable this tweak irrespective of the
+  SMTP server being used.
+--
++
+If not sepcified, the default behaviour will be to enable the tweak only if the
+SMTP server is 'smtp.office365.com' or 'smtp-mail.outlook.com'.
+
 Administering
 ~~~~~~~~~~~~~
 
diff --git a/git-send-email.perl b/git-send-email.perl
index 618474916e..0fb3ee98cf 100755
--- a/git-send-email.perl
+++ b/git-send-email.perl
@@ -60,6 +60,8 @@ sub usage {
     --smtp-user             <str>  * Username for SMTP-AUTH.
     --smtp-pass             <str>  * Password for SMTP-AUTH; not necessary.
     --smtp-encryption       <str>  * tls or ssl; anything else disables.
+    --smtp-outlook-id-tweak <0|1>  * This server munges Message-ID. Retrive it from
+                                     the server and assign to \$message_id.
     --smtp-ssl                     * Deprecated. Use '--smtp-encryption ssl'.
     --smtp-ssl-cert-path    <str>  * Path to ca-certificates (either directory or file).
                                      Pass an empty string to disable certificate
@@ -290,6 +292,7 @@ sub do_edit {
 my $mailmap = 0;
 my $target_xfer_encoding = 'auto';
 my $forbid_sendmail_variables = 1;
+my $outlook_id_tweak = -1;
 
 my %config_bool_settings = (
     "thread" => \$thread,
@@ -305,6 +308,7 @@ sub do_edit {
     "xmailer" => \$use_xmailer,
     "forbidsendmailvariables" => \$forbid_sendmail_variables,
     "mailmap" => \$mailmap,
+    "outlookidtweak" => \$outlook_id_tweak,
 );
 
 my %config_settings = (
@@ -518,6 +522,7 @@ sub config_regexp {
 		    "smtp-pass:s" => \$smtp_authpass,
 		    "smtp-ssl" => sub { $smtp_encryption = 'ssl' },
 		    "smtp-encryption=s" => \$smtp_encryption,
+		    "smtp-outlook-id-tweak!" => \$outlook_id_tweak,
 		    "smtp-ssl-cert-path=s" => \$smtp_ssl_cert_path,
 		    "smtp-debug:i" => \$debug_net_smtp,
 		    "smtp-domain:s" => \$smtp_domain,
@@ -1576,7 +1581,11 @@ sub gen_header {
 
 sub is_outlook {
 	my ($host) = @_;
-	return ($host eq 'smtp.office365.com' || $host eq 'smtp-mail.outlook.com');
+	if ($outlook_id_tweak < 0) {
+		$outlook_id_tweak =
+			($host eq 'smtp.office365.com' ||
+			 $host eq 'smtp-mail.outlook.com') ? 1 : 0;
+	}	return $outlook_id_tweak;
 }
 
 # Prepares the email, then asks the user what to do.
-- 
2.49.0

------


If you want I can officially submit this as well. Would also have to add your Signed-off-by?

Cheers!
Aditya

Re: [PATCH v6 1/1] send-email: retrieve Message-ID from outlook SMTP server

[Eric Sunshine]

On Sat, Apr 26, 2025 at 4:37 AM Aditya Garg <gargaditya08@live.com> wrote:
> Add an option to allow users to specifically enable or disable
> retrieving the Message-ID from the Outlook SMTP server. This can be
> for other hosts mimicking the behaviour of Outlook, or for users who set
> a custom domain to be a CNAME for the Outlook SMTP server.
>
> Co-authored-by: Aditya Garg <gargaditya08@live.com>
> Signed-off-by: Aditya Garg <gargaditya08@live.com>
> ---
> diff --git a/Documentation/git-send-email.adoc b/Documentation/git-send-email.adoc
> @@ -421,6 +421,21 @@ recipient's MUA.
> +--[no-]smtp-outlook-id-tweak::
> +       Outlook servers discard the Message-ID sent via email and assign a
> +       new random Message-ID, thus breaking threads.
> ++
> +--
> +- '--smtp-outlook-id-tweak' will attempt to retrieve the ID from the server
> +  irrespective of the SMTP server being used. Use only if Microsoft is your
> +  email provider.
> +- '--no-smtp-outlook-id-tweak' will disable this tweak irrespective of the
> +  SMTP server being used.
> +--
> ++
> +If not sepcified, the default behaviour will be to enable the tweak only if the
> +SMTP server is 'smtp.office365.com' or 'smtp-mail.outlook.com'.

s/sepcified/specified/

> diff --git a/git-send-email.perl b/git-send-email.perl
> @@ -60,6 +60,8 @@ sub usage {
>      --smtp-encryption       <str>  * tls or ssl; anything else disables.
> +    --smtp-outlook-id-tweak <0|1>  * This server munges Message-ID. Retrive it from
> +                                     the server and assign to \$message_id.

s/Retrive/Retrieve/

As this is a user-facing help message, it seems unusual and unhelpful
for it to be talking about a variable ($message_id) which is internal
to the script. I realize that there is a slight precedent in which the
help for the --foo-cmd options talk about $patch_path, but those cases
are semantically different. Thus, it probably would be better to drop
mention of $message_id and just present a short and sweet explanation,
such as:

    Retrieve Message-ID from the server.

If the user needs more information than that, then he or she can
consult the full documentation.

Re: [PATCH v6 1/1] send-email: retrieve Message-ID from outlook SMTP server

[Aditya Garg]

Hi Eric> On Sat, Apr 26, 2025 at 4:37 AM Aditya Garg <gargaditya08@live.com> wrote:
>> Add an option to allow users to specifically enable or disable
>> retrieving the Message-ID from the Outlook SMTP server. This can be
>> for other hosts mimicking the behaviour of Outlook, or for users who set
>> a custom domain to be a CNAME for the Outlook SMTP server.
>>
>> Co-authored-by: Aditya Garg <gargaditya08@live.com>
>> Signed-off-by: Aditya Garg <gargaditya08@live.com>
>> ---
>> diff --git a/Documentation/git-send-email.adoc b/Documentation/git-send-email.adoc
>> @@ -421,6 +421,21 @@ recipient's MUA.
>> +--[no-]smtp-outlook-id-tweak::
>> +       Outlook servers discard the Message-ID sent via email and assign a
>> +       new random Message-ID, thus breaking threads.
>> ++
>> +--
>> +- '--smtp-outlook-id-tweak' will attempt to retrieve the ID from the server
>> +  irrespective of the SMTP server being used. Use only if Microsoft is your
>> +  email provider.
>> +- '--no-smtp-outlook-id-tweak' will disable this tweak irrespective of the
>> +  SMTP server being used.
>> +--
>> ++
>> +If not sepcified, the default behaviour will be to enable the tweak only if the
>> +SMTP server is 'smtp.office365.com' or 'smtp-mail.outlook.com'.
> 
> s/sepcified/specified/
> 
>> diff --git a/git-send-email.perl b/git-send-email.perl
>> @@ -60,6 +60,8 @@ sub usage {
>>      --smtp-encryption       <str>  * tls or ssl; anything else disables.
>> +    --smtp-outlook-id-tweak <0|1>  * This server munges Message-ID. Retrive it from
>> +                                     the server and assign to \$message_id.
> 
> s/Retrive/Retrieve/
> 
> As this is a user-facing help message, it seems unusual and unhelpful
> for it to be talking about a variable ($message_id) which is internal
> to the script. I realize that there is a slight precedent in which the
> help for the --foo-cmd options talk about $patch_path, but those cases
> are semantically different. Thus, it probably would be better to drop
> mention of $message_id and just present a short and sweet explanation,
> such as:
> 
>     Retrieve Message-ID from the server.

Thanks for the review, I would wait for a go ahead so that I can submit it
officially as well, with your suggestions incorporated :)

Re: [PATCH v6 1/1] send-email: retrieve Message-ID from outlook SMTP server

[Junio C Hamano]

Aditya Garg <gargaditya08@live.com> writes:

> I tested this patch and it works well. I've added some docs as
> well and it now looks like the one below.
>
> --->8---
> From e06ccabb5a0cef100e50e2b9d6d3c0a1769bda59 Mon Sep 17 00:00:00 2001
> From: Aditya Garg <gargaditya08@live.com>
> Date: Sat, 26 Apr 2025 08:25:25 +0000
> Subject: [PATCH] send-email: add --[no-]smtp-outlook-id-tweak option
>
> From: Junio C Hamano <gitster@pobox.com>
>
> Add an option to allow users to specifically enable or disable
> retrieving the Message-ID from the Outlook SMTP server. This can be
> for other hosts mimicking the behaviour of Outlook, or for users who set
> a custom domain to be a CNAME for the Outlook SMTP server.
>
> Co-authored-by: Aditya Garg <gargaditya08@live.com>
> Signed-off-by: Aditya Garg <gargaditya08@live.com>
> ---

Take the authorship yourself.  My involvement is at most Helped-by:
level, I would think.

> @@ -290,6 +292,7 @@ sub do_edit {

>  my $mailmap = 0;
>  my $target_xfer_encoding = 'auto';
>  my $forbid_sendmail_variables = 1;
> +my $outlook_id_tweak = -1;

These lines we see around here are all in a section of variable
enumeration titled:

    # Variables with corresponding config settings & hardcoded defaults

I think this "-1" deserves a bit of an explanation.  It is neither
true or false (and Perl's "is this true?" check on that particular
value would say "true", but that is not how we want it to be taken
and we special case -1 ourselves in the code).

Alternatively perhaps we could initialize it to 'auto' (without any
extra comment here) and then ... 

>  sub is_outlook {
>  	my ($host) = @_;
> -	return ($host eq 'smtp.office365.com' || $host eq 'smtp-mail.outlook.com');
> +	if ($outlook_id_tweak < 0) {

... change this to "eq 'auto'"?  Then the value would be
self-evident.

> +		$outlook_id_tweak =
> +			($host eq 'smtp.office365.com' ||
> +			 $host eq 'smtp-mail.outlook.com') ? 1 : 0;
> +	}	return $outlook_id_tweak;
>  }

Somebody in the near-by thread mentioned that we could enable it
always (and if we do not find a replaced message-id where Outlook
may place one, we keep the original message-id we assigned), but I
personally think it is a poor design taste.  We do not know what a
random SMTP server implementation would do in that response, and all
we examined with any care during this discussion is how an Outlook
server responds.  Once we find a server that gives a random string
there that is not the replacement message-id at all, we would need a
separate knob to opt out of the feature---so let's not go there.

Thanks.

[PATCH] send-email: add --smtp-outlook-id-tweak option

[Aditya Garg]

Add an option to allow users to specifically enable or disable
retrieving the Message-ID from the Outlook SMTP server. This can be used
for other hosts mimicking the behaviour of Outlook, or for users who set
a custom domain to be a CNAME for the Outlook SMTP server.

Helped-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Aditya Garg <gargaditya08@live.com>
---
 Documentation/git-send-email.adoc | 14 ++++++++++++++
 git-send-email.perl               | 14 +++++++++++++-
 2 files changed, 27 insertions(+), 1 deletion(-)

diff --git a/Documentation/git-send-email.adoc b/Documentation/git-send-email.adoc
index 7f223db42d..20f804e4c7 100644
--- a/Documentation/git-send-email.adoc
+++ b/Documentation/git-send-email.adoc
@@ -153,6 +153,20 @@ Note that no attempts whatsoever are made to validate the encoding.
 Default is the value of the `sendemail.transferEncoding` configuration
 value; if that is unspecified, default to `auto`.
 
+--smtp-outlook-id-tweak=(always|never|auto)::
+	Outlook servers discard the Message-ID sent via email and assign a
+	new random Message-ID, thus breaking threads.
++
+--
+- 'auto' will attempt to retrieve the ID from the server only if the SMTP
+  server is 'smtp.office365.com' or 'smtp-mail.outlook.com'.
+- 'always' will attempt to retrieve the ID from the server irrespective of
+  the SMTP server being used. Use only if Microsoft is your email provider.
+- 'never' will disable this tweak irrespective of theSMTP server being used.
+--
++
+If not specified, the default behaviour will be that of 'auto'.
+
 --xmailer::
 --no-xmailer::
 	Add (or prevent adding) the "X-Mailer:" header.  By default,
diff --git a/git-send-email.perl b/git-send-email.perl
index 618474916e..81ff965844 100755
--- a/git-send-email.perl
+++ b/git-send-email.perl
@@ -60,6 +60,8 @@ sub usage {
     --smtp-user             <str>  * Username for SMTP-AUTH.
     --smtp-pass             <str>  * Password for SMTP-AUTH; not necessary.
     --smtp-encryption       <str>  * tls or ssl; anything else disables.
+    --smtp-outlook-id-tweak <str>  * This server munges Message-ID. Retrieve it from
+                                     the server.
     --smtp-ssl                     * Deprecated. Use '--smtp-encryption ssl'.
     --smtp-ssl-cert-path    <str>  * Path to ca-certificates (either directory or file).
                                      Pass an empty string to disable certificate
@@ -290,6 +292,7 @@ sub do_edit {
 my $mailmap = 0;
 my $target_xfer_encoding = 'auto';
 my $forbid_sendmail_variables = 1;
+my $outlook_id_tweak = 'auto';
 
 my %config_bool_settings = (
     "thread" => \$thread,
@@ -333,6 +336,7 @@ sub do_edit {
     "composeencoding" => \$compose_encoding,
     "transferencoding" => \$target_xfer_encoding,
     "sendmailcmd" => \$sendmail_cmd,
+	"outlookidtweak" => \$outlook_id_tweak,
 );
 
 my %config_path_settings = (
@@ -518,6 +522,7 @@ sub config_regexp {
 		    "smtp-pass:s" => \$smtp_authpass,
 		    "smtp-ssl" => sub { $smtp_encryption = 'ssl' },
 		    "smtp-encryption=s" => \$smtp_encryption,
+		    "smtp-outlook-id-tweak=s" => \$outlook_id_tweak,
 		    "smtp-ssl-cert-path=s" => \$smtp_ssl_cert_path,
 		    "smtp-debug:i" => \$debug_net_smtp,
 		    "smtp-domain:s" => \$smtp_domain,
@@ -1576,7 +1581,14 @@ sub gen_header {
 
 sub is_outlook {
 	my ($host) = @_;
-	return ($host eq 'smtp.office365.com' || $host eq 'smtp-mail.outlook.com');
+	if ($outlook_id_tweak eq 'always') {
+		return 1;
+	} elsif ($outlook_id_tweak eq 'never') {
+		return 0;
+	} else {
+		return ($host eq 'smtp.office365.com' ||
+			$host eq 'smtp-mail.outlook.com');
+	}
 }
 
 # Prepares the email, then asks the user what to do.
-- 
2.49.0

[PATCH v2] send-email: add --smtp-outlook-id-tweak option

[Aditya Garg]

Add an option to allow users to specifically enable or disable
retrieving the Message-ID from the Outlook SMTP server. This can be used
for other hosts mimicking the behaviour of Outlook, or for users who set
a custom domain to be a CNAME for the Outlook SMTP server.

Helped-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Aditya Garg <gargaditya08@live.com>
---
v2: Replace tab with spaces in "outlookidtweak" => \$outlook_id_tweak,

 Documentation/git-send-email.adoc | 14 ++++++++++++++
 git-send-email.perl               | 14 +++++++++++++-
 2 files changed, 27 insertions(+), 1 deletion(-)

diff --git a/Documentation/git-send-email.adoc b/Documentation/git-send-email.adoc
index 7f223db42d..20f804e4c7 100644
--- a/Documentation/git-send-email.adoc
+++ b/Documentation/git-send-email.adoc
@@ -153,6 +153,20 @@ Note that no attempts whatsoever are made to validate the encoding.
 Default is the value of the `sendemail.transferEncoding` configuration
 value; if that is unspecified, default to `auto`.
 
+--smtp-outlook-id-tweak=(always|never|auto)::
+	Outlook servers discard the Message-ID sent via email and assign a
+	new random Message-ID, thus breaking threads.
++
+--
+- 'auto' will attempt to retrieve the ID from the server only if the SMTP
+  server is 'smtp.office365.com' or 'smtp-mail.outlook.com'.
+- 'always' will attempt to retrieve the ID from the server irrespective of
+  the SMTP server being used. Use only if Microsoft is your email provider.
+- 'never' will disable this tweak irrespective of theSMTP server being used.
+--
++
+If not specified, the default behaviour will be that of 'auto'.
+
 --xmailer::
 --no-xmailer::
 	Add (or prevent adding) the "X-Mailer:" header.  By default,
diff --git a/git-send-email.perl b/git-send-email.perl
index 618474916e..20cc460ed6 100755
--- a/git-send-email.perl
+++ b/git-send-email.perl
@@ -60,6 +60,8 @@ sub usage {
     --smtp-user             <str>  * Username for SMTP-AUTH.
     --smtp-pass             <str>  * Password for SMTP-AUTH; not necessary.
     --smtp-encryption       <str>  * tls or ssl; anything else disables.
+    --smtp-outlook-id-tweak <str>  * This server munges Message-ID. Retrieve it from
+                                     the server.
     --smtp-ssl                     * Deprecated. Use '--smtp-encryption ssl'.
     --smtp-ssl-cert-path    <str>  * Path to ca-certificates (either directory or file).
                                      Pass an empty string to disable certificate
@@ -290,6 +292,7 @@ sub do_edit {
 my $mailmap = 0;
 my $target_xfer_encoding = 'auto';
 my $forbid_sendmail_variables = 1;
+my $outlook_id_tweak = 'auto';
 
 my %config_bool_settings = (
     "thread" => \$thread,
@@ -333,6 +336,7 @@ sub do_edit {
     "composeencoding" => \$compose_encoding,
     "transferencoding" => \$target_xfer_encoding,
     "sendmailcmd" => \$sendmail_cmd,
+    "outlookidtweak" => \$outlook_id_tweak,
 );
 
 my %config_path_settings = (
@@ -518,6 +522,7 @@ sub config_regexp {
 		    "smtp-pass:s" => \$smtp_authpass,
 		    "smtp-ssl" => sub { $smtp_encryption = 'ssl' },
 		    "smtp-encryption=s" => \$smtp_encryption,
+		    "smtp-outlook-id-tweak=s" => \$outlook_id_tweak,
 		    "smtp-ssl-cert-path=s" => \$smtp_ssl_cert_path,
 		    "smtp-debug:i" => \$debug_net_smtp,
 		    "smtp-domain:s" => \$smtp_domain,
@@ -1576,7 +1581,14 @@ sub gen_header {
 
 sub is_outlook {
 	my ($host) = @_;
-	return ($host eq 'smtp.office365.com' || $host eq 'smtp-mail.outlook.com');
+	if ($outlook_id_tweak eq 'always') {
+		return 1;
+	} elsif ($outlook_id_tweak eq 'never') {
+		return 0;
+	} else {
+		return ($host eq 'smtp.office365.com' ||
+			$host eq 'smtp-mail.outlook.com');
+	}
 }
 
 # Prepares the email, then asks the user what to do.
-- 
2.49.0

Re: [PATCH v2] send-email: add --smtp-outlook-id-tweak option

[Junio C Hamano]

Aditya Garg <gargaditya08@live.com> writes:

> Add an option to allow users to specifically enable or disable
> retrieving the Message-ID from the Outlook SMTP server. This can be used
> for other hosts mimicking the behaviour of Outlook, or for users who set
> a custom domain to be a CNAME for the Outlook SMTP server.
>
> Helped-by: Junio C Hamano <gitster@pobox.com>
> Signed-off-by: Aditya Garg <gargaditya08@live.com>
> ---
> v2: Replace tab with spaces in "outlookidtweak" => \$outlook_id_tweak,

Good eyes ;-)

> +--smtp-outlook-id-tweak=(always|never|auto)::
> +	Outlook servers discard the Message-ID sent via email and assign a
> +	new random Message-ID, thus breaking threads.
> ++
> +--
> +- 'auto' will attempt to retrieve the ID from the server only if the SMTP
> +  server is 'smtp.office365.com' or 'smtp-mail.outlook.com'.
> +- 'always' will attempt to retrieve the ID from the server irrespective of
> +  the SMTP server being used. Use only if Microsoft is your email provider.
> +- 'never' will disable this tweak irrespective of theSMTP server being used.

It is a shame that this is not just a simple boolean.  Those who
expect 'true' to kick in would be disappointed to find that their
'true' means the same thing as 'auto'.

If I were designing this feature, I would rather make it say:

    --[no-]outlook-message-id-fix::

	Outlook servers [DO THIS].  Giving this option reads the
	message-id assigned by the Outlook server and use it as the
	In-Reply-To message ID for subsequent messages.  Without the
	option, connections to only ... and ... automatically gets
	this tweak.  Pass `--no-outlook-message-id-fix` to disable
	the fix even for these hosts.

and make the implementation behave that way.

Note that this is a command line option that is specific to
git-send-email, so I omitted "smtp" from the name and instead
replaced 'id' with 'message-id' to make it more explicit what
gets munged.

But I do not care too deeply either way.  Let me queue the patch
as-is for now.

Thanks.

Re: [PATCH v2] send-email: add --smtp-outlook-id-tweak option

[Aditya Garg]

> On 29 Apr 2025, at 2:17 AM, Junio C Hamano <gitster@pobox.com> wrote:
> 
> Aditya Garg <gargaditya08@live.com> writes:
> 
>> Add an option to allow users to specifically enable or disable
>> retrieving the Message-ID from the Outlook SMTP server. This can be used
>> for other hosts mimicking the behaviour of Outlook, or for users who set
>> a custom domain to be a CNAME for the Outlook SMTP server.
>> 
>> Helped-by: Junio C Hamano <gitster@pobox.com>
>> Signed-off-by: Aditya Garg <gargaditya08@live.com>
>> ---
>> v2: Replace tab with spaces in "outlookidtweak" => \$outlook_id_tweak,
> 
> Good eyes ;-)
> 
>> +--smtp-outlook-id-tweak=(always|never|auto)::
>> +    Outlook servers discard the Message-ID sent via email and assign a
>> +    new random Message-ID, thus breaking threads.
>> ++
>> +--
>> +- 'auto' will attempt to retrieve the ID from the server only if the SMTP
>> +  server is 'smtp.office365.com' or 'smtp-mail.outlook.com'.
>> +- 'always' will attempt to retrieve the ID from the server irrespective of
>> +  the SMTP server being used. Use only if Microsoft is your email provider.
>> +- 'never' will disable this tweak irrespective of theSMTP server being used.
> 
> It is a shame that this is not just a simple boolean.  Those who
> expect 'true' to kick in would be disappointed to find that their
> 'true' means the same thing as 'auto'.
> 
> If I were designing this feature, I would rather make it say:
> 
>    --[no-]outlook-message-id-fix::

I am trying to implement this feature, since a boolean makes sense.

But, I feel the parameter is too long, and will have to rewrite the whole sub usage due to this.
> 
>    Outlook servers [DO THIS].  Giving this option reads the
>    message-id assigned by the Outlook server and use it as the
>    In-Reply-To message ID for subsequent messages.  Without the
>    option, connections to only ... and ... automatically gets
>    this tweak.  Pass `--no-outlook-message-id-fix` to disable
>    the fix even for these hosts.
> 
> and make the implementation behave that way.
> 
> Note that this is a command line option that is specific to
> git-send-email, so I omitted "smtp" from the name and instead
> replaced 'id' with 'message-id' to make it more explicit what
> gets munged.
> 
> But I do not care too deeply either way.  Let me queue the patch
> as-is for now.
> 
> Thanks.

[PATCH v3] send-email: add --[no-]outlook-id-fix option

[Aditya Garg]

Add an option to allow users to specifically enable or disable
retrieving the Message-ID from the Outlook SMTP server. This can be used
for other hosts mimicking the behaviour of Outlook, or for users who set
a custom domain to be a CNAME for the Outlook SMTP server.

Helped-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Aditya Garg <gargaditya08@live.com>
---
v2: Replace tab with spaces in "outlookidtweak" => \$outlook_id_tweak,
v3: Rename to --[no-]outlook-id-fix and make it bool. Also add missing
    * in description of --no-smtp-auth.

 Documentation/git-send-email.adoc | 15 +++++++++++++++
 git-send-email.perl               | 13 +++++++++++--
 2 files changed, 26 insertions(+), 2 deletions(-)

diff --git a/Documentation/git-send-email.adoc b/Documentation/git-send-email.adoc
index 7f223db42d..5760248893 100644
--- a/Documentation/git-send-email.adoc
+++ b/Documentation/git-send-email.adoc
@@ -115,6 +115,21 @@ illustration below where `[PATCH v2 0/3]` is in reply to `[PATCH 0/2]`:
 Only necessary if --compose is also set.  If --compose
 is not set, this will be prompted for.
 
+--[no-]outlook-id-fix::
+	Outlook servers discard the Message-ID sent via email and assign a
+	new random Message-ID, thus breaking threads.
++
+--
+- '--outlook-id-fix' will attempt to retrieve the ID from the server
+  irrespective of the SMTP server being used. Use only if Microsoft is your
+  email provider.
+- '--no-outlook-id-fix' will disable this tweak irrespective of the SMTP
+  server being used.
+--
++
+If not specified, the default behaviour will be to enable the tweak only if the
+SMTP server is 'smtp.office365.com' or 'smtp-mail.outlook.com'.
+
 --subject=<string>::
 	Specify the initial subject of the email thread.
 	Only necessary if --compose is also set.  If --compose
diff --git a/git-send-email.perl b/git-send-email.perl
index 618474916e..ed707bfa46 100755
--- a/git-send-email.perl
+++ b/git-send-email.perl
@@ -41,6 +41,8 @@ sub usage {
     --subject               <str>  * Email "Subject:"
     --reply-to              <str>  * Email "Reply-To:"
     --in-reply-to           <str>  * Email "In-Reply-To:"
+    --[no-]outlook-id-fix          * This server munges Message-ID. Retrieve it from
+                                     the server.
     --[no-]xmailer                 * Add "X-Mailer:" header (default).
     --[no-]annotate                * Review each patch that will be sent in an editor.
     --compose                      * Open an editor for introduction.
@@ -68,7 +70,7 @@ sub usage {
     --smtp-auth             <str>  * Space-separated list of allowed AUTH mechanisms, or
                                      "none" to disable authentication.
                                      This setting forces to use one of the listed mechanisms.
-    --no-smtp-auth                   Disable SMTP authentication. Shorthand for
+    --no-smtp-auth                 * Disable SMTP authentication. Shorthand for
                                      `--smtp-auth=none`
     --smtp-debug            <0|1>  * Disable, enable Net::SMTP debug.
 
@@ -290,6 +292,7 @@ sub do_edit {
 my $mailmap = 0;
 my $target_xfer_encoding = 'auto';
 my $forbid_sendmail_variables = 1;
+my $outlook_id_fix = 'auto';
 
 my %config_bool_settings = (
     "thread" => \$thread,
@@ -305,6 +308,7 @@ sub do_edit {
     "xmailer" => \$use_xmailer,
     "forbidsendmailvariables" => \$forbid_sendmail_variables,
     "mailmap" => \$mailmap,
+    "outlookidfix" => \$outlook_id_fix,
 );
 
 my %config_settings = (
@@ -551,6 +555,7 @@ sub config_regexp {
 		    "relogin-delay=i" => \$relogin_delay,
 		    "git-completion-helper" => \$git_completion_helper,
 		    "v=s" => \$reroll_count,
+		    "outlook-id-fix!" => \$outlook_id_fix,
 );
 $rc = GetOptions(%options);
 
@@ -1576,7 +1581,11 @@ sub gen_header {
 
 sub is_outlook {
 	my ($host) = @_;
-	return ($host eq 'smtp.office365.com' || $host eq 'smtp-mail.outlook.com');
+	if ($outlook_id_fix eq 'auto') {
+		$outlook_id_fix =
+			($host eq 'smtp.office365.com' ||
+			 $host eq 'smtp-mail.outlook.com') ? 1 : 0;
+	}	return $outlook_id_fix;
 }
 
 # Prepares the email, then asks the user what to do.
-- 
2.49.0

Re: [PATCH v3] send-email: add --[no-]outlook-id-fix option

[Aditya Garg]

On 29-04-2025 04:22 pm, Aditya Garg wrote:
> Add an option to allow users to specifically enable or disable
> retrieving the Message-ID from the Outlook SMTP server. This can be used
> for other hosts mimicking the behaviour of Outlook, or for users who set
> a custom domain to be a CNAME for the Outlook SMTP server.
> 
> Helped-by: Junio C Hamano <gitster@pobox.com>
> Signed-off-by: Aditya Garg <gargaditya08@live.com>
> ---

BTW, I am not sure whether I need to send this patch in this thread too. It
is a bug fix, which I encountered with my adventures with git send-email.

Link: https://lore.kernel.org/git/PN0PR01MB9588EBBF200EA002E558D4E0B8872@PN0PR01MB9588.INDPRD01.PROD.OUTLOOK.COM/

Re: [PATCH v3] send-email: add --[no-]outlook-id-fix option

[Junio C Hamano]

Aditya Garg <gargaditya08@live.com> writes:

> +    --[no-]outlook-id-fix          * This server munges Message-ID. Retrieve it from
> +                                     the server.

We know how to retrieve the new message-id only from Outlook server,
but I fear that the above does not convey it (unless we count the
substring "outlook" in the option name).  I came up with

    --[no-]outlook-id-fix          * The smtp host is an Outlook server that
                                     munges the Message-ID.

as a replacement, but I am not sure if it is much better.

> -    --no-smtp-auth                   Disable SMTP authentication. Shorthand for
> +    --no-smtp-auth                 * Disable SMTP authentication. Shorthand for
>                                       `--smtp-auth=none`

Good eyes.

As long as it is mentioned in the proposed log message, it is OK to
make such a small and unrelated correction "while at it".

    $ git log -p --grep=While.at.it

will find many existing examples.

>  sub is_outlook {
>  	my ($host) = @_;
> -	return ($host eq 'smtp.office365.com' || $host eq 'smtp-mail.outlook.com');
> +	if ($outlook_id_fix eq 'auto') {
> +		$outlook_id_fix =
> +			($host eq 'smtp.office365.com' ||
> +			 $host eq 'smtp-mail.outlook.com') ? 1 : 0;
> +	}	return $outlook_id_fix;

No syntax error here, but let's have the final "return $outlook_id_fix;"
on its own line after "if (...) { ... }" statement.

Re: [PATCH v3] send-email: add --[no-]outlook-id-fix option

[Junio C Hamano]

Aditya Garg <gargaditya08@live.com> writes:

> +--[no-]outlook-id-fix::
> +	Outlook servers discard the Message-ID sent via email and assign a
> +	new random Message-ID, thus breaking threads.
> ++
> +--
> +- '--outlook-id-fix' will attempt to retrieve the ID from the server
> +  irrespective of the SMTP server being used. Use only if Microsoft is your
> +  email provider.
> +- '--no-outlook-id-fix' will disable this tweak irrespective of the SMTP
> +  server being used.
> +--
> ++
> +If not specified, the default behaviour will be to enable the tweak only if the
> +SMTP server is 'smtp.office365.com' or 'smtp-mail.outlook.com'.

I found "irrespective of the SMTP server being used" part a bit
confusing, which seemed to contradict with the next sentence to tell
you to use it only against Outlook based servers.

After reading the "If not specified" part, I can sort of guess that
you wanted to "irrespective" mean "even when we are (or are not)
talking to outlook.com", but it is still a confusing description.

Would the following work better, I wonder...?

	Microsoft Outlook SMTP servers ..., thus breaking threads.

	With `--outlook-id-fix`, "git send-email" uses a mechanism
	specific to Outlook servers to learn the Message-ID the
	server assigned to fix the threading.  Use it only when you
	know that the server reports the rewritten Message-ID the
	same way as Outlook servers do.

	Without this option specified, the fix is done by default
	when talking to smtp.office365.com or smtp-mail.outlook.com.
	Use `--no-outlook-id-fix` to disable even when talking to
	these two servers.

Thanks.

Re: [PATCH v3] send-email: add --[no-]outlook-id-fix option

[Aditya Garg]

On 29-04-2025 09:54 pm, Junio C Hamano wrote:
> Aditya Garg <gargaditya08@live.com> writes:
> 
>> +--[no-]outlook-id-fix::
>> +	Outlook servers discard the Message-ID sent via email and assign a
>> +	new random Message-ID, thus breaking threads.
>> ++
>> +--
>> +- '--outlook-id-fix' will attempt to retrieve the ID from the server
>> +  irrespective of the SMTP server being used. Use only if Microsoft is your
>> +  email provider.
>> +- '--no-outlook-id-fix' will disable this tweak irrespective of the SMTP
>> +  server being used.
>> +--
>> ++
>> +If not specified, the default behaviour will be to enable the tweak only if the
>> +SMTP server is 'smtp.office365.com' or 'smtp-mail.outlook.com'.
> 
> I found "irrespective of the SMTP server being used" part a bit
> confusing, which seemed to contradict with the next sentence to tell
> you to use it only against Outlook based servers.
> 
> After reading the "If not specified" part, I can sort of guess that
> you wanted to "irrespective" mean "even when we are (or are not)
> talking to outlook.com", but it is still a confusing description.
> 
> Would the following work better, I wonder...?
> 
> 	Microsoft Outlook SMTP servers ..., thus breaking threads.
> 
> 	With `--outlook-id-fix`, "git send-email" uses a mechanism
> 	specific to Outlook servers to learn the Message-ID the
> 	server assigned to fix the threading.  Use it only when you
> 	know that the server reports the rewritten Message-ID the
> 	same way as Outlook servers do.
> 
> 	Without this option specified, the fix is done by default
> 	when talking to smtp.office365.com or smtp-mail.outlook.com.
> 	Use `--no-outlook-id-fix` to disable even when talking to
> 	these two servers.

I'll just add this.

[PATCH v4] send-email: add --[no-]outlook-id-fix option

[Aditya Garg]

Add an option to allow users to specifically enable or disable
retrieving the Message-ID from the Outlook SMTP server. This can be used
for other hosts mimicking the behaviour of Outlook, or for users who set
a custom domain to be a CNAME for the Outlook SMTP server.

While at it, lets also add missing * in description of --no-smtp-auth.

Helped-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Aditya Garg <gargaditya08@live.com>
---
v2: Replace tab with spaces in "outlookidtweak" => \$outlook_id_tweak,
v3: Rename to --[no-]outlook-id-fix and make it bool. Also add missing
    * in description of --no-smtp-auth.
v4: Improve documentation and the log message. Also, have the final
    "return $outlook_id_fix;" on its own line.

 Documentation/git-send-email.adoc | 13 +++++++++++++
 git-send-email.perl               | 14 ++++++++++++--
 2 files changed, 25 insertions(+), 2 deletions(-)

diff --git a/Documentation/git-send-email.adoc b/Documentation/git-send-email.adoc
index 7f223db42d..7ccca16296 100644
--- a/Documentation/git-send-email.adoc
+++ b/Documentation/git-send-email.adoc
@@ -115,6 +115,19 @@ illustration below where `[PATCH v2 0/3]` is in reply to `[PATCH 0/2]`:
 Only necessary if --compose is also set.  If --compose
 is not set, this will be prompted for.
 
+--[no-]outlook-id-fix::
+	Microsoft Outlook SMTP servers discard the Message-ID sent via email and
+	assign a new random Message-ID, thus breaking threads.
+
+	With `--outlook-id-fix`, 'git send-email' uses a mechanism specific to
+	Outlook servers to learn the Message-ID the server assigned to fix the
+	threading. Use it only when you know that the server reports the
+	rewritten Message-ID the same way as Outlook servers do.
+
+	Without this option specified, the fix is done by default when talking
+	to 'smtp.office365.com' or 'smtp-mail.outlook.com'. Use
+	`--no-outlook-id-fix` to disable even when talking to these two servers.
+
 --subject=<string>::
 	Specify the initial subject of the email thread.
 	Only necessary if --compose is also set.  If --compose
diff --git a/git-send-email.perl b/git-send-email.perl
index 618474916e..4215f8f7e9 100755
--- a/git-send-email.perl
+++ b/git-send-email.perl
@@ -41,6 +41,8 @@ sub usage {
     --subject               <str>  * Email "Subject:"
     --reply-to              <str>  * Email "Reply-To:"
     --in-reply-to           <str>  * Email "In-Reply-To:"
+    --[no-]outlook-id-fix          * The SMTP host is an Outlook server that munges the
+                                     Message-ID. Retrieve it from the server.
     --[no-]xmailer                 * Add "X-Mailer:" header (default).
     --[no-]annotate                * Review each patch that will be sent in an editor.
     --compose                      * Open an editor for introduction.
@@ -68,7 +70,7 @@ sub usage {
     --smtp-auth             <str>  * Space-separated list of allowed AUTH mechanisms, or
                                      "none" to disable authentication.
                                      This setting forces to use one of the listed mechanisms.
-    --no-smtp-auth                   Disable SMTP authentication. Shorthand for
+    --no-smtp-auth                 * Disable SMTP authentication. Shorthand for
                                      `--smtp-auth=none`
     --smtp-debug            <0|1>  * Disable, enable Net::SMTP debug.
 
@@ -290,6 +292,7 @@ sub do_edit {
 my $mailmap = 0;
 my $target_xfer_encoding = 'auto';
 my $forbid_sendmail_variables = 1;
+my $outlook_id_fix = 'auto';
 
 my %config_bool_settings = (
     "thread" => \$thread,
@@ -305,6 +308,7 @@ sub do_edit {
     "xmailer" => \$use_xmailer,
     "forbidsendmailvariables" => \$forbid_sendmail_variables,
     "mailmap" => \$mailmap,
+    "outlookidfix" => \$outlook_id_fix,
 );
 
 my %config_settings = (
@@ -551,6 +555,7 @@ sub config_regexp {
 		    "relogin-delay=i" => \$relogin_delay,
 		    "git-completion-helper" => \$git_completion_helper,
 		    "v=s" => \$reroll_count,
+		    "outlook-id-fix!" => \$outlook_id_fix,
 );
 $rc = GetOptions(%options);
 
@@ -1576,7 +1581,12 @@ sub gen_header {
 
 sub is_outlook {
 	my ($host) = @_;
-	return ($host eq 'smtp.office365.com' || $host eq 'smtp-mail.outlook.com');
+	if ($outlook_id_fix eq 'auto') {
+		$outlook_id_fix =
+			($host eq 'smtp.office365.com' ||
+			 $host eq 'smtp-mail.outlook.com') ? 1 : 0;
+	}
+	return $outlook_id_fix;
 }
 
 # Prepares the email, then asks the user what to do.
-- 
2.49.0

Re: [PATCH v4] send-email: add --[no-]outlook-id-fix option

[Junio C Hamano]

Aditya Garg <gargaditya08@live.com> writes:

> Add an option to allow users to specifically enable or disable
> retrieving the Message-ID from the Outlook SMTP server. This can be used
> for other hosts mimicking the behaviour of Outlook, or for users who set
> a custom domain to be a CNAME for the Outlook SMTP server.
>
> While at it, lets also add missing * in description of --no-smtp-auth.
>
> Helped-by: Junio C Hamano <gitster@pobox.com>
> Signed-off-by: Aditya Garg <gargaditya08@live.com>
> ---
> v2: Replace tab with spaces in "outlookidtweak" => \$outlook_id_tweak,
> v3: Rename to --[no-]outlook-id-fix and make it bool. Also add missing
>     * in description of --no-smtp-auth.
> v4: Improve documentation and the log message. Also, have the final
>     "return $outlook_id_fix;" on its own line.
>
>  Documentation/git-send-email.adoc | 13 +++++++++++++
>  git-send-email.perl               | 14 ++++++++++++--
>  2 files changed, 25 insertions(+), 2 deletions(-)
>
> diff --git a/Documentation/git-send-email.adoc b/Documentation/git-send-email.adoc
> index 7f223db42d..7ccca16296 100644
> --- a/Documentation/git-send-email.adoc
> +++ b/Documentation/git-send-email.adoc
> @@ -115,6 +115,19 @@ illustration below where `[PATCH v2 0/3]` is in reply to `[PATCH 0/2]`:
>  Only necessary if --compose is also set.  If --compose
>  is not set, this will be prompted for.
>  
> +--[no-]outlook-id-fix::
> +	Microsoft Outlook SMTP servers discard the Message-ID sent via email and
> +	assign a new random Message-ID, thus breaking threads.
> +
> +	With `--outlook-id-fix`, 'git send-email' uses a mechanism specific to
> +	Outlook servers to learn the Message-ID the server assigned to fix the
> +	threading. Use it only when you know that the server reports the
> +	rewritten Message-ID the same way as Outlook servers do.
> +
> +	Without this option specified, the fix is done by default when talking
> +	to 'smtp.office365.com' or 'smtp-mail.outlook.com'. Use
> +	`--no-outlook-id-fix` to disable even when talking to these two servers.

You'd need the usual "a line with only + on it as the paragraph
separator, with subsequent paragraphs dedented" mark-up.  I'll tweak
this part on my end.

Re: [PATCH v4] send-email: add --[no-]outlook-id-fix option

[Aditya Garg]

> You'd need the usual "a line with only + on it as the paragraph
> separator, with subsequent paragraphs dedented" mark-up.  I'll tweak
> this part on my end.

Thanks!