[PATCH] mingw: make isatty() recognize MSYS2's pseudo terminals (/dev/pty*)

[PATCH] mingw: make isatty() recognize MSYS2's pseudo terminals (/dev/pty*)

[Johannes Schindelin]

From: Karsten Blees <blees@dcon.de>

MSYS2 emulates pseudo terminals via named pipes, and isatty() returns 0
for such file descriptors. Therefore, some interactive functionality
(such as launching a pager, asking if a failed unlink should be repeated
etc.) doesn't work when run in a terminal emulator that uses MSYS2's
ptys (such as mintty).

However, MSYS2 uses special names for its pty pipes ('msys-*-pty*'),
which allows us to distinguish them from normal piped input / output.

On startup, check if stdin / stdout / stderr are connected to such pipes
using the NtQueryObject API from NTDll.dll. If the names match, adjust
the flags in MSVCRT's ioinfo structure accordingly.

Signed-off-by: Karsten Blees <blees@dcon.de>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
Published-As: https://github.com/dscho/git/releases/tag/windows-isatty-v1
 compat/winansi.c | 58 ++++++++++++++++++++++++++++++++++++++++++++++++++++----
 config.mak.uname |  3 ++-
 2 files changed, 56 insertions(+), 5 deletions(-)

diff --git a/compat/winansi.c b/compat/winansi.c
index 5dfa5ed..3be60ce 100644
--- a/compat/winansi.c
+++ b/compat/winansi.c
@@ -483,6 +483,7 @@ static size_t sizeof_ioinfo = 0;
 #define IOINFO_L2E 5
 #define IOINFO_ARRAY_ELTS (1 << IOINFO_L2E)
 
+#define FPIPE 0x08
 #define FDEV  0x40
 
 static inline ioinfo* _pioinfo(int fd)
@@ -530,6 +531,45 @@ static HANDLE swap_osfhnd(int fd, HANDLE new_handle)
 	return old_handle;
 }
 
+#ifdef DETECT_MSYS_TTY
+
+#include <winternl.h>
+#include <ntstatus.h>
+
+static void detect_msys_tty(int fd)
+{
+	ULONG result;
+	BYTE buffer[1024];
+	POBJECT_NAME_INFORMATION nameinfo = (POBJECT_NAME_INFORMATION) buffer;
+	PWSTR name;
+
+	/* check if fd is a pipe */
+	HANDLE h = (HANDLE) _get_osfhandle(fd);
+	if (GetFileType(h) != FILE_TYPE_PIPE)
+		return;
+
+	/* get pipe name */
+	if (!NT_SUCCESS(NtQueryObject(h, ObjectNameInformation,
+			buffer, sizeof(buffer) - 2, &result)))
+		return;
+	name = nameinfo->Name.Buffer;
+	name[nameinfo->Name.Length] = 0;
+
+	/* check if this could be a MSYS2 pty pipe ('msys-XXXX-ptyN-XX') */
+	if (!wcsstr(name, L"msys-") || !wcsstr(name, L"-pty"))
+		return;
+
+	/* init ioinfo size if we haven't done so */
+	if (init_sizeof_ioinfo())
+		return;
+
+	/* set FDEV flag, reset FPIPE flag */
+	_pioinfo(fd)->osflags &= ~FPIPE;
+	_pioinfo(fd)->osflags |= FDEV;
+}
+
+#endif
+
 void winansi_init(void)
 {
 	int con1, con2;
@@ -538,8 +578,15 @@ void winansi_init(void)
 	/* check if either stdout or stderr is a console output screen buffer */
 	con1 = is_console(1);
 	con2 = is_console(2);
-	if (!con1 && !con2)
+	if (!con1 && !con2) {
+#ifdef DETECT_MSYS_TTY
+		/* check if stdin / stdout / stderr are MSYS2 pty pipes */
+		detect_msys_tty(0);
+		detect_msys_tty(1);
+		detect_msys_tty(2);
+#endif
 		return;
+	}
 
 	/* create a named pipe to communicate with the console thread */
 	xsnprintf(name, sizeof(name), "\\\\.\\pipe\\winansi%lu", GetCurrentProcessId());
@@ -575,8 +622,11 @@ void winansi_init(void)
 HANDLE winansi_get_osfhandle(int fd)
 {
 	HANDLE hnd = (HANDLE) _get_osfhandle(fd);
-	if ((fd == 1 || fd == 2) && isatty(fd)
-	    && GetFileType(hnd) == FILE_TYPE_PIPE)
-		return (fd == 1) ? hconsole1 : hconsole2;
+	if (isatty(fd) && GetFileType(hnd) == FILE_TYPE_PIPE) {
+		if (fd == 1 && hconsole1)
+			return hconsole1;
+		else if (fd == 2 && hconsole2)
+			return hconsole2;
+	}
 	return hnd;
 }
diff --git a/config.mak.uname b/config.mak.uname
index 40d6b29..a88f139 100644
--- a/config.mak.uname
+++ b/config.mak.uname
@@ -557,7 +557,8 @@ else
 			BASIC_LDFLAGS += -Wl,--large-address-aware
 		endif
 		CC = gcc
-		COMPAT_CFLAGS += -D__USE_MINGW_ANSI_STDIO=0
+		COMPAT_CFLAGS += -D__USE_MINGW_ANSI_STDIO=0 -DDETECT_MSYS_TTY
+		EXTLIBS += -lntdll
 		INSTALL = /bin/install
 		NO_R_TO_GCC_LINKER = YesPlease
 		INTERNAL_QSORT = YesPlease
-- 
2.8.1.306.gff998f2

Re: [PATCH] mingw: make isatty() recognize MSYS2's pseudo terminals (/dev/pty*)

[Johannes Schindelin]

Hi Junio,

On Wed, 27 Apr 2016, Johannes Schindelin wrote:

> From: Karsten Blees <blees@dcon.de>
> 
> MSYS2 emulates pseudo terminals via named pipes, and isatty() returns 0
> for such file descriptors. Therefore, some interactive functionality
> (such as launching a pager, asking if a failed unlink should be repeated
> etc.) doesn't work when run in a terminal emulator that uses MSYS2's
> ptys (such as mintty).
> 
> However, MSYS2 uses special names for its pty pipes ('msys-*-pty*'),
> which allows us to distinguish them from normal piped input / output.
> 
> On startup, check if stdin / stdout / stderr are connected to such pipes
> using the NtQueryObject API from NTDll.dll. If the names match, adjust
> the flags in MSVCRT's ioinfo structure accordingly.
> 
> Signed-off-by: Karsten Blees <blees@dcon.de>
> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

For the record, this patch is needed to make the output of `git log` *not*
whizz by when running in MinTTY, the default terminal of Git for Windows
and MSYS2.

I do not see this patch in 'pu'... Anything I can do to get this into
'master' eventually?

Ciao,
Dscho

Re: [PATCH] mingw: make isatty() recognize MSYS2's pseudo terminals (/dev/pty*)

[Junio C Hamano]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

> I do not see this patch in 'pu'... Anything I can do to get this into
> 'master' eventually?

The reason why I left it in my inbox was because I couldn't tell if
this was a final submission with concensus among Git developers on
Windows, or I should be giving a chance to comment to some folks who
work on Windows port but are not necessarily closely communicating
with you.

If the message were Cc'ed to J6t, I would probably have queued it on
'pu' and marked it as "Will merge after waiting for a few days" in
What's cooking.

Thanks.

Re: [PATCH] mingw: make isatty() recognize MSYS2's pseudo terminals (/dev/pty*)

[Johannes Schindelin]

Hi Junio,

On Thu, 26 May 2016, Junio C Hamano wrote:

> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
> 
> > I do not see this patch in 'pu'... Anything I can do to get this into
> > 'master' eventually?
> 
> The reason why I left it in my inbox was because I couldn't tell if
> this was a final submission with concensus among Git developers on
> Windows, or I should be giving a chance to comment to some folks who
> work on Windows port but are not necessarily closely communicating
> with you.
> 
> If the message were Cc'ed to J6t, I would probably have queued it on
> 'pu' and marked it as "Will merge after waiting for a few days" in
> What's cooking.

Hannes does not use Git for Windows' SDK, but instead uses an old
MSys-based development environment that predates even msysGit (which was
the SDK used to develop Git for Windows 1.x). His environment most
notably does *not* use MinTTY, and therefore is unaffected by the bug that
this patch fixes.

That was the entire reasoning why I did not ask Hannes to review this
patch: it is unlikely that he knows about the issue, let alone about the
correctness of the implementation of this patch.

The implementation relies on some really low-level knowledge of the
Windows internals, which Karsten reverse-engineered and I verified.

Thanks,
Dscho