kgdb replacing newline with CRLF in custom query response

[Omar Sandoval <osandov@osandov.com>]

Hi everyone,

Amal is working on adding a custom query packet to kgdb for getting the
kernel's vmcoreinfo. The rationale and details are available here:
https://github.com/osandov/drgn/wiki/GDB-Remote-Protocol-proposal:-linux.vmcoreinfo-query-packet

vmcoreinfo is about 3kB, so we were hoping to avoid hex-encoding the
response and doubling the time it takes to transmit over a slow serial
connection. Instead, we were hoping to use the escaped binary format,
which escapes the characters #$}* and leaves other bytes untouched.

We ran into a problem, though: vmcoreinfo contains newline characters,
which the serial core replaces with CRLF; see commit c7d44a02ac60
("serial_core: Commonalize crlf when working w/ a non open console
port"). This effectively corrupts the data and causes a checksum
mismatch.

We'd love some input on how to work around this, especially from the
kgdb maintainers. Here are a few options, in descending order of my
preference:

1. Disable the LF -> CRLF replacement while sending binary data.
2. Escape the newlines using some other custom scheme.
3. Give up and hex-encode the response.

Any thoughts?

Thanks,
Omar