Re: Fetching an mbox from lore

[Willy Tarreau <w@1wt.eu>]

Hi Maxime,

On Sat, Jul 22, 2023 at 07:12:33PM +0200, Maxime Ripard wrote:
> Hi,
> 
> I've been trying to fetch an mbox from lore with an arbitrary search request.
> 
> I could fetch it fine using curl with the following example:
> 
> curl -XPOST -H "Content-Length:0" -OJ "http://lore.kernel.org/linux-clk/?q=d:1.week.ago..&x=m"
> 
> This returns a gzip'd mbox, everything's fine.

Note that when I do this I get redirected to the https URL and when I
use it, then it works.

> def try_url_redirect(url):
>     headers={"Content-Length": "0"}
>     params={"q": "d:1.week.ago..", "x": "m"}
> 
>     s = Session()
> 
>     req = Request('POST', url, headers=headers, params=params)
>     p = req.prepare()

I don't know about this part in python, but are you certain that
it's not trying to pass the params in the request body ? It would
seem natural to me since you've asked for a POST. In your curl
request, you're not sending arguments as part of the body but as
a query string with an empty body.

It would be useful to strace the output (use http:// to make it
easier) to verify, because I really don't trust the debugging
output which possibly just reassembles the URL as if it were a
GET except that it's not that. With curl (in HTTP) it's what the
request says at least:

   20:43:22.246844 sendto(5, "POST /linux-clk/?q=d:1.week.ago..&x=m HTTP/1.1\r\nHost: lore.kernel.org\r\nUser-Agent: curl/7.81.0\r\nAccept: */*\r\nContent-Length:0\r\n\r\n", 129, MSG_NOSIGNAL, NULL, 0) = 129

You should try to concatenate your arguments just at the end of
the URL and really send nothing with the POST.

Hoping this helps a little bit,
Willy