Implementing "tee" in python asyncio

Implementing "tee" in python asyncio

[John Snow]

Hi folks,

I'm currently wondering how to take a StreamReader as found on
https://docs.python.org/3/library/asyncio-subprocess.html#asyncio.subprocess.Process
and to consume the data while optionally re-streaming it to a
secondary consumer.

What I'd like to do is create a StreamWatcher class that consumes
console data while optionally logging to python logging and/or a file;
but re-buffers the data into an async stream where an additional
consumer is free to use the "standard asyncio API" to consume console
data at their leisure in a way that's unsurprising.

What I'd like this *for* is to be able to do aggressive logging of
stdout/stderr and console data without denying tests the ability to
consume the data as they see fit for their testing purposes. I want to
have my cake and eat it too, and we don't do a good job of managing
this consistently across the board.

I am wondering if there's any way around creating a literal socketpair
and suffering the creation of a full four StreamReader/StreamWriter
instances (one pair per socket...) and then just hanging on to the
"unused" reader/writer per each. It seems kind of foolishly excessive.
It also seems like it might be a pain in the butt if I want
cross-platform compatibility with windows for the machine appliance.

Anyone got any bright ideas?

--js

Re: Implementing "tee" in python asyncio

[Daniel P. Berrangé]

On Wed, Jul 26, 2023 at 04:25:34PM -0400, John Snow wrote:
> Hi folks,
> 
> I'm currently wondering how to take a StreamReader as found on
> https://docs.python.org/3/library/asyncio-subprocess.html#asyncio.subprocess.Process
> and to consume the data while optionally re-streaming it to a
> secondary consumer.
> 
> What I'd like to do is create a StreamWatcher class that consumes
> console data while optionally logging to python logging and/or a file;
> but re-buffers the data into an async stream where an additional
> consumer is free to use the "standard asyncio API" to consume console
> data at their leisure in a way that's unsurprising.
> 
> What I'd like this *for* is to be able to do aggressive logging of
> stdout/stderr and console data without denying tests the ability to
> consume the data as they see fit for their testing purposes. I want to
> have my cake and eat it too, and we don't do a good job of managing
> this consistently across the board.
> 
> I am wondering if there's any way around creating a literal socketpair
> and suffering the creation of a full four StreamReader/StreamWriter
> instances (one pair per socket...) and then just hanging on to the
> "unused" reader/writer per each. It seems kind of foolishly excessive.
> It also seems like it might be a pain in the butt if I want
> cross-platform compatibility with windows for the machine appliance.
> 
> Anyone got any bright ideas?

Don't bother with any of the above, just add 'logfile=/path/to/log'
to the -chardev argument.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Implementing "tee" in python asyncio

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 2302 bytes --]

On Thu, Jul 27, 2023, 6:47 AM Daniel P. Berrangé <berrange@redhat.com>
wrote:

> On Wed, Jul 26, 2023 at 04:25:34PM -0400, John Snow wrote:
> > Hi folks,
> >
> > I'm currently wondering how to take a StreamReader as found on
> >
> https://docs.python.org/3/library/asyncio-subprocess.html#asyncio.subprocess.Process
> > and to consume the data while optionally re-streaming it to a
> > secondary consumer.
> >
> > What I'd like to do is create a StreamWatcher class that consumes
> > console data while optionally logging to python logging and/or a file;
> > but re-buffers the data into an async stream where an additional
> > consumer is free to use the "standard asyncio API" to consume console
> > data at their leisure in a way that's unsurprising.
> >
> > What I'd like this *for* is to be able to do aggressive logging of
> > stdout/stderr and console data without denying tests the ability to
> > consume the data as they see fit for their testing purposes. I want to
> > have my cake and eat it too, and we don't do a good job of managing
> > this consistently across the board.
> >
> > I am wondering if there's any way around creating a literal socketpair
> > and suffering the creation of a full four StreamReader/StreamWriter
> > instances (one pair per socket...) and then just hanging on to the
> > "unused" reader/writer per each. It seems kind of foolishly excessive.
> > It also seems like it might be a pain in the butt if I want
> > cross-platform compatibility with windows for the machine appliance.
> >
> > Anyone got any bright ideas?
>
> Don't bother with any of the above, just add 'logfile=/path/to/log'
> to the -chardev argument.
>

Part of me feels like it'd be a shame to miss out on the chance to do the
logging in the library, but it *would* be a lot easier to just not worry
about it.

>
anyway, this is easy for console, what about stdio? is there a way to run
that through a chardev that's just connected to stdio but we attach a
logfile?


> With regards,
> Daniel
> --
> |: https://berrange.com      -o-
> https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-
> https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-
> https://www.instagram.com/dberrange :|
>
>

[-- Attachment #2: Type: text/html, Size: 3937 bytes --]

Re: Implementing "tee" in python asyncio

[Daniel P. Berrangé]

On Thu, Jul 27, 2023 at 12:49:10PM -0400, John Snow wrote:
> On Thu, Jul 27, 2023, 6:47 AM Daniel P. Berrangé <berrange@redhat.com>
> wrote:
> 
> > On Wed, Jul 26, 2023 at 04:25:34PM -0400, John Snow wrote:
> > > Hi folks,
> > >
> > > I'm currently wondering how to take a StreamReader as found on
> > >
> > https://docs.python.org/3/library/asyncio-subprocess.html#asyncio.subprocess.Process
> > > and to consume the data while optionally re-streaming it to a
> > > secondary consumer.
> > >
> > > What I'd like to do is create a StreamWatcher class that consumes
> > > console data while optionally logging to python logging and/or a file;
> > > but re-buffers the data into an async stream where an additional
> > > consumer is free to use the "standard asyncio API" to consume console
> > > data at their leisure in a way that's unsurprising.
> > >
> > > What I'd like this *for* is to be able to do aggressive logging of
> > > stdout/stderr and console data without denying tests the ability to
> > > consume the data as they see fit for their testing purposes. I want to
> > > have my cake and eat it too, and we don't do a good job of managing
> > > this consistently across the board.
> > >
> > > I am wondering if there's any way around creating a literal socketpair
> > > and suffering the creation of a full four StreamReader/StreamWriter
> > > instances (one pair per socket...) and then just hanging on to the
> > > "unused" reader/writer per each. It seems kind of foolishly excessive.
> > > It also seems like it might be a pain in the butt if I want
> > > cross-platform compatibility with windows for the machine appliance.
> > >
> > > Anyone got any bright ideas?
> >
> > Don't bother with any of the above, just add 'logfile=/path/to/log'
> > to the -chardev argument.
> >
> 
> Part of me feels like it'd be a shame to miss out on the chance to do the
> logging in the library, but it *would* be a lot easier to just not worry
> about it.
> 
> anyway, this is easy for console, what about stdio? is there a way to run
> that through a chardev that's just connected to stdio but we attach a
> logfile?

When you say 'stdio' are you refering to a '-chardev stdio' backend, or
simply whatever the python code dup()s onto stdio when spawnnig QEMU ?
The former case can use ',logfile=...', and in the latter case we could
open a log file and dup() its FD onto stdout/stderr when spawning QEMU.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Implementing "tee" in python asyncio

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 3161 bytes --]

On Thu, Jul 27, 2023, 12:53 PM Daniel P. Berrangé <berrange@redhat.com>
wrote:

> On Thu, Jul 27, 2023 at 12:49:10PM -0400, John Snow wrote:
> > On Thu, Jul 27, 2023, 6:47 AM Daniel P. Berrangé <berrange@redhat.com>
> > wrote:
> >
> > > On Wed, Jul 26, 2023 at 04:25:34PM -0400, John Snow wrote:
> > > > Hi folks,
> > > >
> > > > I'm currently wondering how to take a StreamReader as found on
> > > >
> > >
> https://docs.python.org/3/library/asyncio-subprocess.html#asyncio.subprocess.Process
> > > > and to consume the data while optionally re-streaming it to a
> > > > secondary consumer.
> > > >
> > > > What I'd like to do is create a StreamWatcher class that consumes
> > > > console data while optionally logging to python logging and/or a
> file;
> > > > but re-buffers the data into an async stream where an additional
> > > > consumer is free to use the "standard asyncio API" to consume console
> > > > data at their leisure in a way that's unsurprising.
> > > >
> > > > What I'd like this *for* is to be able to do aggressive logging of
> > > > stdout/stderr and console data without denying tests the ability to
> > > > consume the data as they see fit for their testing purposes. I want
> to
> > > > have my cake and eat it too, and we don't do a good job of managing
> > > > this consistently across the board.
> > > >
> > > > I am wondering if there's any way around creating a literal
> socketpair
> > > > and suffering the creation of a full four StreamReader/StreamWriter
> > > > instances (one pair per socket...) and then just hanging on to the
> > > > "unused" reader/writer per each. It seems kind of foolishly
> excessive.
> > > > It also seems like it might be a pain in the butt if I want
> > > > cross-platform compatibility with windows for the machine appliance.
> > > >
> > > > Anyone got any bright ideas?
> > >
> > > Don't bother with any of the above, just add 'logfile=/path/to/log'
> > > to the -chardev argument.
> > >
> >
> > Part of me feels like it'd be a shame to miss out on the chance to do the
> > logging in the library, but it *would* be a lot easier to just not worry
> > about it.
> >
> > anyway, this is easy for console, what about stdio? is there a way to run
> > that through a chardev that's just connected to stdio but we attach a
> > logfile?
>
> When you say 'stdio' are you refering to a '-chardev stdio' backend, or
> simply whatever the python code dup()s onto stdio when spawnnig QEMU ?
> The former case can use ',logfile=...', and in the latter case we could
> open a log file and dup() its FD onto stdout/stderr when spawning QEMU.
>

I mean stdout and stderr for the QEMU process itself. Some tests like to
watch the output for various strings, but we also need to capture and log
that output to be able to show it during unhandled exceptions, etc.


>
> With regards,
> Daniel
> --
> |: https://berrange.com      -o-
> https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-
> https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-
> https://www.instagram.com/dberrange :|
>
>

[-- Attachment #2: Type: text/html, Size: 4949 bytes --]