* [PULL v2 00/44] Python patches
@ 2021-06-01 23:31 John Snow
2021-06-01 23:31 ` [PULL v2 04/44] Python: add utility function for retrieving port redirection John Snow
2021-06-02 16:07 ` [PULL v2 00/44] Python patches Peter Maydell
0 siblings, 2 replies; 4+ messages in thread
From: John Snow @ 2021-06-01 23:31 UTC (permalink / raw)
To: qemu-devel
Cc: Kevin Wolf, Peter Maydell, Thomas Huth, Eduardo Habkost,
qemu-block, Philippe Mathieu-Daudé,
Philippe Mathieu-Daudé, Wainer dos Santos Moschetta,
Max Reitz, Willian Rampazzo, John Snow, Cleber Rosa,
Alex Bennée, Aurelien Jarno
The following changes since commit 52848929b70dcf92a68aedcfd90207be81ba3274:
Merge remote-tracking branch 'remotes/kraxel/tags/usb-20210528-pull-request' into staging (2021-05-30 20:10:30 +0100)
are available in the Git repository at:
https://gitlab.com/jsnow/qemu.git tags/python-pull-request
for you to fetch changes up to 6b9c277797879ce41ed20deb6737f4156cc279b3:
gitlab: add python linters to CI (2021-06-01 16:21:21 -0400)
----------------------------------------------------------------
Pull request
V2:
- Squashed in fixup for
'Python: add utility function for retrieving port redirection'
- Rebased on today's upstream
CI here:
https://gitlab.com/jsnow/qemu/-/pipelines/313202814
----------------------------------------------------------------
Cleber Rosa (12):
Python: expose QEMUMachine's temporary directory
tests/acceptance/virtiofs_submounts.py: add missing accel tag
tests/acceptance/virtiofs_submounts.py: evaluate string not length
Python: add utility function for retrieving port redirection
Acceptance Tests: move useful ssh methods to base class
Acceptance Tests: add port redirection for ssh by default
Acceptance Tests: make username/password configurable
Acceptance Tests: set up SSH connection by default after boot for
LinuxTest
tests/acceptance/virtiofs_submounts.py: remove launch_vm()
Acceptance Tests: add basic documentation on LinuxTest base class
Acceptance Tests: introduce CPU hotplug test
tests/acceptance/virtiofs_submounts.py: fix setup of SSH pubkey
John Snow (31):
python/console_socket: avoid one-letter variable
python/machine: use subprocess.DEVNULL instead of
open(os.path.devnull)
python/machine: use subprocess.run instead of subprocess.Popen
python/console_socket: Add a pylint ignore
python/machine: Disable pylint warning for open() in _pre_launch
python/machine: disable warning for Popen in _launch()
python/machine: Trim line length to below 80 chars
iotests/297: add --namespace-packages to mypy arguments
python: create qemu packages
python: add qemu package installer
python: add VERSION file
python: add directory structure README.rst files
python: add MANIFEST.in
python: Add pipenv support
python: add pylint import exceptions
python: move pylintrc into setup.cfg
python: add pylint to pipenv
python: move flake8 config to setup.cfg
python: add excluded dirs to flake8 config
python: Add flake8 to pipenv
python: move mypy.ini into setup.cfg
python: add mypy to pipenv
python: move .isort.cfg into setup.cfg
python/qemu: add isort to pipenv
python/qemu: add qemu package itself to pipenv
python: add devel package requirements to setuptools
python: add avocado-framework and tests
python: add Makefile for some common tasks
python: add .gitignore
python: add tox support
gitlab: add python linters to CI
Willian Rampazzo (1):
acceptance tests: bump Avocado version to 88.1
docs/devel/testing.rst | 26 +++
python/PACKAGE.rst | 43 ++++
python/README.rst | 58 +++++
python/qemu/README.rst | 8 +
python/qemu/machine/README.rst | 9 +
python/qemu/qmp/README.rst | 9 +
python/qemu/utils/README.rst | 7 +
.gitlab-ci.d/containers.yml | 5 +
.gitlab-ci.d/static_checks.yml | 21 ++
python/.gitignore | 16 ++
python/MANIFEST.in | 3 +
python/Makefile | 48 ++++
python/Pipfile | 13 ++
python/Pipfile.lock | 231 ++++++++++++++++++++
python/VERSION | 1 +
python/avocado.cfg | 10 +
python/mypy.ini | 4 -
python/qemu/.flake8 | 2 -
python/qemu/.isort.cfg | 7 -
python/qemu/__init__.py | 11 -
python/qemu/machine/__init__.py | 36 +++
python/qemu/{ => machine}/console_socket.py | 11 +-
python/qemu/{ => machine}/machine.py | 68 ++++--
python/qemu/{ => machine}/qtest.py | 9 +-
python/qemu/pylintrc | 58 -----
python/qemu/{qmp.py => qmp/__init__.py} | 12 +-
python/qemu/utils/__init__.py | 45 ++++
python/qemu/{ => utils}/accel.py | 0
python/setup.cfg | 102 +++++++++
python/setup.py | 23 ++
python/tests/flake8.sh | 2 +
python/tests/isort.sh | 2 +
python/tests/mypy.sh | 2 +
python/tests/pylint.sh | 2 +
tests/acceptance/avocado_qemu/__init__.py | 69 +++++-
tests/acceptance/boot_linux.py | 18 +-
tests/acceptance/hotplug_cpu.py | 37 ++++
tests/acceptance/info_usernet.py | 29 +++
tests/acceptance/linux_ssh_mips_malta.py | 42 +---
tests/acceptance/virtio-gpu.py | 2 +-
tests/acceptance/virtiofs_submounts.py | 71 +-----
tests/docker/dockerfiles/python.docker | 18 ++
tests/qemu-iotests/297 | 1 +
tests/qemu-iotests/300 | 4 +-
tests/qemu-iotests/iotests.py | 4 +-
tests/requirements.txt | 2 +-
tests/vm/aarch64vm.py | 2 +-
tests/vm/basevm.py | 12 +-
48 files changed, 967 insertions(+), 248 deletions(-)
create mode 100644 python/PACKAGE.rst
create mode 100644 python/README.rst
create mode 100644 python/qemu/README.rst
create mode 100644 python/qemu/machine/README.rst
create mode 100644 python/qemu/qmp/README.rst
create mode 100644 python/qemu/utils/README.rst
create mode 100644 python/.gitignore
create mode 100644 python/MANIFEST.in
create mode 100644 python/Makefile
create mode 100644 python/Pipfile
create mode 100644 python/Pipfile.lock
create mode 100644 python/VERSION
create mode 100644 python/avocado.cfg
delete mode 100644 python/mypy.ini
delete mode 100644 python/qemu/.flake8
delete mode 100644 python/qemu/.isort.cfg
delete mode 100644 python/qemu/__init__.py
create mode 100644 python/qemu/machine/__init__.py
rename python/qemu/{ => machine}/console_socket.py (94%)
rename python/qemu/{ => machine}/machine.py (93%)
rename python/qemu/{ => machine}/qtest.py (95%)
delete mode 100644 python/qemu/pylintrc
rename python/qemu/{qmp.py => qmp/__init__.py} (96%)
create mode 100644 python/qemu/utils/__init__.py
rename python/qemu/{ => utils}/accel.py (100%)
create mode 100644 python/setup.cfg
create mode 100755 python/setup.py
create mode 100755 python/tests/flake8.sh
create mode 100755 python/tests/isort.sh
create mode 100755 python/tests/mypy.sh
create mode 100755 python/tests/pylint.sh
create mode 100644 tests/acceptance/hotplug_cpu.py
create mode 100644 tests/acceptance/info_usernet.py
create mode 100644 tests/docker/dockerfiles/python.docker
--
2.31.1
^ permalink raw reply [flat|nested] 4+ messages in thread* [PULL v2 04/44] Python: add utility function for retrieving port redirection 2021-06-01 23:31 [PULL v2 00/44] Python patches John Snow @ 2021-06-01 23:31 ` John Snow 2021-06-02 16:07 ` [PULL v2 00/44] Python patches Peter Maydell 1 sibling, 0 replies; 4+ messages in thread From: John Snow @ 2021-06-01 23:31 UTC (permalink / raw) To: qemu-devel Cc: Kevin Wolf, Peter Maydell, Thomas Huth, Eduardo Habkost, qemu-block, Philippe Mathieu-Daudé, Philippe Mathieu-Daudé, Wainer dos Santos Moschetta, Max Reitz, Willian Rampazzo, Eric Auger, John Snow, Cleber Rosa, Alex Bennée, Aurelien Jarno From: Cleber Rosa <crosa@redhat.com> Slightly different versions for the same utility code are currently present on different locations. This unifies them all, giving preference to the version from virtiofs_submounts.py, because of the last tweaks added to it. While at it, this adds a "qemu.utils" module to host the utility function and a test. Signed-off-by: Cleber Rosa <crosa@redhat.com> Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com> Reviewed-by: Eric Auger <eric.auger@redhat.com> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Message-Id: <20210412044644.55083-4-crosa@redhat.com> Signed-off-by: John Snow <jsnow@redhat.com> [Squashed in below fix. --js] Signed-off-by: John Snow <jsnow@redhat.com> Signed-off-by: Cleber Rosa <crosa@redhat.com> Message-Id: <20210601154546.130870-2-crosa@redhat.com> Signed-off-by: John Snow <jsnow@redhat.com> --- python/qemu/utils.py | 33 ++++++++++++++++++++++++ tests/acceptance/info_usernet.py | 29 +++++++++++++++++++++ tests/acceptance/linux_ssh_mips_malta.py | 16 +++++------- tests/acceptance/virtiofs_submounts.py | 21 ++++----------- tests/vm/basevm.py | 11 +++----- 5 files changed, 78 insertions(+), 32 deletions(-) create mode 100644 python/qemu/utils.py create mode 100644 tests/acceptance/info_usernet.py diff --git a/python/qemu/utils.py b/python/qemu/utils.py new file mode 100644 index 00000000000..5ed789275ee --- /dev/null +++ b/python/qemu/utils.py @@ -0,0 +1,33 @@ +""" +QEMU utility library + +This offers miscellaneous utility functions, which may not be easily +distinguishable or numerous to be in their own module. +""" + +# Copyright (C) 2021 Red Hat Inc. +# +# Authors: +# Cleber Rosa <crosa@redhat.com> +# +# This work is licensed under the terms of the GNU GPL, version 2. See +# the COPYING file in the top-level directory. +# + +import re +from typing import Optional + + +def get_info_usernet_hostfwd_port(info_usernet_output: str) -> Optional[int]: + """ + Returns the port given to the hostfwd parameter via info usernet + + :param info_usernet_output: output generated by hmp command "info usernet" + :return: the port number allocated by the hostfwd option + """ + for line in info_usernet_output.split('\r\n'): + regex = r'TCP.HOST_FORWARD.*127\.0\.0\.1\s+(\d+)\s+10\.' + match = re.search(regex, line) + if match is not None: + return int(match[1]) + return None diff --git a/tests/acceptance/info_usernet.py b/tests/acceptance/info_usernet.py new file mode 100644 index 00000000000..9c1fd903a0b --- /dev/null +++ b/tests/acceptance/info_usernet.py @@ -0,0 +1,29 @@ +# Test for the hmp command "info usernet" +# +# Copyright (c) 2021 Red Hat, Inc. +# +# Author: +# Cleber Rosa <crosa@redhat.com> +# +# This work is licensed under the terms of the GNU GPL, version 2 or +# later. See the COPYING file in the top-level directory. + +from avocado_qemu import Test + +from qemu.utils import get_info_usernet_hostfwd_port + + +class InfoUsernet(Test): + + def test_hostfwd(self): + self.vm.add_args('-netdev', 'user,id=vnet,hostfwd=:127.0.0.1:0-:22') + self.vm.launch() + res = self.vm.command('human-monitor-command', + command_line='info usernet') + port = get_info_usernet_hostfwd_port(res) + self.assertIsNotNone(port, + ('"info usernet" output content does not seem to ' + 'contain the redirected port')) + self.assertGreater(port, 0, + ('Found a redirected port that is not greater than' + ' zero')) diff --git a/tests/acceptance/linux_ssh_mips_malta.py b/tests/acceptance/linux_ssh_mips_malta.py index 6dbd02d49d5..052008f02d4 100644 --- a/tests/acceptance/linux_ssh_mips_malta.py +++ b/tests/acceptance/linux_ssh_mips_malta.py @@ -18,6 +18,8 @@ from avocado.utils import archive from avocado.utils import ssh +from qemu.utils import get_info_usernet_hostfwd_port + class LinuxSSH(Test): @@ -70,18 +72,14 @@ def get_kernel_info(self, endianess, wordsize): def setUp(self): super(LinuxSSH, self).setUp() - def get_portfwd(self): + def ssh_connect(self, username, password): + self.ssh_logger = logging.getLogger('ssh') res = self.vm.command('human-monitor-command', command_line='info usernet') - line = res.split('\r\n')[2] - port = re.split(r'.*TCP.HOST_FORWARD.*127\.0\.0\.1 (\d+)\s+10\..*', - line)[1] + port = get_info_usernet_hostfwd_port(res) + if not port: + self.cancel("Failed to retrieve SSH port") self.log.debug("sshd listening on port:" + port) - return port - - def ssh_connect(self, username, password): - self.ssh_logger = logging.getLogger('ssh') - port = self.get_portfwd() self.ssh_session = ssh.Session(self.VM_IP, port=int(port), user=username, password=password) for i in range(10): diff --git a/tests/acceptance/virtiofs_submounts.py b/tests/acceptance/virtiofs_submounts.py index ca64b76301f..57a7047342f 100644 --- a/tests/acceptance/virtiofs_submounts.py +++ b/tests/acceptance/virtiofs_submounts.py @@ -9,6 +9,8 @@ from avocado_qemu import wait_for_console_pattern from avocado.utils import ssh +from qemu.utils import get_info_usernet_hostfwd_port + def run_cmd(args): subp = subprocess.Popen(args, @@ -73,27 +75,14 @@ class VirtiofsSubmountsTest(LinuxTest): :avocado: tags=accel:kvm """ - def get_portfwd(self): - port = None - + def ssh_connect(self, username, keyfile): + self.ssh_logger = logging.getLogger('ssh') res = self.vm.command('human-monitor-command', command_line='info usernet') - for line in res.split('\r\n'): - match = \ - re.search(r'TCP.HOST_FORWARD.*127\.0\.0\.1\s+(\d+)\s+10\.', - line) - if match is not None: - port = int(match[1]) - break - + port = get_info_usernet_hostfwd_port(res) self.assertIsNotNone(port) self.assertGreater(port, 0) self.log.debug('sshd listening on port: %d', port) - return port - - def ssh_connect(self, username, keyfile): - self.ssh_logger = logging.getLogger('ssh') - port = self.get_portfwd() self.ssh_session = ssh.Session('127.0.0.1', port=port, user=username, key=keyfile) for i in range(10): diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py index 00f1d5ca8da..995e642465b 100644 --- a/tests/vm/basevm.py +++ b/tests/vm/basevm.py @@ -21,6 +21,7 @@ sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python')) from qemu.accel import kvm_available from qemu.machine import QEMUMachine +from qemu.utils import get_info_usernet_hostfwd_port import subprocess import hashlib import argparse @@ -227,7 +228,7 @@ def _ssh_do(self, user, cmd, check): "-o", "UserKnownHostsFile=" + os.devnull, "-o", "ConnectTimeout={}".format(self._config["ssh_timeout"]), - "-p", self.ssh_port, "-i", self._ssh_tmp_key_file] + "-p", str(self.ssh_port), "-i", self._ssh_tmp_key_file] # If not in debug mode, set ssh to quiet mode to # avoid printing the results of commands. if not self.debug: @@ -305,12 +306,8 @@ def boot(self, img, extra_args=[]): # Init console so we can start consuming the chars. self.console_init() usernet_info = guest.qmp("human-monitor-command", - command_line="info usernet") - self.ssh_port = None - for l in usernet_info["return"].splitlines(): - fields = l.split() - if "TCP[HOST_FORWARD]" in fields and "22" in fields: - self.ssh_port = l.split()[3] + command_line="info usernet").get("return") + self.ssh_port = get_info_usernet_hostfwd_port(usernet_info) if not self.ssh_port: raise Exception("Cannot find ssh port from 'info usernet':\n%s" % \ usernet_info) -- 2.31.1 ^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [PULL v2 00/44] Python patches 2021-06-01 23:31 [PULL v2 00/44] Python patches John Snow 2021-06-01 23:31 ` [PULL v2 04/44] Python: add utility function for retrieving port redirection John Snow @ 2021-06-02 16:07 ` Peter Maydell 2021-06-02 16:27 ` John Snow 1 sibling, 1 reply; 4+ messages in thread From: Peter Maydell @ 2021-06-02 16:07 UTC (permalink / raw) To: John Snow Cc: Kevin Wolf, Thomas Huth, Eduardo Habkost, Qemu-block, Philippe Mathieu-Daudé, QEMU Developers, Wainer dos Santos Moschetta, Max Reitz, Willian Rampazzo, Cleber Rosa, Alex Bennée, Aurelien Jarno, Philippe Mathieu-Daudé On Wed, 2 Jun 2021 at 00:31, John Snow <jsnow@redhat.com> wrote: > > The following changes since commit 52848929b70dcf92a68aedcfd90207be81ba3274: > > Merge remote-tracking branch 'remotes/kraxel/tags/usb-20210528-pull-request' into staging (2021-05-30 20:10:30 +0100) > > are available in the Git repository at: > > https://gitlab.com/jsnow/qemu.git tags/python-pull-request > > for you to fetch changes up to 6b9c277797879ce41ed20deb6737f4156cc279b3: > > gitlab: add python linters to CI (2021-06-01 16:21:21 -0400) > > ---------------------------------------------------------------- > Pull request > > V2: > - Squashed in fixup for > 'Python: add utility function for retrieving port redirection' > - Rebased on today's upstream > > CI here: > https://gitlab.com/jsnow/qemu/-/pipelines/313202814 > Applied, thanks. Please update the changelog at https://wiki.qemu.org/ChangeLog/6.1 for any user-visible changes. -- PMM ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PULL v2 00/44] Python patches 2021-06-02 16:07 ` [PULL v2 00/44] Python patches Peter Maydell @ 2021-06-02 16:27 ` John Snow 0 siblings, 0 replies; 4+ messages in thread From: John Snow @ 2021-06-02 16:27 UTC (permalink / raw) To: Peter Maydell Cc: Kevin Wolf, Thomas Huth, Eduardo Habkost, Qemu-block, Philippe Mathieu-Daudé, QEMU Developers, Wainer dos Santos Moschetta, Max Reitz, Willian Rampazzo, Cleber Rosa, Alex Bennée, Aurelien Jarno, Philippe Mathieu-Daudé On 6/2/21 12:07 PM, Peter Maydell wrote: > On Wed, 2 Jun 2021 at 00:31, John Snow <jsnow@redhat.com> wrote: >> >> The following changes since commit 52848929b70dcf92a68aedcfd90207be81ba3274: >> >> Merge remote-tracking branch 'remotes/kraxel/tags/usb-20210528-pull-request' into staging (2021-05-30 20:10:30 +0100) >> >> are available in the Git repository at: >> >> https://gitlab.com/jsnow/qemu.git tags/python-pull-request >> >> for you to fetch changes up to 6b9c277797879ce41ed20deb6737f4156cc279b3: >> >> gitlab: add python linters to CI (2021-06-01 16:21:21 -0400) >> >> ---------------------------------------------------------------- >> Pull request >> >> V2: >> - Squashed in fixup for >> 'Python: add utility function for retrieving port redirection' >> - Rebased on today's upstream >> >> CI here: >> https://gitlab.com/jsnow/qemu/-/pipelines/313202814 >> > > > Applied, thanks. > > Please update the changelog at https://wiki.qemu.org/ChangeLog/6.1 > for any user-visible changes. > > -- PMM > Yay!!! Shouldn't be any user-visible changes yet, but there are some developer-visible ones. (The new CI tests now protecting our python code, chiefly.) I don't think it's appropriate matter for the changelog. Thank you! 🎉 ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2021-06-02 16:29 UTC | newest] Thread overview: 4+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2021-06-01 23:31 [PULL v2 00/44] Python patches John Snow 2021-06-01 23:31 ` [PULL v2 04/44] Python: add utility function for retrieving port redirection John Snow 2021-06-02 16:07 ` [PULL v2 00/44] Python patches Peter Maydell 2021-06-02 16:27 ` John Snow
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).