Upstream bug comments on the mailing list and Launchpad

Upstream bug comments on the mailing list and Launchpad

[Kashyap Chamarthy]

Hi, folks —

Many here know that upstream bugs from launchpad get replicated to
'qemu-devel'.  But the catch is that it only happens if you _also_ have
an account on Launchpad with the same email address with which you're
replying on the list.  This is easy to forget.

So a QEMU developer without a Launchpad account might give a detailed
response to a bug on the list.  But the bug reporter isn't subscribed to
the list (which is fine; and is expected behaviour).

    <... a few weeks later ... >

The bug reporter thinks: "no one responded to my bug".  Meanwhile, the
detailed comment from the developer is languishing on the list.

PS: I recall there was discussion on the list of moving to a different
    GitLab tracker.  As Thomas Huth mentioned on IRC, more people seem
    to have account on GitLab than on Launchpad.

-- 
/kashyap

Re: Upstream bug comments on the mailing list and Launchpad

[Thomas Huth]

On 15/04/2021 11.49, Kashyap Chamarthy wrote:
...
> PS: I recall there was discussion on the list of moving to a different
>      GitLab tracker.  As Thomas Huth mentioned on IRC, more people seem
>      to have account on GitLab than on Launchpad.

I think we basically agreed that we want to migrate to gitlab, but e.g. 
Peter suggested that we copy the open bugs on launchpad to the gitlab 
tracker first, so that we don't have two active trackers at the same time... 
thus this needs someone with enough spare time to work on such a convertion 
script first...

See this discussion here for some more details:

 
https://lore.kernel.org/qemu-devel/CAFEAcA_8PKkfeninOXCzPdtY7WVHnC7Pkon758zXe7h9MzS+aw@mail.gmail.com/

  Thomas

Re: Upstream bug comments on the mailing list and Launchpad

[Peter Maydell]

On Mon, 19 Apr 2021 at 16:52, Thomas Huth <thuth@redhat.com> wrote:
>
> On 15/04/2021 11.49, Kashyap Chamarthy wrote:
> ...
> > PS: I recall there was discussion on the list of moving to a different
> >      GitLab tracker.  As Thomas Huth mentioned on IRC, more people seem
> >      to have account on GitLab than on Launchpad.
>
> I think we basically agreed that we want to migrate to gitlab, but e.g.
> Peter suggested that we copy the open bugs on launchpad to the gitlab
> tracker first, so that we don't have two active trackers at the same time...
> thus this needs someone with enough spare time to work on such a convertion
> script first...

In the meantime, can we disable bug reporting on gitlab? It's
confusing to users to have two places, one of which is not
checked by anybody. We already have two stray bugreports:
https://gitlab.com/qemu-project/qemu/-/issues

-- PMM

Transferring bugs from Launchpad to Gitlab (was: Re: Upstream bug comments on the mailing list and Launchpad)

[Thomas Huth]

[-- Attachment #1: Type: text/plain, Size: 3171 bytes --]

On 19/04/2021 18.04, Peter Maydell wrote:
> On Mon, 19 Apr 2021 at 16:52, Thomas Huth <thuth@redhat.com> wrote:
>>
>> On 15/04/2021 11.49, Kashyap Chamarthy wrote:
>> ...
>>> PS: I recall there was discussion on the list of moving to a different
>>>       GitLab tracker.  As Thomas Huth mentioned on IRC, more people seem
>>>       to have account on GitLab than on Launchpad.
>>
>> I think we basically agreed that we want to migrate to gitlab, but e.g.
>> Peter suggested that we copy the open bugs on launchpad to the gitlab
>> tracker first, so that we don't have two active trackers at the same time...
>> thus this needs someone with enough spare time to work on such a convertion
>> script first...
> 
> In the meantime, can we disable bug reporting on gitlab? It's
> confusing to users to have two places, one of which is not
> checked by anybody. We already have two stray bugreports:
> https://gitlab.com/qemu-project/qemu/-/issues

I'd rather like to see us doing the move finally. I'm very much a Python 
ignorant, but since apparently nobody else has spare time to work on this at 
the moment, I've now done some copy-n-paste coding and came up with a script 
that can transfer bugs from Launchpad to Gitlab (see the attachment). You 
need a "private token" passed to the script via the GITLAB_PRIVATE_TOKEN 
environment variable so that it is able to access gitlab (such a token can 
be generated in your private setting pages on gitlab). Some transferred bugs 
can be seen here:

  https://gitlab.com/thuth/trackertest/-/issues

Notes:

- Not all information is transferred, e.g. no attachments yet.
   But considering that there is a link to the original launchpad
   bug, you can always look up the information there, so I think
   that should be fine.

- The script currently also adds the "tags" from the launchpad
   bug as "labels" in gitlab - do we want that, or do we rather
   want to restrict our labels in gitlab to a smaller set?

- Formatting of the bug description and comments is a little bit
   tricky, since the text is pre-formatted ASCII in launchpad and
   markdown on gitlab ... For now, I put the text from launchpad
   into "<pre>" HTML tags on gitlab, that should result in usable
   texts most of the time.

- Biggest problem so far: After transferring 50 bugs or so,
   gitlab refuses to take more bugs automatically with the
   message: 'Your issue has been recognized as spam. Please,
   change the content or solve the reCAPTCHA to proceed.'
   ... and solving captchas in a script is certainly way out
   of scope here. So if we want to copy over all bugs from
   launchpad, we might need to do that in junks.

- Do we want to auto-close the bugs in launchpad after we've
   transferred them? Or should they stay open until the bugs
   have been addressed? I tend to close them (with an appropriate
   comment added to the bug, too), but we could also mark the
   transferred bugs with a tag instead.

- Do we want to transfer also old "Wishlist" bug tickets from
   Launchpad? I think it's quite unlikely that we will ever
   address them... maybe we should close them on Launchpad with
   an appropriate comment?

  Thomas

[-- Attachment #2: launchpad2gitlab.py --]
[-- Type: text/x-python, Size: 3758 bytes --]

#!/usr/bin/env python3

import argparse
import logging
import os
import re
import sys
import time
import gitlab

from launchpadlib.launchpad import Launchpad
import lazr.restfulclient.errors

LOG_FORMAT = "%(levelname)s - %(funcName)s - %(message)s"
logging.basicConfig(format=LOG_FORMAT)
LOG = logging.getLogger(__name__)

parser = argparse.ArgumentParser(description="Close incomplete bug reports.")
parser.add_argument('-l',
                    '--lp-project-name',
                    dest='lp_project_name',
                    default='qemu',
                    help='The Launchpad project name (qemu).')
parser.add_argument('-g',
                    '--gl-project-id',
                    dest='gl_project_id',
                    help='The Gitlab project ID.')
parser.add_argument('--verbose', '-v',
                    help='Enable debug logging.',
                    action="store_true")
parser.add_argument('--search-text', '-s',
                    dest='search_text',
                    help='Look up bugs by searching for text.')
args = parser.parse_args()
LOG.info(args)

LOG.setLevel(logging.DEBUG if args.verbose else logging.INFO)


def get_launchpad():
	cache_dir = os.path.expanduser("~/.launchpadlib/cache/")
	if not os.path.exists(cache_dir):
		os.makedirs(cache_dir, 0o700)
	launchpad = Launchpad.login_anonymously(args.lp_project_name + '-bugs',
	                                        'production', cache_dir)
	return launchpad


def show_bug_task(bug_task):
	print('Bug #%d - %s' % (bug_task.bug.id, str(bug_task.bug.title)))
	if args.verbose:
		print(' - Description: %s' % bug_task.bug.description)
		print(' - Tags: %s' % bug_task.bug.tags)
		print(' - Status: %s' % bug_task.status)
		print(' - Assignee: %s' % bug_task.assignee)
		for message in bug_task.bug.messages:
			print('  - Message: %s' % message.content)


def transfer_to_gitlab(launchpad, project, bug_task):
	bug = bug_task.bug
	desc = "This bug has been copied automatically from: " + bug_task.web_link + \
		"\n\n<pre>" + bug.description.replace("-", "&#45;") + "</pre>"
	issue = project.issues.create({'title': bug.title, 'description': desc})
	issue.labels = bug.tags
	for msg in bug.messages:
		if msg == bug.messages[0] or not msg.content.strip():
			continue
		comment = msg.content.replace("-", "&#45;")
		note = "Comment from '" + re.sub(r'^.*~', '', msg.owner_link) \
		       + "' on Launchpad  (" \
		       + msg.date_created.date().isoformat() + "):\n\n<pre>" \
		       + comment + "</pre>"
		issue.notes.create({'body': note})
		time.sleep(0.2)  # To avoid "spamming"
	issue.save()


def main():
	LOG.info("args: %s", args)

	LOG.info("getting reports...")
	launchpad = get_launchpad()
	lp_project = launchpad.projects[args.lp_project_name]
	if args.search_text:
		bug_tasks = lp_project.searchTasks(
			search_text=args.search_text,
			omit_duplicates=True,
			order_by="datecreated")
	else:
		bug_tasks = lp_project.searchTasks(
			status=["New", "Confirmed", "Triaged", "In Progress"],
			omit_duplicates=True,
			order_by="datecreated")

	if args.gl_project_id:
		try:
			priv_token = os.environ['GITLAB_PRIVATE_TOKEN']
		except Exception as e:
			print("Please set the GITLAB_PRIVATE_TOKEN env variable!")
			return

		gl = gitlab.Gitlab('https://gitlab.com', private_token=priv_token)
		gl.auth()
		project = gl.projects.get(args.gl_project_id)
	else:
		LOG.info("Please provide a Gitlab project ID to transfer the bugs ('-g')")

	for bug_task in bug_tasks:
		show_bug_task(bug_task)
		if args.gl_project_id:
			transfer_to_gitlab(launchpad, project, bug_task)

	LOG.info("All done.")

if __name__ == '__main__':
	main()