* Transferring bugs from Launchpad to Gitlab (was: Re: Upstream bug comments on the mailing list and Launchpad)
2021-04-19 16:04 ` Peter Maydell
@ 2021-04-21 15:07 ` Thomas Huth
0 siblings, 0 replies; 4+ messages in thread
From: Thomas Huth @ 2021-04-21 15:07 UTC (permalink / raw)
To: Peter Maydell, John Snow
Cc: Daniel P. Berrange, QEMU Developers, Stefan Hajnoczi,
Kashyap Chamarthy
[-- Attachment #1: Type: text/plain, Size: 3171 bytes --]
On 19/04/2021 18.04, Peter Maydell wrote:
> On Mon, 19 Apr 2021 at 16:52, Thomas Huth <thuth@redhat.com> wrote:
>>
>> On 15/04/2021 11.49, Kashyap Chamarthy wrote:
>> ...
>>> PS: I recall there was discussion on the list of moving to a different
>>> GitLab tracker. As Thomas Huth mentioned on IRC, more people seem
>>> to have account on GitLab than on Launchpad.
>>
>> I think we basically agreed that we want to migrate to gitlab, but e.g.
>> Peter suggested that we copy the open bugs on launchpad to the gitlab
>> tracker first, so that we don't have two active trackers at the same time...
>> thus this needs someone with enough spare time to work on such a convertion
>> script first...
>
> In the meantime, can we disable bug reporting on gitlab? It's
> confusing to users to have two places, one of which is not
> checked by anybody. We already have two stray bugreports:
> https://gitlab.com/qemu-project/qemu/-/issues
I'd rather like to see us doing the move finally. I'm very much a Python
ignorant, but since apparently nobody else has spare time to work on this at
the moment, I've now done some copy-n-paste coding and came up with a script
that can transfer bugs from Launchpad to Gitlab (see the attachment). You
need a "private token" passed to the script via the GITLAB_PRIVATE_TOKEN
environment variable so that it is able to access gitlab (such a token can
be generated in your private setting pages on gitlab). Some transferred bugs
can be seen here:
https://gitlab.com/thuth/trackertest/-/issues
Notes:
- Not all information is transferred, e.g. no attachments yet.
But considering that there is a link to the original launchpad
bug, you can always look up the information there, so I think
that should be fine.
- The script currently also adds the "tags" from the launchpad
bug as "labels" in gitlab - do we want that, or do we rather
want to restrict our labels in gitlab to a smaller set?
- Formatting of the bug description and comments is a little bit
tricky, since the text is pre-formatted ASCII in launchpad and
markdown on gitlab ... For now, I put the text from launchpad
into "<pre>" HTML tags on gitlab, that should result in usable
texts most of the time.
- Biggest problem so far: After transferring 50 bugs or so,
gitlab refuses to take more bugs automatically with the
message: 'Your issue has been recognized as spam. Please,
change the content or solve the reCAPTCHA to proceed.'
... and solving captchas in a script is certainly way out
of scope here. So if we want to copy over all bugs from
launchpad, we might need to do that in junks.
- Do we want to auto-close the bugs in launchpad after we've
transferred them? Or should they stay open until the bugs
have been addressed? I tend to close them (with an appropriate
comment added to the bug, too), but we could also mark the
transferred bugs with a tag instead.
- Do we want to transfer also old "Wishlist" bug tickets from
Launchpad? I think it's quite unlikely that we will ever
address them... maybe we should close them on Launchpad with
an appropriate comment?
Thomas
[-- Attachment #2: launchpad2gitlab.py --]
[-- Type: text/x-python, Size: 3758 bytes --]
#!/usr/bin/env python3
import argparse
import logging
import os
import re
import sys
import time
import gitlab
from launchpadlib.launchpad import Launchpad
import lazr.restfulclient.errors
LOG_FORMAT = "%(levelname)s - %(funcName)s - %(message)s"
logging.basicConfig(format=LOG_FORMAT)
LOG = logging.getLogger(__name__)
parser = argparse.ArgumentParser(description="Close incomplete bug reports.")
parser.add_argument('-l',
'--lp-project-name',
dest='lp_project_name',
default='qemu',
help='The Launchpad project name (qemu).')
parser.add_argument('-g',
'--gl-project-id',
dest='gl_project_id',
help='The Gitlab project ID.')
parser.add_argument('--verbose', '-v',
help='Enable debug logging.',
action="store_true")
parser.add_argument('--search-text', '-s',
dest='search_text',
help='Look up bugs by searching for text.')
args = parser.parse_args()
LOG.info(args)
LOG.setLevel(logging.DEBUG if args.verbose else logging.INFO)
def get_launchpad():
cache_dir = os.path.expanduser("~/.launchpadlib/cache/")
if not os.path.exists(cache_dir):
os.makedirs(cache_dir, 0o700)
launchpad = Launchpad.login_anonymously(args.lp_project_name + '-bugs',
'production', cache_dir)
return launchpad
def show_bug_task(bug_task):
print('Bug #%d - %s' % (bug_task.bug.id, str(bug_task.bug.title)))
if args.verbose:
print(' - Description: %s' % bug_task.bug.description)
print(' - Tags: %s' % bug_task.bug.tags)
print(' - Status: %s' % bug_task.status)
print(' - Assignee: %s' % bug_task.assignee)
for message in bug_task.bug.messages:
print(' - Message: %s' % message.content)
def transfer_to_gitlab(launchpad, project, bug_task):
bug = bug_task.bug
desc = "This bug has been copied automatically from: " + bug_task.web_link + \
"\n\n<pre>" + bug.description.replace("-", "-") + "</pre>"
issue = project.issues.create({'title': bug.title, 'description': desc})
issue.labels = bug.tags
for msg in bug.messages:
if msg == bug.messages[0] or not msg.content.strip():
continue
comment = msg.content.replace("-", "-")
note = "Comment from '" + re.sub(r'^.*~', '', msg.owner_link) \
+ "' on Launchpad (" \
+ msg.date_created.date().isoformat() + "):\n\n<pre>" \
+ comment + "</pre>"
issue.notes.create({'body': note})
time.sleep(0.2) # To avoid "spamming"
issue.save()
def main():
LOG.info("args: %s", args)
LOG.info("getting reports...")
launchpad = get_launchpad()
lp_project = launchpad.projects[args.lp_project_name]
if args.search_text:
bug_tasks = lp_project.searchTasks(
search_text=args.search_text,
omit_duplicates=True,
order_by="datecreated")
else:
bug_tasks = lp_project.searchTasks(
status=["New", "Confirmed", "Triaged", "In Progress"],
omit_duplicates=True,
order_by="datecreated")
if args.gl_project_id:
try:
priv_token = os.environ['GITLAB_PRIVATE_TOKEN']
except Exception as e:
print("Please set the GITLAB_PRIVATE_TOKEN env variable!")
return
gl = gitlab.Gitlab('https://gitlab.com', private_token=priv_token)
gl.auth()
project = gl.projects.get(args.gl_project_id)
else:
LOG.info("Please provide a Gitlab project ID to transfer the bugs ('-g')")
for bug_task in bug_tasks:
show_bug_task(bug_task)
if args.gl_project_id:
transfer_to_gitlab(launchpad, project, bug_task)
LOG.info("All done.")
if __name__ == '__main__':
main()
^ permalink raw reply [flat|nested] 4+ messages in thread