* [layerindex-web][PATCH 1/7] requirements.txt: bump all to latest
2023-12-30 4:57 [layerindex-web][PATCH 0/7] Fixes; add Update Layer button; expose branch mapping Tim Orling
@ 2023-12-30 4:57 ` Tim Orling
2023-12-30 4:57 ` [layerindex-web][PATCH 2/7] layerindex/views.py: fix parse_view typo Tim Orling
` (5 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Tim Orling @ 2023-12-30 4:57 UTC (permalink / raw)
To: yocto
Pin mysqlclient as greater than 2.1.1 changes behavior
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
requirements.txt | 40 ++++++++++++++++++++--------------------
1 file changed, 20 insertions(+), 20 deletions(-)
diff --git a/requirements.txt b/requirements.txt
index 62b3f7b..20bcd30 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,8 +1,8 @@
-amqp==5.1.1
+amqp==5.2.0
asgiref==3.7.2
beautifulsoup4==4.12.2
-billiard==4.1.0
-celery==5.3.4
+billiard==4.2.0
+celery==5.3.6
click==8.1.7
click-didyoumean==0.3.0
click-plugins==1.1.1
@@ -10,29 +10,29 @@ click-repl==0.3.0
confusable-homoglyphs==3.2.0
diff-match-patch==20230430
Django>=4.2,<4.3
-django-appconf==1.0.5
-django-axes==6.1.0
-django-cors-headers==4.2.0
-django-ipware==5.0.0
+django-appconf==1.0.6
+django-axes==6.2.0
+django-cors-headers==4.3.1
+django-ipware==6.0.3
django-ranged-response==0.2.0
django-registration==3.4
-django-reversion==5.0.4
+django-reversion==5.0.9
django-reversion-compare==0.16.2
-django-simple-captcha==0.5.20
+django-simple-captcha==0.6.0
djangorestframework==3.14.0
-gitdb==4.0.10
-GitPython==3.1.37
-kombu==5.3.2
+gitdb==4.0.11
+GitPython==3.1.40
+kombu==5.3.4
mysqlclient==2.1.1
-Pillow==10.0.1
-prompt-toolkit==3.0.39
+Pillow==10.1.0
+prompt-toolkit==3.0.43
python-dateutil==2.8.2
-pytz==2023.3
+pytz==2023.3.post1
six==1.16.0
-smmap==5.0.0
-soupsieve==2.4.1
+smmap==5.0.1
+soupsieve==2.5
sqlparse==0.4.4
-typing_extensions==4.8.0
+typing_extensions==4.9.0
tzdata==2023.3
-vine==5.0.0
-wcwidth==0.2.8
+vine==5.1.0
+wcwidth==0.2.12
--
2.34.1
^ permalink raw reply related [flat|nested] 8+ messages in thread* [layerindex-web][PATCH 2/7] layerindex/views.py: fix parse_view typo
2023-12-30 4:57 [layerindex-web][PATCH 0/7] Fixes; add Update Layer button; expose branch mapping Tim Orling
2023-12-30 4:57 ` [layerindex-web][PATCH 1/7] requirements.txt: bump all to latest Tim Orling
@ 2023-12-30 4:57 ` Tim Orling
2023-12-30 4:57 ` [layerindex-web][PATCH 3/7] docker/settings.py: fix RABBIT_BROKER TypeError Tim Orling
` (4 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Tim Orling @ 2023-12-30 4:57 UTC (permalink / raw)
To: yocto
The Branch Comparison view would throw an error because of a parser_view instead of parse_view typo.
[YOCTO #15332]
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
layerindex/views.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/layerindex/views.py b/layerindex/views.py
index e660578..2e87181 100644
--- a/layerindex/views.py
+++ b/layerindex/views.py
@@ -1791,7 +1791,7 @@ class BranchCompareView(FormView):
for _, recipe in sorted(recipes.items(), key=lambda item: item[0]):
recipe.from_versions = sorted(recipe.from_versions, key=lambda item: parse_version(item.pv))
from_version_exprs = [x.version_expr() for x in recipe.from_versions]
- recipe.to_versions = sorted(recipe.to_versions, key=lambda item: parser_version(item.pv))
+ recipe.to_versions = sorted(recipe.to_versions, key=lambda item: parse_version(item.pv))
to_version_exprs = [x.version_expr() for x in recipe.to_versions]
if not from_version_exprs:
added.append(recipe)
--
2.34.1
^ permalink raw reply related [flat|nested] 8+ messages in thread* [layerindex-web][PATCH 3/7] docker/settings.py: fix RABBIT_BROKER TypeError
2023-12-30 4:57 [layerindex-web][PATCH 0/7] Fixes; add Update Layer button; expose branch mapping Tim Orling
2023-12-30 4:57 ` [layerindex-web][PATCH 1/7] requirements.txt: bump all to latest Tim Orling
2023-12-30 4:57 ` [layerindex-web][PATCH 2/7] layerindex/views.py: fix parse_view typo Tim Orling
@ 2023-12-30 4:57 ` Tim Orling
2023-12-30 4:57 ` [layerindex-web][PATCH 4/7] dockersetup.py: enable production Let's Encrypt Tim Orling
` (3 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Tim Orling @ 2023-12-30 4:57 UTC (permalink / raw)
To: yocto
Within the layersapp container, if you try to run layerindex/update.py:
File "/opt/layerindex/settings.py", line 280, in <module>
RABBIT_BROKER = 'amqp://' + os.getenv('RABBITMQ_DEFAULT_USER') + ':' + os.getenv('RABBITMQ_DEFAULT_PASS') + '@layersrabbit:5672/'
TypeError: can only concatenate str (not "NoneType") to str
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
docker/settings.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docker/settings.py b/docker/settings.py
index 33ab332..c202585 100644
--- a/docker/settings.py
+++ b/docker/settings.py
@@ -276,7 +276,7 @@ SUBMIT_EMAIL_SUBJECT = 'OE Layerindex layer submission'
SEND_PUBLISH_EMAIL = True
# RabbitMQ settings
-RABBIT_BROKER = 'amqp://' + os.getenv('RABBITMQ_DEFAULT_USER') + ':' + os.getenv('RABBITMQ_DEFAULT_PASS') + '@layersrabbit:5672/'
+RABBIT_BROKER = 'amqp://' + os.getenv('RABBITMQ_DEFAULT_USER', 'guest') + ':' + os.getenv('RABBITMQ_DEFAULT_PASS', 'guest') + '@layersrabbit:5672/'
RABBIT_BACKEND = 'rpc://layersrabbit/'
# Used for fetching repo
--
2.34.1
^ permalink raw reply related [flat|nested] 8+ messages in thread* [layerindex-web][PATCH 4/7] dockersetup.py: enable production Let's Encrypt
2023-12-30 4:57 [layerindex-web][PATCH 0/7] Fixes; add Update Layer button; expose branch mapping Tim Orling
` (2 preceding siblings ...)
2023-12-30 4:57 ` [layerindex-web][PATCH 3/7] docker/settings.py: fix RABBIT_BROKER TypeError Tim Orling
@ 2023-12-30 4:57 ` Tim Orling
2023-12-30 4:57 ` [layerindex-web][PATCH 5/7] layerindex: Add actual_branch to forms and views Tim Orling
` (2 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Tim Orling @ 2023-12-30 4:57 UTC (permalink / raw)
To: yocto
The '--staging' argument to certbot has now been changed
to '--test-cert'. We previously only allowed using the
dockersetup.py tool to create Staging environment certs,
which are still marked as invalid by browsers. Add a
'--letsencrypt-production' knob to allow for valid, trusted
certs to be created. If they already exist in the workspace
and have not expired, re-use them (to avoid hitting rate
limits). Continue to '--force-renewal' for staging certs.
NOTE:
If you have previously created staging certs in your
workspace, you will want to clean docker/certs before
creating production certs for the same domain. Certbot
will not overwrite those staging certs and the newly
created ones will not be in the path passed in by
dockersetup.py.
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
dockersetup.py | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)
diff --git a/dockersetup.py b/dockersetup.py
index d6e8f33..56458d3 100755
--- a/dockersetup.py
+++ b/dockersetup.py
@@ -66,6 +66,7 @@ def get_args():
parser.add_argument('--cert', type=str, help='Existing SSL certificate to use for HTTPS web serving', required=False)
parser.add_argument('--cert-key', type=str, help='Existing SSL certificate key to use for HTTPS web serving', required=False)
parser.add_argument('--letsencrypt', action="store_true", default=False, help='Use Let\'s Encrypt for HTTPS')
+ parser.add_argument('--letsencrypt-production', action="store_true", default=False, help='Use Production server for Let\'s Encrypt. Default is %(default)s')
parser.add_argument('--no-migrate', action="store_true", default=False, help='Skip running database migrations')
parser.add_argument('--no-admin-user', action="store_true", default=False, help='Skip adding admin user')
parser.add_argument('--no-connectivity', action="store_true", default=False, help='Skip checking external network connectivity')
@@ -473,7 +474,7 @@ def edit_dockerfile_web(hostname, no_https):
writefile("Dockerfile.web", ''.join(newlines))
-def setup_https(hostname, http_port, https_port, letsencrypt, cert, cert_key, emailaddr):
+def setup_https(hostname, http_port, https_port, letsencrypt, letsencrypt_production, cert, cert_key, emailaddr):
local_cert_dir = os.path.abspath('docker/certs')
container_cert_dir = '/opt/cert'
if letsencrypt:
@@ -548,7 +549,7 @@ def setup_https(hostname, http_port, https_port, letsencrypt, cert, cert_key, em
shutil.rmtree(tempdir)
# Now run certbot to register SSL certificate
- staging_arg = '--staging'
+ staging_arg = '--force-renewal --test-cert' if not letsencrypt_production else '--keep-until-expiring'
if emailaddr:
email_arg = '--email %s' % quote(emailaddr)
else:
@@ -560,7 +561,7 @@ def setup_https(hostname, http_port, https_port, letsencrypt, cert, cert_key, em
-d %s \
--rsa-key-size 4096 \
--agree-tos \
- --force-renewal" layerscertbot' % (staging_arg, email_arg, quote(hostname)), shell=True)
+ " layerscertbot' % (staging_arg, email_arg, quote(hostname)), shell=True)
if return_code != 0:
print("Running certbot failed")
sys.exit(1)
@@ -757,7 +758,7 @@ else:
edit_options_file(args.project_name)
if not args.no_https:
- setup_https(args.hostname, http_port, https_port, args.letsencrypt, args.cert, args.cert_key, emailaddr)
+ setup_https(args.hostname, http_port, https_port, args.letsencrypt, args.letsencrypt_production, args.cert, args.cert_key, emailaddr)
## Start up containers
return_code = subprocess.call(['docker-compose', 'up', '-d', '--build'], shell=False)
--
2.34.1
^ permalink raw reply related [flat|nested] 8+ messages in thread* [layerindex-web][PATCH 5/7] layerindex: Add actual_branch to forms and views
2023-12-30 4:57 [layerindex-web][PATCH 0/7] Fixes; add Update Layer button; expose branch mapping Tim Orling
` (3 preceding siblings ...)
2023-12-30 4:57 ` [layerindex-web][PATCH 4/7] dockersetup.py: enable production Let's Encrypt Tim Orling
@ 2023-12-30 4:57 ` Tim Orling
2023-12-30 4:57 ` [layerindex-web][PATCH 6/7] layerindex: add Update Layer UI feature Tim Orling
2023-12-30 4:57 ` [layerindex-web][PATCH 7/7] layerindex: improve updates for actual_branch Tim Orling
6 siblings, 0 replies; 8+ messages in thread
From: Tim Orling @ 2023-12-30 4:57 UTC (permalink / raw)
To: yocto
For layers which do not follow standard branch names (including
the inclusive naming move away from "master" to "main") we have
the actual_branch field set in a LayerBranch object. Previously
this was only exposed via the admin interface.
Allow layer maintainers (including upon submitting a new layer)
to set the 'Actual branch' in the web UI.
Add a check to make sure the actual_branch is a valid branch
name using 'git check-ref-format --branch <actual_branch>'
since we are not using full refs.
[YOCTO #8008]
NOTE:
Only existing LayerBranches will be editable. A new layer
can be submitted with a different branch for "master", but
only the "master" LayerBranch will be created.
Further changes to the update.py script will be needed to
make creation of new stable branches with an actual_branch
possible.
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
layerindex/forms.py | 15 +++++++++++++++
layerindex/views.py | 5 +++++
2 files changed, 20 insertions(+)
diff --git a/layerindex/forms.py b/layerindex/forms.py
index 9615859..3b764f1 100644
--- a/layerindex/forms.py
+++ b/layerindex/forms.py
@@ -76,6 +76,7 @@ LayerMaintainerFormSet = inlineformset_factory(LayerBranch, LayerMaintainer, for
class EditLayerForm(StyledModelForm):
# Additional form fields
vcs_subdir = forms.CharField(label='Repository subdirectory', max_length=40, required=False, help_text='Subdirectory within the repository where the layer is located, if not in the root (usually only used if the repository contains more than one layer)')
+ actual_branch = forms.CharField(label='Actual branch', max_length=80, required=False, help_text='Name of the actual branch in the repository matching the core branch (e.g. the development branch is "master" by default)')
deps = forms.ModelMultipleChoiceField(label='Other layers this layer depends upon', queryset=LayerItem.objects.filter(comparison=False), required=False)
captcha = CaptchaField(label='Verification', help_text='Please enter the letters displayed for verification purposes', error_messages={'invalid':'Incorrect entry, please try again'})
@@ -98,11 +99,16 @@ class EditLayerForm(StyledModelForm):
field_order.pop(field_order.index('vcs_subdir'))
name_pos = field_order.index('vcs_url') + 1
field_order.insert(name_pos, 'vcs_subdir')
+ # Ensure actual branch appears after repo subdir
+ field_order.pop(field_order.index('actual_branch'))
+ name_pos = name_pos + 1
+ field_order.insert(name_pos, 'actual_branch')
new_fields = OrderedDict()
for field in field_order:
new_fields[field] = self.fields[field]
self.fields = new_fields
self.fields['vcs_subdir'].initial = layerbranch.vcs_subdir
+ self.fields['actual_branch'].initial = layerbranch.actual_branch
self.was_saved = False
self.allow_base_type = allow_base_type
@@ -178,6 +184,15 @@ class EditLayerForm(StyledModelForm):
val(usage)
return usage
+ def clean_actual_branch(self):
+ import subprocess
+ actual_branch = self.cleaned_data['actual_branch'].strip()
+ process = subprocess.Popen(["git", "check-ref-format", "--branch", actual_branch])
+ exit_status = process.wait()
+ if exit_status != 0:
+ raise forms.ValidationError("Actual branch should be a valid git branch short name")
+ return actual_branch
+
class EditNoteForm(StyledModelForm):
class Meta:
diff --git a/layerindex/views.py b/layerindex/views.py
index 2e87181..1a29dde 100644
--- a/layerindex/views.py
+++ b/layerindex/views.py
@@ -162,6 +162,11 @@ def edit_layer_view(request, template_name, branch='master', slug=None):
layerbranch.vcs_subdir = new_subdir
reset_last_rev = True
layerbranch.save()
+ new_actual_branch = form.cleaned_data['actual_branch']
+ if layerbranch.actual_branch != new_actual_branch:
+ layerbranch.actual_branch = new_actual_branch
+ reset_last_rev = True
+ layerbranch.save()
maintainerformset.save()
if slug:
new_deps = form.cleaned_data['deps']
--
2.34.1
^ permalink raw reply related [flat|nested] 8+ messages in thread* [layerindex-web][PATCH 6/7] layerindex: add Update Layer UI feature
2023-12-30 4:57 [layerindex-web][PATCH 0/7] Fixes; add Update Layer button; expose branch mapping Tim Orling
` (4 preceding siblings ...)
2023-12-30 4:57 ` [layerindex-web][PATCH 5/7] layerindex: Add actual_branch to forms and views Tim Orling
@ 2023-12-30 4:57 ` Tim Orling
2023-12-30 4:57 ` [layerindex-web][PATCH 7/7] layerindex: improve updates for actual_branch Tim Orling
6 siblings, 0 replies; 8+ messages in thread
From: Tim Orling @ 2023-12-30 4:57 UTC (permalink / raw)
To: yocto
* Add an "Update Layer" button to the layer detail view.
- This allows a user that is a member of is_staff to trigger
an update of the current layer (for the current branch)
* Add an "Update Layer" button to the reviewdetail view
- This allows a user that is a member of is_staff and has
publish_layer permissions to trigger an update attempt
of the layer under review (even in the un-published state)
* The update is run as a task with Celery
NOTE:
You must have the RABBITMQ_ and DATABASE_ credentials set
correctly in the docker/settings.py file or set via
environment variables or you will get authentication errors
talking to layersdb or layersrabbit containers.
[YOCTO #12484]
layerindex/views.py: add update_layer_view
layerindex/urls.py: add update_layer_view
layerindex/urls_branch.py: add update_layer_view
templates/layerindex/reviewdetail.html: add Update Layer button
templates/layerindex/detail.html: add Update Layer button
templates/layerindex: add updatelayer.html
TODO:
While the update is happening, the AJAX rendering of the
update.log is showing the b'' characters and not adding
any new lines. If you go back to the same task view
afterwards, the log is rendered as expected.
TODO:
After the update is completed, it would be nice to have a
button to return you to the page from where you called the
"Update Layer".
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
layerindex/urls.py | 2 +-
layerindex/urls_branch.py | 3 +-
layerindex/views.py | 35 ++++++++
templates/layerindex/detail.html | 3 +
templates/layerindex/reviewdetail.html | 3 +
templates/layerindex/updatelayer.html | 119 +++++++++++++++++++++++++
6 files changed, 163 insertions(+), 2 deletions(-)
create mode 100644 templates/layerindex/updatelayer.html
diff --git a/layerindex/urls.py b/layerindex/urls.py
index cee3d2a..1fcd11d 100644
--- a/layerindex/urls.py
+++ b/layerindex/urls.py
@@ -16,7 +16,7 @@ from layerindex.views import LayerListView, LayerReviewListView, LayerReviewDeta
ClassicRecipeSearchView, ClassicRecipeDetailView, ClassicRecipeStatsView, LayerUpdateDetailView, UpdateListView, \
UpdateDetailView, StatsView, publish_view, LayerCheckListView, BBClassCheckListView, TaskStatusView, \
ComparisonRecipeSelectView, ComparisonRecipeSelectDetailView, task_log_view, task_stop_view, email_test_view, \
- BranchCompareView, RecipeDependenciesView
+ BranchCompareView, RecipeDependenciesView, update_layer_view
from layerindex.models import LayerItem, Recipe, RecipeChangeset
from rest_framework import routers
from . import restviews
diff --git a/layerindex/urls_branch.py b/layerindex/urls_branch.py
index 6736f32..40cd915 100644
--- a/layerindex/urls_branch.py
+++ b/layerindex/urls_branch.py
@@ -8,7 +8,7 @@
from django.views.defaults import page_not_found
from django.urls import include, re_path, reverse_lazy
-from layerindex.views import LayerListView, RecipeSearchView, MachineSearchView, DistroSearchView, ClassSearchView, LayerDetailView, edit_layer_view, delete_layer_view, edit_layernote_view, delete_layernote_view, RedirectParamsView, DuplicatesView, LayerUpdateDetailView, layer_export_recipes_csv_view, comparison_update_view
+from layerindex.views import LayerListView, RecipeSearchView, MachineSearchView, DistroSearchView, ClassSearchView, LayerDetailView, edit_layer_view, delete_layer_view, edit_layernote_view, delete_layernote_view, RedirectParamsView, DuplicatesView, LayerUpdateDetailView, layer_export_recipes_csv_view, comparison_update_view, update_layer_view
urlpatterns = [
re_path(r'^$',
@@ -41,6 +41,7 @@ urlpatterns = [
template_name='layerindex/classes.html'),
name='class_search'),
re_path(r'^edit/(?P<slug>[-\w]+)/$', edit_layer_view, {'template_name': 'layerindex/editlayer.html'}, name="edit_layer"),
+ re_path(r'^update/(?P<slug>[-\w]+)/$', update_layer_view, {'template_name': 'layerindex/updatelayer.html'}, name="update_layer"),
re_path(r'^duplicates/$',
DuplicatesView.as_view(
template_name='layerindex/duplicates.html'),
diff --git a/layerindex/views.py b/layerindex/views.py
index 1a29dde..25c045e 100644
--- a/layerindex/views.py
+++ b/layerindex/views.py
@@ -121,6 +121,41 @@ def delete_layer_view(request, template_name, slug):
'cancel_url': layeritem.get_absolute_url()
})
+def update_layer_view(request, template_name, branch='master', slug=None):
+ if not (request.user.is_authenticated and request.user.is_staff):
+ raise PermissionDenied
+ return_url = None
+ branchobj = Branch.objects.filter(name=branch)[:1].get()
+ if slug:
+ # Existing Layer Update
+ layeritem = get_object_or_404(LayerItem, name=slug)
+ layerbranch = get_object_or_404(LayerBranch, layer=layeritem, branch=branchobj)
+ returnto = request.GET.get('returnto', 'layer_item')
+ if returnto:
+ if returnto == 'layer_review':
+ return_url = reverse_lazy(returnto, args=(layeritem.name,))
+ else:
+ return_url = reverse_lazy(returnto, args=(branch, layeritem.name))
+ else:
+ # Pre-Publish Layer Update attempt
+ layeritem = LayerItem()
+ layerbranch = LayerBranch(layer=layeritem, branch=branchobj)
+
+ from celery import uuid
+
+ cmd = 'layerindex/update.py --layer %s --branch %s;' % (layeritem.name, branch)
+
+ task_id = uuid()
+ # Create this here first, because inside the task we don't have all of the required info
+ update = Update(task_id=task_id)
+ update.started = datetime.now()
+ update.triggered_by = request.user
+ update.save()
+
+ res = tasks.run_update_command.apply_async((branch, cmd), task_id=task_id)
+
+ return HttpResponseRedirect(reverse_lazy('task_status', kwargs={'task_id': task_id}))
+
def edit_layer_view(request, template_name, branch='master', slug=None):
return_url = None
branchobj = Branch.objects.filter(name=branch)[:1].get()
diff --git a/templates/layerindex/detail.html b/templates/layerindex/detail.html
index 12ce156..0fa8caf 100644
--- a/templates/layerindex/detail.html
+++ b/templates/layerindex/detail.html
@@ -38,6 +38,9 @@
{% endif %}
{% if user.is_authenticated %}
<span class="pull-right">
+ {% if user.is_staff %}
+ <a href="{% url 'update_layer' url_branch layeritem.name %}" class="btn btn-info">Update layer</a>
+ {% endif %}
{% if perms.layerindex.publish_layer or useredit %}
<a href="{% url 'edit_layer' url_branch layeritem.name %}" class="btn btn-default">Edit layer</a>
{% if layeritem.layernote_set.count == 0 %}
diff --git a/templates/layerindex/reviewdetail.html b/templates/layerindex/reviewdetail.html
index 8baab7c..39501be 100644
--- a/templates/layerindex/reviewdetail.html
+++ b/templates/layerindex/reviewdetail.html
@@ -47,6 +47,9 @@
<a href="{% url 'delete_layer' layeritem.name %}" class="btn btn-warning">Delete layer</a>
<a href="{% url 'publish' layeritem.name %}" class="btn btn-primary">Publish layer</a>
{% endif %}
+ {% if user.is_staff %}
+ <a href="{% url 'update_layer' 'master' layeritem.name %}" class="btn btn-info">Update layer</a>
+ {% endif %}
</span>
{% endif %}
</h1>
diff --git a/templates/layerindex/updatelayer.html b/templates/layerindex/updatelayer.html
new file mode 100644
index 0000000..799fb51
--- /dev/null
+++ b/templates/layerindex/updatelayer.html
@@ -0,0 +1,119 @@
+{% extends "base.html" %}
+{% load i18n %}
+{% load extrafilters %}
+
+{% comment %}
+
+ layerindex-web - update page
+
+ Copyright (C) 2016, 2018 Intel Corporation
+ Licensed under the MIT license, see COPYING.MIT for details
+
+{% endcomment %}
+
+<!--
+{% block title_append %} - {{ update.started }} {% endblock %}
+-->
+
+{% block content %}
+{% autoescape on %}
+
+ <ul class="breadcrumb">
+ <li><a href="{% url 'update_list' %}">Updates</a></li>
+ <li class="active">{{ update.started }}</li>
+ </ul>
+
+
+<h2>{{ update.started }} {% if update.reload %}(reload){% endif %}
+<span id="status-label" class="label {% if update.finished %}{% if update.retcode %}label-danger{% else %}label-success{% endif %}{% endif %} pull-right">{% if update.finished %}{% if update.retcode < 0 %}TERMINATED ({{ update.retcode }}){% elif update.retcode %}FAILED{% endif %}{% endif %}</span>
+</h2>
+
+{% if update.log %}
+ <pre>{{ update.log }}</pre>
+{% endif %}
+
+{% for layerupdate in layerupdates %}
+ <table class="table table-bordered">
+ <thead>
+ {% with layerbranch_exists=layerupdate.layerbranch_exists %}
+ <tr><td{% if layerupdate.errors > 0 or layerupdate.retcode != 0 %} class="error"{% elif layerupdate.warnings %} class="warning"{% endif %}>
+ {% if layerbranch_exists %}<a href="{% url 'layer_item' layerupdate.branch.name layerupdate.layer.name %}">{% endif %}<strong>{{ layerupdate.layer.name }} {{ layerupdate.branch.name }}</strong>{% if layerbranch_exists %}</a>{% endif %}
+ {% if layerupdate.vcs_before_rev != layerupdate.vcs_after_rev %}
+ <span class="pull-right">
+ {% if layerbranch_exists %}
+ {% with before_url=layerupdate.vcs_before_commit_url after_url=layerupdate.vcs_after_commit_url %}
+ {% if before_url %}<a href="{{ before_url }}">{% endif %}{{ layerupdate.vcs_before_rev|truncatesimple:10 }}{% if before_url %}</a>{% endif %} → {% if after_url %}<a href="{{ after_url }}">{% endif %}{{ layerupdate.vcs_after_rev|truncatesimple:10 }}{% if after_url %}</a>{% endif %}
+ {% endwith %}
+ {% else %}
+ {{ layerupdate.vcs_before_rev|truncatesimple:10 }} → {{ layerupdate.vcs_after_rev|truncatesimple:10 }}
+ {% endif %}
+ </span>
+ {% endif %}
+ </td></tr>
+ {% endwith %}
+ </thead>
+ <tbody>
+ {% if layerupdate.log %}
+ <tr><td class="td-pre">
+ <pre class="pre-scrollable pre-plain">{{ layerupdate.log }}</pre>
+ </td></tr>
+ {% endif %}
+ </tbody>
+ </table>
+{% endfor %}
+
+{% if not update.log and not layerupdates %}
+ <p>No messages or layer updates</p>
+{% endif %}
+
+{% if update.comparisonrecipeupdate_set.exists %}
+<h3>Updated comparison recipes</h3>
+<ul>
+{% for recipeupdate in update.comparisonrecipeupdate_set.all %}
+ <li><a href="{% url 'comparison_recipe' recipeupdate.recipe.id %}">{{ recipeupdate.recipe.pn }}</a> {% if recipeupdate.meta_updated and recipeupdate.link_updated %}(meta, link){% elif recipeupdate.link_updated %}(link){% elif recipeupdate.meta_updated %}(meta){% endif %}</li>
+{% endfor %}
+</ul>
+{% endif %}
+
+{% endautoescape %}
+
+{% endblock %}
+{% if updates.count > 0 %}
+<div class="tab-pane" id="updates">
+ <nav class="navbar navbar-default">
+ <div class="container-fluid">
+ <div class="navbar-header">
+ <a class="navbar-brand">{{ layeritem.name }} updates</a>
+ </div>
+ </div>
+ </nav>
+
+ <table class="table table-bordered">
+ <thead>
+ <tr>
+ <th>Date/time</th>
+ <th>Errors</th>
+ <th>Warnings</th>
+ </tr>
+ </thead>
+ <tbody>
+ {% for update in updates %}
+ <tr>
+ <td>
+ {% if update.log %}
+ <a href="{% url 'layerupdate' update.id %}">{{ update.started }}{% if update.update.reload %} (reload){% endif%}</a>
+ {% else %}
+ <span class="text-muted">{{ update.started }}{% if update.update.reload %} (reload){% endif%}</span>
+ {% endif %}
+ </td>
+ <td>{% if update.errors %}<span class="badge badge-important">{{ update.errors }}</span>{% endif %}</td>
+ <td>{% if update.warnings %}<span class="badge badge-warning">{{ update.warnings }}</span>{% endif %}</td>
+ </tr>
+ {% endfor %}
+ </tbody>
+ </table>
+</div>
+{% endif %}
+
+{% block scripts %}
+{% endblock %}
--
2.34.1
^ permalink raw reply related [flat|nested] 8+ messages in thread* [layerindex-web][PATCH 7/7] layerindex: improve updates for actual_branch
2023-12-30 4:57 [layerindex-web][PATCH 0/7] Fixes; add Update Layer button; expose branch mapping Tim Orling
` (5 preceding siblings ...)
2023-12-30 4:57 ` [layerindex-web][PATCH 6/7] layerindex: add Update Layer UI feature Tim Orling
@ 2023-12-30 4:57 ` Tim Orling
6 siblings, 0 replies; 8+ messages in thread
From: Tim Orling @ 2023-12-30 4:57 UTC (permalink / raw)
To: yocto
Some layers now have one branch with many supported
LAYERSERIES_COMPAT. If this branch name does not match
one of the stable releases, LayerBranches might not
have been created. When actual_branch is set, it is
only set in a LayerBranch object. We previously
could not update (create) a stable branch with actual_branch
except manually in the admin interface.
Add --force-create option to be used in conjunction with
--actual-branch (which already requires --branch) in the
update.py script. This tells the script to ignore the
fact that no layerbranch exists already.
Add --actual-branch to update_layer.py so that we can create
(and more importantly checkout) an actual_branch for the
given stable --branch.
Update utils.py to allow checking out of actual_branch when
a LayerBranch does not yet exist.
While we are at it, ensure that any Branch that is marked
as no update will be skipped even with --force-create. The
main reason that a Branch has updates disabled is because the
bitbake or python syntax has changed enough to cause exceptions.
This script can now be run with:
./layerindex/update.py \
--layer meta-weird-one \
--branch kirkstone \
--actual-branch=nonstandard \
--force-create
Which will attempt to create a meta-weird-one:kirkstone layerbranch
checked out at the 'nonstandard' branch from that layer's git repo.
This allows layerindex admins to at least populate the database
without tedious creation of layerbranches in the admin interface.
Helps make the "branch mapping" actually work and be useful:
[YOCTO #8008]
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
layerindex/update.py | 39 ++++++++++++++++++++++++++++----------
layerindex/update_layer.py | 8 ++++++++
layerindex/utils.py | 7 +++++--
3 files changed, 42 insertions(+), 12 deletions(-)
diff --git a/layerindex/update.py b/layerindex/update.py
index f9a216d..c1886bd 100755
--- a/layerindex/update.py
+++ b/layerindex/update.py
@@ -43,6 +43,8 @@ def prepare_update_layer_command(options, branch, layer, initial=False):
else:
cmdprefix = 'python3'
cmd = '%s update_layer.py -l %s -b %s' % (cmdprefix, layer.name, branch.name)
+ if options.actual_branch and options.force_create:
+ cmd += ' --actual-branch=%s' % options.actual_branch
if options.reload:
cmd += ' --reload'
if options.fullreload:
@@ -86,15 +88,18 @@ def update_actual_branch(layerquery, fetchdir, branch, options, update_bitbake,
logger.info("Skipping update actual_branch for %s - branch %s doesn't exist" % (layer.name, actual_branch))
continue
layerbranch = layer.get_layerbranch(branch)
- if not layerbranch:
- logger.info("Skipping update actual_branch for %s - layerbranch %s doesn't exist" % (layer.name, branch))
- continue
- if actual_branch != layerbranch.actual_branch:
- logger.info("%s: %s.actual_branch: %s -> %s" % (layer.name, branch, layerbranch.actual_branch, actual_branch))
- layerbranch.actual_branch = actual_branch
- to_save.add(layerbranch)
+ if not options.force_create:
+ if not layerbranch:
+ logger.info("Skipping update actual_branch for %s - layerbranch %s doesn't exist" % (layer.name, branch))
+ continue
+ if actual_branch != layerbranch.actual_branch:
+ logger.info("%s: %s.actual_branch: %s -> %s" % (layer.name, branch, layerbranch.actual_branch, actual_branch))
+ layerbranch.actual_branch = actual_branch
+ to_save.add(layerbranch)
+ else:
+ logger.info("%s: %s.actual_branch is already %s, so no change" % (layer.name, branch, actual_branch))
else:
- logger.info("%s: %s.actual_branch is already %s, so no change" % (layer.name, branch, actual_branch))
+ logger.info("%s: Allowing branch %s with actual_branch %s to attempt to be created" % (layer.name, branch, actual_branch))
# At last, do the save
if not options.dryrun:
@@ -169,6 +174,9 @@ def main():
parser.add_option("-a", "--actual-branch",
help = "Update actual branch for layer and bitbake",
action="store", dest="actual_branch", default='')
+ parser.add_option("", "--force-create",
+ help = "Create layer branch if it does not already exist",
+ action="store_true", dest="force_create", default=False)
parser.add_option("-d", "--debug",
help = "Enable debug output",
action="store_const", const=logging.DEBUG, dest="loglevel", default=logging.INFO)
@@ -197,6 +205,10 @@ def main():
if not utils.get_branch(branch):
logger.error("Specified branch %s is not valid" % branch)
sys.exit(1)
+ branchquery = Branch.objects.filter(updates_enabled=True).filter(name=branch)
+ if not branchquery.count() > 0:
+ logger.warning("Updates are disabled for specified branch %s" % branch)
+ sys.exit(1)
else:
branchquery = Branch.objects.filter(updates_enabled=True)
branches = [branch.name for branch in branchquery]
@@ -315,7 +327,7 @@ def main():
logger.error("No repositories could be fetched, exiting")
sys.exit(1)
- if options.actual_branch:
+ if options.actual_branch and not options.force_create:
update_actual_branch(layerquery, fetchdir, branches[0], options, update_bitbake, bitbakepath)
return
@@ -389,6 +401,10 @@ def main():
logger.error('Repository %s is bare, not supported' % repodir)
continue
try:
+ # Allow stable branches to be created if actual_branch exists
+ if options.actual_branch:
+ branchname = options.actual_branch
+ branchdesc = "%s (%s)" % (branch, branchname)
# Always get origin/branchname, so it raises error when branch doesn't exist when nocheckout
topcommit = repo.commit('origin/%s' % branchname)
if options.nocheckout:
@@ -419,7 +435,10 @@ def main():
else:
# Check out appropriate branch
if not options.nocheckout:
- utils.checkout_layer_branch(layerbranch, repodir, logger=logger)
+ if not options.actual_branch:
+ utils.checkout_layer_branch(layerbranch, repodir, logger=logger)
+ else:
+ utils.checkout_layer_branch(layerbranch, repodir, actual_branch=options.actual_branch, logger=logger)
layerdir = os.path.join(repodir, layerbranch.vcs_subdir)
if layerbranch.vcs_subdir and not os.path.exists(layerdir):
print_subdir_error(newbranch, layer.name, layerbranch.vcs_subdir, branchdesc)
diff --git a/layerindex/update_layer.py b/layerindex/update_layer.py
index 6d73fad..4a67a50 100644
--- a/layerindex/update_layer.py
+++ b/layerindex/update_layer.py
@@ -272,6 +272,9 @@ def main():
parser.add_option("-i", "--initial",
help = "Print initial values parsed from layer.conf only",
action="store_true")
+ parser.add_option("-a", "--actual-branch",
+ help = "Specify actual_branch for git checkout",
+ action="store", dest="actual_branch", default=None)
parser.add_option("-d", "--debug",
help = "Enable debug output",
action="store_const", const=logging.DEBUG, dest="loglevel", default=logging.INFO)
@@ -326,6 +329,9 @@ def main():
if layerbranch.actual_branch:
branchname = layerbranch.actual_branch
branchdesc = "%s (%s)" % (options.branch, branchname)
+ elif options.actual_branch:
+ branchname = options.actual_branch
+ branchdesc = "%s (%s)" % (options.branch, branchname)
# Collect repo info
repo = git.Repo(repodir)
@@ -347,6 +353,8 @@ def main():
layerbranch = LayerBranch()
layerbranch.layer = layer
layerbranch.branch = branch
+ if options.actual_branch:
+ layerbranch.actual_branch = options.actual_branch
layerbranch_source = layer.get_layerbranch(branch)
if not layerbranch_source:
layerbranch_source = layer.get_layerbranch(None)
diff --git a/layerindex/utils.py b/layerindex/utils.py
index 0ea8e48..8bbd621 100644
--- a/layerindex/utils.py
+++ b/layerindex/utils.py
@@ -297,8 +297,11 @@ def checkout_repo(repodir, commit, logger, force=False):
# Now check out the revision
runcmd(['git', 'checkout', commit], repodir, logger=logger)
-def checkout_layer_branch(layerbranch, repodir, logger=None):
- branchname = layerbranch.get_checkout_branch()
+def checkout_layer_branch(layerbranch, repodir, actual_branch=None, logger=None):
+ if actual_branch:
+ branchname = actual_branch
+ else:
+ branchname = layerbranch.get_checkout_branch()
checkout_repo(repodir, 'origin/%s' % branchname, logger)
def is_layer_valid(layerdir):
--
2.34.1
^ permalink raw reply related [flat|nested] 8+ messages in thread