Scheduled maintenance this week and next week

Scheduled maintenance this week and next week

[Guillaume Tucker]

We're now in the process of moving all the KernelCI Azure
services to a new subscription.  The domain names, port numbers
etc. should remain unchanged, there will only be new IP addresses
for the new VMs and DNS records should be updated in a timely
fashion.

As such, the staging VM is being shut down today.  We'll be
sharing updates as things on staging come back online by the end
of this week at the latest.  Services impacted by this downtime
are:

* https://staging.kernelci.org (web dashboard, backend, Jenkins etc.)
* https://staging.kernelci.org:9000 (new API & Pipeline)
* https://kcidb.kernelci.org (KCIDB Grafana web dashboard)
* https://chromeos.kernelci.org (ChromeOS KernelCI instance)

Once the staging move is complete, all the production services
are scheduled to be moved next week:

* https://kernelci.org (static website)
* https://linux.kernelci.org (web dashboard, backend, Jenkins etc.)

There may be some longer downtime next Monday compared with usual
weekly production deployments.  We'll also be sending regular
updates as things evolve.

Please let us know if this downtime is causing any critical
disruption and we'll see which measures we can take to mitigate
the impact.

Thanks,
Guillaume

Re: Scheduled maintenance this week and next week

[Guillaume Tucker]

On 30/05/2023 10:12, Guillaume Tucker wrote:
> We're now in the process of moving all the KernelCI Azure
> services to a new subscription.  The domain names, port numbers
> etc. should remain unchanged, there will only be new IP addresses
> for the new VMs and DNS records should be updated in a timely
> fashion.
> 
> As such, the staging VM is being shut down today.  We'll be
> sharing updates as things on staging come back online by the end
> of this week at the latest.  Services impacted by this downtime
> are:
> 
> * https://staging.kernelci.org (web dashboard, backend, Jenkins etc.)
> * https://staging.kernelci.org:9000 (new API & Pipeline)
> * https://kcidb.kernelci.org (KCIDB Grafana web dashboard)
> * https://chromeos.kernelci.org (ChromeOS KernelCI instance)

We've made some good progress with transferring the staging VMs
and services are coming back online now.  In principle all of
staging should be back to normal by tomorrow.

> Once the staging move is complete, all the production services
> are scheduled to be moved next week:
> 
> * https://kernelci.org (static website)
> * https://linux.kernelci.org (web dashboard, backend, Jenkins etc.)

Also:

* https://chromeos.kernelci.org (ChromeOS KernelCI instance)

The ChromeOS instance relies on services running both on staging
and production so it will be affected by both updates.  On a side
note, we're now working on removing this dependency with a new
chromeos.staging.kernelci.org instance for ChromeOS KernelCI
development purposes - this will happen after the move to the new
Azure subscription.

> There may be some longer downtime next Monday compared with usual
> weekly production deployments.  We'll also be sending regular
> updates as things evolve.

We're going to have a short production downtime later today to
take a copy of the VMs.  The web dashboard shouldn't be down for
more than a few minutes, and there might be some extra lag before
some kernel git revisions start getting tested.  This should help
reduce the downtime needed on Monday to complete the production
move.  We'll keep sending updates as things progress.

> Please let us know if this downtime is causing any critical
> disruption and we'll see which measures we can take to mitigate
> the impact.

Thanks,
Guillaume

Re: Scheduled maintenance this week and next week

[Guillaume Tucker]

On 01/06/2023 14:21, Guillaume Tucker wrote:
> On 30/05/2023 10:12, Guillaume Tucker wrote:
>> We're now in the process of moving all the KernelCI Azure
>> services to a new subscription.  The domain names, port numbers
>> etc. should remain unchanged, there will only be new IP addresses
>> for the new VMs and DNS records should be updated in a timely
>> fashion.
>>
>> As such, the staging VM is being shut down today.  We'll be
>> sharing updates as things on staging come back online by the end
>> of this week at the latest.  Services impacted by this downtime
>> are:
>>
>> * https://staging.kernelci.org (web dashboard, backend, Jenkins etc.)
>> * https://staging.kernelci.org:9000 (new API & Pipeline)
>> * https://kcidb.kernelci.org (KCIDB Grafana web dashboard)
>> * https://chromeos.kernelci.org (ChromeOS KernelCI instance)
> 
> We've made some good progress with transferring the staging VMs
> and services are coming back online now.  In principle all of
> staging should be back to normal by tomorrow.

All the staging services are now back online and jobs are running
as expected, both the old Jenkins and the new pipeline.

>> Once the staging move is complete, all the production services
>> are scheduled to be moved next week:
>>
>> * https://kernelci.org (static website)
>> * https://linux.kernelci.org (web dashboard, backend, Jenkins etc.)
> 
> Also:
> 
> * https://chromeos.kernelci.org (ChromeOS KernelCI instance)
> 
>> There may be some longer downtime next Monday compared with usual
>> weekly production deployments.  We'll also be sending regular
>> updates as things evolve.
>> 
>> Please let us know if this downtime is causing any critical
>> disruption and we'll see which measures we can take to mitigate
>> the impact.

We've now started transferring VMs for the production move.  It's
still scheduled to happen on Monday as per the usual production
updates, except this time we won't be updating any of the code to
focus solely on the infrastructure.  If things to well, the
downtime shouldn't last more than a couple of hours.  We'll soon
confirm the exact time window for the downtime and provide
regular updates.

Thanks,
Guillaume

Re: Scheduled maintenance this week and next week

[Guillaume Tucker]

On 02/06/2023 09:53, Guillaume Tucker wrote:
>>> Once the staging move is complete, all the production services
>>> are scheduled to be moved next week:
>>>
>>> * https://kernelci.org (static website)
>>> * https://linux.kernelci.org (web dashboard, backend, Jenkins etc.)
>> Also:
>>
>> * https://chromeos.kernelci.org (ChromeOS KernelCI instance)
>>
>>> There may be some longer downtime next Monday compared with usual
>>> weekly production deployments.  We'll also be sending regular
>>> updates as things evolve.
>>>
>>> Please let us know if this downtime is causing any critical
>>> disruption and we'll see which measures we can take to mitigate
>>> the impact.
> We've now started transferring VMs for the production move.  It's
> still scheduled to happen on Monday as per the usual production
> updates, except this time we won't be updating any of the code to
> focus solely on the infrastructure.  If things to well, the
> downtime shouldn't last more than a couple of hours.  We'll soon
> confirm the exact time window for the downtime and provide
> regular updates.

There is now a scheduled maintenance window on Tuesday 6th June
from 18:00 UTC to 22:00 UTC when the kernelci.org production
services will be down as part of the move to the new Azure
subscription.  If things go well, the downtime may be shorter.
There should be no downtime on Monday.

Thanks,
Guillaume

Re: Scheduled maintenance this week and next week

[Guillaume Tucker]

On 02/06/2023 19:08, Guillaume Tucker wrote:
> On 02/06/2023 09:53, Guillaume Tucker wrote:
>>>> Once the staging move is complete, all the production services
>>>> are scheduled to be moved next week:
>>>>
>>>> * https://kernelci.org (static website)
>>>> * https://linux.kernelci.org (web dashboard, backend, Jenkins etc.)
>>> Also:
>>>
>>> * https://chromeos.kernelci.org (ChromeOS KernelCI instance)
>>>
>>>> There may be some longer downtime next Monday compared with usual
>>>> weekly production deployments.  We'll also be sending regular
>>>> updates as things evolve.
>>>>
>>>> Please let us know if this downtime is causing any critical
>>>> disruption and we'll see which measures we can take to mitigate
>>>> the impact.
>> We've now started transferring VMs for the production move.  It's
>> still scheduled to happen on Monday as per the usual production
>> updates, except this time we won't be updating any of the code to
>> focus solely on the infrastructure.  If things to well, the
>> downtime shouldn't last more than a couple of hours.  We'll soon
>> confirm the exact time window for the downtime and provide
>> regular updates.
> 
> There is now a scheduled maintenance window on Tuesday 6th June
> from 18:00 UTC to 22:00 UTC when the kernelci.org production
> services will be down as part of the move to the new Azure
> subscription.  If things go well, the downtime may be shorter.
> There should be no downtime on Monday.

The planned maintenance window is meant to end within 1h, however
we've experienced some complications and it looks like it will
take longer to get all the services back online.  As it's nearing
completion we've opted for extending the maintenance window in
the hope that it gets resolved within the next couple of hours or
so.  We'll send an update once things have settled down.

Thank you for your understanding.

Best wishes,
Guillaume

Re: [kernelci-sysadmin] Scheduled maintenance this week and next week

[Vince Hillier]

On 6/6/23 17:17, Guillaume Tucker via groups.io wrote:
> On 02/06/2023 19:08, Guillaume Tucker wrote:
>> On 02/06/2023 09:53, Guillaume Tucker wrote:
>>>>> Once the staging move is complete, all the production services
>>>>> are scheduled to be moved next week:
>>>>>
>>>>> * https://kernelci.org (static website)
>>>>> * https://linux.kernelci.org (web dashboard, backend, Jenkins etc.)
>>>> Also:
>>>>
>>>> * https://chromeos.kernelci.org (ChromeOS KernelCI instance)
>>>>
>>>>> There may be some longer downtime next Monday compared with usual
>>>>> weekly production deployments.  We'll also be sending regular
>>>>> updates as things evolve.
>>>>>
>>>>> Please let us know if this downtime is causing any critical
>>>>> disruption and we'll see which measures we can take to mitigate
>>>>> the impact.
>>> We've now started transferring VMs for the production move.  It's
>>> still scheduled to happen on Monday as per the usual production
>>> updates, except this time we won't be updating any of the code to
>>> focus solely on the infrastructure.  If things to well, the
>>> downtime shouldn't last more than a couple of hours.  We'll soon
>>> confirm the exact time window for the downtime and provide
>>> regular updates.
>>
>> There is now a scheduled maintenance window on Tuesday 6th June
>> from 18:00 UTC to 22:00 UTC when the kernelci.org production
>> services will be down as part of the move to the new Azure
>> subscription.  If things go well, the downtime may be shorter.
>> There should be no downtime on Monday.
> 
> The planned maintenance window is meant to end within 1h, however
> we've experienced some complications and it looks like it will
> take longer to get all the services back online.  As it's nearing
> completion we've opted for extending the maintenance window in
> the hope that it gets resolved within the next couple of hours or
> so.  We'll send an update once things have settled down.
The production move has now been completed.  Thank you for your patience.

Vince Hillier |  President and Founder
Revenni Inc.  |  https://revenni.com  | 647-812-0131 x301

Re: Scheduled maintenance this week and next week

[Vince Hillier]

Vince Hillier |  President and Founder
Revenni Inc.  |  https://revenni.com  | 647-812-0131 x301

On 6/6/23 21:00, Vince Hillier wrote:
> On 6/6/23 17:17, Guillaume Tucker via groups.io wrote:
>> On 02/06/2023 19:08, Guillaume Tucker wrote:
>>> On 02/06/2023 09:53, Guillaume Tucker wrote:
>>>>>> Once the staging move is complete, all the production services
>>>>>> are scheduled to be moved next week:
>>>>>>
>>>>>> * https://kernelci.org (static website)
>>>>>> * https://linux.kernelci.org (web dashboard, backend, Jenkins etc.)
>>>>> Also:
>>>>>
>>>>> * https://chromeos.kernelci.org (ChromeOS KernelCI instance)
>>>>>
>>>>>> There may be some longer downtime next Monday compared with usual
>>>>>> weekly production deployments.  We'll also be sending regular
>>>>>> updates as things evolve.
>>>>>>
>>>>>> Please let us know if this downtime is causing any critical
>>>>>> disruption and we'll see which measures we can take to mitigate
>>>>>> the impact.
>>>> We've now started transferring VMs for the production move.  It's
>>>> still scheduled to happen on Monday as per the usual production
>>>> updates, except this time we won't be updating any of the code to
>>>> focus solely on the infrastructure.  If things to well, the
>>>> downtime shouldn't last more than a couple of hours.  We'll soon
>>>> confirm the exact time window for the downtime and provide
>>>> regular updates.
>>>
>>> There is now a scheduled maintenance window on Tuesday 6th June
>>> from 18:00 UTC to 22:00 UTC when the kernelci.org production
>>> services will be down as part of the move to the new Azure
>>> subscription.  If things go well, the downtime may be shorter.
>>> There should be no downtime on Monday.
>>
>> The planned maintenance window is meant to end within 1h, however
>> we've experienced some complications and it looks like it will
>> take longer to get all the services back online.  As it's nearing
>> completion we've opted for extending the maintenance window in
>> the hope that it gets resolved within the next couple of hours or
>> so.  We'll send an update once things have settled down.
 >
> The production move has now been completed.  Thank you for your patience.

As we reach the final stages of our migration we'll be moving the Jitsi 
instance at https://meet.kernel.social to our new subscription.  This 
will incur downtime for the service starting Monday June 12th at 5am UTC.

The downtime is dependent on a DNS change that will be effected after 
the migration effort.

Thanks,
Vince