qemu bsd-user plans

qemu bsd-user plans

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 3270 bytes --]

The FreeBSD project has rewritten bsd-user. We've been working on this for
quite some time (the earliest commits date from 2013). Maybe a dozen people
have worked on this over time, and there's 3 or 4 active developers focused
on FreeBSD changes at the moment.

For a while, we'd merge in upstream changes from qemu. This worked great
for us, but left us with a big backlog that was hard to upstream. Each of
the updates took some time, so we got a little behind.

So, a few years ago, I spent several weeks converting the tangled merge
mess into a set of linear patches and started moving that forward. This was
around the time 4.0 was released. I only managed to get the rebase forward
to 3.1 release at the time before I hit problems related to poor testing
environment making it hard to verify newer versions were still working.
Plus, we found a few bugs that took a while to resolve for a number of
reasons. Now that they are resolved, we're able to use qemu-bsd-user to
build ~30k packages for arm, and ~20k for different types of mips in
FreeBSD "ports" system. We now have great confidence that it's working well
again.

Now that those bugs are resolved, I started trying to forward-port the
two-year-old base and immediately found myself hitting a number of
problems. A big problem was that I was re-doing a lot of work that was due
to innoculous changes upstream that I wouldn't have to do if the bsd-user
changes were upstream. These changes get in the way of dealing with the
more substantial structural changes in qemu that have happened.

There had been talk of doing a remove and replace update of bsd-user. This
talk was before I managed to rebase things as far forward as 3.1 even. This
appealed to me because we've accumulated about 150 patches to date, many
quite large, and curating them into a set of maybe 400 or 500 changes to
match the size and scope of most patches I've seen posted to qemu-devel
seemed overwhelming.

However, it's been another year since that plan was hatched, and it's
become clear to me that plan won't end in success. The closest I've been
able to get is 3.1 when 4.1 was current (about 6 months behind). It's time
for a new plan.

So, my new plan is to rebase what changes I can to the tip of master and
submit those for review. I'll work with the developers on the FreeBSD side
to ensure they are included in reviews in addition to the normal qemu-devel
list. This will allow us to pare down the deltas between our code and
upstream to allow us to make progress. The changes will be held to the
standard 'makes things better'. Given how broken bsd-user is today in qemu
upstream, at first that will a very easy standard to make.

The first patch I'll submit will be changing MAINTAINERS to point to me,
since I'm acting as the point person in this effort. I'll then re-submit
some other changes that I've submitted in the past, but CC the FreeBSD
folks that are currently active (they were only CC'd to former developers
who lack the time to review).

But before I get too far down this path, I thought I'd send out what's
going on to qemu-devel so I can get feedback and adjust the plan into
something that's mutually agreeable so time I put towards this is not
wasted.

So, what do people think of these plans?

Warner

[-- Attachment #2: Type: text/html, Size: 3480 bytes --]

Re: qemu bsd-user plans

[Peter Maydell]

Adding the people to the CC list who were on the previous discussion thread...

-- PMM


On Fri, 8 Jan 2021 at 19:43, Warner Losh <imp@bsdimp.com> wrote:
>
> The FreeBSD project has rewritten bsd-user. We've been working on this for quite some time (the earliest commits date from 2013). Maybe a dozen people have worked on this over time, and there's 3 or 4 active developers focused on FreeBSD changes at the moment.
>
> For a while, we'd merge in upstream changes from qemu. This worked great for us, but left us with a big backlog that was hard to upstream. Each of the updates took some time, so we got a little behind.
>
> So, a few years ago, I spent several weeks converting the tangled merge mess into a set of linear patches and started moving that forward. This was around the time 4.0 was released. I only managed to get the rebase forward to 3.1 release at the time before I hit problems related to poor testing environment making it hard to verify newer versions were still working. Plus, we found a few bugs that took a while to resolve for a number of reasons. Now that they are resolved, we're able to use qemu-bsd-user to build ~30k packages for arm, and ~20k for different types of mips in FreeBSD "ports" system. We now have great confidence that it's working well again.
>
> Now that those bugs are resolved, I started trying to forward-port the two-year-old base and immediately found myself hitting a number of problems. A big problem was that I was re-doing a lot of work that was due to innoculous changes upstream that I wouldn't have to do if the bsd-user changes were upstream. These changes get in the way of dealing with the more substantial structural changes in qemu that have happened.
>
> There had been talk of doing a remove and replace update of bsd-user. This talk was before I managed to rebase things as far forward as 3.1 even. This appealed to me because we've accumulated about 150 patches to date, many quite large, and curating them into a set of maybe 400 or 500 changes to match the size and scope of most patches I've seen posted to qemu-devel seemed overwhelming.
>
> However, it's been another year since that plan was hatched, and it's become clear to me that plan won't end in success. The closest I've been able to get is 3.1 when 4.1 was current (about 6 months behind). It's time for a new plan.
>
> So, my new plan is to rebase what changes I can to the tip of master and submit those for review. I'll work with the developers on the FreeBSD side to ensure they are included in reviews in addition to the normal qemu-devel list. This will allow us to pare down the deltas between our code and upstream to allow us to make progress. The changes will be held to the standard 'makes things better'. Given how broken bsd-user is today in qemu upstream, at first that will a very easy standard to make.
>
> The first patch I'll submit will be changing MAINTAINERS to point to me, since I'm acting as the point person in this effort. I'll then re-submit some other changes that I've submitted in the past, but CC the FreeBSD folks that are currently active (they were only CC'd to former developers who lack the time to review).
>
> But before I get too far down this path, I thought I'd send out what's going on to qemu-devel so I can get feedback and adjust the plan into something that's mutually agreeable so time I put towards this is not wasted.
>
> So, what do people think of these plans?
>
> Warner

Re: qemu bsd-user plans

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 3741 bytes --]

And add a couple of FreeBSD people I also forgot to CC.

Warner

On Fri, Jan 8, 2021 at 12:56 PM Peter Maydell <peter.maydell@linaro.org>
wrote:

> Adding the people to the CC list who were on the previous discussion
> thread...
>
> -- PMM
>
>
> On Fri, 8 Jan 2021 at 19:43, Warner Losh <imp@bsdimp.com> wrote:
> >
> > The FreeBSD project has rewritten bsd-user. We've been working on this
> for quite some time (the earliest commits date from 2013). Maybe a dozen
> people have worked on this over time, and there's 3 or 4 active developers
> focused on FreeBSD changes at the moment.
> >
> > For a while, we'd merge in upstream changes from qemu. This worked great
> for us, but left us with a big backlog that was hard to upstream. Each of
> the updates took some time, so we got a little behind.
> >
> > So, a few years ago, I spent several weeks converting the tangled merge
> mess into a set of linear patches and started moving that forward. This was
> around the time 4.0 was released. I only managed to get the rebase forward
> to 3.1 release at the time before I hit problems related to poor testing
> environment making it hard to verify newer versions were still working.
> Plus, we found a few bugs that took a while to resolve for a number of
> reasons. Now that they are resolved, we're able to use qemu-bsd-user to
> build ~30k packages for arm, and ~20k for different types of mips in
> FreeBSD "ports" system. We now have great confidence that it's working well
> again.
> >
> > Now that those bugs are resolved, I started trying to forward-port the
> two-year-old base and immediately found myself hitting a number of
> problems. A big problem was that I was re-doing a lot of work that was due
> to innoculous changes upstream that I wouldn't have to do if the bsd-user
> changes were upstream. These changes get in the way of dealing with the
> more substantial structural changes in qemu that have happened.
> >
> > There had been talk of doing a remove and replace update of bsd-user.
> This talk was before I managed to rebase things as far forward as 3.1 even.
> This appealed to me because we've accumulated about 150 patches to date,
> many quite large, and curating them into a set of maybe 400 or 500 changes
> to match the size and scope of most patches I've seen posted to qemu-devel
> seemed overwhelming.
> >
> > However, it's been another year since that plan was hatched, and it's
> become clear to me that plan won't end in success. The closest I've been
> able to get is 3.1 when 4.1 was current (about 6 months behind). It's time
> for a new plan.
> >
> > So, my new plan is to rebase what changes I can to the tip of master and
> submit those for review. I'll work with the developers on the FreeBSD side
> to ensure they are included in reviews in addition to the normal qemu-devel
> list. This will allow us to pare down the deltas between our code and
> upstream to allow us to make progress. The changes will be held to the
> standard 'makes things better'. Given how broken bsd-user is today in qemu
> upstream, at first that will a very easy standard to make.
> >
> > The first patch I'll submit will be changing MAINTAINERS to point to me,
> since I'm acting as the point person in this effort. I'll then re-submit
> some other changes that I've submitted in the past, but CC the FreeBSD
> folks that are currently active (they were only CC'd to former developers
> who lack the time to review).
> >
> > But before I get too far down this path, I thought I'd send out what's
> going on to qemu-devel so I can get feedback and adjust the plan into
> something that's mutually agreeable so time I put towards this is not
> wasted.
> >
> > So, what do people think of these plans?
> >
> > Warner
>

[-- Attachment #2: Type: text/html, Size: 4282 bytes --]

Re: qemu bsd-user plans

[Kyle Evans]

On Fri, 8 Jan 2021 at 19:43, Warner Losh <imp@bsdimp.com> wrote:
>
> The FreeBSD project has rewritten bsd-user. We've been working on this for quite some time (the earliest commits date from 2013). Maybe a dozen people have worked on this over time, and there's 3 or 4 active developers focused on FreeBSD changes at the moment.
>
> For a while, we'd merge in upstream changes from qemu. This worked great for us, but left us with a big backlog that was hard to upstream. Each of the updates took some time, so we got a little behind.
>
> So, a few years ago, I spent several weeks converting the tangled merge mess into a set of linear patches and started moving that forward. This was around the time 4.0 was released. I only managed to get the rebase forward to 3.1 release at the time before I hit problems related to poor testing environment making it hard to verify newer versions were still working. Plus, we found a few bugs that took a while to resolve for a number of reasons. Now that they are resolved, we're able to use qemu-bsd-user to build ~30k packages for arm, and ~20k for different types of mips in FreeBSD "ports" system. We now have great confidence that it's working well again.
>
> Now that those bugs are resolved, I started trying to forward-port the two-year-old base and immediately found myself hitting a number of problems. A big problem was that I was re-doing a lot of work that was due to innoculous changes upstream that I wouldn't have to do if the bsd-user changes were upstream. These changes get in the way of dealing with the more substantial structural changes in qemu that have happened.
>
> There had been talk of doing a remove and replace update of bsd-user. This talk was before I managed to rebase things as far forward as 3.1 even. This appealed to me because we've accumulated about 150 patches to date, many quite large, and curating them into a set of maybe 400 or 500 changes to match the size and scope of most patches I've seen posted to qemu-devel seemed overwhelming.
>
> However, it's been another year since that plan was hatched, and it's become clear to me that plan won't end in success. The closest I've been able to get is 3.1 when 4.1 was current (about 6 months behind). It's time for a new plan.
>

As one of the developers on the FreeBSD side, I agree with this. In
addition to the reasons cited, reintroducing it would really take a
lot more time and effort and I'm not convinced it would ever be
completed because, IMO, going that route should really entail
redesigning it from the ground-up based on an abstraction of
linux-user. I find that right now I'm playing a lot of catch-up
because we seem to be largely copied from linux-user without later
improvement; there is a really healthy amount of platform-independent
stuff that really should end up ultimately shared in a qemu-user or
something to that effect to mitigate duplication of effort.

> So, my new plan is to rebase what changes I can to the tip of master and submit those for review. I'll work with the developers on the FreeBSD side to ensure they are included in reviews in addition to the normal qemu-devel list. This will allow us to pare down the deltas between our code and upstream to allow us to make progress. The changes will be held to the standard 'makes things better'. Given how broken bsd-user is today in qemu upstream, at first that will a very easy standard to make.
>
> The first patch I'll submit will be changing MAINTAINERS to point to me, since I'm acting as the point person in this effort. I'll then re-submit some other changes that I've submitted in the past, but CC the FreeBSD folks that are currently active (they were only CC'd to former developers who lack the time to review).
>

Thanks for taking this on!


Kyle Evans

Re: qemu bsd-user plans

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 4312 bytes --]

On Sat, Jan 9, 2021 at 10:03 AM Kyle Evans <kevans@freebsd.org> wrote:

> On Fri, 8 Jan 2021 at 19:43, Warner Losh <imp@bsdimp.com> wrote:
> >
> > The FreeBSD project has rewritten bsd-user. We've been working on this
> for quite some time (the earliest commits date from 2013). Maybe a dozen
> people have worked on this over time, and there's 3 or 4 active developers
> focused on FreeBSD changes at the moment.
> >
> > For a while, we'd merge in upstream changes from qemu. This worked great
> for us, but left us with a big backlog that was hard to upstream. Each of
> the updates took some time, so we got a little behind.
> >
> > So, a few years ago, I spent several weeks converting the tangled merge
> mess into a set of linear patches and started moving that forward. This was
> around the time 4.0 was released. I only managed to get the rebase forward
> to 3.1 release at the time before I hit problems related to poor testing
> environment making it hard to verify newer versions were still working.
> Plus, we found a few bugs that took a while to resolve for a number of
> reasons. Now that they are resolved, we're able to use qemu-bsd-user to
> build ~30k packages for arm, and ~20k for different types of mips in
> FreeBSD "ports" system. We now have great confidence that it's working well
> again.
> >
> > Now that those bugs are resolved, I started trying to forward-port the
> two-year-old base and immediately found myself hitting a number of
> problems. A big problem was that I was re-doing a lot of work that was due
> to innoculous changes upstream that I wouldn't have to do if the bsd-user
> changes were upstream. These changes get in the way of dealing with the
> more substantial structural changes in qemu that have happened.
> >
> > There had been talk of doing a remove and replace update of bsd-user.
> This talk was before I managed to rebase things as far forward as 3.1 even.
> This appealed to me because we've accumulated about 150 patches to date,
> many quite large, and curating them into a set of maybe 400 or 500 changes
> to match the size and scope of most patches I've seen posted to qemu-devel
> seemed overwhelming.
> >
> > However, it's been another year since that plan was hatched, and it's
> become clear to me that plan won't end in success. The closest I've been
> able to get is 3.1 when 4.1 was current (about 6 months behind). It's time
> for a new plan.
> >
>
> As one of the developers on the FreeBSD side, I agree with this. In
> addition to the reasons cited, reintroducing it would really take a
> lot more time and effort and I'm not convinced it would ever be
> completed because, IMO, going that route should really entail
> redesigning it from the ground-up based on an abstraction of
> linux-user. I find that right now I'm playing a lot of catch-up
> because we seem to be largely copied from linux-user without later
> improvement; there is a really healthy amount of platform-independent
> stuff that really should end up ultimately shared in a qemu-user or
> something to that effect to mitigate duplication of effort.
>

Yes. Once we have things upstream the normal 'refactoring' work can be done
there, rather than in a side repo where it will quickly go stale. Also,
once it's in upstream, then the NetBSD and OpenBSD folks can come back to
the table with a much smaller delta to get things working again than they
have today.


> > So, my new plan is to rebase what changes I can to the tip of master and
> submit those for review. I'll work with the developers on the FreeBSD side
> to ensure they are included in reviews in addition to the normal qemu-devel
> list. This will allow us to pare down the deltas between our code and
> upstream to allow us to make progress. The changes will be held to the
> standard 'makes things better'. Given how broken bsd-user is today in qemu
> upstream, at first that will a very easy standard to make.
> >
> > The first patch I'll submit will be changing MAINTAINERS to point to me,
> since I'm acting as the point person in this effort. I'll then re-submit
> some other changes that I've submitted in the past, but CC the FreeBSD
> folks that are currently active (they were only CC'd to former developers
> who lack the time to review).
> >
>
> Thanks for taking this on!
>

You bet.

Warner

[-- Attachment #2: Type: text/html, Size: 5078 bytes --]

Re: qemu bsd-user plans

[Thomas Huth]

On 08/01/2021 20.56, Peter Maydell wrote:
> Adding the people to the CC list who were on the previous discussion thread...
> 
> -- PMM
> 
> 
> On Fri, 8 Jan 2021 at 19:43, Warner Losh <imp@bsdimp.com> wrote:
[...]
>> So, my new plan is to rebase what changes I can to the tip of master and submit those for review. I'll work with the developers on the FreeBSD side to ensure they are included in reviews in addition to the normal qemu-devel list. This will allow us to pare down the deltas between our code and upstream to allow us to make progress. The changes will be held to the standard 'makes things better'. Given how broken bsd-user is today in qemu upstream, at first that will a very easy standard to make.
>>
>> The first patch I'll submit will be changing MAINTAINERS to point to me, since I'm acting as the point person in this effort. I'll then re-submit some other changes that I've submitted in the past, but CC the FreeBSD folks that are currently active (they were only CC'd to former developers who lack the time to review).
>>
>> But before I get too far down this path, I thought I'd send out what's going on to qemu-devel so I can get feedback and adjust the plan into something that's mutually agreeable so time I put towards this is not wasted.
>>
>> So, what do people think of these plans?

Sounds like a good way forward. Are you familiar with sending git pull 
requests? I think once the patch to MAINTAINERS has been accepted, it's 
maybe best if you send out pull request on your own once the patches got 
reviewed on the mailing list - otherwise it might get cumbersome to find a 
maintainers tree where the patches can go through.

  Thomas

Re: qemu bsd-user plans

[Daniel P. Berrangé]

On Fri, Jan 08, 2021 at 12:41:30PM -0700, Warner Losh wrote:
> The FreeBSD project has rewritten bsd-user. We've been working on this for
> quite some time (the earliest commits date from 2013). Maybe a dozen people
> have worked on this over time, and there's 3 or 4 active developers focused
> on FreeBSD changes at the moment.

The fact that you have 3-4 people involved in this work is will be very
helpful to you going forward with a QEMU maintenance.

The biggest problem with getting code merged into QEMU is an insufficient
number of reviewers for the amount of patches sent. Since we have a rule
that patches need a review from someone else who isn't the author, if there
are two people with expertize to review patches in a given QEMU subsystem,
then they can become self-sufficient and review each others patches on
qemu-devel, which then makes merging much more productive.

If anyone wants to be automatically CC'd on patches for bsd-user for the
purposes of acting as a designated reviewer, they can added to MAINTAINERS
file to, alongside the primary maintainer(s).


> So, my new plan is to rebase what changes I can to the tip of master and
> submit those for review. I'll work with the developers on the FreeBSD side
> to ensure they are included in reviews in addition to the normal qemu-devel
> list. This will allow us to pare down the deltas between our code and
> upstream to allow us to make progress. The changes will be held to the
> standard 'makes things better'. Given how broken bsd-user is today in qemu
> upstream, at first that will a very easy standard to make.
>
> The first patch I'll submit will be changing MAINTAINERS to point to me,
> since I'm acting as the point person in this effort. I'll then re-submit
> some other changes that I've submitted in the past, but CC the FreeBSD
> folks that are currently active (they were only CC'd to former developers
> who lack the time to review).

> But before I get too far down this path, I thought I'd send out what's
> going on to qemu-devel so I can get feedback and adjust the plan into
> something that's mutually agreeable so time I put towards this is not
> wasted.

No objections from me. Since current bsd-user is orphaned, largely
unusable, and you're volunteering your time to make it better, I'm
supportive of whatever you believe is the most time efficient way
to improve bsd-user.

I presume some of the current QEMU maintainers knowledgable about
linux-user will be able to review the patches, and as mentioned
above, if other BSD devs currently active in bsd-user work can
also provide reviews on qemu-devel that'll be useful long term.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: qemu bsd-user plans

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 3820 bytes --]

On Mon, Jan 11, 2021 at 6:27 AM Daniel P. Berrangé <berrange@redhat.com>
wrote:

> On Fri, Jan 08, 2021 at 12:41:30PM -0700, Warner Losh wrote:
> > The FreeBSD project has rewritten bsd-user. We've been working on this
> for
> > quite some time (the earliest commits date from 2013). Maybe a dozen
> people
> > have worked on this over time, and there's 3 or 4 active developers
> focused
> > on FreeBSD changes at the moment.
>
> The fact that you have 3-4 people involved in this work is will be very
> helpful to you going forward with a QEMU maintenance.
>
> The biggest problem with getting code merged into QEMU is an insufficient
> number of reviewers for the amount of patches sent. Since we have a rule
> that patches need a review from someone else who isn't the author, if there
> are two people with expertize to review patches in a given QEMU subsystem,
> then they can become self-sufficient and review each others patches on
> qemu-devel, which then makes merging much more productive.
>

Yes. That's my hope too. We've been doing this internally for changes we've
been landing lately, so I think expanding our process to also do this with
qemu upstream will be a natural progression...


> If anyone wants to be automatically CC'd on patches for bsd-user for the
> purposes of acting as a designated reviewer, they can added to MAINTAINERS
> file to, alongside the primary maintainer(s).
>

I'll be adding myself, at least, to MAINTAINERS and encouraging the others
that have been working on this to take the time to review when I post. They
are quite willing, but may lack the time, alas, so I'll do what I can to
time the patches such that at least one of them has the time in a
reasonable time frame...


> > So, my new plan is to rebase what changes I can to the tip of master and
> > submit those for review. I'll work with the developers on the FreeBSD
> side
> > to ensure they are included in reviews in addition to the normal
> qemu-devel
> > list. This will allow us to pare down the deltas between our code and
> > upstream to allow us to make progress. The changes will be held to the
> > standard 'makes things better'. Given how broken bsd-user is today in
> qemu
> > upstream, at first that will a very easy standard to make.
> >
> > The first patch I'll submit will be changing MAINTAINERS to point to me,
> > since I'm acting as the point person in this effort. I'll then re-submit
> > some other changes that I've submitted in the past, but CC the FreeBSD
> > folks that are currently active (they were only CC'd to former developers
> > who lack the time to review).
>
> > But before I get too far down this path, I thought I'd send out what's
> > going on to qemu-devel so I can get feedback and adjust the plan into
> > something that's mutually agreeable so time I put towards this is not
> > wasted.
>
> No objections from me. Since current bsd-user is orphaned, largely
> unusable, and you're volunteering your time to make it better, I'm
> supportive of whatever you believe is the most time efficient way
> to improve bsd-user.
>
> I presume some of the current QEMU maintainers knowledgable about
> linux-user will be able to review the patches, and as mentioned
> above, if other BSD devs currently active in bsd-user work can
> also provide reviews on qemu-devel that'll be useful long term.
>

Yes. Many of the patches are copied from there, as well as the initial
version coming largely from there as well (it seems, I wasn't around at the
time and base this entirely on code similarities).

The FreeBSD project makes such heavy use of this, that I really want to
find some way we can stay current and maybe even have better abstractions
that make all user-mode emulation easier...

Warner

[-- Attachment #2: Type: text/html, Size: 4738 bytes --]