From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:35002) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Y0VC9-0001dW-Np for qemu-devel@nongnu.org; Mon, 15 Dec 2014 07:56:18 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Y0VC4-0001H5-Fe for qemu-devel@nongnu.org; Mon, 15 Dec 2014 07:56:13 -0500 Received: from [2001:41d0:8:2b42::1] (port=49828 helo=greensocs.com) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Y0VC3-0001Gk-U7 for qemu-devel@nongnu.org; Mon, 15 Dec 2014 07:56:08 -0500 From: Mark Burton Content-Type: multipart/alternative; boundary="Apple-Mail=_61AE1891-031F-4EE2-9F9A-C151891E6A4F" Date: Mon, 15 Dec 2014 13:56:04 +0100 Message-Id: <5AD68BB8-35E0-4752-BD46-FA3CA9BC7F34@greensocs.com> Mime-Version: 1.0 (Mac OS X Mail 8.1 \(1993\)) Subject: [Qemu-devel] Atomic Instructions - comments please List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: mttcg@listserver.greensocs.com Cc: Developers qemu-devel --Apple-Mail=_61AE1891-031F-4EE2-9F9A-C151891E6A4F Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Comments please=E2=80=A6. Choices for Atomic instructions: The current approach (for ARM at least) for Ld and St exclusive inside = Qemu simply records the address and the value that atomic read = instructions attempt to read from. When an atomic write happens, it = checks the value and address remain the same, otherwise it fails. This just doesn=E2=80=99t match the architecturally defined = functionality. For instance, a write will succeed if another thread = intervenes with a write at the same address with the same (old) value. However - for most people, most of the time, this semantic seems to = work. One proposal is =E2=80=98simply=E2=80=99 to add a mutex around this = code, such that multi-threaded TCG will correctly update/read these = saved address/values. This _should_ maintain the status-quo. Things that were broken before = will remain broken, nothing new should break. The concern is that the = fact that the TCG was previously uni-threaded MAY be masking problems = with this code that we are not taking into account. A second proposal is to somehow re-use the dirty bit mechanism. However - the dirty bit approach seems to be too corse grained = (per page), and (I think) only covers writes, which seems like a = limitation which is not ideal=E2=80=A6. A third proposal is to mark pages as IO when a ld/st ex is performed to = them. Then to augment the memory API to indicate that a ld/st ex is in = hand, allowing the memory chain to decide if it should allow the write = or cause a fault. This would seem to be the closest approach to the real = H/W. However it means marking pages as IO (which could cause us issues = if there was code on that page?, or slow things down) - and it means = adding to the memory API. Cheers Mark. +44 (0)20 7100 3485 x 210 +33 (0)5 33 52 01 77x 210 +33 (0)603762104 mark.burton --Apple-Mail=_61AE1891-031F-4EE2-9F9A-C151891E6A4F Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8
Comments please=E2=80=A6.

Choices for Atomic = instructions:
The current approach (for ARM at = least) for Ld and St exclusive inside Qemu simply records the address = and the value that atomic read instructions attempt to read from. When = an atomic write happens, it checks the value and address remain the = same, otherwise it fails.

This just doesn=E2=80=99t match the architecturally defined = functionality. For instance, a write will succeed if another thread = intervenes with a write at the same address with the same (old) = value.

However = - for most people, most of the time, this semantic seems to = work.

One = proposal is =E2=80=98simply=E2=80=99 to add a mutex around this code, = such that multi-threaded TCG will correctly update/read these saved = address/values.
This _should_ maintain the = status-quo. Things that were broken before will remain broken, nothing = new should break. The concern is that the fact that the TCG was = previously uni-threaded MAY be masking problems with this code that we = are not taking into account.

A second proposal is to somehow re-use = the dirty bit mechanism.
However - = the dirty bit approach seems to be too corse grained (per page), and (I = think) only covers writes, which seems like a limitation which is not = ideal=E2=80=A6.

A third proposal is to mark pages as IO when a ld/st ex is = performed to them. Then to augment the memory API to indicate that a = ld/st ex is in hand, allowing the memory chain to decide if it should = allow the write or cause a fault. This would seem to be the closest = approach to the real H/W. However it means marking pages as IO (which = could cause us issues if there was code on that page?, or slow things = down) - and it means adding to the memory API.

Cheers

Mark.





 +44 (0)20 7100 3485 x = 210
 +33 (0)5 = 33 52 01 77x 210

+33 (0)603762104
mark.burton

= --Apple-Mail=_61AE1891-031F-4EE2-9F9A-C151891E6A4F-- From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:40412) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Y0VWI-0001zp-6v for qemu-devel@nongnu.org; Mon, 15 Dec 2014 08:17:08 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Y0VWC-0000Vs-1u for qemu-devel@nongnu.org; Mon, 15 Dec 2014 08:17:02 -0500 Received: from mx1.redhat.com ([209.132.183.28]:50202) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Y0VWB-0000Vm-QK for qemu-devel@nongnu.org; Mon, 15 Dec 2014 08:16:55 -0500 Message-ID: <548EDF3E.2070407@redhat.com> Date: Mon, 15 Dec 2014 14:16:46 +0100 From: Paolo Bonzini MIME-Version: 1.0 References: <5AD68BB8-35E0-4752-BD46-FA3CA9BC7F34@greensocs.com> In-Reply-To: <5AD68BB8-35E0-4752-BD46-FA3CA9BC7F34@greensocs.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Subject: Re: [Qemu-devel] Atomic Instructions - comments please List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Mark Burton , mttcg@greensocs.com Cc: Developers qemu-devel On 15/12/2014 13:56, Mark Burton wrote: > A third proposal is to mark pages as IO when a ld/st ex is performed to > them. Then to augment the memory API to indicate that a ld/st ex is in > hand, allowing the memory chain to decide if it should allow the write > or cause a fault. This would seem to be the closest approach to the real > H/W. However it means marking pages as IO (which could cause us issues > if there was code on that page?, or slow things down) - and it means > adding to the memory API. What are the intended semantics for reads/writes from the outside world? Should they cause the ll/sc pair to retry or not? If not, it should not need any change to the memory API; you can do it entirely within cputlb.c, roughly the same as the handling of TLB_NOTDIRTY. It also marks pages as I/O, but only internally within TCG. Paolo From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:41805) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Y0Vd5-0005cS-1w for qemu-devel@nongnu.org; Mon, 15 Dec 2014 08:24:09 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Y0Vcy-0002IT-5Q for qemu-devel@nongnu.org; Mon, 15 Dec 2014 08:24:02 -0500 Received: from mail-lb0-f172.google.com ([209.85.217.172]:36307) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Y0Vcx-0002IG-Rv for qemu-devel@nongnu.org; Mon, 15 Dec 2014 08:23:56 -0500 Received: by mail-lb0-f172.google.com with SMTP id u10so9200930lbd.3 for ; Mon, 15 Dec 2014 05:23:55 -0800 (PST) MIME-Version: 1.0 In-Reply-To: <548EDF3E.2070407@redhat.com> References: <5AD68BB8-35E0-4752-BD46-FA3CA9BC7F34@greensocs.com> <548EDF3E.2070407@redhat.com> From: Peter Maydell Date: Mon, 15 Dec 2014 13:23:34 +0000 Message-ID: Content-Type: text/plain; charset=UTF-8 Subject: Re: [Qemu-devel] Atomic Instructions - comments please List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Paolo Bonzini Cc: mttcg@greensocs.com, Mark Burton , Developers qemu-devel On 15 December 2014 at 13:16, Paolo Bonzini wrote: > On 15/12/2014 13:56, Mark Burton wrote: >> A third proposal is to mark pages as IO when a ld/st ex is performed to >> them. Then to augment the memory API to indicate that a ld/st ex is in >> hand, allowing the memory chain to decide if it should allow the write >> or cause a fault. This would seem to be the closest approach to the real >> H/W. However it means marking pages as IO (which could cause us issues >> if there was code on that page?, or slow things down) - and it means >> adding to the memory API. > > What are the intended semantics for reads/writes from the outside world? > Should they cause the ll/sc pair to retry or not? Assuming that by "outside world" you mean "some other CPU in this cluster" [in ARM-speak, some other observer in the same shareability domain]: writes should. Reads should not. -- PMM From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:42582) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Y0VhJ-0007qy-3j for qemu-devel@nongnu.org; Mon, 15 Dec 2014 08:28:31 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Y0VhC-0003Ur-R6 for qemu-devel@nongnu.org; Mon, 15 Dec 2014 08:28:25 -0500 Received: from mx1.redhat.com ([209.132.183.28]:37225) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Y0VhC-0003Ui-Jc for qemu-devel@nongnu.org; Mon, 15 Dec 2014 08:28:18 -0500 Message-ID: <548EE1E7.2090208@redhat.com> Date: Mon, 15 Dec 2014 14:28:07 +0100 From: Paolo Bonzini MIME-Version: 1.0 References: <5AD68BB8-35E0-4752-BD46-FA3CA9BC7F34@greensocs.com> <548EDF3E.2070407@redhat.com> In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] Atomic Instructions - comments please List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Peter Maydell Cc: mttcg@greensocs.com, Mark Burton , Developers qemu-devel On 15/12/2014 14:23, Peter Maydell wrote: > > What are the intended semantics for reads/writes from the outside world? > > Should they cause the ll/sc pair to retry or not? > Assuming that by "outside world" you mean "some other CPU in this > cluster" [in ARM-speak, some other observer in the same shareability > domain]: writes should. Reads should not. No, I mean from DMA. Other CPU of course need to abort reads/writes. The CPU that executes LL can invalidate the physical address on all other CPUs' TLBs. Attempts to write to that address then would fill the TLB entry with the new TLB_LL bit. Paolo From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:42594) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Y0VhP-0007sN-20 for qemu-devel@nongnu.org; Mon, 15 Dec 2014 08:28:35 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Y0VhK-0003WF-Af for qemu-devel@nongnu.org; Mon, 15 Dec 2014 08:28:30 -0500 Received: from mail-la0-f52.google.com ([209.85.215.52]:51647) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Y0VhK-0003W8-3M for qemu-devel@nongnu.org; Mon, 15 Dec 2014 08:28:26 -0500 Received: by mail-la0-f52.google.com with SMTP id hs14so9235661lab.25 for ; Mon, 15 Dec 2014 05:28:25 -0800 (PST) MIME-Version: 1.0 In-Reply-To: <5AD68BB8-35E0-4752-BD46-FA3CA9BC7F34@greensocs.com> References: <5AD68BB8-35E0-4752-BD46-FA3CA9BC7F34@greensocs.com> From: Peter Maydell Date: Mon, 15 Dec 2014 13:28:05 +0000 Message-ID: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Subject: Re: [Qemu-devel] Atomic Instructions - comments please List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Mark Burton Cc: mttcg@listserver.greensocs.com, Developers qemu-devel On 15 December 2014 at 12:56, Mark Burton wrote= : > One proposal is =E2=80=98simply=E2=80=99 to add a mutex around this code,= such > that multi-threaded TCG will correctly update/read these saved > address/values. > This _should_ maintain the status-quo. Things that were broken > before will remain broken, nothing new should break. The concern > is that the fact that the TCG was previously uni-threaded MAY be > masking problems with this code that we are not taking into account. Personally I would start out with this approach. We're going to need a "do this whole sequence atomically wrt other guest CPUs" mechanism anyway, so it's not implementing something we wouldn't otherwise need. And it's the simple thing to do. It's certainly possible to do a more architecturally correct ld/st exclusive implementation along the lines of how we manage TB invalidation with the dirty bitmap, but if we can do without it then we should try to keep the scope of this project constrained: it's a big enough job as it is. -- PMM From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:43352) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Y0Vl8-0004Rg-DG for qemu-devel@nongnu.org; Mon, 15 Dec 2014 08:32:28 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Y0Vl2-0004aR-8h for qemu-devel@nongnu.org; Mon, 15 Dec 2014 08:32:22 -0500 Received: from mx1.redhat.com ([209.132.183.28]:38923) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Y0Vl2-0004aK-2A for qemu-devel@nongnu.org; Mon, 15 Dec 2014 08:32:16 -0500 Message-ID: <548EE2D4.6070603@redhat.com> Date: Mon, 15 Dec 2014 14:32:04 +0100 From: Paolo Bonzini MIME-Version: 1.0 References: <5AD68BB8-35E0-4752-BD46-FA3CA9BC7F34@greensocs.com> In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] Atomic Instructions - comments please List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Peter Maydell , Mark Burton Cc: mttcg@greensocs.com, Developers qemu-devel On 15/12/2014 14:28, Peter Maydell wrote: > Personally I would start out with this approach. We're going to > need a "do this whole sequence atomically wrt other guest CPUs" > mechanism anyway, so it's not implementing something we wouldn't > otherwise need. And it's the simple thing to do. It's certainly > possible to do a more architecturally correct ld/st exclusive > implementation along the lines of how we manage TB invalidation > with the dirty bitmap, but if we can do without it then we > should try to keep the scope of this project constrained: it's > a big enough job as it is. How would "add a mutex" work unless you add a mutex or CAS to each and every qemu_st operation? Paolo From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:44031) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Y0Vod-0007zB-TY for qemu-devel@nongnu.org; Mon, 15 Dec 2014 08:36:04 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Y0VoZ-0005mJ-87 for qemu-devel@nongnu.org; Mon, 15 Dec 2014 08:35:59 -0500 Received: from [2001:41d0:8:2b42::1] (port=50348 helo=greensocs.com) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Y0VoY-0005m9-NZ for qemu-devel@nongnu.org; Mon, 15 Dec 2014 08:35:55 -0500 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 8.1 \(1993\)) From: Mark Burton In-Reply-To: Date: Mon, 15 Dec 2014 14:35:52 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <09D7CFCB-947D-4639-84DF-F93B99811242@greensocs.com> References: <5AD68BB8-35E0-4752-BD46-FA3CA9BC7F34@greensocs.com> Subject: Re: [Qemu-devel] Atomic Instructions - comments please List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Peter Maydell Cc: mttcg@listserver.greensocs.com, Developers qemu-devel We will roll a patch for this approach shortly. For the =E2=80=98better=E2=80=99 approach - I think it=E2=80=99s = something we will consider doing=E2=80=A6. but as you say, one thing at = a time. I dont think it will be too bad to implement, given what already exists = in the tlb=E2=80=99s - (except if we have to protect (for some = architecture or other) against non-atomic reads from an address marked = atomic, I think). I think we can treat this independently (unless we = discover that it won=E2=80=99t work without :-) ) Cheers Mark. > On 15 Dec 2014, at 14:28, Peter Maydell = wrote: >=20 > On 15 December 2014 at 12:56, Mark Burton = wrote: >> One proposal is =E2=80=98simply=E2=80=99 to add a mutex around this = code, such >> that multi-threaded TCG will correctly update/read these saved >> address/values. >> This _should_ maintain the status-quo. Things that were broken >> before will remain broken, nothing new should break. The concern >> is that the fact that the TCG was previously uni-threaded MAY be >> masking problems with this code that we are not taking into account. >=20 > Personally I would start out with this approach. We're going to > need a "do this whole sequence atomically wrt other guest CPUs" > mechanism anyway, so it's not implementing something we wouldn't > otherwise need. And it's the simple thing to do. It's certainly > possible to do a more architecturally correct ld/st exclusive > implementation along the lines of how we manage TB invalidation > with the dirty bitmap, but if we can do without it then we > should try to keep the scope of this project constrained: it's > a big enough job as it is. >=20 > -- PMM +44 (0)20 7100 3485 x 210 +33 (0)5 33 52 01 77x 210 +33 (0)603762104 mark.burton From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:44128) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Y0Vp7-0008Rr-0a for qemu-devel@nongnu.org; Mon, 15 Dec 2014 08:36:35 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Y0Vp0-0005sD-SQ for qemu-devel@nongnu.org; Mon, 15 Dec 2014 08:36:28 -0500 Received: from mail-lb0-f172.google.com ([209.85.217.172]:64884) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Y0Vp0-0005rE-LM for qemu-devel@nongnu.org; Mon, 15 Dec 2014 08:36:22 -0500 Received: by mail-lb0-f172.google.com with SMTP id u10so9207493lbd.17 for ; Mon, 15 Dec 2014 05:36:21 -0800 (PST) MIME-Version: 1.0 In-Reply-To: <548EE1E7.2090208@redhat.com> References: <5AD68BB8-35E0-4752-BD46-FA3CA9BC7F34@greensocs.com> <548EDF3E.2070407@redhat.com> <548EE1E7.2090208@redhat.com> From: Peter Maydell Date: Mon, 15 Dec 2014 13:36:00 +0000 Message-ID: Content-Type: text/plain; charset=UTF-8 Subject: Re: [Qemu-devel] Atomic Instructions - comments please List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Paolo Bonzini Cc: mttcg@greensocs.com, Mark Burton , Developers qemu-devel On 15 December 2014 at 13:28, Paolo Bonzini wrote: > > > On 15/12/2014 14:23, Peter Maydell wrote: >> > What are the intended semantics for reads/writes from the outside world? >> > Should they cause the ll/sc pair to retry or not? >> Assuming that by "outside world" you mean "some other CPU in this >> cluster" [in ARM-speak, some other observer in the same shareability >> domain]: writes should. Reads should not. > > No, I mean from DMA. Other CPU of course need to abort reads/writes. Depends whether the DMAing device is in the same shareability domain. Sufficiently clever DMA devices (probably ones which do cache-coherent DMA; GPUs spring to mind as a possibility) probably would; plain old DMA devices not. In practice anybody trying to do DMA onto an ll/sc mutex location is out of their mind and we can ignore this possibility. -- PMM From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:44366) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Y0Vq7-0001TR-7J for qemu-devel@nongnu.org; Mon, 15 Dec 2014 08:37:36 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Y0Vq1-00067Z-S7 for qemu-devel@nongnu.org; Mon, 15 Dec 2014 08:37:31 -0500 Received: from [2001:41d0:8:2b42::1] (port=50404 helo=greensocs.com) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Y0Vq1-00067R-MK for qemu-devel@nongnu.org; Mon, 15 Dec 2014 08:37:25 -0500 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 8.1 \(1993\)) From: Mark Burton In-Reply-To: <548EE2D4.6070603@redhat.com> Date: Mon, 15 Dec 2014 14:37:24 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <33AF7C69-101D-41C8-BF0D-EBEC60082872@greensocs.com> References: <5AD68BB8-35E0-4752-BD46-FA3CA9BC7F34@greensocs.com> <548EE2D4.6070603@redhat.com> Subject: Re: [Qemu-devel] Atomic Instructions - comments please List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Paolo Bonzini Cc: mttcg@listserver.greensocs.com, Peter Maydell , Developers qemu-devel (not address of mttcg list server) > On 15 Dec 2014, at 14:32, Paolo Bonzini wrote: >=20 >=20 >=20 > On 15/12/2014 14:28, Peter Maydell wrote: >> Personally I would start out with this approach. We're going to >> need a "do this whole sequence atomically wrt other guest CPUs" >> mechanism anyway, so it's not implementing something we wouldn't >> otherwise need. And it's the simple thing to do. It's certainly >> possible to do a more architecturally correct ld/st exclusive >> implementation along the lines of how we manage TB invalidation >> with the dirty bitmap, but if we can do without it then we >> should try to keep the scope of this project constrained: it's >> a big enough job as it is. >=20 > How would "add a mutex" work unless you add a mutex or CAS to each and > every qemu_st operation? it=E2=80=99ll do what it does now=E2=80=A6. e.g. it won=E2=80=99t = =E2=80=98work=E2=80=99 the way the architecture is defined=E2=80=A6. but = it will work as well as it does now=E2=80=A6. Cheers Mark >=20 > Paolo +44 (0)20 7100 3485 x 210 +33 (0)5 33 52 01 77x 210 +33 (0)603762104 mark.burton From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:45660) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Y0Vs3-0004X6-8W for qemu-devel@nongnu.org; Mon, 15 Dec 2014 08:39:36 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Y0Vry-0006zQ-Ip for qemu-devel@nongnu.org; Mon, 15 Dec 2014 08:39:31 -0500 Received: from mail-la0-f41.google.com ([209.85.215.41]:60314) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Y0Vry-0006yr-4P for qemu-devel@nongnu.org; Mon, 15 Dec 2014 08:39:26 -0500 Received: by mail-la0-f41.google.com with SMTP id hv19so9378125lab.0 for ; Mon, 15 Dec 2014 05:39:25 -0800 (PST) MIME-Version: 1.0 In-Reply-To: <548EE2D4.6070603@redhat.com> References: <5AD68BB8-35E0-4752-BD46-FA3CA9BC7F34@greensocs.com> <548EE2D4.6070603@redhat.com> From: Peter Maydell Date: Mon, 15 Dec 2014 13:39:05 +0000 Message-ID: Content-Type: text/plain; charset=UTF-8 Subject: Re: [Qemu-devel] Atomic Instructions - comments please List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Paolo Bonzini Cc: Mark Burton , Developers qemu-devel [I'm getting bounces from mttcg@greensocs.com so have taken them off cc: 550 5.1.1 : Recipient address rejected: User unknown in virtual mailbox table] On 15 December 2014 at 13:32, Paolo Bonzini wrote: > > > On 15/12/2014 14:28, Peter Maydell wrote: >> Personally I would start out with this approach. We're going to >> need a "do this whole sequence atomically wrt other guest CPUs" >> mechanism anyway, so it's not implementing something we wouldn't >> otherwise need. And it's the simple thing to do. It's certainly >> possible to do a more architecturally correct ld/st exclusive >> implementation along the lines of how we manage TB invalidation >> with the dirty bitmap, but if we can do without it then we >> should try to keep the scope of this project constrained: it's >> a big enough job as it is. > > How would "add a mutex" work unless you add a mutex or CAS to each and > every qemu_st operation? Same way our current approach works -- we simply don't implement "stores interrupt ll/sc operations": only a store-conditional can break a load-locked's lock. In practice this works ok because the stereotypical sequences that guests use don't rely on this part of the spec. -- PMM From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:46466) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Y0Vvk-0006yH-71 for qemu-devel@nongnu.org; Mon, 15 Dec 2014 08:43:24 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Y0Vvf-0008SN-Nd for qemu-devel@nongnu.org; Mon, 15 Dec 2014 08:43:20 -0500 Received: from [2001:41d0:8:2b42::1] (port=50476 helo=greensocs.com) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Y0Vvf-0008SI-6l for qemu-devel@nongnu.org; Mon, 15 Dec 2014 08:43:15 -0500 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 8.1 \(1993\)) From: Mark Burton In-Reply-To: Date: Mon, 15 Dec 2014 14:43:13 +0100 Content-Transfer-Encoding: 7bit Message-Id: <06117F83-92B8-4CA8-9328-4418B4E3FB83@greensocs.com> References: <5AD68BB8-35E0-4752-BD46-FA3CA9BC7F34@greensocs.com> <548EE2D4.6070603@redhat.com> Subject: Re: [Qemu-devel] Atomic Instructions - comments please List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Peter Maydell , mttcg@listserver.greensocs.com Cc: Paolo Bonzini , Developers qemu-devel > On 15 Dec 2014, at 14:39, Peter Maydell wrote: > > [I'm getting bounces from mttcg@greensocs.com so have taken > them off cc: > 550 5.1.1 : Recipient address rejected: User > unknown in virtual mailbox table] > the address should be: mttcg@listserver.greensocs.com Not sure who introduced the other address.. Cheers Mark. > On 15 December 2014 at 13:32, Paolo Bonzini wrote: >> >> >> On 15/12/2014 14:28, Peter Maydell wrote: >>> Personally I would start out with this approach. We're going to >>> need a "do this whole sequence atomically wrt other guest CPUs" >>> mechanism anyway, so it's not implementing something we wouldn't >>> otherwise need. And it's the simple thing to do. It's certainly >>> possible to do a more architecturally correct ld/st exclusive >>> implementation along the lines of how we manage TB invalidation >>> with the dirty bitmap, but if we can do without it then we >>> should try to keep the scope of this project constrained: it's >>> a big enough job as it is. >> >> How would "add a mutex" work unless you add a mutex or CAS to each and >> every qemu_st operation? > > Same way our current approach works -- we simply don't implement > "stores interrupt ll/sc operations": only a store-conditional > can break a load-locked's lock. In practice this works ok > because the stereotypical sequences that guests use don't rely > on this part of the spec. > > -- PMM +44 (0)20 7100 3485 x 210 +33 (0)5 33 52 01 77x 210 +33 (0)603762104 mark.burton From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:47252) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Y0VzX-000100-3X for qemu-devel@nongnu.org; Mon, 15 Dec 2014 08:47:21 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Y0VzQ-0001HN-64 for qemu-devel@nongnu.org; Mon, 15 Dec 2014 08:47:15 -0500 Received: from mx1.redhat.com ([209.132.183.28]:33371) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Y0VzP-0001HH-UX for qemu-devel@nongnu.org; Mon, 15 Dec 2014 08:47:08 -0500 Message-ID: <548EE653.9020409@redhat.com> Date: Mon, 15 Dec 2014 14:46:59 +0100 From: Paolo Bonzini MIME-Version: 1.0 References: <5AD68BB8-35E0-4752-BD46-FA3CA9BC7F34@greensocs.com> <548EE2D4.6070603@redhat.com> In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] Atomic Instructions - comments please List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Peter Maydell Cc: Mark Burton , Developers qemu-devel On 15/12/2014 14:39, Peter Maydell wrote: >> > >> > How would "add a mutex" work unless you add a mutex or CAS to each and >> > every qemu_st operation? > Same way our current approach works -- we simply don't implement > "stores interrupt ll/sc operations": only a store-conditional > can break a load-locked's lock. In practice this works ok > because the stereotypical sequences that guests use don't rely > on this part of the spec. Makes sense, thanks! Paolo From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52115) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Y0WLk-0002pP-69 for qemu-devel@nongnu.org; Mon, 15 Dec 2014 09:10:18 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Y0WLe-0003G1-0G for qemu-devel@nongnu.org; Mon, 15 Dec 2014 09:10:12 -0500 Received: from mail-la0-f45.google.com ([209.85.215.45]:50230) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Y0WLd-0003FM-QW for qemu-devel@nongnu.org; Mon, 15 Dec 2014 09:10:05 -0500 Received: by mail-la0-f45.google.com with SMTP id gq15so9397040lab.4 for ; Mon, 15 Dec 2014 06:10:05 -0800 (PST) MIME-Version: 1.0 In-Reply-To: <548EDF3E.2070407@redhat.com> References: <5AD68BB8-35E0-4752-BD46-FA3CA9BC7F34@greensocs.com> <548EDF3E.2070407@redhat.com> From: Peter Maydell Date: Mon, 15 Dec 2014 14:09:44 +0000 Message-ID: Content-Type: text/plain; charset=UTF-8 Subject: Re: [Qemu-devel] Atomic Instructions - comments please List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: mttcg@listserver.greensocs.com, QEMU Developers Cc: Paolo Bonzini , Mark Burton On 15 December 2014 at 13:16, Paolo Bonzini wrote: > If not, it should not need any change to the memory API; you can do it > entirely within cputlb.c, roughly the same as the handling of > TLB_NOTDIRTY. It also marks pages as I/O, but only internally within TCG. Speaking of TLB_NOTDIRTY, I just wrote up a summary of how that works for a private email, so I figured I might as well send it here too so it's in the qemu-devel mail archives; it's probably not new information to anybody involved in this immediate conversation. How we arrange to throw away cached translations when the guest writes to that part of memory: * we have two data structures effectively tracking dirty status: (1) there are a set of bitmaps which track different kinds of dirtiness (the DIRTY_MEMORY_*); the functions for manipulating these are mostly in ram_addr.h. One of the bitmaps is for DIRTY_MEMORY_CODE. (2) where we have an entry in the QEMU TLB for a page which is backed by host RAM, we may set the TLB_NOTDIRTY bit in the addr_write TLB entry field (TLB_NOTDIRTY is one of several low order bits that can be set in what is otherwise a page-aligned virtual address in the TLB structure. TLB_MMIO is another, indicating that the entry is not RAM at all.) TLB entries come and go, but the bitmaps cover all of physical RAM. When a TLB entry is present then the NOTDIRTY flag should be just a cache for "at least one of the dirty bitmaps says this page is not dirty". * when we generate code we call tlb_protect_code() (from tb_alloc_page()): this calls cpu_physical_memory_reset_dirty(), which both updates the dirty bitmap data structure (marking the region as clean in the DIRTY_MEMORY_CODE bitmap) and also calls cpu_tlb_reset_dirty_all() to OR in the TLB_NOTDIRTY flag for any present TLB entries in the range * when we add an entry to the TLB, tlb_set_page() will OR in the TLB_NOTDIRTY bit if the bitmap says this is clean memory, so the two structures stay in sync * tlb_set_page() also calls memory_region_section_get_iotlb() to get an iotlb entry for this RAM, which is what will be used on the slow path. For RAM this will be io_mem_notdirty. * if the guest attempts a read, we don't do anything special because this uses addr_read, not addr_write * for a guest write, the generated code will look at addr_write; it takes the fast path if the low order bits are clear (indicating dirty host RAM). Otherwise we take the slow path (clean RAM, MMIO, nothing present, etc etc). * we then follow the slow path without special casing RAM, which means we'll use the iotlb entry set up when the TLB entry was populated, which is io_mem_notdirty. * notdirty_mem_write() will invalidate the cached TBs if the DIRTY_MEMORY_CODE bitmap says this memory is clean, and do the access the slow way. We then mark the TLB entry as dirty by calling tlb_set_dirty, so next time we'll take the fast path. (There's an optimisation wrinkle here: tb_invalidate_phys_page_fast() is complicated because it tries to avoid simply nuking every TB in the page. So it might need to keep accesses on the slow path. It only calls tlb_unprotect_code_phys() to update the DIRTY_MEMORY_CODE bitmap if every TB on the page has been invalidated. This is why notdirty_mem_write()'s call to tlb_set_dirty() is conditional.) * writes to already-dirty memory can take the fast path, which just writes to the host RAM without calling out or checking any dirty bits. Note that for linux-user mode the mechanism is totally different, because we don't have a softmmu TLB data structure; instead we use mprotect to write-protect the page, and then in the SIGSEGV handler we may throw away cached TBs before un-write-protecting it. -- PMM From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:58518) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Y0aAw-0006Hp-Io for qemu-devel@nongnu.org; Mon, 15 Dec 2014 13:15:24 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Y0aAq-0000tf-FJ for qemu-devel@nongnu.org; Mon, 15 Dec 2014 13:15:18 -0500 Received: from roura.ac.upc.es ([147.83.33.10]:39478) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Y0aAq-0000t3-43 for qemu-devel@nongnu.org; Mon, 15 Dec 2014 13:15:12 -0500 From: =?utf-8?Q?Llu=C3=ADs_Vilanova?= References: <5AD68BB8-35E0-4752-BD46-FA3CA9BC7F34@greensocs.com> <09D7CFCB-947D-4639-84DF-F93B99811242@greensocs.com> Date: Mon, 15 Dec 2014 12:15:02 -0600 In-Reply-To: <09D7CFCB-947D-4639-84DF-F93B99811242@greensocs.com> (Mark Burton's message of "Mon, 15 Dec 2014 14:35:52 +0100") Message-ID: <87d27kok49.fsf@fimbulvetr.bsc.es> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Subject: Re: [Qemu-devel] Atomic Instructions - comments please List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Mark Burton Cc: mttcg@greensocs.com, Peter Maydell , Developers qemu-devel Mark Burton writes: > We will roll a patch for this approach shortly. > For the =E2=80=98better=E2=80=99 approach - I think it=E2=80=99s somethin= g we will consider doing=E2=80=A6. but as you say, one thing at a time. > I dont think it will be too bad to implement, given what already exists i= n the > tlb=E2=80=99s - (except if we have to protect (for some architecture or o= ther) against > non-atomic reads from an address marked atomic, I think). I think we can= treat > this independently (unless we discover that it won=E2=80=99t work without= :-) ) I think something similar was discussed before, but you can also hybridize = the current approach with mprotect. The main problem I see is that every CPU TLB will have to mmap+mprotect a host page (a different host page for every gue= st CPU page table, but not necessarily every guest CPU). On the bright side, that will avoid locks in the fast path. Instead of lock= ing, the fast path can rely on mprotect and only check if the target page contai= ns data or non-translated code in the case of writes. Otherwise you must resor= t to locking, but that can be moved to a non-faulting slow path. This should leave a fast path with the same complexity of today's, but I'm = not sure about the amount of overheads this would add to other paths (and how frequent they'll be). Best, Lluis > Cheers > Mark. >> On 15 Dec 2014, at 14:28, Peter Maydell wrote: >>=20 >> On 15 December 2014 at 12:56, Mark Burton wr= ote: >>> One proposal is =E2=80=98simply=E2=80=99 to add a mutex around this cod= e, such >>> that multi-threaded TCG will correctly update/read these saved >>> address/values. >>> This _should_ maintain the status-quo. Things that were broken >>> before will remain broken, nothing new should break. The concern >>> is that the fact that the TCG was previously uni-threaded MAY be >>> masking problems with this code that we are not taking into account. >>=20 >> Personally I would start out with this approach. We're going to >> need a "do this whole sequence atomically wrt other guest CPUs" >> mechanism anyway, so it's not implementing something we wouldn't >> otherwise need. And it's the simple thing to do. It's certainly >> possible to do a more architecturally correct ld/st exclusive >> implementation along the lines of how we manage TB invalidation >> with the dirty bitmap, but if we can do without it then we >> should try to keep the scope of this project constrained: it's >> a big enough job as it is. >>=20 >> -- PMM --=20 "And it's much the same thing with knowledge, for whenever you learn something new, the whole world becomes that much richer." -- The Princess of Pure Reason, as told by Norton Juster in The Phantom Tollbooth