* Re: IOMMU RMRR for Intel graphic device [not found] ` <237F54289DF84E4997F34151298ABEBC8760DA5D@SHSMSX101.ccr.corp.intel.com> @ 2019-05-10 5:21 ` Lu Baolu 2019-05-10 8:34 ` Daniel Vetter 0 siblings, 1 reply; 3+ messages in thread From: Lu Baolu @ 2019-05-10 5:21 UTC (permalink / raw) To: Zhang, Tina, Yuan, Hang, Tian, Kevin, Zhenyu Wang, Lu, Baolu, Liu, Yi L, Zhang, Xiong Y Cc: intel-gfx, baolu.lu Forward to i915 mailing list and post the question again so people knows what are the key concerns. The background is that Linux community is going to collect and report the reserved memory ranges of an assigned device to VFIO driver, and any mapped address will be checked against the reserved ranges. If there's any conflict, vifo will refuse the map request. Unfortunately, when it comes Intel graphic device, the conflict happens. That means address range mapped through vfio conflicts with the rmrr for graphic device. https://lkml.org/lkml/2018/6/5/760 The questions are 1) will the rmrr for graphic device still needs to be reserved for BIOS or firmware use, when a device is going to be assigned to user level? 2) if no, what's the check point, after which the rmrr is unnecessary anymore? Best regards, Lu Baolu On 5/9/19 5:14 PM, Zhang, Tina wrote: > Hi Baolu, > > +Xiong > > I think the root cause is that guest i915 needs to access RMRR. Xiong cooked a patch to disable the RMRR use in i915 guest, however that patch didn't get landed into the i915 upstream repo due to some concerns from i915 maintainers. Xiong can give us more backgrounds. > > So agreed, need to ask the i915 folk for this. > > BR, > Tina > > >> -----Original Message----- >> From: Yuan, Hang >> Sent: Thursday, May 9, 2019 4:07 PM >> To: Lu Baolu <baolu.lu@linux.intel.com>; Tian, Kevin <kevin.tian@intel.com>; >> Zhenyu Wang <zhenyuw@linux.intel.com>; Zhang, Tina >> <tina.zhang@intel.com>; Lu, Baolu <baolu.lu@intel.com>; Liu, Yi L >> <yi.l.liu@intel.com> >> Subject: RE: IOMMU RMRR for Intel graphic device >> >> Hi Baolu, as Kevin suggested, would you like to ask i915 people in their >> mailing list intel-gfx@lists.freedesktop.org? >> >> Regards, >> Henry >> >>> -----Original Message----- >>> From: Lu Baolu [mailto:baolu.lu@linux.intel.com] >>> Sent: Thursday, May 9, 2019 2:42 PM >>> To: Tian, Kevin <kevin.tian@intel.com>; Zhenyu Wang >>> <zhenyuw@linux.intel.com>; Zhang, Tina <tina.zhang@intel.com>; Yuan, >>> Hang <hang.yuan@intel.com>; Lu, Baolu <baolu.lu@intel.com>; Liu, Yi L >>> <yi.l.liu@intel.com> >>> Cc: baolu.lu@linux.intel.com >>> Subject: Re: IOMMU RMRR for Intel graphic device >>> >>> Hi, >>> >>> +Tina and Henry and cc more people >>> >>> The background is that Linux community is going to collect and report >>> the reserved memory ranges of an assigned device to VFIO driver, and >>> any mapped address will be checked against the reserved ranges. If >>> there's any conflict, vifo will refuse the map request. >>> >>> Unfortunately, when it comes Intel graphic device, the conflict happens. >>> That means address range mapped through vfio conflicts with the rmrr >>> for graphic device. >>> >>> https://lkml.org/lkml/2018/6/5/760 >>> >>> The questions are 1) will the rmrr for graphic device still needs to >>> be reserved for BIOS or firmware use, when a device is going to be >>> assigned to user level? 2) if no, what's the check point, after which >>> the rmrr is unnecessary anymore? >>> >>> Best regards, >>> Lu Baolu >>> >>> On 5/6/19 2:16 PM, Tian, Kevin wrote: >>>> this should better be asked to i915 guys, since it's not >>>> virtualization related. :-) >>>> >>>> One caveat, iirc, i915 driver tries to reuse stolen memory (covered >>>> by >>>> RMRR) even after boot time. take it as if another type of memory >>>> resource. If true I'm afraid this might be a gap to your proposal. >>>> >>>> Since nothing confidential, possibly you can directly discuss in >> community. >>>> >>>>> From: Lu Baolu [mailto:baolu.lu@linux.intel.com] >>>>> Sent: Thursday, May 2, 2019 2:45 PM >>>>> >>>>> Ping... >>>>> >>>>> Any comments? This has been postponed in the community for a long >>> time. >>>>> We need to response this as soon as possible. >>>>> >>>>> Best regards, >>>>> Lu Baolu >>>>> >>>>> On 4/29/19 1:19 PM, Lu Baolu wrote: >>>>>> Hi Zhenyu, >>>>>> >>>>>> As we discussed, BIOS always exports IOMMU reserved memory >> regions >>>>>> for (a.k.a. RMRR in vt-d spec) Intel integrated graphic device. >>>>>> This caused some problems when we pass-through such graphic >>>>>> devices to >>> user level. >>>>>> >>>>>> I am about to propose something to the community so that a RMRR >>>>>> for graphic devices could be explicitly canceled as long as the >>>>>> driver >>>>>> (i915 or vfio) knows that the RMRR will never be used by BIOS >> anymore. >>>>>> >>>>>> The same story happens for USB host controller devices. And since >>>>>> we know that BIOS will stop using that memory region as soon as >>>>>> the driver clears the SMI bits. >>>>>> >>>>>> So the question is, can graphic driver know when the RMRR for >>>>>> graphic could be canceled? >>>>>> >>>>>> Best regards, >>>>>> Lu Baolu >>>>>> _______________________________________________ Intel-gfx mailing list Intel-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/intel-gfx ^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: IOMMU RMRR for Intel graphic device 2019-05-10 5:21 ` IOMMU RMRR for Intel graphic device Lu Baolu @ 2019-05-10 8:34 ` Daniel Vetter 2019-05-16 1:51 ` Lu Baolu 0 siblings, 1 reply; 3+ messages in thread From: Daniel Vetter @ 2019-05-10 8:34 UTC (permalink / raw) To: Lu Baolu; +Cc: Liu, Yi L, Lu, Baolu, intel-gfx, Yuan, Hang On Fri, May 10, 2019 at 01:21:59PM +0800, Lu Baolu wrote: > Forward to i915 mailing list and post the question again so people knows > what are the key concerns. > > The background is that Linux community is going to collect and report > the reserved memory ranges of an assigned device to VFIO driver, and any > mapped address will be checked against the reserved ranges. If there's > any conflict, vifo will refuse the map request. > > Unfortunately, when it comes Intel graphic device, the conflict happens. > That means address range mapped through vfio conflicts with the rmrr for > graphic device. > > https://lkml.org/lkml/2018/6/5/760 > > The questions are 1) will the rmrr for graphic device still needs to be > reserved for BIOS or firmware use, when a device is going to be assigned > to user level? 2) if no, what's the check point, after which the rmrr is > unnecessary anymore? The gfx RMRR isn't for the bios, it's for the driver. It covers stolen memory, and we need it. There's various piles of hacks to disable use of stolen, but they're all somewhat fragile, and afaiui for huc/guc we need it, and huc is part of the uapi exposed by the driver/device combo on gen9+. Until our hw folks come up with a better design I think we're just stuck on this, iow you can't pass-thru modern intel igfx because of this. -Daniel > > Best regards, > Lu Baolu > > On 5/9/19 5:14 PM, Zhang, Tina wrote: > > Hi Baolu, > > > > +Xiong > > > > I think the root cause is that guest i915 needs to access RMRR. Xiong cooked a patch to disable the RMRR use in i915 guest, however that patch didn't get landed into the i915 upstream repo due to some concerns from i915 maintainers. Xiong can give us more backgrounds. > > > > So agreed, need to ask the i915 folk for this. > > > > BR, > > Tina > > > > > > > -----Original Message----- > > > From: Yuan, Hang > > > Sent: Thursday, May 9, 2019 4:07 PM > > > To: Lu Baolu <baolu.lu@linux.intel.com>; Tian, Kevin <kevin.tian@intel.com>; > > > Zhenyu Wang <zhenyuw@linux.intel.com>; Zhang, Tina > > > <tina.zhang@intel.com>; Lu, Baolu <baolu.lu@intel.com>; Liu, Yi L > > > <yi.l.liu@intel.com> > > > Subject: RE: IOMMU RMRR for Intel graphic device > > > > > > Hi Baolu, as Kevin suggested, would you like to ask i915 people in their > > > mailing list intel-gfx@lists.freedesktop.org? > > > > > > Regards, > > > Henry > > > > > > > -----Original Message----- > > > > From: Lu Baolu [mailto:baolu.lu@linux.intel.com] > > > > Sent: Thursday, May 9, 2019 2:42 PM > > > > To: Tian, Kevin <kevin.tian@intel.com>; Zhenyu Wang > > > > <zhenyuw@linux.intel.com>; Zhang, Tina <tina.zhang@intel.com>; Yuan, > > > > Hang <hang.yuan@intel.com>; Lu, Baolu <baolu.lu@intel.com>; Liu, Yi L > > > > <yi.l.liu@intel.com> > > > > Cc: baolu.lu@linux.intel.com > > > > Subject: Re: IOMMU RMRR for Intel graphic device > > > > > > > > Hi, > > > > > > > > +Tina and Henry and cc more people > > > > > > > > The background is that Linux community is going to collect and report > > > > the reserved memory ranges of an assigned device to VFIO driver, and > > > > any mapped address will be checked against the reserved ranges. If > > > > there's any conflict, vifo will refuse the map request. > > > > > > > > Unfortunately, when it comes Intel graphic device, the conflict happens. > > > > That means address range mapped through vfio conflicts with the rmrr > > > > for graphic device. > > > > > > > > https://lkml.org/lkml/2018/6/5/760 > > > > > > > > The questions are 1) will the rmrr for graphic device still needs to > > > > be reserved for BIOS or firmware use, when a device is going to be > > > > assigned to user level? 2) if no, what's the check point, after which > > > > the rmrr is unnecessary anymore? > > > > > > > > Best regards, > > > > Lu Baolu > > > > > > > > On 5/6/19 2:16 PM, Tian, Kevin wrote: > > > > > this should better be asked to i915 guys, since it's not > > > > > virtualization related. :-) > > > > > > > > > > One caveat, iirc, i915 driver tries to reuse stolen memory (covered > > > > > by > > > > > RMRR) even after boot time. take it as if another type of memory > > > > > resource. If true I'm afraid this might be a gap to your proposal. > > > > > > > > > > Since nothing confidential, possibly you can directly discuss in > > > community. > > > > > > > > > > > From: Lu Baolu [mailto:baolu.lu@linux.intel.com] > > > > > > Sent: Thursday, May 2, 2019 2:45 PM > > > > > > > > > > > > Ping... > > > > > > > > > > > > Any comments? This has been postponed in the community for a long > > > > time. > > > > > > We need to response this as soon as possible. > > > > > > > > > > > > Best regards, > > > > > > Lu Baolu > > > > > > > > > > > > On 4/29/19 1:19 PM, Lu Baolu wrote: > > > > > > > Hi Zhenyu, > > > > > > > > > > > > > > As we discussed, BIOS always exports IOMMU reserved memory > > > regions > > > > > > > for (a.k.a. RMRR in vt-d spec) Intel integrated graphic device. > > > > > > > This caused some problems when we pass-through such graphic > > > > > > > devices to > > > > user level. > > > > > > > > > > > > > > I am about to propose something to the community so that a RMRR > > > > > > > for graphic devices could be explicitly canceled as long as the > > > > > > > driver > > > > > > > (i915 or vfio) knows that the RMRR will never be used by BIOS > > > anymore. > > > > > > > > > > > > > > The same story happens for USB host controller devices. And since > > > > > > > we know that BIOS will stop using that memory region as soon as > > > > > > > the driver clears the SMI bits. > > > > > > > > > > > > > > So the question is, can graphic driver know when the RMRR for > > > > > > > graphic could be canceled? > > > > > > > > > > > > > > Best regards, > > > > > > > Lu Baolu > > > > > > > > _______________________________________________ > Intel-gfx mailing list > Intel-gfx@lists.freedesktop.org > https://lists.freedesktop.org/mailman/listinfo/intel-gfx -- Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch _______________________________________________ Intel-gfx mailing list Intel-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/intel-gfx ^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: IOMMU RMRR for Intel graphic device 2019-05-10 8:34 ` Daniel Vetter @ 2019-05-16 1:51 ` Lu Baolu 0 siblings, 0 replies; 3+ messages in thread From: Lu Baolu @ 2019-05-16 1:51 UTC (permalink / raw) To: Daniel Vetter; +Cc: Liu, Yi L, Lu, Baolu, intel-gfx, Yuan, Hang, baolu.lu Hi Daniel, On 5/10/19 4:34 PM, Daniel Vetter wrote: > On Fri, May 10, 2019 at 01:21:59PM +0800, Lu Baolu wrote: >> Forward to i915 mailing list and post the question again so people knows >> what are the key concerns. >> >> The background is that Linux community is going to collect and report >> the reserved memory ranges of an assigned device to VFIO driver, and any >> mapped address will be checked against the reserved ranges. If there's >> any conflict, vifo will refuse the map request. >> >> Unfortunately, when it comes Intel graphic device, the conflict happens. >> That means address range mapped through vfio conflicts with the rmrr for >> graphic device. >> >> https://lkml.org/lkml/2018/6/5/760 >> >> The questions are 1) will the rmrr for graphic device still needs to be >> reserved for BIOS or firmware use, when a device is going to be assigned >> to user level? 2) if no, what's the check point, after which the rmrr is >> unnecessary anymore? > > The gfx RMRR isn't for the bios, it's for the driver. It covers stolen > memory, and we need it. There's various piles of hacks to disable use of > stolen, but they're all somewhat fragile, and afaiui for huc/guc we need > it, and huc is part of the uapi exposed by the driver/device combo on > gen9+. > > Until our hw folks come up with a better design I think we're just stuck > on this, iow you can't pass-thru modern intel igfx because of this. > -Daniel > Thanks for the explanation. It's clear to me now. Best regards, Lu Baolu >> >> Best regards, >> Lu Baolu >> >> On 5/9/19 5:14 PM, Zhang, Tina wrote: >>> Hi Baolu, >>> >>> +Xiong >>> >>> I think the root cause is that guest i915 needs to access RMRR. Xiong cooked a patch to disable the RMRR use in i915 guest, however that patch didn't get landed into the i915 upstream repo due to some concerns from i915 maintainers. Xiong can give us more backgrounds. >>> >>> So agreed, need to ask the i915 folk for this. >>> >>> BR, >>> Tina >>> >>> >>>> -----Original Message----- >>>> From: Yuan, Hang >>>> Sent: Thursday, May 9, 2019 4:07 PM >>>> To: Lu Baolu <baolu.lu@linux.intel.com>; Tian, Kevin <kevin.tian@intel.com>; >>>> Zhenyu Wang <zhenyuw@linux.intel.com>; Zhang, Tina >>>> <tina.zhang@intel.com>; Lu, Baolu <baolu.lu@intel.com>; Liu, Yi L >>>> <yi.l.liu@intel.com> >>>> Subject: RE: IOMMU RMRR for Intel graphic device >>>> >>>> Hi Baolu, as Kevin suggested, would you like to ask i915 people in their >>>> mailing list intel-gfx@lists.freedesktop.org? >>>> >>>> Regards, >>>> Henry >>>> >>>>> -----Original Message----- >>>>> From: Lu Baolu [mailto:baolu.lu@linux.intel.com] >>>>> Sent: Thursday, May 9, 2019 2:42 PM >>>>> To: Tian, Kevin <kevin.tian@intel.com>; Zhenyu Wang >>>>> <zhenyuw@linux.intel.com>; Zhang, Tina <tina.zhang@intel.com>; Yuan, >>>>> Hang <hang.yuan@intel.com>; Lu, Baolu <baolu.lu@intel.com>; Liu, Yi L >>>>> <yi.l.liu@intel.com> >>>>> Cc: baolu.lu@linux.intel.com >>>>> Subject: Re: IOMMU RMRR for Intel graphic device >>>>> >>>>> Hi, >>>>> >>>>> +Tina and Henry and cc more people >>>>> >>>>> The background is that Linux community is going to collect and report >>>>> the reserved memory ranges of an assigned device to VFIO driver, and >>>>> any mapped address will be checked against the reserved ranges. If >>>>> there's any conflict, vifo will refuse the map request. >>>>> >>>>> Unfortunately, when it comes Intel graphic device, the conflict happens. >>>>> That means address range mapped through vfio conflicts with the rmrr >>>>> for graphic device. >>>>> >>>>> https://lkml.org/lkml/2018/6/5/760 >>>>> >>>>> The questions are 1) will the rmrr for graphic device still needs to >>>>> be reserved for BIOS or firmware use, when a device is going to be >>>>> assigned to user level? 2) if no, what's the check point, after which >>>>> the rmrr is unnecessary anymore? >>>>> >>>>> Best regards, >>>>> Lu Baolu >>>>> >>>>> On 5/6/19 2:16 PM, Tian, Kevin wrote: >>>>>> this should better be asked to i915 guys, since it's not >>>>>> virtualization related. :-) >>>>>> >>>>>> One caveat, iirc, i915 driver tries to reuse stolen memory (covered >>>>>> by >>>>>> RMRR) even after boot time. take it as if another type of memory >>>>>> resource. If true I'm afraid this might be a gap to your proposal. >>>>>> >>>>>> Since nothing confidential, possibly you can directly discuss in >>>> community. >>>>>> >>>>>>> From: Lu Baolu [mailto:baolu.lu@linux.intel.com] >>>>>>> Sent: Thursday, May 2, 2019 2:45 PM >>>>>>> >>>>>>> Ping... >>>>>>> >>>>>>> Any comments? This has been postponed in the community for a long >>>>> time. >>>>>>> We need to response this as soon as possible. >>>>>>> >>>>>>> Best regards, >>>>>>> Lu Baolu >>>>>>> >>>>>>> On 4/29/19 1:19 PM, Lu Baolu wrote: >>>>>>>> Hi Zhenyu, >>>>>>>> >>>>>>>> As we discussed, BIOS always exports IOMMU reserved memory >>>> regions >>>>>>>> for (a.k.a. RMRR in vt-d spec) Intel integrated graphic device. >>>>>>>> This caused some problems when we pass-through such graphic >>>>>>>> devices to >>>>> user level. >>>>>>>> >>>>>>>> I am about to propose something to the community so that a RMRR >>>>>>>> for graphic devices could be explicitly canceled as long as the >>>>>>>> driver >>>>>>>> (i915 or vfio) knows that the RMRR will never be used by BIOS >>>> anymore. >>>>>>>> >>>>>>>> The same story happens for USB host controller devices. And since >>>>>>>> we know that BIOS will stop using that memory region as soon as >>>>>>>> the driver clears the SMI bits. >>>>>>>> >>>>>>>> So the question is, can graphic driver know when the RMRR for >>>>>>>> graphic could be canceled? >>>>>>>> >>>>>>>> Best regards, >>>>>>>> Lu Baolu >>>>>>>> >> _______________________________________________ >> Intel-gfx mailing list >> Intel-gfx@lists.freedesktop.org >> https://lists.freedesktop.org/mailman/listinfo/intel-gfx > _______________________________________________ Intel-gfx mailing list Intel-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/intel-gfx ^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2019-05-16 1:58 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <6473ebb1-d44b-1dd7-19c6-cdd93c64fbca@linux.intel.com>
[not found] ` <2a12e17e-16b4-b48c-ea6b-1a25043ac2fe@linux.intel.com>
[not found] ` <AADFC41AFE54684AB9EE6CBC0274A5D19CA2004B@SHSMSX104.ccr.corp.intel.com>
[not found] ` <c425e8fc-d720-9a09-f52b-7a7d140098f7@linux.intel.com>
[not found] ` <C294FBAE55175941838A8D362DCD8AA225EEA42A@SHSMSX104.ccr.corp.intel.com>
[not found] ` <237F54289DF84E4997F34151298ABEBC8760DA5D@SHSMSX101.ccr.corp.intel.com>
2019-05-10 5:21 ` IOMMU RMRR for Intel graphic device Lu Baolu
2019-05-10 8:34 ` Daniel Vetter
2019-05-16 1:51 ` Lu Baolu
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox