Re: [PATCH 5/5] drm/i915: l3 parity sysfs interface

[Ben Widawsky <ben@bwidawsk.net>]

On Fri, 25 May 2012 10:51:19 -0700
Jesse Barnes <jbarnes@virtuousgeek.org> wrote:

> On Fri, 27 Apr 2012 17:40:21 -0700
> Ben Widawsky <ben@bwidawsk.net> wrote:
> 
> > Dumb binary interfaces which allow root-only updates of the cache
> > remapping registers. As mentioned in a previous patch, software using
> > this interface needs to know about HW limits, and other programming
> > considerations as the kernel interface does no checking for these things
> > on the root-only interface.
> > 
> > v1: Drop extra posting reads (Chris)
> > Return negative values in the sysfs interfaces on errors (Chris)
> > 
> > Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
> > ---
> >  drivers/gpu/drm/i915/i915_sysfs.c |  128 ++++++++++++++++++++++++++++++++++++-
> >  1 file changed, 126 insertions(+), 2 deletions(-)
> > 
> > diff --git a/drivers/gpu/drm/i915/i915_sysfs.c b/drivers/gpu/drm/i915/i915_sysfs.c
> > index 79f8344..ed77cbf 100644
> > --- a/drivers/gpu/drm/i915/i915_sysfs.c
> > +++ b/drivers/gpu/drm/i915/i915_sysfs.c
> > @@ -29,6 +29,7 @@
> >  #include <linux/module.h>
> >  #include <linux/stat.h>
> >  #include <linux/sysfs.h>
> > +#include "intel_drv.h"
> >  #include "i915_drv.h"
> >  
> >  static u32 calc_residency(struct drm_device *dev, const u32 reg)
> > @@ -92,20 +93,143 @@ static struct attribute_group rc6_attr_group = {
> >  	.attrs =  rc6_attrs
> >  };
> >  
> > +static int l3_access_valid(struct drm_device *dev, loff_t offset)
> > +{
> > +	if (!IS_IVYBRIDGE(dev))
> > +		return -EPERM;
> 
> -ENODEV?
> 
> > +
> > +	if (offset % 4 != 0)
> > +		return -EPERM;
> 
> -EINVAL?
> 
> > +
> > +	if (offset >= GEN7_L3LOG_SIZE)
> > +		return -ENXIO;
> 
> -E2BIG or -ERANGE?
> 
> And maybe some debug messages here so people can debug their daemons in
> case some of the return values are identical.
> 

Good point, I've gone with EPERM, EINVAL, and ENXIO - so no more
duplication and added the debug message for good measure.

> > +
> > +	return 0;
> > +}
> > +
> > +static ssize_t
> > +i915_l3_read(struct file *filp, struct kobject *kobj,
> > +	     struct bin_attribute *attr, char *buf,
> > +	     loff_t offset, size_t count)
> > +{
> > +	struct device *dev = container_of(kobj, struct device, kobj);
> > +	struct drm_minor *dminor = container_of(dev, struct drm_minor, kdev);
> > +	struct drm_device *drm_dev = dminor->dev;
> > +	struct drm_i915_private *dev_priv = drm_dev->dev_private;
> > +	uint32_t misccpctl;
> > +	int i, ret;
> > +
> > +	ret = l3_access_valid(drm_dev, offset);
> > +	if (ret)
> > +		return ret;
> > +
> > +	ret = i915_mutex_lock_interruptible(drm_dev);
> > +	if (ret)
> > +		return ret;
> > +
> > +	misccpctl = I915_READ(GEN7_MISCCPCTL);
> > +	I915_WRITE(GEN7_MISCCPCTL, misccpctl & ~GEN7_DOP_CLOCK_GATE_ENABLE);
> > +
> > +	for (i = offset; count >= 4 && i < GEN7_L3LOG_SIZE; i += 4, count -= 4)
> > +		*((uint32_t *)(&buf[i])) = I915_READ(GEN7_L3LOG_BASE + i);
> > +
> > +	I915_WRITE(GEN7_MISCCPCTL, misccpctl);
> > +
> > +	mutex_unlock(&drm_dev->struct_mutex);
> > +
> > +	return i - offset;
> > +}
> > +
> > +static ssize_t
> > +i915_l3_write(struct file *filp, struct kobject *kobj,
> > +	      struct bin_attribute *attr, char *buf,
> > +	      loff_t offset, size_t count)
> > +{
> > +	struct device *dev = container_of(kobj, struct device, kobj);
> > +	struct drm_minor *dminor = container_of(dev, struct drm_minor, kdev);
> > +	struct drm_device *drm_dev = dminor->dev;
> > +	struct drm_i915_private *dev_priv = drm_dev->dev_private;
> > +	u32 *temp = NULL;
> > +	int i, ret;
> > +
> > +	ret = l3_access_valid(drm_dev, offset);
> > +	if (ret)
> > +		return ret;
> > +
> > +	ret = i915_mutex_lock_interruptible(drm_dev);
> > +	if (ret)
> > +		return ret;
> > +
> > +	if (!dev_priv->mm.l3_remap_info) {
> > +		temp = kzalloc(GEN7_L3LOG_SIZE, GFP_KERNEL);
> > +		if (!temp) {
> > +			mutex_unlock(&drm_dev->struct_mutex);
> > +			return -ENOMEM;
> > +		}
> > +	}
> 
> Grumble, lazy allocation.  But if we did this at driver load time we'd
> also need a valid bit so we knew whether to try to load it or not...
> 

Just curious what the grumble is about? It is a sysfs entry after all.

> > +
> > +	ret = i915_gpu_idle(drm_dev, true);
> > +	if (ret) {
> > +		kfree(temp);
> > +		mutex_unlock(&drm_dev->struct_mutex);
> > +		return ret;
> > +	}
> > +
> > +	/* TODO: Ideally we really want a GPU reset here to make sure errors
> > +	 * aren't propagated Since I cannot find a stable way to reset the GPU
> > +	 * at this point it is left as a TODO.
> > +	*/
> 
> Do we want a reset?  Or just to restart the app that consumed the
> error?  If the latter, then the current uevent ought to be enough,
> potentially with a configurable SIGBUS or something.
> 

I've thought about this a great deal. I think as long as every app
referencing the buffer is killed, and we do a full pipeline flush after
the remapping, we should be able to avoid a reset. I like the idea of
SIGBUS, but it's a bit of work to track which buffer is actually bad,
and with buffer re-use, potentially harder to find the application that
cared. Therefore, I think reset is the easiest way to just make every
app reset.

> > +
> > +	if (dev_priv->mm.l3_remap_info)
> > +		temp = dev_priv->mm.l3_remap_info;
> > +
> > +	dev_priv->mm.l3_remap_info = temp;
> 
> I'd combine these assignments with the kzalloc block above just to keep
> things together.
> 

Leftover cruft from when I had reset code. Thanks.

> > +
> > +	for (i = offset; count >= 4 && i < GEN7_L3LOG_SIZE; i += 4, count -= 4) {
> > +		u32 remap = I915_READ(GEN7_L3LOG_BASE + i);
> > +		if (remap && remap != *temp)
> > +			DRM_ERROR("0x%x was already programmed to %x\n",
> > +				  GEN7_L3LOG_BASE + i, remap);
> > +		*temp++ = *(uint32_t *)(&buf[i]);
> > +	}
> 
> Is this an error?  Or just a "userspace is schizo" and therefore
> just a DEBUG?
> 

You're right. Thanks.

> > +
> > +	i915_gem_l3_remap(drm_dev);
> 
> You could add the warning/debug info to l3_remap instead, right?
> 

Yes. Fixed.

> > +
> > +	mutex_unlock(&drm_dev->struct_mutex);
> > +
> > +	return offset - i;
> > +}
> > +
> > +static struct bin_attribute dpf_attrs = {
> > +	.attr = {.name = "l3_parity", .mode = (S_IRUSR | S_IWUSR)},
> > +	.size = GEN7_L3LOG_SIZE,
> > +	.read = i915_l3_read,
> > +	.write = i915_l3_write,
> > +	.mmap = NULL
> > +};
> > +
> >  void i915_setup_sysfs(struct drm_device *dev)
> >  {
> >  	int ret;
> >  
> > -	/* ILK doesn't have any residency information */
> > +	/* ILK and below don't yet have relevant sysfs files */
> >  	if (INTEL_INFO(dev)->gen < 6)
> >  		return;
> >  
> >  	ret = sysfs_merge_group(&dev->primary->kdev.kobj, &rc6_attr_group);
> >  	if (ret)
> > -		DRM_ERROR("sysfs setup failed\n");
> > +		DRM_ERROR("RC6 residency sysfs setup failed\n");
> > +
> > +	if (!IS_IVYBRIDGE(dev))
> > +		return;
> > +
> > +	ret = device_create_bin_file(&dev->primary->kdev, &dpf_attrs);
> > +	if (ret)
> > +		DRM_ERROR("l3 parity sysfs setup failed\n");
> >  }
> >  
> >  void i915_teardown_sysfs(struct drm_device *dev)
> >  {
> > +	device_remove_bin_file(&dev->primary->kdev,  &dpf_attrs);
> >  	sysfs_unmerge_group(&dev->primary->kdev.kobj, &rc6_attr_group);
> >  }
> 
> I think Daniel covered the above.

Yep.

> 
> With the above addressed:
> Reviewed-by: Jesse Barnes <jbarnes@virtuousgeek.org>
> 

Thanks.

-- 
Ben Widawsky, Intel Open Source Technology Center