From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753441AbdKSPBb (ORCPT ); Sun, 19 Nov 2017 10:01:31 -0500 Received: from mail.linuxfoundation.org ([140.211.169.12]:54376 "EHLO mail.linuxfoundation.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752561AbdKSPB1 (ORCPT ); Sun, 19 Nov 2017 10:01:27 -0500 From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Gianfranco Costamagna , Michael Thayer , Hans de Goede Subject: [PATCH 4.14 29/31] staging: vboxvideo: Fix reporting invalid suggested-offset-properties Date: Sun, 19 Nov 2017 16:00:03 +0100 Message-Id: <20171119145952.313699606@linuxfoundation.org> X-Mailer: git-send-email 2.15.0 In-Reply-To: <20171119145951.136379453@linuxfoundation.org> References: <20171119145951.136379453@linuxfoundation.org> User-Agent: quilt/0.65 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org 4.14-stable review patch. If anyone has any objections, please let me know. ------------------ From: Hans de Goede commit ce10d7b4e8e3574b9616e54a09d64521b9aeb8b6 upstream. The x and y hints receives from the host are unsigned 32 bit integers and they get set to -1 (0xffffffff) when invalid. Before this commit the vboxvideo driver was storing them in an u16 causing the -1 to be truncated to 65535 which, once reported to userspace, was breaking gnome 3.26+ in Wayland mode. This commit stores the host values in 32 bit variables, removing the truncation and checks for -1, replacing it with 0 as -1 is not a valid suggested-offset-property value. Likewise the properties are now initialized to 0 instead of -1, since -1 is not a valid value. This fixes gnome 3.26+ in Wayland mode not working with the vboxvideo driver. Reported-by: Gianfranco Costamagna Cc: Michael Thayer Signed-off-by: Hans de Goede Signed-off-by: Greg Kroah-Hartman --- drivers/staging/vboxvideo/vbox_drv.h | 8 ++++---- drivers/staging/vboxvideo/vbox_irq.c | 4 ++-- drivers/staging/vboxvideo/vbox_mode.c | 26 ++++++++++++++++++-------- 3 files changed, 24 insertions(+), 14 deletions(-) --- a/drivers/staging/vboxvideo/vbox_drv.h +++ b/drivers/staging/vboxvideo/vbox_drv.h @@ -137,8 +137,8 @@ struct vbox_connector { char name[32]; struct vbox_crtc *vbox_crtc; struct { - u16 width; - u16 height; + u32 width; + u32 height; bool disconnected; } mode_hint; }; @@ -150,8 +150,8 @@ struct vbox_crtc { unsigned int crtc_id; u32 fb_offset; bool cursor_enabled; - u16 x_hint; - u16 y_hint; + u32 x_hint; + u32 y_hint; }; struct vbox_encoder { --- a/drivers/staging/vboxvideo/vbox_irq.c +++ b/drivers/staging/vboxvideo/vbox_irq.c @@ -150,8 +150,8 @@ static void vbox_update_mode_hints(struc disconnected = !(hints->enabled); crtc_id = vbox_conn->vbox_crtc->crtc_id; - vbox_conn->mode_hint.width = hints->cx & 0x8fff; - vbox_conn->mode_hint.height = hints->cy & 0x8fff; + vbox_conn->mode_hint.width = hints->cx; + vbox_conn->mode_hint.height = hints->cy; vbox_conn->vbox_crtc->x_hint = hints->dx; vbox_conn->vbox_crtc->y_hint = hints->dy; vbox_conn->mode_hint.disconnected = disconnected; --- a/drivers/staging/vboxvideo/vbox_mode.c +++ b/drivers/staging/vboxvideo/vbox_mode.c @@ -553,12 +553,22 @@ static int vbox_get_modes(struct drm_con ++num_modes; } vbox_set_edid(connector, preferred_width, preferred_height); - drm_object_property_set_value( - &connector->base, vbox->dev->mode_config.suggested_x_property, - vbox_connector->vbox_crtc->x_hint); - drm_object_property_set_value( - &connector->base, vbox->dev->mode_config.suggested_y_property, - vbox_connector->vbox_crtc->y_hint); + + if (vbox_connector->vbox_crtc->x_hint != -1) + drm_object_property_set_value(&connector->base, + vbox->dev->mode_config.suggested_x_property, + vbox_connector->vbox_crtc->x_hint); + else + drm_object_property_set_value(&connector->base, + vbox->dev->mode_config.suggested_x_property, 0); + + if (vbox_connector->vbox_crtc->y_hint != -1) + drm_object_property_set_value(&connector->base, + vbox->dev->mode_config.suggested_y_property, + vbox_connector->vbox_crtc->y_hint); + else + drm_object_property_set_value(&connector->base, + vbox->dev->mode_config.suggested_y_property, 0); return num_modes; } @@ -640,9 +650,9 @@ static int vbox_connector_init(struct dr drm_mode_create_suggested_offset_properties(dev); drm_object_attach_property(&connector->base, - dev->mode_config.suggested_x_property, -1); + dev->mode_config.suggested_x_property, 0); drm_object_attach_property(&connector->base, - dev->mode_config.suggested_y_property, -1); + dev->mode_config.suggested_y_property, 0); drm_connector_register(connector); drm_mode_connector_attach_encoder(connector, encoder);