[Qemu-devel] [PULL 3/3] vnc-enc-tight: Fix divide-by-zero in tight_detect_smooth_image{16, 24, 32}

[Gerd Hoffmann <kraxel@redhat.com>]

From: Gonglei <arei.gonglei@huawei.com>

Spotted by Coverity:

(1) Event assignment:  Assigning: "pixels" = "0".
(2) Event cond_true:  Condition "y < h", taking true branch
(3) Event cond_false:  Condition "x < w", taking false branch
(4) Event loop_end:  Reached end of loop
(5) Event divide_by_zero:  In expression "(stats[0] + stats[1]) * 100U / pixels",
division by expression "pixels" which may be zero has undefined behavior.

290     DEFINE_DETECT_FUNCTION(16)
291     DEFINE_DETECT_FUNCTION(32)

Signed-off-by: Gonglei <arei.gonglei@huawei.com>
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
 ui/vnc-enc-tight.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/ui/vnc-enc-tight.c b/ui/vnc-enc-tight.c
index 59b59c0..f02352c 100644
--- a/ui/vnc-enc-tight.c
+++ b/ui/vnc-enc-tight.c
@@ -181,6 +181,10 @@ tight_detect_smooth_image24(VncState *vs, int w, int h)
         }
     }
 
+    if (pixels == 0) {
+        return 0;
+    }
+
     /* 95% smooth or more ... */
     if (stats[0] * 33 / pixels >= 95) {
         return 0;
@@ -267,7 +271,9 @@ tight_detect_smooth_image24(VncState *vs, int w, int h)
                 y += w;                                                 \
             }                                                           \
         }                                                               \
-                                                                        \
+        if (pixels == 0) {                                              \
+            return 0;                                                   \
+        }                                                               \
         if ((stats[0] + stats[1]) * 100 / pixels >= 90) {               \
             return 0;                                                   \
         }                                                               \
-- 
1.8.3.1