From: Loic Prylli <lprylli@netflix.com>
Date: Wed, 10 Apr 2019 21:25:35 +0000 (-0700)
Subject: Fix wrap-aware isAfter function in Congestion.cxx
X-Git-Tag: v1.9.90~15^2
X-Git-Url: https://source.dussan.org/?a=commitdiff_plain;h=95c46d8b7b03b1e0fce5169e78474cb2163ccde4;p=tigervnc.git

Fix wrap-aware isAfter function in Congestion.cxx

Result of overflow on signed integer arithmetic is undefined in C/C++ standard.
So in previous version clang was compiling the statement as (int)a > (int)b (i.e. assuming no overflow), which leads to incorrect result.

Correct deterministic behavior means doing overflow arithmetic as unsigned, i.e.
 a != b && a - b <= UINT_MAX / 2
---

diff --git a/common/rfb/Congestion.cxx b/common/rfb/Congestion.cxx
index 4a784522..f3f9ceea 100644
--- a/common/rfb/Congestion.cxx
+++ b/common/rfb/Congestion.cxx
@@ -70,7 +70,7 @@ static const unsigned MAXIMUM_WINDOW = 4194304;
 
 // Compare position even when wrapped around
 static inline bool isAfter(unsigned a, unsigned b) {
-  return (int)a - (int)b > 0;
+  return a != b && a - b <= UINT_MAX / 2;
 }
 
 static LogWriter vlog("Congestion");