vncserver: Display check should confirm UNIX domain socket is still valid

If the X server is not shut down cleanly, it can leave UNIX domain
sockets around that "vncserver" has previously mis-identified as
evidence that the display number is still in use.

Instead of checking for existence of /tmp/.X11-unix/X<n>, the code
will now attempt to connect to the socket to confirm that there is a
server process listening on this UNIX domain socket. This will
eliminate false positives in the case the UNIX domain socket still
exists but is not associated with a listening Xorg server. The Xorg
server does not have a problem with fixing this file when it next
starts up. It is only important to avoid using the port if it is still
in use.

1 files changed, 8 insertions, 3 deletions

diff --git a/unix/vncserver/vncserver.in b/unix/vncserver/vncserver.in
index 6229ea38..25fbbd31 100755
--- a/unix/vncserver/vncserver.in
+++ b/unix/vncserver/vncserver.in
@@ -393,9 +393,14 @@ sub CheckDisplayNumber
     my $x11_unix_domain = "/tmp/.X11-unix/X$n";
 
     if (-e $x11_unix_domain) {
-	warn "\nWarning: $host:$n is taken because of $x11_unix_domain\n";
-	warn "Remove this file if there is no X server $host:$n\n";
-	return 0;
+        # Connect to UNIX domain socket to confirm it is not in use.
+        socket(S, PF_UNIX, SOCK_STREAM, 0) || die "$prog: socket failed: $!\n";
+        if (connect(S, sockaddr_un($x11_unix_domain))) {
+            # UNIX domain socket is in use.
+            close(S);
+            return 0;
+        }
+        close(S);
     }
 
     return 1;