Comment 8 for bug 634571

The problem actually lies in gtk-vnc. Running vinagre with the --gtk-vnc-debug flag shows:

(vinagre:23947): gvnc-DEBUG: vncconnection.c Protocol initialization
(vinagre:23947): gvnc-DEBUG: vncconnection.c Server version: 3.7
(vinagre:23947): gvnc-DEBUG: vncconnection.c Using version: 3.7
(vinagre:23947): gvnc-DEBUG: vncconnection.c Possible auth 18
(vinagre:23947): gvnc-DEBUG: vncconnection.c Possible auth 1
(vinagre:23947): gvnc-DEBUG: vncconnection.c Emit main context 10
(vinagre:23947): gvnc-DEBUG: vncconnection.c Thinking about auth type 18
(vinagre:23947): gvnc-DEBUG: vncconnection.c Decided on auth type 18
(vinagre:23947): gvnc-DEBUG: vncconnection.c Waiting for auth type
(vinagre:23947): gvnc-DEBUG: vncconnection.c Choose auth 18
(vinagre:23947): gvnc-DEBUG: vncconnection.c Do TLS handshake
(vinagre:23947): gvnc-DEBUG: vncconnection.c Handshake was blocking
...
(vinagre:23947): gvnc-DEBUG: vncconnection.c Completed TLS setup
(vinagre:23947): gvnc-DEBUG: vncconnection.c Got 1 subauths
(vinagre:23947): gvnc-DEBUG: vncconnection.c Got 1 subauths
(vinagre:23947): gvnc-DEBUG: vncconnection.c Possible sub-auth 1
(vinagre:23947): gvnc-DEBUG: vncconnection.c Emit main context 10
(vinagre:23947): gvnc-DEBUG: vncconnection.c Thinking about auth type 1
(vinagre:23947): gvnc-DEBUG: vncconnection.c Auth failed

"emit main context 10" means it is doing VNC_AUTH_CHOOSE_TYPE. It calls this twice, but instead should be calling VNC_AUTH_CHOOSE_TYPE followed by VNC_AUTH_CHOOSE_SUBTYPE like it does in Lucid (ie the second call should be saying "emit main context 12").

The following hack in vncconnection.c works around the problem and allows connection:

gboolean vnc_connection_set_auth_type(VncConnection *conn, unsigned int type)
{
 VncConnectionPrivate *priv = conn->priv;

        VNC_DEBUG("Thinking about auth type %u", type);

        // We are incorrectly called twice, once for auth and then again for subtype auth.
        // Because the auth_type is set (eg to TLS), the connection immediately fails.
        // This hack works around the problem:
        if (priv->auth_type != VNC_CONNECTION_AUTH_INVALID) {
            return vnc_connection_set_auth_subtype(conn, type);
        }

        if (priv->auth_type != VNC_CONNECTION_AUTH_INVALID) {