x11vnc throws "X Error of failed request: BadMatch" after upgrade from 17.04

Question

I'm trying to invoke VNC from a remote terminal and no matter the command line I submit I get:

X Error of failed request:  BadMatch (invalid parameter attributes)  
Major opcode of failed request:  73 (X_GetImage)   
Serial number of failed request:  40   
Current serial number in output stream:  40

The session starts like normal...

owen@Thor:~$ x11vnc -storpasswd
05/11/2017 12:01:57 passing arg to libvncserver: -storpasswd
05/11/2017 12:01:57 x11vnc version: 0.9.13 lastmod: 2011-08-10  pid: 13865
05/11/2017 12:01:57 XOpenDisplay("") failed.
05/11/2017 12:01:57 Trying again with XAUTHLOCALHOSTNAME=localhost ...
05/11/2017 12:01:57
05/11/2017 12:01:57 *** XOpenDisplay failed. No -display or DISPLAY.
05/11/2017 12:01:57 *** Trying ":0" in 4 seconds.  Press Ctrl-C to abort.
05/11/2017 12:01:57 *** 1 2 3 4
05/11/2017 12:02:01 *** XOpenDisplay of ":0" successful.
05/11/2017 12:02:01
05/11/2017 12:02:01 Using X display :0
05/11/2017 12:02:01 rootwin: 0x270 reswin: 0x2a00001 dpy: 0xf3ac80f0
05/11/2017 12:02:01
05/11/2017 12:02:01 ------------------ USEFUL INFORMATION ------------------
--snip--
05/11/2017 12:02:01 X FBPM extension not supported.
05/11/2017 12:02:01 X display is not capable of DPMS.
05/11/2017 12:02:01 --------------------------------------------------------
05/11/2017 12:02:01
05/11/2017 12:02:01 Default visual ID: 0x24
X Error of failed request:  BadMatch (invalid parameter attributes)
  Major opcode of failed request:  73 (X_GetImage)
  Serial number of failed request:  40
  Current serial number in output stream:  40

Solution

It seems one of your machines is running on wayland as display manager and many applications including x11vnc depend heavily on Xorg/Xserver and fail to work on wayland.

So a switching to Xorg ad display manager should fix your problem. This you can do by logging out and change your display manager by clicking on the small cogwheel beside the sign in button. (See screen-shot)

Comments

[Videonauth]

What is the result of echo $XDG_SESSION_TYPE on both machines?

[HellionWisp]

It's not populated in the server's ssh session, I'm not sure about in a terminal fired within the X session, and the client is Windows. I set it to "x11" and the error remains the same.

[HellionWisp]

@Videonauth You were on the right track ;). The problem is related to Wayland. I changed to Xorg and x11vnc works again. The problem is not resolved, but I'll live with this workaround.

[HellionWisp]

This is the correct answer!

[Gabriel Glenn]

Your are correct. If you wish to do it without access to the X session, edit etc/gdm3/custom.conf and uncomment #WaylandEnable=false by ssh. @see How do you switch from Wayland back to Xorg in Ubuntu 17.10?

[Forty3]

5 years later on 20.04 - still the only answer that worked. If anyone happens to find a vncserver that works with Wayland, I am all ears!