How should I debug the error "Could not grab keyboard. A malicious client may be eavesdropping on your session."

Question

I am running an installation of Ubuntu 14.04 that I setup more than 6 months. About a week ago, I started getting an error message:

Could not grab keyboard. A malicious client may be eavesdropping on your session.

I have only ever seen it when returning to my computer after being away for quite a while (usually overnight). Several times it has prevented the screen from locking after the set timeout (I have begun actively locking it before leaving).

I am using a usb keyboard (Kinesis Advantage) directly connected to a usb port on the motherboard. I am using a wireless ELECOM mouse.

I am going to try disconnecting the mouse dongle before leaving. How else could I identify if there is a malicious client tracking my keystrokes or if this is a connectivity issue?

Solution 1

Here's how to solve your mystery. The goal is more to teach users "how to fish" by using standard Ubuntu utilities to dig into the details of any process on their system.

Step #1 (for curiosity mostly): identify which program is giving you this error:

# -- You may need to search under more dirs, YMMV
#    List files (incl. binaries) which contain the warning string

$ sudo grep -ral 'malicious client may be eavesdropping' /usr /bin /lib
/usr/lib/openssh/gnome-ssh-askpass

In my env the only program which contains this warning string in its binary is gnome-ssh-askpass. I could search if there's a bug filed on this particular program, and even download its source apt-get source ssh-askpass-gnome (note that the package name is different than the program name) for further inspection.

However, I suspect the root cause is not a problem in gnome-ssh-askpass. Since gnome-ssh-askpass is asking for your passphrase, its developers simply chose to err on the side of caution when failing to grab the keyboard, assume the worst-case scenario, and make the message sound uber-paranoid. But note that typing your passphrase or password into some random web-site dialog-box by accident is probably not a good idea, so in that sense the gnome-ssh-askpass developers have made the right call.

Recently more and more web sites have started engaging in the practice of displaying a popup, fading everything else outside the popup dialog, and aggressively grabbing focus. This might be the root cause for gnome-ssh-askpass failing to grab the keyboard. If your browser being open on such site, closing the browser or navigating away from the aggressive web site may help. If this is the cause, you may be interested in a desktop setting preventing individual processes from grabbing the complete (full desktop) focus. In KDE for example, this setting can be found under (System-Settings -> Window-behavior -> Focus -> Focus stealing prevention). If you feel really paranoid, I would recommend setting it to High or Extreme. Of course, this may also prevent gnome-ssh-askpass itself from grabbing the keyboard, or more accurately: grabbing the X focus.

Step #2: Identify suspicious processes:

Knowing that in Unix, devices appear like files under /dev, the next question is what device represents "the keyboard" in the file-system hierarchy. We can use the lsof (list open files) utility for this.

# look for processes holding devices open, filter out some common ones:
$ sudo lsof | grep /dev | grep -vE '/(null|urandom|zero)'

Notice that most of the processes holding devices open in a typical desktop env are holding /dev/pts/<N> (a pseudo tty) open. These are the "devices" of interest.

Some background on what's going on here:

In a typical Linux graphical desktop, processes don't talk to the keyboard directly. Instead the X program (Xorg) controls all the keyboard events via a device /dev/input/event<N>. X uses an event handler (evdev) which among other things, handles keyboard events. You can also verify this by looking at the X log: /var/log/Xorg.0.log where keyboard is mentioned.

The keyboard events are forwarded from the X event handler to the process that has the mouse pointer focus at any time via the process standard-input which is open on /dev/pts/<N>. Strictly speaking: a process doesn't actually "grab the keyboard", the keyboard is held by X, the process only has (or grabs) "focus" or the attention of X so X can forward keyboard events to it via an open stdin file descriptor on /dev/pts/<N>.

Step #3: which process has the Xorg focus at any particular time?

How to figure which process has the focus at any particular time? Here's a askubuntu question answering this:

find out the application under the mouse

The summary of the answer is to run a script like the following in a terminal while navigating around with the mouse:

#!/bin/bash
# Print the process tree of the window currently in focus.
# prereqs:
#   sudo apt-get install xdotool psmisc

while true; do
   pstree -spaul $(xdotool getwindowpid "$(xdotool getwindowfocus)")
   sleep 2
done

Step #4: dig deeper into process activity

Once you have a suspect process identified, the last step is to investigate this individual process. For that you may turn to the Linux /proc filesystem (man 5 proc).

Almost anything you may want to know about a process is available under /proc. In fact, programs like lsof (list open files), debuggers which examine process state, and process-listing utilities like ps or top, all rely on /proc, which is populated by the kernel, for data.

Using proc you can find where the process executable program is on disk (e.g. any program outside the standard system directories, especially if it is trying to hide under a "don't pay attention to me" kind of name, may be suspect) and using a debugger or system call tracer you can examine what exactly are they doing on the system call level (even if you don't have their source code).

Steps #2 and #3 should give you all the process-IDs (PIDs) that can potentially be reading your keyboard. For each of these PIDS (let's denote each one as $pid) you may:

Map $pid to its full command line:

cat /proc/$pid/cmdline

Map $pid to its on disk executable:

ls -l /proc/$pid/exe

Map $pid to its current working directory:

ls -l /proc/$pid/cwd

Map $pid to its original environment

cat /proc/$pid/environ | tr '\000' '\012'

Trace $pid (and its children-procs) system-call activity in real-time:

strace -f -p $pid

(There's more: see man 5 proc)

If you see an unfamiliar process that reacts to every key-press by storing it into a file (via write) or sending it over the network to via sendto, you may have found a keyboard sniffer.

You can also check which processes have (tcp+udp) network endpoints open:

# See 'man netstat' for details on all options used below
$ sudo netstat -tunapee

Bottom line:

The most likely cause for the error is not malware, but multiple processes trying to get keyboard control at the same time. One of the two is gnome-ssh-askpass (the one printing the error). The other may be an open browser on a site with an aggressive focus-acquiring dialog box.

Even in the remote chance that you actually have some malware installed, the good news is that since you're on Linux, all processes are transparent for you to research and inspect. It would be very hard for malware to really hide from you, or to prevent you from easily locating it using the above techniques, killing its processes, and removing all its files.

Solution 2

My issue was due two concurrent gnome-ssh-askpass windows. I had two rsync jobs to the same server through SSH, and both tried to ask for the password of the SSH certificate. Grouping (and chaining) them together solved for me!

Comments

[Admin]

DO NOT ISSUE SUDO COMMANDS IF YOU THINK YOUR KEYSTROKES ARE BEING LOGGED!!! instead, boot a clean live medium, and go from there.

[Steven C. Howell]

During step #2, I do not see many processes holding /dev/pts/7 (only 3 unique pid values). Scrolling through the results, it seems the most helf device is /dev/pts/15 though some are holding 1, 3, 12, 16, 17, 21, 22, 23, 24, 25, 25, 26, 27, 28, 29, 30, 31, 32, 34. Is the keyboard always 7? How would I identify which of these is my keyboard?

[arielf]

It can be any of the above. The physical keyboard device is actually open by Xorg (/usr/bin/X) as /dev/input/eventN where you can find your N by looking at the string evdev in /var/log/Xorg.0.log. Xorg then "forwards" each keyboard click to the individual process that has the mouse pointer "focus" at that particular instant. When I run ssh-askpass I see it has /dev/pts/3 open but in any env it can be any pseudo tty device. So any one of your /dev/pts/N may be relevant here.

[arielf]

@stvn66 I added a little script telling you which process has "the focus" repeatedly (reference to a related question on askubuntu). HTH.

[Steven C. Howell]

after running the script to identify which processes are holding the mouse, how would I identify a suspect one? It seems to be whatever application I select, e.g., starts as the terminal I ran the script in, switches to {firefox} when I click on firefox, switches again to {thunderbird} when I select thunderbird. Nothing stands out as unexpected. Perhaps this goes to your Bottom line: of the issue not coming from something grabbing the keyboard. I wish was certain this notice is meaningless or could eliminate it.

[arielf]

@stvn66 I hear you :) you can't go back in time and figure out the process that had the focus originally. That process may have exited since then. To be really sure, you need to be able to reproduce. My best guess is it was your browser (firefox) while visiting a web-site with a focus-grabbing pop-up. Unless you regularly download and install software from dubious (non-canonical) sources, I highly doubt you have accidentally installed a keyboard sniffer on Ubuntu. It is good to be a bit paranoid, but there's no need to over-sweat it.