Device misdetected as serial mouse

Question

I'm working on a device which communicates with a PC through a (virtual) serial port. The problem is that the data we are sending occasionally gets incorrectly identified by Windows as a bus mouse, after which the "Microsoft Serial Ballpoint" driver is loaded and the mouse pointer starts jumping around on the screen and randomly clicking on things.

A bit of Googling reveals that is an old and well-known problem with serial devices where the usual work-around is a bit of registry hacking to disable the offending driver. That it is a lot to demand from our users however and I'd rather not have our application messing around with the user's registry. Especially not when the fix is dependent on the Windows version and the user may well be using a bus mouse.

Instead I'd like to avoid the problem by changing our protocol to not send any data which may get us misidentified as a mouse. The only problem is that I'm not quite certain what patterns to avoid. Apparently Microsoft's Mouse protocol consists of packets of four bytes where the MSB of the first is set and that of the last three is clear.

Would sending only 7-bit ASCII suffice? Are there any other devices I need to worry about being detected as?

Solution 1

I just encountered this problem myself on Windows 7 Professional x64, and a solution that worked for me was to go into the registry and edit the following value:

Location: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\sermouse  
Key: Start  
Value: 3

Change Value to 4 and it will stop this problem occurring.

Here is a list of all valid Start values:

0 Boot (loaded by kernel loader). Components of the driver stack for the boot (startup) volume must be loaded by the kernel loader.

1 System (loaded by I/O subsystem). Specifies that the driver is loaded at kernel initialization.

2 Automatic (loaded by Service Control Manager). Specifies that the service is loaded or started automatically.

3 Manual. Specifies that the service does not start until the user starts it manually, such as by using Device Manager.

4 Disabled. Specifies that the service should not be started.

---

A reg edit command would be as follows:

REG ADD "HKLM\SYSTEM\CurrentControlSet\Services\sermouse" /V Start /T REG_DWORD /F /D 4

You then need to restart the computer, which should now start correctly and not attempt to discover a serial mouse.

good luck.

Solution 2

It turns out that mouse detection in Windows is normally handled by the serenum.sys filter driver. This driver implements support for legacy serial mice along with serial plug-and-play. Microsoft has even provided the sourcecode as a WDK sample.

During detection the ports switches to 1200-7-N-1 mode while asserting DTR+RTS to which a response is expected within 200 ms, with a couple of retries in case of failure. Unfortunately for a legacy mouse a single M or B character suffices as identification.

In our case the protocol was reworked to avoid these characters and now appears not to be misidentified anymore.

However we were using a virtual USB serial port and for a traditional serial port this approach may be somewhat difficult as anything sent at a different baud rate is liable to look like line noise. In this case I suppose the easiest workaround is probably, as has already been suggested, to avoid making any unsolicited transmissions.

Alternatively with the serial control signals actually hooked up, or intercepted by a USB CDC device, processing the DTR or RTS signals and holding off on output. Actually implementing the plug-and-play protocol would be an even niftier option. Supposedly there are cheap RS232 cables around without a full complement of control signals though so this approach might still fail.

Solution 3

I also encountered this problem, fixed it by disabling "serial enumerator" in the advanced properties of the FTDI driver (properties of COM ports in Device Manager). This is described in http://www.ftdichip.com/Support/Documents/AppNotes/AN_107_AdvancedDriverOptions_AN_000073.pdf.

Solution 4

I have encountered this Windows bug myself. Here is my own research on the topic:

Microsoft acknowledges this bug: http://support.microsoft.com/kb/819036 Start with downloading their tool and see if it solves the issue.

• Download & install their program.
• Run it from the command prompt from C:\program\Microsoft comdisable\
• Write comdisable /list when executing the program.
• All ports on the computer will be shown.
• Write comdisable /disable COMx where x is the port number.
• Do this for all ports on the computer.
• Reboot.

This should hopefully work as an universal solution.

Alternatively, you can hack in boot.ini but I don't believe this works in Vista/Win 7. I have some app note from Cisco systems describing how to do this. If the above doesn't solve your problem, please let me know.

Solution 5

In my development environment, I've simply disabled Microsoft Serial Mouse from the Device Manager.

This seems to solve the culprit of my problem. Before doing so, the CH340G chip I've used in my design used to lower the DTR five times before initiating the connection, effectively rebooting my Arduino-based board and render it useless.

View more solutions

Comments

[doynax]

Thanks for the suggestion. I'd prefer to avoid any user intervention by changing the protocol on our device so as not to confuse Windows, but if that doesn't work out then at least I have an official workaround (and "bug" acknowledgment) to refer to. I've implemented tinman's suggestion along with repackaging our data as 7-bit ASCII now and have yet to see the problem again, though to be honest it never did appear all that frequently.

[Lundin]

@doynax I'm not sure what kind of obscure protocol this "ballpoint" junk uses, but changing to 7-bit ASCII will might not solve anything. Because Windows assumes that this ancient Microsoft mouse has a certain baudrate. I have encountered the problem when using a faster baudrate than 9600, the data can suddenly get interpreted as something coming from the mouse. When that happens... disaster. The mouse will move all over the screen at the speed of light, clicking everywhere! Anything can happen. I wouldn't risk that, I would use the tool Microsoft recommends in that link.

[doynax]

Thankfully we're using a FTDI chip with a virtual serial port driver instead of a real serial port. So there's no risk of the data being mangled through an invalid baud rate setting.

[tamberg]

@Jon: on Windows 8.1 it helped to open devmgmt.msc, plug in the device, select the "Microsoft Serial Ballpoint" device, right click, and click "Uninstall" to get rid of the driver and the problem

[iforce2d]

Perfect! I needed to restart the OS too.

[Serdalis]

Forgot to mention that, I'll add it to the answer, Thanks.

[Tony Wall]

This is the best solution in my opinion. On my radio link device I could see the host TX/RX light flashing all the time and that was obviously the serial enumerator scanning it. Once disabled and connecting the device (no reboot necessary which is another plus) there was no activity until a program which uses the device is started. Also I had trouble in the past connecting to the device, hit-and-miss, which has gone so must have been the serial enumerator jumping in-between the application's own attempt to lock/open the serial port. Thanks!

[gonatee]

It drove me mad for 2 Hours!

[Günther the Beautiful]

Is there any kind of documentation on this registry entry? Right now, 3 and 4 are totally magic numbers to me. Microsoft has documentation on the sermouse node, but not on the actual Start element itself.

[Serdalis]

@GünthertheBeautiful Here you go! I've added it to the answer for you.

[David Refoua]

This comment is so useful to me! Now I understand why my Arduino-clones reboot a couple of time before powering up (Because the CH340's DTR pin through a 100NF cap is connected to the reset pin.) Thanks for the great explanation!

[Chris Stratton]

That may work for you personally, but it does not address the problem of the original question, which requires a solution that works for end users and not only developers willing to modify the system configuration.

[David Refoua]

@ChrisStratton True, and I may not have posted this as an answer but I merely wanted to provide a quick solution for devs like me Googling this issue. I used a 500ms delay after serial connection to prevent Windows detecting my device as a mouse, instead.

[David Refoua]

@Jubation I'm interested to know, what do you personally think is the better alternative to Serial-to-USB chips (instead of FTDI, CH340G, ATmega32u4, etc...) for a more user-friendly interface?

[Jubatian]

@DRS David Soft If your design can include a proper USB interface using some appropriate device class for your device, then most likely that for end-user things (of course this also demands creating drivers). If it is not, then I think it is better to stick with an ordinary DSUB-9 connector (RS-232), that's more robust if you later wanted to use the device combined with some other microcontroller-driven device for a more autonomous system. Use dongles (or RS-232 interface card) to connect with a PC.

[Neill]

You are my hero! I was ready to throw away my computer I was so frustrated!

[Paul Allsopp]

Worked for me just great! My FTDI adapter was being seen as a Microsoft Ball Pointer. Thx

[Paul Allsopp]

@ChrisStratton This is not a developer fix, this is a fix for anyone. Users should have a basic understanding of what Device Management is.