PySerial non-blocking read loop

Question

I am reading serial data like this:

connected = False
port = 'COM4'
baud = 9600

ser = serial.Serial(port, baud, timeout=0)

while not connected:
    #serin = ser.read()
    connected = True

    while True:
        print("test")
        reading = ser.readline().decode()

The problem is that it prevents anything else from executing including bottle py web framework. Adding sleep() won't help.

Changing "while True"" to "while ser.readline():" doesn't print "test", which is strange since it worked in Python 2.7. Any ideas what could be wrong?

Ideally I should be able to read serial data only when it's available. Data is being sent every 1,000 ms.

Solution 1

Put it in a separate thread, for example:

import threading
import serial

connected = False
port = 'COM4'
baud = 9600

serial_port = serial.Serial(port, baud, timeout=0)

def handle_data(data):
    print(data)

def read_from_port(ser):
    while not connected:
        #serin = ser.read()
        connected = True

        while True:
           print("test")
           reading = ser.readline().decode()
           handle_data(reading)

thread = threading.Thread(target=read_from_port, args=(serial_port,))
thread.start()

http://docs.python.org/3/library/threading

Solution 2

Using a separate thread is totally unnecessary. Just do this for your infinite while loop instead (Tested in Python 3.2.3). I use this technique in my eRCaGuy_PyTerm serial terminal program here (search the code for inWaiting() or in_waiting).

import serial
import time # Optional (required if using time.sleep() below)

ser = serial.Serial(port='COM4', baudrate=9600)

while (True):
    # Check if incoming bytes are waiting to be read from the serial input 
    # buffer.
    # NB: for PySerial v3.0 or later, use property `in_waiting` instead of
    # function `inWaiting()` below!
    if (ser.inWaiting() > 0):
        # read the bytes and convert from binary array to ASCII
        data_str = ser.read(ser.inWaiting()).decode('ascii') 
        # print the incoming string without putting a new-line
        # ('\n') automatically after every print()
        print(data_str, end='') 

    # Put the rest of your code you want here
    
    # Optional, but recommended: sleep 10 ms (0.01 sec) once per loop to let 
    # other threads on your PC run during this time. 
    time.sleep(0.01) 

This way you only read and print if something is there. You said, "Ideally I should be able to read serial data only when it's available." This is exactly what the code above does. If nothing is available to read, it skips on to the rest of your code in the while loop. Totally non-blocking.

(This answer originally posted & debugged here: Python 3 non-blocking read with pySerial (Cannot get pySerial's "in_waiting" property to work))

pySerial documentation: http://pyserial.readthedocs.io/en/latest/pyserial_api.html

UPDATE:

• 27 Dec. 2018: added comment about in_waiting vs inWaiting(). Thanks to @FurkanTürkal for pointing that out in the comments below. See documentation here: https://pyserial.readthedocs.io/en/latest/pyserial_api.html#serial.Serial.in_waiting.
• 27 Oct. 2018: Add sleep to let other threads run.
• Documentation: https://docs.python.org/3/library/time.html#time.sleep
• Thanks to @RufusV2 for bringing this point up in the comments.

Note on multi-threading:

Even though reading serial data, as shown above, does not require using multiple threads, reading keyboard input in a non-blocking manner does. Therefore, to accomplish non-blocking keyboard input reading, I've written this answer: How to read keyboard-input?.

References:

1. Official pySerial serial.Serial() class API - https://pyserial.readthedocs.io/en/latest/pyserial_api.html

Solution 3

I would warn against using blocking IO in a thread. Remember Python has a GIL and at one time only one thread can execute. Now please note that pyserial module is a wrapper over an OS implementation of accessing the serial port. That means it calls code external to the Python. If that code blocks, then the interpreter also get blocked and nothing will execute in the Python program, even the main thread.

This can even happen when using non-blocking IO or timeout based polling if the underlying device driver does not implement timeout well.

A more robust approach is to use multiprocessing module with a queue. Run serial read code in a separate process. This will make sure main and other threads don't block and the program can exit in clean way.

Comments

[Chris Riebschlager]

Thanks! This solution got me out of a tough spot today. I really feel this should be the accepted answer in this case.

[Johnny]

Instead of while(True) I would suggest using while(ser.isOpen())

[Johnny]

for PySerial Version >3 you need to use ser.is_open

[RufusVS]

If you are not blocking, I'd recommend an else clause to the if..inWaiting block, with a time.sleep(0.01) to avoid "pegging" the CPU of your computer if you want anything else to run at the same time.

[Gabriel Staples]

Too see a generic multithreading example with queue usage, see here: stackoverflow.com/questions/5404068/how-to-read-keyboard-inp‌​ut/….

[Dentrax]

Changed in version 3.0: changed to property from inWaiting() to in_waiting() Please see here.

[Gabriel Staples]

@FurkanTürkal, thanks! I've updated it in the answer by making this a comment.

[Dentrax]

@GabrielStaples U'r welcome! Also I am getting this error Any idea? :)

[Gabriel Staples]

I don't have enough context. Please make it a new question, and post a code snippet large enough that anyone can copy and paste it and duplicate your error, but trimmed down enough that it only contains enough code to duplicate the problem.

[PeterT]

or just use pyserial-asyncio and just do something on a callback.

[William Kuipers]

Using a qt5 timer is a great option for me. It provides other IO event handling (keyboard, mouse) and presentation of results in a GUI application. It avoids the use/import of threading.

[rsaxvc]

The GIL is only held in the interpreter. From your link, "Note that potentially blocking or long-running operations, such as I/O, image processing, and NumPy number crunching, happen outside the GIL."

[Mohammad Azim]

The point is about avoiding to wait on something that is happening outside. By using non blocking I/O your program is free to do next useful thing it wants to.