Pyserial buffer fills faster than I can read

Question

I am reading data from a microcontroller via serial, at a baudrate of 921600. I'm reading a large amount of ASCII csv data, and since it comes in so fast, the buffer get's filled and all the rest of the data gets lost before I can read it. I know I could manually edit the pyserial source code for serialwin32 to increase the buffer size, but I was wondering if there is another way around it?

I can only estimate the amount of data I will receive, but it is somewhere around 200kB of data.

Solution 1

There's a "Receive Buffer" slider that's accessible from the com port's Properties Page in Device Manager. It is found by following the Advanced button on the "Port Settings" tab.

More info:

http://support.microsoft.com/kb/131016 under heading Receive Buffer

http://tldp.org/HOWTO/Serial-HOWTO-4.html under heading Interrupts

Try knocking it down a notch or two.

Solution 2

Have you considered reading from the serial interface in a separate thread that is running prior to sending the command to uC to send the data?

This would remove some of the delay after the write command and starting the read. There are other SO users who have had success with this method, granted they weren't having buffer overruns.

If this isn't clear let me know and I can throw something together to show this.

EDIT

Thinking about it a bit more, if you're trying to read from the buffer and write it out to the file system even the standalone thread might not save you. To minimize the processing time you might consider reading say 100 bytes at a time serial.Read(size=100) and pushing that data into a Queue to process it all after the transfer has completed

Pseudo Code Example

def thread_main_loop(myserialobj, data_queue):
    data_queue.put_no_wait(myserialobj.Read(size=100))

def process_queue_when_done(data_queue):
    while(1):
        if len(data_queue) > 0:
            poped_data = data_queue.get_no_wait()
            # Process the data as needed
        else:
            break;

Solution 3

You do not need to manually change pyserial code.

If you run your code on Windows platform, you simply need to add a line in your code

ser.set_buffer_size(rx_size = 12800, tx_size = 12800)

Where 12800 is an arbitrary number I chose. You can make receiving(rx) and transmitting(tx) buffer as big as 2147483647a

---

See also:

https://docs.python.org/3/library/ctypes.html

https://msdn.microsoft.com/en-us/library/system.io.ports.serialport.readbuffersize(v=vs.110).aspx

You might be able to setup the serial port from the DLL // Setup serial

 mySerialPort.BaudRate = 9600;
 mySerialPort.PortName = comPort;
 mySerialPort.Parity = Parity.None;
 mySerialPort.StopBits = StopBits.One;
 mySerialPort.DataBits = 8;
 mySerialPort.Handshake = Handshake.None;
 mySerialPort.RtsEnable = true;
 mySerialPort.ReadBufferSize = 32768;

Property Value Type: System.Int32 The buffer size, in bytes. The default value is 4096; the maximum value is that of a positive int, or 2147483647

And then open and use it in Python

Comments

[Nate]

I will give this a try, though after doing a little math, I still don't know if this will help. currently the buffer will be overrun in 0.03 seconds. And to top that, I'm fairly certain there is no end of stream notification either. The joys of homebrew hardware.

[Nate]

I will give this a try as well. However it sounds like it will just help with the hardware buffer, so while I am probably overrunning that buffer too, I know I am overrunning the pySerial buffer, which is the first concern I have to address.

[jenny]

@Nate That's pretty quick.... Check out my edit for additional ways to speed it up.

[Bob Stein]

On my Windows 7, the setting seems to be: Device Manager | Ports | COMn | Properties | Port Settings | Advanced | Receive(Bytes). This is a drop-down list and I'm already at the max of 4096. My buffer seems to be limited to about 2K.

[krb686]

The reason why your settings don't look like the image supplied by @jon is because that microsoft page applies to windows 95