Processing raw data read from serial port with Python serial library?

Question

I am not a Python programmer but am rather electronic circuit designer, however this time I must process some raw data sent by a microcontroller via RS232 port towards Python script (which is called by PHP script).

I've spent quite a few hours trying to determine the best ways of reading raw bytes from serial (RS232) port using Python and I did get the results - but I would like if someone could clarify certain inconsistencies I noticed during researching and here they are:

1:
I can see a lot of people who asked similar question had been asked whether they are using serial or pySerial module and how did they install the serial library. I can only say I don't really know which module I am using as the module worked out-of-the-box. Somewhere I read serial and pySerial is the same thing but I cannot find if that is true. All I know is I am using Python 2.7.9 with Raspbian OS.

2:
I've read there are read() and readline() methods for reading from the serial port but in the pySerial API docs there is no mention of the readline() method. Futhermore, I discovered the 'number of bytes to read' argument can be passed to readline() method as well as to the read() method (and works the same way, limiting the number of bytes to be read) but I cannot find that to be documented.

3:
When searching for how to determine if all of the data from the RS232 buffer has been read I have here found the following code:

read_byte = ser.read()
while read_byte is not None:
    read_byte = ser.read()
    print '%x' % ord(read_byte)

but that results with the:

Traceback (most recent call last):
  File "./testread.py", line 53, in <module>
    read_all()
  File "./testread.py", line 32, in read_all
    print '%x' % ord(read_byte)
TypeError: ord() expected a character, but string of length 0 found

upon reading the last byte from the buffer and I was able to detect the empty buffer only with the following code:

while True:
    c = rs232.read()
    if len(c) == 0:
        break
    print int(c.encode("hex"), 16), " ",

so I am not sure if the code that didn't work for me is for some serial library that is other than mine. My code for openinig port is BTW:

rs232 = serial.Serial(
    port = '/dev/ttyUSB0',
    baudrate = 2400,
    parity = serial.PARITY_NONE,
    stopbits = serial.STOPBITS_ONE,
    bytesize = serial.EIGHTBITS,
    timeout = 1
)

4:
The data I am receiving from µC is in the format:

0x16 0x02 0x0b 0xc9 ... 0x0d 0x0a

That is some raw bytes + \r\n. Since 'raw bytes' can contain 0x00, can someone confirm that is not a problem regarding reading the bytes into the Python string variable? As I understand that should work well but am not 100% sure.

Solution

PySerial works for me although haven't used it on a Pi.

3: Read() returns a string - this will be zero length if no data is read, so your later version is correct. As a string is not a character, you should use e.g. ord(read_byte[0]) to print the number corresponding to the first character (if the length of the string >0) Your function:

while True:
    c = rs232.read()
    if len(c) == 0:
        break
    print int(c.encode("hex"), 16), " ",

Needs something adding to accumulate the data read, otherwise it is thrown away

rcvd = ""
while True:
    c = rs232.read()
    if len(c) == 0:
        break
    rcvd += c
    for ch in c:
        print ord(ch), " ",

4: Yes you can receive and put nul (0x00) bytes in a string. For example:

a="\x00"
print len(a)

will print length 1

Comments

[Chupo_cro]

Does 'Yes you can but 0 bytes in a string.', mean: 'Yes you can but without zero-bytes in a string.'? What I would like to know is if e.g. 0x00, 0x00, 0x00, 0x00, 0x0d, 0x0a received from RS232 and assigned to a string variable can cause trouble. As I understand the internal string representation are null-terminated C strings but I can't find if zeroes can be stored at the high level representation nevertheless of the internal representation.

[DisappointedByUnaccountableMod]

The internal representation of strings in the usual Python we use on Linux/Windows isn't nul-terminated. You can put any value characters in a string.

[Chupo_cro]

That would mean this voted-up anwer saying: 'Internally, the most popular Python implementation is written in C, so there probably is a NULL-terminated string somewhere under the hood.', is wrong?

[Chupo_cro]

BTW, I can't anymore edit (during first 5 minutes after posting) or delete my own comments since there is no x icon (or anything) on mouse over. The funcion(s) before worked OK but not anymore. The problem is exactly as described here. Unfortunately I don't have enough reputation points to leave the comment to confirm the issue. Note: I don't have many reputation points but I do know how editing/deleting own comments funcion worked before.

[DisappointedByUnaccountableMod]

That answer is only correct in saying that the string implementation is opaque to the python-writing developer, and definitely wrong about the implementation using nul termination, as Josiah Yoder has commented.

[DisappointedByUnaccountableMod]

You can only edit a comment for a few minutes after posting, but you should be able to delete your own comments - I get the X - try different browser?

[Chupo_cro]

Yes, I said the edit is expected to work 'during first 5 minutes after posting'. In the meantime I posted the issue confirmation including screenshots as an 'answer' to the question I mentioned. Since I don't have enough reputation points, posting the 'answer' was the only way I could reply. Judging upon the upvotes I got there are more users affected by this issue. Moreover, a few hours ago the question score was -5 and now the score is +1 meaning others have the same issue. Yes I tried with FF and Chrome and the result was the same - no X anymore.