Converting Comma-Separated String to Numpy Array In Python 3

Question

I'm using Python to read data from an Arduino sensor using PySerial in Python 3.

I want to plot the data using PyQtGraph, so I'm attempting to convert the string that is read from the sensor (288 comma separated values) to a Numpy array. However, I've tried a couple different methods and haven't gotten it to work.

Reading from the sensor works great with this code:

#Read the line of data from the sensor
line = sensor.readline()

#Decode the line to UTF-8 and print it
lineDecoded = line.decode("UTF-8")

print(lineDecoded)

Which gives me the correct output (288 comma separated values):

137,136,151,145,141,133,145,139,140,145,144,139,143,141,139,136,137,138,137,127,142,135,136,137,134,140,127,141,134,128,139,135,136,180,149,147,147,151,156,140,153,143,143,155,163,164,192,250,277,282,275,258,258,248,245,231,215,225,195,195,159,186,175,168,171,173,177,185,213,224,228,231,227,219,261,229,231,231,250,253,262,276,269,274,274,277,276,272,291,303,351,417,483,500,473,399,315,263,255,239,238,244,234,231,231,242,255,272,294,293,299,314,307,306,302,310,319,304,312,327,370,464,507,514,492,425,358,327,313,299,292,291,281,259,245,232,229,224,223,222,216,226,215,211,197,202,199,197,198,193,198,185,190,196,177,198,188,183,201,193,187,159,189,184,186,185,186,185,184,196,195,200,201,198,193,241,189,186,167,179,187,174,188,180,179,169,177,173,172,175,181,175,171,180,175,176,180,184,176,190,182,172,171,179,178,174,188,175,178,167,183,171,168,174,175,171,230,175,177,159,177,170,172,171,173,168,167,169,172,168,171,177,173,167,167,171,163,170,177,172,169,167,163,157,173,161,168,174,162,165,171,165,162,152,165,173,158,193,161,161,147,159,161,159,169,173,168,158,161,159,158,171,167,167,155,159,169,156,159,162,157,165,161,158,147,161,171,159,

From there, figured I'd just be able to pass lineDecoded to PyQtGraph's plot function, but after running this line of code

x = [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,227,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253,254,255,256,257,258,259,260,261,262,263,264,265,266,267,268,269,270,271,272,273,274,275,276,277,278,279,280,281,282,283,284,285,286,287,288]

y = [lineDecoded]

#Plot the data
pg.plot(x, y)

I got the error Exception: X and Y arrays must be the same shape--got (288,) and (1,).

I tried converting the string before passing it to PyQtGraph using various methods like np.fromstring(lineDecoded) and np.array(list(lineDecoded)).ravel() but none of the methods I've tried work.

Any ideas on how this could be accomplished?

EDIT: I've gone through the answers and none of them seem to be working, details in the comments for each response. Any hints would be great.

Solution 1

line.decode() returns a string (a str), not a list. So you are getting back a single entity, not 288 values.

Python has a method associated with strings that will take a string and split it into component parts. Calling the .split() method and giving it the substring to split upon, in this case a ',' should do the trick.

#Read the line of data from the sensor
line = sensor.readline()

#Decode the line to UTF-8 and print it
lineDecoded = line.decode("UTF-8")


values = [int(i) for i in lineDecoded.split(',')]    # <<< this should work
                                                     # added a list
                                                     # comprehension to 
                                                     # convert values to integers


x = range(1,289)                   # <<< this is preferred if you need 
                                   # a range of numbers from 1 to 288

y = values

#Plot the data
pg.plot(x, y)

NOTE: as @umutto mentions in a comment above, for plotting purposes, there should be no need to convert the values to a numpy array. A list should do just fine.

But, if for some reason you find that you want/need an array:

y = np.array(values)

Solution 2

Use the str.split method for converting the line to a list of strings. Also as @Alien suggests, you should use the range function for getting x, instead of enumerating all the values manually.

y = [int(i) for i in lineDecoded.split(',')]
x = range(1,289)

import matplotlib.pyplot as plt
fig = plt.figure()
plt.plot(x, lineDecoded)
fig.show()

Solution 3

With a copy/paste of your string:

In [341]: astring="""137,136,151,145,141,133,145,139,140,145,144,139,143,141,139
     ...: ,136,137,138,137,127,142,135,136,137,134,140,127,141,134,128,139,135,1
     ...: 36,180,149,147,147,151,156,140,153,143,143,155,163,164,192,250,277,282
     ...: ...
,147,159,161,159,169,173,168,158,161,159,158,171,167,167,155,159,169,1
     ...: 56,159,162,157,165,161,158,147,161,171,159"""

split(',') produces a list of strings. If I wrap that in np.array, I get an array of strings:

In [342]: np.array(astring.split(','))
Out[342]: 
array(['137', '136', '151', '145', '141', '133', '145', '139', '140',
       ...
       '162', '157', '165', '161', '158', '147', '161', '171', '159'], 
      dtype='<U3')

I need to tell it to convert the strings to integers:

In [343]: np.array(astring.split(','),int)
Out[343]: 
array([137, 136, 151, 145, 141, 133, 145, 139, 140, 145, 144, 139, 143,
       141, 139, 136, 137, 138, 137, 127, 142, 135, 136, 137, 134, 140,
...., 165, 161, 158, 147, 161,
       171, 159])
In [344]: _.shape
Out[344]: (288,)

The first time I tried it, the string ended with a comma: "8,147,161,171,159,". The split then included an empty string, which np.array could not convert to integer. I edited that out. I could also have removed it from the list of strings with indexing, [:-1].

If it works to pass the list of strings (after split) to the plot function, it's because that function performed this same sort of np.array(..., int) (or float) conversion.

[int(i) for i in astring.split(',')]

can be used to create a list of integers.