Post numpy array with json to flask app with requests

Question

Using Requests, I need to send numpy arrays with json data in a single post to my flask application. How do I do this?

Solution 1

To convert a numpy array arr to json, it can be serialized while preserving dimension with json.dumps(arr.tolist()). Then on the api side, it can be parsed with np.array(json.loads(arr)).

However, when using the requests json parameter, the dumping and loading is handled for you. So arr.tolist() is all that is required on the client, and np.array(arr) on the api. Full example code below.

Client:

params = {'param0': 'param0', 'param1': 'param1'}
arr = np.random.rand(10, 10)
data = {'params': params, 'arr': arr.tolist()}

response = requests.post(url, json=data)

API:

@app.route('/test', methods=['POST'])
def test():
    data = request.json
    params = data['params']
    arr = np.array(data['arr'])
    print(params, arr.shape)
    return "Success"

Output:

{'param0': 'param0', 'param1': 'param1'} (10, 10)

Note: When either the files or data parameter is being used in requests.post, the json parameter is disabled.

Solution 2

The accepted answer gets the job done for small arrays but has huge low performance for large arrays (At least 150% of overhead).

I would recommend using tostring() instead of tolist().

So client would become:

params = {'param0': 'param0', 'param1': 'param1'}
arr = np.random.rand(10, 10)
data = {'params': params, 'arr': arr.tostring()}
response = requests.post(url, json=data)

And API:

@app.route('/test', methods=['POST'])
def test():
    data = request.json
    params = data['params']
    arr = np.fromstring(data['arr'],dtype=float).reshape(10,10)
    print(params, arr.shape)
    return "Success"

You should note that the shape and dtype of the array must be known beforehand or informed in the request body.

Comments

[Julio Milani]

This has incredibly low performance for large arrays. A much better way (at least 50% lighter) is to use tostring() and fromstring()

[zjeffer]

This doesn't work for me, because arr.tostring() returns bytes, which is not JSON serializable, so requests.post throws an error. What is the solution for this?

[Julio Milani]

You can also, encode it in base64: arr = base64.b64encode(np.array([1,2,3,4]).tobytes()).decode('utf-‌​8') # arr = 'AQAAAAAAAAACAAAAAAAAAAMAAAAAAAAABAAAAAAAAAA=' vec = np.frombuffer(base64.b64decode(arr),dtype=int) # vec=[1,2,3,4]