Convert text data from requests object to dataframe with pandas
Solution 1
Try this
import requests
import pandas as pd
import io
urlData = requests.get(url).content
rawData = pd.read_csv(io.StringIO(urlData.decode('utf-8')))
Solution 2
I think you can use read_csv
with url
:
pd.read_csv(url)
filepath_or_buffer : str, pathlib.Path, py._path.local.LocalPath or any object with a read() method (such as a file handle or StringIO)
The string could be a URL. Valid URL schemes include http, ftp, s3, and file. For file URLs, a host is expected. For instance, a local file could be file ://localhost/path/to/table.csv
import pandas as pd
import io
import requests
url = r'http://...'
r = requests.get(url)
df = pd.read_csv(io.StringIO(r))
If it doesnt work, try update last line:
import pandas as pd
import io
import requests
url = r'http://...'
r = requests.get(url)
df = pd.read_csv(io.StringIO(r.text))
Solution 3
Using "read_csv with url" worked:
import requests, csv
import pandas as pd
url = 'https://arte.folha.uol.com.br/ciencia/2020/coronavirus/csv/mundo/dados-bra.csv'
corona_bra = pd.read_csv(url)
print(corona_bra.head())
Solution 4
if the url has no authentication then you can directly use read_csv(url)
if you have authentication you can use request to get it un-pickel and print the csv and make sure the result is CSV and use panda.
You can directly use importing import csv
Related videos on Youtube
sparrow
I take large amounts of data and process it to gain insight and show patterns. I love my job :). My preferred tools are Python, Pandas, Plotly, Folium and Jupyter Notebooks. I'm always looking to grow my software skills. Right now I'm also learning Django. Undergrad - Electrical Engineering SDSU Grad - Hybrid EE/Comp Sci specializing in Wireless Embedded Devices UCSD.
Updated on April 05, 2020Comments
-
sparrow about 4 years
Using requests I am creating an object which is in .csv format. How can I then write that object to a DataFrame with pandas?
To get the requests object in text format:
import requests import pandas as pd url = r'http://test.url' r = requests.get(url) r.text #this will return the data as text in csv format
I tried (doesn't work):
pd.read_csv(r.text) pd.DataFrame.from_csv(r.text)
-
shivsn over 7 yearsdifficult to answer without seeing data.
-
Shijo over 7 yearsMay be you need save the response data to a file and check the file content. Then read the file to csv, check if this approach works. If not then there is something wrong in the data
-
Padraic Cunningham over 7 yearsstackoverflow.com/questions/32400867/pandas-read-csv-from-url/…, no need for requests unless you are posting some data that allows you to access the content
-
Merlin over 7 years@PadraicCunningham, I think you are wrong about requests, Its urllib2 or request. shhh, urllib2 has security flaws. which allow file access. So, requests is safer.
-
sparrow over 7 years@Padraic, I omitted this part in my question but I needed to include a special header format in my request which is why I used requests instead of importing the url directly. headers = {'user1': 'AppInterface'}
-
-
Ravi over 3 yearscan we convert dataframe output to URL. display dataframe table output in the form of url : any one can access it and see the dataframe data by opening that url ?
-
gudé over 2 yearsI guess this will depend on the request URL, I need to pass parameters to get(), for example, requests.get(url,params=dict())
-
Keivan Ipchi Hagh almost 2 years@gudé You can simply convert the parameters to a string using
'&'.join([f'{key}={value}' for key, value in params.items()])
and append to the rest of the URL.