Where can I find historical raw weather data?

Question

Where can I find historical raw weather data for a project I am doing with focus on the USA and Canada. I need temperatures mainly, but other details would be nice. I am having a very hard time finding this data. I really dont want to have to scrape a weather site.

Solution 1

At the United States National Severe Storms Laboratory Historical Weather Data Archive (note: this has since been retired).

Also, the United States National Climatic Data Center Geodata Portal.

The United States National Climatic Data Center Climate Data Online.

The United States National Climatic Data Center Most Popular Products.

Solution 2

I found myself asking this same question, and will share my experience for future Googlers.

Data sources

I wanted raw data, and lots of it... an API wouldn't do. I needed to head directly to the source. The best source for all of that data seemed to be either the NCEP or NCDC NOMADS servers:

http://nomads.ncdc.noaa.gov/dods/ <- good for historical data
http://nomads.ncep.noaa.gov/dods/ <- good for recent data

(Note: A commenter indicated that you must now use https rather than http. I haven't tested it yet, but if you're having issues, try that!)

To give an idea of the amount of data, their data goes all the way back to 1979! If you're looking for Canada and the US, the North American Regional Reanalysis dataset is probably your best answer.

Using the data

I'm a big python user, and either pydap or NetCDF seemed like good tools to use. For no particular reason, I started playing around with pydap.

To give an example of how to get all of the temperature data for a particular location from the nomads website, try the following in python:

from pydap.client import open_url

# setup the connection
url = 'http://nomads.ncdc.noaa.gov/dods/NCEP_NARR_DAILY/197901/197901/narr-a_221_197901dd_hh00_000'
modelconn = open_url(url)
tmp2m = modelconn['tmp2m']

# grab the data
lat_index = 200    # you could tie this to tmp2m.lat[:]
lon_index = 200    # you could tie this to tmp2m.lon[:]
print tmp2m.array[:,lat_index,lon_index] 

The above snippet will get you a time series (every three hours) of data for the entire month of January, 1979! If you needed multiple locations or all of the months, the above code would easily be modified to accommodate.

To super-data... and beyond!

I wasn't happy stopping there. I wanted this data in a SQL database so that I could easily slice and dice it. A great option for doing all of this is the python forecasting module.

Disclosure: I put together the code behind the module. The code is all open source -- you can modify it to better meet your needs (maybe you're forecasting for Mars?) or pull out little snippets for your project.

My goal was to be able to grab the latest forecast from the Rapid Refresh model (your best bet if you want accurate info on current weather):

from forecasting import Model

rap = Model('rap')
rap.connect(database='weather', user='chef')
fields = ['tmp2m']
rap.transfer(fields)

and then to plot the data on a map of the good 'ole USA:

The data for the plot came directly from SQL and could easily modify the query to get out any type of data desired.

If the above example isn't enough, check out the documentation, where you can find more examples.

Solution 3

wunderground.com has a good API. It is free for 500 calls per day.

http://www.wunderground.com/weather/api/

Comments

[iTech]

Check forecast.io, you can get historical data as well as future forecast with very easy to use API

[Elle Najt]

There is data available here: kaggle.com/noaa/gsod

[dl.meteo]

Simple access to the noaa data from here dev.meteostat.net

[Brian]

The url has changed for Climate Data Online

[philshem]

This is good for international weather.

[iTurki]

Historical data NOT included!

[Tom N Tech]

I need weather data for all of 2015, which your link for historical data doesn't have - it stops at 201410. Do you have any ideas for me?

[Lance Fisher]

@Rob I'm not affiliated with wunderground.com Down vote the answer if it's not helpful.

[Rob]

Actually @iturki I apologize, the other poster is wrong, weather underground dose have the only historical projection API I could find!!

[wilsotc]

Weather underground isn't reliable for weather data. They allow weather station data that is highly questionable. Near me there are two weather stations on their site that have a 25F difference in temperature. The two are within half a kilometer of one another.

[wilsotc]

Here's why weatherunderground is a terrible source of weather data: photos.google.com/photo/… . The Lake Butler Florida weather station is showing 27F while every station around it is reporting 80F. Weather underground doesn't care about accuracy. Any idiot that goes to Canada and installs a weather station can report it's location in Florida and host the data on their website.

[Matias Grioni]

All the data coming from the first snippet of code is returning a missing value of 9.999E20. No matter what year I choose or what latitude and longitude index I choose. Any ideas?

[Junier]

Just a quick tip: seems they are only supporting https now; e.g. use url = 'nomads.ncdc.noaa.gov/dods/NCEP_NARR_DAILY/197901/197901/…'

[inspectorG4dget]

share my experience for future Googlers <- this future googler thanks you

[Anagha]

The above link is not accesible

[ignorance]

@wilsotc 27C is 80F. Any idiot from Canada would have known that :). I bet the units were just set wrong. Also, you could have also just discovered a station that included wind chill in their report. Certainly, can't say for sure, but 25F is well within the range of chill factors. It would be interesting to find out. But yes, I agree, WU should fix/report these discrepancies.

[Amit Makashir]

I am probably a little late to this conversation, but can anyone help me get daily rainfall data for a state (say Indiana) for at least the last 10 years.

[Andrew Schultz]

There's an API that you can use to access historical data. Pretty cool it gives you average temps, precipitation per month/year for a location. ncdc.noaa.gov/cdo-web/webservices/v2#gettingStarted

[Anthony Griggs]

We've actually been using wunderground.... Unfortunately however, they just changed their terms of use in that you can only use the API if you yourself are hosting a Weather Station... which is why I am researching an alternative. +1 for wunderground as it worked fine for our needs for many years.

[dl.meteo]

Python API for such data can be found here: dev.meteostat.net