Import netCDF file to Pandas dataframe

Question

Merry Christmas! I am still very new to Python and Pandas, so any help is appreciated. I am trying to read in a netCDF file, which I can do and then import that into a Pandas Dataframe. The netcDF file is 2D so I just want to 'dump it in'. I have tried the DataFrame method but it doesn't recognize the object. Presumably I need to convert the netCDF object to a 2D numpy array? Again, thanks for any ideas on the best way to do this.

Solution 1

The xarray library handles arbitrary-dimensional netCDF data, and retains metadata. Xarray provides a simple method of opening netCDF files, and converting them to pandas dataframes:

import xarray as xr

ds = xr.open_dataset('/path/to/netcdf')
df = ds.to_dataframe()

This will create a dataframe with a multi-index with all of the dimensions in it. Unfortunately, Pandas doesn't support arbitrary metadata, so that will be lost in the conversion, but you can keep the ds around, and use the metadata from that.

Solution 2

If your NetCDF file (or OPeNDAP dataset) follows CF Metadata conventions you can take advantage of them by using the NetCDF4-Python package, which makes accessing them in Pandas really easy. (I'm using the Enthought Python Distribution which includes both Pandas and NetCDF4-Python).

In the example below, the NetCDF file is being served via OPeNDAP, and the NetCDF4-Python library lets you open and work with a remote OPeNDAP dataset just as if it was a local NetCDF file, which is pretty slick. If you want to see the attributes of the NetCDF4 file, point your browser at this link http://geoport-dev.whoi.edu/thredds/dodsC/HUDSON_SVALLEY/5951adc-a1h.nc.html

You should be able to run this without changes:

from matplotlib import pyplot as plt
import pandas as pd
import netCDF4

url='http://geoport-dev.whoi.edu/thredds/dodsC/HUDSON_SVALLEY/5951adc-a1h.nc'
vname = 'Tx_1211'
station = 0

nc = netCDF4.Dataset(url)
h = nc.variables[vname]
times = nc.variables['time']
jd = netCDF4.num2date(times[:],times.units)
hs = pd.Series(h[:,station],index=jd)

fig = plt.figure(figsize=(12,4))
ax = fig.add_subplot(111)
hs.plot(ax=ax,title='%s at %s' % (h.long_name,nc.id))
ax.set_ylabel(h.units)

The result may be seen here in the Ipython Notebook: http://nbviewer.ipython.org/4615153/

Solution 3

You can use a library like PyNIO to read your file into p.e. numpy arrays and feed them to pandas.
PyNIO allows reading several file formats including classic netCDF3 and netCDF4.
netcdf4-python can also read these netCDF formats and is py3.3 compatible