FAQ


How do I use the THREDDS data server (TDS) to download a subset of a dataset that is too large to store locally?

From the TDS site, navigate to the dataset of interest and click on the .ncml file.

Use the “NetcdfSubset” service that can be found in the THREDDS Data Server “Access” list.

Click on the variable of interest.
If you want to do horizontal subsetting, click on “Disable horizontal subsetting” to allow it.
Set the Lat/lon subset Bounding Box.
Click Submit.

The service will return an NCSS Request URL that can be cut and paste to be used in a utility such as wget to download the file. (Note, you may need to put quotes around the URL to get wget to read it correctly.)

For additional information, see “What does the TDS NetcdfSubset message “NCSS response too large or Request too big mean?”

What does the TDS NetcdfSubset message “NCSS response too large or Request too big mean?”

THREDDS data servers (TDS) simplify the discovery of scientific data and metadata, but don’t have the performance necessary to serve large datasets to the desktop.

If you’ve submitted a data request through the NetCDF Subset form and received an NCCS response too large or submitted a data request through the OPeNDAP dataset access form and received a Request too big response, your requested data size exceeds the maximum limit for the server. Requests of more than a year of data can exceed the limit.

If you’re retrieving an exploratory or test file you may want to reduce the coordinates coverage (bounding box), levels or time range and try again. If you’d like the full file, or multiple files, you may want to use a utility (such as wget), an OPeNDAP client or application, such as UV-CDAT, Panoply,Ferret or GrADS, to access the data.

Where do I find the dataset and variable URLs to use with my application, e.g. wget, GrADS?

The easiest way to get your dataset URLs is to let the NetCDF Subset Service (NCSS) build them for you.

1. Navigate to the catalog from the NCCS Dataserver page, Catalog, and click through the options to the data access page for your variable, for example:

NEX-GDDP -> catalog page -> bcsd -> rcp85 -> r1i1p1 -> tasmin -> inmcm4.ncml

2. Under Access, click on the NetCDFSubset link.

This is the NCCS dataset variable access page. The URL is displayed at the bottom of the page and is modified as coordinates (top right) and time subsets (bottom right) are changed. Uncheck the Disable horizontal subsetting box to edit the bounding box coordinates and remember to check the variable box (tasmin).

Most requests will be too large for the server to deliver, but you can cut and paste this URL for use in OPeNDAP-compatible applications, onto the command line or into your favorite text editor for inclusion in a list for further editing. Example URL:

http://dataserver3.nccs.nasa.gov/thredds/ncss/bypass/NEX-GDDP/bcsd/historical/r1i1p1/tasmin/inmcm4.ncml?var=tasmin&horizStride=1&time_start=1950-01-01T12%3A00%3A00Z&time_end=1950-01-31T12%3A00%3A00Z&timeStride=1

How do I use Wget to download data?

Wget is a free network utility you can use to download files referenced by URLs, such as THREDDS and OPeNDAP files. Wget is available in all Linux distributions, UNIX versions, and as a GNU package for many Windows versions. Wget can be used with a single URL or with a text file containing multiple URLs and has a number of useful options, consult your man pages or documentation for availability.

Wget is used on the terminal/command line, followed by the URL:

wget http://dataserver3.nccs.nasa.gov/thredds/ncss/bypass/NEX-GDDP/bcsd/historical/r1i1p1/tasmin/inmcm4.ncml?var=tasmin&horizStride=1&time_start=1950-01-01T12%3A00%3A00Z&time_end=1950-01-31T12%3A00%3A00Z&timeStride=1

For instructions on how to generate the above URL, see the section titled “Where do I find the dataset and variable URLs to use with my application, e.g. wget, GrADS?”

Do I need to use quotes with the URL when running from the command line?

A note on the use of quotes with the URL portion of the wget HTTP request — while not strictly required, it protects the special characters in the URL string from interacting with the shell. One downside to using quotes: if you aren’t renaming the output file, the file name will be long and verbose.

Example of the single output file name when quotes are used:
inmcm4.ncml?var=tasmin&horizStride=1&time_start=1950-01-01T12:00:00Z&time_end=1950-01-31T12:00:00Z&timeStride=1

Example of the single output file name when quotes aren’t used:
inmcm4.ncml?var=tasmin

If using –content-disposition or the -O output file rename options, using quotes will prevent shell interaction. Quotes are not needed when using wget with file lists.

How can I rename the output files with Wget?

If using wget with no options, the file name will have the default format inmcm4.ncml?var=tasmin, example:

wget http://dataserver3.nccs.nasa.gov/thredds/ncss/bypass/NEX-GDDP/bcsd/historical/r1i1p1/tasmin/inmcm4.ncml?var=tasmin&horizStride=1&time_start=1950-01-01T12%3A00%3A00Z&time_end=1950-01-31T12%3A00%3A00Z&timeStride=1

Using wget –content-disposition, the file name will have this format: inmcm4.ncml.nc. Example:

wget --content-disposition "http://dataserver3.nccs.nasa.gov/thredds/ncss/bypass/NEX-GDDP/bcsd/historical/r1i1p1/tasmin/inmcm4.ncml?var=tasmin&horizStride=1&time_start=1950-01-01T12%3A00%3A00Z&time_end=1950-01-31T12%3A00%3A00Z&timeStride=1"

Finally, using the -O option lets you define the output name, for instance, adding the date and other information:

wget -O NEXbcsd-tasmin-195001.nc "http://dataserver3.nccs.nasa.gov/thredds/ncss/bypass/NEX-GDDP/bcsd/historical/r1i1p1/tasmin/inmcm4.ncml?var=tasmin&horizStride=1&time_start=1950-01-01T12%3A00%3A00Z&time_end=1950-01-31T12%3A00%3A00Z&timeStride=1"

How do I retrieve a list of files with Wget?

To download multiple URLs, create a text file with the URLs, one per line, with no trailing spaces or line feeds. The URLs can be copied from the NetCDF Subset Service page and edited. For instructions on how to generate URLs, see the section titled “Where do I find the dataset and variable URLs?” Spreadsheets are handy for formatting and exporting large numbers of URLs.

Below are three example URLs that can be saved as a list, used here as an example file files.txt. Lists should not have trailing spaces or line feeds:

http://dataserver3.nccs.nasa.gov/thredds/ncss/bypass/NEX-GDDP/bcsd/historical/r1i1p1/tasmin/inmcm4.ncml?var=tasmin&horizStride=1&time_start=1950-01-01T12%3A00%3A00Z&time_end=1950-01-31T12%3A00%3A00Z&timeStride=1

http://dataserver3.nccs.nasa.gov/thredds/ncss/bypass/NEX-GDDP/bcsd/historical/r1i1p1/tasmin/inmcm4.ncml?var=tasmin&horizStride=1&time_start=1950-02-01T12%3A00%3A00Z&time_end=1950-02-28T12%3A00%3A00Z&timeStride=1

http://dataserver3.nccs.nasa.gov/thredds/ncss/bypass/NEX-GDDP/bcsd/historical/r1i1p1/tasmin/inmcm4.ncml?var=tasmin&horizStride=1&time_start=1950-03-01T12%3A00%3A00Z&time_end=1950-03-31T12%3A00%3A00Z&timeStride=1

The command to use a file of URLs, for example files.txt, is:

wget --content-disposition -i files.txt

Using –content-disposition writes an output file name in the form inmcm4.ncml.nc.

To use the -O option within a list to rename output files, use wget with a line reading utility, example:

while read -r line; do wget $line; done < files.txt

Here is an example of the content of files.txt used with the -O option:

-O NEXbcsd-tasmin-195001.nc http://dataserver3.nccs.nasa.gov/thredds/ncss/bypass/NEX-GDDP/bcsd/historical/r1i1p1/tasmin/inmcm4.ncml?var=tasmin&horizStride=1&time_start=1950-01-01T12%3A00%3A00Z&time_end=1950-01-31T12%3A00%3A00Z&timeStride=1

-O NEXbcsd-tasmin-195002.nc http://dataserver3.nccs.nasa.gov/thredds/ncss/bypass/NEX-GDDP/bcsd/historical/r1i1p1/tasmin/inmcm4.ncml?var=tasmin&horizStride=1&time_start=1950-02-01T12%3A00%3A00Z&time_end=1950-02-28T12%3A00%3A00Z&timeStride=1

-O NEXbcsd-tasmin-195003.nc http://dataserver3.nccs.nasa.gov/thredds/ncss/bypass/NEX-GDDP/bcsd/historical/r1i1p1/tasmin/inmcm4.ncml?var=tasmin&horizStride=1&time_start=1950-03-01T12%3A00%3A00Z&time_end=1950-03-31T12%3A00%3A00Z&timeStride=1

How to do I get my application to read the NCCS dataserver catalog?

OPeNDAP and THREDDS clients, and applications that analyze, visualize and plot geo-referenced data such as UV-CDAT, Panoply, Ferret or GrADS, can browse the NCCS Dataserver catalog and use the datasets.

At the load from OPeNDAP or Open Remote Catalog (or similar) command, enter:

http://dataserver.nccs.nasa.gov/thredds/catalog.html