home *** CD-ROM | disk | FTP | other *** search
- Newsgroups: sci.data.formats,news.answers,sci.answers
- Path: bloom-beacon.mit.edu!news.kei.com!sol.ctr.columbia.edu!howland.reston.ans.net!agate!ames!ncar!kiowa.scd.ucar.edu!ilana
- From: ilana@kiowa.scd.ucar.edu (Ilana Stern)
- Subject: Scientific Data Format Information FAQ
- Message-ID: <1993Dec29.120011.4167@ncar.ucar.edu>
- Followup-To: sci.data.formats
- Summary: Where to find information on scientific data formats
- Sender: news@ncar.ucar.edu (USENET Maintenance)
- Reply-To: ilana@ncar.ucar.edu
- Organization: NCAR/UCAR
- Date: Wed, 29 Dec 1993 12:00:11 GMT
- Approved: news-answers-request@MIT.Edu
- Expires: Tue, 12 Jan 1993 07:00:00 GMT
- Lines: 294
- Xref: bloom-beacon.mit.edu sci.data.formats:341 news.answers:13435 sci.answers:768
-
- Archive-name: sci-data-formats
- Last-modified: 22 Dec 1993
-
- Recent changes:
-
- ==within last two weeks==
-
- ==within last four weeks==
- Added information about CSCDF
-
-
-
- This is the FAQ for the sci.data.formats newsgroup. Contents:
-
- -2) How to use this document
- -1) How to get a current copy of this document
- 0) Resources for format information
- 1) How to use the data retrieval methods
- 2) Why isn't my favorite format on this list?
-
- Each (major) section has a "Subject:" line, so you can search on the
- subject title above to find the section quickly.
-
- This article is copyright (c) 1993 by Ilana Stern. It may be freely
- distributed provided that this copyright notice and the information
- on retrieving a current copy are not removed.
-
- Comments, corrections, or additions should be sent to Ilana Stern
- at ilana@ncar.ucar.edu.
-
- ---------------
- Subject: How to use this document
-
- Most FAQ (Frequently Asked Questions) documents list many questions and
- their answers. This FAQ is (mostly) devoted to answering only one question:
-
- "Where can I find documentation and software for [X] data format?"
-
- As the amount of information available over the networks has been
- increasing, so have the methods by which this information can be obtained.
- No longer is direct usage of FTP the only, or even the most frequent, method
- of obtaining data; we now have Gopher, Wais, and WWW, as well as many
- site-specific interfaces. Because the information itself may be accessible
- in many different ways, this FAQ will identify resources in terms of
- URLs (Uniform Resource Locators). This will also help us convert this
- FAQ to a hypertext document, so that it can be used with a WWW browser
- to go directly to any of the listed sources.
- Here's a glossary, so you can decode the URLs if necessary to reach
- the sites:
-
- ftp://host.name.domain/directory/[filename] ftp site
- http://host.name.domain/directory/[filename] www server
- telnet://host.name.domain telnet site
- gopher://host.name.domain gopher server
- wais://host.name.domain wais server
- news:newsgroup.name newsgroup
-
- So, for example, if a document is available at ftp://ncardata.ucar.edu/
- it means that you should ftp to ncardata.ucar.edu, and the information is
- in the top-level directory.
- If you don't know what these information retrieval methods are, see
- the section "How to use the data retrieval methods".
-
- ---------------
- Subject: How to get a current copy of this document
-
- If you are reading this document after 12 Jan 1993, you are reading an
- outdated copy. A current copy of this document can be obtained by anonymous
- FTP to ftp://rtfm.mit.edu/pub/usenet/news.answers/sci-data-formats.
- If you don't know what FTP is, see the section "How to use the data retrieval
- methods".
-
- If you can't use FTP, send email to mail-server@rtfm.mit.edu with
- send /pub/usenet/news.answers/sci-data-formats
- as the only text in the message (leave the subject blank).
-
- A current hypertext version of this document can be obtained from
- http://fits.cv.nrao.edu/traffic/scidataformats/faq.html, from
- http://www.ncsa.uiuc.edu/SDG/Software/HDF/SciDataFormatsFAQ.html,
- or (for European users in particular) from
- http://info.mcc.ac.uk:80/CGU/Visualisation/sdf.html. If
- you would like to archive this FAQ in either hypertext or plaintext
- format, and want to receive a new copy automatically at every update,
- please send me email.
-
- ---------------
- Subject: Resources for format information
-
- 1) CDF
- 2) FITS
- 3) GRIB
- 4) HDF
- 5) netCDF
- 6) Miscellaneous graphics formats
-
- 1. CDF
- CDF (Common Data Format) is a library and toolkit for multi-dimensional
- data sets. The basic component of CDF is a software programming interface
- that is a device independent view of the CDF data model. A user's guide
- and software is available from ftp://nssdca.gsfc.nasa.gov/cdf/.
- [Note: this is a VMS FTP site and can't be reached via Xmosaic -- if you
- are reading the hypertext version of this FAQ, use
- ftp://hypatia.gsfc.nasa.gov/nssdc_anon/cdf/ instead.]
- Questions can be directed to cdfsupport@nssdca.gsfc.nasa.gov.
- A client-server software layer called CSCDF, which can be used with
- the CDF library to provide applications access to remote CDF datasets,
- can be obtained from its author, Hillel Steinberg, by email at
- zeus@cs.umd.edu.
-
- 2. FITS
-
- FITS (Flexible Image Transport System) is the standard data interchange
- and archival format of the worldwide astronomy community. The NOST Standard
- and User's Guide, some software, and test files are available from
- ftp://nssdca.gsfc.nasa.gov/fits/ by direct ftp, or from
- ftp://hypatia.gsfc.nasa.gov/nssdc_anon/fits/ via a WWW browser.
- The site ftp://fits.cv.nrao.edu/fits (accessible via WWW at
- http://fits.cv.nrao.edu/) has other software and a different set of test files,
- and electronic copies of FITS proposals that are under development or in the
- international approval process. Archives of news:sci.data.formats and
- news:sci.astro.fits (which is devoted to discussion of FITS) that are of
- interest to astronomers can be found in ftp://fits.cv.nrao.edu/fits/traffic/.
- A WAIS index that can be searched for FITS information is at
- http://info.cern.ch:8001/fits.cv.nrao.edu:210/nrao-fits.
- If you've searched all these resources and still have questions, you
- can direct them to fits@nssdca.gsfc.nasa.gov.
-
- 3. GRIB
-
- GRIB (GRid In Binary) is the World Meteorological Organization (WMO)
- standard for gridded meteorological data. Unfortunately it is still not
- very "standard", as some organizations use their own versions. A format
- description for WMO GRIB can be found at
- ftp://ncardata.ucar.edu/datasets/ds084.5/format_grib,
- and read code is in the file access_grib.f in the same directory.
- If you need GRIB to read ECMWF data, the above format description, along
- with the ECMWF-specific parameter table, and a list of differences between
- WMO and ECMWF GRIB, is in ftp://ncardata.ucar.edu/datasets/ds111.2/format.
- Read code can be found in ftp://ncardata.ucar.edu/datasets/ds111.2/software.
- If all else fails, contact Ilana Stern at ilana@ncar.ucar.edu.
-
- 4. HDF
-
- HDF (Hierarchical Data Format) is a self-defining file format for transfer
- of various types of data between different machines. The HDF library contains
- interfaces for storing and retrieving compressed or uncompressed raster images
- with palettes, and an interface for storing and retrieving n-Dimensional
- scientific datasets together with information about the data, such as labels,
- units, formats, and scales for all dimensions.
- Source code and documentation are on ftp://ftp.ncsa.uiuc.edu/HDF.
- Some general information on HDF, including a FAQ, is available from
- http://www.ncsa.uiuc.edu/SDG/Software/HDF/HDFIntro.html.
-
- 5. netCDF
-
- NetCDF (Network Common Data Form) is an interface for scientific data
- access which implements a machine-independent, self-describing, extendible
- file format. Source code and documentation for the netCDF data access library
- is available from ftp://ftp.unidata.ucar.edu/pub/netcdf or
- gopher://gopher.unidata.ucar.edu:70. A FAQ is here also, as well as pointers
- to other available software packages which use netCDF data. Questions and
- answers about netCDF can be searched in the index
- wais://wais.unidata.ucar.edu:210/unidata-support-netcdf.src.
- A recent paper (Jenter and Signell, 1992) which provides a good introduction
- to netCDF is available as ftp://crusty.er.usgs.gov/pub/netcdf.asce.ps.
- A mailing list, netcdfgroup@unidata.ucar.edu, exists for discussion of the
- netCDF interface, and for announcements of netCDF news: to subscribe, send
- a request to netcdfgroup-adm@unidata.ucar.edu. The archives of netcdfgroup
- are available from ftp://ftp.unidata.ucar.edu/mail-archives/netcdfgroup,
- and also in the searchable index
- wais://wais.unidata.ucar.edu:210/netcdf-group.src.
- For more information, contact support@unidata.ucar.edu.
-
- 6. Miscellaneous graphics formats
-
- These formats for storing graphics files -- TIFF, GIF, JPEG, FLI, CGM,
- and so on -- are more properly discussed in news:comp.graphics. A small
- amount of documentation on these and other graphics formats is on
- ftp://zamenhof.cs.rice.edu/pub/graphics.formats; other archive sites
- are ftp://ftp.ncsa.uiuc.edu/misc/file.formats/graphics.formats, and
- ftp://telva.ccu.uniovi.es/pub/graphics/Image.
- The comp.graphics FAQ and resource file have more information on where
- to find read and conversion programs for these formats. You can find
- them at ftp://rtfm.mit.edu/pub/usenet/news.answers.
- A good (hardcopy) reference for graphics formats is _Graphics
- File Formats_, by David C. Kay and John R. Levine (Windcrest Books,
- ISBN 0-8306-3060-0, about US$30.00 in paperback).
-
- ---------------
- Subject: How to use the data retrieval methods
-
- This section only describes FTP and telnet in any detail; for other
- methods, FTP sites are given, so you can get information on them yourself.
-
- 1) How to use FTP
- 2) How to use telnet
- 3) Gopher information
- 4) Wais information
- 5) WWW information
-
- 1. How to use FTP
-
- FTP (File Transfer Protocol) allows transfer of files between two computers
- which are on the Internet. To access the FTP areas listed here, at your
- system prompt type "ftp" followed by the name of the desired system. For
- example, to access ncardata.ucar.edu you'd type
-
- ftp ncardata.ucar.edu
-
- Use "anonymous" as your login and your email address as the password (if
- requested).
-
- [Note: quotes ("like this") are used to set off names of directories and
- files, or commands you'd type, and are not part of these names.]
-
- Not all FTP systems accept the same commands, but here's a list of the
- most useful:
-
- ls list files in the current directory.
- cd change directory, e.g. "cd wx" changes to the wx directory.
- binary sets binary mode
- ascii sets ascii mode (the default). Use for retrieving text.
- get retrieves a file, e.g. "get readme" gets a file called readme.
- bye exits FTP.
-
- If you can't seem to connect to the site, check to see if it is a telnet
- site. If it is, follow the instructions in the following section instead.
-
- If you can't FTP from your site, use one of the following ftp-by-mail servers:
-
- ftpmail@decwrl.dec.com
- ftpmail@src.doc.ic.ac.uk
- ftpmail@cs.uow.edu.au
- ftpmail@grasp.insa-lyon.fr
-
- Send an e-mail message to the closest address, with the lines:
-
- reply your_address@some.where <- with your email address
- connect ncardata.ucar.edu <- for example
- cd datasets/ds111.2/software
- get access_sun.f
- quit
-
- For complete instructions, send a one-line message reading "help" to the
- server. Please don't ask me for help!
-
- 2. How to use telnet
-
- Type "telnet" followed by the name or IP number of the desired system. These
- publicly accessible systems generally allow you to log in but put you in
- a restricted shell, from which only a certain menu of commands is available.
- The description for the site will include the login to use.
-
- If you can't seem to connect to the site, re-check its description in the
- document; if it's an FTP site, follow the instructions in the previous
- section instead.
-
- 3. Gopher information
-
- Available by ftp at ftp://rtfm.mit.edu/pub/usenet/news.answers/gopher-faq.
-
- 4. Wais information
-
- Available by ftp at
- ftp://rtfm.mit.edu/pub/usenet/news.answers/wais-faq/getting-started.
-
- 5. WWW information
-
- Available by ftp at ftp://rtfm.mit.edu/pub/usenet/news.answers/www/faq.
- WWW is so easy to use that you might as well just hop in and try it, so
- ask your sysadmin if you have a WWW browser such as NCSA Mosaic.
-
- ---------------
- Subject: Why isn't my favorite format on this list?
-
- If you don't see a format you're interested in here, it could be one
- of three reasons. First of all, there are a lot of formats which are
- out of the scope of this newsgroup: it ain't named *sci*.data.formats
- for nuthin', you know. Formats used in commercial spreadsheet and
- word-processing software aren't scientific data formats, and aren't
- discussed in this group.
-
- Second, it may be that nobody has given the FAQ organizer any information
- on sources for information on that format. So ask the newsgroup -- and
- if you do get a response, please let me know what it is!
-
- Finally, you may ask on the net, and hear nothing, because the
- data format description just *isn't* publicly available. For most
- scientific data formats, this is a Bad Thing, and most archivists and
- scientists *want* to have their format information available. If
- you have such information, but don't have resources to make it
- available, please ask around and see if you can get it into an FTP
- area or other resource. Please don't publicize private or proprietary
- formats without the permission of the author, though.
-