\input texinfo @c -*-texinfo-*- 
@c \input /home/zender/texinfo_fedora.tex
@c \input /home/zender/texinfo_ubuntu.tex

@c 19980817: TeX-based systems (texi2dvi, texi2dvi --pdf) require texinfo.tex
@c Hyper-text systems (texi2html, makeinfo) do not require texinfo.tex

@ignore
$Header$

Purpose: TeXInfo documentation for netCDF Operators (NCO)

URL: http://nco.sf.net/nco.texi

Copyright (C) 1995--present Charlie Zender
Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.3
or any later version published by the Free Software Foundation;
with no Invariant Sections, no Front-Cover Texts, and no Back-Cover
Texts. The license is available online at 
http://www.gnu.org/copyleft/fdl.html

The original author of this software, Charlie Zender, seeks to improve
it with your suggestions, contributions, bug-reports, and patches.
Charlie Zender <surname at uci dot edu> (yes, my surname is zender)
Department of Earth System Science
3200 Croul Hall
University of California, Irvine
Irvine, CA 92697-3100

After editing any hyperlink locations, Emacs users update indices with
C-c C-u C-a	texinfo-all-menus-update
C-c C-u C-e	texitnfo-every-node-update

Multiple files:
M-x texinfo-multiple-files-update

Usage:
export TEX='tex --src-specials'
cd ~/nco/doc;texi2dvi nco.texi;makeinfo --html --ifinfo --no-split --output=nco.html nco.texi;makeinfo nco.texi;dvips -o nco.ps nco.dvi;texi2dvi --pdf nco.texi;makeinfo --xml --ifinfo --no-split --output=nco.xml nco.texi;makeinfo --no-headers --output=nco.txt nco.texi
cd ~/nco/doc;/usr/bin/scp index.shtml nco_news.shtml ChangeLog TODO VERSION nco.dvi nco.html nco.info* nco.pdf nco.ps nco.texi nco.txt nco.xml nco_rfr_crd.pdf ../README ../data/ncap.in ../data/ncap2.in zender,nco@web.sf.net:/home/project-web/nco/htdocs;cd -
cd ~/nco/doc;/usr/bin/scp index.shtml nco_news.shtml ChangeLog TODO VERSION nco.dvi nco.html nco.info* nco.pdf nco.ps nco.texi nco.txt nco.xml nco_rfr_crd.pdf ../README ../data/ncap.in ../data/ncap2.in dust.ess.uci.edu:Sites/nco;cd -
dvips -o nco.ps nco.dvi 
dvips -Ppdf -G0 -o nco.ps nco.dvi
makeinfo --html --ifinfo --no-split --output=nco.html nco.texi
makeinfo --no-split --output=nco.info nco.texi
makeinfo --no-headers --no-split --output=nco.txt nco.texi
makeinfo --xml --no-split --output=nco.xml nco.texi
makeinfo --docbook --no-split --output=nco.xml nco.texi
pdftotext nco.pdf nco.txt
ps2pdf -dMaxSubsetPct=100 -dCompatibilityLevel=1.2 -dSubsetFonts=true -dEmbedAllFonts=true nco.ps nco.pdf
texi2dvi --output=nco.dvi nco.texi 
texi2dvi --pdf --output=nco.pdf nco.texi
texi2html -monolithic -verbose nco.texi
texi2html -l2h -l2h_tmp=./l2h_tmp -monolithic -verbose nco.texi # Invoke latex2html
# 20130806 Copy CMIP5 images (takes additional time)
cd ~/nco/doc/xmp;/usr/bin/scp fgr*.png fgr*.eps zender,nco@web.sf.net:/home/project-web/nco/htdocs/xmp;cd -
# 20120203 Copy all nco.html PNG images (takes additional time)
cd ~/nco/doc;/usr/bin/scp nco_*.png zender,nco@web.sf.net:/home/project-web/nco/htdocs;cd -
# 20130801 Copy nco.texi to Ubuntu machine for quick build-tests
scp ~/nco/doc/nco.texi givre.ess.uci.edu:nco/doc

NB: @cindex references in footnotes propagate to PDF file, not to
    HTML files, bug-report sent to makeinfo people 20040229

Producing HTML, makeinfo vs. texi2html:
makeinfo: Better format overall 
          Uses node names for cross references and index
          Acronyms look better
          Excludes @ifinfo sections by default (override with --ifinfo)
texi2html: Index sub-divided by first character
           More fancy options (e.g., latex2html), though none very useful 
           Misprints title
           Includes @ifinfo sections by default

Official TeXInfo documentation:
http://www.gnu.org/software/texinfo/manual/texinfo/texinfo.html

Legend (defined in "highlighting" section of TeXInfo manual):
@b{}: Selects bold face
@i{}: Selects an italic font
@r{}: Selects a Roman font (usefule in example-like environments to have comments in Roman)
@t{}: Selects the fixed-width font used by @code{}
@sansserif{}: Selects a sanserif font
@slanted{}: Selects a slanted font
@caption{}: Caption of floating figures
@code{}: Program text, e.g., @code{if(foo) x=y;}
@command{}: Commands, e.g., @command{ncra}
@dfn{}: Define use of term, e.g., @dfn{supercalifragilisticexpialidocious}
@email{}: E-mail address, e.g., @email{surname at uci dot edu}
@emph{}: Emphasize text, e.g., @emph{important}
@env{}: Environment variable, e.g., @env{HOME}
@file{}: Filename, e.g., @file{in.nc}
@float: Environment for TeX floating figures (e.g., @float Figure,fgr:glb)
@image{drc/nm,sz} Directory, filename (w/o suffix) and figure horizontal size
@html: Text until @end html passed without translation
@ifhtml: Text until @end ifhtml passed with translation
@kbd{}: Keyboard input, e.g., @kbd{ncra in.nc out.nc}
@key{}: Key name, e.g., @key{ESC} (rarely needed)
@option{}: Command-line option, e.g., @option{--dbg}
@samp{}: Extended commands, character sequences, e.g., @samp{ncra in.nc out.nc}
@uref{}: URL with optional alternate text, e.g., @uref{http://nco.sf.net,NCO homepage}
@url{}: URL, synonym for @uref
@var{}: Metasyntactic variable, e.g., @var{fl_in}
@verbatim: Anything goes inside environment (no @'s needed to protect special characters like braces)
@verbatiminclude: Insert contents of file here, e.g., @verbatiminclude{nco.sh}
@example: Quoted environment (@'s needed to protect special characters like braces)
@w{}: Unbreakable text, e.g., @w{of 1}

Use '@*' to force hard carriage return
Use '@:', after periods, questions marks, exclamation marks, or colons
that do not end sentences, e.g., 'vs.@:'
Use '@.', '@!', and `@?' to end sentences that end with single capital letters (e.g., initials)

Outline Hierarchy:
@section
@subsection
@unnumberedsubsec

Resources:
Octave TeXInfo manual shows clean TeXInfo structure
/usr/share/doc/octave-2.1.34/interpreter/octave.texi
@end ignore

@c Start of header

@c No variables may be defined before TeXInfo @setfilename header
@setfilename nco.info

@c Define edition, date, ...
@set nco-edition 5.1.8
@set doc-edition 5.1.8
@set copyright-years 1995--2023
@set update-year 2023
@set update-date 21 August 2023
@set update-month August 2023

@settitle @acronym{NCO} @value{nco-edition} User Guide

@c Uncomment following line to produce guide in smallbook format
@c @smallbook
@c Merge function index into concept index
@syncodeindex fn cp

@c 20090226 Add bibliography capabilities as per
@c http://lists.gnu.org/archive/html//help-texinfo/2004-12/txtPW9h_VG8ez.txt
@include my-bib-macros.texi
@mybibuselist{References}

@c 20150616 Add version info as per
@c http://www.gnu.org/software/automake/manual/html_node/Texinfo.html
@c Seems to require running automake --add-missing on local machine
@c Nightmare because it means clients must run autoconf! Fuggetaboutit
@c @include version.texi

@c end of header

@c TeXInfo macros may not appear before TeXInfo @setfilename header
@c [idx] Index
@macro idx {}
i
@end macro
@c [m s-1] Meridional wind speed
@macro wndmrd {}
v
@end macro
@c [m s-1] Zonal wind speed
@macro wndznl {}
u
@end macro
@macro xxx {}
x
@end macro
@c [K] Temperature
@macro tpt {}
T
@end macro
@c TeX macros may appear anywhere after line 1
@tex
% Define TeX macros to roughly correspond to LaTeX style files
% Use \gdef instead of \def to make definition persistent across TeX blocks
% These should be consistent with any TeXInfo macros
% 0. Alphabet
\gdef\ddd{d} % [ltr] d
\gdef\iii{i} % [ltr] i
\gdef\jjj{j} % [ltr] j
\gdef\kkk{k} % [ltr] k
\gdef\qqq{q} % [ltr] q
\gdef\sss{s} % [ltr] s
\gdef\vvv{v} % [ltr] v
\gdef\xxx{x} % [ltr] x
\gdef\yyy{y} % [ltr] y
% 1. Primary commands
\gdef\cod{{\rm CO}_{2}} % [frc] Math pi
\gdef\dfr{{\rm d}} % [frc] Math differential fxm: upright
\gdef\dmnidx{n} % [idx] Dimension index
\gdef\clmnbr{I} % [nbr] Column number
\gdef\rownbr{J} % [nbr] Row number
\gdef\dmnnbr{N} % [nbr] Dimension number
\gdef\dmn{D} % [dmn] Variable dimension
\gdef\frcdst{f} % [flg] Destination grid fraction
\gdef\frcsgs{s} % [flg] Sub-gridscale fraction
\gdef\idx{i} % [idx] Index
\gdef\lmnidx{i} % [idx] Element index
\gdef\lmnnbr{N} % [nbr] Number of elements in input hyperslab
\gdef\me{{\rm e}} % [frc] Math e fxm: upright
\gdef\mi{{\rm i}} % [frc] Math i fxm: upright
\gdef\mpi{\pi} % [frc] Math pi
\gdef\mpp{\cal{M}} % [map] Map
\gdef\mskflg{m} % [flg] Mask flag
\gdef\mssflg{\mu} % [flg] Missing value flag
\gdef\outnbr{J} % [nbr] Number of elements in output hyperslab
\gdef\prmsbs{\prime} % [sbs] Prime subscript
\gdef\psl{\epsilon} % [frc] epsilon
\gdef\rdridx{r} % [idx] Re-order index
\gdef\rdrnbr{R} % [nbr] Re-order number
\gdef\rdr{R} % [dmn] Re-order dimension
\gdef\shridx{s} % [idx] Share index
\gdef\shrnbr{S} % [nbr] Share number
\gdef\shr{S} % [dmn] Share dimension
\gdef\tllnbr{M} % [nbr] Tally (number of valid elements in input hyperslab)
\gdef\tm{t} % [s] Time
\gdef\wgt{w} % [frc] Weight
\gdef\wndmrd{v} % [m s-1] Meridional wind speed
\gdef\wndznl{u} % [m s-1] Zonal wind speed

% 2. Derived commands
\gdef\pslavg{\bar{\epsilon}} % [frc] Mean error
\gdef\pslmax{\epsilon_{\rm max}} % [frc] Maximum error
\gdef\pslmin{\epsilon_{\rm min}} % [frc] Minimum error
\gdef\pslmabs{\epsilon_{\rm mabs}^{+}} % [frc] Maximum absolute error
\gdef\pslmibs{\epsilon_{\rm mibs}^{+}} % [frc] Minimum absolute error
\gdef\pslmebs{\bar{\epsilon}^{+}} % [frc] Mean absolute error
\gdef\dmnvct{{\bf \dmn}} % [vct] Dimension vector
\gdef\dmnprm{\dmn^{\prmsbs}} % [vct] Dimension prime
\gdef\dmnsubnnn{\dmn_{\dmnidx}} % [dmn] Dimension sub nnn
\gdef\shrsubnnn{\shr_{\dmnidx}} % [dmn] Share dimension sub nnn
\gdef\shrsubsss{\shr_{\shridx}} % [dmn] Share dimension sub sss
\gdef\dmnsubnnnprm{\dmn_{\dmnidx}^{\prmsbs}} % [vct] Dimension prime sub nnn 
\gdef\dmnvctprm{{\bf \dmn}^{\prmsbs}} % [vct] Dimension vector prime
\gdef\rdrvct{{\bf \rdr}} % [vct] Re-order vector
\gdef\shrvct{{\bf \shr}} % [vct] Share vector
\gdef\xxxprm{\xxx^{\prmsbs}} % [ltr] x prime

% 3. Doubly derived commands

@end tex

@c install-info installs NCO info into this category
@dircategory netCDF
@direntry
* NCO::        User Guide for the netCDF Operator suite
@end direntry
@iftex
@tolerance 10000
@end iftex

@c Set smallbook if printing in smallbook format
@c Example of smallbook font is actually written using smallbook
@c In bigbook, a kludge is used for TeX output
@c set smallbook
@clear smallbook

@tex
% fxm: Try to get thumbnails working with texinfo --pdf -generated PDF files
% \input thumbpdf.sty
% Experiment with smaller amounts of whitespace between chapters and sections
\global\chapheadingskip = 15pt plus 4pt minus 2pt 
\global\secheadingskip = 12pt plus 3pt minus 2pt
\global\subsecheadingskip = 9pt plus 2pt minus 2pt
@end tex

@c Experiment with smaller amounts of whitespace between paragraphs in the 8.5 by 11 inch format
@ifclear smallbook
@tex
\global\parskip 6pt plus 1pt
@end tex
@end ifclear

@c Uncomment next line to remove ugly TeX warning blocks from overfull hboxes
@finalout

@ifinfo
This file documents @acronym{NCO}, a collection of utilities to
manipulate and analyze netCDF files.

Copyright @copyright{} @value{copyright-years} Charlie Zender

This is the first edition of the @cite{NCO User Guide},@*
and is consistent with @w{version 2} of @file{texinfo.tex}.

Permission is granted to copy, distribute and/or modify this document 
under the terms of the @acronym{GNU} Free Documentation License, @w{Version 1.3}
or any later version published by the Free Software Foundation;
with no Invariant Sections, no Front-Cover Texts, and no Back-Cover
Texts. The license is available online at 
@uref{http://www.gnu.org/copyleft/fdl.html}

The original author of this software, Charlie Zender, wants to improve it
with the help of your suggestions, improvements, bug-reports, and patches.@*
Charlie Zender <surname at uci dot edu> (yes, my surname is zender)@*
3200 Croul Hall@*
Department of Earth System Science@*
University of California, Irvine@*
Irvine, CA 92697-3100@*

@ignore
Permission is granted to process this file through TeX and print the 
results, provided the printed document carries copying permission
notice identical to this one except for the removal of this paragraph
(this paragraph not being relevant to the printed manual).
@end ignore
@end ifinfo

@setchapternewpage odd

@titlepage
@html
<meta name="Author" content="Charlie Zender">
<meta name="Keywords" content="NCO documentation, NCO User Guide,
netCDF, operator, GCM, CCM, scientific data, ncbo, ncclimo, ncfe, ncecat,
ncflint, ncks, ncra, ncrcat, ncremap, ncrename, ncwa">
@end html
@html
<body text="#000000" link="#0000EF" vlink="#008080" alink="#FF0000">
<font face="Arial">
@end html
@title NCO User Guide
@subtitle A suite of netCDF operators
@subtitle Edition @value{doc-edition}, for @acronym{NCO} Version @value{nco-edition}
@subtitle @value{update-month}

@author by Charlie Zender
@author Departments of Earth System Science and Computer Science
@author University of California, Irvine
@html
<p>WWW readers: Having trouble finding the section you want?</p> 
<p>Search for keywords in the <a href="#index">(hyper) index</a> at the end</p>
@end html

@c Include Distribution inside titlepage so that headings are turned off
@page
@vskip 0pt plus 1filll
Copyright @copyright{} @value{copyright-years} Charlie Zender.

@sp 2
This is the first edition of the @cite{NCO User Guide},@*
and is consistent with @w{version 2} of @file{texinfo.tex}.
@sp 2

Published by Charlie Zender@*
Department of Earth System Science@*
3200 Croul Hall@*
University of California, Irvine@*
Irvine, CA 92697-3100 USA@*

Permission is granted to copy, distribute and/or modify this document
under the terms of the @acronym{GNU} Free Documentation License, @w{Version 1.3}
or any later version published by the Free Software Foundation;
with no Invariant Sections, no Front-Cover Texts, and no Back-Cover
Texts. The license is available online at 
@uref{http://www.gnu.org/copyleft/fdl.html}
@sp 2
We gratefully acknowledge support for @acronym{NCO} development and
maintenance provided by these institutions and programs:
@acronym{DOE} @acronym{ACME} @acronym{DE-SC0012998},
@acronym{LLNL-B625903}, @acronym{LLNL-B632442}, 
@acronym{NASA} @acronym{ACCESS} @acronym{NNX12AF48A} and
@acronym{NNX14AH55A}, and @acronym{NSF} @acronym{SEI}
@acronym{IIS-0431203}, @acronym{AGS-1541031}, and @acronym{OAC-2004993}. 
This research was supported as part of the Energy Exascale Earth System
Model (@acronym{E3SM}) project, formerly known as Accelerated Climate
Modeling for Energy (@acronym{ACME}),
funded by the U.S. Department of Energy, Office of Science, Office of
Biological and Environmental Research.
This material is based upon work supported by the National Science
Foundation.
@sp 2
The original author of this software, Charlie Zender, wants to improve it
with the help of your suggestions, improvements, bug-reports, and patches.@*
Charlie Zender <surname at uci dot edu> (yes, my surname is zender)@*
Department of Earth System Science@*
3200 Croul Hall@*
University of California, Irvine@*
Irvine, CA 92697-3100@*
@sp 2
@end titlepage

@c Print table of contents
@contents

@node Top, Foreword, (dir), (dir)
@ifinfo
@top NCO User Guide
@c node format: node-name, next, previous, up

@ifhtml
@cartouche
@html
<p><b>Note to readers of the NCO User Guide in HTML format</b>: 
<b>The <a href="./nco.pdf">NCO User Guide in PDF format</a> 
(also on <a href="http://nco.sf.net/nco.pdf">SourceForge</a>)
contains the complete NCO documentation.</b> 
<br>This HTML documentation is equivalent except it refers you to the
printed (i.e., DVI, PostScript, and PDF) documentation for description
of complex mathematical expressions. Also, images appear only in the
PDF document due to SourceForge limitations.
@end html
@end cartouche
@end ifhtml
@ifnothtml
@emph{Note to readers of the NCO User Guide in Info format}: 
@emph{The @uref{./nco.pdf,NCO User Guide in PDF format}
(also on @uref{http://nco.sf.net/nco.pdf,SourceForge})
contains the complete @acronym{NCO} documentation.}
This Info documentation is equivalent except it refers you to the 
printed (i.e., DVI, PostScript, and PDF) documentation for description 
of complex mathematical expressions. Also, images appear only in the
PDF document due to SourceForge limitations.
@end ifnothtml

The netCDF Operators, or @acronym{NCO}, are a suite of programs known as 
operators. 
The operators facilitate manipulation and analysis of data stored in the 
self-describing netCDF format, available from
(@uref{http://www.unidata.ucar.edu/software/netcdf}).
Each @acronym{NCO} operator (e.g., @command{ncks}) takes netCDF input
file(s), performs an operation (e.g., averaging, hyperslabbing, or
renaming), and outputs a processed netCDF file. 
Although most users of netCDF data are involved in scientific research,
these data formats, and thus @acronym{NCO}, are generic and are equally
useful in fields from agriculture to zoology.
The @acronym{NCO} User Guide illustrates @acronym{NCO} use with
examples from the field of climate modeling and analysis. 
The @acronym{NCO} homepage is @uref{http://nco.sf.net}, and the 
source code is maintained at @uref{http://github.com/nco/nco}.

This documentation is for @acronym{NCO} version @value{nco-edition}.
It was last updated @value{update-date}.
Corrections, additions, and rewrites of this documentation are
gratefully welcome.

Enjoy,@*
Charlie Zender
@end ifinfo

@menu
* Foreword::
* Summary::
* Introduction::
* Strategies::
* Shared features::
* Reference Manual::
* Contributing::
* Quick Start::
* CMIP5 Example::
* Parallel::
* CCSM Example::
* mybibnode::
* General Index::
@end menu

@html
<a name="fwd"></a> <!-- http://nco.sf.net/nco.html#fwd -->
@end html
@node Foreword, Summary, Top, Top
@unnumbered Foreword
@cindex foreword
@cindex Charlie Zender
@acronym{NCO} is the result of software needs that arose while I worked
on projects funded by @acronym{NCAR}, @acronym{NASA}, and @acronym{ARM}.
Thinking they might prove useful as tools or templates to others,  
it is my pleasure to provide them freely to the scientific community.  
Many users (most of whom I have never met) have encouraged the
development of @acronym{NCO}.  
Thanks espcially to Jan Polcher, Keith Lindsay, Arlindo @w{da Silva},
John Sheldon, and William Weibel for stimulating suggestions and
correspondence. 
Your encouragment motivated me to complete the @cite{NCO User Guide}.
So if you like @acronym{NCO}, send me a note!
@w{I should} mention that @acronym{NCO} is not connected to or
officially endorsed by Unidata, @acronym{ACD}, @acronym{ASP},
@acronym{CGD}, or Nike.@* 
@sp 1
@noindent
Charlie Zender@*
May 1997@*
Boulder, Colorado@*

@sp 2
Major feature improvements entitle me to write another Foreword.
In the last five years a lot of work has been done to refine
@acronym{NCO}. 
@cindex open source
@acronym{NCO} is now an open source project and appears to be much
healthier for it. 
The list of illustrious institutions that do not endorse @acronym{NCO}     
continues to grow, and now includes @acronym{UCI}.@* 
@sp 1
@noindent
Charlie Zender@*
October 2000@*
Irvine, California@*

@sp 2
The most remarkable advances in @acronym{NCO} capabilities in the last  
few years are due to contributions from the Open Source community.
Especially noteworthy are the contributions of Henry Butowsky and Rorik 
Peterson.@* 
@sp 1
@noindent
Charlie Zender@*
January 2003@*
Irvine, California@*

@sp 2
@acronym{NCO} was generously supported from 2004--2008 by US 
National Science Foundation (@acronym{NSF}) grant 
@uref{http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0431203,IIS-0431203}. 
This support allowed me to maintain and extend core @acronym{NCO} code,
and others to advance @acronym{NCO} in new directions: 
Gayathri Venkitachalam helped implement @acronym{MPI};
Harry Mangalam improved regression testing and benchmarking;
Daniel Wang developed the server-side capability, @acronym{SWAMP};
and Henry Butowsky, a long-time contributor, developed @command{ncap2}.
This support also led @acronym{NCO} to debut in professional journals
and meetings.  
The personal and professional contacts made during this evolution have
been immensely rewarding.@*
@sp 1
@noindent
Charlie Zender@*
March 2008@*
Grenoble, France@*

@sp 2
The end of the @acronym{NSF} @acronym{SEI} grant in August, 2008 curtailed
@acronym{NCO} development.  
Fortunately we could justify supporting Henry Butowsky on other research  
grants until May, 2010 while he developed the key @command{ncap2}
features used in our climate research.
And recently the @acronym{NASA} @acronym{ACCESS} program commenced
funding us to support netCDF4 group functionality.
Thus @acronym{NCO} will grow and evade bit-rot for the foreseeable
future. 

I continue to receive with gratitude the thanks of @acronym{NCO} users
at nearly every scientific meeting I attend.  
People introduce themselves, shake my hand and extol @acronym{NCO},
often effusively, while I grin in stupid embarassment. 
These exchanges lighten me like anti-gravity.
Sometimes I daydream how many hours @acronym{NCO} has turned from grunt
work to productive research for researchers world-wide, or from research
into early happy-hours. 
It's a cool feeling.

@sp 1
@noindent
Charlie Zender@*
April, 2012@*
Irvine, California@*

@sp 2
@c 20150619 Test values automagically installed by version.texi, which is a nightmare
@c Test print @value{EDITION}, @value{VERSION}, @value{UPDATED}, @value{UPDATED-MONTH}.
The @acronym{NASA} @acronym{ACCESS} 2011 program generously supported 
(Cooperative Agreement NNX12AF48A) @acronym{NCO} from 2012--2014.
This allowed us to produce the first iteration of a Group-oriented 
Data Analysis and Distribution (@acronym{GODAD}) software ecosystem. 
Shifting more geoscience data analysis to @acronym{GODAD} is a 
long-term plan.
Then the @acronym{NASA} @acronym{ACCESS} 2013 program agreed to
support (Cooperative Agreement NNX14AH55A) @acronym{NCO} from
2014--2016.  
This support permits us to implement support for Swath-like Data
(@acronym{SLD}).
Most recently, the @acronym{DOE} has funded me to implement 
@acronym{NCO} re-gridding and parallelization in support of their 
@acronym{ACME} program.
After many years of crafting @acronym{NCO} as an after-hours hobby,
I finally have the cushion necessary to give it some real attention. 
And I'm looking forward to this next, and most intense yet, phase of
@acronym{NCO} development.
@sp 1
@noindent
Charlie Zender@*
June, 2015@*
Irvine, California@*

The @acronym{DOE} Energy Exascale Earth System Model (@acronym{E3SM})
project (formerly @acronym{ACME}) has generously supported @acronym{NCO}
development for the past four years.
Supporting @acronym{NCO} for a mission-driven, high-performance
climate model development effort has brought unprecedented challenges 
and opportunities.
After so many years of staid progress, the recent development speed
has been both exhilirating and terrifying.

@sp 1
@noindent
Charlie Zender@*
May, 2019@*
Laguna Beach, California@*

The @acronym{DOE} @acronym{E3SM} project has supported @acronym{NCO}
development and maintenance since 2015. 
This is an eternity in the world of research funding!
Their reliable support has enabled us to add cutting-edge features
including quantization, vertical interpolation, and support for
multiple regridding weight-generators.
Recently @acronym{NSF} supported us to enable user-friendly support
for modern compression algorithms that can make geoscience data 
analysis greener by reducing dataset size, and thereby storage,
power, and associated greenhouse gas emissions.
I am grateful for this this agency support that inspires me to create
new features that help my amazing colleagues pursue their scientific
ideas.  

@sp 1
@noindent
Charlie Zender@*
July, 2022@*
Laguna Beach, California@*
@ignore
@end ignore

@html
<a name="smr"></a> <!-- http://nco.sf.net/nco.html#smr -->
@end html
@node Summary, Introduction, Foreword, Top
@unnumbered Summary
@cindex operators
@cindex summary
This manual describes @acronym{NCO}, which stands for netCDF Operators.
@acronym{NCO} is a suite of programs known as @dfn{operators}.
Each operator is a standalone, command line program executed at the
shell-level like, e.g., @command{ls} or @command{mkdir}.  
The operators take netCDF files (including @acronym{HDF5} files
constructed using the netCDF @acronym{API}) as input, perform an
operation (e.g., averaging or hyperslabbing), and produce a netCDF file 
as output.  
The operators are primarily designed to aid manipulation and analysis of 
data.
The examples in this documentation are typical applications of the
operators for processing climate model output. 
This stems from their origin, though the operators are as general as
netCDF itself.

@html
<a name="ntr"></a> <!-- http://nco.sf.net/nco.html#ntr -->
@end html
@node Introduction, Strategies, Summary, Top
@chapter Introduction
@cindex introduction

@menu
* Availability::
* How to Use This guide::
* Compatability::
* Symbolic Links::
* Libraries::
* netCDF2/3/4 and HDF4/5 Support::
* Help Requests and Bug Reports::
@end menu

@node Availability, How to Use This guide, Introduction, Introduction
@section Availability
@cindex @acronym{NCO} availability
@cindex source code
The complete @acronym{NCO} source distribution is currently distributed
as a @dfn{compressed tarfile} from
@uref{http://sf.net/projects/nco}
and from
@uref{http://dust.ess.uci.edu/nco/nco.tar.gz}.
The compressed tarfile must be uncompressed and untarred before building
@acronym{NCO}.
Uncompress the file with @samp{gunzip nco.tar.gz}. 
Extract the source files from the resulting tarfile with @samp{tar -xvf
nco.tar}.    
@acronym{GNU} @code{tar} lets you perform both operations in one step
with @samp{tar -xvzf nco.tar.gz}. 

@cindex documentation 
@cindex @acronym{WWW} documentation
@cindex on-line documentation
@cindex @acronym{HTML}
@cindex @TeX{}info
@cindex Info
@cindex @cite{User Guide}
@cindex @cite{NCO User Guide}
The documentation for @acronym{NCO} is called the 
@cite{NCO User Guide}. 
The @cite{User Guide} is available in @acronym{PDF}, Postscript,
@acronym{HTML}, @acronym{DVI}, @TeX{}info, and Info formats.
These formats are included in the source distribution in the files
@file{nco.pdf}, @file{nco.ps}, @file{nco.html}, @file{nco.dvi},
@file{nco.texi}, and @file{nco.info*}, respectively.
All the documentation descends from a single source file,
@file{nco.texi}
@footnote{   
To produce these formats, @file{nco.texi} was simply run through the
freely available programs @code{texi2dvi}, @code{dvips},
@code{texi2html}, and @code{makeinfo}.    
Due to a bug in @TeX{}, the resulting Postscript file, @file{nco.ps},
contains the Table of Contents as the final pages. 
Thus if you print @file{nco.ps}, remember to insert the Table of
Contents after the cover sheet before you staple the manual.
}.
Hence the documentation in every format is very similar.
However, some of the complex mathematical expressions needed to describe
@command{ncwa} can only be displayed in @acronym{DVI}, Postscript, and 
@acronym{PDF} formats. 

@cindex publications
@cindex presentations
A complete list of papers and publications on/about @acronym{NCO} 
is available on the @acronym{NCO} homepage.
Most of these are freely available. 
@c 20130526 fxm Replace mybibnode with @mybibnode{}
@c 20130526 Doing so, unfortunately, produces error "TeX capacity exceeded, sorry [input stack size=5000]."
@c 20130526 Denude document of @mybibcite{} until it works
@c The primary refereed publications are @mybibcite{ZeM06} and @mybibcite{Zen08}. 
The primary refereed publications are ZeM06 and Zen08. 
These contain copyright restrictions which limit their redistribution,
but they are freely available in preprint form from the @acronym{NCO}.

@cindex @acronym{NCO} homepage
If you want to quickly see what the latest improvements in @acronym{NCO}
are (without downloading the entire source distribution), visit the
@acronym{NCO} homepage at 
@uref{http://nco.sf.net}.
The @acronym{HTML} version of the @cite{User Guide} is also available 
online through the World Wide Web at @acronym{URL}
@uref{http://nco.sf.net/nco.html}.
@cindex netCDF
To build and use @acronym{NCO}, you must have netCDF installed.
The netCDF homepage is
@uref{http://www.unidata.ucar.edu/software/netcdf}.

New @acronym{NCO} releases are announced on the netCDF list 
and on the @code{nco-announce} mailing list 
@uref{http://lists.sf.net/mailman/listinfo/nco-announce}.

@ignore
This tests incorporates an image using the @code{@@image} command.
@image{/data/zender/ps/odxc,6in,}
@end ignore

@node How to Use This guide, Compatability, Availability, Introduction
@section How to Use This Guide
@cindex contents
@cindex introduction
Detailed instructions about
@uref{http://nco.sf.net/#Source, how to download the newest version}, 
and @uref{http://nco.sf.net/#bld, how to complie source code},
as well as a @uref{http://nco.sf.net/#FAQ, @acronym{FAQ}} and 
descriptions of @uref{http://nco.sf.net/#bug, Known Problems} etc.
are on our homepage 
(@uref{http://nco.sf.net/}).

There are twelve operators in the current version (@value{nco-edition}).
The function of each is explained in @ref{Reference Manual, Reference Manual}.
Many of the tasks that @acronym{NCO} can accomplish are described during
the explanation of common @acronym{NCO} Features (@pxref{Shared features}).
More specific use examples for each operator can be seen by visiting the
operator-specific examples in the @ref{Reference Manual}.
These can be found directly by prepending the operator name with the
@code{xmp_} tag, e.g., @uref{http://nco.sf.net/nco.html#xmp_ncks}.
Also, users can type the operator name on the shell command line to 
see all the available options, or type, e.g., @samp{man ncks} to see
a help man-page.

@acronym{NCO} is a command-line language.
You may either use an operator after the prompt (e.g., @samp{$} here),
like, 
@example
$ @command{operator} @option{[options]} @file{input} @file{[output]}
@end example
or write all commands lines into a shell script, as in
the @acronym{CMIP5} Example (@pxref{CMIP5 Example}).

If you are new to @acronym{NCO}, the Quick Start (@pxref{Quick Start})
shows simple examples about how to use @acronym{NCO} on different kinds
of data files.  
More detailed ``real-world'' examples are in the
@ref{CMIP5 Example, @acronym{CMIP5} Example}. 
The @ref{General Index, Index} is presents multiple keyword entries for
the same subject. 
If these resources do not help enough, please 
@pxref{Help Requests and Bug Reports}.

@node Compatability, Symbolic Links, How to Use This guide, Introduction
@section Operating systems compatible with @acronym{NCO}
@cindex @acronym{OS}
@cindex @acronym{IBM}
@cindex @acronym{NEC}
@cindex @acronym{SGI}
@cindex @acronym{HP}
@cindex @acronym{DEC}
@cindex @acronym{PGI}
@cindex Cray
@cindex Digital
@cindex Sun
@cindex Intel
@cindex Comeau
@cindex Compaq
@cindex Macintosh
@cindex Microsoft
@cindex Windows
@cindex PathScale
@cindex QLogic
@cindex compatability
@cindex portability
@cindex installation
In its time on Earth, @acronym{NCO} has been successfully ported and
tested on so many 32- and 64-bit platforms that if we did not write
them down here we would forget their names:  
@c alphabetize by OS name
@acronym{IBM AIX} 4.x, 5.x,
FreeBSD 4.x, 
@acronym{GNU}/Linux 2.x, LinuxPPC, LinuxAlpha, LinuxARM, LinuxSparc64,
LinuxAMD64, 
@acronym{SGI IRIX} 5.x and 6.x,
@w{MacOS X} 10.x, 
@acronym{DEC OSF}, 
@acronym{NEC} Super-UX 10.x, 
Sun SunOS 4.1.x, Solaris 2.x, 
@acronym{Cray UNICOS} 8.x--10.x,
and Microsoft Windows (95, 98, @acronym{NT}, 2000, @acronym{XP}, Vista,
7, 8, 10).
If you port the code to a new operating system, please send me a note 
and any patches you required.

@cindex @acronym{UNIX}
@cindex Unidata
@cindex UDUnits
The major prerequisite for installing @acronym{NCO} on a particular
platform is the successful, prior installation of the netCDF library
(and, as of 2003, the UDUnits library).
Unidata has shown a commitment to maintaining netCDF and UDUnits on all
popular @acronym{UNIX} platforms, and is moving towards full support for 
the Microsoft Windows operating system (@acronym{OS}).
Given this, the only difficulty in implementing @acronym{NCO} on a
particular platform is standardization of various @w{C}-language API
system calls. 
@acronym{NCO} code is tested for @acronym{ANSI} compliance by
compiling with @w{C99 compilers} including those from 
@cindex @command{CC}
@cindex @command{c++}
@cindex @command{cc}
@cindex @command{clang}
@cindex @command{como}
@cindex @command{cxx}
@cindex @command{gcc}
@cindex @command{g++}
@cindex @command{icc}
@cindex @command{MVS}
@cindex @command{pgcc}
@cindex @command{pgCC}
@cindex @command{pathcc}
@cindex @command{pathCC}
@cindex @command{xlC}
@cindex @command{xlc}
@acronym{GNU} (@samp{gcc -std=c99 -pedantic -D_BSD_SOURCE -D_POSIX_SOURCE} -Wall)
@footnote{
The @samp{_BSD_SOURCE} token is required on some Linux platforms where 
@command{gcc} dislikes the network header files like
@file{netinet/in.h}).},
Comeau Computing (@samp{como --c99}),
Cray (@samp{cc}),
@acronym{HP}/Compaq/@acronym{DEC} (@samp{cc}),
@acronym{IBM} (@samp{xlc -c -qlanglvl=extc99}),
Intel (@samp{icc -std=c99}),
@cindex @acronym{LLVM}
@acronym{LLVM} (@samp{clang}),
@acronym{NEC} (@samp{cc}),
PathScale (QLogic) (@samp{pathcc -std=c99}),
@acronym{PGI} (@samp{pgcc -c9x}),
@acronym{SGI} (@samp{cc -c99}),
and
Sun (@samp{cc}).
@cindex C++
@cindex @acronym{ISO}
@cindex @command{libnco}
@acronym{NCO} (all commands and the @command{libnco} library) and
the C++ interface to netCDF (called @command{libnco_c++}) comply with
the @acronym{ISO} C++ standards as implemented by
Comeau Computing (@samp{como}),
Cray (@samp{CC}),
@acronym{GNU} (@samp{g++ -Wall}),
@acronym{HP}/Compaq/@acronym{DEC} (@samp{cxx}),
@acronym{IBM} (@samp{xlC}),
Intel (@samp{icc}),
Microsoft (@samp{MVS}),
@acronym{NEC} (@samp{c++}),
PathScale (Qlogic) (@samp{pathCC}),
@acronym{PGI} (@samp{pgCC}),
@acronym{SGI} (@samp{CC -LANG:std}),
and
Sun (@samp{CC -LANG:std}).
@cindex @file{Makefile}
See @file{nco/bld/Makefile} and @file{nco/src/nco_c++/Makefile.old} for
more details and exact settings. 

@cindex @acronym{ANSI}
@cindex C89
@cindex @code{printf} 
Until recently (and not even yet), @acronym{ANSI}-compliant has meant
compliance with the 1989 @acronym{ISO} C-standard, usually called C89 (with
minor revisions made in 1994 and 1995).
C89 lacks variable-size arrays, restricted pointers, some useful
@code{printf} formats, and many mathematical special functions.
@cindex C99
These are valuable features of C99, the 1999 @acronym{ISO} C-standard. 
@acronym{NCO} is C99-compliant where possible and C89-compliant where
necessary. 
Certain branches in the code are required to satisfy the native
@acronym{SGI} and SunOS @w{C compilers}, which are strictly @acronym{ANSI} 
C89 compliant, and cannot benefit from C99 features.
However, C99 features are fully supported by modern @acronym{AIX},
@acronym{GNU}, Intel, @acronym{NEC}, Solaris, and @acronym{UNICOS}
compilers. 
@acronym{NCO} requires a C99-compliant compiler as of @acronym{NCO}
@w{version 2.9.8}, released in August, 2004.

The most time-intensive portion of @acronym{NCO} execution is spent in
arithmetic operations, e.g., multiplication, averaging, subtraction.
These operations were performed in Fortran by default until August,
1999.
This was a design decision based on the relative speed of Fortran-based
object code vs.@: C-based object code in late 1994.
@w{C compiler} vectorization capabilities have dramatically improved 
since 1994.
We have accordingly replaced all Fortran subroutines with @w{C functions}.
This greatly simplifies the task of building @acronym{NCO} on nominally
unsupported platforms.  
@cindex C language
As of August 1999, @acronym{NCO} built entirely @w{in C} by default.
This allowed @acronym{NCO} to compile on any machine with an
@acronym{ANSI} @w{C compiler}. 
@cindex C99
@cindex C89
@cindex @code{restrict}
In August 2004, the first C99 feature, the @code{restrict} type
qualifier, entered @acronym{NCO} in version 2.9.8. 
@w{C compilers} can obtain better performance with C99 restricted 
pointers since they inform the compiler when it may make Fortran-like
assumptions regarding pointer contents alteration.
Subsequently, @acronym{NCO} requires a C99 compiler to build correctly
@footnote{@acronym{NCO} may still build with an 
@acronym{ANSI} or @acronym{ISO} C89 or C94/95-compliant compiler if the
@w{C pre-processor} undefines the @code{restrict} type qualifier, e.g.,
by invoking the compiler with @samp{-Drestrict=''}.}.

@cindex @acronym{GSL}
@findex ncap2
In January 2009, @acronym{NCO} version 3.9.6 was the first to link
to the @acronym{GNU} Scientific Library (@acronym{GSL}).
@acronym{GSL} must be @w{version 1.4} or later. 
@acronym{NCO}, in particular @command{ncap2}, uses the @acronym{GSL}
special function library to evaluate geoscience-relevant mathematics
such as Bessel functions, Legendre polynomials, and incomplete gamma
functions (@pxref{GSL special functions}).
 
@cindex @var{gamma}
In June 2005, @acronym{NCO} version 3.0.1 began to take advantage
of C99 mathematical special functions.
These include the standarized gamma function (called @code{tgamma()} 
for ``true gamma''). 
@cindex automagic
@acronym{NCO} automagically takes advantage of some @acronym{GNU}
Compiler Collection (@acronym{GCC}) extensions to @w{@acronym{ANSI} C}.

As of July 2000 and @acronym{NCO} @w{version 1.2}, @acronym{NCO} no
longer performs arithmetic operations in Fortran.
We decided to sacrifice executable speed for code maintainability.
Since no objective statistics were ever performed to quantify 
the difference in speed between the Fortran and @w{C code},
the performance penalty incurred by this decision is unknown.
Supporting Fortran involves maintaining two sets of routines for every
arithmetic operation. 
The @code{USE_FORTRAN_ARITHMETIC} flag is still retained in the
@file{Makefile}.
The file containing the Fortran code, @file{nco_fortran.F}, has been
deprecated but a volunteer (@w{Dr.@: Frankenstein}?) could resurrect it.
If you would like to volunteer to maintain @file{nco_fortran.F} please 
contact me. 

@c Following section is obsolete
@ignore
It is still possible to request Fortran routines to perform arithmetic
operations, however.
@cindex preprocessor tokens
@cindex @code{USE_FORTRAN_ARITHMETIC}
This can be accomplished by defining the preprocessor token
@code{USE_FORTRAN_ARITHMETIC} and rebuilding @acronym{NCO}.
@cindex performance
As its name suggests, the @code{USE_FORTRAN_ARITHMETIC} token instructs
@acronym{NCO} to attempts to interface the @w{C routines} with Fortran
arithmetic. 
Although using Fortran calls instead @w{of C} reduces the portability and
and increases the maintenance of the @acronym{NCO} operators, it may
also increase the performance of the numeric operators.
Presumably this will depend on your machine type, the quality of @w{the C}
and Fortran compilers, and the size of the data files
@footnote{If you decide to test the efficiency of the averagers compiled
with @code{USE_FORTRAN_ARITHMETIC} versus the default @w{C averagers} I
would be most interested to hear the results.
Please E-mail me the results including the size of the datasets, the
platform, and the change in the wallclock time for execution.}.
@end ignore

@menu
* Windows Operating System::
@end menu

@html
<a name="wnd"></a> <!-- http://nco.sf.net/nco.html#wnd -->
<a name="windows"></a> <!-- http://nco.sf.net/nco.html#windows -->
<a name="qt"></a> <!-- http://nco.sf.net/nco.html#qt -->
@end html
@node Windows Operating System,  , Compatability, Compatability
@subsection Compiling @acronym{NCO} for Microsoft Windows @acronym{OS}
@cindex Windows
@cindex Microsoft
@cindex @acronym{XP} (Microsoft operating system)
@cindex @acronym{NT} (Microsoft operating system)
@cindex Vista (Microsoft operating system)
@cindex @acronym{MVS}
@cindex Microsoft Visual Studio

@acronym{NCO} has been successfully ported and tested on most Microsoft  
Windows operating systems including: @acronym{XP} SP2/Vista/7/10.
Support is provided for compiling either native Windows executables,
using the Microsoft Visual Studio Compiler (@acronym{MVSC}), or with
Cygwin, the @acronym{UNIX}-emulating compatibility layer with the
@acronym{GNU} toolchain. 
The switches necessary to accomplish both are included in the standard
distribution of @acronym{NCO}.

@cindex C99
@cindex CMake
@html
<a name="CMake"></a> <!-- http://nco.sf.net/nco.html#CMake -->
<a name="cmake"></a> <!-- http://nco.sf.net/nco.html#cmake -->
<a name="cmk"></a> <!-- http://nco.sf.net/nco.html#cmk -->
@end html
With Microsoft Visual Studio compiler, one must build @acronym{NCO}
with C++ since @acronym{MVSC} does not support C99.
Support for Qt, a convenient integrated development environment, was
deprecated in 2017.
As of @acronym{NCO} version 4.6.9 (September, 2017) please build native
Windows executables with CMake:
@example
@verbatim
cd ~/nco/cmake
cmake .. -DCMAKE_INSTALL_PREFIX=${HOME}
make install
@end verbatim
@end example
The file @file{nco/cmake/build.bat} shows how deal with various path
issues. 

@cindex Anaconda
@cindex Conda
@html
<a name="anaconda"></a> <!-- http://nco.sf.net/nco.html#anaconda -->
<a name="cnd"></a> <!-- http://nco.sf.net/nco.html#cnd -->
<a name="conda"></a> <!-- http://nco.sf.net/nco.html#conda -->
@end html
As of @acronym{NCO} version 4.7.1 (December, 2017) the Conda package
for @acronym{NCO} is available from the @code{conda-forge} channel on
all three smithies: Linux, MacOS, and Windows. 
@example
@verbatim
# Recommended install with Conda
conda config --add channels conda-forge # Permananently add conda-forge
conda install nco
# Or, specify conda-forge explicitly as a one-off:
conda install -c conda-forge nco
@end verbatim
@end example

Using the freely available Cygwin (formerly gnu-win32) development
environment  
@footnote{The Cygwin package is available from@*
@code{http://sourceware.redhat.com/cygwin}@*
@cindex @code{gcc}
@cindex @code{g++}
Currently, @w{Cygwin 20.x} comes with the @acronym{GNU} C/C++
compilers (@command{gcc}, @command{g++}.
These @acronym{GNU} compilers may be used to build the netCDF
distribution itself.}, the compilation process is very similar to
installing @acronym{NCO} on a @acronym{UNIX} system.  
@cindex preprocessor tokens
@cindex Cygwin
@cindex @code{gnu-win32}
@cindex @code{WIN32}
@cindex @file{GNUmakefile}
@cindex @file{Makefile}
@cindex @code{f90}
Set the @code{PVM_ARCH} preprocessor token to @code{WIN32}.  
Note that defining @code{WIN32} has the side effect of disabling
Internet features of @acronym{NCO} (see below). 
@acronym{NCO} should now build like it does on @acronym{UNIX}.

@cindex @acronym{UNIX}
@cindex @code{getuid}
@cindex @code{gethostname}
@findex @file{<arpa/nameser.h>}
@findex @file{<resolv.h>}
The least portable section of the code is the use of standard
@acronym{UNIX} and Internet protocols (e.g., @code{ftp}, @code{rcp},
@code{scp}, @code{sftp}, @code{getuid}, @code{gethostname}, and header
files @file{<arpa/nameser.h>} and 
@file{<resolv.h>}). 
@cindex @code{ftp}
@cindex @code{sftp}
@cindex @code{rcp}
@cindex @code{scp}
@cindex @acronym{SSH}
@cindex remote files
Fortunately, these @acronym{UNIX}-y calls are only invoked by the single  
@acronym{NCO} subroutine which is responsible for retrieving files
stored on remote systems (@pxref{Remote storage}).
In order to support @acronym{NCO} on the Microsoft Windows platforms,
this single feature was disabled (on Windows @acronym{OS} only).
This was required by @w{Cygwin 18.x}---newer versions of Cygwin may
support these protocols (let me know if this is the case).
The @acronym{NCO} operators should behave identically on Windows and
@acronym{UNIX} platforms in all other respects.

@html
<a name="sym"></a> <!-- http://nco.sf.net/nco.html#sym -->
@end html
@node Symbolic Links, Libraries, Compatability, Introduction
@section Symbolic Links
@cindex symbolic links
@acronym{NCO} relies on a common set of underlying algorithms.
To minimize duplication of source code, multiple operators sometimes
share the same underlying source.
This is accomplished by symbolic links from a single underlying
executable program to one or more invoked executable names.
For example, @command{nces} and @command{ncrcat} are symbolically linked  
to the @command{ncra} executable.
The @command{ncra} executable behaves slightly differently based on its
invocation name (i.e., @samp{argv[0]}), which can be 
@command{nces}, @command{ncra}, or @command{ncrcat}.
Logically, these are three different operators that happen to share 
the same executable.

@cindex Cygwin
@cindex synonym
@cindex pseudonym
@cindex @code{--pseudonym}
For historical reasons, and to be more user friendly, multiple synonyms 
(or pseudonyms) may refer to the same operator invoked with different
switches. 
For example, @command{ncdiff} is the same as @command{ncbo} and
@command{ncpack} is the same as @command{ncpdq}.
We implement the symbolic links and synonyms by the executing the
following @acronym{UNIX} commands in the directory where the
@acronym{NCO} executables are installed.
@example
ln -s -f ncbo ncdiff    # ncbo --op_typ='-'
ln -s -f ncra nces      # ncra --pseudonym='nces'
ln -s -f ncra ncrcat    # ncra --pseudonym='ncrcat'
ln -s -f ncbo ncadd     # ncbo --op_typ='+'
ln -s -f ncbo ncsubtract # ncbo --op_typ='-'
ln -s -f ncbo ncmultiply # ncbo --op_typ='*'
ln -s -f ncbo ncdivide   # ncbo --op_typ='/'
ln -s -f ncpdq ncpack    # ncpdq
ln -s -f ncpdq ncunpack  # ncpdq --unpack
# NB: Windows/Cygwin executable/link names have '.exe' suffix, e.g.,
ln -s -f ncbo.exe ncdiff.exe
...
@end example
The imputed command called by the link is given after the comment.
As can be seen, some these links impute the passing of a command line
argument to further modify the behavior of the underlying executable.
For example, @command{ncdivide} is a pseudonym for
@command{ncbo --op_typ='/'}.

@html
<a name="lbr"></a> <!-- http://nco.sf.net/nco.html#lbr -->
@end html
@node Libraries, netCDF2/3/4 and HDF4/5 Support, Symbolic Links, Introduction
@section Libraries
@cindex libraries
@cindex @code{LD_LIBRARY_PATH}
@cindex dynamic linking
@cindex static linking
Like all executables, the @acronym{NCO} operators can be built using dynamic
linking. 
@cindex performance
@cindex operator speed
@cindex speed
@cindex execution time
This reduces the size of the executable and can result in significant
performance enhancements on multiuser systems.
Unfortunately, if your library search path (usually the
@env{LD_LIBRARY_PATH} environment variable) is not set correctly, or if
the system libraries have been moved, renamed, or deleted since
@acronym{NCO} was installed, it is possible @acronym{NCO} operators
will fail with a message that they cannot find a dynamically loaded (aka
@dfn{shared object} or @samp{.so}) library. 
This will produce a distinctive error message, such as
@samp{ld.so.1:@- /usr/local/bin/nces:@- fatal:@- libsunmath.@-so.1:@- can't
open@- file:@- errno@-=2}.   
If you received an error message like this, ask your system 
administrator to diagnose whether the library is truly missing
@footnote{The @command{ldd} command, if it is available on your system,
will tell you where the executable is looking for each dynamically
loaded library. Use, e.g., @code{ldd `which nces`}.}, or whether you
simply need to alter your library search path.
As a final remedy, you may re-compile and install @acronym{NCO} with all
operators statically linked.  

@node netCDF2/3/4 and HDF4/5 Support, Help Requests and Bug Reports, Libraries, Introduction
@section netCDF2/3/4 and HDF4/5 Support
@cindex netCDF2
@cindex netCDF3
netCDF @w{version 2} was released in 1993.
@acronym{NCO} (specifically @command{ncks}) began soon after this in 1994.  
@w{netCDF 3.0} was released in 1996, and we were not exactly eager to 
convert all code to the newer, less tested netCDF implementation.
One @w{netCDF3} interface call (@code{nc_inq_libvers}) was added to 
@acronym{NCO} in January, 1998, to aid in maintainance and debugging. 
In March, 2001, the final @acronym{NCO} conversion to @w{netCDF3} 
was completed (coincidentally on the same day @w{netCDF 3.5} was
released). 
@acronym{NCO} @w{versions 2.0} and higher are built with the
@code{-DNO_NETCDF_2} flag to ensure no @w{netCDF2} interface calls   
are used.
@cindex @code{NO_NETCDF_2}

@cindex @acronym{HDF}
@cindex Hierarchical Data Format
@cindex Mike Folk
However, the ability to compile @acronym{NCO} with only @w{netCDF2}
calls is worth maintaining because @acronym{HDF} @w{version 4}, 
aka @acronym{HDF4} or simply @acronym{HDF}, 
@footnote{The Hierarchical Data Format, or @acronym{HDF}, is another
self-describing data format similar to, but more elaborate than,
netCDF. 
@acronym{HDF} comes in two flavors, @acronym{HDF4} and @acronym{HDF5}. 
Often people use the shorthand @acronym{HDF} to refer to the older
format @acronym{HDF4}.
People almost always use @acronym{HDF5} to refer to @acronym{HDF5}.} 
(available from @uref{http://hdfgroup.org, HDF})
supports only the @w{netCDF2} library calls
(see @uref{http://hdfgroup.org/UG41r3_html/SDS_SD.fm12.html#47784}).
There are two versions of @acronym{HDF}.
Currently @acronym{HDF} @w{version 4.x} supports the full @w{netCDF2}
@acronym{API} and thus @acronym{NCO} @w{version 1.2.x}. 
If @acronym{NCO} @w{version 1.2.x} (or earlier) is built with only
@w{netCDF2} calls then all @acronym{NCO} operators should work with 
@acronym{HDF4} files as well as netCDF files
@footnote{One must link the @acronym{NCO} code to the @acronym{HDF4}
@acronym{MFHDF} library instead of the usual netCDF library. 
Apparently @samp{MF} stands for Multi-file not for Mike Folk.
In any case, until about 2007 the @acronym{MFHDF} library only
supported @w{netCDF2} calls. 
Most people will never again install @acronym{NCO} 1.2.x and so will
never use @acronym{NCO} to write @acronym{HDF4} files.
It is simply too much trouble.}.
@cindex @code{NETCDF2_ONLY}
The preprocessor token @code{NETCDF2_ONLY} exists
in @acronym{NCO} @w{version 1.2.x} to eliminate all @w{netCDF3}
calls.  
Only versions of @acronym{NCO} numbered 1.2.x and earlier have this
capability. 

@cindex Unidata
@cindex @acronym{NCSA}
@cindex netCDF4
@cindex @acronym{HDF5}
@acronym{HDF} @w{version 5} became available in 1999, but did not
support netCDF (or, for that matter, Fortran) as of December 1999.
By early 2001, @acronym{HDF5} did support Fortran90.
Thanks to an @acronym{NSF}-funded ``harmonization'' partnership,
@acronym{HDF} began to fully support the @w{netCDF3} read interface
(which is employed by @w{@acronym{NCO} 2.x} and later). 
In 2004, Unidata and @acronym{THG} began a project to implement
the @acronym{HDF5} features necessary to support the netCDF API.
@acronym{NCO} version 3.0.3 added support for reading/writing
netCDF4-formatted @acronym{HDF5} files in October, 2005.
See @ref{File Formats and Conversion} for more details.

HDF support for netCDF was completed with HDF5 version 
@w{version 1.8} in 2007. 
The netCDF front-end that uses this @acronym{HDF5} back-end 
was completed and released soon after as netCDF @w{version 4}.
Download it from the
@uref{http://my.unidata.ucar.edu/content/software/netcdf/netcdf-4,netCDF4}
website. 

@html
<a name="nco4"></a> <!-- http://nco.sf.net/nco.html#nco4 -->
@end html
@acronym{NCO} version 3.9.0, released in May, 2007, added support for
all netCDF4 atomic data types except @code{NC_STRING}.
Support for @code{NC_STRING}, including ragged arrays of strings,
was finally added in version 3.9.9, released in June, 2009.
Support for additional netCDF4 features has been incremental.
We add one netCDF4 feature at a time.
You must build @acronym{NCO} with netCDF4 to obtain this support.

@cindex @code{NC_UBYTE}
@cindex @code{NC_USHORT}
@cindex @code{NC_UINT}
@cindex @code{NC_INT64}
@cindex @code{NC_UINT64}
@acronym{NCO} supports many netCDF4 features including atomic data
types, Lempel-Ziv compression (deflation), chunking, and groups.  
The new atomic data types are @code{NC_UBYTE}, @code{NC_USHORT}, 
@code{NC_UINT}, @code{NC_INT64}, and @code{NC_UINT64}.
Eight-byte integer support is an especially useful improvement from
netCDF3. 
All @acronym{NCO} operators support these types, e.g., @command{ncks}
copies and prints them, @command{ncra} averages them, and
@command{ncap2} processes algebraic scripts with them.
@command{ncks} prints compression information, if any, to screen.

@cindex deflation
@acronym{NCO} version 3.9.1 (June, 2007) added support for netCDF4 
Lempel-Ziv deflation.
Lempel-Ziv deflation is a lossless compression technique.
See @ref{Deflation} for more details.

@cindex chunking
@acronym{NCO} version 3.9.9 (June, 2009) added support for netCDF4
chunking in @command{ncks} and @command{ncecat}.
@acronym{NCO} version 4.0.4 (September, 2010) completed support for
netCDF4 chunking in the remaining operators.
See @ref{Chunking} for more details.

@cindex groups
@acronym{NCO} version 4.2.2 (October, 2012) added support for netCDF4
groups in @command{ncks} and @command{ncecat}.
Group support for these operators was complete (e.g., regular
expressions to select groups and Group Path Editing) as of 
@acronym{NCO} version 4.2.6 (March, 2013).
See @ref{Group Path Editing} for more details.
Group support for all other operators was finished in the
@acronym{NCO} version 4.3.x series completed in December, 2013.

@cindex broadcasting groups
Support for netCDF4 in the first arithmetic operator, @command{ncbo},
was introduced in @acronym{NCO} version 4.3.0 (March, 2013).
@acronym{NCO} version 4.3.1 (May, 2013) completed this support and
introduced the first example of automatic group broadcasting.
See @ref{ncbo netCDF Binary Operator} for more details.

@cindex @acronym{HDF5}
@cindex @code{-4}
@cindex @code{-3}
netCDF4-enabled @acronym{NCO} handles netCDF3 files without change.
In addition, it automagically handles netCDF4 (@acronym{HDF5}) files:
If you feed @acronym{NCO} netCDF3 files, it produces netCDF3 output.
If you feed @acronym{NCO} netCDF4 files, it produces netCDF4 output.
Use the handy-dandy @samp{-4} switch to request netCDF4 output from 
netCDF3 input, i.e., to convert netCDF3 to netCDF4.
See @ref{File Formats and Conversion} for more details.

@html
<a name="hdf4"></a> <!-- http://nco.sf.net/nco.html#hdf4 -->
<a name="HDF4"></a> <!-- http://nco.sf.net/nco.html#HDF4 -->
@end html
@cindex @acronym{HDF4}
@cindex @samp{--hdf4}
When linked to a netCDF library that was built with @acronym{HDF4}
support
@footnote{The procedure for doing this is documented at
@uref{http://www.unidata.ucar.edu/software/netcdf/docs/build_hdf4.html}.},
@acronym{NCO} automatically supports reading @acronym{HDF4} 
files and writing them as netCDF3/netCDF4/@acronym{HDF5} files.
@acronym{NCO} can only write through the netCDF @acronym{API}, which
can only write netCDF3/netCDF4/@acronym{HDF5} files. 
So @acronym{NCO} can @emph{read} @acronym{HDF4} files, perform
manipulations and calculations, and then it must @emph{write} the
results in netCDF format. 

@acronym{NCO} support for @acronym{HDF4} has been quite functional since
December, 2013.
For best results install @acronym{NCO} versions 4.4.0 or later on top of 
netCDF versions 4.3.1 or later. 
Getting to this point has been an iterative effort where Unidata
improved netCDF library capabilities in response to our requests.
@acronym{NCO} versions 4.3.6 and earlier do not explicitly support
@acronym{HDF4}, yet should work with @acronym{HDF4} if compiled with 
a version of netCDF (4.3.2 or later?) that does not unexpectedly die
when probing @acronym{HDF4} files with standard netCDF calls.
@acronym{NCO} versions 4.3.7--4.3.9 (October--December, 2013)
use a special flag to circumvent netCDF @acronym{HDF4} issues.
The user must tell these versions of @acronym{NCO} that an input file is 
@acronym{HDF4} format by using the @samp{--hdf4} switch. 

@cindex @code{HDF4_UNKNOWN}
When compiled with netCDF version 4.3.1 (20140116) or later, 
@acronym{NCO} versions 4.4.0 (January, 2014) and later more gracefully 
handle @acronym{HDF4} files.
In particular, the @samp{--hdf4} switch is obsolete. 
Current versions of @acronym{NCO} use netCDF to determine automatically
whether the underlying file is @acronym{HDF4}, and then take appropriate
precautions to avoid netCDF4 @acronym{API} calls that fail when applied
to @acronym{HDF4} files (e.g., @code{nc_inq_var_chunking()},
@code{nc_inq_var_deflate()}).  
When compiled with netCDF version 4.3.2 (20140423) or earlier,
@acronym{NCO} will report that chunking and deflation properties of
@acronym{HDF4} files as @code{HDF4_UNKNOWN}, because determining 
those properties was impossible.
When compiled with netCDF version 4.3.3-rc2 (20140925) or later, 
@acronym{NCO} versions 4.4.6 (October, 2014) and later fully support  
chunking and deflation features of @acronym{HDF4} files.
Unfortunately, netCDF version 4.7.4 (20200327) introduced a regression 
that breaks this functionality for all @acronym{NCO} versions until
we first noticed the regression a year later and implemented a
workaround to restore this functionality as of 4.9.9-alpha02
(20210327). 
The @samp{--hdf4} switch is supported (for backwards compatibility) yet
redundant (i.e., does no harm) with current versions of @acronym{NCO}
and netCDF. 

Converting @acronym{HDF4} files to netCDF:
Since @acronym{NCO} reads @acronym{HDF4} files natively, it is now easy  
to convert @acronym{HDF4} files to netCDF files directly, e.g.,
@example
ncks        fl.hdf fl.nc # Convert HDF4->netCDF4 (NCO 4.4.0+, netCDF 4.3.1+)
ncks --hdf4 fl.hdf fl.nc # Convert HDF4->netCDF4 (NCO 4.3.7-4.3.9)
@end example
The most efficient and accurate way to convert @acronym{HDF4} data to
netCDF format is to convert to netCDF4 using @acronym{NCO} as above.
Many @acronym{HDF4} producers (@acronym{NASA}!) love to use netCDF4
types, e.g., unsigned bytes, so this procedure is the most typical.
Conversion of @acronym{HDF4} to netCDF4 as above suffices when the data
will only be processed by @acronym{NCO} and other netCDF4-aware tools.  

@cindex @command{ncl_convert2nc}
@cindex @command{nc3tonc4}
However, many tools are not fully netCDF4-aware, and so conversion to
netCDF3 may be desirable.
Obtaining any netCDF file from an @acronym{HDF4} is easy:
@example
ncks -3 fl.hdf fl.nc      # HDF4->netCDF3 (NCO 4.4.0+, netCDF 4.3.1+)
ncks -4 fl.hdf fl.nc      # HDF4->netCDF4 (NCO 4.4.0+, netCDF 4.3.1+)
ncks -6 fl.hdf fl.nc      # HDF4->netCDF3 64-bit  (NCO 4.4.0+, ...)
ncks -7 -L 1 fl.hdf fl.nc # HDF4->netCDF4 classic (NCO 4.4.0+, ...)
ncks --hdf4 -3 fl.hdf fl.nc # HDF4->netCDF3 (netCDF 4.3.0-)
ncks --hdf4 -4 fl.hdf fl.nc # HDF4->netCDF4 (netCDF 4.3.0-)
ncks --hdf4 -6 fl.hdf fl.nc # HDF4->netCDF3 64-bit  (netCDF 4.3.0-)
ncks --hdf4 -7 fl.hdf fl.nc # HDF4->netCDF4 classic (netCDF 4.3.0-)
@end example
As of @acronym{NCO} version 4.4.0 (January, 2014), these commands work
even when the @acronym{HDF4} file contains netCDF4 atomic types (e.g.,
unsigned bytes, 64-bit integers) because @acronym{NCO} can autoconvert
everything to atomic types supported by netCDF3
@footnote{
Prior to @acronym{NCO} version 4.4.0 (January, 2014), we recommended the 
@command{ncl_convert2nc} tool to convert @acronym{HDF} to netCDF3 when
both these are true: @w{1. You} must have netCDF3 and @w{2. the}
@acronym{HDF} file contains netCDF4 atomic types. 
More recent versions of @acronym{NCO} handle this problem fine, and
include other advantages so we no longer recommend
@command{ncl_convert2nc} because @command{ncks} is faster and more
space-efficient. 
Both automatically convert netCDF4 types to netCDF3 types, yet
@command{ncl_convert2nc} cannot produce full netCDF4 files.
In contrast, @command{ncks} will happily convert @acronym{HDF} straight
to netCDF4 files with netCDF4 types. 
Hence @command{ncks} can and does preserve the variable types.
Unsigned bytes stay unsigned bytes. 
64-bit integers stay 64-bit integers. 
Strings stay strings. 
Hence, @command{ncks} conversions often result in smaller files than
@command{ncl_convert2nc} conversions.
Another tool useful for converting netCDF3 to netCDF4 files, and whose
functionality is, we think, also matched or exceeded by @command{ncks},
is the Python script @command{nc3tonc4} by Jeff Whitaker.}. 

@cindex @code{hdf_name}
@cindex illegal names
As of @acronym{NCO} version 4.4.4 (May, 2014) both
@command{ncl_convert2nc} and @acronym{NCO} have built-in, automatic
workarounds to handle element names that contain characters that are
legal in @acronym{HDF} though are illegal in @acronym{netCDF}. 
For example, slashes and leading special characters are are legal in
@acronym{HDF} and illegal in @acronym{netCDF} element (i.e., group,
variable, dimension, and attribute) names.
@acronym{NCO} converts these forbidden characters to underscores, and
retains the original names of variables in automatically produced
attributes named @code{hdf_name}
@footnote{Two real-world examples: @acronym{NCO} translates the
@acronym{NASA} @acronym{CERES} dimension @code{(FOV) Footprints} to
@code{_FOV_ Footprints}, and 
@code{Cloud & Aerosol, Cloud Only, Clear Sky w/Aerosol, and Clear Sky} 
(yes, the dimension name includes whitespace and special characters) to 
@code{Cloud & Aerosol, Cloud Only, Clear Sky w_Aerosol, and Clear Sky}
@command{ncl_convert2nc} makes the element name netCDF-safe in a
slightly different manner, and also stores the original name in the
@code{hdf_name} attribute.}.

@cindex @acronym{H4CF}
@cindex @command{h4tonccf}
Finally, in February 2014, we learned that the @acronym{HDF} group
has a project called @acronym{H4CF} 
(described @uref{http://hdfeos.org/software/h4cflib.php, here})
whose goal is to make @acronym{HDF4} files accessible to @acronym{CF}
tools and conventions. 
Their project includes a tool named @command{h4tonccf} that converts
@acronym{HDF4} files to netCDF3 or netCDF4 files.
We are not yet sure what advantages or features @command{h4tonccf} has
that are not in @acronym{NCO}, though we suspect both methods have their
own advantages. Corrections welcome.

@cindex @acronym{RPM}
@cindex Debian
As of 2012, netCDF4 is relatively stable software.
Problems with netCDF4 and @acronym{HDF} libraries have mainly been fixed.
Binary @acronym{NCO} distributions shipped as @acronym{RPM}s and as debs
have used the netCDF4 library since 2010 and 2011, respectively.

@cindex @code{NETCDF4_ROOT}
One must often build @acronym{NCO} from source to obtain netCDF4
support. 
Typically, one specifies the root of the netCDF4
installation directory. Do this with the @code{NETCDF4_ROOT} variable.
Then use your preferred @acronym{NCO} build mechanism, e.g.,
@example
export NETCDF4_ROOT=/usr/local/netcdf4 # Set netCDF4 location
cd ~/nco;./configure --enable-netcdf4  # Configure mechanism -or-
cd ~/nco/bld;./make NETCDF4=Y allinone # Old Makefile mechanism
@end example

We carefully track the netCDF4 releases, and keep the netCDF4 atomic
type support and other features working.
Our long term goal is to utilize more of the extensive new netCDF4
feature set. The next major netCDF4 feature we are likely to utilize
is parallel I/O. We will enable this in the @acronym{MPI} netCDF
operators. 

@html
<a name="help"></a> <!-- http://nco.sf.net/nco.html#help -->
<a name="hlp"></a> <!-- http://nco.sf.net/nco.html#hlp -->
<a name="bug"></a> <!-- http://nco.sf.net/nco.html#bug -->
@end html
@node Help Requests and Bug Reports,  , netCDF2/3/4 and HDF4/5 Support, Introduction
@section Help Requests and Bug Reports
@cindex reporting bugs
@cindex bugs, reporting
@cindex core dump
@cindex help
@cindex features, requesting
We generally receive three categories of mail from users: help requests,
bug reports, and feature requests.
Notes saying the equivalent of ``Hey, @acronym{NCO} continues to work
great and it saves me more time everyday than it took to write this
note'' are a distant fourth.

There is a different protocol for each type of request.
The preferred etiquette for all communications is via @acronym{NCO}
Project Forums. 
Do not contact project members via personal e-mail unless your request
comes with money or you have damaging information about our personal
lives.
@emph{Please use the Forums}---they preserve a record of the questions
and answers so that others can learn from our exchange.
Also, since @acronym{NCO} is both volunteer-driven and
government-funded, this record helps us provide program officers with 
information they need to evaluate our project. 

Before posting to the @acronym{NCO} forums described below, you might
first @uref{https://sf.net/account/register.php, register}
your name and email address with SourceForge.net or else all of your
postings will be attributed to @emph{nobody}.
Once registered you may choose to @emph{monitor} any forum and to receive
(or not) email when there are any postings including responses to your
questions.
We usually reply to the forum message, not to the original poster.

If you want us to include a new feature in @acronym{NCO}, please 
consider implementing the feature yourself and sending us the patch.
If that is beyond your ken, then send a note to the
@uref{http://sf.net/p/nco/discussion/9829, NCO Discussion forum}.

Read the manual before reporting a bug or posting a help request.
Sending questions whose answers are not in the manual is the best
way to motivate us to write more documentation.  
We would also like to accentuate the contrapositive of this statement.  
If you think you have found a real bug @emph{the most helpful thing you 
can do is simplify the problem to a manageable size and then report it}.
The first thing to do is to make sure you are running the latest
publicly released version of @acronym{NCO}.  

Once you have read the manual, if you are still unable to get
@acronym{NCO} to perform a documented function, submit a help request.
Follow the same procedure as described below for reporting bugs
(after all, it might be a bug).
@cindex debugging
@cindex @code{-r}
@cindex @code{-D}
That is, describe what you are trying to do, and include the complete
commands (run with @samp{-D 5}), error messages, and version of
@acronym{NCO} (with @samp{-r}).  
Some commands behave differently depending on the exact order and
rank of dimensions in the pertinent variables.
In such cases we need you to provide that metadata, e.g., the text
results of @samp{ncks -m} on your input and/or output files.
Post your help request to the 
@uref{http://sf.net/p/nco/discussion/9830, NCO Help forum}.

If you think you used the right command when @acronym{NCO} misbehaves,
then you might have found a bug.  
Incorrect numerical answers are the highest priority.
We usually fix those within one or two days.
Core dumps and sementation violations receive lower priority.
They are always fixed, eventually. 

How do you simplify a problem that reveal a bug?
Cut out extraneous variables, dimensions, and metadata from the
offending files and re-run the command until it no longer breaks.  
Then back up one step and report the problem.
Usually the file(s) will be very small, i.e., one variable with one or
two small dimensions ought to suffice.
@html
<a name="dbg"></a> <!-- http://nco.sf.net/nco.html#dbg -->
<a name="-D"></a> <!-- http://nco.sf.net/nco.html#-D -->
@end html
@cindex @code{-r}
@cindex @code{--revision}
@cindex @code{--version}
@cindex @code{--vrs}
@cindex @code{-D @var{debug-level}}
@cindex @code{--debug-level @var{debug-level}}
@cindex @code{--dbg_lvl @var{debug-level}}
@cindex @var{debug-level}
@cindex @var{dbg_lvl}
Run the operator with @samp{-r} and then run the command with 
@samp{-D 5} to increase the verbosity of the debugging output.
It is very important that your report contain the exact error messages 
and compile-time environment.
Include a copy of your sample input file, or place one on a 
publicly accessible location, of the file(s).
If you are sure it is a bug, post the full report to the 
@uref{http://sf.net/p/nco/bugs, NCO Project buglist}.
Otherwise post all the information to 
@uref{http://sf.net/p/nco/discussion/9830, NCO Help forum}.

@cindex installation
@cindex @command{autoconf}
@cindex @file{nco.configure.$@{GNU_TRP@}.foo}
@cindex @file{nco.config.log.$@{GNU_TRP@}.foo}
@cindex @file{nco.make.$@{GNU_TRP@}.foo}
@cindex @file{config.guess}
@cindex @file{configure.eg}
Build failures count as bugs.
Our limited machine access means we cannot fix all build failures.
The information we need to diagnose, and often fix, build failures
are the three files output by @acronym{GNU} build tools,  
@file{nco.config.log.$@{GNU_TRP@}.foo},
@file{nco.configure.$@{GNU_TRP@}.foo}, 
and @file{nco.make.$@{GNU_TRP@}.foo}.
The file @file{configure.eg} shows how to produce these files.
Here @code{$@{GNU_TRP@}} is the ``@acronym{GNU} architecture triplet'',
the @var{chip-vendor-OS} string returned by @file{config.guess}.
Please send us your improvements to the examples supplied in
@file{configure.eg}.
@cindex regressions archive
The regressions archive at @url{http://dust.ess.uci.edu/nco/rgr}
contains the build output from our standard test systems.
You may find you can solve the build problem yourself by examining the
differences between these files and your own.

@html
<a name="str"></a> <!-- http://nco.sf.net/nco.html#str -->
@end html
@node Strategies, Shared features, Introduction, Top
@chapter Operator Strategies

@menu
* Philosophy::
* Climate Model Paradigm::
* Temporary Output Files::
* Appending Variables::
* Simple Arithmetic and Interpolation::
* Statistics vs Concatenation::
* Large Numbers of Files::
* Large Datasets::
* Memory Requirements::
* Performance::
@end menu

@html
<a name="phl"></a> <!-- http://nco.sf.net/nco.html#phl -->
@end html
@node Philosophy, Climate Model Paradigm, Strategies, Strategies
@section Philosophy
@cindex philosophy
@cindex climate model

The main design goal is command line operators which perform useful,
scriptable operations on netCDF files.  
Many scientists work with models and observations which produce too much
data to analyze in tabular format.
Thus, it is often natural to reduce and massage this raw or primary
level data into summary, or second level data, e.g., temporal or spatial
averages. 
These second level data may become the inputs to graphical and
statistical packages, and are often more suitable for archival and
dissemination to the scientific community.
@acronym{NCO} performs a suite of operations useful in manipulating data
from the primary to the second level state.
@cindex @acronym{IDL}
@cindex Matlab
@cindex @acronym{NCL}
@cindex Perl
@cindex Yorick
Higher level interpretive languages (e.g., @acronym{IDL}, Yorick,
Matlab, @acronym{NCL}, Perl, Python),
and lower level compiled languages (e.g., C, Fortran) can always perform  
any task performed by @acronym{NCO}, but often with more overhead.
NCO, on the other hand, is limited to a much smaller set of arithmetic
and metadata operations than these full blown languages.

@cindex command line switches
Another goal has been to implement enough command line switches so that 
frequently used sequences of these operators can be executed from a
shell script or batch file.
Finally, @acronym{NCO} was written to consume the absolute minimum
amount of system memory required to perform a given job.
The arithmetic operators are extremely efficient; their exact memory
usage is detailed in @ref{Memory Requirements}.

@html
<a name="clm_mdl"></a> <!-- http://nco.sf.net/nco.html#clm_mdl -->
@end html
@node Climate Model Paradigm, Temporary Output Files, Philosophy, Strategies
@section Climate Model Paradigm
@cindex climate model
@cindex @acronym{NCAR}
@cindex @acronym{GCM}

@acronym{NCO} was developed at @acronym{NCAR} to aid analysis and
manipulation of datasets produced by General Circulation Models
(@acronym{GCM}s).  
@acronym{GCM} datasets share many features with other gridded scientific
datasets and so provide a useful paradigm for the explication of the
@acronym{NCO} operator set. 
Examples in this manual use a @acronym{GCM} paradigm because latitude,
longitude, time, temperature and other fields related to our natural
environment are as easy to visualize for the layman as the expert.

@html
<a name="out"></a> <!-- http://nco.sf.net/nco.html#out -->
@end html
@node Temporary Output Files, Appending Variables, Climate Model Paradigm, Strategies
@section Temporary Output Files 
@cindex data safety
@cindex error tolerance
@cindex safeguards
@cindex temporary output files
@cindex temporary files
@acronym{NCO} operators are designed to be reasonably fault tolerant, so
that a system failure or user-abort of the operation (e.g., with
@kbd{C-c}) does not cause loss of data.
The user-specified @var{output-file} is only created upon successful
completion of the operation  
@footnote{The @command{ncrename} and @command{ncatted} operators are
exceptions to this rule.
@xref{ncrename netCDF Renamer}.}.
This is accomplished by performing all operations in a temporary copy
of @var{output-file}.
The name of the temporary output file is constructed by appending
@code{.pid@var{<process ID>}.@var{<operator name>}.tmp} to the
user-specified @var{output-file} name.  
When the operator completes its task with no fatal errors, the temporary
output file is moved to the user-specified @var{output-file}.
This imbues the process with fault-tolerance since fatal error
(e.g., disk space fills up) affect only the temporary output file,
leaving the final output file not created if it did not already exist. 
Note the construction of a temporary output file uses more disk space
than just overwriting existing files ``in place'' (because there may be
two copies of the same file on disk until the @acronym{NCO} operation
successfully concludes and the temporary output file overwrites the
existing @var{output-file}).  
@cindex performance
@cindex operator speed
@cindex speed
@cindex execution time
Also, note this feature increases the execution time of the operator
by approximately the time it takes to copy the @var{output-file}
@footnote{The OS-specific system move command is used.
This is @command{mv} for @acronym{UNIX}, and @command{move} for Windows.}.
Finally, note this fault-tolerant feature allows the @var{output-file}
to be the same as the @var{input-file} without any danger of
``overlap''. 

@html
<a name="tmp_fl"></a> <!-- http://nco.sf.net/nco.html#tmp_fl -->
<a name="no_tmp_fl"></a> <!-- http://nco.sf.net/nco.html#no_tmp_fl -->
<a name="wrt_tmp_fl"></a> <!-- http://nco.sf.net/nco.html#wrt_tmp_fl -->
@end html
@cindex @code{--no_tmp_fl}
@cindex @code{--wrt_tmp_fl}
@cindex @code{--write_tmp_fl}
@cindex @code{--create_ram}
@cindex @code{--open_ram}
@cindex @acronym{RAM} disks
@cindex @acronym{RAM} files
Over time many ``power users'' have requested a way to turn-off the
fault-tolerance safety feature that automatically creates a temporary
file. 
Often these users build and execute production data analysis scripts
that are repeated frequently on large datasets.
Obviating an extra file write can then conserve significant disk space
and time.  
For this purpose @acronym{NCO} has, since version 4.2.1 in August, 2012, 
made configurable the controls over temporary file creation.
The @samp{--wrt_tmp_fl} and equivalent @samp{--write_tmp_fl} switches 
ensure @acronym{NCO} writes output to an intermediate temporary file.
This is and has always been the default behavior so there is currently
no need to specify these switches.
However, the default may change some day, especially since writing to
RAM disks (@pxref{RAM disks}) may some day become the default.
The @samp{--no_tmp_fl} switch causes @acronym{NCO} to write directly to
the final output file instead of to an intermediate temporary file. 
``Power users'' may wish to invoke this switch to increase performance
(i.e., reduce wallclock time) when manipulating large files. 
When eschewing temporary files, users may forsake the ability to have
the same name for both @var{output-file} and @var{input-file} since, as   
described above, the temporary file prevented overlap issues.
However, if the user creates the output file in @acronym{RAM}
(@pxref{RAM disks}) then it is still possible to have the same name
for both @var{output-file} and @var{input-file}.
@example
ncks in.nc out.nc # Default: create out.pid.tmp.nc then move to out.nc
ncks --wrt_tmp_fl in.nc out.nc # Same as default
ncks --no_tmp_fl in.nc out.nc # Create out.nc directly on disk
ncks --no_tmp_fl in.nc in.nc # ERROR-prone! Overwrite in.nc with itself
ncks --create_ram --no_tmp_fl in.nc in.nc # Create in RAM, write to disk
ncks --open_ram --no_tmp_fl in.nc in.nc # Read into RAM, write to disk
@end example
@noindent
There is no reason to expect the fourth example to work.
The behavior of overwriting a file while reading from the same file is
undefined, much as is the shell command @samp{cat foo > foo}.
Although it may ``work'' in some cases, it is unreliable.
One way around this is to use @samp{--create_ram} so that the
output file is not written to disk until the input file is closed,
@xref{RAM disks}.
However, as of 20130328, the behavior of the @samp{--create_ram} and
@samp{--open_ram} examples has not been thoroughly tested.

The @acronym{NCO} authors have seen compelling use cases for utilizing
the @acronym{RAM} switches, though not (yet) for combining them with
@samp{--no_tmp_fl}. 
@acronym{NCO} implements both options because they are largely
independent of eachother.
It is up to ``power users'' to discover which best fit their needs.
We welcome accounts of your experiences posted to the forums.

@html
<a name="-A"></a> <!-- http://nco.sf.net/nco.html#-A -->
<a name="-O"></a> <!-- http://nco.sf.net/nco.html#-O -->
@end html
@cindex @code{-A}
@cindex @code{-O}
@cindex @code{--apn}
@cindex @code{--append}
@cindex @code{--ovr}
@cindex @code{--overwrite}
@cindex overwriting files
@cindex appending variables
@cindex appending to files
Other safeguards exist to protect the user from inadvertently
overwriting data.
If the @var{output-file} specified for a command is a pre-existing file,
then the operator will prompt the user whether to overwrite (erase) the
existing @var{output-file}, attempt to append to it, or abort the
operation. 
However, in processing large amounts of data, too many interactive
questions slows productivity.
Therefore @acronym{NCO} also implements two ways to override its own
safety features, the @samp{-O} and @samp{-A} switches.
Specifying @samp{-O} tells the operator to overwrite any existing
@var{output-file} without prompting the user interactively.
Specifying @samp{-A} tells the operator to attempt to append to any
existing @var{output-file} without prompting the user interactively.
These switches are useful in batch environments because they suppress
interactive keyboard input.

@html
<a name="apn"></a> <!-- http://nco.sf.net/nco.html#apn -->
<a name="append"></a> <!-- http://nco.sf.net/nco.html#append -->
@end html
@node Appending Variables, Simple Arithmetic and Interpolation, Temporary Output Files, Strategies
@section Appending Variables
Adding variables from one file to another is often desirable.
@cindex concatenation
@cindex appending variables
@cindex merging files
@cindex pasting variables
This is referred to as @dfn{appending}, although some prefer the
terminology @dfn{merging} @footnote{The terminology @dfn{merging} is
reserved for an (unwritten) operator which replaces hyperslabs of a
variable in one file with hyperslabs of the same variable from another 
file} or @dfn{pasting}. 
Appending is often confused with what @acronym{NCO} calls
@dfn{concatenation}. 
@cindex record dimension
In @acronym{NCO}, concatenation refers to splicing a variable
along the record dimension.  
The length along the record dimension of the output is the sum of the
lengths of the input files. 
Appending, on the other hand, refers to copying a variable from one file
to another file which may or may not already contain the variable 
@footnote{Yes, the terminology is confusing. 
By all means mail me if you think of a better nomenclature.
Should @acronym{NCO} use @dfn{paste} instead of @dfn{append}?
}. 
@acronym{NCO} can append or concatenate just one variable, or all the
variables in a file at the same time.

In this sense, @command{ncks} can append variables from one file to
another file. 
This capability is invoked by naming two files on the command line,
@var{input-file} and @var{output-file}. 
When @var{output-file} already exists, the user is prompted whether to
@dfn{overwrite}, @dfn{append/replace}, or @dfn{exit} from the command.
Selecting @dfn{overwrite} tells the operator to erase the existing
@var{output-file} and replace it with the results of the operation.
Selecting @dfn{exit} causes the operator to exit---the @var{output-file}
will not be touched in this case.
Selecting @dfn{append/replace} causes the operator to attempt to place
the results of the operation in the existing @var{output-file}, 
@xref{ncks netCDF Kitchen Sink}.

@html
<a name="unn"></a> <!-- http://nco.sf.net/nco.html#unn -->
@end html
@cindex union of files
@cindex disjoint files
The simplest way to create the union of two files is
@example
ncks -A fl_1.nc fl_2.nc
@end example
This puts the contents of @file{fl_1.nc} into @file{fl_2.nc}. 
The @samp{-A} is optional. 
On output, @file{fl_2.nc} is the union of the input files,
regardless of whether they share dimensions and variables, 
or are completely disjoint.
The append fails if the input files have differently named record
dimensions (since netCDF supports only one), or have dimensions of the
same name but different sizes.

@html
<a name="bnr"></a> <!-- http://nco.sf.net/nco.html#bnr -->
@end html
@node Simple Arithmetic and Interpolation, Statistics vs Concatenation, Appending Variables, Strategies
@section Simple Arithmetic and Interpolation

Users comfortable with @acronym{NCO} semantics may find it easier to
perform some simple mathematical operations in @acronym{NCO} rather than  
higher level languages. 
@command{ncbo} (@pxref{ncbo netCDF Binary Operator}) does file
addition, subtraction, multiplication, division, and broadcasting. 
It even does group broadcasting.
@command{ncflint} (@pxref{ncflint netCDF File Interpolator}) does
file addition, subtraction, multiplication and interpolation. 
Sequences of these commands can accomplish simple yet powerful
operations from the command line.

@html
<a name="statisticians"></a> <!-- http://nco.sf.net/nco.html#statisticians -->
<a name="averagers"></a> <!-- http://nco.sf.net/nco.html#averagers -->
@end html
@node Statistics vs Concatenation, Large Numbers of Files, Simple Arithmetic and Interpolation, Strategies
@section Statistics vs Concatenation

@html
<a name="sym_ncea"></a> <!-- http://nco.sf.net/nco.html#sym_ncea -->
<a name="sym_nces"></a> <!-- http://nco.sf.net/nco.html#sym_nces -->
<a name="sym_ncrcat"></a> <!-- http://nco.sf.net/nco.html#sym_ncrcat -->
@end html
@cindex symbolic links
The most frequently used operators of @acronym{NCO} are probably the
@dfn{statisticians} (i.e., tools that do statistics) and concatenators.
Because there are so many types of statistics like averaging (e.g.,
across files, within a file, over the record dimension, over other
dimensions, with or without weights and masks) and of concatenating
(across files, along the record dimension, along other dimensions),
there are currently no fewer than five operators which tackle these two
purposes: @command{ncra}, @command{nces}, @command{ncwa},
@command{ncrcat}, and @command{ncecat}.  
These operators do share many capabilities @footnote{Currently
@command{nces} and @command{ncrcat} are symbolically linked to the
@command{ncra} executable, which behaves slightly differently based on
its invocation name (i.e., @samp{argv[0]}). 
These three operators share the same source code, and merely have
different inner loops.}, though each has its unique specialty.
Two of these operators, @command{ncrcat} and @command{ncecat}, 
concatenate hyperslabs across files. 
The other two operators, @command{ncra} and @command{nces}, compute
statistics across (and/or within) files 
@footnote{The third averaging operator, @command{ncwa}, is the most
sophisticated averager in @acronym{NCO}. 
However, @command{ncwa} is in a different class than @command{ncra} and
@command{nces} because it operates on a single file per invocation (as
opposed to multiple files).    
On that single file, however, @command{ncwa} provides a richer set of 
averaging options---including weighting, masking, and broadcasting.}.  
First, let's describe the concatenators, then the statistics tools.

@menu
* Concatenation::
* Averaging::
* Interpolating::
@end menu

@html
<a name="cnc"></a> <!-- http://nco.sf.net/nco.html#cnc -->
@end html
@node Concatenation, Averaging, Statistics vs Concatenation, Statistics vs Concatenation
@subsection Concatenators @command{ncrcat} and @command{ncecat}
@cindex @command{ncecat}
@cindex @command{ncrcat}

Joining together independent files along a common record dimension is
called @dfn{concatenation}.    
@command{ncrcat} is designed for concatenating record variables, while
@command{ncecat} is designed for concatenating fixed length variables.
Consider five files, @file{85.nc}, @file{86.nc}, 
@w{@dots{} @file{89.nc}} each containing a year's worth of data.  
Say you wish to create from them a single file, @file{8589.nc}
containing all the data, i.e., spanning all five years.
If the annual files make use of the same record variable, then
@command{ncrcat} will do the job nicely with, e.g., 
@code{ncrcat 8?.nc 8589.nc}. 
The number of records in the input files is arbitrary and can vary from
file to file. 
@xref{ncrcat netCDF Record Concatenator}, for a complete description of
@command{ncrcat}. 

However, suppose the annual files have no record variable, and thus
their data are all fixed length. 
@cindex ensemble
@cindex climate model
For example, the files may not be conceptually sequential, but rather
members of the same group, or @dfn{ensemble}. 
Members of an ensemble may have no reason to contain a record dimension.
@command{ncecat} will create a new record dimension (named @var{record}
by default) with which to glue together the individual files into the
single ensemble file.
If @command{ncecat} is used on files which contain an existing record
dimension, that record dimension is converted to a fixed-length
dimension of the same name and a new record dimension (named
@code{record}) is created.  
Consider five realizations, @file{85a.nc}, @file{85b.nc}, 
@w{@dots{} @file{85e.nc}} of 1985 predictions from the same climate
model. 
Then @code{ncecat 85?.nc 85_ens.nc} glues together the individual
realizations into the single file, @file{85_ens.nc}. 
If an input variable was dimensioned [@code{lat},@code{lon}], it will
have dimensions [@code{record},@code{lat},@code{lon}] in the output file.
@w{A restriction} of @command{ncecat} is that the hyperslabs of the
processed variables must be the same from file to file.
Normally this means all the input files are the same size, and contain 
data on different realizations of the same variables.
@xref{ncecat netCDF Ensemble Concatenator}, for a complete description
of @command{ncecat}. 

@cindex @command{ncpdq}
@html
<a name="dmn_cat"></a> <!-- http://nco.sf.net/nco.html#dmn_cat -->
@end html
@command{ncpdq} makes it possible to concatenate files along any
dimension, not just the record dimension.
First, use @command{ncpdq} to convert the dimension to be concatenated
(i.e., extended with data from other files) into the record dimension. 
Second, use @command{ncrcat} to concatenate these files.
Finally, if desirable, use @command{ncpdq} to revert to the original
dimensionality.
As a concrete example, say that files @file{x_01.nc}, @file{x_02.nc},
@w{@dots{} @file{x_10.nc}} contain time-evolving datasets from spatially
adjacent regions.
The time and spatial coordinates are @code{time} and @code{x}, respectively.
Initially the record dimension is @code{time}.
Our goal is to create a single file that contains joins all the
spatially adjacent regions into one single time-evolving dataset.
@example
@verbatim
for idx in 01 02 03 04 05 06 07 08 09 10; do # Bourne Shell
  ncpdq -a x,time x_${idx}.nc foo_${idx}.nc  # Make x record dimension
done
ncrcat foo_??.nc out.nc       # Concatenate along x
ncpdq -a time,x out.nc out.nc # Revert to time as record dimension
@end verbatim
@end example

Note that @command{ncrcat} will not concatenate fixed-length variables, 
whereas @command{ncecat} concatenates both fixed-length and record
variables along a new record variable.
To conserve system memory, use @command{ncrcat} where possible.

@html
<a name="avg"></a> <!-- http://nco.sf.net/nco.html#avg -->
@end html
@node Averaging, Interpolating, Concatenation, Statistics vs Concatenation
@subsection Averagers @command{nces}, @command{ncra}, and @command{ncwa} 
@cindex @command{nces}
@cindex @command{ncra}
@cindex @command{ncwa}

The differences between the averagers @command{ncra} and @command{nces}
are analogous to the differences between the concatenators.
@command{ncra} is designed for averaging record variables from at least
one file, while @command{nces} is designed for averaging fixed length
variables from multiple files.
@command{ncra} performs a simple arithmetic average over the record
dimension of all the input files, with each record having an equal
weight in the average. 
@command{nces} performs a simple arithmetic average of all the input
files, with each file having an equal weight in the average. 
Note that @command{ncra} cannot average fixed-length variables,
but @command{nces} can average both fixed-length and record variables.  
To conserve system memory, use @command{ncra} rather than
@command{nces} where possible (e.g., if each @var{input-file} is one
record long). 
The file output from @command{nces} will have the same dimensions
(meaning dimension names as well as sizes) as the input hyperslabs  
(@pxref{nces netCDF Ensemble Statistics}, for a complete description of 
@command{nces}).  
The file output from @command{ncra} will have the same dimensions as
the input hyperslabs except for the record dimension, which will have a   
size @w{of 1} (@pxref{ncra netCDF Record Averager}, for a complete
description of @command{ncra}). 

@html
<a name="ntp"></a> <!-- http://nco.sf.net/nco.html#ntp -->
@end html
@node Interpolating,  , Averaging, Statistics vs Concatenation
@subsection Interpolator @command{ncflint}
@cindex @command{ncflint}

@command{ncflint} can interpolate data between or two files.
Since no other operators have this ability, the description of
interpolation is given fully on the @command{ncflint} reference page
(@pxref{ncflint netCDF File Interpolator}). 
Note that this capability also allows @command{ncflint} to linearly
rescale any data in a netCDF file, e.g., to convert between differing
units. 

@html
<a name="lrg"></a> <!-- http://nco.sf.net/nco.html#lrg -->
<a name="input_files"></a> <!-- http://nco.sf.net/nco.html#input_files -->
@end html
@node Large Numbers of Files, Large Datasets, Statistics vs Concatenation, Strategies
@section Large Numbers of Files
@cindex files, numerous input
@cindex @code{-n @var{loop}}

Occasionally one desires to digest (i.e., concatenate or average)
hundreds or thousands of input files.
@cindex automagic
@cindex @acronym{NASA EOSDIS}
Unfortunately, data archives (e.g., @acronym{NASA EOSDIS}) may not
name netCDF files in a format understood by the @samp{-n @var{loop}}
switch (@pxref{Specifying Input Files}) that automagically generates
arbitrary numbers of input filenames. 
The @samp{-n @var{loop}} switch has the virtue of being concise,
and of minimizing the command line.
This helps keeps output file small since the command line is stored
as metadata in the @code{history} attribute 
(@pxref{History Attribute}). 
However, the @samp{-n @var{loop}} switch is useless when there is no
simple, arithmetic pattern to the input filenames (e.g.,
@file{h00001.nc}, @file{h00002.nc}, @w{@dots{} @file{h90210.nc}}).
Moreover, filename globbing does not work when the input files are too
numerous or their names are too lengthy (when strung together as a
single argument) to be passed by the calling shell to the @acronym{NCO}
operator
@footnote{The exact length which exceeds the operating system internal
limit for command line lengths varies across @acronym{OS}s and shells.
@acronym{GNU} @code{bash} may not have any arbitrary fixed limits to the
size of command line arguments. 
Many @acronym{OS}s cannot handle command line arguments (including
results of file globbing) exceeding 4096 characters.}.
When this occurs, the @acronym{ANSI} C-standard @code{argc}-@code{argv} 
method of passing arguments from the calling shell to a C-program (i.e.,
an @acronym{NCO} operator) breaks down. 
There are (at least) three alternative methods of specifying the input 
filenames to @acronym{NCO} in environment-limited situations.

@html
<a name="stdin"></a> <!-- http://nco.sf.net/nco.html#stdin -->
@end html
@cindex standard input
@cindex provenance
@cindex @code{stdin}
The recommended method for sending very large numbers (hundreds or
more, typically) of input filenames to the multi-file operators is
to pass the filenames with the @acronym{UNIX} @dfn{standard input}
feature, aka @code{stdin}: 
@example
@verbatim
# Pipe large numbers of filenames to stdin
/bin/ls | grep ${CASEID}_'......'.nc | ncecat -o foo.nc
@end verbatim
@end example
This method avoids all constraints on command line size imposed by
the operating system. 
A drawback to this method is that the @code{history} attribute
(@pxref{History Attribute}) does not record the name of any input 
files since the names were not passed as positional arguments
on the command line.
This makes it difficult later to determine the data provenance.
@cindex @code{nco_input_file_number}
@cindex @code{nco_input_file_list}
@cindex global attributes
@cindex attributes, global
To remedy this situation, operators store the number of input files in
the @code{nco_input_file_number} global attribute and the input file
list itself in the @code{nco_input_file_list} global attribute
(@pxref{File List Attributes}). 
Although this does not preserve the exact command used to generate the
file, it does retains all the information required to reconstruct the
command and determine the data provenance.

As of @acronym{NCO} @w{version 5.1.1}, released in November, 2022,
all operators support specifying input files via @code{stdin}
(@pxref{Specifying Input Files}), and also store such input filenames
in the @ref{File List Attributes}).

@cindex globbing
@cindex shell
@cindex extended regular expressions
@cindex regular expressions
@cindex pattern matching
@cindex @command{xargs}
@cindex @acronym{UNIX}
A second option is to use the @acronym{UNIX} @command{xargs} command.
This simple example selects as input to @command{xargs} all the
filenames in the current directory that match a given pattern.
For illustration, consider a user trying to average millions of 
files which each have a six character filename. 
If the shell buffer cannot hold the results of the corresponding
globbing operator, @file{??????.nc}, then the filename globbing
technique will fail. 
Instead we express the filename pattern as an extended regular 
expression, @file{......\.nc} (@pxref{Subsetting Files}).
We use @command{grep} to filter the directory listing for this pattern
and to pipe the results to @command{xargs} which, in turn, passes the
matching filenames to an @acronym{NCO} multi-file operator, e.g.,
@command{ncecat}.
@example
@verbatim
# Use xargs to transfer filenames on the command line
/bin/ls | grep ${CASEID}_'......'.nc | xargs -x ncecat -o foo.nc
@end verbatim
@end example
@cindex pipes
The single quotes protect the only sensitive parts of the extended
regular expression (the @command{grep} argument), and allow shell
interpolation (the @code{$@{CASEID@}} variable substitution) to
proceed unhindered on the rest of the command.
@command{xargs} uses the @acronym{UNIX} pipe feature to append the
suitably filtered input file list to the end of the @command{ncecat}
command options.  
@cindex output file
@cindex input files
@cindex @code{-o @var{fl_out}}
The @code{-o foo.nc} switch ensures that the input files supplied by
@command{xargs} are not confused with the output file name. 
@command{xargs} does, unfortunately, have its own limit (usually about 
20,000 characters) on the size of command lines it can pass.
Give @command{xargs} the @samp{-x} switch to ensure it dies if it
reaches this internal limit.
When this occurs, use either the @code{stdin} method above, or the
symbolic link presented next.

@cindex symbolic links
@cindex Perl
Even when its internal limits have not been reached, the
@command{xargs} technique may not be flexible enough to handle 
all situations. 
A full scripting language like Perl or Python can handle any level of
complexity of filtering input filenames, and any number of filenames.
The technique of last resort is to write a script that creates symbolic 
links between the irregular input filenames and a set of regular,
arithmetic filenames that the @samp{-n @var{loop}} switch understands. 
@cindex Perl
For example, the following Perl script creates a monotonically
enumerated symbolic link to up to one million @file{.nc} files in a
directory. If there are 999,999 netCDF files present, the links are
named @file{000001.nc} to @file{999999.nc}: 
@cindex @code{-n @var{loop}}
@example
@verbatim
# Create enumerated symbolic links
/bin/ls | grep \.nc | perl -e \
'$idx=1;while(<STDIN>){chop;symlink $_,sprintf("%06d.nc",$idx++);}'
ncecat -n 999999,6,1 000001.nc foo.nc
# Remove symbolic links when finished
/bin/rm ??????.nc
@end verbatim
@end example
The @samp{-n @var{loop}} option tells the @acronym{NCO} operator to
automatically generate the filnames of the symbolic links.
This circumvents any @acronym{OS} and shell limits on command-line size.
The symbolic links are easily removed once @acronym{NCO} is finished.
@cindex @code{history}
@cindex provenance
One drawback to this method is that the @code{history} attribute
(@pxref{History Attribute}) retains the filename list of the symbolic
links, rather than the data files themselves. 
This makes it difficult to determine the data provenance at a later
date. 

@node Large Datasets, Memory Requirements, Large Numbers of Files, Strategies
@section Large Datasets
@cindex large datasets
@cindex @acronym{LFS}
@cindex Large File Support

@dfn{Large datasets} are those files that are comparable in size to the
amount of random access memory (@acronym{RAM}) in your computer.
Many users of @acronym{NCO} work with files larger than @w{100 MB}.
Files this large not only push the current edge of storage technology, 
they present special problems for programs which attempt to access the  
entire file at once, such as @command{nces} and @command{ncecat}.
@cindex swap space
If you work with a @w{300 MB} files on a machine with only @w{32 MB} of
memory then you will need large amounts of swap space (virtual memory on
disk) and @acronym{NCO} will work slowly, or even fail. 
There is no easy solution for this.
The best strategy is to work on a machine with sufficient amounts of
memory and swap space. 
Since about 2004, many users have begun to produce or analyze files
exceeding @w{2 GB} in size. 
These users should familiarize themselves with @acronym{NCO}'s Large
File Support (@acronym{LFS}) capabilities (@pxref{Large File Support}).
The next section will increase your familiarity with @acronym{NCO}'s
memory requirements.
With this knowledge you may re-design your data reduction approach to
divide the problem into pieces solvable in memory-limited situations.   

@html
<a name="ulimit"></a> <!-- http://nco.sf.net/nco.html#ulimit -->
@end html
@cindex server
@cindex @acronym{UNICOS}
@cindex Cray
@cindex @acronym{GNU}/Linux
@cindex @code{ulimit}
@cindex @code{core dump}
If your local machine has problems working with large files, try running
@acronym{NCO} from a more powerful machine, such as a network server.  
If you get a memory-related core dump 
(e.g., @samp{Error exit (core dumped)}) on a @acronym{GNU}/Linux system,
or the operation ends before the entire output file is written,
try increasing the process-available memory with @code{ulimit}:
@example
ulimit -f unlimited
@end example
This may solve constraints on clusters where sufficient hardware
resources exist yet where system administrators felt it wise to prevent 
any individual user from consuming too much of resource. 
Certain machine architectures, e.g., Cray @acronym{UNICOS}, have special 
commands which allow one to increase the amount of interactive memory.
@cindex @code{ilimit}
On Cray systems, try to increase the available memory with the
@code{ilimit} command.    

@cindex speed
The speed of the @acronym{NCO} operators also depends on file size.
When processing large files the operators may appear to hang, or do
nothing, for large periods of time.
In order to see what the operator is actually doing, it is useful to
activate a more verbose output mode.
This is accomplished by supplying a number greater @w{than 0} to the
@samp{-D @var{debug-level}} (or @samp{--debug-level}, or
@samp{--dbg_lvl}) switch.
@cindex @code{-D @var{debug-level}}
@cindex @code{--debug-level @var{debug-level}}
@cindex @code{--dbg_lvl @var{debug-level}}
@cindex @var{debug-level}
@cindex @var{dbg_lvl}
@cindex debugging
When the @var{debug-level} is non-zero, the operators report their
current status to the terminal through the @var{stderr} facility.
Using @samp{-D} does not slow the operators down. 
Choose a @var{debug-level} @w{between 1} @w{and 3} for most situations,
e.g., @code{nces -D 2 85.nc 86.nc 8586.nc}.
@w{A full} description of how to estimate the actual amount of memory the
multi-file @acronym{NCO} operators consume is given in 
@ref{Memory Requirements}. 

@html
<a name="mmr"></a> <!-- http://nco.sf.net/nco.html#mmr -->
@end html
@node Memory Requirements, Performance, Large Datasets, Strategies
@section Memory Requirements
@cindex memory requirements
@cindex memory available
@cindex @acronym{RAM}
@cindex swap space
@cindex peak memory usage
@cindex @code{--ram_all}
@cindex @code{--open_ram}
@cindex @code{--diskless_all}

Many people use @acronym{NCO} on gargantuan files which dwarf the
memory available (free @acronym{RAM} plus swap space) even on today's powerful
machines. 
These users want @acronym{NCO} to consume the least memory possible
so that their scripts do not have to tediously cut files into smaller
pieces that fit into memory. 
We commend these greedy users for pushing @acronym{NCO} to its limits!

@cindex threads
@cindex OpenMP
@cindex shared memory machines
This section describes the memory @acronym{NCO} requires during
operation.
The required memory depends on the underlying algorithms, datatypes,
and compression, if any. 
The description below is the memory usage per thread.
Users with shared memory machines may use the threaded @acronym{NCO}
operators (@pxref{OpenMP Threading}).
The peak and sustained memory usage will scale accordingly,
i.e., by the number of threads.
In all cases the memory use refers to the @emph{uncompressed} size of
the data. 
The netCDF4 library automatically decompresses variables during reads. 
The filesize can easily belie the true size of the uncompressed data.
In other words, the usage below can be taken at face value for netCDF3
datasets only.
Chunking will also affect memory usage on netCDF4 operations.
Memory consumption patterns of all operators are similar, with
the exception of @command{ncap2}.

@menu
* Single and Multi-file Operators::
* Memory for ncap2::
@end menu

@node Single and Multi-file Operators, Memory for ncap2, Memory Requirements, Memory Requirements
@subsection Single and Multi-file Operators

@cindex multi-file operators
The multi-file operators currently comprise the record operators,
@command{ncra} and @command{ncrcat}, and the ensemble operators,
@command{nces} and @command{ncecat}. 
The record operators require @emph{much less} memory than the ensemble 
operators. 
This is because the record operators operate on one single record (i.e.,
time-slice) at a time, whereas the ensemble operators retrieve the
entire variable into memory. 
Let @math{MS} be the peak sustained memory demand of an operator,
@math{FT} be the memory required to store the entire contents of all the 
variables to be processed in an input file,
@math{FR} be the memory required to store the entire contents of a
single record of each of the variables to be processed in an input file, 
@math{VR} be the memory required to store a single record of the
largest record variable to be processed in an input file, 
@math{VT} be the memory required to store the largest variable 
to be processed in an input file,
@math{VI} be the memory required to store the largest variable 
which is not processed, but is copied from the initial file to the
output file. 
All operators require @math{MI = VI} during the initial copying of
variables from the first input file to the output file. 
This is the @emph{initial} (and transient) memory demand.
The @emph{sustained} memory demand is that memory required by the
operators during the processing (i.e., averaging, concatenation)
phase which lasts until all the input files have been processed.
The operators have the following memory requirements: 
@command{ncrcat} requires @math{MS <= VR}. 
@command{ncecat} requires @math{MS <= VT}. 
@command{ncra} requires @math{MS = 2FR + VR}. 
@command{nces} requires @math{MS = 2FT + VT}. 
@command{ncbo} requires @math{MS <= 3VT} 
(both input variables and the output variable).
@command{ncflint} requires @math{MS <= 3VT}
(both input variables and the output variable).
@command{ncpdq} requires @math{MS <= 2VT}
(one input variable and the output variable).
@command{ncwa} requires @math{MS <= 8VT} (see below).
Note that only variables that are processed, e.g., averaged,
concatenated, or differenced, contribute to @math{MS}. 
Variables that do not appear in the output file 
(@pxref{Subsetting Files}) are never read and contribute nothing
to the memory requirements. 

Further note that some operators perform internal type-promotion on some
variables prior to arithmetic (@pxref{Type Conversion}).
For example, @command{ncra}, @command{nces}, and @command{ncwa} all
promote integer types to double-precision floating-point prior to
arithmetic, then perform the arithmetic, then demote back to the
original integer type after arithmetic.
This preserves the on-disk storage type while obtaining the precision
advantages of double-precision floating-point arithmetic. 
Since version 4.3.6 (released in September, 2013), @acronym{NCO} also
by default converts single-precision floating-point to double-precision
prior to arithmetic, which incurs the same @acronym{RAM} penalty.
Hence, the sustained memory required for integer variables and
single-precision floats are two or four-times their on-disk,
uncompressed, unpacked sizes if they meet the rules for automatic
internal promotion. 
Put another way, disabling auto-promotion of single-precision variables
(with @samp{--flt}) considerably reduces the @acronym{RAM} footprint 
of arithmetic operators.

The @samp{--open_ram} switch (and switches that invoke it like
@samp{--ram_all} and @samp{--diskless_all}) incurs a @acronym{RAM}
penalty.
These switches cause each input file to be copied to @acronym{RAM} upon
opening. 
Hence any operator invoking these switches utilizes an additional
@math{FT} of @acronym{RAM} (i.e., @math{MS += FT}).
See @ref{RAM disks} for further details. 

@html
<a name="mmr_ncwa"></a> <!-- http://nco.sf.net/nco.html#mmr_ncwa -->
@end html
@command{ncwa} consumes between two and eight times the memory of an
@code{NC_DOUBLE} variable in order to process it. 
Peak consumption occurs when storing simultaneously in memory 
one input variable, one tally array,
one input weight, one conformed/working weight, one weight tally, 
one input mask, one conformed/working mask, and
one output variable. 
@acronym{NCO}'s tally arrays are of type C-type @code{long}, whose size 
is eight-bytes on all modern computers, the same as @code{NC_DOUBLE}
@footnote{By contrast @code{NC_INT} and its deprecated synonym
@code{NC_LONG} are only four-bytes.
Perhaps this is one reason why the @code{NC_LONG} token is
deprecated.}. 
When invoked, the weighting and masking features contribute up to
three-eighths and two-eighths of these requirements apiece.
If weights and masks are @emph{not} specified 
(i.e., no @samp{-w} or @samp{-a} options)
then @command{ncwa} requirements drop to @math{MS <= 3VT}
(one input variable, one tally array, and the output variable). 
The output variable is the same size as the input variable when 
averaging only over a degenerate dimension.
However, normally the output variable is much smaller than the input, 
and is often a simple scalar, in which case the memory requirements
drop by @math{1VT} since the output array requires essentially no
memory. 

All of this is subject to the type promotion rules mentioned above.
For example, @command{ncwa} averaging a variable of type
@code{NC_FLOAT} requires @math{MS <= 16VT} (rather than @math{MS <= 8VT}) 
since all arrays are (at least temporarily) composed of eight-byte
elements, twice the size of the values on disk.
Without mask or weights, the requirements for @code{NC_FLOAT} are
@math{MS <= 6VT} (rather than @math{MS <= 3VT} as for @code{NC_DOUBLE}) 
due to temporary internal promotion of both the input variable and the
output variable to type @code{NC_DOUBLE}. 
The @samp{--flt} option that suppresses promotion reduces this to
@math{MS <= 4VT} (the tally elements do not change size), and to
@math{MS <= 3VT} when the output array is a scalar.

@cindex OpenMP
@cindex threads
The above memory requirements must be multiplied by the number of
threads @var{thr_nbr} (@pxref{OpenMP Threading}).
@cindex @code{-t @var{thr_nbr}}
If this causes problems then reduce (with @samp{-t @var{thr_nbr}}) the
number of threads.

@node Memory for ncap2,  , Single and Multi-file Operators, Memory Requirements
@subsection Memory for @command{ncap2}
@cindex @command{ncap2}
@cindex binary operations
@cindex unary operations
@cindex memory leaks
@cindex left hand casting
@command{ncap2} has unique memory requirements due its ability to process
arbitrarily long scripts of any complexity.
All scripts acceptable to @command{ncap2} are ultimately processed as a
sequence of binary or unary operations.
@command{ncap2} requires @math{MS <= 2VT} under most conditions.
An exception to this is when left hand casting (@pxref{Left hand
casting}) is used to stretch the size of derived variables beyond the
size of any input variables.
Let @math{VC} be the memory required to store the largest variable
defined by left hand casting.
In this case, @math{MS <= 2VC}.

@findex malloc()
@command{ncap2} scripts are complete dynamic and may be of arbitrary
length. 
A script that contains many thousands of operations, may uncover a
slow memory leak even though each single operation consumes little
additional memory. 
Memory leaks are usually identifiable by their memory usage signature.
Leaks cause peak memory usage to increase monotonically with time
regardless of script complexity. 
Slow leaks are very difficult to find.
Sometimes a @command{malloc()} (or @command{new[]}) failure is the
only noticeable clue to their existence.
If you have good reasons to believe that a memory allocation failure  
is ultimately due to an @acronym{NCO} memory leak (rather than
inadequate @acronym{RAM} on your system), then we would be very
interested in receiving a detailed bug report. 

@node Performance,  , Memory Requirements, Strategies
@section Performance

@cindex papers
@cindex overview
An overview of @acronym{NCO} capabilities as of about 2006 is in
Zender, C. S. (2008), 
``Analysis of Self-describing Gridded Geoscience Data with netCDF Operators (NCO)'',
Environ. Modell. Softw., doi:10.1016/j.envsoft.2008.03.004.
This paper is also available at
@url{http://dust.ess.uci.edu/ppr/ppr_Zen08.pdf}.

@cindex scaling
@cindex performance
@acronym{NCO} performance and scaling for arithmetic operations is
described in 
Zender, C. S., and H. J. Mangalam (2007), 
``Scaling Properties of Common Statistical Operators for Gridded Datasets'', 
Int. @w{J. High} Perform. Comput. Appl., 21(4), 485-498,
doi:10.1177/1094342007083802. 
This paper is also available at
@url{http://dust.ess.uci.edu/ppr/ppr_ZeM07.pdf}.

It is helpful to be aware of the aspects of @acronym{NCO} design 
that can limit its performance:
@enumerate
@item 
@cindex buffering
No data buffering is performed during @command{nc_get_var} and
@command{nc_put_var} operations.  
@cindex performance
@cindex operator speed
@cindex speed
@cindex execution time
Hyperslabs too large to hold in core memory will suffer substantial
performance penalties because of this. 

@item 
@cindex monotonic coordinates
Since coordinate variables are assumed to be monotonic, the search for 
bracketing the user-specified limits should employ a quicker algorithm,
like bisection, than the two-sided incremental search currently
implemented.  

@item 
@cindex @var{C_format}
@cindex @var{FORTRAN_format}
@cindex @var{signedness}
@cindex @var{scale_format} 
@cindex @var{add_offset} 
@var{C_format}, @var{FORTRAN_format}, @var{signedness},
@var{scale_format} and @var{add_offset} attributes are ignored by
@command{ncks} when printing variables to screen. 

@item
@cindex Yorick
In the late 1990s it was discovered that some random access operations
on large files on certain architectures (e.g., @acronym{UNICOS}) were
much slower with @acronym{NCO} than with similar operations performed
using languages that bypass the netCDF interface (e.g., Yorick).    
This may have been a penalty of unnecessary byte-swapping in the netCDF 
interface.  
It is unclear whether such problems exist in present day (2007)
netCDF/@acronym{NCO} environments, where unnecessary byte-swapping has
been reduced or eliminated.
@end enumerate

@html
<a name="ftr"></a> <!-- http://nco.sf.net/nco.html#ftr -->
@end html
@node Shared features, Reference Manual, Strategies, Top
@chapter Shared Features

Many features have been implemented in more than one operator and are
described here for brevity. 
The description of each feature is preceded by a box listing the
operators for which the feature is implemented. 
@cindex command line switches
Command line switches for a given feature are consistent across all
operators wherever possible. 
If no ``key switches'' are listed for a feature, then that particular
feature is automatic and cannot be controlled by the user. 

@menu
* Internationalization::
* Metadata Optimization::
* OpenMP Threading::
* Command Line Options::
* Sanitization of Input::
* Specifying Input Files::
* Specifying Output Files::
* Remote storage::
* Retaining Retrieved Files::
* File Formats and Conversion::
* Zarr and NCZarr::
* Large File Support::
* Subsetting Files::
* Subsetting Coordinate Variables::
* Group Path Editing::
* C and Fortran Index Conventions::
* Hyperslabs::
* Stride::
* Record Appending::
* Subcycle::
* Interleave::
* Multislabs::
* Wrapped Coordinates::
* Auxiliary Coordinates::
* Grid Generation::
* Regridding::
* Climatology and Bounds Support::
* UDUnits Support::
* Rebasing Time Coordinate::
* Multiple Record Dimensions::
* Missing Values::
* Chunking::
* Compression::
* Deflation::
* MD5 digests::
* Buffer sizes::
* RAM disks::
* Unbuffered I/O::
* Packed data::
* Operation Types::
* Type Conversion::
* Batch Mode::
* Global Attribute Addition::
* History Attribute::
* File List Attributes::
* CF Conventions::
* ARM Conventions::
* Operator Version::
@end menu

@html
<a name="i18n"></a> <!-- http://nco.sf.net/nco.html#i18n -->
@end html
@node Internationalization, Metadata Optimization, Shared features, Shared features
@section Internationalization
@cindex Internationalization
@cindex I18N
@cartouche
Availability: All operators@*
@end cartouche
@cindex L10N
@acronym{NCO} support for @dfn{internationalization} of textual input
and output (e.g., Warning messages) is nascent.
We introduced the first foreign language string catalogues (French and
Spanish) in 2004, yet did not activate these in distributions because 
the catalogues were nearly empty.
We seek volunteers to populate our templates with translations for their
favorite languages.
@c fxm: Work on this section

@html
<a name="hdr"></a> <!-- http://nco.sf.net/nco.html#hdr -->
<a name="hdr_pad"></a> <!-- http://nco.sf.net/nco.html#hdr_pad -->
@end html
@node Metadata Optimization, OpenMP Threading, Internationalization, Shared features
@section Metadata Optimization
@cindex metadata optimization
@cindex performance
@cindex operator speed
@cindex speed
@cindex execution time
@cindex @code{nc__enddef()}
@cindex @code{--hdr_pad @var{hdr_pad}}
@cindex @code{--header_pad @var{hdr_pad}}
@cartouche
Availability: All operators@*
Short options: None@*
Long options: @samp{--hdr_pad}, @samp{--header_pad}@* 
@end cartouche
@acronym{NCO} supports padding headers to improve the speed of future
metadata operations.
Use the @samp{--hdr_pad} and @samp{--header_pad} switches to request
that @var{hdr_pad} bytes be inserted into the metadata section of the
output file.
There is little downside to padding a header with kilobyte of space,
since subsequent manipulation of the file will annotate the
@code{history} attribute with all commands, let alone any explicit
metadata additions with @command{ncatted}.
@example
ncks --hdr_pad=1000  in.nc out.nc # Pad header with  1 kB space
ncks --hdr_pad=10000 in.nc out.nc # Pad header with 10 kB space
@end example
Future metadata expansions will not incur the netCDF3 performance
penalty of copying the entire output file unless the expansion exceeds
the amount of header padding.
This can be beneficial when it is known that some metadata will be added
at a future date.
The operators that benefit most from judicious use of header padding
are @command{ncatted} and @command{ncrename}, since they only alter
metadata. 

This optimization exploits the netCDF library @code{nc__enddef()}
function.
This function behaves differently with different storage formats.
It will improve speed of future metadata expansion with @code{CLASSIC}
and @code{64bit} netCDF files, though not necessarily with @code{NETCDF4} 
files, i.e., those created by the netCDF interface to the @acronym{HDF5}
library (@pxref{File Formats and Conversion}).
netCDF3 formats use a simple sequential ordering that requires copying
the file if the size of new metadata exceeds the available padding.
netCDF4 files use internal file pointers that allow flexibility at
inserting and removing data without necessitating copying the whole
file.

@html
<a name="omp"></a> <!-- http://nco.sf.net/nco.html#omp -->
<a name="openmp"></a> <!-- http://nco.sf.net/nco.html#openmp -->
@end html
@node OpenMP Threading, Command Line Options, Metadata Optimization, Shared features
@section OpenMP Threading
@cindex OpenMP
@cindex threads
@cindex @acronym{SMP}
@cindex shared memory parallelism
@cindex parallelism
@cindex @code{nco_openmp_thread_number}
@cindex @code{--thr_nbr @var{thr_nbr}}
@cindex @code{--threads @var{thr_nbr}}
@cindex @code{--omp_num_threads @var{thr_nbr}}
@cindex @code{-t @var{thr_nbr}}
@cartouche
Availability: @command{ncclimo}, @command{ncks}, @command{ncremap}@*
Short options: @samp{-t}@*
Long options: @samp{--thr_nbr}, @samp{--threads},
@samp{--omp_num_threads}@* 
@end cartouche
@acronym{NCO} supports shared memory parallelism (@acronym{SMP}) when
compiled with an OpenMP-enabled compiler.
Threads requests and allocations occur in two stages.
First, users may request a specific number of threads @var{thr_nbr} with
the @samp{-t} switch (or its long option equivalents, @samp{--thr_nbr},
@samp{--threads}, and @samp{--omp_num_threads}).
If not user-specified, OpenMP obtains @var{thr_nbr} from the
@env{OMP_NUM_THREADS} environment variable, if present, or from the
@acronym{OS}, if not.

@cartouche
Caveat:
Unfortunately, threading does not improve @acronym{NCO} throughput (i.e.,
wallclock time) because nearly all @acronym{NCO} operations are
I/O-bound. 
This means that @acronym{NCO} spends negligible time doing anything
compared to reading and writing. 
The only exception is regridding with @command{ncremap} which uses
@command{ncks} under-the-hood.
As of 2017, threading works only for regridding, thus this section is
relevant only to @command{ncclimo}, @command{ncks}, and
@command{ncremap}. 
We have seen some and can imagine other use cases where @command{ncwa},
@command{ncpdq}, and @command{ncap2} (with long scripts) will complete
faster due to threading.  
The main benefits of threading so far have been to isolate the serial
from parallel portions of code. 
This parallelism is now exploited by OpenMP but then runs into the I/O
bottleneck during output. 
The bottleneck will be ameliorated for large files by the use of
MPI-enabled calls in the netCDF4 library when the underlying filesystem
is parallel (e.g., @acronym{PVFS} or @acronym{JFS}).
Implementation of the parallel output calls in @acronym{NCO} is not a
goal of our current funding and would require new volunteers or funding.   
@end cartouche

@cindex @var{thr_nbr}
@cindex @env{OMP_NUM_THREADS}
@cindex @command{ncrcat}
@cindex @command{ncwa}
@cindex @command{ncap2}
@cindex @command{ncpdq}
@cindex large datasets
@acronym{NCO} may modify @var{thr_nbr} according to its own internal
settings before it requests any threads from the system.
Certain operators contain hard-code limits to the number of threads they
request.
We base these limits on our experience and common sense, and to reduce
potentially wasteful system usage by inexperienced users.
For example, @code{ncrcat} is extremely I/O-intensive so we restrict
@math{@var{thr_nbr} <= 2} for @code{ncrcat}.
This is based on the notion that the best performance that can be
expected from an operator which does no arithmetic is to have one thread
reading and one thread writing simultaneously.
In the future (perhaps with netCDF4), we hope to demonstrate significant
threading improvements with operators like @code{ncrcat} by performing
multiple simultaneous writes. 

Compute-intensive operators (@code{ncremap}) benefit most from
threading. 
The greatest increases in throughput due to threading occur on
large datasets where each thread performs millions, at least,
of floating-point operations.
Otherwise, the system overhead of setting up threads probably outweighs 
the speed enhancements due to @acronym{SMP} parallelism.
However, we have not yet demonstrated that the @acronym{SMP} parallelism 
scales beyond four threads for these operators.
Hence we restrict @math{@var{thr_nbr} <= 4} for all operators.
We encourage users to play with these limits (edit file
@file{nco_omp.c}) and send us their feedback.

@cindex debugging
@cindex @var{dbg_lvl}
Once the initial @var{thr_nbr} has been modified for any
operator-specific limits, @acronym{NCO} requests the system to allocate 
a team of @var{thr_nbr} threads for the body of the code.
The operating system then decides how many threads to allocate
based on this request.
Users may keep track of this information by running the operator with
@math{@var{dbg_lvl} > 0}.

By default, threaded operators attach one global attribute,
@code{nco_openmp_thread_number}, to any file they create or modify. 
This attribute contains the number of threads the operator used to
process the input files. 
This information helps to verify that the answers with threaded and
non-threaded operators are equal to within machine precision.
@cindex benchmarks
This information is also useful for benchmarking.

@html
<a name="cmd_ln"></a> <!-- http://nco.sf.net/nco.html#cmd_ln -->
@end html
@node Command Line Options, Sanitization of Input, OpenMP Threading, Shared features
@section Command Line Options
@cindex command line options
@cartouche
Availability: All operators@*
@end cartouche
@cindex @acronym{POSIX}
@cindex @acronym{UNIX}
@cindex @acronym{GNU}
@cindex switches
@acronym{NCO} achieves flexibility by using @dfn{command line options}.
These options are implemented in all traditional @acronym{UNIX} commands 
as single letter @dfn{switches}, e.g., @samp{ls -l}.
For many years @acronym{NCO} used only single letter option names.
In late 2002, we implemented @acronym{GNU}/@acronym{POSIX} extended
or long option names for all options.
This was done in a backward compatible way such that the full
functionality of @acronym{NCO} is still available through the familiar 
single letter options.
Many features of @acronym{NCO} introduced since 2002 now require the 
use of long options, simply because we have nearly run out of single
letter options.
More importantly, mnemonics for single letter options are often
non-intuitive so that long options provide a more natural way of
expressing intent.

@cindex long options
Extended options, also called long options, are implemented using the
system-supplied @file{getopt.h} header file, if possible. 
@cindex @code{BSD}
@cindex @code{getopt}
@cindex @code{getopt_long}
@cindex @file{getopt.h}
This provides the @command{getopt_long} function to @acronym{NCO}
@footnote{
If a @command{getopt_long} function cannot be found on the system, 
@acronym{NCO} will use the @command{getopt_long} from the
@command{my_getopt} package by Benjamin Sittler
@email{bsittler@@iname.com}.
This is @acronym{BSD}-licensed software available from  
@uref{http://www.geocities.com/ResearchTriangle/Node/9405/#my_getopt}.}. 

@cindex @code{-D @var{debug-level}}
@cindex @code{--dbg_lvl @var{debug-level}}
The syntax of @dfn{short options} (single letter options) is
@kbd{-@var{key} @var{value}} (dash-key-space-value).
Here, @var{key} is the single letter option name, e.g., 
@samp{-D 2}.

The syntax of @dfn{long options} (multi-letter options) is 
@kbd{--@var{long_name} @var{value}}
(dash-dash-key-space-value), e.g., @samp{--dbg_lvl 2} or
@kbd{--@var{long_name}=@var{value}}
(dash-dash-key-equal-value), e.g., @samp{--dbg_lvl=2}.
Thus the following are all valid for the @samp{-D} (short version)
or @samp{--dbg_lvl} (long version) command line option.
@example
ncks -D 3 in.nc        # Short option, preferred form
ncks -D3 in.nc         # Short option, alternate form
ncks --dbg_lvl=3 in.nc # Long option, preferred form
ncks --dbg_lvl 3 in.nc # Long option, alternate form
@end example
@noindent
The third example is preferred for two reasons.
First, @samp{--dbg_lvl} is more specific and less ambiguous than
@samp{-D}.
The long option format makes scripts more self documenting and less
error-prone.   
Often long options are named after the source code variable whose value 
they carry.
Second, the equals sign @kbd{=} joins the key (i.e., @var{long_name}) to   
the value in an uninterruptible text block. 
Experience shows that users are less likely to mis-parse commands when
restricted to this form.

@menu
* Truncating Long Options::
* Multi-arguments::
@end menu

@node Truncating Long Options, Multi-arguments, Command Line Options, Command Line Options
@subsection Truncating Long Options
@cindex Truncating options
@cindex Options, truncating
@acronym{GNU} implements a superset of the @acronym{POSIX} standard.
Their superset accepts any unambiguous truncation of a valid option:
@example
ncks -D 3 in.nc        # Short option
ncks --dbg_lvl=3 in.nc # Long option, full form
ncks --dbg=3 in.nc     # Long option, OK unambiguous truncation
ncks --db=3 in.nc      # Long option, OK unambiguous truncation
ncks --d=3 in.nc       # Long option, ERROR ambiguous truncation
@end example
@noindent
The first four examples are equivalent and will work as expected.
The final example will exit with an error since @command{ncks} cannot
disambiguate whether @samp{--d} is intended as a truncation of
@samp{--dbg_lvl}, of @samp{--dimension}, or of some other long option.  

@acronym{NCO} provides many long options for common switches.
For example, the debugging level may be set in all operators with any
of the switches @samp{-D}, @samp{--debug-level}, or @samp{--dbg_lvl}.
This flexibility allows users to choose their favorite mnemonic.
For some, it will be @samp{--debug} (an unambiguous truncation of
@samp{--debug-level}, and other will prefer @samp{--dbg}.
Interactive users usually prefer the minimal amount of typing, i.e.,
@samp{-D}.
We recommend that re-usable scripts employ long options to facilitate
self-documentation and maintainability.  

This manual generally uses the short option syntax in examples.
This is for historical reasons and to conserve space in printed output.
Users are expected to pick the unambiguous truncation of each option
name that most suits their taste.

@html
<a name="MTA"></a> <!-- http://nco.sf.net/nco.html#MTA -->
<a name="mta"></a> <!-- http://nco.sf.net/nco.html#mta -->
<a name="multi-arguments"></a> <!-- http://nco.sf.net/nco.html#multi-arguments -->
<a name="Multi-arguments"></a> <!-- http://nco.sf.net/nco.html#Multi-arguments -->
@end html
@node Multi-arguments,  , Truncating Long Options, Command Line Options
@subsection Multi-arguments
@cindex Multi-arguments
@cindex @acronym{MTA}
@cindex Options, multi-argument
@cindex @code{--gaa @var{key}=@var{val}}
@cindex @code{--ppc @var{key}=@var{val}}
@cindex @code{--rgr @var{key}=@var{val}}
@cindex @code{--trr @var{key}=@var{val}}
As of @acronym{NCO} version 4.6.2 (November, 2016), @acronym{NCO} accepts
multiple key-value pair options for a single feature to be joined
together into a single extended argument called a @dfn{multi-argument},
sometimes abbreviated @acronym{MTA}. 
Only four @acronym{NCO} features accept multiple key-value pairs that
can be aggregated into multi-arguments. 
These features are:
Global Attribute Addition options indicated via @samp{--gaa} (@pxref{Global Attribute Addition});
Image Manipulation indicated via @samp{--trr}@footnote{
@html
<a name="envi"></a> <!-- http://nco.sf.net/nco.html#envi -->
<a name="ENVI"></a> <!-- http://nco.sf.net/nco.html#ENVI -->
<a name="BIL"></a> <!-- http://nco.sf.net/nco.html#BIL -->
<a name="BSQ"></a> <!-- http://nco.sf.net/nco.html#BSQ -->
<a name="BIP"></a> <!-- http://nco.sf.net/nco.html#BIP -->
<a name="bil"></a> <!-- http://nco.sf.net/nco.html#bil -->
<a name="bsq"></a> <!-- http://nco.sf.net/nco.html#bsq -->
<a name="bip"></a> <!-- http://nco.sf.net/nco.html#bip -->
@end html
@cindex @acronym{ENVI}
@cindex @acronym{DOE}
@cindex @acronym{BIL} format
@cindex @acronym{BSQ} format
@cindex @acronym{BIP} format
@cindex Terraref
@cindex @samp{--trr}
@acronym{NCO} supports decoding @acronym{ENVI} images in support of the
@acronym{DOE} Terraref project.
These options are indicated via the @command{ncks} @samp{--trr} switch,
and are otherwise undocumented.
Please contact us if more support and documentation of handling of
@acronym{ENVI} @acronym{BIL}, @acronym{BSQ}, and @acronym{BIP} images
would be helpful},
Precision-Preserving Compression options are indicated via
@samp{--ppc} (@pxref{Precision-Preserving Compression}); and 
Regridding options are indicated via @samp{--rgr} (@pxref{Regridding}).
Arguments to these four indicator options take the form of key-value
pairs, e.g., @samp{--rgr @var{key}=@var{val}}. 
These four features have so many options that making each key its own
command line option would pollute the namespace of @acronym{NCO}'s
global options. 
Yet supplying multiple options to each indicator option one-at-a-time
can result in command lines overpopulated with indicator switches (e.g.,
@samp{--rgr}): 
@example
@verbatim
ncks --rgr grd_ttl='Title' --rgr grid=grd.nc --rgr latlon=129,256 \
     --rgr lat_typ=fv --rgr lon_typ=grn_ctr ...
@end verbatim
@end example

Multi-arguments combine all the indicator options into one option that
receives a single argument that comprises all the original arguments
glued together by a delimiter, which is, by default, @samp{#}.
Thus the multi-argument version of the above example is
@example
@verbatim
ncks --rgr grd_ttl='Title'#grid=grd.nc#latlon=129,256#lat_typ=fv#lon_typ=grn_ctr
@end verbatim
@end example
Note the aggregation of all @var{key}=@var{val} pairs into a single
argument.
@acronym{NCO} simply splits this argument at each delimiter, and
processes the sub-arguments as if they had been passed with their own
indicator option.
Multi-arguments produce the same results, and may be mixed with,
traditional indicator options supplied one-by-one.

As mentioned previously, the multi-argument delimiter string is, by
default, the hash-sign @samp{#}. 
When any @var{key}=@var{val} pair contains the default delimiter, the
user must specify a custom delimiter string so that options are parsed
correctly. 
The options to change the multi-argument delimiter string are
@samp{--mta_dlm=@var{delim_string}} or
@samp{--dlm_mta=@var{delim_string}}, where @var{delim_string} can be any
single or multi-character string that (1) is not contained in any
@var{key} or @var{val} string; and (2) will not confuse the shell.
For example, to use multi-arguments to pass a string that includes 
the hash symbol (the default delimiter is @samp{#}), one must also
change the delimiter so something besides hash, e.g., a colon @samp{:}: 
@example
@verbatim
ncks --dlm=":" --gaa foo=bar:foo2=bar2:foo3,foo4="hash # is in value" 
ncks --dlm=":" --gaa foo=bar:foo2=bar2:foo3,foo4="Thu Sep 15 13\:03\:18 PDT 2016"
ncks --dlm="csz" --gaa foo=barcszfoo2=bar2cszfoo3,foo4="Long text"
@end verbatim
@end example
In the second example, the colons that are escaped with the backslash
become literal characters.  
Many characters have special shell meanings and so must be escaped by a 
single or double backslash or enclosed in single quotes to prevent
interpolation. 
These special characters include @samp{:}, @samp{$}, @samp{%}, @samp{*},
@samp{@@}, and @samp{&}. 
If @var{val} is a long text string that could contain the default
delimiter, then delimit with a unique multi-character string such as
@samp{csz} in the third example. 

As of @acronym{NCO} version 4.6.7 (May, 2017), multi-argument
flags no longer need be specified as key-value pairs.
By definition a flag sets a boolean value to either True or False.
Previously @acronym{MTA} flags had to employ key-value pair syntax,
e.g., @samp{--rgr infer=Y} or @samp{--rgr no_cll_msr=anything} in order 
to parse correctly.
Now the @acronym{MTA} parser accepts flags in the more intuitive syntax
where they are listed by name, i.e., the flag name alone indicates the
flag to set, e.g., @samp{--rgr infer} or @samp{--rgr no_cll_msr} are
valid. 
A consequence of this is that flags in multi-argument strings appear
as straightforward flag names, e.g.,
@samp{--rgr infer#no_cll_msr#latlon=129,256}.
It is also valid to prefix flags in multi-arument strings with
single or double-dashes to make the flags more visible, e.g.,
@samp{--rgr latlon=129,256#--infer#-no_cll_msr}. 

@html
<a name="scr"></a> <!-- http://nco.sf.net/nco.html#scr -->
<a name="sntz"></a> <!-- http://nco.sf.net/nco.html#sntz -->
<a name="security"></a> <!-- http://nco.sf.net/nco.html#security -->
<a name="sanitize"></a> <!-- http://nco.sf.net/nco.html#sanitize -->
<a name="whitelist"></a> <!-- http://nco.sf.net/nco.html#whitelist -->
<a name="wht_lst"></a> <!-- http://nco.sf.net/nco.html#wht_lst -->
@end html
@node Sanitization of Input, Specifying Input Files, Command Line Options, Shared features
@section Sanitization of Input
@cindex sanitize
@cindex security
@cindex whitelist
@cindex blacklist
@cartouche
Availability: All operators@*
@end cartouche
@acronym{NCO} is often installed in system directories (although not
with Conda), and on some production machines it may have escalated
privileges. 
Since @acronym{NCO} manipulates files by using @code{system()} calls (e.g., to 
move and copy them with @code{mv} and @code{cp}) it makes sense to audit it for vulnerabilities and 
protect it from malicious users trying to exploit security gaps.
Securing @acronym{NCO} against malicious attacks is multi-faceted, and
involves careful memory management and auditing of user-input.
In versions 4.7.3--4.7.6 (March-September, 2018), @acronym{NCO}
implements a whitelist of characters allowed in user-specified
filenames.
This whitelist proved unpopular mainly because it proscribed certain
character combinations that could appear in automatically generated
files, and was therefore turned-off in 4.7.7 and following versions.
The whitelist is described here for posterity and for possible
improvement and re-introduction:
The purpose of the whitelist was to prevent malicious users from injecting
filename strings that could be used for attacks. 
The whitelist allowed only these characters:
@example
@verbatim
abcdefghijklmnopqrstuvwxyz
ABCDEFGHIJKLMNOPQRSTUVWXYZ
1234567890_-.@ :%/
@end verbatim
@end example
The backslash character @kbd{\} was also whitelisted for Windows only.
This whitelist allows filenames to be @acronym{URL}s, include username
prefixes, and standard non-alphabetic characters. 
The implied blacklist included these characters
@example
@verbatim
;|<>[](),&*?'"
@end verbatim
@end example
This blacklist rules-out strings that may contain dangerous commands and 
injection attacks.
If you would like any of these characters whitelisted, please contact
us and include a compelling real-world use-case.

The @acronym{DAP} protocol supports accessing files with so-called
``constraint expressions''.
@acronym{NCO} allows access to a wider set of whitelisted characters
for files whose names indicate the @acronym{DAP} protocol.
This is defined as any filename beginning with the string @samp{http://}, @samp{https://}, 
or @samp{dap4://}.
The whitelist for these files is expanded to include these characters:
@example
@verbatim
#=:[];|{}/<>
@end verbatim
@end example

The whitelist method is straightforward, and does not interfere
with @acronym{NCO}'s globbing feature.
The whitelist applies only to filenames because they are handled by
shell commands passed to the @command{system()} function.  
However, the whitelist method is applicable to other user-input such
as variable lists, hyperslab arguments, etc.
Hence, the whitelist could be applied to other user-input in the future.

@html
<a name="fl_in"></a> <!-- http://nco.sf.net/nco.html#fl_in -->
<a name="in"></a> <!-- http://nco.sf.net/nco.html#in -->
<a name="input"></a> <!-- http://nco.sf.net/nco.html#input -->
@end html
@node Specifying Input Files, Specifying Output Files, Sanitization of Input, Shared features
@section Specifying Input Files
@cindex input files
@cindex globbing
@cindex regular expressions
@cindex wildcards
@cindex @code{NINTAP}
@cindex Processor, @acronym{CCM}
@cindex @acronym{CCM} Processor
@cindex standard input
@cindex @code{stdin}
@cindex @code{-n @var{loop}}
@cindex @code{--nintap @var{loop}}
@cindex @code{-p @var{input-path}}
@cindex @code{--pth @var{input-path}}
@cindex @code{--path @var{input-path}}
@cindex @var{input-path}
@cartouche
Availability (@code{-n}): @command{nces}, @command{ncecat}, @command{ncra}, @command{ncrcat}@*
Availability (@code{-p}): All operators@*
Availability (@code{stdin}): All operators@*
Short options: @samp{-n}, @samp{-p}@*
Long options: @samp{--nintap}, @samp{--pth}, @samp{--path}@*
@end cartouche
It is important that users be able to specify multiple input files
without typing every filename in full, often a tedious task even
by graduate student standards.
@cindex @acronym{UNIX}
There are four different ways of specifying input files to @acronym{NCO}:
explicitly typing each, using @acronym{UNIX} shell wildcards, and using
the @acronym{NCO} @samp{-n} and @samp{-p} switches (or their long option
equivalents, @samp{--nintap} or @samp{--pth} and @samp{--path},
respectively). 
Techniques to augment these methods to specify arbitrary numbers (e.g.,
thousands) and patterns of filenames are discussed separately 
(@pxref{Large Numbers of Files}).

To illustrate these methods, consider the simple problem of using
@command{ncra} to average five input files, @file{85.nc}, @file{86.nc},
@w{@dots{} @file{89.nc}}, and store the results in @file{8589.nc}.
Here are the four methods in order.
They produce identical answers.
@example
ncra 85.nc 86.nc 87.nc 88.nc 89.nc 8589.nc
ncra 8[56789].nc 8589.nc
ncra 8?.nc 8589.nc
ncra -p @var{input-path} 85.nc 86.nc 87.nc 88.nc 89.nc 8589.nc
ncra -n 5,2,1 85.nc 8589.nc
@end example
The first method (explicitly specifying all filenames) works by brute 
force. 
The second method relies on the operating system shell to @dfn{glob}
(expand) the @dfn{regular expression} @code{8[56789].nc}.
The shell then passes the valid filenames (those which match the
regular expansion) to @command{ncra}.
In this case @command{ncra} never knows that a regular expression was
used, because the shell intercepts and expands and matches the regular
expression before @command{ncra} is actually invoked.
The third method is uses globbing with a different regular expression
that is less safe (it will also match unwanted files such as
@file{81.nc} and @file{8Z.nc} if present). 
The fourth method uses the @samp{-p @var{input-path}} argument to
specify the directory where all the input files reside.
@acronym{NCO} prepends @var{input-path} (e.g.,
@file{/data/username/model}) to all @var{input-files} (though not to
@var{output-file}).  
Thus, using @samp{-p}, the path to any number of input files need only
be specified once.
Note @var{input-path} need not end with @samp{/}; the @samp{/} is
automatically generated if necessary. 

The last method passes (with @samp{-n}) syntax concisely describing 
the entire set of filenames
@footnote{The @samp{-n} option is a backward-compatible superset of the 
@code{NINTAP} option from the @acronym{NCAR} @acronym{CCM} Processor.
The @acronym{CCM} Processor was custom-written Fortran code maintained
for many years by Lawrence Buja at @acronym{NCAR}, and phased-out in 
the late 1990s.
@acronym{NCO} copied some ideas, like @code{NINTAP}-functionality,
from @acronym{CCM} Processor capabilities.}.
@cindex multi-file operators
@cindex files, multiple
This option is only available with the @dfn{multi-file operators}:
@command{ncra}, @command{ncrcat}, @command{nces}, and @command{ncecat}.
By definition, multi-file operators are able to process an arbitrary
number of @var{input-files}.
This option is very useful for abbreviating lists of filenames
representable as
@var{alphanumeric_prefix}+@var{numeric_suffix}+@file{.}+@var{filetype}
where @var{alphanumeric_prefix} is a string of arbitrary length and
composition, @var{numeric_suffix} is a fixed width field of digits, and
@var{filetype} is a standard filetype indicator. 
For example, in the file @file{ccm3_h0001.nc}, we have
@var{alphanumeric_prefix} = @file{ccm3_h}, @var{numeric_suffix} =
@file{0001}, and @var{filetype} = @file{nc}.

@acronym{NCO} decodes lists of such filenames encoded using the
@samp{-n} syntax. 
The simpler (three-argument) @samp{-n} usage takes the form 
@code{-n @var{file_number},@var{digit_number},@var{numeric_increment}}
where @var{file_number} is the number of files, @var{digit_number} is
the fixed number of numeric digits comprising the @var{numeric_suffix},
and @var{numeric_increment} is the constant, integer-valued difference
between the @var{numeric_suffix} of any two consecutive files.
The value of @var{alphanumeric_prefix} is taken from the input file,
which serves as a template for decoding the filenames.
In the example above, the encoding @code{-n 5,2,1} along with the input
file name @file{85.nc} tells @acronym{NCO} to
construct five (5) filenames identical to the template @file{85.nc}
except that the final two (2) digits are a numeric suffix to be
incremented by one (1) for each successive file.
Currently @var{filetype} may be either be empty, @file{nc}, @file{h5},
@file{cdf}, @file{hdf}, @file{hd5}, or @file{he5}.
If present, these @var{filetype} suffixes (and the preceding @file{.})
are ignored by @acronym{NCO} as it uses the @samp{-n} arguments to
locate, evaluate, and compute the @var{numeric_suffix} component of
filenames. 

@cindex wrapped filenames
@cindex climate model
Recently the @samp{-n} option has been extended to allow convenient
specification of filenames with ``circular'' characteristics.
This means it is now possible for @acronym{NCO} to automatically
generate filenames which increment regularly until a specified maximum
value, and then wrap back to begin again at a specified minimum value. 
The corresponding @samp{-n} usage becomes more complex, taking one or
two additional arguments for a total of four or five, respectively: 
@code{-n
@var{file_number},@var{digit_number},@var{numeric_increment}[,@var{numeric_max}[,@var{numeric_min}]]}
where @var{numeric_max}, if present, is the maximum integer-value of 
@var{numeric_suffix} and @var{numeric_min}, if present, is the minimum
integer-value of @var{numeric_suffix}.
Consider, for example, the problem of specifying non-consecutive input
files where the filename suffixes end with the month index.  
In climate modeling it is common to create summertime and wintertime
averages which contain the averages of the months June--July--August,
and December--January--February, respectively:
@example
ncra -n 3,2,1 85_06.nc 85_0608.nc
ncra -n 3,2,1,12 85_12.nc 85_1202.nc
ncra -n 3,2,1,12,1 85_12.nc 85_1202.nc
@end example
The first example shows that three arguments to the @samp{-n} option
suffice to specify consecutive months (@code{06, 07, 08}) which do not
``wrap'' back to a minimum value.
The second example shows how to use the optional fourth and fifth
elements of the @samp{-n} option to specify a wrap value.
The fourth argument to @samp{-n}, when present, specifies the maximum
integer value of @var{numeric_suffix}.
In the example the maximum value @w{is 12,} and will be formatted as
@file{12} in the filename string. 
The fifth argument to @samp{-n}, when present, specifies the minimum
integer value of @var{numeric_suffix}.
The default minimum filename suffix @w{is 1,} which is formatted as
@file{01} in this case.   
Thus the second and third examples have the same effect, that is, they
automatically generate, in order, the filenames @file{85_12.nc},
@file{85_01.nc}, and @file{85_02.nc} as input to @acronym{NCO}.

As of @acronym{NCO} version 4.5.2 (September, 2015), @acronym{NCO}
supports an optional sixth argument to @samp{-n}, the month-indicator. 
The month-indicator affirms to @acronym{NCO} that the right-most digits
being manipulated in the generated filenames correspond to month numbers 
(with January formatted as @file{01} and December as @file{12}). 
Moreover, it assumes digits to the left of the month are the year.
The full (six-argument) @samp{-n} usage takes the form 
@code{-n @var{file_number},@var{digit_number},@var{month_increment},@var{max_month},@var{min_month},@samp{yyyymm}}.
The @samp{yyyymm} string is a clunky way (can you think of a clearer
way?) to tell @acronym{NCO} to enumerate files in year-month mode.
When present, @samp{yyyymm} string causes @acronym{NCO} to automatically
generate series of filenames whose right-most two digits increment from
@var{min_month} by @var{month_increment} up to @var{max_month} and then
the leftmost digits (i.e., the year) increment by one, and the whole
process is repeated until the @var{file_number} filenames are
generated. 
@example
ncrcat -n 3,6,1,12,1         198512.nc 198512_198502.nc
ncrcat -n 3,6,1,12,1,yyyymm  198512.nc 198512_198602.nc
ncrcat -n 3,6,1,12,12,yyyymm 198512.nc 198512_198712.nc
@end example
The first command above concatenates three files (@file{198512.nc}, 
@file{198501.nc}, @file{198502.nc}) into the output file.
The second command above concatenates three files (@file{198512.nc}, 
@file{198601.nc}, @file{198602.nc}).
The @samp{yyyymm}-indicator causes the left-most digits to
increment each time the right-most two digits reach their
maximum and then wrap.
The first command does not have the indicator so it is always 1985.  
The third command concatenates three files (@file{198512.nc},
@file{198612.nc}, @file{198712.nc}).

As of @acronym{NCO} @w{version 5.1.1}, released in November, 2022,
all operators support specifying input files via @code{stdin}.
This capability was implemented with NCZarr in mind, though it can
also be used with traditional @acronym{POSIX} files.
The @command{ncap2}, @command{ncks}, @command{ncrename}, and
@command{ncatted} operators accept one or two filenames as positional
arguments.
If the input file for these operators is provided via @code{stdin},
then the output file, if any, must be specified with @samp{-o out.nc}
so the operators know whether to check @code{stdin}.
Multi-file operators (@command{ncra}, @command{ncea},
@command{ncrcat}, @command{ncecat}) will continue to identify the last
positional argument as the output file unless the @samp{-o out.nc}
form is used.
The best best practice is to use @samp{-o out.nc} to specify output
filenames when @code{stdin} is used for input filenames: 
@example
@verbatim
echo in.nc | ncks            
echo in.nc | ncks -o out.nc
echo "in1.nc in2.nc" | ncbo -o out.nc
echo "in1.nc in2.nc" | ncflint -o out.nc
@end verbatim
@end example
For the provenance reasons dicussed above
(@pxref{Large Numbers of Files}), all filenames input via @code{stdin}
are stored as global attributes in the @ref{File List Attributes}).

@html
<a name="-o"></a> <!-- http://nco.sf.net/nco.html#-o -->
<a name="--output"></a> <!-- http://nco.sf.net/nco.html#--output -->
<a name="fl_out"></a> <!-- http://nco.sf.net/nco.html#fl_out -->
<a name="out"></a> <!-- http://nco.sf.net/nco.html#out -->
<a name="output"></a> <!-- http://nco.sf.net/nco.html#output -->
@end html
@node Specifying Output Files, Remote storage, Specifying Input Files, Shared features
@section Specifying Output Files
@cindex output file
@cindex input files
@cindex positional arguments
@cindex command line switches
@cindex @code{-o @var{fl_out}}
@cindex @code{--output @var{fl_out}}
@cindex @code{--fl_out @var{fl_out}}
@cartouche
Availability: All operators@*
Short options: @samp{-o}@*
Long options: @samp{--fl_out}, @samp{--output}@*
@end cartouche
@acronym{NCO} commands produce no more than one output file, @var{fl_out}. 
Traditionally, users specify @var{fl_out} as the final argument to the
operator, following all input file names. 
This is the @dfn{positional argument} method of specifying input and
ouput file names.
The positional argument method works well in most applications.
@acronym{NCO} also supports specifying @var{fl_out} using the command
line switch argument method, @samp{-o @var{fl_out}}.

Specifying @var{fl_out} with a switch, rather than as a positional
argument, allows @var{fl_out} to precede input files in the argument
list. 
@cindex multi-file operators
This is particularly useful with multi-file operators for three reasons.
Multi-file operators may be invoked with hundreds (or more) filenames.
Visual or automatic location of @var{fl_out} in such a list is
difficult when the only syntactic distinction between input and output
files is their position.
@cindex @command{xargs}
@cindex input files
Second, specification of a long list of input files may be difficult
(@pxref{Large Numbers of Files}).
Making the input file list the final argument to an operator facilitates 
using @command{xargs} for this purpose.
Some alternatives to @command{xargs} are heinous and undesirable.
Finally, many users are more comfortable specifying output files 
with @samp{-o @var{fl_out}} near the beginning of an argument list.
@cindex compilers
@cindex linkers
Compilers and linkers are usually invoked this way.

Users should specify @var{fl_out} using either (not both) method.
If @var{fl_out} is specified twice (once with the switch and once as
the last positional argument), then the positional argument takes
precedence. 

@html
<a name="rmt"></a> <!-- http://nco.sf.net/nco.html#rmt -->
@end html
@node Remote storage, Retaining Retrieved Files, Specifying Output Files, Shared features
@section Accessing Remote Files
@cindex @code{rcp}
@cindex @code{scp}
@cindex @file{.rhosts}
@cindex @acronym{NCAR MSS}
@cindex @acronym{MSS}
@cindex Mass Store System
@cindex @acronym{URL}
@cindex @code{ftp}
@cindex @code{sftp}
@cindex @code{wget}
@cindex remote files
@cindex synchronous file access
@cindex asynchronous file access
@cindex @code{--pth @var{input-path}}
@cindex @code{--path @var{input-path}}
@cindex @code{--lcl @var{output-path}}
@cindex @code{--local @var{output-path}}
@cindex @code{-l @var{output-path}}
@cindex @file{.netrc}
@cindex @code{history}
@cartouche
Availability: All operators@*
Short options: @samp{-p}, @samp{-l}@*
Long options: @samp{--pth}, @samp{--path}, @samp{--lcl}, @samp{--local}@*
@end cartouche
All @acronym{NCO} operators can retrieve files from remote sites as well 
as from the local file system.
@w{A remote} site can be an anonymous @acronym{FTP} server, a machine on
which the user has @command{rcp}, @command{scp}, or @command{sftp}
privileges, @acronym{NCAR}'s Mass Storage System (@acronym{MSS}), or
an @acronym{OPeNDAP} server.
Examples of each are given below, following a brief description of the 
particular access protocol.

@html
<a name="ftp"></a> <!-- http://nco.sf.net/nco.html#ftp -->
@end html
To access a file via an anonymous @acronym{FTP} server, simply supply
the remote file's @acronym{URL}.
Anonymous @acronym{FTP} usually requires no further credentials,
e.g., no @file{.netrc} file is necessary.
@acronym{FTP} is an intrinsically insecure protocol because it transfers
passwords in plain text format.  
Users should access sites using anonymous @acronym{FTP}, or better yet,
secure @acronym{FTP} (@acronym{SFTP}, see below) when possible. 
Some @acronym{FTP} servers require a login/password combination for a
valid user account.
@acronym{NCO} allows transactions that require additional credentials
so long as the required information is stored in the @file{.netrc} file.  
Usually this information is the remote machine name, login, and
password, in plain text, separated by those very keywords, e.g.,
@example
machine dust.ess.uci.edu login zender password bushlied
@end example
Eschew using valuable passwords for @acronym{FTP} transactions, since
@file{.netrc} passwords are potentially exposed to eavesdropping
software
@footnote{@acronym{NCO} does not implement command line options to
specify @acronym{FTP} logins and passwords because copying those data
into the @code{history} global attribute in the output file (done by
default) poses an unacceptable security risk. 
}. 

@html
<a name="sftp"></a> <!-- http://nco.sf.net/nco.html#sftp -->
@end html
@acronym{SFTP}, i.e., secure @acronym{FTP}, uses @acronym{SSH}-based 
security protocols that solve the security issues associated with
plain @acronym{FTP}.  
@acronym{NCO} supports @acronym{SFTP} protocol access to files
specified with a homebrew syntax of the form
@example
sftp://machine.domain.tld:/path/to/filename
@end example
Note the second colon following the top-level-domain, @code{tld}.
This syntax is a hybrid between an @acronym{FTP URL} and standard
remote file syntax.

@html
<a name="rcp"></a> <!-- http://nco.sf.net/nco.html#rcp -->
<a name="scp"></a> <!-- http://nco.sf.net/nco.html#scp -->
@end html
To access a file using @command{rcp} or @command{scp}, specify the
Internet address of the remote file.
Of course in this case you must have @command{rcp} or @command{scp}
privileges which allow transparent (no password entry required) access
to the remote machine. 
This means that @file{~/.rhosts} or @file{~/ssh/authorized_keys} must
be set accordingly on both local and remote machines.   

@cindex @acronym{HPSS}
@cindex @command{hsi}
@cindex @command{msrcp}
@cindex @command{msread}
@cindex @command{nrnet}
@html
<a name="hpss"></a> <!-- http://nco.sf.net/nco.html#hpss -->
<a name="HPSS"></a> <!-- http://nco.sf.net/nco.html#HPSS -->
<a name="hsi"></a> <!-- http://nco.sf.net/nco.html#hsi -->
<a name="HSI"></a> <!-- http://nco.sf.net/nco.html#HSI -->
<a name="msrcp"></a> <!-- http://nco.sf.net/nco.html#msrcp -->
<a name="msread"></a> <!-- http://nco.sf.net/nco.html#msread -->
<a name="nrnet"></a> <!-- http://nco.sf.net/nco.html#nrnet -->
@end html
To access a file on a High Performance Storage System (@acronym{HPSS}) 
(such as that at @acronym{NCAR}, @acronym{ECMWF}, @acronym{LANL},
@acronym{DKRZ}, @acronym{LLNL}) specify the full @acronym{HPSS} pathname
of the remote file and use the @samp{--hpss} flag.
Then @acronym{NCO} will attempt to detect whether the local machine has direct 
(synchronous) @acronym{HPSS} access. 
If so, @acronym{NCO} attempts to use the Hierarchical Storage
Interface (@acronym{HSI}) command @command{hsi get}
@footnote{The @command{hsi} command must be in the user's path in one of
the following directories: @code{/usr/local/bin}, @code{/opt/hpss/bin},
or @code{/ncar/opt/hpss/hsi}.
Tell us if the @acronym{HPSS} installation at your site places the
@command{hsi} command in a different location, and we will add that
location to the list of acceptable paths to search for @command{hsi}.
}.

The following examples show how one might analyze files stored on  
remote systems.
@c HPSS syntax at http://www2.cisl.ucar.edu/docs/hpss/hsi
@example
ncks -l . ftp://dust.ess.uci.edu/pub/zender/nco/in.nc
ncks -l . sftp://dust.ess.uci.edu:/home/ftp/pub/zender/nco/in.nc
ncks -l . dust.ess.uci.edu:/home/zender/nco/data/in.nc
ncks -l . /ZENDER/nco/in.nc # NCAR (broken old MSS path)
ncks -l . --hpss /home/zender/nco/in.nc # NCAR HPSS
ncks -l . http://thredds-test.ucar.edu/thredds/dodsC/testdods/in.nc 
@end example
@noindent
@c ncks -l . /home/z/zender/in.nc # NERSC
The first example works verbatim if your system is connected to the
Internet and is not behind a firewall. 
The second example works if you have @command{sftp} access to the
machine @code{dust.ess.uci.edu}.
The third example works if you have @command{rcp} or @command{scp}
access to the machine @code{dust.ess.uci.edu}. 
The fourth and fifth examples work on @acronym{NCAR} computers with
local access to the @acronym{HPSS} @command{hsi get} command
@footnote{@acronym{NCO} supported the old @acronym{NCAR} Mass Storage
System (@acronym{MSS}) until version 4.0.7 in April, 2011.
@acronym{NCO} supported @acronym{MSS}-retrievals via a variety of
mechanisms including the @command{msread}, @command{msrcp}, and
@command{nrnet} commands invoked either automatically or with sentinels
like @command{ncks -p mss:/ZENDER/nco -l . in.nc}.
Once the @acronym{MSS} was decommissioned in March, 2011, support for
these retrieval mechanisms was replaced by support for @acronym{HPSS}.
}.
The sixth command works if your local version of @acronym{NCO} is
@acronym{OPeNDAP}-enabled (this is fully described in @ref{OPeNDAP}),
or if the remote file is accessible via @command{wget}.
The above commands can be rewritten using the @samp{-p @var{input-path}} 
option as follows: 
@cindex @code{-p @var{input-path}}
@cindex @var{input-path}
@cindex @code{-l @var{output-path}}
@cindex @var{output-path}
@example
ncks -p ftp://dust.ess.uci.edu/pub/zender/nco -l . in.nc
ncks -p sftp://dust.ess.uci.edu:/home/ftp/pub/zender/nco -l . in.nc
ncks -p dust.ess.uci.edu:/home/zender/nco -l . in.nc
ncks -p /ZENDER/nco -l . in.nc
ncks -p /home/zender/nco -l . --hpss in.nc # HPSS
ncks -p http://thredds-test.ucar.edu/thredds/dodsC/testdods \ 
     -l . in.nc
@end example
@noindent
Using @samp{-p} is recommended because it clearly separates the
@var{input-path} from the filename itself, sometimes called the
@dfn{stub}. 
@cindex stub
When @var{input-path} is not explicitly specified using @samp{-p},
@acronym{NCO} internally generates an @var{input-path} from the first
input filename.  
The automatically generated @var{input-path} is constructed by stripping 
the input filename of everything following the final @samp{/} character
(i.e., removing the stub).
The @samp{-l @var{output-path}} option tells @acronym{NCO} where to
store the remotely retrieved file.
It has no effect on locally-retrieved files, or on the output file.
Often the path to a remotely retrieved file is quite different than the
path on the local machine where you would like to store the file.
If @samp{-l} is not specified then @acronym{NCO} internally generates an
@var{output-path} by simply setting @var{output-path} equal to
@var{input-path} stripped of any machine names.
If @samp{-l} is not specified and the remote file resides on a detected
@acronym{HPSS} system, then the leading character of
@var{input-path}, @samp{/}, is also stripped from @var{output-path}.
Specifying @var{output-path} as @samp{-l ./} tells @acronym{NCO} to
store the remotely retrieved file and the output file in the current
directory. 
Note that @samp{-l .} is equivalent to @samp{-l ./} though the latter is
syntactically more clear.

@menu
* OPeNDAP::
@end menu

@html
<a name="dap"></a> <!-- http://nco.sf.net/nco.html#dap -->
<a name="DAP"></a> <!-- http://nco.sf.net/nco.html#DAP -->
<a name="DODS"></a> <!-- http://nco.sf.net/nco.html#DODS -->
<a name="OPeNDAP"></a> <!-- http://nco.sf.net/nco.html#OPeNDAP -->
<a name="dods"></a> <!-- http://nco.sf.net/nco.html#dods -->
<a name="opendap"></a> <!-- http://nco.sf.net/nco.html#opendap -->
@end html
@node OPeNDAP,  , Remote storage, Remote storage
@subsection @acronym{OPeNDAP}
@cindex @acronym{DAP}
@cindex @acronym{DODS}
@cindex @acronym{HTTP} protocol
@cindex @env{DODS_ROOT}
@cindex Distributed Oceanographic Data System
@cindex oceanography
@cindex data access protocol
@cindex Open-source Project for a Network Data Access Protocol
@cindex @acronym{OPeNDAP}.
@cindex server
@cindex client-server
The Distributed Oceanographic Data System (@acronym{DODS}) provides
useful replacements for common data interface libraries like netCDF.
The @acronym{DODS} versions of these libraries implement network
transparent access to data via a client-server data access protocol
that uses the @acronym{HTTP} protocol for communication.
Although @acronym{DODS}-technology originated with oceanography data,
it applyies to virtually all scientific data.
In recognition of this, the data access protocol underlying
@acronym{DODS} (which is what @acronym{NCO} cares about) has been 
renamed the Open-source Project for a Network Data Access Protocol, 
@acronym{OPeNDAP}.
We use the terms @acronym{DODS} and @acronym{OPeNDAP} interchangeably,
and often write @acronym{OPeNDAP}/@acronym{DODS} for now. 
In the future we will deprecate @acronym{DODS} in favor of
@acronym{DAP} or @acronym{OPeNDAP}, as appropriate
@footnote{
@cindex @acronym{NVODS}
@cindex National Virtual Ocean Data System
@cindex open source
@acronym{DODS} is being deprecated because it is ambiguous, referring
both to a protocol and to a collection of (oceanography) data.
It is superceded by two terms.
@acronym{DAP} is the discipline-neutral Data Access Protocol at the
heart of @acronym{DODS}.
The National Virtual Ocean Data System (@acronym{NVODS}) refers to the
collection of oceanography data and oceanographic extensions to
@acronym{DAP}. 
In other words, @acronym{NVODS} is implemented with @acronym{OPeNDAP}.
@acronym{OPeNDAP} is @emph{also} the open source project which
maintains, develops, and promulgates the @acronym{DAP} standard. 
@acronym{OPeNDAP} and @acronym{DAP} really are interchangeable.
Got it yet?}.

@acronym{NCO} may be @acronym{DAP}-enabled by linking
@acronym{NCO} to the @acronym{OPeNDAP} libraries. 
@cindex @file{Makefile}
This is described in the @acronym{OPeNDAP} documentation and
automagically implemented in @acronym{NCO} build mechanisms
@footnote{
Automagic support for @acronym{DODS} version 3.2.x was deprecated in 
December, 2003 after @acronym{NCO} version 2.8.4.
@acronym{NCO} support for @acronym{OPeNDAP} versions 3.4.x commenced in
December, 2003, with @acronym{NCO} version 2.8.5.
@acronym{NCO} support for @acronym{OPeNDAP} versions 3.5.x commenced in
June, 2005, with @acronym{NCO} version 3.0.1.
@acronym{NCO} support for @acronym{OPeNDAP} versions 3.6.x commenced in
June, 2006, with @acronym{NCO} version 3.1.3.
@acronym{NCO} support for @acronym{OPeNDAP} versions 3.7.x commenced in
January, 2007, with @acronym{NCO} version 3.1.9.}.
The @file{./configure} mechanism automatically enables @acronym{NCO} as
@acronym{OPeNDAP} clients if it can find the required
@acronym{OPeNDAP} libraries.
Since about 2010 the netCDF library can be configured (with
@code{--enable-dap}) to build @acronym{DAP} directly into the netCDF
library, which @acronym{NCO} automatically links to, so @acronym{DAP}
need not be installed as a third-party library.
It has been so many years since @acronym{NCO} has needed to support
linking to @acronym{DAP} installed outside of the netCDF library that
is is unclear whether this configuration 
@footnote{
The minimal set of libraries required to build @acronym{NCO} as
@acronym{OPeNDAP} clients, where @acronym{OPeNDAP} is supplied as a
separate library apart from @file{libnetcdf.a}, are, in link order,
@file{libnc-dap.a}, @file{libdap.a}, and 
@file{libxml2} and @file{libcurl.a}.}.
still works.
The @env{$DODS_ROOT} environment variable may be used to override the 
default @acronym{OPeNDAP} library location at @acronym{NCO}
compile-time.  
Building @acronym{NCO} with @file{bld/Makefile} and the command
@code{make DODS=Y} adds the (non-intuitive) commands to link to the
@acronym{OPeNDAP} libraries installed in the @env{$DODS_ROOT}
directory.  
The file @file{doc/opendap.sh} contains a generic script intended to help
users install @acronym{OPeNDAP} before building @acronym{NCO}.
The documentation at the 
@uref{http://www.opendap.org, OPeNDAP Homepage}
is voluminous.
Check there and on the
@uref{http://www.unidata.ucar.edu/software/dods/home/mailLists/, DODS mail lists}.
to learn more about the extensive capabilities of @acronym{OPeNDAP}
@footnote{
We are most familiar with the @acronym{OPeNDAP} ability to enable 
network-transparent data access.
@cindex constraint expressions
@cindex server-side processing
@acronym{OPeNDAP} has many other features, including sophisticated
hyperslabbing and server-side processing via @dfn{constraint expressions}.
If you know more about this, please consider writing a section
on ``@acronym{OPeNDAP} Capabilities of Interest to @acronym{NCO} Users''
for incorporation in the @cite{NCO User Guide}.}.

Once @acronym{NCO} is @acronym{DAP}-enabled the operators are
@acronym{OPeNDAP} clients.  
All @acronym{OPeNDAP} clients have network transparent access to
any files controlled by a @acronym{OPeNDAP} server. 
Simply specify the input file path(s) in @acronym{URL} notation and all 
@acronym{NCO} operations may be performed on remote files made
accessible by a @acronym{OPeNDAP} server. 
This command tests the basic functionality of @acronym{OPeNDAP}-enabled  
@acronym{NCO} clients: 
@example
% ncks -O -o ~/foo.nc -C -H -v one -l /tmp \
  -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc
% ncks -H -v one ~/foo.nc
one = 1
@end example
The @code{one = 1} outputs confirm (first) that @command{ncks} correctly
retrieved data via the  @acronym{OPeNDAP} protocol and (second) that 
@command{ncks} created a valid local copy of the subsetted remote file.
With minor changes to the above command, netCDF4 can be used as both the
input and output file format:
@example
% ncks -4 -O -o ~/foo.nc -C -H -v one -l /tmp \
  -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in_4.nc
% ncks -H -v one ~/foo.nc
one = 1
@end example
And, of course, @acronym{OPeNDAP}-enabled @acronym{NCO} clients continue
to support orthogonal features such as UDUnits 
(@pxref{UDUnits Support}):
@example
% ncks -u -C -H -v wvl -d wvl,'0.4 micron','0.7 micron' \
  -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in_4.nc
% wvl[0]=5e-07 meter
@end example

The next command is a more advanced example which demonstrates the real
power of @acronym{OPeNDAP}-enabled @acronym{NCO} clients.
The @command{ncwa} client requests an equatorial hyperslab from remotely
stored @acronym{NCEP reanalyses data} of the @w{year 1969}.
The @acronym{NOAA} @acronym{OPeNDAP} server (hopefully!) serves these data. 
The local @command{ncwa} client then computes and stores (locally) the
regional mean surface pressure @w{(in Pa)}. 
@example
ncwa -O -C -a lat,lon,time -d lon,-10.,10. -d lat,-10.,10. \
http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis.dailyavgs/surface/pres.sfc.1969.nc ~/foo.nc
@end example
@noindent
@cindex packing
@cindex unpacking
All with one command!
The data in this particular input file also happen to be packed
(@pxref{Methods and functions}), although this complication is
transparent to the user since @acronym{NCO} automatically unpacks data
before attempting arithmetic. 

@acronym{NCO} obtains remote files from the @acronym{OPeNDAP} server
(e.g., @file{www.cdc.noaa.gov}) rather than the local machine. 
Input files are first copied to the local machine, then processed.
The @acronym{OPeNDAP} server performs data access, hyperslabbing,
and transfer to the local machine.
@cindex I/O
This allows the I/O to appear to @acronym{NCO} as if the input files
were local.  
The local machine performs all arithmetic operations.
Only the hyperslabbed output data are transferred over the network (to
the local machine) for the number-crunching to begin.
The advantages of this are obvious if you are examining small parts of
large files stored at remote locations.

Natually there are many versions of @acronym{OPeNDAP} servers supplying
data and bugs in the server can appear to be bugs in @acronym{NCO}.
However, with very few exceptions
@footnote{For example, @acronym{DAP} servers do not like variables with
periods (``.'') in their names even though this is perfectly legal with
netCDF. 
Such names may cause the @acronym{DAP} service to fail because 
@acronym{DAP} interprets the period as structure delimiter in an 
@acronym{HTTP} query string.} an @acronym{NCO} command that works
on a local file must work across an @acronym{OPeNDAP} connection or else 
there is a bug in the server. 
This is because @acronym{NCO} does nothing special to handle files
served by @acronym{OPeNDAP}, the whole process is (supposed to be)
completely transparent to the client @acronym{NCO} software.
Therefore it is often useful to try @acronym{NCO} commands on various
@acronym{OPeNDAP} servers in order to isolate whether a problem may be
due to a bug in the @acronym{OPeNDAP} server on a particular machine.
For this purpose, one might try variations of the following commands
that access files on public @acronym{OPeNDAP} servers:
@example
# Strided access to HDF5 file
ncks -v Time -d Time,0,10,2 http://eosdap.hdfgroup.uiuc.edu:8080/opendap/data/NASAFILES/hdf5/BUV-Nimbus04_L3zm_v01-00-2012m0203t144121.h5
# Strided access to netCDF3 file
ncks -O -D 1 -d time,1 -d lev,0 -d lat,0,100,10 -d lon,0,100,10 -v u_velocity http://nomads.ncep.noaa.gov:9090/dods/rtofs/rtofs_global20140303/rtofs_glo_2ds_forecast_daily_prog ~/foo.nc
@end example
@noindent
These servers were operational at the time of writing, March 2014.
Unfortunately, administrators often move or rename path directories.
Recommendations for additional public @acronym{OPeNDAP} servers on
which to test @acronym{NCO} are welcome.

@html
<a name="rtn"></a> <!-- http://nco.sf.net/nco.html#rtn -->
@end html
@node Retaining Retrieved Files, File Formats and Conversion, Remote storage, Shared features
@section Retaining Retrieved Files
@cindex file deletion
@cindex file removal
@cindex file retention
@cindex @code{-R}
@cindex @code{--rtn}
@cindex @code{--retain}
@cartouche
Availability: All operators@*
Short options: @samp{-R}@*
Long options: @samp{--rtn}, @samp{--retain}@*
@end cartouche
In order to conserve local file system space, files retrieved from
remote locations are automatically deleted from the local file system 
once they have been processed.
Many @acronym{NCO} operators were constructed to work with numerous
large (e.g., @w{200 MB}) files. 
Retrieval of multiple files from remote locations is done serially. 
Each file is retrieved, processed, then deleted before the cycle
repeats.  
In cases where it is useful to keep the remotely-retrieved files on the
local file system after processing, the automatic removal feature may be 
disabled by specifying @samp{-R} on the command line.

Invoking @code{-R} disables the default printing behavior of
@command{ncks}.
This allows @command{ncks} to retrieve remote files without
automatically trying to print them.
See @ref{ncks netCDF Kitchen Sink}, for more details.

@cindex @acronym{FTP}
@cindex @acronym{SSH}
@cindex @acronym{msrcp}
Note that the remote retrieval features of @acronym{NCO} can always be
used to retrieve @emph{any} file, including non-netCDF files, via
@command{SSH}, anonymous @acronym{FTP}, or @command{msrcp}.
Often this method is quicker than using a browser, or running an
@acronym{FTP} session from a shell window yourself.
@cindex server
For example, say you want to obtain a @acronym{JPEG} file from a weather
server. 
@example
ncks -R -p ftp://weather.edu/pub/pix/jpeg -l . storm.jpg
@end example
@noindent
In this example, @command{ncks} automatically performs an anonymous
@acronym{FTP} login to the remote machine and retrieves the specified
file. 
When @command{ncks} attempts to read the local copy of @file{storm.jpg}
as a netCDF file, it fails and exits, leaving  @file{storm.jpg} in
the current directory.

@cindex @acronym{DODS}
@cindex server
If your @acronym{NCO} is @acronym{DAP}-enabled (@pxref{OPeNDAP}),
then you may use @acronym{NCO} to retrieve any files (including netCDF,
@acronym{HDF}, etc.) served by an @acronym{OPeNDAP} server to your local 
machine. 
For example, 
@example
ncks -R -l . -p \
http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis.dailyavgs/surface \
  pres.sfc.1969.nc
@end example
It may occasionally be useful to use @acronym{NCO} to transfer files
when your other preferred methods are not available locally.

@html
<a name="conversion"></a> <!-- http://nco.sf.net/nco.html#conversion -->
<a name="fl_fmt"></a> <!-- http://nco.sf.net/nco.html#fl_fmt -->
<a name="hdf"></a> <!-- http://nco.sf.net/nco.html#hdf -->
<a name="cdf5"></a> <!-- http://nco.sf.net/nco.html#cdf5 -->
<a name="64bit"></a> <!-- http://nco.sf.net/nco.html#64bit -->
<a name="64bit-data"></a> <!-- http://nco.sf.net/nco.html#64bit-data -->
<a name="64bit-offset"></a> <!-- http://nco.sf.net/nco.html#64bit-offset -->
<a name="netcdf4"></a> <!-- http://nco.sf.net/nco.html#netcdf4 -->
@end html
@node File Formats and Conversion, Zarr and NCZarr, Retaining Retrieved Files, Shared features
@section File Formats and Conversion
@cindex @acronym{HDF}
@cindex CDF5
@cindex netCDF2
@cindex netCDF3
@cindex netCDF4
@cindex @code{NETCDF4_CLASSIC} files
@cindex @code{NETCDF4} files
@cindex @code{CLASSIC} files
@cindex @code{64BIT_OFFSET} files
@cindex @code{64BIT_DATA} files
@cindex @code{CDF5} files
@cindex @code{--3}
@cindex @code{-3}
@cindex @code{-4}
@cindex @code{-5}
@cindex @code{-6}
@cindex @code{-7}
@cindex @code{--4}
@cindex @code{--5}
@cindex @code{--6}
@cindex @code{--7}
@cindex @code{--netcdf4}
@cindex @code{--fl_fmt}
@cindex @code{--file_format}
@cindex @code{--64bit_offset}
@cindex @code{--64bit_data}
@cartouche
Availability: @command{ncap2}, @command{nces},
@command{ncecat}, @command{ncflint}, @command{ncks}, @command{ncpdq},
@command{ncra}, @command{ncrcat}, @command{ncwa}@*
Short options: @samp{-3}, @samp{-4}, @samp{-5}, @samp{-6}, @samp{-7}@*
Long options: @samp{--3}, @samp{--4}, @samp{--5}, @samp{--6}, @samp{--64bit_offset}, @samp{--7}, @samp{--fl_fmt},
@samp{--netcdf4}@*  
@end cartouche
All @acronym{NCO} operators support (read and write) all three (or four, 
depending on how one counts) file formats supported by netCDF4.
The default output file format for all operators is the input file
format. 
The operators listed under ``Availability'' above allow the user to
specify the output file format independent of the input file format. 
These operators allow the user to convert between the various file
formats. 
(The operators @command{ncatted} and @command{ncrename} do not support
these switches so they always write the output netCDF file in the same
format as the input netCDF file.) 

@menu
* File Formats::
* Determining File Format::
* File Conversion::
* Autoconversion::
@end menu

@node File Formats, Determining File Format, File Formats and Conversion, File Formats and Conversion
@subsection File Formats
netCDF supports five types of files: @code{CLASSIC},
@code{64BIT_OFFSET}, @code{64BIT_DATA}, @code{NETCDF4}, and
@code{NETCDF4_CLASSIC}.
The @code{CLASSIC} (aka @code{CDF1}) format is the traditional 32-bit offset written by netCDF2 and netCDF3. 
As of 2005, nearly all netCDF datasets were in @code{CLASSIC} format. 
The @code{64BIT_OFFSET} (originally called plain old @code{64BIT})
(aka @code{CDF2}) format was added in Fall, 2004. 
As of 2010, many netCDF datasets were in @code{64BIT_OFFSET} format. 
As of 2013, an increasing number of netCDF datasets were in
@code{NETCDF4_CLASSIC} format.
The @code{64BIT_DATA} (aka @code{CDF5} or @code{PNETCDF})
format was added to netCDF in January, 2016 and immediately supported
by @acronym{NCO}.
Support for @ref{Zarr and NCZarr} backend storage formats was added to  
netCDF in 2021 and supported by @acronym{NCO} in 2022.

The @code{NETCDF4} format uses @acronym{HDF5} as the file storage layer. 
The files are (usually) created, accessed, and manipulated using the 
traditional netCDF3 @acronym{API} (with numerous extensions).
The @code{NETCDF4_CLASSIC} format refers to netCDF4 files created with
the @code{NC_CLASSIC_MODEL} mask.
Such files use @acronym{HDF5} as the back-end storage format (unlike
netCDF3), though they incorporate only netCDF3 features.
Hence @code{NETCDF4_CLASSIC} files are entirely readable by applications 
that use only the netCDF3 @acronym{API} (though the applications must be
linked with the netCDF4 library).
@acronym{NCO} must be built with @w{netCDF4} to write files in the new
@code{NETCDF4} and @code{NETCDF4_CLASSIC} formats, and to read files in
these formats. 
Datasets in the default @code{CLASSIC} or the newer @code{64BIT_OFFSET}
formats have maximum backwards-compatibility with older applications. 
@acronym{NCO} has deep support for @code{NETCDF4} formats.
If backwards compatibility is important, and your datasets are too large
for netCDF3, use @code{NETCDF4_CLASSIC} instead of @code{CLASSIC} format
files.  
@acronym{NCO} support for the @code{NETCDF4} format is complete and many
high-performance disk/@acronym{RAM} efficient workflows utilize this
format.   

As mentioned above, all operators write use the input file format for
output files unless told otherwise.
Toggling the short option @samp{-6} or the long option @samp{--6} or
@samp{--64bit_offset} (or their @var{key}-@var{value} equivalent
@samp{--fl_fmt=64bit_offset}) produces the netCDF3 64-bit offset format 
named @code{64BIT_OFFSET}.
@acronym{NCO} must be built with @w{netCDF 3.6} or higher to produce
a @code{64BIT_OFFSET} file.
As of @acronym{NCO} version 4.6.9 (September, 2017), toggling the short
option @samp{-5} or the long options @samp{--5}, @samp{--64bit_data},
@samp{--cdf5}, or @samp{--pnetcdf} (or their @var{key}-@var{value}
equivalent @samp{--fl_fmt=64bit_data}) produces the netCDF3 64-bit data
format named @code{64BIT_DATA}. 
This format is widely used by @acronym{MPI}-enabled modeling codes
because of its long association with PnetCDF.
@acronym{NCO} must be built with @w{netCDF 4.4} or higher to produce
a @code{64BIT_DATA} file.

Using the @samp{-4} switch (or its long option equivalents
@samp{--4} or @samp{--netcdf4}), or setting its @var{key}-@var{value}
equivalent @samp{--fl_fmt=netcdf4} produces a @code{NETCDF4} file
(i.e., with all supported @acronym{HDF5} features).
Using the @samp{-7} switch (or its long option equivalent
@samp{--7}
@footnote{
The reason (and mnemonic) for @samp{-7} is that @code{NETCDF4_CLASSIC}
files include great features of both netCDF3 (compatibility) and
netCDF4 (compression, chunking) and, well, @math{3+4=7}.}, or 
setting its @var{key}-@var{value} equivalent
@samp{--fl_fmt=netcdf4_classic} produces a @code{NETCDF4_CLASSIC}  
file (i.e., with all supported @acronym{HDF5} features like compression
and chunking but without groups or new atomic types).
Operators given the @samp{-3} (or @samp{--3}) switch without arguments
will (attempt to) produce netCDF3 @code{CLASSIC} output, even from
netCDF4 input files.  

Note that @code{NETCDF4} and @code{NETCDF4_CLASSIC} are the same
binary format. 
The latter simply causes a writing application to fail if it attempts to 
write a @code{NETCDF4} file that cannot be completely read by the
netCDF3 library. 
Conversely, @code{NETCDF4_CLASSIC} indicates to a reading application
that all of the file contents are readable with the netCDF3 library. 
@acronym{NCO} has supported reading/writing basic @code{NETCDF4} and
@code{NETCDF4_CLASSIC} files since October, 2005.

@html
<a name="fmt_inq"></a> <!-- http://nco.sf.net/nco.html#fmt_inq -->
@end html
@node Determining File Format, File Conversion, File Formats, File Formats and Conversion
@subsection Determining File Format
Input files often end with the generic @code{.nc} suffix that leaves
(perhaps by intention) the internal file format ambiguous.
There are at least three ways to discover the internal format of a
netCDF-supported file.
These methods determine whether it is a classic (32-bit offset) or newer
64-bit offset netCDF3 format, or is a netCDF4 format. 
Each method returns the information using slightly different terminology 
that becomes easier to understand with practice.

First, examine the first line of global metadata output by @samp{ncks -M}:  
@cindex netCDF3 classic file format
@cindex netCDF4 classic file format
@cindex netCDF4 file format
@cindex 32-bit offset file format
@cindex 64-bit offset file format
@cindex 64-bit data file format
@cindex PnetCDF file format
@cindex @command{ncks} 
@cindex @code{-M}
@example
% ncks -M foo_3.nc
Summary of foo_3.nc: filetype = NC_FORMAT_CLASSIC, 0 groups ...
% ncks -M foo_6.nc
Summary of foo_6.nc: filetype = NC_FORMAT_64BIT_OFFSET, 0 groups ...
% ncks -M foo_5.nc
Summary of foo_5.nc: filetype = NC_FORMAT_CDF5, 0 groups ...
% ncks -M foo_7.nc
Summary of foo_7.nc: filetype = NC_FORMAT_NETCDF4_CLASSIC, 0 groups ...
% ncks -M foo_4.nc
Summary of foo_4.nc: filetype = NC_FORMAT_NETCDF4, 0 groups ...
@end example
This method requires a netCDF4-enabled @acronym{NCO} version 3.9.0+
(i.e., from 2007 or later).
@cindex extended file format
@cindex underlying file format
@cindex @code{NC_FORMATX_NC3}
@cindex @code{NC_FORMATX_NC_HDF5}
@cindex @code{NC_FORMATX_NC_HDF4}
@cindex @code{NC_FORMATX_PNETCDF}
@cindex @code{NC_FORMATX_DAP2}
@cindex @code{NC_FORMATX_DAP4}
As of @acronym{NCO} version 4.4.0 (January, 2014), @command{ncks} will
also print the extended or underlying format of the input file.
The extended filetype will be one of the six underlying formats that
are accessible through the netCDF @acronym{API}.
These formats are
@code{NC_FORMATX_NC3} (classic and 64-bit versions of netCDF3 formats),
@code{NC_FORMATX_NC_HDF5} (classic and extended versions of netCDF4, and
``pure'' HDF5 format),
@code{NC_FORMATX_NC_HDF4} (HDF4 format),
@code{NC_FORMATX_PNETCDF} (PnetCDF format),
@code{NC_FORMATX_DAP2} (accessed via DAP2 protocol), and
@code{NC_FORMATX_DAP4} (accessed via DAP4 protocol).
For example,
@example
% ncks -D 2 -M hdf.hdf
Summary of hdf.hdf: filetype = NC_FORMAT_NETCDF4 (representation of \
  extended/underlying filetype NC_FORMAT_HDF4), 0 groups ...
% ncks -D 2 -M http://thredds-test.ucar.edu/thredds/dodsC/testdods/in.nc
Summary of http://thredds-test.ucar.edu/thredds/dodsC/testdods/in.nc: \
  filetype = NC_FORMAT_CLASSIC (representation of extended/underlying \
  filetype NC_FORMATX_DAP2), 0 groups  
% ncks -D 2 -M foo_4.nc
Summary of foo_4.nc: filetype = NC_FORMAT_NETCDF4 (representation of \
  extended/underlying filetype NC_FORMAT_HDF5), 0 groups  
@end example
The extended filetype determines some of the capabilities that netCDF
has to alter the file.

Second, query the file with @samp{ncdump -k}:
@cindex @command{ncdump} 
@example
% ncdump -k foo_3.nc
classic
% ncdump -k foo_6.nc
64-bit offset
% ncdump -k foo_5.nc
cdf5
% ncdump -k foo_7.nc
netCDF-4 classic model
% ncdump -k foo_4.nc
netCDF-4
@end example
This method requires a netCDF4-enabled @acronym{netCDF} 3.6.2+
(i.e., from 2007 or later).

The third option uses the POSIX-standard @command{od} (octal dump)
command:   
@cindex @command{od} 
@cindex octal dump
@example
% od -An -c -N4 foo_3.nc
   C   D   F 001
% od -An -c -N4 foo_6.nc
   C   D   F 002
% od -An -c -N4 foo_5.nc
   C   D   F 005
% od -An -c -N4 foo_7.nc
 211   H   D   F
% od -An -c -N4 foo_4.nc
 211   H   D   F
@end example
This option works without @acronym{NCO} and @command{ncdump}.
Values of @samp{C D F 001} and @samp{C D F 002} indicate 32-bit
(classic) and 64-bit netCDF3 formats, respectively, while values of
@samp{211 H D F} indicate either of the newer netCDF4 file formats.

@html
<a name="hdf2nc"></a> <!-- http://nco.sf.net/nco.html#hdf2nc -->
<a name="3to4"></a> <!-- http://nco.sf.net/nco.html#3to4 -->
<a name="4to3"></a> <!-- http://nco.sf.net/nco.html#4to3 -->
@end html
@node File Conversion, Autoconversion, Determining File Format, File Formats and Conversion
@subsection File Conversion
Let us demonstrate converting a file from any netCDF-supported
input format into any netCDF output format (subject to limits of the
output format).  
Here the input file @file{in.nc} may be in any of these formats:
netCDF3 (classic, 64bit_offset, 64bit_data), netCDF4 (classic and
extended), HDF4, HDF5, HDF-EOS (version 2 or 5), and DAP. 
The switch determines the output format written in the comment:
@footnote{The switches @samp{-5}, @samp{--5}, and @samp{pnetcdf} are
reserved for PnetCDF files, i.e., @code{NC_FORMAT_CDF5}.
Such files are similar to netCDF3 classic files, yet also support
64-bit offsets and the additional netCDF4 atomic types.}
@example
ncks --fl_fmt=classic in.nc foo_3.nc # netCDF3 classic
ncks --fl_fmt=64bit_offset in.nc foo_6.nc # netCDF3 64bit-offset
ncks --fl_fmt=64bit_data in.nc foo_5.nc # netCDF3 64bit-data
ncks --fl_fmt=cdf5 in.nc foo_5.nc # netCDF3 64bit-data
ncks --fl_fmt=netcdf4_classic in.nc foo_7.nc # netCDF4 classic
ncks --fl_fmt=netcdf4 in.nc foo_4.nc # netCDF4 
ncks -3 in.nc foo_3.nc # netCDF3 classic
ncks --3 in.nc foo_3.nc # netCDF3 classic
ncks -6 in.nc foo_6.nc # netCDF3 64bit-offset
ncks --64 in.nc foo_6.nc # netCDF3 64bit-offset
ncks -5 in.nc foo_5.nc # netCDF3 64bit-data
ncks --5 in.nc foo_5.nc # netCDF3 64bit-data
ncks -4 in.nc foo_4.nc # netCDF4 
ncks --4 in.nc foo_4.nc # netCDF4 
ncks -7 in.nc foo_7.nc # netCDF4 classic
ncks --7 in.nc foo_7.nc # netCDF4 classic
@end example
Of course since most operators support these switches, the
``conversions'' can be done at the output stage of arithmetic
or metadata processing rather than requiring a separate step.
Producing (netCDF3) @code{CLASSIC} or @code{64BIT_OFFSET} or
@code{64BIT_DATA} files from @code{NETCDF4_CLASSIC} files always 
works.  

@html
<a name="autoconversion"></a> <!-- http://nco.sf.net/nco.html#autoconversion -->
<a name="autocnv"></a> <!-- http://nco.sf.net/nco.html#autocnv -->
@end html
@node Autoconversion,  , File Conversion, File Formats and Conversion
@subsection Autoconversion
Because of the dearth of support for netCDF4 amongst tools and user
communities (including the @acronym{CF} conventions), it is often useful
to convert netCDF4 to netCDF3 for certain applications.
Until @acronym{NCO} version 4.4.0 (January, 2014), producing netCDF3
files from netCDF4 files only worked if the input files contained no 
netCDF4-specific features (e.g., atomic types, multiple record
dimensions, or groups). 
As of @acronym{NCO} version 4.4.0, @command{ncks} supports
@dfn{autoconversion} of many netCDF4 features to their closest
netCDF3-compatible representations.
Since converting netCDF4 to netCDF3 results in loss of features, 
``automatic down-conversion'' may be a more precise description of what  
we term autoconversion.

@acronym{NCO} employs three algorithms to downconvert netCDF4 to netCDF3:
@enumerate
@item 
@cindex autoconversion
Autoconversion of atomic types:
Autoconversion automatically promotes @code{NC_UBYTE} to @code{NC_SHORT}, 
and @code{NC_USHORT} to @code{NC_INT}.
It automatically demotes the three types @code{NC_UINT},
@code{NC_UINT64}, and @code{NC_INT64} to @code{NC_INT}. 
And it converts @code{NC_STRING} to @code{NC_CHAR}.
All numeric conversions work for attributes and variables of any rank.
Two numeric types (@code{NC_UBYTE} and @code{NC_USHORT}) are
@emph{promoted} to types with greater range (and greater storage). 
This extra range is often not used so promotion perhaps conveys
the wrong impression.
However, promotion never truncates values or loses data (this perhaps
justifies the extra storage).
Three numeric types (@code{NC_UINT}, @code{NC_UINT64} and
@code{NC_INT64}) are @emph{demoted}.
Since the input range is larger than the output range, demotion can
result in numeric truncation and thus loss of data. 
In such cases, it would possible to convert the data to floating-point
values instead. 
If this feature interests you, please be the squeaky wheel and let us
know.

String conversions (to @code{NC_CHAR}) work for all attributes, but
not for variables.
This is because attributes are at most one-dimensional and may be of any
size whereas variables require gridded dimensions that usually do not
fit the ragged sizes of text strings.
Hence scalar @code{NC_STRING} attributes are correctly converted to and
stored as @code{NC_CHAR} attributes in the netCDF3 output file, but
@code{NC_STRING} variables are not correctly converted. 
If this limitation annoys or enrages you, please let us know by being
the squeaky wheel.

@item
@cindex @code{--fix_rec_dmn all}
Convert multiple record dimensions to fixed-size dimensions.
Many netCDF4 and @acronym{HDF5} datasets have multiple unlimited
dimensions.
Since a netCDF3 file may have at most one unlimited dimension, all but
possibly one unlimited dimension from the input file must be converted
to fixed-length dimensions prior to storing netCDF4 input as netCDF3
output.
By invoking @code{--fix_rec_dmn all} the user ensures the output file
will adhere to netCDF3 conventions and the user need not know the names
of the specific record dimensions to fix.
See @ref{ncks netCDF Kitchen Sink} for a description of the
@samp{--fix_rec_dmn} option. 

@item
@cindex flattening
Flattening (removal) of groups.
Many netCDF4 and @acronym{HDF5} datasets have group hierarchies.
Since a netCDF3 file may not have any groups, groups in the input file
must be removed.
This is also called ``flattening'' the hierarchical file.
See @ref{Group Path Editing} for a description of the @acronym{GPE}
option @samp{-G :} to flatten files.
@end enumerate

Putting the three algorithms together, one sees that the recipe to 
convert netCDF4 to netCDF4 becomes increasingly complex as the netCDF4
features in the input file become more elaborate:
@example
# Convert file with netCDF4 atomic types
ncks -3 in.nc4 out.nc3
# Convert file with multiple record dimensions + netCDF4 atomic types
ncks -3 --fix_rec_dmn=all in.nc4 out.nc3
# Convert file with groups, multiple record dimensions + netCDF4 atomic types
ncks -3 -G : --fix_rec_dmn=all in.nc4 out.nc3
@end example
Future versions of @acronym{NCO} may automatically invoke the record
dimension fixation and group flattening when converting to netCDF3
(rather than requiring it be specified manually).
If this feature would interest you, please let us know.

@html
<a name="Zarr"></a> <!-- http://nco.sf.net/nco.html#Zarr -->
<a name="NCZarr"></a> <!-- http://nco.sf.net/nco.html#NCZarr -->
<a name="zarr"></a> <!-- http://nco.sf.net/nco.html#zarr -->
<a name="nczarr"></a> <!-- http://nco.sf.net/nco.html#nczarr -->
<a name="conversion"></a> <!-- http://nco.sf.net/nco.html#conversion -->
<a name="zarr"></a> <!-- http://nco.sf.net/nco.html#ncz -->
<a name="ncz2psx"></a> <!-- http://nco.sf.net/nco.html#ncz2psx -->
<a name="fl_fmt"></a> <!-- http://nco.sf.net/nco.html#fl_fmt -->
@end html
@cindex Zarr
@cindex NCZarr
@cindex @command{ncz2psx}
@cindex @acronym{POSIX}
@cindex @acronym{S3}
@cindex @code{stdin}
@node Zarr and NCZarr, Large File Support, File Formats and Conversion, Shared features
@section Zarr and NCZarr
@cartouche
Availability: All Operators@*
@end cartouche
As of @w{version 5.1.1} (November 2022), all @acronym{NCO} operators
support NCZarr I/O.
This support is currently limited to the @code{file://} scheme.
Support for the @acronym{S3} scheme is next.
All @acronym{NCO} commands should work as expected independent of the
back-end storage format of the I/O.
Operators can ingest and output @code{POSIX} or Zarr backend file
formats:
@example
@verbatim
in_ncz="file://${HOME}/in_zarr4#mode=nczarr,file"
in_psx="${HOME}/in_zarr4.nc"
out_ncz="file://${HOME}/foo#mode=nczarr,file"
out_psx="${HOME}/foo.nc"

ncks ${in_ncz} # Print contents of Zarr file
ncks -O -v var ${in_psx} ${out_psx} # POSIX input to POSIX output
ncks -O -v var ${in_psx} ${out_ncz} # POSIX input to Zarr output
ncks -O -v var ${in_ncz} ${out_psx} # Zarr input to  POSIX output
ncks -O -v var ${in_ncz} ${out_ncz} # Zarr input to Zarr output
ncks -O --cmp='gbr|shf|zst' ${in_psx} ${out_ncz} # Quantize/Compress
ncks -O --cmp='gbr|shf|zst' ${in_ncz} ${out_ncz} # Quantize/Compress
@end verbatim
@end example
Note that NCZarr only supports the netCDF4 (not netCDF3) data model.
This is because NCZarr needs to know chunking and compression
information by default (it is not optional).
Hence if the input format is netCDF3, then the user must explicitly
specify a netCDF4 format for the output NCZarr storage:
@example
@verbatim
in_psx="${HOME}/in_zarr3.nc" # As above, but a netCDF3 input file

ncks -O -4 ${in_psx} ${out_ncz} # netCDF3 POSIX input to Zarr output
ncks -O -7 ${in_psx} ${out_ncz} # netCDF3 POSIX input to Zarr output
@end verbatim
@end example
Furthermore, the current NCZarr library (version 4.9.2, March 2023)
does not yet support record dimensions (this is a significant and high
priority netCDF library limitation, not an @acronym{NCO} limitation).
All dimensions must be fixed, not record
To workaround this limitation, first fix any record dimensions with,
e.g., 
@example
@verbatim
ncks -O --fix_rec_dmn=all ${in_psx} ${in_psx}
@end verbatim
@end example

Commands with Zarr I/O behave mostly as expected.
@acronym{NCO} treats Zarr and @acronym{POSIX} files identically once
they are opened via the netCDF @acronym{API}.
The main difference between Zarr and @acronym{POSIX}, from the
viewpoint of @acronym{NCO}, is in handling the filenames.
By default @acronym{NCO} performs operations in temporary files that
it moves to a final destination once the rest of the command
succeeds.
Supporting Zarr in @acronym{NCO} requires applying the correct
procedures to create, copy, move/rename, and delete files and
directories correctly depending on the backend format.

Many @acronym{NCO} users rely on @acronym{POSIX} filename globbing for
multi-file operations, e.g., @samp{ncra in*.nc out.nc}.
Globbing returns matches in @acronym{POSIX} format (e.g.,
@code{in1.nc in2.nc in3.nc}) which lacks the @code{scheme://}
indicator and the @code{#mode=...} fragment that the netCDF
@acronym{API} needs to open a Zarr store.
There is no perfect solution to this.

A partial solution is available by judiciously using @acronym{NCO}'s
new @code{stdin} capabilities for all operators
(@pxref{Specifying Input Files}).
The procedure uses the @command{ls} command (instead of globbing) to
identify the desired Zarr stores, and pipes the
(@acronym{POSIX}-style) results of that through the newly supplied 
@acronym{NCO} filter-script @command{ncz2psx} that will prepend the
desired scheme and append the desired fragment to the matched Zarr
stores, and pipe those results onward to an @acronym{NCO} operator: 
@example
@verbatim
nces in*.nc out.nc      # POSIX input files via globbing
ls in*.nc | nces out.nc # POSIX input files via stdin
ls -d in* | ncz2psx | nces out.nc # Zarr input via stdin
ls -d in* | ncz2psx --scheme=file --mode=nczarr,file | nces out.nc
@end verbatim
@end example

@html
<a name="lfs"></a> <!-- http://nco.sf.net/nco.html#lfs -->
@end html
@node Large File Support, Subsetting Files, Zarr and NCZarr, Shared features
@section Large File Support
@cindex @acronym{LFS}
@cindex Large File Support
@cartouche
Availability: All operators@*
Short options: none@*
Long options: none@*
@end cartouche
@acronym{NCO} has Large File Support (@acronym{LFS}), meaning that 
@acronym{NCO} can write files larger than @w{2 GB} on some 32-bit
operating systems with netCDF libraries earlier than @w{version 3.6}. 
If desired, @acronym{LFS} support must be configured when both netCDF and
@acronym{NCO} are installed.
netCDF @w{versions 3.6} and higher support 64-bit file addresses as part
of the netCDF standard.
We recommend that users ignore @acronym{LFS} support which is difficult to
configure and is implemented in @acronym{NCO} only to support netCDF
versions prior @w{to 3.6}.  
This obviates the need for configuring explicit @acronym{LFS} support in
applications (such as @acronym{NCO}) that now support 64-bit files   
directly through the netCDF interface.
See @ref{File Formats and Conversion} for instructions on accessing 
the different file formats, including 64-bit files, supported by the
modern netCDF interface. 

If you are still interested in explicit @acronym{LFS} support for netCDF versions
prior @w{to 3.6}, know that @acronym{LFS} support depends on a complex,
interlocking set of operating system  
@footnote{
Linux and @acronym{AIX} do support @acronym{LFS}.}
and netCDF support issues.
The netCDF @acronym{LFS} 
@uref{http://my.unidata.ucar.edu/content/software/netcdf/faq-lfs.html,FAQ}
describes the various file size limitations imposed by different
versions of the netCDF standard.
@acronym{NCO} and netCDF automatically attempt to configure @acronym{LFS} at
build time. 

@html
<a name="x"></a> <!-- http://nco.sf.net/nco.html#x -->
<a name="grp"></a> <!-- http://nco.sf.net/nco.html#grp -->
<a name="var"></a> <!-- http://nco.sf.net/nco.html#var -->
<a name="xcl"></a> <!-- http://nco.sf.net/nco.html#xcl -->
<a name="exclude"></a> <!-- http://nco.sf.net/nco.html#exclude -->
<a name="sbs"></a> <!-- http://nco.sf.net/nco.html#sbs -->
<a name="subset"></a> <!-- http://nco.sf.net/nco.html#subset -->
@end html
@node Subsetting Files, Subsetting Coordinate Variables, Large File Support, Shared features
@section Subsetting Files
@cindex subsetting
@cindex union
@cindex intersection
@cindex exclusion
@cindex extraction
@cindex @code{-v @var{var}}
@cindex @code{--variable @var{var}}
@cindex @code{-g @var{grp}}
@cindex @code{--grp @var{grp}}
@cindex @code{--group @var{grp}}
@cindex @code{-x}
@cindex @code{--exclude}
@cindex @code{--xcl}
@cindex @code{--unn}
@cindex @code{--union}
@cindex @code{--gxvx}
@cindex @code{--grp_xtr_var_xcl}
@cartouche
Options @code{-g @var{grp}}@*
Availability: @command{ncbo}, @command{nces},
@command{ncecat}, @command{ncflint}, @command{ncks}, @command{ncpdq},
@command{ncra}, @command{ncrcat}, @command{ncwa}@* 
Short options: @samp{-g}@*
Long options: @samp{--grp} and @samp{--group}@*
Options @code{-v @var{var}} and @code{-x}@*
Availability: (@command{ncap2}), @command{ncbo}, @command{nces},
@command{ncecat}, @command{ncflint}, @command{ncks}, @command{ncpdq},
@command{ncra}, @command{ncrcat}, @command{ncwa}@*
Short options: @samp{-v}, @samp{-x}@*
Long options: @samp{--variable}, @samp{--exclude} or @samp{--xcl}@*
Options @code{--unn}@*
Availability: @command{ncbo}, @command{nces},
@command{ncecat}, @command{ncflint}, @command{ncks}, @command{ncpdq},
@command{ncra}, @command{ncrcat}, @command{ncwa}@* 
Short options: @*
Long options: @samp{--unn} and @samp{--union}@*
Options @code{--grp_xtr_var_xcl}@*
Availability: @command{ncks}@*
Short options: @*
Long options: @samp{--gxvx} and @samp{--grp_xtr_var_xcl}@*
@end cartouche
Subsetting variables refers to explicitly specifying variables and
groups to be included or excluded from operator actions.
Subsetting is controlled by the @samp{-v @var{var}[,@dots{}]} and
@samp{-x} options for directly specifying variables.  
Specifying groups, whether in addition to or instead of variables,
is quite similar and is controlled by the @samp{-g @var{grp}[,@dots{}]}
and @samp{-x} options.
@w{A list} of variables or groups to extract is specified following the
@samp{-v} and @samp{-g} options, e.g., @samp{-v time,lat,lon} or
@samp{-g grp1,grp2}.
Both options may be specified simultaneously and @acronym{NCO} will
extract the intersection of the lists, i.e., only variables of the
specified names found in groups of the specified names.
The @samp{--unn} option causes @acronym{NCO} to extract the
union, rather than the intersection, of the specified groups and
variables. 
Not using the @samp{-v} or @samp{-g} option is equivalent to specifying
all variables or groupp, respectively.  

The @samp{-x} option causes the list of variables specified with
@samp{-v} to be @emph{excluded} rather than @emph{extracted}.
Thus @samp{-x} saves typing when you only want to extract fewer than
half of the variables in a file.
@example       
@verbatim
ncks -x -v v1,v2 in.nc out.nc # Extract all variables except v1, v2
ncks -C -x -v lat,lon in.nc out.nc # Extract all except lat, lon
@end verbatim
@end example
The first example above shows the typical use of @samp{-x} to
subset all variables except a few into the output.
Note that @code{v1} and @code{v2} will be retained in the
output if they are coordinate-like variables
(@pxref{Subsetting Coordinate Variables})
associated with any extracted variable.
If one wishes to exclude coordinate-like variables despite their
being referenced by extracted variables, one must use the @samp{-C}  
(or synonym @samp{--xcl_ass_var}) option as shown in the second
example.

Variables or groups explicitly specified for extraction with
@samp{-v @var{var}[,@dots{}]} or @samp{-g @var{grp}[,@dots{}]}
@emph{must} be present in the input file or an error will result.
Variables explicitly specified for @emph{exclusion} with
@samp{-x -v @var{var}[,@dots{}]} need not be present in the input
file.
To accord with the sophistication of the underlying hierarchy,
group subsetting is controlled by a few powerful yet subtle syntactical
distinctions.
When learning this syntax it is helpful to keep in mind the similarity
between group hierarchies and directory structures. 

@html
<a name="gxvx"></a> <!-- http://nco.sf.net/nco.html#gxvx -->
<a name="grp_xtr_var_xcl"></a> <!-- http://nco.sf.net/nco.html#grp_xtr_var_xcl -->
@end html
As of @acronym{NCO} 4.4.4 (June, 2014), @command{ncks} (alone) supports 
an option to include specified groups yet exclude specified variables.
The @samp{--grp_xtr_var_xcl} switch (with long option equivalent
@samp{--gxvx}) extracts all contents of groups given as arguments to
@samp{-g @var{grp}[,@dots{}]}, except for variables given as arguments
to @samp{-v @var{var}[,@dots{}]}.
Use this when one or a few variables in hierarchical files are not to be
extracted, and all other variables are.  
This is useful when coercing netCDF4 files into netCDF3 files such as
with converting, flattening, or dismembering files 
(see @ref{Flattening Groups}).
@example       
ncks --grp_xtr_var_xcl -g g1 -v v1 # Extract all of group g1 except v1
@end example

@cindex @command{mv}
@cindex @command{cp}
@cindex recursion
@cindex recursive
@cindex anchor
@cindex anchoring
@html
<a name="rcr"></a> <!-- http://nco.sf.net/nco.html#rcr -->
<a name="recursion"></a> <!-- http://nco.sf.net/nco.html#recursion -->
<a name="recursive"></a> <!-- http://nco.sf.net/nco.html#recursive -->
<a name="ncr"></a> <!-- http://nco.sf.net/nco.html#ncr -->
<a name="anchor"></a> <!-- http://nco.sf.net/nco.html#anchor -->
<a name="anchoring"></a> <!-- http://nco.sf.net/nco.html#anchoring -->
@end html
Two properties of subsetting, recursion and anchoring, are best
illustrated by reminding the user of their @acronym{UNIX} equivalents.
The @acronym{UNIX} command @command{mv src dst} moves @file{src}
@emph{and all its subdirectories} (and all their subdirectories etc.)
to @file{dst}.
In other words @command{mv} is, by default, @emph{recursive}.
In contrast, the @acronym{UNIX} command @command{cp src dst} moves
@file{src}, and only @file{src}, to @file{dst},
If @file{src} is a directory, not a file, then that command fails.
One must explicitly request to copy directories recursively, i.e.,
with @command{cp -r src dst}.
In @acronym{NCO} recursive extraction (and copying) of groups is the
default (like with @command{mv}, not with @command{cp}).
Recursion is turned off by appending a trailing slash to the path.

These @acronym{UNIX} commands also illustrate a property we call
@emph{anchoring}. 
The command @command{mv src dst} moves (recursively) the source
directory @file{src} to the destination directory @file{dst}. 
If @file{src} begins with the slash character then the specified path is
relative to the root directory, otherwise the path is relative to the
current working directory. 
In other words, an initial slash character anchors the subsequent path
to the root directory.
In @acronym{NCO} an initial slash anchors the path at the root group.
Paths that begin and end with slash characters (e.g., @file{//},
@file{/g1/}, and @file{/g1/g2/}) are both anchored and non-recursive. 

Consider the following commands, all of which may be assumed to end with
@samp{in.nc out.nc}:
@example       
ncks -g  g1  # Extract, recursively, all groups with a g1 component
ncks -g  g1/ # Extract, non-recursively, all groups terminating in g1
ncks -g /g1  # Extract, recursively, root group g1
ncks -g /g1/ # Extract, non-recursively root group g1
ncks -g //   # Extract, non-recursively the root group
@end example
The first command is probably the most useful and common.
It would extract these groups, if present, and all their direct
ancestors and children:
@file{/g1}, @file{/g2/g1}, and @file{/g3/g1/g2}.
In other words, the simplest form of @samp{-g grp} grabs all groups that 
(and their direct ancestors and children, recursively) that have
@file{grp} as a complete component of their path.
A simple string match is insufficient, @var{grp} must be a complete 
component (i.e., group name) in the path.
The option @samp{-g g1} would not extract these groups because @file{g1} 
is not a complete component of the path: @file{/g12}, @file{/fg1}, and
@file{/g1g1}.
The second command above shows how a terminating slash character
@kbd{/} cancels the recursive copying of groups.
An argument to @samp{-g} which terminates with a slash character
extracts the group and its direct ancestors, but none of its children. 
The third command above shows how an initial slash character @kbd{/}
anchors the argument to the root group.
The third command would not extract the group @file{/g2/g1} because 
the @file{g1} group is not at the root level, but it would extract,
any group @file{/g1} at the root level and all its children,
recursively.  
The fourth command is the non-recursive version of the third command.
The fifth command is a special case of the fourth command.

@html
<a name="unn"></a> <!-- http://nco.sf.net/nco.html#unn -->
<a name="nsx"></a> <!-- http://nco.sf.net/nco.html#nsx -->
<a name="union"></a> <!-- http://nco.sf.net/nco.html#union -->
<a name="intersection"></a> <!-- http://nco.sf.net/nco.html#intersection -->
@end html
@cindex union
@cindex intersection
@cindex @code{--unn}
@cindex @code{--union}
@cindex @code{--nsx}
@cindex @code{--intersection}
As mentioned above, both @samp{-v} and @samp{-g} options may be
specified simultaneously and @acronym{NCO} will, by default, extract the
intersection of the lists, i.e., the specified variables found in the
specified groups
@footnote{
Intersection-mode can also be explicitly invoked with the @samp{--nsx}
or @samp{--intersection} switches.
These switches are supplied for clarity and consistency and do
absolutely nothing since intersection-mode is the default.}.
The @samp{--unn} option causes @acronym{NCO} to extract the
union, rather than the intersection, of the specified groups and
variables. 
Consider the following commands (which may be assumed to end with
@samp{in.nc out.nc}):
@example       
# Intersection-mode subsetting (default)
ncks -g  g1  -v v1 # Yes: /g1/v1, /g2/g1/v1. No: /v1, /g2/v1
ncks -g /g1  -v v1 # Yes: /g1/v1, /g1/g2/v1. No: /v1, /g2/v1, /g2/g1/v1
ncks -g  g1/ -v v1 # Yes: /g1/v1, /g2/g1/v1. No: /v1, /g2/v1, /g1/g2/v1
ncks -v  g1/v1     # Yes: /g1/v1, /g2/g1/v1. No: /v1, /g2/v1, /g1/g2/v1
ncks -g /g1/ -v v1 # Yes: /g1/v1. No: /g2/g1/v1, /v1, /g2/v1 ...
ncks -v /g1/v1     # Yes: /g1/v1. No: /g2/g1/v1, /v1, /g2/v1 ...

# Union-mode subsetting (invoke with --unn or --union)
ncks -g  g1  -v v1 --unn # All variables in  g1 or progeny, or named v1
ncks -g /g1  -v v1 --unn # All variables in /g1 or progeny, or named v1
ncks -g  g1/ -v v1 --unn # All variables in  g1 or named v1
ncks -g /g1/ -v v1 --unn # All variables in /g1 or named v1
@end example
The first command (@samp{-g g1 -v v1}) extracts the variable @file{v1}
from any group named @file{g1} or descendent @file{g1}.
The second command extracts @file{v1} from any root group
named @file{g1} and any descendent groups as well.
The third and fourth commands are equivalent ways of extracting
@file{v1} only from the root group named @file{g1} (not its
descendents). 
The fifth and sixth commands are equivalent ways of extracting the 
variable @file{v1} only from the root group named @file{g1}.
Subsetting in union-mode (with @samp{--unn}) causes all variables to be
extracted which meet either one or both of the specifications of the 
variable and group specifications.
Union-mode subsetting is simply the logical ``OR'' of intersection-mode
subsetting. 
As discussed below, the group and variable specifications may be comma
separated lists of regular expressions for added control over
subsetting. 

@cindex memory requirements
Remember, if averaging or concatenating large files stresses your
systems memory or disk resources, then the easiest solution is often to
subset (with @samp{-g} and/or @samp{-v}) to retain only the most
important variables (@pxref{Memory Requirements}).
@example       
ncks          in.nc out.nc # Extract all groups and variables
ncks -v scl   # Extract variable scl from all groups
ncks -g g1    # Extract group g1 and descendents
ncks -x -g g1 # Extract all groups except g1 and descendents
ncks -g g2,g3 -v scl # Extract scl from groups g2 and g3
@end example
Overwriting and appending work as expected:
@example
# Replace scl in group g2 in out.nc with scl from group g2 from in.nc
ncks -A -g g2 -v scl in.nc out.nc
@end example

Due to its special capabilities, @command{ncap2} interprets the
@samp{-v} switch differently 
(@pxref{ncap2 netCDF Arithmetic Processor}). 
For @command{ncap2}, the @samp{-v} switch takes no arguments and
indicates that @emph{only} user-defined variables should be output. 
@command{ncap2} neither accepts nor understands the @var{-x} and
@var{-g} switches.

@html
<a name="rx"></a> <!-- http://nco.sf.net/nco.html#rx -->
<a name="wildcarding"></a> <!-- http://nco.sf.net/nco.html#wildcarding -->
@end html
@cindex extended regular expressions
@cindex regular expressions
@cindex pattern matching
@cindex wildcards
@cindex @command{grep -E}
@cindex @command{egrep}
@cindex @command{ncatted}
@cindex @acronym{GNU}
Regular expressions the syntax that @acronym{NCO} use pattern-match
object names in netCDF file against user requests.
The user can select all variables beginning with the string @samp{DST}
from an input file by supplying the regular expression @samp{^DST} to
the @samp{-v} switch, i.e., @samp{-v '^DST'}. 
The meta-characters used to express pattern matching operations are
@samp{^$+?.*[]@{@}|}. 
If the regular expression pattern matches @emph{any} part of a variable 
name then that variable is selected.
This capability is also called @dfn{wildcarding}, and is very useful for 
sub-setting large data files.

Extended regular expressions are defined by the @acronym{POSIX}
@command{grep -E} (aka @command{egrep}) command.  
As of @acronym{NCO} 2.8.1 (August, 2003), variable name arguments
to the @samp{-v} switch may contain @dfn{extended regular expressions}.
As of @acronym{NCO} 3.9.6 (January, 2009), variable names arguments 
to @command{ncatted} may contain @dfn{extended regular expressions}. 
As of @acronym{NCO} 4.2.4 (November, 2012), group name arguments 
to the @samp{-g} switch may contain @dfn{extended regular expressions}.

@cindex @acronym{POSIX}
@cindex @code{regex}
Because of its wide availability, @acronym{NCO} uses the @acronym{POSIX}  
regular expression library @code{regex}.  
Regular expressions of arbitary complexity may be used.
Since netCDF variable names are relatively simple constructs, only a 
few varieties of variable wildcards are likely to be useful.
For convenience, we define the most useful pattern matching operators
here: 
@cindex @code{.} (wildcard character)
@cindex @code{$} (wildcard character)
@cindex @code{^} (wildcard character)
@cindex @code{?} (filename expansion)
@cindex @code{*} (filename expansion)
@table @samp
@item ^
Matches the beginning of a string
@item $
Matches the end of a string
@item .
Matches any single character
@end table
@noindent
The most useful repetition and combination operators are
@cindex @code{?} (wildcard character)
@cindex @code{*} (wildcard character)
@cindex @code{+} (wildcard character)
@cindex @code{|} (wildcard character)
@table @samp
@item ?
The preceding regular expression is optional and matched at most once
@item *
The preceding regular expression will be matched zero or more times
@item +
The preceding regular expression will be matched one or more times
@item |
The preceding regular expression will be joined to the following regular
expression.
The resulting regular expression matches any string matching either
subexpression. 
@end table
@noindent

To illustrate the use of these operators in extracting variables
and groups, consider file @file{in_grp.nc} with groups
@code{g0}--@code{g9}, and subgroups @code{s0}--@code{s9}, in each of
those groups, and file @file{in.nc} with variables @code{Q},
@code{Q01}--@code{Q99}, @code{Q100}, @code{QAA}--@code{QZZ},
@code{Q_H2O}, @code{X_H2O}, @code{Q_CO2}, @code{X_CO2}.  
@example
@verbatim
ncks -v '.+' in.nc               # All variables (default)
ncks -v 'Q.?' in.nc              # Variables that contain Q
ncks -v '^Q.?' in.nc             # Variables that start with Q
ncks -v '^Q+.?.' in.nc           # Q, Q0--Q9, Q01--Q99, QAA--QZZ, etc.
ncks -v '^Q..' in.nc             # Q01--Q99, QAA--QZZ, etc.
ncks -v '^Q[0-9][0-9]' in.nc     # Q01--Q99, Q100
ncks -v '^Q[[:digit:]]{2}' in.nc # Q01--Q99
ncks -v 'H2O$' in.nc             # Q_H2O, X_H2O 
ncks -v 'H2O$|CO2$' in.nc        # Q_H2O, X_H2O, Q_CO2, X_CO2 
ncks -v '^Q[0-9][0-9]$' in.nc    # Q01--Q99
ncks -v '^Q[0-6][0-9]|7[0-3]' in.nc # Q01--Q73, Q100
ncks -v '(Q[0-6][0-9]|7[0-3])$' in.nc # Q01--Q73
ncks -v '^[a-z]_[a-z]{3}$' in.nc # Q_H2O, X_H2O, Q_CO2, X_CO2
ncks -g 'g.' in_grp.nc           # 10 Groups g0-g9
ncks -g 's.' in_grp.nc       # 100 sub-groups g0/s0, g0/s1, ... g9/s9
ncks -g 'g.' -v 'v.' in_grp.nc   # All variables 'v.' in groups 'g.'
@end verbatim
@end example
Beware---two of the most frequently used repetition pattern matching
operators, @samp{*} and @samp{?}, are also valid pattern matching
operators for filename expansion (globbing) at the shell-level.
Confusingly, their meanings in extended regular expressions and in
shell-level filename expansion are significantly different.
In an extended regular expression, @samp{*} matches zero or more
occurences of the preceding regular expression. 
Thus @samp{Q*} selects all variables, and @samp{Q+.*} selects all
variables containing @samp{Q} (the @samp{+} ensures the preceding item 
matches at least once).
To match zero or one occurence of the preceding regular expression,   
use @samp{?}.
Documentation for the @acronym{UNIX} @command{egrep} command details the
extended regular expressions which @acronym{NCO} supports.

@html
<a name="globbing"></a> <!-- http://nco.sf.net/nco.html#globbing -->
<a name="glb"></a> <!-- http://nco.sf.net/nco.html#glb -->
@end html
@cindex globbing
@cindex shell
@cindex @command{bash}
@cindex @command{csh}
@cindex quotes
One must be careful to protect any special characters in the regular
expression specification from being interpreted (globbed) by the shell.
This is accomplish by enclosing special characters within single or
double quotes
@example
ncra -v Q?? in.nc out.nc   # Error: Shell attempts to glob wildcards
ncra -v '^Q+..' in.nc out.nc # Correct: NCO interprets wildcards
ncra -v '^Q+..' in*.nc out.nc # Correct: NCO interprets, Shell globs 
@end example
The final example shows that commands may use a combination of variable
wildcarding and shell filename expansion (globbing).
For globbing, @samp{*} and @samp{?} @emph{have nothing to do} with the 
preceding regular expression!
In shell-level filename expansion, @samp{*} matches any string,
including the null string and @samp{?} matches any single character. 
Documentation for @command{bash} and @command{csh} describe the rules of
filename expansion (globbing).

@html
<a name="no_coord"></a> <!-- http://nco.sf.net/nco.html#no_coord -->
<a name="no_crd"></a> <!-- http://nco.sf.net/nco.html#no_crd -->
<a name="crd"></a> <!-- http://nco.sf.net/nco.html#crd -->
<a name="-C"></a> <!-- http://nco.sf.net/nco.html#-C -->
<a name="-c"></a> <!-- http://nco.sf.net/nco.html#-c -->
<a name="xcl_ass_var"></a> <!-- http://nco.sf.net/nco.html#xcl_ass_var -->
<a name="xtr_ass_var"></a> <!-- http://nco.sf.net/nco.html#xtr_ass_var -->
@end html
@node Subsetting Coordinate Variables, Group Path Editing, Subsetting Files, Shared features
@section Subsetting Coordinate Variables
@cindex subsetting
@cindex @code{-C}
@cindex @code{-c}
@cindex @code{--xcl_ass_var}
@cindex @code{--xtr_ass_var}
@cindex @code{--no_coords}
@cindex @code{--no_crd}
@cindex @code{--coords}
@cindex @code{--crd}
@cartouche
Availability: @command{ncap2}, @command{ncbo}, @command{nces},
@command{ncecat}, @command{ncflint}, @command{ncks}, @command{ncpdq},
@command{ncra}, @command{ncrcat}, @command{ncwa}@* 
Short options: @samp{-C}, @samp{-c}@*
Long options: @samp{--no_coords}, @samp{--no_crd}, @samp{--xcl_ass_var}, 
@samp{--crd}, @samp{--coords}, @samp{--xtr_ass_var}@*
@end cartouche
By default, coordinates variables associated with any variable appearing
in the @var{input-file} will be placed in the @var{output-file}, even
if they are not explicitly specified, e.g., with the @samp{-v} switch.
Thus variables with a latitude coordinate @code{lat} always carry the
values of @code{lat} with them into the @var{output-file}.
This automatic inclusion feature can be disabled with @samp{-C}, which
causes @acronym{NCO} to exclude (or, more precisely, not to
automatically include) coordinates and associated variables from the
extraction list.
However, using @samp{-C} does not preclude the user from including some
coordinates in the output files simply by explicitly selecting the
coordinates and associated variables with the @var{-v} option.
The @samp{-c} option, on the other hand, is a shorthand way of
automatically specifying that @emph{all} coordinate and associated
variables in @var{input-files} should appear in @var{output-file}.
The user can thereby select all coordinate variables without even
knowing their names.  

The meaning of ``coordinates'' in these two options has expanded since
about 2009 from simple one dimensional coordinates (per the
@acronym{NUG}) definition) to any and all associated variables.
This includes multi-dimensional coordinates as well as a menagerie
of associated variables defined by the @acronym{CF} metadata
conventions: 
@cindex @acronym{CF} conventions
As of @acronym{NCO} version 4.4.5 (July, 2014) 
both @samp{-c} and @samp{-C} honor the @acronym{CF} @code{ancillary_variables}
convention described in @ref{CF Conventions}. 
As of @acronym{NCO} version 4.0.8 (April, 2011) 
both @samp{-c} and @samp{-C} honor the @acronym{CF} @code{bounds}
convention described in @ref{CF Conventions}. 
As of @acronym{NCO} version 4.6.4 (January, 2017) 
both @samp{-c} and @samp{-C} honor the @acronym{CF} @code{cell_measures}
convention described in @ref{CF Conventions}. 
As of @acronym{NCO} version 4.4.9 (May, 2015) 
both @samp{-c} and @samp{-C} honor the @acronym{CF} @code{climatology}
convention described in @ref{CF Conventions}. 
As of @acronym{NCO} version 3.9.6 (January, 2009) 
both @samp{-c} and @samp{-C} honor the @acronym{CF} @code{coordinates}
convention described in @ref{CF Conventions}.
As of @acronym{NCO} version 4.6.4 (January, 2017) 
both @samp{-c} and @samp{-C} honor the @acronym{CF} @code{formula_terms}
convention described in @ref{CF Conventions}. 
As of @acronym{NCO} version 4.6.0 (May, 2016) 
both @samp{-c} and @samp{-C} honor the @acronym{CF} @code{grid_mapping}
convention described in @ref{CF Conventions}. 

The expanded categories of variables controlled by @samp{-c} and
@samp{-C} justified adding a more descriptive switch.
As of @acronym{NCO} version 4.8.0 (May, 2019) the switch
@samp{--xcl_ass_var}, which stands for ``exclude associated variables'', 
is synonymous with @samp{-C} and @samp{--xtr_ass_var}, which stands
for ``extract associated variables'', is synonymous with @samp{-c}.

@html
<a name="gpe"></a> <!-- http://nco.sf.net/nco.html#gpe -->
@end html
@node Group Path Editing, C and Fortran Index Conventions, Subsetting Coordinate Variables, Shared features
@section Group Path Editing
@cindex @code{-G @var{gpe_dsc}}
@cindex @code{--gpe @var{gpe_dsc}}
@cartouche
Options @code{-G @var{gpe_dsc}}@*
Availability: @command{ncbo}, @command{ncecat}, @command{nces},
@command{ncflint}, @command{ncks}, @command{ncpdq}, @command{ncra},
@command{ncrcat}, @command{ncwa}@*
Short options: @samp{-G}@*
Long options: @samp{--gpe}@*
@end cartouche

@dfn{Group Path Editing}, or @acronym{GPE}, allows the user to
restructure (i.e., add, remove, and rename groups) in the output file 
relative to the input file based on the instructions they provide.
As of  @acronym{NCO} 4.2.3 (November, 2012), all operators that accept  
netCDF4 files with groups accept the @samp{-G} switch, or its
long-option equivalent @samp{--gpe}.
To master @acronym{GPE} one must understand the meaning of the
required @var{gpe_dsc} structure/argument that specifies the
transformation of input-to-output group paths.

Each @var{gpe_dsc} contains up to three elements (two are optional) in
the following order:@*   
@var{gpe_dsc} = @var{grp_pth}:@var{lvl_nbr} or @var{grp_pth}@@@var{lvl_nbr}

@table @var
@item grp_pth
Group Path.
@cindex group path
This (optional) component specifies the output group path that should be 
appended after any editing (i.e., deletion or truncation) of the input
path is performed.
@item lvl_nbr
The number of levels to delete (from the head) or truncate (from the
tail) of the input path.
@end table
@noindent
If both components of the argument are present, then a single character,
either the colon or at-sign (@code{:} or @code{@@}), must separate them.
If only @var{grp_pth} is specifed, the separator character may be
omitted, e.g., @samp{-G g1}.
If only @var{lvl_nbr} is specifed, the separator character is still
required to indicate it is a @var{lvl_nbr} arugment and not a
@var{grp_pth}, e.g., @samp{-G :-1} or @samp{-G @@1}.

If the at-sign separator character @code{@@} is used instead of the colon
separator character @code{:}, then the following @var{lvl_nbr} arugment 
must be positive and it will be assumed to refer to Truncation-Mode.
Hence, @samp{-G :-1} is the same as @samp{-G @@1}. 
This is simply a way of making the @var{lvl_nbr} argument
positive-definite. 

@menu
* Flattening Groups::
* Moving Groups::
* Dismembering Files::
* Checking CF-compliance::
@end menu
@node Flattening Groups, Moving Groups, Group Path Editing, Group Path Editing
@subsection Deletion, Truncation, and Flattening of Groups
@html
<a name="flatten"></a> <!-- http://nco.sf.net/nco.html#flatten -->
<a name="delete"></a> <!-- http://nco.sf.net/nco.html#delete -->
<a name="truncate"></a> <!-- http://nco.sf.net/nco.html#truncate -->
@end html
@cindex @code{@@} (separator character)
@cindex @code{:} (separator character)
@cindex delete (groups)
@cindex truncate (groups)
@cindex flatten (groups)
@acronym{GPE} has three editing modes: Delete, Truncate, and
Flatten.
Select one of @acronym{GPE}'s three editing modes by supplying a
@var{lvl_nbr} that is positive, negative, or zero for Delete-, 
Truncate- and Flatten-mode, respectively. 

In Delete-mode, @var{lvl_nbr} is a positive integer which specifies
the maximum number of group path components (i.e., groups) that
@acronym{GPE} will try to delete from the head of @var{grp_pth}. 
For example @math{@var{lvl_nbr} = 3} changes the input path
@file{/g1/g2/g3/g4/g5} to the output path @file{/g4/g5}.
Input paths with @var{lvl_nbr} or fewer components (groups)
are completely erased and the output path commences from the root  
level. 

In other words, @acronym{GPE} is tolerant of specifying too many group
components to delete. 
It deletes as many as possible, without complaint, and then begins to
flatten the file (which fails if namespace conflicts arise).

In Truncate-mode, @var{lvl_nbr} is a negative integer which specifies
the maximum number of group path components (i.e., groups) that
@acronym{GPE} will try to truncate from the tail of @var{grp_pth}. 
For example @math{@var{lvl_nbr} = -3} changes the input path
@file{/g1/g2/g3/g4/g5} to the output path @file{/g1/g2}.
Input paths with @var{lvl_nbr} or fewer components (groups)
are completely erased and the output path commences from the root  
level. 

In Flatten-mode, indicated by the separator character alone
or with @math{@var{lvl_nbr} = 0}, @acronym{GPE} removes the entire group
path from the input file and constructs the output path beginning at the
root level.  
For example @code{-G :0} and @code{-G :} are identical and change the
input path @file{/g1/g2/g3/g4/g5} to the output path @file{/} whereas
@code{-G g1:0} and @code{-G g1:} are identical and result in the output 
path @file{/g1} for all variables.

Subsequent to the alteration of the input path by the specified
editing mode, if any, @acronym{GPE} prepends (in Delete Mode)
or Appends (in Truncate-mode) any specifed @var{grp_pth} to the output
path. 
For example @code{-G g2} changes the input paths @file{/} and @file{/g1}
to @file{/g2} and @file{/g1/g2}, respectively.
Likewise, @code{-G g2/g3} changes the input paths @file{/} and @file{/g1}
to @file{/g2/g3} and @file{/g1/g2/g3}, respectively.
When @var{grp_pth} and @var{lvl_nbr} are both specified, the editing
actions are taken in sequence so that, e.g., @code{-G g1/g2:2} 
changes the input paths @file{/} and @file{/h1/h2/h3/h4}
to @file{/g1/g2} and @file{/g1/g2/h3/h4}, respectively.
Likewise, @code{-G g1/g2:-2} changes the input paths @file{/} and
@file{/h1/h2/h3/h4} to @file{/g1/g2} and @file{/h1/h2/g1/g2},
respectively. 

Combining @acronym{GPE} with subsetting (@pxref{Subsetting Files}) 
yields powerful control over the extracted (or excluded) variables and
groups and their placement in the output file as shown by the following
commands. 
All commands below may be assumed to end with @samp{in.nc out.nc}.
@example       
@verbatim
# Prepending paths without editing:
ncks                   # /g?/v? -> /g?/v?
ncks             -v v1 # /g?/v1 -> /g?/v1
ncks       -g g1       # /g1/v? -> /g1/v?
ncks -G o1             # /g?/v? -> /o1/g?/v?
ncks -G o1 -g g1       # /g1/v? -> /o1/g1/v?
ncks       -g g1 -v v1 # /g1/v1 -> /g1/v1
ncks -G o1       -v v1 # /g?/v1 -> /o1/g?/v1
ncks -G o1 -g g1 -v v1 # /g1/v1 -> /o1/g1/v1
ncks -G g1 -g /  -v v1 # /v1    -> /g1/v1
ncks -G g1/g2    -v v1 # /g?/v1 -> /g1/g2/g?/v1
# Delete-mode: Delete from and Prepend to path head
# Syntax: -G [ppn]:lvl_nbr = # of levels to delete
ncks -G :1    -g g1    -v v1 # /g1/v1    -> /v1
ncks -G :1    -g g1/g1 -v v1 # /g1/g1/v1 -> /g1/v1
ncks -G :2    -g g1/g1 -v v1 # /g1/g1/v1 -> /v1
ncks -G :2    -g g1    -v v1 # /g1/v1    -> /v1
ncks -G g2:1  -g g1    -v v1 # /g1/v1    -> /g2/v1
ncks -G g2:2  -g g1/g1 -v v1 # /g1/g1/v1 -> /g2/v1
ncks -G g2:1  -g /     -v v1 # /v1       -> /g2/v1
ncks -G g2:1           -v v1 # /v1       -> /g2/v1
ncks -G g2:1  -g g1/g1 -v v1 # /g1/g1/v1 -> /g2/g1/v1
# Flatten-mode: Remove all input path components
# Syntax: -G [apn]: colon without numerical argument
ncks -G :            -v v1 # /g?/v1    -> /v1
ncks -G :   -g g1    -v v1 # /g1/v1    -> /v1
ncks -G :   -g g1/g1 -v v1 # /g1/g1/v1 -> /v1
ncks -G g2:          -v v1 # /g?/v1    -> /g2/v1
ncks -G g2:                # /g?/v?    -> /g2/v?
ncks -G g2: -g g1/g1 -v v1 # /g1/g1/v1 -> /g2/v1
# Truncate-mode: Truncate from and Append to path tail
# Syntax: -G [apn]:-lvl_nbr = # of levels to truncate
# NB: -G [apn]:-lvl_nbr is equivalent to -G [apn]@lvl_nbr
ncks -G :-1   -g g1    -v v1 # /g1/v1    -> /v1
ncks -G :-1   -g g1/g2 -v v1 # /g1/g2/v1 -> /g1/v1
ncks -G :-2   -g g1/g2 -v v1 # /g1/g2/v1 -> /v1
ncks -G :-2   -g g1    -v v1 # /g1/v1    -> /v1
ncks -G g2:-1          -v v1 # /g?/v1    -> /g2/v1
ncks -G g2:-1 -g g1    -v v1 # /g1/v1    -> /g2/v1
ncks -G g1:-1 -g g1/g2 -v v1 # /g1/g2/v1 -> /g1/g1/v1
@end verbatim
@end example

@html
<a name="mv"></a> <!-- http://nco.sf.net/nco.html#mv -->
<a name="move"></a> <!-- http://nco.sf.net/nco.html#move -->
@end html
@node Moving Groups, Dismembering Files, Flattening Groups, Group Path Editing
@subsection Moving Groups
@cindex move groups
@cindex groups, moving
@cindex rename groups
@cindex groups, renaming
Until fall 2013 (netCDF version 4.3.1-pre1), netCDF contained no library
function for renaming groups, and therefore @command{ncrename} cannot
rename groups.
However, @acronym{NCO} built on earlier versions of netCDF than 4.3.1
can use a @acronym{GPE}-based workaround mechanism to ``rename''
groups. 
The @acronym{GPE} mechanism actually @emph{moves} (i.e., copies to a new
location) groups, a more arduous procedure than simply renaming them.
@acronym{GPE} applies to all selected groups, so, in the general case,
one must move only the desired group to a new file, and then merge that
new file with the original to obtain a file where the desired group has
been ``renamed'' and all else is unchanged.
Here is how to ``rename'' group @file{/g4} to group @file{/f4} with
@acronym{GPE} instead of @command{ncrename}
@example
ncks -O -G f4:1 -g g4 ~/nco/data/in_grp.nc ~/tmp.nc # Move /g4 to /f4
ncks -O -x -g g4 ~/nco/data/in_grp.nc ~/out.nc # Excise /g4
ncks -A ~/tmp.nc ~/out.nc # Add /f4 to new file
@end example
If the original group @file{g4} is not excised from @file{out.nc} (step
two above), then the final output file would contain both @file{g4} and
a copy named @file{f4}.
Thus GPE can be used to both ``rename'' and copy groups.
The recommended way to rename groups when when netCDF version 4.3.1 is
availale is to use @command{ncrename} (@pxref{ncrename netCDF Renamer}).

@html
<a name="xmp_flt"></a> <!-- http://nco.sf.net/nco.html#xmp_flt -->
@end html
One may wish to flatten hierarchical group files for many reasons.
These include @w{1. To} obtain flat netCDF3 files for use with tools
that do not work with netCDF4 files, @w{2. To} split-apart
hierarchies to re-assemble into different hierarchies, and
@w{3. To} provide a subset of a hierarchical file with the simplest
possible storage structure.
@example
ncks -O -G : -g cesm -3 ~/nco/data/cmip5.nc ~/cesm.nc # Extract /cesm to /
@end example
The @option{-3} switch
@footnote{Note that the @option{-3} switch should appear @emph{after} the
@option{-G} and @option{-g} switches. 
This is due to an artifact of the @acronym{GPE} implementation which we
wish to remove in the future.}
specifies the output dataset should be in netCDF3
format, the @option{-G :} option flattens all extracted groups, and the
@option{-g cesm} option extracts only the @code{cesm} group and leaves
all other groups (e.g., @code{ecmwf}, @code{giss}).

@html
<a name="dismember"></a> <!-- http://nco.sf.net/nco.html#dismember -->
<a name="disaggregate"></a> <!-- http://nco.sf.net/nco.html#disaggregate -->
<a name="ncdismember"></a> <!-- http://nco.sf.net/nco.html#ncdismember -->
@end html
@node Dismembering Files, Checking CF-compliance, Moving Groups, Group Path Editing
@subsection Dismembering Files
@cindex disaggregate
@cindex dismember
@findex ncdismember
Let us show how to completely disaggregate (or, more memorably)
@emph{dismember} a hierarchical dataset.
For now we take this to mean: store each group as a standalone flat
dataset in netCDF3 format.
This can be accomplished by looping the previous example over all
groups. 
This script @file{ncdismember} dismembers the input file @var{fl_in}
specified in the first argument and places the resulting files in the
directory @var{drc_out} specified by the second argument:
@example
@verbatim
cat > ~/ncdismember << 'EOF'
#!/bin/sh

# Purpose: Dismember netCDF4/HDF5 hierarchical files. CF-check them.
# Place each input file group in separate netCDF3 output file
# Described in NCO User Guide at http://nco.sf.net/nco.html#dismember
# Requirements: NCO 4.3.x+, UNIX shell utilities awk, grep, sed
# Optional: Decker CFchecker https://bitbucket.org/mde_/cfchecker

# Usage:
# ncdismember <fl_in> <drc_out> [cf_chk] [cf_vrs] [opt]
# where fl_in is input file/URL to dismember, drc_out is output directory
# CF-compliance check is performed when optional third argument is not '0'
# Default checker is Decker's cfchecker installed locally
# Specify cf_chk=nerc for smallified uploads to NERC checker
# Optional fourth argument cf_vrs is CF version to check
# Optional fifth argument opt passes straight-through to ncks
# Arguments must not use shell expansion/globbing
# NB: ncdismember does not clean-up output directory, so user must
# chmod a+x ~/sh/ncdismember
# Examples:
# ncdismember ~/nco/data/mdl_1.nc /data/zender/tmp
# ncdismember http://dust.ess.uci.edu/nco/mdl_1.nc /tmp
# ncdismember http://thredds-test.ucar.edu/thredds/dodsC/testdods/foo.nc /tmp
# ncdismember ~/nco/data/mdl_1.nc /data/zender/nco/tmp cf
# ncdismember ~/nco/data/mdl_1.nc /data/zender/nco/tmp nerc
# ncdismember ~/nco/data/mdl_1.nc /data/zender/nco/tmp cf 1.3
# ncdismember ~/nco/data/mdl_1.nc /data/zender/nco/tmp cf 1.5 --fix_rec_dmn=all

# Command-line argument defaults
fl_in="${HOME}/nco/data/mdl_1.nc" # [sng] Input file to dismember/check
drc_out="${DATA}/nco/tmp" # [sng] Output directory
cf_chk='0' # [flg] Perform CF-compliance check? Which checker?
cf_vrs='1.5' # [sng] Compliance-check this CF version (e.g., '1.5')
opt='' # [flg] Additional ncks options (e.g., '--fix_rec_dmn=all')
# Use single quotes to pass multiple arguments to opt=${5}
# Otherwise arguments would be seen as ${5}, ${6}, ${7} ...

# Command-line argument option parsing
if [ -n "${1}" ]; then fl_in=${1}; fi
if [ -n "${2}" ]; then drc_out=${2}; fi
if [ -n "${3}" ]; then cf_chk=${3}; fi
if [ -n "${4}" ]; then cf_vrs=${4}; fi
if [ -n "${5}" ]; then opt=${5}; fi

# Prepare output directory
echo "NCO dismembering file ${fl_in}"
fl_stb=$(basename ${fl_in})
drc_out=${drc_out}/${fl_stb}
mkdir -p ${drc_out}
cd ${drc_out}
chk_dck='n'
chk_nrc='n'
if [ ${cf_chk} = 'nerc' ]; then
    chk_nrc='y'
fi # chk_nrc
if [ ${cf_chk} != '0' ] && [ ${cf_chk} != 'nerc' ]; then
    chk_dck='y'
    hash cfchecker 2>/dev/null || { echo >&2 "Local cfchecker command not found, will smallify and upload to NERC checker instead"; chk_nrc='y'; chk_dck='n'; }
fi # !cf_chk
# Obtain group list
grp_lst=`ncks -m ${fl_in} | grep '// group' | awk '{$1=$2=$3="";sub(/^  */,"",$0);print}'`
IFS=$'\n' # Change Internal-Field-Separator from <Space><Tab><Newline> to <Newline>
for grp_in in ${grp_lst} ; do
    # Replace slashes by dots for output group filenames
    grp_out=`echo ${grp_in} | sed 's/\///' | sed 's/\//./g'`
    if [ "${grp_out}" = '' ]; then grp_out='root' ; fi
    # Tell older NCO/netCDF if HDF4 with --hdf4 switch (signified by .hdf/.HDF suffix)
    hdf4=`echo ${fl_in} | awk '{if(match(tolower($1),".hdf$")) hdf4="--hdf4"; print hdf4}'`
    # Flatten to netCDF3, anchor, no history, no temporary file, padding, HDF4 flag, options
    cmd="ncks -O -3 -G : -g ${grp_in}/ -h --no_tmp_fl --hdr_pad=40 ${hdf4} ${opt} ${fl_in} ${drc_out}/${grp_out}.nc"
    # Use eval in case ${opt} contains multiple arguments separated by whitespace
    eval ${cmd}
    if [ ${chk_dck} = 'y' ]; then
       # Decker checker needs Conventions <= 1.6
       no_bck_sls=`echo ${drc_out}/${grp_out} | sed 's/\\\ / /g'`
       ncatted -h -a Conventions,global,o,c,CF-${cf_vrs} ${no_bck_sls}.nc
    else # !chk_dck
       echo ${drc_out}/${grp_out}.nc
    fi # !chk_dck
done
if [ ${chk_dck} = 'y' ]; then
    echo 'Decker CFchecker reports CF-compliance of each group in flat netCDF3 format'
    cfchecker -c ${cf_vrs} *.nc
fi
if [ ${chk_nrc} = 'y' ]; then
    # Smallification and NERC upload from qdcf script by Phil Rasch (PJR)
    echo 'Using remote CFchecker http://puma.nerc.ac.uk/cgi-bin/cf-checker.pl'
    cf_lcn='http://puma.nerc.ac.uk/cgi-bin/cf-checker.pl'
    for fl in ${drc_out}/*.nc ; do
	fl_sml=${fl}
	cf_out=${fl%.nc}.html
	dmns=`ncdump -h ${fl_in} | sed -n -e '/dimensions/,/variables/p' | grep = | sed -e 's/=.*//'`
	hyp_sml=''
	for dmn in ${dmns}; do
	    dmn_lc=`echo ${dmn} | tr "[:upper:]" "[:lower:]"`
	    if [ ${dmn_lc} = 'lat' ] || [ ${dmn_lc} = 'latitude' ] || [ ${dmn_lc} = 'lon' ] || [ ${dmn_lc} = 'longitude' ] || [ ${dmn_lc} = 'time' ]; then
		hyp_sml=`echo ${hyp_sml}" -d ${dmn},0"`
	    fi # !dmn_lc
	done
	# Create small version of input file by sampling only first element of lat, lon, time
	ncks -O ${hyp_sml} ${fl} ${fl_sml}
	# Send small file to NERC checker
	curl --form cfversion=1.6 --form upload=@${fl_sml} --form press="Check%20file" ${cf_lcn} -o ${cf_out}
	# Strip most HTML to improve readability
	cat ${cf_out} | sed -e "s/<[^>]*>//g" -e "/DOCTYPE/,/\]\]/d" -e "s/CF-Convention//g" -e "s/Output of//g" -e "s/Compliance Checker//g" -e "s/Check another//g" -e "s/CF-Checker follows//g" -e "s/Received//g" -e "s/for NetCDF//g" -e "s/NetCDF format//g" -e "s/against CF version 1//g" -e "s/\.\.\.//g"
	echo "Full NERC compliance-check log for ${fl} in ${cf_out}"
    done
fi # !nerc
EOF
chmod 755 ~/ncdismember # Make command executable
/bin/mv -f ~/ncdismember ~/sh # Store in location on $PATH, e.g., /usr/local/bin

zender@roulee:~$ ncdismember ~/nco/data/mdl_1.nc ${DATA}/nco/tmp
NCO dismembering file /home/zender/nco/data/mdl_1.nc
/data/zender/nco/tmp/mdl_1.nc/cesm.cesm_01.nc
/data/zender/nco/tmp/mdl_1.nc/cesm.cesm_02.nc
/data/zender/nco/tmp/mdl_1.nc/cesm.nc
/data/zender/nco/tmp/mdl_1.nc/ecmwf.ecmwf_01.nc
/data/zender/nco/tmp/mdl_1.nc/ecmwf.ecmwf_02.nc
/data/zender/nco/tmp/mdl_1.nc/ecmwf.nc
/data/zender/nco/tmp/mdl_1.nc/root.nc
@end verbatim
@end example
A (potentially more portable) binary executable could be written to
dismember all groups with a single invocation, yet dismembering without
loss of information is possible now with this simple script on all 
platforms with @acronym{UNIX}y utilities.
Note that all dimensions inherited by groups in the input file are
correctly placed by @command{ncdismember} into the flat files.
Moreover, each output file preserves the group metadata of all ancestor
groups, including the global metadata from the input file.
As written, the script could fail on groups that contain advanced
netCDF4 features because the user requests (with the @samp{-3} switch)
that output be netCDF3 classic format.  
However, @command{ncks} detects many format incompatibilities in advance
and works around them.
For example, @command{ncks} autoconverts netCDF4-only atomic-types (such
as @code{NC_STRING} and @code{NC_UBYTE}) to corresponding netCDF3
atomic types (@code{NC_CHAR} and @code{NC_SHORT}) when the output format
is netCDF3. 

@html
<a name="cf-compliance"></a> <!-- http://nco.sf.net/nco.html#cf-compliance -->
<a name="nccf"></a> <!-- http://nco.sf.net/nco.html#nccf -->
@end html
@node Checking CF-compliance,  , Dismembering Files, Group Path Editing
@subsection Checking CF-compliance
@cindex @acronym{CF} compliance checker
@findex cfchecker
@findex ncdismember
@cindex compliance checker
@cindex Martin Schultz
@cindex Michael Decker
One application of dismembering is to check the @acronym{CF}-compliance of each 
group in a file. 
When invoked with the optional third argumnt @samp{cf},
@command{ncdismember} passes each file it generates to freely available 
compliance checkers, such as @command{cfchecker}
@footnote{CFchecker is developed by Michael Decker and Martin Schultz at
Forschungszentrum J@"ulich and distributed at
@uref{https://bitbucket.org/mde_/cfchecker}.}.
@example
@verbatim
zender@roulee:~$ ncdismember ~/nco/data/mdl_1.nc /data/zender/nco/tmp cf
NCO dismembering file /home/zender/nco/data/mdl_1.nc
CFchecker reports CF-compliance of each group in flat netCDF3 format
WARNING: Using the default (non-CF) Udunits database
cesm.cesm_01.nc: 
INFO: INIT:     running CFchecker version 1.5.15
INFO: INIT:     checking compliance with convention CF-1.5
INFO: INIT:     using standard name table version: 25, last modified: 2013-07-05T05:40:30Z
INFO: INIT:     using area type table version: 2, date: 10 July 2013
INFO: 2.4:      no axis information found in dimension variables, not checking dimension order
WARNING: 3:     variable "tas1" contains neither long_name nor standard_name attribute
WARNING: 3:     variable "tas2" contains neither long_name nor standard_name attribute
INFO: 3.1:      variable "tas1" does not contain units attribute
INFO: 3.1:      variable "tas2" does not contain units attribute
--------------------------------------------------
cesm.cesm_02.nc: 
...
@end verbatim
@end example
By default the @acronym{CF} version checked is determined automatically by
@command{cfchecker}. 
The user can override this default by supplying a supported @acronym{CF}
version, e.g., @samp{1.3}, as an optional fourth argument to
@command{ncdismember}. 
Current valid @acronym{CF} options are @samp{1.0}, @samp{1.1},
@samp{1.2}, @samp{1.3}, @samp{1.4}, and @samp{1.5}. 

@html
<a name="diwg"></a> <!-- http://nco.sf.net/nco.html#diwg -->
@end html
Our development and testing of @command{ncdismember} is funded by our
involvement in @acronym{NASA}'s Dataset Interoperability Working Group 
(@uref{https://wiki.earthdata.nasa.gov/display/ESDSWG/Dataset+Interoperability+Working+Group,
DIWG}), though our interest extends beyond @acronym{NASA} datasets.
Taken together, @acronym{NCO}'s features (autoconversion to netCDF3  
atomic types, fixing multiple record dimensions, autosensing
@acronym{HDF4} input, scoping rules for CF conventions) make 
@command{ncdismember} reliable and friendly for both dismembering 
hierarchical files and for @acronym{CF}-compliance checks. 
Most @acronym{HDF4} and @acronym{HDF5} datasets can be checked for 
@acronym{CF}-compliance with a one-line command. 
Example compliance checks of common @acronym{NASA} datasets are at
@uref{http://dust.ess.uci.edu/diwg}.
Our long-term goal is to enrich the hierarchical data model with the 
expressivity and syntactic power of @acronym{CF} conventions.

@html
<a name="daac"></a> <!-- http://nco.sf.net/nco.html#daac -->
@end html
@acronym{NASA} asked the @acronym{DIWG} to prepare a one-page summary
of the procedure necessary to check @acronym{HDF} files for
@acronym{CF}-compliance: 
@example
@verbatim
cat > ~/ncdismember.txt << 'EOF'
    Preparing an RPM-based OS to Test HDF & netCDF Files for CF-Compliance

By Charlie Zender, UCI & NASA Dataset Interoperability Working Group (DIWG)

Installation Summary:
1. HDF4 [with internal netCDF support _disabled_]
2. HDF5
3. netCDF [with external HDF4 support _enabled_]
4. NCO
5. numpy
6. netcdf4-python
7. python-lxml
8. CFunits-python
9. CFChecker
10. ncdismember

All 10 packages can use default installs _except_ HDF4 and netCDF.
Following instructions for Fedora Core 20 (FC20), an RPM-based Linux OS
Feedback and changes for other Linux-based OS's welcome to zender at uci.edu
${H4DIR}, ${H5DIR}, ${NETCDFDIR}, ${NCODIR}, may all be different
For simplicity CZ sets them all to /usr/local

# 1. HDF4. Build in non-default manner. Turn-off its own netCDF support.
# Per http://www.unidata.ucar.edu/software/netcdf/docs/build_hdf4.html
# HDF4 support not necessary though it makes ncdismember more comprehensive
wget -c http://www.hdfgroup.org/ftp/HDF/HDF_Current/src/hdf-4.2.9.tar.gz
tar xvzf hdf-4.2.9.tar.gz
cd hdf-4.2.9
./configure --enable-shared --disable-netcdf --disable-fortran --prefix=${H4DIR}
make && make check && make install

# 2. HDF5. Build normally. RPM may work too. Please let me know if so.
# HDF5 is a necessary pre-requisite for netCDF4
wget -c ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf-4/hdf5-1.8.11.tar.gz
tar xvzf hdf5-1.8.11.tar.gz
cd hdf5-1.8.11
./configure --enable-shared --prefix=${H5DIR}
make && make check && make install

# 3. netCDF version 4.3.1 or later. Build in non-default manner with HDF4.
# Per http://www.unidata.ucar.edu/software/netcdf/docs/build_hdf4.html
# Earlier versions of netCDF may fail checking some HDF4 files
wget -c ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf-4.3.2.tar.gz
tar xvzf netcdf-4.3.2.tar.gz
cd netcdf-4.3.2
CPPFLAGS="-I${H5DIR}/include -I${H4DIR}/include" \
LDFLAGS="-L${H5DIR}/lib -L${H4DIR}/lib" \
./configure --enable-hdf4 --enable-hdf4-file-tests
make && make check && make install

# 4. NCO version 4.4.0 or later. Some RPMs available. Or install by hand.
# Later versions of NCO have much better support for ncdismember
wget http://nco.sourceforge.net/src/nco-4.4.4.tar.gz .
tar xvzf nco-4.4.4.tar.gz
cd nco-4.4.4
./configure --prefix=${NCODIR}
make && make install

# 5. numpy
sudo yum install numpy -y

# 6. netcdf4-python
sudo yum install netcdf4-python -y

# 7. python-lxml
sudo yum install python-lxml -y

# 8. CFunits-python. No RPM available. Must install by hand.
# http://code.google.com/p/cfunits-python/
wget http://cfunits-python.googlecode.com/files/cfunits-0.9.6.tar.gz .
tar xvzf cfunits-0.9.6.tar.gz
cd cfunits-0.9.6
sudo python setup.py install

# 9. CFChecker. No RPM available. Must install by hand.
# https://bitbucket.org/mde_/cfchecker
wget https://bitbucket.org/mde_/cfchecker/downloads/CFchecker-1.5.15.tar.bz2 . 
tar xvjf CFchecker-1.5.15.tar.bz2 
cd CFchecker
sudo python setup.py install

# 10. ncdismember. Copy script from http://nco.sf.net/nco.html#ncdismember
# Store dismembered files somewhere, e.g., ${DATA}/nco/tmp/hdf
mkdir -p ${DATA}/nco/tmp/hdf
# Many datasets work with a simpler command...
ncdismember ~/nco/data/in.nc ${DATA}/nco/tmp/hdf cf 1.5
ncdismember ~/nco/data/mdl_1.nc ${DATA}/nco/tmp/hdf cf 1.5
ncdismember ${DATA}/hdf/AMSR_E_L2_Rain_V10_200905312326_A.hdf \
            ${DATA}/nco/tmp/hdf cf 1.5
ncdismember ${DATA}/hdf/BUV-Nimbus04_L3zm_v01-00-2012m0203t144121.h5 \
            ${DATA}/nco/tmp/hdf cf 1.5
ncdismember ${DATA}/hdf/HIRDLS-Aura_L3ZAD_v06-00-00-c02_2005d022-2008d077.he5 ${DATA}/nco/tmp/hdf cf 1.5
# Some datasets, typically .h5, require the --fix_rec_dmn=all argument
ncdismember_${DATA}/hdf/GATMO_npp_d20100906_t1935191_e1935505_b00012_c20110707155932065809_noaa_ops.h5 ${DATA}/nco/tmp/hdf cf 1.5 --fix_rec_dmn=all
ncdismember ${DATA}/hdf/mabel_l2_20130927t201800_008_1.h5 \
            ${DATA}/nco/tmp/hdf cf 1.5 --fix_rec_dmn=all
EOF
@end verbatim
@end example
@c scp ~/ncdismember.pdf dust.ess.uci.edu:/var/www/html/diwg
A @acronym{PDF} version of these instructions is available
@uref{http://dust.ess.uci.edu/diwg/ncdismember.pdf, here}.

@html
<a name="fortran"></a> <!-- http://nco.sf.net/nco.html#fortran -->
<a name="ftn"></a> <!-- http://nco.sf.net/nco.html#ftn -->
<a name="-F"></a> <!-- http://nco.sf.net/nco.html#-F -->
@end html
@node C and Fortran Index Conventions, Hyperslabs, Group Path Editing, Shared features
@section C and Fortran Index conventions
@cindex index convention
@cindex Fortran index convention
@cindex C index convention
@cindex @code{-F}
@cindex @code{--fortran}
@cartouche
Availability: @command{ncbo}, @command{nces}, @command{ncecat},
@command{ncflint}, @command{ncks}, @command{ncpdq}, @command{ncra},
@command{ncrcat}, @command{ncwa}@* 
Short options: @samp{-F}@*
Long options: @samp{--fortran}@*
@end cartouche
@cindex I/O
The @samp{-F} switch changes @acronym{NCO} to read and write with
the Fortran index convention. 
By default, @acronym{NCO} uses C-style (0-based) indices for all I/O. 
@w{In C}, indices count @w{from 0} (rather @w{than 1}), and
dimensions are ordered from slowest (inner-most) to fastest
(outer-most) varying.
In Fortran, indices count @w{from 1} (rather @w{than 0}), and
dimensions are ordered from fastest (inner-most) to slowest 
(outer-most) varying.  
@cindex transpose
Hence @w{C and} Fortran data storage conventions represent mathematical
transposes of eachother.
@cindex record variable
Note that record variables contain the record dimension as the most
slowly varying dimension.  
See @ref{ncpdq netCDF Permute Dimensions Quickly} for techniques
to re-order (including transpose) dimensions and to reverse data
storage order.

@cindex record dimension
Consider a file @file{85.nc} containing @w{12 months} of data in the
record dimension @code{time}.
The following hyperslab operations produce identical results, a
June-July-August average of the data:
@example
ncra -d time,5,7 85.nc 85_JJA.nc
ncra -F -d time,6,8 85.nc 85_JJA.nc
@end example

Printing variable @var{three_dmn_var} in file @file{in.nc} first with
the @w{C indexing} convention, then with Fortran indexing convention
results in the following output formats: 
@example
% ncks --trd -v three_dmn_var in.nc
lat[0]=-90 lev[0]=1000 lon[0]=-180 three_dmn_var[0]=0 
...
% ncks --trd -F -v three_dmn_var in.nc
lon(1)=0 lev(1)=100 lat(1)=-90 three_dmn_var(1)=0 
...
@end example

@html
<a name="-d"></a> <!-- http://nco.sf.net/nco.html#-d -->
<a name="-0"></a> <!-- http://nco.sf.net/nco.html#-0 -->
<a name="dmn"></a> <!-- http://nco.sf.net/nco.html#dmn -->
<a name="hyp"></a> <!-- http://nco.sf.net/nco.html#hyp -->
<a name="hyperslab"></a> <!-- http://nco.sf.net/nco.html#hyperslab -->
@end html
@node Hyperslabs, Stride, C and Fortran Index Conventions, Shared features
@section Hyperslabs 
@cindex hyperslab
@cindex dimension limits
@cindex coordinate limits
@cindex @code{-0}
@cindex @code{-d @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]}
@cindex @code{--dimension @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]}
@cindex @code{--dmn @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]}
@cartouche
Availability: @command{ncbo}, @command{nces}, @command{ncecat},
@command{ncflint}, @command{ncks}, @command{ncpdq}, @command{ncra},
@command{ncrcat}, @command{ncwa}@* 
Short options: @samp{-d @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]}@*
Long options: 
@samp{--dimension @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]},@* 
@samp{--dmn @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]}@*
@end cartouche
@w{A @dfn{hyperslab}} is a subset of a variable's data.
The coordinates of a hyperslab are specified with the 
@code{-d @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]} short
option (or with the same arguments to the @samp{--dimension} or
@samp{--dmn} long options).   
At least one hyperslab argument (@var{min}, @var{max}, or @var{stride})
must be present. 
The bounds of the hyperslab to be extracted are specified by the
associated @var{min} and @var{max} values. 
@w{A half}-open range is specified by omitting either the @var{min} or
@var{max} parameter.
The separating comma must be present to indicate the omission of one of
these arguments.
The unspecified limit is interpreted as the maximum or minimum value in 
the unspecified direction.  
@w{A cross}-section at a specific coordinate is extracted by specifying only
the @var{min} limit and omitting a trailing comma. 
Dimensions not mentioned are passed with no reduction in range.
The dimensionality of variables is not reduced (in the case of a
cross-section, the size of the constant dimension will be one). 
@example
# First and second longitudes
ncks -F -d lon,1,2 in.nc out.nc
# Second and third longitudes
ncks -d lon,1,2 in.nc out.nc
@end example

As of version 4.2.1 (August, 2012), @acronym{NCO} allows one to extract
the last @var{N} elements of a hyperslab.
Negative integers as @var{min} or @var{max} elements of a hyperslab
specification indicate offsets from the end (Python also uses this
convention). 
Consistent with this convention, the value @samp{-1} (negative one)
indicates the last element of a dimension, and negative zero is
algebraically equivalent to zero and so indicates the first element of a
dimension. 
Previously, for example, @samp{-d time,-2,-1} caused a domain error. 
Now it means select the penultimate and last timesteps, independent of
the size of the @code{time} dimension.
Select only the first and last timesteps, respectively, with 
@samp{-d time,0} and @samp{-d time,-1}.
Negative integers work for @var{min} and @var{max} indices, though not
for @var{stride}. 
@example
# Second through penultimate longitudes
ncks -d lon,1,-2 in.nc out.nc
# Second through last longitude
ncks -d lon,1,-1 in.nc out.nc
# Second-to-last to last longitude
ncks -d lon,-3,-1 in.nc out.nc
# Second-to-last to last longitude 
ncks -d lon,-3, in.nc out.nc
@end example
@noindent
The @samp{-F} argument, if any, applies the Fortran index convention
only to indices specified as positive integers:
@example
# First through penultimate longitudes
ncks -F -d lon,1,-2 in.nc out.nc (-F affects only start index)
# First through last longitude
ncks -F -d lon,1,-1 in.nc out.nc
# Second-to-last to penultimate longitude (-F has no effect)
ncks -F -d lon,-3,-1 in.nc out.nc
# Second-to-last to last longitude (-F has no effect)
ncks -F -d lon,-3, in.nc out.nc
@end example

@cindex stride
Coordinate values should be specified using real notation with a decimal 
point required in the value, whereas dimension indices are specified
using integer notation without a decimal point. 
This convention serves only to differentiate coordinate values from
dimension indices.
It is independent of the type of any netCDF coordinate variables.
In other words, even if coordinates are defined as integers, specify
them with decimal points to have the command interpret them as values,
rather than indices.
For a given dimension, the specified limits must both be coordinate
values (with decimal points) or dimension indices (no decimal points).

If values of a coordinate-variable are used to specify a range or
cross-section, then the coordinate variable must be monotonic (values
either increasing or decreasing). 
In this case, command-line values need not exactly match coordinate
values for the specified dimension. 
Ranges are determined by seeking the first coordinate value to occur in
the closed range [@var{min},@var{max}] and including all subsequent
values until one falls outside the range. 
The coordinate value for a cross-section is the coordinate-variable
value closest to the specified value and must lie within the range or
coordinate-variable values. 
The @var{stride} argument, if any, must be a dimension index, not a
coordinate value.
@xref{Stride}, for more information on the @var{stride} option.
@example
# All longitude values between 1 and 2 degrees
ncks -d lon,1.0,2.0 in.nc out.nc
# All longitude values between 1 and 2 degrees
ncks -F -d lon,1.0,2.0 in.nc out.nc
# Every other longitude value between 0 and 90 degrees
ncks -F -d lon,0.0,90.0,2 in.nc out.nc
@end example
As shown, we recommend using a full floating-point suffix of @code{.0}
instead of simply @code{.} in order to make obvious the selection of
hyperslab elements based on coordinate value rather than index.

@cindex @code{NC_CHAR}
User-specified coordinate limits are promoted to double-precision values 
while searching for the indices which bracket the range. 
Thus, hyperslabs on coordinates of type @code{NC_CHAR} are computed
numerically rather than lexically, so the results are unpredictable. 

@cindex wrapped coordinates
The relative magnitude of @var{min} and @var{max} indicate to the
operator whether to expect a @dfn{wrapped coordinate}
(@pxref{Wrapped Coordinates}), such as longitude.
If @math{@var{min} > @var{max}}, the @acronym{NCO} expects the
coordinate to be wrapped, and a warning message will be printed.
When this occurs, @acronym{NCO} selects all values outside the domain
[@math{@var{max} < @var{min}}], i.e., all the values exclusive of the
values which would have been selected if @var{min} and @var{max} were
swapped. 
If this seems confusing, test your command on just the coordinate
variables with @command{ncks}, and then examine the output to ensure
@acronym{NCO} selected the hyperslab you expected (coordinate wrapping
is currently only supported by @command{ncks}). 

Because of the way wrapped coordinates are interpreted, it is very
important to make sure you always specify hyperslabs in the
monotonically increasing sense, i.e., @math{@var{min} < @var{max}}
(even if the underlying coordinate variable is monotonically
decreasing). 
The only exception to this is when you are indeed specifying a wrapped
coordinate.  
The distinction is crucial to understand because the points selected by, 
e.g., @code{-d longitude,50.,340.}, are exactly the complement of the
points selected by @code{-d longitude,340.,50.}.

Not specifying any hyperslab option is equivalent to specifying full
ranges of all dimensions. 
This option may be specified more than once in a single command 
(each hyperslabbed dimension requires its own @code{-d} option).

@html
<a name="srd"></a> <!-- http://nco.sf.net/nco.html#srd -->
<a name="stride"></a> <!-- http://nco.sf.net/nco.html#stride -->
@end html
@node Stride, Record Appending, Hyperslabs, Shared features
@section Stride 
@cindex stride
@cindex @code{-d @var{dim},[@var{min}],[@var{max}],@var{stride}}
@cindex @code{--dimension @var{dim},[@var{min}],[@var{max}],@var{stride}}
@cindex @code{--dmn @var{dim},[@var{min}],[@var{max}],@var{stride}}
@cartouche
Availability: @command{ncbo}, @command{nces}, @command{ncecat},
@command{ncflint}, @command{ncks}, @command{ncpdq}, @command{ncra},
@command{ncrcat}, @command{ncwa}@* 
Short options: @samp{-d @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]}@*
Long options: 
@samp{--dimension @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]},@* 
@samp{--dmn @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]}@*
@end cartouche
All data operators support specifying a @dfn{stride} for any and all
dimensions at the same time.
The @var{stride} is the spacing between consecutive points in a
hyperslab. 
@w{A @var{stride}} @w{of 1} picks all the elements of the hyperslab, and
a @var{stride} @w{of 2} skips every other element, etc.@.
@command{ncks} multislabs support strides, and are more powerful than
the regular hyperslabs supported by the other operators
(@pxref{Multislabs}).
Using the @var{stride} option for the record dimension with
@command{ncra} and @command{ncrcat} makes it possible, for instance, to
average or concatenate regular intervals across multi-file input data sets.

The @var{stride} is specified as the optional fourth argument to the
@samp{-d} hyperslab specification:  
@code{-d @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]}.
Specify @var{stride} as an integer (i.e., no decimal point) following
the third comma in the @samp{-d} argument.  
There is no default value for @var{stride}. 
Thus using @samp{-d time,,,2} is valid but @samp{-d time,,,2.0} and
@samp{-d time,,,} are not.
When @var{stride} is specified but @var{min} is not, there is an
ambiguity as to whether the extracted hyperslab should begin with (using
C-style, 0-based indexes) @w{element 0} or element @samp{stride-1}.
@acronym{NCO} must resolve this ambiguity and it chooses @w{element 0}
as the first element of the hyperslab when @var{min} is not specified.
Thus @samp{-d time,,,@var{stride}} is syntactically equivalent to
@samp{-d time,0,,@var{stride}}.
This means, for example, that specifying the operation 
@samp{-d time,,,2} on the array @samp{1,2,3,4,5} selects the hyperslab
@samp{1,3,5}. 
To obtain the hyperslab @samp{2,4} instead, simply explicitly specify
the starting index @w{as 1,} i.e., @samp{-d time,1,,2}. 

For example, consider a file @file{8501_8912.nc} which contains 60
consecutive months of data. 
Say you wish to obtain just the March data from this file.
Using 0-based subscripts (@pxref{C and Fortran Index Conventions}) these 
data are stored in records @w{2, 14, @dots{} 50} so the desired
@var{stride} @w{is 12.}
Without the @var{stride} option, the procedure is very awkward.
One could use @command{ncks} five times and then use @command{ncrcat} to  
concatenate the resulting files together:
@cindex Bourne Shell
@cindex C Shell
@example
@verbatim
for idx in 02 14 26 38 50; do # Bourne Shell
  ncks -d time,${idx} 8501_8912.nc foo.${idx}
done
foreach idx (02 14 26 38 50) # C Shell
  ncks -d time,${idx} 8501_8912.nc foo.${idx}
end
ncrcat foo.?? 8589_03.nc
rm foo.??
@end verbatim
@end example
With the @var{stride} option, @command{ncks} performs this hyperslab
extraction in one operation:
@example
ncks -d time,2,,12 8501_8912.nc 8589_03.nc
@end example
@xref{ncks netCDF Kitchen Sink}, for more information on @command{ncks}.

Applying the @var{stride} option to the record dimension in
@command{ncra} and @command{ncrcat} makes it possible, for instance, to
average or concatenate regular intervals across multi-file input data
sets. 
@example
ncra -F -d time,3,,12 85.nc 86.nc 87.nc 88.nc 89.nc 8589_03.nc
ncrcat -F -d time,3,,12 85.nc 86.nc 87.nc 88.nc 89.nc 8503_8903.nc
@end example

@html
<a name="rec_apn"></a> <!-- http://nco.sf.net/nco.html#rec_apn -->
<a name="record_append"></a> <!-- http://nco.sf.net/nco.html#record_append -->
@end html
@node Record Appending, Subcycle, Stride, Shared features
@section Record Appending
@cindex record append
@cindex @code{--rec_apn}
@cindex @code{--record_append}
@cartouche
Availability: @command{ncra}, @command{ncrcat}@* 
Short options: None@*
Long options: 
@samp{--rec_apn}, @samp{--record_append}@*
@end cartouche
As of version 4.2.6 (March, 2013), @acronym{NCO} allows both
Multi-File, Multi-Record operators (@command{ncra} and @command{ncrcat})
to append their output directly to the end of an existing file.
This feature may be used to augment a target file, rather than construct
it from scratch. 
This helps, for example, when a timeseries is concatenated from input
data that becomes available in stages rather than all at once.
In such cases this switch significantly speeds writing.

Consider the use case where one wishes to preserve the contents of
@file{fl_1.nc}, and add to them new records contained in
@file{fl_2.nc}. 
Previously the output had to be placed in a third file, @file{fl_3.nc}
(which could also safely be named @file{fl_2.nc}), via
@example
ncrcat -O fl_1.nc fl_2.nc fl_3.nc
@end example
Under the hood this operation copies all information in
@file{fl_1.nc} and @file{fl_2.nc} not once but twice.
The first copy is performed through the netCDF interface, as all
data from @file{fl_1.nc} and @file{fl_2.nc} are extracted and placed in
the output file.
The second copy occurs (usually much) more quickly as the (by default)
temporary output file is copied (sometimes a quick re-link suffices) to
the final output file (@pxref{Temporary Output Files}).
All this copying is expensive for large files. 

The @samp{--record_append} switch appends all records in @file{fl_2.nc} 
to the end (after the last record) of @file{fl_1.nc}: 
@example
ncrcat --rec_apn fl_2.nc fl_1.nc
@end example
The ordering of the filename arguments may seem non-intuitive.
If the record variable represents time in these files, then the
values in @file{fl_1.nc} precede those in @file{fl_2.nc}, so why
do the files appear in the reverse order on the command line?
@file{fl_1.nc} is the last file named because it is the pre-existing
output file to which we will append all the other input files listed
(in this case only @file{fl_2.nc}).
The contents of @file{fl_1.nc} are completely preserved, and only
values in @file{fl_2.nc} (and any other input files) are copied. 
This switch avoids the necessity of copying all of @file{fl_1.nc} 
through the netCDF interface to a new output file.
The @samp{--rec_apn} switch automatically puts @acronym{NCO} into 
append mode (@pxref{Appending Variables}), so specifying @samp{-A} is
redundant, and simultaneously specifying overwrite mode with @samp{-O}
causes an error.  
By default, NCO works in an intermediate temporary file.
Power users may combine @samp{--rec_apn} with the @samp{--no_tmp_fl}
switch (@pxref{Temporary Output Files}):
@example
ncrcat --rec_apn --no_tmp_fl fl_2.nc fl_1.nc
@end example
This avoids creating an intermediate file, and copies only the
minimal amount of data (i.e., all of @file{fl_2.nc}). 
Hence, it is fast.
We recommend users try to understand the safety trade-offs involved. 

One side-effect of @samp{--rec_apn} to be aware of is how attributes
are handled.
When appending files, @acronym{NCO} typically overwrites attributes
for existing variables in the destination file with the corresponding
attributes from the same variable in the source file.
The exception to this rule is when @samp{--rec_apn} is invoked.
As of version 4.7.9 (January, 2019), @acronym{NCO} leaves unchanged
the attributes for existing variables in the destination file.
This is primarily to ensure that calendar attributes (e.g.,
@code{units}, @code{calendar}) of the record coordinate, if any, are
maintained, so that the data appended to them can be re-based to the
existing units.
Otherwise rebasing would fail or require rewriting the entire file
which is counter to the purpose of @samp{--rec_apn}.

@html
<a name="subcycle"></a> <!-- http://nco.sf.net/nco.html#subcycle -->
<a name="ssc"></a> <!-- http://nco.sf.net/nco.html#ssc -->
<a name="duration"></a> <!-- http://nco.sf.net/nco.html#duration -->
<a name="drn"></a> <!-- http://nco.sf.net/nco.html#drn -->
<a name="mro"></a> <!-- http://nco.sf.net/nco.html#mro -->
@end html
@node Subcycle, Interleave, Record Appending, Shared features
@section Subcycle
@cindex duration
@cindex sub-cycle
@cindex subcycle
@cindex MRO
@cindex Multi-Record Operator
@cindex @code{--mro}
@cindex @code{-d @var{dim},[@var{min}],[@var{max}],[@var{stride}],[@var{subcycle}]}
@cindex @code{--dimension @var{dim},[@var{min}],[@var{max}],[@var{stride}],[@var{subcycle}]}
@cindex @code{--dmn @var{dim},[@var{min}],[@var{max}],[@var{stride}],@var{subcycle}]}
@cartouche
Availability: @command{ncra}, @command{ncrcat}@* 
Short options: @samp{-d @var{dim},[@var{min}][,[@var{max}][,[@var{stride}][,[@var{subcycle}]]]]}@*
Long options: 
@samp{--mro}
@samp{--dimension @var{dim},[@var{min}][,[@var{max}][,[@var{stride}][,[@var{subcycle}]]]]}@*
@samp{--dmn @var{dim},[@var{min}][,[@var{max}][,[@var{stride}][,[@var{subcycle}]]]]}@*
@end cartouche
As of version 4.2.1 (August, 2012), @acronym{NCO} allows both Multi-File,
Multi-Record operators, @command{ncra} and @command{ncrcat}, to extract
and operate on multiple groups of records. 
These groups may be connected to physical @emph{sub-cycles} of a
periodic nature, e.g., months of a year, or hours of a day. 
Or they may be thought of as groups of a specifed duration.
We call this the @dfn{subcycle feature}, sometimes abbreviated
@acronym{SSC}
@footnote{When originally released in 2012 this was called the
@dfn{duration feature}, and was abbreviated @acronym{DRN}.}.

The subcycle feature allows processing of groups of records separated
by regular intervals of records. 
It is perhaps best illustrated by an extended example that describes 
how to solve the same problem both with and without the @acronym{SSC}
feature. 

Creating seasonal cycles is a common task in climate data processing.
Suppose a 150-year climate simulation produces 150 output files, each
comprising 12 records, each record a monthly mean:
@file{1850.nc}, @file{1851.nc}, ... @file{1999.nc}.
Our goal is to create a single file that contains the climatological
summertime (June, July, and August, aka JJA) mean.
Traditionally, we would first compute the climatological monthly
mean for each month of summer. 
Each of these is a 150-year mean, i.e., 
@example
@verbatim
# Step 1: Create climatological monthly files clm06.nc..clm08.nc
for mth in {6..8}; do
  mm=`printf "%02d" $mth`
  ncra -O -F -d time,${mm},,12 -n 150,4,1 1850.nc clm${mm}.nc
done
# Step 2: Average climatological monthly files into summertime mean
ncra -O clm06 clm07.nc clm08.nc clm_JJA.nc
@end verbatim
@end example
@noindent
So far, nothing is unusual and this task can be performed by any
@acronym{NCO} version. 
The @acronym{SSC} feature makes obsolete the need for the shell loop
used in @w{Step 1} above. 

The new @acronym{SSC} option aggregates more than one input record at
a time before performing arithmetic operations, and, with an
additional switch, allows archival of those results in
multiple-record output (MRO) files.  
This reduces the task of producing the climatological summertime
mean to @emph{one} step:
@example
# Step 1: Compute climatological summertime mean
ncra -O -F -d time,6,,12,3 -n 150,4,1 1850.nc clm_JJA.nc
@end example
@noindent
The @acronym{SSC} option instructs @command{ncra} (or @command{ncrcat})
to process files in groups of three records. 
To better understand the meaning of each argument to the @samp{-d}
hyperslab option, read it this way: ``for the time dimension start with
the sixth record, continue without end, repeat the process every twelfth
record, and define a sub-cycle as three consecutive records''. 

A separate option, @samp{--mro}, instructs @command{ncra} to output 
its results from each sub-group, and to produce a @dfn{Multi-Record Output}
(@acronym{MRO}) file rather than a @dfn{Single-Record Output}
(@acronym{SRO}) file. 
Unless Multi-Record-Output is indicated (either with @samp{--mro} or
implicitly, as with interleave-mode), @command{ncra} collects together 
all sub-groups, operates on their ensemble, and produces a single
output record. 
Adding @samp{--mro} to the above example causes @command{ncra} to
archive all (150) annual summertime means to one file: 
@example
# Step 1: Archive all 150 summertime means in one file
ncra --mro -O -F -d time,6,,12,3 -n 150,4,1 1850.nc 1850_2009_JJA.nc
# ...or all (150) annual means...
ncra --mro -O -d time,,,12,12 -n 150,4,1 1850.nc 1850_2009.nc
@end example
@noindent
These operations generate and require no intermediate files.
This contrasts to previous @acronym{NCO} methods, which require
generating, averaging, then catenating 150 files.  
The @samp{--mro} option only works on @command{ncra} and has no effect
on (or rather is redundant for) @command{ncrcat}, since
@command{ncrcat} always outputs all selected records.

@html
<a name="interleave"></a> <!-- http://nco.sf.net/nco.html#interleave -->
<a name="ilv"></a> <!-- http://nco.sf.net/nco.html#ilv -->
@end html
@node Interleave, Multislabs, Subcycle, Shared features
@section Interleave
@cindex interleave
@cindex @code{--mro}
@cindex @code{-d @var{dim},[@var{min}],[@var{max}],[@var{stride}],[@var{subcycle}],[@var{interleave}]}
@cindex @code{--dimension @var{dim},[@var{min}],[@var{max}],[@var{stride}],[@var{subcycle}],[@var{interleave}]}
@cindex @code{--dmn @var{dim},[@var{min}],[@var{max}],[@var{stride}],@var{subcycle}],[@var{interleave}]}
@cartouche
Availability: @command{ncra}, @command{ncrcat}@* 
Short options: @samp{-d @var{dim},[@var{min}][,[@var{max}][,[@var{stride}][,[@var{subcycle}][,[@var{interleave}]]]]]}@*
Long options: 
@samp{--mro}
@samp{--dimension @var{dim},[@var{min}][,[@var{max}][,[@var{stride}][,[@var{subcycle}][,[@var{interleave}]]]]]}@*
@samp{--dmn @var{dim},[@var{min}][,[@var{max}][,[@var{stride}][,[@var{subcycle}][,[@var{interleave}]]]]]}@*
@end cartouche
As of version 4.9.4 (September, 2020), @acronym{NCO} allows both Multi-File,
Multi-Record operators, @command{ncra} and @command{ncrcat}, to
extract, interleave, and operate on multiple groups of records. 
Interleaving (or de-interleaving, depending on one's perspective)
means altering the order of records in a group to be processed.
Specifically, the interleaving feature (sometimes abbreviated
@acronym{ILV}) causes the operator to treat as sequential records
those that are separated by multiples of the specified
@var{interleave} parameter within a group or sub-cycle of records.

The interleave feature sequences records with respect to their
position relative to the beginning of each sub-cycle.
Records a multiple of @var{interleave} from sub-cycle beginning
are first extracted (@command{ncrcat}) or reduced (@command{ncra}),
then records offset from these by one, two, et cetera up to
@math{@var{interleave}-1}. 
In this manner interleaving extracts an inner (intra-sub-cycle) loop  
that preserves high-frequency signals relative to the longer
stride between sub-cycles.
Thus interleaving allows deconvolution of periodic phenomena within a 
time-series.

Processing simple arithmetic sequences is a helpful way to 
understand what interleaving does. 
Here are some examples to reify the abstract.
Let @file{in1.nc} contain the record-array [1..10],
@file{in2.nc} contain [11..20], and @file{in12.nc} contain [1..20].
@example
@verbatim
ncra   -d time,,,,10,5 in1.nc ~/foo.nc # 3.5, 4.5, 5.5, 6.5, 7.5
ncrcat -d time,0,4,,6,2 in1.nc ~/foo.nc # 1, 3, 5, 2, 4, 6 (+WARNING)
ncrcat -d time,2,,10,4,2 in12.nc ~/foo.nc # 3, 5, 4, 6, 13, 15, 14, 16
ncra   -d time,2,,10,4,2 in12.nc ~/foo.nc # 4, 5, 14, 15
ncra   -d time,,,,10,2 in1.nc in2.nc ~/foo.nc # 5, 6, 15, 16
ncra   -d time,,,,10,2 in12.nc ~/foo.nc # 5, 6, 15, 16
@end verbatim
@end example

Interleaving is perhaps best illustrated by an extended example that
describes how to solve the same problem both with and without the
@acronym{ILV} feature. 
Consider as an example an interannual timeseries archived at a
high-enough temporal frequency to resolve the diurnal cycle with
@var{tpd} timesteps-per-day.
Many climate models and re-analyses are archived at hourly,
tri-hourly, or six-hourly resolution yielding
@math{@var{tpd} = 24, 8}, or @math{6}, respectively.
Our goal is to extract a monthly mean diurnal cycle from this
timeseries.

Suppose a 150-year climate simulation produces 150 output files, each
comprising 365 days of hourly data, or 8760 records, each record an hourly mean:
@file{1850.nc}, @file{1851.nc}, ... @file{1999.nc}.
Our goal is to create a single file that contains the climatological
monthly mean diurnal cycle for, say, March, which contains @w{31 days}
or @w{744 hourly} records that commence on the 60th day of the 356-day
year, with record index 1416.
Traditionally, we might first compute the climatological monthly
mean for hour of the day, then combine those into a full diurnal cycle:
@example
@verbatim
# Step 1: Create climatological hourly files hr00.nc..hr23.nc 
for hr in {0..23}; do
  hh=`printf "%02d" $hr`
  let srt=${hr}+1416
  # Alternatively, use UDUnits by setting srt=1850-03-01T00:00:01 
  ncra -O -d time,${srt},,8760 -n 150,4,1 1850.nc hr${hh}.nc
done
# Step 2: Concatenate climatological hourly files into diurnal cycle
ncrcata -O hr??.nc clm_drn.nc
@end verbatim
@end example
@noindent
So far, nothing is unusual and this task can be performed by any
@acronym{NCO} version. 
The @acronym{ILV} feature obsoletes the need for the shell loop
used in @w{Step 1} above. 

The new @acronym{ILV} option aggregates more than one input record at
a time before performing arithmetic operations, and, with an
additional switch, allows archival of those results in
multiple-record output (MRO) files.  
This reduces the task of producing the climatological summertime
mean to @emph{one} step:
@example
# Step 1: Archive all 150 March-mean diurnal cycles in one file
ncra -O -d time,1850-03-01T00:00:01,,8760,744,24 -n 150,4,1 1850.nc clm_drn.nc
@end example
@noindent
The @acronym{ILV} option instructs @command{ncra} (or @command{ncrcat})
to process files in groups of 31 days (744 hourly records) interleaved
with a 24-record cycle.
The end result will have 150 sets of 24-timesteps representing the
diurnal cycle of March in every year.
A given timestep is the mean of the same hour of the day for every day
in March of that year.

@html
<a name="multislab"></a> <!-- http://nco.sf.net/nco.html#multislab -->
<a name="multislabs"></a> <!-- http://nco.sf.net/nco.html#multislabs -->
<a name="msa"></a> <!-- http://nco.sf.net/nco.html#msa -->
<a name="mlt"></a> <!-- http://nco.sf.net/nco.html#mlt -->
@end html
@node Multislabs, Wrapped Coordinates, Interleave, Shared features
@section Multislabs 
@cindex multislab
@cindex multi-hyperslab
@cindex @acronym{MSA}
@cindex @code{-d @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]}
@cindex @code{--dimension @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]}
@cindex @code{--dmn @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]}
@cindex @code{--msa}
@cindex @code{--msa_usr_rdr}
@cindex @code{--msa_user_order}
@cartouche
Availability: @command{ncbo}, @command{nces}, @command{ncecat},
@command{ncflint}, @command{ncks}, @command{ncpdq}, @command{ncra},
@command{ncrcat}@* 
Short options: @samp{-d @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]}@*
Long options: 
@samp{--dimension @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]},@* 
@samp{--dmn @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]}@*
@samp{--msa_usr_rdr}, @samp{--msa_user_order}@*
@end cartouche
A @w{multislab} is a union of one or more hyperslabs.
One defines multislabs by chaining together hyperslab commands, i.e., 
@kbd{-d} options (@pxref{Hyperslabs}).
Support for specifying a @dfn{multi-hyperslab} or @dfn{multislab} for
any variable was first added to @command{ncks} in late 2002.
The other operators received these capabilities in April 2008.
Multi-slabbing is often referred to by the acronym @acronym{MSA},
which stands for ``Multi-Slabbing Algorithm''.
As explained below, the user may additionally request that the
multislabs be returned in the user-specified order, rather than the
on-disk storage order. 
Although @acronym{MSA} user-ordering has been available in all operators
since 2008, most users were unaware of it since the documentation
(below, and in the man pages) was not written until July 2013.

Multislabs overcome many restraints that limit simple hyperslabs.
@w{A single} @kbd{-d} option can only specify a contiguous and/or
a regularly spaced multi-dimensional data array.
Multislabs are constructed from multiple @kbd{-d} options and may
therefore have non-regularly spaced arrays.
For example, suppose it is desired to operate on all longitudes
from 10.0 to 20.0 and from 80.0 to @w{90.0 degrees}.
The combined range of longitudes is not selectable in a single 
hyperslab specfication of the form 
@samp{-d @var{dimension},@var{min},@var{max}} or  
@samp{-d @var{dimension},@var{min},@var{max},@var{stride}} because its
elements are irregularly spaced in coordinate space (and presumably 
in index space too). 
The multislab specification for obtaining these values is simply
the union of the hyperslabs specifications that comprise the multislab,
i.e., 
@example
ncks -d lon,10.,20. -d lon,80.,90. in.nc out.nc
ncks -d lon,10.,15. -d lon,15.,20. -d lon,80.,90. in.nc out.nc
@end example
@noindent
Any number of hyperslabs specifications may be chained together
to specify the multislab.
@acronym{MSA} creates an output dimension equal in size to the sum of
the sizes of the multislabs.
This can be used to extend and or pad coordinate grids.

@cindex stride
Users may specify redundant ranges of indices in a multislab, e.g., 
@example
ncks -d lon,0,4 -d lon,2,9,2 in.nc out.nc
@end example
@noindent
This command retrieves the first five longitudes, and then every other
longitude value up to the tenth.
Elements 0, 2, @w{and 4} are specified by both hyperslab arguments (hence
this is redundant) but will count only once if an arithmetic operation
is being performed.  
This example uses index-based (not coordinate-based) multislabs because
the @var{stride} option only supports index-based hyper-slabbing. 
@xref{Stride}, for more information on the @var{stride} option.

Multislabs are more efficient than the alternative of sequentially
performing hyperslab operations and concatenating the results.
@cindex I/O
This is because @acronym{NCO} employs a novel multislab algorithm to
minimize the number of I/O operations when retrieving irregularly spaced
data from disk.
The @acronym{NCO} multislab algorithm retrieves each element from disk
once and only once.
Thus users may take some shortcuts in specifying multislabs and the
algorithm will obtain the intended values.
Specifying redundant ranges is not encouraged, but may be useful on
occasion and will not result in unintended consequences.

Suppose the @var{Q} variable contains three dimensional arrays of
distinct chemical constituents in no particular order.
We are interested in the NOy species in a certain geographic range. 
Say that NO, NO2, and N2O5 are @w{elements 0}, 1, @w{and 5} of the
@var{species} dimension of @var{Q}.
The multislab specification might look something like
@example
ncks -d species,0,1 -d species,5 -d lon,0,4 -d lon,2,9,2 in.nc out.nc
@end example
@noindent
Multislabs are powerful because they may be specified for every
dimension at the same time.
Thus multislabs obsolete the need to execute multiple @command{ncks}
commands to gather the desired range of data.

@html
<a name="msa_usr_rdr"></a> <!-- http://nco.sf.net/nco.html#msa_usr_rdr -->
<a name="rotate_longitude"></a> <!-- http://nco.sf.net/nco.html#rotate_longitude -->
@end html
The @acronym{MSA} user-order switch @samp{--msa_usr_rdr} (or
@samp{--msa_user_order}, both of which shorten to @samp{--msa}) 
requests that the multislabs be output in the user-specified
order from the command-line, rather than in the input-file on-disk
storage order.  
This allows the user to perform complex data re-ordering in one
operation that would otherwise require cumbersome steps of
hyperslabbing, concatenating, and permuting. 
Consider the example of converting datasets stored with the longitude
coordinate @code{Lon} ranging from [@minus{}180,180) to datasets that
follow the [0,360) convention.
@example
% ncks -H -v Lon in.nc
Lon[0]=-180
Lon[1]=-90
Lon[2]=0
Lon[3]=90
@end example
@noindent
What is needed is a simple way to rotate longitudes.
Although simple in theory, this task requires both mathematics to
change the numerical value of the longitude coordinate, data
hyperslabbing to split the input on-disk arrays at Greenwich, and data
re-ordering within to stitch the western hemisphere onto the eastern
hemisphere at the date-line.
The @samp{--msa} user-order switch overrides the default that data are
output in the same order in which they are stored on-disk in the input
file, and instead stores them in the same order as the multi-slabs are
given to the command line.
This default is intuitive and is not important in most uses.
However, the @acronym{MSA} user-order switch allows users to meet
their output order needs by specifying multi-slabs in a certain order.
Compare the results of default ordering to user-ordering for longitude:
@example
% ncks -O -H       -v Lon -d Lon,0.,180. -d Lon,-180.,-1.0 in.nc
Lon[0]=-180 
Lon[1]=-90 
Lon[2]=0 
Lon[3]=90 
% ncks -O -H --msa -v Lon -d Lon,0.,180. -d Lon,-180.,-1.0 in.nc
Lon[0]=0 
Lon[1]=90 
Lon[2]=-180 
Lon[3]=-90 
@end example
@noindent
The two multi-slabs are the same but they can be presented to screen,
or to an output file, in either order. 
The second example shows how to place the western hemisphere after the
eastern hemisphere, although they are stored in the opposite order in
the input file. 

With this background, one sees that the following commands suffice to
rotate the input file by @w{180 degrees} longitude:
@example
% ncks -O -v LatLon --msa -d Lon,0.,180. -d Lon,-180.,-1.0 in.nc out.nc
% ncap2 -O -s 'where(Lon < 0) Lon=Lon+360' out.nc out.nc
% ncks --trd -C -H -v LatLon ~/nco/data/in.nc
Lat[0]=-45 Lon[0]=-180 LatLon[0]=0 
Lat[0]=-45 Lon[1]=-90 LatLon[1]=1 
Lat[0]=-45 Lon[2]=0 LatLon[2]=2 
Lat[0]=-45 Lon[3]=90 LatLon[3]=3 
Lat[1]=45 Lon[0]=-180 LatLon[4]=4 
Lat[1]=45 Lon[1]=-90 LatLon[5]=5 
Lat[1]=45 Lon[2]=0 LatLon[6]=6 
Lat[1]=45 Lon[3]=90 LatLon[7]=7 
% ncks --trd -C -H -v LatLon ~/out.nc
Lat[0]=-45 Lon[0]=0 LatLon[0]=2 
Lat[0]=-45 Lon[1]=90 LatLon[1]=3 
Lat[0]=-45 Lon[2]=180 LatLon[2]=0 
Lat[0]=-45 Lon[3]=270 LatLon[3]=1 
Lat[1]=45 Lon[0]=0 LatLon[4]=6 
Lat[1]=45 Lon[1]=90 LatLon[5]=7 
Lat[1]=45 Lon[2]=180 LatLon[6]=4 
Lat[1]=45 Lon[3]=270 LatLon[7]=5 
@end example
@noindent
The analogous commands to rotate all fields in a global dataset by
@w{180 degrees} in the other direction, i.e., from [0,360) to
[@minus{}180,180), are:
@example
ncks -O --msa -d lon,181.,360. -d lon,0.,180.0 in.nc out.nc
ncap2 -O -s 'where(lon > 180) lon=lon-360' out.nc out.nc
@end example

There are other workable, valid methods to rotate data, yet
none are simpler nor more efficient than utilizing @acronym{MSA}
user-ordering. 
Some final comments on applying this algorithm:
Be careful to specify hemispheres that do not overlap, e.g., by
inadvertently specifying coordinate ranges that both include Greenwich
or the date-line.
Some users will find using index-based rather than coordinate-based
hyperslabs makes this clearer.

@html
<a name="wrp"></a> <!-- http://nco.sf.net/nco.html#wrp -->
@end html
@node Wrapped Coordinates, Auxiliary Coordinates, Multislabs, Shared features
@section Wrapped Coordinates
@cindex wrapped coordinates
@cindex longitude
@cindex @code{-d @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]}
@cindex @code{--dimension @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]}
@cindex @code{--dmn @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]}
@cartouche
Availability: @command{ncks}@*
Short options: @samp{-d @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]}@*
Long options: 
@samp{--dimension @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]},@* 
@samp{--dmn @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]}@*
@end cartouche
@w{A @dfn{wrapped coordinate}} is a coordinate whose values increase or
decrease monotonically (nothing unusual so far), but which represents a
dimension that ends where it begins (i.e., wraps around on itself).
Longitude (i.e., degrees on a circle) is a familiar example of a wrapped
coordinate.
Longitude increases to the East of Greenwich, England, where it is
defined to be zero.
Halfway around the globe, the longitude is @w{180 degrees} East (or West). 
Continuing eastward, longitude increases to @w{360 degrees} East at
Greenwich. 
The longitude values of most geophysical data are either in the range
[0,360), or [@minus{}180,180).
In either case, the Westernmost and Easternmost longitudes are
numerically separated by @w{360 degrees}, but represent contiguous
regions on the globe.
For example, the Saharan desert stretches from roughly 340 to 
@w{50 degrees} East.
Extracting the hyperslab of data representing the Sahara from a global
dataset presents special problems when the global dataset is stored
consecutively in longitude from 0 to @w{360 degrees}.
This is because the data for the Sahara will not be contiguous in the
@var{input-file} but is expected by the user to be contiguous in the
@var{output-file}. 
In this case, @command{ncks} must invoke special software routines to
assemble the desired output hyperslab from multiple reads of the
@var{input-file}. 

Assume the domain of the monotonically increasing longitude coordinate
@code{lon} is @math{0 < @var{lon} < 360}. 
@command{ncks} will extract a hyperslab which crosses the Greenwich
meridian simply by specifying the westernmost longitude as @var{min} and
the easternmost longitude as @var{max}.
The following commands extract a hyperslab containing the Saharan desert:
@example
ncks -d lon,340.,50. in.nc out.nc
ncks -d lon,340.,50. -d lat,10.,35. in.nc out.nc
@end example
@noindent
The first example selects data in the same longitude range as the Sahara. 
The second example further constrains the data to having the same
latitude as the Sahara.
The coordinate @code{lon} in the @var{output-file}, @file{out.nc}, will
no longer be monotonic! 
The values of @code{lon} will be, e.g., @samp{340, 350, 0, 10, 20, 30,
40, 50}. 
This can have serious implications should you run @file{out.nc} through
another operation which expects the @code{lon} coordinate to be
monotonically increasing.
Fortunately, the chances of this happening are slim, since @code{lon}
has already been hyperslabbed, there should be no reason to hyperslab
@code{lon} again.
Should you need to hyperslab @code{lon} again, be sure to give
dimensional indices as the hyperslab arguments, rather than coordinate
values (@pxref{Hyperslabs}).

@html
<a name="aux"></a> <!-- http://nco.sf.net/nco.html#aux -->
<a name="auxiliary"></a> <!-- http://nco.sf.net/nco.html#auxiliary -->
<a name="-X"></a> <!-- http://nco.sf.net/nco.html#-X -->
<a name="std_nm"></a> <!-- http://nco.sf.net/nco.html#std_nm -->
<a name="standard_name"></a> <!-- http://nco.sf.net/nco.html#standard_name -->
@end html
@node Auxiliary Coordinates, Grid Generation, Wrapped Coordinates, Shared features
@section Auxiliary Coordinates
@cindex @code{-X}
@cindex @code{--auxiliary}
@cindex @code{standard_name}
@cindex @code{coordinates}
@cindex auxiliary coordinates
@cindex @acronym{CF} conventions
@cindex @code{-X @var{lon_min},@var{lon_max},@var{lat_min},@var{lat_max}}
@cindex @code{--auxiliary @var{lon_min},@var{lon_max},@var{lat_min},@var{lat_max}}
@cartouche
Availability: @command{ncbo}, @command{nces}, @command{ncecat},
@command{ncflint}, @command{ncks}, @command{ncpdq}, @command{ncra},
@command{ncrcat}@* 
Short options: @samp{-X @var{lon_min},@var{lon_max},@var{lat_min},@var{lat_max}}@*
Long options: 
@samp{--auxiliary @var{lon_min},@var{lon_max},@var{lat_min},@var{lat_max}}@*
@end cartouche
Utilize auxiliary coordinates specified in values of the coordinate
variable's @code{standard_name} attributes, if any, when interpreting 
hyperslab and multi-slab options. 
Also @samp{--auxiliary}.
This switch supports hyperslabbing cell-based grids (aka unstructured
grids) over coordinate ranges. 
When these grids are stored as 1D-arrays of cell data, this feature is
helpful at hyperslabbing and/or performing arithmetic on selected
geographic regions.
This feature cannot be used to select regions of 2D grids (instead use
the @command{ncap2} @code{where} statement for such grids 
@ref{Where statement}).
This feature works on datasets that associate coordinate variables to 
grid-mappings using the @acronym{CF}-convention (@pxref{CF Conventions})   
@code{coordinates} and @code{standard_name} attributes described 
@uref{http://cfconventions.org/cf-conventions/cf-conventions.html#coordinate-system, here}. 
Currently, @acronym{NCO} understands auxiliary coordinate variables 
pointed to by the @code{standard_name} attributes for @var{latitude} and  
@var{longitude}.   
Cells that contain a value within the user-specified
West-East-South-North (aka @acronym{WESN}) bounding box 
[@var{lon_min},@var{lon_max},@var{lat_min},@var{lat_max}] are
included in the output hyperslab.

The sides of the @acronym{WESN}) bounding box must be specified in
degrees (not radians).
The specified coordinates must be within the valid data range.
This includes boxes that wrap the origin of the longitude coordinate.
For example, if the longitude coordinate is stored in [0,360], then
a bounding box that straddles the Greenwich meridian in Africa would
be specified as, e.g., @math{[350,10,-20,20]}, not as
@math{[350,370,-20,20]}. 

@cindex unstructured grid
@cindex cell-based grid
A cell-based or unstructured grid collapses the horizontal spatial
information  (latitude and longitude) and stores it along a one-dimensional
coordinate that has a one-to-one mapping to both latitude and longitude
coordinates. 
Rectangular (in longitude and latitude) horizontal hyperslabs cannot
be selected using the typical procedure (@pxref{Hyperslabs}) of
separately specifying @samp{-d} arguments for longitude and latitude.
Instead, when the @samp{-X} is used, @acronym{NCO} learns the names of
the latitude and longitude coordinates by searching the
@code{standard_name} attribute of all variables until it finds
the two variables whose @code{standard_name}'s are ``latitude'' and 
``longitude'', respectively. 
This @code{standard_name} attribute for latitude and longitude
coordinates follows the @acronym{CF}-convention  
(@pxref{CF Conventions}). 

Putting it all together, consider a variable @var{gds_3dvar} output from 
simulations on a cell-based geodesic grid. 
Although the variable contains three dimensions of data (time, latitude,
and longitude), it is stored in the netCDF file with only two dimensions,
@code{time} and @code{gds_crd}.  
@example
% ncks -m -C -v gds_3dvar ~/nco/data/in.nc
gds_3dvar: type NC_FLOAT, 2 dimensions, 4 attributes, chunked? no, \
 compressed? no, packed? no, ID = 41
gds_3dvar RAM size is 10*8*sizeof(NC_FLOAT) = 80*4 = 320 bytes
gds_3dvar dimension 0: time, size = 10 NC_DOUBLE, dim. ID = 20 \ 
 (CRD)(REC)
gds_3dvar dimension 1: gds_crd, size = 8 NC_FLOAT, dim. ID = 17 (CRD)
gds_3dvar attribute 0: long_name, size = 17 NC_CHAR, value = \ 
 Geodesic variable
gds_3dvar attribute 1: units, size = 5 NC_CHAR, value = meter
gds_3dvar attribute 2: coordinates, size = 15 NC_CHAR, value = \
 lat_gds lon_gds
gds_3dvar attribute 3: purpose, size = 64 NC_CHAR, value = \ 
 Test auxiliary coordinates like those that define geodesic grids
@end example
The @code{coordinates} attribute lists the names of the latitude and
longitude coordinates, @code{lat_gds} and @code{lon_gds}, respectively. 
The @code{coordinates} attribute is recommended though optional.
With it, the user can immediately identify which variables contain
the latitude and longitude coordinates.
Without a @code{coordinates} attribute it would be unclear at first
glance whether a variable resides on a cell-based grid.
In this example, @code{time} is a normal record dimension and
@code{gds_crd} is the cell-based dimension.

The cell-based grid file must contain two variables whose
@code{standard_name} attributes are ``latitude'', and ``longitude'':
@example
% ncks -m -C -v lat_gds,lon_gds ~/nco/data/in.nc
lat_gds: type NC_DOUBLE, 1 dimensions, 4 attributes, \
 chunked? no, compressed? no, packed? no, ID = 37
lat_gds RAM size is 8*sizeof(NC_DOUBLE) = 8*8 = 64 bytes
lat_gds dimension 0: gds_crd, size = 8 NC_FLOAT, dim. ID = 17 (CRD)
lat_gds attribute 0: long_name, size = 8 NC_CHAR, value = Latitude
lat_gds attribute 1: standard_name, size = 8 NC_CHAR, value = latitude
lat_gds attribute 2: units, size = 6 NC_CHAR, value = degree
lat_gds attribute 3: purpose, size = 62 NC_CHAR, value = \ 
 1-D latitude coordinate referred to by geodesic grid variables

lon_gds: type NC_DOUBLE, 1 dimensions, 4 attributes, \
 chunked? no, compressed? no, packed? no, ID = 38
lon_gds RAM size is 8*sizeof(NC_DOUBLE) = 8*8 = 64 bytes
lon_gds dimension 0: gds_crd, size = 8 NC_FLOAT, dim. ID = 17 (CRD)
lon_gds attribute 0: long_name, size = 9 NC_CHAR, value = Longitude
lon_gds attribute 1: standard_name, size = 9 NC_CHAR, value = longitude
lon_gds attribute 2: units, size = 6 NC_CHAR, value = degree
lon_gds attribute 3: purpose, size = 63 NC_CHAR, value = \
 1-D longitude coordinate referred to by geodesic grid variables
@end example
In this example @code{lat_gds} and @code{lon_gds} represent the 
latitude or longitude, respectively, of cell-based variables.
These coordinates (must) have the same single dimension (@code{gds_crd},
in this case) as the cell-based variables.
And the coordinates must be one-dimensional---multidimensional
coordinates will not work.

This infrastructure allows @acronym{NCO} to identify, interpret, and
process (i.e., hyperslab) the variables on cell-based grids as easily
as it works with regular grids.
To time-average all the values between zero and @w{180 degrees}
longitude and between plus and minus @w{30 degress} latitude, we use
@example
ncra -O -X 0.,180.,-30.,30. -v gds_3dvar in.nc out.nc
@end example
@acronym{NCO} accepts multiple @samp{-X} arguments for cell-based grid
multi-slabs, just as it accepts multiple @samp{-d} arguments for 
multi-slabs of regular coordinates.
@example
ncra -O -X 0.,180.,-30.,30. -X 270.,315.,45.,90. in.nc out.nc
@end example
The arguments to @samp{-X} are always interpreted as floating-point
numbers, i.e., as coordinate values rather than dimension indices
so that these two commands produce identical results
@example
ncra -X 0.,180.,-30.,30. in.nc out.nc
ncra -X 0,180,-30,30 in.nc out.nc
@end example
By contrast, arguments to @samp{-d} require decimal places to be
recognized as coordinates not indices (@pxref{Hyperslabs}).  
We recommend always using decimal points with @samp{-X} arguments
to avoid confusion.

@html
<a name="scrip"></a> <!-- http://nco.sf.net/nco.html#scrip -->
<a name="grid"></a> <!-- http://nco.sf.net/nco.html#grid -->
<a name="grd"></a> <!-- http://nco.sf.net/nco.html#grd -->
@end html
@node Grid Generation, Regridding, Auxiliary Coordinates, Shared features
@section Grid Generation
@cindex Gaussian grid
@cindex Equi-Angular grid
@cindex FV grid
@cindex FV grid
@cindex CAM-FV grid
@cindex grid, Fixed
@cindex grid, Offset
@cindex grid, FV
@cindex grid, CAM-FV
@cindex grid, Equi-Angular
@cindex grid, Gaussian
@cindex gridfile
@cindex @code{--map}
@cindex @acronym{ESMF}
@cindex @acronym{SCRIP}
@cindex @code{--rgr @var{key}=@var{val}}
@cartouche
Availability: @command{ncks}@* 
Short options: None@*
Long options: 
@samp{--rgr @var{key}=@var{val}} (multiple invocations allowed)@*
@end cartouche

As of @acronym{NCO} version 4.5.2 (August, 2015), @command{ncks}
generates accurate and complete SCRIP-format gridfiles for select grid 
types, including uniform, capped and Gaussian rectangular,
latitude/longitude grids, global or regional.
The grids are stored in an external @var{grid-file}.

All options pertinent to the grid geometry and metadata are passed to
@acronym{NCO} via key-value pairs prefixed by the @samp{--rgr} option,
or its synonym, @samp{--regridding}.
@cindex indicator option
@cindex multi-arguments
The option @samp{--rgr} (and its long option equivalents such
as @samp{--regridding}) indicates the argument syntax will be
@var{key}=@var{val}.
As such, @samp{--rgr} and its synonyms are indicator options that accept
arguments supplied one-by-one like 
@samp{--rgr @var{key1}=@var{val1} --rgr @var{key2}=@var{val2}}, or
aggregated together in multi-argument format like
@samp{--rgr @var{key1}=@var{val1}#@var{key2}=@var{val2}}
(@pxref{Multi-arguments}).

The text strings that describe the grid and name the file are important 
aids to convey the grid geometry to other users.
These arguments, and their corresponding keys, are the grid title
(@var{grd_ttl}), and grid filename (@var{grid}), respectively.
The numbers of latitudes (@var{lat_nbr}) and longitudes (@var{lon_nbr})   
are independent, and together determine the grid storage size.
These four options should be considered mandatory, although
@acronym{NCO} provides defaults for any arguments omitted.

The remaining arguments depend on the whether the grid is global or
regional.
For global grids, one should specify only two more arguments, the
latitude (@var{lat_typ}) and longitude (@var{lon_typ}) grid-types.
These types are chosen as described below from a small selection of
options that together define the most common rectangular global grids.
For regional grids, one must specify the bounding box, i.e., the edges
of the rectangular grid on the North (@var{lat_nrt}), South (@var{lat_sth}), 
East (@var{lat_est}), and West (@var{lat_nrt}) sides. 
Specifying a bounding box for global grids is redundant and will cause
an error to ensure the user intends a global grid.
@acronym{NCO} assumes that regional grids are uniform, though it will 
attempt to produce regional grids of other types if the user specifies
other latitude (@var{lat_typ}) and longitude (@var{lon_typ}) grid-types,
e.g., Gaussian or Cap.
Edges of a regional bounding box may be specified individually, or in
the single-argument forms.

The full description of grid-generation arguments, and their
corresponding keys, is:
@table @dfn
@cindex @var{grd_ttl}
@cindex @samp{--rgr grd_ttl=@var{grd_ttl}}
@item Grid Title: @var{grd_ttl}
It is surprisingly difficult to discern the geometric configuration of
a grid from the coordinates of a @acronym{SCRIP}-format gridfile.  
A human-readable grid description should be placed in @var{grd_ttl}.
Examples include ``CAM-FV scalar grid 129x256'' and ``T42 Gaussian grid''. 

@cindex @var{scrip_grid}
@cindex @samp{--rgr grid=@var{scrip_grid}}
@cindex @samp{--rgr scrip=@var{scrip_grid}}
@item Grid File: @var{scrip_grid}
The grid-generation @acronym{API} was bolted-on to @acronym{NCO}
and contains some temporary kludges.
For example, the output grid filename is distinct from the output
filename of the host @command{ncks} command.  
Specify the output gridfile name @var{scrip_grid} with keywords
@code{grid} or @code{scrip}, e.g., @samp{--rgr grid=@var{scrip_grid}} or  
@samp{--rgr scrip=@file{t42_SCRIP.20150901.nc}}.
It is conventional to include a datestamp in the gridfile name.
This helps users identify up-to-date and out-of-date grids.
Any valid netCDF file may be named as the source (e.g., @file{in.nc}). 
It will not be altered. 
The destination file (e.g., @file{foo.nc}) will be overwritten. 
Its contents are immaterial. 

@cindex @var{lat_typ}
@cindex @var{lon_typ}
@cindex @samp{--rgr lon_typ=@var{lon_typ}}
@cindex @samp{--rgr lat_typ=@var{lat_typ}}
@item Grid Types: @var{lat_typ}, @var{lon_typ}
The keys that hold the longitude and latitude gridtypes (which
are, by the way, independent of eachother) are @var{lon_typ} and
@var{lat_typ}. 
The @var{lat_typ} options for global grids are @samp{uni} for
Uniform, @samp{cap} (or @samp{fv}) for Cap@footnote{
The term @acronym{FV} confusing because it is correct to call
any Finite Volume grid (including arbitrary polygons) an
@acronym{FV} grid.
However, an @acronym{FV} grid has also been used for many years
to described the particular type of rectangular grid with caps
at the poles used to discretize global model grids for use with
the Lin-Rood dynamical core.
To reduce confusion, we use ``Cap grid'' to refer to the latter
and reserv @acronym{FV} as a straightforward acronym for Finite
Volume.},
and @samp{gss} for Gaussian.

These values are all case-independent, so @samp{Gss} and @samp{gss} both
work.
As of version 4.7.7 (September, 2018), @acronym{NCO} generates perfectly
symmetric interface latitudes for Gaussian grids.
Previously the interface latitude generation mechanism could accumulate
small rounding errors (~1.0e-14).
Now symmetry properties are used to ensure perfect symmetry.
All other Gaussian grids we have seen compute interfaces as the
arithmetic mean of the adjacent Gaussian latitudes, which is patently
wrong. 
To our knowledge @acronym{NCO} is the only map software that generates
accurate interface latitudes for a Gaussian grid.
We use a Newton-Raphson iteration technique to identify the interface
latitudes that enclose the area indicated by the Gaussian weight.

As its name suggests, the latitudes in a Uniform-latitude grid are
uniformly spaced  
@footnote{
A Uniform grid in latitude could be called ``equi-angular'' in
latitude, but @acronym{NCO} reserves the term Equi-angular or ``eqa'' 
for grids that have the same uniform spacing in both latitude and
longitude, e.g., 1@textdegree{}x1@textdegree{} or
2@textdegree{}x2@textdegree{}. 
@acronym{NCO} reserves the term Regular to refer to grids that are
monotonic and rectangular grids. 
Confusingly, the angular spacing in a Regular grid need not be uniform,  
it could be irregular, such as in a Gaussian grid.
The term Regular is not too useful in grid-generation, because so many
other parameters (spacing, centering) are necessary to disambiguate it.}. 
The Uniform-latitude grid may have any number of latitudes. 
@acronym{NCO} can only generate longitude grids (below) that are
uniformly spaced, so the Uniform-latitude grids we describe are
also uniform in the 2D sense.
Uniform grids are intuitive, easy to visualize, and simple to program.
Hence their popularity in data exchange, visualization, and archives.
Moreover, regional grids (unless they include the poles), are free
of polar singularities, and thus are well-suited to storage on Uniform 
grids. 
Theoretically, a Uniform-latitude grid could have non-uniform
longitudes, but @acronym{NCO} currently does not implement non-uniform
longitude grids.

Their mathematical properties (convergence and excessive resolution at
the poles, which can appear as singularities) make Uniform grids fraught
for use in global models.  
One purpose Uniform grids serve in modeling is as ``offset'' or
``staggered'' grids, meaning grids whose centers are the interfaces of 
another grid. 
The Finite-Volume (@acronym{FV}) method is often used to represent 
and solve the equations of motion in climate-related fields.
Many @acronym{FV} solutions (including the popular Lin-Rood method as
used in the @acronym{CESM} @acronym{CAM-FV} atmospheric model) evaluate
scalar (i.e., non-vector) fields (e.g., temperature, water vapor) at
gridcell centers of what is therefore called the scalar grid. 
@acronym{FV} methods (like Lin-Rood) that employ an Arakawa C-grid or
D-grid formulation define velocities on the edges of the scalar grid.
This @acronym{CAM-FV} velocity grid is therefore ``staggered'' or
``offset'' from the @acronym{CAM-FV} scalar grid by one-half gridcell.  
The @acronym{CAM-FV} scalar latitude grid has gridpoints (the
``caps'') centered on each pole to avoid singularities.
The offset of a Cap-grid is a Uniform-grid, so the Uniform grid is 
often called an @acronym{FV}-''offset'' or ``staggered'' grid.
Hence an @acronym{NCO} Uniform grid is equivalent to an @acronym{NCL}  
``Fixed Offset'' grid.
For example, a 128x256 Uniform grid is the offset or staggered version
of a 129x256 Cap grid (aka @acronym{FV}-grid).

Referring the saucer-like cap-points at the poles, @acronym{NCO} uses
the term ``Cap grid'' to describe the latitude portion of the
@acronym{FV}-scalar grid as used by the @acronym{CAM-FV} Lin-Rood
dynamics formulation. 
@acronym{NCO} accepts the shorthand @acronym{FV}, and the more
descriptive ``Yarmulke'', as synonyms for Cap. 
A Cap-latitude grid differs from a Uniform-latitude grid in many ways:

Most importantly, Cap grids are 2D-representations of numerical grids
with cap-midpoints instead of zonal-teeth convergence at the poles.  
The rectangular 2D-representation of each cap contains gridcells shaped
like sharp teeth that converge at the poles similar to the Uniform grid,  
but the Cap gridcells are meant to be aggregated into a single cell
centered at the pole in a dynamical transport algorithm.  
In other words, the polar teeth are a convenient way to encode a
non-rectangular grid in memory into a rectangular array on disk.  
Hence Cap grids have the unusual property that the poles are labeled as
being both the centers and the outer interfaces of all polar gridcells. 
Second, Cap grids are uniform in angle except at the poles, where the
latitudes span half the meridional range of the rest of the gridcells. 
Even though in the host dynamical model the Cap grid polar points are
melded into caps uniform (in angle) with the rest of the grid, the disk
representation on disk is not uniform.
Nevertheless, some call the Cap grid a uniform-angle grid because the
information contained at the poles is aggregated in memory to span twice
the range of a single polar gridcell (which has half the normal width). 
@acronym{NCL} uses the term ``Fixed grid'' for a Cap grid.
The ``Fixed'' terminology seems broken.

@html
<a name="gss"></a> <!-- http://nco.sf.net/nco.html#gss -->
@end html
Finally, Gaussian grids are the Cartesian representation of global
spectral transform models. 
Gaussian grids typically have an even number of latitudes and so  
do not have points at the poles.
All three latitude grid-type supported by @acronym{NCO} (Uniform, Cap, 
and Gaussian) are Regular grids in that they are monotonic. 

The @var{lon_typ} options for global grids are @samp{grn_ctr} and
@samp{180_ctr} for the first gridcell centered at Greenwich or 
@w{180 degrees}, respecitvely.   
And @samp{grn_wst} and @samp{180_wst} for Greenwich or @w{180 degress}
lying on the western edge of the first gridcell.
Many global models use the @samp{grn_ctr} longitude grid as their
``scalar grid'' (where, e.g., temperature, humidity, and other scalars
are defined).
The ``staggered'' or ``offset'' grid (where often the dynamics variables
are defined) then must have the @samp{grn_wst} longitude convention.
That way the centers of the scalar grid are the vertices of the offset
grid, and visa versa.

@cindex @var{lat_nbr}
@cindex @var{lon_nbr}
@cindex @samp{--rgr latlon=@var{lat_nbr},@var{lon_nbr}}
@cindex @samp{--rgr lon_nbr=@var{lon_nbr}}
@cindex @samp{--rgr lat_nbr=@var{lat_nbr}}
@item Grid Resolution: @var{lat_nbr}, @var{lon_nbr}
The number of gridcells in the horizontal spatial dimensions
are @var{lat_nbr} and @var{lon_nbr}, respectively.
There are no restrictions on @var{lon_nbr} for any gridtype.
Latitude grids do place some restrictions on @var{lat_nbr} (see above). 
As of @acronym{NCO} @w{version 4.5.3}, released in October, 2015,
the @samp{--rgr latlon=@var{lat_nbr},@var{lon_nbr}} switch may be
used to simultaneously specify both latitude and longitude, e.g., 
@samp{--rgr latlon=180,360}.

@html
<a name="lat_drc"></a> <!-- http://nco.sf.net/nco.html#lat_drc -->
<a name="lat_drc"></a> <!-- http://nco.sf.net/nco.html#n2s -->
<a name="lat_drc"></a> <!-- http://nco.sf.net/nco.html#s2n -->
@end html
@cindex @code{n2s}
@cindex @code{s2n}
@cindex @var{lat_drc}
@cindex @samp{--rgr lat_drc=@var{lat_drc}}
@item Latitude Direction: @var{lat_drc}
The @var{lat_drc} option is specifies whether latitudes monotonically
increase or decrease in rectangular grids.
The two possible values are
@samp{s2n} for grids that begin with the most southerly latitude and end
with the most northerly, and 
@samp{n2s} for grids that begin with the most northerly latitude and end
with the most southerly.
By default @acronym{NCO} creates grids whose latitudes run
south-to-north.
Hence this option is only necessary to create a grid whose latitudes run
north-to-south.

@cindex @var{lat_nrt}
@cindex @var{lat_sth}
@cindex @var{lat_est}
@cindex @var{lat_wst}
@cindex @var{wesn}
@cindex @var{snwe}
@cindex @samp{--rgr wesn=@var{lon_wst},@var{lon_est},@var{lat_sth},@var{lon_nrt}}
@cindex @samp{--rgr snwe=@var{lat_sth},@var{lat_nrt},@var{lon_wst},@var{lon_est}}
@cindex @samp{--rgr lat_nrt=@var{lat_nrt}}
@cindex @samp{--rgr lat_nbr=@var{lat_nbr}}
@item Grid Edges: @var{lon_wst}, @var{lon_est}, @var{lat_sth}, @var{lat_nrt}
The outer edges of a regional rectangular grid are specified by
the North (@var{lat_nrt}), South (@var{lat_sth}), East (@var{lat_est}),
and West (@var{lat_nrt}) sides. 
Latitudes and longigudes must be specified in degrees (not radians).
Latitude edges must be between @w{-90 and 90}.
Longitude edges may be positive or negative and separated by no more
than 360 degrees.
The edges may be specified individually with four arguments, 
consecutively separated by the multi-argument delimiter (@samp{#} by
default), or together in a short list to the pre-ordered options
@samp{wesn} or @samp{snwe}. 
These three specifications are equivalent:
@example
@verbatim
ncks ... --rgr lat_sth=30.0 --rgr lat_nrt=70.0 --rgr lon_wst=-120.0 --rgr lon_est=-90.0 ...
ncks ... --rgr lat_sth=30.0#lat_nrt=70.0#lon_wst=-120.0#lon_est=-90.0 ...
ncks ... --rgr snwe=30.0,70.0,-120.0,-90.0 ...
@end verbatim
@end example
@end table
The first example above supplies the bounding box with four
@var{key}=@var{val} pairs. 
The second example above supplies the bounding box with a single option
in multi-argument format (@pxref{Multi-arguments}).
The third example uses a convenience switch introduced to reduce typing.

@html
<a name="grd_cmd"></a> <!-- http://nco.sf.net/nco.html#grd_cmd -->
@end html
Generating common grids:
@cindex @acronym{ECMWF IFS} grid
@cindex @acronym{ECMWF ERA5} grid
@cindex @acronym{NCEP2} grid
@cindex @acronym{NASA CMG} grid
@cindex @acronym{NASA MERRA2} grid
@cindex @acronym{CAM-FV} grid
@cindex @acronym{Equi-angular} grid
@example
@verbatim
# Through version 4.7.5 (August, 2018), ncks performed grid-generation
# 180x360 (1x1 degree) Equi-Angular grid, first longitude centered at Greenwich
ncks --rgr ttl='Equi-Angular grid 180x360'#latlon=180,360#lat_typ=uni#lon_typ=grn_ctr \
     --rgr scrip=${DATA}/grids/180x360_SCRIP.20150901.nc \
     ~zender/nco/data/in.nc ~/foo.nc

# As of version 4.7.6 (August, 2018), ncremap supports more concise commands
ncremap -G ttl='Equi-Angular grid 180x360'#latlon=180,360#lat_typ=uni#lon_typ=grn_ctr \
        -g ${DATA}/grids/180x360_SCRIP.20180901.nc

# 180x360 (1x1 degree) Equi-Angular grid, first longitude west edge at Greenwich
ncremap -G ttl='Equi-Angular grid 180x360'#latlon=180,360#lat_typ=uni#lon_typ=grn_wst \
        -g ${DATA}/grids/180x360wst_SCRIP.20180301.nc

# 129x256 CAM-FV grid, first longitude centered at Greenwich
ncremap -G ttl='CAM-FV scalar grid 129x256'#latlon=129,256#lat_typ=fv#lon_typ=grn_ctr \
        -g ${DATA}/grids/129x256_SCRIP.20150901.nc

# 192x288 CAM-FV grid, first longitude centered at Greenwich
ncremap -G ttl='CAM-FV scalar grid 192x288'#latlon=192,288#lat_typ=fv#lon_typ=grn_ctr \
        -g ${DATA}/grids/192x288_SCRIP.20160301.nc

# 361x576 NASA MERRA2 FV grid, first longitude centered at DateLine
ncremap -G ttl='NASA MERRA2 Cap grid 361x576'#latlon=361,576#lat_typ=cap#lon_typ=180_ctr \
        -g ${DATA}/grids/merra2_361x576.20201001.nc

# 1441x2880 CAM-FV grid, first longitude centered at Greenwich
ncremap -G ttl='CAM-FV scalar grid 1441x2880'#latlon=1441,2880#lat_typ=fv#lon_typ=grn_ctr \
        -g ${DATA}/grids/1441x2880_SCRIP.20170901.nc

# 1440x2880 ELM/MOSART grid, first longitude west edge at DateLine
ncremap -7 -L 1 \
        -G ttl='ELM/MOSART 1440x2880 one-eighth degree uniform grid (r0125)'#latlon=1440,2880#lat_typ=uni#lon_typ=180_wst \
        -g ${DATA}/grids/r0125_1440x2880.20210401.nc

# 91x180 CAM-FV grid, first longitude centered at Greenwich (2 degree grid)
ncremap -G ttl='CAM-FV scalar grid 91x180'#latlon=91,180#lat_typ=fv#lon_typ=grn_ctr \
        -g ${DATA}/grids/91x180_SCRIP.20170401.nc

# 25x48 CAM-FV grid, first longitude centered at Greenwich (7.5 degree grid)
ncremap -G ttl='CAM-FV scalar grid 25x48'#latlon=25,48#lat_typ=fv#lon_typ=grn_ctr \
        -g ${DATA}/grids/25x48_SCRIP.20170401.nc

# 128x256 Equi-Angular grid, Greenwich west edge of first longitude
# CAM-FV offset grid for 129x256 CAM-FV scalar grid above
ncremap -G ttl='Equi-Angular grid 128x256'#latlon=128,256#lat_typ=uni#lon_typ=grn_wst \
        -g ${DATA}/grids/128x256_SCRIP.20150901.nc

# T42 Gaussian grid, first longitude centered at Greenwich
ncremap -G ttl='T42 Gaussian grid'#latlon=64,128#lat_typ=gss#lon_typ=grn_ctr \
        -g ${DATA}/grids/t42_SCRIP.20180901.nc

# T62 Gaussian grid, first longitude centered at Greenwich, NCEP2 T62 Gaussian grid 
ncremap -G ttl='NCEP2 T62 Gaussian grid'#latlon=94,192#lat_typ=gss#lon_typ=grn_ctr#lat_drc=n2s \
        -g ${DATA}/grids/ncep2_t62_SCRIP.20191001.nc

# F256 Full Gaussian grid, first longitude centered at Greenwich
ncremap -7 -L 1 \
        -G ttl='ECMWF IFS F256 Full Gaussian grid 512x1024'#latlon=512,1024#lat_typ=gss#lon_typ=grn_ctr#lat_drc=n2s \
        -g ${DATA}/grids/f256_scrip.20201001.nc

# 513x1024 FV grid, first longitude centered at Greenwich
ncremap -7 -L 1 \
        -G ttl='FV scalar grid 513x1024'#latlon=513,1024#lat_typ=fv#lon_typ=grn_ctr \
        -g ${DATA}/grids/513x1024_SCRIP.20201001.nc

# 1025x2048 FV grid, first longitude centered at Greenwich
ncremap -7 -L 1 \
        -G ttl='FV scalar grid 1025x2048'#latlon=1025,2048#lat_typ=fv#lon_typ=grn_ctr \
        -g ${DATA}/grids/1025x2048_SCRIP.20201001.nc

# F640 Full Gaussian grid, first longitude centered at Greenwich
ncremap -7 -L 1 \
     -G ttl='ECMWF IFS F640 Full Gaussian grid 1280x2560'#latlon=1280,2560#lat_typ=gss#lon_typ=grn_ctr#lat_drc=n2s \
     -g ${DATA}/grids/f640_scrip.20190601.nc

# NASA Climate Modeling Grid (CMG) 3600x7200 (0.05x0.05 degree) Equi-Angular grid
# Date-line west edge of first longitude, east edge of last longitude
# Write to compressed netCDF4-classic file to reduce filesize ~140x from 2.2 GB to 16 MB
ncremap -7 -L 1 \
     -G ttl='Equi-Angular grid 3600x7200 (NASA CMG)'#latlon=3600,7200#lat_typ=uni#lon_typ=180_wst \
     -g ${DATA}/grids/3600x7200_SCRIP.20160301.nc

# DOE E3SM/ACME High Resolution Topography (1 x 1 km grid) for Elevation Classes
# Write to compressed netCDF4-classic file to reduce filesize from ~85 GB to 607 MB
ncremap -7 -L 1 \
     -G ttl='Global latxlon = 18000x36000 ~1 x 1 km'#latlon=18000,36000#lat_typ=uni#lon_typ=grn_ctr \
     -g ${DATA}/grids/grd_18000x36000_SCRIP.nc

# 1x1 degree Equi-Angular Regional grid over Greenland, centered longitudes
ncremap -G ttl='Equi-Angular Greenland 1x1 degree grid'#latlon=30,90#snwe=55.0,85.0,-90.0,0.0#lat_typ=uni#lon_typ=grn_ctr \
        -g ${HOME}/greenland_1x1.nc

# 721x1440 ECMWF ERA5 resolution in north-to-south order (ERA5/CAMS default order)
ncremap -7 --dfl_lvl=1 -G ttl='Cap/FV ECMWF ERA5 grid 0.25x0.25 degree, dimensions 721x1440, cell centers on Poles/Equator (in north-to-south order) and Prime Meridian/Date Line'#latlon=721,1440#lat_drc=n2s#lat_typ=cap#lon_typ=grn_ctr \
         -g ${DATA}/grids/era5_n2s_721x1440.nc

# 721x1440 ECMWF ERA5 resolution in south-to-north order (E3SM/ELM-offline forcing order)
ncremap -7 --dfl_lvl=1 -G ttl='Cap/FV ECMWF ERA5 grid 0.25x0.25 degree, dimensions 721x1440, cell centers on Poles/Equator (in south-to-north order) and Prime Meridian/Date Line'#latlon=721,1440#lat_drc=s2n#lat_typ=cap#lon_typ=grn_ctr \
        -g ${DATA}/grids/era5_s2n_721x1440.nc

# 360x720 CRUNCEP (E3SM/ELM-offline forcing grid) (NB: CRUNCEP starts at Greenwich, is not r05)
ncremap -7 --dfl_lvl=1 -G ttl='CRUNCEP Equi-Angular 0.5x0.5 degree uniform grid, dimensions 360x720, cell edges on Poles/Equator and Prime Meridian/Date Line'#latlon=360,720#lat_typ=uni#lon_typ=Grn_wst \
        -g ${DATA}/grids/cruncep_360x720.nc

# 360x720 ELM/MOSART grid, first longitude west edge at DateLine (NB: ELM defines r05, starts at Dateline)
ncremap -7 --dfl_lvl=1 -G ttl='Equi-Angular 0.5x0.5 degree uniform grid (r05), dimensions 360x720, cell edges on Poles/Equator and Date Line/Prime Meridian'#latlon=360,720#lat_typ=uni#lon_typ=180_wst \
        -g ${DATA}/grids/r05_360x720.nc

# 105x401 Greenland ERA5
ncremap -G ttl='Equi-Angular Greenland 0.25x0.25 degree ERA5 north-to-south grid'#latlon=105,401#snwe=58.875,85.125,-87.125,13.125#lat_typ=uni#lat_drc=n2s#lon_typ=grn_ctr \
        -g ${DATA}/grids/greenland_0.25x0.25_era5.nc

# Greenland r025 with SNWE = 59,84,-73,-11 (in round numbers) with RACMO ice mask
ncremap -G ttl='Equi-Angular Greenland 0.25x0.25 degree r025 south-to-north grid'#latlon=100,250#snwe=58.875,83.875,-73.25,-10.75#lat_typ=uni#lat_drc=s2n#lon_typ=grn_ctr \
        -g ${DATA}/grids/greenland_r025_100x250.nc

# NASA Climate Modeling Grid (CMG) 3600x7200 (0.05x0.05 degree, 3'x3') Equi-Angular grid
# With land mask derived mainly from GLOBE 30" topography and anywhere Gardner 30" land ice data is valid
# Date-line west edge of first longitude, east edge of last longitude
# Write to compressed netCDF4-classic file to reduce filesize ~140x from 2.2 GB to 16 MB
ncremap -7 -L 1 \
     -G ttl='Equi-Angular grid 3-minute=0.05 degree resolution = 3600x7200, NASA CMG boundaries, with land mask derived mainly from GLOBE 30" topography and anywhere Gardner 30" land ice data is valid'#latlon=3600,7200#lat_typ=uni#lon_typ=180_wst \
     -g ${DATA}/grids/r005_3600x7200_globe_gardner_landmask.20210501.nc
@end verbatim
@end example

@html
<a name="nfr"></a> <!-- http://nco.sf.net/nco.html#nfr -->
<a name="infer"></a> <!-- http://nco.sf.net/nco.html#infer -->
<a name="ugrid"></a> <!-- http://nco.sf.net/nco.html#ugrid -->
@end html
@cindex infer
@cindex @code{--rgr nfr}
@cindex @code{--rgr infer}
Often researchers face the problem not of generating a known, idealized
grid but of understanding an unknown, possibly irregular or curvilinear
grid underlying a dataset produced elsewhere.
@acronym{NCO} will @dfn{infer} the grid of a datafile by examining its
coordinates (and boundaries, if available), reformat that information
as necessary to diagnose gridcell areas, and output the results in
@acronym{SCRIP} format.
As of @acronym{NCO} @w{version 4.5.3}, released in October, 2015,
the @samp{--rgr infer} flag activates the machinery to infer the grid
rather than construct the grid from other user-specified switches.
To infer the grid properties, @acronym{NCO} interrogates
@var{input-file} for horizontal coordinate information, such as the
presence of dimension names rooted in latitude/longitude-naming
traditions and conventions. 
Once @acronym{NCO} identifies the likely horizontal dimensions it looks
for horizontal coordinates and bounds.
If bounds are not found, @acronym{NCO} assumes the underlying grid
comprises quadrilateral cells whose edges are midway between cell
centers, for both rectilinear and curvilinear grids. 
@example
@verbatim
# Infer AIRS swath grid from input, write it to grd_scrip.nc
ncks --rgr infer --rgr scrip=${DATA}/sld/rgr/grd_scrip.nc \
     ${DATA}/sld/raw/AIRS.2014.10.01.202.L2.TSurfStd.Regrid010.1DLatLon.nc ~/foo.nc
@end verbatim
@end example
When inferring grids, the grid file (@file{grd_scrip.nc}) is written
in @acronym{SCRIP} format, the input file (@file{AIRS...nc}) is read,
and the output file (@file{foo.nc}) is overwritten (its contents are
immaterial). 

@cindex @acronym{UGRID}
@cindex @code{--rgr ugrid}
As of @acronym{NCO} @w{version 4.6.6}, released in April, 2017,
inferred 2D rectangular grids may also be written in
@acronym{UGRID}-format (defined
@uref{http://ugrid-conventions.github.io/ugrid-conventions, here}).
Request a @acronym{UGRID} mesh with the option
@samp{--rgr ugrid=@var{fl_ugrid}}.
Currently both @acronym{UGRID} and @acronym{SCRIP} grids must be
requested in order to produce the @acronym{UGRID} output, e.g., 
@example
@verbatim
ncks --rgr infer --rgr ugrid=${HOME}/grd_ugrid.nc \
     --rgr scrip=${HOME}/grd_scrip.nc ~/skl_180x360.nc ~/foo.nc
@end verbatim
@end example

The @acronym{SCRIP} gridfile and @acronym{UGRID} meshfile metadata
produced for the equiangular @w{1-by-1 degree} global grid are:
@example
@verbatim
zender@aerosol:~$ ncks -m ~/grd_scrip.nc 
netcdf grd_scrip {
  dimensions:
    grid_corners = 4 ;
    grid_rank = 2 ;
    grid_size = 64800 ;

  variables:
    double grid_area(grid_size) ;
      grid_area:units = "steradian" ;

    double grid_center_lat(grid_size) ;
      grid_center_lat:units = "degrees" ;

    double grid_center_lon(grid_size) ;
      grid_center_lon:units = "degrees" ;

    double grid_corner_lat(grid_size,grid_corners) ;
      grid_corner_lat:units = "degrees" ;

    double grid_corner_lon(grid_size,grid_corners) ;
      grid_corner_lon:units = "degrees" ;

    int grid_dims(grid_rank) ;

    int grid_imask(grid_size) ;
} // group /

zender@aerosol:~$ ncks -m ~/grd_ugrid.nc 
netcdf grd_ugrid {
  dimensions:
    maxNodesPerFace = 4 ;
    nEdges = 129240 ;
    nFaces = 64800 ;
    nNodes = 64442 ;
    two = 2 ;

  variables:
    int mesh ;
      mesh:cf_role = "mesh_topology" ;
      mesh:standard_name = "mesh_topology" ;
      mesh:long_name = "Topology data" ;
      mesh:topology_dimension = 2 ;
      mesh:node_coordinates = "mesh_node_x mesh_node_y" ;
      mesh:face_node_connectivity = "mesh_face_nodes" ;
      mesh:face_coordinates = "mesh_face_x mesh_face_y" ;
      mesh:face_dimension = "nFaces" ;
      mesh:edge_node_connectivity = "mesh_edge_nodes" ;
      mesh:edge_coordinates = "mesh_edge_x mesh_edge_y" ;
      mesh:edge_dimension = "nEdges" ;

    int mesh_edge_nodes(nEdges,two) ;
      mesh_edge_nodes:cf_role = "edge_node_connectivity" ;
      mesh_edge_nodes:long_name = "Maps every edge to the two nodes that it connects" ;
      mesh_edge_nodes:start_index = 0 ;

    double mesh_edge_x(nEdges) ;
      mesh_edge_x:standard_name = "longitude" ;
      mesh_edge_x:long_name = "Characteristic longitude of 2D mesh face" ;
      mesh_edge_x:units = "degrees_east" ;

    double mesh_edge_y(nEdges) ;
      mesh_edge_y:standard_name = "latitude" ;
      mesh_edge_y:long_name = "Characteristic latitude of 2D mesh face" ;
      mesh_edge_y:units = "degrees_north" ;

    int mesh_face_nodes(nFaces,maxNodesPerFace) ;
      mesh_face_nodes:cf_role = "face_node_connectivity" ;
      mesh_face_nodes:long_name = "Maps every face to its corner nodes" ;
      mesh_face_nodes:start_index = 0 ;
      mesh_face_nodes:_FillValue = -2147483648 ;

    double mesh_face_x(nFaces) ;
      mesh_face_x:standard_name = "longitude" ;
      mesh_face_x:long_name = "Characteristic longitude of 2D mesh edge" ;
      mesh_face_x:units = "degrees_east" ;

    double mesh_face_y(nFaces) ;
      mesh_face_y:standard_name = "latitude" ;
      mesh_face_y:long_name = "Characteristic latitude of 2D mesh edge" ;
      mesh_face_y:units = "degrees_north" ;

    double mesh_node_x(nNodes) ;
      mesh_node_x:standard_name = "longitude" ;
      mesh_node_x:long_name = "Longitude of mesh nodes" ;
      mesh_node_x:units = "degrees_east" ;

    double mesh_node_y(nNodes) ;
      mesh_node_y:standard_name = "latitude" ;
      mesh_node_y:long_name = "Latitude of mesh nodes" ;
      mesh_node_y:units = "degrees_north" ;
} // group /
@end verbatim
@end example

@html
<a name="skl"></a> <!-- http://nco.sf.net/nco.html#skl -->
<a name="skeleton"></a> <!-- http://nco.sf.net/nco.html#skeleton -->
@end html
@cindex skeleton
@cindex @code{--rgr skl}
Another task that arises in regridding is characterizing new grids.
In such cases it can be helpful to have a ``skeleton'' version of a
dataset on the grid, so that grid center and interfaces locations can be
assessed, continental outlines can be examined, or the skeleton can be
manually populated with data rather than relying on a model.
@acronym{SCRIP} files can be difficult to visualize and manipulate, so 
@acronym{NCO} will provide, if requested, a so-called skeleton file on
the user-specified grid.
As of @acronym{NCO} @w{version 4.5.3}, released in October, 2015,
the @samp{--rgr skl=@var{fl_skl}} switch outputs the skeleton file to
@var{fl_skl}.
The skeleton file may then be examined in a dataset viewer, populated
with data, and generally serve as a template for what to expect from
datasets of the same geometry.
@example
@verbatim
# Generate T42 Gaussian grid file t42_SCRIP.nc and skeleton file t42_skl.nc
ncks --rgr skl=${DATA}/grids/t42_skl.nc --rgr scrip=${DATA}/grids/t42_SCRIP.nc \
     --rgr latlon=64,128#lat_typ=gss#lon_typ=Grn_ctr \
     ~zender/nco/data/in.nc ~/foo.nc
@end verbatim
@end example
When generating skeleton files, both the grid file (@file{t42_SCRIP.nc})
and the skeleton file (@file{t42_skl.nc}) are written, the input file
(@file{in.nc}) is ignored, and the output file (@file{foo.nc}) is
overwritten (its contents are immaterial). 

@html
<a name="map"></a> <!-- http://nco.sf.net/nco.html#map -->
<a name="esmf"></a> <!-- http://nco.sf.net/nco.html#esmf -->
<a name="tr"></a> <!-- http://nco.sf.net/nco.html#tr -->
<a name="tempest"></a> <!-- http://nco.sf.net/nco.html#tempest -->
<a name="scrip"></a> <!-- http://nco.sf.net/nco.html#scrip -->
<a name="rgr"></a> <!-- http://nco.sf.net/nco.html#rgr -->
<a name="rgr_map"></a> <!-- http://nco.sf.net/nco.html#rgr_map -->
<a name="regrid"></a> <!-- http://nco.sf.net/nco.html#regrid -->
@end html
@node Regridding, Climatology and Bounds Support, Grid Generation, Shared features
@section Regridding
@cindex map
@cindex grid-file
@cindex map-file
@cindex regridding
@cindex @code{--map}
@cindex @acronym{ESMF}
@cindex @acronym{MOAB}
@cindex @acronym{TR}
@cindex @acronym{SCRIP}
@cindex TempestRemap
@cindex OpenMP
@cindex @code{--rgr @var{key}=@var{val}}
@cindex @code{--rgr_map}
@cindex @var{--map-file}
@cartouche
Availability: @command{ncclimo}, @command{ncks}, @command{ncremap}@*
Short options: None@*
Long options: 
@samp{--map @var{map-file}} or @samp{--rgr_map @var{map-file}}@*
@samp{--rgr @var{key}=@var{val}} (multiple invocations allowed)@*
@samp{--rnr=@var{rnr_thr}} or @samp{--rgr_rnr=@var{rnr_thr}} or 
@samp{--renormalize=@var{rnr_thr}} or
@samp{--renormalization_threshold=@var{rnr_thr}}@*
@end cartouche

@acronym{NCO} includes extensive regridding features in 
@command{ncclimo} (as of version 4.6.0 in May, 2016),
@command{ncremap} (as of version 4.5.4 in November, 2015)
and @command{ncks} (since version 4.5.0 in June, 2015). 
Regridding can involve many choices, options, inputs, and outputs.
The appropriate operator for this workflow is the @command{ncremap}
script which automatically handles many details of regridding and
passes the required commands to @command{ncks} and external programs.
Occasionally users need access to lower-level remapping functionality
present in @command{ncks} and not exposed to direct manipulation
through @command{ncremap} or @command{ncclimo}.
This section describes the lower-level functionality and switches
as implemented in @command{ncks}.
Knowing what these features are will help @command{ncremap} and
@command{ncclimo} users understand the full potential of these
operators. 

@command{ncks} supports horizontal regridding of datasets where the
grids and weights are all stored in an external @var{map-file}.
Use the @samp{--map} or @samp{--rgr_map} options to specify the
@var{map-file}, and @acronym{NCO} will regrid the @var{input-file} to
a new (or possibly the same, aka, an identity mapping) horizontal grid
in the @var{output-file}, using the input and output grids and mapping
weights specified in the @acronym{ESMF}- or @acronym{SCRIP}-format
@var{map-file}. 
Currently @acronym{NCO} understands the mapfile formats pioneered
by @acronym{SCRIP} 
(@url{http://oceans11.lanl.gov/svn/SCRIP/trunk/SCRIP})
and later extended by @acronym{ESMF} 
(@uref{http://www.earthsystemcog.org/projects/regridweightgen}),
and adopted (along with Exodus) by TempestRemap
(@uref{https://github.com/ClimateGlobalChange/tempestremap.git}).
Those references document quirks in their respectively
weight-generation algorithms as to map formats, grid specification,
and weight generation.
@acronym{NCO} itself produces map-files in the format recommended by
@acronym{CMIP6} and described  
@uref{https://docs.google.com/document/d/1BfVVsKAk9MAsOYstwFSWI2ZBt5mrO_Nmcu7rLGDuL08,
here}.
This format differs from @acronym{ESMF} map-file format chiefly in
that its metadata are slightly more evolved, self-descriptive, and
standardized.

Originally @acronym{NCO} supported only weight-application, which is
what most people mean by ``regridding''. 
As of @w{version 4.9.0}, released in December, 2019, @acronym{NCO}
also supports weight-generation by its own conservative algorithm.
Thus @acronym{NCO} can now apply weights generated by @acronym{ESMF}, 
@acronym{NCO}, @acronym{SCRIP}, and TempestRemap.
@acronym{NCO} reads-in pre-stored weights from the @var{map-file} and
applies them to (almost) every variable, thereby creating a regridded
@var{output-file}. 
Specify regridding with a standard @command{ncks} command and options
along with the additional specification of a @var{map-file}: 
@example
@verbatim
# Regrid entire file, same output format as input:
ncks --map=map.nc in.nc out.nc
# Entire file, netCDF4 output:
ncks -4 --map=map.nc in.nc out.nc
# Deflated netCDF4 output
ncks -4 -L 1 --map=map.nc in.nc out.nc
# Selected variables
ncks -v FS.?,T --map=map.nc in.nc out.nc
# Threading
ncks -t 8 --map=map.nc in.nc out.nc
# Deflated netCDF4 output, threading, selected variables:
ncks -4 -L 1 -t 8 -v FS.?,T --map=map.nc in.nc out.nc
@end verbatim
@end example
OpenMP threading works well with regridding large datasets.
Threading improves throughput of regridding @w{1--10 GB} files by 
factors of 2--5.
Options specific to regridding are described below.

@cindex equiangular grid
@cindex Gaussian grid
@cindex cubed-sphere grid
@cindex @acronym{FV} grid
@acronym{NCO} supports 1D@result{}1D, 1D@result{}2D, 2D@result{}1D, and
2D@result{}2D regridding for any unstructured 
1D-grid and any rectangular 2D-grid.  
This has been tested by converting among and between Gaussian,
equiangular, @acronym{FV}, unstructured cubed-sphere grids, and
regionally refined grids.
Support for irregular 2D- and regional grids (e.g., swath-like data) is
planned.  

@unnumberedsubsec Renormalization
@html
<a name="rnr"></a> <!-- http://nco.sf.net/nco.html#rnr -->
<a name="renormalize"></a> <!-- http://nco.sf.net/nco.html#renormalize -->
<a name="renormalization_threshold"></a> <!-- http://nco.sf.net/nco.html#renormalization_threshold -->
<a name="rgr_rnr"></a> <!-- http://nco.sf.net/nco.html#rgr_rnr -->
@end html
@cindex conservative regridding
@cindex renormalized regridding
@cindex missing values
@cindex data, missing 
@cindex @code{missing_value}
@cindex @code{_FillValue}
@cindex @code{--rnr}
@cindex @code{--renormalize}
@cindex @code{--renormalization_threshold}
@cindex @code{--rgr_rnr}
Conservative regridding is, for first-order accurate algorithms,
a straightforward procedure of identifying gridcell overlap and
apportioning values correctly from source to destination.
The presence of missing values forces a decision on how to handle
destination gridcells where some but not all source cells are valid. 
@acronym{NCO} allows the user to choose between two distinct
weight-application algorithms:
``conservative'' and ``renormalized''.
The ``conservative'' algorithm uses all valid data from the input grid
on the output grid once and only once.
Destination cells receive the weighted valid values of the source cells.
This is conservative because the global integrals of the source and
destination fields are equal.
Another name for the ``conservative'' weight-application method is
therefore ``integral-preserving''.
The ``renormalized'' algorithm divides the destination value by the sum
of the valid weights.
This produces values equal to the mean of the valid input values, but
extended to the entire destination gridcell. 
Thus renormalization is equivalent to extrapolating valid data to
missing regions.  
Another name for the ``renormalized'' weight-application method is
therefore ``mean-preserving''.
Input and output integrals are unequal and renormalized regridding is
not conservative. 
Both algorithms produce identical answers when no missing data maps to 
the destination gridcell.

The renormalized algorithm is useful because it solves some problems,
like producing physically unrealistic temperature values, at the expense
of incurring others, like non-conservation.
Many land and ocean modelers eschew unrealistic gridpoint values, and
conservative weight-application often produces ``weird'' values along
coastlines or missing data gaps where state variables are regridded
to/from small fractions of a gridcell.
Renormalization ensures the output values are physically consistent,
although the integral of their value times area is not preserved.

@cindex @var{rnr_thr}
By default, @acronym{NCO} implements the ``conservative'' algorithm
because it has useful properties, is simpler to understand, and requires
no additional parameters.
To employ the ``renormalized'' algorithm instead, use the @samp{--rnr},
@samp{--rgr_rnr}, @samp{--rnr_thr}, or @samp{--renormalize} options to
supply @var{rnr_thr}, the threshold weight for valid destination values.
Valid values must cover at least the fraction @var{rnr_thr} of the
destination gridcell to meet the threshold for a non-missing destination
value. 
When @var{rnr_thr} is exceeded, the mean valid value is renormalized by
the valid area and placed in the destination gridcell. 
If the valid area covers less than @var{rnr_thr}, then the destination
gridcell is assigned the missing value.
Valid values of @var{rnr_thr} range from zero to one.
Keep in mind though, that this threshold is potentially a divisor, and 
values of zero or very near to zero can lead to floating-point underflow
and divide-by-zero errors.
For convenience @acronym{NCO} permits users to specify a 
@math{@var{rnr_thr} = 0.0} threshold weight.
This indicates that any valid data should be represented and
renormalized on the output grid. 
Also, renormalization can be explicitly prevented or turned-off by
setting @var{rnr_thr} to either of the values @samp{off} or @samp{none}: 
@example
@verbatim
ncks           --map=map.nc in.nc out.nc # Conservative (global integral-preserving)
ncks --rnr=off --map=map.nc in.nc out.nc # Conservative (global integral-preserving)
ncks --rnr=0.1 --map=map.nc in.nc out.nc # Renormalized (local mean-preserving with threshold)
ncks --rnr=0.0 --map=map.nc in.nc out.nc # Renormalized (local mean-preserving)
@end verbatim
@end example
The first and second examples use the default conservative algorithm.
The third example specifies that valid values must cover at least 10%
of the destination gridcell to meet the threshold for a non-missing
destination value. 
With valid destination areas of, say 25% or 50%, the renormalized
algorithm would produce destination values greater than the conservative
algorithm by factors of four or two, respectively.

In practice, it may make sense to use the default ``conservative''
algorithm when performing conservative regridding, and the
``renormalized'' algorithm when performing other regridding such as
bilinear interpolation or nearest-neighbor.
Another consideration is whether the fields being regridded are fluxes
or state variables. 
For example, temperature (unlike heat) and concentrations (amount per
unit volume) are not physically conserved quantities under
areal-regridding so it often makes sense to interpolate them in a 
non-conservative fashion, to preserve their fine-scale structure. 
Few researchers can digest the unphysical values of temperature that the  
``conservative'' option will produce in regions rife with missing
values.
A counter-example is fluxes, which should be physically conserved under 
areal-regridding.
One should consider both the type of field and its conservation
properties when choosing a regridding strategy.

@ifhtml
@cartouche
@html
<p><b>Note to readers of the NCO User Guide in HTML format</b>: 
<b>The <a href="./nco.pdf">NCO User Guide in PDF format</a> 
(also on <a href="http://nco.sf.net/nco.pdf">SourceForge</a>)
contains the complete NCO documentation, including complex
mathematical formulae relevant to this section regridding</b>.
@end html
@end cartouche
@end ifhtml

@tex
The regridded value of a variable $\xxx$ at a destination location
$\ddd$ can be generally represented as
$$
\xxx_{\ddd} = {\sum_{\sss=1}^{\sss=\lmnnbr}
\mssflg_{\sss} \sigma_{\sss,\ddd} \xxx_{\sss} \over
\sum_{\sss=1}^{\sss=\lmnnbr} \mssflg_{\sss} \sigma_{\sss,\ddd}}  
$$
where $\xxx_{\ddd}$ is the $\ddd$'th element of the regridded
variable, $\xxx_{\sss}$ is the $\sss$'th element of the raw (native grid)
variable, $\mssflg_{\sss} = 1$ if $\xxx_{\sss}$ is valid and
$\mssflg_{\sss} = 0$ if $\xxx_{\sss}$ is the missing value, and
$\sigma_{\sss,\ddd}$ is the overlap weight of $\sss$'th source gridcell
with the $\ddd$'th destination gridcell, and $\lmnnbr$ is the
total number of source gridcells that overlap (partially or fully)
with the destination gridcell.

The number of overlap gridcells $\lmnnbr$ is a property of the source
and destination grids and the regridding algorithm.
The weight-generation software determines $\lmnnbr$ by
``intersecting'' the grids, taking into account higher-order
(e.g., local gradient) contributions if the algorithm so-demands,
and then generates the overlap weights $\sigma_{\sss,\ddd}$
accordingly.  
Both source and destination grids may indicate valid gridcells with
a mask flag that is binary-valued, zero or one, such that
$\mskflg_{\sss} = 1$ (i.e., unmasked) for source gridcells allowed to
contribute to the destination grid, and $\mskflg_{\sss} = 0$ (i.e.,
masked) for gridcells that are forbidden from contributing to the
destination grid. 
There are subtle distinctions between the mask flag $\mskflg_{\sss}$,
and the missing value flag $\mssflg_{\sss}$.
The mask flag $\mskflg_{\sss}$ does not appear in the formula above
because the weight-generator produces no weights for masked source
gridcells.
Doing otherwise would waste storage space in the map-file, because
such weights are, by definition, zero.
Furthermore the masks $\mskflg_{\sss}$ and $\mskflg_{\ddd}$ are 
time-invariant properties of the grids, whereas missing value fields
$\mssflg_{\sss}$ (and thus $\mssflg_{\ddd}$) are potentially
time-varying characteristics of the fields.
Although $\mssflg_{\sss}$ should in theory be treated the same as
$\mskflg_{\sss}$ when computing mapping weights $\sigma_{\sss,\ddd}$,
in practice this is not done.
Different fields may have different patterns of missing values,
and managing per-field map-files would be difficult, so traditionally 
all fields are remapped with the same map-file.
That said, it can make sense to treat flux fields and state-variable
fields with distinct algorithms, so that a different map-file might
be employed for each class of fields.

The weight-generation software normalizes $\sigma_{\sss,\ddd}$
such that $\sum_{\sss=1}^{\sss=\lmnnbr} \sigma_{\sss,\ddd} = 1$
when unmasked ($\mskflg_{\sss}=1$) source gridcells completely overlap
the destination gridcell. 
In this case we also have
$\sum_{\sss=1}^{\sss=\lmnnbr} \mskflg_{\sss} = \lmnnbr$.
Furthermore, if all contributing gridpoints are valid values
(i.e., not missing values) then $\mssflg_{\sss} = 1$ so that
$\sum_{\sss=1}^{\sss=\lmnnbr} \mssflg_{\sss} = \lmnnbr$.
For complete overlap with no masked values and no missing values,
then $\mssflg_{\sss} = \mskflg_{\sss} = \sum\sigma_{\sss,\ddd} = 1$
and the generic averaging expression above reduces to a simple
weighted mean
$\xxx_{\ddd} = \sum_{\sss=1}^{\sss=\lmnnbr} \sigma_{\sss,\ddd} \xxx_{\sss}$.

$$
\xxx_{\ddd} = {\sum_{\sss=1}^{\sss=\lmnnbr} \mssflg_{\sss}
\frcsgs_{\sss} \sigma_{\sss,\ddd} \xxx_{\sss} \over
\sum_{\sss=1}^{\sss=\lmnnbr} \mssflg_{\sss} \frcsgs_{\sss} \sigma_{\sss,\ddd}}  
$$
@end tex

@unnumberedsubsec Regridder Options Table
@html
<a name="rgr_tbl"></a> <!-- http://nco.sf.net/nco.html#rgr_tbl -->
@end html
@acronym{NCO} automatically annotates the output with relevant metadata
such as coordinate bounds, axes, and vertices (@w{@`{a} la} @acronym{CF}). 
These annotations include
@table @dfn
@cindex @var{lat_dmn_nm}
@cindex @var{lon_dmn_nm}
@cindex @samp{--rgr lon_dmn_nm=@var{lon_dmn_nm}}
@cindex @samp{--rgr lat_dmn_nm=@var{lat_dmn_nm}}
@item Horizontal Dimension Names: @var{lat_dmn}, @var{lon_dmn}
    The name of the horizontal spatial dimensions assumed to represent
    latitude and longitude in 2D rectangular input files are
    @var{lat_dmn_nm} and @var{lon_dmn_nm}, which default to @code{lat}
    and @code{lon}, respectively.  
    Variables that contain a @var{lat_dmn_nm}-dimension and a
    @var{lon_dmn_nm}-dimension on a 2D-rectangular input grid will be 
    regridded, and variables regridded to a 2D-rectangular output grid 
    will all contain the @var{lat_dmn_nm}- and @var{lon_dmn_nm}-dimensions.
    To treat different dimensions as latitude and longitude, use the
    options @samp{--rgr lat_dmn_nm=@var{lat_dmn_nm}} and
    @samp{--rgr lon_dmn_nm=@var{lon_dmn_nm}}.  
    These options applied only to inferring and generating grids until
    @acronym{NCO} version 4.7.9 (February, 2019).
    Since then, these options also determine the dimension names in
    regridded output files. 
@cindex @var{lat_nm}
@cindex @var{lon_nm}
@cindex @samp{--rgr lon_nm=@var{lon_nm}}
@cindex @samp{--rgr lat_nm=@var{lat_nm}}
@item Horizontal Coordinate Names: @var{lat}, @var{lon}
    The name of the horizontal spatial coordinates that represent
    latitude and longitude in input files are @var{lat_nm} and
    @var{lon_nm}, and default to @code{lat} and @code{lon},
    respectively.  
    Variables that contain a @var{lat_dmn_nm}-dimension and a
    @var{lon_dmn_nm}-dimension on a 2D input grid will be 
    regridded, and output regridded variables will all contain the 
    @var{lat_nm}- and @var{lon_nm}-variables.
    Unless the @var{lat_dmn_nm}- and @var{lon_dmn_nm}-dimensions
    are explicitly configured otherwise, they will share the same
    name as the @var{lat_nm}- and @var{lon_nm}-variables.
    Thus variables regridded to a 2D-rectangular output grid usually
    have @var{lat_nm}- and @var{lon_nm} as coordinate variables.
    Variables regridded to a 1D-unstructured output grid will have
    @var{lat_nm} and @var{lon_nm} as auxiliary coordinate variables.
    Variables regridded to a 2D-curvilinear output grid will have
    @var{lat_nm} and @var{lon_nm} as multi-dimensional auxiliary
    coordinate variables.
    To treat different variables as latitude and longitude, use the
    options @samp{--rgr lat_nm=@var{lat_nm}} and
    @samp{--rgr lon_nm=@var{lon_nm}}.   
    Before @acronym{NCO} version 4.7.9 (February, 2019), @var{lat_nm}
    and @var{lon_nm} specified both the variable names @emph{and},
    where applicable (i.e., on 2D-grids), the dimensions of the
    horizontal coordinates in output files.
    Now the horizontal variable and dimension names in output files
    may be separately specified.
@cindex @code{ncol}
@cindex @var{col_nm}
@cindex @samp{--rgr col_nm=@var{col_nm}}
@item Unstructured Dimension Name: @var{col}
    The name of the horizontal spatial dimension assumed to delineate
    an unstructured grid is @var{col_nm}, which defaults to @code{ncol}
    (number of columns), the name @acronym{CAM} employs.
    Other common names for the columns in an unstructured grid include
    @code{lndgrid} (used by @acronym{CLM}), and @code{nCells} (used by 
    @acronym{MPAS-O}). 
    Variables that contain the @var{col_nm}-dimension on an
    unstructured input grid will be regridded, and regridded variables
    written to an unstructured output grid will all contain the 
    @var{col_nm}-dimension.
    To treat a different dimension as unstructured, use the option
    @samp{--rgr col_nm=@var{col_nm}}. 
    Note: Often there is no coordinate variable for the
    @var{col_nm}-dimension, i.e., there is no variable named
    @var{col_nm}, although such a coordinate could contain useful
    information about the unstructured grid.
@cindex @acronym{CF}
@cindex @code{latitude}
@cindex @code{longitude}
@cindex @code{axes}
@cindex @code{standard_name}
@cindex @code{degrees_east}
@cindex @code{degrees_north}
@cindex @code{units}
@cindex @code{X} axis
@cindex @code{Y} axis
@cindex auxiliary coordinates
@item Structured Grid Standard Names and Units
    Longitude and latitude coordinates (both regular and auxiliary,
    i.e., for unstructured grids) receive @acronym{CF}
    @code{standard_name} values of @code{latitude} and @code{longitude},
    @acronym{CF} @code{axes} attributes with values @code{X} and
    @code{Y}, and @code{units} attributes with values
    @code{degrees_east} and @code{degrees_north}, respectively.
@cindex @code{lat}
@cindex @code{lon}
@item Unstructured Grid Auxiliary Coordinates
    Unstructured grid auxiliary coordinates for longitude and latitude
    receive @acronym{CF} @code{coordinates} attributes with values
    @code{lon} and @code{lat}, respectively.
@cindex @code{lon_bnds}
@cindex @code{lat_bnds}
@cindex @code{nbnd}
@cindex @code{time_bnds}
@cindex @var{lat_bnd_nm}
@cindex @var{lon_bnd_nm}
@cindex @samp{--rgr lat_bnd_nm=@var{lat_bnd_nm}}
@cindex @samp{--rgr lon_bnd_nm=@var{lon_bnd_nm}}
@item Structured Grid Bounds Variables: @var{bnd}, @var{lat_bnd}, @var{lon_bnd}
    Structured grids with 1D-coordinates use the dimension
    @var{bnd_nm} (which defaults to @code{nbnd}) with the spatial bounds
    variables in @var{lat_bnd_nm} and @var{lon_bnd_nm} which default to
    @code{lon_bnds} and @code{lat_bnds}, respectively.
    By default spatial bounds for such structured grids parallel the
    oft-used temporal bounds dimension (@code{nbnd=2}) and variable 
    (@code{time_bnds}).
    Bounds are attached to the horizontal spatial dimensions via their
    @code{bounds} attributes. 
    Change the spatial bounds dimension with the option 
    @samp{--rgr bnd_nm=@var{bnd_nm}}.
    Rename the spatial bounds variables with the options 
    @samp{--rgr lat_bnd_nm=@var{lat_bnd_nm}} and 
    @samp{--rgr lon_bnd_nm=@var{lon_bnd_nm}}.
@cindex @code{lat_vertices}
@cindex @code{lon_vertices}
@cindex @code{nv}
@cindex @code{bounds}
@item Unstructured Grid Bounds Variables: @var{bnd}, @var{lat_bnd}, @var{lon_bnd} 
    Unstructured grids with 1D-coordinates use the dimension
    @var{bnd_nm} (which defaults to @code{nv}, number of vertices) 
    for the spatial bounds variables @var{lat_bnd_nm} and
    @var{lon_bnd_nm} which default to @code{lat_vertices} and
    @code{lon_vertices}, respectively. 
    It may be impossible to re-use the temporal bounds dimension (often
    @code{nbnd}) for unstructure grids, because the gridcells are not
    rectangles, and thus require specification of all vertices for each
    gridpoint, rather than only two parallel interfaces per dimension.
    These bounds are attached to the horizontal spatial dimensions
    via their @code{bounds} attributes.
    Change the spatial bounds dimension with the option 
    @samp{--rgr bnd_nm=@var{bnd_nm}}.
    Rename the spatial bounds variables with the options 
    @samp{--rgr lat_bnd_nm=@var{lat_bnd_nm}} and
    @samp{--rgr lon_bnd_nm=@var{lon_bnd_nm}}.
    The temporal bounds dimension in unstructured grid output remains as
    in the @var{input-file}, usually @code{nbnd}.
@cindex @var{lev_dmn_nm}
@cindex @var{ilev_dmn_nm}
@cindex @samp{--rgr ilev_dmn_nm=@var{ilev_dmn_nm}}
@cindex @samp{--rgr lev_dmn_nm=@var{lev_dmn_nm}}
@item Vertical Dimension Names: @var{lev_dmn}, @var{ilev_dmn}
    The name of the dimension(s) associated with the vertical
    coordinate(s) in multi-level input files are @var{lev_dmn_nm} and
    @var{ilev_dmn_nm}, which default to @code{lev} and @code{ilev},
    respectively.
    Variables that contain a @var{lev_dmn_nm}-dimension or an
    @var{ilev_dmn_nm}-dimension will be vertically interpolated to the 
    specified (with @samp{vrt_out=@var{vrt_fl}}) vertical output grid,
    and will all contain the @var{lev_dmn_nm}- and, for
    hybrid-sigma/pressure interface variables,
    @var{ilev_dmn_nm}-dimensions.
    To treat different dimensions as the midlayer and interface level
    dimensions, use the options @samp{--rgr lev_dmn_nm=@var{lev_dmn_nm}}
    and @samp{--rgr ilev_dmn_nm=@var{ilev_dmn_nm}} options.
    Pure-pressure grids should use the
    @samp{--rgr lev_dmn_nm=@var{lev_dmn_nm}} option (to reduce option
    proliferation, there is no @var{plev_dmn_nm} option).
    These options were introduced in @acronym{NCO} version 4.9.0 (December, 2019).
    These options also determine the vertical dimension names in
    vertically interpolated output files. 
@cindex @var{lev_nm}
@cindex @var{ilev_nm}
@cindex @var{plev_nm}
@cindex @samp{--rgr lev_nm=@var{lev_nm}}
@cindex @samp{--rgr ilev_nm=@var{ilev_nm}}
@cindex @samp{--rgr plev_nm=@var{plev_nm}}
@item Vertical Coordinate Names: @var{lev}, @var{ilev}, @var{plev}
    The name of the vertical coordinate variables that represent
    midpoint levels and interface levels in hybrid-sigma/pressuure
    input files are @var{lev_nm} and @var{ilev_nm}, and default to
    @code{lev} and @code{ilev}, respectively.  
    While the vertical coordinate in pure-pressure vertical grid files
    (i.e., the template-file to which data will be interpolated) must 
    be named @code{plev}, the vertical coordinate in pure-pressure
    @emph{data} files (i.e., the files to be interpolated) may be
    changed with the @samp{--rgr plev_nm=@var{plev_nm}} option.
    The name of the vertical coordinate variable that represents
    pressure levels in pure-pressure grid input data files is
    @var{plev_nm}, and it defaults to @code{plev}.
    To reduce proliferation of command-line options and internal code 
    complexity, the variable and dimension options for pure-pressure
    vertical coordinate output names re-use the ``lev'' options, i.e.,
    @samp{--rgr lev_nm_out=@var{lev_nm_out}} option.
    Variables that contain a @var{lev_dmn_nm}-dimension or a
    @var{ilev_dmn_nm}-dimension on hybrid-sigma/pressure input grid,
    or a @var{plev_dmn_nm}-dimension on a pure pressure grid,
    will be regridded, and output in vertically interpolated files on
    a hybrid-sigma/pressure grid will all contain the @var{lev_nm}-
    and @var{ilev_nm}-variables, and output on a pure-pressure grid
    will contain the @var{lev_nm} coordinate.
    Unless the @var{lev_dmn_nm} and @var{ilev_dmn_nm} dimensions
    are explicitly configured otherwise, they will share the same
    name as the @var{lev_nm}/@var{plev_nm} and
    @var{ilev_nm}-variables, respectively.
    Thus variables regridded to a hybrid-sigma/pressure output grid
    usually have @var{lev_nm}- and @var{ilev_nm} as coordinate
    variables. 
    Variables regridded to a pure-pressure output grid will only have 
    a single vertical coordinate variable, @var{lev_nm}, which will be
    an associated coordinate variable if @var{lev_dmn_nm} differs from
    @var{lev_nm}.
    To treat different variables as level and interface-level
    coordinates, use the options @samp{--rgr lev_nm=@var{lev_nm}} and 
    @samp{--rgr ilev_nm=@var{ilev_nm}}.   
    Before @acronym{NCO} version 4.9.0 (December, 2019), @var{lev_nm}
    and @var{ilev_nm} specified both the variable names @emph{and},
    where applicable (i.e., on 2D-grids), the dimensions of the
    vertical coordinates in output files.
    Now the vertical variable and dimension names in output files
    may be separately specified.
@cindex @var{ps_nm}
@cindex @samp{--rgr ps_nm=@var{ps_nm}}
@item Surface Pressure Names: @var{ps}, @var{PS}
    The name of the surface pressure field necessary to reconstruct
    the layer pressures in the hybrid-sigma/pressure coordinate system 
    is @var{ps_nm} which defaults to @code{PS}.
    As of @acronym{NCO} @w{version 5.1.2}, released in November, 2022,
    one may change this with the @samp{--rgr ps_nm=@var{ps_nm}} option.
    There are, in fact, three similar options, one each for the
    surface pressure variable in the input data file, the vertical
    grid file, and in the output (interpolated file).
    The full option key names are @code{ps_nm} (equivalent to
    @code{ps_nm_in}), @code{ps_nm_tpl}, and @code{ps_nm_out},
    respectively.
@cindex @code{area}
@cindex @code{cell_methods}
@cindex @code{cell_area}
@cindex @code{steradian}
@cindex @samp{--no_cll_msr}
@cindex @samp{--rgr no_area_out}
@cindex @samp{--rgr no_area}
@cindex @samp{--rgr area_out=@var{area_nm}}
@cindex @var{area_nm}
@cindex @acronym{SI}
@cindex solid angle
@cindex extensive variable
@item Gridcell Area: @var{area}
    The variable @var{area_nm} (which defaults to @code{area}) is, by 
    default, (re-)created in the @var{output_file} to hold the gridcell 
    area in steradians.     
    To store the area in a different variable, use the option
    @samp{--rgr area=@var{area_nm}}.
    The @var{area_nm} variable receives a @code{standard_name} attribute 
    of @code{cell_area}, a @code{units} attribute of @code{steradian}
    (the @acronym{SI} unit of solid angle),
    and a @code{cell_methods} attribute with value @code{lat, lon: sum},
    which indicates that @var{area_nm} is @dfn{extensive}, meaning 
    that its value depends on the gridcell boundaries.  
    Since @var{area_nm} is a property of the grid, it is read directly
    from the @var{map-file} rather than regridded itself. 
    To omit the area variable from the output file, set the
    @var{no_area_out} flag.
    The @code{--no_cll_msr} switch to @command{ncremap} and
    @command{ncclimo} does this automatically.
@cindex @code{frc}
@cindex @samp{--rgr frc_nm=@var{frc_nm}}
@cindex @var{frc_nm}
@cindex extensive variable
@item Gridcell Fraction: @var{frc}
    The variable @var{frc_nm} (which defaults to @code{frac_b}) is
    automatically copied to the @var{output_file} to hold the valid
    fraction of each gridcell when certain conditions are met.
    First, the regridding method must be conservative.
    Second, at least one value of @var{frc_nm} must be non-unity.
    These conditions ensure that whenever fractional gridcells affect
    the regridding, they are also placed in the output file.
    To store the fraction in a different variable, use the option
    @samp{--rgr frc_nm=@var{frc_nm}}.
    The @var{frc_nm} variable receives a @code{cell_methods} attribute
    with value @code{lat, lon: sum}, which indicates that @var{frc_nm}
    is @dfn{extensive}, meaning that its value depends on the gridcell
    boundaries.   
    Since @var{frc_nm} is a property of the grid, it is read directly
    from the @var{map-file} rather than regridded itself. 
@cindex @code{mask}
@cindex @samp{--no_msk}
@cindex @samp{--no_mask}
@cindex @samp{--msk_out}
@cindex @samp{--mask_out}
@cindex @samp{--rgr no_msk_out}
@cindex @samp{--rgr no_mask}
@cindex @samp{--rgr msk_nm=@var{msk_nm}}
@cindex infer
@cindex @var{msk_nm}
@item Gridcell Mask: @var{mask}
    The variable @var{msk_nm} (which defaults to @code{mask}) can, if
    present, be copied from the @var{map-file} to hold the gridcell mask 
    on the destination grid in @var{output-file}.
    To store the mask in a different variable, use the option
    @samp{--rgr msk_nm=@var{msk_nm}}.
    Since @var{msk_nm} is a property of the grid, it is read directly
    from the @var{map-file} rather than regridded itself. 
    To include the mask variable in the output file, set the
    @var{msk_out} flag.
    To omit the mask variable from the output file, set the
    @var{no_msk_out} flag.
    In grid inferral and map-generation modes, this option tells the
    regridder to generate an integer mask map from the variable
    @var{msk_nm}. 
    The mask will be one (i.e., points at that location will contribute
    to regridding weights) where @var{msk_nm} has valid values. 
    The mask will be zero (i.e., points at that location will not
    contribute to regridding weights) where @var{msk_nm} has a missing
    value.
    This feature is useful when creating weights between masked grids,
    e.g., ocean-only points or land-only points.
@cindex @code{gw}
@cindex @samp{--rgr lat_weight=@var{lat_wgt_nm}}
@cindex @var{lat_wgt_nm}
@cindex Gaussian weight
@item Latitude weights: @var{lat_wgt}
    Rectangular 2D-grids use the variable @var{lat_wgt_nm}, which
    defaults to @code{gw} (originally for ``Gaussian weight''), to store 
    the 1D-weight appropriate for area-weighting the latitude grid.
    To store the latitude weight in a different variable, use the option
    @samp{--rgr lat_wgt=@var{lat_wgt_nm}}.
    The @var{lat_wgt_nm} variable will not appear in 1D-grid output.
    Weighting statistics by latitude (i.e., by @var{lat_wgt_nm} will
    produce the same answers (up-to round-off error) as weighting by
    area (i.e., by @var{area_nm}) in grids that have both variables.
    The former requires less memory because @var{lat_wgt_nm} is 1D), 
    whereas the latter is more general because @var{area_nm} works on
    @emph{any} grid.
@cindex @code{mapping_file}
@cindex @code{source_file}
@item Provenance Attributes
    The @var{map-file} and @var{input-file} names are stored in the
    @var{output-file} global attributes @code{mapping_file} and
    @code{source_file}, respectively.
@html
<a name="slat"></a> <!-- http://nco.sf.net/nco.html#slat -->
<a name="slon"></a> <!-- http://nco.sf.net/nco.html#slon -->
<a name="stg"></a> <!-- http://nco.sf.net/nco.html#stg -->
<a name="stagger"></a> <!-- http://nco.sf.net/nco.html#stagger -->
<a name="no_stagger"></a> <!-- http://nco.sf.net/nco.html#no_stagger -->
@end html
@cindex @code{--no_stg}
@cindex @code{--no_stagger}
@cindex @code{--no_stg_grd}
@cindex @code{--stg}
@cindex @code{--stagger}
@cindex @code{--stg_grd}
@cindex @code{slat}
@cindex @code{slon}
@cindex @code{w_stag}
@cindex @acronym{CAM}
@cindex @acronym{FV}
@cindex @acronym{AMWG}
@cindex staggered-grid
@cindex no stagger
@cindex stagger
@item Staggered Grid Coordinates and Weights
    Owing to its heritage as an early @acronym{CCM} analysis tool,
    @acronym{NCO} tries to create output interoperable with other
    @acronym{CESM} analysis tools. 
    Like many models, @acronym{CAM} computes and archives thermodynamic
    state variables on gridcell centers, and computes dynamics variables
    (zonal and meridional winds @var{U} and @var{V}, respectively) on
    gridcell edges (interfaces). 
    The dual-grid, sometimes called the ``staggered grid'', formed
    by connecting edge centers is thus the natural location for 
    storing output dynamics variables.
    Most dynamical cores of @acronym{CAM} archives horizontal winds at
    gridcell centers under the names @code{U}, and @code{V}.
    For @acronym{CAM-FV}, these are interpolated from the computed
    interface winds archived as @code{US}, and @code{VS} (which are
    on the staggered grid coordinate system). 
    Some analysis packages, such as the @acronym{AMWG} diagnostics, 
    require access to these dual-grid coordinates with the names
    @code{slat} and @code{slon} (for ``staggered'' latitude and
    longitude).
    Until @acronym{NCO} version 4.9.8 (released March, 2021),
    the @acronym{NCO} regridder output these coordinates,
    along with the latitude weights (called @code{w_stag}), by default
    when the input was on a cap (aka @acronym{FV}) grid so that the
    result could be processed by @acronym{AMWG} diagnostics. 
    Setting the @var{no_stagger} flag turns-off archiving the
    staggered grid (i.e., @code{slat}, @code{slon}, and
    @code{w_stag}). 
    Do this with the @code{--no_stg_grd} flag in @command{ncremap}.
    @command{ncclimo} always sets this @code{--no_stagger} flag.
    As of @acronym{NCO} version 4.9.8 (released March, 2021),
    the default @command{ncremap} and @command{ncclimo} behavior
    is to omit the staggered grid.
    The new flag @code{--stg_grd} turns-on outputting the staggered
    grid, and thus recovers the previous default behavior. 
@end table

One may supply muliple @samp{--rgr @var{key}=@var{value}} options
to simultaneously customize multiple grid-field names.
The following examples may all be assumed to end with the standard
options @samp{--map=map.nc in.nc out.nc}.
@example
@verbatim
ncks --rgr lat_nm=latitude --rgr lon_nm=longitude
ncks --rgr col_nm=column --rgr lat_wgt=lat_wgt
ncks --rgr bnd_nm=bounds --rgr lat_bnd_nm=lat_bounds --rgr lon_bnd_nm=lon_bounds
ncks --rgr bnd_nm=vertices --rgr lat_bnd_nm=lat_vrt --rgr lon_bnd_nm=lon_vrt
@end verbatim
@end example
The first command causes the regridder to associate the latitude and
longitude dimensions with the dimension names @code{latitude} and
@code{longitude} (instead of the defaults, @code{lat} and @code{lon}). 
The second command causes the regridder to associate the independent
columns in an unstructured grid with the dimension name @code{column}
(instead of the default, @code{ncol}) and the variable containing
latitude weights to be named @code{lat_wgt} (instead of the default,
@code{gw}). 
The third command associates the latitude and longitude bounds with the
dimension @code{bounds} (instead of the default, @code{nbnd}) and the
variables @code{lat_bounds} and @code{lon_bounds} (instead of the
defaults, @code{lat_bnds} and  @code{lon_bnds}, respectively). 
The fourth command associates the latitude and longitude bounds with the
dimension @code{vertices} (instead of the default, @code{nv}) and the
variables @code{lat_vrt} and @code{lon_vrt} (instead of the defaults,
@code{lat_vertices} and @code{lon_vertices}, respectively). 

When used with an identity remapping files, regridding can signficantly
enhance the metadata and therefore the dataset usability.
Consider these selected metadata (those unchanged are not shown for
brevity) associated with the variable @code{FSNT} from typical
unstructured grid (@acronym{CAM-SE} cubed-sphere) output before and
after an identity regridding: 
@example
@verbatim
# Raw model output before regridding
netcdf ne30_FSNT {
  dimensions:
    nbnd = 2 ;
    ncol = 48602 ;
    time = UNLIMITED ; // (1 currently)

  variables:
    float FSNT(time,ncol) ;
      FSNT:long_name = "Net solar flux at top of model" ;

    double time(time) ;
      time:long_name = "time" ;
      time:bounds = "time_bnds" ;

    double time_bnds(time,nbnd) ;
      time_bnds:long_name = "time interval endpoints" ;
} // group /

# Same model output after identity regridding
netcdf dogfood {
  dimensions:
    nbnd = 2 ;
    ncol = 48602 ;
    nv = 5 ;
    time = 1 ;

  variables:
    float FSNT(time,ncol) ;
      FSNT:long_name = "Net solar flux at top of model" ;
      FSNT:coordinates = "lat lon" ;

    double lat(ncol) ;
      lat:long_name = "latitude" ;
      lat:standard_name = "latitude" ;
      lat:units = "degrees_north" ;
      lat:axis = "Y" ;
      lat:bounds = "lat_vertices" ;
      lat:coordinates = "lat lon" ;

    double lat_vertices(ncol,nv) ;
      lat_vertices:long_name = "gridcell latitude vertices" ;

    double lon(ncol) ;
      lon:long_name = "longitude" ;
      lon:standard_name = "longitude" ;
      lon:units = "degrees_east" ;
      lon:axis = "X" ;
      lon:bounds = "lon_vertices" ;
      lon:coordinates = "lat lon" ;

    double lon_vertices(ncol,nv) ;
      lon_vertices:long_name = "gridcell longitude vertices" ;

    double time(time) ;
      time:long_name = "time" ;
      time:bounds = "time_bnds" ;

    double time_bnds(time,nbnd) ;
      time_bnds:long_name = "time interval endpoints" ;
} // group /
@end verbatim
@end example
The raw model output lacks the @acronym{CF} @code{coordinates} and
@code{bounds} attributes that the regridder adds.
The metadata turns @code{lat} and @code{lon} into auxiliary coordinate
variables (@pxref{Auxiliary Coordinates}) which can then be hyperslabbed
(with @samp{-X}) using latitude/longitude coordinates bounding the
region of interest:
@example
@verbatim
% ncks -u -H -X 314.6,315.3,-35.6,-35.1 -v FSNT dogfood.nc
time[0]=31 ncol[0] FSNT[0]=344.575 W/m2

ncol[0] lat[0]=-35.2643896828 degrees_north

ncol[0] nv[0] lat_vertices[0]=-35.5977213708 
ncol[0] nv[1] lat_vertices[1]=-35.5977213708 
ncol[0] nv[2] lat_vertices[2]=-35.0972113817 
ncol[0] nv[3] lat_vertices[3]=-35.0972113817 
ncol[0] nv[4] lat_vertices[4]=-35.0972113817 

ncol[0] lon[0]=315 degrees_east

ncol[0] nv[0] lon_vertices[0]=315 
ncol[0] nv[1] lon_vertices[1]=315 
ncol[0] nv[2] lon_vertices[2]=315.352825437 
ncol[0] nv[3] lon_vertices[3]=314.647174563 
ncol[0] nv[4] lon_vertices[4]=314.647174563 

time[0]=31 days since 1979-01-01 00:00:00

time[0]=31 nbnd[0] time_bnds[0]=0 
time[0]=31 nbnd[1] time_bnds[1]=31 
@end verbatim
@end example
Thus auxiliary coordinate variables help to structure unstructured grids.
The expanded metadata annotations from an identity regridding may
obviate the need to place unstructured data on a rectangular grid. 
For example, statistics for regions that can be expressed as unions of 
rectangular regions can now be performed on the native (unstructured)
grid. 

Here are some quick examples of regridding from common models.
All examples require @samp{in.nc out.nc} at the end.
@example
@verbatim
# Identity re-map E3SM/ACME CAM-SE Cubed-Sphere output (to improve metadata)
ncks --map=${DATA}/maps/map_ne30np4_to_ne30np4_aave.20150603.nc
# Convert E3SM/ACME CAM-SE Cubed Sphere output to rectangular lat/lon
ncks --map=${DATA}/maps/map_ne30np4_to_fv129x256_aave.150418.nc
# Convert CAM3 T42 output to Cubed-Sphere grid
ncks --map=${DATA}/maps/map_ne30np4_to_t42_aave.20150601.nc
@end verbatim
@end example

@html
<a name="clm_nfo"></a> <!-- http://nco.sf.net/nco.html#clm_nfo -->
<a name="cb"></a> <!-- http://nco.sf.net/nco.html#cb -->
<a name="clm_bnd"></a> <!-- http://nco.sf.net/nco.html#clm_bnd -->
<a name="climatology_information"></a> <!-- http://nco.sf.net/nco.html#climatology_information -->
@end html
@node Climatology and Bounds Support, UDUnits Support, Regridding, Shared features
@section Climatology and Bounds Support
@cindex @code{--clm_nfo}
@cindex @code{--cb}
@cindex @code{--clm_bnd}
@cindex @code{--climatology_information}
Availability: @command{nces}, @command{ncra}, @command{ncrcat}@*
Short options: None@*
Long options: 
@samp{--cb=@var{yr_srt},@var{yr_end},@var{mth_srt},@var{mth_end},@var{tpd}}@*
@samp{--clm_bnd=@var{yr_srt},@var{yr_end},@var{mth_srt},@var{mth_end},@var{tpd}}@*
@samp{--clm_nfo=@var{yr_srt},@var{yr_end},@var{mth_srt},@var{mth_end},@var{tpd}}@*
@samp{--climatology_information=@var{yr_srt},@var{yr_end},@var{mth_srt},@var{mth_end},@var{tpd}}@*

(NB: This section describes support for generating 
@acronym{CF}-compliant bounds variables and attributes, i.e.,
metadata. For instructions on constructing climatologies themselves,
see the @command{ncclimo} documentation).
As of @acronym{NCO} version 4.9.4 (September, 2020) @command{ncra}
introduces the @samp{--clm_bnd} option, a powerful method to
fully implement the @acronym{CF} @code{bounds}, @code{climatology},
and @code{cell_methods} attributes defined by @ref{CF Conventions}.
The new method updates the previous @samp{--cb} and @samp{--c2b}
methods introduced in version 4.6.0 which only worked for monthly mean 
data.
The newer @code{--cb} method also works for climatological diurnally
resolved input, and for datasets that contain more than more than one
record. 
This option takes as argument a comma-separated list of five relevant
input parameters: 
@samp{--cb=@var{yr_srt},@var{yr_end},@var{mth_srt},@var{mth_end},@var{tpd}}, 
where
@var{yr_srt} is the climatology start-year,
@var{yr_end} is the climatology end-year,
@var{mth_srt} is the climatology start-month (in @code{[1..12]} format),
@var{mth_end} is the climatology end-month (in @code{[1..12]} format), and
@var{tpd} is the number of timestpes per day (with the special
exception that @math{@var{tpd}=0} indicates monthly data, not
diurnally-resolved data).
For example, a seasonal summer climatology created from monthly mean
input data spanning June, 2000 to August, 2020 should call
@command{ncra} with @samp{--clm_bnd=2000,2020,6,8,0}, whereas a
diurnally resolved climatology of the same period with 6-hourly input
data resolution would use @samp{--clm_bnd=2000,2020,6,8,4}.
The @command{ncclimo} command internally uses @code{--clm_bnd}
extensively. 

@example
# Average monthly means into a climatological month
ncra --cb=2014,2016,1,1,0 2014_01.nc 2015_01.nc 2016_01.nc clm_JAN.nc
# Average seasonally contiguous climatological monthly means into NH winter
ncra --cb=2013,2016,12,2,0 -w 31,31,28 DEC.nc JAN.nc FEB.nc DJF.nc
# Average seasonally discontiguous climatological means into NH winter
ncra --cb=2014,2016,1,12,0 -w 31,28,31 JAN.nc FEB.nc DEC.nc JFD.nc
# Reduce four climatological seasons to make an annual climatology
ncra --cb=2014,2016,1,12,0 -w 92,92,91,90 MAM.nc JJA.nc SON.nc DJF.nc ANN.nc
# Reduce twelve monthly climatologies to make into an annual climatology
ncra --cb=2014,2016,1,12,0 -w 31,28,31,30,31,30,31,31,30,31,30,31 clm_??.nc ANN.nc
@end example
In the fourth and fifth examples, @acronym{NCO} uses the number of
input files (3 and 4, respectively) to discriminate between seasonal
and annual climatologies since the other arguments to @samp{--cb}
are identical.

When using this option, @acronym{NCO} expects each output file to
contain @code{max(1,@var{tpd})} records.
@command{nces} and @command{ncra} both accept the @samp{--cb} option.
While @command{ncra} almost always reduces the input dataset over the
record dimension, @command{nces} never does.
This makes it easy to use @command{nces} to combine and create
climatologies of diurnally resolved input files.
@example
# Average diurnally resolved monthly means into a climatology
nces --cb=2014,2016,1,1,8 2014_01.nc 2015_01.nc 2016_01.nc clm_JAN.nc
# Average seasonally contiguous diurnally resolved means into a season
nces --cb=2013,2016,12,2,8 -w 31,31,28 DEC.nc JAN.nc FEB.nc DJF.nc
# Average seasonally discontiguous diurnally resolved means into a season
nces --cb=2014,2016,1,12,8 -w 31,28,31 JAN.nc FEB.nc DEC.nc JFD.nc
# Reduce four diurnally resolved seasons to make an annual climatology
nces --cb=2014,2016,1,12,8 -w 92,92,91,90 MAM.nc JJA.nc SON.nc DJF.nc ANN.nc
# Reduce twelve diurnally resolved months to make into an annual climatology
nces --cb=2014,2016,1,12,8 -w 31,28,31,30,31,30,31,31,30,31,30,31 clm_??.nc ANN.nc
@end example
Every input in the above set of examples must have eight records,
and that number will appear in the output as well.

@html
<a name="udunits"></a> <!-- http://nco.sf.net/nco.html#udunits -->
<a name="UDUnits"></a> <!-- http://nco.sf.net/nco.html#UDUnits -->
<a name="UDUnits2"></a> <!-- http://nco.sf.net/nco.html#UDUnits2 -->
@end html
@node UDUnits Support, Rebasing Time Coordinate, Climatology and Bounds Support, Shared features
@section UDUnits Support 
@cindex UDUnits
@cindex Unidata
@cindex @code{units}
@cindex attribute, @code{units}
@cindex @code{-d @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]}
@cindex @code{--dimension @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]}
@cindex @code{--dmn @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]}
@cartouche
Availability: @command{ncbo}, @command{nces}, @command{ncecat},
@command{ncflint}, @command{ncks}, @command{ncpdq}, @command{ncra},
@command{ncrcat}, @command{ncwa}@* 
Short options: @samp{-d @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]}@*
Long options: 
@samp{--dimension @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]},@* 
@samp{--dmn @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]}@*
@end cartouche
There is more than one way to hyperskin a cat.
The @uref{http://www.unidata.ucar.edu/software/udunits, UDUnits} package 
provides a library which, if present, @acronym{NCO} uses to translate
user-specified physical dimensions into the physical dimensions of data
stored in netCDF files.
Unidata provides UDUnits under the same terms as netCDF, so sites should
install both.
Compiling @acronym{NCO} with UDUnits support is currently optional but
may become required in a future version of @acronym{NCO}.

Two examples suffice to demonstrate the power and convenience of UDUnits  
support. 
@cindex MKS units
First, consider extraction of a variable containing non-record
coordinates with physical dimensions stored in MKS units.
In the following example, the user extracts all wavelengths
in the visible portion of the spectrum in terms of the units
very frequently used in visible spectroscopy, microns:
@example
% ncks --trd -C -H -v wvl -d wvl,"0.4 micron","0.7 micron" in.nc
wvl[0]=5e-07 meter
@end example
@noindent
@cindex @code{units}
The hyperslab returns the correct values because the @var{wvl} variable
is stored on disk with a length dimension that UDUnits recognizes in the 
@code{units} attribute.
The automagical algorithm that implements this functionality is worth
describing since understanding it helps one avoid some potential
pitfalls. 
First, the user includes the physical units of the hyperslab dimensions 
she supplies, separated by a simple space from the numerical values of
the hyperslab limits.
She encloses each coordinate specifications in quotes so that the shell
does not break the @emph{value-space-unit} string into separate
arguments before passing them to @acronym{NCO}. 
Double quotes (@kbd{"foo"}) or single quotes (@kbd{'foo'}) are equally
valid for this purpose. 
Second, @acronym{NCO} recognizes that units translation is requested
because each hyperslab argument contains text characters and non-initial
spaces.  
Third, @acronym{NCO} determines whether the @var{wvl} is dimensioned
with a coordinate variable that has a @code{units} attribute. 
@cindex coordinate variable 
In this case, @var{wvl} itself is a coordinate variable.
The value of its @code{units} attribute is @code{meter}. 
Thus @var{wvl} passes this test so UDUnits conversion is attempted. 
If the coordinate associated with the variable does not contain a 
@code{units} attribute, then @acronym{NCO} aborts.
Fourth, @acronym{NCO} passes the specified and desired dimension strings  
(microns are specified by the user, meters are required by
@acronym{NCO}) to the UDUnits library.
Fifth, the UDUnits library that these dimension are commensurate
and it returns the appropriate linear scaling factors to convert from 
microns to meters to @acronym{NCO}.
If the units are incommensurate (i.e., not expressible in the same
fundamental MKS units), or are not listed in the UDUnits database, then 
NCO aborts since it cannot determine the user's intent.
Finally, @acronym{NCO} uses the scaling information to convert the
user-specified hyperslab limits into the same physical dimensions as
those of the corresponding cooridinate variable on disk.
At this point, @acronym{NCO} can perform a coordinate hyperslab using
the same algorithm as if the user had specified the hyperslab without
requesting units conversion.

@cindex @code{units}
@cindex @code{time} 
The translation and dimensional interpretation of time coordinates
shows a more powerful, and probably more common, UDUnits application.
In this example, the user prints all data between @w{4 PM} and @w{7 PM}
on @w{December 8}, 1999, from a variable whose time dimension is hours 
since the year 1900:
@example
% ncks -u -H -C -v time_udunits -d time_udunits,"1999-12-08 \
  16:00:0.0","1999-12-08 19:00:0.0" in.nc
time_udunits[1]=876018 hours since 1900-01-01 00:00:0.0
@end example
@noindent
@cindex stride
@cindex whitespace
Here, the user invokes the stride (@pxref{Stride}) capability to obtain 
every other timeslice.
This is possible because the UDUnits feature is additive, not
exclusive---it works in conjunction with all other hyperslabbing
(@pxref{Hyperslabs}) options and in all operators which support
hyperslabbing.
The following example shows how one might average data in a 
time period spread across multiple input files
@example
ncra -d time,"1939-09-09 12:00:0.0","1945-05-08 00:00:0.0" \
  in1.nc in2.nc in3.nc out.nc
@end example
@noindent
Note that there is no excess whitespace before or after the individual
elements of the @samp{-d} argument.
@cindex shell 
This is important since, as far as the shell knows, @samp{-d} takes
only @emph{one} command-line argument.
Parsing this argument into its component
@code{@var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]} elements 
(@pxref{Hyperslabs}) is the job of @acronym{NCO}.
When unquoted whitespace is present between these elements, the shell
passes @acronym{NCO} arugment fragments which will not parse as
intended. 

@acronym{NCO} implemented support for the UDUnits2 library with version   
3.9.2 (August, 2007).
The
@uref{http://www.unidata.ucar.edu/software/udunits/udunits-2/udunits2.html,
UDUnits2} package supports non-ASCII characters and logarithmic units. 
We are interested in user-feedback on these features.

@html
<a name="UDUNITS2_XML_PATH"></a> <!-- http://nco.sf.net/nco.html#UDUNITS2_XML_PATH -->
@end html
@cindex @code{UDUNITS2_XML_PATH}
One aspect that deserves mention is that UDUnits, and thus
@acronym{NCO}, supports run-time definition of the location of the
relevant UDUnits databases. 
UDUnits2 (specifically, the function @code{ut_read_xml()}) uses the
environment variable @code{UDUNITS2_XML_PATH}, if any, to find its
all-important @acronym{XML} database, named @file{udunits2.xml} by
default.   
If @code{UDUNITS2_XML_PATH} is undefined, then UDUnits2 looks in the
fall-back default initial location that was hardcoded when the
UDUnits2 library was built. 
This location varies depending upon your operating system and UDUnits2
ncompilation settings. 
If UDUnits2 is correctly linked yet cannot find the @acronym{XML}
database in either of these locations, then @acronym{NCO} will report
that the UDUnits2 library has failed to initialize. 
To fix this, export the full location (path+name) of the UDUnits2
@acronym{XML} database file @file{udunits2.xml} to the shell:
@example
@verbatim
# Example UDUnits2 XML database locations:
export UDUNITS2_XML_PATH='/opt/homebrew/share/udunits/udunits2.xml' # Homebrew
export UDUNITS2_XML_PATH='/opt/local/share/udunits/udunits2.xml' # MacPorts
export UDUNITS2_XML_PATH="${HOME}/anaconda/share/udunits/udunits2.xml" # Anaconda
@end verbatim
@end example
One can then invoke (without recompilation) @acronym{NCO} again, and
UDUnits2 should work. 
This run-time flexibility can enable the full functionality of
pre-built binaries on machines with libraries in different locations.

@ignore
@cindex @code{UDUNITS_PATH}
@acronym{NCO} supported for UDUnits @w{version 1} for many years, 
and deprecated this feature in about 2015.
Some UDUnits @w{version 1} feature may still work (we do not know).
If you are stuck with a UDUnits @w{version 1} installation
you might try to specify the directory which contains the UDUnits
database, @file{udunits.dat}, via the @code{UDUNITS_PATH} environment
variable: 
@example
# UDUnits1
export UDUNITS_PATH='/unusual/location/share/udunits'
@end example
@end ignore

@cindex Climate and Forecast Metadata Convention
@cindex @acronym{CF} conventions
The @uref{http://www.unidata.ucar.edu/software/udunits, UDUnits}
package documentation describes the supported formats of time
dimensions. 
Among the metadata conventions that adhere to these formats are the  
@uref{http://cf-pcmdi.llnl.gov, 
Climate and Forecast (CF) Conventions} and the 
@uref{http://ferret.wrc.noaa.gov/noaa_coop/coop_cdf_profile.html,
Cooperative Ocean/Atmosphere Research Data Service (COARDS) Conventions}.
The following @samp{-d arguments} extract the same data using 
commonly encountered time dimension formats: 
@c fxm add more formats here
@example
-d time,'1918-11-11 00:00:0.0','1939-09-09 00:00:0.0'
-d time,'1918-11-11 00:00:0.0','1939-09-09 00:00:0.0'
-d time,'1918-11-11T00:00:0.0Z','1939-09-09T00:00:0.0Z'
-d time,'1918-11-11','1939-09-09'
-d time,'1918-11-11','1939-9-9'
@end example
@noindent
All of these formats include at least one dash @kbd{-} in a
non-leading character position (a dash in a leading character position 
is a negative sign). 
@acronym{NCO} assumes that a space, colon, or non-leading dash in a
limit string indicates that a UDUnits units conversion is requested.
Some date formats like YYYYMMDD that are valid in UDUnits are ambiguous
to @acronym{NCO} because it cannot distinguish a purely numerical date
(i.e., no dashes or text characters in it) from a coordinate or index
value: 
@example
-d time,1918-11-11 # Interpreted as the date November 11, 1918
-d time,19181111   # Interpreted as time-dimension index 19181111
-d time,19181111.  # Interpreted as time-coordinate value 19181111.0
@end example
Hence, use the YYYY-MM-DD format rather than YYYYMMDD for dates.

@noindent
As of version 4.0.0 (January, 2010), @acronym{NCO} supports some
calendar attributes specified by the @acronym{CF} conventions. 
@table @asis
@item @strong{Supported types:} 
"365_day"/"noleap", "360_day", "gregorian", "standard" 
@item  @strong{Unsupported types:} 
"366_day"/"all_leap","proleptic_gregorian","julian","none" 
@end table
Unsupported types default to mixed Gregorian/Julian as defined by  
UDUnits. 

@noindent An Example: Consider the following netCDF variable
@example
variables:
  double lon_cal(lon_cal) ;
    lon_cal:long_name = "lon_cal" ;
    lon_cal:units = "days since 1964-2-28 0:0:0" ;
    lon_cal:calendar = "365_day" ;
data:
  lon_cal = 1,2,3,4,5,6,7,8,9,10;
@end example 
@samp{ncks -v lon_cal -d lon_cal,'1964-3-1 0:00:0.0','1964-3-4 00:00:0.0'}
results in @code{lon_cal=1,2,3,4}.

@cindex MKS units
@cindex God
netCDF variables should always be stored with @acronym{MKS} (i.e.,
God's) units, so that application programs may assume @acronym{MKS}
dimensions apply to all input variables. 
The UDUnits feature is intended to alleviate @acronym{NCO} users'
pain when handling @acronym{MKS} units. 
It connects users who think in human-friendly units (e.g.,
miles, millibars, days) to extract data which are always stored in
God's units, @acronym{MKS} (e.g., meters, Pascals, seconds). 
The feature is not intended to encourage writers to store data in 
esoteric units (e.g., furlongs, pounds per square inch, fortnights). 

@html
<a name="time_rebase"></a> <!-- http://nco.sf.net/nco.html#time_rebase -->
<a name="rbs"></a> <!-- http://nco.sf.net/nco.html#rbs -->
@end html
@node Rebasing Time Coordinate, Multiple Record Dimensions, UDUnits Support, Shared features
@section Rebasing Time Coordinate
@cartouche
Availability: 
@command{ncra}, @command{ncrcat} 
Short options: None@*
@end cartouche

Time rebasing is invoked when numerous files share a common record
coordinate, and the record coordinate basetime (not the time increment,
e.g., days or hours) changes among input files. 
The rebasing is performed automatically if and only if UDUnits is
installed. 
Rebasing occurs when the record coordinate is a time-based variable, and
times are recorded in units of a time-since-basetime, and the basetime
changes from file to file. 
Since the output file can have only one unit (i.e., one basetime) for
the record coordinate, @acronym{NCO}, in such cases, chooses the units
of the first input file to be the units of the output file.
It is necessary to ``rebase'' all the input record variables to this
output time unit in order for the output file to have the correct
values.  

For example suppose the time coordinate is in hours and each day in
January is stored in its own daily file.
Each daily file records the temperature variable @code{tpt(time)} 
with an (unadjusted) @code{time} coordinate value between 0--23 hours,
and uses the @code{units} attribute to advance the base time:
@example
file01.nc time:units="hours since 1990-1-1"   
file02.nc time:units="hours since 1990-1-2"   
...
file31.nc time:units="hours since 1990-1-31"   
@end example

@example
// Mean noontime temperature in January
ncra -v tpt -d time,"1990-1-1 12:00:00","1990-1-31 23:59:59",24 \
      file??.nc noon.nc    

// Concatenate day2 noon through day3 noon records
ncrcat -v tpt -d time,"1990-1-2 12:00:00","1990-1-3 11:59:59" \ 
      file01.nc file02.nc file03.nc noon.nc    

// Results: time is "re-based" to the time units in "file01.nc"
time=36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, \
     51, 52, 53, 54, 55, 56, 57, 58, 59 ;
  
// If we repeat the above command but with only two input files...
ncrcat -v tpt -d time,"1990-1-2 12:00:00","1990-1-3 11:59:59" \
      file02.nc file03 noon.nc    

// ...then output time coordinate is based on time units in "file02.nc"
time = 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, \ 
     26, 27, 28, 29, 30, 31, 32, 33, 34, 35 ;
@end example

As of @acronym{NCO} version 4.2.1 (August, 2012), @acronym{NCO}
automatically rebases not only the record coordinate (@code{time}, here) 
but also any cell boundaries associated with the record coordinate
(e.g., @code{time_bnds}) (@pxref{CF Conventions}). 

As of @acronym{NCO} version 4.4.9 (May, 2015), @acronym{NCO}
also rebases any climatology boundaries associated with the record
coordinate (e.g., @code{climatology_bounds}) (@pxref{CF Conventions}). 

As of @acronym{NCO} version 4.6.3 (December, 2016), @acronym{NCO}
also rebases the time coordinate when the units differ between files.
For example the first file may have @code{units="days since 2014-03-01"} 
and the second file @code{units="hours since 2014-03-10 00:00"}.

@html
<a name="mrd"></a> <!-- http://nco.sf.net/nco.html#mrd -->
@end html
@node Multiple Record Dimensions, Missing Values, Rebasing Time Coordinate, Shared features
@section Multiple Record Dimensions
@cindex netCDF4
@cindex @code{--mrd}
@cindex @code{--multiple_record_dimensions}
@cartouche
Availability: 
@command{ncecat}, @command{ncpdq} 
Short options: None@*
Long options: @samp{--mrd}@*
@end cartouche
The netCDF3 file format allows only one record dimension, and that
dimension must be the first dimension (i.e., the least rapidly varying 
dimension) of any variable in which it appears.
This imposes certain rules on how operators must perform operations
that alter the ordering of dimensions or the number of record variables.
The netCDF4 file format has no such restrictions.
Files and variables may have any number of record dimensions in any
order.
This additional flexibility of netCDF4 can only be realized by
selectively abandoning the constraints that would make operations
behave completely consistently between netCDF3 and netCDF4 files.

@acronym{NCO} chooses, by default, to impose netCDF3-based constraints
on netCDF4 files. 
This reduces the number of unanticipated consequences and keeps the
operators functioning in a familiar way.
Put another way, @acronym{NCO} limits production of additional record
dimensions so processing netCDF4 files leads to the same results as
processing netCDF3 files.
Users can override this default with the @samp{--mrd} (or 
@samp{--multiple_record_dimension}) switch, which enables netCDF4
variables to accumulate additional record dimensions.

How can additional record dimensions be produced?
Most commonly @command{ncecat} (in record-aggregate mode) defines a new
leading record dimension.
In netCDF4 files this becomes an additional record dimension unless the
original record dimension is changed to a fixed dimension (as must be
done in netCDF3 files). 
Also when @command{ncpdq} reorders dimensions it can preserve the
``record'' property of record variables.
@command{ncpdq} tries to define as a record dimension whichever
dimension ends up first in a record variable, and, in netCDF4 files,
this becomes an additional record dimension unless the original record
dimension is changed to a fixed dimension (as must be done in netCDF3
files). 
It it easier if @command{ncpdq} and @command{ncecat} do not increase
the number of record dimensions in a variable so that is the default.
Use @samp{--mrd} to override this.

@html
<a name="missing_value"></a> <!-- http://nco.sf.net/nco.html#missing_value -->
<a name="_FillValue"></a> <!-- http://nco.sf.net/nco.html#_FillValue -->
<a name="fll_val"></a> <!-- http://nco.sf.net/nco.html#fll_val -->
<a name="mss_val"></a> <!-- http://nco.sf.net/nco.html#mss_val -->
<a name="mss"></a> <!-- http://nco.sf.net/nco.html#mss -->
@end html
@node Missing Values, Chunking, Multiple Record Dimensions, Shared features
@section Missing values
@cindex missing values
@cindex data, missing 
@cindex averaging data
@cindex @code{missing_value}
@cindex @code{_FillValue}
@cartouche
Availability: @command{ncap2}, @command{ncbo}, @command{ncclimo},
@command{nces}, @command{ncflint}, @command{ncpdq}, @command{ncra},  
@command{ncremap}, @command{ncwa}@* 
Short options: None@*
@end cartouche

The phrase @dfn{missing data} refers to data points that are missing,
invalid, or for any reason not intended to be arithmetically processed
in the same fashion as valid data.  
@cindex arithmetic operators
All @acronym{NCO} arithmetic operators attempt to handle missing data in 
an intelligent fashion. 
There are four steps in the @acronym{NCO} treatment of missing data:
@enumerate
@item 
Identifying variables that may contain missing data. 

@acronym{NCO} follows the convention that missing data should be stored
with the @var{_FillValue} specified in the variable's @code{_FillValue} 
attributes. 
The @emph{only} way @acronym{NCO} recognizes that a variable @emph{may}
contain missing data is if the variable has a @code{_FillValue}
attribute. 
In this case, any elements of the variable which are numerically equal
to the @var{_FillValue} are treated as missing data.

@acronym{NCO} adopted the behavior that the default attribute name, if 
any, assumed to specify the value of data to ignore is @code{_FillValue} 
with version 3.9.2 (August, 2007).
Prior to that, the @code{missing_value} attribute, if any, was assumed to  
specify the value of data to ignore.
Supporting both of these attributes simultaneously is not practical.
Hence the behavior @acronym{NCO} once applied to @var{missing_value} it
now applies to any @var{_FillValue}. 
@acronym{NCO} now treats any @var{missing_value} as normal data 
@footnote{
The old functionality, i.e., where the ignored values are indicated by
@code{missing_value} not @code{_FillValue}, may still be selected 
@emph{at @acronym{NCO} build time} by compiling @acronym{NCO} 
with the token definition 
@c @kbd{CPPFLAGS='-DNCO_MSS_VAL_SNG=missing_value'}.
@kbd{CPPFLAGS='-UNCO_USE_FILL_VALUE'}.
}.

@findex ncrename
@findex ncatted
It has been and remains most advisable to create both @code{_FillValue} 
and @code{missing_value} attributes with identical values in datasets.
Many legacy datasets contain only @code{missing_value} attributes.
@acronym{NCO} can help migrating datasets between these conventions.
One may use @command{ncrename} (@pxref{ncrename netCDF Renamer}) to
rename all @code{missing_value} attributes to @code{_FillValue}:
@example
ncrename -a .missing_value,_FillValue inout.nc
@end example
Alternatively, one may use
@command{ncatted} (@pxref{ncatted netCDF Attribute Editor}) to
add a @code{_FillValue} attribute to all variables
@example
ncatted -O -a _FillValue,,o,f,1.0e36 inout.nc
@end example

@item 
Converting the @var{_FillValue} to the type of the variable, if
neccessary. 

Consider a variable @var{var} of type @var{var_type} with a
@code{_FillValue} attribute of type @var{att_type} containing the
value @var{_FillValue}.  
As a guideline, the type of the @code{_FillValue} attribute should be
the same as the type of the variable it is attached to.
If @var{var_type} equals @var{att_type} then @acronym{NCO}
straightforwardly compares each value of @var{var} to
@var{_FillValue} to determine which elements of @var{var} are to be
treated as missing data. 
@cindex C language
If not, then @acronym{NCO} converts @var{_FillValue} from
@var{att_type} to @var{var_type} by using the implicit conversion rules
@w{of C}, or, if @var{att_type} is @code{NC_CHAR}
@footnote{For example, the @acronym{DOE} @acronym{ARM} program often
uses @var{att_type} = @code{NC_CHAR} and @var{_FillValue} =
@samp{-99999.}. 
}, by typecasting the results of the @w{C function}
@code{strtod(@var{_FillValue})}. 
@cindex @command{ncatted}
You may use the @acronym{NCO} operator @command{ncatted} to change the
@code{_FillValue} attribute and all data whose data is
@var{_FillValue} to a new value
(@pxref{ncatted netCDF Attribute Editor}).

@item 
Identifying missing data during arithmetic operations.

@cindex performance
@cindex operator speed
@cindex speed
@cindex execution time
@cindex arithmetic operators
When an @acronym{NCO} arithmetic operator processes a variable @var{var}
with a @code{_FillValue} attribute, it compares each value of
@var{var} to @var{_FillValue} before performing an operation.
Note the @var{_FillValue} comparison imposes a performance penalty
on the operator.
Arithmetic processing of variables which contain the
@code{_FillValue} attribute always incurs this penalty, even when
none of the data are missing.
Conversely, arithmetic processing of variables which do not contain the
@code{_FillValue} attribute never incurs this penalty.
In other words, do not attach a @code{_FillValue} attribute to a
variable which does not contain missing data.
This exhortation can usually be obeyed for model generated data, but it
may be harder to know in advance whether all observational data will be
valid or not.

@item 
Treatment of any data identified as missing in arithmetic operators.

@cindex @command{nces}
@cindex @command{ncra}
@cindex @command{ncwa}
@cindex @command{ncbo}
@cindex @command{ncflint}
@acronym{NCO} averagers (@command{ncra}, @command{nces}, @command{ncwa})
do not count any element with the value @var{_FillValue} towards the
average. 
@command{ncbo} and @command{ncflint} define a @var{_FillValue} result  
when either of the input values is a @var{_FillValue}.
Sometimes the @var{_FillValue} may change from file to file in a
multi-file operator, e.g., @command{ncra}.
@acronym{NCO} is written to account for this (it always compares a
variable to the @var{_FillValue} assigned to that variable in the
current file). 
Suffice it to say that, in all known cases, @acronym{NCO} does ``the
right thing''. 

It is impossible to determine and store the correct result of a binary  
operation in a single variable.
One such corner case occurs when both operands have differing
@var{_FillValue} attributes, i.e., attributes with different
numerical values.
Since the output (result) of the operation can only have one
@var{_FillValue}, some information may be lost.
In this case, @acronym{NCO} always defines the output variable to have
the same @var{_FillValue} as the first input variable.
Prior to performing the arithmetic operation, all values of the second
operand equal to the second @var{_FillValue} are replaced with the
first @var{_FillValue}.
Then the arithmetic operation proceeds as normal, comparing each element 
of each operand to a single @var{_FillValue}.
Comparing each element to two distinct @var{_FillValue}'s would be
much slower and would be no likelier to yield a more satisfactory
answer. 
In practice, judicious choice of @var{_FillValue} values prevents any
important information from being lost.
@end enumerate

@html
<a name="chunking"></a> <!-- http://nco.sf.net/nco.html#chunking -->
<a name="cnk"></a> <!-- http://nco.sf.net/nco.html#cnk -->
<a name="cnk_sz"></a> <!-- http://nco.sf.net/nco.html#cnk_sz -->
<a name="chunk_size"></a> <!-- http://nco.sf.net/nco.html#chunk_size -->
@end html
@node Chunking, Compression, Missing Values, Shared features
@section Chunking
@cindex @code{--cnk_byt}
@cindex @code{--cnk_csh}
@cindex @code{--cnk_dmn}
@cindex @code{--cnk_map}
@cindex @code{--cnk_min}
@cindex @code{--cnk_plc}
@cindex @code{--cnk_scl}
@cindex @code{--chunk_byte}
@cindex @code{--chunk_cache}
@cindex @code{--chunk_dimension}
@cindex @code{--chunk_map}
@cindex @code{--chunk_min}
@cindex @code{--chunk_policy}
@cindex @code{--chunk_scalar}
@cindex chunking
@cartouche
Availability: @command{ncap2}, @command{ncbo}, @command{nces},
@command{ncecat}, @command{ncflint}, @command{ncks}, @command{ncpdq},
@command{ncra}, @command{ncrcat}, @command{ncwa}@*
Short options: none@*
Long options: 
@samp{--cnk_byt @var{sz_byt}}, @samp{--chunk_byte @var{sz_byt}}@*
@samp{--cnk_csh @var{sz_byt}}, @samp{--chunk_cache @var{sz_byt}}@*
@samp{--cnk_dmn @var{dmn_nm},@var{sz_lmn}},
@samp{--chunk_dimension @var{dmn_nm},@var{sz_lmn}}@*,
@samp{--cnk_map @var{cnk_map}}, @samp{--chunk_map @var{cnk_map}},@*
@samp{--cnk_min @var{sz_byt}}, @samp{--chunk_min @var{sz_byt}},@*
@samp{--cnk_plc @var{cnk_plc}}, @samp{--chunk_policy @var{cnk_plc}},@*
@samp{--cnk_scl @var{sz_lmn}}, @samp{--chunk_scalar @var{sz_lmn}}@*
@end cartouche

All netCDF4-enabled @acronym{NCO} operators that define variables 
support a plethora of chunksize options.
Chunking can significantly accelerate or degrade read/write access
to large datasets.
Dataset chunking issues are described by @acronym{THG} and Unidata 
@uref{http://www.hdfgroup.org/HDF5/doc/H5.user/Chunking.html,here},
@uref{http://www.unidata.ucar.edu/blogs/developer/en/entry/chunking_data_why_it_matters,here},
and
@uref{http://www.unidata.ucar.edu/blogs/developer/en/entry/chunking_data_choosing_shapes,here}.
@acronym{NCO} authors are working on generalized algorithms and
applications of chunking strategies (stay tuned for more in 2018).

@html
<a name="chunk_cache"></a> <!-- http://nco.sf.net/nco.html#chunk_cache -->
<a name="cnk_csh"></a> <!-- http://nco.sf.net/nco.html#cnk_csh -->
<a name="cache"></a> <!-- http://nco.sf.net/nco.html#cache -->
<a name="csh"></a> <!-- http://nco.sf.net/nco.html#csh -->
@end html
@cindex chunk cache size
@cindex cache size
@cindex @code{--cnk_csh @var{sz}}
@cindex @code{--chunk_cache @var{sz}}
As of @acronym{NCO} version 4.6.5 (March, 2017), @acronym{NCO} supports
run-time alteration of the chunk cache size.
By default, the cache size is set (by the @code{--with-chunk-cache-size}
option to @command{configure}) at netCDF compile time. 
The @code{--cnk_csh @var{sz}} option sets the cache size to @var{sz}
bytes for all variables.
When the debugging level is set (with @code{-D @var{dbg_lvl}}) to three
or higher, @acronym{NCO} prints the current value of the cache settings
for informational purposes. 
Also @samp{--chunk_cache}.

Increasing cache size from the default can dramatically accelerate time
to aggregate and rechunk multiple large input datasets, e.g.,
@example
ncrcat -4 -L 1 --cnk_csh=1000000000 --cnk_plc=g3d --cnk_dmn=time,365 \
       --cnk_dmn=lat,1800 --cnk_dmn=lon,3600 in*.nc4 out.nc
@end example
In this example all 3D variables the input datasets (which may or may
not be chunked already) are re-chunked to a size of 365 along the time
dimension. 
Because the default chunk cache size of about @w{4 MB} is too small to
manipulate the large chunks, we reset the cache to @w{1 GB}.
The operation completes much faster, and subsequent reads along the
time dimension will be much more rapid.

@html
<a name="blocksize"></a> <!-- http://nco.sf.net/nco.html#blocksize -->
<a name="blk"></a> <!-- http://nco.sf.net/nco.html#blk -->
@end html
@cindex chunking policy
@cindex chunking map
@cindex chunksize
@cindex blocksize
The @acronym{NCO} chunking implementation is designed to be flexible. 
Users control four aspects of the chunking implementation.
These are the @dfn{chunking policy}, @dfn{chunking map},
@dfn{chunksize}, and @dfn{minimum chunksize}.
The chunking policy determines @emph{which} variables to chunk, and
the chunking map determines how (with what exact sizes) to chunk
those variables.
These are high-level mechanisms that apply to an entire file and all
variables and dimensions.
The chunksize option allows per-dimension specification of sizes that 
will override the selected (or default) chunking map.

The distinction between elements and bytes is subtle yet crucial to
understand. 
Elements refers to values of an array, whereas bytes refers to the
memory size required to hold the elements. 
These measures differ by a factor of four or eight for @code{NC_FLOAT}
or @code{NC_DOUBLE}, respectively. 
The option @samp{--cnk_scl} takes an argument @var{sz_lmn} measured in
elements.
The options @samp{--cnk_byt}, @samp{--cnk_csh}, and @samp{--cnk_min}
take arguments @var{sz_byt} measured in bytes.

Use the @samp{--cnk_min=@var{sz_byt}} option to set the minimum size in
bytes (not elements) of variables to chunk.  
This threshold is intended to restrict use of chunking to variables
for which it is efficient. 
By default this minimum variable size for chunking is twice the system 
blocksize (when available) and is @w{8192 bytes} otherwise. 
Users may set this to any value with the @samp{--cnk_min=@var{sz_byt}}
switch. 
To guarantee that chunking is performed on all arrays, regardless of
size, set the minimum size to one byte (not to zero bytes).

@cindex hyperslab
@findex ncpdq
@cindex packing
The chunking implementation is similar to a hybrid of the
@command{ncpdq} packing policies 
(@pxref{ncpdq netCDF Permute Dimensions Quickly}) and hyperslab
specifications (@pxref{Hyperslabs}).  
Each aspect is intended to have a sensible default, so that many users 
only need to set one switch to obtain sensible chunking. 
Power users can tune chunking with the three switches in tandem to
obtain optimal performance. 

By default, @acronym{NCO} preserves the chunking characteristics
of the input file in the output file
@footnote{This behavior became the default in November 2014 with
@acronym{NCO} version 4.4.7.
Prior versions would always use netCDF default chunking in the output
file when no @acronym{NCO} chunking switches were activated, regardless
of the chunking in the input file.}.
In other words, preserving chunking requires no switches or user
intervention.

Users specify the desired chunking policy with the @samp{-P} switch 
(or its long option equivalents, @samp{--cnk_plc} and
@samp{--chunk_policy}) and its @var{cnk_plc} argument.
As of August, 2014, six chunking policies are implemented:@*
@cindex @samp{all}
@cindex @samp{g2d}
@cindex @samp{g3d}
@cindex @samp{r1d}
@cindex @samp{xpl}
@cindex @samp{xst}
@cindex @samp{cnk_all}
@cindex @samp{cnk_g2d}
@cindex @samp{cnk_g3d}
@cindex @samp{cnk_r1d}
@cindex @samp{cnk_xpl}
@cindex @samp{cnk_xst}
@cindex @samp{plc_all}
@cindex @samp{plc_g2d}
@cindex @samp{plc_g3d}
@cindex @samp{plc_r1d}
@cindex @samp{plc_xpl}
@cindex @samp{plc_xst}
@table @dfn
@item Chunk All Variables
Definition: Chunk all variables possible.
For obvious reasons, scalar variables cannot be chunked.@*
Alternate invocation: @code{ncchunk}@*
@var{cnk_plc} key values: @samp{all}, @samp{cnk_all}, @samp{plc_all}@*
Mnemonic: All@*
@item Chunk Variables with at least Two Dimensions [@emph{default}]
Definition: Chunk all variables possible with at least two dimensions@*
Alternate invocation: none@*
@var{cnk_plc} key values: @samp{g2d}, @samp{cnk_g2d}, @samp{plc_g2d}@*
Mnemonic: @emph{G}reater than or equal to @emph{2} @emph{D}imensions@*
@item Chunk Variables with at least Three Dimensions
Definition: Chunk all variables possible with at least three dimensions@*
Alternate invocation: none@*
@var{cnk_plc} key values: @samp{g3d}, @samp{cnk_g3d}, @samp{plc_g3d}@*
Mnemonic: @emph{G}reater than or equal to @emph{3} @emph{D}imensions@*
@item Chunk One-Dimensional Record Variables
Definition: Chunk all 1-D record variables@*
Alternate invocation: none@*
Any specified (with @samp{--cnk_dmn}) record dimension chunksizes will
be applied only to 1-D record variables (and to no other variables).
Other dimensions may be chunked with their own @samp{--cnk_dmn} options 
that will apply to all variables. 
@var{cnk_plc} key values: @samp{r1d}, @samp{cnk_r1d}, @samp{plc_r1d}@*
Mnemonic: @emph{R}ecord @emph{1}-@emph{D} variables@*
@item Chunk Variables Containing Explicitly Chunked Dimensions
Definition: Chunk all variables possible that contain at least one
dimension whose chunksize was explicitly set with the @samp{--cnk_dmn} option.
Alternate invocation: none@*
@var{cnk_plc} key values: @samp{xpl}, @samp{cnk_xpl}, @samp{plc_xpl}@*
Mnemonic: E@emph{XPL}icitly specified dimensions@*
@item Chunk Variables that are already Chunked
Definition: Chunk only variables that are already chunked in the input
file. 
When used in conjunction with @samp{cnk_map=xst} this option preserves
and copies the chunking parameters from the input to the output file.
Alternate invocation: none@*
@var{cnk_plc} key values: @samp{xst}, @samp{cnk_xst}, @samp{plc_xst}@*
Mnemonic: E@emph{X}i@emph{ST}ing chunked variables@*
@item Chunk Variables with @acronym{NCO} recommendations
Definition: Chunk all variables according to @acronym{NCO} best
practices. 
This is a virtual option that ensures the chunking policy is (in the 
subjective opinion of the authors) the best policy for typical usage.
As of @acronym{NCO} version 4.4.8 (February, 2015), this virtual policy
implements @samp{map_rew} for 3-D variables and @samp{map_lfp} for all
other variables.@*
Alternate invocation: none@*
@var{cnk_plc} key values: @samp{nco}, @samp{cnk_nco}, @samp{plc_nco}@*
Mnemonic: @emph{N}et@emph{C}DF@emph{O}perator@*
@item Unchunking
Definition: Unchunk all variables possible. 
The @acronym{HDF5} storge layer requires that record variables (i.e.,
variables that contain at least one record dimension) must be chunked.
Also variables that are compressed or use checksums must be chunked.
Such variables cannot be unchunked.@*  
Alternate invocation: @code{ncunchunk}@*
@var{cnk_plc} key values: @samp{uck}, @samp{cnk_uck}, @samp{plc_uck}, @samp{none}, @samp{unchunk}@*
Mnemonic: @emph{U}n@emph{C}hun@emph{K}@*
@end table
@noindent
Equivalent key values are fully interchangeable.
Multiple equivalent options are provided to satisfy disparate needs
and tastes of @acronym{NCO} users working with scripts and from the
command line.

@cindex chunking map
@cindex degenerate dimension
@cindex @var{cnk_map}
@cindex @code{-M @var{cnk_map}}
@cindex @code{--cnk_map @var{cnk_map}}
@cindex @code{--map @var{cnk_map}}
The chunking algorithms must know the chunksizes of each dimension of
each variable to be chunked.
The correspondence between the input variable shape and the chunksizes
is called the @dfn{chunking map}. 
The user specifies the desired chunking map with the @samp{-M} switch
(or its long option equivalents, @samp{--cnk_map} and
@samp{--chunk_map}) and its @var{cnk_map} argument.
Nine chunking maps are currently implemented:@*
@cindex @samp{dmn}
@cindex @samp{scl}
@cindex @samp{prd}
@cindex @samp{lfp}
@cindex @samp{rd1}
@cindex @samp{xst}
@cindex @samp{rew}
@cindex @samp{nc4}
@cindex @samp{nco}
@cindex @samp{cnk_dmn}
@cindex @samp{cnk_scl}
@cindex @samp{cnk_prd}
@cindex @samp{cnk_lfp}
@cindex @samp{cnk_rd1}
@cindex @samp{cnk_xst}
@cindex @samp{cnk_rew}
@cindex @samp{cnk_nc4}
@cindex @samp{cnk_nco}
@cindex @samp{map_dmn}
@cindex @samp{map_scl}
@cindex @samp{map_prd}
@cindex @samp{map_lfp}
@cindex @samp{map_rd1}
@cindex @samp{map_xst}
@cindex @samp{map_rew}
@cindex @samp{map_nc4}
@cindex @samp{map_nco}
@cindex Chris Barker
@table @dfn
@item Chunksize Equals Dimension Size
Definition: Chunksize defaults to dimension size. 
Explicitly specify chunksizes for particular dimensions with
@samp{--cnk_dmn} option.
In most cases this chunksize will be applied in all variables
that contain the specified dimension.
Some chunking policies noted above allow (fxm), and others (fxm) 
prevent this chunksize from applying to all variables.@*
@var{cnk_map} key values: @samp{dmn}, @samp{cnk_dmn}, @samp{map_dmn}@*
Mnemonic: @emph{D}i@emph{M}e@emph{N}sion@*
@item Chunksize Equals Dimension Size except Record Dimension
Definition: Chunksize equals dimension size except record dimension has size one.
Explicitly specify chunksizes for particular dimensions with
@samp{--cnk_dmn} option.@*
@var{cnk_map} key values: @samp{rd1}, @samp{cnk_rd1}, @samp{map_rd1}@*
Mnemonic: @emph{R}ecord @emph{D}imension size @emph{1}@*
@item Chunksize Equals Scalar Size Specified
Definition: Chunksize for all dimensions is set with the
@samp{--cnk_scl=@var{sz_lmn}} option.
For this map @var{sz_lmn} itself becomes the chunksize of each
dimension.  
This is in contrast to the @var{cnk_prd} map, where the @var{r}th root
of @var{sz_lmn}) becomes the chunksize of each dimension.@* 
@var{cnk_map} key values: @samp{scl}, @samp{cnk_scl}, @samp{map_scl}@*
Mnemonic: @emph{SC}a@emph{L}ar@*
@var{cnk_map} key values: @samp{xpl}, @samp{cnk_xpl}, @samp{map_xpl}@*
Mnemonic: E@emph{XPL}icitly specified dimensions@*
@item Chunksize Product Matches Scalar Size Specified
Definition: The product of the chunksizes for each variable
matches (approximately equals) the size specified with the
@samp{--cnk_scl=@var{sz_lmn}} option.
A dimension of size one is said to be @emph{degenerate}.
For a variable of rank @var{R} (i.e., with @var{R} non-degenerate
dimensions), the chunksize in each non-degenerate dimension is
(approximately) the @var{R}th root of @var{sz_lmn}.
This is in contrast to the @var{cnk_scl} map, where @var{sz_lmn} itself
becomes the chunksize of each dimension.@* 
@var{cnk_map} key values: @samp{prd}, @samp{cnk_prd}, @samp{map_prd}@*
Mnemonic: @emph{PR}o@emph{D}uct@*
@item Chunksize Lefter Product Matches Scalar Size Specified
Definition: The product of the chunksizes for each variable
(approximately) equals the size specified with the
@samp{--cnk_byt=@var{sz_byt}} (not @samp{--cnk_dfl}) option.
This is accomplished by using dimension sizes as chunksizes for the
rightmost (most rapidly varying) dimensions, and then ``flexing'' the
chunksize of the leftmost (least rapidly varying) dimensions such that 
the product of all chunksizes matches the specified size.
All @var{L}-dimensions to the left of and including the first record 
dimension define the left-hand side.
To be precise, if the total size (in bytes) of the variable is
@var{var_sz}, and if the specified (with @samp{--cnk_byt}) product of
the @var{R} ``righter'' dimensions (those that vary more rapidly than
the first record dimension) is @var{sz_byt}, then chunksize (in bytes)
of each of the @var{L} lefter dimensions is (approximately) the
@var{L}th root of @math{@var{var_sz}/@var{sz_byt}}.
This map was first proposed by Chris Barker.@*
@var{cnk_map} key values: @samp{lfp}, @samp{cnk_lfp}, @samp{map_lfp}@*
Mnemonic: @emph{L}e@emph{F}ter @emph{P}roduct@*
@item Chunksize Equals Existing Chunksize
Definition: Chunksizes are copied from the input to the output
file for every variable that is chunked in the input file.
Variables not chunked in the input file will be chunked with 
default mappings.@*
@var{cnk_map} key values: @samp{xst}, @samp{cnk_xst}, @samp{map_xst}@*
Mnemonic: E@emph{X}i@emph{ST}@*
@item Chunksize Balances 1D and (N-1)-D Access to N-D Variable [@emph{default for netCDF4 input}]
Definition: Chunksizes are chosen so that 1-D and (@var{N-1})-D
hyperslabs of @var{3}-D variables (e.g., point-timeseries or
latitude/longitude surfaces of 3-D fields) both require approximately
the same number of chunks. 
Hence their access time should be balanced.
Russ Rew explains the motivation and derivation for this strategy 
@uref{http://www.unidata.ucar.edu/blogs/developer/en/entry/chunking_data_choosing_shapes,
here}.@*
@var{cnk_map} key values: @samp{rew}, @samp{cnk_rew}, @samp{map_rew}@*
Mnemonic: Russ @emph{REW}@*
@item Chunksizes use netCDF4 defaults
Definition: Chunksizes are determined by the underlying netCDF library.
All variables selected by the current chunking policy have their
chunksizes determined by netCDF library defaults. 
The default algorithm netCDF uses to determine chunksizes has changed
through the years, and thus depends on the netCDF library version.
This map can be used to reset (portions of) previously chunked files to
default chunking values.@*
@var{cnk_map} key values: @samp{nc4}, @samp{cnk_nc4}, @samp{map_nc4}@*
Mnemonic: @emph{N}et@emph{C}DF@emph{4}@*
@item Chunksizes use @acronym{NCO} recommendations [@emph{default for netCDF3 input}]
Definition: Chunksizes are determined by the currently recommended
@acronym{NCO} map.
This is a virtual option that ensures the chunking map is (in the 
subjective opinion of the authors) the best map for typical usage.
As of @acronym{NCO} version 4.4.9 (May, 2015), this virtual map
calls @samp{map_lfp}.@*
@c for 3-D variables and @samp{map_lfp} for all others.@*  
@var{cnk_map} key values: @samp{nco}, @samp{cnk_nco}, @samp{map_nco}@*
Mnemonic: @emph{N}et@emph{C}DF@emph{O}perator@*

@end table
@noindent
It is possible to combine the above chunking map algorithms with
user-specified per-dimension (though not per-variable) chunksizes that 
override specific chunksizes determined by the maps above. 
The user specifies the per-dimension chunksizes with the (equivalent) 
long options @samp{--cnk_dmn} or @samp{--chunk_dimension}).
The option takes two comma-separated arguments,
@var{dmn_nm},@var{sz_lmn}, which are the dimension name and its
chunksize (in elements, not bytes), respectively. 
The @samp{--cnk_dmn} option may be used as many times as necessary.

The default behavior of chunking depends on several factors.
As mentioned above, when no chunking options are explicitly specified by
the user, then @acronym{NCO} preserves the chunking characteristics of
the input file in the output file.
This is equivalent to specifying both @var{cnk_plc} and @var{cnk_map} 
as ``existing'', i.e., @samp{--cnk_plc=xst --cnk_map=xst}.
If 
output netCDF4 files are chunked with the default behavior of the 
netCDF4 library.

When any chunking parameter @emph{except} @samp{cnk_plc} or
@samp{cnk_map} is specified (such as @samp{cnk_dmn} or
@samp{cnk_scl}), then the ``existing'' policy and map are
retained and the output chunksizes are modified where necessary in
accord with the user-specified parameter.
When @samp{cnk_map} is specified and @samp{cnk_plc} is not, then
@acronym{NCO} picks (what it thinks is) the optimal chunking policy. 
This has always been policy @samp{map_g2d}.
When @samp{cnk_plc} is specified and @samp{cnk_map} is not, then
@acronym{NCO} picks (what it thinks is) the optimal chunking map.
This has always been map @samp{map_rd1}.

To start afresh and return to netCDF4 chunking defaults, select
@samp{cnk_map=nc4}. 

@html
<a name="xmp_cnk"></a> <!-- http://nco.sf.net/nco.html#xmp_cnk -->
<a name="xmp_chunk"></a> <!-- http://nco.sf.net/nco.html#xmp_chunk -->
@end html
@example
# Simple chunking and unchunking
ncks -O -4 --cnk_plc=all     in.nc out.nc # Chunk in.nc
ncks -O -4 --cnk_plc=unchunk in.nc out.nc # Unchunk in.nc

# Chunk data then unchunk it, printing informative metadata
ncks -O -4 -D 4 --cnk_plc=all ~/nco/data/in.nc ~/foo.nc
ncks -O -4 -D 4 --cnk_plc=uck ~/foo.nc ~/foo.nc

# Set total chunksize to 8192 B
ncks -O -4 -D 4 --cnk_plc=all --cnk_byt=8192 ~/nco/data/in.nc ~/foo.nc

# More complex chunking procedures, with informative metadata
ncks -O -4 -D 4 --cnk_scl=8 ~/nco/data/in.nc ~/foo.nc
ncks -O -4 -D 4 --cnk_scl=8 dstmch90_clm.nc ~/foo.nc
ncks -O -4 -D 4 --cnk_dmn lat,64 --cnk_dmn lon,128 dstmch90_clm.nc \ 
 ~/foo.nc 
ncks -O -4 -D 4 --cnk_plc=uck ~/foo.nc ~/foo.nc
ncks -O -4 -D 4 --cnk_plc=g2d --cnk_map=rd1 --cnk_dmn lat,32 \
 --cnk_dmn lon,128 dstmch90_clm_0112.nc ~/foo.nc

# Chunking works with all operators...
ncap2 -O -4 -D 4 --cnk_scl=8 -S ~/nco/data/ncap2_tst.nco \ 
 ~/nco/data/in.nc ~/foo.nc
ncbo -O -4 -D 4 --cnk_scl=8 -p ~/nco/data in.nc in.nc ~/foo.nc
ncecat -O -4 -D 4 -n 12,2,1 --cnk_dmn lat,32 \ 
 -p /data/zender/dstmch90 dstmch90_clm01.nc ~/foo.nc
ncflint -O -4 -D 4 --cnk_scl=8 ~/nco/data/in.nc ~/foo.nc
ncpdq -O -4 -D 4 -P all_new --cnk_scl=8 -L 5 ~/nco/data/in.nc ~/foo.nc
ncrcat -O -4 -D 4 -n 12,2,1 --cnk_dmn lat,32 \ 
 -p /data/zender/dstmch90 dstmch90_clm01.nc ~/foo.nc
ncwa -O -4 -D 4 -a time --cnk_plc=g2d --cnk_map=rd1 --cnk_dmn lat,32 \ 
 --cnk_dmn lon,128 dstmch90_clm_0112.nc ~/foo.nc
@end example

@html
<a name="r1d"></a> <!-- http://nco.sf.net/nco.html#r1d -->
@end html
Chunking policy @samp{r1d} changes the chunksize of 1-D record variables
(and no other variables) to that specified (with @samp{--cnk_dmn})
chunksize. 
Any specified record dimension chunksizes will be applied to 1-D
record variables only. 
Other dimensions may be chunked with their own @samp{--cnk_dmn} options
that will apply to all variables. 
For example, 
@example
ncks --cnk_plc=r1d --cnk_dmn=time,1000. in.nc out.nc
@end example
This sets @code{time} chunks to 1000 only in 1-D record variables. 
Without the @samp{r1d} policy, @code{time} chunks would change in all 
variables.   

@cindex record dimension
It is appropriate to conclude by informing users about an aspect of
chunking that may not be expected.
Three types of variables are @emph{always} chunked: Record variables,
Deflated (compressed) variables, and Checksummed variables.
Hence all variables that contain a record dimension are also chunked
(since data must be chunked in all dimensions, not just one).
Unless otherwise specified by the user, the other (fixed, non-record) 
dimensions of record variables are assigned default chunk sizes. 
The @acronym{HDF5} layer does all this automatically to optimize the
on-disk variable/file storage geometry of record variables.
Do not be surprised to learn that files created without any explicit
instructions to activate chunking nevertheless contain chunked
variables. 

@html
<a name="bg"></a> <!-- http://nco.sf.net/nco.html#bg -->
<a name="bitgrooming"></a> <!-- http://nco.sf.net/nco.html#bitgrooming -->
<a name="Bit-Grooming"></a> <!-- http://nco.sf.net/nco.html#Bit-Grooming -->
<a name="BG"></a> <!-- http://nco.sf.net/nco.html#BG -->
<a name="ppc"></a> <!-- http://nco.sf.net/nco.html#ppc -->
<a name="PPC"></a> <!-- http://nco.sf.net/nco.html#PPC -->
<a name="compression"></a> <!-- http://nco.sf.net/nco.html#compression -->
<a name="cmp"></a> <!-- http://nco.sf.net/nco.html#cmp -->
<a name="nsd"></a> <!-- http://nco.sf.net/nco.html#nsd -->
<a name="NSD"></a> <!-- http://nco.sf.net/nco.html#NSD -->
<a name="dsd"></a> <!-- http://nco.sf.net/nco.html#dsd -->
<a name="DSD"></a> <!-- http://nco.sf.net/nco.html#DSD -->
<a name="qnt"></a> <!-- http://nco.sf.net/nco.html#qnt -->
@end html
@node Compression, Deflation, Chunking, Shared features
@section Compression
@cindex @code{--cmp_sng}
@cindex @code{--compression}
@cindex codecs
@cindex Zstandard
@cindex Bzip2
@cindex Granular BitRound
@cindex @code{--ppc}
@cindex @code{--precision_preserving_compression}
@cindex @code{--quantize}
@cindex lossy compression
@cindex quantization
@cindex rounding
@cindex Huffman coding
@cindex @command{gzip}
@cindex @command{zlib}
@cartouche
Availability: @command{ncbo}, @command{ncecat}, @command{nces},
@command{ncflint}, @command{ncks}, @command{ncpdq}, @command{ncra},
@command{ncrcat}, @command{ncwa}@* 
Short options: None@*
Long options: @samp{--ppc @var{var1}[,@var{var2}[,...]]=@var{prc}},@*
@samp{--precision_preserving_compression @var{var1}[,@var{var2}[,...]]=@var{prc}},@*
@samp{--quantize @var{var1}[,@var{var2}[,...]]=@var{prc}}@*
@samp{--cmp cmp_sng}@*
@samp{--cmp @var{codec1}[,@var{params1}[|@var{codec2}[,@var{params2}[|...]]]]}@*
@samp{--codec @var{codec1}[,@var{params1}[|@var{codec2}[,@var{params2}[|...]]]]}@*
@end cartouche

Compression is a rapidly developing area in geoscientific software,
and @acronym{NCO} is no exception.
Documentation of these features can quickly become out-of-date.
A brief review of compression support from the early days until now:
@acronym{NCO} first supported Linear Packing with @command{ncpdq}
in 2004.
The advent of netCDF4 allowed @acronym{NCO} to support lossless
compression with the @acronym{DEFLATE} algorithm beginning in 2007.
Nearly a decade elapsed before the next features came in 2015 when,
thanks to support from the @acronym{DOE}, we developed the lossy
BitGroom quantization algorithm.
@acronym{NCO} soon introduced a flexible per-variable @acronym{API}
(@samp{--ppc}) to support it, and its relatives BitShave, and BitSet,
in all arithmetic operators. 
This work helped spur interest and research on other Bit Adjustment
Algorithms (@acronym{BAA}s) that perform quantization.

In 2020 we reduced the quantization error of BitGroom by implementing
@acronym{IEEE}-rounding (aka BitRound), and newer quantization
algorithms including BitGroomRound, HalfShave, and DigitRound
@footnote{R. Kouznetsov contributed the masking techinques used in
BitRound, BitGroomRound, and HalfShave. Thanks Rostislav!}.
These algorithms are all accessible via the @samp{--baa} option.
In 2020 @acronym{NSF} awarded support for us to develop the Community 
Codec Respository to put a friendlier @acronym{API} on the
@acronym{HDF5} shared-library filter mechanism so that all netCDF
users could shift to more modern and efficient codecs than
@acronym{DEFLATE}. 
This strategy aligned with needs of operational forecasters supported
by software engineers at the @acronym{NOAA} Environmental Modeling
Center (@acronym{EMC}), who contributed to developing and testing the
@acronym{CCR} @footnote{E. Hartnett of @acronym{NOAA} @acronym{EMC} is 
co-founder and co-maintainer of the @acronym{CCR}. Thanks Ed!}.
By the end of 2021 the @acronym{CCR} supported codecs for Bzip2,
Zstandard, BitGroom, and Granular BitRound. 
The @acronym{CCR} performance helped persuade the netCDF team at
Unidata of the importance and practicality of expanding compression
options in the base netCDF C-library beyond the venerable
@acronym{DEFLATE} algorithm.
Together, we merged the quantization, Bzip2, and Zstandard codecs and
@acronym{API} from the @acronym{CCR} into what became (in June, 2022)
netCDF version 4.9.0
@footnote{D. Heimbigner (Unidata) helped implement all these features
into netCDF. Thanks Dennis!}.
@acronym{NCO} version 5.1.0 (released in July, 2022) unified these
advances by fully supporting its own quantization methods,
@acronym{CCR} codecs, generic (i.e., non-@acronym{CCR}) @acronym{HDF5}
codecs, and the quantization algorithms and modern lossless codecs in
netCDF 4.9.0.  

Access to the quantization and compression options is available
through three complementary, backwards-compatible, and overlapping
@acronym{API}s designed to accomodate the successive generations of
compression features. 
The original generation of compression options remain accessible
through the standard @command{ncpdq} (for Linear Packing) and
@acronym{NCO}-wide @code{-L} option (or its synonyms @code{--deflate}
and @code{--dfl_lvl}) for @acronym{DEFLATE}. 
The second generation of compression options refers to the
@code{--ppc} and @code{--baa} (and related synonyms) options that
control the type and level of quantization algorithm, and the
variables to operate on.
These options call quantization and rounding routines implemented
within @acronym{NCO} itself, rather than in external libraries.
The new @code{--cmp_sng} (and synonyms) option provides an
@acronym{API} for @acronym{NCO} to invoke all lossy and lossless
codecs in external libraries, including the netCDF C-library, the
@acronym{CCR}, and generic @acronym{HDF5} codecs.

The @code{--cmp_sng} (and synonyms @code{--cmp} and
@code{--compression}) options take as argument a string @var{cmp_sng}
which contains a list of quantization and compression algorithms and
their respective parameters.
The @var{cmp_sng} must adhere to a superset of the filter-list
@acronym{API} introduced by the @command{nccopy} command and reported
in the netCDF @code{_Filter} attribute.
This @acronym{API} uses the @acronym{UNIX} pipe symbole @code{|} to
separate the codecs applied as @acronym{HDF5} filters to a variable:
@example
@verbatim
% ncks --hdn -m in.nc | grep _Filter
      u:_Filter = "307,2|32015,3" ;
      U:_Filter = "32001,2,2,4,4185932,5,1,1" ;
% ncdump -h -s in.nc | grep _Filter
		u:_Filter = "307,2|32015,3" ;
                U:_Filter = "32001,2,2,4,4185932,5,1,1" ;
@end verbatim
@end example
The above example shows variables compressed with two successive
codecs.
The variable @code{u} was compressed with codecs with @acronym{HDF5}
filter @acronym{ID}s 307 and 32015, respectively.  
@acronym{NCO} translates these mysterious @acronym{HDF5} numeric
filter @acronym{ID}s into filter names in the @acronym{CDL} comments
when invoked with a higher debugging level: 
@example
@verbatim
% ncks -D 2 --hdn -m in.nc | grep _Filter
      u:_Filter = "307,2|32015,3" ; // char codec(s): Bzip2, Zstandard
      U:_Filter = "32001,2,2,4,4185932,5,1,1" ; // char codec(s): \
                                              Blosc Shuffle, Blosc LZ4
@end verbatim
@end example

@html
<a name="dfl"></a> <!-- http://nco.sf.net/nco.html#dfl -->
<a name="zst"></a> <!-- http://nco.sf.net/nco.html#zst -->
<a name="bz2"></a> <!-- http://nco.sf.net/nco.html#bz2 -->
<a name="shf"></a> <!-- http://nco.sf.net/nco.html#shf -->
<a name="gbr"></a> <!-- http://nco.sf.net/nco.html#gbr -->
<a name="btg"></a> <!-- http://nco.sf.net/nco.html#btg -->
<a name="bgr"></a> <!-- http://nco.sf.net/nco.html#bgr -->
<a name="btr"></a> <!-- http://nco.sf.net/nco.html#btr -->
<a name="f32"></a> <!-- http://nco.sf.net/nco.html#f32 -->
<a name="dns"></a> <!-- http://nco.sf.net/nco.html#dns -->
@end html
@cindex @code{dfl}
@cindex @code{zst}
@cindex @code{bz2}
@cindex @code{shf}
@cindex @code{gbr}
@cindex @code{btg}
@cindex @code{bgr}
@cindex @code{f32}
@cindex @code{dns}
@cindex Shuffle
@cindex Fletcher32
You may provide any operator (besides @command{ncrename} and
@command{ncatted}) with a @var{cmp_sng} comprised of an arbitrary
number of @acronym{HDF5} filters specified either by numeric
@acronym{ID} or by name, including @acronym{NCO}-supported synonyms:
@example
@verbatim
% ncks --cmp="307,2|32015,3" in.nc out.nc # Filter numeric IDs
% ncks --cmp="Bzip2,2|Zstandard,3" in.nc out.nc # Filter names
% ncks --cmp="bzp,2|zst,3" in.nc out.nc # Filter abbreviations
@end verbatim
@end example
@acronym{NCO} also uses this @acronym{API} to invoke the netCDF
quantization algorithms such as Granular BitGroom and BitRound. 
netCDF records the operation of quantization algorithms in a
@code{_Quantize} attribute.
@example
@verbatim
% ncks --cmp="gbr,2|zst,3" in.nc out.nc 
% ncks -D 2 --hdn -m out.nc | grep 'Filt|Quant'
      u:_QuantizeGranularBitRoundNumberOfSignificantDigits = 2 ;
      u:_Filter = "32015,3" ; // char Codec(s): Zstandard
@end verbatim
@end example
@acronym{NCO} calls the filters in the order specified.
Thus, it is important to follow the above example and to specify
compression pre-filters like quantization and Shuffle prior to any
lossless codecs.  
However, the netCDF library imposes rules that can override the
user-specified order for the Shuffle and Fletcher32 filters as
described below (these are always helpful in real-world situations).

The @samp{--cmp} option specifies a global compression configuration.
This is fine for lossless codecs, since there is little evidence to
motivate per-variable lossless compression levels for real-world data.
By contrast, it is often desirable to configure quantization levels on
a per-variable basis.
This is because the real information content of geophysical variables
can differ strongly due to a multitude of factors including the field
meaning, spatial location, and dimensional units.
@acronym{NCO} applies any quantization filter specified in
@var{cmp_sng} uniformly to all variables encountered (the @samp{--ppc}
quantization option/@acronym{API} is still available for per-variable
quantization parameters, as discussed below). 
The one exception is that @acronym{NCO} prohibits quantization of
``coordinate-like variables''.
Variables that are tradiational (1-dimensional) coordinates, or that
are mentioned in the values of @acronym{CF} @code{bounds},
@code{climatology}, @code{coordinates}, or @code{grid_mapping}
attributes, all count as coordinate-like variables.
Such variables include quantities like gridcell areas and boundaries.
@acronym{NCO} eschews quantizing such variables to avoid unforeseen
or unanticipated degradation of numerical accuracy due to propagation
of quantization errors in post-processing.

Specifying the compression string with codec names should make
lossy and lossless compression easier for users to understand and
employ. 
There are, however, a few wrinkles and legacy conventions that
might surprise users when first encountered.
For instance, the codecs for @acronym{DEFLATE}, Shuffle, and
Fletcher32 have always been built-in to netCDF.
Users can instruct @acronym{NCO} to apply these filters with the
new @acronym{API}, yet their application to a dataset will still
be reported using the ``old'' per-filter attributes:
@example
@verbatim
% ncks --cmp="fletcher32|shuffle|deflate" in.nc out.nc
% ncks --hdn -m out.nc
...
      u:_DeflateLevel = 1 ;
      u:_Shuffle = "true" ;
      u:_Fletcher32 = "true" ;
...
@end verbatim
@end example
As this example shows, it is not required to give filter parameter
arguments to all filters.
When the user omits filter parameters (e.g., compression level,
@acronym{NSD}, @acronym{NSB}, or other filter configurator) 
for select filters that require such a parameter, @acronym{NCO}
automatically inserts an appropriate default filter parameter.
@acronym{NCO} assumes default parameters 1, 4, 1, and 3, for the
lossless filters @acronym{DEFLATE}, Shuffle, Bzip2, and Zstandard,
respectively.   
@acronym{NCO} assumes default parameters 3, 3, and 9, for the lossy
quantization algorithms BitGroom, Granular BitGroom, and BitRound,
respectively.   
Note that the netCDF filter for the Fletcher32 checksum algorithm
does not accept a parameter argument, and @acronym{NCO} ignores
any parameters provided to this filter.
@example
@verbatim
% ncks --cmp="fletcher32|shuffle|granularbr|deflate|zstandard" ...
% ncks --cmp="f32|shf|gbr|dfl|zst" ... # Shorthand, default parameters
% ncks --cmp="f32|shf,4|gbr,3|dfl,1|zst,3" ... # Explicit parameters
@end verbatim
@end example

The @code{cmd_sng} option supports an arbitrary number of filters.
An example that compressess then reads a file with most
netCDF-supported algorithms shows the mixture of filter-specific
attributes (@code{_Shuffle}, @code{_DeflateLevel}, @code{_Fletcher32})
and generic filter attributes (@code{_Filter}) that result:
@example
@verbatim
% ncks --cmp='f32|shf|gbr|dfl|bz2|zst' in.nc out.nc
% ncks --hdn -m out.nc
    float u(time) ;
      u:long_name = "Zonal wind speed" ;
      u:units = "meter second-1" ;
      u:_QuantizeGranularBitRoundNumberOfSignificantDigits = 3 ;
      u:_Storage = "chunked" ;
      u:_ChunkSizes = 1024 ;
      u:_Filter = "307,1|32015,3" ;
      u:_DeflateLevel = 1 ;
      u:_Shuffle = "true" ;
      u:_Fletcher32 = "true" ;
      u:_Endianness = "little" ;
@end verbatim
@end example
Note that the @code{_Filter} value is a valid @var{cmp_sng} for
use as an argument to the @code{--cmp_sng} option.
This enables users to easily duplicate the compression settings
of one dataset in another dataset.
One can also find the global @var{cmp_sng} that @acronym{NCO} used to 
compress a dataset in the global @code{history} attribute.

In the absence of instructions to the contrary, @acronym{NCO}
preserves the compression settings of datasets that it copies or
subsets. 
It simply defaults to copying the per-variable compression settings 
from the input file.
If the copy or subset command includes global compression instructions
(i.e., the @samp{--cmp} or @samp{-L} options), those instructions will
override the per-variable settings in the input file.
The user can eliminate all compression filters by setting
@var{cmp_sng} to the special value @code{none} (or to its synonyms
@code{uncompress}, @code{decompress}, @code{defilter}, or
@code{unset}).  
@example
@verbatim
% ncks --cmp='f32|shf|gbr|dfl|bz2|zst' in.nc out.nc
% ncks --cmp='none' out.nc none.nc
% ncks --hdn -m none.nc
    float u(time) ;
      u:long_name = "Zonal wind speed" ;
      u:units = "meter second-1" ;
      u:_QuantizeGranularBitRoundNumberOfSignificantDigits = 3 ;
      u:_Storage = "chunked" ;
      u:_ChunkSizes = 1024 ;
      u:_Endianness = "little" ;
@end verbatim
@end example
The uncompressed copy has no filter attributes remaining because all
filters have been removed.
The @code{_Quantize} attribute remains because the quantization was
applied as an internal algorithm not as an @acronym{HDF5} filter.
In contrast to lossless compression filters, lossy quantization
algorithms can never be ``removed'' much less undone because, by
definition, they are lossy.
Removing compression is ``all or nothing'' in that there is currently
no way to remove only one or a few codecs and leave the rest in place.
To do that, the user must instead recompress the dataset using a
@var{cmp_sng} that includes only the desired codecs.
@example
@verbatim
% ncks --cmp='shf|zst' ... # Compress
% ncks --cmp='none' ...    # Decompress (remove Shuffle and Zstd)
% ncks --cmp=zst,4 ...     # Recompress to new Zstd level, no Shuffle
@end verbatim
@end example

Shuffle is an important filter than can boost lossless compression
ratios of geoscientific data by 10--20% (@pxref{fgr:qnt_cr_dfl}).
@float Figure,fgr:qnt_cr_dfl
@image{fgr/qnt_cr_dfl,5.5in} 
@caption{Quantization and then compression by @acronym{DEFLATE},
including the contribution of Shuffle.}
@end float
Both the netCDF library and @acronym{NCO} continue to treat the
Shuffle filter specially. 
If the Shuffle and @acronym{DEFLATE} algorithms are both invoked
through the standard netCDF @acronym{API} (i.e.,
@code{nc_def_var_deflate()}), then the netCDF library ensures that the
Shuffle filter is called before @acronym{DEFLATE}, indeed before any
filter except @code{Fletcher32} (which performs a checksum, not
compression).  
This behavior is welcome as it avoids inadvertent mis-use of Shuffle.
Prior to version 5.1.0, @acronym{NCO} always invoked Shuffle with
@acronym{DEFLATE}, and did not expose the Shuffle filter to user
control.
@acronym{NCO} now exposes Shuffle to user control for all filters.
To preserve backward compatibility, invoking the @acronym{DEFLATE}
algorithm with the @code{dfl} or @code{deflate} names (or with the
numeric @acronym{HDF5} filter @acronym{ID} of @code{1}) still sets
the Shuffle filter to true.
To invoke @acronym{DEFLATE} without Shuffle, use the special filter
names @code{dns} or @code{DEFLATE No Shuffe}.
Specifying the Shuffle filter along with any Blosc compressor causes
@acronym{NCO} to invoke the Blosc version of Shuffle instead of the
@acronym{HDF5} version of Shuffle.
The Blosc Shuffle should execute more rapidly because it takes
advantage of @acronym{AVX2} instructions, etc.
In summary, users must explicitly request Shuffle for all
non-@acronym{DEFLATE} codecs, otherwise Shuffle will not be
employed prior to those codecs.
@example
@verbatim
% ncks --cmp='dfl' ...     # Invoke Shuffle then DEFLATE
% ncks --cmp='shf|dfl' ... # Same (default Shuffle stride 4-bytes)
% ncks --cmp='dns' ...     # Invoke DEFLATE (no Shuffle)
% ncks --cmp='shf|dns' ... # Invoke Shuffle then DEFLATE
% ncks --cmp='zst' ...     # Invoke Zstandard (no Shuffle)
% ncks --cmp='shf|zst' ... # Invoke Shuffle then Zstandard
% ncks --cmp='zst|shf' ... # Same (netCDF enforces Shuffle-first rule)
% ncks --cmp='shf' ...     # Invoke only Shuffle
% ncks --cmp='shf,8|zst' ... # Shuffle stride 8-bytes then Zstandard
% ncks --cmp='shf,8|dfl' ... # Shuffle stride remains default 4-bytes
% ncks --cmp='bls' ...     # Invoke Blosc (LZ by default) without Shuffle
% ncks --cmp='shf|bls' ... # Invoke Blosc (not HDF5) Shuffle, then Blosc LZ
@end verbatim
@end example
The last example above shows how to invoke Shuffle with non-default
byte stride
@footnote{Full disclosure:
Documentation of the meaning of the Shuffle parameter is scarce.
I think though am not certain that the Shuffle parameter refers to the
number of contiguous byte-groups that the algorithm rearranges a chunk
of data into. 
I call this the stride.
Thus the default stride of 4 means that Shuffle rearranges a chunk of
4-byte integers into four consecutive sequences, the first comprises
all the leading bytes, the second comprises all the second bytes, etc.
A well-behaved stride should evenly divide the number of bytes in a
data chunk.}.
The netCDF library, and thus @acronym{NCO}, always uses the default
Shuffle stride (4 bytes) with the @acronym{DEFLATE} filter.
Specifying a different stride only has an effect with
non-@acronym{DEFLATE} filters.  
This ensures @acronym{NCO}'s default behavior with @acronym{DEFLATE}
does not change.
As explained above, invoke @acronym{DEFLATE} with @samp{--cmp=dns}
instead of @samp{--cmp=dfl} if you wish to suppress the otherwise
automatic invocation of Shuffle.

Note that @acronym{NCO} and netCDF implement their quantization
algorithms internally, whereas the @acronym{CCR} implements them as
external shared-library codecs (valid only with netCDF4 files).
Since these quantization algorithms leave data in @acronym{IEEE}
format no codec/filter is required to read (or write) them.
Quantization therefore works with netCDF3 datasets, not just netCDF4.
netCDF applications that attempt to read data compressed with
shared-library filters must be linked to the same shared filters 
or the ``decompression'' step will fail.
Datasets produced with netCDF or @acronym{CCR}-supported codecs
(Bzip2, @acronym{DEFLATE}, Zstandard) will be readable by all users
who upgrade to netCDF 4.9.0 or later or who install the
@acronym{CCR}.
There is no difference between a losslessly compressed dataset
produced with a @acronym{CCR}-supplied vs.@: a netCDF-supplied
filter.
However, reading a dataset quantized by a @acronym{CCR} filter (e.g.,
BitGroom or GranularBG) requires access to the @acronym{CCR} filter,
which forces users to take an extra step to install the
@acronym{CCR}. 
This is an unfortunate artifact of implementing quantization as a
codec (which the @acronym{CCR} must do) vs.@: an internal numerical
function (which netCDF and @acronym{NCO} do).
Where possible, people should encode datasets with netCDF-supported 
algorithms and codecs in preference to @acronym{CCR} or raw
@acronym{HDF5} codecs. 
Doing so will increase dataset portability.

@unnumberedsubsec Limitations of Current Compression @acronym{API}
The @acronym{NCO} filter @acronym{API} will evolve in a (hopefully)
backward-compatible manner as experience and feedback are gained.
All filter parameters are eventually passed to @acronym{HDF5} as
unsigned integers. 
By contrast, the current @acronym{API} treats all input arguments in
@var{cmp_sng} signed integers for ease-of-use.
For example, it is easier to specify a Zstandard filter level of
negative one as @samp{zstd,-1} than as @samp{zstd,4294967295}. 
@acronym{NCO} currently has no easy way to specify
the floating-point configuration parameters required by some
@acronym{CCR} filters and many external filters.
That said, most of the code to support the filter parameter syntax
documented at
@url{https://docs.unidata.ucar.edu/netcdf-c/current/filters.html}
and implemented in @command{ncgen} and @command{nccopy} is ready and
we expect to support that syntax in @acronym{NCO} 5.0.9.
That syntax is backward compatible with the integer-only input
assumptions already embedded in @acronym{NCO} 5.1.0.
In the meantime, we request your feedback, use-cases, and suggestions
before adopting a final approach.

Another limitation of @acronym{NCO} 5.1.0 was that the Blosc filters
would fail without explanation.
This is why the manual does not yet document much about Blosc filters.
The Blosc issues were fixed upstream in netCDF version 4.9.1.

netCDF 4.9.0 contained some other inadvertent mistakes that were fixed 
in 4.9.1.
First, the quantization algorithms internal to netCDF work only on
datasets of type @acronym{NETCDF4} in netCDF 4.9.0.
A recently discovered bug prevents them from working on
@acronym{NETCDF4_CLASSIC}-format datasets 
@footnote{Quantization may never be implemented in netCDF for any 
@acronym{CLASSIC} or other netCDF3 formats since there is no
compression advantage to doing so.
Use the @acronym{NCO} implementation to quantize to netCDF3 output
formats.}.
The fix to the @acronym{NETCDF4_CLASSIC} bug was officially released
in netCDF 4.9.1.
Note that this bug did not affect the same quantization algorithms as
implemented in @acronym{NCO} (or in the @acronym{CCR}, for that
matter). 
In other words, quantization to netCDF3 and
@acronym{NETCDF4_CLASSIC}-format output files @emph{always} works
when invoked through the @samp{--ppc} (not @samp{--cmp}) option. 
This restriction will only affects netCDF prior to 4.9.1. 
Per-variable quantization settings must also be invoked through
@samp{--ppc} (not @samp{--cmp}) for all output formats, until and
unless this feature is migrated to @samp{--cmp} (there are no plans
to do so).

Another sticky wicket expected that was fixed in netCDF 4.9.1
was the use of the Blosc codec.
NCZarr uses Blosc internally, however the @acronym{HDF5} Blosc codec
in netCDF 4.9.0 was not robust.
We plan to advertise the advantages of Blosc more fully in future
versions of @acronym{NCO}.
Feedback on any or all of these constraints is welcome.

@unnumberedsubsec Best Practices for Real World Lossy Compression
The workflow to compress large-scale, user-facing datasets requires 
consideration of multiple factors including storage space, speed,
accuracy, information content, portability, and user-friendliness.
We have found that this blend is best obtained by using per-variable
quantization together with global lossless compression.
@acronym{NCO} can configure per-variable quantization levels together
with global lossless filters when the quantization algorithm is
specified with the @samp{--ppc} option/@acronym{API} and the lossless
filters are specified with the @samp{--cmp_sng} option/@acronym{API}.
Granular BitRound (GranularBR) and BitRound are state-of-the-art
quantization algorithms that are configured with the number of
significant decimal digits (@acronym{NSD}) or number of significant
bits (@acronym{NSB}), respectively.

One can devise an optimal approach in about four steps: 
First, select a global lossless codec that produces a reasonable
tradeoff between compression ratio (@acronym{CR}) and speed.
The speed of the final workflow will depend mostly on the lossless
codec employed and its compression settings.
Try a few standard codecs with Shuffle to explore this tradeoff:
@example
@verbatim
ncks -7 --cmp='shf|zst' ...
ncks -7 --cmp='shf|dfl' ...
ncks -7 --cmp='shf|bz2' ...
@end verbatim
@end example
The @samp{-7} switch creates output in @acronym{NETCDF4_CLASSIC}
format.
This highly portable format supports codecs and is also mandated by
many archives such as @acronym{CMIP6}.
The only other viable format choice is @samp{-4} for @acronym{NETCDF4}. 
That format must be used if any variables make use of the extended
netCDF4 atomic types.
Our experience with @acronym{ESM} data shows that Bzip2 often yields
the best @acronym{CR} (@pxref{fgr:qnt_cr_bz2}).
However Bzip2 is much slower than Zstandard which yields a comparable
@acronym{CR}.
@float Figure,fgr:qnt_cr_bz2
@image{fgr/qnt_cr_bz2,5.5in} 
@caption{Quantization and then compression by Bzip2,
including the contribution of Shuffle.}
@end float

The second step is to choose the quantization algorithm and
its default level.
Quantization can significantly improve the @acronym{CR} without
sacrificing any scientifically meaningful data.
Lossless algorithms are unlikely to significantly alter the workflow
throughput unless applied so agressively that they considerably
reduce the entropy seen by the lossless codec.
The goal in this step is to choose the strongest quantization that
preserves all the meaningful precision of @emph{most} fields, and
that dials-in the @acronym{CR} to the desired value.
@example
@verbatim
ncks -7 --ppc default=3 ...          # GranularBR, NSD=3
ncks -7 --baa=4 --ppc default=3 ...  # Same
ncks -7 --baa=4 --ppc dfl=3 ...      # Same
ncks -7 --baa=8 --ppc default=9  ... # BitRound, NSB=9
@end verbatim
@end example
As an argument to @samp{--ppc}, the keyword @code{dfl} is just a
synonym for @code{default} and has nothing to do with
@acronym{DEFLATE}.  

The third step is to develop per-variable exceptions to the
default quantization of the previous step.
This can be a process of trial-and-error, or semi-automated through
techniques such as determining an acceptable information content
threshold for each variable
@footnote{See, e.g., the procedure described in ``Compressing
atmospheric data into its real information content'' by M.~Klower et
al., available at @url{https://doi.org/10.1038/s43588-021-00156-2}.}.
The per-variable arguments to @samp{--ppc} can take many forms:
@example
@verbatim
ncks --ppc p,w,z=5 --ppc q,RH=4 --ppc T,u,v=3 # Multiple options
ncks --ppc p,w,z=5#q,RH=4#T,u,v=3 ...  # Combined option
ncks --ppc Q.?=5#FS.?,FL.?=4#RH=.3 ... # Regular expressions
ncks --baa=8 --ppc p,w,z=15#q,RH=12#T,u,v=9 ... # BitRound (NSB)
ncks --baa=8 --ppc CO2=15#AER.?=12#U,V=6 ... # Klower et al. 
@end verbatim
@end example
No compression is necessary in this step, which presumably involves 
evaluating the adequacy of the quantized values at matching
observations or at meeting other error metrics.

The fourth and final step combines the lossless and lossy algorithms
to produce the final workflow:
@example
@verbatim
ncks -7 --ppc dfl=3 --cmp='shf|zst' ... # A useful starting point?
ncks -7 --ppc default=3#Q.?=5#FS.?,FL.?=4 --cmp='shf|zst' ...
ncks -7 --baa=4 --ppc default=3#Q.?=5#FS.?,FL.?=4 --cmp='shf|zst' ...
ncks -7 --baa=8 --ppc default=9#Q.?=15#FS.?,FL.?=12 --cmp='shf|zst' ...
@end verbatim
@end example
The example above uses Zstandard (@pxref{fgr:qnt_cr_zst}) because it
is significant faster than other codecs with comparable @acronym{CR}s,
e.g., Bzip2. 
@float Figure,fgr:qnt_cr_zst
@image{fgr/qnt_cr_zst,5.5in} 
@caption{Quantization and then compression by Zstandard,
including the contribution of Shuffle.}
@end float

@unnumberedsubsec Older Compression @acronym{API}
@acronym{NCO} implements or accesses four different compression
algorithms, the standard lossless @acronym{DEFLATE} algorithm
and three lossy compression algorithms.
All four algorithms reduce the on-disk size of a dataset while
sacrificing no (lossless) or a tolerable amount (lossy) of precision. 
First, @acronym{NCO} can access the lossless @acronym{DEFLATE} algorithm,
a combination of Lempel-Ziv encoding and Huffman coding, algorithm on
any netCDF4 dataset (@pxref{Deflation}). 
Because it is lossless, this algorithm re-inflates deflated data to
their full original precision. 
This algorithm is accessed via the @acronym{HDF5} library layer
(which itself calls the @command{zlib} library also used by
@command{gzip}), and is unavailable with netCDF3.   

@menu
* Linear Packing::
* Precision-Preserving Compression::
@end menu

@node Linear Packing, Precision-Preserving Compression, Compression, Compression
@subsection Linear Packing
The three lossy compression algorithms are Linear Packing 
(@pxref{Packed data}), and two precision-preserving algorithms.
Linear packing quantizes data of a higher precision type into a lower
precision type (often @code{NC_SHORT}) that thus stores a fewer (though
constant) number of bytes per value. 
Linearly packed data unpacks into a (much) smaller dynamic range than the
floating-point data can represent.  
The type-conversion and reduced dynamic range of the data allows packing
to eliminate bits typically used to store an exponent, thus improving 
its packing efficiency.
Packed data also can also be deflated for additional space savings. 

A limitation of linear packing is that unpacking data stored as integers 
into the linear range defined by @code{scale_factor} and
@code{add_offset} rapidly loses precision outside of a narrow range of
floating-point values.  
Variables packed as @code{NC_SHORT}, for example, can represent only
about 64000 discrete values in the range 
@math{-32768*scale_factor+add_offset} to
@math{32767*scale_factor+add_offset}. 
The precision of packed data equals the value of @code{scale_factor},
and @code{scale_factor} is usually chosen to span the range of valid
data, not to represent the intrinsic precision of the variable.
In other words, the precision of packed data cannot be specified in
advance because it depends on the range of values to quantize.

@node Precision-Preserving Compression,  , Linear Packing, Compression
@subsection Precision-Preserving Compression
@cindex @acronym{PPC}
@cindex @acronym{LSD}
@cindex Least Significant Digit
@cindex quantization
@acronym{NCO} implemented the final two lossy compression algorithms
in version 4.4.8 (February, 2015).
These are both @dfn{Precision-Preserving Compression} (@acronym{PPC})
algorithms and since standard terminology for precision is remarkably 
imprecise, so is our nomenclature. 
The operational definition of ``significant digit'' in our precision
preserving algorithms is that the exact value, before rounding or
quantization, is within one-half the value of the decimal place occupied
by the @dfn{Least Significant Digit} (@acronym{LSD}) of the rounded value.
For example, the value @math{pi = 3.14} correctly represents the exact
mathematical constant @var{pi} to three significant digits because the
@acronym{LSD} of the rounded value @w{(i.e., 4)} is in the one-hundredths 
digit place, and the difference between the exact value and the rounded
value is less than one-half of one one-hundredth, i.e., 
(@math{3.14159265358979323844 - 3.14 = 0.00159 < 0.005}).

@cindex Number of Significant Digits
@cindex @acronym{NSD}
@cindex Bit-Grooming
One @acronym{PPC} algorithm preserves the specified total 
@dfn{Number of Signifcant Digits} (@acronym{NSD}) of the value.
For example there is only one significant digit in the weight of
most ``eight-hundred pound gorillas'' that you will encounter, 
i.e., so @math{@var{nsd=1}}.
This is the most straightforward measure of precision, and thus
@acronym{NSD} is the default @acronym{PPC} algorithm.

@cindex Decimal Significant Digits
@cindex @acronym{DSD}
The other @acronym{PPC} algorithm preserves the number of 
@dfn{Decimal Significant Digits} (@acronym{DSD}), i.e., the number of
significant digits following (positive, by convention) or preceding
(negative) the decimal point.  
For example, @samp{0.008} and @samp{800} have, respectively, three and 
negative two digits digits following the decimal point, corresponding 
to @math{@var{dsd=3}} and @math{@var{dsd=-2}}. 

The only justifiable @acronym{NSD} for a given value depends on 
intrinsic accuracy and error characteristics of the model or
measurements, and not on the units with which the value is stored.
The appropriate @acronym{DSD} for a given value depends on these
intrinsic characteristics and, in addition, the units of storage.
This is the fundamental difference between the @acronym{NSD} and 
@acronym{DSD} approaches. 
The eight-hundred pound gorilla always has @math{@var{nsd=1}} regardless 
of whether the value is stored in pounds or in some other unit.
@acronym{DSD} corresponding to this weight is @math{@var{dsd=-2}} if the
value is stored in pounds, @math{@var{dsd=4}} if stored in megapounds.

Users may wish to express the precision to be preserved as either
@acronym{NSD} or @acronym{DSD}.
Invoke @acronym{PPC} with the long option @samp{--ppc var=prc}, or give
the same arguments to the synonyms
@samp{--precision_preserving_compression}, or to @samp{--quantize}. 
Here @var{var} is the variable to quantize, and @var{prc} is its
precision.
@cindex indicator option
@cindex multi-arguments
The option @samp{--ppc} (and its long option equivalents such
as @samp{--quantize}) indicates the argument syntax will be
@var{key}=@var{val}.
As such, @samp{--ppc} and its synonyms are indicator options that accept
arguments supplied one-by-one like 
@samp{--ppc @var{key1}=@var{val1} --ppc @var{key2}=@var{val2}}, or
aggregated together in multi-argument format like
@samp{--ppc @var{key1}=@var{val1}#@var{key2}=@var{val2}}
(@pxref{Multi-arguments}).
The default algorithm assumes @var{prc} specifies @acronym{NSD}
precision, e.g., @samp{T=2} means @math{@var{nsd=2}}.
Prepend @var{prc} with a decimal point to specify @acronym{DSD}
precision, e.g., @samp{T=.2} means @math{@var{dsd=2}}.
@acronym{NSD} precision must be specified as a positive integer.
@acronym{DSD} precision may be a positive or negative integer;
and is specified as the negative @w{base 10} logarithm of the desired 
precision, in accord with common usage. 
For example, specifying @samp{T=.3} or @samp{T=.-2} tells the
@acronym{DSD} algorithm to store only enough bits to preserve the  
value of @var{T} rounded to the nearest thousandth or hundred,
respectively. 

@cindex @acronym{CF} conventions
@cindex @code{coordinates} attribute
@cindex @code{climatology} attribute
@cindex @code{bounds} attribute
Setting @var{var} to @code{default} has the special meaning of applying 
the associated @acronym{NSD} or @acronym{DSD} algorithm to all floating
point variables except coordinate variables.
Variables @emph{not affected} by @code{default} include integer and
non-numeric atomic types, coordinates, and variables mentioned in 
the @code{bounds}, @code{climatology}, or @code{coordinates} attribute
of any variable. 
@acronym{NCO} applies @acronym{PPC} to coordinate variables only if 
those variables are explicitly specified (i.e., not with the
@samp{default=@var{prc}} mechanism. 
@acronym{NCO} applies @acronym{PPC} to integer-type variables only if 
those variables are explicitly specified (i.e., not with the
@samp{default=@var{prc}}, and only if the @acronym{DSD} algorithm is
invoked with a negative @var{prc}.
To prevent @acronym{PPC} from applying to certain non-coordinate
variables (e.g., @code{gridcell_area} or @code{gaussian_weight}),
explicitly specify a precision @w{exceeding 7} (for @code{NC_FLOAT})
@w{or 15} (for @code{NC_DOUBLE}) for those variables. 
Since these are the maximum representable precisions in decimal digits,
@acronym{NCO} @emph{turns-off} @acronym{PPC} (i.e., does nothing)
when more precision is requested.

@cindex bitmask
@cindex significand
@cindex @w{@acronym{IEEE} 754}
The time-penalty for compressing and uncompressing data varies according 
to the algorithm.
The Number of Significant Digit (@acronym{NSD}) algorithm quantizes by
bitmasking, and employs no floating-point math.
The Decimal Significant Digit (@acronym{DSD}) algorithm quantizes by
rounding, which does require floating-point math.
Hence @acronym{NSD} is likely faster than @acronym{DSD}, though
the difference has not been measured.
@acronym{NSD} creates a bitmask to alter the @dfn{significand} of
@w{@acronym{IEEE} 754} floating-point data.
The bitmask is one for all bits to be retained and zero or one for all
bits to be ignored.
The algorithm assumes that the number of binary digits (i.e., bits)
necessary to represent a single base-10 digit is 
@math{ln(10)/ln(2) = 3.32}.
The exact numbers of bits @var{Nbit} retained for single and double
precision values are @math{ceil(3.32*@var{nsd})+1} and
@math{ceil(3.32*@var{nsd})+2}, respectively.
Once these @w{reach 23} @w{and 53}, respectively, bitmasking is
completely ineffective. 
This occurs at @math{@var{nsd}=6.3} @w{and 15.4}, respectively.

The @acronym{DSD} algorithm, by contrast, uses rounding to remove
undesired precision.  
@cindex @command{rint()}
@cindex C99
@cindex @acronym{IEEE}
@cindex significand
The rounding 
@footnote{
Rounding is performed by the internal math library @command{rint()} 
family of functions that were standardized in C99.
The exact alorithm employed is
@math{@var{val} := rint(@var{scale}*@var{val})/@var{scale}} where
@var{scale} is the nearest power @w{of 2} that exceeds 
@math{10**@var{prc}}, and the inverse of @var{scale} is used when 
@math{@var{prc} < 0}.
For @math{@var{ppc} = 3} or @math{@var{ppc} = -2}, for example, we have
@math{@var{scale} = 1024} and @math{@var{scale} = 1/128}.}
zeroes the greatest number of significand bits consistent with
the desired precision.

To demonstrate the change in @acronym{IEEE} representation caused by
@acronym{PPC} rounding algorithms, consider again the case of @var{pi},
represented as an @code{NC_FLOAT}.
The @w{@acronym{IEEE} 754} single precision representations of the exact
value (3.141592...), the value with only three significant digits treated
as exact (3.140000...), and the value as stored (3.140625) after
@acronym{PPC}-rounding with either the @acronym{NSD} (@math{@var{prc}=3})
or @acronym{DSD} (@math{@var{prc}=2}) algorithm are, respectively,
@ignore
ncks -O -v pi --ppc pi=1  ~/nco/data/in.nc ~/foo_nsd1.nc 
ncks -O -v pi --ppc pi=2  ~/nco/data/in.nc ~/foo_nsd2.nc 
ncks -O -v pi --ppc pi=3  ~/nco/data/in.nc ~/foo_nsd3.nc 
ncks -O -v pi --ppc pi=4  ~/nco/data/in.nc ~/foo_nsd4.nc 
ncks -O -v pi --ppc pi=5  ~/nco/data/in.nc ~/foo_nsd5.nc 
ncks -O -v pi --ppc pi=6  ~/nco/data/in.nc ~/foo_nsd6.nc 
ncks -O -v pi --ppc pi=7  ~/nco/data/in.nc ~/foo_nsd7.nc 
ncks -O -v pi --ppc pi=8  ~/nco/data/in.nc ~/foo_nsd8.nc 
ncks -O -v pi --ppc pi=9  ~/nco/data/in.nc ~/foo_nsd9.nc 
ncks -O -v pi --ppc pi=.2 ~/nco/data/in.nc ~/foo_dsd2.nc 
ncks -s %20.16e -C -H ~/foo_nsd.nc
ncks -s %20.16e -C -H ~/foo_dsd.nc
ccc --tst=bnr --flt_foo=3.14159265358979323844 2> /dev/null | grep "Binary of float"
ccc --tst=bnr --flt_foo=3.14000000000000000000 2> /dev/null | grep "Binary of float"
ccc --tst=bnr --flt_foo=3.00000000000000000000 2> /dev/null | grep "Binary of float"
ccc --tst=bnr --flt_foo=3.12500000000000000000 2> /dev/null | grep "Binary of float"
ccc --tst=bnr --flt_foo=3.14062500000000000000 2> /dev/null | grep "Binary of float"
ccc --tst=bnr --flt_foo=3.14062500000000000000 2> /dev/null | grep "Binary of float"
ccc --tst=bnr --flt_foo=3.14147949218750000000 2> /dev/null | grep "Binary of float"
@end ignore
@example
S Exponent  Fraction (Significand)   Decimal    Notes
0 100000001 0010010000111111011011 # 3.14159265 Exact
0 100000001 0010001111010111000011 # 3.14000000
0 100000001 0010010000000000000000 # 3.14062500 NSD = 3
0 100000001 0010010000000000000000 # 3.14062500 DSD = 2
@end example
The string of trailing zero-bits in the rounded values facilitates
byte-stream compression. 
Note that the @acronym{NSD} and @acronym{DSD} algorithms do not always
produce results that are bit-for-bit identical, although they do in this
particular case.

@html
<a name="ppc_tbl_bit"></a> <!-- http://nco.sf.net/nco.html#ppc_tbl_bit -->
@end html
Reducing the preserved precision of @acronym{NSD}-rounding produces
increasingly long strings of identical-bits amenable to compression: 
@example
S Exponent  Fraction (Significand)   Decimal    Notes
0 100000001 0010010000111111011011 # 3.14159265 Exact
0 100000001 0010010000111111011011 # 3.14159265 NSD = 8
0 100000001 0010010000111111011010 # 3.14159262 NSD = 7
0 100000001 0010010000111111011000 # 3.14159203 NSD = 6
0 100000001 0010010000111111000000 # 3.14158630 NSD = 5
0 100000001 0010010000111100000000 # 3.14154053 NSD = 4
0 100000001 0010010000000000000000 # 3.14062500 NSD = 3
0 100000001 0010010000000000000000 # 3.14062500 NSD = 2
0 100000001 0010000000000000000000 # 3.12500000 NSD = 1
@end example
The consumption of about @w{3 bits} per digit of base-10 precision is
evident, as is the coincidence of a quantized value that greatly
exceeds the mandated precision for @math{@acronym{NSD} = 2}.
Although the @acronym{NSD} algorithm generally masks some bits for all 
@math{@var{nsd} <= 7} (for @code{NC_FLOAT}), compression algorithms like
@acronym{DEFLATE} may need byte-size-or-greater (i.e., at least
eight-bit) bit patterns before their algorithms can take advantage of
of encoding such patterns for compression.
Do not expect significantly enhanced compression from 
@math{@var{nsd} > 5} (for @code{NC_FLOAT}) or @math{@var{nsd} > 14} (for
@code{NC_DOUBLE}). 
Clearly values stored as @code{NC_DOUBLE} (i.e., eight-bytes) are
susceptible to much greater compression than @code{NC_FLOAT} for a given
precision because their significands explicitly contain @w{53 bits}
rather than @w{23 bits}.

Maintaining non-biased statistical properties during lossy compression
requires special attention. 
The @acronym{DSD} algorithm uses @code{rint()}, which rounds toward the
nearest even integer. 
Thus @acronym{DSD} has no systematic bias.
However, the @acronym{NSD} algorithm uses a bitmask technique
susceptible to statistical bias.
Zeroing all non-significant bits is guaranteed to produce numbers 
quantized to the specified tolerance, i.e., half of the decimal value
of the position occupied by the @acronym{LSD}. 
However, always zeroing the non-significant bits results in quantized
numbers that never exceed the exact number.
This would produce a negative bias in statistical quantities (e.g., the
average) subsequently derived from the quantized numbers. 
To avoid this bias, our @acronym{NSD} implementation rounds
non-significant bits down (to zero) or up (to one) in an alternating
fashion when processing array data.
In general, the first element is rounded down, the second up, and so
on. 
This results in a mean bias quite close to zero.
The only exception is that the floating-point value of zero is never 
quantized upwards.
For simplicity, @acronym{NSD} always rounds scalars downwards.

@cindex @acronym{DEFLATE}
@cindex @acronym{HDF5}
@cindex @command{gzip}
@cindex @command{gunzip}
Although @acronym{NSD} or @acronym{DSD} are different algorithms under
the hood, they both replace the (unwanted) least siginificant bits of
the @acronym{IEEE} significand with a string of consecutive zeroes.  
Byte-stream compression techniques, such as the @command{gzip}
@acronym{DEFLATE} algorithm compression available in @acronym{HDF5}, 
always compress zero-strings more efficiently than random digits. 
The net result is netCDF files that utilize compression can be
significantly reduced in size.  
This feature only works when the data are compressed, either internally 
(by netCDF) or externally (by another user-supplied mechanism).
It is most straightfoward to compress data internally using the built-in 
compression and decompression supported by netCDF4.
For convenience, @acronym{NCO} automatically activates file-wide
Lempel-Ziv deflation (@pxref{Deflation}) level one (i.e., @samp{-L 1})
when @acronym{PPC} is invoked on any variable in a netCDF4 output file. 
This makes @acronym{PPC} easier to use effectively, since the user
need not explicitly specify deflation.
Any explicitly specified deflation (including no deflation, @samp{-L 0}) 
will override the @acronym{PPC} deflation default.
If the output file is a netCDF3 format, @acronym{NCO} will emit a
message suggesting internal netCDF4 or external netCDF3 compression. 
netCDF3 files compressed by an external utility such as @command{gzip}
accrue approximately the same benefits (shrinkage) as netCDF4, although
with netCDF3 the user or provider must uncompress (e.g.,
@command{gunzip}) the file before accessing the data. 
There is no benefit to rounding numbers and storing them in netCDF3
files unless such custom compression/decompression is employed.
Without that, one may as well maintain the undesired precision. 

The user accesses @acronym{PPC} through a single switch, @samp{--ppc},
repeated as many times as necessary.
To apply the @acronym{NSD} algorithm to variable @var{u} use, e.g., 
@example
ncks -7 --ppc u=2 in.nc out.nc
@end example
The output file will preserve only two significant digits of @var{u}.
The options @samp{-4} or @samp{-7} ensure a netCDF4-format output 
(regardless of the input file format) to support internal compression.
It is recommended though not required to write netCDF4 files after 
@acronym{PPC}.
For clarity the @samp{-4/-7} switches are omitted in subsequent
examples. 
@html
<a name="QuantizeBitGroomNumberOfSignificantDigits"></a> <!-- http://nco.sf.net/nco.html#QuantizeBitGroomNumberOfSignificantDigits -->
<a name="QuantizeGranularBitRoundNumberOfSignificantDigits"></a> <!-- http://nco.sf.net/nco.html#QuantizeGranularBitRoundNumberOfSignificantDigits -->
<a name="QuantizeBitShaveNumberOfSignificantDigits"></a> <!-- http://nco.sf.net/nco.html#QuantizeBitShaveNumberOfSignificantDigits -->
<a name="QuantizeBitSetNumberOfSignificantDigits"></a> <!-- http://nco.sf.net/nco.html#QuantizeBitSetNumberOfSignificantDigits -->
<a name="QuantizeDigitRoundNumberOfSignificantDigits"></a> <!-- http://nco.sf.net/nco.html#QuantizeDigitRoundNumberOfSignificantDigits -->
<a name="QuantizeBitGroomRoundNumberOfSignificantDigits"></a> <!-- http://nco.sf.net/nco.html#QuantizeBitGroomRoundNumberOfSignificantDigits -->
<a name="QuantizeHalfShaveNumberOfSignificantBits"></a> <!-- http://nco.sf.net/nco.html#QuantizeHalfShaveNumberOfSignificantBits -->
<a name="QuantizeBruteForceNumberOfSignificantDigits"></a> <!-- http://nco.sf.net/nco.html#QuantizeBruteForceNumberOfSignificantDigits -->
<a name="QuantizeBitRoundNumberOfSignificantBits"></a> <!-- http://nco.sf.net/nco.html#QuantizeBitRoundNumberOfSignificantBits -->
<a name="number_of_significant_digits"></a> <!-- http://nco.sf.net/nco.html#number_of_significant_digits -->
<a name="least_significant_digit"></a> <!-- http://nco.sf.net/nco.html#least_significant_digit -->
@end html
@cindex @code{QuantizeBitGroomNumberOfSignificantDigits}
@cindex @code{QuantizeBitSetNumberOfSignificantDigits}
@cindex @code{QuantizeBitShaveNumberOfSignificantDigits}
@cindex @code{QuantizeDigitRoundNumberOfSignificantDigits}
@cindex @code{QuantizeGranularBitRoundNumberOfSignificantDigits}
@cindex @code{QuantizeBitGroomRoundNumberOfSignificantDigits}
@cindex @code{QuantizeHalfShaveNumberOfSignificantBits}
@cindex @code{QuantizeBruteForceNumberOfSignificantDigits}
@cindex @code{QuantizeBitRoundNumberOfSignificantBits}
@cindex @code{number_of_significant_digits}
@cindex @code{number_of_significant_bits}
@cindex @code{least_significant_digit}
@cindex Jeff Whitaker
@cindex Rich Signell
@cindex John Caron
@acronym{NCO} attaches attributes that indicate the algorithm used and
degree of precision retained for each variable affected by
@acronym{PPC}. 
The @acronym{NSD} and @acronym{DSD} algorithms store the attributes
@code{QuantizeBitGroomNumberOfSignificantDigits}
@footnote{
Prior to @acronym{NCO} version 5.0.3 (October, 2021), @acronym{NCO}
stored the @acronym{NSD} attribute
@code{number_of_significant_digits}.
However, this was deemed too ambiguous, given the increasing number of
supported quantization methods.
The new attribute names better disambiguate which algorithm was used
to quantize the data.
They also harmonize better with the metadata produced by the upcoming
netCDF library quantization features.
}
and @code{least_significant_digit}
@footnote{
A suggestion by Rich Signell and the @command{nc3tonc4} tool by Jeff
Whitaker inspired @acronym{NCO} to implement @acronym{PPC}.
Note that @acronym{NCO} implements a different @acronym{DSD} algorithm
than @command{nc3tonc4}, and produces slightly different (not
bit-for-bit) though self-consistent and equivalent results.
@command{nc3tonc4} records the precision of its @acronym{DSD} algorithm 
in the attribute @code{least_significant_digit} and @acronym{NCO} 
does the same for consistency.
The Unidata blog 
@uref{http://www.unidata.ucar.edu/blogs/developer/en/entry/compression_by_bit_shaving,  
here} also shows how to compress @acronym{IEEE} floating-point data by
zeroing insignificant bits.
The author, John Caron, writes that the technique has been called
``bit-shaving''.
We call the algorithm of always rounding-up ``bit-setting''.
And we named the algorithm produced by alternately rounding up and
down (with a few other bells and whistles) ``bit-grooming''.
Imagine orthogonally raking an already-groomed Japanese rock garden.
The criss-crossing tracks increase the pattern's entropy, and this
entropy produces self-compensating instead of accumulating errors
during statistical operations.}, respectively.

@html
<a name="ppq"></a> <!-- http://nco.sf.net/nco.html#ppq -->
<a name="PPQ"></a> <!-- http://nco.sf.net/nco.html#PPQ -->
<a name="gbr"></a> <!-- http://nco.sf.net/nco.html#gbr -->
<a name="GBR"></a> <!-- http://nco.sf.net/nco.html#GBR -->
<a name="BitGroom"></a> <!-- http://nco.sf.net/nco.html#BitGroom -->
<a name="GranularBitRound"></a> <!-- http://nco.sf.net/nco.html#GranularBitRound -->
<a name="BitShave"></a> <!-- http://nco.sf.net/nco.html#BitShave -->
<a name="BitSet"></a> <!-- http://nco.sf.net/nco.html#BitSet -->
<a name="DigitRound"></a> <!-- http://nco.sf.net/nco.html#DigitRound -->
<a name="BitGroomRound"></a> <!-- http://nco.sf.net/nco.html#BitGroomRound -->
<a name="HalfShave"></a> <!-- http://nco.sf.net/nco.html#HalfShave -->
<a name="BruteForce"></a> <!-- http://nco.sf.net/nco.html#BruteForce -->
<a name="BitRound"></a> <!-- http://nco.sf.net/nco.html#BitRound -->
@end html
@cindex @acronym{PPQ}
@cindex @acronym{GBR}
@cindex BitGroom
@cindex Granular BitRound
@cindex BitShave
@cindex BitSet
@cindex DigitRound
@cindex BitGroomRound
@cindex HalfShave
@cindex BruteForce
@cindex BitRound
As of @w{version 5.0.3} (October 2021), @acronym{NCO} supports a more
complete set of Precision-Preserving Quantization (@acronym{PPQ})
filters than has been documented here. 
The default algorithm has been changed from BitGroom with BitRound
masks from R. Kouznetsov (2021), to what we call Granular BitRound
(@acronym{GBR}).
@acronym{GBR} combines features of BitGroom, BitRound, and DigitRound by
Delaunay et al. (2019).
@acronym{GBR} improves compression ratios by ~20% relative to BitGroom
for @acronym{NSD}=3 on our benchmark @w{1 GB} climate model output
dataset.
Since it quantizes a few more bits than BitGroom (and BitGroomRound)
for a given @acronym{NSD}, @acronym{GBR} produces significantly larger
quantization errors than those algorithms as well.
@example
@verbatim
ncks -7 -L 1         --ppc default=3 in.nc out.nc # Granular BitRound (NSD)
ncks -7 -L 1 --baa=0 --ppc default=3 in.nc out.nc # BitGroom (NSD)
ncks -7 -L 1 --baa=1 --ppc default=3 in.nc out.nc # BitShave (NSD)
ncks -7 -L 1 --baa=2 --ppc default=3 in.nc out.nc # BitSet (NSD)
ncks -7 -L 1 --baa=3 --ppc default=3 in.nc out.nc # DigitRound (NSD)
ncks -7 -L 1 --baa=4 --ppc default=3 in.nc out.nc # Granular BitRound (NSD)
ncks -7 -L 1 --baa=5 --ppc default=3 in.nc out.nc # BitGroomRound (NSD)
ncks -7 -L 1 --baa=6 --ppc default=9 in.nc out.nc # HalfShave (NSB)
ncks -7 -L 1 --baa=7 --ppc default=3 in.nc out.nc # BruteForce (NSD)
ncks -7 -L 1 --baa=8 --ppc default=9 in.nc out.nc # BitRound (NSB)
@end verbatim
@end example
These @acronym{NSD} algorithms write an algorithm-specific attribute,
e.g., @code{QuantizeGranularBitRoundNumberOfSignificantDigits} or
@code{QuantizeDigitRoundNumberOfSignificantDigits}.
Documentation on these algorithms is best found in the literature.
While Bit-Groom/Shave/Set are described above, documentation on
Bit-Adjustment-Algorithms (@acronym{BAA}) 3--8 will be improved in the
future.  

@c keepbits=number_significant_bits-1
@c number_significant_bits=keepbits+1
@c keepbits=number_stored_bits (NSB) = number_explicit_significant_bits (NESB)

As of @w{version 5.0.5} (January 2022), @acronym{NCO} supports
two quantization codecs (BitRound and HalfShave) that expect a
user-specified number of explicit significant bits (@acronym{NSB},
or ``keepbits'') to retain
@footnote{
The terminology of significant bits (not to mention digits) can be
confusing.
The @acronym{IEEE} standard devotes 24 and 53 bits, respectively,
to the mantissas that determine the precision of single and double
precision floating-point numbers. 
However, the first (i.e., the most significant) of those bits is
@emph{implicit}, and is not explicitly stored.
Its value is one unless all of the exponent bits are zero.
The implicit bit is @emph{significant} thought it is not explicitly stored
and so cannot be quantized.
Therefore single and double precision floats have only 23 and 52
explicitly stored bits, respectively, that can be ``kept'' and
therefore quantized. 
Each explicit bit kept is as significant as the implicit bit.
Thus the number of ``keepbits'' is one less than the number of
significant bits, i.e., the bits that contribute to the precision of an
@acronym{IEEE} value. 
The BitRound quantization algorithm in @acronym{NCO} and in netCDF
accept as an input parameter the number of keepbits, i.e., the number
of explicit significant bits @acronym{NESB} to retain (i.e., not mask
to zero). 
Unfortunately the acronym @acronym{NSB} has been used instead of
the more accurate acronym @acronym{NESB}, and at this point it is
difficult to change. 
Therefore the @acronym{NSB} acronym and parameter as used by
@acronym{NCO} and netCDF should be interpreted as ``number of stored
bits'' (i.e., keepbits) not the ``number of significant bits''.
}.
The @acronym{NSB} argument contrasts with the number of significant 
digits (@acronym{NSD}) parameter expected by the other codecs
(like BitGroom, DigitRound, and GranularBR).
The valid ranges of @acronym{NSD} are 1--7 for @acronym{NC_FLOAT}
and 1--15 for @acronym{NC_DOUBLE}.
The valid ranges of @acronym{NSB} are 1--23 for @acronym{NC_FLOAT}
and 1--52 for @acronym{NC_DOUBLE}.
The upper limits of @acronym{NSB} are the number of explicitly
represented bits in the @acronym{IEEE} single- and double-precision
formats (the implicit bit does not count).

It is safe to attempt @acronym{PPC} on input that has already been
rounded. 
Variables can be made rounder, not sharper, i.e., variables cannot be
``un-rounded''. 
Thus @acronym{PPC} attempted on an input variable with an existing
@acronym{PPC} attribute proceeds only if the new rounding level exceeds
the old, otherwise no new rounding occurs (i.e., a ``no-op''), and the
original @acronym{PPC} attribute is retained rather than replaced with
the newer value of @var{prc}.

To request, say, five significant digits (@math{@var{nsd=5}}) for all
fields, except, say, wind speeds which are only known to integer values
(@math{@var{dsd=0}}) in the supplied units, requires @samp{--ppc} twice: 
@example
ncks -4 --ppc default=5 --ppc u,v=.0 in.nc out.nc
@end example
To preserve five digits in all variables except coordinate variables
and @var{u} and @var{v}, first specify a default value with the
@samp{default} keyword (or synonym keywords @samp{dfl}, @samp{global},
or @samp{glb}), and separately specify the exceptions with
per-variable key-value pairs: 
@example
ncks --ppc default=5 --ppc u,v=20 in.nc out.nc
@end example
The @samp{--ppc} option may be specified any number of times to
support varying precision types and levels, and each option may
aggregate all the variables with the same precision
@example
ncks --ppc p,w,z=5 --ppc q,RH=4 --ppc T,u,v=3 in.nc out.nc
ncks --ppc p,w,z=5#q,RH=4#T,u,v=3 in.nc out.nc # Multi-argument format
@end example
Any @var{var} argument may be a regular expression.
This simplifies generating lists of related variables:
@example
ncks --ppc Q.?=5 --ppc FS.?,FL.?=4 --ppc RH=.3 in.nc out.nc
ncks --ppc Q.?=5#FS.?,FL.?=4#RH=.3 in.nc out.nc # Multi-argument format
@end example
Although @acronym{PPC}-rounding instantly reduces data precision,
on-disk storage reduction only occurs once the data are compressed.  

@html
<a name="ppc_tbl_ffc"></a> <!-- http://nco.sf.net/nco.html#ppc_tbl_ffc -->
@end html
How can one be sure the lossy data are sufficiently precise?
@acronym{PPC} preserves all significant digits of every value.
The @acronym{DSD} algorithm uses floating-point math to round each 
value optimally so that it has the maximum number of zeroed bits
that preserve the specified precision.
The @acronym{NSD} algorithm uses a theoretical approach (3.2 bits per
base-10 digit), tuned and tested to ensure the @emph{worst} case
quantization error is less than half the value of the minimum increment
in the least significant digit. 

@ifhtml
@cartouche
@html
<p><b>Note for HTML users</b>: 
<br>The definition of error metrics relies heavily on mathematical
expressions that cannot easily be represented in HTML.  
<b>See the <a href="./nco.pdf">printed manual</a> for much more detailed
and complete documentation of this subject.</b>
@end html
@end cartouche
@end ifhtml
@ifnothtml
@ifinfo
@emph{Note for Info users}: 
The definition of error metrics relies heavily on mathematical
expressions which cannot be easily represented in Info.
@emph{See the @uref{./nco.pdf, printed manual} for much more detailed
and complete documentation of this subject.}
@end ifinfo
@end ifnothtml

@tex
We define several metrics to quantify the quantization error.
The mean error~$\pslavg$ and mean absolute error~$\pslmebs$ incurred in
quantizing a variable from its true values~$\xxx_{\idx}$ to quantized
values~$\qqq_{\idx}$ are, respectively,
$$
\pslavg = {\sum_{\idx=1}^{\idx=\lmnnbr} \mssflg_{\idx}
\mskflg_{\idx} \wgt_{\idx} ( \xxx_{\idx} - \qqq_{\idx} ) \over
\sum_{\idx=1}^{\idx=\lmnnbr} \mssflg_{\idx} \mskflg_{\idx} \wgt_{\idx}}\qquad\hbox{and}\qquad
\pslmebs = {\sum_{\idx=1}^{\idx=\lmnnbr} \mssflg_{\idx}
\mskflg_{\idx} \wgt_{\idx} | \xxx_{\idx} - \qqq_{\idx} | \over
\sum_{\idx=1}^{\idx=\lmnnbr} \mssflg_{\idx} \mskflg_{\idx} \wgt_{\idx}}  
$$
where $\mssflg_{\idx}$ is~1 unless $\xxx_{\idx}$~equals the missing
value, $\mskflg_{\idx}$ is~1 unless $\xxx_{\idx}$~is masked, and
$\wgt_{\idx}$~is the weight.
The maximum and minimum errors $\pslmax$ and~$\pslmin$ are both signed 
$$
\pslmax = {\rm max}(\xxx_{\idx} - \qqq_{\idx})\qquad\hbox{and}\qquad
\pslmin = {\rm min}(\xxx_{\idx} - \qqq_{\idx})
$$
while the maximum and minimum absolute errors $\pslmabs$ and $\pslmibs$
are positive-definite. 
$$
\pslmabs = {\rm max}|\xxx_{\idx} - \qqq_{\idx}| = {\rm max}(|\pslmax|,|\pslmin|)
$$
$$
\pslmibs = {\rm min}|\xxx_{\idx} - \qqq_{\idx}| = {\rm min}(|\pslmax|,|\pslmin|)
$$
Typically $\pslmibs = 0$ for quantization, since many exact values
need no quantization.
Bit-shifting zeros into the least significant bits (@acronym{LSB}s)
always underestimates true values so that $\pslmax = 0$.
Conversely, bit-shifting ones into the @acronym{LSB}s always
overestimates true values so that $\pslmin = 0$. 
Our @acronym{NSD} algorithm is balanced because it alternates
bit-shifting zeroes and ones.
Balanced algorithms should yield $\pslmax \approx -\pslmin$,
$\pslmabs \approx \pslmibs$, and $\pslavg \approx 0$.

The three most important error metrics for quantization are 
$\pslmabs$, $\pslmebs$, and~$\pslavg$.
The upper bound (worst case) quantization performance is~$\pslmabs$.
$\pslmebs$~measures the absolute mean accuracy of quantization, and does
not allow positive and negative offsets to compensate eachother and
conceal poor performance.
The difference bewtween~$\pslmabs$ and~$\pslmebs$ indicates how much 
of an outlier the worst case is.
The mean accuracy~$\pslavg$ indicates whether statistical properties of 
quantized numbers will accurately reflect the true values. 
@end tex

All three metrics are expressed in terms of the fraction of the ten's
place occupied by the @acronym{LSD}.
If the @acronym{LSD} is the hundreds digit or the thousandths digit,
then the metrics are fractions @w{of 100}, or @w{of 1/100},
respectively. 
@acronym{PPC} algorithms should produce maximum absolute errors no
greater @w{than 0.5} in these units.
If the @acronym{LSD} is the hundreds digit, then quantized versions of
true values will be within fifty of the true value.
It is much easier to satisfy this tolerance for a true value 
@w{of 100} (only 50% accuracy required) than @w{for 999} 
@w{(5% accuracy} required). 
Thus the minimum accuracy guaranteed for @math{@var{nsd}=1} ranges
from 5--50%.
For this reason, the best and worst cast performance usually occurs for
true values whose @acronym{LSD} value is close to one and nine,
respectively. 
Of course most users prefer @math{@var{prc} > 1} because accuracies
increase exponentially with @var{prc}.
Continuing the previous example to @math{@var{prc}=2}, 
quantized versions of true values from 1000--9999 will also be 
@w{within 50} of the true value, i.e., have accuracies from 0.5--5%.
In other words, only two significant digits are necessary to guarantee 
better @w{than 5%} accuracy in quantization. 
We recommend that dataset producers and users consider quantizing
datasets with @math{@var{nsd}=3}.
This guarantees accuracy of 0.05--0.5% for individual values.
Statistics computed from ensembles of quantized values will, assuming
the mean error 
@set flg
@tex
$\pslavg$
@clear flg
@end tex
@ifinfo
@math{@var{Emean}}
@clear flg
@end ifinfo
@ifset flg
@c texi2html does not like @math{}
@var{Emean}
@clear flg
@end ifset
is small, have much better accuracy @w{than 0.5%}.
This accuracy is the most that can be justified for many applications.   

To demonstrate these principles we conduct error analyses on an
artificial, reproducible dataset, and on an actual dataset of
observational analysis values. 
@footnote{The artificial dataset employed is one million evenly spaced
values from 1.0--2.0.
The analysis data are @math{N=13934592} values of the temperature field
from the @acronym{NASA} @acronym{MERRA} analysis of 20130601.}
The table summarizes quantization accuracy based on the three metrics.
@table @code
@item NSD
Number of Significant Digits.
@item Emabs
Maximum absolute error.
@item Emebs
Mean absolute error.
@item Emean
Mean error.
@end table

@ignore
9.9692099683868690e+36f = 0 1111100 111100000000000000000000 NC_FILL_FLOAT
2.3509885615147286E-38f = 0 0000000 111111111111111111111111 Recommended
9.9692099683868690e+36  = 0 1000111100 11110000000000000000000000000000000000000000000000000 NC_FILL_DOUBLE
2.84809453888921745415348453979E-306 = 0 0000000111 11111111111111111111111111111111111111111111111111111 Recommended

2.22507385850720088902458687609E-308 = 0 0000000000 11111111111111111111111111111111111111111111111111111
1.17549435082228737746264717725E-38  = 0 0111000000 11111111111111111111111111111111111111111111111111111
1.17549435082228737746264717725E-38  = 0 0000000    100000000000000000000000

# Accuracy tests

# Satellite:
ncks -3 -O -v T ${DATA}/hdf/MERRA300.prod.assim.inst3_3d_asm_Cp.20130601.nc ~/foo_T.nc
ncatted -a scale_factor,,d,, -a add_offset,,d,, ~/foo_T.nc
ncap2 -O -v -s 'ppc=T;nsd1=ppc;nsd2=ppc;nsd3=ppc;nsd4=ppc;nsd5=ppc;nsd6=ppc;nsd7=ppc' ~/foo_T.nc ~/foo_ppc_in.nc
# Artificial double
ncap2 -O -v -s 'defdim("dmn",1000000);ppc=array(1.0,1.e-6,$dmn);nsd1=ppc;nsd2=ppc;nsd3=ppc;nsd4=ppc;nsd5=ppc;nsd6=ppc;nsd7=ppc' ~/nco/data/in.nc ~/foo_ppc_in.nc
# Artificial float
ncap2 -O -v -s 'defdim("dmn",1000000);ppc=float(array(1.0,1.e-6,$dmn));nsd1=ppc;nsd2=ppc;nsd3=ppc;nsd4=ppc;nsd5=ppc;nsd6=ppc;nsd7=ppc' ~/nco/data/in.nc ~/foo_ppc_in.nc

# nc3tonc4
nc3tonc4 -o --quantize=nsd1=1,nsd2=2,nsd3=3,nsd4=4,nsd5=5,nsd6=6,nsd7=7 --quiet=1 ~/foo_ppc_in.nc ~/foo_ppc_out.nc
# DSD
ncks -O -C --ppc nsd1=.1 --ppc nsd2=.2 --ppc nsd3=.3 --ppc nsd4=.4 --ppc nsd5=.5 --ppc nsd6=.6 --ppc nsd7=.7 ~/foo_ppc_in.nc ~/foo_ppc_out.nc
# NSD
ncks -O -C --ppc nsd1=1 --ppc nsd2=2 --ppc nsd3=3 --ppc nsd4=4 --ppc nsd5=5 --ppc nsd6=6 --ppc nsd7=7 ~/foo_ppc_in.nc ~/foo_ppc_out.nc

# Generic rounding test:
ncbo -O -C ~/foo_ppc_out.nc ~/foo_ppc_in.nc ~/foo_ppc_dff.nc
ncbo -O -C -y dvd ~/foo_ppc_dff.nc ~/foo_ppc_in.nc ~/foo_ppc_rat.nc
ncap2 -O -v -s 'nsd1*=10;nsd2*=100;nsd3*=1000;nsd4*=10000;nsd5*=100000;nsd6*=1000000;nsd7*=10000000' ~/foo_ppc_rat.nc ~/foo_ppc_rat_scl.nc
ncwa -O -C -y avg ~/foo_ppc_rat_scl.nc ~/foo_ppc_avg.nc
ncwa -O -C -y max ~/foo_ppc_rat_scl.nc ~/foo_ppc_max.nc
ncwa -O -C -y min ~/foo_ppc_rat_scl.nc ~/foo_ppc_min.nc
ncwa -O -C -y mabs ~/foo_ppc_rat_scl.nc ~/foo_ppc_mabs.nc
ncwa -O -C -y mebs ~/foo_ppc_rat_scl.nc ~/foo_ppc_mebs.nc
ncwa -O -C -y mibs ~/foo_ppc_rat_scl.nc ~/foo_ppc_mibs.nc
@end ignore
@example
Artificial Data: N=1000000 values in [1.0,2.0) in steps of 1.0e-6
Single-Precision        Double-Precision   Single-Precision
NSD Emabs Emebs Emean   Emabs Emebs Emean  DSD Emabs Emebs Emean
 1  0.31  0.11  4.1e-4  0.31  0.11  4.0e-4  1  0.30  0.11 -8.1e-4  
 2  0.39  0.14  6.8e-5  0.39  0.14  5.5e-5  2  0.39  0.14 -1.3e-4
 3  0.49  0.17  1.0e-6  0.49  0.17 -5.5e-7  3  0.49  0.17 -2.0e-5
 4  0.30  0.11  3.2e-7  0.30  0.11 -6.1e-6  4  0.30  0.11  5.1e-8
 5  0.37  0.13  3.1e-7  0.38  0.13 -5.6e-6  5  0.38  0.13  2.6e-6
 6  0.36  0.12 -4.4e-7  0.48  0.17 -4.1e-7  6  0.48  0.17  7.2e-6
 7  0.00  0.00  0.0     0.30  0.10  1.5e-7  7  0.00  0.00  0.0     

Observational Analysis: N=13934592 values MERRA Temperature 20130601
Single-Precision        
NSD Emabs Emebs Emean   
 1  0.31  0.11  2.4e-3
 2  0.39  0.14  3.8e-4
 3  0.49  0.17 -9.6e-5 
 4  0.30  0.11  2.3e-3
 5  0.37  0.13  2.2e-3
 6  0.36  0.13  1.7e-2
 7  0.00  0.00  0.0     
@end example
All results show that @acronym{PPC} quantization performs as expected.
Absolute maximum errors @math{@var{Emabs} < 0.5} for all @var{prc}.
For @math{1 <= @var{prc} <= 6}, quantization results in comparable
maximum absolute and mean absolute errors @var{Emabs} and @var{Emebs},
respectively. 
Mean errors @var{Emean} are orders of magnitude smaller because 
quantization produces over- and under-estimated values in balance.
When @math{@var{prc}=7}, quantization of single-precision values is
ineffective, because all available bits are used to represent the 
maximum precision of seven digits.
The maximum and mean absolute errors @var{Emabs} and @var{Emebs} are
nearly identical across algorithms, precisions, and dataset types.
This is consistent with both the artificial data and empirical data
being random, and thus exercising equally strengths and weaknesses of
the algorithms over the course of millions of input values.
We generated artificial arrays with many different starting values
and interval spacing and all gave qualitatively similar results.
The results presented are the worst obtained.

The artificial data has much smaller mean error @var{Emean} than the
observational analysis. 
The reason why is unclear.
It may be because the temperature field is concentrated in particular
ranges of values (and associated quantization errors) prevalent on
Earth, e.g., @math{200 < @var{T} < 320}. 
It is worth noting that the mean error @math{@var{Emean} < 0.01} for
@math{1 <= @var{prc} < 6}, and that @var{Emean} is typically at least 
two or more orders of magnitude less than @var{Emabs}.
Thus quantized values with precisions as low as @math{@var{prc}=1} 
still yield highly significant statistics by contemporary scientific
standards. 

Testing shows that @acronym{PPC} quantization enhances compression of 
typical climate datasets. 
The degree of enhancement depends, of course, on the required
precision. 
Model results are often computed as @code{NC_DOUBLE} then archived 
as @code{NC_FLOAT} to save space. 
@ignore
# Unidata Compression Blog:
# http://www.unidata.ucar.edu/blogs/developer/en/entry/netcdf_compression

# Table entries
fl=${DATA}/dstmch90/dstmch90_clm.nc
fl=${DATA}/hdf/b1850c5cn_doe_polar_merged_0_cesm1_2_0_HD+MAM4+tun2b.hp.e003.cam.h0.0001-01.nc
fl=${DATA}/hdf/MERRA300.prod.assim.inst3_3d_asm_Cp.20130601.nc
fl=${DATA}/hdf/OMI-Aura_L2-OMIAuraSO2_2012m1222-o44888_v01-00-2014m0107t114720.h5

# ncks PPC only
sz_rgn=`ls -l ${fl} | cut -d ' ' -f 5`
fmt_fnc(){ gawk '{print "old=" $1/1000000 " MB, new=" $2/1000000 " MB, cmp=" 100*$2/$1 "%"}';} 
cmd="ls -l ${fl}";sz_new=`${cmd} | cut -d ' ' -f 5`;fmt_fnc ${sz_rgn} ${sz_new}

sz_new=`bzip2 -1 -f ${fl};ls -l ${fl}.bz2 | cut -d ' ' -f 5`;bunzip2 ${fl}.bz2;echo ${sz_rgn} ${sz_new} | gawk '{print "old=" $1/1000000 " MB, new=" $2/1000000 " MB, cmp=" 100*$2/$1 "%"}'
sz_new=`bzip2 -9 -f ${fl};ls -l ${fl}.bz2 | cut -d ' ' -f 5`;bunzip2 ${fl}.bz2;echo ${sz_rgn} ${sz_new} | gawk '{print "old=" $1/1000000 " MB, new=" $2/1000000 " MB, cmp=" 100*$2/$1 "%"}'
sz_new=`ncks -O -7 -L 0 ${fl} ~/foo.nc;ls -l ~/foo.nc | cut -d ' ' -f 5`;echo ${sz_rgn} ${sz_new} | gawk '{print "old=" $1/1000000 " MB, new=" $2/1000000 " MB, cmp=" 100*$2/$1 "%"}'
sz_new=`ncks -O -7 -L 1 ${fl} ~/foo.nc;ls -l ~/foo.nc | cut -d ' ' -f 5`;echo ${sz_rgn} ${sz_new} | gawk '{print "old=" $1/1000000 " MB, new=" $2/1000000 " MB, cmp=" 100*$2/$1 "%"}'
sz_new=`ncks -O -7 -L 9 ${fl} ~/foo.nc;ls -l ~/foo.nc | cut -d ' ' -f 5`;echo ${sz_rgn} ${sz_new} | gawk '{print "old=" $1/1000000 " MB, new=" $2/1000000 " MB, cmp=" 100*$2/$1 "%"}'
sz_new=`ncpdq -O -7 -L 0 ${fl} ~/foo.nc;ls -l ~/foo.nc | cut -d ' ' -f 5`;echo ${sz_rgn} ${sz_new} | gawk '{print "old=" $1/1000000 " MB, new=" $2/1000000 " MB, cmp=" 100*$2/$1 "%"}'
sz_new=`ncpdq -O -7 -L 1 ${fl} ~/foo.nc;ls -l ~/foo.nc | cut -d ' ' -f 5`;echo ${sz_rgn} ${sz_new} | gawk '{print "old=" $1/1000000 " MB, new=" $2/1000000 " MB, cmp=" 100*$2/$1 "%"}'
sz_new=`ncks -O -7 -L 1 --ppc default=7 ${fl} ~/foo.nc;ls -l ~/foo.nc | cut -d ' ' -f 5`;echo ${sz_rgn} ${sz_new} | gawk '{print "old=" $1/1000000 " MB, new=" $2/1000000 " MB, cmp=" 100*$2/$1 "%"}'
sz_new=`ncks -O -7 -L 1 --ppc default=6 ${fl} ~/foo.nc;ls -l ~/foo.nc | cut -d ' ' -f 5`;echo ${sz_rgn} ${sz_new} | gawk '{print "old=" $1/1000000 " MB, new=" $2/1000000 " MB, cmp=" 100*$2/$1 "%"}'
sz_new=`ncks -O -7 -L 1 --ppc default=5 ${fl} ~/foo.nc;ls -l ~/foo.nc | cut -d ' ' -f 5`;echo ${sz_rgn} ${sz_new} | gawk '{print "old=" $1/1000000 " MB, new=" $2/1000000 " MB, cmp=" 100*$2/$1 "%"}'
sz_new=`ncks -O -7 -L 1 --ppc default=4 ${fl} ~/foo.nc;ls -l ~/foo.nc | cut -d ' ' -f 5`;echo ${sz_rgn} ${sz_new} | gawk '{print "old=" $1/1000000 " MB, new=" $2/1000000 " MB, cmp=" 100*$2/$1 "%"}'
sz_new=`ncks -O -7 -L 1 --ppc default=3 ${fl} ~/foo.nc;ls -l ~/foo.nc | cut -d ' ' -f 5`;echo ${sz_rgn} ${sz_new} | gawk '{print "old=" $1/1000000 " MB, new=" $2/1000000 " MB, cmp=" 100*$2/$1 "%"}'
sz_new=`ncks -O -7 -L 1 --ppc default=2 ${fl} ~/foo.nc;ls -l ~/foo.nc | cut -d ' ' -f 5`;echo ${sz_rgn} ${sz_new} | gawk '{print "old=" $1/1000000 " MB, new=" $2/1000000 " MB, cmp=" 100*$2/$1 "%"}'
sz_new=`ncks -O -7 -L 1 --ppc default=1 ${fl} ~/foo.nc;ls -l ~/foo.nc | cut -d ' ' -f 5`;echo ${sz_rgn} ${sz_new} | gawk '{print "old=" $1/1000000 " MB, new=" $2/1000000 " MB, cmp=" 100*$2/$1 "%"}'
sz_new=`ncpdq -O -7 -L 1 --ppc default=4 ${fl} ~/foo.nc;ls -l ~/foo.nc | cut -d ' ' -f 5`;echo ${sz_rgn} ${sz_new} | gawk '{print "old=" $1/1000000 " MB, new=" $2/1000000 " MB, cmp=" 100*$2/$1 "%"}'
sz_new=`ncpdq -O -7 -L 9 --ppc default=4 ${fl} ~/foo.nc;ls -l ~/foo.nc | cut -d ' ' -f 5`;echo ${sz_rgn} ${sz_new} | gawk '{print "old=" $1/1000000 " MB, new=" $2/1000000 " MB, cmp=" 100*$2/$1 "%"}'
sz_new=`ncpdq -O -3 --ppc default=4 ${fl} ~/foo.nc;bzip2 -1 -f ~/foo.nc;ls -l ~/foo.nc.bz2 | cut -d ' ' -f 5`;echo ${sz_rgn} ${sz_new} | gawk '{print "old=" $1/1000000 " MB, new=" $2/1000000 " MB, cmp=" 100*$2/$1 "%"}'
@end ignore
@html
<a name="ppc_tbl_nc"></a> <!-- http://nco.sf.net/nco.html#ppc_tbl_nc -->
@end html
@cindex Burrows-Wheeler algorithm
@cindex @command{bzip2}
This table summarizes the performance of lossless and lossy compression
on two typical, or at least random, netCDF data files.  
The files were taken from representative model-simulated and
satellite-retrieved datasets.
Only floating-point data were compressed.
No attempt was made to compress integer-type variables as they occupy an 
insignificant fraction of every dataset.
The columns are
@table @code
@item Type 
File-type: 
@kbd{N3} for netCDF @code{CLASSIC},
@kbd{N4} for @code{NETCDF4}, 
@kbd{N7} for @code{NETCDF4_CLASSIC} (which comprises netCDF3
data types and structures with netCDF4 storage features like
compression),  
@kbd{H4} for @acronym{HDF4}, and
@kbd{H5} for @acronym{HDF5}.
@kbd{N4/7} means results apply to both @kbd{N4} and @kbd{N7} filetypes.
@item LLC
Type of lossless compression employed, if any.
Bare numbers refer to the strength of the @acronym{DEFLATE} algorithm
employed internally by netCDF4/@acronym{HDF5}, while numbers prefixed
with @kbd{B} refer to the block size employed by the Burrows-Wheeler
algorithm in @command{bzip2}. 
@item PPC 
Number of significant digits retained by the precision-preserving
compression @acronym{NSD} algorithm.  
@item Pck 
@kbd{Y} if the default @command{ncpdq} packing algorithm (convert
floating-point types to @code{NC_SHORT}) was employed.
@item Size
Resulting filesize in @acronym{MB}.
@item %
Compression ratio, i.e., resulting filesize relative to original size,
in percent. 
In some cases the original files is already losslessly compressed. 
The compression ratios reported are relative to the size of the original
file as distributed, not as optimally losslessly compressed. 
@end table
A dash (@kbd{-}) indicates the associated compression feature
was not employed.
@ignore
# Lines trimmed from table as redundant/misleading
  N7   1   4  Y    7.9  22.9  ncpdq --ppc default=4
  N7   9   4  Y    7.7  22.1  ncpdq -L 9 --ppc default=4
  N3  B1   4  Y    6.3  18.1  ncpdq --ppc default=4 bzip2 -1

  N7   1   4  Y   26.3  21.9  ncpdq --ppc default=4
  N7   9   4  Y   25.6  21.4  ncpdq -L 9 --ppc default=4
  N3  B1   4  Y   20.9  17.4  ncpdq --ppc default=4 bzip2 -1

N4/7   1   4  Y  133.6  54.7  ncpdq -L 1 --ppc default=4
N4/7   9   4  Y  127.0  52.0  ncpdq -L 9 --ppc default=4
  N3  B1   4  Y  114.0  46.7  ncpdq --ppc default=4 bzip2 -1

  N4   1   4  Y   13.0  44.0  ncpdq -L 1 --ppc default=4
  N4   9   4  Y   12.5  42.5  ncpdq -L 9 --ppc default=4
@end ignore
@example
# dstmch90_clm.nc
Type LLC PPC Pck  Size   %    Flags and Notes
  N3   -   -  -   34.7 100.0  Original is not compressed
  N3  B1   -  -   28.9  83.2  bzip2 -1
  N3  B9   -  -   29.3  84.4  bzip2 -9
  N7   -   -  -   35.0 101.0     
  N7   1   -  -   28.2  81.3  -L 1
  N7   9   -  -   28.0  80.8  -L 9
  N7   -   -  Y   17.6  50.9  ncpdq -L 0
  N7   1   -  Y    7.9  22.8  ncpdq -L 1
  N7   1   7  -   28.2  81.3  --ppc default=7
  N7   1   6  -   27.9  80.6  --ppc default=6
  N7   1   5  -   25.9  74.6  --ppc default=5
  N7   1   4  -   22.3  64.3  --ppc default=4
  N7   1   3  -   18.9  54.6  --ppc default=3
  N7   1   2  -   14.5  43.2  --ppc default=2
  N7   1   1  -   10.0  29.0  --ppc default=1

# b1850c5cn_doe_polar_merged_0_cesm1_2_0_HD+MAM4+tun2b.hp.e003.cam.h0.0001-01.nc
Type LLC PPC Pck  Size   %    Flags and Notes
  N3   -   -  -  119.8 100.0  Original is not compressed
  N3  B1   -  -   84.2  70.3  bzip2 -1
  N3  B9   -  -   84.8  70.9  bzip2 -9
  N7   -   -  -  120.5 100.7     
  N7   1   -  -   82.6  69.0  -L 1
  N7   9   -  -   82.1  68.6  -L 9
  N7   -   -  Y   60.7  50.7  ncpdq -L 0
  N7   1   -  Y   26.0  21.8  ncpdq -L 1
  N7   1   7  -   82.6  69.0  --ppc default=7
  N7   1   6  -   81.9  68.4  --ppc default=6
  N7   1   5  -   77.2  64.5  --ppc default=5
  N7   1   4  -   69.0  57.6  --ppc default=4
  N7   1   3  -   59.3  49.5  --ppc default=3
  N7   1   2  -   49.5  41.3  --ppc default=2
  N7   1   1  -   38.2  31.9  --ppc default=1

# MERRA300.prod.assim.inst3_3d_asm_Cp.20130601.hdf
Type LLC PPC Pck  Size   %    Flags and Notes
  H4   5   -  -  244.3 100.0  Original is compressed
  H4  B1   -  -  244.7 100.1  bzip2 -1
  N4   5   -  -  214.5  87.8
  N7   5   -  -  210.6  86.2  
  N4  B1   -  -  215.4  88.2  bzip2 -1
  N4  B9   -  -  214.8  87.9  bzip2 -9
  N3   -   -  -  617.1 252.6
N4/7   -   -  -  694.0 284.0  -L 0
N4/7   1   -  -  223.2  91.3  -L 1
N4/7   9   -  -  207.3  84.9  -L 9
N4/7   -   -  Y  347.1 142.1  ncpdq -L 0
N4/7   1   -  Y  133.6  54.7  ncpdq -L 1
N4/7   1   7  -  223.1  91.3  --ppc default=7
N4/7   1   6  -  225.1  92.1  --ppc default=6
N4/7   1   5  -  221.4  90.6  --ppc default=5
N4/7   1   4  -  201.4  82.4  --ppc default=4
N4/7   1   3  -  185.3  75.9  --ppc default=3
N4/7   1   2  -  150.0  61.4  --ppc default=2
N4/7   1   1  -  100.8  41.3  --ppc default=1

# OMI-Aura_L2-OMIAuraSO2_2012m1222-o44888_v01-00-2014m0107t114720.h5
Type LLC PPC Pck  Size   %    Flags and Notes
  H5   5   -  -   29.5 100.0  Original is compressed
  H5  B1   -  -   29.3  99.6  bzip2 -1
  N4   5   -  -   29.5 100.0
  N4  B1   -  -   29.3  99.6  bzip2 -1
  N4  B9   -  -   29.3  99.4  bzip2 -9
  N4   -   -  -   50.7 172.3  -L 0
  N4   1   -  -   29.8 101.3  -L 1
  N4   9   -  -   29.4  99.8  -L 9
  N4   -   -  Y   27.7  94.0  ncpdq -L 0
  N4   1   -  Y   12.9  43.9  ncpdq -L 1
  N4   1   7  -   29.7 100.7  --ppc default=7
  N4   1   6  -   29.7 100.8  --ppc default=6
  N4   1   5  -   27.3  92.8  --ppc default=5
  N4   1   4  -   23.8  80.7  --ppc default=4
  N4   1   3  -   20.3  69.0  --ppc default=3
  N4   1   2  -   15.1  51.2  --ppc default=2
  N4   1   1  -    9.9  33.6  --ppc default=1
@end example

@cindex @acronym{NCAR}
@cindex @acronym{CAM}
@cindex @acronym{GCM}
A selective, per-variable approach to @acronym{PPC} yields the
best balance of precision and compression yet requires the dataset
producer to understand the intrinsic precision of each variable.
Such a specification for a @acronym{GCM} dataset might look like this 
(using names for the @acronym{NCAR} @acronym{CAM} model):
@example
# Be conservative on non-explicit quantities, so default=5
# Some quantities deserve four significant digits
# Many quantities, such as aerosol optical depths and burdens, are 
# highly uncertain and only useful to three significant digits.
ncks -7 -O \
--ppc default=5 \
--ppc AN.?,AQ.?=4 \
--ppc AER.?,AOD.?,ARE.?,AW.?,BURDEN.?=3 \
ncar_cam.nc ~/foo.nc
@end example

@html
<a name="dfl_lvl"></a> <!-- http://nco.sf.net/nco.html#dfl_lvl -->
<a name="dfl"></a> <!-- http://nco.sf.net/nco.html#dfl -->
<a name="lz"></a> <!-- http://nco.sf.net/nco.html#lz -->
<a name="lz77"></a> <!-- http://nco.sf.net/nco.html#lz77 -->
<a name="deflate"></a> <!-- http://nco.sf.net/nco.html#deflate -->
<a name="deflation"></a> <!-- http://nco.sf.net/nco.html#deflation -->
@end html
@node Deflation, MD5 digests, Compression, Shared features
@section Deflation
@cindex @code{-L}
@cindex @code{--deflate}
@cindex @code{--dfl_lvl}
@cindex Lempel-Ziv deflation
@cindex compression
@cindex deflation
@cartouche
Availability: @command{ncap2}, @command{ncbo}, @command{ncclimo}, @command{nces},
@command{ncecat}, @command{ncflint}, @command{ncks}, @command{ncpdq},
@command{ncra}, @command{ncrcat}, @command{ncremap}, @command{ncwa}@*
Short options: @samp{-L}@*
Long options: @samp{--dfl_lvl}, @samp{--deflate}@*  
@end cartouche

All @acronym{NCO} operators that define variables support the netCDF4
feature of storing variables compressed with the lossless
@acronym{DEFLATE} compression algorithm.
@acronym{DEFLATE} combines the Lempel-Ziv encoding with Huffman coding.
The specific version used by netCDF4/@acronym{HDF5} is that implemented
in the @command{zlib} library used by @command{gzip}.
Activate deflation with the @code{-L @var{dfl_lvl}} short option (or
with the same argument to the @samp{--dfl_lvl} or @samp{--deflate} long
options). 
Specify the deflation level @var{dfl_lvl} on a scale from no deflation
(@var{dfl_lvl = 0}) to maximum deflation (@var{dfl_lvl = 9}).
Under the hood, this selects the compression blocksize.
Minimal deflation (@var{dfl_lvl = 1}) achieves considerable storage
compression with little time penalty.
Higher deflation levels require more time for compression.
File sizes resulting from minimal (@var{dfl_lvl = 1}) and maximal   
(@var{dfl_lvl = 9}) deflation levels typically differ by less 
@w{than 10%} in size. 

To compress an entire file using deflation, use
@example
ncks -4 -L 0 in.nc out.nc # No deflation (fast, no time penalty)
ncks -4 -L 1 in.nc out.nc # Minimal deflation (little time penalty)
ncks -4 -L 9 in.nc out.nc # Maximal deflation (much slower)
@end example

Unscientific testing shows that deflation compresses typical climate
datasets by 30-60%.  
Packing, a lossy compression technique available for all netCDF files 
(see @ref{Packed data}), can easily compress files by 50%.
Packed data may be deflated to squeeze datasets by about 80%:
@example
ncks  -4 -L 1 in.nc out.nc # Minimal deflation (~30-60% compression)
ncks  -4 -L 9 in.nc out.nc # Maximal deflation (~31-63% compression)
ncpdq         in.nc out.nc # Standard packing  (~50% compression)
ncpdq -4 -L 9 in.nc out.nc # Deflated packing  (~80% compression)
@end example
@findex ncks
@command{ncks} prints deflation parameters, if any, to screen
(@pxref{ncks netCDF Kitchen Sink}).

@html
<a name="md5"></a> <!-- http://nco.sf.net/nco.html#md5 -->
<a name="digest"></a> <!-- http://nco.sf.net/nco.html#digest -->
<a name="hash"></a> <!-- http://nco.sf.net/nco.html#hash -->
<a name="integrity"></a> <!-- http://nco.sf.net/nco.html#integrity -->
<a name="security"></a> <!-- http://nco.sf.net/nco.html#security -->
@end html
@node MD5 digests, Buffer sizes, Deflation, Shared features
@section MD5 digests
@cindex @code{--md5_digest}
@cindex @code{--md5_dgs}
@cindex @code{--md5_wrt_att}
@cindex @code{--md5_write_attribute}
@cindex integrity
@cindex security
@cindex digest
@cindex hash
@cindex MD5 digest
@cartouche
Availability: 
@command{ncecat}, @command{ncks}, @command{ncrcat}@*
Short options: @*
Long options: @samp{--md5_dgs}, @samp{--md5_digest}, @samp{--md5_wrt_att}, @samp{--md5_write_attribute}@*
@end cartouche

As of @acronym{NCO} version 4.1.0 (April, 2012), @acronym{NCO} 
supports data integrity verification using the @acronym{MD5} digest
algorithm. 
This support is currently implemented in @command{ncks} and in the
multi-file concatenators @command{ncecat} and @command{ncrcat}.
Activate it with the @samp{--md5_dgs} or @samp{--md5_digest} long
options. 
As of @acronym{NCO} version 4.3.3 (July, 2013), @acronym{NCO} 
will write the @acronym{MD5} digest of each variable as an
@code{NC_CHAR} attribute named @code{MD5}.
This support is currently implemented in @command{ncks} and in the
multi-file concatenators @command{ncecat} and @command{ncrcat}.
Activate it with the @samp{--md5_wrt_att} or
@samp{--md5_write_attribute} long options.

The behavior and verbosity of the @acronym{MD5} digest is
operator-dependent.
@acronym{MD5} digests may be activated in both @command{ncks} invocation
types, the one-filename argument mode for printing sub-setted and
hyperslabbed data, and the two-filename argument mode for copying that
data to a new file. 
Both modes will incur minor overhead from performing the hash
algorithm for each variable read, and each variable written will
have an additional attribute named @code{MD5}.
When activating @acronym{MD5} digests with @command{ncks} it is assumed
that the user wishes to print the digest of every variable when the
debugging level exceeds one. 

@command{ncks} displays an @acronym{MD5} digest as a 32-character
hexadecimal string in which each two characters represent one byte of 
the 16-byte digest: 
@example
> ncks --trd -D 2 -C --md5 -v md5_a,md5_abc ~/nco/data/in.nc
...
ncks: INFO MD5(md5_a) = 0cc175b9c0f1b6a831c399e269772661
md5_a = 'a' 
ncks: INFO MD5(md5_abc) = 900150983cd24fb0d6963f7d28e17f72
lev[0]=100 md5_abc[0--2]='abc' 
> ncks --trd -D 2 -C -d lev,0 --md5 -v md5_a,md5_abc ~/nco/data/in.nc
...
ncks: INFO MD5(md5_a) = 0cc175b9c0f1b6a831c399e269772661
md5_a = 'a' 
ncks: INFO MD5(md5_abc) = 0cc175b9c0f1b6a831c399e269772661
lev[0]=100 md5_abc[0--0]='a' 
@end example
In fact these examples demonstrate the validity of the hash algorithm
since the @acronym{MD5} hashes of the strings ``a'' and ``abc'' are
widely known.
The second example shows that the hyperslab of variable @code{md5_abc}
(= ``abc'') consisting of only its first letter (= ``a'') has the same
hash as the variable @code{md5_a} (``a'').
This illustrates that @acronym{MD5} digests act only on variable data,
not on metadata. 

When activating @acronym{MD5} digests with @command{ncecat} or
@command{ncrcat} it is assumed that the user wishes to verify
that every variable written to disk has the same @acronym{MD5} digest 
as when it is subsequently read from disk.
This incurs the major additional overhead of reading in each variable
after it is written and performing the hash algorithm again on that to
compare to the original hash.
Moreover, it is assumed that such operations are generally done in
``production mode'' where the user is not interested in actually
examining the digests herself.
The digests proceed silently unless the debugging level exceeds three:
@example
> ncecat -O -D 4 --md5 -p ~/nco/data in.nc in.nc ~/foo.nc | grep MD5
...
ncecat: INFO MD5(wnd_spd) = bec190dd944f2ce2794a7a4abf224b28
ncecat: INFO MD5 digests of RAM and disk contents for wnd_spd agree
> ncrcat -O -D 4 --md5 -p ~/nco/data in.nc in.nc ~/foo.nc | grep MD5
...
ncrcat: INFO MD5(wnd_spd) = 74699bb0a72b7f16456badb2c995f1a1
ncrcat: INFO MD5 digests of RAM and disk contents for wnd_spd agree
@end example
Regardless of the debugging level, an error is returned when the digests
of the variable read from the source file and from the output file
disagree.  

These rules may further evolve as @acronym{NCO} pays more attention to 
data integrity. 
We welcome feedback and suggestions from users.

@html
<a name="bfr_sz_hnt"></a> <!-- http://nco.sf.net/nco.html#bfr_sz_hnt -->
<a name="bfr"></a> <!-- http://nco.sf.net/nco.html#bfr -->
<a name="buffer"></a> <!-- http://nco.sf.net/nco.html#buffer -->
@end html
@node Buffer sizes, RAM disks, MD5 digests, Shared features
@section Buffer sizes
@cindex @code{--bfr_sz_hnt}
@cindex Buffer sizes
@cindex File buffers
@cindex @command{stat() system call}
@cindex I/O block size
@cindex System calls
@cartouche
Availability: All operators@*
Short options: @*
Long options: @samp{--bfr_sz_hnt}, @samp{--buffer_size_hint}@*  
@end cartouche

As of @acronym{NCO} version 4.2.0 (May, 2012), @acronym{NCO} 
allows the user to request specific buffer sizes to allocate for reading 
and writing files.
This buffer size determines how many system calls the netCDF layer must
invoke to read and write files.
By default, netCDF uses the preferred I/O block size returned as the
@samp{st_blksize} member of the @samp{stat} structure returned by the
@command{stat()} system call
@footnote{
On modern Linux systems the block size defaults to @w{8192 B}.
The GLADE filesystem at NCAR has a block size of @w{512 kB}.}.
Otherwise, netCDF uses twice the system pagesize.
Larger sizes can increase access speed by reducing the number of 
system calls netCDF makes to read/write data from/to disk.
Because netCDF cannot guarantee the buffer size request will be met, the 
actual buffer size granted by the system is printed as an INFO
statement. 
@example
# Request 2 MB file buffer instead of default 8 kB buffer
> ncks -O -D 3 --bfr_sz=2097152 ~/nco/data/in.nc ~/foo.nc
...
ncks: INFO nc__open() will request file buffer size = 2097152 bytes
ncks: INFO nc__open() opened file with buffer size = 2097152 bytes
...
@end example

@html
<a name="ram_all"></a> <!-- http://nco.sf.net/nco.html#ram_all -->
<a name="ram"></a> <!-- http://nco.sf.net/nco.html#ram -->
<a name="diskless"></a> <!-- http://nco.sf.net/nco.html#diskless -->
@end html
@node RAM disks, Unbuffered I/O, Buffer sizes, Shared features
@section RAM disks
@cindex @code{--ram_all}
@cindex @code{--create_ram}
@cindex @code{--open_ram}
@cindex @code{--diskless_all}
@cindex @acronym{RAM} disks
@cindex @acronym{RAM} files
@cindex @code{NC_DISKLESS}
@cindex diskless files
@cindex memory requirements
@cindex memory available
@cindex @acronym{RAM}
@cindex swap space
@cindex peak memory usage
@cartouche
Availability: All operators (works with netCDF3 files only)@*
Short options: @*
Long options: @samp{--ram_all}, @samp{--create_ram}, @samp{--open_ram},
@samp{--diskless_all}@*   
@end cartouche

As of @acronym{NCO} version 4.2.1 (August, 2012), @acronym{NCO} supports
the use of diskless files, aka @acronym{RAM} disks, for access and
creation of netCDF3 files (these options have no effect on netCDF4 files). 
Two independent switches, @samp{--open_ram} and @samp{--create_ram},
control this feature. 
Before describing the specifics of these switches, we describe why many 
@acronym{NCO} operations will not benefit from them.
Essentially, reading/writing from/to @acronym{RAM} rather than disk only hastens 
the task when reads/writes to disk are avoided.
Most @acronym{NCO} operations are simple enough that they require a
single read-from/write-to disk for every block of input/output. 
Diskless access does not change this, but it does add an extra
read-from/write-to @acronym{RAM}. 
However this extra @acronym{RAM} write/read does avoid contention for limited
system resources like disk-head access.
Operators which may benefit from @acronym{RAM} disks include
@command{ncwa}, which may need to read weighting variables multiple
times, the multi-file  operators @command{ncra}, @command{ncrcat}, and
@command{ncecat}, which may try to write output at least once per input
file, and @command{ncap2} scripts which may be arbitrarily long and
convoluted.  

The @samp{--open_ram} switch causes input files to copied to @acronym{RAM} when
opened. 
All further metadata and data access occurs in @acronym{RAM} and thus avoids
access time delays caused by disk-head movement.
Usually input data is read at most once so it is unlikely that
requesting input files be stored in @acronym{RAM} will save much time.
The likeliest exceptions are files that are accessed numerous times,
such as those repeatedly analyzed by @command{ncap2}. 

Invoking @samp{--open_ram}, @samp{--ram_all}, or @samp{--diskless_all}
uses much more system memory.
To copy the input file to @acronym{RAM} increases the sustained
memory use by exactly the on-disk filesize of the input file, i.e.,
@math{MS += FT}.
For large input files this can be a huge memory burden that starves
the rest of the @acronym{NCO} analysis of sufficient @acronym{RAM}.
To be safe, use @samp{--open_ram}, @samp{--ram_all}, or
@samp{--diskless_all} only on files that are much (say at least a factor
of four) smaller than your available system @acronym{RAM}.
See @ref{Memory Requirements} for further details. 

@cindex @acronym{RAM} variables
The @samp{--create_ram} switch causes output files to be created in
@acronym{RAM}, rather than on disk. 
These files are copied to disk only when closed, i.e., when the
operator completes.
Creating files in @acronym{RAM} may save time, especially with
@command{ncap2} computations that are iterative, e.g., loops, and for
multi-file operators that write output every record (timestep) or file. 
@acronym{RAM} files provide many of the same benefits as @acronym{RAM}
variables in such cases (@pxref{RAM variables}). 

Two switches, @samp{--ram_all} and @samp{--diskless_all}, are convenient
shortcuts for specifying both @samp{--create_ram} and @samp{--open_ram}. 
Thus
@example
ncks in.nc out.nc # Default: Open in.nc on disk, write out.nc to disk
ncks --open_ram in.nc out.nc # Open in.nc in RAM, write out.nc to disk
ncks --create_ram in.nc out.nc # Create out.nc in RAM, write to disk
# Open in.nc in RAM, create out.nc in RAM, then write out.nc to disk
ncks --open_ram --create_ram in.nc out.nc
ncks --ram_all in.nc out.nc # Same as above
ncks --diskless_all in.nc out.nc # Same as above
@end example

It is straightforward to demonstrate the efficacy of @acronym{RAM}
disks.
For @acronym{NASA} we constructed a test that employs @command{ncecat} 
an arbitrary number (set to one hundred thousand) of files that are all 
symbolically linked to the same file. 
Everything is on the local filesystem (not @acronym{DAP}).
@example
@verbatim
# Create symbolic links for benchmark
cd ${DATA}/nco # Do all work here
for idx in {1..99999}; do
  idx_fmt=`printf "%05d" ${idx}`
  /bin/ln -s ${DATA}/nco/LPRM-AMSR_E_L3_D_SOILM3_V002-20120512T111931Z_20020619.nc \
             ${DATA}/nco/${idx_fmt}.nc
done
# Benchmark time to ncecat one hundred thousand files
time ncecat --create_ram -O -u time -v ts -d Latitude,40.0 \ 
 -d Longitude,-105.0 -p ${DATA}/nco -n 99999,5,1 00001.nc ~/foo.nc
@end verbatim
@end example
Run normally on a laptop in 201303, this completes in @w{21 seconds}.
The @samp{--create_ram} reduces the elapsed time to @w{9 seconds}.
Some of this speed may be due to using symlinks and caching.
However, the efficacy of @samp{--create_ram} is clear.
Placing the output file in @acronym{RAM} avoids thousands of disk
writes.
It is not unreasonable to for @acronym{NCO} to process a million files 
like this in a few minutes. 
However, there is no substitute for benchmarking with real files.

@cindex temporary output files
@cindex temporary files
@cindex @code{--no_tmp_fl}
A completely independent way to reduce time spent writing files is 
to refrain from writing temporary output files.
This is accomplished with the @samp{--no_tmp_fl} switch 
(@pxref{Temporary Output Files}).

@html
<a name="share_all"></a> <!-- http://nco.sf.net/nco.html#share_all -->
<a name="share"></a> <!-- http://nco.sf.net/nco.html#share -->
<a name="create_share"></a> <!-- http://nco.sf.net/nco.html#create_share -->
<a name="open_share"></a> <!-- http://nco.sf.net/nco.html#open_share -->
<a name="unbuffered_io"></a> <!-- http://nco.sf.net/nco.html#unbuffered_io -->
<a name="unbuffered"></a> <!-- http://nco.sf.net/nco.html#unbuffered -->
<a name="uio"></a> <!-- http://nco.sf.net/nco.html#uio -->
@end html
@node Unbuffered I/O, Packed data, RAM disks, Shared features
@section Unbuffered I/O
@cindex @code{--share_all}
@cindex @code{--create_share}
@cindex @code{--open_share}
@cindex @code{--unbuffered_io}
@cindex @code{--uio}
@cindex unbuffered I/O
@cindex @code{NC_SHARE}
@cindex concurrent access
@cindex shared access
@cindex caching
@cartouche
Availability: All operators (works on netCDF3 files only)@*
Short options: @*
Long options: @samp{--share_all}, @samp{--create_share},
@samp{--open_share}, @samp{--unbuffered_io}, @samp{--uio}@*   
@end cartouche

As of @acronym{NCO} version 4.9.4 (July, 2020), @acronym{NCO} supports
unbuffered I/O with netCDF3 files when requested with the
@samp{--unbuffered_io} flag, or its synonyms @samp{--uio} or
@samp{--share_all}. 
(Note that these options work only with netCDF3 files and have no
affect on netCDF4 files). 
These flags turn-off the default I/O buffering mode for both newly
created and existing datasets.
For finer-grained control, use the @code{--create_share} switch to
request unbuffered I/O only for newly created datasets, and the
@code{--open_share} switch to request unbuffered I/O only for
existing datasets.
Typically these options only significantly reduce throughput time
when large record variables are written or read.
Normal I/O buffering copies the data to be read/written into an
intermediate buffer in order to avoid numerous small reads/writes.
Unbuffered I/O avoids this intermediate step and can therefore
execute (sometimes much) faster when read/write lengths are large.

@html
<a name="pck"></a> <!-- http://nco.sf.net/nco.html#pck -->
<a name="pack"></a> <!-- http://nco.sf.net/nco.html#pack -->
@end html
@node Packed data, Operation Types, Unbuffered I/O, Shared features
@section Packed data
@cindex packing
@cindex unpacking
@cindex @code{add_offset}
@cindex @code{scale_factor}
@cindex @code{missing_value}
@cindex @code{_FillValue}
@cindex @command{pack(x)}
@cindex @command{unpack(x)}
@cindex @code{--hdf_upk}
@cindex @code{--hdf_unpack}
@cartouche
Availability: @command{ncap2}, @command{ncbo}, @command{nces},
@command{ncflint}, @command{ncpdq}, @command{ncra}, @command{ncwa}@* 
Short options: None@*
Long options: @samp{--hdf_upk}, @samp{--hdf_unpack}@*
@end cartouche

The phrase @dfn{packed data} refers to data which are stored in the
standard netCDF3 lossy linear packing format.
See @ref{ncks netCDF Kitchen Sink} for a description of deflation, a 
lossless compression technique available with netCDF4 only.
Packed data may be deflated to save additional space.

@unnumberedsubsec Standard Packing Algorithm
@dfn{Packing}
The standard netCDF linear packing algorithm (described
@uref{http://www.unidata.ucar.edu/software/netcdf/docs/netcdf/Attribute-Conventions.html, here})
produces packed data with the same dynamic range as the original but
which requires no more than half the space to store.
@acronym{NCO} will always use this algorithm for packing.
Like all packing algorithms, linear packing is @emph{lossy}.
Just how lossy depends on the values themselves, especially their range.
The packed variable is stored (usually) as type @code{NC_SHORT}
with the two attributes required to unpack the variable,
@code{scale_factor} and @code{add_offset}, stored at the original
(unpacked) precision of the variable
@footnote{Although not a part of the standard, @acronym{NCO} enforces
the policy that the @code{_FillValue} attribute, if any, of a packed
variable is also stored at the original precision.}.
Let @var{min} and @var{max} be the minimum and maximum values 
@w{of @var{x}.} 
@tex
$$
\rm
\eqalign{\hbox{scale\_factor} &= (\hbox{max}-\hbox{min})/\hbox{ndrv}\cr
\hbox{add\_offset} &= (\hbox{min}+\hbox{max})/2\cr
\hbox{pck} &= (\hbox{upk}-\hbox{add\_offset})/\hbox{scale\_factor}\cr
 &= {\hbox{ndrv}\times[\hbox{upk}-(\hbox{min}+\hbox{max})/2]\over\hbox{max}-\hbox{min}}\cr}
$$
@end tex
@ifnottex
@sp 1
@var{scale_factor} = (@var{max}-@var{min})/@var{ndrv}@*
@var{add_offset} = 0.5*(@var{min}+@var{max})@*
@var{pck} = (@var{upk}-@var{add_offset})/@var{scale_factor} = (@var{upk}-0.5*(@var{min}+@var{max}))*@var{ndrv}/(@var{max}-@var{min})@* 
@sp 1
@end ifnottex
where @var{ndrv} is the number of discrete representable values for
given type of packed variable.
The theoretical maximum value for @var{ndrv} is two raised to the
number of bits used to store the packed variable.
Thus if the variable is packed into type @code{NC_SHORT}, a two-byte
datatype, then there are at most @math{2^{16} = 65536} distinct values
representable.
In practice, the number of discretely representible values is taken
to be two less than the theoretical maximum.
This leaves space for a missing value and solves potential problems with
rounding that may occur during the unpacking of the variable.
Thus for @code{NC_SHORT}, @math{ndrv = 65536 - 2 = 65534}.
Less often, the variable may be packed into type @code{NC_CHAR}, 
where @math{ndrv = 2^{8} - 2 = 256 - 2 = 254}, or type @code{NC_INT} where
where @math{ndrv = 2^{32} - 2 = 4294967295 - 2 = 4294967293}.
One useful feature of the (lossy) netCDF packing algorithm is that 
lossless packing algorithms perform well on top of it. 

@html
<a name="upk"></a> <!-- http://nco.sf.net/nco.html#upk -->
<a name="unpack"></a> <!-- http://nco.sf.net/nco.html#unpack -->
@end html
@unnumberedsubsec Standard (Default) Unpacking Algorithm
@dfn{Unpacking}
The unpacking algorithm depends on the presence of two attributes,
@code{scale_factor} and @code{add_offset}.
If @code{scale_factor} is present for a variable, the data are
multiplied by the value @var{scale_factor} after the data are read.
If @code{add_offset} is present for a variable, then the
@var{add_offset} value is added to the data after the data are read.
If both @code{scale_factor} and @code{add_offset} attributes are
present, the data are first scaled by @var{scale_factor} before the
offset @var{add_offset} is added.   
@tex
$$
\rm
\eqalign{\hbox{upk} &= \hbox{scale\_factor}\times\hbox{pck} + \hbox{add\_offset}\cr 
&= {\hbox{pck}\times(\hbox{max}-\hbox{min})\over\hbox{ndrv}} + {\hbox{min}+\hbox{max}\over2}\cr} 
$$
@end tex
@ifnottex
@sp 1
@var{upk} = @var{scale_factor}*@var{pck} + @var{add_offset} = (@var{max}-@var{min})*@var{pck}/@var{ndrv} + 0.5*(@var{min}+@var{max})@*
@sp 1
@end ifnottex
@acronym{NCO} will use this algorithm for unpacking unless told
otherwise as described below.
When @code{scale_factor} and @code{add_offset} are used for packing, the
associated variable (containing the packed data) is typically of type
@code{byte} or @code{short}, whereas the unpacked values are intended to
be of type @code{int}, @code{float}, or @code{double}. 
An attribute's @code{scale_factor} and @code{add_offset} and
@code{_FillValue}, if any, should all be of the type intended for the
unpacked data, i.e., @code{int}, @code{float} or @code{double}. 

@unnumberedsubsec Non-Standard Packing and Unpacking Algorithms
@html
<a name="hdf_upk"></a> <!-- http://nco.sf.net/nco.html#hdf_upk -->
<a name="hdf_unpack"></a> <!-- http://nco.sf.net/nco.html#hdf_unpack -->
@end html
@cindex interoperability
@cindex @acronym{HDF} unpacking
Many (most?) files originally written in @acronym{HDF4} format use
poorly documented packing/unpacking algorithms that are incompatible and
easily confused with the netCDF packing algorithm described above.   
The unpacking component of the ``conventional'' @acronym{HDF} algorithm
(described @uref{http://www.hdfgroup.org/HDF5/doc/UG/UG_frame10Datasets.html, here}
and in Section 3.10.6 of the @acronym{HDF4} Users Guide
@uref{http://www.hdfgroup.org/release4/doc/UsrGuide_html/UG_PDF.pdf,
here}, 
and in the @acronym{FAQ} for @acronym{MODIS} @acronym{MOD08} data
@uref{http://modis-atmos.gsfc.nasa.gov/MOD08_D3/faq.html, here}) 
is
@tex
$$
\rm
\hbox{upk} = \hbox{scale\_factor}\times(\hbox{pck} - \hbox{add\_offset})
$$
@end tex
@ifnottex
@sp 1
@var{upk} = @var{scale_factor}*(@var{pck} - @var{add_offset})@*
@sp 1
@end ifnottex

The unpacking component of the @acronym{HDF} algorithm employed
for @acronym{MODIS} @acronym{MOD13} data
is
@tex
$$
\rm
\hbox{upk} = (\hbox{pck} - \hbox{add\_offset})/\hbox{scale\_factor}
$$
@end tex
@ifnottex
@sp 1
@var{upk} = (@var{pck} - @var{add_offset})/@var{scale_factor}@*
@sp 1
@end ifnottex

The unpacking component of the @acronym{HDF} algorithm employed
for @acronym{MODIS} @acronym{MOD04} data is the same as the netCDF
algorithm.

@ignore
@acronym{HDF5} only implements D-scaling, aka, decimal-scaling
bit-packing. 
D-Scaling uses @code{add_offset} as a minimum data value, and
@code{scale_factor} as the integer power @w{of 10} by which the
@code{add_offset} corrected data are divided before storage.
@end ignore
Confusingly, the (incompatible) netCDF and @acronym{HDF} algorithms both 
store their parameters in attributes with the same names
(@code{scale_factor} and @code{add_offset}).
Data packed with one algorithm should never be unpacked with the other;
doing so will result in incorrect answers.
Unfortunately, few users are aware that their datasets may be packed,
and fewer know the details of the packing algorithm employed.
This is what we in the ``bizness'' call an @dfn{interoperability} issue
because it hampers data analysis performed on heterogeneous systems.

As described below, @acronym{NCO} automatically unpacks data before
performing arithmetic.
This automatic unpacking occurs silently since there is usually no
reason to bother users with these details. 
There is as yet no generic way for @acronym{NCO} to know which
packing convention was used, so @acronym{NCO} @emph{assumes} the netCDF 
convention was used. 
@acronym{NCO} uses the same convention for unpacking unless explicitly
told otherwise with the @samp{--hdf_upk} (also @samp{--hdf_unpack})
switch. 
Until and unless a method of automatically detecting the packing method 
is devised, it must remain the user's responsibility to tell
@acronym{NCO} when to use the @acronym{HDF} convention instead of the
netCDF convention to unpack. 

If your data originally came from an @acronym{HDF} file (e.g.,
@acronym{NASA} @acronym{EOS}) then it was likely packed with the
@acronym{HDF} convention and must be unpacked with the same convention.
Our recommendation is to only request @acronym{HDF} unpacking when you 
are certain. 
Most packed datasets encountered by @acronym{NCO} will have used the
netCDF convention.
Those that were not will hopefully produce noticeably weird values when
unpacked by the wrong algorithm.
Before or after panicking, treat this as a clue to re-try your commands
with the @samp{--hdf_upk} switch.
See @ref{ncpdq netCDF Permute Dimensions Quickly} for an easy technique
to unpack data packed with the @acronym{HDF} convention, and then
re-pack it with the netCDF convention.

@unnumberedsubsec Handling of Packed Data by Other Operators
All @acronym{NCO} arithmetic operators understand packed data.
The operators automatically unpack any packed variable in the input 
file which will be arithmetically processed.
For example, @command{ncra} unpacks all record variables, 
and @command{ncwa} unpacks all variable which contain a dimension to 
be averaged.
These variables are stored unpacked in the output file.

On the other hand, arithmetic operators do not unpack non-processed
variables. 
For example, @command{ncra} leaves all non-record variables packed, 
and @command{ncwa} leaves packed all variables lacking an averaged
dimension.  
These variables (called fixed variables) are passed unaltered from the
input to the output file. 
Hence fixed variables which are packed in input files remain packed in
output files.
Completely packing and unpacking files is easily accomplished with
@command{ncpdq} (@pxref{ncpdq netCDF Permute Dimensions Quickly}).
Pack and unpack individual variables with @command{ncpdq} and the
@command{ncap2} @command{pack()} and @command{unpack()} functions
(@pxref{Methods and functions}).

@html
<a name="op_typ"></a> <!-- http://nco.sf.net/nco.html#op_typ -->
@end html
@node Operation Types, Type Conversion, Packed data, Shared features
@section Operation Types
@cindex operation types
@cindex @code{avg}
@cindex @code{sqravg}
@cindex @code{avgsqr}
@cindex @code{min}
@cindex @code{max}
@cindex @code{mabs}
@cindex @code{mebs}
@cindex @code{mibs}
@cindex @code{rmssdn}
@cindex @code{rms}
@cindex @code{tabs}
@cindex @code{ttl}
@cindex @code{sqrt}
@cindex average
@cindex mean
@cindex total
@cindex minimum
@cindex maximum
@cindex root-mean-square
@cindex standard deviation
@cindex variance
@cindex @code{-y @var{op_typ}}
@cindex @code{--operation @var{op_typ}}
@cindex @code{--op_typ @var{op_typ}}
@cartouche
Availability: @command{ncap2}, @command{ncra}, @command{nces}, @command{ncwa}@*
Short options: @samp{-y}@*
Long options: @samp{--operation}, @samp{--op_typ}@*
@end cartouche
@noindent
The @samp{-y @var{op_typ}} switch allows specification of many different
types of operations 
Set @var{op_typ} to the abbreviated key for the corresponding operation:
@table @code
@item avg
Mean value
@item sqravg
Square of the mean
@item avgsqr
Mean of sum of squares
@item max
Maximum value
@item min
Minimum value
@item mabs
Maximum absolute value
@item mebs
Mean absolute value
@item mibs
Minimum absolute value
@item rms
Root-mean-square (normalized by @var{N})
@item rmssdn
Root-mean square (normalized by @var{N-1})
@item sqrt
Square root of the mean
@item tabs
Sum of absolute values
@item ttl
Sum of values
@end table
@noindent
@cindex coordinate variable 
@acronym{NCO} assumes coordinate variables represent grid axes, e.g.,
longitude. 
The only rank-reduction which makes sense for coordinate variables
is averaging.
Hence @acronym{NCO} implements the operation type requested with
@samp{-y} on all non-coordinate variables, not on coordinate variables.  
When an operation requires a coordinate variable to be reduced in
rank, i.e., from one dimension to a scalar or from one dimension to
a degenerate (single value) array, then @acronym{NCO} 
@emph{always averages} the coordinate variable regardless of the
arithmetic operation type performed on the non-coordinate variables.

The mathematical definition of each arithmetic operation is given below. 
@xref{ncwa netCDF Weighted Averager}, for additional information on
masks and normalization.
If an operation type is not specified with @samp{-y} then the operator
performs an arithmetic average by default.
Averaging is described first so the terminology for the other operations
is familiar. 

@ifhtml
@cartouche
@html
<p><b>Note for HTML users</b>: 
<br>The definition of mathematical operations involving rank reduction
(e.g., averaging) relies heavily on mathematical expressions which
cannot easily be represented in HTML.  
<b>See the <a href="./nco.pdf">printed manual</a> for much more detailed
and complete documentation of this subject.</b>
@end html
@end cartouche
@end ifhtml
@ifnothtml
@ifinfo
@emph{Note for Info users}: 
The definition of mathematical operations involving rank reduction
(e.g., averaging) relies heavily on mathematical expressions which
cannot be easily represented in Info.
@emph{See the @uref{./nco.pdf, printed manual} for much more detailed and
complete documentation of this subject.}
@end ifinfo
@end ifnothtml

@tex
The masked, weighted average of a variable $\xxx$ can be generally
represented as
$$
\bar \xxx_{\jjj} = {\sum_{\idx=1}^{\idx=\lmnnbr} \mssflg_{\idx}
\mskflg_{\idx} \wgt_{\idx} \xxx_{\idx} \over
\sum_{\idx=1}^{\idx=\lmnnbr} \mssflg_{\idx} \mskflg_{\idx} \wgt_{\idx}}  
$$
where $\bar \xxx_{\jjj}$ is the $\jjj$'th element of the output
hyperslab, $\xxx_{\idx}$ is the $\idx$'th element of the input
hyperslab, $\mssflg_{\idx}$ is~1 unless $\xxx_{\idx}$ equals the missing  
value, $\mskflg_{\idx}$ is~1 unless $\xxx_{\idx}$ is masked, and
$\wgt_{\idx}$ is the weight.  
This formiddable formula represents a simple weighted average
whose bells and whistles are all explained below. 
It is not too early to note, however, that when 
$\mssflg_{\idx} = \mskflg_{\idx} = \wgt_{\idx} = 1$, the 
generic averaging expression above reduces to a simple arithmetic
average.  
Furthermore, $\mskflg_{\idx} = \wgt_{\idx} = 1$ for all operators
except @command{ncwa}.
These variables are included in the discussion below for completeness,
and for possible future use in other operators. 

The size $\outnbr$ of the output hyperslab for a given variable is the
product of all the dimensions of the input variable which are not
averaged over. 
The size $\lmnnbr$ of the input hyperslab contributing to each 
$\bar \xxx_{\jjj}$ is simply the product of the sizes of all dimensions
which are averaged over (i.e., dimensions specified with @samp{-a}).  
Thus $\lmnnbr$ is the number of input elements which @emph{potentially} 
contribute to each output element.
An input element $\xxx_{\idx}$ contributes to the output element
$\xxx_{\jjj}$ except in two conditions:  
@cindex missing values
@enumerate
@item $\xxx_{\idx}$ equals the @var{missing value} 
(@pxref{Missing Values}) for the variable. 
@item $\xxx_{\idx}$ is located at a point where the mask condition 
(@pxref{Mask condition}) is false.
@end enumerate
Points $\xxx_{\idx}$ in either of these two categories do not contribute 
to $\xxx_{\jjj}$---they are ignored.
We now define these criteria more rigorously.

Each~$\xxx_{\idx}$ has an associated Boolean weight~$\mssflg_{\idx}$
whose value is~0 or~1 (false or true). 
The value of~$\mssflg_{\idx}$ is~1 (true) unless $\xxx_{\idx}$ equals
the @var{missing value} (@pxref{Missing Values}) for the variable. 
Thus, for a variable with no @code{_FillValue} attribute,
$\mssflg_{\idx}$~is always~1.
All @acronym{NCO} arithmetic operators (@command{ncbo},
@command{ncra}, @command{nces}, @command{ncflint}, @command{ncwa}) treat  
missing values analogously. 

Besides (weighted) averaging, @command{ncwa}, @command{ncra}, and
@command{nces} also compute some common non-linear operations which may
be specified with the @samp{-y} switch (@pxref{Operation Types}).
The other rank-reducing operations are simple variations of the generic
weighted mean described above.
The total value of~$\xxx$ (@code{-y ttl}) is 
$$
\bar \xxx_{\jjj} = \sum_{\idx=1}^{\idx=\lmnnbr} \mssflg_{\idx}
\mskflg_{\idx} \wgt_{\idx} \xxx_{\idx}  
$$
@cindex @code{-N}
@cindex @code{numerator}
Note that the total is the same as the numerator of the mean
of~$\xxx$, and may also be obtained in @command{ncwa} by using the
@samp{-N} switch (@pxref{ncwa netCDF Weighted Averager}).

The minimum value of~$\xxx$ (@code{-y min}) is 
$$
\bar \xxx_{\jjj} = \min [ \mssflg_{1} \mskflg_{1} \wgt_{1} \xxx_{1},
\mssflg_{2} \mskflg_{2} \wgt_{2} \xxx_{2}, \ldots, \mssflg_{\lmnnbr} 
\mskflg_{\lmnnbr} \wgt_{\lmnnbr} \xxx_{\lmnnbr} ] 
$$
Analogously, the maximum value of~$\xxx$ (@code{-y max}) is 
$$
\bar \xxx_{\jjj} = \max [ \mssflg_{1} \mskflg_{1} \wgt_{1} \xxx_{1},
\mssflg_{2} \mskflg_{2} \wgt_{2} \xxx_{2}, \ldots, \mssflg_{\lmnnbr} 
\mskflg_{\lmnnbr} \wgt_{\lmnnbr} \xxx_{\lmnnbr} ] 
$$
Thus the minima and maxima are determined after any weights are applied.

The total absolute value of~$\xxx$ (@code{-y tabs}) is 
$$
\bar \xxx_{\jjj} = \sum_{\idx=1}^{\idx=\lmnnbr} \mssflg_{\idx}
\mskflg_{\idx} \wgt_{\idx} |\xxx_{\idx}|
$$

The minimum absolute value of~$\xxx$ (@code{-y mibs}) is 
$$
\bar \xxx_{\jjj} = \min [ \mssflg_{1} \mskflg_{1} \wgt_{1} |\xxx_{1}|,
\mssflg_{2} \mskflg_{2} \wgt_{2} |\xxx_{2}|, \ldots, \mssflg_{\lmnnbr} 
\mskflg_{\lmnnbr} \wgt_{\lmnnbr} |\xxx_{\lmnnbr}| ] 
$$
Analogously, the maximum absolute value of~$\xxx$ (@code{-y mabs}) is 
$$
\bar \xxx_{\jjj} = \max [ \mssflg_{1} \mskflg_{1} \wgt_{1} |\xxx_{1}|,
\mssflg_{2} \mskflg_{2} \wgt_{2} |\xxx_{2}|, \ldots, \mssflg_{\lmnnbr} 
\mskflg_{\lmnnbr} \wgt_{\lmnnbr} |\xxx_{\lmnnbr}| ] 
$$
Thus the minimum and maximum absolute values are determined after any weights are applied.
The mean absolute value of~$\xxx$ (@code{-y mebs}) is 
$$
\bar \xxx_{\jjj} = {\sum_{\idx=1}^{\idx=\lmnnbr}
\mssflg_{\idx} \mskflg_{\idx} \wgt_{\idx} |\xxx_{\idx}| \over
\sum_{\idx=1}^{\idx=\lmnnbr} \mssflg_{\idx} \mskflg_{\idx} \wgt_{\idx}} 
$$

The square of the mean value of~$\xxx$ (@code{-y sqravg}) is 
$$
\bar \xxx_{\jjj} = \left( {\sum_{\idx=1}^{\idx=\lmnnbr}
\mssflg_{\idx} \mskflg_{\idx} \wgt_{\idx} \xxx_{\idx} \over
\sum_{\idx=1}^{\idx=\lmnnbr} \mssflg_{\idx} \mskflg_{\idx} \wgt_{\idx}} 
\right)^{2}    
$$
The mean of the sum of squares of~$\xxx$ (@code{-y avgsqr}) is 
$$
\bar \xxx_{\jjj} = {\sum_{\idx=1}^{\idx=\lmnnbr} \mssflg_{\idx}
\mskflg_{\idx} \wgt_{\idx} \xxx^{2}_{\idx} \over
\sum_{\idx=1}^{\idx=\lmnnbr} \mssflg_{\idx} \mskflg_{\idx} \wgt_{\idx}} 
$$
If $\xxx$ represents a deviation from the mean of another variable,
$\xxx_{\idx} = \yyy_{\idx} - \bar{\yyy}$ (possibly created by
@command{ncbo} in a previous step), then applying @code{avgsqr} to
$\xxx$ computes the approximate variance of $\yyy$. 
Computing the true variance of~$\yyy$ requires subtracting~1 from the
denominator, discussed below.
For a large sample size however, the two results will be nearly
indistinguishable. 

The root mean square of~$\xxx$ (@code{-y rms}) is 
$$
\bar \xxx_{\jjj} = \sqrt{ {\sum_{\idx=1}^{\idx=\lmnnbr} \mssflg_{\idx}
\mskflg_{\idx} \wgt_{\idx} x^{2}_{\idx} \over
\sum_{\idx=1}^{\idx=\lmnnbr} \mssflg_{\idx} \mskflg_{\idx} \wgt_{\idx}}} 
$$
Thus @code{rms} simply computes the squareroot of the quantity computed
by @code{avgsqr}.

The root mean square of~$\xxx$ with standard-deviation-like
normalization (@code{-y rmssdn}) is implemented as follows.
When weights are not specified, this function is the same as the root
mean square of~$\xxx$ except one is subtracted from the sum in the
denominator 
$$
\bar \xxx_{\jjj} = \sqrt{ {\sum_{\idx=1}^{\idx=\lmnnbr} \mssflg_{\idx} 
\mskflg_{\idx} x^{2}_{\idx} \over -1 + \sum_{\idx=1}^{\idx=\lmnnbr}
\mssflg_{\idx} \mskflg_{\idx}} } 
$$
If $\xxx$ represents the deviation from the mean of another variable, 
$\xxx_{\idx} = \yyy_{\idx} - \bar{\yyy}$, then applying @code{rmssdn} to
$\xxx$ computes the standard deviation of~$\yyy$.
In this case the $-1$ in the denominator compensates for the degree of
freedom already used in computing $\bar{\yyy}$ in the numerator.
Consult a statistics book for more details.

When weights are specified it is unclear how to compensate for this
extra degree of freedom.
Weighting the numerator and denominator of the above by $\wgt_{\idx}$
and subtracting one from the denominator is only appropriate when all
the weights are~1.0.
When the weights are arbitrary (e.g., Gaussian weights), subtracting one 
from the sum in the denominator does not necessarily remove one degree
of freedom. 
Therefore when @code{-y rmssdn} is requested and weights are specified,
@command{ncwa} actually implements the @code{rms} procedure.
@command{nces} and @command{ncra}, which do not allow weights to be
specified, always implement the @code{rmssdn} procedure when asked.

@ignore
20130827: Fedora Core 19 (FC19) broke here with "./nco.texi:6394: Missing dollarsign inserted."
Ubuntu always built nco.texi fine
Adding a dollarsign character right here breaks Ubuntu builds too
Hence I must carefully spell-out the word dollarsign instead
20130829: Making many smaller TeX environments does not solve problem
20130910: Using latest texinfo.tex from GNU does not solve problem
20130910: Karl Berry solved problem by fixing bug in texinfo.tex
Bug was triggered in Fedora by apostrophe in "User's Guide" (manual title)
Bug not present in texinfo.tex version 2008-04-18.10 (used by Ubuntu 13.04)
Bug     present in texinfo.tex version 2013-02-01.11 (used by FC19)
Bug just fixed  in texinfo.tex version 2013-09-11 (committed by Karl)
nco/autobld/texinfo.tex now contains fixed version
Breakage always occurs near here
@end ignore
The square root of the mean of~$\xxx$ (@code{-y sqrt}) is 
$$
\bar \xxx_{\jjj} = \sqrt{ {\sum_{\idx=1}^{\idx=\lmnnbr} \mssflg_{\idx}
\mskflg_{\idx} \wgt_{\idx} \xxx_{\idx} \over
\sum_{\idx=1}^{\idx=\lmnnbr} \mssflg_{\idx} \mskflg_{\idx} \wgt_{\idx}}}  
$$
@end tex
The definitions of some of these operations are not universally useful.
Mostly they were chosen to facilitate standard statistical
computations within the @acronym{NCO} framework.
We are open to redefining and or adding to the above. 
If you are interested in having other statistical quantities
defined in @acronym{NCO} please contact the @acronym{NCO} project
(@pxref{Help Requests and Bug Reports}).  

@noindent
EXAMPLES

@html
<a name="mabs"></a> <!-- http://nco.sf.net/nco.html#mabs -->
<a name="mebs"></a> <!-- http://nco.sf.net/nco.html#mebs -->
<a name="mibs"></a> <!-- http://nco.sf.net/nco.html#mibs -->
<a name="min"></a> <!-- http://nco.sf.net/nco.html#min -->
<a name="max"></a> <!-- http://nco.sf.net/nco.html#max -->
<a name="rms"></a> <!-- http://nco.sf.net/nco.html#rms -->
@end html
@noindent
Suppose you wish to examine the variable @code{prs_sfc(time,lat,lon)} 
which contains a time series of the surface pressure as a function of
latitude and longitude.
Find the minimum value of @code{prs_sfc} over all dimensions:
@example
ncwa -y min -v prs_sfc in.nc foo.nc 
@end example
@noindent
Find the maximum value of @code{prs_sfc} at each time interval for each
latitude: 
@example
ncwa -y max -v prs_sfc -a lon in.nc foo.nc
@end example
@noindent
Find the root-mean-square value of the time-series of @code{prs_sfc} at
every gridpoint:
@example
ncra -y rms -v prs_sfc in.nc foo.nc
ncwa -y rms -v prs_sfc -a time in.nc foo.nc
@end example
@noindent
The previous two commands give the same answer but @command{ncra} is
preferred because it has a smaller memory footprint.
@cindex degenerate dimension
A dimension of size one is said to be @dfn{degenerate}.
By default, @command{ncra} leaves the (degenerate) @code{time}
dimension in the output file (which is usually useful) whereas
@command{ncwa} removes the @code{time} dimension (unless @samp{-b} is
given).

@noindent
These operations work as expected in multi-file operators.
Suppose that @code{prs_sfc} is stored in multiple timesteps per file
across multiple files, say @file{jan.nc}, @file{feb.nc},
@file{march.nc}.  
We can now find the three month maximum surface pressure at every point.
@example
nces -y max -v prs_sfc jan.nc feb.nc march.nc out.nc
@end example

@html
<a name="standard_deviation"></a> <!-- http://nco.sf.net/nco.html#standard_deviation -->
<a name="stddvn"></a> <!-- http://nco.sf.net/nco.html#stddvn -->
<a name="sdn"></a> <!-- http://nco.sf.net/nco.html#sdn -->
<a name="sdv"></a> <!-- http://nco.sf.net/nco.html#sdv -->
<a name="xmp_sdn"></a> <!-- http://nco.sf.net/nco.html#xmp_sdn -->
@end html
@noindent
@cindex standard deviation
It is possible to use a combination of these operations to compute
the variance and standard deviation of a field stored in a single file
or across multiple files.
The procedure to compute the temporal standard deviation of the surface
pressure at all points in a single file @file{in.nc} involves three
steps. 
@example
ncwa -O -v prs_sfc -a time in.nc out.nc
ncbo -O -v prs_sfc in.nc out.nc out.nc 
ncra -O -y rmssdn out.nc out.nc
@end example
First construct the temporal mean of @code{prs_sfc} in the file
@file{out.nc}.
Next overwrite @file{out.nc} with the anomaly (deviation from the mean).
Finally overwrite @file{out.nc} with the root-mean-square of itself. 
Note the use of @samp{-y rmssdn} (rather than @samp{-y rms}) in the
final step. 
This ensures the standard deviation is correctly normalized by one fewer
than the number of time samples.
The procedure to compute the variance is identical except for the use of 
@samp{-y avgsqr} instead of @samp{-y rmssdn} in the final step.

@command{ncap2} can also compute statistics like standard deviations.
Brute-force implementation of formulae is one option, e.g.,
@example
ncap2 -s 'prs_sfc_sdn=sqrt((prs_sfc-prs_sfc.avg($time)^2). \
      total($time)/($time.size-1))' in.nc out.nc
@end example
The operation may, of course, be broken into multiple steps in order  
to archive intermediate quantities, such as the time-anomalies
@example
ncap2 -s 'prs_sfc_anm=prs_sfc-prs_sfc.avg($time)' \
      -s 'prs_sfc_sdn=sqrt((prs_sfc_anm^2).total($time)/($time.size-1))' \
      in.nc out.nc
@end example

@command{ncap2} supports intrinsic standard deviation functions
(@pxref{Operation Types}) which simplify the above expression to
@example
ncap2 -s 'prs_sfc_sdn=(prs_sfc-prs_sfc.avg($time)).rmssdn($time)' in.nc out.nc
@end example
These instrinsic functions compute the answer quickly and concisely.

The procedure to compute the spatial standard deviation of a field
in a single file @file{in.nc} involves three steps.
@example
ncwa -O -v prs_sfc,gw -a lat,lon -w gw in.nc out.nc
ncbo -O -v prs_sfc,gw in.nc out.nc out.nc
ncwa -O -y rmssdn -v prs_sfc -a lat,lon -w gw out.nc out.nc
@end example
First the spatially weighted (by @samp{-w gw}) mean values
are written to the output file, as are the mean weights.
The initial output file is then overwritten with the gridpoint
deviations from the spatial mean.
It is important that the output file after the second line contain
the original, non-averaged weights.
This will be the case if the weights are named so that @acronym{NCO}
treats them like a coordinate (@pxref{CF Conventions}).
One such name is @code{gw}, and any variable whose name begins with
@code{msk_} (for ``mask'') or @code{wgt_} (for ``weight'') will likewise
be treated as a coordinate, and will be copied (not differenced)
straight from @file{in.nc} to @file{out.nc} in the second step.
When using weights to compute standard deviations one must remember to
include the weights in the initial output files so that they may be used
again in the final step. 
Finally the root-mean-square of the appropriately weighted spatial
deviations is taken.  

No elegant @command{ncap2} solution exists to compute weighted standard
deviations.  
Those brave of heart may try to formulate one.
A general formula should allow weights to have fewer than and variables
to have more than the minimal spatial dimensions (latitude and
longitude). 
@ignore
@c Until 20151229 ncap2 example was FUBAR, and did not handle weights in denominator
@example
ncap2 -s 'prs_sfc_sdn=((gw*(prs_sfc-((gw*prs_sfc).ttl($lat,$lon)/gw.ttl($lat,$lon)))^2).ttl($lat,$lon)/ \ 
      gw.ttl()).sqrt()' in.nc out.nc
@end example
Note how the weight multiplies the variable prior to computing the
the anomalies and the standard deviation.
@end ignore

The procedure to compute the standard deviation of a time-series across
multiple files involves one extra step since all the input must first be
collected into one file. 
@example
ncrcat -O -v tpt in.nc in.nc foo1.nc
ncwa -O -a time foo1.nc foo2.nc
ncbo -O -v tpt foo1.nc foo2.nc foo3.nc
ncra -O -y rmssdn foo3.nc out.nc
@end example
The first step assembles all the data into a single file.
Though this may consume a lot of temporary disk space, it is more or
less required by the @command{ncbo} operation in the third step.

@html
<a name="typ_cnv"></a> <!-- http://nco.sf.net/nco.html#typ_cnv -->
@end html
@node Type Conversion, Batch Mode, Operation Types, Shared features
@section Type Conversion
@cindex type conversion
@cartouche
Availability (automatic type conversion): @command{ncap2}, @command{ncbo}, @command{nces},
@command{ncflint}, @command{ncra}, @command{ncwa}@* 
Short options: None (it's @emph{automatic})@*
Availability (manual type conversion): @command{nces}, @command{ncra}, @command{ncwa}@* 
Short options: None@*
Long options: @samp{--dbl}, @samp{--flt}, @samp{--rth_dbl}, @samp{--rth_flt}@* 
@end cartouche
@cindex promotion
@cindex demotion
@cindex automatic type conversion
@cindex manual type conversion
Type conversion refers to the casting or coercion of one fundamental or
atomic data type to another, e.g., converting @code{NC_SHORT} (two
bytes) to @code{NC_DOUBLE} (eight bytes).  
Type conversion always @dfn{promotes} or @dfn{demotes} the range and/or 
precision of the values a variable can hold.
Type conversion is automatic when the language carries out this
promotion according to an internal set of rules without explicit user 
intervention. 
In contrast, manual type conversion refers to explicit user commands to
change the type of a variable or attribute.
Most type conversion happens automatically, yet there are situations in
which manual type conversion is advantageous.

@menu
* Automatic type conversion::
* Promoting Single-precision to Double::
* Manual type conversion::
@end menu

@node Automatic type conversion, Promoting Single-precision to Double, Type Conversion, Type Conversion
@subsection Automatic type conversion

There are at least two reasons to avoid type conversions.
First, type conversions are expensive since they require creating
(temporary) buffers and casting each element of a variable from its 
storage type to some other type and then, often, converting it back.
Second, a dataset's creator perhaps had a good reason for storing 
data as, say, @code{NC_FLOAT} rather than @code{NC_DOUBLE}.  
In a scientific framework there is no reason to store data with more
precision than the observations merit.
Normally this is single-precision, which guarantees 6--9 digits of
precision. 
Reasons to engage in type conversion include avoiding rounding 
errors and out-of-range limitations of less-precise types.
This is the case with most integers.
Thus @acronym{NCO} defaults to automatically promote integer types to
floating-point when performing lengthy arithmetic, yet @acronym{NCO}
defaults to not promoting single to double-precision floats.

Before discussing the more subtle floating-point issues, we first
examine integer promotion. 
We will show how following parsimonious conversion rules dogmatically
can cause problems, and what @acronym{NCO} does about that. 
That said, there are situations in which implicit conversion of
single- to double-precision is also warranted.
Understanding the narrowness of these situations takes time, and we
hope the reader appreciates the following detailed discussion.

Consider the average of the two @code{NC_SHORT}s @code{17000s} and
@code{17000s}.
A straightforward average without promotion results in garbage since the
intermediate value which holds their sum is also of type @code{NC_SHORT}
and thus overflows on (i.e., cannot represent) values greater than
32,767 
@footnote{
@set flg
@tex
$32767 = 2^{15}-1$
@clear flg
@end tex
@ifinfo
@math{32767 = 2^15-1}
@clear flg
@end ifinfo
@ifset flg
@c texi2html does not like @math{}
32767 = 2^15@minus{}1
@clear flg
@end ifset
}.
There are valid reasons for expecting this operation to succeed and 
the @acronym{NCO} philosophy is to make operators do what you want, not
what is purest.
Thus, unlike C and Fortran, but like many other higher level interpreted
languages, @acronym{NCO} arithmetic operators will perform automatic type
conversion on integers when all the following conditions are met
@footnote{Operators began performing automatic type conversions before
arithmetic in @acronym{NCO} @w{version 1.2}, August, 2000. 
Previous versions never performed unnecessary type conversion for
arithmetic.}: 
@enumerate
@item The requested operation is arithmetic.
This is why type conversion is limited to the operators @command{ncap2}, 
@command{ncbo}, @command{nces}, @command{ncflint}, @command{ncra}, and
@command{ncwa}.   
@item The arithmetic operation could benefit from type conversion.
Operations that could benefit include averaging, summation, or any
``hard'' arithmetic that could overflow or underflow.  
Larger representable sums help avoid overflow, and more precision
helps to avoid underflow.
Type conversion does not benefit searching for minima and maxima
(@samp{-y min}, or @samp{-y max}).
@item The variable on disk is of type @code{NC_BYTE}, @code{NC_CHAR},
@code{NC_SHORT}, or @code{NC_INT}.
Type @code{NC_DOUBLE} is not promoted because there is no type of
higher precision.
Conversion of type @code{NC_FLOAT} is discussed in detail below. 
When it occurs, it follows the same procedure (promotion then
arithmetic then demotion) as conversion of integer types.
@end enumerate

When these criteria are all met, the operator promotes the variable in
question to type @code{NC_DOUBLE}, performs all the arithmetic
operations, casts the @code{NC_DOUBLE} type back to the original type,
and finally writes the result to disk. 
The result written to disk may not be what you expect, because of
incommensurate ranges represented by different types, and because of
(lack of) rounding.
First, continuing the above example, the average (e.g., @samp{-y avg})
of @code{17000s} and @code{17000s} is written to disk as @code{17000s}. 
The type conversion feature of @acronym{NCO} makes this possible since
the arithmetic and intermediate values are stored as @code{NC_DOUBLE}s,
i.e., @code{34000.0d} and only the final result must be represented
as an @code{NC_SHORT}.
Without the type conversion feature of @acronym{NCO}, the average would
have been garbage (albeit predictable garbage near @code{-15768s}). 
Similarly, the total (e.g., @samp{-y ttl}) of @code{17000s} and
@code{17000s} written to disk is garbage (actually @code{-31536s}) since 
the final result (the true total) of @math{34000} is outside the range
of type @code{NC_SHORT}.  

@cindex @code{trunc()}
After arithmetic is computed in double-precision for promoted variables,
the intermediate double-precision values must be demoted to the
variables' original storage type (e.g., from @code{NC_DOUBLE} to
@code{NC_SHORT}). 
@acronym{NCO} has handled this demotion in three ways in its history.
Prior to October, 2011 (version 4.0.8), @acronym{NCO} employed the
@w{C library} truncate function, @code{trunc()}
@footnote{
@cindex C language
The actual type conversions with trunction were handled by intrinsic
type conversion, so the @code{trunc()} function was never explicitly
called, although the results would be the same if it were.}.
Truncation rounds @var{x} to the nearest integer not larger in absolute 
value.
For example, truncation rounds @code{1.0d}, @code{1.5d}, and
@code{1.8d} to the same value, @code{1s}. 
Clearly, truncation does not round floating-point numbers to the nearest
integer! 
Yet truncation is how the @w{C language} performs implicit conversion of 
real numbers to integers.

@cindex Neil Davis
@acronym{NCO} stopped using truncation for demotion when an alert user
(Neil Davis) informed us that this caused a small bias in the packing
algorithm employed by @command{ncpdq}.
This led to @acronym{NCO} adopting rounding functions for demotion.
Rounding functions eliminated the small bias in the packing algorithm.

@findex @code{lround()}. 
From February, 2012 through March, 2013 (versions 4.0.9--4.2.6),
@acronym{NCO} employed the @w{C library} family of rounding functions,
@code{lround()}. 
These functions round @var{x} to the nearest integer, halfway cases away
from zero.
The problem with @code{lround()} is that it always rounds real values
ending in @code{.5} away from zero.
This rounds, for example, @code{1.5d} and @code{2.5d} to @code{2s}
and @code{3s}, respectively.

@findex @code{lrint()}. 
@cindex @acronym{IEEE}
Since April, 2013 (version 4.3.0), @acronym{NCO} has employed the 
other @w{C library} family of rounding functions, @code{lrint()}. 
This algorithm rounds @var{x} to the nearest integer, using the current 
rounding direction.
Halfway cases are rounded to the nearest even integer.
This rounds, for example, both @code{1.5d} and @code{2.5d} to the same
value, @code{2s}, as recommended by the @acronym{IEEE}.
This rounding is symmetric: up half the time, down half the time.
This is the current and hopefully final demotion algorithm employed by  
@acronym{NCO}.

Hence because of automatic conversion, @acronym{NCO} will compute the
average of @code{2s} and @code{3s} in double-precision arithmetic as 
@math{(@code{2.0d} + @code{3.0d})/@code{2.0d}) = @code{2.5d}}.
It then demotes this intermediate result back to @code{NC_SHORT} and
stores it on disk as  
@math{@code{trunc(2.5d)} = @code{2s}} (versions up to 4.0.8), 
@math{@code{lround(2.5d)} = @code{3s}} (versions 4.0.9--4.2.6), and
@math{@code{lrint(2.5d)} = @code{2s}} (versions 4.3.0 and later).

@html
<a name="sp_dp"></a> <!-- http://nco.sf.net/nco.html#sp_dp -->
<a name="dbl"></a> <!-- http://nco.sf.net/nco.html#dbl -->
@end html
@node Promoting Single-precision to Double, Manual type conversion, Automatic type conversion, Type Conversion
@subsection Promoting Single-precision to Double
@cindex promotion
@cindex implicit conversion
@cindex @code{--dbl}
@cindex @code{--rth_dbl}
@cindex @code{--flt}
@cindex @code{--rth_flt}
Promotion of real numbers from single- to double-precision is
fundamental to scientific computing.
When it should occur depends on the precision of the inputs and the
number of operations.
Single-precision (four-byte) numbers contain about seven significant
figures, while double-precision contain about sixteen.
More, err, precisely, the @acronym{IEEE} single-precision representation
gives from @w{6 to 9} significant decimal digits precision
@footnote{According to Wikipedia's summary of @acronym{IEEE} 
@w{standard 754}, ``If a decimal string with at most @w{6 significant} 
digits is converted to @w{@acronym{IEEE} 754} single-precision and then
converted back to the same number of significant decimal, then the final
string should match the original; and if an @w{@acronym{IEEE} 754}
single-precision is converted to a decimal string with at @w{leastn 9}
significant decimal and then converted back to single, then the final
number must match the original''.}. 
And the @acronym{IEEE} double-precision representation
gives from @w{15 to 17} significant decimal digits precision
@footnote{According to Wikipedia's summary of @acronym{IEEE} 
@w{standard 754}, ``If a decimal string with at most @w{15 significant}
digits is converted to @w{@acronym{IEEE} 754} double-precision
representation and then converted back to a string with the same number
of significant digits, then the final string should match the original;
and if an @w{@acronym{IEEE} 754} double precision is converted to a
decimal string with at least @w{17 significant} digits and then
converted back to double, then the final number must match the
original''.}.  
Hence double-precision numbers represent about nine digits more
precision than single-precision numbers.

Given these properties, there are at least two possible arithmetic
conventions for the treatment of real numbers:
@cindex C language
@cindex Fortran
@enumerate
@item Conservative, aka Fortran Convention
Automatic type conversion during arithmetic in the Fortran language is,
by default, performed only when necessary. 
All operands in an operation are converted to the most precise type
involved the operation before the arithmetic operation.
Expressions which involve only single-precision numbers are computed
entirely in single-precision.
Expressions involving mixed precision types are computed in the type
of higher precision.
@acronym{NCO} by default employs the Fortan Convention for promotion.
@item Aggressive, aka C Convention
The @w{C language} is by default much more aggressive (and thus
wasteful) than Fortran, and will always implicitly convert single- 
to double-precision numbers, even when there is no good reason.
All real-number standard @w{C library} functions are double-precision, 
and @w{C programmers} must take extra steps to only utilize single
precision arithmetic.
The high-level interpreted data analysis languages @acronym{IDL},
Matlab, and @acronym{NCL} all adopt the @w{C Convention}.
@end enumerate

@acronym{NCO} does not automatically promote @code{NC_FLOAT} because, in
our judgement, the performance penalty of always doing so would outweigh
the potential benefits. 
The now-classic text ``Numerical Recipes @w{in C}'' discusses this point
under the section ``Implicit Conversion of Float to Double''
@footnote{See @w{page 21} in Section 1.2 of the First edition for this
gem:
@quotation
One does not need much experience in scientific computing to recognize
that the implicit conversion rules are, in fact, sheer madness!
In effect, they make it impossible to write efficient numerical
programs. 
@end quotation
}.
That said, such promotion is warranted in some circumstances.

For example, rounding errors can accumulate to worrisome levels during
arithmetic performed on large arrays of single-precision floats. 
This use-case occurs often in geoscientific studies of climate where
thousands-to-millions of gridpoints may contribute to a single average.
If the inputs are all single-precision, then so should be the output.
However the intermediate results where running sums are accumulated may  
suffer from too much rounding or from underflow unless computed in
double-precision. 

The order of operations matters to floating-point math even when the
analytic expressions are equal. 
Cautious users feel disquieted when results from equally valid analyses
differ in the final bits instead of agreeing bit-for-bit.
For example, averaging arrays in multiple stages produces different
answers than averaging them in one step. 
This is easily seen in the computation of ensemble averages by two
different methods.
The @acronym{NCO} test file @file{in.nc} contains single- and
double-precision representations of the same temperature timeseries as 
@code{tpt_flt} and @code{tpt_dbl}.  
Pretend each datapoint in this timeseries represents a monthly-mean
temperature. 
We will mimic the derivation of a fifteen-year ensemble-mean January
temperature by concatenating the input file five times, and then
averaging the datapoints representing January two different ways.
In @w{Method 1} we derive the 15-year ensemble January average in two 
steps, as the average of three five-year averages.
This method is naturally used when each input file contains multiple
years and multiple input files are needed
@footnote{For example, the @acronym{CMIP5} archive tends to distribute
monthly average timeseries in 50-year chunks.}.
In @w{Method 2} we obtain 15-year ensemble January average in a single
step, by averaging all 15 Januaries at one time:
@example
# tpt_flt and tpt_dbl are identical except for precision
ncks -C -v tpt_flt,tpt_dbl ~/nco/data/in.nc
# tpt_dbl = 273.1, 273.2, 273.3, 273.4, 273.5, 273.6, 273.7, 273.8, 273.9, 274
# tpt_flt = 273.1, 273.2, 273.3, 273.4, 273.5, 273.6, 273.7, 273.8, 273.9, 274
# Create file with five "ten-month years" (i.e., 50 timesteps) of temperature data
ncrcat -O -v tpt_flt,tpt_dbl -p ~/nco/data in.nc in.nc in.nc in.nc in.nc ~/foo.nc
# Average 1st five "Januaries" (elements 1, 11, 21, 31, 41)
ncra --flt -O -F -d time,1,,10 ~/foo.nc ~/foo_avg1.nc
# Average 2nd five "Januaries" (elements 2, 12, 22, 32, 42)
ncra --flt -O -F -d time,2,,10 ~/foo.nc ~/foo_avg2.nc
# Average 3rd five "Januaries" (elements 3, 13, 23, 33, 43)
ncra --flt -O -F -d time,3,,10 ~/foo.nc ~/foo_avg3.nc
# Method 1: Obtain ensemble January average by averaging the averages
ncra --flt -O ~/foo_avg1.nc ~/foo_avg2.nc ~/foo_avg3.nc ~/foo_avg_mth1.nc
# Method 2: Obtain ensemble January average by averaging the raw data
# Employ ncra's "subcycle" feature (http://nco.sf.net/nco.html#ssc)
ncra --flt -O -F -d time,1,,10,3 ~/foo.nc ~/foo_avg_mth2.nc
# Difference the two methods
ncbo -O ~/foo_avg_mth1.nc ~/foo_avg_mth2.nc ~/foo_avg_dff.nc
ncks ~/foo_avg_dff.nc
# tpt_dbl = 5.6843418860808e-14 ;
# tpt_flt = -3.051758e-05 ;
@end example
Although the two methods are arithmetically equivalent, they produce
slightly different answers due to the different order of operations.
Moreover, it appears at first glance that the single-precision
answers suffer from greater error than the double-precision answers.
In fact both precisions suffer from non-zero rounding errors.
The answers differ negligibly to machine precision, which is about 
seven significant figures for single precision floats (@code{tpt_flt}),
and sixteen significant figures for double precision (@code{tpt_dbl}).
The input precision determines the answer precision.

@acronym{IEEE} arithmetic guarantees that two methods will produce
bit-for-bit identical answers only if they compute the same operations
in the same order.  
Bit-for-bit identical answers may also occur by happenstance when 
rounding errors exactly compensate one another.
This is demonstrated by repeating the example above with the
@samp{--dbl} (or @samp{--rth_dbl} for clarity) option which forces
conversion of single-precision numbers to double-precision prior to
arithmetic. 
Now @command{ncra} will treat the first value of @code{tpt_flt},
@code{273.1000f}, as @code{273.1000000000000d}. 
Arithmetic on @code{tpt_flt} then proceeds in double-precision until the
final answer, which is converted back to single-precision for final
storage. 
@example
# Average 1st five "Januaries" (elements 1, 11, 21, 31, 41)
ncra --dbl -O -F -d time,1,,10 ~/foo.nc ~/foo_avg1.nc
# Average 2nd five "Januaries" (elements 2, 12, 22, 32, 42)
ncra --dbl -O -F -d time,2,,10 ~/foo.nc ~/foo_avg2.nc
# Average 3rd five "Januaries" (elements 3, 13, 23, 33, 43)
ncra --dbl -O -F -d time,3,,10 ~/foo.nc ~/foo_avg3.nc
# Method 1: Obtain ensemble January average by averaging the averages
ncra --dbl -O ~/foo_avg1.nc ~/foo_avg2.nc ~/foo_avg3.nc ~/foo_avg_mth1.nc
# Method 2: Obtain ensemble January average by averaging the raw data
# Employ ncra's "subcycle" feature (http://nco.sf.net/nco.html#ssc)
ncra --dbl -O -F -d time,1,,10,3 ~/foo.nc ~/foo_avg_mth2.nc
# Difference the two methods
ncbo -O ~/foo_avg_mth1.nc ~/foo_avg_mth2.nc ~/foo_avg_dff.nc
# Show differences
ncks ~/foo_avg_dff.nc
# tpt_dbl = 5.6843418860808e-14 ;
# tpt_flt = 0 ;
@end example
The @samp{--dbl} switch has no effect on the results computed from
double-precision inputs.
But now the two methods produce bit-for-bit identical results from the
single-precision inputs!
This is due to the happenstance of rounding along with the effects of 
the @samp{--dbl} switch.
The @samp{--flt} and @samp{--rth_flt} switches are provided for
symmetry.
They enforce the traditional @acronym{NCO} and Fortran convention of
keeping single-precision arithmetic in single-precision unless a
double-precision number is explicitly involved.

We have shown that forced promotion of single- to double-precision
prior to arithmetic has advantages and disadvantages. 
The primary disadvantages are speed and size. 
Double-precision arithmetic is 10--60% slower than, and requires
twice the memory of single-precision arithmetic. 
The primary advantage is that rounding errors in double-precision are 
much less likely to accumulate to values near the precision of the 
underlying geophysical variable. 

For example, if we know temperature to five significant digits, then a
rounding error of 1-bit could affect the least precise digit of
temperature after 1,000--10,000 consecutive one-sided rounding
errors under the worst possible scenario.
Many geophysical grids have tens-of-thousands to millions of points
that must be summed prior to normalization to compute an average.
It is possible for single-precision rouding errors to accumulate and
degrade the precision in such situtations. 
Double-precision arithmetic mititgates this problem, so @samp{--dbl}
would be warranted. 

@cindex @acronym{TREFHT}
@cindex @acronym{CAM3}
@cindex @acronym{GCM}
This can be seen with another example, averaging a global surface
temperature field with @command{ncwa}.
The input contains a single-precision global temperature field
(stored in @code{TREFHT}) produced by the @acronym{CAM3} general
circulation model (@acronym{GCM}) run and stored at @w{1.9 by 2.5}
degrees resolution. 
This requires @w{94 latitudes} and @w{144 longitudes}, or @math{13,824} 
total surface gridpoints, a typical GCM resolution in 2008--2013.
These input characteristics are provided only to show the context
to the interested reader, equivalent results would be found in 
statistics of any dataset of comparable size.
Models often represent Earth on a spherical grid where global averages 
must be created by weighting each gridcell by its latitude-dependent
weight (e.g., a Gaussian weight stored in @code{gw}), or by the
surface area of each contributing gridpoint (stored in @code{area}).

Like many geophysical models and most @acronym{GCM}s, @acronym{CAM3}
runs completely in double-precision yet stores its archival output in
single-precision to save space.
In practice such models usually save multi-dimensional prognostic and
diagnostic fields (like @code{TREFHT(lat,lon)}) as single-precision,
while saving all one-dimensional coordinates and weights (here
@code{lat}, @code{lon}, and @code{gw(lon)}) as double-precision.  
The gridcell area @code{area(lat,lon)} is an extensive grid property
that should be, but often is not, stored as double-precision.
To obtain pure double-precision arithmetic @emph{and} storage of the 
globla mean temperature, we first create and store double-precision
versions of the single-precision fields:
@example
ncap2 -O -s 'TREFHT_dbl=double(TREFHT);area_dbl=double(area)' in.nc in.nc
@end example
The single- and double-precision temperatures may each be averaged
globally using four permutations for the precision of the weight
and of the intermediate arithmetic representation:
@enumerate
@item Single-precision weight (@code{area}), single-precision arithmetic
@item Double-precision weight (@code{gw}),   single-precision arithmetic
@item Single-precision weight (@code{area}), double-precision arithmetic
@item Double-precision weight (@code{gw}),   double-precision arithmetic
@end enumerate
@example
# NB: Values below are printed with C-format %5.6f using
# ncks -H -C -s '%5.6f' -v TREFHT,TREFHT_dbl out.nc
# Single-precision weight (area), single-precision arithmetic
ncwa --flt -O -a lat,lon -w area in.nc out.nc
# TREFHT     = 289.246735 
# TREFHT_dbl = 289.239964
# Double-precision weight (gw),   single-precision arithmetic
ncwa --flt -O -a lat,lon -w gw   in.nc out.nc
# TREFHT     = 289.226135
# TREFHT_dbl = 289.239964
# Single-precision weight (area), double-precision arithmetic
ncwa --dbl -O -a lat,lon -w area in.nc out.nc
# TREFHT     = 289.239960
# TREFHT_dbl = 289.239964
# Double-precision weight (gw),   double-precision arithmetic
ncwa --dbl -O -a lat,lon -w gw   in.nc out.nc
# TREFHT     = 289.239960
# TREFHT_dbl = 289.239964
@end example
First note that the @code{TREFHT_dbl} average never changes because 
@code{TREFHT_dbl(lat,lon)} is double-precision in the input file.
As described above, @acronym{NCO} automatically converts all operands
involving to the highest precision involved in the operation.
So specifying @samp{--dbl} is redundant for double-precision inputs.

Second, the single-precision arithmetic averages of the single-precision
input @code{TREFHT} differ by @math{289.246735 - 289.226135 = 0.0206}
from eachother, and, more importantly, by as much as 
@math{289.239964 - 289.226135 = 0.013829} from the correct
(double-precision) answer.
These averages differ in the fifth digit, i.e., they agree only to four
significant figures!
Given that climate scientists are concerned about global temperature
variations of a tenth of a degree or less, this difference is large.
Global mean temperature changes significant to climate scientists are  
comparable in size to the numerical artifacts produced by the averaging
procedure.  

@cindex rounding
@cindex random walk
Why are the single-precision numerical artifacts so large?
Each global average is the result of multiplying almost 15,000 elements
each by its weight, summing those, and then dividing by the summed 
weights.  
Thus about 50,000 single-precision floating-point operations caused
the loss of two to three significant digits of precision.
The net error of a series of independent rounding errors is a random
walk phenomena
@footnote{
@cindex Michael Prather
Thanks to @w{Michael J.} Prather for explaining this to me.}.
Successive rounding errors displace the answer further from the truth.
An ensemble of such averages will, on average, have no net bias.
In other words, the expectation value of a series of @acronym{IEEE}
rounding errors is zero.
And the error of any given sequence of rounding errors obeys, for large 
series, a Gaussian distribution centered on zero.

@cindex mantissa
@cindex exponent
Single-precision numbers use three of their four eight-bit bytes to
represent the mantissa so the smallest representable single-precision
mantissa is @math{\epsilon \equiv 2^{-23} = 1.19209 \times 10^{-7}}.
This @math{\epsilon} is the smallest @var{x} such that 
@math{1.0 + x \ne 1.0}.
This is the rounding error for non-exact precision-numbers.
Applying random walk theory to rounding, it can be shown that the
expected rounding error after @var{n} inexact operations is
@math{\sqrt{2n/\pi}} for @w{large @var{n}}.
The expected (i.e., mean absolute) rounding error in our example with
@math{13,824} additions is about 
@math{\sqrt{2 \times 13824 / \pi} = 91.96}.
Hence, addition alone of about fifteen thousand single-precision floats
is expected to consume about two significant digits of precision.
This neglects the error due to the inner product (weights times values)
and normalization (division by tally) aspects of a weighted average.
The ratio of two numbers each containing a numerical bias can magnify
the size of the bias. 
In summary, a global mean number computed from about 15,000 gridpoints
each with weights can be expected to lose up to three significant digits.
Since single-precision starts with about seven significant digits, we
should not expect to retain more than four significant digits after
computing weighted averages in single-precision.
The above example with @code{TREFHT} shows the expected four digits of
agreement. 
@c For example, 50,000 coin flips would lead to 25,500 or more ``heads''
@c only a small percentage of the time.
@c P(k,n)= 50000_C_25500 p^k(1-p)^(n-k)
@c P(25500,50000)= 50000_C_25500 (0.5)^(25500)(0.5)^(24500)
@c P(>=25500,50000)= ?
@c fxm: Use Gaussian distribution/Random Walk

@cindex beer
The @acronym{NCO} results have been independently validated to the
extent possible in three other languages: 
@w{C}, Matlab, and @acronym{NCL}. 
C and @acronym{NCO} are the only languages that permit single-precision 
numbers to be treated with single precision arithmetic:
@example
# Double-precision weight (gw),   single-precision arithmetic (C)
ncwa_3528514.exe
# TREFHT     = 289.240112
# Double-precision weight (gw),   double-precision arithmetic (C)
# TREFHT     = 289.239964
# Single-precision weight (area), double-precision arithmetic (Matlab)
# TREFHT     = 289.239964
# Double-precision weight (gw),   double-precision arithmetic (Matlab)
# TREFHT     = 289.239964
# Single-precision weight (area), double-precision arithmetic (NCL)
ncl < ncwa_3528514.ncl
# TREFHT     = 289.239960
# TREFHT_dbl = 289.239964
# Double-precision weight (gw),   double-precision arithmetic (NCL)
# TREFHT     = 289.239960
# TREFHT_dbl = 289.239964
@end example
All languages tested (C, Matlab, @acronym{NCL}, and @acronym{NCO}) agree
to machine precision with double-precision arithmetic.
Users are fortunate to have a variety of high quality software that 
liberates them from the drudgery of coding their own.
Many packages are free (as in beer)!
As shown above @acronym{NCO} permits one to shift to their
float-promotion preferences as desired. 
No other language allows this with a simple switch.

To summarize, until version 4.3.6 (September, 2013), the default
arithmetic convention of @acronym{NCO} adhered to Fortran behavior,
and automatically promoted single-precision to double-precision in all 
mixed-precision expressions, and left-alone pure single-precision
expressions.  
This is faster and more memory efficient than other conventions.
However, pure single-precision arithmetic can lose too much precision
when used to condense (e.g., average) large arrays.
Statistics involving about @math{n = 10,000} single-precision inputs
will lose about @w{2--3} digits if not promoted to double-precision
prior to arithmetic. 
The loss scales with the squareroot @w{of @var{n}}. 
For larger @var{n}, users should promote floats with the @samp{--dbl} 
option if they want to preserve more than four significant digits in
their results. 

The @samp{--dbl} and @samp{--flt} switches are only available with the
@acronym{NCO} arithmetic operators that could potentially perform more
than a few  single-precision floating-point operations per result.
These are @command{nces}, @command{ncra}, and @command{ncwa}.
Each is capable of thousands to millions or more operations per result. 
By contrast, the arithmetic operators @command{ncbo} and
@command{ncflint} perform at most one floating-point operation per
result. 
Providing the @samp{--dbl} option for such trivial operations makes 
little sense, so the option is not currently made available.

We are interested in users' opinions on these matters. 
The default behavior was changed from @samp{--flt} to @samp{--dbl}
with the release of @acronym{NCO} version 4.3.6 (October 2013).
We will change the default back to @samp{--flt} if users prefer.
Or we could set a threshold (e.g., @math{n \ge 10000}) after which
single- to double-precision promotion is automatically invoked.
Or we could make the default promotion convention settable via an
environment variable (@acronym{GSL} does this a lot).
Please let us know what you think of the selected defaults and options.

@node Manual type conversion,  , Promoting Single-precision to Double, Type Conversion
@subsection Manual type conversion
@cindex @command{ncap2}
@command{ncap2} provides intrinsic functions for performing manual type
conversions.
This, for example, converts variable @code{tpt} to external type
@code{NC_SHORT} (a C-type @code{short}), and variable @code{prs} to
external type @code{NC_DOUBLE} (a C-type @code{double}). 
@example
ncap2 -s 'tpt=short(tpt);prs=double(prs)' in.nc out.nc
@end example
With ncap2 there also is the @code{convert()} method that takes an integer 
argument. For example the above statements become:
@example
ncap2 -s 'tpt=tpt.convert(NC_SHORT);prs=prs.convert(NC_DOUBLE)' in.nc out.nc
@end example
Can also use @code{convert()} in combination with @code{type()} so to make variable 
@code{ilev_new} the same type as @code{ilev} just do:
@example
ncap2 -s 'ilev_new=ilev_new.convert(ilev.type())' in.nc out.nc
@end example


@xref{ncap2 netCDF Arithmetic Processor}, for more details.

@html
<a name="ovr"></a> <!-- http://nco.sf.net/nco.html#ovr -->
<a name="-O"></a> <!-- http://nco.sf.net/nco.html#-O -->
@end html
@node Batch Mode, Global Attribute Addition, Type Conversion, Shared features
@section Batch Mode
@cindex batch mode
@cindex overwriting files
@cindex appending to files
@cindex force overwrite
@cindex force append
@cindex @code{-O}
@cindex @code{-A}
@cindex @code{--overwrite}
@cindex @code{--ovr}
@cindex @code{--apn}
@cindex @code{--append}
@cartouche
Availability: All operators@*
Short options: @samp{-O}, @samp{-A}@*
Long options: @samp{--ovr}, @samp{--overwrite}, @samp{--apn}, @samp{--append}@*
@end cartouche
If the @var{output-file} specified for a command is a pre-existing file,
then the operator will prompt the user whether to overwrite (erase) the
existing @var{output-file}, attempt to append to it, or abort the
operation. 
However, interactive questions reduce productivity when processing large
amounts of data.
Therefore @acronym{NCO} also implements two ways to override its own safety
features, the @samp{-O} and @samp{-A} switches.
Specifying @samp{-O} tells the operator to overwrite any existing
@var{output-file} without prompting the user interactively.
Specifying @samp{-A} tells the operator to attempt to append to any
existing @var{output-file} without prompting the user interactively.
These switches are useful in batch environments because they suppress
interactive keyboard input.
NB: As of 20120515, @command{ncap2} is unable to append to files that
already contain the appended dimensions. 

@html
<a name="gaa"></a> <!-- http://nco.sf.net/nco.html#gaa -->
<a name="glb"></a> <!-- http://nco.sf.net/nco.html#glb -->
<a name="glb_att_add"></a> <!-- http://nco.sf.net/nco.html#glb_att_add -->
@end html
@node Global Attribute Addition, History Attribute, Batch Mode, Shared features
@section Global Attribute Addition
@cindex global attributes
@cindex attributes, global
@cindex @code{--gaa}
@cindex @code{--glb @var{att_nm}=@var{att_val}}
@cindex @code{--glb}
@cindex @code{--glb_att_add}
@cindex @command{ncatted}
@cartouche
Availability: All operators@*
Short options: None@*
Long options: @samp{--glb}, @samp{--gaa}, @samp{--glb_att_add}@*
@samp{--glb @var{att_nm}=@var{att_val}} (multiple invocations allowed)@*
@end cartouche
All operators can add user-specified global attributes to output files.
As of @acronym{NCO} version 4.5.2 (July, 2015), @acronym{NCO} supports
multiple uses of the @samp{--glb} (or equivalent @samp{--gaa} or
@samp{--glb_att_add}) switch.
@cindex indicator option
@cindex multi-arguments
The option @samp{--gaa} (and its long option equivalents such as
@samp{--glb_att_add}) indicates the argument syntax will be
@var{key}=@var{val}. 
As such, @samp{--gaa} and its synonyms are indicator options that accept
arguments supplied one-by-one like 
@samp{--gaa @var{key1}=@var{val1} --gaa @var{key2}=@var{val2}}, or
aggregated together in multi-argument format like
@samp{--gaa @var{key1}=@var{val1}#@var{key2}=@var{val2}}
(@pxref{Multi-arguments}).

The switch takes mandatory arguments 
@samp{--glb @var{att_nm}=@var{att_val}}   
where @var{att_nm} is the desired name of the global attribute to add, 
and @var{att_val} is its value.
Currently only text attributes are supported (recorded as type
@code{NC_CHAR}), and regular expressions are not allowed (unlike
@pxref{ncatted netCDF Attribute Editor}).    
Attributes are added in ``Append'' mode, meaning that values are
appended to pre-existing values, if any. 
Multiple invocations can simplify the annotation of output file at
creation (or modification) time:
@example
@verbatim
ncra --glb machine=${HOSTNAME} --glb created_by=${USER} in*.nc out.nc
@end verbatim
@end example
As of @acronym{NCO} version 4.6.2 (October, 2016), one may instead
combine the separate invocations into a single list of invocations
separated by colons:
@example
@verbatim
ncra --glb machine=${HOSTNAME}:created_by=${USER} in*.nc out.nc
@end verbatim
@end example
The list may contain any number of key-value pairs.
Special care must be taken should a key or value contain a delimiter
(i.e., a colon) otherwise @command{NCO} will interpret the colon as
a delimiter and will attempt to create a new attribute.
To protect a colon from being interpreted as an argument delimiter,
precede it with a backslash.

The global attribution addition feature helps to avoid the performance
penalty incurred by using @command{ncatted} separately to annotate large files. 
Should users emit a loud hue and cry, we will consider ading the
functionality of ncatted to the front-end of all operators, i.e.,
accepting valid @command{ncatted} arguments to modify attributes of any
type and to apply regular expressions.

@html
<a name="hst"></a> <!-- http://nco.sf.net/nco.html#hst -->
<a name="history"></a> <!-- http://nco.sf.net/nco.html#history -->
<a name="-h"></a> <!-- http://nco.sf.net/nco.html#-h -->
@end html
@node History Attribute, File List Attributes, Global Attribute Addition, Shared features
@section History Attribute
@cindex @code{history}
@cindex timestamp
@cindex global attributes
@cindex attributes, global
@cindex @code{-h}
@cindex @code{--hst}
@cindex @code{--history}
@cartouche
Availability: All operators@*
Short options: @samp{-h}@*
Long options: @samp{--hst}, @samp{--history}@*
@end cartouche
All operators automatically append a @code{history} global attribute to
any file they create or modify.
The @code{history} attribute consists of a timestamp and the full string
of the invocation command to the operator, e.g., @samp{Mon May 26 20:10:24
1997: ncks in.nc out.nc}.
The full contents of an existing @code{history} attribute are copied
from the first @var{input-file} to the @var{output-file}.
The timestamps appear in reverse chronological order, with the most
recent timestamp appearing first in the @code{history} attribute.
Since @acronym{NCO} adheres to the @code{history} convention, the entire
data processing path of a given netCDF file may often be deduced from
examination of its @code{history} attribute. 
As of May, 2002, @acronym{NCO} is case-insensitive to the spelling
of the @code{history} attribute name.
Thus attributes named @code{History} or @code{HISTORY} (which are
non-standard and not recommended) will be treated as valid history
attributes. 
When more than one global attribute fits the case-insensitive search
for ``history'', the first one found is used.
@cindex @command{ncatted}
To avoid information overkill, all operators have an optional switch
(@samp{-h}, @samp{--hst}, or @samp{--history}) to override
automatically appending the @code{history} attribute 
(@pxref{ncatted netCDF Attribute Editor}).   
Note that the @samp{-h} switch also turns off writing the
@code{nco_input_file_list}-attribute for multi-file operators
(@pxref{File List Attributes}).

@cindex @code{history_of_appended_files}
@cindex provenance
As of @acronym{NCO} version 4.5.0 (June, 2015), @acronym{NCO} 
supports its own convention to retain the @code{history}-attribute 
contents of all files that were appended to a file
@footnote{Note that before version 4.5.0, @acronym{NCO} could,
in append (@samp{-A}) mode only, inadvertently overwrite the global
metadata (including  @code{history}) of the output file with that of the
input file. 
This is opposite the behavior most would want.}.
This convention stores those contents in the
@code{history_of_appended_files} attribute, which complements
the @code{history}-attribute to provide a more complete provenance.
These attributes may appear something like this in output:
@example
@verbatim
// global attributes:
:history = "Thu Jun  4 14:19:04 2015: ncks -A /home/zender/foo3.nc /home/zender/tmp.nc\n",
  "Thu Jun  4 14:19:04 2015: ncks -A /home/zender/foo2.nc /home/zender/tmp.nc\n",
  "Thu Jun  4 14:19:04 2015: ncatted -O -a att1,global,o,c,global metadata only in foo1 /home/zender/foo1.nc\n",
  "original history from the ur-file serving as the basis for subsequent appends." ;
:history_of_appended_files = "Thu Jun  4 14:19:04 2015: Appended file \
  /home/zender/foo3.nc had following \"history\" attribute:\n",
  "Thu Jun  4 14:19:04 2015: ncatted -O -a att2,global,o,c,global metadata only in foo3 /home/zender/foo3.nc\n",
  "history from foo3 from which data was appended to foo1 after data from foo2 was appended\n",
  "Thu Jun  4 14:19:04 2015: Appended file /home/zender/foo2.nc had following \"history\" attribute:\n",
  "Thu Jun  4 14:19:04 2015: ncatted -O -a att2,global,o,c,global metadata only in foo2 /home/zender/foo2.nc\n",
  "history of some totally different file foo2 from which data was appended to foo1 before foo3 was appended\n",
:att1 = "global metadata only in foo1" ;
@end verbatim
@end example
Note that the @code{history_of_appended_files}-attribute is only
created, and will only exist, in a file that is, or descends from
a file that was, appended to.
The optional switch @samp{-h} (or @samp{--hst} or @samp{--history})  
also overrides automatically appending the
@code{history_of_appended_files} attribute.

@html
<a name="fl_lst_in_att"></a> <!-- http://nco.sf.net/nco.html#fl_lst_in_att -->
@end html
@node File List Attributes, CF Conventions, History Attribute, Shared features
@section File List Attributes
@cindex @code{nco_input_file_list}
@cindex @code{nco_input_file_number}
@cindex @code{stdin}
@cindex global attributes
@cindex attributes, global
@cindex @code{-H}
@cindex @code{--fl_lst_in}
@cindex @code{--file_list}
@cartouche
Availability: All binary executables@*
Short options: @samp{-H}@*
Long options: @samp{--fl_lst_in}, @samp{--file_list}@*
@end cartouche
Many methods of specifying large numbers of input file names pass
these names via pipes, encodings, or argument transfer programs
(@pxref{Large Numbers of Files}).
When these methods are used, the input file list is not explicitly
passed on the command line.
This results in a loss of information since the @code{history}
attribute no longer contains the complete input information from which
the file was created.

@acronym{NCO} solves this dilemma by archiving an attribute that
contains the input file list.
When the input file list to an operator is specified via @code{stdin},
the operator, by default, attaches two global attributes to any file(s)
they create or modify. 
The @code{nco_input_file_number} global attribute contains the number of
input files, and @code{nco_input_file_list} contains the file names,
specified as standard input to the multi-file operator. 
This information helps to verify that all input files the user thinks
were piped through @code{stdin} actually arrived.
Without the @code{nco_input_file_list} attribute, the information is lost
forever and the ``chain of evidence'' would be broken.

The @samp{-H} switch overrides (turns off) the default behavior of
writing the input file list global attributes when input is from
@code{stdin}. 
The @samp{-h} switch does this too, and turns off the @code{history}
attribute as well (@pxref{History Attribute}).
Hence both switches allows space-conscious users to avoid storing what
may amount to many thousands of filenames in a metadata attribute.

@html
<a name="CF"></a> <!-- http://nco.sf.net/nco.html#CF -->
<a name="cnv"></a> <!-- http://nco.sf.net/nco.html#cnv -->
<a name="cnv_ACME"></a> <!-- http://nco.sf.net/nco.html#cnv_ACME -->
<a name="ACME"></a> <!-- http://nco.sf.net/nco.html#ACME -->
<a name="acme"></a> <!-- http://nco.sf.net/nco.html#acme -->
<a name="cnv_E3SM"></a> <!-- http://nco.sf.net/nco.html#cnv_E3SM -->
<a name="E3SM"></a> <!-- http://nco.sf.net/nco.html#E3SM -->
<a name="e3sm"></a> <!-- http://nco.sf.net/nco.html#e3sm -->
<a name="cnv_CF"></a> <!-- http://nco.sf.net/nco.html#cnv_CF -->
<a name="cnv_CCSM"></a> <!-- http://nco.sf.net/nco.html#cnv_CCSM -->
@end html
@node CF Conventions, ARM Conventions, File List Attributes, Shared features
@section @acronym{CF} Conventions
@cindex @acronym{ACME} conventions
@cindex @acronym{E3SM} conventions
@cindex @acronym{CF} conventions
@cindex @acronym{CCSM} conventions
@cindex @acronym{MPAS} conventions
@cindex UDUnits
@c CESM:
@cindex @code{ORO}
@cindex @code{area}
@cindex @code{datesec}
@cindex @code{date}
@cindex @code{gw}
@cindex @code{hyai}
@cindex @code{hyam}
@cindex @code{hybi}
@cindex @code{hybm}
@cindex @code{lat_bnds}
@cindex @code{lon_bnds}
@cindex @code{msk_*}
@cindex @code{wgt_*}
@c MPAS:
@cindex @code{angleEdge}
@cindex @code{areaCell}
@cindex @code{areaTriangle}
@cindex @code{cellMask}
@cindex @code{cellsOnCell}
@cindex @code{cellsOnEdge}
@cindex @code{cellsOnVertex}
@cindex @code{dcEdge}
@cindex @code{dvEdge}
@cindex @code{edgesOnCell}
@cindex @code{edgesOnEdge}
@cindex @code{edgesOnVertex}
@cindex @code{indexToCellID}
@cindex @code{indexToEdgeID}
@cindex @code{indexToVertexID}
@cindex @code{kiteAreasOnVertex}
@cindex @code{latCell}
@cindex @code{latEdge}
@cindex @code{latVertex}
@cindex @code{lonCell}
@cindex @code{lonEdge}
@cindex @code{lonVertex}
@cindex @code{maxLevelCell}
@cindex @code{meshDensity}
@cindex @code{nEdgesOnCell}
@cindex @code{nEdgesOnEdge}
@cindex @code{vertexMask}
@cindex @code{verticesOnCell}
@cindex @code{verticesOnEdge}
@cindex @code{weightsOnEdge}
@cindex @code{xCell}
@cindex @code{xEdge}
@cindex @code{xVertex}
@cindex @code{yCell}
@cindex @code{yEdge}
@cindex @code{yVertex}
@cindex @code{zCell}
@cindex @code{zEdge}
@cindex @code{zVertex}
@cartouche
Availability: @command{ncbo}, @command{nces}, @command{ncecat},
@command{ncflint}, @command{ncpdq}, @command{ncra}, @command{ncwa}@*
Short options: None@*
@end cartouche
@acronym{NCO} recognizes some Climate and Forecast (@acronym{CF})
metadata conventions, and applies special rules to such data.
@acronym{NCO} was contemporaneous with @acronym{COARDS} and still
contains some rules to handle older model datasets that pre-date
@acronym{CF}, such as @acronym{NCAR} @acronym{CCM} and early
@acronym{CCSM} datasets.
Such datasets may not contain an explicit @code{Conventions} attribute
(e.g., @samp{CF-1.0}). 
Nevertheless, we refer to all such metadata collectively as @acronym{CF}  
metadata. 
Skip this section if you never work with @acronym{CF} metadata.

The latest @acronym{CF} netCDF conventions are described 
@uref{http://cfconventions.org/1.10.html, here}. 
Most @acronym{CF} netCDF conventions are transparent to @acronym{NCO}.
There are no known pitfalls associated with using any @acronym{NCO}
operator on files adhering to these conventions.
@acronym{NCO} applies some rules that are not in @acronym{CF}, or
anywhere else, because experience shows that they simplify
data analysis, and stay true to the @acronym{NCO} mantra to do what 
users want.

Here is a general sense of @acronym{NCO}'s @acronym{CF}-support: 
@itemize @bullet
@item Understand and implement @acronym{NUG} recommendations such
as the history attribute, packing conventions, and attention to units. 
@item Special handling of variables designated as coordinates, bounds,
or ancillary variables, so that users subsetting a certain variable
automatically obtain all related variables.
@item Special handling and prevention of meaningless operations 
(e.g., the root-mean-square of latitude) so that coordinates and bounds
preserve meaningful information even as normal (non-coordinate) fields
are statistically transformed.
@item Understand units and certain calendars so that hyperslabs
may be specified in physical units, and so that user needs not manually
decode per-file time specifications.
@item Understand auxiliary coordinates so that irregular hyperslabs may
be specified on complex geometric grids.
@item Check for CF-compliance on netCDF3 and netCDF4 and @acronym{HDF}
files. 
@item Convert netCDF4 and @acronym{HDF} files to netCDF3 for strict
@acronym{CF}-compliance. 
@end itemize
Finally, a main use of @acronym{NCO} is to ``produce @acronym{CF}'', 
i.e., to improve @acronym{CF}-compliance by annotating metadata,
renaming objects (attributes, variables, and dimensions), permuting and
inverting dimensions, recomputing values, and data compression.

@html
<a name="prc_xcp"></a> <!-- http://nco.sf.net/nco.html#prc_xcp -->
@end html
Currently, @acronym{NCO} determines whether a datafile is a
@acronym{CF} output datafile simply by checking (case-insensitively)
whether the value of the global attribute @code{Conventions} (if any)
equals @samp{CF-1.0} or @samp{NCAR-CSM} 
Should @code{Conventions} equal either of these in the (first)
@var{input-file}, @acronym{NCO} will apply special rules to certain
variables because of their usual meaning in @acronym{CF} files. 
@acronym{NCO} will not average the following variables often found in
@acronym{CF} files: 
@code{ntrm}, @code{ntrn}, @code{ntrk}, @code{ndbase}, @code{nsbase},
@code{nbdate}, @code{nbsec}, @code{mdt}, @code{mhisf}.
These variables contain scalar metadata such as the resolution of the
host geophysical model and it makes no sense to change their values.

@cindex non-coordinate grid properties
Furthermore, the @dfn{size and rank-preserving arithmetic operators} try
not to operate on certain grid properties.
These operators are @command{ncap2}, @command{ncbo}, @command{nces},
@command{ncflint}, and @command{ncpdq} (when used for packing, not for
permutation).  
These operators do not operate, by default, on (i.e., add, subtract,
pack, etc.) the following variables:   
@code{ORO}, 
@code{area}, 
@code{datesec}, 
@code{date}, 
@code{gw}, 
@code{hyai}, 
@code{hyam},
@code{hybi}. 
@code{hybm}, 
@code{lat_bnds}, 
@code{lon_bnds},
@code{msk_*}, and
@code{wgt_*}.
These variables represent Gaussian weights, land/sea masks,
time fields, hybrid pressure coefficients, and latititude/longitude
boundaries.
We call these fields non-coordinate @dfn{grid properties}.
Coordinate grid properties are easy to identify because they are 
coordinate variables such as @code{latitude} and @code{longitude}.

Users usually want @emph{all} grid properties to remain unaltered in the 
output file. 
To be treated as a grid property, the variable name must @emph{exactly}
match a name in the above list, or be a coordinate variable. 
Handling of @code{msk_*} and @code{wgt_*} is exceptional in that
@emph{any} variable whose name starts with @code{msk_} or @code{wgt_} 
is considered to be a ``mask'' or a ``weight'' and is thus preserved
(not operated on when arithmetic can be avoided). 

As of @acronym{NCO} version 4.7.7 (September, 2018), @acronym{NCO} began
to explicitly identify files adhering to the @acronym{MPAS} convention.
These files have a global attribute @code{Conventions} attribute that
contains the string or @samp{MPAS}.
Size and rank-preserving arithmetic operators will not operate on these
@acronym{MPAS} non-coordinate grid properties:
@code{angleEdge}, 
@code{areaCell}, 
@code{areaTriangle}, 
@code{cellMask}, 
@code{cellsOnCell}, 
@code{cellsOnEdge}, 
@code{cellsOnVertex}, 
@code{dcEdge}, 
@code{dvEdge}, 
@code{edgesOnCell}, 
@code{edgesOnEdge}, 
@code{edgesOnVertex}, 
@code{indexToCellID}, 
@code{indexToEdgeID}, 
@code{indexToVertexID}, 
@code{kiteAreasOnVertex}, 
@code{latCell}, 
@code{latEdge}, 
@code{latVertex}, 
@code{lonCell}, 
@code{lonEdge}, 
@code{lonVertex}, 
@code{maxLevelCell}, 
@code{meshDensity}, 
@code{nEdgesOnCell}, 
@code{nEdgesOnEdge}, 
@code{vertexMask}, 
@code{verticesOnCell}, 
@code{verticesOnEdge}, 
@code{weightsOnEdge}, 
@code{xCell}, 
@code{xEdge}, 
@code{xVertex}, 
@code{yCell}, 
@code{yEdge}, 
@code{yVertex}, 
@code{zCell}, 
@code{zEdge}, and
@code{zVertex}.

As of @acronym{NCO} version 4.5.0 (June, 2015), @acronym{NCO} began
to support behavior required for the @acronym{DOE} @acronym{E3SM/ACME}
program, and we refer to these rules collectively as the @acronym{E3SM/ACME} 
convention. 
@cindex @acronym{GMT}
@cindex @code{date_written}
@cindex @code{time_written}
@cindex @code{gmtime()}
The first @acronym{E3SM/ACME} rule implemented is that the contents of
@var{input-file} variables named @code{date_written} and
@code{time_written}, if any, will be updated to the current 
system-supplied (with @code{gmtime()}) @acronym{GMT}-time as the
variables are copied to the @var{output-file}.

You must spoof @acronym{NCO} if you would like any grid properties
or other special @acronym{CF} fields processed normally.
For example rename the variables first with @command{ncrename}, 
or alter the @code{Conventions} attribute.

@html
<a name="cnv_CF_bnd"></a> <!-- http://nco.sf.net/nco.html#cnv_CF_bnd -->
<a name="bnd"></a> <!-- http://nco.sf.net/nco.html#bnd -->
<a name="bounds"></a> <!-- http://nco.sf.net/nco.html#bounds -->
@end html
@cindex @code{bounds} attribute
@cindex bounds convention
As of @acronym{NCO} version 4.0.8 (April, 2011), @acronym{NCO} 
supports the @acronym{CF} @code{bounds} convention for cell boundaries 
described 
@uref{http://cfconventions.org/cf-conventions/cf-conventions.html#cell-boundaries, here}.
This convention allows coordinate variables (including multidimensional
coordinates) to describe the boundaries of their cells.
This is done by naming the variable which contains the bounds in
in the @code{bounds} attribute. 
Note that coordinates of rank @math{N} have bounds of rank @math{N+1}.
NCO-generated subsets of @acronym{CF}-compliant files with @code{bounds}
attributes will include the coordinates specified by the @code{bounds}
attribute, if any.  
Hence the subsets will themselves be @acronym{CF}-compliant.
Bounds are subject to the user-specified override switches
(including @samp{-c} and @samp{-C}) described in 
@ref{Subsetting Coordinate Variables}. 

@cindex @code{lev}
@cindex @code{ilev}
@cindex @code{hyai}
@cindex @code{hybi}
The @acronym{CAM}/@acronym{EAM} family of atmospheric models does not
output a @code{bounds} variable or attribute corresponding to the
@code{lev} coordinate.
This prevents @acronym{NCO} from activating its @acronym{CF} bounds
machinery when @code{lev} is extracted.
As of version 4.7.7 (September, 2018), @acronym{NCO} works around this
by outputting the @code{ilev} coordinate (and @code{hyai}, @code{hybi})
whenever the @code{lev} coordinate is also output. 

@html
<a name="cnv_CF_clm"></a> <!-- http://nco.sf.net/nco.html#cnv_CF_clm -->
<a name="clm"></a> <!-- http://nco.sf.net/nco.html#clm -->
<a name="climatology"></a> <!-- http://nco.sf.net/nco.html#climatology -->
@end html
@cindex @code{climatology} attribute
@cindex climatology convention
As of @acronym{NCO} version 4.4.9 (May, 2015), @acronym{NCO} 
supports the @acronym{CF} @code{climatology} convention for climatological 
statistics described 
@uref{http://cfconventions.org/Data/cf-conventions/cf-conventions-1.7/build/cf-conventions.html#climatological-statistics, here}.
This convention allows coordinate variables (including multidimensional
coordinates) to describe the (possibly nested) periods and statistical
methods of their associated statistics.
This is done by naming the variable which contains the periods and
methods in the @code{climatology} attribute. 
Note that coordinates of rank @math{N} have climatology bounds of rank 
@math{N+1}. 
NCO-generated subsets of @acronym{CF}-compliant files with @code{climatology}
attributes will include the variables specified by the @code{climatology}
attribute, if any.  
Hence the subsets will themselves be @acronym{CF}-compliant.
Climatology variables are subject to the user-specified override switches 
(including @samp{-c} and @samp{-C}) described in 
@ref{Subsetting Coordinate Variables}. 

@html
<a name="cnv_CF_ncl"></a> <!-- http://nco.sf.net/nco.html#cnv_CF_ncl -->
<a name="ncl"></a> <!-- http://nco.sf.net/nco.html#ncl -->
<a name="ancillary"></a> <!-- http://nco.sf.net/nco.html#ancillary -->
@end html
@cindex @code{ancillary_variables} attribute
@cindex ancillary variables convention
As of @acronym{NCO} version 4.4.5 (July, 2014), @acronym{NCO} 
supports the @acronym{CF} @code{ancillary_variables} convention for 
described 
@uref{http://cfconventions.org/cf-conventions/cf-conventions.html#ancillary-data, here}.
This convention allows ancillary variables to be associated with one or
more primary variables.
@acronym{NCO} attaches any such variables to the extraction list along 
with the primary variable and its usual (one-dimensional) coordinates,
if any. 
Ancillary variables are subject to the user-specified override switches 
(including @samp{-c} and @samp{-C}) described in 
@ref{Subsetting Coordinate Variables}. 

@html
<a name="cnv_CF_cll_msr"></a> <!-- http://nco.sf.net/nco.html#cnv_CF_cll_msr -->
<a name="cll_msr"></a> <!-- http://nco.sf.net/nco.html#cll_msr -->
<a name="cell_measures"></a> <!-- http://nco.sf.net/nco.html#cell_measures -->
<a name="no_cll_msr"></a> <!-- http://nco.sf.net/nco.html#no_cll_msr -->
<a name="no_cell_measures"></a> <!-- http://nco.sf.net/nco.html#no_cell_measures -->
@end html
@cindex @code{cell_measures} attribute
@cindex cell measures convention
As of @acronym{NCO} version 4.6.4 (January, 2017), @acronym{NCO} 
supports the @acronym{CF} @code{cell_measures} convention 
described 
@uref{http://cfconventions.org/cf-conventions/cf-conventions.html#cell-measures, here}.
This convention allows variables to indicate which other variable or
variables contains area or volume information about a gridcell.
These measures variables are pointed to by the @code{cell_measures}
attribute.
The @acronym{CDL} specification of a measures variable for area looks
like 
@example
orog:cell_measures = "area: areacella"
@end example
where @code{areacella} is the name of the measures variable.
Unless the default behavior is overridden, @acronym{NCO} attaches any
measures variables to the extraction list along with the primary
variable and other associated variables. 
By definition, measures variables are a subset of the rank of the
variable they measure.
The most common case is that the measures variable for area is the same
size as @w{2D fields} (like surface air temperature) and much smaller
than @w{3D fields} (like full air temperature). 
In such cases the measures variable might occupy 50% of the space of a
dataset consisting of only one @w{2D field}.
Extraction of measures variables is subject to the user-specified
override switches (including @samp{-c} and @samp{-C}) described in  
@ref{Subsetting Coordinate Variables}. 
To conserve space without sacrificing too much metadata, @acronym{NCO}
makes it possible to override the extraction of measures variables
independent of extracting other associated variables.
Override the default with @samp{--no_cell_measures} or
@samp{--no_cll_msr}. 
These options are available in all operators that perform subsetting
(i.e., all operators except @command{ncatted} and @command{ncrename}).

@html
<a name="cnv_CF_frm_trm"></a> <!-- http://nco.sf.net/nco.html#cnv_CF_frm_trm -->
<a name="frm_trm"></a> <!-- http://nco.sf.net/nco.html#frm_trm -->
<a name="formula_terms"></a> <!-- http://nco.sf.net/nco.html#formula_terms -->
<a name="no_frm_trm"></a> <!-- http://nco.sf.net/nco.html#no_frm_trm -->
<a name="no_formula_terms"></a> <!-- http://nco.sf.net/nco.html#no_formula_terms -->
@end html
@cindex @code{formula_terms} attribute
@cindex cell measures convention
As of @acronym{NCO} version 4.6.4 (January, 2017), @acronym{NCO} 
supports the @acronym{CF} @code{formula_terms} convention 
described 
@uref{http://cfconventions.org/cf-conventions/cf-conventions.html#formula-terms, here}.
This convention encodes formulas used to construct (usually vertical)
coordinate grids.
The @acronym{CDL} specification of a vertical coordinate formula for
looks like 
@example
lev:standard_name = "atmosphere_hybrid_sigma_pressure_coordinate"
lev:formula_terms = "a: hyam b: hybm p0: P0 ps: PS"
@end example
where @code{standard_name} contains the standardized name of the
formula variable and @code{formula_terms} contains a list of the
variables used, called formula variables.
Above the formula variables are @code{hyam}, @code{hybm}, @code{P0}, and
@code{PS}. 
Unless the default behavior is overridden, @acronym{NCO} attaches any 
formula variables to the extraction list along with the primary
variable and other associated variables. 
By definition, formula variables are a subset of the rank of the
variable they define.
One common case is that the formula variables for constructing a 
@w{3D height} grid involves a 2D variable (like surface pressure,
or elevation).
In such cases the formula variables typically constitute only a 
small fraction of a dataset consisting of one @w{3D field}.
Extraction of formula variables is subject to the user-specified
override switches (including @samp{-c} and @samp{-C}) described in  
@ref{Subsetting Coordinate Variables}. 
To conserve space without sacrificing too much metadata, @acronym{NCO}
makes it possible to override the extraction of formula variables
independent of extracting other associated variables.
Override the default with @samp{--no_formula_terms} or
@samp{--no_frm_trm}. 
These options are available in all operators that perform subsetting
(i.e., all operators except @command{ncatted} and @command{ncrename}).

@html
<a name="cnv_CF_grd"></a> <!-- http://nco.sf.net/nco.html#cnv_CF_grd -->
<a name="grd_map"></a> <!-- http://nco.sf.net/nco.html#grd_map -->
<a name="grid_mapping"></a> <!-- http://nco.sf.net/nco.html#grid_mapping -->
@end html
@cindex @code{grid_mapping} attribute
@cindex ancillary variables convention
As of @acronym{NCO} version 4.6.0 (May, 2016), @acronym{NCO} 
supports the @acronym{CF} @code{grid_mapping} convention for 
described 
@uref{http://cfconventions.org/cf-conventions/cf-conventions.html#grid-mappings-and-projections, here}.
This convention allows descriptions of map-projections to be associated    
with variables.
@acronym{NCO} attaches any such map-projection variables to the
extraction list along with the primary variable and its usual
(one-dimensional) coordinates, if any. 
Map-projection variables are subject to the user-specified override
switches (including @samp{-c} and @samp{-C}) described in 
@ref{Subsetting Coordinate Variables}. 

@html
<a name="cnv_CF_crd"></a> <!-- http://nco.sf.net/nco.html#cnv_CF_crd -->
<a name="coordinates"></a> <!-- http://nco.sf.net/nco.html#coordinates -->
@end html
@cindex @code{coordinates}
@cindex coordinates convention
@cindex coordinate variable 
@cindex auxiliary coordinates
@cindex subsetting
@cindex @code{-C}
@cindex @code{-c}
@cindex @code{--no_coords}
@cindex @code{--no_crd}
@cindex @code{--coords}
@cindex @code{--crd}
As of @acronym{NCO} version 3.9.6 (January, 2009), @acronym{NCO}
supports the @acronym{CF} @code{coordinates} convention described 
@uref{http://cfconventions.org/cf-conventions/cf-conventions.html#coordinate-system, here}. 
This convention allows variables to specify additional coordinates
(including mult-idimensional coordinates) in a space-separated string 
attribute named @code{coordinates}. 
@acronym{NCO} attaches any such coordinates to the extraction list along
with the variable and its usual (one-dimensional) coordinates, if any. 
These auxiliary coordinates are subject to the user-specified override
switches (including @samp{-c} and @samp{-C}) described in 
@ref{Subsetting Coordinate Variables}. 

Elimination of reduced dimensions from the @code{coordinates} attribute
helps ensure that rank-reduced variables become completely independent
from their former dimensions.
As of @acronym{NCO} version 4.4.9 (May, 2015), @acronym{NCO}
may modify the @code{coordinates} attribute to assist this.
In particular, @command{ncwa} eliminates from the @code{coordinates}
attribute any dimension that it collapses, e.g., by averaging.
The former presence of this dimension will usually be indicated by the 
@acronym{CF} @code{cell_methods} convention described 
@uref{http://cfconventions.org/cf-conventions/cf-conventions.html#cell-methods, here}.
Hence the @acronym{CF} @code{cell_methods} and @code{coordinates} 
conventions can be said to work in tandem to characterize the state and
history of a variable's analysis.

@html
<a name="cnv_CF_cll_mth"></a> <!-- http://nco.sf.net/nco.html#cnv_CF_cll_mth -->
<a name="cll_mth"></a> <!-- http://nco.sf.net/nco.html#cll_mth -->
<a name="cell_methods"></a> <!-- http://nco.sf.net/nco.html#cell_methods -->
<a name="no_cll_mth"></a> <!-- http://nco.sf.net/nco.html#no_cll_mth -->
<a name="no_cell_methods"></a> <!-- http://nco.sf.net/nco.html#no_cell_methods -->
@end html
@cindex @code{cell_methods}
@cindex @code{--cll_mth}
@cindex @code{--no_cll_mth}
@cindex @code{--cell_methods}
@cindex @code{--no_cell_methods}
@cindex cell methods convention
As of @acronym{NCO} version 4.4.2 (February, 2014), @acronym{NCO} 
supports some of the @acronym{CF} @code{cell_methods} 
@uref{http://cfconventions.org/cf-conventions/cf-conventions.html#cell-methods, convention}
to describe the analysis procedures that have been applied to data.
The convention creates (or appends to an existing) @code{cell_methods}
attribute a space-separated list of couplets of the form @var{dmn: op}
where @var{dmn} is a comma-separated list of dimensions previously
contained in the variable that have been reduced by the arithmetic
operation @var{op}.
For example, the @code{cell_methods} value @code{time: mean} says that
the variable in question was averaged over the @code{time} dimension.
In such cases @code{time} will either be a scalar variable or a
degenerate dimension or coordinate. 
This simply means that it has been averaged-over.
The value @code{time, lon: mean lat: max} says that the variable in
question is the maximum zonal mean of the time averaged original
variable.
Which is to say that the variable was first averaged over time and
longitude, and then the residual latitudinal array was reduced by
choosing the maximum value.
Since the @code{cell methods} convention may alter metadata in an
undesirable (or possibly incorrect) fashion, we provide switches
to ensure it is always or never used.
Use long-options @samp{--cll_mth} or @samp{--cell_methods} to invoke the
algorithm (true by default), and options @samp{--no_cll_mth} or 
@samp{--no_cell_methods} to turn it off.
These options are only available in the operators @command{ncwa} and
@command{ncra}.

@html
<a name="cnv_ARM"></a> <!-- http://nco.sf.net/nco.html#cnv_ARM -->
<a name="ARM"></a> <!-- http://nco.sf.net/nco.html#ARM -->
<a name="arm"></a> <!-- http://nco.sf.net/nco.html#arm -->
@end html
@node ARM Conventions, Operator Version, CF Conventions, Shared features
@section @acronym{ARM} Conventions
@cindex @acronym{ARM} conventions
@cindex @code{time_offset}
@cindex @code{base_time}
@cindex @code{time}
@cartouche
Availability: @command{ncrcat}@*
Short options: None@*
@end cartouche
@command{ncrcat} has been programmed to correctly handle data files
which utilize the Atmospheric Radiation Measurement (@acronym{ARM})
Program @uref{http://www.arm.gov/data/time.stm,convention} for
time and time offsets.
If you do not work with @acronym{ARM} data then you may skip this
section. 
@acronym{ARM} data files store time information in two variables, a
scalar, @code{base_time}, and a record variable, @code{time_offset}.
Subtle but serious problems can arise when these type of files are
blindly concatenated without @acronym{CF} or @acronym{ARM} support. 
@command{NCO} implements rebasing (@pxref{Rebasing Time Coordinate}) 
as necessary on both @acronym{CF} and @acronym{ARM} files.
Rebasing chains together consecutive @var{input-files} and produces an
@var{output-file} which contains the correct time information. 
For @acronym{ARM} files this is expecially complex because the time
coordinates are often stored as type @code{NC_CHAR}.
Currently, @command{ncrcat} determines whether a datafile is an
@acronym{ARM} datafile simply by testing for the existence of the
variables @code{base_time}, @code{time_offset}, and the dimension
@code{time}.  
If these are found in the @var{input-file} then @command{ncrcat} will 
automatically perform two non-standard, but hopefully useful,
procedures. 
First, @command{ncrcat} will ensure that values of @code{time_offset}
appearing in the @var{output-file} are relative to the @code{base_time}
appearing in the first @var{input-file} (and presumably, though not
necessarily, also appearing in the @var{output-file}).
Second, if a coordinate variable named @code{time} is not found in the
@var{input-files}, then @command{ncrcat} automatically creates the
@code{time} coordinate in the @var{output-file}.  
The values of @code{time} are defined by the @acronym{ARM} conventions 
@math{@var{time} = @var{base_time} + @var{time_offset}}.
Thus, if @var{output-file} contains the @code{time_offset}
variable, it will also contain the @code{time} coordinate.
@cindex @code{history}
@cindex global attributes
@cindex attributes, global
@w{A short} message is added to the @code{history} global attribute
whenever these @acronym{ARM}-specific procedures are executed.

@html
<a name="vrs"></a> <!-- http://nco.sf.net/nco.html#vrs -->
<a name="version"></a> <!-- http://nco.sf.net/nco.html#version -->
@end html
@node Operator Version,  , ARM Conventions, Shared features
@section Operator Version
@cindex version
@cindex @acronym{RCS}
@cindex @code{-r}
@cindex @code{--revision}
@cindex @code{--version}
@cindex @code{--vrs}
@cartouche
Availability: All operators@*
Short options: @samp{-r}@*
Long options: @samp{--revision}, @samp{--version}, or @samp{--vrs}@*
@end cartouche
All operators can be told to print their version information,
library version, copyright notice, and compile-time configuration 
with the @samp{-r} switch, or its long-option equivalent
@samp{revision}. 
The @samp{--version} or @samp{--vrs} switches print the operator
version information only.
The internal version number varies between operators, and indicates the 
most recent change to a particular operator's source code.
This is useful in making sure you are working with the most recent
operators.
The version of @acronym{NCO} you are using might be, e.g., @code{3.9.5}.
Using @samp{-r} on, say, @command{ncks}, produces something like 
@samp{NCO netCDF Operators version "3.9.5" last modified 2008/05/11 built May 12 2008 on neige by zender 
Copyright (C) 1995--2008 Charlie Zender
ncks version 20090918}.
This tells you that @command{ncks} contains all patches up to version 
@code{3.9.5}, which dates from @w{May 11}, 2008.
@html
<a name="rfr"></a> <!-- http://nco.sf.net/nco.html#rfr -->
@end html
@node Reference Manual, Contributing, Shared features, Top
@chapter Reference Manual

This chapter presents reference pages for each of the operators
individually. 
The operators are presented in alphabetical order.
@cindex command line switches
All valid command line switches are included in the syntax statement.
Recall that descriptions of many of these command line switches are
provided only in @ref{Shared features}, to avoid redundancy.
Only options specific to, or most useful with, a particular operator are 
described in any detail in the sections below.  

@menu
* ncap2 netCDF Arithmetic Processor::
* ncatted netCDF Attribute Editor::
* ncbo netCDF Binary Operator::
* ncclimo netCDF Climatology Generator::
* ncecat netCDF Ensemble Concatenator::
* nces netCDF Ensemble Statistics::
* ncflint netCDF File Interpolator::
* ncks netCDF Kitchen Sink::
* ncpdq netCDF Permute Dimensions Quickly::
* ncra netCDF Record Averager::
* ncrcat netCDF Record Concatenator::
* ncremap netCDF Remapper::
* ncrename netCDF Renamer::
* ncwa netCDF Weighted Averager::
@end menu

@page
@html
<a name="ncap"></a> <!-- http://nco.sf.net/nco.html#ncap -->
<a name="ncap2"></a> <!-- http://nco.sf.net/nco.html#ncap2 -->
@end html
@node ncap2 netCDF Arithmetic Processor, ncatted netCDF Attribute Editor, Reference Manual, Reference Manual
@section @command{ncap2} netCDF Arithmetic Processor
@cindex parser
@cindex lexer
@cindex arithmetic processor
@findex ncap
@findex ncap2
@cartouche
@command{ncap2} understands a relatively full-featured 
language of operations, including loops, conditionals, arrays,
and math functions.
@command{ncap2} is the most rapidly changing @acronym{NCO} operator and
its documentation is incomplete.
The distribution file @file{data/ncap2_tst.nco} contains an up-to-date
overview of its syntax and capabilities. 
The @file{data/*.nco} distribution files (especially
@file{bin_cnt.nco}, @file{psd_wrf.nco}, and @file{rgr.nco}) contain
in-depth examples of @command{ncap2} solutions to complex problems.
@end cartouche

@c fxm: TODO nco549 hyper-link all switches to explanatory sections?
@c Problem is that only works well in HTML mode
@c TeXInfo has no native mode for concise hyperlinks in text mode
@c Currently in TeX/PDF mode, TeXInfo opens browser to find link,
@c rather than jumping to internal link within document
@noindent
SYNTAX
@example
ncap2 [-3] [-4] [-5] [-6] [-7] [@uref{http://nco.sf.net/nco.html#-A,,-A}] [-C] [-c] 
[-D @var{dbg}] [-F] [-f] [--glb ...] [-H] [-h] [--hdf] [--hdr_pad @var{nbr}] [--hpss]
[-L @var{dfl_lvl}] [-l @var{path}] [--no_tmp_fl] [-O] [-o @var{output-file}]
[-p @var{path}] [-R] [-r] [--ram_all]
[-s @var{algebra}] [-S @var{fl.nco}] [-t @var{thr_nbr}] [-v]
[@var{input-file}] [@var{output-file}]  
@end example
 
@noindent
DESCRIPTION

@command{ncap2} arithmetically processes netCDF files.
@command{ncap2} is the successor to @command{ncap} which was put into
maintenance mode in November, 2006, and completely removed from
@acronym{NCO} in March, 2018. 
This documentation refers to @command{ncap2} implements its own
domain-specific language to produc a powerful superset
@command{ncap}-functionality.
@command{ncap2} may be renamed @command{ncap} one day!
@cindex script file
@cindex @code{--script-file}
@cindex @code{--fl_spt}
@cindex @code{--script}
@cindex @code{--spt}
The processing instructions are contained either in the @acronym{NCO}
script file @file{fl.nco} or in a sequence of command line arguments.
The options @samp{-s} (or long options @samp{--spt} or @samp{--script})
are used for in-line scripts and @samp{-S} (or long options
@samp{--fl_spt}, @samp{--nco_script}, or @samp{--script-file}) are used to provide the
filename where (usually multiple) scripting commands are pre-stored.   
@command{ncap2} was written to perform arbitrary algebraic
transformations of data and archive the results as easily as
possible. 
@cindex derived fields
@xref{Missing Values}, for treatment of missing values.
The results of the algebraic manipulations are called 
@dfn{derived fields}. 

Unlike the other operators, @command{ncap2} does not accept a list of
variables to be operated on as an argument to @samp{-v} 
(@pxref{Subsetting Files}).
Rather, the @samp{-v} switch takes no arguments and indicates
that @command{ncap2} should output @emph{only} user-defined variables.
@command{ncap2} neither accepts nor understands the @var{-x} switch.
@cindex appending variables
NB: As of 20120515, @command{ncap2} is unable to append to files that
already contain the appended dimensions. 

Providing a name for @var{output-file} is optional if @var{input-file}
is a netCDF3 format, in which case @command{ncap2} attempts to write
modifications directly to @var{input-file} (similar to the behavior of
@command{ncrename} and @command{ncatted}).
Format-constraints prevent this type of appending from working on a
netCDF4 format @var{input-file}.
In any case, reading and writing the same file can be risky and lead
to unexpected consequences (since the file is being both read and
written), so in normal usage we recommend providing @var{output-file}
(which can be the same as @var{input-file} since the changes are first
written to an intermediate file).

As of @acronym{NCO} version 4.8.0 (released May, 2019),
@command{ncap2} does not require that @var{input-file} be specified
when @var{output-file} has no dependency on it.
Prior to this, @command{ncap2} required users to specify a dummy
@var{input-file} even if it was not used to construct
@var{output-file}.
Input files are always read by @command{ncap2}, and dummy input
files are read though not used for anything nor modified.
Now 
@example
@verbatim
ncap2 -s 'quark=1' ~/foo.nc # Create new foo.nc
ncap2 -s 'print(quark)' ~/foo.nc # Print existing foo.nc
ncap2 -O -s 'quark=1' ~/foo.nc # Overwrite old with new foo.nc
ncap2 -s 'quark=1' ~/foo.nc ~/foo.nc # Add to old foo.nc
@end verbatim
@end example

@c @subsection Scripting Mathematical Processing with @command{ncap2}

@menu
* Syntax of ncap2 statements::
* Expressions::
* Dimensions::
* Left hand casting::
* Arrays and hyperslabs::
* Attributes::
* Value List::
* Number literals::
* if statement::
* Print & String methods::
* Missing values ncap2::
* Methods and functions::
* RAM variables::
* Where statement::
* Loops::
* Include files::
* Sort methods::
* UDUnits script::
* Vpointer::
* Irregular grids::
* Bilinear interpolation::
* GSL special functions::
* GSL interpolation::
* GSL least-squares fitting::
* GSL statistics::
* GSL random number generation::
* Examples ncap2::
* Intrinsic mathematical methods::
* Operator precedence and associativity ::
* ID Quoting::
* make_bounds() function::
* solar_zenith_angle function::
@end menu

@html
<a name="att_prp"></a> <!-- http://nco.sf.net/nco.html#att_prp -->
@end html
Defining new variables in terms of existing variables is a powerful
feature of @command{ncap2}. 
@cindex derived fields
Derived fields inherit the metadata (i.e., attributes) of their
ancestors, if any, in the script or input file. 
When the derived field is completely new (no identically-named ancestors
exist), then it inherits the metadata (if any) of the left-most variable
on the right hand side of the defining expression.
This metadata inheritance is called @dfn{attribute propagation}.
Attribute propagation is intended to facilitate well-documented 
data analysis, and we welcome suggestions to improve this feature.

The only exception to this rule of attribute propagation is in cases of
left hand casting (@pxref{Left hand casting}).
The user must manually define the proper metadata for variables defined
using left hand casting. 

@html
<a name="syn"></a> <!-- http://nco.sf.net/nco.html#syn -->
@end html
@node Syntax of ncap2 statements, Expressions, ncap2 netCDF Arithmetic Processor, ncap2 netCDF Arithmetic Processor
@subsection Syntax of @command{ncap2} statements
@cindex statement
@cindex syntax
Mastering @command{ncap2} is relatively simple.
Each valid statement @var{statement} consists of standard forward
algebraic expression. 
The @file{fl.nco}, if present, is simply a list of such statements,
whitespace, and comments.
@cindex C language
The syntax of statements is most like the computer @w{language C}.
The following characteristics @w{of C} are preserved:
@table @asis
@item Array syntax
@cindex array syntax
@cindex @code{[]} (array delimiters)
Arrays elements are placed within @code{[]} characters;
@item Array indexing
@cindex array indexing
Arrays are 0-based;
@item Array storage
@cindex array storage
Last dimension is most rapidly varying;
@item Assignment statements
@cindex assignment statement
@cindex semi-colon
@cindex @code{;} (end of statement)
@w{A semi}-colon @samp{;} indicates the end of an assignment statement.
@item Comments
@cindex comments
@cindex @code{/*...*/} (comment)
@cindex @code{//} (comment)
Multi-line comments are enclosed within @code{/* */} characters.
Single line comments are preceded by @code{//} characters.
@item Nesting
@cindex including files
@cindex nesting
@cindex @code{#include}
Files may be nested in scripts using @code{#include @var{script}}.
The @code{#include} command is not followed by a semi-colon because it
is a pre-processor directive, not an assignment statement. 
The filename @file{script} is interpreted relative to the run directory.
@item Attribute syntax
@cindex attribute syntax
@cindex @code{@@} (attribute)
The at-sign @code{@@} is used to delineate an attribute name from a
variable name.
@end table 

@html
<a name="ncap_xpr"></a> <!-- http://nco.sf.net/nco.html#ncap_xpr -->
@end html
@node Expressions, Dimensions, Syntax of ncap2 statements, ncap2 netCDF Arithmetic Processor
@cindex expressions
@subsection Expressions
Expressions are the fundamental building block of @command{ncap2}. 
Expressions are composed of variables, numbers, literals, and
attributes. 
@cindex C language
The following @w{C operators} are ``overloaded'' and work with scalars 
and multi-dimensional arrays:
@example
Arithmetic Operators: * / % + - ^
Binary Operators:     > >= < <= == != == || && >> <<
Unary Operators:      + - ++ -- !
Conditional Operator: exp1 ? exp2 : exp3
Assign Operators:     = += -= /= *=
@end example

In the following section a @dfn{variable} also refers to a 
number literal which is read in as a scalar variable:

@strong{Arithmetic and Binary Operators }

Consider @emph{var1 'op' var2}

@strong{Precision}
@itemize @bullet
@item When both operands are variables, the result has the precision of the higher precision operand.
@item When one operand is a variable and the other an attribute, the result has the precision of the variable. 
@item When both operands are attributes, the result has the precision of the more precise attribute.
@item The exponentiation operator ``^'' is an exception to the above rules. 
When both operands have type less than @code{NC_FLOAT}, the result is @code{NC_FLOAT}. 
When either type is @code{NC_DOUBLE}, the result is also @code{NC_DOUBLE}. 
@end itemize
@c csz got to here editing

@cindex broadcasting variables
@cindex rank
@strong{Rank}
@itemize @bullet 
@item The Rank of the result is generally equal to Rank of the operand
that has the greatest number of dimensions.  
@item If the dimensions in var2 are a subset of the dimensions in var1
then its possible to  make var2 conform to var1 through broadcasting and
or dimension reordering.  
@item Broadcasting a variable means creating data in non-existing
dimensions by copying data in existing dimensions. 
@item More specifically: If the numbers of dimensions in var1 is greater 
than or equal to the number of dimensions in var2 then an attempt is
made to make var2 conform to var1 ,else var1 is made to conform to
var2. If conformance  is not possible then an error message will be
emitted and script execution will cease.@* 
@end itemize

@noindent Even though the logical operators return True(1) or False(0)
they are treated in the same way as the arithmetic operators with regard
to precision and rank.@* 
Examples:

@example
@verbatim
dimensions: time=10, lat=2, lon=4
Suppose we have the two variables:

double  P(time,lat,lon);
float   PZ0(lon,lat);  // PZ0=1,2,3,4,5,6,7,8;

Consider now the expression:
 PZ=P-PZ0

PZ0 is made to conform to P and the result is
PZ0 =
   1,3,5,7,2,4,6,8,
   1,3,5,7,2,4,6,8,
   1,3,5,7,2,4,6,8,
   1,3,5,7,2,4,6,8,
   1,3,5,7,2,4,6,8,
   1,3,5,7,2,4,6,8,
   1,3,5,7,2,4,6,8,
   1,3,5,7,2,4,6,8,
   1,3,5,7,2,4,6,8,
   1,3,5,7,2,4,6,8,

Once the expression is evaluated then PZ will be of type double;

Consider now 
 start=four-att_var@double_att;  // start =-69  and is of type intger;
 four_pow=four^3.0f               // four_pow=64 and is of type float  
 three_nw=three_dmn_var_sht*1.0f; // type is now float
 start@n1=att_var@short_att*att_var@int_att; 
                                  // start@n1=5329 and is type int 
@end verbatim
@end example

@noindent @strong{Binary Operators} @* 
@cindex binary Operators
Unlike C the binary operators return an array of values. 
There is no such thing as short circuiting with the AND/OR operators. 
Missing values are carried into the result in the same way they are with
the arithmetic operators. 
When an expression is evaluated in an if() the missing values are
treated as true.@*  
The binary operators are, in order of precedence: 
@example
	
!   Logical Not
----------------------------
<<  Less Than Selection
>>  Greater Than Selection
----------------------------
>   Greater than
>=  Greater than or equal to
<   Less than
<=  Less than or equal to
----------------------------
==  Equal to
!=  Not equal to
----------------------------
&&  Logical AND
----------------------------
||  Logical OR
----------------------------
@end example

To see all operators: @pxref{Operator precedence and associativity} 
Examples:
@example
tm1=time>2 && time <7;  // tm1=0, 0, 1, 1, 1, 1, 0, 0, 0, 0 double
tm2=time==3 || time>=6; // tm2=0, 0, 1, 0, 0, 1, 1, 1, 1, 1 double
tm3=int(!tm1);          // tm3=1, 1, 0, 0, 0, 0, 1, 1, 1, 1 int
tm4=tm1 && tm2;         // tm4=0, 0, 1, 0, 0, 1, 0, 0, 0, 0 double
tm5=!tm4;               // tm5=1, 1, 0, 1, 1, 0, 1, 1, 1, 1 double
@end example
	
@noindent @strong{Regular Assign Operator}@*
@noindent @emph{var1 '=' exp1} @*
If var1 does not already exist in Input or Output then var1 is written to Output with the values, type and dimensions from expr1. If var1 is in Input only it is copied to Output first. Once the var is in Ouptut then the only reqirement on expr1 is that the number of elements must match the number already on disk. The type of expr1 is converted as necessary to the disk type.

If you wish to change the type or shape of a variable in Input then you
must cast the variable.
See @pxref{Left hand casting}
@example
time[time]=time.int();
three_dmn_var_dbl[time,lon,lat]=666L;
@end example

@noindent @strong{ Other Assign Operators +=,-=,*=./= }@*
        @noindent  @emph{var1 'ass_op' exp1 }@*
if exp1 is a variable and it doesn't conform to var1 then an attempt is made to make it conform to var1. If exp1 is an attribute it must have unity size or else have the same number of elements as var1. If expr1 has a different type to var1 the it is converted to the var1 type.
@example
z1=four+=one*=10 // z1=14 four=14 one=10;	
time-=2          // time= -1,0,1,2,3,4,5,6,7,8
@end example

@noindent @strong{Increment/Decrement Operators @*} 
These work in a similar fashion to their regular C counterparts. If say the variable @code{four} is input only then the statement @code{++four} effectively means read @code{four} from input increment each element by one, then write the new values to Output;

Example:
@example
n2=++four;   n2=5, four=5 
n3=one--+20; n3=21  one=0;	 
n4=--time;   n4=time=0.,1.,2.,3.,4.,5.,6.,7.,8.,9.;
@end example

@noindent @strong{Conditional Operator ?:}@*
@cindex conditional Operator
@noindent @emph{exp1 ? exp2 : exp3 } @*
The conditional operator (or ternary Operator) is a succinct way
of writing an if/then/else. If exp1 evaluates to true then exp2 is
returned else exp3 is returned. 

Example:
@example
@verbatim
weight_avg=weight.avg();
weight_avg@units= (weight_avg == 1 ? "kilo" : "kilos");  
PS_nw=PS-(PS.min() > 100000 ? 100000 : 0);
@end verbatim
@end example

@html
<a name="clp"></a> <!-- http://nco.sf.net/nco.html#clp -->
<a name="clipping"></a> <!-- http://nco.sf.net/nco.html#clipping -->
@end html
@noindent @strong{Clipping Operators}
@cindex clipping operators
@table @asis
@item << Less-than Clipping@*
For arrays, the less-than selection operator selects all values in the
left operand that are less than the corresponding value in the right
operand. 
If the value of the left side is greater than or equal to the
corresponding value of the right side, then the right side value is 
placed in the result	 
@item >> Greater-than Clipping@*
For arrays, the greater-than selection operator selects all values in
the left operand that are greater than the corresponding value in the
right operand. 
If the value of the left side is less than or equal to the corresponding
value of the right side, then the right side value is placed in the
result.  
@end table

Example:
@example
RDM2=RDM >> 100.0 // 100,100,100,100,126,126,100,100,100,100 double
RDM2=RDM <<  90s  // 1, 9, 36, 84, 90, 90, 84, 36, 9, 1 int
@end example

@html
<a name="ncap_dims"></a> <!-- http://nco.sf.net/nco.html#ncap_dims -->
<a name="defdim"></a> <!-- http://nco.sf.net/nco.html#defdim -->
@end html
@node Dimensions, Left hand casting, Expressions, ncap2 netCDF Arithmetic Processor
@subsection Dimensions
@cindex defining dimensions in @command{ncap2}
@cindex @code{defdim()}
Dimensions are defined in Output using the @code{defdim()} function.
@example
defdim("cnt",10); # Dimension size is fixed by default
defdim("cnt",10,NC_UNLIMITED); # Dimension is unlimited (record dimension)
defdim("cnt",10,0); # Dimension is unlimited (record dimension)
defdim("cnt",10,1); # Dimension size is fixed
defdim("cnt",10,737); # All non-zero values indicate dimension size is fixed
@end example

This dimension name must then be prefixed with a dollar-sign @samp{$}
when referred to in method arguments or left-hand-casting, e.g.,
@example
new_var[$cnt]=time;
temperature[$time,$lat,$lon]=35.5;
temp_avg=temperature.avg($time);
@end example

The @code{size} method allows dimension sizes to be used in 
arithmetic expressions:
@example
time_avg=time.total()/$time.size;
@end example

Increase the size of a new variable by one and set new member to zero:
@example
defdim("cnt_new",$cnt.size+1);
new_var[$cnt_new]=0.0;
new_var(0:($cnt_new.size-2))=old_var;
@end example

To define an unlimited dimension, simply set the size to zero
@example
defdim("time2",0)
@end example

@noindent @strong{Dimension Abbreviations @*}
It is possible to use dimension abbreviations as method arguments:@*
@code{$0} is the first dimension of a variable@*
@code{$1} is the second dimension of a variable@*
@code{$n} is the n+1 dimension of a variable@*

@example
float four_dmn_rec_var(time,lat,lev,lon);
double three_dmn_var_dbl(time,lat,lon);

four_nw=four_dmn_rev_var.reverse($time,$lon)
four_nw=four_dmn_rec_var.reverse($0,$3);

four_avg=four_dmn_rec_var.avg($lat,$lev);  
four_avg=four_dmn_rec_var.avg($1,$2);  

three_mw=three_dmn_var_dbl.permute($time,$lon,$lat);
three_mw=three_dmn_var_dbl.permute($0,$2,$1);
@end example

@noindent @strong{ID Quoting @*}
If the dimension name contains non-regular characters use ID quoting:
See @pxref{ID Quoting}
@example
defdim("a--list.A",10);
A1['$a--list.A']=30.0;
@end example

@noindent @strong{GOTCHA @*}
@noindent It is not possible to manually define in Output any dimensions that exist in Input. When a variable from Input appears in an expression or statement its  dimensions in Input are  automagically copied to Output (if they are not already present)

@html
<a name="lhc"></a> <!-- http://nco.sf.net/nco.html#lhc -->
<a name="lhs"></a> <!-- http://nco.sf.net/nco.html#lhs -->
@end html
@node Left hand casting, Arrays and hyperslabs, Dimensions, ncap2 netCDF Arithmetic Processor
@subsection Left hand casting
@cindex hybrid coordinate system 
@cindex left hand casting
@cindex @acronym{LHS}
The following examples demonstrate the utility of the 
@dfn{left hand casting} ability of @command{ncap2}.
Consider first this simple, artificial, example.
If @var{lat} and @var{lon} are one dimensional coordinates of
dimensions @var{lat} and @var{lon}, respectively, then addition
of these two one-dimensional arrays is intrinsically ill-defined because 
whether @var{lat_lon} should be dimensioned @var{lat} by @var{lon}
or @var{lon} by @var{lat} is ambiguous (assuming that addition is to
remain a @dfn{commutative} procedure, i.e., one that does not depend on 
the order of its arguments).
Differing dimensions are said to be @dfn{orthogonal} to one another,
and sets of dimensions which are mutually exclusive are orthogonal
as a set and any arithmetic operation between variables in orthogonal
dimensional spaces is ambiguous without further information.

The ambiguity may be resolved by enumerating the desired dimension 
ordering of the output expression inside square brackets on the
left hand side (@acronym{LHS}) of the equals sign.
This is called @dfn{left hand casting} because the user resolves the 
dimensional ordering of the @acronym{RHS} of the expression by
specifying the desired ordering on the @acronym{LHS}.
@example
ncap2 -s 'lat_lon[lat,lon]=lat+lon' in.nc out.nc
ncap2 -s 'lon_lat[lon,lat]=lat+lon' in.nc out.nc
@end example
The explicit list of dimensions on the @acronym{LHS}, @code{[lat,lon]}
resolves the otherwise ambiguous ordering of dimensions in
@var{lat_lon}. 
In effect, the @acronym{LHS} @dfn{casts} its rank properties onto the 
@acronym{RHS}.
Without @acronym{LHS} casting, the dimensional ordering of @var{lat_lon}
would be undefined and, hopefully, @command{ncap2} would print an error
message. 

@html
<a name="prs_mdp"></a> <!-- http://nco.sf.net/nco.html#prs_mdp -->
@end html
Consider now a slightly more complex example.
In geophysical models, a coordinate system based on 
a blend of terrain-following and density-following surfaces is 
called a @dfn{hybrid coordinate system}.
In this coordinate system, four variables must be manipulated to
obtain the pressure of the vertical coordinate:
@var{PO} is the domain-mean surface pressure offset (a scalar),
@var{PS} is the local (time-varying) surface pressure (usually two
horizontal spatial dimensions, i.e. latitude by longitude), @var{hyam}
is the weight given to surfaces of constant density (one spatial
dimension, pressure, which is orthogonal to the horizontal
dimensions), and @var{hybm} is the weight given to surfaces of
constant elevation (also one spatial dimension). 
This command constructs a four-dimensional pressure @code{prs_mdp}
from the four input variables of mixed rank and orthogonality:
@example
ncap2 -s 'prs_mdp[time,lat,lon,lev]=P0*hyam+PS*hybm' in.nc out.nc
@end example
Manipulating the four fields which define the pressure in a hybrid
coordinate system is easy with left hand casting.

@html
<a name="pdel"></a> <!-- http://nco.sf.net/nco.html#pdel -->
<a name="prs_ntf"></a> <!-- http://nco.sf.net/nco.html#prs_ntf -->
@end html
Finally, we show how to use interface quantities to define midpoint
quantities.
In particular, we will define interface pressures using the standard
@acronym{CESM} output hybrid coordinate parameters, and then difference
those interface pressures to obtain the pressure difference between
the interfaces. 
The pressure difference is necessary obtain gridcell mass path
and density (which are midpoint quantities).
Definitions are as in the above example, with new variables
@var{hyai} and @var{hybi} defined at grid cell vertical
interfaces (rather than midpoints like @var{hyam} and @var{hybm}).
The approach naturally fits into two lines:
@example
@verbatim
cat > ~/pdel.nco << 'EOF'
*prs_ntf[time,lat,lon,ilev]=P0*hyai+PS*hybi;
// Requires NCO 4.5.4 and later:
prs_dlt[time,lat,lon,lev]=prs_ntf(:,:,:,1:$ilev.size-1)-prs_ntf(:,:,:,0:$ilev.size-2);
// Derived variable that require pressure thickness:
// Divide by gravity to obtain total mass path in layer aka mpl [kg m-2] 
mpl=prs_dlt/grv_sfc;
// Multiply by mass mixing ratio to obtain mass path of constituent
mpl_CO2=mpl*mmr_CO2;
EOF
ncap2 -O -v -S ~/pdel.nco ~/nco/data/in.nc ~/foo.nc
ncks -O -C -v prs_dlt ~/foo.nc
@end verbatim
@end example
The first line defines the four-dimensional interface pressures 
@code{prs_ntf} as a @acronym{RAM} variable because those are not desired 
in the output file. 
The second differences each pressure level from the pressure above it
to obtain the pressure difference. 
This line employs both left-hand casting and array hyperslabbing.
However, this syntax only works with @acronym{NCO} version 4.5.4
(November, 2015) and later because earlier versions require that 
@acronym{LHS} and @acronym{RHS} dimension names (not just sizes) match. 
From the pressure differences, one can obtain the mass path in each
layer as shown.

Another reason to cast a variable is to modify the shape or type 
of a variable already in Input 
@example
gds_var[gds_crd]=gds_var.double();
three_dmn_var_crd[lat,lon,lev]=10.0d;
four[]=four.int();
@end example

@html
<a name="ncap_arr"></a> <!-- http://nco.sf.net/nco.html#ncap_arr -->
<a name="array"></a> <!-- http://nco.sf.net/nco.html#array -->
@end html
@node Arrays and hyperslabs, Attributes, Left hand casting, ncap2 netCDF Arithmetic Processor
@subsection Arrays and hyperslabs

@findex array
@cindex @code{array} function
@cindex arrays
@cindex findgen-equivalent
@cindex indgen-equivalent
Generating a regularly spaced n-dimensional array with @command{ncap2}
is simple with the @code{array()} function. 
@noindent The function comes in three (overloaded) forms
@example
(A) var_out=array(val_srt,val_inc,$dmn_nm); // One-dimensional output
(B) var_out=array(val_srt,val_inc,var_tpl); // Multi-dimensional output
(C) var_out=array(val_srt,val_inc,/$dmn1,$dmn2...,$dmnN/); // Multi-dimensional output
@end example
@noindent

@table @dfn
@item val_srt
Starting value of the array. The @var{type} of the array will be the @var{type} of this starting value.
@item val_inc
Spacing (or increment) between elements. 
@item var_tpl
Variable from which the array can derive its shape 1D or nD
@end table

@noindent @strong{One-Dimensional Arrays@*}
Use form (A) or (B) above for 1D arrays:
@example
# var_out will be NC_DOUBLE:
var_out=array(10.0,2,$time) // 10.5,12.5,14.5,16.5,18.5,20.5,22.5,24.5,26.5,28.5

// var_out will be NC_UINT, and "shape" will duplicate "ilev"
var_out=array(0ul,2,ilev) // 0,2,4,6

// var_out will be NC_FLOAT
var_out=array(99.0f,2.5,$lon) // 99,101.5,104,106.5

// Create an array of zeros 
var_out=array(0,0,$time) // 0,0,0,0,0,0,0,0,0,0 

// Create array of ones
var_out=array(1.0,0.0,$lon) // 1.0,1.0,1.0,1.0 
@end example

@noindent @strong{n-Dimensional Arrays@*}
Use form (B) or (C) for creating n-D arrays.@* 
NB: In (C) the final argument is a list of dimensions
@example
// These are equivalent
var_out=array(1.0,2.0,three_dmn_var);
var_out=array(1.0,2.0,/$lat,$lev,$lon/);

// var_out is NC_BYTE
var_out=array(20b, -4, /$lat,$lon/); // 20,16,12,8,4,0,-4,-8  

srt=3.14159f;
inc=srt/2.0f;
var_out(srt,inc,var_2D_rrg);
// 3.14159, 4.712385, 6.28318, 7.853975, 9.42477, 10.99557, 12.56636, 14.13716 ; 
@end example

@cindex hyperslabs
Hyperslabs in @command{ncap2} are more limited than hyperslabs with the
other @acronym{NCO} operators. 
@command{ncap2} does not understand the shell command-line syntax
used to specify multi-slabs, wrapped co-ordinates, negative stride or
coordinate value limits.
However with a bit of syntactic magic they are all are possible. 
@command{ncap2} accepts (in fact, it requires) @var{N}-hyperslab
arguments for a variable of rank @var{N}:
@example
var1(arg1,arg2 ... argN);
@end example
where each hyperslab argument is of the form
@example
start:end:stride 
@end example
and the arguments for different dimensions are separated by commas.
@noindent 
If @var{start} is omitted, it defaults to zero.
If @var{end} is omitted, it defaults to dimension size minus one.
If @var{stride} is omitted, it defaults to one.
@sp 1
@noindent If a single value is present then it is assumed that that
dimension collapses to a single value (i.e., a cross-section). 
The number of hyperslab arguments MUST equal the variable's rank.
@sp 1

@noindent @strong{Hyperslabs on the Right Hand Side of an assign@*}

A simple 1D example:
@example 
@verbatim
($time.size=10)
od[$time]={20,22,24,26,28,30,32,34,36,38};

od(7);     // 34
od(7:);    // 34,36,38
od(:7);    // 20,22,24,26,28,30,32,34 
od(::4);   // 20,28,36
od(1:6:2)  // 22,26,30
od(:)      // 20,22,24,26,28,30,32,34,36,38 
@end verbatim
@end example

A more complex three dimensional example:
@example
@verbatim
($lat.size=2,$lon.size=4)
th[$time,$lat,$lon]=      
                          {1, 2, 3, 4, 5, 6, 7, 8,
                          9,10,11,12,13,14,15,16,
                          17,18,19,20,21,22,23,24,
                          -99,-99,-99,-99,-99,-99,-99,-99,
                          33,34,35,36,37,38,39,40,
                          41,42,43,44,45,46,47,48,
                          49,50,51,52,53,54,55,56,
                          -99,58,59,60,61,62,63,64,
                          65,66,67,68,69,70,71,72,
                          -99,74,75,76,77,78,79,-99 };

th(1,1,3);        // 16
th(2,0,:);        // 17, 18, 19, 20
th(:,1,3);        // 8, 16, 24, -99, 40, 48, 56, 64, 72, -99 
th(::5,:,0:3:2); // 1, 3, 5, 7, 41, 43, 45, 47
@end verbatim
@end example

If hyperslab arguments collapse to a single value (a cross-section has
been specified), then that dimension is removed from the returned
variable. 
If all the values collapse then a scalar variable is returned.
So, for example, the following is valid: 
@example
th_nw=th(0,:,:)+th(9,:,:); 
// th_nw has dimensions $lon,$lat 
// NB: the time dimension has become degenerate
@end example

The following is invalid:
@example
th_nw=th(0,:,0:1)+th(9,:,0:1);
@end example
because the @code{$lon} dimension now only has two elements.
The above can be calculated by using a LHS cast with 
@code{$lon_nw} as replacement dim for @code{$lon}: 
@example
defdim("lon_nw",2);
th_nw[$lat,$lon_nw]=th(0,:,0:1)+th(9,:,0:1);
@end example

@noindent @strong{Hyperslabs on the Left Hand Side of an assign@*}
@noindent When hyperslabing on the LHS, the expression on the RHS must 
evaluate to a scalar or a variable/attribute with the same number of 
elements as the LHS hyperslab.
Set all elements of the last record to zero:
@example
th(9,:,:)=0.0;
@end example
Set first element of each lon element to 1.0:
@example
th(:,:,0)=1.0;
@end example
One may hyperslab on both sides of an assign.
For example, this sets the last record to the first record:
@example
th(9,:,:)=th(0,:,:);
@end example
Say @var{th0} represents pressure at height=0 and 
@var{th1} represents pressure at height=1.
Then it is possible to insert these hyperslabs into the records
@example
prs[$time,$height,$lat,$lon]=0.0;
prs(:,0,:,:)=th0;
prs(:,1,:,:)=th1;
@end example

@noindent @strong{Reverse method}@*
@cindex reverse()
Use the @code{reverse()} method to reverse a dimension's elements in a
variable with at least one dimension.
This is equivalent to a negative stride, e.g., 
@example 
@verbatim
th_rv=th(1,:,:).reverse($lon); // {12,11,10,9 }, {16,15,14,13}
od_rv=od.reverse($time);        // {38,36,34,32,30,28,26,24,22,20}
@end verbatim
@end example

@noindent @strong{Permute method}p@*
@cindex permute()
Use the @code{permute()} method to swap the dimensions of a variable.
The number and names of dimension arguments must match the dimensions in
the variable. 
If the first dimension in the variable is of record type then this must
remain the first dimension. 
If you want to change the record dimension then consider using
@command{ncpdq}. 

Consider the variable:
@example
float three_dmn_var(lat,lev,lon);
three_dmn_var_prm=three_dmn_var.permute($lon,$lat,$lev);
// The permuted values are
three_dmn_var_prm= 
  0,4,8,
  12,16,20,
  1,5,9,
  13,17,21,
  2,6,10,
  14,18,22,
  3,7,11,
  15,19,23;
@end example

@html
<a name="ncap_att"></a> <!-- http://nco.sf.net/nco.html#ncap_att -->
@end html
@node Attributes, Value List, Arrays and hyperslabs, ncap2 netCDF Arithmetic Processor
@subsection Attributes
@cindex attributes@command{ncap2}
Refer to attributes with @emph{var_nm@@att_nm}.
The following are all valid statements:
@example
@verbatim
global@text="Test Attributes"; /* Assign a global variable attribute */
a1[$time]=time*20;
a1@long_name="Kelvin";
a1@min=a1.min();
a1@max=a1.max();
a1@min++;
--a1@max; 
a1(0)=a1@min;
a1($time.size-1)=a1@max;
@end verbatim
@end example

NetCDF allows all attribute types to have a size between one
@code{NC_MAX_ATTRS}. 
Here is the metadata for variable @var{a1}:
@example 
double a1(time) ;
  a1:long_name = "Kelvin" ;
  a1:max = 199. ;
  a1:min = 21. ;
  a1:trip1 = 1, 2, 3 ;
  a1:triplet = 21., 110., 199. ;
@end example

These basic methods can be used with attributes:
@code{size()}, @code{type()}, and @code{exists()}.
For example, to save an attribute text string in a variable:
@example
@verbatim
defdim("sng_len",a1@long_name.size());
sng_arr[$sng_len]=a1@long_name; // sng_arr now contains "Kelvin" 
@end verbatim
@end example
Attributes defined in a script are stored in memory and are written to
the output file after script completion. 
To stop the attribute being written use the @code{ram_delete()} method
or use a bogus variable name. 

@noindent @strong{Attribute Propagation and Inheritance}
@cindex attribute propagation
@cindex attribute inheritance
@itemize @bullet
  @item Attribute propagation occurs in a regular assign statement. The variable being defined on the LHS gets copies of the attributes from the leftermost variable on the RHS.
  @item Attribute Inheritance: The LHS variable ``inherits'' attributes from an Input variable with the same name
  @item It is possible to have a regular assign statement for which both propagation and inheritance occur.
@end itemize

@example
// prs_mdp inherits attributes from P0:
prs_mdp[time,lat,lon,lev]=P0*hyam+hybm*PS;
// th_min inherits attributes from three_dmn_var_dbl:
th_min=1.0 + 2*three_dmn_var_dbl.min($time);
@end example

@noindent @strong{Attribute Concatenation@*}
@cindex attribute concatenation
@cindex push

The push() function concatenates attributes, or appends an ``expression'' to a pre-existing attribute.
It comes in two forms
@example
(A) att_new=push(att_exp, expr)
(B) att_size=push(&att_nm,expr)
@end example

@noindent In form (A) The first argument should be an attribute
identifier or an expression that evaluates to an attribute. 
The second argument can evalute to an attribute or a variable. 
The second argument is then converted to the type of @var{att_exp}; 
and appended to @var{att_exp} ; and the resulting attribute is returned.@*

@noindent In form (B) the first argument is a call-by-reference
attribute identifier (which may not yet exist). 
The second argument is then evaluated (and type-converted as needed) and
appended to the call-by-reference atttribute. 
The final size of the attribute is then returned. 

@example
@verbatim
temp@range=-10.0;
push(&temp@range,12.0); // temp@range=-10.0,12.0

numbers@squares=push(1,4);
numbers@squares=push(numbers@squares,9);
push(&number@squares,16.0); 
push(&number@squares,25ull); // numbers@squares=1,4,9,16,25  
@end verbatim
@end example

@noindent Now some text examples.@* 
Remember, an atttribute identifier that begins with @@ implies a global
attribute.
For example, '@@institution' is short for 'global@@institution'.
@example
@verbatim
global@greetings=push("hello"," world !!");
global@greek={"alpha"s,"beta"s,"gamma"s};
// Append an NC_STRING
push(&@greek,"delta"s);
// Pushing an NC_CHAR to a NC_STRING attribute is allowed, it is converted to an an NC_CHAR
@e="epsilon";
push(&@greek,@e);
push(&@greek,"zeta"); 

// Pushing a single NC_STRING to an NC_CHAR is not allowed
@h="hello";
push(&@h," again"s); // BAD PUSH
@end verbatim
@end example

If the attribute name contains non-regular characters use ID quoting:
@example
@verbatim
'b..m1@c--lost'=23;
@end verbatim
@end example
See @pxref{ID Quoting}.

@html
<a name="value_list"></a> <!-- http://nco.sf.net/nco.html#value_list -->
<a name="initialize"></a> <!-- http://nco.sf.net/nco.html#initialize -->
@end html
@node Value List, Number literals, Attributes, ncap2 netCDF Arithmetic Processor
@subsection Value List
@cindex value list@command{ncap2}

A @emph{value list} is a special type of attribute. 
It can only be used on the RHS of the assign family of statements.@*
That is @emph{=, +=, -=, *=, /=}@*
A value list CANNOT be involved in any logical, binary, or arithmetical operations (except those above).@* 
A value list CANNOT be used as a function argument.@*
A value list CANNOT have nested value lists.@*
The type of a value list is the type of the member with the highest type.@*

@cindex value list
@example
@verbatim
a1@trip={1,2,3};
a1@trip+={3,2,1}; // 4,4,4
a1@triplet={a1@min,(a1@min+a1@max)/2,a1@max}; 
lon[lon]={0.0,90.0,180.0,270.0};
lon*={1.0,1.1,1.2,1.3} 
dlon[lon]={1b,2s,3ull,4.0f}; // final type NC_FLOAT

a1@ind={1,2,3}+{4,4,4}; // BAD
a1@s=sin({1.0,16.0}); // BAD
@end verbatim
@end example

@noindent One can also use a value_list to create an attribute of type
NC_STRING. 
Remember, a literal string of type NC_STRING has a postfix 's'. 
A value list of NC_CHAR has no semantic meaning and is plain wrong.   
@example
@verbatim
array[lon]={1.0,2.,4.0,7.0};
array@numbers={"one"s, "two"s, "four"s, "seven"s}; // GOOD

ar[lat]={0,20} 
ar@numbers={"zero","twenty"}; // BAD
@end verbatim
@end example

@html
<a name="ncap_num"></a> <!-- http://nco.sf.net/nco.html#ncap_num -->
<a name="ncap_string"></a> <!-- http://nco.sf.net/nco.html#ncap_string -->
@end html
@node Number literals, if statement, Value List, ncap2 netCDF Arithmetic Processor
@subsection Number literals
@cindex number literals @command{ncap2}
The table below lists the postfix character(s) to add to a number
literal (aka, a naked constant) for explicit type specification.
The same type-specification rules are used for variables and
attributes. 
A floating-point number without a postfix defaults to @code{NC_DOUBLE}, 
while an integer without a postfix defaults to type @code{NC_INT}:
@example
@verbatim
var[$rlev]=0.1;     // Variable will be type NC_DOUBLE
var[$lon_grd]=2.0;  // Variable will be type NC_DOUBLE
var[$gds_crd]=2e3;  // Variable will be type NC_DOUBLE
var[$gds_crd]=2.0f; // Variable will be type NC_FLOAT (note "f")
var[$gds_crd]=2e3f; // Variable will be type NC_FLOAT (note "f")
var[$gds_crd]=2;    // Variable will be type NC_INT
var[$gds_crd]=-3;   // Variable will be type NC_INT
var[$gds_crd]=2s;   // Variable will be type NC_SHORT
var[$gds_crd]=-3s;  // Variable will be type NC_SHORT
var@att=41.;        // Attribute will be type NC_DOUBLE
var@att=41.f;       // Attribute will be type NC_FLOAT
var@att=41;         // Attribute will be type NC_INT
var@att=-21s;       // Attribute will be type NC_SHORT  
var@units="kelvin"; // Attribute will be type NC_CHAR
@end verbatim
@end example 
There is no postfix for characters, use a quoted string instead for 
@code{NC_CHAR}.
@command{ncap2} interprets a standard double-quoted string as a value
of type @code{NC_CHAR}. 
In this case, any receiving variable must be dimensioned as an array
of @code{NC_CHAR} long enough to hold the value.

To use the newer netCDF4 types @acronym{NCO} must be compiled/linked to
the netCDF4 library and the output file must be of type @code{NETCDF4}:
@example
@verbatim
var[$time]=1UL;    // Variable will be type @code{NC_UINT}
var[$lon]=4b;      // Variable will be type @code{NC_BYTE}
var[$lat]=5ull;    // Variable will be type @code{NC_UINT64}  
var[$lat]=5ll;     // Variable will be type @code{NC_INT64}  
var@att=6.0d;      // Attribute will be type @code{NC_DOUBLE}
var@att=-666L;     // Attribute will be type @code{NC_INT}
var@att="kelvin"s; // Attribute will be type @code{NC_STRING} (note the "s")
@end verbatim
@end example
@cindex @code{NC_CHAR}
@cindex @code{NC_STRING}
Use a post-quote @samp{s} for @code{NC_STRING}.
Place the letter @samp{s} immediately following the double-quoted string
to indicate that the value is of type @code{NC_STRING}.
In this case, the receiving variable need not have any memory allocated
to hold the string because netCDF4 handles that memory allocation.

Suppose one creates a file containing an ensemble of model results, and
wishes to label the record coordinate with the name of each model.
The @code{NC_STRING} type is well-suited to this because it facilitates
storing arrays of strings of arbitrary length.
This is sophisticated, though easy with @command{ncap2}:
@example 
@verbatim
% ncecat -O -u model cesm.nc ecmwf.nc giss.nc out.nc
% ncap2 -4 -O -s 'model[$model]={"cesm"s,"ecmwf"s,"giss"s}' out.nc out.nc
@end verbatim
@end example 
The key here to place an @samp{s} character after each double-quoted
string value to indicate an @code{NC_STRING} type. 
The @samp{-4} ensures the output filetype is netCDF4 in case the input
filetype is not. 

@table @asis
@item @strong{netCDF3/4 Types}
@item b|B	 
  @code{NC_BYTE}, a signed 1-byte integer 
@item none	 
  @code{NC_CHAR}, an ISO/ASCII character 
@item s|S	 
  @code{NC_SHORT}, a signed 2-byte integer 
@item l|L	 
  @code{NC_INT}, a signed 4-byte integer 
@item f|F	 
  @code{NC_FLOAT}, a single-precision (4-byte) floating-point number 
@item d|D
  @code{NC_DOUBLE}, a double-precision (8-byte) floating-point number 
@item @strong{netCDF4 Types}
@item ub|UB	 
  @code{NC_UBYTE}, an unsigned 1-byte integer 
@item us|US 
  @code{NC_USHORT}, an unsigned 2-byte integer 
@item u|U|ul|UL	 
  @code{NC_UINT}, an unsigned 4-byte integer 
@item ll|LL	 
  @code{NC_INT64}, a signed 8-byte integer 
@item ull|ULL 
  @code{NC_UINT64}, an unsigned 8-byte integer 
@item s 
  @code{NC_STRING}, a string of arbitrary length
@end table

@html
<a name="ncap_if"></a> <!-- http://nco.sf.net/nco.html#ncap_if -->
@end html
@node if statement, Print & String methods, Number literals, ncap2 netCDF Arithmetic Processor
@subsection if statement
@cindex if() 
The syntax of the if statement is similar to its C counterpart. 
The @emph{Conditional Operator (ternary operator)} has also been
implemented. 
@example
@verbatim
if(exp1)
   stmt1;
else if(exp2)     
   stmt2;
else
   stmt3;

# Can use code blocks as well:
if(exp1){
   stmt1;
   stmt1a;
   stmt1b;
}else if(exp2)     
   stmt2; 
else{
   stmt3;
   stmt3a;
   stmt3b;
}   
@end verbatim
@end example

@comment Truth
@noindent For a variable or attribute expression to be logically true
all its non-missing value elements must be logically true, i.e.,
non-zero. 
The expression can be of any type. 
@w{Unlike C} there is no short-circuiting of an expression with the 
OR (@code{||}) and AND (@code{&&}) operators. 
The whole expression is evaluated regardless if one of the AND/OR
operands are True/False.
@example
@verbatim
# Simple example
if(time > 0)
  print("All values of time are greater than zero\n");
else if(time < 0)
  print("All values of time are less than zero\n");   
else {
  time_max=time.max();
  time_min=time.min();
  print("min value of time=");print(time_min,"%f");
  print("max value of time=");print(time_max,"%f");
}

# Example from ddra.nco
if(fl_typ == fl_typ_gcm){
  var_nbr_apx=32;
  lmn_nbr=1.0*var_nbr_apx*varsz_gcm_4D; /* [nbr] Variable size */
  if(nco_op_typ==nco_op_typ_avg){
    lmn_nbr_avg=1.0*var_nbr_apx*varsz_gcm_4D; // Block size
    lmn_nbr_wgt=dmnsz_gcm_lat; /* [nbr] Weight size */
  } // !nco_op_typ_avg
}else if(fl_typ == fl_typ_stl){
  var_nbr_apx=8;
  lmn_nbr=1.0*var_nbr_apx*varsz_stl_2D; /* [nbr] Variable size */
  if(nco_op_typ==nco_op_typ_avg){
    lmn_nbr_avg=1.0*var_nbr_apx*varsz_stl_2D; // Block size
    lmn_nbr_wgt=dmnsz_stl_lat; /* [nbr] Weight size */
  } // !nco_op_typ_avg
} // !fl_typ
@end verbatim
@end example

@noindent @strong{Conditional Operator@*}
@example
// netCDF4 needed for this example
th_nw=(three_dmn_var_sht >= 0 ? three_dmn_var_sht.uint() : \
       three_dmn_var_sht.int()); 
@end example

@html
<a name="ncap_prn"></a> <!-- http://nco.sf.net/nco.html#ncap_prn -->
@end html
@node Print & String methods, Missing values ncap2, if statement, ncap2 netCDF Arithmetic Processor
@subsection Print & String methods
@cindex print() @command{ncap2}

The print statement comes in a variety of forms:
@example
(A)   print(variable_name, format string?);
(A1)  print(expression/string, format string?);

(B)   sprint(expression/string, format string?);
(B1)  sprint4(expression/string, format string?);
@end example  

@noindent @strong{print() @*@*}
@noindent If the variable exists in I/O then it is printed in a similar fashion to @code{ncks -H}.
@example
@verbatim
print(lon);
lon[0]=0 
lon[1]=90 
lon[2]=180 
lon[3]=270 

print(byt_2D)
lat[0]=-90 lon[0]=0 byt_2D[0]=0 
lat[0]=-90 lon[1]=90 byt_2D[1]=1 
lat[0]=-90 lon[2]=180 byt_2D[2]=2 
lat[0]=-90 lon[3]=270 byt_2D[3]=3 
lat[1]=90 lon[0]=0 byt_2D[4]=4 
lat[1]=90 lon[1]=90 byt_2D[5]=5 
lat[1]=90 lon[2]=180 byt_2D[6]=6 
lat[1]=90 lon[3]=270 byt_2D[7]=7 
@end verbatim
@end example

@noindent If the first argument is NOT a variable the form (A1) is invoked.
@example
@verbatim
print(mss_val_fst@_FillValue);
mss_val_fst@_FillValue, size = 1 NC_FLOAT, value = -999

print("This function \t is monotonic\n");
This function is 	  monotonic

print(att_var@float_att)
att_var@float_att, size = 7 NC_FLOAT, value = 73, 72, 71, 70.01, 69.001, 68.01, 67.01

print(lon*10.0)
lon, size = 4 NC_DOUBLE, value = 0, 900, 1800, 2700
@end verbatim
@end example

@noindent If the format string is specified then the results from (A) and (A1) forms are the same
@example
@verbatim
print(lon_2D_rrg,"%3.2f,");
0.00,0.00,180.00,0.00,180.00,0.00,180.00,0.00,

print(lon*10.0,"%g,")
0,900,1800,2700,

print(att_var@float_att,"%g,")
73,72,71,70.01,69.001,68.01,67.01,
@end verbatim
@end example

@noindent @strong{sprint() & sprint4() @*@*}
These functions work in an identical fashion to (A1) except that @code{sprint()} outputs a regular netCDF3 @code{NC_CHAR} attribute 
and @code{sprint4()} outputs a netCDF4 @code{NC_STRING} attribute
@example
@verbatim
time@units=sprint(yyyy,"days since %d-1-1")
bnd@num=sprint4(bnd_idx,"Band number=%d")

time@arr=sprint4(time,"%.2f,") // "1.00,2.00,3.00,4.00,5.00,6.00,7.00,8.00,9.00,10.00,"
@end verbatim
@end example

@noindent You can also use @code{sprint4()} to convert a @code{NC_CHAR} string to a @code{NC_STRING} string 
and @code{sprint()} to convert a @code{NC_STRING} to a @code{NC_CHAR}
@example
@verbatim
lat_1D_rct@long_name = "Latitude for 2D rectangular grid stored as 1D arrays"; // 

// convert to NC_STRING
lat_1D_rct@long_name = sprint4(lat_1D_rct@long_name) 
@end verbatim
@end example

@noindent @strong{hyperslab a netCDF string @*@*}
Its possible to index-into a NC_CHAR string. Just like a C-String. Remember  an NC_CHAR string is  has no terminating null.
You CANNOT index into a NC_STRING. You have to convert to an NC_CHAR first. 
@example
@verbatim
global@greeting="hello world!!!"
@h=@greeting(0:4);  // "hello"
@w=@greeting(6:11); // "world"

// can use negative inidices
@x=@greeting(-3:-1);  // "!!!"

// can  use stride
@n=@greeting(::2);  // "hlowrd!"

// concatenation
global@new_greeting=push(@h, " users !!!"); // "hello users!!!"

@institution="hotel california"s; 
@h=@institution(0:4); // BAD 

// convert NC_STRING to NC_CHAR
@is=sprint(@institution);
@h=@is(0:4);  // "hotel"

// convert NC_CHAR to NC_STRING
@h=sprint4(@h);
@end verbatim
@end example

@noindent @strong{get_vars_in() & get_vars_out()}
@example
att_lst=get_vars_in(att_regexp?) 
att_lst=get_vars_out(att_regexp?) 
@end example

These functions are used to create a list of vars in Input or Output. The optional arg 'att_regexp'. Can be an NC_CHAR att or a NC_STRING att. If NC_CHAR then only a single reg-exp can be specified. If NC_STRING then multiple reg-exp can be specified.  The output is allways an NC_STRING att. The matching works in an identical fashion to the -v switch in ncks. if there is no arg then all vars are returned.
@example
@verbatim
@slist=get_vars_in("^time");  // "time", "time_bnds", "time_lon", "time_udunits"
// Use NC_STRINGS
@regExp={".*_bnd"s,".*_grd"s}
@slist=get_vars_in(@regExp);  // "lat_bnd", "lat_grd", "lev_bnd", "lon_grd", "time_bnds", "cnv_CF_grd"
@end verbatim
@end example

@html
<a name="ncap_miss"></a> <!-- http://nco.sf.net/nco.html#ncap_miss -->
@end html
@node Missing values ncap2, Methods and functions, Print & String methods, ncap2 netCDF Arithmetic Processor
@subsection Missing values ncap2
@cindex missing values ncap2
Missing values operate slightly differently in @command{ncap2} 
Consider the expression where op is any of the following operators (excluding '=')
@example
Arithmetic operators ( * / % + - ^ )
Binary Operators     ( > >= < <= == != == || && >> << ) 
Assign Operators     ( += -= /= *= ) 

var1 'op' var2
@end example

@noindent If var1 has a missing value then this is the value used in the 
operation, otherwise the missing value for var2 is used. 
If during the element-by-element operation an element from either
operand is equal to the missing value then the missing value is carried 
through. 
In this way missing values 'percolate' or propagate through an
expression.@*  
Missing values associated with Output variables are stored in memory and
are written to disk after the script finishes. 
During script execution its possible (and legal) for the missing value
of a variable to take on several different values. 
@example 
# Consider the variable:
int rec_var_int_mss_val_int(time); =-999,2,3,4,5,6,7,8,-999,-999;
rec_var_int_mss_val_int:_FillValue = -999;

n2=rec_var_int_mss_val_int + rec_var_int_mss_val_int.reverse($time); 

n2=-999,-999,11,11,11,11,11,11,999,-999;
@end example

@html
<a name="missing"></a> <!-- http://nco.sf.net/nco.html#missing -->
<a name="mask_miss"></a> <!-- http://nco.sf.net/nco.html#mask_miss -->
<a name="has_miss"></a> <!-- http://nco.sf.net/nco.html#has_miss -->
<a name="delete_miss"></a> <!-- http://nco.sf.net/nco.html#delete_miss -->
<a name="set_miss"></a> <!-- http://nco.sf.net/nco.html#set_miss -->
<a name="get_miss"></a> <!-- http://nco.sf.net/nco.html#get_miss -->
<a name="change_miss"></a> <!-- http://nco.sf.net/nco.html#change_miss -->
<a name="number_miss"></a> <!-- http://nco.sf.net/nco.html#number_miss -->
@end html

The following methods query or manipulate missing value (aka
@code{_FillValue} information associated with a variable.
The methods that ``manipulate'' only succeed on variables in Output.
@table @code
@item set_miss(expr)
@cindex @code{set_miss()}
 The numeric argument @var{expr} becomes the new missing value,
 overwriting the old missing value, if any.
 The argument given is converted if necessary to the variable's type.
 NB: This only changes the missing value attribute.
 Missing values in the original variable remain unchanged, and thus 
 are no long considered missing values.
 They are effectively ``orphaned''.
 Thus @code{set_miss()} is normally used only when creating new
 variables.
 The intrinsic function @code{change_miss()} (see below) is typically 
 used to edit values of existing variables.
@item change_miss(expr)
@cindex @code{change_miss()}
 Sets or changes (any pre-existing) missing value attribute and missing 
 data values to @var{expr}. 
 NB: This is an expensive function since all values must be examined. 
 Use this function when changing missing values for pre-existing
 variables. 
@item get_miss() 
@cindex @code{get_miss()}
 Returns the missing value of a variable. 
 If the variable exists in Input and Output then the missing value of
 the variable in Output is returned. 
 If the variable has no missing value then an error is returned.   
@item delete_miss()
@cindex @code{delete_miss()}
 Delete the missing value associated with a variable.
@item number_miss()
@cindex @code{number_miss()}
 Count the number of missing values a variable contains.
@item has_miss()
@cindex @code{has_miss()}
Returns 1 (True) if the variable has a missing value associated with it. 
else returns 0 (False)
@item missing()
@cindex @code{missing(), mask_miss()}
This function creates a True/False mask array of where the missing value is set.
It is syntatically equivalent to @code{(var_in == var_in.get_miss())},
except that requires deleting the missing value before-hand.
@end table

@example
@verbatim
th=three_dmn_var_dbl;
th.change_miss(-1e10d);
/* Set values less than 0 or greater than 50 to missing value */
where(th < 0.0 || th > 50.0) th=th.get_miss();

# Another example:
new[$time,$lat,$lon]=1.0;
new.set_miss(-997.0);

// Extract all elements evenly divisible by 3
where (three_dmn_var_dbl%3 == 0)
     new=three_dmn_var_dbl; 
elsewhere
     new=new.get_miss();   

// Print missing value and variable summary
mss_val_nbr=three_dmn_var_dbl.number_miss();
print(three_dmn_var_dbl@_FillValue);
print("Number of missing values in three_dmn_var_dbl: ");
print(mss_val_nbr,"%d");
print(three_dmn_var_dbl);

// Find total number of missing values along dims $lat and $lon
mss_ttl=three_dmn_var_dbl.missing().ttl($lat,$lon);
print(mss_ttl); // 0, 0, 0, 8, 0, 0, 0, 1, 0, 2 ;
@end verbatim
@end example

@table @code
@item simple_fill_miss(var)
@cindex @code{simple_fill_miss()}
This function takes a variable and attempts to fill missing values using an average
of up to the 4 nearest neighbour grid points. The method used is iterative (up to 1000 cycles).
For very large areas of missing values results can be unpredictable.
The given variable must be at least 2D; and the algorithm assumes that the last two dims are lat/lon
or y/x
@item weighted_fill_miss(var)
@cindex @code{weighted_fill_miss()}
Weighted_fill_miss is more sophisticated. Up to 8 nearest neighbours  are used to calculate a weighted average.
The weighting used is the inverse  square of distance. Again the method is iterative (up to 1000 cycles).
The area filled is defined by the final  two dims of the variable. In addition this function assumes the existance of
coordinate vars the same name as the last two dims. if it doesn't find these dims it will gently exit with warning.
@end table

@html
<a name="ncap_mth"></a> <!-- http://nco.sf.net/nco.html#ncap_mth -->
<a name="ncap2_mth"></a> <!-- http://nco.sf.net/nco.html#ncap2_mth -->
@end html
@node Methods and functions, RAM variables, Missing values ncap2, ncap2 netCDF Arithmetic Processor
@subsection Methods and functions

The convention within this document is that methods can be used as 
functions. 
However, functions are not and cannot be used as methods.
Methods can be daisy-chained d and their syntax is cleaner than functions. 
Method names are reserved words and CANNOT be used as variable names.  
The command @code{ncap2 -f} shows the complete list of methods available
on your build. 
@example
n2=sin(theta) 
n2=theta.sin() 
n2=sin(theta)^2 + cos(theta)^2 
n2=theta.sin().pow(2) + theta.cos()^2
@end example

This statement chains together methods to convert three_dmn_var_sht to
type double, average it, then convert this back to type short: 
@example
three_avg=three_dmn_var_sht.double().avg().short();
@end example

@sp 1
@noindent @strong{Aggregate Methods @*} 
@noindent These methods mirror the averaging types available in @command{ncwa}. The arguments to the methods are the dimensions to average over. Specifying no dimensions is equivalent to specifying all dimensions i.e., averaging over all dimensions. A masking variable and a weighting variable can be manually created and applied as needed.

@table @code
@item avg()
@cindex avg()
Mean value 
@item sqravg()
@cindex sqravg()
Square of the mean
@item avgsqr()
Mean of sum of squares
@item max()
@cindex max()
Maximum value
@item min()
@cindex min()
Minimum value
@item mabs()
@cindex mabs()
Maximum absolute value
@item mebs()
@cindex mebs()
Mean absolute value
@item mibs()
@cindex mibs()
Minimum absolute value
@item rms()
Root-mean-square (normalize by @var{N})
@item rmssdn()
@cindex rmssdn()
Root-mean square (normalize by @var{N-1})
@item tabs() or ttlabs()
@cindex tabs()
Sum of absolute values
@item ttl() or total() or sum()
@cindex ttl()
Sum of values
@end table

@example
// Average a variable over time
four_time_avg=four_dmn_rec_var($time);
@end example

@sp 1
@noindent @strong{Packing Methods @*}
For more information see @pxref{Packed data} and @pxref{ncpdq netCDF Permute Dimensions Quickly}@*
@table @code
@item pack() & pack_short()
@cindex pack()
The default packing algorithm is applied and variable is packed to @code{NC_SHORT}
@item pack_byte()
@cindex pack_byte()
Variable is packed to @code{NC_BYTE}
@item pack_short()
@cindex pack_short()
Variable is packed to @code{NC_SHORT}
@item pack_int()
@cindex pack_int()
Variable is packed to @code{NC_INT}
@item unpack()
@cindex unpack()
The standard unpacking algorithm is applied. 
@end table
@acronym{NCO} automatically unpacks packed data before arithmetically
modifying it. 
After modification @acronym{NCO} stores the unpacked data.
To store it as packed data again, repack it with, e.g., the 
@code{pack()} function.
To ensure that @code{temperature} is packed in the output file,
regardless of whether it is packed in the input file, one uses, e.g.,
@example
ncap2 -s 'temperature=pack(temperature-273.15)' in.nc out.nc
@end example

All the above pack functions also take the additional two arguments
@code{scale_factor, add_offset}.
Both arguments must be included:
@example
ncap2 -v -O -s 'rec_pck=pack(three_dmn_rec_var,-0.001,40.0);' in.nc foo.nc
@end example

@strong{Basic Methods @*}
These methods work with variables and attributes. They have no arguments.
@table @code
@item size()	
@cindex size()
Total number of elements 
@item ndims()
@cindex ndims()
Number of dimensions in variable
@item type() 
@cindex type()
Returns the netcdf type (see previous section)
@item exists()
@cindex exists()
Return 1 (true) if var or att is present in I/O else return 0 (false)
@item getdims()
@cindex getdims()
Returns an NC_STRING attribute of all the dim names of a variable
@end table

@sp 1
@noindent @strong{Utility Methods @*}
These functions are used to manipulate missing values and @acronym{RAM} variables.
@pxref{Missing values ncap2} 

@table @code
@item set_miss(expr)
 Takes one argument, the missing value. Sets or overwrites the existing
 missing value. The argument given is converted if necessary to the
 variable type. (NB: pre-existing missing values, if any, are not converted).
@item change_miss(expr)
 Changes the missing value elements of the variable to the new missing
 value (NB: an expensive function).
@item get_miss() 
 Returns the missing value of a variable in Input or Output  
@item delete_miss()
 Deletes the missing value associated with a variable.
@item has_miss()
 Returns 1 (True) if the variable has a missing else returns 0 (False)
@item number_miss
 Returns the number of missing values a variable contains
@item ram_write()
 Writes a @acronym{RAM} variable to disk i.e., converts it to a regular disk type variable
@item ram_delete()
 Deletes a @acronym{RAM} variable or an attribute 
@end table

@sp 1
@noindent @strong{PDQ Methods@*}
See @pxref{ncpdq netCDF Permute Dimensions Quickly}
@table @code
@item reverse(dim args)
 Reverse the dimension ordering of elements in a variable. 
@item permute(dim args)
 Re-shape variables by re-ordering the dimensions. 
 All the dimensions of the variable must be specified in the
 arguments. 
 A limitation of this permute (unlike @command{ncpdq}) is that the
 record dimension cannot be re-assigned.  
@end table 
// Swap dimensions about and reorder along lon
@example
lat_2D_rrg_new=lat_2D_rrg.permute($lon,$lat).reverse($lon);
lat_2D_rrg_new=0,90,-30,30,-30,30,-90,0
@end example

@sp 1
@noindent @strong{Type Conversion Methods and Functions@*}
@noindent These methods allow @command{ncap2} to convert variables and
attributes to the different netCDF types. 
For more details on automatic and manual type conversion see
(@pxref{Type Conversion}). 
netCDF4 types are only available if you have compiled/links
@acronym{NCO} with the netCDF4 library and the Output file is
@acronym{HDF5}.
@table @code
@item @strong{netCDF3/4 Types}
@item byte()	 
@cindex byte()
 convert to @code{NC_BYTE},  a signed 1-byte integer 
@item char()
@cindex char()	 
 convert to @code{NC_CHAR},  an ISO/ASCII character
@item short()	
@cindex sshort() 
 convert to @code{NC_SHORT}, a signed 2-byte integer 
@item int()	 
@cindex int()
 convert to @code{NC_INT},   a signed 4-byte integer 
@item float()
@cindex float()	 
 convert to @code{NC_FLOAT}, a single-precision (4-byte) floating-point number 
@item double() 
@cindex double()
 convert to @code{NC_DOUBLE}, a double-precision (8-byte) floating-point number 
@item @strong{netCDF4 Types}
@item ubyte()	 
@cindex ubyte()
 convert to @code{NC_UBYTE}, an unsigned 1-byte integer 
@item ushort() 
@cindex ushort()
 convert to @code{NC_USHORT}, an unsigned 2-byte integer 
@item uint()
@cindex uint()	 
 convert to @code{NC_UINT}, an unsigned 4-byte integer 
@item int64()	
@cindex int64() 
 convert to @code{NC_INT64}, a signed 8-byte integer 
@item uint64() 
@cindex unit64()
 convert to @code{NC_UINT64}, an unsigned 8-byte integer
@end table

You can also use the @code{convert()} method to do type conversion. 
This takes an integer agument. 
For convenience, @command{ncap2} defines the netCDF pre-processor tokens
as @acronym{RAM} variables. 
For example you may wish to convert a non-floating point variable to the
same type as another variable. 
@example
lon_type=lon.type();
if(time.type() != NC_DOUBLE && time.type() != NC_FLOAT) 
   time=time.convert(lon_type);
@end example

@noindent @strong{Intrinsic Mathematical Methods @*}
The list of mathematical methods is system dependant.
For the full list @pxref{Intrinsic mathematical methods} 

All the mathematical methods take a single argument except @code{atan2()}
and @code{pow()} which take two. 
If the operand type is less than @emph{float} then the result will be of
type @emph{float}. 
Arguments of type @emph{double} yield results of type @emph{double}. 
Like the other methods, you are free to use the mathematical methods as functions. 

@example
n1=pow(2,3.0f)    // n1 type float
n2=atan2(2,3.0)   // n2 type double
n3=1/(three_dmn_var_dbl.cos().pow(2))-tan(three_dmn_var_dbl)^2; // n3 type double
@end example

@html
<a name="ncap_ram"></a> <!-- http://nco.sf.net/nco.html#ncap_ram -->
@end html
@cindex @acronym{RAM} variables
@node RAM variables, Where statement, Methods and functions, ncap2 netCDF Arithmetic Processor
@subsection @acronym{RAM} variables
Unlike regular variables, @acronym{RAM} variables are never written to disk.
Hence using @acronym{RAM} variables in place of regular variables (especially
within loops) significantly increases execution speed.
Variables that are frequently accessed within @code{for} or @code{where}
clauses provide the greatest opportunities for optimization. 
To declare and define a @acronym{RAM} variable simply prefix the variable name
with an asterisk (@code{*}) when the variable is declared/initialized.
To delete @acronym{RAM} variables (and recover their memory) use the
@code{ram_delete()} method. 
To write a @acronym{RAM} variable to disk (like a regular variable) use
@code{ram_write()}. 
@cindex ram_write()
@cindex ram_delete()
@example
*temp[$time,$lat,$lon]=10.0;     // Cast
*temp_avg=temp.avg($time);      // Regular assign
temp.ram_delete();              // Delete RAM variable
temp_avg.ram_write();           // Write Variable to output

// Create and increment a RAM variable from "one" in Input
*one++;   
// Create RAM variables from the variables three and four in Input.
// Multiply three by 10 and add it to four. 
*four+=*three*=10; // three=30, four=34 
@end example

@html
<a name="where"></a> <!-- http://nco.sf.net/nco.html#where -->
<a name="ncap_whr"></a> <!-- http://nco.sf.net/nco.html#ncap_whr -->
<a name="ncap_where"></a> <!-- http://nco.sf.net/nco.html#ncap_where -->
@end html
@node Where statement, Loops, RAM variables, ncap2 netCDF Arithmetic Processor
@subsection Where statement
@cindex where()
The @code{where()} statement combines the definition and application of
a mask and can lead to succinct code.  
The syntax of a @code{where()} statement is:
@example
@verbatim
// Single assign ('elsewhere' is optional)
where(mask) 
   var1=expr1;
elsewhere
   var1=expr2;	   	

// Multiple assigns
where(mask){
    var1=expr1;
    var2=expr2;
    ...
}elsewhere{
    var1=expr3
    var2=expr4
    var3=expr5;
    ...
}
@end verbatim
@end example

@itemize @bullet
@item The only expression allowed in the predicate of a where is assign,
i.e., 'var=expr'. 
This assign differs from a regular @command{ncap2} assign. 
The LHS var must already exist in Input or Output. 
The RHS expression must evaluate to a scalar or a variable/attribute of
the same size as the LHS variable.
@item Consider when both the LHS and RHS are variables: 
For every element where mask condition is True, the corresponding LHS
variable element is re-assigned to its partner element on the RHS. 
In the elsewhere part the mask is logically inverted and the assign
process proceeds as before.
@item If the mask dimensions are a subset of the LHS variable's
dimensions, then it is made to conform; if it cannot be made to conform 
then script execution halts.   
@item Missing values in the mask evaluate to False in the where 
code/block statement and to True in the elsewhere block/statement. 
@item LHS variable elements set to missing value are treated just like any other
 elements and can be re-assigned as the mask dictates
@item LHS variable cannot include subscripts.
If they do script execution will terminate. 
See below example for work-araound.
@end itemize

Consider the variables @code{float lon_2D_rct(lat,lon);} and
@code{float var_msk(lat,lon);}. 
Suppose we wish to multiply by two the elements for which @code{var_msk} 
@w{equals 1}: 
@example
where(var_msk == 1) lon_2D_rct=2*lon_2D_rct;
@end example
Suppose that we have the variable @code{int RDM(time)} and that we want
to set its values less than 8 or greater than 80 @w{to 0}:
@example
where(RDM < 8 || RDM > 80) RDM=0;          
@end example

To use @command{where} on a variable hyperslab, define and use a temporary
variable, e.g., 
@example
@verbatim
*var_tmp=var2(:,0,:,:); 
where (var1 < 0.5) var_tmp=1234; 
var2(;,0,:,;)=var_tmp;
ram_delete(var_tmp);
@end verbatim
@end example

@html
<a name="WRF"></a> <!-- http://nco.sf.net/nco.html#WRF -->
<a name="SLD"></a> <!-- http://nco.sf.net/nco.html#SLD -->
<a name="wrf"></a> <!-- http://nco.sf.net/nco.html#wrf -->
<a name="sld"></a> <!-- http://nco.sf.net/nco.html#sld -->
@end html
@cindex Weather and Research Forecast (@acronym{WRF}) Model
@cindex Swath-like Data (@acronym{SLD})
@cindex @acronym{WRF} (Weather and Research Forecast Model)
@cindex @acronym{SLD} (Swath-like Data)
Consider irregularly gridded data, described using @w{rank 2} coordinates: 
@code{double lat(south_north,east_west)},
@code{double lon(south_north,east_west)}, 
@code{double temperature(south_north,east_west)}.
This type of structure is often found in regional weather/climate model
(such as @acronym{WRF}) output, and in satellite swath data.
For this reason we call it ``Swath-like Data'', or @acronym{SLD}.
To find the average temperature in a region bounded by
[@var{lat_min},@var{lat_max}] and [@var{lon_min},@var{lon_max}]:
@example
@verbatim
temperature_msk[$south_north,$east_west]=0.0;
where((lat >= lat_min && lat <= lat_max) && (lon >= lon_min && lon <= lon_max))
  temperature_msk=temperature;	
elsewhere
  temperature_msk=temperature@_FillValue;

temp_avg=temperature_msk.avg();
temp_max=temperature.max();
@end verbatim
@end example

@html
<a name="NARR"></a> <!-- http://nco.sf.net/nco.html#NARR -->
<a name="narr"></a> <!-- http://nco.sf.net/nco.html#narr -->
@end html
@cindex @acronym{NARR} (North American Regional Reanalysis)a
@cindex North American Regional Reanalysis (@acronym{NARR})
For North American Regional Reanalysis (@acronym{NARR}) data
(example
@uref{http://dust.ess.uci.edu/diwg/narr_uwnd.199605.nc, dataset})
the procedure looks like this
@example
@verbatim
ncap2 -O -v -S ~/narr.nco ${DATA}/hdf/narr_uwnd.199605.nc ~/foo.nc
@end verbatim
@end example
where @file{narr.nco} is an @command{ncap2} script like this:
@example
@verbatim
/* North American Regional Reanalysis (NARR) Statistics
   NARR stores grids with 2-D latitude and longitude, aka Swath-like Data (SLD) 
   Here we work with three variables:
   lat(y,x), lon(y,x), and uwnd(time,level,y,x);
   To study sub-regions of SLD, we use masking techniques:
   1. Define mask as zero times variable to be masked
      Then mask automatically inherits variable attributes
      And average below will inherit mask attributes
   2. Optionally, create mask as RAM variable (as below with asterisk *)
      NCO does not write RAM variable to output
      Masks are often unwanted, and can be big, so this speeds execution
   3. Example could be extended to preserve mean lat and lon of sub-region
      Follow uwnd example to do this: lat_msk=0.0*lat ... lat_avg=lat.avg($y,$x) */
*uwnd_msk=0.0*uwnd;
where((lat >= 35.6 && lat <= 37.0) && (lon >= -100.5 && lon <= -99.0))
  uwnd_msk=uwnd;
elsewhere
  uwnd_msk=uwnd@_FillValue;

// Average only over horizontal dimensions x and y (preserve level and time)
uwnd_avg=uwnd_msk.avg($y,$x); 
@end verbatim
@end example
Stripped of comments and formatting, this example is a three-statement
script executed by a one-line command. 
@acronym{NCO} needs only this meagre input to unpack and copy the input
data and attributes, compute the statistics, and then define and write
the output file.  
Unless the comments pointed out that wind variable (@code{uwnd}) was
four-dimensional and the latitude/longitude grid variables were both
two-dimensional, there would be no way to tell.
This shows how @acronym{NCO} hides from the user the complexity of
analyzing multi-dimensional @acronym{SLD}. 
We plan to extend such @acronym{SLD} features to more operators soon.

@html
<a name="ncap_lop"></a> <!-- http://nco.sf.net/nco.html#ncap_lop -->
@end html
@node Loops, Include files, Where statement, ncap2 netCDF Arithmetic Processor
@subsection Loops
@cindex while()
@cindex for()
@command{ncap2} supplies @command{for()} loops and @command{while()} loops. 
They are completely unoptimized so use them only with @acronym{RAM} 
variables unless you want thrash your disk to death. 
To break out of a loop use the @command{break} command. 
To iterate to the next cycle use the @command{continue} command. 

@example
@verbatim
// Set elements in variable double temp(time,lat) 
// If element < 0 set to 0, if element > 100 set to 100
*sz_idx=$time.size;
*sz_jdx=$lat.size;

for(*idx=0;idx<sz_idx;idx++)
  for(*jdx=0;jdx<sz_jdx;jdx++)
    if(temp(idx,jdx) > 100) temp(idx,jdx)=100.0; 
      else if(temp(idx,jdx) < 0) temp(idx,jdx)=0.0;

// Are values of co-ordinate variable double lat(lat) monotonic?
*sz=$lat.size;

for(*idx=1;idx<sz;idx++)
  if(lat(idx)-lat(idx-1) < 0.0) break;

if(idx == sz) print("lat co-ordinate is monotonic\n");
   else print("lat co-ordinate is NOT monotonic\n");

// Sum odd elements	
*idx=0;
*sz=$lat_nw.size;
*sum=0.0;

while(idx<sz){
  if(lat(idx)%2) sum+=lat(idx);
  idx++;
}

ram_write(sum);
print("Total of odd elements ");print(sum);print("\n"); 
@end verbatim
@end example

@html
<a name="ncap_inc"></a> <!-- http://nco.sf.net/nco.html#ncap_inc -->
@end html
@node Include files, Sort methods, Loops, ncap2 netCDF Arithmetic Processor
@subsection Include files
@cindex @command{include}
The syntax of an @var{include-file} is:
@example
#include "script.nco"
#include "/opt/SOURCES/nco/data/tst.nco"
@end example
If the filename is relative and not absolute then the directory searched is relative to the run-time directory. 
It is possible to nest include files to an arbitrary depth. 
A handy use of inlcude files is to store often used constants. 
Use @acronym{RAM} variables if you do not want these constants written to nc-file.

@var{output-file}.  
@example
// script.nco
// Sample file to #include in ncap2 script
*pi=3.1415926535; // RAM variable, not written to output
*h=6.62607095e-34; // RAM variable, not written to output
e=2.71828; // Regular (disk) variable, written to output
@end example

As of @acronym{NCO} version 4.6.3 (December, 2016), The user can specify the directory(s) to be searched by specifing them in the UNIX environment var @code{NCO_PATH}. The format used is identical to the UNIX @code{PATH}. The directory(s) are only searched if the include filename is relative.   

@example
export NCO_PATH=":/home/henryb/bin/:/usr/local/scripts:/opt/SOURCES/nco/data:"
@end example

@html
<a name="srt"></a> <!-- http://nco.sf.net/nco.html#srt -->
<a name="sort"></a> <!-- http://nco.sf.net/nco.html#sort -->
<a name="remap"></a> <!-- http://nco.sf.net/nco.html#remap -->
@end html
@node Sort methods, UDUnits script, Include files, ncap2 netCDF Arithmetic Processor
@subsection @command{sort} methods
@cindex @command{sort}
@cindex @command{asort}
@cindex @command{dsort}
@cindex @command{remap}
@cindex @command{unmap}
@cindex @command{invert_map}
In @acronym{ncap2} there are multiple ways to sort data. 
Beginning with @acronym{NCO} 4.1.0 (March, 2012), @acronym{ncap2} 
support six sorting functions:
@example
var_out=sort(var_in,&srt_map); // Ascending sort
var_out=asort(var_in,&srt_map); // Accending sort 
var_out=dsort(var_in,&srt_map); // Desending sort     
var_out=remap(var_in,srt_map); // Apply srt_map to var_in
var_out=unmap(var_in,srt_map); // Reverse what srt_map did to var_in
dsr_map=invert_map(srt_map); // Produce "de-sort" map that inverts srt_map
@end example
The first two functions, @command{sort()} and @command{asort()}
sort, in ascending order, all the elements of @var{var_in} (which can be
a variable or attribute) without regard to any dimensions.
The third function, @command{dsort()} does the same but sorts in
descending order. 
Remember that ascending and descending sorts are specified by
@command{asort()} and @command{dsort()}, respectively.

These three functions are overloaded to take a second, optional argument 
called the sort map @var{srt_map}, which should be supplied as a
call-by-reference variable, i.e., preceded with an ampersand.
If the sort map does not yet exist, then it will be created and 
returned as an integer type the same shape as the input variable.

The output @var{var_out} of each sort function is a sorted version of
the input, @var{var_in}.
The output @var{var_out} of the two mapping functions the result of
applying (with @command{remap()} or un-applying (with @command{unmap()}) 
the sort map @var{srt_map} to the input @var{var_in}.
To apply the sort map with @command{remap()} the size of the variable
must be exactly divisible by the size of the sort map. 

The final function @command{invert_map()} returns the so-called
de-sorting map @var{dsr_map} which is the inverse of the input map
@var{srt_map}. 
This gives the user access to both the forward and inverse sorting maps: 
@example
@verbatim
a1[$time]={10,2,3,4,6,5,7,3,4,1};
a1_sort=sort(a1);
print(a1_sort);
// 1, 2, 3, 3, 4, 4, 5, 6, 7, 10;

a2[$lon]={2,1,4,3};
a2_sort=sort(a2,&a2_map);
print(a2);
// 1, 2, 3, 4
print(a2_map);
// 1, 0, 3, 2;
@end verbatim
@end example  

If the map variable does not exist prior to the @command{sort()} call,
then it will be created with the same shape as the input variable and be
of type @code{NC_INT}. 
If the map variable already exists, then the only restriction is that it
be of at least the same size as the input variable. 
To apply a map use @code{remap(var_in,srt_map)}. 
@example
@verbatim
defdim("nlat",5);

a3[$lon]={2,5,3,7};
a4[$nlat,$lon]={
 1, 2, 3, 4, 
 5, 6, 7, 8,
 9,10,11,12,
 13,14,15,16,
 17,18,19,20};

a3_sort=sort(a3,&a3_map);
print(a3_map);
// 0, 2, 1, 3;

a4_sort=remap(a4,a3_map);
print(a4_sort);
// 1, 3, 2, 4,
// 5, 7, 6, 8,
// 9,11,10,12,
// 13,15,14,16,
// 17,19,18,20;

a3_map2[$nlat]={4,3,0,2,1};

a4_sort2=remap(a4,a3_map2);
print(a4_sort2);
// 3, 5, 4, 2, 1
// 8, 10, 9,7, 6, 
// 13,15,14,12,11, 
// 18,20,19,17,16
@end verbatim
@end example
As in the above example you may create your own sort map.
To sort in descending order, apply the @code{reverse()} method after the
@command{sort()}.    

Here is an extended example of how to use @command{ncap2} features to
hyperslab an irregular region based on the values of a variable not a
coordinate. 
The distinction is crucial: hyperslabbing based on dimensional indices
or coordinate values is straightforward.
Using the values of single or multi-dimensional variable to define a
hyperslab is quite different.
@example
cat > ~/ncap2_foo.nco << 'EOF'
// Purpose: Save irregular 1-D regions based on variable values

// Included in NCO User Guide at http://nco.sf.net/nco.html#sort

/* NB: Single quotes around EOF above turn off shell parameter 
    expansion in "here documents". This in turn prevents the
    need for protecting dollarsign characters in NCO scripts with
    backslashes when the script is cut-and-pasted (aka "moused") 
    from an editor or e-mail into a shell console window */

/* Copy coordinates and variable(s) of interest into RAM variable(s)
   Benefits:
   1. ncap2 defines writes all variables on LHS of expression to disk
      Only exception is RAM variables, which are stored in RAM only
      Repeated operations on regular variables takes more time, 
      because changes are written to disk copy after every change.
      RAM variables are only changed in RAM so script works faster
      RAM variables can be written to disk at end with ram_write()
   2. Script permutes variables of interest during processing
      Safer to work with copies that have different names
      This discourages accidental, mistaken use of permuted versions
   3. Makes this script a more generic template:
      var_in instead of specific variable names everywhere */
*var_in=one_dmn_rec_var;
*crd_in=time;
*dmn_in_sz=$time.size; // [nbr] Size of input arrays

/* Create all other "intermediate" variables as RAM variables 
   to prevent them from cluttering the output file.
   Mask flag and sort map are same size as variable of interest */
*msk_flg=var_in;
*srt_map=var_in;

/* In this example we mask for all values evenly divisible by 3
   This is the key, problem-specific portion of the template
   Replace this where() condition by that for your problem
   Mask variable is Boolean: 1=Meets condition, 0=Fails condition */
where(var_in % 3 == 0) msk_flg=1; elsewhere msk_flg=0;

// print("msk_flg = ");print(msk_flg); // For debugging...

/* The sort() routine is overloaded, and takes one or two arguments
   The second argument (optional) is the "sort map" (srt_map below)
   Pass the sort map by reference, i.e., prefix with an ampersand
   If the sort map does not yet exist, then it will be created and 
   returned as an integer type the same shape as the input variable.
   The output of sort(), on the LHS, is a sorted version of the input
   msk_flg is not needed in its original order after sort()
   Hence we use msk_flg as both input to and output from sort()
   Doing this prevents the need to define a new, unneeded variable */
msk_flg=sort(msk_flg,&srt_map);

// Count number of valid points in mask by summing the one's
*msk_nbr=msk_flg.total();

// Define output dimension equal in size to number of valid points
defdim("crd_out",msk_nbr);

/* Now sort the variable of interest using the sort map and remap()
   The output, on the LHS, is the input re-arranged so that all points
   meeting the mask condition are contiguous at the end of the array
   Use same srt_map to hyperslab multiple variables of the same shape
   Remember to apply srt_map to the coordinate variables */
crd_in=remap(crd_in,srt_map);
var_in=remap(var_in,srt_map);

/* Hyperslab last msk_nbr values of variable(s) of interest */
crd_out[crd_out]=crd_in((dmn_in_sz-msk_nbr):(dmn_in_sz-1));
var_out[crd_out]=var_in((dmn_in_sz-msk_nbr):(dmn_in_sz-1));

/* NB: Even though we created all variables possible as RAM variables,
   the original coordinate of interest, time, is written to the ouput.
   I'm not exactly sure why. For now, delete it from the output with: 
   ncks -O -x -v time ~/foo.nc ~/foo.nc
   */ 
EOF
ncap2 -O -v -S ~/ncap2_foo.nco ~/nco/data/in.nc ~/foo.nc
ncks -O -x -v time ~/foo.nc ~/foo.nc
ncks ~/foo.nc
@end example

Here is an extended example of how to use @command{ncap2} features to
sort multi-dimensional arrays based on the coordinate values along a
single dimension. 
@example
cat > ~/ncap2_foo.nco << 'EOF'
/* Purpose: Sort multi-dimensional array based on coordinate values
   This example sorts the variable three_dmn_rec_var(time,lat,lon)
   based on the values of the time coordinate. */

// Included in NCO User Guide at http://nco.sf.net/nco.html#sort

// Randomize the time coordinate
time=10.0*gsl_rng_uniform(time);
//print("original randomized time = \n");print(time);

/* The sort() routine is overloaded, and takes one or two arguments
   The first argument is a one dimensional array
   The second argument (optional) is the "sort map" (srt_map below)
   Pass the sort map by reference, i.e., prefix with an ampersand
   If the sort map does not yet exist, then it will be created and 
   returned as an integer type the same shape as the input variable.
   The output of sort(), on the LHS, is a sorted version of the input */

time=sort(time,&srt_map);
//print("sorted time (ascending order) and associated sort map =\n");print(time);print(srt_map);

/* sort() always sorts in ascending order
   The associated sort map therefore re-arranges the original,
   randomized time array into ascending order.
   There are two methods to obtain the descending order the user wants
   1) We could solve the problem in ascending order (the default)
   and then apply the reverse() method to re-arrange the results.
   2) We could change the sort map to return things in descending
   order of time and solve the problem directly in descending order. */

// Following shows how to do method one:

/* Expand the sort map to srt_map_3d, the size of the data array
   1. Use data array to provide right shape for the expanded sort map
   2. Coerce data array into an integer so srt_map_3d is an integer
   3. Multiply data array by zero so 3-d map elements are all zero
   4. Add the 1-d sort map to the 3-d sort map (NCO automatically resizes)
   5. Add the spatial (lat,lon) offsets to each time index 
   6. de-sort using the srt_map_3d
   7. Use reverse to obtain descending in time order
   Loops could accomplish the same thing (exercise left for reader)
   However, loops are slow for large datasets */

/* Following index manipulation requires understanding correspondence
   between 1-d (unrolled, memory order of storage) and access into that
   memory as a multidimensional (3-d, in this case) rectangular array.
   Key idea to understand is how dimensionality affects offsets */ 
// Copy 1-d sort map into 3-d sort map
srt_map_3d=(0*int(three_dmn_rec_var))+srt_map;
// Multiply base offset by factorial of lesser dimensions
srt_map_3d*=$lat.size*$lon.size;
lon_idx=array(0,1,$lon);
lat_idx=array(0,1,$lat)*$lon.size;
lat_lon_idx[$lat,$lon]=lat_idx+lon_idx;
srt_map_3d+=lat_lon_idx;

print("sort map 3d =\n");print(srt_map_3d);

// Use remap() to re-map the data
three_dmn_rec_var=remap(three_dmn_rec_var,srt_map_3d);

// Finally, reverse data so time coordinate is descending
time=time.reverse($time);
//print("sorted time (descending order) =\n");print(time);
three_dmn_rec_var=three_dmn_rec_var.reverse($time);

// Method two: Key difference is srt_map=$time.size-srt_map-1;
EOF
ncap2 -O -v -S ~/ncap2_foo.nco ~/nco/data/in.nc ~/foo.nc
@end example

@html
<a name="udunits_fnc"></a> <!-- http://nco.sf.net/nco.html#udunits_fnc -->
<a name="units_cnv"></a> <!-- http://nco.sf.net/nco.html#units_cnv -->
@end html
@node UDUnits script, Vpointer, Sort methods, ncap2 netCDF Arithmetic Processor
@subsection UDUnits script
@cindex UDUnits

As of @acronym{NCO} version 4.6.3 (December, 2016), @acronym{ncap2}
includes support for UDUnits conversions. 
The function is called @code{udunits}. 
Its syntax is
@example
varOut=udunits(varIn,"UnitsOutString")
@end example

The @code{udunits()} function looks for the attribute of
@code{varIn@@units} and fails if it is not found. 
A quirk of this function that due to attribute propagation
@code{varOut@@units} will be overwritten by @code{varIn@@units}. 
It is best to re-initialize this attribute AFTER the call. 
In addition if @code{varIn@@units} is of the form 
@code{"time_interval since basetime"} then the calendar attribute
@code{varIn@@calendar} will read it.
If it does not exist then the calendar used defaults to mixed
Gregorian/Julian as defined by UDUnits. 
 
If @code{varIn} is not a floating-point type then it is promoted to
@code{NC_DOUBLE} for the system call in the UDUnits library,   
and then demoted back to its original type after.
@example
@verbatim
T[lon]={0.0,100.0,150.0,200.0};
T@units="Celsius";
// Overwrite variable
T=udunits(T,"kelvin"); 
print(T);  
// 273.15, 373.15, 423.15, 473.15 ;
T@units="kelvin";

// Rebase coordinate days to hours 
timeOld=time;
print(timeOld);
// 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ;
timeOld@units="days since 2012-01-30";

@units="hours since 2012-02-01 01:00";
timeNew=udunits(timeOld, @units);
timeNew@units=@units;
print(timeNew);
// -25, -1, 23, 47, 71, 95, 119, 143, 167, 191 ;

tOld=time;
// nb in this calendar NO Leap year
tOld@calendar="365_day";
tOld@units="minutes since 2012-02-28 23:58:00.00";

@units="seconds since 2012-03-01 00:00";
tNew=udunits(tOld, @units);
tNew@units=@units;
print(tNew);
// -60, 0, 60, 120, 180, 240, 300, 360, 420, 480 
@end verbatim
@end example

@noindent strftime()
The @code{var_str=strtime(var_time,fmt_sng)} method takes a time-based variable and a format string and returns an @code{NC_STRING} variable (of the same shape as var_time) of time-stamps in the form specified by 'fmt_sng'. In order to run this command output type must be netCDF4.
@example 
@verbatim
ncap2 -4  -v -O -s 'time_str=strftime(time,"%Y-%m-%d");' in.nc foo.nc

time_str="1964-03-13", "1964-03-14", "1964-03-15", "1964-03-16", 
         "1964-03-17", "1964-03-18", "1964-03-19", "1964-03-20", 
         "1964-03-21", "1964-03-22" ;
@end verbatim
@end example

Under the hood there are  a few steps invoved:
First the method reads @code{var_time@@units} and
@code{var_time@@calendar} (if present) then converts @code{var_time} to
@code{seconds since 1970-01-01}.
It then converts these possibly UTC seconds to the standard struture
@code{struct *tm}.
Finally @code{strftime()} is called with @code{fmt_sng} and the
@code{*tm} struct.
The C-standard @code{strftime()} is used as defined in @file{time.h}.
If the method is called without @var{fmt_sng} then the following default is
used: @code{"%Y-%m-%d %H:%M:%S"}.
The method @code{regular} takes a single var argument and uses the
above default string. 
@example 
@verbatim
ncap2 -4  -v -O -s 'time_str=regular(time);' in.nc foo.nc

time_str = "1964-03-13 21:09:00", "1964-03-14 21:09:00", "1964-03-15 21:09:00", 
           "1964-03-16 21:09:00", "1964-03-17 21:09:00", "1964-03-18 21:09:00", 
           "1964-03-19 21:09:00", "1964-03-20 21:09:00", "1964-03-21 21:09:00", 
           "1964-03-22 21:09:00" ;
@end verbatim
@end example

@noindent Another working example
@example
@verbatim
ncap2 -v -O -s 'ts=strftime(frametime(0),"%Y-%m-%d/envlog_netcdf_L1_ua-mac_%Y-%m-%d.nc");' in.nc out.nc
ts="2017-08-11/envlog_netcdf_L1_ua-mac_2017-08-11.nc" 
@end verbatim
@end example

@html
<a name="vpointer"></a> <!-- http://nco.sf.net/nco.html#vpointer -->
<a name="vp"></a> <!-- http://nco.sf.net/nco.html#vp -->
@end html
@node Vpointer, Irregular grids, UDUnits script, ncap2 netCDF Arithmetic Processor
@subsection Vpointer
@cindex vpointer

@noindent A variable-pointer or @emph{vpointer} is a pointer to a
variable or attribute.
It is most useful when one needs to apply a set of operations on a list
of variables.
For example, after regular processing one may wish to set the
@code{_FillValue} of all @code{NC_FLOAT} variables to a particular
value, or to create min/max attributes for all 3D variables of type
@code{NC_DOUBLE}.
A vpointer is not a 'pointer' to a memory location in the C/C++
sense.
Rather the vpointer is a text attribute that contains the name of a
variable.
To use the pointer simply prefix the pointer with @code{*}.
Then, most places where you use @code{VAR_ID} you can use
*vpointer_nm.
There are a variety of ways to maintain a list of strings in
@command{ncap2}.
The easiest method is to use an @code{NC_STRING} attribute.  

Below is a simple illustration that uses a vpointer of type
@code{NC_CHAR}. 
Remember an attribute starting with @code{@@} implies 'global', e.g.,  
@code{@@vpx} is short for @code{global@@vpx}. 
@example
@verbatim
idx=9;
idy=20;
t2=time;

global@vpx="idx";

// Increment idx by one
*global@vpx++;  
print(idx);

// Multiply by 5
*@vpx*=5; // idx now 50
print(idx);

// Add 200 (long method)
*@vpx=*@vpx+200; //idx now 250
print(idx);

@vpy="idy";

// Add idx idy to get idz
idz=*@vpx+*@vpy; // idz == 270
print(idz);

// We can also reference variables in the input file
// Can use an existing attribute pointer since attributes are not written
// to the netCDF file until after the script has finished.
@vpx="three_dmn_var";

// We can convert this variable to type NC_DOUBLE and
// write it to ouptut all at once
*@vpx=*@vpx.double();
@end verbatim
@end example

The following script writes to the output files all variables that are
of type @code{NC_DOUBLE} and that have at least two dimensions.
It then changes their @code{_FillValue} to @code{1.0E-9}.  
The function @code{get_vars_in()} creates an @code{NC_STRING} attribute
that contains all of the variable names in the input file.
Note that a vpointer must be a plain attribute, NOT an a attribute
expression. 
Thus in the below script using @code{*all(idx)} would be a fundamental
mistake.
In the below example the vpointer @code{var_nm} is of type
@code{NC_STRING}. 
@example
@verbatim
@all=get_vars_in();

*sz=@all.size();
*idx=0;

for(idx=0;idx<sz;idx++){
  // @var_nm is of type NC_STRING
  @var_nm=@all(idx);
 
  if(*@var_nm.type() == NC_DOUBLE && *@var_nm.ndims() >= 2){
     *@var_nm=*@var_nm; 
     *@var_nm.change_miss(1e-9d);
  }
}
@end verbatim
@end example 

The following script writes to the output file all 3D/4D
variables of type @code{NC_FLOAT}.
Then for each variable it calculates a @code{range} attribute that 
contains the maximum and minimum values, and a @code{total} attribute
that is the sum of all the elements.
In this example vpointers are used to 'point' to attributes.
@example
@verbatim
@all=get_vars_in();
*sz=@all.size();

for(*idx=0;idx<sz;idx++){
  @var_nm=@all(idx);
  if(*@var_nm.ndims() >= 3){
    *@var_nm=*@var_nm.float();
    // The push function also takes a call-by-ref attribute: if it does not already exist then it will be created
    // The call below pushes a NC_STRING to an att so the end result is a list of NC_STRINGS   
    push(&@prc,@var_nm); 
  }
} 

*sz=@prc.size();

for(*idx=0;idx<sz;idx++){
  @var_nm=@prc(idx);

  // We can work with attribute pointers as well 
  // sprint - ouptut is of type NC_CHAR
  @att_total=sprint(@var_nm,"%s@total"); 
  @att_range=sprint(@var_nm,"%s@range"); 

  // If you are still confused then print out the atts 
  print(@att_total); 
  print(@att_range);
 
  *@att_total= *@var_nm.total();
  *@att_range={ min(*@var_nm), max(*@var_nm)}; 
} 
@end verbatim
@end example

@noindent This is an ncdump of one of the variables that has been processed by the above script
@example
@verbatim
float three_dmn_var_int(time, lat, lon) ;
three_dmn_var_int:_FillValue = -99.f ;
three_dmn_var_int:long_name = "three dimensional record variable of type int" ;
three_dmn_var_int:range = 1.f, 80.f ;
three_dmn_var_int:total = 2701.f ;
three_dmn_var_int:units = "watt meter-2" ;
@end verbatim
@end example

@html
<a name="rct"></a> <!-- http://nco.sf.net/nco.html#rct -->
@end html

@node Irregular grids, Bilinear interpolation, Vpointer, ncap2 netCDF Arithmetic Processor
@subsection Irregular Grids
@cindex irregular grids
@cindex rectangular grids
@cindex non-rectangular grids
@cindex non-standard grids
@cindex mask
@c fxm need to edit rrg sxn beginning here
@acronym{NCO} is capable of analyzing datasets for many different
underlying coordinate grid types.
netCDF was developed for and initially used with grids comprised of
orthogonal dimensions forming a rectangular coordinate system.
We call such grids @emph{standard} grids.
It is increasingly common for datasets to use metadata to describe
much more complex grids.
Let us first define three important coordinate grid properties:
regularity, rectangularity, and structure.

Grids are @emph{regular} if the spacing between adjacent is constant. 
For example, a 4-by-5 degree latitude-longitude grid is regular
because the spacings between adjacent latitudes (@w{4 degrees}) are
constant as are the (@w{5 degrees}) spacings between adjacent
longitudes. 
Spacing in @emph{irregular} grids depends on the location along the
coordinate. 
Grids such as Gaussian grids have uneven spacing in latitude (points 
cluster near the equator) and so are irregular.

Grids are @emph{rectangular} if the number of elements in any
dimension is not a function of any other dimension.
For example, a T42 Gaussian latitude-longitude grid is rectangular
because there are the same number of longitudes (128) for each of the 
(64) latitudes.
Grids are @emph{non-rectangular} if the elements in any dimension
depend on another dimension.
Non-rectangular grids present many special challenges to 
analysis software like @acronym{NCO}.

Grids are @emph{structured} if they are represented as functions
of two horizontal spatial dimensions.
For example, grids with latitude and longitude dimensions are
structured, and so are curvilinear grids with along-track and
cross-track dimensions.
A grid with a single dimension is @emph{unstructured}.
For example, icosohedral grids are usually unstructured, as are
@acronym{MPAS} grids.

Wrapped coordinates (@pxref{Wrapped Coordinates}), such as longitude,
are independent of these grid properties (regularity, rectangularity,
structure). 

@cindex wrapped coordinates
The preferred @acronym{NCO} technique to analyze data on non-standard
coordinate grids is to create a region mask with @command{ncap2}, and
then to use the mask within @command{ncap2} for variable-specific
processing, and/or with other operators (e.g., @command{ncwa},
@command{ncdiff}) for entire file processing. 

Before describing the construction of masks, let us review how
irregularly gridded geoscience data are described.
Say that latitude and longitude are stored as @var{R}-dimensional
arrays and the product of the dimension sizes is the total number of  
elements N in the other variables.
Geoscience applications tend to use @math{@var{R}=1}, 
@math{@var{R}=2}, and @math{@var{R}=3}.

If the grid is has no simple representation (e.g., discontinuous) then
it makes sense to store all coordinates as 1D arrays with the same
size as the number of grid points. 
These gridpoints can be completely independent of all the other (own
weight, area, etc.).  

@var{R}=1: lat(number_of_gridpoints) and lon(number_of_gridpoints)

If the horizontal grid is time-invariant then @var{R}=2 is common:

@var{R}=2: lat(south_north,east_west) and lon(south_north,east_west)

The Weather and Research Forecast (@acronym{WRF}) model uses @var{R}=3:

@var{R}=3: lat(time,south_north,east_west), lon(time,south_north,east_west)

and so supports grids that change with time.

Grids with @var{R} > 1 often use missing values to indicated empty points.
For example, so-called ``staggered grids'' will use fewer east_west
points near the poles and more near the equator. netCDF only accepts
rectangular arrays so space must be allocated for the maximum number
of east_west points at all latitudes. Then the application writes
missing values into the unused points near the poles.

We demonstrate the @command{ncap2} analysis technique for irregular
regions by constructing a mask for an @var{R}=2 grid.
We wish to find, say, the mean temperature within 
[@var{lat_min},@var{lat_max}] and [@var{lon_min},@var{lon_max}]: 
@example
ncap2 -s 'mask_var= (lat >= lat_min && lat <= lat_max) && \
                    (lon >= lon_min && lon <= lon_max);' in.nc out.nc
@end example
Arbitrarily shaped regions can be defined by more complex conditional
statements. 
Once defined, masks can be applied to specific variables,
and to entire files:
@example
ncap2 -s 'temperature_avg=(temperature*mask_var).avg()' in.nc out.nc
ncwa -a lat,lon -m mask_var -w area in.nc out.nc
@end example
Crafting such commands on the command line is possible though unwieldy.
In such cases, a script is often cleaner and allows you to document the
procedure:
@example
@verbatim
cat > ncap2.in << 'EOF'
mask_var = (lat >= lat_min && lat <= lat_max) && (lon >= lon_min && > lon <= lon_max);
if(mask_var.total() > 0){ // Check that mask contains some valid values
  temperature_avg=(temperature*mask_var).avg(); // Average temperature
  temperature_max=(temperature*mask_var).max(); // Maximum temperature
}
EOF
ncap2 -S ncap2.in in.nc out.nc
@end verbatim
@end example

@ignore
http://foehn.colorado.edu/wrfout_to_cf/wrfout_to_cf.ncl
ncl 'file_in="wrfout.nc"' 'file_out="wrfpost.nc"' wrfout_to_cf.ncl
ncl 'file_in="wrfout_d02_2013-10-04_20:00:00"' 'file_out="wrfout_d02_2013-10-04_20:00:00_cf.nc"' wrfout_to_cf.ncl
ncl 'file_in="wrfout_v2_Lambert"' 'file_out="wrfout_v2_Lambert.nc"' wrfout_to_cf.ncl
@end ignore
@cindex @acronym{WRF}
Grids like those produced by the @acronym{WRF} model are complex because
one must use global metadata to determine the grid staggering and
offsets to translate @code{XLAT} and @code{XLONG} into real latitudes, 
longitudes, and missing points. 
The @acronym{WRF} grid documentation should describe this.
For @acronym{WRF} files creating regional masks looks, in general, like 
@example
mask_var = (XLAT >= lat_min && XLAT <= lat_max) && (XLONG >= lon_min && XLONG <= lon_max);
@end example

A few notes:
Irregular regions are the union of arrays of lat/lon min/max's. 
The mask procedure is identical for all @var{R}.
@c fxm need to edit rrg sxn down to here

@html
<a name="bln_ntp"></a> <!-- http://nco.sf.net/nco.html#bln_ntp -->
<a name="bil_int"></a> <!-- http://nco.sf.net/nco.html#bil_int -->
@end html
@node Bilinear interpolation, GSL special functions, Irregular grids, ncap2 netCDF Arithmetic Processor
@subsection Bilinear interpolation
@noindent As of version 4.0.0 @acronym{NCO} has internal routines to
perform bilinear interpolation on gridded data sets.
In mathematics, bilinear interpolation is an extension of linear
interpolation for interpolating functions of two variables on a regular
grid. 
The idea is to perform linear interpolation first in one direction, and
then again in the other direction.

Suppose we have an irregular grid of data @code{temperature[lat,lon]},
with co-ordinate vars @code{lat[lat], lon[lon]}. 
We wish to find the temperature at an arbitary point [@var{X},@var{Y}]
within the grid. 
If we can locate lat_min,lat_max and lon_min,lon_max such that 
@code{lat_min <= X <= lat_max} and @code{lon_min <= Y <= lon_max} 
then we can interpolate in two dimensions the temperature at
[@var{X},@var{Y}]. 

The general form of the @command{ncap2} interpolation function is
@example
var_out=bilinear_interp(grid_in,grid_out,grid_out_x,grid_out_y,grid_in_x,grid_in_y)
@end example
where
@table @code
@item grid_in
Input function data. 
Usually a two dimensional variable. 
It must be of size @code{grid_in_x.size()*grid_in_y.size()}        
@item grid_out
This variable is the shape of @code{var_out}. 
Usually a two dimensional variable. 
It must be of size @code{grid_out_x.size()*grid_out_y.size()}
@item grid_out_x
@var{X} output values 
@item grid_out_y
@var{Y} output values 
@item grid_in_x
@var{X} input values values. Must be monotonic (increasing or decreasing).
@item grid_in_y
@var{Y} input values values. Must be monotonic (increasing or decreasing).
@end table
@noindent
Prior to calculations all arguments are converted to type
@code{NC_DOUBLE}.
After calculations @code{var_out} is converted to the input type of
@code{grid_in}. 

Suppose the first part of an @command{ncap2} script is
@example
@verbatim
defdim("X",4);
defdim("Y",5);

// Temperature
T_in[$X,$Y]=
 {100, 200, 300, 400, 500,
  101, 202, 303, 404, 505,
  102, 204, 306, 408, 510,
  103, 206, 309, 412, 515.0 };

// Coordinate variables
x_in[$X]={0.0,1.0,2.0,3.01};
y_in[$Y]={1.0,2.0,3.0,4.0,5};
@end verbatim
@end example
Now we interpolate with the following variables:
@example
@verbatim
defdim("Xn",3);
defdim("Yn",4); 
T_out[$Xn,$Yn]=0.0;
x_out[$Xn]={0.0,0.02,3.01};
y_out[$Yn]={1.1,2.0,3,4};

var_out=bilinear_interp(T_in,T_out,x_out,y_out,x_in,y_in);
print(var_out);
// 110, 200, 300, 400,
// 110.022, 200.04, 300.06, 400.08,
// 113.3, 206, 309, 412 ;
@end verbatim
@end example 

It is possible to interpolate a single point:
@example
var_out=bilinear_interp(T_in,0.0,3.0,4.99,x_in,y_in);
print(var_out);
// 513.920594059406
@end example

@noindent @strong{Wrapping and Extrapolation} @*
@noindent The function @code{bilinear_interp_wrap()} takes the same
arguments as @code{bilinear_interp()} but performs wrapping (@var{Y})
and extrapolation (@var{X}) for points off the edge of the grid.
If the given range of longitude is say (25-335) and we have a point at
20 degrees, then the endpoints of the range are used for the
interpolation. 
This is what wrapping means.   
For wrapping to occur @var{Y} must be longitude and must be in the range
(0,360) or (-180,180). 
There are no restrictions on the longitude (@var{X}) values, though
typically these are in the range (-90,90).
This @command{ncap2} script illustrates both wrapping and extrapolation
of end points:
@example
@verbatim
defdim("lat_in",6);
defdim("lon_in",5);

// Coordinate input vars
lat_in[$lat_in]={-80,-40,0,30,60.0,85.0};
lon_in[$lon_in]={30, 110, 190, 270, 350.0};

T_in[$lat_in,$lon_in]=
  {10,40,50,30,15,   
    12,43,52,31,16,   
    14,46,54,32,17,   
    16,49,56,33,18,   
    18,52,58,34,19,   
    20,55,60,35,20.0 };
   
defdim("lat_out",4);
defdim("lon_out",3);

// Coordinate variables
lat_out[$lat_out]={-90,0,70,88.0};   
lon_out[$lon_out]={0,190,355.0};

T_out[$lat_out,$lon_out]=0.0;

T_out=bilinear_interp_wrap(T_in,T_out,lat_out,lon_out,lat_in,lon_in);
print(T_out); 
// 13.4375, 49.5, 14.09375,
// 16.25, 54, 16.625,
// 19.25, 58.8, 19.325,
// 20.15, 60.24, 20.135 ;
@end verbatim
@end example

@html
<a name="gsl"></a> <!-- http://nco.sf.net/nco.html#gsl -->
@end html
@node GSL special functions, GSL interpolation, Bilinear interpolation, ncap2 netCDF Arithmetic Processor
@subsection GSL special functions
@cindex @acronym{GSL}
@noindent As of version 3.9.6 (released January, 2009), @acronym{NCO} 
can link to the @acronym{GNU} Scientific Library (@acronym{GSL}). 
@command{ncap2} can access most @acronym{GSL} special functions including
Airy, Bessel, error, gamma, beta, hypergeometric, and Legendre functions
and elliptical integrals. 
@acronym{GSL} must be @w{version 1.4} or later. 
To list the @acronym{GSL} functions available with your @acronym{NCO} 
build, use @command{ncap2 -f | grep ^gsl}.

@noindent The function names used by @acronym{ncap2} mirror their
@acronym{GSL} names.
The @acronym{NCO} wrappers for @acronym{GSL} functions automatically
call the error-handling version of the @acronym{GSL} function when
available  
@footnote{   
These are the @acronym{GSL} standard function names postfixed with
@code{_e}.  
@acronym{NCO} calls these functions automatically, without the 
@acronym{NCO} command having to specifically indicate the @code{_e}
function suffix.
}.
This allows @acronym{NCO} to return a missing value when the
@acronym{GSL} library encounters a domain error or a floating-point 
exception. 
The slow-down due to calling the error-handling version of the 
@acronym{GSL} numerical functions was found to be negligible (please let
us know if you find otherwise).

@cindex gamma function
@cindex @var{gsl_sf_gamma}
@noindent Consider the gamma function.@*
@noindent The @acronym{GSL} function prototype is @*
@code{int gsl_sf_gamma_e(const double x, gsl_sf_result * result)}
The @command{ncap2} script would be:
@example
@verbatim
lon_in[lon]={-1,0.1,0,2,0.3};
lon_out=gsl_sf_gamma(lon_in);
lon_out= _, 9.5135, 4.5908, 2.9915 
@end verbatim
@end example

@noindent The first value is set to @code{_FillValue} since the gamma
function is undefined for negative integers.
If the input variable has a missing value then this value is used.
Otherwise, the default double fill value is used
(defined in the netCDF header @file{netcdf.h} as 
@code{NC_FILL_DOUBLE = 9.969e+36}).

@cindex Bessel function
@cindex @var{gsl_sf_bessel_Jn}
@noindent Consider a call to a Bessel function with @acronym{GSL}
prototype@* 
@code{int gsl_sf_bessel_Jn_e(int n, double x, gsl_sf_result * result)} 

An @command{ncap2} script would be
@example
lon_out=gsl_sf_bessel_Jn(2,lon_in);  
lon_out=0.11490, 0.0012, 0.00498, 0.011165
@end example
This computes the Bessel function of order @var{n=2} for every value in
@code{lon_in}.
The Bessel order argument, an integer, can also be a non-scalar
variable, i.e., an array.  
@example
@verbatim
n_in[lon]={0,1,2,3};
lon_out=gsl_sf_bessel_Jn(n_in,0.5);
lon_out= 0.93846, 0.24226, 0.03060, 0.00256
@end verbatim
@end example

@noindent Arguments to @acronym{GSL} wrapper functions in @command{ncap2}
must conform to one another, i.e., they must share the same sub-set of
dimensions.  
For example: @code{three_out=gsl_sf_bessel_Jn(n_in,three_dmn_var_dbl)}
is valid because the variable @code{three_dmn_var_dbl} has a @var{lon} 
dimension, so @code{n_in} in can be broadcast to conform to
@code{three_dmn_var_dbl}.  
However @code{time_out=gsl_sf_bessel_Jn(n_in,time)} is invalid.

@cindex Elliptic integrals
Consider the elliptical integral with prototype
@code{int gsl_sf_ellint_RD_e(double x, double y, double z, gsl_mode_t mode, gsl_sf_result * result)}
@example
three_out=gsl_sf_ellint_RD(0.5,time,three_dmn_var_dbl);
@end example

@noindent The three arguments are all conformable so the above @command{ncap2} call is valid. The mode argument in the function prototype controls the convergence of the algorithm. It also appears  in the Airy Function prototypes. It can be set by defining the environment variable @code{GSL_PREC_MODE}. If unset it defaults to the value @code{GSL_PREC_DOUBLE}. See the @acronym{GSL} manual for more details. 
@example
export GSL_PREC_MODE=0 // GSL_PREC_DOUBLE
export GSL_PREC_MODE=1 // GSL_PREC_SINGLE
export GSL_PREC_MODE=2 // GSL_PREC_APPROX
@end example

@noindent The @command{ncap2} wrappers to the array functions are
slightly different. 
Consider the following @acronym{GSL} prototype @* 
@code{int gsl_sf_bessel_Jn_array(int nmin, int nmax, double x, double *result_array)}
@example
b1=lon.double();
x=0.5;
status=gsl_sf_bessel_Jn_array(1,4,x,&b1);
print(status);
b1=0.24226,0.0306,0.00256,0.00016;
@end example
@noindent This calculates the Bessel function of @var{x}=0.5 for
@var{n}=1 to 4. 
The first three arguments are scalar values. 
If a non-scalar variable is supplied as an argument then only the first
value is used. 
The final argument is the variable where the results are stored (NB: the
@code{&} indicates this is a call by reference). 
This final argument must be of type @code{double} and must be of least
size @var{nmax-nmin+1}. 
If either of these conditions is not met then then the function 
returns an error message. 
The function/wrapper returns a status flag. 
Zero indicates success. 

@noindent Consider another array function @* 
@code{int gsl_sf_legendre_Pl_array(int lmax, double x, double *result_array);}
@cindex Legendre polynomial
@findex gsl_sf_legendre_Pl
@example
a1=time.double();
x=0.3;
status=gsl_sf_legendre_Pl_array(a1.size()-1, x,&a1);  
print(status);
@end example
@noindent This call calculates @var{P_l}(0.3) for @var{l}=0..9. 
Note that @var{|x|<=1}, otherwise there will be a domain error. 
See the @acronym{GSL} 
documentation for more details.  

@noindent The @acronym{GSL} functions implemented in @acronym{NCO} are 
listed in the table below. 
This table is correct for @acronym{GSL} version 1.10. 
To see what functions are available on your build run the command
@command{ncap2 -f |grep ^gsl} . 
To see this table along with the @acronym{GSL} @w{C-function}
prototypes look at the spreadsheet @strong{doc/nco_gsl.ods}. @* @* 

@multitable @columnfractions .35 .05 .60 
@item @strong{GSL NAME} @tab @strong{I} @tab @strong{NCAP FUNCTION CALL}
@item gsl_sf_airy_Ai_e @tab Y @tab gsl_sf_airy_Ai(dbl_expr)
@item gsl_sf_airy_Bi_e @tab Y @tab gsl_sf_airy_Bi(dbl_expr)
@item gsl_sf_airy_Ai_scaled_e @tab Y @tab gsl_sf_airy_Ai_scaled(dbl_expr)
@item gsl_sf_airy_Bi_scaled_e @tab Y @tab gsl_sf_airy_Bi_scaled(dbl_expr)
@item gsl_sf_airy_Ai_deriv_e @tab Y @tab gsl_sf_airy_Ai_deriv(dbl_expr)
@item gsl_sf_airy_Bi_deriv_e @tab Y @tab gsl_sf_airy_Bi_deriv(dbl_expr)
@item gsl_sf_airy_Ai_deriv_scaled_e @tab Y @tab gsl_sf_airy_Ai_deriv_scaled(dbl_expr)
@item gsl_sf_airy_Bi_deriv_scaled_e @tab Y @tab gsl_sf_airy_Bi_deriv_scaled(dbl_expr)
@item gsl_sf_airy_zero_Ai_e @tab Y @tab gsl_sf_airy_zero_Ai(uint_expr)
@item gsl_sf_airy_zero_Bi_e @tab Y @tab gsl_sf_airy_zero_Bi(uint_expr)
@item gsl_sf_airy_zero_Ai_deriv_e @tab Y @tab gsl_sf_airy_zero_Ai_deriv(uint_expr)
@item gsl_sf_airy_zero_Bi_deriv_e @tab Y @tab gsl_sf_airy_zero_Bi_deriv(uint_expr)
@item gsl_sf_bessel_J0_e @tab Y @tab gsl_sf_bessel_J0(dbl_expr)
@item gsl_sf_bessel_J1_e @tab Y @tab gsl_sf_bessel_J1(dbl_expr)
@item gsl_sf_bessel_Jn_e @tab Y @tab gsl_sf_bessel_Jn(int_expr,dbl_expr)
@item gsl_sf_bessel_Jn_array @tab Y @tab status=gsl_sf_bessel_Jn_array(int,int,double,&var_out)
@item gsl_sf_bessel_Y0_e @tab Y @tab gsl_sf_bessel_Y0(dbl_expr)
@item gsl_sf_bessel_Y1_e @tab Y @tab gsl_sf_bessel_Y1(dbl_expr)
@item gsl_sf_bessel_Yn_e @tab Y @tab gsl_sf_bessel_Yn(int_expr,dbl_expr)
@item gsl_sf_bessel_Yn_array @tab Y @tab gsl_sf_bessel_Yn_array
@item gsl_sf_bessel_I0_e @tab Y @tab gsl_sf_bessel_I0(dbl_expr)
@item gsl_sf_bessel_I1_e @tab Y @tab gsl_sf_bessel_I1(dbl_expr)
@item gsl_sf_bessel_In_e @tab Y @tab gsl_sf_bessel_In(int_expr,dbl_expr)
@item gsl_sf_bessel_In_array @tab Y @tab status=gsl_sf_bessel_In_array(int,int,double,&var_out)
@item gsl_sf_bessel_I0_scaled_e @tab Y @tab gsl_sf_bessel_I0_scaled(dbl_expr)
@item gsl_sf_bessel_I1_scaled_e @tab Y @tab gsl_sf_bessel_I1_scaled(dbl_expr)
@item gsl_sf_bessel_In_scaled_e @tab Y @tab gsl_sf_bessel_In_scaled(int_expr,dbl_expr)
@item gsl_sf_bessel_In_scaled_array @tab Y @tab staus=gsl_sf_bessel_In_scaled_array(int,int,double,&var_out)
@item gsl_sf_bessel_K0_e @tab Y @tab gsl_sf_bessel_K0(dbl_expr)
@item gsl_sf_bessel_K1_e @tab Y @tab gsl_sf_bessel_K1(dbl_expr)
@item gsl_sf_bessel_Kn_e @tab Y @tab gsl_sf_bessel_Kn(int_expr,dbl_expr)
@item gsl_sf_bessel_Kn_array @tab Y @tab status=gsl_sf_bessel_Kn_array(int,int,double,&var_out)
@item gsl_sf_bessel_K0_scaled_e @tab Y @tab gsl_sf_bessel_K0_scaled(dbl_expr)
@item gsl_sf_bessel_K1_scaled_e  @tab Y @tab gsl_sf_bessel_K1_scaled(dbl_expr)
@item gsl_sf_bessel_Kn_scaled_e @tab Y @tab gsl_sf_bessel_Kn_scaled(int_expr,dbl_expr)
@item gsl_sf_bessel_Kn_scaled_array @tab Y @tab status=gsl_sf_bessel_Kn_scaled_array(int,int,double,&var_out)
@item gsl_sf_bessel_j0_e @tab Y @tab gsl_sf_bessel_J0(dbl_expr)
@item gsl_sf_bessel_j1_e @tab Y @tab gsl_sf_bessel_J1(dbl_expr)
@item gsl_sf_bessel_j2_e @tab Y @tab gsl_sf_bessel_j2(dbl_expr)
@item gsl_sf_bessel_jl_e @tab Y @tab gsl_sf_bessel_jl(int_expr,dbl_expr)
@item gsl_sf_bessel_jl_array @tab Y @tab status=gsl_sf_bessel_jl_array(int,double,&var_out)
@item gsl_sf_bessel_jl_steed_array @tab Y @tab gsl_sf_bessel_jl_steed_array
@item gsl_sf_bessel_y0_e @tab Y @tab gsl_sf_bessel_Y0(dbl_expr)
@item gsl_sf_bessel_y1_e @tab Y @tab gsl_sf_bessel_Y1(dbl_expr)
@item gsl_sf_bessel_y2_e @tab Y @tab gsl_sf_bessel_y2(dbl_expr)
@item gsl_sf_bessel_yl_e @tab Y @tab gsl_sf_bessel_yl(int_expr,dbl_expr)
@item gsl_sf_bessel_yl_array @tab Y @tab status=gsl_sf_bessel_yl_array(int,double,&var_out)
@item gsl_sf_bessel_i0_scaled_e @tab Y @tab gsl_sf_bessel_I0_scaled(dbl_expr)
@item gsl_sf_bessel_i1_scaled_e @tab Y @tab gsl_sf_bessel_I1_scaled(dbl_expr)
@item gsl_sf_bessel_i2_scaled_e @tab Y @tab gsl_sf_bessel_i2_scaled(dbl_expr)
@item gsl_sf_bessel_il_scaled_e @tab Y @tab gsl_sf_bessel_il_scaled(int_expr,dbl_expr)
@item gsl_sf_bessel_il_scaled_array @tab Y @tab status=gsl_sf_bessel_il_scaled_array(int,double,&var_out)
@item gsl_sf_bessel_k0_scaled_e @tab Y @tab gsl_sf_bessel_K0_scaled(dbl_expr)
@item gsl_sf_bessel_k1_scaled_e @tab Y @tab gsl_sf_bessel_K1_scaled(dbl_expr)
@item gsl_sf_bessel_k2_scaled_e @tab Y @tab gsl_sf_bessel_k2_scaled(dbl_expr)
@item gsl_sf_bessel_kl_scaled_e @tab Y @tab gsl_sf_bessel_kl_scaled(int_expr,dbl_expr)
@item gsl_sf_bessel_kl_scaled_array @tab Y @tab status=gsl_sf_bessel_kl_scaled_array(int,double,&var_out)
@item gsl_sf_bessel_Jnu_e @tab Y @tab gsl_sf_bessel_Jnu(dbl_expr,dbl_expr)
@item gsl_sf_bessel_Ynu_e @tab Y @tab gsl_sf_bessel_Ynu(dbl_expr,dbl_expr)
@item gsl_sf_bessel_sequence_Jnu_e @tab N @tab gsl_sf_bessel_sequence_Jnu
@item gsl_sf_bessel_Inu_scaled_e @tab Y @tab gsl_sf_bessel_Inu_scaled(dbl_expr,dbl_expr)
@item gsl_sf_bessel_Inu_e @tab Y @tab gsl_sf_bessel_Inu(dbl_expr,dbl_expr)
@item gsl_sf_bessel_Knu_scaled_e @tab Y @tab gsl_sf_bessel_Knu_scaled(dbl_expr,dbl_expr)
@item gsl_sf_bessel_Knu_e @tab Y @tab gsl_sf_bessel_Knu(dbl_expr,dbl_expr)
@item gsl_sf_bessel_lnKnu_e @tab Y @tab gsl_sf_bessel_lnKnu(dbl_expr,dbl_expr)
@item gsl_sf_bessel_zero_J0_e @tab Y @tab gsl_sf_bessel_zero_J0(uint_expr)
@item gsl_sf_bessel_zero_J1_e @tab Y @tab gsl_sf_bessel_zero_J1(uint_expr)
@item gsl_sf_bessel_zero_Jnu_e @tab N @tab gsl_sf_bessel_zero_Jnu
@item gsl_sf_clausen_e @tab Y @tab gsl_sf_clausen(dbl_expr)
@item gsl_sf_hydrogenicR_1_e @tab N @tab gsl_sf_hydrogenicR_1
@item gsl_sf_hydrogenicR_e @tab N @tab gsl_sf_hydrogenicR
@item gsl_sf_coulomb_wave_FG_e @tab N @tab gsl_sf_coulomb_wave_FG
@item gsl_sf_coulomb_wave_F_array @tab N @tab gsl_sf_coulomb_wave_F_array
@item gsl_sf_coulomb_wave_FG_array @tab N @tab gsl_sf_coulomb_wave_FG_array
@item gsl_sf_coulomb_wave_FGp_array @tab N @tab gsl_sf_coulomb_wave_FGp_array
@item gsl_sf_coulomb_wave_sphF_array  @tab N @tab gsl_sf_coulomb_wave_sphF_array 
@item gsl_sf_coulomb_CL_e @tab N @tab gsl_sf_coulomb_CL
@item gsl_sf_coulomb_CL_array @tab N @tab gsl_sf_coulomb_CL_array
@item gsl_sf_coupling_3j_e @tab N @tab gsl_sf_coupling_3j
@item gsl_sf_coupling_6j_e @tab N @tab gsl_sf_coupling_6j
@item gsl_sf_coupling_RacahW_e @tab N @tab gsl_sf_coupling_RacahW
@item gsl_sf_coupling_9j_e @tab N @tab gsl_sf_coupling_9j
@item gsl_sf_coupling_6j_INCORRECT_e @tab N  @tab gsl_sf_coupling_6j_INCORRECT
@item gsl_sf_dawson_e @tab Y @tab gsl_sf_dawson(dbl_expr)
@item gsl_sf_debye_1_e @tab Y @tab gsl_sf_debye_1(dbl_expr)
@item gsl_sf_debye_2_e @tab Y @tab gsl_sf_debye_2(dbl_expr)
@item gsl_sf_debye_3_e @tab Y @tab gsl_sf_debye_3(dbl_expr)
@item gsl_sf_debye_4_e @tab Y @tab gsl_sf_debye_4(dbl_expr)
@item gsl_sf_debye_5_e @tab Y @tab gsl_sf_debye_5(dbl_expr)
@item gsl_sf_debye_6_e @tab Y @tab gsl_sf_debye_6(dbl_expr)
@item gsl_sf_dilog_e @tab N @tab gsl_sf_dilog
@item gsl_sf_complex_dilog_xy_e  @tab N @tab gsl_sf_complex_dilog_xy_e 
@item gsl_sf_complex_dilog_e @tab N @tab gsl_sf_complex_dilog
@item gsl_sf_complex_spence_xy_e   @tab N @tab gsl_sf_complex_spence_xy_e  
@item gsl_sf_multiply_e @tab N @tab gsl_sf_multiply
@item gsl_sf_multiply_err_e @tab N @tab gsl_sf_multiply_err
@item gsl_sf_ellint_Kcomp_e @tab Y @tab gsl_sf_ellint_Kcomp(dbl_expr)
@item gsl_sf_ellint_Ecomp_e @tab Y @tab gsl_sf_ellint_Ecomp(dbl_expr)
@item gsl_sf_ellint_Pcomp_e @tab Y @tab gsl_sf_ellint_Pcomp(dbl_expr,dbl_expr)
@item gsl_sf_ellint_Dcomp_e @tab Y @tab gsl_sf_ellint_Dcomp(dbl_expr)
@item gsl_sf_ellint_F_e @tab Y @tab gsl_sf_ellint_F(dbl_expr,dbl_expr)
@item gsl_sf_ellint_E_e @tab Y @tab gsl_sf_ellint_E(dbl_expr,dbl_expr)
@item gsl_sf_ellint_P_e @tab Y @tab gsl_sf_ellint_P(dbl_expr,dbl_expr,dbl_expr)
@item gsl_sf_ellint_D_e @tab Y @tab gsl_sf_ellint_D(dbl_expr,dbl_expr,dbl_expr)
@item gsl_sf_ellint_RC_e @tab Y @tab gsl_sf_ellint_RC(dbl_expr,dbl_expr)
@item gsl_sf_ellint_RD_e @tab Y @tab gsl_sf_ellint_RD(dbl_expr,dbl_expr,dbl_expr)
@item gsl_sf_ellint_RF_e @tab Y @tab gsl_sf_ellint_RF(dbl_expr,dbl_expr,dbl_expr)
@item gsl_sf_ellint_RJ_e @tab Y @tab gsl_sf_ellint_RJ(dbl_expr,dbl_expr,dbl_expr,dbl_expr)
@item gsl_sf_elljac_e @tab N @tab gsl_sf_elljac
@item gsl_sf_erfc_e @tab Y @tab gsl_sf_erfc(dbl_expr)
@item gsl_sf_log_erfc_e @tab Y @tab gsl_sf_log_erfc(dbl_expr)
@item gsl_sf_erf_e @tab Y @tab gsl_sf_erf(dbl_expr)
@item gsl_sf_erf_Z_e @tab Y @tab gsl_sf_erf_Z(dbl_expr)
@item gsl_sf_erf_Q_e @tab Y @tab gsl_sf_erf_Q(dbl_expr)
@item gsl_sf_hazard_e @tab Y @tab gsl_sf_hazard(dbl_expr)
@item gsl_sf_exp_e @tab Y @tab gsl_sf_exp(dbl_expr)
@item gsl_sf_exp_e10_e @tab N @tab gsl_sf_exp_e10
@item gsl_sf_exp_mult_e @tab Y @tab gsl_sf_exp_mult(dbl_expr,dbl_expr)
@item gsl_sf_exp_mult_e10_e @tab N @tab gsl_sf_exp_mult_e10
@item gsl_sf_expm1_e @tab Y @tab gsl_sf_expm1(dbl_expr)
@item gsl_sf_exprel_e @tab Y @tab gsl_sf_exprel(dbl_expr)
@item gsl_sf_exprel_2_e @tab Y @tab gsl_sf_exprel_2(dbl_expr)
@item gsl_sf_exprel_n_e @tab Y @tab gsl_sf_exprel_n(int_expr,dbl_expr)
@item gsl_sf_exp_err_e @tab Y @tab gsl_sf_exp_err(dbl_expr,dbl_expr)
@item gsl_sf_exp_err_e10_e @tab N @tab gsl_sf_exp_err_e10
@item gsl_sf_exp_mult_err_e @tab N @tab gsl_sf_exp_mult_err
@item gsl_sf_exp_mult_err_e10_e @tab N @tab gsl_sf_exp_mult_err_e10
@item gsl_sf_expint_E1_e @tab Y @tab gsl_sf_expint_E1(dbl_expr)
@item gsl_sf_expint_E2_e @tab Y @tab gsl_sf_expint_E2(dbl_expr)
@item gsl_sf_expint_En_e @tab Y @tab gsl_sf_expint_En(int_expr,dbl_expr)
@item gsl_sf_expint_E1_scaled_e @tab Y @tab gsl_sf_expint_E1_scaled(dbl_expr)
@item gsl_sf_expint_E2_scaled_e @tab Y @tab gsl_sf_expint_E2_scaled(dbl_expr)
@item gsl_sf_expint_En_scaled_e @tab Y @tab gsl_sf_expint_En_scaled(int_expr,dbl_expr)
@item gsl_sf_expint_Ei_e @tab Y @tab gsl_sf_expint_Ei(dbl_expr)
@item gsl_sf_expint_Ei_scaled_e @tab Y @tab gsl_sf_expint_Ei_scaled(dbl_expr)
@item gsl_sf_Shi_e @tab Y @tab gsl_sf_Shi(dbl_expr)
@item gsl_sf_Chi_e @tab Y @tab gsl_sf_Chi(dbl_expr)
@item gsl_sf_expint_3_e @tab Y @tab gsl_sf_expint_3(dbl_expr)
@item gsl_sf_Si_e @tab Y @tab gsl_sf_Si(dbl_expr)
@item gsl_sf_Ci_e @tab Y @tab gsl_sf_Ci(dbl_expr)
@item gsl_sf_atanint_e @tab Y @tab gsl_sf_atanint(dbl_expr)
@item gsl_sf_fermi_dirac_m1_e @tab Y @tab gsl_sf_fermi_dirac_m1(dbl_expr)
@item gsl_sf_fermi_dirac_0_e @tab Y @tab gsl_sf_fermi_dirac_0(dbl_expr)
@item gsl_sf_fermi_dirac_1_e @tab Y @tab gsl_sf_fermi_dirac_1(dbl_expr)
@item gsl_sf_fermi_dirac_2_e @tab Y @tab gsl_sf_fermi_dirac_2(dbl_expr)
@item gsl_sf_fermi_dirac_int_e @tab Y @tab gsl_sf_fermi_dirac_int(int_expr,dbl_expr)
@item gsl_sf_fermi_dirac_mhalf_e @tab Y @tab gsl_sf_fermi_dirac_mhalf(dbl_expr)
@item gsl_sf_fermi_dirac_half_e @tab Y @tab gsl_sf_fermi_dirac_half(dbl_expr)
@item gsl_sf_fermi_dirac_3half_e @tab Y @tab gsl_sf_fermi_dirac_3half(dbl_expr)
@item gsl_sf_fermi_dirac_inc_0_e @tab Y @tab gsl_sf_fermi_dirac_inc_0(dbl_expr,dbl_expr)
@item gsl_sf_lngamma_e @tab Y @tab gsl_sf_lngamma(dbl_expr)
@item gsl_sf_lngamma_sgn_e @tab N @tab gsl_sf_lngamma_sgn 
@item gsl_sf_gamma_e @tab Y @tab gsl_sf_gamma(dbl_expr)
@item gsl_sf_gammastar_e @tab Y @tab gsl_sf_gammastar(dbl_expr)
@item gsl_sf_gammainv_e @tab Y @tab gsl_sf_gammainv(dbl_expr)
@item gsl_sf_lngamma_complex_e @tab N @tab gsl_sf_lngamma_complex
@item gsl_sf_taylorcoeff_e @tab Y @tab gsl_sf_taylorcoeff(int_expr,dbl_expr)
@item gsl_sf_fact_e @tab Y @tab gsl_sf_fact(uint_expr)
@item gsl_sf_doublefact_e @tab Y @tab gsl_sf_doublefact(uint_expr)
@item gsl_sf_lnfact_e @tab Y @tab gsl_sf_lnfact(uint_expr)
@item gsl_sf_lndoublefact_e @tab Y @tab gsl_sf_lndoublefact(uint_expr)
@item gsl_sf_lnchoose_e @tab N @tab gsl_sf_lnchoose
@item gsl_sf_choose_e @tab N @tab gsl_sf_choose
@item gsl_sf_lnpoch_e @tab Y @tab gsl_sf_lnpoch(dbl_expr,dbl_expr)
@item gsl_sf_lnpoch_sgn_e @tab N @tab gsl_sf_lnpoch_sgn
@item gsl_sf_poch_e @tab Y @tab gsl_sf_poch(dbl_expr,dbl_expr)
@item gsl_sf_pochrel_e @tab Y @tab gsl_sf_pochrel(dbl_expr,dbl_expr)
@item gsl_sf_gamma_inc_Q_e @tab Y @tab gsl_sf_gamma_inc_Q(dbl_expr,dbl_expr)
@item gsl_sf_gamma_inc_P_e @tab Y @tab gsl_sf_gamma_inc_P(dbl_expr,dbl_expr)
@item gsl_sf_gamma_inc_e @tab Y @tab gsl_sf_gamma_inc(dbl_expr,dbl_expr)
@item gsl_sf_lnbeta_e @tab Y @tab gsl_sf_lnbeta(dbl_expr,dbl_expr)
@item gsl_sf_lnbeta_sgn_e @tab N @tab gsl_sf_lnbeta_sgn
@item gsl_sf_beta_e @tab Y @tab gsl_sf_beta(dbl_expr,dbl_expr)
@item gsl_sf_beta_inc_e @tab N @tab gsl_sf_beta_inc
@item gsl_sf_gegenpoly_1_e @tab Y @tab gsl_sf_gegenpoly_1(dbl_expr,dbl_expr)
@item gsl_sf_gegenpoly_2_e @tab Y @tab gsl_sf_gegenpoly_2(dbl_expr,dbl_expr)
@item gsl_sf_gegenpoly_3_e @tab Y @tab gsl_sf_gegenpoly_3(dbl_expr,dbl_expr)
@item gsl_sf_gegenpoly_n_e @tab N @tab gsl_sf_gegenpoly_n
@item gsl_sf_gegenpoly_array @tab Y @tab gsl_sf_gegenpoly_array
@item gsl_sf_hyperg_0F1_e @tab Y @tab gsl_sf_hyperg_0F1(dbl_expr,dbl_expr)
@item gsl_sf_hyperg_1F1_int_e @tab Y @tab gsl_sf_hyperg_1F1_int(int_expr,int_expr,dbl_expr)
@item gsl_sf_hyperg_1F1_e @tab Y @tab gsl_sf_hyperg_1F1(dbl_expr,dbl_expr,dbl_expr)
@item gsl_sf_hyperg_U_int_e @tab Y @tab gsl_sf_hyperg_U_int(int_expr,int_expr,dbl_expr)
@item gsl_sf_hyperg_U_int_e10_e @tab N @tab gsl_sf_hyperg_U_int_e10
@item gsl_sf_hyperg_U_e @tab Y @tab gsl_sf_hyperg_U(dbl_expr,dbl_expr,dbl_expr)
@item gsl_sf_hyperg_U_e10_e @tab N @tab gsl_sf_hyperg_U_e10
@item gsl_sf_hyperg_2F1_e @tab Y @tab gsl_sf_hyperg_2F1(dbl_expr,dbl_expr,dbl_expr,dbl_expr)
@item gsl_sf_hyperg_2F1_conj_e @tab Y @tab gsl_sf_hyperg_2F1_conj(dbl_expr,dbl_expr,dbl_expr,dbl_expr)
@item gsl_sf_hyperg_2F1_renorm_e @tab Y @tab gsl_sf_hyperg_2F1_renorm(dbl_expr,dbl_expr,dbl_expr,dbl_expr)
@item gsl_sf_hyperg_2F1_conj_renorm_e @tab Y @tab gsl_sf_hyperg_2F1_conj_renorm(dbl_expr,dbl_expr,dbl_expr,dbl_expr)
@item gsl_sf_hyperg_2F0_e @tab Y @tab gsl_sf_hyperg_2F0(dbl_expr,dbl_expr,dbl_expr)
@item gsl_sf_laguerre_1_e @tab Y @tab gsl_sf_laguerre_1(dbl_expr,dbl_expr)
@item gsl_sf_laguerre_2_e @tab Y @tab gsl_sf_laguerre_2(dbl_expr,dbl_expr)
@item gsl_sf_laguerre_3_e @tab Y @tab gsl_sf_laguerre_3(dbl_expr,dbl_expr)
@item gsl_sf_laguerre_n_e @tab Y @tab gsl_sf_laguerre_n(int_expr,dbl_expr,dbl_expr)
@item gsl_sf_lambert_W0_e @tab Y @tab gsl_sf_lambert_W0(dbl_expr)
@item gsl_sf_lambert_Wm1_e @tab Y @tab gsl_sf_lambert_Wm1(dbl_expr)
@item gsl_sf_legendre_Pl_e @tab Y @tab gsl_sf_legendre_Pl(int_expr,dbl_expr)
@item gsl_sf_legendre_Pl_array @tab Y @tab status=gsl_sf_legendre_Pl_array(int,double,&var_out)
@item gsl_sf_legendre_Pl_deriv_array @tab N @tab gsl_sf_legendre_Pl_deriv_array
@item gsl_sf_legendre_P1_e @tab Y @tab gsl_sf_legendre_P1(dbl_expr)
@item gsl_sf_legendre_P2_e @tab Y @tab gsl_sf_legendre_P2(dbl_expr)
@item gsl_sf_legendre_P3_e @tab Y @tab gsl_sf_legendre_P3(dbl_expr)
@item gsl_sf_legendre_Q0_e @tab Y @tab gsl_sf_legendre_Q0(dbl_expr)
@item gsl_sf_legendre_Q1_e @tab Y @tab gsl_sf_legendre_Q1(dbl_expr)
@item gsl_sf_legendre_Ql_e @tab Y @tab gsl_sf_legendre_Ql(int_expr,dbl_expr)
@item gsl_sf_legendre_Plm_e @tab Y @tab gsl_sf_legendre_Plm(int_expr,int_expr,dbl_expr)
@item gsl_sf_legendre_Plm_array  @tab Y @tab status=gsl_sf_legendre_Plm_array(int,int,double,&var_out) 
@item gsl_sf_legendre_Plm_deriv_array @tab N @tab gsl_sf_legendre_Plm_deriv_array
@item gsl_sf_legendre_sphPlm_e @tab Y @tab gsl_sf_legendre_sphPlm(int_expr,int_expr,dbl_expr)
@item gsl_sf_legendre_sphPlm_array @tab Y @tab status=gsl_sf_legendre_sphPlm_array(int,int,double,&var_out)
@item gsl_sf_legendre_sphPlm_deriv_array @tab N @tab gsl_sf_legendre_sphPlm_deriv_array
@item gsl_sf_legendre_array_size @tab N @tab gsl_sf_legendre_array_size
@item gsl_sf_conicalP_half_e @tab Y @tab gsl_sf_conicalP_half(dbl_expr,dbl_expr)
@item gsl_sf_conicalP_mhalf_e @tab Y @tab gsl_sf_conicalP_mhalf(dbl_expr,dbl_expr)
@item gsl_sf_conicalP_0_e @tab Y @tab gsl_sf_conicalP_0(dbl_expr,dbl_expr)
@item gsl_sf_conicalP_1_e @tab Y @tab gsl_sf_conicalP_1(dbl_expr,dbl_expr)
@item gsl_sf_conicalP_sph_reg_e @tab Y @tab gsl_sf_conicalP_sph_reg(int_expr,dbl_expr,dbl_expr)
@item gsl_sf_conicalP_cyl_reg_e @tab Y @tab gsl_sf_conicalP_cyl_reg(int_expr,dbl_expr,dbl_expr)
@item gsl_sf_legendre_H3d_0_e @tab Y @tab gsl_sf_legendre_H3d_0(dbl_expr,dbl_expr)
@item gsl_sf_legendre_H3d_1_e @tab Y @tab gsl_sf_legendre_H3d_1(dbl_expr,dbl_expr)
@item gsl_sf_legendre_H3d_e @tab Y @tab gsl_sf_legendre_H3d(int_expr,dbl_expr,dbl_expr)
@item gsl_sf_legendre_H3d_array @tab N @tab gsl_sf_legendre_H3d_array
@item gsl_sf_legendre_array_size @tab N @tab gsl_sf_legendre_array_size
@item gsl_sf_log_e @tab Y @tab gsl_sf_log(dbl_expr)
@item gsl_sf_log_abs_e @tab Y @tab gsl_sf_log_abs(dbl_expr)
@item gsl_sf_complex_log_e @tab N @tab gsl_sf_complex_log
@item gsl_sf_log_1plusx_e @tab Y @tab gsl_sf_log_1plusx(dbl_expr)
@item gsl_sf_log_1plusx_mx_e @tab Y @tab gsl_sf_log_1plusx_mx(dbl_expr)
@item gsl_sf_mathieu_a_array @tab N @tab gsl_sf_mathieu_a_array
@item gsl_sf_mathieu_b_array @tab N @tab gsl_sf_mathieu_b_array
@item gsl_sf_mathieu_a @tab N @tab gsl_sf_mathieu_a
@item gsl_sf_mathieu_b @tab N @tab gsl_sf_mathieu_b
@item gsl_sf_mathieu_a_coeff @tab N @tab gsl_sf_mathieu_a_coeff
@item gsl_sf_mathieu_b_coeff @tab N @tab gsl_sf_mathieu_b_coeff
@item gsl_sf_mathieu_ce @tab N @tab gsl_sf_mathieu_ce
@item gsl_sf_mathieu_se @tab N @tab gsl_sf_mathieu_se
@item gsl_sf_mathieu_ce_array @tab N @tab gsl_sf_mathieu_ce_array
@item gsl_sf_mathieu_se_array @tab N @tab gsl_sf_mathieu_se_array
@item gsl_sf_mathieu_Mc @tab N @tab gsl_sf_mathieu_Mc
@item gsl_sf_mathieu_Ms @tab N @tab gsl_sf_mathieu_Ms
@item gsl_sf_mathieu_Mc_array @tab N @tab gsl_sf_mathieu_Mc_array
@item gsl_sf_mathieu_Ms_array @tab N @tab gsl_sf_mathieu_Ms_array
@item gsl_sf_pow_int_e @tab N @tab gsl_sf_pow_int
@item gsl_sf_psi_int_e @tab Y @tab gsl_sf_psi_int(int_expr)
@item gsl_sf_psi_e @tab Y @tab gsl_sf_psi(dbl_expr)
@item gsl_sf_psi_1piy_e @tab Y @tab gsl_sf_psi_1piy(dbl_expr)
@item gsl_sf_complex_psi_e @tab N @tab gsl_sf_complex_psi
@item gsl_sf_psi_1_int_e @tab Y @tab gsl_sf_psi_1_int(int_expr)
@item gsl_sf_psi_1_e @tab Y @tab gsl_sf_psi_1(dbl_expr)
@item gsl_sf_psi_n_e @tab Y @tab gsl_sf_psi_n(int_expr,dbl_expr)
@item gsl_sf_synchrotron_1_e @tab Y @tab gsl_sf_synchrotron_1(dbl_expr)
@item gsl_sf_synchrotron_2_e @tab Y @tab gsl_sf_synchrotron_2(dbl_expr)
@item gsl_sf_transport_2_e @tab Y @tab gsl_sf_transport_2(dbl_expr)
@item gsl_sf_transport_3_e @tab Y @tab gsl_sf_transport_3(dbl_expr)
@item gsl_sf_transport_4_e @tab Y @tab gsl_sf_transport_4(dbl_expr)
@item gsl_sf_transport_5_e @tab Y @tab gsl_sf_transport_5(dbl_expr)
@item gsl_sf_sin_e @tab N @tab gsl_sf_sin
@item gsl_sf_cos_e @tab N @tab gsl_sf_cos
@item gsl_sf_hypot_e @tab N @tab gsl_sf_hypot
@item gsl_sf_complex_sin_e @tab N @tab gsl_sf_complex_sin
@item gsl_sf_complex_cos_e @tab N @tab gsl_sf_complex_cos
@item gsl_sf_complex_logsin_e @tab N @tab gsl_sf_complex_logsin
@item gsl_sf_sinc_e @tab N @tab gsl_sf_sinc
@item gsl_sf_lnsinh_e @tab N @tab gsl_sf_lnsinh
@item gsl_sf_lncosh_e @tab N @tab gsl_sf_lncosh
@item gsl_sf_polar_to_rect @tab N @tab gsl_sf_polar_to_rect
@item gsl_sf_rect_to_polar @tab N @tab gsl_sf_rect_to_polar
@item gsl_sf_sin_err_e @tab N @tab gsl_sf_sin_err
@item gsl_sf_cos_err_e @tab N @tab gsl_sf_cos_err
@item gsl_sf_angle_restrict_symm_e @tab N @tab gsl_sf_angle_restrict_symm
@item gsl_sf_angle_restrict_pos_e @tab N @tab gsl_sf_angle_restrict_pos
@item gsl_sf_angle_restrict_symm_err_e @tab N @tab gsl_sf_angle_restrict_symm_err
@item gsl_sf_angle_restrict_pos_err_e @tab N @tab gsl_sf_angle_restrict_pos_err
@item gsl_sf_zeta_int_e @tab Y @tab gsl_sf_zeta_int(int_expr)
@item gsl_sf_zeta_e @tab Y @tab gsl_sf_zeta(dbl_expr)
@item gsl_sf_zetam1_e @tab Y @tab gsl_sf_zetam1(dbl_expr)
@item gsl_sf_zetam1_int_e @tab Y @tab gsl_sf_zetam1_int(int_expr)
@item gsl_sf_hzeta_e @tab Y @tab gsl_sf_hzeta(dbl_expr,dbl_expr)
@item gsl_sf_eta_int_e @tab Y @tab gsl_sf_eta_int(int_expr)
@item gsl_sf_eta_e @tab Y @tab gsl_sf_eta(dbl_expr)
@end multitable

@html
<a name="gsl_int"></a> <!-- http://nco.sf.net/nco.html#gsl_int -->
@end html
@node GSL interpolation, GSL least-squares fitting, GSL special functions, ncap2 netCDF Arithmetic Processor
@subsection GSL interpolation
@cindex @acronym{GSL}
@noindent As of version 3.9.9 (released July, 2009), @acronym{NCO} has wrappers to the @acronym{GSL} interpolation functions.

@noindent  Given a set of data points (x1,y1)...(xn, yn) the @acronym{GSL} functions computes a continuous interpolating function @acronym{Y(x)} such that @acronym{Y(xi) = yi}. The interpolation is piecewise smooth, and its behavior at the end-points is determined by the type of interpolation used. For more information consult the @acronym{GSL} manual. 

@noindent Interpolation with @command{ncap2} is a two stage process. In the first stage, a @acronym{RAM} variable is created from the chosen interpolating function and the data set. This @acronym{RAM} variable holds in memory a @acronym{GSL} interpolation object. In the second stage, points along the interpolating function are calculated. If you have a very large data set or are interpolating many sets then consider deleting the @acronym{RAM} variable when it is redundant. Use the command @command{ram_delete(var_nm)}.   

@noindent A simple example

@example
@verbatim
x_in[$lon]={1.0,2.0,3.0,4.0};
y_in[$lon]={1.1,1.2,1.5,1.8};

// Ram variable is declared and defined here 
gsl_interp_cspline(&ram_sp,x_in,y_in);

x_out[$lon_grd]={1.1,2.0,3.0,3.1,3.99};

y_out=gsl_spline_eval(ram_sp,x_out);
y2=gsl_spline_eval(ram_sp,1.3);
y3=gsl_spline_eval(ram_sp,0.0);
ram_delete(ram_sp);

print(y_out); // 1.10472, 1.2, 1.4, 1.42658, 1.69680002 
print(y2);    // 1.12454 
print(y3);    // '_' 
@end verbatim
@end example

@noindent Note in the above example y3 is set to 'missing value' because 0.0 isn't within the input X range.  

@strong{@acronym{GSL} Interpolation Types}@*
@noindent All the interpolation functions have been implemented. These are:@*
gsl_interp_linear() @* gsl_interp_polynomial() @* gsl_interp_cspline()@*
gsl_interp_cspline_periodic()@* gsl_interp_akima() @* gsl_interp_akima_periodic() @*

@* @*
@strong{ Evaluation of Interpolating Types } @*
@noindent @strong{Implemented} @*
gsl_spline_eval() @*
@noindent @strong{Unimplemented} @*
gsl_spline_deriv()@*
gsl_spline_deriv2()@*
gsl_spline_integ()@*

@html
<a name="ncap_lsqf"></a> <!-- http://nco.sf.net/nco.html#ncap_lsqf -->
@end html
@node GSL least-squares fitting, GSL statistics, GSL interpolation, ncap2 netCDF Arithmetic Processor
@subsection  GSL least-squares fitting       
Least Squares fitting is a method of calculating a straight line
through a set of experimental data points in the XY plane.
Data may be weighted or unweighted.
For more information please refer to the @acronym{GSL} manual. 

@noindent These @acronym{GSL} functions fall into three categories:@*
@strong{A)} Fitting data to Y=c0+c1*X@*
@strong{B)} Fitting data (through the origin) Y=c1*X@*
@strong{C)} Multi-parameter fitting (not yet implemented)@* 

@strong{Section A} @*
@code{status=@strong{gsl_fit_linear} (data_x,stride_x,data_y,stride_y,n,&co,&c1,&cov00,&cov01,&cov11,&sumsq)}

@noindent @strong{Input variables}: data_x, stride_x, data_y, stride_y, n @* 
From the above variables an X and Y vector both of length 'n' are derived. 
If data_x or data_y is less than type double then it is converted to type @code{double}. 
It is up to you to do bounds checking on the input data. 
For example if stride_x=3 and n=8 then the size of data_x must be at least 24

@noindent @strong{Output variables}: c0, c1, cov00, cov01, cov11,sumsq @* 
The '&' prefix indicates that these are call-by-reference variables.
If any of the output variables don't exist prior to the call then they are created on the fly as scalar variables of type @code{double}. If they already exist then their existing value is overwritten. If the function call is successful then @code{status=0}. 

@code{status= @strong{gsl_fit_wlinear}(data_x,stride_x,data_w,stride_w,data_y,stride_y,n,&co,&c1,&cov00,&cov01,&cov11,&chisq) }

@noindent Similar to the above call except it creates an additional weighting vector from the variables data_w, stride_w, n  


@code{ data_y_out=@strong{gsl_fit_linear_est}(data_x,c0,c1,cov00,cov01,cov11) }

@noindent This function calculates y values along the line Y=c0+c1*X @* @*

@strong{Section B} @*
@code{status=@strong{gsl_fit_mul}(data_x,stride_x,data_y,stride_y,n,&c1,&cov11,&sumsq)}

@noindent @strong{Input variables}: data_x, stride_x, data_y, stride_y, n @* 
From the above variables an X and Y vector both of length 'n' are derived. 
If data_x or data_y is less than type @code{double} then it is converted to type @code{double}. @*

@noindent @strong{Output variables}: c1,cov11,sumsq @* 

@code{status= @strong{gsl_fit_wmul}(data_x,stride_x,data_w,stride_w,data_y,stride_y,n,&c1,&cov11,&sumsq)}

@noindent Similar to the above call except it creates an additional weighting vector from the variables data_w, stride_w, n  

@code{data_y_out=@strong{gsl_fit_mul_est}(data_x,c0,c1,cov11)}

@noindent This function calculates y values along the line Y=c1*X @*

@noindent The below example shows @strong{gsl_fit_linear()} in action
@example
@verbatim
defdim("d1",10);
xin[d1]={1,2,3,4,5,6,7,8,9,10.0};
yin[d1]={3.1,6.2,9.1,12.2,15.1,18.2,21.3,24.0,27.0,30.0};
gsl_fit_linear(xin,1,yin,1,$d1.size,&c0,&c1,&cov00,&cov01,&cov11,&sumsq);
print(c0);  // 0.2
print(c1);  // 2.98545454545

defdim("e1",4);
xout[e1]={1.0,3.0,4.0,11};
yout[e1]=0.0;

yout=gsl_fit_linear_est(xout,c0,c1,cov00,cov01,cov11,sumsq);

print(yout); // 3.18545454545, 9.15636363636, 12.1418181818, 33.04
@end verbatim
@end example

@noindent The following code does linear regression of sst(time,lat,lon) for each time-step@*
@example
@verbatim
// Declare variables
c0[$lat, $lon]=0.; // Intercept
c1[$lat, $lon]=0.; // Slope
sdv[$lat, $lon]=0.; // Standard deviation
covxy[$lat, $lon]=0.; // Covariance
for (i=0;i<$lat.size;i++) // Loop over lat
{
  for (j=0;j<$lon.size;j++) // Loop over lon
  {
      // Linear regression function
      gsl_fit_linear(time,1,sst(:, i, j),1,$time.size,&tc0,&tc1,&cov00,&cov01,&cov11,&sumsq); 
      c0(i,j)=tc0; // Output results
      c1(i,j)=tc1; // Output results
      // Covariance function
      covxy(i,j)=gsl_stats_covariance(time,1,$time.size,double(sst(:,i,j)),1,$time.size); 
      // Standard deviation function
      sdv(i,j)=gsl_stats_sd(sst(:,i,j),1,$time.size); 
  }
 }
// slope (c1) missing values are set to '0', change to -999. (variable c0 intercept value)
where(c0 == -999) c1=-999;
@end verbatim
@end example

@html
<a name="ncap_stat"></a> <!-- http://nco.sf.net/nco.html#ncap_stat -->
@end html

@node GSL statistics, GSL random number generation, GSL least-squares fitting, ncap2 netCDF Arithmetic Processor
@subsection GSL statistics

@noindent Wrappers for most of the @acronym{GSL} Statistical functions have been implemented. The @acronym{GSL} function names include a type specifier (except for type double functions). To obtain the equivalent @acronym{NCO} name simply remove the type specifier; then depending on the data type the appropriate @acronym{GSL} function  is called. The weighed statistical functions e.g., @code{ gsl_stats_wvariance()} are only defined in @acronym{GSL} for floating-point types; so your data must of type @code{float} or @code{double} otherwise ncap2 will emit an error message. To view the implemented functions use the shell command @command{ncap2 -f|grep _stats}   

@noindent @acronym{GSL} Functions
@example
short gsl_stats_max (short data[], size_t stride, size_t n);
double gsl_stats_int_mean (int data[], size_t stride, size_t n);
double gsl_stats_short_sd_with_fixed_mean (short data[], size_t stride, size_t n, double mean);
double gsl_stats_wmean (double w[], size_t wstride, double data[], size_t stride, size_t n);
double gsl_stats_quantile_from_sorted_data (double sorted_data[], size_t stride, size_t n, double f) ;
@end example

@noindent Equivalent ncap2 wrapper functions
@example
short gsl_stats_max (var_data, data_stride, n);
double gsl_stats_mean (var_data, data_stride, n);
double gsl_stats_sd_with_fixed_mean (var_data, data_stride, n, var_mean);
double gsl_stats_wmean (var_weight, weight_stride, var_data, data_stride, n, var_mean);
double gsl_stats_quantile_from_sorted_data (var_sorted_data, data_stride, n, var_f) ;
@end example

@noindent @acronym{GSL} has no notion of missing values or dimensionality beyond one. If your data has missing values which you want ignored in the calculations then use the @command{ncap2} built in aggregate functions(@ref{Methods and functions}). The @acronym{GSL} functions operate on a vector of values created from the var_data/stride/n arguments. The ncap wrappers check that there is no bounding error with regard to the size of the data and the final value in the vector. 
@example
@verbatim
a1[time]={1,2,3,4,5,6,7,8,9,10};

a1_avg=gsl_stats_mean(a1,1,10);
print(a1_avg); // 5.5

a1_var=gsl_stats_variance(a1,4,3);
print(a1_var); // 16.0

// bounding error, vector attempts to access element a1(10)
a1_sd=gsl_stats_sd(a1,5,3); 
@end verbatim
@end example

@noindent For functions with the signature 
@strong{func_nm(var_data,data_stride,n)}, 
one may omit the second or third arguments. 
The default value for @var{stride} is @code{1}. 
The default value for @var{n} is @code{1+(data.size()-1)/stride}.

@example
// Following statements are equvalent
n2=gsl_stats_max(a1,1,10)
n2=gsl_stats_max(a1,1);
n2=gsl_stats_max(a1);

// Following statements are equvalent
n3=gsl_stats_median_from_sorted_data(a1,2,5);
n3=gsl_stats_median_from_sorted_data(a1,2);

// Following statements are NOT equvalent
n4=gsl_stats_kurtosis(a1,3,2);
n4=gsl_stats_kurtosis(a1,3); //default n=4
@end example 

The following example illustrates some of the weighted functions.
The data are randomly generated. 
In this case the value of the weight for each datum is either 0.0 or 1.0   
@example
defdim("r1",2000);
data[r1]=1.0;

// Fill with random numbers [0.0,10.0)
data=10.0*gsl_rng_uniform(data);

// Create a weighting variable
weight=(data>4.0);

wmean=gsl_stats_wmean(weight,1,data,1,$r1.size);
print(wmean);

wsd=gsl_stats_wsd(weight,1,data,1,$r1.size);
print(wsd);

// number of values in data that are greater than 4
weight_size=weight.total();
print(weight_size);

// print min/max of data 
dmin=data.gsl_stats_min();
dmax=data.gsl_stats_max();
print(dmin);print(dmax);
@end example

@html
<a name="ncap_rng"></a> <!-- http://nco.sf.net/nco.html#ncap_rng -->
@end html

@node GSL random number generation, Examples ncap2, GSL statistics, ncap2 netCDF Arithmetic Processor
@subsection GSL random number generation
The @acronym{GSL} library has a large number of random number generators. In addition there are a large set of functions for turning uniform random numbers into discrete or continuous probabilty distributions. The random number generator algorithms vary in terms of quality numbers output, speed of execution and maximum number output. For more information see the @acronym{GSL} documentation. The algorithm and seed are set via environment variables, these are picked up by the @code{ncap2} code.

@noindent @strong{Setup} @*  
The number algorithm is set by the environment variable @code{GSL_RNG_TYPE}. If this variable isn't set then the default rng algorithm is gsl_rng_19937. The seed is set with the environment variable @code{GSL_RNG_SEED}. The following wrapper functions in ncap2 provide information about the chosen algorithm. @* 

@table @code
@item gsl_rng_min() 
the minimum value returned by the rng algorithm.
@item gsl_rng_max()
the maximum value returned by the rng algorithm. 
@end table

@noindent @strong{Uniformly Distributed Random Numbers} 
@table @code
@item gsl_rng_get(var_in)
This function returns var_in with integers from the chosen rng algorithm. The min and max values depend uoon the chosen rng algorthm.
@item gsl_rng_uniform_int(var_in)
This function returns var_in with random integers from 0 to n-1. The value n must be less than or equal to the maximum value of the chosen rng algorithm. 
@item gsl_rng_uniform(var_in)
This function returns var_in with double-precision numbers in the range [0.0,1). The range includes 0.0 and excludes 1.0.
@item gsl_rng_uniform_pos(var_in)
This function returns var_in with double-precision numbers in the range (0.0,1), excluding both 0.0 and 1.0.
@end table

@noindent Below are examples of @code{gsl_rng_get()} and @code{gsl_rng_uniform_int()} in action.

@example
export GSL_RNG_TYPE=ranlux
export GSL_RNG_SEED=10
ncap2 -v -O -s 'a1[time]=0;a2=gsl_rng_get(a1);' in.nc foo.nc 
// 10 random numbers from the range 0 - 16777215
// a2=9056646, 12776696, 1011656, 13354708, 5139066, 1388751, 11163902, 7730127, 15531355, 10387694 ;

ncap2 -v -O -s 'a1[time]=21;a2=gsl_rng_uniform_int(a1).sort();' in.nc foo.nc
// 10 random numbers from the range 0 - 20
a2 = 1, 1, 6, 9, 11, 13, 13, 15, 16, 19 ;

@end example

@noindent The following example produces an @code{ncap2} runtime error. This is because the chose rng algorithm has a maximum value greater than @code{NC_MAX_INT=2147483647}; the wrapper functions to @code{gsl_rng_get()} and @code{gsl_rng_uniform_int()} return variable of type @code{NC_INT}. Please be aware of this when using random number distribution functions functions from the @acronym{GSL} library which return @code{unsigned int}. Examples of these are @code{gsl_ran_geometric()} and @code{gsl_ran_pascal()}. 

@example
export GSL_RNG_TYPE=mt19937
ncap2 -v -O -s 'a1[time]=0;a2=gsl_rng_get(a1);' in.nc foo.nc 
@end example

@noindent To find the maximum value of the chosen rng algorithm use the following code snippet.
@example
ncap2 -v -O -s 'rng_max=gsl_rng_max();print(rng_max)' in.nc foo.nc
@end example

@noindent @strong{Random Number Distributions} @*
The @acronym{GSL} library has a rich set of random number disribution functions. The library also provides cumulative distribution functions and inverse cumulative distribution functions sometimes referred to a quantile functions. To see whats available on your build use the shell command @code{ncap2 -f|grep -e _ran -e _cdf}.

@noindent The following examples all return variables of type @code{NC_INT} @*
@example
defdim("out",15);
a1[$out]=0.5;
a2=gsl_ran_binomial(a1,30).sort();
//a2 = 10, 11, 12, 12, 13, 14, 14, 15, 15, 16, 16, 16, 16, 17, 22 ;
a3=gsl_ran_geometric(a2).sort();
//a2 = 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 3, 4, 5 ;
a4=gsl_ran_pascal(a2,50);
//a5 = 37, 40, 40, 42, 43, 45, 46, 49, 52, 58, 60, 62, 62, 65, 67 ;
@end example

@noindent The following all return variables of type @code{NC_DOUBLE};
@example
defdim("b1",1000);
b1[$b1]=0.8;
b2=gsl_ran_exponential(b1);
b2_avg=b2.avg();
print(b2_avg);
// b2_avg = 0.756047976787

b3=gsl_ran_gaussian(b1);
b3_avg=b3.avg();
b3_rms=b3.rms();
print(b3_avg);
// b3_avg = -0.00903446534258;
print(b3_rms);
// b3_rms = 0.81162979889;

b4[$b1]=10.0;
b5[$b1]=20.0;
b6=gsl_ran_flat(b4,b5);
b6_avg=b6.avg();
print(b6_avg);
// b6_avg=15.0588129413
@end example

@html
<a name="ncap_emp"></a> <!-- http://nco.sf.net/nco.html#ncap_emp -->
@end html
@node Examples ncap2, Intrinsic mathematical methods, GSL random number generation, ncap2 netCDF Arithmetic Processor
@subsection Examples ncap2

See the @file{ncap.in} and @file{ncap2.in} scripts released with @acronym{NCO} 
for more complete demonstrations of @command{ncap2} functionality
(script available on-line at @url{http://nco.sf.net/ncap2.in}).

Define new attribute @var{new} for existing variable @var{one}
as twice the existing attribute @var{double_att} of variable
@var{att_var}: 
@example
@verbatim
ncap2 -s 'one@new=2*att_var@double_att' in.nc out.nc
@end verbatim
@end example

Average variables of mixed types (result is of type @code{double}):
@example
ncap2 -s 'average=(var_float+var_double+var_int)/3' in.nc out.nc 
@end example

Multiple commands may be given to @command{ncap2} in three ways.
First, the commands may be placed in a script which is executed, e.g.,
@file{tst.nco}. 
Second, the commands may be individually specified with multiple
@samp{-s} arguments to the same @command{ncap2} invocation.
Third, the commands may be chained into a single @samp{-s}
argument to @command{ncap2}.
Assuming the file @file{tst.nco} contains the commands
@code{a=3;b=4;c=sqrt(a^2+b^2);}, then the following @command{ncap2}
invocations produce identical results:
@example
ncap2 -v -S tst.nco in.nc out.nc
ncap2 -v -s 'a=3' -s 'b=4' -s 'c=sqrt(a^2+b^2)' in.nc out.nc
ncap2 -v -s 'a=3;b=4;c=sqrt(a^2+b^2)' in.nc out.nc
@end example
The second and third examples show that @command{ncap2} does not require
that a trailing semi-colon @samp{;} be placed at the end of a @samp{-s}
argument, although a trailing semi-colon @samp{;} is always allowed.
However, semi-colons are required to separate individual assignment
statements chained together as a single @samp{-s} argument. 

@html
<a name="xmp_grw"></a> <!-- http://nco.sf.net/nco.html#xmp_grw -->
@end html
@cindex growing dimensions
@cindex dimensions, growing
@command{ncap2} may be used to ``grow'' dimensions, i.e., to increase
dimension sizes without altering existing data.
Say @file{in.nc} has @code{ORO(lat,lon)} and the user wishes a new
file with @code{new_ORO(new_lat,new_lon)} that contains zeros in the
undefined portions of the new grid.
@example
defdim("new_lat",$lat.size+1); // Define new dimension sizes
defdim("new_lon",$lon.size+1);
new_ORO[$new_lat,$new_lon]=0.0f; // Initialize to zero
new_ORO(0:$lat.size-1,0:$lon.size-1)=ORO; // Fill valid data
@end example
The commands to define new coordinate variables @code{new_lat}
and @code{new_lon} in the output file follow a similar pattern.
One would might store these commands in a script @file{grow.nco}
and then execute the script with
@example
ncap2 -v -S grow.nco in.nc out.nc
@end example

@html
<a name="flg"></a> <!-- http://nco.sf.net/nco.html#flg -->
@end html
@cindex flags
Imagine you wish to create a binary flag based on the value of 
an array.
The flag should have @w{value 1.0} where the array @w{exceeds 1.0},
and @w{value 0.0} elsewhere.
This example creates the binary flag @code{ORO_flg} in @file{out.nc}
from the continuous array named @code{ORO} in @file{in.nc}.
@example
ncap2 -s 'ORO_flg=(ORO > 1.0)' in.nc out.nc
@end example
Suppose your task is to change all values of @code{ORO} which 
@w{equal 2.0} to the new @w{value 3.0}:
@example
ncap2 -s 'ORO_msk=(ORO==2.0);ORO=ORO_msk*3.0+!ORO_msk*ORO' in.nc out.nc
@end example
@cindex mask
This creates and uses @code{ORO_msk} to mask the subsequent arithmetic
operation.
Values of @code{ORO} are only changed where @code{ORO_msk} is true,
i.e., where @code{ORO} @w{equals 2.0} @*
Using the @code{where} statement the above code simplifies to :
@example
ncap2 -s 'where(ORO == 2.0) ORO=3.0;' in.nc foo.nc
@end example

@html
<a name="cvr"></a> <!-- http://nco.sf.net/nco.html#cvr -->
@end html
This example uses @command{ncap2} to compute the covariance of two
variables. 
Let the variables @var{@wndznl{}} and @var{@wndmrd{}} be the horizontal 
wind components. 
@cindex covariance
@c fxm 20030423: texi2html 1.64 has problems with this legal syntax but makeinfo --html does not
The @dfn{covariance} of @var{@wndznl{}} and @var{@wndmrd{}} is defined
as the time mean product of the deviations of @var{@wndznl{}} and
@var{@wndmrd{}} from their respective time means.
Symbolically, the covariance 
@set flg
@tex
$[\wndznl^{\prime} \wndmrd^{\prime}] =
[\wndznl\wndmrd]-[\wndznl][\wndmrd]$ where $[\xxx]$ denotes the
time-average of~$\xxx$, 
$[\xxx] \equiv {1 \over \tau} \int_{\tm=0}^{\tm=\tau} \xxx(\tm) \,\dfr\tm$
and $\xxx^{\prime}$ 
@clear flg 
@end tex
@ifinfo
@math{[@var{@wndznl{}'@wndmrd{}'}] =
[@var{@wndznl{}@wndmrd{}}]-[@var{@wndznl{}}][@var{@wndmrd{}}]} 
where @math{[@var{@xxx{}}]} denotes the time-average of
@math{@var{@xxx{}}} and @math{@var{@xxx{}'}} 
@clear flg
@end ifinfo
@ifset flg
@c texi2html does not like @math{}
[@var{@wndznl{}'@wndmrd{}'}] =
[@var{@wndznl{}@wndmrd{}}]-[@var{@wndznl{}}][@var{@wndmrd{}}] where
[@xxx{}] denotes the time-average of @var{@xxx{}} and @var{@xxx{}'}
@clear flg
@end ifset
denotes the deviation from the time-mean. 
The covariance tells us how much of the correlation of two signals
arises from the signal fluctuations versus the mean signals.
@cindex eddy covariance
Sometimes this is called the @dfn{eddy covariance}.
We will store the covariance in the variable @code{uprmvprm}.
@example
ncwa -O -a time -v u,v in.nc foo.nc # Compute time mean of u,v
ncrename -O -v u,uavg -v v,vavg foo.nc # Rename to avoid conflict
ncks -A -v uavg,vavg foo.nc in.nc # Place time means with originals
ncap2 -O -s 'uprmvprm=u*v-uavg*vavg' in.nc in.nc # Covariance
ncra -O -v uprmvprm in.nc foo.nc # Time-mean covariance
@end example
The mathematically inclined will note that the same covariance would be
obtained by replacing the step involving @command{ncap2} with
@example
ncap2 -O -s 'uprmvprm=(u-uavg)*(v-vavg)' foo.nc foo.nc # Covariance
@end example

As of @acronym{NCO} version 3.1.8 (December, 2006), @command{ncap2}
can compute averages, and thus covariances, by itself:
@example
ncap2 -s 'uavg=u.avg($time);vavg=v.avg($time);uprmvprm=u*v-uavg*vavg' \
      -s 'uprmvrpmavg=uprmvprm.avg($time)' in.nc foo.nc
@end example
We have not seen a simpler method to script and execute powerful
arithmetic than @command{ncap2}.

@cindex globbing
@cindex shell
@cindex quotes
@cindex extended regular expressions
@cindex regular expressions
@command{ncap2} utilizes many meta-characters 
(e.g., @samp{$}, @samp{?}, @samp{;}, @samp{()}, @samp{[]})
that can confuse the command-line shell if not quoted properly.
The issues are the same as those which arise in utilizing extended
regular expressions to subset variables (@pxref{Subsetting Files}).
The example above will fail with no quotes and with double quotes.
This is because shell globbing tries to @dfn{interpolate} the value of
@code{$time} from the shell environment unless it is quoted:
@example
ncap2 -s 'uavg=u.avg($time)'  in.nc foo.nc # Correct (recommended)
ncap2 -s  uavg=u.avg('$time') in.nc foo.nc # Correct (and dangerous)
ncap2 -s  uavg=u.avg($time)   in.nc foo.nc # Wrong ($time = '')
ncap2 -s "uavg=u.avg($time)"  in.nc foo.nc # Wrong ($time = '')
@end example
Without the single quotes, the shell replaces @code{$time} with an
empty string.
The command @command{ncap2} receives from the shell is
@code{uavg=u.avg()}. 
This causes @command{ncap2} to average over all dimensions rather than
just the @var{time} dimension, and unintended consequence.

We recommend using single quotes to protect @command{ncap2}
command-line scripts from the shell, even when such protection is not
strictly necessary. 
Expert users may violate this rule to exploit the ability to use shell
variables in @command{ncap2} command-line scripts 
(@pxref{CCSM Example}). 
In such cases it may be necessary to use the shell backslash character
@samp{\} to protect the @command{ncap2} meta-character.


@cindex appending data
@cindex time-averaging
@findex ncks
@findex ncwa
@findex ncra
@cindex degenerate dimension
@cindex @samp{-b}
A dimension of size one is said to be @emph{degenerate}.
Whether a degenerate record dimension is desirable or not
depends on the application.
Often a degenerate @var{time} dimension is useful, e.g., for
concatentating, but it may cause problems with arithmetic.
Such is the case in the above example, where the first step employs
@command{ncwa} rather than @command{ncra} for the time-averaging.
Of course the numerical results are the same with both operators.
The difference is that, unless @samp{-b} is specified, @command{ncwa}
writes no @var{time} dimension to the output file, while @command{ncra}
defaults to keeping @var{time} as a degenerate (@w{size 1}) dimension. 
Appending @code{u} and @code{v} to the output file would cause
@command{ncks} to try to expand the degenerate time axis of @code{uavg}
and @code{vavg} to the size of the non-degenerate @var{time} dimension
in the input file.
Thus the append (@command{ncks -A}) command would be undefined (and
should fail) in this case. 
@cindex @code{-C}
Equally important is the @samp{-C} argument 
(@pxref{Subsetting Coordinate Variables}) to @command{ncwa} to prevent
any scalar @var{time} variable from being written to the output file.  
Knowing when to use @command{ncwa -a time} rather than the default
@command{ncra} for time-averaging takes, well, time.

@html
<a name="mth"></a> <!-- http://nco.sf.net/nco.html#mth -->
@end html
@node Intrinsic mathematical methods, Operator precedence and associativity , Examples ncap2, ncap2 netCDF Arithmetic Processor
@subsection Intrinsic mathematical methods
@command{ncap2} supports the standard mathematical functions supplied with
most operating systems.
@cindex addition
@cindex subtraction
@cindex multiplication
@cindex division
@cindex exponentiation
@cindex power
@cindex modulus
@cindex @code{+} (addition)
@cindex @code{-} (subtraction)
@cindex @code{*} (multiplication)
@cindex @code{/} (division)
@cindex @code{^} (power)
@cindex @code{%} (modulus)
Standard calculator notation is used for addition @kbd{+}, subtraction
@kbd{-}, multiplication @kbd{*}, division @kbd{/}, exponentiation
@kbd{^}, and modulus @kbd{%}.
The available elementary mathematical functions are: 
@cindex @var{abs}
@cindex @var{acosh}
@cindex @var{acos}
@cindex @var{asinh}
@cindex @var{asin}
@cindex @var{atanh}
@cindex @var{atan}
@cindex @var{ceil}
@cindex @var{cosh}
@cindex @var{cos}
@cindex @var{erfc}
@cindex @var{erf}
@cindex @var{exp}
@cindex @var{floor}
@cindex @var{gamma}
@cindex @var{ln}
@cindex @var{log10}
@cindex @var{log}
@cindex @var{nearbyint}
@cindex @var{pow}
@cindex @var{rint}
@cindex @var{round}
@cindex @var{sinh}
@cindex @var{sin}
@cindex @var{sqrt}
@cindex @var{tanh}
@cindex @var{tan}
@cindex @var{trunc}
@cindex mathematical functions
@cindex nearest integer function (inexact)
@cindex nearest integer function (exact)
@cindex rounding functions
@cindex truncation function
@cindex absolute value
@cindex arccosine function
@cindex arcsine function
@cindex arctangent function
@cindex ceiling function
@cindex complementary error function
@cindex cosine function
@cindex error function
@cindex exponentiation function
@cindex floor function
@cindex gamma function
@cindex hyperbolic arccosine function
@cindex hyperbolic arcsine function
@cindex hyperbolic arctangent function
@cindex hyperbolic cosine function
@cindex hyperbolic sine function
@cindex hyperbolic tangent
@cindex logarithm, base 10
@cindex logarithm, natural
@cindex power function
@cindex sine function
@cindex square root function
@table @code
@item abs(x)
@dfn{Absolute value}
@tex
Absolute value of $x$, $|x|$.
@end tex
@ifinfo
Absolute value of @var{x}.
@end ifinfo
Example: 
@tex
abs$(-1) = 1$
@end tex
@ifinfo
@math{abs(-1) = 1}
@end ifinfo
@item acos(x)
@dfn{Arc-cosine}
Arc-cosine of @var{x} where @var{x} is specified in radians.
Example: 
@tex
acos$(1.0) = 0.0$
@end tex
@ifinfo
@math{acos(1.0) = 0.0}
@end ifinfo
@item acosh(x)
@dfn{Hyperbolic arc-cosine}
Hyperbolic arc-cosine of @var{x} where @var{x} is specified in radians.
Example: 
@tex
acosh$(1.0) = 0.0$
@end tex
@ifinfo
@math{acosh(1.0) = 0.0}
@end ifinfo
@item asin(x)
@dfn{Arc-sine}
Arc-sine of @var{x} where @var{x} is specified in radians.
Example: 
@tex
asin$(1.0) = 1.57079632679489661922$
@end tex
@ifinfo
@math{asin(1.0) = 1.57079632679489661922}
@end ifinfo
@item asinh(x)
@dfn{Hyperbolic arc-sine}
Hyperbolic arc-sine of @var{x} where @var{x} is specified in radians.
Example: 
@tex
asinh$(1.0) = 0.88137358702$
@end tex
@ifinfo
@math{asinh(1.0) = 0.88137358702}
@end ifinfo
@item atan(x)
@dfn{Arc-tangent}
Arc-tangent of @var{x} where @var{x} is specified in radians between 
@tex
$-\pi/2$ and $\pi/2$.
@end tex
@ifinfo
@math{-pi/2} and @math{pi/2}.
@end ifinfo
Example: 
@tex
atan$(1.0) = 0.78539816339744830961$
@end tex
@ifinfo
@math{atan(1.0) = 0.78539816339744830961}
@end ifinfo

@item atan2(y,x)
@dfn{Arc-tangent2}
Arc-tangent of @var{y/x} 
@ifinfo
@math{:Example atan2(1,3) =  0.321689857}
@end ifinfo

@item atanh(x)
@dfn{Hyperbolic arc-tangent}
Hyperbolic arc-tangent of @var{x} where @var{x} is specified in radians between 
@tex
$-\pi/2$ and $\pi/2$.
@end tex
@ifinfo
@math{-pi/2} and @math{pi/2}.
@end ifinfo
Example:
@tex
atanh$(3.14159265358979323844) = 1.0$
@end tex
@ifinfo
@math{atanh(3.14159265358979323844) = 1.0}
@end ifinfo
@item ceil(x)
@dfn{Ceil}
Ceiling of @var{x}. Smallest integral value not less than argument.
Example: 
@tex
ceil$(0.1) = 1.0$
@end tex
@ifinfo
@math{ceil(0.1) = 1.0}
@end ifinfo
@item cos(x)
@dfn{Cosine}
Cosine of @var{x} where @var{x} is specified in radians.
Example: 
@tex
cos$(0.0) = 1.0$
@end tex
@ifinfo
@math{cos(0.0) = 1.0}
@end ifinfo
@item cosh(x)
@dfn{Hyperbolic cosine}
Hyperbolic cosine of @var{x} where @var{x} is specified in radians.
Example: 
@tex
cosh$(0.0) = 1.0$
@end tex
@ifinfo
@math{cosh(0.0) = 1.0}
@end ifinfo
@item erf(x)
@dfn{Error function}
Error function of @var{x} where @var{x} is specified between
@tex
$-1$ and $1$.
@end tex
@ifinfo
@math{-1} and @math{1}.
@end ifinfo
Example: 
@tex
erf$(1.0) = 0.842701$
@end tex
@ifinfo
@math{erf(1.0) = 0.842701}
@end ifinfo
@item erfc(x)
@dfn{Complementary error function}
Complementary error function of @var{x} where @var{x} is specified between
@tex
$-1$ and $1$.
@end tex
@ifinfo
@math{-1} and @math{1}.
@end ifinfo
Example: 
@tex
erfc$(1.0) = 0.15729920705$
@end tex
@ifinfo
@math{erfc(1.0) = 0.15729920705}
@end ifinfo
@item exp(x)
@dfn{Exponential}
Exponential of @var{x},
@tex
$e^{x}$.
@end tex
@ifinfo
@math{e^x}.
@end ifinfo
Example: 
@tex
exp$(1.0) = 2.71828182845904523536$
@end tex
@ifinfo
@math{exp(1.0) = 2.71828182845904523536}
@end ifinfo
@item floor(x)
@dfn{Floor}
Floor of @var{x}. Largest integral value not greater than argument.
Example: 
@tex
floor$(1.9) = 1$
@end tex
@ifinfo
@math{floor(1.9) = 1}
@end ifinfo
@item gamma(x)
@dfn{Gamma function}
Gamma function of @var{x},
@tex
$\Gamma(x)$.
@end tex
@ifinfo
@math{Gamma(x)}.
@end ifinfo
The well-known and loved continuous factorial function.
Example: 
@tex
gamma$(0.5) = \sqrt{\pi}$
@end tex
@ifinfo
@math{gamma(0.5) = sqrt(pi)}
@end ifinfo
@item gamma_inc_P(x)
@dfn{Incomplete Gamma function}
Incomplete Gamma function of parameter @var{a} and variable @var{x},
@tex
$P(a,x)$.
@end tex
@ifinfo
@math{gamma_inc_P(a,x)}.
@end ifinfo
One of the four incomplete gamma functions.
Example: 
@tex
gamma\_inc\_P$(1,1) = 1-\me^{-1}$
@end tex
@ifinfo
@math{gamma_inc_P(1,1) = 1-1/e}
@end ifinfo
@item ln(x)
@dfn{Natural Logarithm}
Natural logarithm of @var{x},
@tex
$\ln(x)$.
@end tex
@ifinfo
@math{ln(x)}.
@end ifinfo
Example: 
@tex
ln$(2.71828182845904523536) = 1.0$
@end tex
@ifinfo
@math{ln(2.71828182845904523536) = 1.0}
@end ifinfo
@item log(x)
@dfn{Natural Logarithm}
Exact synonym for @code{ln(x)}.
@item log10(x)
@dfn{Base 10 Logarithm}
@w{Base 10} logarithm of @var{x}, 
@tex
$\log_{10}(x)$.
@end tex
@ifinfo
@math{log10(x)}.
@end ifinfo
Example: 
@tex
log$(10.0) = 1.0$
@end tex
@ifinfo
@math{log(10.0) = 1.0}
@end ifinfo
@item nearbyint(x)
@dfn{Round inexactly}
Nearest integer to @var{x} is returned in floating-point format.
@cindex inexact conversion
No exceptions are raised for @dfn{inexact conversions}.
Example: 
@tex
nearbyint$(0.1) = 0.0$
@end tex
@ifinfo
@math{nearbyint(0.1) = 0.0}
@end ifinfo
@item pow(x,y)
@dfn{Power}
@cindex promotion
@cindex automatic type conversion
Value of @var{x} is raised to the power of @var{y}.
Exceptions are raised for @dfn{domain errors}.
Due to type-limitations in the @w{C language} @code{pow} function,
integer arguments are promoted (@pxref{Type Conversion}) to type
@code{NC_FLOAT} before evaluation. 
Example: 
@tex
pow$(2,3) = 8$
@end tex
@ifinfo
@math{pow(2,3) = 8}
@end ifinfo
@item rint(x)
@dfn{Round exactly}
Nearest integer to @var{x} is returned in floating-point format.
Exceptions are raised for @dfn{inexact conversions}.
Example: 
@tex
rint$(0.1) = 0.0$
@end tex
@ifinfo
@math{rint(0.1) = 0}
@end ifinfo
@item round(x)
@dfn{Round}
Nearest integer to @var{x} is returned in floating-point format.
Round halfway cases away from zero, regardless of current @acronym{IEEE}
rounding direction.  
Example: 
@tex
round$(0.5) = 1.0$
@end tex
@ifinfo
@math{round(0.5) = 1.0}
@end ifinfo
@item sin(x)
@dfn{Sine}
Sine of @var{x} where @var{x} is specified in radians.
Example: 
@tex
sin$(1.57079632679489661922) = 1.0$
@end tex
@ifinfo
@math{sin(1.57079632679489661922) = 1.0}
@end ifinfo
@item sinh(x)
@dfn{Hyperbolic sine}
Hyperbolic sine of @var{x} where @var{x} is specified in radians.
Example: 
@tex
sinh$(1.0) = 1.1752$
@end tex
@ifinfo
@math{sinh(1.0) = 1.1752}
@end ifinfo
@item sqrt(x)
@dfn{Square Root}
Square Root of @var{x},
@tex
$\sqrt{x}$.
@end tex
@ifinfo
@math{sqrt(x)}.
@end ifinfo
Example: 
@tex
sqrt$(4.0) = 2.0$
@end tex
@ifinfo
@math{sqrt(4.0) = 2.0}
@end ifinfo
@item tan(x)
@dfn{Tangent}
Tangent of @var{x} where @var{x} is specified in radians.
Example: 
@tex
tan$(0.78539816339744830961) = 1.0$
@end tex
@ifinfo
@math{tan(0.78539816339744830961) = 1.0}
@end ifinfo
@item tanh(x)
@dfn{Hyperbolic tangent}
Hyperbolic tangent of @var{x} where @var{x} is specified in radians.
Example: 
@tex
tanh$(1.0) = 0.761594155956$
@end tex
@ifinfo
@math{tanh(1.0) = 0.761594155956}
@end ifinfo
@item trunc(x)
@dfn{Truncate}
Nearest integer to @var{x} is returned in floating-point format.
Round halfway cases toward zero, regardless of current @acronym{IEEE}
rounding direction.  
Example: 
@tex
trunc$(0.5) = 0.0$
@end tex
@ifinfo
@math{trunc(0.5) = 0.0}
@end ifinfo
@end table
@noindent
The complete list of mathematical functions supported is
platform-specific.  
Functions mandated by @w{ANSI C} are @emph{guaranteed} to be present
and are indicated with an asterisk 
@c fxm No they are not, not yet
@cindex @code{ANSI C}
@cindex @code{float}
@cindex precision
@cindex quadruple-precision
@cindex single-precision
@cindex double-precision
@cindex @code{long double}
@cindex @code{NC_DOUBLE}
@footnote{
@w{ANSI C} compilers are guaranteed to support double-precision versions
of these functions.
These functions normally operate on netCDF variables of type @code{NC_DOUBLE}
without having to perform intrinsic conversions.
For example, @acronym{ANSI} compilers provide @code{sin} for the sine of C-type
@code{double} variables. 
The @acronym{ANSI} standard does not require, but many compilers provide,
an extended set of mathematical functions that apply to single
(@code{float}) and quadruple (@code{long double}) precision variables. 
Using these functions (e.g., @code{sinf} for @code{float}, 
@code{sinl} for @code{long double}), when available, is (presumably)
more efficient than casting variables to type @code{double},
performing the operation, and then re-casting.
@acronym{NCO} uses the faster intrinsic functions when they are
available, and uses the casting method when they are not.
}.
and are indicated with an asterisk. 
@cindex @code{-f}
@cindex @code{--prn_fnc_tbl}
@cindex @code{--fnc_tbl}
Use the @samp{-f} (or @samp{fnc_tbl} or @samp{prn_fnc_tbl}) switch
to print a complete list of functions supported on your platform.
@cindex Linux
@footnote{Linux supports more of these intrinsic functions than
other OSs.}

@noindent
@html
<a name="xmp_ncap"></a> <!-- http://nco.sf.net/nco.html#xmp_ncap -->
<a name="xmp_ncap2"></a> <!-- http://nco.sf.net/nco.html#xmp_ncap2 -->
@end html

@c Begin HMB documentation

@html
<a name="ncap_opts"></a> <!-- http://nco.sf.net/nco.html#ncap_opts -->
@end html
@node Operator precedence and associativity , ID Quoting, Intrinsic mathematical methods, ncap2 netCDF Arithmetic Processor
@subsection Operator precedence and associativity
This page lists the @command{ncap2} operators in order of precedence (highest to lowest). Their associativity indicates in what order operators of equal precedence in an expression are applied.

@multitable @columnfractions .18 .62 .20
@headitem Operator @tab Description @tab Associativity
@item @code{++ --} @tab Postfix Increment/Decrement @tab Right to Left
@item @code{()} @tab Parentheses (function call)
@item @code{.}	@tab Method call
@item
@item @code{++ --} @tab Prefix Increment/Decrement @tab Right to Left
@item @code{+ -} @tab  Unary  Plus/Minus
@item @code{!} @tab Logical Not
@item
@item @code{^} @tab Power of Operator @tab Right to Left
@item
@item @code{* / %} @tab Multiply/Divide/Modulus @tab Left To Right
@item
@item @code{+ -} @tab Addition/Subtraction @tab Left To Right
@item
@item @code{>> <<} @tab Fortran style array clipping @tab Left to Right
@item
@item
@item @code{< <=} @tab Less than/Less than or equal to @tab Left to Right
@item @code{> >=} @tab Greater than/Greater than or equal to
@item
@item @code{== !=} @tab Equal to/Not equal to @tab Left to Right
@item 
@item @code{&&}	@tab Logical AND @tab Left to Right
@item
@item @code{||}	@tab Logical OR @tab Left to Right
@item
@item @code{?:} @tab Ternary Operator @tab Right to Left
@item
@item @code{=}	@tab Assignment	@tab Right to Left
@item @code{+= -=} @tab	Addition/subtraction assignment	
@item @code{*= /=} @tab	Multiplication/division assignment
@end multitable

@html
<a name="ncap_nmc"></a> <!-- http://nco.sf.net/nco.html#ncap_nmc -->
@end html
@node ID Quoting, make_bounds() function, Operator precedence and associativity , ncap2 netCDF Arithmetic Processor
@subsection ID Quoting
@cindex ID Quoting
In this section a name refers to a variable, attribute, or dimension name.
The allowed characters in a valid netCDF name vary from release to
release. (See end section).
To use metacharacters in a name, or to use a method name as a variable
name, the name must be quoted wherever it occurs. 

@noindent The default @acronym{NCO} name is specified by the regular expressions:

@example
DGT:     ('0'..'9');
LPH:     ( 'a'..'z' | 'A'..'Z' | '_' );
name:    (LPH)(LPH|DGT)+
@end example

@noindent The first character of a valid name must be alphabetic or the underscore.
Subsequent characters must be alphanumeric or underscore, e.g., a1, _23, hell_is_666.

@noindent The valid characters in a quoted name are specified by the regular expressions:
@example
LPHDGT:  ( 'a'..'z' | 'A'..'Z' | '_' | '0'..'9');
name:    (LPHDGT|'-'|'+'|'.'|'('|')'|':' )+  ;      
@end example

@noindent Quote a variable:@*
'avg' , '10_+10','set_miss' '+-90field' , '--test'=10.0d@* @* 
Quote an attribute:@*
'three@@10', 'set_mss@@+10', '666@@hell', 't1@@+units'="kelvin" @* @*
Quote a dimension: @*
'$10', '$t1--', '$--odd', c1['$10','$t1--']=23.0d @* 

@sp 1
The following comments are from the netCDF library definitions and
detail the naming conventions for each release. 
@noindent netcdf-3.5.1 @*
netcdf-3.6.0-p1 @*
netcdf-3.6.1 @*
netcdf-3.6.2 @*
@example
@verbatim
/*
 * ( [a-zA-Z]|[0-9]|'_'|'-'|'+'|'.'|'|':'|'@'|'('|')' )+
 * Verify that name string is valid CDL syntax, i.e., all characters are
 * alphanumeric, '-', '_', '+', or '.'.
 * Also permit ':', '@', '(', or ')' in names for chemists currently making 
 * use of these characters, but don't document until ncgen and ncdump can 
 * also handle these characters in names.
 */
@end verbatim
@end example

@noindent netcdf-3.6.3@*
netcdf-4.0 Final  2008/08/28@*
@example
@verbatim
/*
 * Verify that a name string is valid syntax.  The allowed name
 * syntax (in RE form) is:
 *
 * ([a-zA-Z_]|{UTF8})([^\x00-\x1F\x7F/]|{UTF8})*
 *
 * where UTF8 represents a multibyte UTF-8 encoding.  Also, no
 * trailing spaces are permitted in names.  This definition
 * must be consistent with the one in ncgen.l.  We do not allow '/'
 * because HDF5 does not permit slashes in names as slash is used as a
 * group separator.  If UTF-8 is supported, then a multi-byte UTF-8
 * character can occur anywhere within an identifier.  We later
 * normalize UTF-8 strings to NFC to facilitate matching and queries.
 */ 
@end verbatim
@end example
@c End HMB documentation

@html
<a name="make_bounds"></a> <!-- http://nco.sf.net/nco.html#make_bounds -->
@end html
@node make_bounds() function, solar_zenith_angle function, ID Quoting, ncap2 netCDF Arithmetic Processor
@subsection make_bounds() function
@cindex make_bounds() function
The @command{ncap2} custom function 'make_bounds()' takes any monotonic 1D coordinate variable with regular or irregular (e.g., Gaussian) spacing and creates a bounds variable.

@emph{<bounds_var_out>=make_bounds( <coordinate_var_in>, <dim in>, <string>)}

@table @var
@item 1st Argument
The name of the input coordinate variable.
@item 2nd Argument
The second dimension of the output variable, referenced as a dimension
(i.e., the name preceded by a dollarsign) not as a string name.
The size of this dimension should always @w{be 2}.
If the dimension does not yet exist create it first using @code{defdim()}. 
@item 3rd Argument
This optional string argument will be placed in the "bounds" attribute
that will be created in the input coordinate variable. 
Normally this is the name of the bounds variable:
@end table
Typical usage:
@example
defdim("nv",2);
longitude_bounds=make_bounds(longitude,$nv,"longitude_bounds");
@end example

Another common CF convention:
@example
defdim("nv",2);
climatology_bounds=make_bounds(time,$nv,"climatology_bounds");
@end example

@node solar_zenith_angle function,  , make_bounds() function, ncap2 netCDF Arithmetic Processor
@subsection solar_zenith_angle function
@cindex solar_zenith_angle function

@emph{<zenith_out>=solar_zenith_angle( <time_in>, <latitude in>)}  

This function takes two arguments, mean local solar time and latitude.
Calculation and output is done with type @code{NC_DOUBLE}.
The calendar attribute for <time_in> in is NOT read and is assumed to be
Gregorian (this is the calendar that UDUnits uses).
As part of the calculation <time_in> is converted to days since start of
year.
For some  input units e.g., seconds, this function may produce
gobbledygook.
The output <zenith_out> is in @code{degrees}. 
For more details of the algorithm used please examine the function
@code{solar_geometry()} in @code{fmc_all_cls.cc}.
Note that this routine does not account for the equation of time,
and so can be in error by the angular equivalent of up to about fifteen
minutes time depending on the day of year.

@example
my_time[time]=@{10.50, 11.0, 11.50, 12.0, 12.5, 13.0, 13.5, 14.0, 14.50, 15.00@};  
my_time@@units="hours since 2017-06-21";

// Assume we are at Equator
latitude=0.0;

// 32.05428, 27.61159, 24.55934, 23.45467, 24.55947, 27.61184, 32.05458, 37.39353, 43.29914, 49.55782 ;
zenith=solar_zenith_angle(my_time,latitude);
@end example

@page
@html
<a name="ncatted"></a> <!-- http://nco.sf.net/nco.html#ncatted -->
@end html
@node ncatted netCDF Attribute Editor, ncbo netCDF Binary Operator, ncap2 netCDF Arithmetic Processor, Reference Manual
@section @command{ncatted} netCDF Attribute Editor
@cindex attributes
@cindex attribute names
@cindex editing attributes
@findex ncatted

@noindent
SYNTAX
@example
ncatted [-a @var{att_dsc}] [-a @dots{}] [-D @var{dbg}]
[-H] [-h] [--hdr_pad @var{nbr}] [--hpss] 
[-l @var{path}] [-O] [-o @var{output-file}] [-p @var{path}]
[-R] [-r] [--ram_all] [-t] @var{input-file} [[@var{output-file}]]
@end example
 
@noindent
DESCRIPTION

@cindex @command{ncattget}
@command{ncatted} edits attributes in a netCDF file.  
If you are editing attributes then you are spending too much time in the
world of metadata, and @command{ncatted} was written to get you back out as
quickly and painlessly as possible.
@command{ncatted} can @dfn{append}, @dfn{create}, @dfn{delete},
@dfn{modify}, and @dfn{overwrite} attributes (all explained below).  
@command{ncatted} allows each editing operation to be applied
to every variable in a file.
This saves time when changing attribute conventions throughout a file. 
@command{ncatted} is for @emph{writing} attributes.
To @emph{read} attribute values in plain text, use @command{ncks -m -M},
or define something like @command{ncattget} as a shell command
(@pxref{Filters for @command{ncks}}). 

@cindex @code{history}
@cindex @code{-h}
Because repeated use of @command{ncatted} can considerably increase the size
of the @code{history} global attribute (@pxref{History Attribute}), the
@samp{-h} switch is provided to override automatically appending the
command to the @code{history} global attribute in the @var{output-file}.

@cindex performance
@cindex operator speed
@cindex speed
@cindex execution time
According to the @cite{netCDF User Guide}, altering metadata in
netCDF files does not incur the penalty of recopying the entire file
when the new metadata occupies less space than the old metadata.
Thus @command{ncatted} may run much faster (at least on netCDF3 files)
if judicious use of header padding (@pxref{Metadata Optimization}) was
made when producing the @var{input-file}.
Similarly, using the @samp{--hdr_pad} option with @command{ncatted}
helps ensure that future metadata changes to @var{output-file} occur
as swiftly as possible.

@cindex missing values
@cindex data, missing 
@cindex @code{_FillValue}
When @command{ncatted} is used to change the @code{_FillValue} attribute,
it changes the associated missing data self-consistently.
If the internal floating-point representation of a missing value, 
e.g., 1.0e36, differs between two machines then netCDF files produced 
on those machines will have incompatible missing values.
This allows @command{ncatted} to change the missing values in files from 
different machines to a single value so that the files may then be 
concatenated, e.g., by @command{ncrcat}, without losing information.   
@xref{Missing Values}, for more information.

To master @command{ncatted} one must understand the meaning of the
structure that describes the attribute modification, @var{att_dsc} 
specified by the required option @samp{-a} or @samp{--attribute}. 
This option is repeatable and may be used multiple time in a single
@command{ncatted} invocation to increase the efficiency of altering
multiple attributes.
Each @var{att_dsc} contains five elements.
This makes using @command{ncatted} somewhat complicated, though
powerful. 
The @var{att_dsc} fields are in the following order:@* 

@var{att_dsc} = @var{att_nm}, @var{var_nm}, @var{mode}, @var{att_type},
@var{att_val}@*

@table @var
@item att_nm
Attribute name. 
Example: @code{units}
As of @acronym{NCO} 4.5.1 (July, 2015), @command{ncatted} accepts 
regular expressions (@pxref{Subsetting Files}) for attribute names
(it has ``always'' accepted regular expressions for variable names). 
Regular expressions will select all matching attribute names. 
@item var_nm
Variable name. 
Example: @code{pressure}, @code{'^H2O'}.
@cindex extended regular expressions
@cindex regular expressions
@cindex pattern matching
@cindex wildcards
Regular expressions (@pxref{Subsetting Files}) are accepted and will 
select all matching variable (and/or group) names. 
The names @code{global} and @code{group} have special meaning.
@item mode
Edit mode abbreviation. 
Example: @code{a}. 
See below for complete listing of valid values of @var{mode}.
@item att_type
Attribute type abbreviation. 
Example: @code{c}. 
See below for complete listing of valid values of @var{att_type}.
@item att_val
Attribute value. 
Example: @code{pascal}. 
@end table
@noindent
There should be no empty space between these five consecutive
arguments. 
The description of these arguments follows in their order of
appearance. 

The value of @var{att_nm} is the name of the attribute to edit. 
The meaning of this should be clear to all @command{ncatted} users.
Both @var{att_nm} and @var{var_nm} may be specified as regular
expressions.
If @var{att_nm} is omitted (i.e., left blank) and @dfn{Delete} mode is 
selected, then all attributes associated with the specified variable
will be deleted. 

@cindex global attributes
@cindex attributes, global
The value of @var{var_nm} is the name of the variable containing the
attribute (named @var{att_nm}) that you want to edit.
There are three very important and useful exceptions to this rule.
The value of @var{var_nm} can also be used to direct @command{ncatted}
to edit global attributes, or to repeat the editing operation for every 
group or variable in a file.
@w{A value} of @var{var_nm} of @code{global} indicates that @var{att_nm}
refers to a global (i.e., root-level) attribute, rather than to a
particular variable's attribute. 
This is the method @command{ncatted} supports for editing global
attributes.
@w{A value} of @var{var_nm} of @code{group} indicates that @var{att_nm}
refers to all groups, rather than to a particular variable's or group's
attribute.  
The operation will proceed to edit group metadata for every group.
Finally, if @var{var_nm} is left blank, then @command{ncatted} 
attempts to perform the editing operation on every variable in the file.
This option may be convenient to use if you decide to change the
conventions you use for describing the data.
As of @acronym{NCO} 4.6.0 (May, 2016), @command{ncatted} accepts 
the @samp{-t} (or long-option equivalent @samp{--typ_mch} or
@samp{--type_match}) option.
This causes @command{ncatted} to perform the editing operation only on
variables that are the same type as the specified attribute.

@html
<a name="mode"></a> <!-- http://nco.sf.net/nco.html#mode -->
<a name="att_mode"></a> <!-- http://nco.sf.net/nco.html#att_mode -->
<a name="att_edit"></a> <!-- http://nco.sf.net/nco.html#att_edit -->
<a name="att_append"></a> <!-- http://nco.sf.net/nco.html#att_append -->
<a name="att_create"></a> <!-- http://nco.sf.net/nco.html#att_create -->
<a name="att_delete"></a> <!-- http://nco.sf.net/nco.html#att_delete -->
<a name="att_modify"></a> <!-- http://nco.sf.net/nco.html#att_modify -->
<a name="att_nappend"></a> <!-- http://nco.sf.net/nco.html#att_nappend -->
<a name="att_overwrite"></a> <!-- http://nco.sf.net/nco.html#att_overwrite -->
<a name="att_prepend"></a> <!-- http://nco.sf.net/nco.html#att_prepend -->
@end html
The value of @var{mode} is a single character abbreviation (@code{a},
@code{c}, @code{d}, @code{m}, @code{n}, @code{o}, or @code{p})
standing for one of seven editing modes:@*
@cindex attribute, edit
@cindex attribute, append
@cindex attribute, create
@cindex attribute, delete
@cindex attribute, modify
@cindex attribute, nappend
@cindex attribute, overwrite
@cindex attribute, prepend
@table @code
@item a 
@dfn{Append}.
Append value @var{att_val} to current @var{var_nm} attribute
@var{att_nm} value @var{att_val}, if any.  
If @var{var_nm} does not already have an existing attribute
@var{att_nm}, it is created with the value @var{att_val}.
@item c
@dfn{Create}.
Create variable @var{var_nm} attribute @var{att_nm} with @var{att_val}
if @var{att_nm} does not yet exist.  
If @var{var_nm} already has an attribute @var{att_nm}, there is no
effect, so the existing attribute is preserved without change.
@item d
@dfn{Delete}.
Delete current @var{var_nm} attribute @var{att_nm}.
If @var{var_nm} does not have an attribute @var{att_nm}, there is no
effect. 
If @var{att_nm} is omitted (left blank), then all attributes associated
with the specified variable are automatically deleted. 
When @dfn{Delete} mode is selected, the @var{att_type} and @var{att_val}
arguments are superfluous and may be left blank.
@item m
@dfn{Modify}.
Change value of current @var{var_nm} attribute @var{att_nm} to value
@var{att_val}. 
If @var{var_nm} does not have an attribute @var{att_nm}, there is no
effect. 
@item n 
@dfn{Nappend}.
Append value @var{att_val} to @var{var_nm} attribute @var{att_nm} value 
@var{att_val} if @var{att_nm} already exists.
If @var{var_nm} does not have an attribute @var{att_nm}, there is no
effect. 
In other words, if @var{att_nm} already exists, Nappend behaves like
Append otherwise it does nothing.
The mnenomic is ``non-create append''.
Nappend mode was added to @command{ncatted} in version 4.6.0 (May,
2016). 
@item o
@dfn{Overwrite}.
Write attribute @var{att_nm} with value @var{att_val} to variable
@var{var_nm}, overwriting existing attribute @var{att_nm}, if any. 
This is the default mode.
@item p 
@dfn{Prepend}.
Prepend value @var{att_val} to @var{var_nm} attribute @var{att_nm} value 
@var{att_val} if @var{att_nm} already exists.
If @var{var_nm} does not have an attribute @var{att_nm}, there is no
effect. 
Prepend mode was added to @command{ncatted} in version 5.0.5 (January, 
2022). 
@end table

@html
<a name="att_typ"></a> <!-- http://nco.sf.net/nco.html#att_typ -->
@end html
The value of @var{att_type} is a single character abbreviation 
(@code{f}, @code{d}, @code{l}, @code{i}, @code{s}, @code{c}, 
@code{b}, @code{u}) or a short string standing for one of the twelve
primitive netCDF data types:@*  
@table @code
@item f
@dfn{Float}.
Value(s) specified in @var{att_val} will be stored as netCDF intrinsic
type @code{NC_FLOAT}. 
@item d
@dfn{Double}.
Value(s) specified in @var{att_val} will be stored as netCDF intrinsic
type @code{NC_DOUBLE}.
@item i, l
@dfn{Integer} or (its now deprecated synonym) @dfn{Long}.
Value(s) specified in @var{att_val} will be stored as netCDF intrinsic
type @code{NC_INT}.
@item s
@dfn{Short}.
Value(s) specified in @var{att_val} will be stored as netCDF intrinsic
type @code{NC_SHORT}.
@item c
@dfn{Char}.
Value(s) specified in @var{att_val} will be stored as netCDF intrinsic
type @code{NC_CHAR}.
@item b
@dfn{Byte}.
Value(s) specified in @var{att_val} will be stored as netCDF intrinsic
type @code{NC_BYTE}.
@item ub
@dfn{Unsigned Byte}.
Value(s) specified in @var{att_val} will be stored as netCDF intrinsic
type @code{NC_UBYTE}.
@item us
@dfn{Unsigned Short}.
Value(s) specified in @var{att_val} will be stored as netCDF intrinsic
type @code{NC_USHORT}.
@item u, ui, ul
@dfn{Unsigned Int}.
Value(s) specified in @var{att_val} will be stored as netCDF intrinsic
type @code{NC_UINT}.
@item ll, int64
@dfn{Int64}.
Value(s) specified in @var{att_val} will be stored as netCDF intrinsic
type @code{NC_INT64}.
@item ull, uint64
@dfn{Uint64}.
Value(s) specified in @var{att_val} will be stored as netCDF intrinsic
type @code{NC_UINT64}.
@item sng, string
@dfn{String}.
Value(s) specified in @var{att_val} will be stored as netCDF intrinsic
type @code{NC_STRING}.
Note that @command{ncatted} handles type @code{NC_STRING} attributes
correctly beginning with version 4.3.3 released in July, 2013. 
Earlier versions fail when asked to handle @code{NC_STRING} attributes.  
@end table
@noindent
In @dfn{Delete} mode the specification of @var{att_type} is optional
(and is ignored if supplied).

The value of @var{att_val} is what you want to change attribute
@var{att_nm} to contain.
The specification of @var{att_val} is optional in @dfn{Delete} (and is
ignored) mode. 
Attribute values for all types besides @code{NC_CHAR} must have an
attribute length of at least one.
Thus @var{att_val} may be a single value or one-dimensional array of
elements of type @code{att_type}.
If the @var{att_val} is not set or is set to empty space,
and the @var{att_type} is @code{NC_CHAR}, e.g., @code{-a units,T,o,c,""}
or @code{-a units,T,o,c,}, then the corresponding attribute is set to 
have zero length.
When specifying an array of values, it is safest to enclose
@var{att_val} in single or double quotes, e.g., 
@code{-a levels,T,o,s,"1,2,3,4"} or   
@code{-a levels,T,o,s,'1,2,3,4'}.
The quotes are strictly unnecessary around @var{att_val} except 
when @var{att_val} contains characters which would confuse the calling
shell, such as spaces, commas, and wildcard characters. 

@cindex Perl
@cindex @acronym{ASCII}
@acronym{NCO} processing of @code{NC_CHAR} attributes is a bit like Perl in
that it attempts to do what you want by default (but this sometimes
causes unexpected results if you want unusual data storage).
@cindex @code{printf()}
@cindex @code{\n} (@acronym{ASCII} LF, linefeed)
@cindex characters, special
@cindex @code{\t} (@acronym{ASCII} HT, horizontal tab)
If the @var{att_type} is @code{NC_CHAR} then the argument is interpreted as a
string and it may contain C-language escape sequences, e.g., @code{\n},
which @acronym{NCO} will interpret before writing anything to disk.
@acronym{NCO} translates valid escape sequences and stores the
appropriate @acronym{ASCII} code instead.
Since two byte escape sequences, e.g., @code{\n}, represent one-byte
@acronym{ASCII} codes, e.g., @acronym{ASCII} 10 (decimal), the stored
string attribute is one byte shorter than the input string length for
each embedded escape sequence. 
The most frequently used C-language escape sequences are @code{\n} (for
linefeed) and @code{\t} (for horizontal tab).
These sequences in particular allow convenient editing of formatted text
attributes. 
@cindex @code{\a} (@acronym{ASCII} BEL, bell)
@cindex @code{\b} (@acronym{ASCII} BS, backspace)
@cindex @code{\f} (@acronym{ASCII} FF, formfeed)
@cindex @code{\r} (@acronym{ASCII} CR, carriage return)
@cindex @code{\v} (@acronym{ASCII} VT, vertical tab)
@cindex @code{\\} (@acronym{ASCII} \, backslash)
The other valid @acronym{ASCII} codes are @code{\a}, @code{\b}, @code{\f},
@code{\r}, @code{\v}, and @code{\\}. 
@xref{ncks netCDF Kitchen Sink}, for more examples of string formatting
(with the @command{ncks} @samp{-s} option) with special characters. 

@cindex @code{\'} (protected end quote)
@cindex @code{\"} (protected double quote)
@cindex @code{\?} (protected question mark)
@cindex @code{\\} (protected backslash)
@cindex @code{'} (end quote)
@cindex @code{"} (double quote)
@cindex @code{?} (question mark)
@cindex @code{\} (backslash)
@cindex special characters
@cindex @acronym{ASCII}
Analogous to @code{printf}, other special characters are also allowed by 
@command{ncatted} if they are ``protected'' by a backslash.
The characters @code{"}, @code{'}, @code{?}, and @code{\} may be 
input to the shell as @code{\"}, @code{\'}, @code{\?}, and @code{\\}.
@acronym{NCO} simply strips away the leading backslash from these
characters before editing the attribute.
No other characters require protection by a backslash.
Backslashes which precede any other character (e.g., @code{3}, @code{m},
@code{$}, @code{|}, @code{&}, @code{@@}, @code{%}, @code{@{}, and
@code{@}}) will not be filtered and will be included in the attribute.

@cindex strings
@cindex NUL-termination
@cindex NUL
@cindex C language
@cindex @code{0} (NUL)
Note that the NUL character @code{\0} which terminates @w{C language}
strings is assumed and need not be explicitly specified.
@comment If @code{\0} is input, it will not be translated (because it would
@comment terminate the string in an additional location).
@comment 20101007 Before today, \0 was not translated to NUL
@comment 20101007 As of today,  \0 is      translated to NUL
If @code{\0} is input, it is translated to the NUL character.
However, this will make the subsequent portion of the string, if any,
invisible to @w{C standard} library string functions. 
And that may cause unintended consequences.
Because of these context-sensitive rules, one must use @command{ncatted}
with care in order to store data, rather than text strings, in an 
attribute of type @code{NC_CHAR}.

Note that @command{ncatted} interprets character attributes
(i.e., attributes of type @code{NC_CHAR}) as strings.
@noindent
@html
<a name="xmp_ncatted"></a> <!-- http://nco.sf.net/nco.html#xmp_ncatted -->
@end html
EXAMPLES

Append the string @code{Data version 2.0.\n} to the global attribute
@code{history}: 
@example
ncatted -a history,global,a,c,'Data version 2.0\n' in.nc 
@end example
Note the use of embedded @w{C language} @code{printf()}-style escape 
sequences. 

Change the value of the @code{long_name} attribute for variable @code{T}
from whatever it currently is to ``temperature'':
@example
ncatted -a long_name,T,o,c,temperature in.nc
@end example

@html
<a name="MPAS"></a> <!-- http://nco.sf.net/nco.html#MPAS -->
<a name="typ_mch"></a> <!-- http://nco.sf.net/nco.html#typ_mch -->
<a name="type_match"></a> <!-- http://nco.sf.net/nco.html#type_match -->
@end html
@cindex MPAS
@cindex @code{_FillValue}
Many model and observational datasets use missing values that are not
annotated in the standard manner.
For example, at the time (2015--2018) of this writing,
the @acronym{MPAS} ocean and ice models use
@math{-9.99999979021476795361e+33} as the missing value, yet do not
store a @code{_FillValue} attribute with any variables.
To prevent arithmetic from treating these values as normal, designate
this value as the @code{_FillValue} attribute:
@example
ncatted    -a _FillValue,,o,d,-9.99999979021476795361e+33 in.nc
ncatted -t -a _FillValue,,o,d,-9.99999979021476795361e+33 in.nc
ncatted -t -a _FillValue,,o,d,-9.99999979021476795361e+33 \
           -a _FillValue,,o,f,1.0e36 -a _FillValue,,o,i,-999 in.nc
@end example
The first example adds the attribute to all variables. 
The @samp{-t} switch causes the second example to add the attribute only
to double precision variables.
This is often more useful, and can be used to provide distinct missing
value attributes to each numeric type, as in the third example.

@html
<a name="NaNf"></a> <!-- http://nco.sf.net/nco.html#NaNf -->
<a name="NaN"></a> <!-- http://nco.sf.net/nco.html#NaN -->
<a name="nan"></a> <!-- http://nco.sf.net/nco.html#nan -->
@end html
@cindex NaN
@cindex NaNf
@cindex @acronym{IEEE} NaN, NaNf
@cindex Not-a-Number
@acronym{NCO} arithmetic operators may not work as expected on
@acronym{IEEE} NaN (short for Not-a-Number) and NaN-like numbers such
as positive infinity and negative infinity
@footnote{NaN is a special floating point value (not a string).
Arithmetic comparisons to NaN and NaN-like numbers always
return False, contrary to the behavior of all other numbers.
This behavior is difficult to intuit, yet @w{@acronym{IEEE} 754}
mandates it.  
To correctly handle NaNs during arithmetic, code must use special 
math library macros (e.g., @command{isnormal()}) to determine whether 
any operand requires special treatment. 
If so, additional logic must be added to correctly perform the
arithmetic. 
This is in addition to the normal handling incurred to correctly
handle missing values. 
Handling field and missing values (either or both of which may be NaN) 
in binary operators thus incurs four-to-eight extra code paths.
Each code path slows down arithmetic relative to normal numbers.
This makes supporting NaN arithmetic costly and inefficient.
Hence @acronym{NCO} supports NaN only to the extent necessary to
replace it with a normal number.
Although using NaN for the missing value (or any value) in datasets is 
legal in netCDF, we discourage it.
We recommend avoiding NaN entirely.}. 
One way to work-around this problem is to change @acronym{IEEE} NaNs
to normal missing values. 
As of @acronym{NCO} 4.1.0 (March, 2012), @command{ncatted} works with 
NaNs (though none of @acronym{NCO}'s arithmetic operators do).
This limited support enables users to change NaN to a normal number
before performing arithmetic or propagating a NaN-tainted dataset.
First set the missing value (i.e., the value of the @code{_FillValue}
attribute) for the variable(s) in question to the @acronym{IEEE} NaN
value.  
@example
ncatted -a _FillValue,,o,f,NaN in.nc
@end example
Then change the missing value from the @acronym{IEEE} NaN value to a
normal @acronym{IEEE} number, like 1.0e36 (or to whatever the original
missing value was). 
@example
ncatted -a _FillValue,,m,f,1.0e36 in.nc
@end example
Some @acronym{NASA} @acronym{MODIS} datasets provide a real-world
example. 
@example
ncatted -O -a _FillValue,,m,d,1.0e36 -a missing_value,,m,d,1.0e36 \
        MODIS_L2N_20140304T1120.nc MODIS_L2N_20140304T1120_noNaN.nc
@end example

Delete all existing @code{units} attributes:
@example
ncatted -a units,,d,, in.nc
@end example
@noindent
The value of @var{var_nm} was left blank in order to select all
variables in the file.
The values of @var{att_type} and @var{att_val} were left blank because
they are superfluous in @dfn{Delete} mode. 

@cindex @code{global} attribute
@cindex global attributes
@cindex attributes, global
Delete all attributes associated with the @code{tpt} variable, and
delete all global attributes
@example
ncatted -a ,tpt,d,, -a ,global,d,, in.nc
@end example
@noindent
The value of @var{att_nm} was left blank in order to select all
attributes associated with the variable.
To delete all global attributes, simply replace @code{tpt} with
@code{global} in the above.

@cindex @code{units}
Modify all existing @code{units} attributes to @code{meter second-1}:
@example
ncatted -a units,,m,c,'meter second-1' in.nc
@end example

@cindex @code{units}
Add a @code{units} attribute of @code{kilogram kilogram-1} to all
variables whose first three characters are @samp{H2O}:
@example
ncatted -a units,'^H2O',c,c,'kilogram kilogram-1' in.nc
@end example

Overwrite the @code{quanta} attribute of variable
@code{energy} to an array of four integers. 
@example
ncatted -a quanta,energy,o,s,'010,101,111,121' in.nc
@end example

@cindex extended regular expressions
@cindex regular expressions
@cindex pattern matching
@cindex wildcards
As of @acronym{NCO} 3.9.6 (January, 2009), @command{ncatted} accepts
@dfn{extended regular expressions} as arguments for variable names,
and, since @acronym{NCO} 4.5.1 (July, 2015), for attribute names.
@example
ncatted -a isotope,'^H2O*',c,s,'18' in.nc
ncatted -a '.?_iso19115$','^H2O*',d,, in.nc
@end example
The first example creates @code{isotope} attributes for all variables
whose names contain @samp{H2O}.
The second deletes all attributes whose names end in @code{_iso19115}
from all variables whose names contain @samp{H2O}.
See @ref{Subsetting Files} for more details on using regular
expressions. 

@cindex groups
As of @acronym{NCO} 4.3.8 (November, 2013), @command{ncatted} 
accepts full and partial group paths in names of attributes,
variables, dimensions, and groups.
@c ncks -m -v lon ~/in_grp.nc
@example
# Overwrite units attribute of specific 'lon' variable
ncatted -O -a units,/g1/lon,o,c,'degrees_west' in_grp.nc
# Overwrite units attribute of all 'lon' variables
ncatted -O -a units,lon,o,c,'degrees_west' in_grp.nc
# Delete units attribute of all 'lon' variables
ncatted -O -a units,lon,d,, in_grp.nc
# Overwrite units attribute with new type for specific 'lon' variable
ncatted -O -a units,/g1/lon,o,sng,'degrees_west' in_grp.nc
# Add new_att attribute to all variables
ncatted -O -a new_att,,c,sng,'new variable attribute' in_grp.nc
# Add new_grp_att group attribute to all groups
ncatted -O -a new_grp_att,group,c,sng,'new group attribute' in_grp.nc
# Add new_grp_att group attribute to single group
ncatted -O -a g1_grp_att,g1,c,sng,'new group attribute' in_grp.nc
# Add new_glb_att global attribute to root group
ncatted -O -a new_glb_att,global,c,sng,'new global attribute' in_grp.nc
@end example
Note that regular expressions work well in conjuction with group
path support. 
In other words, the variable name (including group path component) and
the attribute names may both be extended regular expressions.

Demonstrate input of C-language escape sequences (e.g., @code{\n}) and
other special characters (e.g., @code{\"}) 
@example
ncatted -h -a special,global,o,c,
'\nDouble quote: \"\nTwo consecutive double quotes: \"\"\n
Single quote: Beyond my shell abilities!\nBackslash: \\\n
Two consecutive backslashes: \\\\\nQuestion mark: \?\n' in.nc
@end example
Note that the entire attribute is protected from the shell by single
quotes. 
These outer single quotes are necessary for interactive use, but may be
omitted in batch scripts.

@cindex standard input
@cindex pipes
@cindex shell
@cindex @command{xargs}
Although @command{ncatted} accepts multiple @samp{-a @var{att_dst}}
options simultaneously, modifying lengthy commands can become
unwieldy.  
To preserve simplicity in storing/modifying multiple attribute edits, 
consider storing the options separately in a text file and
assembling them at run-time to generate and submit the correct
command.
One such method uses the @command{xargs} command to intermediate
between an on-disk list attributes to change and the @command{ncatted}
command.
For example, use an intermediate file named @file{options.txt}
to store one option per line thusly
@example
@verbatim
cat > opt.txt << EOF
-a institution,global,m,c,\"Super Cool University\"
-a source,global,c,c,\"My Awesome Radar\"
-a contributors,global,c,c,\"Enrico Fermi, Galileo Galilei, Leonardo Da Vinci\"
...
EOF
@end verbatim
@end example
The backslashes preserve the whitespace in the individual attributes
for correct parsing by the shell.
Simply substituting the expansion of this file through @command{xargs}
directly on the command line fails to work (why?).
However, a simple workaround is to use @command{xargs} to construct
the command string, and execute that string with @command{eval}:
@example
@verbatim
opt=$(cat opt.txt | xargs)
cmd="ncatted -O ${opt} in.nc out.nc"
eval $cmd
@end verbatim
@end example
This procedure can by modified to employ more complex option
pre-processing using other tools such as Awk, Perl, or Python.

@page
@html
<a name="ncbo"></a> <!-- http://nco.sf.net/nco.html#ncbo -->
<a name="ncdiff"></a> <!-- http://nco.sf.net/nco.html#ncdiff -->
<a name="ncadd"></a> <!-- http://nco.sf.net/nco.html#ncadd -->
<a name="ncsub"></a> <!-- http://nco.sf.net/nco.html#ncsub -->
<a name="ncsubtract"></a> <!-- http://nco.sf.net/nco.html#ncsubtract -->
<a name="ncmult"></a> <!-- http://nco.sf.net/nco.html#ncmult -->
<a name="ncmultiply"></a> <!-- http://nco.sf.net/nco.html#ncmultiply -->
<a name="ncdivide"></a> <!-- http://nco.sf.net/nco.html#ncdivide -->
@end html
@node ncbo netCDF Binary Operator, ncclimo netCDF Climatology Generator, ncatted netCDF Attribute Editor, Reference Manual
@section @command{ncbo} netCDF Binary Operator
@findex ncbo
@findex ncdiff
@findex ncadd
@findex ncsub
@findex ncsubtract
@findex ncmult
@findex ncmultiply
@findex ncdivide
@cindex binary operations
@cindex addition
@cindex subtraction
@cindex multiplication
@cindex adding data
@cindex subtracting data
@cindex multiplying data
@cindex dividing data

@noindent
SYNTAX
@example 
ncbo [-3] [-4] [-5] [-6] [-7] [-A] [-C] [-c] [--cmp @var{cmp_sng}]
[--cnk_byt @var{sz_byt}] [--cnk_csh @var{sz_byt}] [--cnk_dmn @var{nm},@var{sz_lmn}]
[--cnk_map @var{map}] [--cnk_min @var{sz_byt}] [--cnk_plc @var{plc}] [--cnk_scl @var{sz_lmn}]
[-D @var{dbg}] [-d @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]] [-F] [--fl_fmt @var{fl_fmt}]
[-G @var{gpe_dsc}] [-g @var{grp}[,@dots{}]] [--glb ...] [-H] [-h] [--hdr_pad @var{nbr}] [--hpss]
[-L @var{dfl_lvl}] [-l @var{path}] [--no_cll_msr] [--no_frm_trm] [--no_tmp_fl] 
[-O] [-o @var{file_3}] [-p @var{path}] [-R] [-r] [--ram_all]
[-t @var{thr_nbr}] [--unn] [-v @var{var}[,@dots{}]] [-X ...] [-x] [-y @var{op_typ}]
@var{file_1} @var{file_2} [@var{file_3}]
@end example

@noindent
DESCRIPTION

@command{ncbo} performs binary operations on variables in @var{file_1}
and the corresponding variables (those with the same name) in
@var{file_2} and stores the results in @var{file_3}. 
The binary operation operates on the entire files (modulo any excluded
variables). 
@xref{Missing Values}, for treatment of missing values.
One of the four standard arithmetic binary operations currently
supported must be selected with the @samp{-y @var{op_typ}} switch (or
long options @samp{--op_typ} or @samp{--operation}).
@cindex @code{add}
@cindex @code{subtract}
@cindex @code{multiply}
@cindex @code{divide}
@cindex @code{+}
@cindex @code{-}
@cindex @code{*}
@cindex @code{/}
@cindex @code{-y @var{op_typ}}
@cindex @code{--operation @var{op_typ}}
@cindex @code{--op_typ @var{op_typ}}
@cindex alternate invocations
The valid binary operations for @command{ncbo}, their definitions, 
corresponding values of the @var{op_typ} key, and alternate invocations
are:  
@table @dfn
@item Addition
@c Internal operation code: @{nco_op_add}@*
Definition: @var{file_3} = @var{file_1} + @var{file_2}@*
Alternate invocation: @command{ncadd}@*
@var{op_typ} key values: @samp{add}, @samp{+}, @samp{addition}@*
Examples: @samp{ncbo --op_typ=add 1.nc 2.nc 3.nc}, @samp{ncadd 1.nc 2.nc 3.nc}@*
@item Subtraction
Definition: @var{file_3} = @var{file_1} - @var{file_2}@*
Alternate invocations: @command{ncdiff}, @command{ncsub}, @command{ncsubtract}@*
@var{op_typ} key values: @samp{sbt}, @samp{-}, @samp{dff}, @samp{diff}, @samp{sub}, @samp{subtract}, @samp{subtraction}@*
Examples: @samp{ncbo --op_typ=- 1.nc 2.nc 3.nc}, @samp{ncdiff 1.nc 2.nc 3.nc}@*
@item Multiplication
Definition: @var{file_3} = @var{file_1} * @var{file_2}@* 
Alternate invocations: @command{ncmult}, @command{ncmultiply}@* 
@var{op_typ} key values: @samp{mlt}, @samp{*}, @samp{mult}, @samp{multiply}, @samp{multiplication}@*
Examples: @samp{ncbo --op_typ=mlt 1.nc 2.nc 3.nc}, @samp{ncmult 1.nc 2.nc 3.nc}@*
@item Division
Definition: @var{file_3} = @var{file_1} / @var{file_2}@* 
Alternate invocation: @command{ncdivide}@*
@var{op_typ} key values: @samp{dvd}, @samp{/}, @samp{divide}, @samp{division}@*
Examples: @samp{ncbo --op_typ=/ 1.nc 2.nc 3.nc}, @samp{ncdivide 1.nc 2.nc 3.nc}@*
@end table
@noindent
Care should be taken when using the shortest form of key values,
i.e., @samp{+}, @samp{-}, @samp{*}, @w{and @samp{/}}.
Some of these single characters may have special meanings to the shell
@cindex naked characters
@footnote{@w{A naked} (i.e., unprotected or unquoted) @samp{*} is a
wildcard character.  
@w{A naked} @samp{-} may confuse the command line parser.
@w{A naked} @samp{+} and @samp{/} are relatively harmless.}.
@cindex Bash shell
Place these characters inside quotes to keep them from being interpreted 
(globbed) by the shell
@footnote{The widely used shell Bash correctly interprets all these
special characters even when they are not quoted. 
That is, Bash does not prevent @acronym{NCO} from correctly interpreting 
the intended arithmetic operation when the following arguments are given
(without quotes) to @command{ncbo}:
@samp{--op_typ=+}, @samp{--op_typ=-}, @samp{--op_typ=*},
and @samp{--op_typ=/}}. 
@cindex globbing
@cindex shell
@cindex quotes
For example, the following commands are equivalent
@example
ncbo --op_typ=* 1.nc 2.nc 3.nc # Dangerous (shell may try to glob)
ncbo --op_typ='*' 1.nc 2.nc 3.nc # Safe ('*' protected from shell)
ncbo --op_typ="*" 1.nc 2.nc 3.nc # Safe ('*' protected from shell)
ncbo --op_typ=mlt 1.nc 2.nc 3.nc
ncbo --op_typ=mult 1.nc 2.nc 3.nc
ncbo --op_typ=multiply 1.nc 2.nc 3.nc
ncbo --op_typ=multiplication 1.nc 2.nc 3.nc
ncmult 1.nc 2.nc 3.nc # First do 'ln -s ncbo ncmult'
ncmultiply 1.nc 2.nc 3.nc # First do 'ln -s ncbo ncmultiply'
@end example
No particular argument or invocation form is preferred.
Users are encouraged to use the forms which are most intuitive to them.

@cindex @command{alias}
@cindex @command{ln -s}
@cindex symbolic links
Normally, @command{ncbo} will fail unless an operation type is specified
with @samp{-y} (equivalent to @samp{--op_typ}).
You may create exceptions to this rule to suit your particular tastes,
in conformance with your site's policy on @dfn{symbolic links} to
executables (files of a different name point to the actual executable).
For many years, @command{ncdiff} was the main binary file operator.
As a result, many users prefer to continue invoking @command{ncdiff}
rather than memorizing a new command (@samp{ncbo -y @var{sbt}}) which
behaves identically to the original @command{ncdiff} command.
However, from a software maintenance standpoint, maintaining a distinct 
executable for each binary operation (e.g., @command{ncadd}) is untenable,
and a single executable, @command{ncbo}, is desirable.
To maintain backward compatibility, therefore, @acronym{NCO}
automatically creates a symbolic link from @command{ncbo} to
@command{ncdiff}.  
Thus @command{ncdiff} is called an @dfn{alternate invocation} of
@command{ncbo}. 
@command{ncbo} supports many additional alternate invocations which must
be manually activated.
Should users or system adminitrators decide to activate them, the
procedure is simple. 
For example, to use @samp{ncadd} instead of @samp{ncbo --op_typ=add}, 
simply create a symbolic link from @command{ncbo} to @command{ncadd}
@footnote{The command to do this is @samp{ln -s -f ncbo ncadd}}.
The alternatate invocations supported for each operation type are listed
above. 
Alternatively, users may always define @samp{ncadd} as an @dfn{alias} to 
@samp{ncbo --op_typ=add}
@footnote{The command to do this is @samp{alias ncadd='ncbo --op_typ=add'}}.

It is important to maintain portability in @acronym{NCO} scripts.
Therefore we recommend that site-specfic invocations (e.g.,
@samp{ncadd}) be used only in interactive sessions from the
command-line.
For scripts, we recommend using the full invocation (e.g., 
@samp{ncbo --op_typ=add}).
This ensures portability of scripts between users and sites.

@html
<a name="brd_var"></a> <!-- http://nco.sf.net/nco.html#brd_var -->
@end html
@cindex broadcasting variables
@cindex rank
@command{ncbo} operates (e.g., adds) variables in @var{file_2} with the
corresponding variables (those with the same name) in @var{file_1} and
stores the results in @var{file_3}. 
@cindex broadcasting variables
Variables in @var{file_1} or @var{file_2} are @dfn{broadcast} to conform
to the corresponding variable in the other input file if
necessary@footnote{
Prior to @acronym{NCO} version 4.3.1 (May, 2013), @command{ncbo}
would only broadcast variables in @var{file_2} to conform to
@var{file_1}. 
Variables in @var{file_1} were @emph{never} broadcast to conform to the 
dimensions in @var{file_2}.}. 
Now @command{ncbo} is completely symmetric with respect to @var{file_1}
and @var{file_2}, i.e., 
@set flg
@tex
$\rm{file}_1 - \rm{file}_2 = -(\rm{file}_2-\rm{file}_1)$.
@clear flg
@end tex
@ifinfo
@math{@var{file_1} - @var{file_2} = - (@var{file_2} - @var{file_1}}.
@clear flg
@end ifinfo
@ifset flg
@c texi2html does not like @math{}
@var{file_1} - @var{file_2} = -(@var{file_2} - @var{file_1}.
@clear flg
@end ifset

Broadcasting a variable means creating data in non-existing dimensions
by copying data in existing dimensions.
For example, a two dimensional variable in @var{file_2} can be
subtracted from a four, three, or two (not one or zero)
dimensional variable (of the same name) in @code{file_1}. 
@cindex anomalies
This functionality allows the user to compute anomalies from the mean.
In the future, we will broadcast variables in @var{file_1}, if necessary
to conform to their counterparts in @var{file_2}.
@c TODO #268
@cindex rank
Thus, presently, the number of dimensions, or @dfn{rank}, of any
processed variable in @var{file_1} must be greater than or equal to the
rank of the same variable in @var{file_2}. 
Of course, the size of all dimensions common to both @var{file_1} and
@var{file_2} must be equal. 

When computing anomalies from the mean it is often the case that
@var{file_2} was created by applying an averaging operator to a file
with initially the same dimensions as @var{file_1} (often @var{file_1}
itself).  
In these cases, creating @var{file_2} with @command{ncra} rather than
@command{ncwa} will cause the @command{ncbo} operation to fail.
For concreteness say the record dimension in @code{file_1} is
@code{time}.  
If @var{file_2} was created by averaging @var{file_1} over the
@code{time} dimension with the @command{ncra} operator (rather than with
the @command{ncwa} operator), then @var{file_2} will have a @code{time}
dimension of @w{size 1} rather than having no @code{time} dimension at
all 
@cindex degenerate dimension
@cindex @samp{-b}
@footnote{This is because @command{ncra} collapses the record dimension
to a size @w{of 1} (making it a @dfn{degenerate} dimension), but does
not remove it, while, unless @samp{-b} is given, @command{ncwa} removes
all averaged dimensions.
In other words, by default @command{ncra} changes variable size though
not rank, while, @command{ncwa} changes both variable size and rank.}.   
In this case the input files to @command{ncbo}, @var{file_1} and
@var{file_2}, will have unequally sized @code{time} dimensions which
causes @command{ncbo} to fail.
To prevent this from occurring, use @command{ncwa} to remove the
@code{time} dimension from @var{file_2}.
See the example below.

@cindex coordinate variable 
@cindex @code{NC_CHAR}
@command{ncbo} never operates on coordinate variables or variables
of type @code{NC_CHAR} or @code{NC_STRING}. 
This ensures that coordinates like (e.g., latitude and longitude) are 
physically meaningful in the output file, @var{file_3}. 
This behavior is hardcoded.
@cindex @acronym{CF} conventions
@command{ncbo} applies special rules to some 
@acronym{CF}-defined (and/or @acronym{NCAR CCSM} or @acronym{NCAR CCM} 
fields) such as @code{ORO}.
See @ref{CF Conventions} for a complete description.
Finally, we note that @command{ncflint} (@pxref{ncflint netCDF File
Interpolator}) is designed for file interpolation.
As such, it also performs file subtraction, addition, multiplication,
albeit in a more convoluted way than @command{ncbo}.

@html
<a name="grp_brd"></a> <!-- http://nco.sf.net/nco.html#grp_brd -->
<a name="brd_grp"></a> <!-- http://nco.sf.net/nco.html#brd_grp -->
<a name="gb"></a> <!-- http://nco.sf.net/nco.html#gb -->
<a name="GB"></a> <!-- http://nco.sf.net/nco.html#GB -->
@end html
@cindex broadcasting groups
Beginning with @acronym{NCO} version 4.3.1 (May, 2013), @command{ncbo} 
supports @dfn{group broadcasting}.
Group broadcasting means processing data based on group patterns in the
input file(s) and automatically transferring or transforming groups to
the output file. 
Consider the case where @var{file_1} contains multiple groups each with
the variable @var{v1}, while @var{file_2} contains @var{v1} only in its 
top-level (i.e., root) group.
Then @command{ncbo} will replicate the group structure of @var{file_1}
in the output file, @var{file_3}.
Each group in @var{file_3} contains the output of the corresponding
group in @var{file_1} operating on the data in the single group in
@var{file_2}. 
An example is provided below.

@noindent
@html
<a name="xmp_ncbo"></a> <!-- http://nco.sf.net/nco.html#xmp_ncbo -->
<a name="xmp_ncdiff"></a> <!-- http://nco.sf.net/nco.html#xmp_ncdiff -->
@end html
EXAMPLES

Say files @file{85_0112.nc} and @file{86_0112.nc} each contain 12 months
of data.
Compute the change in the monthly averages from 1985 to 1986:
@example
ncbo   86_0112.nc 85_0112.nc 86m85_0112.nc
ncdiff 86_0112.nc 85_0112.nc 86m85_0112.nc
ncbo --op_typ=sub 86_0112.nc 85_0112.nc 86m85_0112.nc
ncbo --op_typ='-' 86_0112.nc 85_0112.nc 86m85_0112.nc
@end example
@noindent
These commands are all different ways of expressing the same thing.

@cindex broadcasting
@cindex rank
The following examples demonstrate the broadcasting feature of
@command{ncbo}.  
Say we wish to compute the monthly anomalies of @code{T} from the yearly
average of @code{T} for the year 1985.
First we create the 1985 average from the monthly data, which is stored
with the record dimension @code{time}.
@example
ncra 85_0112.nc 85.nc
ncwa -O -a time 85.nc 85.nc
@end example
@noindent
The second command, @command{ncwa}, gets rid of the @code{time} dimension
of @w{size 1} that @command{ncra} left in @file{85.nc}. 
Now none of the variables in @file{85.nc} has a @code{time} dimension.
@w{A quicker} way to accomplish this is to use @command{ncwa} from the
beginning:  
@example
ncwa -a time 85_0112.nc 85.nc
@end example
@noindent
We are now ready to use @command{ncbo} to compute the anomalies for 1985:
@example
ncdiff -v T 85_0112.nc 85.nc t_anm_85_0112.nc
@end example
@noindent
Each of the 12 records in @file{t_anm_85_0112.nc} now contains the
monthly deviation of @code{T} from the annual mean of @code{T} for each 
gridpoint. 

Say we wish to compute the monthly gridpoint anomalies from the zonal
annual mean. 
@w{A @dfn{zonal mean}} is a quantity that has been averaged over the
longitudinal (or @var{x}) direction.
First we use @command{ncwa} to average over longitudinal direction
@code{lon}, creating @file{85_x.nc}, the zonal mean of @file{85.nc}. 
Then we use @command{ncbo} to subtract the zonal annual means from the
monthly gridpoint data:
@example
ncwa -a lon 85.nc 85_x.nc
ncdiff 85_0112.nc 85_x.nc tx_anm_85_0112.nc
@end example
@noindent
This examples works assuming @file{85_0112.nc} has dimensions
@code{time} and @code{lon}, and that @file{85_x.nc} has no @code{time}
or @code{lon} dimension.

@cindex broadcasting groups
Group broadcasting simplifies evaluation of multiple models against
observations.
Consider the input file @file{cmip5.nc} which contains multiple
top-level groups @code{cesm}, @code{ecmwf}, and @code{giss}, each of
which contains the surface air temperature field @code{tas}.
We wish to compare these models to observations stored in @file{obs.nc} 
which contains @code{tas} only in its top-level (i.e., root) group.
It is often the case that many models and/or model simulations exist,
whereas only one observational dataset does.
We evaluate the models and obtain the bias (difference) between models
and observations by subtracting @file{obs.nc} from @file{cmip5.nc}.
Then @command{ncbo} ``broadcasts'' (i.e., replicates) the observational
data to match the group structure of @file{cmip5.nc}, subtracts,
and then stores the results in the output file, @file{bias.nc}
which has the same group structure as @file{cmip5.nc}.
@example
% ncbo -O cmip5.nc obs.nc bias.nc
% ncks -H -v tas -d time,3 bias.nc
/cesm/tas
time[3] tas[3]=-1 
/ecmwf/tas
time[3] tas[3]=0 
/giss/tas
time[3] tas[3]=1 
@end example
@noindent

@html
<a name="csn_anm"></a> <!-- http://nco.sf.net/nco.html#csn_anm -->
@end html
As a final example, say we have five years of monthly data (i.e., 
@w{60 months}) stored in @file{8501_8912.nc} and we wish to create a
file which contains the twelve month seasonal cycle of the average
monthly anomaly from the five-year mean of this data. 
The following method is just one permutation of many which will
accomplish the same result.
First use @command{ncwa} to create the five-year mean: 
@example
ncwa -a time 8501_8912.nc 8589.nc
@end example
@noindent
Next use @command{ncbo} to create a file containing the difference of
each month's data from the five-year mean:
@example
ncbo 8501_8912.nc 8589.nc t_anm_8501_8912.nc
@end example
@noindent
Now use @command{ncks} to group together the five January anomalies in
one file, and use @command{ncra} to create the average anomaly for all
five Januarys. 
These commands are embedded in a shell loop so they are repeated for all
twelve months:
@cindex Bash Shell
@cindex Bourne Shell
@cindex C Shell
@example
@verbatim
for idx in {1..12}; do # Bash Shell (version 3.0+) 
  idx=`printf "%02d" ${idx}` # Zero-pad to preserve order
  ncks -F -d time,${idx},,12 t_anm_8501_8912.nc foo.${idx}
  ncra foo.${idx} t_anm_8589_${idx}.nc
done
for idx in 01 02 03 04 05 06 07 08 09 10 11 12; do # Bourne Shell
  ncks -F -d time,${idx},,12 t_anm_8501_8912.nc foo.${idx}
  ncra foo.${idx} t_anm_8589_${idx}.nc
done
foreach idx (01 02 03 04 05 06 07 08 09 10 11 12) # C Shell
  ncks -F -d time,${idx},,12 t_anm_8501_8912.nc foo.${idx}
  ncra foo.${idx} t_anm_8589_${idx}.nc
end
@end verbatim
@end example
@noindent
Note that @command{ncra} understands the @code{stride} argument so the
two commands inside the loop may be combined into the single command 
@example
@verbatim
ncra -F -d time,${idx},,12 t_anm_8501_8912.nc foo.${idx}
@end verbatim
@end example
@noindent
Finally, use @command{ncrcat} to concatenate the @w{12 average} monthly  
anomaly files into one twelve-record file which contains the entire
seasonal cycle of the monthly anomalies:
@example
ncrcat t_anm_8589_??.nc t_anm_8589_0112.nc
@end example
@noindent

@page
@html
<a name="ncclimo"></a> <!-- http://nco.sf.net/nco.html#ncclimo -->
<a name="ncsplit"></a> <!-- http://nco.sf.net/nco.html#ncsplit -->
<a name="ncsplit"></a> <!-- http://nco.sf.net/nco.html#splitter -->
@end html
@node ncclimo netCDF Climatology Generator, ncecat netCDF Ensemble Concatenator, ncbo netCDF Binary Operator, Reference Manual
@section @command{ncclimo} netCDF Climatology Generator
@cindex climo
@cindex climatology
@findex ncclimo

@noindent
SYNTAX
@example
ncclimo [-3] [-4] [-5] [-6] [-7] 
[-a @var{wnt_md}] [-C @var{clm_md}] [-c @var{caseid}]
[-d @var{dbg_lvl}] [--d2f] [--dpf=@var{dpf}] [--dpt_fl=@var{dpt_fl}] [-E @var{yr_prv}] [-e @var{yr_end}]
[-f @var{fml_nm}] [--fl_fmt=@var{fl_fmt}] [--glb_avg] [-h @var{hst_nm}] [-i @var{drc_in}]
[-j @var{job_nbr}] [-L @var{dfl_lvl}] [-l @var{lnk_flg}]
[-m @var{mdl_nm}] [--mth_end=@var{mth_end}] [--mth_srt=@var{mth_srt}]
[-n @var{nco_opt}] [--no_cll_msr] [--no_frm_trm] [--no_ntv_tms] [--no_stg_grd] [--no_stdin] 
[-O @var{drc_rgr}] [-o @var{drc_out}] [-P @var{prc_typ}] [-p @var{par_typ}] [--ppc=@var{ppc_prc}]
[-R @var{rgr_opt}] [-r @var{rgr_map}]
[-S @var{yr_prv}] [-s @var{yr_srt}] [--seasons=@var{csn_lst}] [--sgs_frc=@var{sgs_frc}] [--split]
[-t @var{thr_nbr}] [--tpd=@var{tpd}] [--uio] [-v @var{var_lst}] [--var_xtr=@var{var_xtr}] [--version] 
[--vrt_out=@var{vrt_fl}] [--vrt_xtr=@var{vrt_xtr}]
[-X @var{drc_xtn}] [-x @var{drc_prv}] [--xcl_var]
[-Y @var{rgr_xtn}] [-y @var{rgr_prv}] [--ypf=@var{ypf_max}]
@end example

@noindent
DESCRIPTION

In climatology generation mode, @command{ncclimo} ingests ``raw'' data
consisting of interannual sets of files, each containing sub-daily
(diurnal), daily, monthly, or yearly averages, and from these
produces climatological daily, monthly, seasonal, and/or annual
means.
Alternatively, in timeseries reshaping (aka ``splitter'') mode,
@command{ncclimo} will subset and temporally split the input raw data
timeseries into per-variable files spanning the entire period. 
@command{ncclimo} can optionally (call @command{ncremap} to) regrid
all output files in either mode.
Unlike the rest of @acronym{NCO}, @command{ncclimo} and
@command{ncremap} are shell scripts, not compiled binaries@footnote{ 
This means that newer (including user-modified) versions of
@command{ncclimo} work fine without re-compiling @acronym{NCO}.
Re-compiling is only necessary to take advantage of new features or
fixes in the @acronym{NCO} binaries, not to improve @command{ncclimo}.
One may download and give executable permissions to the latest source 
at @url{https://github.com/nco/nco/tree/master/data/ncclimo} without
re-installing the rest of @acronym{NCO}.}. 
@html
<a name="HDF5_USE_FILE_LOCKING"></a> <!-- http://nco.sf.net/nco.html#HDF5_USE_FILE_LOCKING -->
@end html
@cindex @code{HDF5_USE_FILE_LOCKING}
As of @acronym{NCO} 4.9.2 (February, 2020), the @command{ncclimo}
and @command{ncremap} scripts export the environment variable
@code{HDF5_USE_FILE_LOCKING} with a value of @code{FALSE}.
This prevents failures of these operators that can occur with some
versions of the underlying HDF library that attempt to lock files
on file systems that cannot, or do not, support it.

There are five (usually) required options (@samp{-c}, @samp{-s},
@samp{-e}, @samp{-i}, and @samp{-o})) to generate climatologies, and
many more options are available to customize the processing.
Options are similar to @command{ncremap} options.
Standard @command{ncclimo} usage for climatology generation looks like
@example
ncclimo            -c caseid -s srt_yr -e end_yr -i drc_in -o drc_out
ncclimo -m mdl_nm  -c caseid -s srt_yr -e end_yr -i drc_in -o drc_out
ncclimo -v var_lst -c caseid -s srt_yr -e end_yr -i drc_in -o drc_out
ncclimo --case=caseid --start=srt_yr --end=end_yr --input=drc_in --output=drc_out
@end example
In climatology generation mode, @command{ncclimo} constructs the list
of input filenames from the arguments to the caseid, date, and
model-type options.
As of @acronym{NCO} version 4.9.4 (September, 2020), @command{ncclimo}
can produce climatologies of high-frequency input data supplied via
standard input, positional command-line options, or directory
contents, all input methods traditionally supported only in splitter
mode. 
Instead of using the @code{caseid} option to help generate the input
filenames as it does for normal (monthly) climos, @command{ncclimo}
uses the @code{caseid} option, when provided, to rename the output
files for high-frequency climos.
@example
@verbatim
# Generate diurnal climos from high-frequency CMIP6 timeseries
cd ${drc_in};ls ${caseid}*.h4.nc | ncclimo --clm_md=hfc \
  -c ${caseid} --yr_srt=2001 --yr_end=2002 --drc_out=${HOME}
@end verbatim
@end example

@cindex standard input
@cindex @code{stdin}
@command{ncclimo} automatically switches to timeseries reshaping mode
if it receives a list of files from @code{stdin}, or, alternatively,
placed as positional arguments (after the last command-line option), or
if neither of these is done and no @var{caseid} is specified, in which
case it assumes all @code{*.nc} files in @var{drc_in} constitute the
input file list. 
@example
@verbatim
# Split monthly timeseries into CMIP-like timeseries
cd ${drc_in};ls ${caseid}*.h4.nc | ncclimo              -v=T \
  --ypf=1 --yr_srt=56 --yr_end=76 --drc_out=${HOME}
# Split high-frequency timeseries into CMIP-like timeseries
cd ${drc_in};ls ${caseid}*.h4.nc | ncclimo --clm_md=hfs -v=T \
  --ypf=1 --yr_srt=56 --yr_end=76 --drc_out=${HOME}
@end verbatim
@end example

Options for @command{ncclimo} and @command{ncremap} come in both short
(single-letter) and long forms. 
The handful of long-option synonyms for each option allows the user 
to imbue the commands with a level of verbosity and precision that suits
her taste. 
A complete description of all options is given below, in alphabetical
order of the short option letter.
Long option synonyms are given just after the letter.
When invoked without options, @command{ncclimo} and @command{ncremap}
print a succinct table of all options and some examples.
All valid options for both operators are listed in their command
syntax above but, for brevity, options that @command{ncclimo} passes
straight through to @command{ncremap} are only fully described in the
table of @command{ncremap} options.
@table @option
@html
<a name="winter_mode"></a> <!-- http://nco.sf.net/nco.html#winter_mode -->
<a name="wnt_md"></a> <!-- http://nco.sf.net/nco.html#wnt_md -->
<a name="dec_md"></a> <!-- http://nco.sf.net/nco.html#dec_md -->
<a name="dcm_md"></a> <!-- http://nco.sf.net/nco.html#dcm_md -->
<a name="december_mode"></a> <!-- http://nco.sf.net/nco.html#december_mode -->
<a name="djf"></a> <!-- http://nco.sf.net/nco.html#djf -->
<a name="DJF"></a> <!-- http://nco.sf.net/nco.html#DJF -->
<a name="jfd"></a> <!-- http://nco.sf.net/nco.html#jfd -->
<a name="JFD"></a> <!-- http://nco.sf.net/nco.html#JFD -->
@end html
@cindex @code{-a @var{wnt_md}}
@cindex @code{djf}
@cindex @code{DJF}
@cindex @code{JFD}
@cindex @code{jfd}
@cindex @code{--dec_md}
@cindex @code{--dcm_md}
@cindex @code{--december_mode}
@cindex @code{--wnt_md}
@cindex @code{--winter_mode}
@cindex @var{dec_md}
@item -a @var{dec_md} (@code{--dec_md}, @code{--dcm_md}, @code{--december_mode}, @code{--dec_mode})
Winter mode aka December mode determines the start and end months of
the climatology and the type of NH winter seasonal average.
Two valid arguments are @code{jfd} (default, or synonyms @code{sdd}
and @code{JFD}) and @code{djf} (or synonyms @code{scd} and
@code{DJF}). 
@acronym{DJF}-mode is the same as @acronym{SCD}-mode which
stands for ``Seasonally Continuous December''. 
The first month used is December of the year before the start year
specified with @samp{-s}.
The last month is November of the end year specified with @samp{-e}.
In @acronym{DJF}-mode the Northern Hemisphere winter seasonal
climatology will be computed with sets of the three consecutive
months December, January, and February (DJF) where the calendar
year of the December months is always one less than the calendar
year of January and February.
@acronym{JFD}-mode is the same as @acronym{SDD}-mode which stands for
``Seasonally Discontinuous December''.
The first month used is January of the specified start year. 
The last month is December of the end year specified with @samp{-e}.
In @acronym{JFD}-mode the Northern Hemisphere winter seasonal
climatology will be computed with sets of the three non-consecutive
months January, February, and December (JFD) from each calendar year. 

@html
<a name="clm_md"></a> <!-- http://nco.sf.net/nco.html#clm_md -->
@end html
@cindex @code{-C @var{clm_md}}
@cindex @var{clm_md}
@cindex @code{--clm_md}
@cindex @code{--climatology_mode}
@cindex @code{--mode}
@cindex @code{--climatology}
@item -C @var{clm_md} (@code{--clm_md}, @code{--climatology_mode}, @code{--mode}, @code{--climatology})
Climatology mode.
Valid values for @var{clm_md} are
@code{ann} (or synonyms @code{annual}, @code{annual}, @code{yearly}, or @code{year})
for annual-mode climatologies,
@code{dly} (or synonyms @code{daily}, @code{doy}, or @code{day})
for daily-mode climatologies,
@code{hfc} (or synonyms @code{high_frequency_climo} or @code{hgh_frq_clm})
for high-frequency (diurnally resolved) climos, 
@code{hfs} (or synonyms @code{high_frequency_splitter} or @code{hgh_frq_spl})
for high-frequency splitting, and 
@code{mth} (or synonyms @code{month} or @code{monthly})
for monthly climotologies.
The value indicates the timespan of each input file for annual and
monthly climatologies.
The default mode is @samp{mth}, which means input files are monthly
averages.  
Use @samp{ann} when the input files are a series of annual means 
(a common temporal resolution for ice-sheet simulations).
The value @samp{dly} is used only input files whose temporal
resolution is daily or finer, and when the desired output is a
day-of-year climatology where the means are output for each
day of a @w{365 day} year.
Day-of-year climatologies are uncommon, yet useful for showing
daily variability.
The value @samp{hfc} indicates a high-frequency climatology where
the output will be a traditional set of climatological monthly,
seasonal, or annual means similar to monthly climos, except that
each file will have the same number of timesteps-per-day as the
input data to resolve the diurnal cycle.
The value @samp{hfs} indicates a high-frequency splitting operation
where an interannual input timeseries will be split into regular
size segments of a given number of years, similar to @acronym{CMIP}
timeseries.

The climatology generator and splitter do not require that daily-mode
input files begin or end on daily boundaries.
These tools hyperslab the input files using the date information
required to performed their analysis.
This facilitates analyzing datasets with varying numbers of days per
input file.

@cindex @code{stdin}
@cindex @command{crontab}
@cindex @code{nohup}
@cindex Python
Explicitly specifying @samp{--clm_md=mth} serves a secondary purpose,
namely invoking the default setting on systems that control
@code{stdin}.
When @command{ncclimo} detects that @code{stdin} is not attached to the
terminal (keyboard) it automatically expects a list of files on
@code{stdin}. 
Some environments, however, hijack @code{stdin} for their purposes
and thereby confuse @code{ncclimo} into expecting a list argument.
Users have encountered this issue when attempting to run
@command{ncclimo} in Python parallel environments, via inclusion in 
@command{crontab}, and in @command{nohup}-mode (whatever that is!).
In such cases, explicitly specify @samp{--clm_md=mth} (or @code{ann} or
@code{day}) to persuade @command{ncclimo} to compute a normal
climatology. 

@html
<a name="caseid"></a> <!-- http://nco.sf.net/nco.html#caseid -->
@end html
@cindex @code{-c @var{caseid}}
@cindex @var{caseid}
@cindex @code{--caseid}
@cindex @code{--case}
@item -c @var{caseid} (@code{--case}, @code{--caseid}, @code{--case_id})
Simulation name, or any input filename for non-@acronym{CESM}'ish files.  
The use of @var{caseid} is required in climate generation mode (unless
equivalent information is provided through other options), where
@var{caseid} is used to construct both input and output filenames.
For @acronym{CESM}'ish input files like
@file{famipc5_ne30_v0.3_00001.cam.h0.1980-01.nc}, 
specify @samp{-c famipc5_ne30_v0.3_00001}. 
The @samp{.cam.} and @samp{.h0.} bits are added internally to produce
the input filenames.
Modify these via the @option{-m @var{mdl_nm}} and @option{-h
@var{hst_nm}} options if needed.   
For input files named slightly differently than standard
@acronym{CESM}'ish names, supply the filename (excluding the path
component) as the @var{caseid} and then @command{ncclimo} will attempt
to parse that by matching to a database of regular expressions known
to be favored by various other datasets.
These expressions are all various formats of dates at the end of the
filenames, and adhere to the general convention
@var{prefix}[.-]@var{YYYY}[-]@var{MM}[-]@var{DD}[-]@var{SSSSS}.@var{suffix}.
The particular formats currently supported, as of @acronym{NCO} version
5.1.6 (May, 2023) are:
@var{prefix}@code{_YYYYMM}.@var{suffix}, 
@var{prefix}@code{.YYYY-MM}.@var{suffix}, 
@var{prefix}@code{.YYYY-MM-01}.@var{suffix}, and
@var{prefix}@code{.YYYY-MM-01-00000}.@var{suffix}.
For example, input files like @file{merra2_198001.nc} (i.e., the six
digits that precede the suffix are @var{YYYYMM}-format), specify
@samp{-c merra2_198001.nc} and the prefix (@code{merra2}) will be
automatically abstracted and used to template and generate all the
filenames based on the specified @var{yr_srt} and @var{yr_end}. 
Please tell us any dataset filename regular expressions that you would
like added to @command{ncclimo}'s internal database.

The @samp{--caseid=@var{caseid}} option is not mandatory in
the High-Frequency-Splitter (@var{clm_md}=@code{hfs}) and
High-Frequency-Climatology (@var{clm_md}=@code{hfc}) modes.
Those modes expect all input filenames to be entered from the
command-line so there is no internal need to create filenames
from the @var{caseid} variable.
Instead, when @var{caseid} is specified in a high-freqency mode,
its value is used to name the output files in a similar manner
to the @samp{-f @var{fml_nm}} option.

@cindex @code{-D @var{dbg_lvl}}
@cindex @var{dbg_lvl}
@cindex @code{--dbg_lvl}
@cindex @code{--debug_level}
@item -D @var{dbg_lvl} (@code{--dbg_lvl}, @code{--dbg}, @code{--debug}, @code{--debug_level})
Specifies a debugging level similar to the rest of @acronym{NCO}.
If @math{@var{dbg_lvl} = 1}, @command{ncclimo} prints more extensive
diagnostics of its behavior.
If @math{@var{dbg_lvl} = 2}, @command{ncclimo} prints the commands
it would execute at any higher or lower debugging level, but does
not execute these commands.
If @math{@var{dbg_lvl} > 2}, @command{ncclimo} prints the diagnostic
information, executes all commands, and passes-through the debugging
level to the regridder (@command{ncks}) for additional diagnostics.

@cindex @code{d2f}
@cindex @code{--d2f}
@cindex @code{--d2s}
@cindex @code{--dbl_flt}
@cindex @code{--dbl_sgl}
@cindex @code{--double_float}
@cindex @code{ncpdq}
@cindex Precision
@item --d2f (@code{--d2f}, @code{--d2s}, @code{--dbl_flt}, @code{--dbl_sgl}, @code{--double_float})
This switch (which takes no argument) causes @command{ncclimo}
to invoke @command{ncremap} with the same switch, so that
@command{ncremap} converts all double precision non-coordinate
variables to single precision in the regridded file.
This switch has no effect on files that are not regridded.
To demote the precision in such files, use @command{ncpdq} to apply
the @code{dbl_flt} packing map to the file directly.

@html
<a name="dpf"></a> <!-- http://nco.sf.net/nco.html#dpf -->
@end html
@cindex @code{--dpf=@var{dpf}}
@cindex @var{dpf}
@cindex @code{--dpf}
@cindex @code{--days-per-file}
@item --dpf=@var{dpf} (@code{--dpf}, @code{--days_per_file})
The number of days-per-file in files ingested by @command{ncclimo}.
It can sometimes be difficult for @command{ncclimo} to infer the
number of days-per-file in high-frequency input files, i.e., those
with 1 or more timesteps-per-day.
In such cases, users may override the inferred value by explicitly
specifying @code{--dpf=@var{dpf}}.

@cindex @code{--dpt_fl=@var{dpt_fl}}
@cindex @var{dpt_fl}
@cindex @code{--mpas_fl}
@cindex @code{--mpas_depth}
@cindex @code{--depth_file}
@cindex @command{add_depth.py}
@cindex Depth
@cindex @acronym{MPAS}
@item --dpt_fl=@var{dpt_fl} (@code{--dpt_fl}, @code{--depth_file}, @code{--mpas_fl}, @code{--mpas_depth})
The @samp{--dpt_fl=@var{dpt_fl}} triggers the addition of a depth
coordinate to @acronym{MPAS} ocean datasets that will undergo
regridding.
@command{ncclimo} passes this option through to @command{ncremap},
and this option has no effect when @command{ncclimo} does not invoke
@command{ncremap}. 
The @command{ncremap} documentation contains the full description of
this option.

@html
<a name="end_yr"></a> <!-- http://nco.sf.net/nco.html#end_yr -->
<a name="yr_end"></a> <!-- http://nco.sf.net/nco.html#yr_end -->
@end html
@cindex @code{-e @var{end_yr}}
@cindex @var{end_yr}
@cindex @code{--end_yr}
@cindex @code{--end_year}
@cindex @code{--year_end}
@cindex @code{--end}
@item -e @var{end_yr} (@code{--end_yr}, @code{--yr_end}, @code{--end_year}, @code{--year_end}, @code{--end})
End year (example: 2000). 
By default, the last month is December of the specified end year.  
If @samp{-a scd} is specified, the last month used is November of the
specified end year.  

@html
<a name="fml_nm"></a> <!-- http://nco.sf.net/nco.html#fml_nm -->
@end html
@cindex @code{-f @var{fml_nm}}
@cindex @var{fml_nm}
@cindex @code{--fml_nm}
@cindex @code{--fml}
@cindex @code{--family}
@cindex @code{--family_name}
@item -f @var{fml_nm} (@code{--fml_nm}, @code{--fml}, @code{--family}, @code{--family_name})
Family name (nickname) of output files.
In climate generation mode, output climo file names are constructed by
default with the same @var{caseid} as the input files.
The @var{fml_nm}, if supplied, replaces @var{caseid} in output climo
names, which are of the form
@var{fml_nm}_XX_YYYYMM_YYYYMM.nc where @var{XX} is the month or seasonal
abbreviation.
Use @samp{-f @var{fml_nm}} to simplify long names, avoid overlap, etc.
Example values of @var{fml_nm} are @samp{control}, @samp{experiment},
and (for a single-variable climo) @samp{FSNT}.
In timeseries reshaping mode, @var{fml_nm} will be used, if supplied,
as an additional string in the output filename.
For example, specifying @samp{-f control} would cause
@file{T_000101_000912.nc} to be instead named 
@file{T_control_000101_000912.nc}.

@html
<a name="hst_nm"></a> <!-- http://nco.sf.net/nco.html#hst_nm -->
@end html
@cindex @code{-h @var{hst_nm}}
@cindex @var{hst_nm}
@cindex @code{--history_name}
@cindex @code{--hst_nm}
@cindex @code{--history}
@item -h @var{hst_nm} (@code{--hst_nm}, @code{--history_name}, @code{--history})
History volume name of file used to generate climatologies.
This referring to the @var{hst_nm} character sequence used to construct
input file names: @code{caseid.mdl_nm.}@var{hst_nm}@code{.YYYY-MM.nc}.
By default input climo file names are constructed from the @var{caseid} 
of the input files, together with the model name @var{mdl_nm} (specified
with @samp{-m}) and the date range.
Use @samp{-h @var{hst_nm}} to specify alternative history volumes.
Examples include @samp{h0} (default, works for @acronym{CAM},
@acronym{CLM/CTSM/ELM}), @samp{h1}, and @samp{h} (for @acronym{CISM}).

@html
<a name="drc_in"></a> <!-- http://nco.sf.net/nco.html#drc_in -->
@end html
@cindex @code{-i @var{drc_in}}
@cindex @var{drc_in}
@cindex @code{--drc_in}
@cindex @code{--in_drc}
@cindex @code{--dir_in}
@cindex @code{--input}
@item -i @var{drc_in} (@code{--drc_in}, @code{--in_drc}, @code{--dir_in}, @code{--input})
Directory containing all monthly mean files to read as input to the
climatology.
The use of @var{drc_in} is mandatory in climate generation mode and is 
optional in timeseries reshaping mode.
In timeseries reshaping mode, @command{ncclimo} uses all netCDF files
(meaning files with suffixes @code{.nc}, @code{.nc3}, @code{.nc4},
@code{.nc5}, @code{.nc6}, @code{.nc7},
@code{.cdf}, @code{.hdf}, @code{.he5}, or @code{.h5}) in @var{drc_in} to 
create the list of input files when no list is provided through
@code{stdin} or as positional arguments to the command-line. 

@html
<a name="job_nbr"></a> <!-- http://nco.sf.net/nco.html#job_nbr -->
<a name="job_nbr_ncclimo"></a> <!-- http://nco.sf.net/nco.html#job_nbr_ncclimo -->
@end html
@cindex @code{-j @var{job_nbr}}
@cindex @var{job_nbr}
@cindex @code{--job_nbr}
@cindex @code{--job_number}
@cindex @code{--jobs}
@item -j @var{job_nbr} (@code{--job_nbr}, @code{--job_number}, @code{--jobs})
The @var{job_nbr} parameter controls the parallelism granularity of
both timeseries reshaping (aka splitting) and climatology generation.
These modes parallelize over different types of tasks, so we describe
the effects of @var{job_nbr} separately, first for climatologies, then
for splitting. 
However, for both modes, @var{job_nbr} specifies the total number of
simultaneous processes to run in parallel either on the local node for
Background parallelism, or across all the nodes for @acronym{MPI}
parallelism (i.e., @var{job_nbr} is the simultaneous total across all
nodes, it is not the simultaneous number per node).

For climatology generation, @var{job_nbr} specifies the number of
averaging tasks to perform simultaneously on the local node for
Background parallelism, or spread across all nodes for
@acronym{MPI}-parallelism.
By default @command{ncclimo} sets @math{@var{job_nbr} = 12} for
background parallelism mode.
This number ensures that monthly averages for all individual
months complete more-or-less simultaneously, so that all seasonal
averages can then be computed.
However, many nodes are too small to simultaneously average multiple
distinct months (January, February, etc.).
Hence @var{job_nbr} may be set to any factor of 12, i.e., 1, 2, 3, 4,
6, or 12.
For Background parallelism, setting @math{@var{job_nbr} = 4} causes
four-months to be averaged at one time.
After three batches of four-months complete, the climatology generator
then moves on to seasonal averaging and regridding.
In @acronym{MPI}-mode, @command{ncclimo} defaults to 
@math{@var{job_nbr} = @var{nd_nbr}} unless the user explicitly sets
@var{job_nbr} to a different value. 
For the biggest jobs, when a single-month nearly exhausts the
@acronym{RAM} on a node, this default value
@math{@var{job_nbr} = @var{nd_nbr}} ensures that each node gets only
one job at a time. 
To override this default for @acronym{MPI}-parallelism, set
@math{@var{job_nbr} >= @var{nd_nbr}} otherwise some nodes will be idle 
for the entire time.
If a node can handle averaging three distinct months simultaneously,
then try @math{@var{job_nbr} = 3*@var{nd_nbr}}.
Never set @math{@var{job_nbr} > 12} in climatology mode, since there
are at most only twelve jobs that can be performed in parallel.

For splitting, @var{job_nbr} specifies the number of simultaneous
subsetting processes to spawn during parallel execution for both
Background and @acronym{MPI}-parallelism.  
In both parallelism modes @command{ncclimo} spawns processes in
batches of @var{job_nbr} jobs, then waits for those processes to
complete. 
Once a batch finishes, @command{ncclimo} spawns the next batch.
For Background-parallelism, all jobs are spawned to the local node.
For @acronym{MPI}-parallelism, all jobs are spawned in round-robin
fashion to all available nodes until @var{job_nbr} jobs are running.
Rinse, lather, repeat until all variables have been split.
The splitter chooses its default value of @var{job_nbr} based on
on the parallelism mode.
For Background parallelism, @var{job_nbr} defaults to the number of
variables to be split, so that not specifying @var{job_nbr} results 
in launching @var{var_nbr} simultaneous splitter tasks.
This scales well to over a hundred variables in our tests
@footnote{At least one known environment (the @acronym{E3SM-Unified}
Anaconda environment at @acronym{NERSC}) prevents users from spawning
scores of processes and may report @acronym{OpenBLAS/pthread} or
@code{RLIMIT_NPROC}-related errors.
A solution seems to be executing @samp{ulimit -u unlimited}}.
In practice, splitting timeseries consumes minimal memory, since
@command{ncrcat} (which underlies the splitter) only holds one
record (timestep) of a variable in memory @ref{Memory Requirements}.

However, if splitting consumes so much @acronym{RAM} (e.g., because 
variables are large and/or the number of jobs is large) that a
single node can perform only one or a few subsetting jobs at a time,
then it is reasonable to use @acronym{MPI}-mode to split the datasets. 
For @acronym{MPI}-parallelism, @var{job_nbr} defaults to the number of 
nodes requested. 
This helps prevent users from overloading nodes with too many jobs.
Usually, however, nodes can usually subset (and then regrid, if
requested) multiple variables simultaneously.
In summary, the splitter defaults to
@math{@var{job_nbr} = @var{var_nbr}} in Background mode, and to
@math{@var{job_nbr} = @var{node_nbr}} in @acronym{MPI} mode. 
Subject to the availability of adequate @acronym{RAM}, expand the
number of jobs-per-node by increasing @var{job_nbr} until overall
throughput peaks.

The main throughput bottleneck in timeseries reshaping mode is I/O. 
Increasing @var{job_nbr} may reduce throughput once the maximum I/O
bandwidth of the node is reached, due to contention for I/O
resources. 
Regridding requires math that can relieve some I/O contention and
allows for some throughput gains with increasing @var{job_nbr}.
One strategy that seems sensible is to set @var{job_nbr} equal to
the number of nodes times the number of cores per node, and increase
or decrease as necessary until throughput peaks.

@cindex @code{-L}
@cindex @code{--dfl_lvl}
@cindex @code{--dfl}
@cindex @code{--deflate}
@item -L (@code{--dfl_lvl}, @code{--dfl}, @code{--deflate})
Activate deflation (i.e., lossless compress, see @ref{Deflation}) with
the @code{-L @var{dfl_lvl}} short option (or with the same argument to 
the @samp{--dfl_lvl} or @samp{--deflate} long options). 
Specify deflation level @var{dfl_lvl} on a scale from no deflation
(@var{dfl_lvl = 0}, the default) to maximum deflation
(@var{dfl_lvl = 9}).

@html
<a name="lnk_flg"></a> <!-- http://nco.sf.net/nco.html#lnk_flg -->
@end html
@cindex @code{-l}
@cindex @code{--lnk_flg}
@cindex @code{--link_flag}
@cindex @code{--no_amwg_links}
@cindex @code{--no_amwg}
@cindex @code{--amwg_links}
@item -l (@code{--lnk_flg}, @code{--link_flag})
@item --no_amwg_link (@code{--no_amwg_link}, @code{--no_amwg_links}, @code{--no_amwg}, @code{--no_AMWG_link}, @code{--no_AMWG_links})
@item --amwg_link (@code{--amwg_link}, @code{--amwg_links}, @code{--AMWG_link}, @code{--AMWG_links})
These options turn-on or turn-off the linking of
@acronym{E3SM/ACME}-climo to @acronym{AMWG}-climo filenames.
@acronym{AMWG} omits the @acronym{YYYYMM} components of climo filenames,
resulting in shorter names.
By default @command{ncclimo} symbolically links the full
@acronym{E3SM/ACME} filename (which is always) created to a file with
the shorter (@acronym{AMWG}) name whose creation is optional. 
@acronym{AMWG} diagnostics scripts can produce plots directly from
the linked @acronym{AMWG} filenames. 
The @samp{-l} (and @samp{--lnk_flg} and @samp{--link_flag} long-option
synonmyms) are true options that require an argument of either
@samp{Yes} or @samp{No}.
The remaining synonyms are switches that take no arguments.
The @samp{--amwg_link} switch and its synonyms cause the creation
of symbolic links with @acronym{AMWG} filenames.
The @samp{--no_amwg_link} switch and its synonyms prevent the creation
of symbolic links with @acronym{AMWG} filenames.
If you do not need @acronym{AMWG} filenames, turn-off linking to
reduce file proliferation in the output directories.

@html
<a name="mdl_nm"></a> <!-- http://nco.sf.net/nco.html#mdl_nm -->
@end html
@cindex @code{-m @var{mdl_nm}}
@cindex @var{mdl_nm}
@cindex @code{--model_name}
@cindex @code{--mdl}
@cindex @code{--model}
@cindex @code{--mdl_nm}
@item -m @var{mdl_nm} (@code{--mdl_nm}, @code{--mdl}, @code{--model_name}, @code{--model})
Model name (as embedded in monthly input filenames). 
Default is @samp{cam}. Other options are @samp{clm2}, @samp{ocn},
@samp{ice}, @samp{cism}, @samp{cice}, @samp{pop}. 

@html
<a name="nco_opt"></a> <!-- http://nco.sf.net/nco.html#nco_opt -->
@end html
@cindex @code{-n @var{nco_opt}}
@cindex @var{nco_opt}
@cindex @code{--nco_opt}
@cindex @code{--nco_options}
@cindex @code{--nco}
@item -n @var{nco_opt} (@code{nco_opt}, @code{nco}, @code{nco_options})
Specifies a string of options to pass-through unaltered to
@command{ncks}. 
@var{nco_opt} defaults to @samp{--no_tmp_fl}.
Note that @command{ncclimo} passes its @var{nco_opt} to
@command{ncremap}.
This can cause unexpected results, so use the front-end options to
@command{ncclimo} when possible, rather than attempting to subvert
them with @var{nco_opt}.

@html
<a name="drc_rgr"></a> <!-- http://nco.sf.net/nco.html#drc_rgr -->
@end html
@cindex @code{-O @var{drc_rgr}}
@cindex @var{drc_rgr}
@cindex @code{--drc_rgr}
@cindex @code{--rgr_drc}
@cindex @code{--dir_rgr}
@cindex @code{--regrid}
@item -O @var{drc_rgr} (@code{--drc_rgr}, @code{--rgr_drc}, @code{--dir_rgr}, @code{--regrid})
Directory to hold regridded climo files.
Regridded climos are placed in @var{drc_out} unless a separate
directory for them is specified with @samp{-O} (NB: capital ``O''). 

@html
<a name="no_cll_msr"></a> <!-- http://nco.sf.net/nco.html#no_cll_msr -->
@end html
@cindex @code{--no_cll_msr}
@cindex @code{--no_cll}
@cindex @code{--no_cell_measures}
@cindex @code{--no_area}
@cindex @code{cell_measures} attribute
@item --no_cll_msr (@code{--no_cll_msr}, @code{--no_cll}, @code{--no_cell_measures}, @code{--no_area})
This switch (which takes no argument) controls whether @command{ncclimo}
and @command{ncremap} add measures variables to the extraction list
along with the primary variable and other associated variables. 
See @ref{CF Conventions} for a detailed description.

@html
<a name="no_frm_trm"></a> <!-- http://nco.sf.net/nco.html#no_frm_trm -->
@end html
@cindex @code{--no_frm_trm}
@cindex @code{--no_frm}
@cindex @code{--no_formula_terms}
@cindex @code{formula_terms} attribute
@item --no_frm_trm (@code{--no_frm_trm}, @code{--no_frm}, @code{--no_formula_terms})
This switch (which takes no argument) controls whether @command{ncclimo}
and @command{ncremap} add formula variables to the extraction list along
with the primary variable and other associated variables. 
See @ref{CF Conventions} for a detailed description.

@html
<a name="hms_avg"></a> <!-- http://nco.sf.net/nco.html#hms_avg -->
<a name="rgn_avg"></a> <!-- http://nco.sf.net/nco.html#rgn_avg -->
<a name="glb_avg"></a> <!-- http://nco.sf.net/nco.html#glb_avg -->
<a name="global_average"></a> <!-- http://nco.sf.net/nco.html#global_average -->
<a name="regional_average"></a> <!-- http://nco.sf.net/nco.html#regional_average -->
@end html
@cindex @code{--glb_avg}
@cindex @code{--global_average}
@cindex @code{--rgn_avg}
@cindex @code{--regional_average}
@cindex @code{--hms_avg}
@cindex @code{--hemispheric_average}
@cindex global average
@cindex regional average
@cindex hemispheric average
@item --glb_avg (@code{--glb_avg}, @code{--global_average}) (deprecated) 
@item --rgn_avg (@code{--rgn_avg}, @code{--region_average})
When introduced in @acronym{NCO} version 4.9.1 (released December,
2019), this switch (which takes no argument) caused the splitter to
output global horizontally spatially averaged timeseries files instead of
raw, native-grid timeseries.
This switch changed behavior in @acronym{NCO} version 5.1.1
(released November, 2022).
It now causes the splitter to output three horizontally spatially
averaged timeseries.
First is the global average (as before), next is the northern
hemisphere average, followed by the southern hemisphere average.
The three timeseries are now saved in a two-dimensional (time
by region) array with a ``region dimension'' named @code{rgn}.
The region names are stored in the variable named @code{region_name}.

This switch only has effect in timeseries splitting mode.
This is useful, for example, to quickly diagnose the behavior of ongoing
model simulations prior to a full-blown analysis. 
Thus the spatial mean files will be in the same location and have the same
name as the native grid timeseries would have been and had, respectively.
Note that this switch does not alter the capability of also outputting
the full regridded timeseries, if requested, at the same time.

Because the switch now outputs global and regional averages, the best
practice is to invoke with @samp{--rgn_avg} instead of
@samp{--glb_avg}. 

@html
<a name="no_ntv_tms"></a> <!-- http://nco.sf.net/nco.html#no_ntv_tms -->
<a name="no_ntv"></a> <!-- http://nco.sf.net/nco.html#no_ntv -->
<a name="no_native"></a> <!-- http://nco.sf.net/nco.html#no_native -->
@end html
@cindex @code{--no_ntv_tms}
@cindex @code{--no_ntv}
@cindex @code{--no_native}
@cindex @code{--remove_native}
@item --no_ntv_tms (@code{--no_ntv_tms}, @code{--no_ntv}, @code{--no_native}, @code{--remove_native})
This switch (which takes no argument) controls whether the splitter
retains native grid split files, which it does by default, or deletes them.
@command{ncclimo} can split model output from multi-variable native grid
files into per-variable timeseries files and regrid those onto a
so-called analysis grid. 
That is the typical format in which Model Intercomparison Projects
(@acronym{MIP}s) request and disseminate contributions.
When the data producer has no use for the split timeseries on the native
grid, he/she can invoke this flag to cause @command{ncclimo} to
delete the native grid timeseries (not the raw native grid datafiles).
This functionality is implemented by first creating the native grid 
timeseries, regridding it, and then overwriting the native grid
timeseries with the regridded timeseries.
Thus the regridded files will be in the same location and have the same
name as the native grid timeseries would have been and had, respectively.

@html
<a name="no_stg_grd"></a> <!-- http://nco.sf.net/nco.html#no_stg_grd -->
<a name="no_stg"></a> <!-- http://nco.sf.net/nco.html#no_stg -->
@end html
@cindex @code{slat}
@cindex @code{slon}
@cindex @code{w_stag}
@cindex @code{--no_stg_grd}
@cindex @code{--no_stg}
@cindex @code{--no_stagger}
@cindex @code{--no_staggered_grid}
@item --no_stg_grd (@code{--no_stg_grd}, @code{--no_stg}, @code{--no_stagger}, @code{--no_staggered_grid})
This switch (which takes no argument) controls whether
regridded output will contain the staggered grid coordinates 
@code{slat}, @code{slon}, and @code{w_stag} (@pxref{Regridding}). 
By default the staggered grid is output for all files regridded from 
a Cap (aka @acronym{FV}) grid, except when the regridding is performed
as part of splitting (reshaping) into timeseries.

@html
<a name="drc_out"></a> <!-- http://nco.sf.net/nco.html#drc_out -->
@end html
@cindex @code{-o @var{drc_out}}
@cindex @var{drc_out}
@cindex @code{--drc_out}
@cindex @code{--out_drc}
@cindex @code{--dir_out}
@cindex @code{--output}
@item -o @var{drc_out} (@code{--drc_out}, @code{--out_drc}, @code{--dir_out}, @code{--output})
Directory to hold computed (output) native grid climo files.
Regridded climos are also placed here unless a separate directory
for them is specified with @samp{-O} (NB: capital ``O''). 

@html
<a name="par_typ"></a> <!-- http://nco.sf.net/nco.html#par_typ -->
<a name="par_typ_ncclimo"></a> <!-- http://nco.sf.net/nco.html#par_typ_ncclimo -->
@end html
@cindex @code{-p @var{par_typ}}
@cindex @var{par_typ}
@cindex @code{--par_typ}
@cindex @code{--par_md}
@cindex @code{--parallel_type}
@cindex @code{--parallel_mode}
@cindex @code{--parallel}
@item -p @var{par_typ} (@code{--par_typ}, @code{--par_md}, @code{--parallel_type}, @code{--parallel_mode}, @code{--parallel})
Specifies the parallelism mode desired.
The options are serial mode (@samp{-p srl}, @samp{-p serial}, or @samp{-p nil}), 
background mode parallelism (@samp{-p bck} or @samp{-p background})),
and @acronym{MPI} parallelism (@samp{-p mpi} or @samp{-p MPI}). 
The default is background-mode parallelism.
The default @var{par_typ} is @samp{background}, which means
@command{ncclimo} spawns up to twelve (one for each month) parallel
processes at a time.
See discussion below under Memory Considerations.

@html
<a name="ppc_prc"></a> <!-- http://nco.sf.net/nco.html#ppc_prc -->
@end html
@cindex @code{--ppc=@var{ppc_prc}}
@cindex @var{ppc_prc}
@cindex @code{--ppc}
@cindex @code{--ppc_prc}
@cindex @code{--precision}
@cindex @code{--quantize}
@item --ppc=@var{ppc_prc} (@code{--ppc}, @code{--ppc_prc}, @code{--precision}, @code{--quantize})
@cindex Decimal Significant Digits
@cindex Number of Significant Digits
@cindex Bit-Grooming
Specifies the precision of the Precision-Preserving Compression
algorithm (@pxref{Precision-Preserving Compression}).
A positive integer is interpreted as the Number of Significant Digits
for the Bit-Grooming algorithm, and is equivalent to specifying
@samp{--ppc default=@var{ppc_prc}} to a binary operator.
A positive or negative integer preceded by a period, e.g., @samp{.-2} is 
interpreted as the number of Decimal Significant Digits for the
rounding algorithm and is equivalent to specifying
@samp{--ppc default=.@var{ppc_prc}} to a binary operator.
This option applies one precision algorithm and a uniform precision
for the entire file. 
To specify variable-by-variable precision options, pass the desired
options as a quoted string directly with @samp{-n @var{nco_opt}},
e.g., @samp{-n '--ppc FSNT,TREFHT=4 --ppc CLOUD=2'}.

@html
<a name="rgr_opt"></a> <!-- http://nco.sf.net/nco.html#rgr_opt -->
@end html
@cindex @code{-R @var{rgr_opt}}
@cindex @var{rgr_opt}
@cindex @code{--rgr_opt}
@cindex @code{--regrid_options}
@item -R @var{rgr_opt} (@code{rgr_opt}, @code{regrid_options})
Specifies a string of options to pass-through unaltered to
@command{ncks}. 
@var{rgr_opt} defaults to @samp{-O --no_tmp_fl}.

@html
<a name="rgr_map"></a> <!-- http://nco.sf.net/nco.html#rgr_map -->
@end html
@cindex @code{-r @var{rgr_map}}
@cindex @var{rgr_map}
@cindex @code{--rgr_map}
@cindex @code{--regrid_map}
@cindex @code{--map}
@item -r @var{rgr_map} (@code{--rgr_map}, @code{--regrid_map}, @code{--map})
Regridding map.
Unless @samp{-r} is specified @command{ncclimo} produces only a 
climatology on the native grid of the input datasets.
The @var{rgr_map} specifies how to (quickly) transform the native
grid into the desired analysis grid.
@command{ncclimo} will (call @command{ncremap} to) apply the given map
to the native grid climatology and produce a second climatology on the
analysis grid.
Options intended exclusively for the regridder may be passed as
arguments to the @samp{-R} switch.
See below the discussion on regridding.

@html
<a name="mth_srt"></a> <!-- http://nco.sf.net/nco.html#mth_srt -->
<a name="srt_mth"></a> <!-- http://nco.sf.net/nco.html#srt_mth -->
<a name="mth_end"></a> <!-- http://nco.sf.net/nco.html#mth_end -->
<a name="end_mth"></a> <!-- http://nco.sf.net/nco.html#end_mth -->
@end html
@cindex @code{--mth_srt}
@cindex @code{--srt_mth}
@cindex @code{--start_month}
@cindex @code{--month_start}
@item --mth_srt=@var{mth_srt} (@code{--mth_srt}, @code{--srt_mth}, @code{--month_start}, @code{--start_month})
@cindex @code{--mth_end}
@cindex @code{--end_mth}
@cindex @code{--end_month}
@cindex @code{--month_end}
@item --mth_end=@var{mth_end} (@code{--mth_end}, @code{--end_mth}, @code{--month_end}, @code{--end_month})
Start month (example: 4), and end month (example: 11).
The starting month of monthly timeseries extracted by the splitter
defaults to January of the specified start year, and the ending
month defaults to December of the specified end year.
As of @acronym{NCO} @w{version 4.9.8}, released in March, 2021,
the splitter mode of @command{ncclimo} accepts user-specified
start and end months with the @samp{--mth_srt} and @samp{--mth_end}
options, respectively.
Months are input as one-based integers so January @w{is 1} and
December is @w{is 12}.
To extract 14-month timeseries from individual monthly input files one
could use, e.g., 
@example
ncclimo --yr_srt=1 --yr_end=2 --mth_srt=4 --mth_end=5 ...
@end example
Note that @code{mth_srt} and @code{mth_end} only affect the splitter,
and that they play no role in climatology generation.

@html
<a name="srt_yr"></a> <!-- http://nco.sf.net/nco.html#srt_yr -->
<a name="yr_srt"></a> <!-- http://nco.sf.net/nco.html#yr_srt -->
@end html
@cindex @code{-s @var{srt_yr}}
@cindex @var{srt_yr}
@cindex @code{--srt_yr}
@cindex @code{--start_year}
@cindex @code{--year_start}
@cindex @code{--start}
@item -s @var{srt_yr} (@code{--srt_yr}, @code{--yr_srt}, @code{--start_year}, @code{--year_start}, @code{--start})
Start year (example: 1980). 
By default, the first month used is January of the specified start year. 
If @samp{-a scd} is specified, the first month used will be December
of the year before the start year (to allow for contiguous
@acronym{DJF} climos).

@html
<a name="csn_lst"></a> <!-- http://nco.sf.net/nco.html#csn_lst -->
@end html
@cindex @code{--seasons=@var{csn_lst}}
@cindex @var{csn_lst}
@cindex @code{--seasons}
@cindex @code{--csn}
@cindex @code{--csn_lst}
@item --seasons=@var{csn_lst} (@code{--seasons}, @code{--csn_lst}, @code{--csn})
Seasons for @command{ncclimo} to compute in monthly climatology
generation mode. 
The list of seasons, @var{csn_lst}, is a comma-separated,
case-insensitive, unordered subset of the abbreviations for the eleven
(so far) defined seasons: 
@code{jfm}, @code{amj}, @code{jas}, @code{ond}, @code{on}, @code{fm},
@code{djf}, @code{mam}, @code{jja}, @code{son}, and @code{ann}.
By default @code{@var{csn_lst}=mam,jja,son,djf}.
Moreover, @command{ncclimo} automatically computes the climatological
annual mean, @code{ANN}, is always computed when MAM, JJA, SON, and DJF
are all requested (which is the default).
The ANN computed automatically is the time-weighted average of the four
seasons, rather than as the time-weighted average of the twelve monthly
climatologies. 
Users who need ANN but not DJF, MAM, JJA, and SON should instead
explicitly specify ANN as a season in @var{csn_lst}.
The ANN computed as a season is the time-weighted average of the twelve
monthly climatologies, rather than the time-weighted average of four
seasonal climatologies. 
Specifying the four seasons and ANN in @var{csn_lst} (e.g.,
@code{@var{csn_lst}=mam,jja,son,djf,ann}) is legal though redundant
and wasteful.
It cause ANN to be computed twice, first as the average of the twelve
monthly climatologies, then as the average of the four seasons.
The special value @code{@var{csn_lst}=none} turns-off computation of
seasonal (and annual) climatologies. 
@example
ncclimo --seasons=none ...            # Produce only monthly climos
ncclimo --seasons=mam,jja,son,djf ... # Monthly + MAM,JJA,SON,DJF,ANN
ncclimo --seasons=jfm,jas,ann ...     # Monthly + JFM,JAS,ANN
ncclimo --seasons=fm,on ...           # Monthly + FM,ON
@end example

@html
<a name="split"></a> <!-- http://nco.sf.net/nco.html#split -->
<a name="splitter"></a> <!-- http://nco.sf.net/nco.html#splitter -->
<a name="tms_flg"></a> <!-- http://nco.sf.net/nco.html#tms_flg -->
<a name="timeseries"></a> <!-- http://nco.sf.net/nco.html#timeseries -->
@end html
@cindex @code{--split}
@cindex @code{--splitter}
@cindex @code{--tms_flg}
@cindex @code{--timeseries}
@item --split (@code{--split}, @code{--splitter}, @code{--tms_flg}, @code{--timeseries})
This switch (which takes no argument) explicitly instructs
@command{ncclimo} to split the input multi-variable raw datasets into 
per-variable timeseries spanning the entire period.
The @code{--split} switch, and its synonyms @code{--splitter},
@code{--tms_flg}, and @code{--timeseries}, were introduced in
@acronym{NCO} version 5.0.4 (released December, 2021).
Previously, the splitter was automatically invoked whenever the input
files were provided via @code{stdin}, globbing, or positional
command-line arguments, with some exceptions.
That older method became ambiguous and untenable once it was decided
to also allow climos to be generated from files provided via
@code{stdin}, globbing, or positional command-line arguments.
Now there are three methods to invoke the splitter:
@w{1) Use} the @samp{--split} flag:
this is the most explicit way to invoke the splitter.
@w{2) Select} @samp{clm_md=hfs}:
the high-frequency splitter mode by definition invokes the splitter,
so a more explicit option than this is not necessary.
@w{3) Set} the years-per-file option, e.g., @samp{--ypf=25}:
the @code{ypf_max} option is only useful to the splitter, and has
thus been used in many scripts.
Since this option still causes the splitter to be invoked, those
will perform as before the @acronym{API} change.

These three splitter invocations methods are non-exclusive, i.e., more
than one can be used, and there is no harm in doing so.
While the @acronym{API} change in version 5.0.4 does proscribe the
former practice of passively invoking the splitter by simply piping
files to @code{stdin} or similar, it enables much more flexibility
for future features, including the possibility of automatically
generating timeseries filenames for the splitter, and of piping
files to @code{stdin} or similar for climo generation.

@html
<a name="no_stdin"></a> <!-- http://nco.sf.net/nco.html#no_stdin -->
@end html
@cindex Chrysalis
@cindex Common Workflow Language
@cindex Azure @acronym{CI}
@cindex @acronym{SLURM}
@cindex @acronym{CWL}
@cindex @code{--no_stdin}
@cindex @code{--no_inp_std}
@cindex @code{--no_standard_input}
@cindex @code{--no_redirect}
@item --no_stdin (@code{--no_stdin}, @code{--no_inp_std}, @code{--no_redirect}, @code{--no_standard_input})
First introduced in @acronym{NCO} version 4.8.0 (released May, 2019),
this switch (which takes no argument) disables checking standard input
(aka @code{stdin}) for input files. 
This is useful because @command{ncclimo} and @command{ncremap} may
mistakenly expect input to be provided on @code{stdin} in environments
that use @code{stdin} for other purposes. 
Some non-interactive environments (e.g., @command{crontab},
@code{nohup}, Azure @acronym{CI}, @acronym{CWL}), may
use standard input for their own purposes, and thus confuse
@acronym{NCO} into thinking that you provided the input files names
via the @code{stdin} mechanism. 
In such cases users may disable the automatic checks for standard
input by explicitly invoking the @samp{--no_stdin} flag.
This switch is usually not required for jobs in an interactive shell.
Interactive @acronym{SLURM} shells can also commandeer @code{stdin},
as is the case on the @code{DOE} machine named Chrysalis.
This behavior appears to vary depending on the @acronym{SLURM}
implementation.
@example
@verbatim
ncclimo --no_stdin -v T -s 2000 -e 2001 --ypf=10 -i in -o out
@end verbatim
@end example

@cindex @code{-t @var{thr_nbr}}
@cindex @var{thr_nbr}
@cindex @code{--thr_nbr}
@cindex @code{--thr}
@cindex @code{--threads}
@cindex @code{--thread_number}
@item -t @var{thr_nbr} (@code{--thr_nbr}, @code{--thr}, @code{--thread_number}, @code{--threads})
Specifies the number of threads used per regridding process
(@pxref{OpenMP Threading}).
The @acronym{NCO} regridder scales well to 8--16 threads.
However, regridding with the maximum number of threads can interfere
with climatology generation in parallel climatology mode (i.e., when
@math{@var{par_typ}} = @code{mpi} or @code{bck}). 
Hence @command{ncclimo} defaults to @var{thr_nbr}=2.

@html
<a name="tpd"></a> <!-- http://nco.sf.net/nco.html#tpd -->
@end html
@cindex @code{--tpd@var{tpd_out}}
@cindex @code{--tpd}
@cindex @code{--tpd_out}
@cindex @code{--timesteps_per_day}
@cindex @var{tpd}
@item --tpd=@var{tpd} (@code{--tpd_out}, @code{--tpd}, @code{--timesteps_per_day})
Normally, the number of timesteps-per-day in files ingested by
@command{ncclimo}.
It can sometimes be difficult for @command{ncclimo} to infer the
number of timesteps-per-day in high-frequency input files, i.e., those
with 1 or more timesteps-per-day.
In such cases, users may override the inferred value by explicitly
specifying @code{--tpd=@var{tpd}}.

The value of @var{tpd_out} in daily-average climatology mode
@code{clm_md=dly} (which is generally not used outside of ice-sheet
models) is different, and actually refers to the number of timesteps
per day that ncclimo will output, regardless of its value in the input
files.
Hence in daily-average mode (only), we refer to this variable as
@var{tpd_out}.

The climatology output from input files at daily or sub-daily resolution
is, by default, averaged to daily resolution, i.e., @var{tpd_out}=1. 
If the number of timesteps per day in each input file is @var{tpd_in},
then the user may select any value of @var{tpd_out} that is smaller 
than and integrally divides @var{tpd_in}.
For example, an input timeseries with @var{tpd_in}=8 (i.e., 3-hourly
resolution), can be used to produce climatological output at 3, 6,
or 12-hourly resolution by setting @var{tpd_out} to 8, 4, or 2,
respectively. 
This option only takes effect in daily-average climatology mode.

For full generality, the @code{--tpd} option should probably be split
into separate options @code{--tpd_in} and @code{--tpd_out}.
However, because it is unlikely that anyone will need to specify these
to different values, we leave only one option.
If this hinders you, please let us know and we will split the options.

@cindex @code{-v @var{var_lst}}
@cindex @var{var_lst}
@cindex @code{--var_lst}
@cindex @code{--var}
@cindex @code{--vars}
@cindex @code{--variables}
@cindex @code{--variable_list}
@item -v @var{var_lst} (@code{--var_lst}, @code{--var}, @code{--vars}, @code{--variables}, @code{--variable_list})
Variables to subset or to split.
Same behavior as @ref{Subsetting Files}.
The use of @var{var_lst} is optional in clim-generation mode.
We suggest using this feature to test whether an @command{ncclimo}
command, especially one that is lengthy and/or time-consuming, works as
intended on one or a few variables with, e.g., @samp{-v T,FSNT} before
generating the full climatology (by omitting this option). 
Invoking this switch was required in the original splitter released in 
version 4.6.5 (March, 2017), and became optional as of version 4.6.6
(May, 2017). 
This option is recommended in timeseries reshaping mode to prevent
inadvertently copying the results of an entire model simulation. 
Regular expressions are allowed so, e.g., @samp{PREC.?} extracts
the variables @samp{PRECC,PRECL,PRECSC,PRECSL} if present.
Currently in reshaping mode all matches to a regular expression are
placed in the same output file.
We hope to remove this limitation in the future.

@html
<a name="var_xtr"></a> <!-- http://nco.sf.net/nco.html#var_xtr -->
@end html
@cindex @var{var_xtr}
@cindex @code{--var_xtr}
@cindex @code{--var_extra}
@cindex @code{--extra_variables}
@cindex @code{--variables_extra}
@item --var_xtr=@var{var_xtr} (@code{--var_xtr}, @code{--var_xtr}, @code{--var_extra}, @code{--variables_extra}, @code{--extra_variables})
The @samp{--var_xtr} option causes @command{ncclimo} to include the extra
variables list in @var{var_xtr} in every timeseries split from the raw
data. 
This is useful when extra variables are desired in timeseries.
There are no limits on the extra variables---they may be of any rank
and may be timeseries themselves.
One useful application of this option, is to ensure that the area
variable is included with each timeseries, e.g.,
@samp{--var_xtr=area}. 

@cindex @code{--version}
@cindex @code{--vrs}
@cindex @code{--config}
@cindex @code{--configuration}
@cindex @code{--cnf}
@item --version (@code{--version}, @code{--vrs}, @code{--config}, @code{--configuration}, @code{--cnf})
This switch (which takes no argument) causes the operator to print
its version and configuration.
This includes the copyright notice, @acronym{URL}s to the @acronym{BSD}
and @acronym{NCO} license, directories from which the @acronym{NCO}
scripts and binaries are running, and the locations of any separate 
executables that may be used by the script.

@html
<a name="xcl_var"></a> <!-- http://nco.sf.net/nco.html#xcl_var -->
@end html
@cindex @code{--xcl_var}
@cindex @code{--xcl}
@cindex @code{--exclude}
@cindex @code{--exclude_variables}
@item --xcl_var (@code{--xcl_var}, @code{--xcl}, @code{--exclude}, @code{--exclude_variables})
This flag (which takes no argument) changes @var{var_lst},
as set by the @code{--var_lst} option, from an extraction list to an
exclusion list so that variables in @var{var_lst} will not be
processed, and variables not in @var{var_lst} will be processed.
Thus the option @samp{-v @var{var_lst}} must also be present for this
flag to take effect.
Variables explicitly specified for exclusion by
@samp{--xcl --vars=@var{var_lst}[,@dots{}]} need not be present in the
input file.

@html
<a name="ypf"></a> <!-- http://nco.sf.net/nco.html#ypf -->
@end html
@cindex @code{--ypf_max @var{ypf_max}}
@cindex @var{ypf_max}
@cindex @code{--ypf_max}
@item --ypf_max @var{ypf_max} (@code{--ypf}, @code{--years}, @code{--years_per_file})
Specifies the maximum number of years-per-file output by
@command{ncclimo}'s splitting operation.  
When @command{ncclimo} subsets and splits a collection of input files
spanning a timerseries, it places each subset variable in its own output
file. 
The maximum length, in years, of each output file is @var{ypf_max},
which defaults to @var{ypf_max}=50.
If an input timeseries spans @w{237 years} and @var{ypf_max}=50, then
@command{ncclimo} will generate four output files of length @w{50 years}  
and one output file of length @w{37 years}.
Note that invoking this option @emph{causes} @command{ncclimo} to enter
timeseries reshaping mode.
In fact, one @emph{must} use @samp{--ypf} to turn-on splitter mode when
the input files are specified by using the @samp{-i drc_in} method.
Otherwise it would be ambiguous whether to generate a climatology from 
or to split the input files. 
@end table
@noindent

@unnumberedsubsec Timeseries Reshaping mode, aka Splitting
This section of the @command{ncclimo} documentation applies only to
resphaping mode, whereas all subsequent sections apply to climatology
generation mode. 
In splitter mode, @command{ncclimo} @emph{reshapes} the input so that
the outputs are continuous timeseries of each variable taken from all
input files.  
As of @acronym{NCO} version 5.0.4 (released December, 2021), 
@command{ncclimo} enters splitter mode when invoked with the
@code{--split} switch (or its synonyms @code{--splitter}, 
@code{--tms_flg}, or @code{--timeseries}) or with the @code{--ypf_max} 
option. 
Then @command{ncclimo} will create per-variable timeseries from the
list of files supplied via @code{stdin}, or, alternatively,
placed as positional arguments (after the last command-line option), or
if neither of these is done and no @var{caseid} is specified, in which
case it assumes all @code{*.nc} files in @var{drc_in} constitute the
input file list. 
These examples invoke reshaping mode in the four possible ways:
@example
# Pipe list to stdin
cd $drc_in;ls *mdl*000[1-9]*.nc | ncclimo --split -v T,Q,RH -s 1 -e 9 -o $drc_out
# Redirect list from file to stdin
cd $drc_in;ls *mdl*000[1-9]*.nc > foo;ncclimo --split -v T,Q,RH -s 1 -e 9 -o $drc_out < foo
# List as positional arguments
ncclimo --split -v T,Q,RH -s 1 -e 9 -o $drc_out $drc_in/*mdl*000[1-9]*.nc
# Glob directory
ncclimo --split -v T,Q,RH -s 1 -e 9 -i $drc_in -o $drc_out
@end example
Assuming each input file is a monthly average comprising the variables
@var{T}, @var{Q}, and @var{RH}, then the output will be files
@file{T_000101_000912.nc}, 
@file{Q_000101_000912.nc}, and
@file{RH_000101_000912.nc}. 
When necessary, the output is split into segments each containing no
more than @var{ypf_max} (default 50) years of input, i.e., 
@file{T_000101_005012.nc}, @file{T_005101_009912.nc},
@file{T_010001_014912.nc}, etc.

@unnumberedsubsec @acronym{MPAS-O/I/L} considerations
@acronym{MPAS} ocean and ice models currently have their own
(non-CESM'ish) naming convention that guarantees output files have the
same names for all simulations. 
By default @command{ncclimo} analyzes the ``timeSeriesStatsMonthly''
analysis member output (tell us if you want options for other analysis
members). 
@command{ncclimo} and @command{ncremap} recognize input files as being 
@acronym{MPAS}-style when invoked with @samp{-P mpas} or with the
more expressive synonym  @samp{--prc_typ=mpas}.
The the generic @samp{-P mpas} invocation works for generating
climatologies for any @acronym{MPAS} model.
However, some regridder options are model-specific and therefore it is  
smarter to specify which @acronym{MPAS} model produced the input
data with 
@samp{-P mpasocean}, (or @samp{-P mpaso} for short),
@samp{-P mpasseaice}, (or @samp{-P mpassi} for short), or
@samp{-P mali}, like this: 
@example
@verbatim
ncclimo -P mpaso  -s 1980 -e 1983 -i $drc_in -o $drc_out # MPAS-O
ncclimo -P mpassi -s 1980 -e 1983 -i $drc_in -o $drc_out # MPAS-SI
ncclimo -P mali   -s 1980 -e 1983 -i $drc_in -o $drc_out # MPAS-LI
@end verbatim
@end example

Raw output data from all @acronym{MPAS} models does not contain
missing value attributes @footnote{
We submitted pull-requests to implement the @code{_FillValue}
attribute in all @acronym{MPAS}-ocean output in July, 2020.
The status of this @acronym{PR} may be tracked at
@url{https://github.com/MPAS-Dev/MPAS-Model/pull/677}.
Once this @acronym{PR} is merged to master, we will do the same
for the @acronym{MPAS}-Seaice and @acronym{MPAS}-Landice models.}.
These attributes must be manually added before sending the data as
input to @command{ncclimo} or @command{ncremap}.
We recommend that simulation producers annotate all floating point
variables with the appropriate @code{_FillValue} prior to invoking
@command{ncclimo}. 
Run something like this once in the history-file directory: 
@example
@verbatim
for fl in `ls hist.*` ; do
  ncatted -O -t -a _FillValue,,o,d,-9.99999979021476795361e+33 ${fl}
done
@end verbatim
@end example
If/when @acronym{MPAS-O/I} generates the @code{_FillValue} attributes
itself, this step can and should be skipped. 
All other @command{ncclimo} features like regridding (below) are invoked 
identically for @acronym{MPAS} as for @acronym{CAM}/@acronym{CLM} users
although under-the-hood @command{ncclimo} does do some special
pre-processing (dimension permutation, metadata annotation) for
@acronym{MPAS}.  
A five-year oEC60to30 @acronym{MPAS-O} climo with regridding to T62
takes less than @w{10 minutes} on the machine @file{rhea}. 

@unnumberedsubsec Annual climos
Not all model or observed history files are created as monthly means. 
To create a climatological annual mean from a series of annual mean
inputs, select @command{ncclimo}'s annual climatology mode with 
the @samp{-C ann} option:  
@example
@verbatim
ncclimo -C ann -m cism -h h -c caseid -s 1851 -e 1900 -i drc_in -o drc_out
@end verbatim
@end example
The options @samp{-m mdl_nm} and @samp{-h hst_nm} (that default to 
@code{cam} and @code{h0}, respectively) tell @command{ncclimo} how to
construct the input filenames. 
The above formula names the files
@code{caseid.cism.h.1851-01-01-00000.nc},
@code{caseid.cism.h.1852-01-01-00000.nc}, 
and so on. 
Annual climatology mode produces a single output file (or two if
regridding is selected), and in all other respects behaves the same as
monthly climatology mode.  

@unnumberedsubsec Regridding Climos and Other Files
@command{ncclimo} will (optionally) regrid during climatology generation 
and produce climatology files on both native and analysis grids. 
This regridding is virtually free, because it is performed on idle
nodes/cores after monthly climatologies have been computed and while
seasonal climatologies are being computed.  
This load-balancing can save half-an-hour on ne120 datasets. 
To regrid, simply pass the desired mapfile name with @samp{-r map.nc},
e.g., @samp{-r maps/map_ne120np4_to_fv257x512_aave.20150901.nc}. 
Although this should not be necessary for normal use, you may pass any
options specific to regridding with @samp{-R opt1 opt2}.

Specifying @samp{-O @var{drc_rgr}} (NB: uppercase @samp{O}) causes
@command{ncclimo} to place the regridded files in the directory
@var{drc_rgr}.  
These files have the same names as the native grid climos from which
they were derived. 
There is no namespace conflict because they are in separate
directories. 
These files also have symbolic links to their @acronym{AMWG} filenames. 
If @samp{-O @var{drc_rgr}} is not specified, @command{ncclimo} places 
all regridded files in the native grid climo output directory,
@var{drc_out}, specified by @samp{-o @var{drc_out}} (NB: lowercase 
@samp{o}). 
To avoid namespace conflicts when both climos are stored in the same
directory, the names of regridded files are suffixed by the destination
geometry string obtained from the mapfile, e.g., 
@file{*_climo_fv257x512_bilin.nc}. 
These files also have symbolic links to their @acronym{AMWG} filenames. 
@example
ncclimo -c amip_xpt -s 1980 -e 1983 -i drc_in -o drc_out
ncclimo -c amip_xpt -s 1980 -e 1983 -i drc_in -o drc_out -r map_fl
ncclimo -c amip_xpt -s 1980 -e 1983 -i drc_in -o drc_out -r map_fl -O drc_rgr
@end example
@noindent
The above commands perform a climatology without regridding, then with
regridding (all climos stored in @var{drc_out}), then with regridding and
storing regridded files separately. 
Paths specified by @var{drc_in}, @var{drc_out}, and @var{drc_rgr} may be
relative or absolute.  
An alternative to regridding during climatology generation is to regrid
afterwards with @command{ncremap}, which has more special features
built-in for regridding. 
To use @command{ncremap} to regrid a climatology in @var{drc_out} and
place the results in @var{drc_rgr}, use something like
@example
ncremap -I drc_out -m map.nc -O drc_rgr
ls drc_out/*climo* | ncremap -m map.nc -O drc_rgr
@end example
See @ref{ncremap netCDF Remapper} for more details (including
@acronym{MPAS}!).  
 
@cindex incremental climatology (climo)
@cindex binary climatology (climo)
@cindex extended climatology (climo)
@cindex previous climatology (climo)
@cindex current climatology (climo)
@unnumberedsubsec Extended Climatologies
@command{ncclimo} supports two methods for generating extended
climatologies: Binary and Incremental.
Both methods lengthen a climatology without requiring access to 
all the raw monthly data spanning the time period.
The binary method combines, with appropriate weighting, two previously
computed climatologies into a single climatology. 
No raw monthly data are employed.
The incremental method computes a climatology from raw monthly data
and (with appropriate weighting) combines that with a previously
computed climatology that ends the month prior to raw data.
The incremental method was introduced in @acronym{NCO} version 4.6.1
(released August, 2016), and the binary method was introduced in
@acronym{NCO} version 4.6.3 (released December, 2016).

@c fxm edit to distinguish extended from binary
Both methods, binary and incremental, compute the so-called ``extended
climo'' as a weighted mean of two shorter climatologies,
called the ``previous'' and ``current'' climos.
The incremental method uses the original monthly input to compute the
curent climo, which must immediately follow in time the previous climo 
which has been pre-computed.
The binary method use pre-computed climos for both the previous and
current climos, and these climos need not be sequential nor
chronological. 
Both previous and current climos for both binary and incremental methods
may be of any length (in years); their weights will be automatically
adjusted in computing the extended climo.

The use of pre-computed climos permits ongoing simulations (or lengthy
observations) to be analyzed in shorter segments combined piecemeal,
instead of requiring all raw, native-grid data to be simultaneously
accessible.
Without extended climatology capability, generating a one-hundred
year climatology requires that one-hundred years of monthly data be
available on disk. 
Disk-space requirements for large datasets may make this untenable.
Extended climo methods permits a one-hundred year climo to be
generated as the weighted mean of, say, the current ten year climatology
(weighted at 10%) combined with the pre-computed climatology of the
previous 90-years (weighted at 90%).
The 90-year climo could itself have been generated incrementally or
binary-wise, and so on. 
Climatologies occupy at most 17/(12@var{N}) the amount of space
of @var{N} years of monthly data, so the extended methods vastly
reduce disk-space requirements. 

Incremental mode is selected by specifying @samp{-S}, the start year
of the pre-computed, previous climo.
The argument to @samp{-S}) is the previous climo start year.
That, together with the current climo end year, determines the extended
climo range.
@command{ncclimo} assumes that the previous climo ends the month before
the current climo begins.
In incremental mode, @command{ncclimo} first generates the current
climatology from the current monthly input files then weights  
that current climo with the previous climo to produce the extended
climo.

Binary mode is selected by specifying both @samp{-S} and @samp{-E}, the 
end year of the pre-computed, previous climo.
In binary mode, the previous and current climatologies can be of any
length, and from any time-period, even overlapping.
Most users will run extended clmos the same way they run regular climos
in terms of parallelism and regridding, although that is not required.  
Both climos must treat Decembers same way (or else previous climo files
will not be found), and if subsetting (i.e., @samp{-v var_lst}) is
performed, then the subset must remain the same, and if nicknames (i.e.,
@samp{-f fml_nm}) are employed, then the nickname must remain the same. 

As of 20161129, the @code{climatology_bounds} attributes of extended
climos are incorrect. 
This is a work in progress...

Options:
@table @option
@cindex @code{-E @var{yr_end_prv}}
@cindex @var{yr_end_prv}
@cindex @code{--yr_end_prv}
@cindex @code{--prv_yr_end}
@cindex @code{--previous_end}
@item -E @var{yr_end_prv} (@code{--yr_end_prv}, @code{--prv_yr_end}, @code{--previous_end})
The ending year of the previous climo. 
This argument is required to trigger binary climatologies,
and should not be used for incremental climatologies.

@cindex @code{-S @var{yr_srt_prv}}
@cindex @var{yr_srt_prv}
@cindex @code{--yr_srt_prv}
@cindex @code{--prv_yr_srt}
@cindex @code{--previous_start}
@item -S @var{yr_srt_prv} (@code{--yr_srt_prv}, @code{--prv_yr_srt}, @code{--previous_start})
The starting year of the previous climo. 
This argument is required to trigger incremental climatologies,
and is also mandatory for binary climatologies.

@cindex @code{-X @var{drc_xtn}}
@cindex @var{drc_xtn}
@cindex @code{--drc_xtn}
@cindex @code{--xtn_drc}
@cindex @code{--extended}
@item -X @var{drc_xtn} (@code{--drc_xtn}, @code{--xtn_drc}, @code{--extended})
Directory in which the extended native grid climo files will be stored
for an extended climatology.
Default value is @var{drc_prv}. 
Unless a separate directory is specified (with @samp{-Y}) for the
extended climo on the analysis grid, it will be stored in @var{drc_xtn},
too.  

@cindex @code{-x @var{drc_prv}}
@cindex @var{drc_prv}
@cindex @code{--drc_prv}
@cindex @code{--prv_drc}
@cindex @code{--previous}
@item -x @var{drc_prv} (@code{--drc_prv}, @code{--prv_drc}, @code{--previous})
Directory in which the previous native grid climo files reside for an
incremental climatology. 
Default value is @var{drc_out}. 
Unless a separate directory is specified (with @samp{-y}) for the
previous climo on the analysis grid, it is assumed to reside in
@var{drc_prv}, too. 

@cindex @code{-Y @var{drc_rgr_xtn}}
@cindex @var{drc_rgr_xtn}
@cindex @code{--drc_rgr_xtn}
@cindex @code{--drc_xtn_rgr}
@cindex @code{--extended_regridded}
@cindex @code{--regridded_extended}
@item -Y @var{drc_rgr_xtn} (@code{--drc_rgr_xtn}, @code{--drc_xtn_rgr}, @code{--extended_regridded}, @code{--regridded_extended})
Directory in which the extended analysis grid climo files will be
stored in an incremental climatology. 
Default value is @var{drc_xtn}.

@cindex @code{-y @var{drc_rgr_prv}}
@cindex @var{drc_rgr_prv}
@cindex @code{--drc_rgr_prv}
@cindex @code{--drc_prv_rgr}
@cindex @code{--regridded_previous}
@cindex @code{--previous_regridded}
@item -y @var{drc_rgr_prv} (@code{--drc_rgr_prv}, @code{--drc_prv_rgr}, @code{--regridded_previous}, @code{--previous_regridded})
Directory in which the previous climo on the analysis grid resides in an
incremental climatology. 
Default value is @var{drc_prv}.
@end table
@noindent

Incremental method climatologies can be as simple as providing a start
year for the previous climo, e.g., 
@example
ncclimo -v FSNT,AODVIS -c caseid -s 1980 -e 1981 -i raw -o clm -r map.nc
ncclimo -v FSNT,AODVIS -c caseid -s 1982 -e 1983 -i raw -o clm -r map.nc -S 1980
@end example
By default @command{ncclimo} stores all native and analysis grid climos
in one directory so the above ``just works''.
There are no namespace clashes because all climos are for distinct
years, and regridded files have a suffix based on their grid resolution.  
However, there can be only one set of @acronym{AMWG} filename links
due to @acronym{AMWG} filename convention.
Thus @acronym{AMWG} filename links, if any, point to the latest extended 
climo in a given directory. 

Many researchers segregate (with @samp{-O @var{drc_rgr}}) native-grid
from analysis-grid climos. 
Incrementally generated climos must be consistent in this regard.
In other words, all climos contributing to an extended climo must have
their native-grid and analysis-grid files in the same (per-climo)
directory, or all climos must segregate their native from their analysis
grid files.
Do not segregate the grids in one climo, and combine them in another.
Such climos cannot be incrementally aggregated.
Thus incrementing climos can require from zero to four additional
options that specify all the previous and extended climatologies for
both native and analysis grids.
The example below constructs the current climo in @var{crr},
then combines the weighted average of that with the previous climo in
@var{prv}, and places the resulting extended climatology in @var{xtn}.
Here the native and analysis climos are combined in one directory per 
climo:
@example
ncclimo -v FSNT,AODVIS -c caseid -s 1980 -e 1981 -i raw -o prv -r map.nc
ncclimo -v FSNT,AODVIS -c caseid -s 1982 -e 1983 -i raw -o clm -r map.nc \
        -S 1980 -x prv -X xtn
@end example
If the native and analysis grid climo directories are segregated, 
then those directories must be specified, too:
@example
ncclimo -v FSNT,AODVIS -c caseid -s 1980 -e 1981 -i raw -o prv -O rgr_prv -r map.nc
ncclimo -v FSNT,AODVIS -c caseid -s 1982 -e 1983 -i raw -o clm -O rgr -r map.nc \
        -S 1980 -x prv -X xtn -y rgr_prv -Y rgr_xtn
@end example

@command{ncclimo} does not know whether a pre-computed climo is on a
native grid or an analysis grid, i.e., whether it has been regridded.
In binary mode, @command{ncclimo} may be pointed to two pre-computed
native grid climatologies, or to two pre-computed analysis grid
climatologies.
In other words, it is not necessary to maintain native grid
climatologies for use in creating extended climatologies.
It is sufficient to generate climatologies on the analysis grid, and
feed them to @command{ncclimo} in binary mode, without a mapping file:
@example
ncclimo -c caseid -S 1980 -E 1981 -x prv -s 1980 -e 1981 -i crr -o clm 
@end example

@unnumberedsubsec Coupled Runs
@command{ncclimo} works on all @acronym{E3SM/ACME} and @acronym{CESM} models. 
It can simultaneously generate climatologies for a coupled run, where
climatologies mean both native and regridded monthly, seasonal, and
annual averages as per @acronym{E3SM/ACME} specifications (which mandate
the inclusion of certain helpful metadata and provenance information).
Here are template commands for a recent simulation:
@example
@verbatim
caseid=20160121.A_B2000ATMMOD.ne30_oEC.titan.a00
drc_in=/scratch/simulations/$caseid/run
drc_out=${DATA}/acme
map_atm=${DATA}/maps/map_ne30np4_to_fv129x256_aave.20150901.nc
map_lnd=$map_atm
map_ocn=${DATA}/maps/map_oEC60to30_to_t62_bilin.20160301.nc
map_ice=$map_ocn
ncclimo -p mpi -c $caseid -m cam  -s 2 -e 5 -i $drc_in -r $map_atm -o $drc_out/atm
ncclimo        -c $caseid -m clm2 -s 2 -e 5 -i $drc_in -r $map_lnd -o $drc_out/lnd
ncclimo -p mpi -m mpaso           -s 2 -e 5 -i $drc_in -r $map_ocn -o $drc_out/ocn 
ncclimo        -m mpassi          -s 2 -e 5 -i $drc_in -r $map_ice -o $drc_out/ice
@end verbatim
@end example
Atmosphere and ocean model output is typically larger than land and ice
model output.  
These commands recognize that by using different parallelization
strategies that may (@file{rhea} standard queue) or may not
(@file{cooley}, or @file{rhea}'s @code{bigmem} queue) be required,
depending on the fatness of the analysis nodes, as explained below. 

@unnumberedsubsec Memory Considerations
It is important to employ the optimal @command{ncclimo} parallelization
strategy for your computer hardware resources. 
Select from the three available choices with the 
@option{-p @var{par_typ}} switch.
The options are serial mode (@samp{-p srl}, @samp{-p serial}, or @samp{-p nil}), 
background mode parallelism (@samp{-p bck}, or @samp{-p background}),
and @acronym{MPI} parallelism (@samp{-p mpi} or @samp{-p MPI}).
The default is background-mode parallelism.
This is appropriate for lower resolution (e.g., ne30L30) simulations on
most nodes at high-performance computer centers. 
Use (or at least start with) serial mode on personal
laptops/workstations.   
Serial mode requires twelve times less @acronym{RAM} than the parallel
modes, and is much less likely to deadlock or cause @acronym{OOM}
(out-of-memory) conditions on your personal computer. 
If the available @acronym{RAM} (plus swap) is 
@math{< 12*4*}@code{sizeof(}monthly input file@code{)}, then try serial
mode first (12 is the optimal number of parallel processes for monthly
climos, the computational overhead is a factor of four). 
@acronym{EAM-SE} ne30L30 output is about @w{1 GB/month} so each month
requires about @w{4 GB} of @acronym{RAM}. 
@acronym{EAM-SE} ne30L72 output (with @acronym{LINOZ}) is about 
@w{10 GB/month} so each month requires about @w{40 GB} @acronym{RAM}. 
@acronym{EAM-SE} ne120 output is about @w{12 GB/month} so each month
requires about @w{48 GB} @acronym{RAM}. 
The computer does not actually use all this memory at one time, and many
kernels compress @acronym{RAM} usage to below what top reports, so the
actual physical usage is hard to pin-down, but may be a factor of
2.5--3.0 (rather than a factor of four) times the size of the input
file. 
For instance, my @w{16 GB} 2014 MacBookPro successfully runs an ne30L30 
climatology (that requests @w{48 GB} @acronym{RAM}) in background mode. 
However the laptop is slow and unresponsive for other uses until it
finishes (in 6--8 minutes) the climos. 
Experiment and choose the parallelization option that performs best.

Serial-mode, as its name implies, uses one core at a time for climos,
and proceeds sequentially from months to seasons to annual
climatologies. 
Serial mode means that climos are performed serially, while regridding
still employs OpenMP threading (up to @w{16 cores}) on platforms that
support it. 
By design each month and each season is independent of the others, so
all months can be computed in parallel, then each season can be computed
in parallel (using monthly climatologies), from which annual average is
computed. 
Background parallelization mode exploits this parallelism and executes
the climos in parallel as background processes on a single node, so that
twelve cores are simultaneously employed for monthly climatologies, four
for seasonal, and one for annual. 
The optional regridding will employ, by default, up to two cores per
process. 
The @acronym{MPI} parallelism mode executes the climatologies on
different nodes so that up to (optimally) twelve nodes compute monthly 
climos. 
The full memory of each node is available for each individual climo. 
The optional regridding employs, by default, up to eight cores per node
in @acronym{MPI}-mode.
@acronym{MPI}-mode or serial-mode must be used to process ne30L72 and
ne120L30 climos on all but the fattest @acronym{DOE} nodes. 
An ne120L30 climo in background mode on @file{rhea} (i.e., on one @w{128
GB} compute node) fails due to @acronym{OOM}. 
(Unfortunately @acronym{OOM} errors do not produce useful return codes
so if your climo processes die without printing useful information, the
cause may be @acronym{OOM}). 
However the same climo in background-mode succeeds when executed on a
single big-memory (@w{1 TB}) node on @file{rhea} (use
@samp{-lpartition=gpu}, as shown below). 
Or @acronym{MPI}-mode can be used for any climatology. 
The same ne120L30 climo will also finish blazingly fast in background
mode on @file{cooley} (i.e., on one @w{384 GB} compute node), so
@acronym{MPI}-mode is unnecessary on @file{cooley}. 
In general, the fatter the memory, the better the performance.  

@unnumberedsubsec Single, Dedicated Nodes at @acronym{LCF}s
The basic approach above (running the script from a standard terminal
window) that works well for small cases can be unpleasantly slow on
login nodes of @acronym{LCF}s and for longer or higher resolution (e.g.,
ne120) climatologies. 
As a baseline, generating a climatology of @w{5 years} of ne30 (~1x1
degree) @acronym{EAM-SE} output with @command{ncclimo} takes 1--2
minutes on @file{rhea} (at a time with little contention), and 6--8
minutes on a 2014 MacBook Pro. 
To make things a bit faster at @acronym{LCF}s, request a dedicated node
(this only makes sense on supercomputers or clusters with
job-schedulers).  
On @file{rhea} or @file{titan}, which use the @acronym{PBS} scheduler,
do this with 
@example
# Standard node (128 GB), PBS scheduler
qsub -I -A CLI115 -V -l nodes=1 -l walltime=00:10:00 -N ncclimo
# Bigmem node (1 TB), PBS scheduler
qsub -I -A CLI115 -V -l nodes=1 -l walltime=00:10:00 -lpartition=gpu -N ncclimo
@end example
The equivalent requests on @file{cooley} or @file{mira} (Cobalt
scheduler) and @file{cori} or @file{titan} (@acronym{SLURM} scheduler)
are:  
@example
# Cooley node (384 GB) with Cobalt
qsub -I -A HiRes_EarthSys --nodecount=1 --time=00:10:00 --jobname=ncclimo 
# Cori node (128 GB) with SLURM
salloc  -A acme --nodes=1 --partition=debug --time=00:10:00 --job-name=ncclimo
@end example
@noindent
Flags used and their meanings:
@table @option
@item -I 
Submit in interactive mode. 
This returns a new terminal shell rather than running a program. 
@item --time
How long to keep this dedicated node for. 
Unless you kill the shell created by the @command{qsub} command, the
shell will exist for this amount of time, then die suddenly. 
In the above examples, @w{10 minutes} is requested. 
@item -l nodes=1 
@acronym{PBS} syntax (e.g., on @file{rhea}) for nodes.
@item --nodecount 1 
Cobalt syntax (e.g., on @file{cooley}) for nodes.
@item --nodes=1
@acronym{SLURM} syntax (e.g., on @file{cori} or @file{edison}) for nodes.
These scheduler-dependent variations request a quantity of nodes. 
Request @w{1 node} for Serial or Background-mode, and up to @w{12 nodes} 
for @acronym{MPI}-mode parallelism.
In all cases @command{ncclimo} will use multiple cores per node if
available. 
@item -V
Export existing environmental variables into the new interactive shell.
This may not actually be needed.  
@item -q name
Queue name.
This is needed for locations like @file{edison} that have multiple
queues with no default queue. 
@item -A 
Name of account to charge for time used. 
@end table
Acquiring a dedicated node is useful for any workflow, not just creating climos.
This command returns a prompt once nodes are assigned (the prompt is
returned in your home directory so you may then have to @command{cd} to
the location you meant to run from). 
Then run your code with the basic @command{ncclimo} invocation.
The is faster because the node is exclusively dedicated to @command{ncclimo}.
Again, ne30L30 climos only require @w{< 2} minutes, so the 
@w{10 minutes} requested in the example is excessive and conservative.  
Tune it with experience. 

@unnumberedsubsec @w{12 node} @acronym{MPI}-mode Jobs
The above parallel approaches will fail when a single node lacks enough 
@acronym{RAM} (plus swap) to store all twelve monthly input files, plus
extra @acronym{RAM} for computations. 
One should employ @acronym{MPI} multinode parallelism @samp{-p mpi}
on nodes with less @acronym{RAM} than
@math{12*3*}@code{sizeof(}input@code{)}. 
The longest an ne120 climo will take is less than half an hour (~25
minutes on @file{edison} or @file{rhea}), so the simplest method to run
@acronym{MPI} jobs is to request 12-interactive nodes using the above
commands (though remember to add @samp{-p mpi}), then execute the script
at the command line. 

It is also possible, and sometimes preferable, to request
non-interactive compute nodes in a batch queue. 
Executing an @acronym{MPI}-mode climo (on machines with job scheduling
and, optimally, @w{12 nodes}) in a batch queue can be done in two
commands. 
First, write an executable file which calls the @command{ncclimo} script
with appropriate arguments. 
We do this below by echoing to a file, @file{ncclimo.pbs}.
@example
@verbatim
echo "ncclimo -p mpi -c $caseid -s 1 -e 20 -i $drc_in -o $drc_out" > ncclimo.pbs
@end verbatim
@end example
The only new argument here is @samp{-p mpi} that tells @command{ncclimo}
to use @acronym{MPI} parallelism. 
Then execute this command file with a @w{12 node} non-interactive job: 
@example
qsub -A CLI115 -V -l nodes=12 -l walltime=00:30:00 -j oe -m e -N ncclimo \
     -o ncclimo.out ncclimo.pbs
@end example
This script adds new flags:
@samp{-j oe} (combine output and error streams into standard error),
@samp{-m e} (send email to the job submitter when the job ends),
@samp{-o ncclimo.out} (write all output to @file{ncclimo.out}).
The above commands are meant for @acronym{PBS} schedulers like on
@file{rhea}.  
Equivalent commands for @file{cooley}/@file{mira} (Cobalt) and
@file{cori}/@file{edison} (@acronym{SLURM}) are
@example
@verbatim
# Cooley (Cobalt scheduler)
/bin/rm -f ncclimo.err ncclimo.out
echo '#!/bin/bash' > ncclimo.cobalt
echo "ncclimo -p mpi -c $caseid -s 1 -e 20 -i $drc_in -o $drc_out" >> ncclimo.cobalt
chmod a+x ncclimo.cobalt
qsub -A HiRes_EarthSys --nodecount=12 --time=00:30:00 --jobname ncclimo \
     --error ncclimo.err --output ncclimo.out --notify zender@uci.edu ncclimo.cobalt

# Cori/Edison (SLURM scheduler)
echo "ncclimo -p mpi -c $caseid -s 1 -e 20 -i $drc_in -o $drc_out -r $map_fl" \
      > ncclimo.pbs
chmod a+x ncclimo.slurm
sbatch -A acme --nodes=12 --time=03:00:00 --partition=regular --job-name=ncclimo \
       --mail-type=END --error=ncclimo.err --output=ncclimo.out ncclimo.slurm
@end verbatim
@end example
Notice that Cobalt and @acronym{SLURM} require the introductory
shebang-interpreter line (@code{#!/bin/bash}) which @acronym{PBS} does
not need.  
Set only the scheduler batch queue parameters mentioned above. 
In @acronym{MPI}-mode, @command{ncclimo} determines the appropriate
number of tasks-per-node based on the number of nodes available and
script internals (like load-balancing for regridding). 
Hence do not set a tasks-per-node parameter with scheduler configuration
parameters as this could cause conflicts.

@unnumberedsubsec What does @command{ncclimo} do?
For monthly climatologies (e.g., @acronym{JAN}), @command{ncclimo} passes
the list of all relevant January monthly files to @acronym{NCO}'s
@command{ncra} command, which averages each variable in these monthly
files over their time-dimension (if it exists) or copies the value from
the first month unchanged (if no time-axis exists). 
Seasonal climos are then created by taking the average of the monthly
climo files using @command{ncra}. 
To account for differing numbers of days per month, the @command{ncra}
@samp{-w} flag is used, followed by the number of days in the relevant
months. 
For example, the @acronym{MAM} climo is computed with 
@samp{ncra -w 31,30,31 MAR_climo.nc APR_climo.nc MAY_climo.nc MAM_climo.nc} 
(details about file names and other optimization flags have been
stripped here to make the concept easier to follow). 
The annual (@acronym{ANN}) climo is then computed as a weighted
average of the seasonal climos.

@unnumberedsubsec Assumptions, Approximations, and Algorithms (@acronym{AAA}) Employed: 
A climatology embodies many algorithmic choices, and regridding from the
native to the analysis grid involves still more choices. 
A separate method should reproduce the @command{ncclimo} and
@acronym{NCO} answers to round-off precision if it implements the same
algorithmic choices. 
For example, @command{ncclimo} agrees to round-off with @acronym{AMWG}
diagnostics when making the same (sometimes questionable) choices. 
The most important choices have to do with converting single- to
double-precision (@acronym{SP} and @acronym{DP}, respectively),
treatment of missing values, and generation/application of regridding
weights. 
For concreteness and clarity we describe the algorithmic choices made in
processing a @acronym{EAM-SE} monthly output into a climatological
annual mean (@acronym{ANN}) and then regridding that. 
Other climatologies (e.g., daily to monthly, or
annual-to-climatological) involve similar choices. 

@acronym{E3SM/ACME} (and @acronym{CESM}) computes fields in @acronym{DP} and
outputs history (not restart) files as monthly means in @acronym{SP}. 
The @acronym{NCO} climatology generator (@command{ncclimo}) processes
these data in four stages. 
Stage @var{N} accesses input only from stage @math{@var{N-1}}, never
from stage @math{@var{N-2}} or earlier. 
Thus the (on-disk) files from stage @var{N} determine the highest
precision achievable by stage @math{@var{N+1}}. 
The general principal is to perform math (addition, weighting,
normalization) in @acronym{DP} and output results to disk in the same
precision in which they were input from disk (usually @acronym{SP}). 
In @w{Stage 1}, @acronym{NCO} ingests @w{Stage 0} monthly means (raw
@acronym{EAM-SE} output), converts @acronym{SP} input to @acronym{DP},
performs the average across all years, then converts the answer from
@acronym{DP} to @acronym{SP} for storage on-disk as the climatological
monthly mean. 
In @w{Stage 2}, @acronym{NCO} ingests @w{Stage 1} climatological monthly
means, converts @acronym{SP} input to @acronym{DP}, performs the average
across all months in the season (e.g., @acronym{DJF}), then converts the  
answer from @acronym{DP} to @acronym{SP} for storage on-disk as the
climatological seasonal mean. 
In @w{Stage 3}, @acronym{NCO} ingests @w{Stage 2} climatological
seasonal means, converts @acronym{SP} input to @acronym{DP}, performs
the average across all four seasons (@acronym{DJF}, @acronym{MAM},
@acronym{JJA}, @acronym{SON}), then converts the answer from
@acronym{DP} to @acronym{SP} for storage on-disk as the climatological
annual mean.  

@w{Stage 2} weights each input month by its number of days (e.g., 31 for 
January), and @w{Stage 3} weights each input season by its number of days 
(e.g., 92 for @acronym{MAM}). 
@acronym{E3SM/ACME} runs @acronym{EAM-SE} with a 365-day calendar, so these
weights are independent of year and never change. 
The treatment of missing values in @w{Stages 1--3} is limited by the
lack of missing value tallies provided by @w{Stage 0} (model) output. 
@w{Stage 0} records a value as missing if it is missing for the entire
month, and present if the value is valid for one or more timesteps. 
@w{Stage 0} does not record the missing value tally (number of valid
timesteps) for each spatial point. 
Thus a point with a single valid timestep during a month is weighted the
same in @w{Stages 1--4} as a point with 100% valid timesteps during the
month. 
The absence of tallies inexorably degrades the accuracy of subsequent
statistics by an amount that varies in time and space. 
On the positive side, it reduces the output size (by a factor of two) 
and complexity of analyzing fields that contain missing values. 
Due to the ambiguous nature of missing values, it is debatable whether
they merit efforts to treat them more exactly. 

The vast majority of fields undergo three promotion/demotion cycles 
between @acronym{EAM-SE} and @acronym{ANN}. 
No promotion/demotion cycles occur for history fields that
@acronym{EAM-SE} outputs in @acronym{DP} rather than @acronym{SP}, nor
for fields without a time dimension. 
Typically these fields are grid coordinates (e.g., longitude, latitude) 
or model constants (e.g., 
@set flg
@tex
$\cod$
@clear flg
@end tex
@ifinfo
CO2
@clear flg
@end ifinfo
@ifset flg
CO2
@clear flg
@end ifset
mixing ratio). 
@acronym{NCO} never performs any arithmetic on grid coordinates or
non-time-varying input, regardless of whether they are @acronym{SP} or
@acronym{DP}. 
Instead, @acronym{NCO} copies these fields directly from the first
input file.  
@w{Stage 4} uses a mapfile to regrid climos from the native to the
desired analysis grid. 
@acronym{E3SM/ACME} currently uses mapfiles generated by
@command{ESMF_RegridWeightGen} (@acronym{ERWG}) and by TempestRemap.  

The algorithmic choices, approximations, and commands used to generate
mapfiles from input gridfiles are separate issues.
We mention only some of these issues here for brevity.
Input gridfiles used by @acronym{E3SM/ACME} until ~20150901, and by
@acronym{CESM} (then and currently, at least for Gaussian grids)
contained flaws that effectively reduced their precision, especially at
regional scales, and especially for Gaussian grids. 
@acronym{E3SM/ACME} (and @acronym{CESM}) mapfiles continue to approximate
grids as connected by great circles, whereas most analysis grids (and
some models) use great circles for longitude and small circles for
latitude. 
The great circle assumption may be removed in the future. 
Constraints imposed by @acronym{ERWG} during weight-generation ensure
that global integrals of fields undergoing conservative regridding are
exactly conserved. 

Application of weights from the mapfile to regrid the native data to the
analysis grid is straightforward. 
Grid fields (e.g., latitude, longitude, area) are not regridded. 
Instead they are copied (and area is reconstructed if absent) directly
from the mapfile. 
@acronym{NCO} ingests all other native grid (source) fields, converts
@acronym{SP} to @acronym{DP}, and accumulates destination gridcell
values as the sum of the @acronym{DP} weight (from the sparse matrix in
the mapfile) times the (usually @acronym{SP}-promoted-to-@acronym{DP})
source values. 
Fields without missing values are then stored to disk in their
original precision. 
Fields with missing values are treated (by default) with what
@acronym{NCO} calls the ``conservative'' algorithm. 
This algorithm uses all valid data from the source grid on the
destination grid once and only once. 
Destination cells receive the weighted valid values of the source
cells. 
This is conservative because the global integrals of the source and
destination fields are equal. 
See @ref{ncremap netCDF Remapper} for more description of the
conservative and of the optional (``renormalized'') algorithm. 

@noindent
@html
<a name="xmp_ncclimo"></a> <!-- http://nco.sf.net/nco.html#xmp_ncclimo -->
@end html
EXAMPLES

@html
<a name="merra2"></a> <!-- http://nco.sf.net/nco.html#merra2 -->
@end html
How to create a climo from a collection of monthly
non-@acronym{CESM}'ish files?  
This is a two-step procedure:
First be sure the names are arranged with a @acronym{YYYYMM}-format date
preceding the suffix (usually @samp{.nc}).
Then give @emph{any} monthly input filename to @command{ncclimo}.
Consider the @acronym{MERRA2} collection, for example.
As retrieved from @acronym{NASA}, @acronym{MERRA2} files have names like
@file{svc_MERRA2_300.tavgM_2d_aer_Nx.200903.nc4}. 
While the sub-string @samp{200903} is easy to recognize as a month in
@acronym{YYYYMM} format, other parts (specifically the @samp{300} code)
of the filename also change with date. 
We can use Bash regular expressions to extract dates and create symbolic
links to simpler filenames with regularly patterned @acronym{YYYYMM}
strings like @file{merra2_200903.nc4}:
@example
@verbatim
for fl in `ls *.nc4` ; do
# Convert svc_MERRA2_300.tavgM_2d_aer_Nx.YYYYMM.nc4 to merra2_YYYYMM.nc4
    sfx_out=`expr match "${fl}" '.*_Nx.\(.*.nc4\)'`
    fl_out="merra2_${sfx_out}"
    ln -s ${fl} ${fl_out}
done
@end verbatim
@end example
Then call @command{ncclimo} with any standard format filename, e.g., 
@file{merra2_200903.nc4}, as as the @var{caseid}: 
@example
ncclimo -c merra2_200903.nc4 -s 1980 -e 2016 -i $drc_in -o $drc_out
@end example

@html
<a name="ecmwf_ifs"></a> <!-- http://nco.sf.net/nco.html#ecmwf_ifs -->
@end html
In the default monthly climo generation mode, @command{ncclimo} expects 
each input file to contain one single record that is the monthly average 
of all fields.
Another example of of wrangling observed datasets into a
@acronym{CESM}ish format is @acronym{ECMWF} Integrated Forecasting
System (@acronym{IFS}) output that contains twelve months per file,
rather than the one month per file that @command{ncclimo} expects.
@example
@verbatim
for yr in {1979..2016}; do
# Convert ifs_YYYY01-YYYY12.nc to ifs_YYYYMM.nc
    yyyy=`printf "%04d" $yr`
    for mth in {1..12}; do
        mm=`printf "%02d" $mth`
        ncks -O -F -d time,${mth} ifs_${yyyy}01-${yyyy}12.nc ifs_${yyyy}${mm}.nc
    done
done
@end verbatim
@end example
Then call @command{ncclimo} with @file{ifs_197901.nc} as @var{caseid}:  
@example
ncclimo -c ifs_197901.nc -s 1979 -e 2016 -i $drc_in -o $drc_out
@end example
@acronym{ncclimo} does not recognize all combinations imaginable of
records per file and files per year. 
However, support can be added for the most prevalent combinations 
so that @acronym{ncclimo}, rather than the user, does any necessary data
wrangling.
Contact us if there is a common input data format you would like
supported as a custom option.

Often one wishes to create a climatology of a single variable.
The @samp{-f @var{fml_nm}} option to @command{ncclimo} makes this easy.
Consider a series of single-variable climos for the fields @code{FSNT}, 
and @code{FLNT}
@example
@verbatim
ncclimo -v FSNT -f FSNT -c amip_xpt -s 1980 -e 1983 -i drc_in -o drc_out
ncclimo -v FLNT -f FLNT -c amip_xpt -s 1980 -e 1983 -i drc_in -o drc_out
@end verbatim
@end example
These climos use the @samp{-f} option and so their output files will
have no namespace conflicts. 
Moreover, the climatologies can be generated in parallel.

@page
@html
<a name="ncecat"></a> <!-- http://nco.sf.net/nco.html#ncecat -->
@end html
@node ncecat netCDF Ensemble Concatenator, nces netCDF Ensemble Statistics, ncclimo netCDF Climatology Generator, Reference Manual
@section @command{ncecat} netCDF Ensemble Concatenator
@cindex concatenation
@cindex ensemble concatenation
@findex ncecat

@noindent
SYNTAX
@example
ncecat [-3] [-4] [-5] [-6] [-7] [-A] [-C] [-c] [--cmp @var{cmp_sng}]
[--cnk_byt @var{sz_byt}] [--cnk_csh @var{sz_byt}] [--cnk_dmn @var{nm},@var{sz_lmn}]
[--cnk_map @var{map}] [--cnk_min @var{sz_byt}] [--cnk_plc @var{plc}] [--cnk_scl @var{sz_lmn}]
[-D @var{dbg}] [-d @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]] [-F] [--fl_fmt @var{fl_fmt}]
[-G @var{gpe_dsc}] [-g @var{grp}[,@dots{}]] [--gag] [--glb ...]
[-H] [-h] [--hdf] [--hdr_pad @var{nbr}] [--hpss] 
[-L @var{dfl_lvl}] [-l @var{path}] [-M] [--md5_digest] [--mrd] [-n @var{loop}]
[--no_cll_msr] [--no_frm_trm] [--no_tmp_fl] 
[-O] [-o @var{output-file}] [-p @var{path}] [--ppc ...] [-R] [-r] [--ram_all] 
[-t @var{thr_nbr}] [-u @var{ulm_nm}] [--unn] [-v @var{var}[,@dots{}]] [-X ...] [-x] 
[@var{input-files}] [@var{output-file}]
@end example

@noindent
DESCRIPTION

@command{ncecat} aggregates an arbitrary number of input files into a 
single output file using using one of two methods.
@dfn{Record AGgregation} (@acronym{RAG}), the traditional method employed on
(flat) netCDF3 files and still the default method, stores
@var{input-files} as consecutive records in the @var{output-file}.
@dfn{Group AGgregation} (@acronym{GAG}) stores @var{input-files} as top-level
groups in the netCDF4 @var{output-file}.
Record Aggregation (@acronym{RAG}) makes numerous assumptions about the
structure of input files whereas Group Aggregation (@acronym{GAG}) makes
none. 
Both methods are described in detail below.
Since @command{ncecat} aggregates all the contents of the input files,
it can easily produce large output files so it is often helpful to invoke
subsetting simultaneously (@pxref{Subsetting Files}).

@html
<a name="rag"></a> <!-- http://nco.sf.net/nco.html#rag -->
@end html
@cindex record aggregation
@cindex @acronym{RAG}
@acronym{RAG} makes each variable (except coordinate variables) in each
input file into a single record of the same variable in the output file.  
Coordinate variables are not concatenated, they are instead simply
copied from the first input file to the @var{output-file}.
All @var{input-files} must contain all extracted variables (or else
there would be ``gaps'' in the output file).

A new record dimension is the glue which binds together the input file
data.
The new record dimension is defined in the root group of the output file
so it is visible to all sub-groups.
Its name is, by default, ``record''.
@cindex unlimited dimension
@cindex record dimension
@cindex @samp{-u @var{ulm_nm}}
@cindex @samp{--ulm_nm @var{ulm_nm}}
@cindex @samp{--rcd_nm @var{ulm_nm}}
This default name can be overridden with the @samp{-u @var{ulm_nm}}
short option (or the @samp{--ulm_nm} or @samp{rcd_nm} long options).

Each extracted variable must be constant in size and rank across all
@var{input-files}. 
@cindex record dimension
@cindex hyperslab
The only exception is that @command{ncecat} allows files to differ in
the record dimension size if the requested record hyperslab
(@pxref{Hyperslabs}) resolves to the same size for all files. 
@cindex @acronym{CMIP}
This allows easier gluing/averaging of unequal length timeseries from 
simulation ensembles (e.g., the @acronym{CMIP} archive). 

@cindex fixed dimension
@cindex fix record dimension
Classic (i.e., all netCDF3 and @code{NETCDF4_CLASSIC}) output files
can contain only one record dimension.
@command{ncecat} makes room for the new glue record dimension by
changing the pre-existing record dimension, if any, in the input files
into a fixed dimension in the output file. 
netCDF4 output files may contain any number of record dimensions, so
@command{ncecat} need not and does not alter the record dimensions,
if any, of the input files as it copies them to the output file. 

@html
<a name="gag"></a> <!-- http://nco.sf.net/nco.html#gag -->
@end html
@cindex group aggregation
@cindex @acronym{GAG}
@dfn{Group AGgregation} (@acronym{GAG}) stores @var{input-files} as
top-level groups in the @var{output-file}.
No assumption is made about the size or shape or type of a given 
object (variable or dimension or group) in the input file.
The entire contents of the extracted portion of each input file
is placed in its own top-level group in @var{output-file}, which
is automatically made as a netCDF4-format file.

@cindex @option{--gag}
@acronym{GAG} has two methods to specify group names for the
@var{output-file}.    
The @samp{-G} option, or its long-option equivalent @samp{--gpe},
takes as argument a group path editing description @var{gpe_dsc} of
where to place the results.
Each input file needs a distinct output group name to avoid namespace
conflicts in the @var{output-file}. 
Hence @command{ncecat} automatically creates unique output group names
based on either the input filenames or the @var{gpe_dsc} arguments.
When the user provides @var{gpe_dsc} (i.e., with @samp{-G}), then the
output groups are formed by enumerating sequential two-digit numeric
suffixes starting with zero, and appending them to the specified group
path (@pxref{Group Path Editing}).
When @var{gpe_dsc} is not provided (i.e., user requests @acronym{GAG} with
@samp{--gag} instead of @samp{-G}), then @command{ncecat} forms the
output groups by stripping the input file name of any type-suffix
(e.g., @code{.nc}), and all but the final component of the full
filename. 
@example
ncecat --gag 85.nc 86.nc 87.nc 8587.nc # Output groups 85, 86, 87
ncecat -G 85_ a.nc b.nc c.nc 8589.nc # Output groups 85_00, 85_01, 85_02
ncecat -G 85/ a.nc b.nc c.nc 8589.nc # Output groups 85/00, 85/01, 85/02
@end example

With both @acronym{RAG} and @acronym{GAG} the @var{output-file} size is
the sum of the sizes of the extracted variables in the input files. 
@xref{Statistics vs Concatenation}, for a description of the
distinctions between the various statistics tools and concatenators. 
@cindex multi-file operators
@cindex standard input
@cindex @code{stdin}
As a multi-file operator, @command{ncecat} will read the list of
@var{input-files} from @code{stdin} if they are not specified 
as positional arguments on the command line 
(@pxref{Large Numbers of Files}).

@cindex @code{-M}
@cindex @code{--no_glb_mtd}
@cindex @code{--suppress_global_metadata}
@cindex @code{history}
@cindex provenance
@cindex metadata, global 
Suppress global metadata copying.
By default @acronym{NCO}'s multi-file operators copy the global metadata
from the first input file into @var{output-file}.  
This helps to preserve the provenance of the output data.
However, the use of metadata is burgeoning and sometimes one
encounters files with excessive amounts of extraneous metadata.
Extracting small bits of data from such files leads to output files
which are much larger than necessary due to the automatically copied
metadata.
@command{ncecat} supports turning off the default copying of global
metadata via the @samp{-M} switch (or its long option equivalents,
@samp{--no_glb_mtd} and @samp{--suppress_global_metadata}). 

@cindex climate model
Consider five realizations, @file{85a.nc}, @file{85b.nc}, 
@w{@dots{} @file{85e.nc}} of 1985 predictions from the same climate
model. 
Then @code{ncecat 85?.nc 85_ens.nc} glues together the individual
realizations into the single file, @file{85_ens.nc}. 
If an input variable was dimensioned [@code{lat},@code{lon}], it will
by default have dimensions [@code{record},@code{lat},@code{lon}] in
the output file. 
@w{A restriction} of @command{ncecat} is that the hyperslabs of the
processed variables must be the same from file to file.
Normally this means all the input files are the same size, and contain 
data on different realizations of the same variables.

@findex ncpdq
@cindex packing
@cindex unpacking
@cindex @code{add_offset}
@cindex @code{scale_factor}
Concatenating a variable packed with different scales across multiple
datasets is beyond the capabilities of @command{ncecat} (and
@command{ncrcat}, the other concatenator (@ref{Concatenation}).
@command{ncecat} does not unpack data, it simply @emph{copies} the data
from the @var{input-files}, and the metadata from the @emph{first}
@var{input-file}, to the @var{output-file}. 
This means that data compressed with a packing convention must use
the identical packing parameters (e.g., @code{scale_factor} and
@code{add_offset}) for a given variable across @emph{all} input files.
Otherwise the concatenated dataset will not unpack correctly.
The workaround for cases where the packing parameters differ across
@var{input-files} requires three steps:
First, unpack the data using @command{ncpdq}.
Second, concatenate the unpacked data using @command{ncecat}, 
Third, re-pack the result with @command{ncpdq}.

@noindent
@html
<a name="xmp_ncecat"></a> <!-- http://nco.sf.net/nco.html#xmp_ncecat -->
@end html
EXAMPLES

Consider a model experiment which generated five realizations of one
year of data, say 1985.
You can imagine that the experimenter slightly perturbs the
initial conditions of the problem before generating each new solution.  
Assume each file contains all twelve months (a seasonal cycle) of data
and we want to produce a single file containing all the seasonal
cycles. 
Here the numeric filename suffix denotes the experiment number
(@emph{not} the month):
@example
ncecat 85_01.nc 85_02.nc 85_03.nc 85_04.nc 85_05.nc 85.nc
ncecat 85_0[1-5].nc 85.nc
ncecat -n 5,2,1 85_01.nc 85.nc
@end example
@noindent
These three commands produce identical answers.
@xref{Specifying Input Files}, for an explanation of the distinctions
between these methods.
The output file, @file{85.nc}, is five times the size as a single
@var{input-file}. 
It contains @w{60 months} of data.

@html
<a name="ncecat_rnm"></a> <!-- http://nco.sf.net/nco.html#ncecat_rnm -->
@end html
One often prefers that the (new) record dimension have a more
descriptive, context-based name than simply ``record''. 
This is easily accomplished with the @samp{-u @var{ulm_nm}} switch.
To add a new record dimension named ``time'' to all variables
@example
ncecat -u time in.nc out.nc
@end example
To glue together multiple files with a new record variable named
``realization'' 
@example
ncecat -u realization 85_0[1-5].nc 85.nc
@end example
@noindent
Users are more likely to understand the data processing history when
such descriptive coordinates are used. 

@html
<a name="dmn_rcd_rm"></a> <!-- http://nco.sf.net/nco.html#dmn_rcd_rm -->
@end html
@cindex record dimension
@cindex fixed dimension
@cindex fix record dimension
@cindex @code{--mk_rec_dmn @var{dim}}
Consider a file with an existing record dimension named @code{time}. 
and suppose the user wishes to convert @code{time} from a record
dimension to a non-record dimension.
This may be useful, for example, when the user has another use for the
record variable.
The simplest method is to use @samp{ncks --fix_rec_dmn}, and another
possibility is to use @command{ncecat} followed by 
@command{ncwa}: 
@cindex degenerate dimension
@example
ncecat in.nc out.nc # Convert time to non-record dimension
ncwa -a record in.nc out.nc # Remove new degenerate record dimension
@end example
@noindent
The second step removes the degenerate record dimension.
See @ref{ncpdq netCDF Permute Dimensions Quickly} and
@ref{ncks netCDF Kitchen Sink} for other methods of
of changing variable dimensionality, including the record dimension.

@page
@html
<a name="nces"></a> <!-- http://nco.sf.net/nco.html#nces -->
<a name="ncea"></a> <!-- http://nco.sf.net/nco.html#ncea -->
@end html
@node nces netCDF Ensemble Statistics, ncflint netCDF File Interpolator, ncecat netCDF Ensemble Concatenator, Reference Manual
@section @command{nces} netCDF Ensemble Statistics
@cindex averaging data
@cindex ensemble average
@findex nces

@noindent
SYNTAX
@example
nces [-3] [-4] [-5] [-6] [-7] [-A] [-C] [-c] [--cb @var{y1},@var{y2},@var{m1},@var{m2},@var{tpd}]
[--cmp @var{cmp_sng}] [--cnk_byt @var{sz_byt}] [--cnk_csh @var{sz_byt}] [--cnk_dmn @var{nm},@var{sz_lmn}]
[--cnk_map @var{map}] [--cnk_min @var{sz_byt}] [--cnk_plc @var{plc}] [--cnk_scl @var{sz_lmn}]
[-D @var{dbg}] [-d @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]] [-F]
[-G @var{gpe_dsc}] [-g @var{grp}[,@dots{}]] [--glb ...]
[-H] [-h] [--hdf] [--hdr_pad @var{nbr}] [--hpss] 
[-L @var{dfl_lvl}] [-l @var{path}] [-n @var{loop}]
[--no_cll_msr] [--no_frm_trm] [--no_tmp_fl] [--nsm_fl|grp] [--nsm_sfx sfx]
[-O] [-o @var{output-file}] [-p @var{path}] [--ppc ...] [-R] [-r] [--ram_all] [--rth_dbl|flt]
[-t @var{thr_nbr}] [--unn] [-v @var{var}[,@dots{}]] [-w wgt] [-X ...] [-x] [-y @var{op_typ}]
[@var{input-files}] [@var{output-file}]
@end example

@noindent
DESCRIPTION

@command{nces} performs gridpoint statistics (including, but not limited
to, averages) on variables across an arbitrary number (an
@dfn{ensemble}) of @var{input-files} and/or of input groups within each
file. 
Each file (or group) receives an equal weight by default.
@command{nces} was formerly (until @acronym{NCO} version 4.3.9,
released December, 2013) known as @command{ncea} (netCDF Ensemble
Averager)@footnote{
The old ncea command was deprecated in @acronym{NCO} version 4.3.9,  
released December, 2013.
@acronym{NCO} will attempt to maintain back-compatibility and work
as expected with invocations of @command{ncea} for as long as possible.
Please replace @command{ncea} by @command{nces} in all future work.}.
@cindex ensemble
For example, @command{nces} will average a set of files or groups,
weighting each file or group evenly by default. 
This is distinct from @command{ncra}, which performs statistics only
over the record dimension(s) (e.g., @var{time}), and weights each record 
in each record dimension evenly.

The file or group is the logical unit of organization for the results of
many scientific studies.
Often one wishes to generate a file or group which is the statistical
product (e.g., average) of many separate files or groups. 
This may be to reduce statistical noise by combining the results of a
large number of experiments, or it may simply be a step in a procedure 
whose goal is to compute anomalies from a mean state. 
In any case, when one desires to generate a file whose statistical
properties are influenced by all the inputs, then use @command{nces}.

@cindex weights
@cindex weighted average
As of @acronym{NCO} @w{version 4.9.4}, released in July, 2020,
@command{nces} accepts user-specified weights with the @samp{-w} 
(or long-option equivalent @samp{--wgt}, @samp{--wgt_var},
or @samp{--weight}) switch. 
The user must specify one weight per input file on the command line,
or the name of a (scalar or degenerate 1-D array) variable in each
input file that contains a single value to weight that file.
When no weight is specified, @command{nces} weights each file
(e.g., ensemble) in the @var{input-files} equally.

Variables in the @var{output-file} are the same size as the variable
hyperslab in each input file or group, and each input file or group
must be the same size after hyperslabbing
@footnote{As of @acronym{NCO} version 4.4.2 (released February, 2014)
@command{nces} allows hyperslabs in all dimensions so long as the
hyperslabs resolve to the same size. 
The fixed (i.e., non-record) dimensions should be the same size in
all ensemble members both before and after hyperslabbing, although
the hyperslabs may (and usually do) change the size of the dimensions
from the input to the output files.
Prior to this, @command{nces} was only guaranteed to work on hyperslabs
in the record dimension that resolved to the same size.}
@cindex record dimension
@cindex hyperslab
@command{nces} does allow files to differ in the input record
dimension size if the requested record hyperslab (@pxref{Hyperslabs})
resolves to the same size for all files.  
@command{nces} recomputes the record dimension hyperslab limits for
each input file so that coordinate limits may be used to select equal
length timeseries from unequal length files.
@cindex @acronym{IPCC}
@cindex @acronym{AR4}
@cindex @acronym{CMIP}
This simplifies analysis of unequal length timeseries from simulation
ensembles (e.g., the @acronym{CMIP3} @acronym{IPCC} @acronym{AR4}
archive).   

@html
<a name="nsm_fl"></a> <!-- http://nco.sf.net/nco.html#nsm_fl -->
<a name="nsm_grp"></a> <!-- http://nco.sf.net/nco.html#nsm_grp -->
<a name="nsm_sfx"></a> <!-- http://nco.sf.net/nco.html#nsm_sfx -->
@end html
@cindex @code{--nsm_fl}
@cindex @code{--nsm_grp}
@cindex @code{--ensemble_file}
@cindex @code{--ensemble_group}
@cindex @code{--nsm_sfx}
@cindex @code{--ensemble_suffix}
@command{nces} works in one of two modes, file ensembles 
or group ensembles.
File ensembles are the default (equivalent to the old @command{ncea}) 
and may also be explicitly specified by the @samp{--nsm_fl} or
@samp{--ensemble_file} switches.
To perform statistics on ensembles of groups, a newer feature, use
@samp{--nsm_grp} or @samp{--ensemble_group}.
Members of a group ensemble are groups that share the same structure,
parent group, and nesting level. 
Members must be @dfn{leaf groups}, i.e., not contain any sub-groups.
Their contents usually have different values because they are
realizations of replicated experiments.  
In group ensemble mode @command{nces} computes the statistics across 
the ensemble, which may span multiple input files. 
Files may contain members of multiple, distinct ensembles. 
However, all ensembles must have at least one member in the first input 
file. 
Group ensembles behave as an unlimited dimension of datasets: 
they may contain an arbitrary and extensible number of realizations in
each file, and may be composed from multiple files. 

Output statistics in group ensemble mode are stored in the parent group
by default. 
If the ensemble members are @file{/cesm/cesm_01} and
@file{/cesm/cesm_02}, then the computed statistic will be in
@file{/cesm} in the output file.   
The @samp{--nsm_sfx} option instructs nces to instead store output in  
a new child group of the parent created by attaching the suffix
to the parent group's name, e.g., @samp{--nsm_sfx='_avg'} would store
results in the output group @file{/cesm/cesm_avg}:
@example
nces --nsm_grp                  mdl1.nc mdl2.nc mdl3.nc out.nc
nces --nsm_grp --nsm_sfx='_avg' mdl1.nc mdl2.nc mdl3.nc out.nc
@end example

@xref{Statistics vs Concatenation}, for a description of the
distinctions between the statistics tools and concatenators. 
@cindex multi-file operators
@cindex standard input
@cindex @code{stdin}
As a multi-file operator, @command{nces} will read the list of
@var{input-files} from @code{stdin} if they are not specified 
as positional arguments on the command line 
(@pxref{Large Numbers of Files}).

Like @command{ncra} and @command{ncwa}, @command{nces} treats coordinate
variables as a special case.
Coordinate variables are assumed to be the same in all ensemble members,
so @command{nces} simply copies the coordinate variables that appear in 
ensemble members directly to the output file.
This has the same effect as averaging the coordinate variable across the
ensemble, yet does not incur the time- or precision- penalties of
actually averaging them.
@command{ncra} and @command{ncwa} allow coordinate variables to be
processed only by the linear average operation, regardless of the
arithmetic operation type performed on the non-coordinate variables
(@pxref{Operation Types}). 
Thus it can be said that the three operators (@command{ncra},
@command{ncwa}, and @command{nces}) all average coordinate variables
(even though @command{nces} simply copies them).
All other requested arithmetic operations (e.g., maximization,
square-root, RMS) are applied only to non-coordinate variables.
In these cases the linear average of the coordinate variable will be
returned.

@noindent
@html
<a name="xmp_ncea"></a> <!-- http://nco.sf.net/nco.html#xmp_ncea -->
<a name="xmp_nces"></a> <!-- http://nco.sf.net/nco.html#xmp_nces -->
@end html
EXAMPLES

Consider a model experiment which generated five realizations of one
year of data, say 1985.
Imagine that the experimenter slightly perturbs the initial conditions
of the problem before generating each new solution.   
Assume each file contains all twelve months (a seasonal cycle) of data
and we want to produce a single file containing the ensemble average
(mean) seasonal cycle.  
Here the numeric filename suffix denotes the realization number
(@emph{not} the month):
@example
nces 85_01.nc 85_02.nc 85_03.nc 85_04.nc 85_05.nc 85.nc
nces 85_0[1-5].nc 85.nc
nces -n 5,2,1 85_01.nc 85.nc
@end example
@noindent
These three commands produce identical answers.
@xref{Specifying Input Files}, for an explanation of the distinctions
between these methods.
The output file, @file{85.nc}, is the same size as the inputs files.
It contains 12 months of data (which might or might not be stored in the 
record dimension, depending on the input files), but each value in the
output file is the average of the five values in the input files.

In the previous example, the user could have obtained the ensemble
average values in a particular spatio-temporal region by adding a 
hyperslab argument to the command, e.g.,
@example
nces -d time,0,2 -d lat,-23.5,23.5 85_??.nc 85.nc
@end example
@noindent
In this case the output file would contain only three slices of data in
the @var{time} dimension. 
These three slices are the average of the first three slices from the
input files.
Additionally, only data inside the tropics is included.

As of @acronym{NCO} version 4.3.9 (released December, 2013)
@command{nces} also works with groups (rather than files) as the
fundamental unit of the ensemble.
Consider two ensembles, @code{/ecmwf} and @code{/cesm} stored across
three input files @file{mdl1.nc}, @file{mdl2.nc}, and @file{mdl3.nc}.
Ensemble members would be leaf groups with names like @code{/ecmwf/01},
@code{/ecmwf/02} etc. and @code{/cesm/01}, @code{/cesm/02}, etc.
These commands average both ensembles: 
@example
nces --nsm_grp mdl1.nc mdl2.nc mdl3.nc out.nc
nces --nsm_grp --nsm_sfx='_min' --op_typ=min -n 3,1,1 mdl1.nc out.nc
nces --nsm_grp -g cesm -v tas -d time,0,3 -n 3,1,1 mdl1.nc out.nc
@end example
@example
nces --nsm_grp mdl1.nc mdl2.nc mdl3.nc out.nc
nces --nsm_grp --nsm_sfx='_min' --op_typ=min -n 3,1,1 mdl1.nc out.nc
nces --nsm_grp -g cesm -v tas -d time,0,3 -n 3,1,1 mdl1.nc out.nc
@end example
The first command stores averages in the output groups @file{/cesm} and 
@file{/ecmwf}, while the second stores minima in the output groups
@file{/cesm/cesm_min} and @file{/ecmwf/ecmwf_min}:
The third command demonstrates that sub-setting and hyperslabbing work
as expected.
Note that each input file may contain different numbers of members
of each ensemble, as long as all distinct ensembles contain at least one
member in the first file.

@cindex weights
@cindex weighted average
As of @acronym{NCO} @w{version 4.9.4}, released in July, 2020,
@command{nces} accepts user-specified weights with the @samp{-w} 
(or long-option equivalent @samp{--wgt}, @samp{--wgt_var},
or @samp{--weight}) switch: 
@example
# Construct input variables with values of 1 and 2
ncks -O -M -v one ~/nco/data/in.nc ~/1.nc
ncrename -O -v one,var ~/1.nc
ncap2 -O -s 'var=2' ~/1.nc ~/2.nc

# Three methods of weighting input files unevenly
# 1. Old-method: specify input files multiple times
# 2. New-method: specify one weight per input file
# 3. New-method: specify weight variable in each input file
nces -O ~/1.nc ~/2.nc ~/2.nc ~/out.nc # Clumsy, limited to integer weights
nces -O -w 1,2 ~/1.nc ~/2.nc ~/out.nc # Flexible, works for any weight
nces -O -w var ~/1.nc ~/2.nc ~/out.nc # Flexible, works for any weight
# All three methods produce same answer: var=(1*1+2*2)/3=5/3=1.67
ncks ~/out.nc
@end example

@page
@html
<a name="ncflint"></a> <!-- http://nco.sf.net/nco.html#ncflint -->
@end html
@node ncflint netCDF File Interpolator, ncks netCDF Kitchen Sink, nces netCDF Ensemble Statistics, Reference Manual
@section @command{ncflint} netCDF File Interpolator
@cindex interpolation
@cindex adding data
@cindex multiplying data
@cindex addition
@findex ncflint

@noindent
SYNTAX
@example
ncflint [-3] [-4] [-5] [-6] [-7] [-A] [-C] [-c]  [--cmp @var{cmp_sng}]
[--cnk_byt @var{sz_byt}] [--cnk_csh @var{sz_byt}] [--cnk_dmn @var{nm},@var{sz_lmn}]
[--cnk_map @var{map}] [--cnk_min @var{sz_byt}] [--cnk_plc @var{plc}] [--cnk_scl @var{sz_lmn}]
[-D @var{dbg}] [-d @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]] [--fl_fmt @var{fl_fmt}]
[-F] [--fix_rec_crd] [-G @var{gpe_dsc}] [-g @var{grp}[,@dots{}]] [--glb ...]
[-H] [-h] [--hdr_pad @var{nbr}] [--hpss] 
[-i @var{var},@var{val3}] [-L @var{dfl_lvl}] [-l @var{path}] [-N]
[--no_cll_msr] [--no_frm_trm] [--no_tmp_fl] 
[-O] [-o @var{file_3}] [-p @var{path}] [--ppc ...] [-R] [-r] [--ram_all] 
[-t @var{thr_nbr}] [--unn] [-v @var{var}[,@dots{}]] [-w @var{wgt1}[,@var{wgt2}]] [-X ...] [-x]
@var{file_1} @var{file_2} [@var{file_3}]
@end example

@noindent
DESCRIPTION

@command{ncflint} creates an output file that is a linear combination of 
the input files.
This linear combination is a weighted average, a normalized weighted
average, or an interpolation of the input files.
Coordinate variables are not acted upon in any case, they are simply
copied from @var{file_1}.

There are two conceptually distinct methods of using @command{ncflint}.
The first method is to specify the weight each input file contributes to 
the output file.
In this method, the value @var{val3} of a variable in the output file
@var{file_3} is determined from its values @var{val1} and @var{val2} in
the two input files according to 
@set flg
@tex
$val3 = wgt1 \times val1 + wgt2 \times val2$
@clear flg
@end tex
@ifinfo
@math{@var{val3} = @var{wgt1}*@var{val1} + @var{wgt2}*@var{val2}} 
@clear flg
@end ifinfo
@ifset flg
@c texi2html does not like @math{}
@var{val3} = @var{wgt1}*@var{val1} + @var{wgt2}*@var{val2} 
@clear flg
@end ifset
.
Here at least @var{wgt1}, and, optionally, @var{wgt2}, are specified on 
the command line with the @samp{-w} (or @samp{--weight} or
@samp{--wgt_var}) switch.
@cindex @code{-w @var{wgt1}[,@var{wgt2}]}
@cindex @code{--weight @var{wgt1}[,@var{wgt2}]}
@cindex @code{--wgt_var @var{wgt1}[,@var{wgt2}]}
If only @var{wgt1} is specified then @var{wgt2} is automatically
computed as @math{@var{wgt2} = 1 @minus{} @var{wgt1}}.
Note that weights larger @w{than 1} are allowed. 
Thus it is possible to specify @math{@var{wgt1} = 2} and
@math{@var{wgt2} = -3}.
One can use this functionality to multiply all values in a given
file by a constant.

@cindex normalization
@cindex @code{-N}
@cindex @code{--nrm}
@cindex @code{--normalize}
As of @acronym{NCO} version 4.6.1 (July, 2016), the @samp{-N} switch
(or long-option equivalents @samp{--nrm} or @samp{--normalize})
implements a variation of this method. 
This switch instructs @command{ncflint} to internally normalize the two
supplied (or one supplied and one inferred) weights so that 
@set flg
@tex
$wgt1 = wgt1 / (wgt1 + wgt2)$ and $wgt2 = wgt2 / (wgt1 + wgt2)$
@clear flg
@end tex
@ifinfo
@math{@var{wgt1} = @var{wgt1}/(@var{wgt1} + @var{wgt2}} and
@math{@var{wgt2} = @var{wgt2}/(@var{wgt1} + @var{wgt2}} and
@clear flg
@end ifinfo
@ifset flg
@c texi2html does not like @math{}
@var{wgt1} = @var{wgt1}/(@var{wgt1} + @var{wgt2} and
@var{wgt2} = @var{wgt2}/(@var{wgt1} + @var{wgt2}
@clear flg
@end ifset
.
This allows the user to input integral weights, say, and to delegate 
the chore of normalizing them to @command{ncflint}.
Be careful that @samp{-N} means what you think, since the same
switch means something quite different in @command{ncwa}.

The second method of using @command{ncflint} is to specify the
interpolation option @w{with @samp{-i}} (or with the @samp{--ntp} or 
@samp{--interpolate} long options). 
This is the inverse of the first method in the following sense: 
When the user specifies the weights directly, @command{ncflint} has no
work to do besides multiplying the input values by their respective
weights and adding together the results to produce the output values.  
It makes sense to use this when the weights are known 
@emph{@w{a priori}}.

@cindex arrival value 
Another class of problems has the @dfn{arrival value} (i.e., @var{val3})
of a particular variable @var{var} known @emph{@w{a priori}}. 
In this case, the implied weights can always be inferred by examining
the values of @var{var} in the input files. 
This results in one equation in two unknowns, @var{wgt1} and @var{wgt2}:  
@set flg
@tex
$val3 = wgt1 \times val1 + wgt2 \times val2$
@clear flg
@end tex
@ifinfo
@math{@var{val3} = @var{wgt1}*@var{val1} + @var{wgt2}*@var{val2}} 
@clear flg
@end ifinfo
@ifset flg
@c texi2html does not like @math{}
@var{val3} = @var{wgt1}*@var{val1} + @var{wgt2}*@var{val2} 
@clear flg
@end ifset
.
Unique determination of the weights requires imposing the additional
constraint of normalization on the weights:
@math{@var{wgt1} + @var{wgt2} = 1}.
Thus, to use the interpolation option, the user specifies @var{var}
and @var{val3} with the @samp{-i} option.
@command{ncflint} then computes @var{wgt1} and @var{wgt2}, and uses these
weights on all variables to generate the output file.
Although @var{var} may have any number of dimensions in the input
files, it must represent a single, scalar value.  
@cindex degenerate dimension
Thus any dimensions associated with @var{var} must be @dfn{degenerate},
i.e., of size one.

If neither @samp{-i} nor @samp{-w} is specified on the command line,
@command{ncflint} defaults to weighting each input file equally in the
output file.
This is equivalent to specifying @samp{-w 0.5} or @samp{-w 0.5,0.5}.
Attempting to specify both @samp{-i} and @samp{-w} methods in the same
command is an error. 

@command{ncflint} does not interpolate variables of type @code{NC_CHAR}
and @code{NC_STRING}. 
This behavior is hardcoded.

By default @command{ncflint} interpolates or multiplies record
coordinate variables (e.g., time is often stored as a record coordinate) 
not other coordinate variables (e.g., latitude and longitude). 
This is because @command{ncflint} is often used to time-interpolate
between existing files, but is rarely used to spatially interpolate.
Sometimes however, users wish to multiply entire files by a constant
that does not multiply any coordinate variables.
The @samp{--fix_rec_crd} switch was implemented for this purpose
in @acronym{NCO} version 4.2.6 (March, 2013).
It prevents @command{ncflint} from multiplying or interpolating any
coordinate variables, including record coordinate variables. 

@cindex missing values
@cindex @code{_FillValue}
Depending on your intuition, @command{ncflint} may treat missing values
unexpectedly.
Consider a point where the value in one input file, say @var{val1},
equals the missing value @var{mss_val_1} and, at the same point,
the corresponding value in the other input file @var{val2} is not
misssing (i.e., does not equal @var{mss_val_2}).
There are three plausible answers, and this creates ambiguity.

@w{Option one} is to set @math{@var{val3} = @var{mss_val_1}}.
The rationale is that @command{ncflint} is, at heart, an interpolator
and interpolation involving a missing value is intrinsically undefined.
@command{ncflint} currently implements this behavior since it is the
most conservative and least likely to lead to misinterpretation.

@w{Option two} is to output the weighted valid data point, i.e.,
@set flg
@tex
$val3 = wgt2 \times val2$
@clear flg
@end tex
@ifinfo
@math{@var{val3} = @var{wgt2}*@var{val2}} 
@clear flg
@end ifinfo
@ifset flg
@c texi2html does not like @math{}
@var{val3} = @var{wgt2}*@var{val2} 
@clear flg
@end ifset
.
The rationale for this behavior is that interpolation is really a
weighted average of known points, so @command{ncflint} should weight the
valid point. 

@w{Option three} is to return the @emph{unweighted} valid point, i.e.,
@math{@var{val3} = @var{val2}}.
This behavior would appeal to those who use @command{ncflint} to
estimate data using the closest available data. 
When a point is not bracketed by valid data on both sides, it is better
to return the known datum than no datum at all.

The current implementation uses the first approach, @w{Option one}.
If you have strong opinions on this matter, let us know, since we are
willing to implement the other approaches as options if there is enough
interest. 

@noindent
@html
<a name="xmp_ncflint"></a> <!-- http://nco.sf.net/nco.html#xmp_ncflint -->
@end html
EXAMPLES

Although it has other uses, the interpolation feature was designed 
to interpolate @var{file_3} to a time between existing files.
Consider input files @file{85.nc} and @file{87.nc} containing variables 
describing the state of a physical system at times @math{@code{time} =
85} and @math{@code{time} = 87}.
Assume each file contains its timestamp in the scalar variable
@code{time}.  
Then, to linearly interpolate to a file @file{86.nc} which describes
the state of the system at time at @code{time} = 86, we would use
@example
ncflint -i time,86 85.nc 87.nc 86.nc
@end example

Say you have observational data covering January and April 1985 in two
files named @file{85_01.nc} and @file{85_04.nc}, respectively.
Then you can estimate the values for February and March by interpolating
the existing data as follows.
Combine @file{85_01.nc} and @file{85_04.nc} in a 2:1 ratio to make
@file{85_02.nc}:  
@example
ncflint -w 0.667 85_01.nc 85_04.nc 85_02.nc
ncflint -w 0.667,0.333 85_01.nc 85_04.nc 85_02.nc
@end example

Multiply @file{85.nc} @w{by 3} and @w{by @minus{}2} and add them
together to make @file{tst.nc}: 
@example
ncflint -w 3,-2 85.nc 85.nc tst.nc
@end example
@noindent
@cindex null operation
This is an example of a null operation, so @file{tst.nc} should be
identical (within machine precision) to @file{85.nc}.

@cindex multiplication
@cindex file multiplication
@cindex scaling
Multiply all the variables except the coordinate variables in the file
@file{emissions.nc} by @w{by 0.8}:
@example
ncflint --fix_rec_crd -w 0.8,0.0 emissions.nc emissions.nc scaled_emissions.nc
@end example
@noindent
The use of @samp{--fix_rec_crd} ensures, e.g., that the @code{time}
coordinate, if any, is not scaled (i.e., multiplied).

Add @file{85.nc} to @file{86.nc} to obtain @file{85p86.nc},
then subtract @file{86.nc} from @file{85.nc} to obtain @file{85m86.nc} 
@example
ncflint -w 1,1 85.nc 86.nc 85p86.nc
ncflint -w 1,-1 85.nc 86.nc 85m86.nc
ncdiff 85.nc 86.nc 85m86.nc
@end example
@noindent
Thus @command{ncflint} can be used to mimic some @command{ncbo}
operations. 
@cindex broadcasting variables
However this is not a good idea in practice because @command{ncflint}
does not broadcast (@pxref{ncbo netCDF Binary Operator}) conforming
variables during arithmetic. 
Thus the final two commands would produce identical results except that    
@command{ncflint} would fail if any variables needed to be broadcast.

@cindex @code{units}
Rescale the dimensional units of the surface pressure @code{prs_sfc}
from Pascals to hectopascals (millibars)
@example
ncflint -C -v prs_sfc -w 0.01,0.0 in.nc in.nc out.nc
ncatted -a units,prs_sfc,o,c,millibar out.nc
@end example
@noindent

@page
@html
<a name="ncks"></a> <!-- http://nco.sf.net/nco.html#ncks -->
@end html
@node ncks netCDF Kitchen Sink, ncpdq netCDF Permute Dimensions Quickly, ncflint netCDF File Interpolator, Reference Manual
@section @command{ncks} netCDF Kitchen Sink
@cindex kitchen sink
@cindex printing files contents
@cindex printing variables
@findex ncks

@noindent
SYNTAX
@example
ncks [-3] [-4] [-5] [-6] [-7] [-A] [-a] [--area_wgt] [-b @var{fl_bnr}]
[-C] [-c] [--cdl] [--chk_chr] [--chk_map] [--chk_mss] [--chk_nan] [--cmp @var{cmp_sng}]
[--cnk_byt @var{sz_byt}] [--cnk_csh @var{sz_byt}] [--cnk_dmn @var{nm},@var{sz_lmn}]
[--cnk_map @var{map}] [--cnk_min @var{sz_byt}] [--cnk_plc @var{plc}] [--cnk_scl @var{sz_lmn}]
[-D @var{dbg}] [-d @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]
[-F] [--fix_rec_dmn @var{dim}] [--fl_fmt @var{fl_fmt}] [--fmt_val @var{format}]
[-G @var{gpe_dsc}] [-g @var{grp}[,@dots{}]] [--glb ...] [--grp_xtr_var_xcl]
[-H] [-h] [--hdn] [--hdr_pad @var{nbr}] [--hpss] [--jsn] [--jsn_fmt @var{lvl}] 
[-L @var{dfl_lvl}] [-l @var{path}]
[-M] [-m] [--map @var{map-file}] [--md5] [--mk_rec_dmn @var{dim}]
[--no_blank] [--no_cll_msr] [--no_frm_trm] [--no_tmp_fl] 
[-O] [-o @var{output-file}] [-P] [-p @var{path}] [--ppc ...] [--prn_fl @var{print-file}]
[-Q] [-q] [-R] [-r] [--rad] [--ram_all] [--rgr ...] [--rnr=wgt]
[-s @var{format}] [-u] [--unn] [-V] [-v @var{var}[,@dots{}]] [--vrt @var{vrt-file}] 
[-X ...] [-x] [--xml] @var{input-file} [[@var{output-file}]]
@end example

@noindent
DESCRIPTION

@cindex @command{ncextr}
The nickname ``kitchen sink'' is a catch-all because @command{ncks}
combines most features of @command{ncdump} and @command{nccopy} with
extra features to extract, hyperslab, multi-slab, sub-set, and translate  
into one versatile utility. 
@command{ncks} extracts (a subset of the) data from @var{input-file},
regrids it according to @var{map-file} if specified,
then writes in netCDF format to @var{output-file}, and 
optionally writes it in flat binary format to @file{fl_bnr}, and
optionally prints it to screen.

@cindex @code{--trd}
@cindex @code{--traditional}
@cindex traditional
@command{ncks} prints netCDF input data in @acronym{ASCII},
@acronym{CDL}, @acronym{JSON}, or @acronym{NcML/XML} text formats to
@code{stdout}, like (an extended version of) @command{ncdump}.
By default @command{ncks} prints @acronym{CDL} format.
Option @samp{-s} (or long options @samp{--sng_fmt} and @samp{--string}) 
permits the user to format data using C-style format strings, while
option @samp{--cdl} outputs @acronym{CDL},
option @samp{--jsn} (or @samp{json}) outputs @acronym{JSON},
option @samp{--trd} (or @samp{traditional}) outputs ``traditional'' format,
and option @samp{--xml} (or @samp{ncml}) outputs @acronym{NcML}. 
The ``traditional'' tabular format is intended to be
easy to search for the data you want, one datum per screen line, with
all dimension subscripts and coordinate values (if any) preceding the
datum. 
@command{ncks} exposes many flexible controls over printed output,
including @acronym{CDL}, @acronym{JSON}, and @acronym{NcML}.

Options @samp{-a}, @samp{--cdl}, @samp{-F}, @samp{--fmt_val}, @samp{-H},
@samp{--hdn}, @samp{--jsn}, @samp{-M}, @samp{-m}, @samp{-P},
@samp{--prn_fl}, @samp{-Q}, @samp{-q}, @samp{-s}, @samp{--trd},
@samp{-u}, @samp{-V}, and @samp{--xml} (and their long option
counterparts) control the presence of data and metadata and their
formatted location and appearance when printed. 

@cindex global attributes
@cindex attributes, global
@command{ncks} extracts (and optionally creates a new netCDF file
comprised of) only selected variables from the input file
(similar to the old @command{ncextr} specification).
Only variables and coordinates may be specifically included or
excluded---all global attributes and any attribute associated with an
extracted variable are copied to the screen and/or output netCDF file. 
Options @samp{-c}, @samp{-C}, @samp{-v}, and @samp{-x} (and their long 
option synonyms) control which variables are extracted.

@command{ncks} extracts hyperslabs from the specified variables
(@command{ncks} implements the original @command{nccut} specification). 
Option @samp{-d} controls the hyperslab specification.
Input dimensions that are not associated with any output variable do
not appear in the output netCDF.
This feature removes superfluous dimensions from netCDF files. 

@cindex appending data
@cindex merging files
@command{ncks} will append variables and attributes from the
@var{input-file} to @var{output-file} if @var{output-file} is a
pre-existing netCDF file whose relevant dimensions conform to dimension
sizes of @var{input-file}. 
The append features of @command{ncks} are intended to provide a
rudimentary means of adding data from one netCDF file to another,
conforming, netCDF file. 
If naming conflicts exist between the two files, data in
@var{output-file} is usually overwritten by the corresponding data from 
@var{input-file}.  
Thus, when appending, the user should backup @var{output-file} in case 
valuable data are inadvertantly overwritten.

If @var{output-file} exists, the user will be queried whether to
@dfn{overwrite}, @dfn{append}, or @dfn{exit} the @command{ncks} call
completely.  
Choosing @dfn{overwrite} destroys the existing @var{output-file} and
create an entirely new one from the output of the @command{ncks} call.  
Append has differing effects depending on the uniqueness of the
variables and attributes output by @command{ncks}: If a variable or
attribute extracted from @var{input-file} does not have a name conflict
with the members of @var{output-file} then it will be added to
@var{output-file} without overwriting any of the existing contents of
@var{output-file}.  
In this case the relevant dimensions must agree (conform) between the
two files; new dimensions are created in @var{output-file} as required. 
@cindex global attributes
@cindex attributes, global
When a name conflict occurs, a global attribute from @var{input-file}
will overwrite the corresponding global attribute from
@var{output-file}.  
If the name conflict occurs for a non-record variable, then the
dimensions and type of the variable (and of its coordinate dimensions,
if any) must agree (conform) in both files. 
Then the variable values (and any coordinate dimension values)
from @var{input-file} will overwrite the corresponding variable values
(and coordinate dimension values, if any) in @var{output-file} 
@footnote{
Those familiar with netCDF mechanics might wish to know what is
happening here: @command{ncks} does not attempt to redefine the variable
in @var{output-file} to match its definition in @var{input-file},
@command{ncks} merely copies the values of the variable and its
coordinate dimensions, if any, from @var{input-file} to
@var{output-file}. 
}.

Since there can only be one record dimension in a file, the record
dimension must have the same name (though not necessarily the same size) in 
both files if a record dimension variable is to be appended. 
If the record dimensions are of differing sizes, the record dimension of
@var{output-file} will become the greater of the two record dimension
sizes, the record variable from @var{input-file} will overwrite any
counterpart in @var{output-file} and fill values will be written to any
gaps left in the rest of the record variables (I think). 
In all cases variable attributes in @var{output-file} are superseded by
attributes of the same name from @var{input-file}, and left alone if
there is no name conflict. 

Some users may wish to avoid interactive @command{ncks} queries about
whether to overwrite existing data.
For example, batch scripts will fail if @command{ncks} does not receive 
responses to its queries. 
Options @samp{-O} and @samp{-A} are available to force overwriting
existing files, and appending existing variables, respectively. 

@unnumberedsubsec Options specific to @command{ncks}

The following summarizes features unique to @command{ncks}.  
Features common to many operators are described in 
@ref{Shared features}. 

@table @samp

@html
<a name="abc"></a> <!-- http://nco.sf.net/nco.html#abc -->
<a name="alphabetize"></a> <!-- http://nco.sf.net/nco.html#alphabetize -->
@end html
@cindex alphabetization
@cindex sort alphabetically
@cindex @code{--abc}
@cindex @code{--alphabetize}
@cindex @code{--no_abc}
@cindex @code{--no_alphabetize}
@cindex @code{--no-abc}
@cindex @code{--no-alphabetize}
@item -a
Switches @samp{-a}, @samp{--abc}, and @samp{--alphabetize}
@emph{turn-off} the default alphbetization of extracted fields in
@command{ncks} only.
These switches are misleadingly named and were deprecated in
@command{ncks} as of @acronym{NCO} version 4.7.1 (December, 2017).  

This is the default behavior so these switches are no-ops included only
for completeness.
By default, @acronym{NCO} extracts, prints, and writes specified output
variables to disk in alphabetical order. 
This tends to make long output lists easier to search for particular
variables. 
Again, no option is necessary to write output in alphabetical order.
Until @acronym{NCO} version 4.7.1 (December, 2017), @command{ncks}
used the @code{-a}, @code{--abc}, or @code{--alphabetize} switches to
@emph{turn-off} the default alphabetization.
These names were counter-intuitive and needlessly confusing.
As of @acronym{NCO} version 4.7.1, @command{ncks} uses the new switches
@code{--no_abc}, @code{--no-abc}, @code{--no_alphabetize}, or
@code{--no-alphabetize}, all of which are equivalent.
The @code{--abc} and @code{--alphabetize} switches are now no-ops,
i.e., they write the output in the unsorted order of the input.
The @code{-a} switch is now completely deprecated in favor of the
clearer long option switches.

@html
<a name="bnr"></a> <!-- http://nco.sf.net/nco.html#bnr -->
<a name="binary"></a> <!-- http://nco.sf.net/nco.html#binary -->
@end html
@cindex binary format
@cindex @code{-b}
@cindex @code{--fl_bnr}
@cindex @code{--bnr}
@cindex @code{--binary}
@item -b @file{file}
Activate native machine binary output writing to binary file
@file{file}.
Also @samp{--fl_bnr} and @samp{--binary-file}.
Writing packed variables in binary format is not supported.
Metadata is never output to the binary file.
Examine the netCDF output file to see the variables in the binary file. 
Use the @samp{-C} switch, if necessary, to avoid wanting unwanted
coordinates to the binary file:
@example
% ncks -O -v one_dmn_rec_var -b bnr.dat -p ~/nco/data in.nc out.nc
% ls -l bnr.dat | cut -d ' ' -f 5 # 200 B contains time and one_dmn_rec_var
200
% ls -l bnr.dat
% ncks -C -O -v one_dmn_rec_var -b bnr.dat -p ~/nco/data in.nc out.nc
% ls -l bnr.dat | cut -d ' ' -f # 40 B contains one_dmn_rec_var only
40
@end example

@html
<a name="cal"></a> <!-- http://nco.sf.net/nco.html#cal -->
<a name="cln"></a> <!-- http://nco.sf.net/nco.html#cln -->
<a name="date_format"></a> <!-- http://nco.sf.net/nco.html#date_format -->
<a name="dt_fmt"></a> <!-- http://nco.sf.net/nco.html#dt_fmt -->
<a name="calendar"></a> <!-- http://nco.sf.net/nco.html#calendar -->
@end html
@cindex calendar dates
@cindex date formats
@cindex Gregorian dates
@cindex @code{--calendar}
@cindex @code{--cln_lgb}
@cindex @code{--prn_lgb}
@cindex @code{--dt_fmt}
@cindex @code{--date_format}
@cindex @code{--datestamp}
@item --cal 
As of @acronym{NCO} version 4.6.5 (March, 2017), @command{ncks} can
print human-legible calendar strings corresponding to time values with 
UDUnits-compatible date units of the form time-since-basetime, e.g.,
@samp{days since 2000-01-01} and a @acronym{CF} calendar attribute, if
any. 
Enact this with the @samp{--calendar} (also @samp{--cln},
@samp{--prn_lgb}, and @samp{--datestamp}) option when printing in any mode.
Invoking this option when @math{@var{dbg_lvl} >= 1} in @acronym{CDL}
mode prints both the value and the calendar string (one in comments):
@example
@verbatim
zender@aerosol:~$ ncks -D 1 --cal -v tm_365 ~/nco/data/in.nc
...
  variables:
    double tm_365 ;
      tm_365:units = "days since 2013-01-01" ; // char
      tm_365:calendar = "365_day" ; // char

  data:
    tm_365 = "2013-03-01"; // double value: 59
...
zender@aerosol:~$ ncks -D 1 -v tm_365 ~/nco/data/in.nc
...
    tm_365 = 59; // calendar format: "2013-03-01"
...
@end verbatim
@end example
This option is similar to the @command{ncdump} @samp{-t} option.
As of @acronym{NCO} version 4.6.8 (August, 2017), @command{ncks} 
@acronym{CDL} printing supports finer-grained control of date formats
with the @samp{--dt_fmt=@var{dt_fmt}} (or @samp{--date_format}) option. 
The @var{dt_fmt} is an enumerated integer from 0--3.
Values @math{@var{dt_fmt}=0} or 1 correspond to the short format for
dates that are the default.
The value @math{@var{dt_fmt}=2} requests the ``regular'' format for
dates, @math{@var{dt_fmt}=3} requests the full @acronym{ISO}-8601 format
with the ``T'' separator and the comma:
@example
@verbatim
ncks -H -m -v time_bnds -C --dt_fmt=value ~/nco/data/in.nc
# Value:    Output:
# 0,1       1964-03-13 09:08:16        # Default, short format
# 2         1964-03-13 09:08:16.000000 # Regular format
# 3         1964-03-13T09:08:16.000000 # ISO8601 'T' format
@end verbatim
@end example
Note that @samp{--dt_fmt} automatically implies @samp{--cal}
makes that options superfluous.

As of @acronym{NCO} version 4.9.4 (September, 2020), invoking
the @samp{--dt_fmt} option now applies equally well to @acronym{JSON}
and @acronym{XML} output as to @acronym{CDL} output:
@example
% ncks -d time,0 -v time --cdl --dt_fmt=3 ~/nco/data/in.nc
...
time = "1964-03-13T21:09:0.000000" ;
...
% ncks -d time,0 -v time --json --dt_fmt=3 ~/nco/data/in.nc
...
"data": ["1964-03-13T21:09:0.000000"]
...
% ncks -d time,0 -v time --xml --dt_fmt=3 ~/nco/data/in.nc
...
<ncml:values separator="*">1964-03-13T21:09:0.000000</ncml:values>
...
@end example

@html
<a name="chk_map"></a> <!-- http://nco.sf.net/nco.html#chk_map -->
<a name="area_wgt"></a> <!-- http://nco.sf.net/nco.html#area_wgt -->
<a name="frac_b_nrm"></a> <!-- http://nco.sf.net/nco.html#frac_b_nrm -->
@end html
@cindex @code{--area_wgt}
@cindex @code{--chk_map}
@cindex @code{--frac_b_nrm}
@cindex map-file
@item --chk_map
As of @acronym{NCO} version 4.9.0 (December, 2019), invoking
@samp{--chk_map} causes @command{ncks} to evaluate the quality of
regridding weights in the map-file provided as @var{input-file}.
This option works with map-files (not grid-files) in
@acronym{ESMF}/@acronym{CMIP6}-compliant format (i.e., a sparse matrix
variable named @code{S} and coordinates @code{[xy][ab]_[cv]}.
When invoked with the additional @samp{--area_wgt} option, the
evaluation statistics are area-weighted and thus exactly represent
the global-mean/min/max/mebs/rms/sdn biases expected when regridding a
globally uniform field.
This tool makes it easier to objectively assess weight-generation
algorithms, and will hopefully assist in their improvement.
Thanks to Mark Taylor of Saturday Night Live (@acronym{SNL}) and Paul 
Ullrich of UC Davis for this suggestion and early prototypes.
@example
@verbatim
$ ncks --chk_map map.nc            # Unweighted statistics
$ ncks --chk_map --dbg=2 map.nc    # Additional diagnostics
$ ncks --chk_map --area_wgt map.nc # Area-weighted statistics
@end verbatim
@end example

The map-checker performs numerous checks and reports numerous
statistics, probably more than you care about. 
Be assured that each piece of provided information has in the past
proved useful to developers of weight-generation and regridding
algorithms.
Most of the time, users can learn whether the examined map is of
sufficient quality for their purposes by examing only a few of these
statistics.
Before defining these primary statistics, it is helpful to understand
the meaning of the weight-array @var{S} (stored in a map-file as the
variable @code{S}), and the terminology of rows and columns.

A remapping (aka regridding) transforms a field on an input grid to an
an output grid while conserving to the extent possible or desired the
local and global properties of the field.
The map @var{S} is a matrix of @var{M} rows and @var{N} columns of
weights, where @var{M} is the number of gridcells (or degrees of
freedom, @acronym{DOF}s) in the destination grid, and @var{N} is the
number of gridcells (or @acronym{DOFs}) in the source grid. 
An individual weight @var{S}(@var{m},@var{n}) represents the
fractional contribution to destination gridcell @var{m} by source
gridcell @var{n}. 
By convention the weights are normalized to sum to unity in each row 
(destination gridcell) that completely overlaps the input grid.
Thus the weights in a single row are all equivalent to the fractional
destination areas that the same destination gridcell (we will drop the
@acronym{DOF} terminology hereafter for conciseness) receives from
each source gridcell. 
Regardless of the values of the individual weights, it is intuitive
that their row-sum should never exceed unity because that would be
physically equivalent to an output gridcell receiving more than its
own area from the source grid.
Map-files typically store these row-sum statistics for each
destination gridcell in the @code{frac_b} variable described further
below. 

Likewise the weights in a single column represent the fractional
destination areas that a single source gridcell contributes to  
every output gridcell.
Each output gridcell in a column may have a different area so
column-sums need not, and in general do not, sum to unity.
However, a source gridcell ought to contribute to the destination
grid a total area equal to its own area.
Thus a constraint on column-sums is that their weights, themselves
weighted by the destination gridcell area corresponding to each row,
should sum exactly to the source gridcell area.
In other words, the destination-area-weighted column-sum divided by
the source gridcell area would be unity (in a perfect first order
map) for every source gridcell that completely overlaps valid
destination gridcells.  
Map-files typically store these area-weighted-column-sum-ratio
statistics for each gridcell in the @code{frac_a} variable described
further below.  

Storing the entire weight-matrix @b{S} is unnecessary because only a
relative handful of gridcells in the source grid contribute to a given 
destination gridcell, and visa versa.
Instead, map-files store only the non-zero @var{S}(@var{m},@var{n}),
and encode them as a sparse-matrix.
Storing @b{S} as a sparse matrix rather than a full matrix reduces
overall storage sizes by a factor on the order of the ratio of the
product of the grid sizes to their sum, or about 10,000 for grids with
horizontal resolution near one degree, and more for finer resolutions.
The sparse-matrix representation is a one-dimensional array of weights 
@code{S}, together with two ancillary arrays, @code{row}
and @code{column}, that contain the one-dimensional row and column
indices, respectively, corresponding to the destination and source
gridcells of the associated weight.
By convention, map-files store the row and column indices using the
1-based convention in common use in the 1990s when regridding software
was all written in Fortran.
The map-checker prints cell locations with 1-based indices as well:
@example
@verbatim
% ncks --chk_map map_ne30np4_to_cmip6_180x360_nco.20190601.nc
Characterization of map-file map_ne30np4_to_cmip6_180x360_nco.20190601.nc
Cell triplet elements : [Fortran (1-based) index, center latitude, center longitude]
Sparse matrix size n_s: 246659
Weight min S(190813):  5.1827201764857658e-25 from cell \
                       [33796,-45.7998,+136.437] to [15975,-45.5,+134.5]
Weight max S( 67391):  1.0000000000000000e+00 from cell \
                       [33671,-54.4442,+189.645] to [12790,-54.5,+189.5]
Ignored weights (S=0.0): 0
...
@end verbatim
@end example
Here the map-file weights span twenty-five orders of magnitude.
This may seem large though in practice is typical for high-resolution
intersection grids.
The Fortran-convention index of each weight extreme is followed by 
its geographic latitude and longitude.
Reporting the locations of extrema, and of gridcells whose metrics
miss their target values by more than a specificied tolerance, are
prime map-checker features.

As mentioned above, the two statistics most telling about map quality 
are the weighted column-sums @var{frac_a} and the row-sums @var{frac_b}.
The short-hand names for what these metrics quantify are
Conservation and Consistency, respectively. 
Conservation means the total fraction of an input gridcell that
contributes to the output grid.
For global input and output grids that completely tile the sphere,
the entirety of each input gridcell should contribute (i.e., map to)
the output grid.
The same concept that applies locally to conservation of a gridcell
value applies globally to the overall conservation of an input field.
Thus a perfectly conservative mapping between global grids that tile
the sphere would have @math{@var{frac_a} = 1.0} for every input
gridcell, and for the mean of all input gridcells.

The map-checker computes Conservation (@var{frac_a}) from the stored
variables @code{S}, @code{row}, @code{column}, @code{area_a}, and
@code{area_b} in the map-file, and then compares those values to the
@code{frac_a} values (if any) on-disk, and warns of any disagreements
@footnote{
As of @w{version 5.1.1} (November 2022), the map checker diagnoses
from the global attributes @code{map_method}, @code{no_conserve},
or @code{noconserve} (in that order, if present) whether the mapping
weights are intended to be conservative (as opposed to, e.g.,
bilinear). 
Weights deemed non-conservative by design are no longer flagged with
dire @emph{WARNING} messages.}. 
By definition, conservation is perfect to first order if the sum of
the destination-gridcell-area-weighted weights (which is an area)
equals the source gridcell area, and so their ratio (@var{frac_a}) is
unity.
Computing the area-weighted-column-sum-ratios and comparing those
@var{frac_a} to the stored @code{frac_a} catches any discrepancies. 
The analysis sounds an alarm when discrepancies exceed a tolerance
(currently 5.0e-16).  
More importantly, the map-checker reports the summary statistics of
the computed @var{frac_a} metrics and their imputed errors, including
the grid mean, minimum, maximum, mean-absolute bias, root-mean-square
bias, and standard deviation.
@example
@verbatim
% ncks --chk_map map_ne30np4_to_cmip6_180x360_nco.20190601.nc
...
Conservation metrics (column-sums of area_b-weighted weights normalized by area_a) and errors---
Perfect metrics for global Grid B are avg = min = max = 1.0, mbs = rms = sdn = 0.0:
frac_a avg: 1.0000000000000000 = 1.0-0.0e+00 // Mean
frac_a min: 0.9999999999991109 = 1.0-8.9e-13 // Minimum in grid A cell [45328,+77.3747,+225]
frac_a max: 1.0000000000002398 = 1.0+2.4e-13 // Maximum in grid A cell [47582,+49.8351,+135]
frac_a mbs: 0.0000000000000096 =     9.6e-15 // Mean absolute bias from 1.0
frac_a rms: 0.0000000000000167 =     1.7e-14 // RMS relative to 1.0
frac_a sdn: 0.0000000000000167 =     1.7e-14 // Standard deviation
...
@end verbatim
@end example
The values of the @code{frac_a} metric are generally imperfect (not
1.0) for global grids.
The bias is the deviation from the target metric shown in the second
floating-point column in each row above (e.g., 8.9e-13). 
These biases should be vanishingly small with respect to unity.
Mean biases as large as 1.0e-08 may be considered acceptable for
off-line analyses (i.e., a single regridding of raw data) though the
acceptable tolerance should be more stringent for on-line use such as
in a coupler where forward and reverse mappings may be applied tens of 
thousands of times.
The mean biases for such on-line regridding should be close to 1.0e-15
in order for tens-of-thousands of repetitions to still conserve
mass/energy to full double-precision.

The minimum and maximum gridcell biases indicate the worst performing
locations of the mapping.
These are generally much (a few orders of magnitude) greater than the
mean biases.
Observe that the minimum and maximum biases in the examples above
and below occur at longitudes that are multiples of @w{45 degrees}.
This is characteristic of mappings to/from for cube-square grids
whose faces have edges, and thus additional complexity, at multiples
of @w{45 degrees}.
This illustrates how intersection grid geometry influences biases.
More complex, finer-scale structures, produce greater biases.
The Root-Mean-Square (@acronym{RMS}) and standard deviation metrics
characterize the distribution of biases throughout the entire
intersection grid, and are thus complementary information to the
minimum and maximum biases. 

Consistency expresses the total fraction of an output gridcell that
receives contributions from the input grid. 
Thus Consistency is directly analogous to Conservation, only applied
to the output grid. 
Conservation is the extent to which the mapping preserves the local
and grid-wide integrals of input fields, while Consistency is the
extent to which the mapping correctly aligns the input and output
grids so that each destination cell receives the appropriate
proportion of the input integrals.
The mapping will produce an acceptably faithful reproduction of the
input on the output grid only if all local and global Conservation and
Consistency metrics meet the acceptable error tolerances. 

The map-checker computes the Consistency (@var{frac_b}) as row-sums of 
the weights stored in @code{S} and compares these to the stored values
of @code{frac_b}. 
(Note how the definition of weights @var{S}(@var{m},@var{n}) as the
fractional contribution to destination gridcell @var{m} by source
gridcell @var{n} makes calculation of @var{frac_b} almost trivial in
comparison to @var{frac_a}). 
Nevertheless, @code{frac_b} in the file may differ from the computed
row-sum for example if the map-file generator artificially limits the
stored @code{frac_b} value for any cell @w{to 1.0} for those row-sums
that @w{exceed 1.0}.  
The map-checker raises an alarm when discrepancies between computed
and stored @code{frac_b} exceed a tolerance (currently 5.0e-16).  
There are semi-valid reasons a map-generator might do this, so this
does not necessarily indicate an error.
The alarm simply informs the user that applying the weights will lead
to a slightly different Consistency than indicated by the stored
@code{frac_b}.

As with @code{frac_a}, the values of @code{frac_b} are generally
imperfect @w{(not 1.0)} for global grids:
@example
@verbatim
% ncks --chk_map map_ne30np4_to_cmip6_180x360_nco.20190601.nc
...
Consistency metrics (row-sums of weights) and errors---
Perfect metrics for global Grid A are avg = min = max = 1.0, mbs = rms = sdn = 0.0:
frac_b avg: 0.9999999999999999 = 1.0-1.1e-16 // Mean
frac_b min: 0.9999999999985523 = 1.0-1.4e-12 // Minimum in grid B cell [59446,+75.5,+45.5]
frac_b max: 1.0000000000004521 = 1.0+4.5e-13 // Maximum in grid B cell [63766,+87.5,+45.5]
frac_b mbs: 0.0000000000000065 =     6.5e-15 // Mean absolute bias from 1.0
frac_b rms: 0.0000000000000190 =     1.9e-14 // RMS relative to 1.0
frac_b sdn: 0.0000000000000190 =     1.9e-14 // Standard deviation
...
@end verbatim
@end example
This example shows that @var{frac_b} has the greatest local errors
at similar boundaries (multiples of @w{45 degrees} longitude) as
@var{frac_a}. It is typical for Conservation and Consistency to
degrade in intricate areas of the intersection grid, and these
areas occur at multiples of @w{45 degrees} longitude for cubed-sphere 
mappings. 

The map-checker will produce area-weighted metrics when invoked
with the @code{--area_wgt} flag, e.g.,
@samp{ncks --area_wgt in.nc}.
Area-weighted statistics show the exact local and global results to
expect with real-world grids in which large consistency/conservation
errors in small gridcells may be less important than smaller errors in
larger gridcells. 
Global-weighted mean statistics will of course differ from unweighted
statistics, although the minimum and maximum do not change:
@example
@verbatim
% ncks --area_wgt map_ne30np4_to_cmip6_180x360_nco.20190601.nc
...
Conservation metrics (column-sums of area_b-weighted weights normalized by area_a) and errors---
Perfect metrics for global Grid B are avg = min = max = 1.0, mbs = rms = sdn = 0.0:
frac_a avg: 1.0000000000000009 = 1.0+8.9e-16 // Area-weighted mean
frac_a min: 0.9999999999999236 = 1.0-7.6e-14 // Minimum in grid A cell [12810,+3.44654,+293.25]
frac_a max: 1.0000000000001146 = 1.0+1.1e-13 // Maximum in grid A cell [16203,-45.7267,+272.31]
frac_a mbs: 0.0000000000000067 =     6.7e-15 // Area-weighted mean absolute bias from 1.0
frac_a rms: 0.0000000000000102 =     1.0e-14 // Area-weighted RMS relative to 1.0
frac_a sdn: 0.0000000000000103 =     1.0e-14 // Standard deviation

Consistency metrics (row-sums of weights) and errors---
Perfect metrics for global Grid A are avg = min = max = 1.0, mbs = rms = sdn = 0.0:
frac_b avg: 1.0000000000000047 = 1.0+4.7e-15 // Area-weighted mean
frac_b min: 0.9999999999998442 = 1.0-1.6e-13 // Minimum in grid B cell [48415,+44.5,+174.5]
frac_b max: 1.0000000000002611 = 1.0+2.6e-13 // Maximum in grid B cell [16558,-44.5,+357.5]
frac_b mbs: 0.0000000000000065 =     6.5e-15 // Area-weighted mean absolute bias from 1.0
frac_b rms: 0.0000000000000129 =     1.3e-14 // Area-weighted RMS relative to 1.0
frac_b sdn: 0.0000000000000133 =     1.3e-14 // Standard deviation
...
@end verbatim
@end example
The examples above show no outstanding differences (besides rounding) 
between the unweighted and area-weighted statistics.
The absence of degradation between the global unweighted statistics
(further up the page) and the global weighted statistics (just above)
demonstrates there are no important correlations between local weight
biases and gridcell areas.
The area-weighted mean @var{frac_b} statistic deserves special
mention.
Its value is the exact factor by which the mapping will shift the
global mean of a spatially uniform input field.
This metric is, therefore, first among equals when evaluating 
the quality of maps under consideration for use in time-stepping
models where global conservation (e.g., of mass or energy) is
crucial. 

@cindex @code{--frac_b_nrm}
As of @acronym{NCO} version 4.9.2 (March, 2020), adding the
@samp{--frac_b_nrm} flag changes the map-checker into a read-write
algorithm that first diagnoses the map-file statistics described
above and then re-writes the weights (and weight-derived statistics
@var{frac_a} and @var{frac_b}) to compensate or ``fix'' issues
that poor-quality input grids can cause.
Input grids can and often do have regions that are not tiled by
any portion of any input gridcell.
For example, many @acronym{FV} ocean grids (such as @acronym{MPAS})
are empty (have no gridcells) in land regions beyond the coasts.
Some @acronym{FV} ocean grids have gridcells everywhere and mask
(i.e., screen-out) the non-ocean gridcells by setting the mask value
to zero.
Both these designs are perfectly legal.
What is illegal, yet sometimes encountered in practice, is overlapping
gridcells on the same input grid.
Such an input grid is said to be self-overlapping.

The surface topography dataset grid
@file{SCRIPgrid_1km-merge-10min_HYDRO1K-merge-nomask_c130402.nc}
(hereafter the @acronym{HYDRO1K} grid for short) used by
@acronym{E3SM} and @acronym{CESM} is self-overlapping.
Weight-generators that receive the same input location twice might
(if they do not take precaustions to idenfity the issue, which no
known weight-generators do) double-weight the self-overlapped
region(s).
In other words, self-overlapping input grids can lead
weight-generators to produce values @math{@var{frac_b} >> 1.0}.
Applying these weights would lead to exaggerated values on the
destination grid.

The best solution to this issue is to adjust the input grid to
avoid self-overlap.
However, this solution may be difficult or impractical where the
original data, producer, or algorithm are unavailable or unclear.
In such cases, the @code{--frac_b_nrm} flag provides a workaround. 
Please understand that @samp{ncks --frac_b_nrm map.nc} is designed to
alter @file{map.nc} in-xsplace, so backup the original file first.
@c ncks --frac_b_nrm $DATA/maps/map_hydro1k_to_ne1024np4_nco.20200301.nc > foo 2>&1
@example
@verbatim
% ncks --frac_b_nrm map_hydro1k_to_ne1024np4_nco.20200301.nc
...
...
@end verbatim
@end example

@html
<a name="chk_chr"></a> <!-- http://nco.sf.net/nco.html#chk_chr -->
<a name="check_char"></a> <!-- http://nco.sf.net/nco.html#check_char -->
@end html
@cindex @code{--chk_chr}
@cindex @code{--check_char}
@cindex @code{missing_value}
@cindex @acronym{CF}
@cindex @acronym{DIWG}
@item --chk_chr
The @emph{identifiers} in a netCDF file are the set of dimension,
group, variable, and attribute names it contains.
As of @acronym{NCO} version 5.1.8 (September, 2023), @command{ncks}
can report all identifiers that violate the @acronym{CF} Convention
that identifiers ``should begin with a letter and be composed of
letters, digits, and underscores.''
System or library-defined identifiers (such as @code{_FillValue})
are not subject to this (user-land) rule.
@acronym{NASA}'s Dataset Interoperability Working Group 
(@uref{https://wiki.earthdata.nasa.gov/display/ESDSWG/Dataset+Interoperability+Working+Group,
DIWG}) supports this convention.
This option reports which identifiers do not comply with this
convention, so that a file can be easily checked for compliance with
the @acronym{DIWG} recommendation and the underlying @acronym{CF}
Convention: 
@example
@verbatim
$ ncks --chk_chr ~/nco/data/in.nc
...
ncks: WARNING nco_chk_chr() reports variable att_var_spc_chr attribute name "at_in_name@" is not CF-compliant
ncks: WARNING nco_chk_chr() reports variable name "var_nm-dash" is not CF-compliant
ncks: WARNING nco_chk_chr() reports variable var_nm-dash attribute name "att_nm-dash" is not CF-compliant
ncks: INFO nco_chk_chr() reports total number of identifiers with CF non-compliant names is 26
@end verbatim
@end example

@html
<a name="chk_mss"></a> <!-- http://nco.sf.net/nco.html#chk_mss -->
<a name="check_missing_value"></a> <!-- http://nco.sf.net/nco.html#check_missing_value -->
@end html
@cindex @code{--chk_mss}
@cindex @code{--check_missing_value}
@cindex @code{missing_value}
@cindex @acronym{CF}
@cindex @acronym{DIWG}
@item --chk_mss
As of @acronym{NCO} version 5.1.8 (September, 2023), @command{ncks}
can report all variables and groups that contain a
@code{missing_value} attribute.
@acronym{NASA}'s Dataset Interoperability Working Group 
(@uref{https://wiki.earthdata.nasa.gov/display/ESDSWG/Dataset+Interoperability+Working+Group,
DIWG}) notes that the @code{missing_value} attribute has been
semi-deprecated, and recommends that it should not be used in new
Earth Science data products.
This option reports which variables (and groups) contain a
@code{missing_value} attribute, so that a file can be easily
checked for compliance with the @acronym{DIWG} recommendation:
@example
@verbatim
$ ncks --chk_mss ~/nco/data/in.nc
ncks: WARNING nco_chk_mss() reports variable fll_val_mss_val contains "missing_value" attribute
ncks: WARNING nco_chk_mss() reports variable one_dmn_rec_var_missing_value contains "missing_value" attribute
...
ncks: WARNING nco_chk_mss() reports variable rec_var_int_mss_val_flt contains "missing_value" attribute
ncks: INFO nco_chk_mss() reports total number of variables and/or groups with "missing_value" attribute is 11
@end verbatim
@end example

@html
<a name="chk_nan"></a> <!-- http://nco.sf.net/nco.html#chk_nan -->
<a name="check_nan"></a> <!-- http://nco.sf.net/nco.html#check_nan -->
@end html
@cindex @code{--chk_nan}
@cindex @code{--check_nan}
@cindex @code{NaN}
@cindex @code{NaNf}
@cindex @acronym{DIWG}
@item --chk_nan
As of @acronym{NCO} version 4.8.0 (May, 2019), @command{ncks} can
locate @code{NaN} or @code{NaNf} in double- and single-precision
floating-point variables, respectively.
@acronym{NCO} prints the location of the first @code{NaN} (if any)
encountered in each variable. 
@acronym{NASA}'s Dataset Interoperability Working Group 
(@uref{https://wiki.earthdata.nasa.gov/display/ESDSWG/Dataset+Interoperability+Working+Group,
DIWG}) notes that the @code{missing_value} attribute has been
semi-deprecated, and recommends that it should not be used in new
Earth Science data products.
This option reports allows users to easily check whether all the
floating point variables in a file comply with the @acronym{DIWG}
recommendation: 
@example
@verbatim
$ ncks --chk_nan ~/nco/data/in_4.nc
ncks: WARNING nco_chk_nan() reports variable /nan_arr has first NaNf at hyperslab element 1
ncks: WARNING nco_chk_nan() reports variable /nan_scl has first NaNf at hyperslab element 0
ncks: INFO nco_chk_nan() reports total number of floating-point variables with NaN elements is 2
@end verbatim
@end example
Thanks to Matthew Thompson of @acronym{NASA} for originally suggesting
this feature.

@html
<a name="dmn_fix_mk"></a> <!-- http://nco.sf.net/nco.html#dmn_fix_mk -->
<a name="fix_rec_dmn"></a> <!-- http://nco.sf.net/nco.html#fix_rec_dmn -->
@end html
@cindex record dimension
@cindex fixed dimension
@cindex @code{--fix_rec_dmn @var{dim}}
@cindex @code{--no_rec_dmn @var{dim}}
@item --fix_rec_dmn
Change record dimension @var{dim} in the input file into a fixed
dimension in the output file. 
Also @samp{--no_rec_dmn}.
Before @acronym{NCO} version 4.2.5 (January, 2013), the syntax for 
@code{--fix_rec_dmn} did not permit or require the specification of
the dimension name @var{dim}. 
This is because the feature only worked on netCDF3 files, which support
only one record dimension, so specifying its name was unnecessary.
netCDF4 files allow an arbitrary number of record dimensions, so the
user must specify which record dimension to fix.
The decision was made that starting with @acronym{NCO} version 4.2.5
(January, 2013), it is always required to specify the dimension name to
fix regardless of the netCDF file type.
This keeps the code simple, and is symmetric with the syntax for
@code{--mk_rec_dmn}, described next.

As of @acronym{NCO} version 4.4.0 (January, 2014), the argument
@code{all} may be given to @samp{--fix_rec_dmn} to convert @emph{all} 
record dimensions to fixed dimensions in the output file. 
Previously, @samp{--fix_rec_dmn} only allowed one option, the name of a
single record dimension to be fixed. 
Now it is simple to simultaneously fix all record dimensions.
This is useful (and nearly mandatory) when flattening netCDF4 files that
have multiple record dimensions per group into netCDF3 files (which are
limited to at most one record dimension) (@pxref{Group Path Editing}). 

@html
<a name="hdn"></a> <!-- http://nco.sf.net/nco.html#hdn -->
<a name="hidden"></a> <!-- http://nco.sf.net/nco.html#hidden -->
@end html
@cindex hidden attributes
@cindex special attributes
@cindex @code{--hdn}
@cindex @code{--hidden}
@cindex @code{_SOURCE_FORMAT}
@cindex @code{_Format}
@cindex @code{_DeflateLevel}
@cindex @code{_Shuffle}
@cindex @code{_Storage}
@cindex @code{_ChunkSizes}
@cindex @code{_Endianness}
@cindex @code{_Fletcher32}
@cindex @code{_NOFILL}
@cindex @code{_NCProperties}
@cindex @code{_IsNetcdf4}
@cindex @code{_SuperblockVersion}
@item --hdn
As of @acronym{NCO} version 4.4.0 (January, 2014), the @samp{--hdn}
or @samp{--hidden} options print hidden (aka special) attributes.
This is equivalent to @samp{ncdump -s}.
Hidden attributes include: @code{_Format}, @code{_DeflateLevel},
@code{_Shuffle}, @code{_Storage}, @code{_ChunkSizes},
@code{_Endianness}, @code{_Fletcher32}, and @code{_NOFILL}. 
Previously @command{ncks} ignored all these attributes in
@acronym{CDL}/@acronym{XML} modes. 
Now it prints these attributes as appropriate in all modes. 
As of @acronym{NCO} version 4.4.6 (September, 2014), @samp{--hdn} 
also prints the extended file format (i.e., the format of the file
or server supplying the data) as @code{_SOURCE_FORMAT}.
As of @acronym{NCO} version 4.6.1 (August, 2016), @samp{--hdn} 
also prints the hidden attributes @code{_NCProperties},
@code{_IsNetcdf4}, and @code{_SuperblockVersion} for netCDF4 files so 
long as @acronym{NCO} is linked against netCDF library version 4.4.1 or
later. 
Users are referred to the
@uref{http://www.unidata.ucar.edu/software/netcdf/docs, Unidata netCDF Documentation},
or the man pages for @command{ncgen} or @command{ncdump}, for
detailed descriptions of the meanings of these hidden attributes. 

@html
<a name="cdl"></a> <!-- http://nco.sf.net/nco.html#cdl -->
<a name="hdp"></a> <!-- http://nco.sf.net/nco.html#hdp -->
<a name="hncgen"></a> <!-- http://nco.sf.net/nco.html#hncgen -->
<a name="h4_ncgen"></a> <!-- http://nco.sf.net/nco.html#h4_ncgen -->
<a name="ncgen-hdf"></a> <!-- http://nco.sf.net/nco.html#ncgen-hdf -->
@end html
@cindex @command{hdp}
@cindex @command{ncgen}
@cindex @command{ncgen-hdf}
@cindex @command{hncgen}
@cindex @command{h4_ncgen}
@cindex @command{ncdump}
@cindex @code{--cdl}
@cindex @acronym{CDL}
@cindex @acronym{HDF}
@cindex @acronym{HDF4}
@item --cdl 
As of @acronym{NCO} version 4.3.3 (July, 2013), @command{ncks} can
print extracted data and metadata to screen (i.e., @code{stdout}) as
valid @acronym{CDL} (network Common data form Description Language). 
@acronym{CDL} is the human-readable ``lingua franca'' of netCDF ingested by
@command{ncgen} and excreted by @command{ncdump}.
As of @acronym{NCO} version 4.6.9 (September, 2017), @command{ncks} 
prints @acronym{CDL} by default, and the ``traditional'' mode must
be explicitly selected with @samp{--trd}.
Compare @command{ncks} ``traditional'' with @acronym{CDL} printing:
@example
@verbatim
zender@roulee:~$ ncks --trd -v one ~/nco/data/in.nc
one: type NC_FLOAT, 0 dimensions, 1 attribute, chunked? no, compressed? no, packed? no
one size (RAM) = 1*sizeof(NC_FLOAT) = 1*4 = 4 bytes
one attribute 0: long_name, size = 3 NC_CHAR, value = one

one = 1 

zender@roulee:~$ ncks --cdl -v one ~/nco/data/in.nc
netcdf in {

  variables:
    float one ;
    one:long_name = "one" ;

  data:
    one = 1 ;

} // group /
@end verbatim
@end example
Users should note the @acronym{NCO}'s @acronym{CDL} mode outputs
successively more verbose additional diagnostic information in
@acronym{CDL} comments as the level of debugging increases from
zero to two.
For example printing the above with @samp{-D 2} yields
@example
@verbatim
zender@roulee:~$ ncks -D 2 --cdl -v one ~/nco/data/in.nc
netcdf in {
  // ncgen -k classic -b -o in.nc in.cdl

  variables:
    float one ; // RAM size = 1*sizeof(NC_FLOAT) = 1*4 = 4 bytes, ID = 147
      one:long_name = "one" ; // char

  data:
    one = 1 ; 

} // group /
@end verbatim
@end example

@command{ncgen} converts @acronym{CDL}-mode output into a netCDF file:
@example
ncks -v one ~/nco/data/in.nc > ~/in.cdl
ncgen -k netCDF-4 -b -o ~/in.nc ~/in.cdl
ncks -v one ~/in.nc
@end example
The @acronym{HDF4} version of @command{ncgen}, often named
@command{hncgen}, @command{h4_ncgen}, or @command{ncgen-hdf},
(usually) converts netCDF3 @acronym{CDL} into an @acronym{HDF} file:
@example
cd ~/nco/data
ncgen      -b -o hdf.hdf hdf.cdl # HDF ncgen is sometimes named...ncgen
ncgen      -b -o in.hdf  in.cdl  # Fails: Some valid netCDF CDL breaks HDF ncgen
hncgen     -b -o hdf.hdf hdf.cdl # HDF ncgen is hncgen in some RPM packages
h4_ncgen   -b -o hdf.hdf hdf.cdl # HDF ncgen is h4_ncgen in Anaconda packages
ncgen-hdf  -b -o hdf.hdf hdf.cdl # HDF ncgen is ncgen-hdf in some Debian packages
hdp dumpsds hdf.hdf              # ncdump/h5dump-equivalent for HDF4
h4_ncdump dumpsds hdf.hdf        # ncdump/h5dump-equivalent for HDF4
@end example
Note that @acronym{HDF4} does not support netCDF-style groups, so the 
above commands fail when the input file contains groups.
Only netCDF4 and @acronym{HDF5} support groups.
In our experience the @acronym{HDF} @command{ncgen} command, by
whatever name installed, is not robust and fails on many valid netCDF3 
@acronym{CDL} constructs.
The @acronym{HDF4} version of @command{ncgen} will definitely fail on
the default @acronym{NCO} input file @file{nco/data/in.cdl}.
The @acronym{NCO} source code distribution provides
@file{nco/data/hdf.cdl} that works with the @acronym{HDF4} version
of @command{ncgen}, and can be used to test @acronym{HDF} files.

@html
<a name="dmn_rec_mk"></a> <!-- http://nco.sf.net/nco.html#dmn_rec_mk -->
<a name="mk_rec_dmn"></a> <!-- http://nco.sf.net/nco.html#mk_rec_dmn -->
@end html
@cindex record dimension
@cindex fixed dimension
@cindex fix record dimension
@cindex @code{--mk_rec_dmn @var{dim}}
@item --mk_rec_dmn @var{dim}
Change existing dimension @var{dim} to a record dimension in the output file.
This is the most straightforward way of changing a dimension to a/the
record dimension, and works fine in most cases.
See @ref{ncecat netCDF Ensemble Concatenator} and 
@ref{ncpdq netCDF Permute Dimensions Quickly} for other methods of
changing variable dimensionality, including the record dimension. 

@html
<a name="H"></a> <!-- http://nco.sf.net/nco.html#H -->
<a name="data"></a> <!-- http://nco.sf.net/nco.html#data -->
@end html
@cindex @code{-H}
@cindex @code{--data}
@cindex @code{--hieronymus}
@item -H 
Toggle (turn-on or turn-off) default behavior of printing data (not
metadata) to screen or copying data to disk.
Also activated using @samp{--print} or @samp{--prn}.
By default @command{ncks} prints all metadata but no data to screen 
when no netCDF @var{output-file} is specified.
And if @var{output-file} is specified, @command{ncks} copies all
metadata and all data to it.
In other words, the printing/copying default is context-sensitive,
and @samp{-H} toggles the default behavior.
Hence, use @samp{-H} to turn-off copying data (not metadata) to an
output file. 
(It is occasionally useful to write all metadata to a file, so that
the file has allocated the required disk space to hold the data, yet
to withold writing the data itself). 
And use @samp{-H} to turn-on printing data (not metadata) to screen.
Unless otherwise specified (with @code{-s}), each element of the data
hyperslab prints on a separate line containing the names, indices,
and, values, if any, of all of the variables dimensions.
The dimension and variable indices refer to the location of the
corresponding data element with respect to the variable as stored on
disk (i.e., not the hyperslab).
@example
% ncks --trd -C -v three_dmn_var in.nc
lat[0]=-90 lev[0]=100 lon[0]=0 three_dmn_var[0]=0 
lat[0]=-90 lev[0]=100 lon[1]=90 three_dmn_var[1]=1 
lat[0]=-90 lev[0]=100 lon[2]=180 three_dmn_var[2]=2 
...
lat[1]=90 lev[2]=1000 lon[1]=90 three_dmn_var[21]=21 
lat[1]=90 lev[2]=1000 lon[2]=180 three_dmn_var[22]=22 
lat[1]=90 lev[2]=1000 lon[3]=270 three_dmn_var[23]=23 
@end example
Printing the same variable with the @samp{-F} option shows the same
variable indexed with Fortran conventions
@example
% ncks -F -C -v three_dmn_var in.nc
lon(1)=0 lev(1)=100 lat(1)=-90 three_dmn_var(1)=0 
lon(2)=90 lev(1)=100 lat(1)=-90 three_dmn_var(2)=1 
lon(3)=180 lev(1)=100 lat(1)=-90 three_dmn_var(3)=2 
...
@end example
Printing a hyperslab does not affect the variable or dimension indices
since these indices are relative to the full variable (as stored in the
input file), and the input file has not changed.
However, if the hyperslab is saved to an output file and those values
are printed, the indices will change:
@c fxm: replace with new MSA output style
@example
% ncks --trd -H -d lat,90.0 -d lev,1000.0 -v three_dmn_var in.nc out.nc
...
lat[1]=90 lev[2]=1000 lon[0]=0 three_dmn_var[20]=20 
lat[1]=90 lev[2]=1000 lon[1]=90 three_dmn_var[21]=21 
lat[1]=90 lev[2]=1000 lon[2]=180 three_dmn_var[22]=22 
lat[1]=90 lev[2]=1000 lon[3]=270 three_dmn_var[23]=23 
% ncks --trd -C -v three_dmn_var out.nc
lat[0]=90 lev[0]=1000 lon[0]=0 three_dmn_var[0]=20 
lat[0]=90 lev[0]=1000 lon[1]=90 three_dmn_var[1]=21 
lat[0]=90 lev[0]=1000 lon[2]=180 three_dmn_var[2]=22 
lat[0]=90 lev[0]=1000 lon[3]=270 three_dmn_var[3]=23 
@end example

@html
<a name="jsn"></a> <!-- http://nco.sf.net/nco.html#jsn -->
<a name="json"></a> <!-- http://nco.sf.net/nco.html#json -->
<a name="JSON"></a> <!-- http://nco.sf.net/nco.html#JSON -->
@end html
@cindex @code{--jsn_fmt}
@cindex @code{--jsn}
@cindex @code{--json}
@cindex @acronym{JSN}
@cindex @acronym{JSON}
@cindex JavaScript
@item --jsn, --json
As of @acronym{NCO} version 4.6.2 (November, 2016), @command{ncks} can
print extracted metadata and data to screen (i.e., @code{stdout}) as
@acronym{JSON}, JavaScript Object Notation, defined 
@uref{http://www.json.org, here}.
@command{ncks} supports @acronym{JSON} output more completely, flexibly,
and robustly than any other tool to our knowledge.
With @command{ncks} one can translate entire netCDF3 and netCDF4 files
into @acronym{JSON}, including metadata and data, using all 
@acronym{NCO}'s subsetting and hyperslabbing capabilities.
@acronym{NCO} uses a @acronym{JSON} format we developed ourselves,
during a year of discussion among interested researchers.
Some refer to this format as @acronym{NCO-JSON}, to disambiguate it from 
other @acronym{JSON} formats for netCDF data.
Other projects have since adopted, and some can generate, 
@acronym{NCO-JSON}.
@cindex @acronym{ERDDAP}
@cindex @acronym{CF-JSON}
Projects that support @acronym{NCO-JSON} include @acronym{ERDDAP} 
(@url{https://coastwatch.pfeg.noaa.gov/erddap/index.html}, choose output
filetype @code{.ncoJson} from this 
@uref{https://coastwatch.pfeg.noaa.gov/erddap/griddap/documentation.html#fileType, table}) 
and @acronym{CF-JSON} (@url{http://cf-json.org}).

Behold @acronym{JSON} output in default mode:
@example
@verbatim
zender@aerosol:~$ ncks --jsn -v one ~/nco/data/in.nc
{
  "variables": {
    "one": {
      "type": "float",
      "attributes": {
        "long_name": "one"
      },
      "data": 1.0
    }
  }
}
@end verbatim
@end example
@acronym{NCO} converts to (using commonsense rules) and prints all
@acronym{NC_TYPE}s as one of three atomic types distinguishable as 
@acronym{JSON} values: @code{float}, @code{string}, and @code{int}
@footnote{The @acronym{JSON} boolean atomic type is not (yet) supported
as there is no obvious netCDF-equivalent to this type.}.
Floating-point types (@code{NC_FLOAT} and @code{NC_DOUBLE})
are printed with a decimal point and at least one signficant digit
following the decimal point, e.g., @code{1.0} rather than @code{1.} or
@code{1}.    
Integer types (e.g., @code{NC_INT}, @code{NC_UINT64}) are printed
with no decimal point. 
String types (@code{NC_CHAR} and @code{NC_STRING}) are enclosed 
in double-quotes.

The @acronym{JSON} specification allows many possible output formats for  
netCDF files.  
@acronym{NCO} developers implemented a working prototype in Octoboer,
2016 and, after discussing options with interested parties 
@uref{https://sourceforge.net/p/nco/discussion/9829/thread/8c4d7e72, here}, 
finalized the emitted @acronym{JSON} syntax a few weeks later.
The resulting @acronym{JSON} backend supports three levels of
pedanticness, ordered from more concise, flexible, and human-readable to 
more verbose, restrictive, and 1-to-1 reproducible.
@acronym{JSON}-specific switches access these modes and other features. 
Each @acronym{JSON} configuration option automatically triggers 
@acronym{JSON} printing, so that specifying @samp{--json} in addition to
a @acronym{JSON} configuration option is redundant and unnecessary. 

Request a specific format level with the pedantic level argument to
the @samp{--jsn_fmt @var{lvl}} option.
As of @acronym{NCO} version 4.6.3 (December, 2016), the option formerly
known as @samp{--jsn_att_fmt} was renamed simply @samp{--jsn_fmt}.
The more general name reflects the fact that the option controls
all @acronym{JSON} formatting, not just attribute formatting.
As of version 4.6.3, @acronym{NCO} defaults to demarcate inner 
dimensions of variable data with (nested) square brackets rather than 
printing data as an unrolled single dimensional array.
An array with C-ordered dimensionality [2,3,4] prints as:
@example
@verbatim
% ncks --jsn -v three_dmn_var ~/nco/data/in.nc
...
"data": [[[0.0, 1.0, 2.0, 3.0], [4.0, 5.0, 6.0, 7.0], [8.0, 9.0, 10.0,11.0]], [[12.0, 13.0, 14.0, 15.0], [16.0, 17.0, 18.0, 19.0], [20.0,21.0, 22.0, 23.0]]]
...
% ncks --jsn_fmt=4 -v three_dmn_var ~/nco/data/in.nc
...
"data": [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0, 11.0, 12.0, 13.0, 14.0, 15.0, 16.0, 17.0, 18.0, 19.0, 20.0, 21.0,22.0, 23.0]
...
@end verbatim
@end example
One can recover the former behavior (and omit the brackets) by adding
four to the base pedantic level @var{lvl} (as shown above).
Besides the potential offset of four, @var{lvl} may take one of three 
values between 0--2: 
@itemize @bullet
@item @math{@var{lvl} = 0} is the default mode, and is also explicitly
selectable with @samp{--jsn_fmt=0}.
All values are output without the original @acronym{NC_TYPE} token.
This allows attributes to print as @acronym{JSON} name-value pairs,
rather than as more complex objects: 
@example
@verbatim
% ncks --jsn_fmt=0 -v att_var ~/nco/data/in_grp.nc
...
"att_var": {
  "shape": ["time"],
  "type": "float",
  "attributes": {
    "byte_att": [0, 1, 2, 127, -128, -127, -2, -1],
    "char_att": "Sentence one.\nSentence two.\n",
    "short_att": 37,
    "int_att": 73,
    "long_att": 73,
    "float_att": [73.0, 72.0, 71.0, 70.010, 69.0010, 68.010, 67.010],
    "double_att": [73.0, 72.0, 71.0, 70.010, 69.0010, 68.010, 67.0100010]
  },
  "data": [10.0, 10.10, 10.20, 10.30, 10.40101, 10.50, 10.60, 10.70, 10.80, 10.990]
...
@end verbatim
@end example

This least pedantic mode produces the most easily read results, and
suffices for many (most?) purposes.  
Any downstream parser is expected to assign an appropriate type as
indicated by @acronym{JSON} syntax rules.
Because the original attributes' @code{NC_TYPE} are not output, 
a downstream parser may not exactly reproduce the input file datatypes. 
For example, whether the original attribute string was stored as
@code{NC_CHAR} or @code{NC_STRING} will be unknown to a downstream
parser.
Distinctions between @code{NC_FLOAT} and @code{NC_DOUBLE} are similarly
lost, as are all distinctions among the integer types.

In our experience, these distinctions are immaterial for attributes,
which are intended for metadata not for large-scale storage.   
Type-distinctions can, however, significantly impact the size of
variable data, responsible for nearly all the storage required by
datasets.
For instance, storing or transferring an @code{NC_SHORT} field as
@code{NC_DOUBLE} would waste a factor of four in space or bandwidth. 
This is why @acronym{NCO} @emph{always} prints the @code{NC_TYPE} 
of variable data.
Downstream parsers can (but are not required to) take advantage of the
variable's @code{NC_TYPE} to choose the most efficient storage type. 

The Shape member of the variable object is an ordered array of
dimension names such as @code{"shape": ["lat","lon"]}, similar to its
use in NcML.
Each name corresponds to a previously defined Dimension object
that, taken together, define the rank, shape, and size of the
variable. 
Variables are assumed to be scalar by default.
Hence the Shape member is mandatory for arrays, and is always omitted
for scalars (by contrast, NcML requires an empty shape string to
indicate scalars).  

@item @math{@var{lvl} = 1} is a medium-pedantic level that prints all
attributes as objects (with explicit types) unless the attribute type
match the simplest default @acronym{JSON} value types.
In other words, attributes of type @code{NC_FLOAT}, @code{NC_CHAR},
@code{NC_SHORT}, and @code{NC_BYTE} are printed as objects with an
explicit type so that parsers do not use the default type. 
Attributes of type @code{NC_DOUBLE}, @code{NC_STRING}, and @code{NC_INT}
are printed as @acronym{JSON} arrays, as in the @math{@var{lvl} = 0}
above: 
@example
@verbatim
% ncks --jsn_fmt=1 -v att_var ~/nco/data/in.nc
...
"att_var": {
  "shape": ["time"],
  "type": "float",
  "attributes": {
    "byte_att": { "type": "byte", "data": [0, 1, 2, 127, -128, -127, -2, -1]},
    "char_att": "Sentence one.\nSentence two.\n",
    "short_att": { "type": "short", "data": 37},
    "int_att": 73,
    "long_att": 73,
    "float_att": [73.0, 72.0, 71.0, 70.010, 69.0010, 68.010, 67.010],
    "double_att": { "type": "double", "data": [73.0, 72.0, 71.0, 70.010, 69.0010, 68.010, 67.0100010]}
  },
  "data": [10.0, 10.10, 10.20, 10.30, 10.40101, 10.50, 10.60, 10.70, 10.80, 10.990]
...
@end verbatim
@end example
The attributes of type @code{NC_BYTE}, @code{NC_SHORT}, and
@code{NC_DOUBLE} are printed as @acronym{JSON} objects that comprise an
@code{NC_TYPE} and a value list, because their values could conceivably 
not be representable, or would waste space, if interpreted as
@code{NC_INT} or @code{NC_FLOAT}, respectively.
All other attributes may be naturally mapped to the type indicated by
the @acronym{JSON} syntax of the value, where numbers are assumed to
correspond to @code{NC_FLOAT} for floating-point, @code{NC_INT} for
integers, and @code{NC_CHAR} or @code{NC_STRING} for strings.
This minimal increase in verbosity allows a downstream parser to
re-construct the original dataset with nearly identical attributes types
to the original.

@item @math{@var{lvl} = 2} is the most pedantic mode, and should be used
when preserving all input types (e.g., to ensure exact reproducibility
of the input file) is important. 
This mode always prints attributes as @acronym{JSON} objects with a type
value so that any downstream parser can (though it need not) guarantee
exact reproduction of the original dataset:
@example
@verbatim
% ncks --jsn_fmt=2 -v att_var ~/nco/data/in.nc
...
"att_var": {
  "shape": ["time"],
  "type": "float",
  "attributes": {
    "byte_att": { "type": "byte", "data": [0, 1, 2, 127, -128, -127, -2, -1]},
    "char_att": { "type": "char", "data": "Sentence one.\nSentence two.\n"},
    "short_att": { "type": "short", "data": 37},
    "int_att": { "type": "int", "data": 73},
    "long_att": { "type": "int", "data": 73},
    "float_att": { "type": "float", "data": [73.0, 72.0, 71.0, 70.010, 69.0010, 68.010, 67.010]},
    "double_att": { "type": "double", "data": [73.0, 72.0, 71.0, 70.010, 69.0010, 68.010, 67.0100010]}
  },
  "data": [10.0, 10.10, 10.20, 10.30, 10.40101, 10.50, 10.60, 10.70, 10.80, 10.990]
...
@end verbatim
@end example
@end itemize

@cindex @command{jsonlint}
That @acronym{ncks} produces correct translations of for all supported
datatypes may be verified by a @acronym{JSON} syntax checker command
like @command{jsonlint}. 
Please let us know how to improve @acronym{JSON} features for your
application.  

There 

@html
<a name="Metadata"></a> <!-- http://nco.sf.net/nco.html#Metadata -->
<a name="M"></a> <!-- http://nco.sf.net/nco.html#M -->
@end html
@cindex @code{-M}
@cindex @code{--Mtd}
@cindex @code{--Metadata}
@cindex metadata, global 
@item -M
Turn-on printing to screen or turn-off copying global and group metadata.
This includes file summary information and global and group attributes.
Also @samp{--Mtd} and @samp{--Metadata}.
By default @command{ncks} prints global metadata to screen if no netCDF
output file and no variable extraction list is specified (with @samp{-v}).  
Use @samp{-M} to print global metadata to screen if a netCDF output is 
specified, or if a variable extraction list is specified (with @samp{-v}). 
Use @samp{-M} to turn-off copying of global and group metadata when
copying, subsetting, or appending to an output file.

@html
<a name="prn_tbl"></a> <!-- http://nco.sf.net/nco.html#prn_tbl -->
@end html
The various combinations of printing switches can be confusing.
In an attempt to anticipate what most users want to do, @command{ncks}
uses context-sensitive defaults for printing.
Our goal is to minimize the use of switches required to accomplish the
common operations.
We assume that users creating a new file or overwriting (e.g., with
@samp{-O}) an existing file usually wish to copy all global and
variable-specific attributes to the new file.
In contrast, we assume that users appending (e.g., with @samp{-A} an
explicit variable list from one file to another usually wish to copy
only the variable-specific attributes to the output file.
The switches @samp{-H}, @samp{-M}, and @samp{-m} switches are
implemented as toggles which reverse the default behavior.
The most confusing aspect of this is that @samp{-M} inhibits copying
global metadata in overwrite mode and causes copying of global
metadata in append mode.
@example
ncks                 in.nc        # Print  VAs and GAs
ncks          -v one in.nc        # Print  VAs not GAs
ncks    -M    -v one in.nc        # Print  GAs only
ncks       -m -v one in.nc        # Print  VAs only
ncks    -M -m -v one in.nc        # Print  VAs and GAs
ncks -O              in.nc out.nc # Copy   VAs and GAs
ncks -O       -v one in.nc out.nc # Copy   VAs and GAs
ncks -O -M    -v one in.nc out.nc # Copy   VAs not GAs
ncks -O    -m -v one in.nc out.nc # Copy   GAs not VAs
ncks -O -M -m -v one in.nc out.nc # Copy   only data (no atts)
ncks -A              in.nc out.nc # Append VAs and GAs
ncks -A       -v one in.nc out.nc # Append VAs not GAs
ncks -A -M    -v one in.nc out.nc # Append VAs and GAs
ncks -A    -m -v one in.nc out.nc # Append only data (no atts)
ncks -A -M -m -v one in.nc out.nc # Append GAs not VAs
@end example
where @code{VAs} and @code{GAs} denote variable and group/global
attributes, respectively. 

@html
<a name="-m"></a> <!-- http://nco.sf.net/nco.html#-m -->
<a name="m"></a> <!-- http://nco.sf.net/nco.html#m -->
<a name="mtd"></a> <!-- http://nco.sf.net/nco.html#mtd -->
<a name="metadata"></a> <!-- http://nco.sf.net/nco.html#metadata -->
@end html
@cindex @command{ncdump}
@cindex @code{-m}
@cindex @code{--mtd}
@cindex @code{--metadata}
@cindex metadata
@item -m
Turn-on printing to screen or turn-off copying variable metadata.
Using @samp{-m} will print variable metadata to screen (similar to
@kbd{ncdump -h}).  
This displays all metadata pertaining to each variable, one variable
at a time.
@cindex chunking
@cindex compression
@cindex deflation
This includes information on the storage properties of the variable,
such as whether it employs chunking, compression, or packing.
Also activated using @samp{--mtd} and @samp{--metadata}.
The @command{ncks} default behavior is to print variable metadata to
screen if no netCDF output file is specified.
Use @samp{-m} to print variable metadata to screen if a netCDF output is  
specified. 
Also use @samp{-m} to turn-off copying of variable metadata to an output
file.

@html
<a name="no_blank"></a> <!-- http://nco.sf.net/nco.html#no_blank -->
<a name="noblank"></a> <!-- http://nco.sf.net/nco.html#noblank -->
<a name="no-blank"></a> <!-- http://nco.sf.net/nco.html#no-blank -->
@end html
@cindex @code{--no_blank}
@cindex @code{--noblank}
@cindex @code{--no-blank}
@cindex blank
@cindex missing values
@item --no_blank
Print numeric representation of missing values.
As of @acronym{NCO} version 4.2.2 (October, 2012), @acronym{NCO} prints
missing values as blanks (i.e., the underscore character @samp{_}) by default.
To enable the old behavior of printing the numeric representation of
missing values (e.g., @code{1.0e36}), use the @samp{--no_blank} switch.
Also activated using @samp{--noblank} or @samp{--no-blank}.

@html
<a name="P"></a> <!-- http://nco.sf.net/nco.html#P -->
<a name="prn"></a> <!-- http://nco.sf.net/nco.html#prn -->
@end html
@cindex @code{-P}
@cindex @code{--print}
@cindex @code{--prn}
@item -P 
Print data, metadata, and units to screen.
The @samp{-P} switch is a convenience abbreviation for 
@samp{-C -H -M -m -u}.
Also activated using @samp{--print} or @samp{--prn}.
This set of switches is useful for exploring file contents.

@html
<a name="fl_prn"></a> <!-- http://nco.sf.net/nco.html#fl_prn -->
<a name="prn_fl"></a> <!-- http://nco.sf.net/nco.html#prn_fl -->
@end html
@cindex print file
@cindex @code{--fl_prn}
@cindex @code{--prn_fl}
@item --prn_fl @file{print-file}
Activate printing formatted output to file @file{print-file}.
Also @samp{--print_file}, @samp{--fl_prn}, and @samp{--file_print}.
One can achieve the same result by redirecting stdout to a named file.
However, it is slightly faster to print formatted output directly to a
file than to stdout:
@example
ncks --fl_prn=foo.txt --jsn in.nc
@end example

@html
<a name="Q"></a> <!-- http://nco.sf.net/nco.html#Q -->
<a name="quiet"></a> <!-- http://nco.sf.net/nco.html#quiet -->
@end html
@cindex @code{-Q}
@cindex @code{--quiet}
@item -Q
Print quietly, meaning omit dimension names, indices, and coordinate
values when printing arrays. 
Variable (not dimension) indices are printed.
Variable names appear flush left in the output:
@example
@verbatim
zender@roulee:~$ ncks --trd -Q -v three_dmn_rec_var -C -H ~/nco/data/in.nc              
three_dmn_rec_var[0]=1 
...
@end verbatim
@end example
This helps locate specific variables in lists with many variables and 
different dimensions. 
See also the @samp{-V} option, which omits all names and indices and
prints only variable values. 

@html
<a name="q"></a> <!-- http://nco.sf.net/nco.html#q -->
<a name="quench"></a> <!-- http://nco.sf.net/nco.html#quench -->
@end html
@cindex @code{-q}
@cindex @code{--quench}
@cindex quench
@item -q 
Quench (turn-off) all printing to screen.
This overrides the setting of all print-related switches, equivalent to
@kbd{-H -M -m} when in single-file printing mode. 
When invoked with @code{-R} (@pxref{Retaining Retrieved Files}), @command{ncks}
automatically sets @code{-q}. 
This allows @command{ncks} to retrieve remote files without
automatically trying to print them.
Also @samp{--quench}.

@html
<a name="rad"></a> <!-- http://nco.sf.net/nco.html#rad -->
<a name="orphan"></a> <!-- http://nco.sf.net/nco.html#orphan -->
<a name="rph_dmn"></a> <!-- http://nco.sf.net/nco.html#rph_dmn -->
@end html
@cindex @code{--rad}
@cindex orphan dimensions
@cindex @code{--retain_all_dimensions}
@cindex @code{--orphan_dimensions}
@cindex @code{--rph_dmn}
@item --rad
Retain all dimensions.
When invoked with @code{--rad} (Retain All Dimensions),
@command{ncks} copies each dimension in the input file to the output
file, regardless of whether the dimension is utilized by any variables. 
Normally @command{ncks} discards ``orphan dimensions'', i.e., dimensions
not referenced by any variables.
This switch allows users to keep non-referenced dimensions in the workflow.
When invoked in printing mode, causes orphaned dimensions to be printed
(they are not printed by default).
Also @samp{--retain_all_dimensions}, @samp{--orphan_dimensions}, and
@samp{--rph_dmn}. 

@html
<a name="s"></a> <!-- http://nco.sf.net/nco.html#s -->
<a name="sng_fmt"></a> <!-- http://nco.sf.net/nco.html#sng_fmt -->
<a name="string"></a> <!-- http://nco.sf.net/nco.html#string -->
@end html
@cindex @code{-s}
@cindex @code{--string}
@cindex @code{--sng_fmt}
@cindex @code{printf()}
@cindex C language
@item -s @var{format}
String format for text output. 
Accepts @w{C language} escape sequences and @code{printf()} formats. 
Also @samp{--string}  and @samp{--sng_fmt}.
This option is only intended for use with traditional (@acronym{TRD})
printing, and thus automatically invokes the @samp{--trd} switch.

@html
<a name="fmt_val"></a> <!-- http://nco.sf.net/nco.html#fmt_val -->
<a name="val_fmt"></a> <!-- http://nco.sf.net/nco.html#val_fmt -->
<a name="value_format"></a> <!-- http://nco.sf.net/nco.html#value_format -->
@end html
@cindex @code{--value_format}
@cindex @code{--fmt_val}
@cindex @code{--val_fmt}
@cindex @code{printf()}
@cindex C language
@item --fmt_val @var{format}
Supply a @code{printf()}-style format for printed output, i.e., in
@acronym{CDL}, @acronym{JSON}, @acronym{TRD}, or @acronym{XML} modes. 
Also @samp{--val_fmt} and @samp{--value_format}.
One use for this option is to reduce the printed precision of floating
point values: 
@example
# Default printing of original double precision values
# 0.0,0.1,0.12,0.123,0.1234,0.12345,0.123456,0.1234567,0.12345678,0.123456789
% ncks -C -v ppc_dbl ~/nco/data/in.nc
...
ppc_dbl = 0, 0.1, 0.12, 0.123, 0.1234, 0.12345, 0.123456, 0.1234567, 0.12345678, 0.123456789 ;
...
# Restrict printing to three digits after the decimal
% ncks --fmt_val=%.3f -C -v ppc_dbl ~/nco/data/in.nc
...
ppc_dbl = 0., 0.1, 0.12, 0.123, 0.123, 0.123, 0.123, 0.123, 0.123, 0.123 ;
...
@end example
@noindent
The supplied @var{format} only applies to floating point variable values 
(@code{NC_FLOAT} or @code{NC_DOUBLE}), and not to other types or to
attributes.
For reference, the default @code{printf()} @var{format} for
@acronym{CDL}, @acronym{JSON}, @acronym{TRD}, and @acronym{XML} modes is
@code{%#.7gf}, @code{%#.7g},  @code{%g},    and @code{%#.7g},
respectively, for single-precision data, and, for double-precision data is
@code{%#.15g}, @code{%#.15g}, @code{%.12g}, and @code{%#.15g},
respectively.
@acronym{NCO} introduced this feature in version 4.7.3 (March, 2018).
We would appreciate your feedback on whether and how to extend this
feature to make it more useful. 

@html
<a name="ssh"></a> <!-- http://nco.sf.net/nco.html#ssh -->
<a name="scr"></a> <!-- http://nco.sf.net/nco.html#scr -->
<a name="secret"></a> <!-- http://nco.sf.net/nco.html#secret -->
@end html
@cindex @code{--ssh}
@cindex @code{--scr}
@cindex @code{--secret}
@cindex hidden features
@item --secret
Print summary of @command{ncks} hidden features.
These hidden or secret features are used mainly by developers.
They are not supported for general use and may change at any time.
This demonstrates conclusively that I cannot keep a secret.
Also @samp{--ssh} and @samp{--scr}. 

@html
<a name="trd"></a> <!-- http://nco.sf.net/nco.html#trd -->
<a name="traditional"></a> <!-- http://nco.sf.net/nco.html#traditional -->
@end html
@cindex @code{--trd}
@cindex @code{--traditional}
@item --trd, --traditional
From 1995--2017 @command{ncks} dumped the @acronym{ASCII} text
representation of netCDF files in what we now call ``traditional''
mode. 
Much of this manual contains output printed in traditional mode,
which places one value per line, with complete dimensional
information.
Traditional-mode metadata output includes lower-level information,
such as @acronym{RAM} usage and internal variable IDs, than
@acronym{CDL}.  
While this is useful for some developers and user, @acronym{CDL} has,
over the years, become more useful than traditional mode for most
users.  
As of @acronym{NCO} version 4.6.9 (September, 2017) @acronym{CDL} became 
the default printing mode. 
Traditional printing mode is accessed via the @samp{--trd} option.  

@html
<a name="units"></a> <!-- http://nco.sf.net/nco.html#units -->
<a name="u"></a> <!-- http://nco.sf.net/nco.html#u -->
@end html
@cindex @code{-u}
@cindex @code{--units}
@item -u 
Toggle the printing of a variable's @code{units} attribute, if any, 
with its values.
Also @samp{--units}.

@html
<a name="V"></a> <!-- http://nco.sf.net/nco.html#V -->
<a name="val_var"></a> <!-- http://nco.sf.net/nco.html#val_var -->
@end html
@cindex @code{-V}
@cindex @code{--val_var}
@cindex @code{--no_dmn_var_nm}
@cindex @code{--no_nm_prn}
@item -V
Print variable values only.
Do not print variable and dimension names, indices, and coordinate
values when printing arrays. 
@example
@verbatim
zender@roulee:~$ ncks --trd -V -v three_dmn_rec_var -C -H ~/nco/data/in.nc
1
...
@end verbatim
@end example
See also the @samp{-Q} option, which prints variable names and indices,
but not dimension names, indices, or coordinate values when printing
arrays. 
Using @samp{-V} is the same as specifying @samp{-Q --no_nm_prn}.

@html
<a name="xml"></a> <!-- http://nco.sf.net/nco.html#xml -->
<a name="ncml"></a> <!-- http://nco.sf.net/nco.html#ncml -->
@end html
@cindex @code{--xml}
@cindex @code{--ncml}
@cindex @command{ncdump}
@cindex @acronym{XML}
@cindex @acronym{NcML}
@item --xml, --ncml
As of @acronym{NCO} version 4.3.3 (July, 2013), @command{ncks} can
print extracted data and metadata to screen (i.e., @code{stdout}) as 
@acronym{XML} in @acronym{NcML}, the netCDF Markup Language.  
@command{ncks} supports @acronym{XML} more completely than 
@samp{ncdump -x}.
With @command{ncks} one can translate entire netCDF3 and netCDF4 files
into @acronym{NcML}, including metadata and data, using all 
@acronym{NCO}'s subsetting and hyperslabbing capabilities.
Compare @command{ncks} ``traditional'' with @acronym{XML} printing:
@example
@verbatim
zender@roulee:~$ ncks --trd -v one ~/nco/data/in.nc
one: type NC_FLOAT, 0 dimensions, 1 attribute, chunked? no, compressed? no, packed? no
one size (RAM) = 1*sizeof(NC_FLOAT) = 1*4 = 4 bytes
one attribute 0: long_name, size = 3 NC_CHAR, value = one

one = 1 

zender@roulee:~$ ncks --xml -v one ~/nco/data/in.nc
<?xml version="1.0" encoding="UTF-8"?>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2" location="/home/zender/nco/data/in.nc">
  <variable name="one" type="float" shape="">
    <attribute name="long_name" separator="*" value="one" />
    <values>1.</values>
  </variable>
</netcdf>
@end verbatim
@end example
@acronym{XML}-mode prints variable metadata and, as of 
@acronym{NCO} version 4.3.7 (October, 2013), variable data and, as of
@acronym{NCO} version 4.4.0 (January, 2014), hidden attributes.
That @acronym{ncks} produces correct @acronym{NcML} translations of
@acronym{CDM} files for all supported datatypes is verified by
comparison to output from Unidata's @command{toolsUI} Java program.
Please let us know how to improve @acronym{XML}/@acronym{NcML}
features. 

@cindex @code{--xml_no_location}
@cindex @code{--xml_spr_chr}
@cindex @code{--xml_spr_nmr}
@cindex separator
@command{ncks} provides additional options to configure @acronym{NcML}
output: @samp{--xml_no_location}, @samp{--xml_spr_chr}, and
@samp{--xml_spr_nmr}. 
Every @acronym{NcML} configuration option automatically triggers
@acronym{NcML} printing, so that specifying @samp{--xml} in addition
to a configuration option is redundant and unnecessary.
The @samp{--xml_no_location} switch prevents output of the
@acronym{NcML} @code{location} element.
By default the location element is printed with a value equal to the
location of the input dataset, e.g.,
@code{location="/home/zender/in.nc"}.
The @samp{--xml_spr_chr} and @samp{--xml_spr_nmr} options customize
the strings used as @acronym{NcML} separators for attributes and
variables of character-type and numeric-type, respectively.
Their default separators are @code{*} and ``@code{ }'' (a space):
@example
@verbatim
zender@roulee:~$ ncks --xml -d time,0,3 -v two_dmn_rec_var_sng in.nc
...
   <values separator="*">abc*bcd*cde*def</values>
 ...
 zender@roulee:~$ ncks --xml_spr_chr=', ' -v two_dmn_rec_var_sng in.nc
...
<values separator=", ">abc, bcd, cde, def, efg, fgh, ghi, hij, jkl, klm</values>
...
zender@roulee:~$ ncks --xml -v one_dmn_rec_var in.nc
...
<values>1 2 3 4 5 6 7 8 9 10</values>
...
zender@roulee:~$ ncks --xml_spr_nmr=', ' -v one_dmn_rec_var in.nc
...
<values separator=", ">1, 2, 3, 4, 5, 6, 7, 8, 9, 10</values>
...
@end verbatim
@end example
Separator elements for strings are a thorny issue.
One must be sure that the separator element is not mistaken as a portion
of the string. 
@acronym{NCO} attempts to produce valid @acronym{NcML} and supplies the
@samp{--xml_spr_chr} option to work around any difficulties.
@acronym{NCO} performs precautionary checks with
@code{strstr(@var{val},@var{spr})} to identify presence of the separator
string (@var{spr}) in data (@var{val}) and, when it detects a match,
automatically switches to a backup separator string (@code{*|*}). 
However limitations of @code{strstr()} may lead to false negatives 
when the separator string occurs in data beyond the first string in
multi-dimensional @code{NC_CHAR} arrays. 
Hence, results may be ambiguous to NcML parsers. 
If problems arise, use @samp{--xml_spr_chr} to specify a multi-character
separator that does not appear in the string array and that does not
include an NcML formatting characters (e.g., commas, angles, quotes).
@end table

@html
<a name="ncattget"></a> <!-- http://nco.sf.net/nco.html#ncattget -->
<a name="nclst"></a> <!-- http://nco.sf.net/nco.html#nclst -->
<a name="ncvarlst"></a> <!-- http://nco.sf.net/nco.html#ncvarlst -->
<a name="ncvardmnlst"></a> <!-- http://nco.sf.net/nco.html#ncvardmnlst -->
<a name="ncdmnlst"></a> <!-- http://nco.sf.net/nco.html#ncdmnlst -->
<a name="ncdmnsz"></a> <!-- http://nco.sf.net/nco.html#ncdmnsz -->
<a name="ncgrplst"></a> <!-- http://nco.sf.net/nco.html#ncgrplst -->
<a name="ncrecsz"></a> <!-- http://nco.sf.net/nco.html#ncrecsz -->
<a name="ncmax"></a> <!-- http://nco.sf.net/nco.html#ncmax -->
<a name="ncmdn"></a> <!-- http://nco.sf.net/nco.html#ncmdn -->
<a name="ncavg"></a> <!-- http://nco.sf.net/nco.html#ncavg -->
<a name="ncrng"></a> <!-- http://nco.sf.net/nco.html#ncrng -->
<a name="ncunits"></a> <!-- http://nco.sf.net/nco.html#ncunits -->
<a name="alias"></a> <!-- http://nco.sf.net/nco.html#alias -->
<a name="filters"></a> <!-- http://nco.sf.net/nco.html#filters -->
<a name="filter"></a> <!-- http://nco.sf.net/nco.html#filter -->
<a name="flt"></a> <!-- http://nco.sf.net/nco.html#flt -->
@end html
@menu
* Filters for @command{ncks}::
@end menu

@node Filters for @command{ncks},  , ncks netCDF Kitchen Sink, ncks netCDF Kitchen Sink
@subsection Filters for @command{ncks}
@cindex @acronym{UNIX}
@cindex @command{ncattget}
@cindex @command{ncavg}
@cindex @command{ncdmnlst}
@cindex @command{ncvardmnlst}
@cindex @command{ncvardmnlatlon}
@cindex @command{ncdmnsz}
@cindex @command{ncgrplst}
@cindex @command{nclst}
@cindex @command{nclst}
@cindex @command{ncmax}
@cindex @command{ncmdn}
@cindex @command{ncmin}
@cindex @command{ncrecsz}
@cindex @command{ncrng}
@cindex @command{nctypget}
@cindex @command{ncunits}
@cindex @file{.bashrc}
@cindex filters
@cindex alias
@cindex shell
@cindex Bash shell
@cindex Csh shell
@cindex Sh shell
@cindex @command{bash}
We encourage the use of standard @acronym{UNIX} pipes and filters to
narrow the verbose output of @command{ncks} into more precise targets.
For example, to obtain an uncluttered listing of the variables in a file
try 
@example
ncks --trd -m in.nc | grep -E ': type' | cut -f 1 -d ' ' | sed 's/://' | sort
@end example
A Bash user could alias the previous filter to the shell command
@command{ncvarlst} as shown below.
More complex examples could involve command line arguments.
For example, a user may frequently be interested in obtaining the value
of an attribute, e.g., for textual file examination or for passing to
another shell command.
Say the attribute is @code{purpose}, the variable is @code{z}, and the
file is @code{in.nc}.
In this example, @command{ncks --trd -m -v z} is too verbose so a robust
@command{grep} and @command{cut} filter is desirable, such as
@example
ncks --trd -M -m in.nc | grep -E -i "^z attribute [0-9]+: purpose" | cut -f 11- -d ' ' | sort
@end example
The filters are clearly too complex to remember on-the-fly so the entire 
procedure could be implemented as a shell command or function called,
say, @command{ncattget}
@example
@verbatim
function ncattget { ncks --trd -M -m ${3} | grep -E -i "^${2} attribute [0-9]+: ${1}" | cut -f 11- -d ' ' | sort ; }
@end verbatim
@end example
The shell @command{ncattget} is invoked with three arugments that are,
in order, the names of the attribute, variable, and file to examine.
Global attributes are indicated by using a variable name of @code{global}.
This definition yields the following results
@example
% ncattget purpose z in.nc
Height stored with a monotonically increasing coordinate
% ncattget Purpose Z in.nc
Height stored with a monotonically increasing coordinate
% ncattget history z in.nc
% ncattget history global in.nc
History global attribute.
@end example
Note that case sensitivity has been turned off for the variable and
attribute names (and could be turned on by removing the @samp{-i} switch
to @command{grep}).
Furthermore, extended regular expressions may be used for both the
variable and attribute names.
The next two commands illustrate this by searching for the values
of attribute @code{purpose} in all variables, and then for all
attributes of the variable @code{z}:
@example
% ncattget purpose .+ in.nc
1-D latitude coordinate referred to by geodesic grid variables
1-D longitude coordinate referred to by geodesic grid variables
...
% ncattget .+ Z in.nc
Height
Height stored with a monotonically increasing coordinate
meter
@end example

Extended filters are best stored as shell commands if they are
used frequently.
Shell commands may be re-used when they are defined in shell
configuration files.
These files are usually named @file{.bashrc}, @file{.cshrc}, and
@file{.profile} for the Bash, Csh, and Sh shells, respectively.
@cindex minimum
@cindex maximum
@cindex median
@cindex mode
@cindex range
@html
<a name="median"></a> <!-- http://nco.sf.net/nco.html#median -->
<a name="mode"></a> <!-- http://nco.sf.net/nco.html#mode -->
<a name="range"></a> <!-- http://nco.sf.net/nco.html#range -->
@end html
@example
@verbatim
# NB: Untested on Csh, Ksh, Sh, Zsh! Send us feedback!
# Bash shell (/bin/bash), .bashrc examples
# ncattget $att_nm $var_nm $fl_nm : What attributes does variable have?
function ncattget { ncks --trd -M -m ${3} | grep -E -i "^${2} attribute [0-9]+: ${1}" | cut -f 11- -d ' ' | sort ; }
# ncunits $att_val $fl_nm : Which variables have given units?
function ncunits { ncks --trd -m ${2} | grep -E -i " attribute [0-9]+: units.+ ${1}" | cut -f 1 -d ' ' | sort ; }
# ncavg $var_nm $fl_nm : What is mean of variable?
function ncavg { ncwa -y avg -O -C -v ${1} ${2} ~/foo.nc ; ncks --trd -H -C -v ${1} ~/foo.nc | cut -f 3- -d ' ' ; }
# ncavg $var_nm $fl_nm : What is mean of variable?
function ncavg { ncap2 -O -C -v -s "foo=${1}.avg();print(foo)" ${2} ~/foo.nc | cut -f 3- -d ' ' ; }
# ncdmnlst $fl_nm : What dimensions are in file?
function ncdmnlst { ncks --cdl -m ${1} | cut -d ':' -f 1 | cut -d '=' -s -f 1 ; }
# ncvardmnlst $var_nm $fl_nm : What dimensions are in a variable?
function ncvardmnlst { ncks --trd -m -v ${1} ${2} | grep -E -i "^${1} dimension [0-9]+: " | cut -f 4 -d ' ' | sed 's/,//' ; }
# ncvardmnlatlon $var_nm $fl_nm : Does variable contain both lat and lon dimensions?
function ncvardmnlatlon { flg=`ncks -C -v ${1} -m ${2} | grep -E -i "${1}\(" | grep -E "lat.*lon|lon.*lat"` ; [[ ! -z "$flg" ]] && echo "Yes, ${1} has both lat and lon dimensions" || echo "No, ${1} does not have both lat and lon dimensions" }
# ncdmnsz $dmn_nm $fl_nm : What is dimension size?
function ncdmnsz { ncks --trd -m -M ${2} | grep -E -i ": ${1}, size =" | cut -f 7 -d ' ' | uniq ; }
# ncgrplst $fl_nm : What groups are in file?
function ncgrplst { ncks -m ${1} | grep 'group:' | cut -d ':' -f 2 | cut -d ' ' -f 2 | sort ; }
# ncvarlst $fl_nm : What variables are in file?
function ncvarlst { ncks --trd -m ${1} | grep -E ': type' | cut -f 1 -d ' ' | sed 's/://' | sort ; }
# ncmax $var_nm $fl_nm : What is maximum of variable?
function ncmax { ncwa -y max -O -C -v ${1} ${2} ~/foo.nc ; ncks --trd -H -C -v ${1} ~/foo.nc | cut -f 3- -d ' ' ; }
# ncmax $var_nm $fl_nm : What is maximum of variable?
function ncmax { ncap2 -O -C -v -s "foo=${1}.max();print(foo)" ${2} ~/foo.nc | cut -f 3- -d ' ' ; }
# ncmdn $var_nm $fl_nm : What is median of variable?
function ncmdn { ncap2 -O -C -v -s "foo=gsl_stats_median_from_sorted_data(${1}.sort());print(foo)" ${2} ~/foo.nc | cut -f 3- -d ' ' ; }
# ncmin $var_nm $fl_nm : What is minimum of variable?
function ncmin { ncap2 -O -C -v -s "foo=${1}.min();print(foo)" ${2} ~/foo.nc | cut -f 3- -d ' ' ; }
# ncrng $var_nm $fl_nm : What is range of variable?
function ncrng { ncap2 -O -C -v -s "foo_min=${1}.min();foo_max=${1}.max();print(foo_min,\"%f\");print(\" to \");print(foo_max,\"%f\")" ${2} ~/foo.nc ; }
# ncmode $var_nm $fl_nm : What is mode of variable?
function ncmode { ncap2 -O -C -v -s "foo=gsl_stats_median_from_sorted_data(${1}.sort());print(foo)" ${2} ~/foo.nc | cut -f 3- -d ' ' ; }
# ncrecsz $fl_nm : What is record dimension size?
function ncrecsz { ncks --trd -M ${1} | grep -E -i "^Root record dimension 0:" | cut -f 10- -d ' ' ; }
# nctypget $var_nm $fl_nm : What type is variable?
function nctypget { ncks --trd -m -v ${1} ${2} | grep -E -i "^${1}: type" | cut -f 3 -d ' ' | cut -f 1 -d ',' ; }

# Csh shell (/bin/csh), .cshrc examples (derive others from Bash definitions):
ncattget() { ncks --trd -M -m -v ${3} | grep -E -i "^${2} attribute [0-9]+: ${1}" | cut -f 11- -d ' ' | sort ; }
ncdmnsz() { ncks --trd -m -M ${2} | grep -E -i ": ${1}, size =" | cut -f 7 -d ' ' | uniq ; }
ncvarlst() { ncks --trd -m ${1} | grep -E ': type' | cut -f 1 -d ' ' | sed 's/://' | sort ; }
ncrecsz() { ncks --trd -M ${1} | grep -E -i "^Record dimension:" | cut -f 8- -d ' ' ; }

# Sh shell (/bin/sh), .profile examples (derive others from Bash definitions):
ncattget() { ncks --trd -M -m ${3} | grep -E -i "^${2} attribute [0-9]+: ${1}" | cut -f 11- -d ' ' | sort ; }
ncdmnsz() { ncks --trd -m -M ${2} | grep -E -i ": ${1}, size =" | cut -f 7 -d ' ' | uniq ; }
ncvarlst() { ncks --trd -m ${1} | grep -E ': type' | cut -f 1 -d ' ' | sed 's/://' | sort ; }
ncrecsz() { ncks --trd -M ${1} | grep -E -i "^Record dimension:" | cut -f 8- -d ' ' ; }
@end verbatim
@end example

@noindent
@html
<a name="xmp_ncks"></a> <!-- http://nco.sf.net/nco.html#xmp_ncks -->
@end html
EXAMPLES

View all data in netCDF @file{in.nc}, printed with Fortran indexing
conventions: 
@example
ncks -F in.nc
@end example

Copy the netCDF file @file{in.nc} to file @file{out.nc}.
@example
ncks in.nc out.nc
@end example
Now the file @file{out.nc} contains all the data from @file{in.nc}.
There are, however, two differences between @file{in.nc} and
@file{out.nc}.
@cindex @code{history}
First, the @code{history} global attribute (@pxref{History Attribute})
will contain the command used to create @file{out.nc}.
@cindex alphabetize output
@cindex sort alphabetically
@cindex @code{-a}
Second, the variables in @file{out.nc} will be defined in alphabetical
order.
Of course the internal storage of variable in a netCDF file should be
transparent to the user, but there are cases when alphabetizing a file 
is useful (see description of @code{-a} switch).

@html
<a name="xmp_att_glb_cpy"></a> <!-- http://nco.sf.net/nco.html#xmp_att_glb_cpy -->
@end html
@cindex global attributes
@cindex attributes, global
@cindex subsetting
@cindex exclusion
@cindex extraction
@cindex @code{-v @var{var}}
@cindex @code{--variable @var{var}}
@cindex @code{-x}
@cindex @code{--exclude}
@cindex @code{--xcl}
Copy all global attributes (and no variables) from @file{in.nc} to
@file{out.nc}: 
@example
ncks -A -x ~/nco/data/in.nc ~/out.nc
@end example
The @samp{-x} switch tells @acronym{NCO} to use the complement of the
extraction list (@pxref{Subsetting Files}). 
Since no extraction list is explicitly specified (with @samp{-v}),
the default is to extract all variables.
The complement of all variables is no variables.
@cindex @code{-A}
@cindex @code{--apn}
@cindex @code{--append}
@cindex appending to files
Without any variables to extract, the append (@samp{-A}) command
(@pxref{Appending Variables}) has only to extract and copy
(i.e., append) global attributes to the output file.

@html
<a name="xmp_att_glb_cpy"></a> <!-- http://nco.sf.net/nco.html#xmp_att_var_cpy -->
@end html
Copy/append metadata (not data) from variables in one file to
variables in a second file.
When copying/subsetting/appending files (as opposed to printing them),
the copying of data, variable metadata, and global/group metadata are
now turned OFF by @samp{-H}, @samp{-m}, and @samp{-M}, respectively. 
This is the opposite sense in which these switches work when
@emph{printing} a file. 
One can use these switches to easily replace data or metadata in one
file with data or metadata from another:
@example
# Extract naked (data-only) copies of two variables
ncks -h -M -m -O -C -v one,three_dmn_rec_var ~/nco/data/in.nc ~/out.nc
# Change values to be sure original values are not copied in following step
ncap2 -O -v -s 'one*=2;three_dmn_rec_var*=0' ~/nco/data/in.nc ~/in2.nc
# Append in2.nc metadata (not data!) to out.nc
ncks -A -C -H -v one,three_dmn_rec_var ~/in2.nc ~/out.nc
@end example
Variables in @file{out.nc} now contain data (not metadata) from
@file{in.nc} and metadata (not data) from @file{in2.nc}.

@cindex @code{-s}
@cindex @code{--string}
@cindex @code{--sng_fmt}
@cindex @code{printf()}
@cindex @code{\n} (linefeed)
@cindex @code{\t} (horizontal tab)
Print variable @code{three_dmn_var} from file @file{in.nc} with
default notations. 
Next print @code{three_dmn_var} as an un-annotated text column.
Then print @code{three_dmn_var} signed with very high precision.
Finally, print @code{three_dmn_var} as a comma-separated list:
@example
% ncks --trd -C -v three_dmn_var in.nc
lat[0]=-90 lev[0]=100 lon[0]=0 three_dmn_var[0]=0 
lat[0]=-90 lev[0]=100 lon[1]=90 three_dmn_var[1]=1 
...
lat[1]=90 lev[2]=1000 lon[3]=270 three_dmn_var[23]=23 
% ncks --trd -s '%f\n' -C -v three_dmn_var in.nc
0.000000
1.000000
...
23.000000
% ncks --trd -s '%+16.10f\n' -C -v three_dmn_var in.nc
   +0.0000000000
   +1.0000000000
...
  +23.0000000000
% ncks --trd -s '%f, ' -C -v three_dmn_var in.nc
0.000000, 1.000000, ..., 23.000000,
@end example
@noindent
Programmers will recognize these as the venerable @w{C language} 
@code{printf()} formatting strings. 
The second and third options are useful when pasting data into text
files like reports or papers.  
@xref{ncatted netCDF Attribute Editor}, for more details on string
formatting and special characters.

@cindex @code{--no_blank}
As of @acronym{NCO} version 4.2.2 (October, 2012), @acronym{NCO} prints
missing values as blanks (i.e., the underscore character @samp{_}) by
default: 
@example
% ncks --trd -C -H -v mss_val in.nc
lon[0]=0 mss_val[0]=73 
lon[1]=90 mss_val[1]=_ 
lon[2]=180 mss_val[2]=73 
lon[3]=270 mss_val[3]=_ 
% ncks -s '%+5.1f, ' -H -C -v mss_val in.nc
+73.0, _, +73.0, _, 
@end example
To print the numeric value of the missing value instead of a blank,
use the @samp{--no_blank} option.

@cindex @code{-Q}
@cindex @code{--quiet}
@cindex @code{-V}
@cindex @code{--val_var}
@cindex @code{--no_dmn_var_nm}
@cindex @code{--no_nm_prn}
@command{ncks} prints in a verbose fashion by default and supplies a
number of switches to pare-down (or even spruce-up) the output.
The interplay of the @samp{-Q}, @samp{-V}, and (otherwise undocumented) 
@samp{--no_nm_prn} switches yields most desired verbosities:
@example
@verbatim
% ncks -v three_dmn_rec_var -C -H ~/nco/data/in.nc
time[0]=1 lat[0]=-90 lon[0]=0 three_dmn_rec_var[0]=1 
% ncks -Q -v three_dmn_rec_var -C -H ~/nco/data/in.nc              
three_dmn_rec_var[0]=1 
% ncks -V -v three_dmn_rec_var -C -H ~/nco/data/in.nc
1
% ncks -Q --no_nm_prn -v three_dmn_rec_var -C -H ~/nco/data/in.nc
1
% ncks --no_nm_prn -v three_dmn_rec_var -C -H ~/nco/data/in.nc
1 -90 0 1
@end verbatim
@end example

One dimensional arrays of characters stored as netCDF variables are 
automatically printed as strings, whether or not they are
NUL-terminated, e.g.,
@example
ncks -v fl_nm in.nc
@end example
@noindent
The @code{%c} formatting code is useful for printing 
multidimensional arrays of characters representing fixed length strings
@example
ncks -s '%c' -v fl_nm_arr in.nc
@end example
@noindent
@cindex @code{core dump}
Using the @code{%s} format code on strings which are not NUL-terminated 
(and thus not technically strings) is likely to result in a core dump.

@html
<a name="xmp_xtr_xcl"></a> <!-- http://nco.sf.net/nco.html#xmp_xtr_xcl -->
@end html
@cindex subsetting
@cindex exclusion
@cindex extraction
@cindex @code{-x}
@cindex @acronym{CF} conventions
@cindex @code{coordinates} attribute
@cindex @code{climatology} attribute
@cindex @code{bounds} attribute
@cindex @code{ancillary_variables} attribute
@cindex @code{grid_mapping} attribute
Create netCDF @file{out.nc} containing all variables, and any associated 
coordinates, except variable @code{time}, from netCDF @file{in.nc}:
@example
ncks -x -v time in.nc out.nc
@end example
As a special case of this, consider how to remove a 
variable such as @code{time_bounds} that is identified in a
@acronym{CF} Convention (@pxref{CF Conventions}) compliant
@code{ancillary_variables}, @code{bounds}, @code{climatology}, 
@code{coordinates}, or @code{grid_mapping} attribute. 
@acronym{NCO} subsetting assumes the user wants all ancillary variables,
axes, bounds and coordinates associated with all extracted variables 
(@pxref{Subsetting Coordinate Variables}).
Hence to exclude a @code{ancillary_variables}, @code{bounds},
@code{climatology}, @code{coordinates}, or @code{grid_mapping} variable
while retaining the ``parent'' variable (here @code{time}), one must use
the @samp{-C} switch:  
@example
ncks -C -x -v time_bounds in.nc out.nc
@end example
The @samp{-C} switch tells the operator @emph{NOT} to necessarily
include all the @acronym{CF} ancillary variables, axes, bounds, and
coordinates.
Hence the output file will contain @code{time} and not
@code{time_bounds}. 

Extract variables @code{time} and @code{pressure} from netCDF
@file{in.nc}.  
If @file{out.nc} does not exist it will be created.
Otherwise the you will be prompted whether to append to or to
overwrite @file{out.nc}: 
@example
ncks -v time,pressure in.nc out.nc
ncks -C -v time,pressure in.nc out.nc
@end example
@noindent
The first version of the command creates an @file{out.nc} which contains
@code{time}, @code{pressure}, and any coordinate variables associated
with @var{pressure}. 
The @file{out.nc} from the second version is guaranteed to contain only 
two variables @code{time} and @code{pressure}.  

Create netCDF @file{out.nc} containing all variables from file
@file{in.nc}.  
Restrict the dimensions of these variables to a hyperslab. 
@c Print (with @code{-H}) the hyperslabs to the screen for good measure.  
The specified hyperslab is: the fifth value in dimension @code{time};
the 
half-open range @math{@var{lat} > 0.} in coordinate @code{lat}; the
half-open range @math{@var{lon} < 330.} in coordinate @code{lon}; the
closed interval @math{0.3 < @var{band} < 0.5} in coordinate @code{band};
and cross-section closest to 1000.@: in coordinate @code{lev}.  
Note that limits applied to coordinate values are specified with a
decimal point, and limits applied to dimension indices do not have a 
decimal point @xref{Hyperslabs}.
@example
ncks -d time,5 -d lat,,0.0 -d lon,330.0, -d band,0.3,0.5 
-d lev,1000.0 in.nc out.nc 
@end example

@cindex wrapped coordinates
Assume the domain of the monotonically increasing longitude coordinate
@code{lon} is @math{0 < @var{lon} < 360}. 
Here, @code{lon} is an example of a wrapped coordinate.
@command{ncks} will extract a hyperslab which crosses the Greenwich
meridian simply by specifying the westernmost longitude as @var{min} and 
the easternmost longitude as @var{max}, as follows:
@example
ncks -d lon,260.0,45.0 in.nc out.nc
@end example
For more details @xref{Wrapped Coordinates}.

@page
@html
<a name="ncpdq"></a> <!-- http://nco.sf.net/nco.html#ncpdq -->
<a name="ncpack"></a> <!-- http://nco.sf.net/nco.html#ncpack -->
<a name="ncunpack"></a> <!-- http://nco.sf.net/nco.html#ncunpack -->
@end html
@node ncpdq netCDF Permute Dimensions Quickly, ncra netCDF Record Averager, ncks netCDF Kitchen Sink, Reference Manual
@section @command{ncpdq} netCDF Permute Dimensions Quickly
@findex ncpdq
@findex ncpack
@findex ncunpack
@cindex reshape variables
@cindex permute dimensions
@cindex reverse dimensions
@cindex re-order dimensions
@cindex re-dimension
@cindex packing
@cindex unpacking

@noindent
SYNTAX
@example
ncpdq [-3] [-4] [-5] [-6] [-7] [-A] [-a [-]@var{dim}[,@dots{}]]
[-C] [-c] [--cmp @var{cmp_sng}]
[--cnk_byt @var{sz_byt}] [--cnk_csh @var{sz_byt}] [--cnk_dmn @var{nm},@var{sz_lmn}]
[--cnk_map @var{map}] [--cnk_min @var{sz_byt}] [--cnk_plc @var{plc}] [--cnk_scl @var{sz_lmn}]
[-D @var{dbg}] [-d @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]] [-F] [--fl_fmt @var{fl_fmt}]
[-G @var{gpe_dsc}] [-g @var{grp}[,@dots{}]] [--glb ...]
[-H] [-h] [--hdf] [--hdr_pad @var{nbr}] [--hpss] 
[-L @var{dfl_lvl}] [-l @var{path}] [-M @var{pck_map}] [--mrd]
[--no_cll_msr] [--no_frm_trm] [--no_tmp_fl] 
[-O] [-o @var{output-file}] [-P @var{pck_plc}] [-p @var{path}] [--ppc ...]
[-R] [-r] [--ram_all] [-t @var{thr_nbr}] [-U] [--unn] [-v @var{var}[,@dots{}]] [-X ...] [-x]
@var{input-file} [@var{output-file}]
@end example

@noindent
DESCRIPTION

@command{ncpdq} performs one (not both) of two distinct functions
per invocation: packing or dimension permutation.
Without any options, @command{ncpdq} will pack data with default
parameters.
The @samp{-a} option tells @command{ncpdq} to permute dimensions
accordingly, otherwise @command{ncpdq} will pack data as
instructed/controlled by the @samp{-M} and @samp{-P} options.
@command{ncpdq} is optimized to perform these actions in a parallel
fashion with a minimum of time and memory.
The @dfn{pdq} may stand for ``Permute Dimensions Quickly'', 
``Pack Data Quietly'', ``Pillory Dan Quayle'', or other silly uses.

@cindex @code{add_offset}
@cindex @code{scale_factor}
@cindex @command{ncap2}
@cindex packing policy
@unnumberedsubsec Packing and Unpacking Functions
The @command{ncpdq} packing (and unpacking) algorithms are described 
in @ref{Methods and functions}, and are also implemented in
@command{ncap2}. 
@command{ncpdq} extends the functionality of these algorithms by 
providing high level control of the @dfn{packing policy} so that
users can consistently pack (and unpack) entire files with one command. 
@cindex @var{pck_plc}
@cindex @code{-P @var{pck_plc}}
@cindex @code{--pck_plc @var{pck_plc}}
@cindex @code{--pack_policy @var{pck_plc}}
The user specifies the desired packing policy with the @samp{-P} switch
(or its long option equivalents, @samp{--pck_plc} and
@samp{--pack_policy}) and its @var{pck_plc} argument.
Four packing policies are currently implemented:@*   
@table @dfn
@item Packing (and Re-Packing) Variables [@emph{default}]
Definition: Pack unpacked variables, re-pack packed variables@*
Alternate invocation: @code{ncpack}@*
@var{pck_plc} key values: @samp{all_new}, @samp{pck_all_new_att}@*
@item Packing (and not Re-Packing) Variables
Definition: Pack unpacked variables, copy packed variables@*
Alternate invocation: none@*
@var{pck_plc} key values: @samp{all_xst}, @samp{pck_all_xst_att}@*
@item Re-Packing Variables
Definition: Re-pack packed variables, copy unpacked variables@*
Alternate invocation: none@*
@var{pck_plc} key values: @samp{xst_new}, @samp{pck_xst_new_att}@*
@item Unpacking
Definition: Unpack packed variables, copy unpacked variables@*
Alternate invocation: @code{ncunpack}@*
@var{pck_plc} key values: @samp{upk}, @samp{unpack}, @samp{pck_upk}@*
@end table
@noindent
Equivalent key values are fully interchangeable.
Multiple equivalent options are provided to satisfy disparate needs
and tastes of @acronym{NCO} users working with scripts and from the
command line.

Regardless of the packing policy selected, @command{ncpdq} 
no longer (as of @acronym{NCO} version 4.0.4 in October, 2010)
packs coordinate variables, or the special variables, weights, 
and other grid properties described in @ref{CF Conventions}.
Prior @command{ncpdq} versions treated coordinate variables and
grid properties no differently from other variables.
However, coordinate variables are one-dimensional, so packing saves
little space on large files, and the resulting files are difficult for
humans to read. 
@command{ncpdq} will, of course, @emph{unpack} coordinate variables and
weights, for example, in case some other, non-@acronym{NCO} software
packed them in the first place.

Concurrently, Gaussian and area weights and other grid properties are
often used to derive fields in re-inflated (unpacked) files, so packing
such grid properties causes a considerable loss of precision in 
downstream data processing.
If users express strong wishes to pack grid properties, we will
implement new packing policies.
An immediate workaround for those needing to pack grid properties
now, is to use the @command{ncap2} packing functions or to rename the
grid properties prior to calling @command{ncpdq}. 
We welcome your feedback. 

To reduce required memorization of these complex policy switches, 
@command{ncpdq} may also be invoked via a synonym or with switches
that imply a particular policy.
@command{ncpack} is a synonym for @command{ncpdq} and behaves the same 
in all respects.
Both @command{ncpdq} and @command{ncpack} assume a default packing
policy request of @samp{all_new}.
Hence @command{ncpack} may be invoked without any @samp{-P} switch,
unlike @command{ncpdq}.
Similarly, @command{ncunpack} is a synonym for @command{ncpdq} 
except that @command{ncpack} implicitly assumes a request to unpack, 
i.e., @samp{-P pck_upk}.
@cindex @code{-U}
@cindex @code{--unpack}
Finally, the @command{ncpdq} @samp{-U} switch (or its long option
equivalents @samp{--unpack}) requires no argument.
It simply requests unpacking.

Given the menagerie of synonyms, equivalent options, and implied
options, a short list of some equivalent commands is appropriate.
The following commands are equivalent for packing:
@code{ncpdq -P all_new}, @code{ncpdq --pck_plc=all_new}, and
@code{ncpack}.
The following commands are equivalent for unpacking:
@code{ncpdq -P upk}, @code{ncpdq -U}, @code{ncpdq --pck_plc=unpack}, 
and @code{ncunpack}.
Equivalent commands for other packing policies, e.g., @samp{all_xst}, 
follow by analogy. 
@cindex @command{alias}
@cindex @command{ln -s}
@cindex symbolic links
Note that @command{ncpdq} synonyms are subject to the same constraints 
and recommendations discussed in the secion on @command{ncbo} synonyms
(@pxref{ncbo netCDF Binary Operator}).
That is, symbolic links must exist from the synonym to @command{ncpdq},
or else the user must define an @command{alias}.

@html
<a name="pck_map"></a> <!-- http://nco.sf.net/nco.html#pck_map -->
<a name="hgh_sht"></a> <!-- http://nco.sf.net/nco.html#hgh_sht -->
<a name="hgh_byt"></a> <!-- http://nco.sf.net/nco.html#hgh_byt -->
<a name="flt_byt"></a> <!-- http://nco.sf.net/nco.html#flt_byt -->
<a name="nxt_lsr"></a> <!-- http://nco.sf.net/nco.html#nxt_lsr -->
<a name="dbl_flt"></a> <!-- http://nco.sf.net/nco.html#dbl_flt -->
<a name="flt_dbl"></a> <!-- http://nco.sf.net/nco.html#flt_dbl -->
@end html
@cindex packing map
@cindex @var{pck_map}
@cindex @code{-M @var{pck_map}}
@cindex @code{--pck_map @var{pck_map}}
@cindex @code{--map @var{pck_map}}
The @command{ncpdq} packing algorithms must know to which type
particular types of input variables are to be packed.
The correspondence between the input variable type and the output,
packed type, is called the @dfn{packing map}.
The user specifies the desired packing map with the @samp{-M} switch
(or its long option equivalents, @samp{--pck_map} and
@samp{--map}) and its @var{pck_map} argument.
Six packing maps are currently implemented:@*
@cindex @samp{hgh_sht}
@cindex @samp{hgh_byt}
@cindex @samp{flt_sht}
@cindex @samp{flt_byt}
@cindex @samp{nxt_lsr}
@cindex @samp{dbl_flt}
@cindex @samp{flt_dbl}
@cindex @code{NC_DOUBLE}
@cindex @code{NC_FLOAT}
@cindex @code{NC_INT64}
@cindex @code{NC_UINT64}
@cindex @code{NC_INT}
@cindex @code{NC_UINT}
@cindex @code{NC_SHORT}
@cindex @code{NC_USHORT}
@cindex @code{NC_CHAR}
@cindex @code{NC_BYTE}
@cindex @code{NC_UBYTE}
@table @dfn
@item Pack Floating Precisions to @code{NC_SHORT} [@emph{default}]
Definition: Pack floating precision types to @code{NC_SHORT}@*
Map: Pack [@code{NC_DOUBLE},@code{NC_FLOAT}] to @code{NC_SHORT}@*
Types copied instead of packed: [@code{NC_INT64},@code{NC_UINT64},@code{NC_INT},@code{NC_UINT},@code{NC_SHORT},@code{NC_USHORT},@code{NC_CHAR},@code{NC_BYTE},@code{NC_UBYTE}]@*
@var{pck_map} key values: @samp{flt_sht}, @samp{pck_map_flt_sht}@*
@item Pack Floating Precisions to @code{NC_BYTE}
Definition: Pack floating precision types to @code{NC_BYTE}@*
Map: Pack [@code{NC_DOUBLE},@code{NC_FLOAT}] to @code{NC_BYTE}@* 
Types copied instead of packed: [@code{NC_INT64},@code{NC_UINT64},@code{NC_INT},@code{NC_UINT},@code{NC_SHORT},@code{NC_USHORT},@code{NC_CHAR},@code{NC_BYTE},@code{NC_UBYTE}]@*
@var{pck_map} key values: @samp{flt_byt}, @samp{pck_map_flt_byt}@*
@item Pack Higher Precisions to @code{NC_SHORT}
Definition: Pack higher precision types to @code{NC_SHORT}@*
Map: 
Pack [@code{NC_DOUBLE},@code{NC_FLOAT},@code{NC_INT64},@code{NC_UINT64},@code{NC_INT},@code{NC_UINT}] to @code{NC_SHORT}@*
Types copied instead of packed: [@code{NC_SHORT},@code{NC_USHORT},@code{NC_CHAR},@code{NC_BYTE},@code{NC_UBYTE}]@*
@var{pck_map} key values: @samp{hgh_sht}, @samp{pck_map_hgh_sht}@*
@item Pack Higher Precisions to @code{NC_BYTE}
Definition: Pack higher precision types to @code{NC_BYTE}@*
Map: 
Pack [@code{NC_DOUBLE},@code{NC_FLOAT},@code{NC_INT64},@code{NC_UINT64},@code{NC_INT},@code{NC_UINT},@code{NC_SHORT},@code{NC_USHORT}] to @code{NC_BYTE}@*
Types copied instead of packed: [@code{NC_CHAR},@code{NC_BYTE},@code{NC_UBYTE}]@*
@var{pck_map} key values: @samp{hgh_byt}, @samp{pck_map_hgh_byt}@*
@item Pack to Next Lesser Precision
Definition: Pack each type to type of next lesser size@*
Map: Pack [@code{NC_DOUBLE},@code{NC_INT64},@code{NC_UINT64}] to @code{NC_INT}. 
Pack [@code{NC_FLOAT},@code{NC_INT},@code{NC_UINT}] to @code{NC_SHORT}.
Pack [@code{NC_SHORT},@code{NC_USHORT}] to @code{NC_BYTE}.@*
Types copied instead of packed: [@code{NC_CHAR},@code{NC_BYTE},@code{NC_UBYTE}]@*
@var{pck_map} key values: @samp{nxt_lsr}, @samp{pck_map_nxt_lsr}@*
@item Pack Doubles to Floats
Definition: Demote (via type-conversion, @emph{not packing}) double-precision variables to single-precision@*
Map: Demote @code{NC_DOUBLE} to @code{NC_FLOAT}. 
Types copied instead of packed: All except @code{NC_DOUBLE}@*
@var{pck_map} key values: @samp{dbl_flt}, @samp{pck_map_dbl_flt}, @samp{dbl_sgl}, @samp{pck_map_dbl_sgl}@*
The @code{dbl_flt} map was introduced in @acronym{NCO} version 4.7.7 (September, 2018).
@item Promote Floats to Doubles
Definition: Promote (via type-conversion, @emph{not packing}) single-precision variables to double-precision@*
Map: Promote @code{NC_FLOAT} to @code{NC_DOUBLE}.
Types copied instead of packed: All except @code{NC_FLOAT}@*
@var{pck_map} key values: @samp{flt_dbl}, @samp{pck_map_flt_dbl}, @samp{sgl_dbl}, @samp{pck_map_sgl_dbl}@*
The @code{flt_dbl} map was introduced in @acronym{NCO} version 4.9.1
(December, 2019). 
@end table
@noindent
The default @samp{all_new} packing policy with the default
@samp{flt_sht} packing map reduces the typical @code{NC_FLOAT}-dominated
file size by @w{about 50%.}
@samp{flt_byt} packing reduces an @code{NC_DOUBLE}-dominated file by
@w{about 87%.}

@cindex @code{d2f}
The ``packing map'' @samp{pck_map_dbl_flt} does a pure type-conversion
(no packing is involved) from @code{NC_DOUBLE} to @code{NC_FLOAT}.
The resulting variables are not packed, they are just single-precision
floating point instead of double-precision floating point.
This operation is irreversible, and no attributes are created, modified,
or deleted for these variables.
Note that coordinate and coordinate-like variables will not be demoted
as best practices dictate maintaining coordinates in the highest
possible precision.

The ``packing map'' @samp{pck_map_flt_dbl} does a pure type-conversion
(no packing is involved) from @code{NC_FLOAT} to @code{NC_DOUBLE}.
The resulting variables are not packed, they are just double-precision
floating point instead of single-precision floating point.
This operation is irreversible, and no attributes are created, modified,
or deleted for these variables.
All single-precision variables, including coordinates, are promoted. 
Note that this map can double the size of a dataset.

@cindex @var{_FillValue}
@cindex @code{_FillValue}
@cindex @code{NUL}
The netCDF packing algorithm (@pxref{Methods and functions}) is
lossy---once packed, the exact original data cannot be recovered without
a full backup. 
Hence users should be aware of some packing caveats:
First, the interaction of packing and data equal to the
@var{_FillValue} is complex.
Test the @code{_FillValue} behavior by performing a pack/unpack cycle 
to ensure data that are missing @emph{stay} missing and data that are
not misssing do not join the Air National Guard and go missing.
This may lead you to elect a new @var{_FillValue}.
Second, @code{ncpdq} actually allows packing into @code{NC_CHAR} (with,
e.g., @samp{flt_chr}).
However, the intrinsic conversion of @code{signed char} to higher
precision types is tricky for values equal to zero, i.e., for
@code{NUL}.  
Hence packing to @code{NC_CHAR} is not documented or advertised.  
Pack into @code{NC_BYTE} (with, e.g., @samp{flt_byt}) instead.

@html
<a name="rvr"></a> <!-- http://nco.sf.net/nco.html#rvr -->
@end html
@unnumberedsubsec Dimension Permutation
@command{ncpdq} re-shapes variables in @var{input-file} by re-ordering
and/or reversing dimensions specified in the dimension list.
The dimension list is a whitespace-free, comma separated list of
dimension names, optionally prefixed by negative signs, that follows the 
@samp{-a} (or long options @samp{--arrange}, @samp{--permute},
@samp{--re-order}, or @samp{--rdr}) switch.  
To re-order variables by a subset of their dimensions, specify
these dimensions in a comma-separated list following @samp{-a}, e.g., 
@samp{-a lon,lat}. 
To reverse a dimension, prefix its name with a negative sign in the
dimension list, e.g., @samp{-a -lat}. 
Re-ordering and reversal may be performed simultaneously, e.g.,
@samp{-a lon,-lat,time,-lev}. 

@cindex record dimension
Users may specify any permutation of dimensions, including
permutations which change the record dimension identity.
The record dimension is re-ordered like any other dimension.
@cindex concatenation
@cindex record dimension
This unique @command{ncpdq} capability makes it possible to concatenate
files along any dimension.
See @ref{Concatenation} for a detailed example.
@cindex record variable
The record dimension is always the most slowly varying dimension in a
record variable (@pxref{C and Fortran Index Conventions}).
The specified re-ordering fails if it requires creating more than
one record dimension amongst all the output variables
@footnote{This limitation, imposed by the netCDF storage layer,
may be relaxed in the future with netCDF4.}.

Two special cases of dimension re-ordering and reversal deserve special
mention. 
First, it may be desirable to completely reverse the storage order of a
variable. 
To do this, include all the variable's dimensions in the dimension
re-order list in their original order, and prefix each dimension name
with the negative sign.  
@cindex transpose
Second, it may useful to transpose a variable's storage order, e.g.,
@w{from C} to Fortran data storage order 
(@pxref{C and Fortran Index Conventions}).
To do this, include all the variable's dimensions in the dimension
re-order list in reversed order.
Explicit examples of these two techniques appear below.

@tex
NB: fxm ncpdq documentation will evolve through Fall 2004.
I will upload updates to documentation linked to by the NCO homepage.
ncpdq is a powerful operator, and 
I am unfamiliar with the terminology needed to describe what ncpdq does.
Sequences, sets, sheesh!
I just know that it does ``The right thing'' according to my gut feelings.
Now do you feel more comfortable using it?

Let $\dmnvct(\xxx)$ represent the dimensionality of the variable $\xxx$. 
Dimensionality describes the order and sizes of dimensions.
If $\xxx$ has rank $\dmnnbr$, then we may write $\dmnvct(\xxx)$ as the
$\dmnnbr$-element vector 
$$
\dmnvct(\xxx) = [ \dmn_{1}, \dmn_{2}, \dmn_{3}, \ldots, 
\dmn_{\dmnidx-1}, \dmn_{\dmnidx}, \dmn_{\dmnidx+1}, 
\ldots, \dmn_{\dmnnbr-2}, \dmn_{\dmnnbr-1}, \dmn_{\dmnnbr} ] 
$$
where $\dmn_{\dmnidx}$ is the size of the $\dmnidx$'th dimension.

The dimension re-order list specified with @samp{-a} is the
$\rdrnbr$-element vector 
$$
\rdrvct = [ \rdr_{1}, \rdr_{2}, \rdr_{3}, \ldots, 
\rdr_{\rdridx-1}, \rdr_{\rdridx}, \rdr_{\rdridx+1}, 
\ldots, \rdr_{\rdrnbr-2}, \rdr_{\rdrnbr-1}, \rdr_{\rdrnbr} ] 
$$
There need be no relation between $\dmnnbr$ and $\rdrnbr$.
Let the $\shrnbr$-element vector $\shrvct$ be the intersection
(i.e., the ordered set of unique shared dimensions) of $\dmnvct$ and 
$\rdrvct$
Then
$$
\eqalign{{\shrvct} &= \rdrvct \cap \dmnvct \cr
&= [ \shr_{1}, \shr_{2}, \shr_{3}, \ldots, 
\shr_{\shridx-1}, \shr_{\shridx}, \shr_{\shridx+1}, 
\ldots, \shr_{\shrnbr-2}, \shr_{\shrnbr-1}, \shr_{\shrnbr} ]} 
$$
$\shrvct$ is empty if $\rdrvct \notin \dmnvct$.

Re-ordering (or re-shaping) a variable means mapping the input state
with dimensionality $\dmnvct(\xxx)$ to the output state with
dimensionality $\dmnvctprm(\xxxprm)$.  
In practice, mapping occurs in three logically distinct steps.
First, we tranlate the user input to a one-to-one mapping $\mpp$ 
between input and output dimensions, 
$\dmnvct \mapsto \dmnvctprm$.
This tentative map is final unless external constraints (typically
netCDF restrictions) impose themselves.   
Second, we check and, if necessary, refine the tentative mapping so that
the re-shaped variables will co-exist in the same file without violating 
netCDF-imposed storage restrictions. 
This refined map specifies the final (output) dimensionality.
Third, we translate the output dimensionality into one-dimensional  
memory offsets for each datum according to the @w{C language} convention 
for multi-dimensional array storage.
Dimension reversal changes the ordering of data, though not the
rank or dimensionality, and so is part of the third step. 

Dimensions $\rdr$ disjoint from $\dmnvct$ play no role in re-ordering.
The first step taken to re-order a variable is to determine $\shrvct$. 
$\rdrvct$ is constant for all variables, whereas $\dmnvct$, and hence
$\shrvct$, is variable-specific.
$\shrvct$ is empty if $\rdrvct \notin \dmnvct$.
This may be the case for some extracted variables.
The user may explicitly specify the one-to-one mapping of input
to output dimension order by supplying (with @samp{-a}) a re-order list
$\rdrvct$ such that $\shrnbr = \dmnnbr$. 
In this case $\dmnsubnnnprm = \shrsubnnn$.  
The degenerate case occurs when $\dmnvct = \shrvct$.
This produces the identity mapping $\dmnsubnnnprm = \dmnsubnnn$.  

The mapping of input to output dimension order is more complex
when $\shrnbr \ne \dmnnbr$. 
In this case $\dmnsubnnnprm = \dmnsubnnn$ for the $\dmnnbr-\shrnbr$
dimensions $\dmnsubnnnprm \notin \shrvct$.
For the $\shrnbr$ dimensions $\dmnsubnnnprm \in \shrvct$, 
$\dmnsubnnnprm = \shrsubsss$.  
@end tex

@c fxm: discuss netCDF-imposed constraints here

@noindent
@html
<a name="xmp_ncpdq"></a> <!-- http://nco.sf.net/nco.html#xmp_ncpdq -->
@end html
EXAMPLES

Pack and unpack all variables in file @file{in.nc} and store the results
in @file{out.nc}:  
@example
ncpdq in.nc out.nc # Same as ncpack in.nc out.nc
ncpdq -P all_new -M flt_sht in.nc out.nc # Defaults
ncpdq -P all_xst in.nc out.nc
ncpdq -P upk in.nc out.nc # Same as ncunpack in.nc out.nc
ncpdq -U in.nc out.nc # Same as ncunpack in.nc out.nc
@end example
The first two commands pack any unpacked variable in the input file.
They also unpack and then re-pack every packed variable.
The third command only packs unpacked variables in the input file.
If a variable is already packed, the third command copies it unchanged
to the output file. 
The fourth and fifth commands unpack any packed variables.
If a variable is not packed, the third command copies it unchanged.

The previous examples all utilized the default packing map.
Suppose you wish to archive all data that are currently unpacked 
into a form which only preserves 256 distinct values.
Then you could specify the packing map @var{pck_map} as @samp{hgh_byt}
and the packing policy @var{pck_plc} as @samp{all_xst}:
@example
ncpdq -P all_xst -M hgh_byt in.nc out.nc
@end example
@cindex appending variables
@cindex @samp{-A}
@cindex @samp{-v}
Many different packing maps may be used to construct a given file 
by performing the packing on subsets of variables (e.g., with @samp{-v}) 
and using the append feature with @samp{-A} (@pxref{Appending Variables}).

Users may wish to unpack data packed with the @acronym{HDF} convention,
and then re-pack it with the netCDF convention so that all their
datasets use the same packing convention prior to intercomparison.
@cindex @command{ncl_convert2nc}
@cindex @acronym{NCL}
@example
# One-step procedure: For NCO 4.4.0+, netCDF 4.3.1+
# 1. Convert, unpack, and repack HDF file into netCDF file
ncpdq --hdf_upk -P xst_new modis.hdf modis.nc # HDF4 files
ncpdq --hdf_upk -P xst_new modis.h5  modis.nc # HDF5 files

# One-step procedure: For NCO 4.3.7--4.3.9
# 1. Convert, unpack, and repack HDF file into netCDF file
ncpdq --hdf4 --hdf_upk -P xst_new modis.hdf modis.nc # HDF4
ncpdq        --hdf_upk -P xst_new modis.h5  modis.nc # HDF5

# Two-step procedure: For NCO 4.3.6 and earlier
# 1. Convert HDF file to netCDF file
ncl_convert2nc modis.hdf
# 2. Unpack using HDF convention and repack using netCDF convention
ncpdq --hdf_upk -P xst_new modis.nc modis.nc
@end example
@acronym{NCO} now
@footnote{
Prior to @acronym{NCO} 4.4.0 and netCDF 4.3.1 (January, 2014),
@acronym{NCO} requires the @samp{--hdf4} switch to correctly read
HDF4 input files.
For example, 
@samp{ncpdq --hdf4 --hdf_upk -P xst_new modis.hdf modis.nc}. 
That switch is now obsolete, though harmless for backwards
compatibility. 
Prior to version 4.3.7 (October, 2013), @acronym{NCO} lacked the
software necessary to circumvent netCDF library flaws handling
@acronym{HDF4} files, and thus @acronym{NCO} failed to convert
@acronym{HDF4} files to netCDF files.
In those cases, use the @command{ncl_convert2nc} command distributed
with @acronym{NCL} to convert @acronym{HDF4} files to netCDF.}
automatically detects @acronym{HDF4} files.  
In this case it produces an output file @file{modis.nc} which preserves
the @acronym{HDF} packing used in the input file.
The @command{ncpdq} command first unpacks all packed variables using the
@acronym{HDF} unpacking algorithm (as specified by @samp{--hdf_upk}), 
and then repacks those same variables using the netCDF algorithm
(because that is the only algorithm @acronym{NCO} packs with).
As described above the @samp{--P xst_new} packing policy only repacks
variables that are already packed. 
Not-packed variables are copied directly without loss of precision
@footnote{@command{ncpdq} does not support packing data using the
@acronym{HDF} convention.
Although it is now straightforward to support this, we think it might
sow more confusion than it reaps. 
Let us know if you disagree and would like @acronym{NCO} to support
packing data with @acronym{HDF} algorithm.}.

Re-order file @file{in.nc} so that the dimension @code{lon} always
precedes the dimension @code{lat} and store the results in
@file{out.nc}:  
@example
ncpdq -a lon,lat in.nc out.nc
ncpdq -v three_dmn_var -a lon,lat in.nc out.nc
@end example
The first command re-orders every variable in the input file.
The second command extracts and re-orders only the variable
@code{three_dmn_var}. 

@cindex reverse dimensions
Suppose the dimension @code{lat} represents latitude and monotonically 
increases increases from south to north. 
Reversing the @code{lat} dimension means re-ordering the data so that
latitude values decrease monotonically from north to south.
Accomplish this with
@example
% ncpdq -a -lat in.nc out.nc
% ncks --trd -C -v lat in.nc
lat[0]=-90
lat[1]=90
% ncks --trd -C -v lat out.nc
lat[0]=90
lat[1]=-90
@end example
This operation reversed the latitude dimension of all variables.
Whitespace immediately preceding the negative sign that specifies
dimension reversal may be dangerous.
@cindex long options
@cindex quotes
Quotes and long options can help protect negative signs that should
indicate dimension reversal from being interpreted by the shell as
dashes that indicate new command line switches.
@example
ncpdq -a -lat in.nc out.nc # Dangerous? Whitespace before "-lat"
ncpdq -a '-lat' in.nc out.nc # OK. Quotes protect "-" in "-lat"
ncpdq -a lon,-lat in.nc out.nc # OK. No whitespace before "-"
ncpdq --rdr=-lat in.nc out.nc # Preferred. Uses "=" not whitespace
@end example

@cindex transpose
@cindex reverse dimensions
@html
<a name="ncpdq_trn"></a> <!-- http://nco.sf.net/nco.html#ncpdq_trn -->
<a name="transpose"></a> <!-- http://nco.sf.net/nco.html#transpose -->
@end html
To create the mathematical transpose of a variable, place all its
dimensions in the dimension re-order list in reversed order.
This example creates the transpose of @code{three_dmn_var}: 
@example
% ncpdq -a lon,lev,lat -v three_dmn_var in.nc out.nc
% ncks --trd -C -v three_dmn_var in.nc
lat[0]=-90 lev[0]=100 lon[0]=0 three_dmn_var[0]=0 
lat[0]=-90 lev[0]=100 lon[1]=90 three_dmn_var[1]=1 
lat[0]=-90 lev[0]=100 lon[2]=180 three_dmn_var[2]=2 
...
lat[1]=90 lev[2]=1000 lon[1]=90 three_dmn_var[21]=21 
lat[1]=90 lev[2]=1000 lon[2]=180 three_dmn_var[22]=22 
lat[1]=90 lev[2]=1000 lon[3]=270 three_dmn_var[23]=23 
% ncks --trd -C -v three_dmn_var out.nc
lon[0]=0 lev[0]=100 lat[0]=-90 three_dmn_var[0]=0
lon[0]=0 lev[0]=100 lat[1]=90 three_dmn_var[1]=12
lon[0]=0 lev[1]=500 lat[0]=-90 three_dmn_var[2]=4
...
lon[3]=270 lev[1]=500 lat[1]=90 three_dmn_var[21]=19
lon[3]=270 lev[2]=1000 lat[0]=-90 three_dmn_var[22]=11
lon[3]=270 lev[2]=1000 lat[1]=90 three_dmn_var[23]=23
@end example

@cindex reverse data
To completely reverse the storage order of a variable, include
all its dimensions in the re-order list, each prefixed by a negative
sign. 
This example reverses the storage order of @code{three_dmn_var}: 
@example
% ncpdq -a -lat,-lev,-lon -v three_dmn_var in.nc out.nc
% ncks --trd -C -v three_dmn_var in.nc
lat[0]=-90 lev[0]=100 lon[0]=0 three_dmn_var[0]=0 
lat[0]=-90 lev[0]=100 lon[1]=90 three_dmn_var[1]=1 
lat[0]=-90 lev[0]=100 lon[2]=180 three_dmn_var[2]=2 
...
lat[1]=90 lev[2]=1000 lon[1]=90 three_dmn_var[21]=21 
lat[1]=90 lev[2]=1000 lon[2]=180 three_dmn_var[22]=22 
lat[1]=90 lev[2]=1000 lon[3]=270 three_dmn_var[23]=23 
% ncks --trd -C -v three_dmn_var out.nc
lat[0]=90 lev[0]=1000 lon[0]=270 three_dmn_var[0]=23
lat[0]=90 lev[0]=1000 lon[1]=180 three_dmn_var[1]=22
lat[0]=90 lev[0]=1000 lon[2]=90 three_dmn_var[2]=21
...
lat[1]=-90 lev[2]=100 lon[1]=180 three_dmn_var[21]=2
lat[1]=-90 lev[2]=100 lon[2]=90 three_dmn_var[22]=1
lat[1]=-90 lev[2]=100 lon[3]=0 three_dmn_var[23]=0
@end example

@html
<a name="dmn_rcd_mk"></a> <!-- http://nco.sf.net/nco.html#dmn_rcd_mk -->
<a name="mk_rcd_dmn"></a> <!-- http://nco.sf.net/nco.html#mk_rcd_dmn -->
@end html
Creating a record dimension named, e.g., @code{time}, in a file which
has no existing record dimension is simple with @command{ncecat}:
@example
ncecat -O -u time in.nc out.nc # Create degenerate record dimension named "time"
@end example

@cindex record dimension
Now consider a file with all dimensions, including @code{time}, fixed
(non-record).
Suppose the user wishes to convert @code{time} from a fixed dimension to  
a record dimension. 
This may be useful, for example, when the user wishes to append
additional time slices to the data.
As of @acronym{NCO} version 4.0.1 (April, 2010) the preferred method for
doing this is with @command{ncks}:
@example
ncks -O --mk_rec_dmn time in.nc out.nc # Change "time" to record dimension
@end example

Prior to 4.0.1, the procedure to change an existing fixed dimension into
a record dimension required three separate commands,
@command{ncecat} followed by @command{ncpdq}, and then @command{ncwa}.
The recommended method is now to use @samp{ncks --fix_rec_dmn}, yet it
is still instructive to present the original procedure, as it shows how
multiple operators can achieve the same ends by different means: 
@cindex degenerate dimension
@example
ncecat -O in.nc out.nc # Add degenerate record dimension named "record"
ncpdq -O -a time,record out.nc out.nc # Switch "record" and "time"
ncwa -O -a record out.nc out.nc # Remove (degenerate) "record"
@end example
@noindent
The first step creates a degenerate (size equals one) record dimension
named (by default) @code{record}. 
The second step swaps the ordering of the dimensions named @code{time}
and @code{record}.
Since @code{time} now occupies the position of the first (least rapidly
varying) dimension, it becomes the record dimension.
The dimension named @code{record} is no longer a record dimension.
The third step averages over this degenerate @code{record} dimension.
Averaging over a degenerate dimension does not alter the data.
The ordering of other dimensions in the file (@code{lat}, @code{lon},
etc.) is immaterial to this procedure. 
See @ref{ncecat netCDF Ensemble Concatenator} and 
@ref{ncks netCDF Kitchen Sink} for other methods of
changing variable dimensionality, including the record dimension. 

@page
@html
<a name="ncra"></a> <!-- http://nco.sf.net/nco.html#ncra -->
@end html
@node ncra netCDF Record Averager, ncrcat netCDF Record Concatenator, ncpdq netCDF Permute Dimensions Quickly, Reference Manual
@section @command{ncra} netCDF Record Averager
@cindex averaging data
@cindex record average
@cindex record dimension
@cindex running average
@findex ncra

@noindent
SYNTAX
@example
ncra [-3] [-4] [-5] [-6] [-7] [-A] [-C] [-c]
[--cb @var{y1},@var{y2},@var{m1},@var{m2},@var{tpd}] [--cmp @var{cmp_sng}]
[--cnk_byt @var{sz_byt}] [--cnk_csh @var{sz_byt}] [--cnk_dmn @var{nm},@var{sz_lmn}]
[--cnk_map @var{map}] [--cnk_min @var{sz_byt}] [--cnk_plc @var{plc}] [--cnk_scl @var{sz_lmn}]
[-D @var{dbg}] [-d @var{dim},[@var{min}][,[@var{max}][,[@var{stride}][,[@var{subcycle}][,[@var{interleave}]]]]]
[-F] [--fl_fmt @var{fl_fmt}]
[-G @var{gpe_dsc}] [-g @var{grp}[,@dots{}]] [--glb ...]
[-H] [-h] [--hdf] [--hdr_pad @var{nbr}] [--hpss] 
[-L @var{dfl_lvl}] [-l @var{path}] [--mro] [-N] [-n @var{loop}] 
[--no_cll_msr] [--no_cll_mth] [--no_frm_trm] [--no_tmp_fl] 
[-O] [-o @var{output-file}] [-p @var{path}] [--ppc ...] [--prm_int] [--prw wgt_arr]
[-R] [-r] [--ram_all] [--rec_apn] [--rth_dbl|flt]
[-t @var{thr_nbr}] [--unn] [-v @var{var}[,@dots{}]] [-w wgt] [-X ...] [-x] [-y @var{op_typ}]
[@var{input-files}] [@var{output-file}]
@end example

@noindent
DESCRIPTION

@command{ncra} computes statistics (including, though not limited to,
averages) of record variables across an arbitrary number of  
@var{input-files}.
@cindex degenerate dimension
@cindex record dimension
The record dimension is, by default, retained as a degenerate 
@w{(size 1)} dimension in the output variables.
@xref{Statistics vs Concatenation}, for a description of the
distinctions between the various statistics tools and concatenators. 
@cindex multi-file operators
@cindex standard input
@cindex @code{stdin}
As a multi-file operator, @command{ncra} will read the list of
@var{input-files} from @code{stdin} if they are not specified 
as positional arguments on the command line 
(@pxref{Large Numbers of Files}).

Input files may vary in size, but each must have a record dimension.
The record coordinate, if any, should be monotonic (or else non-fatal
warnings may be generated). 
@cindex hyperslab
Hyperslabs of the record dimension which include more than one file 
work correctly.
@cindex stride
@command{ncra} supports the @var{stride} argument to the @samp{-d}
hyperslab option (@pxref{Hyperslabs}) for the record dimension only,
@var{stride} is not supported for non-record dimensions.
@cindex operation types
@command{ncra} @emph{always averages} coordinate variables (e.g., 
@code{time}) regardless of the arithmetic operation type performed on
non-coordinate variables (@pxref{Operation Types}).

As of @acronym{NCO} @w{version 4.4.9}, released in May, 2015,
@command{ncra} accepts user-specified weights with the @samp{-w} 
(or long-option equivalent @samp{--wgt}, @samp{--wgt_var},
or @samp{--weight}) switch. 
When no weight is specified, @command{ncra} weights each record (e.g.,
time slice) in the @var{input-files} equally.
@command{ncra} does not attempt to see if, say, the @code{time}
coordinate is irregularly spaced and thus would require a weighted
average in order to be a true time-average.
Specifying unequal weights is entirely the user's responsibility.

Weights specified with @samp{-w wgt} may take one of two forms.
In the first form, the @samp{wgt} argument is a comma-separated list of
values by which to weight each @emph{file} (recall that files may have
multiple timesteps). 
In this form the number of weights specified must equal the number of
files specified in the input file list, or else the program will exit.
In the second form, the @samp{wgt} argument is the name of a weighting
variable present in every input file.
The variable may be a scalar or a one-dimensional record variable.
Scalar weights are applied uniformly to the entire file (i.e., this
produces the same arithmetic result as supplying the same value
as a per-file weight option on the command-line).
One-dimensional weights apply to each corresponding record (i.e.,
per-record weights), and are suitable for dynamically changing
timesteps. 

By default, any weights specified (whether by value or by variable name)
are normalized to unity by dividing each specified weight by the sum of
all the weights.
This means, for example, that, @samp{-w 0.25,0.75} is equivalent to 
@samp{-w 2.0,6.0} since both are equal when normalized.
This behavior simplifies specifying weights based on countable items.
For example, time-weighting monthly averages for March, April, and May
to obtain a spring seasonal average can be done with 
@samp{-w 31,30,31} instead of 
@samp{-w 0.33695652173913043478,0.32608695652173913043,0.33695652173913043478}. 

However, sometimes one wishes to use weights in ``dot-product mode'',
i.e., multiply by the (non-normalized) weights.
As of @acronym{NCO} @w{version 4.5.2}, released in July, 2015,
@command{ncra} accepts the @samp{-N} (or long-option equivalent
@samp{--no_nrm_by_wgt}) switch that prevents automatic weight
normalization. 
When this switch is used, the weights will not be normalized (unless the
user provides them as normalized), and the numerator of the weighted
average will not be divided by the sum of the weights (which is one for
normalized weights).

@html
<a name="prw"></a> <!-- http://nco.sf.net/nco.html#prw -->
<a name="per_record_weights"></a> <!-- http://nco.sf.net/nco.html#per_record_weights -->
@end html
@cindex @code{--per_record_weights}
@cindex @code{--prw}
@cindex weights
@cindex weighted average
@cindex per-record-weights
@cindex record average
As of @acronym{NCO} @w{version 4.9.4}, released in September, 2020,
@command{ncra} supports the @samp{--per_record_weights} (or
@samp{--prw}) flag to utilize the command-line weights separately
specified by @samp{-w @var{wgt_arr}} (or @samp{--wgt @var{wgt_arr}})
for per-record weights instead of per-file-weights, where
@var{wgt_arr} is a 1-D array of weights.
This is useful when computing weighted averages with cyclically
varying weights, since the weights given on the command line will be 
repeated for the length of the timeseries.
Consider, for example, a @acronym{CMIP6} timeseries of historical
monthly mean emissions that one wishes to convert to a timeseries of
annual-mean emissions.
One can now weight each month by its number of days via:
@example
ncra --per_record_weights --mro -d time,,,12,12 --wgt \
  31,28,31,30,31,30,31,31,30,31,30,31 ~/monthly.nc ~/annual.nc
@end example
Note that the twelve weights will be implicitly repeated throughtout
the duration of the input file(s), which in this case may therefore
specify an interannual monthly timeseries that is reduced to a
timeseries of annual-means in the output.

Bear these exceptions in mind when weighting input:
First, @command{ncra} only applies weights if the arithmetic operation
type is averaging (@pxref{Operation Types}), i.e., for timeseries mean
and for timeseries mean absolute value.
Weights are never applied for minimization, square-roots, etc.
Second, @command{ncra} @emph{never weights} coordinate variables (e.g., 
@code{time}) regardless of the weighting performed on non-coordinate
variables.

@html
<a name="prm_ints"></a> <!-- http://nco.sf.net/nco.html#prm_ints -->
<a name="promote_ints"></a> <!-- http://nco.sf.net/nco.html#promote_ints -->
@end html
@cindex @code{--promote_ints}
@cindex @code{--prm_ints}
@cindex promotion
@cindex Boolean values
As of @acronym{NCO} @w{version 4.9.4}, released in September, 2020,
@command{ncra} supports the @samp{--promote_ints} (or @samp{prm_ints}) 
flags to output statistics of integer-valued input variables in
floating-point precision in the output file.
By default, arithmetic operators such as @command{ncra} auto-promote
integers to double-precision prior to arithmetic, then conduct the
arithmetic, then demote the values back to integers for final output.
The final stage (demotion) of this default behavior quantizes the
mantissa of the values and prevents, e.g., retaining the statisitical
means of Boolean (0 or 1-valued) input data as floating point data.
The @samp{--promote_ints} flag eliminates the demotion and causes the
statistical means of integer (@code{NC_BYTE}, @code{NC_SHORT},
@code{NC_INT}, @code{NC_INT64}) inputs to be output as
single-precision floating point (@code{NC_FLOAT}) variables.
This allows useful arithmetic to be performed on Boolean values stored
in the space-conserving @code{NC_BYTE} (single-byte) format.
@example
ncra --prm_ints in*.nc out.nc
@end example

@ignore
@c Old, and possibly future, documentation
@html
<a name="cb"></a> <!-- http://nco.sf.net/nco.html#cb -->
<a name="c2b"></a> <!-- http://nco.sf.net/nco.html#c2b -->
<a name="clm_bnd"></a> <!-- http://nco.sf.net/nco.html#clm_bnd -->
<a name="clm2bnd"></a> <!-- http://nco.sf.net/nco.html#clm2bnd -->
@end html
@cindex @code{--cb}
@cindex @code{--c2b}
@cindex @code{--clm_bnd}
@cindex @code{--clm2bnd}
As of @acronym{NCO} version 4.6.0 (May, 2016) @command{ncra} can honor 
the @acronym{CF} @code{climatology} and climatological statistics
conventions described in @ref{CF Conventions}.
This functionality only works when each input file contains only a
single record (timestep).
Currently this is opt-in with the @samp{--cb} flag (or long-option
equivalent @samp{--clm_bnd}), or with the @samp{--c2b} flag (or its
long-option equivalent @samp{--clm2bnd}) switches. 
Invoking @samp{--cb} causes @command{ncra} to:
@enumerate
@item Add a @code{climatology} attribute with value
``climatology_bounds'' the time coordinate, if necessary
@item Remove the @code{bounds} attribute from the time coordinate, if
necessary 
@item Output a variable named @code{climatology_bounds} with values that
      are minima/maxima of the input time coordinate @code{bounds}
      variable. 
@item Omit any input time coordinate @code{bounds} attribute and
      variable 
@item Ensure the @code{cell_methods} attribute for all variables is
      appropriate for climatologies within and over years.
      Climatologies within days will have incorrect units (the switch is
      currently opt-in so that incorrect units are not inadvertently generated).
      Please contact the authors if this functionality is important to you
      (The omission of climatologies within days is mainly a matter of
      trying to keep the switches and interface clean).
@end enumerate
Use the @samp{--c2b} flag (instead of @samp{--cb}) to convert the input
@code{climatology} bounds to a non-climatology @code{bounds} in the
output.  
In other words, use @samp{--c2b} when averaging sub-sampled
climatologies together to produce a continuous (non-climatologically
sub-sampled) mean. 
@example
# Use --cb to average months into a climatological month
ncra --cb 2014_01.nc 2015_01.nc 2016_01.nc clm_JAN.nc
# Use --cb to average climatological months into a climatological season
ncra --cb clm_DEC.nc clm_JAN.nc clm_FEB.nc clm_DJF.nc
# Four seasons make a complete year so use --c2b
ncra --c2b clm_DJF.nc clm_MAM.nc clm_JJA.nc clm_SON.nc clm_ANN.nc
@end example
Currently this functionality only works with climatologies within and
over years (not within or over days).
@end ignore

@noindent
@html
<a name="xmp_ncra"></a> <!-- http://nco.sf.net/nco.html#xmp_ncra -->
@end html
EXAMPLES

Average files @file{85.nc}, @file{86.nc}, @w{@dots{} @file{89.nc}}
along the record dimension, and store the results in @file{8589.nc}: 
@cindex globbing
@cindex @code{NINTAP}
@cindex Processor
@cindex @acronym{CCM} Processor
@example
ncra 85.nc 86.nc 87.nc 88.nc 89.nc 8589.nc
ncra 8[56789].nc 8589.nc
ncra -n 5,2,1 85.nc 8589.nc
@end example
These three methods produce identical answers.
@xref{Specifying Input Files}, for an explanation of the distinctions
between these methods.

@cindex Fortran
Assume the files @file{85.nc}, @file{86.nc}, @w{@dots{} @file{89.nc}}
each contain a record coordinate @var{time} of length 12 defined such
that the third record in @file{86.nc} contains data from March 1986,
etc. 
@acronym{NCO} knows how to hyperslab the record dimension across files.
Thus, to average data from December, 1985 through February, 1986:
@example
ncra -d time,11,13 85.nc 86.nc 87.nc 8512_8602.nc
ncra -F -d time,12,14 85.nc 86.nc 87.nc 8512_8602.nc
@end example
@noindent
The file @file{87.nc} is superfluous, but does not cause an error.
The @samp{-F} turns on the Fortran (1-based) indexing convention.
@cindex stride
The following uses the @var{stride} option to average all the March
temperature data from multiple input files into a single output file
@example
ncra -F -d time,3,,12 -v temperature 85.nc 86.nc 87.nc 858687_03.nc
@end example
@xref{Stride}, for a description of the @var{stride} argument.

Assume the @var{time} coordinate is incrementally numbered such that
January, @w{@math{1985 = 1}} and December, @w{@math{1989 = 60}}.
Assuming @samp{??} only expands to the five desired files, the following 
averages June, 1985--June, 1989: 
@example
ncra -d time,6.,54. ??.nc 8506_8906.nc
ncra -y max -d time,6.,54. ??.nc 8506_8906.nc
@end example
The second example identifies the maximum instead of averaging.
@xref{Operation Types}, for a description of all available statistical 
operations. 

@cindex seasonal average
@cindex average
@cindex subcycle
@cindex time-averaging
@command{ncra} includes the powerful subcycle and multi-record output
features (@pxref{Subcycle}). 
This example uses these features to compute and output winter
(@acronym{DJF}) averages for all winter seasons beginning with year
1990 and continuing to the end of the input file:
@example
ncra -O --mro -d time,"1990-12-01",,12,3 in.nc out.nc
@end example

The @samp{-w wgt} option weights input data @emph{per-file}
when explicit numeric weights are given on the command-line, or 
@emph{per-timestep} when the argument is a record variable that
resides in the file:
@example
ncra -w 31,31,28 dec.nc jan.nc feb.nc out.nc # Per-file weights
ncra -w delta_t in1.nc in2.nc in3.nc out.nc # Per-timestep weights
@end example
The first example weights the input differently per-file to produce
correctly weighted winter seasonal mean statistics.
The second example weights the input per-timestep to produce correctly 
weighted mean statistics.

@page
@html
<a name="ncrcat"></a> <!-- http://nco.sf.net/nco.html#ncrcat -->
@end html
@node ncrcat netCDF Record Concatenator, ncremap netCDF Remapper, ncra netCDF Record Averager, Reference Manual
@section @command{ncrcat} netCDF Record Concatenator
@cindex concatenation
@cindex record concatenation
@findex ncrcat

@noindent
SYNTAX
@example
ncrcat [-3] [-4] [-5] [-6] [-7] [-A] [-C] [-c] [--cmp @var{cmp_sng}]
[--cnk_byt @var{sz_byt}] [--cnk_csh @var{sz_byt}] [--cnk_dmn @var{nm},@var{sz_lmn}]
[--cnk_map @var{map}] [--cnk_min @var{sz_byt}] [--cnk_plc @var{plc}] [--cnk_scl @var{sz_lmn}]
[-D @var{dbg}] [-d @var{dim},[@var{min}][,[@var{max}][,[@var{stride}][,[@var{subcycle}][,[@var{interleave}]]]]]
[-F] [--fl_fmt @var{fl_fmt}]
[-G @var{gpe_dsc}] [-g @var{grp}[,@dots{}]] [--glb ...]
[-H] [-h] [--hdr_pad @var{nbr}] [--hpss] 
[-L @var{dfl_lvl}] [-l @var{path}] [--md5_digest] [-n @var{loop}]
[--no_tmp_fl] [--no_cll_msr] [--no_frm_trm] [--no_tmp_fl] 
[-O] [-o @var{output-file}] [-p @var{path}] [--ppc ...] [-R] [-r] [--ram_all] [--rec_apn]
[-t @var{thr_nbr}] [--unn] [-v @var{var}[,@dots{}]] [-X ...] [-x] 
[@var{input-files}] [@var{output-file}]
@end example

@noindent
DESCRIPTION

@command{ncrcat} concatenates record variables across an arbitrary
number of @var{input-files}.
@cindex record dimension
The final record dimension is by default the sum of the lengths of the 
record dimensions in the input files.
@xref{Statistics vs Concatenation}, for a description of the
distinctions between the various statistics tools and concatenators. 
@cindex multi-file operators
@cindex standard input
@cindex @code{stdin}
As a multi-file operator, @command{ncrcat} will read the list of
@var{input-files} from @code{stdin} if they are not specified 
as positional arguments on the command line 
(@pxref{Large Numbers of Files}).

Input files may vary in size, but each must have a record dimension.
The record coordinate, if any, should be monotonic (or else non-fatal
warnings may be generated).
@cindex hyperslab
Hyperslabs along the record dimension that span more than one file are  
handled correctly.
@cindex stride
@command{ncra} supports the @var{stride} argument to the @samp{-d}
hyperslab option for the record dimension only, @var{stride} is not
supported for non-record dimensions.

@findex ncpdq
@cindex packing
@cindex unpacking
@cindex @code{add_offset}
@cindex @code{scale_factor}
Concatenating a variable packed with different scales multiple datasets  
is beyond the capabilities of @command{ncrcat} (and @command{ncecat},
the other concatenator (@ref{Concatenation}).
@command{ncrcat} does not unpack data, it simply @emph{copies} the data
from the @var{input-files}, and the metadata from the @emph{first}
@var{input-file}, to the @var{output-file}. 
This means that data compressed with a packing convention must use
the identical packing parameters (e.g., @code{scale_factor} and
@code{add_offset}) for a given variable across @emph{all} input files.
Otherwise the concatenated dataset will not unpack correctly.
The workaround for cases where the packing parameters differ across
@var{input-files} requires three steps:
First, unpack the data using @command{ncpdq}.
Second, concatenate the unpacked data using @command{ncrcat}, 
Third, re-pack the result with @command{ncpdq}.

@cindex ARM conventions
@command{ncrcat} applies special rules to @acronym{ARM} convention time
fields (e.g., @code{time_offset}).
See @ref{ARM Conventions} for a complete description.

@noindent
@html
<a name="xmp_ncrcat"></a> <!-- http://nco.sf.net/nco.html#xmp_ncrcat -->
@end html
EXAMPLES

Concatenate files @file{85.nc}, @file{86.nc}, @w{@dots{} @file{89.nc}}
along the record dimension, and store the results in @file{8589.nc}: 
@cindex globbing
@cindex @code{NINTAP}
@cindex Processor
@cindex @acronym{CCM} Processor
@example
ncrcat 85.nc 86.nc 87.nc 88.nc 89.nc 8589.nc
ncrcat 8[56789].nc 8589.nc
ncrcat -n 5,2,1 85.nc 8589.nc
@end example
@noindent
These three methods produce identical answers.
@xref{Specifying Input Files}, for an explanation of the distinctions
between these methods.

@cindex Fortran
Assume the files @file{85.nc}, @file{86.nc}, @w{@dots{} @file{89.nc}}
each contain a record coordinate @var{time} of @w{length 12} defined
such that the third record in @file{86.nc} contains data from March
1986, etc. 
@acronym{NCO} knows how to hyperslab the record dimension across files. 
Thus, to concatenate data from December, 1985--February, 1986:
@example
ncrcat -d time,11,13 85.nc 86.nc 87.nc 8512_8602.nc
ncrcat -F -d time,12,14 85.nc 86.nc 87.nc 8512_8602.nc
@end example
@noindent
The file @file{87.nc} is superfluous, but does not cause an error.
When @command{ncra} and @command{ncrcat} encounter a file which does 
contain any records that meet the specified hyperslab criteria, they
disregard the file and proceed to the next file without failing.
The @samp{-F} turns on the Fortran (1-based) indexing convention.
@cindex stride

The following uses the @var{stride} option to concatenate all the March 
temperature data from multiple input files into a single output file
@example
ncrcat -F -d time,3,,12 -v temperature 85.nc 86.nc 87.nc 858687_03.nc
@end example
@xref{Stride}, for a description of the @var{stride} argument.

Assume the @var{time} coordinate is incrementally numbered such that
January, @w{1985 = 1} and December, @w{1989 = 60.}
Assuming @code{??} only expands to the five desired files, the following 
concatenates June, 1985--June, 1989: 
@example
ncrcat -d time,6.,54. ??.nc 8506_8906.nc
@end example

@page
@html
<a name="ncremap"></a> <!-- http://nco.sf.net/nco.html#ncremap -->
@end html
@node ncremap netCDF Remapper, ncrename netCDF Renamer, ncrcat netCDF Record Concatenator, Reference Manual
@section @command{ncremap} netCDF Remapper
@cindex remap
@cindex regrid
@findex ncremap

@noindent
SYNTAX
@example
ncremap [-3] [-4] [-5] [-6] [-7]
[-a @var{alg_typ}] [--a2o] [--add_fll] [--area_dgn] 
[-D @var{dbg_lvl}] [-d @var{dst_fl}] [--d2f] [--dpt] [--dpt_fl=@var{dpt_fl}]
[--dt_sng=@var{dt_sng}] [--esmf_typ=@var{esmf_typ}] 
[--fl_fmt=@var{fl_fmt}] [-G @var{grd_sng}] [-g @var{grd_dst}]
[-I @var{drc_in}] [-i @var{input-file}] [-j @var{job_nbr}] [-L @var{dfl_lvl}]
[-M] [-m @var{map_fl}] [--mpi_nbr=@var{mpi_nbr}] [--mpi_pfx=@var{mpi_pfx}] [--msh_fl=@var{msh_fl}]
[--msk_apl] [--msk_dst=@var{msk_dst}] [--msk_out=@var{msk_out}] [--msk_src=@var{msk_src}] [--mss_val=@var{mss_val}]
[-n @var{nco_opt}] [--nm_dst=@var{nm_dst}] [--nm_src=@var{nm_src}] 
[--no_add_fll] [--no_cll_msr] [--no_frm_trm] [--no_permute] [--no_stdin] [--no_stg_grd]
[-O @var{drc_out}] [-o @var{output-file}] [-P @var{prc_typ}] [-p @var{par_typ}]
[--pdq=@var{pdq_opt}] [--ppc=@var{ppc_opt}] [--preserve=@var{prs_stt}] [--ps_nm=@var{ps_nm}]
[-R @var{rgr_opt}] [--rgn_dst] [--rgn_src] [--rnr_thr=@var{rnr_thr}] 
[--rrg_bb_wesn=@var{bb_wesn}] [--rrg_dat_glb=@var{dat_glb}] [--rrg_grd_glb=@var{grd_glb}]
[--rrg_grd_rgn=@var{grd_rgn}] [--rrg_rnm_sng=@var{rnm_sng}]
[-s @var{grd_src}] [--sgs_frc=@var{sgs_frc}] [--sgs_msk=@var{sgs_msk}] [--sgs_nrm=@var{sgs_nrm}]
[--skl=@var{skl-file}] [--stdin] [-T @var{drc_tmp}] [-t @var{thr_nbr}]
[-U] [-u @var{unq_sfx}] [--ugrid=@var{ugrid-file}] [--uio]
[-V @var{rgr_var}] [-v @var{var_lst}[,@dots{}]] [--version] [--vrb=@var{vrb_lvl}] 
[--vrt_in=@var{vrt_fl}] [--vrt_out=@var{vrt_fl}] [--vrt_ntp=@var{vrt_ntp}] [--vrt_xtr=@var{vrt_xtr}]
[-W @var{wgt_opt}] [-w @var{wgt_cmd}] [-x @var{xtn_lst}[,@dots{}]] [--xcl_var]
[--xtr_nsp=@var{xtr_nsp}] [--xtr_xpn=@var{xtr_xpn}]
[@var{input-files}] [@var{output-file}]
@end example

@noindent
DESCRIPTION

@command{ncremap} remaps the data file(s) in @var{input-file}, in
@var{drc_in}, or piped through standard input, to the horizontal grid
specified by (in descending order of precedence) @var{map_fl},
@var{grd_dst}, or @var{dst_fl} and stores the result in
@var{output-file}(s).  
If a vertical grid @var{vrt_fl} is provided, @command{ncremap} will
(also) vertically interpolate the input file(s) to that grid.
When no @var{input-file} is provided, @command{ncremap} operates in
``map-only'' mode where it exits after producing an annotated map-file. 
@command{ncremap} was introduced to @acronym{NCO} in version 4.5.4
(December, 2015). 

@command{ncremap} is a ``super-operator'' that orchestrates the
regridding features of several different programs including other
@acronym{NCO} operators.
Under the hood @acronym{NCO} applies pre-computed remapping weights or, 
when necessary, generates and infers grids, generates remapping
weights itself or calls external programs to generate the weights,
and then applies the weights (i.e., regrids).

@cindex @command{ncremap}
@cindex @command{ESMF_RegridWeightGen}
@cindex @command{GenerateOfflineMap}
@cindex @command{GenerateOverlapMesh}
@cindex @command{mbtempest}
@cindex @command{mbconvert}
@cindex @command{mbpart}
@cindex @acronym{ERWG}
@cindex @acronym{MOAB}
@cindex TempestRemap
@cindex @acronym{NCL}
@cindex Conda
Unlike the rest of @acronym{NCO}, @command{ncremap} and
@command{ncclimo} are shell scripts, not compiled binaries@footnote{ 
This means that newer (including user-modified) versions of
@command{ncremap} work fine without re-compiling @acronym{NCO}.
Re-compiling is only necessary to take advantage of new features or
fixes in the @acronym{NCO} binaries, not to improve @command{ncremap}.
One may download and give executable permissions to the latest source 
at @url{https://github.com/nco/nco/tree/master/data/ncremap} without
re-installing the rest of @acronym{NCO}.}. 
@html
<a name="HDF5_USE_FILE_LOCKING"></a> <!-- http://nco.sf.net/nco.html#HDF5_USE_FILE_LOCKING -->
@end html
@cindex @code{HDF5_USE_FILE_LOCKING}
As of @acronym{NCO} 4.9.2 (February, 2020), the @command{ncclimo}
and @command{ncremap} scripts export the environment variable
@code{HDF5_USE_FILE_LOCKING} with a value of @code{FALSE}.
This prevents failures of these operators that can occur with some
versions of the underlying HDF library that attempt to lock files
on file systems that cannot or do not support it.
@command{ncremap} wraps the underlying regridder (@command{ncks}) and
external executables to produce a friendly interface to regridding.
Without any external dependencies, @command{ncremap} applies weights
from a pre-exisiting map-file to a source data file to produce a
regridded dataset. 
Source and destination datasets may be on any Swath, Curvilinear,
Rectangular, or Unstructured Data (@acronym{SCRUD}) grid. 
@command{ncremap} will also use its own algorithms or, when requested,  
external programs @acronym{ESMF}'s @command{ESMF_RegridWeightGen}
(@acronym{ERWG}) or
@acronym{MOAB}'s
@command{mbconvert}/@command{mbpart}/@command{mbtempest}, 
TempestRemap's
@command{GenerateOverlapMesh}/@command{GenerateOfflineMap}) to
generate weights and mapfiles.
In order to use the weight-generation options, either invoke an
internal @acronym{NCO} weight-generation algorithm (e.g.,
@samp{--alg_typ=nco}), or ensure that the desired external
weight-generation package is installed and on your @code{$PATH}.
The recommended way to obtain @acronym{ERWG} is as distributed in
binary format.  
Many @acronym{NCO} users already have @acronym{NCL} on their
system(s), and @acronym{NCL} usually comes with @acronym{ERWG}. 
Since about June, 2016, the Conda @acronym{NCO} package will also
install @acronym{ERWG}
@footnote{   
Install the Conda @acronym{NCO} package with
@samp{conda install -c conda-forge nco}.}. 
Then be sure the directory containing the @acronym{ERWG} executable is
on your @code{$PATH} before using @command{ncremap}.
As a fallback, @acronym{ERWG} may also be installed from source:
@url{https://earthsystemcog.org/projects/esmf/download_last_public}.
@command{ncremap} can also generate and utilize mapfiles created by
TempestRemap, 
@url{https://github.com/ClimateGlobalChange/tempestremap}.
Until about April, 2019, TempestRemap had to be built from source
because there were no binary distributions of it.
As of @acronym{NCO} @w{version 4.8.0}, released in May, 2019, the
Conda @acronym{NCO} package automatically installs the new
TempestRemap Conda package so building from source is not necessary. 
Please contact those projects for support on building and installing
their software, which makes @command{ncremap} more functional and
user-friendly.
As of @acronym{NCO} @w{version 5.0.2} from September, 2021,
@command{ncremap} users can also use the @acronym{MOAB} regridding
toolchain.
@acronym{MOAB} and @acronym{ERWG} perform best in an @acronym{MPI}
environment. 
One can easily obtain such an environment with Conda
@footnote{
Install the Conda @acronym{MPI} versions of the
@acronym{ERWG} and @acronym{MOAB} packages with
@samp{conda install -c conda-forge moab=5.3.0=*mpich_tempest* esmf}.}.
Please ensure you have the latest version of @acronym{ERWG},
@acronym{MOAB}, and/or TempestRemap before reporting any related 
problems to @acronym{NCO}.  

As mentioned above, @command{ncremap} orchestrates the regridding
features of several different programs.
@command{ncremap} runs most quickly when it is supplied with a
pre-computed mapfile.  
However, @command{ncremap} will also (call other programs to) compute
mapfiles when necessary and when given sufficient grid information.
Thus it is helpful to understand when @command{ncremap} will and will 
not internally generate a mapfile.
Supplying input data files and a pre-computed mapfile @emph{without}
any other grid information causes @command{ncremap} to regrid the data
files without first pausing to internally generate a mapfile.
On the other hand, supplying any grid information (i.e., using any of
the @samp{-d}, @samp{-G}, @samp{-g}, or @samp{-s} switches described
below), causes @command{ncremap} to internally (re-)generate the
mapfile by combining the supplied and inferred grid information.
A generated mapfile is given a default name unless a user-specified name
is supplied with @samp{-m @var{map_fl}}. 

@unnumberedsubsec Fields not regridded by @command{ncremap}

Most people ultimately use @command{ncremap} to regrid data, yet not all
data can or should be regridded in the sense of applying a sparse-matrix
of weights to an input field to produce and output field.
Certain fields (e.g., the longitude coordinate) specify the grid.
These fields must be provided in order to compute the weights that are
used to regrid. 
The regridded usually copies these fields ``as is'' directly into
regridded files, where they describe the destination grid, and replace
or supercede the source grid information.
Other fields are extensive grid properties (e.g., the number of cells
adjacent to a given cell) that may apply only to the source (not the
destination) grid, or be too difficult to re-compute for the destination
grid. 
@command{ncremap} contains an internal database of fields that it will
not propagate or regrid.
First are variables with names identical to the coordinate names found
in an ever-growing collection of publicly available geoscience datasets
(@acronym{CMIP}, @acronym{NASA}, etc.):@*

@cindex @code{CO_Latitude}
@cindex @code{CO_Longitude}
@cindex @code{LAT}
@cindex @code{LON}
@cindex @code{LatitudeCornerpoints}
@cindex @code{Latitude}
@cindex @code{LongitudeCornerpoints}
@cindex @code{Longitude}
@cindex @code{S1_Latitude}
@cindex @code{S1_Longitude}
@cindex @code{TLAT}
@cindex @code{TLONG}
@cindex @code{TLON}
@cindex @code{ULAT}
@cindex @code{ULONG}
@cindex @code{ULON}
@cindex @code{XLAT_M}
@cindex @code{XLAT}
@cindex @code{XLONG_M}
@cindex @code{XLONG}
@cindex @code{area}
@cindex @code{bounds_lat}
@cindex @code{bounds_lon}
@cindex @code{global_latitude0}
@cindex @code{global_longitude0}
@cindex @code{gridcell_area}
@cindex @code{gw}
@cindex @code{lat_bnds}
@cindex @code{lat_vertices}
@cindex @code{latitude0}
@cindex @code{latitude_bnds}
@cindex @code{latitude}
@cindex @code{latt_bounds}
@cindex @code{latu_bounds}
@cindex @code{lat}
@cindex @code{lon_bnds}
@cindex @code{lon_vertices}
@cindex @code{longitude0}
@cindex @code{longitude_bnds}
@cindex @code{longitude}
@cindex @code{lont_bounds}
@cindex @code{lonu_bounds}
@cindex @code{lon}
@cindex @code{nav_lat}
@cindex @code{nav_lon}
@cindex @code{slat}
@cindex @code{slon}
@cindex @code{w_stag}

@code{area}, @code{gridcell_area}, @code{gw}, @code{LAT}, @code{lat},
@code{Latitude}, @code{latitude}, @code{nav_lat},
@code{global_latitude0}, @code{latitude0}, @code{slat}, @code{TLAT},
@code{ULAT}, @code{XLAT}, @code{XLAT_M}, @code{CO_Latitude},
@code{S1_Latitude}, @code{lat_bnds}, @code{lat_vertices},
@code{latt_bounds}, @code{latu_bounds}, @code{latitude_bnds},
@code{LatitudeCornerpoints}, @code{bounds_lat}, @code{LON}, @code{lon},
@code{Longitude}, @code{longitude}, @code{nav_lon},
@code{global_longitude0}, @code{longitude0}, @code{slon}, @code{TLON},
@code{TLONG}, @code{ULON}, @code{ULONG}, @code{XLONG}, @code{XLONG_M},
@code{CO_Longitude}, @code{S1_Longitude}, @code{lon_bnds},
@code{lon_vertices}, @code{lont_bounds}, @code{lonu_bounds},
@code{longitude_bnds}, @code{LongitudeCornerpoints}, @code{bounds_lon},
and @code{w_stag}.

Files produced by @acronym{MPAS} models may contain these variables that
will not be regridded:@*

@cindex @code{angleEdge}
@cindex @code{areaTriangle}
@cindex @code{cellsOnCell}
@cindex @code{cellsOnEdge}
@cindex @code{cellsOnVertex}
@cindex @code{dcEdge}
@cindex @code{dvEdge}
@cindex @code{edgeMask}
@cindex @code{edgesOnCell}
@cindex @code{edgesOnEdge}
@cindex @code{edgesOnVertex}
@cindex @code{indexToCellID}
@cindex @code{indexToEdgeID}
@cindex @code{indexToVertexID}
@cindex @code{kiteAreasOnVertex}
@cindex @code{latCell}
@cindex @code{latEdge}
@cindex @code{latVertex}
@cindex @code{lonCell}
@cindex @code{lonEdge}
@cindex @code{lonVertex}
@cindex @code{maxLevelEdgeTop}
@cindex @code{meshDensity}
@cindex @code{nEdgesOnCell}
@cindex @code{nEdgesOnEdge}
@cindex @code{vertexMask}
@cindex @code{verticesOnCell}
@cindex @code{verticesOnEdge}
@cindex @code{weightsOnEdge}
@cindex @code{xEdge}
@cindex @code{yEdge}
@cindex @code{zEdge}
@cindex @code{xVertex}
@cindex @code{yVertex}
@cindex @code{zVertex}

@code{angleEdge}, @code{areaTriangle}, @code{cellsOnCell},
@code{cellsOnEdge}, @code{cellsOnVertex}, @code{dcEdge}, @code{dvEdge},
@code{edgeMask}, @code{edgesOnCell}, @code{edgesOnEdge},
@code{edgesOnVertex}, @code{indexToCellID}, @code{indexToEdgeID},
@code{indexToVertexID}, @code{kiteAreasOnVertex}, @code{latCell},
@code{latEdge}, @code{latVertex}, @code{lonCell}, @code{lonEdge},
@code{lonVertex}, @code{maxLevelEdgeTop}, @code{meshDensity},
@code{nEdgesOnCell}, @code{nEdgesOnEdge}, @code{vertexMask},
@code{verticesOnCell}, @code{verticesOnEdge}, @code{weightsOnEdge},
@code{xEdge}, @code{yEdge}, @code{zEdge}, @code{xVertex},
@code{yVertex}, and @code{zVertex}.

Most of these fields that @command{ncremap} will not regrid are also
fields that @acronym{NCO} size-and-rank-preserving operators will not
modify, as described in @ref{CF Conventions}. 

@unnumberedsubsec Options specific to @command{ncremap}

The following summarizes features unique to @command{ncremap}.  
Features common to many operators are described in 
@ref{Shared features}. 

@table @samp

@html
<a name="alg_typ"></a> <!-- http://nco.sf.net/nco.html#alg_typ -->
<a name="neareststod"></a> <!-- http://nco.sf.net/nco.html#neareststod -->
<a name="nstod"></a> <!-- http://nco.sf.net/nco.html#nstod -->
<a name="stod"></a> <!-- http://nco.sf.net/nco.html#stod -->
<a name="nearestdtos"></a> <!-- http://nco.sf.net/nco.html#nearestdtos -->
<a name="ndtos"></a> <!-- http://nco.sf.net/nco.html#ndtos -->
<a name="dtos"></a> <!-- http://nco.sf.net/nco.html#dtos -->
<a name="nds"></a> <!-- http://nco.sf.net/nco.html#nds -->
<a name="patch"></a> <!-- http://nco.sf.net/nco.html#patch -->
<a name="patc"></a> <!-- http://nco.sf.net/nco.html#patc -->
<a name="pch"></a> <!-- http://nco.sf.net/nco.html#pch -->
<a name="bilinear"></a> <!-- http://nco.sf.net/nco.html#bilinear -->
<a name="bilin"></a> <!-- http://nco.sf.net/nco.html#bilin -->
<a name="blin"></a> <!-- http://nco.sf.net/nco.html#blin -->
<a name="conserve"></a> <!-- http://nco.sf.net/nco.html#conserve -->
<a name="aave"></a> <!-- http://nco.sf.net/nco.html#aave -->
<a name="fv2se_flx"></a> <!-- http://nco.sf.net/nco.html#fv2se_flx -->
<a name="se2fv_flx"></a> <!-- http://nco.sf.net/nco.html#se2fv_flx -->
<a name="fv2se_stt"></a> <!-- http://nco.sf.net/nco.html#fv2se_stt -->
<a name="se2fv_stt"></a> <!-- http://nco.sf.net/nco.html#se2fv_stt -->
<a name="fv2se_alt"></a> <!-- http://nco.sf.net/nco.html#fv2se_alt -->
<a name="se2fv_alt"></a> <!-- http://nco.sf.net/nco.html#se2fv_alt -->
<a name="se2se"></a> <!-- http://nco.sf.net/nco.html#se2se -->
<a name="cs2cs"></a> <!-- http://nco.sf.net/nco.html#cs2cs -->
<a name="fv2fv"></a> <!-- http://nco.sf.net/nco.html#fv2fv -->
<a name="fv2fv_flx"></a> <!-- http://nco.sf.net/nco.html#fv2fv_flx -->
<a name="fv2fv_stt"></a> <!-- http://nco.sf.net/nco.html#fv2fv_stt -->
<a name="rll2rll"></a> <!-- http://nco.sf.net/nco.html#rll2rll -->
<a name="nco_con"></a> <!-- http://nco.sf.net/nco.html#nco_con -->
<a name="nco_dwe"></a> <!-- http://nco.sf.net/nco.html#nco_dwe -->
<a name="nco_idw"></a> <!-- http://nco.sf.net/nco.html#nco_idw -->
@end html
@cindex @code{-a @var{alg_typ}}
@cindex @var{alg_typ}
@cindex @code{--alg_typ}
@cindex @code{--algorithm}
@cindex @code{--regrid_algorithm}
@cindex @code{dwe}
@cindex @code{idw}
@cindex @code{neareststod}
@cindex @code{nstod}
@cindex @code{stod}
@cindex @code{nsd}
@cindex @code{nearestdtos}
@cindex @code{ndtos}
@cindex @code{dtos}
@cindex @code{nds}
@cindex @code{patch}
@cindex @code{patc}
@cindex @code{pch}
@cindex @code{bilinear}
@cindex @code{bilin}
@cindex @code{blin}
@cindex @code{aave}
@cindex @code{conserve}
@cindex @code{conserve2nd}
@item -a @var{alg_typ} (@code{--alg_typ}, @code{--algorithm}, @code{--regrid_algorithm})
Specifies the interpolation algorithm for weight-generation for use by
@command{ESMF_RegridWeightGen} (@acronym{ERWG}), @acronym{MOAB},
@acronym{NCO}, and/or TempestRemap. 
@command{ncremap} unbundles this algorithm choice from the rest of
the weight-generator invocation syntax because users more frequently
change interpolation algorithms than other options
(that can be changed with @samp{-W @var{wgt_opt}}).
@command{ncremap} can invoke all seven @acronym{ERWG} weight
generation algorithms, one @acronym{NCO} algorithm, and eight
TempestRemap algorithms (with both @acronym{TR} and @acronym{MOAB}). 

The seven @acronym{ERWG} weight generation algorithms are:
@code{bilinear} (default, acceptable abbreviations are @code{bilin}, @code{blin}, @code{bln}), 
@code{conserve} (or @code{conservative}, @code{cns}, @code{c1}, or @code{aave}), 
@code{conserve2nd} (or @code{conservative2nd}, @code{c2}, or @code{c2nd})
(@acronym{NCO} supports @code{conserve2nd} as of version 4.7.4 (April, 2018)), 
@code{nearestdtos} (or @code{nds} or @code{dtos} or @code{ndtos}), 
@code{neareststod} (or @code{nsd} or @code{stod} or @code{nstod}), 
and @code{patch} (or @code{pch} or @code{patc}). 
See @acronym{ERWG} documentation
@uref{http://www.earthsystemmodeling.org/esmf_releases/public/ESMF_6_3_0rp1/ESMF_refdoc/node3.html#SECTION03020000000000000000, here}
for detailed descriptions of @acronym{ERWG} algorithms.
@cindex @code{nco_con}
@cindex @code{nco_cns}
@cindex @code{nco_conserve}
@cindex @code{nco}

@html
<a name="idw"></a> <!-- http://nco.sf.net/nco.html#idw -->
<a name="dwe"></a> <!-- http://nco.sf.net/nco.html#dwe -->
<a name="nco_idw"></a> <!-- http://nco.sf.net/nco.html#nco_idw -->
<a name="nco_dwe"></a> <!-- http://nco.sf.net/nco.html#nco_dwe -->
<a name="inverse_distance"></a> <!-- http://nco.sf.net/nco.html#inverse_distance -->
<a name="inverse_distance_weighted"></a> <!-- http://nco.sf.net/nco.html#inverse_distance_weighted -->
<a name="distance_weighted"></a> <!-- http://nco.sf.net/nco.html#distance_weighted -->
<a name="nearest_neighbor"></a> <!-- http://nco.sf.net/nco.html#nearest_neighbor -->
@end html
@cindex @code{nco_idw}
@cindex @code{nco_dwe}
@cindex @code{idw}
@cindex @code{dwe}
@cindex @code{inverse_distance_weighted}
@cindex @code{distance_weighted}
@cindex @code{nco_nearest_neighbor}
@cindex @acronym{DWE}
@cindex @acronym{IDW}
@cindex inverse-distance-weighted interpolation/extrapolation
@cindex distance-weighted extrapolation
@cindex nearest-neighbor extrapolation
@command{ncremap} implements its own internal weight-generation
algorithm as of @acronym{NCO} version 4.8.0 (May, 2019).
The first @acronym{NCO}-native algorithm is a first-order conservative
algorithm that competes well in accuracy with similar algorithms
(e.g., @acronym{ERWG}'s conservative algorithm).
This algorithm is built-in to @acronym{NCO} and requires no external
software so it is the default weight generation algorithm and is
recommended for everyday use. 
The algorithm may also be explicitly invoked with @code{nco_con} (or
@code{nco_cns}, @code{nco_conservative}, or simply @code{nco}).

As of @acronym{NCO} version 4.9.4 (September, 2019) @command{ncremap}
supports a second internal weight-generation algorithm based on
inverse-distance-weighted (@acronym{IDW}) interpolation/extrapolation. 
@acronym{IDW} is similar to the @acronym{ERWG} @code{nearestidavg}
extrapolation alorithm, and accepts the same two parameters as input:
@samp{--xtr_xpn @var{xtr_xpn}} sets the (absolute value of) the
exponent used in inverse distance weighting (default @w{is 2.0}), and 
@samp{--xtr_nsp @var{xtr_nsp}} sets the number of source points used
in the extrapolation (default @w{is 8}).
@command{ncremap} applies @command{NCO}'s @acronym{IDW} to the entire
destination grid, not just to points with missing/masked values,
whereas @acronym{ERWG} uses distance-weighted-extrapolation
(@acronym{DWE}) solely for extrapolation to missing data points. 
Thus @command{NCO}'s @acronym{IDW} is more often used as an
alternative to bilinear interpolation since it interpolates between
known regions and extrapolates to unknown regions.
@example
@verbatim
ncremap --alg_typ=nco_idw -s src.nc -d dst.nc -m map.nc
ncremap -a nco_idw --xtr_xpn=1.0 -s src.nc -d dst.nc -m map.nc
ncremap -a nco_idw --xtr_nsp=1   -s src.nc -d dst.nc -m map.nc
@end verbatim
@end example

@ignore
It promises to free users from the bondage of spherically-connected cell edges.
We are working to make it geometrically exact for gridcells with
small-circle edges, such as the latitude circles in @acronym{RLL}
grids. 
@end ignore
@html
<a name="moab"></a> <!-- http://nco.sf.net/nco.html#moab -->
<a name="mbtr"></a> <!-- http://nco.sf.net/nco.html#mbtr -->
<a name="MOAB"></a> <!-- http://nco.sf.net/nco.html#MOAB -->
@end html
@cindex @acronym{MOAB}
@cindex TempestRemap
@cindex Tempest2
@cindex @code{se2fv_flx}
@cindex @code{fv2se_flx}
@cindex @code{se2fv_stt}
@cindex @code{fv2se_stt}
@cindex @code{se2fv_alt}
@cindex @code{fv2se_alt}
@cindex @code{se2se}
@cindex @code{cs2cs}
@cindex @code{fv2fv}
@cindex @code{fv2fv_flx}
@cindex @code{fv2fv_stt}
@cindex @code{rll2rll}
@cindex @code{mono_se2fv}
@cindex @code{conservative_monotone_se2fv}
@cindex @code{monotr_fv2se}
@cindex @code{conservative_monotone_fv2se}
@cindex @code{highorder_se2fv}
@cindex @code{accurate_conservative_nonmonotone_se2fv}
@cindex @code{highorder_fv2se}
@cindex @code{accurate_conservative_nonmonotone_fv2se}
@cindex @code{intbilin_se2fv}
@cindex @code{accurate_monotone_nonconservative_se2fv}
@cindex @code{mono_fv2se}
@cindex @code{conservative_monotone_fv2se_alt}
@command{ncremap} can invoke eight preconfigured TempestRemap 
weight-generation algorithms, and one generic algorithm
(@code{tempest}) for which users should provide their own 
options. 
As of @acronym{NCO} version 4.7.2 (January, 2018), @command{ncremap}
implemented the six @acronym{E3SM}-recommended TempestRemap mapping
algorithms between @acronym{FV} and @acronym{SE} flux, state, and
other variables.
@command{ncremap} originated some (we hope) common-sense names
for these algorithms (@code{se2fv_flx}, @code{se2fv_stt},
@code{se2fv_alt}, @code{fv2se_flx}, @code{fv2se_stt}, and
@code{fv2se_alt}), and also allows more mathematically precise
synonyms (shown below).
As of @acronym{NCO} version 4.9.0 (December, 2019), @command{ncremap} 
added two further boutique mappings (@code{fv2fv_flx} and
@code{fv2fv_stt}). 
The @samp{-a tempest} algorithm can be specified with the precise
TempestRemap options as arguments to the @samp{-W} (or
@samp{--wgt_opt}) option.  
Support for the named algorithms requires TempestRemap version 2.0.0
or later (some option combinations fail with earlier versions).
The @acronym{MOAB} algorithms are identical TempestRemap algorithms
@footnote{
However, mapping weights generated by
Although @acronym{MOAB} and TempestRemap use the same numerical
algorithms, they are likely to produce slightly different weights
due to round-off differences. 
@acronym{MOAB} is heavily parallelized and computes and adds terms
together in an unpredictable order compared to the serial TempestRemap.}.
Use the same algorithm names to select them.
Passing the @code{--mpi_nbr} option to  @command{ncremap} causes it
to invoke the @acronym{MOAB} toolchain to compute weights for any
TempestRemap algorithm (otherwise the @acronym{TR} toolchain is
used).

Generate and use the recommended weights to remap fluxes from
@acronym{FV} to @acronym{FV} grids, for example, with
@example
@verbatim
ncremap -a fv2fv_flx --src_grd=src.g --dst_grd=dst.nc -m map.nc
ncremap -m map.nc in.nc out.nc
@end verbatim
@end example
This causes @command{ncremap} to automatically invoke TempestRemap with
the boutique options 
@samp{--in_type fv --in_np 1 --out_type fv --out_np 1}
that are recommended by @acronym{E3SM} for conservative and monotone  
remapping of fluxes.
Invoke @acronym{MOAB} to compute these weights by adding the
@samp{--mpi_nbr=@var{mpi_nbr}} option:
@example
@verbatim
ncremap --mpi_nbr=8 -a fv2fv_flx --src_grd=src.g --dst_grd=dst.nc -m map.nc
@end verbatim
@end example
This causes @command{ncremap} to automatically invoke multiple
components of the @acronym{MOAB} toolchain:
@example
@verbatim
mbconvert -B -o PARALLEL=WRITE_PART -O PARALLEL=BCAST_DELETE \
  -O PARTITION=TRIVIAL -O PARALLEL_RESOLVE_SHARED_ENTS \
  "src.g" "src.h5m"
mbconvert -B -o PARALLEL=WRITE_PART -O PARALLEL=BCAST_DELETE \
  -O PARTITION=TRIVIAL -O PARALLEL_RESOLVE_SHARED_ENTS \
  "dst.nc" "dst.h5m"
mbpart 8 --zoltan RCB "src.h5m" "src_8p.h5m"
mbpart 8 --zoltan RCB --recompute_rcb_box --scale_sphere \
  --project_on_sphere 2 "dst.h5m" "dst_8p.h5m"
mpirun -n 8 mbtempest --type 5 --weights \
  --load "src_8p.h5m" --load "dst_8p.h5m" \
  --method fv --order 1 --method fv --order 1 \
  --file "map.nc"
ncatted -O --gaa <command lines> map.nc map.nc
@end verbatim
@end example
The @acronym{MOAB} toolchain should produce a map-file identical, to
rounding precision, to one produced by @acronym{TR}.
When speed matters (i.e., large grids), and the algorithm is supported
(i.e., @code{fv2fv_flx}), invoke @acronym{MOAB}, otherwise invoke
@acronym{TR}. 

@cindex Galerkin methods
@cindex @code{cgll}
@cindex @code{dgll}
@cindex @code{mono}
@cindex @code{--mono}
TempestRemap options have the following meanings:
@code{mono} specifies a monotone remapping, i.e., one that does not
generate any new extrema in the field variables.a
@code{cgll} indicates the input or output are represented by a
continuous Galerkin method on Gauss-Lobatto-Legendre nodes.
This is appropriate for spectral element datasets.
(TempestRemap also supports, although @acronym{NCO} does not invoke,
the @code{dgll} option for a discontinuous Galerkin method on
Gauss-Lobatto-Legendre nodes.)
It is equivalent to, yet simpler to remember and to invoke than 
@example
@verbatim
ncremap -a tempest --src_grd=se.g --dst_grd=fv.nc -m map.nc \
        -W '--in_type cgll --in_np 4 --out_type fv --mono'
@end verbatim
@end example
Specifying @samp{-a tempest} without additional options in the
@samp{-W} clause causes TempestRemap to employ defaults.
The default configuration requires both input and output grids to be
@acronym{FV}, and produces a conservative, non-monotonic mapping.
The @samp{-a fv2fv} option described below may produce more desirable
results than this default for many users.
Using @samp{-a tempest} alone without other options for spectral
element grids will lead to undefined and likely unintentional
results. 
In other words, @samp{-a tempest} is intended to be used in
conjunction with a @samp{-W} option clause to supply your own
combination of TempestRemap options that does not duplicate one of the 
boutique option collections that already has its own name.

The full list of supported canonical algorithm names, their synonyms,
and boutique options passed to @command{GenerateOfflineMap} or to
@command{mbtempest} follow.

@html
<a name="bug_moab"></a> <!-- http://nco.sf.net/nco.html#bug_moab -->
@end html
@cartouche
Caveat lector:
As of September, 2021 @acronym{MOAB}-generated weights are only
trustworthy for the @code{fv2fv_flx} algorithm. 
The options for all other algorithms are implemented as indicated
though they should be invoked for testing purposes only.
High order and spectral element maps are completely unsupported.
@acronym{MOAB} anticipates supporting more TempestRemap algorithms
in the future.
Always use the map-checker to test maps before use, e.g., with
@samp{ncks --chk_map map.nc}.
@end cartouche

@table @asis
@item @code{se2fv_flx} (synonyms @code{mono_se2fv}, @code{conservative_monotone_se2fv})
@acronym{TR} options: @samp{--in_type cgll --in_np 4 --out_type fv --mono}@*
@acronym{MOAB} options: @samp{--method cgll --order 4 --global_id GLOBAL_DOFS --method fv --monotonic 1 --global_id GLOBAL_ID}
@item @code{fv2se_flx} (synonyms @code{monotr_fv2se}, @code{conservative_monotone_fv2se}),
@acronym{TR} options: @samp{--in_type cgll --in_np 4 --out_type fv --mono}.
For @code{fv2se_flx} the weights are generated with options identical to
@code{se2fv_flx}, and then the transpose of the resulting weight matrix
is employed.@*
@acronym{MOAB} options: @samp{--method cgll --order 4 --method fv --monotonic 1}
@item @code{se2fv_stt} (synonyms @code{highorder_se2fv}, @code{accurate_conservative_nonmonotone_se2fv}),
@acronym{TR} options: @samp{--in_type cgll --in_np 4 --out_type fv}@*
@acronym{MOAB} options: @samp{--method cgll --order 4 --method fv}
@item @code{fv2se_stt} (synonyms @code{highorder_fv2se}, @code{accurate_conservative_nonmonotone_fv2se}),
@acronym{TR} options: @samp{--in_type fv --in_np 2 --out_type cgll --out_np 4}@*
@acronym{MOAB} options: @samp{--method fv --order 2 --method cgll --order 4}
@item @code{se2fv_alt} (synonyms @code{intbilin_se2fv}, @code{accurate_monotone_nonconservative_se2fv}),
@acronym{TR} options: @samp{--in_type cgll --in_np 4 --out_type fv --method mono3 --noconserve}@*
@acronym{MOAB} options: @samp{--method cgll --order 4 --method fv --monotonic 3 --noconserve}
@item @code{fv2se_alt} (synonyms @code{mono_fv2se}, @code{conservative_monotone_fv2se_alt}),
@acronym{TR} options: @samp{--in_type fv --in_np 1 --out_type cgll --out_np 4 --mono}@*
@acronym{MOAB} options: @samp{--method fv --order 1 --method cgll --order 4 --monotonic 1}
@item @code{se2se} (synonyms @code{cs2cs}, @code{conservative_monotone_se2se}),
@acronym{TR} options: @samp{--in_type cgll --in_np 4 --out_type cgll --out_np 4 --mono}@*
@acronym{MOAB} options: @samp{--method cgll --order 4 --method cgll --order 4 --monotonic 1}
@item @code{fv2fv} (synonyms @code{rll2rll}),
@acronym{TR} options: @samp{--in_type fv --in_np 2 --out_type fv}@*
@acronym{MOAB} options: @samp{--method fv --order 2 --method fv}
@item @code{fv2fv_flx} (synonyms @code{fv2fv_mono}, @code{conservative_monotone_fv2fv}),
@acronym{TR} options: @samp{--in_type fv --in_np 1 --out_type fv --out_np 1}@*
@acronym{MOAB} options: @samp{--method fv --order 1 --method fv --order 1}
@item @code{fv2fv_stt} (synonyms @code{fv2fv_highorder}, @code{accurate_conservative_nonmonotone_fv2fv}),
@acronym{TR} options: @samp{--in_type fv --in_np 2 --out_type fv}@*
@acronym{MOAB} options: @samp{--method fv --order 2 --method fv}
@end table
Thus these boutique options are specialized for @acronym{SE} grids with
fourth order resolution (@math{@var{np} = 4}).
Full documentation of the @acronym{E3SM}-recommended boutique options
for TempestRemap is
@uref{https://acme-climate.atlassian.net/wiki/spaces/Docs/pages/178848194/Transition+to+TempestRemap+for+Atmosphere+grids, here}
(may require @acronym{E3SM}-authorization to view).
Let us know if you would like other boutique TempestRemap option sets 
added as canonical options for @command{ncremap}.

@html
<a name="a2o"></a> <!-- http://nco.sf.net/nco.html#a2o -->
<a name="atm2ocn"></a> <!-- http://nco.sf.net/nco.html#atm2ocn -->
<a name="b2l"></a> <!-- http://nco.sf.net/nco.html#b2l -->
<a name="big2ltl"></a> <!-- http://nco.sf.net/nco.html#big2ltl -->
<a name="l2s"></a> <!-- http://nco.sf.net/nco.html#l2s -->
<a name="lrg2sml"></a> <!-- http://nco.sf.net/nco.html#lrg2sml -->
<a name="allow_no_overlap"></a> <!-- http://nco.sf.net/nco.html#allow_no_overlap -->
@end html
@cindex @command{GenerateOverlapMesh}
@cindex @var{a2o}
@cindex @code{--a2o}
@cindex @code{--atm2ocn}
@cindex @code{--b2l}
@cindex @code{--big2ltl}
@cindex @code{--l2s}
@cindex @code{--lrg2sml}
@cindex @code{--allow_no_overlap}
@item --a2o (@code{--a2o}, @code{--atm2ocn}, @code{--b2l}, @code{--big2ltl}, @code{--l2s}, @code{--lrg2sml})
Use one of these flags (that take no arguments) to cause TempestRemap to
generate mapping weights from a source grid that has more coverage than
the destination grid, i.e., the destination grid is a subset of the
source.
When computing the intersection of two meshes, TempestRemap uses an
algorithm (in an executable named @command{GenerateOverlapMesh}) that
expects the mesh with less coverage to be the first grid, and the
grid with greater coverage to be the second, regardless of the mapping
direction.
By default, @command{ncremap} supplies the source grid first and the
destination second, but this order causes @command{GenerateOverlapMesh}  
(which is agnostic about ordering for grids of equal coverage)
to fail when the source grid covers regions not in the destination grid.  
For example, a global atmosphere grid has more coverage than a global
ocean grid, so that remapping from atmosphere-to-ocean would require
invoking the @samp{--atm2ocn} switch:
@example
# Use --a2o to generate weights for "big" to "little" remaps:
ncremap --a2o -a se2fv_flx --src_grd=atm_se_grd.nc \
              --dst_grd=ocn_fv_grd.nc -m map.nc
# Otherwise, omit it:
ncremap       -a fv2se_flx --src_grd=ocn_fv_grd.nc \
              --dst_grd=atm_se_grd.nc -m map.nc
ncremap       -a se2fv_flx --src_grd=atm_se_grd.nc \
              --dst_grd=atm_fv_grd.nc -m map.nc
# Only necessary when generating, not applying, weights:
ncremap -m atm2ocn.nc in.nc out.nc
@end example
As shown in the second example above, remapping from global
ocean-to-atmosphere grids does not require (and should not invoke) this
switch. 
The third example shows that the switch is only needed when
@emph{generating} weights, not when applying them. 
The switch is never needed (and is ignored) when generating weights
with @acronym{ERWG} (which constructs the intersection mesh with a
different algorithm than TempestRemap).
Attempting to remap a larger source grid to a subset destination grid
without using @samp{--a2o} causes @command{GenerateOverlapMesh} to emit
an error (and a potential workaround) like this:
@example
....Nearest target face 130767
....ERROR: No overlapping face found
....This may be caused by mesh B being a subset of mesh A
....Try swapping order of mesh A and B, or override with \
    --allow_no_overlap
....EXCEPTION (../src/OverlapMesh.cpp, Line 1738) Exiting
@end example
The @samp{--a2o} switch and its synonyms are available in 
version 4.7.3 (March, 2018) and later.
As of @acronym{NCO} version 4.9.9 (May, 2021), @code{ncremap}
automatically transmits the flag @samp{--allow_no_overlap} to
@command{GenerateOverlapMesh} so that regional meshes that do
not completely overlap may be intersected.
This is thought to have no effect on global mappings.
Please let us know if these capabilities do not work for you.

@html
<a name="add_fill_value"></a> <!-- http://nco.sf.net/nco.html#add_fill_value -->
<a name="add_fll"></a> <!-- http://nco.sf.net/nco.html#add_fll -->
<a name="fill_empty"></a> <!-- http://nco.sf.net/nco.html#fill_empty -->
<a name="fll_mpt"></a> <!-- http://nco.sf.net/nco.html#fll_mpt -->
@end html
@cindex missing value
@cindex @code{_FillValue}
@cindex @code{--add_fll}
@cindex @code{--no_add_fll}
@cindex @code{--add_fill_value}
@cindex @code{--fll_mpt}
@cindex @code{--no_fll_mpt}
@cindex @code{--fill_empty}
@cindex @code{--no_fill_empty}
@cindex @code{--msk_apl}
@cindex @code{--mask_apply}
@item --add_fll (@code{--add_fll}, @code{--add_fill_value}, @code{--fll_mpt}, @code{--fill_empty})
Introduced in @acronym{NCO} version 5.0.0 (released June, 2021),
this switch (which takes no argument) causes the regridder to add a
@code{_FillValue} attribute to fields with empty destination cells.
The corresponding @code{--no_add_fll} switches, introduced in
@acronym{NCO} version 5.1.1 (released November, 2022),
do the opposite and prevent the regridder from adding a
@code{_FillValue} attribute to fields with empty destination cells. 

There are two varieties of empty destination cells:
First are those cells with no non-zero weights from the source grids.
If all source grid contributions to the a particular cell are zero,
then no field will ever be mapped into that cell.
For example, if an ocean model source grid only contains ocean
gridcells (e.g., like @acronym{MPAS}-Ocean), then all continental
interior gridcells in a destination grid will be empty.
The second type of empty gridcell can occur in conjunction with
sub-gridscale (@acronym{SGS}) fractions.
All destination gridcells with @acronym{SGS} fraction equal to zero
will always be empty.
For example, sea-ice models often employ time-varying @acronym{SGS}
fractions which are zero everywhere except where/when sea ice is
present.
These gridcells are disjoint from continental interior gridcells
whose locations can be determined mapping weights alone.

When a contiguous geophysical field (e.g., air temperature) without a
@code{_FillValue} is mapped to such a destination grid, the empty
destination values are normally set to zero (because no source grid
cells contribute).
However, zero is a valid value for many geophysical fields.
Use this switch to ensure that empty destination gridcells are always 
set to @code{_FillValue}.
The default @code{_FillValue} will be used in the output file for
input fields that lack an explicit @code{_FillValue}.
This flag has no effect on fields that have any input values equal
to an explicitly or implicitly defined @code{_FillValue}.
The flag does affect fields that have valid input values everywhere
on the source grid, yet for some reason (e.g., disjoint grids or
zero sub-gridscale fractions) there are unmapped destination
gridcells. 
@example
@verbatim
ncremap ...           # No renormalization/masking
ncremap --sgs_frc=sgs --add_fll ... # Mask cells missing 100% 
ncremap --rnr=0.1 ... # Mask cells missing > 10% 
ncremap --rnr=0.1 --sgs_frc=sgs ... # Mask missing > 10%
ncremap --rnr=0.1 --sgs_frc=sgs --add_fll ... # Mask missing > 90% or sgs < 10% 
ncremap -P mpas... # --add_fll implicit, mask where sgs=0.0
ncremap -P mpas... --no_add_fll # --add_fll explicitly turned-off, no masking
ncremap -P mpas... --rnr=0.1 # Mask missing > 90% or sgs < 10% 
ncremap -P elm...  # --add_fll not implicit, no masking
@end verbatim
@end example
Note that @code{--add_fll} is automatically triggered by
@code{--msk_apl} to ensure that masked fields regridded with
TempestRemap-generated map-files have @code{_FillValue}s consistent 
with map-files generated by @acronym{ESMF} and @acronym{NCO}. 

@html
<a name="area_dgn"></a> <!-- http://nco.sf.net/nco.html#area_dgn -->
<a name="dgn_area"></a> <!-- http://nco.sf.net/nco.html#dgn_area -->
<a name="area_diagnose"></a> <!-- http://nco.sf.net/nco.html#area_diagnose -->
<a name="diagnose_area"></a> <!-- http://nco.sf.net/nco.html#diagnose_area -->
@end html
@cindex diagnose area
@cindex @code{--area_dgn}
@cindex @code{--dgn_area}
@cindex @code{--area_diagnose}
@cindex @code{--diagnose_area}
@item --area_dgn (@code{--area_dgn}, @code{--area_diagnose}, @code{--dgn_area}, @code{--diagnose_area})
Introduced in @acronym{NCO} version 5.0.4 (released December, 2021),
this switch (which takes no argument) causes the regridder to diagnose 
(rather than copy) the area of each gridcell to an inferred grid-file.
By default, @command{ncremap} simply copies the area variable (whose
name defaults to @code{area} and can be explicitly specified with
@samp{-R '--rgr area_nm=name'}) into the @code{grid_area} variable of
the inferred grid-file.
When @code{--area_dgn} is invoked, @command{ncremap} instead computes
the values of @code{grid_area} based on the cell boundaries in the
input template file.
@example
@verbatim
ncremap --area_dgn -d dst.nc -g grid.nc
@end verbatim
@end example
Note that @code{--area_dgn} has no effect on any mapping weights
subsequently generated from the grid-file because most
weight-generators base their weights on internally computed cell
areas (although @acronym{ERWG} has an option, @code{--user_areas}, to
override this behavior).

@html
<a name="config"></a> <!-- http://nco.sf.net/nco.html#config -->
<a name="cnf"></a> <!-- http://nco.sf.net/nco.html#cnf -->
@end html
@cindex @code{--version}
@cindex @code{--vrs}
@cindex @code{--config}
@cindex @code{--configuration}
@cindex @code{--cnf}
@item --version (@code{--version}, @code{--vrs}, @code{--config}, @code{--configuration}, @code{--cnf})
This switch (which takes no argument) causes the operator to print
its version and configuration.
This includes the copyright notice, @acronym{URL}s to the @acronym{BSD}
and @acronym{NCO} license, directories from which the @acronym{NCO}
scripts and binaries are running, and the locations of any separate 
executables that may be used by the script.

@html
<a name="d2f"></a> <!-- http://nco.sf.net/nco.html#d2f -->
@end html
@cindex @code{d2f}
@cindex @code{--d2f}
@cindex @code{--d2s}
@cindex @code{--dbl_flt}
@cindex @code{--dbl_sgl}
@cindex @code{--double_float}
@cindex @code{ncpdq}
@cindex Precision
@item --d2f (@code{--d2f}, @code{--d2s}, @code{--dbl_flt}, @code{--dbl_sgl}, @code{--double_float})
This switch (which takes no argument) demotes all double precision
non-coordinate variables to single precision.
Internally @command{ncremap} invokes @command{ncpdq} to apply the
@code{dbl_flt} packing map to an intermediate version of the input
file before regridding it.
This switch has no effect on files that are not regridded.
To demote the precision in such files, use @command{ncpdq} to apply 
the @code{dbl_flt} packing map to the file directly.
Files without any double precision fields will be unaltered.

@html
<a name="dbg_lvl"></a> <!-- http://nco.sf.net/nco.html#dbg_lvl -->
@end html
@cindex @code{-D @var{dbg_lvl}}
@cindex @var{dbg_lvl}
@cindex @code{--dbg_lvl}
@cindex @code{--dbg}
@cindex @code{--debug}
@cindex @code{--debug_level}
@item -D @var{dbg_lvl} (@code{--dbg_lvl}, @code{--dbg}, @code{--debug}, @code{--debug_level})
Specifies a debugging level similar to the rest of @acronym{NCO}.
If @math{@var{dbg_lvl} = 1}, @command{ncremap} prints more extensive
diagnostics of its behavior.
If @math{@var{dbg_lvl} = 2}, @command{ncremap} prints the commands
it would execute at any higher or lower debugging level, but does
not execute these commands.
If @math{@var{dbg_lvl} > 2}, @command{ncremap} prints the diagnostic
information, executes all commands, and passes-through the debugging
level to the regridder (@command{ncks}) for additional diagnostics.

@html
<a name="devnull"></a> <!-- http://nco.sf.net/nco.html#devnull -->
@end html
@cindex @code{--devnull}
@cindex @code{--dev_nll}
@cindex @code{--dvn_flg}
@item @code{--devnull=@var{dvn_flg}} (@code{--devnull}, @code{--dev_nll}, @code{--dvn_flg})
The @var{dvn_flg} controls whether @command{ncremap} suppresses
regridder output or sends it to @code{/dev/null}.
The default value of @var{dvn_flg} is ``Yes'', so that
@command{ncremap} prints little output to the terminal.
Set @var{dvn_flg} to ``No'' to allow the internal regridder
executables (mainly @command{ncks}) to send their output to the
terminal.

@html
<a name="dpt"></a> <!-- http://nco.sf.net/nco.html#dpt -->
<a name="dpt_fl"></a> <!-- http://nco.sf.net/nco.html#dpt_fl -->
@end html
@cindex @code{dpt}
@cindex @code{--dpt}
@cindex @code{--add_dpt}
@cindex @code{--depth}
@cindex @code{--add_depth}
@cindex @code{--dpt_fl=@var{dpt_fl}}
@cindex @var{dpt_fl}
@cindex @code{--mpas_fl}
@cindex @code{--mpas_depth}
@cindex @code{--depth_file}
@cindex @command{add_depth.py}
@cindex Depth
@cindex @acronym{MPAS}
@item --dpt (@code{--dpt}, @code{--add_dpt}, @code{--depth}, @code{--add_depth})
@item --dpt_fl=@var{dpt_fl} (@code{--dpt_fl}, @code{--depth_file}, @code{--mpas_fl}, @code{--mpas_depth})
The @samp{--dpt} switch (which takes no argument) and the
@samp{--dpt_fl=@var{dpt_fl}} option which automatically sets the
switch and also takes a filename argument, both control the addition
of a depth coordinate to @acronym{MPAS} ocean datasets.
Depth is the vertical distance below sea surface and, like pressure
in the atmosphere, is an important vertical coordinate whose explicit
values are often omitted from datasets yet may be computed from other
variables (gridbox thickness, pressure difference) and grid information.
Moreover, users are often more interested in the approximate depth,
aka reference depth, of a given ocean layer independent of its
horizontal position.
To invoke either of these options first obtain and place the
@command{add_depth.py} command on the executable path (i.e.,
@code{$PATH}), and use @command{ncremap --config} to verify that it is
found.
These options tell @command{ncremap} to invoke @command{add_depth.py}
which uses the @code{refBottomDepth} variable in the current data file
or, if specified, the @var{dpt_fl}, to create and add a depth
coordinate to the current file (before regridding).

As of @acronym{NCO} version 4.7.9 (February, 2019), the depth
coordinate is an approximate, one-dimensional, globally uniform 
coordinate that neglects horizontal variations in depth that can occur
near strong bathymetry or under ice shelves.
Like its atmospheric counterpart in many models, the @code{lev}
pressure-coordinate, @code{depth} is useful for plotting purposes and
global studies. 
It would not be difficult to modify these options to add other depth
information based on the 3D cell-thickness field to ocean files
(please ask Charlie if interested in this).

@html
<a name="dst_fl"></a> <!-- http://nco.sf.net/nco.html#dst_fl -->
@end html
@cindex @code{-d @var{dst_fl}}
@cindex @var{dst_fl}
@cindex @code{--dst_fl}
@cindex @code{--destination_file}
@cindex @code{--tpl}
@cindex @code{--tpl_fl}
@cindex @code{--template}
@cindex @code{--template_file}
@item -d @var{dst_fl} (@code{--dst_fl}, @code{--destination_file}, @code{--tpl}, @code{tpl_fl}, @code{--template_file}, @code{--template})
Specifies a data file to serve as a template for inferring the
destination grid. 
Currently @var{dst_fl} must be a data file (not a gridfile,
@acronym{SCRIP} or otherwise) from which @acronym{NCO} can
infer the destination grid. 
The more coordinate and boundary information and metadata 
the better @acronym{NCO} will do at inferring the grid.
If @var{dst_fl} has cell boundaries then @acronym{NCO} will use those.
If @var{dst_fl} has only cell-center coordinates (and no edges),
then @acronym{NCO} will guess-at (for rectangular grids) or interpolate
(for curvilinear grids) the edges. 
Unstructured grids must supply cell boundary information, as it cannot
be interpolated or guessed-at.
@acronym{NCO} only reads coordinate and grid data and metadata from
@var{dst_fl}. 
@var{dst_fl} is not modified, and may have read-only permissions.

@html
<a name="dt_sng"></a> <!-- http://nco.sf.net/nco.html#dt_sng -->
@end html
@cindex @code{--dt_sng=@var{dt_sng}}
@cindex @var{dt_sng}
@cindex @code{--dt_sng}
@cindex @code{--date_string}
@item --dt_sng=@var{dt_sng} (@code{--dt_sng}, @code{--date_string})
Specifies the date-string use in the full name of map-files created in
@acronym{MWF} mode.  
Map-file names include, by convention, a string to indicate the
approximate date (and thus algorithm versions employed) of weight
generation. 
@command{ncremap} uses the @var{dt_sng} argument to encode the date
into output map-file names of this format:  
@code{map_@var{nm_src}_to_@var{nm_dst}_@var{alg_typ}.@var{dt_sng}.nc}.
@acronym{MWF} mode defaults @var{dt_sng} to the current date in
@code{YYYYMMDD}-format. 

@html
<a name="esmf_typ"></a> <!-- http://nco.sf.net/nco.html#esmf_typ -->
@end html
@cindex @code{--esmf_typ=@var{esmf_typ}}
@cindex @var{esmf_typ}
@cindex @code{--esmf_typ}
@cindex @code{--esmf_mth}
@cindex @code{--esmf_extrap_type}
@cindex @code{--esmf_extrap_method}
@cindex distance-weighted extrapolation
@item --esmf_typ=@var{esmf_typ} (@code{--esmf_typ}, @code{--esmf_mth}, @code{--esmf_extrap_type}, @code{--esmf_extrap_method})
Specifies the extrapolation method used to compute unmapped
destination point values with the @acronym{ERWG} weight generator.
Valid values, their synonyms, and their meanings are
@code{neareststod} (synonyms @code{stod} and @code{nsd}) which
uses the nearest valid source value,
@code{nearestidavg} (synonyms @code{idavg} and @code{id}) which
uses an inverse distance-weighted (with an exponent of @var{xtr_xpn})
average of the nearest @var{xtr_nsp} valid source values, and 
@code{none} (synonyms @code{nil} and @code{nowaydude}) which forbids 
extrapolation. 
Default is @math{@var{esmf_typ} = @code{none}}.
The arguments to options @samp{--xtr_xpn=@var{xtr_xpn}} (which
defaults to 2.0) and @samp{--xtr_nsp=@var{xtr_nsp}} (which defaults
to 8) set the parameters that control the extrapolation
@code{nearestidavg} algorithm.   
For more information on @acronym{ERWG} extrapolation, see documentation
@uref{http://www.earthsystemmodeling.org/esmf_releases/last_built/ESMF_refdoc/node3.html#SECTION03022300000000000000, here}.
@acronym{NCO} supports this feature as of version 4.7.4 (April, 2018). 

@html
<a name="xtr_nsp"></a> <!-- http://nco.sf.net/nco.html#xtr_nsp -->
@end html
@cindex extrapolation
@cindex @code{--xtr_nsp=@var{xtr_nsp}}
@cindex @var{xtr_nsp}
@cindex @code{--xtr_nsp}
@cindex @code{--esmf_pnt_src_nbr}
@cindex @code{--esmf_extrap_num_src_pnts}
@item --xtr_nsp=@var{xtr_nsp} (@code{--xtr_nsp}, @code{--esmf_pnt_src_nbr}, @code{--esmf_extrap_num_src_pnts})
Specifies the number of source points to use in extrapolating unmapped
destination point values with the @acronym{ERWG} weight generator.
This option is only useful in conjunction with explicitly requested
extrapolation types @math{@var{esmf_typ} = @code{neareststod}} and
@math{@var{esmf_typ} = @code{nearestidavg}}. 
Default is @math{@var{xtr_nsp} = 8}.
For more information on @acronym{ERWG} extrapolation, see documentation
@uref{http://www.earthsystemmodeling.org/esmf_releases/last_built/ESMF_refdoc/node3.html#SECTION03022300000000000000, here}.
@acronym{NCO} supports this feature as of version 4.7.4 (April, 2018). 

@html
<a name="xtr_xpn"></a> <!-- http://nco.sf.net/nco.html#xtr_xpn -->
@end html
@cindex @code{--xtr_xpn=@var{xtr_xpn}}
@cindex @var{xtr_xpn}
@cindex @code{--xtr_xpn}
@cindex @code{--esmf_pnt_src_nbr}
@cindex @code{--esmf_extrap_num_src_pnts}
@item --xtr_xpn=@var{xtr_xpn} (@code{--xtr_xpn}, @code{--esmf_pnt_src_nbr}, @code{--esmf_extrap_num_src_pnts})
Specifies the number of source points to use in extrapolating unmapped
destination point values with the @acronym{ERWG} weight generator.
This option is only useful in conjunction with explicitly requested
extrapolation types @math{@var{esmf_typ} = @code{neareststod}} and
@math{@var{esmf_typ} = @code{nearestidavg}}. 
Default is @math{@var{xtr_xpn} = 2.0}.
For more information on @acronym{ERWG} extrapolation, see documentation
@uref{http://www.earthsystemmodeling.org/esmf_releases/last_built/ESMF_refdoc/node3.html#SECTION03022300000000000000, here}.
@acronym{NCO} supports this feature as of version 4.7.4 (April, 2018). 

@html
<a name="grd_dst"></a> <!-- http://nco.sf.net/nco.html#grd_dst -->
@end html
@cindex @code{-g @var{grd_dst}}
@cindex @var{grd_dst}
@cindex @code{--grd_dst}
@cindex @code{--grid_dest}
@cindex @code{--dest_grid}
@cindex @code{--destination_grid}
@cindex infer
@item -g @var{grd_dst} (@code{--grd_dst}, @code{--grid_dest}, @code{--dest_grid}, @code{--destination_grid})
Specifies the destination gridfile.
An existing gridfile may be in any format accepted by the weight generator.
@acronym{NCO} will use @acronym{ERWG} or TempestRemap to combine
@var{grd_dst} with a source gridfile (either inferred from
@var{input-file}, supplied with @samp{-s @var{grd_src}}, or generated
from @samp{-G @var{grd_sng}}) to produce remapping weights. 
When @var{grd_dst} is used as input, it is not modified,
and may have read-only permissions.
When @var{grd_dst} is inferred from @var{input-file} or created
from @var{grd_sng}, it will be generated in @acronym{SCRIP} format. 

@html
<a name="fl_fmt_ncremap"></a> <!-- http://nco.sf.net/nco.html#fl_fmt_ncremap -->
@end html
@cindex @code{--fl_fmt_ncremap}
@cindex @code{--file_format_ncremap}
@cindex @code{-3}
@cindex @code{-4}
@cindex @code{-5}
@cindex @code{-6}
@cindex @code{-7}
As of @acronym{NCO} version 4.6.8 (August, 2017), @command{ncremap}
supports most of the file format options that the rest of @acronym{NCO}
has long supported (@pxref{File Formats and Conversion}).
This includes short flags (e.g., @samp{-4}) and key-value options (e.g., 
@samp{--fl_fmt=netcdf4}) though not long-flags without values
(e.g., @samp{--netcdf4}).
However, @command{ncremap} can only apply the full suite of file format 
options to files that it creates, i.e., regridded files.
The weight generators (@acronym{ERWG} and TempestRemap) are limited
in the file formats that they read and write.
Currently (August, 2017), @acronym{ERWG} supports @code{CLASSIC},
@code{64BIT_OFFSET}, and @code{NETCDF4}, while TempestRemap
supports only @code{CLASSIC}.
These can of course be converted to other formats using @command{ncks}
(@pxref{File Formats and Conversion}).
However, map-files @emph{produced} in other non-@code{CLASSIC} formats
can remap significantly larger grids than @code{CLASSIC}-format
map-files. 

@html
<a name="grd_sng"></a> <!-- http://nco.sf.net/nco.html#grd_sng -->
@end html
@cindex @code{-G @var{grd_sng}}
@cindex @var{--grd_sng}
@cindex @code{--grd_sng}
@cindex @code{--grid_generation}
@cindex @code{--grid_gen}
@cindex @code{--grid_string}
@item -G @var{grd_sng} (@code{--grd_sng}, @code{--grid_generation}, @code{--grid_gen}, @code{--grid_string})
Specifies, with together with other options, a source gridfile to 
create@footnote{ 
As of version 4.7.6 (August, 2018)), @acronym{NCO}'s syntax for
gridfile generation is much improved and streamlined, and is the
syntax described here.
This is also called ``Manual Grid-file Generation''.
An earlier syntax (described at @pxref{Grid Generation}) accessed
through @command{ncks} options still underlies the new syntax, though 
it is less user-friendly. 
Both old and new syntax work well and produce finer rectangular
grids than any other software we know of.}.
@command{ncremap} creates the gridfile in @acronym{SCRIP} format by
default, and then, should the requisite options for regridding
be present, combines that with the destination grid (either
inferred from @var{input-file} or supplied with
@samp{-g @var{grd_dst}} and generates mapping weights.
Manual grid-file generation is not frequently used since
@command{ncremap} can infer many grids directly from the
@var{input-file}, and few users wish to keep track of @acronym{SCRIP}
grids when they can be easily regenerated as intermediate files.
This option also allows one to visually tune a grid by rapidly
generating candidates and inspecting the results.

If a desired grid-file is unavailable, and no dataset on that grid is
available (so inferral cannot be used), then one must manually create
a new grid. 
Users create new grids for many reasons including dataset
intercomparisons, regional studies, and fine-tuned graphics.
@acronym{NCO} and @command{ncremap} support manual generation of the
most common rectangular grids as @acronym{SCRIP}-format grid-files.
Create a grid by supplying @command{ncremap} with a grid-file name and
``grid-formula'' (@var{grd_sng}) that contains, at a minimum, the
grid-resolution.
The grid-formula is a single hash-separated string of name-value pairs
each representing a grid parameter.
All parameters except grid resolution have reasonable defaults, so a
grid-formula can be as simple as @samp{latlon=180,360}: 
@example
@verbatim
ncremap -g grd.nc -G latlon=180,360
@end verbatim
@end example
The @acronym{SCRIP}-format grid-file @file{grd.nc} is a valid source
or destination grid for @command{ncremap} and other regridders. 

Grid-file generation documentation in the @acronym{NCO} Users Guide at
@url{http://nco.sf.net/nco.html#grid} describes all the grid
parameters and contains many examples.
Note that the examples in this section use grid generation
@acronym{API} for @command{ncremap} version 4.7.6 (August, 2018) and 
later.
Earlier versions can use the @command{ncks} @acronym{API} explained
at @ref{Grid Generation} in the Users Guide.  

The most useful grid parameters (besides resolution) are latitude type
(@var{lat_typ}), longitude type (@var{lon_typ}), title (@var{ttl}),
and, for regional grids, the @acronym{SNWE} bounding box
(@var{snwe}).
The three supported varieties of global rectangular grids are
Uniform/equiangular (@math{@var{lat_typ}=@code{uni}}), Cap/FV
(@math{@var{lat_typ}=@code{cap}}), and Gaussian 
(@math{@var{lat_typ}=@code{gss}}).
The four supported varieties of longitude types are the first
(westernmost) gridcell centered at Greenwich
(@math{@var{lon_typ}=@code{grn_ctr}}), western edge at Greenwish
(@code{grn_wst}), or at the Dateline
(@math{@var{lon_typ}=@code{180_ctr}} and
@math{@var{lon_typ}=@code{180_wst}}, respectively).
Grids are global, uniform, and have their first longitude centered at 
Greenwich by default.
The grid-formula for this is @samp{lat_typ=uni#lon_typ=grn_ctr}. 
Some examples (remember, this @acronym{API} requires @acronym{NCO}
4.7.6+): 
@example
@cindex @acronym{NCEP2} grid
@verbatim
ncremap -g grd.nc -G latlon=180,360                 # 1x1 Uniform grid
ncremap -g grd.nc -G latlon=180,360#lat_drc=n2s     # 1x1 Uniform grid, N->S not S->N
ncremap -g grd.nc -G latlon=180,360#lon_typ=grn_wst # 1x1 Uniform grid, Greenwich-west edge
ncremap -g grd.nc -G latlon=129,256#lat_typ=cap     # 1.4x1.4  FV grid
ncremap -g grd.nc -G latlon=94,192#lat_typ=gss      # T62 Gaussian grid
ncremap -g grd.nc -G latlon=361,576#lat_typ=cap#lon_typ=180_ctr # MERRA2 FV grid
ncremap -g grd.nc -G latlon=94,192#lat_typ=gss#lat_drc=n2s # NCEP2 T62 Gaussian grid 
@end verbatim
@end example
Regional grids are a powerful tool in regional process analyses, and
can be much smaller in size than global datasets.
Regional grids are always uniform. Specify the rectangular bounding
box, i.e., the outside edges of the region, in @acronym{SNWE} order:
@example
@verbatim
ncremap -g grd.nc -G ttl="Equi-Angular 1x1 Greenland grid"#latlon=30,90#snwe=55.0,85.0,-90.0,0.0
@end verbatim
@end example
The @var{grd_sng} argument to @samp{-G} or @samp{--grd_sng} must be 
a @emph{single} hash-separated string of name-value pairs, e.g.,
@code{latlon=....#lat_typ=...#ttl="My title"}.
@command{ncremap} will not correctly parse any other format, such
as multiple separate name-value pairs without hashes.

@html
<a name="in_drc"></a> <!-- http://nco.sf.net/nco.html#in_drc -->
@end html
@cindex @code{-I @var{in_drc}}
@cindex @var{in_drc}
@cindex @code{stdin}
@cindex @code{--in_drc}
@cindex @code{--drc_in}
@cindex @code{--dir_in}
@cindex @code{--in_dir}
@cindex @code{--input}
@item -I @var{in_drc} (@code{--in_drc}, @code{--drc_in}, @code{--dir_in}, @code{--in_dir}, @code{input})
Specifies the input directory, i.e., the directory which contains
the input file(s).
If @var{in_fl} is also specified, then the input filepath is
constructed by appending a slash and the filename to the directory:
@samp{@var{in_drc}/@var{in_fl}}.
Specifying @var{in_drc} without @var{in_fl} causes @command{ncremap}
to attempt to remap every file in @var{in_drc} that ends with one of
these suffixes: @code{.nc}, @code{.nc3}, @code{.nc4}, @code{.nc5},
@code{.nc6}, @code{.nc7}, @code{.cdf},
@code{.hdf}, @code{.he5}, or @code{.h5}.
When multiple files are regridded, each output file takes the name
of the corresponding input file.
There is no namespace conflict because the input and output files are in
separate directories. 
Note that @command{ncremap} can instead accept a list of input files
through standard input (e.g., @samp{ls *.nc | ncremap ...}) or as
positional command-line arguments (e.g., @samp{ncremap in1.nc in2.nc ...}).

@html
<a name="in_fl"></a> <!-- http://nco.sf.net/nco.html#in_fl -->
@end html
@cindex @code{-i @var{in_fl}}
@cindex @var{in_fl}
@cindex @code{--in_fl}
@cindex @code{--in_file}
@cindex @code{--input_file}
@item -i @var{in_fl} (@code{--in_fl}, @code{--in_file}, @code{--input_file})
Specifies the file containing data on the source grid to be remapped
to the destination grid.
When provided with the optional @var{map_fl}, @command{ncremap} 
only reads data from @var{in_fl} in order to regrid it.
Without the optional @var{map_fl} or @var{src_grd}, @command{ncremap}
will try to infer the source grid from @var{in_fl}, and so must read
coordinate and metatdata information from @var{in_fl}.  
In this case the more coordinate and boundary information and metadata,
the better @acronym{NCO} will do at inferring the source grid.
If @var{in_fl} has cell boundaries then @acronym{NCO} will use those. 
If @var{in_fl} has only cell-center coordinates (and no edges),
then @acronym{NCO} will guess (for rectangular grids) or interpolate
(for curvilinear grids) the edges.  
Unstructured grids must supply cell boundary information, as it cannot
be interpolated or guessed-at.
@var{in_fl} is not modified, and may have read-only permissions.
Note that @command{ncremap} can instead accept input file name(s)
through standard input (e.g., @samp{ls *.nc | ncremap ...}) or as
positional command-line arguments (e.g.,
@samp{ncremap in1.nc in2.nc ...}).
When one or three-or-more positional arguments are given, they are
all interpreted as input filename(s).  
Two positional arguments are interpreted as a single @var{input-file}
and its corresponding @var{output-file}.

@html
<a name="job_nbr"></a> <!-- http://nco.sf.net/nco.html#job_nbr -->
<a name="job_nbr_ncremap"></a> <!-- http://nco.sf.net/nco.html#job_nbr_ncremap -->
@end html
@cindex @code{-j @var{job_nbr}}
@cindex @var{job_nbr}
@cindex @code{--job_nbr}
@cindex @code{--job_number}
@cindex @code{--job_number}
@cindex @code{--jobs}
@item -j @var{job_nbr} (@code{--job_nbr}, @code{--job_number}, @code{--jobs})
Specifies the number of simultaneous regridding processes to spawn  
during parallel execution for both Background and @acronym{MPI} modes. 
In both parallel modes @command{ncremap} spawns processes in batches 
of @var{job_nbr} jobs, then waits for those processes to complete.
Once a batch finishes, @command{ncremap} spawns the next batch.
In Background mode, all jobs are spawned to the local node.
In @acronym{MPI} mode, all jobs are spawned in round-robin fashion
to all available nodes until @var{job_nbr} jobs are running.

If regridding consumes so much @acronym{RAM} (e.g., because
variables are large and/or the number of threads is large) that a
single node can perform only one regridding job at a time, then a
reasonable value for @var{job_nbr} is the number of nodes,
@var{node_nbr}. 
Often, however, nodes can regrid multiple files simultaneously. 
It can be more efficient to spawn multiple jobs per node than to
increase the threading per job because I/O contention for write
access to a single file prevents threading from scaling indefinitely. 

By default @math{@var{job_nbr} = 2} in Background mode, and
@math{@var{job_nbr} = @var{node_nbr}} in @acronym{MPI} mode.
This helps prevent users from overloading nodes with too many jobs.
Subject to the availability of adequate @acronym{RAM},
expand the number of jobs per node by increasing @var{job_nbr}
until, ideally, each core on the node is used. 
Remember that processes and threading are multiplicative in core use.
Four jobs each with four threads each consumes sixteen cores.

As an example, consider regridding @w{100 files} with a single map.
Say you have a five-node cluster, and each node has @w{16 cores}
and can simultaneously regrid two files using eight threads each.
(One needs to test a bit to optimize these parameters.)
Then an optimal (in terms of wallclock time) invocation would
request five nodes with @w{10 simultaneous} jobs of eight threads.
On @acronym{PBS} or @acronym{SLURM} batch systems this would involve a
scheduler command like @samp{qsub -l nodes=5 ...} or 
@samp{sbatch --nodes=5 ...}, respectively, followed by 
@samp{ncremap --par_typ=mpi --job_nbr=10 --thr_nbr=8 ...}.
This job will likely complete between five and ten-times faster than a  
serial-mode invocation of @command{ncremap} to regrid the same files.
The uncertainty range is due to unforeseeable, system-dependent
load and I/O charateristics.
Nodes that can simultaneously write to more than one file fare better
with multiple jobs per node.
Nodes with only one I/O channel to disk may be better exploited
by utilizing more threads per process.

@html
<a name="map_mlt"></a> <!-- http://nco.sf.net/nco.html#map_mlt -->
@end html
@cindex @code{-M}
@cindex @code{--mlt_map}
@cindex @code{--multimap}
@cindex @code{--no_multimap}
@cindex @code{--nomultimap}
@item -M (@code{--mlt_map}, @code{--multimap}, @code{--no_multimap}, @code{--nomultimap})
@command{ncremap} assumes that every input file is on a unique grid
unless a source gridfile is specified (with @samp{-s @var{grd_src}})
or multiple-mapfile generation is explicitly turned-off (with
@samp{-M}). 
The @samp{-M} switch is a toggle, it requires and accepts no argument. 
Toggling @samp{-M} tells @command{ncremap} to generate at most one
mapfile regardless of the number of input files. 
If @samp{-M} is not toggled (and neither 
@samp{-m @var{map_fl}} nor @samp{-s @var{grd_src}} is invoked)   
then @command{ncremap} will generate a new mapfile for each input file.
Generating new mapfiles for each input file is necessary for processing
batches of data on different grids (e.g., swath-like data), and slow,
tedious, and unnecessary when batch processing data on the same grids.

@html
<a name="map_fl"></a> <!-- http://nco.sf.net/nco.html#map_fl -->
@end html
@cindex @code{-m @var{map_fl}}
@cindex @var{map_fl}
@cindex @code{--map_fl}
@cindex @code{--map}
@cindex @code{--map_file}
@cindex @code{--rgr_map}
@cindex @code{--regrid_map}
@item -m @var{map_fl} (@code{--map_fl}, @code{--map}, @code{--map_file}, @code{--rgr_map}, @code{--regrid_map})
Specifies a mapfile (i.e., weight-file) to remap the source to
destination grid. 
If @var{map_fl} is specified in conjunction with any of the @samp{-d},
@samp{-G}, @samp{-g}, or @samp{-s} switches, then @command{ncremap}
will name the internally generated mapfile @var{map_fl}.
Otherwise (i.e., if none of the source-grid switches are used),
@command{ncremap} assumes that @var{map_fl} is a pre-computed mapfile. 
In that case, the @var{map_fl} must be in @acronym{SCRIP} format,
although it may have been produced by any application (usually
@acronym{ERWG} or TempestRemap). 
If @var{map_fl} has only cell-center coordinates (and no edges),
then @acronym{NCO} will guess-at or interpolate the edges.
If @var{map_fl} has cell boundaries then @acronym{NCO} will use those.
A pre-computed @var{map_fl} is not modified, and may have read-only
permissions. 
The user will be prompted to confirm if a newly generated map-file  
named @var{map_fl} would overwrite an existing file.
@command{ncremap} adds provenance information to any newly generated
map-file whose name was specified with @samp{-m @var{map_fl}}.
This provenance includes a @code{history} attribute that contains
the command invoking @command{ncremap}, and the map-generating command
invoked by @command{ncremap}.

@html
<a name="mpi_pfx"></a> <!-- http://nco.sf.net/nco.html#mpi_pfx -->
<a name="mpi_prefix"></a> <!-- http://nco.sf.net/nco.html#mpi_prefix -->
<a name="srun_cmd"></a> <!-- http://nco.sf.net/nco.html#srun_cmd -->
<a name="srun_command"></a> <!-- http://nco.sf.net/nco.html#srun_command -->
<a name="mpi_nbr"></a> <!-- http://nco.sf.net/nco.html#mpi_nbr -->
<a name="mpi_number"></a> <!-- http://nco.sf.net/nco.html#mpi_number -->
<a name="tsk_nbr"></a> <!-- http://nco.sf.net/nco.html#tsk_nbr -->
<a name="task_number"></a> <!-- http://nco.sf.net/nco.html#task_number -->
@end html
@cindex @code{--mpi_pfx=@var{mpi_pfx}}
@cindex @var{mpi_pfx}
@cindex @code{--mpi_pfx}
@cindex @code{--mpi_prefix}
@cindex @code{--srun_cmd}
@cindex @code{--srun_command}
@item @code{--mpi_pfx=@var{mpi_pfx}} (@code{--mpi_pfx}, @code{--mpi_prefix}, @code{--srun_cmd}, @code{--srun_command})
@cindex @code{--mpi_nbr=@var{mpi_nbr}}
@cindex @var{mpi_nbr}
@cindex @code{--mpi_nbr}
@cindex @code{--mpi_number}
@cindex @code{--tsk_nbr}
@cindex @code{--task_nbr}
@item @code{--mpi_nbr=@var{mpi_nbr}} (@code{--mpi_nbr}, @code{--mpi_number}, @code{--tsk_nbr}, @code{--task_number})
The @samp{--mpi_pfx=@var{mpi_pfx}} option specifies an appropriate job
scheduler prefix for @acronym{MPI}-enabled weight-generation
executables such as @acronym{ESMF}'s @command{ESMF_RegridWeightGen}
and @acronym{MOAB}'s @command{mbtempest}.
Other weight generators (@command{ncks}, @command{GenerateOfflineMap}) 
are unaffected by this option since they are not
@acronym{MPI}-enabled. 
@var{mpi_pfx} defaults to @code{mpirun -n $@{@var{mpi_nbr}@}} on all
machines except those whose @code{$HOSTNAME} matches an internal
database of @acronym{DOE}-operated supercomputers where 
@var{mpi_pfx} usually defaults to @code{srun -n $@{@var{mpi_nbr}@}}
When invoking @samp{--mpi_pfx}, be sure to explicitly define the
number of @acronym{MPI} tasks-per-node, e.g.,
@example
@verbatim
ncremap --mpi_pfx='srun -n 16' ...
ncremap --mpi_pfx='srun --mpi=pmi2 -n 4' ...
@end verbatim
@end example
The separate @samp{--mpi_nbr=@var{mpi_nbr}} option specifies the
number of tasks-per-node that @acronym{MPI}-enabled weight generators
will request.
It preserves the default job scheduler prefix (@command{srun} or
@command{mpirun}):
@example
@verbatim
ncremap --mpi_nbr=4 ... # 16 MPI tasks-per-node for ERWG/mbtempest 
ncremap --mpi_nbr=16 ... # 4 MPI tasks-per-node for ERWG/mbtempest 
@end verbatim
@end example
Thus @samp{--mpi_nbr=@var{mpi_nbr}} can be used to create
host-independent @command{ncremap} commands to facilitate benchmarking
the scaling of weight-generators across hosts that work with the
default value of @var{mpi_pfx}.
The @samp{--mpi_pfx} option will prevail and @samp{--mpi_nbr} will be
ignored if both are used in the same @command{ncremap} invocation.
Note that @samp{mpi_pfx} is only used internally by @command{ncremap}
to exploit the @acronym{MPI} capabilities of select weight-generators.
It is not used to control and does not affect the distribution of
multiple @command{ncremap} commands among a cluster of nodes.

@html
<a name="msh_fl"></a> <!-- http://nco.sf.net/nco.html#msh_fl -->
@end html
@cindex @code{--msh_fl=@var{msh_fl}}
@cindex @var{msh_fl}
@cindex @code{--msh_fl}
@cindex @code{--msh}
@cindex @code{--mesh}
@cindex @code{--mesh_file}
@item @code{--msh_fl=@var{msh_fl}} (@code{--msh_fl}, @code{--msh}, @code{--mesh}, @code{--mesh_file})
Specifies a meshfile (aka intersection mesh, aka overlap mesh) that
stores the grid formed by the intersection of the source and 
destination grids.
If not specified then @command{ncremap} will name any internally
generated meshfile with a temporary name and delete the file prior
to exiting.
@acronym{NCO} and TempestRemap support archiving the meshfile,
and @acronym{ERWG} does not.
@acronym{NCO} stores the meshfile in @acronym{SCRIP} format, while
TempestRemap stores it in Exodus format (with a @samp{.g} suffix).
@command{ncremap} adds provenance information to any newly generated
mesh-file whose name was specified with @samp{--msh_fl=@var{msh_fl}}.
This provenance includes a @code{history} attribute that contains
the command invoking @command{ncremap}, and the map-generating command
invoked by @command{ncremap}.

@html
<a name="msk_apl"></a> <!-- http://nco.sf.net/nco.html#msk_apl -->
<a name="mask_apply"></a> <!-- http://nco.sf.net/nco.html#mask_apply -->
<a name="msk_app"></a> <!-- http://nco.sf.net/nco.html#msk_app -->
<a name="fill_empty"></a> <!-- http://nco.sf.net/nco.html#fill_empty -->
@end html
@cindex missing value
@cindex @code{--add_fill_value}
@cindex @code{--add_fll}
@cindex @code{_FillValue}
@cindex @code{--msk_apl}
@cindex @code{--mask_apply}
@cindex @code{--msk_app}
@item --msk_apl (@code{--msk_apl}, @code{--mask_apply}, @code{--msk_app})
Introduced in @acronym{NCO} version 5.0.0 (released June, 2021),
this switch (which takes no argument) causes the regridder to
apply @var{msk_out} (i.e., @code{mask_b}) to variables after
regridding.
Some weight generators (e.g., TempestRemap) ignore masks and thus
produce non-zero weights for masked destination cells, and/or from
masked source cells.
This flag causes regridded files produced with such map-files to
adhere to the destination mask rules (though source mask rules may
still be violated).
This feature is especially useful in placing missing values (aka,
@code{_FillValue}) in destination cells that should be empty, so that 
regridded files have @code{_FillValue} distributions identical with
output from other weight-generators such as @acronym{ESMF} and
@acronym{NCO}. 
@example
@verbatim
ncremap --msk_apl           -v FLNS -m map.nc in.nc out.nc
ncremap --msk_apl --add_fll -v FLNS -m map.nc in.nc out.nc # Equivalent
@end verbatim
@end example
By itself, @code{--msk_apl} would only mask cells based on the
@code{mask_b} field in the map-file.
This is conceptually independent of the actual intersection mesh.
However, @code{--msk_apl} automatically triggers @code{--add_fll},
which also masks fields based on the computed intersection mesh
(i.e., @code{--frac_b}).
This combinations ensures that masked fields regridded with
TempestRemap-generated map-files have @code{_FillValue}s consistent
with map-files generated by @acronym{ESMF} and @acronym{NCO}. 

@html
<a name="msk_dst"></a> <!-- http://nco.sf.net/nco.html#msk_dst -->
@end html
@cindex @code{--msk_dst=@var{msk_dst}}
@cindex @var{msk_dst}
@cindex @code{--msk_dst}
@cindex @code{--dst_msk}
@cindex @code{--mask_destination}
@cindex @code{--mask_dst}
@item --msk_dst=@var{msk_dst} (@code{--msk_dst}, @code{--dst_msk}, @code{--mask_destination}, @code{--mask_dst})
Specifies a template variable to use for the integer mask of the
destination grid when inferring grid files and/or creating
map-files (i.e., generating weights). 
Any variable on the same horizontal grid as a data file can serve as a
mask template for that grid. 
The mask will be one (i.e., gridcells will participate in regridding) 
where @var{msk_dst} has valid, non-zero values in the data file from which
@acronym{NCO} infers the destination grid.   
The mask will be zero (i.e., gridcells will not participate in
regridding) where @var{msk_nm} has a missing value or is zero.  
A typical example of this option would be to use Sea-surface Temperature
(@acronym{SST}) as a template variable for an ocean mask because
@acronym{SST} is often defined only over ocean, and missing values
might denote locations to which regridded quantities should never be
placed. 
The special value @math{@var{msk_dst} = @code{none}} prevents the
regridder from inferring and treating any variable (even one named,
e.g., @code{mask}) in a source file as a mask variable.
This guarantees that all points in the inferred destination grid will be 
unmasked.  
@var{msk_dst}, @var{msk_out}, and @var{msk_src} are related yet distinct:
@var{msk_dst} is the mask template variable in the destination file (whose grid will be inferred),
@var{msk_out} is the name to give the destination mask in regridded data files, and 
@var{msk_src} is the mask template variable in the source file (whose grid will be inferred).
@var{msk_src} and @var{msk_dst} only affect inferred grid files for
the source and destination grids, respectively, whereas @var{msk_out}
only affects regridded files.

@html
<a name="msk_out"></a> <!-- http://nco.sf.net/nco.html#msk_out -->
@end html
@cindex @code{--msk_out=@var{msk_out}}
@cindex @var{msk_out}
@cindex @code{--msk_out}
@cindex @code{--out_msk}
@cindex @code{--mask_destination}
@cindex @code{--mask_out}
@item --msk_out=@var{msk_out} (@code{--msk_out}, @code{--out_msk}, @code{--mask_destination}, @code{--mask_out})
Use of this option tells @command{ncremap} to include a variable named 
@var{msk_out} in any regridded file.
The variable @var{msk_out} will contain the integer-valued
regridding mask on the destination grid.
The mask will be one (i.e., fields may have valid values in this
gridcell) or zero (i.e., fields will have missing values in this
gridcell). 
By default, @command{ncremap} does not output the destination mask to
the regridded file.
This option changes that default behavior and causes @command{ncremap}
to ingest the default destination mask variable contained in the
@var{map-file}. 
@acronym{ERWG} generates @acronym{SCRIP}-format map-files that contain
the destination mask in the variable named @code{mask_b}.
@acronym{SCRIP} generates map-files that contain the destination mask in
the variable named @code{dst_grid_imask}. 
The @code{msk_out} option works with map-files that adhere to either of
these conventions. 
Tempest generates map-files that do not typically contain the
destination mask, and so the @code{msk_out} option has no effect on
files that Tempest regrids.
@var{msk_dst}, @var{msk_out}, and @var{msk_src} are related yet distinct:
@var{msk_dst} is the mask template variable in the destination file (whose grid will be inferred),
@var{msk_out} is the name to give the destination mask in regridded data files, and 
@var{msk_src} is the mask template variable in the source file (whose grid will be inferred).
@var{msk_src} and @var{msk_dst} only affect inferred grid files for
the source and destination grids, respectively, whereas @var{msk_out}
only affects regridded files.

@html
<a name="msk_src"></a> <!-- http://nco.sf.net/nco.html#msk_src -->
@end html
@cindex @code{--msk_src=@var{msk_src}}
@cindex @var{msk_src}
@cindex @code{--msk_src}
@cindex @code{--src_msk}
@cindex @code{--mask_source}
@cindex @code{--mask_src}
@item --msk_src=@var{msk_src} (@code{--msk_src}, @code{--src_msk}, @code{--mask_source}, @code{--mask_src})
Specifies a template variable to use for the integer mask of the
source grid when inferring grid files and/or creating
map-files (i.e., generating weights). 
Any variable on the same horizontal grid as a data file can serve as a
mask template for that grid. 
The mask will be one (i.e., gridcells will participate in regridding) 
where @var{msk_src} has valid, non-zero values in the data file from which
@acronym{NCO} infers the source grid.   
The mask will be zero (i.e., gridcells will not participate in
regridding) where @var{msk_nm} has a missing value or is zero.  
A typical example of this option would be to use Sea-surface Temperature
(SST) as a template variable for an ocean mask because SST is often
defined only over ocean, and missing values might denote locations from
which regridded quantities should emanate.
The special value @math{@var{msk_src} = @code{none}} prevents the
regridder from inferring and treating any variable (even one named,
e.g., @code{mask}) in a source file as a mask variable.
This guarantees that all points in the inferred source grid will be 
unmasked.  
@var{msk_dst}, @var{msk_out}, and @var{msk_src} are related yet distinct:
@var{msk_dst} is the mask template variable in the destination file (whose grid will be inferred),
@var{msk_out} is the name to give the destination mask in regridded data files, and
@var{msk_src} is the mask template variable in the source file (whose grid will be inferred).
@var{msk_src} and @var{msk_dst} only affect inferred grid files for
the source and destination grids, respectively, whereas @var{msk_out}
only affects regridded files.

@html
<a name="--mss_val"></a> <!-- http://nco.sf.net/nco.html#--mss_val -->
<a name="mss_val_ncremap"></a> <!-- http://nco.sf.net/nco.html#mss_val_ncremap -->
@end html
@cindex @code{--mss_val=@var{mss_val}}
@cindex @var{mss_val}
@cindex @code{--mss_val}
@cindex @code{--fll_val}
@cindex @code{--missing_value}
@cindex @code{--fill_value}
@item --mss_val=@var{mss_val} (@code{--mss_val}, @code{--fll_val}, @code{--missing_value}, @code{--fill_value})
Specifies the numeric value that indicates missing data when processing
@acronym{MPAS} datasets, i.e., when @samp{-P mpas} is invoked.
The default missing value is @code{-9.99999979021476795361e+33} which is
correct for the @acronym{MPAS} ocean and sea-ice models.
Currently (January, 2018) the @acronym{MPAS} land-ice model uses
@code{-1.0e36} for missing values.
Hence this option is usually invoked as @samp{--mss_val=-1.0e36} to
facilitate processing of @acronym{MPAS} land-ice datasets.

@html
<a name="nco_opt"></a> <!-- http://nco.sf.net/nco.html#nco_opt -->
@end html
@cindex @code{-n @var{nco_opt}}
@cindex @var{nco_opt}
@cindex @code{--nco_opt}
@cindex @code{--nco_options}
@cindex @code{--nco}
@item -n @var{nco_opt} (@code{--nco_opt}, @code{--nco_options}, @code{--nco})
Specifies a string of options to pass-through unaltered to
@command{ncks}. 
@var{nco_opt} defaults to @samp{-O --no_tmp_fl}.

@html
<a name="nm_dst"></a> <!-- http://nco.sf.net/nco.html#nm_dst -->
@end html
@cindex @code{--nm_dst=@var{nm_dst}}
@cindex @var{nm_dst}
@cindex @code{--nm_dst}
@cindex @code{--name_dst}
@cindex @code{--nm_sht_dst}
@cindex @code{--name_short_destination}
@item --nm_dst=@var{nm_dst} (@code{--nm_dst}, @code{--name_dst}, @code{--name_short_destination}, @code{--nm_sht_dst})
Specifies the short name for the destination grid to use in the full
name of map-files created in @acronym{MWF} mode. 
Map-file names include, by convention, shortened versions of both the
source and destination grids.
@command{ncremap} uses the @var{nm_dst} argument to encode the
destination grid name into the output map-file name of this format: 
@code{map_@var{nm_src}_to_@var{nm_dst}_@var{alg_typ}.@var{dt_sng}.nc}.
@acronym{MWF} mode requires this argument, there is no default.

@html
<a name="nm_src"></a> <!-- http://nco.sf.net/nco.html#nm_src -->
@end html
@cindex @code{--nm_src=@var{nm_src}}
@cindex @var{nm_src}
@cindex @code{--nm_src}
@cindex @code{--name_src}
@cindex @code{--nm_sht_src}
@cindex @code{--name_short_source}
@item --nm_src=@var{nm_src} (@code{--nm_src}, @code{--name_src}, @code{--name_short_source}, @code{--nm_sht_src})
Specifies the short name for the source grid to use in the full
name of map-files created in @acronym{MWF} mode. 
Map-file names include, by convention, shortened versions of both the
source and destination grids.
@command{ncremap} uses the @var{nm_dst} argument to encode the
source grid name into the output map-file name of this format: 
@code{map_@var{nm_src}_to_@var{nm_dst}_@var{alg_typ}.@var{dt_sng}.nc}.
@acronym{MWF} mode requires this argument, there is no default.

@cindex @code{--no_cll_msr}
@cindex @code{--no_cll}
@cindex @code{--no_cell_measures}
@cindex @code{--no_area}
@cindex @code{cell_measures} attribute
@item --no_cll_msr (@code{--no_cll_msr}, @code{--no_cll}, @code{--no_cell_measures}, @code{--no_area})
This switch (which takes no argument) controls whether @command{ncclimo}
and @command{ncremap} add measures variables to the extraction list
along with the primary variable and other associated variables. 
See @ref{CF Conventions} for a detailed description.

@cindex @code{--no_frm_trm}
@cindex @code{--no_frm}
@cindex @code{--no_formula_terms}
@cindex @code{formula_terms} attribute
@item --no_frm_trm (@code{--no_frm_trm}, @code{--no_frm}, @code{--no_formula_terms})
This switch (which takes no argument) controls whether @command{ncclimo}
and @command{ncremap} add formula variables to the extraction list along
with the primary variable and other associated variables. 
See @ref{CF Conventions} for a detailed description.

@cindex @code{slat}
@cindex @code{slon}
@cindex @code{w_stag}
@cindex @code{--no_stg_grd}
@cindex @code{--no_stg}
@cindex @code{--no_stagger}
@cindex @code{--no_staggered_grid}
@item --no_stg_grd (@code{--no_stg_grd}, @code{--no_stg}, @code{--no_stagger}, @code{--no_staggered_grid})
This switch (which takes no argument) controls whether
regridded output will contain the staggered grid coordinates 
@code{slat}, @code{slon}, and @code{w_stag} (@pxref{Regridding}). 
Originally, the staggered grid was output for all files regridded from 
a Cap (aka @acronym{FV}) grid, except when the regridding was performed
as part of splitting (reshaping) into timeseries.
As of (roughly, I forget) @acronym{NCO} @w{version 4.9.4}, released in
July, 2020, outputging the staggered grid information is turned-off
for all workflows and must be proactively turned-on (with
@code{--stg_grd}).
Thus the @code{--no_stg_grd} switch is obsolete and is intened
only to preserve backward-compatibility of previous workflows.

@html
<a name="out_drc"></a> <!-- http://nco.sf.net/nco.html#out_drc -->
@end html
@cindex @code{-O @var{out_drc}}
@cindex @var{out_drc}
@cindex @code{--out_drc}
@cindex @code{--drc_out}
@cindex @code{--dir_out}
@cindex @code{--out_dir}
@cindex @code{--output}
@item -O @var{out_drc} (@code{--out_drc}, @code{--drc_out}, @code{--dir_out}, @code{--out_dir}, @code{--output})
Specifies the output directory, i.e., the directory name to contain 
the output file(s).
If @var{out_fl} is also specified, then the output filepath is
constructed by appending a slash and the filename to the directory:
@samp{@var{out_drc}/@var{out_fl}}.
Specifying @var{out_drc} without @var{out_fl} causes @command{ncremap}
to name each output file the same as the corresponding input file.
There is no namespace conflict because the input and output files will
be in separate directories.

@html
<a name="out_fl"></a> <!-- http://nco.sf.net/nco.html#out_fl -->
@end html
@cindex @code{-o @var{out_fl}}
@cindex @var{--out_fl}
@cindex @code{--out_fl}
@cindex @code{--output_file}
@cindex @code{--out_file}
@item -o @var{out_fl} (@code{--out_fl}, @code{--output_file}, @code{--out_file})
Specifies the output filename, i.e., the name of the file to contain
the data from @var{in_fl} remapped to the destination grid.
If @var{out_fl} already exists it will be overwritten.
Specifying @var{out_fl} when there are multiple input files (i.e., from
using @samp{-I @var{in_drc}} or standard input) generates an error 
(output files will be named the same as input files).
Two positional arguments are interpreted as a single @var{input-file}
and its corresponding @var{output-file}.

@html
<a name="mpas"></a> <!-- http://nco.sf.net/nco.html#mpas -->
<a name="prc_typ"></a> <!-- http://nco.sf.net/nco.html#prc_typ -->
<a name="rrg"></a> <!-- http://nco.sf.net/nco.html#rrg -->
<a name="eamxx"></a> <!-- http://nco.sf.net/nco.html#eamxx -->
<a name="scream"></a> <!-- http://nco.sf.net/nco.html#scream -->
@end html
@cindex @code{-P @var{prc_typ}}
@cindex @var{prc_typ}
@cindex @code{--prc_typ}
@cindex @code{--prm_typ}
@cindex @code{--procedure}
@item -P @var{prc_typ} (@code{--prc_typ}, @code{--prm_typ}, @code{--procedure})
Specifies the permutation mode desired.
As of @acronym{NCO} version 4.5.5 (February, 2016), one can tell
@command{ncremap} to invoke special processing procedures for different
types of input data. 
For instance, to automatically permute the dimensions in the data file
prior to regridding for a limited (though growing) number of data-file
types that encounter the @command{ncremap} limitation concerning
dimension ordering. 
Valid procedure types include @samp{airs} for @acronym{NASA AIRS} satellite data, 
@samp{eam} or @samp{cam} for @acronym{DOE EAM} and @acronym{NCAR CAM} model data,
@samp{eamxx} for @acronym{DOE EAMxx} (aka, @acronym{SCREAM}) model data,
@samp{elm} or @samp{clm} for @acronym{DOE ELM} and @acronym{NCAR CLM} model data,
@samp{cice} for @acronym{CICE} ice model data (must be on 2D grids),
@samp{cism} for @acronym{NCAR CISM} land ice model data,
@samp{mpascice}, @samp{mpasseaice}, or @samp{mpassi} for @acronym{MPAS} sea-ice model data,
@samp{mpaso} or @samp{mpasocean} for @acronym{MPAS} ocean model data,
@samp{mod04} for @w{Level 2} @acronym{MODIS} @acronym{MOD04} product,
@samp{mwf} for making all weight-files for a pair of grids, 
@samp{sgs} for datasets containing sub-gridscale (@acronym{SGS}) data
(such as @acronym{CLM/CTSM/ELM} land model data and
@acronym{CICE/MPAS-Seaice} sea-ice model data), 
and @samp{nil} (for none).
The default @var{prc_typ} is @samp{nil}, which means @command{ncremap} 
does not perform any special procedures prior to regridding.
The @acronym{AIRS} procedure calls @command{ncpdq} to permute dimensions
from their order in the input file to this order: 
@code{StdPressureLev,GeoTrack,GeoXTrack}.
The @acronym{ELM}, @acronym{CLM}, and @acronym{CICE} procedures set 
idiosyncratic model values and then invoke the Sub-gridscale
(@acronym{SGS}) procedure (see below). 
The @acronym{MOD04} procedure unpacks input data.
The @acronym{EAMxx} procedures permute input data dimensions into this
order prior to horizontal regridding:
@code{ilev,lev,dim2,ncol}, and cause the vertical interpolation
routine to look for surface pressure under the name @code{ps} instead
of @code{PS}.
The @acronym{MPAS} procedures permute input data dimensions into this
order: 
@code{Time,depth,nVertInterfaces,nVertLevels,nVertLevelsP1,nZBGCTracers,nBioLayersP1,nAlgaeIceLayers,nDisIronIceLayers,nIceLayers,maxEdges,MaxEdges2,nCategories,R3,ONE,TWO,FOUR,nEdges,nCells},
and invokes renormalization.
An @acronym{MPAS} dataset that contains any other dimensions will fail
to regrid until/unless those dimensions are added to the
@command{ncremap} dimension permutation option.
@*
@*
@b{@acronym{MWF}-mode}:@*
@cindex Make-Weight-Files (@acronym{MWF}) 
@cindex @acronym{MWF}-mode
As mentioned above in other options, @command{ncremap} includes an
@acronym{MWF}-mode (for ``Make All Weight Files'') that generates and
names, with one command and in a self-consistent manner, all
combinations of (for instance, @acronym{E3SM} or @acronym{CESM})
global atmosphere<->ocean maps with both @acronym{ERWG} and Tempest.
@acronym{MWF}-mode automates the laborious and error-prone process of
generating numerous map-files with various switches.
Its chief use occurs when developing and testing new global grid-pairs
for the @acronym{E3SM} atmosphere and ocean components.
Invoke @acronym{MWF}-mode with a number of specialized options to
control the naming of the output map-files: 
@example
@verbatim
ncremap -P mwf -s grd_ocn -g grd_atm --nm_src=ocn_nm \
        --nm_dst=atm_nm --dt_sng=date
@end verbatim
@end example
where @var{grd_ocn} is the "global" ocean grid, @var{grd_atm}, is the
global atmosphere grid, @var{nm_src} sets the shortened name for the
source (ocean) grid as it will appear in the output map-files,
@var{nm_dst} sets, similarly, the shortend named for the destination
(atmosphere) grid, and @var{dt_sng} sets the date-stamp in the output
map-file name
@file{map_$@{@var{nm_src}@}_to_$@{@var{nm_dst}@}_$@{@var{alg_typ}@}.$@{@var{dt_sng}@}.nc}.
Setting @var{nm_src}, @var{nm_dst}, and @var{dt_sng}, is optional
though highly recommended.
For example,  
@example
@verbatim
ncremap -P mwf -s ocean.RRS.30-10km_scrip_150722.nc \
  -g t62_SCRIP.20150901.nc --nm_src=oRRS30to10 --nm_dst=T62 \
  --dt_sng=20180901 
@end verbatim
@end example
produces the 10 @acronym{ERWG} map-files:
@enumerate
@item @file{map_oRRS30to10_to_T62_aave.20180901.nc}
@item @file{map_oRRS30to10_to_T62_blin.20180901.nc}
@item @file{map_oRRS30to10_to_T62_ndtos.20180901.nc}
@item @file{map_oRRS30to10_to_T62_nstod.20180901.nc}
@item @file{map_oRRS30to10_to_T62_patc.20180901.nc}
@item @file{map_T62_to_oRRS30to10_aave.20180901.nc}
@item @file{map_T62_to_oRRS30to10_blin.20180901.nc}
@item @file{map_T62_to_oRRS30to10_ndtos.20180901.nc}
@item @file{map_T62_to_oRRS30to10_nstod.20180901.nc}
@item @file{map_T62_to_oRRS30to10_patc.20180901.nc}
@end enumerate
The ordering of source and destination grids is immaterial for
@acronym{ERWG} maps since @acronym{MWF}-mode produces all map
combinations.
However, as described above in the TempestRemap section, the Tempest
overlap-mesh generator must be called with the smaller grid preceding
the larger grid.
For this reason, always invoke @acronym{MWF}-mode with the smaller
grid (i.e., the ocean) as the source, otherwise some Tempest map-file
will fail to generate.
The six optimized @acronym{SE}<->@acronym{FV} Tempest maps described
above in the TempestRemap section will be generated when the
destination grid has a @samp{.g} suffix which @command{ncremap}
interprets as indicating an Exodus-format @acronym{SE} grid (NB: this
assumption is an implementation convenience that can be modified if
necessary).
For example,  
@example
@verbatim
ncremap -P mwf -s ocean.RRS.30-10km_scrip_150722.nc -g ne30.g \
        --nm_src=oRRS30to10 --nm_dst=ne30np4 --dt_sng=20180901
@end verbatim
@end example
produces the @w{6 TempestRemap} map-files:
@enumerate
@item @file{map_oRRS30to10_to_ne30np4_monotr.20180901.nc}
@item @file{map_oRRS30to10_to_ne30np4_highorder.20180901.nc}
@item @file{map_oRRS30to10_to_ne30np4_mono.20180901.nc}
@item @file{map_ne30np4_to_oRRS30to10_mono.20180901.nc}
@item @file{map_ne30np4_to_oRRS30to10_highorder.20180901.nc}
@item @file{map_ne30np4_to_oRRS30to10_intbilin.20180901.nc}
@end enumerate
@acronym{MWF}-mode takes significant time to complete (~20 minutes on
my MacBookPro) for the above grids.
To accelerate this, consider installing the @acronym{MPI}-enabled
instead of the serial version of @acronym{ERWG}.
Then use the @samp{--wgt_cmd} option to tell @command{ncremap} the
@acronym{MPI} configuration to invoke @acronym{ERWG} with, for
example:  
@example
@verbatim
ncremap -P mwf --wgt_cmd='mpirun -np 12 ESMF_RegridWeightGen' \
  -s ocean.RRS.30-10km_scrip_150722.nc -g t62_SCRIP.20150901.nc \
  --nm_src=oRRS30to10 --nm_dst=T62 --dt_sng=20180901
@end verbatim
@end example
Background and distributed node parallelism (as described above in the
the Parallelism section) of @acronym{MWF}-mode are possible though not
yet implemented.
Please let us know if this feature is desired. 
@*
@*
@b{@acronym{RRG}-mode}:@*
@cindex Regridding ReGional Data (@acronym{RRG}) 
@cindex @acronym{RRG}-mode
@cindex @code{rrg_bb_wesn}
@cindex @code{rrg_dat_glb}
@cindex @code{rrg_grd_glb}
@cindex @code{rrg_grd_rgn}
@cindex @code{rrg_rnm_sng}
@cindex @var{bb_wesn}
@cindex @var{dat_glb}
@cindex @var{grd_glb}
@cindex @var{grd_rgn}
@cindex @var{rnm_sng}
@cindex @code{--rrg_bb_wesn=@var{bb_wesn}}
@cindex @code{--rrg_dat_glb=@var{dat_glb}}
@cindex @code{--rrg_grd_glb=@var{grd_glb}}
@cindex @code{--rrg_grd_rgn=@var{grd_rgn}}
@cindex @code{--rrg_rnm_sng=@var{rnm_sng}}
@acronym{EAM} and @acronym{CAM-SE} will produce regional output if
requested to with the @code{finclNlonlat} namelist parameter.
Output for a single region can be higher temporal resolution than the
host global simulation.
This facilitates detailed yet economical regional process studies.
Regional output files are in a special format that we call
@acronym{RRG} (for ``regional regridding'').
An @acronym{RRG} file may contain any number of rectangular regions.
However, @command{ncremap} can process only one region per invocation
(change the argument to the @samp{--rnm_sng} option, described below,
in each invocation).
The coordinates and variables for one region do not interfere with
other (possibly overlapping) regions because all variables and
dimensions are named with a per-region suffix string, e.g.,
@code{lat_128e_to_134e_9s_to_16s}.
@command{ncremap} can easily regrid @acronym{RRG} output from an
atmospheric @acronym{FV}-dycore because @command{ncremap} can infer
(as discussed above) the regional grid from any rectangular
@acronym{FV} data file.
Regridding regional @acronym{SE} data, however, is more complex
because @acronym{SE} gridcells are essentially weights without
vertices and @acronym{SE} weight-generators are not yet flexible
enough to output regional weights.
To summarize, regridding @acronym{RRG} data leads to three
@acronym{SE}-specific difficulties (#1--3 below) and two difficulties 
(#4--5) shared with @acronym{FV} @acronym{RRG} files: 

@enumerate
@item @acronym{RRG} files contain only regional gridcell center
locations, not weights 
@item Global @acronym{SE} grids have well-defined weights not vertices
for each gridpoint 
@item Grid generation software (@acronym{ESMF} and TempestRemap) only create
global not regional @acronym{SE} grid files 
@item Non-standard variable names and dimension names
@item Regional files can contain multiple regions
@end enumerate

@command{ncremap}'s @acronym{RRG} mode resolves these issues to allow
trouble-free regridding of @acronym{SE} @acronym{RRG} files. The user
must provide two additional input arguments,
@samp{--dat_glb=@var{dat_glb}} (or synonynms @samp{--rrg_dat_glb},
@samp{--data_global}, or @samp{--global_data}) and
@samp{--grd_glb=@var{grd_glb}} (or synonyms @samp{--rrg_grd_glb},
@samp{--grid_global}, or @samp{global_grid}) that point to a global
@acronym{SE} dataset and grid, respectively, of the same resolution as
the model that generated the @acronym{RRG} datasets.
Hence a typical @acronym{RRG} regridding invocation is: 
@example
@verbatim
ncremap --dat_glb=dat_glb.nc --grd_glb=grd_glb.nc -g grd_rgn.nc \
        dat_rgn.nc dat_rgr.nc
@end verbatim
@end example
Here @file{grd_rgn.nc} is a regional destination grid-file,
@file{dat_rgn.nc} is the @acronym{RRG} file to regrid, and
@file{dat_rgr.nc} is the regridded output.
Typically @file{grd_rgn.nc} is a uniform rectangular grid covering the
same region as the @acronym{RRG} file.
Generate this as described in the last example in the section that
describes Manual Grid-file Generation with the @samp{-G} option.
@file{grd_glb.nc} is the standard dual-grid grid-file for the
@acronym{SE} resolution, e.g.,
@file{ne30np4_pentagons.091226.nc}.
@command{ncremap} regrids the
global data file @file{dat_glb.nc} to the global dual-grid in order to 
produce a intermediate global file annotated with gridcell
vertices.
Then it hyperslabs the lat/lon coordinates (and vertices) from the
regional domain to use with regridding the @acronym{RRG} file.
A @file{grd_glb.nc} file with only one 2D field suffices (and is
fastest) for producing the information needed by the @acronym{RRG} 
procedure.
One can prepare an optimal @file{dat_glb.nc} file by subsetting any 2D
variable from any full global @acronym{SE} output dataset with, 
e.g., @samp{ncks -v FSNT in.nc dat_glb.nc}. 

@command{ncremap} @acronym{RRG} mode supports two additional options
to override internal parameters.
First, the per-region suffix string
may be set with @samp{--rnm_sng=rnm_sng} (or synonyms
@samp{--rrg_rnm_sng} or @samp{--rename_string}).
@acronym{RRG} mode will, by default, regrid the first region it finds
in an @acronym{RRG} file.
Explicitly set the desired region with @var{rnm_sng} for files with
multiple regions, e.g., @samp{--rnm_sng=_128e_to_134e_9s_to_16s}.
Second, the regional bounding-box may be explicitly set with
@samp{--bb_wesn=lon_wst,lon_est,lat_sth,lat_nrt}.
The normal parsing of the bounding-box string from the suffix string
may fail in (as yet undiscovered) corner cases, and the
@samp{--bb_wesn} option provides a workaround should that occur.
The bounding-box string must include the entire @acronym{RRG} region
(not a subset thereof), specified in WESN order.
The two override options may be used independently or together, as in:
@example
@verbatim
ncremap --rnm_sng='_128e_to_134e_9s_to_16s' --bb_wesn='128,134,-16,-9' \
        --dat_glb=dat_glb.nc --grd_glb=grd_glb.nc -g grd_rgn.nc \
        dat_rgn.nc dat_rgr.nc
@end verbatim
@end example

@acronym{RRG}-mode supports most normal @command{ncremap} options,
including input and output methods and regridding algorithms. 
@*
@html
<a name="alm"></a> <!-- http://nco.sf.net/nco.html#alm -->
<a name="cice"></a> <!-- http://nco.sf.net/nco.html#cice -->
<a name="clm"></a> <!-- http://nco.sf.net/nco.html#clm -->
<a name="ctsm"></a> <!-- http://nco.sf.net/nco.html#ctsm -->
<a name="elm"></a> <!-- http://nco.sf.net/nco.html#elm -->
<a name="mpasseaice"></a> <!-- http://nco.sf.net/nco.html#mpasseaice -->
<a name="sgs"></a> <!-- http://nco.sf.net/nco.html#sgs -->
<a name="sgs_frc"></a> <!-- http://nco.sf.net/nco.html#sgs_frc -->
<a name="sgs_msk"></a> <!-- http://nco.sf.net/nco.html#sgs_msk -->
<a name="sub-grid"></a> <!-- http://nco.sf.net/nco.html#sub-grid -->
<a name="sub-gridscale"></a> <!-- http://nco.sf.net/nco.html#sub-gridscale -->
<a name="subgrid"></a> <!-- http://nco.sf.net/nco.html#subgrid -->
@end html
@b{@acronym{SGS}-mode}:@*
@cindex Sub-gridscale (@acronym{SGS}) data
@cindex @acronym{SGS}-mode
@cindex @var{sgs_frc}
@cindex @var{sgs_msk}
@cindex @var{sgs_nrm}
@cindex @code{--sgs_frc=@var{sgs_frc}}
@cindex @code{--sgs_msk=@var{sgs_msk}}
@cindex @code{--sgs_nrm=@var{sgs_nrm}}
@command{ncremap} has a sub-gridscale (@acronym{SGS}) mode that
performs the special pre-processing and weighting necessary to
to conserve fields that represent fractional spatial portions of a
gridcell, and/or fractional temporal periods of the analysis.
Spatial fields output by most geophysical models are intensive, 
and so by default the regridder attempts to conserve the integral of the
area times the field value such that the integral is equal on source and
destination grids. 
However some models (like @acronym{ELM}, @acronym{CLM},
@acronym{CICE}, and @acronym{MPAS-Seaice}) output gridcell values
intended to apply to only a fraction @var{sgs_frc} (for
``sub-gridscale fraction'') of the gridcell.  
The sub-gridscale (@acronym{SGS}) fraction usually changes spatially
with the distribution of land and ocean, and spatiotemporally with
the distribution of sea ice and possibly vegetation.
For concreteness consider a sub-grid field that represents the land
fraction.
Land fraction is less than one in gridcells that resolve coastlines or 
islands.  
@acronym{ELM} and @acronym{CLM} happily output temperature values
valid only for a small (i.e., @math{@var{sgs_frc} << 1}) island within
the larger gridcell.
Model architecture dictates this behavior and savvy researchers expect
it.
The goal of the @acronym{NCO} weight-application algorithm is to treat
@acronym{SGS} fields as seamlessly as possible so that those less
familiar with sub-gridscale models can easily regrid them correctly.

Fortunately, models like @acronym{ELM} and @acronym{CLM} that run on
the same horizontal grid as the overlying atmosphere can use the same
mapping-file as the atmosphere, so long as the @acronym{SGS}
weight-application procedure is invoked.
Not invoking an @acronym{SGS}-aware weight application algorithm is
equivalent to assuming @math{@var{sgs_frc} = 1} everywhere.
Regridding sub-grid values correctly versus incorrectly (e.g., with
and without @acronym{SGS}-mode) alters global-mean answers for
land-based quantities by @w{about 1%} for horizontal grid resolutions
of about one degree. 
The resulting biases are in intricately shaped regions (coasts, lakes,
sea-ice floes) and so are easy to overlook.

To invoke @acronym{SGS} mode and correctly regrid sub-gridscale data, 
specify the names of the fractional area @var{sgs_frc} and, if
applicable, the mask variable @var{sgs_msk} (strictly, this is only
necessary if these names differ from their 
respective defaults @code{landfrac} and @code{landmask}).  
Trouble will ensue if @var{sgs_frc} is a percentage or an absolute
area rather than a fractional area (between zero and one).
@command{ncremap} must know the normalization factor @var{sgs_nrm} by
which @var{sgs_frc} must be @emph{divided} (not multiplied) to obtain
a true, normalized fraction.
Datasets (such as those from @acronym{CICE}) that store @var{sgs_frc} 
in percent should specify the option
@samp{--sgs_nrm=100} to instruct @command{ncremap} to normalize the
sub-grid area appropriately before regridding.
@command{ncremap} will re-derive @var{sgs_msk} based on the regridded 
values of @var{sgs_frc}: @math{@var{sgs_msk} = 1} is assigned to
destination gridcells with @math{@var{sgs_frc} > 0.0}, and all others
@math{@var{sgs_msk} = 0}.
As of @acronym{NCO} version 4.6.8 (released June, 2017), invoking any
of the options @samp{--sgs_frc}, @samp{--sgs_msk}, or @samp{--sgs_nrm},
automatically triggers @acronym{SGS}-mode, so that also invoking 
@samp{-P sgs} is redundant though legal.
As of @acronym{NCO} version 4.9.0 (released December, 2019), the values 
of the @var{sgs_frc} and @var{sgs_msk} variables should be explicitly
specified.
In previous versions they defaulted to @code{landfrac} and
@code{landmask}, respectively, when @samp{-P sgs} was selected.
This behavior still exists but will likely be deprecated in a future
version. 

The @code{area} and @var{sgs_frc} fields in the regridded file will be
in units of sterradians and fraction, respectively.
However, @command{ncremap} offers custom options to reproduce the
idiosyncratic data and metadata format of two particular models,
@acronym{ELM} and @acronym{CICE}.
When invoked with @samp{-P elm} (or @samp{-P clm}), a final step
converts the output @code{area} from sterradians to square kilometers. 
When invoked with @samp{-P cice}, the final step converts the output
@code{area} from sterradians to square meters, and the output
@code{sgs_frc} from a fraction to a percent.
@example
# ELM/CLM: output "area" in [sr]
ncremap --sgs_frc=landfrac --sgs_msk=landmask in.nc out.nc
ncremap -P sgs in.nc out.nc
# ELM/CLM pedantic format: output "area" in [km2]
ncremap -P elm in.nc out.nc # Same as -P clm, alm, ctsm

# CICE: output "area" in [sr]
ncremap --sgs_frc=aice --sgs_msk=tmask --sgs_nrm=100 in.nc out.nc
# CICE pedantic format: output "area" in [m2], "aice" in [%]
ncremap -P cice in.nc out.nc

# MPAS-Seaice: both commands are equivalent
ncremap -P mpasseaice in.nc out.nc
ncremap --sgs_frc=timeMonthly_avg_iceAreaCell in.nc out.nc
@end example
It is sometimes convenient to store the @var{sgs_frc} field in
an external file from the field(s) to be regridded.
For example, @acronym{CMIP}-style timeseries are often written
with only one variable per file.
@acronym{NCO} supports this organization by accepting @var{sgs_frc}
arguments in the form of a filename followed by a slash and then a
variable name: 
@example
ncremap --sgs_frc=sgs_landfrac_ne30.nc/landfrac -m map.nc in.nc out.nc
@end example

Files regridded using explicitly specified @acronym{SGS} options will
differ slightly from those regridded using the @samp{-P elm} or 
@samp{-P cice} options. 
The former will have an @code{area} field in sterradians, the generic
units used internally by the regridder. 
The latter produces model-specific @code{area} fields in square 
kilometers (for @acronym{ELM}) or square meters (for @acronym{CICE}),
as expected in the raw output from these two models.
To convert from angular to areal values, @acronym{NCO} assumes a
spherical Earth with radius @w{6,371,220 m} or @w{6,371,229 m},
for @acronym{ELM} and @acronym{CICE}, respectively.
The ouput @var{sgs_frc} field is expressed as a decimal fraction in all
cases except for @samp{-P cice} which stores the fraction in percent.
Thus the generic @acronym{SGS} and model-specific convenience options
produce equivalent results, and the latter is intended to be
indistinguishable (in terms of metadata and units) to raw model
output.  
This makes it more interoperable with many existing analysis scripts. 

@html
<a name="par_typ"></a> <!-- http://nco.sf.net/nco.html#par_typ -->
<a name="par_typ_ncremap"></a> <!-- http://nco.sf.net/nco.html#par_typ_ncremap -->
@end html
@cindex @code{-p @var{par_typ}}
@cindex @var{par_typ}
@cindex @code{--par_typ}
@cindex @code{--par_md}
@cindex @code{--parallel_type}
@cindex @code{--parallel_mode}
@cindex @code{--parallel}
@item -p @var{par_typ} (@code{--par_typ}, @code{--par_md}, @code{--parallel_type}, @code{--parallel_mode}, @code{--parallel})
Specifies the desired file-level parallelism mode, either Background,
@acronym{MPI}, or Serial. 
File-level parallelism accelerates throughput when regridding multiple
files in one @command{ncremap} invocation, and has no effect when only
one file is to be regridded.
Note that the @command{ncclimo} and @command{ncremap} semantics for
selecting file-level parallelism are identical, though their defaults
differ (Background mode for @command{ncclimo} and Serial mode for
@command{ncremap}). 
Select the desired mode with the argument to
@samp{--par_typ=@var{par_typ}}.
Explicitly select Background mode with @var{par_typ} values of
@code{bck}, @code{background}, or @code{Background}.
The values @code{mpi} or @code{MPI} select @acronym{MPI} mode, and
the @code{srl}, @code{serial}, @code{Serial}, @code{nil}, or
@code{none} will select Serial mode (which disables file-level
parallelism, though still allows intra-file OpenMP parallelism). 

The default file-level parallelism for @command{ncremap} is Serial
mode (i.e., no file-level parallelism), in which @command{ncremap}
processes one input file at a time.
Background and @acronym{MPI} modes implement true file-level
parallelism. 
Typically both these parallel modes scale well with sufficent
memory unless and until I/O contention becomes the bottleneck. 
In Background mode @command{ncremap} issues all commands to regrid
the input file list as @acronym{UNIX} background processes on the
local node. 
Nodes with mutiple cores and sufficient @acronym{RAM} take advantage
of this to simultaneously regrid multiple files.
In @acronym{MPI} mode @command{ncremap} issues commands to regrid
the input file list in round-robin fashion to all available compute
nodes. 
Prior to @acronym{NCO} version 4.9.0 (released December, 2019),
Background and @acronym{MPI} parallelism modes both regridded all
the input files at one time and there was no way to limit the number 
of files being simultaneously regridded.
Subsequent versions allow finer grained parallelism by introducing
the ability to limit the number of discrete workflow elements or
``jobs'' (i.e., file regriddings) to perform simultaneously within an
@command{ncremap} invocation or ``workflow''.

As of @acronym{NCO} version 4.9.0 (released December, 2019),
the @samp{--job_nbr=@var{job_nbr}} option specifies the maximum number
of files to regrid simultaneously on all nodes being harnessed by the
workflow.
Thus @var{job_nbr} is an additional parameter to fine-tune file level
parallelism (it has no effect in Serial mode).
Please see the @command{ncremap} @var{job_nbr} documentation for more
details.

@html
<a name="pdq_opt"></a> <!-- http://nco.sf.net/nco.html#pdq_opt -->
<a name="prm_opt"></a> <!-- http://nco.sf.net/nco.html#prm_opt -->
<a name="prm"></a> <!-- http://nco.sf.net/nco.html#prm -->
<a name="pdq"></a> <!-- http://nco.sf.net/nco.html#pdq -->
<a name="permute"></a> <!-- http://nco.sf.net/nco.html#permute -->
@end html
@cindex @var{pdq_opt}
@cindex @code{--pdq_opt}
@cindex @code{--pdq}
@cindex @code{--prm_opt}
@cindex @code{--prm}
@cindex @code{--permute}
@item --pdq_opt @var{pdq_opt} (@code{--pdq}, @code{--prm_opt}, @code{--prm}, @code{--permute})
Specifies the dimension permutation option used by @command{ncpdq}
prior to regridding. 
Synonyms include @samp{--pdq}, @samp{--prm}, @samp{--prm_opt}, and
@samp{--permute}. 
Files to be regridded must have their horizontal spatial dimension(s) 
in the last (most-rapidly-varying) position. 
Most data files store variables with dimensions arranged in this
order, and @command{ncremap} internally sets the permutation option
for datasets known (via the @code{--prc_typ} option) to require
permutation.
Use @samp{--permute=@var{pdq_opt}} to override the internally preset
defaults.
This is useful when regridding files that contain new dimensions that 
@command{ncremap} has not encountered before.
For example, if a development version of an @acronym{MPAS} model
inserts a new dimension @code{new_dim} after the horizontal spatial
dimension @code{nCells} in some variables, that would prevent the
regridder from working because the horizontal dimension(s) must
be the last dimension(s).
The workaround is to instruct @command{ncremap} what the permutation
option to @command{ncpdq} should be in order to place the horizontal
spatial dimension(s) at the end of all variables:
@example
ncremap --permute=Time,new_dim,nCells  --map=map.nc in.nc out.nc
ncremap --permute=time,new_dim,lat,lon --map=map.nc in.nc out.nc
@end example
The @acronym{API} for this option changed in @acronym{NCO} version
5.0.4 (released December, 2021).
Prior to this, the option argument needed to include the entire
option string to be passed to @command{ncpdq} @emph{including the
@samp{-a}}, e.g., @samp{--permute='-a time,new_dim,lat,lon'}.
Now @command{ncremap} supplies the implicit @samp{-a} internally
so the user does not need to know the @command{ncpdq} syntax.

@html
<a name="no_permute"></a> <!-- http://nco.sf.net/nco.html#no_permute -->
@end html
@cindex permute
@cindex @command{ncpdq}
@cindex @acronym{SLURM}
@cindex @acronym{CWL}
@cindex @code{--no_permute}
@item --no_permute (@code{--no_permute}, @code{--no_prm}, @code{--no_pdq}, @code{--no_ncpdq})
Introduced in @acronym{NCO} version 5.0.0 (released June, 2021),
this switch (which takes no argument) causes the regridder to skip the
default permutation of dimensions before regridding (notably
@acronym{MPAS}) datasets known to store data with non-horizontal
most-rapidly varying dimensions.
@command{ncremap} normally ensures that input fields are stored in the 
shape expected by regridder weights (horizontal dimensions last)
by permuting the dimensions with @command{ncpdq}.
However, permutation consumes time and generates an extra intermediate
file. 
Avoid this time penalty by using the @samp{--no_permute} flag if the
input fields are known to already have trailing horizontal
dimensions. 

@html
<a name="preserve"></a> <!-- http://nco.sf.net/nco.html#preserve -->
@end html
@cindex @code{--preserve=@var{prs_stt}}
@cindex @var{prs_stt}
@cindex @code{--preserve}
@cindex @code{--prs_stt}
@cindex @code{--preserve_statistic}
@item --preserve=@var{prs_stt} (@code{--preserve}, @code{--prs_stt}, @code{--preserve_statistic})
This is a simple, intuitive option to specify how weight application
should treat destination gridcells that are not completely overlapped
by source gridcells with valid values.
Destination gridcells that are completely overlapped by valid source
values are unaffected.
The two statistics that can be preserved for incompletely overlapped
gridcells are the local mean and/or the global integral of the source
values. 
Hence the valid values for this option are @samp{integral}
(and, as of @acronym{NCO} @w{version 5.0.2}, released in September,
2021, its synonyms @samp{global} and @samp{global_integral}) and
@samp{mean} (or its synonyms @samp{local}, @samp{local_mean},
@samp{gridcell}, @samp{gridcell_mean}, @samp{natural_values}).
@acronym{NCO} @w{version 5.1.5}, released in March, 2023, fixed a
longstanding problem with the implmentation of this option, which
had essentially been broken since its inception.
The option finally works as documented.

Specifying @code{--preserve=@var{integral}} sets the destination
gridcell equal to the sum of the valid source values times their
source weights.
This sum is @emph{not} renormalized by the (valid) fractional area
covered. 
This is exactly equivalent to setting @code{--rnr=off}, i.e.,
no renormalization (see @ref{Regridding}).
If the weights were generated by a conservative algorithm then the
output will be conservative, and will conserve the global integral of 
the input field in all cases.
This is often desired for regridding quantities that should be
conserved, e.g., fluxes, and is the default weight application method 
in @command{ncremap} (except in @acronym{MPAS}-mode).
Specifying @code{--preserve=@var{mean}} sets the destination
gridcell equal to the mean of the valid source values times
their source weights.
This is exactly equivalent to setting @code{--rnr=0.0}, i.e.,
renormalizing the integral value by the (valid) fractional area
covered (see @ref{Regridding}). 
This is often desired for regridding state variables, e.g.,
temperature, though it is not the default behavior and must be
explicitly requested (except in @acronym{MPAS}-mode).
These two types of preserved statistics, integral and mean, produce
identical output in all gridcells where there are no missing data,
i.e., where valid data completely tile the gridcell.
By extension, these two statistics produce identical global means
if valid data completely tile the sphere.

@html
<a name="rgr_opt"></a> <!-- http://nco.sf.net/nco.html#rgr_opt -->
@end html
@cindex @code{-R @var{rgr_opt}}
@cindex @var{rgr_opt}
@cindex @code{--rgr_opt}
@cindex @code{--regrid_options}
@item -R @var{rgr_opt} (@code{--rgr_opt}, @code{--regrid_options})
@command{ncremap} passes @var{rgr_opt} directly through to the
regridder.  
This is useful to customize output grids and metadata.
One use is to rename output variables and dimensions from the defaults
provided by or derived from input data.
The default value is @samp{--rgr lat_nm_out=lat --rgr lon_nm_out=lon},
i.e., by default @command{ncremap} always names latitude and longitude
``lat'' and ``lon'', respectively, regardless of their input names.
Users might use this option to set different canonical axes names,
e.g., @samp{--rgr lat_nm_out=y --rgr lon_nm_out=x}.

@html
<a name="rnr_thr"></a> <!-- http://nco.sf.net/nco.html#rnr_thr -->
@end html
@cindex @code{-r @var{rnr_thr}}
@cindex @var{rnr_thr}
@cindex @code{--rnr_thr}
@cindex @code{--thr_rnr}
@cindex @code{--rnr}
@cindex @code{--renormalize}
@cindex @code{--renormalization_threshold}
@item -r @var{rnr_thr} (@code{--rnr_thr}, @code{--thr_rnr}, @code{--rnr}, @code{--renormalize}, @code{--renormalization_threshold})
Use this option to request renormalized (see @ref{Regridding})
weight-application and to specify the weight threshold, if any.
For example, @samp{-r 0.9} tells the regridder to renormalize
with a weight threshold of 90%, so that all destination gridcells
with at least 90% of their area contributed by valid source gridcells
will be contain valid (not missing) values that are the area-weighted
mean of the valid source values.
If the weights are conservative, then the output gridcells on the
destination grid will preserve the mean of the input gridcells.
Specifying @samp{-r 0.9} and @samp{--rnr_thr=0.9} are equivalent.
Renormalization can be explicitly turned-off by setting @var{rnr_thr}
to either of the values @samp{off}, or @samp{none}. 
The @samp{--preserve=@var{prs_stt}} option performs the same task
as this option except it does not allow setting an arbitrary threshold
fraction.  

@html
<a name="rgn_dst"></a> <!-- http://nco.sf.net/nco.html#rgn_dst -->
<a name="rgn_src"></a> <!-- http://nco.sf.net/nco.html#rgn_src -->
@end html
@cindex @var{rgn_dst}
@cindex @code{--rgn_dst}
@cindex @code{--dst_rgn}
@cindex @code{--regional_destination}
@item --rgn_dst (@code{--rgn_dst}, @code{--dst_rgn}, @code{--regional_destination})
@cindex @var{rgn_src}
@cindex @code{--rgn_src}
@cindex @code{--src_rgn}
@cindex @code{--regional_source}
@item --rgn_src (@code{--rgn_src}, @code{--src_rgn}, @code{--regional_source})
Use these flags which take no argument to indicate that a user-supplied 
(i.e., with @samp{-s @var{grd_src}} or @samp{-g @var{grd_dst}})
grid is regional.
The @acronym{ERWG} weight-generator (at least all versions
@w{before 8.0}) needs to be told whether the source, destination, or
both grids are regional or global in order to optimize weight
production. 
@command{ncremap} supplies this information to the regridder for grids
it automatically infers from data files. 
However, the regridder needs to be explicitly told if user-supplied
(i.e., with either @samp{-s @var{grd_src}} or @samp{-g @var{grd_dst}})
grids are regional because the regridder does not examine supplied grids 
before calling @acronym{ERWG} which assumes, unless told otherwise,
that grids are global in extent.
The sole effect of these flags is to add the arguments
@samp{--src_regional} and/or @samp{--dst_regional} to @acronym{ERWG} 
calls.
Supplying regional grids without invoking these flags may dramatically 
increase the map-file size and time to compute.
According to @acronym{E3SM} @acronym{MPAS} documentation, @acronym{ERWG}
``considers a mesh to be regional when the mesh is not a full sphere
(including if it is planar and does not cover the full sphere).
In other words, all @acronym{MPAS-O} and @acronym{MPAS-LI} grids are
regional'' to @acronym{ERWG}.

@html
<a name="grd_src"></a> <!-- http://nco.sf.net/nco.html#grd_src -->
@end html
@cindex @code{-s @var{grd_src}}
@cindex @var{grd_src}
@cindex @code{--grd_src}
@cindex @code{--grid_source}
@cindex @code{--source_grid}
@cindex @code{--src_grd}
@item -s @var{grd_src} (@code{--grd_src}, @code{--grid_source}, @code{--source_grid}, @code{--src_grd})
Specifies the source gridfile.
@acronym{NCO} will use @acronym{ERWG} or TempestRemap weight-generator 
to combine this with a destination gridfile (either inferred from
@var{dst_fl}, or generated by supplying a  @samp{-G @var{grd_sng}}
option) to generate remapping weights. 
@var{grd_src} is not modified, and may have read-only permissions.
One appropriate circumstance to specify @var{grd_src} is when the
@var{input-file}(s) do not contain sufficient information for 
@acronym{NCO} to infer an accurate or complete source grid.
(Unfortunately many dataset producers do not record information like
cell edges/vertices in their datasets. 
This is problematic for non-rectangular grids.)
@acronym{NCO} assumes that @var{grd_src}, when supplied, applies to 
every @var{input-file}.
Thus @acronym{NCO} will call the weight generator only once, and will
use that @var{map_fl} to regrid every @var{input-file}.

Although @command{ncremap} usually uses the contents of a pre-existing
@var{grd_src} to create mapping weights, there are some situations
where @command{ncremap} creates the file specified by @var{grd_src}
(i.e., treats it as a location for storing output).
When a source grid is inferred or created from other user-specified
input, @command{ncremap} will store it in the location specified by
@var{grd_src}.
This allows users to, for example, name the grid on which an input
dataset is stored when that grid is not known @emph{@w{a priori}}.
This functionality is only available for @acronym{SCRIP}-format grids. 

@html
<a name="skl_fl"></a> <!-- http://nco.sf.net/nco.html#skl_fl -->
@end html
@cindex @code{--skl_fl=@var{skl_fl}}
@cindex @var{skl_fl}
@cindex @code{--skl_fl}
@cindex @code{--skl}
@cindex @code{--skeleton}
@cindex @code{--skeleton_file}
@cindex skeleton
@item --skl_fl=@var{skl_fl} (@code{--skl_fl}, @code{--skl}, @code{--skl_fl})
Normally @command{ncremap} only creates a @acronym{SCRIP}-format
gridfile named @var{grd_dst} when it receives the @code{grd_sng}
option.
The @samp{--skl} option instructs @command{ncremap} to also produce a  
``skeleton'' file based on the @code{grd_sng} argument.
A skeleton file is a bare-bones datafile on the specified grid.
It contains the complete latitude/longitude grid and an area field.
Skeleton files are useful for validating that the grid-creation
instructions in @var{grd_sng} perform as expected.

@html
<a name="stdin"></a> <!-- http://nco.sf.net/nco.html#stdin -->
@end html
@cindex @acronym{SLURM}
@cindex @acronym{PBS}
@cindex @code{--stdin}
@cindex @code{--inp_std}
@cindex @code{--std_flg}
@cindex @code{--redirect}
@cindex @code{--standard_input}
@item --stdin (@code{--stdin}, @code{--inp_std}, @code{--std_flg}, @code{--redirect}, @code{--standard_input})
This switch (which takes no argument) explicitly indicates that 
input file lists are provided via @code{stdin}, i.e., standard input.
In interactive environments, @command{ncremap} can automatically
(i.e., without any switch) detect whether input is provided via
@code{stdin}.  
This switch is never required for jobs run in an interactive shell. 
However, non-interactive batch jobs (such as those submitted to the 
@acronym{SLURM} and @acronym{PBS} schedulers) make it impossible to 
unambiguously determine whether input has been provided via
@code{stdin}. 
Specifically, the @samp{--stdin} switch @emph{must} be used with
@command{ncremap} in non-interactive batch jobs on @acronym{PBS}
when the input files are piped to @code{stdin}, and on @acronym{SLURM}
when the input files are redirected from a file to @code{stdin}
@footnote{
Until version 5.0.4 (December, 2021) the @samp{--stdin} was
also supported by @command{ncclimo}, and used for the same reasons
as it still is for @command{ncclimo}.
At that time, the @samp{--split} switch superceded the @samp{--stdin} 
switch in @command{ncclimo}, where it is now deprecated.}.
Using @samp{--stdin} in any other context (e.g., interactive shells)
is optional.

In some other non-interactive environments (e.g., @command{crontab},
@code{nohup}, Azure @acronym{CI}, @acronym{CWL}),
@command{ncremap} may mistakenly expect input to be provided on
@code{stdin} simply because the environment is using @code{stdin} for
other purposes. 
In such cases users may disable checking @code{stdin} by explicitly
invoking the @samp{--no_stdin} flag (described next), which works for
both @command{ncclimo} and @command{ncremap}. 

@html
<a name="no_stdin"></a> <!-- http://nco.sf.net/nco.html#no_stdin -->
@end html
@cindex Chrysalis
@cindex Common Workflow Language
@cindex Azure @acronym{CI}
@cindex @acronym{SLURM}
@cindex @acronym{CWL}
@cindex @code{--no_stdin}
@cindex @code{--no_inp_std}
@cindex @code{--no_standard_input}
@cindex @code{--no_redirect}
@item --no_stdin (@code{--no_stdin}, @code{--no_inp_std}, @code{--no_redirect}, @code{--no_standard_input})
First introduced in @acronym{NCO} version 4.8.0 (released May, 2019),
this switch (which takes no argument) disables checking standard input
(aka @code{stdin}) for input files. 
This is useful because @command{ncclimo} and @command{ncremap} may
mistakenly expect input to be provided on @code{stdin} in environments
that use @code{stdin} for other purposes. 
Some non-interactive environments (e.g., @command{crontab},
@code{nohup}, Azure @acronym{CI}, @acronym{CWL}), may
use standard input for their own purposes, and thus confuse
@acronym{NCO} into thinking that you provided the input files names
via the @code{stdin} mechanism. 
In such cases users may disable the automatic checks for standard
input by explicitly invoking the @samp{--no_stdin} flag.
This switch is usually not required for jobs in an interactive shell.
Interactive @acronym{SLURM} shells can also commandeer @code{stdin},
as is the case on the @code{DOE} machine named Chrysalis.
This behavior appears to vary depending on the @acronym{SLURM}
implementation.
@example
@verbatim
srun -N 1 -n 1 ncremap --no_stdin -m map.nc in.nc out.nc
@end verbatim
@end example

@html
<a name="tmp_drc"></a> <!-- http://nco.sf.net/nco.html#tmp_drc -->
@end html
@cindex @code{-T @var{tmp_drc}}
@cindex @var{tmp_drc}
@cindex @code{--tmp_drc}
@cindex @code{--drc_tmp}
@cindex @code{--tmp_dir}
@cindex @code{--dir_tmp}
@cindex @code{--tmp}
@item -T @var{tmp_drc} (@code{--tmp_drc}, @code{--drc_tmp}, @code{--tmp_dir}, @code{--dir_tmp}, @code{--tmp_drc})
Specifies the directory in which to place intermediate output files.
Depending on how it is invoked, @command{ncremap} may generate
a few or many intermediate files (grids and maps) that it will, by
default, remove upon successful completion.
These files can be large, so the option to set @var{tmp_drc} is offered
to ensure their location is convenient to the system.
If the user does not specify @var{tmp_drc}, then @command{ncremap} uses
the value of @code{$TMPDIR}, if any, or else @file{/tmp} if it exists,
or else it uses the current working director (@code{$PWD}).

@html
<a name="thr_nbr"></a> <!-- http://nco.sf.net/nco.html#thr_nbr -->
<a name="thr_nbr_ncremap"></a> <!-- http://nco.sf.net/nco.html#thr_nbr_ncremap -->
@end html
@cindex @code{-t @var{thr_nbr}}
@cindex @var{thr_nbr}
@cindex @code{--thr}
@cindex @code{--thr_nbr}
@cindex @code{--thread_number}
@cindex @code{--threads}
@item -t @var{thr_nbr} (@code{--thr_nbr}, @code{--thr}, @code{--thread_number}, @code{--threads})
Specifies the number of threads used per regridding process
(@pxref{OpenMP Threading}).
@command{ncremap} can use OpenMP shared-memory techniques to
simultaneosly regrid multiple variables within a single file.
This shared memory parallelism is quite efficient because it
uses a single copy of the regridding weights in physical memory
to regrid multiple variable simultaneously.
Even so, simultaneously regridding multiple variables, especially at
high resolution, may be memory-limited, meaning that the insufficient
@acronym{RAM} can often limit the number of variables that the system
can simultaneously regrid.
By convention all variables to be regridded share the same
regridding weights stored in a map-file, so that only one copy
of the weights needs to be in memory, just as in Serial mode.
However, the per-thread (i.e., per-variable) OpenMP memory demands 
are considerable, with the memory required to regrid variables
amounting to no less than about 5--7 times (for type @code{NC_FLOAT})
and 2.5--3.5 times (for type @code{NC_DOUBLE}) the size of the
uncompressed variable, respectively.
Memory requirements are so high because the regridder performs
all arithmetic in double precision to retain the highest accuracy,
and must allocate separate buffers to hold the input and output
(regridded) variable, a tally array to count the number of missing
values and an array to sum the of the weights contributing to each
output gridcell (the last two arrays are only necessary for variables
with a @code{_FillValue} attribute).
The input, output, and weight-sum arrays are always double precision,
and the tally array is composed of four-byte integers.
Given the high memory demands, one strategy to optimize @var{thr_nbr}
for repetitious workflows is to increase it to keep doubling it (1, 2,
4, @dots{}) until throughput stops improving. 
With sufficient @acronym{RAM}, the @acronym{NCO} regridder scales well
up to 8--16 threads. 

@html
<a name="upk_inp"></a> <!-- http://nco.sf.net/nco.html#upk_inp -->
@end html
@cindex @code{-U}
@cindex @code{--unpack}
@cindex @code{--upk}
@cindex @code{--upk_inp}
@item -U (@code{--unpack}, @code{--upk}, @code{--upk_inp})
This switch (which takes no argument) causes @command{ncremap} to
unpack (see @ref{Packed data}) input data before regridding it.
This switch causes unpacking at the regridding stage that occurs
after map generation.
Hence this switch does not benefit grid inferral.
Grid inferral examines only the coordinate variables in a dataset.
If coordinates are packed (a terrible practice) in a file from which a
grid will be inferred, users should first manually unpack the file (this
option will not help). 
Fortunately, coordinate variables are usually not packed, even in files
with other packed data.

Many institutions (like @acronym{NASA}) pack datasets to conserve space
before distributing them. 
This option allows one to regrid input data without having
to manually unpack it first.
Beware that @acronym{NASA} uses at least three different and
incompatible versions of packing in its @acronym{L2} datasets.
The unpacking algorithm employed by this option is the default netCDF
algorithm, which is appropriate for @acronym{MOD04} and is inappropriate
for @acronym{MOD08} and @acronym{MOD13}. 
See @ref{Packed data} for more details and workarounds.

@html
<a name="ugrid_fl"></a> <!-- http://nco.sf.net/nco.html#ugrid_fl -->
@end html
@cindex @code{--ugrid_fl=@var{ugrid_fl}}
@cindex @var{ugrid_fl}
@cindex @code{--ugrid_fl}
@cindex @code{--ugrid}
@cindex @code{--ugrid_file}
@cindex infer
@item --ugrid_fl=@var{ugrid_fl} (@code{--ugrid_fl}, @code{--ugrid}, @code{--ugrid_fl})
Normally @command{ncremap} only infers a gridfile named @var{grd_dst} in 
@acronym{SCRIP}-format.
The @samp{ugrid_fl} option instructs @command{ncremap} to infer both a 
@acronym{SCRIP}-format gridfile named @var{grd_dst} and a
@acronym{UGRID}-format gridfile named @var{ugrid_fl}.
This is an experimental feature and the @acronym{UGRID} file is
only expected to be valid for global rectangular grids. 

@html
<a name="unq_sfx"></a> <!-- http://nco.sf.net/nco.html#unq_sfx -->
@end html
@cindex @code{-u @var{unq_sfx}}
@cindex @var{unq_sfx}
@cindex noclean
@cindex @code{--unq_sfx}
@cindex @code{--unique_suffix}
@cindex @code{--suffix}
@item -u @var{unq_sfx} (@code{--unq_sfx}, @code{--unique_suffix}, @code{--suffix})
Specifies the suffix used to label intermediate (internal) files
generated by the regridding workflow.
Unique names are required to avoid interference among parallel
invocations of @command{ncremap}. 
The default @var{unq_sfx} generated internally by @command{ncremap} is
@samp{.pid@var{PID}} where @var{PID} is the process ID.
Applications can provide their own more or less informative suffixes
using the @samp{--unq_sfx=@var{unq_sfx}} option.
The suffix should be unique so that no two simultaneously executing 
instances of @command{ncremap} can generate the same file.
For example, when the @command{ncclimo} climatology script issues a
dozen @command{ncremap} commands to regrid all twelve months 
simultaneously, it uses @samp{--unq_sfx=@var{mth_idx}} to encode the
climatological month index in the unique suffix.
Note that the controlling process @var{PID} is insufficient to
disambiguate all the similar temporary files when the input file list 
is divided into multiple concurrent jobs (controlled by the
@samp{--job_nbr=@var{job_nbr}} option).
Those files have their user-provided or internally generated
@var{unq_sfx} extended by @var{fl_idx}, their position in the input 
file list, so that their full suffix is
@samp{.pid@var{PID}.@var{fl_idx}}. 
Finally, a special value of @var{unq_sfx} is available to aid
developers: if @var{unq_sfx} is @samp{noclean} then @command{ncremap}
retains (not removes) all intermediate files after completion.

@html
<a name="var_lst"></a> <!-- http://nco.sf.net/nco.html#var_lst -->
@end html
@cindex @code{-v @var{var_lst}}
@cindex @var{var_lst}
@cindex @code{--var_lst}
@cindex @code{--var}
@cindex @code{--vars}
@cindex @code{--variables}
@cindex @code{--variable_list}
@item -v @var{var_lst} (@code{--var_lst}, @code{--var}, @code{--vars}, @code{--variables}, @code{--variable_list})
The @samp{-v} option causes @command{ncremap} to regrid only the
variables in @var{var_lst}. 
It behaves like subsetting (@pxref{Subsetting Files}) in the rest of
@acronym{NCO}.   

@html
<a name="rgr_var"></a> <!-- http://nco.sf.net/nco.html#rgr_var -->
@end html
@cindex @code{-V @var{rgr_var}}
@cindex @var{rgr_var}
@cindex @code{--rgr_var}
@cindex @code{--var_rgr}
@cindex @code{--var_cf}
@cindex @code{--cf_var}
@cindex @code{--cf_variable}
@item -V @var{var_rgr} (@code{--var_rgr}, @code{--rgr_var}, @code{--var_cf}, @code{--cf_var}, @code{cf_variable})
The @samp{-V} option tells @command{ncremap} to use the same grid as 
@var{var_rgr} in the input file.
If @var{var_rgr} adheres to the @acronym{CF} @code{coordinates}
convention described 
@uref{http://cfconventions.org/cf-conventions/cf-conventions.html#coordinate-system, here},
then @command{ncclimo} will infer the grid as represented by those
coordinate variables.
This option simplifies inferring grids when the grid coordinate names 
are unknown, since @command{ncclimo} will follow the @acronym{CF}
convention to learn the identity of the grid coordinates.

Until @acronym{NCO} version 4.6.0 (May, 2016), @command{ncremap} would 
not follow @acronym{CF} conventions to identify coordinate variables.
Instead, @command{ncremap} used an internal database of ``usual
suspects'' to identify latitude and longitude coordinate variables.
Now, if @var{var_rgr} is @acronym{CF}-compliant, then @command{ncremap}
will automatically identify the horizontal spatial dimensions. 
If @var{var_rgr} is supplied but is not @acronym{CF}-compliant, then
@command{ncremap} will still attempt to identify horizontal spatial
dimensions using its internal database of ``likely names''.
If both these automated methods fail, manually supply @command{ncremap}
with the names of the horizontal spatial dimensions 
@example
# Method used to obtain horizontal spatial coordinates:
ncremap -V var_rgr -d dst.nc -O ~/rgr in.nc # CF coordinates convention
ncremap -d dst.nc -O ~/rgr in.nc # Internal database
ncremap -R "--rgr lat_nm=xq --rgr lon_nm=zj" -d dst.nc -O ~/rgr in.nc # Manual
@end example
@noindent

@html
<a name="vrb_lvl"></a> <!-- http://nco.sf.net/nco.html#vrb_lvl -->
@end html
@cindex @code{--vrb=@var{vrb_lvl}}
@cindex @var{vrb_lvl}
@cindex @code{--vrb_lvl}
@cindex @code{--vrb}
@cindex @code{--verbosity}
@cindex @code{--verbosity_level}
@item --vrb=@var{vrb_lvl} (@code{--vrb_lvl}, @code{--vrb}, @code{--verbosity}, @code{--verbosity_level})
Specifies a verbosity level similar to the rest of @acronym{NCO}.
If @math{@var{vrb_lvl} = 0}, @command{ncremap} prints nothing except
potentially serious warnings.
If @math{@var{vrb_lvl} = 1}, @command{ncremap} prints the basic
filenames involved in the remapping.
If @math{@var{vrb_lvl} = 2}, @command{ncremap} prints helpful comments
about the code path taken.
If @math{@var{vrb_lvl} > 2}, @command{ncremap} prints even more detailed
information.
Note that @var{vrb_lvl} is distinct from @var{dbg_lvl} which is
passed to the regridder (@command{ncks}) for additional diagnostics.

@html
<a name="vrt"></a> <!-- http://nco.sf.net/nco.html#vrt -->
<a name="vrt_fl"></a> <!-- http://nco.sf.net/nco.html#vrt_fl -->
@end html
@cindex @code{vrt}
@cindex @code{--vrt}
@cindex @code{--vrt_crd}
@cindex @code{--vrt_fl_out}
@cindex @code{--vrt_out=@var{vrt_fl}}
@cindex @var{vrt_fl}
@cindex Vertical coordinate
@cindex @acronym{MPAS}
@item --vrt_out=@var{vrt_fl} (@code{--vrt_out}, @code{--vrt_fl}, @code{--vrt}, @code{--vrt_crd}, @code{--vrt_grd_out})
The @samp{--vrt_out=@var{vrt_fl}} option instructs @command{ncremap}
to vertically interpolate the input file to the vertical coordinate
grid contained in the file @var{vrt_fl}.
This option first appeared in @acronym{NCO} @w{version 4.8.0},
released in May, 2019.
The vertical gridfile @var{vrt_fl} must specify a vertical gridtype
that @command{ncremap} understands, currently either pure-pressure or
hybrid-coordinate pressure.
We plan to add pure-sigma coordinates in the future.

Besides the vertical grid-type, the main assumptions, constraints, and
priorities for future development of vertical regridding are:
@enumerate
@item When originally released, the vertical interpolation feature
      required that the input datasets have netCDF (and thus C-based)
      dimension-ordering @emph{all other dimensions, a single vertical 
      dimension, then one or two horizontal dimensions, if any} so
      that the horizontal dimension(s) vary most rapidly.
      @acronym{NCO} @w{version 5.1.4}, released in January, 2023,
      eliminated this constraint.
      Now the regridder automatically determines whether the vertical
      or the horizontal dimensions vary most rapidly, and adjusts its
      algorithms accordingly.
@item The vertical interpolation algorithm defaults to linear in
      log(pressure) for pure pressure and for hybrid sigma-pressure
      coordinates.  
      This assumption is more natural for gases (like the atmosphere)
      than for condensed media (like oceans or Earth's interior).
      To instead interpolate linearly in the vertical coordinate, use
      the @samp{ntp_mth=lin} options (as of @acronym{NCO} 4.9.0).
      @acronym{NCO} @w{version 5.1.4}, released in January, 2023,
      supports interpolation of data structured with a geometric
      depth/height grid (such as ocean data usually uses).
      Data on a depth/height grid defaults to linear interpolation.
@item Vertical interpolation and horizontal regridding may be invoked
      simultaneously (as of @acronym{NCO} 4.9.0) by the user simply by
      supplying both a (horizontal) map-file and a vertical grid-file
      to @command{ncremap}. 
      When this occurs, @command{ncremap} internally performs the
      vertical interpolation prior to the horizontal regridding.
      Exploiting this feature can have some unintended consequences.
      For example, horizontal regridding requires that the horizontal
      spatial dimension vary most rapidly, whereas vertical
      interpolation makes no such assumption.
      When the regridder needs to permute the dimension ordering of
      the input dataset in order to perform horizontal regridding,
      this step actually precedes the vertical regridding.
      This order of operations is problematic and we are working
      to address the issues in future updates.
@item The default extrapolation method uses nearest neighbor except
      for temperature and geopotential (those extrapolation methods
      are described below).
      These defaults are well-suited to extrapolate valid initial
      conditions from data on older vertical grids.
      Note that the default approximation used for geopotential is
      inaccurate in cold regions.
      As of July 2019 and @acronym{NCO} @w{version 4.8.1}, one may
      instead set points outside the input domain to missing-values
      with the @samp{--vrt_xtr=mss_val} option.
      More extrapolation options, exposed to user-access as of
      November 2022 in @acronym{NCO} @w{version 5.1.1}, are:
      linear extrapolation (@samp{--vrt_xtr=linear}),
      setting @w{to 0.0} (@samp{--vrt_xtr=zero}).
      Linear extrapolation does exactly what you think: Values outside
      the input domain are linearly extrapolated from the nearest two
      values inside the input domain.
      Zero extrapolation sets values outside the extrapoloation domain
      @w{to 0.0}.  
      Supporting other methods, or improving the existing
      special-case approximations for temperature or geopotential,
      will remain low priority until we are lobbied with compelling
      use-cases for other algorithms.  
@item Missing values may not always be treated correctly
      Eliminating this constraint was not originally a priority
      because atmospheric datasets often contain no missing data.
      However, now that vertical regridding can be applied to ocean
      data with a depth coordinate, we need to verify that using
      missing values to indicate bathymetry works as expected.
@item Time-varying vertical grids are only allowed for hybrid grids
      (not pure pressure grids), and these must store the time
      dimension as a record dimension.
      This constraint applies to the vertical grid only, not to the
      other fields in the dataset.
      Hence this does not preclude interpolating timeseries to/from
      time-invariant vertical grids.
      For example, time-varying hybrid grid data such as temperature
      may be interpolated to timeseries on a time-invariant pressure
      grid.
      Eliminating this constraint will not be a priority unless/until
      an important use-case is identified.  
@item Variable names for input and output vertical grids must match
      @acronym{E3SM/CESM}, @acronym{ECMWF}, @acronym{MPAS}, and
      @acronym{NCEP} implementations. 
      These names include @code{hyai}, @code{hyam}, @code{hybi},
      @code{hybm}, @code{ilev}, @code{lev}, @code{P0}, and @code{PS}
      (for @acronym{E3SM/CESM} hybrid grids), @code{lev},
      @code{lev_2}, and @code{lnsp} (for @acronym{ECMWF} hybrid grids
      only), @code{depth}, @code{timeMonthly_avg_zMid} (for
      @acronym{MPAS} depth grids), and @code{plev} (for pure-pressure
      grids).  
      The infrastructure to provide alternate names for any of these
      input/output variables names is straightforward, and is heavily 
      used for horizontal spatial regridding.
      Allowing this functionality will not be a priority until we are 
      presented with a compelling use-case.
@end enumerate

@html
<a name="vrt_prs_mk"></a> <!-- http://nco.sf.net/nco.html#vrt_prs_mk -->
<a name="vrt_prs"></a> <!-- http://nco.sf.net/nco.html#vrt_prs -->
<a name="vrt_ncep"></a> <!-- http://nco.sf.net/nco.html#vrt_ncep -->
@end html
The simplest vertical grid-type, a pure-pressure grid, contains
the horizontally uniform vertical pressure levels in a one-dimensional 
coordinate array named (by default) @code{plev}. 
The @code{plev} dimension may have any number of levels and the values 
must monotonically increase or decrease.
A 17-level NCEP pressure grid, for example, is easy to create:
@example
@verbatim
# Construct monotonically decreasing 17-level NCEP pressure grid
ncap2 -O -v -s 'defdim("plev",17);plev[$plev]={100000,92500,85000, \
  70000,60000,50000,40000,30000,25000,20000,15000,10000,7000,5000, \
  3000,2000,1000};' vrt_prs_ncep_L17.nc
@end verbatim
@end example
@command{ncremap} will search the supplied vertical grid file for
the coordinate named @code{plev}, or, for any coordinate name
specified by with the @code{plev_nm_in} option to the regridder,
e.g., @samp{--plev_nm_in=z}.

@html
<a name="vrt_hyb_mk"></a> <!-- http://nco.sf.net/nco.html#vrt_hyb_mk -->
<a name="vrt_hyb"></a> <!-- http://nco.sf.net/nco.html#vrt_hyb -->
@end html
Hybrid-coordinate grids are a hybrid between a sigma-coordinate
grid (where each pressure level is a fixed fraction of a
spatiotemporally varying surface pressure) and a pure-pressure grid
that is spatially invariant (as described above). 
The so-called hybrid @var{A} and @var{B} coefficients specify the
fractional weight of the pure-pressure and sigma-grids, respectively, 
at each level.
@cindex @code{P0}
@cindex @code{PS}
@cindex @code{hyai}
@cindex @code{hyam}
@cindex @code{hybi}
@cindex @code{hybm}
@cindex @code{formula_terms} attribute
The hybrid gridfile must specify @var{A} and @var{B} coefficients for 
both layer midpoints and interfaces with these standard
(as employed by @acronym{CESM} and @acronym{E3SM}) names and
dimensions: @code{hyai(ilev)}, @code{hybi(ilev)}, @code{hyam(lev)}, 
and @code{hybm(lev)}.
The reference pressure and surface pressure must be named
@code{P0} and @code{PS}, respectively.
The pressures at all midpoints and interfaces are then defined as 
@example
@verbatim
prs_mdp[time,lev, lat,lon]=hyam*P0+hybm*PS # Midlayer
prs_ntf[time,ilev,lat,lon]=hyai*P0+hybi*PS # Interface
@end verbatim
@end example
The scalar reference pressure @code{P0} is typically @w{100000 Pa}
(or @w{1000 hPa}) while the surface pressure @code{PS} is a (possibly
time-varying) array with one or two spatial dimensions, and its values 
are in the same dimensional units (e.g., Pa or hPa) as @code{P0}.

It is often useful to create a vertical grid file from existing
model or reanalysis output.
We call vertical grid files ``skinny'' if they contain only the
vertical information.
Skinny grid-files are easy to create with @command{ncks}, e.g.,
@example
@verbatim
ncks -C -v hyai,hyam,hybi,hybm,P0 in_L128.nc vrt_hyb_L128.nc
@end verbatim
@end example
Such files are extremely small and portable, and represent
all the hybrid files created by the model because the vertical
grid parameters are time-invariant.
A ``fat'' vertical grid file would also include the time-varying
grid information, i.e., the surface pressure field.
Fat grid-files are also easy to create with @command{ncks}, e.g.,
@example
@verbatim
ncks -C -v hyai,hyam,hybi,hybm,P0,PS in_L128.nc vrt_hyb_L128.nc
@end verbatim
@end example
The full (layer-midpoint) and half (layer-interface) pressure fields
@code{prs_mdp} and @code{prs_ntf}, respectively, can be reconstructed
from any fat grid-file with an @command{ncap2} command:
@example
@verbatim
ncap2 -s 'prs_mdp[time,lat,lon,lev]=P0*hyam+PS*hybm' \
      -s 'prs_ntf[time,lat,lon,ilev]=P0*hyai+PS*hybi' in.nc out.nc
@end verbatim
@end example

Hybrid-coordinate grids define a pure-sigma or pure-pressure grid when
either their @var{A} or @var{B} coefficients are zero, respectively.
For example, the following creates the hybrid-coordinate
representation of a pure-pressure grid with midpoints every @w{100 hPa}
from @w{100 hPa} to @w{1000 hPa}:
@example
@verbatim
ncap2 -O -v -s 'defdim("ilev",11);defdim("lev",10);P0=100000.0; \
  hyai=array(0.05,0.1,$ilev);hyam=array(0.1,0.1,$lev); \
  hybi=0.0*hyai;hybm=0.0*hyam;' vrt_hyb_L10.nc
@end verbatim
@end example
@acronym{NCO} currently has no other means of representing pure sigma
vertical grids (as opposed to pure pressure grids).

As of July 2019 and @acronym{NCO} @w{version 4.8.1}, @acronym{NCO} 
supports regridding @acronym{ECMWF} datasets in @acronym{IFS} hybrid
vertical coordinate format to @acronym{CESM}/@acronym{E3SM}-format
hybrid vertical grids. 
The native @acronym{IFS} hybrid datasets that we have seen store
pressure coordinates in terms of a slightly different formula that
employs the log of surface pressure (@code{lnsp}) instead of surface
pressure @code{PS}, that redefines @code{hyai} and @code{hyam} to be
pure-pressure offsets (rather than coefficients), and that omits
@code{P0}: 
@example
@verbatim
prs_mdp[time,lev,  lat,lon]=hyam+hybm*exp(lnsp) # Midlayer
prs_ntf[time,lev_2,lat,lon]=hyai+hybi*exp(lnsp) # Interface
@end verbatim
@end example
Note that @acronym{ECMWF} also alters the names of the vertical
half-layer coordinate and employs distinct dimensions (@code{nhym} and 
@code{nhyi}) for the hybrid variables @code{hyai(nhyi)},
@code{hybi(nhyi)}, @code{hyam(nhym)}, and @code{hybm(nhym)}.
@acronym{ECMWF} uses the vertical coordinates @code{lev} and
@code{lev_2} for full-layer (i.e., midlayer) and  half-layer
(i.e., interface) for all other variables.

The @code{lev} and @code{ilev} coordinates of a hybrid grid are
defined by the hybrid coefficients and reference pressure, and are
by convention stored in millibars (not Pascals) as follows:
@example
@verbatim
ilev[ilev]=P0*(hyai+hybi)/100.0;
lev[lev]=P0*(hyam+hybm)/100.0;
@end verbatim
@end example

A vertical hybrid grid file @var{vrt_fl} must contain at least
@code{hyai}, @code{hybi}, @code{hyam}, @code{hybm(lev)} and
@code{P0}; @code{PS}, @code{lev}, and @code{ilev} are optional. 
(Exceptions for @acronym{ECMWF} grids are noted above).
All hybrid-coordinate data files must contain @code{PS}.
Interpolating a pure-pressure coordinate data file to hybrid
coordinates requires, therefore, that the hybrid-coordinate
@var{vrt_fl} must contain @code{PS} and/or the input data file
must contain @code{PS}.
If both contain @code{PS} then the @code{PS} from the @var{vrt_fl}
takes precedence and will be used to construct the hybrid grid
and then copied without to the output file.

In all cases @code{lev} and @code{ilev} are optional in input
hybrid-coordinate data files and vertical grid-files.  
They are diagnosed from the other parameters using the above
definitions. 
The minimal requirements---a @code{plev} coordinate for a
pure-pressure grid or five parameters for a hybrid grid---allow 
vertical gridfiles to be much smaller than horizontal gridfiles
such as @acronym{SCRIP} files.
Moreover, data files from @acronym{ESM}s or analyses (@acronym{NCEP},
@acronym{MERRA2}, @acronym{ERA5}) are also valid gridfiles.
The flexibility in gridfile structure makes it easy to intercompare
data from the same or different sources.

@command{ncremap} supports vertical interpolation between all
combinations of pure-pressure and hybrid-pressure grids.
The input and output (aka source and destination) pressure grids may
monotonically increase or decrease independently of eachother (i.e.,
one may increase and the other may decrease). 
When an output pressure level is outside the input pressure range
for that column, then all variables must be extrapolated (not
interpolated) to that/those level(s).
By default @command{ncremap} sets all extrapolated values to the
nearest valid value.

Temperature and geopotential height are exceptions to this rule.
Temperature variables (those named @code{T} or @code{ta}, anyway) are
extrapolated upwards towards space using the nearest neighbor
assumption, and downwards beneath the surface assuming a moist 
adiabatic lapse rate of 6.5 degrees centigrade per 100 millibars.
Geopotential variables (those named @code{Z3} or @code{zg}, anyway)
are extrapolated upwards and downwards using the hypsometric equation
@footnote{@math{Z_2-Z_1=(R_d*T_v/g_0)*ln(p_1/p_2)=(R_d*T_v/g_0)*(ln(p_1)-ln(p_2))}}
@c Z2-Z1=(Rd*Tv/g0)*ln(p1/p2)=(Rd*Tv/g0)*(ln(p1)-ln(p2))} 
with constant global mean virtual temperature
@math{@var{@tpt{}} = 288}K.
This assumption leads to unrealistic values where @tpt{} differs
significantly from the global mean surface temperature.
Using the local @tpt{} itself would be a much better approximation,
yet would require a time-consuming implementation.
Please let us know if accurate surface geopotential extrapolation in
cold regions is important to you.

Interpolation to and from hybrid coordinate grids works on both
midpoint and interface fields (i.e., on variables with @code{lev} or
@code{ilev} dimensions), while interpolation to and from pure-pressure
grids applies to fields with, or places output of fields on, a
@code{plev} dimension.  
All other fields pass through the interpolation procedure unscathed.
Input can be rectangular (aka @acronym{RLL}), curvilinear, or
unstructured.  

@html
<a name="ps_nm"></a> <!-- http://nco.sf.net/nco.html#ps_nm -->
<a name="ps"></a> <!-- http://nco.sf.net/nco.html#ps -->
@end html
@cindex @code{--ps_nm=@var{ps_nm}}
@cindex @var{ps_nm}
@cindex @code{--ps_nm}
@cindex @code{PS}
@cindex surface pressure
@item --ps_nm=@var{ps_nm} (@code{--ps_nm}, @code{--ps_name}, @code{--vrt_ps}, @code{--ps})
It is sometimes convenient to store the @var{ps_nm} field in
an external file from the field(s) to be regridded.
For example, @acronym{CMIP}-style timeseries are often written
with only one variable per file.
@acronym{NCO} supports this organization by accepting @var{ps_nm}
arguments in the form of a filename followed by a slash and then a
variable name: 
@example
ncremap --vrt_in=vrt.nc --ps_nm=ps in.nc out.nc # Use ps not PS
ncremap --vrt_in=vrt.nc --ps_nm=/external/file.nc/ps in.nc out.nc
@end example
This same functionality (of optionally embedding a filename into
the variable name) is also implemented for the @var{sgs_frc} variable.

@html
<a name="ps_rtn"></a> <!-- http://nco.sf.net/nco.html#ps_rtn -->
<a name="rtn_sfc_prs"></a> <!-- http://nco.sf.net/nco.html#rtn_sfc_prs -->
<a name="retain_surface_pressure"></a> <!-- http://nco.sf.net/nco.html#retain_surface_pressure -->
@end html
@cindex @code{--ps_rtn}
@cindex @var{ps_nm}
@cindex @code{--ps_rtn}
@cindex @code{--retain_surface_pressure}
@cindex @code{--rtn_sfc_prs}
@cindex surface pressure, retaining
@item --ps_rtn (@code{--ps_rtn}, @code{--rtn_sfc_prs}, @code{--retain_surface_pressure})
As of @acronym{NCO} version 5.1.5 (March, 2023), @command{ncremap} 
includes a @code{--ps_rtn} switch (with long-option equivalents
@code{--rtn_sfc_prs} and @code{--retain_surface_pressure}) to
facilitate ``round-trip'' vertical interpolation such as
hybrid-to-pressure followed by pressure-to-hybrid interpolation.
By default @command{ncremap} excludes the surface pressure field named
@var{ps_nm} from the output after hybrid-to-pressure interpolation. 
The @code{--ps_rtn} switch (which takes no argument) instructs the
regridder to retain the surface pressure field after
hybrid-to-pressure interpolation.
The surface pressure field is then available for subsequent
interpolation back to a hybrid vertical coordinate:
@example
ncremap --ps_rtn --ps_nm=ps --vrt_out=ncep.nc in.nc out_ncep.nc
ncremap --ps_rtn -v T,Q,U,PS --vrt_out=ncep.nc in.nc out_ncep.nc
ncremap --vrt_out=hybrid.nc out_ncep.nc out_hybrid.nc
@end example

@html
<a name="vrt_ntp"></a> <!-- http://nco.sf.net/nco.html#vrt_ntp -->
@end html
@cindex @code{--vrt_ntp=@var{vrt_ntp}}
@cindex @var{vrt_ntp}
@cindex @code{--vrt_ntp}
@cindex @code{--ntp_mth}
@cindex @code{--interpolation_type}
@cindex @code{--interpolation_method}
@item --vrt_ntp=@var{vrt_ntp} (@code{--vrt_ntp}, @code{--ntp_mth}, @code{--interpolation_type}, @code{--interpolation_method})
Specifies the interpolation method for destination points within the 
vertical range of the input data during vertical interpolation.
Valid values and their synonyms are
@code{lin} (synonyms @code{linear} and @code{lnr}), and
@code{log} (synonyms @code{logarithmic} and @code{lgr}).
Default is @math{@var{vrt_ntp} = @code{log}}.
The vertical interpolation algorithm defaults to linear in
log(pressure). 
Logarithmic interpolation is more natural for gases like the
atmosphere, because it is compressible, than for condensed media like
oceans or  Earth's interior, which are incompressible.
To instead interpolate linearly in the vertical coordinate, use
the @samp{ntp_mth=lin} option.
@acronym{NCO} supports this feature as of version 4.9.0 (December,
2019). 

@html
<a name="vrt_xtr"></a> <!-- http://nco.sf.net/nco.html#vrt_xtr -->
@end html
@cindex @code{--vrt_xtr=@var{vrt_xtr}}
@cindex @var{vrt_xtr}
@cindex @code{--vrt_xtr}
@cindex @code{--xtr_mth}
@cindex @code{--extrapolation_type}
@cindex @code{--extrapolation_method}
@item --vrt_xtr=@var{vrt_xtr} (@code{--vrt_xtr}, @code{--xtr_mth}, @code{--extrapolation_type}, @code{--extrapolation_method})
Specifies the extrapolation method for destination points outside the
vertical range of the input data during vertical interpolation.
Valid values and their synonyms are
@code{linear} (synonyms @code{lnr} and @code{lin}), 
@code{mss_val} (synonyms @code{msv} and @code{missing_value}), 
@code{nrs_ngh} (synonyms @code{nn} and @code{nearest_neighbor}), and
@code{zero} (synonym @code{nil}).
Default is @math{@var{vrt_xtr} = @code{nrs_ngh}}.
@acronym{NCO} supports this feature as of version 4.8.1 (July, 2019).

@html
<a name="wgt_opt"></a> <!-- http://nco.sf.net/nco.html#wgt_opt -->
@end html
@cindex @code{-W @var{wgt_opt}}
@cindex @var{wgt_opt}
@cindex @code{--wgt_opt}
@cindex @code{--weight_options}
@cindex @code{--tps_opt}
@cindex @code{--tempest_options}
@cindex @code{--esmf_opt}
@cindex @code{--esmf_options}
@item -W @var{wgt_opt} (@code{--wgt_opt}, @code{--weight_options}, @code{--esmf_opt}, @code{--esmf_options}, @code{--tps_opt}, @code{--tempest_options})
@command{ncremap} passes @var{wgt_opt} directly through to the 
weight-generator (currently @acronym{ERWG} or
TempestRemap's @command{GenerateOfflineMap}) (and not to
@command{GenerateOverlapMesh}).
The user-specified contents of @var{wgt_opt}, if any, supercede the
default contents for the weight-generator.
The default option for @acronym{ERWG} is @samp{--ignore_unmapped}).
@command{ncremap} 4.7.7 and later additionally set the @acronym{ERWG}
@samp{--ignore_degenerate} option, though if the run-time @acronym{ERWG}
reports its version is 7.0 (March, 2018) or later. 
This is done to preserve backwards compatibility since, @acronym{ERWG}
7.1.0r and later require @samp{--ignore_degenerate} to successfully
regrid some datasets (e.g., @acronym{CICE}) that previous @acronym{ERWG}
versions handle fine. 
Users of earlier versions of @command{ncremap} that call @acronym{ESMF}
7.1.0r and later can explicitly pass the base @acronym{ERWG} options
with @command{ncremap}'s @samp{--esmf_opt} option:
@example
@verbatim
# Use when NCO <= 4.7.6 and ERWG >= 7.1.0r
ncremap --esmf_opt='--ignore_unmapped --ignore_degenerate' ...
@end verbatim
@end example

The @acronym{ERWG} and TempestRemap documentation shows all available
options. 
For example, to cause @acronym{ERWG} to output to a netCDF4 file,  
pass @samp{-W "--netcdf4"} to @command{ncremap}.

By default, @command{ncremap} runs @command{GenerateOfflineMap} without
any options. 
To cause @command{GenerateOfflineMap} to use a @code{_FillValue} of
@math{-1}, pass @samp{-W '--fillvalue -1.0'} to @command{ncremap}.
Other common options include enforcing monotonicity (which is not the
default in TempestRemap) constraints. 
To guarantee monotonicity in regridding from Finite Volume @acronym{FV} 
to @acronym{FV} maps (e.g., @acronym{MPAS}-to-rectangular), pass 
@samp{-W '-in_np 1'} to @command{ncremap}.
To guarantee monotonicity in regridding from Finite Element @acronym{FE} 
to @acronym{FV} maps, pass @samp{-W '--mono'}.
Common sets of specialized options recommended for TempestRemap are
collected into six boutique algorithms invokable with @samp{--alg_typ}
as described above.

@html
<a name="wgt_cmd"></a> <!-- http://nco.sf.net/nco.html#wgt_cmd -->
@end html
@cindex @code{-w @var{wgt_cmd}}
@cindex @var{wgt_cmd}
@cindex @code{--wgt_cmd}
@cindex @code{--wgt_gnr}
@cindex @code{--weight_command}
@cindex @code{--weight_generator}
@item -w @var{wgt_cmd} (@code{--wgt_cmd}, @code{--weight_command}, @code{--wgt_gnr}, @code{--weight_generator})
Specifies a (possibly extended) command to use to run the
weight-generator when a map-file is not provided.  
This command overrides the default executable executable for the
weight generator, which is @command{ESMF_RegridWeightGen} for
@acronym{ESMF} and @command{GenerateOfflineMap} for TempestRemap.
(There is currently no way to override @command{GenerateOverlapMesh}  
for TempestRemap). 
The @var{wgt_cmd} must accept the same arguments as the default
command. 
Examples include @samp{mpirun -np 24 ESMF_RegridWeightGen},
@samp{mpirun-openmpi-mp -np 16 ESMF_RegridWeightGen}, and other
ways of exploiting parallelism that are system-dependent.
Specifying @var{wgt_cmd} and supplying (with @samp{-m}) a map-file
is not permitted (since the weight-generator would not be used).

@html
<a name="xcl_var"></a> <!-- http://nco.sf.net/nco.html#xcl_var -->
@end html
@cindex @code{--xcl_var}
@cindex @code{--xcl}
@cindex @code{--exclude}
@cindex @code{--exclude_variables}
@item --xcl_var (@code{--xcl_var}, @code{--xcl}, @code{--exclude}, @code{--exclude_variables})
This flag (which takes no argument) changes @var{var_lst},
as set by the @code{--var_lst} option, from an extraction list to an
exclusion list so that variables in @var{var_lst} will not be
processed, and variables not in @var{var_lst} will be processed.
Thus the option @samp{-v @var{var_lst}} must also be present for this
flag to take effect.
Variables explicitly specified for exclusion by
@samp{--xcl --vars=@var{var_lst}[,@dots{}]} need not be present in the
input file.

@html
<a name="xtn_lst"></a> <!-- http://nco.sf.net/nco.html#xtn_lst -->
@end html
@cindex @code{-v @var{xtn_lst}}
@cindex @var{xtn_lst}
@cindex @code{--xtn_lst}
@cindex @code{--xtn_var}
@cindex @code{--var_xtn}
@cindex @code{--extensive}
@cindex @code{--extensive_variables}
@item -x @var{xtn_lst} (@code{--xtn_lst}, @code{--xtn_var}, @code{--var_xtn}, @code{--extensive}, @code{--extensive_variables})
The @samp{-x} option causes @command{ncremap} to treat the variables in
@var{xtn_lst} as @dfn{extensive}, meaning that their value depends on
the gridcell boundaries. 
Support for extensive variables during regridding is nascent.
Currently variables marked as extensive are summed, not regridded.
We are interested in ``real-world'' situations that require regridding
extensive variables, please contact us if you have one.

@end table

@unnumberedsubsec Limitations to @command{ncremap}

@command{ncremap} has two significant limitations to be aware of.
First, for two-dimensional input grids the fields to be regridded must
have latitude and longitude, or, in the case of curvilinear data, the
two equivalent horizontal dimensions, as the final two dimensions in
@var{in_fl}.
Fields with other dimension orders (e.g., @samp{lat,lev,lon}) will not
regrid properly. 
To circumvent this limitation one can employ 
@command{ncpdq} (@pxref{ncpdq netCDF Permute Dimensions Quickly})
to permute the dimensions before (and un-permute them after) regridding.
@command{ncremap} utilizes this method internally for some common
input grids.
For example,
@example
# AIRS Level2 vertical profiles
ncpdq -a StdPressureLev,GeoTrack,GeoXTrack AIRS_L2.hdf AIRS_L2_ncpdq.nc
ncremap -i AIRS_L2_ncpdq.nc -d dst_1x1.nc -O ~/rgr
# MPAS-O fields
ncpdq -a Time,nVertLevels,maxEdges,MaxEdges2,nEdges,nCells mpas.nc mpas_ncpdq.nc
ncremap -R "--rgr col_nm=nCells" -i mpas_ncpdq.nc -m mpas120_to_t62.nc -O ~/rgr
@end example
@noindent
The previous two examples occur so frequently that @command{ncremap} has
been specially equipped to handle @acronym{AIRS} and @acronym{MPAS}
files. 
As of @acronym{NCO} version 4.5.5 (February, 2016), the following
@command{ncremap} commands with the @samp{-P @var{prc_typ}} option
automagically perform all required permutation and renaming necessary: 
@example
# AIRS Level2 vertical profiles
ncremap -P airs -i AIRS_L2.nc -d dst_1x1.nc -O ~/rgr
# MPAS-O/I fields
ncremap -P mpas -i mpas.nc -m mpas120_to_t62.nc -O ~/rgr
@end example
@noindent
The machinery to handle permutations and special options for other
datafiles is relatively easy to extend with new @var{prc_typ} options. 
If you work with common datasets that could benefit from their own
pre-processing options, contact us and we will try to implement them.  

The second limitation is that to perform regridding, @command{ncremap}
must read weights from an on-disk mapfile, and cannot yet compute
weights itself and use them directly from @acronym{RAM}.  
This makes @command{ncremap} an ``offline regridder'' and unnecessarily
slow compared to an ``integrated regridder'' that computes weights and
immediately applies them in @acronym{RAM} without any disk-access.
In practice, the difference is most noticeable when the weights are 
easily computable ``on the fly'', e.g., rectangular-to-rectangular
mappings. 
Otherwise the weight-generation takes much more time than the
weight-application, at which @command{ncremap} is quite fast.
As of @acronym{NCO} @w{version 4.9.0}, released in December, 2019,
regridder supports generation of intersection grids and overlap
weights for all finite volume grid combinations.
However these weights are first stored in an offline mapfile, are not
usable otherwise.

One side-effect of @command{ncremap} being an offline regridder is
that, when necessary, it can generate files to store intermediate
versions of grids, maps, and data. 
These files are named, by default, 
@file{ncremap_tmp_att.nc}@file{$@{unq_sfx@}},
@file{ncremap_tmp_d2f.nc}@file{$@{unq_sfx@}},
@file{ncremap_tmp_grd_dst.nc}@file{$@{unq_sfx@}},
@file{ncremap_tmp_grd_src.nc}@file{$@{unq_sfx@}},
@file{ncremap_tmp_gnr_out.nc}@file{$@{unq_sfx@}},
@file{ncremap_tmp_map_*.nc}@file{$@{unq_sfx@}},
@file{ncremap_tmp_msh_ovr_*.nc}@file{$@{unq_sfx@}}, and
@file{ncremap_tmp_pdq.nc}@file{$@{unq_sfx@}}.
They are placed in @var{drc_out} with the output file(s).
In general, no intermediate grid or map files are generated when the
map-file is provided. 
Intermediate files are always generated when the @samp{-P @var{prm_typ}}
option is invoked. 
By default these files are automatically removed upon successful
completion of the script, unless @command{ncremap} was invoked by
@samp{--unq_sfx=noclean} to explitly override this ``self-cleaning''
behavior.  
Nevertheless, early or unexpected termination of @command{ncremap}
will almost always leave behind a collection of these intermediate
files.
Should intermediate files proliferate and/or annoy you, locate and/or
remove all such files under the current directory with 
@example
find . -name 'ncremap_tmp*'
rm `find . -name 'ncremap_tmp*'`
@end example
@noindent

@noindent
@html
<a name="xmp_ncremap"></a> <!-- http://nco.sf.net/nco.html#xmp_ncremap -->
@end html
EXAMPLES

Regrid input file @file{in.nc} to the spatial grid in file @file{dst.nc}
and write the output to @file{out.nc}: 
@example
ncremap -d dst.nc in.nc out.nc
ncremap -d dst.nc -i in.nc -o out.nc
ncremap -d dst.nc -O regrid in.nc out.nc
ncremap -d dst.nc in.nc regrid/out.nc
ncremap -d dst.nc -O regrid in.nc # output named in.nc
@end example
@noindent
@acronym{NCO} infers the destination spatial grid from @file{dst.nc} by
reading its coordinate variables and @acronym{CF} attributes.
In the first example, @command{ncremap} places the output in
@file{out.nc}.
In the second and third examples, the output file is
@file{regrid/out.nc}. 
In the fourth example, @command{ncremap} places the output in the
specified output directory.
Since no output filename is provided, the output file will be named
@file{regrid/in.nc}.

Generate a mapfile with @command{ncremap} and store it for later re-use. 
A pre-computed mapfile (supplied with @samp{-m @var{map_fl}}) eliminates 
time-consuming weight-generation, and thus considerably reduces
wallclock time: 
@example
ncremap -m map.nc in.nc out.nc
ncremap -m map.nc -I drc_in -O regrid
@end example

As of @acronym{NCO} version 4.7.2 (January, 2018), @command{ncremap}
supports ``canonical'' argument ordering of command line arguments most
frequently desired for one-off regridding, where a single input and
output filename are supplied as command-line positional arguments
without switches, pipes, or redirection: 
@example
ncremap -m map.nc in.nc out.nc # Requires 4.7.2+
ncremap -m map.nc -i in.nc -o out.nc
ncremap -m map.nc -o out.nc in.nc
ncremap -m map.nc -O out_dir in1.nc in2.nc
ncremap -m map.nc -o out.nc < in.nc
ls in.nc | ncremap -m map.nc -o out.nc
@end example
These are all equivalent methods, but the canonical ordering shown in
the first example only works in @acronym{NCO} version 4.7.2 and later.

@command{ncremap} annotates the gridfiles and mapfiles that it creates
with helpful metadata containing the full provenance of the command.
Consequently, @command{ncremap} is a sensible tool for generating
mapfiles for later use. 
To generate a mapfile with the specified (non-default) name
@file{map.nc}, and then regrid a single file, 
@example
ncremap -d dst.nc -m map.nc in.nc out.nc
@end example

To test the remapping workflow, regrid only one or a few variables
instead of the entire file:  
@example
ncremap -v T,Q,FSNT -m map.nc in.nc out.nc
@end example
Regridding generally scales linearly with the size of data to be
regridded, so eliminating unnecessary variables produces a snappier
response.  

Regrid multiple input files with a single mapfile @file{map.nc} 
and write the output to the @file{regrid} directory:
@example
ncremap -m map.nc -I drc_in -O regrid
ls drc_in/*.nc | ncremap -m map.nc -O regrid
@end example
@noindent
The three ways @acronym{NCO} obtains the destination spatial grid are,
in decreasing order of precedence, 
from @var{map_fl} (specified with @samp{-m}), 
from @var{grd_dst} (specified with @samp{-g}), and
(inferred) from @var{dst_fl} (specified with @samp{-d}).
In the first example all likely data files from @var{drc_in} are
regridded using the same specified mapfile, @var{map_fl} = @file{map.nc}.
Each output file is written to @var{drc_out} = @file{regrid} with the
same name as the corresponding input file.
The second example obtains the input file list from standard input,
and uses the mapfile and output directory as before.

If multiple input files are on the same grid, yet the mapfile does not
exist in advance, one can still regrid all input files without incurring
the time-penalty of generating multiple mapfiles. 
To do so, provide the (known-in-advance) source gridfile or toggle the
@samp{-M} switch: 
@example
ncremap -M -I drc_in -d dst.nc -O regrid
ls drc_in/*.nc | ncremap -M -d dst.nc -O regrid
ncremap -I drc_in -s grd_src.nc -d dst.nc -O regrid
ls drc_in/*.nc | ncremap -s grd_src.nc -d dst.nc -O regrid
ncremap -I drc_in -s grd_src.nc -g grd_dst.nc -O regrid
ls drc_in/*.nc | ncremap -s grd_src.nc -g grd_dst.nc -O regrid
@end example
@noindent
The first two examples explicitly toggle the multi-map-generation switch
(with @samp{-M}), so that @command{ncremap} refrains from generating
multiple mapfiles. 
In this case the source grid is inferred from the first input file,
the destination grid is inferred from @file{dst.nc}, and
@command{ncremap} uses @acronym{ERWG} to generate a single mapfile and
uses that to regrid every input file.
The next four examples are variants on this theme.
In these cases, the user provides (with @samp{-s grd_src.nc}) the source  
gridfile, which will be used directly instead of being inferred.
Any of these styles works well when each input file is known in advance
to be on the same grid, e.g., model data for successive time periods in
a simulation. 

The most powerful, time-consuming (yet simultaneously time-saving!)
feature of @command{ncremap} is its ability to regrid multiple input
files on unique grids.
Both input and output can be on any @acronym{CRUD} grid.
@example
ncremap -I drc_in -d dst.nc -O regrid
ls drc_in/*.nc | ncremap -d dst.nc -O regrid
ncremap -I drc_in -g grd_dst.nc -O regrid
ls drc_in/*.nc | ncremap -g grd_dst.nc -O regrid
@end example
@noindent
There is no pre-supplied @var{map_fl} or @var{grd_src} in these
examples, so @command{ncremap} first infers the output grid from
@file{dst.nc} (first two examples), or directly uses the supplied
gridfile @file{grd_dst} (second two examples), and calls @acronym{ERWG}
to generate a new mapfile for each input file, whose grid it infers.
This is necessary when each input file is on a unique grid, e.g.,
swath-like data from satellite observations or models with time-varying
grids. 
These examples require remarkably little input, since @command{ncremap}
automates most of the work.

Finally, @command{ncremap} uses the parallelization options 
@samp{-p @var{par_typ}} and @samp{-j @var{job_nbr}} to help manage
high-volume workflow. 
On a single node such as a local workstation, use Background mode
to regrid multiple files in parallel
@example
ls drc_in/*.nc | ncremap -p bck -d dst.nc -O regrid
ls drc_in/*.nc | ncremap -p bck -j 4 -d dst.nc -O regrid
@end example
@noindent
Both examples will eventually regrid all input files.
The first example regrids two at a time because two is the default
batch size @command{ncremap} employs.
The second example regrids files in batches of four at a time.
Increasing @var{job_nbr} will increase throughput so long as the node
is not I/O-limited.

Multi-node clusters can exploit inter-node parallelism in
@acronym{MPI}-mode:
@example
qsub -I -A CLI115 -V -l nodes=4 -l walltime=03:00:00 -N ncremap
ls drc_in/*.nc | ncremap -p mpi -j 4 -d dst.nc -O regrid
@end example
@noindent
This example shows a typical request for four compute nodes.
After receiving the login prompt from the interactive master node,
execute the @command{ncremap} command with @samp{-p mpi}.
@command{ncremap} will send regridding jobs in round-robin fashion
to all available compute nodes until all jobs finish.
It does this by internally prepending an @acronym{MPI} execution
command, like @samp{mpirun -H @var{node_name} -npernode 1 -n 1},
to the usual regridding command.
@acronym{MPI}-mode typically has excellent scaling because most
nodes have independent access to hard storage.
This is the easiest way to speed your cumbersome job by factors
of ten or more.
As mentioned above under Limitations, parallelism is currently only
supported when all regridding uses the same map-file.

@page
@html
<a name="ncrename"></a> <!-- http://nco.sf.net/nco.html#ncrename -->
@end html
@node ncrename netCDF Renamer, ncwa netCDF Weighted Averager, ncremap netCDF Remapper, Reference Manual
@section @command{ncrename} netCDF Renamer
@cindex renaming variables
@cindex renaming groups
@cindex renaming dimensions
@cindex renaming attributes
@cindex variable names
@cindex dimension names
@cindex attribute names
@cindex group names
@findex ncrename

@noindent
SYNTAX
@example
ncrename [-a @var{old_name},@var{new_name}] [-a @dots{}] [-D @var{dbg}] 
[-d @var{old_name},@var{new_name}] [-d @dots{}] [-g @var{old_name},@var{new_name}] [-g @dots{}] 
[--glb ...] [-H] [-h] [--hdf] [--hdr_pad @var{nbr}] [--hpss] 
[-l @var{path}] [-O] [-o @var{output-file}] [-p @var{path}] [-R] [-r] 
[-v @var{old_name},@var{new_name}] [-v @dots{}]
@var{input-file} [[@var{output-file}]]
@end example
 
@noindent
DESCRIPTION

@cindex @kbd{.}
@command{ncrename} renames netCDF dimensions, variables, attributes, and
groups. 
Each object that has a name in the list of old names is renamed using
the corresponding name in the list of new names. 
All the new names must be unique. 
Every old name must exist in the input file, unless the old name is
preceded by the period (or ``dot'') character @samp{.}. 
The validity of @var{old_name} is not checked prior to the renaming. 
Thus, if @var{old_name} is specified without the @samp{.} prefix that
indicates the presence of @var{old_name} is optional, and @var{old_name}  
is not present in @var{input-file}, then @command{ncrename} will abort.  
The @var{new_name} should never be prefixed by a @samp{.} (or else the
period will be included as part of the new name).
As of @acronym{NCO} version 4.4.6 (released October, 2014), the
@var{old_name} and @var{new_name} arguments may include (or be, for
groups) partial or full group paths. 
The OPTIONS and EXAMPLES show how to select specific variables
whose attributes are to be renamed.

@html
<a name="bug_nc4_rename"></a> <!-- http://nco.sf.net/nco.html#bug_nc4_rename -->
@end html
@cartouche
Caveat lector: Unforunately from 2007--present (August, 2023) the
netCDF library (versions 4.0.0--4.9.3) contains bugs or limitations
that sometimes prevent @acronym{NCO} from correctly renaming coordinate
variables, dimensions, and groups in netCDF4 files. 
(To our knowledge the netCDF library calls for renaming always work
well on netCDF3 files so one workaround to many netCDF4 issues is
convert to netCDF3, rename, then convert back). 
To understand the renaming limitations associated with particular
netCDF versions, read the @command{ncrename} documentation below in 
its entirety. 
@end cartouche

Although @command{ncrename} supports full pathnames for both
@var{old_name} and @var{new_name}, this is really ``window dressing''.
The full-path to @var{new_name} must be identical to the full-path to 
@var{old_name} in all classes of objects (attributes, variables,
dimensions, or groups).
In other words, @command{ncrename} can change only the local names
of objects, it cannot change the location of the object in the group
hierarchy within the file.
Hence using a full-path in @var{new_name} is redundant. 
The object name is the terminal path component of @var{new_name} and
this object must already exist in the group specified by the 
@var{old_name} path.

@cindex data safety
@cindex safeguards
@cindex temporary output files
@command{ncrename} is an exception to the normal @acronym{NCO} rule that
the user will be interactively prompted before an existing file is
changed, and that a temporary copy of an output file is constructed
during the operation. 
If only @var{input-file} is specified, then @command{ncrename} changes
object names in the @var{input-file} in place without prompting and
without creating a temporary copy of @code{input-file}.
This is because the renaming operation is considered reversible if the
user makes a mistake.
The @var{new_name} can easily be changed back to @var{old_name} by using 
@command{ncrename} one more time.

Note that renaming a dimension to the name of a dependent variable can
be used to invert the relationship between an independent coordinate
variable and a dependent variable. 
In this case, the named dependent variable must be one-dimensional and
should have no missing values. 
Such a variable will become a coordinate variable.

@cindex performance
@cindex operator speed
@cindex speed
@cindex execution time
According to the @cite{netCDF User Guide}, renaming objects in netCDF
files does not incur the penalty of recopying the entire file when the
@var{new_name} is shorter than the @var{old_name}. 
Thus @command{ncrename} may run much faster (at least on netCDF3 files)
if judicious use of header padding (@pxref{Metadata Optimization}) was
made when producing the @var{input-file}.
Similarly, using the @samp{--hdr_pad} option with @command{ncrename}
helps ensure that future metadata changes to @var{output-file} occur
as swifly as possible.

@noindent
OPTIONS

@table @samp
@item -a @var{old_name},@var{new_name}
Attribute renaming. 
The old and new names of the attribute are specified with @samp{-a}
(or @samp{--attribute}) by the associated @var{old_name} and
@var{new_name} values.  
@cindex @code{global} attribute
@cindex global attributes
@cindex attributes, global
Global attributes are treated no differently than variable attributes.
This option may be specified more than once.
As mentioned above, all occurrences of the attribute of a given name
will be renamed unless the @samp{.} form is used, with one exception.
To change the attribute name for a particular variable, specify 
the @var{old_name} in the format @var{old_var_name@@old_att_name}.
The @samp{@@} symbol delimits the variable from the attribute name.
If the attribute is uniquely named (no other variables contain the
attribute) then the @var{old_var_name@@old_att_name} syntax is
redundant. 
The @var{old_var_name} variable names @code{global} and @code{group}
have special significance.
They indicate that @var{old_att_nm} should only be renamed where it
occurs as a global (i.e., root group) metadata attribute (for
@code{global}), or (for @code{group}) as @emph{any} group attribute, and
not where it occurs as a variable attribute.
The @var{var_name@@att_name} syntax is accepted, though not required,
for the @var{new_name}.

@item -d @var{old_name},@var{new_name}
Dimension renaming. 
The old and new names of the dimension are specified with @samp{-d}
(or @samp{--dmn}, @samp{--dimension}) by the associated @var{old_name}
and @var{new_name} values.  
This option may be specified more than once.
 
@item -g @var{old_name},@var{new_name}
Group renaming. 
The old and new names of the group are specified with @samp{-g}
(or @samp{--grp}, @samp{--group}) by the associated @var{old_name}
and @var{new_name} values.  
This option may be specified more than once.
This functionality is only available in @acronym{NCO} version 4.3.7
(October, 2013) or later, and only when built on netCDF library version
4.3.1-rc1 (August, 2013) or later. 
 
@item -v @var{old_name},@var{new_name}
Variable renaming. 
The old and new names of the variable are specified with @samp{-v}
(or @samp{--variable}) by the associated @var{old_name} and
@var{new_name} values.  
This option may be specified more than once.
@end table

@noindent
@html
<a name="xmp_ncrename"></a> <!-- http://nco.sf.net/nco.html#xmp_ncrename -->
@end html
EXAMPLES

Rename the variable @code{p} to @code{pressure} and @code{t} to
@code{temperature} in netCDF @file{in.nc}. 
In this case @code{p} must exist in the input file (or
@command{ncrename} will abort), but the presence of @code{t} is optional:
@example
ncrename -v p,pressure -v .t,temperature in.nc
@end example

Rename the attribute @code{long_name} to @code{largo_nombre} in the
variable @code{u}, and no other variables in netCDF @file{in.nc}. 
@example
@verbatim
ncrename -a u@long_name,largo_nombre in.nc
@end verbatim
@end example
 
Rename the group @code{g8} to @code{g20} in netCDF4 file
@file{in_grp.nc}:   
@example
ncrename -g g8,g20 in_grp.nc
@end example
 
Rename the variable @code{/g1/lon} to @code{longitude} in netCDF4
@file{in_grp.nc}:
@example
ncrename -v /g1/lon,longitude in_grp.nc
ncrename -v /g1/lon,/g1/longitude in_grp.nc # Alternate
@end example
 
@html
<a name="ncrename_crd"></a> <!-- http://nco.sf.net/nco.html#ncrename_crd -->
@end html
@cindex coordinate variables
@command{ncrename} does not automatically attach dimensions to variables of
the same name.
This is done to make renaming an easy way to change whether a variable
is a coordinate.
If you want to rename a coordinate variable so that it remains a
coordinate variable, you must separately rename both the dimension and
the variable: 
@example
ncrename -d lon,longitude -v lon,longitude in.nc
@end example
Unfortunately, the netCDF4 library had a longstanding bug (all versions
until 4.3.1-rc5 released in December, 2013) that crashed @acronym{NCO}
when performing this operation. 
Simultaneously renaming variables and dimensions in netCDF4 files with
earlier versions of netCDF is impossible; it must instead be done in two 
separate @command{ncrename} invocations (e.g., first rename the
variable, then rename the dimension) to avoid triggering the libary
bug. 

A related bug causes unintended side-effects with @command{ncrename} 
also built with all versions of the netCDF4 library until 4.3.1-rc5
released in December, 2013):
This bug caused renaming @emph{either} a dimension @emph{or} its
associated coordinate variable (not both, which would fail as above) in
a netCDF4 file to inadvertently rename both:
@example
# Demonstrate bug in netCDF4/HDF5 library prior to netCDF-4.3.1-rc5
ncks -O -h -m -M -4 -v lat_T42 ~/nco/data/in.nc ~/foo.nc
ncrename -O -v lat_T42,lat ~/foo.nc ~/foo2.nc # Also renames dimension
ncrename -O -d lat_T42,lat ~/foo.nc ~/foo2.nc # Also renames variable
@end example
To avoid this faulty behavior, either build @acronym{NCO} with netCDF
version 4.3.1-rc5 or later, or convert the file to netCDF3 first,
then rename as intended, then convert back.
Unforunately while this bug and the related coordinate renaming bug 
were fixed in 4.3.1-rc5 (released in December, 2013), a new and related
bug was discovered in October 2014.

Another netCDF4 bug that causes unintended side-effects with
@command{ncrename} affects (at least) versions 4.3.1--4.3.2 and all 
snapshots of the netCDF4 library until January, 2015.
This bug (fixed in 4.3.3 in February, 2015) corrupts values or renamed
netCDF4 coordinate variables (i.e., variables with underlying dimensions
of the same name) and other (non-coordinate) variables that include an
underlying dimension that was renamed. 
In other words, @emph{renaming} coordinate variables and dimensions
succeeds yet it corrupts the values contained by the affected array
variables. 
This bug corrupts affected variables by replacing their values with the
default @code{_FillValue} for that variable's type: 
@example
# Demonstrate bug in netCDF4 libraries prior to version 4.3.3
ncks -O -4 -C -M -v lat ~/nco/data/in.nc ~/bug.nc
ncrename -O -v lat,tal ~/bug.nc ~/foo.nc # Broken until netCDF-4.3.3
ncrename -O -d lat,tal ~/bug.nc ~/foo.nc # Broken until netCDF-4.3.3
ncrename -O -d lat,tal -v lat,tal ~/bug.nc ~/foo.nc # Broken too
ncks ~/foo.nc
@end example
To avoid this faulty behavior, either build @acronym{NCO} with netCDF 
version 4.3.3 or later, or convert the file to netCDF3 first, then
rename as intended, then convert back.  
This bug does not affect renaming of groups or of attributes.

Yet another netCDF4 bug that causes unintended side-effects with
@command{ncrename} affects only snapshots from January--February, 2015,
and released version 4.3.3 (February, 2015).
It was fixed in (and was the reason for releasing) netCDF version
4.3.3.1 (March, 2015). 
This bug causes renamed attributes of coordinate variables in netCDF4 
to files to disappear: 
@example
@verbatim
# Demonstrate bug in netCDF4 library version 4.3.3
ncrename -O -h -a /g1/lon@units,new_units ~/nco/data/in_grp.nc ~/foo.nc 
ncks -v /g1/lon ~/foo.nc # Shows units and new_units are both gone
@end verbatim
@end example

Clearly, renaming coordinates in netCDF4 files is non-trivial.
The penultimate chapter in this saga is a netCDF4 bug discovered in
September, 2015, and present in versions 4.3.3.1 (and possibly earlier 
versions too) and later.
As of this writing (February, 2018), this bug is still present in netCDF4
version 4.6.0.1-development. 
This bug causes @command{ncrename} to create corrupted output files
when attempting to rename two or more dimensions simultaneously.
The workaround is to rename the dimensions sequentially, in two separate  
@command{ncrename} calls.
@example
@verbatim
# Demonstrate bug in netCDF4 library versions 4.3.3.1--4.6.1+
ncrename -O -d lev,z -d lat,y -d lon,x ~/nco/data/in_grp.nc ~/foo.nc # Completes but file is unreadable
ncks -v one ~/foo.nc # File is unreadable (multiple dimensions with same ID?)
@end verbatim
@end example

A new netCDF4 renaming bug was discovered in March, 2017.
It is present in versions 4.4.1--4.6.0 (and possibly earlier versions).
This bug was fixed in netCDF4 version 4.6.1 (Yay Ed!). 
This bug caused @command{ncrename} to fail to rename a variable when the
result would become a coordinate.
@example
@verbatim
# Demonstrate bug in netCDF4 library versions 4.4.1--4.6.0
ncrename -O -v non_coord,coord ~/nco/data/in_grp.nc ~/foo.nc # Fails (HDF error)
@end verbatim
@end example
The fix is to upgrade to netCDF version 4.6.1.
The workaround is to convert to netCDF3, then rename, then convert back
to netCDF4.

A potentially new netCDF4 bug was discovered in November, 2017 and is
now fixed.
It is present in versions 4.4.1.1--4.6.0 (and possibly earlier versions too).
This bug causes @command{ncrename} to fail to rename a variable when the
result would become a coordinate.
Oddly this issue shows that simultaneously renaming a dimension and
coordinate can succeed (in contrast to a bug described above), and that
separating that into two steps can fail.
@example
@verbatim
# Demonstrate bug in netCDF4 library versions 4.4.1--4.6.0
# 20171107: https://github.com/Unidata/netcdf-c/issues/597
# Create test dataset
ncks -O -C -v lon ~/nco/data/in_grp.nc ~/in_grp.nc
ncks -O -x -g g1,g2 ~/in_grp.nc ~/in_grp.nc
# Rename dimension then variable
ncrename -d lon,longitude ~/in_grp.nc # works
ncrename -v lon,longitude ~/in_grp.nc # borken "HDF error"
# Rename variable then dimension
ncrename -v lon,longitude ~/in_grp.nc # works
ncrename -d lon,longitude ~/in_grp.nc # borken "nc4_reform_coord_var: Assertion `dim_datasetid > 0' failed."
# Oddly renaming both simultaneously works:
ncrename -d lon,longitude -v lon,longitude ~/in_grp.nc # works
@end verbatim
@end example
The fix is to upgrade to netCDF version 4.6.1.
The workaround is to convert to netCDF3, then rename, then convert back
to netCDF4.

A new netCDF3 bug was discovered in April, 2018 and is now fixed.
It is present in netCDF versions 4.4.1--4.6.0 (and possibly earlier versions too).
This bug caused @command{ncrename} to fail to rename many coordinates
and dimensions simultaneously.
This bug affects netCDF3 @code{64BIT_OFFSET} files and possibly other
formats as well.
As such it is the first and so far only bug we have identified that
affects netCDF3 files.
@example
@verbatim
cp /glade/scratch/gus/GFDL/exp/CM3_test/pp/0001/0001.land_month_crop.AllD.nc ~/correa_in.nc   
ncrename -O -d grid_xt,lon -d grid_yt,lat -v grid_xt,lon -v grid_yt,lat \
         -v grid_xt_bnds,lon_bnds -v grid_yt_bnds,lat_bnds ~/correa_in.nc ~/correa_out.nc 
@end verbatim
@end example
The fix is to upgrade to netCDF version 4.6.1.

@cindex global attributes
@cindex group attributes
@cindex attributes, global
@cindex @code{_FillValue}
@cindex @code{missing_value}
Create netCDF @file{out.nc} identical to @file{in.nc} except the
attribute @code{_FillValue} is changed to @code{missing_value}, 
the attribute @code{units} is changed to @code{CGS_units} (but only in
those variables which possess it), the attribute @code{hieght} is
changed to @code{height} in the variable @code{tpt}, and in the
variable @code{prs_sfc}, if it exists.
@example
@verbatim
ncrename -a _FillValue,missing_value -a .units,CGS_units \
  -a tpt@hieght,height -a prs_sfc@.hieght,height in.nc out.nc 
@end verbatim
@end example
The presence and absence of the @samp{.} and @samp{@@} features
cause this command to execute successfully only if a number of 
conditions are met. 
All variables @emph{must} have a @code{_FillValue} attribute @emph{and} 
@code{_FillValue} must also be a global attribute.
The @code{units} attribute, on the other hand, will be renamed to
@code{CGS_units} wherever it is found but need not be present in
the file at all (either as a global or a variable attribute).
The variable @code{tpt} must contain the @code{hieght} attribute.
The variable @code{prs_sfc} need not exist, and need not contain the
@code{hieght} attribute.

Rename the global or group attribute @code{Convention} to
@code{Conventions}
@example
@verbatim
ncrename -a Convention,Conventions  in.nc # Variable and group atts.
ncrename -a .Convention,Conventions in.nc # Variable and group atts.
ncrename -a @Convention,Conventions  in.nc # Group atts. only
ncrename -a @.Convention,Conventions in.nc # Group atts. only
ncrename -a global@Convention,Conventions   in.nc # Group atts. only
ncrename -a .global@.Convention,Conventions in.nc # Group atts. only
ncrename -a global@Convention,Conventions   in.nc # Global atts. only
ncrename -a .global@.Convention,Conventions in.nc # Global atts. only
@end verbatim
@end example
The examples without the @code{@@} character attempt to change the
attribute name in both Global or Group and variable attributes.
The examples with the @code{@@} character attempt to change only 
global and group @code{Convention} attributes, and leave unchanged any
@code{Convention} attributes attached directly to variables.
Attributes prefixed with a period (@code{.Convention}) need not be
present. 
Attributes not prefixed with a period (@code{Convention}) must be
present. 
Variables prefixed with a period (@code{.} or @code{.global}) need not 
be present.  
Variables not prefixed with a period (@code{global}) must be present.  

@page
@html
<a name="ncwa"></a> <!-- http://nco.sf.net/nco.html#ncwa -->
@end html
@node ncwa netCDF Weighted Averager,  , ncrename netCDF Renamer, Reference Manual
@section @command{ncwa} netCDF Weighted Averager
@cindex averaging data
@cindex weights
@cindex weighted average
@cindex masked average
@cindex broadcasting variables
@findex ncwa

@noindent
SYNTAX
@example
ncwa [-3] [-4] [-5] [-6] [-7] [-A] [-a @var{dim}[,@dots{}]]
[-B @var{mask_cond}] [-b] [-C] [-c] [--cmp @var{cmp_sng}]
[--cnk_byt @var{sz_byt}] [--cnk_csh @var{sz_byt}] [--cnk_dmn @var{nm},@var{sz_lmn}]
[--cnk_map @var{map}] [--cnk_min @var{sz_byt}] [--cnk_plc @var{plc}] [--cnk_scl @var{sz_lmn}]
[-D @var{dbg}] [-d @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]] [-F] [--fl_fmt @var{fl_fmt}]
[-G @var{gpe_dsc}] [-g @var{grp}[,@dots{}]] [--glb ...] [-H] [-h] [--hdr_pad @var{nbr}] [--hpss] [-I]
[-L @var{dfl_lvl}] [-l @var{path}] [-M @var{mask_val}] [-m @var{mask_var}] [-N] 
[--no_cll_msr] [--no_cll_mth] [--no_frm_trm] [--no_tmp_fl] 
[-O] [-o @var{output-file}] [-p @var{path}] [--ppc ...] [-R] [-r] [--ram_all] [--rth_dbl|flt]
[-T @var{mask_comp}] [-t @var{thr_nbr}] [--unn] [-v @var{var}[,@dots{}]] [-w @var{weight}]
[-X ...] [-x] [-y @var{op_typ}]
@var{input-file} [@var{output-file}]
@end example

@noindent
DESCRIPTION

@command{ncwa} performs statistics (including, but not limited to,
averages) on variables in a single file over arbitrary dimensions, with
options to specify weights, masks, and normalization.
@xref{Statistics vs Concatenation}, for a description of the
distinctions between the various statistics tools and concatenators. 
The default behavior of @command{ncwa} is to arithmetically average
every numerical variable over all dimensions and to produce a scalar 
result for each. 

@cindex degenerate dimension
Averaged dimensions are, by default, eliminated as dimensions.
Their corresponding coordinates, if any, are output as scalar
variables. 
The @samp{-b} switch (and its long option equivalents @samp{--rdd} and 
@samp{--retain-degenerate-dimensions}) causes @command{ncwa} to retain
averaged dimensions as degenerate (@w{size 1}) dimensions.
This maintains the association between a dimension (or coordinate) and
variables after averaging and simplifies, for instance, later
concatenation along the degenerate dimension. 

To average variables over only a subset of their dimensions, specify
these dimensions in a comma-separated list following @samp{-a}, e.g.,
@samp{-a time,lat,lon}. 
@cindex arithmetic operators
@cindex hyperslab
@cindex @code{-d @var{dim},[@var{min}][,[@var{max}]]}
As with all arithmetic operators, the operation may be restricted to
an arbitrary hyperslab by employing the @samp{-d} option
(@pxref{Hyperslabs}). 
@command{ncwa} also handles values matching the variable's
@code{_FillValue} attribute correctly. 
Moreover, @command{ncwa} understands how to manipulate user-specified
weights, masks, and normalization options.
With these options, @command{ncwa} can compute sophisticated averages
(and integrals) from the command line. 

@html
<a name="-w"></a> <!-- http://nco.sf.net/nco.html#-w -->
<a name="wgt"></a> <!-- http://nco.sf.net/nco.html#wgt -->
@end html
@cindex @code{-w @var{weight}}
@cindex @code{--weight @var{weight}}
@cindex @code{--wgt_var @var{weight}}
@cindex @code{-m @var{mask_var}}
@cindex @code{--mask-variable @var{mask_var}}
@cindex @code{--mask_variable @var{mask_var}}
@cindex @code{--msk_nm @var{mask_var}}
@cindex @code{--msk_var @var{mask_var}}
@cindex @code{-B @var{mask_cond}}
@cindex @code{--msk_cnd @var{mask_cond}}
@cindex @code{--mask_condition @var{mask_cond}}
@var{mask_var} and @var{weight}, if specified, are broadcast to conform
to the variables being averaged. 
@cindex rank
The rank of variables is reduced by the number of dimensions which they
are averaged over.  
Thus arrays which are one dimensional in the @var{input-file} and are
averaged by @command{ncwa} appear in the @var{output-file} as scalars.
This allows the user to infer which dimensions may have been averaged.
Note that that it is impossible for @command{ncwa} to make make a
@var{weight} or @var{mask_var} of rank @var{W} conform to a @var{var} of
rank @var{V} if @var{W > V}.
This situation often arises when coordinate variables (which, by
definition, are one dimensional) are weighted and averaged.
@command{ncwa} assumes you know this is impossible and so @command{ncwa}
does not attempt to broadcast @var{weight} or @var{mask_var} to conform
to @var{var} in this case, nor does @command{ncwa} print a warning
message telling you this, because it is so common.  
Specifying @var{dbg > 2} does cause @command{ncwa} to emit warnings in
these situations, however.

Non-coordinate variables are always masked and weighted if specified.
Coordinate variables, however, may be treated specially.
By default, an averaged coordinate variable, e.g., @code{latitude},
appears in @var{output-file} averaged the same way as any other variable 
containing an averaged dimension.
In other words, by default @command{ncwa} weights and masks
coordinate variables like all other variables.  
This design decision was intended to be helpful but for some
applications it may be preferable not to weight or mask coordinate
variables just like all other variables.   
Consider the following arguments to @command{ncwa}: 
@code{-a latitude -w lat_wgt -d latitude,0.,90.} where @code{lat_wgt} is
a weight in the @code{latitude} dimension.
Since, by default @command{ncwa} weights coordinate variables, the
value of @code{latitude} in the @var{output-file} depends on the weights 
in @var{lat_wgt} and is not likely to @w{be 45.0}, the midpoint latitude
of the hyperslab.
@cindex coordinate variable
@cindex @code{-I}
Option @samp{-I} overrides this default behavior and causes
@command{ncwa} not to weight or mask coordinate variables
@footnote{The default behavior of (@samp{-I}) changed on
19981201---before this date the default was not to weight or mask
coordinate variables.}.
In the above case, this causes the value of @code{latitude} in the
@var{output-file} to @w{be 45.0}, an appealing result.
Thus, @samp{-I} specifies simple arithmetic averages for the coordinate
variables. 
In the case of latitude, @samp{-I} specifies that you prefer to archive
the arithmetic mean latitude of the averaged hyperslabs rather than the 
area-weighted mean latitude.
@footnote{If @code{lat_wgt} contains Gaussian weights then the value of 
@code{latitude} in the @var{output-file} will be the area-weighted
centroid of the hyperslab. 
For the example given, this is about @w{30 degrees.}}.  

@cindex average
@cindex operation types
As explained in @xref{Operation Types}, @command{ncwa} 
@emph{always averages} coordinate variables regardless of the arithmetic
operation type performed on the non-coordinate variables. 
This is independent of the setting of the @samp{-I} option.
The mathematical definition of operations involving rank reduction 
is given above (@pxref{Operation Types}).

@menu
* Mask condition::
* Normalization and Integration::
@end menu

@html
<a name="mask"></a> <!-- http://nco.sf.net/nco.html#mask -->
<a name="msk"></a> <!-- http://nco.sf.net/nco.html#msk -->
@end html
@node Mask condition, Normalization and Integration, ncwa netCDF Weighted Averager, ncwa netCDF Weighted Averager
@subsection Mask condition
@cindex mask condition
@cindex truth condition
@tex
Each $\xxx_{\idx}$ also has an associated masking
weight~$\mskflg_{\idx}$ whose value is~0 or~1 (false or true).
The value of~$\mskflg_{\idx}$ is always~1 unless a @var{mask\_var} is
specified (with 
@samp{-m}).
As noted above, @var{mask\_var} is broadcast, if possible, to conform 
to the variable being averaged.  
In this case, the value of~$\mskflg_{\idx}$ depends on the 
@dfn{mask condition} also known as the @dfn{truth condition}.
As expected, $\mskflg_{\idx} = 1$ when the mask condition is
@dfn{true} and $\mskflg_{\idx} = 0$ otherwise.   
@end tex

@cindex @code{--op_rlt @var{mask_comp}}
@cindex @code{--mask_comparator @var{mask_comp}}
@cindex @code{--msk_cmp_typ @var{mask_comp}}
@cindex @code{--msk_cnd_sng @var{mask_cond}}
@cindex @code{--mask_condition @var{mask_cond}}
@cindex @code{-B @var{mask_cond}}
The mask condition has the syntax @math{@var{mask_var}}
@math{@var{mask_comp}} @math{@var{mask_val}}. 
The preferred method to specify the mask condition is in one string with  
the @samp{-B} or @samp{--mask_condition} switches.
The older method is to use the three switches @samp{-m}, @samp{-T}, and
@samp{-M} to specify the @var{mask_var}, @var{mask_comp}, and 
@var{mask_val}, respectively.  
@footnote{The three switches @samp{-m}, @samp{-T}, and @samp{-M} are
maintained for backward compatibility and may be deprecated in the
future.
It is safest to write scripts using @samp{--mask_condition}.}.
The @var{mask_condition} string is automatically parsed into its three
constituents @var{mask_var}, @var{mask_comp}, and @var{mask_val}.

@cindex comparator
Here @var{mask_var} is the name of the masking variable (specified with 
@samp{-m}, @samp{--mask-variable}, @samp{--mask_variable},
@samp{--msk_nm}, or @samp{--msk_var}).  
The truth @var{mask_comp} argument (specified with @samp{-T},
@samp{--mask_comparator}, @samp{--msk_cmp_typ}, or @samp{--op_rlt} may 
be any one of the six arithmetic comparators: @kbd{eq}, @kbd{ne},
@kbd{gt}, @kbd{lt}, @kbd{ge}, @kbd{le}. 
@set flg
@tex
These are the Fortran-style character abbreviations for the logical 
comparisons $=$, $\neq$, $>$, $<$, $\ge$, $\le$. 
@clear flg
@end tex
@ifinfo
These are the Fortran-style character abbreviations for the logical 
comparisons @math{==}, @math{!=}, @math{>}, @math{<}, @math{>=},
@math{<=}. 
@clear flg
@end ifinfo
@ifset flg
@c texi2html does not like @math{}
These are the Fortran-style character abbreviations for the logical 
comparisons @kbd{==}, @kbd{!=}, @kbd{>}, @kbd{<}, @kbd{>=},
@clear flg
@end ifset
The mask comparator defaults to @kbd{eq} (equality).
@cindex @code{--mask-value @var{mask_val}}
@cindex @code{--mask_value @var{mask_val}}
@cindex @code{--msk_val @var{mask_val}}
The @var{mask_val} argument to @samp{-M} (or @samp{--mask-value}, or
@samp{--msk_val}) is the right hand side of the
@dfn{mask condition}.
Thus for the @var{i}'th element of the hyperslab to be averaged,
the mask condition is 
@set flg
@tex
{\it mask$_{\idx}$ mask\_comp mask\_val}.
@clear flg
@end tex
@ifinfo
@math{mask(i)} @var{mask_comp} @var{mask_val}.
@clear flg
@end ifinfo
@ifset flg
@c texi2html does not like @math{}
@var{mask}(@var{i}) @var{mask_comp} @var{mask_val}.
@clear flg
@end ifset

@tex
Each~$\xxx_{\idx}$ is also associated with an additional
weight~$\wgt_{\idx}$ whose value may be user-specified.
The value of~$\wgt_{\idx}$ is identically~1 unless the user specifies a 
weighting variable @var{weight} (with @samp{-w}, @samp{--weight}, or
@samp{--wgt\_var}). 
In this case, the value of~$\wgt_{\idx}$ is determined by the
@var{weight} variable in the @var{input-file}. 
As noted above, @var{weight} is broadcast, if possible, to conform 
to the variable being averaged.  

$\tllnbr$ is the number of input elements $\xxx_{\idx}$ which actually
contribute to output element $\xxx_{\jjj}$.
$\tllnbr$ is also known as the @dfn{tally} and is defined as 
$$
\tllnbr = \sum_{\idx=1}^{\idx=\lmnnbr} \mssflg_{\idx} \mskflg_{\idx} 
$$
$\tllnbr$ is identical to the denominator of the generic averaging
expression except for the omission of the weight $\wgt_{\idx}$.
Thus $\tllnbr = \lmnnbr$ whenever no input points are missing values or
are masked.  
Whether an element contributes to the output, and thus increments
$\tllnbr$ by one, has more to do with the above two criteria (missing
value and masking) than with the numeric value of the element per se.
For example, $\xxx_{\idx}=0.0$ does contribute to $\xxx_{\jjj}$
(assuming the @code{_FillValue} attribute is not~0.0 and location
$\idx$ is not masked). 
The value $\xxx_{\idx}=0.0$ will not change the numerator of the generic 
averaging expression, but it will change the denominator (unless its
weight $\wgt_{\idx}=0.0$ as well).
@end tex

@html
<a name="nrm"></a> <!-- http://nco.sf.net/nco.html#nrm -->
<a name="ntg"></a> <!-- http://nco.sf.net/nco.html#ntg -->
@end html
@node Normalization and Integration,  , Mask condition, ncwa netCDF Weighted Averager
@subsection Normalization and Integration
@cindex normalization
@cindex @code{-N}
@cindex @code{numerator}
@cindex integration
@cindex dot product
@command{ncwa} has one switch which controls the normalization of the
averages appearing in the @var{output-file}.
Short option @samp{-N} (or long options @samp{--nmr} or
@samp{--numerator}) prevents @command{ncwa} from dividing the weighted
sum of the variable (the numerator in the averaging expression) by the
weighted sum of the weights (the denominator in the averaging
expression).   
Thus @samp{-N} tells @command{ncwa} to return just the numerator of the
arithmetic expression defining the operation (@pxref{Operation Types}). 

With this normalization option, @command{ncwa} can integrate variables.
Averages are first computed as sums, and then normalized to obtain the
average. 
The original sum (i.e., the numerator of the expression in
@ref{Operation Types}) is output if default normalization is turned off
(@w{with @samp{-N}}). 
This sum is the integral (not the average) over the specified 
(@w{with @samp{-a}}, or all, if none are specified) dimensions.
The weighting variable, if specified (@w{with @samp{-w}}), plays the
role of the differential increment and thus permits more sophisticated 
integrals (i.e., weighted sums) to be output.
For example, consider the variable 
@code{lev} where @math{@var{lev} = [100,500,1000]} weighted by
the weight @code{lev_wgt} where @math{@var{lev_wgt} = [10,2,1]}.
@cindex dot product
The vertical integral of @code{lev}, weighted by @code{lev_wgt}, 
is the dot product of @var{lev} and @var{lev_wgt}. 
That this is @w{is 3000.0} can be seen by inspection and verified with 
the integration command
@example
ncwa -N -a lev -v lev -w lev_wgt in.nc foo.nc;ncks foo.nc
@end example

@ignore
fxm TODO nco702
As explained in @xref{Operation Types}, @command{ncwa} 
@emph{always averages} coordinate variables regardless of the arithmetic 
operation type performed on the non-coordinate variables. 
The single exception is shown in the above example.
The @samp{-N} switch turns off normalization so variables which are to be
averaged, including coordinate variables, are not normalized.
This is equivalent to summation or integration.
@end ignore

@ignore
@c NB: these masking features are deprecated
The second normalization option tells @command{ncwa} to multiply the
weighted average the variable (given by the averaging expression)
by the @w{tally, @var{M}}.
Thus this option is similar to integration---multiplying the mean value
of a quantity by the number of gridpoints to which it applies.

The third normalization option is equivalent to specifying the first two
options simultaneously. 
In other words this option causes @command{ncwa} to return 
@w{@var{M} times} the numerator of the generic averaging expression.  
With these normalization options, @command{ncwa} can compute
sophisticated averages (and integrals) from the command line.
@end ignore

@noindent
@html
<a name="xmp_ncwa"></a> <!-- http://nco.sf.net/nco.html#xmp_ncwa -->
@end html
EXAMPLES

Given file @file{85_0112.nc}:
@example
@verbatim
netcdf 85_0112 {
dimensions:
        lat = 64 ;
        lev = 18 ;
        lon = 128 ;
        time = UNLIMITED ; // (12 currently)
variables:
        float lat(lat) ;
        float lev(lev) ;
        float lon(lon) ;
        float time(time) ;
        float scalar_var ;
        float three_dmn_var(lat, lev, lon) ;
        float two_dmn_var(lat, lev) ;
        float mask(lat, lon) ;
        float gw(lat) ;
} 
@end verbatim
@end example

Average all variables in @file{in.nc} over all dimensions and store
results in @file{out.nc}:
@example
ncwa in.nc out.nc
@end example
@noindent
All variables in @file{in.nc} are reduced to scalars in @file{out.nc} 
since @command{ncwa} averages over all dimensions unless otherwise
specified (with @samp{-a}).

Store the zonal (longitudinal) mean of @file{in.nc} in @file{out.nc}:
@example
ncwa -a lon in.nc out1.nc
ncwa -a lon -b in.nc out2.nc
@end example
@noindent
@cindex degenerate dimension
The first command turns @code{lon} into a scalar and the second retains 
@code{lon} as a degenerate dimension in all variables.
@example
% ncks --trd -C -H -v lon out1.nc
lon = 135
% ncks --trd -C -H -v lon out2.nc
lon[0] = 135
@end example
In either case the tally is simply the size of @code{lon}, i.e., 180
for the @file{85_0112.nc} file described by the sample header above.

@cindex @code{gw}
@cindex Gaussian weights
@cindex climate model
Compute the meridional (latitudinal) mean, with values weighted by
the corresponding element of @var{gw}
@footnote{@code{gw} stands for @dfn{Gaussian weight} in many
climate models.}: 
@example
ncwa -w gw -a lat in.nc out.nc
@end example
@noindent
Here the tally is simply the size of @code{lat}, @w{or 64.}
The sum of the Gaussian weights @w{is 2.0.}

Compute the area mean over the tropical Pacific:
@example
ncwa -w gw -a lat,lon -d lat,-20.,20. -d lon,120.,270. in.nc out.nc
@end example
@noindent
Here the tally is 
@set flg
@tex
$64 \times 128 = 8192$.
@clear flg
@end tex
@ifset flg
64 times 128 = 8192.
@clear flg
@end ifset

@cindex @code{ORO}
@cindex climate model
Compute the area-mean over the globe using only points for which 
@set flg
@tex
$ORO < 0.5$
@clear flg
@end tex
@ifset flg
@var{ORO} < 0.5
@clear flg
@end ifset
@footnote{@code{ORO} stands for @dfn{Orography} in some climate models
and in those models @math{@var{ORO} < 0.5} selects ocean gridpoints.}: 
@example
ncwa -B 'ORO < 0.5'      -w gw -a lat,lon in.nc out.nc
ncwa -m ORO -M 0.5 -T lt -w gw -a lat,lon in.nc out.nc
@end example
@noindent
It is considerably simpler to specify the complete @var{mask_cond} with
the single string argument to @samp{-B} than with the three separate
switches @samp{-m}, @samp{-T}, and @samp{-M}
@footnote{Unfortunately the @samp{-B} and @samp{--mask_condition}
options are unsupported on Windows (with the @acronym{MVS} compiler),
which lacks a free, standard parser and lexer.}. 
If in doubt, enclose the @var{mask_cond} within quotes since some
of the comparators have special meanings to the shell.

Assuming 70% of the gridpoints are maritime, then here the tally is
@set flg
@tex
$0.70 \times 8192 \approx 5734$.
@clear flg
@end tex
@ifset flg
0.70 times 8192 = 5734.
@clear flg
@end ifset

Compute the global annual mean over the maritime tropical Pacific:
@example
ncwa -B 'ORO < 0.5'      -w gw -a lat,lon,time \
  -d lat,-20.0,20.0 -d lon,120.0,270.0 in.nc out.nc
ncwa -m ORO -M 0.5 -T lt -w gw -a lat,lon,time \
  -d lat,-20.0,20.0 -d lon,120.0,270.0 in.nc out.nc
@end example
Further examples will use the one-switch specification of
@var{mask_cond}.  

Determine the total area of the maritime tropical Pacific, assuming
the variable @var{area} contains the area of each gridcell
@example
ncwa -N -v area -B 'ORO < 0.5' -a lat,lon \
  -d lat,-20.0,20.0 -d lon,120.0,270.0 in.nc out.nc
@end example
Weighting @var{area} (e.g., by @var{gw}) is not appropriate because
@var{area} is @emph{already} area-weighted by definition.
Thus the @samp{-N} switch, or, equivalently, the @samp{-y ttl} switch, 
correctly integrate the cell areas into a total regional area.

@cindex mask condition
@cindex truth condition
Mask a file to contain @var{_FillValue} everywhere except where
@math{@var{thr_min} <= @var{msk_var} <= @var{thr_max}}:
@example
@verbatim
# Set masking variable and its scalar thresholds
export msk_var='three_dmn_var_dbl' # Masking variable
export thr_max='20' # Maximum allowed value
export thr_min='10' # Minimum allowed value
ncecat -O in.nc out.nc # Wrap out.nc in degenerate "record" dimension
ncwa -O -a record -B "${msk_var} <= ${thr_max}" out.nc out.nc
ncecat -O out.nc out.nc # Wrap out.nc in degenerate "record" dimension
ncwa -O -a record -B "${msk_var} >= ${thr_min}" out.nc out.nc
@end verbatim
@end example
After the first use of @command{ncwa}, @file{out.nc} contains
@var{_FillValue} where @code{$@{msk_var@} >= $@{thr_max@}}.
The process is then repeated on the remaining data to filter out 
points where @code{$@{msk_var@} <= $@{thr_min@}}.
The resulting @file{out.nc} contains valid data only
where @math{@var{thr_min} <= @var{msk_var} <= @var{thr_max}}.

@html
<a name="ctr"></a> <!-- http://nco.sf.net/nco.html#ctr -->
@end html
@node Contributing, Quick Start, Reference Manual, Top
@chapter Contributing
@cindex contributing
We welcome contributions from anyone.
The project homepage at @uref{https://sf.net/projects/nco}
contains more information on how to contribute. 

@cindex PayPal
Financial contributions to @acronym{NCO} development may be made through  
@uref{https://www.paypal.com/xclick/business=zender%40uci.edu&item_name=NCO+development&item_number=nco_dnt_dvl&no_note=1&tax=0&currency_code=USD, PayPal}.
@acronym{NCO} has been shared for over @w{10 years} yet only two 
users have contributed any money to the developers
@footnote{
@cindex chocolate
Happy users have sent me a few gifts, though.
This includes a box of imported chocolate.
Mmm.
Appreciation and gifts are definitely better than money.
Naturally, I'm too lazy to split and send gifts to the other developers.
However, unlike some @acronym{NCO} developers, I have a steady "real job".
My intent is to split monetary donations among the active developers
and to send them their shares via PayPal.}. 
So you could be the third!

@html
<a name="dvl"></a> <!-- http://nco.sf.net/nco.html#dvl -->
<a name="cnt"></a> <!-- http://nco.sf.net/nco.html#cnt -->
<a name="ppl"></a> <!-- http://nco.sf.net/nco.html#ppl -->
@end html
@menu
* Contributors::
* Citation::
* Proposals for Institutional Funding::
@end menu

@node Contributors, Citation, Contributing, Contributing
@section Contributors
@acronym{NCO} would not exist without the dedicated efforts of the
remarkable software engineers who conceive, develop, and
maintain netCDF, UDUnits, and OPeNDAP.
@cindex Russ Rew
@cindex John Caron
@cindex Glenn Davis
@cindex Steve Emmerson
@cindex Ward Fisher
@cindex James Gallagher
@cindex Ed Hartnett
@cindex Dennis Heimbigner
Since 1995 @acronym{NCO} has received support from nearly the entire
staff of all these projects, including  
Russ Rew, 
John Caron,
Glenn Davis, 
Steve Emmerson, 
Ward Fisher,
James Gallagher, 
Ed Hartnett, 
and Dennis Heimbigner.
In addition to their roles in maintaining the software stack on which
@acronym{NCO} perches, Yertl-like, some of these gentlemen have advised
or contributed to @acronym{NCO} specifically. That support is
acknowledged separately below. 

@cindex contributors
The primary contributors to @acronym{NCO} development have been:
@table @asis
@cindex Charlie Zender
@item Charlie Zender
All concept, design and implementation from 1995--2000.
Since then autotools, bug-squashing, @acronym{CDL}, chunking,
documentation, anchoring, recursion, @acronym{GPE}, packing,
regridding, @acronym{CDL}/@acronym{XML} backends, compression,
@acronym{NCO} library redesign, @command{ncap2} features,
@command{ncbo}, @command{ncpdq}, @acronym{SMP} threading and @acronym{MPI} parallelization,
netCDF4 integration, external funding, project management, science
research, releases. 
@cindex Henry Butowsky
@item Henry Butowsky
Non-linear operations and @code{min()}, @code{max()}, @code{total()}
support in @command{ncra} and @command{ncwa}. 
Type conversion for arithmetic.
Migration to netCDF3 @acronym{API}.
@command{ncap2} parser, lexer, @acronym{GSL}-support, @w{and I/O}.
Multislabbing algorithm.
Variable wildcarding.
@acronym{JSON} backend.
Numerous hacks.
@command{ncap2} language.
@cindex Rorik Peterson
@item Rorik Peterson
Original autotools build support. 
Long command-line options.
Original UDUnits support.
Debianization.
Numerous bug-fixes.
@cindex Joe Hamman
@item Joe Hamman
Python bindings (PyNCO).
@cindex Milan Klower
@cindex Rostislav Kouznetsov
@item Milan Klower, Rostislav Kouznetsov
Quantization by rounding 
@cindex Daniel Wang
@item Daniel Wang
Script Workflow Analysis for MultiProcessing (@acronym{SWAMP}).
@acronym{RPM} support.
@cindex Harry Mangalam
@item Harry Mangalam
Benchmarking.
OPeNDAP configuration.
@cindex Pedro Vicente
@item Pedro Vicente
Windows Visual Studio support.
netCDF4 groups.
CMake build-engine.
@cindex Jerome Mao
@item Jerome Mao
Multi-argument parsing.
@cindex Joseph O'Rourke
@item Joseph O'Rourke
Routines from his book ``Computational Geometry in C''.
@cindex Russ Rew
@item Russ Rew
Advice on @acronym{NCO} structural algorithms.
@cindex Brian Mays
@item Brian Mays
Original packaging for Debian @acronym{GNU}/Linux, @command{nroff} man pages.
@cindex George Shapovalov
@item George Shapovalov
Packaging for Gentoo @acronym{GNU}/Linux.
@cindex Bill Kocik
@item Bill Kocik
Memory management.
@cindex Len Makin
@item Len Makin
NEC SX architecture support.
@cindex Jim Edwards
@item Jim Edwards
AIX architecture support.
@cindex Juliana Rew
@item Juliana Rew
Compatibility with large @acronym{PID}s.
@cindex Karen Schuchardt
@item Karen Schuchardt
Auxiliary coordinate support.
@cindex Gayathri Venkitachalam
@item Gayathri Venkitachalam
@acronym{MPI} implementation.
@cindex Scott Capps
@item Scott Capps
Large work-load testing
@cindex Xylar Asay-Davis
@cindex Sterling Baldwin 
@cindex Dave Blodgett
@cindex Peter Campbell
@cindex Peter Caldwell
@cindex Philip Cameron-Smith
@cindex Martin Dix
@cindex Mark Flanner
@cindex Chris Golaz
@cindex Barron Henderson
@cindex Ben Hillman
@cindex Aleksandar Jelenak
@cindex Markus Liebig
@cindex Keith Lindsay
@cindex Daniel Macks, 
@cindex Stu Muller
@cindex Daniel Neumann
@cindex Mike Page
@cindex Martin Schmidt
@cindex Lori Sentman
@cindex Michael Schulz
@cindex Rich Signell
@cindex Bob Simons
@cindex Gary Strand
@cindex Qi Tang
@cindex Mark Taylor
@cindex Matthew Thompson
@cindex Adrian Tompkins
@cindex Paul Ullrich
@cindex Andrew Wittenberg
@cindex George White
@cindex Min Xu
@cindex Remik Ziemlinski
@item Xylar Asay-Davis, Sterling Baldwin, Dave Blodgett, Peter Caldwell, Philip Cameron-Smith, Peter Campbell, Martin Dix, Mark Flanner, Chris Golaz, Barron Henderson, Ben Hillman, Aleksandar Jelenak, Markus Liebig, Keith Lindsay, Daniel Macks, Daniel Neumann, Mike Page, Martin Schmidt, Michael Schulz, Lori Sentman, Rich Signell, Bob Simons, Gary Strand, Mark Taylor, Matthew Thompson, Qi Tang, Adrian Tompkins, Paul Ullrich, George White, Andrew Wittenberg, Min Xu, Remik Ziemlinski
Excellent bug reports and feature requests.
@cindex Xylar Asay-Davis
@cindex Filipe Fernandes
@cindex Isuru Fernando
@cindex Craig MacLachlan
@cindex Hugo Oliveira
@cindex Rich Signell
@cindex Kyle Wilcox
@cindex Klaus Zimmermann
@item Filipe Fernandes, Isuru Fernando, Craig MacLachlan, Hugo Oliveira, Rich Signell, Kyle Wilcox, Klaus Zimmermann
Anaconda packaging
@cindex Xylar Asay-Davis
@cindex Daniel Baumann
@cindex Nick Bower
@cindex Luk Claes
@cindex Barry deFreese
@cindex Francesco Lovergine
@cindex Bas Couwenberg
@cindex Matej Vela
@item Xylar Asay-Davis, Daniel Baumann, Nick Bower, Luk Claebs, Bas Couwenberg, Barry deFreese, Francesco Lovergine, Matej Vela
Cygwin packaging
@cindex Marco Atzeri
@item Marco Atzeri
Debian packaging
@cindex Patrice Dumas
@cindex Ed Hill
@cindex Orion Poplawski
@item Patrice Dumas, Ed Hill, Orion Poplawski
Gentoo packaging
@cindex Filipe Fernandes
@item Filipe Fernandes
OpenSuse packaging
@cindex Takeshi Enomoto
@cindex Alexander Hansen
@cindex Ian Lancaster
@cindex Alejandro Soto
@item Takeshi Enomoto, Alexander Hansen, Ian Lancaster, Alejandro Soto
Mac OS packaging
@cindex Eric Blake
@item Eric Blake
PyNCO Anaconda and Pip packaging, bug fixes
@cindex Tim Heap
@item Tim Heap
RedHat packaging
@cindex George Shapavalov
@cindex Patrick Kursawe
@cindex Manfred Schwarb
@item George Shapavalov, Patrick Kursawe, Manfred Schwarb
Autoconf/M4 help
@cindex Gavin Burris
@cindex Kyle Wilcox
@item Gavin Burris, Kyle Wilcox
RHEL and CentOS build scripts and bug reports.
@cindex Andrea Cimatoribus
@item Andrea Cimatoribus
@acronym{NCO} Spiral Logo
@cindex Martin Otte
@cindex Etienne Tourigny
@item Martin Otte, Etienne Tourigny
Single bug reports and fixes
@cindex Wenshan Wang
@item Wenshan Wang
@acronym{CMIP5} and @acronym{MODIS} processing documentation, reference card
@cindex Todd Mitchell
@item Todd Mitchell
Sustained acknowledgement via financial donations
@end table
Please let me know if your name was omitted!

@html
<a name="ctt"></a> <!-- http://nco.sf.net/nco.html#ctt -->
@end html
@node Citation, Proposals for Institutional Funding, Contributors, Contributing
@section Citation
@cindex citation
The recommended citations for @acronym{NCO} software are
@example
@verbatim
Zender, C. S. (2008), Analysis of Self-describing Gridded Geoscience
Data with netCDF Operators (NCO), Environ. Modell. Softw., 23(10),
1338-1342, doi:10.1016/j.envsoft.2008.03.004. 

Zender, C. S. and H. J. Mangalam (2007), Scaling Properties of Common
Statistical Operators for Gridded Datasets, Int. J. High
Perform. Comput. Appl., 21(4), 485-498, doi:10.1177/1094342007083802.

Zender, C. S. (2016), Bit Grooming: Statistically accurate
precision-preserving quantization with compression, evaluated in the
netCDF Operators (NCO, v4.4.8+), Geosci. Model Dev., 9, 3199-3211,
doi:10.5194/gmd-9-3199-2016.

Zender, C. S. (Year), netCDF Operator (NCO) User Guide, 
http://nco.sf.net/nco.pdf. 
@end verbatim
@end example
Use the first when referring to overall design, purpose, and 
optimization of @acronym{NCO}, the second for the speed and throughput
of @acronym{NCO}, the third for compressions, and the fourth for
specific features and/or the User Guide itself, or in a non-academic
setting. 
A complete list of @acronym{NCO} publications and presentations is at
@url{http://nco.sf.net#pub}.
This list links to the full papers and seminars themselves.

@html
<a name="prp"></a> <!-- http://nco.sf.net/nco.html#prp -->
<a name="prp_sei"></a> <!-- http://nco.sf.net/nco.html#prp_sei -->
<a name="fnd"></a> <!-- http://nco.sf.net/nco.html#fnd -->
@end html
@menu
* Proposals for Institutional Funding::  
@end menu

@node Proposals for Institutional Funding,  , Citation, Contributing
@section Proposals for Institutional Funding
@cindex funding
@cindex proposals
@cindex @acronym{NSF}
@cindex server-side processing
@cindex Distributed Data Reduction & Analysis
@cindex Scientific Data Operators
@cindex @acronym{DDRA}
@cindex Server-Side Distributed Data Reduction & Analysis
@cindex @acronym{SSDDRA}
@cindex @acronym{CCSM}
@cindex @acronym{IPCC}
@cindex @acronym{NSF}
@cindex @acronym{SDO}
@cindex @acronym{SEIII}
@cindex OptIPuter
@cindex parallelism
From 2004--2007, @acronym{NSF} funded a
@uref{http://nco.sf.net#prp_sei, project}
to improve Distributed Data Reduction & Analysis (@acronym{DDRA}) by
evolving @acronym{NCO} parallelism (OpenMP, @acronym{MPI}) and
Server-Side @acronym{DDRA} (@acronym{SSDDRA}) implemented through
extensions to @acronym{OPeNDAP} and netCDF4. 
The @acronym{SSDDRA} features were implemented in @acronym{SWAMP},
the PhD Thesis of Daniel Wang.
@acronym{SWAMP} dramatically reduced bandwidth usage for @acronym{NCO} 
between client and server.

@cindex @acronym{NASA}
@cindex @acronym{NRA}
@cindex @acronym{HDF}
With this first @acronym{NCO} proposal funded, the content of the
next @acronym{NCO} proposal became clear.
We had long been interested in obtaining @acronym{NASA} support for 
@acronym{HDF}-specific enhancements to @acronym{NCO}. 
From 2012--2015 the @acronym{NASA} @acronym{ACCESS} program funded us to
implement support support netCDF4 group functionality. 
Thus @acronym{NCO} will grow and evade bit-rot for the foreseeable
future. 

We are considering other interesting ideas for still more proposals.
Please contact us if you wish to be involved with any future
@acronym{NCO}-related proposals.  
Comments on the proposals and letters of support are also very welcome.

@html
<a name="quicksrt"></a> <!-- http://nco.sf.net/nco.html#quicksrt -->
@end html
@node Quick Start, CMIP5 Example, Contributing, Top
@chapter Quick Start
@cindex Quick Start
Simple examples in Bash shell scripts showing how to average data with
different file structures.  
Here we include monthly, seasonal and annual average with daily or
monthly data in either one file or multiple files. 

@menu
* Daily data in one file::
* Monthly data in one file::
* One time point one file::
* Multiple files with multiple time points::
@end menu

@node Daily data in one file, Monthly data in one file, Quick Start, Quick Start
@section Daily data in one file
@cindex daily data
Suppose we have daily data from Jan 1st, 1990 to Dec. 31, 2005 in the
file of @file{in.nc} with the record dimension as @code{time}.

@noindent
@strong{Monthly average:}
@cindex monthly average
@cindex average
@cindex time-averaging
@example
@verbatim
for yyyy in {1990..2005}; do      # Loop over years
  for moy in {1..12}; do          # Loop over months
    mm=$( printf "%02d" ${moy} )  # Change to 2-digit format

    # Average specific month yyyy-mm
    ncra -O -d time,"${yyyy}-${mm}-01","${yyyy}-${mm}-31" \
         in.nc in_${yyyy}${mm}.nc
  done
done

# Concatenate monthly files together
ncrcat -O in_??????.nc out.nc
@end verbatim
@end example

@noindent
@strong{Annual average:}
@cindex annual average from daily data
@cindex average
@cindex time-averaging
@example
@verbatim
for yyyy in {1990..2005}; do      # Loop over years
  ncra -O -d time,"${yyyy}-01-01","${yyyy}-12-31" in.nc in_${yyyy}.nc
done

# Concatenate annual files together
ncrcat -O in_????.nc out.nc
@end verbatim
@end example
The @option{-O} switch means to overwrite the pre-existing files (@pxref{Batch Mode}).
The @option{-d} option is to specify the range of hyperslabs (@pxref{Hyperslabs}).
There are detailed instructions on @command{ncra} (@pxref{ncra netCDF Record Averager} and @command{ncrcat} (@pxref{ncrcat netCDF Record Concatenator}).
@acronym{NCO} supports UDUnits so that we can use readable dates as time dimension (@pxref{UDUnits Support}).

@node Monthly data in one file, One time point one file, Daily data in one file, Quick Start
@section Monthly data in one file
@cindex monthly data
Inside the input file @file{in.nc}, the record dimension @code{time} is from Jan 1990 to Dec 2005.

@noindent
@strong{Seasonal average (e.g., @acronym{DJF}):}
@cindex seasonal average
@cindex average
@cindex time-averaging
@example
ncra -O --mro -d time,"1990-12-01",,12,3 in.nc out.nc
@end example

@noindent
@strong{Annual average:}
@cindex annual average from monthly data
@cindex average
@cindex time-averaging
@example
ncra -O --mro -d time,,,12,12 in.nc out.nc
@end example
Here we use the subcycle feature (i.e., the number after the fourth
comma: @samp{3} in the seasonal example and the second @samp{12} in
the annual example) to retrieve groups of records separated by regular
intervals (@pxref{Subcycle}).
The option @option{--mro} switches @command{ncra} to produce a
Multi-Record Output instead of a single-record output. 
For example, assume @var{snd} is a 3D array with dimensions
@code{time} * @code{latitude} * @code{longitude} and @code{time}
includes every month from Jan. 1990 to Dec. 2005, 192 months in total, 
or 16 years. 
Consider the following two command lines:
@example
ncra --mro -v snd -d time,"1990-12-01",,12,3 in.nc out_mro.nc
ncra -v snd -d time,"1990-12-01",,12,3 in.nc out_sro.nc
@end example
In the first output file, @file{out_mro.nc}, @var{snd} is still a 3D
array with dimensions @code{time} * @code{latitude} * @code{longitude}, 
but the length of @code{time} now is 16, meaning 16 winters.
In the second output file, @file{out_sro.nc}, the length of
@code{time} is @w{only 1}, which contains the average of all 16 winters.

When using @samp{-d @var{dim},min[,max]} to specify the hyperslabs, 
you can leave it blank if you want to include the minimum or the
maximum of the data, like we did above. 

@node One time point one file, Multiple files with multiple time points, Monthly data in one file, Quick Start
@section One time point one file
@cindex daily data
@cindex monthly data
@cindex average
@cindex time-averaging
This means if you have daily data of 30 days, there will be 30 data files.
Or if you have monthly data of 12 months, there will be 12 data files.
Dealing with this kind of files, you need to specify the file names in shell scripts and pass them to NCO operators.
For example, your daily data files may look like @file{snd_19900101.nc}, @file{snd_19900102.nc}, @file{snd_19900103.nc} ...
If you want to know the monthly average of Jan 1990, you can write like,
@example
ncra -O snd_199001??.nc out.nc
@end example
You might want to use loop if you need the average of each month.
@example
@verbatim
for moy in {1..12}; do          # Loop over months
  mm=$( printf "%02d" ${moy} )  # Change to 2-digit format

  ncra -O snd_????${mm}??.nc out_${mm}.nc
done
@end verbatim
@end example

@node Multiple files with multiple time points,  , One time point one file, Quick Start
@section Multiple files with multiple time points
@cindex daily data
@cindex monthly data
Similar as the last one, it's more about shell scripts.
Suppose you have daily data with one month of them in one data file.
The monthly average is simply to apply @command{ncra} on the specific data file.
And for seasonal averages, you can specify the three months by shell scripts.

@html
<a name="cmip5"></a> <!-- http://nco.sf.net/nco.html#cmip5 -->
<a name="godad"></a> <!-- http://nco.sf.net/nco.html#godad -->
@end html
@node CMIP5 Example, Parallel, Quick Start, Top
@chapter @acronym{CMIP5} Example
@cindex @acronym{CMIP5}
@cindex @acronym{GODAD}
@ignore
This @uref{http://nco.sf.net/xmp_cesm.html,Wonderful CMIP5 Documentation}
shows complete processing of the @acronym{CMIP5} dataset.
@end ignore

The fifth phase of the Coupled Model Intercomparison Project 
(@uref{http://cmip-pcmdi.llnl.gov/cmip5/index.html?submenuheader=0, @acronym{CMIP5}}) 
provides a multi-model framework for comparing the mechanisms and
responses of climate models from around the world.   
However, it is a tremendous workload to retrieve a single climate
statistic from all these models, each of which includes several ensemble 
members.  
Not only that, it is too often a tedious process that impedes new
research and hypothesis testing.  
Our @acronym{NASA} @acronym{ACCESS} 2011 project simplified and
accelerated this process.   

Traditional geoscience data analysis requires users to work with
numerous flat (data in one level or namespace) files. 
In that paradigm instruments or models produce, and then repositories
archive and distribute, and then researchers request and analyze,
collections of flat files.
@acronym{NCO} works well with that paradigm, yet it also embodies the
necessary algorithms to transition geoscience data analysis from relying
solely on traditional (or ``flat'') datasets to allowing newer
hierarchical (or ``nested'') datasets.  

Hierarchical datasets support and enable combining all datastreams that
meet user-specified criteria into a single or small number of files that
hold @emph{all} the science-relevant data.
@acronym{NCO} (and no other software to our knowledge) exploits this
capability now.
Data and metadata may be aggregated into and analyzed in hierarchical
structures.
We call the resulting data storage, distribution, and analysis
paradigm Group-Oriented Data Analysis and Distribution
(@acronym{GODAD}). 
@acronym{GODAD} lets the scientific question organize the data, not the  
@emph{ad hoc} granularity of all relevant datasets.
This chapter illustrates @acronym{GODAD} techniques applied to 
analysis of the @acronym{CMIP5} dataset.

To begin, we document below a prototypical example of @acronym{CMIP5} 
analysis and evaluation using traditional @acronym{NCO} commands on
netCDF3-format model and @acronym{HDF-EOS} format observational
(@acronym{NASA} @acronym{MODIS} satellite instrument) datasets.
These examples complement the @acronym{NCO} User Guide by detailing
in-depth data analysis in a frequently encountered ``real world''
context.   
Graphical representations of the results (@acronym{NCL} scripts
available upon request) are provided to illustrate physical meaning of
the analysis.
@ignore
In 2013 we added scripts which make use of new @acronym{NCO} features
that combine all the loops in the analysis into single commands by
exploiting @acronym{NCO}'s new group aggregation and arithmetic
features.   
@end ignore
Since @acronym{NCO} can process hierarchical datasets, i.e., datasets
stored with netCDF4 groups, we present sample scripts illustrating
group-based processing as well. 

@menu
* Combine Files::
* Global Distribution of Long-term Average::
* Annual Average over Regions::
* Monthly Cycle::
* Regrid MODIS Data::
* Add Coordinates to MODIS Data::
* Permute MODIS Coordinates::
@end menu

@node Combine Files, Global Distribution of Long-term Average, CMIP5 Example, CMIP5 Example
@section Combine Files
@cindex file combination
Sometimes, the data of one ensemble member will be stored in several
files to reduce single file size.
It is more convenient to concatenate these files into a single
timeseries, and the following script illustrates how.
Key steps include: 
@enumerate
@item Obtain number and names (or partial names) of files in a directory
@item Concatenate files along the record dimension (usually time) using 
@command{ncrcat} (@pxref{ncrcat netCDF Record Concatenator}).
@end enumerate
@example
@verbatiminclude xmp/cmb_fl.sh
@end example

@acronym{CMIP5} model data downloaded from the Earth System Grid
Federation (@uref{http://pcmdi9.llnl.gov/esgf-web-fe/, @acronym{ESGF}}) 
does not contain group features yet. 
Therefore users must aggregate flat files into hierarchical ones themselves.
The following script shows how.
Each dataset becomes a group in the output file.
There can be several levels of groups.
In this example, we employ two experiments (``scenarios'') as the top-level.
The second-level comprises different models (e.g., @acronym{CCSM4}, @acronym{CESM1-BGC}).
Many models are run multiple times with slight perturbed initial
conditions to produce an ensemble of realizations.
These ensemble members comprise the third level of the hierarchy.
The script selects two variables, @var{snc} and @var{snd} (snow cover
and snow depth).
@cindex @option{--gag}
@cindex aggregation
@cindex group aggregation
@cindex groups, creating
@example
@verbatiminclude xmp/cmb_fl_grp.sh
@end example

@node Global Distribution of Long-term Average, Annual Average over Regions, Combine Files, CMIP5 Example
@section Global Distribution of Long-term Average
@cindex spatial distribution
@cindex long-term average
@cindex average
@cindex time-averaging
@float Figure,fgr:glb
@image{xmp/fgr1,3.5in} 
@caption{Global Distribution of Long-term Average.}
@end float
@noindent
This section illustrates how to calculate the global distribution of
long-term average (@pxref{fgr:glb}) with either flat files or 
@uref{http://nco.sourceforge.net/nco.html#index-groups, group file}.
Key steps include: 
@enumerate
@item Average ensemble members of each model using @command{nces} (@pxref{nces netCDF Ensemble Statistics})
@item Average the record dimension using @command{ncra} (@pxref{ncra netCDF Record Averager})
@item Store results of each model as a distinct group in a single output file using @command{ncecat} (@pxref{ncrcat netCDF Record Concatenator}) with the @option{--gag} option
@end enumerate
The first example shows how to process flat files.
@example
@verbatiminclude xmp/glb_avg.sh
@end example

With the use of @key{group}, the above script will be shortened to @w{ONE LINE}.
@cindex groups, averaging
@example
# Data from cmb_fl_grp.sh
# ensemble averaging
nces -O --nsm_grp --nsm_sfx='_avg' \
sn_LImon_all-mdl_all-xpt_all-nsm_200001-200512.nc \
  sn_LImon_all-mdl_all-xpt_nsm-avg.nc
@end example
The input file,
@file{sn_LImon_all-mdl_all-xpt_all-nsm_200001-200512.nc}, produced by
@file{cmb_fl_grp.sh}, includes all the ensemble members as groups.
The option @samp{--nsm_grp} denotes 
that we are using @uref{http://nco.sf.net/nco.html#nsm_grp, group ensembles mode} of @command{nces}, 
instead of @uref{http://nco.sf.net/nco.html#nsm_fl, file ensembles mode}, @samp{--nsm_fl}.
The option @samp{--nsm_sfx='_avg'} instructs @command{nces} 
to store the output as a new child group @file{/[model]/[model name]_avg/var};
otherwise, the output will be stored directly in the parent group @file{/[model]/var}. 
In the final output file, @file{sn_LImon_all-mdl_all-xpt_nsm-avg_tm-avg.nc}, 
sub-groups with a suffix of `avg' are the long-term averages of each model.
One thing to notice is that for now, 
ensembles with only one ensemble member will be left untouched.

@node Annual Average over Regions, Monthly Cycle, Global Distribution of Long-term Average, CMIP5 Example
@section Annual Average over Regions
@cindex annual average
@cindex average
@cindex time-averaging
@cindex area-averaging
@cindex dimension order
@cindex anomalies
@cindex standard deviation
@cindex renaming variables
@cindex attributes, editing
@cindex	attributes, modifying
@cindex	attributes, overwriting
@cindex regression
@cindex nco script file
@cindex variables, appending
@float Figure,fgr:anl
@image{xmp/fgr2,4in}
@caption{Annual Average over Regions.}
@end float
@noindent
This section illustrates how to calculate the annual average over
specific regions (@pxref{fgr:anl}).
Key steps include: 
@enumerate
@item Spatial average using @command{ncap2} (@pxref{ncap2 netCDF Arithmetic Processor}) and @command{ncwa} (@pxref{ncwa netCDF Weighted Averager}); 
@item Change dimension order using @command{ncpdq} (@pxref{ncpdq netCDF Permute Dimensions Quickly});
@item Annual average using @command{ncra} (@pxref{ncra netCDF Record Averager});
@item Anomaly from long-term average using @command{ncbo} (@pxref{ncbo netCDF Binary Operator});
@item Standard deviation using @command{ncbo} (@pxref{ncbo netCDF Binary Operator}) and @command{nces} (@pxref{nces netCDF Ensemble Statistics});
@item Rename variables using @command{ncrename} (@pxref{ncrename netCDF Renamer});
@item Edit attributions using @command{ncatted} (@pxref{ncatted netCDF Attribute Editor});
@item Linear regression using @command{ncap2} (@pxref{ncap2 netCDF Arithmetic Processor});
@item Use @command{ncap2} (@pxref{ncap2 netCDF Arithmetic Processor}) with nco script file (i.e., @file{.nco} file);
@item Move variables around using @command{ncks} (@pxref{ncks netCDF Kitchen Sink}).
@end enumerate
@strong{Flat files example}
@example
@verbatiminclude xmp/ann_avg.sh
@end example
@strong{gsl_rgr.nco}
@example
@verbatiminclude xmp/gsl_rgr.nco
@end example

With the @key{group} feature, 
all the loops over experiments, models and ensemble members can be omitted.
As we are working on implementing @key{group} feature in all @acronym{NCO} operators,
some functions (e.g., regression and standard deviation over ensemble members) 
may have to wait until the new versions.
@cindex group, spatial averaging
@cindex group, temporal averaging
@cindex group, anomaly
@cindex group, standard deviation
@cindex group, aggregation
@cindex group, dimension permutation
@example
@verbatiminclude xmp/ann_avg_grp.sh
@end example

@node Monthly Cycle, Regrid MODIS Data, Annual Average over Regions, CMIP5 Example
@section Monthly Cycle
@cindex monthly average
@cindex average
@cindex time-averaging
@cindex anomalies
@cindex geographical weight
@cindex weighted average
@float Figure,fgr:mon
@image{xmp/fgr3,4in}
@caption{Monthly Cycle.}
@end float
@noindent
This script illustrates how to calculate the monthly anomaly from the
annual average (@pxref{fgr:mon}). 
In order to keep only the monthly cycle,
we will subtract the annual average of each year from the monthly data,
instead of subtracting the long-term average.
This is a little more complicated in coding since we need to loop over years. 

@strong{Flat files example}
@example
@verbatiminclude xmp/mcc.sh
@end example
Using @key{group} feature and @uref{http://nco.sourceforge.net/nco.html#Hyperslabs, hyperslabs} of @command{ncbo},
the script will be shortened.
@example
@verbatiminclude xmp/mcc_grp.sh
@end example

@node Regrid MODIS Data, Add Coordinates to MODIS Data, Monthly Cycle, CMIP5 Example
@section Regrid @acronym{MODIS} Data
@cindex regrid
@cindex MODIS
@cindex bilinear interpolation
@cindex interpolation
@cindex renaming variables
@cindex renaming attributes
@cindex renaming dimensions
@cindex attributes, editing
@cindex	attributes, modifying
@cindex	attributes, overwriting
In order to compare the results between @acronym{MODIS} and
@acronym{CMIP5} models, one usually regrids one or both datasets so 
that the spatial resolutions match. 
Here, the script illustrates how to regrid @acronym{MODIS} data.
Key steps include:
@enumerate
@item Regrid using bilinear interpolation (@pxref{Bilinear interpolation})
@item Rename variables, dimensions and attributions using @command{ncrename} (@pxref{ncrename netCDF Renamer}).
@end enumerate
@strong{Main Script}
@example
@verbatiminclude xmp/rgr.sh
@end example
@strong{bi_interp.nco}
@example
@verbatiminclude xmp/bi_interp.nco
@end example

@node Add Coordinates to MODIS Data, Permute MODIS Coordinates, Regrid MODIS Data, CMIP5 Example
@section Add Coordinates to @acronym{MODIS} Data
@cindex MODIS
@cindex coordinates
@strong{Main Script}
@example
@verbatiminclude xmp/add_crd.sh
@end example
@strong{crd.nco}
@example
@verbatiminclude xmp/crd.nco
@end example

@node Permute MODIS Coordinates,  , Add Coordinates to MODIS Data, CMIP5 Example
@section Permute @acronym{MODIS} Coordinates
@cindex coordinates, modifying
@c NB: 20140130: @textdegree is in TeXInfo 4.12 but Mac OS X 10.8 ships with 4.8
@acronym{MODIS} orders latitude data from 90@textdegree{}N to
-90@textdegree{}N, and longitude from -180@textdegree{}E to
180@textdegree{}E.   
However, @acronym{CMIP5} orders latitude from -90@textdegree{}N to
90@textdegree{}N, and longitude from 0@textdegree{}E to
360@textdegree{}E.  
This script changes the @acronym{MODIS} coordinates to follow the
@acronym{CMIP5} convention.
@example
@verbatiminclude xmp/pmt_crd.sh
@end example

@ignore
@node Hierarchical Data Files, Parallel, CMIP5 Example, Top
section Hierarchical Data Files
@cindex hierarchical data
@cindex groups
Hierarchical Data Files support arbitrarily nested groups.
The following @acronym{NCO} operators now can work recursively through all groups:
@itemize @bullet
@item @command{ncbo} (@pxref{ncbo netCDF Binary Operator})
@item @command{ncecat} (@pxref{ncecat netCDF Ensemble Concatenator})
@item @command{ncks} (@pxref{ncks netCDF Kitchen Sink})
@item @command{ncpdq} (@pxref{ncpdq netCDF Permute Dimensions Quickly})
@item @command{ncwa} (@pxref{ncwa netCDF Weighted Averager})
@end itemize
Here is an example showing:
@enumerate
@item How to create a hierarchical data file from multiple files using @command{ncecat} (@pxref{ncecat netCDF Ensemble Concatenator}) or @command{ncks} (@pxref{ncks netCDF Kitchen Sink});
@item Hyperslabs using @command{ncks} (@pxref{ncks netCDF Kitchen Sink});
@item Spatial average and time average using @command{ncwa} (@pxref{ncwa netCDF Weighted Averager});
@item Anomaly from average using @command{ncbo} (@pxref{ncbo netCDF Binary Operator}).
@end enumerate
@example
@verbatiminclude xmp/grp.sh
@end example
@end ignore

@html
<a name="parallel"></a> <!-- http://nco.sf.net/nco.html#parallel -->
<a name="prl"></a> <!-- http://nco.sf.net/nco.html#prl -->
<a name="swift"></a> <!-- http://nco.sf.net/nco.html#swift -->
<a name="swf"></a> <!-- http://nco.sf.net/nco.html#swf -->
<a name="Parallel"></a> <!-- http://nco.sf.net/nco.html#Parallel -->
<a name="Swift"></a> <!-- http://nco.sf.net/nco.html#Swift -->
@end html
@node Parallel, CCSM Example, CMIP5 Example, Top
@chapter Parallel
@cindex Parallel
@cindex Swift
@cindex @command{parallel}
This section will describe @acronym{NCO} scripting strategies.
Many techniques can be used to exploit script-level parallelism,
including @acronym{GNU} Parallel and Swift.
@example
@verbatim
ls *historical*.nc | parallel ncks -O -d time,"1950-01-01","2000-01-01" {} 50y/{}
@end verbatim
@end example

@html
<a name="ccsm"></a> <!-- http://nco.sf.net/nco.html#ccsm -->
@end html
@node CCSM Example, mybibnode, Parallel, Top
@chapter CCSM Example
@cindex CCSM

This chapter illustrates how to use @acronym{NCO} to
process and analyze the results of a @acronym{CCSM} climate simulation.
@example
@verbatim
************************************************************************
Task 0: Finding input files
x************************************************************************
The CCSM model outputs files to a local directory like:

/ptmp/zender/archive/T42x1_40

Each component model has its own subdirectory, e.g., 

/ptmp/zender/archive/T42x1_40/atm
/ptmp/zender/archive/T42x1_40/cpl
/ptmp/zender/archive/T42x1_40/ice
/ptmp/zender/archive/T42x1_40/lnd
/ptmp/zender/archive/T42x1_40/ocn

within which model output is tagged with the particular model name

/ptmp/zender/archive/T42x1_40/atm/T42x1_40.cam2.h0.0001-01.nc
/ptmp/zender/archive/T42x1_40/atm/T42x1_40.cam2.h0.0001-02.nc
/ptmp/zender/archive/T42x1_40/atm/T42x1_40.cam2.h0.0001-03.nc
...
/ptmp/zender/archive/T42x1_40/atm/T42x1_40.cam2.h0.0001-12.nc
/ptmp/zender/archive/T42x1_40/atm/T42x1_40.cam2.h0.0002-01.nc
/ptmp/zender/archive/T42x1_40/atm/T42x1_40.cam2.h0.0002-02.nc
...

or 

/ptmp/zender/archive/T42x1_40/lnd/T42x1_40.clm2.h0.0001-01.nc
/ptmp/zender/archive/T42x1_40/lnd/T42x1_40.clm2.h0.0001-02.nc
/ptmp/zender/archive/T42x1_40/lnd/T42x1_40.clm2.h0.0001-03.nc
...

************************************************************************
Task 1: Regional processing
************************************************************************
A common task in data processing is often creating seasonal cycles.
Imagine a 100-year simulation with its 1200 monthly mean files.
Our goal is to create a single file containing 12 months of data.
Each month in the output file is the mean of 100 input files.

Normally, we store the "reduced" data in a smaller, local directory.

caseid='T42x1_40'
#drc_in="${DATA}/archive/${caseid}/atm"
drc_in="${DATA}/${caseid}"
drc_out="${DATA}/${caseid}"
mkdir -p ${drc_out}
cd ${drc_out}

Method 1: Assume all data in directory applies
for mth in {1..12}; do
  mm=`printf "%02d" $mth`
  ncra -O -D 1 -o ${drc_out}/${caseid}_clm${mm}.nc \
    ${drc_in}/${caseid}.cam2.h0.*-${mm}.nc 
done # end loop over mth

Method 2: Use shell 'globbing' to construct input filenames
for mth in {1..12}; do
  mm=`printf "%02d" $mth`
  ncra -O -D 1 -o ${drc_out}/${caseid}_clm${mm}.nc \
    ${drc_in}/${caseid}.cam2.h0.00??-${mm}.nc \
    ${drc_in}/${caseid}.cam2.h0.0100-${mm}.nc
done # end loop over mth

Method 3: Construct input filename list explicitly
for mth in {1..12}; do
  mm=`printf "%02d" $mth`
  fl_lst_in=''
  for yr in {1..100}; do
    yyyy=`printf "%04d" $yr`
    fl_in=${caseid}.cam2.h0.${yyyy}-${mm}.nc
    fl_lst_in="${fl_lst_in} ${caseid}.cam2.h0.${yyyy}-${mm}.nc"
  done # end loop over yr
  ncra -O -D 1 -o ${drc_out}/${caseid}_clm${mm}.nc -p ${drc_in} \
    ${fl_lst_in}
done # end loop over mth

Make sure the output file averages correct input files!
ncks --trd -M prints global metadata: 

  ncks --trd -M ${drc_out}/${caseid}_clm01.nc

The input files ncra used to create the climatological monthly mean
will appear in the global attribute named 'history'.

Use ncrcat to aggregate the climatological monthly means

  ncrcat -O -D 1 \
    ${drc_out}/${caseid}_clm??.nc ${drc_out}/${caseid}_clm_0112.nc

Finally, create climatological means for reference.
The climatological time-mean:

  ncra -O -D 1 \
    ${drc_out}/${caseid}_clm_0112.nc ${drc_out}/${caseid}_clm.nc

The climatological zonal-mean:

  ncwa -O -D 1 -a lon \
    ${drc_out}/${caseid}_clm.nc ${drc_out}/${caseid}_clm_x.nc

The climatological time- and spatial-mean:

  ncwa -O -D 1 -a lon,lat,time -w gw \
    ${drc_out}/${caseid}_clm.nc ${drc_out}/${caseid}_clm_xyt.nc

This file contains only scalars, e.g., "global mean temperature",
used for summarizing global results of a climate experiment.

Climatological monthly anomalies = Annual Cycle: 
Subtract climatological mean from climatological monthly means. 
Result is annual cycle, i.e., climate-mean has been removed.

  ncbo -O -D 1 -o ${drc_out}/${caseid}_clm_0112_anm.nc \
    ${drc_out}/${caseid}_clm_0112.nc ${drc_out}/${caseid}_clm_xyt.nc

************************************************************************
Task 2: Correcting monthly averages
************************************************************************
The previous step appoximates all months as being equal, so, e.g.,
February weighs slightly too much in the climatological mean.
This approximation can be removed by weighting months appropriately.
We must add the number of days per month to the monthly mean files.
First, create a shell variable dpm:

unset dpm # Days per month
declare -a dpm
dpm=(0 31 28.25 31 30 31 30 31 31 30 31 30 31) # Allows 1-based indexing

Method 1: Create dpm directly in climatological monthly means
for mth in {1..12}; do
  mm=`printf "%02d" ${mth}`
  ncap2 -O -s "dpm=0.0*date+${dpm[${mth}]}" \
    ${drc_out}/${caseid}_clm${mm}.nc ${drc_out}/${caseid}_clm${mm}.nc
done # end loop over mth

Method 2: Create dpm by aggregating small files
for mth in {1..12}; do
  mm=`printf "%02d" ${mth}`
  ncap2 -O -v -s "dpm=${dpm[${mth}]}" ~/nco/data/in.nc \
    ${drc_out}/foo_${mm}.nc
done # end loop over mth
ncecat -O -D 1 -p ${drc_out} -n 12,2,2 foo_${mm}.nc foo.nc
ncrename -O -D 1 -d record,time ${drc_out}/foo.nc
ncatted -O -h \
  -a long_name,dpm,o,c,"Days per month" \
  -a units,dpm,o,c,"days" \
  ${drc_out}/${caseid}_clm_0112.nc
ncks -A -v dpm ${drc_out}/foo.nc ${drc_out}/${caseid}_clm_0112.nc

Method 3: Create small netCDF file using ncgen
cat > foo.cdl << 'EOF'
netcdf foo { 
dimensions:
	time=unlimited;
variables:
	float dpm(time);
	dpm:long_name="Days per month";
	dpm:units="days";
data:
	dpm=31,28.25,31,30,31,30,31,31,30,31,30,31;
}
EOF
ncgen -b -o foo.nc foo.cdl
ncks -A -v dpm ${drc_out}/foo.nc ${drc_out}/${caseid}_clm_0112.nc

Another way to get correct monthly weighting is to average daily
output files, if available.  

************************************************************************
Task 3: Regional processing
************************************************************************
Let's say you are interested in examining the California region.
Hyperslab your dataset to isolate the appropriate latitude/longitudes.

ncks -O -D 1 -d lat,30.0,37.0 -d lon,240.0,270.0 \ 
    ${drc_out}/${caseid}_clm_0112.nc \
    ${drc_out}/${caseid}_clm_0112_Cal.nc

The dataset is now much smaller!
To examine particular metrics.

************************************************************************
Task 4: Accessing data stored remotely
************************************************************************
OPeNDAP server examples:

UCI DAP servers:
ncks --trd -M -p http://dust.ess.uci.edu/cgi-bin/dods/nph-dods/dodsdata in.nc
ncrcat -O -C -D 3 \
  -p http://dust.ess.uci.edu/cgi-bin/dods/nph-dods/dodsdata \
  -l /tmp in.nc in.nc ~/foo.nc

Unidata DAP servers:
ncks --trd -M -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc
ncrcat -O -C -D 3 \
  -p http://thredds-test.ucar.edu/thredds/dodsC/testdods \
  -l /tmp in.nc in.nc ~/foo.nc

NOAA DAP servers:
ncwa -O -C -a lat,lon,time -d lon,-10.,10. -d lat,-10.,10. -l /tmp -p \
http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis.dailyavgs/surface \
pres.sfc.1969.nc ~/foo.nc

LLNL PCMDI IPCC OPeNDAP Data Portal: 
ncks --trd -M -p http://username:password@esgcet.llnl.gov/cgi-bin/dap-cgi.py/ipcc4/sresa1b/ncar_ccsm3_0 pcmdi.ipcc4.ncar_ccsm3_0.sresa1b.run1.atm.mo.xml

Earth System Grid (ESG): http://www.earthsystemgrid.org

caseid='b30.025.ES01' 
CCSM3.0 1% increasing CO2 run, T42_gx1v3, 200 years starting in year 400
Atmospheric post-processed data, monthly averages, e.g.,
/data/zender/tmp/b30.025.ES01.cam2.h0.TREFHT.0400-01_cat_0449-12.nc
/data/zender/tmp/b30.025.ES01.cam2.h0.TREFHT.0400-01_cat_0599-12.nc

ESG supports password-protected FTP access by registered users
NCO uses the .netrc file, if present, for password-protected FTP access 
Syntax for accessing single file is, e.g.,
ncks -O -D 3 \
  -p ftp://climate.llnl.gov/sresa1b/atm/mo/tas/ncar_ccsm3_0/run1 \
  -l /tmp tas_A1.SRESA1B_1.CCSM.atmm.2000-01_cat_2099-12.nc ~/foo.nc 

# Average surface air temperature tas for SRESA1B scenario
# This loop is illustrative and will not work until NCO correctly
# translates '*' to FTP 'mget' all remote files
for var in 'tas'; do
for scn in 'sresa1b'; do
for mdl in 'cccma_cgcm3_1 cccma_cgcm3_1_t63 cnrm_cm3 csiro_mk3_0 \
gfdl_cm2_0 gfdl_cm2_1 giss_aom giss_model_e_h giss_model_e_r \
iap_fgoals1_0_g inmcm3_0 ipsl_cm4 miroc3_2_hires miroc3_2_medres \
miub_echo_g mpi_echam5 mri_cgcm2_3_2a ncar_ccsm3_0 ncar_pcm1 \
ukmo_hadcm3 ukmo_hadgem1'; do
for run in '1'; do
        ncks -R -O -D 3 -p ftp://climate.llnl.gov/${scn}/atm/mo/${var}/${mdl}/run${run} -l ${DATA}/${scn}/atm/mo/${var}/${mdl}/run${run} '*' ${scn}_${mdl}_${run}_${var}_${yyyymm}_${yyyymm}.nc
done # end loop over run
done # end loop over mdl
done # end loop over scn
done # end loop over var

cd sresa1b/atm/mo/tas/ukmo_hadcm3/run1/
ncks -H -m -v lat,lon,lat_bnds,lon_bnds -M tas_A1.nc | m
bds -x 096 -y 073 -m 33 -o ${DATA}/data/dst_3.75x2.5.nc # ukmo_hadcm3
ncview ${DATA}/data/dst_3.75x2.5.nc

# msk_rgn is California mask on ukmo_hadcm3 grid
# area is correct area weight on ukmo_hadcm3 grid
ncks -A -v area,msk_rgn ${DATA}/data/dst_3.75x2.5.nc \
${DATA}/sresa1b/atm/mo/tas/ukmo_hadcm3/run1/area_msk_ukmo_hadcm3.nc 

Template for standardized data:
${scn}_${mdl}_${run}_${var}_${yyyymm}_${yyyymm}.nc

e.g., raw data
${DATA}/sresa1b/atm/mo/tas/ukmo_hadcm3/run1/tas_A1.nc
becomes standardized data

Level 0: raw from IPCC site--no changes except for name 
         Make symbolic link name match raw data
Template: ${scn}_${mdl}_${run}_${var}_${yyyymm}_${yyyymm}.nc

ln -s -f tas_A1.nc sresa1b_ukmo_hadcm3_run1_tas_200101_209911.nc
area_msk_ukmo_hadcm3.nc

Level I: Add all variables (not standardized in time)
         to file containing msk_rgn and area
Template: ${scn}_${mdl}_${run}_${yyyymm}_${yyyymm}.nc

/bin/cp area_msk_ukmo_hadcm3.nc sresa1b_ukmo_hadcm3_run1_200101_209911.nc
ncks -A -v tas sresa1b_ukmo_hadcm3_run1_tas_200101_209911.nc \
               sresa1b_ukmo_hadcm3_run1_200101_209911.nc
ncks -A -v pr  sresa1b_ukmo_hadcm3_run1_pr_200101_209911.nc \
               sresa1b_ukmo_hadcm3_run1_200101_209911.nc

If already have file then:
mv sresa1b_ukmo_hadcm3_run1_200101_209911.nc foo.nc
/bin/cp area_msk_ukmo_hadcm3.nc sresa1b_ukmo_hadcm3_run1_200101_209911.nc
ncks -A -v tas,pr foo.nc sresa1b_ukmo_hadcm3_run1_200101_209911.nc

Level II: Correct # years, months
Template: ${scn}_${mdl}_${run}_${var}_${yyyymm}_${yyyymm}.nc

ncks -d time,....... file1.nc file2.nc 
ncrcat file2.nc file3.nc sresa1b_ukmo_hadcm3_run1_200001_209912.nc

Level III: Many derived products from level II, e.g., 

      A. Global mean timeseries
      ncwa -w area -a lat,lon \
           sresa1b_ukmo_hadcm3_run1_200001_209912.nc \
	   sresa1b_ukmo_hadcm3_run1_200001_209912_xy.nc

      B. Califoria average timeseries
      ncwa -m msk_rgn -w area -a lat,lon \
           sresa1b_ukmo_hadcm3_run1_200001_209912.nc \
	   sresa1b_ukmo_hadcm3_run1_200001_209912_xy_Cal.nc
@end verbatim
@end example

@html
<a name="bibliography"></a> <!-- http://nco.sf.net/nco.html#bibliography -->
<a name="bib"></a> <!-- http://nco.sf.net/nco.html#bib -->
@end html
@node mybibnode, General Index, CCSM Example, Top
@chapter References
@itemize
@mybibitem{ZeM07} Zender, C. S., and H. J. Mangalam (2007), Scaling Properties of Common Statistical Operators for Gridded Datasets, Int. J. High Perform. Comput. Appl., 21(4), 485-498, doi:10.1177/1094342007083802.
@mybibitem{Zen08} Zender, C. S. (2008), Analysis of Self-describing Gridded Geoscience Data with netCDF Operators (NCO), Environ. Modell. Softw., 23(10), 1338-1342, doi:10.1016/j.envsoft.2008.03.004.
@mybibitem{WZJ07} Wang, D. L., C. S. Zender, and S. F. Jenks (2007), DAP-enabled Server-side Data Reduction and Analysis, Proceedings of the 23rd AMS Conference on Interactive Information and Processing Systems (IIPS) for Meteorology, Oceanography, and Hydrology, Paper 3B.2, January 14-18, San Antonio, TX. American Meteorological Society, AMS Press, Boston, MA.
@mybibitem{ZMW06} Zender, C. S., H. Mangalam, and D. L. Wang (2006), Improving Scaling Properties of Common Statistical Operators for Gridded Geoscience Datasets, Eos Trans. AGU, 87(52), Fall Meet. Suppl., Abstract IN53B-0827.
@mybibitem{ZeW07} Zender, C. S., and D. L. Wang (2007), High performance distributed data reduction and analysis with the netCDF Operators (NCO), Proceedings of the 23rd AMS Conference on Interactive Information and Processing Systems (IIPS) for Meteorology, Oceanography, and Hydrology, Paper 3B.4, January 14-18, San Antonio, TX. American Meteorological Society, AMS Press, Boston, MA.
@mybibitem{WZJ06} Wang, D. L., C. S. Zender, and S. F. Jenks (2006), Server-side netCDF Data Reduction and Analysis, Eos Trans. AGU, 87(52), Fall Meet. Suppl., Abstract IN53B-0826.
@mybibitem{WZJ073} Wang, D. L., C. S. Zender, and S. F. Jenks (2007), Server-side parallel data reduction and analysis, in Advances in Grid and Pervasive Computing, Second International Conference, GPC 2007, Paris, France, May 2-4, 2007, Proceedings. IEEE Lecture Notes in Computer Science, vol. 4459, edited by C. Cerin and K.-C. Li, pp. 744-750, Springer-Verlag, Berlin/Heidelberg, doi:10.1007/978-3-540-72360-8_67.
@mybibitem{WZJ074} Wang, D. L., C. S. Zender and S. F. Jenks (2007), A System for Scripted Data Analysis at Remote Data Centers, Eos Trans. AGU, 88(52), Fall Meet. Suppl., Abstract IN11B-0469.
@mybibitem{WZJ081} Wang, D. L., C. S. Zender and S. F. Jenks (2008), Cluster Workflow Execution of Retargeted Data Analysis Scripts, Proceedings of the 8th IEEE Int'l Symposium on Cluster Computing and the Grid (IEEE CCGRID '08), pp. 449-458, Lyon, France, May 2008.
@mybibitem{WZJ091} Wang, D. L., C. S. Zender, and S. F. Jenks (2009), Efficient Clustered Server-side Data Analysis Workflows using SWAMP, Earth Sci. Inform., 2(3), 141-155, doi:10.1007/s12145-009-0021-z.
@mybibitem{PFT88} Press, Flannery, Teukolsky, and Vetterling (1988), Numerical Recipes in C, Cambridge Univ. Press, New York, NY.
@end itemize

@c @node Name Index, General Index, Operators, Top
@c @unnumbered Function and Variable Index

@c @printindex fn

@html
<a name="index"></a> <!-- http://nco.sf.net/nco.html#index -->
<a name="idx"></a> <!-- http://nco.sf.net/nco.html#idx -->
@end html
@node General Index,  , mybibnode, Top
@unnumbered General Index

@syncodeindex fn cp
@printindex cp

@c TTFN (Ta ta for now)
@bye
