easel.tex

The easel (esl) module implements a small set of functionality shared
by all the modules: notably, the error-handling system.

\section{Error handling conventions}

Easel might be used in applications ranging from small command line
utilities to complex graphical user interfaces and parallel
systems. Simple and complex applications have different needs for how
errors should be handled by a library.

In a simple application, we don't want to write a lot of code to
checking return codes for unexpected problems. We would prefer to have
Easel crash out with an appropriate message to \ccode{stderr} -- after
all, that's all a simple application would do anyway.

On the other hand, there are certain problems that even the simplest
command-line applications should handle gracefully. Errors involving
user input (including typos in command line arguments, bad file
formats, nonexistent files, or bad file permissions) are ``normal''
and should be expected. Users will do anything.

In a complex application, we may want to guarantee that execution
never terminates within a library routine. In this case, library
functions always need to return control to the application, even in
the most unexpected circumstances, so the application can fail
gracefully. A failure in an Easel routine should not suddenly crash a
whole graphical user environment, for example. Additionally, because a
complex application may not even be associated with a terminal, a
library cannot count on printing error messages directly to
\ccode{stderr}.

These considerations motivate Easel's error handling conventions.
Most Easel procedures return an integer status code. An \ccode{eslOK}
code indicates that the procedure succeeded. A nonzero code indicates
an error. Easel distinguishes two kinds of errors:

\begin{itemize}
\item \textbf{Failures} include normal ``errors'' (like a read failing
  when the end of a file is reached), and errors that are the user's
  fault, such as bad input (which are also normal, because users will
  do anything.) We say that failures are \textbf{returned} by Easel
  functions. All applications should check the return status of any
  Easel function that might return a failure code. Relatively few
  Easel functions can return failure codes. The ones that do are
  generally functions having to do with reading user input.

\item \textbf{Exceptions} are errors that are the fault of Easel (bugs
in my code) or your application (bugs in your code) or the system
(resource allocation failures). We say that exceptions are
\textbf{thrown} by Easel functions. By default, exceptions result in
immediate termination of your program. Optionally, you may provide
your own exception handler, in which case Easel functions may return
nonzero exception codes (in addition to any nonzero failure codes).
\end{itemize}

The documentation for each Easel function lists what failure codes it
may return, as well as what exception codes it may throw (if a
nonfatal exception handler has been registered), in addition to the
\ccode{eslOK} normal status code. The list of possible status codes is
shown in Table~\ref{tbl:statuscodes}. There is no intrinsic
distinction between failure codes and exception codes. Codes that
indicate failures in one function may indicate exceptions in another
function.

\begin{table}
\begin{center}
\input{cexcerpts/statuscodes}
\end{center}
\caption{List of all status codes that might be returned by Easel functions.}
\label{tbl:statuscodes}
\end{table}

Not all Easel functions return status codes. \ccode{*\_Create()}
functions that allocate and create new objects usually follow a
convention of returning a valid pointer on success, and \ccode{NULL}
on failure; these are functions that only fail by memory allocation
failure. Destructor functions (\ccode{*\_Destroy()}) always return
\ccode{void}, and must have no points of failure of their own, because
destructors can be called when we're already handling an
exception. Functions with names containing \ccode{Is}, such as
\ccode{esl\_abc\_XIsValid()}, are tests that return \ccode{TRUE} or
\ccode{FALSE}. Finally, there are some ``true'' functions that simply
return an answer, rather than a status code; these must be functions
that have no points of failure.

\subsection{Failure messages} 

When failures occur, often the failure status code is sufficient for
your application to know what went wrong. For instance, \ccode{eslEOF}
means end-of-file, so your application might report \ccode{"premature
end of file"} if it receives such a status code unexpectedly. But for
failures involving a file format syntax problem (for instance) a terse
\ccode{eslESYNTAX} return code is not as useful as knowing
\ccode{"Parse failed at line 42 of file foo.data, where I expected to
see an integer, but I saw nothing"}. When your application might want
more information to format an informative failure message for the
user, the Easel API provides (somewhere) a message buffer called
\ccode{errbuf[]}.

In many cases, file parsers in Easel are encapsulated in objects. In
these cases, the object itself allocates an \ccode{errbuf[]} message
string. (For instance, see the \eslmod{sqio} module and its
\ccode{ESL\_SQFILE} object for sequence file parsing.)  In a few
cases, the \ccode{errbuf[]} is part of the procedure's call API, and
space is provided by the caller. In such cases, the caller either
passes \ccode{NULL} (no failure message is requested) or a pointer to
allocated space for at least \ccode{eslERRBUFSIZE} chars. (For
instance, see the \eslmod{tree} module and the
\ccode{esl\_tree\_ReadNewick()} parser.)

Easel uses \ccode{sprintf()} to format the messages in
\ccode{errbuf[]}'s. Each individual call guarantees that the size of
its message cannot overflow \ccode{eslERRBUFSIZE} chars, so none of
these \ccode{sprintf()} calls represent possible security
vulnerabilities (buffer overrun attacks).


\subsection{Exception handling}

Easel's default exception handler prints a message to \ccode{stderr}
and aborts execution of your program, as in:

\begin{cchunk}
   Easel exception: Memory allocation failed.
   Aborted at file sqio.c, line 42. 
\end{cchunk}

Therefore, by default, Easel handles its own exceptions internally,
and exception status codes are not returned to your
application. Simple applications don't need to worry about checking
for exceptions.

If your application wants to handle exceptions itself -- for instance,
if you want a guarantee that execution will never terminate from
within Easel -- or even if you simply want to change the format of
these messages, you can register a custom exception handler which will
catch the information from Easel and react appropriately. If your
exception handler prints a message and exits, Easel will still just
abort without returning exception codes. If your exception handler is
nonfatal (returning \ccode{void}), Easel procedures then percolate the
exception code up through the call stack until the exception code is
returned to your application.

To provide your own exception handler, you define your exception
handler with the following prototype:

\begin{cchunk}
extern void my_exception_handler(int code, char *file, int line, char *format, va_list arg);
\end{cchunk}

An example implementation of a nonfatal exception handler:

\begin{cchunk}
#include <stdarg.h>

void
my_exception_handler(int code, char *file, int line, char *format, va_list arg)
{
  fprintf(stderr, ``Easel threw an exception (code %d):\n'', code);
  if (format != NULL) vfprintf(stderr, format, arg);
  fprintf(stderr, ``at line %d, file %s\b'', line, file);
  return;
}
\end{cchunk}

The \ccode{code}, \ccode{file}, and \ccode{line} are always
present. The formatted message (the \ccode{format} and \ccode{va\_list
arg}) is optional; the \ccode{format} might be
\ccode{NULL}. (\ccode{NULL} messages are used when percolating
exceptions up a stack trace, for example.)

Then, to register your exception handler, you call
\ccode{esl\_exception\_SetHandler(\&my\_error\_handler)} in your
application. Normally you would do this before calling any other Easel
functions. However, in principle, you can change error handlers at any
time. You can also restore the default handler at any time with
\ccode{esl\_exception\_RestoreDefaultHandler()}.

The implementation of the exception handler relies on a static
function pointer that is not threadsafe. If you are writing a threaded
program, you need to make sure that multiple threads do not try to
change the handler at the same time.

Because Easel functions call other Easel functions, the function that
first throws an exception may not be the function that your
application called.  If you implement a nonfatal handler, an exception
may result in a partial or complete stack trace of exceptions, as the
original exception percolates back to your application. Your exception
handler should be able to deal with a stack trace. The first exception
code and message will be the most relevant. Subsequent codes and
messages arise from that exception percolating upwards. 

For example, a sophisticated replacement exception handler might push
each code/message pair into a FIFO queue. When your application
receives an exception code from an Easel call, your application can
might then access this queue, and see where the exception occurred in
Easel, and what messages Easel left for you. A less sophisticated
replacement exception handler might just register the first
code/message pair, and ignore the subsequent exceptions from
percolating up the stack trace. Note the difference between the
exception handler that you register with Easel (which operates inside
Easel, and must obey Easel's conventions) and any error handling you
do in your own application after Easel returns a nonzero status code
to you (which is your own business).

Although each function's documentation \emph{in principle} lists all
thrown exceptions, \emph{in practice}, you should not trust this
list. Because of exceptions percolating up from other Easel calls, it
is too easy to forget to document all possible exception
codes.\footnote{Someday we should combine a static code analyzer with
a script that understands Easel's exception conventions, and automate
the enumeration of all possible codes.} If you are catching
exceptions, you should program defensively here, and always have a
failsafe catch for any nonzero return status. For example, a minimal
try/catch idiom for an application calling a Easel function is
something like:

\begin{cchunk}
     int status;
     if ((status = esl_foo_function()) != eslOK)  my_failure();
\end{cchunk}

Or, a little more complex one that catches some specific errors, but
has a failsafe for everything else, is:

\begin{cchunk}
     int status;
     status = esl_foo_function();
     if      (status == eslEMEM) my_failure("Memory allocation failure");
     else if (status != eslOK)   my_failure("Unexpected exception %d\n\", status);
\end{cchunk}


\subsection{Violations}

Internally, Easel also distinguishes a third class of error, termed a
\textbf{fatal violation}. Violations never arise in production code;
they are used to catch bugs during development and testing. Violations
always result in immediate program termination. They are generated by
two mechanisms: from assertions that can be optionally enabled in
development code, or from test harnesses that call the always-fatal
\ccode{esl\_fatal()} function when they detect a problem they're
testing for.


\subsection{Internal API for error handling}

You only need to understand this section if you want to understand
Easel's source code (or other code that uses Easel conventions, like
HMMER), or if you want to use Easel's error conventions in your own
source code.

The potentially tricky design issue is the following. One the one
hand, you want to be able to return an error or throw an exception
``quickly'' (in less than a line of code). On the other hand, it might
require several lines of code to free any resources, set an
appropriate return state, and set the appropriate nonzero status code
before leaving the function. 

Easel uses the following error-handling macros:

\begin{center}
{\small
\begin{tabular}{|ll|}\hline
\ccode{ESL\_FAIL(code, errbuf, mesg, ...)}   & Format errbuf, return failure code. \\
\ccode{ESL\_EXCEPTION(code, mesg, ...)}      & Throw an exception, return exception code. \\
\ccode{ESL\_XFAIL(code, errbuf, mesg, ...)}  & A failure message, with cleanup convention.\\
\ccode{ESL\_XEXCEPTION(code, mesg, ...)}     & An exception, with cleanup convention.\\
\hline
\end{tabular}
}
\end{center}

They are implementated in \ccode{easel.h} as:

\input{cexcerpts/error_macros}

The \ccode{ESL\_FAIL} and \ccode{ESL\_XFAIL} macros are only used when
a failure message needs to be formatted. For the simpler case where we
just return an error code, Easel simply uses \ccode{return code;} or
\ccode{status = code; goto ERROR;}, respectively.

The \ccode{X} versions, with the cleanup convention, are sure to
offend some programmers' sensibilities. They require the function to
provide an \ccode{int status} variable in scope, and they require an
\ccode{ERROR:} target for a \ccode{goto}. But if you can stomach that,
they provide for a fairly clean idiom for catching exceptions and
cleaning up, and cleanly setting different return variable states on
success versus failure, as illustrated by this pseudoexample:

\begin{cchunk}
int 
foo(char **ret_buf, char **ret_fp)
{
    int status;
    char *buf = NULL;
    FILE *fp  = NULL;

    if ((buf = malloc(100))  == NULL) ESL_XEXCEPTION(eslEMEM,      "malloc failed");
    if ((fp  = fopen("foo")) == NULL) ESL_XEXCEPTION(eslENOTFOUND, "file open failed");

    *ret_buf = buf;
    *ret_fp  = fp;
    return eslOK;

  ERROR:
    if (buf != NULL) free(buf);  *ret_buf = NULL;
    if (fp  != NULL) fclose(fp); *ret_fp  = NULL;
    return status;
}
\end{cchunk}

Additionally, for memory allocation and reallocation, Easel implements
two macros \ccode{ESL\_ALLOC()} and \ccode{ESL\_RALLOC()}, which
encapsulate standard \ccode{malloc()} and \ccode{realloc()} calls
inside Easel's exception-throwing convention.


\vspace*{\fill}
\begin{quote}
\emph{Only a complete outsider could ask your question. Are there
control authorities? There are nothing but control authorities. Of
course, their purpose is not to uncover errors in the ordinary meaning
of the word, since errors do not occur and even when an error does in
fact occur, as in your case, who can say conclusively that it is an
error?}\\ \hspace*{\fill} -- Franz Kafka, \emph{The Castle}
\end{quote}     


\section{Memory management}


\section{Replacements for C library functions}


\section{Standard banner for Easel miniapplications}


\section{File and path name manipulation}


\subsection{Secure temporary files}

A program may need to write and read temporary files.  Many of the
methods for creating temporary files, even using standard library
calls, are known to create exploitable security holes
\citep{Wheeler03,ChenDeanWagner04}.

Easel provides a secure and portable POSIX procedure for obtaining an
open temporary file handle, \ccode{esl\_tmpfile()}. This replaces the
ANSI C \ccode{tmpfile()} function, which is said to be insecurely
implemented on some platforms.  Because closing and reopening a
temporary file can create an exploitable race condition under certain
circumstances, \ccode{esl\_tmpfile()} does not return the name of the
invisible file it creates, only an open \ccode{FILE *} handle to
it. The tmpfile is not persistent, meaning that it automatically
vanishes when the \ccode{FILE *} handle is closed. The tmpfile is
created in the usual system world-writable temporary directory, as
indicated by \ccode{TMPDIR} or \ccode{TMP} environment variables, or
\ccode{/tmp} if neither environment variable is defined.

Still, it is sometimes useful, even necessary, to close and reopen a
temporary file. For example, Easel's own test suites generate a
variety of input files for testing input parsers.  Easel also provides
the \ccode{esl\_tmpfile\_named()} procedure for creating a persistent
tmpfile, which returns both an open \ccode{<FILE *>} handle and the
name of the file. Because the tmpfile name is known, the file may be
closed and reopened.  \ccode{esl\_tmpfile\_named()} creates its files
relative to the current working directory, not in \ccode{TMPDIR}, in
order to reduce the chances of creating the file in a shared directory
where a race condition might be exploited. Nonetheless, secure use of
\ccode{esl\_tmpfile\_named()} requires that you must only reopen a
tmpfile for reading only, not for writing, and moreover, you must not
trust the contents.  (It may be possible for an attacker to replace
the tmpfile with a symlink to another file.)

An example that shows both tmpfile mechanisms:

\input{cexcerpts/easel_example_tmpfiles}

\section{Internals}

\subsection{Input maps}

An \esldef{input map} is for converting input ASCII symbols to
internal encodings. It is a many-to-one mapping of the 128 7-bit ASCII
symbol codes (0..127) onto new ASCII symbol codes. It is defined as
an \ccode{unsigned char inmap[128]} or a \ccode{unsigned char *}
allocated for 128 entries.

Input maps are used in two contexts: for filtering ASCII text input
into internal text strings, and for converting ASCII input or internal
ASCII strings into internal digitized sequences (an \eslmod{alphabet}
object contains an input map that it uses for digitization).

The rationale for input maps is the following. The ASCII strings that
represent biosequence data require frequent massaging. An input file
might have sequence data mixed up with numerical coordinates and
punctuation for human readability. We might want to distinguish
characters that represent residues (that should be input) from
characters for coordinates and punctuation (that should be ignored)
from characters that aren't supposed to be present at all (that should
trigger an error or warning). Also, in representing a sequence string
internally, we might want to map the symbols in an input string onto a
smaller internal alphabet. For example, we might want to be
case-insensitive (allow both T and t to represent thymine), or we
might want to allow an input T to mean U in a program that deals with
RNA sequence analysis, so that input files can either contain RNA or
DNA sequence data.  Easel reuses the input map concept in routines
involved in reading and representing input character sequences, for
example in the \eslmod{alphabet}, \eslmod{sqio}, and \eslmod{msa}
modules.