There should be more nodes than listed below, but at least the chapters are isolated.

Introduction
************

This document explains how to use the Test package to test GNU Emacs packages.  It is an extension of Chapter 22 [Compiling and Testing Programs] of the GNU Emacs Reference Manual.

Purpose of the Package
======================

This package provides tools to aid in developing regression tests for Emacs packages and to assess the efficacy of those tests.  GNU Emacs packages, written in a dialect of Lisp, provide extensions to the basic text editing features of Emacs.  Most of these packages perform state-changing operations.  For example, a package may create new buffers, modify text, change the locations of the cursor and various marks, write files or modify anything else accessible to Emacs (which is just about everything in Unix). Traditional Lisp debugging strategies are not sufficient to adequately test a package that performs extensive state-manipulation.  The Test package provides a strategy that is.

Test Approach
=============

The Test system is designed to support structural (or ``white-box'') testing.  (Another commonly used test approach is functional, or ``black-box'' testing.)  It automates the process of running a number of tests cases designed to analyze ``coverage'' of the package (that is, find parts of the code that are not exercised) and ``expression'' (that is, what values are returned by each expression, and which expressions always returned the same values).

Testing Method
==============

Test provides mechanisms to ease the creation and execution of sets of tests.  Some of the features are designed to simplify testing activity, while others are designed to aid the analysis of testing.

Test supports coverage testing.  That is, it provides instrumentation mechanisms designed to check for inadequacies of testing of the following two forms:

  * Failure to execute an expression.

  * Failure to produce more than one value for an expression.

In each case, the fault may be in the tests (i.e., not enough testing was performed), or the fault may be in the program (i.e., an ``unnecessarily complex'' program has been written).  (There is also the possibility that no fault exists, such as the use of ``constant'' functions for readability or style.  In practice these cases are easy to identify and ignore.)

You should use the following strategy in creating tests with this system:

 1. Create a simple set of tests that appear to be necessary to test the basic functionality of the package.

 2. Add additional tests to ensure that every expression is executed, and that each expression produces at least two values over the entire set of tests.

 3. Run the set of tests with instrumentation to check for inadequacies.

 4. If there are reported inadequacies, then correct the code or improve the testing, and go back to the previous step.

There is no guarantee that a set of tests that passes these coverage checks will find all errors.  But, such a set of tests has the property that every expression in the code has been exercised in a minimally significant way (i.e., every expression has produced at least two different values).  In practice, a great number of errors can be detected with this method.

The importance of coverage testing increases as a package is modified. Changes to one part of a package may have unexpected effects on other (unconsidered) parts.  Coverage testing ensures that all parts of a package are tested, at least minimally.  Coverage tests also serve as a specification of the functionality of a package.

Figure 1 illustrates the overall flow of events using this test approach.

             +------------+                         +--------------+
             | Instrument |                         | Generate     |
             | o Control  |                         | o Test Cases |
             | o Data     |                         | o Script(s)  |
             +------------+                         +--------------+
                           \                       /     ^
                            \                     /      |
                             \                   /       |
                              \                 /        |
                               \               /         |
                                \             /          |
                                +-------------+          |
                                |   Execute   |          |
                                |     the     |          |
                                |   Test(s)   |          |
                                +-------------+          |
                                       |                 |
                                       |                 |
                                +-------------+          |
                                |   Analyze,  |          |
                                |   Display   |          |
                                |   Results   |          |
                                +-------------+          |
                                       |                 |
                                       |                 |
                                       ^                 |
                                     /   \               |
                                    /     \  No          |
                                   < Done? >-------------+
                                    \     /
                                     \   /
                                      \ /
                                       v
                                       | Yes
                                       |(Test Goals Achieved)
                                       V

Figure 1.  General Test Algorithm

Overview
========

The Test product consists of four major subsystems:

  * Regression Subsystem -- a set of functions to support regression testing of Emacs packages.

  * Coverage Subsystem -- a set of functions to support coverage testing of Emacs packages.

  * Annotation Subsystem -- a set of functions to provide support for creating, using and displaying information about Emacs Lisp code.

  * Equality Subsystem -- a set of functions used by the testing subsystems that provide different interpretations of equality.

The following sections explain the use of these functions to support the steps in the testing process:

  * Getting Started

  * Instrumenting a package

  * Testing a Package

  * Analyzing Test Results

  * Displaying Test Results

The system is designed to test Emacs-Lisp programs, and is implemented as a minor mode of Emacs-Lisp mode (See Section 28.1 [Minor Modes] of the GNU Emacs Reference Manual).

Getting Started
***************

This section describes how to begin using the Test package to test Emacs packages.

Loading and Initialization
==========================

To start testing you probably want a fresh Emacs session with no extraneous buffers or processes hanging around.  Non-essential extras will just slow down the state capture and achieve functions.  If you have a busy .emacs file you may want to keep a trimmed back copy just for test sessions.

Once in Emacs visit the file containing Emacs-lisp code to be tested, then enter:

     M-x load-file
     test.el
     M-x test-mode

The mode line at the bottom of your window should show ``(Emacs-lisp Test)'' in the mode field.  You are now in Test mode.

Probably the first thing to do at this point is to use C-c n to generate a new buffer containing a test script template.  This is a good starting point for creating a new test script; the template is shown in section 2.3 of this document.  The task now confronting you is to develop a test suite that will exercise your package throughly, not only to test it now but also to have an automated regression test.  The remainder of this document will guide you through this task.

Figure 2 illustrates the overall flow of data in the Test system.

             +-------------+
             |Source Text  |
             +-------------+
                            \
                             \
                              \
                               Instrument
                              /
                             /
                            /
             +-------------+                     +-------------+
             |Instrumented |                     | Test Cases  |
             |    Text     |                     +-------------+
             +-------------+                    /
                            \                  /
                             \                /
                              \              /
                               \            /
                                   Execute
                                            \
                                             \
                                              \
                                               \
                                                 +-------------+
                                                 |Test Results |
                                                 +-------------+
                                                /        ^
                                               /         |
                                              /          |
                                             /           v
                                      Display         Analyze

Figure 2.  Test Data Flow

Test Mode
=========

The package `test.el' provides several functions to the user:

C-c n
     Start a new test script (tst-new-script-buffer)

C-c i
     (Re-)instrument a test program buffer  (tst-instrument).

C-c x
     Test the program, using the test script  (tst-execute).

C-c c
     Record a GNU Emacs state for later use (tst-capture-state).

C-c a
     Achieve a previously saved GNU Emacs state (tst-achieve-state).

C-c C-a
     Achieve a state previously saved-to-file (tst-achieve-state-from-file).

C-c C-c
     Capture current emacs state to file (tst-capture-state-to-file).

C-c C-r
     Read a previously saved-to-file state into a variable (tst-read-state-from-file).

C-c C-w
     Write a state previously captured in a variable to a file (tst-write-state-to-file).

C-c f
     Analyze (filter) the results of running the test(s)  (tst-analyze).

C-c d
     Display the annotations about the test  (tst-display-mode).

C-c q
     Exit test mode  (test-mode).

Contents of the *test script* Buffer
====================================

     (defun test-script ()
       (interactive)
     					; Local Variables
       (let (pre post actual)
     					; Body
         (switch-to-buffer foo)
         (erase-buffer)
         (insert-file foo)
         (goto-char (point-min))
         (emacs-lisp-mode)
         (test-mode)
         (tst-instrument)
     					; 1st test run
         (tst-read-state-from-file 'post expected-state-file1)
         (tst-achieve-state-from-file 'pre initial-state-file1)
         (foo args)
         (tst-capture-state 'actual nil nil)
     ;;; If you have a lot of tests, consider a while loop
       ) ; let
     ) ; defun

The Example -- box.el
=====================

The package `box.el' provides two functions to the user: box-region and unbox. The first function encloses the current region with a box:

     /********************************************\
     * A boxed region                             *
     * may have lines of text of varying lengths. *
     * A side effect of boxing is to normalize    *
     * the length of lines.                       *
     \********************************************/

The second function undoes the effect of the first by removing any boxes within the region.  This simple package will be used in this document as an example of a package to be tested using the Test package.

Testing Your Package
********************

As mentioned at the beginning of this document, Emacs packages are generally state changing operations.  You invoke the package with a session that is in some arbitrary initial state.  The package modifies selected aspects of this initial state in a defined manner and leaves the session in some final state.  The state of a GNU Emacs session consists of information on buffers, windows, processes, and other global information.  The components of the state that are used in the Test package are listed later in this section.

Regression Testing is a technique for verifying that changes in a package do not unintentionally degrade the quality of the code.  Any changes in one part of the package should not alter the functionality of other parts of the package.  State information plays two roles in regression testing of Emacs packages.  These are:

 1. initiating the run of a package in a consistent and pre-determined initial state, and

 2. comparing the actual final state after the completion of a package to a desired final state.

The regression testing components of test.el allow you to capture session states for later use, achieve these states to set up a test run, and test the equality of states for package verification.

The following example demonstrates these steps for regressions testing an arbitrary package.  Assume that a working version of some package exists. Now suppose you wish to add some new function that requires some modifications in the original code. After adding the new function you need to verify that the original functionality of the package has not changed. In other words, you need to verify that the code has not regressed from its original functionality.  To perform this testing, you capture the initial state of a session, run the original package, and capture the final state.  Following this, you achieve the initial state that you previously captured, run the new version of the package, and perform an equality test to compare the new final state to the one that was captured after the run of the original package.  The states should be equivalent if the test is successful.

Two extensions to this state comparison paradigm are:

 1. Testing for partial equivalence - Each Emacs package is supposed to change certain aspects of a session state, but leave others unchanged.  For example, a package may change the text of the current buffer but should leave the window configuration of the session intact.  Partial equivalence testing consists of the following steps:

      1. Capture the initial state.

      2. Run the function to be tested.

      3. Compare the final state to the initial state.  The states should be equivalent in all areas except those that the test function is supposed to change.

 2. Testing functions that undo the effects of other functions - Some emacs packages have a function that perform some state change and a corresponding function that undoes that change.  Testing these types of packages consists of the following steps:

      1. Capture the initial state.

      2. Run the function that performs the state change.

      3. Run the function that undoes the state change.

      4. Compare the final state to the initial state.  They should be equivalent.

The remainder of this section describes the elements of the Test package that implement these aspects of regression testing. In summary, these are:

  * capture - saves the current state of the Emacs session in either a bound variable or UNIX file.

  * partial capture - save part of the current state of the Emacs session.  Some or all global variables may be excluded, and a subset of buffers may be selected.

  * achieve - sets the state of the current session to one that is either in a bound variable or a UNIX file.

  * equality - compares two emacs sessions states for equivalence.

  * partial equality - compares selected aspects of two emacs session states for equivalence.

  * inequality - using user defined hooks, compares two session states within certain degrees of inequality (e.g. compares buffer contents ignoring whitespace).

Capturing and Achieving Emacs States
====================================

GNU Emacs State Definition
--------------------------

The capture and achieve functions of test.el represent an Emacs state as the hierarchy of data shown below.

  * Global Bound Symbols - Most of the information about an Emacs session is contained in the values of symbols global to the session. In conventional programming language terms, this session information can be viewed as the values of global variables.  Some examples of common global bound symbols are:

       * global-abbrev-table - the list of abbreviations global to the session and their meanings.

       * global-map - a vector describing the mappings of all keys to Emacs commands.

       * debug-on-error - a flag that indicates that the Emacs debugger should be called on an error (if non-nil).

  * Buffer Information - Each Emacs session contains one or more buffers.  The capture functions store the following information for each active buffer:

       * buffer name - a unique name that identifies the buffer.

       * file name - if the buffer is associated with a UNIX file, the name of that file.

       * point - the value of the cursor point in the buffer.

       * mark - the  value of the buffer's mark.

       * contents - the contents of the buffer represented as a text string.

       * modified - a flag indicating if the buffer has been modified since last saved.

       * local map - the local key map for this buffer.

       * local bound symbols - symbols (variables) that are local to this buffer (e.g., the current-mode of the buffer).

  * Window Information - At any time an Emacs session has one or more windows visible.  The capture functions store the following information for each window:

       * edges - the upper left and lower right corner of the window in screen coordinates.

       * buffer - the buffer occupying this window.

       * start - the integer position in the buffer where display starts in the respective window.

       * point - the integer position in the buffer where the point for this window sits.

       * current flag - a flag that is non-nil if this is the window in which the screen cursor currently is located.

  * Process Information - An Emacs session may have invoked several UNIX processes during its lifetime.  These processes may have completed or may still be running.  Two common processes that run during an Emacs sessions are the shell and the process that displays the time, date and system load on the mode line.  The Test capture functions store the follow information for each process:

       * buffer - the name of the buffer to which the process is attached (nil if the process is not related to a buffer, e.g., the display time process).

       * process mark - the marker for the end of the last output from the process.

       * command - the UNIX command that invoked the process.

       * exit status - if the process has completed, the UNIX return code from that process.

       * filter - the name of an Emacs lisp function that receives all output of the process.

       * name - a unique name of the program invoked in the process.

       * sentinel - the name of an Emacs lisp function that is called when the process changes state (e.g. completes).

       * status - indicates whether the process is running, complete, etc.

A significant item that is excluded from this state information is the current definition of bound functions.  Recall that an Emacs user can define any new function with a defun and load that function. Furthermore, the user can redefine any existing function (e.g. forward-char !) in this manner.  Saving the current definition of all bound functions would result in a huge state vector, and is therefore not done.

Capturing an Emacs Session State
--------------------------------

When regression testing an Emacs package, you will mainly use the state capture functions of Test in two ways:

 1. to save a known initial state to use as the base state for a run of an Emacs package, and

 2. to save a final state to later compare to the state of the session after the run of a modified package.

Commands for Capturing State
............................

tst-capture-state-to-file
     Writes the current state of the Emacs session to a file.  You are prompted for the name of the file, the exclude-variables and the buffer-list.  Refer to the Usage Notes below for details on these last two items.  If the file you specify already exists it is overwritten.

tst-write-state-to-file
     Writes a state that is saved in a bound variable to a file.  You are prompted for the state variable name (which was created with tst-capture-state) and the name of the file.  If the file already exists it is overwritten.

tst-capture-state
     Saves the state of the Emacs session in a state variable.  You are prompted for the name of the file, the exclude-variables and the buffer-list.  Refer to the Usage Notes below for more details on these last two items.  If the bound variable you specify already exists, it is overwritten.

Usage Notes
...........

The amount of information saved by tst-capture-state-to-file and tst-capture-state is controlled by your responses to the command prompts.  These are:

List of buffers to capture:
     There are three possible responses to this prompt:

      1. nil to indicate that state information on all buffers should be saved (the default).

      2. A list of of buffer names indicating for which buffers state information should be saved.  For example, to save only the states of buffers bar and foo you enter:

                      ("bar" "foo")

      3. A singleton list that is the name of a non-existent buffer if you do not wish to save any buffer state information.  For example, if no buffer xxx exists you could enter

                      ("xxx")

          to specify that capture not save any buffer state information.

List global vars to exclude:
     There are three possible responses to this prompt:

      1. nil to capture all global variables.

      2. all to exclude all global variables from the captured state.

      3. A list of global variable names to exclude from the captured state.  For example, if you enter

                       ("obarray" "values")

          the two global variables, obarray and values, are not captured.  This value is the default for the prompt since these two variables have very large values that are rarely of use in testing.

Achieving an Emacs State
------------------------

When using Test you will mainly use the achieve state functions to return your session to a known initial state.  With these functions you can make repeated test runs by simply achieving the initial state between each run.

Commands for State Achieve
..........................

tst-read-state-from-file
     Reads a session state from a file into a bound symbol.  You are prompted for the name of the file and the name of the bound symbol.  The file should have been created by capture-state-to-file or write-state-to-file.  If the bound symbol already exists, it will be overwritten.

tst-achieve-state-from-file
     Sets the state of the emacs session to that saved in a file.  You are prompted for the name of the file.  The file should have been created by capture-state-to-file or write-state-to-file.

tst-achieve-state
     Sets the state of the emacs session to that saved in a bound symbol.  You are prompted for the name of the bound symbol.  The symbol must have been created by tst-capture-state.

Usage Notes
...........

Achieving the state of a previous Emacs session is a non-trivial task that can have many unexpected side-effects.  These are listed below:

 1. If you want achieve to preserve all existing buffers that were not part of the state you are achieving, you may set the variable tst-achieve-buffers-nondestructively non nil.

 2. Achieving a previous state completely resets your Emacs session.  Any information in your current state will be overwritten by the achieved state.  For example, any changes in buffers will not exist after the achieve process.  You should, therefore, make sure that you save any changes to disk before running achieve.

 3. If you are not careful you may find that previous versions of disk files overwrite current changed versions.  The following scenario demonstrates this problem.

      1. Suppose you have a file mytest that is in a buffer of an emacs session at the time you run tst-capture-state-to-file.

      2. Now suppose you make changes to this file over a period of a few days, and then run tst-achieve-state-from-file.

      3. Your emacs session will now contain the old version of the file.

      4. If written to disk, this old version will overwrite the current version of mytest.

     You can avoid this problem by testing packages only on dummy test files, or using a special test directory with copies of files that are permanently stored in other directories, or by not capturing buffer mytest and setting tst-achieve-buffers-nondestructively to t.

 4. The state of an emacs session is partially dependent on the state of your entire UNIX session.  Therefore, the achieve state functions may not be able to restore your state to the exact one that was captured.  For example, a process can not be started if the run image for the program no longer exists on disk.  When testing and comparing states, it is best to rely only on the more common aspects of an Emacs session (e.g. buffers, windows).

 5. As explained earlier, the capture functions do not save all aspects of the session.  A significant example is the current definition of all bound functions.  Therefore, the achieve functions will not restore these aspects of the session state.

Testing for equality
====================

The equality package of Test compares two Emacs states for equivalence by invoking special equality functions on each of the data components captured in the state. Each of these equality functions compares only a small part of the entire state. The results of these comparisons are written into the buffer *equal-log*. If two states are not equal then you can determine where components differ by looking at this buffer.

The section Equality Functions provides a list of all of the equivalence testing functions.  The section Partial Equality explains how you can accomplish testing a subset of a state. This is useful if you are only interested in differences between certain aspects of state, such as only the contents of buffers, but you do not care to hear about differences in other components, such as windows.  The section Inequality explains how to add hooks to the equality functions. These hooks are called by the equality functions when two components are not equal but before a result is returned. A hook that you write can be inserted at this point and change a ``not equal'' result into an ``equal.''

Equality Functions
------------------

The functions to compare states for equality are divided into two categories.  One set of functions compares complete states, the second set of functions only uses the buffer components.

The following state equality functions are defined for interactive use.

tst-equ-state
     Compares two states for equality.  Each component in the first state is compared to its corresponding component in the second state.

tst-equ-sessions
     Compares the sessions component of two states.  The sessions components contains all global variables.  These includes those that define key-maps and syntax tables.

tst-equ-buffers
     Compares the buffers components of two states. Compares all of the buffers in one state with those of a second state.

tst-equ-windows
     Compares the windows components of two states.

tst-equ-process
     Compares the process components of two states.

The following functions operate on just the buffer components of states.  Objects of this type can be extracted from state objects by the function tst-equ-find-buff-with-name.

tst-equ-buffer-state
     Compares two buffers for equality. The following components are checked for equality: point, mark, contents, modified, file, local-variables.

tst-equ-point
     Compares the point component of two buffers.

tst-equ-mark
     Compares the mark component of two buffers.

tst-equ-file
     Compares the file component of two buffers.

tst-equ-modified
     Compares the modified component of two buffers.

tst-equ-contents
     Compares the contents component of two buffers.  Contents are compared as two single strings.

tst-equ-contents-line
     Compares the contents component of two buffers.  Contents are viewed as composed of lines of text, and compared line-by-line.

tst-equ-contents-region
     Compares the contents component of two buffers. Only the contents within a region are compared. Region is delimited by point and mark.

Partial Equality
----------------

If you are interested in only checking the equality of a subset of an Emacs state, then there are four methods you can use.

tst-capture-state
     You can use this function to capture only a subset of state.  Once this has been done, equality testing will only be performed on those parts of the state that where captured.

tst-equ-state-functions
     This variable contains a list of equality functions to be invoked when comparing two states.  You can change it to test only those components of state that interest you.  As an example, you might want to remove tst-equ-windows if you are not interested in differences in windows between states.

tst-equ-buff-state-functions
     This variable contains a list of equality functions to be invoked when comparing two buffer states.  You can change it to test only those components of buffer state that interest you.  As an example, you might want to remove tst-equ-point and tst-equ-mark if you are not interested in those parts of the states.

tst-equ-find-buffer-with-name
     Use this function to obtain only the part of state that is associated with a particular buffer.  You can then pass the result of this function to buffer testing equality functions.

Inequality
----------

Sometimes the comparison of two objects yields ``not equal'' when wish to ignore certain inequalities.  Each of the equality functions will execute a hook, if one is defined, and the comparison of the two objects yields ``not equal.''  Within the hook you can write your own test for equality and change the result of the comparison if you so desire.

For each equality function, there exists a hook symbol that, when defined, will be run if the two objects are not equal.  The name of the hook symbol can be found by adding -hook to the equality function name.  The value of this hook should then be set to the name of the hook function you wish to execute.  From within the the hook the two objects being compared can be accessed by adding 1 and 2 to the equality function name.  The variable tst-equ-result should be set to t or nil from within your hook.

The following example shows a hook, called ignore-zero-points, that is executed whenever the result of comparing the point components of two buffers is not equal.  From within this function the two point components are accessed as tst-equ-point1 and tst-equ-point2.  The hook changes the result of a point comparison from nil to t if one of the points is at position zero.

     (setq tst-equ-point-hook 'ignore-zero-points)

     (defun ignore-zero-points ()
          "Equality point hook, changes result to t if one point
           is at position zero."

     	(if (or (equal tst-equ-point1 0) (equal tst-equ-point2 0))
     	   (setq tst-equ-result t)  ; return t if one is zero
     	;else
     	   (setq tst-equ-result nil); otherwise return nil
             )
     )

Of particular usefulness are the hooks that are associated with the contents of two buffers. The equality function tst-equ-bs-contents compares the contents of two buffers by comparing strings that contain the entire contents of each buffer. Should this comparison fail it is sometimes useful to compare the two strings after eliminating all white-space from each string. The following hook, provided with this package, does such a comparison.

     (setq tst-equ-contents-hook 'ignore-white-space)

     defun ignore-white-space()
        " Compares the contents of two buffers after removing all
          white-space from each."

     	(setq tst-equ-result (string-equal-less-regexp "\\s "
     		tst-equ-contents1 tst-equ-contents2))
     )

A second contents hook of interest is the one associated with the tst-equ-contents-line function. This function compares the contents of two functions on a line-by-line basis.  The hook for this function differs from all other hooks, in that it is called once per line, instead of once per call to its associated function. The two lines are accessed as tst-equ-line1 and tst-equ-line2.

Testing example
===============

The example that follows illustrates how to write an emacs-lisp function that uses the Test package to test another function you have written.  In the example the two user supplied functions under test are: box-region and unbox.

     ;;; test-box uses the Test package to test the functions box and unbox
     ;;;
      test-box ()
       (interactive)
     					; Local Variables
       (let (initial-state		        ; to hold states
              boxed-state
              final-state
              capture-buffer-list
     	)

         ; First create a buffer to use the functions on
         (get-buffer-create "box.junk")
         (set-buffer "box.junk")
         (erase-buffer)
         (insert-file "/project/gnutest/test/box.junk")

         ; Save the initial state
         ; Since we're only concerned with one buffer include
         ;     include that in the capture list
         (setq capture-buffer-list '("box.junk"))
         (tst-capture-state 'initial-state capture-buffer-list nil)

         ; execute the box function
         (goto-char (point-min))
         (set-mark (point-max))
         (box-region nil)

         ; capture the boxed state
         (tst-capture-state 'boxed-state capture-buffer-list nil)

         ; unbox and capture again
         (unbox nil)
         (tst-capture-state 'final-state capture-buffer-list nil)

         ; compare the initial state to the final state
         ; they should be exactly equal
         (if (tst-equ-state initial-state final-state
                            "Compare before box to after unbox")
     	(message "Box/unbox test passed")
     	(message "Box/unbox test failed"))

         )					; let
       )					; defun

Contents of *equal-log*
-----------------------

Each of the equality functions writes the results of its comparison into a buffer named ``*equal-log*''.  If this buffer does not exist, then it is created.  The contents of the buffer are never erased nor is the buffer ever deleted, these actions are left to the user.

The buffer is created in Outline Mode. This allows selective hiding of comparisons that are not of interest to the user.

If the result of a comparison is nil (i.e. not equal) then that entry in the log is flagged with a question mark (?).  In most cases, the two differing objects are also logged as in the following example.

     *  ?State comparison: Compare before box to after unbox

     **    ?Sessions state
     ***	? Global symbols
     	    data-bytes-free   5350948 5176868
     	    this-command   nil kill-region
     	    data-bytes-used   383452 557532
     	    ?post not found in second state
     	    post   nil nil
     	    ?file not found in second state
     	    file
     	    statevar   state post

     **    ?Buffers state

     **    Comparison of buffers named: *scratch*
     ***	 point: "1" "1"
     ***	 mark: nil nil
     ***	 contents: contents equal
     ***	 modified: nil nil
     ***	 file: nil nil
     ***	 local-vars: local variables are equal

     **    Comparison of buffers named:  *Minibuf-0*
     ***	 point: "1" "1"
     ***	 mark: nil nil
     ***	 contents: contents equal
     ***	 modified: nil nil
     ***	 file: nil nil
     ***	 local-vars: local variables are equal

     **    Comparison of buffers named: box.el
     ***	 point: "1" "1"
     ***	 mark: 4480 4480
     ***	 contents: contents equal
     ***	 modified: t t
     ***	 file: nil nil
     ***	 local-vars: local variables are equal

     **    Comparison of buffers named: box.el-instrumented
     ***	 point: "5608" "5608"
     ***	 mark: 5608 5608
     ***	 contents: contents equal
     ***	 modified: t t
     ***	 file: nil nil
     ***	 local-vars: local variables are equal

     **    ?Comparison of buffers named: box.junk
     ***	 ?point: "1" "234"
     ***	 ?mark: 235 1
     ***	 ?contents: contents not equal
     ***	 modified: t t
     ***	 file: nil nil
     ***	 local-vars: local variables are equal

     **    Processes state

     **    Window state

     ***	 window
     ****	     window-edges: (0 0 10 9) (0 0 10 9)
     ****	     window-buffer: "box.junk" "box.junk"
     ****	     window-start: 1 1
     ****	     window-point: 1 1
     ****	     current-window: t t

Instrumenting a Package
***********************

Use the tst-instrument command for instrumenting Lisp code. Instrumenting copies the contents of the current buffer to a new buffer named for the current buffer concatenated with -instrumented.  The new buffer is set to emacs-lisp-mode.  Instrumenting initializes the annotation data base, adds the instrumentation to the copied code, and finally evaluates the entire copied buffer.

Instrumentation Probes
======================

Instrumentation acts on certain lists that represent functions within a defun by adding an instrumentation probe.  Candidate lists are identified and Lisp code is added around the list.  A list representing a function that has the form, (function arg1 arg2), after insertion of the instrumentation probe will have the form, (tst-cover #id (function arg1 arg2)).  Instrumentation assigns the line number of the function within the buffer as the #id.

Cover Function
==============

The instrumentation probe consists of the invocation of the function tst-cover with two arguments, a unique identifier and the function that was instrumented.  Tst-cover uses the identifier as a key into the annotation data base, stores the result of the instrumented function, increments an invocation counter, and returns the result of the instrumented function as its function value.

Instrumentation Example
=======================

Original Code:

     (defun factorial (n)
       (let
           (result)
         (if
             (< n 2)
             (setq result 1)
           (setq result
              (* n
                 (factorial
                  (1- n)))))
         result))

Instrumented Code:

     (defun factorial (n)
       (tst-cover 2 (let
           (result)
         (tst-cover 4 (if
             (tst-cover 5 (< n 2))
             (tst-cover 6 (setq result 1))
           (tst-cover 7 (setq result
              (tst-cover 8 (* n
                 (tst-cover 9 (factorial
                  (tst-cover 10 (1- n))))))))))
         result)))

Analyzing the Test Results
**************************

This section explains how to use the analyze function

Analysis Functions
==================

The Coverage Analysis package is designed to run after a series of tests has been performed using the instrumented version of the code under test. It retrieves data from the annotation database and detects the following testing anomalies:

  * Failure to execute an expression.

  * Failure to produce more than one value for an expression.

When these conditions are detected additional attributes are stored in the annotation database in a format suitable for extraction by the display component of Test.

Analysis is invoked by the command:

     	tst-analyze

If an expression is not executed, the attribute ``zero'' is inserted with a constant value.  This constant value is defined by the following variable:

     	tst-anl-zero-counts

If an expression returns a constant value, the attribute ``constant'' is inserted with the value of the constant.  If an expression is only executed once, it is deemed to return a constant result.

You can re-run the analysis function without re-instrumenting the code under test.  If, for example, an initial set of tests indicates a number of unexecuted expressions, you can run additional tests and analyze the combined results without re-running the complete set of tests.  Note however, that if you re-instrument the code, the annotation database is reinitialized and the results of previous tests, including any analysis results will be lost (unless you explicitly save the annotation database using tst-ann-get-db).

Sample Analysis Results
=======================

Normally you would view the results of an analysis using the display package on-line.  However, the following indicates the information produced by the analysis.

     # Test analysis of box.el
     #   (lines which were never evaluated during tests, indicated by
     #    the string ``NEVER->>'',  or which returned the same value
     #    every time they were evaluated, including the value.)

     box.el:31== nil
     box.el:33== nil
     box.el:34== nil
     box.el:35== nil
     box.el:37== 1
     box.el:38== 0
     box.el:39== nil
     box.el:41== nil
     box.el:42== nil
     box.el:49== 1
     box.el:50== nil

     ...

     box.el:94== nil
     box.el:95== nil
     box.el:109== #<marker at 1 in box.junk>
     box.el:115== nil
     box.el:116== t
     box.el:121== NEVER->>
     box.el:122== NEVER->>
     box.el:128== nil
     box.el:129== nil
     box.el:134== NEVER->>
     box.el:135== NEVER->>
     box.el:140== nil

     ...

Displaying Test Results
***********************

The coverage component of the Test package is useful for measuring how thoroughly a test script exercises an emacs-lisp package.  The interactive Test display mode lets you browse the annotated code.  The batch display function creates a summary report which may be browsed later using the Emacs ``compilation'' mode.

Display Mode---creating annotation windows
==========================================

Evaluating an instrumented buffer of emacs-lisp code creates a database of annotations for each instrumented line.  The two data stored during evaluation are: count, the number of times each line has been evaluated, and values, a list of the resultant values.  Analyzing the database adds two new annotations to various lines in the database: zero, the line was never evaluated, and constant, the value returned was the same every time it was evaluated.  Note that a line evaluated exactly once will be flagged as having a constant value.

Display mode puts up these annotations beside the lisp code in one or more annotation windows which are linked to the lisp window so that all windows scroll together.  The functions available for creating annotation windows are as follows:

C-c c
     Open an annotation window which highlights code which returned the same value during testing (tst-display-constant).

C-c z
     Open an annotation window which highlights code which was never evaluated during testing (tst-display-zero).

M-x tst-display-open-buffer
     Create a buffer containing the database values for a particular attribute over all lines of the lisp buffer.  Valid attributes are count, the count of how many times that line was evaluated, and values, a list with each element the result of one evaluation of that line.

M-x tst-display-open-window
     Open an annotation window onto a buffer created by a prior evaluation of tst-display-open-buffer.

C-c C-h
     Open a help window showing the key bindings for display mode (tst-display-mode-help).

C-c q
     Exit display mode (tst-display-mode-exit).

Typically the first thing you will do after entering display mode is to type C-c c C-c z to create two annotation windows which point out any lines in the lisp buffer which either were never evaluated or always returned the same value.  In the first case you probably need to add to your test script to either call an unreferenced function or exercise the other side of a conditional expression.  The second case may require some investigation.  One possibility is that the line is something like:

     (setq very-important-variable nil).

This expression, of course, always returns the same value.  Another possibility is that the line was only evaluated once, in which case you might want to add to your test script to hit it again.  The third possibility is that you overlooked a parameter when composing your test script.

If you want to look at the other annotations in the database, you may do so with the commands M-x tst-display-open-buffer and M-x tst-display-open-window.  You must use them in that order, and each one will prompt for an ``attribute name'' which may be either of count or values.

Display Mode---moving within annotation windows
===============================================

The annotation windows are kept in step with the lisp window through several functions which take the place of the usual cursor movement commands while in display mode.  These are described below:

C-n
     Move down one line vertically in the lisp window and all associated annotation windows.  On reaching the bottom of the window, scroll windows together (tst-display-next-line).

C-p
     Move up one line vertically in the lisp window and all associated annotation windows.  On reaching the top of the window, scroll windows together (tst-display-previous-line).

C-v
     Scroll forward in the lisp window and any associated annotation windows keeping them aligned (tst-display-scroll-up).

M-v
     Scroll backward in the lisp window and any associated annotation windows keeping them aligned (tst-display-scroll-down).

C-c l
     Clear screen and redisplay, scrolling the lisp window and any associated annotation windows together to center the line containing point (tst-display-redraw).

The usual cursor up and down keys are C-n and C-p, respectively.  In display mode these move point in all annotation windows simultaneously so that when the limits of the screen are reached all windows will scroll together.  Similarly the scroll-up and scroll-down keys C-v and M-v are rebound in display mode to functions which keep track of what annotation windows are open and scroll them in step with the lisp buffer.  It is still possible for the windows to get out of step (e.g. after using M-<), so the C-c l key will resynchronize all annotation windows while recentering the line containing point just as the C-l key does for a single window.

Batch Display of Analysis
=========================

Quite separate from the display mode is a display function that is designed to give a concise summary of test coverage without your interaction.  This function can be run in batch or interactively.  (It was originally intended for only batch operation, hence the name.)

M-x tst-display-batch
     Generate a compilation style buffer containing zero and constant analyses from the database.

You need not be in display mode to use tst-display-batch.  You are most likely to use it in a test script run from batch mode to dump out a summary of test coverage at the end of a run.  However it is also useful from interactive mode because it takes advantage of the Emacs function C-x` (next-error), often used to view compiler error messages.  The first invocation of C-x` parses the error messages in the buffer named *compilation* then places point in one window on the line of code referenced by the error message shown at the top of the other window.  Successive C-x` keystrokes advance to successive error messages and the corresponding lines in the code buffer.

The M-x tst-display-batch command writes into the *compilation* buffer a line for every line in the lisp buffer that has either a zero or constant annotation.  Subsequent uses of C-x` advance to the next error and the corresponding line in the lisp buffer.

The Example
===========

The following is the output of the evaluation of tst-display-batch in our example test script.  Notice that some deficiencies in the test script are pointed out: error conditions were not exercised (see lines 121-122 and 134-135 in Appendix A); and not enough different input texts were ``boxed'' to really stress the package under test (e.g. line 40, input text always same width).

     # Test analysis of box.el
     #   (lines which were never evaluated during tests or returned
     #    the same value every time they were evaluated.)
     box.el:31==  nil
     box.el:33==  nil
     box.el:34==  nil
     box.el:35==  nil
     box.el:37==  1
     box.el:38==  0
     box.el:39==  nil
     box.el:40==  41
     box.el:41==  nil
     box.el:42==  nil
     box.el:43==  42
     box.el:47==  44
     box.el:49==  1
     box.el:50==  nil
     box.el:51==  nil
     box.el:52==  nil
     box.el:53==  nil
     box.el:55==  nil
     box.el:56==  nil
     box.el:57==  nil
     box.el:58==  nil
     box.el:59==  nil
     box.el:62==  nil
     box.el:63==  nil
     box.el:64==  nil
     box.el:65==  nil
     box.el:67==  nil
     box.el:68==  nil
     box.el:69==  1
     box.el:70==  12
     box.el:71==  nil
     box.el:72==  nil
     box.el:73==  nil
     box.el:74==  12
     box.el:75==  nil
     box.el:80==  nil
     box.el:91==  nil
     box.el:93==  nil
     box.el:94==  nil
     box.el:95==  nil
     box.el:109==  #<marker at 6 in box.junk>
     box.el:115==  nil
     box.el:116==  t
     box.el:121==  NEVER->>
     box.el:122==  NEVER->>
     box.el:128==  nil
     box.el:129==  nil
     box.el:134==  NEVER->>
     box.el:135==  NEVER->>
     box.el:140==  nil
     box.el:141==  nil
     box.el:143==  nil
     box.el:144==  t
     box.el:145==  NEVER->>
     box.el:147==  t
     box.el:149==  nil
     box.el:150==  nil
     box.el:151==  nil
     box.el:153==  nil
     box.el:154==  nil
     box.el:157==  nil
     box.el:159==  nil
     box.el:160==  #<marker at 6 in box.junk>

Running Test in Batch Mode
**************************

Preparation
===========

Before running a set of tests, you must prepare the following items:

  * A package to test.

  * A test script, usually written as a single function.

  * A set of precondition states---one for each test (although many tests may share the same precondition).

  * A set of expected postcondition states---one for each test (some sharing is possible).

Test scripts are discussed in the next section.

You will probably want to use the state-capturing functions (e.g., ``tst-capture-state'', ``tst-write-state-to-file'') to prepare and save states in files.  In doing this, strive for minimality of states.  That is, avoid complicating the states with extra components, particularly buffers and windows, that do not contribute to the functionality being tested. This will make it easier to read and comprehend the testing states, and will result in more efficient test scripts.  (The instrumentation and analysis components of Test are designed to assist you in discovering the inadequacies of your test scripts.  It is best to start with too little testing, and add additional tests as the analysis dictates.)

Below is an example scenario for preparing the states used in a testing script discussed later.  Note that the functions would be invoked interactively, even though they are shown in ``program invocation'' style.

     (load "test.el")
     (load "box.el")
     (find-file "sample")
     ... go to an appropriate location ...
     (set-mark)
     ... go to an appropriate location ...
     (tst-capture-state-to-file "pre-box")
     (box-region)
     (tst-capture-state-to-file "exp-box-1")
     (unbox)
     (tst-capture-state-to-file "exp-unbox-1")
     (box-region t)
     (tst-capture-state-to-file "exp-box-2")
     (unbox t)
     (tst-capture-state-to-file "exp-unbox-2")

Note that in this scenario the expected result of applying a function is created by applying the function to the precondition state.  It is hard to imagine how the test could fail, but it might for unexpected reasons.  A better method of creating the test states is to use an independent method, such as repeated application of simpler functions (e.g., ``insert'', ``next-line'').  Still, ``safe'' tests, such as that shown above, are useful for regression testing.

Batch Test Scripts
==================

A test run usually consists of the following steps:

 1. Read testing states from files.

 2. Initialize the testing environment:

      1. Find the object package (the code to be tested).

      2. Instrument the package.

 3. Run each test:

      1. Achieve the precondition state of the test.

      2. Execute the function to be tested.

      3. Capture the postcondition state.

      4. Compare the postcondition state to the expected postcondition state.

 4. Save the results of testing in a file.

 5. Analyze the testing (filter the collected data) and save the results.

Note that a set of tests may reuse testing states, especially the preconditions of tests.  That is why it is wise to read all of the needed testing states from files first.  Also, it is good style to minimize state changes during testing (except for the execution of tested functions, of course), so that the test script may be modified without unforeseen side effects.

Below is an example test script for testing functions ``box-region'' and ``unbox'' in package ``box.el''.  This script does not contain enough tests, which the instrumentation should expose.

     (load "test.el")

     (defun script ()
       (let (post-state pre-box exp-box-1 exp-box-2
                        pre-unbox exp-unbox-1 exp-unbox-2)
                                             ; Load states from files
         (tst-read-state-from-file 'pre-box "pre-box")
         (tst-read-state-from-file 'exp-box-1 "exp-box-1")
         (tst-read-state-from-file 'exp-box-2 "exp-box-2")
         (tst-read-state-from-file 'pre-unbox "pre-unbox")
         (tst-read-state-from-file 'exp-unbox-1 "exp-unbox-1")
         (tst-read-state-from-file 'exp-unbox-2 "exp-unbox-2")
                                             ; Instrument package
         (find-file "box.el")
         (tst-instrument)
                                             ; Initialize test environment
         (find-file "sample")
                                             ; Tests
         ; TEST box-1
         (tst-achieve-state pre-box)
         (box-region nil)
         (tst-capture-state 'post-state)
         (tst-equ-state post-state exp-box-1)
         ; TEST box-2
         (tst-achieve-state pre-box)
         (box-region t)
         (tst-capture-state 'post-state)
         (tst-equ-state post-state exp-box-2)
         ; TEST unbox-1
         (tst-achieve-state pre-unbox)
         (unbox nil)
         (tst-capture-state 'post-state)
         (tst-equ-state post-state exp-unbox-1)
         ; TEST unbox-2
         (tst-achieve-state pre-unbox)
         (unbox t)
         (tst-capture-state 'post-state)
         (tst-equ-state post-state exp-unbox-2)
                                             ; Results of tests
         (set-buffer "*equal-log*")
         (save-file "equal-log-sample")
                                             ; Analysis of testing
         (tst-analyze)
         (tst-display-batch)
         ) ; let
     ) ; defun script

Batch Invocation of Test
========================

To invoke the test script from batch, use the following Unix command:

     % gnuemacs -batch -l script.el -f script -kill &

Note that the test script explicitly loads ``test.el'' at the beginning of ``script.el''.

An instrumented package runs considerably slower than an uninstrumented package.  It is a good idea to test your script without instrumentation (and probably without many of the tests).  Once you are satisfied with the script, run it at lower priority and/or at non-peak hours to avoid inconveniencing other users.

Advanced Batch Use -- Filtering
===============================

The state information available for ``capture'ing and ``achieve''ing is usually more than needed for any particular test.  You can improve the performance of testing by filtering out the unneeded components of states. There are mechanisms for doing this before and after testing:

  * The ``capture'' functions use global options to determine the components to capture.

  * The ``achieve'' functions are similarly parameterized.

  * The ``equality'' functions use global options to determine which components to compare.  Also, they can be weakened (i.e. return ``true'' more often) through the use of hooks and replacement functions.

  * The ``analysis'' functions reduce the instrumentation data to two special cases: lack of evaluation and ``constant value across testing''.

  * The ``*equal-log*'' buffer may be viewed in ``outline'' mode, with uninteresting cases suppressed through elision.

  * The batch display function produces output that may be viewed with the ``next-error'' function of ``compilation''.

In using these filters you should be aware that too much filtering defeats the purpose of testing.

The Epilogue
************

How to Extend the Package
=========================

Equality Tests
--------------

If you want to extend the equality package to compare other aspects of your environment, simply write your own comparison function and add it to the 'function-vector' for the appropriate area.  For example, if you were interested in comparing the foobar attribute of buffers, first write a function, tst-equ-foobar, and then add it to the list of functions executed for a buffer, tst-equ-buff-state-functions.

Similarly, if you wanted to test the foo attribute of a state, add the new function to the list of functions executed for a state, tst-equ-state-functions.

You can also extend the equality tests by the use of hooks.  The method is defined in more detail in the section on Inequality.

Instrumentation
---------------

Display Functions
-----------------

The interactive display mode attempts to provide the fiction of annotation windows which are part of the code window.  In retrospect it may be that simply inserting the text into a new buffer along with the code would be as useful.  It also may be that the batch style report in conjunction with the ``compilation'' error parsing is just as useful in practice.

There are certainly other types of annotation which could usefully be displayed, like the number of times each line was executed, etc.

It might make sense to bind tst-display-batch to a key in test-mode.  It also might make sense to remove the binding for tst-analyze and call it only from within the display functions.

Annotation Functions
--------------------

The annotation database is accessed twice every time an instrumented line of code is evaluated.  This code contributed substantially to the slow-down of instrumented code and time spent making it more efficient would be rewarding.

Analysis Functions
------------------

If you want to add further functions to the Analysis Package, you will have to modify the main function, ``tst-analyze'', which invokes the individual analysis functions.  It currently invokes ``tst-anl-zero-counts'' and ``tst-anl-constant-values''.  Your new function should use the same idiom as these functions, i.e., use ``tst-ann-get-lines'' to retrieve the list of lines for which annotation holds information and then use ``mapcar'' to apply your single line analysis function to each line's data.  You can then use ``tst-ann-put'' to store your results.

The Complete Example
********************

     [file example not included]