| [ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
This document explains how to use the Test package to test GNU Emacs packages. It is an extension of Chapter 22 [Compiling and Testing Programs] of the GNU Emacs Reference Manual.
| [ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
This package provides tools to aid in developing regression tests for Emacs packages and to assess the efficacy of those tests. GNU Emacs packages, written in a dialect of Lisp, provide extensions to the basic text editing features of Emacs. Most of these packages perform state-changing operations. For example, a package may create new buffers, modify text, change the locations of the cursor and various marks, write files or modify anything else accessible to Emacs (which is just about everything in Unix). Traditional Lisp debugging strategies are not sufficient to adequately test a package that performs extensive state-manipulation. The Test package provides a strategy that is.
| [ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
The Test system is designed to support structural (or “white-box”) testing. (Another commonly used test approach is functional, or “black-box” testing.) It automates the process of running a number of tests cases designed to analyze “coverage” of the package (that is, find parts of the code that are not exercised) and “expression” (that is, what values are returned by each expression, and which expressions always returned the same values).
| [ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
Test provides mechanisms to ease the creation and execution of sets of tests. Some of the features are designed to simplify testing activity, while others are designed to aid the analysis of testing.
Test supports coverage testing. That is, it provides instrumentation mechanisms designed to check for inadequacies of testing of the following two forms:
In each case, the fault may be in the tests (i.e., not enough testing was performed), or the fault may be in the program (i.e., an “unnecessarily complex” program has been written). (There is also the possibility that no fault exists, such as the use of “constant” functions for readability or style. In practice these cases are easy to identify and ignore.)
You should use the following strategy in creating tests with this system:
There is no guarantee that a set of tests that passes these coverage checks will find all errors. But, such a set of tests has the property that every expression in the code has been exercised in a minimally significant way (i.e., every expression has produced at least two different values). In practice, a great number of errors can be detected with this method.
The importance of coverage testing increases as a package is modified. Changes to one part of a package may have unexpected effects on other (unconsidered) parts. Coverage testing ensures that all parts of a package are tested, at least minimally. Coverage tests also serve as a specification of the functionality of a package.
Figure 1 illustrates the overall flow of events using this test approach.
+------------+ +--------------+
| Instrument | | Generate |
| o Control | | o Test Cases |
| o Data | | o Script(s) |
+------------+ +--------------+
\ / ^
\ / |
\ / |
\ / |
\ / |
\ / |
+-------------+ |
| Execute | |
| the | |
| Test(s) | |
+-------------+ |
| |
| |
+-------------+ |
| Analyze, | |
| Display | |
| Results | |
+-------------+ |
| |
| |
^ |
/ \ |
/ \ No |
< Done? >-------------+
\ /
\ /
\ /
v
| Yes
|(Test Goals Achieved)
V
Figure 1. General Test Algorithm
| [ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
The Test product consists of four major subsystems:
The following sections explain the use of these functions to support the steps in the testing process:
The system is designed to test Emacs-Lisp programs, and is implemented as a minor mode of Emacs-Lisp mode (See Section 28.1 [Minor Modes] of the GNU Emacs Reference Manual).
| [ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
This section describes how to begin using the Test package to test Emacs packages.
| [ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
To start testing you probably want a fresh Emacs session with no extraneous buffers or processes hanging around. Non-essential extras will just slow down the state capture and achieve functions. If you have a busy ‘.emacs’ file you may want to keep a trimmed back copy just for test sessions.
Once in Emacs visit the file containing Emacs-lisp code to be tested, then enter:
M-x load-file test.el M-x test-mode
The mode line at the bottom of your window should show “(Emacs-lisp Test)” in the mode field. You are now in Test mode.
Probably the first thing to do at this point is to use ‘C-c n’ to generate a new buffer containing a test script template. This is a good starting point for creating a new test script; the template is shown in section 2.3 of this document. The task now confronting you is to develop a test suite that will exercise your package throughly, not only to test it now but also to have an automated regression test. The remainder of this document will guide you through this task.
Figure 2 illustrates the overall flow of data in the Test system.
+-------------+
|Source Text |
+-------------+
\
\
\
Instrument
/
/
/
+-------------+ +-------------+
|Instrumented | | Test Cases |
| Text | +-------------+
+-------------+ /
\ /
\ /
\ /
\ /
Execute
\
\
\
\
+-------------+
|Test Results |
+-------------+
/ ^
/ |
/ |
/ v
Display Analyze
Figure 2. Test Data Flow
| [ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
The package ‘‘test.el’’ provides several functions to the user:
Start a new test script
(tst-new-script-buffer)
(Re-)instrument a test program buffer (tst-instrument).
Test the program, using the test script (tst-execute).
Record a GNU Emacs state for later use
(tst-capture-state).
Achieve a previously saved GNU Emacs state
(tst-achieve-state).
Achieve a state previously saved-to-file
(tst-achieve-state-from-file).
Capture current emacs state to file
(tst-capture-state-to-file).
Read a previously saved-to-file state into a variable
(tst-read-state-from-file).
Write a state previously captured in a variable to a file
(tst-write-state-to-file).
Analyze (filter) the results of running the test(s) (tst-analyze).
Display the annotations about the test (tst-display-mode).
Exit test mode (test-mode).
| [ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
(defun test-script ()
(interactive)
; Local Variables
(let (pre post actual)
; Body
(switch-to-buffer foo)
(erase-buffer)
(insert-file foo)
(goto-char (point-min))
(emacs-lisp-mode)
(test-mode)
(tst-instrument)
; 1st test run
(tst-read-state-from-file 'post expected-state-file1)
(tst-achieve-state-from-file 'pre initial-state-file1)
(foo args)
(tst-capture-state 'actual nil nil)
;;; If you have a lot of tests, consider a while loop
) ; let
) ; defun
| [ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
The package ‘‘box.el’’ provides two functions to the user:
box-region and unbox.
The first function encloses the current region with a box:
/********************************************\ * A boxed region * * may have lines of text of varying lengths. * * A side effect of boxing is to normalize * * the length of lines. * \********************************************/
The second function undoes the effect of the first by removing any boxes within the region. This simple package will be used in this document as an example of a package to be tested using the Test package.
| [ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
As mentioned at the beginning of this document, Emacs packages are generally state changing operations. You invoke the package with a session that is in some arbitrary initial state. The package modifies selected aspects of this initial state in a defined manner and leaves the session in some final state. The state of a GNU Emacs session consists of information on buffers, windows, processes, and other global information. The components of the state that are used in the Test package are listed later in this section.
Regression Testing is a technique for verifying that changes in a package do not unintentionally degrade the quality of the code. Any changes in one part of the package should not alter the functionality of other parts of the package. State information plays two roles in regression testing of Emacs packages. These are:
The regression testing components of ‘test.el’ allow you to capture session states for later use, achieve these states to set up a test run, and test the equality of states for package verification.
The following example demonstrates these steps for regressions testing an arbitrary package. Assume that a working version of some package exists. Now suppose you wish to add some new function that requires some modifications in the original code. After adding the new function you need to verify that the original functionality of the package has not changed. In other words, you need to verify that the code has not regressed from its original functionality. To perform this testing, you capture the initial state of a session, run the original package, and capture the final state. Following this, you achieve the initial state that you previously captured, run the new version of the package, and perform an equality test to compare the new final state to the one that was captured after the run of the original package. The states should be equivalent if the test is successful.
Two extensions to this state comparison paradigm are:
The remainder of this section describes the elements of the Test package that implement these aspects of regression testing. In summary, these are:
capture - saves the current state of the Emacs session
in either a bound variable or UNIX file.
partial capture - save part of the current state of the
Emacs session. Some or all global variables may be excluded, and a subset
of buffers may be selected.
achieve - sets the state of the current session to one
that is either in a bound variable or a UNIX file.
equality - compares two emacs sessions states for
equivalence.
partial equality - compares selected aspects of two emacs
session states for equivalence.
inequality - using user defined hooks, compares two
session states within certain degrees of inequality (e.g. compares buffer
contents ignoring whitespace).
| [ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
| [ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
The capture and achieve functions of ‘test.el’ represent an Emacs state as the hierarchy of data shown below.
Global Bound Symbols - Most of the information about an
Emacs session is contained in the values of symbols global to the session.
In conventional programming language terms, this session information can be
viewed as the values of global variables. Some examples of common global
bound symbols are:
Buffer Information - Each Emacs session contains one or
more buffers. The capture functions store the following information for
each active buffer:
Window Information - At any time an Emacs session has
one or more windows visible. The capture functions store the following
information for each window:
Process Information - An Emacs session may have invoked
several UNIX processes during its lifetime. These processes may have
completed or may still be running. Two common processes that run during an
Emacs sessions are the ‘shell’ and the process that displays the time,
date and system load on the mode line. The ‘Test’ capture
functions store the follow information for each process:
A significant item that is excluded from this state information is the
current definition of bound functions. Recall that an Emacs user can
define any new function with a ‘defun’ and load that function.
Furthermore, the user can redefine any existing function (e.g.
forward-char !) in this manner. Saving the current definition of
all bound functions would result in a huge state vector, and is therefore
not done.
| [ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
When regression testing an Emacs package, you will mainly use the state capture functions of Test in two ways:
| [ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
Writes the current state of the Emacs session to a file. You are prompted
for the name of the file, the exclude-variables and the
buffer-list. Refer to the Usage Notes below for details on
these last two items. If the file you specify already exists it is
overwritten.
Writes a state that is saved in a bound variable to a file. You are prompted for the state variable name (which was created with ‘tst-capture-state’) and the name of the file. If the file already exists it is overwritten.
Saves the state of the Emacs session in a state variable. You are prompted
for the name of the file, the exclude-variables and the
buffer-list. Refer to the Usage Notes below for more details
on these last two items. If the bound variable you specify already exists,
it is overwritten.
| [ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
The amount of information saved by ‘tst-capture-state-to-file’ and ‘tst-capture-state’ is controlled by your responses to the command prompts. These are:
There are three possible responses to this prompt:
("bar" "foo")
("xxx")
to specify that capture not save any buffer state information.
There are three possible responses to this prompt:
("obarray" "values")
the two global variables, ‘obarray’ and ‘values’, are not captured. This value is the default for the prompt since these two variables have very large values that are rarely of use in testing.
| [ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
When using Test you will mainly use the achieve state functions to return your session to a known initial state. With these functions you can make repeated test runs by simply achieving the initial state between each run.
| [ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
Reads a session state from a file into a bound symbol. You are prompted for the name of the file and the name of the bound symbol. The file should have been created by ‘capture-state-to-file’ or ‘write-state-to-file’. If the bound symbol already exists, it will be overwritten.
Sets the state of the emacs session to that saved in a file. You are prompted for the name of the file. The file should have been created by ‘capture-state-to-file’ or ‘write-state-to-file’.
Sets the state of the emacs session to that saved in a bound symbol. You are prompted for the name of the bound symbol. The symbol must have been created by ‘tst-capture-state’.
| [ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
Achieving the state of a previous Emacs session is a non-trivial task that can have many unexpected side-effects. These are listed below:
tst-capture-state-to-file.
tst-achieve-state-from-file.
You can avoid this problem by testing packages only on dummy test files, or
using a special test directory with copies of files that are permanently
stored in other directories, or by not capturing buffer ‘mytest’ and
setting tst-achieve-buffers-nondestructively to t.
| [ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
The equality package of Test compares two Emacs states for equivalence by invoking special equality functions on each of the data components captured in the state. Each of these equality functions compares only a small part of the entire state. The results of these comparisons are written into the buffer ‘*equal-log*’. If two states are not equal then you can determine where components differ by looking at this buffer.
The section Equality Functions provides a list of all of the equivalence testing functions. The section Partial Equality explains how you can accomplish testing a subset of a state. This is useful if you are only interested in differences between certain aspects of state, such as only the contents of buffers, but you do not care to hear about differences in other components, such as windows. The section Inequality explains how to add hooks to the equality functions. These hooks are called by the equality functions when two components are not equal but before a result is returned. A hook that you write can be inserted at this point and change a “not equal” result into an “equal.”
| [ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
The functions to compare states for equality are divided into two categories. One set of functions compares complete states, the second set of functions only uses the buffer components.
The following state equality functions are defined for interactive use.
tst-equ-stateCompares two states for equality. Each component in the first state is compared to its corresponding component in the second state.
tst-equ-sessionsCompares the sessions component of two states. The sessions components contains all global variables. These includes those that define key-maps and syntax tables.
tst-equ-buffersCompares the buffers components of two states. Compares all of the buffers in one state with those of a second state.
tst-equ-windowsCompares the windows components of two states.
tst-equ-processCompares the process components of two states.
The following functions operate on just the buffer components of
states. Objects of this type can be extracted from state objects by the
function tst-equ-find-buff-with-name.
tst-equ-buffer-stateCompares two buffers for equality. The following components are checked for equality: point, mark, contents, modified, file, local-variables.
tst-equ-pointCompares the point component of two buffers.
tst-equ-markCompares the mark component of two buffers.
tst-equ-fileCompares the file component of two buffers.
tst-equ-modifiedCompares the modified component of two buffers.
tst-equ-contentsCompares the contents component of two buffers. Contents are compared as two single strings.
tst-equ-contents-lineCompares the contents component of two buffers. Contents are viewed as composed of lines of text, and compared line-by-line.
tst-equ-contents-regionCompares the contents component of two buffers. Only the contents within a region are compared. Region is delimited by point and mark.
| [ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
If you are interested in only checking the equality of a subset of an Emacs state, then there are four methods you can use.
tst-capture-stateYou can use this function to capture only a subset of state. Once this has been done, equality testing will only be performed on those parts of the state that where captured.
tst-equ-state-functionsThis variable contains a list of equality functions to be invoked when comparing two states. You can change it to test only those components of state that interest you. As an example, you might want to remove ‘tst-equ-windows’ if you are not interested in differences in windows between states.
tst-equ-buff-state-functionsThis variable contains a list of equality functions to be invoked when comparing two buffer states. You can change it to test only those components of buffer state that interest you. As an example, you might want to remove ‘tst-equ-point’ and ‘tst-equ-mark’ if you are not interested in those parts of the states.
tst-equ-find-buffer-with-nameUse this function to obtain only the part of state that is associated with a particular buffer. You can then pass the result of this function to buffer testing equality functions.
| [ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
Sometimes the comparison of two objects yields “not equal” when wish to ignore certain inequalities. Each of the equality functions will execute a hook, if one is defined, and the comparison of the two objects yields “not equal.” Within the hook you can write your own test for equality and change the result of the comparison if you so desire.
For each equality function, there exists a hook symbol that, when defined,
will be run if the two objects are not equal. The name of the hook symbol
can be found by adding ‘-hook’ to the equality function name. The
value of this hook should then be set to the name of the hook function you
wish to execute. From within the the hook the two objects being compared
can be accessed by adding ‘1’ and ‘2’ to the equality function
name. The variable tst-equ-result should be set to ‘t’ or
‘nil’ from within your hook.
The following example shows a hook, called ignore-zero-points, that
is executed whenever the result of comparing the point components of two
buffers is not equal. From within this function the two point components
are accessed as ‘tst-equ-point1’ and ‘tst-equ-point2’. The hook
changes the result of a point comparison from ‘nil’ to ‘t’ if one
of the points is at position zero.
(setq tst-equ-point-hook 'ignore-zero-points)
(defun ignore-zero-points ()
"Equality point hook, changes result to t if one point
is at position zero."
(if (or (equal tst-equ-point1 0) (equal tst-equ-point2 0))
(setq tst-equ-result t) ; return t if one is zero
;else
(setq tst-equ-result nil); otherwise return nil
)
)
Of particular usefulness are the hooks that are associated with the contents of two buffers. The equality function ‘tst-equ-bs-contents’ compares the contents of two buffers by comparing strings that contain the entire contents of each buffer. Should this comparison fail it is sometimes useful to compare the two strings after eliminating all white-space from each string. The following hook, provided with this package, does such a comparison.
(setq tst-equ-contents-hook 'ignore-white-space)
defun ignore-white-space()
" Compares the contents of two buffers after removing all
white-space from each."
(setq tst-equ-result (string-equal-less-regexp "\\s "
tst-equ-contents1 tst-equ-contents2))
)
A second contents hook of interest is the one associated with the ‘tst-equ-contents-line function’. This function compares the contents of two functions on a line-by-line basis. The hook for this function differs from all other hooks, in that it is called once per line, instead of once per call to its associated function. The two lines are accessed as ‘tst-equ-line1’ and ‘tst-equ-line2’.
| [ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
The example that follows illustrates how to write an emacs-lisp function that uses the Test package to test another function you have written. In the example the two user supplied functions under test are: ‘box-region’ and ‘unbox’.
;;; test-box uses the Test package to test the functions box and unbox
;;;
test-box ()
(interactive)
; Local Variables
(let (initial-state ; to hold states
boxed-state
final-state
capture-buffer-list
)
; First create a buffer to use the functions on
(get-buffer-create "box.junk")
(set-buffer "box.junk")
(erase-buffer)
(insert-file "/project/gnutest/test/box.junk")
; Save the initial state
; Since we're only concerned with one buffer include
; include that in the capture list
(setq capture-buffer-list '("box.junk"))
(tst-capture-state 'initial-state capture-buffer-list nil)
; execute the box function
(goto-char (point-min))
(set-mark (point-max))
(box-region nil)
; capture the boxed state
(tst-capture-state 'boxed-state capture-buffer-list nil)
; unbox and capture again
(unbox nil)
(tst-capture-state 'final-state capture-buffer-list nil)
; compare the initial state to the final state
; they should be exactly equal
(if (tst-equ-state initial-state final-state
"Compare before box to after unbox")
(message "Box/unbox test passed")
(message "Box/unbox test failed"))
) ; let
) ; defun
| [ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
Each of the equality functions writes the results of its comparison into a buffer named “*equal-log*”. If this buffer does not exist, then it is created. The contents of the buffer are never erased nor is the buffer ever deleted, these actions are left to the user.
The buffer is created in Outline Mode. This allows selective hiding of comparisons that are not of interest to the user.
If the result of a comparison is nil (i.e. not equal) then that entry in the log is flagged with a question mark (?). In most cases, the two differing objects are also logged as in the following example.
* ?State comparison: Compare before box to after unbox ** ?Sessions state *** ? Global symbols data-bytes-free 5350948 5176868 this-command nil kill-region data-bytes-used 383452 557532 ?post not found in second state post nil nil ?file not found in second state file statevar state post ** ?Buffers state ** Comparison of buffers named: *scratch* *** point: "1" "1" *** mark: nil nil *** contents: contents equal *** modified: nil nil *** file: nil nil *** local-vars: local variables are equal ** Comparison of buffers named: *Minibuf-0* *** point: "1" "1" *** mark: nil nil *** contents: contents equal *** modified: nil nil *** file: nil nil *** local-vars: local variables are equal ** Comparison of buffers named: box.el *** point: "1" "1" *** mark: 4480 4480 *** contents: contents equal *** modified: t t *** file: nil nil *** local-vars: local variables are equal ** Comparison of buffers named: box.el-instrumented *** point: "5608" "5608" *** mark: 5608 5608 *** contents: contents equal *** modified: t t *** file: nil nil *** local-vars: local variables are equal ** ?Comparison of buffers named: box.junk *** ?point: "1" "234" *** ?mark: 235 1 *** ?contents: contents not equal *** modified: t t *** file: nil nil *** local-vars: local variables are equal ** Processes state ** Window state *** window **** window-edges: (0 0 10 9) (0 0 10 9) **** window-buffer: "box.junk" "box.junk" **** window-start: 1 1 **** window-point: 1 1 **** current-window: t t
| [ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
Use the tst-instrument command for instrumenting Lisp code.
Instrumenting copies the contents of the current buffer to a new buffer
named for the current buffer concatenated with ‘-instrumented’. The
new buffer is set to emacs-lisp-mode. Instrumenting initializes the
annotation data base, adds the instrumentation to the copied code, and
finally evaluates the entire copied buffer.
| [ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
Instrumentation acts on certain lists that represent functions within a
defun by adding an instrumentation probe. Candidate lists are
identified and Lisp code is added around the list. A list representing a
function that has the form, ‘(function arg1 arg2)’, after insertion of
the instrumentation probe will have the form, ‘(tst-cover #id
(function arg1 arg2))’. Instrumentation assigns the line number of the
function within the buffer as the #id.
| [ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
The instrumentation probe consists of the invocation of the function
tst-cover with two arguments, a unique identifier and the function
that was instrumented. Tst-cover uses the identifier as a key into
the annotation data base, stores the result of the instrumented function,
increments an invocation counter, and returns the result of the
instrumented function as its function value.
| [ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
Original Code:
(defun factorial (n)
(let
(result)
(if
(< n 2)
(setq result 1)
(setq result
(* n
(factorial
(1- n)))))
result))
Instrumented Code:
(defun factorial (n)
(tst-cover 2 (let
(result)
(tst-cover 4 (if
(tst-cover 5 (< n 2))
(tst-cover 6 (setq result 1))
(tst-cover 7 (setq result
(tst-cover 8 (* n
(tst-cover 9 (factorial
(tst-cover 10 (1- n))))))))))
result)))
| [ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
This section explains how to use the analyze function
| [ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
The Coverage Analysis package is designed to run after a series of tests has been performed using the instrumented version of the code under test. It retrieves data from the annotation database and detects the following testing anomalies:
When these conditions are detected additional attributes are stored in the annotation database in a format suitable for extraction by the display component of Test.
Analysis is invoked by the command:
tst-analyze
If an expression is not executed, the attribute “zero” is inserted with a constant value. This constant value is defined by the following variable:
tst-anl-zero-counts
If an expression returns a constant value, the attribute “constant” is inserted with the value of the constant. If an expression is only executed once, it is deemed to return a constant result.
You can re-run the analysis function without re-instrumenting the code under test. If, for example, an initial set of tests indicates a number of unexecuted expressions, you can run additional tests and analyze the combined results without re-running the complete set of tests. Note however, that if you re-instrument the code, the annotation database is reinitialized and the results of previous tests, including any analysis results will be lost (unless you explicitly save the annotation database using tst-ann-get-db).
| [ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
Normally you would view the results of an analysis using the display package on-line. However, the following indicates the information produced by the analysis.
# Test analysis of box.el # (lines which were never evaluated during tests, indicated by # the string ``NEVER->>'', or which returned the same value # every time they were evaluated, including the value.) box.el:31== nil box.el:33== nil box.el:34== nil box.el:35== nil box.el:37== 1 box.el:38== 0 box.el:39== nil box.el:41== nil box.el:42== nil box.el:49== 1 box.el:50== nil ... box.el:94== nil box.el:95== nil box.el:109== #<marker at 1 in box.junk> box.el:115== nil box.el:116== t box.el:121== NEVER->> box.el:122== NEVER->> box.el:128== nil box.el:129== nil box.el:134== NEVER->> box.el:135== NEVER->> box.el:140== nil ...
| [ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
The coverage component of the Test package is useful for measuring how thoroughly a test script exercises an emacs-lisp package. The interactive Test display mode lets you browse the annotated code. The batch display function creates a summary report which may be browsed later using the Emacs “compilation” mode.
| [ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
Evaluating an instrumented buffer of emacs-lisp code creates a database of annotations for each instrumented line. The two data stored during evaluation are: ‘count’, the number of times each line has been evaluated, and ‘values’, a list of the resultant values. Analyzing the database adds two new annotations to various lines in the database: ‘zero’, the line was never evaluated, and ‘constant’, the value returned was the same every time it was evaluated. Note that a line evaluated exactly once will be flagged as having a constant value.
Display mode puts up these annotations beside the lisp code in one or more annotation windows which are linked to the lisp window so that all windows scroll together. The functions available for creating annotation windows are as follows:
Open an annotation window which highlights code which returned the same value during testing (‘tst-display-constant’).
Open an annotation window which highlights code which was never evaluated during testing (‘tst-display-zero’).
Create a buffer containing the database values for a particular attribute over all lines of the lisp buffer. Valid attributes are ‘count’, the count of how many times that line was evaluated, and ‘values’, a list with each element the result of one evaluation of that line.
Open an annotation window onto a buffer created by a prior evaluation of ‘tst-display-open-buffer’.
Open a help window showing the key bindings for display mode (‘tst-display-mode-help’).
Exit display mode (‘tst-display-mode-exit’).
Typically the first thing you will do after entering display mode is to type ‘C-c c C-c z’ to create two annotation windows which point out any lines in the lisp buffer which either were never evaluated or always returned the same value. In the first case you probably need to add to your test script to either call an unreferenced function or exercise the other side of a conditional expression. The second case may require some investigation. One possibility is that the line is something like:
(setq very-important-variable nil).
This expression, of course, always returns the same value. Another possibility is that the line was only evaluated once, in which case you might want to add to your test script to hit it again. The third possibility is that you overlooked a parameter when composing your test script.
If you want to look at the other annotations in the database, you may do so with the commands ‘M-x tst-display-open-buffer’ and ‘M-x tst-display-open-window’. You must use them in that order, and each one will prompt for an “attribute name” which may be either of ‘count’ or ‘values’.
| [ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
The annotation windows are kept in step with the lisp window through several functions which take the place of the usual cursor movement commands while in display mode. These are described below:
Move down one line vertically in the lisp window and all associated annotation windows. On reaching the bottom of the window, scroll windows together (‘tst-display-next-line’).
Move up one line vertically in the lisp window and all associated annotation windows. On reaching the top of the window, scroll windows together (‘tst-display-previous-line’).
Scroll forward in the lisp window and any associated annotation windows keeping them aligned (‘tst-display-scroll-up’).
Scroll backward in the lisp window and any associated annotation windows keeping them aligned (‘tst-display-scroll-down’).
Clear screen and redisplay, scrolling the lisp window and any associated annotation windows together to center the line containing point (‘tst-display-redraw’).
The usual cursor up and down keys are ‘C-n’ and ‘C-p’, respectively. In display mode these move point in all annotation windows simultaneously so that when the limits of the screen are reached all windows will scroll together. Similarly the scroll-up and scroll-down keys ‘C-v’ and ‘M-v’ are rebound in display mode to functions which keep track of what annotation windows are open and scroll them in step with the lisp buffer. It is still possible for the windows to get out of step (e.g. after using ‘M-<’), so the ‘C-c l’ key will resynchronize all annotation windows while recentering the line containing point just as the ‘C-l’ key does for a single window.
| [ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
Quite separate from the display mode is a display function that is designed to give a concise summary of test coverage without your interaction. This function can be run in batch or interactively. (It was originally intended for only batch operation, hence the name.)
Generate a compilation style buffer containing zero and constant analyses from the database.
You need not be in display mode to use ‘tst-display-batch’. You are most likely to use it in a test script run from batch mode to dump out a summary of test coverage at the end of a run. However it is also useful from interactive mode because it takes advantage of the Emacs function ‘C-x`’ (‘next-error’), often used to view compiler error messages. The first invocation of ‘C-x`’ parses the error messages in the buffer named ‘*compilation*’ then places point in one window on the line of code referenced by the error message shown at the top of the other window. Successive ‘C-x`’ keystrokes advance to successive error messages and the corresponding lines in the code buffer.
The ‘M-x tst-display-batch’ command writes into the ‘*compilation*’ buffer a line for every line in the lisp buffer that has either a ‘zero’ or ‘constant’ annotation. Subsequent uses of ‘C-x`’ advance to the next error and the corresponding line in the lisp buffer.
| [ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
The following is the output of the evaluation of ‘tst-display-batch’ in our example test script. Notice that some deficiencies in the test script are pointed out: error conditions were not exercised (see lines 121-122 and 134-135 in Appendix A); and not enough different input texts were “boxed” to really stress the package under test (e.g. line 40, input text always same width).
# Test analysis of box.el # (lines which were never evaluated during tests or returned # the same value every time they were evaluated.) box.el:31== nil box.el:33== nil box.el:34== nil box.el:35== nil box.el:37== 1 box.el:38== 0 box.el:39== nil box.el:40== 41 box.el:41== nil box.el:42== nil box.el:43== 42 box.el:47== 44 box.el:49== 1 box.el:50== nil box.el:51== nil box.el:52== nil box.el:53== nil box.el:55== nil box.el:56== nil box.el:57== nil box.el:58== nil box.el:59== nil box.el:62== nil box.el:63== nil box.el:64== nil box.el:65== nil box.el:67== nil box.el:68== nil box.el:69== 1 box.el:70== 12 box.el:71== nil box.el:72== nil box.el:73== nil box.el:74== 12 box.el:75== nil box.el:80== nil box.el:91== nil box.el:93== nil box.el:94== nil box.el:95== nil box.el:109== #<marker at 6 in box.junk> box.el:115== nil box.el:116== t box.el:121== NEVER->> box.el:122== NEVER->> box.el:128== nil box.el:129== nil box.el:134== NEVER->> box.el:135== NEVER->> box.el:140== nil box.el:141== nil box.el:143== nil box.el:144== t box.el:145== NEVER->> box.el:147== t box.el:149== nil box.el:150== nil box.el:151== nil box.el:153== nil box.el:154== nil box.el:157== nil box.el:159== nil box.el:160== #<marker at 6 in box.junk>
| [ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
| [ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
Before running a set of tests, you must prepare the following items:
Test scripts are discussed in the next section.
You will probably want to use the state-capturing functions (e.g., “tst-capture-state”, “tst-write-state-to-file”) to prepare and save states in files. In doing this, strive for minimality of states. That is, avoid complicating the states with extra components, particularly buffers and windows, that do not contribute to the functionality being tested. This will make it easier to read and comprehend the testing states, and will result in more efficient test scripts. (The instrumentation and analysis components of Test are designed to assist you in discovering the inadequacies of your test scripts. It is best to start with too little testing, and add additional tests as the analysis dictates.)
Below is an example scenario for preparing the states used in a testing script discussed later. Note that the functions would be invoked interactively, even though they are shown in “program invocation” style.
(load "test.el") (load "box.el") (find-file "sample") ... go to an appropriate location ... (set-mark) ... go to an appropriate location ... (tst-capture-state-to-file "pre-box") (box-region) (tst-capture-state-to-file "exp-box-1") (unbox) (tst-capture-state-to-file "exp-unbox-1") (box-region t) (tst-capture-state-to-file "exp-box-2") (unbox t) (tst-capture-state-to-file "exp-unbox-2")
Note that in this scenario the expected result of applying a function is created by applying the function to the precondition state. It is hard to imagine how the test could fail, but it might for unexpected reasons. A better method of creating the test states is to use an independent method, such as repeated application of simpler functions (e.g., “insert”, “next-line”). Still, “safe” tests, such as that shown above, are useful for regression testing.
| [ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
A test run usually consists of the following steps:
Note that a set of tests may reuse testing states, especially the preconditions of tests. That is why it is wise to read all of the needed testing states from files first. Also, it is good style to minimize state changes during testing (except for the execution of tested functions, of course), so that the test script may be modified without unforeseen side effects.
Below is an example test script for testing functions “box-region” and “unbox” in package “box.el”. This script does not contain enough tests, which the instrumentation should expose.
(load "test.el")
(defun script ()
(let (post-state pre-box exp-box-1 exp-box-2
pre-unbox exp-unbox-1 exp-unbox-2)
; Load states from files
(tst-read-state-from-file 'pre-box "pre-box")
(tst-read-state-from-file 'exp-box-1 "exp-box-1")
(tst-read-state-from-file 'exp-box-2 "exp-box-2")
(tst-read-state-from-file 'pre-unbox "pre-unbox")
(tst-read-state-from-file 'exp-unbox-1 "exp-unbox-1")
(tst-read-state-from-file 'exp-unbox-2 "exp-unbox-2")
; Instrument package
(find-file "box.el")
(tst-instrument)
; Initialize test environment
(find-file "sample")
; Tests
; TEST box-1
(tst-achieve-state pre-box)
(box-region nil)
(tst-capture-state 'post-state)
(tst-equ-state post-state exp-box-1)
; TEST box-2
(tst-achieve-state pre-box)
(box-region t)
(tst-capture-state 'post-state)
(tst-equ-state post-state exp-box-2)
; TEST unbox-1
(tst-achieve-state pre-unbox)
(unbox nil)
(tst-capture-state 'post-state)
(tst-equ-state post-state exp-unbox-1)
; TEST unbox-2
(tst-achieve-state pre-unbox)
(unbox t)
(tst-capture-state 'post-state)
(tst-equ-state post-state exp-unbox-2)
; Results of tests
(set-buffer "*equal-log*")
(save-file "equal-log-sample")
; Analysis of testing
(tst-analyze)
(tst-display-batch)
) ; let
) ; defun script
| [ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
To invoke the test script from batch, use the following Unix command:
% gnuemacs -batch -l script.el -f script -kill &
Note that the test script explicitly loads “test.el” at the beginning of “script.el”.
An instrumented package runs considerably slower than an uninstrumented package. It is a good idea to test your script without instrumentation (and probably without many of the tests). Once you are satisfied with the script, run it at lower priority and/or at non-peak hours to avoid inconveniencing other users.
| [ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
The state information available for “capture’ing and “achieve”ing is usually more than needed for any particular test. You can improve the performance of testing by filtering out the unneeded components of states. There are mechanisms for doing this before and after testing:
In using these filters you should be aware that too much filtering defeats the purpose of testing.
| [ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
| [ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
| [ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
If you want to extend the equality package to compare other aspects of your
environment, simply write your own comparison function and add it to the
’function-vector’ for the appropriate area. For example, if you were
interested in comparing the foobar attribute of buffers, first write
a function, tst-equ-foobar, and then add it to the list of functions
executed for a buffer, tst-equ-buff-state-functions.
Similarly, if you wanted to test the foo attribute of a state, add
the new function to the list of functions executed for a state,
tst-equ-state-functions.
You can also extend the equality tests by the use of hooks. The method is defined in more detail in the section on Inequality.
| [ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
| [ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
The interactive display mode attempts to provide the fiction of annotation windows which are part of the code window. In retrospect it may be that simply inserting the text into a new buffer along with the code would be as useful. It also may be that the batch style report in conjunction with the “compilation” error parsing is just as useful in practice.
There are certainly other types of annotation which could usefully be displayed, like the number of times each line was executed, etc.
It might make sense to bind ‘tst-display-batch’ to a key in
test-mode. It also might make sense to remove the binding for
tst-analyze and call it only from within the display functions.
| [ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
The annotation database is accessed twice every time an instrumented line of code is evaluated. This code contributed substantially to the slow-down of instrumented code and time spent making it more efficient would be rewarding.
| [ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
If you want to add further functions to the Analysis Package, you will have to modify the main function, “tst-analyze”, which invokes the individual analysis functions. It currently invokes “tst-anl-zero-counts” and “tst-anl-constant-values”. Your new function should use the same idiom as these functions, i.e., use “tst-ann-get-lines” to retrieve the list of lines for which annotation holds information and then use “mapcar” to apply your single line analysis function to each line’s data. You can then use “tst-ann-put” to store your results.
| [ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
| [Top] | [Contents] | [Index] | [ ? ] |
This document was generated on December 12, 2024 using texi2html 5.0.
The buttons in the navigation panels have the following meaning:
| Button | Name | Go to | From 1.2.3 go to |
|---|---|---|---|
| [ << ] | FastBack | Beginning of this chapter or previous chapter | 1 |
| [ < ] | Back | Previous section in reading order | 1.2.2 |
| [ Up ] | Up | Up section | 1.2 |
| [ > ] | Forward | Next section in reading order | 1.2.4 |
| [ >> ] | FastForward | Next chapter | 2 |
| [Top] | Top | Cover (top) of document | |
| [Contents] | Contents | Table of contents | |
| [Index] | Index | Index | |
| [ ? ] | About | About (help) |
where the Example assumes that the current position is at Subsubsection One-Two-Three of a document of the following structure:
This document was generated on December 12, 2024 using texi2html 5.0.