More or less orthogonal to the issue of how to map AIL operations to the Python language is the question of how they should be implemented.
In principle it would be possible to use the same strategy that is used for C: add an interface to Amoeba's low-level RPC primitives to Python and generate Python code to marshal parameters into and out of a buffer. However, Python's high-level data types are not well suited for marshalling: byte-level operations are clumsy and expensive, with the result that marshalling a single byte of data can take several Python statements. This would mean that a large amount of code would be needed to implement a stub, which would cost a lot of time to parse and take up a lot of space in `compiled' form (as parse tree or pseudo code). Execution of the marshalling code would be sluggish as well.
We therefore chose an alternate approach, writing the marshalling in C, which is efficient at such byte-level operations. While it is easy enough to generate C code that can be linked with the Python interpreter, it would obviously not stimulate the use of Python for server testing if each change to an interface required relinking the interpreter (dynamic loading of C code is not yet available on Amoeba). This is circumvented by the following solution: the marshalling is handled by a simple virtual machine, and AIL generates instructions for this machine. An interpreter for the machine is linked into the Python interpreter and reads its instructions from a file written by AIL.
The machine language for our virtual machine is dubbed Stubcode. Stubcode is a super-specialized language. There are two sets of of about a dozen instructions each: one set marshals Python objects representing parameters into a buffer, the other set (similar but not quite symmetric) unmarshals results from a buffer into Python objects. The Stubcode interpreter uses a stack to hold Python intermediate results. Other state elements are an Amoeba header and buffer, a pointer indicating the current position in the buffer, and of course a program counter. Besides (un)marshalling, the virtual machine must also implement type checking, and raise a Python exception when a parameter does not have the expected type.
The Stubcode interpreter marshals Python data types very efficiently, since each instruction can marshal a large amount of data. For instance, a whole Python string is marshalled by a single Stubcode instruction, which (after some checking) executes the most efficient byte-copying loop possible — it calls memcpy().
Construction details of the Stubcode interpreter are straightforward. Most complications are caused by the peculiarities of AIL's strategy module and Python's type system. By far the most complex single instruction is the `loop' instruction, which is used to marshal arrays.
As an example, here is the complete Stubcode program (with spaces and
comments added for clarity) generated for the function some_stub() of the example above. The stack contains pointers to
Python objects, and its initial contents is the parameter to the
function, the string buf. The final stack contents will be the
function return value, the tuple (n_done, status). The name
header refers to the fixed size Amoeba RPC header structure.
| BufSize | 1000 | Allocate RPC buffer of 1000 bytes |
| Dup | 1 | Duplicate stack top |
| StringS | Replace stack top by its string size | |
| PutI | h_extra int32 | Store top element in header.h_extra |
| TStringSlt | 1000 | Assert string size less than 1000 |
| PutVS | Marshal variable-size string | |
| Trans | 1234 | Execute the RPC (request code 1234) |
| GetI | h_extra int32 | Push integer from header.h_extra |
| GetI | h_size int32 | Push integer from header.h_size |
| Pack | 2 | Pack top 2 elements into a tuple |
As much work as possible is done by the Python back-end in AIL, rather than in the Stubcode interpreter, to make the latter both simple and fast. For instance, the decision to eliminate an array size parameter from the Python parameter list is taken by AIL, and Stubcode instructions are generated to recover the size from the actual parameter and to marshal it properly. Similarly, there is a special alignment instruction (not used in the example) to meet alignment requirements.
Communication between AIL and the Stubcode generator is via the file system. For each stub function, AIL creates a file in its output directory, named after the stub with a specific suffix. This file contains a machine-readable version of the Stubcode program for the stub. The Python user can specify a search path containing directories which the interpreter searches for a Stubcode file the first time the definition for a particular stub is needed.
The transformations on the parameter list and data types needed to map AIL data types to Python data types make it necessary to help the Python programmer a bit in figuring out the parameters to a call. Although in most cases the rules are simple enough, it is sometimes hard to figure out exactly what the parameter and return values of a particular stub are. There are two sources of help in this case: first, the exception contains enough information so that the user can figure what type was expected; second, AIL's Python back-end optionally generates a human-readable `interface specification' file.