Unpublished draft
Interactive programming environments for fully dynamic languages (such as Lisp and Smalltalk) allow developers to experiment with their programs through an interpreter or other interface while they are constructing them. This kind of activity, which I call dynamic interaction, helps support rapid prototyping and the use of the programming environment as a conversational medium. Dynamic interaction can be implemented in many ways, but at minimum allows a programmer to interactively create and manipulate program objects; to invoke arbitrary parts of the program; and to observe the results of program activity. Dynamic languages are not necessary to dynamic interaction, but make supporting it much easier than it would be otherwise. Such languages generally feature typed objects but untyped variables, late binding, incremental programming, and garbage collection. Environments for dynamic languages usually support the ability to modify code on the fly, the ability to recover flexibly from errors, and full interactive access to the live computational environment.
Languages that are based on a strictly compiled model tend not to support this level of interactivity. Java, while it absorbed some of the ideas of dynamic languages (i.e., garbage collection and self-describing objects), omitted others (such as untyped variables and a uniform object model). Among the left-out features was the ability to interact dynamically with the environment of a running program. This omission is probably due to the fact that Java is both strongly typed and by definition compiled to bytecode, facts which tend to work against interactive interpreters.
Scripting languages share many of the same requirements as dynamic interaction. They too are generally untyped, interpreted, and permit on-the-fly modification of code for rapid prototyping. These days, a great deal of software development takes the form of scripted components: that is, a combination of some prefabricated components and a scripting language for gluing them together into an application. In such a world, where components may be poorly documented black boxes, it seems particularly important to provide developers with the means to perform interactive experiments on them during the course of development.
In this paper I describe some efforts to provide scripting and dynamic interaction capabilities in the Java environment. I constructed a variety of tools that permit dynamic interaction, including a scripting language, a web-based interface, and a variety of graphic object inspectors. All of these interfaces run from within a standard Java VM, and can run alongside existing Java applications such as debugging and scripting tools.
Direct manipulation tools such as WYSIWYG text editors and spreadsheets use interface technology to remove barriers between programmer and application objects. They allow the user to have the feeling of directly interacting with the system under development. This style of interaction is now the norm for end-user applications such as word processors and drawing programs, but is not yet available in most programming environments. Boxer [diSessa and Abelson 86], LiveWorld [Travers 96], and Self [Ungar and Smith 87] are some environments that do provide this kind of interface, which for the purposes of this paper I will call dynamic interaction.
Dynamic interaction is not quite the same as a dynamic language, but the notions are closely related and it is much easier to provide dynamic interaction for a language which is itself dynamic.
It can be useful to contrast the idea of dynamic interaction environments with some related techniques for integrating visual interfaces and programming. Visual builders and most visual programming languages are visual only "before the fact": they allow the user to construct portions of a program through an interactive graphic interface, but the program itself runs separately from the construction phase. Software visualization, in contrast, usually produces its graphic component "after the fact". A program runs and produces a trace which is then turned into a graphic display or animation.
Dynamic interaction, by contrast, involves a continuous interactive relationship with a programming environment. Graphical interface techniques can certainly help maintain this relationship, but are not essential to it.
To make dynamic reflection useful, it requires not only the capability to do arbitrary runtime operations but a suitable user interface to this functionality. I built several different interfaces; the most developed (and the basis for the others) is an interpreter for an extended version of the Scheme language, called Skij. This provides a powerful text-based interface to dynamic interaction. A variety of other tools that used graphic or web-based interfaces were also constructed.
Scripting languages are also used within single applications as extension/customization tools that allow users to customize, connect, and control the components of the application. Examples of extension languages include Emacs Lisp, JavaScript, LotusScript, SIOD (a Scheme dialect, used as an extension language for Gimp, an open source image editing program).
The case for scripting languages as fundamentally different from "system programming languages" has been made prominently in [Ousterhout]. In his conceptual scheme, scripting languages are interpreted, loosely typed, and capable of more abstraction than system programming languages (a category which includes both C and Java). Scripting languages are suited for gluing together components, implementing user interfaces, string manipulation, and rapid prototyping. System programming languages are more appropriate when speed is essential or when programs must implement complex algorithms or data structures.
Contrary to Ousterhout's view, it is possible to use a single language for both high-level scripting and system programming. Examples of systems designed in this way include the MIT Lisp Machine and descendants, which used Lisp at all levels of implementation [Symbolics+++]; and Squeak, a Smalltalk system in which the internal VM is itself written in the high-level language [Ingalls et al 97]. These systems can be enormously powerful development tools, but they tend to create closed worlds, in which everything developed in Lisp or Smalltalk can conveniently interact with everything else; but communication with programs in other languages is awkward at best. Scripting languages, in contrast, are designed to operate in heterogeneous environments where communication between loosely coupled modules is the norm.
Lisp is a protean language which can exist in many forms. It has most recently been been used as the basis for large, closed-world systems and environments; but is now becoming more widely used as a small versatile scripting engine.
Lisp performs computations by evaluating expressions to produce values. A Lisp expression is typically a list. The first element of the list indicates a function, while the remaining elements indicate arguments. Numbers are self evaluating; symbols are used as variables but can designate themselves if quoted. Thus the expression(a b c 1 2 3) ((this is) (a list) (of lists)) (define height (* hypotenuse (sin theta))) ()
means find the value of x, then apply the function + to that and the number 1. Functions are defined by means of lambda expressions, and names are bound to objects by means of define. Thus(+ x 1)
defines a new function, average, that will return the average of its two numerical arguments:(define average (lambda (x y) (/ (+ x y) 2)))
Note that the syntax for invoking a built-in primitive function such as + is the same as that for invoking a user-defined function.(average 10 100) ==> 55
A Lisp listener allows the user to enter expressions from the keyboard and observe the result.
Scheme is in widespread use as a scripting and extension language. It has several advantages in this area:
Since Skij was intended primarily for debugging and experimentation, execution speed was not a primary consideration. In its current state, Skij is implemented as an interpreter and is thus rather slow. Compiling Scheme into Java bytecode is possible (see Kawa), but was not thought necessary for the tasks Skij was designed to serve.
Another disadvantage of Scheme is that it is often seen as a strange or alien language by programmers used to more syntax-heavy languages like C. This problem is partially addressed by a Java-like front-end to Skij (see Jive).
The first stage of this effort involved writing a minimal Scheme interpreter with only a few types and primitives, and no tail-recursion, but with the capability to access Java objects and methods. This produced a proof-of-concept implementation with about a month's worth of work. After that, the implementation was extended to implement all of Scheme, and to provide access to various other aspects of Java functionality such as threads and events.
The basic design principles which Skij uses to integrate Scheme and Java are:
Skij extends Scheme by allowing any Java object (usually including null) to be a Scheme value, used as an element of a list, etc. Internally, Skij typically uses variables whose static type is Object, and casts their values to specific types when necessary.
The Scheme printed representation of a Java object (other than one that corresponds to a Scheme type) consists of the string returned by the Java toString method, quoted by "#< ... >". This ensures that an attempt to read back in a random Java object will generate an error rather than cause unpredictable behavior.
(new class [args]*)
| Package com.ibm.jikes.skij:
Classes for Scheme types: Cons Nil EOFObject InputPort OutputPort Procedure (abstract) CompoundProcedure Macro PrimProcedure (abstract) ~70 individual PrimProcedures (inner classes) Continuation Symbol Internal classes:
|
Package com.ibm.jikes.skij.util:
Invoke Tracer ExtendableClassLoader (extends java.lang.ClassLoader) Package com.ibm.jikes.skij.misc:
"Package" com.ibm.jikes.skij.lib:
|
Eval is still called recursively when necessary. When a tail-call is done, the evaluator does a reduction rather than a new call to eval. That is, it stays within the current method, replacing the original form with a new one. In the fragment below, which implements the Scheme if operator, the predicate is evaluated with a recursive call to eval, while the then and else clauses are evaluated by reduction:
static Object eval1(Object exp, Environment env) throws SchemeException {
while (true) {
...
Cons list = (Cons)exp;
Object qproc = list.car;
Cons args = (Cons)list.cdr;
if (qproc instanceof Symbol) { // check for special forms
String sym = ((Symbol)qproc).name;
...
if (sym == "if") {
Object predval = eval(args.car, env);
if (schemeTrue(predval))
exp = args.cadr(); // reduction, fall through
else
exp = args.caddr(); // reduction, fall through
}
...
}}}
This technique requires that any part of evaluation that performs in a
tail-call must done within the body of the while loop. This forces some
violation in modularity. In particular, the implementation of the apply
method for CompoundProcedure must be copied so that it can run within the
loop.
Each primitive procedure is defined as an inner class of the basic PrimProcedure class, with its own apply method. For example, this defines the Scheme set-car! primitive procedure:
new PrimProcedure("set-car!") {
public Object apply(Environment env, Cons args) throws SchemeException {
Cons mycons = (Cons)args.car;
mycons.car = args.cadr();
return null; }} ;
Compound procedures are simple instances of the CompoundProcedure
class, and have their own body, args, and environment attributes. Every
evaluation of a Scheme lambda expression produces a CompoundProcedure object.
While CompoundProcedures implement apply, more typically they
are invoked from the evaluator in the interests of maintaining proper
tail-calling. Macros are a subclass of CompoundProcedure and work identically;
differing only in the manner of their invocation (see Evaluator).
Continuations have an apply method that throws a ContinuationException. This exception is caught by the invocation of call-with-current-continuation that generated the Continuation object.
Note: Skij does not implement full Scheme continuations. Continuations may only be used as escape procedures; they do not persist indefinitely nor do they allow exited code to be re-entered. This limitation is fairly common in Scheme implementations that use the underlying stack structure for the Scheme stack. Full call/cc requires that control information be heap-allocated or that the stack be copiable.
ProcEnvironments keep track of their bindings in the form of two lists, one holding the names of bound variables, and the other holding the corresponding values. This lets them be created by directly copying the list structures used by the evaluator. Lookup time will be linear in the size of the environment, but since most ProcEnvironments are small this is acceptable.
Although there can be multiple top environments, they all share the same bindings. Rather than keeping them in a linear list, or a hashtable (which has a slow constant lookup time), they use a special form of lookup. Each symbol is given an extra integer-valued index field, which is non-zero and unique for symbols with top-level bindings. When lookup up the binding of a symbol, this value is used to index into a special static array. This produces fast lookup for global bindings (which includes most procedure names). [This idea is due to Peter Norvig].
To handle these cases, Skij provides a function for generating vectors of types other than Object[]:
(%make-vector n class) [Procedure]
creates a vector of length n whose elements are of type class.The fact that vectors might be arrays of primitives forces Skij to use the Reflection API (class java.lang.reflect.Array) for vector access, rather than the more obvious method that would apply only to arrays containing reference types. This reflects a decision to simplify the language; a more efficient design might have separate access primitives for arrays of primitive type.
Even with autoloading, the process of reading in Scheme files can be slow (or buggy in some Java implementations, especially those found in web browsers). To get around these problems, it is possible to translate the Scheme files into equivalent Java source files which can then be compiled into Java class files. This is not compilation, but merely puts the textual Scheme representation into a format that is faster and more reliable to load.
Any Scheme function can be implemented as a primitive (in Java) or in Scheme. Initially, I tried to minimize the number of functions implemented as primitives, to keep implementation time and size small. However, as Skij evolved, some additional frequently-used functions were made into primitives (and some primitives turned out to be unnecessary, and were rewritten in Scheme).
Normal method invocation in Java relies upon compile-time static type information. Method dispatch is done using a combination of dynamic type information (for the target object) and static type information (for the arguments). In a dynamic interaction environment, static type information is not available. The obvious solution is to substitute the dynamic types of the arguments for the nonexistent static types.
Method invocation in Skij involves the following steps (constructor invocation is similar):
In Java 2, this procedure is slightly different, since it is possible to invoke non-public methods. Instead of using the getMethods call, Invoke uses getDeclaredMethods the target class and on all of its ancestors. When a method resolution is found, it is set to be accessible using the new AccessibleObject interface.
(catch . body) [Macro]
Evaluate body. If an exception occurs during the evaluation, the exception is trapped and the exception is returned as the value of the catch form. Otherwise the value returned by the last body form is returned.(%catch thunk) [Primitive]
The primitive on which catch depends. A thunk is a procedure of no arguments. (catch body) is expanded to:
(%catch (lambda () body))
(throw exception) [Primitive]
Causes exception to be thrown.
(error string)
Create and throw an exception containing the message string.
(synchronize object . body) [Macro]
(%synchronize object thunk) [Primitive]
These functions arrange for body or thunk to be called within a context in which object is locked.
(in-own-thread . body) [Macro]
(run-in-thread thunk) [Procedure]
Run body or thunk in its own thread.
run-in-thread is defined as follows:
(define (run-in-thread thunk) (define thread (new 'java.lang.Thread thunk)) (invoke thread 'start) thread)
(new java.awt.Button()).addMouseListener(listener);We'd like to be able to specify the response to a mouse event in Skij. To accomplish this, Skij includes a Java class called GenericCallback that implements all of the AWT Listener interfaces. A GenericCallback object contains a Skij procedure of one argument. When the callback object receives an event, the procedure is applied to the event object.
Thus the following Skij commands will create a button that prints a message when pressed:
(define b (new 'java.awt.Button "Press Me")) (define listener (new 'com.ibm.jikes.skij.misc.GenericCallback (lambda (evt) (print "Thanks, that felt good")))) (invoke b 'addActionListener listener)It would be possible to define a separate callback class for each type of event listener, but it was not necessary since events are self-describing.
Most objects in the java package have few or no publicly accessible
fields, and instead present their state to external users through bean-style
get
methods. Thus, the JDK 1.1 version of the inspector shows the values of
all get methods (defined to be zero-argument methods that begin with
get
or is). In Java 2, non-public fields of objects can be accessed
and can be displayed directly.
A typical use of the inspector is to traverse the object structure
starting from a known object until the object of interest is found. Often
one wishes to perform operations on this object. Skij includes a variable,
inspected,
that is always bound to the object contained in the topmost inspect window.
This lets the user refer to an object obtained via the inspector from a
Skij listener.
In some cases the slot/method view of an object is not very useful. For example, inspecting a list in this fashion would only show the car and cdr of the first pair. So, for certain object types, the inspector shows the contents in a different form. The object types treated specially include lists, arrays, Vectors, Enumerations, and Hashtables.
The inspector gives a view of selected Java objects, but all fields of interest in Java are associated with objects. Static fields and methods are associated with classes rather than individual instances. In order to make these accessible via the inspector, they are displayed as additional fields when inspecting a Class object.
The graph inspector represents objects as rectangles. Color is used to indicate object type. Relations are indicated by labeled lines. Objects and relationships are created on demand, by the user, using a pop-up menu. The graph inspector can be linked to the tabular inspector, so that browsing through one will also bring up objects in the other.
The graph inspector is also useful for revealing hidden facts about object identity. Any given object is drawn only once in the object graph, so the fact that two access paths lead to the same object is readily visible. It's also easy to spot unexpected multiple objects (for instance, I was surprised to learn that the getGraphics() method on AWT components always returns a unique new Graphics object every time it is called, in at least one Java implementation. This fact would be difficult to notice in a textual interface, sine the objects print identically).
WebSpect presents interfaces to operations by means of interface templates that correspond to allowed constructor or method calls. When one of these operations requires an argument, it provides an appropriate interface widget. Strings and numbers can be typed directly into text boxes; booleans are entered by means of a check box. Other types of object are more problematic. How do you specify, say, a Locale? The solution is suggested by the fact that the server already tracks known objects, so for any argument, it can find all the objects of that type that it knows about, and offer them in a pop-up menu.
WebSpect arguments also contain a link to the class of the argument, which allows a user to go off and create an object of the appropriate type, come back, and use it as an argument.
http://fury.watson.ibm.com:2341/webspect/inspect?object=19indicates a request to inspect the object with index 19. A more complex request can include multiple parameters:
http://fury.watson.ibm.com:2341/webspect/create?class=14&po0=19&po1=22indicates a request to create a new object from the class whose index is 14 and with two parameters with indices 19 and 22.
The server translates the incoming URLs to calls to special page-generating procedures, i.e.:
(define-command create (class) (set! class (code-object class)) ; turn the index into a class object (define arguments (decode-parameters params)) ; translate the p0n style arguments (define object (apply new class arguments)) ... generate a page describing the new object ... )
Jive is built from a Scheme-based parser generator (written by Mark Johnson) and a LALR(1) grammar that translates Jive expressions into Skij forms. The Jive listener reads Jive expressions, translates them using the grammar, and evaluates them.
One way in which SILK differs from Skij is that it uses the Java null value to represent the Scheme empty list. Many loops in Scheme code or internal loops of the interpreter are terminated on the empty list, and testing for null is generally fast in Java, so this is a speed advantage. Any Java variable of reference type can contain the null value, but null is otherwise atypical. A null "object" cannot be the target of a method invocation and cannot be stored in a hashtable. Thus using the Java null value for Scheme's empty list can complicate the interpreter and give rise to irregularities.
Ousterhout, John K. Scripting: Higher Level programming for the 21st Century, IEEE Computer, March 1998, http://www.scriptics.com/people/john.ousterhout/scripting.html
Travers, M, Programming with Agents: New Metaphors for Thinking about Computation, Ph.D. dissertation, MIT Media Lab 1996.
diSessa, A. A. and H. Abelson (1986). "Boxer: A Reconstructible Computational Medium." CACM 29(9): 859-868.
The Java Language Specification, Gosling, Joy, Steele, 1996.
Scheme standard, Clinger et al.
Ungar, D. and R. B. Smith (1987). Self: The Power of Simplicity. OOPSLA '87, Orlando, Florida, ACM Press.
Back to the Future: The Story of Squeak, a Practical Smalltalk Written in Itself, Dan Ingalls, Ted Kaehler, John Maloney, Scott Wallace, Alan Kay, OOPSLA 97.