To process a human language text effectively, it is necessary to have some understanding of what the text means.  The Theatre of the Mind (TOM) project is my attempt to display text meaning in a way that is not just another text:  TOM provides a “virtual stage” on which animated characters act out the meaning of the given text.

The first implementation, called “JackAndJill”, began with the nursery rhyme “Jack and Jill went up the hill to fetch a pail of water…”.  It displays a stage and backdrop against which one can set up certain locations (e.g. the town called “London”, the hill called “High-Hill”) and characters (the boy “Jack”, the girl “Jill”, the boy “Sam”).  The programme accepts texts such as “Jack went up the hill” or “Jill has gone to London”, and produces semantic structures (“scripts” in our terminology) corresponding to the successful parse or parses.  The scripts can be “played” and when played, the characters move appropriately on the stage.  (see Figure 1)

Figure 1. Jack and Jill on the Virtual Stage.  Jack is going to London.

Our objective was not to create state-of-the-art animation, so the techniques used are elementary; nonetheless, the characters move in 3-dimensional space, orient themselves correctly for the path they are following, and move their arms and legs.  Some of these body-motions (i.e. change of configuration of body parts vs. translation of the whole character to a new location) are used to distinguish one kind of action from another (e.g. “go” vs “dance”; with “dance”, the character twirls around as well as moves forward).  So, the animation is as sophisticated as needed to capture the semantic distinctions in the text and to do it in a non-text medium.

I have always been concerned that semantic explanations which (for example) translate “John runs”  into “run(john)” are overlooking some of the necessary details.  To be fair, when “run(john)” is a statement in a formal logic, it has a well-defined meaning.  As such, it is capturing (in a well-defined way) something about the meaning of the text, and in principle, the logic statement could drive other representations such as  our animation.  But there is the uneasy feeling that moving from one text notation to another text notation can simply hide the challenging parts of the interpretation;  when one has to see the meaning acted out, it is intuitively clear what has (or has not) been captured.

The JackAndJill programme uses a parser that maps strings of characters (divided into words by the presence of blanks or line ends – so called white space)  into constituent structure trees and uses these trees to compose elementary semantic units (called “intentions”) into a composite semantic structure (a “script”). We will describe these in more detail in subsequent notes.   Here we note that the process allows for multiple parses and therefore multiple interpretations of a text: the lexicon (mapping strings of characters to syntactic categories and their associated semantic intentions) can have multiple entries for the same string;   the syntactic rules can apply in multiple ways to the same input ([Jack talked to [the girl] in London] vs [Jack talked to [the girl in London]]) and the virtual stage can provide different scenarios for the same string, depending on the prior configuration of characters (The performance of “Jack goes to London” varies depending on where Jack is placed at the start).   A more enhanced version of the JackAndJill programme would allow the user to manually adjust the state of the stage.

We envision the JackAndJill programme and other like implementations being used in various ways:

  1. It leads us to build a computational model of language and language processing.  As such, it is an aid to our understanding of natural language, on the one hand, and of the various academic theories about natural language on the other hand.   Existing theories often are isolated from one another (e.g. a theory of syntax not relating to any theory of semantics and vice versa)  but to make a successful JackAndJill programme, it is absolutely necessary to have all the parts of the system working with one another.
  2. A working system can be used in practical settings, for example, to teach language by allowing a user to try constructions and see what they mean.  English “in” and “on” are not easy to decipher: (in my dialect at least) one lives “on a street” but “in a town”, although one could play  “in the street” or “on a team”
  3. A more robust version of JackAndJill could be used by authors to create stories that are acted out on stage (What would William Shakespeare have done with such a tool?).  More mundanely, the tool might help the authors of work instructions, standards documents, and regulations to visualize what their writing actually says.  The visualization could also be an adjunct to the written document to aid the reader as well.

All of these applications require a breadth of coverage and a depth of function that is not currently there, but with the current state of information technology, they remain real possibilities.

We intend to make the JackAndJill grammar handle the core features of English, such as the syntactic structures (phrases, optional and mobile elements, embedded clauses, etc, — the English auxiliary construction is particularly interesting computationally), the use of closed class categories (determiners, quantifiers, prepositions, particles, conjunctions and so on) and “basic” open class words (including representative verbs of motion, perception etc.), so that the grammar can be extended to a particular field or application by adding more open class items on the model of  what already exists. Thus “walk”, “trot”, “canter” and “gallop” could all been seen as extensions of “go” (or perhaps “dance”) by specifying the mode of going.

We also hope to explore some of the corresponding semantic fields from a computational perspective, such as the language-user’s implicit notions of space, position and motion (driven by the semantics of prepositions).

All these these things will be discussed in more detail on subsequent notes to this blog.  Readers who have comments or contributions to make are invited to write me at:

wheeler <at>  (where you replace <at> with @)


Wheeler, Eric S. 2009. Theatre of the Mind: A Project to Animate the Language of Thought and Communication. in E-Learning and Digital Media. 6.3 Special Edition. Sept 2009. 272-273.

Wheeler, Eric S. 2009. Visualizing Language. Presentation to the International Linguistic Association. New York. April 2009.