User Interface in Interactive Storytelling

It seems that I've gone AWOL for about a year. I don't think anyone is particularly saddened, but I have some more posts to do now that I have the time.

I have been doing research on Interactive Storytelling for the past few months in preparation for my senior honors dissertation in computer science. One of the things that I've been researching, originally for the aim of finding examples of how stories can be formalized, is semiotics. I ended up coming across a website for Algebraic Semiotics, a project in which sign systems are mathematically formalized as the basis for creating a science of user interfaces.

Having now seen the use of a mathematical application of semiotics to both the structure of a story and creating user interfaces, I was brought back to a central problem of interactive storytelling: creating seamless interaction between the user and the story that she* perceives. This "perceived story" (the way in which the user interprets what comes onto the screen) is linguistic, not spatial (my interactive storytelling engine is language-based) and must therefore be able to allow the user to give feedback appropriate to the words that appear on the screen. It's important to note that the design needs to be geared towards how the reader reads into a given sentence as far as short-term statements go; intent means little in design and nothing in aesthetics.

What this pointed me to is that the interface demands for an interactive storytelling engine are unusually high. What the user sees is the only thing that the user is going to get, because the entire story is a real-time experience. This is a lot different than say, Microsoft Word, in which the final product (in hard copy, for simplicity's sake) is independent of what the user sees on the interface. All the interface is, in the context of wanting to print a hard copy, is something on the surface that gives clues as to what you will get for a final product. In Interactive Storytelling, the output of the interface is the ultimate end.

There is an additional demand, then, on the interface to an interactive storytelling engine. There has to be a harmony between the internal state of the software and the interface in which the interface could hardly be seen as a layer of buttons and switches; perhaps not entirely—the computer could be imagining and entirely different scenario than the person is and the person could still be getting output that makes for a good perceived story; but in terms of functionality, the interface has to have some kind of direct correspondence.

In addition, the user is not only pushing buttons on the screen for some type of desired feedback, she's also pushing these buttons with her input as part of the experience of the story. The perceived story is the entirety of the user interface. Anything going on below the boilerplate is propping up this real-time surface interaction.

To put it another way, every decision the user makes has to have results that "make sense" to her (I say "makes sense" because there are some things where it may not make sense to the user because she is not thinking it through in people terms, such as if in a romance story, she punches her friend in the face and then wonders why he called her a bitch*; on the other hand, to fit my example, it should never happen that she punches him in the face and he does not react.) Notice how by "react", it means that something appropriate be displayed on the screen.

Because of the need for functional equivalence between the internal engine and the interface (if the interface is doing one thing but telling the user another, and it works the same as if it were thinking the same thing as the interface, then it is functionally equivalent to thinking the same thing as the interface), the engine has to think in terms of the interface (a.k.a. in terms of the story that the user is perceiving.) Therefore, the engine must think in the same language that the interface is using, as each language has a unique syntax and thus expresses its own set of ideas in its own way. The engine has to think in this language and then speak back in this language.

This does not necessarily mean simply "English"; books themselves have a language that defines their ideas. This means that the dramatic space that the engine explores has to be in the literary language of the story, which will make for a very different way of formulating a world. Nonetheless, it is the only way to be in line with the user's perceived story—there cannot be some hidden "spatial universe" independent of the linguistic interface as there is no tidy metaphor that exists between these conceptions of the world. Trying to translate between the ideas of the spatial (or any other space of ideas) and the space of the story's dramatic language, that which the user inputs to and sees output from, would lead to an entirely dissatisfying story.

I will say quickly that I don't mean to invalidate having no basic conceptions of space and time: there is a deep structure to drama, I believe, just as there is a deep structure to lingual syntax and so it would make sense for the computer to receive a sentence, interpret it into some sort of "semantics", come up with new "actions" and then translate said "actions" back into words; but we must be careful to remember that first and foremost the engine is producing a sentence.

I'll try to summarize all this here. The chronologically ordered set of all inputs and outputs that the user enters and perceives respectively is the entirety of the story. Therefore, the engine must think entirely in terms of these elements, as the story and the interface are one in the same. This means creating an engine that sees stories in a dramatic language and expresses them in that same language, and a giant headache over the course of the next ten months.





*Why did I use the "she" pronoun? Two reasons. 1) I refuse to write "he/she", the political-correctness police try to ignore the fact that gender seems to naturally arise in language and sometimes in far more ridiculous ways than English speakers can imagine (if you don't believe me, take a look at Swahili.) 2) Nonetheless, my projects are meant to appeal to a base that is not the traditional masculine gamer, and so I'd like to keep as much emphasis as possible on that. Just a note.

*I warned you, I'm not politically correct.