From Physical Narration
Revision as of 12:57, 11 September 2012 by Emilie (Talk | contribs)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search


Data Ecologies 2012: The Map and the Territory

1 1 1.jpg

Day 1: 15th June 2012

Welcome by Simone Boria

Data Ecologies is a series of seminars which deal with the way that real and abstracted systems interrelate. The previous one took place in 2009 in Brussels with Money as an abstraction (Data Ecologies 2009), which explored issues around the use of money and how the investment and maintenance of non-money assets work whilst also exploring value. The aim of the Data Ecologies series is to link theory with artistic practice and create a space for change, reflection and simulation. This years subject explores notation techniques and their possibilities of using artistic practice, with the intention to search for ways to write and in the wider sense, invent tasks and processes in order to create a toolkit for transmission. In a nutshell, Data Ecologies was created as a platform for exchange, reflection and simulation with input from experts, who work in diverse areas related to the topic. The eight experts presented their research and findings related to the topic from theory and practical point of views.

1 2.jpg

Introduction by Tim Boykett


Tim briefly discussed why everyone was chosen to come to the seminar and what they would like to be covering over the next couple of days. The Data Ecologies series is a nice way to bring together people that Time's Up would like to learn something from and have a discussion with. They were partly looking forward to The map and the territory discussion so as to pick people's brains about ideas which they're using, which can then be discussed. One of the non speakers, but with an important presence nonetheless is Gerhard Dirmoser, who kindly brought along some of his diagrammatic representations of his exhibition work and also material from an exhibition currently showing at Nordico. One of the diagrams collates an exhibition with 600 pieces and reduces it to a diagram, abstracting the structures in the show.

2 0.1.jpg

2 0.2.jpg

2 0.3.jpg

To begin his presentation, Tim started with maps. One of the problems with flat maps is, that they are not very representational.


Tim talked about different kind of maps and what people prefer. An example being a navigator map called the World Mercator Projection Map, as it helped with sailing:


There are maps such as this one which try to reflect land-mass and try to show the world in a different way rather than just showing it as a rectangle:


This became important for Buckminster Fuller's maps of the globe, The Fuller Projection, which is all about the land masses:


Tim showed us a map by Athelstan Frederick Spilhaus, which is about the representation of the water rather than the land and the water masses of the earth. In the map below you can very much see Africa but not so much of America:


We also looked at a map by Chuck Clark, which shows the world's watersheds, but also shows all the continents at their true scale:


A lot of guides will have complicated textual descriptions - and they are sometimes very accurate, but they don't always help you to work out where you are. They do tell you about nice little things which a map might not tell you though.

One of the other areas of interest is the theatre and how they use maps. This representation is done just through words to say what is going on. The map takes the form of the text showing what people are doing and saying, which is a way of reducing a piece of theatre to a reproducible textual object, which works because theatre is textual. Tim was sure that as we would probably see in Toxic Dream's presentation, the way in which you leave behind the textually can make things become more complicated.

We've moved into the modern film age, which has become very important and with this the storyboard has become an important part of it:


The storyboard contains instructions as to how camera and objects are moved in order to coherently help a film team establish what is going to happen.

Code/scripts also offer themselves as a way of abstracting something which can happen into a code. Without comments and nicely labeled variables, it can be difficult to establish what a piece of code does. People often use inappropriate names for variables making it difficult. There is even a competition called the Obfuscated Contest [1] in which people create code which is even more unintelligible than it needs to be.


Tim then showed us an example of some code which is laid out to look like a labyrinth and in fact when you compile it, you have something which creates labyrinths. In doing this the code is allowing you to see what it happening within it.


Code is also about mapping and trying to figure out what's going on. One problem with it is, that you need to abstract away from it in order to figure out what is going on.

Flow charts show us the process of starting somewhere through to making decisions and doing things:


You could turn it into a piece of code, but a flow chart much more reflects human behavior.

Tim made the point that one of the types of people, who weren't at the forum were business people. It's apparently good to often have flow charts and diagrams in business environments as it makes processes more understandable. It shows the movement of information and objects in space.

He moved on to talk about the MONIAC Computer (Monetary National Income Analogue Computer), which was created to model the national economic process of the UK. It was made by William Phillips in 1949 and reflects the idea of money moving around like liquid. Many things about the machine are instructional and it was a way of helping people to see how things fitted together such as interest rates, costs of money etc:


Another set of instructions and process based things that people do is making recipes, which provide a code/abstraction/diagram of the food to be created. There is an idea that engineers find them hard to follow and so special recipes have been created just for them, which list the ingredients on one side and the instructions of how you combine them over time on the other side:


This is a interesting way of showing an object and how you transform it over time.

Another example is data flow diagrams. This can be seen in programs such as Max MSP and pure Data. Here is a diagram of the flow of food:


We next looked at an example of musical notation, something which Elisabeth Schimana would be touching on again later in her presentation. Tim referenced Andre Smirnov from the Theremin Centre in Moscow, discussing some images which Smirnov had collected showing music conducted by Evgeny Sholpo for early Soviet movies. The music in this was not created by orchestras, but rather electronic:


With this score, the music can't be composed with notes as there is a different ability to make sound. This example shows again a difference of how notation can work.

Referring back to the idea of who was missing from the seminar, people who make comics were one of them. Comics are very much about abstraction and have very little representation going on, but they do just enough to make it work. Another group of people missing, whose presence would have been good was people who make technical drawings. They look at how to show instructive ways to abstract machine bits out into diagrams. The other people who would have been welcomed at the seminar were people in film production and as mentioned before, business.

Time's Up: Physical Narration

3 0.jpg

Andreas Mayrhofer started by introducing Time's Up to the audience and explained why they ended up being interested in notation. The group has been together for 16 years and there are various interests involved with different people and approaches and so one strength of the group is to bring it all together to build something. They investigate how people interact with each other and explore their physical surroundings, whilst discovering, learning and communicating as they do so. Time's Up's research is based upon constructing interactive situations which are similar to our normal, physical world and their presentation for Data Ecologies explored this work.

3 1.jpg

Time's Up's practice began by building huge mechanical installations, creating pieces which were almost like playgrounds where the audience had to do something in order to engage with the work. It wasn't just installation work which they focused on, but also creating social gatherings that create a place to have a dialogue about different approaches. They are interested in exploring narratives. In 2007 they had the idea to build a city which would be a 2000 sqm installation, and this project didn't happen but from it came a lot of different ideas one being that they realised that narrative elements were very important to them. This led them to focus on that branch. Below is a diagram depicting the direction of their work between 1996 and 2010:

3 2.jpg

Tim spoke about how Time's Up goes about building these worlds, which work in the the same way as real ones, even though they are not real. They're very physical and one easy thing to use in order to build them is a map.

3 3.jpg

The map above is from 1997 and shows the layout of objects in an installation which was featured in Ars Electronica called Hyperfitness [2]. The map depicts the route which the audience would go as they passed through the space. During this they had choices of paths which they could follow, and they could interact with elements which would control parts of the installation. The map was a simple way of taking the spaces and talking about peoples experience with regards to how they correlate backwards and forwards.

Tim moved on to discuss a piece called Reality Shift which consists of 16 cylinders that rotate around their own axis in groups of 4 using handle devices.

3 4.jpg

3 5.jpg

As the cylinders turn the audience can move from one to the other through doors, the piece being like a dynamic labyrinth. Tim spoke about how when you have worked out how to build something so that it does what you want it to do, the difficulty is trying to get everything working together in a certain way. The hand drawn diagrams below show how the cylinders work, this notation being a neccessity when starting to think about more than four of them working at a time:

3 6.png

3 7.png

What can be seen is the view above of the blocks of cylinders. The cylinders are grouped together in a group of 4 and rotate against one another. One important factor was that when someone leaves the block of cylinders, they shouldn't be in the same state as when entered, otherwise no one would be able to get into them afterwards. This means that there is only one position possible where people can get into and out of the blocks. They also didn't want it to be too obvious how to get through it, leading them to look into the idea of the cylinders moving against each other. Safety was a concern, as if the doors moved in contrary directions to one another as this would make a chopping action when the doors closed. One surprising reaction from people when engaging with the piece was a competitiveness which took over, leading Time's Up to restrict the number of people to one in there at a time. The diagram helped with establishing how to move them around. The second diagram above shows the path to take in order to navigate through it. What was nice about the dynamics of the piece was that the spinning made people lose their points of reference as all there was were the lights from above.

Tim next spoke about a more abstracted, media-rich environment piece called Black Box Sessions [3], which they worked on with Alex Davies. The audience goes into a completely black environment and the only light source is a hole the size of a pin prick, which when looked through can be seen a screen showing an infrared image of them in the space. The audience sees themselves from behind as well as other things around them, including the performers walking in. The image below shows a performer called Patrick Huber interacting with an audience member:

3 8.jpg

The interesting thing was that they had created a 'green box' scenario where they did recordings of the performers acting as if they were in the dark. They wanted to be able to synchronize it all so it worked properly, leading Time's Up to sit down and work out everything which could happen in the space, including playing the video and merging it with the live footage from the camera. To help realise this, they created a diagram showing the ideal things which would happen and the not so ideal:

3 10.jpg

This enabled the Max MSP patch to do everything at the correct time. They also created a numbered diagram to assist with working out this process:

3 11.jpg

This piece was one of the more simple pieces, working out how you make things work the way you want them to, programming the computers, what can go wrong and with this was created a lot of notation. This helps you to create it - it's all very process based.

The next piece which Time's Up created was Domestic Bliss [4], which was shown in the House of Stories in Linz:

They created a whole set for it and the story was set in the early 60s revolving around a crime scene. The story was told on small TV screens around the space. There were objects in the space which were interactive and acted within the story. For example when the light went off, rain and thunder would start. People would go into the story, being invited in by a narrator and then they would go into the story.

This was Times Up's first experience of great narrative storytelling, it was very linear and was multilayered with the actors and the environment. They wanted to involved the audience more and more into the room, so a key part of the piece was creating the atmosphere. This piece is quite difficult to define as a medium, it's not quite theatre or cinema, it's a mixture. It has the ability to really trick people. For example, they hear the rain when in the space and so think that when they exit they will need an umbrella. After three weeks preparing the space, Andi would still get caught out by the shaking cupboard.

There are so many questions at the beginning of creating a piece like this. Time's Up looked a lot into the semiotics of objects in order to establish what to use. They had to think about whether they needed a concrete story and whether everything in the room needed to be related to the story. It's also about learning where to focus.

Tim spoke about the piece 20 seconds into the Future [5]:

3 12.jpg

They wanted to make the piece accessible to a large number of people, and so they had a presentation on the wall created by a fictional scientist. There were papers on his desk, a mathematical genealogy on the wall, photos of people he knew and a pong game. All these things fitted together. There were also letters from fictional other characters in the world:

3 13.jpg

The red objects in the diagram were the ones which Time's Up thought might be the entry points for people to start engaging. They anticipated that the network of objects would help the audience to explore the story more and more. Some people might be more interested in the objects and deciphering the narrative, whilst others would be drawn more to the radio or the diagram on the wall. This was an attempt to explore the whole world of the fictional scientist.

The next piece discussed was Stored in a Bank Vault [6], which took the focus on physical narratives further. It's about a heist, where a group of people are breaking into a bank but as the audience you're not entirely sure at first why they have done it. When the audience walks into the space they can hear a man talking on phone, who is taking a bath as he had to climb into the sewers during the heist. There is evidence of computer hacking, and bits and pieces lying around the space such as documents from a forgery workshop created to get the people into the bank, and on the wall there are photos of the banks and plans of the heist. As the audience explores more and more, they establish that it isn't money being stolen but rather seeds which have mythical properties. There are many ways in which the audience can move through the space. Time's Up even added information to Wikipedia in order to allow people to see what was going on in the narrative. This ties into alternative reality narratives where people create semi fake universes which connect into ours.

The piece also contained multiple timelines, unlike Domestic Bliss which had one timeline. This mirrored the style of timeline found in 20 Seconds into the Future, with small story elements floating around. They created a version of the piece called Im Tresor [7], which followed one timeline and was very theatrical. To synchronize everything in it, Time's Up built a complicated timeline programming environment, for which they used a graphical notation.

Andi introduced Unattended Luggage [8], which is smaller than all of the other pieces discussed, taking the form of a wardrobe which contains relics that tell a chapter of the story embedded in it. The wardrobe contains a family story from over four generations, containing 'episodes' with each family member's story being in a drawer. This allowed to play with multiple layers of storytelling.

From discussion, Andi said that they hoped to build a vocabulary around multilayered installations.

Professor Robert Rotenberg: The Agency of the Map


Professor Robert Rotenberg is based at DePaul University in the USA and teaches courses in many aspects of anthropology, with research interests that involve the city, the museum and material agency. At Data Ecologies' he presented a paper to us entitled The Agency of the Map, exploring objects and how those which are not intended to be maps in a formal sense, can still serve as a directional guide toward a desired goal.


Robert began by discussing what we might think about when presented with the term 'material agency'. He suggested we might think of Stephen King novels where automobiles and toy dolls become alive, or we might think of notes coming off a music score and manipulating the musician's fingers to make the right sounds. He told us of how a lot of his colleagues, especially in archeology have a mystical approach to the notion of material agency. However to Robert it isn't that mysterious, it's something which just people have.

For Data Ecologies Robert wanted to talk about the agency of the map as when he was reading the abstract to the conference, he found himself paying less attention to notation and more to the idea of mapping and territory and their relationship to each other. He then came back to the problem of notation as you can't have a relationship between mapping and territory except through notation.

Robert wrote a paper for his presentation, along with a PowerPoint. His starting point was to talk about how material agency works, and to discuss an object which has nothing to do with maps and territories but which we all use: a dinner plate. Below is his paper along with images to accompany it.



Agency is a human quality. It refers to any action that is intentionally performed by someone and understood by others. Material objects cannot have agency because they cannot have intentions. They are not conscious. However, for the last fifteen years or so, anthropologists have shown that objects contain implicit messages that can direct, modify or augment the actor’s actions. Agency in material objects inhabits a world of ambiguity and indeterminacy, frustrating our efforts to grasp it as if it were concrete and fixed. Like the particle and the wave in quantum physics, the object and its agency coexist in the same moment but are apprehended separately. When that object is observed, its agency disappears and when the agency is observed the object disappears. I see my task today to make the implicit agency of maps explicit. Maps, for me, today, include any representation of space, both in time and through time.


We can begin with a example of how agency is implicit in an object that is not a map: the simple dinner plate. The plate holds our food and facilitates an orderly tidying up when the meal is concluded. This is the plate’s function. When we look at the plate, we think of the food it holds while we are eating. This is merely the beginning of the role the plate will play in our social lives. The plate has three implicit messages for us that change over time as we use it. First, the implicit message of the empty plate is “You are now free to arrange food on me as you choose, but do not exceed my perimeter." When food is served on platters at the table, and the diners can arrange the plates as they like, no two plates look the same. Are there rules for arranging the food on the plate? Certainly for some diners, strict separation of elements is a rule. For others it is not. When food is served on plates from the kitchen, the cook creates an arrangement that takes several different features into account: Portion size, texture, color, viscosity, or aroma all play a role. After the plate is filled, the canvas falls away as 'ground' to the food’s 'figure'. Just as with a picture on the wall, we see the painting, but not the canvas.


This is not yet the full extent of the plate’s messages for us. When placed on the table in front of a diner, we feel the plate’s perimeter. That edge separates our food from the food of others. The plate becomes an extension of our bodies. It doesn’t matter that the food came from the same pot, or that our forks or fingers have not yet touched a morsel, the moment the plate is placed in front of us, that small bit of territory becomes as personal to us as the inside of our mouths. While children may delight in taking food from the plates of their siblings, adults respect the boundary of the plate. Only people who are on intimate social standing share food in public. Once this perimeter is acknowledged, it disappears. The diner proceeds to eat the food that is knowingly hers.


The plate’s messages to us continues to a final phase. When the meal is finished, the plate circumscribes our waste. The food that remains on it is now transformed from the desired to the rejected, from food to dirt. Unlike the remains themselves, the implicit message of the plate at this moment is “Wash me and reuse me." The plate’s resilience as an object is never more evident than when it is ready to be washed. That message is washed away with the food scraps, restoring the plate to its initial state: a canvas waiting to be filled.

Where do these messages reside? Plates do not come with a set of instructions. Therefore, they must reside with our comprehension of the plate. There is no mistaking a plate’s condition. It is dirty or clean, empty or full, set before the diner or stacked in the pantry. The substance of the plate itself is not transformed by our use. We respond as if messages emanated from the plate, but plates can not talk. We talk for the plates. For this reason, material agency can best be characterized as a non-verbal dialogue between people about objects.


Maps, Plans, Patterns and Recipes

My goal today is to use this idea of a non-verbal dialogue between people to make our production and consumption of maps clearer. We can begin with maps that only take shape in language. That is, maps that have no material form other than the sound of words. In a classic article about how residents of New York City describe their flats, Linde and Labov (1975) discovered that all the descriptions fell into only two types, The first type is some variation of the following: “The bedroom is next to the kitchen”; the second type sounds like this: “You turn right and come into the living room.” These are labeled, respectively, the 'map' and the 'tour'. In this particular study, only three percent of the people interviewed chose to describe their flats using the 'map' style. All the rest chose the 'tour' style.

These two types of descriptions, the map and tour, illustrate a longstanding and critical difference in how people understand their environment: seeing vs. going, presenting a tableau vs. organising someone’s movements. The tour is a more aggressive dialogue between person and object. It limits choices and insists on keeping to a specific path. It makes sense that it would dominate in people's descriptions of their flats. They want us to see their homes in a specific series of impressions in a particular sequence. The tour includes effects (‘you will see . . .’), limits (‘there is a wall’), possibilities (‘there is a door’), and directives (‘look to your left’). The tour produces a representation of the flat in our minds as the resident wants us to experience it. It brings the flat into social existence. This is the tour's agency.

The map form of the description affords more choice. It is heavy with positional pointers: 'this is next to that,' or 'this is before that,' but is lacking in action or performance. The resident is saying to guest, "Here is the floor plan. You may move through anyway you wish." The resident is in dialogue with the map as he presents the tour. From it, he selects a path and narrates its landmarks. These landmarks and all possible paths are potential in the map. This is the map's agency. As spatial practices, then, both maps and tours are dialogues about a territory, but with different intentions to direct, modify or augment an experience.


Wool and Pastry

Beyond language, we also experience tours and maps as sensory representations of different kinds of spaces. All of these focus on the important places or moments in a territory and leave out the unimportant ones. I am always surprised at how many different kinds of time-based experiences can be described in maps and tours. I want to focus on two that we rarely associate as either spatial or time-based, wool and pastry. My reason for doing so is to make the variety of maps and tours people produce explicit. I hope that by exploring these unfamiliar and exotic territories, we will see the maps and tours in a new light.


Let's begin with wool. Cleaned, separated and spun into yarn, wool is woven into garments by various techniques. The most complex of these fabrications is knitting, a craft practiced by hundreds of thousands of people throughout the world and an art practiced by hundreds of people. The experience of knitting a garment is one that unfolds in time and space. The garment is the territory that is created as one moves through it, one stitch at a time, until the garment is complete. Knitting a single sock takes about 20 hours and involves making 34,000 individual stitches. Knitting a man's sweater takes over 200 hours and involves making 312,000 stitches. In both cases, one starts out on the journey with a single stitch. After a while, the stitch pattern may change, or the garment might require shaping, a feat accomplished by tactically increasing or decreasing the number of stitches in a row. Holes for arms may require dividing the work. Heels may require special techniques to create the 'turn'. The journey ends by stitching the various pieces together, or closing up the tube. Since it is a journey with a starting point and destination, this journey ought to be describable with maps and tours. Indeed it is.


This is a representation of a stitch pattern (die Strickanleitung) for a jumper, a Guernsey frock design by the British fabric artists Kim Hargreaves, as one would find it in a contemporary European and American instructions for making a garment. This is a map. It is constructed on a grid were each box represents one stitch. The jumper is built up of only two stitches, a knit stitch (rechte Masche) that results in a bit of the wool being pushed forward and indicated by a empty grid square, and a purl stitch (linke Masche) that forces a bit of wool to be pulled backward and represented by a square with a dot in it. These symbols vary a bit in different European languages. It is the alternation of these two stitches that produce the brocade decoration for the jumper. The map begins in the lower right corner and moves row by row to the top left. This particular map superimposes the boundaries for seven different sizes. There is also a gap between the bottom edge pattern and the chest pattern in which one knits row after row of knit stitches only until a specific length is completed. Then, one begins the chest pattern.


It would be possible to fashion the jumper using only this map. Doing so affords the knitter many more choices of how to accomplish the pattern. However, all European and American instructions include a tour of the territory of the garment that uses only words to describe the space. This is the tour description for knitting the back of the jumper. Like all tours it is highly directive and removes options from the knitter. It assumes that the front and back will be knitted separately and sewn together when completed. Knitters could knit both pieces together as a wide, round tube until the armholes. It invokes a knowledge of landmarks that are specific to territory of jumpers, such as side vents, arm holes, right sides and wrong sides (WS), garter stitch and straight stitch (st st). Finally, it connects to the map with its references to Chart A and Chart B. The tour of the jumper is a document of the designer's knitting practices. Most craft knitters will follow her tour to the letter. Fabric artists will use her tour as an introduction to the territory of the garment, study the map, and then adjust the tour to their own practices.


The wool example so far has emphasized European and American garment territories. Switching to Japanese practices, we find that map and the tour can be combined. This is a pattern for a vest. The annotations in English were added by Western fabric artist who was attempting to teach non-Japanese readers how to interpret the patterns. This is very useful for us, as well. The map of the garment begins in the same place, the lower right. But instead of a grid, we see the external and internal boundaries of the territory only. The figure on the left is the back of the vest, while the one on the right is front of the vest. The knitter knows that both halves of the back or front are knitted together. Then the front and back are sewn together. The lines in the lower half of the vest are rows where the stitch pattern changes. That stitch pattern is given as a specific formula: x number of stitches for y number of rows repeated z number of times. The particular kind of stitch is indicated by a character or can be seen in the photo. The characters in the band at the bottom of the vest translate as 1x1 ribbing. The numbers separated by hyphens indicate changes in the number of stitches per row when shaping the arm holes and neck. Binding off ties off a stitch and reduces the number of stitches in a row. So, for example 2-2-2 means "every two rows, bind off two stitches at the arm hole edge and do this twice." Then the next instruction is 2-1-5, or "every two rows, bind off one stitch and do this five times." This continues until the knitter reaches the top of the arm hole.

As with the Euro-American tour and map, this combined representation assumes that the knitter is familiar with its conventions and landmarks. For example, the knitter is supposed to know that 1x1 ribbing is always knitted with needles two sizes smaller than the rest of the garment. The knitter is expected to understand how increases and decreases in stitch counts affect the overall shape of the vest. The chart patterns of the jumper are rendered as a formula rather than boxers on a grid. In add, the amount of information available in the representation of the vest is dramatically reduced and more knowledge of getting around is shifted to the knitter. This pattern is far more map-like and requires more background knowledge and experience to orient oneself and reach one's goal. This is not necessarily more liberating than the tour of the jumper. It is in even more constraining, more conventional. The decoding of the path through the garment particularly of complex patterns, like Hargreaves brocade in the Jumper, would be so complex as to confound all but the most experienced Japanese knitter. Baking pastry is at least as old as knitting wool. Pastry connects us to the agricultural revolution and the year round availability of milk, eggs, flour and sweet and savory flavorings. We all enjoy eating these confections, but few of us take the time to learn to make them. Modern food systems take the effort out of these preparations. Just as we no longer shear, wash, card, and spin wool, we also no longer have to churn butter, grind and sieve flour, feed chickens and collect honey. All of these ingredients are now staples in any pantry easily purchased in any market.


I want to focus on one form of the pastry arts. Called in American, layer cake, in British, pudding, in French entremet, and in German, Dessert or Nachspeise, this confection involves layering different textures and flavors in such a way that the consumer enjoys a concert of tastes. In the hands of the home cook, this will be a flavored pastry cream spooned onto a baked crust of dough of some sort, and topped with whipped cream, fruit or sweetened sauce. In the hands of a pastry chef, the entremet is the concerto of the pastry arts, often comprising five or more distinct layers uncommonly combined flavors. The dish is a territory that becomes explicit as layers are created and assembled. The journey ends with the confection ready to be sliced and served.


An example of a high-end entremet is one developed by Juliette Lelchuk, a chef-instructor at the San Francisco Baking Institute. This is a spiced milk chocolate flavor meringue layered with orange infused pastry cream and topped with blood orange jelly.


The map of this territory takes the form of a set of recipes, one for each of the layers. The recipes include the form and weight of the ingredient and the method for combining them. As with the Japanese knitting pattern, the map makes assumptions that the user knows what the various methods entail. One cannot whip cream 'to a soft peak' without some prior experience in the stages of whipping cream. Thus, the recipe is a map because it uses words as icons, as parts that represent an implicit whole of ingredients and actions. Navigating the map requires having been to similar 'places' before and therefore, knowing what do when one gets there.


Once the map has been read and the layers are ready, assembly can begin. Here, the chef leaves the word-based map behind and employs a sequence of pictures of her actions as the entremet takes shape.

Narration of the photos:

• Spread the praline milk chocolate ganache evenly over the meringue.


• Sprinkle the chopped hazelnuts over the spread ganache.


• Fold the Italian meringue into the orange pastry cream mixture in three additions.


• Pour the pastry cream over the spread ganache and nuts


• Smooth the orange pastry cream flush with the frame.


• Spread the jelly mixture and arrange the orange segments so they spread over the entire surface.


• Garnish the entremet with a bit of gold leaf and triangular dark chocolate bands to enhance the shine and add height.

The effect of the photos is similar to that of the tour. The chef, in selecting the shots to be photographed and published, is demonstrating the professional techniques required to construct the entremet. The resulting confection is out of proportion with the needs of a home cook. The amount produced in the photos yields 20 portions. This would feed a large party. But the proportions are not the only constraint in the tour.


Very few home cooks would invest in a set of matching frame extenders in multiple heights (notice the black corners in this photo and the yellow corners in this one) that the chef is using to construct the entremet. The frame is rectangular, a shape that lends itself to commercial baking but is not commonly found in home kitchens. The photos show most of the layers as finished. Only the lightening of the pastry cream is demonstrated. In fact, the photos represent the least difficult parts of constructing the dessert, assuming you have the right equipment. The more difficult parts, making a tender dacquoise (first layer), getting the ganache at the right consistency, and making a blood orange gelée that will jell and not run are all techniques that require considerable practice. That practice cannot be photographed. The photographs constrain anyone who wishes to make this dessert by only revealing how an experienced chef in well equipped kitchen would make it. She shows us only her version of the dessert, not a version we ourselves would make.

Here we discover that it is not the time-series itself that constrains the choices of the cook, but the selection of practices and tools that are photographed. That is where the implicit messaging lies.

The Agency of Sets

Maps, tours, recipes and patterns are not unique objects. Instead, each must be seen as comprised of a set of places, stitches, or ingredients. Therein lies a further complication to understanding their agency: the map and the tour are political. They are produced by one person to influence the perceptions or actions of others. To see this political potential, Deleuze and Guattari (1987) suggest that we use the French word 'assemblage,' in place of he word 'set'. Assemblage is a way of ordering and arranging knowledge that can occupy two different states simultaneous: it can be a whole or it can be the parts of the whole. The thinking here is analogous to the way we understand bodies. They, too, can be understood as wholes, or as parts. When understood as wholes, the body occupies one kind of space and time. When understood as parts, the body occupies a different kind of space and time. The whole is animate. It moves, and in the process of doing so occupies or constitutes space. The parts are inanimate. The threshold that permits the parts to function as a body is their cooperation. More importantly, what distinguishes one body from another is the specific way the parts work together to be or act in the world.

This approach is less abstract than it appears. Assemblage is valuable as a tool of analysis precisely because it reflects real experience. I wish to take advantage of this with an assemblage that exist in every one of our lives and show how this tool works. I want to offer you a menu for a dinner party. Let’s say that I want to make Chef Lelchuk’s entremet for my guests. How do I go about creating the assemblage of courses that will highlight the dessert appropriately. Chef Lelchuk has created a dessert that self consciously references Spain: She named it La Naranjita, the little orange. She combined quintessential Mediterranean flavors like blood oranges and hazelnuts with references Spain’s colonial past with chocolate, cloves, nutmeg and cardamon. To highlight this dessert, I want my guests to experience more of the flavors of Spain. To do otherwise, for example, to have the entremet conclude a Thai meal, or a South Indian meal would do violence to the flavor combination. It would render it meaningless. To preserve its meaning, I want to preserve its context. For the creator of the confection, the context is Spanish.


I might start with macona almonds, cubes of manchego cheese with slivers of guava paste, cured olives and sparkling cavé wine as the guests arrive.


At the table, they will enjoy a chilled tomato and apricot gazpacho and a fino sherry.


A large dish of paella arrives redolent with sausages, chicken and shellfish. It’s aroma of garlic and smoked pimentón fills the air. It is served with a manzanilla sherry.


Finally, the chocolate and blood orange entremet is served with a cream sherry.

Assemblage is intended to used as an analytical tool to help us dig beneath the surface of any practice and discover the different processes working in concert to give meaning and shape to our actions as producers of maps and tours. The following observations belong to my practices and choices as the designer of the menu, rather than the experience of my guests.


First, this collection of recipes is an ad hoc grouping. I could invent several menus that accomplish as ‘Spanish’ meal without repeating any of the items here. The origins of this collection are real. Each ingredient, each method of preparation, each wine is an ordinary food or meal of thousands of Spaniards every day. Their rice, olive oil, wines, spices and fish are locally produced. They have many choices of what to eat. Social relationships around the sources of these foods often determine which flavors they taste. Since I am not a Spaniard, I learned these recipes under specific circumstances in the United States, using my local ingredients and equipment, and at a specific time in my life where I had mastered some cooking skills, but not others. If my food tasted like home cooking to a Spaniard, it was a accident. The rule underlying the collection, if it can be said to even be a rule-based collection, is one of contingency: there was the opportunity, the means, and the desire. However, there was none of the necessity a Spaniard experiences every day. Yet the origin of the menu in no way undermines or weakens the efficacy of it for constructing a Spanish eating experience for non-Spaniards. For my guests, what I served to them was a Spanish meal.


Second, there are several ways in which this collection can be said to be coherent. The number and variety of recipes ‘as a whole’ generate possibilities for action that go beyond any one of the recipes. It is the repetition of the Spanish theme across several different foods assumed to have Spanish roots that builds this potential. The collection permits me to choose this or that recipe (gazpacho can be red, white, or green, and involve many different combinations of ingredients), or to mix recipes with different traditions.


In Spain, however, these dishes are not understood as Spanish, the way a schnitzel a Austria, bangers and mash is British or the hamburger is American. Instead, they are understood as regional dishes. Manchego is, of course from La Mancha. Paella is from the east coast, from Valencia primarily and Cataluña. Gazpacho is an ancient and widely distributed soup based on stale bread, garlic and olive oil, but in Spain, it is identified as a southern dish, primarily from Córdoba and Seville. Because of the strength of the regional identities of these dishes, they would not ordinarily be served at the same meal. To do so would violate the logic of the set. Even the particular combination I chose for the paella would rarely, if ever, be served in Spain. They would prefer to focus on how a particular ingredient, the chicken or the sausage or the seafood contributed to the overall flavor of the rice. I violate the logic of the set even further by adding what from a Spaniard’s point of view are random recipes. For example, the entremet is ‘Spanish’ because Juliette Lelchuk says it is, not because it was ever eaten in Spain. Deleuze and Guattari would identify this menu tactics as countercultural since they exceed and confound the coherence of the menu as it is ordinarily practiced in Spain.


Third, each of parts of the collection maintains it’s specific history, which influences how I choose with which recipes to work. My experience with Spanish restaurants in the U.S., the representations of Spanish food in mass media, and my assumption that my guests would have had similar experiences were the primary factors influencing my choice of menu. In spite to my efforts, each course has it’s own trajectory or movement through the web of the food experiences, its own way of interacting or crossing the trajectories of the other courses. These trajectories shape which courses will translate as Spanish and which will not. Gazpacho in particular can be confused with other cold vegetable soups originating in Latin America. Paella can, and often is prepared in a way that resembles North American or Caribbean rice dishes like burgoo or arroz con pollo. Deleuze and Guattari would say that because of this ambiguity, the power (of affect) is not equally distributed across the assemblage. This means that the persuasiveness of the 'Spanish' in each dish is not consistent. There is a middle ground of cosmopolitan knowledge in which a guest would be willing to make the journey to Spain through these foods, while those who know too little, or those who know to much, would not.


Fourth, there is no recipe that completely dominates the menu, in spite of my efforts to set up menu to highlight the entremet. No one course has sufficient competence to fully determine the consequences of the activities of the assemblage. When I consider how I have worked with these recipes in the past, I recollect that each one has been served to other guests. I also recollect that at one time I served different paellas in four or five dinner parties in the same season. I was working my way through a specialty cookbook on paellas. These are the ‘generalists,’ so to speak. The starters and the gazpacho have also served many different guests and were combined with very main courses. I stopped cooking these dishes ten years ago, choosing to explore other traditions. The only reason, I come back to them for this example is to demonstrate the agency of sets.


Finally, a particular set of recipes is a captured flow of objects for which organizational forms outside my household are implicated. The organizational forms implicated in this collection are present in the ways this menu marks what Deleuze and Guattari call instances where breaks or divisions in otherwise free-flowing phenomena are evident. For my recipes, these breaks and divisions are represented through my act of purchasing the ingredients, some of which were actually produced in Spain, and dividing them from the much larger set of ingredient supplies that existed at the industrial firms who produced them for my purchase, from the retail store at its specific location and from its collection of merchandise, and in placing them in my pantry in my house. The organizational forms that permitted me to capture these objects also include larger organizational forms, such as the global market in food products, the lack of barriers to export and import, and the legitimacy of private property. That, too, is a part of assemblage: the food I serve my guests belongs to me. Before I purchased these ingredients, they were in flux. They were produced for purchase, but were not yet purchased. They traveled to various places where they could be compared to similar products and bought. Such flows always exist prior to any particular assemblage coming into existence. The flows are terminated in time and space when the sets are placed in the assemblage. That is, the moment when we can finally say that a object has been consumed.


To sum up, Deleuze and Guattari call assemblages ‘desiring machines'. They move us to see ourselves and our relations to others in a desirable way. Solomon and Assael observe that we put objects into assemblages because the unified whole carries role information. That is, for those few hours, my guests chose to believe that they could have been in Spain.


It was a theater, to be sure, but it was also a shared fiction that was made possible because the menu mapped the territory of Spain in their imaginations and provided an fanciful tour for their taste buds. To play the role successfully, the person cannot consume only one item, but rather must acquire the entire complement. This way of understanding assemblage as role activation lies at the core of the agency of sets. The knitting pattern collects stitches into a garment, imagined through the accompanying photograph, that when knitted and worn, imbues the person with a particular ‘look’ and all that the look implies. The recipes collect butter, eggs, flour, and flavorings into forms that, when assembled and eaten, produce a delight for the eye and the mouth that marks the cook with a particular level of competence shared by very few other people.


We began to explore maps and tours as dialogues about a territory, but with different intentions to direct, modify or augment an experience, by observing that all material objects, like the dinner plate, can contain implicit messages that can direct, modify or augment the actions of the people that use them. The objects enter a dialogue with the people, instructing them of the actions that need to be taken.


We then looked at patterns for knitting wool as examples of how this dialogue can unfold in real time, creating a territory that will ultimately transform itself into a garment. We noted that the territory is accessible through either the map, which offers greater options for action, or the tour, which is considerably more determined and channeled.


Turning to Japan, we noted that a hybrid map-tour tended to reinforce the constraining features of tours, while limiting the liberating qualities of maps. Both the knitting pattern and the pastry recipe require a knowledgeable reader, familiar with the landmarks of ingredient quality, craft practices, and opportunities for 'side trips and excursions'.


Looking at the series of photographs of the chef assembling the pastry brought us closer to the ordinary experience of time-based media. These photographs show that is not the time series itself that constrains the choices of the cook, but the selection of practices and tools that are photographed.



Where do we go from here? What is the use of thinking about maps in this way?

After reading his paper Robert suggested that we might be wondering why he would talk about maps and tools in this way, and he thinks that the larger message he has for us is that all design is socially embedded, and that it's all about a conversation that the designer is having with the audience. The map or the notation, the figure we create to document our design, is a part of that conversation. This issue with the material agency is about the conversation occurring at two different levels: the explicit level which is about the design itself, and the implicit level which is all these extraneous features and implied message going on because of the way we've constructed the notation of the map. It says as much about who we are as designers and who our audience are as consumers as it does about the subject of our design itself.

We then moved into Q&A. An audience member went back to the idea which Tim had previously presented about a recipe for an engineer. They asked that if they decided to cook the dessert in a more 'mixed' dinner would the reading then be different or would you use the same system? Robert responded by saying that any assemblage you put together must out of necessity be an ad-hoc assemble, all the elements will have a randomness to them and no matter how much you try to be consistent, the fact that you have to assemble them is itself an act of randomness. The only alternative would be to cook unconsciously. Any attempted desire, whether it's in theatre, dance or multimedia involves selection. It's this selection which creates the assemblage and an element of this must be random, this is according to Deleuze and Guattari. Simone asked how it would work with a food fight. Robert responded that by interpreting this as a performance would be one thing and that the notation of the food fight is where the assemblage comes in. It's only when we try to document who threw what at whom, that we run into the kinds of issues that he wrote about in his paper; he's talking about the messaging around the design.


Someone in the audience came back to the part of the paper on which Robert discussed how people describe their apartments and asked how does the distinction in map and tour correspond to this concept of affordance, which is embedded into a certain medium? Robert asked if other disciplines used affordances? The previous speaker made the point that if you give people a piece of paper they will more likely draw a map instead of a tour, and about how Robert spoke a lot of the social determination of the narrative which is embedded into the material itself. Robert answered by telling of how he has discussions with one of his colleagues, an anthropologist who teaches in an architecture school, and how they discuss this issue of where do these affordances fit into this pattern, but they haven't reached a resolution yet. He thinks that if you view specifically people's use of spaces, and the person who is narrating their experience of the space as embedded in that space that narrative is going to generate affordances. If they're trying to do it in an abstract response to spaces in general, maybe a park they've visited or a new public building, you're still going to get the map vs. the tour destination but you're not going to be loaded down with the evaluations which residents make about their personal space. Another way to talk about affordances is how space satisfies life required of the people who occupy it, and this gets bound up with how they narrate it. With the tour, it dominates 97% of all of these responses, it's about focusing the visitor's attention to one thing at a time and it seems to duplicate narrations and affordances quite explicitly.

Robert explained about how with this discussion, there is still a lot to explore for him, as there are things which he doesn't even understand yet, since he's been working on this problem for only a few years. It is connected with a larger area of research on which he's focusing on exploring, the social embeddedness of design activities in general across all the design disciplines.

Marc Downie brought in a point about how when Robert produced a map on screen it was to embody control or transfer problems, and that he talked at the end about design and selection. One of the things which hadn't been mentioned, but which is important to what a map means in Marc's practice is that he has a course to map making to name and locate landmarks that he's bumped into in order to find his way back to them. This isn't something which maps easily onto Robert's cooking, because he's good at cooking. He doesn't think that cooking is a formal system that he cannot think through in advance and simply implement and explore. Robert responded by saying that he is always making a destination between talking, and designers which are concerned with craft and those which are concerned with art. Marc just described the destination, as some designers enjoy replicating patterns again and again and getting the same result. It's about perfection and techniques, which is a basic for everything we do; we need to be craft people before we can do anything else (who said this? A contentious statement at least). However, to use these techniques to explore unknown territory, for example the way a fabric artist designs a sweater, you just start doing something and in their case the yarn will lead them how to do it. The product of this is very intimate. The same is with the pastry chef.

Tim suggested that this can be looked at as a map where you've been and a map which shows the empty spaces, the white surfaces which used to be there on maps of the world which hadn't been explored. Robert suggested that the oceans have been un-discovered. One area which is different to those discussed is the area of mathematical research, where you go and you try something out that you then and try to summarise in order to present what you've done so that you don't need to do those calculations again. This is a type of mapping.

Elisabeth Schimana: Digital Nirvana


Elisabeth Schimana is a composer, performer and radio artist who through her work explores ideas surrounding space, the body and electronics. Her presentation for Data Ecologies explored the idea of what we leave for others with regards to code and addressed this scenario within electronic music as often there is a lack of notation within this field, due to the composer and performer being the same person. Elisabeth presented to us a paper around this topic:

5 0.jpg

Digital Nirvana

What code do we leave for the others? Files? Patches? Description? In the long tradition of musical culture a powerful tradition is interpretation. For interpretation one media has to be translated into another. But very rare this process is going on within the electronic or digital music community. Most of the time the performer is the composer and only she or he knows. So one could span a space from digital nirvana to grave symbols in stone. I’m gliding in that space.



The concept of moksa, presupposes an existence of infinite eternal souls, who alone are doers (Handelnde), enjoyers and responsible for their actions. Thus, all souls are entangled (verwoben) in the mundane (alltäglich) worldly activities, bound to karmas since beginningless time and transmigrating and reincarnating from one existence to another. According to Jainism, all souls can bring an end to this repeated cycle of births and deaths and attain liberation, that is moksa.

• Where there is neither pain nor pleasure, neither suffering nor obstacle, neither birth nor death, there is emancipation.(617) • Where there are neither sense organs, nor surprise, nor sleep, nor thirst, nor hunger, there is emancipation.(618) • Where there is neither Karma, nor quasi-Karma nor the worry, nor any type of thinking......there is Nirvâna. (619)

There is a safe place in view of all, but difficult of approach, where there is no old age nor death, no pain nor disease. It is what is called 'Nirvâna', or freedom from pain, or perfection, which is in view of all; it is the safe, happy, and quiet place which the great sages (Weisen) reach. That is the eternal place, in view of all, but difficult of approach. Those sages who reach it are free from sorrows, they have put an end to the stream of existence. (81-4).

Digital Nirvana

Is the place of all stored data. There is neither pain nor pleasure, neither suffering nor obstacle, neither birth nor death, neither sense organs, nor surprise, nor sleep, nor thirst, nor hunger, neither Karma, nor quasi-Karma nor the worry, nor any type of thinking. No meaning. There is a safe place in view of all, but difficult of approach. The digital grave of a disappearing culture. We are out of cultural reincarnation. Bane or mercy?


The word 'tradition' itself derives from the Latin 'tradere' or 'traderer' literally meaning to transmit, to hand over, to give for safekeeping.

It is presumed that at least two transmissions over three generations are required for a practice, belief or object to be seen as traditional.[8]

Tradition changes slowly, with changes from one generation to the next, not being seen as significant.[13] Thus, those carrying out the traditions will not be consciously aware of the change, and even if a tradition undergoes major changes over many generations, it will be seen as unchanged.

So the main question is: do I want tradere literally meaning to transmit, to hand over, to give for safekeeping? Or is it enough to enjoy this one moment in time and space of a performance, an action, a common emotional space, an interaction? Assuming that you all store your data, the data will remain, but who can decode it, give the data meaning, I would say even in the nearest future?

So if I have the feeling that I want to leave something I need to think about data translation and to find a language not depending on any particular soft or hardware.


Notation in general is defined as written communication. A translation from one media into another - from sound to written symbols - to sound.

Music notation or musical notation is any system that represents aurally perceived music, through the use of written symbols. The earliest form of musical notation can be found in a cuneiform tablet that was created at Nippur, Iraq in about 2000 B.C. The tablet represents fragmentary instructions for performing music, that the music was composed in harmonies of thirds, and that it was written using a diatonic scale.[1]

Cuneiform 5 1.jpg

Because of conventional musical notation limitations, many present-day composers in various genres prefer to compose music that is either not notated, or notated only through the computer language of digital recording.

Now I switch right away from music to sound and into the field of digital or electronic music. There notation also includes the recorded sound piece.

There are two main reasons for notation, of sound structures in that genre: 1. For analysis and graphical expression of sound structures 2. For interpretation

For analysis and graphical expression of sound structures

One example is Spectromorphology developed by Denis Smalley in 1986, 'Spectromorphology and Structuring Processes'. He refers to the ideas of Pierre Schaeffer, who developed a Typo and Morphology to describe categories of basic characteristics of sound. It is a method for acousmatic music.

Smalley distinguishes 4 steps in spectromorphology analyses:
1. Spectral typology - from tone to noise
2. Spectromorphologycal archetypes - a profile of sound
3. Motion and growth processes
4. Structural connections - the connections between different sounds

Elisabeth told us about what Shaeffer did; in his book Traité des objets musicaux in 1966, he wrote about the topic of layers of perception influenced by Henry Bergson, a French philosopher of the early 20th century. We can see the similarities of Smalley and Shaeffer:

- Perception first order - profil général - means the envelope of the sound
- Perception second order - allure - means the inner structure of the sound (types of vibratos)
- Perception third order - grain - means the tactile sensibility

Schaeffer's work at INA/GRM (Institut National Audiovisuel, Groupe de Recherches Musicales) in Paris, beginning in the late 1940s, culminated in the publication of Traité des objets musicaux. For Schaeffer it never was important to create a written score, it was just the opposite. He believed in the recording as the perfect notation of acousmatic music. This even was a condition of his concept of 'reductive listening', which means to disrobe the perception of sound from everything which is not the sound itself. So the music has to be stored on a media to make repeated listening possible.

Spectromorphology was developed further for notation by Manuela Blackburn.

Manuela Blackburn

Since its conception, Denis Smalley's spectromorphology has equipped listeners and practitioners of electroacoustic music with appropriate and relevant vocabulary to describe the sound-shapes, sensations and evocations associated with experiences of acousmatic sound. This liberation has facilitated and permitted much-needed discussion about sound events, structures and other significant sonic detail. More than 20 years on, it is safe to assume that within the electroacoustic music community there is an agreed and collective understanding of spectromorphological vocabulary and its descriptive application.

Spectromorphology's influence has been far reaching, inciting approaches to electroacoustic music analysis (Thoresen 2007), notation (Patton 2007), composition and education through its flexible functionality and accessible pool of vocabulary.

Elisabeth showed us images from The Visual Sound-Shapes of Spectromorphology : an illustrative guide to composition by Manuella Blackburn which shows spectomorphological notation:

5 2.jpg

There are seven different sound types. Elisabeth said that maybe this is the type of notation which you would find after; you would play the piece and then find a graphic representation of what is going on in the piece.

5 3.jpg

The image above shows the scores for the instruments and then below there is a graphical interpretation. Elisabeth questions this as you don't really know what it going on with it. The graphic is a representation rather than interpretation, which is different.

5 4.jpg

The image above is another example. There is a spectrogram which shows the frequencies within the spectrum of the sound.

For interpretation

Interpretation in music is a very powerful tool. It’s creating. It’s redefining a given structure in endless variations, changing over time and space, depending on actual aestethics and performance practices.

So how to leave a structure in digital music for interpretation? Is it the patch, the script...? I guess no. Software we experience nowadays will disappear, systems, file formats too. They all go to digital nirvana. Technical development is fast. So I started to think about these facts.

Till now there are 3 quite different approaches.

Elisabeth told us about a project she worked on with Seppo Gründler from 2001 to 2006 called The Great Score [9].

The basic idea for the composition and its performances is rooted in the historical context of electronic music. The concerts follow a precisely defined structure - the score. However the performing artists Seppo Gründler and Elisabeth Schimana don’t proceed according to a score in the traditional sense based on notes, but due to a temporal and functional structure, which was left to other artists for interpretation in the eighth concert.

They played seven different concerts in seven different cities and in each of the cities where they performed, they asked other artists for interpretation for the eighth concert. So what they left was a base structure, which means that the performers do material creation in the first part, freezing (creating loops) in the second part and regulation in the third part. In parallel the computer would do acquisition, which is where it would record parts, during freezing the computer would analyse the recorded parts and in the third part it would play the synthesis of the analysis. The composer is essentially just sitting there regulating the sound but the whole process was going on during this one performance. There was also a visual score:

5 5.jpg

Above was the visual score for part 1a. It was not like this, but rather drawn during the 28 minutes of the performance. This would give them the sign that the machine was recording.

5 6.jpg

Above was the visual score for part 1b

5 7.jpg

Above was the visual score for part 2

5 8.jpg

Above was the visual score for part 3

The visual representations came one after the other. This was from the performance in Vienna.

In the eighth performance they had different artists in different locations and they performed a night concert for a live radio show:

5 9.jpg

Elisabeth played us Synthesis:

File:01 grp net.mp3

The second approach was to leave musical score for flute and live electronic and to try and find a language for live electronic musician interpretation. She had never found people before who wanted to interpret as people are normally so focused on their own stuff. Elisabeth wondered if this would be enough information to do an interpretation, no matter what technology you used. No matter what it was, it shouldn't have made a difference. This was the score:

5 10.jpg

And this the recording:

File:On tesar part1.mp3

She explained a bit about the score (above). There is a hidden layer of phrases by Heinz Tesar, who is an architect, and then in shows what the flute and electronics are doing, and how the sound has to be in space and which symbols mean what. This is a very classical notation of a piece.

Virus #1 - The media is the score from sound to sound

Elisabeth then spoke to us about a series of works which she is focusing on at the moment, called Virus [10] where the medium is the score from sound to sound. It's a composition for live generated electronic resonating body and in the first place it was a string quintet, and then after this for other instruments including a piano and two drum sets. She generates the score and the musicians need to play what they hear in a moment. It's about a reaction in a moment and it's amazing to see how fast people are when relating to sound. It's a very deep concentration process as you need to follow what is happening each second. This means that there is no time to think.

Elisabeth concluded - So I end up with the paradox: no translation from one media into another, the media itself, in this case the live generated electronic sound body is the score for interpretation in the very moment of NOW.


We moved onto the Q&A around Elisabeth's work. Someone asked if Elisabeth thought there would ever be a notation for recurring music. She thought that perhaps yes, if it's needed and wanted. If we want to create a language which we can leave then yes. She would like to do it because what she finds so nice in music, is that interpretation is creation. In years to come, people can do it in a different way. The questioner was thinking that if you could formulate it, then it can help you with what you are planning to do. It also gives you restraint which can be useful, you need notation for this. Elisabeth confirmed that he was referring to the idea of writing the concept, so that you have more control of how you're going to realise it but she believes that this is different to the idea of leaving something. He thinks it's the same, as we don't even have the language to describe sound.

5 12.jpg

An attendee of the seminar made the point that music is always changing, how it is perceived in 20 years time is different. You wouldn't be able to write down the simplest produced music in all its content because you have to write down so many parameters. Elisabeth believes you always need a cultural reference, as the design is embedded in it. Any kind of musical score is embedded into cultural practice. You need the notation and the cultural background otherwise the notation won't make sense. Elisabeth said how she can't read the stuff she made 30 years ago, everything changes.

Someone made the point that in ten years time, you will be able to do data mining on music. Tim chipped into this thread, asking, can you play a pop song you've heard on a musical instrumental? You can still play a version of it on a different instrument, but it's all down to interpretation. Someone made a point about describing sounds - for example we don't know how to describe a piano sound, but we don't need to as we know what it is from experience. If you start changing the waveforms though then we're lost. We don't have an abstraction for it. Elisabeth said that you could just record it, but you always need to be careful when copying it to a new format, everything is continuously about copying to new formats which is going very fast. She thinks that somehow this digital culture will disappear.

Someone asked whether Elisabeth thinks we can reduce music to two systems: 1) notation (the algorithm) and 2) the sound, and that the algorithm triggers the sound? Elisabeth thinks it's about how you describe the quality of sound itself. The man suggested, what is the influence of the sound on the time structure? Elisabeth said that sound is time, and so this something which always gets discussed.

We moved onto the idea of how the file becomes its own representation, so the notation becomes an exercise of analysing or describing the sound. Elisabeth responded by saying that this comes back to the approach of the recording being the score, but as soon as you start to work with acoustic instruments it is a different situation. The culture of recording is still going on, but also so is the culture of producing something in the moment.

A question was asked about the interface of a notation: what is the interface to the machine? It's easy to record the process because it's software. Someone made the point, is a guitar the notation? It's a simple instrument which is why you don't need complex notation. In certain instances the instrument is the notation. Nik Gaffney from FoAM made the point that there are two ways to look at notation, the first is the notation of the sound produced by the instrument and the second is the notation of how to construct things from the instrument. With music you're playing as much with the instrument themselves, or the system for constructing them as you are playing the sounds produced. The reason it's very to interpret these spectrum notations is that you have to design the instrument, and there is no instructions that tell you how to design an instrument which makes specific spectrum patterns.

Marc asked, what as a composer do you want? Say in 100 years from now someone finds something written down and plays something which is yours/good/theirs/useful to them - he wanted to know what would be satisfying to her. He thinks this puts some limits on what she wants to specify. Yosi stepped in and referred back to what Elisabeth had said about wanting to leave something for someone to interpret. Elisabeth answered with discussing about how she leaves a structure or a description of structure which is able to integrate with totally different tools in 20 or 40 years. These tools could design whatever, or disappear, but she thinks it is possible to leave a structure and if she wants to do this she has to think about how to leave something. She's sure in even 100 years people will understand frequency and other musical aspects, they will be part of our cultural heritage. So it's about how to find a language around this. She gave an example of Bach's work, and how that was brought back but it is probably played very differently to how he would have composed it. However, this is something which she finds to be very beautiful, that this interpretation is connected to the aesthetics and culture of a certain time. Marc suggested that it doesn't need to sound like your music it just needs to be useful to them. Elisabeth thinks this gives you some kind of input and inspiration.

Tim referenced a band that interprets a lot of electronic music for classical instruments, and his question was that is everyone who makes this electronic music picking up a punk sensibility and is it all about doing it yourself? People want to do their own thing; has this affected contemporary music? Elisabeth said that with regards to classical music we have education, but there is no education for electronic musicians and composers and so the questions is, will it come in the future? Once we have this we'll have people who have found how to do interpretation, which she thinks is coming. Tim referred to a man who within his own practice froze his hardware, he didn't change it after a certain point. He therefore got very good at using it. This is the same with software, there is no time to become an expert as it's constantly changing. Tim's last question was about the image which Elisabeth had showed us that showed the assemblage of different things that have classical musical notation, the physical placement of sounds and spectrographic texts. She told us of how it's difficult to read, there is a booklet with a score in it which she offered to show it . She said it would be interesting to see if people who work with sound can read it. We went back to re-interpretation and Tim spoke about how they want to organise an evening of re-interpretation performances.

Simone left us with a thought that if she invented a kind of representation which wasn't a graphical one, something that would be something like a recording or text or somebody speaking, the question is could this be done? Music is very linked to graphical interpretation.

Day 2: 16th June 2012

Toxic Dreams: Mise en scène


Toxic Dreams was founded in 1997 by Kornelia Kilga (producer) and Yosi Wanunu (director) & Michael Strohmann, Irene Coticchio, Anna Mendelssohn are all associate members. The group puts on shows which deal with the not knowing and that busy themselves with the slippery nature of reality. They feel that the best way to view their shows is for the audience to accept confusion as part of the experience of sitting in the theatre and they expect the audience to make up their own mind.

6 0.jpg

Mise en scène

Toxic Dreams's presentation for Data Ecologies was called Mise en scène, regarding the reading and staging of political acts and interpretation as an exercise of power. Toxic Dreams visualises the play of meaning across a field of knowledge over a number of productions, and the process that grants certain aggressive voices the status of cultural history and excludes the voices of the disenfranchised.

Yosi began by telling us a bit about the group. They are a mixed, not traditional performance group in which the members also come from media, visual art and music backgrounds and the work they produce spans installations and traditional theatre shows. Their next piece is called The Church of Warren Buffett [11], which is based on an article and will be performed like a sermon. After this they will be working on part two of a long-term project about the assassination of JFK and in-between they are working on a project with the Wien Museum, which will be a one night exhibition of presenting parts from their archives. The items will be presented in a performative way, showing how exhibitions can be presented differently.

Toxic Dreams started from a very specific place; they are not designing a show without the theatre in mind. When they work in a place, everything around it is part of the concept. From the beginning on there is the theatre room, and then there is the one they create; it's like a room within a room. They always start with a very specific setting, which is normally a public setting. Yosi doesn't like the idea of intimate settings in theatre, like a set being made to look like a living room, for him theatre is about a public meeting where people come to discuss with each other. Where the performance is therefore always in public spaces, like a hotel lobby or a train station. They make it clear to the audience that it is a public meeting, and that they are there to talk. A text at the beginning of the performance declares a bit about the space as well as the audience's role. It sets up the relationship which they are going to have.

Entrance and exit are the most important things. The audience come in and they leave the light on. In their performance about JFK, when the audience come in, someone is sat at a big table reading a newspaper, which is like a committee meeting. Here, they are there to investigate the death of JFK. On the video behind them the screen shows pictures of them reading the newspaper on a tram in Vienna as well as pictures from the time of the assassination, of people reading about the incident in the newspaper in New York in 1963. The audience can track down the references to the image which they see as they enter the theatre. Yosi doesn't like the idea of going into a theatre, the light goes down and the performance starts. The problem he has with intimate places is that it allows the audience to take themselves out and feel security. He compared it to something he read in an article about business people asking someone to design their bedroom like a hotel room they might stay in, it's like a false security.

In the space, the performers and the audience take their places with no particular order. There is no rigid setup, people sit where they want and they start. He's interested how people arrange themselves in the space without any particular mise en scène or blocking. People tend to organise themselves in the space in places which are warm to them. You will have the performers themselves, who feel comfortable being close to the mic, or people who have never been to this type of thing before who take a chair to the side. The people start to inhibit the space in a way you just can't imagine if proper blocking was part of it. This has become part of the process with everyone mapping their own territory and sitting where is comfortable for them. There is a natural aspect to it, people can just get up and get a drink if they want to, Toxic Dreams rehearses for this. The movement around the territory is organic. The performers know better about the space than Yosi, as they inhibit it. He therefore let's them go by how they feel. Since they start in a very clear room at the beginning people know where they are.

Yosi then told us a bit about how they work with the text for the performance. They have to work it out by seeing that it fits and there is a lot of trial and error. Yosi is a fan of Gertrude Stein, who wrote about theatre that the problem was that you were in front of it or before it, but never in the moment. For Toxic Dreams, there is this idea of beginning again, which she called 'continual presence' and means that they get bored very fast and then restart. Yosi realised that something was missing as sometimes they didn't know how to continue. He therefore came up with something called 'office party', which is when every time someone is stuck, they'll go into a dance. It's a senseless release of energy into the room, and then they continue. The problem was that office party became the main thing, with the text becoming the second thing which made them re-think it. He thinks the audience got a bit confused about why they were doing the dances between serious texts. The other thing, with regards to the text, is that in theatre it is important to the interpretation of the play. In that respect, Yosi thinks he is disrespectful to the big writer. They'll never do a play fully from start to end. If it doesn't fit, they'll change things or re-write it. They work in cycles, defining an area of interest (they're currently in the political cycle), and then for the next four years they will work with this area. All their performances are geared to this question. They look at ideas like how can you mix academic text with speeches, etc.

They worked on something called 'the theatre cycle', where they asked themselves what to do, as in theatre nowadays it's not cool to be a theatre artist as it's seen as old fashioned. One piece they recently worked on was Uncle Vanya by Anton Chekhov which they redid over and over for years. One of the questions they asked was: What is realism? And they went to Russia to film the play in the place it was written, in Chekhov's place. They wanted to see what it was like to perform it in a real place. They then extracted the text to the bone and wrote an opera called Pink Vanya [12], where they reduced it to the basic things. It was called Pink Vanya because they wanted to change it into lovely pink music. Many people in theatre will take a text and spend lots of time building a certain act, and then sit in the rehearsal trying to find it, making it very predictable. However, with this approach, which is influenced by the Wooster Group it's about task orientated acting. You take a text, and you start to chew it; the more familiar you get with it the more you chew it. The same text can be different on each night. The problem with this is, that it's very risky because an audience might love it one night, but the next night they might hate it. This makes the performers insecure.

Yosi then passed over to Michael, who discussed an opera piece which Toxic Dreams worked on last year, Collapsonomics [13], set as a talk show between Friedrich August Hayek and John Maynard Keynes. The actress playing the host, couldn't sing so they had to figure out a way for the music to work with her talking. Michael used the invisible hand orchestra which is a computer controlled piano. He came across a piece by Peter Ablingers called Deus Cantando [14] within the Ars Electronica Centre which is a speech recording, making the pianos talk. Michael thought that if the actress and the piano could talk together it would be good. He had to take the spoken text from Anna, the actress, and then transform the analogue audio into midi data so that piano could understand it, and then played this back with the actress being in sync with what she said before.

6 0.1.jpg

He used some software called Spear which does a free transformation of the audio stream into a different format.

Michael played some of Anna's dialogue for us, followed by what it sounded like as a piano. If you knew what she was saying then you could understand it by the piano. Michael had to tweak it a bit to sound more like the other parts of the music, pitching the notes a bit, to sound more harmonic. He showed us a video of the results:

It was difficult for the performers to be precise with the time frame. The show was written in stone, in a timeline which they had to be restrained to. This process of devising the map out of the territory, where the territory would be the audio file and reducing it to the midi data, Michael also used this to feed rhythmic stuff into it. This helped him with ideas he wouldn't have come up with beforehand. He showed us the notation, which he thought no-one would really come up with:

6 1.jpg

Or like this, which he said is a bit easier. It's classical notation but he doesn't think it helps anybody:

6 2.jpg


We then moved into a Q&A about Toxic Dream's presentation. Elisabeth asked about why Michael thought nobody would come up with the type of notation that he showed us. Michael said that it's complex to write and impossible to read. She said that there are people doing this, writing things which nobody can read or is able to perform, but it comes from the other way, it is their intention to write it this complex. It will initiate a process in the musician to create something. Michael responded by saying that the output is normally beyond not playable. Elisabeth asked what the map and the territory was for him. He said that the territory is the speech said by the performer, and the map was the transformation into midi. For the two opera singers he would compose separately.

6 2.1.jpg

Tim asked about conductors, as Michael said they normally don't have one. He referenced how Michael had said the day before that they use lights, and asked if this was something which they use as conductive signs, telling the actors where to be. Michael said they don't in this case, but they have done before when they used more actors and people who couldn't read notes. Also, there was a band brought in, which made them get a bit lost. They used different types of musicians (jazz and classical) who count differently and they had to learn it off by heart. It would be easier to have a composer but the mise en scène doesn't allow it.

A member of the audience asked a technical question, about the one step which Michael couldn't show earlier when demonstrating the process of the audio change. Michael wanted to play the Spear output alongside the midi output as you can hear the transformation.

Marc Downie asked Yosi about when he was talking about making to make the audience insecure and go against the comfort of the darkness, which you can take as an audience member. He said that if he had to choose a label as an artist he would probably be a film-maker and that he likes the theatre to be really dark, and so the opposite to Yosi. He asked why should he be insecure? Yosi said that he takes the Brechtian idea that he feels that if the audience are secure they take themselves out of the situation, and by making them realise that it's 'them' or 'us' and that we're part of a system, we can discuss the show afterwards. There is a certain necessity on his part to keep pointing out that we're in the theatre and are there to discuss. The other way which Richard Foreman uses is when people have a tendency to dream, he flashes white light into the audience to wake people up. It's not a fantasy movie.

Elisabeth asked, how much is the audience involved? Can people have a discussion during the show. Yosi said yes, in some of them. They did a format where they performed their own take on a film called My Dinner with Andre [15] in a restaurant and they sat around a table for two hours, and in-between the storytelling when the food came, they would have a conversation. In other shows it's about forcing the audience to participate, using the technique of TV, for example signs telling people to laugh, forcing a reaction. Some things you just ask them. By setting up the situation and coming back and relating to it, you don't leave the audience alone. Certain people hate it and when they started in Vienna people would leave the theatre, angry at them thinking they were arrogant. It took time to show them that they weren't provoking for the sake of it, or trying to be arrogant. Elisabeth wasn't sure if she would like it. Kornelia said it's about developing techniques to be in the moment and the necessity of presence. It's not necessarily a hook for action. Yosi said it's difficult to talk about, it's about the comfort which you create in the room over time. It's a time when anything can happen, there is an openness to it. The audience that comes now knows this and there is no fear. He hates how in the 60s audience participation was destroyed. Elisabeth explained that the reason she wouldn't like it is because she doesn't like being told to do something. Yosi said that he thinks if they do it though, it's interesting if he can confront Elisabeth with this, her constant need to feel the master of something and ask someone why they have a resistance to it.

6 3.jpg

Yosi mentioned with regards to the opera, it was to see if in a political level if they could choose one pure economic text and stick with it, with no story-line but that. A lot of people think they based it on Richard Quest from CNN as he talks like that, but actually they based it on a BBC show called HARDtalk [16]. The idea in the opera, was that every personal detail was taken out. They wanted to tell the political story without anything personal in it, as normally that stuff is in there. They wanted to focus on the theory. Marc asked why the audience felt insecure within the piece and asked how it worked. Yosi said they had a character called the Dictionary Scrutinizer who would stop the performers using certain words where a light would be flashed, and also into the audience. There was also loud modern electronic music which he thinks the audience had a problem with.

Simone asked how they negotiate the process within the production? Everything from the performers, to the music to the video. Yosi said sometimes things start with a clear setup and then they know where to have video projection, the music, the video etc. They talk about it and try it with trial and error. Simone asked if it's an organic process and Yosi said it's hard to say and it changes from show to show. When they worked on a piece around King Kong, they met a year earlier to discuss and do research so that when they came to rehearse they'd be prepared. But then with another show there might be a different level of knowledge within the group, for example this was different in the JFK piece. Someone once came to view one of their rehearsals and said it's amazing how knowledgeable they all are; things look different from the outside.

Marc Downie: Art as inquiry in the open studio


Marc Downie is primarily a visual artist and is a Co-founder of OpenEndedGroup, a digital arts collective based in New York. His presentation for Data Ecologies discussed artwork pieces and some of the processes behind them as well as the tools which he has built in order to make them and help other people. He said that he hoped to contextualise it with the framework of this meaning, whilst highlighting certain aspects of his process and aesthetics, which has something to do with notation and mapping and spaces, real or imagined.



The first piece he chose to tell us about was one from 2007, firstly because it has a lot to do with notation and secondly because it was commissioned for Mozart's 250th birthday. It was commissioned by the Lincoln Centre in New York. It was an opening commission, with them being asked to do something about Mozart. They asked the conductor of the festival, Louis Langrée, what the hardest bit of Mozart's work is to play, and he said that the last 30 or so bars of Mozart's last symphony, Jupiter:

7 1.jpg

In the last moments of this piece Mozart reveals that all the thematic material you've heard so far in the movement fits together in a five-part invertible counterpoint. He states the counterpoint, does the cannon, and then he's out. It's a fantastic ridiculous stunt of composition.

OpenEndedGroup built a series of computer programs to analyse this score and, as Marc is not one for classical musical theory, but rather for machine learning and statistics, instead of trying to codify the knowledge of music theory from a top down first principle approach, they took a data driven approach. They took the data of the piece of music, and tried to build up a machine learning based idea of what was idiomatically correct:

7 2.jpg

7 3.jpg

You quickly get a computer program which can tease apart the last fragment of this music and split it into themes. You can apply this analysis to this piece of music as a whole, teasing out when the themes appear and what the permissible harmonic structures are that lie underneath. When working on this, OpenEndedGroup started on a course of how to figure out how to draw music as visual artists, making them think how to render music visually. A study they did early on in the piece, can be seen below:

7 4.jpg

A progression of ways of drawing a passage played by a violin. At the top is a piano notation (pitch goes upwards, time goes across) and the next line is the piano role notation played as a performer. They took each musician in the orchestra and recorded them in the studio to build up a multi-layered recording, and segmented that automatically so that each sample represented the note played. You can stretch and deform the timing of the piece of music, to draw not the way it was notated but rather the way it was played at tempo. They also video-taped the performances as they were being recorded in the studio. You can then analyse the video of the note being played. With stringed instruments you can tease apart the way it is being bowed and so build notations off of that in the last two lines, and show the physical attention that goes into playing.

The piece they constructed out of it was a live real time, never-repeating computational cluster:

Ten computers spent six weeks re-composing this piece of music from scratch. The first few notes are random, with the rest of the piece emerging out. After a short while, the themes are pieced together which are taken from nearby computers. You end up with a CD quality recoding which is searched for by the computer. It took the computers 20 -30 minutes to build up a piece.

One of the rules they set themselves was that as the piece got closer and closer to Mozart, the physical movements of the performer would leek into the notation system, so that by the end of the piece it looks like an animated abstracted performance.

The piece touches upon techniques and aesthetics which are present throughout Marc's work. One thing he is constantly trying to think about is an axis between the hand and the code - something which is handcrafted (in a physical and theoretical sense) and something which is algorithmic, a formal idea which has to be executed to see its consequences. Once you have the below axis, you can play with putting other things on it, for example Photoshop would sit to the left:

7 5.jpg

Algorithmic is something you might call experiential; you have to experience it and then tease something out. Because of this axis Marc found it took a while to self-identify as an artist. He originally comes from a physics background, which helped him understand the idea of the handmade which he was encountering. What he finds entrancing about the hand is the physics of the situation, for example the charcoal and the paper push back. There is something uncontrollable and outside the body to this experience. To have a relationship with the algorithm is artistically enjoyable and to do this he needed to develope mechanisms of control with computers, that provoked a long engagement with artificial intelligence. Artificial Intelligence can be seen as an attempt to get computers to do clever things, when we are in-able to express to them in their terms but we want to get them to do things.

Marc did a series of collaborations with Choreographers and started off by showing us a diagram from Trisha Brown's notebook:

7 6.jpg

In the 60's/70's she was making pieces in a fascinating way, essentially constructing machines for making movement and also writing recipes/instructions. She wrote out sentences, or rather programs, which contained tasks for her to achieve. The picture above shows a hypothetical cube around her body with letters contained inside the cube. A performance of this piece, Locus [17], is an incredible high-wire act of the human body, of physics, and this had a great amount to offer for Marc.

Inspired by working with Trisha Brown, OpenEndedGroup has worked with a number of choreographers. He spoke about a piece called Loops [18], a portrait of Merce Cunningham:

7 7.jpg

They took him into a motion capture studio, capturing essentially his last performance, which was with his hands. From this they made an installation:

It's composed from Merce's voice, of him reading from his diary about his first visit to New York, which is propelled mathematically into some prepared John Cage piano samples so that it is matched with his speech. There were 44 motion capture points which moved with the data and decided what other points to connect to. This makes a real time system which OpenEndedGroup could sit in-front of and express their preferences. Once again, you have this code like system with its own rules and formal ideas. You implement it, and as an artist your job is to tease out what is hidden within it. When they made this piece they found that they started producing scores. They needed additional notational strategies just to keep a handle on the larger aspects of the piece. The scores were made by hand but as time went on they realised that there was a tremendous opportunity to make those scores actionable and executable. The desire to be able to score on a computer, to score a code, resulted into an investigation of what tools we can have to write code on computers.

Marc spoke about their rejection of motion capture which OpenEndedGroup developed a year ago, after they'd exhausted it. Motion capture gives you a high resolution capture of motion, but it deletes the look of dance, the clothing, the breath, the sweat. Last year they developed a system for capturing the opposite of motion capture, the surface. The piece is called Stairwell [19] and it was commissioned by the Hayward Gallery to work with Wayne McGregor:

7 8.jpg

The space they were given was a concrete stairwell. They took the challenge and shot the footage for the piece, which were improvisations by McGregor, shot in the space itself. They used a three way camera capture system, with which they could capture the surface geometry of the scene in a way where the computer could work out the difference between the figure and the ground. The visuals for the piece, which ended up as stereoscopic cinema, came up out of this constant attempt of the computer trying to figure out what was dancer and what was architecture:

This indicated that they could have a serious investigation around this. In order to survive the code aspects of the pieces and to survive several pieces they made in the theatre, they made their own tools, something Marc started doing as a PhD student even before constructing OpenEndedGroup. For the Loops piece, they got concerned about the legacy of their work, and that of Merce Cunningham. The dance he did for that piece was the only one he didn't set on anyone else and it was his last. The Melon Foundation was interested in the preservation of their work and commissioned a project where OpenEndedGroup tried to preserve their portrait of Merce Cunningham. For this they used an excuse to open-source an artwork and all the tools constructed to make it. This initiated a project called Field [20]. They got Merce to open-source his performance. A few years later, a museum commissioned a series of re-mixes of their piece, which speaks to the idea of what you leave behind. Five well known media artists re-mixed their work in-front of them, which was terrifying! Below is a screenshot from Field:

7 10.jpg

It is very much a chameleon environment. It is a code writing one, and it likes you to write python. There is an additional canvas where you can make ad-hoc notational systems. The code on the right produces lines, which then go in the window to the left:

7 11.jpg

The lines on the left can be edited with the mouse and the code on the right can still edit it, after you have. These ways of editing are constantly being merged together. This approach that you can have multiple accesses onto a date is something which is echoed throughout the environment. A lot of people are interested in it, including architects, who think it is an interesting take on parametric modeling. But the ultimate goal is not to produce a competitor for an already existing product, it's to think about what a product actually is. It's difficult to keep track of such a large project with music that allows you to align pieces of code and notation. When you try to patch it together with conventional tools you find strategies embodied in products. There are timelines, like in software packages like Myer and Flash and this duplication of redundancy is offensive to Marc as an engineer and as an artist. It's problematic that you have these representations that are not interesting, nor can they talk to each other. The same problem exists with data flow systems. What is the tool where you can consume these products whole, in order to cause them to integrate and what is the environment that you need to make your own ad-hoc notational paradigms? Field, allows you to talk to environments like Max MSP and Processing but you can also duplicate parts, and hack them together. The sense of product begins to dissolve in a fascinating way.


We moved onto the Q&A. Elisabeth asked about the communication tool (Field), between the types of programs and whether it existed and can be downloaded which Marc answered with yes. But he also warned that when you dissolve the idea of a product you also dissolve its usefulness. It's an invitation to write code and stitch things together but it's more an invitation to write code. Elisabeth asked what you need to know to use it, Marc said you need to be able to write in python as well as some other languages.

Tim asked if there are any other people using Field and what they have done with it, that Marc didn't expect? Apparently there are people making 2D prints in it, and using it for offline video editing. The architecture community are using it as a lot of them don't like their current tools, due to not being experimental enough. The other exciting thing is that there is a movement in science, with it being dragged by data, and this type of inductive science means that they are in a position like an artist at a computer. They have some material which has potential in it, with algorithms which they did not write and they want to apply it to the material. OpenEndedGroup are getting interest from this group of people and also a climate scientist is currently using Field. There is a goal to make your investigations visual, transparent and understandable.

A member of the audience asked about the relationship between dance and technology. Dancers often see their bodies as a tool, and the relationship with a computer can be a long distance, so he wanted to ask Marc about his collaborations. Marc responded with discussing how one of the things they're privileged in is that they always have a choreographer to dance for them in their projects and if they like what they do they get invited back into their world. They exploit the real time-ness aspect of their tools. The initial threat as a toolmaker came through working with Trisha Brown, as he wanted to see the dancers as computers played out in front of his eyes. He saw that she could turn a dancer's head with a single sentence whilst he's sitting there typing away in java and so he saw this as a sign that he needed to change this approach.

Someone asked if Marc was interested in dancing diagrams. Marc said yes, he's fascinated by all notational strategies in dance. He'll sometimes find someone who is obsessed with one of them, who thinks they have the one which is the best. He said that all choreographers have amazing notebooks, with amazing notational strategies. These are taken very seriously, but are abandoned when the piece is over.

7 12.jpg

A women brought forward an open question, saying that she's very interested about the AI elements in some of the pieces. She wondered how do you work with it as a tool or material for creating a visual for a piece, not having complete control over it? Marc explained how it's all about control and especially when he was growing up there was a real rhetoric around emergence, the new math, and that cross-pollinated into art. There were a lot of people saying they were working emergently, making complicated systems but not knowing what they were doing. The danger with being that uncontrolled is that when something amazing happens you can't take advantage of it. You're left with the question, what form of systems are that unpredictable that they can give something back but are controllable enough so you can take advantage of them? There are certain techniques in AI that offer answers to this which are artificial agencies, where you can choose what can happen and not happen again. Robert asked if it was like this for the Mozart piece, Marc said it was more in a machine learning type of way rather than AI, it learns it rather than just doing it.

Kate Sicchio: Hacking Choreography


Kate Sicchio is a choreographer, media artist and performer whose work spans dance performances, installations web and video projects. She has recently completed her PhD at the University of East London and is a senior lecturer at the University of Lincoln.

Her topic for discussion at Data Ecologies was Hacking Choreography, exploring the overlapping of programming and choreography and the principles of computer hacking such as re-proposing or subverting. The purpose of her research is to find the parallels between code and choreography as well as to find implications in areas such as live notation and live coding.

8 0.jpg

Hacking Choreography

Kate's perspective on hacking explores the idea of how to change something and re-purpose it.

Most of Kate's work involves live realtime projection in live performance, which she programs, choreographs as well as performing in. She normally creates all the elements of the pieces. In her head she doesn't separate these things; they're all choreography to her, including sitting and coding. Her project developed out of the idea of how does code become choreography.

One of the first things which you need to do in order to think about the idea of choreography as being programming is to expand the definition of choreography. You need to separate dance and choreography as they're actually two distinct things. She told us about dance analysis systems and notation systems, starting with Rudolf Laban and his principles about choreutics (spatial principles), kinetography. His systems and ways of discussing dance revolve around the body. He didn't have a way of notation or describing what he called the 'general space of performance' and he focused on the body. As dance discourse developed Valerie Preston-Dunlop and Ana Sanchez-Colberg looked at choreography as the 'nexus of the strands of the dance medium' and started to look beyond the body making these movements and defined choreography as a relationship between 4 strands which are:

- Performer
- Choreographer
- Sound
- Space

The idea is that the choreography is when these things have an interrelationship.

More recently this idea has been developed further. One project which takes this further is William Forsythe Choreographic Objects, where his dance score has been taken and re-produced in several different mediums, looking at what else can become the 'performer' in the choreography and so removing the body.

Marten Spangberg in his book Spanbergism has said that dance and choreography are completely separate. His book started as a blog, which he said was choreography. He set it up as if it was a choreographic practice, planning on spending a certain amount of time on it each day as if he were going into the studio with dancers.

Another person Kate told us about was Siobhan Davies who has come up with different choreographic patterns and giving them to crafts people to work with, for example ceramics, which has been presented as choreography.

All these examples show that when we remove the body from choreography, we have a wider potential for what can be choreography. However, Kate said that we're not quite removing it. She told us a quote by William Forsythe (2009):

“A choreographic object is not a substitute for the body, but rather an alternative site for the understanding of potential instigation and organization of action to reside. Ideally, choreographic ideas in this form would draw an attentive, diverse readership that would eventually understand and, hopefully, champion the innumerable manifestations, old and new, of choreographic thinking”

So, we're not substituting the body, there is still a relationship between the body and choreography, it's just not centered around the body anymore like Laben's work and the focus has moved a little away from it. Forsythe proposes that choreography is a way of thinking. One thing which Kate doesn't quite agree with is his use of the word 'organisation'. She likes to think of choreography as relationships rather than this.

Kate went on to reference Deleuze and Guattari's 'plane of composition' which is the idea of how interrelationships are what makes a composition. A plane of composition is very similar to how Kate is defining choreography. She thinks that once you move away from the body you can move into new territories of space, which is what she did on her PhD. She focused on something she called 'Choreotopology', exploring how choreography can create non-euclidean spaces and how topology could be used to describe her choreographic practice where there is a dancer in a physical space, with a camera translating it, and then it's in a projection but it's in its overall composition within a live performance, all this is happening simultaneously. She decided the relationship of this was topological and the choreography of this is choreotopology. Another manifestation of this idea of separating dance with choreography is the idea of the code becoming choreography as well.

There are a lot of approaches to this idea, it isn't a new thing. This idea of computational elements within performance can be found in the work of Merce Cunningham and Judson Church Dance Theatre, including Trisha Brown, and William Forsythe, who has worked on a piece which is all created by algorithms and then danced. Scott deLahunta created a piece called Software for Dancers, which had mixed reactions - a lot of people thought is was before its time. Kate has become involved in the world of live coding and Nick Collins does this without computers. A lot of this is scores for dancers, but Kate says he doesn't realise this. She thinks he's a performance artist trapped in a live coder's body. She went on to show us an example from the Software for Dancers project. It is the code for the ballet movement 'do glissade':

8 1.jpg

In the code you can see they've defined their terms and it shows how code and choreography can overlap. This project has inspired one of Kate's pieces, and within her own work she wanted to find something similar to this which she could do with scores, and start to change them in realtime. With her video tracking systems she jacked a Kinect and was interested in how she could hack the movement system. She began with existing scores, taking this as the code and then changed it to make the performance. Below is the first hack she started with, taken from the Fluxus Performance Workbook [21]:

8 2.jpg

And here it is being performed by Kate:

It was an experiment at University of Lincoln. In this score you label objects and she created hers with paper and stickers. The first part is her following the score, interpreting it rather than hacking it and she used audience members within the piece as the score said you could. She did this with very little preparation, the idea being that the composition would emerge in realtime. This is because she wanted the hack to be in realtime. There isn't a lot of choice within the score of what to do, so she wanted to make her decisions in a realtime, pressured situation. The initial way she decided to hack it was to rip up the paper and categories, and making new ones out of the language which existed. She ended the piece when she got frustrated with not knowing what she had done, fed up with making the decisions in realtime.

The next choreographic hack was when she made her own code. It's made to look like java script, but also made so a dancer could read it:

8 3.jpg

8 4.jpg

8 5.jpg

She had her setup with her two dancers, defining the movement she was interested in. The dancers had to interpret this. The choreography section is the relationships created. She then gave the dancers an order to run it in, giving the code to them the day before the performance, without any other instructions and so they had to work them out themselves. Halfway through the performance Kate put up new instructions, this was the hack, and the dancer's had to change their relationship depending on this. Kate did actually give them the hack an hour before the performance, she they knew it was coming but they had to figure it out in the moment:

The order they performed it in was the movements straight through, then they performed it with the relationships and then they performed the hack when it appeared on the screen.

One of Kate's colleagues in the computer science department got very excited about the piece and wrote the code up actually in Java script as opposed to Kate's pseudo script, but it is very similar:

8 6.jpg

She thinks that the dancer's would have understood this too.

The last one Kate made was about the dancer being a hacker, where she gave verbal instructions:

What was interesting about this one was, that the score started with what Kate was saying, but then the dancer had to slowly change what she was doing so she wasn't doing what Kate was instructing. The score isn't what Kate was saying at all, just at the beginning, making the dancer the hacker. This experience led Kate to a community of live coders. What she has in common with them is live compositional processes, in that there is some kind of system or score that is set up and within it there is a frame from change. There is also a transparency about it, with the projection of code. This led her to work with Alex McLean on a piece called Prism 11:

8 7.jpg

What they were trying to do, was to build up a feedback loop between them. Kate would do set movements which she had pre-coded and were projected behind her and he was using his system (unfortunately he had brought the wrong one for the performance), which was projected too. Kate would change dynamics and qualities of the movements whilst performing, and he would change the dynamics and qualities of the sound he was producing, which would then in turn make Kate change what she was doing; this created a feedback loop. Both the codes were projected side by side. This was completely un-rehearsed beforehand and most of Kate's choreography doesn't use sound so this was a big challenge. They want to perform it again in a rehearsed manner. There is a score but this live compositional process is the priority.

Kate is taking part in other choreographic hacks. One at the Arnolfini in Bristol, with a live notation group who look at where live code and live art meet. Kate is one of the live coders, and the artists are going to draw Kate a score, which she follows live as it gets created. Another piece is Trio A by Yvonne Rainer, which Kate has hacked before. It's a very specific piece with specific parameters, and Rainer wrote a paper about it comparing it to minimalist sculpture. Kate has used her parameters before to create a choreography. Kate is interested in finding new ways to hack the piece.

8 8jpg.jpg


We then moved onto the Q&A about Kate's work. Elisabeth asked a question about timing, as if you have to read what is on screen surely there is a gap between the action and the movement. Kate said that she lets the gap be there; it's a computational moment and the dancer has to deal with it which is an interesting tension. Elisabeth also asked about the body memory, and how with stuff constantly coming what does it feel like? Kate said it's scary as you can't rely on what you've rehearsed. You have to give up a lot of control which is difficult as a dancer, due to being trained with techniques. The traditional way of notating dance tells you how your body should be and as a choreographer you're giving it all up. Kate feels she does this, as being a control freak, it forces her to give all that up.

8 9.jpg

Tim asked why Kate concentrated on textual descriptions rather than diagrams? He thought this was interesting as she's not really talking about space but movement, which space comes out of. Kate said that this comes out of the topology, that movement defines space, but there is still the frame of the black box in the case of it and the diagram stuff will happen, especially with her work around drawing which she is currently focusing on.

Robert commented that he thought it was a very exciting moment when the two dancers went into the hack and had to focus on each other. He wondered what would happen when you increase the number of dancers, each with different instructions? Kate says she wants to try this as they'll have to realise their relationships rather than just the relationship with the movement and their own body.

A man in the audience told of how there are a few dance companies in Melbourne working with instruction based improvisation. He wondered is it the same thing as hacking choreography and how do you differentiate between them? What is the difference between hacking and instructional improvisation? Kate thinks that with improv in dancing that has a score, or an underscore, there is something which you know is going to happen a lot within the piece even though it is improvisational, the scores are decided on beforehand and the movement or your relationship is improvised. The difference with this is that there is a real time change of this score, and that's the hack, you might do something which wasn't in the original score which you set out to perform. The man referenced work he's seen where people have been invited to come on stage and put on headphones and are invited to perform something first time, and also where performers are wearing an earpiece through which other dancers are telling them what to do. For him there is the hack which is glamorous and anonymous and there is hacking which is about working at something a lot and not knowing what you're doing beforehand. Kate said that here is an app, where the dancer gets feedback which is on interest.

Marc said that is a tradition of problem solving and task based choreography. He thinks that in addition to Kate's work being real time she has a commitment to show her code. His question for her is, what does she get from that? She said that the pragmatic thing is that the dancers need to see it, and that also there is an in-joke that the audience gets to experience, which is that it's a fake java code. Marc made the point that, if you see two dancers performing a piece, you might never see that again and wondered how Kate felt about that? She said that this is also an overlap with live coders. They don't save their code, they throw it away as that moment can never happen again like that so why try to re-create it. Through this project she is subscribing to that.

8 10.jpg

Yosi asked about when projecting a text and looking at a dancer, are you trying to figure out whether their mode of interpretation is right or wrong? Kate said that with the projected text, the main feedback was that the dancers could have performed the initial loop for longer so that the audience can learn it before the hack comes in. Yosi said it's interesting, because even though it's happening live it kind of becomes a script. There is a tension as to who is solving the tension, the dancer or the viewer? Kate said this is something she's not sure about yet but yes the tension is there.

Simone asked where Kate is planning to go with the project? Kate would love to have an evening of 20 choreographic hacks and present it as a show. She has slowly been developing the hacks, having played with the three initial ones and is finding new places to go with them. The one with Alex McLean could stand on it's own as it was almost a 20 minute long piece and uses sound which is different to the rest of her practice. This one might develop into something else.

Tim referenced when Kate spoke about the person who took their influence from the Fluxus Performance Workbook and using that as a basis from what they derived other stuff. This made him think of what Elisabeth referenced the day before about taking a body of work which has been re-interpreted and essentially hacked, like taking old Stockhausen pieces and playing them with modern instruments. He asked are the Fluxus art-pieces the pre-cursor of this? Kate thinks it is a pre-cursor, it's one trajectory, where in these performance pieces there is a score and the performers role up and do it. Working with the live artists on the notation project, they have an idea and do it on the day, this is one influence. Looking at the choreographic tradition, it is less about how it goes on the day but rather about rehearsing and refining the material. Another part is taking from the x but also rebelling and how dance doesn't do that process.

The man who spoke earlier about the dance companies in Melbourne said that he liked the idea of the instruction set, and how this is the dancers crashing and having a segfault. This is something to look at as well, getting trainers to train in a way where the rules are impossible, this would be interesting. He said that when he's worked in live performance, it isn't good to ask them to break the rules as they pseudo perform, they have to come to it themselves. Kate thinks this is about negotiation. She gave the example of the drawing project she's working on. Say one of the artists creates a smudge, how does she smudge her body? It's about negotiation. The man agreed; in a context like this, where you're watching the live code, this element makes the rebellion not as interesting as just an inability of being able to keep up with the chain of command. He thinks this will position the audience more actively within it, as they have to follow it and decide whether it is right or wrong.

A lady asked what Kate's experience of the audience is in these pieces? For the dancers it could be interesting and challenging to perform the hacks but the audience are trying to compare the score with the performance and are perhaps wondering are they reading it wrong. Kate thinks the audience has to negotiate as much as the dancers do, and decide if they're executing the code correctly. She added that the three pieces have not been performed for a general audience yet, but in a dance department and the other one she doesn't count as it went wrong.

Lev Ledit said that the way Kate codes is like the old way of coding and that doing it the new way, with object orientated programming, would fit very well with her work. This is because the program itself is the stage where nothing is happening, the first thing is the event. Kate thinks this is interesting, and that her own programming experience first started with object orientated programming.

Herbert Lachmayer: Staging Knowledge - An Active Decadence


Herbert Lachmayer holds a professorship at the University of Linz in the Department for 'Staging Knowledge and Imaginative Rhetorics' as well as lecturing at the Bauhaus University Weimar. His background is in philosophy, sociology and art history. For Data Ecologies his aim to examine and present the research around Staging Knowledge.

9 1.jpg

Staging Knowledge

Herbert told us about how he developed Staging Knowledge as a format of exhibition making to put together the scientific rational aspect. In Austria they have always had a contradiction, with science being responsible for objection and art as a low aspect for subjectivity. Their aim was to combine this and so he developed an exhibition format where artists and scientist worked together. To do an exhibition it's a lot of money and organisation, but he can bring a scientist into a situation to make a subjective judgement. They can finally say, do you prefer this picture or that picture and this is always a very strange thing, to bring the scientist towards this subjective judgement.

It's a speciality that they have a process of producing an exhibition as one side of a conversation - they are speaking and working, bringing the obsession together to make the concept. They then have to speak to visitors for the next three months, everyone involved has to speak throughout the show. This is to kick off the concept of it within everyday life. With the Mozart exhibition, Herbert was speaking everyday for five hours for the audience. This is a good concept, to make conversation and not only discourses.

They produced wallpapers and carpets as as these things are always together. In his talk Herbert touched on the start in the 18th century because it's important to recognise that around then individuality was invited. For example the emperor wanted to outsource his desires of individuality to the artist, like Mozart or Weimer, who were on the same level as the emperor in a way, because to write a good libretto they have to understand the power fantasies of the emperor. A position of court artist was finally created, for which there was competition. After this you have the romantic genius, and then the bourgeois. The bourgeois society have no idea of power fantasies like the emperor does. Herbert went on to show us a video of the Haydn exhibition in the Esterhazy Palace:

He told us about how he likes castles; they are a repressing architecture built up just for the emperor but when he puts on his exhibitions, they are for everyone, for non aristocratic people. There are 18 monitors integrated and you realise the context. His issue is to have absolute top serious scientific research but on the other hand, a model, a lifestyle and an aesthetic surrounding. He thinks we need to learn the desire of individuality and that this is important right now.

He went on to mention how the time of the French Revolution is always important, and how you can look at some misunderstanding of the 19th century with the 18th century. Surrealism is also important to recognise. Rationality has a surrealistic intension.

He went on to talk about etiquette and showed us a picture of a couple admiring the sunset. The one on the right would be seen as the wrong way to do it as they're too excited:

9 2.jpg

Politeness training was important and it is important to recognise what was happening over the last few hundred years with Romanticism to Absolutism and the history of the 19th century. Things altered by taste. This taste intelligence is a social incompetence and tied in with fantasies about social positions.

In the sciences of the 18th century, there was a real scene within the Royal Academy in London. You have Hogarth, who was very analytical in a rationalistic way. You have a great voyeur because of the change through the French Revolution and afterward. In the 18th century people lived in parallel worlds, there was no ego, which he thinks young people are doing today as well. He then spoke about reading as this was always combined with fantasy and reception. The discovery of the emotion was finally there after the second half of the 18th century.

Eroticism was a real piece of art. The production of meaning and guise was a sublimation act from sexuality. It was not a Madam Bovary, it was a very real thing. Eroticism was high in level, but seducing was not always about sex, it was also about a new idea, about inspired thinking. Corrosion of society can be turned into productivity. The 18th and 19th centuries was highly productive by decadence. In our current bourgeois society decadence isn't really in. For example when you entered the box of an opera it becomes a balcony because inside the opera this was the cosmos of the emperor:

9 3.jpg

He went on to show us another exhibition, and told us of how Lichtenstein's carpets always have the colours closest to this time. He curated an exhibition in Weimar entitled Wozu Braucht Carl August Einen Goethe?. The castle's atmosphere changed, with his exhibition occupying it:

9 4.jpg

9 5.jpg

9 6.jpg

9 7.jpg

Franz West designed wallpaper for the exhibition and even the elderly people liked it, even though it's very psychedelic. Herbert thinks that white walls are very aggressive and so prefers this. He then told us a bit about his exhibition for Mozart which was aimed at children:

9 8.jpg

9 9.jpg

9 10.jpg

This was in the children's museum. It was very interactive and children could dress up and engage with the exhibits. For Herbert, it was one of his most beloved exhibitions.

So, this is all a way of staging knowledge. We are producing this to put together the idea of good feelings and emotions combined with real serious information.

Herbert took us through different aspects of culture reflected in art from sexual aspects to culture consumerism. He then took us through some more images of his exhibitions, including some work which he had produced with students:

9 11.jpg

9 12.jpg

9 13.jpg


We then moved onto Q&A. A lady in the audience asked him to talk about his technique with curation, and how this is connected to the theoretics of interactive art spaces like the Palais de Tokyo. Herbert said that he isn't alone with these techniques, his main issue is to bring in the sciences and subjectivity which we have to reinvent. He invites scientist to come and speak, and use the space for thinking. She asked what would be in contrast to this technique? For example exhibitions at the Getty in Los Angeles, where they limited people's interaction with the information. So comparing the idea of being open and presenting your research to being really abstracted. He responded that it's important for us right now to live in contradictions and contrasts. The mind of the ego is a dream. It's important to change the perspective always, and to recognise information it must change always. He spoke about the Getty, and how because every object is so overly restored, the work looks like copies rather than original pieces. In his exhibitions they change the perspective. This is accepted by young people and children as well as elderly people. The latter even wanted parts of the wallpaper and carpet; they were inspired.

Robert asked whether if in 30 years from now someone wanted to replicate one of these exhibitions, how close could they come to doing so? Is it notated (the objects, the videos, etc)? Herbert spoke about how they have tried to create a very good source of information online so that people can change the pictures. He spoke about an office exhibition which he produced in 1998 in Linz and how the idea of staging is important to him. It's not stupid interactivity like just pushing buttons. He thinks that the whole process should be documented. Since 2003 he's worked on 17 exhibitions.

9 14.jpg

The lady who spoke before asked if he is interested in up-datable concepts. He said yes, and he has lots of scientific stuff which is serious.

Someone said that he doesn't think it's possible to see Herbert's exhibitions again in 30 years as the main thing in the exhibition is him and his tours through the exhibition so without him it doesn't exist. Herbert agreed, his identity isn't repeatable, but making exhibitions is a great thing and a great thing for students to do with him.

Lev Ledit: Forecast the Unpredictable


10 0.jpg

The last presentation of the day was by Lev Ledit, who is CEO and Game Designer at GAME GESTALT. He invented the virtual world Papermint and has previously worked for Rockstar Vienna as a designer. He teaches game design at the University of Vienna. For Data Ecologies he proposed to explore the challenges which game designers face everyday when they want to create a new game.

Forecast the Unpredictable

Lev started by saying that when he talks about games he has to think about who it is he is talking to. When talking to us, he decided he would go a bit deeper and talk a bit about play. When we see a cat playing, we can think about how they have rules, they are pretending to be in a different situation. Tt's like when children play; they are having fun, it's obvious. The question is, where does this fun come from? This is connected to one specific game he's working on and it's about learning. The brain recognises when you accomplish something, and it gives you a good feeling. But how does it know when something is good? How do we know what is good and bad? Nature invented these two feelings and all animals have this, including humans. It comes from a location in the brain:

10 1.jpg

Eating healthy food and having kids, these both give you a good chemistry in the brain. This also happens when you realise that you've learned something, or solved a puzzle or are accepted in a group, or breathe fresh air. The feeling which Lev said he would focus on was the feeling of accomplishment; this is how games work. The feeling tells you when you've achieved something which makes you happy. This doesn't solve the question of why the brain knows what is good. Lev thinks it must be something to do with your parents, and how they reward you when you do something they think is good. You also have old parts in your brain which make you know what is good and not good, it's instinctual. For example we know to be scared of bears and we don't need our parents to tell us that. Games tap into this, we know what to do and we know how the system behind the game works. This is a good ability, to see through a system.

You learn to draw a distinction, to know what is you and not you. You differentiate yourself into one. Over our whole lives we differentiate between things and learn more and more. Through electronic impulses, for example, we construct what a friend is.

So we learn who we are, where to go, what we are, what is interesting, what are our surroundings, and this is the game which GAME GESTALT is working on. It's called Kognition and it starts when you realise that your Facebook page has been eaten up by aliens:

10 2.jpg

The page of information is eaten, and from then on there is no language or signs to assist you as a player, you have to learn everything. It's about trial and error. For example, there is a new language you have to learn that is made up of signs:

10 5.jpg

You then learn how to get photos and that you can move them whilst also collecting more:

10 6.jpg

10 7.jpg

10 8.jpg

There is an infrared camera, and so you learn that you have different eyes. The image from which can be seen on the bottom right in green:

10 9.jpg

You realise that this is all part of an alien invasion puzzle, and you learn to use a camera so you can view space. You learn how to solve puzzles which takes you through a narrative. You also learn that you are one of many individuals; you are a brain part and contribute towards the decision making. Below are some of the test outputs where you have to figure out where you are:

10 12.jpg

10 13.jpg

You slowly realise that you are on a sphere. It turns out that the aliens have built the puzzles to work out who is intelligent and who isn't. Lev still hasn't solved the below puzzle:

10 16.jpg

Below is a part of the server, which knows where to give you your photograph or not, depending on how you do:

10 19.jpg

10 20.jpg

Lev then showed us some different models, which he would show to investors when discussing games:

10 22.jpg

10 23.jpg

10 24.jpg

10 25.jpg

10 26.jpg

10 27.jpg

When you are a designer you need to be able to define the mechanics.

He told us that both players and investors want specific types of fun which are:

1) Sensation
2) Fantasy
3) Narrative
4) Challenge
5) Fellowship
6) Discovery
7) Expression

They may want a narrative game where you feel like you're in a film, playing the lead role. This is how Lev has to talk to investors about the game. In the end, it's a lie because you'll never be in a story when you play a game. It's a theory of the experience and as you play you have the experience but you're not in the theory. It's about the chemistry when you're playing. The story is about theory and the game is about experience. However, these mechanics can lead to dynamics and these dynamics can lead to a challenge that can be compared to the feelings of the main character in a film. This is how they test it. Before making a game they can set a goal of what should happen so that is can be test. For example one thing to test is competitiveness; if the player can't measure this, then they don't know where they stand.

The dynamic of a game changes when you alter a mechanic, so when you make a shooter and keep everything the same but change the amount of ammunition available, the game won't be as rich. But then when you up this, it gives the player more to do and engage with. You can't forecast what a game mechanic can do until you try it out. When you write down a plan for your game, you need to build a prototype and test it.

10 28.jpg


We then moved onto Q&A. Someone asked about a game prototype and asked when does this end and begin to be a real game? Lev said that's a good question as it's hard to figure out what prototype you need for a solution. You can normally bring out some learning achievements, and he always wants a learning effect, which the prototype needs. If it doesn't have this then it's not a prototype. You can make prototype without programming, you just need to know what learning outcome you want. So Lev often prototypes in the 'real' world, without computers, but rather people.

A lady in the audience asked if Lev knows about the concept of 'proprioception' (often linked to dance), which is about your nervous system being able to sense another heat body and where you can organise yourself in space without looking at what is next to you. It's the embodied intelligence to sense another body and to map together organically. It is what stops us from walking into each other. She wondered what Lev thought about that and how you could tap into it with regards to a prototype? He said he would try to make a new sensor out of it. He's not sure what it would give him, it wouldn't perhaps be a picture, but more an instinctual thing. The lady said it's not just about crashing into things but about being able to build non-verbally together, like a shared choreography. It's how improvisational work happens. It's a felt information which is being carried non-verbally. Lev thought this could fit in with a game he will be working on with students for blind people.

Robert said that he wanted to make an argument that the games Lev described are not games. He thinks that the one person who makes the best description about gaming is Johan Huizinga in his book Homo Ludens, in which he describes the rules and aspects of time and space as well as saying that games are about disinterested activity. He thinks that no-one has changed, it is completely neutral to what is important about life. He wanted to make this argument as by getting people to collaborate as a meta-intelligence at the end to solve the problem, Lev is changing the stake. You are requiring people to invest in effort, social aspect and time, and in-fact, whether you win or lose the game, it costs you in real social time, meaning you can't be disinterested in it. You need to give that sense of being disinterested up in order to commit to it. Lev thinks that Huizinga meant that it is not about your real life problems, so you know that you're not really fighting; there is a border and it doesn't touch real life. He thinks that Huizinga never meant it doesn't change yourself, as he doesn't think people would play unless it would have an effect.

Tim brought in a similar discussion that he has heard where either you play and there are no points involved, or there is a game where there is. Lev thinks this is difficult to find the distinction between these. Elisabeth thinks that to play, it has no goal, it is joyful. The idea of winning keeps you in the game and this is the big difference for her. The other thing is, she's not sure if this boundary works, the real and the virtual, as she thinks that everything is real. We are already so much into the virtual world, so to her it's totally reality and it's the same with games. Lev thinks that the difference between just playing and playing a game is about how visible the goal is. He asked us to think of an example of play so he could answer as to what the goal is. Playing music was given as an example, to which he said the goal is to get better. It isn't always about having to win. Michael gave the example of bio-chemistry, you're trying to find life. Marc was wondering what the goal for a kitten playing is. Lev thinks that the goal is to get better with the paw co-ordination.

Someone made the point that often the goal is not known at the time of play. We practice, and then we can impose a sort of goal onto it in retrospect. What's interesting about play, is that there is a self defining process, where you might not know your goal and we don't think about them, we go into a flow space, play becomes a game after we stop playing. It is not a game whilst we play as it feels real and understandable for us. When the cat is playing, they don't know about co-ordination. She said it's like the Schrödinger thing where the cat can be playing and gaming at the same time, super position is possible, there is no line. Lev agreed.

Lev went back to address Elisabeth's questions and discuss the idea of the magic circle of play. You experience something, which is either good or bad and there is a differentiation between when you do things, for example killing a person on a computer game feels very different to killing someone in the real world. Michael gave the example of boxing in a ring and then fighting in 'real' life. Someone suggested that with kids, they find it hard to shake off the gaming world. Elisabeth thinks this threshold is going down. Lev thinks that we are empathic people and we know that a computer character isn't hurt, unlike a real person would be. Someone referenced the idea being discussed about playing being learning, for example cats learning to hunt through play. There is a distinction between the real and not real but there is also the idea of mimicking something.

Yosi gave us the overview of a story in which there is a country where they keep having Olympic games on a monthly/yearly basis and they keep on changing the rules. The people get to the point where they don't know anymore what is real and what isn't. He has a fear that this could happen in our world. Marc says that this story points to the dark side of games, whereas Lev is focusing on the idea of learning which is linked to games and this is a good side. He thinks there is growing sense that reward is attached to it and either learning happens or it doesn't. Reward is therefore something like shooting something, which is problematic and makes people repeat the same action over and over again, which is not a happy or a learning thing. There is an understanding to games and a rhetoric of gamification which is problematic, and you need to understand the stakes of it. The idea that games are fun because you learn stuff just isn't sufficient enough. Someone brought in the idea of learning being fun is insane, as it is very difficult. It's neither good or bad or fun. It's a necessity and it is unavoidable. Lev said that when you realise you learned something it's fun though. It's difficult to get to the fun part, but when you've done it this moment is good. It's about the realisation.

Marc suggested, that when you're a mouse in a box, getting a food pellet by pressing a leaver, the good feeling is the food pellet, not the mastery over the food lever; it's not the learning which is good it's the food. Lev responding by saying that no-one said its just the realisation of learning which is fun, for example making children is fun, but it's not a game, it's real life, there is no magic circle. Many things are good for you, and learning is part of this. Lev thinks that you probably can even make games for a mouse as perhaps even they play. But he doesn't think that getting the food is necessarily a game for the mouse, they don't realise that they've learned something.

Michael said that he was impressed by a King Kong game that he played for research, he wonders what is the wisdom of a game designer and how do you design the levels so the player gets the rush of learning but not getting frustrated. Lev said, this is the magic, a good game has to have interesting decision making as part of it every 15 seconds and that even looking around the environment is a decision making activity.

Simone had a thought about boys playing with monsters and then at the same time dancing to pop tunes. This is her experience of observing game playing, which she saw in a shop. She thought it was very surreal how they could float between these two spaces.

Final Discussion

Tim brought things back to one of the overarching topics of the discussion: notation. He observed that in Kognition, the alien perceptions are some kind of notation or representation of the space and that your job is to work out how these connections work and to see through different eyes. Lev said that the question here is what are the different ways of seeing something? You see it through your eyes and the computer. You can make something appear different, and it's about knowing when it gets to this point. To think about an alien seeing an animal in a different world is very hard, it has to take a lot of things out of the reality we know and bring it back to the game.

Tim asked then whether Kognition is impossible to play properly. Lev said that it is the opposite; when you play it you learn. However, the reason it's still a prototype because the things in it currently are the same as in our world. What they need in it are sensors which we don't even know about but in order to create them they need to know what they are. It's a bit of a tricky situation. Tim brought this back to the earlier discussion about proprioception and maybe that it's about this. Lev argued that this is something which we have in the world already though and that we understand it. Tim brought in one method of sense, which Lev could look to for inspiration. This is where the closeness of an object in a virtual space can be felt through a vibrating glove, and depending on your distance to the object, the vibrations would change. The lady in the audience who had originally brought up proprioception earlier pointed out that it is not just about heat though, it's about bodies in general, therefore objects too. Someone suggested that the movement of air, the sense of smell and other sensations all together are what you have if you want to use this technique, you need to take it all into account. Heat alone wouldn't do it. If you've been running, you can't tell different temperatures. He questioned Lev about their approach to thinking about an alien world, as of course a living creature from another planet would probably not be able to survive in the space which we have on earth. You need to think about different physical conditions and spaces. Lev said that they had spoken about this, and as an example had looked at the internet as that's different to our world as a space. The understanding of our surroundings is so important to puzzle solving. Nik Gaffney made a point that no matter about how unusual the world you're trying to simulate is, you'll always have to try and map it back onto human perception, which includes the dominance of things like sight, sound and sense of balance. He spoke about a game which foAM made where the player experienced the world as a plant might. However, this is from a human's point of view, as it has to be presented in a way which we can understand. For example how do you sense sunlight? If you are a green plant this is important. Do you present it as shifting colors or forms or just light? You have to use the knowledge we have of that world in order to make that mapping from a foreign unintelligible world into something we can work with.

11 0.jpg

A man attending the seminar brought in a text by Humdog [22] which was created in 1994 when she became disillusioned about the internet. Before this she participated in the early internet movement. She then wrote an article where she mapped Marxist ideas over the internet, coming up with a term commodification around the idea of value; everyone on internet forums, and now Facebook crave money. He asked Lev if you could apply this to games? Lev said yes, there are interesting games where players are solving very difficult things. One example has been asking players to find out a chemistry formula and another is getting them to name things in photos, where upon a correct answer they are rewarded. This is however, for no money, just for the satisfaction. The man in the audience made the point though that people who make games do make money.

We went back to the idea of making a game and mapping it back onto human perception (with foAM's game), with a seminar attendee asking Nik Gaffney what is the aim of this mapability? Nik said it's for it to be perceivable, for example a human's experience of ultra violet light is very different to that of a plant's, so the idea was that there would always be some mapping from one limited set of sensory apparatus to another. It's primarily to make it more intelligible. She clarified whether this is one form of intelligence and theoretically another form of intelligence, but that assumption might not be valid? Nik said he wasn't talking about intelligence but rather perception.

Tim followed up on the idea of different worlds which we perceive, and the idea of non-euclidean geometry which Kate brought up earlier with regards to the spaces which she works in. This is linked to the problem of the spaces which we work with and the ones we are trying to map onto, as they are all very euclidean. Kate said that for her it's about the idea of allowing the choreography not to be about the body and therefore being able to choreograph a space or make a space by a movement. This became the primary focus of her work. When using video tracking systems a lot of time as a dancer you're fiercely moving in front of the camera trying to trigger something, but the camera doesn't 'see' you, it sees pixel changes. She had a realisation that choreographing a computer system is actually to choreograph pixel change rather than bodies, which is when topology became pertinent in her work. This is because that's what was connecting the space of the live performer, the camera and live projection. The overall relationship of these in a composition was movement and not her physical body anymore. Lev had an idea that you could simplify his game to a 2D area with just 16 lights which you see, and they could tell you everything about your surroundings. When you click on them, you yourself move in the space. You then learn the behaviour of the lights and the space, and can interact. Tim asked how would visual intelligence work in this case? Marc thought that if you were to make that game, there is a chance it wouldn't be fun. So in order to make it fun you would have to come up with additions like reward. If he was to make an AI program to do this kind of learning which Lev wants to happen, he would come up with a reward structure to guide the learning algorithm. How they would both design these things would be very similar. This reward structure would be impended from the feeling of mastery over the domain, that you really want your reward signal to come from.

Embodiment was brought into the discussion, as the point was made that territory, the earth, the internet, choreography and technology cannot be talked about without thinking about this. We can't be virtual in our bodies, we have to be real in them. Anytime we're dealing with reality we're confronted with our own physicality, which isn't a limitation as our sensoryness is what makes us real. It's interesting to think about the map which we make together in this space. Lev said this was an interesting idea, to get rid of the body in this game. It leads to the fact that you have a puzzle, but there is no 'me'. So when you want to have a game where you are immersed in a world, you need something named 'me', which is the body. We went on with the embodiment discussion with another person touching on the fact that they were surprised about its lack of presence, along with identity. They were surprised that the discussion has focused on reward and learning rather than this, as when he watches someone playing a game, the sense of whether they're enjoying themselves or not is when they can find some kind of identification, and this crosses over into interactive and media art. He gave the example of Shadow Monsters [23], where you apparently don't have to learn much, everyone gets it immediately. He's interested in that point when someone can quickly embody something and identify something which was foreign to them.

Yosi thought that the reward question is bigger than the question of the game and education in general. This is because we are dealing with it in bigger terms. Someone suggested that we can't think about reward without looking at what it looks like on a daily basis. We can't think about this without thinking about what this is for us everyday in our own embodied experience. We are afraid of not getting what we want or losing something which we have.

It was suggested that our lives have become gamified. We get worked up quickly when we discuss games, but in a more negative way than perhaps previously, and this is connected to things like Facebook. We are beginning to react to ideas which were getting us excited before, because it's now become part of this market economy where we are rewarded for friendship etc. It now feels like it's an analogy for life. Lev thinks that there is a feeling that all this is over-simplified. Simone suggested that what is missing from the discussion is experience and this is something which everyone here is dealing with. Reward is just a part of this. For Simone, this doesn't work, but the experience does.

Robert asked if anyone else was connecting Herbert's exhibitions with these games? He thinks that Herbert was reading a similar virtual world, complete with a reward structure, but the stake was how we understand someone like say Mozart or Haydn. In some respects this is a higher stake, connected to our historical identities and our connection to meaningful aesthetic experiences which are manipulative and dangerous in the same ways that we've been imputing to computer games. Yosi went to see the Mozart exhibition with his son, who found it boring but in the last room, there was a trumpet which he liked as he plays it. That is the reward.

Elisabeth was thinking about attention and what is the difference between putting your attention towards reading a book, or playing a computer game or just walking on the street? With regards to the idea of proprioception, she said she just doesn't have this when her attention is elsewhere. She doesn't feel anything else if she is focused on something. It's about this strange thing where you have your concentration, and wondered is there a difference between these different medias? She can't really feel it.

The lady in the audience who initially brought up proprioception, clarified that this was with regards to the idea of being invested with consciousness in something, and about rewards and feeling. Going back to Herbert and the idea of higher aesthetic stakes, this is about how we attend to something. If you go to a museum and are fully engaged compared to not being engaged then you will get something different out of it. Our capacity to perceive something is dependent on our attention to it. This is where we have agency.

Tim said that it is interesting how this expression of engagement or attention of experience has come out when these are much more general things than reward, which is a very specific way of engaging with something. Going back to the idea of Nirvana which was discussed yesterday, about just wanting to be in the experience, whether that is in a musical or theatre thing, this is something we all want from what we're trying to do. One of the things we're trying to do is to get rid of this abstraction.

Marc thinks that dancers and choreographers tend to over-emphasise the importance of embodiment, which is weird for someone like him to say as he used to be very into embodiment in AI. It occurred to him early on that he plays and works to make this body disappear. When he is working well, a great day will be when he doesn't stand up. A lady at the seminar said that it depends as if you're in a tension with your body, you won't feel the difference between it and the room. A man said that when he's at his computer he spends his time doing exactly the same thing and he does call this embodiment. It's about identification and the transformation of your body inside something else. When you're working on something, say AI, you're putting your body into something else. It's a transformation of self into something else. Elisabeth thinks there is no difference between cooking and being in front of a computer, to which the man agreed and said that he doesn't think it's any different to someone performing. It's about being in the thing. It's the point of a mediated sensation. Someone commented that for those of us trying to understand bodies and space in a general way, this isn't the physical body. There are a lot of ways to get to it. We have to look at what territory, location, space body etc. we have, to make individual choices of what these look like, but in the end interconnectivity is a big theme. The man previously speaking said that because he spends a lot of time with code, he doesn't think that physicality is the right word when following its shape, maybe it's a topology. It's coming through the denial of body perhaps. With regards to reward and game, it's kind of being abused, particularly simple games which get people to do work for no pay. An example is Yahoo, where because they didn't have any meta data tagged images they made some games around this to get people to tag them. In one way, people using reward to create experiences and machine learning is nice but other ways of using rewards aren't so good. He wanted to hear a bit more from the people who work with theatre and have an audience, and about them being users.

Yosi said for him there are two things. One is about awareness, a quantity and back to Huizinga, and the other is about playing with a minimum knowledge of the reward of goal. You are constantly aware of the reward, to the point that the game disappears. His point is that he thinks that the audience who attend Toxic Dreams' work now, is a very aware and knowledgeable one. It's about sharing the game with them and playing together. In this sense, and maybe it's an old Brechtian idea, on some level there are certain things you can't do anymore which you could before. Simple things like playing with an image and body position. People now read it very fast and so the interaction has to change. You now have a smart intuitive audience.

Marc brought up the disagreement between him and Yosi about whether the lights should be on in a theatre, as he comes from a world where you turn them all off. He does this because the theatre melts away. He doesn't want to create an immersive experience, he doesn't want to transport you as he actually enjoys the sensation of seeing and inspecting. The camera movements in his work are often orbits as you go around something rather than panning, so you're not trying to create a sense of space which is fake. He said how Yosi wanted to impose a form of complicity onto his audience in order to allow them to have a kind of distance; he couldn't tell if this was Brechtian or anti-Brechtian. Yosi said that every night he comes to meet the people to have an evening together and so to turn off the lights would seem counterproductive; you wouldn't do this if you had guests. On a practical level, they started to talk to people more so this was one of the reasons for doing it, but it also felt like the right way to go. He's not proposing it as a general thing, it's just good for their practice as it opened the room. Someone asked if anyone had seen My Dinner with Andre as it's important due to the idea of the beehive and how we can invite people to come together for an act of happening.

Kate said that within a lot of her work she doesn't think about the audience, to which Elisabeth agreed. An audience member suggested that you can look at map makers who make them for themselves and that this is kind of like that, but then as an artist there is a pressure to please the audience. Kate says that her audience witness the process of someone trying to interpret the map, but what they get from that isn't the focus of what she is doing. Her focus is the performer and the map. The audience is there to witness the process and take away from it what they will.

We came to the end of the discussion and the therefore the end of the last session and the seminar. Tim thanked everyone involved, with a special mention to Simone for stepping in as moderator.

Personal tools