<nettime> ISADORA: in dialogue with Mark Coniglio

[Scott deLahunta <sdela {AT} ahk.nl>]

Hello...

Faster cheaper processors and growing familiarity with software interfaces 
(by learning and/ or design) mean that artists working in live performance, 
dance and/ or theatre are seeing more software show up on their stages in 
the form of 'real time' sampling and cross media synthesis tools. This 
article/ interview tells the story of one of these tools with the aim to be 
accessible to those who are perhaps not as familiar with the practices and 
discourses of software, code, digital and net art as many nettimers.

Below is Part I. Part I and II are available on line at: 
http://huizen.dds.nl/~sdela/sfd/isadora.html.

Scott

***********************************************

ISADORA "almost out of beta": tracing the development of a new software 
tool for artists

Part I: in dialogue with Mark Coniglio

Part II: comments from Jean-Baptiste Barrière, Jem Finer, Armando 
Menicacci, Giorgio Olivero, Steina Vasulka

Edited by Scott deLahunta

15 September 2002

INTRODUCTION:

The field of real time multi-media software tools created by artists for 
artists emerged from the hardware and software developments of the early to 
mid 1980s. Fifteen plus years later hundreds of individuals and groups are 
working consistently with these tools in their own creative practice, often 
to some degree customising them; creating tools within tools and sharing 
these with other practitioners. However, there are a handful of key artist 
programmers who have devoted themselves to developing major software 
contributions to this field; amongst them Tom Demeyer, David Rokeby, 
Netochka Nezvanova and Miller Puckette. This article focuses on one of the 
latest of these contributions, Isadora, which is programmed by Mark 
Coniglio. It is intended to provide some insight for those who may be 
relatively new to this area of software tool development for artists by 
artists; and in particular the development of multi-media tools to be used 
in the context of live performance practices.

BACKGROUND/ BIOGRAPHY:

Mark Coniglio is an artist whose work crosses the disciplines of music, 
dance, theater and interactive media. Dubbed an "interactive performance 
pioneer" by the New York Times, his work has been performed nationally and 
internationally primarily with Troika Ranch, a New York City based 
performance company committed to creating multidisciplinary works of which 
he is Co-director with choreographer Dawn Stoppiello. A native of Nebraska, 
Mark studied at the California Institute of the Arts with electronic music 
pioneer Morton Subotnick and received his degree in music composition in 
1989. He was on the staff of the Center for Experiments in Art, Information 
and Technology at CalArts teaching courses in Interactive Music from 1990 
to 1994. In 1993, they started Troika Ranch while Dawn was dancing for 
Bella Lewitzky, and in 1994 they moved with the company to New York City. 
Besides the rehearsals, performances, symposia appearances and residencies 
that make up the bulk of their creative contribution to the field, Troika 
Ranch regularly conducts their popular Live Interactive (Live-I) workshop 
which gives participants the opportunity to explore the use of interactive 
computer technology in the creation of live performance artworks. 
Participants also have a chance to learn to use the custom hardware and 
software Mark has created.

While the building of new interactive instruments has a robust tradition 
within the electronic music field dating to the early 20th century, Mark is 
one of a handful of artists who have specialised in creating these 
instruments to monitor the movements of dancers and use this information to 
control other media events in real time. The MidiDancer, a wearable device 
Mark first built in 1989, measures the angular change at several joints on 
the dancer's body. This information is then sent over a wireless link to a 
computer where the data (input) is used to control a variety of media 
events (output), e.g. sonic, video, lighting and/ or robotic. A software 
programmer for as long as he has been a composer, Mark has written two 
programs to map this data flow; the first was Interactor and the second is 
Isadora. Interactor was made available for others to use, however it 
required time to learn (made easier with some knowledge of software like 
Max, a graphical programming environment for music and multimedia). Isadora 
(named after the renowned early 20th century modern dance pioneer) has been 
designed to be used by a non-programmer after only the briefest of 
introductions; placing the control of the instrument in the hands of the 
dancers themselves. Another key development is that while Interactor was 
focussed on handling MIDI data (a communications protocol that allows 
electronic musical instruments to interact with each other); Isadora, while 
it can also work with MIDI, is designed primarily for the manipulation of 
video.

PART I: DIALOGUE:

Scott deLahunta: You have been programming actually longer than you have 
been composing. So do you consider your programming a part of your artistic 
practice?

Mark Coniglio: I definitely do, but it does feel a bit secondary to my 
composition and/or media art making in that I see it as more of a "support" 
activity. The software I program allows the creation of the artwork, 
whether sonic and/ or visual in some combination with live performance, 
that I envision. I always seem to resort to musical metaphors for things 
like this. The artwork is like a musical composition; the software tools 
are like the instruments in the orchestra. If you can develop a more 
advanced instrument, you can create more advanced music. The French Horn is 
a good example; early versions required removing a piece of tubing from the 
instrument and replacing it with another piece of a different length to 
change key. About 1815 the modern version of the instrument with valves was 
invented. This technological innovation meant that you could now include 
the horn in passages where there were rapid changes of key. This was 
immediately taken advantage of by composers, notably by Beethoven in his 
symphonies. The difference between Beethoven and me is that he was not 
driving the innovation directly. I am both artist and instrument inventor. 
My artistic ideas drive the development of software and hardware I need to 
realise them; while simultaneously the programming I do expands my world of 
expressive possibilities.

Scott: I understand the analogy with the French Horn, but software 
'instruments' are comprised of another type of material altogether. Instead 
of metals being forged into shapes; one could speak of algorithms, formal 
abstractions and language. As another way to relate programming and art 
making, I am curious about the creative process of writing code compared to 
working with other materials. It seems that most programming is driven by 
the assumption that the software will have a purpose (more like the notion 
of the instrument). Once that purpose is understood and defined so there is 
a goal, only then does the programmer get down to the work of coding. If 
this is true then programming is a very different creative process than, 
say, how the choreographer might work with movement material.

Mark: When writing software it is useful to have a clear goal in mind, and 
this may not be true with other types of art making, as you suggest. Having 
a goal that is too specific can be detrimental to the process of making a 
dance for instance or composing music. I don't think I understood that 
early on, but over the last 3 or 4 years I found myself exploring much more 
organic, open-ended approaches to art making. Being involved in dance in 
particular, which relies on improvisation as a primary source of generating 
material, has profoundly influenced my way of working. Now, my approach is 
a much more, "try everything and follow your nose." By this I mean, try not 
to preconceive as much, make lots of stuff and follow through with the 
material that seems interesting and let the material begin to tell you what 
it is about. Now, it's pretty difficult to program that way, a kind of 
"goalless" coding. The architecture of the microprocessor, from which all 
programming languages derive, is actually antithetical to such behaviour. 
However, this "follow your nose" approach has definitely influenced the way 
I program. I still have a goal, but I don't often plan out the algorithms. 
I simply write towards my goal, improvising my approach to solving the problem.

Scott: There is quite a big discussion about 'software as art' these days 
in Europe and I'm sure it's going on in North America as well. Besides its 
utility, its usefulness in helping to support your art works, do you 
consider Isadora a work of art by itself?

Mark: In a word, I don't - not in and of itself anyway. It's bit like 
asking if the telephone system is a work of art. Does the creation of the 
technology to support that system display an incredibly high level of 
inventive thought and uber-craftsmanship? Definitely. I have the utmost 
respect for the creative people that designed and implemented such a 
robust, complicated, and reliable system. But that particular technology is 
dormant until you pick up the receiver, ring your lover, tell her that you 
no longer desire to see her, and a heated conversation ensues.

Scott: It seems to me that the telephone system is the collective result, 
over time, of a multiplicity of individuals and institutions labouring 
together, and it's difficult to locate individual contributions in that, or 
at least I don't use the telephone system and sense that I am in contact 
with one of its creators. But when someone opens up Isadora and begins to 
build a patch that will map an arm motion to the speed of a video clip, do 
they not encounter your presence directly, as the primary creator, in the 
look and feel of the interface?

Mark: Well, I guess the only way that I WOULD consider Isadora to be an 
artwork is the personal stamp that I have on its design and functionality. 
To take that a bit further, in a broad sense I could say that I collaborate 
with each Isadora user as they use the program. Because I can't totally 
erase myself from the software I create, they have to embrace some of my 
predilections to make use of the program; which is what happens whenever 
you choose to collaborate with another artist. It's just a question of how 
apparent the influence of the software's creator is. That's where software 
designers and artists who make software may differ, I think. A typical 
software designer does everything he or she can to filter out their 
personality and create something that is useful in a general way.

Scott: Maybe we could say that the extent software like Isadora is an 
artwork is dependent on the degree to which the maker tries to remove him 
or herself from it? I suppose Isadora then is more of an artwork than say, 
Photoshop, but is maybe less of an artwork than something like 
Auto-Illustrator by Adrian Ward (which won the Software Art prize at the 
Berlin 2001 Transmediale Festival). Auto-Illustrator looks like a vector 
graphics program, but it doesn't act like one. It misbehaves frequently 
because it has seemingly autonomous behaviours built into it that take over 
for you. Adrian creates these behaviours using certain algorithms when he 
does the coding, so as such his authorship is revealed every time the 
software does something you did not expect it to which is frequently.

But let's talk a little bit about the background of Isadora as a graphic 
programming environment for real time manipulation of media. You made 
something similar with Interactor first didn't you?

Mark: Here's a bit of history. In 1986 my soon-to-be mentor and Interactor 
collaborator Mort Subotnick had just come from a residency at MIT where he 
was using a program called Hookup created by a student there named David 
Levitt. Hookup was the first program I know of that used the "patch-cord" 
metaphor, i.e., modules that manipulate data are linked by virtual wires, 
the connection of which is determined by the user. For those in the world 
of early analog, patch-cord programmed synthesizers, this was a familiar 
interface. Mort was using David's program to do tempo following of MIDI 
instruments -- this allowed him to lock hardware MIDI sequences to the 
tempo of the live performers. I was a composition student at CalArts at the 
time, and word had gotten around that I was a good programmer. So Mort 
contacted me to see if I could hardcode some of the ideas he had 
implemented in Hookup on a Mac, so that he could use them in his next 
performance. That program (used in Mort's 1987 multimedia work "Hungers") 
would eventually become Interactor. Mort designed the functionality of the 
early versions, but I became more influential in the design as time went on.

Scott: I guess the hardware and software development of the early to mid 
1980s where we saw the advent of the personal computer and more importantly 
the graphical user interface (marked by Apple's introduction of the 
Macintosh to the consumer market in 1984) created a context out of which 
the 'computer' could emerge as a creative instrument or tool. The 
electronic music field was already well advanced in analysing and exploring 
the formal and physical properties of music as part of compositional and 
performing practice, so moving to programming real time graphic interfaces 
for this seems like a rather natural progression.

Mark: Yes, that's true and importantly a kind of creative intuition was 
creeping back in through the development of these new visual interface 
possibilities for software. Part of the thing I reacted to in Hookup was 
the way you could easily drop modules into the program and try things; a 
lot like you could do with the patch-cord synthesizers. I may not have 
realized it explicitly then, but this ability to program improvisationally 
allowed for that kind of artful playfulness that is so important. So I set 
out to make a similar user interface for Interactor. The creation of 
Isadora was a natural outgrowth of Interactor. In 1996 Troika Ranch had a 
two-week residency at STEIM, where I first saw Tom Demeyer's real-time 
video processing program Image/ine. I first started using Image/ine in 
concert with Interactor, because Image/ine didn't allow the kind of 
complicated interactive decision making that I was used to having in 
Interactor. So, Interactor would process the MIDI data from my interactive 
sensors, and then tell Image/ine what to do. By 1998 I was using Image/ine 
in a major way in my performances with Troika Ranch.

But, while I loved what Image/ine did, I wasn't fond of its table-based 
interface. And there were problems with crashing during performance, which 
is unacceptable when there is an audience. Furthermore, we were teaching 
our Live-I (Live-Interactive) workshops using Interactor and Image/ine, and 
the students (especially the ones with weak computer backgrounds) found 
learning both programs, and figuring out how to have them communicate with 
each other daunting at best. I wanted Isadora to take the qualities that 
made Interactor and Image/ine great, and put them together in one package 
that was easy to learn but still offered enough depth to satisfy "power" 
users. And, I wanted it to be more affordable to members of my community, 
which I consider to be choreographers, because of my involvement with 
Troika Ranch.

Scott: How does Isadora compare to Max, which is probably the most 
successful and widely used graphic programming environment for controlling 
and mapping data flow?

Mark: Isadora and Max both inherit the modules linked by the patch-cord 
metaphor from Hookup. But unlike Max, each Isadora module shows the 
parameter names and current values for all of its inputs and outputs, and 
many modules give real-time graphic feedback about their operation. This is 
important from the perspective of helping new users understand what's going 
on right away. But perhaps the biggest difference is that Max is a very 
powerful, open-ended programming language in which you could solve any 
number of problems. Isadora isn't that. It is a lot like Interactor in that 
each module is essentially a macro that accomplishes some specific 
function. This approach helps people who are just beginning to do this kind 
of work, as it means that useful functionality is already embodied for you 
and it's very easy to start doing things and getting interesting results 
quickly (like with Image/ine). Max allows the most flexibility, but may be 
somewhat more difficult to program because more things have to be built up 
from scratch. Isadora offers somewhat less flexibility, but is still 
open-ended enough for the user to imprint his or her aesthetic on the result.

Scott: You have told me that your most important Isadora user is yourself. 
How do you use Isadora in Troika Ranch's performance work and in particular 
in connection with the MidiDancer which is the sensor system you built to 
get data input from a moving body.

Mark: I first created MidiDancer in 1989 while I was still a student at 
CalArts. While it is now much smaller and more reliable, the basic 
functionality has not changed much since then. Basically, it is a set of up 
to eight sensors that measure the flexion of joints on a dancer's body. 
Thirty times a second this information is sent over a wireless, radio link 
and a receiver up to 150' away decodes the information, reporting the angle 
of each joint in the form of a MIDI (Musical Instrument Digital Interface) 
message. Any computer with a MIDI interface can accept and process these 
messages as desired.

The problem with MidiDancer is that, to really play it, and for the 
audience to see that the dancers are playing, you need to move like a 
musician. What I mean is that the movement of the dancer needs to be in 
service of the sound or image that they are generating or controlling. We 
have worked hard to find ways to make this work choreographically, but it 
is quite difficult to do. My basic instinct in putting the sensors on the 
"gross" joints (elbows, knees, hips, wrists) was correct, in that these 
angular changes can be clearly seen by the audience. But, I have really 
been seeing lately that this is not the gestalt that we perceive when we 
watch a dancer move. We really see energy -- that's a bit vague, but it's 
the best word I can think of to describe it. We're not looking at the 
individual angles of the joints, but the way that the dancer moves through 
space and the overall articulation of the movement.

Scott: Well, electronic musicians have been building sophisticated playable 
interfaces for a long time, but these tend towards either the 
hyper-instrument (extending an existing musical instruments capabilities) 
or a few unique hand orientated interfaces. But, I've always thought that 
one would need to think quite differently to develop an appropriate system 
for a trained mover or dancer. I think more research is needed with various 
sensor input devices and maybe not always towards the aim of live stage 
performance, but maybe just experimenting much longer with what might 
emerge from the kind of feedback conditions for the senses interactive 
systems can generate.

Mark: That's why we'll be using a residency we have next year to make some 
changes to the MidiDancer. I want to start working with accelerometers in 
addition to the flexion sensors. The act of turning, or stopping suddenly, 
or shaking the whole body, now becomes something that can be measured. My 
instinct is that using this information to manipulate the media will be a 
more natural linkage between what the audience sees in the dancer and the 
resulting sonic or visual manipulation. I can then use the position of the 
limbs to allow the performer to enhance their level of control -- but I 
suspect that being able to sense the impulse of movement may become the 
primary source of manipulation. I think that, not being a dancer or 
choreographer myself, I have been slow to let go of the notion of being a 
musician. In fact, I have often described the MidiDancer as allowing a 
dancer to be "both musician and dancer." I now think that is incorrect. I 
need a device that allows a dancer to be a dancer, with the media taking 
its cue from what it sees the dancer doing.

Scott: Isadora is just about finished with its public beta testing phase 
during which I know you have been working with several artists who have 
been trying out the software for different projects and giving you feedback 
and suggestions for additional functions, etc. I have invited some input 
from some of these individuals [Part II], but first I just wanted to ask 
you to say something about how long you have been working on Isadora and 
your decision to sell the software instead of using an Open Source approach 
(in which software code is released for free into a collaborative 
development environment).

Mark: As I think I've indicated Isadora has grown out of a need Troika 
Ranch had for a reliable relatively easy to use but also sophisticated 
software for both workshops as well as performance. The end result is that 
I have taken two plus years to develop Isadora in my spare time so it has 
grown quite organically. In regard to deciding to sell Isadora, I don't 
have much interest in starting a real business, so I am feeling out my 
progress quite slowly. But in the U.S., arts funding is very difficult to 
come by, and discovering ways to supplement what we can get from the 
government or foundations is essential. I have always had to hold a day 
job, so I wanted to see if I could make a tool that would be: 1) useful to 
me; 2) useful to others and; 3) perhaps generate enough income to help me 
spend more of my time making my artwork. Obviously, an Open Source model 
would not generate any income, and thus wouldn't help to support my 
artistic pursuits. As I have mentioned, it is important to me that the 
Isadora is affordable to those in the dance community, so I figure, for 
what the program does (and will do as it grows) the single license fee is a 
reasonable amount. I have yet to make a final determination about site 
licenses for schools etc, but they will definitely get a break if they 
purchase a 5-seat license for example. I have no idea at this point if this 
whole plan will work, and if the burden of supporting a growing user base 
will actually be more work than the day job, but I am hopeful.

Scott: Thanks Mark. Now, what I would like to do is to invite some comments 
from some of the artists who have been working with Isadora and are 
familiar with the history and evolution of artist software tools to reflect 
on some of our dialogue and to talk about how it is they use or may use 
Isadora in their work.

*****************************************************************************
See Part II on line at: http://huizen.dds.nl/~sdela/sfd/isadora2.html
*****************************************************************************

---------------------------------------------------==|
---------------------------------------------------==|
   Scott deLahunta
   Writing Research Associates, NL
   Sarphatipark 26-3, 1072 PB Amsterdam, NL
   mobile: +44 (0)797 741 2060 [voice messages too]
   fax: +44 (0)870 121 9311
   [there is no fax signal, begin faxing anytime]
   email: sdela {AT} ahk.nl
   www: http://huizen.dds.nl/~sdela/

---------------------------------------------------==|
---------------------------------------------------==| 

#  distributed via <nettime>: no commercial use without permission
#  <nettime> is a moderated mailing list for net criticism,
#  collaborative text filtering and cultural politics of the nets
#  more info: majordomo {AT} bbs.thing.net and "info nettime-l" in the msg body
#  archive: http://www.nettime.org contact: nettime {AT} bbs.thing.net