MOCKINGBIRD: A COMPOSER'S AMANUENSIS by John Turner Maxwell III and Severo M. Ornstein* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - * The authors are both members of the Xerox PARC Computer Science Laboratory. At the time Mockingbird editor was constructed, Mr. Maxwell was completing his studies at the Massachusetts Institute of Technology. The Mockingbird editor, described herein is the subject of his MIT Master's thesis. It grew out of deep, long-term, personal interests of the two authors. Although the system design was a joint project, Mr. Maxwell did all the programming. Mr. Ornstein acted primarily in a supervisory/advisory capacity and designed and built the hardware interface to the Synthesizer. The existence of this editor should not be construed as indicating any special interest on the part of Xerox PARC in music editors; specifically there is no "music laboratory" or "music project" within PARC. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - INTRODUCTION Our intention, in building Mockingbird, was to try to provide a useful tool for composers to help them in notating their ideas. The problem is that music is quite a complicated structure which is in motion as it passes through a composer's mind. The complexity of the structure can be measured by the fact that it can easily take an hour to write down a minute's worth of music. Complex pages go by rapidly as music is performed. So writing music, especially where the phrases are long and involved, requires an exceptional memory and often much is lost in the process. We discussed this problem with a number of composers, among them Aaron Copland, Virgil Thomson, Samuel Barber, Leonard Bernstein, Leo Ornstein, and Andrew Imbrie. We reached the conclusion that two things were needed: first, a mechanism for rapidly and flexibly capturing the music in an accessible form, (i.e. a form in which it could later be manipulated, modified, and massaged into a score); second, a powerful and convenient, interactive (graphical) music-notation editor for doing the manipulation and for printing the resulting score. What are the alternative methods of capturing musical ideas? Clearly we couldn't reach directly into the composer's head. We could get him to sing or play something and try to analyze the resulting sound waveforms, but even figuring out from the waveforms what notes are intended is a difficult problem. Based on our sample and on what we know of past composers we observed that, although not universally true, many serious composers are at least adept enough at keyboard playing that, with sufficient incentive, they can produce a rough rendition of what they want to put down. In fact much writing includes "trying something out" on a piano. We thus decided that a keyboard instrument connected to the computer would provide a suitably flexible means of capture. The composer must be able to play whatever he wishes in whatever way he wants. The computer simply captures every note depression, together with its time. The editor had to be able to accept such a record of note depressions as input and in addition, because modifications and additions would be required, it also had to allow the user to insert material using standard graphical facilities. This means that Mockingbird is also useful to a composer who does not want to "play in" his material. The editor is highly interactive, presenting the illusion that the user can reach in and move things around as he desires. This illusion is supported by the fact that the detail of the score is always shown exactly as it might be printed. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - HISTORY AND REFLECTION The idea of building a graphical music notation editor is hardly new. One of us set out to construct a music "recognizer" in 1958 using the TX-2 computer at the MIT Lincoln Laboratory. Its purpose, identical to that of Mockingbird, was to relieve the composer of as much as possible of the mundane work of writing down music. The ambitious idea of that period was to feed the audio of piano playing directly into the computer's analog-to-digital input and, with some clever programming, produce from this a printed score as output. It didn't take long to come up against the very first problem: determining which notes had been played - and when. This led to further study which convinced us that, although we could overcome that initial obstacle by using sensors on the individual keys, there were much more serious problems in disambiguating a collection of raw notestrokes. To produce a proper score involved determining the time of individual notes (which we soon realized was only vaguely connected to the length of time the key was held down), identifying complex n-tuplets, grace notes, and rests, determinings rythm and detecting rythm changes, identifying measure lines, determing how many staffs to use, which staff to put notes on, how to group the notes into chords, how to combine notes into beams, when to switch clefs or use octava, in short - determining ALL of the complex structural notational devices which have been developed over the years to help the reader parse music scores into comprehensible elements. As we studied the problem, poring over scores, we found that complexity often gave way to ambiguity - that many decisions about how to notate something ended up as a matter of personal taste. We thus concluded that if we were going to build a tool that a real contemporary composer might want to use, it would have to permit him to modify any and all decisions that the computer had made about what the composer intended. This led directly to a guiding design principal which we adopted in building Mockingbird: since the user would have to be able to modify all decisions the program made, we would FIRST construct a straightforward editor - and LATER incorporate any useful software aids which we could construict. We feel that steadfast adherence to this strategy was an important element in our success. But at that point the question still remained: would such an editor be of any use to a real composer. Furthermore, would a system which attempted to utilize a piano, or piano-like keyboard, for rapid input be useful to composers. In an attempt to answer such questions we spoke with a number of contemporary composers including Aaron Copland, Samuel Barber, Leonard Bernstein, Virgil Thomson, Leo Ornstein (the father of the author of this paper), and Andrew Imbrie. The conclusions were ambiguous; what we were proposing was so unfamiliar to these people that they had no way of making a sensible evaluation. For the most part they encouraged us and indicated that we were attacking what was, for them, a very real problem. Although much of their present writing takes place with pencil and paper, away from any instrument, they are all excellent pianists and would certainly be able to utilize a piano to enter their material if it could significantly speed up the notating process. Armed with this hope we decided to proceed. From the outset we felt that speed would be vital; the system would be uninteresting unless it was responsive to the user. This prompted our decisions concerning both what machine and what language to use. The fastest machine we had available was a Dorado and it seemed the obvious choice. We were tempted to use the Smalltalk language because of the ease with which one could experiment in building such a system. But in the end we were afraid of the speed issue and were concerned about the stability of the language upon which we would become totally dependent. We therefore decided to build it in Mesa, Xerox's burgeoning development language. In addition to greater speed, this would have the added benefit that if we succeeded, it might more easily be adopted someday as a candidate "product". When we started we were focussed on the capture of input and its conversion into roughly standard notation. We felt that we would be happy if we could handle simple pieces in a general fashion, but had only vague hopes of being able to deal with complex n-tuplets, etc. In particular we never anticipated that we would be able to represent the score graphically in such a close approximation to normal printed music. In general, things turned out to be even easier than we had expected ONCE A NUMBER OF IMPORTANT DESIGN DECISIONS WERE MADE. The result has been that a much larger fraction of our energies than we expected has gone into creating "pretty" output (both on the screen and in the printed output) and into various features that we initially considered frills.