MOCKINGBIRD: A COMPOSER'S AMANUENSIS

      by

   John Turner Maxwell III and Severo M. Ornstein*

 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
 * The authors are both members of the Xerox PARC Computer Science
 Laboratory. At the time Mockingbird editor was constructed, Mr. Maxwell
 was completing his studies at the Massachusetts Institute of Technology.
 The Mockingbird editor, described herein is the subject of his MIT
 Master's thesis. Although the system design was a joint project, Mr. Maxwell
 did all the programming while Mr. Ornstein acted primarily in a
 supervisory/advisory capacity and designed the hardware
 interface to the synthesizer. The existence of this editor should not be
 construed as indicating any special interest on the part of Xerox PARC in
 music editors; specifically there is no "music laboratory" or "music project"
 within PARC.
 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -



     ABSTRACT

Mockingbird is a computer-based, display-oriented, music notation editor. It is especially focussed on helping the composer capture his ideas. It can accept both graphical input and input played on a synthesizer keyboard attached to the computer. It can play scores on the synthesizer as well as displaying and printing them in standard notational form. Played input must be edited to turn it into this form, and much of Mockingbird's interest lies in the methods by which this conversion is accomplished. The editor is highly interactive, presenting the illusion that the user can reach in and move things around as he desires. This illusion is supported by the fact that the detail of the score is always shown exactly as it might be printed.


INTRODUCTION


Mockingbird is a composer's amanuensis, a computer program designed to aid the composer with the capture, editing, and printing of musical ideas. The basic goal of Mockingbird is not to replace the composer, but to aid him by speeding up the process of notating music. Mockingbird is not a publisher's aid, although it does print music; nor is it a performer's aid, although it can play; it is strictly focussed on the needs of the composer.

Mockingbird is an interactive music notation editor. It knows nothing about the rhythmic, harmonic, or melodic aspects of music except insofar as they are represented in common music notation. To narrow the problem, we have concentrated on handling piano music; Mockingbird cannot presently handle orchestral scores or music for instruments that require their own notational devices.

It is somewhat surprising that no-one has previously built such an obviously interesting system. We believe that there are two principal reasons: first, we had at out disposal an unusually powerful set of hardware and software facilities, and second, we made a number of key decisions which allowed us to by-pass some extremely difficult problems.

The reader should remember that Mockingbird is only a "research prototype". Many features are still missing and in general we only did enough to show that our ideas were feasible.

Underlying Facilities

Mockingbird is a software package written in Mesa, an experimental language developed at Xerox PARC. Mockingbird runs on a general purpose computer called the Dorado, which is an extremely powerful, experimental, single-user machine also developed at PARC. The Dorado has a 60 nanosecond instruction cycle, a large memory (typically two to eight megabytes of RAM), and an 80 megabyte disk. It also includes a large, high-resolution bitmap-display, a keyboard, and a special graphical pointing device called a mouse. These features make the Dorado a particularly suitable tool for music editing. The only special hardware we provided was an interface to a Yamaha CP-30 electronic synthesizer in order to be able to use its keyboard to "play in" music and to allow the computer to "play-back" music without having to synthesize the sound waveforms by program.

In addition to these hardware resources, Mockingbird relies heavily on a general purpose graphics software package [1] which provides simple commands to display characters and draw both lines and curves. In addition it provides a common interface for both displaying material on the screen and printing a high-resolution hardcopy. ("A Device Independent Graphics Imaging for Use with Raster Devices" Warnock, J. and Wyatt, D. Computer Graphics VOL 16 # 3 July '82 pp. 313-320. (Sigraph '82)).

Key Ideas and Decisions

1. Amanuensis vs. Automatic Transcriber - We decided not to try to write a program that converted synthesizer keystrokes directly into a score. We made the decision for a number of reasons. First, we weren't sure that it could be done for the class of music we were interested in. Rather than pursue that question, we wanted to produce a tool that worked. Second, we knew that an editor would be needed anyway, both to correct mistakes and to satisfy the composer who did not want to use the synthesizer keyboard to enter material. So instead of a recognizer, we built an amanuensis or scribe, which provides a human transcriber with powerful editing tools. Our strategy was to build the editing tools first and then work on automatic heuristics to augment the editing process.

These tools can assist either in performing the conversion from played-input to score, or in entering scores graphically.

2. Data Structure - We believe that one of the most important decisions we made was the choice of data structures. Mockingbird treats music simply as a sequence of events. This allows us to handle simultaneously raw, ("played in") material and more finished ("structured") material. It is furthermore convenient for presenting the material in its various external manifestations - displaying, printing, and playing.

3. User Interface - In Mockingbird, rather than doing a lot of typing on the computer's keyboard, the user operates directly on the picture of the music that appears on the screen. To do this, we make heavy use of the mouse. Furthermore, there is a strong correlation between the internal representation of music, its visual display, and how it is played. All of the elements of the data structure are displayed, and everything displayed on the screen corresponds to some part of the data structure. If the user moves something on the screen, the data structure is immediately updated to reflect it; if the data structure is changed, the screen is immediately repainted. Not only is the picture faithful to the data structure, but so is the synthesizer "performance". For example, if the user puts a trill marking on a note, Mockingbird will trill it when it is played.

4. Voices - Music is broken "vertically" into separate parts or voices. Such partitioning is obvious in multi-instrument music but it is also present in some single-instrument (e.g. piano) scores as an essential structural feature. The recognition of this fact, and its explicit representation in Mockingbird, greatly facilitates the editing and formating of scores.

* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *

DESCRIPTION - OVERVIEW

Mockingbird is a software package that runs on a general purpose computer called the Dorado. The Dorado is an experimental computer developed at Xerox PARC which has a large raster display, a keyboard, and a special pointing device called the mouse. The mouse is moved around beside the keyboard on the table while a cursor appears on the display to indicate its position. When Mockingbird is run, a score appears on the display with staffs, notes, beams, and so on. The mouse is used to point at particular elements of the score. It has three buttons on the top for issuing commands (sometimes in conjunction with keyboard keys).

Optionally attached to the Dorado is a Yamaha CP-30 electronic synthesizer. The Dorado can sense key positions and simulate key strokes on the synthesizer. Thus it can be used to play and record the music that appears on the screen. It doesn't have to be connected to use the editor.

Mockingbird is designed to handle classical piano music notation. It knows how to display and play such things as notes, rests, accidentals, beams, chords, and ties. It knows how to display common music objects such as measure lines, time signatures, key signatures, and clef switches. It can also handle some of the more esoteric devices such as n-tuplets, ottava, trills, grace notes, and mordents.

<FOOTNOTE> There are a number of notational devices which we never got around to incorporating into Mockingbird. These include such things as rolled chords, staccato markings, fermata, and so on. Furthermore, although Mockingbird can handle ties, it can't handle the more general slurs. Nor can it display text such as lyrics or tempo markings.

THE EDITOR

Mockingbird consists of a number of functionally distinct parts integrated into one editor. The editor allows the user to record, edit, play, and print a single piece of music. Commands are issued by making selections and typing characters. Some commands are invoked by pointing the mouse at an object on the screen and clicking a button. When the user has finished with the piece of music, he can name it and file it away in the Dorado's filing system. Later it can be retrieved by name.

On the display there appears a score that looks much like a piece of sheet music. There are usually four to six staff sets (lines) each composed of two to four staffs. At the left of each staff there is a bass or treble clef sign with an appropriate key signature. Scattered over the staffs are notes, chords, beams, measure lines and other symbols commonly found in music.

Only a small section of the score can appear on the screen at a time, so there are commands that allow the user to look at different sections. "Scrolling" will cause the current section of the score to be moved up or down so that neighboring lines can appear. The user can scroll from a single line to an entire page. "Thumbing" allows the user to jump to an arbitrary point in the score. To thumb, the user specifies approximately how far into the score he would like to be, and the program moves the display to that point. Both thumbing and scrolling are accomplished by moving the cursor into a special "scroll bar" at the left of the score and clicking one of the mouse buttons.

The score can be edited with Mockingbird just as documents are edited with word processors. The usual paradigm for making an edit is to "select" some portion of the music and then issue a command. The command issued will only apply to the selected portion. The display is immediately updated to reflect the changes. If it is appropriate, several commands can be issued before a new selection is made.

SELECTIONS

There are two types of selections available to the user. He may either select a contiguous section of the score, or an arbitrary collection of individual notes. Both types of selections are made by moving the mouse over the desired objects while holding down a button.

Section selections are typically used for gross editing operations that apply to a section of the score. It is indicated by video reversing the section (black to white and white to black). The section selected may be as small as a portion of a measure or as large as the entire score. It encompasses all of the notation that appears on all of the staffs.

Note selections are typically used for operations that only apply to notes. They are indicated by painting the note heads grey. A collection of notes can be selected by sweeping the mouse over the note heads.

The user can also change the view of the score so that only one voice appears. The rest of the score is displayed in a very light grey for reference. Editing commands don't affect the part shown in grey.

Figure 1 illustrates what we mean by the term "voice". The notes that are in black all belong to the same voice, whereas the notes in grey belong to other voices. Note that a voice may contain chords as well as individual notes, and that a voice is not restricted to one staff. Also note that a voice does not always completely fill the rhythmic structure of a measure. There may be gaps.

Measure 3 shows why explicit voices are important. In order to know when to play a note, it is often necessary to know which voice it belongs to. (Vertical alignment with other notes is just a useful hint.) But voices are only implicitly defined in the score by clues such as beaming, stem direction and staffing. It is up to the musician to determine the voicing and then play accordingly. Although Mockingbird could try to figure out the voicing just as the musician does, we decided that it was better for the composer to have explicit control over it.

EDITING COMMANDS

Typical commands for editing the score include assigning note values, transposing notes, changing their spelling, or changing the staff on which they appear. It also includes commands that group notes together with beams, chords, or slurs. Many of these commands work with either type of selection. For instance, the user may transpose a single note down an octave or an entire voice up a fifth. The first action would be accomplished by selecting the note and issuing the command. The second would be accomplished by first changing the view to look at just the voice desired, then selecting the entire score, and finally issuing the command.

The user may also rearrange large sections of the score with ease. (This is sometimes called "cutting and pasting"). To replace one section of the score with another, the user first selects the section to be replaced (called the primary selection), then the section to be copied (called the secondary selection), and finally issues the replace command. The primary selection doesn't have to be the same size as the secondary selection. If there is no secondary selection, the resulting operation is a deletion. If the primary selection merely points at "empty space" in the score, the resulting operation is an insertion.

In addition to changing the structure of the music, the user can change the way it appears on the sheet. For instance, the number of staffs for each line can be changed on a line by line basis. The user can switch a staff's clef in the middle of a measure, or designate a section to be displayed in ottava notation. Key signatures and time signatures can be changed within the score wherever necessary.

The user can add new material to the score by picking up items from a pop-up menu. The menu appears under the cursor whenever a particular button of the mouse is held down. On the menu are found a note icon, a rest icon, measure lines, and various markings such as trills, accidentals, etc. As the cursor is moved over the menu its shape changes to correspond to the icon immediately underneath it. When the mouse button is released, the cursor retains the last shape. The user can then insert instances of that icon by pointing to a place in the score and clicking another mouse button. (Mockingbird automatically selects the last inserted note or rest for the user's convenience. This allows the user immediately to issue commands that affect the note).

In addition to the fundamental commands that allow the user to rearrange the score and change the structure of the music, there are also a number of fairly sophisticated commands that help the user produce a complete score. For instance, Mockingbird can check that all the measures add up to the correct metrical values. So if, for example, one of the voices were missing a rest, Mockingbird would complain by marking the measure with a stipple pattern on the display. Other such commands are discussed in the following sections.

JUSTIFICATION

A particularly useful command is the one that "justifies" sections of the score, such as that shown in Figure Xa. Justifying a section of the score involves several things: making the voices consistent relative to one another, laying out the graphical elements of the score in an aesthetic arrangement, and making sure that an integral number of measures falls on each line. It is only concerned with the horizontal placement of objects; things like the height and tilt of the beams are outside of its domain. It also doesn't perturb structural elements such as stem directions and staffing.

First of all, the justifier goes through each measure and makes sure that all of the voices are consistently ordered relative to one another. If they aren't, it moves the notes around until they are. Two voices would be inconsistent if separately they added up to the time signature but together they added up to more than the time signature. Figure Xb shows the result of this first step.

After the voices are made consistent, the justifier re-determines the horizontal placement of the graphical elements of the score. The horizontal distance is determined based on the types of elements (measure line, note, clef sign, accidental, etc.), the voicing, and the need to keep things from overlapping. The user can also give a parameter which determines about how "dense" the justification will be. The justifier then squeezes things together as close as possible based on these constraints. Figure Xc shows the result of this step.

Finally, the justifier stretchs out the spacing in the material to make an integral number of measures fit on each line. Figure Xd shows the result of this step. Figure Xe shows the process repeated with a different density parameter.

If the user doesn't like the results, he can manually move things around to improve its appearance. However, the justifier produces a surprisingly good layout for the section. This means that the user is freed from a lot of detail work. A common style of use is to enter music a voice at a time, not worrying about the spacing between the notes. When a line's worth has been entered, the user justifies it and goes on to the next line. The justification takes care of the properly aligning the voices and producing a suitable layout.

The justifier is also helpful in determining page layout and page breaks. The user can indicate that a particular measure must fall at the end of the line. The justifier takes this into account when deciding how many measures to put on the line. With this feature the user can control how many pages the score fills. He can also make sure that the end of the score falls at the end of a page.

THE SYNTHESIZER

As the user completes a section of the score, he can "proofread" it by playing it through the synthesizer. To do this, Mockingbird "reads" the score and plays it by simulating key strokes on the synthesizer. As the synthesizer plays the notes, a pointer tracks the performance on the displayed score. Mockingbird's rendition will correctly handle polyphonic music, taking into account such things as grace notes, n-tuplets, trills, ottava, and metrenome markings. Thus it gives the composer some ability to listen to what he has written. Although the performance sounds a little mechanical, it is sufficient to catch erroneous note values and pitches. The music can moreover be played at double speed for rapid scanning or half speed for careful analysis.

The synthesizer is also used to capture music played by the composer. As the user plays, Mockingbird observes the synthesizer keys going up and down. It records these in a "pianoroll" form, which specifies when every note was struck and when it was released. Music in this form may be displayed either as a direct time plot, as illustrated in Figure ***a, or in the "notehead" form illustrated in Figure ***b. (Mockingbird chooses default staffing and spelling.) Figure ***d shows the final score for this same section.

The composer isn't restricted to a single melodic line, nor must he follow a metrenome; he may play whatever he wants as freely as he wishes. He thereby captures his idea in rough form which, although a far cry from a standard score, nonetheless contains enough information to reconstruct his original intent. At this point, the user could go on to capture more music, or he could start transforming (by editing) the pianoroll into a score.

Raw pianorolls can be freely mixed in with standard music notation, both on a measure-by-measure basis and within a single measure as indicated in Fig. ***c. All of the commands that apply to standard music notation can also be applied to the pianoroll or mixed sections. Thus the composer can rearrange material, put notes into different voices, specify the durations of the notes, and add structural elements such as beams and chords. The ability to mix pianoroll and standard music notation gives the user a lot of freedom; he can work on the score in whatever order pleases him. Mockingbird will even play correctly across the boundaries of mixed sections of pianoroll and standard music notation.

Another feature of Mockingbird is that it is possible to "play against" previously entered material. While Mockingbird plays, the user can play along with it on the synthesizer. Both of the performances will be performed by the synthesizer. Mockingbird will record the notes and merge them in with what it is playing. This allows the composer to compose a piece one voice at a time, or to lay in a new melodic line over an existing score.

CONVERTING PIANOROLLS

Raw pianorolls are hard for humans to read; there are no chords or beams to group things, nor have the notes been separated into their proper voices. Because part of the process of composition includes specifying such syntactic structuring, it was necessary to go beyond pianoroll notation. For reasons mentioned earlier and discussed later on, the process of converting from pianoroll to standard music notation is left up to the user.

Although Mockingbird does not attempt to convert pianorolls automatically to standard music notation, it does provide a number of heuristics that aid the user with the transformation. When the user plays a pianoroll back, he can "beat in" measure lines by striking a key on the first beat of each measure. There is also a heuristic that runs through the pianoroll and aligns notes that are almost simultaneous. This tends to remove the inevitable imprecision in playing the notes of chords, and so on.

Typically, the next step is to assign notes to their respective voices. The user can assign an individual note to a voice by first selecting it and then indicating which voice it belongs to. Collections of notes can be voiced all at once by selecting them together and then issuing one voicing command. Similarly, notes can be moved to a staff different from their default assignment, if desired.

At this point, the user could go through and manually assign time values to the notes and designate chords and beams. However, Mockingbird has a number of heuristics to help him with these tasks. After the user assigns notes to their proper voices and gives a time signature, he may ask Mockingbird to guess the time values of the notes, group them into chords, and assign beams. Although the heuristics used are only about 80% accurate, they save the user a lot of work. In addition, what remains is easier to deal with since it is in a more familiar form. **** FIGURE? ***** Mistakes can be found by inspecting the score or listening to it through the synthesizer. They can then be quickly corrected. The combination of simple heuristics and easy editing is at the heart of the notion of an amanuensis.

By the time these steps are completed, the pianoroll has become a score. As such, it can be edited with the commands described above. Thus it can be justified, played back, and printed. More pianoroll could be entered and added to the score.

It is worth remembering that playing music on the synthesizer is not the only method of entering music into the editor. If the user does not want to use the synthesizer, he can always enter music directly with the mouse and menu. Since Mockingbird allows pianorolls and standard music notation to be mixed, the two methods can also be used together.

* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *

DATA STRUCTURES

This section describes the data structures that Mockingbird uses to represent a score. We felt that it was important to include this section because it took us a long time to arrive at a final design. In this discussion we will describe our design and explain why we think that it is the best one for our purposes.

Our first design was a structured, hierarchical data structure which closely matched the formal structure we saw in music. But we kept on running into problems with it because it didn't match the needs of an editor. After much discussion, we settled on an unstructured, sequential data structure. This surprised us, because at the beginning we had thought that the hiearchical design was the obvious choice. However, experience has convinced us that the sequential design is vastly superior.

There are three considerations in designing a data structure: representational power, programming convenience, and performance. Representational power has to do with how much of the domain is covered by the data structure, and how easy it is to represent different aspects of the domain in the structure. Programming convenience has to do with how easy it is to write algorithms that deal with the data structure. This depends a great deal on what the algorithms do. In Mockingbird we are mostly concerned with playing, editing, and displaying the score. (As opposed to structural analysis or automatic composition). Performance also has to do with the algorithms chosen. Even if a data structure is convenient, it may not be efficient. Sometimes there is a trade-off between structural complexity, space usage, and speed. In Mockingbird, space wasn't very important but speed was. You can't support the illusion of editable music unless the display reacts crisply to edits the user makes.

HIERARCHICAL VS. SEQUENTIAL

There are many possible "hierarchical" data structures that might be used to represent music. We use the term loosely to describe a class of data structures which implicitly incorporate musical structure in the design. Thus one might imagine a data structure that had a separate part for each measure, or that had a separate data structure for each voice. A "sequential" data structure, on the other hand, is simply a sequence of undifferentiated entities. No attempt is made to incorporate musical structure into the design. Instead, it is up to the algorithms to determine the structure from the entities.

On the surface the hierarchical design seems obviously better. If the musical structure is built into the data structure, then one can guarantee a uniform interpretation over all of the algorithms. Not only that, but algorithms won't have to derive the built in information.

Unfortunately, basing the data structure on the musical structure was too constraining. The first problem we ran into was that we wanted a uniform representation for both common music notation and pianorolls. We wanted the user to be able to mix both types of music freely. Although it might be possible to keep a separate data structure for pianorolls, it would make the algorithms for editing, displaying and playing much more difficult.

The second problem had to do with the fact that all of our algorithms for playing, displaying, editing and justification needed to know what things were near one another. A simple example is redisplaying the score after a small edit has been made. For efficiency one would like to redisplay as little of the score as possible. But that requires knowing what was near the entity that was edited. In the hierarchical design an entity that is close physically may be logically far away. It might be in another measure, in another voice, in a different chord, or on a different staff. Enumerating all of the possibilities is incovenient.

****A good example of this is the rule that says that all of the notes in a measure must add up to the time signature for that measure. Unfortunately, a musical phrase occasionally crosses a measure boundary. And if the phrase contains n-tuplets, it may temporarily violate the sanctity of the time signature (See Figure xxx). ****

The last problem was that we kept finding exceptions. Many of the rules of notation that are presumably inviolable turn out to be violated when the composer finds the notation too constraining. Figure xxx shows several examples of this. A design which had fixed rules about the structure of music built into it wouldn't be able to handle such exceptions. Even if the exceptions were ruled out, one would still have problems with the inconsistent structures that arise temporarily during editing. We wanted our design to be tolerant of such exceptions and inconsistencies.

A sequential design doesn't have these problems. It allows pianorolls to be mixed with standard music notation because they are both represented as an ordered sequence of notes. Finding things that are nearby is easy, because things that are near one another on the screen are near one another in the data structure. And finally, since the data structure is so unstructured, it is flexible enough to handle a wide range of exceptions.

A SEQUENTIAL DATA STRUCTURE

A "sequential" data structure is simply a sequence of events ordered by time. An event might be a measure line, a collection of notes, a time signature, a clef switch, or a switch in the number of staffs per line. The events which contain notes are called "syncs" because they synchronize all of the notes in the event. (That is, all of the notes in the sync are played or displayed together.) The notes may belong to different voices or chords, but they all have the same "time". The editor automatically syncs together notes that are very close to simultaneous whenever notes are entered or moved. Occasionally this will introduce an error, which can be fixed by the user.

Syncs are important because they keep simutaneous notes together while the score is being edited. Usually, if the composer plays several notes at the "same time", he wants them to stay together unless he explicitly says otherwise. Inserting a note before a sync shouldn't break up the sync, even if the one of the notes in the sync belongs to the same voice as the inserted note. If any of the notes in a sync move, they should all move. (The justifier sometimes violates this rule, but only when it is obvious that the notes have been incorrectly synced.)

There are three ways of measuring the "time" of an event: as seconds from the start of play, as beats from the start of the score, and as inches from the first measure line. Although these notions of time are very different, they co-exist nicely because the order of one is usually the order of the other. Thus if note A is displayed to the left of note B, it is most likely played before note B. In general we can therefore use the order of the notes as they appear on the display to determine the order in which they should be played. However, there are a few exceptions which must be handled properly. Embellishments such as trills and grace notes are not always played in the order that they are displayed. Conversely, notes that are logically simultaneous may be separated slightly on the display so that the note heads can be distinguished.

Beams and chords are handled separately from the rest of the data structure, since they are only a visual aid for the human performer and aren't otherwise crucial to the score. That is, beams and chords are just horizontal and vertical parenthesis. Their only function is to group notes together. If all the beams and the chords were removed from the score, it would affect its readability but not the way it is played. In Mockingbird, each beam and chord knows what notes belong to it, and each note knows what beam or chord it belongs to. In addition, chords have a stem direction and beams have a tilt and vertical position.

THE SHEET

So far we have been assuming that a score is a long sequence of measures that appeared on one line. But since music is printed on rectangular sheets of paper, this long line must be broken up into shorter lines. Rather than make our data structure more complex, we decided to use a separate data structure to map between the linear data structure and the two-dimensional piece of paper. This data structure keeps track of how long each line is, how many lines fit on each page, how much of the line must be devoted to a key signature, and which section of the score goes on which line. Only the displayer and the justifier need to make use of this representation of the sheet; all of the other algorithms can manipulate the simpler data structure directly.

* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *

DISCUSSION OF AUTOMATIC RECOGNITION

Our intention, in building Mockingbird, was to construct a useful tool for composers to help them in notating their ideas. The idea is not new. In 1958 Ornstein set out to construct a music "recognizer" using the TX-2 computer at the MIT Lincoln Laboratory. Its purpose, identical to that of Mockingbird, was to relieve the composer of as much as possible of the mundane work of writing down music. The (naive) idea at that time was to feed an audio recording of piano playing directly into the computer's analog-to-digital input and, with a clever program, produce a printed score. It didn't take long to come up against the very first problem: extracting the note occurrences from the sound waveform. But there were even more serious problems. To produce a proper score involved determining the time value of individual notes (which we soon realized was often totally uncorrelated with how long the key was held down). It also involved identifying complex n-tuplets, grace notes, and rests, determining voicing and rhythm and detecting rhythmic changes, identifying measure lines, determing how many staffs to use, which staff to put notes on, how to group the notes into chords, how to combine notes into beams, when to switch clefs or use octava, in short - determining ALL of the complex structural notational devices which composers use to help performers read scores.

As we studied more and more scores, we found that complexity often gave way to ambiguity - that decisions about notation were often a matter of personal taste. (See Fig - Chopin Etude)

POSTSCRIPT

When we started we never expected that we would be able to produce scores close to normal printed music. In this we were wrong and the result has been that large fraction of our energies has gone into creating "pretty" output (both on the screen and in the printed copy) and into various features that we initially considered frills. In dwelling on these later stages, we failed to pursue some of our early ideas.

For instance: we realized that if we succeeded in what we were trying to do, very soon a composer could be overwhelmed by a large number of sections of captured material. There would be a filing and retrieval problem and using text names to identify the bits and pieces seemed unlikely to be helpful. So we discussed the idea of keeping an index of short sections of the beginnings of each piece or segment. Music volumes containing multiple works often use this device to guide the reader, but in our proposed useage the number of entries would be very large. We felt that we might be able to do pattern matching searches of such an index in order to find for the user that entry which approximately matched some "key-tune". But we never got around to trying out this idea.



correct spelling



CONCLUSION

We hope that by presenting this material we will encourage others to pursue similar endeavors. Music editing is already being done on home computers and while it will be some years before machines as powerful as a Dorado are in every living room, soon useful tools will become feasible - even on a home machine of modest cost. It would seem that a display of reasonable resolution and a mouse are prerequisites. But if one eschews the temptation to make "pretty" scores and sticks to providing a simple "cut and paste" editor of piano roll material, then a reasonable composing tool should soon become possible. The storage and retrieval problem mentioned above would have to be addressed, but seems quite tractable.

ACKNOWLEDGEMENTS

Mockingbird was made possible by a fortuitous convergence of people, attitudes, and facilities. The environment at Xerox PARC in general and within the Computer Science Laboratory (CSL) in particular provided the necessary encouragement and support for this work. The existence of and access to a Dorado was absolutely essential and R.W. Taylor, head of CSL, provided us with this and all other necessary facilities. Will Crowther worked with us closely on the initial design and helped us to get started in the right directions. John Warnock and Doug Wyatt provided us with access to the Cedar Graphics software upon which Mockingbird is so dependent. Gene McDaniel wrote the special Dorado microcode for dealing with the Synthesizer and Mike Overton built the hardware interface. Last and most gratifying of all was the enthusiastic support which we received from our colleagues, whose vicarious pleasure in seeing Mockingbird come alive cheered us along the way.