INTRODUCTION
Mockingbird is a software package that runs on a general purpose computer called the Dorado. The Dorado is an experimental computer developed at Xerox PARC which has a large raster display, a keyboard, and a special pointing device called the mouse. The mouse is moved around beside the keyboard on the table while a cursor appears on the display to indicate its position. When Mockingbird is run, a score appears on the display with staffs, notes, beams, and so on. The mouse is used to point at particular elements of the score. It has three buttons on the top for issuing commands (sometimes in conjunction with keyboard keys).
Optionally attached to the Dorado is a Yamaha CP-30 electronic synthesizer. The Dorado can sense key positions and simulate key strokes on the synthesizer. Thus it can be used to play and record the music that appears on the screen. It doesn't have to be connected to use the editor.
THE EDITOR
The score that appears on the screen looks exactly like a piece of normal sheet music. There are usually four to six staff sets (lines) on the display each composed of two to four staffs. At the left of each staff there is a bass or treble clef sign with an appropriate key signature. Scattered over the staffs are notes, chords, beams, measure lines and other symbols commonly found in music.
Only a small amount of the score can appear on the screen at a time, so there are commands that allow the user to look at different sections. "Scrolling" will cause the current section of the score to be moved up or down some amount so that neighboring lines can appear. The user can scroll from a single line to an entire page. "Thumbing" allows the user to jump to an arbitrary point in the score. This is done in a way analogous to how people thumb through a book. The user specifies approximately how far into the score he would like to be, and the program modifies the display accordingly. Both thumbing and scrolling are accomplished by moving the cursor into a special "scroll bar" at the left of the score and clicking one of the mouse buttons.
Mockingbird has a notion of "voice" that is similar to the parts found in an orchestral score. If you think of the different "parts" that occur in piano music, it is clear that voices may contain chords as well as individual notes. Voices are not restricted to one staff, but may wander up and down as needed. The user can change the view of the score so that only one of the voices appears. The remainder of the score is displayed in a very light grey for reference, but isn't affected by any of the commands.
The score can be edited with Mockingbird in ways analogous to how documents are edited with word processors. The usual paradigm for making an edit is to "select" some portion of the music and then issue a command. The command issued will only apply to the selected section. The display is immediately updated to reflect the changes. If it is appropriate, several commands can be issued before a new selection is made.
There are two types of selections available to the user. He may either select a contiguous section of the score, or an arbitrary collection of individual notes. A selected section is indicated by video reversing the section (black to white and white to black). The section selected may be as small as a portion of a measure or as large as the entire score. It encompasses all of the notation that appears on all of the staffs. On the other hand, a note selection applies only to the particular notes that are selected. Note selection is indicated by painting the note heads grey.
EDITING
Typical commands for editing the score include beaming or chording a collection of notes together, making an n-tuplet, transposing notes, changing their spelling, or changing the staff on which they appear. Many of these commands work with either type of selection. For instance, the user may transpose a single note down an octave or an entire voice up a fifth. The first action would be accomplished by selecting the note and issuing the command. The second would be accomplished by first changing the view to look at just the voice desired, then selecting the entire score, and finally issuing the command.
The user may also rearrange large sections of the score with ease. To replace one section of the score with another, the user first selects the section to be replaced (called the primary selection), then the section to be copied (called the secondary selection), and finally issues the replace command. The primary selection doesn't have to be the same size as the secondary selection. This facility can be used to delete, insert, and duplicate arbitrary sections of the score in arbitrary places.
In addition to changing the structure of the music, the user can change the way it appears on the sheet. For instance, the number of staffs for each line can be changed on a line by line basis. The user can switch a staff's clef in the middle of a measure, or make a section octava. Key signatures and time signatures can be changed within the score wherever necessary.
The user can add new material to the score by picking up items from a pop-up menu. The menu appears under the cursor whenever a particular button of the mouse is held down. On it are found a note icon, a rest icon, measure lines, and various other markings such as trills, etc. As the cursor is moved over the menu its shape changes to correspond to the icon immediately under it. When the mouse button is released, the cursor retains the last shape. The user can then insert instances of that icon by pointing to a place in the score and clicking another mouse button. Notes and rests are selected after they are inserted for the user's convenience (this allows the user to immediately issue commands that affect the note).
In addition to the fundamental commands that allow the user to rearrange the score and change the structure of the music, there are also a number of fairly sophisticated commands that help the user produce a complete score. For instance, Mockingbird will check that all the measures add up to the correct metrical values. So if, for example, one of the voices is missing a rest, Mockingbird will complain by marking the measure with a stipple pattern on the display.
JUSTIFICATION
A particularly useful command is the one that "justifies" sections of the score. Justifying a section of the score involves several things. First of all, the justifier goes through each measure and makes sure that all of the voices are consistently aligned with one another. If they aren't, it moves the notes around until they are. Two voices would be inconsistent if separately they added up to the time signature but together they added up to more than the time signature. This might happen if voice A consisted of quarter notes and voice B half notes and the half notes of voice B were aligned with the second and fourth quarter notes of voice A rather than the first and third notes. In this case, the half notes would be moved.
After the voices are made consistent, the justifier re-determines the horizontal distances between the different elements of the score. The horizontal distance is determined based on the types of the elements (whether measure line, whole note, or quarter note), the voicing, and how things touch each other. The user can also give a parameter which determines about how "tight" the justification will be. The justifier the squeezes things together as close as possible based on these constraints.
After this, the justifier decides how many measures will fit on each line. If there is room left over, the measures are stretched to fill the space. The result is that an integral number of measures appear on each line.
If the user doesn't like the results, he can edit the score to improve its appearance. However, the justifier produces a surprisingly good layout for the section. This means that the user is freed from a lot of detail work. A common style of use is to enter music a voice at a time, not worrying about the spacing between the notes. When a line's worth has been entered, the user justifies it and starts again. The justification takes care of the properly aligning the voices and producing a suitable layout. All the user has to concern himself with is the structure of the music (and the height and tilt of the beams, which are outside of the domain of the justifier).
The justifier is also helpful in determining page layout and page breaks. The user can indicate that a particular measure must fall at the end of the line. The justifier takes this into account when it decides how many measures to put on the line. There is also a density parameter to the justification command which gives the user a little control over how many measures will appear on each line. With these two features the user can control where the score ends and how many pages it fills.
THE SYNTHESIZER
As the user completes sections of the score, he can "proofread" them by playing them through the optional synthesizer. To do this, Mockingbird "reads" the score and passes key strokes to the synthesizer. As the synthesizer plays the notes, the cursor tracks the performance on the display. Mockingbird's rendition will correctly take into account grace notes, n-tuplets, trills, polyphonic passages, and changes in the metrenome. Although the music produced sounds a little mechanical, it is sufficient to catch erroneous note values and pitches. It also gives the composer feedback on how his composition is going. The music can be played at double speed for rapid scanning or half speed for careful analysis.
The synthesizer can also be used to capture music played by the composer. As the user plays, Mockingbird observes the synthesizer keys going up and down. It records these in a "pianoroll" form, capturing when notes were struck and when they were released. The composer isn't restricted to a single melodic line, nor must he follow a metrenome. He may play as freely as he wishes.
Once the user has entered some music through the synthesizer, he has captured his idea in a form that cannot be lost. Although pianorolls are a far cry from standard scores, they contain enough information to reconstruct the composer's original intent. At this point, the user could go on to capture more music, or he could start transforming (by editing) the pianoroll into a score.
The pianoroll that is produced can be freely mixed in with standard music notation, or it may be kept separately. All of the commands that apply to standard music notation can also be applied to the pianoroll or mixed sections. Thus the composer can rearrange sections of the pianoroll, put notes into different voices, specify the durations of the notes, and add structural elements such as beams and chords. He may also have the synthesizer perform the pianoroll version.
The ability to mix pianorolls and standard music notation gives the user a lot of freedom. He can work on the score in whatever order pleases him. Mockingbird will even play correctly across the boundaries of mixed sections of pianoroll and standard music notation.
CONVERTING PIANOROLLS
Raw pianorolls are hard to read. There are no chords or beams to group things, nor have the notes been separated into their proper voices. The time axis of a pianoroll is very different from the time axis of standard music notation. Pianorolls measure durations in seconds (displayed as inches), whereas the notation measures durations in beats (an abstraction). The mapping between the two isn't simple since the performer may have altered passages for aesthetic reasons. For all these reasons it is very difficult to automatically determine the proper notational form for a pianoroll. This is left to the user to do manually.
Although Mockingbird does not attempt to automatically convert pianorolls to standard music notation, it does provide a number of heuristics that aid the user with the transformation. When the user plays a pianoroll back, he can "beat in" measures by striking a key on the first beat. There is also a command that runs through the pianoroll and aligns notes that are within a certain distance of each other. This is useful when the notes of a chord were not all struck together.
After the user assigned notes to their proper voices and gives a time signature, he may ask Mockingbird to make its best guess as to what the values of the notes will be. Mockingbird will then use a simple heuristic to scan the measures one by one and assign note values for each voice. Although this heuristic is only about 80% accurate, it saves the user a lot of work. It also transforms the pianoroll into a more familiar form.
Many of the heuristic's mistakes can be found by inspecting the score or listening to it through the synthesizer. As mistakes are found, they can be quickly corrected. The combination of simple heuristics and easy editing helps the user rapidly enter a score.
It is worth remembering that playing music on the synthesizer is not the only means of entering music into the editor. If the user cannot play a piano or is copying music from an existing score, he can always enter music directly with the mouse and menu. Since Mockingbird allows pianorolls and standard music notation to be mixed, the two methods can also be used together.