An essential feature of a good integrated editor is uniformity. Uniformity means that the user learns a consistent set of methods for selecting a variety of visual objects, and a consistent set of commands, each of which will have a similar effect on the selected objects. This common interface hides from the user any implementation differences required to represent the behavior of the different visual media.

One example, the editor supplied with the Xerox 8010 (Star) workstation [15], will suffice to illustrate this concept. In Star, the same sequence of keystrokes and mouse-button clicks can be used to move a portion of text from one textual window to another, to move a geometric figure or a digitized photograph from one location to another within a graphics window, or to move an entire document from one `file folder' to another. Deletion of any of these objects can be accomplished by moving the selected object to an iconic representation of a waste basket.

Our basic approach, like that of most of the voice editing systems we have studied, is to extend the uniform editing concepts of an editor for visual media to encompass voice as well. But we believe it is equally important not to stretch this principle to its logical limit. Instead, we need to determine those concepts that cannot beneficially be applied uniformly to voice, and to add some specialized operations dealing with characteristics of voice that are unparalleled in other media. We believe the result has some important advantages over earlier integrated voice editors.

To insert a graphical illustration into a textual document, a common approach using an integrated editor is to create a graphics subwindow within the text, then to create and modify the illustration directly within that subwindow. Textual annotation of the illustration may be added by creating a further text subwindow within the graphics subwindow; and so on. This model is perfectly natural and understandable to the user, and is reasonably straightforward to implement.

Straightforward annotations to text can be made simply by recording single utterances from a microphone and marking their presence pictorially. Playback of complete annotations is equally straightforward. The time will come, however, when a user wishes to extend or modify a voice annotation, or when the amount of voice to be included exceeds what can be dictated all at once. What is needed is a means for editing voice.

Nearly all of the systems that support voice annotations also include voice editors. Most of these voice editors function by presenting the user with some form of visual representation of the voice, and operations on this representation that allow manipulation of the voice. Taking advantage of the static, spatial representation of what is intrinsically a temporal phenomenon, the user can access any part of an utterance, designate ranges, specify reorderings, etc., much more conveniently than would ever be possible using conventional telephone-based voice editing facilities. We have adopted this eminently sensible approach as well.

Before turning to a description of our voice editor, we mention one more philosophical point that underlies our design. Even the best possible graphical representation of a recorded utterance will fail to convey much of its content. Therefore, it is easy to lose your place when dealing with such representations. A good editor should supply numerous methods for marking significant places in the representation. Simple temporary markers are useful for keeping track of important boundaries during editing operations. Permanent markers can be used to find significant locations within extended transcriptions.

Figure 1 shows a text document window, or viewer, from a Cedar workstation screen. Its caption defines the various regions of the viewer, indicating how one selects objects of interest and how one performs various operations on those objects. We will use the terminology defined in Figure 1 throughout the discussion.

Any single text character within a Tioga document can be annotated with an audio recording of arbitrary length. To add an annotation, the user simply selects the desired character within a text viewer and buttons AddVoice in that viewer's menu. Recording begins immediately, using either a hands-free microphone or the telephone handset, and continues until the user buttons STOP. As recording begins, a distinctive iconic indication of the presence of a voice annotation is displayed as a sort of decoration of the selected character. Currently, this voice icon is an outline in the shape of a comic-strip dialog `balloon' surrounding the entire character. The second line of text in the first paragraph shown in Figure 1 contains a voice icon.

Adding a voice icon does not alter the layout of a document in any way. Thus, voice annotations can be used to comment on the content, format, or appearance of formatted text. Moreover, programs such as compilers can read the text, ignoring voice icons just as they ignore font information. Voice annotations may be used, for example, to explain portions of a program text without affecting the ability to compile it. Like font information, voice icons are copied along with the text they annotate when editing operations move or copy that text to other locations, either within the same document or from one document to another.

A voice annotation becomes a permanent part of a Tioga document. Copies of the document may be stored on shared file servers or sent directly to other users as electronic mail. To listen to voice, a user selects a region containing one or more voice icons and buttons PlayVoice. Since playback takes some time, the user may queue up additional PlayVoice requests during playback. These will then be played in sequence. The STOP button can be used to halt the whole process at any time.

The procedure outlined above is fine for short annotations, but for more complex annotations the user will need facilities to edit portions of voice. To keep things simple for rapid annotation, all that appeared in the text was an icon representing a complete voice utterance. To perform editing operations, the user selects a region of text and buttons EditVoice. A voice viewer opens up for each voice icon within the selection. Each of the selected voice icons at this point changes its appearance to that of an opened voice icon — it now displays a number that identifies the corresponding voice viewer. Each voice viewer is labelled with its number, so that the user can easily see the associations between voice viewers and positions in text viewers. Figure 2 shows what the voice icon of Figure 1 looks like when opened, along with the corresponding voice viewer.

We have already expressed our desire to steer the user toward phrase-level editing and away from editing at a finer grain. For this reason, we have avoided a type of display that many other designers have used [2, 8, 13, 18], which is a graphical profile of the sound energy plotted against time. Such a display encourages people to identify visually the individual consonants, phonemes, or words within voice. Our viewers look essentially like `capillary tubes' — long horizontal rectangular tubes filled in to indicate positions of sound and unshaded where there is enough silence to identify a phrase or sentence break. As with other voice editors, the length of a capillary tube corresponds linearly to the duration of sound that it represents. The appearance of the voice viewers is very similar to the graphical depiction used in the Sydis VoiceStation [9].

Although this format heightens the tendency to think in terms of phrase-level editing, it is sufficiently uninformative that users will not remember which capillary segment represents which portion of speech. When editing text one tends to focus in turn on several portions of text, then to locate them again to make a specific edit. We must be able to do the same with voice. This seems to be what has led other designers to the energy-graph approach, although it is only moderately successful as a means of providing context. We have chosen instead to supply a number of marking techniques that we think can provide this context more successfully, without the unwanted side-effect of encouraging the user to focus on an undesirable level of detail when editing.

How are editing operations accomplished? In Tioga, the user copies text from one position to another by making two selections with the mouse, then pressing a particular key on the keyboard. The uniform editing philosophy says that exactly the same actions should be used to copy voice from one position in a voice viewer to another. In fact, all of the standard Tioga text-editing operations are used also for editing voice — operations such as deleting, moving, copying or transposing portions of voice within or among voice viewers.

Some operations have analogous rather than identical meanings within voice and text viewers. `Selection granularities' are one example. We have referred several times to `making a selection'. Tioga functions are mainly achieved by selecting some region of text using the mouse, and then indicating a function to be applied to it. For convenient and fast editing, Tioga offers easy selection of different units or granularities of text — single characters, complete words, entire paragraphs, or entire sections. A selection, once made, can be extended to include contiguous units at the same granularity. Now in voice viewers our smallest units are not characters or words, but phrases or sentences. Therefore, we have set voice selection granularities to phrase-sized units or larger, so that the user is steered towards making edits at an appropriate level. A finer-level selection is available, providing resolution to within about a quarter of a second, but its use is discouraged for ordinary editing. (We remarked above that edits at the phoneme level, if attempted, invariably sound bad.)

Inserting new voice into a voice viewer is analogous to generating new text at a keyboard for conventional documents. The user selects a position within the voice representation, buttons AddVoice, then speaks into the microphone. While the new information is being recorded, a series of arrowhead symbols is inserted into the capillary tube beginning at the insertion point. New arrowhead symbols are added at the proper rate to indicate in terms of the time scale of the capillary display how long the speaker has been talking. Once STOP is invoked, the arrowheads are replaced by the normal capillary display (possibly enhanced visually using the techniques described in Section 5.4, below).

Finally, the user can save a set of voice edits, buttoning Save to preserve them as the new `contents' of the corresponding voice icon (if there was one), or Store to add the voice as an annotation at any selected position in a text viewer. The use of these buttons is analogous to their interpretation in text viewers to store edited text into permanent files. When the Store button is used, an open voice icon is created at the new position to represent the changed information. In this case, the voice annotation at the original text position (if any) retains its original value; its open icon reverts to the standard `balloon' form.

Useful lessons can be learned by examining how conventional dictation machines are used. Executives use the dictation machine as an input device because they can speak faster than they can write or type. Dictation is given to a secretary to type up and no attempt is made to play the whole thing back before then, simply because one can read faster than one can listen to voice. Dictations on conventional machines often include editing directives such as "in the first paragraph about artichokes, please change `would like you to' to `urgently require you to'", and so forth. The rather more manipulable approach to dictation set out in this paper should encourage more direct editing of the voice instead of such postfixed amendments.

For dictation applications, voice editing tools as described in this paper may eventually become obsolete. As voice recognition systems become more sophisticated, and in particular when they become generally applicable and affordable enough for inclusion in the personal workstation environment, we expect that the entry of voice, translated instantaneously into text for fast reading, will become a normal mode of use. Multimedia editors may well then become integrated to the extent that alterations to textual portions of a document may be done by vocal input.

Accurate translation of continuous speech into text is unlikely to be possible for some time. However, systems that instantly translate dictation into reasonably-accurate text that the user can then correct should be available much sooner. A secretary using such a system to transcribe other people's dictations would benefit from the ability to listen to the original voice while correcting the automatic translation. However, there are some uses of voice annotation that would not benefit from a translation to text. Inflection, emphasis, and the personal touch of a familiar voice are all part of the value of a spoken annotation.

While the incorporation of accurate speech recognition into a multi-media document system is still far from being a reality, the concept helps highlight the need for a more generalized document structure and editing interface than we have set out here. There are several reasons for wanting to extend document architectures beyond what has yet been accomplished.

For example, we have limited ourselves to vocal annotation of documents, with simple textual annotation of the voice but no further nesting of structure. There are three obvious directions in which we might extend these ideas. First, there is no reason to restrict annotation to vocal annotation. Just as a voice viewer can be `popped up' by selecting and manipulating a voice icon, a visual entity could be `popped up' and manipulated in an analogous way. This might be a textual annotation of a textual document, analogous to 'scribbling in the margin'. Annotations using computer animation and video are also possibilities. Secondly, there is no need to restrict the kinds of information that can be annotated. For example, voice annotation of illustrations and scanned objects could be as useful as annotated text. Extending in a different direction, it might be useful to apply annotations to a range of text or other objects, if a clear way could be found to indicate the presence of overlapping annotations to the user. Finally, there is no need to restrict annotation to a fixed number of levels. For example, an annotation hierarchy, with voice annotating the text that annotates a video commentary of a multi-media document, makes quite good sense. An on-line documentation system for computer systems is an application where a complex system of documents annotating one another could be used to help guide the user quickly to the desired information. The general model is that documents can be nested within other documents, either by direct inclusion or by annotation.

We have also considered higher-level applications that would use annotation as a component. We suggested in the introduction an advanced use of voice (and potentially video) annotation, in which a document would become an automatically-narrated audio-visual presentation. The document would contain timing and sequencing information to scroll specified areas into view on cue as the recorded narrative paced the presentation.

Most of the extensions suggested here could be accommodated within the framework of existing uniform user interfaces — the user would not have to learn large numbers of new concepts. We do expect, however, that each new medium will introduce the need for a few specialized concepts and operations, as we discovered in the present work. For instance, the behavior of a voice annotation still needs to be slightly different from that of other `pop-up' annotations, because there is a need for both a simple playback capability and the more heavyweight opening of a voice viewer to edit the voice. There is no direct parallel to these two separate requirements in the case of visual media.

As we indicated in Section 7, we consider it impractical to include digitized voice directly in annotated documents because of its size, choosing instead to store references to the voice. The inclusion of scanned images into documents introduces another medium that, like voice, involves a substantial amount of data that will be manipulated in large units. This suggests that such images might also be represented in documents by embedded references, to reduce the time and storage required to copy the documents. Real-time video annotations and cross-references among documents could also be implemented using this approach.