Gaining General Acceptance for UIMSs Brad A. Myers Dynamic Graphics Project Computer Systems Research Institute University of Toronto Myers@Toronto.edu Abstract User Interface Management Systems (UIMSs) have now gained acceptance in the research and business communities. Unfortunately, they are still not widely available or used. This paper proposes three reasons for this: that they are too hard to use, that they are still too limited in the types of interfaces that they can create, and that they are not portable with respect to different machines, operating systems and graphics systems. UIMSs can be made easier to use by avoiding programming-language-like techniques, and they can be made more functional by providing the ability to create Direct Manipulation interaction techniques and to integrate them into modern interfaces. UIMSs can be made more portable by inventing new input models and graphics packages that are appropriate for highly-interactive interfaces and that can work in various environments. Introduction User Interface Management Systems (UIMSs) [17,19,24] have gained general acceptance in the research and business communities. This can be seen by the large number of papers on UIMSs that appear in the SIGGRAPH and SIGCHI conferences and the rising number of UIMS products. Unfortunately, most user interfaces are still created by hand without the use of appropriate tools. Why is this? This paper proposes three unsolved problems with current UIMSs which, I believe, strongly contribute to their lack of popularity. The first is that current UIMSs are very hard to use and often require the user interface (UI) designer to learn a special purpose programming-like language. The second problem is that UIMSs typically are fairly limited in the types of interfaces that they can create. The third problem is that most UIMSs are not portable. One avowed goal of UIMSs is to allow professional UI designers who are not programmers to create user interfaces. Most UIMSs, however, use a textual specification with a rigid syntax that is much like a programming language. This problem might be addressed by using techniques such as those found in easy-to-use Direct Manipulation [21] interfaces in new UIMSs. Unfortunately, it is hardest to make UIMSs that can create the modern, Direct Manipulation style of interfaces that are popular for most new systems. For example, more work needs to be done so that UIMSs can be used to create the interaction techniques used in Direct Manipulation interfaces. In addition, it must be made easier for the user interface designer to control multiple interaction techniques that are available and operating at the same time. Another important problem is that UIMSs typically only work on one particular machine with one particular operating system and with one type of graphics package. This is typically because there is no standard graphics package that provides the necessary primitives needed by UIMSs and because existing device independent graphics packages are too slow for use in Direct Manipulation interfaces. Definitions There does not seem to be any general agreement about what should be called a User Interface Management System, beyond a general feeling that it is something that helps build user interfaces. The model for UIMSs presented by Tanner and Buxton [23] shown in Figure 1 is an important start and shows where various parts fit into the general picture. For example, the graphics supplied with most window systems and the Macintosh Toolbox [1] can be classified as the module library by itself. Is this enough to qualify as a UIMS? Does it matter? .sp 2.625i Figure 1. Model for User Interface Management Systems (from [23]). Making UIMSs easier to use UIMSs can also be classified on a different axis by how they allow the designer to specify the interfaces (see Figure 2). This is closely tied to how easy the system is to use since non-programmer UI designers will have to learn programming techniques (if not actual programming languages) for the systems towards the left of the axis. .sp 1.5i Figure 2. A classification of systems for creating user interfaces by how much programming skills the UI designer needs to know to use them (and therefore how hard they are to use for non-programmers). Example systems are shown below the line. Some systems provide a library and require the designer to program in a conventional language (as in typical window managers). This would usually not be classified as a UIMS. However, some other systems provide some sort of framework that directs and helps the designer in this programming. These systems are often classified as UIMSs even though conventional, general-purpose programming languages are used. Examples of this are object-oriented systems like GWUIMS [22] and Apple's MacApp [20] (which creates Macintosh programs). These provide standard objects for handling user interfaces which the designer modifies to create a particular user interface. The next class of UIMSs invents a language of its own that the designer must learn in order to create the user interface. This language may be very abstract such as BNFs [16], or it may be more ``direct'' and declarative as in COUSIN [8]. The next class has tools that allow the designer to present a graphical description of the desired interface. These range from very program-like (e.g. Transition Diagrams [11]) to very demonstrational (e.g. Peridot [13] which uses Direct Manipulation techniques to create Direct Manipulation interfaces). The assumption is that systems to the right of the axis will be easiest for UI designers to use. Using UIMSs to create direct manipulation interfaces. A problem with most available UIMSs is that they do not address Direct Manipulation (DM) interfaces. These interfaces are characterized by: (a) a general lack of syntax, (b) multiple interaction techniques that are available at the same time (e.g. a number of menus and the user can choose which one to use), (c) a reliance on novel and rapid use of non-keyboard input devices (e.g. a mouse), (d) semantic feedback inside inner loops of interaction techniques, and (e) elaborate graphics. UIMSs which support Direct Manipulation interfaces must deal with the following issues, some of which are elaborated on in this section: · support for different types of interaction techniques, · the ability to create new and different types of interaction techniques, · allowing Direct Manipulation methods to be used for specifying those portions of the interface that can best be created graphically (such as the placement of interaction techniques), · using an appropriate, easy-to-use language for programming the parts that cannot be easily specified graphically, · allowing multiple interaction techniques to be available at the same time, · allowing multiple interaction techniques to be operating at the same time, · providing semantic feedback, semantic error checking and semantic defaults that work at a sufficient speed, and · having an appropriate mechanism for attaching to application programs. Some UIMSs support DM interfaces by providing a hand-coded library of interaction techniques and just allowing the designer to ``glue'' them together (see Figure 1). Some of these require the designer to use a textual language to specify where the interaction techniques are positioned on the screenCOUSIN [8], and Apollo's Dialoguewhereas others, such as Menulay [4], Trillium [9] and GRINS [18], allow this to be done graphically. Another approach is taken by the Peridot UIMS [13]. Peridot allows the designer to create the interaction techniques themselves using examples to demonstrate what the screen picture should look like and what the end user should do to perform the interaction (see Figure 3). The system attempts to infer what the general case is that covers the specific examples using some simple rule-based Artificial Intelligence techniques. Peridot was inspired by work in the areas of Visual Programming and Programming by Example [12]. Examples of other interesting recent systems in these areas include Editing-by-Example [15], Graphical Thinglab [3], Rehearsal World [6], and SmallStar [7]. .sp 3.0i Figure 3. Creating a scroll bar using Peridot. In (a), the background graphics have been created. The grey bar will represent percent of file visible in the window. The two extremes of the full file (b) and none of the file (c) are demonstrated. This will depend on the active value ScrollPercent which ranges from 100 to 0 (d). Next, the two extremes of seeing the end of the file (d) and the beginning of the file (e) are demonstrated. The active value WhereInFile (which varies from the value of the parameter CharsInFile to 1) controls this (f). The designer then demonstrates (f) that the bar should follow the mouse when the middle button is down using the ``simulated mouse.'' From [13]. Work is also needed to provide appropriate mechanisms to handle multiple interaction techniques that are available at the same time (so the user can choose which one is desired), and also multiple interactions that are operating at the same time (such as different devices under each hand). Having multiple interactions available is a cornerstone of mode-free interfaces, and using multiple devices in parallel has been shown to increase users' efficiency and satisfaction with systems in some experiments [5]. This concurrency problem is compounded by the fact that the application program must be structured so that it can deal with the parallelism. This might require research into appropriate operating system structures for the kind of parallelism needed by user interfaces. Hill's Sassafras system [10], which uses a production-like system, is one experiment in this area, and another is Peridot, which uses ``active values'' [13] which are like variables except that dependent graphics are updated immediately when the associated values change. A related issue that is not usually handled by UIMSs is to support semantic feedback [14], which is when the application is involved in feedback during mouse movement and inside other ``inner loops'' of interactions. For example, an object being dragged might jump to legal points (this is often called ``gravity'' and an advanced form is ``Snap-Dragging'' [2]). These types of feedback are sufficiently complicated that the UIMS cannot provide them locally without access to the application's data structures. This is closely related to issues of semantic error checking and semantic default values which are not usually handled by existing UIMSs. Overall, this can be classified as the desire for fine-grain control whereas most UIMSs only allow coarse-grain control. Constraints as used in GRINS [18] and the active values of Peridot [13] will help alleviate this problem, but an important efficiency question remains: can the UIMS provide for semantic feedback that is fast enough to keep up with the mouse and other input devices? Making UIMSs more portable Almost every UIMS that exists today is tightly tied to a particular operating system, computer, and graphics package. This is because there are no appropriate high level interfaces that can shield the UIMS from having to directly manage the various input devices such as the mouse and keyboard and from directly creating pictures on the screen. To make UIMSs be more portable, we must deal with the following issues, some of which are elaborated on in this section: · creating a more powerful, abstract, and appropriate input model for handling the various input devices, · creating an appropriate output model for graphics (probably by using or modifying an existing standard), · getting these two models incorporated into a standard (either de facto or de jure), and · creating and possibly standardizing a window manager interface that incorporates or supports these models. One important reason for the lack of portability is that none of the existing or proposed graphics standards (CORE, GKS, PHIGS, CGI/VDI, etc.) are appropriate for creating Direct Manipulation interfaces. Most are too slow to get an appropriate level of feedback, and too large to fit easily into many computers, but more importantly, the input model in all standards is essentially the same and is widely perceived as wrong for highly-interactive interfaces. Unfortunately, there seems to be no good alternative model, so current UIMSs must directly manage the mouse to provide Direct Manipulation interfaces. The UIMS community should be investigating new models for input devices so a graphics package can be created that provides a higher level of abstraction. Obviously, an appropriate model for output graphics must also be provided, but this seems easier to generate. The most difficult problem here is providing a high enough level of abstraction along with sufficient speed, but the appropriate general principles seem to exist. Along with the input and output models, a standard interface to a window manager will probably be needed for portability. The current effort at creating a Window Manager standard (X3H3.6) will probably help with this, but it will also need new input and output models. In addition, it would benefit from the advice of the UIMS community since they will probably be affecting the user interfaces that can be created. Conclusions User Interface Management Systems have arrived. There are a fairly large number of them available and a number of products have been created using them. The UIMS community now has three important goals: to make UIMSs easier to use by designers, to increase the functionality of UIMSs so that they can create the most popular Direct Manipulation styles of interfaces, and to allow them to be more widely available. To achieve the first two goals, more work needs to be done on graphical specification techniques and parallelism. The third goal requires that more work be done on making UIMSs portable which includes defining appropriate input models for highly interactive interfaces. These efforts will hopefully allow User Interface Management Systems to be as widespread and essential for creating user interfaces as parser-generators are for creating compilers. Acknowledgements I want to thank Xerox Canada, Inc. for the donation of the Xerox workstations and Interlisp environment which supported the work on Peridot. This research was also partially funded by the National Science and Engineering Research Council (NSERC) of Canada. I would also like to thank SIGGRAPH for funding my attendance at the UIMS conference. For help and support with this paper, I would like to especially thank my advisor, Bill Buxton, and also Bernita Myers, Peter Rowley, Ralph Hill, and Ron Baecker. References [1] Apple Computer, Inc. Inside Macintosh. Addison-Wesley, 1985. [2] Bier E. and Stone, M. Snap-Dragging. Proceedings of SIGGRAPH'86 (Dallas, Texas, August 1822, 1986). In Computer Graphics 20, 4 (August 1986), 233240. [3] Borning, A. Defining Constraints Graphically. In Proceedings CHI'86 Conference on Human Factors in Computing Systems (Boston, April 1317, 1986), ACM, New York, 137143. [4] Buxton, W., Lamb, M.R., Sherman, D. and Smith, K.C. Towards a Comprehensive User Interface Management System, Proceedings of SIGGRAPH'83 (Detroit, Mich., July 2529, 1983). In Computer Graphics 17, 3 (July 1983), 3542. [5] Buxton, W. and Myers B.A. A study in two-handed input. In Proceedings CHI'86 Conference on Human Factors in Computing Systems (Boston, April 1317, 1986), ACM, New York, 321-326. [6] Gould, L. and Finzer, W. Programming by Rehearsal, Byte 9, 6 (June 1984). Also Technical Report SCL841, Xerox Palo Alto Research Center, May, 1984. 133 pages. [7] Halbert, D.C. Programming by Example. PhD thesis. University of California, Berkeley. 1984. Also Technical Report OSDT8402. Xerox Office Systems Division, Systems Development Department, December, 1984. 83 pages. [8] Hayes, P.J., Szekely, P.A. and Lerner R.A. Design Alternatives for User Interface Management Systems Based on Experience with COUSIN. In Proceedings CHI'85 Human Factors in Computing Systems (San Francisco, April 1418, 1985), ACM, New York, 169175. [9] Henderson, D.A. Jr. The Trillium User Interface Design Environment. In Proceedings CHI'86 Human Factors in Computing Systems (Boston, April 1317, 1986), ACM, New York, 221227. [10] Hill, R.D. Supporting concurrency, communication and synchronization in human-computer interaction the Sassafras UIMS. To appear in ACM Transactions on Graphics, 1987. [11] Jacob, R.J.K. A State Transition Diagram Language for Visual Programming, IEEE Computer 18, 8 (August 1985), 5159. [12] Myers, B.A. Visual Programming, Programming by Example, and Program Visualization; A Taxonomy. In Proceedings CHI'86 Conference on Human Factors in Computing Systems (Boston, April 1317, 1986), ACM, New York, 5966. [13] Myers, B.A. and Buxton, W. Creating Highly Interactive and Graphical User Interfaces by Demonstration, Proceedings of SIGGRAPH'86 (Dallas, Texas, August 1822, 1986). In Computer Graphics 20, 4 (August 1986), 249258. [13] Myers, B.A. Creating dynamic interaction techniques by demonstration. To appear in Proceedings of CHI+GI'87 (Toronto, April 59, 1987), ACM, New York. [14] Myers, B.A. The issue of feedback. In preparation. [15] Nix, R.P. Editing by Example. ACM Transactions on Programming Languages and Systems 7, 4. (October 1985), 600621. [16] Olsen, D.R. Jr. and Dempsey, E. SYNGRAPH: A Graphical User Interface Generator, Proceedings of SIGGRAPH'83 (Detroit, Mich., July 2529, 1983). In Computer Graphics 17, 3 (July 1983), 4350. [17] Olsen, D., Buxton, W., Ehrich, R., Kasik, D., Rhyne, J. and Sibert, J. A Context for User Interface Management, IEEE Computer Graphics and Applications 4, 12, (Dec 1984), 3342. [18] Olsen, D.R., Dempsey E.P. and Rogge R. Input-Output Linkage in a User Interface Management System, Proceedings of SIGGRAPH'85 (San Francisco, Calif., July 2226, 1985). In Computer Graphics 19, 3 (July 1985), 191197. [19] Pfaff, G.E., ed. User Interface Management Systems, (Proceedings of the IFIP/EG Workshop on User Interface Management Systems, Seeheim, Federal Republic of Germany, October 1983), Springer-Verlag, 1985. [20] Schmucker, K.J. MacApp: An Application Framework, Byte, (August 1986), 189192. [21] Shneiderman, B. Direct manipulation: a step beyond programming languages, IEEE Computer 16, 8 (1983), 5769. [22] Sibert, J.L., Hurley, W.D. and Bleser, T.W. An Object-Oriented User Interface Management System, Proceedings of SIGGRAPH'86 (Dallas, Texas, August 1822, 1986). In Computer Graphics 20, 4 (August 1986), 259268. [23] Tanner, P.P. and Buxton, W.A.S. Some Issues in Future User Interface Management System (UIMS) Development. In User Interface Management Systems, Pfaff G.R. (ed.), Springer-Verlag, Berlin, 1985, 6779. [24] Thomas, J.J. and Hamlin, G. Graphical input interaction technique (GIIT) workshop summary, Computer Graphics 17, 1 (January 1983), 530. B14Myers.tioga Rick Beach, February 13, 1987 8:58:56 pm PST Ê<–"computergraphics" style˜•MarkcenterHeader™ J–centerFooter™,—Ititle˜$Iauthors˜rhead˜Ibody˜Þ—˜N˜ÎN˜ëN˜ÈN˜‹—˜˜¡Idisplay˜ OšÏb5˜5Oš˜——˜˜ÐO˜Ošô˜ô—Nšœ´Ïiœå˜œ —˜5N˜é˜ˆIbulletšœ7˜8PšœI˜JPšœ·˜¸Pšœr˜sPšœ0ž œ˜LPšœ0ž œ˜LPšœp˜qPšœG˜H—N˜³˜¬Ošœ˜Oš†Ðbi¬˜½—Nšœiž œiž œ»˜ŸNš œCžœ=žœ¨ž œ'žœ”˜—˜N˜Ù˜xPšœh˜iPšœi˜jPšœ?žœžœ˜YPšœk˜l—N˜þN˜‘N˜¡—˜N˜å—˜N˜ü—šœ ˜ I referencešœžœ˜@Qšœlžœžœ˜›Qšœ5žCœ5˜Qšœ´žœžœ˜ßQšœ@žCœ5˜¸Qšœ7žœg˜¤QšœÚ˜ÚQšœž5œ<˜þQšœKž5œ5˜µQšœ‹žœ˜®QšœOžœ˜xQšœgžCœ3˜ÝQšœ¯žœžœ˜ÞQšœXžœ+˜›Qšœ7˜7Qšœ#ž5œžœ˜wQšœ—žœžœ˜ÂQšœuž)œ˜¶Qšœ±žœžœ˜ÞQšœž!œ˜˜ÏQšœ7žœ˜TQšœOžœ˜qQšœ©žœžœ˜ØQšœsž!œ9˜ÍQšœ`žœžœ˜——…—J N