Formative evaluation is a key methodological process by means of which the current conceptual view of the target system is subjected to criticism and analysis to determine needs, inconsistencies, weaknesses and other inadequacies. It also involves monitoring the process and products of system development and gathering enduser feedback for use in refinement and further system development. Summative evaluation is the counterpart process of examining enduser behavior after the target system has been designed and implemented. For a discussion of the differences between formative and summative evaluation, together with some illustrative examples, see [11].

At the requirements acquisition and conceptual design stage the primary concern is still focused on the determination of a consistent framework that accounts for the system requirements. As this framework evolves, additional questions will be raised about the goals and requirements of the target system until the point is finally reached where design can begin and the UIMS tools can take on an appropriate embodiment.

Members of development teams are becoming increasingly sophisticated in their understanding of issues in ``consistency.'' The everyday notion is becoming refined as considerations of consistent technology, appearance and enduser processes are included in trade-off design decisions. In order to allow for evolution of tool power with increasing innovation in technology and in interaction techniques, it is clear that ``identical'' interfaces are unduly restrictive. What is needed is a consistency that preserves those properties of the enduser interface that support transfer of already acquired factual and process knowledge (about the application, about the use of tools in general). Consistency within the interface enhances both learnability and usability by reducing the frequency of unexpected events that require additional enduser attention. People are willing to acquire new patterns, but only when the advantages provided clearly exceed the cost of learning.

A simple example can illustrate the role of consistency in system specification. Imagine a dialogue specification for move in the sequence: object selection, command selection, target selection. Further imagine a dialogue specification for copy with the sequence: command selection, target selection, object selection.

Functional requirements are met, but conformance with enduser expectations arising from the conceptual similarity of the two commands are not. Now, no team would make such an obvious mistake, but less obvious mistakes are common, even in well designed interfaces. There are several dimensions of consistency:

There are doubtless other kinds of consistency not mentioned here. The issues which must be addressed are the impact of each kind of consistency and the interactions among them [1].

Integration of interfaces is another notion that is frequently used but badly understood. It is useful to modularize function in applications, as this permits the endusers to compose functional modules in ways best suited to their task requirements. This is especially true when a system must support many endusers with diverse needs, or when task requirements are continually changing. Relying on endusers to do their own, high level application development may be the only way to avoid the omnipresent application development backlog. The interface should permit endusers to apply their general knowledge about interaction techniques to all modules. It should permit the enduser to move effortlessly from one module to another (flow of control), and to move data effortlessly from one module to another (flow of data). There is a great deal of current focus on rudimentary characteristics of integration, such as whether modules may execute concurrently, how they communicate and share data, whether multiple interfaces (windows) to the enduser are required. It seems clear that such characteristics will be possessed by future systems, and that our ability to build suitable enduser interfaces will be limited by our understanding of enduser problem solving behavior.

Part of understanding the function of the target system involves understanding the needs of the endusers (the designer of the interface, the person using an application through the interface). Given these needs (which might involve learnability, speed of interaction, error rates, expertise level, physical capabilities or environment), there are common performance baselines that have to be considered in the system formulation in order to lead to good designs later on. For example, if there are requirements for very rapid interaction with low error rates, one might begin to formulate ways to make use of direct manipulation concepts. Direct manipulation allows immediate feedback on many kinds of errors so that they are seen by the enduser and fixed immediately.

The major obstacle in this area is simple lack of information. Lots of claims are made, but surveys of the human factors literature reveals that very little is known about enduser preferences or the degree to which particular interaction technologies satisfy performance needs.

Another aspect of conceptual formulation of a system involves an analysis of alternative media for the system. Given the needs of the enduser, the system designer can make use of baseline data and technological expertise to select alternative physical media that may lead to viable solutions. For example, if the enduser is a worker in a noisy factory environment, many types of voice interfaces might be ruled out. However, there may be new sensor technology known to the designer that might permit limited vocabularies in a voice interface. This information then leads to thumbnail sketches of possible interface instances. As these sketches are constructed, additional information may be needed to determine whether certain technologies are indeed consistent with other system requirements.

Once a representation has been captured in text and data, experience shows that it will need to be revised. Finding bugs in concepts, correcting descriptions and discovering omissions are all sensitive to the fact that there are no accepted formats that communicate to both designers and endusers. Repeated iterations with the endusers will be needed to compensate for this lack.

Requirements acquisition has as its goal the creation of a model of the enduser and organizational requirements for the interface. Model forms include information flow, process representation, task assignment and enduser behavior. There are many techniques used for data collection, only a few of which are amenable to computer support (e.g. instrumentation of existing interfaces).

A few attempts to structure the requirements acquisition process around a model have had some success (e.g. the ISDOS project, Jackson Design). The fact that none of these attempts have gained widespread acceptance probably means that a variety of requirements acquisition techniques will be used in the future. Rather than focus on a semantic model for task analysis, it may be more fruitful to examine general, customizable data organization and retrieval tools. Particular problems that need to be addressed are high level query facilities, rule based tools for analysis, and tools for re-organizing the data when the initial organization proves incomplete (this will inevitably happen).

Interfaces are always developed in conjunction with the application. It is probably not possible to cleanly separate requirements for the interface from requirements for the application. This means that the interface task analysis tools must be identical to or fit nicely into the set of tools used for task analysis of the application function.

Conceptual design is the generation and testing of hypotheses about interface presentation and behavior. These hypotheses can exist in two forms, one is the designers' technical specification of the design, and the other is a realization of the design for inspection by the endusers. The designers' technical specification may employ terms like direct manipulation, multiple dialogue threads or iconic presentation which will have little meaning to the endusers. The endusers can react best to example presentation and behavior. An interesting, open problem is a method for stating hypotheses and a tool for realizing these hypotheses as enduser interaction scenarios.

Storyboards are often used to help endusers visualize a set of design hypotheses. General drawing programs can be used to create the presentations for storyboards, but their end product is hardcopy and neither the descriptions behind the presentations, nor the storyboard scenarios are typically recorded with the design data. To this end, mixed media databases could be very helpful, along with tools for animation of the presentations. The animation specifications should be recorded in the database along with the presentations.

The storyboards will be revised, and there will be a need for tools and techniques to keep track of various revisions, tied to observations which prompted the revision. Finally, the tools must be efficient, as conceptual design is a fast paced activity, especially in a group environment.

A more dynamic visualization may be obtained through very high level prototyping, perhaps using a WYSIGYG interface builder such as Peridot [15], or Trillium [10]. Such prototyping is only feasible on very high performance workstations because the flexibility of the tools trades against the use of lots of CPU cycles. We conjecture that design effectiveness can be increased with investment of capital. Present day examples include the growing popularity of LISP and SMALLTALK machines as prototyping vehicles for software development. Given the dramatic increases in hardware performance, it does not seem out of place to throw MIPS, Megabytes, Megapels and Megamessages at the design problem.

The lack of standard interfaces to input devices is a minor, but serious problem. The ubiquitous asynchronous communication interface lacks speed and places a decoding burden on the realtime software. Perhaps a standard I/O bus or a message based mini-LAN will emerge as the solution. The lack of standard interfaces to display hardware is being solved by standards such as CGI, and by the growing consensus on function in window managers.

User performance models seem to occur in two flavors: loosely organized collections of facts about enduser behaviors, and formal models of enduser behavior. The success or failure of one of these types of user model depends on the skill with which it is used; examples of good and bad interfaces done with and without employing user models abound.

Loosely organized fact collections are stored in the minds of experienced designers, and result from experience and apprenticeship. A few attempts have been made to codify such knowledge, but by the time this is done, changes in the interface technology may render it obsolete. It would seem that the same kinds of video/audio/text data networks previously mentioned could be used to collect and index user behavior facts.

Formal models exist to predict behavior. Those models which have been thus far developed focus on a relatively low level of enduser behavior (GOMS, Riesner's BNF Grammars), or are constrained to predicting enduser strategy in uncomplicated tasks. There are research problems here of the greatest difficulty. Some researchers believe that Expert Systems tools may be used to build formal enduser models [22]

The requirements and the capabilities created during the execution of this component must match. Integration of tools in a toolkit can be achieved when: all the tools can be used in the same environment, tools are modular, so that new tools can be added and old ones deleted and existing ones modified. In the case of interactive tools, of course, common presentation and interaction techniques should be used so that enduser habits transfer among the tools

Links between requirements and capabilities ought to be recorded so that the completeness of the design and the consistency of the requirements can be assessed. Such links should convey both positive and negative support (i.e. when a design feature works against a requirement). It is extremely important that requirement/capability links carry an explanation and rationale. That a particular set of capabilities meets a requirement may not be obvious at all, especially when the requirement is a general one, such as a specific limit on training required for an enduser to reach a certain level of proficiency with the interface. The linking of requirements and capabilities suggests that similar information models (knowledge bases, semantic models) be used. The development of information models for this purpose is an open research area, scarcely touched at present.

Most development is a group process. Software engineering has lately been concerned with group development facilities, though the focus has been on tools and procedures for information sharing. Such tools will also be useful for interface design, but interface design has stronger requirements for visualization and high level modeling. Special presentation facilities, such as multiple projection displays, will be useful. Example interaction devices should also be present to allow storyboards to be played out with some realism.

It is extremely hard to maintain design consistency with the requirements and the conceptual design. Organization factors, such as design being done by a different team, make it difficult to carry the vision and understanding across the boundary between these two phases. The question of whether a particular design decision is consistent with the requirements or the conceptual design phase often cannot be answered because it falls into an area left uncovered, or because the facts needed to decide the question are difficult to extract from the RA/CD documentation. Systematic approaches require compatible, formal representations of both the conceptual design and the design, and this introduces overhead and loss of flexibility in the design process.

Often, the enduser is brought into the design process in an attempt to circumvent these problems. The enduser is shown the proposed design, and asked to determine if this meets the requirements. This places all of the burden on the enduser.

Solutions to these problems involve both methodology and tools. Tools, in particular, help maintain the fine-grained consistency throughout an interface by enforcing the many details that have to be considered to produce a well-crafted implementation. The methodology must also incorporate procedures for consistency checking, and the practitioners must carry these out.

One approach that has hardly been explored is to treat the consistency checking task as having three components: information retrieval, relevancy determination and consistency analysis. The computer may be used to speed up the retrieval task, with people determining the relevance and consistency of the retrieved facts. This process does not require a formal model, but preserves the flexibility of the designers by using plain or structured text to record information.

Whatever procedures and tools are chosen for managing consistency with the requirements and conceptual design may also be used to check the internal consistency of the design. Formal methods may well be worth their inherent overhead, especially if some semi-automated tools for deriving implementations from the formal representation are available (e.g. interaction techniques with displayed objects derived from known constraints).

As many have pointed out, it is important that the enduser be involved in the design process. Designers, as creative individuals, tend to see a personal view of the needs and behavior of the enduser, and the criticism by the enduser of the emerging design moderates this tendency. Likewise, the enduser may learn something that facilitates creative suggestions regarding the design, i.e. overlooked needs and opportunities. The maintenance of contact through this phase cements the bond of understanding between the designer and the enduser, making a satisfactory conclusion to the project much more likely.

As with conceptual design, the enduser best understands concrete examples of interface scenarios. Demonstrations are very useful, but the effort of their implementation should not exceed the probable utility, and there is some danger the enduser will erroneously conclude the project is near to completion. Prose and picture descriptions are useful as well, and far easier to produce.

One method for effectively using endusers in design is called iterative refinement and has been reported by Whiteside to produce excellent results [27]. The technique consists of having subjects use a partly implemented application system, augmented by a hidden investigator who attempts to map unexpected enduser actions into valid system functions. Such enduser inputs are later analyzed and used to produce an improved interface which is tested again.

Prototyping is frequently done, both to provide a demonstration vehicle, and to test the adequacy and feasibility of a design approach. The prototype is constructed to determine whether enduser performance meets expectations before a full implementation is done. In some cases, a prototype may be used to test the viability of the underlying hardware/software system in supporting the desired interface. Prototypes are very valuable when it is otherwise difficult to provide a rationale for a design decision. The acceptance of the prototype stands in lieu of the rationale for the design decision. This situation usually occurs when a decision has been made to consider a design approach that is novel. Such decisions are made for marketing, or internal research and development reasons, and the prototype provides a way to resolve the risk before too much time and resource has been spent.

Prototyping is only effective if a prototype can be developed much more quickly than a full implementation. A frequently made mistake, especially with workstations, is to prototype on the implementation hardware and system. Unless the prototype is seriously restricted in function, its development time is unlikely to be much shorter than a full implementation. In fact, many systems began life as a prototype and grew into a product. Extensibility and maintainability problems with such systems are common and well known.

Tools have usually focused on a particular design representation, such as Jackson Design or the Entity-Attribute-Relationship model. These is much controversy about the adequacy of such representations, and their flexibility in the design process. Flexibility and adequacy may be preserved by representing information as plain or structured text. The software tools attempt to recover text from the requirements, conceptual design and design documents based on semantically related keywords and phrases, leaving the task of relevancy determination and consistency analysis to the tool user.

The task can be done interactively, if the information database is online, and semantic retrieval tools are available. Portions of the text related to a consistency tester's query are shown on an interactive screen, where they may be dismissed as irrelevant or noted as consistent or contradictory by the touch of a finger. It will be possible and desirable to keep an audit trail of such decisions, both to measure the coverage of the work, and to uncover mistakes.

The presentations were roughed out in the conceptual design stage, and details must now be added to them. It will be useful if the tools used to create storyboards and the associated presentations also generate descriptions for the implementation phase to follow. A number of research systems include WYSIWYG generators of presentations (Trillium [10], Menulay, [5], Peridot [15]), whose internal representation of the interface might automatically be converted to implementation code. A number of research problems must be solved, however, before such systems can be widely used.

Prototyping is best done in an environment free from concerns about system limitations and interactive performance. It should be possible to quickly alter design decisions are create a new working prototype. This is easier if very powerful, interpretive software is used on a computer with several times the speed and size of the actual enduser machine.

Low level interaction techniques are typically closed packages of procedures and data. There are two reasons for this: (1) a uniformity of base interaction techniques in an interface is encouraged; the tight coupling of input and output gives better performance. Dragging and menu selection are examples of low level interaction techniques.

The choice of a low level interaction technique is governed by the task to be performed and the availability of devices on the enduser system. The decision to include another hardware interaction device in the system must be supported by reasoning that the benefit to the enduser outweighs the confusion and clutter caused by too many interaction devices.

Good programming practices should insure clean interfaces to all techniques in the library. Moreover, the techniques should be classified according to the interaction devices employed, the basic interaction semantics implemented (types of data manipulated), feedback requirements (function and performance of displays) and enduser indications and contra-indications.

It should be easy to add support for new interactions or new input and display devices. The most effective paradigm for this seems to be copying and modifying an existing interaction technique.

Higher level interactions may include multiple states, simultaneous interaction events, complex constraints, communication with the application and elaborate feedback. The complexity of the design problem at this level precludes the simple assembly of interaction technique modules.

Tools employed at this level include dialogue languages, interaction specification by example and constraint systems. The best possible world would allow the designer to create an interaction configuration in whichever tool mode best matched the design task of the moment. However, many hard problems are yet unsolved in the quest to view an interface design in all of these modes,

A proposed design must be tested, by building a prototype, by analysis of the design specification, or by the construction of a user model. These approaches are not exclusive, but the latter two require both a formal model and the skills to analyze it, and are rarely practiced. There is some opportunity here for tools which employ a particular formal model and its analysis, but these tools will require training of the designer. In the case a prototype is built, it will be useful to obtain ``hostile'' users as a test group, inviting them to try to ``break'' the interface to the system [11].

An important task, almost never done, is the estimation of interactive performance of the interface. Design errors resulting in poor performance are almost never caught until implementation is nearly finished, at which time a great deal of effort and expense is required to correct them.

Expert systems technology may permit the construction of intelligent assistants, capable of reducing the burden of making mundane design choices, and of minor customization of chosen techniques. A few research projects are beginning to explore these technologies.

Experiences of other designers, and human factors work provides valuable guidelines for designers. A well indexed knowledge base of experiments and reports would be extremely useful, but this does not presently exist, and would be a mammoth undertaking.

There is a considerable advantage in having an integrated data base of analysis, specification and design, if for no other reason than recording relationships among objects and assertions at the different levels in the design process. Whether the individual tools used against this database are integrated is not clear at present.

It is clear that the interface designer is also an enduser, albeit with special requirements. A common complaint about current interface design and implementation tools is that they are difficult to use. This stems partly from the difficulty of the problem, partly from the considerable programming skills required by today's tools, and partly because interface designers, like the cobbler's children, suffer from badly designed and implemented interfaces.

The interface designer's interface should provide both a final form presentation and an underlying form presentation for the interface objects being designed, and allow the designer to switch rapidly from design definition to design evaluation. WYSIWYG design tools may be very effective here, together with tools to convert the design specification into code. This is obviously possible for example when a drawing package is used to create graphic objects whose interface implementation will be by calls to procedures in a standard graphics package such as GKS or CORE.

The problem of associating action and application function with interface events is only partly solved in WYSIWYG design systems. Virtual input devices that can be manipulated as if they were real make it possible to do some association by the example of performing an operation on the virtual input device and then performing the corresponding operation on the presentation object. Where the association is formulaic, or dependent on the state of other objects, exemplary association does not work well.

Some symbolic specification of interface behavior appears inescapable for the near future. This will require some programming skill on the part of the interface designer, but the task may be made easier by better specification paradigms, such as active objects and declarative specifications.

In this section we try to isolate the research issues underlying the further development of the UI design tools described above.

We lack formal tools to assist in the conceptual design phase as well as to provide a usable guide for further stages of design and evaluation. First on the agenda would seem to be development of the necessary formalisms for describing conceptual models. This is clearly related to developments in knowledge representation in AI. Given the development of adequate formalisms, the next step is to capture them in automated tools.

We need to find ways to incorporate developments in knowledge based systems into the interface design process and into end user interfaces. Two possible means of accomplishing this are:

Currently tools exist for use during design, implementation, execution and evaluation of user interfaces. While further developments are needed in all of these areas, it is also important to consider the integration of these tools over the entire life cycle.

We were unable to agree among ourselves about the meaning of the increasingly popular term ``direct manipulation.'' This leads us to believe that we are in need of a more useful and formal definition than currently exists.

The interface designer is faced with an increasing range of choices of input and output media for user interfaces. What constitutes a ``good'' choice of media for a given application? How can the selection of ``natural'' or ``appropriate'' media be encouraged?

Adaptive interfaces change dynamically depending on recent actions by the end user. Issues involved in adaptive interfaces include more than just how to build them. How does one avoid violating the ``principle of least astonishment''? Is it possible to build an adaptive interface that is inobtrusive, that does not make the user feel he is being controlled by the computer?

There is a continuing need for better tools to analyze performance with user interfaces as part of the evaluation process. This includes data intensive operations such as analyzing event history traces. This appears to be another area where techniques developed in some branches of AI might be useful.

The interface designers toolkit can be viewed as a collection of separate tools which are used independently. What is the value of integrating these tools, and what level of integration is necessary.

There are a host of documentation issues which are related to user interface tool research. Examples are use of on line documentation, automated documentation generation, and testing documentation for understandability and for correctness related to the interface design.

There are many research questions which we feel are best attacked by those in other fields. Following are a few examples of such issues.

In order for us to build better on line help systems we must understand the kinds of help which prove most useful to end users. This involves understanding potential sources of confusion, as well as a better understanding of the learning process. Similar knowledge is needed by documentation designers and by those of us who would like to build automated documentation aids.

We need better models of end users, both cognitive and behavioral models. Our current tools are primarily based on an implied user model rather than an explicit one. This makes it difficult to evaluate the effectiveness of the tools in terms of satisfying the end users needs.