The file servers and databases at PARC currently have a capacity of about 20 gigabytes. There will be a significant increase in our demands on mass"storage for digitized audio, scanned images, and video. An additional 10 gigabytes of mass storage should suffice for the near term (we hope).

Our file servers need to speak a widely understood protocol, so that new machines can be added to the network with a minimum of expense. Code needs to be written for existing systems to enable them to act as clients under this protocol.

We have some database capability from a networked Oracle system in SCL. In CSL, the Cypress database system is used primarily for mail storage. Because there is no simple network protocol from the D"Machines to use these databases, there is no experience in using these databases outside of their original context. Although we could ignore this problem, it might be useful to try to make these services available on a wider basis.

We cannot tolerate servers that are unreliable or difficult to maintain. File servers that depend on non"standard hardware should be avoided. Recovery from a crash on the file server should be rapid and graceful.

Of course, we want the maximum performance possible. The SUN file system has a raw transfer speed of up to 250 kilobytes per second (read) and 60 kilobytes per seconds (write). We should not settle for performance significantly worse than this.

Buying more disk space to increase the storage capacity of our existing IFSs is only a partial solution. It does not address the issues of reliability, interoperability, or performance. Similarly, expanding the capacity of our product file servers leaves a number of problems unsolved.

In fact we do have a small networked Oracle system in SCL. There are lots of problems to do with publicity, protocols, performance, user education, size, system administration, and so forth. In addition, these systems do not serve well to store files.

Alpine lacks a real backup system, other administrative software, and must run on Trident disks. By expending about a man"year, we could fix all of these problems. However, we don't have the people free and don't believe it is a good investment in time to pursue this.

The SUN Network File System (NFS) is an integral part of the SUN operating system. It is a moderately high performance implementation, and the protocol is understood by a large number of machines. Hardware for the SUN is (relatively) cheap and easily available. Finally, PARC already has a fair number of SUNs. It has the drawbacks that it does not support file versions, authentication, or file locking (although there is a service that can be used to provide file locking).

Symbolics machines implement a proven, robust, remote file system that performs well. They support file locking, file versions, and a number of kinds of hardware. Symbolics machines already understand a number of ªstandardº protocols. However, the prospects for continued health of the company are not good, and this may argue against this option.

Apollo Domain workstations also implement a distributed file system as part of their operating system. They implement a compaction procedure that greatly increases the effective storage capacity of the system. We did not explore this in any detail. A similar alternative is to explore the use of a VAX (NFS) file server. The principal arguments against these is the proliferation of the type of hardware that must be supported in the building.

To store and manipulate electronic documents today, one has to deal with file systems and/or databases with very simple data models. There are restrictions on the size of fields, types of interconnections, enforcement of naming, system cost and performance, and the size of the information space.

Multiple types of objects must be stored. These should include plain text, structured text, scanned image, mail message, digitized voice, digitized video, bitmaps, uninterpreted byte arrays. If the system distinguishes types, it can provide special purpose compression and data search algorithms.

We need to manipulate links between different data elements. This is often called the hypertext model. Supporting links between text, pictures, and voice makes this more like ªhypermediaº. With this data model in the underlying system, it can provide more efficient storage and retrieval, and will match users' models of stored information. This should allow support of current projects such as Notecards and Colab. It is also compatible with ideas developed for the Object Service.

Documents by their very nature evolve. As they evolve, new versions of them are created. In addition, alternatives to a document may be created. Version and alternative trees must be supported in the document base system.

We want to support multiple access styles in the document base. By allowing users to give documents names, we can support standard file server needs. By allowing attributes and keywords on documents, with appropriate indices, we can support simple rapid access by document content or history.

Documents must be sharable. Several clients must be able to access to the same document simultaneously.

Documents will be stored on servers found on the internet. Servers must be able to exchange data, support different protocols for users, and share data. The server must support transactions. Locking at the granularity of at least a document is required. Very large objects require that they have page level access (or access to document fragments).

Alerters are a simple form of triggers. Some class of events trigger the sending of a ªmessageº to clients. Events can include insert, delete, read, lock, or update of some class of objects. Alerters provide a ªhookº for other systems (e.g., Notecards or Colab). Alerters can be used to assist cache and screen maintenance on workstations, and to trigger some service at document creation (e.g., document recognition).

The requirements for digital on"line storage in this time frame is very large. We want to store very large files (e.g. images, video sequences etc). To not be limited in the next few years we should probably plan on increasing our storage available by at least a factor of10. This requires between 500 and 1000 gigabytes.

A storage hierarchy is required to get acceptable performance with reasonable cost. Not all of the documents must be ªimmediatelyº accessible. Delays of seconds are acceptable for infrequently referenced or archived documents. Archival storage is required to keep all relevant versions of objects. Optical media are an appropriate bottom end.

Data compression is a useful adjunct for the system. For many classes of document (e.g., scanned images), data compression radically changes the amount of storage available for a fixed cost (an order of magnitude or more).

While we are unable to give exact number, we must have a system that allows easy experimentation with information retrieval strategies. In the same way that Dorados were higher performance personal machines than anyone else had available at the time they came on board, the document base server must initially overwhelm the needs of the problem.

To support a service, the system must be robust, available, and replicable. It must provide administrative functions such as debugging, backup, historical logging, and monitoring. Security and access control must be provided. No server should be isolated. It must cooperate with other server and foreign data systems.

Conventional database management systems such as Oracle and RTI Ingres have a number of important properties:

However these relational systems won't fill our document storage and retrieval needs because they are limited in:

Many desirable document operations, such as finding the transitive closure of a bibliography, will inevitably take too long for users.

Optical disk servers are (or shortly will be) sold by some companies. Although the jukebox and optical disk hardware are interesting, the software layer they sell does not satisfy most of our needs.

Purchase of a full system, such as from Filenet, is not an option. The scanners, printers, and workstations may not be those that integrate with Xerox's long term product plans.

Jukeboxs and optical disk hardware are not commodity items. Other technologies may make them obsolete in a few years. Service, lifetime of product, and maintenance are questionable. We think it will probably be the right thing to get one of these as a bottom end for our project, but for the long term there is some risk in dealing with startup companies. OEM prices for a Filenet jukebox are about ¤160K while Cygnet is somewhat cheaper at ¤115K, though the features are somewhat different.

The Object Service has different goals and time scale from those presented here; the Object Service is exploring a higher level execution based model of storage, and the utility of persistent processes. It also is exploring a seamless integration of long term storage with the Smalltalk programming system. Rather than trying to combine the current needs for a document base with the long term goals of the Object Service project, it seems more appropriate in the medium term to have separate projects. The server discussed here may be able to act as a storage backend for the Object Service.

This is what we recommend. The server would implement the data model developed jointly between CSL and ISL over the last few months. For large volume storage, the server would use either a jukebox optical disk unit directly connected to the server, a network server with a jukebox optical disk, or use a relational database system in conjunction with a jukebox or a networked jukebox. Hagmann and Kent have agreed to act as coordinators (leaders?) of this new project. A choice must be made whether to implement the service in their natural language (Cedar) or to explore other options. The viability of this choice is dependent on having a porting strategy of Cedar to new hardware, and the availability of new hardware. To make this system usable on our Dorados, we will need 10 mHz Ethernet boards on them. This is another source of risk as well as a cost item.

To build this service requires:

The server would act as a front end, provide the data model, and manage the short, medium, and long term storage. It would most likely be written in Cedar and initially run on Dorados. However, it must be portable and must move to the server architecture that will result from the ªComputational Baseº committee.

The staffing of the project is critical to any time table we might propose. However, to give an idea of phasing, some sort of milestones were requested by the OCM. With reservations, we propose the following milestones taken from ªstart of codingº:

This conclusion rests, in part, on the assumptions we have made. To be complete, we should understand the implications of our assumptions and validate our assumptions. Unfortunately, time and manpower have not allowed us to be as complete as we would like to be. We have relied principally on our own experiences and understanding of external research and events. It may be appropriate to have our conclusion reviewed by an outside consultant.