DRAGON INTERRUPT MANAGEMENT
DRAGON INTERRUPT MANAGEMENT
DRAGON INTERRUPT MANAGEMENT
DRAGON PROJECT — FOR INTERNAL XEROX USE ONLY
DRAGON PROJECT — FOR INTERNAL XEROX USE ONLY
DRAGON PROJECT — FOR INTERNAL XEROX USE ONLY
Dragon Interrupt Management
Jean-Marc Frailong, Pradeep Sindhu, Russ Atkinson
Dragon-86-21 Written: August 14, 1986  Revised: October 23, 1986
© Copyright 1986 Xerox Corporation. All rights reserved.
Abstract: This document describes the interrupt mechanism for the June 87 Dragon. This mechanism supports up to 32 interrupt sources, accommodates both device and interprocessor interrupts, and allows interrupts to be either broadcast or directed to individual processors.
The implementation is distributed across the IFU, the small cache, the SloBridge, and other I/O devices (e.g. the display). The IFU has a single Reschedule line that causes the processor to take an interrupt to a fixed location. The small cache implements I/O registers that cause the reschedule line to be pulsed when the appropriate IOWrite or BIOWrite transaction appears on the DynaBus. Finally, the SloBridge has hardware to convert SloBus device interrupts into an IOWrite or BIOWrite that writes the interrupt registers for one or more caches. Other DynaBus devices may post Dragon interrupts using the same IOWrite or BIOWrite transactions.
Keywords: Dragon, June87, Interrupts, Interrupt Management
FileName: /Indigo/Dragon/Interrupts/InterruptsDoc.tioga, .interpress
XEROX  Xerox Corporation
   Palo Alto Research Center
   3333 Coyote Hill Road
   Palo Alto, California 94304



Dragon Project - For Internal Xerox Use Only
Contents
1. Basics
2. Interrupt Support in the IFU
3. Interrupt Support in the Small Cache
4. Interrupt support in the SloBridge
5. Other interrupt sources
6. The story of an interrupt
7. References
ChangeLog
1. Basics
The Dragon processor interrupt mechanism is provided by the IFU Reschedule line. When this line is asserted, the processor takes an interrupt through some fixed location. This mechanism alone is sufficient to implement interrupts in a multiple processor system (for example by tying the Reschedule lines of all processors together and asserting this line whenever any interrupt happens), but it suffers from two drawbacks. First, it is not selective in that it always interrupts all processors; typically there is no reason to do this, and in fact doing this can be detrimental because it tends to put processors in lock step. Second, it requires devices generating interrupts to be smart in that they have to write the reason for the interrupt into shared memory so the interrupted processors can determine who generated the interrupt and take appropriate action.
These problems can be overcome by providing a more flexible way to communicate interrupts from their source to the Reschedule lines of various processors. The mechanism described here uses the DynaBus for this communication. Each small cache provides I/O registers that respond to certain IOWrite and BIOWrite transactions on the DynaBus (see the DynaBus specification [1] and implementation guidelines [2]) by asserting the Reschedule line for that cache's processor. Since there is a level of interpretation between the DynaBus transaction and the pulling of the Reschedule line, it is easy to provide the facility for handling multiple interrupt sources and the ability to handle directed and broadcast interrupts. Each interrupt source initiates a DynaBus transaction for each interrupt event. In the June 87 machine, there are (at least) three kinds of sources, SloBridge(s), display controller(s), and Dragon processors.
As used in this document, the term interrupt source is used in two ways: for a software (and processor) initiated class of interrupts, or for a device that is directly connected to the DynaBus and issues interrupt requests directly to the small caches. In this sense, the SloBridge is an interrupt source, yet devices connected to the SloBridge are not interrupt sources. The term interrupt source index denotes a compact indexing scheme that identifies index sources by a 5 bit index (with values 0 through 31).
As used in this document, the term interrupt request is used to indicate a single request for interrupt servicing that is derived from either an I/O device or a processor. It is a more generic term than interrupt source, and its meaning should be clear from context.
2. Interrupt Support in the IFU
Various aspects of interrupt support in the IFU have been described elsewhere (see the References section), but we will summarize the description here for convenience.
The interrupt hardware in the IFU consists of a Reschedule line to signal interrupts, a Status register to record (among other things) state information about interrupts, and some logic to cause the necessary changes in the processor's control flow. Two of the bits in the Status register are relevant to interrupts: reschedule and trapsEnabled; the first bit records a pending reschedule, while the second provides the ability to mask reschedules. Thus, asserting the Reschedule line sets the reschedule bit. If trapsEnabled is set, the processor takes a reschedule trap immediately after the current instruction is completed (taking the trap is semantically equivalent to executing a procedure call). Once the IFU decides to take the trap, it clears trapsEnabled before actually taking the trap to avoid infinite regress. It is the task of the trap routine to re-enable traps.
A cache asserts the Reschedule line by pulsing it for one PhA/PhB cycle. This avoids the problem of having the IFU tell the cache when to lower the Reschedule line. The timing of Reschedule is shown below:
[Artwork node; type 'ArtworkInterpress on' to command tool]
Reschedule Timing
3. Interrupt Support in the Small Cache
Small caches have two 32-bit registers, InterruptStatus and InterruptMask for interrupt support. Each bit position in the two registers corresponds to a different interrupt source; sources are treated equally as far as the hardware is concerned, but the software can implement any level structure it wants on top since it has complete freedom in choosing the servicing order. The bits in InterruptStatus correspond to pending interrupts, while those in InterruptMask allow interrupts from individual reasons to be masked (a 0 value masks). Masked interrupts are still recorded in InterruptStatus, but they do not result in a Reschedule. On reset, a cache sets the InterruptStatus and InterruptMask registers to zero to indicate no interrupts pending and all interrupts disabled.
The cache's Reschedule output will be pulsed for one Dragon cycle (signaling an interrupt to its IFU) whenever the OR of (InterruptStatus AND InterruptMask) changes from 0 to 1.
Pulsing the Reschedule output is preferable to raising the Reschedule output since there is no need for the IFU to perform any action to lower the output. This not only reduces the instructions required, but also eliminates several possible race conditions.
All of the operations listed below are atomic system-wide. All EU cache interrupt operations are implemented via I/O transactions, where the ID field specifies the cache, the I/O address specifies which operation is to be performed, and the I/O data (low-order 32 bits) supplies an additional parameter for operations that may modify the EU cache state. The I/O transaction can be either initiated from the Dynabus side or the processor side of the EU cache. The *Broadcast operations differ from the other operations in that a BIOWrite transaction is used instead of an IORead or IOWrite transaction. The *Self operations are meaningful only from a processor to its EU cache and allow a processor to handle its own interrupt status without knowing the DynaBus DeviceID of its EU cache.
The following summarizes the interrupt management operations available for the EU cache (refer to CacheOps.mesa [4] for an exact Cedar interface):
InterruptStatus: 32 bits, Read-Write. Bits may be set/cleared individually. Reschedule is pulsed if a new reason is added.
SetInterruptStatus: OR bits into InterruptStatus of specified cache and pulse Reschedule for specified cache if new reason set.
SetInterruptStatusBroadcast: OR bits into InterruptStatus of all caches and pulse Reschedule in each cache where a new reason is set.
SetInterruptStatusSelf: OR bits into InterruptStatus of the processor's EU cache and pulse Reschedule if new reason is set.
ClearInterruptStatus: remove bits from InterruptStatus of specified cache.
ClearInterruptStatusBroadcast: remove bits from InterruptStatus of all caches.
ClearInterruptStatusSelf: remove bits from InterruptStatus of the processor's EU cache.
ReadInterruptStatus: return InterruptStatus for the specified cache.
ReadInterruptStatusSelf: return InterruptStatus of the connected processor's EU cache.
InterruptMask: 32 bits, Read-Write. Reschedule is pulsed if the new mask makes visible a pending, but previously masked, interrupt reason.
WriteInterruptMask: sets the value of InterruptMask for the specified cache and pulse Reschedule for specified cache if a pending interrupt is thus made visible.
WriteInterruptMaskBroadcast: sets the value of InterruptMask for all caches and pulse Reschedule in each cache where a pending interrupt is thus made visible.
WriteInterruptMaskSelf: sets the value of InterruptMask for the processor's EU cache and pulse Reschedule if a pending interrupt is thus made visible.
ReadInterruptMask: returns the value of InterruptMask for the specified cache.
ReadInterruptMaskSelf: returns the value of InterruptMask for the processor's EU cache.
A processor may manipulate its interrupt status without knowing its own ID (through the *Self operations), which simplifies the software a little in some critical sections. The *Self operations above are only implemented via I/O instructions executed by the Dragon processors. As an optimization to reduce Dynabus traffic, the EU cache does not need to issue a Dynabus transaction. The *Self operations differ from their directed counterparts only in that the 24 MSB of the I/O instruction operand is 0, which is interpreted as directing the request to the connected EU cache.
Although both the IFU and EU caches support the above operations, the IFU's Reschedule line is connected only to the EU cache. Thus, it is meaningful to direct operations only to the EU cache.
4. Interrupt Support in the SloBridge
The SloBridge receives a number of physical interrupt lines from the outside world and from on-chip sources. The SloBridge must provide a method for interrupt one (or all) Dragon processors to provide device service. Since the SloBridge only provides one interrupt source for all of the devices it must also provide a means to distinguish which devices are requesting service.
A SloBridge has 3 interrupt support registers: InterruptStatus, InterruptMask and InterruptAction. InterruptStatus is a read-only copy of the current state of the physical interrupt lines. InterruptMask acts as a mask on each physical interrupt line (one mask bit per interrupt line, 0 meaning masked). InterruptAction specifies what should be done to notify a cache that a SloBridge interrupt needs servicing: either a single cache or all of them may be notified, and the reason bit to be raised is programmable (more details given below).
The SloBridge contains an internal 1-bit flag (ServiceRequired) indicating that service has been required, but has not been provided. This bit is set when a new interrupt line becomes visible (because the physical line raises, or because the mask makes visible an active line previously hidden). If ServiceRequired was not previously set, an interrupt request is sent according to InterruptAction. When a processor reads InterruptStatus, ServiceRequired is cleared. This scheme ensures that a processor does not get spurious interrupts (in fact, spurious interrupts may still occur, although with a much lower probability).
If InterruptAction is changed then the InterruptStatus register should be sampled both before and after the change to ensure that no devices are ignored. The current software plans do not include changes to this register after initialization.
All SloBridge interrupt operations are invoked via I/O instructions, where the ID field indicates which SloBridge is being selected, the I/O address specifies the operation, and the I/O data supplies an additional parameter (which is ignored for Read* operations). The following summarizes the operations available (refer to SloBridgeOps.mesa [3] for an exact Cedar interface) for each SloBridge register:
InterruptStatus: 32 bits, Read-Only. Each bit reflects the state of the corresponding physical interrupt line. Bits in this register are cleared by an appropriate action on the corresponding device causing it to rescind its interrupt line. The correspondence between bit position in this register and interrupt line will be partly through external wiring for interruption originating in the SloBus, partly through internal wiring for internal interrupt sources.
ReadInterruptStatus: returns the value of InterruptStatus for the specified SloBridge.
InterruptMask: 32 bits, Read-Write. One bit per bit in interrupt status. A 1-bit enables the corresponding interrupt line. It is set to 0 upon reset, disabling all interrupt lines.
ReadInterruptMask: returns the value of InterruptMask for the specified SloBridge.
WriteInterruptMask: sets the value of InterruptMask for the specified SloBridge. An interrupt request will be sent to cache(s) if the new mask reveals a previously masked pending interrupt and ServiceRequired was not previously set.
WriteInterruptMaskBroadcast: sets the value of InterruptMask for all SloBridges. An interrupt request will be sent to cache(s) if the new mask reveals a previously masked pending interrupt and ServiceRequired was not previously set.
InterruptAction: 32 bits, Read-Write. The least significant 10 bits are the ID of the small cache to which the interrupt should be sent. The next bit is 1 when a broadcast interrupt is required. In that case, the low-order 10 bits should be set to all 1's (requirement of the SmallCache). The next 5 bits encode the interrupt source index. The 16 most-significant bits are currently unused. They are ignored on write, and return an unspecified value on read.
ReadInterruptAction: returns the value of InterruptAction for the specified SloBridge.
WriteInterruptAction: sets the value of InterruptAction for the specified SloBridge.
WriteInterruptActionBroadcast: sets the value of InterruptAction for all SloBridge.
5. Other interrupt sources
5.1. Guidelines for interrupt generating DynaBus devices
Devices that generate interrupts on the DynaBus should adhere to some basic guidelines to ensure that software support will be reasonably easy:
1. Each device should use as few interrupt source indexes as possible, since they are a scarce resource. One is the preferred number.
2. The interrupt request sent by a device on the DynaBus should be definable via DynaBus (B)IOWRqst, i.e. the cache ID (or broadcast) to which the interrupt should be sent and the interrupt source index to be raised.
3. Each device should have a register indicating the interrupts that have been posted, but not yet handled by the software. This register should contain 1 bit per interrupt source. The bits should be cleared individually through some positive action in the device driver that removes the cause of the interrupt (such as reading data from a received data register, or explicitly clearing the interrupt bit).
4. Each device should also have an interrupt mask, with one bit position per interrupt source in the same order as in the interrupt register mentioned in point 3, to allow individual interrupt sources to be ignored.
Reading the specifications for the small cache and the SloBridge should provide an example of how these guidelines may be applied.
5.2. Software interrupts
It may be necessary in software to have inter-processor interrupts. Any processor may interrupt another processor or all of them by executing the I/O commands SetInterruptStatus and/or SetInterruptStatusBroadcast. The software should adhere to the guidelines described in 5.1 by using memory locations under a lock to further describe interrupts.
5.3. Memory controller
To be defined
5.4. Display controller
To be defined
6. The story of an interrupt
This section traces the handling performed for a single interrupt. We give the steps for a "normal" device interrupt in chronological order.
Although the hardware supports masking of interrupt sources, our current plans for interrupt handling do not use this feature. All processors are equivalent, and all interrupt sources are permitted.
In general, each interrupt request is handled by notifying a condition variable associated with each source. This notification causes a process switch to the handler, which then performs the necessary data handling and clears the interrupt request.
This description is subject to change, particularly the parts involving the software.
6.1. At the device
The device hardware detects an interrupt condition (for example, a packet arrives at the Ethernet interface). In response it raises an appropriate interrupt line that is connected to the SloBridge chip. This interrupt line is typically left in the requesting state until some service is performed by software.
6.2. At the SloBridge chip
When the leading edge of the signal on the interrupt line reaches the SloBridge, and the SloBridge has not already requested an interrupt (see the above description of the ServiceRequired bit), then the SloBridge sets the ServiceRequired bit to 1 and issues a SetInterruptStatusBroadcast operation.
This operation can also result when an interrupt line is held high and the mask is changed to allow that interrupt. As mentioned above, current software plans do not depend on this feature.
The SetInterruptStatusBroadcast operation is implemented as a BIOWrite (Broadcast I/O Write) transaction directed at all EU caches. The address indicates that it is a SetInterruptStatusBroadcast operation, and the data indicates the interrupt source (a maximum of 32 interrupt sources are allowed on a Dragon system). The SloBridge InterruptAction register permits a SloBridge to map into only one interrupt source. By software convention, each interrupt source is associated with at most a single SloBridge.
Each SloBridge has a register (InterruptStatus) that directly reflects the interrupt lines attached to the SloBridge. Once the SloBridge requesting service has been identified, the InterruptStatus register is used to determine which devices attached to the SloBridge are requesting service.
Since reading the SloBridge InterruptStatus register clears the ServiceRequired bits to allow more interrupts, then the software must be prepared to handle all pending interrupt requests for any given SloBridge whenever it handles an interrupt request.
6.3. At the EU cache
When the SetInterruptStatusBroadcast transaction arrives at each EU cache the request data is OR'd into the InterruptStatus register in each EU cache. If the AND of the changed bits and the InterruptMask register is not all 0, then the EU cache pulses the Reschedule line to the connected IFU.
The SetInterruptStatusBroadcast operation is implemented as a BIOWrite transaction where the I/O address indicates that the request is a SetInterruptStatusBroadcast operation, and the data indicates which interrupt source is requesting the interrupt.
Since the IFU cache and the EU cache use identical cache chips the IFU cache performs the same actions, but the Reschedule line is not connected, and so has no effect.
It is possible to read the InterruptStatus register without affecting its contents. This is the purpose of the ReadInterruptStatusSelf operation. This is used in the software to avoid trying to process requests that have already been satisfied.
6.4. At the IFU
When the Reschedule line is pulsed by the EU cache the IFU sets the rescheduleRequested bit to 1. At the next instruction (after pipeline delay) when the rescheduleRequested bit = 1 and traps are enabled the IFU will take a Reschedule trap, which as a side effect clears the rescheduleRequested bit in the IFU. Traps are, of course, disabled during the execution of the Reschedule trap handler.
6.5. At the Reschedule trap handler (software)
When the Reschedule trap handler starts execution some processor acquires the scheduler lock (processors running low priority processes have an advantage in acquiring the scheduler). The processor that acquires the scheduler lock reads the InterruptStatus register in the EU cache to determine which interrupt sources are requesting interrupts. The value read is then used to clear the requesting bits via the ClearInterruptStatusBroadcast operation.
6.6. Back at the EU cache
The ClearInterruptStatusBroadcast operation clears specific bits in the InterruptStatus register for all EU caches. This is used to acknowledge that the specific interrupt requests have been seen. It is important that clearing these bits be performed atomically with respect to the SetInterruptStatusBroadcast operation to avoid losing interrupts.
The ClearInterruptStatusBroadcast operation is implemented as a BIOWrite transaction directed at all EU caches. The I/O location is different than the one used for SetInterruptStatusBroadcast, even though it affects the same register in the EU cache.
The reason to clear specific bits is so that interrupt requests that arrive between the ReadInterruptStatusSelf and the ClearInterruptStatusBroadcast operations are not ignored. The use of SetInterruptStatusBroadcast and ClearInterruptStatusBroadcast ensure that all EU caches have the same values in their InterruptStatus registers at any given moment, which makes it easier to allow any processor to handle the interrupts.
6.7. Back at the Reschedule trap handler (software)
After the ClearInterruptStatusBroadcast operation has finished the trap handler must read the SloBridge InterruptStatus register for each requesting SloBridge to determine which devices are actually requesting service. For each device there is a corresponding interrupt condition variable, which is notified (delicately) to wake up the process (if any) waiting for the interrupt. After all requests have been turned into notifications the scheduler reassigns processes to processors to allow the newly ready processes (if any) to run. Then the scheduler lock is released, and the processor that handled the Reschedule request either continues executing the interrupted process or switches to one of the newly ready processes.
For some special interrupt sources it would be possible to completely handle the interrupt in the Reschedule handler without causing any process switches. We would do this for devices that have high interrupt rates or severe real-time response requirements. No such devices are anticipated for the June87 Dragon.
To avoid excess notifications, a record is kept in software for which device interrupts have been seen by the scheduler but not cleared by the device handlers. When testing for interrupt requests only those devices that are newly requesting interrupts should result in notifications.
6.8. At the device handler (software)
The device handler performs the appropriate device servicing, which also clears the associated interrupt request bit from the SloBridge InterruptStatus register. This usually indicates that the device can proceed to issue additional interrupts as necessary. This handling may also include setting up the next task for the device to accomplish (the next packet to send, block to transfer, or whatever).
The device handler must also indicate to the software that the device interrupt has been serviced so the scheduler will again notify the associated condition variable.
The means for transferring data to the user will differ from device to device, and will also depend on which approaches are taken to address spaces and buffering in the operating system. We do not choose to make these decisions here, since they do not affect the basic interrupt handling.
The scheme adopted here is not completely foolproof, insofar as it may infrequently generate spurious interrupts since an interrupt may be reflected in the InterruptStatus register before it has been sent to a cache. A possible solution would be to provide another register indicating which interrupt lines have been posted to caches and are still pending. If only this function was available, polling for interrupts would not be possible.
7. References
[1] Sindhu: "DynaBus Logical Specifications"
[Indigo]<Dragon>DynaBus>DynaBusLogicalSpecifications.tioga, .interpress
[2] Frailong: "DynaBus Implementation Guidelines"
[Indigo]<Dragon>DynaBus>DynaBusGuidelines.tioga, .interpress
[3] Frailong: SloBridgeOps.mesa
[Indigo]<Dragon>Interrupts>SloBridgeOps.mesa -- Will migrate to SloBridge directory
[4] Sindhu, Frailong: CacheOps.mesa
[Indigo]<Dragon>Interrupts>CacheOps.mesa -- Will migrate to SmallCache directory
ChangeLog
Jean-Marc Frailong October 22, 1986 10:45:46 am PDT
Major editorial rewriting. Separated clean Cedar interfaces. Added sections on various interrupting devices on DynaBus. Left placeholder for RRA to give some details on software interrupt processing in Dragon Cedar.
changes to: almost everything...
Russ Atkinson (RRA) October 23, 1986 6:22:39 pm PDT
Added the "story of an interrupt" section, and made minor corrections to other sections.
changes to: lots, Dragon
Jean-Marc Frailong October 24, 1986 11:30:01 am PDT
Minor corrections after proofreading. Added a few references.
changes to: minor editorial changes, References