This note gives instructions and advice to the Release Coordinator about how to make a Cedar release.
What does a Release Coordinator do?
A Cedar Release Coordinator shepherds a Cedar release from its planning to its completion. Depending on the amount of authority that a Release Coordinator has, he or she will make more or fewer decisions. However, the operational details are handled by the Release Coordinator.
The Release Coordinator should be conscientious, polite, and somewhat knowledgeable about the innards of Cedar. It is often a time-consuming job, so it is usually a mistake to have a manager of people also be Release Coordinator. It is also a mistake for the Release Coordinator to have insufficient authority or information to make decisions about the release.
Planning a new release
In planning a release it is important to determine the objectives and scope of the release. A release may be as simple as a few bug fixes, or it may involve a change to the nucleus of Cedar. The amount of change in a release is normally related to how many commonly used interfaces change. A release is more difficult if there is bootstrapping to be done, especially for BCD or instruction set changes.
Another important part of planning a release is to have an estimated time table. Past experience shows that most estimates are optimistic. A prudent Release Coordinator will not take these estimates literally. Unexpected bugs, vacations, and emergencies of various forms all play a part in degrading their accuracy.
Preparing a new release
1. Get a new root directory made
Your local file system administrator should be asked to make a directory for the new release with at least enough space for two versions of all files in the release plus some contingency space. As of Cedar 6.0 this amount is 60000 IFS pages. The name of the directory will be
Cedar followed by the version number and the server is
Cyan (prior to Cedar 6.0 it was
Indigo). The owner of the directory is CedarAdministrators^.pa, which is a list that must contain the Release Coordinator. The defaults should be set as follows:
Default file protection:
R: Owner World;
W: CedarAdministrators^.pa CedarImplementors^.pa Owner;
A: CedarAdministrators^.pa CedarImplementors^.pa Owner
Create permission: CedarAdministrators^.pa CedarImplementors^.pa
Connect permission: CedarAdministrators^.pa
In most cases a new CedarChest directory will be necessary as well. The conventions are similar to those of Cedar. As of Cedar 6.0, 40000 IFS pages were desirable. The release coordinator may or may not also have responsibility for CedarChest.
2. Run the UnReleaseTool
The UnReleaseTool is accessed through ReleaseTool.df. It will move the DF files for the old release to the directory for the new release, and will fix the names and dates of the DF files to refer to the new directory. The other files will be left on the directory for the old release.
3. Make a new basic boot file
Most releases do not involve low-level interface changes. For such a release we construct a boot file named
BasicCedarDorado.boot so implementors can develop higher-level modules. When this is the first boot file of the release this construction is more complicated than normal.
3.0 - Bring over all files from BootEssentials.df, BasicCedar.df and UserCredentials.df. Also bring over the public files from MakeBoot.df, Compiler.df, and Binder.df.
3.1 - Edit BasicCedar.bootmesa (from BasicCedar.df) to change the version number. In most cases this involves incrementing the the minor part of the release number.
3.2 - Edit and compile DefaultRemoteNamesImpl.mesa (from UserCredentials.df). This file has the remote names that the boot file will access during its installation and initialization.
3.3 - Make the boot file. In most cases this is as simple as running the command file MakeBasicCedar.cm (from BasicCedar.df).
3.4 - Edit User.profile (from BootEssentials.df) to refer to the new directory names. Also make your own profile point to the new versions of any necessary files, and store it to [Ivy]<user>version>user.profile (where user is your user name and version is the version number).
3.5 - Store the changes to the new root directory. You will probably need to edit the directory names in BootEssentials.df, BasicCedar.df and UserCredentials.df before performing the SModel.
3.6 - Test the boot file.
4. Send a message to indicate that the release has started
In this message you need to give the version number and describe any departures from previous release practices the implementors are expected to know. You should indicate what the goals of the release are, what changes are permitted or encouraged, and what the schedule is likely to be.
Development phase
Making new basic boot files
In the normal cycle of making basic boot files there is relatively little to change.
1 - Bring over all files from BasicCedar.df. Also, if you have any reason to suspect that it has changed, bring over the public files from MakeBoot.df.
2 - Execute the MakeBasicCedar.cm command file.
3 - SModel BasicCedar.df.
4 - Test the boot file.
Making version maps
Version maps are files that provide fast translation from version stamps to long path names for files, and for fast translation from short names to long names. They are constructed by the program VersionMapBuilder. If the version maps must be created from scratch, the command GenCedarMap should be used. If the current version maps are mostly correct, then the command MergeCedarMap will be faster and just as correct. After creating the version maps, SModel SaveMap.df to store the maps.
Keep the directory under quota
It will periodically be necessary to use Chat to keep the release directory under quota. Keeping no more two versions of all files is normally safe, and implementors must assume that this can happen at any time.
Manage the "torches"
A person who has the torch for a release component has permission to make changes in that component. That person also has the responsibility of passing the torch back to the Release Coordinator when the changes have been made (and preferably tested). Managing who has which torches is important in a community where numerous people have the capability to make changes in system components. Otherwise there can be chaos when simultaneous development of a component is attempted.
Components that are understood to be under development by a single person, or by a group with its own torch manager, may have an implicit torch, which does not need to be managed by the Release Coordinator.
It is possible to have torch management performed by someone other than the Release Coordinator, but this often leaves the Release Coordinator with a harder job, since the Release Coordinator has less information about how to keep the release consistent.
Coordinate compilation order
In releases where there are significant interface changes the compilation order needs coordination. Here the Release Coordinator may perform critical compilations to ensure the proper order, or may simply tell the various torch holders when it is safe (or necessary) to recompile. The Release Coordinator is normally responsible for compiling modules for absent implementors.
All components within the LifeSupport level have command files that have the appropriate commands to remake the components. In almost all cases, to remake component X one should BringOver X.df (using the DF Tool), execute MakeX.cm (using the Command Tool), then SModel and Verify X.df (using the DF Tool).
Encourage and monitor testing
The more testing a release gets the more reliable it is likely to be. The Release Coordinator should encourage people to test out their changes.
Integration phase
The integration phase of the release is when we try to get a release to converge to a stable and useful state. During this phase the Release Coordinator must continue to keep the system state consistent, keep the release directory under quota, manage the torches, and coordinate compilation order. There are other duties, however, that are necessary to get the release to converge.
The integration phase of the release is somewhat more structured than the development phase. During the integration phase components are submitted for release, which means that the components are considered to be frozen (occasional thawing for bug fixes may be necessary). A component is submitted by sending a message to the Release Coordinator, with a copy to each implementor. Such a message indicates that the component's implementor considers the package complete, and has used the DFTool to verify its consistency.
1. Send a message calling for release submissions
Sending the request for submissions message signals the start of the integration phase. The message should contain the form to be used for submissions. Be prepared for components being submitted multiple times.
2. Periodically run the global Verify phase of the ReleaseTool
It is possible for a release to fail to converge when development of components continues, even though other components may be invalidated by that development. One tool for detecting this problem is the Verification phase of the ReleaseTool (using the VerifyRelease command registered by the ReleaseTool), which checks for global consistency of compiled modules in the release. Problems in the DF files should be corrected (as much as possible) by the implementors, both to keep the Release Coordinator's load low and to keep the implementor's consistency high.
3. Keep the release message up to date
At this point in the release process a draft of the release message should be started. It is best to start with previous release messages. As submission messages arrive the draft release message should be kept current. The state of the release message reflects the state of the release, so the release message is a critical tool in managing the release. Properly used, it can indicate which components need to be submitted, and which components are being submitted as correction to earlier submissions.
As new catalog entries are received in the submission entries, you should also update the file Catalog.tioga, which describes the entire release.
Making the release
1. Make candidate boot files & version maps
These boot files and version maps will be the ones released unless truly nasty problems crop up. The Release Coordinator should install them on at least one machine from scratch to test the installation sequence. To make full boot files, follow these steps:
1.1 - Using the DFTool, BringOver all files from BasicCedar.df. Also, if you have any reason to suspect that it has changed, bringover the public files from MakeBoot.df.
1.2 - Execute the MakeBasicCedar.cm command file.
1.3 - Using the DFTool, SModel BasicCedar.df.
1.4 - Test the boot file.
Version maps are constructed in the same manner as during the development phase.
2. Send messages about the impending release
A message to implementors announces the expected time of the release, and the availability of candidate boot files. Implementors are urged to try out the system for a while.
A separate message to users should alert them to the impending release.
3. Run the ReleaseTool
If no disabling problems are found, it is time to verify that the release is consistent using the ReleaseTool. If problems are found by the ReleaseTool, they must be fixed. The severity of the problem determines who should fix it (normally either the Release Coordinator or the component maintainer). Of course, if the ReleaseTool has been used frequently during the submission phase these problems should be both rare and minor.
4. Update CedarRelease.df
4.1 - ReleaseMessage.tioga & ReleaseMessage.press - These are the final versions of the description of the release.
4.2 - Catalog.tioga & Catalog.press - These are the final versions of the complete system description at the component level.
4.3 - ReleaseCoordinator.tioga - This file, which you are now reading, should be updated whenever a change is made to the release procedure.
4.4 - Released.switch - If this file is present the system considers itself to be released. Otherwise the system considers itself to be in development. When the system is in development it examines BootEssentials.df at every boot (the code is in InstallerImpl). When the system has been released it only examines BootEssentials.df when its create date changes.
5. Make & store DFIncludes.* & Dependencies.*
The file
CurrentCedar.DFIncludes gives the DF file dependencies, and results from the DFIncludes commands:
DFIncludes CurrentCedar.df
rename CurrentCedar.DFIncludes ← DFIncludes.txt
The file
CurrentCedar.includes lists the BCD file dependencies, and results from the Dependencies command:
Dependencies CurrentCedar.df
Since these files are useful in managing the release, they should be generated whenever some significant structural change in the release is complete. They take a long time to generate, so overnight generation is recommended.
6. Make & store version maps
These should be the final version maps. Their only incompleteness should be the version maps themselves and the containing DF file (SaveMap.df).
7. Restrict stores to release directory
At this point the release is considered to be final. Create access to the release directory should be limited to CedarAdministrators^.pa.
8. Send the release message
The release message announces the release to the user community. It is important to provide sufficient instructions on installation so that novices can install the release.
9. Install the release on boot servers
The boot , germ, and microcode files need to be installed on boot servers. The server to start with is Nebula, which will propagate the files to other boot servers. The program BootSmash (from
ReleaseTool.df) should be used to install the files. Here are command lines for the various machines:
BootSmash -s Dexter BasicCedarDorado.pb CedarDorado.eg CedarDorado.eb
The above line ships the boot, germ, and microcode files for the Dorado to the boot file server s Dexter (indicated by the S switch). The dates for the files are changed by BootSmash to be the current date (and time) to force proper propagation.
BootSmash -s Dexter BasicCedarD0.pb CedarD0.eg CedarD0.eb
The above commands ship the boot, germ, and microcode files for the Dolphin to the boot file server Dexter.
rename CedarDLion20.germ ← DLion.germ
rename BasicCedarDLion20.pb ← BasicCedarDLion.pb
rename CedarDLion20.db ← CedarDLion.db
BootSmash -s Dexter -~d CedarDLion20.db -cd BasicCedarDLion20.pb -~d CedarDLion20.germ
DLions are quite funny, since they do not have standard boot file names, and two of the files (boot and germ) need their checksums set (indicated by the C switch). The date for CedarDLion.germ should (we think) not be set (indicated by the ~D switch). The 20 refers to the sub-boot number (21 is available for experimental releases).
If you are only shipping new boot files things are a little easier, so the following commands should work:
rename BasicCedarDLion20.pb ← BasicCedarDLion.pb
BootSmash -n BasicCedarDorado.pb BasicCedarD0.pb -c BasicCedarDLion20.pb
Incidentally, you will need to be on the Dexter.internet list in order to have permission to store on Dexter. Have the owner of that list add you via Maintain if you need to.
Dexter should automatically ship the boot and germ files over to Ivy, although it may take quite some time (a day?). Once there, they should have the following names:
[Ivy]<System>Boot>45003-BasicCedarD0.pb
[Ivy]<System>Boot>3425-CedarD0.eb
[Ivy]<System>Boot>5000-CedarD0.eg
[Ivy]<System>Boot>45103-BasicCedarDorado.pb
[Ivy]<System>Boot>3113-CedarDorado.eb
[Ivy]<System>Boot>5100-CedarDorado.eg
10. Install the release on public machines
This is normally done by support personnel, although it is a good idea to test out installation before sending the release message. The Release Coordinator is responsible for providing directions on how to install the release.
The rules of the game
1. DF file conventions
In order to preserve the sanity of the various Cedar users and implementors, the following conventions are used for release components. It is a good idea to remind implementors of these rules from time to time.
In the following rules we use the current release (6.0) as the release version, although the reader is encouraged to generalize where applicable. Further, Component denotes the component name.
General
All submitted components must have passed verification (no errors) by the Verify command of the DFTool. This is the most crucial step in making a timely release.
All submitted components must have their top-level DF files on [Cedar]<Cedar—+Ë>Top>.
For each component named Component, there should be a command file named MakeComponent.cm (not an export). This file should contain commands that drive the compiler and binder (and whatever else is needed) to reconstruct the component from its sources.
Imports
Imported DF files should always come from some subdirectory of [Cedar]<Cedar—+Ë>, and almost always come from [Cedar]<Cedar—+Ë>Top>.
Imports should use ~= as their date format.
Exports
The
Component.df file itself should be exported, using:
... [Cedar]<Cedar—+Ë>Top>
The Tioga form of the documentation files should be exported, using
... [Cedar]<Cedar—+Ë>Documentation>
All exported files except top-level DF files and documentation files should use
... [Cedar]<Cedar—+Ë>Component>
The major public interface files should be exported, both source and object.
Files required by the implementation (such as *.tip, *.style, *.icons) to be in the local name space should be exported. These files should be commented as such in the DF file.
If the package is not in the boot file, there are two more requirements:
The main package implementation file (e.g. Compiler.bcd) should be exported only if it is not in the full boot file.
For each command that would be registered with Commander by the component's start code, e.g. "Solve", a text file named
Solve.load should be exported. This file should contain the commands necessary to get that command registered (and perhaps others too). For example,
Compiler.load contains a single command:
Run Compiler.bcd
No other files should be exported.
Documentation
The primary documentation file for Component should be named ComponentDoc.tioga.
Documentation files need only have the Tioga form.
2. Changes to components
Typically a release goes through stages where the allowed changes to components become progressively restricted. Notice should be given for every additional restriction. Approval should be sought for each exception.
Component
DF file: Component.df
Documentation: Component.tioga | see Maintainer
Maintainer: MaintainerName
This paragraph outlines the changes that the component has undergone since the last release. More complete documentation should be included in Component.tioga.
Catalog entry
This paragraph provides an abstract for the component. It is only present if the component is new or the abstract needs alteration since the last release.
Critical changes
This paragraph provides a warning about especially important changes since the last release. The Release Coordinator will place this paragraph near the front of the release message to increase the liklihood that users will notice it.