Inter-Office MemorandumToCommunication ProtocolsDateNovember 21, 1980FromEd Taft, David Boggs, Hal MurrayLocationPalo AltoSubjectEthernet Boot ProtocolsOrganizationParc/CSL, OPD/SDDXEROX Filed on: [Maxc1]EtherBoot.pressThis memo describes the protocol by which Altos are boot-loaded over the Ethernet, the protocolused by Dolphins and Dorados for loading microcode, the protocol for discovering what boot filesare available, the protocol for distributing the most recent version of a boot file to all boot servers,and the protocol for getting a boot server's statistics. It supersedes [Maxc]AltoBoot.presswhich contains only the subset of this information that is available to the universities. The protocolsdescribed in this memo have been developed over several years, so don't be surprised if thingsappear a bit haphazard.Because gateways are up 24 hours a day and are often located at places in the internet where manyEthernets come together, most gateways contain a boot server. However it is important tounderstand that boot servers and gateways are logically two very different things which are onlyphysically co-located for convenience. There are gateways which aren't boot servers (e.g. Maxc1),and boot servers which aren't gateways (e.g. Peek).Breath of LifeA Boot server periodically (every 5 seconds or so) sends a BreathOfLife packet on each directly-connected Ethernet. This is not a Pup: it is a raw Ethernet packet with the Ethernet destinationaddress set to a special value. The remainder of the packet is an Alto Ethernet boot loaderprogram.When an Alto is booted with the BS key depressed, the boot microcode enables the Ethernet receiverto accept packets directed to host 377B and copies them into memory beginning at location 1. Whena packet of type 602B is received without error, the Alto then begins executing instructions atlocation 3.The current Alto Ethernet boot loader is contained in EtherBoot.asm.MaydayAn Alto which wants to be boot-loaded broadcasts a Pup of type BootFileRequest to MiscellaneousServices sockets of all hosts on the directly-connected Ethernet. The low-order 16 bits of the Pup IDis the number of the boot file desired. The Pup source port is the one to which the Alto wants theboot file sent.Since the BootFileRequest Pup may be lost, it should be periodically retransmitted up to somemaximum number of retries if no response is received at first. EtherBoot retransmits about once a]gpi c8q]rX-q7Br \q]r-q7Br VDq]r-q 7BrOsr I% CR Bl(8 @Z ?da =J <\W : 7&; 6oF 4Q 3gG 13 -Vt *rr#u r (*7 'iG % #qr- !}&qr& qr: t P t r2 ur [q r V  ur? * T  >/^Ethernet Booting Protocols2second for about 30 seconds. The reason for giving up after a while is that perhaps it is getting noanswer because its Ethernet interface is broken and is polluting the net whenever it transmits.The standard boot loader, when started at its normal address (3), reads one of the Alto keyboardwords to determine the desired boot file number. All keys up (except BS) corresponds to boot filenumber 0. One-bits in this word are selected by depressing combinations of the following keys,listed most-significant bit first:3 2 W Q S A 9 I X O L , " ] BLANK-MIDDLE BLANK-TOPThe Dolphin microcode uses an alternate starting address (5) to get to the NetExec after themicrocode has been loaded from the Ethernet. This requires that the desired boot file number (10Bfor the NetExec) be in AC0.EFTPThe actual transfer of a boot file is accomplished using the Pup EFTP protocol. A boot serverreceiving a request for a boot file it is willing to supply simply attempts to EFTP that file to the portfrom which the BootFileRequest Pup arose.Since several boot servers may respond to a single request, a server should be prepared for the EFTPtransmit attempt to fail. When the Ethernet boot loader receives the first EFTPData Pup in responseto its BootFileRequest, it locks on to that source and ignores Pups from everywhere else. Due to spacelimitations (254 words), it is unable to respond to other EFTP transmissions with EFTPAbort Pups, asspecified by protocol.There are two timeouts of interest here: the abort timeout, within which an ack must be received orthe transfer is aborted, and the retransmission timeout, after which if an ack has not arrived thecurrent data block is sent again. Ideally the retransmission timeout should be adaptive: about 2times the average response time, exponentially aged over the last 8 samples. In any case it shouldbe such as to retransmit a few times before the abort timeout takes. The abort timeout should be afunction of the available bandwidth of the path between the sender and receiver. The table belowlists recommended values.First BlockSubsequent BlocksAbortRetransAbortRetransFast net 500 100 50001000Slow net100002500100002500(I need to double check these. /HGM May-80)The timeouts for a slow net are suitable down to about 2400 bits/sec. The retransmission timeoutslisted are for EFTP implementations which do not use an adaptive algorithm; the initial adaptiveretransmission timeout may have to be reduced from its default value (typically 1 second) for thefirst block on a fast net. A reasonable simplification is to assume that all nets except Ethernets areslow. Even on an unloaded 9600 bits/sec line it takes several minutes to send a full core image bootfile. Boot servers should be able to boot an Alto over an Ethernet while simultaneously updating aboot server at the end of a slow phone line.Boot File Names and NumbersString names and 16 bit numbers are both used to refer to boot files. Servers deal mostly in bootfile numbers: requests to send a boot file refer to it by number; servers compare the creation datesof files with identical names and numbers when distributing new versions. The NetExec sorts itsdirectory by name, keeping the number, date, and server host address as auxiliary information.Boot file numbers less than 100000B have a uniform meaning throughout the network, are updatedautomatically and are assigned by administrative fiat. The remaining numbers are available for local frG bb `6) ]G \1>qr ZK Y)"VgqA Sr'5 R"20 P Lt I-rAqr G)&qr F$ur C@;%q ArLur @7ur< >:qr ur =/ :Kv r) 8 vr+ 7B U 5L 4:c 2a 12.M . ,#.5  )$.6` (=$.6` &q, $r(: "qr= !%< M  U a , vt rR  N X L  qr; 05 U>/]tEthernet Booting Protocols3use and do not propagate. [Maxc]BootDirectory.txt is the master directory of registeredboot files.Most gateways have a boot file with the name .boot, with number 100000B. This isintended for use by people developing new boot files. A test version of a boot file stored therewon't propagate. It is also handy for experimenting and glitch chasing since there can't be anydoubt about which boot server it came from.Boot Directory InformationWhen the EtherBoot mechanism was first developed it was only expected to handle a small numberof files -- DMT, Scavenger, FTP and a few others -- and key combinations were picked that were easyto remember and convenient for people with two hands and ten fingers. Even so, it was difficult toremember the keys and the number of files grew to the point where this scheme was getting out ofhand, so the NetExec was developed. The NetExec was assigned one of the last convenient keycombinations and it is now the standard way for humans to invoke other boot files.The NetExec discovers what boot files are available by broadcasting a Pup of type BootDirRequest toMiscellaneous Services sockets on all hosts on the directly-connected Ethernet. Hosts that are bootservers respond with packets of type BootDirReply containing tuples. A boot server with lots of boot files may fill several BootDirReply Pups. The NetExec buildsa directory of tuples from these responses.Boot File UpdateAt present there are two programs which implement boot servers: Gateways and Peek. There areabout 20 gateways in operation, and the number is growing. It takes a few minutes per gateway toupdate one boot file (most gateways are at the end of slow phone lines). Not all gateways are up allof the time. There are probably 50 Peek disks in the world, each with some subset of the boot filesthat existed when the disk was built. The owners of Peek disks are exhorted to rebuild their disksabout once a month. The result of this anarchy is that old versions of boot files persisted in theinternet for years.A Boot file now includes the date on which it was built. Boot servers periodically exchange bootfile directories which include these dates. When a new version of a boot file is stored onto any bootserver, all other boot servers will soon discover this and automatically update their local copies. Theprotocol is similar to that used by name servers to update the network directory.About once an hour and each time a new boot file arrives, each boot server broadcasts its boot filedirectory in a BootDirReply Pup to miscellaneous services sockets on all directly connected networks.When a boot server receives one of these, it compares the dates of its local boot files with the datesin the Pup. If the sender has a more recent version, then the receiver requests a copy using aBootFileRequest Pup. If the receiver has a more recent version, then it should send a BootDirReply tocause the sender to update. The same EFTP protocol that is used to boot an Alto is used to movenew versions among boot servers.If the date comparison is unguarded, a damaged file with a bogus date far in the future couldpropagate everywhere and it would be impossible to purge. To protect against this, a file whichclaims to have been created in the future should be treated as if it had a date of zero, thus making itelegible for update by anyone.The automatic update mechanism has turned out to be quite robust. In fact, it is almost impossibleto switch back to an old version of a boot file if a new one doesn't work correctly. If, for example,you use FTP to store it on a boot server, as soon as that server gets restarted it will notice that it hasan old version and overwrite it with the (buggy) newer version from some nearby boot server. Toavoid this problem, GateControl smashes the date to "now" when it sends a file with a name endingin .boot. There is no corresponding trick for microcode files or Pilot boot files. You must rerun the frG bU ` ]:qr \1N Z?! Y)+ Tt QrB P4 qr qrD N4/ M,M KT J#R G?3u r E+9 D7%u r B@u r A.Z /V!4Ethernet Booting Protocols5Booting Dolphin MicrocodeThe hard part of loading microcode into a Dolphin is deciding which microcode to load. Theoptions are two dimensional: which microcode device drivers, and which emulator(s)? To avoidwhat might otherwise become unmanagable chaos, the emulator microcode is packaged intointeresting combinations, and loading microcode is split into three phases.When you poke the boot button, the Boot microcode is copied from an EProm into control store. Itchecks the processor a bit, and then tries to load the Initial microcode from the disk. If it runs intoany problem, it tries to load Initial from the Ethernet. The normal trick for forcing a Dolphin to Etherbootis to cycle power off for a few seconds. This makes the disk appear not ready for a minute or so. In either case,the same code (except for a different starting address) is loaded directly into control store. Initialsets up memory, and if it was Etherbooted, it figures out which variant of the final microcode toload. Currently it does this by assuming that you want the Alto emulator, and looking at the clock speed on thedisplay controller to decide if you are running on a SDD Dolphin, a CSL D0, or a TOR machine. The desiredemulator microcode is then loaded into main memory. It is started using LoadRam. LoadRam hasbeen standardized across all microcode variants to simplify this process.EtherBoot is the module in the EProm and loads Initial from the Ethernet. When control is passedto EtherBoot, it first dallys for a second or so to let you read the MP. If the BootReason registercontains anything other than a push button boot, the delay is lengthened to a minute or so to avoidflooding the Ethernet if the Ethernet board is sick. (If the Ethernet board causes an H4 parityerror, for example, the fault handler will put a number into the MP, dally for a second, and thenreboot the machine.) EtherBoot then locates the Ethernet board, and sets up the task asignments.The emulator task, sets a timer to 10 seconds, starts the input task (which will initialize itself andwait for a packet), and then pokes the output task. The output task broadcasts a request formicrocode file number 0, and then goes to sleep. (It's collision processor is quite primitive.) Whenthe first valid packet arrives, the input task will latch onto the the server's host number. Sincememory has not yet been set up, the data words are stored directly into control store. Notice thatthe EProm must be careful to avoid overwriting itself in case the packet is bad. The problem isthat the data must be stored somewhere before the status of the packet can be checked. In order tosimplify things for EtherBoot, the boot server sends Pups that contain 3*n data words. Each time a packet arrivescorrectly, the input task resets the timer to 2 seconds. As the data arrives, a checksum isaccumulated. When a jump block is encountered and the checksum is ok, Etherboot jumps to thedesignated location. If anything goes wrong, for example a CRC error, the input task puts anumber in the MP and goes back to the beginning. In the background, the emulator task isdecrementing the timer. If it expires, the emulator will retry everything up to 15 times.EtherLoad is the module in Initial that loads the emulator microcode from the Ethernet. It is verysimilar to EtherBoot, in particular, it does not use the normal Ethernet microcode. EtherLoad's taskis a bit simpler that EtherBoot's because it simply places the data into memory. When an empty (0data bytes) Pup arrives, EtherLoad checks the checksum, and it if is ok, it jumps to LoadRam.Booting Dorado MicrocodeNeedless to say, booting a Dorado is quite complicated. This description is quite brief. For moreinformation, see Ed Taft's memo.When a Dorado is powered up, a microcomputer in the baseboard goes into action. If all goes well,it manages to get Initial (which is stored as data in a prom in it's address space) into the Dorado.Initial then initializes the map and main memory, turns on the normal IO devices, and inspects thekeyboard to determine which variant of the emulator microcode to load. Initial then uses themicrocode booting protocol to load the desired microcode into main memory and then usesLoadRam to load it into control store. The emulator microcode will boot the software from the diskor Ethernet by testing the keyboard just like the Alto does. frG bt ^rC ]nT [2$ ZfK W;v r0 U7vr T3. q3 R4.r Q+70 OD N#q'B LAr Kvr: II FkJ DP CcK A5+ @[D >$= =Sb ; S :Kf 8F 7BO 5Q 4:Xq 2.(r 12H /K .*J ,%4 +"Z 'I &s [ $b #jA t rY u P  L F &7 M Z x<R 1?/YyEthernet Booting Protocols6Microcode FilesBy convention, the names of microcode boot files end in .eb. In order to use the normal updatemechanism to propogate new versions of microcode around the internet, microcode files aredisguised as normal Alto format boot files. See the previous section on Boot File Formats for thedetails of the first few words. For simplicity, the server skips over the whole first page before itsends out the microcode, so there is plenty of spare room for other things. The version 1 serverrequires that the first word of the boot file contain a 1. This restriction is likely to get removed or updated tocontain more information, for example, the type of the target machine. The format of the rest of the filedepends, of course, on the target machine.To avoid bolting arbitrary large constants into Proms, the boot server adds an offset to themicrocode file number to translate it into a boot file number. Currently the offset is 3000B. Thecurrently assigned microcode file numbers, the corresponding boot file numbers, and the file namesare: 0 3000 Initial.eb (9 packets) 1 3001 AltoLF.eb (45 packets) 2 3002 AltoCSL.eb 3 3003 PilotLF.eb 4 3004 PilotCSL.eb 5 3005 AltoTOR.eb 7 3007 PilotTOR.eb100 3100 DoradoMesa.eb101 3101 DoradoSmalltalk.eb102 3102 DoradoLisp.eb103 3103 DoradoCedar.ebPilot Boot Files and the Ether GermFor bootstrapping and/or working on a disk that has not yet been formatted, it is very convient tobe able to load Pilot boot files over the Ethernet. To support this, the Germ (Pilot boot loader) hasan optional Ethernet driver. It uses the normal BootFileRequest/EFTP protocol. Pilot boot files arefrequently larger than a full Alto memory. This is harmless since they won't run on an Altoanyway. By convention, the names of Pilot format boot files end in .pb and the Germ is calledPilot.eg.MesaNetExecThe MesaNetExec is very similar to the BCPL NetExec. In addition, it contains logic to load Pilotboot files. If the desired file name ends in .pb, the MesaNetExec doesn't actually load it, butinstead it uses the normal BootFileRequest/EFTP protocol to load the Germ and the appropriatemicrocode into main memory. Next, it moves the Germ to hyperspace and then uses the LoadRaminstruction to load the control store with the new microcode. When the new microcode is started, itwill start running the Germ which will finally load the specified Pilot boot file.If your Dolphin normally runs in Alto mode, this is the way to get to Othello so that you can installnew microcode on your disk. You can also use the MesaNetExec's SetVersions command to selectspecial debugging variants of the Germ and/or microcode.Server StatisticsBoot servers may optionally keep statistics on their activities and make them available through thenet. A program requests a boot server's statistics by sending a Pup of type BootStatsRequest to themiscellaneous services socket, and the boot server responds by sending a Pup of type BootStatsReplycontaining the statistics. The first word of the reply is a format version number which isincremented whenever the format changes. frG bt _9rQ ]&3 \1E Z [ Y)C W": " V !6R Q7. -q r I8 t r_ T3ur ?u LrA ( R >/ZCEthernet Booting Protocols7Pointers to other DocumentationWhen an Alto is hardware-booted over the Ethernet, all three of the steps (BreathOfLife, MayDay,EFTP) are executed. A software-initiated boot may be accomplished by copying the boot loader intopage 0 and jumping into it, thereby starting at the "Mayday" stage with the boot file number andhost as optional arguments. Further information may be found in the "EtherBoot" packagedocumentation on [Maxc]EtherBoot.tty. The standard Alto Ethernet boot loader can load only B-format boot files. A boot server musttransform S0-format files into B-format files (by rearranging pages) before sending them. Furtherinformation may be found in the "BuildBoot" subsystem documentation on[Maxc]BuildBoot.tty.[Ivy]DoradoBooting.press contains detailed information about booting Dorados,including descriptions of when and how they get microcode and/or boot files from boot servers.DetailsA Breath of life packet is a raw (non-Pup) Ethernet packet:Destination:377BType:602BContents:A boot loader program.The starting address of the boot loader is the the third word of the packet (first contentword) which will be address 3 in Alto memory. The total packet length must not exceed256 words.Boot servers listen on the Miscellaneous Services socket (4) and handle some or all of the Pup typeslisted below.BootFileRequestPup Type:244BPup ID:Low 16 bits are the boot file number desiredPup SPort:The port to which the boot file should be EFTPedPup DPort:Miscellaneous servicesPup Contents:NoneThis packet is generated in several contexts: 1) by the Ether boot loader while booting anAlto, 2) by a boot server to update a local copy of a boot file, 3) by the MesaNetExec to getcopies of the Germ and the appropiate microcode, and 4) by the Germ to load a Pilot bootfile.MicrocodeRequestPup Type:264BPup ID:High 16 bits are the protocol version number (currently 1)Low 16 bits are the microcode file number desiredPup SPort:The port to which the microcode file sentPup DPort:Miscellaneous servicesPup Contents:NoneThis packet is generated by Dolphins and Dorados that need microcode.MicrocodeReplyPup Type:265BPup ID:High 16 bits are the protocol version number (currently 1) frG bt _9r(8 ]qr1- \1 V ZH Y)0 VD= TZ S<8+9 Q N<6=# MO@ Ht Er!v rB qAurq?r= K;.(: 7V 5 2u/rq.Mrr,, 'qr+E  ) r&?%X R#M "P kurqrr:~r1 &v   r E )u  Drq rr: y>/]AEthernet Booting Protocols8Low 16 bits are the packet sequence numberPup SPort:The port from which the microcode request was sentPup DPort:Miscellaneous servicesPup Contents:data, or empty for an end markerA cluster of these packets is generated by boot servers in response to a packet of typeMicrocodeRequest.BootDirRequestPup Type:257BPup DPort:Miscellaneous servicesPup Contents:NoneThis packet is generated by the NetExec to discover who the boot servers are and what filesthey have.BootDirReplyPup Type:260BPup ID:if it is in reply to a BootDirRequest, the ID should match the request.Pup Contents:1 or more blocks of the following format: A boot file number (thenumber that goes in the low 16 bits of a BootFileRequest Pup), an Altoformat date (2 words), a boot file name in BCPL string format.This packet is generated 1) in response to a BootDirRequest, 2) gratuitously broadcast everyhour, and 3) in response to a BootDirReply advertising an older version of a local file.KissOfDeathPup Type:247BPup DPort:Miscellaneous servicesPup SPort:The BootFileRequest Pup is sent to SPort.host on the local Ethernet. IfSPort.host is zero, it is broadcast.Pup ID:The low 16 bits contain the boot file number to put in theBootFileRequest Pup.Pup Contents:NoneDMT is the only program which currently responds to KissOfDeath Pups and is used now onlyto run tests on the Ethernet. We have a multiprocessor with about 125 6-MIP CPUs on thesecond floor of Parc which is idle 16 hours a day just waiting for someone to figure out howto use it.BootStatsRequestPup Type:253BPup DPort:Miscellaneous servicesPup Contents:NoneBootStatsReplyPup Type:254BPupID:same as the BootStatsRequest that triggered it.Pup Contents:A version number to identify the format of the following words(current version = 1). Followed by the number of boot files sent,followed by the number of boot directories sent, both in BCPL doubleprecision format. frGbr*` /_  ] rZWY)ur VDu S_rrqQr PW MrHK I u F$rqDru rqrC )A&ur @ qr=/,u r!;u r 8u 5rq4^r 2 ur11U$/.Mur, )qr$ u r(` =qrqr&M%X "surq r   u rq9r ur .1<#qr) 99Y)eEthernet Booting Protocols9Revision HistoryOctober 17, 1976First releaseMarch 9, 1978Boot directory protocol added.December 31, 1978Automatic update protocol added.February 13, 1979Boot server statistics protocol added.May 15, 1980Microcode booting added, file name changed from AltoBoot to EterBoot.November 21, 1980Dolphin EProm requires second word of source socket to be 4. frG bt _9r\T Yo V SP MJ& H E-E BI?d<z =0*rD TIMESROMAN  TIMESROMAN TIMESROMAN LOGO TIMESROMAN  HELVETICA TIMESROMAN GACHA   $ / 8?EGj/J HDEtherBoot.bRAVO Murray, HalNovember 21, 1980 2:06 AM