-- PilotGV: Update.log, HGM, 16-Dec-84  1:57:10

Klamath update and first round of (trivial) fixes - August '84:

changes required (and things to note like short ptrs and strings):
 
  General fixes:
	*Make start up all be interactive rather than going to the debugger. 
	*Make heap size be a parameter.	(remove boot switch)
		This code assumes .data file already exists (but not chain).
			So GVSwitches creates the heap file for you.
	*Display Inboxes will now display an individual mailbox.
	*IdleImpl - copy new code from activity.Impl.bcd
  	VM leaks: Log.mesa  DoSpill needs some unwinds to ReleasePate 
	   Not sure about archiving stuff (mailboxAlloc and Spiller).
	   ReadPage, AllocatePage, and UsePage need Release or Deactivate
	*"Hello" message on the Chat connection now includes the up time. 
	      That will make HostWatcher messages a bit more informative.  
	*Log msg for Archiver starting, Archiver finished. 
	   (start msg says how many days it's going to trim to.)
	*Removed all STOP statements and recompiled all w/ released compiler 
	*Increment the number of processes
	*Eliminate duplicate server config.
	*Recompiled with initialized checking. 
	  (perhaps check for things that count on init to 0 - didn't do).
		
 Specific changes:
  
  BitMapDefs removed Bravo formatting, has a short pointer.
  BodyDefs	has short STRINGs
  BTree2Defs: 	has short POINTERs
  BTreeSupportDefs:	 removed Bravo formatting 
  	(renamed it so that internal name same as file name), 
	 has short pointers
  FrameDefs.IsBound should take a LONG UNSPECIFIED rather than an UNSPECIFIED
  GlassDefs: has short STRINGS and ptrs
  HeapFileDefs: added some procs from HeapFile for use by HeapRestart.
  IODefs: has shorts
  Leaf.AnswerObject has op multiply defined
  LogDefs: has STRINGS, PTRs, and opens
  RestartDefs: MTP and RegRestart now export two init procs instead of a program.
  SequinPrivate - BufferDefs => Buffer (Buffer & QueueObject types should be same). 
  	Took out RequeueClosure stuff and made Requeue itself public
 	 new var bufferPool (pool of buffers created in init)
  	has LONG ptr
  Storage - has short POINTERS and STRINGS 
  TimeExtra - has STRINGs
  TimeExtraDefs looking for TimeExtra.PackedTimeFromString (LONG STRING)
    sol'n: add TimeExtra (on Ibis/grapvine>ftp) to cm/df (wasn't there)
  TimeExtraImpl looking for TimeExtra (exports PackedTimeFromString),
    rework of TimeZones: zone went from +8 to 8 west (?)
    (changes E,C,M,P to 5-8west else to GMT)
  VMDefs haslong strs and ptrs
  
  Bitmap: has  short prs
  BTree2:  has short ptrs
  BTreeDefs - remove Bravo formatting
  BTreeSupport:  has short pts
  EnquiryDefs: changed def of MailboxCount to do specify indiv. mailbox
  FileDefs:  has short ptrs
  GlassImpl:	has short ptrs
  IdleTime - Completly rewrote based on ActivityImpl.
  Log:		has shorts
  	add unwind to release VMPages if remote file transfer fails.
  LogPrivateDefs:	"
  MaintainDefs		"
  NameInfoDefs: has short strings
  ProtocolDefs		"
  RetrieveDefs:	"
  SendDefs:	"
  SequinImplA:  BufferDefs => Buffer, pup changes (creating a buffer pool)
  	these impls now use Buffer.BufferObject.requeueData instead of RequeueClosure stuff.
	has short pointers
  SequinImplB:  BufferDefs => Buffer, pup changes
  StorageGVImpl need to change POINTERS and STRINGS to LONGs 
  VMStorageMgr:  Space use needs to be updated. Changed Create/Map to
     Allocate/MapAt (this might want to be redesigned to just Map.
     has short POINTERs (so uses MDS)
     gives warings about something being truncated.  Should be ok. ?
  
  Enquiry: this references a proc. MaintainDefs.DoIt which no one includes
   	(But it seems prepared for this as it checks to see if bound first).
  	added a line to Chat hello msg to print uptime.
	changed call to DisplayInboxes to specify one or all.
  	has short STRINGS, proc var change size, LOOPHOLE for CheckImpl args.
  HeapDefs: has short pointer in Buffer: Reader/Writer Handle, Buffer.
  HeapFile: made some procs public so HeapRestart need not share this module
  	Incorporating our fixes for AR 7507 -
		
  	rmoved InitialisingHeap error, will not do to debugger now unless
	  client types N to "DoYouKnowWhatYou'reDoingandWantToProceed" query
  Locate: pup changes, needs a buffer pool.
  MDSWatcher:  Buffer and Space changes.  has short strs   
      this has a process which runs thru MDS and calculates how much is free.  No easy way to do this so will leave out the figure on MDS space.
     this also tells you how many pup buffers are avail. - calc from BufferMgr.sysPool.  
     initialize var storageFaults.
  RListDefs: has short pointer
  PilotFileSystem:	-- file access rework, space rework, made look more like NSFileSystem (esp. PageIOProcess).  Had to move UNWIND to outer block in Open.
  
  IFSFileControl:  Runtime loopholes
  LocalName:	has short strings
  NameInfo: has shorts
  ObjectDir:  made some proc types and vars so HeapRestart can share this module and access some procs.
  Restart: won't accept '\ so put in 134C.
  	LOOPHOLEs for Runtime.IsBound
	proc calls instead of START/RESTART for MTPServer and RegRestart
	remove a parameter to PupDefs.AdjustBufferParms
	* this references a module Test which is not around.  But explicitly checks if bound so ok.
  RetrievePoll: PupDefs rework	(created a BufferPool, correct nos.?)
  VMControl: Runtime loopholes
  HeapRestart:  can't import procs from modules any more, so work around.
  RegAccessDefs:
  RegServerDefs: change export of Registration program (no more STOPs)
  SLDefs:  add thing for SLRestart to use.
  
  GVSwitches:  MFileExtra => SpecialMFile.  New parameter to MFile.Acquire
  	Removed 'l switch -
	 it now prompts for heap size if booted with 'i switch
	  (so you must not set that as a permanent switch else nuisance)
  MailboxAlloc:  proc. var's to allow MailboxRestart to call internal procs.
  	changed MailboxCount to work for an indiv as well as all mailboxes.
  MiscSoc:	BufferDefs stuff    (created a BufferPool, correct nos.?)
  MTPServer: 	compiler error	(doesn't like STOP - Brian made new compiler but can't use (not released) Igor>Ramona>compiler>backup>compiler.bcd) so replaced with two init procs.
  ReadForward:	BufferDefs, created a bufferPool.
  RegAccess: remove stops and make them procs.
  Registration: PupDefs.ReturnFreePupBuffer
  	remove stops and make them procs.
  RegMail: remove stops and make them procs.
  RegRestart: remove STOPs and RESTARTs and replace with proc. calls.
  SLQueueImpl:	renamed file SLQueue to SLQueueImpl. Moved restart proc to here 'cause to complicated to export short pointers needed.
  RegRestart:  Removing the signal IsThisTheFirstRServerInTheWorld and making it an interactive query which will then call InitializeWorld on its own
  MailboxRestart:  reference new procs. in MailboxAlloc
 SLQueueRestart:	moved restart code to SLQueueImpl.
  MyKluge:	exports some proc that someone calls but doesn't need.
  VM.config:	rename BufferDefs => Buffer, remove Transaction
  ServerMisc:			"		"
  tweaks to various configs; Removed extra copy of server code; 
  new bootmesa based on new pilot.bootmesa;

Second round of changes (mid August '84):

  General:
     ServerProd:  made it not answer unless willing to accept mail 
       required fixing Policy.CheckOp to not always set
     Incorrect handling of updates for domains not served (so can't remove a domain) 
	    If registry doesn't exist then delete msg.  
	    If just entry in a known domain doesn't exist, must keep. 
     MCHCommon fixes (AR 7507)
	  Writer.SetWriterOffset - must put gap prior to moving offset
	    doesn't seem to be a problem here as they call TerminateObject.
	  HeapFile.CommitObject - make sure endOfChain ptr 
            (end of new chain which points to -1) is forced out.
     Message timeouts when server been down long period of time:
	  Disabled timeouts for first 6 hours anytime server is rebooted. 
	  This doesn't hold problems for old updates does it? 
	  Should we have some max limit which won't allow more than?
     Echo prompted information.
	
  Specific:	
    PolicyDefs/Policy: give CheckOp and additional arg to not set PC (default to do so).
    HeapRestart and RegRestart: echo info obtained with TTY.GetChar.
    ReaderAlloc: HeapStartRead gets unwind since we've seen the ML disappear
    MiscSoc: AcceptPoll, echoMe takes into account value of serverInput P.C.
    HeapFile:	make sure commit writes out end of new chain.
  	looks like there might be a problem with singlePage writers -
		 rewrite with our new stuff?
    RegServerDefs: UpdateSublist has new return value, everyone ignores it
  	 except AcceptBriefUpdate which needs that info
    RegServerImpl: change decl and use of UpdateSublist to return whether the
  	 registry is served locally if entry not found.
    RegMail:  change AcceptBriefUpdate to pass back new info so
  	 RegMailReader can throw away updates for non-local registries.
    ReadInput: add code to not timeout messages 'til server up for 6 hours.

Third round of changes:
  General:
    Add heap expansion option at restart (boot switch 'l)
    Rearrange grapevine directories on Ibis.  
      Pull over all unforked modules (not many left)
  Specific:
    GVSwitches: rearranged some things
    HeapRestart:  added heap expansion code.
    Enquiry:  change call from FrameDefs to Runtime.IsBound 
    FrameDefs:  get rid of. (not nec.)
    RestartXDefs:  gone away
    RestartDefs:  merged RestartXDefs into (one decl)
    MDSWatcher now exports to RestartDefs not RestartXDefs.
    Restart now uses RestartDefs not RestartXDefs
    TimeExtraDefs:  deleted
    FTPServer: use of TimeExtraDefs changed to TimeExtra

Fourth round of changes:
   Fix my error in MiscSoc relative to new ServerProd response
   Fix Locate.FindNearestServer to only let three callers thru at a time.
     This calls call things which fork off lots of processes which don't get joined 'til done and only one call can complete at a time anyway.  Will allow 3 so set up and clean up can overlap.
   
Fifth round of changes (EPW):
   Fix above previous fix to Locate/NameInfo bug in GVUser package.
   Set MTP Policy control to FALSE.
   Fix UNWIND bug in Log$DoSpill.  Was incorrectly releasing "from" not "to".

Sixth round of changes:
  General:
    Remove MTP?  
		DLFetcher (just for it), MailBoxDefs, MTPServer, Policy,
	        PolicyDefs, ReadForward, RetrieveDefs, RetrieveInit,
	        RetrieveMail, RetrieveMTP, RetrievePoll, RetrieveXDefs,
	        ReturnDefs, ReturnMail, SendDefs, ServerAlloc, ServerDefs,
	        SiteCache, SiteCacheDefs, Spiller.
    remove FTP and replace with Hal's STP?  
		See what we use it for (MTPServer)
		see if it uses our stuff (leave it out and see what happens)
		(check Idun>apilot>PupGateway.df for imports: STP and STPs.bcd  
		rewrite GateFTPServer for our use.??  
		See if STP server runs w/o window stuff.  
		Add bootswitch to start with only STPServer running.
		FTPServer is the only file that is different.
		Will try leaving FTP stuff out and see if anyone relies on it.

  Specific:
  
  
Things left to do:
	Make PilotGV run with a LS instead of a display.
	Cache 2 streams in NameInfo, 1 local 1 remote.
	Keep MS reading it's mail from hogging policy control.  
	  Make it wait at least once around each loop of reading.
	  RegMail.RegMailWatcher and MSMail.FetchMail 
	    are in a Wait, Do, End loop but maybe just too fast? 
	      Want to put a pause of a minute in there?
	    Think problem is that FetchMail fails and loops until empty.
	    Need more data on just where the problem is before can fix.
	Priority given to PendingQ instead of Input - change?
	Find all other 'normal' calls to debugger and eliminate 
	  in possible effort to eliminate need for a debugger volume?.
	Display InBoxes should list the file name(s) for archived mail (what mean?)
	Get Randy to point out connection leaks.
	Bitmap.FindFree - 
	  maybe client should check to see if page returned is valid.
	Why does Hello msg often come out incorrectly on pilotgv servers?
		something to do with auto login packet confusing server
		  (sst of 3 looks like a control C  ???)
	there appears to be something wrong with ServerProd such that doesn't
	      always mark as up even tho' other guy answered 
	      (did packet get back and was just not processed?).
        June 19th msg about FTP/MTP	(where?)
	We've seen the ML lost in PilotFileSystem (all have unwinds now),
	   SiteCache (don't see how) and ReaderAlloc (added one unwind).
	Also seen a SS leak on Riesling - free map was full but freeCount = 2
	Remove obsolete/unused stuff? temp statements (tricky), uncalled procs?
		RegPurger, what is, how work, do right thing?
		Msg of Jul 26	- regPurger, what's it spending it's time on?
	Convert to LONG STRINGS and LONG POINTERs?  (VM problem for one).
	  see about specifics for which modules have short strs and ptrs
	Allow reading of mail while its reading its own?  
		(Does this have problems with CHS inconsistencies 
		 and it not being able to ever get caught up?)
	 July 18 msg from Murray about reading its mail:

	   "abandoned reading" log msg means that the free heap is < 5%. 
	   The MSMailReader crawls into a hole for a while and waits. 
	   When it comes out, it starts all over at the beginning of it's mailbox.
	     (Andrew mentioned this, I just didn't recognize it the first time.)

	   When the remailer (aka MSMailReader) finds a broken mailbox, the IFS
	   gets marked as down. The remailer then skips over any entrys with
	   archived mail (but doesn't delete the msg in Merlot.ms) and plunges on.

	   After a while, the IFS gets marked up. 
	   That lets the remailer continue processing archived mailboxes again.

	   Evenutally, it gets to the end of it's mailbox. Ha. 
	   We've only gotten half way through. With 4000 messages to process, 
	   it always finds more work to do, and/or trips over a sick mailbox,
	   and/or runs out of disk space, and/or ... 

	   I think our best plan is to let Merlot crunch/thrash as best it can.
	   As long as it's making progress and isn't mucking things up too much
	   we are gaining. Then what's left will be broken mailboxes or mailboxes
	   in the shadow of a broken one. We can find them by watching the log.
	   I've already  trimmed the shadow several times by poking the
	   ProdServers process.

		

HGM, 13-Nov-84  4:04:16
	Convert to Calaveras: Run FixArrows and Formatter
	 Find '''­ *.mesa =>
	  CreateMail(Express check)
	  DLFetcher(FetchDL: defaulting registry)
	  Enquiry(Enable: Transport^.ms)
	  FTPServer(CheckAccess: Transport^.ms, LogReaders^.ms)
	  GVPServer(CheckAccess: Transport^.ms)
	  MTPServer(Recursion test ??)
	  RegServerDefs(upArrow comment)
	  RegServerImpl(several...)
	  SiteCache(several...)
	Enquiry: Add SendNow to Display Inboxes. CR for All.
	MSMail: Add 3 minute wait after processing mail.
	RegMail: Add 3 minute wait after processing mail.
	ReadInput/Restart: Activate calls to DisplayNumber, defer StatisticsOn.
	
HGM, 15-Nov-84 18:20:57
	More work on Restart/Stats timing.
	  (trying to get 3 DisplayNumbers from ReadInput to work.)
	Line up columns in Display Statistics.
	Added ShowLine and ShowNumber to LogDefs.
	
HGM, 11-Dec-84  0:52:29
	More printout to the TTY.
	Added statistics printout to Compactor.
	Flushed Lily hooks.
	Flushed MTP (LOTS OF CHANGES)!
	More Sequin Buffers.
		
HGM, 16-Dec-84  1:57:05
	Add PupEcho, PupRoutingTable and PupStatistics commands.
	Add Disk Error Watcher.
	Major rework of Policy stuff for Express/Pending.
	  Pending now waits if forwardQueue > 10.
	  Express now waits if inputQueue > 0 or forwardQueue > 5.