Number: 2204

Date: 19-Sep-84 18':00':18

Submitter: Sannella.PA

Source: Newman.pasa

Subject: Extreme thrashing in LFG sysout

Assigned To: 

Attn: Kaplan, Lisp

Status: Open

In/By: 

Problem Type: Performance

Impact: Moderate

Difficulty: 

Frequency: 

Priority: Perhaps

System: Operating System

Subsystem: Virtual Memory

Machine: 1132

Disk: 

Lisp Version: 19-Sep-84 13':04':52

Source Files: 

Microcode Version: 5124

Memory Size: 4096

File Server: 

Server Software Version: 

Disposition: 

Description: '
Date': 12 Sep 84 17':16 PDT'
From': Newman.pasa'
Subject': Lisp': SLOW SYSOUT'
To': LispSupport.pa'
cc': Newman.pasa'
Lisp-System-Date': 21-Jun-84 10':50':28'
Machine-Type': Dandelion'
'
WHO KNOWS WHAT MIGHT CAUSE A RELATIVELY SMALL SYSOUT  TO GET SLOW?  We are talking so slow that I eventually booted out of it because I was getting absolutely no response. It was generated from {rosebowl}<lisp>carol>full.sysout, and its VMEMSIZE was no more than 9100 pages the last time I checked before bailing out.  In addition, the last time I got a response from control-T (and several times previous to that) I was seeing numbers approaching 98% in the Swap category and 1% in the Util Category. The sysout seemed fine when I first came into it, but got progressively worse and worse. I am pretty sure that the size of the Virtualmemory was not the problem because I have seen sysouts with larger Vmemsizes which ran faster than that one did.'
'
'
   HELP !'
'
'
  >>Dave'
'
-----'
'
Date': 13 Sep 84 10':23 PDT'
Sender': Sannella.pa'
Subject': Re': Lisp': SLOW SYSOUT'
In-reply-to': Newman.pasa''s message of 12 Sep 84 17':16 PDT'
To': Newman.pasa'
cc': LispSupport.pa'
From': LispSupport.pa'
Reply-To': LispSupport.pa'
'
When the system spends an overwhelming amount of its time swapping ("thrashing"), this indicates that the working set is fragmented.  In other words, the information used in a computation is spread over so many pages that the system has to keep swapping to access all of the data.  The larger the virtual memory size, the more likely that you will have working set fragmentation, but it is not NECESSARY to have a large vmem for this to occur --- it can occur whenever the virtual memory is larger than the real memory.'
'
It is impossible to diagnose what caused your system to swap so much, without more information.  How long had you been working in the sysout before it got slow?  What computations were running when its response got so slow --- could there be other processes that are causing a lot of swapping?  Do you have a set of files which, when loaded, cause this behavior?'
'
-----'
'
Date': 13 Sep 84 12':49 PDT'
From': Newman.pasa'
Subject': Re': Lisp': SLOW SYSOUT'
In-reply-to': LispSupport.pa''s message of 13 Sep 84 10':23 PDT'
To': LispSupport.pa'
cc': Newman.pasa,Sannella.pa'
'
I had been in the sysout for only one day. Actually this has happened to me twice. Both times I loaded the sysout and was having severe problems by the end of the day. The sysout is {dante}<linguini>lfg.sysout, and it contains {rosebowl}<lisp>carol>full.sysout and all of Ron Kaplan''s LFG files (the dcom versions without databases). It also includes some files from the linguini directory on dante. (linguinieditaux and linguinideletemenu) At the time the sysouts got slow, they also included some stuff that I am working on, but were not very large.  Is there a function that one can call to de-fragment the memory?'
'
 The process running were LAFITE and the basic system processes and TEDIT. I don''t know if the processes were competing with each other so much that they caused the swapping or not ... might this indicate a problem with the schedluer algorithim?'
'
  Lastly, this happens only intermittently with this sysout as far as I am aware.'
'
'
  >>Dave'
'
-----'
'
Date': 18 Sep 84 17':18 PDT'
From': Newman.pasa'
Subject': THRASHING'
To': Kaplan.pa,Sannella.pa,Lispsupport.pa'
cc': Newman.pasa,Teditsupport.pa'
'
I think that the thrashing problem that I have been experiencing in the LFG sysout that I built has been caused by TEDIT and LAFITE. In using the STORAGE function, I think that I see increasing amounts of storage lost as TEDIT and LAFITE are being used (in what I think are  TEDIT datatypes).'
'
Correct me if my interpretation of STORAGE''s output is incorrect':'
'
  Assigned pages --> Number of pages of virtual memory assigned to store items of a particular datatype.'
'
  Assigned items --> The number of items that the assigned pages can store of a particular datatype.'
'
  Free Items --> Number of items of the datatype on the free list.'
'
  In use --> Number of items of the datatype currently in use by the system.'
'
  Total alloc --> Number of items allocated by the system.'
'
'
Hence, the  difference between Total Alloc and In use is the number of items of the datatype that are "lost". The datatypes that I am looking at are ': PIECE, SELECTION, LINEDESCRIPTOR, CHARLOOKS,  FMTSPEC, IMAGEOBJ, TEDITCARET, and LINECACHE.'
'
'
Is this a reasonable interpretation ???'
'
   >>Dave'
'
-----'
'
Date': 18 Sep 84 17':32 PDT'
From': Kaplan.pa'
Subject': Re': THRASHING'
In-reply-to': Newman.pasa''s message of 18 Sep 84 17':18 PDT'
To': Newman.pasa'
cc': Kaplan.pa, Sannella.pa, Lispsupport.pa, Teditsupport.pa'
'
Almost but not quite.  The Total Alloc column is the number of times that the system was told to allocate items of that type.  I.e., if you allocate 2,000 FMTSPECs but immediately throw them on the floor, the Total alloc will be bumped up by 2,000 but (after a sufficient number of gc''s), the in-use will return to what it was before you started.'
'
The number of items that are "lost" are the ones that are "in-use" that you can''t really account for, by open windows, current processes, etc.'
'
--Ron'
'


Workaround: 

Test Case: 

Edit-By: 

Edit-Date: