(FILECREATED " 4-Apr-85 15:48:54" ("compiled on " {PHYLUM}NGRAMS>NG-SYSTEM.;1) "19-Mar-85 14:34:17" tcompl'd in "INTERLISP-D 19-Mar-85 ..." dated "19-Mar-85 14:48:35") (FILECREATED "21-Mar-85 04:34:51" {DSK}NG-SYSTEM.;40 44013 changes to: (FNS NG.ACCEPT-LINK-DENSITY-OF-CLUSTER NG.FORM-CLUSTERS NG.CLUSTER NG.DELETE-WEAKER-CLUSTER-LINKS NG.PUT-LARGEST-ITEM-FIRST NG.PRUNE-CLUSTER-LINKS NG.PUT-LONGEST-LIST-FIRST NG.INSERT-IN-ORDERED-LIST NG.MERGE-ORDERED-LISTS NG.OBTAIN-CLUSTER-LINKS NG.GENERATE-UPPER-TRIANGLE-ROW-OF-CLUSTER-LINKS NG.BACK-FILL-LINK-MATRIX) (VARS NG-SYSTEMCOMS) previous date: "17-Mar-85 22:32:35" {DSK}NG-SYSTEM.;35) NG.ACCEPT-LINK-DENSITY-OF-CLUSTER D1 (I 1 LINKSUMS I 0 CLUSTER F 5 cluster-link-density-threshold) 2@j(@ kÙÖ@AHµ I×UýóºAJ(¼K“KLó“L»J¹HX°â(7 LENGTH) NIL () NG.ACCUML D1 (P 3 LAST-CHAR P 2 COUNTS P 1 TOKEN-LIST P 0 TEXT-STRING I 0 TEXT F 4 number-of-grams) '@@ Xg[TkØgjdZYµJ ¿°ô(42Q NG.EXTRACT-AND-COUNT-GRAMS 32Q NG.GET-ENCODED-TOKEN-LIST 26Q ARRAY 6 CONCAT) (21Q BYTE 12Q ?) () NG.ADD-GRAM-FREQUENCIES D1 (P 5 Y P 0 VL I 2 COEFFICIENT I 1 Y-PROFILE-BODY I 0 X-PROFILE-BODY) ´@³>AAIµJ]k(BÖjØXjð³HµMkH*¿M¼KµLh[ºIY°×L&»°ø ²ä°ñA¡@@dj(Aj(ð²k@k(Ak(BÖjØÔ*@@ABj(Aj(ýóš@@ABAk(BÖjØXjð³HµAkH*¿A@AB ²ìAb±ÿP(250Q \FZEROP 243Q NG.ADD-GRAM-FREQUENCIES 200Q NG.ADD-GRAM-FREQUENCIES 153Q NG.ADD-GRAM-FREQUENCIES 75Q \FZEROP) NIL () NG.BACK-FILL-LINK-MATRIX D1 (L (1 ROW 0 LINKROWLISTS) F 6 number-of-grams) b@VkØ@QH\µI²ýALð²=LL]hðLdL¿ŸL@ýóœ@L„Ld¿©M»JµKhZ¹L°°K&º°ùNIL NIL () NG.DELETE-WEAKER-CLUSTER-LINKS D1 (L (2 ASSIGNMENT 1 LINKROWS 0 CLUSTER)) ^@@j(@k(@h(ÕoÖoÔÔHµhºBJh*¿AJAJ(1KµL*¿HX°ÞdIýó¨¾Mµ Nh]\¿K[°àN&]°÷NIL NIL ( 27Q .01 22Q .02) NG.GRAM-SETUP D1 (F 3 number-of-extents F 4 EXTENT-BASES F 5 TRANSFORM F 6 TWOGRAMS F 7 BIT-VALUES F 10Q BITS-SET F 11Q ALPHAN-HASH F 12Q number-of-grams F 13Q vector-scaling F 14Q SPACE F 15Q STOPHASH F 16Q STOPWORDS) )n³clclcSkØgmÿkcokHµ‚±™lgmÿkc okHµ‚±ŒlªgjdcojHµ‚±€lgmÿjcojHµvl€ojdcojHµnocd cgcl( cojHµZiºTIJ*¿IkÔ¹HX±ÿUºUIJ*¿IkÔ¹HX±ÿbºVIJ*¿IkÔ¹HX±ÿnºWIJ*¿IkÔ¹HX±ÿzºWIJ*¿IkÔ¹HX°‚IW¿IkÔ¹HX°–(435Q PUTHASH 260Q HARRAY 244Q NG.GENERATE-STOP-HASH 213Q ARRAY 161Q ARRAY 123Q ARRAY 66Q ARRAY 30Q ARRAY) (251Q % 153Q FIXP 116Q BYTE 60Q FIXP 22Q FIXP) ( 266Q (A B C D E F G H I J K L M N O P Q R S T U V W X Y Z 0 1 2 3 4 5 6 7 10Q 11Q) 236Q ("THE" "A" "IT" "OR" "AND") 221Q (0 1 1 2 1 2 2 3 1 2 2 3 2 3 3 4 1 2 2 3 2 3 3 4 2 3 3 4 3 4 4 5 1 2 2 3 2 3 3 4 2 3 3 4 3 4 4 5 2 3 3 4 3 4 4 5 3 4 4 5 4 5 5 6 1 2 2 3 2 3 3 4 2 3 3 4 3 4 4 5 2 3 3 4 3 4 4 5 3 4 4 5 4 5 5 6 2 3 3 4 3 4 4 5 3 4 4 5 4 5 5 6 3 4 4 5 4 5 5 6 4 5 5 6 5 6 6 7) 206Q (BITS 10Q) 167Q (1 2 4 10Q 20Q 40Q 100Q 200Q) 131Q (116Q 275Q 76Q 105Q 44Q 131Q 120Q 331Q 244Q 101Q 104Q 222Q 4 175Q 371Q 357Q 304Q 44Q 111Q 20Q 231Q 46Q 100Q 104Q 20Q 6 177Q 370Q 56Q 0 0 100Q 0 1 2 100Q 106Q 22Q 6 23Q 321Q 20Q 164Q 5 161Q 340Q 222Q 357Q 107Q 144Q 262Q 6 0 0 20Q 164Q 305Q 371Q 124Q 231Q 314Q 101Q 144Q 222Q 6 36Q 271Q 16Q 104Q 4 1 20Q 31Q 6 0 0 40Q 0 0 0 0 0 0 0 0 0 4 12Q 17Q 22Q 24Q 30Q 32Q 37Q 42Q 44Q 46Q 51Q 52Q 60Q 66Q 75Q 100Q 102Q 105Q 106Q 112Q 115Q 116Q 120Q 121Q 123Q 132Q 137Q 143Q 143Q 143Q 144Q 144Q 145Q 146Q 147Q 152Q 154Q 156Q 161Q 165Q 166Q 172Q 174Q 200Q 203Q 206Q 215Q 221Q 224Q 230Q 232Q 232Q 232Q 233Q 237Q 243Q 251Q 254Q 260Q 264Q 266Q 271Q 274Q 276Q 302Q 307Q 312Q 314Q 315Q 316Q 317Q 322Q 324Q 324Q 324Q 325Q 325Q 325Q 325Q 325Q 325Q 325Q 325Q) 74Q (20Q 26Q 33Q 40Q 43Q 47Q 52Q 55Q 60Q 62Q 65Q 67Q 71Q 73Q 75Q 100Q) 36Q (0 1633Q 3466Q 5321Q 7154Q 11007Q 12642Q 14512Q 15263Q)) NG.VECTOR-SIMILARITY-PARAMETERS D1 (P 11Q WS2 P 10Q WS1 P 7 PARAMETERS P 6 COVARIANCE-INCREMENT P 5 COVARIANCE-SUM P 4 P-FOURTH-SUM P 3 P-CUBE-SUM P 2 P-SQUARE-SUM P 1 EXPECTED-SUM P 0 WEIGHTED-P-SQUARE I 2 W-SUM I 1 W I 0 P) êÀ@ gokXoYoZo[o\o]o^@ k_¿_¿OOó³]HOAO(@O(@O(êê*¿IHO(è¹JAO(HO(êèºKAO(HO(@O(êêè»LHO(HO(êè¼MHO(Nêè½NHO(è¾OkÔ_°¢oBë_ddê_lgoj_dIOêj*OJOêk*OKOêl*OLOêl*OMdèOêl*O(267Q ARRAY 64Q ARRAYSIZE 21Q ARRAY 6 ARRAYSIZE) (257Q FLOATP 11Q FLOATP) ( 263Q 0.0 241Q 1.0 57Q 0.0 52Q 0.0 45Q 0.0 40Q 0.0 33Q 0.0 26Q 0.0 15Q 0.0) NG.VECTOR-TO-COUNTS D1 (P 2 BODY P 1 I I 0 VECTOR F 12Q BOUNDS F 13Q EXTENT-BASES F 14Q FULLVECTOR F 15Q number-of-extents) † Wk@3IHó¤JdI(¡hI([jð³fK•K ³\WI(WI(kØ½¼MLó«WMj*¿MkÔ]°óJI(kJI(WI( ONó³#OOO(Ø_¿WOOOkØ(*¿OlÔ_°ÝWIO*IkÔY°†(42Q \FZEROP) NIL () NG.VECTOR-TO-PROFILE-BODY D1 (P 12Q GRAM P 11Q EXTENT P 10Q N P 5 BODY P 4 I P 0 INDEX-PAIR I 0 VECTOR F 16Q number-of-extents F 17Q EXTENT-BASES) ²0Wk@LKó¤MdL(¡NL(Yjð’±I—I ’±NML(kML(WL(4 OOó³aOOO(Ø_²UOOkØ(Zjð³8J•J ³.lgjdXdjO*¿HkOOkØ(*¿_¿OµOh__¿OlÔ_°¦O&_¿°ñO ¾LkÔ\±ÿ^(250Q \NCONC2 157Q ARRAY 143Q \FZEROP 47Q \FZEROP) (152Q FIXP) () NG.FIND-GRAM D1 (P 1 BIT P 0 BYTE I 1 GRAMTABLE I 0 GRAMVALUE F 2 BIT-VALUES F 3 BITS-SET) / @lÛX@låYAH(RI(åjó²AHlUØ(SAH(RI( (Ømÿ(47Q IMOD) NIL () NG.FORM-CLUSTERS D1 (P 7 STATISTICS P 6 Q P 5 P P 4 L P 1 I I 3 ASSIGNMENT I 2 LINKSUMS I 1 LINKROWS I 0 RANGE F 10Q RESULTS) å@!HµW¹AI(AKµHX°ê\Ió³õCI(]µ4lgoj_²"Ojo*²OIh]´CIM*—BIo*¿MjMj(L *¿MkMk(L *¿CL(^µ#LM ²DCLM*¿BLo*°3dºCJM*¿µöMjMj(Nj( *¿MkMk(Nk( *¿MN ¿BIBI(LÔ*¿BLBL(LÔ*¿K[±ÿ3(306Q NG.MERGE-ORDERED-LISTS 277Q MAX 257Q MIN 203Q NG.INSERT-IN-ORDERED-LIST 164Q MIN 146Q MIN 66Q ARRAY) (56Q FLOATP) ( 222Q 0.0 130Q 0.0 101Q 10000.0 62Q 0.0) NG.NEXT-TOKEN D1 (P 3 CHARLIST P 2 CHAR P 1 FIRST-CHAR P 0 TOKEN F 7 ALPHAN-HASH F 10Q TEXT-STRING F 11Q LAST-CHAR) mo1WZµHžJdW©¿W Z°òJ¹0W ZdW³#¿L»HIK ¸Jo ²W Z°¾Mµ Nh]¼°ÊN&½°ÅJW³ºJc¿°¢(114Q GNC 105Q STRPOS 74Q CONCAT 71Q CONCATLIST 47Q GNC 32Q GNC) NIL ( 102Q ".,-/'" 3 "") NG.INFLEX-STEMMER D1 (P 114Q g4 P 113Q g3 P 112Q g2 P 111Q g1 P 110Q v2 P 107Q v1 P 106Q v0 P 105Q sp P 104Q gt P 103Q ne P 102Q eq P 101Q lt P 100Q is P 77Q ndbl P 76Q nlen P 75Q nisa P 74Q nbeg P 73Q nsuc P 72Q nfai P 71Q z4 P 70Q z3 P 67Q z2 P 66Q z1 P 65Q y1 P 64Q y9 P 63Q y8 P 62Q y7 P 61Q y6 P 60Q y5 P 57Q y4 P 56Q y3 P 55Q y2 P 54Q y1 P 53Q y0 P 52Q no P 51Q p6 P 50Q p5 P 47Q p4 P 46Q p3 P 45Q p2 P 44Q p1 P 43Q zz P 42Q m6 P 41Q m5 P 40Q m4 P 37Q m3 P 36Q m2 P 35Q m1 P 34Q xz P 33Q xy P 32Q xx P 31Q xw P 30Q xv P 27Q xu P 26Q xt P 25Q xs P 24Q xr P 23Q xq P 22Q xp P 21Q xo P 20Q xn P 17Q xm P 16Q xl P 15Q xk P 14Q xj P 13Q xi P 12Q xh P 11Q xg P 10Q xf P 7 xe P 6 xd P 5 xc P 4 xb P 3 xa P 2 CHARACTER P 1 CONTINUANCE P 0 S-LIST I 0 TOKEN) Mððð,ð;ðJ Lg[g\g]g^g_g_g_g_g_g_g_g_g_g_ g_"g_$g_&g_(g_*g_,g_.g_0g_2g_4g_6g_8mÿ_:mþ_mü_@mû_Bmú_Dh_Fk_Hl_Jl_Ll_Nl_Pl_Rh_Tmÿ_Vmþmý_Zmü_\mû_^mú_`mù_bmø_dm÷_fmö_hmõ_Xk_ll_nl_pl_rh_tmÿ_vl_xl_zl_|l _~l_€l_‚l_„l_†l_ˆl_Šl(_Œl)_Žl*_l½_’li_”lt_–l£_˜oXiY@ Zdgðdgð§dgð¡@@ bd§@ I²÷h¹@b°í(577Q PACK 574Q REVERSE 563Q REVERSE 560Q UNPACK 526Q LAST 523Q UNPACK) (550Q G 542Q D 534Q S 215Q z 210Q y 203Q x 176Q w 171Q v 164Q u 157Q t 152Q s 145Q r 140Q q 133Q p 126Q o 121Q n 114Q m 107Q l 102Q k 75Q j 70Q i 63Q h 56Q g 51Q f 44Q e 40Q d 34Q C 30Q b 24Q a) ( 514Q (1 xs z1 g1 xe z1 53Q xi eq 4 2 y0 z4 6 xt xr xo xs y0 z4 6 xr xe xt xo y0 z2 4 xv xo y0 z1 16Q xt is 7 2 xi xu y7 p1 xy eq 6 2 y0 y7 p1 xy z1 56Q xo eq 5 2 y0 is 6 2 xd xg y8 zz x2 4 xn xa y0 z2 4 xr xh y0 eq 4 2 y0 is 4 1 xh y0 z1 13Q xt is 7 3 xa xs xt y8 zz y0 y8 zz z1 50Q xs z2 5 xa xg y8 zz z3 6 xu xc xo y8 zz is 10Q 2 xe xy y8 p2 xi xs z3 10Q xi xr xc y8 p2 xi xs is 5 1 xs y8 zz y0 z1 45Q xh is 4 1 xt y0 z1 33Q xc eq 5 2 y0 eq 6 23Q z2 5 xi xr y8 zz is 5 1 xn y8 zz is 4 1 xe no y0 y8 zz is 5 1 xx y8 zz z2 5 xz xt y8 zz eq 3 2 no y0 z1 15Q xi eq 4 2 no is 5 2 xx xn y0 no z2 22Q xu xn is 4 1 xg y0 lt 5 2 no z2 4 xe xm y0 no z3 5 xu xa xe y0 is 5 2 xs xu no z1 6 xa lt 4 2 no y0)) NG.INSERT-IN-ORDERED-LIST D1 (P 1 L I 1 ORDEREDLIST I 0 ITEM) .AHYd›¿@Iýó£I°óIµA@h AdI¿I@°ô(36Q \NCONC2) NIL () NG.EXTRACT-AND-COUNT-GRAMS D1 (P 4 THIRDCHAR P 3 SECONDCHAR P 2 FIRSTCHAR P 1 GRAMLENGTH P 0 GRAMINDEX I 0 TOKEN-LIST F 5 COUNTS F 6 TWOGRAMS) PlY@—@d³!¿Ilð´@l$Ú@n£ØØ¸UHUH(kØ*hº@»@¼lJó²*lKó²$lLó²JlÚKØV XmÿóœHlÚLkØØ¸l°Ilð’j¸‹Jl$ÚKn£ØØX’l¹UHUH(kØ*¿@b±ÿ{(125Q NG.FIND-GRAM) NIL () NG.GENERATE-PROFILE D1 (P 17Q PP P 12Q I P 6 PARAMETERS P 5 BODY P 4 PACKED-WEIGHTS P 3 PACKED-PROBABILITIES P 2 PAIR P 1 NUMBER P 0 SUM-OF-WEIGHTS I 1 PROBABILITIES I 0 COUNTS F 20Q number-of-grams F 21Q TRANSFORM) ±ojrkW OOó³E@O(_jð³1O–O ³%IkØ¹lgjdZM½JjO*¿JkW"@O((*¿OmÿÔ_¿°¸IMAIMKLH ^dMAOµO_k(_jð³ O–O ³O_¿OµOh__¿O_°ÊO&_¿°ó(214Q \FZEROP 147Q NG.PROFILE-SIMILARITY-PARAMETERS 136Q NG.PACK-PROFILE-PROBABILITIES-AND-WEIGHTS 70Q ARRAY 50Q \FZEROP) (63Q FIXP) ( 3 0.0) NG.GENERATE-UPPER-TRIANGLE-ROW-OF-CLUSTER-LINKS D1 (P 13Q S P 12Q VS P 11Q ES P 10Q C2 P 7 C1 P 6 SUM P 5 V P 1 J I 11Q FSMP22 I 10Q FSMP4 I 7 FSMP3 I 6 FSMP2 I 5 FSMP I 4 FSM I 3 VECTORS I 2 BOUNDS I 1 FULLVECTOR I 0 ROW F 14Q vector-scaling F 15Q clustering-significance-threshold) q C@kØ¢HµJ]W×¾DNÔkÙ_kÙ_¿NEÖ_¿NFOGÖOGÖ ÔÔÖNOGÖÖÕ_¿MABOO _Wýó¬IO¼KµLh[ºIkÔ¹HX°ŸL&»°ô(115Q NG.COMPUTE-CLUSTER-SIMILARITY-LINK 64Q MINUS) NIL () NG.GENERATE-VECTOR D1 (P 11Q GRAM P 6 GRAMDELTA P 5 EXTENT-SIZE P 4 EXTENT P 3 VECTOR P 2 LAST-GRAM P 1 VECTORSUM P 0 EXTENT-NUMBER I 0 COUNTS F 12Q EXTENT-BASES F 13Q number-of-extents F 14Q number-of-grams F 15Q TRANSFORM) ™ kjdSWghk[h\Wk OOó³n@O(_jð³OO–O ³COdWHkØ(ó²¿KHL *¿h¼HkØ¸WH(º°àJÙ¾Ndlÿó³¿W@O((NL¼ILØ¹OºOkÔ_°žlÿÙ¾jlÿL¼°ÑKHL *IK(222Q NG.GENERATE-VECTOR-EXTENT 110Q NG.GENERATE-VECTOR-EXTENT 63Q \FZEROP 17Q ARRAY) (12Q POINTER) () NG.GENERATE-VECTOR-EXTENT D1 (P 1 EXTENT-SIZE P 0 EXTENT-VECTOR I 0 EXTENT-GRAM-LIST) ,0@ YdgjkXIº@£IHHJ@*¿JkÙº@b°ì(20Q ARRAY 6 LENGTH) (13Q BYTE) () NG.GET-ENCODED-TOKEN-LIST D1 (P 2 TOKENLIST P 1 TOKEN P 0 CHAR F 6 STOPHASH F 7 ALPHAN-HASH) 60Y V³÷0I XµKZd“²åJW½Lµ Mh\»°ãM&¼°Þ(24Q GNC 11Q MKATOM 5 NG.NEXT-TOKEN) NIL () NG.NORMALIZED-SIMILARITY-MEASURE D1 (P 2 SCALING P 1 STANDARD-DEVIATION P 0 EXPECTED-VALUE I 2 VECTOR-SUM I 1 PARAMETERS I 0 INNER-PRODUCT F 3 vector-scaling) &0oSëZBJAj(ÖÖXBJAk(ÖÖ Y@JÖHéIë(33Q SQRT) NIL ( 6 1.0) NG.GENERATE-STOP-HASH D1 (P 3 WORD P 2 HASHVALUE P 0 STOPHASH I 0 STOPWORDLIST) /@ lÚ X@jIµH»JkØºK JH¿IY°è(47Q PUTHASH 42Q MKATOM 14Q HARRAY 6 LENGTH) NIL () NG.ANALYZE D1 (I 0 TEXT) @ (6 NG.GENERATE-VECTOR 3 NG.ACCUML) NIL () NG.CLUSTER D1 (P 11Q CLUSTERLIST P 10Q ASSIGNMENT P 7 LINKSUMS P 6 LINKROWS P 5 RANGE I 1 PROBABILITIES I 0 VECTORLIST F 12Q VECTOR) ˜ W k2IdHó²O¿J@A @ gok@ ghk MN ^›MNOO_¿O _d™O ²O¼Kµ Lh[ºIkÔY°¡L&»°÷ONO¿ON ¿ONOOO _°µ(222Q \APPEND2 214Q NG.FORM-CLUSTERS 200Q NG.PRUNE-CLUSTER-LINKS 170Q NG.DELETE-WEAKER-CLUSTER-LINKS 126Q NG.ACCEPT-LINK-DENSITY-OF-CLUSTER 114Q NG.PUT-LONGEST-LIST-FIRST 104Q NG.FORM-CLUSTERS 71Q NG.PRUNE-CLUSTER-LINKS 61Q ARRAY 51Q LENGTH 45Q ARRAY 32Q LENGTH 26Q NG.OBTAIN-CLUSTER-LINKS 4 LENGTH) (54Q POINTER 35Q FLOATP) ( 41Q 0.0) NG.OBTAIN-CLUSTER-LINKS D1 (P 14Q VL P 13Q I P 11Q ROW P 10Q ROWBACKLIST P 7 ROWINDEX P 6 VECTORSUM P 5 MODEL-SUMS P 4 SUM-OF-WEIGHTS P 3 PACKED-WEIGHTS P 2 PACKED-PROBABILITIES P 1 BOUNDS P 0 FULLVECTOR I 1 PROBABILITIES I 0 VECTORLIST F 15Q number-of-grams F 16Q number-of-extents F 17Q n-gram-count F 20Q vector-scaling) Ã @ kØh_@ ghk_WgjkXWgjkYWA JKL]@kO_µO ¿OW ×¾OHIONdMj(ÖNMk(ÖNMl(ÖNMl(ÖNMl(Ö _¿OOOO O *¿OOOh ¿OOýó—OO¿OkÔ_¿O°…(242Q \NCONC2 224Q \APPEND2 217Q NG.BACK-FILL-LINK-MATRIX 201Q NG.GENERATE-UPPER-TRIANGLE-ROW-OF-CLUSTER-LINKS 120Q NG.VECTOR-TO-COUNTS 74Q NG.VECTOR-SIMILARITY-PARAMETERS 66Q NG.PACK-VECTOR-PROBABILITIES-AND-WEIGHTS 57Q ARRAY 44Q ARRAY 30Q ARRAY 20Q LENGTH 6 LENGTH) (52Q FIXP 37Q BYTE 23Q POINTER) () NG.COMPUTE-CLUSTER-SIMILARITY-LINK D1 (P 12Q GRAM P 11Q EXTENT P 10Q N P 6 BODY P 5 I P 1 S P 0 INNER-PRODUCT I 4 VS I 3 ES I 2 BOUNDS I 1 FULLVECTOR I 0 VECTOR F 13Q number-of-extents F 14Q EXTENT-BASES F 15Q vector-scaling) Šj1Wk@MLó³lBM(Zjð³^J•J ³TNM(kNM(WM( OOó³