lex in interactive mode

lex usually reads input from a filestream.  When the stream reaches EOF, lex returns to the caller. How it cooperates with a parser?

The parser calls lex via "to top" (the entry function of the Som parser).

parse
  startlex
  lex           // get the first token
  while ! EOF
    e = top     //  call parser
    if 'TO or ':
      genex e       // generate code
    else 
      append im e   //  collect im lines

in interactive mode

lex will read stdin and wait for one line input, as it is called from "top" it will return when "top" terminates. Because the parser always looks one token ahead lex will always get stuck and waiting for an input line and have no chance to "return".

To force lex to return, one can append "EOF" at the end of input buffer (a line).  So when lex reaches the end of line it will return to the caller. This will not work for the source that is longer than one line as "top" will parse until a unit is completed.

top := to ... | : ... | exp

That is, the end of a function definition, a macro definition or an expression.  To allow multi-line input, a special "continue" character (usually \) can be appended at the end of input line to indicate the continuation to the next line.

So, lex must have two modes: batch and line.  The batch mode allows lex to read until EOF.  In the line mode, when lex reads a new input line, it will append EOF and the end of line.  This is done via system area 
  M[lexmode_ads] = {line 1, batch 2}

In interactive mode lexmode is set to "line". When "loadfile" lex must be in batch mode to read the whole file. 

How lexmode is set?

interactive mode lexmode is "line".  compile mode, lexmode is "batch".  when "loadfile", lexmode is "batch". in execute mode, lexmode is set "batch" by default but users can set it from user programs (that use lex).

the behaviour during interactive mode is: lex is "line" until "loadfile" is called, then lex is "batch".  After loadfile terminates (read the whole file), lex returns to r-e-p loop which sets lexmode to "line" again.

In compile mode, lex is always in "batch".

25 Oct 2009

 