


FLEX(1)			 USER COMMANDS			  FLEX(1)



NAME
     flex - fast lexical analyzer generator

SYNOPSIS
     flex [-bcdfinpstvFILT8 -C[efmF] -Sskeleton] [filename ...]

DESCRIPTION
     flex is a	tool  for  generating  scanners:  programs  which
     recognized	 lexical  patterns in text.  flex reads	the given
     input files, or its standard input	 if  no	 file  names  are
     given,  for  a  description  of  a	scanner	to generate.  The
     description is in the form	of pairs of  regular  expressions
     and  C  code,  called  rules.  flex  generates as output a	C
     source file, lex.yy.c, which defines a routine yylex(). This
     file is compiled and linked with the -lfl library to produce
     an	executable.  When the executable is run, it analyzes  its
     input  for	occurrences of the regular expressions.	 Whenever
     it	finds one, it executes the corresponding C code.

     For full documentation, see flexdoc(1). This manual entry is
     intended for use as a quick reference.

OPTIONS
     flex has the following options:

     -b	  Generate  backtracking  information  to  lex.backtrack.
	  This	is  a  list of scanner states which require back-
	  tracking and the input characters on which they do  so.
	  By adding rules one can remove backtracking states.  If
	  all backtracking states are eliminated and -f	or -F  is
	  used,	the generated scanner will run faster.

     -c	  is a do-nothing, deprecated option included  for  POSIX
	  compliance.

	  NOTE:	in previous releases of	flex -c	specified  table-
	  compression  options.	  This functionality is	now given
	  by the -C flag.  To ease the the impact of this change,
	  when	flex encounters	-c, it currently issues	a warning
	  message and assumes that -C was  desired  instead.   In
	  the future this "promotion" of -c to -C will go away in
	  the name of full POSIX  compliance  (unless  the  POSIX
	  meaning is removed first).

     -d	  makes	the generated scanner run in debug  mode.   When-
	  ever	 a   pattern   is   recognized	 and  the  global
	  yy_flex_debug	is non-zero (which is the  default),  the
	  scanner will write to	stderr a line of the form:

	      --accepting rule at line 53 ("the	matched	text")

	  The line number refers to the	location of the	 rule  in



Version	2.3	    Last change: 26 May	1990			1






FLEX(1)			 USER COMMANDS			  FLEX(1)



	  the  file defining the scanner (i.e.,	the file that was
	  fed to flex).	 Messages are  also  generated	when  the
	  scanner  backtracks,	accepts	the default rule, reaches
	  the end of its input buffer (or encounters a	NUL;  the
	  two  look  the same as far as	the scanner's concerned),
	  or reaches an	end-of-file.

     -f	  specifies (take your pick) full table	or fast	 scanner.
	  No  table compression	is done.  The result is	large but
	  fast.	 This option is	equivalent to -Cf (see below).

     -i	  instructs flex to generate a case-insensitive	 scanner.
	  The  case  of	 letters given in the flex input patterns
	  will be ignored,  and	 tokens	 in  the  input	 will  be
	  matched  regardless of case.	The matched text given in
	  yytext will have the preserved case (i.e., it	will  not
	  be folded).

     -n	  is another do-nothing, deprecated option included  only
	  for POSIX compliance.

     -p	  generates a performance report to stderr.   The  report
	  consists  of	comments  regarding  features of the flex
	  input	file which will	cause a	loss  of  performance  in
	  the resulting	scanner.

     -s	  causes the default rule (that	unmatched  scanner  input
	  is  echoed to	stdout)	to be suppressed.  If the scanner
	  encounters input that	does not match any of its  rules,
	  it aborts with an error.

     -t	  instructs flex to write the  scanner	it  generates  to
	  standard output instead of lex.yy.c.

     -v	  specifies that flex should write to stderr a summary of
	  statistics regarding the scanner it generates.

     -F	  specifies that the fast  scanner  table  representation
	  should  be  used.  This representation is about as fast
	  as the full table representation  (-f),  and	for  some
	  sets	of patterns will be considerably smaller (and for
	  others, larger).  See	flexdoc(1) for details.

	  This option is equivalent to -CF (see	below).

     -I	  instructs flex to generate an	interactive scanner, that
	  is, a	scanner	which stops immediately	rather than look-
	  ing ahead if it knows	that the currently  scanned  text
	  cannot  be  part  of a longer	rule's match.  Again, see
	  flexdoc(1) for details.

	  Note,	-I cannot be used in  conjunction  with	 full  or



Version	2.3	    Last change: 26 May	1990			2






FLEX(1)			 USER COMMANDS			  FLEX(1)



	  fast tables, i.e., the -f, -F, -Cf, or -CF flags.

     -L	  instructs flex not  to  generate  #line  directives  in
	  lex.yy.c. The	default	is to generate such directives so
	  error	messages in the	actions	will be	correctly located
	  with	respect	 to the	original flex input file, and not
	  to the fairly	meaningless line numbers of lex.yy.c.

     -T	  makes	flex run in trace mode.	 It will generate  a  lot
	  of  messages to stdout concerning the	form of	the input
	  and the resultant non-deterministic  and  deterministic
	  finite  automata.   This  option  is	mostly for use in
	  maintaining flex.

     -8	  instructs flex to generate an	8-bit scanner.	 On  some
	  sites,  this is the default.	On others, the default is
	  7-bit	characters.  To	see which is the case, check  the
	  verbose  (-v)	output for "equivalence	classes	created".
	  If the denominator of	the number shown is 128, then  by
	  default  flex	is generating 7-bit characters.	 If it is
	  256, then the	default	is 8-bit characters.

     -C[efmF]
	  controls the degree of table compression.

	  -Ce directs  flex  to	 construct  equivalence	 classes,
	  i.e.,	 sets  of characters which have	identical lexical
	  properties.  Equivalence classes usually give	 dramatic
	  reductions  in the final table/object	file sizes (typi-
	  cally	 a  factor  of	2-5)   and   are   pretty   cheap
	  performance-wise   (one  array  look-up  per	character
	  scanned).

	  -Cf specifies	that the full scanner  tables  should  be
	  generated - flex should not compress the tables by tak-
	  ing advantages of similar transition functions for dif-
	  ferent states.

	  -CF specifies	that the alternate fast	scanner	represen-
	  tation (described in flexdoc(1)) should be used.

	  -Cm directs flex to construct	meta-equivalence classes,
	  which	 are  sets of equivalence classes (or characters,
	  if equivalence classes are not  being	 used)	that  are
	  commonly  used  together.  Meta-equivalence classes are
	  often	a big win when using compressed	tables,	but  they
	  have	a  moderate  performance  impact (one or two "if"
	  tests	and one	array look-up per character scanned).

	  A lone -C specifies that the scanner tables  should  be
	  compressed  but  neither  equivalence	classes	nor meta-
	  equivalence classes should be	used.



Version	2.3	    Last change: 26 May	1990			3






FLEX(1)			 USER COMMANDS			  FLEX(1)



	  The options -Cf or  -CF  and	-Cm  do	 not  make  sense
	  together - there is no opportunity for meta-equivalence
	  classes if the table is not being  compressed.   Other-
	  wise the options may be freely mixed.

	  The default setting is -Cem, which specifies that  flex
	  should   generate   equivalence   classes   and   meta-
	  equivalence classes.	This setting provides the highest
	  degree   of  table  compression.   You  can  trade  off
	  faster-executing scanners at the cost	of larger  tables
	  with the following generally being true:

	      slowest &	smallest
		    -Cem
		    -Cm
		    -Ce
		    -C
		    -C{f,F}e
		    -C{f,F}
	      fastest &	largest


	  -C options are not cumulative;  whenever  the	 flag  is
	  encountered, the previous -C settings	are forgotten.

     -Sskeleton_file
	  overrides the	default	skeleton  file	from  which  flex
	  constructs its scanners.  You'll never need this option
	  unless you are doing flex maintenance	or development.

SUMMARY	OF FLEX	REGULAR	EXPRESSIONS
     The patterns in the input are written using an extended  set
     of	regular	expressions.  These are:

	 x	    match the character	'x'
	 .	    any	character except newline
	 [xyz]	    a "character class"; in this case, the pattern
		      matches either an	'x', a 'y', or a 'z'
	 [abj-oZ]   a "character class"	with a range in	it; matches
		      an 'a', a	'b', any letter	from 'j' through 'o',
		      or a 'Z'
	 [^A-Z]	    a "negated character class", i.e., any character
		      but those	in the class.  In this case, any
		      character	EXCEPT an uppercase letter.
	 [^A-Z\n]   any	character EXCEPT an uppercase letter or
		      a	newline
	 r*	    zero or more r's, where r is any regular expression
	 r+	    one	or more	r's
	 r?	    zero or one	r's (that is, "an optional r")
	 r{2,5}	    anywhere from two to five r's
	 r{2,}	    two	or more	r's
	 r{4}	    exactly 4 r's



Version	2.3	    Last change: 26 May	1990			4






FLEX(1)			 USER COMMANDS			  FLEX(1)



	 {name}	    the	expansion of the "name"	definition
		    (see above)
	 "[xyz]\"foo"
		    the	literal	string:	[xyz]"foo
	 \X	    if X is an 'a', 'b', 'f', 'n', 'r',	't', or	'v',
		      then the ANSI-C interpretation of	\x.
		      Otherwise, a literal 'X' (used to	escape
		      operators	such as	'*')
	 \123	    the	character with octal value 123
	 \x2a	    the	character with hexadecimal value 2a
	 (r)	    match an r;	parentheses are	used to	override
		      precedence (see below)


	 rs	    the	regular	expression r followed by the
		      regular expression s; called "concatenation"


	 r|s	    either an r	or an s


	 r/s	    an r but only if it	is followed by an s.  The
		      s	is not part of the matched text.  This type
		      of pattern is called as "trailing	context".
	 ^r	    an r, but only at the beginning of a line
	 r$	    an r, but only at the end of a line.  Equivalent
		      to "r/\n".


	 <s>r	    an r, but only in start condition s	(see
		    below for discussion of start conditions)
	 <s1,s2,s3>r
		    same, but in any of	start conditions s1,
		    s2,	or s3


	 <<EOF>>    an end-of-file
	 <s1,s2><<EOF>>
		    an end-of-file when	in start condition s1 or s2

     The regular expressions listed above are  grouped	according
     to	 precedence, from highest precedence at	the top	to lowest
     at	the bottom.   Those  grouped  together	have  equal  pre-
     cedence.

     Some notes	on patterns:

     -	  Negated character classes match  newlines  unless  "\n"
	  (or  an equivalent escape sequence) is one of	the char-
	  acters explicitly  present  in  the  negated	character
	  class	(e.g., "[^A-Z\n]").




Version	2.3	    Last change: 26 May	1990			5






FLEX(1)			 USER COMMANDS			  FLEX(1)



     -	  A rule can have at most one instance of  trailing  con-
	  text (the '/'	operator or the	'$' operator).	The start
	  condition, '^', and "<<EOF>>"	patterns can  only  occur
	  at the beginning of a	pattern, and, as well as with '/'
	  and '$', cannot be  grouped  inside  parentheses.   The
	  following are	all illegal:

	      foo/bar$
	      foo|(bar$)
	      foo|^bar
	      <sc1>foo<sc2>bar


SUMMARY	OF SPECIAL ACTIONS
     In	addition to arbitrary C	code, the following can	appear in
     actions:

     -	  ECHO copies yytext to	the scanner's output.

     -	  BEGIN	followed by the	name of	a start	condition  places
	  the scanner in the corresponding start condition.

     -	  REJECT directs the scanner to	proceed	on to the "second
	  best"	 rule which matched the	input (or a prefix of the
	  input).  yytext and yyleng are  set  up  appropriately.
	  Note that REJECT is a	particularly expensive feature in
	  terms	scanner	performance; if	it is used in any of  the
	  scanner's   actions  it  will	 slow  down  all  of  the
	  scanner's matching.  Furthermore, REJECT cannot be used
	  with the -f or -F options.

	  Note also that unlike	the other special actions, REJECT
	  is  a	 branch;  code	immediately  following	it in the
	  action will not be executed.

     -	  yymore() tells  the  scanner	that  the  next	 time  it
	  matches  a  rule,  the  corresponding	 token	should be
	  appended onto	the current value of yytext  rather  than
	  replacing it.

     -	  yyless(n) returns all	but the	first n	characters of the
	  current token	back to	the input stream, where	they will
	  be rescanned when the	scanner	looks for the next match.
	  yytext  and  yyleng  are  adjusted appropriately (e.g.,
	  yyleng will now be equal to n	).

     -	  unput(c) puts	the  character	c  back	 onto  the  input
	  stream.  It will be the next character scanned.

     -	  input() reads	the next character from	the input  stream
	  (this	 routine  is  called  yyinput()	if the scanner is
	  compiled using C++).



Version	2.3	    Last change: 26 May	1990			6






FLEX(1)			 USER COMMANDS			  FLEX(1)



     -	  yyterminate()	can be used in lieu of a return	statement
	  in  an action.  It terminates	the scanner and	returns	a
	  0 to the scanner's caller, indicating	"all done".

	  By default, yyterminate() is also called when	 an  end-
	  of-file is encountered.  It is a macro and may be rede-
	  fined.

     -	  YY_NEW_FILE is an  action  available	only  in  <<EOF>>
	  rules.   It  means "Okay, I've set up	a new input file,
	  continue scanning".

     -	  yy_create_buffer( file, size ) takes a FILE pointer and
	  an integer size. It returns a	YY_BUFFER_STATE	handle to
	  a new	input buffer  large  enough  to	 accomodate  size
	  characters and associated with the given file.  When in
	  doubt, use YY_BUF_SIZE for the size.

     -	  yy_switch_to_buffer(	 new_buffer   )	  switches    the
	  scanner's  processing	to scan	for tokens from	the given
	  buffer, which	must be	a YY_BUFFER_STATE.

     -	  yy_delete_buffer( buffer ) deletes the given buffer.

VALUES AVAILABLE TO THE	USER
     -	  char *yytext holds the text of the current  token.   It
	  may not be modified.

     -	  int yyleng holds the length of the current  token.   It
	  may not be modified.

     -	  FILE *yyin is	the file  which	 by  default  flex  reads
	  from.	  It  may  be  redefined  but doing so only makes
	  sense	before scanning	begins.	 Changing it in	the  mid-
	  dle of scanning will have unexpected results since flex
	  buffers its input.  Once scanning terminates because an
	  end-of-file	has   been  seen,  void	 yyrestart(  FILE
	  *new_file ) may be called to	point  yyin  at	 the  new
	  input	file.

     -	  FILE *yyout is the file to which ECHO	actions	are done.
	  It can be reassigned by the user.

     -	  YY_CURRENT_BUFFER returns a YY_BUFFER_STATE  handle  to
	  the current buffer.

MACROS THE USER	CAN REDEFINE
     -	  YY_DECL controls how the scanning routine is	declared.
	  By  default, it is "int yylex()", or,	if prototypes are
	  being	used, "int yylex(void)".  This definition may  be
	  changed  by  redefining the "YY_DECL"	macro.	Note that
	  if you give arguments	to the scanning	routine	 using	a



Version	2.3	    Last change: 26 May	1990			7






FLEX(1)			 USER COMMANDS			  FLEX(1)



	  K&R-style/non-prototyped function declaration, you must
	  terminate the	definition with	a semi-colon (;).

     -	  The nature of	how the	scanner	gets  its  input  can  be
	  controlled	by   redefining	  the	YY_INPUT   macro.
	  YY_INPUT's	     calling	     sequence	       is
	  "YY_INPUT(buf,result,max_size)".    Its  action  is  to
	  place	up to max_size characters in the character  array
	  buf  and  return  in the integer variable result either
	  the number of	characters read	or the	constant  YY_NULL
	  (0  on  Unix	systems)  to  indicate	EOF.  The default
	  YY_INPUT reads from the global file-pointer "yyin".	A
	  sample  redefinition	of  YY_INPUT  (in the definitions
	  section of the input file):

	      %{
	      #undef YY_INPUT
	      #define YY_INPUT(buf,result,max_size) \
		  { \
		  int c	= getchar(); \
		  result = (c == EOF) ?	YY_NULL	: (buf[0] = c, 1); \
		  }
	      %}


     -	  When the scanner  receives  an  end-of-file  indication
	  from	YY_INPUT,  it  then checks the yywrap()	function.
	  If yywrap() returns false (zero), then  it  is  assumed
	  that	the  function  has  gone ahead and set up yyin to
	  point	to another input file,	and  scanning  continues.
	  If  it  returns  true	(non-zero), then the scanner ter-
	  minates, returning 0 to its caller.

	  The default yywrap() always returns 1.   Presently,  to
	  redefine  it	you  must first	"#undef	yywrap", as it is
	  currently implemented	as a macro.  It	 is  likely  that
	  yywrap()  will  soon be defined to be	a function rather
	  than a macro.

     -	  YY_USER_ACTION can be	redefined to  provide  an  action
	  which	 is  always  executed prior to the matched rule's
	  action.

     -	  The macro YY_USER_INIT may be	redefined to  provide  an
	  action which is always executed before the first scan.

     -	  In the generated scanner, the	actions	are all	 gathered
	  in  one  large  switch  statement  and  separated using
	  YY_BREAK, which may be redefined.  By	 default,  it  is
	  simply  a  "break", to separate each rule's action from
	  the following	rule's.




Version	2.3	    Last change: 26 May	1990			8






FLEX(1)			 USER COMMANDS			  FLEX(1)



FILES
     flex.skel
	  skeleton scanner.

     lex.yy.c
	  generated scanner (called lexyy.c on some systems).

     lex.backtrack
	  backtracking information for -b flag (called lex.bck on
	  some systems).

     -lfl library with which to	link the scanners.

SEE ALSO
     flexdoc(1), lex(1), yacc(1), sed(1), awk(1).

     M.	E. Lesk	and E. Schmidt,	LEX - Lexical Analyzer Generator

DIAGNOSTICS
     reject_used_but_not_detected undefined or

     yymore_used_but_not_detected undefined -  These  errors  can
     occur  at compile time.  They indicate that the scanner uses
     REJECT or yymore()	but that flex failed to	notice the  fact,
     meaning that flex scanned the first two sections looking for
     occurrences of these actions and failed  to  find	any,  but
     somehow  you  snuck  some in (via a #include file,	for exam-
     ple).  Make an explicit reference to the action in	your flex
     input   file.    (Note  that  previously  flex  supported	a
     %used/%unused mechanism for dealing with this problem;  this
     feature  is  still	supported but now deprecated, and will go
     away soon unless the author hears from people who can  argue
     compellingly that they need it.)

     flex scanner jammed - a scanner compiled with -s has encoun-
     tered  an	input  string  which wasn't matched by any of its
     rules.

     flex input	buffer overflowed -  a	scanner	 rule  matched	a
     string  long enough to overflow the scanner's internal input
     buffer  (16K   bytes   -	controlled   by	  YY_BUF_MAX   in
     "flex.skel").

     scanner  requires	-8  flag  -  Your  scanner  specification
     includes  recognizing  8-bit  characters  and  you	 did  not
     specify the -8 flag (and your site	has  not  installed  flex
     with -8 as	the default).

     fatal flex	scanner	internal error--end of	buffer	missed	-
     This  can	occur  in  an  scanner which is	reentered after	a
     long-jump has jumped out (or over)	the scanner's  activation
     frame.  Before reentering the scanner, use:



Version	2.3	    Last change: 26 May	1990			9






FLEX(1)			 USER COMMANDS			  FLEX(1)



	 yyrestart( yyin );


     too many %t classes! - You	managed	to put every single char-
     acter  into  its  own %t class.  flex requires that at least
     one of the	classes	share characters.

AUTHOR
     Vern Paxson, with the help	of many	ideas and  much	 inspira-
     tion from Van Jacobson.  Original version by Jef Poskanzer.

     See flexdoc(1) for	additional credits  and	 the  address  to
     send comments to.

DEFICIENCIES / BUGS
     Some trailing context patterns cannot  be	properly  matched
     and  generate  warning  messages  ("Dangerous  trailing con-
     text").  These are	patterns where the ending  of  the  first
     part  of  the rule	matches	the beginning of the second part,
     such as "zx*/xy*",	where the 'x*' matches	the  'x'  at  the
     beginning	of  the	 trailing  context.  (Note that	the POSIX
     draft states that the text	matched	by such	patterns is unde-
     fined.)

     For some trailing context rules, parts  which  are	 actually
     fixed-length  are	not  recognized	 as  such, leading to the
     abovementioned performance	loss.  In particular, parts using
     '|'   or  {n}  (such  as  "foo{3}")  are  always  considered
     variable-length.

     Combining trailing	context	with the special '|'  action  can
     result  in	fixed trailing context being turned into the more
     expensive variable	trailing context.  For example,	this hap-
     pens in the following example:

	 %%
	 abc	  |
	 xyz/def


     Use of unput() invalidates	yytext and yyleng.

     Use of unput() to push back more text than	was  matched  can
     result  in	the pushed-back	text matching a	beginning-of-line
     ('^') rule	even though it didn't come at  the  beginning  of
     the line (though this is rare!).

     Pattern-matching  of  NUL's  is  substantially  slower  than
     matching other characters.

     flex does not generate correct  #line  directives	for  code
     internal to the scanner; thus, bugs in flex.skel yield bogus



Version	2.3	    Last change: 26 May	1990		       10






FLEX(1)			 USER COMMANDS			  FLEX(1)



     line numbers.

     Due to both buffering of input and	 read-ahead,  you  cannot
     intermix  calls to	<stdio.h> routines, such as, for example,
     getchar(),	with flex rules	and  expect  it	 to  work.   Call
     input() instead.

     The total table entries listed by the -v flag  excludes  the
     number  of	 table	entries	needed to determine what rule has
     been matched.  The	number of entries is equal to the  number
     of	 DFA states if the scanner does	not use	REJECT,	and some-
     what greater than the number of states if it does.

     REJECT cannot be used with	the -f or -F options.

     Some of the macros, such as  yywrap(),  may  in  the  future
     become  functions which live in the -lfl library.	This will
     doubtless break a lot of  code,  but  may	be  required  for
     POSIX-compliance.

     The flex internal algorithms need documentation.


































Version	2.3	    Last change: 26 May	1990		       11



