---

Contents

• Overview
• Format of a Flex Specification
  • Regular Expression Rules
    • Self-Study #1
  • Flex Directives
    • Self-Study #2
  • Comments
  • States
    • Self-Study #3
• yyline and yytext
• A Small Example
• Quick Reference Guide

Overview

Flex is a scanner generator that produces C++ code. Here's a picture illustrating how to create and run a program using Flex:

The input to Flex is a specification that includes a set of regular expressions and associated actions. By default, the output of Flex is a C source program that runs a scanner. Using the options we will be using, Flex will output a C++ source file that defines a class encapsulating the scanner. In case of the C program, there is a function yylex. In case of the C++ program, the yylex function will be a member of the scanner class. yylex includes a constructor that allows for an input stream and/or output stream (of class istream and ostream, respectively) It also includes a method called yylex, which returns the next token in the input.

The picture above assumes that a class named P2 has been defined that contains the core program of interest. That program will declare an object of (abstract) type FlexLexer, and initialize it to the concrete subclass yyFlexLexer. The program then repeatedly calls the yylex function to get tokens until the FlexLexer runs out of tokens.

Format of a Flex Specification

A Flex specification has three parts, separated by double percent signs:

1. User code: this code is plopped into the output code and is appropriate for code definitions used in the actions.
2. Flex directives: This includes macro definitions (described below). See the Flex Reference Manual for more information about this part of the specification.
3. Regular expression rules: These rules specify how to divide up the input into tokens. Each rule includes an optional state list, a regular expression, and an associated action.

We will discuss the regular expression rules part first.

Regular Expressions Rules

The state-list part of a rule is discussed below. Ignoring state-lists for now, the form of a regular expression rule is:

When the scanner's yylex method is called, it repeats:

1. Find the longest sequence of characters in the input (starting with the current character) that matches a regular-expression pattern.
2. Perform the associated action.

until an action causes the yylex method to return. If there are several patterns that match the same (longest) sequence of characters, then the first such pattern is considered to be matched (so the order of the regular-expression rules can be important).

If an input character is not matched in any pattern, the scanner throws an exception. It is not good to have a scanner that can "crash" on bad input, so it is important to make sure that there can be no such unmatched characters!

The regular expressions are similar to the ones discussed in the scanner notes. Here's how they are used to match the input:

• Most characters match themselves. For example:
  • abc
  • ==
  • while
  are three patterns that match exactly those sequences of characters (note that writing one character after another means "followed by" as usual).

• Characters (even special characters, except backslash) enclosed in double quotes match themselves. For example, the following patterns are equivalent to the three given above:
  • "abc"
  • "=="
  • "while"
  And the following pattern:

  "a|b"

  matches the three-character sequence: a then | then b, rather than matching a single a or a single b.

• The following characters have the usual special meanings as regular expression operators:

  |	means "or"
  *	means zero or more instances of
  +	means one or more instances of
  ?	means zero or one instance of
  ( )	are used for grouping

• The dot character matches any character except the newline character. It is usually used in the last rule in the specification, to match all "bad" characters (and the associated action issues an error message).

• The backslash is a special escape character:

  \n	newline
  \t	tab
  \"	double quote

  To match a backslash character, put two backslashes in a character class (see below). See the Flex Reference Manual for a complete list of the special characters escaped by a backslash.

• The carat and dollar-sign characters: ^ and $, are special characters. When the carat is used as the first character in a pattern, it causes the pattern to match only at the beginning of a line (i.e., only if the previous character was a newline). When the dollar sign is used as the last character in a pattern, it causes the pattern to match only at the end of a line (i.e., only if the next character is a newline).

• The regular expression can include character classes, delimited by square brackets:
  • A character class will match one character.
  • If no special characters are used inside the character class, then the character class matches any of the characters it includes inside its square brackets. For example: [abc] matches an a, or a b, or a c, so it is the same as: a|b|c.
  • Here are the characters that are "special" inside a character class:

    -	means a range of characters; e.g., a-z means "a to z".
    ^	is only a special character if it is the first character in the square brackets; it means not any of the following characters. So for example, [^abc] matches any character other than an a, or a b, or a c.
    \	is used as an escape character with n, b, ", etc as usual; it can also be used to escape the characters that are special inside a character class (e.g., [a\-z] matches an a or a - or a z, and [\\] matches a backslash.
    "	can be used around characters that are special inside a character class to make them match themselves (e.g., ["\"] matches a backslash, and ["-"] matches a hyphen. To include a double-quote character in a character class, escape it with a backslash.

Note that whitespace only matches itself if it is inside quotes or in a character class; otherwise, it ends the current pattern. So the two rules:

are equivalent; each matches an a, or a space, or a b, or a c, while the rule:

causes an error when you try to process your specification.

---

You Try #1

Question 1: The character class [a-zA-Z] matches any letter. Write a character class that matches any letter or any digit.

Question 2: Write a pattern that matches any Pascal identifier (a sequence of one or more letters and/or digits, starting with a letter).

Question 3: Write a pattern that matches any C identifier (a sequence of one or more letters and/or digits and/or underscores, starting with a letter or underscore).

Question 4: Write a pattern that matches any C identifier that does not end with an underscore.

solution

---

Flex directives

Recall that the second part of a Flex specification contains directives. This can include specifying the value that should be returned on end-of-file, specifying that line counting should be turned on, and specifying that the scanner will be used with the Bison parser generator. (See the Flex Reference Manual for more information about directives.)

The directives part also includes macro definitions. The form of a macro definition is:

where name is any valid C identifier, and regular-expression is any regular expression as defined above.

DIGIT [0-9]
DOT   "."

%x DOTSTATE

%%

<INITIAL>{DIGIT}+{DOT}  { BEGIN(DOTSTATE);
                              -- save the value so far --
                        }

<DOTSTATE>{DIGIT}+      { BEGIN(INITIAL);
                              -- return the saved value and the new one --
                        }

DIGIT		[0-9]
LETTER		[a-zA-Z]
WHITESPACE	[ \t\n]

%option c++

%%

{LETTER}({LETTER}|{DIGIT})* { std::cout << "ID " << yytext;}
{DIGIT}+                    { std::cout << "INT";}
"="                         { std::cout << "ASSIGN";}
"=="                        { std::cout << "EQUALS";}
{WHITESPACE}*               { }
.               	    { std::cout << "bad char";}

%%

int main(){
	FlexLexer * lexer = new yyFlexLexer();
	lexer->yylex();
}