CS374: Programming Language Principles - Tokens and Scanning Lexemes

Activity Goals

The goals of this activity are:
  1. To define a lexeme and to define tokens based on lexemes
  2. To differentiate between types, reserved words, and identifiers in a scanner
  3. To read a file and identify defined tokens

The Activity


Consider the activity models and answer the questions provided. First reflect on these questions on your own briefly, before discussing and comparing your thoughts with your group. Appoint one member of your group to discuss your findings with the class, and the rest of the group should help that member prepare their response. Answer each question individually from the activity, and compare with your group to prepare for our whole-class discussion. After class, think about the questions in the reflective prompt and respond to those individually in your notebook. Report out on areas of disagreement or items for which you and your group identified alternative approaches. Write down and report out questions you encountered along the way for group discussion.

Model 1: Scanning Lexemes for Tokens with flex


  1. What is a lexeme? What is a token?
  2. Describe an algorithm to read a String from left to right, returning tokens as they are identified. What familiar programming construct or paradigm do you see (hint - look at the loop, conditional, and check for the current character!)?
  3. How would you modify this scanner configuration to support an assignment operator and an identifier (variable) token?
  4. Notice the actual text of the token is not typically returned directly by the scanner. Rather, it is stored in a variable (here, yylval). What is returned instead, and why not simply return yytext directly?
  5. Sketch the symbol table for the lexemes above. For each token type, assign a constant value to the type, and describe the type and format of the resulting token value.


I encourage you to submit your answers to the questions (and ask your own questions!) using the Class Activity Questions discussion board. You may also respond to questions or comments made by others, or ask follow-up questions there. Answer any reflective prompt questions in the Reflective Journal section of your OneNote Classroom personal section. You can find the link to the class notebook on the syllabus.