1

JavaCC

Programación de Sistemas

2

Que es un generador de parsers?

Total

= precio

+ iva ;

Scanner

Parser

precio

id + id

Expr

asignación

=Total

iva

T o t a l = p r e c i o + i v a ;

Parser generator (JavaCC)

Especificación

lexica+gramatical

3

JavaCC

• JavaCC (Java Compiler Compiler) es un generador de scanner y parser

• Producir un scanner y/o parser escrito en java, mismo que está escrito en Java;

• Hay muchos generadores de parsers– yacc (Yet Another Compiler-Compiler) para el lenguaje

de programación C– Bison de gnu.org

• Hay también muchos generadores de parsers escritos en Java¿– JavaCUP; – ANTLR;– SableCC

4

Más sobre la clasificación de generadores de parsers en java

• Herramientas generadoras de Parsers ascendentes. – JavaCUP;– jay, YACC for Java www.inf.uos.de/bernd/jay– SableCC, The Sable Compiler Compiler www.sablecc.org

• Herramientas generadoras de Parsers descendentes– ANTLR, Another Tool for Language Recognition

www.antlr.org– JavaCC, Java Compiler Compiler

www.webgain.com/java_cc

5

Características de JavaCC

• Generador de Parsers descendentes LL(K)• Especificación Lexica y gramática en un archivo• Procesador Tree Building

– con JJTree

• Extremadamente Ajustable– Muchas opciones diferentes seleccionables

• Generación de Documentación– Utilizando JJDoc

• Internacionalización– Puede manejar unicode completo

• Lookahead Sintáctico y Semántico

6

Características de JavaCC (cont.)

• Permite especificaciones extendidas en BNF– Puede utilizar | * ? + () en RHS.

• Estados y acciones Lexicas.• Análisis léxico sensitivo a mayúsculas y

minúsculas• Capacidad de depuración extensiva• Tokens especiales• Reporteador de error muy bueno

7

Instalación de JavaCC

• Descargar el archivo javacc-3.X.zip desde https://javacc.dev.java.net/

• Seguir el enlace que dice Download o ir directamente a https://javacc.dev.java.net/servlets/ProjectDocumentList

• unzip javacc-3.X.zip en un directorio %JCC_HOME%

• add %JCC_HOME\bin directory to your %path%.– javacc, jjtree, jjdoc may now be invoked directly from

the command line.

8

Pasos para usar JavaCC

• Escribir una especificación JavaCC (.jj file)– Define la gramática y acciones en un archivo (digamos,

calc.jj)

• Ejecutar javaCC para generar un scanner y un parser– javacc calc.jj– Generará el parser, scanner, token,… java sources

• Escribe el programa que utilice el parser– Por ejemplo, UseParser.java

• Compilar y ejecutar el programa– javac -classpath . *.java– java -cp . mainpackage.MainClass

9

Ejemplo 1

• Grammar : re.jj• Ejemplo

– % todas las cadenas terminan en "ab"– (a|b)*ab;– aba;– ababb;

• Nuestras tareas:– Por cada cadena de entrada (Linea 3,4) determinar

cuando coincida con la expresión regular (linea 2).

Parsear una especificación de expresiones regulares y que coincidan con las cadenas de entrada

10

La película completa

% comentario

(a|b)*ab;

a;

ab;

REParserTokenManager

javaCC

REParser

re.jj

tokens

resultado

MainClass

11

Formato de una gramática de entrada para JavaCC

• javacc_options

• PARSER_BEGIN ( <IDENTIFIER>1 )

unidad_de_compilación_de_java PARSER_END ( <IDENTIFIER>2 )

• ( produccion )*

12

El archivo de especificación de entrada (re.jj)

options {

USER_TOKEN_MANAGER=false;

BUILD_TOKEN_MANAGER=true;

OUTPUT_DIRECTORY="./reparser";

STATIC=false;

}

re.jj PARSER_BEGIN(REParser) package reparser;

import java.lang.*; … import dfa.*;public class REParser { public FA tg = new FA(); // mensaje de error con la linea actual public static void msg(String s) {

System.out.println("ERROR"+s); } public static void main(String args[]) throws Exception

REParser reparser = new REParser(System.in); reparser.S(); }}PARSER_END(REParser)

14

re.jj (Definición de tokens)

TOKEN : { <SYMBOL: ["0"-"9","a"-"z","A"-"Z"] > | <EPSILON: "epsilon" > | <LPAREN: "(“ > | <RPAREN: ")“ > | <OR: "|" > | <STAR: "*“ > | <SEMI: ";“ >}

SKIP: { < ( [" ","\t","\n","\r","\f"] )+ > |< "%" ( ~ ["\n"] )* "\n" > { System.out.println(image); }}

15

re.jj (producciones)

void S() : { FA d1; }{ d1 = R() <SEMI> { tg = d1; System.out.println("------NFA");

tg.print();

System.out.println("------DFA");tg = tg.NFAtoDFA(); tg.print();

System.out.println("------Minimizar");tg = tg.minimize(); tg.print();

System.out.println("------Renumerar");tg=tg.renumber(); tg.print();

System.out.println("------Ejecutar"); } testCases()

}

16

re.jj

void testCases() : {}{ (testCase() )+ }

void testCase(): { String testInput ;}{ testInput = symbols() <SEMI> { tg.execute( testInput) ; }}

String symbols() :{Token token = null; StringBuffer result = new

StringBuffer(); }{ ( token = <SYMBOL> { result.append( token.image) ; } )* { return result.toString(); }}

17

re.jj (expresiones regulares)

// R --> RUnit | RConcat | RChoice

FA R() : {FA result ;} { result = RChoice() { return result; } }

FA RUnit() : { FA result ; Token d1; }{ ( <LPAREN> result = RChoice() <RPAREN> |<EPSILON> { result = tg.epsilon(); } | d1 = <SYMBOL> { result = tg.symbol( d1.image ); } ) { return result ; } }

18

re.jj

FA RChoice() : { FA result, temp ;} { result = RConcat() ( <OR> temp = RConcat() { result =

result.choice( temp ) ;} )* {return result ; } }

FA RConcat() : { FA result, temp ;} { result = RStar() ( temp = RStar() { result =

result.concat( temp ) ;} )* {return result ; } }

FA RStar() : {FA result;} { result = RUnit() ( <STAR> { result = result.closure();} )* { return result; } }

19

Formato de una gramática de entrada de JavaCC

javacc_input ::= javacc_options PARSER_BEGIN ( <IDENTIFIER>1 )

unidad_de_compilacion_de_java PARSER_END ( <IDENTIFIER>2 )

( production )* <EOF>

Codigo de color:– azul --- no-terminal– <naranja> – un tipo de token– morado --- lexema ( palabra reservada; – I.e., consistente de la literal en sí misma)– negro -- meta simbolos

20

Notas

• <IDENTIFIER> significa cualquier identificador de Java como var, class2, …– IDENTIFIER significa solamente IDENTIFIER.

• <IDENTIFIER>1 debe ser igual a <IDENTIFIER>2

• unidad_de_compilacio_de_java es cualquier codigo de java que como un todo puede aparecer legalmente en un archivo.– Debe contener una declaración de clase principal con el mismo

nombre que <IDENTIFIER>1 .• Ejemplo:PARSER_BEGIN ( MiParser ) package mipackage; import miotropackage….; public class MiParser { … } class MiOtraClase { … } …PARSER_END (MiParser)

La entrada y salida de javacc

javaccPARSER_BEGIN ( MiParser ) package mipackage; import miotropackage….; public class MiParser { … } class MiOtraClase { … } …PARSER_END (MiParser)

(MiEspecifLeng.jj )

MyParser.java

MyParserTokenManager.javaMyParserCostant.java

Token.java

ParserError.java

22

Notes:

• Token.java y ParseError.jar son los mismos para todas las entradas y pueden ser reutilizados.

• package declaration in *.jj are copied to all 3 outputs.• import declarations in *.jj are copied to the parser and

token manager files.

• parser file is assigned the file name <IDENTIFIER>1 .java

• The parser file has contents: …class MiParser { … //generated parser is inserted here. … }• The generated token manager provides one public method: Token getNextToken() throws ParseError;

23

Especificación Léxica con JavaCC

24

javacc options

javacc_options ::=[ options { ( option_binding )* } ]

• option_binding es de la forma :– <IDENTIFIER>3 = <java_literal> ;

– donde <IDENTIFIER>3 no es sensible a mayúsculas y minúsculas.

• Ejemplo: options { USER_TOKEN_MANAGER=true; BUILD_TOKEN_MANAGER=false; OUTPUT_DIRECTORY="./sax2jcc/personnel"; STATIC=false; }

More Options• LOOKAHEAD

– java_integer_literal (1)

• CHOICE_AMBIGUITY_CHECK– java_integer_literal (2) for A | B … | C

• OTHER_AMBIGUITY_CHECK– java_integer_literal (1) for (A)*, (A)+ and (A)?

• STATIC (true)• DEBUG_PARSER (false)• DEBUG_LOOKAHEAD (false)• DEBUG_TOKEN_MANAGER (false)• OPTIMIZE_TOKEN_MANAGER

– java_boolean_literal (false)

• OUTPUT_DIRECTORY (current directory)• ERROR_REPORTING (true)

More Options• JAVA_UNICODE_ESCAPE (false)

– replace \u2245 to actual unicode (6 char 1 char)• UNICODE_INPUT (false)

– input strearm is in unicode form • IGNORE_CASE (false)• USER_TOKEN_MANAGER (false)

– generate TokenManager interface for user’s own scanner• USER_CHAR_STREAM (false)

– generate CharStream.java interface for user’s own inputStream• BUILD_PARSER (true)

– java_boolean_literal • BUILD_TOKEN_MANAGER (true)• SANITY_CHECK (true)• FORCE_LA_CHECK (false)• COMMON_TOKEN_ACTION (false)

– invoke void CommonTokenAction(Token t) after every getNextToken()• CACHE_TOKENS (false)

Ejemplo: Figura 2.21. if IF2. [a-z][a-z0-9]* ID3. [0-9]+ NUM4. ([0-9]+”.”[0-9]*) | ([0-9]*”.”[0-9]+) REAL5. (“--”[a-z]*”\n”) | (“ “|”\n” | “\t” )+ nonToken,

WS6. . error• Notaciones javacc 1. “if” or “i” “f” or [“i”][“f”]2. [“a”-”z”]([“a”-”z”,”0”-”9”])*3. ([“0”-”9”])+4. ([“0”-”9”])+ “.” ( [“0”-”9”] ) * | ([“0”-”9”])* ”.” ([“0”-”9”])+

Especificación JavaCC para algunos Tokens

PARSER_BEGIN(MiParser) class MiParser{}PARSER_END(MiParser)/* Para la expresión regular en la derecha, se

retornará el token a la izquierda */TOKEN : { < IF: “if” > | < #DIGIT: [“0”-”9”] >|< ID: [“a”-”z”] ( [“a”-”z”] | <DIGIT>)* >|< NUM: (<DIGIT>)+ >|< REAL: ( (<DIGIT>)+ “.” (<DIGIT>)* ) | ( <DIGIT>+ “.” (<DIGIT>)* ) >

}

Continuación/* Las expresiones regulares aquí serán omitidas

durante el análisis léxico */SKIP : { < “ “> | <“\t”> |<“\n”> }/* como SKIP pero el texto saltado es accesible

desde la acción del parser */SPECIAL_TOKEN : {<“--” ([“a”-”z”])* (“\n” | “\r” | “\n\r” ) >}/* . Para cualquier subcadena que no coincida con la

especificación léxica, javacc lanzara un error *//* regla principal */void start() : {}{ (<IF> | <ID> |<NUM> |<REAL>)* }

30

Especificación de Gramática con JavaCC

31

La forma de una Producción

java_return_type java_identifier ( java_parameter_list ) :

java_block {opciones_de_expansion }

• Ejemplo :void XMLDocument(Logger logger): { int msg = 0; }{ <StartDoc> { print(token); } Element(logger) <EndDoc> { print(token); } | else() }

32

Ejemplo ( Gramática )

1. P L2. S id := id3. S while id do S4. S begin L end5. S if id then S6. S if id then S else S7. L S8. L L;S

1,7,8 : P S (;S)*

33

JavaCC Version of Grammar 3.30

PARSER_BEGIN(MiParser) pulic class MiParser{}PARSRE_END(MiParser)

SKIP : {“ “ | “\t” | “\n” }

TOKEN: { <WHILE: “while”> | <BEGIN: “begin”> | <END:”end”>| <DO:”do”> | <IF:”if”> | <THEN : “then”> | <ELSE:”else”> | <SEMI: “;”> | <ASSIGN: “=“>|<#LETTER: [“a”-”z”]>| <ID: <LETTER>(<LETTER> | [“0”-”9”] )* >

}

34

JavaCC Version of Grammar 3.30 (cont’d)

void Prog() : { } { StmList() <EOF> }

void StmList(): { } { Stm() (“;” Stm() ) * }

void Stm(): { } { <ID> “=“ <ID>| “while” <ID> “do” Stm()| <BEGIN> StmList() <END>| “if” <ID> “then” Stm() [ LOOKAHEAD(1) “else” Stm() ]

}

35

Tipos de producciones

• production ::= javacode_production | regulr_expr_production

| bnf_production| token_manager_decl

Note: 1,3 se utilizan para definir gramáticas. 2 se usa para definir tokens

4 se usa para incrustar código en el token manager.

36

JAVACODE production

• javacode_production ::= “JAVACODE” java-return_type iava_id “(“ java_param_list “)” java_block

• Note:– Se utiliza para definir no-terminales para reconocer Used

to define nonterminals for recognizing sth that is hard to parse using normal production.

37

Example JAVACODE

JAVACODE void skip_to_matching_brace() { Token tok; int nesting = 1; while (true) { tok = getToken(1); if (tok.kind == LBRACE) nesting++; if (tok.kind == RBRACE) { nesting--; if (nesting == 0) break; } tok = getNextToken(); } }

38

Note:

• Do not use nonterminal defined by JAVACODE at choice point without giving LOOKHEAD.

• void NT() : {} { skip_to_matching_brace() | some_other_production() } • void NT() : {} { "{" skip_to_matching_brace() | "(" parameter_list() ")" }

39

TOKEN_MANAGER_DECLS

token_manager_decls ::= TOKEN_MGR_DECLS : java_block

• The token manager declarations starts with the reserved word "TOKEN_MGR_DECLS" followed by a ":" and then a set of Java declarations and statements (the Java block).

• These declarations and statements are written into the generated token manager (MyParserTokenManager.java) and are accessible from within lexical actions.

• There can only be one token manager declaration in a JavaCC grammar file.

40

regular_expression_production

regular_expr_production ::= [ lexical_state_list ] regexpr_kind [ [ IGNORE_CASE ] ] : { regexpr_spec ( | regexpr_spec )* }

• regexpr_kind::= TOKEN | SPECIAL_TOKEN | SKIP | MORE

• TOKEN is used to define normal tokens• SKIP is used to define skipped tokens (not passed to later

parser)• MORE is used to define semi-tokens (I.e. only part of a token).• SPECIAL_TOKEN is between TOKEN and SKIP tokens in that it

is passed on to the parser and accessible to the parser action but is ignored by production rules (not counted as an token). Useful for representing comments.

41

lexical_state_list

lexical_state_list::= < * > | < java_identifier ( , java_identifier )* >• The lexical state list describes the set of lexical

states for which the corresponding regular expression production applies.

• If this is written as "<*>", the regular expression production applies to all lexical states. Otherwise, it applies to all the lexical states in the identifier list within the angular brackets.

• if omitted, then a DEFAULT lexical state is assumed.

42

regexpr_spec

regexpr_spec::=regular_expression1 [ java_block ] [ : java_identifier ]

• Meaning:

• When a regular_expression1 is matched then– if java_block exists then execute it– if java_identifier appears, then transition to that lexical

state.

43

regular_expression

regular_expression ::= java_string_literal | < [ [#] java_identifier : ] complex_regular_expression_choices

> | <java_identifier> | <EOF>

• <EOF> is matched by end-of-file character only.• (3) <java_identifier> is a reference to other labeled

regular_expression.– used in bnf_production

• java_string_literal is matched only by the string denoted by itself.• (2) is used to defined a labled regular_expr and not visible to

outside the current TOKEN section if # occurs.• (1) for unnamed tokens

44

Example

<DEFAULT, LEX_ST2> TOKEN IGNORE_CASE : { < FLOATING_POINT_LITERAL: (["0"-"9"])+ "." (["0"-"9"])* (<EXPONENT>)?

(["f","F","d","D"])? | "." (["0"-"9"])+ (<EXPONENT>)? (["f","F","d","D"])? | (["0"-"9"])+ <EXPONENT> (["f","F","d","D"])? | (["0"-"9"])+ (<EXPONENT>)? ["f","F","d","D"] > { // do Something } : LEX_ST1 | < #EXPONENT: ["e","E"] (["+","-"])? (["0"-"9"])+ > } • Note: if # is omitted, E123 will be recognized erroneously as a token of kind EXPONENT.

45

Structure of complex_regular_expression

• complex_regular_expression_choices::= complex_regular_expression (|

complex_regular_expression )*• complex_regular_expression ::= ( complex_regular_expression_unit )*• complex_regular_expression_unit ::= java_string_literal | "<" java_identifier ">" | character_list | ( complex_regular_expression_choices ) [+|*|?]

• Note: unit concatenation;juxtapositionconcatenation;juxtaposition

complex_regular_expression choice; | choice; | complex_regular_expression_choice (.)[+|*|?](.)[+|*|?]

unit

46

character_list

character_list::= [~] [ [ character_descriptor ( , character_descriptor )*

] ]character_descriptor::= java_string_literal [ - java_string_literal ]java_string_literal ::= // reference to java grammar “ singleCharString* “ note: java_sting_literal here is restricted to length 1.ex:

– ~[“a”,”b”] --- all chars but a and b.– [“a”-”f”, “0”-”9”, “A”,”B”,”C”,”D”,”E”,”F”] --- hexadecimal

digit.– [“a”,”b”]+ is not a regular_expression_unit. Why ?

• should be written ( [“a”,”b”] )+ instead.

47

bnf_production

• bnf_production::=java_return_type java_identifier "("

java_parameter_list ")" ":"java_block "{" expansion_choices "}“

• expansion_choices::= expansion ( "|" expansion )*

• expansion::= ( expansion_unit )*

48

expansion_unit• expansion_unit::= local_lookahead| java_block| "(" expansion_choices ")" [ "+" | "*" | "?" ]| "[" expansion_choices "]"| [ java_assignment_lhs "=" ]

regular_expression| [ java_assignment_lhs "=" ] java_identifier "(" java_expression_list ")“Notes:1 is for lookahead; 2 is for semantic action4 = ( …)?5 is for token match6. is for match of other nonterminal

49

lookahead

• local_lookahead::= "LOOKAHEAD" "(" [ java_integer_literal ] [ "," ] [ expansion_choices ] [ "," ] [ "{" java_expression "}" ] ")“

• Notes:• 3 componets: max # lookahead + syntax +

semantics• examples:

– LOOKHEAD(3)– LOOKAHEAD(5, Expr() <INT> | <REAL> , { true} )

• More on LOOKAHEAD– see minitutorial

50

JavaCC API

• Non-Terminals in the Input Grammar• NT is a nonterminal =>

returntype NT(parameters) throws ParseError; is generated in the parser class

• API for Parser Actions• Token token;

– variable always holds the last token and can be used in parser actions.

– exactly the same as the token returned by getToken(0).

– two other methods - getToken(int i) and getNextToken() can also be used in actions to traverse the token list.

51

Token class

• public int kind;– 0 for <EOF>

• public int beginLine, beginColumn, endLine, endColumn;

• public String image;• public Token next;• public Token specialToken;• public String toString()• { return image; }• public static final Token newToken(int ofKind)

52

Error reporting and recovery

• It is not user friendly to throw an exception and exit the parsing once encountering a syntax error.

• two Exceptions– ParseException . can be recovered– TokenMgrError not expected to be recovered

• Error reporting– modify ParseExcpetion.java or TokenMgrError.java– generateParseException method is always invokable in

parser action to report error

53

Error Recovery in JavaCC:

• Shallow Error Recovery• Deep Error Recovery

• Shallow Error Recovery • Ex:void Stm() : {} { IfStm() | WhileStm() }

if getToken(1) != “if” or “while” => shallow error

54

Shallow recoverycan be recovered by additional choice:void Stm() : {} { IfStm() | WhileStm() | error_skipto(SEMICOLON) } whereJAVACODE void error_skipto(int kind) { ParseException e = generateParseException(); // generate the

exception object. System.out.println(e.toString()); // print the error message Token t; do { t = getNextToken(); } while (t.kind != kind);}

55

Deep Error Recovery

• Same example: void Stm() : {} { IfStm() | WhileStm() }

• But this time the error occurs during paring inside IfStmt() or WhileStmt() instead of the lookahead entry.

• The approach: use java try-catch construct.void Stm() : {} { try { ( IfStm() | WhileStm() ) } catch (ParseException e) { error_skipto(SEMICOLON); } } note: the new syntax for javacc bnf_production.

56

More Examples

• There are plenty examples on the net– http://www.vorlesungen.uni-

osnabrueck.de/informatik/compilerbau98/code/JavaCC/examples/

– JavaCC Grammar Repository• http://www.cobase.cs.ucla.edu/pub/javacc/

57

References

• http://xml.cs.nccu.edu.tw/courses/compiler/cp2003Fall/slides/javaCC.ppt

• Compilers Principles, Techniques and Tools, Aho, Sethi, and Ullman