JavaScript 2.0 Lexer Semantics

JavaScript 2.0

Formal Description

Lexer Semantics

Thursday, July 22, 1999

The lexer semantics describe the actions the lexer takes in order to transform an input stream of Unicode characters into a stream of tokens. For convenience, the lexer grammar is repeated here. See also the description of the semantic notation.

This document is also available as a Word 98 rtf file.

The start symbols are NextToken^re and NextToken^div depending on whether a / should be interpreted as a regular expression or division.

Unicode Character Classes

Syntax

UnicodeCharacter Any Unicode character

UnicodeInitialAlphabetic Any Unicode initial alphabetic character (includes ASCII A-Z and a-z)

UnicodeAlphanumeric Any Unicode alphabetic or decimal digit character (includes ASCII 0-9, A-Z, and a-z)

WhiteSpaceCharacter

«TAB» | «VT» | «FF» | «SP» | «u00A0»

| «u2000» | «u2001» | «u2002» | «u2003» | «u2004» | «u2005» | «u2006» | «u2007»

| «u2008» | «u2009» | «u200A» | «u200B»

| «u3000»

LineTerminator «LF» | «CR» | «u2028» | «u2029»

ASCIIDigit 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9

Semantics

action DecimalValue[ASCIIDigit] : Integer = digitValue(ASCIIDigit)

Comments

Syntax

LineComment / / LineCommentCharacters

LineCommentCharacters

«empty»

| LineCommentCharacters NonTerminator

NonTerminator UnicodeCharacter except LineTerminator

BlockComment / * BlockCommentCharacters * /

BlockCommentCharacters

«empty»

| BlockCommentCharacters NonSlash

| PreSlashCharacters /

PreSlashCharacters

«empty»

| BlockCommentCharacters NonAsteriskOrSlash

| PreSlashCharacters /

NonSlash UnicodeCharacter except /

NonAsteriskOrSlash UnicodeCharacter except * | /

White space

Syntax

WhiteSpace

«empty»

| WhiteSpace WhiteSpaceCharacter

| WhiteSpace LineTerminator

| WhiteSpace LineComment LineTerminator

| WhiteSpace BlockComment

Tokens

Syntax

t {re, div}

NextToken^t WhiteSpace Token^t

Token^re

IdentifierOrReservedWord

| Punctuator

| NumericLiteral

| QuantityLiteral

| StringLiteral

| RegExpLiteral^slash

| RegExpLiteral^guillemet

| EndOfInput

Token^div

IdentifierOrReservedWord

| Punctuator

| DivisionPunctuator

| NumericLiteral

| QuantityLiteral

| StringLiteral

| RegExpLiteral^guillemet

| EndOfInput

EndOfInput

End

| LineComment End

Semantics

type RegExp = tuple {reBody: String; reFlags: String}

type Quantity = tuple {amount: Double; unit: String}

type Token
  = oneof {
           identifier: String;
           keyword: String;
           punctuator: String;
           number: Double;
           quantity: Quantity;
           string: String;
           regularExpression: RegExp;
           end}

action Token[NextToken^t] : Token

Token[NextToken^t WhiteSpace Token^t] = Token[Token^t]

action RegExpMayFollow[NextToken^t] : Boolean

RegExpMayFollow[NextToken^t WhiteSpace Token^t] = RegExpMayFollow[Token^t]

action Token[Token^t] : Token

Token[Token^t IdentifierOrReservedWord] = Token[IdentifierOrReservedWord]

Token[Token^t Punctuator] = Token[Punctuator]

Token[Token^div DivisionPunctuator] = punctuator Punctuator[DivisionPunctuator]

Token[Token^t NumericLiteral] = number DoubleValue[NumericLiteral]

Token[Token^t QuantityLiteral] = quantity QuantityValue[QuantityLiteral]

Token[Token^t StringLiteral] = string StringValue[StringLiteral]

Token[Token^re RegExpLiteral^slash] = regularExpression REValue[RegExpLiteral^slash]

Token[Token^t RegExpLiteral^guillemet]
= regularExpression REValue[RegExpLiteral^guillemet]

Token[Token^t EndOfInput] = end

action RegExpMayFollow[Token^t] : Boolean

RegExpMayFollow[Token^t IdentifierOrReservedWord]
= RegExpMayFollow[IdentifierOrReservedWord]

RegExpMayFollow[Token^t Punctuator] = RegExpMayFollow[Punctuator]

RegExpMayFollow[Token^div DivisionPunctuator] = true

RegExpMayFollow[Token^t NumericLiteral] = false

RegExpMayFollow[Token^t QuantityLiteral] = false

RegExpMayFollow[Token^t StringLiteral] = false

RegExpMayFollow[Token^re RegExpLiteral^slash] = false

RegExpMayFollow[Token^t RegExpLiteral^guillemet] = false

RegExpMayFollow[Token^t EndOfInput] = true

Keywords and identifiers

Syntax

IdentifierName

InitialIdentifierCharacter

| IdentifierName ContinuingIdentifierCharacter

InitialIdentifierCharacter

OrdinaryInitialIdentifierCharacter

| \ HexEscape

OrdinaryInitialIdentifierCharacter UnicodeInitialAlphabetic | $ | _

ContinuingIdentifierCharacter

OrdinaryContinuingIdentifierCharacter

| \ HexEscape

OrdinaryContinuingIdentifierCharacter UnicodeAlphanumeric | $ | _

Semantics

action Name[IdentifierName] : String

Name[IdentifierName InitialIdentifierCharacter]
= [CharacterValue[InitialIdentifierCharacter]]

Name[IdentifierName IdentifierName₁ ContinuingIdentifierCharacter]
= Name[IdentifierName₁] [CharacterValue[ContinuingIdentifierCharacter]]

action ContainsEscapes[IdentifierName] : Boolean

ContainsEscapes[IdentifierName InitialIdentifierCharacter]
= ContainsEscapes[InitialIdentifierCharacter]

ContainsEscapes[IdentifierName IdentifierName₁ ContinuingIdentifierCharacter]
= ContainsEscapes[IdentifierName₁] or ContainsEscapes[ContinuingIdentifierCharacter]

action CharacterValue[InitialIdentifierCharacter] : Character

CharacterValue[InitialIdentifierCharacter OrdinaryInitialIdentifierCharacter]
= OrdinaryInitialIdentifierCharacter

CharacterValue[InitialIdentifierCharacter  \ HexEscape]
  = if isOrdinaryInitialIdentifierCharacter(CharacterValue[HexEscape])
     then CharacterValue[HexEscape]
     else

action ContainsEscapes[InitialIdentifierCharacter] : Boolean

ContainsEscapes[InitialIdentifierCharacter OrdinaryInitialIdentifierCharacter] = false

ContainsEscapes[InitialIdentifierCharacter \ HexEscape] = true

action CharacterValue[ContinuingIdentifierCharacter] : Character

CharacterValue[ContinuingIdentifierCharacter OrdinaryContinuingIdentifierCharacter]
= OrdinaryContinuingIdentifierCharacter

CharacterValue[ContinuingIdentifierCharacter  \ HexEscape]
  = if isOrdinaryContinuingIdentifierCharacter(CharacterValue[HexEscape])
     then CharacterValue[HexEscape]
     else

action ContainsEscapes[ContinuingIdentifierCharacter] : Boolean

ContainsEscapes[ContinuingIdentifierCharacter OrdinaryContinuingIdentifierCharacter]
= false

ContainsEscapes[ContinuingIdentifierCharacter \ HexEscape] = true

reservedWordsRE : String[]
  = [“abstract”,
      “break”,
      “case”,
      “catch”,
      “class”,
      “const”,
      “continue”,
      “debugger”,
      “default”,
      “delete”,
      “do”,
      “else”,
      “enum”,
      “eval”,
      “export”,
      “extends”,
      “field”,
      “final”,
      “finally”,
      “for”,
      “function”,
      “goto”,
      “if”,
      “implements”,
      “import”,
      “in”,
      “instanceof”,
      “native”,
      “new”,
      “package”,
      “private”,
      “protected”,
      “public”,
      “return”,
      “static”,
      “switch”,
      “synchronized”,
      “throw”,
      “throws”,
      “transient”,
      “try”,
      “typeof”,
      “var”,
      “volatile”,
      “while”,
      “with”]

reservedWordsDiv : String[] = [“false”, “null”, “super”, “this”, “true”]

nonReservedWords : String[]
= [“constructor”, “getter”, “method”, “override”, “setter”, “traditional”, “version”]

keywords : String[] = reservedWordsRE reservedWordsDiv nonReservedWords

member(id: String, list: String[]) : Boolean
  = if |list| = 0
     then false
     else if id = list[0]
     then true
     else member(id, list[1 ...])

Syntax

IdentifierOrReservedWord IdentifierName

Semantics

action Token[IdentifierOrReservedWord] : Token

Token[IdentifierOrReservedWord  IdentifierName]
  = let id: String = Name[IdentifierName]
     in if member(id, keywords) and not ContainsEscapes[IdentifierName]
         then keyword id
         else identifier id

action RegExpMayFollow[IdentifierOrReservedWord] : Boolean

RegExpMayFollow[IdentifierOrReservedWord  IdentifierName]
  = let id: String = Name[IdentifierName]
     in member(id, reservedWordsRE) and not ContainsEscapes[IdentifierName]

Punctuators

Syntax

Punctuator

PunctuatorRE

| PunctuatorDiv

PunctuatorRE

!

| ! =

| ! = =

| #

| %

| % =

| &

| & &

| & & =

| & =

| (

| *

| * =

| +

| + =

| ,

| -

| - =

| - >

| .

| . .

| . . .

| :

| : :

| ;

| <

| < <

| < < =

| < =

| =

| = =

| = = =

| >

| > =

| > >

| > > =

| > > >

| > > > =

| ?

| @

| [

| ^

| ^ =

| ^ ^

| ^ ^ =

| {

| |

| | =

| | |

| | | =

| ~

PunctuatorDiv

)

| + +

| - -

| ]

| }

DivisionPunctuator

/

| / =

Semantics

action Token[Punctuator] : Token

Token[Punctuator PunctuatorRE] = punctuator Punctuator[PunctuatorRE]

Token[Punctuator PunctuatorDiv] = punctuator Punctuator[PunctuatorDiv]

action RegExpMayFollow[Punctuator] : Boolean

RegExpMayFollow[Punctuator PunctuatorRE] = true

RegExpMayFollow[Punctuator PunctuatorDiv] = false

action Punctuator[PunctuatorRE] : String

Punctuator[PunctuatorRE !] = “!”

Punctuator[PunctuatorRE ! =] = “!=”

Punctuator[PunctuatorRE ! = =] = “!==”

Punctuator[PunctuatorRE #] = “#”

Punctuator[PunctuatorRE %] = “%”

Punctuator[PunctuatorRE % =] = “%=”

Punctuator[PunctuatorRE &] = “&”

Punctuator[PunctuatorRE & &] = “&&”

Punctuator[PunctuatorRE & & =] = “&&=”

Punctuator[PunctuatorRE & =] = “&=”

Punctuator[PunctuatorRE (] = “(”

Punctuator[PunctuatorRE *] = “*”

Punctuator[PunctuatorRE * =] = “*=”

Punctuator[PunctuatorRE +] = “+”

Punctuator[PunctuatorRE + =] = “+=”

Punctuator[PunctuatorRE ,] = “,”

Punctuator[PunctuatorRE -] = “-”

Punctuator[PunctuatorRE - =] = “-=”

Punctuator[PunctuatorRE - >] = “->”

Punctuator[PunctuatorRE .] = “.”

Punctuator[PunctuatorRE . .] = “..”

Punctuator[PunctuatorRE . . .] = “...”

Punctuator[PunctuatorRE :] = “:”

Punctuator[PunctuatorRE : :] = “::”

Punctuator[PunctuatorRE ;] = “;”

Punctuator[PunctuatorRE <] = “<”

Punctuator[PunctuatorRE < <] = “<<”

Punctuator[PunctuatorRE < < =] = “<<=”

Punctuator[PunctuatorRE < =] = “<=”

Punctuator[PunctuatorRE =] = “=”

Punctuator[PunctuatorRE = =] = “==”

Punctuator[PunctuatorRE = = =] = “===”

Punctuator[PunctuatorRE >] = “>”

Punctuator[PunctuatorRE > =] = “>=”

Punctuator[PunctuatorRE > >] = “>>”

Punctuator[PunctuatorRE > > =] = “>>=”

Punctuator[PunctuatorRE > > >] = “>>>”

Punctuator[PunctuatorRE > > > =] = “>>>=”

Punctuator[PunctuatorRE ?] = “?”

Punctuator[PunctuatorRE @] = “@”

Punctuator[PunctuatorRE [] = “[”

Punctuator[PunctuatorRE ^] = “^”

Punctuator[PunctuatorRE ^ =] = “^=”

Punctuator[PunctuatorRE ^ ^] = “^^”

Punctuator[PunctuatorRE ^ ^ =] = “^^=”

Punctuator[PunctuatorRE {] = “{”

Punctuator[PunctuatorRE |] = “|”

Punctuator[PunctuatorRE | =] = “|=”

Punctuator[PunctuatorRE | |] = “||”

Punctuator[PunctuatorRE | | =] = “||=”

Punctuator[PunctuatorRE ~] = “~”

action Punctuator[PunctuatorDiv] : String

Punctuator[PunctuatorDiv )] = “)”

Punctuator[PunctuatorDiv + +] = “++”

Punctuator[PunctuatorDiv - -] = “--”

Punctuator[PunctuatorDiv ]] = “]”

Punctuator[PunctuatorDiv }] = “}”

action Punctuator[DivisionPunctuator] : String

Punctuator[DivisionPunctuator /] = “/”

Punctuator[DivisionPunctuator / =] = “/=”

Numeric literals

Syntax

NumericLiteral

DecimalLiteral

| HexIntegerLiteral [lookahead{HexDigit}]

| OctalIntegerLiteral

Semantics

action DoubleValue[NumericLiteral] : Double

DoubleValue[NumericLiteral DecimalLiteral]
= rationalToDouble(RationalValue[DecimalLiteral])

DoubleValue[NumericLiteral HexIntegerLiteral [lookahead{HexDigit}]]
= rationalToDouble(IntegerValue[HexIntegerLiteral])

DoubleValue[NumericLiteral OctalIntegerLiteral]
= rationalToDouble(IntegerValue[OctalIntegerLiteral])

expt(base: Rational, exponent: Integer) : Rational
  = if exponent = 0
     then 1
     else if exponent < 0
     then 1/expt(base, -exponent)
     else base*expt(base, exponent - 1)

Syntax

DecimalLiteral

Mantissa

| Mantissa LetterE SignedInteger

LetterE E | e

Mantissa

DecimalIntegerLiteral

| DecimalIntegerLiteral .

| DecimalIntegerLiteral . Fraction

| . Fraction

DecimalIntegerLiteral

0

| NonZeroDecimalDigits

NonZeroDecimalDigits

NonZeroDigit

| NonZeroDecimalDigits ASCIIDigit

NonZeroDigit 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9

Fraction DecimalDigits

Semantics

action RationalValue[DecimalLiteral] : Rational

RationalValue[DecimalLiteral Mantissa] = RationalValue[Mantissa]

RationalValue[DecimalLiteral Mantissa LetterE SignedInteger]
= RationalValue[Mantissa]*expt(10, IntegerValue[SignedInteger])

action RationalValue[Mantissa] : Rational

RationalValue[Mantissa DecimalIntegerLiteral] = IntegerValue[DecimalIntegerLiteral]

RationalValue[Mantissa DecimalIntegerLiteral .] = IntegerValue[DecimalIntegerLiteral]

RationalValue[Mantissa DecimalIntegerLiteral . Fraction]
= IntegerValue[DecimalIntegerLiteral] + RationalValue[Fraction]

RationalValue[Mantissa . Fraction] = RationalValue[Fraction]

action IntegerValue[DecimalIntegerLiteral] : Integer

IntegerValue[DecimalIntegerLiteral 0] = 0

IntegerValue[DecimalIntegerLiteral NonZeroDecimalDigits]
= IntegerValue[NonZeroDecimalDigits]

action IntegerValue[NonZeroDecimalDigits] : Integer

IntegerValue[NonZeroDecimalDigits NonZeroDigit] = DecimalValue[NonZeroDigit]

IntegerValue[NonZeroDecimalDigits NonZeroDecimalDigits₁ ASCIIDigit]
= 10*IntegerValue[NonZeroDecimalDigits₁] + DecimalValue[ASCIIDigit]

action DecimalValue[NonZeroDigit] : Integer = digitValue(NonZeroDigit)

action RationalValue[Fraction] : Rational

RationalValue[Fraction DecimalDigits]
= IntegerValue[DecimalDigits]/expt(10, NDigits[DecimalDigits])

Syntax

SignedInteger

DecimalDigits

| + DecimalDigits

| - DecimalDigits

Semantics

action IntegerValue[SignedInteger] : Integer

IntegerValue[SignedInteger DecimalDigits] = IntegerValue[DecimalDigits]

IntegerValue[SignedInteger + DecimalDigits] = IntegerValue[DecimalDigits]

IntegerValue[SignedInteger - DecimalDigits] = -IntegerValue[DecimalDigits]

Syntax

DecimalDigits

ASCIIDigit

| DecimalDigits ASCIIDigit

Semantics

action IntegerValue[DecimalDigits] : Integer

IntegerValue[DecimalDigits ASCIIDigit] = DecimalValue[ASCIIDigit]

IntegerValue[DecimalDigits DecimalDigits₁ ASCIIDigit]
= 10*IntegerValue[DecimalDigits₁] + DecimalValue[ASCIIDigit]

action NDigits[DecimalDigits] : Integer

NDigits[DecimalDigits ASCIIDigit] = 1

NDigits[DecimalDigits DecimalDigits₁ ASCIIDigit] = NDigits[DecimalDigits₁] + 1

Syntax

HexIntegerLiteral

0 LetterX HexDigit

| HexIntegerLiteral HexDigit

LetterX X | x

HexDigit 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | A | B | C | D | E | F | a | b | c | d | e | f

OctalIntegerLiteral

0 OctalDigit

| OctalIntegerLiteral OctalDigit

OctalDigit 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7

Semantics

action IntegerValue[HexIntegerLiteral] : Integer

IntegerValue[HexIntegerLiteral 0 LetterX HexDigit] = HexValue[HexDigit]

IntegerValue[HexIntegerLiteral HexIntegerLiteral₁ HexDigit]
= 16*IntegerValue[HexIntegerLiteral₁] + HexValue[HexDigit]

action HexValue[HexDigit] : Integer = digitValue(HexDigit)

action IntegerValue[OctalIntegerLiteral] : Integer

IntegerValue[OctalIntegerLiteral 0 OctalDigit] = OctalValue[OctalDigit]

IntegerValue[OctalIntegerLiteral OctalIntegerLiteral₁ OctalDigit]
= 8*IntegerValue[OctalIntegerLiteral₁] + OctalValue[OctalDigit]

action OctalValue[OctalDigit] : Integer = digitValue(OctalDigit)

Quantity literals

Syntax

QuantityLiteral NumericLiteral QuantityName

QuantityName [lookahead{LetterE, LetterX}] IdentifierName

Semantics

action QuantityValue[QuantityLiteral] : Quantity

QuantityValue[QuantityLiteral NumericLiteral QuantityName]
= amount DoubleValue[NumericLiteral], unit Name[QuantityName]

action Name[QuantityName] : String

Name[QuantityName [lookahead{LetterE, LetterX}] IdentifierName]
= Name[IdentifierName]

String literals

Syntax

q {single, double}

StringLiteral

' StringChars^single '

| " StringChars^double "

Semantics

action StringValue[StringLiteral] : String

StringValue[StringLiteral ' StringChars^single '] = StringValue[StringChars^single]

StringValue[StringLiteral " StringChars^double "] = StringValue[StringChars^double]

Syntax

StringChars^q

«empty»

| StringChars^q StringChar^q

StringChar^q

LiteralStringChar^q

| \ StringEscape

LiteralStringChar^single UnicodeCharacter except ' | \ | LineTerminator

LiteralStringChar^double UnicodeCharacter except " | \ | LineTerminator

Semantics

action StringValue[StringChars^q] : String

StringValue[StringChars^q «empty»] = “”

StringValue[StringChars^q StringChars^q₁ StringChar^q]
= StringValue[StringChars^q₁] [CharacterValue[StringChar^q]]

action CharacterValue[StringChar^q] : Character

CharacterValue[StringChar^q LiteralStringChar^q] = LiteralStringChar^q

CharacterValue[StringChar^q \ StringEscape] = CharacterValue[StringEscape]

Syntax

StringEscape

ControlEscape

| OctalEscape

| HexEscape

| IdentityEscape

IdentityEscape NonTerminator except UnicodeAlphanumeric

Semantics

action CharacterValue[StringEscape] : Character

CharacterValue[StringEscape ControlEscape] = CharacterValue[ControlEscape]

CharacterValue[StringEscape OctalEscape] = CharacterValue[OctalEscape]

CharacterValue[StringEscape HexEscape] = CharacterValue[HexEscape]

CharacterValue[StringEscape IdentityEscape] = IdentityEscape

Syntax

ControlEscape

b

| f

| n

| r

| t

| v

Semantics

action CharacterValue[ControlEscape] : Character

CharacterValue[ControlEscape b] = ‘«BS»’

CharacterValue[ControlEscape f] = ‘«FF»’

CharacterValue[ControlEscape n] = ‘«LF»’

CharacterValue[ControlEscape r] = ‘«CR»’

CharacterValue[ControlEscape t] = ‘«TAB»’

CharacterValue[ControlEscape v] = ‘«VT»’

Syntax

OctalEscape

OctalDigit [lookahead{OctalDigit}]

| ZeroToThree OctalDigit [lookahead{OctalDigit}]

| FourToSeven OctalDigit

| ZeroToThree OctalDigit OctalDigit

ZeroToThree 0 | 1 | 2 | 3

FourToSeven 4 | 5 | 6 | 7

Semantics

action CharacterValue[OctalEscape] : Character

CharacterValue[OctalEscape OctalDigit [lookahead{OctalDigit}]]
= codeToCharacter(OctalValue[OctalDigit])

CharacterValue[OctalEscape ZeroToThree OctalDigit [lookahead{OctalDigit}]]
= codeToCharacter(8*OctalValue[ZeroToThree] + OctalValue[OctalDigit])

CharacterValue[OctalEscape FourToSeven OctalDigit]
= codeToCharacter(8*OctalValue[FourToSeven] + OctalValue[OctalDigit])

CharacterValue[OctalEscape  ZeroToThree OctalDigit₁ OctalDigit₂]
  = codeToCharacter(
         64*OctalValue[ZeroToThree] + 8*OctalValue[OctalDigit₁] + OctalValue[OctalDigit₂])

action OctalValue[ZeroToThree] : Integer = digitValue(ZeroToThree)

action OctalValue[FourToSeven] : Integer = digitValue(FourToSeven)

Syntax

HexEscape

x HexDigit HexDigit

| u HexDigit HexDigit HexDigit HexDigit

Semantics

action CharacterValue[HexEscape] : Character

CharacterValue[HexEscape x HexDigit₁ HexDigit₂]
= codeToCharacter(16*HexValue[HexDigit₁] + HexValue[HexDigit₂])

CharacterValue[HexEscape  u HexDigit₁ HexDigit₂ HexDigit₃ HexDigit₄]
  = codeToCharacter(
         4096*HexValue[HexDigit₁] + 256*HexValue[HexDigit₂] + 16*HexValue[HexDigit₃] +
         HexValue[HexDigit₄])