JavaScript 2.0 Lexer Semantics

JavaScript 2.0

Formal Description

Lexer Semantics

Saturday, May 15, 1999

The lexer semantics describe the actions the lexer takes in order to transform an input stream of Unicode characters into a stream of tokens. For convenience, the lexer grammar is repeated here. See also the description of the semantic notation.

This document is also available as a Word 98 rtf file.

The start symbols are NextToken^re and NextToken^div depending on whether a / should be interpreted as a regular expression or division.

Unicode Character Classes

Syntax

UnicodeCharacter Any Unicode character

UnicodeInitialAlphabetic Any Unicode initial alphabetic character (includes ASCII A-Z and a-z)

UnicodeAlphanumeric Any Unicode alphabetic or decimal digit character (includes ASCII 0-9, A-Z, and a-z)

WhiteSpaceCharacter

«TAB» | «VT» | «FF» | «SP» | «u00A0»

| «u2000» | «u2001» | «u2002» | «u2003» | «u2004» | «u2005» | «u2006» | «u2007»

| «u2008» | «u2009» | «u200A» | «u200B»

| «u3000»

LineTerminator «LF» | «CR» | «u2028» | «u2029»

ASCIIDigit 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9

Semantics

action DigitValue[ASCIIDigit] : Integer = digitValue(ASCIIDigit)

Comments

Syntax

LineComment / / LineCommentCharacters

LineCommentCharacters

«empty»

| LineCommentCharacters NonTerminator

NonTerminator UnicodeCharacter except LineTerminator

BlockComment / * BlockCommentCharacters * /

BlockCommentCharacters

«empty»

| BlockCommentCharacters NonSlash

| PreSlashCharacters /

PreSlashCharacters

«empty»

| BlockCommentCharacters NonAsteriskOrSlash

| PreSlashCharacters /

NonSlash UnicodeCharacter except /

NonAsteriskOrSlash UnicodeCharacter except * | /

White space

Syntax

WhiteSpace

«empty»

| WhiteSpace WhiteSpaceCharacter

| WhiteSpace LineTerminator

| WhiteSpace LineComment LineTerminator

| WhiteSpace BlockComment

Tokens

Syntax

t {re, div}

NextToken^t WhiteSpace Token^t

Token^re

IdentifierOrReservedWord

| Punctuator

| NumericLiteral

| QuantityLiteral

| StringLiteral

| RegExpLiteral^slash

| RegExpLiteral^guillemet

| EndOfInput

Token^div

IdentifierOrReservedWord

| Punctuator

| DivisionPunctuator

| NumericLiteral

| QuantityLiteral

| StringLiteral

| RegExpLiteral^guillemet

| EndOfInput

EndOfInput

End

| LineComment End

Semantics

type RegExp = tuple {reBody: String; reFlags: String}

type Quantity = tuple {amount: Double; unit: String}

type Token
  = oneof {
           identifier: String;
           keyword: String;
           punctuator: String;
           number: Double;
           quantity: Quantity;
           string: String;
           regularExpression: RegExp;
           end}

action Token[NextToken^t] : Token

Token[NextToken^t WhiteSpace Token^t] = Token[Token^t]

action RegExpMayFollow[NextToken^t] : Boolean

RegExpMayFollow[NextToken^t WhiteSpace Token^t] = RegExpMayFollow[Token^t]

action Token[Token^t] : Token

Token[Token^t IdentifierOrReservedWord] = Token[IdentifierOrReservedWord]

Token[Token^t Punctuator] = Token[Punctuator]

Token[Token^div DivisionPunctuator] = punctuator Punctuator[DivisionPunctuator]

Token[Token^t NumericLiteral] = number DoubleValue[NumericLiteral]

Token[Token^t QuantityLiteral] = quantity QuantityValue[QuantityLiteral]

Token[Token^t StringLiteral] = string StringValue[StringLiteral]

Token[Token^re RegExpLiteral^slash] = regularExpression REValue[RegExpLiteral^slash]

Token[Token^t RegExpLiteral^guillemet]
= regularExpression REValue[RegExpLiteral^guillemet]

Token[Token^t EndOfInput] = end

action RegExpMayFollow[Token^t] : Boolean

RegExpMayFollow[Token^t IdentifierOrReservedWord]
= RegExpMayFollow[IdentifierOrReservedWord]

RegExpMayFollow[Token^t Punctuator] = RegExpMayFollow[Punctuator]

RegExpMayFollow[Token^div DivisionPunctuator] = true

RegExpMayFollow[Token^t NumericLiteral] = false

RegExpMayFollow[Token^t QuantityLiteral] = false

RegExpMayFollow[Token^t StringLiteral] = false

RegExpMayFollow[Token^re RegExpLiteral^slash] = false

RegExpMayFollow[Token^t RegExpLiteral^guillemet] = false

RegExpMayFollow[Token^t EndOfInput] = true

Keywords and identifiers

Syntax

IdentifierName

InitialIdentifierCharacter

| IdentifierName ContinuingIdentifierCharacter

InitialIdentifierCharacter

OrdinaryInitialIdentifierCharacter

| HexEscape

OrdinaryInitialIdentifierCharacter UnicodeInitialAlphabetic | $ | _

ContinuingIdentifierCharacter

OrdinaryContinuingIdentifierCharacter

| HexEscape

OrdinaryContinuingIdentifierCharacter UnicodeAlphanumeric | $ | _

Semantics

action Name[IdentifierName] : String

Name[IdentifierName InitialIdentifierCharacter]
= [CharacterValue[InitialIdentifierCharacter]]

Name[IdentifierName IdentifierName₁ ContinuingIdentifierCharacter]
= Name[IdentifierName₁] [CharacterValue[ContinuingIdentifierCharacter]]

action ContainsEscapes[IdentifierName] : Boolean

ContainsEscapes[IdentifierName InitialIdentifierCharacter]
= ContainsEscapes[InitialIdentifierCharacter]

ContainsEscapes[IdentifierName IdentifierName₁ ContinuingIdentifierCharacter]
= ContainsEscapes[IdentifierName₁] or ContainsEscapes[ContinuingIdentifierCharacter]

action CharacterValue[InitialIdentifierCharacter] : Character

CharacterValue[InitialIdentifierCharacter OrdinaryInitialIdentifierCharacter]
= OrdinaryInitialIdentifierCharacter

CharacterValue[InitialIdentifierCharacter  HexEscape]
  = if isOrdinaryInitialIdentifierCharacter(CharacterValue[HexEscape])
     then CharacterValue[HexEscape]
     else

action ContainsEscapes[InitialIdentifierCharacter] : Boolean

ContainsEscapes[InitialIdentifierCharacter OrdinaryInitialIdentifierCharacter] = false

ContainsEscapes[InitialIdentifierCharacter HexEscape] = true

action CharacterValue[ContinuingIdentifierCharacter] : Character

CharacterValue[ContinuingIdentifierCharacter OrdinaryContinuingIdentifierCharacter]
= OrdinaryContinuingIdentifierCharacter

CharacterValue[ContinuingIdentifierCharacter  HexEscape]
  = if isOrdinaryContinuingIdentifierCharacter(CharacterValue[HexEscape])
     then CharacterValue[HexEscape]
     else

action ContainsEscapes[ContinuingIdentifierCharacter] : Boolean

ContainsEscapes[ContinuingIdentifierCharacter OrdinaryContinuingIdentifierCharacter]
= false

ContainsEscapes[ContinuingIdentifierCharacter HexEscape] = true

reservedWordsRE : String[]
  = [“abstract”,
      “break”,
      “case”,
      “catch”,
      “class”,
      “const”,
      “continue”,
      “debugger”,
      “default”,
      “delete”,
      “do”,
      “else”,
      “enum”,
      “eval”,
      “export”,
      “extends”,
      “field”,
      “final”,
      “finally”,
      “for”,
      “function”,
      “goto”,
      “if”,
      “implements”,
      “import”,
      “in”,
      “instanceof”,
      “native”,
      “new”,
      “package”,
      “private”,
      “protected”,
      “public”,
      “return”,
      “static”,
      “switch”,
      “synchronized”,
      “throw”,
      “throws”,
      “transient”,
      “try”,
      “typeof”,
      “var”,
      “volatile”,
      “while”,
      “with”]

reservedWordsDiv : String[] = [“false”, “null”, “super”, “this”, “true”]

nonReservedWords : String[]
= [“constructor”, “getter”, “method”, “override”, “setter”, “traditional”, “version”]

keywords : String[] = reservedWordsRE reservedWordsDiv nonReservedWords

member(id: String, list: String[]) : Boolean
  = if empty(list)
     then false
     else let s: String = first(list)
           in if id = s
               then true
               else member(id, rest(list))

Syntax

IdentifierOrReservedWord IdentifierName

Semantics

action Token[IdentifierOrReservedWord] : Token

Token[IdentifierOrReservedWord  IdentifierName]
  = let id: String = Name[IdentifierName]
     in if member(id, keywords) and (not ContainsEscapes[IdentifierName])
         then keyword id
         else identifier id

action RegExpMayFollow[IdentifierOrReservedWord] : Boolean

RegExpMayFollow[IdentifierOrReservedWord  IdentifierName]
  = let id: String = Name[IdentifierName]
     in member(id, reservedWordsRE) and (not ContainsEscapes[IdentifierName])

Punctuators

Syntax

Punctuator

PunctuatorRE

| PunctuatorDiv

PunctuatorRE

!

| ! =

| ! = =

| #

| %

| % =

| &

| & &

| & & =

| & =

| (

| *

| * =

| +

| + =

| ,

| -

| - =

| - >

| .

| . .

| . . .

| :

| : :

| ;

| <

| < <

| < < =

| < =

| =

| = =

| = = =

| >

| > =

| > >

| > > =

| > > >

| > > > =

| ?

| @

| [

| ^

| ^ =

| ^ ^

| ^ ^ =

| {

| |

| | =

| | |

| | | =

| ~

PunctuatorDiv

)

| + +

| - -

| ]

| }

DivisionPunctuator

/

| / =

Semantics

action Token[Punctuator] : Token

Token[Punctuator PunctuatorRE] = punctuator Punctuator[PunctuatorRE]

Token[Punctuator PunctuatorDiv] = punctuator Punctuator[PunctuatorDiv]

action RegExpMayFollow[Punctuator] : Boolean

RegExpMayFollow[Punctuator PunctuatorRE] = true

RegExpMayFollow[Punctuator PunctuatorDiv] = false

action Punctuator[PunctuatorRE] : String

Punctuator[PunctuatorRE !] = “!”

Punctuator[PunctuatorRE ! =] = “!=”

Punctuator[PunctuatorRE ! = =] = “!==”

Punctuator[PunctuatorRE #] = “#”

Punctuator[PunctuatorRE %] = “%”

Punctuator[PunctuatorRE % =] = “%=”

Punctuator[PunctuatorRE &] = “&”

Punctuator[PunctuatorRE & &] = “&&”

Punctuator[PunctuatorRE & & =] = “&&=”

Punctuator[PunctuatorRE & =] = “&=”

Punctuator[PunctuatorRE (] = “(”

Punctuator[PunctuatorRE *] = “*”

Punctuator[PunctuatorRE * =] = “*=”

Punctuator[PunctuatorRE +] = “+”

Punctuator[PunctuatorRE + =] = “+=”

Punctuator[PunctuatorRE ,] = “,”

Punctuator[PunctuatorRE -] = “-”

Punctuator[PunctuatorRE - =] = “-=”

Punctuator[PunctuatorRE - >] = “->”

Punctuator[PunctuatorRE .] = “.”

Punctuator[PunctuatorRE . .] = “..”

Punctuator[PunctuatorRE . . .] = “...”

Punctuator[PunctuatorRE :] = “:”

Punctuator[PunctuatorRE : :] = “::”

Punctuator[PunctuatorRE ;] = “;”

Punctuator[PunctuatorRE <] = “<”

Punctuator[PunctuatorRE < <] = “<<”

Punctuator[PunctuatorRE < < =] = “<<=”

Punctuator[PunctuatorRE < =] = “<=”

Punctuator[PunctuatorRE =] = “=”

Punctuator[PunctuatorRE = =] = “==”

Punctuator[PunctuatorRE = = =] = “===”

Punctuator[PunctuatorRE >] = “>”

Punctuator[PunctuatorRE > =] = “>=”

Punctuator[PunctuatorRE > >] = “>>”

Punctuator[PunctuatorRE > > =] = “>>=”

Punctuator[PunctuatorRE > > >] = “>>>”

Punctuator[PunctuatorRE > > > =] = “>>>=”

Punctuator[PunctuatorRE ?] = “?”

Punctuator[PunctuatorRE @] = “@”

Punctuator[PunctuatorRE [] = “[”

Punctuator[PunctuatorRE ^] = “^”

Punctuator[PunctuatorRE ^ =] = “^=”

Punctuator[PunctuatorRE ^ ^] = “^^”

Punctuator[PunctuatorRE ^ ^ =] = “^^=”

Punctuator[PunctuatorRE {] = “{”

Punctuator[PunctuatorRE |] = “|”

Punctuator[PunctuatorRE | =] = “|=”

Punctuator[PunctuatorRE | |] = “||”

Punctuator[PunctuatorRE | | =] = “||=”

Punctuator[PunctuatorRE ~] = “~”

action Punctuator[PunctuatorDiv] : String

Punctuator[PunctuatorDiv )] = “)”

Punctuator[PunctuatorDiv + +] = “++”

Punctuator[PunctuatorDiv - -] = “--”

Punctuator[PunctuatorDiv ]] = “]”

Punctuator[PunctuatorDiv }] = “}”

action Punctuator[DivisionPunctuator] : String

Punctuator[DivisionPunctuator /] = “/”

Punctuator[DivisionPunctuator / =] = “/=”

Numeric literals

Syntax

NumericLiteral

DecimalLiteral

| HexIntegerLiteral

| OctalIntegerLiteral

Semantics

action DoubleValue[NumericLiteral] : Double

DoubleValue[NumericLiteral DecimalLiteral]
= rationalToDouble(RationalValue[DecimalLiteral])

DoubleValue[NumericLiteral HexIntegerLiteral]
= rationalToDouble(IntegerValue[HexIntegerLiteral])

DoubleValue[NumericLiteral OctalIntegerLiteral]
= rationalToDouble(IntegerValue[OctalIntegerLiteral])

expt(base: Rational, exponent: Integer) : Rational
  = if exponent = 0
     then 1
     else if exponent < 0
     then 1/expt(base, -exponent)
     else base*expt(base, exponent - 1)

Syntax

DecimalLiteral Mantissa Exponent

Mantissa

DecimalIntegerLiteral

| DecimalIntegerLiteral .

| DecimalIntegerLiteral . Fraction

| . Fraction

DecimalIntegerLiteral

0

| NonZeroDecimalDigits

NonZeroDecimalDigits

NonZeroDigit

| NonZeroDecimalDigits ASCIIDigit

NonZeroDigit 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9

Fraction DecimalDigits

Semantics

action RationalValue[DecimalLiteral] : Rational

RationalValue[DecimalLiteral Mantissa Exponent]
= RationalValue[Mantissa]*expt(10, IntegerValue[Exponent])

action RationalValue[Mantissa] : Rational

RationalValue[Mantissa DecimalIntegerLiteral] = IntegerValue[DecimalIntegerLiteral]

RationalValue[Mantissa DecimalIntegerLiteral .] = IntegerValue[DecimalIntegerLiteral]

RationalValue[Mantissa DecimalIntegerLiteral . Fraction]
= IntegerValue[DecimalIntegerLiteral] + RationalValue[Fraction]

RationalValue[Mantissa . Fraction] = RationalValue[Fraction]

action IntegerValue[DecimalIntegerLiteral] : Integer

IntegerValue[DecimalIntegerLiteral 0] = 0

IntegerValue[DecimalIntegerLiteral NonZeroDecimalDigits]
= IntegerValue[NonZeroDecimalDigits]

action IntegerValue[NonZeroDecimalDigits] : Integer

IntegerValue[NonZeroDecimalDigits NonZeroDigit] = DecimalValue[NonZeroDigit]

IntegerValue[NonZeroDecimalDigits NonZeroDecimalDigits₁ ASCIIDigit]
= 10*IntegerValue[NonZeroDecimalDigits₁] + DecimalValue[ASCIIDigit]

action DigitValue[NonZeroDigit] : Integer = digitValue(NonZeroDigit)

action RationalValue[Fraction] : Rational

RationalValue[Fraction DecimalDigits]
= IntegerValue[DecimalDigits]/expt(10, NDigits[DecimalDigits])

Syntax

Exponent

«empty»

| ExponentIndicator SignedInteger

ExponentIndicator E | e

SignedInteger

DecimalDigits

| + DecimalDigits

| - DecimalDigits

Semantics

action IntegerValue[Exponent] : Integer

IntegerValue[Exponent «empty»] = 0

IntegerValue[Exponent ExponentIndicator SignedInteger] = IntegerValue[SignedInteger]

action IntegerValue[SignedInteger] : Integer

IntegerValue[SignedInteger DecimalDigits] = IntegerValue[DecimalDigits]

IntegerValue[SignedInteger + DecimalDigits] = IntegerValue[DecimalDigits]

IntegerValue[SignedInteger - DecimalDigits] = -IntegerValue[DecimalDigits]

Syntax

DecimalDigits

ASCIIDigit

| DecimalDigits ASCIIDigit

Semantics

action IntegerValue[DecimalDigits] : Integer

IntegerValue[DecimalDigits ASCIIDigit] = DecimalValue[ASCIIDigit]

IntegerValue[DecimalDigits DecimalDigits₁ ASCIIDigit]
= 10*IntegerValue[DecimalDigits₁] + DecimalValue[ASCIIDigit]

action NDigits[DecimalDigits] : Integer

NDigits[DecimalDigits ASCIIDigit] = 1

NDigits[DecimalDigits DecimalDigits₁ ASCIIDigit] = NDigits[DecimalDigits₁] + 1

Syntax

HexIntegerLiteral

0 HexIndicator HexDigit

| HexIntegerLiteral HexDigit

HexIndicator X | x

HexDigit 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | A | B | C | D | E | F | a | b | c | d | e | f

OctalIntegerLiteral

0 OctalDigit

| OctalIntegerLiteral OctalDigit

OctalDigit 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7

Semantics

action IntegerValue[HexIntegerLiteral] : Integer

IntegerValue[HexIntegerLiteral 0 HexIndicator HexDigit] = HexValue[HexDigit]

IntegerValue[HexIntegerLiteral HexIntegerLiteral₁ HexDigit]
= 16*IntegerValue[HexIntegerLiteral₁] + HexValue[HexDigit]

action DigitValue[HexDigit] : Integer = digitValue(HexDigit)

action IntegerValue[OctalIntegerLiteral] : Integer

IntegerValue[OctalIntegerLiteral 0 OctalDigit] = OctalValue[OctalDigit]

IntegerValue[OctalIntegerLiteral OctalIntegerLiteral₁ OctalDigit]
= 8*IntegerValue[OctalIntegerLiteral₁] + OctalValue[OctalDigit]

action DigitValue[OctalDigit] : Integer = digitValue(OctalDigit)

Quantity literals

Syntax

QuantityLiteral NumericLiteral _ IdentifierName

Semantics

action QuantityValue[QuantityLiteral] : Quantity

QuantityValue[QuantityLiteral NumericLiteral _ IdentifierName]
= amount DoubleValue[NumericLiteral], unit Name[IdentifierName]

String literals

Syntax

q {single, double}

StringLiteral

' StringChars^single '

| " StringChars^double "

Semantics

action StringValue[StringLiteral] : String

StringValue[StringLiteral ' StringChars^single '] = StringValue[StringChars^single]

StringValue[StringLiteral " StringChars^double "] = StringValue[StringChars^double]

Syntax

StringChars^q

OrdinaryStringChars^q

| StringChars^q ShortOctalEscape

OrdinaryStringChars^q

«empty»

| StringChars^q PlainStringChar^q

| OrdinaryStringChars^q OctalDigit

| StringChars^q OrdinaryEscape

PlainStringChar^single UnicodeCharacter except ' | \ | OctalDigit | LineTerminator

PlainStringChar^double UnicodeCharacter except " | \ | OctalDigit | LineTerminator

Semantics

action StringValue[StringChars^q] : String

StringValue[StringChars^q OrdinaryStringChars^q] = StringValue[OrdinaryStringChars^q]

StringValue[StringChars^q StringChars^q₁ ShortOctalEscape]
= StringValue[StringChars^q₁] [CharacterValue[ShortOctalEscape]]

action StringValue[OrdinaryStringChars^q] : String

StringValue[OrdinaryStringChars^q «empty»] = “”

StringValue[OrdinaryStringChars^q StringChars^q PlainStringChar^q]
= StringValue[StringChars^q] [PlainStringChar^q]

StringValue[OrdinaryStringChars^q OrdinaryStringChars^q₁ OctalDigit]
= StringValue[OrdinaryStringChars^q₁] [OctalDigit]

StringValue[OrdinaryStringChars^q StringChars^q OrdinaryEscape]
= StringValue[StringChars^q] [CharacterValue[OrdinaryEscape]]

Syntax

OrdinaryEscape

ControlEscape

| FullOctalEscape

| HexEscape

| \ IdentityEscape

IdentityEscape NonTerminator except UnicodeAlphanumeric

Semantics

action CharacterValue[OrdinaryEscape] : Character

CharacterValue[OrdinaryEscape ControlEscape] = CharacterValue[ControlEscape]

CharacterValue[OrdinaryEscape FullOctalEscape] = CharacterValue[FullOctalEscape]

CharacterValue[OrdinaryEscape HexEscape] = CharacterValue[HexEscape]

CharacterValue[OrdinaryEscape \ IdentityEscape] = IdentityEscape

Syntax

ControlEscape

\ b

| \ f

| \ n

| \ r

| \ t

| \ v

Semantics

action CharacterValue[ControlEscape] : Character

CharacterValue[ControlEscape \ b] = ‘«BS»’

CharacterValue[ControlEscape \ f] = ‘«FF»’

CharacterValue[ControlEscape \ n] = ‘«LF»’

CharacterValue[ControlEscape \ r] = ‘«CR»’

CharacterValue[ControlEscape \ t] = ‘«TAB»’

CharacterValue[ControlEscape \ v] = ‘«VT»’

Syntax

ShortOctalEscape

\ OctalDigit

| \ ZeroToThree OctalDigit

FullOctalEscape

\ FourToSeven OctalDigit

| \ ZeroToThree OctalDigit OctalDigit

ZeroToThree 0 | 1 | 2 | 3

FourToSeven 4 | 5 | 6 | 7

Semantics

action CharacterValue[ShortOctalEscape] : Character

CharacterValue[ShortOctalEscape \ OctalDigit] = codeToCharacter(OctalValue[OctalDigit])

CharacterValue[ShortOctalEscape \ ZeroToThree OctalDigit]
= codeToCharacter(8*OctalValue[ZeroToThree] + OctalValue[OctalDigit])

action CharacterValue[FullOctalEscape] : Character

CharacterValue[FullOctalEscape \ FourToSeven OctalDigit]
= codeToCharacter(8*OctalValue[FourToSeven] + OctalValue[OctalDigit])

CharacterValue[FullOctalEscape  \ ZeroToThree OctalDigit₁ OctalDigit₂]
  = codeToCharacter(
         64*OctalValue[ZeroToThree] + 8*OctalValue[OctalDigit₁] + OctalValue[OctalDigit₂])

action DigitValue[ZeroToThree] : Integer = digitValue(ZeroToThree)

action DigitValue[FourToSeven] : Integer = digitValue(FourToSeven)

Syntax

HexEscape

\ x HexDigit HexDigit

| \ u HexDigit HexDigit HexDigit HexDigit

Semantics

action CharacterValue[HexEscape] : Character

CharacterValue[HexEscape \ x HexDigit₁ HexDigit₂]
= codeToCharacter(16*HexValue[HexDigit₁] + HexValue[HexDigit₂])

CharacterValue[HexEscape  \ u HexDigit₁ HexDigit₂ HexDigit₃ HexDigit₄]
  = codeToCharacter(
         4096*HexValue[HexDigit₁] + 256*HexValue[HexDigit₂] + 16*HexValue[HexDigit₃] +
         HexValue[HexDigit₄])