OK Script (OysterKit script) is a light weight string format to specify tokenisers (or single states). It can be defined using itself.
{ <'"',{({*!"\"\\"->character, "\\"."trn\"\\"->character} )->Char} >->quote, <'\'',{({*!"'\\"->character, "\\"."'\\"->character} )->delimiter} >->single-quote, "!"->not, "-".">"->token, "*"->loop, "."->then, "{"->start-branch, "}"->end-branch, "("->start-repeat, ")"->end-repeat, "<"->start-delimited, ">"->end-delimited, ","->comma, { *"0123456789"->integer, "+-".*"0123456789"->integer }, "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ".*"abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_-"->variable, *" \t\r\n"->whitespace, !"\x04" }
OK Script is generated by all TokenizationStates when they generated their string representation. For example, the above definition is generated from the original Swift definition of the language
self.branch( Delimited(delimiter: "\"", states: Repeat(state:Branch().branch( LoopingChar(except: "\"\\").token("character"), Char(from:"\\").branch( Char(from:"trn\"\\").token("character") ) ), min: 1, max: nil).token("Char") ).token("quote"), Delimited(delimiter: "'", states: Repeat(state:Branch().branch( LoopingChar(except: "'\\").token("character"), Char(from:"\\").branch( Char(from:"'\\").token("character") ) ), min: 1).token("delimiter") ).token("single-quote"), Char(from: "!").token("not"), Char(from: "-").sequence( Char(from:">").token("token") ), Char(from:"*").token("loop"), Char(from:".").token("then"), Char(from:"{").token("start-branch"), Char(from:"}").token("end-branch"), Char(from:"(").token("start-repeat"), Char(from:")").token("end-repeat"), Char(from:"<").token("start-delimited"), Char(from:">").token("end-delimited"), Char(from:",").token("comma"), OysterKit.number, OysterKit.Code.variableName, OysterKit.whiteSpaces, Char(except: "\x04") )
In order to create a tokenizer, or a single state, from a String containing OK script simply call OysterKit.parseState and parseTokenizer.
OysterKit.parseState("\"0123456789\"->digit")
Would create a state that generates digit tokens.
Swift is very young... does its performance have a significant impact on your code's performance? Do we need to wait until they tune the compiler? SPOILER ALERT: No, it's your fault dummy.