ch37.md 6.3 KB
Chapter 37. strscans
The tokenize iterator, as it was implemented in the previous chapter, is straight-forward but imperative code, and the task of parsing comes up frequently in day to day programming. Ideally we want to program in a declarative way; only describing how to extract the desired data via patterns and not how to advance any required auxiliary cursors, for example. The standard library offers the relatively unknown module strscans that helps with this task. strscans.scanTuple can be used to extract data into a custom tuple type. The tuple type depends on the pattern that is tried to match. For example: import std / strscans const InputData = "1000-01-01 00:00:00" 1 let (ok, year, month, day, time) = scanTuple(InputData, "$i-$i-$i$s$+") 2 if ok:
assert year == 1000 3
assert month == 1 4
assert day == 1 5
assert time == "00:00:00" 6
1 InputData contains a date a clock time that we seek to parse. 2 The string "$i-$i-$i$s$"` is a description of how to extract the data.
Characters are matched verbatim except for substrings starting with
`$`. `$i` means to expect a string substring that can be parsed into
an `int`. `$s` means to skip optional whitespace. `$ matches the rest
of the input.
3 year is inferred to be of type int because it corresponds to the first $i
pattern. For InputData its value is 1000.
243
4 month is inferred to be of type int because it corresponds to the second $i
pattern. For InputData its value is 1.
5 day is inferred to be of type int because it corresponds to the third $i
pattern. For InputData its value is 1.
6 time is inferred to be of type string because it corresponds to the $+
pattern. For InputData its value is "00:00:00".
scanTuple needs to produce a tuple of variable length depending on the pattern that we pass to it. The first component of the tuple is always of type bool and contains the information if the parse was successful. In order to simplify the task that scanTuple has to do, we first create a couple of helper routines operating on a parsing State: import std / parseutils type
State = object 1
i: int
err: bool
proc matchChar(s: string; c: var State; ch: char) = 2
if not c.err:
if c.i < s.len and s[c.i] == ch:
inc c.i
else:
c.err = true
proc skipWhitespace(s: string; c: var State) = 3
if not c.err:
while c.i < s.len and s[c.i] in {' ', '\t', '\n', '\r'}: inc c.i
proc matchInt(s: string; res: var int; c: var State) = 4
if not c.err:
let span = parseInt(s, res, c.i)
if span > 0:
inc c.i, span
else:
c.err = true
proc matchRest(s: string; res: var string; c: var State) = 5
if not c.err:
res = s.substr(c.i)
1 The parsing State consists of the current parsing position i and an error
flag called err. Once err is true, it is never reset to false.
244 2 matchChar tries to match a single character ch. 3 skipWhitespace skips optional whitespace. 4 matchInt tries to match the input at position c.i against an integer. It does
so with the help of the standard library’s parseutils.parseInt function.
5 matchRest matches the rest of the input string and stores it into res. This design with an explicit error state allows us to emit sequential code rather than (potentially deeply) nested if statements: import std / macros macro scanTuple*(input: string; pattern: static[string]): untyped = 1 var i = 0 var body = newTree(nnkStmtList) 2 var tup = newTree(nnkTupleConstr) 3 tup.add newLit(true) let stateVar = genSym(nskVar, "stateVar") let res = genSym(nskVar, "scanResult") while i < pattern.len:
if pattern[i] == '$':
inc i
case pattern[i]
of 'i': 4
body.add newCall(bindSym"matchInt", input,
newTree(nnkBracketExpr, res, newLit(tup.len)), stateVar)
tup.add newLit(0)
of 's': 5
body.add newCall(bindSym"skipWhitespace", input, stateVar)
of '+': 6
body.add newCall(bindSym"matchRest", input,
newTree(nnkBracketExpr, res, newLit(tup.len)), stateVar)
tup.add newLit("")
else:
error "invalid pattern"
inc i
else: 7
body.add newCall(bindSym"matchChar", input,
stateVar, newLit(pattern[i]))
inc i
result = newTree(nnkStmtListExpr, 8
newVarStmt(res, tup),
newVarStmt(stateVar, newTree(nnkObjConstr, bindSym"State")),
body,
newAssignment(newTree(nnkBracketExpr, res, newLit(0)),
newCall(bindSym"not", newDotExpr(stateVar, ident"err"))),
res)
245
when defined(debugScanTuple):
echo repr result 9
1 Because the exact return tuple type depends on pattern, only untyped can
be used as the return type.
2 body collects the list of statements that contains the calls to the helpers
matchChar, skipWhitespace, matchInt, and matchRest.
3 tup collects the resulting tuple value (not the tuple type!). 4 We map the pattern $i to a call to matchInt. 5 We map the pattern $s to a call to skipWhitespace. 6 We map the pattern $+ to a call to matchRest. 7 Every other character in pattern is mapped to matchChar. 8 The result of scanTuple is a statement list expression roughly like (var
scanResult = (false, ...); var stateVar = State(); body; scanResult[0]
= not stateVar.err; scanResult).
9 The when ... section allows us to inspect the produced code easily. If we compile the program with the switch --define:debugScanTuple it enables the line echo repr result so at compile-time the produced AST is written to standard output: var scanResult_123 = (true, 0, 0, 0, "") var stateVar_456 = State() matchInt(InputData, scanResult_123[1], stateVar_456) matchChar(InputData, stateVar_456, '-') matchInt(InputData, scanResult_123[2], stateVar_456) matchChar(InputData, stateVar_456, '-') matchInt(InputData, scanResult_123[3], stateVar_456) skipWhitespace(InputData, stateVar_456) matchRest(InputData, scanResult_123[4], stateVar_456) scanResult_123[0] = not(stateVar_456.err) scanResult_123 echo repr result is an idiom worth remembering; it is important for debugging macro code and also allows for an easier development process.