nimcity/doc/part03/ch36.md

Chapter 36. strformat

Macros can be used to translate mini languages embedded inside string literals into Nim code. A good example for this is the standard library’s strformat module. Instead of a & " " & $b & " " & c you can write fmt"{a} {b} {c}". Inside the string literal, curly braces enclose a Nim expression. The expression is turned into a string via $. The standard library’s fmt supports many features for formatting strings, integers and floats, their precision and alignment. However, in our reimplementation we keep things simple: We only support curly braces and use macros.parseExpr to do the hard part of parsing the Nim subexpressions into Nim’s AST. It is good style to split up the tasks “parsing” and “synthesis” into different routines. Only the synthesis uses Nim’s AST API. The parser/tokenizer is implemented as an iterator: type TokenKind = enum 1

Literal 2
NimExpr 3

iterator tokenize(s: string): (TokenKind, string) = 4

var i = 0
var tok = Literal 5
while i < s.len:
   let start = i
   case tok
   of Literal:
     while i < s.len and s[i] != '{': inc i 6
   of NimExpr:
     while i < s.len and s[i] != '}': inc i 7
   yield (tok, s.substr(start, i-1))
   tok = if tok == Literal: NimExpr else: Literal 8
   inc i
                                                                        239

1 The tokenizer distinguishes only between two kinds of tokens. 2 A Literal token means it should be interpreted literally. For example, the "abc" part from "{x}abc". 3 A NimExpr needs to be parsed as a Nim expression. For example, the "x" part from "{x}abc". 4 tokenize yields the determined tokens. A token is a pair of (TokenKind, string). 5 The tokenizer starts in the state Literal. 6 If the tokenizer is in the state Literal it needs to proceed until either the end of the string is reached or until a '{' is found. 7 If the tokenizer is in the state NimExpr it needs to proceed until either the end of the string is reached or until a '}' is found. 8 After a Literal token a NimExpr token must follow and vice versa. The fmt macro uses this tokenize iterator: import macros macro fmt*(pattern: static[string]): string = 1 var args = newTree(nnkBracket) 2 for (k, s) in tokenize(pattern): 3

 case k
 of Literal:
   if s != "":
     args.add newLit(s) 4
 of NimExpr:
   args.add newCall(bindSym"$", parseExpr(s)) 5

if args.len == 0: 6

 result = newLit("")

else:

 result = nestList(bindSym"&", args) 7

var x = 0.9 var y = "abc" echo fmt"{x} {y}" 8 1 fmt takes a static[string] as input. This means that inside the macro body pattern really is of type string and not of NimNode making the data easier to access. 2 args collects all the arguments that we pass to the & operator. 240 3 We use the tokenize iterator and unpack the token tuple into k and s. 4 If the token is a Literal and not the empty string, we can append s to args. But we need to convert s to a NimNode first via newLit(s). 5 If the token is a NimExpr we use macros.parseExpr to parse it into a NimNode. We then wrap the node and use it as an argument to a call of the $ operator. Thus fmt supports any expression that can be turned into a string via $. 6 For a call like fmt"" it is possible that args remains empty. We map this case to the empty string literal "". 7 Else we call the concatenation operator & with args. However & only accepts two arguments so we need to turn &[a, b, c] to (a & b) & c. This nesting of arguments is performed by macros.nestList. 8 Produces the output: "0.9 abc".