llvm-journey

commit 8ddcb6f0b36378a848efe265869639d7a69b7eb5
parent 90b9c7410e85cb7c6f0376a1bd762419dadb039e
Author: Mohammad-Reza Nabipoor <m.nabipoor@yahoo.com>
Date:   Fri,  7 Aug 2020 03:22:47 +0430

README.md: Add documentation for kal::lex

Diffstat:
M
README.md
 | 
26
++++++++++++++++++++++++++

1 file changed, 26 insertions(+), 0 deletions(-)
diff --git a/README.md b/README.md
@@ -12,3 +12,29 @@ My journey through
 The lexer has been implemented as a STL-like algorithm. It accepts a pair of
 `ForwardIterator`s and a function object; for each token, the function object
 will be called as follow: `funcobj(kal::TkToken, FwdIt tkBegin, FwdIt tkEnd)`.
+
+### `kal::lex` function template
+
+Tokenize the input sequence.
+
+```c++
+template<typename FwdIt, typename TokenOp>
+FwdIt
+lex(FwdIt f, FwdIt l, TokenOp op);
+```
+
+The function template tokenize the range `[f, l)`, and will call `op` with
+the following arguments:
+- Type of token (`kal::TkToken`)
+- Begining of the token (`FwdIt`)
+- End of the token (`FwdIt`)
+
+#### Parameters
+
+- `f` Beginning of the source code sequence
+- `l` End of the source code sequence
+- `op` Function object that will be called for each recognized token
+
+#### Return value
+
+Returns the end of input range (`l`).