diff options
| author | Vladimir Azarov <avm@intermediate-node.net> | 2025-03-24 21:51:16 +0100 |
|---|---|---|
| committer | Vladimir Azarov <avm@intermediate-node.net> | 2025-03-24 21:51:16 +0100 |
| commit | 87217fe5ba58f5199d30586b5d9bec104dece445 (patch) | |
| tree | 00a43d11ebdfbb65750e80758ce1925e4c6a1a3e /tokenizer.sig | |
| parent | e40727b58e357f123256557af50666aa42c2caa4 (diff) | |
Partition into structures
Diffstat (limited to 'tokenizer.sig')
| -rw-r--r-- | tokenizer.sig | 12 |
1 files changed, 12 insertions, 0 deletions
diff --git a/tokenizer.sig b/tokenizer.sig new file mode 100644 index 0000000..c31a7e6 --- /dev/null +++ b/tokenizer.sig @@ -0,0 +1,12 @@ +signature TOKENIZER = sig + type token + type fullToken = Stream.pos * token + + (* Fatal. both may be thrown by tokenize *) + exception FsmTableIsTooSmall + exception TkErrorAug of Stream.convPos * string + + val tokenize: Stream.t -> fullToken list -> fullToken list * Stream.filesInfo + + val printToken: token -> unit +end |