Tokenizer for python ast
tokenize_ast(script_path, path_prefix = ".")
| script_path | absolute paths from scripts to tokenize. |
|---|---|
| path_prefix | leading folders from absolute path to omit from names. |
a named list, compatible with quanteda::tokenize()