Lines Matching refs:tokens
4 Python tokens. It decodes the bytes according to PEP-0263 for
18 that it produces COMMENT tokens for comments and gives type OP for all
169 self.tokens = []
181 self.tokens.append("\\\n" * row_offset)
185 self.tokens.append(" " * col_offset)
213 self.tokens.append(indent)
217 self.tokens.append(token)
222 return "".join(self.tokens)
226 toks_append = self.tokens.append
262 """Transform tokens back into Python source code.
268 only two tokens are passed, the resulting output is poor.
530 if pseudomatch: # scan for tokens
656 tokens = list(tokenize(f.readline))
659 tokens = _tokenize(sys.stdin.readline, None)
662 for token in tokens: