Lines Matching refs:tokens
7 text into Python tokens. It accepts a readline-like method which is called
18 that it produces COMMENT tokens for comments and gives type OP for all
24 are the same, except instead of generating tokens, tokeneater is a callback
174 self.tokens = []
183 self.tokens.append(" " * col_offset)
192 self.tokens.append(token)
197 return "".join(self.tokens)
202 toks_append = self.tokens.append
315 """Transform tokens back into Python source code.
319 only two tokens are passed, the resulting output is poor.
448 if pseudomatch: # scan for tokens