Lines Matching refs:tokenize
23 tokenize(readline, tokeneater=printtoken)
37 __all__ = [x for x in dir(token) if x[0] != '_'] + ["tokenize",
148 def tokenize(readline, tokeneater=printtoken):
150 The tokenize() function accepts two parameters: one representing the
151 input stream, and one providing an output mechanism for tokenize().
245 in the same way as the tokenize() generator.
325 # Output text will tokenize the back to the input
426 ("<tokenize>", lnum, pos, line))
563 if len(sys.argv) > 1: tokenize(open(sys.argv[1]).readline)
564 else: tokenize(sys.stdin.readline)