Lines Matching refs:untokenize
3 from tokenize import (tokenize, _tokenize, untokenize, NUMBER, NAME, OP,
971 return untokenize(result).decode('utf-8')
1507 self.assertEqual(u.untokenize(iter([token])), 'Hello ')
1509 self.assertEqual(u.untokenize(iter(tokens)), 'Hello ')
1511 self.assertEqual(untokenize(iter(tokens)), b'Hello ')
1518 Test roundtrip for `untokenize`. `f` is an open file or a string.
1521 tokenize.untokenize(), and the latter tokenized again to 2-tuples.
1524 When untokenize bugs are fixed, untokenize with 5-tuples should
1538 bytes_from2 = untokenize(tokens2)
1543 bytes_from5 = untokenize(tokens5)
1557 # untokenize a little trickier. Note that this test involves trailing
1624 # With roundtrip extended to test the 5-tuple mode of untokenize,
1645 return untokenize(tokenize(BytesIO(code).readline)).decode('utf-8')