1:mod:`codecs` --- Codec registry and base classes 2================================================= 3 4.. module:: codecs 5 :synopsis: Encode and decode data and streams. 6 7.. moduleauthor:: Marc-André Lemburg <mal@lemburg.com> 8.. sectionauthor:: Marc-André Lemburg <mal@lemburg.com> 9.. sectionauthor:: Martin v. Löwis <martin@v.loewis.de> 10 11**Source code:** :source:`Lib/codecs.py` 12 13.. index:: 14 single: Unicode 15 single: Codecs 16 pair: Codecs; encode 17 pair: Codecs; decode 18 single: streams 19 pair: stackable; streams 20 21-------------- 22 23This module defines base classes for standard Python codecs (encoders and 24decoders) and provides access to the internal Python codec registry, which 25manages the codec and error handling lookup process. Most standard codecs 26are :term:`text encodings <text encoding>`, which encode text to bytes (and 27decode bytes to text), but there are also codecs provided that encode text to 28text, and bytes to bytes. Custom codecs may encode and decode between arbitrary 29types, but some module features are restricted to be used specifically with 30:term:`text encodings <text encoding>` or with codecs that encode to 31:class:`bytes`. 32 33The module defines the following functions for encoding and decoding with 34any codec: 35 36.. function:: encode(obj, encoding='utf-8', errors='strict') 37 38 Encodes *obj* using the codec registered for *encoding*. 39 40 *Errors* may be given to set the desired error handling scheme. The 41 default error handler is ``'strict'`` meaning that encoding errors raise 42 :exc:`ValueError` (or a more codec specific subclass, such as 43 :exc:`UnicodeEncodeError`). Refer to :ref:`codec-base-classes` for more 44 information on codec error handling. 45 46.. function:: decode(obj, encoding='utf-8', errors='strict') 47 48 Decodes *obj* using the codec registered for *encoding*. 49 50 *Errors* may be given to set the desired error handling scheme. The 51 default error handler is ``'strict'`` meaning that decoding errors raise 52 :exc:`ValueError` (or a more codec specific subclass, such as 53 :exc:`UnicodeDecodeError`). Refer to :ref:`codec-base-classes` for more 54 information on codec error handling. 55 56The full details for each codec can also be looked up directly: 57 58.. function:: lookup(encoding) 59 60 Looks up the codec info in the Python codec registry and returns a 61 :class:`CodecInfo` object as defined below. 62 63 Encodings are first looked up in the registry's cache. If not found, the list of 64 registered search functions is scanned. If no :class:`CodecInfo` object is 65 found, a :exc:`LookupError` is raised. Otherwise, the :class:`CodecInfo` object 66 is stored in the cache and returned to the caller. 67 68.. class:: CodecInfo(encode, decode, streamreader=None, streamwriter=None, incrementalencoder=None, incrementaldecoder=None, name=None) 69 70 Codec details when looking up the codec registry. The constructor 71 arguments are stored in attributes of the same name: 72 73 74 .. attribute:: name 75 76 The name of the encoding. 77 78 79 .. attribute:: encode 80 decode 81 82 The stateless encoding and decoding functions. These must be 83 functions or methods which have the same interface as 84 the :meth:`~Codec.encode` and :meth:`~Codec.decode` methods of Codec 85 instances (see :ref:`Codec Interface <codec-objects>`). 86 The functions or methods are expected to work in a stateless mode. 87 88 89 .. attribute:: incrementalencoder 90 incrementaldecoder 91 92 Incremental encoder and decoder classes or factory functions. 93 These have to provide the interface defined by the base classes 94 :class:`IncrementalEncoder` and :class:`IncrementalDecoder`, 95 respectively. Incremental codecs can maintain state. 96 97 98 .. attribute:: streamwriter 99 streamreader 100 101 Stream writer and reader classes or factory functions. These have to 102 provide the interface defined by the base classes 103 :class:`StreamWriter` and :class:`StreamReader`, respectively. 104 Stream codecs can maintain state. 105 106To simplify access to the various codec components, the module provides 107these additional functions which use :func:`lookup` for the codec lookup: 108 109.. function:: getencoder(encoding) 110 111 Look up the codec for the given encoding and return its encoder function. 112 113 Raises a :exc:`LookupError` in case the encoding cannot be found. 114 115 116.. function:: getdecoder(encoding) 117 118 Look up the codec for the given encoding and return its decoder function. 119 120 Raises a :exc:`LookupError` in case the encoding cannot be found. 121 122 123.. function:: getincrementalencoder(encoding) 124 125 Look up the codec for the given encoding and return its incremental encoder 126 class or factory function. 127 128 Raises a :exc:`LookupError` in case the encoding cannot be found or the codec 129 doesn't support an incremental encoder. 130 131 132.. function:: getincrementaldecoder(encoding) 133 134 Look up the codec for the given encoding and return its incremental decoder 135 class or factory function. 136 137 Raises a :exc:`LookupError` in case the encoding cannot be found or the codec 138 doesn't support an incremental decoder. 139 140 141.. function:: getreader(encoding) 142 143 Look up the codec for the given encoding and return its :class:`StreamReader` 144 class or factory function. 145 146 Raises a :exc:`LookupError` in case the encoding cannot be found. 147 148 149.. function:: getwriter(encoding) 150 151 Look up the codec for the given encoding and return its :class:`StreamWriter` 152 class or factory function. 153 154 Raises a :exc:`LookupError` in case the encoding cannot be found. 155 156Custom codecs are made available by registering a suitable codec search 157function: 158 159.. function:: register(search_function) 160 161 Register a codec search function. Search functions are expected to take one 162 argument, being the encoding name in all lower case letters with hyphens 163 and spaces converted to underscores, and return a :class:`CodecInfo` object. 164 In case a search function cannot find a given encoding, it should return 165 ``None``. 166 167 .. versionchanged:: 3.9 168 Hyphens and spaces are converted to underscore. 169 170 171.. function:: unregister(search_function) 172 173 Unregister a codec search function and clear the registry's cache. 174 If the search function is not registered, do nothing. 175 176 .. versionadded:: 3.10 177 178 179While the builtin :func:`open` and the associated :mod:`io` module are the 180recommended approach for working with encoded text files, this module 181provides additional utility functions and classes that allow the use of a 182wider range of codecs when working with binary files: 183 184.. function:: open(filename, mode='r', encoding=None, errors='strict', buffering=-1) 185 186 Open an encoded file using the given *mode* and return an instance of 187 :class:`StreamReaderWriter`, providing transparent encoding/decoding. 188 The default file mode is ``'r'``, meaning to open the file in read mode. 189 190 .. note:: 191 192 If *encoding* is not ``None``, then the 193 underlying encoded files are always opened in binary mode. 194 No automatic conversion of ``'\n'`` is done on reading and writing. 195 The *mode* argument may be any binary mode acceptable to the built-in 196 :func:`open` function; the ``'b'`` is automatically added. 197 198 *encoding* specifies the encoding which is to be used for the file. 199 Any encoding that encodes to and decodes from bytes is allowed, and 200 the data types supported by the file methods depend on the codec used. 201 202 *errors* may be given to define the error handling. It defaults to ``'strict'`` 203 which causes a :exc:`ValueError` to be raised in case an encoding error occurs. 204 205 *buffering* has the same meaning as for the built-in :func:`open` function. 206 It defaults to -1 which means that the default buffer size will be used. 207 208 .. versionchanged:: 3.11 209 The ``'U'`` mode has been removed. 210 211 212.. function:: EncodedFile(file, data_encoding, file_encoding=None, errors='strict') 213 214 Return a :class:`StreamRecoder` instance, a wrapped version of *file* 215 which provides transparent transcoding. The original file is closed 216 when the wrapped version is closed. 217 218 Data written to the wrapped file is decoded according to the given 219 *data_encoding* and then written to the original file as bytes using 220 *file_encoding*. Bytes read from the original file are decoded 221 according to *file_encoding*, and the result is encoded 222 using *data_encoding*. 223 224 If *file_encoding* is not given, it defaults to *data_encoding*. 225 226 *errors* may be given to define the error handling. It defaults to 227 ``'strict'``, which causes :exc:`ValueError` to be raised in case an encoding 228 error occurs. 229 230 231.. function:: iterencode(iterator, encoding, errors='strict', **kwargs) 232 233 Uses an incremental encoder to iteratively encode the input provided by 234 *iterator*. This function is a :term:`generator`. 235 The *errors* argument (as well as any 236 other keyword argument) is passed through to the incremental encoder. 237 238 This function requires that the codec accept text :class:`str` objects 239 to encode. Therefore it does not support bytes-to-bytes encoders such as 240 ``base64_codec``. 241 242 243.. function:: iterdecode(iterator, encoding, errors='strict', **kwargs) 244 245 Uses an incremental decoder to iteratively decode the input provided by 246 *iterator*. This function is a :term:`generator`. 247 The *errors* argument (as well as any 248 other keyword argument) is passed through to the incremental decoder. 249 250 This function requires that the codec accept :class:`bytes` objects 251 to decode. Therefore it does not support text-to-text encoders such as 252 ``rot_13``, although ``rot_13`` may be used equivalently with 253 :func:`iterencode`. 254 255 256The module also provides the following constants which are useful for reading 257and writing to platform dependent files: 258 259 260.. data:: BOM 261 BOM_BE 262 BOM_LE 263 BOM_UTF8 264 BOM_UTF16 265 BOM_UTF16_BE 266 BOM_UTF16_LE 267 BOM_UTF32 268 BOM_UTF32_BE 269 BOM_UTF32_LE 270 271 These constants define various byte sequences, 272 being Unicode byte order marks (BOMs) for several encodings. They are 273 used in UTF-16 and UTF-32 data streams to indicate the byte order used, 274 and in UTF-8 as a Unicode signature. :const:`BOM_UTF16` is either 275 :const:`BOM_UTF16_BE` or :const:`BOM_UTF16_LE` depending on the platform's 276 native byte order, :const:`BOM` is an alias for :const:`BOM_UTF16`, 277 :const:`BOM_LE` for :const:`BOM_UTF16_LE` and :const:`BOM_BE` for 278 :const:`BOM_UTF16_BE`. The others represent the BOM in UTF-8 and UTF-32 279 encodings. 280 281 282.. _codec-base-classes: 283 284Codec Base Classes 285------------------ 286 287The :mod:`codecs` module defines a set of base classes which define the 288interfaces for working with codec objects, and can also be used as the basis 289for custom codec implementations. 290 291Each codec has to define four interfaces to make it usable as codec in Python: 292stateless encoder, stateless decoder, stream reader and stream writer. The 293stream reader and writers typically reuse the stateless encoder/decoder to 294implement the file protocols. Codec authors also need to define how the 295codec will handle encoding and decoding errors. 296 297 298.. _surrogateescape: 299.. _error-handlers: 300 301Error Handlers 302^^^^^^^^^^^^^^ 303 304To simplify and standardize error handling, codecs may implement different 305error handling schemes by accepting the *errors* string argument: 306 307 >>> 'German ß, ♬'.encode(encoding='ascii', errors='backslashreplace') 308 b'German \\xdf, \\u266c' 309 >>> 'German ß, ♬'.encode(encoding='ascii', errors='xmlcharrefreplace') 310 b'German ß, ♬' 311 312.. index:: 313 pair: strict; error handler's name 314 pair: ignore; error handler's name 315 pair: replace; error handler's name 316 pair: backslashreplace; error handler's name 317 pair: surrogateescape; error handler's name 318 single: ? (question mark); replacement character 319 single: \ (backslash); escape sequence 320 single: \x; escape sequence 321 single: \u; escape sequence 322 single: \U; escape sequence 323 324The following error handlers can be used with all Python 325:ref:`standard-encodings` codecs: 326 327.. tabularcolumns:: |l|L| 328 329+-------------------------+-----------------------------------------------+ 330| Value | Meaning | 331+=========================+===============================================+ 332| ``'strict'`` | Raise :exc:`UnicodeError` (or a subclass), | 333| | this is the default. Implemented in | 334| | :func:`strict_errors`. | 335+-------------------------+-----------------------------------------------+ 336| ``'ignore'`` | Ignore the malformed data and continue without| 337| | further notice. Implemented in | 338| | :func:`ignore_errors`. | 339+-------------------------+-----------------------------------------------+ 340| ``'replace'`` | Replace with a replacement marker. On | 341| | encoding, use ``?`` (ASCII character). On | 342| | decoding, use ``�`` (U+FFFD, the official | 343| | REPLACEMENT CHARACTER). Implemented in | 344| | :func:`replace_errors`. | 345+-------------------------+-----------------------------------------------+ 346| ``'backslashreplace'`` | Replace with backslashed escape sequences. | 347| | On encoding, use hexadecimal form of Unicode | 348| | code point with formats ``\xhh`` ``\uxxxx`` | 349| | ``\Uxxxxxxxx``. On decoding, use hexadecimal | 350| | form of byte value with format ``\xhh``. | 351| | Implemented in | 352| | :func:`backslashreplace_errors`. | 353+-------------------------+-----------------------------------------------+ 354| ``'surrogateescape'`` | On decoding, replace byte with individual | 355| | surrogate code ranging from ``U+DC80`` to | 356| | ``U+DCFF``. This code will then be turned | 357| | back into the same byte when the | 358| | ``'surrogateescape'`` error handler is used | 359| | when encoding the data. (See :pep:`383` for | 360| | more.) | 361+-------------------------+-----------------------------------------------+ 362 363.. index:: 364 pair: xmlcharrefreplace; error handler's name 365 pair: namereplace; error handler's name 366 single: \N; escape sequence 367 368The following error handlers are only applicable to encoding (within 369:term:`text encodings <text encoding>`): 370 371+-------------------------+-----------------------------------------------+ 372| Value | Meaning | 373+=========================+===============================================+ 374| ``'xmlcharrefreplace'`` | Replace with XML/HTML numeric character | 375| | reference, which is a decimal form of Unicode | 376| | code point with format ``&#num;`` Implemented | 377| | in :func:`xmlcharrefreplace_errors`. | 378+-------------------------+-----------------------------------------------+ 379| ``'namereplace'`` | Replace with ``\N{...}`` escape sequences, | 380| | what appears in the braces is the Name | 381| | property from Unicode Character Database. | 382| | Implemented in :func:`namereplace_errors`. | 383+-------------------------+-----------------------------------------------+ 384 385.. index:: 386 pair: surrogatepass; error handler's name 387 388In addition, the following error handler is specific to the given codecs: 389 390+-------------------+------------------------+-------------------------------------------+ 391| Value | Codecs | Meaning | 392+===================+========================+===========================================+ 393|``'surrogatepass'``| utf-8, utf-16, utf-32, | Allow encoding and decoding surrogate code| 394| | utf-16-be, utf-16-le, | point (``U+D800`` - ``U+DFFF``) as normal | 395| | utf-32-be, utf-32-le | code point. Otherwise these codecs treat | 396| | | the presence of surrogate code point in | 397| | | :class:`str` as an error. | 398+-------------------+------------------------+-------------------------------------------+ 399 400.. versionadded:: 3.1 401 The ``'surrogateescape'`` and ``'surrogatepass'`` error handlers. 402 403.. versionchanged:: 3.4 404 The ``'surrogatepass'`` error handler now works with utf-16\* and utf-32\* 405 codecs. 406 407.. versionadded:: 3.5 408 The ``'namereplace'`` error handler. 409 410.. versionchanged:: 3.5 411 The ``'backslashreplace'`` error handler now works with decoding and 412 translating. 413 414The set of allowed values can be extended by registering a new named error 415handler: 416 417.. function:: register_error(name, error_handler) 418 419 Register the error handling function *error_handler* under the name *name*. 420 The *error_handler* argument will be called during encoding and decoding 421 in case of an error, when *name* is specified as the errors parameter. 422 423 For encoding, *error_handler* will be called with a :exc:`UnicodeEncodeError` 424 instance, which contains information about the location of the error. The 425 error handler must either raise this or a different exception, or return a 426 tuple with a replacement for the unencodable part of the input and a position 427 where encoding should continue. The replacement may be either :class:`str` or 428 :class:`bytes`. If the replacement is bytes, the encoder will simply copy 429 them into the output buffer. If the replacement is a string, the encoder will 430 encode the replacement. Encoding continues on original input at the 431 specified position. Negative position values will be treated as being 432 relative to the end of the input string. If the resulting position is out of 433 bound an :exc:`IndexError` will be raised. 434 435 Decoding and translating works similarly, except :exc:`UnicodeDecodeError` or 436 :exc:`UnicodeTranslateError` will be passed to the handler and that the 437 replacement from the error handler will be put into the output directly. 438 439 440Previously registered error handlers (including the standard error handlers) 441can be looked up by name: 442 443.. function:: lookup_error(name) 444 445 Return the error handler previously registered under the name *name*. 446 447 Raises a :exc:`LookupError` in case the handler cannot be found. 448 449The following standard error handlers are also made available as module level 450functions: 451 452.. function:: strict_errors(exception) 453 454 Implements the ``'strict'`` error handling. 455 456 Each encoding or decoding error raises a :exc:`UnicodeError`. 457 458 459.. function:: ignore_errors(exception) 460 461 Implements the ``'ignore'`` error handling. 462 463 Malformed data is ignored; encoding or decoding is continued without 464 further notice. 465 466 467.. function:: replace_errors(exception) 468 469 Implements the ``'replace'`` error handling. 470 471 Substitutes ``?`` (ASCII character) for encoding errors or ``�`` (U+FFFD, 472 the official REPLACEMENT CHARACTER) for decoding errors. 473 474 475.. function:: backslashreplace_errors(exception) 476 477 Implements the ``'backslashreplace'`` error handling. 478 479 Malformed data is replaced by a backslashed escape sequence. 480 On encoding, use the hexadecimal form of Unicode code point with formats 481 ``\xhh`` ``\uxxxx`` ``\Uxxxxxxxx``. On decoding, use the hexadecimal form of 482 byte value with format ``\xhh``. 483 484 .. versionchanged:: 3.5 485 Works with decoding and translating. 486 487 488.. function:: xmlcharrefreplace_errors(exception) 489 490 Implements the ``'xmlcharrefreplace'`` error handling (for encoding within 491 :term:`text encoding` only). 492 493 The unencodable character is replaced by an appropriate XML/HTML numeric 494 character reference, which is a decimal form of Unicode code point with 495 format ``&#num;`` . 496 497 498.. function:: namereplace_errors(exception) 499 500 Implements the ``'namereplace'`` error handling (for encoding within 501 :term:`text encoding` only). 502 503 The unencodable character is replaced by a ``\N{...}`` escape sequence. The 504 set of characters that appear in the braces is the Name property from 505 Unicode Character Database. For example, the German lowercase letter ``'ß'`` 506 will be converted to byte sequence ``\N{LATIN SMALL LETTER SHARP S}`` . 507 508 .. versionadded:: 3.5 509 510 511.. _codec-objects: 512 513Stateless Encoding and Decoding 514^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 515 516The base :class:`Codec` class defines these methods which also define the 517function interfaces of the stateless encoder and decoder: 518 519 520.. method:: Codec.encode(input, errors='strict') 521 522 Encodes the object *input* and returns a tuple (output object, length consumed). 523 For instance, :term:`text encoding` converts 524 a string object to a bytes object using a particular 525 character set encoding (e.g., ``cp1252`` or ``iso-8859-1``). 526 527 The *errors* argument defines the error handling to apply. 528 It defaults to ``'strict'`` handling. 529 530 The method may not store state in the :class:`Codec` instance. Use 531 :class:`StreamWriter` for codecs which have to keep state in order to make 532 encoding efficient. 533 534 The encoder must be able to handle zero length input and return an empty object 535 of the output object type in this situation. 536 537 538.. method:: Codec.decode(input, errors='strict') 539 540 Decodes the object *input* and returns a tuple (output object, length 541 consumed). For instance, for a :term:`text encoding`, decoding converts 542 a bytes object encoded using a particular 543 character set encoding to a string object. 544 545 For text encodings and bytes-to-bytes codecs, 546 *input* must be a bytes object or one which provides the read-only 547 buffer interface -- for example, buffer objects and memory mapped files. 548 549 The *errors* argument defines the error handling to apply. 550 It defaults to ``'strict'`` handling. 551 552 The method may not store state in the :class:`Codec` instance. Use 553 :class:`StreamReader` for codecs which have to keep state in order to make 554 decoding efficient. 555 556 The decoder must be able to handle zero length input and return an empty object 557 of the output object type in this situation. 558 559 560Incremental Encoding and Decoding 561^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 562 563The :class:`IncrementalEncoder` and :class:`IncrementalDecoder` classes provide 564the basic interface for incremental encoding and decoding. Encoding/decoding the 565input isn't done with one call to the stateless encoder/decoder function, but 566with multiple calls to the 567:meth:`~IncrementalEncoder.encode`/:meth:`~IncrementalDecoder.decode` method of 568the incremental encoder/decoder. The incremental encoder/decoder keeps track of 569the encoding/decoding process during method calls. 570 571The joined output of calls to the 572:meth:`~IncrementalEncoder.encode`/:meth:`~IncrementalDecoder.decode` method is 573the same as if all the single inputs were joined into one, and this input was 574encoded/decoded with the stateless encoder/decoder. 575 576 577.. _incremental-encoder-objects: 578 579IncrementalEncoder Objects 580~~~~~~~~~~~~~~~~~~~~~~~~~~ 581 582The :class:`IncrementalEncoder` class is used for encoding an input in multiple 583steps. It defines the following methods which every incremental encoder must 584define in order to be compatible with the Python codec registry. 585 586 587.. class:: IncrementalEncoder(errors='strict') 588 589 Constructor for an :class:`IncrementalEncoder` instance. 590 591 All incremental encoders must provide this constructor interface. They are free 592 to add additional keyword arguments, but only the ones defined here are used by 593 the Python codec registry. 594 595 The :class:`IncrementalEncoder` may implement different error handling schemes 596 by providing the *errors* keyword argument. See :ref:`error-handlers` for 597 possible values. 598 599 The *errors* argument will be assigned to an attribute of the same name. 600 Assigning to this attribute makes it possible to switch between different error 601 handling strategies during the lifetime of the :class:`IncrementalEncoder` 602 object. 603 604 605 .. method:: encode(object, final=False) 606 607 Encodes *object* (taking the current state of the encoder into account) 608 and returns the resulting encoded object. If this is the last call to 609 :meth:`encode` *final* must be true (the default is false). 610 611 612 .. method:: reset() 613 614 Reset the encoder to the initial state. The output is discarded: call 615 ``.encode(object, final=True)``, passing an empty byte or text string 616 if necessary, to reset the encoder and to get the output. 617 618 619 .. method:: getstate() 620 621 Return the current state of the encoder which must be an integer. The 622 implementation should make sure that ``0`` is the most common 623 state. (States that are more complicated than integers can be converted 624 into an integer by marshaling/pickling the state and encoding the bytes 625 of the resulting string into an integer.) 626 627 628 .. method:: setstate(state) 629 630 Set the state of the encoder to *state*. *state* must be an encoder state 631 returned by :meth:`getstate`. 632 633 634.. _incremental-decoder-objects: 635 636IncrementalDecoder Objects 637~~~~~~~~~~~~~~~~~~~~~~~~~~ 638 639The :class:`IncrementalDecoder` class is used for decoding an input in multiple 640steps. It defines the following methods which every incremental decoder must 641define in order to be compatible with the Python codec registry. 642 643 644.. class:: IncrementalDecoder(errors='strict') 645 646 Constructor for an :class:`IncrementalDecoder` instance. 647 648 All incremental decoders must provide this constructor interface. They are free 649 to add additional keyword arguments, but only the ones defined here are used by 650 the Python codec registry. 651 652 The :class:`IncrementalDecoder` may implement different error handling schemes 653 by providing the *errors* keyword argument. See :ref:`error-handlers` for 654 possible values. 655 656 The *errors* argument will be assigned to an attribute of the same name. 657 Assigning to this attribute makes it possible to switch between different error 658 handling strategies during the lifetime of the :class:`IncrementalDecoder` 659 object. 660 661 662 .. method:: decode(object, final=False) 663 664 Decodes *object* (taking the current state of the decoder into account) 665 and returns the resulting decoded object. If this is the last call to 666 :meth:`decode` *final* must be true (the default is false). If *final* is 667 true the decoder must decode the input completely and must flush all 668 buffers. If this isn't possible (e.g. because of incomplete byte sequences 669 at the end of the input) it must initiate error handling just like in the 670 stateless case (which might raise an exception). 671 672 673 .. method:: reset() 674 675 Reset the decoder to the initial state. 676 677 678 .. method:: getstate() 679 680 Return the current state of the decoder. This must be a tuple with two 681 items, the first must be the buffer containing the still undecoded 682 input. The second must be an integer and can be additional state 683 info. (The implementation should make sure that ``0`` is the most common 684 additional state info.) If this additional state info is ``0`` it must be 685 possible to set the decoder to the state which has no input buffered and 686 ``0`` as the additional state info, so that feeding the previously 687 buffered input to the decoder returns it to the previous state without 688 producing any output. (Additional state info that is more complicated than 689 integers can be converted into an integer by marshaling/pickling the info 690 and encoding the bytes of the resulting string into an integer.) 691 692 693 .. method:: setstate(state) 694 695 Set the state of the decoder to *state*. *state* must be a decoder state 696 returned by :meth:`getstate`. 697 698 699Stream Encoding and Decoding 700^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 701 702 703The :class:`StreamWriter` and :class:`StreamReader` classes provide generic 704working interfaces which can be used to implement new encoding submodules very 705easily. See :mod:`encodings.utf_8` for an example of how this is done. 706 707 708.. _stream-writer-objects: 709 710StreamWriter Objects 711~~~~~~~~~~~~~~~~~~~~ 712 713The :class:`StreamWriter` class is a subclass of :class:`Codec` and defines the 714following methods which every stream writer must define in order to be 715compatible with the Python codec registry. 716 717 718.. class:: StreamWriter(stream, errors='strict') 719 720 Constructor for a :class:`StreamWriter` instance. 721 722 All stream writers must provide this constructor interface. They are free to add 723 additional keyword arguments, but only the ones defined here are used by the 724 Python codec registry. 725 726 The *stream* argument must be a file-like object open for writing 727 text or binary data, as appropriate for the specific codec. 728 729 The :class:`StreamWriter` may implement different error handling schemes by 730 providing the *errors* keyword argument. See :ref:`error-handlers` for 731 the standard error handlers the underlying stream codec may support. 732 733 The *errors* argument will be assigned to an attribute of the same name. 734 Assigning to this attribute makes it possible to switch between different error 735 handling strategies during the lifetime of the :class:`StreamWriter` object. 736 737 .. method:: write(object) 738 739 Writes the object's contents encoded to the stream. 740 741 742 .. method:: writelines(list) 743 744 Writes the concatenated iterable of strings to the stream (possibly by reusing 745 the :meth:`write` method). Infinite or 746 very large iterables are not supported. The standard bytes-to-bytes codecs 747 do not support this method. 748 749 750 .. method:: reset() 751 752 Resets the codec buffers used for keeping internal state. 753 754 Calling this method should ensure that the data on the output is put into 755 a clean state that allows appending of new fresh data without having to 756 rescan the whole stream to recover state. 757 758 759In addition to the above methods, the :class:`StreamWriter` must also inherit 760all other methods and attributes from the underlying stream. 761 762 763.. _stream-reader-objects: 764 765StreamReader Objects 766~~~~~~~~~~~~~~~~~~~~ 767 768The :class:`StreamReader` class is a subclass of :class:`Codec` and defines the 769following methods which every stream reader must define in order to be 770compatible with the Python codec registry. 771 772 773.. class:: StreamReader(stream, errors='strict') 774 775 Constructor for a :class:`StreamReader` instance. 776 777 All stream readers must provide this constructor interface. They are free to add 778 additional keyword arguments, but only the ones defined here are used by the 779 Python codec registry. 780 781 The *stream* argument must be a file-like object open for reading 782 text or binary data, as appropriate for the specific codec. 783 784 The :class:`StreamReader` may implement different error handling schemes by 785 providing the *errors* keyword argument. See :ref:`error-handlers` for 786 the standard error handlers the underlying stream codec may support. 787 788 The *errors* argument will be assigned to an attribute of the same name. 789 Assigning to this attribute makes it possible to switch between different error 790 handling strategies during the lifetime of the :class:`StreamReader` object. 791 792 The set of allowed values for the *errors* argument can be extended with 793 :func:`register_error`. 794 795 796 .. method:: read(size=-1, chars=-1, firstline=False) 797 798 Decodes data from the stream and returns the resulting object. 799 800 The *chars* argument indicates the number of decoded 801 code points or bytes to return. The :func:`read` method will 802 never return more data than requested, but it might return less, 803 if there is not enough available. 804 805 The *size* argument indicates the approximate maximum 806 number of encoded bytes or code points to read 807 for decoding. The decoder can modify this setting as 808 appropriate. The default value -1 indicates to read and decode as much as 809 possible. This parameter is intended to 810 prevent having to decode huge files in one step. 811 812 The *firstline* flag indicates that 813 it would be sufficient to only return the first 814 line, if there are decoding errors on later lines. 815 816 The method should use a greedy read strategy meaning that it should read 817 as much data as is allowed within the definition of the encoding and the 818 given size, e.g. if optional encoding endings or state markers are 819 available on the stream, these should be read too. 820 821 822 .. method:: readline(size=None, keepends=True) 823 824 Read one line from the input stream and return the decoded data. 825 826 *size*, if given, is passed as size argument to the stream's 827 :meth:`read` method. 828 829 If *keepends* is false line-endings will be stripped from the lines 830 returned. 831 832 833 .. method:: readlines(sizehint=None, keepends=True) 834 835 Read all lines available on the input stream and return them as a list of 836 lines. 837 838 Line-endings are implemented using the codec's :meth:`decode` method and 839 are included in the list entries if *keepends* is true. 840 841 *sizehint*, if given, is passed as the *size* argument to the stream's 842 :meth:`read` method. 843 844 845 .. method:: reset() 846 847 Resets the codec buffers used for keeping internal state. 848 849 Note that no stream repositioning should take place. This method is 850 primarily intended to be able to recover from decoding errors. 851 852 853In addition to the above methods, the :class:`StreamReader` must also inherit 854all other methods and attributes from the underlying stream. 855 856.. _stream-reader-writer: 857 858StreamReaderWriter Objects 859~~~~~~~~~~~~~~~~~~~~~~~~~~ 860 861The :class:`StreamReaderWriter` is a convenience class that allows wrapping 862streams which work in both read and write modes. 863 864The design is such that one can use the factory functions returned by the 865:func:`lookup` function to construct the instance. 866 867 868.. class:: StreamReaderWriter(stream, Reader, Writer, errors='strict') 869 870 Creates a :class:`StreamReaderWriter` instance. *stream* must be a file-like 871 object. *Reader* and *Writer* must be factory functions or classes providing the 872 :class:`StreamReader` and :class:`StreamWriter` interface resp. Error handling 873 is done in the same way as defined for the stream readers and writers. 874 875:class:`StreamReaderWriter` instances define the combined interfaces of 876:class:`StreamReader` and :class:`StreamWriter` classes. They inherit all other 877methods and attributes from the underlying stream. 878 879 880.. _stream-recoder-objects: 881 882StreamRecoder Objects 883~~~~~~~~~~~~~~~~~~~~~ 884 885The :class:`StreamRecoder` translates data from one encoding to another, 886which is sometimes useful when dealing with different encoding environments. 887 888The design is such that one can use the factory functions returned by the 889:func:`lookup` function to construct the instance. 890 891 892.. class:: StreamRecoder(stream, encode, decode, Reader, Writer, errors='strict') 893 894 Creates a :class:`StreamRecoder` instance which implements a two-way conversion: 895 *encode* and *decode* work on the frontend — the data visible to 896 code calling :meth:`read` and :meth:`write`, while *Reader* and *Writer* 897 work on the backend — the data in *stream*. 898 899 You can use these objects to do transparent transcodings, e.g., from Latin-1 900 to UTF-8 and back. 901 902 The *stream* argument must be a file-like object. 903 904 The *encode* and *decode* arguments must 905 adhere to the :class:`Codec` interface. *Reader* and 906 *Writer* must be factory functions or classes providing objects of the 907 :class:`StreamReader` and :class:`StreamWriter` interface respectively. 908 909 Error handling is done in the same way as defined for the stream readers and 910 writers. 911 912 913:class:`StreamRecoder` instances define the combined interfaces of 914:class:`StreamReader` and :class:`StreamWriter` classes. They inherit all other 915methods and attributes from the underlying stream. 916 917 918.. _encodings-overview: 919 920Encodings and Unicode 921--------------------- 922 923Strings are stored internally as sequences of code points in 924range ``U+0000``--``U+10FFFF``. (See :pep:`393` for 925more details about the implementation.) 926Once a string object is used outside of CPU and memory, endianness 927and how these arrays are stored as bytes become an issue. As with other 928codecs, serialising a string into a sequence of bytes is known as *encoding*, 929and recreating the string from the sequence of bytes is known as *decoding*. 930There are a variety of different text serialisation codecs, which are 931collectivity referred to as :term:`text encodings <text encoding>`. 932 933The simplest text encoding (called ``'latin-1'`` or ``'iso-8859-1'``) maps 934the code points 0--255 to the bytes ``0x0``--``0xff``, which means that a string 935object that contains code points above ``U+00FF`` can't be encoded with this 936codec. Doing so will raise a :exc:`UnicodeEncodeError` that looks 937like the following (although the details of the error message may differ): 938``UnicodeEncodeError: 'latin-1' codec can't encode character '\u1234' in 939position 3: ordinal not in range(256)``. 940 941There's another group of encodings (the so called charmap encodings) that choose 942a different subset of all Unicode code points and how these code points are 943mapped to the bytes ``0x0``--``0xff``. To see how this is done simply open 944e.g. :file:`encodings/cp1252.py` (which is an encoding that is used primarily on 945Windows). There's a string constant with 256 characters that shows you which 946character is mapped to which byte value. 947 948All of these encodings can only encode 256 of the 1114112 code points 949defined in Unicode. A simple and straightforward way that can store each Unicode 950code point, is to store each code point as four consecutive bytes. There are two 951possibilities: store the bytes in big endian or in little endian order. These 952two encodings are called ``UTF-32-BE`` and ``UTF-32-LE`` respectively. Their 953disadvantage is that if e.g. you use ``UTF-32-BE`` on a little endian machine you 954will always have to swap bytes on encoding and decoding. ``UTF-32`` avoids this 955problem: bytes will always be in natural endianness. When these bytes are read 956by a CPU with a different endianness, then bytes have to be swapped though. To 957be able to detect the endianness of a ``UTF-16`` or ``UTF-32`` byte sequence, 958there's the so called BOM ("Byte Order Mark"). This is the Unicode character 959``U+FEFF``. This character can be prepended to every ``UTF-16`` or ``UTF-32`` 960byte sequence. The byte swapped version of this character (``0xFFFE``) is an 961illegal character that may not appear in a Unicode text. So when the 962first character in a ``UTF-16`` or ``UTF-32`` byte sequence 963appears to be a ``U+FFFE`` the bytes have to be swapped on decoding. 964Unfortunately the character ``U+FEFF`` had a second purpose as 965a ``ZERO WIDTH NO-BREAK SPACE``: a character that has no width and doesn't allow 966a word to be split. It can e.g. be used to give hints to a ligature algorithm. 967With Unicode 4.0 using ``U+FEFF`` as a ``ZERO WIDTH NO-BREAK SPACE`` has been 968deprecated (with ``U+2060`` (``WORD JOINER``) assuming this role). Nevertheless 969Unicode software still must be able to handle ``U+FEFF`` in both roles: as a BOM 970it's a device to determine the storage layout of the encoded bytes, and vanishes 971once the byte sequence has been decoded into a string; as a ``ZERO WIDTH 972NO-BREAK SPACE`` it's a normal character that will be decoded like any other. 973 974There's another encoding that is able to encode the full range of Unicode 975characters: UTF-8. UTF-8 is an 8-bit encoding, which means there are no issues 976with byte order in UTF-8. Each byte in a UTF-8 byte sequence consists of two 977parts: marker bits (the most significant bits) and payload bits. The marker bits 978are a sequence of zero to four ``1`` bits followed by a ``0`` bit. Unicode characters are 979encoded like this (with x being payload bits, which when concatenated give the 980Unicode character): 981 982+-----------------------------------+----------------------------------------------+ 983| Range | Encoding | 984+===================================+==============================================+ 985| ``U-00000000`` ... ``U-0000007F`` | 0xxxxxxx | 986+-----------------------------------+----------------------------------------------+ 987| ``U-00000080`` ... ``U-000007FF`` | 110xxxxx 10xxxxxx | 988+-----------------------------------+----------------------------------------------+ 989| ``U-00000800`` ... ``U-0000FFFF`` | 1110xxxx 10xxxxxx 10xxxxxx | 990+-----------------------------------+----------------------------------------------+ 991| ``U-00010000`` ... ``U-0010FFFF`` | 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx | 992+-----------------------------------+----------------------------------------------+ 993 994The least significant bit of the Unicode character is the rightmost x bit. 995 996As UTF-8 is an 8-bit encoding no BOM is required and any ``U+FEFF`` character in 997the decoded string (even if it's the first character) is treated as a ``ZERO 998WIDTH NO-BREAK SPACE``. 999 1000Without external information it's impossible to reliably determine which 1001encoding was used for encoding a string. Each charmap encoding can 1002decode any random byte sequence. However that's not possible with UTF-8, as 1003UTF-8 byte sequences have a structure that doesn't allow arbitrary byte 1004sequences. To increase the reliability with which a UTF-8 encoding can be 1005detected, Microsoft invented a variant of UTF-8 (that Python calls 1006``"utf-8-sig"``) for its Notepad program: Before any of the Unicode characters 1007is written to the file, a UTF-8 encoded BOM (which looks like this as a byte 1008sequence: ``0xef``, ``0xbb``, ``0xbf``) is written. As it's rather improbable 1009that any charmap encoded file starts with these byte values (which would e.g. 1010map to 1011 1012 | LATIN SMALL LETTER I WITH DIAERESIS 1013 | RIGHT-POINTING DOUBLE ANGLE QUOTATION MARK 1014 | INVERTED QUESTION MARK 1015 1016in iso-8859-1), this increases the probability that a ``utf-8-sig`` encoding can be 1017correctly guessed from the byte sequence. So here the BOM is not used to be able 1018to determine the byte order used for generating the byte sequence, but as a 1019signature that helps in guessing the encoding. On encoding the utf-8-sig codec 1020will write ``0xef``, ``0xbb``, ``0xbf`` as the first three bytes to the file. On 1021decoding ``utf-8-sig`` will skip those three bytes if they appear as the first 1022three bytes in the file. In UTF-8, the use of the BOM is discouraged and 1023should generally be avoided. 1024 1025 1026.. _standard-encodings: 1027 1028Standard Encodings 1029------------------ 1030 1031Python comes with a number of codecs built-in, either implemented as C functions 1032or with dictionaries as mapping tables. The following table lists the codecs by 1033name, together with a few common aliases, and the languages for which the 1034encoding is likely used. Neither the list of aliases nor the list of languages 1035is meant to be exhaustive. Notice that spelling alternatives that only differ in 1036case or use a hyphen instead of an underscore are also valid aliases; therefore, 1037e.g. ``'utf-8'`` is a valid alias for the ``'utf_8'`` codec. 1038 1039.. impl-detail:: 1040 1041 Some common encodings can bypass the codecs lookup machinery to 1042 improve performance. These optimization opportunities are only 1043 recognized by CPython for a limited set of (case insensitive) 1044 aliases: utf-8, utf8, latin-1, latin1, iso-8859-1, iso8859-1, mbcs 1045 (Windows only), ascii, us-ascii, utf-16, utf16, utf-32, utf32, and 1046 the same using underscores instead of dashes. Using alternative 1047 aliases for these encodings may result in slower execution. 1048 1049 .. versionchanged:: 3.6 1050 Optimization opportunity recognized for us-ascii. 1051 1052Many of the character sets support the same languages. They vary in individual 1053characters (e.g. whether the EURO SIGN is supported or not), and in the 1054assignment of characters to code positions. For the European languages in 1055particular, the following variants typically exist: 1056 1057* an ISO 8859 codeset 1058 1059* a Microsoft Windows code page, which is typically derived from an 8859 codeset, 1060 but replaces control characters with additional graphic characters 1061 1062* an IBM EBCDIC code page 1063 1064* an IBM PC code page, which is ASCII compatible 1065 1066.. tabularcolumns:: |l|p{0.3\linewidth}|p{0.3\linewidth}| 1067 1068+-----------------+--------------------------------+--------------------------------+ 1069| Codec | Aliases | Languages | 1070+=================+================================+================================+ 1071| ascii | 646, us-ascii | English | 1072+-----------------+--------------------------------+--------------------------------+ 1073| big5 | big5-tw, csbig5 | Traditional Chinese | 1074+-----------------+--------------------------------+--------------------------------+ 1075| big5hkscs | big5-hkscs, hkscs | Traditional Chinese | 1076+-----------------+--------------------------------+--------------------------------+ 1077| cp037 | IBM037, IBM039 | English | 1078+-----------------+--------------------------------+--------------------------------+ 1079| cp273 | 273, IBM273, csIBM273 | German | 1080| | | | 1081| | | .. versionadded:: 3.4 | 1082+-----------------+--------------------------------+--------------------------------+ 1083| cp424 | EBCDIC-CP-HE, IBM424 | Hebrew | 1084+-----------------+--------------------------------+--------------------------------+ 1085| cp437 | 437, IBM437 | English | 1086+-----------------+--------------------------------+--------------------------------+ 1087| cp500 | EBCDIC-CP-BE, EBCDIC-CP-CH, | Western Europe | 1088| | IBM500 | | 1089+-----------------+--------------------------------+--------------------------------+ 1090| cp720 | | Arabic | 1091+-----------------+--------------------------------+--------------------------------+ 1092| cp737 | | Greek | 1093+-----------------+--------------------------------+--------------------------------+ 1094| cp775 | IBM775 | Baltic languages | 1095+-----------------+--------------------------------+--------------------------------+ 1096| cp850 | 850, IBM850 | Western Europe | 1097+-----------------+--------------------------------+--------------------------------+ 1098| cp852 | 852, IBM852 | Central and Eastern Europe | 1099+-----------------+--------------------------------+--------------------------------+ 1100| cp855 | 855, IBM855 | Bulgarian, Byelorussian, | 1101| | | Macedonian, Russian, Serbian | 1102+-----------------+--------------------------------+--------------------------------+ 1103| cp856 | | Hebrew | 1104+-----------------+--------------------------------+--------------------------------+ 1105| cp857 | 857, IBM857 | Turkish | 1106+-----------------+--------------------------------+--------------------------------+ 1107| cp858 | 858, IBM858 | Western Europe | 1108+-----------------+--------------------------------+--------------------------------+ 1109| cp860 | 860, IBM860 | Portuguese | 1110+-----------------+--------------------------------+--------------------------------+ 1111| cp861 | 861, CP-IS, IBM861 | Icelandic | 1112+-----------------+--------------------------------+--------------------------------+ 1113| cp862 | 862, IBM862 | Hebrew | 1114+-----------------+--------------------------------+--------------------------------+ 1115| cp863 | 863, IBM863 | Canadian | 1116+-----------------+--------------------------------+--------------------------------+ 1117| cp864 | IBM864 | Arabic | 1118+-----------------+--------------------------------+--------------------------------+ 1119| cp865 | 865, IBM865 | Danish, Norwegian | 1120+-----------------+--------------------------------+--------------------------------+ 1121| cp866 | 866, IBM866 | Russian | 1122+-----------------+--------------------------------+--------------------------------+ 1123| cp869 | 869, CP-GR, IBM869 | Greek | 1124+-----------------+--------------------------------+--------------------------------+ 1125| cp874 | | Thai | 1126+-----------------+--------------------------------+--------------------------------+ 1127| cp875 | | Greek | 1128+-----------------+--------------------------------+--------------------------------+ 1129| cp932 | 932, ms932, mskanji, ms-kanji | Japanese | 1130+-----------------+--------------------------------+--------------------------------+ 1131| cp949 | 949, ms949, uhc | Korean | 1132+-----------------+--------------------------------+--------------------------------+ 1133| cp950 | 950, ms950 | Traditional Chinese | 1134+-----------------+--------------------------------+--------------------------------+ 1135| cp1006 | | Urdu | 1136+-----------------+--------------------------------+--------------------------------+ 1137| cp1026 | ibm1026 | Turkish | 1138+-----------------+--------------------------------+--------------------------------+ 1139| cp1125 | 1125, ibm1125, cp866u, ruscii | Ukrainian | 1140| | | | 1141| | | .. versionadded:: 3.4 | 1142+-----------------+--------------------------------+--------------------------------+ 1143| cp1140 | ibm1140 | Western Europe | 1144+-----------------+--------------------------------+--------------------------------+ 1145| cp1250 | windows-1250 | Central and Eastern Europe | 1146+-----------------+--------------------------------+--------------------------------+ 1147| cp1251 | windows-1251 | Bulgarian, Byelorussian, | 1148| | | Macedonian, Russian, Serbian | 1149+-----------------+--------------------------------+--------------------------------+ 1150| cp1252 | windows-1252 | Western Europe | 1151+-----------------+--------------------------------+--------------------------------+ 1152| cp1253 | windows-1253 | Greek | 1153+-----------------+--------------------------------+--------------------------------+ 1154| cp1254 | windows-1254 | Turkish | 1155+-----------------+--------------------------------+--------------------------------+ 1156| cp1255 | windows-1255 | Hebrew | 1157+-----------------+--------------------------------+--------------------------------+ 1158| cp1256 | windows-1256 | Arabic | 1159+-----------------+--------------------------------+--------------------------------+ 1160| cp1257 | windows-1257 | Baltic languages | 1161+-----------------+--------------------------------+--------------------------------+ 1162| cp1258 | windows-1258 | Vietnamese | 1163+-----------------+--------------------------------+--------------------------------+ 1164| euc_jp | eucjp, ujis, u-jis | Japanese | 1165+-----------------+--------------------------------+--------------------------------+ 1166| euc_jis_2004 | jisx0213, eucjis2004 | Japanese | 1167+-----------------+--------------------------------+--------------------------------+ 1168| euc_jisx0213 | eucjisx0213 | Japanese | 1169+-----------------+--------------------------------+--------------------------------+ 1170| euc_kr | euckr, korean, ksc5601, | Korean | 1171| | ks_c-5601, ks_c-5601-1987, | | 1172| | ksx1001, ks_x-1001 | | 1173+-----------------+--------------------------------+--------------------------------+ 1174| gb2312 | chinese, csiso58gb231280, | Simplified Chinese | 1175| | euc-cn, euccn, eucgb2312-cn, | | 1176| | gb2312-1980, gb2312-80, | | 1177| | iso-ir-58 | | 1178+-----------------+--------------------------------+--------------------------------+ 1179| gbk | 936, cp936, ms936 | Unified Chinese | 1180+-----------------+--------------------------------+--------------------------------+ 1181| gb18030 | gb18030-2000 | Unified Chinese | 1182+-----------------+--------------------------------+--------------------------------+ 1183| hz | hzgb, hz-gb, hz-gb-2312 | Simplified Chinese | 1184+-----------------+--------------------------------+--------------------------------+ 1185| iso2022_jp | csiso2022jp, iso2022jp, | Japanese | 1186| | iso-2022-jp | | 1187+-----------------+--------------------------------+--------------------------------+ 1188| iso2022_jp_1 | iso2022jp-1, iso-2022-jp-1 | Japanese | 1189+-----------------+--------------------------------+--------------------------------+ 1190| iso2022_jp_2 | iso2022jp-2, iso-2022-jp-2 | Japanese, Korean, Simplified | 1191| | | Chinese, Western Europe, Greek | 1192+-----------------+--------------------------------+--------------------------------+ 1193| iso2022_jp_2004 | iso2022jp-2004, | Japanese | 1194| | iso-2022-jp-2004 | | 1195+-----------------+--------------------------------+--------------------------------+ 1196| iso2022_jp_3 | iso2022jp-3, iso-2022-jp-3 | Japanese | 1197+-----------------+--------------------------------+--------------------------------+ 1198| iso2022_jp_ext | iso2022jp-ext, iso-2022-jp-ext | Japanese | 1199+-----------------+--------------------------------+--------------------------------+ 1200| iso2022_kr | csiso2022kr, iso2022kr, | Korean | 1201| | iso-2022-kr | | 1202+-----------------+--------------------------------+--------------------------------+ 1203| latin_1 | iso-8859-1, iso8859-1, 8859, | Western Europe | 1204| | cp819, latin, latin1, L1 | | 1205+-----------------+--------------------------------+--------------------------------+ 1206| iso8859_2 | iso-8859-2, latin2, L2 | Central and Eastern Europe | 1207+-----------------+--------------------------------+--------------------------------+ 1208| iso8859_3 | iso-8859-3, latin3, L3 | Esperanto, Maltese | 1209+-----------------+--------------------------------+--------------------------------+ 1210| iso8859_4 | iso-8859-4, latin4, L4 | Baltic languages | 1211+-----------------+--------------------------------+--------------------------------+ 1212| iso8859_5 | iso-8859-5, cyrillic | Bulgarian, Byelorussian, | 1213| | | Macedonian, Russian, Serbian | 1214+-----------------+--------------------------------+--------------------------------+ 1215| iso8859_6 | iso-8859-6, arabic | Arabic | 1216+-----------------+--------------------------------+--------------------------------+ 1217| iso8859_7 | iso-8859-7, greek, greek8 | Greek | 1218+-----------------+--------------------------------+--------------------------------+ 1219| iso8859_8 | iso-8859-8, hebrew | Hebrew | 1220+-----------------+--------------------------------+--------------------------------+ 1221| iso8859_9 | iso-8859-9, latin5, L5 | Turkish | 1222+-----------------+--------------------------------+--------------------------------+ 1223| iso8859_10 | iso-8859-10, latin6, L6 | Nordic languages | 1224+-----------------+--------------------------------+--------------------------------+ 1225| iso8859_11 | iso-8859-11, thai | Thai languages | 1226+-----------------+--------------------------------+--------------------------------+ 1227| iso8859_13 | iso-8859-13, latin7, L7 | Baltic languages | 1228+-----------------+--------------------------------+--------------------------------+ 1229| iso8859_14 | iso-8859-14, latin8, L8 | Celtic languages | 1230+-----------------+--------------------------------+--------------------------------+ 1231| iso8859_15 | iso-8859-15, latin9, L9 | Western Europe | 1232+-----------------+--------------------------------+--------------------------------+ 1233| iso8859_16 | iso-8859-16, latin10, L10 | South-Eastern Europe | 1234+-----------------+--------------------------------+--------------------------------+ 1235| johab | cp1361, ms1361 | Korean | 1236+-----------------+--------------------------------+--------------------------------+ 1237| koi8_r | | Russian | 1238+-----------------+--------------------------------+--------------------------------+ 1239| koi8_t | | Tajik | 1240| | | | 1241| | | .. versionadded:: 3.5 | 1242+-----------------+--------------------------------+--------------------------------+ 1243| koi8_u | | Ukrainian | 1244+-----------------+--------------------------------+--------------------------------+ 1245| kz1048 | kz_1048, strk1048_2002, rk1048 | Kazakh | 1246| | | | 1247| | | .. versionadded:: 3.5 | 1248+-----------------+--------------------------------+--------------------------------+ 1249| mac_cyrillic | maccyrillic | Bulgarian, Byelorussian, | 1250| | | Macedonian, Russian, Serbian | 1251+-----------------+--------------------------------+--------------------------------+ 1252| mac_greek | macgreek | Greek | 1253+-----------------+--------------------------------+--------------------------------+ 1254| mac_iceland | maciceland | Icelandic | 1255+-----------------+--------------------------------+--------------------------------+ 1256| mac_latin2 | maclatin2, maccentraleurope, | Central and Eastern Europe | 1257| | mac_centeuro | | 1258+-----------------+--------------------------------+--------------------------------+ 1259| mac_roman | macroman, macintosh | Western Europe | 1260+-----------------+--------------------------------+--------------------------------+ 1261| mac_turkish | macturkish | Turkish | 1262+-----------------+--------------------------------+--------------------------------+ 1263| ptcp154 | csptcp154, pt154, cp154, | Kazakh | 1264| | cyrillic-asian | | 1265+-----------------+--------------------------------+--------------------------------+ 1266| shift_jis | csshiftjis, shiftjis, sjis, | Japanese | 1267| | s_jis | | 1268+-----------------+--------------------------------+--------------------------------+ 1269| shift_jis_2004 | shiftjis2004, sjis_2004, | Japanese | 1270| | sjis2004 | | 1271+-----------------+--------------------------------+--------------------------------+ 1272| shift_jisx0213 | shiftjisx0213, sjisx0213, | Japanese | 1273| | s_jisx0213 | | 1274+-----------------+--------------------------------+--------------------------------+ 1275| utf_32 | U32, utf32 | all languages | 1276+-----------------+--------------------------------+--------------------------------+ 1277| utf_32_be | UTF-32BE | all languages | 1278+-----------------+--------------------------------+--------------------------------+ 1279| utf_32_le | UTF-32LE | all languages | 1280+-----------------+--------------------------------+--------------------------------+ 1281| utf_16 | U16, utf16 | all languages | 1282+-----------------+--------------------------------+--------------------------------+ 1283| utf_16_be | UTF-16BE | all languages | 1284+-----------------+--------------------------------+--------------------------------+ 1285| utf_16_le | UTF-16LE | all languages | 1286+-----------------+--------------------------------+--------------------------------+ 1287| utf_7 | U7, unicode-1-1-utf-7 | all languages | 1288+-----------------+--------------------------------+--------------------------------+ 1289| utf_8 | U8, UTF, utf8, cp65001 | all languages | 1290+-----------------+--------------------------------+--------------------------------+ 1291| utf_8_sig | | all languages | 1292+-----------------+--------------------------------+--------------------------------+ 1293 1294.. versionchanged:: 3.4 1295 The utf-16\* and utf-32\* encoders no longer allow surrogate code points 1296 (``U+D800``--``U+DFFF``) to be encoded. 1297 The utf-32\* decoders no longer decode 1298 byte sequences that correspond to surrogate code points. 1299 1300.. versionchanged:: 3.8 1301 ``cp65001`` is now an alias to ``utf_8``. 1302 1303 1304Python Specific Encodings 1305------------------------- 1306 1307A number of predefined codecs are specific to Python, so their codec names have 1308no meaning outside Python. These are listed in the tables below based on the 1309expected input and output types (note that while text encodings are the most 1310common use case for codecs, the underlying codec infrastructure supports 1311arbitrary data transforms rather than just text encodings). For asymmetric 1312codecs, the stated meaning describes the encoding direction. 1313 1314Text Encodings 1315^^^^^^^^^^^^^^ 1316 1317The following codecs provide :class:`str` to :class:`bytes` encoding and 1318:term:`bytes-like object` to :class:`str` decoding, similar to the Unicode text 1319encodings. 1320 1321.. tabularcolumns:: |l|p{0.3\linewidth}|p{0.3\linewidth}| 1322 1323+--------------------+---------+---------------------------+ 1324| Codec | Aliases | Meaning | 1325+====================+=========+===========================+ 1326| idna | | Implement :rfc:`3490`, | 1327| | | see also | 1328| | | :mod:`encodings.idna`. | 1329| | | Only ``errors='strict'`` | 1330| | | is supported. | 1331+--------------------+---------+---------------------------+ 1332| mbcs | ansi, | Windows only: Encode the | 1333| | dbcs | operand according to the | 1334| | | ANSI codepage (CP_ACP). | 1335+--------------------+---------+---------------------------+ 1336| oem | | Windows only: Encode the | 1337| | | operand according to the | 1338| | | OEM codepage (CP_OEMCP). | 1339| | | | 1340| | | .. versionadded:: 3.6 | 1341+--------------------+---------+---------------------------+ 1342| palmos | | Encoding of PalmOS 3.5. | 1343+--------------------+---------+---------------------------+ 1344| punycode | | Implement :rfc:`3492`. | 1345| | | Stateful codecs are not | 1346| | | supported. | 1347+--------------------+---------+---------------------------+ 1348| raw_unicode_escape | | Latin-1 encoding with | 1349| | | ``\uXXXX`` and | 1350| | | ``\UXXXXXXXX`` for other | 1351| | | code points. Existing | 1352| | | backslashes are not | 1353| | | escaped in any way. | 1354| | | It is used in the Python | 1355| | | pickle protocol. | 1356+--------------------+---------+---------------------------+ 1357| undefined | | Raise an exception for | 1358| | | all conversions, even | 1359| | | empty strings. The error | 1360| | | handler is ignored. | 1361+--------------------+---------+---------------------------+ 1362| unicode_escape | | Encoding suitable as the | 1363| | | contents of a Unicode | 1364| | | literal in ASCII-encoded | 1365| | | Python source code, | 1366| | | except that quotes are | 1367| | | not escaped. Decode | 1368| | | from Latin-1 source code. | 1369| | | Beware that Python source | 1370| | | code actually uses UTF-8 | 1371| | | by default. | 1372+--------------------+---------+---------------------------+ 1373 1374.. versionchanged:: 3.8 1375 "unicode_internal" codec is removed. 1376 1377 1378.. _binary-transforms: 1379 1380Binary Transforms 1381^^^^^^^^^^^^^^^^^ 1382 1383The following codecs provide binary transforms: :term:`bytes-like object` 1384to :class:`bytes` mappings. They are not supported by :meth:`bytes.decode` 1385(which only produces :class:`str` output). 1386 1387 1388.. tabularcolumns:: |l|L|L|L| 1389 1390+----------------------+------------------+------------------------------+------------------------------+ 1391| Codec | Aliases | Meaning | Encoder / decoder | 1392+======================+==================+==============================+==============================+ 1393| base64_codec [#b64]_ | base64, base_64 | Convert the operand to | :meth:`base64.encodebytes` / | 1394| | | multiline MIME base64 (the | :meth:`base64.decodebytes` | 1395| | | result always includes a | | 1396| | | trailing ``'\n'``). | | 1397| | | | | 1398| | | .. versionchanged:: 3.4 | | 1399| | | accepts any | | 1400| | | :term:`bytes-like object` | | 1401| | | as input for encoding and | | 1402| | | decoding | | 1403+----------------------+------------------+------------------------------+------------------------------+ 1404| bz2_codec | bz2 | Compress the operand using | :meth:`bz2.compress` / | 1405| | | bz2. | :meth:`bz2.decompress` | 1406+----------------------+------------------+------------------------------+------------------------------+ 1407| hex_codec | hex | Convert the operand to | :meth:`binascii.b2a_hex` / | 1408| | | hexadecimal | :meth:`binascii.a2b_hex` | 1409| | | representation, with two | | 1410| | | digits per byte. | | 1411+----------------------+------------------+------------------------------+------------------------------+ 1412| quopri_codec | quopri, | Convert the operand to MIME | :meth:`quopri.encode` with | 1413| | quotedprintable, | quoted printable. | ``quotetabs=True`` / | 1414| | quoted_printable | | :meth:`quopri.decode` | 1415+----------------------+------------------+------------------------------+------------------------------+ 1416| uu_codec | uu | Convert the operand using | :meth:`uu.encode` / | 1417| | | uuencode. | :meth:`uu.decode` | 1418+----------------------+------------------+------------------------------+------------------------------+ 1419| zlib_codec | zip, zlib | Compress the operand using | :meth:`zlib.compress` / | 1420| | | gzip. | :meth:`zlib.decompress` | 1421+----------------------+------------------+------------------------------+------------------------------+ 1422 1423.. [#b64] In addition to :term:`bytes-like objects <bytes-like object>`, 1424 ``'base64_codec'`` also accepts ASCII-only instances of :class:`str` for 1425 decoding 1426 1427.. versionadded:: 3.2 1428 Restoration of the binary transforms. 1429 1430.. versionchanged:: 3.4 1431 Restoration of the aliases for the binary transforms. 1432 1433 1434.. _text-transforms: 1435 1436Text Transforms 1437^^^^^^^^^^^^^^^ 1438 1439The following codec provides a text transform: a :class:`str` to :class:`str` 1440mapping. It is not supported by :meth:`str.encode` (which only produces 1441:class:`bytes` output). 1442 1443.. tabularcolumns:: |l|l|L| 1444 1445+--------------------+---------+---------------------------+ 1446| Codec | Aliases | Meaning | 1447+====================+=========+===========================+ 1448| rot_13 | rot13 | Return the Caesar-cypher | 1449| | | encryption of the | 1450| | | operand. | 1451+--------------------+---------+---------------------------+ 1452 1453.. versionadded:: 3.2 1454 Restoration of the ``rot_13`` text transform. 1455 1456.. versionchanged:: 3.4 1457 Restoration of the ``rot13`` alias. 1458 1459 1460:mod:`encodings.idna` --- Internationalized Domain Names in Applications 1461------------------------------------------------------------------------ 1462 1463.. module:: encodings.idna 1464 :synopsis: Internationalized Domain Names implementation 1465.. moduleauthor:: Martin v. Löwis 1466 1467This module implements :rfc:`3490` (Internationalized Domain Names in 1468Applications) and :rfc:`3492` (Nameprep: A Stringprep Profile for 1469Internationalized Domain Names (IDN)). It builds upon the ``punycode`` encoding 1470and :mod:`stringprep`. 1471 1472If you need the IDNA 2008 standard from :rfc:`5891` and :rfc:`5895`, use the 1473third-party `idna module <https://pypi.org/project/idna/>`_. 1474 1475These RFCs together define a protocol to support non-ASCII characters in domain 1476names. A domain name containing non-ASCII characters (such as 1477``www.Alliancefrançaise.nu``) is converted into an ASCII-compatible encoding 1478(ACE, such as ``www.xn--alliancefranaise-npb.nu``). The ACE form of the domain 1479name is then used in all places where arbitrary characters are not allowed by 1480the protocol, such as DNS queries, HTTP :mailheader:`Host` fields, and so 1481on. This conversion is carried out in the application; if possible invisible to 1482the user: The application should transparently convert Unicode domain labels to 1483IDNA on the wire, and convert back ACE labels to Unicode before presenting them 1484to the user. 1485 1486Python supports this conversion in several ways: the ``idna`` codec performs 1487conversion between Unicode and ACE, separating an input string into labels 1488based on the separator characters defined in :rfc:`section 3.1 of RFC 3490 <3490#section-3.1>` 1489and converting each label to ACE as required, and conversely separating an input 1490byte string into labels based on the ``.`` separator and converting any ACE 1491labels found into unicode. Furthermore, the :mod:`socket` module 1492transparently converts Unicode host names to ACE, so that applications need not 1493be concerned about converting host names themselves when they pass them to the 1494socket module. On top of that, modules that have host names as function 1495parameters, such as :mod:`http.client` and :mod:`ftplib`, accept Unicode host 1496names (:mod:`http.client` then also transparently sends an IDNA hostname in the 1497:mailheader:`Host` field if it sends that field at all). 1498 1499When receiving host names from the wire (such as in reverse name lookup), no 1500automatic conversion to Unicode is performed: applications wishing to present 1501such host names to the user should decode them to Unicode. 1502 1503The module :mod:`encodings.idna` also implements the nameprep procedure, which 1504performs certain normalizations on host names, to achieve case-insensitivity of 1505international domain names, and to unify similar characters. The nameprep 1506functions can be used directly if desired. 1507 1508 1509.. function:: nameprep(label) 1510 1511 Return the nameprepped version of *label*. The implementation currently assumes 1512 query strings, so ``AllowUnassigned`` is true. 1513 1514 1515.. function:: ToASCII(label) 1516 1517 Convert a label to ASCII, as specified in :rfc:`3490`. ``UseSTD3ASCIIRules`` is 1518 assumed to be false. 1519 1520 1521.. function:: ToUnicode(label) 1522 1523 Convert a label to Unicode, as specified in :rfc:`3490`. 1524 1525 1526:mod:`encodings.mbcs` --- Windows ANSI codepage 1527----------------------------------------------- 1528 1529.. module:: encodings.mbcs 1530 :synopsis: Windows ANSI codepage 1531 1532This module implements the ANSI codepage (CP_ACP). 1533 1534.. availability:: Windows. 1535 1536.. versionchanged:: 3.3 1537 Support any error handler. 1538 1539.. versionchanged:: 3.2 1540 Before 3.2, the *errors* argument was ignored; ``'replace'`` was always used 1541 to encode, and ``'ignore'`` to decode. 1542 1543 1544:mod:`encodings.utf_8_sig` --- UTF-8 codec with BOM signature 1545------------------------------------------------------------- 1546 1547.. module:: encodings.utf_8_sig 1548 :synopsis: UTF-8 codec with BOM signature 1549.. moduleauthor:: Walter Dörwald 1550 1551This module implements a variant of the UTF-8 codec. On encoding, a UTF-8 encoded 1552BOM will be prepended to the UTF-8 encoded bytes. For the stateful encoder this 1553is only done once (on the first write to the byte stream). On decoding, an 1554optional UTF-8 encoded BOM at the start of the data will be skipped. 1555