xref: /third_party/python/Doc/library/compileall.rst

:mod:`compileall` --- Byte-compile Python libraries
===================================================

.. module:: compileall
   :synopsis: Tools for byte-compiling all Python source files in a directory tree.

**Source code:** :source:`Lib/compileall.py`

--------------

This module provides some utility functions to support installing Python
libraries.  These functions compile Python source files in a directory tree.
This module can be used to create the cached byte-code files at library
installation time, which makes them available for use even by users who don't
have write permission to the library directories.

.. include:: ../includes/wasm-notavail.rst

Command-line use
----------------

This module can work as a script (using :program:`python -m compileall`) to
compile Python sources.

.. program:: compileall

.. cmdoption:: directory ...
               file ...

   Positional arguments are files to compile or directories that contain
   source files, traversed recursively.  If no argument is given, behave as if
   the command line was ``-l <directories from sys.path>``.

.. cmdoption:: -l

   Do not recurse into subdirectories, only compile source code files directly
   contained in the named or implied directories.

.. cmdoption:: -f

   Force rebuild even if timestamps are up-to-date.

.. cmdoption:: -q

   Do not print the list of files compiled. If passed once, error messages will
   still be printed. If passed twice (``-qq``), all output is suppressed.

.. cmdoption:: -d destdir

   Directory prepended to the path to each file being compiled.  This will
   appear in compilation time tracebacks, and is also compiled in to the
   byte-code file, where it will be used in tracebacks and other messages in
   cases where the source file does not exist at the time the byte-code file is
   executed.

.. cmdoption:: -s strip_prefix
.. cmdoption:: -p prepend_prefix

   Remove (``-s``) or append (``-p``) the given prefix of paths
   recorded in the ``.pyc`` files.
   Cannot be combined with ``-d``.

.. cmdoption:: -x regex

   regex is used to search the full path to each file considered for
   compilation, and if the regex produces a match, the file is skipped.

.. cmdoption:: -i list

   Read the file ``list`` and add each line that it contains to the list of
   files and directories to compile.  If ``list`` is ``-``, read lines from
   ``stdin``.

.. cmdoption:: -b

   Write the byte-code files to their legacy locations and names, which may
   overwrite byte-code files created by another version of Python.  The default
   is to write files to their :pep:`3147` locations and names, which allows
   byte-code files from multiple versions of Python to coexist.

.. cmdoption:: -r

   Control the maximum recursion level for subdirectories.
   If this is given, then ``-l`` option will not be taken into account.
   :program:`python -m compileall <directory> -r 0` is equivalent to
   :program:`python -m compileall <directory> -l`.

.. cmdoption:: -j N

   Use *N* workers to compile the files within the given directory.
   If ``0`` is used, then the result of :func:`os.cpu_count()`
   will be used.

.. cmdoption:: --invalidation-mode [timestamp|checked-hash|unchecked-hash]

   Control how the generated byte-code files are invalidated at runtime.
   The ``timestamp`` value, means that ``.pyc`` files with the source timestamp
   and size embedded will be generated. The ``checked-hash`` and
   ``unchecked-hash`` values cause hash-based pycs to be generated. Hash-based
   pycs embed a hash of the source file contents rather than a timestamp. See
   :ref:`pyc-invalidation` for more information on how Python validates
   bytecode cache files at runtime.
   The default is ``timestamp`` if the :envvar:`SOURCE_DATE_EPOCH` environment
   variable is not set, and ``checked-hash`` if the ``SOURCE_DATE_EPOCH``
   environment variable is set.

.. cmdoption:: -o level

   Compile with the given optimization level. May be used multiple times
   to compile for multiple levels at a time (for example,
   ``compileall -o 1 -o 2``).

.. cmdoption:: -e dir

   Ignore symlinks pointing outside the given directory.

.. cmdoption:: --hardlink-dupes

   If two ``.pyc`` files with different optimization level have
   the same content, use hard links to consolidate duplicate files.

.. versionchanged:: 3.2
   Added the ``-i``, ``-b`` and ``-h`` options.

.. versionchanged:: 3.5
   Added the  ``-j``, ``-r``, and ``-qq`` options.  ``-q`` option
   was changed to a multilevel value.  ``-b`` will always produce a
   byte-code file ending in ``.pyc``, never ``.pyo``.

.. versionchanged:: 3.7
   Added the ``--invalidation-mode`` option.

.. versionchanged:: 3.9
   Added the ``-s``, ``-p``, ``-e`` and ``--hardlink-dupes`` options.
   Raised the default recursion limit from 10 to
   :py:func:`sys.getrecursionlimit()`.
   Added the possibility to specify the ``-o`` option multiple times.


There is no command-line option to control the optimization level used by the
:func:`compile` function, because the Python interpreter itself already
provides the option: :program:`python -O -m compileall`.

Similarly, the :func:`compile` function respects the :attr:`sys.pycache_prefix`
setting. The generated bytecode cache will only be useful if :func:`compile` is
run with the same :attr:`sys.pycache_prefix` (if any) that will be used at
runtime.

Public functions
----------------

.. function:: compile_dir(dir, maxlevels=sys.getrecursionlimit(), ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=-1, workers=1, invalidation_mode=None, *, stripdir=None, prependdir=None, limit_sl_dest=None, hardlink_dupes=False)

   Recursively descend the directory tree named by *dir*, compiling all :file:`.py`
   files along the way. Return a true value if all the files compiled successfully,
   and a false value otherwise.

   The *maxlevels* parameter is used to limit the depth of the recursion; it
   defaults to ``sys.getrecursionlimit()``.

   If *ddir* is given, it is prepended to the path to each file being compiled
   for use in compilation time tracebacks, and is also compiled in to the
   byte-code file, where it will be used in tracebacks and other messages in
   cases where the source file does not exist at the time the byte-code file is
   executed.

   If *force* is true, modules are re-compiled even if the timestamps are up to
   date.

   If *rx* is given, its ``search`` method is called on the complete path to each
   file considered for compilation, and if it returns a true value, the file
   is skipped. This can be used to exclude files matching a regular expression,
   given as a :ref:`re.Pattern <re-objects>` object.

   If *quiet* is ``False`` or ``0`` (the default), the filenames and other
   information are printed to standard out. Set to ``1``, only errors are
   printed. Set to ``2``, all output is suppressed.

   If *legacy* is true, byte-code files are written to their legacy locations
   and names, which may overwrite byte-code files created by another version of
   Python.  The default is to write files to their :pep:`3147` locations and
   names, which allows byte-code files from multiple versions of Python to
   coexist.

   *optimize* specifies the optimization level for the compiler.  It is passed to
   the built-in :func:`compile` function. Accepts also a sequence of optimization
   levels which lead to multiple compilations of one :file:`.py` file in one call.

   The argument *workers* specifies how many workers are used to
   compile files in parallel. The default is to not use multiple workers.
   If the platform can't use multiple workers and *workers* argument is given,
   then sequential compilation will be used as a fallback.  If *workers*
   is 0, the number of cores in the system is used.  If *workers* is
   lower than ``0``, a :exc:`ValueError` will be raised.

   *invalidation_mode* should be a member of the
   :class:`py_compile.PycInvalidationMode` enum and controls how the generated
   pycs are invalidated at runtime.

   The *stripdir*, *prependdir* and *limit_sl_dest* arguments correspond to
   the ``-s``, ``-p`` and ``-e`` options described above.
   They may be specified as ``str`` or :py:class:`os.PathLike`.

   If *hardlink_dupes* is true and two ``.pyc`` files with different optimization
   level have the same content, use hard links to consolidate duplicate files.

   .. versionchanged:: 3.2
      Added the *legacy* and *optimize* parameter.

   .. versionchanged:: 3.5
      Added the *workers* parameter.

   .. versionchanged:: 3.5
      *quiet* parameter was changed to a multilevel value.

   .. versionchanged:: 3.5
      The *legacy* parameter only writes out ``.pyc`` files, not ``.pyo`` files
      no matter what the value of *optimize* is.

   .. versionchanged:: 3.6
      Accepts a :term:`path-like object`.

   .. versionchanged:: 3.7
      The *invalidation_mode* parameter was added.

   .. versionchanged:: 3.7.2
      The *invalidation_mode* parameter's default value is updated to None.

   .. versionchanged:: 3.8
      Setting *workers* to 0 now chooses the optimal number of cores.

   .. versionchanged:: 3.9
      Added *stripdir*, *prependdir*, *limit_sl_dest* and *hardlink_dupes* arguments.
      Default value of *maxlevels* was changed from ``10`` to ``sys.getrecursionlimit()``

.. function:: compile_file(fullname, ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=-1, invalidation_mode=None, *, stripdir=None, prependdir=None, limit_sl_dest=None, hardlink_dupes=False)

   Compile the file with path *fullname*. Return a true value if the file
   compiled successfully, and a false value otherwise.

   If *ddir* is given, it is prepended to the path to the file being compiled
   for use in compilation time tracebacks, and is also compiled in to the
   byte-code file, where it will be used in tracebacks and other messages in
   cases where the source file does not exist at the time the byte-code file is
   executed.

   If *rx* is given, its ``search`` method is passed the full path name to the
   file being compiled, and if it returns a true value, the file is not
   compiled and ``True`` is returned. This can be used to exclude files matching
   a regular expression, given as a :ref:`re.Pattern <re-objects>` object.

   If *quiet* is ``False`` or ``0`` (the default), the filenames and other
   information are printed to standard out. Set to ``1``, only errors are
   printed. Set to ``2``, all output is suppressed.

   If *legacy* is true, byte-code files are written to their legacy locations
   and names, which may overwrite byte-code files created by another version of
   Python.  The default is to write files to their :pep:`3147` locations and
   names, which allows byte-code files from multiple versions of Python to
   coexist.

   *optimize* specifies the optimization level for the compiler.  It is passed to
   the built-in :func:`compile` function. Accepts also a sequence of optimization
   levels which lead to multiple compilations of one :file:`.py` file in one call.

   *invalidation_mode* should be a member of the
   :class:`py_compile.PycInvalidationMode` enum and controls how the generated
   pycs are invalidated at runtime.

   The *stripdir*, *prependdir* and *limit_sl_dest* arguments correspond to
   the ``-s``, ``-p`` and ``-e`` options described above.
   They may be specified as ``str`` or :py:class:`os.PathLike`.

   If *hardlink_dupes* is true and two ``.pyc`` files with different optimization
   level have the same content, use hard links to consolidate duplicate files.

   .. versionadded:: 3.2

   .. versionchanged:: 3.5
      *quiet* parameter was changed to a multilevel value.

   .. versionchanged:: 3.5
      The *legacy* parameter only writes out ``.pyc`` files, not ``.pyo`` files
      no matter what the value of *optimize* is.

   .. versionchanged:: 3.7
      The *invalidation_mode* parameter was added.

   .. versionchanged:: 3.7.2
      The *invalidation_mode* parameter's default value is updated to None.

   .. versionchanged:: 3.9
      Added *stripdir*, *prependdir*, *limit_sl_dest* and *hardlink_dupes* arguments.

.. function:: compile_path(skip_curdir=True, maxlevels=0, force=False, quiet=0, legacy=False, optimize=-1, invalidation_mode=None)

   Byte-compile all the :file:`.py` files found along ``sys.path``. Return a
   true value if all the files compiled successfully, and a false value otherwise.

   If *skip_curdir* is true (the default), the current directory is not included
   in the search.  All other parameters are passed to the :func:`compile_dir`
   function.  Note that unlike the other compile functions, ``maxlevels``
   defaults to ``0``.

   .. versionchanged:: 3.2
      Added the *legacy* and *optimize* parameter.

   .. versionchanged:: 3.5
      *quiet* parameter was changed to a multilevel value.

   .. versionchanged:: 3.5
      The *legacy* parameter only writes out ``.pyc`` files, not ``.pyo`` files
      no matter what the value of *optimize* is.

   .. versionchanged:: 3.7
      The *invalidation_mode* parameter was added.

   .. versionchanged:: 3.7.2
      The *invalidation_mode* parameter's default value is updated to None.

To force a recompile of all the :file:`.py` files in the :file:`Lib/`
subdirectory and all its subdirectories::

   import compileall

   compileall.compile_dir('Lib/', force=True)

   # Perform same compilation, excluding files in .svn directories.
   import re
   compileall.compile_dir('Lib/', rx=re.compile(r'[/\\][.]svn'), force=True)

   # pathlib.Path objects can also be used.
   import pathlib
   compileall.compile_dir(pathlib.Path('Lib/'), force=True)

.. seealso::

   Module :mod:`py_compile`
      Byte-compile a single source file.