xref: /third_party/EGL/extensions/KHR/EGL_KHR_stream_producer_eglsurface.txt

Name

    KHR_stream_producer_eglsurface

Name Strings

    EGL_KHR_stream_producer_eglsurface

Contributors

    Acorn Pooley
    Jamie Gennis
    Marcus Lorentzon

Contacts

    Acorn Pooley, NVIDIA  (apooley 'at' nvidia.com)

Notice

    Copyright (c) 2011-2013 The Khronos Group Inc. Copyright terms at
        http://www.khronos.org/registry/speccopyright.html

Status

    Complete.
    Approved by the Khronos Board of Promoters on December 2, 2011.

Version

    Version 11, June 18, 2012

Number

    EGL Extension #34

Dependencies

    Requires EGL 1.2.
    Requires OpenGL ES 1.1 or OpenGL ES 2.0.

    Requires the EGL_KHR_stream extension.

Overview

    This extension allows an EGLSurface to be created as a producer of
    images to an EGLStream.  Each call to eglSwapBuffers posts a new
    image frame into the EGLStream.

New Procedures and Functions

    EGLSurface eglCreateStreamProducerSurfaceKHR(
                        EGLDisplay dpy,
                        EGLConfig config,
                        EGLStreamKHR stream,
                        const EGLint *attrib_list)

New Tokens

    Bit that can appear in the EGL_SURFACE_TYPE of an EGLConfig:

    EGL_STREAM_BIT_KHR                         0x0800




Add a row to "Table 3.2: Types of surfaces supported by an EGLConfig"
in the EGL spec, right after the EGL_PBUFFER_BIT row:

        EGL Token Name         Description
        --------------         --------------------------
        EGL_STREAM_BIT_KHR     EGLConfig supports streams


In the second paragraph of section "Other EGLConfig Attribute
Description" in the EGL spec, replace
        EGL_WINDOW_BIT | EGL_PIXMAP_BIT | EGL_PBUFFER_BIT
with
        EGL_WINDOW_BIT | EGL_PIXMAP_BIT | EGL_PBUFFER_BIT | EGL_STREAM_BIT_KHR
and replace
        "...cannot be used to create a pbuffer or pixmap."
with
        "...cannot be used to create a pbuffer, pixmap, or stream."


Replace section "3.10.3.1 No way to connect producer to EGLStream" in
the EGL_KHR_stream extension with this:

    3.10.3.1 Stream Surface Producer

    Call

        EGLSurface eglCreateStreamProducerSurfaceKHR(
                        EGLDisplay dpy,
                        EGLConfig config,
                        EGLStreamKHR stream,
                        const EGLint *attrib_list)

    to create an EGLSurface and connect it as the producer of
    <stream>.

    <attrib_list> specifies a list of attributes for <stream>. The
    list has the same structure as described for eglChooseConfig. The
    attributes EGL_WIDTH and EGL_HEIGHT must both be specified in the
    <attrib_list>.

    EGL_WIDTH and EGL_HEIGHT indicate the width and height
    (respectively) of the images that makes up the stream.

    The EGLSurface producer inserts an image frame into <stream> once
    for each time it is passed to eglSwapBuffers().  The image frame
    is inserted after the GL has finished previous rendering commands.
    Refer to section "3.10.5 EGLStream operation" in the
    EGL_KHR_stream extension specification for operation of the
    EGLStream when an image frame is inserted into it.

    If <stream> is not in the EGL_STREAM_STATE_EMPTY_KHR,
    EGL_STREAM_STATE_NEW_FRAME_AVAILABLE_KHR, or
    EGL_STREAM_STATE_OLD_FRAME_AVAILABLE_KHR when passed to
    eglSwapBuffers(), then eglSwapBuffers will return EGL_FALSE and
    generate an EGL_BAD_CURRENT_SURFACE error.

    If the application would like to have the results of rendering
    appear on the screen at a particular time then it must query the
    value of EGL_CONSUMER_LATENCY_USEC_KHR after calling
    eglCreateStreamProducerSurfaceKHR.  This is the estimated time that
    will elapse between the time the image frame is inserted into the
    EGLStream and the time that the image frame will appear to the
    user.

    The image frame is not inserted into the EGLStream until the GL
    has finished rendering it.  Therefore predicting exactly when the
    image frame will be inserted into the stream is nontrivial.

    If it is critical that this frame of data reach the screen at a
    particular point in time, then the application can
        - render the frame (using GL/GLES commands)
        - call glFinish (or use other synchronization techniques to
           ensure rendering has completed).
        - wait until the time that the frame should appear to the user
           MINUS the value of EGL_CONSUMER_LATENCY_USEC_KHR.
        - call eglSwapBuffers
    This will allow the image frame to be inserted into the EGLStream
    at the correct time ("Image Frame Intended Display Time" minus
    "Consumer Latency") so that it will be displayed ("Image Frame
    Actual Display Time" as close as possible to the desired time.

    However, this will cause the GPU to operate in lockstep with the
    CPU which can cause poor performance.  In most cases it will be
    more important for the image frame to appear to the user "as soon
    as possible" rather than at a specific point in time.  So in most
    cases the application can ignore the value of
    EGL_CONSUMER_LATENCY_USEC_KHR, not call glFinish, and not wait
    before calling eglSwapBuffers.

    On failure eglCreateStreamProducerSurfaceKHR returns EGL_NO_SURFACE
    and generates an error.

        - EGL_BAD_PARAMETER if EGL_WIDTH is not specified or is specified
          with a value less than 1.

        - EGL_BAD_PARAMETER if EGL_HEIGHT is not specified or is specified
          with a value less than 1.

        - EGL_BAD_STATE_KHR is generated if <stream> is not in state
          EGL_STREAM_STATE_CONNECTING_KHR.

        - EGL_BAD_MATCH is generated if <config> does not have the
          EGL_STREAM_BIT_KHR set in EGL_SURFACE_TYPE.

        - EGL_BAD_MATCH is generated if the implementation is not able to
          convert color buffers described by <config> into image frames
          that are acceptable by the consumer that is connected to
          <stream>.

        - EGL_BAD_STREAM_KHR is generated if <stream> is not a valid
          EGLStream created for <dpy>.

        - EGL_BAD_DISPLAY is generated if <dpy> is not a valid,
          initialized EGLDisplay.

Add a section preceding "3.9.3 Posting Semantics" in the EGL
specification:

    3.9.x Posting to a Stream

    To post the color buffer to an EGLStream with an EGLSurface
    producer, call

        EGLBoolean eglSwapBuffers(
                            EGLDisplay dpy,
                            EGLSurface surface);

    If <surface> is the producer of an EGLStream then the
    contents of the color buffer are inserted as a new image frame
    into the EGLStream.

    When eglSwapBuffers returns the contents of the color buffer will
    have been inserted into the EGLStream as described in section
    "3.10.5 EGLStream operation" in the EGL_KHR_stream extension
    specification, and the EGL_PRODUCER_FRAME_KHR attribute and
    EGL_STREAM_STATE_KHR attribute values will reflect this.

    The contents of the color buffer and all ancillary buffers are
    always undefined after calling eglSwapBuffers.

    eglSwapBuffers is never synchronized to a video frame when
    <surface> is the producer for an EGLStream (it is as if the
    swapinterval (set by eglSwapInterval, see below section "3.9.3
    Posting Semantics") is 0).

    It is implementation dependent whether eglSwapBuffers actually
    waits for rendering to the color buffer to complete before
    returning, but except for timing it must appear to the application
    that all rendering to the EGLSurface (e.g. all previous gl
    commands) completed before the image frame was inserted into the
    EGLStream and eglSwapBuffers returned (as described below in
    section "3.9.3 Posting Semantics").


Add to section "3.9.4 Posting Errors" in the EGL specification a new
sentence as the 2nd to last sentence in the first paragraph:

    If eglSwapBuffers is called and the EGLStream associated with
    surface is no longer valid, an EGL_BAD_STREAM_KHR error is
    generated.


Issues
    1.  How many image frame buffers should be used?

        DISCUSSION:
        - leave up to implementation?
        - leave up to producer?
        - need hints from consumer?
        - In practice 1, 2, and 3 buffers mean different semantics
          which are visible to both the producer and consumer.  Each
          may be useful.  I cannot think of a use for more than 3
          buffers for EGL_KHR_stream_surface.  (For a video producer
          more than 3 often does make sense, but that is a different
          extension.)

        One possibility: expose EGL_BUFFER_COUNT_KHR to application.

        It probably does not make sense to ever use more or less than
        3 buffers.  One that is the EGLSurface back buffer.  One that
        is waiting for the consumer to acquire.  And one that the
        consumer has acquired and is actively consuming.

        RESOLVED: remove the EGL_BUFFER_COUNT_KHR parameter and always
        use 3 buffers.  This attribute can be added back with a
        layered extension later if needed.

    2.  How is the resolution (width/height) of image frames set?

        RESOLVED: The width and height are set with the required
        EGL_WIDTH and EGL_HEIGHT attributes.  These do not change for
        the life of <stream>.

    3.  How is the image format, zbuffering, etc set?

        RESOLVED: These are all determined by the <config>.  These do
        not change for the life of <stream>.

    4.  How does eglSwapBuffers act if there are already image frames
        in the EGLStream when it is called.

        RESOLVED: Frames are inserted into the EGLStream as described
        in section "3.10.5 EGLStream operation" in the EGL_KHR_stream
        extension specification.  In particular:

            If the value of EGL_STREAM_FIFO_LENGTH_KHR is 0 or if the
            EGL_KHR_stream_fifo extension is not supported then the
            new frame replaces any frames that already exist in the
            EGLStream.  If the consumer is already consuming a frame
            then it continues to consume that same frame, but the next
            time the consumer begins to consume a frame (e.g. the
            next time eglStreamConsumerAcquireKHR() is called for a
            gltexture consumer) the newly rendered image frame will be
            consumed.  (This is the standard behavior for ANY producer
            when EGL_STREAM_FIFO_LENGTH_KHR is 0, described as "mailbox
            mode").

            If the EGL_KHR_stream_fifo extension is supported and the
            value of EGL_STREAM_FIFO_LENGTH_KHR is greater than 0 then
            the newly rendered frame will be inserted into the
            EGLStream.  If the EGLStream is full (already contains
            EGL_STREAM_FIFO_LENGTH_KHR frames) then eglSwapBuffers
            will block until there is room in the fifo.  Note that
            this can deadlock if the consumer is running in the same
            thread as the producer since the consumer will never be
            able to consume a frame if the thread is blocked waiting
            for room in the fifo.  This fifo-related behavior is
            described in the EGL_KHR_stream_fifo specification (this
            behavior is not specific to this producer; it works the
            same for all producers and all consumers).

        All rendering commands must complete before the color
        buffer is inserted into the EGLStream, or at least this is how
        the behavior must appear to the application.

        To be precise: when eglSwapBuffers returns the rendering
        commands may or may not actually be complete, but the
        following must all be true:
            - The EGL_PRODUCER_FRAME_KHR value reflects the frame that
                was just swapped by eglSwapBuffers
            - The EGL_STREAM_STATE_KHR indicates that the image frame
                is available (i.e. its value is
                EGL_STREAM_STATE_NEW_FRAME_AVAILABLE_KHR)
            - In mailbox mode if the consumer consumes a new frame it
                will get this new frame (not an older frame).  (For
                example, with a EGL_NV_stream_consumer_gltexture
                consumer, a call to eglStreamConsumerAcquireKHR() will
                latch this new frame.)
            - In fifo mode (see EGL_KHR_stream_fifo extension) if the
                consumer consumes a new frame and all previous frames
                have been consumed it will get this new frame (not an
                older frame).  (For example, with a
                EGL_NV_stream_consumer_gltexture consumer, a call to
                eglStreamConsumerAcquireKHR() will latch this new
                frame.)
            - If a consumer consumes the swapped frame, all GL (and
                other API) commands called prior to eglSwapBuffers
                will take effect on the image frame before the
                consumer consumes it.  In other words, the consumer
                will never consume a partially rendered frame.  (For
                example, with EGL_NV_stream_consumer_gltexture
                consumer, if the app does this:
                    eglSwapBuffers()               // swap the producer EGLSurface
                    eglStreamConsumerAcquireKHR()  // acquire the swapped image
                    glDrawArrays()                 // draw something using the texture
                then the texture used in the glDrawArrays() command
                will contain the image rendered by all gl (and/or
                other API) commands preceding the eglSwapBuffers call
                as if the app had called glFinish and/or eglWaitClient
                just before calling eglSwapBuffers (but note that this
                is implicit in eglSwapBuffers; the app does NOT need
                to actually call glFinish or any other synchronization
                functions in order to get this effect, and in fact
                explicitly calling glFinish and/or eglWaitClient there
                may significantly and negatively affect performance).)

Revision History

    #11 (June 18. 2012) Acorn Pooley
        - Replace EGLStream with EGLStreamKHR in function prototypes.

    #10 (June 15, 2012) Acorn Pooley
        - Fix eglCreateStreamProducerSurfaceKHR name (was missing KHR)

    #9 (October 17, 2011) Acorn Pooley
        - Clarify issue 4

    #8 (October 12, 2011) Acorn Pooley
        - remove interactions with EGL_KHR_stream_fifo extension (they
          are already decribed in that extension).

    #7 (October 11, 2011) Acorn Pooley
        - Add issue 4
        - add changes to section 3.9 of the EGL spec to clarify
          eglSwapBuffer behavior

    #6 (October 4, 2011) Acorn Pooley
        - Convert from an NV extension to a KHR extension

    #5 (September 30, 2011) Acorn Pooley
        - Remove EGL_BUFFER_COUNT_NV (0x321D) attribute and resolve issue 1.

    #4 (September 27, 2011) Acorn Pooley
        - Assign enum values (bug 8064)

    #3 (July 6, 2011) Acorn Pooley
        - Rename EGL_KHR_image_stream to EGL_KHR_stream

    #2  (June 30, 2011) Acorn Pooley
        - remove dependence on EGLImage
        - clarify overview
        - remove glossary (it can be seen in EGL_KHR_stream ext)
        - Add EGL_STREAM_BIT
        - clarify description
        - describe attribute

    #1  (April 20, 2011) Acorn Pooley
        - initial draft

# vim:ai:ts=4:sts=4:expandtab:textwidth=70