xref: /third_party/openGLES/extensions/EXT/EXT_YUV_target.txt

Name

    EXT_YUV_target

Name Strings

    GL_EXT_YUV_target

Contributors

    Amit Bansal
    Arshad Bebal
    Jeff Leger
    Jing Zhou
    Maurice Ribble
    Prabindh Sundareson
    Jan-Harald Fredriksen
    Daniel Koch
    Jamie Gennis
    Mark Callow
    Pat Brown
    Andrew Garrard
    Etay Meiri
    Weifeng Zhang

Contacts

    Jeff Leger  <jleger@qti.qualcomm.com>

Status

    Complete.

Version

    April 6, 2018 (version 17)

Number

    OpenGL ES Extension #222

Dependencies

    Requires OpenGL ES 3.0 and ESSL 3.0.
    Requires EGL 1.4 and following EGL extension:

    EGL_KHR_image_base
    OES_EGL_image_external

    This extension is written based on the wording of the OpenGL ES 3.0
    Specification and OpenGL ES Shading Language 3.0.

    This extension interacts with OpenGL ES 3.1. See issue 1 and 13 for
    more details.

    This extension interacts with OES_tessellation_shader &
    OES_geometry shader see first issue for more details.

    This extension borrows texture function names from the
    OES_EGL_image_external_essl3. This extension is compatible with
    OES_EGL_image_external_essl3 and can coexist but does not require it.

    This extension interacts with EXT_shader_framebuffer_fetch. see
    issue 7 for more details.

Overview

    This extension adds support for three new YUV related items: first
    rendering to YUV images, second sampling from YUV images while keeping the
    data in YUV space, third it defines a new built in function that does
    conversion from RGB to YUV with controls to choose ITU-R BT.601-7,
    ITU-R BT.601-7 Full range (JFIF images), or ITU-R BT.709-5 standard.

    This new functionality is layered on top of the OES_EGL_image_external
    extension.

    To perform the YUV rendering capability in this extension an application
    will attach a texture to the framebuffer object as the color attachment.
    If the texture has a target type of TEXTURE_EXTERNAL_OES with YUV color
    format then the GL driver can use this framebuffer object as the render
    target, TEXTURE_EXTERNAL_OES target with RGB color format are not allowed
    with this extension.

New Types & Keywords

    A new OpenGL GLSL extension flag is added:

        #extension GL_EXT_YUV_target

    When the above GLSL extension is specified, one new sampler type
    will be available for sampling the 2D texture:

        __samplerExternal2DY2YEXT

    The "__samplerExternal2DY2YEXT" is used to sample a YUV texture image and
    output color value without any color conversion.

    Whenever a YUV sample is output from the sampler, the format of the YUV
    will be as if YUV 4:4:4 format is output. This also means that the Y
    sample maps to component R, the U sample maps to component G, the V sample
    maps to component B, and the component A will be 1.0f.

    The RGB sample output will be the same as in OpenGL ES specification.

    Here is one example:

        uniform __samplerExternal2DY2YEXT   u_sTexture;

    When the above GLSL extension is specified, one new additional type will be
    available to specify color space standard formula for yuv to rgb transformation.

        yuvCscStandardEXT

    The value of new type can be specified using one of three new keywords
    itu_601, itu_601_full_range and itu_709.

    For example:

    yuvCscStandardEXT conv_standard = itu_601;
    yuvCscStandardEXT conv_standard = itu_601_full_range;
    yuvCscStandardEXT conv_standard = itu_709;

    Keyword itu_601 means color conversion is required using the formula
    specified in the ITU-R BT.601. Similarly keyword itu_601_full_range
    means conversion is done using ITU BT.601 full range formula as specified
    in JFIF standard while keyword itu_709 specify ITU-R BT.709 standard. Note
    that new yuvCscStandardEXT type can't be use for any other purpose apart
    from specifying color standard value to newly introduced color conversion
    built in functions mentioned below.

New Built-in function

    When the new GLSL extension is specified, two new built in functions
    will be available for rgb to yuv or yuv to rgb color space conversion.

        vec3 rgb_2_yuv(vec3 color, yuvCscStandardEXT conv_standard);

    The function rgb_2_yuv will apply rgb to yuv color conversion
    transformation on "color" value using the formula specified as per new type
    yuvCscStandardEXT variable. The first input parameter supposed to specify
    rgb value using x, y & z channels of a vec3 variable, correspondingly
    return value of this function will have transformed y, u and v value in its
    x, y and z channel. Precision of the input color will define the precision
    used for color space conversion and for output yuv color value.

        vec3 yuv_2_rgb (vec3 color, yuvCscStandardEXT conv_standard);

    The function yuv_2_rgb will apply yuv to rgb color conversion
    transformation on "color" value using the formula specified as per new type
    yuvCscStandardEXT variable. The first input parameter supposed to specify
    yuv value using x, y & z channels of a vec3 variable, correspondingly
    return value of this function will have transformed r, g and b value in its
    x, y and z channel. Precision of the input color will define the precision
    used for color space conversion and for output yuv color value.

New layout qualifier

    A shader which produces yuv format color output must qualify the fragment
    shader output variable with new yuv layout qualifier as described below.

    layout (yuv) out vec4 color;

    The new yuv layout qualifier can't be combined with any other layout qualifier,
    can only be used with fragment shader outputs and would be available only when
    the new GLSL extension is specified. Additionally if the shader qualifies
    fragment shader output with the new yuv qualifier and write depth or multiple
    color output, it would cause compilation failure.

New Tokens

    Returned in the <type> parameter of GetActiveUniform:

        SAMPLER_EXTERNAL_2D_Y2Y_EXT                         0x8BE7

Reuse tokens as in extension "OES_EGL_image_external."

    Accepted as a target in the <target> parameter of BindTexture,
    EGLImageTargetTexture2DOES, and FramebufferTexture2D:

        TEXTURE_EXTERNAL_OES                                0x8D65

    Accepted as <value> in GetIntegerv() and GetFloatv() queries:

        TEXTURE_BINDING_EXTERNAL_OES                        0x8D67

    Accepted as <value> in GetTexParameter*() queries:

        REQUIRED_TEXTURE_IMAGE_UNITS_OES                    0x8D68

Addition to Chapter 3 of the OpenGL ES 3.0 specification

    Add a new section: "3.8.18 External Textures," which follows section
    "3.8.17 Shader Exponent Texture Color Conversion." The wording will
    be the same as in extension "OES_EGL_image_external."

Changes to Section 4.4.2 "Attaching Images to Framebuffer Objects"

    At the end of subsection "Attaching Texture Images to a Framebuffer," add
    the following paragraphs:

      from
        "If texture is not zero, then texture must either name an existing
        two dimensional texture object and textarget must be TEXTURE_2D or
        texture must name an existing cube map...."

      to
        "If texture is not zero, then texture must either name an existing
        two dimensional texture object and textarget must be TEXTURE_2D or
        TEXTURE_EXTERNAL_OES or texture must name an existing cube map...."

      from
        "If textarget is TEXTURE_2D, level must be greater than or equal to
        zero and no larger than log2 of the value of MAX_TEXTURE_SIZE.
        Otherwise, an INVALID_VALUE error is generated.
          The command..."
      to
        "If textarget is TEXTURE_2D, level must be greater than or equal to
        zero and no larger than log2 of the value of MAX_TEXTURE_SIZE.
        Otherwise, an INVALID_VALUE error is generated. If textarget is
        TEXTURE_EXTERNAL_OES and attachment is other than COLOR_ATTACHMENT0,
        an INVALID_OPERATION error is generated.
          The command..."

     Add an additional sentence at the end of above paragraph:

        "Since TEXTURE_EXTERNAL_OES texture target can only be specified
        using an EGLImage, yuv formats can be rendered by attaching
        TEXTURE_EXTERNAL_OES textarget via EGLImage."

Changes to Section 4.2.3 "Clearing the Buffers"
        from
        "Unsigned normalized fixed-point RGBA color buffers are cleared to color
        values derived by clamping each component of the clear color to the range
        [0; 1], then converting the (possibly sRGB converted and/or dithered) color
        to fixed-point using equations 2.3 or 2.4, respectively. The result of
        clearing integer color buffers with Clear is undefined."

        to
        "Unsigned normalized fixed-point RGBA color buffers are cleared to color
        values derived by clamping each component of the clear color to the range
        [0; 1], then converting the (possibly sRGB converted and/or dithered) color
        to fixed-point using equations 2.3 or 2.4, respectively. When clearing YUV
        Color Buffers, clear color should be defined in yuv color space and so
        floating point r, g, and b value will be mapped to corresponding y, u and v
        value and alpha channel will be ignored. The result of clearing integer
        color buffers with Clear is undefined."

Changes to Section 4.4.3 "Fine Control of Buffer Updates"

        from
        "ColorMask is used to mask the writing of R, G, B and A values to all active
         draw buffers. r, g, b, and a indicate whether R, G, B, or A values, respectively,
         are written or not (a value of TRUE means that the corresponding value is written).
         In the initial state, all color values are enabled for writing for all draw
         buffers..."

        to
        "ColorMask is used to mask the writing of R, G, B and A values to all active
         draw buffers. r, g, b, and a indicate whether R, G, B, or A values, respectively,
         are written or not (a value of TRUE means that the corresponding value is written)
         except when the color buffer is in YUV color space, in that case setting any one
         of the r, g, b, and a values as false would generate draw time INVALID_OPERATION error.
         In the initial state, all color values are enabled for writing for all draw
         buffers..."

Changes to Section 4.1.7 "Blending"

        from
        "Blending applies only if the color buffer has a fixed-point format. If the color
         buffer has an integer format, proceed to the next operation."

        to
        "Blending applies only if the color buffer has a fixed-point format and non YUV space
         format color buffer. If the color buffer has an integer, proceed to the next
         operation. If color buffer format is  YUV color space, it will generate
         draw time INVALID_OPERATION error."

Changes to 4.2.1 "Selecting Buffers for Writing"
        from
        "If some, but not all user-defined output variables are written, the values of
         fragment colors corresponding to unwritten variables are similarly undefined.
         The order of writes..."

        to
        "If some, but not all user-defined output variables are written, the values of
         fragment colors corresponding to unwritten variables are similarly undefined unless
         any of the fragment output is qualified with yuv format qualifier. In later case
         compiler would restrict usage of yuv format qualifier up to a single color output.
         The order of writes..."

Changes to section 4.4.4 ("Framebuffer Completeness")
        from
        "An internal format is color-renderable if it is one of the formats from table
         3.12 noted as color-renderable or if it is unsized format RGBA or RGB. No
         other formats,"

        to
        "An internal format is color-renderable if it is one of the formats from table
         3.12 noted as color-renderable or if it is unsized format RGBA or RGB or if it is
         YUV color space format supported by implementation. No other formats,"

Changes to section 3.7 ("Keywords") of the OpenGL ES Shading Language
specification

    - Add to the list of keywords:

        __samplerExternal2DY2YEXT
        yuvCscStandardExt 
        itu_601 
        itu_601_full_range 
        itu_709
        
        

Changes to section 4.1 ("Basic Types") of the OpenGL ES Shading Language
specification
    - Add to the list of Transparent types:
        "yuvCscStandardExt    a value type to specify color space standard formula for 
        transformation between yuv and rgb, taking values of itu_601, itu_601_full_range, 
        itu_709"

    - Add to the list of Floating Point Sampler Types (opaque):

        "__samplerExternal2DY2YEXT   a handle for accessing an external 2D texture
        whose underneath format is YUV"

    - Add a paragraph at the end of the section:

        "The __samplerExternal2DY2YEXT and yuvCscStandardExt types are optional and must 
        be enabled by

         #extension GL_EXT_YUV_target : enable
         
         No type constructor or type conversion is allowed on yuvCscStandardExt. A value
         of itu_601, itu_601_full_range, itu_709, or a variable typed as yuvCscStandardExt 
         can be used as an argument to user-defined functions and the built-in functions yuv_2_rgb and 
         rgb_2_yuv. Other than the assignment operator, any other operations on this type 
         of values or variables will be treated as a compilation error."

Changes to section 4.5.4 ("Default Precision Qualifiers") of the OpenGL ES
Shading Language specification

    - Add to the list of "vertex language has the following predeclared
    globally scoped default precision statements:"

        "precision lowp __samplerExternal2DY2YEXT;"

    - Add to the list of "fragment language has the following predeclared
    globally scoped default precision statements:"

        "precision lowp __samplerExternal2DY2YEXT;"

Changes to section 8.8 ("Texture Lookup Functions") of the OpenGL ES Shading
Language specification

    - Add to the table the following sampler functions:

      "highp ivec2 textureSize(__samplerExternal2DY2YEXT sampler, int lod)
       vec4 texture(__samplerExternal2DY2YEXT sampler, vec2 P [, float bias] )
       vec4 textureProj(__samplerExternal2DY2YEXT sampler, vec3 P [, float bias] )
       vec4 textureProj(__samplerExternal2DY2YEXT sampler, vec4 P [, float bias])
       vec4 texelFetch(__samplerExternal2DY2YEXT sampler, ivec2 P, int lod)"

Add a new section 8.10 "Conversion Functions between YUV and RGB"

    - Add a new table containing 2 entries:
        
        vec3 rgb_2_yuv(vec3 color, yuvCscStandardEXT conv_standard);
        apply rgb to yuv color conversion transformation on "color" value 
        using the formula specified by conv_standard. 

        vec3 yuv_2_rgb(vec3 color, yuvCscStandardEXT conv_standard);
        apply yuv to rgb color conversion transformation on "color" value
        using the formula specified by conv_standard.
    
Compatibility with OpenGL ES 3.1 and ESSL 3.10

    This extension is expected to work without modification with OpenGL ES 3.1
    and  ESSL 3.10. Some further clarification in issue 1, 9 and 13.

Issues

    1. Do we want to explicitly disallow "__samplerExternal2DY2YEXT" from use in other
    shader apart from fragment shader e.g. vertex, compute, geometry and tessellation
    shaders?

    RESOLVED: No, modern unified shader hardware supports YUV texture sampling in all
    of the shader stages, we should keep this enable irrespective of shader stage.

    2. What happens if the input source isn't in the expected color space for built in
    functions or lets say there is cross conversion between different color space
    conversion standards e.g. 601 RGB to 709 YUV etc or yuv layout qualified color
    output is actually written with in rgb color space?

    RESOLVED: Implementation can't determine how the input pixel has been
    manipulated before producing the shader outputs, so results would be
    undefined in this case.

    3. What happens if the source formats aren't the expected format for sampler?

    RESOLVED: Non-YUV format will generate a draw time INVALID_OPERATION error.

    4. Is the depth or stencil behaviour changed when rendering to yuv surfaces.

    RESOLVED: The depth and stencil buffers will work the same when rendering
    to RGB or YUV surfaces, however when shader output is qualified with yuv
    layout qualifier, writing out gl_FragDepth would cause compilation failure.

    5. Should this extension include support for renderbuffers or just
    renderable textures?

    RESOLVED: Renderbuffers doesn't add any functionality so it will not be
    included.

    6. Is BlitFramebuffer or CopyTex[Sub]Image supported with YUV renderable
    surfaces?

    RESOLVED: No. There is a lot of driver complexity in supporting and
    testing case. Using these calls with a YUV source or destination
    will cause an INVALID_OPERATION error.

    7. Does EXT_shader_framebuffer_fetch work with YUV renderable surfaces?

    RESOLVED: Yes. The next question is the result before or after
    sub-sampling?  To give maximum flexibility to hardware either of these
    options is acceptable.

    8. Should we add support for BT.2020?

    RESOLVED: Currently No, If require BT.2020 functionality can be achieved
    using __samplerExternal2DY2YEXT and user defined matrix. To keep the spec
    simple, we just want to have only three currently.

    9. Should sampling return samples converted in to linear space or raw
    samples?

    RESOLVED: This extension will return raw samples. If the YUV data is
    encoded in a non-linear space, as is common, the returned samples will be
    in the non-linear space. If the data is encoded linearly, the returned
    samples will be linear.

    10. Are there specific requirements on how a lower resolution UV plane must
    be sampled during rendering?

    RESOLVED: To allow maximum flexibility in hardware at this time this
    extension does not define how this sampling must be performed. Even if we
    decide to define this behaviour in future, it would be more appropriate to
    do it via a separate EGL extension.

    11. How goes gl_FragCoord work with a lower resolution UV plane?

    RESOLVED: It is expected to work as if that plane was rendered at full
    resolution.  The exception to this is talked about in issue #11 where
    there is some flexibility in subpixel sampling.

    12. Should we add YUV format support for image load store in this extension or
    is it valid to use new yuv format layout qualifier with image load store ?

    RESOLVED: No, Since that would require major changes, we should do this with
    a separate extension, currently using new yuv format qualifier with image load
    store would cause compilation failure.

    13. How can we defines color standard matrix other than provided via built-in
    functions e.g. full range BT.709 or adding compensation for the different colour
    primaries of the different standards ?

    RESOLVED: This extension defines built in utility functions for some of the
    common color conversion standard, however its no way limit an app to certain
    standard of output data. App can define own color matrix for such purpose
    and use with either __samplerExternal2DY2YEXT or sampler2D.

    14. What happens if shader produce a color output qualified with yuv layout
    qualifier and current draw FBO format isn't YUV or draw FBO format is in
    YUV color space but shader output isn't qualified with yuv layout qualifier.

    RESOLVED: In order to write YUV color output, shader color output must be
    qualified with yuv layout qualifier and draw FBO format needs to be in YUV
    color space otherwise lack of any of these would cause INVALID_OPERATION
    error.

    15. What happens if shader produce yuv color output which is out of range
    as per definition of color space format of rendering surface e.g. full range
    output when the output color space is limited or similarly input to one of
    built in color conversion functions or to the new __samplerExternal2DY2YEXT
    is out of range.

    RESOLVED: If shader yuv color output or input values to one of the built in
    color conversion functions or to the new __samplerExternal2DY2YEXT are not
    within the specified range of the format, results would be undefined values.

   16. How ReadPixels operate for YUV render target?

   RESOLVED: ReadPixels will convert YUV to RGBA under the current spec
   requirement to support format RGBA and type UNSIGNED_BYTE combination  for
   the normalized fixed point surfaces depending upon format and color space
   standard of currently bound rendering surface.

Known Quality Issues

    Since YUV is a lossy compressed format and UV values represent sampling at
    even locations, there may be minor quality issues along the primitive edges
    or for scissor when its X or Y set to an odd value. In case of YUV
    primitive rendering if geometry is animated, there may be more artifacts as
    edges might flicker and sparkle frame to frame.  Another way to explain
    this corruption is that sometimes the different planes are stored at
    different resolutions so the different interpolation for different planes
    causes these artifacts.

    Due to nature of CbCr subsampling, there might be artifacts along the primitive
    edges if there are  partially covered pixels. Since commonly used 4:2:0 YUV
    formats writes out 1 pixel for a block of 2x2 pixels, depending upon how
    subsampled pixel output is calculated, there may always be artifacts around the
    primitive edges for partially covered pixels e.g. implementation might always
    decide to choose CbCr subsample cosited with top left pixel 2x2 Y plane block
    based on this single pixel coverage. This could help to avoid artifacts in more
    key cases e.g. along the diagonal of quad when its drawn using triangle strip
    but it may cause artifacts in other cases when we have partially covered
    pixels around the primitive edges visible after using very high zoom factor.

Dependencies on EGL_KHR_image, EGL_KHR_image_base, and EGL 1.4

    If EGL 1.2 is not supported, or if neither the EGL_OES_image nor
    EGL_OES_image_base extensions is supported, all discussion of
    EGLImages should be ignored, and any calls to either
    EGLImageTargetTexture2DOES or FramebufferTexture2D with
    TEXTURE_EXTERNAL_OES textarget should generate the error
    INVALID_OPERATION.

Dependencies on OES_EGL_image_external

    This extension is built on top of the OES_EGL_image_external extension.
    It can coexist with the OES_EGL_image_external extension.

Revision History
    #17 (April 6, 2018, H1Gdev)
        - Fix typo.
    #16 (June 8, 2016, Jeff Leger)
        - Add yuvCscStandardExt itu_* to the list of keywords
        - Add yuvCscStandardExt as a basic type and clarify that this type
          can only be used in assignments or as a function argument.
        - Add rgb_2_yuv and yuv_2_rgb as built-in functions
          (Bug 15704).

    #15 (April 23, 2015, Jon Leech)
        - Fix typo EGLImageTargetTexImage2DOES -> EGLImageTargetTexture2DOES
          (Bug 8114).

    #14 (February 10, 2015, Amit Bansal)
        - Removed YUV_EXT and Y_EXT tokens hard requirement for ReadPixels,
          current plan is to propose it later as separate extension once we
          have more corresponding compelling use cases.
        - Adding interaction wording with EXT_shader_framebuffer_fetch
        - Removed duplicated image load store issue.
        - Further clarification about usage of yuv layout qualifier

    #13 (January 23, 2015, Amit Bansal)
        - added texture built in functions from OES_EGL_image_external_essl3
        - Added explicit clarification in overview section about RGB external images
        - Interaction with geometry & tessellation shaders w.r.t. issue 1

    #12 (December 12, 2014, Amit Bansal)
        - Added new yuv qualifiers and corresponding draw time checks
          requirement issues
        - Clarification regarding yuvCscStandardEXT type usage
        - Output clamping requirement issue
        - New YUV_EXT and Y_EXT tokens for ReadPixels
        - Changed blending & color mask behaviour to draw time error.
        - Some language clean up

    #11 (September 30, 2014, Amit Bansal)
        - Changed couple of issues to spec language
        - Reworded some issues after recent spec updates

    #10 (August 12, 2013, Maurice Ribble)
       - Changed from QCOM to EXT extension
       - Reworded issue 16 based on Mark Callow's suggestion
       - Updated names of SAMPLER_EXTERNAL_2D_Y2Y_EXT,
         __samplerExternal2DY2YEXT, and yuvCscStandardEXT

    #9  (August 6, 2013, Maurice Ribble)
       - Added issues 15-18

    #8  (July 12, 2013, Maurice Ribble)
       - Some language cleanup
       - Changed some edge case errors to be more consistent
       - Updates based on feedback from other khronos members
       - Added issues 10-14

    #7  (June 27, 2013, Maurice Ribble)
       - Some language cleanup

    #6  (May 16, 2013, Amit Bansal)
       - Correcting spacing
       - Modified overview to keep only one sampler and add new built-in
       - Defined new types for built in and removed previous layouts
       - Removed the separate GL functions for external attachment
       - Modified description of sections meant for spec
       - Modified issues to reflect layout and built in function update
       - Changed proposed to resolved for finalized issues
       - Move precision issue to the description of built in function

    #5  (March 14, 2013, Jing zhou)
       - Minor rewording on same of proposals for the Issues
       - Assign new token IDs from reserved QCOM ID block

    #4  (March 4, 2013, Maurice Ribble)
       - First round of cleanup and todos

    #3  (February 19, 2013, Amit Bansal)
       - Modified glReadPixels()issue section

    #2  (January 9, 2013, Amit Bansal)
       - Changed the OES suffix to QCOM as extension is still pending to be
         approved.
       - Removed EGLImageTargetTexture2DOES from New Procedures and Functions
         category
       - Corrected error in FramebufferRenderbuffer2DExternalQCOM name at line
         167.
       - Modified description of FramebufferTexture2DExternalQCOM to reflect
         argument changes correctly
       - Corrected typo extension to extension
       - Finished the documentation of final proposal for issue 2 & 3.
       - Added a new Known Quality Issue section and text corresponding to it

    #1 - (Oct 1, 2012) Original Draft