xref: /third_party/openGLES/extensions/QCOM/QCOM_framebuffer_foveated.txt

Name

    QCOM_framebuffer_foveated

Name Strings

    GL_QCOM_framebuffer_foveated

Contributors

    Skyler Saleh
    Maurice Ribble
    Tate Hornbeck
    Jonathan Wicks
    Robert VanReenen

Contact

    Jeff Leger - jleger 'at' qti.qualcomm.com

Status

    Complete

Version

    Last Modified Date: May 10, 2017
    Revision: #11

Number

    OpenGL ES Extension #273

Dependencies

    OpenGL ES 2.0 is required.  This extension is written against OpenGL ES 3.2.

Overview

    Foveated rendering is a technique that aims to reduce fragment processing
    workload and bandwidth by reducing the average resolution of a framebuffer.
    Perceived image quality is kept high by leaving the focal point of
    rendering at full resolution.

    It exists in two major forms:

        - Static foveated(lens matched) rendering: where the gaze point is
        fixed with a large fovea region and designed to match up with the lens
        characteristics.
        - Eye-tracked foveated rendering: where the gaze point is continuously
        tracked by a sensor to allow a smaller fovea region (further reducing
        average resolution)

    Traditionally foveated rendering involves breaking a framebuffer's area
    into smaller regions such as bins, tiles, viewports, or layers which are
    rendered to individually. Each of these regions has the geometry projected
    or scaled differently so that the net resolution of these layers is less
    than the original framebuffer's resolution. When these regions are mapped
    back to the original framebuffer, they create a rendered result with
    decreased quality as pixels get further from the focal point.

    Foveated rendering is currently achieved by large modifications to an
    applications render pipelines to manually implement the required geometry
    amplifications, blits, and projection changes.  This presents a large
    implementation cost to an application developer and is generally
    inefficient as it can not make use of a platforms unique hardware features
    or optimized software paths. This extension aims to address these problems
    by exposing foveated rendering in an explicit and vendor neutral way, and by
    providing an interface with minimal changes to how an application specifies
    its framebuffer.

New Tokens

    Allowed in the config input in FramebufferFoveationConfigQCOM:

        FOVEATION_ENABLE_BIT_QCOM                    0x1
        FOVEATION_SCALED_BIN_METHOD_BIT_QCOM         0x2

New Procedures and Functions

    void FramebufferFoveationConfigQCOM(uint fbo,
                                       uint numLayers,
                                       uint focalPointsPerLayer,
                                       uint requestedFeatures,
                                       uint *providedFeatures);

    void FramebufferFoveationParametersQCOM(uint fbo,
                                           uint layer,
                                           uint focalPoint,
                                           float focalX,
                                           float focalY,
                                           float gainX,
                                           float gainY,
                                           float foveaArea);

Additions to Chapter 9 of the OpenGL ES 3.2 Specification

    The command

    void FramebufferFoveationConfigQCOM( uint fbo, uint numLayers,
    uint focalPointsPerLayer, uint requestedFeatures, uint *providedFeatures);

    is used to configure foveated rendering for the framebuffer object 'fbo'
    and to instruct the implementation to allocate any additional
    intermediate resources needed for foveated rendering.

    In order to enable foveation, this call must be issued prior to any
    operation which causes data to be written to a framebuffer attachment.
    Once this call is made for a framebuffer object, the fbo will remain a
    "foveated fbo". The following scenarios are unsupported conditions:

        1. Rendering to a non foveated fbo, then calling
           FramebufferFoveationConfigQCOM results in current framebuffer content
           becoming undefined.

        2. Rendering to a foveated fbo then switching attachments results in an
           error.

    Each layer of a foveated framebuffer, the max of which is specified by
    'numLayers', can have multiple focal points as controlled by
    'focalPointsPerLayer'. This enables applications that make use of double
    wide rendering to utilize foveated rendering and also allows more complex
    falloff characteristics to be modeled with multiple overlapping focal
    points. There are limitations to the number of focal points that an
    implementation can support with different enabled features or framebuffer
    formats. When an implementation can not support having as many focal points
    per layer as was specified in this function, it will fail with the error
    INVALID_VALUE. It is recommended that an application utilize as
    few focal points per layer as possible.

    The 'requestedFeatures' bitfield is used to specify which features an
    application would like to use.

    An explanation of each of the features is below:

        FOVEATION_ENABLE_BIT_QCOM: Is used to enable foveated rendering, if
        this bit is not specified foveated rendering will not be used.

        FOVEATION_SCALED_BIN_METHOD_BIT_QCOM: Requests that the implementation
        perform foveated rendering by dividing the framebuffer into a grid of
        subregions. Each subregions will be greater than or equal to one pixel
        and less than or equal to the full framebuffer. Then rendering the geometry
        to each of these regions with a different projection or scale. Then, finally
        upscaling the subregion to the native full resolution framebuffer.
        Quality in the scaled bin method is defined as a minimum pixel density
        which is the ratio of the resolution rendered compared to the native
        framebuffer.

    In the future it is expected that more features will be added, without
    breaking backwards compatibility.

    'providedFeatures' is a pointer to a uint that will be set to a new bitfield
    that tells the application which features the implementation provided for the
    current foveation configuration, in the same format as used in the 'requested
    Features' bitfield. This may include more or fewer features than the application
    requested.  The FOVEATION_ENABLE_BIT_QCOM will not be included if the
    implementation has to fallback to non-foveated rendering.

    If an application tries to make use of a feature that is not included in the
    'providedFeatures' bitfield, the results of the operation are implementation
    defined, but should not yield application termination.

    If this command is called with 'requestedFeatures' equal to zero, then the value
    of 'providedFeatures' will have FOVEATION_ENABLE_BIT_QCOM unset, and the other
    bits will be set or unset to indicate which foveation features are supported
    by the implementation.

    The command

    void FramebufferFoveationParametersQCOM(uint fbo,uint layer, uint focalPoint,
    float focalX, float focalY, float gainX, float gainY, float foveaArea);

    is used to control the falloff of the foveated rendering of 'focalPoint'
    for layer  'layer' in the framebuffer object 'fbo'. Multiple focal points
    per layer are provided to enable foveated rendering when different regions
    of a framebuffer represent different views, such as with double wide
    rendering. Values of 0 to the framebuffers focalPointsPerLayer-1 are valid
    for the 'focalPoint' input and specify which focal point's data to update
    for the layer.

    'focalX' and 'focalY' is used to specify the x and y coordinate
    of the focal point of the foveated framebuffer in normalized device
    coordinates. 'gainX' and 'gainY' are used to control how quickly the
    quality falls off as you get further away from the focal point in each
    axis. The larger these values are the faster the quality degrades.
    'foveaArea' is used to control the minimum size of the fovea region, the
    area before the quality starts to fall off. These parameters should be
    modified to match the lens characteristics.

    For the scaled bin method, these parameters define the minimum pixel
    density allowed for a given focal point at the location (px,py) on a
    framebuffer layer in NDC as:

    min_pixel_density=0.;
    for(int i=0;i<focalPointsPerLayer;++i){
        focal_point_density = 1./max((focalX[i]-px)^2*gainX[i]^2+
                            (focalY[i]-py)^2*gainY[i]^2-foveaArea[i],1.);
        min_pixel_density=max(min_pixel_density,focal_point_density);
    }

    While this function is continuous, it is worth noting that an
    implementation is allowed to decimate to a fixed number of supported
    quality levels, and it is allowed to group pixels into larger regions of
    constant quality level, as long as the implementation at least provides
    the quality level given in the above equation.

    Future supported foveation methods could have different definitions of
    quality.

    The default values for each of the focal points in a layer is:

    focalX=focalY=0;
    gainX=gainY=0;
    foveaArea=0;

    Which requires the entire framebuffer to be rendered at full quality.

    By specifying these constraints an application can fully constrain its
    render quality while leaving the implementation enough flexibility to
    render efficiently.

Errors

    OUT_OF_MEMORY is generated by FramebufferFoveationConfigQCOM if an
    implementation runs out of memory when trying to reserve the needed
    additional resources for the foveated framebuffer.

    INVALID_VALUE is generated by FramebufferFoveationConfigQCOM if 'fbo' is
    not a valid framebuffer.

    INVALID_VALUE is generated by FramebufferFoveationConfigQCOM if 'numLayers'
    is greater than GL_MAX_ARRAY_TEXTURE_LAYERS - 1.

    INVALID_VALUE is generated by FramebufferFoveationConfigQCOM if
    'numFocalPoints' is greater than implementation can support.

    INVALID_OPERATION is generated by FramebufferFoveationConfigQCOM if it is
    called for a fbo that has already been cofigured for foveated rendering.

    INVALID_VALUE is generated by FramebufferFoveationParametersQCOM if 'fbo'
    is not a valid framebuffer.

    INVALID_OPERATION is generated by FramebufferFoveationParametersQCOM if
    'fbo' has not been configured for foveated rendering.

    INVALID_VALUE is generated by FramebufferFoveationParametersQCOM if
    'layer' is greater than or equal to the numLayers that the fbo was
    previously configured for in FramebufferFoveationConfigQCOM.

    INVALID_VALUE is generated by FramebufferFoveationParametersQCOM if
    'numFocalPoints' is greater than implementation can support.

    INVALID_OPERATION is generated by any API call which causes a framebuffer
    attachment to be written to if the framebuffer attachments have changed for
    a foveated fbo.

    INVALID_OPERATION is generated if a rendering command is issued and the
    current bound program uses tessellation or geometry shaders.

Issues

    1. Are layered framebuffers supported?

    Layered framebuffers are supported to enable stereoscopic foveated
    rendering which is a main use case of this extension. When a layered
    framebuffer is used each layer has its own set of foveation parameters.

    2. How is foveation performed?

    How foveated rendering is performed to the framebuffer is implementation
    defined. However, if 'FOVEATION_SCALED_BIN_METHOD_BIT_QCOM' is set the
    implementation must perform foveation by dividing the framebuffer into a
    grid of subregions. Then rendering the geometry to each of these regions
    with a different projection or scale. And finally upscaling the subregion
    to the native full resolution framebuffer.

    When that bit is not set the implementation can use any algorithm it
    wants for foveated rendering as long as it meets the application
    requested features.

    3. How are MRTs handled?

    Every framebuffer attachment uses the same quality in a given region.

    4. How does gl_FragCoord work?

    When using the scaled bin method, gl_FragCoord will be scaled to match
    the relative location in a foveated framebuffer. This means the absolute
    value of gl_FragCoord will not be correct in lower resolution areas,
    but the value relative to the full resolution will be consistent.

    5. How is depth handled?

    Depth surfaces can be used during foveated rendering, but the contents
    of the depth buffer will not be resolved out. The implementation can
    do an implicit discard of the depth buffer. The reasoning here is that
    the upsample of depth during resolve would produce irrelevant results.

    6. How does unresolving from a foveated framebuffer work?

    Loading from a foveated framebuffer is undefined, so the app must be
    sure not to trigger mid frame flushes. In the dynamic foveation case
    the focal point can move constantly. If a region A of a frame 0 was
    rendered at a lower quality because the focal point was far away,
    then the focal point moved to cover region A during frame 1, the
    unresolve could not reconstruct the full quality region A. The
    app must be careful to fully clear the surface and remove mid frame
    flushes to prevent unresolves.

Examples:

    (1) Initialize a foveated framebuffer

        // Allocate and initialize a regular framebuffer and attachments
        GLuint fbo = createFramebufferAndAttachments();
        GLuint providedFeatures;
        glFramebufferFoveationConfigQCOM(fbo,1,1, GL_FOVEATION_ENABLE_BIT_QCOM, &providedFeatures);
        if(!(providedFeatures & GL_FOVEATION_ENABLE_BIT_QCOM)) {
            // Failed to enable foveation
        }

    (2) Setup static foveated rendering

        // Insert code from #1
        GLfloat focalX=0.f, focalY=0.f;  // Setup focal point at the center of screen
        GLfloat gainX=4.f, gainY=4.f;  // Increase these for stronger foveation
        glFramebufferFoveationParametersQCOM(fbo, 0, 0, focalX, focalY, gainX, gainY, 0.f);

    (3) Change eye position for eye tracked foveated rendering

        // Code called whenever the eye position changes
        // It is best to position this call both before rendering anything to
        //   a fbo and right before Flush or changing FBO since some
        //   some implementations can apply this state late by patching command
        //   buffers.
        glFramebufferFoveationParametersQCOM(fbo, 0, 0, focalX, focalY, gainX, gainY, 0.f);

    (4) Setting parameters for a multiview stereo framebuffer

        //focalPointsPerLayer should be 1
        float focalX1=0.f,focalY1=0.f;  // Gaze of left eye
        float focalX2=0.f,focalY2=0.f;  // Gaze of right eye
        float gain_x=10.f,gain_y=10.f;  // Strong foveation
        glFramebufferFoveationParametersQCOM(fbo, 0, 0, focalX1, focalY1,gainX, gainY, 0.f);
        glFramebufferFoveationParametersQCOM(fbo, 1, 0, focalX2, focalY2,gainX, gainY, 0.f);

    (5) Setting parameters for a double wide stereo framebuffer

        //focalPointsPerLayer should be 2
        float focalX1=0.f,focalY1=0.f;  // Gaze of left eye
        float focalX2=0.f,focalY2=0.f;  // Gaze of right eye
        float gainX=10.f,gainY=10.f;
        glFramebufferFoveationParametersQCOM(fbo, 0, 0, focalX1*0.5f-0.5f, focalY1, gainX*2.f ,gainY, 0.f);
        glFramebufferFoveationParametersQCOM(fbo, 0, 1, focalX2*0.5f+0.5f, focalY2, gainX*2.f ,gainY, 0.f);


Revision History

    Rev.    Date     Author    Changes
    ----  --------  --------  ----------------------------------------------
     1    05/19/16  ssaleh    Initial draft.
     2    05/27/16  ssaleh    Made the extension much more explicit.
     3    07/08/16  ssaleh    Further refinements.
     4    08/11/16  ssaleh    Specified bitfield values
     5    08/19/16  ssaleh    Added support for double wide rendering
     6    08/24/16  mribble   Name changes and cleanup
     7    08/24/16  ssaleh    Add examples
     8    10/14/16  tateh     Clarify gl_FragCoord
     9    01/04/17  tateh     Update entry points and cleanup
    10    02/17/17  jleger    Convert from EXT to QCOM extension.
    11    05/10/17  tateh     Minor cleanup