neuray Services API Programmer's Manual

Render Loops

The traditional rendering mechanism used in neuray Services introduces a degree of latency that severely impacts image convergence time when progressive rendering. Typically an application will render an image, download and display it and then make another request to render the next, more refined, image. The overhead of image download, and making another request is significant and increases the time it takes for convergence. In addition, render resources are idle during this period. Render loops remove this latency and provide more efficient use of resources and faster convergence.

A render loop is a long lived service which continuously renders the scene in a tight loop rather than relying on the client to trigger each individual render. Images are obtained by polling the loop to retrieve images (and progress information) as required. When the image converges the loop automatically pauses rendering (freeing up resources for other users) until it is marked as dirty to tell it that changes have been made. A render loop will run until it is explicitly stopped or if the client has not made contact with it within a timeout period. This means that a render loop does not have to be specifically shutdown to release the resources, however it will continue to take up resources until it expires.

Implementation

The render loop system is made up of four components:
  • The render loop manager IRender_loop_manager is a neuray library API component and handles all the creation, destruction and manipulation of render loops. Additionally, user implemented render handlers are registered here and then made available for use by render loops.
  • A render loop IRender_loop represents an actual render loop. These are identified by name and created by the render loop manager by providing the render loop handler to use. Once started the loop runs in a separate thread calling the handlers' render method in a tight loop to perform actual rendering. Render loops can pause on request and when rendering converges then be started again when marked dirty.
  • Render loop handlers IRender_loop_handler performs the actual rendering for a render loop. These are user implemented classes which can perform a number of additional tasks depending on the capabilities of the handler. neuray Services comes with two render loop handlers, default and falsecolor.
  • Render loop handler factories IRender_loop_handler_factory are responsible for creating instances of specific render loop handlers for a render loop. These are registered with the render loop manager to make handlers available for use.
  • Render loop counter callbacks IRender_loop_render_counter_callback are callbacks that can be used to know when a certain number of render calls have been made, or when the next render is complete. These can be used to ensure that a given render contains changes previously made to the scene.

Workflow

This shows a top level example of the process of how to use a render loop.
  • Get the render loop manager from neuray API.
  • Create a new render loop with a unique name and the name of the handler to use.
  • Set any initial parameters on the render loop before calling initialize and start_loop. Note that the initialize method requires a timeout. If no methods are called on the render loop for that many seconds then the loop is considered abandoned and is automatically shut down to prevent resource hogging.
  • In a separate thread get the render loop manager and then retrieve the newly started render loop by name. Call get_canvas for the most recent render and get_last_render_result for the render result.
  • Whenever a change is made to the scene call set_dirty on the loop. This will ensure that rendering will start again if the scene has converged.
  • When finished rendering either stop using the loop to let it timeout, or call shutdown_render_loop on the manager to stop the loop manually.

Commands

The render loop system provides a suite of commands that allow controlling all aspects of render loops by simply calling service commands. This is the recommended way of using the system. All commands start with the render_loop prefix and can be found in the render_loop command group.

Default Handler

The default handler ships with neuray Services and is used by providing 'default' as the render handler to use when creating a render loop. This handler provides basic rendering services. It supports the following parameters:
renderer
The renderer to be used, ie "iray", "irt", etc. Note: This cannot be changed after the render loop has been initialized.
canvas_pixel_type
The pixel type used by the canvas, ie "Rgba", "Color", etc.
canvas_content
The render content to be rendered to the canvas, ie "result", "depth", etc.

Falsecolor Handler

The falsecolor handler ships with neuray Services and is used by providing 'falsecolor' as the render handler to use when creating a render loop. In addition to basic rendering services it provides many advanced features. It supports the following parameters:
renderer
The renderer to be used, ie "iray", "irt", etc. Note: This cannot be changed after the render loop has been initialized.
canvas_pixel_type
The pixel type used by the canvas, ie "Rgba", "Color", etc.
canvas_content
The render content to be rendered to the canvas, ie "result", "depth", etc.

Falsecolor Tonemapping

The handler can perform falsecolor tonemapping which produces a heatmap image representing luminance in a gradient from blue to yellow to red. Falsecolor tonemapping is enabled by setting a combination of render loop parameters and camera attributes as follows:
  1. Set the canvas_pixel_type parameter on the render loop to Color to ensure that HDR images are rendered.
  2. On the scene's camera set the following attributes to enable false color tonemapping:
    • String tm_tonemapper "falsecolor"
    • Float32 mif_min value this luminance is mapped to blue.
    • Float32 mif_max value this luminance is mapped to red.
Once this is done falsecolor images will be produced. To disable falsecolor simply set tm_tonemapper and canvas_pixel_type back to their previous values.
Falsecolor tonemapping also supports the following parameter:
irradiance
When set to "on" this changes the rendering from a radiance render to an irradiance render. This will override the canvas_pixel_type and canvas_content parameters.

Outline Rendering

The handler is able to highlight one or more objects within a scene by drawing outlines around them. These outlines will persist while the feature is enabled and the outlines will update when the camera or objects move within the scene.

Outline rendering is enabled by setting the following parameter:
outline
r,g,b;outline_instance(,outline_instance)*(;(watch_instance)?(,watch_instance)*(;disable_instance(,disable_instance)*)?)?
The string value consists of up to 4 ; separated sections:
  1. r,g,b color to draw the outline in (values are in 0->1 range)
  2. outline_instance(,outline_instance)* comma separated list of instance names to outline
  3. (watch_instance)?(,watch_instance)* optional comma separated list of instances to watch for changes. If any of these instances move the outline is recalculated. The outlined instances are automatically watched. This list can be empty if you need to specify disabled instances.
  4. disable_instance(,disable_instance)* optional comma separated list of instances to disable when generating outlines. By default object outlines will be cut where other objects overlap them. Listing overlapping objects here will prevent them from cutting outlines. Disables instances are also added to the watch list.
Literal ; , and \ characters in names can be escaped with the \ character.

To disable outline rendering simply set the parameter to an empty string.

For example:

1,0,0;sphere_instance Outlines sphere_instance in red. Updates the outline drawing when the camera or sphere_instance is moved.

0,1,0;sphere_instance,cube_instance;teapot_instance Outlines sphere_instance and cube_instance in green. Updates the outline drawing when the camera, sphere_instance, cube_instance or teapot_instance is moved.

0,0,1;sphere_instance,cube_instance;teapot_instance;buddha_instance,rabbit_instance Outlines sphere_instance and cube_instance in blue but does not allow the outline to be obscured by buddha_instance or rabbit_instance. Updates the outline drawing when the camera, sphere_instance, cube_instance, teapot_instance, buddha_instance or rabbit_instance is moved.

0,0,1;sphere_instance,cube_instance;;buddha_instance,rabbit_instance Outlines sphere_instance and cube_instance in blue but does not allow the outline to be obscured by buddha_instance or rabbit_instance. Has no specific watch instance list so updates the outline drawing when the camera, sphere_instance, cube_instance, buddha_instance or rabbit_instance is moved.

Autoexposure

The falsecolor handler supports a camera auto exposure system which interacts with the built in photographic tonemapper. Triggering an auto expose will internally render an untonemapped HDR image and then calculate the appropriate tonemapper settings to produce a visible image. These settings can either be automatically applied to the camera or just set as separate attributes to be retrieved by the user. In addition statistical luminance information is produced.

Autoexposure is controlled by setting attributes on the scene's camera. As render loop rendering is an asynchronous process a request/response ID system is used to trigger an auto expose request and to idenfify when it is complete.

For example, when the client wants to perform an auto exposure and get the values back they must first set ae_request on the camera to a value higher than the auto exposure request counter in the render loop. As the counter in the render loop always begins at zero the client should first set this to 1 and then increment it further whenever another auto exposure is required. Once the auto exposure is complete, the render loop will set the ae_response attribute on the camera to the same value as ae_request. The client can poll the camera for this value and when it is the same as the original ae_request know the request is complete and read off the values.

Note that if ae_request is set greater than 0 before the render loop is started then an auto exposure will be calculated before rendering begins. This allows for the first image returned to be correctly exposed.

Auto exposure uses settings familar to anyone who has used an SLR camera and are consistent with the photographic tonemapper nomenclature. It is controlled by the following camera attributes:
  • Uint32 ae_request increment this to trigger an auto exposure.
  • Uint32 ae_resolution_x Sets the horizontal resolution of the auto exposure render, a higher value will result in a more accurate auto exposure but will take longer to complete. Defaults to: 150.
  • Uint32 ae_resolution_y Sets the vertical resolution of the auto exposure render, a higher value will result in a more accurate auto exposure but will take longer to complete. Defaults to: 80.
  • Uint32 ae_render_samples Sets the number of rendering samples to perform for the auto exposure render. A higher value will result in a more accurate auto exposure but will take longer to complete. Defaults to: 30.
  • Float32 ae_exposure_compensation Sets an fstop value to offset the auto exposured result by. Defaults to: 0.
  • String ae_metering_mode Sets the way the auto exposure is calculated. These are the available modes:
    • full_image Takes all pixels in the auto exposed render into account equally.
    • spot Takes only a circular cutout in the center of the render into account. The radius is defined by ae_metering_radius. Pixels outside of this circle are ignored.
    • center_weighted The same as spot, however the values inside the circle are weighted by ae_center_weight.
    Defaults to: full_image.
  • Float32 ae_metering_radius Sets the size of the metering circle used by spot and center_weighted metering modes. This is a percentage of the smallest resolution (width or height) value. A value of 1.0 will be equal to the width or height. Defaults to: 0.5.
  • Float32 ae_center_weight Sets the weight of the circle when using center_weighted metering mode. This is a value between 0.0 and 1.0. Values inside the circle are multiplied by ae_center_weight and values outside the circle are multiplied by (1.0 - ae_center_weight). Defaults to: 0.8
  • Float32 ae_f_stop_maximum Sets the maximum that the fstop can be set to. If fstop needs to be greater than this then the shutter speed is changed to compensate. Defaults to: 32.0.
  • Float32 ae_f_stop_minimum Sets the minimum that the fstop can be set to. If fstop needs to be less than this then the shutter speed is changed to compensate. Defaults to: 0.0.
  • Boolean ae_irradiance When true, the auto exposure uses an irradiance render rather than radiance. Default to: false.
  • Uint32 ae_set_onto_camera Controls whether the calculated exposure values should be set onto the camera or not. This allows the tonemappers to automatically be updated with the new values so the next render contains an auto exposed image. This value is set by ORing together the following:
    • 0x00 Do not set anything onto the camera.
    • 0x01 Set the luminance min and max onto the camera, for use by the falsecolour tonemapper.
    • 0x02 Set the f number and camera shutter onto the camera, for use by the photographic tonemapper.
    • 0x04 Used when doing falsecolor renders. When set this will automatically set the falsecolor tonemapper to irradiance mode if an irradiance auto expose was calculated. If a radiance auto expose was done then the tonemapper will be set to radiance mode.
    Defaults to: 0x02.
  • Color ae_whitepoint If set then this value will be transferred to the photographic tonemappers whitepoint attribute (mip_whitepoint) once the auto expose is complete.
  • String ae_tonemapper If set then this value will be used as the tonemapper once the auto expose is complete.
Once auto exposure is complete the following attributes are set back onto the camera:
  • Uint32 ae_response set to the same value as ae_request so we can tell that the auto expose is complete.
  • Float32 ae_max The max luminance or irradiance value found. Can be used by falsecolor mode.
  • Float32 ae_min The min luminance or irradiance value found. Can be used by falsecolor mode.
  • Float32 ae_average The average luminance or irradiance value calculated.
  • Float32 ae_sd The standard deviation of the ae_min, ae_max and ae_average values.
  • Float32 ae_f_number The calculated f number.
  • Float32 ae_camera_shutter The calculated shutter speed.
  • Float32 mip_f_number Same as ae_f_number, only set if ae_set_onto_camera contains the value 0x02.
  • Float32 mip_camera_shutter Same as ae_camera_shutter, only set if ae_set_onto_camera contains the value 0x02.
  • Float32 mif_max Same as ae_max, only set if ae_set_onto_camera contains the value 0x01.
  • Float32 mif_min Same as ae_min, only set if ae_set_onto_camera contains the value 0x01.

Picking

Since the render loop render contexts are not exposed to the rest of neuray Services it is not possible to do a pick on them. The falsecolor handler gets around this by providing a mechanism to perform picks. Note: This system will only return the first pick result from the pick.

Picks are performed in the same way as auto exposures, request attributes are set on the camera, a request value is incremented and the response set once the pick is complete. Picks are performed by setting the following attributes:
  • Uint32 pick_request increment this to trigger a pick.
  • Float32<2> pick_position the position in pixels to pick.
  • Float32<2> pick_size the size of the pick area. Defaults to 1,1
Once the pick is complete the following attributes are set back onto the camera:
  • Uint32 pick_response set to the same value as pick_request so we can tell that the pick is complete.
  • Float32<3> pick_result_position the world point where the pick hit an object.
  • String pick_result_object_name the name of the picked object.
  • String pick_result_error if there was an error, such as a pick on the environment this will be a non empty string. pick_result_position and pick_result_object_name will be unchanged.
  • String[] pick_result_path string array that contains the pick path down to pick_result_object_name

Limitations

It is important to note that the render loop system has limitations.
  • The only way to send several scene changes at once and definitely get a render containing those changes in it is to use an IRender_loop_render_counter_callback or to use the associated render_loop_get_next_render command. Changes made to a scene when committing a transaction will not appear until the next render is complete.
  • There is no support for custom tonemapper operators only the built in ones will work. The only exception is the falsecolor handler which supports a falsecolor tone mapper.
  • There is currently no way of limiting the rate at which the render loop will render.
  • Any commands that require a render context name will not be able to access a render loops' render context.