The process of producing renders are extensively more in demand with the amount of iterations of shots that are being sent to render. In DNEG, the procedure will more likely send render jobs to render farms in which is processed in the background. Distributing these jobs across different clusters of computers and nodes ensures that the time it takes to render will be considerably less than just rendering in one machine. Render farms encapsulate the grid framework that are made up of multiple nodes which are assigned jobs that are put together to create a network of nodes that help communicate and compute areas that require time to respond.
Tractor is used to manage the render jobs that have been submitted by artists from every department. Render job priorities can also be requested to ensure that the render that urgent tasks are being processed and sent through. Errors and issues that might occur will have an error log provided and will need to be flagged to the supervisor/lead in charge as well as other respective artists and the tech team that will look into the problem through the ticketing system. This support ticket created will directly link the job or task with the error which the tech team can easily find.
Monitoring the farm and being able to track how performance heavy rendered jobs are distributed can help manage the render queues. Tractor has an in-built graph that show server and software metrics. More considerably useful to track renders through Grafana which is a production friendly farm statistics system. On occasion I do check the live resource usage reports especially when shows are nearing delivery to keep track and just out of curiosity, really.
When handling with rendered material, artists are able to send their renders to the render farm which are managed and distributed depending on the level of prioritisation from the scale of 1-5 (5 being extremely urgent) to ensure that the renders will be processed. Working within the show pipeline, I had to ensure that I can submit my deliverable on time and by changing my render priority to 5, allowed me to submit my rendered pass for dailies. This procedure immediately rendered my shot quickly along with the tools that enabled to produce the rendered outcome which was a slap comp of the shotsculpt with all the model, lighting, textures and plates that have been published within the shot.
An option to render locally is necessary when working outside of the scope of the project, i.e. test renders, producing work for training purposes or for personal projects. This is to ensure that the farm renders only deal with show specific tasks rather than separate entities that have no relation to the work within the scope of the production.
Necessary action is required to limit and optimise the render jobs to run in the farm. Each show are given their balance and fair share of render slots. Soft show limits are given in a sense that will provide more share once the farm is less busy and full to accommodate the needs of certain shows. To facilitate that enough resources are provided, brief meetings are conducted to can offer to temporarily boost the show’s render shares for a day or two. For heavy renders, background renders from fastest and less memory intensive workstations which happens when staff are not in the facility during weekdays and all throughout the weekend.
What’s incredible is that renders from other artist’s work can run in your workstation. Since each machine are referred to as a slave using the processing power of the workstations within the studio. The nimby (i.e. not in my backyard) command ran in the shell will facilitate background renders onto unnimbied machines by using the nimby -u command. This is extraordinarily helpful for artist renders that have been struggling to render out their jobs.
RENDER JOBS
Within my line of work rendering work would be for the purpose of checking if geometry is cached out right since I’m mostly dealing with alembic geometry caches for shotsculpt work. After I’ve sent my work to render through the farm, it will output a playblast version and slap lighting comp version of my cache since lighting and comp have already got versions published and are fed into the shot data which will render out my sequence as if it’s already close to being finaled. This is to ensure that the look will corresponding to the look and colour of the show. Another purpose in terms of rendering the animation of rig for testing and QC (quality check) is to make sure that the cache sets (what’s being outputted and rendered out on the virtual camera) like the rendered geometry or sim geometry for CFX (cloth) work are visible and that the rig behaves and deforms accordingly.
RENDER ERRORS
Render Wranglers (or Render TDs) are usually helping to debug failed renders. Failed renders are distinguishable in Tractor and are displayed in red. Usually looking into the render node and checking the log traces which will highlight the reason. Rerunning the render and checking to see if the error happens again. Usually I would look through the ticketing system and enter some of the traceback error words to see if it has happened before or if there are any solutions. If not it needs to be highlighted and tracked by the pipeline and render TDs so that it can be resolved.
RENDER OPTIMISATIONS
There are many different ways of approaching how to optimise a render. Most of the visual effects work nowadays will have high resolution geometries that will eat up a considerable amount of memory. Splitting up longer sequences into multiple chunks is a way to speed up jobs at the cost of CPUs. This is called chunking and it is normally done by either splitting render sequences into smaller chunks of frames or by rendering different segments of an image as multiple tasks. For example the digi-double will be rendered separately from the environment that’s been decimated into bits and pieces and comped together in Nuke.
What’s also put in place is having multiple versions of the asset but are saved as different level of detail. One can have the lowest poly-count, while another version will have substantially more refined details and another version will have the highest poly count with extremely detailed displacements and texture maps. Objects that are furthest away from the camera will be at the lowest possible level of detail while hero characters and objects (closest to the camera) will have the highest level of detail.
ericka's blog 