Terabytes of large-scale data are dealt and processed on a regular basis within VFX facilities. In DNEG, keeping track of all the asset or shot data are published, stored and managed through asset management system. The file system management and database would follow a standardised structure (hierarchal naming convention) to ensure that it follows the procedure and is accessible throughout the pipeline.
DATA STORAGE TYPES
Assets and footage scans from client that have been stored in an online asset management system and available for artist within the facility once granted access by the production coordinators. These data are accessible anytime and are available immediately once pushed through the pipeline.
Backups and archived data that are infrequently used are stored onto LTO drivers and other near-line storage. These are not immediately accessible but are more cost-effective for data that’s not constantly using disk spinning power.
DIGITAL IMAGES AND MANIPULATION TECHNIQUES
Digital images can be classed as an array of pixels which each pixel contains a set of RGB (red, green and blue) value that when put together output colour information. Images can also store information not related to colour such as alpha (opacity), motion vectors (distance of object from camera) and other arbitrary sets of image channels.
Moving images that are dealt within production are stored as individual still image frames. These when combined are displayed at rates at either 25 or 30 frames per second (fps) which are processed to create a full motion picture sequence. Having to produce, longer more complex visuals require having to read and write image sequences with the expense of very high data transfer rates between computer and disk drives.
Video codecs which are a compression technologies are used to lower the quality of the captured data sequence. By generating proxy footage which a lower resolution, artists can manipulate and work on their tasks without being slowed down by higher resolution outputs. These proxy footage are usually used as a reference of where the asset that’s being worked on during shot work is aligning with the camera.
Converting, resizing and renaming of files within the asset management stack are mostly initiated by the Editorial and the ATDs which tasks I am not able to perform as a Creature apprentice.
DISK MANAGEMENT
Every data contains a metadata that consists of elements such as the shot name, the date added, status, location, application and etc. And essentially the end result will be a rendered data output that will have the exrs and metadata.
Managing files and identifying when that take up server space which the Pipeline TDs would send an alert to the artist to request a delete submission. Monitoring disk usage regularly allows Pipe TDs/ATDs to ensure that the servers don’t quickly fill up data. Identifying which areas that take up server space and submitting delete request for older versions of the playblasts or specific tasks to the artist will help free up disk space. Whenever there are emails regarding the server space filling up, I instinctively check and monitor my data usage to ensure that I can free up space as much as possible.
Auto-deletion systems will regularly use a rule-based system to configure and run a script that interacts with the database. This script will analyse the versions and mark data as deletable if they meet the criteria such as element not approved, if it hasn’t been submitted to client, if it hasn’t been submitted to dailies or if it’s been 5 versions behind the latest version.
BACKUPS & ARCHIVES
Backed up copies are safely stored in an event that live data might be lost, infected or corrupted. This risk-prevention measure is constantly being ran on a regular basis since loss of data will lead to a loss of money and artist time. And the loss of a client’s trust which perhaps can become the most devastating thing that can happen. Snapshot systems are solutions for backing up data. Changes (snapshots) made to a machine’s file system are recorded and stored at regular intervals. To limit storage use the unwanted backup snapshots are deleted after a set period of time.
Archiving is another important data-protection measure that stores data long-term once it’s no longer being in use. The process of archiving is not always time-critical therefore files can be indexed and unnecessary data can be pruned. LTO (Linear Tape-One) drives are storage mediums (holding up to 12TB of data) commonly used in visual effects.
I am unable to perform these archival tasks and restoring data from tape. But I do have an understanding that archives to tape are performed once the shot are finaled but keeping the final versions to ensure that the downstream dependencies are protected.
DATA SECURITY
Sensitive data is being handled within VFX studios which is why data security measures are taken seriously, especially within the digital age when data and personal information can end up being distributed. Due to the competitive nature of data security systems that prevent unauthorised access to information. Data security compliance training and certification is enforced that reflect the legal systems by protecting laws and written contracts holding people and corporations accountable.
Encryption is the process of encoding data using a mathematical cryptographic algorithm (substituting certain characters) that makes it unreadable to anyone without the right keys that may intercept.
DATA ACCESS & PERMISSIONS
Artists aren’t able to access specific files if they aren’t granted access or permission by a specific show’s production coordinator. This is to ensure that sensitive material is exclusive to the team working directly on the show. Show related files stored in the asset management system and Shotgun pages also restrict access.
ericka's blog 