Reducing the Time and Footprint of Video Exports

When you export a clip as a file sequence (using the Video Options), or when you export it as part of a Sequence Publish where the media is exported as a file sequence, you can use the Link Media options to accelerate the export and minimize used storage space.

Because unedited frames are not recreated but only referenced in the exported sequence, the export takes less time to complete, and saves storage space by avoiding media duplication. The mechanism used to create links is described at How Media Linking Works.

Link Media Export Options

Simple Link Media (Bit Depth) example:

  1. Import a sequence of 50 .dpx files, with a bit depth of 10-bit.
  2. Of that sequence, you colour correct only the first 10 frames.
  3. Export that sequence as a Sequence Publish, using the format preset Source Media Publish (10-bit DPX and WAVE). Notice how in the menu Video Options, the box Link Original Media is set to Link Media (Bit Depth).
  4. The result of the export is 50 .dpx files:
    • 10 new .dpx files created by Flame
    • 40 .dpx files referencing the original files

    The last 40 files are referenced, and because they were neither created nor moved, the export process was that much faster.

Because media linking uses tools enabled by the operating system (see below), the following conditions must also be met.

Additional requirements for successful media linking:

If the source media is not available, Flame can link to the media cache instead, but on the following conditions:

Important: The export never fails because of failed media links. Flame always completes the export. When it cannot link to the source media files, it copies the files. And if it cannot copy, the export creates new files.

How Media Linking Works

For the link media mechanism, Flame uses the hard link mechanism found in POSIX-compliant operating systems such as the Linux and the macOS X operating systems.

To understand the hard link, you must first understand how a file is stored on a volume. On a file system, for reasons of efficiency, data and the organization of files on a volume are managed separately.

Consider a .dpx file named original.dpx. When the operating system ask the file system to write this file to disk, the following happens:

  1. The file system sends the image content of original.dpx to the hard drive and asks it to write it down somewhere. This data consumes a certain amount of storage.
  2. The hard drive sends back to the file system the physical address where the data is stored.
  3. The file system stores this address, called inode, along with other relevant metadata (file name, file location within the directory hierarchy, creation date) in what we call a file, that is displayed by the operating system as original.dpx.

For the user a file is the data. But for the file system, file and data are two separate entities. It is the operating system that creates the illusion for the user. But this illusion is quite useful: moving a file between folders changes the path in the file's metadata, without actually moving the data. The file hierarchy displayed by the file system (folders and files) bears little resemblance with how the data is stored on the volume: moving a file around does not change its inode, or address.

Going further, one can create a new file, called linked_file.dpx. This file has its own file name, creation date, file path. But it is assigned the same inode as original.dpx: both files point to the same content. But no actual data is duplicated. The operating system presents both files separately. This is what's called creating hard links.

And that is what happens with the Media Linking. By assigning existing inodes to new files for those frames that have not changed, Flame avoids data duplication. And saves time, since creating a new file is much faster than writing a whole media frame.

Notes about hard links:

Setting Link Media to Work Across Different Volumes

Hard links used by Link Media require that the original media file and the hard link are both created on the same volume. This limitation is inherited from the file system: inodes (essential to media link) cannot be shared between file systems. In cases where you absolutely need to create links across file systems, you can use soft links. On macOS, a soft-link is also known as an alias.

A soft link is an indirect link to a file: if file linked_file.dpx is soft linked to file original.dpx, file linked_file.dpx essentially stores the path to file original.dpx. It does not duplicate any inode information! This means that:

  • If file original.dpx is moved, file linked_file.dpx cannot access its contents anymore.
  • If file original.dpx is renamed, file linked_file.dpx cannot access its contents anymore.
  • If file original.dpx is erased, file linked_file.dpx cannot access its contents anymore.
  • If file original.dpx is overwritten, file linked_file.dpx now uses the new content.

In the end, you only want to use soft links because your workflow requires using multiple volumes. And as long as you are aware of the limitations, this workflow can work.

Soft links are disabled by default.

To enable soft links:

  1. Open the file /opt/Autodesk/sw/cfg/stone+wire.cfg.
  2. Search for the line # SymlinkAcrossFilesystems=True
  3. Remove the # to uncomment it.
  4. Save and close the file.
  5. From the command line, restart Stone+Wire. Enter:

    /opt/Autodesk/sw/cfg/sw_restart

Note: Soft links are only used to link to imported media. No soft links can be made to or from managed media.