Reducing the Time and Footprint of Video Exports

When you export a clip as a file sequence (using the Video Options), or when you export it as part of a Sequence Publish where the media is exported as a file sequence, you can use the Link Media options to accelerate the export and minimize used storage space.

Because unedited frames are not recreated but only referenced in the exported sequence, the export takes less time to complete, and saves storage space by avoiding media duplication. The mechanism used to create links is described at How Media Linking Works.

Link Media Export Options

• Link Media: Use this option to publish content and link back to the source media files when possible. This option creates links based on the destination file format, no matter the resolution, bit depth, or compression. This is the most permissive Link Media option.

• Link Media (Resolution): Creates links instead of new media for frames that are unedited, and with a destination resolution matching that of the source media files. Edited frames or frames with a different resolution from the source media files result in Flame writing new media files to disk. The resolution is set in the Resolution Presets box in the Media Export window; Same as Clip uses the resolution of the exported sequence, not the resolution of the source media.

• Link Media (Bit Depth): Similar to the option Link Original Media available in previous versions (2018 and earlier). Creates links instead of new media for frames that are unedited, and with a destination resolution and bit depth matching that of the source media files. Edited frames or frames with a different resolution or bit depth from the source media files result in Flame writing new media files to disk. The bit depth is set in the Bit Depth box in the Media Export window.

  Important: If you use a Media Export preset created in a previous version of the application, Link Media (Bit Depth) option is used to replicate the behavior of that legacy preset.

• Link Media (Compression): Creates links instead of new media for frames that are unedited, and with a destination resolution, bit depth, and compression matching that of the source media files. Edited frames or frames with a different resolution, bit depth, or compression from the source media files result in Flame writing new media files to disk.

• Generate Media: Always create new media, no matter what. Use this option when the exported files' metadata, such as timecode, must be accurate.

Simple Link Media (Bit Depth) example:

1. Import a sequence of 50 .dpx files, with a bit depth of 10-bit.

2. Of that sequence, you colour correct only the first 10 frames.

3. Export that sequence as a Sequence Publish, using the format preset Source Media Publish (10-bit DPX and WAVE). Notice how in the menu Video Options, the box Link Original Media is set to Link Media (Bit Depth).

4. The result of the export is 50 .dpx files:

   • 10 new .dpx files created by Flame
   • 40 .dpx files referencing the original files

   The last 40 files are referenced, and because they were neither created nor moved, the export process was that much faster.

Because media linking uses tools enabled by the operating system (see below), the following conditions must also be met.

Additional requirements for successful media linking:

• The sequence must be exported to the volume that stores the source media files. It can be a different directory, but the volume must be the same. If the export destination is a different volume, then Flame copies to the new volume those files that would have linked in the right circumstances.
• The sequence must be exported in the file format of the source media: DPX to DPX, OpenEXR to OpenEXR.

If the source media is not available, Flame can link to the media cache instead, but on the following conditions:

• You export the sequence to the volume that holds the media cache.
• You select Link Media (Bit Depth) or Link Media (Compression).
• You select the compression indicated by the asterisk ( * ) in the Compression drop-down box. The asterisk indicates the compression format used by the media cache.

How Media Linking Works

For the link media mechanism, Flame uses the hard link mechanism found in POSIX-compliant operating systems such as the Linux and the macOS operating systems.

To understand the hard link, you must first understand how a file is stored on a volume. On a file system, for reasons of efficiency, data and the organization of files on a volume are managed separately.

Consider a .dpx file named original.dpx. When the operating system ask the file system to write this file to disk, the following happens:

1. The file system sends the image content of original.dpx to the hard drive and asks it to write it down somewhere. This data consumes a certain amount of storage.
2. The hard drive sends back to the file system the physical address where the data is stored.
3. The file system stores this address, called inode, along with other relevant metadata (file name, file location within the directory hierarchy, creation date) in what we call a file, that is displayed by the operating system as original.dpx.

For the user, a file is the data. But for the file system, file and data are two separate entities. It is the operating system that creates the illusion for the user. But this illusion is quite useful: moving a file between folders changes the path in the file's metadata, without actually moving the data. The file hierarchy displayed by the file system (folders and files) bears little resemblance with how the data is stored on the volume: moving a file around does not change its inode, or address.

Going further, one can create a new file, called linked_file.dpx. This file has its own file name, creation date, file path. But it is assigned the same inode as original.dpx: both files point to the same content. But no actual data is duplicated. The operating system presents both files separately. This is what's called creating hard links.

• You can easily see how many reference counts are on a file by typing the following in a shell: ls -li

• The command returns something similar to the following lines:

   -rw -rw -rw 1 user users 8.0M Jan 5 09:47 solo_file.dpx
   -rw -rw -rw 2 user users 8.0M Jan 5 09:49 linked_file.dpx

• The specifics are not important. What is important is the number next to the Unix file permission (-rw -rw -rw): it displays the number of files that refer to the same inode. The number 1 on the solo_file.dpx line indicates this is the sole file to refer to its inode. Deleting this file actually frees up storage.

• For linked_file.dpx, the 2 indicates the inode is referenced by 2 files: linked_file.dpx and another one. Deleting linked_file.dpx does not delete the original reference, nor does it free space on the file system: you need to delete both linked_file.dpx and original.dpx. An inode is only deleted when every file referencing it has been deleted.

And that is what happens with the Media Linking. By assigning existing inodes to new files for those frames that have not changed, Flame avoids data duplication. And saves time, since creating a new file is much faster than writing a whole media frame.

Notes about hard links:

• Hard links can only be done on the file system where the original files are located.

• A hard linked file shares the same clip properties as the original file.

  For example, you wish to publish DPX content that was modified in Flame. But changes to the files' metadata such as timecode and tape name are not reflected in the hard link media.

  Also, managed media created in Flame (renders, caching, etc.) do not contain any timecode nor tape name when created as DPX or OpenEXR. Creating Hard Links to be used in third party applications might need to be used with file name information for timecode since the files will not contain the original media attributes.

Setting Link Media to Work Across Different Volumes

Hard links used by Link Media require that the original media file and the hard link are both created on the same volume. This limitation is inherited from the file system: inodes (essential to media link) cannot be shared between file systems. In cases where you absolutely need to create links across file systems, you can use soft links. On macOS, a soft-link is also known as an alias.

A soft link is an indirect link to a file: if file linked_file.dpx is soft linked to file original.dpx, file linked_file.dpx essentially stores the path to file original.dpx. It does not duplicate any inode information! This means that:

• If file original.dpx is moved, file linked_file.dpx cannot access its contents anymore.
• If file original.dpx is renamed, file linked_file.dpx cannot access its contents anymore.
• If file original.dpx is erased, file linked_file.dpx cannot access its contents anymore.
• If file original.dpx is overwritten, file linked_file.dpx now uses the new content.

In the end, you only want to use soft links because your workflow requires using multiple volumes. And as long as you are aware of the limitations, this workflow can work.

Soft links are disabled by default.

To enable soft links:

1. Open the file /opt/Autodesk/sw/cfg/stone+wire.cfg.

2. Search for the line # SymlinkAcrossFilesystems=True

3. Remove the # to uncomment it.

4. Save and close the file.

5. From the command line, restart Stone+Wire. Enter:

   /opt/Autodesk/sw/cfg/sw_restart