Stone and Wire

Stone and Wire is a layer that sits between the application and the file system. It is required and installed with Creative Finishing applications, except Lustre. It handles local management of media and the transfer of media between workstations. Wire enables the transfer of uncompressed video, film and audio between workstations over networks.

Five processes that must be running for Stone and Wire to work: sw_serverd, sw_probed, sw_dbd, sw_bwmgr, ifffsWiretapServer. This process is started by the sw_wiretapd script.

All Stone and Wire processes should start when the system is restarted. If they are not running, check that they are configured to run and then restart Stone and Wire. To start Stone and Wire processes, edit /etc/sysconfig/stone+wire and ensure that it contains the following:

Restart Stone and Wire with /usr/discreet/sw/sw_restart.

Logging and errors

Every Stone and Wire process has a log file at /usr/discreet/sw/log.

The current log file is named <process>.log, where <process> is the name of the Stone and Wire process or daemon. The next time Stone and Wire creates a log file for the process, it renames the previous log file by adding a number to the file name. For example, the sw_served process log file is named sw_served.log. The next time the process is launched, the first log file is renamed to sw_served.log.1. Each time Stone and Wire creates a new log file, the old log file is renamed by incrementing the number that appears at the end of the filename.

Use the terminal to list the most recent version of the log files: ls -altr

Use swr_set_debug to enable verbose reporting of Stone and Wire driver errors. When error reporting is on, Stone and Wire outputs error messages to /var/log/messages and the console window.

Disable verbose error reporting to save on storage. Run: /usr/discreet/sw/tools/swr_set_debug <level> where <level> is the verbosity of the report: 0 is off, 1 is default, and 2 logs all operations and errors.

Stone and Wire configuration

The configuration file is /usr/discreet/sw/cfg/stone+wire.cfg. In the case of software upgrades, if stone+wire.cfg exists, then it is not overwritten. However a new stone+wire.cfg.sample file is written which includes any new functionality added in that release, including for example support for new file formats.

sw_framestore_map

/usr/discreet/sw/cfg/sw_framestore_map is only available after the installation of the Autodesk product. It is configured when the application is installed. Normally, it should not be necessary to edit this file.

It has two sections

  • [FRAMESTORES]: contains information on the media storage volume. If the workstation uses DHCP to get its IP address, use localhost as the value for the HADDR parameter.
  • [INTERFACES]: keywords that define the network interfaces on the system that are available for Wire transfers.

Media pre-allocation

By default, disk space for video files is allocated on-the-fly as the files are rendered or written. Concurrent I/O operations from multiple clients or background processes can generate frames with poor locality of reference, which leads to clip fragmentation and decreases playback performance.

To prevent this problem, Stone and Wire provides media file pre-allocation mechanisms that reserve the necessary space on the disk before the actual writing of the media files takes place. Pre-allocation reserves as many available contiguous disk sectors as needed for the file sequence to be written.

There are two media pre-allocation methods:

FALLOCATE
The disk space needed for the file is allocated and filled with zeroes. On most filesystems, the performance impact of this operation is as serious as when writing the actual file.
RESERVE
The disk space needed for the file is reserved, but not written to. The apparent size of the frame remains zero. This operation is much faster than FALLOCATE, but is supported only on certain filesystems, such as XFS.

By default, pre-allocation is turned off in Stone and Wire (the method is set to NONE). To enable it, modify a parameter in stone+wire.cfg. This is not a global setting. Pre-allocation must be individually enabled for each partition. On an XFS system on which unwritten extent flagging was disabled when creating the filesystem, the RESERVE pre-allocation method is not available for it.

To enable media pre-allocation for a partition edit the Preallocation parameter in /usr/discreet/sw/cfg/stone+wire.cfg Save the configuration file and restart Stone and Wire..

sw_io_perf_tool

Stone and Wire comes with a command line tool to measure filesystem performance called sw_io_perf_tool. This tool simulates I/O requests (audio and/or video) that would be issued by the Creative Finishing application, and reports the throughput and maximum latency of all the requested I/O operations. Use sw_io_perf_tool to measure read and write performance of a volume declared and mounted by Stone and Wire. Several options are available to simulate many file formats and writing/reading patterns. Launching the command will display a list of available options.

To list all options available for sw_io_perf_tool, open a terminal and type /usr/discreet/sw/tools/sw_io_perf_tool -h .

To test a standard filesystem partition's performance for a specific resolution, Run /usr/discreet/sw/tools/sw_io_perf_tool <-p #> <options> where <-p #> specifies a partition ID (0 to 7), and<options> can be a combination of several media format attributes.

sw_io_perf_tool with custom resolutions may be used. For example, to test read performances for frames with a width of 1600 pixels and a height of 900 pixels: Max Latency: 335.284 ms

The command can be run while other operations are ongoing to simulate how the application will behave under these conditions. The media used by sw_io_perf_tool is allocated and freed automatically. sw_io_perf_tool will not reserve any bandwidth on the storage device. Partitions/volumes must be properly configured in the Stone and Wire configuration file.

wiretapgateway

Imports compressed media into Stone and Wire partitions. Controls the transactions between the Stone and Wire server and remote applications. Its configuration file is: /usr/discreet/wiretapgateway/cfg/wiretapgateway.cfg.

sw_probed

Uses a self-discovery mechanism to locate all other Wire hosts and their framestores on the Wire network. Stores the configuration required for communication between two hosts in memory. Its configuration file is: /usr/discreet/sw/cfg/sw_probed.cfg. To enable multicasting:

  1. On the render node, open /usr/discreet/sw/cfg/sw_probed.cfg for editing.
  2. Set SelfDiscovery to Yes. Now sw_probed runs in self-discovery mode and it will automatically probe the network for other systems. This is set to Yes by default when Stone and Wire is installed on the render node.
  3. The Scope parameter defines the scope for the multicast. This parameter setting must be the same for all machines on the network.
    • For networks with one subnet, set to LinkLocal.
    • For networks with subnets, use an appropriate value and router configuration, either OrganizationLocal, or GlobalWorld.
  4. If the workstations and nodes in the facility are on separate networks connected through routers, use the ttl parameter in the file to specify the number of router hops for a multicast. Transfers across multiple routers may cause bottlenecks at network bridges, especially with jobs involving film frames. Using the ttlparameter may reduce multicast-related traffic and improve general network performance in the facility.
  5. Save and close sw_probed.cfg then restart the sw_probed daemon with /etc/init.d/stone+wire restart.

sw_serverd

The Wire server daemon. Controls the transactions between the Stone and Wire filesystem and remote applications. Its configuration file is: /usr/discreet/sw/cfg/sw_framestore_map.

sw_dbd

Responsible for mapping frame IDs from the entries in the swdb database to the location of the media on disk. Database entries are created for all managed media and soft-imported media. One Standard FS media database file exists for each Standard FS partition. Its configuration file is: /usr/discreet/sw/cfg/sw_dbd.cfg. Logs are located in: /usr/discreet/sw/log/

Configure the Stone and Wire database to automatically shut off when there is too little free space:

  1. As root, edit /usr/discreet/sw/cfg/sw_dbd.cfg, and in the [Disk Check] section uncomment the Enabled and Minimum keywords.
  2. Set the value of Minimum to the minimum allowed amount of free space on the disk (in Megabytes). If the amount of free space on the disk falls below this threshold, the Stone and Wire database daemon automatically shuts off.
  3. Optionally uncomment the Timeout keyword, and set how often the Stone and Wire daemon should check the available disk space. The default value is 30 seconds.
  4. Save and close the configuration file, the restart Stone and Wire: /etc/init.d/stone+wire restart.

Bandwidth reservation: sw_bwmgr

Reserves storage bandwidth for Autodesk applications and tools based on its configuration file /usr/discreet/sw/cfg/sw_bwmgr.cfg. Documentation is in /usr/discreet/sw/cfg/sw_bwmgr.cfg.sample.

The Stone and Wire bandwidth reservation mechanism tries dynamically to improve bandwidth distribution to Autodesk applications connecting to a workstation’s storage. It cannot protect the storage from I/O requests from non-Autodesk applications and tools, since it doesn't handle them.

In low bandwidth situations, instead of reducing all reservations proportionally towards zero, it first reduces the bandwidth of each process towards the low-bandwidth value specified for that process. If the bandwidth continues to degrade, it starts reducing bandwidth proportionally towards zero.

Configure bandwidth reservation

Additional standard filesystem mount points (such as mount points to SAN volumes or to any other standard filesystem volume that is not present in /usr/discreet/sw/cfg/stone+wire.cfg) can be managed by the Bandwidth Manager. Each partition managed by the Bandwidth Manager must have a [Device] section associated to it in /usr/discreet/sw/cfg/sw_bwmgr.cfg. To add a new device to the Bandwidth Manager:

  1. Open /usr/discreet/sw/cfg/sw_bwmgr.cfg and add a new [Device] section at the end of the file, as follows:
    • <device_number>: the number of the [Device] section, e.g. [Device0].
    • Path: the partition's mount point. Since a partition can have several paths, <path_number> is the number of the current path, starting at 0 for each device, e.g.: Path0=/mnt/XYZ.
    • TotalAvailableReadBandwidth: the estimated total reading bandwidth of the device, in megabytes per second. Based on this value, and on the amount of active reservations for the current partition, the Bandwidth Manager decides whether new reservation requests are granted or rejected.
    • TotalAvailableWriteBandwidth: the maximum estimated writing bandwidth of the device, in megabytes per second. If unspecified, its value defaults to the value of TotalAvailableReadBandwidth. The total bandwidth parameters are estimates of the theoretical maximum bandwidth of the partition. The actual bandwidth is affected by several factors, including multiple applications trying to concurrently read or write to it. The Bandwidth Manager continuously measures partition performance and dynamically maintains the actual total available bandwidth for each partition.
    • DefaultReservation: the bandwidth assigned by default to applications that are denied a reservation request or that don't have individual or group reservations associated to them. The value is expressed in megabytes per second, e.g.: DefaultReservation=10
  2. Define reservations for the current partition.
  3. Save and close the configuration file, and restart Stone and Wire: /etc/init.d/stone+wire restart

Define groups of applications

Bandwidth reservation can be done by groups of applications.

  1. In /usr/discreet/sw/cfg/sw_bwmgr.cfg, [Groups] section, add a line for each group of applications to be defined, as in this example:

    <group_name>=<application_name> <application_name> where:

    • <group_name> is the custom name of the group. The group name must not contain spaces and must not be the same as one of the predefined application names.
    • Each <application_name> represents the name of an application in the group. This parameter can take one of the following predefined values: “inferno”, “flame”, “flint”, “smoke”, “stonifiseTool”, “publishTool”, “S+W Server”, “IFFFS Wiretap Server”, or “IFFFSTool” for other Creative Finishing command-line tools.
  2. In the [Device] section add a Reservation line for the new group. The syntax is the same as when adding a reservation for a single application. For example (using the “IFFFSTools” group from the previous step): Reservation1=IFFFSTools 50 50
  3. Save and close the configuration file, and restart Stone and Wire: /etc/init.d/stone+wire restart

For more information see the comments inside the configuration file.

Remove invalid frame references from the Stone and Wire database

Stone and Wire has a database inconsistency detection mechanism. If the framestore ID of a Stone filesystem or standard filesystem partition in the system is changed without reformatting the Stone and Wire database, the inconsistency check mechanism detects invalid frame references in the database, and prevents the Stone and Wire database server sw_dbd from starting.

To remove invalid frame references from the Stone and Wire database:

  1. In a terminal, as root, search the /usr/discreet/sw/log/sw_dbd.log file for the text “Bad framestore ID” e.g.: cat /usr/discreet/sw/log/sw_dbd.log | grep -i bad/ framestore/ id
    • If the search returns one or more lines, the problem is related to invalid frame references. Perform the remaining steps in this procedure to clean up the Stone and Wire database.
    • If the search returns no results, the problem is not related to invalid frame references. Check the /usr/discreet/sw/log/sw_dbd.log file to determine the cause of the problem.
  2. Stop Stone and Wire: /etc/init.d/stone+wire stop
  3. Clean up the Stone and Wire database: /usr/discreet/sw/sw_dbd -C. A prompt offers to remove invalid entries from the Stone and Wire database.
  4. Restart Stone and Wire: /etc/init.d/stone+wire start

Check disk usage

To see if the filesystem is full, run /usr/discreet/sw/sw_df. If the filesystem is full or almost full, consider deleting or archiving older material from clip libraries to free disk space. VTR archiving in Creative Finishing applications requires free space on the framestore to generate slates, test patterns, headers, and audio encoding.

Clear the Undo Buffer to free up space. Clearing the Undo Buffer cancels any undoable operation, so use carefully. This is available in the Preferences menu, Undo section.

Solve a framestore ID conflict

All hosts on the Wire network must have a unique framestore ID. By default the ID is generated from the IP address. It is possible that there could be an ID conflict, where an ID is not unique. In this case Flame will not run, because the Wiretap server shuts down after detecting the conflict. The log file /usr/discreet/log/ifffsWiretapServer.log will contain an error.

Solve a partition ID conflict

Each partition must have a different partition ID. To solve a Partition ID conflict:

  1. Stop Stone and Wire: /etc/init.d/stone+wire stop.
  2. In /usr/discreet/sw/cfg/stone+wire.cfg, find the [PARTITION#] section, where # is the ID listed in the error message. Change the # for an unused ID. Partition identifiers go from 0 to 7, giving a total of 8 possible partitions. Stone filesystem and standard filesystem partitions share the same pool of identifiers, but only standard filesystem ones are listed in stone+wire.cfg. It is recommended to start numbering standard filesystem partitions at 7, decrementing with each additional standard filesystem partition.
  3. Save and close the stone+wire.cfg file, then start Stone and Wire: /etc/init.d/stone+wire start.