This page gives information about the suggested Flow Production Tracking update process for Flow Production Tracking Enterprise.

Glossary

Flow Production Tracking Operations Team - Team responsible for supporting Flow Production Tracking Enterprise clients.

Application Server - The server on which the Web Application components are installed.

Database Server - The server on which the database server is installed. It can be the same server as the database server.

Staging - Refers to any component of Flow Production Tracking used for development and testing purposes.

Production - Refers to any component of Flow Production Tracking used for production purposes.

Update procedure overview

Updating Flow Production Tracking consists of deploying a new version of flow production tracking-app container. That container contains the libraries the Flow Production Tracking Web App needs to run, plus the Flow Production Tracking front-end and back-end code. Using the new Flow Production Tracking coder requires performing migrations to the database data.

Depending on your cluster topology, the different Flow Production Tracking components can live on the same server, or on different servers.

During the update, Flow Production Tracking will be in maintenance. That means that for your Production update, you will have to schedule downtime at your studio. This downtime can be considerable, sometimes many hours, depending on the version updated to or from. Keep that in mind when scheduling your update at your Studio. Updating every year can help reduce the downtime period.

Staging and Production

Flow Production Tracking enforces an update to your Staging environment prior to any Production update. We enforce this for many reasons:

1. Like any complex system, regressions are always possible. Upgrading staging allows you to test exhaustively before you upgrade Production. You should test:

2. Performance regressions.

3. Web UI, at least pages that you use the most.

4. Critical production scripts.

5. Integration to the different tools and DCCs you use.

6. Most Flow Production Tracking upgrades require some migration steps on the database’s data. Flow Production Tracking is a powerful software that allows clients to customize their data’s schemas. Because of this, data migration issues are always possible and hard to completely prevent. Synchronizing Staging with Production, then testing the upgrade process on Staging, reduces the risk of having a failure during the Production upgrade.

7. Flow Production Tracking is a critical part of a lot of studios’ pipelines. Our objective is always to reduce the risk of having Production downtime. We always make this choice at the expense of execution time.

Flow Production Tracking update workflow

Here is the workflow that we enforce when updating Flow Production Tracking. Using this workflow will help you minimize risks and get the best out of Flow Production Tracking.

1. Planning

   1. Determine with your Production team when downtime would be acceptable. Plan for your staging upgrade to happen at least two weeks prior to that date. Sometimes, knowing how much downtime will be required is useful, so you may want to do the staging upgrade before planning production downtime.

2. Staging upgrade

   1. Production event trim - We suggest removing events older than six months. This can help improve Flow Production Tracking's performance. The database is backed up first. Also, trimmed events are kept available, in CSV format, allowing for later consultation.
   2. Production to Staging sync - We will sync your production data to your Staging site. This will ensure your testing is as relevant as possible. This also ensures that your data can be migrated without issues to a new Flow Production Tracking version.
   3. Media sync - Optional step. Only needed if your testing requires the media to be available. Flow Production Tracking will work without thumbnails and media assets.
   4. Staging upgrade - Update and test the latest features!

3. Exhaustive testing - To prevent any issues in production, we highly suggest you test all your common and critical Flow Production Tracking workflows:

   1. Performance regressions.
   2. Web UI, at least pages that you use the most.
   3. Critical production scripts.
   4. Integration to the different tools and DCCs you use.

4. Production upgrade - Once your testing is conclusive, we can confirm the production update time our Support Team scheduled.

Planning for downtime

Updating Flow Production Tracking is an intrusive procedure that requires planning. It will require putting Flow Production Tracking in maintenance for a possibly long period of time.

When planning how much downtime will be required, one must take the worst-case scenario into account. The worst-case scenario in the case of an update is to have to restore the database backup taken before the tentative update.

How to evaluate the downtime duration

The downtime period can be computed using the following formula:

Downtime = Backup Time + Update Time + Restore Time

Backup time - The backup time can be estimated by performing a backup of the production site.

Update time - The update time highly depends on the update path. It also depends on the data migrations included in the release, and the database content. A rule of thumb is to account for about 15 minutes per update jump. For example, updating from Flow Production Tracking 6.2.0 to 7.0.8.2 will take on average 45 minutes (6.2.0 -> 6.2.7 -> 6.3.18.0 -> 7.0.8.2 equals three jumps, for a total of about 45 minutes). The update time will be known with more precision after the staging upgrade, as the operation should take about the same time for staging as for production.

Restore time - The restore time is about three times the backup time. A more precise estimation can be obtained at the production to staging sync step. Restoring the backup is an exceptional measure that happens very rarely, but it should be accounted for to prevent unexpected impact on your production.

Details on the Different Steps

Event trimming

We highly recommend trimming events as the first step of the update sequence. Trimming events has many advantages:

• It helps improve Flow Production Tracking performance.
• It reduces the database size.
• It lowers the downtime required for the production upgrade.
• It reduces the impact of non-optimized queries on the Event Log Entries table.

Event trimming can take a while and can be intrusive. We suggest you trim logs in batches if you fear it will take too much time, and to run it outside production hours.

Production to staging sync

Before executing the operation, make sure that you have enough space on the main production app server and on the staging application server to store the backup. When doing a sync, the backup is first stored on the main production application server, and then copied over to the main staging application server.

Upgrading version by version

Flow Production Tracking enforces an upgrade path from version to version. You will need to download each of the required versions from https://shotgrid.autodesk.com/ (learn more here). Then, you’ll need to sequentially update your instance from a given version to the next version in the upgrade path.

Starting with Flow Production Tracking 7.x, an upgrade path from the last minor of a major to the latest minor of a major is possible. For example, updating Flow Production Tracking from 6.3.18.0 to 7.1.1 can be done in one step. However, upgrading to 8.x, when available, will require updating to the latest version of 7.y. See Flow Production Tracking official update path for more details.

Please note that if a major change happens in a version, for example introducing a new version of Ruby, an additional jump may be required.

Updating a Flow Production Tracking cluster

Updating a Flow Production Tracking cluster with multiple application containers requires additional steps.

1. Before starting the update sequence, all the application containers must be put in maintenance mode.
2. Choose a container that will be used for the step by step update sequence.
3. The secondary application containers can be updated after the main application container has been updated to the target version. Secondary application container updates don’t have to follow the version-by-version update.

Step-by-step update procedure

Every update, staging or production, is performed the same way. Here is a detailed overview of that process’ steps:

1. Just before the update

   1. Client to make sure all any scripts accessing directly the database, if any, are stopped.

2. Flow Production Tracking shutdown

   1. Flow Production Tracking application container(s) are stopped.

3. Production backup

4. Main App Server upgrade

   1. Upgrade in many iterations, version by version, depending on the version going from or to.

5. API and Secondary App Server upgrade

   1. Secondary servers don’t need to be updated in steps, because the data migration has already been done for the Main Application Server.

6. Surface testing (Sysadmin)

7. In-depth testing (Production Team)

FAQ

Are backups intrusive?

Backups are usually non-intrusive, both at the database and system level. A backup will add a bit of load on the database, but this is usually unnoticeable.

The only time a database backup can be slightly intrusive is when it coincides with table modifications (ALTER). In Flow Production Tracking, tables are modified when new fields are added to an entity. When doing a backup, it is better to prevent field additions. Even then, the impact should be limited, since we timeout lock acquisition for field additions very quickly.

Production and staging are hosted on the same database. Will a restore or sync or upgrade of my staging site impact production?

Even if stored on the same hardware, different sites are segregated. However, some load may be added on the server during the operation. If your servers are already accessed heavily, it may be preferable to execute the backup outside peak hours.

Are event trims intrusive?

Yes, they can be intrusive, especially if the quantity of events trimmed is important. Removing a huge quantity of data will likely trigger PostgreSQL vacuuming. That can add significant load on the system. It is therefore preferable to notify the team, and to execute it outside production hours.