Scaleform GFx FAQ: Graphics Rendering and Special Effects

This section covers topics including rendering artifacts, anti-aliasing, masks, fillers, and buffers.

What type of vertex, geometry, and pixel shaders are used in GFx?

Scaleform GFx uses vertex shaders to perform 2D shape transformations and pixel shaders to apply color effects and implement EdgeAA. The pixel shaders used are usually very simple, with pixel shader model 1.1 being adequate to implement all of the Flash blending functionality. Different shader implementations are provided in HLSL, Cg,, PS 1.1, and PS 2.0 assembly formats, depending on the applicable platforms. On older hardware a fixed-function (FF) pipeline is used, which is usually adequate for most of Flash rendering. With an FF pipeline enabled, texture EdgeAA and some color blending effects may be lost. If you plan to target older video cards, we recommend testing your content on them to make sure that it appears as intended.

Is it possible to apply user-defined shaders to shapes or UI elements?

The easiest and recommended way to leverage pixel shader effects is by first rendering the Flash content to a texture and then applying the pixel shader as a second pass.

Custom pixel, vertex, and geometry shaders can be integrated into the GFx rendering process if the game implements its own GRenderer class. The later modifications may, however, be rather involved because GFx already has a number of shaders that are selected dynamically based on EdgeAA mode and other settings. We are investigating approaches that would allow you to stitch your own custom shaders on top of our shader logic in the future SDK versions, but do not have an exact date for it yet.

Is Anti-Aliasing (AA) possible without Full-Scene Anti-Aliasing (FSAA)?

Yes! Scaleform has developed a new breakthrough vector graphics anti-aliasing technology we call Edge AA, which is compatible with all modern graphics hardware and does not use FSAA or complex pixel shaders. EdgeAA rendering is enabled when the GFxRenderConfig::RF_EdgeAA flag is set.

In contrast to shapes, small text output is always anti-aliased in GFx, even when EdgeAA is not enabled. To achieve high quality text rendering GFx caches glyph textures and uses a highly optimized software renderer similar to the one found in AGG and FreeType.

What about AA with transparency?

Scaleform GFx Edge AA technology works with all supported blending modes and partially transparent shapes, without using separate bitmap buffers. The architecture supports AA compositing of Flash files with transparencies on top of an existing image or 3D scene.

Do you support AA with masks?

No. It is not possible to implement anti-aliased masks by utilizing the stencil/depth buffer method without full-screen AA; anti-aliased mask implementation would be complex and significantly more expensive. We intend on implementing this and the solution is expected in GFx 4.0.

How can I use masks with GFx?

It is possible to use masks when working with GFx. However, our usual recommendation is to keep the total number of masks on screen in single digits—the fewer the better—only using them to clip content when absolutely necessary. Note that in many cases the effect that artists use masks for does not actually require a mask. In particular, it is common to use a mask to cut a shape out of a bitmap. The same thing can be achieved much more efficiently by simply applying a bitmap to a shape directly in the Flash Studio, in which case you also get anti-aliasing. Multi-bit stencil buffer is required for nested masks. We support using depth buffer for masks if you don't have stencil buffer available; however, if you use depth buffer you will only get one level of masking and no nested masks.

Is there Normal mapped UI support?

There is no built-in bump mapping support; however, it is possible to apply custom shaders to a flash-rendered scene with an additional rendering pass, as discussed earlier.

What Z-buffer writing operations exist in Scaleform GFx?

The Z-Buffer is used for masks when stencil is not available. See the stencil topic below.

Is it possible to change the Z-order of the elements (e.g., to draw a Flash sprite partially behind a 3D object), or is Scaleform GFx always on top of the screen?

Using Scaleform GFx, developers can render Flash on any surface, both flat 2D interfaces and texture mapped onto 3D objects in the scene. Since the current renderer does not rely on Z-Buffer, developers can draw objects at the user-specified Z level. In GFx 2.2 we introduced MovieClip.rendererFloat and MovieClip.rendererString extension properties accessible from ActionScript. The value of these properties is passed to the GRenderer::SetUserData virtual function and can be used to pass Z-data or drive user-defined special effects.

Is Stencil buffer required?

Stencil buffer is used to support SWF mask layers. It is possible to develop Flash files which do not use masks, in which case stencil is not used. Since rendering surfaces are configured externally by the developer, the current renderer implementation will use the stencil buffer if it is available and simply drop mask layer support if it is not. Z-buffer can be used for masks when stencil is not available.

Do you work with Alpha buffer?

The current renderer does not make use of an Alpha buffer.

Is it possible to avoid using Stencil for clipping? I am trying to carve out a part of a large texture with a mask and am concerned about performance.

There is no way to do clipping in Flash besides using the stencil. For images, the other option you have is using the ActionScript MovieClip.beginBitmapFill to draw a shape with a BitmapData object loaded from your game code through BitmapData.loadBitmap("img://name"). Bitmap fills allow you to specify the matrix transform applied to the image, which you could modify as you scroll. As described in GFx forum posts, passing the "img://" prefix causes the image to be pulled through GFxImageLoader and thus allows you to bind it to an in-game texture.

Please note that using the Drawing APIs may incur significant CPU overhead if done every frame, so you need to test it to make sure you are comfortable with the results.

We are encountering "seams" (thin horizontal and vertical lines) in GFxPlayer, but we don't see them in Flash Player. Why is this happening and how can we fix it?

These seams are being caused by the EdgeAA, where two bitmaps or disconnected shapes are aligned in Flash. Seams are fundamental to EdgeAA and cannot be avoided when it is enabled, unless your coordinates coincide with the screen pixel grid. Any time two shapes meet in such a way that their edges do not exactly fall on the edges of the screen pixel grid, there can be a seam.

To avoid seams and keep EdgeAA, you can expand all your triangles by half a pixel, so that they overlap each other a little bit. The only limitation here is that any shape drawn this way cannot be alpha-blended. Your stitching will work flawlessly if the images meet at integer pixel edges, but those must be screen buffer integer pixels and not the original shape coordinates. This means that if you are scaling the viewport (as our player does by default unless Stage.scaleMode is set to "noScale"), or the shape, or positioning any of the parent movie clips at non-integer coordinates (as you do in your FLA), you will get seams.

If you turn off EdgeAA, many seams will go away; however, you could still get pixel gaps unless you make sure that your coordinate calculations are absolutely identical for adjacent triangles. An alternative is using the ActionScript Drawing API. If you draw your shape with a single path there will be no seams.