Porting Selection from Viewport 1 to 2

Introduction

Porting selection functionality in your plug-ins can be seen as a separate task from porting drawing, depending on the interfaces used and the version of Maya used.

As of Maya 2016, native Viewport 2 selection is available.

Before the 2016 release, selection could only be performed using existing Viewport 1 interfaces. The Viewport 1 interface drawing must use OpenGL even if the draw code for Viewport 2 is using a different drawing API (such as DirectX11).

The Viewport 1 interfaces are dependent on OpenGL software pick buffers for selection. Software pick buffers are not available when you are using a core profile context. If a legacy OpenGL profile is available, then this functionality may be available.

A choice of OpenGL profiles is available on the Windows and Linux platforms. On certain configurations of Mac OS X, only core profile contexts can be instantiated.

Interfaces

Manipulators

Manipulators are not selectable at the object level for both Viewport 1 and Viewport 2 as they do not exist as scene elements.

The selection level that is available for MPxManipulatorNode instances is defined at the "sub-object" level, also called the "handle level".

Manipulator containers (MPxManipContainer) that are composed from existing manipulators require no additional code to support selection as this support is defined at the manipulator level. This is true for both Viewport 1 and Viewport 2. It is possible to add additional draw (and hence selection) for a container as desired.

Locators

Locators (MPxLocatorNode) are only selectable at the object level.

For locators that draw very simple geometry, or when there will be very few occurrences of a given locator in a scene, it is acceptable to use the MUIDrawManager interface for drawing and selection.

For "click" or closest point single selection, there is no change in the interface logic used. The members MPxLocatorNode::useClosestPointForSelection() and MPxLocatorNode::closestPoint() should be implemented for Viewport 1 selection. For Viewport 2 selection, the closest point is computed automatically from the geometry data and hence the APIs are not used.

Anything that is more complicated or requires better performance falls into the category of "shapes", covered in the next section. They should be implemented using an MPxGeometryOverride or MPxSubSceneOverride

Shapes

More complex object types such as MPxSurfaceShape and MPxComponentShape, (or MPxLocatorNodes that require more complex or persistent drawing) should use render item (MRenderItem) -based interfaces: MPxGeometryOverride or MPxSubSceneOverride.

Using MUIDrawManager for complex objects is not recommended, though Viewport 2 selection can still work.

Using MPxDrawOverride if Viewport 2 selection is required is not recommended as it does not support raw draw code.

Viewport 2 Selection Handling of Components

When Viewport 2 selection is enabled (as of Maya 2016) the selection interfaces on MPxSurfaceShapeUI are no longer called.

Functionality in the following Viewport 1 interfaces:

• MPxSurfaceShapeUI::select()
• MPxSurfaceShapeUI::selectUV()

are handled by the following Viewport 2 interfaces:

• MSelectionContext: Information about the state of selection including selection level.
• MIntersection: Hit intersection information
• MSelectionInfo: Used with MPxGeometryOverride::refineSelectionPath()
• MPxComponentConverter : utility to handle MIntersections and convert index buffers to MFnComponents.

  Registered / deregistered on MDrawRegistry using registerComponentConverter() / deregisterComponentConverter()

• For MRenderItem:

  A "selection" mode (DrawMode) allows indication of whether a render item should be used for selection.

  There is the ability to set a selection mask (MSelectionMask) to indicate the selection type.

• MPxSurfaceShape: has been extended with the ability to return a selection mask (MSelectionMask) to indicate that component selection is supported in Viewport 2.

Notes for Complex Shaders

For hardware shaders that displace input geometry (for example, a mesh that is drawn at a different location, or with more details, such as tessellation), or that generate complex geometries at the shader level (for example, shaders that use a geometry stage to draw complex geometry such as a particle blob on a very basic geometry such as a point):

• For the Viewport 2 selection path, when creating the selection render items, custom hardware selection effects can be added that replicate the displacement/generation of the beauty pass, but that draw flat shaded geometries using a color defined by the hardware selection protocol. These items will then be selectable using camera based selection mode in a WYSIWYG manner in the viewport.
• For the non-Viewport 2.0 selection path, draw code in Viewport 1 interfaces which displaces geometry is always ignored.

Notes for Geometry

It is possible to reference or extract out the geometry used for render items (MRenderItems) and to use raw resource handle to draw with. This can help to avoid duplicating memory if using Viewport 2 draw and Viewport 1 selection.

Camera based selection

In camera based selection mode, the scene is painted using a single plain color for each distinct scene element that can be selected. The shader should use the following formula to generate the plain color used for selection:

int colorID = [HWS_PrimitiveBase];
if (![HWS_ObjectLevel])
    colorID += PrimitiveID;
if ([HWS_InstancedDraw])
    colorID += InstanceID * [HWS_PrimitiveCountPerInstance];
float4 color;
color.x = float(colorID & 0x000000FF) / 255.0;
color.y = float((colorID & 0x0000FF00) >> 8) / 255.0;
color.z = float((colorID & 0x00FF0000) >> 16) / 255.0;
color.w = float((colorID & 0xFF000000) >> 24) / 255.0;

PrimitiveID and InstanceID are system values that can be retrieved using the correct GLSL/HLSL API (SV_PrimitiveID and SV_InstanceID for DX11, and gl_PrimitiveID and gl_InstanceID for GLSL). Square brackets are used to indicate any variable that is declared using the aforementioned semantics.

Refer to the NormalView_Selection technique definition in the WaterSimulation shader example for a demonstration of how to select displaced geometry, and to draw vertex, edge, and face components at their correct displaced positions. For the GLSL version, see the WaterSimulation.ogsfx and PS_HWSelection.ogsfh files in the presets\GLSL\examples folder of the Maya installation directory. For the DX11 version, see the WaterSimulation.fx and PS_HWSelection.fxh files in the presets\HLSL11\examples folder.

Selection mask usage

MPxLocatorNode::getShapeSelectionMask() or MPxSurfaceShape::getShapeSelectionMask() can be overridden to return the desired selection mask for use in Viewport 2.0.

This can be either an existing internal mask or a custom one that is registered by using MSelectionMask::registerSelectionType(). For example, if a custom type called "my_selection_type" is registered, the getShapeSelectionMask() would return MSelectionMask("my_selection_type").

By default, custom selection types are disabled. To enable a custom selection type, do as follows:

"selectType -byName \"my_selection_type\" 1"

where "my_selection_type" is your custom type.

Also available with a custom selection type is the ability to set the priority via the selectPriority command. In this example, assume that the registration originally set the priority to be 2.

// Change it to be lower in priority 
selectPriority -byName "my_selection_type" 1;

// Change it to be higher in priority 
selectPriority -byName "my_selection_type" 3;

An example use case is when an environment encompasses the scene. The user may either not be able to or accidentally select the environment instead of the objects in the scene. To mitigate this, the environment's selection priority can be lowered.

Selection of a particular type can also be disabled to disallow selection:

"selectType -byName "my_selection_type" 0;

See the footPrintNode geometry override implementation (footPrintNode_GeometryOverride Developer Kit example) for an example use of MPxLocatorNode and a custom selection type. See apiDirectionalLightShape and apiMeshShape for example use of internal masks.