Sub-Object Hit Testing

The definition of a sub-object element depends on the modifier. For example it may be a modifier's gizmo or part of its gizmo. Consider when an FFD modifier is in vertex sub-selection mode, the sub-object elements that are being hit tested are the control points of the lattice. Edit modifiers usually hit test components of the object in the pipeline such as vertices or faces.

When a modifier's HitTest() method is called, it traverses the sub-elements of the current sub-object selection level and checks each one to see if it has been hit. If so, it registers a hit record with the active viewport. A hit record contains the following information:

• A pointer to the node that was hit. Modifiers may be instanced across multiple nodes, so the instance that was hit must be identified.
• A pointer to the ModContext.
• The 'distance' of the hit. To classify as a hit, the sub-object component must be within some threshold distance of the mouse. This distance is recorded in the hit record so that of all the hits below the threshold, the one that is the closest can be identified. What the distance actually represents depends on the rendering level. For wireframe modes, it refers to the distance in the screen XY plane from the mouse to the sub-object component. In a shaded mode, it refers to the Z depth of the sub-object component. In both cases, smaller distances indicate that the sub-object element is 'closer' to the mouse cursor.
• A general unsigned long value. Most modifiers will just need this to identify the sub-object element. The Edit Mesh modifier uses the value to store the index of the vertex or face that was hit.
• In case the 4 bytes above isn't enough space to identify the sub-object element, a pointer to a HitData class is included. To use this, a developer would define a class derived from this class that would contain the necessary data. The HitData class has one member function, a virtual destructor, so the derived class can be properly deleted when the HitRecord instance is deleted.

When Modifier::HitTest() is called by 3ds Max on an edit modifier, the edit modifier needs to hit test elements of the object that is flowing through the pipeline. This presents a problem since a modifier usually only has a pointer to this object when its ModifyObject() method is called. In order to be able to hit test the object, a modifier must keep the object cached. Caching the object the modifier is modifying is generally useful as well. For example, the Edit Mesh modifier keeps a cache of the mesh it is editing. When the Edit Mesh modifier is evaluated, if it has a cache it can simply return the cache. This way, as the user applies incremental edits to the mesh, all of the previous edits don't need to be reapplied. Instead, when an edit is made it is applied to the cache and stored away in some form in case the modifier needs to be reapplied.

Modifiers can be instanced, so storing an instance-specific cache in the modifier might not be the best idea. It is the ModContext specifically in the ModContext::localData field, where instance-specific modifier data belongs. This is where edit modifiers can put a cache of the object that it modifies.