Although the mental ray renderer is relatively easy to use once you’ve set it up correctly, there are several "gotchas" that you might encounter immediately, especially if you’re primarily accustomed to the 3ds Max scanline renderer and its workflow. For example, see 3ds Max Materials in mental ray Renderings. Following are some basic rules of thumb for using mental ray in 3ds Max:
Using Lights with the mental ray Renderer
When you set up a scene for rendering with the mental ray renderer, keep the following tips in mind:
- The Overshoot parameter for lights doesn't work when you use mental ray to render shadow-mapped shadows. To use Overshoot, use ray-traced shadows.
- Excluding an object from shadow casting doesn't work when you use mental ray to render shadow-mapped shadows. To exclude objects from shadow casting, use ray-traced shadows. (The Exclude button is on a light's General Parameters rollout.)
- When you assign a map to object shadows in the light's Shadow Parameters rollout, the mental ray renderer does not recognize the toggle for the map (to the left of the Map button), and renders the map whether the toggle is on or off. To stop using the map, you must click the Map button and in the Material/Map Browser, assign NONE as the map type.
- Using the default scanline renderer, you can set a light to have a value of zero, with a shadow color of white, and a shadow density of −1. With these settings, the light casts shadows but does not illuminate the scene. To get the same effect using the mental ray renderer, the light value must not be zero. Instead, set it to a value close to zero (for example, 0.001 or 0.001).
- The mental ray renderer disregards the bias parameters in the Shadow Map Params rollout and the Ray Traced Shadow Params rollout.
Ray Tracing
The mental ray ray tracer is fast and provides excellent quality images, but it’s important to use it correctly.
The mental ray renderer does not fully support cubic maps for Reflect/Refract maps. It uses them if they have already been generated by the default scanline renderer, but it does not generate them. If Source 
From File is active and the mental ray renderer can find the six cubic maps, it uses them. If Source Automatic is active, or if the cubic maps cannot be found, the mental ray renderer generates ray-traced reflections or refractions instead.
Ray Tracing Setup
On the rendering menu, Ray Tracer Settings and Raytrace Global Include/Exclude are disabled while the mental ray renderer is active. These controls adjust ray-trace settings for the scanline renderer only. The settings of these controls have no impact on the mental ray renderer. The ray-tracing controls for mental ray appear on the Renderer panel Rendering Algorithms rollout.
Ray Tracing Rules of Thumb
Say you’re rendering a (lathed) wineglass, with an inner and outer surface and a piece of geometry representing the wine. The wine geometry is just slightly smaller than the inner surfaces of the wineglass, and capped with a flat top. Now, you go to render the glass. After rendering the scene, however, there’s something wrong: the inner surfaces of the glass don’t seem reflective enough, and the wine isn’t refracting properly. What’s wrong?
It’s possible that you have the number of reflections and refractions set too low for the number of surfaces you have. To check this, go to the Renderer panel Rendering Algorithms rollout and look at the Maximum Trace Depth settings. If you havent changed the parameters, then you should see Max. Reflections and Max. Refractions set to the default of 6, and Max. Depth set to 6.
There’s the problem: you actually have six surfaces that need to be traced by the light rays for both reflections and refractions. The way to always calculate the number of rays needed for a scene is to take the ray-traced objects in your scene and draw an imaginary line through them, originating at the point of view. Then, count the number of surfaces the line intersects.
For the wineglass and wine, you need at least six reflections and refractions that correspond to the following surfaces:
- Near outer glass surface (“near” relative to your Camera viewpoint)
- Near inner glass surface
- Near wine surface
- Far wine surface
- Far inner glass surface
- Far outer glass surface
Therefore, increase the value of Max. Depth to 12.
Caustics and Global Illumination
Before rendering with caustics, there are several things you need to set up in your scene:
- For caustics to work properly, the generating object must use a material that contains some degree of shininess, reflectivity, or refraction. Assign a Raytrace or other map as either a Reflection map or Refraction map before you render caustics.
- Most often, you’ll be using very shiny, highly reflective materials (such as chrome and other metals), or transparent or translucent materials (such as glass goblets or water), to generate caustics in your scene. If you’re using a glassy material, make sure it’s double-sided to create the proper results.
- Make sure you have object properties set to Receive Caustics or Generate Caustics (or both). To set up these properties, right-click an object and choose Properties. For example, if you’re rendering a wineglass on a tabletop, you probably want the wineglass both to generate and receive caustics (so that caustics are scattered within the glass itself), and the tabletop only to receive caustics (unless it’s chrome, say, instead of wood).
- If the rendering of your scene is washed out by light, double-check the Multiplier settings: one in the Basic group of the Final Gather rollout, and one each in the Caustics and Global Illumination (GI) groups of the Indirect Illumination panel Caustics And Global Illumination rollout. These apply to all lights in the scene. Reducing the Multiplier values can eliminate washout.
If a single light object is causing the problem, you can reduce the Energy multiplier's value in that light object's mental ray Indirect Illumination rollout, available on the Modifier panel.
- To improve the quality of caustics, go to the Caustics group of the Caustics And Global Illumination rollout and increase the Max Num. Photons Per Sample setting.
- Be careful of the total number of photons you’re emitting: A very high number (100,000 and above) can dramatically increase your rendering time. Then again, for some simple scenes, you might actually be able to set these to 1,000,000 and still render in an acceptable amount of time. Warning: The number of photons specified for each light indicates the number of photons that need to be stored for each light, not the number of photons to be shot. This is an important distinction: If a light points in a direction where there is no surface, the mental ray renderer might shoot photons forever. In the Messages Window, the mental ray renderer displays warnings that no photons are being stored. To avoid the slowdowns related to this issue, make sure that every light points in the direction of a surface (this is sometimes impossible to do with omni lights). Another way to avoid this problem is to add a big sphere around your entire model.
- In general, use an exposure control. The mr Photographic Exposure Control works particularly well for adjusting overall exposure.
Coincident Faces
When it encounters coincident faces, the mental ray renderer can produce artifacts, because it can't decide which face is nearer the camera (neither is). To fix this, move or scale one of the objects so faces are no longer coincident.
Backface Culling
mental ray rendering correctly performs backface culling, and renders one-sided faces much as the scanline renderer does.