The mental ray renderer can generate shadows by ray tracing. Ray tracing traces the path of rays sampled from the light source. Shadows appear where rays have been blocked by objects. Ray-traced shadows have sharp edges.
Ray-traced shadows
Turning off caustics makes the outlines of shadows in this scene easier to see.
You can tell the mental ray renderer to use shadow maps instead of ray-traced shadows. This can improve performance at a cost of accuracy.
Shadow controls are on the Render Setup Dialog 
Renderer panel Shadows & Displacement rollout.
Shadow Generators and the mental ray Renderer
Light objects in 3ds Max let you choose a shadow generator: Ray Traced, Advanced Ray Traced, Shadow Map, and so on. Because the mental ray renderer supports only two kinds of shadow generation, ray tracing and shadow maps, some of the 3ds Max shadow generators aren't fully supported.
In 3ds Max, a special shadow generator type, mental ray Shadow Map, is provided to support the mental ray renderer. If shadows are enabled (on the Shadows & Displacement rollout of the Render Setup dialog) but shadow maps are not enabled, then shadows for all lights are generated using the mental ray ray-tracing algorithm. If shadow maps are enabled, then shadow generation is based on each light’s choice of shadow generator:
- mental ray Shadow Map Shadows are generated using the mental ray shadow-map algorithm.
- Shadow Map Settings on the Shadow Parameters rollout are translated into a mental ray equivalent before shadows are generated. The quality of shadows generated this way might not always meet expectations.
- Area Shadows, Advanced Ray Traced Shadows, or Ray Traced Shadows Shadows are generated using the mental ray ray-tracing algorithm.