Fine displacement critically depends on the specification of a cache size limit, because otherwise the fine tessellation results would not flow through the cache but accumulate until memory runs out. mental ray has a default cache limit of 512 MB.
A good choice is half the amount of physical RAM, or 500-800 MB on 32-bit machines, whichever is smaller. If the number is too large, the operating system may run out of virtual address space; if it is too small, mental ray will perform too many cache flush operations.