I’ve been messing around with using Ptex in VRay for Maya and ran into a particularly weird little problem involving a normal map exported from Mudbox. It’s probably easier to show it than to tell about it (Fig. 1):
Fig. 1. Check out the wax wrapper up top… all chunky and gross.
That wax wrapper looks terrible… not at all like the original sculpt. The mesh is set to render as a subdivision surface (using the custom VRay attributes from the Attribute Editor), but the details are all mangled like it’s not subdividing the surface at all.
Even weirder is what happens when I only render a small region of the image (Fig. 2):
Fig. 2. The details are suddenly cleaner! All I changed was enabling region render.
This was really confusing, and although I’m still not 100% sure what’s going on in VRay to cause this (I’m fairly certain this is a bug), there is at least a solution.
What’s probably happening here is that choosing to render this mesh as a subdivision surface is changing the point count of the object BEFORE any Ptex information is applied to the mesh during rendering. Ptex is very sensitive to your geometry… any change in point count could potentially break things. You’re not allowed to polySmooth objects that will have Ptex textures, for example, or Maya won’t know what points to assign Ptex information to.
The way to get around this is to put the object into a VRay displacement set. Assign displacement and subdivision attributes to the displacement set like you normally would, but don’t attach a displacement map, and make sure the Displace Amount is set to 0. Rendering this way gets you the more correct image (Fig. 3):
Fig. 3. This looks a lot more like the original sculpt from Mudbox.
This seems like buggy behavior to me more than some technical thing I’ve overlooked, but thankfully the workaround is pretty minimally difficult. If you have any better insight as to what exactly is happening here or if there’s a less hacked way to prevent it, I’d love to hear it.