automated path substitution in maya.

I’m trying to build a system right now that can automatically substitute environment variables for paths on a per-scene basis. When a user opens a Maya file, I want to parse that file and set temporary environment variables called $SHOW, $SEQ and $SHOT that point to the show’s root folder, the sequence’s folder in that show, or the shot in that sequence. The variable names and paths I’m trying to get are pretty specific to the current pipeline I’m working on, but this essentially lets me use relative paths in Maya but WITHOUT using Maya’s outdated and inflexible workspace system. I don’t want an individual Maya project for every single shot and asset in the entire show, and I don’t want all of my Maya scenes to be forced to live within a single workspace, either.

I’ve solved this problem before by using symbolic links to basically force all of the Maya projects generated for every single shot and asset to link to the same place. This makes for a pretty busy-looking file system, though, and symlinks only work on Unix-based servers (i.e. not Windows). This system I’m building now looks a lot more like Houdini’s global variables… as long as I define $SHOW or $SHOT or whatever, I can path a reference or a texture to $SHOT/assets/whatever.xyz and it doesn’t necessarily have to be within a Maya workspace, and it’s also not an absolute path.

Read more below…

You can set environment variables in Maya using the command putenv. If you define an environment var using putenv butt /path/to/butt, from then on you can substitute $BUTT for /path/to/butt. Great!

The trick here is that these variables need to change on a per-scene basis. Any time a Maya user opens a different scene, these variables could be entirely different depending on what show, or part of a show, they’re working on. Even worse, if the file a user is opening contains references, these variables need to be set immediately upon loading the scene, BEFORE any references are loaded at all!

Maya’s scriptJob and scriptNode commands initially seemed like the right bet, but unfortunately neither of these commands can actually interrupt the loading of a file by the time it tries to start resolving file reference paths. Instead we’re going to have to get ugly with OpenMaya API callbacks.

I’m completely worthless at C++, so I tried my hand at doing this with the OpenMaya library for Python. There’s a lot of little gotchas when dealing with Maya at this level, so I’ll try to document everything I can.

First, I define a procedure that I want to run before any file is loaded. We’ll call this setJobGlobals(). The pseudocode looks more or less like this:

import pymel.core as pm
import maya.cmds as cmds

def setJobGlobals():
    path = cmds.file(q=1,sn=1)
    show = getShowName() # this would get the show name by parsing "path"
    seq = getSeqName()
    shot = getShotName()
    # now set env vars
    pm.util.putenv('SHOW',show)
    pm.util.putenv('SEQ',seq)
    pm.util.putenv('SHOT',shot)

The pymel.core.util.putenv() command creates environment variables that will last for as long as the Maya session until overwritten. Next, we have to use the API to create a callback.

import maya.OpenMaya as api
callbackID = api.MSceneMessage.addCheckFileCallback(api.MSceneMessage.kBeforeOpenCheck, setJobGlobals)

Let’s break down that giant command. The MSceneMessage class in the API contains a bunch of functions for generating callbacks for scene-related messages. AddCheckFileCallback can cause a function of your choice to run during certain events related to files. The event we want is the MSceneMessage.kBeforeOpenCheck, which operates BEFORE a file is opened. So now we’re running our setJobGlobals() script before any file would open.

If you tried to run this, though, it would fail, or at least behave unexpectedly. Since this command is technically running BEFORE we open this new scene, the command cmds.file(q=1,sn=1) is getting the path of your OLD file (or no file at all if we’re in a blank scene). So this won’t work. Fortunately, the addCheckFileCallback command will pass some parameters to our setJobGlobals() function. If you check the docs for this function, it passes along three parameters: retCode, fileObject and clientData. The parameter we care about is fileObject, a member of the MFileObject class. We can get the filename from this object in our little function. So let’s go back and edit it:

import pymel.core as pm
import maya.OpenMaya as api

def setJobGlobals(retCode,fileObject,clientData):
    path = fileObject.rawFullName()
    path = str(path) # convert rawFullName into a normal Python string
    # blah blah blah, same as above

callbackID = api.MSceneMessage.addCheckFileCallback(api.MSceneMessage.kBeforeOpenCheck, setJobGlobals)

This will almost work. There’s one or two more gotchas. First, the callback is going to interrupt the normal opening of a Maya scene, so if you run this, your environment variables might get set all nicely, but no file will actually open. So we have to explicitly tell Maya to open the goddamn file. We can use this API command at the end of our original function:

api.MFileIO.open(path)

However, now we run into another problem… the callback is going to run again as soon as we command Maya to open a file. The callback will run recursively until Maya gives up and hits its recursion limit. So we will need to run the function, DELETE the callback, open the Maya file we want, then regenerate the callback (for the next scene we might open). Annoying, right?

To remove a callback, we just need the callback’s ID. We get this when we create the callback in the first place. We can keep this number laying around as a global variable (as much as I hate them). Here’s what the final code might look like:

callbackID = -1
  
def setJobGlobals(retCode,fileObject,clientData):
    ''' Set SHOW, SEQ, SHOT vars based on scene path. '''
    global callbackID
    path = fileObject.rawFullName()
    path = str(path)
    show = getShowName() # this would get the show name by parsing "path"
    seq = getSeqName()
    shot = getShotName()
    # now set env vars
    pm.util.putenv('SHOW',show)
    pm.util.putenv('SEQ',seq)
    pm.util.putenv('SHOT',shot)
    
    # now force load of the file, since maya seems to not want to do it anymore.
    # to prevent infinite recursion, we need to nuke the callback, then load the file, then regen the callback.
    removeCallback(callbackID)
    api.MFileIO.open(path,None,True) # flags here will guess maya format and force file load 
    callbackID = api.MSceneMessage.addCheckFileCallback(api.MSceneMessage.kBeforeOpenCheck, setJobGlobals)

def removeCallback(id):
    try:
        api.MMessage.removeCallback(id)
    except:
        print 'failed to remove callback %s' % (str(id))

# RUN THIS AUTOMATICALLY AT START. CALLBACK RUNS BEFORE ANY FILE IS OPENED.
callbackID = api.MSceneMessage.addCheckFileCallback(api.MSceneMessage.kBeforeOpenCheck, setJobGlobals)

We could save this as our userSetup.py, and now this callback will be run anytime a scene is opened, BEFORE the references are loaded! Instead of setting job globals, we could substitute paths based on our operating system (to fix Mac/Windows path inconsistencies), or really do any other damn thing we wanted to do to a file before Maya gets to it.

UPDATE: Roy Nieterau pointed out that there’s no need to manually open the file, and thus no need to destroy and rebuild the callback. The first parameter of setJobGlobals (retCode) is a pointer to a boolean that the callback is looking for. If the boolean is True, Maya will open the file. The catch is that because we’re dealing with a pointer (or a reference, I’m honestly not sure here), we have to use the API’s MScriptUtil class to set the value, instead of just setting it the Python way. MScriptUtil pops up pretty often when you’re dealing with the Python API.

Instead of all that code at the bottom where we remove the callback, open the file, and rebuild the callback, we just need one line of code:

api.MScriptUtil.setBool(retCode,True)

This sets the retCode parameter that’s automatically passed in and out of the callback function by the kBeforeOpenCheck callback itself. As long as this is True, Maya will assume everything’s OK in the callback and open the file. Otherwise it does nothing (assuming that you would catch the error in your callback and provide feedback accordingly).