some non-maya pipeline UI stuff…

I’ve been working for the last few months on a pipeline system for Wolf & Crow here in Los Angeles. My efforts are typically focused on Maya pipelines, which I’ll be documenting later, but I wanted to show off a little something I made that’s usually outside my repertoire… web programming.

The company needed an interface to handle mundane tasks like creating new jobs and assigning job numbers to them, creating new assets, and creating new sequences & shots for projects. Normally this would be as easy as duplicating a template folder, but the pipeline here makes extensive use of symbolic links in order to tie multiple Maya workspaces together in a way that’s as efficient as possible (for example, the “sourceimages” folder of every Maya project in a job can be symlinked to a single “textures” repository). Symlinks can’t be created in Windows, although Windows will happily follow symlinks served from a Linux server. So the bulk of the work has to be done on the server itself via Python or Bash scripts.

Prior to the web app, the standard procedure was to run plink.exe (an SSH command-line utility) inside a Python script using the subprocess module (see my earlier post on subprocess), which would pass a command over to the Linux file server to create all the symlinks. Or you could just SSH into the server using putty.exe and run the command yourself. This was clumsy, though, since you either needed to have Maya open to run the Python script through an interface, or you had to be comfortable enough with SSH and the Linux command line to run these scripts.

Instead, there’s a (sort of) sexy web interface that anyone can run from a browser! Here’s some examples.

Click below to see a whole bunch of code.

There are three languages at play here: JavaScript/JQuery handles the front-end and forms, including using AJAX to load each section into the main grey <DIV> when the user navigates; PHP handles user authentication, and Python does all the heavy lifting on the server in the background. Each of the sub-sections that are loaded have a PHP include at the very beginning that checks to see if the user’s session is valid and immediately redirects back to the login page if they’re not. The Python scripts are being run as CGI via AJAX calls from JQuery.

Here’s an example of the JQuery function that runs when the form is submitted:

function submitJob() {
    // gather form vars and send to python
    var jobInput = $('#projectInput').val();
    var jobNum = $('#jobNumber').val();
    var jobName = jobInput + '_' + jobNum;
    var serverName = $('#serverSelect').val();
    // run ajax query
    $.ajax({
        type: "POST",
        data: {
            jobName: jobName,
            serverName: serverName,
        },
        url: "/cgi-bin/createProject.py",
        cache: false,
        error: function(xhr, status, errorThrown) {
            alert("Submission error! Please wait a minute and try again.");
            console.log("Error: " + errorThrown);
            console.log("Status: " + status);
            console.dir(xhr);
        },
        timeout: 5000,
        success: function(result) {
            // result is a string coming from the python script.
            // set #resultsText and show the element.
            $('#resultsText').html(result);
            $('#resultsText').fadeIn(600);
            $('#submitBtn').prop('disabled',true);
            // get job number again to prevent conflict
            getJobNumber();
            // $('#projectInput').val("");
       }
    }); // ajax
}; // submitJob

The form passes all of its values via HTTP GET to a Python script running as CGI. (Apache servers don’t do this by default; you need to configure them specifically to allow .py as an extension). Here’s what Python is doing:

#!/usr/local/bin/python2.7

import os, sys, cgi, re
import subprocess

# header for http POST/GET
httpHeader = "Content-Type: text/html\n"

def createAsset(serverRoot,project,category,asset):
    # run shell script in the appropriate directory.
    output = ''
    path = os.path.join(serverRoot,project,'assets',category)
    shellcmd1 = 'cd ' + path + '\n'
    shellcmd2 = 'addAsset -auto ' + asset + '\n'
    # symlinking won't work from a remote mount point, so we have to
    # SSH into the host machine and run the commands from there.
    # we'll create the subprocess to SSH into the correct server, then
    # use stdin.write() to send commands.
    sshcmd = ''
    if serverRoot == '/mnt/SERVER/WOLF':
        sshcmd = ['ssh', '-T', 'user@192.168.1.100']
    else:
        sshcmd = ['ssh', '-T', 'user@192.168.1.101']
    sshProc = subprocess.Popen(sshcmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
    # execute commands
    sshProc.stdin.write(shellcmd1)
    sshProc.stdin.write(shellcmd2)
    sshProc.stdin.close()
    # read output
    for line in sshProc.stdout.readlines():
        output += (line + '\n')
    sshProc.stdout.close()
    return output

if __name__ == "__main__":
    # get HTTP GET vars
    form = cgi.FieldStorage()
    server = form.getfirst('server', 'empty')
    project = form.getfirst('project', 'empty')
    category = form.getfirst('category', 'empty')
    asset = form.getfirst('asset', 'empty')
    server = cgi.escape(server)
    project = cgi.escape(project)
    category = cgi.escape(category)
    asset = cgi.escape(asset)
    
    # parse server
    if int(server) == 1:
        server = '/mnt/SERVER/WOLF3'
    else:
        server = '/mnt/SERVER/WOLF4'

    ret = createAsset(server,project,category,asset)
    print httpHeader
    results = 'Response from server: \n' + ret
    print results

There’s some weird stuff going on here. Most importantly, the web server that’s running these operations is NOT the same as the file servers! This causes some problems with creating symbolic links, and I couldn’t figure out an easy workaround to this problem. The answer was to SSH from the web server to the file server and run the “addAsset” command (which actually creates all the folders and symlinks) directly from there. I set up public/private SSH authentication between the web server’s Apache user account and each of the file servers so that it could log in without a password.

The subprocess here is just writing each of the shell commands needed (“cd /some/dir”; “addAsset assetName”) in order, followed by ‘\n’ which means “enter.” It writes these to “stdin” of the subprocess, so it’s as if a user typed in the commands. After these commands are done, the stdin is closed, and I return the output (stdout) of the script.

The if __name__ == "__main__" condition is pretty common for Python scripts that are meant to act from the command line… it just means that if the script isn’t being imported, it should run the following commands immediately. Since the script is being executed by the Apache server as CGI, the commands inside the conditional will run.

The cgi module can get form variables via HTTP GET or POST by name. I run them through cgi.escape() in order to somewhat safeguard against form injection or other potential weirdness, and then send those variables right to the createAsset() function.

Just before I print the results, I have to print an HTTP header so that the web server recognizes the return data as valid HTML. Apache sees the line “Content-Type: text/html\n” printed at the top of the output here, is satisfied that it’s actually HTML, and so it will send the information back to JQuery without a hitch. The JQuery, upon a successful transaction, prints the results to ‘#resultsText’ so the user can get useful feedback.

Next up I’ll post some examples of what the Maya-centric pipeline is doing.