Upholding Standards

One of the reasons visual effects work is because of standardization.

Here's how I see standardization in visual effects...

Still photographers (we are talking mid-1800's onward) shot in whatever format (aspect ratio) they wanted or could afford. Most still cameras were custom built or built in very limited production runs (as fast as one guy could make them). Same was true for early motion picture cameras until Bell & Howell came out with a mass-produced film camera in the 1920's.

What made Bell & Howell camera especially important to the development of visual effects was that it increased the ability of filmmakers to shoot multiple images within the same frame of film through a variety of clever masking techniques. You could block the light from hitting part of the film the first time the film went through the camera, mask the part you just exposed, and run it through the camera again, this time exposing the part of the film you covered in the first run through the camera.

Capturing motion pictures with any kind of assurance that you would get an image required an agreement between the people flogging cameras and projectors and those flogging the film to go in the cameras. Slot A had to fit into Tab B (within a reasonable tolerance). Things had to fit because it made good business sense -- a weavy, wobbly image would make audiences heave, head for the door or both.

Later when motion picture film size was standardized for a variety of true and apocryphal reasons, camera operators cranked their cameras at speeds which suited them, speeds may or may not have been replicated by the projectionist who may have been taking his or correspondence course in modern electronics between reel changes. This meant that there was no guarantee that the exhibitor would run the film at the speed the filmmaker intended.

Then along came sound in the late 1920's and the speed of the film going through the camera *had* to be standardized so that every character didn't sound like a schoolkid hopped up on helium or Paul Robeson waking up after a long night partying.

So film speed was standardized to a specific frame rate which turned out to be 24 frames per second.

The need for things to "fit" is especially important in creating the illusions of visual effects.

Much of the early history of visual effects involves developing increasingly tighter registration and synchronization of cameras used to shoot the elements for a shot.

Not just the light had to match among elements.

The movement of the elements in the shot -- and the movement of the film through the camera as well -- had to match. The elements had to tell the story without drawing attention to themselves -- and without being copied readily by the joker in the studio down the boulevard. (Mark Cotto Vaz and Craig Barron's highly readable Invisible Art, a loving history of what has come to be called "matte painting," tells of the lengths early filmmakers went to guard their particular tricks of the trade.)

As time progressed, developments in optics and the electromechanical components of film recording and projection permitted closer and closer registration and synchronization, both during the shoot and during the post-production process (in processes such as optical printing).

Nowadays, organizations like SMPTE help keep technical standards in line and confusion to a minimum.

This is all by way of introducing the key visual effects concept of matching.

Though the human visual system does have its drawbacks (for example, we suck at judging distance and scale in the absence of other visual cues). it is very good at noticing things that break a pattern, appear misaligned, wonky, skewed, or "somehow" not right.

To be successful at pulling off a "trick" shot you had to have control over the frame.

My view that the human trait of noticing patterns (even making patterns in one's mind when a pattern really does not exist -- see, for example, Al Seckel's work in optical illusions) has been selected through evolution. The creatures who were more likely to notice the change in patterns were probably better at spotting food and predators, were therefore more likely to survive and pass their genetic makeup to their offspring.

One of the great advantages of computers is that one can write a set of instructions that enable optical, electromechanical and digital equipment to tirelessly repeat a series of actions within tolerances than a human cannot maintain over time. These are sometimes called "passes" and there can be dozens of them in a single visual effects shot which are then combined into a single visual effects shot. Though I am loathe to introduce vfx jargon in this blog, this is a term probably you need to know: motion control.

In motion control, a camera can shoot a variety of elements which can be combined into a single image by creating the illusion that all of the elements were photographed by the camera at the same place and time instead of being shot separately as separate passes and composited together.

(We see here again the world of visual effects playing with the fourth dimension of time.)

Of course, the moving picture itself is an illusion in the sense that it is succession of still frames because of the peculiarity of the human visual system called persistence of vision.

Matching, synchronization, standardization -- all attempts by filmakers to get control over the frame and the movement of elements within the frame.

The 2D / 3D Convention

To understand visual effects its important to understand the issues surrounding the representation of the "real" three-dimensional world on a two-dimensional surface and have that surface convey some notion of the three-dimensional world to audiences.

What seemed OK for a cave dweller -- brushing out a few buffalo-looking smudges on the firelit cave wall and calling it a day -- was not OK for the rampantly curious Renaissance person eager to understand the natural world -- including the heavens.

To the enlightened Renaissance picture-maker picture-making had to be done, well, scientifically, mathematically.

We have been wrestling with geometric perspective (the set of tools and conventions dealing with the represention a three-dimensional scene on a two-dimensional surface) ever since. Perspective and its fun-loving peers -- parallax, scale, apparent motion, depth, aerial perspective (things look bluer and less contrasty the further away they are) have become encoded into software tools and into creative discussions about shots. But we've got to remember that the 2D/3D distinctions are conventions.

Motion pictures are themselves illusions which take advantage of the inability of the human imaging system to distinguish between continuous movement and the illusion of continuous movement caused by a succession of still images present in the same space at a certain standard rate per second.

One might say the scientific, industrial, technological, economic, artistic, scholarly developments until our own day have been but a series of mini-Renaissances. In my view, we are experiencing another mini-Renaissance as digital distribution enables the mass exhibition of 3D versions of effects-driven and animated CG features in what had been strictly 2D venues.

The 2D/3D distinction continues in visual effects today in larger vfx shows where departments are called "the 2D team" and "the 3D team" -- 2D generally referring to compositing of live and virtual elements, 3D generally referring to the creation and animation of virtual characters, objects and enviroments.

Someone once said something like "movies are like life with all the boring bits cut out."

Filmmaking in general and visual effects in particular concerns itself manipulating our conventional, everyday sense of height, width and depth as it tweaks with our conventional understanding of the fourth dimension: time.

Visual effects work because they rely on the ability to deceive the human imaging system (for example the difficulty humans have with shared contours) as well as through a careful manipulation of the audience's expectations and assumptions.

Quite a feat, really.

A Conceptual Approach to VFX

As I mentioned in an earlier post, I think it might be valuable to take a conceptual approach to the history of visual effects.

For an approach that deals with visual effects techniques and methods, my favorite sources (other than from other people working in the business!) are Mitchell's Visual Effects for Film and Television, Fielding's The Technique of Special Effects Cinematography and Rickitt's Special Effects.

These are great if you want to learn the so-called "optical effects," bipack cameras, motion control, glass (matte) painting and our old pal, the Schuftan Process. Mitchell's book in presents an interesting distinction between "special" and "visual" effects -- special effects modify the real world whereas visual effects modify the virtual world.

For a look at digital visual effects methods, check out the historical sections of Mason's CG 101, Alias' The Art of Maya and Kerlow's The Art of Computer Animation and Imaging. And, of course, the yearbook of the vfx industry Cinefex. There are other good sources.

Some people dig this stuff, others do not. I dig it. However, my claim: You do not need to know this stuff to understand what you need to know about visual effects and where it has taken and will take us as visual creators.

Here I will try to explain visual effects in terms of concepts you may already know from your own visual creative endeavors including, but never limited to [and secular and sacred] painting, sculpture, mosiac, sketching, camouflage, stagecraft, collage, graphics, architecture, interior design, textile design, sacred design, fashion, computer programming, printing, calligraphy, music, dance, filmmaking, still photography, stage magic, industrial, scientific and medical imaging and, of course, storytelling.

Which also suggests that people who do visual effects are interested in many things. In my experience visual effects people (at least the ones that've had the most impact on me) have been hyper-curious, inventor types, who make connections between and across technologies and methods. Sometimes you make money in it, sometimes you don't. Nearly always there is a reward at seeing the illusion you created and the story the illusion tells.

I've already discussed the role of story in visual effects. In a nutshell: No matter how groovily high-tech the effects on the screen, all were conceived and created by humans in pursuit of a story created by humans.

Which is why I'm reluctant to use the term "computer-generated imagery."

A computer no more generates imagery than a hammer builds a house.

Let There Be Light

The curious among us have wondered how and why we see what exists around us. Galileo, DaVinci, Aristotle, Einstein, my sixth grade science teacher Mr. Moore.

In addition to creating The Calculus (along with Mr. Leibniz) and practicing a kind of alchemy bizarre even for alchemy, Sir Issac was heavily involved in optics, specifically. the composition of visible light and documenting light's behavior on earth and in space near Earth in predictable mathematical terms.

Much of visual effects is about the creation, manipulation, transport and recording of visible light. its compposition and its behavior. (Of course, visible light is itself a relatively small chunk of the entire electromagnetic spectrum. But that's another post. Or maybe another blog)

So at a fundamental level, the history of visual effects is in some way also the history of humanity's understanding of light's interaction with materials, environments, characters and objects, the various visual elements of the story we are telling.

Not surprisingly, much of the creative discussion during the capture of a live-action visual effect is how to light the shot so that it does not look like a visual effect to the audience.

Much of the creative discussion of a visual effects shot as it moves through a production pipeline is about light and its close pals color and shadow.

Chances are much of the creative discussion surrounding your own creative visual endeavors addresses issues of light and light's role in conveying your intention or story to your audience. You choose to color something a certain way or emphasize some parts or elements of your creation above other parts and elements. Sometimes "reality/verite" is most important, other times the mood is more important. It depends on what you are tyring to convey -- your story.

The nearly continuous study of optics since ancient times (with a little hiccup in the so-called Dark Ages) led to development of perspective during the Renaissance. Many of the conventions established by perspective -- the representation of three-dimensional objects, characters and environments on a two-dimensional surface -- are still with us today in visual effects in terms of the separation between 2D and 3D.

Thankfully, humanity's constant curiosity about light has led to the development of imaging systems based on how we currently understand the properties of light and our ability to generate, capture and store it on a physical (film, tape) or virtual (computer memory) medium.

Farewell to MONSTER HOUSE!

So it’s time I told you all about something amazing...

For the past eight months I’ve been working on a show called MONSTER HOUSE at Sony Pictures Imageworks. My role in the show wraps next week and I will move onto my next adventure in vfx…

MONSTER HOUSE is a performance-capture show (some people call it motion capture, or “mocap” for short, but, hey, the industry still uses “M.O.S” in screenplays, so why not a little hip anachronism for us vfx geeks?). I'll write more about the advantages and risks of performance capture Sony-style in a future post.

I come from a “live action” vfx background, where you get a plate and jazz it up with cool 2D and 3D gags using digital and analog methods. (I hope I’m not getting too technical here! :-)) Something less than all of the shots in the show are vfx shots.

But MONSTER HOUSE is ALL visual effects.

Everything has to be created. No, seriously, I mean every thing.

Every character, object or environment must fit into the world along with all the other characters, objects or environments while at the same time making sure these elements also tell a coherent story and generate an emotional response in the audience. And you’ve got to put these characters, objects or environments in 1200+ shots, each of which may need to touched by multiple people in different orders depending on the needs of the shot and the particular story element it is meant to convey.

Since nobody wants to see what they’ve seen before and all storytellers (ncluding the MONSTER HOUSE team) want to “push the envelope” you have to adopt technologies that may be only a few months older than your greenlight.

So you create all of these characters, objects or environments using technology which was probably not designed to do what you want it to do (by definition, because you want to do something that has never been done before).

And the software vendors upgrade in the middle of your production cycle. And servers crash and…

You also have to do it in far less time than you’d like with far less money than you need.

Only rare humans are capable, much less adept, at surfing the deluge of information that a production entity generates while still maintaining a creative and critical eye.

I have worked with many of these rare people at Imageworks on MONSTER HOUSE.

The work on MONSTER HOUSE has undergone a remarkable transformation even in the short time I’ve been on the project.

As part of my work for the show (my title was Sr. Production Coordinator - gathering and coordinating the bidding estimates of person-days the team projects it will take to create a particular shot which then, you guessed it, translates into mega-dollars), I found myself reviewing the shots and sequences as they progressed. The characters' lives flashed before my eyes again and again on my Linux box.

I have seen the MONSTER HOUSE characters respond to adversity again and again. I feel that I've known some of the characters for years longer than I’ve really “known” them. The world of Mayville exists in some part of my brain as somewhere not real but not imaginary either. But it's definitely a place I will remember.

I'm also sure what usually happens when I'm among smart, motivated and talented people has happened during my tenure on the show -I've learned things that I don't even know I learned.

One of the lasting rewards of production that you get to live in another world in a way that not even the most rabid fan of “just” the movie gets to experience. I believe this desire to be part of more than just the movie is what drives many people into visual effects (it drove me through a long, rambling route to vfx after seeing 2001: A SPACE ODYSSEY in 1970s re-release).

Production is a collective dream that begins, grows and fades with the show as the lasting artifact of that dream.

I feel incredibly fortunate to have been a part of the dream that was, is and will be MONSTER HOUSE.

To VFX or Not to VFX

To recap: Visual effects serves storytelling. A visual effect without a story is a "test."

If you're considering visual effects for your project, ask yourself:

1. What story (or story point) do we need to convey?

2. Can we convey the story without images? If not…

3. Can we create the images required without using visual effects? If not…

4. Can we create the images by using a combination of “practical” and “visual effects” methods?

5. What human, financial and technical resources do we need to create the images?

6. How have other practitioners addressed this challenge before us?

7. What is the least expensive way (time = money) to create the images at the quality level required? [Remember, it called "show-business" for a reason.]

A Working Definition of Visual Effects

Here is my current working definition of visual effects...

"Practices, methods and technologies relating to the creation and manipulation of elements within moving images that enable storytellers to guide an audience’s conception of time, space and/or reality, thereby eliciting a desired emotional response and/or conveying critical story information."

Beyond Words - Story & Visual Effects

Which scenes first come to mind when you think of STAR WARS, STARSHIP TROOPERS, THE MATRIX, KING KONG (the 30's one), CITIZEN KANE, GONE WITH THE WIND, THE WIZARD OF OZ, TITANIC, METROPOLIS, JURASSIC PARK, HARRY POTTER, SHREK GLADIATOR, LORD OF THE RINGS?

Chances are pretty good what you remember is a scene which relied heavily on the art and science of visual effects.

There's also a visual effects approach/category called "invisible effects" which basically means those shots or sequences in which you are unaware of visual effects in the presentation.

There is another more complex (and usually more expensive) approach to visual effects which combines image capture expertise (cinematography), post-production work (washing out colors via a digital intermediate, for example) and visual effects to tell the story.

Many practitioners are now using a combination of practical and digital effects in an effort to keep the human visual system (developed over millions of years) concentrated on the emotions the image is trying to generate in the audience rather than on the technology used to achieve it (vfx geeks like myself find both pretty cool!).

The masters of the art know which expertise to bring to bear to create an image that -- until excruciatingly late in the process -- becomes the public's lasting vision of that shot or sequence and their representation of it in their mind's eye...

Yes, it can be done wrong. Yes, sometimes creative impulses collide instead of blend. Yes, sometimes even withing the same show some work can be great and other work can be mediocre.

But when it works, it is a remarkable almost alchemic transformation of human thought, human communication and technology into archival images of the way things are, were or might be.

Visual effects is about trade-offs, about getting the best image possible that will serve the story under the financial, schedule and political/team circumstances in which you find yourself.

Visual effects is fundamentally about the people you work with to solve problems, not the specific technology you employ to solve the problem. There are multiple solutions to any visual effects problem. The solution chosen has everything to do with the human and technical resources available at the time.

Visual effects practitioners most often work in teams using resources at hand. Though magical and effortless-appearing in the end, it's often a long, hard (but often very soul-rewarding) slog to get there.

The field has its “rock stars,” of course, but these personalities most often represent teams of other people who take on and relinquish many roles over the course of a visual effects project.

The way I see it, visual effects is akin to old time stage magic, trompe l'oeil painting, camouflage, scientific demonstration, theatre, architecture (especially the architecture of sacred places such as temples, cathedrals and so on, collage, sculpture...

Rather than looking at the evolution of visual effects as a linear progression of people technology and methods (the textbook approach), we will look at visual effects as a succession of solutions to various problems, many of which exist in motion picture production to this day.

Since the beginning of the era of recorded visual entertainment, practitioners have solved problems on many levels in demanding and chaotic project-based ad-hoc organizations.

On one level, visual effects deals with the problems of human perception – finding ways to take advantage of the strengths and weakness in the human visual system. Cues such as scale, perspective, depth, color, pattern, texture, intensity, shadow, parallax each of us instinctively use to place ourselves in the world. Visual effects take advantage of these cues

On another level, motion pictures are themselves illusions in that they depend on a visual circumstance called persiustence of vision which enables successive frames of still images to be perceived as continuous action.

On still another level are the technical problems related to the tools and materials used to develop a visual effects solution. But no matter what the eventual technical hurdles, visual effects starts with the written representation of the story -- the script -- and very quickly moves into iterative visual representations of the story.

The bottom line always is the story: depictions of real or imagined events with characters, objects and environments illustrating some aspect of the human experience.

Tis light makes color visible; at night
Red, green and russet vanish from thy sight.
So too the light by darkness is made known.
All hid things by their contraries are shown.
Mevlana C. Rumi (1207-1273)