The Heavens Have Foretold Your Doom

At one time or another, many computer animation people have worked to create an illusion of the night sky from earth or of its cousin, a “star field”, which is an imaginary view of the stars from space. Whether this was for their own amusement, for visual effects purposes, or for scientific visualization, these innocents would approach the problem with the assumption that it was going to be easy. How hard could it be, its just a bunch of random white dots, after all. Imagine their surprise when they discovered that doing excellent starfields is far from trivial.

A classic traditional technique to create starfields is to create a cyc, or curved screen, painted black and with very small holes punched in it. Then behind this screen was a curved light source, usually florescent tubes. The camera would be at the center of the implied sphere of the screen and when the room was darkened and the backlight illuminated, you had a curved space of very bright, very small light sources which could be photographed with long exposures when the camera was moving. The result was excellent motion blurred, perfectly antialiased, very high contrast star fields. But ultimately there were certain moves that the motion control camera could not easily do, such as tumble end over end for example, so there was a need to synthetically generate these elements.

Another time honored technique which looked excellent was the painting on glass. Most of the times you saw stars in Close Encounters of the Third Kind (1977) you were seeing an optical composite of a live action element or motion control shot with a matte painting on glass.

Since everyone seems to have to go through the same learning curve, I am providing notes here for what some of the issues facing 3D technical directors as they produce their first starfield and I have written it as a letter to my younger self.

September 19, 1983

Oh, unwary traveler, so proud of your 3D knowledge, your knowledge of geometric modeling, or of animation whether scripted or procedural, and of global illumination; do you think to encompass the heavens with these pathetic tools? Fool, your doom is assured. There are more things in heaven and on earth than are encompassed in your philosophy, or so I have heard, and when you approach the field of scientific visualization you must unlearn what you have learned and embrace the esoteric wisdom. You must open your eyes in order to see the light.

What perils await the unwary, the arrogant, the unlettered?

The first peril is the vast expanse of space. There is the scale of mortal man, then the scale of the solar system, then the scale of one single galaxy, and then beyond. These differences in scales are way beyond what most software packages can handle, so using the 3D positions of everything in a naive fashion is unlikely to work.

And that renderer you are so proud of.  Does it do all its calculations of space in 64 bit floating point or even higher precision?   Most renderers, with a few notable exceptions,  do the majority of their work using single precision floating point which may be adequate for a giant robot or two, but falls apart in the vast distances of space.  

The second peril involves the issue of filtering of what is very untypical samples.  Most scenes render surfaces with various lighting applied.   But a great deal of what you wish to render are stars but what are stars? Stars are huge things, but they are (for all practical purposes) infinitely bright and infinitely small (on the screen). The amount of energy concentrated in a single pixel may be immense, but the pixel next to it may have very little or no energy at all. And what happens under those circumstances when you move the camera? Well, it aliases, of course, terribly. Furthermore, if one has modeled stars very far away and you are using point sampling of one form or another to simulate area sampling, then if you are not careful, some of your samples will miss and you will have aliasing again.

Part of the solution is to use a good filter and lots of samples and in the choice of filter lurks another threat since as we know a "good" filter, perhaps a 7x7 sinc for example, is likely to have negative lobes, and instead of throwing those values out, you should keep them until the end and even then you should not throw them away. What then to do with them is a mystery left as an exercise for the reader.  The best solution of course would be to have a display that could absorb light as well as emit it, but we wait in vain for the display manufacturers to come to our aid.

And what about those overly bright stars? Will you generate glows and other artifacts? After all we are not just trying to simulate realistic stars, we are often trying to simulate realistic stars as the audience has seen them, and expects to see them.

Although the sky is filled with stars, that is not the only thing that there is. There are also great fuzzy areas known as nebulae and sometimes other galaxies. It turns out that if there is any data for that, it is likely to be volume data. But even if there is no data and you create your own, volume rendering is the best way to render a nebula one might argue. Does your renderer of choice do volume rendering?

Review the following image of the earthling's galaxy.

Do you notice the great areas of darkness? That of course is the infamous "space dust", the so-called Interstellar Media or ISM which must surely exist to hide from us the center of our galaxy where no doubt an entity of great evil exists. Surely you do not think it a coincidence that the space dust would hide what is arguably the most spectacular sight in our little neighborhood? Since most star catalogs do not have the ISM modeled, you may wish to develop a model of ISM in your spare time. If not, the galaxy will not look right unless you simply leave out the stars that are in those areas (which may or may not be be in the catalog anyway as they are impossible to view from earth, at least in the visible bands).

Because you are rendering stars, no doubt you have studied scotopic vision.  It goes without saying that whenever the biped mammals have watched the stars they have, generally, been night adapted. And yet they see color sometimes, perhaps they see Angry Red Planet Mars or Betelgeuse and they perceive the color red.   How then are they seeing color? It may help the seeker of knowledge to realize that “scotopic” is named for the Skoptsy sect of religious devotees whose most notable doctrine is of male castration.  (see link below)

Of course I am sure when you move the camera you will motion blur everything. Oh yes, what do you plan to do with the speed of light issue? I am sure you will come up with something.

So, foolish mortal, you have been warned.

These are just the first of the issues you must address for a proper starfield.

Fools may go where wise people fear to tread.

Sincerely,
A Friend.

___________________________________________________________

Scotopic Vision

http://en.wikipedia.org/wiki/Scotopic_vision

The Skoptsy

http://en.wikipedia.org/wiki/Skoptsy

Close Encounters of the Third Kind (1977) on IMDB

http://www.imdb.com/title/tt0075860/

Two VFX Examples From The Bourne Identity (2002)

This post will showcase two very elegant visual effects shots/sequences in the movie The Bourne Identity (2002).

Sadly none of the case studies today feature giant robots or things that explode loudly.   This is because of the filmmakers' failure to understand that giant robots and exploding things seasoned with an occasional space battle or cute furry aliens are the most important elements when crafting a significant visual effects project.  Instead what we have here is an odd little film with a few likable characters and a lot of violence, but very few explosions.  Even so, we can use this project as a modest example of how visual effects can improve a film or lower the cost of shooting something without an explosion per se.

There is a well-known aphorism from film editing that goes something like "Good film editing should not be noticed by the audience."  The editing should be subliminal, you should not (consciously) notice it.  In a similar way, if the audience thinks "wow, what a great visual effect!" that is not the desired result. They should think it is real.  Of course there has to be some level of suspension of disbelief for that to work in many cases when, for example, one is showing a giant robot eating an alien world or some other subject not drawn from day-to-day experience.

But in the case of a film that takes place in a contemporary setting and in which there are not overtly fantastical elements, then hopefully the viewer will just be involved in watching the film and not think that he or she has just witnessed a visual effect. There are some surprisingly effective and useful visual effects that are completely unnoticable unless someone points them out to you.

To illustrate this, I am going to showcase, as best I can, two shots/sequences from the first Bourne movie: The Bourne Identity (2002). I think that both of these sequences work very well and both of them are implemented in a remarkably simple manner.   In both cases, digital technology made the shots easier to execute.  In the second case, the shot could not have been done without a digital technique.

Sequence 1: The train at night

In this sequence, Jason Bourne, who is suffering from amnesia and does not know who he is, has very little money, is travelling on a train from France to Switzerland.

The shots break down as follows (times are approximate):

1. A five second shot of a modern train going into a tunnel,

2. A twelve second shot with a slowly moving camera of Bourne looking out a window of a train, either looking at the tunnel moving outside the window or at his own reflection,

3. A three second shot of Bourne's hand fiddling with a plot device,

4. A thirteen second shot of Bourne at his destination outside the train looking lost.

One of the unusual things about this sequence is that it moves very slowly.  In general, we do not like people to have the time to study the effect, as they can usually see through it if we give them more than a few seconds.  But in this case, we stare right at the effect for 12 seconds and it works fine.

In the second shot, the one with the moving camera, the original element was shot in a train that was not moving, with the window blacked out (the reflection of his face and seat are there, but it is black otherwise with no sensation of movement). There is a light on the set illuminating Bourne's face intermittantly to simulate the idea that the train is passing something that is giving off light, such as a signal, but there was nothing beyond the window but black in the principle photography.  The camera was tracked in 3D using some early tracking software and a 3D element of some abstract, dark, tunnel-like textures rendered moving past the camera at high speed (e.g. with a lot of motion blur) and rendered with the tracking camera move. This was then composited against the original shot using a simple hold out matte generated of the outline of the window. The element was basically just overlayed on top of the shot in the area of the window, you did not mind that the textures were visible "under" the reflection of Bourne.

The end result of this is that you completely buy that Bourne is on a train moving at night. One 3D track, one simple 3D element, one travelling hold out matte, and a simple additive composite within the hold out area. I think it works perfectly and it was very inexpensive to execute.  Without it, I don't think the sequence would have been as believable (in other words, had Bourne been looking at his reflection against a black background without any sensation of motion). Had it been shot in reality, e.g. a train moving at night, it would have been much more expensive.

Alternatively, one could have used rear projection to do a similar shot, but you would not have been able to move the camera that far off axis in a rear projection situation.   One could have done a similar shot with a moving camera and traditional techniques, I think, but it would have been more difficult.  Using traditional techniques, I would have shot the principle photography using a motion control or motion tracked camera and then reused that move to control a motion control camera to shoot additional elements, in particular to  rephotograph rear projection art work which had previously been created with a suitable blur of movement (for the movement of the train past the window, not the movement of the camera).   Either I would have shot blue/green screen outside the window of the train in the principle photography, or if I was using motion control to shoot the plate, repeated the movement with a green screen in order to get a hold out of the window.  Then I would have optically composited and it all would have worked.  Here the digital techniques really do make this shot straightforward, however, and less costly to execute.

Here is the sequence online.
http://youtu.be/qyAcJEU4xco

Sequence 2: The incident in the park

After Bourne arrives in Zurich, we have one establishing shot of him alone, at night, in Zurich with snow falling which is about 7 seconds long. We cut to Bourne sleeping in the snow on a bench. Two police officers wake him up, ask to see his ID, and tell him he can not sleep there. One of them gestures with his nightstick, and Jason grabs it. The two police officers are standing above him, he is sitting on the bench, unarmed.

The next six or seven shots (depending on how you count) are each very short and appear almost continuous, even though they are not.   In these shots,  Jason disarms both men and knocks them both unconscious as well as taking one of the police officers revolver.  When he is done, Bourne stands puzzled over the two unconscious men and seems to wonder what happened.

It looks completely natural and Jason does not even appear to be working very hard.

It reminded me, as it was intended to, of when I have watched a dancer or gymnast perform: it looks as if what they are doing is easy even though you realize that what they are doing is impossible.

What they did is as follows. First, a martial artist working for the production choreographed the actions of Bourne and the two officers moving very slowly. As shot, the actors moved at a comfortable speed and did not try to maintain a constant rate.  The camera changes position during the shot so presumably it was shot several times from different positions.   You will also notice that Jason appears to move in what seems like clean, deliberate motions with brief pauses between them. The speed of the performers was not constant, to get the effect of the police officer on the right being knocked to the ground, for example, the (presumably) stand in had to basically throw himself onto the snow so it would react properly.  

Then the effects supervisor, Peter Donen,  took the shot(s) and digitally retimed them, varying the apparent speed continuously through the shot(s). There is also some very good film editing going on.  The sequence that results looks flawless to me, and as I have mentioned, almost appears continuous, even though in actuality there are several cuts.  In this case, digital retiming which makes use of a variety of image processing technologies involving motion analysis (image flow) between frames enabled this approach.  Previous to this digital technique, the traditional techniques could do retiming but only in specific increments of the frame rate, e.g. one could skip frames and double the speed of the shot, but that would not have been sufficiently flexible and continuous (e.g. moving at fractional speeds).  Keep in mind also that this retiming technique could only work in this situation as long as one keeps to very short cuts because we have snow falling.  Assuming that this is real snow (and it very well might be, or practical snow on the set) then it will appear to change speed if we do retiming on longer sequences and just allow that to be viewed.  So this technique has to be used in very short segments or elements like the snow have to be added later.

Here is the sequence online.
http://youtu.be/kl1uGvPAJEQ

So here we have two examples of visual effects used to serve the story that were both elegant and inexpensive to do. The second sequence is an example of making something that is inherently fantastical look natural and realistic.  We can forgive the lack of a cute furry alien or a giant robot since the filmmakers have executed their inferior robot-less vision with such skill. 

The effects supervisor and my friend, Peter Donen, passed away about four years after these sequences were done, tragically of a heart attack in his mid-50s. What was especially sad was that his career was just taking off after decades of struggle. He had the misfortune of being the son of a very famous man, the director Stanley Donen.