It seems like they might have used something like Frischluft Lensfeed, which shows live camera feed inside of AE, and something like PictureReady, which offers the ability to record a QuickTime movie which is available immediately, during capture. It would be fun to find out the actual specifics.
There wasn't a mention in Studio Daily's The Avatar Plug-in List, and quick looks at a few other resources so far only mentioned Shake and Nuke: the Jan 09, 2010 episode of fxguide's series on Avatar, segments of show Science of the Movies, CG Society's Perfection in Blue (via Topher Welsh), David Stripinis on modo & Avatar with Brad Peebler, and the current Cinefex (#120).
Update: there's a few more tidbits on AE and "simulCAM" in an article linked by Michael Coleman, and and more from Adobe's Serge Jespers,
"AFTER EFFECTS had a multitude of uses as you can probably imagine:
- In the early stages of production, AE was used to place flowing camera moves on still artwork for concept art reels.
- On the live and mocap sets, AE was used to quickly create test composites to see if the live-action and “simulcam” green screen shots were lining up with the virtual animation and backgrounds. Within a few minutes of each take being shot, artists were able to mock-up composites for James Cameron to look at and approve or re-shoot.
- Automated rough comping of the video of the facial performances of the actors (recorded head via mounted miniature video cameras) into CG character face textures for the previs portion of the production. This allowed Jim a quick and easy way to know if the actor’s facial performances were going to be OK or if a take needed re-shooting.
- AE is being used by some of the vendors hired for VFX work to create 3D stereo composites for finished shots, to create complex motion graphics for use in the 3D holographic screens in the various control room scenes, and to create “heads-up” displays for various high-tech vehicles in the film.
- Adobe provides documentation to write complex procedural scripts for AE, and the AVATAR artists took full advantage of this. They created an automated rendering pipeline for comping the previs shots using AE’s scripting capability.
PREMIERE PRO was also used in different ways:
- Each time Jim Cameron does a review/critique of VFX work, an HD recording is made of the session. Premiere Pro is used to take the edited recording and format into various packages for digital distribution. The process is automated, with typically 8 hours or more a day of reviews that need to be processed.
- Premiere Pro was used extensively on set in combination with AE to check rough composites in playback context with animated sequences. A/B comparisons of VFX vendor work are easily set up and reviewed.
- Although the creative editing for AVATAR is done using an AVID based system, Premiere Pro is able to read in cut-lists and other important metadata using its built-in AVID “AAF” import feature. Rather than bog down AVID editors with having to render out updated sequences for the animation department, the AVID editor can export the virtual “recipe” for the new cut to a small file that Premiere Pro then uses to automatically assemble the cut to match the AVID. Digital video files of the shots that are online in the master shot database are sourced as clips by Premiere Pro."
Update 3: not for AE info, but fxguide's January 15 podcast for which interviews John Knoll, who goes into specifics, including a discussion of stereo 3D techniques. Also this,