Better compression on the Mac with x264

Jim Geduldick (@FILMBOT) has an new x264 Encoder Tutorial for as an alternative to the QuickTime codec for producing H.264 files on the Mac (embedded below).

This video tutorial is on Already Been Done, a skateboarding resource. As usual, there's are other tutorials, for example one by Satya Meka, Quick Tip – Better Compression With X264. And there were tips and utilities before that, some found in More on Quicktime’s H.264 gamma bug at AE Portal.


Jim Geduldick: H.264 files / x264 Encoder Tutorial from Already Been Done on Vimeo.

AE FAQS: memory usage, gamma shifts + more

Today Todd Kopriva added some useful posts on FAQs for After Effects and ProRes 4444 colors and gamma shift when working with After Effects and Final Cut Pro [Todd later added more on ProRes on the AE-List].

Many of us will find these reminders useful, like FAQ: Why doesn't After Effects see and use all of my RAM? In another example, here's a portion from RAM and disk caches that explains counterintuitive memory use of new systems with 8 GB and more of RAM:

"In the Memory & Multiprocessing preferences, After Effects reports the size of the RAM cache---the amount of RAM that can be filled with frames for RAM preview. With Render Multiple Frames Simultaneously turned off, this value is reported as Total After Effects Memory Usage. With Render Multiple Frames Simultaneously turned on, this value is reported as Foreground Memory Usage.

This value will not be the full amount of RAM allocated to the foreground application. Rather, it is just the part that can be filled with frames for RAM preview.

For example, on a Mac, this value might be 1.79GB on my Mac. 4GB is the theoretical limit for an After Effects process (because it's 32-bit). The OS takes a bite, getting you down to ~3GB. The RAM cache is only allowed to use 60% of that, which gets you down to 1.79GB.

The limitation of the RAM cache to 60% of the memory allocated to the foreground process is to avoid fragmentation problems and therefore out-of-memory errors.

The terminology can be confusing in part because in After Effects CS3 we had a Maximum Memory Usage item and a Maximum RAM Cache Size item, and in After Effects CS4 the thing called Foreground Memory Usage maps more closely to the latter, not the former."

Update: it's December and Todd is back with a reminder on RAM, Performance tip: Don't starve your software of RAM.

25 years of using gamma 1.8

A comment on John Nack's blog post Why your Web content will look darker on Snow Leopard explains the origin of the old Mac gamma. While the gamma change is hardly news to After Effects users, there's more in the post, like a quote by Adobe Principal Scientist Lars Borg, "the movie studios' Digital Cinema Initiative selected a gamma of 2.6 as providing the best perceptual quality for 12-bit cinema projection" now in home theater.

Ted Dillard (11:19 AM on September 4, 2009; trimmed here): '... I think I got it from as near the "horse's mouth" as you can- Bill Atkinson, who not only pioneered ColorSync, but built QuickDraw and helped develop the first laser printer for Apple.

The story he tells is that they were trying to get the Laserwriter to print close to what the screen was showing- in monochrome, mind you, and the only way they could get it to work was to jack the gamma to 1.8. In his words... "it was the silliest thing."

Color Management wasn't even a thought at the time, and it set the stage for decades of Color Management mismatch and confusion. I asked him about it because I read a story about gamma, and that, back in the mid-90s this team took a bunch of cards and tested them- Mac and PC, and they all tested out to around 2.1 - 2.2.'

Fun maybe just for me, I resold a copy of Bill Atkinson's photography book Within the Stone to Aardvaark Books in San Francisco last year; it was inscribed by a Bill Atkinson to the Dalai Lama.

More on Quicktime’s H.264 gamma bug

In The Quicktime Conundrum, Art Adams riffs off of Chris Meyer advice (using X264 codec popular on the PC) on how to get around Quicktime’s H.264 gamma bug. In the 2nd post on the topic, Art talks about reader comments:

"...Brandon Cory suggests that exporting a Quicktime reference movie from Final Cut Pro in the Animation codec and then running that through Compressor’s H.264 encoder should retain the proper gamma settings...Apparently using the right codec, one that doesn’t mess with gamma on export, is still a key to successful H.264 encoding.

david@kosmos pointed out ...the easiest way to make an H.264 Flash file from an H.264 Quicktime file is to change the extension from “.mov” to “.f4v”. That’s it. On the Mac I had to go into the H.264-encoded file’s “Get Info” properties and change the “Open with” attribute to Adobe Media Player..."

Just passing along anecdotal reports (untested here); the original poster didn't mention reference movie just an export. But hopefully no more fancy footwork or gamma stripper will needed to fix h.264 gamma.

Update: The QuickTime Gamma Bug, posted on December 31, 2010 at vitrolite, is a thoughtful and deltailed an any article I've seen

Brightness Issues with H.264 QuickTime

Chris and Trish Meyer have a good post on the gamma problem:

Brightness Issues with H.264 QuickTime Movies: Solutions - good and bad - to a long-standing problem.

They nicely summarize actual solutions, including use of a gamma stripper app from FuelFX mentioned earlier and using the x264 codec.

Update: Chris added a few details to follow up on a thread on the AE-List (Search "h.264 compression"),

The workaround Alan and Jarle mentioned works - on Windows. Which is why it didn't work for me. The procedure is slightly different inside QT on the Mac. Unfortunately, it doesn't work for playback in all apps, and has other feature and performance hits that are unpleasant (for example, it supposedly kills Fast Load).

Brian: Having a custom color profile indeed greatly aggravated the issue. Although I did still see a slight problem even with the default Cinema color profile. What is interesting is that when I switch to my custom profile (which is darker than the default Cinema profile), the untagged movies get darker as well, whereas the Apple gamma tagged movies change very little. This points further to some correction Apple is doing under the hood - which, however well intended, is going awry in many real-world situations.

Thanks again to everyone for their suggestions. The x264 codec ended up being the way to go.

QTGammaStripper for OS X PPC and Win32

Fuel International has free unsupported apps, both called QTGammaStripper, for OS X PPC and Win32 to strip QuickTime gamma tags. Look under the Software link.

The problem and other solutions were discussed in QuicktimeGammaStripper utility for Windows and elsewhere.

QuicktimeGammaStripper utility for Windows

Besides the usual gamma differences and Windows graphics acceleration, gamma tags in QuickTime files can change appearance of files across platforms. There's more on this in a previous post, Quicktime Gamma Stripper.

Frantic Films Software "created a small tool that strips the 'gama' tag out of offending quicktimes. It's a very simple tool which operates on all the file names and folders given to it."

This QuicktimeGammaStripper utility (for Windows) disappeared in the last few months, but it popped up here; I'm not sure if or when the link expires.

Update: Some websites directly copy postings (cf. here & Windows Live) and/or hide their sources, but at least some of them give something if only to keep viewers from leaving their site. In one case, The DV Show follows but at least posts the utility as QTGammaStripper.

QuickTime Pro gamma fix

Video Copilot posted another workaround that's been kicked around for well over a year to fix the QuickTime gamma wash out; you'd need QT Pro though:

After rendering into a QuickTime/h.264 file, open it up in QuickTime and select “Show Movie Properties.” Highlight the video track then click on the “Visual Settings” tab. Towards the bottom left you should see “Transparency” with a drop-down box next to it. Select “Blend” from the menu then move the “Transparency Level” slider to 100%. Choose “Straight Alpha” from the same drop-down and close the properties window and finally “Save.”

Update: More in comments at Video Copilot, "VisualHub users can circumvent the problem by selecting “Force: FFmpeg Decoding” when compressing H.264 material. StaxRip and MpegGUI are great Windows H.264 encoding solutions that also don’t exhibit the issue on files they produce." ...and... from Geoffe "Now what I have found certainly does work is export the Quicktime as a 100% Motion-JPEG encoded movie, and then from Quicktime Pro export that as h.264. This looks pretty much just like the original (gamma-wise)." The Animation codec would work too.

Quicktime Gamma Stripper

Besides the usual gamma differences and Windows graphics acceleration, gamma tags in QuickTime files can change appearance of files across platforms.

Frantic Films Software explains: "Quicktime has a feature, mostly hidden from users, which is designed to adjust the display gamma of quicktime movies on different machines to compensate for display difference. Deep within the file, there is sometimes a little tag called 'gama' lurking which tells the Quicktime player what gamma correction the file was encoded with. While this is well-intentioned, motivated by the difference in display gamma between PCs and Macs, the Quicktime player offers no way to view this tag and change it.

To work around this issue, we have created a small tool that strips the 'gama' tag out of offending quicktimes. It's a very simple tool which operates on all the file names and folders given to it."

There's more info as well as the QuicktimeGammaStripper utility at Frantic Films Software. It's for Windows but there appears to be instructions for Mac too.

Adobe has a technote for strictly After Effects-related issues: QuickTime movies exported from After Effects CS3 are darker or lighter than expected.

Update: this utility appears to be missing from Frantic; I'll look around for it.

Update 2: File found on internet; see QuicktimeGammaStripper utility for Windows.

Update 3: Some websites directly copy postings (cf. here & Windows Live) and/or hide their sources, but at least some of them give something if only to keep viewers from leaving their site. In one case, The DV Show follows but at least posts QTGammaStripper.

Update 4: Check out updates QTGammaStripper for OS X PPC and Win32 (another gamma stripper app) and Brightness Issues with H.264 QuickTime, a survey and x264 solution by Chris & Trish Meyer.

Definitive answers on QuickTime and gamma

As noted in 2 previous posts, there have been concerns about unexpected color shifts when looking at movies in QuickTime Player and other apps like AE and FCP.

Over on the AE-List, Todd Kopriva (keeper of AE LiveDocs) announced a new technote on QuickTime, gamma, and After Effects which provide "the definitive answers" from Adobe Technical Support. The technote is titled "QuickTime movies exported from After Effects CS3 are darker or lighter than expected" and is also available at http://www.adobe.com/go/learn_ae_quicktimegamma.

Gamma gamma QT

Mark Christiansen, whose new book Adobe After Effects CS 3 Professional Studio Techniques is nearing release, posted Fun with Gamma and Quicktime which briefly touches on file display issues in various apps. Hopefully Mark, as well as Trish and Chris Meyer, will have much more to say on the subject of color management in their new books and web postings.

An earlier post here, FreshDV notes Apple's 'color compatibility', attempted to summarize the same issue from the armchair.

Update: on the AE-List 3 Nov 2007, Chris Meyer alerted users to a not ready for primetime link to that should clarify things,
http://www.adobe.com/go/learn_ae_quicktimegamma
"... will be updated _significantly_ in the next week or two with specific instructions of what to do in various situations, and what to expect as a result."

Color bars FCP<->Color

Again from the armchair... not only are there now sometimes color display or conversion issues among AE, QT Player, FCP, and Shake (see FreshDV notes Apple's 'color compatibility'), but also between Apple Color and FCP. Could be another reason for a Universal Color Metadata format. For more see various threads on the ColorList, for example this one: anyone try sending color bars back and forth from FCP?

Update: BTW, Walter Biscardi has a new video tutorial on Apple Color: Building Vignettes in the ColorFX Room, with other at Creative Cow.

FreshDV notes Apple's 'color compatibility'

Apart from reported issues in parts of CS3 with Leopard or with understanding the new Color Management, some users might have have issues with gamma changes in QuickTime renders when viewed in various applications. These sorts of problems are not related to color management in AE per se, and can manifest as luminance differences displaying the same file in AE, Shake, FCP, or QT Player.

FreshDV notes a fix in the Mac QuickTime 7.2 update, where "Apple has added a new tickbox in the Quicktime Preferences labeled 'Enable Final Cut Studio color compatibility,' with the description 'When enabled, video is not displayed using ColorSync. Source colors are read with the 2.2 gamma and are displayed in a color space with 1.8 gamma.'"

Unfortunately, gamma behavior is not consistently handled for all codecs (e.g., DVCProHD) on all platforms (or even the "visual context" of OpenGL or Direct3D in QT). Gamma tags have not always attached or read, and problems seem to be Apple's rather than something caused by Adobe's MediaCore file handling and conversion components. One issue was fixed by AE 8.0.1, because with FCP Studio 2, Apple's Uncompressed 4:2:2 10-bit QuickTime codec reads and writes pixel data differently than prior versions of the codec.

In CS3 Adobe attempted to deal with gamma changes across platforms by adding a switch in Project Settings->Color Management called "Match Legacy After Effects QuickTime Gamma Adjustments." The Legacy QT checkbox defaults to On when you open a legacy project. Generally keeping color management off and the Legacy switch on will keep things as they were in AE 7, except on Mac OS running on a PowerPC processor, where QuickTime codecs are used for some formats (including DV, 2vuy, and v210), whether or not you select Match Legacy.

Using Match Legacy mostly works fine, except when cross-platform round-tripping. Some reports have claimed acceptable gamma matches in problem situations by enabling Color Management with the project working space set to sRGB, but that could just be just for previewing files in QT Player.

There seems to be uncertainty here (from the armchair) whether the new Quicktime preference will make gamma the same in FCP, QT Player, and other apps for all codecs, but things will probably be clarified or cleared up soon for the few problematic situations.

Update: Martin Baker added a comment to the FreshDV article: "This preference just changes the display of movies in QuickTime Player so they match the appearance in FCP’s Viewer or Canvas. In FCP6, Apple extended the import gamma preference (User Preferences > Editing tab) to affect movies as well as stills and you can also change the gamma of a clip individually in the Browser."

Color management in AE CS3

Mark Christiansen's After Effects 7.0 Studio Techniques has a great discussion of film, HDR, and 32-bit compositing, but AE8 adds a few wrinkles with a complete color management workflow through Project Working Space options, to preview color and simulate final output in different mediums like broadcast monitors or film stock.

As in Photoshop, I'd keep Color Management off to avoid inadvertent color shifts from embedding or converting to default profiles unless I've worked out a specific workflow. Files embedded with Adobe default profile sRGB might sneak in, so getting color management under control for all codecs and file types might take some serious thought. I had standardized on Adobe RGB (1998) and ruthlessly convert graphics from others if they use a profile. There's even a new utility filter to convert per layer, Color Profile Converter (pictured). I don't have a firm handle on the new color management yet (I'm fine as an NTSC island) but hopefully these Adobe white papers will help clear things up: Color Management Workflow in Adobe After Effects CS3 and Color workflows for Adobe Creative Suite 3 .

There's a basic video tutorial on color management in the Video Workshop. And you can also flip through the documentation for color and color managment at AE LiveDocs, for example High dynamic range color and Why you should use color management.

Update: this is a hairy issue...according to Studio Daily, "Post Logic Studios announced the launch of its Image Science Division...[to offer]...clients “the assurance of consistent color management across every stage of a production, regardless of acquisition format or output medium.”

Update 2: On the AE-List (18 Aug 2007) Chris Meyer responded to a question about a gamma shift between AE8 and AE7, "The short answer, in most "AE CS3 does not match AE7" cases where Color Management is _not_ enabled for the project (Project Working Space = None), is to enable the Legacy Gamma switch in the Project Prefs. The long answer is...a long answer. (I'm writing that chapter this week, as a matter of fact.) The text in the Project Prefs, Interpret Footage, and Output Module dialogs help, as does the Adobe white paper..."