Scaling in Premiere Pro CS5

Todd Kopriva and Steve Hoeg share some details about scaling with CPU and CUDA in Premiere Pro CS5, which AE users can only look on with envy unless there's a Premiere render path through Dynamic Link or import. At least there's another option than Photoshop. Note also that deinterlacing and blending modes are GPU-accelerated, and there are GPU-accelerated effects [update: see a list in link comments]. Here's an excerpt on scaling:

"When Premiere Pro CS5 is just using the CPU for the processing of scaling operations, it uses the following scaling methods:

• playback: bilinear
• paused: Gaussian low-pass sampled with bilinear
• high-quality export (Maximum Render Quality off): Gaussian low-pass sampled with bilinear
• Maximum Render Quality export: variable-radius bicubic (never used on unrendered footage)

The variable-radius bicubic scaling done on the CPU is very similar to the standard bicubic mode in Photoshop, though the Premiere Pro CS5 version is multi-threaded and optimized with some SSE instructions. Even with these optimizations, it is still extremely slow. For high-quality scaling at faster-than-real-time processing, you need to use a CUDA card.

When Premiere Pro CS5 is using CUDA on the GPU to accelerate the processing of scaling operations, it uses the following scaling methods:

• playback: bilinear
• paused: Lanczos 2 low-pass sampled with bicubic
• export: Lanczos 2 low-pass sampled with bicubic

For export, scaling with CUDA is always at maximum quality, regardless of quality settings. (This only applies to scaling done on the GPU.) Maximum Render Quality can still make a difference with CUDA-accelerated exports for any parts of the render that are processed on the CPU. Over time, we are working on reducing the list of exceptions to what can be processed on the GPU. For an example of a limitation that can cause some rendering to fall back to the CPU, see this article: “Maxium dimensions in Premiere Pro CS5″.

When rendering is done on the CPU with Maximum Render Quality enabled, processing is done in a linear color space (i.e., gamma = 1.0) at 32 bits per channel (bpc), which results in more realistic results, finer gradations in color, and better results for midtones. CUDA-accelerated processing is always performed in a 32-bpc linear color space. To have results match between CPU rendering and GPU rendering, enable Maximum Render Quality."

Read the rest at Premiere Pro work area.

Zoom in and enhance

While most of us have seen Let's Enhance, a rerun of The Simpsons provides another algorithm for now:

Wider aspect: directional content adaptive scaling

Future video-oriented versions of Content Aware Scaling might eliminate the need for "directional content aware scaling," if the problems encountered in scaling video can be solved.

It seems that the Food Network, HGTV, and other HD channels sometimes use a lens adapter or other technique to stretch SD content to HD. You can especially see the scaling in pans, where the center holds but the sides are stretched.

A post on the FCP-List mentioned that Teranex conversion and conditioning hardware executes a "center spread." Teranex (at NAB 2009, Booth SU5025) calls it "directional content adaptive scaling" and explains:

"Working with Turner Networks, Teranex developed a method of aspect ratio conversion using a non-linear anamorphic function. This technique, named FlexView, performs its aspect ratio conversion by leaving the picture information in the center of the image relatively undisturbed and then applies progressively more ‘stretch’ to the image as it gets closer to the edges. This process, while largely done in the horizontal domain also has a vertical component to help maintain correct geometry.

The basic premise is that most of the important content in a scene (the material that your eye is drawn to) is in the center of an image and the information on the edges is generally less important. Flexview leaves the center portion undisturbed and stretches the image most aggressively around the edges where there is less important material."

For more on this see the Flexview Application Note from Teranex.

You can do this right now with a free Final Cut filter (via The Editblog) from Andy Mees called Andy’s Elastic Aspect. 

And it seems like you might do this without too much trouble in After Effects or Premiere Pro with an appropriate gradient and a Displacement filter. Premiere with CUDA has better scaling quality, but the drawback is that the lower quality algorithms in popular software are unlikely to measure up to patented multipurpose hardware like Blackmagic Teranex.

For other options, see the nice thread on Creative Cow.

Note: It might even be possible to use the Protect options in Content Aware Scaling in later versions of Photoshop.

Protect options in Content Aware Scaling

Here's Russell Brown explaining how to protect details with Content-Aware Scale tool options. There's a QT version available too.




In addition to the original presentation of the research on content aware scaling (paper), there's another presentation from SIGGRAPH 2008 that's slanted toward video. Let's hope that some of this will appear in future implementations within After Effects, and maybe even some improved upscaling. Here's Improved Seam Carving for Video Retargeting:

Seam Carving hits the New York Times

Seam Carving, noted in previous posts, hit the New York Times yesterday, as noted by Create Digital Motion with added value in "Savvy Stretching: Free Pixel-Resizing Tools, But What About Real-Time, Video?"

It would be nice if After Effects got better resizing and scaling,

Seam Carving feature requests coming

John Nack mentioned this SIGGRAPH video a few days ago and John Dowdell later commented that the lead author of this technology has joined Adobe!

Seam Carving for Content-Aware Image Resizing

Update: Video Thing has some comments and links, including one to "An Informal Catalogue of Slit-Scan Video Artworks."

Update 2: In Imaging heavy hitters join Adobe John Nack details a number of interesting additions to the Adobe staff.

Update 3: Picutel is releasing a PS plug-in for this sort of image resizing (September).

Update 4: another tool Liquid Resize was noted in comments, and Studio Daily added info in Seam Carving for Content-Aware Image Resizing.

Update 5: CNET added a newer video from the developers.
http://reviews.cnet.com/4665-10621_7-6804136.html

Update 6: Gantico tested Resizer, a Flash app in this category; see Test: my first liquidly resized picture.

Transforming Scale

Image transforms like scaling and rotation can degrade images, and involve trade-offs between artifacts like aliasing, blockies, blurring, edge halos, ringing, and moire. Background on image transform can be found at Cambridge in Colour's Understanding Digital Image Interpolation, Optimizing Digital Photo Enlargement, and A Closer Look at Resizing an Image for the Web & Email.

FxGuide posted a comparison of transforms in hi-end apps (including AE) and concluded there is no single magic bullet to automagically improve quality for every image, or a single approach for every app. It does help to use high quality scaling and deinterlace algorithms, pre- and post-process with blur (when scaling down) and sharpen (when enlarging) filters, and deartifact or denoise with a specialized filter (or with AE Remove Grain or by brute force with a Median filter). You're on your own when animating slowly from 98% to 102%!

Some applications take scaling quality seriously, though many desktop editing and compression software does not. AE uses simple bilinear interpolation, or a weighted box filter, even at Best Quality. Applications that use bicubic interpolation and variants include Photoshop, Commotion, Shake, or Combustion. The semi-official workaround, AE's Magnify filter, is not a solution, although some filters that scale may use high quality algorithms. Note that resampling in AE when there's no geometric transform is due to subpixel positioning. You can avoid this by positioning layers in 1 pixel increments.

You can access different scaling algorithms inside AE by using Resizer from Digital Anarchy. ReSizer also has a deinterlacer that uses motion estimation. Check out DA's before-and-after mug shots, though they don't compare results with the better algorithms in Photoshop.

Other AE filters that offer transform and enhancement features include Algolith, Topaz Enhance, and others by RE:Vision Effects or Red Giant Software. Debabelizer ("Sharp Sine" scaling on non-interlaced footage), IrfanView, and Satori Paint are also said to be good. Fractal-based PS plug-ins like Genuine Fractals (better with flat colors or clean images), PhotoZoom Pro or Pxl SmartScale sometimes work well. Digital photographers have investigated different methods that may be better than the fractal filters -- see the solutions from Fred Miranda (FM Stair Interpolation, Resize Pro) and tests from Roger Cavanagh.

Open source algorithm for denoising, interpolation and resizing

Hey, where's the free AE version?

GREYCstoration is an open source algorithm for image denoising, interpolation and resizing, using state-of-the-art image processing techniques. The original GREYCstoration algorithm is distributed as a command line program. There are some variants done by other people, including a Windows stand-alone GUI and a GiMP plug-in:

http://www.greyc.ensicaen.fr/~dtschump/greycstoration/index.html
.

Fit to comp AE script

Question on list:
Is there a way to do Cmd-opt-F (fit to comp) in a script? People like to work at 720x540 square pixels, but need to final output D1 aspect ratio 720x486.

Answer:
Lloyd Alvarez's brainchild script was a real collaborative effort, and is now detailed at AEnhancers.com. There's also a new version of the script that puts in a user-definable effect (like Broadcast Colors or Broadcast Spec, but it could be any Effect).

.