Streaming Media has posted all presentations from their recent conferences, with video and PDF handouts. Here's Jan Ozer giving an overview of H.264 (PDF). Missing background can be found in articles by Ozer and Andy Beach.
Hulu type quality isn't easy to achieve; for nice Flash compression examples, look at the work of Fabio Sonnati.
Update: EDN's Brian Dipert had insights into Hulu tech earlier this year; see:
NAB 2008: Is Full-Screen Hulu Jerky For You, Too?
H.264 Blues: Why Isn't Starbuck Smooth?
H.264 and Hulu: Day Two, Additional Data
Update 2: Geeky Mac people can cobble together similar Open Source tools as on Windows, including x264, with the help of Diego's Rants.