Codec shoot-out 2003 - 1st installment

Table of content:

1: Introduction
2: Test setup
3: Test 1: The Matrix
4: Test 2: Saving Private Ryan
5: Test 3: Futurama
6: Conclusion
7: Future outlook

Phew. For the past 3 weeks I've literally spent every free minute in preparation for a new codec comparison. The new comparison was supposed to be the biggest one so far, increasing the number of sources and codecs to a previously unseen level. I also meant to include a H.264 (AKA MPEG-4 AVC) codec into the test, but having spoken to the developers of the prospective candidate (Vanguard Software H.264 beta2) I decided to postpone the test because this codec currently does not have any postprocessing, and does not correspond to the latest revision of the standard (the standard has been finalized a few weeks back). Beta3, which should fix those "issues" was supposed to be released in time for the comparison, but is still not out at the time I'm writing this. This left me with 7 codecs to be tested on the two well known sources and a new one: Futurama (an animated series from the creators of the Simpsons).

Now a quick overview of what has happened since the last comparison: 3ivX finally has delivered a usable codec for multiple platforms. The codec also comes with MP4 playback filters, which finally makes MP4 files usable without having to use a specialized player (look forward to a MP4 creation guide update;). DivX5 has gotten multipass encoding (where multi means more than one, and can mean more than two), contains various bugfixes and supports profile for compatibility with DivX hardware devices. RealNetworks partially open sourced their product and created the Helix community for creating products capable of creating, editing and playing Real media files. The Helix community released a number of updates of the producer application, fixing a couple of bugs and increasing encoding speed. The German company Dicas has released their own mpegable brand of MPEG-4 encoding tools / codecs, which also support MP4 files. Microsoft has released Windows Media V9 to the general public. Recently, they've manage to gain some hardware support but mostly for their audio format. Artisan has also released one dual DVD set featuring a HD-DVD based on WMV9 and Terminator 2 will also be released on HD-DVD. Needless to say that those DVDs can only be played on a computer capable of playing WMV9 and that the term HD is misleading because not the full HDTV resolution was used to encode those movies. A few weeks back, Microsoft has also released the video part of WindowsMedia 9 with a Video for Windows (VfW) interface, allowing it to the used in regular encoding applications like Premiere or VirtualDub. Last but not least, XviD is getting close to a 1.0 release and many of the advanced features like B-frames, GMC, QPel, etc. are now pretty stable and have been available in the major codec distributions for a while.

The missing contestants: The following codecs were supposed to be included but didn't make it for various reasons: Dicas mpegable X4, the commercial encoding application by Dicas, comes as a 30-day trial. However, that trial ran out after only starting the program twice and I couldn't find a way to circumvent the limits (OK, reinstalling would've done it but that's too much to ask for). I got an unlocked version by Dicas, which puts a logo in every frame and which I did not deem acceptable for a production environment. I requested a key to unlock my locked 30 day trial but I never got one. I also asked On2 a 3rd time to evaluate VP5 and I did not get a reply for the 3rd time. On2 has publicly announced VP6 to be released in mid-May, we'll see if there's a change in their stance towards public scrutiny by then. I also did not include the Sigma Designs MPEG-4 codec. Forst of all, the license violation issue has never been properly settled, and it's nothing more than a rebranded XviD codec.

Last but not least: In order to present each codec the best way possible, I contacted each codec maker 1-2 weeks prior to the start of encoding to give them a chance to suggest the best settings for the movies used in this test. So, if you think my testing methods and settings are flawed, you're welcome to take it up with the developers because I stuck to their recommendations and if they gave sub-optimal settings it's their loss. At this point I'd like to thank all people who've worked with me in this area. The only codecs where I did not get any settings for were 3ivX and WMV9, despite having contacted the two companies.

Now let me present our contestants:

3ivX D4 4.0.3
Dicas mpegable AVI 2.0.3
DivX3 in the form of SBC (Smart Bitrate Control) as done in Nandub (hereafter referred to as SBC)
DivX5.05
RealNetworks RealVideo9 based on HelixProducer 9.1 M6 Gold (hereafter referred to as RV9)
Windows Media Video V9 (VCM beta1, hereafter referred to as WMV9)
XviD (Isibaar's development build dated 4/24, Isibaar branch in the CVS)

All codecs were tested in a 2 pass setup where this was applicable (3ivX and mpegable AVI only offer 1 pass encoding).

Test setup:

All testing was done on the following hardware:

AMD Athlon XP 2800+
Shuttle SN41 mainboard (part of the Shuttle SN41G2 barebone system)
2x256MB PC2700 Apacer CL2.5 DDR RAM
Creative 3d Blaster 4 Titanium4600 (NVidia Geforce4 TI4600)
Samsung SyncMaster 191T-P Display connected via DVI

To encode I used the following software:

DVD2AVI 1.76
mpeg2dec3.dll
Avisynth 2.51 April 16. for frameserving
VirtualDubMod 1.4.13.2b2 for DivX5/XviD encoding, VirtualDub 1.5.1 for 3ivX, mpegable AVI and WMV9
Nandub 1.0 RC2 lumafix ecffix for SBC
Helix Producer 9.1 M6 gold (build number 9.1.0.92)

To simplify things I used GordianKnot for the DivX3/5/XviD encoding and AutoRV9 for the RealVideo encoding.

Movies I encoded:

Matrix - Region1, NTSC, length: 2h16
Saving Private Ryan - Region1, NTSC, length: 2h49. I will call this movie SPR from here on
Futurama Season 2 Episode 1 - Region2, PAL, length: 0h21

Encoding parameters:

I used a 128kbit/s ABR audio track created in lame 3.93 (using --alt-profile 128 as parameter) for all movies, then tried to put Matrix onto 1 700MB CD, SPR onto 2 700MB CDs and 4 Futurama Episodes onto one 700 MB CD (resulting in 175MB for an episode). The size of the audio track was 124221KB, 157455KB and 20582KB respectively. This resulted in an effective video bitrate of 581kbit/s for Matrix, 1016kbit/s for SPR and 987kbit/s for Futurama (here kbit = 1000bit, DivX3 uses kbit = 1024bits so you have to divide that bitrate by 1.024 to get the effective DivX3 bitrate, all other codecs use k = 1000). Please note that multiplexing a VBR MP3 with an AVI gives more overhead than a regular CBR MP3 file. I used GKnot to calculate the bitrate as it's known to be accurate in such situations.

As you may know, not every rate control mechanism is perfect so here are the final movie sizes I got

Note that 700MB equals 716'800KB, 1400MB equals 1'433'600KB and 175 MB equals 179'200 KB. SBC is still best to match a given target size, closely followed by XviD and DivX5. RV9 still gets oversized (the longer the movie, the more), so did 3ivX and WMV9. As you can see the results for mpegable AVI are catastrophic. This has a reason: the WfV configuration interface is buggy (the people at Dicas admit that) and it happens quite often that your set bitrate gets reset to the default bitrate of 800 kbit/s. This explains why Matrix got oversized and SPR got undersized. As the difference in bitrate is considerable, it would not have been fair to compare mpegable AVI to the other codecs, and it was therefore excluded from the comparison at this point. Bottom line: If you need size predictability (who doesn't?), the codec choice narrows considerably already at this point.

Codec settings:

In 3ivX I used average bitrate mode, with minimum quality QP12. For SBC I used the GKnot defaults (no anti-shit). For DivX5 I used the following settings: Home theater profile (with B-frames), psy settings off, performance/quality slowest, bitrate modulation: constant quality and no mv file. In mpegable AVI I used skip probability: smooth motion and set the keyframe interval to 10 seconds. In WMV9 I set compression control to Two-pass Bit rate VBR, keyframe interval to 8 seconds, decoder complexity to Main and performance to level 4 of 5. In XviD I used the following settings for Matrix and SPR: motion search precision 6, quant type: MPEG, VHQ mode: 1, Quarterpel and chroma motion checked, Maximum B-frames: 2, B-frame quantizer ratio: 180, B-frame quantizer offset: 0, Below i-frame distance: 0, Bitrate payback delay: 500 and proportional payback. For SPR I additionally limited the quantizer range to 3-13. For Futurama I used different settings: Motion search precision: 6, quant type: MPEG, VHQ mode: 1, Mimimum I-frame interval: 0, Cartoon mode checked, Below i-frame distance: 0 and proportional payback. Note that you have to press "load defaults", then apply the described settings to get all the correct settings. For RV9 I used max bitrate = 2.1x average bitrate, startup latency 25 seconds, keyframe interval 10 seconds and audio format: 132kbit/s Atrac3

Encoding speed:

Here I take Matrix as an example. The resolution of SPR was bigger, hence the performance also dropped. Note that you should take these values with a grain of salt. Only the first 10'000 frames were used for speed measures so the overall average FPS might be a bit different. Furthermore, I did not reset the PC before each measurement and I did not perform several measurements per codec. For good absolute values, more testing would be required, but these values should nevertheless give you a good impression of the relative speed difference between codecs. Note that the values are averaged over the two passes where applicable.

3ivX: 51.54 fps
DivX5: 40.65 fps
RV9: 33.27 fps
SBC: 46.62 fps
XviD: 21.66 fps
WMV9: 17.43 fps

As you can see, speed has augmented considerably for both DivX5 and SBC as compared to the last comparison. This is due to a much faster CPU and the fact that all processing is now done in YV12 mode (AviSynth 2.51). A word about XviD encoding speed: Futurama was encoded faster than the other two movies, mainly because I didn't use B-frames, QPel and Chroma motion (all options that have a tendency to slow things down considerably). It is to be expected that once all the current features are stable, programmers will work on optimizing the codec for speed. If you recall from the last comparison (where these features were not present), XviD was the fastest codec of the bunch.

Other important stuff:

Here's the Avisynth script I used to encode the movies, so that you can perfectly reproduce my results. I used force film in DVD2AVI in Matrix and SPR. Futurama was interlaced, I used Decomb's FieldDeinterlace with no blending to get a progressive image both movies. Note that I used neutral bicubic resizing for every clip. I don't like the blurryness of the bilinear resize filter and the neutral setting is a good compromise between crispness and better compressibility but it doesn't blur.

Matrix script:

LoadPlugin("C:\PROGRA~1\GORDIA~1\mpeg2dec3.dll")
mpeg2source("D:\DVD\Projects\THE_MATRIX_16X9LB_N_AMERICA_VTS_02_PGC1\THE_MATRIX_16X9LB_N_AMERICA_ForceFilm.d2v")
crop(0,60,718,356)
BicubicResize(640,272,0,0.5)

SPR script:

LoadPlugin("C:\PROGRA~1\GORDIA~1\mpeg2dec3.dll")
mpeg2source("D:\DVD\Projects\SAVING_PRIVATE_RYAN_VTS_01_PGC1\SAVING_PRIVATE_RYAN_ForceFilm.d2v")
crop(2,8,714,464)
BicubicResize(640,352,0,0.5)

Futurama script:

LoadPlugin("C:\PROGRA~1\GORDIA~1\mpeg2dec3.dll")
LoadPlugin("C:\PROGRA~1\GORDIA~1\decomb.dll")
mpeg2source("D:\DVD\Projects\FUTURAMA_SEASON_2_DISC_1_VTS_01_PGC1\FUTURAMA_SEASON_2_DISC_1.d2v")
FieldDeinterlace(blend=false)
crop(10,4,702,568)
BicubicResize(608,464,0,0.5)

Playback:

For SBC and DivX5 I used the DivX5 filter at strength 4. I have been using the DivX5 filter to play back DivX3 movies a lot in the past and whenever I thought that something was odd I switched back to the DivX3 filter only to find out that the problem was the source, not the filter. I have also done the same during the comparison but I'm confident to say that the results would not be any different had I used the DivX3 filter for SBC playback (except for being more of a hassle to compare and make screenshots). For XviD I used ffdhow and set it to use XviD rather than ffdshow's own decoding routines. The build I got had internal XivD postprocessing turned on by default, in future versions you might have to turn on deblocking and deringing in the decoder options in the codec setup. Note that this postprocessing is brand new and the lead programmer suggesed that I used it for my comparisons. In SPR and Futurama, the CPU usage was above 90% so if you don't have a 2.8 GHz or higher CPU you should use Nic's filter (ffdshow has/had a bug when playing back XviD files where QPel was used). The other codecs do not permit postprocessing configuration.

All files were reviewed using Media Player Classic as it can play back AVIs and RV9 files (also QT by the way). Furthermore, the player can forward to any frame in the video which literally saved me days when making screenshots. Thank you Gabest for making such a great player! As I have a flatscreen monitor, I used the monitor's native resolution of 1280x1024 to review all clips. Flatscreens are not good at displaying lower resolutions full screen, plus zooming the video helps to accentuate any problems a codec might have (and which I would've certainly missed when reviewing at a low quality 640x480 resolution). Note that the screen used has a reaction time lower than 25ms which allows for flawless movie playback (and 3D shooters are okay, too).

A note about the screenshots: I did my best to present shots from the exact same frame. Some codecs exhibited delays (for instance in codec A you take frame X, in codec B you have to take frame X+1), and RV9 behaved rather erratically. Sometime I was unable to locate the frame that corresponded to the one used in all other codecs.

Now proceed to the first test.

This document was last updated on May 5, 2003