Introducing AV Foundation and the future of QuickTime [Updated]

Introduction to AV Foundation http://slidesha.re/aYEJfR To be honest I don’t know why this isn’t hidden behind an NDA, but it’s not and until someone has it taken down, and asks me to do the same, I’ll consider it public knowledge.

Now, AV Foundation is the iOS media system, so we’re not talking about QuickTime per se but I have to wonder.

QuickTime – the real OS-centric media framework, not the little sub applications that function as players – is transitioning from C APIs (Carbon) to Cocoa via QTKit. Trouble is, QTKit got a lot of work around QuickTime 7’s release, but not so much in recent years. And yet Final Cut Pro needs a lot of what’s not written, before it can release a Cocoa version of Final Cut Pro.

Actually, Apple could do what Adobe have done for Premiere Pro CS5. In rewriting their core media handling engine, Adobe retained QuickTime support by spinning it off into a 32 bit thread, but that’s a complex workaround that does nothing for performance, nothing positive anyway.

When you consider slide 9… Even though it was only introduced in iOS 2.2, extended in iOS 3 and “completed” in iOS 4 (consider the reference framework growth in slides 6, 7 and 8), AV Foundation has 56 Classes and 460 Methods (the more you have of these, the more you can do with it). QTKit has 24 Classes (less than half) and 360 Methods. Compare that with the (very mature) QuickTime for Java with 576 Classes and more than 10,000 Methods. Something tells me that QTKit is not in favor at Apple.

Not that I think QuickTime is going away, at least not as a brand for their media players and the overall technology. I say that because, although the code that’s in iPhone OS shows a simplified player, that was all that was originally released and it shared no “QT Classes or Frameworks”. So, the QuickTime brand is likely to be retained.

If I was extrapolating from this presentation, and I am extrapolating wildly from a small amount of data, I’d guess that the direction within Apple was toward the more modern Classes and Methods of AV Foundation, and that, eventually, AV Foundation, Core Audio, Core Animation and Core Media will replace what we currently have under QuickTime on OS X: Core Audio, Core Video (well, just a subclass of Core Image) and a lot of deprecated (do not use) C APIs.

If you consider slide 14, and the similarity of Classes between QTKit and AV Foundation it makes no sense to build two technologies in the company that were essentially doing the same thing.  Slide 29 shows how similar an AVAsset is to a QTMovie. The other Classes all seem to duplicate functionality that’s in QuickTime now, but in efficient, new, modern code. Capture, editing, playback, media formats… they all seem to be in AV Foundation duplicating work done (or not yet done) in QuickTime’s QTKit.

Importantly Core Media Time is in “n’ths of a second” not “ticks” or “events”. Media based on time will be better for video frame rate uses than one based on ticks or events, which caused the “Long Frames” problems of earlier versions of Final Cut Pro.

In support of my hypothesis I offer slide 42: specific references to AVAssetExportSession.h being available in OS X with 10.7 and likewise CMTime.h has a reference to becoming available in 10.7.

So, I’ll go on a limb and suggest that QuickTime as we’ve known it is somewhat dead; long live a new QuickTime. QuickTime will continue being the branding, but everything “below that” will transition to new architectures essentially ported from iOS to OS X.

This would be a very good thing. A completely new, modern, efficient (you see what it does on the iPhone) underpinning for QuickTime down below that QTKit layer.

Who wouldn’t want to use that in an modern NLE, even if it means waiting for OS X 10.7, which hasn’t been announced yet? It would make it much easier for the Final Cut Pro team to create a much more powerful media engine than it has now; one that really understands time and not events and one that mimics the power of Adobe’s Mercury Engine. Let’s face it, media performance on a 1 GHz A4 chip is in some ways better than the performance on 8 core processors. iMovie for iOS, built on these frameworks (if slide 24 is to be believed) can edit Long GOP H.264, which Final Cut Pro can’t! (And in both cases the H.264 playback is accelerated by hardware: dedicated chips in the iPhone, on the graphics card in OS X.)

As always, conjecture on my part, and this time based solely on what I’ve learnt from the quoted slide show. Chris Adamson does not work for Apple but he does claim expertise in iOS and QuickTime. Other posts on his blog indicate some differences between AV Foundation and QuickTime; and Classes still missing from AV Foundation that are in the current version of QuickTime. That shakes my confidence in the hypothesis a little, but given how little work has been done on QTKit in the last two years, and the need to have the foundations for QuickTime modernized, it still seems like the most likely path Apple will take.

Another data point is that the QuickTime X player was promoted thusly:

Using media technology pioneered in OS X iPhone™, Snow Leopard introduces QuickTime X, which optimizes support for modern audio and video formats resulting in extremely efficient media playback. Snow Leopard also includes Safari® with the fastest implementation of JavaScript ever, increasing performance by 53 percent, making Web 2.0 applications feel more responsive.*

Pioneering the technology under iOS, and then porting it to Mac OS X has happened already.

UPDATE: Chris Adamson, who did the presentation I referred to, clarified many of the points I get wrong or wrongish, including the fact that AV Foundation is not under NDA. His Connecting the Dots post is essential reading if you’ve got this far!