Amplifying Editing

I frequently get asked if I think there will ever be AI “Automated Editing.” It’s somewhat of a moot point, as there is already a whole lot of automated editing going on! It’s been going on for a long time.

In an April 2016 blog post I wrote of four automated editing tools already in use.

Wibbitz, a tech startup in New York, has developed a program to create videos automatically by adding footage and stills to an audio track that it generates. Link
With directr you choose the template and the app then tells you what shots to take. It then builds them into a video based on the template Link
SundaySky create up to 1.4 million unique videos a month: SundaySky, Tel Aviv/New York-based start-up with 50 employees, with no video camera or production staff, will produce 1.4 million video clips this month for a range of big retail and real estate corporate customers including Overstock.com and the History Channel. The company pulls customer data into a customized template which creates videos with movement, music, narration and graphics and video. Link

Of those five year old examples, directr is no longer with us, but Wibbitz is a current competitor to Magisto. Technically none were AI or ML based back then.

If your source material is highly structured then template driven editing is fairly straightforward. Our company, Intelligent Assistance Software created a custom app for a client that built FCP Classic Sequences based on a template XML and their repository of highly structured files. I imagine SundaySky were doing something similar.

Moving in to the ML era, you may already have experienced automatic editing. Richter Studios’ Blog AI and the Next Decade of Video Production (which we are now four years into) talks about:

“Apple’s “Memories” movie feature (in their iOS “Photos” App) provides a great glimpse where AI is taking editing. In very little time, you can create and select a photo album, add a music style, determine the video length (Short, Medium or Long) and – PRESTO – your video is created in just a few seconds”

Automated personalized videos. Of course, Apple now have the Clips app, which I confess I’d never used until now. At a minimum, clips is using ML for Speech-to-titles, and for scanning for Augmented Reality effects, like the light up dance floor I’ve always wanted in my office!

Clips is targeted directly at consumers, while Wibbitz and Magisto are focused on a corporate market, but open to individuals. My associate Cirina Catania (a producer-editor herself) was very impressed with the results from Magistro:

I did nothing except shoot/pick the clips, graphic style and music- they cut!
Cirina Catania

Magistro – using ML Models – chose the parts of the uploaded clips with the most action, and matched them into templates. Cirina and I were both impressed with the results. As recently as 2016, choosing clips based on action (and emotion) was a co-operative research project at IBM Watson and 20th Century Fox to pull selects for the Morgan trailer. Now it’s part of a mainstream app creating hundreds of videos a day.

Magistro, and its competitors, are certainly templates at their core, but intelligently selecting the shots that go into those templates is where ML is adding significant value.

Temploratization itself is not a new topic. Terence Curren and I discussed the topic in a 2011 episode of The Terence and Philip Show.

The trend toward basing creative endeavors on templates has been a trend for many years, culminating in Hollywood’s use of its history as templates for its current production.
Whether this is a good or bad thing depends on whether you value your personal creativity, or you’re pushing a budget to get a project finished.

Honestly, if you’ve watched any significant amount of the House Hunters franchise on HGTV the template would be obvious. All that’s needed is a little real-time logging and the right ML Models, and at least the first assemble is done ready for final shot selection and trimming.

Sports is another area where automated editing is making strong inroads into finding and building custom highlight or recap packages. Asset Management specialist Evolphin added the ability to “automatically edit videos using AI-generated metadata”. Back in 2019 Evolphin VP of Video Product Management Evan Michals told StudioDaily:

“The killer app, of course, is AI video editing — the ability to automatically assemble exactly the clips that are needed for a given recap or highlight reel. Instead of having an editorial staffer pore over footage and manually select relevant clips, Evolphin says, the system can be used to search for the face of a given player at specific moments in a game, such as goals or penalties, or for types of action, like cars drifting on a race track.
With the correct filters applied, Zoom will retrieve the corresponding moments and generate a sequence that can be saved as a video file or exported as XML for use with any video editing software that recognizes XML files.”

In an Adobe Blog article from 2019 The State of AI in Video Tatiana Mejia writes

Video intelligence platforms using machine learning help marketers gain an increasingly granular understanding of where their audience is watching and why they’re watching. Such insights enhance a marketer’s ability to interpret and act on their audience’s wants, needs, and goals.
That ability to speak directly to people’s priorities will only increase as hyper-personalization becomes more routine — in future years, marketers may be able create AI-driven video content targeted to an individual viewer.
Other future potentialities in AI for video include gesture control, use- and context-specific tagging to improve product discovery, and neuromarketing and biometric sensing to monitor viewer response. Context-aware marketing, which is also gaining traction, uses natural language processing to better place video ads against relevant video content.

At this point I can hear voices (and I can identify some of them) and they’re saying “but that’s not real editing!” In the sense that those voices mean it, I will agree. While all these types of automated editing have their place, they are not creative editing in any form.

Clips, Magistro, Memories, etc have their uses. If I’d used Magistro, maybe the video from our last Zion National Park trip – about eight years ago – would have been edited before we went back again. This editor never got to the personal project, so the automated tool would be a better result, because it would be a result!

Birthday party videos would be edited promptly, instead of when someone can “get around to it,” or in time for their 21st birthday compilation video!

Super smart templatorization could conceivably assemble the basic cut of a highly formulaic show like House Hunters, ready for a pass by the “craft” editor.

Outside, the highly formulaic that lends itself to templatorization, there’s no suggestion, or research that suggests creative editors and creative editing will become obsolete in the foreseeable future, but a studio camera switcher might feel a little threatened, particularly after seeing the demise of studio camera operators!

Back in 2014, before the modern ML era, Disney Researchers “…have developed a groundbreaking program that delivers automated edits from multi-camera footage based on cinematic criteria.”

An interesting research project, but not in the least “real” editing, and also not particularly practical.

It’s going to take a couple of quantum leaps forward for any sort of AI that can do the creative editing of even a junior level editor. Thirteen years later nothing comes close to our own First Cuts, which was most definitely not AI

The obstacle to an autonomous creative editor are phenomenal. The sheer complexity of the task; the lack of a training set, and the lack of any other suitable training method pushes the likelihood way off into the future. In the present we can use all the organization advantages of visual and speech metadata to amplify our personal creativity as an editor.

Introduction, AI and ML, Amplified Creativity