Adobe is heading into AI/Machine Learning full steam ahead. A private room NAB demo showed picture search and dialog search powered by IBM Watson, and now this collaboration with Stanford organizes takes, matches them to lines of dialog, recognizes voices, faces and emotions, camera framing, and more and then builds dialog-focused stories in any style desired.
What makes this exciting is that this system is that it “understands” film idiom – that we start with a wide shot, for example – and that there are different styles of editing.
You can also use leisurely or fast pacing, emphasize a certain character, intensify emotions or keep shot types (like wide or closeup) consistent. Such idioms are generally used to best tell the story in the way the director intended.
The system then assembles a cut very quickly: a 71s cut in just three seconds!
We’re moving in this direction, and I don’t know how long it will be between a “collaboration with Stanford” and a shipping product, but regardless, these types of tools will increasingly affect our workflows. For now, there are limits:
The system only works for dialogue, and not action or other types of sequences. It also has no way to judge the quality of the performance, naturalism and emotional beats in take. Editors, producers and directors still have to examine all the video that was shot, so AI is not going to take those jobs away anytime soon. However it looks like it’s about ready to replace the assistant editors who organize all the materials, or at least do a good chunk of their work.
I added the emphasis, because – as Terence Curren and I have been discussing on The Terence and Philip Show, these type of job losses are somewhat inevitable. It also emphasizes that it’s not true ‘creativity’ but 90% of what we do isn’t.
One reply on “New “AI Film Editor” Cuts Scenes in “Your Style””
Inspiring and a little scary! But I’m not ready to bury assistant editors just yet.
Nobody talks about the craft of organization in editing, just “selection, arrangement, and pacing.” Good org guarantees shots are always located, then selected, arranged and paced, no matter what. The AE’s job is the backbone of any cutting room.
One of Philip and Greg’s IA products was doing a lot of this auto cutting years ago! It also prefigured a way of identifying shots, rather like Roles, now used in FCPX. Amazing.
I would like to see a product which helps automate the organization of difficult-to-peg documentary material as well as more conventional dramatic work. I’m not sure how that would work. A smart robot might help.
To explain, I had a very nice gig recently, a series of museum videos, my first in this genre, loved doing them, and the edits were good, but it took me several tries at organization before shots appeared where I needed them.
The script scenes weren’t numbered! (Yeah, I should have flagged it, but it was a client I always wanted to work with, when in Rome, etc). The raw material was precut newsreel clips, and I’d go haywire because shots contained in many of those precuts were useful for several scenes! So spoiled was I by scene numbers, slates, scenes, preferred takes and original footage.
After dumping page numbers as a scheme, (scenes would usually float between revision pages) I settled on bins named after topics in each script- if I’d had scene numbers my work would have been cut out for me (or an assistant on a larger assignment of this kind). A parent /child number scheme could account for most all changes, such as new material– I.E, Scene 5 is now followed by Scene 5A, expanded material. Simple scene numbering: easy to overlook. Topic folders are another way to cope. But even with topics identified, with each script revision, missing shots helped destroy my schedule.
Maybe by describing what I settled on, some ideas for a bot which could be deployed by an editor or assistant may arise. It would;t be a bad idea to have the bot interview the assistant to tabulate parameters, sort of like TurboTax.
I tried subclipping, just grabbing the hot shots for each topic, but tossed it, because it’s useful to examine adjacent material immediately and not have to constantly release clip limits, match frame, etc.
So, since most of these were short subjects, one-reelers, I settled on full master clips containing material for several topics, and in Premiere Pro CC used markers (locators in Avid). This way, when duplicating those clips they became independent affiliates pointing to the same media file, but in each dupe, I was able to wipe out the previous set of markers and mark new material relevant to another scene and drop it into that topic folder. Believe it or not, this obvious solution took time to discover.
In general this is why good assistants get the work. They set up the conventions for organizing a particular kind of show. Many are not beginners. Some are older than the editors!
It may be that future assistant work entails highly creative programming of highly mutable automation software to organize a particular kind of production, to set up bin folders and the like. I just don’t see the work of AE’s going away entirely, but like trained machinists who once ran huge lathes and milling machines under direct control, today they use C&C numeric control and lay out the cut for the machine to follow.
Machine vision to recognize content will up the ante, but it’s still GIGO, garbage in , garbage out– you must accurately set up the required search patterns, identify the content roles, etc.
I doubt the job will be eliminated, it’ll morph into a new era using astounding new tools which still require human interpretation to get the most efficient result and keep editing a happy craft.