AI and Production: The Future

It’s relatively easy to get an overview of the current state of Artificial Intelligence (AI). It’s probably easier to understand the benefits of machine learning, particularlyMachine Learning (ML) that’s already applied to common tasks that we can benefit from now because we’re fitting those new technologies within existing frameworks.

What is much harder to determine, is how machine learning will be directly applied to post production processes, and what role AI will take in our collective production future.

ML has been most successful where it’s assigned a particular task to learn. So, an all purpose “editing” machine is unlikely in the near future, but ML will almost certainly be applied to automate ‘formulaic’ editing in the near future.

Truth be told, much editing is at least somewhat formulaic. Wedding, Corporate, Education, News and more follow a pattern, but are unable to be automated with a template because they are never perfectly consistent in the shots.

Back in January 2011, I wrote about SundaySky producing – at that time – 1.4 million video clips a month. No-one was claiming that this is high art. These are promotional pieces for big retail and real estate corporate customers including Overstock.com and the History Channel, and built on templates, with different shots dropped in for each product or home.

If we say that a lot of production is not highly creative, what do we mean by “creativity”?:

The use of the imagination or original ideas, especially in the production of an artistic work.

While a wedding video may be beautifully produced, I’ve shot and edited enough to know that no two are identical, but they all follow a pattern, and there is a certain consistency to the content across all weddings.

Similarly, most production is not highly creative, but hard to templatorize. Templates work  well where there is a very strict consistency between shots. For example, we’re working with a client to create a simple tool that builds videos from a template. Into each version of the template, the same shots from a different version of the product are inserted. This works for SundaySky and for our client because their workflow is highly regimented, with strong consistency across every shoot, so the shots slot into the template comfortably.

But in Wedding, News, Corporate or Education, there are strong patterns, but nothing is rigidly controlled enough to fit a template. This is where ML will be used.

ML learns from examples, so – for example – we could expose one to a large number of wedding video examples, and have it learn to edit wedding videos. Similarly, expose ML to enough examples of corporate, education, news or other genre and the machine will learn to make the same types of edits, themselves.

This is inevitable. Right now this would be a little beyond the capability of today’s ML, but given the exponential growth of these technologies, today’s impossible, is next years automated editor.

At this point the response is usually, “Machines may be able to do the routine, but they’ll never be truly creative”.  Using the definition of  “imagination or original ideas” I agree.

Many years ago I was editing a documentary around a young ‘triple-threat’ performer (singer, actor, dancer). I had footage of his performance as Rolf singing 16 going on 17  in a professional production of The Sound of Music, and footage of his next day charity appearance singing Hanky Panky. I doubt that any ML is ever going to think to intercut those two performances, but it worked brilliantly.

Is the writing of a theater musical creative? Beyond the Fence is billed as the first stage musical written by computers. Like most of these articles, that’s a little stretch, in that no one computer or algorithm wrote the musical:

For this project, Till (the author) and a team of researchers from around the world designed a series of algorithms to create the “magic formula” for a hit musical. Each part of the production—the premise, the narrative, the lyrics, the score, and even the size of the cast—was determined by a computer system

Like the trailer for Morgan being ‘edited’ by computer, the computers functioned as very, very smart assistants.

There is not a single song in this show, not a single moment, that wasn’t at some point inspired by or written by a computer,” said Till. Altogether, computers fully determined the premise and generated 25 percent of the music and lyrics.

When you dive into the article there are a series of algorithms in play: from analysis of stage musicals for the success formula, to writing part of the music, there is a lot of computer power in there:

Till wrote the first draft of Beyond the Fence in a few weeks, and they finished the entire show in four months. For comparison, he spent over a year just researching and writing the first draft of previous musicals.

This stage musical certainly has elements of computer creativity in it: the plot points were computer generated as the most likely to be successful, after all.

But that spark, that moment of creating something never before seen, is a long way off.  But never say never! Almost all the technology we’ve been discussing in these three articles was impossible five years ago. Everything I do day by day was impossible when I started in video.

Also, keep in mind those music composition examples in part 1.

What is important is to note that Beyond the Fence was completed with a human/machine cooperation in four months. Three times faster than just the research phase of other musical projects from the same team.

This is the ultimate bottom line: computer algorithms and humans working together can be creative, faster. The bottom line drives all production, so innovations that speed the process will be used and take over.

While computers in general aren’t likely to be imaginative or original without some human input, they will be very effective creative partners.

These technologies will help creative people achieve their outcomes faster.

It will also enable the near-complete automation of formulaic production, which will eliminate a lot of editing positions.

Change is inevitable. Our response to it is where we have control. We can ignore or fight-off the incursion of AI and ML into our world, or we can embrace the increase in productivity and the how we can focus on the truly creative – imaginative & original – parts of what we do.

It will also make it easier for creative producers/preditors to be more self sufficient in realizing their vision. AI assisted human creativity is our immediate future. I’m keen to ride whatever ‘bicycle of the mind’ is available to me.