Amplifying Pre-production

Before a single frame of a feature film or television series is shot, it is likely ML has been involved. Writers focus on whether or not an AI can write a script, but that’s probably the wrong question. Attempts at storytelling are the subject of the last section, but we can safely say that script writing will – at a very minimum – need input from humans for a very long time to come.

The right question would be whether there are ML based smart assistants that are in use, or proposed, for pre-production. None of these write scripts, but like all creative amplifications, these tools are freeing creatives to spend more time on what they alone can do. Whether it’s helping decide which projects to green light, to automated storyboarding tools in development, to breakdowns and budgets, or voice casting, ML is already creating Amplified Creatives.

Greenlighting

Even before a script hits pre-production data science and ML have already decided which scripts could be turned into blockbusters.

“Netflix uses machine learning to determine expected hours of viewing for each piece of content, estimate the cost per hour viewed, and compare it with that of similar content deals [7]. Additionally, the firm uses predictive models to understand customers, such that there is a large enough set of content that meets their preferences without necessitating the renewal of any specific title [7]. This cost-effectiveness is particularly important as increased competition bids up licensing and renewal agreements.

While ML’s attempts at writing scripts are amusing and somewhat more detailed in the next section, they are of no practical value. Priya Kantaria makes that point in AI in the movie industry has a way to go to produce the next set of Oscar-winners, but as he also notes that they use AI to plan scripts:

“They use similar techniques to predictive marketing in planning scripts, to see what’s being talked about, what’s popular. Trying to identify for scripts what might resonate with the audience,” he explains. “But we haven’t quite got to the stage yet where we can replace scriptwriters and even really good actors.”

Here again is the Amplified Creative approach using the best of human and machine together.

Movies and TV Projects have always tailored themselves to the audience. It is, after all, the hallmark of The Hallmark Channel. It’s also the point of focus group advance screenings. Movies have entirely new endings added after that feedback, notably Sweet Charity. When projects meet their audiences’ needs, they make money. Creatives who make money often get the opportunity to do it again!

Tailoring scripts to customize them to audiences is ubiquitous. That we do it better with the help of machines is no surprise.

Storyboarding

While essential to the creative process, storyboards are challenging because they entail a lot of work that won’t ever be seen outside the production. What would be helpful, is for a smart tool to analyze the script and automatically generate (useful) storyboards.

People have been working on that problem for a long time, right back to the paper Towards Automatic Animated Storyboards from 2008. Eleven years later and Disney have a prototype automatic script to storyboard tool.

“Disney Research and Rutgers take the idea one step further with an end-to-end model that can create a rough storyboard and video depicting text from movie screenplays. Specifically, their text-to-animation model produces animations without the need for annotated data or a pre-training step, given input text describing certain activities.”

Not yet a production tool, but a whole lot closer than in 2008. The article is worth reading because it explains some of the challenges of extracting the right information from the script. This type of useful smart tool takes multiple layers of analysis, understanding and visualizing, all incredibly complex in themselves.

It’s also worth reading just to note how quickly these fields are advancing, In an aside at the start of the article they quote a precedent to the current automatic tool from 2018, just a year earlier:

“Last year, researchers detailed a system that tapped a pair of neural networks — layers of mathematical functions modeled after biological neurons — to generate videos 32 frames long and 64×64 pixels in size from descriptions like “playing golf on the grass.”

This is a computer generating imagery and animations from text descriptions (now) parsed from Disney’s scripts. Still need the human to write the script, but I wouldn’t invest a lot in a storyboarding career!

Breakdowns and Budgeting

Another task that seems well suited to automation is the process of breaking down a script into cast, locations, crew, etc. It is an important adjunct to the creative production process that is essential for the financial well-being of the project.

Startup RivetAI develops ML-infused moviemaking tools designed to streamline preproduction. One of three cofounders Debajyoti Ray explains:

“It helps people to create content much faster,” Ray told VentureBeat in a video conference, “by using AI to augment creativity. Everything starts with data.”

Once again, the goal is to augment creativity, not to replace it. Anyone who has broken down a script into storyboards, shot list generation, optimizing schedules, and creating budgets knows that these are tedious tasks that are essential, but not at the core of creativity. Necessary to create the framework where creativity happens, but in the way of creativity.

RivetAI automates that process from the finished script, changing a multi-week task to one done in minutes, with the option of human over-ride because it’s still not perfect.

From the breakdowns RivetAI calculates a budget:

“With no more than a script, it can estimate the required number of shoot days and prep days (down to the length of scenes), predict a project’s total budget, and spit out a line-item, department-by-department bill of materials.”

The article makes it clear that this is intended as a ballpark figure, not one to raise funds on, but automating so much tedious work out of the process, frees resources to go into the production itself. The co-founders are also:

“… adamant that RivetAI’s suite of tools only aid, rather than interfere with, the creative process.”

You can find a discussion of the data science and philosophy of RivetAI by Debajyoti Ray at medium.

Voice Casting

This example isn’t a generalized voice casting agent replacement, but rather a tool for matching talent recording localized versions of movies and television to the vocal qualities of the original voice.

“Exploring Automated Voice Casting for Content Localization using Deep Learning” by Aansh Malik is a technical paper that explores the use of deep learning to automate what is now a largely manual workflow for casting voiceover talent across languages and cultures. … Malik considers ways to leverage developments in deep learning for text-independent speaker verification (TI-SV) to enable computer-aided voice casting. https://ieeexplore.ieee.org/document/9395671

In what I expect will become a common story, by the time automatic voice casting for localization becomes a readily available tool, it will have become irrelevant. Given advances in automatic translation, voice synthesis and facial manipulation, the more likely future is a suite of automated tools that translate the script, synthesize the new language in the voice of the original actor, and manipulate the mouth and face of the actor from the original shoot, so they mouth the new translated performance. You’ll visit some of the research projects attempting to make that a reality, in a later part of this article.

Introduction, AI and ML, Amplified Creativity