Amplified Storytelling

Everything starts with a story, so the obvious question is, “can Machines tell compelling stories.” Could a machine write or direct a “movie”? When it comes to results so far, we should be asking if they can even tell coherent stories!

People have been working on these questions for years. There have been experimental projects where a machine has been trained on “science fiction scripts before writing their own.” The result was Sunspring, which is definitely a swing and a miss! Each of the projects I’ve mentioned will be discussed in detail in this section, with links.

Another experimental had an “AI” create a highly stereotypical car commercial. Speaking of stereotypical, the Eclipse project used AI to create a music video. While there have been lots of experiments, left to themselves, the machines create laughable, or incoherent results.

Where there has been success, it has been in machine/man synergy, or Amplified Creativity. On Impossible Things the machine suggested the storyline and some oddly specific requirements for the Trailer, but the script was written by humans!

All these experimental projects are interesting. None of them suggests that creative storytelling by humans is at all threatened.

West End Musical

While not a movie or television project, an excellent example is a collaborative effort between composer and machine: Beyond the Fence. It was billed as the first stage musical “written by computers.” Like most of these articles, that’s a little stretch, in that no one computer or algorithm wrote the musical. Benjamin Till, credited as the creator, worked with a series of ML models.

“There is not a single song in this show, not a single moment, that wasn’t at some point inspired by or written by a computer,” said Till. Altogether, computers fully determined the premise and generated 25 percent of the music and lyrics.

Till turned himself into an amplified creative who finished the first draft of Beyond the Fence in a few weeks, and the team finished the entire show in four months. By comparison, the “unamplified” creative spent over a year just researching and writing the first draft of his previous musicals. That’s roughly a 13x increase in creative outputs: a pretty serious amplification of Till’s creativity.

Or course, it’s one thing to write a musical, but was it any good? One reviewer described it as “risibly stereotypical but pleasant as a milky drink.” Even an associated research paper described it as “a plausible and acceptable musical.”

Before we dismiss it though, “stereotypically bland” describes a lot of television production. It’s not high art, so maybe machines do have a role in creating hundreds of hours of “stereotypically bland!”

Even Amplified Creatives should focus more on the creativity than the “amplified.” Don’t use Machine Learning just because you can.

This is also a good time to re-iterate that very little is created by only one person. Filmmaking, stage production, television production, etc are all team sports, so it’s probably unreasonable to expect AI to do it alone.

Looking at some examples will give us a fuller idea of how various creatives have experimented with ML for storytelling.

Car Commercial

As an experiment, filmmaker Kevin Macdonald directed a 60 second commercial using AI to generate the script:

“The computer produced a script featuring a sentient-like Lexus ES that hits the open road, whizzing by stunning vistas of shoreline and forests before saving itself from a dramatic crash.
“I thought this was something amazing that had all these ambiguities in it and strangeness in it,” Macdonald said. “It’s only a matter of time where the formula of what makes up a great story, a great character can be learned by a computer.”

The LA Times article teases with the “AI writes script” trope, but in fact re-iterates the marketing and script reviewing tools I discussed above. The example – a 60 second car commercial with sweeping vistas – hardly makes the case for AI originality.

Science Fiction

In a more experimental mode, Columbia University School of Arts’ Digital Storytelling Lab has been working on AI in more experimental filmmaking with projects like the aforementioned Sunspring and Frankenstein AI: A Monster Made by Many. While way out of the mainstream, these type of experimental projects are valuable for opening the field of thought, and perhaps seeding an idea further into the future.

At the current stage of development these projects end up being hilariously bad (Sunspring) or rely on humans to complete the project .

On Impossible Thingsthe machine did the data analysis and suggested the story points that would resonate with an audience of horror movie fans. Apart from suggesting that the general story be about ghosts and family relationships, the machine made very specific instructions for the trailer (“needs a piano and a bathtub”). The actual writing was done by talented humans.

Music Video

Eclipse is a pop video featuring a French Band that had “AI as its director.” While what was done was interesting and more than a little amazing, it’s a stretch to claim the AI “directed.” There was no AI auteur involved, A series of Machine Leaning models were used for different processes along the way, starting with the music track.

From there:

The first machine gave script and a treatment, derived from the music
Machine 2, took the treatment and directed cameras on drones for the shooting
During editing another machine decided where cuts should be made according to the beat of the music and its emotional content.

Even the cast were selected by examining EEC brain scan data from actors and matched them to the emotions detected in the singer on the original track. Interestingly, the producers (independently) made the same casting choices. Perhaps it’s not only voice casting that could fall to the machines.

The final comment in the BBC article sums it up:

Eclipse will not win any awards for creative film-making, and its director won’t be snapped up to make a Hollywood blockbuster.
But AI is advancing every day – and a decade from now, actors may find a computer sitting in the director’s chair shouting: “Cut!”

The Black Box

In a 2020 article, A.i.Can Make Music, Screenplays and Poetry. What About a Movie?, Mike Pearl conjectures a future where he describes a putative future “Black Box” where you select from options like genres, plots, types of character, locations etc, and even content keywords, and “Black Box” will create a fully “produced” unique movie that fits all the desired characteristics. He describes the theoretical process that could make this happen, and pretty much every quoted authority agrees it’s highly theoretical.

Duke University AI scientist Davit Carlson Ph.D. thinks that someone will eventually try to achieve the Black Box, but even he says:

“That’s a long way off. You know, years at the minimum.”

And ultimately the article ends up where everyone else has: it will be a collaboration, another form of amplified creativity.

Documentary Production

Lance Weiler wrote a very good and wide ranging 2018 article Here comes the Automation, in which he describes Tommy Pallota’s More Human than Human (original page unreachable) documentary where Mr Pollota documents his journey to try and replace himself as director with AI Tommy Pallota is responsible for the rotoscoping that brought Waking Life and A Scanner Darkly into existence, and is a multi Emmy award winning director himself.

Interwoven with his efforts to collaborate with (and ultimately be replaced by) an intelligent machine are an examination of AI’s rapid acceleration and its impact on humanity.

The project was elaborate, involving robotic arms, robots interviewing people and being part of the edit team.

The robot could formulate questions, read emotions, detect body temperature, determine whether someone was lying as well as recognize a human’s age.

I will note, that I do not detect body temperature, reliably tell if someone is lying, or reliably recognize a human’s age, so the machine has that going for it!

Ultimately with the current stage of AI’s involvement in storytelling is best summed up by Mr Pallota: