How will (or can) data analysis change production?

I am fascinated by most technology, particularly how it can be applied to the things I am interested in professionally: the production, distribution and marketing of video/filmic entertainment. Two recent articles have stuck in my mind. The Japanese research quoted in the excerpt and How cold, hard numbers can be used to foretell the battle where researchers are using Wikileaks information, among other sources, to predict where violence will break out.

But first the Japanese research.

The team from Tottori University devised a set of mathematical models that measure how much money was spent on advertising before a movie is released, over what period of time, and how much talk the film generated in social media.

Using the models, they predicted the popularity of a variety of blockbusters, including the Da Vinci Code, Spider-Man 3 and Avatar, which they compared to actual revenue generated.

”They appeared to match very well, meaning the calculations could provide a fairly good prediction of how successful a movie could be even before it is released,” said a statement from the Institute of Physics, which published the paper in the Journal of Physics yesterday.

The practical application of the algorithm would help producers and distributors to work out when to optimally spend money on promotion to give the best audience return.

What fascinates me about these “big data” ideas is how analysis of really large amounts of raw data can be turned into useful predictors of behavior. The Japanese research is one way: by analysis historic data and comparing it to results, they can use that information to predict the future.

The team hopes to apply its model to other commercial markets such as online music, food snacks, soft drinks and event organising.

Lead author Akira Ishii said a central benefit of the formula was that it enabled a company to determine the best time it should spend its advertising dollars.

The cost of promotion in distribution is significant, even for modest projects. If that can be more targeted to the optimal time and amount to spend on advertising, then the benefit is clear: bigger audiences for less advertising expenditure, meaning a bigger return on the project for its producers and investors.

My other example, the use of leaked data via Wikileaks, is of more general interest:

The availability of huge amounts of data combined with steady increases in computing power has prompted experts to bring the rigour of objective quantitative analysis to realms that were once considered fundamentally subjective, including literature and the study of social groups.

The article is quite long but that’s the primary point. By analyzing big enough data sets, patterns emerge. The choice of data and the patterns that it reveals, of course, can be quite variable. What’s particularly interesting is that, given a large enough set of data, “subjective” responses can be predicted. Not at the individual level, but at the aggregate level. No-one is yet to the Minority Report level of predicting individual future crimes, but in terms of understanding “subjective” content perhaps has application in the generation of algorithms that can assemble some sorts of video content automatically. (Note I did NOT say “edit”.)

I hypothesize that it should be theoretically possible to get a reading on how successful a project will be, before it even gets into production. And that would be valuable data for anyone.

Ben Balser wrote the “finish” to the post with his comment and I reproduce it here because it completes the thought I was trying to convey.

 Coming from an educational psychology and heavy neurolinquistc scholastic background, I am always fascinated at how good editing simply mimics how our brains perceive and process data in our daily lives. I teach an editing seminar based on this. The more an editor studies psychology and sensory perception models, the more accurate an editor one can become.

Thus, there is a “language” to editing (and that language is cultural), as set of rules. The rules governing how I write this to relate data to you, are the exact same rules Shakespeare used to write his plays. He just knew how to use them more fluently than I. The brain is a huge computer, period. So eventually yes, we will get closer and closer to understanding how our individual and cultural (micro/macrocosms) work. Thus predictions will be more accurate, and automated editing can, in fact, technically, according to how our own brains work, be totally possible.

Though at this point in time that is a huge threat to many egos. Just like astronomy was a huge threat to many religious and governmental egos. In the end, the science will rule, as it always has. Nothing to be afraid of or threatened by, it has always, and will always push humans to achieve greater and greater things. Science has always done that. We will always become greater because of it. I personally welcome it. Great post, Philip. Thanks for helping us keep up these things!

Ben’s thoughts gel exactly my own: ultimately anything we do can – and will – be reduced to the basic thought process. Once analyzed the problem is really about getting the source metadata to feed the algorithm. But, I don’t think anyone currently working in the industry need feel threatened. This process will take years.