Thanks to an introduction by a mutual friend, I had the opportunity to chat with Alex LoVerde of SyncOnSet, and it stuck me that the best technology is one driven by a direct, and often personal need. It also struck me how different two ostensibly similar “metadata companies” can be.
I don’t always cross post my appearances on Larry Jordan’s Digital Production BuZZ, but I thought I did a particularly good explanation of the basics of AI and Machine Learning and how they might apply in production, that I thought I’d share this one.
https://www.digitalproductionbuzz.com/interview/philip-hodgetts-the-basics-of-ai-explained/#.W9iRBi2ZN1Q
Greg and I will be at IBC 2018 and we’re looking forward to seeing you there.
If you’d like to pick our brains for up to an hour, then schedule a meeting with us. We’ll run through your workflow and offer suggestions on where there might be efficiencies, or we’re happy to demonstrate the innovative Lumberjack Builder. If you’re a Lumberjack customer, we’d love to hear how you’ve been using it and how it could be better for you. We’ll even buy you a beer!
Other than those meetings we’ll be mostly hanging around the Atomos ProRes RAW Theater as that seems to be the center of FCP X action this year.
As there’s no Supermeet this year, those of us who’d see each other there, are celebrating the Not Very Supermeet so come join us there.
Web APIs (Application Programming Interface) allow us to send data to a remote service and get a result back. Machine learning tools and Cognitive Services like speech-to-text and image recognition are mostly online APIs. Trained machines can be integrated into apps, but in general these services operate through an API.
The big advantage is that they keep getting better, without the local developer getting involved.
Nearly two years ago I wrote of my experience with SpeedScriber*, which was the first of the machine learning based transcription apps on the market. At the time I was impressed that I could get the results of a 16 minute interview back in less than 16 minutes, including prep and upload time. Usually the overall time was around the run time of the file.
Upload time is the downside of of web based APIs and is significantly holding back image recognition on video. That is why high quality proxy files are created for audio to be transcribed, which reduces upload time.
My most recent example sourced from a 36 minute WAV, took around one minute to convert to archival quality m4a which reduced the file size from 419 MB to 71MB. The five times faster upload – now 2’15” – compared with more than 12 minutes to upload the original, more than compensates for the small prep time for the m4a.
The result was emailed back to me 2’30.” That’s 36 minutes of speech transcribed with about 98% accuracy, in 2.5 minutes. That’s more than 14x real time. The entire time from instigating the upload to finished transcript back was 5’45” for 36 minutes of interview.
These APIs keep getting faster and can run on much “heavier iron” than my local iMac which is no doubt part of the reason they are so fast, but that’s just another reason they’re good for developers. Plus, every time the speech-to-text algorithm gets improved, every app that calls on the API gets the improvement for free.
*I have’t used SpeedScriber recently but I would expect that it has similarly benefited from improvements on the service side of the API they work with.
For a book project I recorded a 46 minute interview and had it transcribed by Speechmatics.com (as part of our testing for Lumberjack Builder). The interview was about 8600 words raw.
The good news is that it was over 99.98% accurate. I corrected 15 words out of a final 8100. The interview had good audio. I’m sure an audio perfectionist would have made it better, as would recording in a perfect environment, but this was pretty typical of most interview setups. It was recorded to a Zoom H1N as a WAV file. No compression.
Naturally, my off-mic questions and commentary was not transcribed accurately but it was never expected or intended to be. Although, to be fair, it was clear enough that a human transcriber would probably have got closer.
The less good news: my one female speaker was identified as about 15 different people! If I wanted a perfect transcript I probably would have cleaned up the punctuations as it wasn’t completely clean. But reality is that people do not speak in nice, neat sentences.
But neither the speaker identification nor the punctuation matter for the uses I’m going to make. I recognize that accurate punctuation would be needed for Closed (or open) Captioning for an output, but for production purposes perfect reproduction of the words is enough.
Multiple speakers will be handled in Builder’s Keyword Manager and reduced to one there. SpeedScriber has a feature to eliminate the speaker ID totally, which I would have used if a perfect output was my goal. For this project I simply eliminated any speaker ID.
The punctuation would also not be an issue in Builder, where we break on periods, but you can combine and break paragraphs with simple keystrokes. It’s not a problem for the book project as it will mostly be rewritten from spoken form to a more formal written style.
Most importantly for our needs, near perfect text is the perfect input for keyword, concept and emotion extraction.
On the night of the Supermeet 2011 Final Cut Pro X preview I was told that this was the “foundation for the next 10 years.” Well, as of last week, seven of the ten have elapsed. I do not, for one minute, think that Apple intended to convey a ten year limit to Final Cut Pro X’s ongoing development, but maybe it’s smart to plan obsolescence. To limit the time an app continues to be developed before its suitability for the task is re-evaluated.
At Lumberjack System, we frequently get push back that it’s “too hard” to log during the shoot. If you’re manipulating a camera (reframing etc) then sure you can’t log. And if you’re holding a boom, ditto. But if you’re monitoring audio during recording, or running the interview, then it is totally possible to log during the shoot with no added stress.
I do it all the time. For my family history project I set up cameras, mics and audio records and run the interview and log. Lunch with Philip and Greg is much the same, with the added complication of eating.
The best approach is to work in back-time mode and relax. Back-time eliminates the stress of anticipating when an answer starts because you’re logging “in the past”
I typically work with the 5 second back time on by default. This means I can be fully engaged with the subject while asking the question, and continue to be engaged with them as they start their answer. I can glance down after a few seconds (fewer than five!) and tap on the keyword start.
This takes the stress and tension out of having to “get it right on the moment.â€
Back time also allows us to add a new keyword and log it from up to 90 seconds in the past. Keyword range end is always the current time.
Two recent announcement place IBM’s Artificial Intelligence play, Watson, right in the sports spotlight.
Watson is being used for tagging World Cup coverage, and the relationship with Wimbledon from picking highlights and enhancing user experience to, this year, designing the poster!
This is the first time I’ve taken a deep look at a TV show and worked out what I think would be the perfect metadata workflow from shoot to edit bay. I chose to look at Pie Town’s House Hunters franchise because it is so built on a (obviously winning) formulae, and I thought that might make it easier for automation or Artificial Intelligence approaches.
But first a disclaimer. I am in no way associated with Pie Town Productions. I know for certain they are not a Lumberjack System customer and am also pretty sure they – like the rest of Hollywood – build their post on Avid Media Composer (and apparently Media Central as well). This is purely a thought exercise built around a readily available example and our Lumberjack System’s capabilities.
In a new Terence and Philip Show we start with the question “Should Apple be present at Trade Shows like NAB?” and then extend discussion to question whether there is still a role for big trade shows like NAB and IBC.
http://www.theterenceandphilipshow.com/episode-80-shoul…o-to-trade-shows/