The Long Tail of Artificial Intelligence
I was first introduced to Chris Anderson at the World Innovation Forum over a decade ago in New York City. Chris is the editor of one of my favorite tech publications, Wired Magazine and the author of a truly mind twisting book called: “The Long Tail: Why the Future of Business Is Selling Less of More”, published by Hyperion in the summer of 2006.
Starting with a memorable case study on N-Sync, Chris argued that products in low demand or that have a low sales volume can collectively build a better market share than their rivals, or exceed the relatively few current bestsellers and blockbusters, provided the store or distribution channel is large enough — i.e. the internet. Since then, the term long tail has gained popularity as describing a new retail paradigm whereby (mostly tech) companies sell a huge variety of different products or services in relatively small quantities. Just think of Netflix, who’s US catalogue boasts 6,494 movies and 1,609 TV shows — there really is something for almost every taste. As the world’s premium OTT (Over The Top) media distribution provider (and a top production company), Netflix pioneered targeting thousands of niche audiences, turning them into millions of viewers. Yes, it has top flagship shows like House of Cards and Stranger Things which bring in millions of viewers, but Netflix also has plenty of fringe standup comedy and indie content as well. In my book, one thousand groups of one thousand discerning viewers is better than a mundane milieu of one million mainstream muppets. This is especially true in a world of (Big Data) Predictive Analytics. This philosophy is also relevant when its comes to the world of Big Pharma and the so-called orphan drug epidemic.
Now, Netflix does an OKish job of recommending what to watch next based on crude meta-tags (something Amazon is also famous for), but I believe this is where we in the global tech industry of the 21st Century have yet to scratch the surface. Most of us are not even at the Machine Learning stage of Artificial Intelligence when it comes to the way we try to improve the way our users consume our products and services. Most tech companies are more like a Mechanical Turk than a (sentient) AI.
The future of consumption is born when companies start to actively mine the zettabytes of data they generate from their customers (users) to help predict each individual’s users preference. Companies can then stretch the long tail out towards infinity instead of just relying on simple reinforcement learning. I want Netflix to create an AI that can then create a TV show or film (now that AI has the potential to create actors) that maybe I myself and only I will enjoy to the max. I want a world in which Tesla’s AI can send me a schematic to 3D print a car that is uber-customized to my motoring tastes and needs. I don’t want to show up to my favorite Italian restaurant and browse the menu, I want the Menu AI to browse through my previous curated Italian culinary experiences and come up with the most amazing pasta dish ever — something I would never have come up with using the mere 100 billion neurons in my noggin.
This future, might be here sooner than we realize.
In fact, back in the summer of 2016, researchers at NYU’s Interactive Telecommunications Program created an AI called Benjamin (formerly known as Jetson). Benjamin is a long short term memory (LSTM) recurrent neural network, a type of AI that is often used in text recognition. To train Benjamin, the researchers fed it (him?) with a corpus of dozens of sci-fi screenplays from the internet — mostly movies from the 1980s and 90s. Benjamin then dissected them down to the letter, learning to predict which letters tended to follow each other and from there which words and phrases tended to occur together. Now that’s Big Data! Not to get too mathematical, but this is worth noting that the advantage of an LSTM algorithm over let’s say a Markov chain, is that it can sample much longer strings of letters, making Benjamin better at predicting whole paragraphs rather than just a few words. LSTM is also better when it comes to generating original sentences rather than cutting and pasting sentences together from its corpus (so perhaps there is an opportunity for a digital AI upgrade to CRISPR/Cas-9 application of genetic editing).
Impressively, and in just 48 hours, Benjamin quickly learned to imitate the structure of a screenplay, producing stage directions and well-formatted character lines. The only thing the AI couldn’t learn were proper names, because they aren’t used like other words and are very unpredictable. (I wonder if that’s why they decided to rename the AI to Benjamin).
So screenwriters can rest assured, if Sunspring is anything to judge the power of AI when it comes to screenplay production, these so-called robo-writers still have a long way to go.