The Prediction Business

Businesses are in the prediction and risk business, a bold statement if there ever was one. No matter what, when running a business, there are going to be things that are known and things that are unknown. We can control the things that are known to us and act on them in clear ways. But the unknowns are a bit trickier because they are unknown. We just don’t know what they are there. For example we don’t know how our customers or the market are going to respond to a new product. Or if we should be doing a heavy amount of fundraising based on the customer growth curve and whether or not we have sustainable operations to continue heading down this path. This is the risky part of the business.

In order to better understand how to predict these variables, more companies should be collecting as much data as they can about their businesses and understand it the best they can. Data is the 21st century oil. Reading it is the 21st century literacy. Big Data is what the press is calling it. Almost all companies collect data now and because of this, it is much better to understand how it works, what is the information is trying to convey, and how the information can be used to make a forecast, tell a story, or just inform. General statistics has always been used to mine and learn from data in the past. But with faster computer speeds and access to an almost unlimited bandwidth, it is now much easier to run much more advanced algorithms over the data using what is known as machine learning. What is machine learning? It is applying statistical models to the data you or your company has so that smarter predictions can be made about the data you don’t have.

With each new set of data we encounter, new uses for algorithms must be found. And while machine learning has a great amount of potential to the way companies approach their business problems and the way entire industries operate, it can still be thought of as a branch of statistics that is to be used on big data. And the tools that machine learning bring are designed to make better use of that data.

How can we approach thinking about big data and machine learning along with it? The enormous scale of data available to firms can be challenging; using machine learning is as much about data analysis as it is about adapting to the sheer size of any particular data set. A great way to think about data is how long and wide it is. What is meant by this statement is our data set will be long depending on how many rows it happens to have. Let’s say we are analyzing a large company’s data and what it will look like; we can imagine each row being one unique customer. Depending on the size of the company there could be up to millions or even billions of customers. So with that line of thought, the more customers there are, the longer or higher our data set will be. Width then corresponds to the number of columns in the data set. So in our case, each column is considered a unique variable assigned to our customers. For example, our columns can be purchase and browser history, mouse clicks, and even text. This data set can become rather large and overbearing and this is where machine learning makes use of a tool set to better analyze wide data.

We can further refine our initial question down further by asking what machine learning is used for? The most common application is to make predictions and this is why it is becoming so important to businesses. Being able to make predictions about data that isn’t available can be used to formulate sales, marketing, operations, and financial strategy. Here are a few examples of how it is used in industry today:

  • Personalized recommendations for each customer. (Amazon product recommendations, Spotify and Pandora recommending new music, and Netflix movie recommendations)
  • Forecasting customer loyalty (How often they shop with a company down to the time and what they consistently spend their money on.)
  •  Fraud detection and credit card risk (More banks and insurance companies are using their data to make predictions about what customers may be a moral hazard)
  • Facial recognition software (Facebook makes great use of this when it recommends who should be tagged in a photo)
  •  Advertisements that create their own copy and images (M&C Saatchi partnered with Clear Channel UK and company called Postercope to create these ads.)
  • Personalized assistants (Apple’s Siri, Google’s Now, and Microsoft’s Cortana are just the big name examples of what can be accomplished. There will be many, better assistants down the road.

The common identifier is the need for a unique business process and the decision that must be acted upon to get to that accurate prediction. Each of these examples come from complex environments where a correct decision depends on many different variables. (Our wide data). And each prediction will ultimately lead to an outcome with whatever it is helping the model become continuously better.The business value of machine learning is enormous even with its limitations are taken into consideration. It is focused on prediction which means the model of the environment might be all that is needed to make the right decision.

So let’s get into how machine learning can be used in practice. Within each machine learning algorithm there are generally three broad concepts. They are:

  • Feature Extraction: This determines what data to use in the model.
  • Regularization: Used to determine how the data are weighted.
  • Cross-Validation: Tests the accuracy of the model.

What each of these concepts does is separate the “signal” from the “noise” which is common in most every data set and helps sort through the mix to get to better predictions.

Feature extraction is the process where the variables that the model will use are discovered. There are times where all features are dumped in to a model and used but more often than not this doesn’t happen due to overfitting. Features help aggregate important signals that are spread out over the data. For example, if your company runs an online music store, each feature could correspond to musical genre, record label, or even the artist’s home county. Once these data points are collected they are combined through automation that clusters the features together and the model can then analyze customer predictions. A very well-known business case is Netflix’s movie recommendation algorithm. The more each customer uses their product, the more data points they are able to collect about that user and the company is better able to predict what movie or television show the customer is interested in watching.

After we have our features chosen we must understand if the data we have been collection and what it is being combined reflects a signal or noise. So we begin by playing it safe with the model using regularization. This is a way to split the difference between a flexible model and a conservative model. For example, one effect is known as “selection” which happens when the models algorithm focuses on a smaller number of features that contain the best signal, discarding all other features. Regularization helps the model stay away from overfitting, (overfitting is when a model learns patterns from the data that ultimately are not helpful and won’t hold up in future cases) and helps it learn from both signal and noise.

In order to test the accuracy of the models predictions, a process is used called cross-validation. To test that the model is “out-of-sample”, which is when predictions are made on data we don’t have based on data we do have, our initial definition of machine learning. This is done by splitting the data into two sets called the training and test data. The model is first built using the training data and then more tests are done with the use of the test data. Keeping a clear partition between the two sets is instrumental in not over estimating how good the model actually is.  

There are many examples of machine learning being used in production that we use on a daily basis. In some cases, we might not even be aware that our technology is using it in the background. Netflix was used as an example of a business that makes great use of its data. Amazon is also extremely data driven with their product recommendations being used skillfully with each customer who shops with them. However, the company that probably uses machine learning the most right now is Alphabet, Inc. (The Company formerly known as Google.) Machine learning not only guides how their search engine works so efficiently, but is also used in Google Translate, Nest, their self-driving cars, Google Now, and many other products they offer. The more data they collect from us, the better they will be able to fine tune their algorithms in their products so that they will interact with us seamlessly.

One final intriguing example is how a digital agency is using artificial intelligence to create ‘self-writing’ campaigns in London. How it works is the ad itself is placed on a bus stop and has a camera connected to it. This camera registers commuters’ engagement based on whether they look happy, sad, or neutral. Then, an algorithm executes various responses based on the commuters’ responses to the ad. This campaign in particular only used a fake coffee brand since it was more of a test than anything. But if the proof of concept works, we may start seeing more interactive billboards out and about.

Being able to make the correct forecast and predictions for your company isn’t something that can be done with 100% accuracy, but businesses that do utilize the data they are collecting to its fullest potential find they are better able to cope with the uncertainty of variables they can control. Forecasting isn’t about getting the answer to your questions correct, because that isn’t going to happen. Forecasting is about being able to make sound judgement from the data and the algorithms used to mine that data will help anybody or business get a bit closer to the answers they are looking to find.

The Creation and Capture of Value

In order for a business to create value after cost it must create and distribute that value in the most efficient way possible. It can be considered the starting point for any and all businesses which leads to our first question to think about: how is that value created? Simply put, it is created through work. That work could be anything from administrative tasks (such as filling out the right paper work for customer orders), technical (deploying code to servers), and creative (marketing copy, product and/or logo design, etc.). The business can then create value through that work, sell or trade it to a customer base, and capture some of that value through profit. Based on this definition, we can begin to clearly see that businesses add value in more ways than just making a product and selling it. Every moving piece of that business should be moving towards the end goal of creating and capturing that value.

Now let’s start thinking about the types of value that can be created. First, let’s remember that not all types of value are created equal by any means. A value that is considered a commodity is easily replaceable in the minds of customers. For example, if your products aren’t distinguishable from your competitors, then that competitor will be primed to take your place should your business falter by any means. There are ways around this though; if your company is able to create a new and more efficient process for doing business or in possession of unique skills focused on the creation of value, then you will be able to more readily differentiate from those trying to eat your lunch. Having any of these things is a competitive advantage and should be fully utilized towards the goal of value creation.

Measuring value creation is important if only so that you understand what the value is that your business is making. The first method is by measuring revenue. Revenue tells you that the way your business creates value was worthwhile to your customer base since they are willing to pay for it. Notice how I didn’t say profit. Many businesses successfully create revenues but no profits (think Amazon), but many will not be able to do this for very long. A business needs a profit in order to survive and sooner or later a lack of making any profits will bring that business down, no matter how great the product or service. (Amazon has been able to successfully navigate this pasture by shrewdly reinvesting its revenues into future initiatives such as AWS.)

Then there is perceived and exchange value, which are interrelated. The exchange value is straightforward in that it is the amount of value exchanged between a buyer and seller for a product or service. For example, if you go to the store and purchase a pair of shoes, the price you pay for those shoes is the exchange value.  The perceived value is defined by our perceptions of usefulness of the product or service. In economics there is a consumer surplus (C.S.); when the consumer surplus is greater than zero (C.S. > 0), then the customer is better off making a purchase than not. So value is created when the perceived value of a product or service has a certain degree of usefulness, consumer surplus is greater than zero (C.S. > 0) and that same value is then exchanged to the seller.  

As we can see, it’s incredibly important for a business to create value. It won’t survive for very long if it doesn’t create value by differentiating itself in the marketplace or not making a profit in the long run. In order for a business to survive, it must capture a portion of that value it is creating. If businesses ultimately want to succeed, they must think clearly about how they are going to capture the value they are creating. Businesses that don’t do this may be leaving money on the table.

There are a number of different ways to capture value with some being more common than others. For instance, price based on value changes according to the offerings worth to the customer. What this means is businesses don’t set their prices based on what their competitors are doing. They might also discontinue the process of marking up prices based on production costs. Instead, what they are doing is looking at what their customers want and setting their prices accordingly.  What is important is that the customers’ perception of value must be discovered. This is obviously different with each customer but there are different models of discovering this missing piece. An obvious example is auctioning, which doesn’t work with all business arrangements but is incredibly effective when it does. The most common example of an auction is with online advertising, where each buyer sets the price and the seller can choose whether or not to take that offer. Of course this is controlled less and less by human interaction and is guided more efficient using algorithms to guide the software. There are some downsides such as prices that may be less than satisfactory for the seller, but must be honored regardless.

Another model is known as demand-driven pricing; what this does is let the price change due to the fluctuations in demand for a product or service. The most common example to date is Uber and it’s constantly changing prices. Although there can be a large amount of complaints if the price is too high, there are reasons for it, and Uber acts on those reasons with a mechanical efficiency. The business is a money making machine. How it works is the company raises and lowers its prices based on demand for rides. Other factors that are taken could be day of the week, whether it is a holiday or not, weather, and even city, etc. Using these variables (and many, many others), the company can maximize its profitability by knowing how many cars to have on the road and in what location at any given time. Demand-driven pricing at its finest.

A further form of value capture model enables customers to set their own prices. Although this doesn’t take place in most industries, it is especially prevalent in the travel industry where buyers can decide what price they want to pay and sellers can take it or leave it. Unlike auctions though, this transaction is mostly kept between the buyers and sellers. Doing this lets the seller maintain their prices even if they are discounting for incremental sales.

Next up is for companies to capture value by using two-sided market forces to their advantage. Although the name may not ring a bell to most people, it is a model everyone has seen in action and is used efficiently by media companies. For example, many publications are free for the general public to take (think local periodicals in many major cities) and they make their money, in turn, by charging advertisers to place ads in their publications. The money is then used to subsidize the content that is created for the publications and the process is continued. Although Vice Media is a much larger entity now, when it was just a magazine, it was free to anyone (who could find it) and the vast amount of value was captured by charging advertisers (especially American Apparel). Of course they made money with subscriptions, but most people just searched high and low for a free copy.

Then there are those businesses that use what’s known as the “price carrier” in their offerings. This is the experience that businesses will hang a price tag on while customers may not be coming through the doors just for that in the first place; they may be coming for something else entirely. Think about it, we will sometimes purchase a product or service to gain access to something else the company is offering, but has no price tag on. Starbucks is a great example of this. The majority of their customers walk through their doors for a quick coffee before work in morning, but there are those people who come in just for the wi-fi. Using it isn’t for sale, but purchasing a coffee or pastry will give you access to it. So a good question to ask yourself is whether or not your business is hanging the price tag in the right place. What would happen if you moved it to something else?

One of the most prominent examples of value capture is called the razor-and-blades model, which was pioneered by Gillette. It’s a rather simple model: customers get the razor handle for free, but must purchase blades continuously since they get dull rather easily. And charging for the blades is where the value capture happens. This model was also utilized by technology companies who sold printers and printer ink replacement cartridges. The printers are always cheaply priced; the ink is not.

The final model to talk about is one we all know well since the phone carriers are incredibly efficient at using it; it is known as bundling. How it works is the price of the new phone is subsidized by any extra hardware, software, and data features we purchase with it. The phone is cheap; the options that are bundled with the purchase are where the value capture takes place. Car dealerships also excel at this too; when you go to purchase a car, the sales people will usually bundle in many products or services you may or may not even need since those add-ons are where the money is then made.

Both value creation and value capture are incredibly important to keep your business running smoothly over a long period of time; more so than your competitors. Both are equally important; both need to be studied rigorously for the best understanding of how the complement one another. Value creation is the work that a business needs in order to create value to offer to their current and potential customer base; capturing that value is what will keep customers happy and the business running smoothly. Combining the two will be what ultimately either keeps a business alive. And that is something to constantly been thinking about when focusing on your own business.

Broken Windows

Technology is changing a large aspect of how we live our daily lives without giving us the time to slow down and catch our collective breath. This aspect of technological change should be enough to make anyone pause, but there doesn't seem to be any end in sight. This isn't a negative, it's just what it is at the moment. I'm also not saying that this is a good or a bad thing; it just is. Where my concerns are starting to focus on is what the trade-offs we may not be paying attention to are. Technological change is a Faustian bargain and it seems to be happening before our eyes without an consideration to the trade-offs of living our lives around it.

What am I talking about regarding trade-offs? Technology gives and technology takes; there is always a hidden price to pay for letting it become the centerpiece of our lives. This isn't at all obvious to most people. For most, the price they pay may be of greater importance than the trade-off. For example, take Facebook. In order to use the service we must sign up and agree to the terms before we can begin using it. That agreement is the price right there; we are agreeing to let Facebook do whatever they want with the data that they collect about us. And there are a variety of resources they use that data for such as marketing and advertising, making their product stickier to customers, and user experiments. The ultimate price we pay is our online privacy. I'm not condemning or condoning it; again, it's just what it is.

Online services aren't the only technologies with major trade-offs. Another example is smart-phones and their now ubiquitous nature. They are fantastic devices that many people in the world use on a daily basis, but at what price? How difficult is it to have a conversation with someone without them staring at their phone in the middle of you talking with them? How has it changed our social interactions with other people, and is that a good thing? Automobiles are another example; we were suddenly able to get somewhere faster and more conveniently than ever before and that revolutionized the start of the 20th century due to not only how they were made (Ford lines) but to how it gave people the ability to travel easily. Now we know that they are responsible for contributing to air pollution and traffic gridlock. They have done an incredible amount of damage to our environment and that will not be easy to fix, if it ever can be.

Technological change is our modern day version of Bastiat's Broken Window Fallacy. We buy into a new technology and can only see one side of the benefits. For example, we might start using an app that hires people to deliver us goods (such as food or clothes) and/or services (such as taxis). By doing so, we revel in the new found freedom from the convenience it provides our lives and the time it now affords us, allowing anyone to now focus on much more important parts of their lives. But this is only one side of the story. We aren't opening our eyes or even looking for the trade-offs that are taking place. And this needs to change.

Very rarely do we ask what this technology is undoing for us. What is the other side to that trade-off? How does using this new technology effect the people who are working for it (such as drivers for Uber or deliveries from Postmates). Are the laws being changed to accommodate these new companies and should they even be changed in the first place? What will be the long term effects of those changes to our economy and even our own jobs? There are some very interesting critiques of both Postmates and Uber.

These are all questions that are difficult to answer but the very real point that I am arguing about in this essay is that most of us know very little about the social and psychological effects that new technologies have on our society. And it's in our best interest to really start thinking about, and understanding the costs that these technologies bring to us. 

Predicting through Randomness

I wrote this back in January of 2015. I wanted to wait until closer to the end of the tournament until I posted it. What I found interesting about it was that although I didn't answer all of the questions, those questions I did answer gave me a fairly low brier score, which is used to show how accurate the forecast happened to be. Sticking with financial markets and indexes seems to suit me. Political maneuvers we're a bit harder to tackle. In the end though, forecasting is a great tool to have in the toolkit, but it should only be used by those who are going to focus their entire attention span on the subject.

7/3/15 - Just received my feedback report and I placed in the top 5-6% of the tournament. Will have to keep focused on this. 


Forecasting is hard. There is no way around it. When you are trying to forecast the outcome of a certain idea or situation there is so much back and forth information that is super hard to do. There are people who make entire careers out of forecasting, some more on force of personality than actual hard analysis, and some on actual deep dive analysis. I've been thinking about forecasting now for the past few months since I was accepted into the Good Judgement Project tournament. And all it once it is a fantastic way to learn of what is happening in the world around us and quickly become humbled by what we know and think we know. I've been learning quite a few lessons since I've joined including being humbled by what is really nothing more than my opinion on something. Forecasting, if done correctly, forces you to look hard at the reality of what is happening and make judgments from there. And many times, we have to redo what we were thinking about. 

For a quick history lesson, the Good Judgement Project was started in 2011 by Philip Tetlock and funding from IARPA. Over the course of the next few years, the researchers found that accurate training plus a strong dose of statistics, pyschology, game theory, and interactions among team members helped create highly accurate forecasters. These people were not government trained at all. Most of them were only getting their news from online sources. The same same sources that most anyone has access too. However, these amateur forecasters were displaying a 30% more accurate prediction than those in the intelligence community with access to classified information. I thought this was pretty exciting and applied quickly thinking that it would be great way to finally apply some of the game and probability theory I learned in college. But that wasn't the only reason I wanted to join. 

Thanks to the influx of more technology driven businesses there has been larger amounts of data available not only on open source, but for companies as well. Most companies have been using data to understand their customers for years (the credit card companies can paint an accurate portrait of a consumer and do it with ease), but there are new techniques being used to accurately mine that data. These techniques come from a brand of artificial intelligence and are focused mainly in an area called machine learning. What machine learning does is construct algorithms, study the results of those algorithms, and learn from that data. For example, there is one algorithm called "Association Learning or the Apriori Algorithm". It is generally known as the first algorithm data miners try. What it does is learn interesting relationships among data in large sets. An easy example would be the groceries you purchase from the store. Management can mine each transaction that is made and discover if there is any correlation between items being purchased. Perhaps people who buy apples also buy cheese. Management can then market their products in order to keep that relationship continuing and perhaps even discover more relationships. Many companies that do this can learn quite a bit about their customers (even though some argue this can't be done right now, with the way technology is growing, it probably will soon.)

For me, gaining more confidence in my ability to apply mathematical probabilities to real life events is something I've been wanting to do but never had the proper outlet to succeed at doing. And having spent the past five years learning all I can about computer programming, this finally feels like an excellent outlet for me to pursue. And sure enough, once I started learning how to apply data mining techniques to large datasets, hours started flying by as I immersed myself into this new world. Gaining the right information to then apply as knowledge is becoming the new currency in the world today. There is so much raw data that anyone who can sift through it and correctly analyze and accurately communicate what it means will easily by a one-eyed king in the kingdom of the blind. 

There are a few rules and although I'm not going to give a rundown of the entire training materials, I will explain a few ways it is currently being used.

  • It is a foolish to ask for predictions for the fundamentally unpredictable. What I mean by this is that although there is a method and process for accurately forecasting the probability of an event happening, there are still mostly events that cannot be predicted. The best we can do is ask ourselves the right questions and move forward from there. But the big key is knowing the right questions to ask and when to ask them. Trying to predict where the ball will land in a game of roulette on every play is mathematically impossible (unless there is cheating involved because the probability of guessing correctly just three times in a row is .00182%). However, if you know what the odds are against you (trust me, the house has a 5.26% advantage on every spin) and what your long term expected value for continual play is, then game becomes much more manageable. The movement of the financial markets could also be thought of as unpredictable and if you are going to play that game, your best bet is to have a strategy and system in place. (I'm not going to go into details on this, if you really are interested there is an entire industry dedicated to systems trading. Whether they work or not is another story.)
  • Forecasters need positive and negative feedback. When there is a question you have been thinking about, the worst thing you can do is make your forecast and then walk away when you are done. Forecasting requires constant iteration over the course of the questions lifecycle. The outcomes change constantly and the best thing the forecaster can do is constantly take in the feedback they are getting and apply it the best they can. Not listening to the feedback, even if it's negative can be detrimental to the overall outcome. Always take in the feedback and move forward from there. 
  • Prove yourself wrong. If there is a event of some sort that you are trying to prevent and the evidence seems to point overwhelmingly to a specific answer, try finding evidence that will prove that outcome wrong. Maybe it's just an opinion piece or from a new source that may not be entirely credible, but find it and analyze it anyway. As I stated in the last point, outcomes can change in the blink of an eye, even entirely predictable outcomes. But confirmation bias can be even more detrimental to a proper forecast analysis. Knowing all sides to a story is always a good bet. 

 It isn't just the financial industry that has been using forecasting successfully. Although Everyone from George Soros, Ray Dalio, to James Simmon's Renaissance Technologies are incredibly effective in short term (milliseconds for Ren-Tech) and longer term thinking (macro thinking application in the case of Dalio and Soros), other industries such as technology to sports are effectively using these techniques to model better outcomes for their businesses. The big five tech companies, Google, Apple, Facebook, Amazon, and Microsoft use and offer forecasting techniques in a myriad of ways. Google is by far the most innovative technology wise (type anything into the search input box and watch it "magically" try to predict what you are going to type), but all of these companies use forecasting in ways that weren't even possible 10 years ago. Amazon can predict what you are going to buy and Facebook can predict how you will react to a status update. That is just the basic level of forecasting these companies can accomplish. Then there is sports and more specifically the "Moneyball" techniques that were used in baseball with the Oakland A's and Boston Red Sox. (It helped that the Red Sox owner, John W. Henry owns a CTA fund and has successfully applied forecast and trend techniques to make money in the markets which is how he could afford to buy the Red Sox in the first place). I am convinced that other industries will start getting better at collecting and applying the data they get far more effectively.

Which leads me to write about how using the analytics techniques I mentioned above with more of the technological models (machine + deep learning) to get to effective forecasting. These two types of forecasting aren't mutually exclusive and should in fact be combined. In fact, that is probably one of the best ways that businesses (and human beings in particular) can really start to effectively use artificial intelligence; by using them together to try to forecast better results. Machine learning algorithms can model data faster and more effectively than a human being can but a human being can use quick judgement to understand when to use and not to use certain data. Weak A.I. is still the most prevalent in the world today (think Apples Siri, it's artificial intelligence, but it isn't very intelligent.) but we may start to see the emergence of stronger A.I. over the next 10 to 20 years (machines that can learn, apply, learn some more. Like how a child learns by doing.) As more and more data becomes available, there will more and more of a need to accurately forecast outcomes using that data. Data science and analysis plus applications of new technology in the field of A.I. will play a huge roll in pushing this forward.

From everything I have learned since I joined the tournament last August, I am extremely excited about what will be possible. For me, learning how to apply forecasting techniques has opened my eyes to my own capabilities and what I would like to do with them. I've always been fascinated by the emergence of A.I. and have wanted to somehow work in the field and am started to find ways to become more and more involved. I had already been pushing forward learning as much as I could about machine learning (yes, I took the Coursera class, but I still want to learn more applications) and since I have started to mine more open datasets, how these algorithms really work is becoming more clear to me. It is a fascinating and exciting field to be in and I can't wait to see what's to come.  

Truth

“When you’re young, you look at television and think, There’s a conspiracy. The networks have conspired to dumb us down. But when you get a little older, you realize that’s not true. The networks are in business to give people exactly what they want. That’s a far more depressing thought. Conspiracy is optimistic! You can shoot the bastards! We can have a revolution! But the networks are really in business to give people what they want. It’s the truth.”

- Steve Jobs

This quote is from 1996 but it still rings very true today, especially with algorithms controlling much of the content that we are seeing. It was important back then and doubly important now but we must learn to think critically for ourselves as opposed to just accepting what is being fed to us. Some of it may be true and some of it may be false, but the best we can do is make that decision for ourselves, not by a large company.