We live in a world where the term “Big Data” really means large swaths of data. However, have you ever thought to think about where this large swaths of data exists? Or who has access to it? Or if a single group of people could view all the data at once? Thinking about these questions presents the problem of partial information. The idea that while there exists massive amounts of data on a subject out there in the world, very few people have access to the entire set. Without the entire set, we as operators on this partial data set are forced in to making sub optimal decisions based off of a sub optimal data set. In fields like healthcare, medicine, financial trading, and other high pressure quick reaction fields, partial information could be the cause of your failure as much as the cause for your success.
Before diving into how we can solve for partial information, let us explore what exactly it represents. Assume that you are a VC looking at a portfolio of companies to invest in. You are a competent VC and have devoted your life to exploring the investment tactics of your fellow competing VC firms as well as your firm’s historical investing pattern. You have a data set of your own portfolio and your competitors portfolios. You know where the money comes into your firm from and where it goes out. You know where your portfolio companies put money and what type of people they invest in. You know the products these companies build and how that creates wealth or use for the end user, but that is where your chain of information ends. You have no way of knowing how this wealth (Wealth from this point forward defines anything gained by the end user including a positive experience) is spent by the end user and transmitted to the next step in the chain, or how the money is spent from that hop onwards. You have no sensors to pick up on that information. This is a case of partial information. Had you as the VC known the entire chain of money in the industry you were focused on you would be able to map far more accurately how spending tactics and behaviors were changing and because of that where the next “Big” idea would come from. Partial information is knowing a facet or a portion of the entire data set and making conclusions from it. If we don’t solve for partial information, we are learning to make bets and predictions and assertive statements without understanding the complete picture. I.e only a little better than shooting darts in the dark.
How can we solve for partial information? Its a tricky problem that I believe cannot be solved with a blanket solution. It will not be a one size fits all solution. Furthermore, it may be impossible to ever have a full data set, and have complete information about a given topic. However, what isn’t impossible is building platforms that allow us to crowdsource the completion of the partial information we all individually own. Each industry will have a different way of accomplishing the sharing, however, by sharing the entire industry will be better off. Take for example health care:
Each individual patient knows who they feel is a good doctor in x y z areas. Each doctor knows that they can trust some small subset of x,y,z doctor. However, neither party may know anything about a doctor alpha because neither party has information to base decisions for that doctor. Yet, there may exist an individual that the patient knows and trusts that knows doctor alpha and has simply not revealed that they have that data point. The data exists, but cannot be surfaced because it is in multiple locations. What if we turned this situation on its head, and created a platform that surfaced that the data existed within your grasp? Say, we had told this patient that they had a friend that knew doctor alpha. They would have been able to make a much more informed decision if that was the case than they could without the information.
One might ask, why didn’t the patient just ask the other trusted party about doctor alpha and discover that the trusted party had a data point on the issue? My response is with another question: How would the patient know to ask this trusted party about Doctor alpha when they don’t know that the trusted party knows anything about doctor alpha? It would be like asking a 5 year old to solve a calculus problem. Unless you knew they were a prodigy you wouldn’t even attempt, because you would reason there was only a very slim chance that they could answer your question.
This concept of partial information is one I think will hamper our success in making judgements from information until it is effectively solved for all possible situations. We need to learn how to effectively share our knowledge, as that is the only way we can continue to make decisions and growth from that knowledge.