TechRepublic / 7 billion hypotheses later, SparkBeyond employs an alternative way of applying AI

What if there was a way to automate idea creation? What if you could crawl the web for disparate data and combine it all to provide previously unseen connections and hypotheses? This is where SparkBeyond comes in.

“We view SparkBeyond as a platform to make planetary scale impact on the world. It’s not a search engine. It’s a research engine,” said CEO and co-founder Sagie Davidovich.

“Can AI be applied in a very different way than how it is applied today? Can it invent? Can it ask questions? Can it conduct research? This is the question we started with six years ago when we embarked on this journey.”

The AI company has been making waves since it was founded in Israel by Davidovich and Ron Karidi in 2013. After nearly six years fine-tuning it, SparkBeyond’s platform has generated more than seven billion hypotheses and has had a $1 billion impact on the companies it works with, Davidovich said.

According to Davidovich, SparkBeyond’s AI platform cost “six to seven figures” to use and dozens of Fortune 500 companies, including MetLife and Anheuser-Busch have started work with them. Consulting firm McKinsey & Company has cited the company’s work helping clients using SparkBeyond’s AI tool.

“This is not a theoretical exercise. We have delivered more than 150 meaningful client engagements together and well over $1 billion in bottom line impact for clients,” McKinsey’s head of analytics ventures Erez Raanan said in July at the Innovate’19 Summit in London.

During an Oxford medical conference, SparkBeyond presented the results of its work with a large healthcare organization in helping doctors increase their ability to detect colon cancer early based on historical medical data, lab tests, prescriptions, doctor visits and other data.

“The machine basically found very intricate patterns related to changes in your hemoglobin level over time and other factors which generally you might have a vague notion that ‘yes, they are related,’ but understanding exactly how to combine them and how to abstract them is something that you would not have necessarily tried. The machine explored tens of millions of hypotheses and ideas.”

‍

Davidovich demonstrated just how granular the AI technology is. When you ask the machine what affects housing prices in Brooklyn, it quickly spits out a list of nearby landmarks that increase home values like a bay, pier, healthcare facilities or even gas stations with diesel.

In SparkBeyond’s work with a Japanese retailer to optimize store location decisions, the system figured out that the best places to open new stores were next to laundromats, he said. It makes perfect sense when you think about it—people have time to kill as they wait for their clothes—but these kinds of insights may have been overlooked by decision-makers relying on traditional processes.

Data only tells one part of the story, Davidovich said. By taking data and contextualizing it with external data sources, one can connect the dots to find patterns that would otherwise seem like some anomalies.

The project grew out of an idea Davidovich and Karidi had to create a platform that could crawl the internet for all of the code available on the web. They create one of the world’s largest libraries of open-source algorithms and SparkBeyond can now generate four million hypotheses per minute—a feat that allows the platform to work through hundreds of good and bad ideas every second.

The two founders now believe that challenges like climate change, sustainability, cancer and other persistent problems can be solved with the help of their platform. Davidovich said so far, AI has been used to replicate the cognitive functions of humans but had never been built to come up with creative ideas.

“You start with a problem, you identify root causes, you explore existing solutions, you explore deficiencies in those solutions, through these you identify opportunities for innovation. This is what the world says about your problem and this is what the data says. It’s interesting to see where they overlap,” Davidovich said.

It requires a bit of a learning curve to use because it provides users with results that are not websites but actual answers connected to chains of research.

SparkBeyond’s platform is multilevel and shows you the distribution of sources it’s pulling information from—a sort of visual representation of its thought process.

The results are ranked according to the number of sources that agree and there is a credibility score for each source that is also taken into account. You can blacklist or exclude certain sources if you like and overlay different data streams to find new correlations.

‍

The platform is linked to a repository of 1.2 million data sets and can quantify the strength of the relationship between two points of information based on how much evidence there is on the web.

The company pulls the data on its platform from a network of data partners, proprietary sources and publicly available information sources on the web. But Davidovich stressed that the platform goes far beyond predictions or data to generate insights and models.

Davidovich and his team designed the platform to perform like evolution, splicing disparate data and merging it to create new solutions.

“These are two paradigms, one is inductive and the other is deductive. One is quantitative and one is qualitative. These two worlds have never been bridged before. One part is data one part is knowledge,” Davidovich said

SparkBeyond’s platform is now used in more than 23 industries and has a powerful partner in Microsoft, which has repeatedly praised the innovative nature of the AI system.

“Essentially, the hypothesis engine that we built extracts knowledge from data,” Davidovich said.

“If you can identify the root cause, reverse engineer the system and discover mechanisms of action, then you can recommend actions that solve the problem rather than just saying ‘OK, let’s build a predictive model.’ This complementary approach has allowed us to grow fast because it’s very synergistic with the existing AI approaches.”

The company has grown rapidly in the past few years, opening or expanding offices in London, New York, Tel Aviv, Singapore, Melbourne, and there are now employees working remotely in Poland and Tokyo.

“SparkBeyond allows you to ask lots of questions, to get insights on a daily basis as explainable as possible,” Raanan said. “If you’re a smart corporation, you’ll use that to encourage collaboration between the business side and the data side and the analytics side. Some of the insights are irrelevant and some of them are impactful. Some of them are actionable problems that you can do something about.”

‍

SparkBeyond’s hypothesis engine has been running since 2014 but the web knowledge mining aspect of the work started in 2018. The company is now looking to expand its footprint and make even more partners to help validate ideas, increase their domain knowledge and help implement some of the insights generated by SparkBeyond’s platform.

The company now works with some of the top insurance companies and banks on everything from underwriting to customer churn, employee retention, fraud and credit scoring, Davidovich said. It has helped retailers with location optimization and even assortment pricing.

“We hope that SparkBeyond can move the needle on some of the meaningful problems that impact billions of people. But we don’t want to be delusional. We know that none of these problems can be solved alone. We need all the help we can get in order to make this possible and this is our call for action. We want partners to join us and we want to create a community of data partnerships.”

“There is not enough time left to wait centuries for someone to come up with the right ideas. We should not leave these things to luck. This is about research, problem solving, optimization and in general getting past the cognitive bottleneck.”

For the future, Davidovich said the company imagines a time when it can take the last hundred years of inventions and extract universal principles that can be used to solve modern issues.

He demonstrated a tool that could take the insights from hypothesis engine and try to come up with new inventions. While many of the ideas generated are half-baked and nonsense, they provide useful starting points for human innovators to jump from.

As an example, Davidovich listed some of the United Nations’ Sustainable Development Goals and asked whether they could use the invention machine to create millions of ideas or solutions that could be sorted and improved with the help of experts.

“We want to mine the knowledge of human intelligence,” Davidovich said.

“We’re harnessing humanity’s collective intelligence to solve the world’s current challenges using different sources of intelligence.”

‍