Artificial Intelligence Index Measures Technological Progress – AI100

What We Need Is An Artificial Intelligence Index Of Some Kind!

If you listen to some of the more famous entrepreneurs and scientist you would think that we are on the verge of creating thinking self aware machines that may or may not look like humans and may or may not want to rule us. There is more than a little evidence to suggest that artificial intelligence science may indeed be going through a growth spurt. After all it has been a thing since the 1970’s and needed to be patient until computers became fast, powerful and inexpensive.

This does beg the question though since we all are aware of the possible risk’s for artificial intelligence. and we do not want to find out we are heading in a bad direction only after it is too late, of how we manage it. We need some sort of index to help us measure we are on the terror versus happiness scale of A.I.

Stanford University has taken the lead this will a multitude of A.I. stakeholders to create something called the AI100 Index. It is made up of companies like Google,Facebook, Amazon and Microsoft just to name drop a few. There is a complete report of course but here is the summary of findings after the first index measurement in the fall of 2016.

This Stanford-led AI index reveals a dramatic increase in AI startups and investment as well as significant improvements in the technology’s ability to mimic human performance.

“The AI100 effort realized that in order to supplement its regular review of AI, a more continuous set of collected metrics would be incredibly useful,” said Russ Altman, a professor of bioengineering and the faculty director of AI100.

We were very happy to seed the AI Index, which will inform the AI100 as we move forward.

The AI100 was set in motion three years ago by a charitable gift from Eric Horvitz, a Stanford alumnus and former president of the Association for the Advancement of Artificial Intelligence. Its first report, released in the fall of 2016, sought to anticipate the likely effects of AI in an urban environment in the year 2030.

Among the key findings in the new index are a dramatic increase in AI startups and investment as well as significant improvements in the technology’s ability to mimic human performance.

Baseline metrics

The AI Index tracks and measures at least 18 independent vectors in academia, industry, open-source software and public interest, plus technical assessments of progress toward what the authors call “human-level performance” in areas such as speech recognition, question-answering and computer vision – algorithms that can identify objects and activities in 2D images. Specific metrics in the index include evaluations of academic papers published, course enrollment, AI-related startups, job openings, search-term frequency and media mentions, among others.

“In many ways, we are flying blind in our discussions about artificial Intelligence and lack the data we need to credibly evaluate activity,” said Yoav Shoham, professor emeritus of computer science.

The goal of the AI Index is to provide a fact-based measuring stick against which we can chart progress and fuel a deeper conversation about the future of the field.

Shoham conceived of the index and assembled a steering committee including Ray Perrault from SRI International, Erik Brynjolfsson of the Massachusetts Institute of Technology and Jack Clark from OpenAI. The committee subsequently hired Calvin LeGassick as project manager.

“The AI Index will succeed only if it becomes a community effort,” Shoham said.

Although the authors say the AI Index is the first index to track either scientific or technological progress, there are many other non-financial indexes that provide valuable insight into equally hard-to-quantify fields. Examples include the Social Progress Index, the Middle East peace index and the Bangladesh empowerment index, which measure factors as wide-ranging as nutrition, sanitation, workload, leisure time, public sentiment and even public speaking opportunities.

Intriguing findings

Among the findings of this inaugural index is that the number of active AI startups has increased 14-fold since 2000. Venture capital investment has increased six times in the same period. In academia, publishing in AI has increased a similarly impressive nine times in the last 20 years while course enrollment has soared. Enrollment in the introductory AI-related machine learning course at Stanford, for instance, has grown 45-fold in the last 30 years.

In technical metrics, image and speech recognition are both approaching, if not surpassing, human-level performance. The authors noted that AI systems have excelled in such real-world applications as object detection, the ability to understand and answer questions and classification of photographic images of skin cancer cells.

Shoham noted that the report is still very U.S.-centric and will need a greater international presence as well as a greater diversity of voices. He said he also sees opportunities to fold in government and corporate investment in addition to the venture capital funds that are currently included.

In terms of human-level performance, the AI Index suggests that in some ways AI has already arrived. This is true in game-playing applications including chess, the Jeopardy! game show and, most recently, the game of Go. Nonetheless, the authors note that computers continue to lag considerably in the ability to generalize specific information into deeper meaning.

Shoham said:

AI has made truly amazing strides in the past decade, but computers still can’t exhibit the common sense or the general intelligence of even a 5-year-old.

Just to keep it in perspective we thought we would include Dr. Steven Hawkings 2017 discussion on the pros and cons of Artificial Intelligence Science.

About the AI100 Artificial Intelligence Index.

The AI Index was made possible by funding from AI100, Google, Microsoft and Toutiao. Data supporting the various metrics were provided by Elsevier, TrendKite,,, the Google Trends Team, the Google Brain Team, Sand Hill Econometrics, VentureSource, Crunchbase, Electronic Frontier Foundation, EuroMatrix, Geoff Sutcliffe, Kevin Leyton-Brown and Holger Hoose.

A Stanford-led AI index reveals a dramatic increase in AI startups and investment as well as significant improvements in the technology’s ability to mimic human performance

(Image credit: Tricia Seibold)


(Visited 89 times, 1 visits today)

Artificial Intelligence Index Measures Technological Progress – AI100

1 Comment

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

This site uses Akismet to reduce spam. Learn how your comment data is processed.