This question comes up briefly in Guns, Germs, and Steel, but the author brushes it aside. Understandable, since it's one of the most difficult, most divisive questions around these days. But it's one that I'm extremely interested in, one I think about all the time, mostly because I'm interested in eventually trying to develop intelligent systems.

So there are lots of angles to come at this question, but here's the main one I take: If I wanted to develop an intelligent agent, what behavior would I want it to exhibit?

Alan Turing took at shot at this question in his famous 1950 essay "Computing Machinery and Intelligence". If you're not familiar with the "Turing Test", it basically goes like this:

The new form of the problem can be described' in terms of a game which we call the 'imitation game'. It is played with three people, a man (A), a woman (B), and an interrogator (C) who may be of either sex. The interrogator stays in a room apart from the other two. The object of the game for the interrogator is to determine which of the other two is the man and which is the woman. He knows them by labels X and Y, and at the end of the game he says either 'X is A and Y is B' or 'X is B and Y is A'.

In the case of machine intelligence, the judge concludes that X is either a human or an artificial intelligence. Y would be whatever X is not.

But this test is embarassingly bad for lots of reasons. As prerequisites for intelligence it presumes:

• natural language proficiency
  
• cultural knowledge
  
• adeptness at deception
  

If I were designing an intelligence system, why would I necessarily want my system to exhibit these skills? You can see how the test itself might drive the design of the system.

So what if I wanted to create an agent that could exhibit the following behaviors:

• spatial awareness
  
• self awareness
  
• progressive learning
  
• symbol acquisition and manipulation
  
• causal reasoning
  
• inference
  
• mathematical ability
  
• memory and recall
  
• physical dexterity
  

Which of the above do you associate with intelligence? Which should you associate with intelligence?

Would the Turing Test be useful in measuring or detecting any or all of the attributes listed above? Not really.

I think it's important for AI researchers to do two things, before real progress is made in the field:

1) A reasonable definition of intelligence, including its primary facets, must be developed
2) A good test, or rather battery of tests, must be developed to measure the various facets of intelligence

I think such a test would probably be very complex itself, composed of dozens of tasks, questions, and passive observations of subjects. I think such a test would necessarily need to be as objective as possible, normalizing biases of the evaluator. Such a test would also need to account for the variability in conditions and in the sensory apparatus of the subject (for example, an intelligent system may perceive the world through sonar and infrared).

Actually trying to conceive of such a generalized, comprehensive intelligence test seems daunting, but I think it is vital to frame the way people think about and approach AI.

I think the basic attributes of intelligent behavior can be approached by researchers in an incremental way. For example, most people would agree that a mouse is smarter than a fly, that a monkey is smarter than a mouse, and that the average human is smarter than a monkey. Upon what criteria do we make that evaluation? I personally think it is based upon a wide array of cognitive skills, many of which I mentioned in the list above.

If we can work to develop systems along this scale, I think the fruits of AI research will be realistic. If we try, out of the gate, to create a chatbot that can fool a judge into thinking it is human, researchers are ludicrously wasting their time and effort.