An issue in cognitive science that's been popping up in many different things I've been reading lately is what is known as the type-token distinction.

A "type" is basically a category or class, like "cats" or "gifts" or "things you take to the beach".

A "token" is a particular instance of a class, like "my cat Murray" or "your grandfather's pocket watch" or "my half-empty bottle of suntan lotion".

One place where this issue has cropped up, though it wasn't stated in terms of the type-token issue, was in Stephen Pinker's book Words and Rules. The book uses the extended example of the way in which English speakers learn the past tense for verbs to generalize about how language works.

Here's the general pattern of learning the past tense of verbs in kids:



It's a u-shaped curve, which is unusual, and therefore interesting. What this figure reflects is that in the early stages of learning verbs, when they've only acquired a small amount of verbs, children conjugate the verbs with a high degree of accuracy (including regular and irregular verbs). For example, they say things like "Daddy went home" and "I walked" (point A). However, as they begin to learn more verbs, their accuracy in using the past tense decreases, and they start making more mistakes, usually by overgeneralizing the past tense. For example, they say things like "Daddy goed home" and "The doggy taked the bone" (point B). Then as a larger amount of verbs are learned, children's accuracy increases again (point C), and they learn not to overuse the rule for regular verbs (just add "-ed").

Pinker's explanation for this pattern is that when children begin to learn the rule for regular verbs, they begin to try to apply it to everything. Since most verbs are regular, this makes a certain amount of sense. Pinker points out that when we create novel verbs, by turning a noun into a verb (e.g., "they really Clintoned that one") we invariably use the rule for regular verbs. So regular verbs are made by applying the rule, but what about irregular verbs? Pinker argues that they're stored as particular instances. Thus, language is made up of words and rules.

He criticizes a neural network model by Rumelhart and McClelland from 1986 which tried to explain past tense acquisition in children. The model showed a u-shaped learning pattern, but the modelers trained the network by tailoring the input, so that irregular verbs were trained first, followed by regular verbs. This isn't how things work with children, who are exposed to the full range of verbs from day one. Interestingly, the network seems to be unable to apply regularization to novel verbs. It seems like it is explicitly learning every present-past tense pair (e.g. walk-walked, get-got, take-took). The network seems incapable of generalizing.

In other words, the network is learning to memorize tokens, instead of learning types (such as regular and irregular verbs). I think what's going on with children is that they are learning specific instances of verbs early on. The dip in performance comes, not from learning an explicit rule, but from learning the higher-order concept of a regular verb. I think they tend to clump all verbs into that type, and that the eventual rise in performance is due to subsequently learning the higher-order concept of irregular verb, and being able to distinguish it from regular ones.

Another place types and tokens has cropped up is in this new BBS article, "Darwin’s mistake: Explaining the discontinuity between human and nonhuman minds". The authors argue that there is no solid evidence to suggest that non-human animals can make the type-token distinction. Instead, they say there is evidence that animals like chimps can form categories, but that those categories are based on similarities in the features of the instances. For example, chimps are probably able to represent a type like "banana" based on the similarity in features of all bananas (color, shape, etc.). However, it is unlikely that they are able to represent a type like "gift", whose tokens may display a wide range of observable features (e.g. a flower looks very different from a watch, but they may both be gifts).

It may very well be that as the owner of a very large neocortex, you are able to learn the past tense of verbs and represent abstract types like "things at a picnic" because your cortical hierarchy includes levels both larger and more numerous. This may allow you to have representations of abstract, higher-level concepts in a way that non-humans do not, simply because of hardware.

I think the key to a neural network model that is able to handle both regular and irregular verbs in a realistic way is hierarchy. If I weren't already going down a particular research path, this is one I'd definitely explore.