About the Author

 

People, including KQED's Michael Krasny who has interviewed everyone, ask me where I get the chutzpah. My best answer is that I was the first grandchild in the extended family and apparently was always being told how smart and special I was. The neural connections that constitute this belief have not been totally suppressed by the years of experience to the contrary. This is hard on people who need to deal with me, but it does seem prerequisite for anyone who would attempt to unify cognitive science.

 

I don't know when I first became intrigued by computers, the brain, and the mind, but it was well before my graduation from Rochester in 1960. I had a student job with a professor of physiology and vividly remember him arguing that the true path was through neuroscience.  But his research was testing the damage to the isolated crayfish giant axon caused by different UV frequencies, funded by the Atomic Energy Commission.  Another defining undergraduate moment was when, in my physics major, we reached areas of physics where intuition was (and is) not available. My memory is of deciding that, if it was all math anyway, why not just switch to a math major.

 

After one exploratory year in industry, I joined the Math department of (then) Carnegie Tech. To my good fortune, the department head was Alan Perlis, one of the pioneers of the field that would become Computer Science. He said roughly: "you are bright and ambitious but not smart enough to have a big impact on mathematics, but there is this new field where nothing is yet known." I did a thesis with him on the semantics of programming languages, but was more drawn to the AI effort at Carnegie led by Allen Newell and Herb Simon.

 

The rest of the story is outlined in the preface of the book. Cognitive Science literally involves the efforts of tens of thousands of scientists. The NTL group's work over the years has been primarily the product of the wonderful students that we are privileged to see at Berkeley.

 

Preface to "From Molecule to Metaphor"

 

 

Many years ago, I was browsing through books on learning how to draw. One of them said, after a brief introduction, put down this book and start drawing. This book is like that - it will frequently suggest a simple mental exercise to help you personally experience a phenomenon. If this appeals to you, you might like the book.

 

By now, virtually everyone agrees that the scientific explanation for human language and cognition will be based on our bodies, brains and experiences. The major exception is Noam Chomsky's, whose dominance of 20th Century linguistics is unparalleled in any other academic field. I will later quote from Chomsky's 1993 book, Language and Thought , and the same idea was stated repeatedly in his 2003 Berkeley lectures: We don't know nearly enough about the brain for cognitive science to take it seriously. Chomsky has focused on linguistic form; since this book deals first with meaning, we won't encounter him again until Chapter 22.

 

As a first mental exercise, try expressing to yourself what you know about how your own thoughts work. How do our brains compute our minds? When I ask Berkeley students, on the first day of class, to write a page on this question, most of the students express mystification. Even among people who know a great deal about neuroscience, psychology, linguistics, philosophy, and artificial intelligence there is often no clear idea of how the findings of these fields could combine to yield even a preliminary understanding of how language is embodied in us.

 

The purpose of this book is to propose the skeleton of a theory that integrates current insights from many disciplines into a coherent neural theory of language. It might seem that no such effort is needed - isn't language obviously a function of our brains - what else could it be? Certainly other human abilities such as motor control, hearing, and especially vision have been studied as neural systems for many decades. But language is still often treated as an abstract symbol system not particularly tied to human brains or experience.

 

A great deal of permanent value has been learned by formal studies of language, but it is surprising that the disembodied notion of language persists. This is partly historical, but also arises from the fact that other animals share our abilities in vision, etc. but not in language. Much of the progress in neural theories of vision, motor control, etc. have come from invasive animal experiments that are thankfully prohibited on people. Until recently, there has been very little known about how language is processed by our brains.

 

It is still true that no one currently knows the details of how words or sentences are processed in the brain and there is no known methodology for finding out.  Many scientists believe that it is premature (perhaps by centuries) to formulate explicit theories linking language to neural computation. Even theoreticians are usually content with suggestive models, which can't actually be right, but do suggest interesting experiments. However, the cognitive sciences are revealing a great deal about how our brains produce language and thought. And there is a long and productive tradition, going back at least to the Greek atomic theories of matter, of postulating "bridging theories" in advance of the detailed evidence. Brian Greene's, The Elegant Universe, is a wonderful description of the science of the fundamental nature of matter, where there might never be experimental verification.

 

In contemporary science, it is not unusual to have quite extensive knowledge at two ends of a causal chain and to build and test theories that try to explicate the bridging links.  For example, astrophysics is concerned with linking fundamental particle physics with astronomy. In economics and other social sciences, a principle concern is how individual preferences give rise to group behavior. Similarly, much of molecular biology is concerned with how genetic material yields the various proteins and the resultant organisms. Higher levels of biology are also trying to develop bridging theories. The search for a neural theory of language can be seen as one of these attempts, albeit unusually ambitious. These bridging theories are often developed as computer simulations and the book will follow this tradition.

 

I treat the question of mind as a biological one - language and thought are adaptations that extend abilities that we share with other animals. For well over a century, this has been the standard scientific approach to other mental capacities such as vision and motor control. But language and thought, even now, are usually studied as abstract formal systems that just happen to be implemented in our brains. Instead, we will pursue the great ethologist Nico Tinbergen's (Tinbergen  1963) four questions that must be asked of any biological ability:

 

1. How does it work?
2. How does it improve fitness?
3. How does it develop and adapt?
4. How did it evolve?

The first three of these questions will be covered in considerable detail. The origins of language are still largely unknown and will be discussed briefly in Chapter 26.

 

There is a sufficiently large gap between brain and language to contain ecological niches for many theories, especially if their proponents are satisfied to ignore inconvenient findings. Understanding language and thought requires combining findings from biology, computer science, linguistics, and psychology.  A theory that seems perfectly adequate from one perspective may contradict what is known in another field. Problems that seem intractable in one discipline might be quite approachable from a different direction. Taking all the constraints seriously is the only way to get it right.

 

But this requires us to understand the essential ideas from several quite different scientific domains. In any of these fields, keeping up with technical advances and doing original work is extremely demanding and requires focused effort. There are some endeavors at the boundaries between subfields, but very little scientific work that attempts to encompass the full range needed for our task. I will need to synthesize a bridging theory from separate fields, all of which have their focus elsewhere. My approach is to pick out key findings and theories from various disciplines and show how, in combination, they constrain the possible bridging theories of language to a narrow family of possibilities.

 

Each discussion is an over-simplification of some research field, often involving thousands of active investigators, and thus is inherently incomplete. There are the usual references suggesting more detailed discussions of various points, but these will be most useful as key words for search engines. By the time you read this book, there will be important new developments in each of these areas. A list of books for further reading is included for people who would like additional background in one or another direction.

 

While we are far from having a complete neural theory of language, there have been enormous scientific advances in all the relevant fields. Taken together, these developments provide a framework in which everything that we know fits together nicely. The goal of this book is simple; I would like you, at the end, to say: This all makes sense. It could explain how people understand language.  There will be no attempt to convince you that other theories are wrong - in fact, I will assume that most of them are partially right. The book can be seen as part of a general effort to construct a Unified Cognitive Science that can lead the effort to understand our brains and minds. I will try to present a story that is consistent with all the existing scientific data and that also seems plausible to you as a description of your own mind.

 

Except for one thing. There is one part of our mental life that is still scientifically inexplicable - subjective experience. Why do we experience everything in the way that we do? The pleasure of beauty, the pain of disappointment, and even the awareness of being alive ... these do not feel to us like they are reducible to neural firings and chemical reactions. Almost everyone believes that his or her own personal experience has a quality that goes beyond what this book, and science in general, can describe.  If I had anything technical to say about subjective experience, it would be the highlight of the book, to say the least.

 

People use terms like personal experience, subjective experience, and phenomenology to label this idea. Philosophers have coined a technical term, qualia, to refer to these phenomena that are currently beyond scientific explanation. Antonio Damasio (Damasio 2003), who in my opinion is doing the best scientific work on subjective experience, distinguishes measurable emotions from subjective feelings. Aside from a brief discussion in Chapter 26, this book will focus on what can be learned from studying the physiological and behavioral correlates of experience - i.e., what can be measured and modeled objectively.

 

My undertaking of this quixotic enterprise came as the result of a year of explicit soul-searching around the time of my fortieth birthday. I had the good luck of entering the field of Computer Science in its infancy and believed that this gave me the opportunity to move in almost any direction, exploiting insights into information processing not available to previous generations. Having long-term interests in language and the brain and having worked on various computer systems including some of the earliest robots, I was led to focus on the question that I just asked you - How does the brain compute the mind? Twenty-five years later, due to advances in all fields that were inconceivable to me at the time, the outlines of an answer seem to be emerging.

 

A Brief Guide to the Book

 

The book is designed to be read in order; each chapter provides some of the underpinnings for later ideas. But it should also be possible to look first at the parts that interest you most and then decide how much effort you wish to exert. There are many forward and backward pointers that may help integrate the material.

 

Information processing is the organizing theme of the book. Language and thought are inherently about how information is acquired, used, and transmitted. Chapter 1 lays out some of the richness of language and its relation to experience. The central mechanism in my approach to the Neural Language Problem is neural computation. Chapters 2-3 provide a general introduction to neural computation. Chapters 4-6 provide the minimal biological background on neurons, neural circuits and how they develop. We focus on those properties of molecules, cells, and brain circuits that determine the character of our thinking and language.

 

Chapters 7 and 8 consider thought from the external perspective and look at the brain/mind as a behaving system. With all of this background, Chapter 9 introduces the technical tools that will be used to model how various components of language and thought are realized in the brain. A fair amount of mechanism is required for my approach, which involves building computational models that actually exhibit the required behavior while remaining consistent with the findings from all disciplines. I refer to such systems as adequate computational models and believe that such models are the only hope for scientifically linking brain and behavior. There is no guarantee that an adequate model is correct, but any correct model must be adequate in the sense defined above.

 

The specific demonstrations begin with a study of how children learn their first words. This involves some general review (Chapter 10) and a more thorough study of conceptual structure (Chapter 11) that is needed for word learning. The first detailed model is presented in Chapter 12, which describes Terry Regier's program that learns words for spatial relation concepts across languages. This theme of concrete word learning is then extended to cover words for simple actions in Chapters 13 and 14, which describes David Bailey's demonstration system.

 

The next section extends the discussion to words for abstract and metaphorical concepts. In Chapters 15 and 16, we look further at the structure of conceptual systems and how they arise through metaphorical mappings from direct experience. Chapter 17 takes the informal idea of understanding as imaginative simulation and shows how it can be made the basis for a concrete theory. This theory is shown in Chapter 18 to be sufficiently rich to describe linguistic aspect - the shape of events. This is enough to capture the direct effects of hearing a sentence, but for the indirect consequences, we need one more computational abstraction of neural activity - belief networks, described in Chapter 19. All of these ideas are brought together in Srinivas Narayanan's program for understanding news stories, discussed in Chapter 20.

 

Chapters 21 - 25 are about language form  i.e., grammar - how grammar is learned and how grammatical processing works. Chapter 21 lays out the basic facts about the form of language that any theory must explain. Chapter 22 is partly a digression; it discusses the hot-button issues surrounding how much of human grammar is innate. We see that classical questions become much different in an explicitly embodied neural theory of language and that such theories can be expressed in standard formalisms (Chapter 23).

 

Chapter 24 shows how the formalized version of neural grammar can be used scientifically and to build software systems for understanding natural language. The poster child for the entire theory is Nancy Chang's program (Chapter 25) that models how children learn their early grammar - as explicit mappings (constructions) relating linguistic form to meaning. Chapter 26 discusses two questions that are not currently answerable: the evolution of language and the nature of subjective experience. Finally, Chapter 27 summarizes the book, and suggests that further progress will require a broadly based Unified Cognitive Science. But the scientific progress to date does support a range of practical and intellectual applications and should allow us to understand ourselves a bit better.

 

A version of the material in this book has been taught to hundreds of undergraduate students at UC Berkeley over the years. There were weekly assignments and most of the students actually did them. The course did not work for all the students, but a significant number of them came out of the class with the basic insights of a neural theory of language. If you want to understand how our brains create thought and language, there is a fair chance that the book can help.