Smell was primal, while thinking became important much later.
Luo and colleagues came to this conclusion from studying rare intact skulls of two small transitional animals, morganucodon and hadrocodium, which are thought to be part way between modern mammals and a reptilian ancestor called a cynodont.
The researchers used CT scanning to produce 3-D images of what these animals' brains might have resembled.
What they saw was that the brains of these two creatures were proportionally larger than those of their reptilian predecessors, but most of the growth was in the part that processes scents - the olfactory bulb, Luo said. There was also some expansion in the brain area used to sense touch, suggesting that this was another important adaptation for early mammals.
They published their work in Friday's issue of the journal Science.
"This is the first time we know the relative size of brains of intermediate forms," said neurobiologist Glenn Northcutt of the Scripps Institute of Oceanography in La Jolla, Calif., who was not part of the team.
"Just looking at the relative development of the various brain parts tells us what was increasing at that time.. . . These are brain missing links."
These early pre-mammals were tiny - morganucodon was the size of a small chipmunk, while hadrocodium was smaller than a grasshopper. Luo said these animals were probably covered in fur and were warm-blooded, but still laid eggs.
Both date to the early Jurassic period - when the world was stalked by dinosaurs. Fossils of early mammals and pre-mammals from this era are exceedingly rare, said the paper's lead author, Timothy Rowe of the University of Texas, Austin.
More than 99 percent of the known fossils have crushed skulls. "There are more Vermeer paintings than there are intact skulls of Mesozoic animals," he said, referring to the longer era that includes the Jurassic period.
Luo was able to obtain the fossils from China, where they were discovered in the late 1980s. Back then, however, the only way to use a skull to understand the brain was to make a cast, and then destroy the original fossil.
Texas' Rowe said as soon as these fossils were found he knew they held answers to the evolution of the mammalian brain. "But they were too precious to break open," he said.
He said scientists can make inferences about the brains from these scans thanks to the way the brains of mammals conform to the skull. "Your head is jam-packed with brains, and we know the same is true for morganucodon."
For comparison, they also had scans of the cynodont - which is considered ancestral to all modern mammals. "They were nasty, narrow-brained little bite-y creatures," Rowe said. "We have very humble beginnings."
From scanning the two early mammals that came later, the researchers saw that the brains had grown 50 percent bigger relative to body size. And most of that growth was in the smell center.
"The single major driving factor for that was the development of the olfactory system," said Rowe.
To survive in a world dominated by dinosaurs, these proto-mammals became largely nocturnal, the researchers say. "They had to navigate through an olfactory landscape rather than a visual one," said Rowe, smelling the presence of prey, predators, allies, and potential mates.
Scripps' Northcutt said there was one other branch of the animal kingdom that was getting brainier at the same time - the theropod dinosaurs, which are ancestral to birds.
In their case it was driven more by the need for good vision and hearing, he said.
Of course, humans are not a shining example of sniffing power compared with mice or dogs. We and other apes went a different direction, developing good hearing and vision. Thinking was probably important at some stage, but Luo believes that for most of the last 200-some-odd million years, brain evolution was driven more by the advantage of good senses.
"To have this mental capacity to think is a secondary consequence of that."
Go online to discuss evolution with Faye Flam at www.philly.com/evolution
Contact staff writer Faye Flam
at 215-854-4977 or firstname.lastname@example.org.