Posts Tagged ‘methods’

Surveying Change

Social historians studying the twentieth century have an advantage specialists in earlier centuries do not. Survey research, which began seriously in the 1930s, allows the former to know what average people reported about their attitudes and actions in ways that no documentary archive can even approximate (here, for example). To track changes over the decades in attitudes and action accurately, not only should the samples drawn in different eras be comparable, the questions asked should be the same – whether they are about church attendance, political participation, racial views, whatever. As the noted sociologist Otis Dudley Duncan reportedly stated, “If you want to measure change, don’t change the measure” – i.e., the wording of question.

Wise advice. But there is a problem: Sometimes the words themselves change meaning.

I was sharply reminded of this issue recently when leading a team that was putting together a survey. I had jotted down a phrase to use in a question: “in order to keep things straight.” Graduate students quickly objected. You can’t use straight because of its sexual connotations. I was well-aware that the word gay had been transformed. Tom W. Smith, a dean of survey research, noted that the Gallup Poll’s 1954 question, “Which American city do you think has the gayest night life?,” did not mean the same thing just 30 years later. Now, neither does straight.

Survey designers cannot fully rely on fixed meanings. Paradoxically, the pollsters’ craft requires judgments about social change in order to write the questions to measure social change. (For related discussions of how words’ histories can affect psychological testing, see this earlier post and here.)


Read Full Post »

Word Counts and What Counts

A post back in June on “digital humanities” discussed the promises and perils of turning to “Big Data” to answer questions about American history. I focused there on a study that looked specifically at the history of American literature. A paper in Psychological Science this August uses the same tool – the Ngram function in Google that counts a word in the company’s sample of over 1 million books ever published in U.S. and calculates the percentage of all words it represents – to make broad claims about historical changes in American character.

Patricia Greenfield, an eminent UCLA psychologist who has conducted terrific research on cognitive development, changes in cognitive skills, and cultural differences in thinking, much of it based on her work in rural Mexico (mentioned in this 2012 post), uses Ngram to argue that there was a major shift in America over two centuries from a communal to a self-centered culture. Ngram word counts in American books from 1800 to 2000 show, she claims, that Americans changed from being group-oriented and sharing to being individualistic and self-absorbed. Maybe. But there are a lot of issues to consider before accepting the claim. These concerns show, once again, the pitfalls in using such statistical methods ahistorically.


Read Full Post »

The Survey Crisis

At this time in the presidential election cycle, we are inundated by surveys, almost moment-by-moment, battleground state by battleground state. But surveys are far more important than just serving to handicap elections. It is through scientific surveys – that is, asking standardized questions of representative samples of the population – that researchers and policymakers in the last 70 or so years have been able to get an accurate sense of Americans’ lives and opinions. It is how we know, for example, who is having trouble making ends meet, or people’s views about big policy issues.

(There are many survey cynics out there, I know. But, well-done surveys are roughly accurate – and roughly accurate is better than other ways of figuring out what is going on, like extrapolating from one’s friends’ experiences and opinions.)

Telephone Interviewers (source)

For the last several years, however, survey researchers have faced escalating challenges, in particular, the problem of getting Americans to answer their phone calls. (See this excellent National Journal article.) High-quality, face-to-face surveys are still done, but the costs have shot through the roof. Only the government and well-funded academic projects can afford them. The more common way of doing surveys – say, the way Gallup or Pew does them – is by telephone. And that has become difficult. I know one small survey business that closed from general dispirit.


Read Full Post »

Depressing Comparisons

An August post on a sociology blog began, “For the last several decades, depression rates have been on the rise at a rapid pace.”

Source: Andrew Mason via flickr

That assertion has appeared in many places over recent years. The blogger provided no reference for the assertion. I think I know the initial source of the claim; most writers who declare that depression has been rising probably read it in – of course – The New York Times.

Research indicates, however, that there was no rise in depression rates over the last several decades. The key studies relied on for the claim that depression rose had one or more important  flaws. Understanding those flaws helps us understand the difficulties of discovering and making historical claims.


Read Full Post »