Surveying Change

(source)

Social historians studying the twentieth century have an advantage specialists in earlier centuries do not. Survey research, which began seriously in the 1930s, allows the former to know what average people reported about their attitudes and actions in ways that no documentary archive can even approximate (here, for example). To track changes over the decades in attitudes and action accurately, not only should the samples drawn in different eras be comparable, the questions asked should be the same – whether they are about church attendance, political participation, racial views, whatever. As the noted sociologist Otis Dudley Duncan reportedly stated, “If you want to measure change, don’t change the measure” – i.e., the wording of question.

Wise advice. But there is a problem: Sometimes the words themselves change meaning.

I was sharply reminded of this issue recently when leading a team that was putting together a survey. I had jotted down a phrase to use in a question: “in order to keep things straight.” Graduate students quickly objected. You can’t use straight because of its sexual connotations. I was well-aware that the word gay had been transformed. Tom W. Smith, a dean of survey research, noted that the Gallup Poll’s 1954 question, “Which American city do you think has the gayest night life?,” did not mean the same thing just 30 years later. Now, neither does straight.

Survey designers cannot fully rely on fixed meanings. Paradoxically, the pollsters’ craft requires judgments about social change in order to write the questions to measure social change. (For related discussions of how words’ histories can affect psychological testing, see this earlier post and here.)

Cases

An historically dramatic case of words changing their social meanings – and survey researchers’ efforts to keep up with those changes – is the term for the population we currently name African-American or Black. Pollsters have used Negro, colored, and Afro-American at various times, trying to keep up with the changing connotations of the labels. Or take the word partner as meaning a person in an established couple. It does not show up as such in Gallup Poll questions until the early 1980s and in the General Social Survey until 1987, even though living together was not a new – just a growing – phenomenon.

The word welfare has come to have a strongly negative connotation for many Americans, suggesting being on the dole, but it was not always so. The Gallup Poll used the term from 1939 through the 1940s either to ask about whether there should be a federal “Department of Public Welfare” or whether the Republican party was sufficiently concerned with the “welfare of the people.” (I take these items from the iPoll data bank.) In 1949, Gallup started asking whether America should move in the direction of a “welfare state” (respondents were about 2:1 opposed). Finally in 1961, even before expansion of Aid to Dependent Children or the Great Society, Gallup asked, “Do you think that the government in Washington is spending too much tax money for welfare programs to help the people in the United States or do you think it isn’t spending enough?” More respondents said not enough than said too much. However, by 1978, in answer to a similar question, Americans answering that welfare should be “cut back” well outnumbered those who said “expand” it. The realities of welfare had changed, but probably as importantly, what respondents understood by the term had also changed.

New contexts reshape respondents’ interpretations. Smith points out that the 1937 question “Do you object to movie scenes of women smoking?” probably measured respondents’ opinions about proper behavior for women, but that question today might instead measure respondents’ concerns about health. One heavily-discussed topic among experts is whether the old questions that were used to assess racial and ethnic prejudice no longer work as accurate measures because Americans have learned through the media that some answers — such as whether blacks are inherently inferior — are widely thought inappropriate. Scholars of “the new racism” argue that different sorts of questions, mainly about policies, or assessments of implicit altitudes are now more accurate. As a final example, about a quarter-century ago, if you asked people about their social networks, they were likely to tell you about their family and friends; now they are likely to discuss their Facebook lists.

Survey researchers would love to believe that our measuring instruments are reliable yardsticks for assessing social change, not unlike the weather stations climate scientists use to describe global warming. For the most part, survey questions hold up pretty well. But, alas, some of our yardsticks morph a bit even as we use them. To adjust, to alter wordings, we are forced to make assumptions about the very social changes that we are trying to measure.

 

[[ Scheduling Note: I have generally posted on this blog weekly for years. Because of other commitments, I’ll be going to alternate weeks at least until May.]]