Methodologies | Public Diplomacy

Ask the Survey Doctor: Evaluating Public Diplomacy

By Joshua Marshall, US State Dept | Insights | Series II | No. 5 | July 2014

Q: I know that polls, focus groups, and other opinion research techniques are often used to evaluate foreign aid programs. But what about that other tool of “soft power,” diplomacy? Can survey research be used to evaluate diplomacy?

A: Evaluating high-level diplomacy – negotiations between governments, carried out deep in the corridors of foreign ministries – is a task for diplomatic historians; what survey research can evaluate is what the international affairs world calls public diplomacy: the programs and communications one country uses to reach the people of another, rather than their government.

Examples include educational exchanges and international broadcasting (e.g. Voice of America); small-scale engagements, such as “open houses” hosted by an embassy; and large-scale information campaigns about a country’s history, culture, or values. All of these are public diplomacy because their goal is to shape how foreign populations view the sponsoring country.

That, in a sentence, is what public diplomacy does: it shapes people’s views. And that is why it can be evaluated by survey research, the science of observing what people think and why.

In fact, several countries are already using survey research to evaluate their public diplomacy.

For example, for its bi-yearly “Public Diplomacy Impact” study, the U.S. State Department surveys two groups of foreign individuals: individuals who have participated in/been exposed to U.S.-sponsored public diplomacy and a matching sample of individuals who have not – a “test group” and “control group.” State then compares the views of these two groups along metrics such as “understanding of the U.S.” Differences in the two groups’ views are considered the impact of U.S. public diplomacy.

Those differences – for example, an X% difference on metric Y – are data points against which the U.S. can assess the effectiveness of its public diplomacy, both now and in the future. That is, they are benchmarks, used just like benchmarks are in the private sector. Indeed, the PD Impact study is a benchmarking study – one of several that are conducted by various countries in order to evaluate their public diplomacy, usually through the same test group-control group methodology.

Most benchmarking studies evaluate the aggregate effect of a country’s public diplomacy: the impact of all of a country’s public diplomacy programs and communications taken together. Evaluating one particular program or communication calls for another type of study, a program evaluation.

Program evaluations done for public diplomacy are very similar to program evaluations conducted by aid agencies and NGOs. Their focus is one particular program or communications campaign which they examine most often through feedback surveys, focus groups, and one-on-one interviews.

Basic data analytics can also be used to evaluate public diplomacy. For example, many countries collect demographic information on their exchange participants, broadcast listeners, embassy visitors, etc. Performing a simple segmentation/CHAID analysis on that data can determine whether the demographics the country is targeting have participated/listened/visited in hoped-for numbers – that is, whether the country’s public diplomacy is engaging the “right” audiences.

Admittedly, data analytics are not quite the same thing as opinion research. But I mention data analytics because they and opinion research are very closely related: both disciplines use the same techniques (such as segmentation/CHAID analysis) to find meaning in data.

And finding meaning in the data is what makes an evaluation valuable – whether it is a benchmarking study, a program evaluation, data analytics, or any other type of evaluation. A book of raw crosstabs has little utility for the diplomats who actually do public diplomacy – they are experts in public diplomacy, not in data interpretation. But a smart analysis of those crosstabs, one that tells the story in the numbers, can inform diplomatic decision-making.