Government Using BI Software To Measure Public Diplomacy

President-elect Barack Obama has said he wants to cut the fat out of government and at a Nov. 25 press conference indicated that the government's own spending watchdog, the Office of Management and Budget, will be "more efficient and more effective at serving the American people."

That likely means government agencies will need to provide the OMB with even more detailed answers on where their dollars go.

A State Department division that runs public diplomacy programs overseas could prove to be a model to its peers with its use of business intelligence software, popular with the private sector, to demonstrate the return on investment of its expenditures. Its latest project is a pilot program to develop algorithms that better show correlations between the department's goals and its expenditures, using SAP Business Objects XI business intelligence platform and planning applications.

The State Department last year spent $357 million on diplomacy programs designed to create a positive image of the United States in other parts of the world. These include summer camp programs for kids in the Middle East, the American Corners information libraries at various U.S. embassies, and speaking engagements by American celebrities.

But in 2006, the OMB gave the State Department a poor rating on its ability to measure the effectiveness of those diplomacy programs.

"The OMB has been very clear that performance measurement is something they're placing an emphasis on," said Cherreka Montgomery, acting director of the evaluation and measurement unit in the Office for Policy, Planning, and Resources for the undersecretary of state for public diplomacy, in an interview Tuesday.

The division's first step was to reduce its number of performance measures from an unmanageable 898 down to 15, and to develop six outcome measurements. One of those six is "Initiation of positive change to local communities"; another is "Reduced anti-Americanism."

In 2007, staffers visited the State Department's outposts in Japan, Israel, Germany, Nigeria, Ecuador, Palestinian territories, and India. It surveyed a sample of 1,800 foreign audience members, some of whom had participated in a diplomacy program and some who had not, for a comparison study to measure attitudes about the United States. Those results were then used to develop focus groups to get more qualitative, detailed data.

All of this resulted in a difficult-to-digest 300-page report. So the division created a Public Diplomacy Impact dashboard accessible on the State Department's intranet, based on Business Objects Xcelsius data-visualization software. The dashboard provides State Department executives with budget details, plus how far it's come in achieving its six outcome measures based on survey data.

For example, 64% of survey respondents with influential community roles -- such as youth, community and religious leaders, academics, journalists, bloggers, and even cartoonists -- said they were taking concrete measures to initiate positive change in their communities following their involvement in diplomacy efforts.

The State Department's now back in good standing with the OBM on its public diplomacy measurements, but Montgomery hopes for more -- she wants to use BI efforts to demonstrate the need for additional resources.

"The dashboard shows that while public diplomacy is having a measurable and tangible impact among foreign audience members, we can further our impact if we have more resources to implement public diplomacy overseas," she said. The ability to "change the hearts and minds" of any country that leans toward anti-Americanism, she added, can happen only if "we can engage repeatedly over time." The division did a similar survey this year and expects to present its findings to the OMB within a few months.

Now the public diplomacy division has started a pilot program to get more detailed expenditure information from its various outposts around the world. Its goal is to demonstrate, for example, why youth summer camps in the Middle East require more funding than some in other countries because of higher transportation, staffing, and translation resources costs.

Montgomery said the pilot program is a delicate balance, as the recently established performance and measurement unit she runs tries to collect such data as transportation, advertising, equipment, and utilities costs, and details on staffing numbers and hours, without overburdening staffers outside the United States with data collection.

Technology is helping with the effort: Staffers are inputting the budget data into a Microsoft Excel interface in Business Objects planning application. Montgomery's unit will then develop the algorithms to show relationships between money spent and the division's six outcome measures.

The results of that work will later appear under a tab on the Public Diplomacy Impact dashboard call What-If Analysis, or what the State Department could do if it received even more funding for diplomacy efforts.

The OMB needs to recognize that BI efforts cost money, too, Montgomery said. "We're doing our best, but to do a true-blue performance budget effort requires massive amounts of resources."

If the Obama administration is serious about cutting costs from federal programs, however, the State Department's public diplomacy division may find itself in a long handout line. Surely there are many agencies that believe they can better serve the United States and its citizens, if only they had more money to do it. But the public diplomacy division is on the right track: Those responsible for cutting the checks will want to see sharp analysis and presentation of where the money goes.