The Challenges of Evaluation

Measuring the effectiveness of any communications effort raises serious challenges that should be acknowledged up front. Evaluation of nonprofit communications is still a relatively new field, and emerging evaluation techniques are still grappling with how to deal with the following types of challenges:

  • To date, standard and widely accepted guidelines for communications evaluation have not existed in either the for-profit or nonprofit worlds.
  • Nonprofit organizations and the campaigns they implement are often unique, making the creation and adoption of standard evaluating guidelines difficult.
  • It can be difficult to disaggregate the impact of communications efforts and their value added from that of other social change strategies being implemented at the same time, like grassroots organizing.
  • Public communication campaigns often aim for complex and hard-to-achieve change (for example, changing public will by affecting norms, expectations and public support, or changing behavior through skill teaching, positive reinforcement and rewards). Campaigns can also often aim for change at multiple levels of society (community, state, national or international).
  • Some methods useful to communications evaluation are too costly for many nonprofits (for example, polling) or may require staff time or expertise that is not readily available.
  • Communicators and evaluators don’t always speak the same language. Most evaluators don’t understand communications theory and practice and communications people don’t understand evaluation language or methods. One result is that the evaluation’s focus can sometimes be misguided. For example, the evaluation may focus only on “placement” of stories in the media as the primary measure of a campaign’s success, ignoring the importance of informing supporters and allies though internal communications efforts such as newsletters, e-mails, briefing calls and meetings.
  • Often, the goal of nonprofit campaigns is to ensure that an organization’s efforts to define a social problem and its proposed solution capture the awareness of those who hold the power to allocate resources and choose appropriate policy alternatives. This is a high standard for success, with implications for evaluation design and data interpretation.
  • Sometimes communications resources dedicated to achieving an impact are too limited to be effective. Also, sometimes a campaign is not ready to be evaluated.
  • Some campaigns seek incremental change. They are implemented in stages, and initial stages may be modest in impact.