What is Evaluation Policy and Why Does it Matter?
Evaluation policy is the explicit and implicit principles or norms that an organization uses to guide evaluation decisions and actions (Trochim, 2009). Put a simpler way, it’s how organizations think about evaluation, and what criteria they put in place for how evaluation is done. It is often informed by an organization’s values as well as any relevant policies that might impact one’s practice.
For example, governmental organizations like NASA or the NSF must abide by federal policies and legislation. Organizations that fall outside of stringent federal policy oversight will likely rely more on their internal values and norms to develop their evaluation policy. Whatever the primary focus, evaluation policy will likely have “substantial implications for when and how evaluation is done, who it is for, and whether and how it is used” (Hart & Mark, 2022).
Evaluation policy can encompass a great number of things. Hart and Mark (2022) list the following areas it can influence: rules, procedures, approaches, resources, infrastructure, personnel, and direction for evaluation. Now these might not be the most familiar to folks who don’t practice evaluation on the daily, so let’s break each of those down.
Rules may refer to the framing of evaluation and why it is important to the organization. For example, does an organization commit to a set of criteria to guide evaluation development and use, such as relevance, effectiveness, and sustainability?
Procedures could mean the processes through which an organization designs and implements an evaluation or approves it for use. Are there specific timelines for evaluation that the organization uses?
Approaches can refer to how the evaluation is carried out. There are many different approaches to evaluation that can be used, including equitable evaluation, culturally responsive evaluation, utilization-focused evaluation, and participatory evaluation (to name just a few). It could also refer to the methodologies. Is an organization prioritizing mixed methods approaches, or solely quantitative?
Resources might refer to the financial resources an organization is willing to spend on evaluation, or how much staff time can be devoted to evaluation. Oftentimes, a budget of approximately 8-12% of program funds is suggested for a robust evaluation.
Infrastructure can refer to how evaluation is built into the organizational operations or capacity. Is the organization investing in data management and communication tools?
Personnel of course refers to the “who” behind the evaluation. Is an organization investing in full-time staff or bringing in consultants? Must these consultants have certain qualifications?
Direction for evaluation might mean how the organization thinks about evaluation overall. Is evaluation a priority and a part of the organization’s dedication to continuous learning and improvement?
Now that you’ve seen the wide-reaching influence that evaluation policy can have, let’s address the arguably more important piece - why should you care? While it might seem like a rather dry subject area with little actual influence on your work, evaluation policy is enormously important for informal STEM education (ISE) organizations. And not just in the sense of “you should develop your own policies” (and yes, maybe you should), but rather -
evaluation policy is a commitment to certain values. Values of the organization, values of the individuals with power in those organizations, values that communicate and uphold certain standards in not just the field of evaluation, but ISE as a whole.
Let’s talk about how these values can present themselves, using two super fancy words that your coworkers will be impressed to hear you say: axiology and epistemology.
Axiology is the study of values and the intrinsic worth of things. It can include ideas like the priorities and importance of beliefs. In the space of ISE evaluation, this might look like:
setting priorities for the outcomes you’d like to track (e.g., workforce development over social-emotional learning) or the evaluation questions to include in the evaluation plan.
Epistemology is the study of knowledge and beliefs. It can include ideas like what constitutes knowledge, what knowledge is valuable, and how it can be acquired. In ISE evaluation, this might look like:
valuing certain knowledge over others (e.g., “rigorous” quantitative research over Indigenous knowledge), or who is allowed to carry out an evaluation.
You might be wondering why I put the word “rigorous” in quotations in the last paragraph. Well, concepts like rigor, outcomes, and what constitutes the “value” of a program may vary drastically between stakeholders, and is culturally informed. Some have even argued that rigorous requirements (including evaluation) set by foundation staff serve as barriers that perpetuate inequity among nonprofits and the communities they serve (Villanueva, 2018).
So, it is clear that evaluation policy, which conveys values and impacts practice, has real and important implications for our field. The first step to understanding evaluation policy and to improving it in service of equity and better outcomes for all is to be able to identify where policy exists, both explicitly and implicitly, and to ensure those values are communicated clearly.
Since much of evaluation policy is implicit (meaning, it’s “known” within the organization but not written down or communicated outwardly), a great step at your own organization might be to get really clear about what your evaluation policies are, and what they convey about your organization’s values. Then, make them clear to others.
Consider producing something like the National Science Foundation’s Evaluation Policy document that outlines many of the areas we covered at the beginning. It’s a great step towards equity, and a great way to improve your own evaluation knowledge and practice.
References:
Hart, N.R., & Mark, M.M. (2022). Evaluation policy: An introduction. New Directions for Evaluation, 9-16. https://doi.org/10.1002/ev.20492
National Science Foundation. (September 2020). National Science Foundation Evaluation Policy, 1-4.
Trochim, W.M.K. (2009) Evaluation policy and evaluation practice. In W.M.K. Trocim, M.M. Mark, & L.J. Cooksy (Eds.), Evaluation policy and evaluation practice. New Directions for Evaluation, 123, 13-32.
Villanueva, E. (2018). Decolonizing wealth: Indigenous wisdom to heal divides and restore balance. Oakland: Berrett-Koehler Publishers, Inc.
If you enjoyed this post, follow along with Improved Insights by signing up for our monthly newsletter. Subscribers get first access to our blog posts, as well as Improved Insights updates and our 60-Second Suggestions. Join us!