Evaluation Policy Frameworks
Overhead shot of three coworkers drafting plans at a long conference table.
Welcome back to our Insights! We’ll be continuing today with our series examining Dr. Sarah Dunifon’s research on funding priorities and evaluation policies of informal STEM learning funding organizations.
The following content is based on or excerpted from Dr. Dunifon’s dissertation, An Examination of Evaluation Policies and Funding Priorities in Informal STEM Education. The research cited in this piece was conducted between April 2022 and March 2023. You can access the full dissertation here.
In an earlier piece, we explored evaluation policies and how they influence informal STEM education. Now, let’s dig deeper into how major players in the field are thinking about evaluation policies for themselves and their grantees. In this piece, we’ll explore evaluation policy frameworks from two notable organizations: the American Evaluation Association and the National Science Foundation.
Evaluation policy is an increasingly important consideration for large organizations and federal agencies. The American Evaluation Association, a national professional organization for evaluators, has created an Evaluation Policy Taskforce to assist the organization “in developing an ongoing capability to influence evaluation policies that are critically important to the practice of evaluation in the federal government.”
Evaluation policy points that the Taskforce informs include things like requirements of evaluation (e.g., “When are evaluations required? What programs or entities are required to have evaluations? How often are evaluations scheduled? What procedures are used to determine when or whether evaluation takes place?”), evaluation methods (e.g., “What approaches or methods of evaluation are recommended or required by legislation or regulation; for what types of programs or initiatives?”), and evaluation budgets (e.g., “What are the standards for budgeting for evaluation work?”).
Defining each of these details in advance of a project might feel like overkill to non-evaluators, but those of us in the field recognize them as crucial to setting up evaluation policies that accomplish an organization’s goals.
The National Science Foundation (NSF) introduced its own evaluation policies, informed by the Foundations for Evidence-Based Policymaking Act of 2018, which required federal agencies to draft and adopt evaluation policies to guide their work. NSF’s 2023 Evaluation Policies, for example, focused on six principles:
Relevance. “Evaluations supported by NSF must address questions of importance and serve the information needs of stakeholders. These evaluations should present findings that are clear, actionable, and timely to inform agency activities and actions such as program improvement, accountability, management, and policy development.”
High Quality and Rigor. “Evaluations must produce credible evidence—findings that NSF and its stakeholders can confidently rely upon. To produce such evidence while balancing constraints like time and resources, evaluations must rely on the most appropriate designs and methods to answer key questions; provide clear explanations of limitations; ensure appropriate execution of the study design with quality controls and regular risk monitoring; and promote proper interpretation of findings.”
Independence. “NSF is committed to supporting evaluations that are objective and unbiased. This is essential to produce credible findings that stakeholders may rely upon in making decisions. Evaluators shall strive for objectivity and avoid bias in planning, conducting, interpreting, and disseminating findings from the studies in which they participate. The implementation of evaluation activities, including how evaluators are selected and operate, shall be insulated from undue influences that may affect their objectivity, impartiality, and professional judgment.”
Transparency. “NSF promotes transparency in the planning, implementation, and reporting phases of evaluation activities to promote dialogue that enhances quality, enables accountability, and prevents tailoring that influences findings. Transparency is crucial to support reproducibility and contribute to advancing knowledge. Whenever possible, completed evaluations will be released in a timely manner and with sufficient detail to support use of findings (including comparability to the existing literature) and replication.”
Ethics. “Evaluations supported by NSF will be conducted with the highest ethical standards to protect the public and maintain public trust in the agency’s evidence-building efforts. Evaluations shall be planned and implemented to safeguard the dignity, rights, safety, and privacy of participants and other stakeholders and affected entities, abiding by regulations governing research involving human subjects. Evaluations will be equitable, fair, just, and consider contextual factors that could influence the findings, their interpretation, or their use.”
Equity. “NSF is committed to supporting evaluations that contribute to the common good and advance an equitable and just society. To this end, evaluators must seek to include the perspectives of diverse populations, particularly from populations that are underrepresented and/or underserved. Relevant data collection efforts should be rigorous, inclusive, and culturally appropriate. Appropriate methods should be used to produce credible findings applicable to broad and specific populations, whenever possible.”
While there has been work to establish these frameworks for developing evaluation policies, less has been conducted on assessing evaluation policies in the informal STEM education space specifically. Informal STEM learning organizations can take inspiration from these frameworks when considering their own processes and evaluation policies.
References:
Dunifon, S. M. (2024). An examination of evaluation policies and funding priorities in informal STEM education funding organizations (Doctoral dissertation). University of Pittsburgh.
Evaluation policy (April 2023). (n.d.). NSF - National Science Foundation. https://new.nsf.gov/od/oia/eac/evaluation-policy-april-2023
If you enjoyed this post, follow along with Improved Insights by signing up for our monthly newsletter. Subscribers get first access to our blog posts, as well as Improved Insights updates and our 60-Second Suggestions. Join us!