Evaluating an Assessment Instrument for the Oregon Environmental Literacy Plan


Susan Mae Duncan

Click here for a full PDF file.


This mixed methods study evaluated the validity, and reliability of an instrument

designed to assess a middle school student’s proficiency in systems thinking as described

in the 2010 Oregon Environmental Literacy Plan. In Stage 1, a forum of middle school

students, formal, and non-formal educators used the Delphi technique to reach consensus

regarding which skills were important to include in a scoring guide for systems thinking.

In Stage 2, the scoring guide was field tested by formal and non-formal educators using a

sample of students’ work. The two groups’ scores were compared using Cohen’s kappa to

make inferences regarding inter-rater reliability. Concurrently, an autoethnographic

narrative was written to explore issues of equity related to the assessment of

environmental literacy.

The commonalities between formal and non-formal educators revealed a high

level of validity for the construct of proficiency with systems thinking, and a moderate

level of reliability between the scores assigned by two groups of educators. In the words

of the middle school students, formal, and non-formal educators, who volunteered to

create the scoring guide, the ability to make responsible decisions with natural systems,

community, and the future in mind involves: creating solutions for systems that are not in

balance, presenting the complex inner workings of a system in a simple and succinct way,

collaborating, exploring multiple solutions, and sharing ideas in a way that people will

understand you.