Evaluation
Formative and summative evaluation of the program proposed here is important as it will allow us to make judgements and decisions about the students, our effectiveness as teachers, and the effectiveness of the field experiment in meeting the pedagogical and ecological objectives (Harmon 1995). As suggested by TenBrink (1986) ,we will use several techniques for gathering this information. Leeming et al. (1993) have recently reviewed the problems associated with the evaluation of environmental education programs.
Pedagogical Objectives
Authentic assessment strategies will be developed and used to assess the pedagogical objectives. Students will be asked to interpret data sets drawn from the ecological literature, ask questions based on observations and establish hypotheses based on these observations, and establish a basic research protocol to test a hypothesis. The emphasis in this phase of the evaluation will be on student's ability to use the processes of ecological field research. Evaluation of the program will be based on a comparison of pre-test and post-test performance.
Ecological Objectives
Lisowksi & Disinger (1991) provide a Student Ecology Assessment (SEA) instrument for obtaining information about student's understandings of concepts related to ecology. Although developed principally for testing plant-animal feeding relationships, this model is readily adapted for the objectives of our study. The SEA consists of 40 items incorporated into eight concept clusters within which items progress from concrete to abstract, from familiar to unfamiliar, and from fact-based to higher-order questions. A student background (e.g., number of previous science courses taken in high school and college) and attitude survey (e.g., science self-efficacy) is administered along with the SEA to allow the effect of factors to be either removed from subsequent analysis (e.g., using Analysis of Covariance) or used as the basis of interpretation. The summative evaluation will consist of a non-equivalent groups pre-test-post-test design (Cook & Campbell 1979). Current GPA and pretest scores will be used to control pre-existing group differences on post-test results. The formative evaluation will consist of regular assessments of research participants over a two-year period as they progress through the environmental studies major.
The SAE will be administered on the following groups of students:
- All students taking PLB 301I Environmental Issues (long-term study participants) and ZOOL 312I Conservation of Natural Resources (long-term study non-participants). Students taking PLB 301I will be taken on weekend field trips to the field course (see section C.5 C 2, above), whereas students taking ZOOL 312I will not visit the field experiment or receive any instruction on the value of long-term field experiments. The students taking either course are drawn from the same student population. Both courses are one of the options in the Interdisciplinary section of the University Core Curriculum that all undergraduate majors are required to take. Students will not take both courses, unless they first take ZOOL 312I and then decide to become an Environmental Studies major in which case they will have to take PLB 301I. ZOOL 312I will thus form a control group in the evaluation. Since both courses will be offered in all fall and spring semesters it will be possible to track any changes in the effectiveness of the field experiment versus the control groups over the course of several years using a repeated measures analysis of variance.
- Environmental Studies majors. These students will start their program in PLB 301I, and progress to take PLB 307. Many will also take PLB 337, 440 and 451. In each of these courses, students will be exposed to the field experiment to a greater or lessor extent, and will take the SAE. Their changing facility in meeting the ecological and pedagogical objectives can be assessed as they individually progress through the program.
- Retention. Life science students who take PLB 200, but do not pursue ecological or environmental science tracks will be retested with the SEA (after first taking in PLB 200, see below) in their senior year. These delayed post-test, follow up tests will allow us to determine whether the observed effects from the long-term field experiment experience in PLB 200 persist beyond the semester of the experience.
- Pre-test - Post-test evaluations. All students participating in courses which utilize the long-term field experiment and students in ZOOL 312I will be given the SAE, background and attitude survey in the beginning of the semester before exposure to the field experiment intervention. The post-test SAE and attitude survey will be given following the completion of the field experiment (end of the semester), and again one year later.
The evaluation study, then, involves four groups of students:
- PLB 301I students who become environmental or ecology minors (hereafter, minors)
- PLB 301I students who do not become minors
- ZOOL 312I students (nonminors)
- PLB 200 students (non minors; but Life Science majors)
Technical Advisory Committee (TAC)
A committee of experts in science education, ecology and life sciences will be convened annually to provide an overview of the program in meeting the ecological and pedagogical objectives. This committee will review the results of administering the SAE, and review written and oral reports and depositions from the projects PI's. The following individuals have provisionally agreed to serve on the TAC:
- Dr. John Yopp: Dean, SIUC Graduate School, & Professor of Plant Biology
- Mr. Mark Cosgrove: Director, SIUC Touch of Nature
- Dr. Carolyn Donow: Research Project Specialist, SIUC Office of Research Development & Administration
- Dr. Diane Ebert-May: Director & Associate Professor, Biology, Science and Mathematics Learning Center, Northern Arizona University
- Dr. George Feldhamer: Associate Professor of Zoology, Director, SIUC Environmental Studies Program
Independent Assessment
Dr. Margaret Stockdale is an assistant professor in the Department of Psychology, College of Liberal Arts at SIUC. She holds a Ph.D. in Industrial and Organizational Psychology and has 5 years of post-doctoral experience in evaluation. She is the instructor for Psyc 569: Applied Research Consultants (ARC) which is a practicum for graduate students in applied experimental psychology. ARC provides evaluation consulting services to the University and to local, regional, and national organizations. ARC has conducted several needs assessments, organizational studies, and both summative and formative evaluations for a wide variety of issues. Recent projects include an international evaluation of a systems engineer certification program, an evaluation of a university faculty mentoring program and a needs assessment of health and social service needs in seven rural counties.
External Assessment
An annual evaluation by an NSF National Visiting Team (NVT) will provide an external evaluation of the program. The members of the NVT will be chosen by NSF from science educators nationally.