BGSU :: Offices :: Student Achievement Assessment Committee

(We propose the following outcomes as a refinement of #2 above. We plan to formally adopt these in Spring 2007)

Assessment Methods and Procedures:
The department conducts exit surveys of graduating senior majors each semester. In addition, the department assesses the achievement of students in two courses. POLS 400, the Capstone Seminar in Political Science, is populated mainly with senior POLS majors. POLS 290, Intro to Political Inquiry, taught in both Fall and Spring semester, has 30-35 majors, usually in their sophomore or junior year. POLS 290, a course required for majors, teaches research methods and statistics.
Our goal is to compare the level of student performance in POLS 290, taken in the middle of a student's college career, to the level achieved in POLS 400, taken near graduation. We hope to see strong performance in both courses, with an improved level in POLS 400.
The capstone seminar has been adapted based on previous assessment of the course. It provides students with a review of research methods in the discipline, and focuses on a broad topic of interest across 2 or more subfields in the discipline. In Spring 2006 the seminar was taught by Dr. Jeffrey Peake as a topics course under the title "Capstone Seminar: Democracy and Leadership". Enrollment is limited to 25 students or fewer. The course required students to complete a significant original research project. This project is the basis of the assessment.
In POLS 290, students complete an extensive (10 page) research design. While they do not carry out the research, they write an extensive literature review on their topic and must demonstrate a command of the research process. Thus, with the exception of oral communication, our learning outcomes can be assessed using this project.
In the capstone seminar, students wrote a 15-20 page research paper, and described their results in a 15 minute oral presentation to the class and visiting faculty members who were able to evaluate the oral presentation for assessment purposes.

The rubric for assessment of the Department's learning outcomes was applied to the project from both courses. We developed this rubric for assessing outcome #2 by using our two new proposed learning outcomes (3 and 4).
2. Apply critical thinking abilities and disciplinary theories/frameworks to solve problems.

Assessment Rubric: Rate each paper as high, medium, or low performance in the following categories:

Please note that the students in POLS 290 are not the same students as those in the Capstone seminar. At this point we are not able to evaluate learning outcomes for each student over time, rather we are taking a cross-section of majors at each level and comparing their performance.
Our results, based on the oral and written presentations, are as follows:

Inferences from Assessments:
A. Data
POLS 400 Capstone: all 11 student papers read. All papers evaluated the nature and quality of the leadership of an important President or US leader. Because this did not involve statistics, we were unable to rate in category 4b.
POLS 290: of the 30 students in each semester, a random sample of 10 student papers were rated, for a total of 20. The research design papers did not require student to use their own data to evaluate their question, so few were rated high on 4b.
Table 1. Results (number of POLS 290students, number of Capstone students; then %290, % Capstone)

Data from the graduating senior surveys from this year again showed nearly equal levels of performance (as perceived by the students) in the areas of writing, critical thinking, and speaking skills. All were under 2.0 on a 1-5 scale, (1=strong improvement, 5--no improvement). We are seeing a declining number of students return this survey-only about 15% of those surveyed. As these surveys measure student satisfaction and perception of learning rather than actual learning outcomes, we are inclined to revise or discontinue them.
B. Inferences
The department believes that our increased emphasis on interactive teaching and the opportunities for speaking in case studies, role playing simulation, and presentations is having a positive effect on oral skills, which had been weaker during earlier years of this assessment. Though it was not the focus of assessment this year, students gave strong oral presentations in the capstone class, and the papers in both 290 and the capstone showed solid writing skills. What is disturbing in this year's assessment is the level of performance in the critical thinking and research categories as presented above.
The data in table 1 above show little or no improvement over time between the POLS 290 cohort and the capstone cohort, except in the area of theory(3a). In qualitative research design (4a), the capstone students perform more poorly than those in 290. Furthermore, the percentages of students rated "low" in both courses needs to be reduced. The area most in need of improvement relate to the connections of theories and concepts to hypotheses and data. We need to improve their ability to state and test research questions, and to show a more sophisticated understanding of the use of data in the construction of their argument.

Actions taken
POLS 400, our capstone seminar, is not yet required of majors. We have redesigned this course based on assessment results from last year as well as student comments on course evaluations and faculty input. The main project for the course—a major research paper and oral presentation—remained the same, but the subject matter has been shifted. In the past the course attempted to cover the entire political science discipline, using guest lectures from many faculty members. Now we are focusing on one or two broad, major topics in the discipline that cut across subdisciplinary boundaries so that they will interest nearly all students. Dr. Peake taught the course as "Capstone: Democracy and Leadership" in the spring 2006. Based on student evaluations and assessment of the research projects, this course design appears to be a significant improvement. Students were more motivated to do readings, discussions in class were improved, and research projects were more focused. We will teach this again under an international relations/comparative government topic (Seminar in War and Peace) in Fall 2006.
We will spend more time helping students to review research design in the Capstone class in an effort to improve the quality of the research projects. We will also explore ways in which the connections between theory, hypotheses, and empirical data can be strengthened in other courses in our curriculum.
Ongoing Actions:
The department has continued initiatives that provide our students more opportunities for interactive learning and oral communication in their courses. In the Fall 2004 semester, we added 30-student breakout discussion sections to our largest POLS 110 Intro to American Government course, which had 150 or more students. These discussion sections are led by our strongest graduate students (both in the MPA program, and under an exchange agreement, from American Culture Studies). We have had positive student evaluations in these sections for 4 semesters. However, we need to do more detailed assessment to determine whether the student learning outcomes for POLS 110 lecture/discussion section courses are significantly better than those that are lecture only. Based on student evaluations and the perceptions of faculty members who teach the courses, there is some concern that the differences may be small.
Our department continues to offer 5-6 small BGX sections each fall, as well 1-2 small honors sections per semester when possible. The department faculty have noted a concern that student performance (based on grades and professor opinion) in BGX sections is significantly poorer than in non-BGX sections. This may be a function of the lack of upperclass students in BGX sections, or the lack of interest of BGX students in the course topic. (They are forced to take 1 BGX class, and our fear is that they are taking a course based on scheduling availability instead of interest in the subject matter).
In 8 of our regularly taught courses (POLS 172, 475, 476, 345, 454, 400, 473, and 479) our faculty members use role playing simulation, debate, or case study methods to provide students with structured opportunities for oral presentation. Based on senior exit surveys and observation of the capstone oral presentations, this appears to have reduced the difference between student skills in writing vs. oral communication.

To improve written communication skills, department faculty are experimenting with a new technique, suggested and implemented by Dr. Shannon Orr and Dr. Candace Archer, of using short in-class writing assignments. These are based on reading assignments, and used to encourage student to do the readings. However, feedback provided on these short assignments has proven to be a very useful way to improve student writing on subsequent research paper assignments. This has been a problem in courses where only one long paper was assigned, and students essentially did not get feedback on their written work except for that paper. Thus we could not clearly evaluate improvement in writing ability. Given the apparent success of this technique (of course it would be best to actually assess this!), others in the department are beginning to implement the technique.
Plan for 2006-07:
The department will teach the newly designed capstone seminar in Fall 2006 under a title related to International Relations/Comparative politics (seminar in Peace and War). Following recommendations from program review, the department will review and adopt the 2 new learning outcomes below, which are more discipline-specific, to replace or supplement outcome #2.

Ability to apply theoretical perspectives to identify and explain domestic/global political problems.
Ability to understand the basic techniques of political inquiry, including statistical analysis and qualitative methods.

The rubric used to assess POLS 290 (a sophomore level research methods course, required of majors) will be further developed to make it compatible with the rubric used in the capstone and with the new learning outcomes.
Since the exit surveys have low return rates and are not showing significant changes in patterns of data, they will be re-designed or discontinued. The department is considering adoption of the new teaching evaluation survey instrument from the IDEA center which is being piloted in several departments in Spring 2007. This may make regular teaching evaluations a more effective assessment tool.
POLS 490 (capstone) seminar will be proposed as a required course in the curriculum for all majors provided that the new design continues to bring positive results.
The department will make an effort to organize an electronic portfolio/archive of POLS 290 and capstone papers. Our belief is that students will not retain these on their own, but rather would be better served if the department maintained the archive.
Finally, the department's graduate committee will develop a full assessment plan for its MPA program as part of a major curriculum revision that the department has adopted and which is pending at the graduate college.