Student Achievement Assessment Committee

Learning Outcomes:

At the completion of baccalaureate degree studies in Psychology, students will:

• Exhibit broad knowledge about human behavior from a variety of psychological perspectives (e.g., biological, cognitive, developmental, social).
• Have the necessary skills in research and other forms of inquiry in order to develop new knowledge about behavior.
• Be able to communicate their knowledge of psychology to others.
• Have the necessary skills and content knowledge to be an informed and critical consumer of existing knowledge.
• Be prepared for post-baccalaureate studies in psychology or related disciplines, or for entering the workforce in areas related to their training.

Annual Report

1.  Learning Outcomes assessed this year 

During 2005-06, we assessed all five learning outcomes using the Psychological Skills Inventory, an instrument that wedeveloped and piloted last year. The inventory lists a set of 57 specific, objective skills that we feel are important for our majors to acquire. We take evidence of these skills being acquired as evidence that the learning outcomes have been achieved. The skills are grouped into six categories. The categories, and the learning outcome or outcomes most relevant to those categories, are: Written/Oral Communication (Learning Outcome 3); Research Skills (Learning Outcomes 2, 4, and 5); Engagement (Learning Outcome 5); Critical Thinking/Problem Solving (all five learning outcomes); Ethics/Values (Learning Outcome 5, and to a lesser extent, 1); and Integration of Knowledge (Learning Outcomes 1, 3, and 5).

Last year, we planned to use an “extreme groups” approach to pilot test the inventory, comparing responses from seniors to those from first-year students. Because we had so few responses from the latter group, we combined data from first- and second-year students and compare them to those obtained from fourth-year students. For five of the six categories of skill, seniors had significantly higher scores than the combined first- and second-year students; the scores were the same for both groups in the Engagement category. Not surprisingly, the seniors reported taking three times the number of psychology classes taken by the first- and second-year students. Taken together, these results are consistent with the notion that by taking more psychology classes one acquires more skills in nearly all categories.

The goal this year was to obtain data from additional students, increasing the sample size so that we could make comparisons among students by class standing on the categories of skills that are acquired. We added two items to the inventory, asking students to indicate which courses they had transferred from other schools and to report the schools where those courses were taken. As described in last year’s report, we are interested in determining if there is value added – in terms of acquiring important skills – to coming to BGSU for three or four years rather than only one or two.

2.  Assessment Methods and Procedures

The Inventory was administered at the end of the spring semester to students who were taking our 300-level lab courses. Students who came in for advising appointments in the latter portion of the spring semester were also asked to fill out the inventory. Accordingly, nearly all of our respondents are psychology majors. We collected data at the end of the academic year so that our primary independent variable -- class standing -- would be as meaningful as possible. (A second-semester sophomore and a first-semester junior may have taken the exact same psychology classes, but they would be coded differently where no real difference exists. The potential for second-semester sophomores and juniors to have taken the same classes is much lower by comparison. Our approach assumes that the progress of most students is in synch with the academic year.)

The Psychology Skills Inventory is a list of skills, and students are asked to check those items that apply to them. Students are told that they are not expected to have acquired all 57 skills, particularly if they hadn't taken many psychology classes. In an attempt to reduce the demand characteristic to check every item, students are told "You may find that you'll check a lot of items in some areas, fewer in others, and possibly none in still other areas." The clusters of items from the Research Skills category are shown below:

A. Searching for information used in research.                                                                                       
__ I have used psychology-related databases (e.g., PsycInfo) to search for materials in the library.
__ I have used the Internet to obtain information for a psychology project.
__ I have gathered information by interviewing or e-mailing experts in a psychology-related field.
B. Reading the professional literature (scientific/professional books and journal articles).
__ I have read at least 3 articles in scholarly psychology journals.
__ I have prepared an annotated bibliography (citations with summaries) for a psychology project.
__ I have read at least 3 chapters in psychology-related scholarly books (not textbooks).
C. Constructing a survey or psychological scale.                                                                                     
__ I have helped conduct a formal psychology-related survey project.
__ I have written questions for a psychology-related written questionnaire or structured interview.
__ I have designed and administered a psychology-related survey to a large (25+) group of individuals. 
D. Designing and conducting research.                                                                                             
 __ I have helped design a psychological research project.
__ I have helped conduct a psychological research project.
__ I have independently designed and conducted a psychological research project.
E. Analyzing and interpreting research results.
__ I have used statistical software programs such as SPSS or SAS to analyze data.
__ I have answered research questions using archival data bases (e.g., census data).
__ I have performed a content analysis of qualitative data (e.g., answers to open ended questions).
F. Using technical skills to do psychological research.
__ I have used laboratory equipment to gather psychological data from research participants.
__ I have learned to write or modify software (e.g., E-prime, web-based survey software) in order to collect data by computer.
__ I have used a coding scheme to code the behavior of research participants from video tape.

Students are also asked to indicate which psychology courses they have taken (and where if they were transferred from another school), their year in school, gender, GPA, major, and minor. We report the data from 193 students, 89 seniors (46%), 68 juniors (35%), 22 sophomores (11%) and 14 first-year students (7%).

3.   Inferences from Assessment

The number of items checked in a skill category was tallied for each student, giving a score for each category. These scores were submitted to a MANOVA, with class standing as the input factor and the scores on the six factors plus the number of classes taken as output variables. The table below shows the average score in each skill category (and the number of classes taken) by class standing. Within a row, means with different superscripts are significantly different from each other at the .05 level. MANOVA summary tables and post hoc tests are available upon request.                                   

Number of Classes Taken. As one would expect, the number of classes taken increases with class standing. As the superscripts associated with the means indicate, seniors took significantly more classes than juniors, who took significantly more than sophomores, who took significantly more than first-year students.

Written/Oral Communication. These findings indicate that with each successive year, students acquire significantly more communication skills. The biggest jump occurs as a junior, when most students have taken smaller classes where writing is emphasized (e.g., PSYC 290, 300-level labs). There was a small but significant correlation between scores in this category and self-reported GPA [r = .162, p < .05], indicating that better students tend to have better communication skills than poorer ones.

Research Skills. These results parallel those obtained with communication skills: students acquire significantly more research skills every year, with the biggest jump occurring during the junior year. Unlike with communication skills, where the difference between seniors and juniors is small (albeit significant), research skills continue to be acquired at a high rate through the senior year. This was a particularly gratifying finding given the importance we place on research skills.

Engagement. Scores in this category remain at the same level for the first three years and then increase significantly during the senior year. The increase in engagement as a senior seems appropriate: we wouldn't expect (or necessarily want) students with minimal preparation working in the community. Mirroring last year’s findings, the scores in this category are low across the board. We either may be setting too high a bar for our students in this category or not communicating the importance of these skills to our students.

Critical Thinking/Problem Solving. The acquisition of skills in this category increases steadily across one’s time in school although at a slower rate than for both communication and research skills. Scores from successive years do not differ from one another, but those from every other year do. In last year’s sample there was a correlation between scores in this category and self-reported GPA; the correlation in this year’s sample was significant only at the p < .10 level.

Ethics/Values. Skills associated with ethics and values appear to be acquired in three stages: during the first, second, and fourth years, with no increase from the sophomore to junior year. One possible reason for this is that research ethics are covered most heavily in the research methods course (PSYC 290), a course that is taken by both sophomores and juniors.

Integration of Knowledge. Scores in this category remain constant during the first two years and then increase to a higher level during the last two years. There was a small but significant correlation between scores in this category and self-reported GPA [r = .149, p < .05], suggesting that better students exhibit these skills more readily than poorer ones.

In general, we are pleased with the results of this more extensive test of the inventory. Unlike what we found in last year’s pilot test, scores were significantly higher for seniors than for first-year students for every skill category. That skill acquisition in four of the categories is unrelated to GPA is a sign that all our majors – not simply the better ones – are acquiring these skills. The two categories (Communication, Integration of Knowledge) in which we found modest relationships between GPA and skill acquisition may reflect a more general scholarly aptitude. We are also pleased to find that the inventory is sensitive enough to capture differences in skill level by year, albeit for only some skill categories. Whether the inability to do so for all skill categories is due to small (and unequal) sample sizes at each grade level or to different time courses for the acquisition of certain skills is unclear. Certainly, increasing the sample size would be beneficial.

Increasing the sample size will allow us to examine at a finer level what different categories of students – other than year in school -- are learning. For example, preliminary findings indicate that there is a possible advantage to spending four years at BGSU compared to coming in as a transfer student. These results were obtained by splitting the students into two groups: those who transferred at most one course from another institution and those who transferred more than one course (irrespective of whether the courses were taken at a two- or four-year college). There were 183 students in the first group, 10 in the second. Of these 10 transfer students, nine were either juniors or seniors. Consequently, we compared scores obtained on the categories of skill by “native” juniors and seniors to those obtained from our nine transfer students.

For five of the skill categories, the scores of the natives and transfer students did not differ statistically. In the Research Skills category, however, the native students averaged 11.3 skills acquired whereas the transfer students’ average was significantly lower at 8.56 [F(1, 155) = 5.825, p < .05]. A subsequent analysis of the skills within the category revealed that transfer students scored significantly lower than native students on clusters A, C, and F (see pp. 2-3). Whereas many of the skills contained in clusters B, D, and E are those that can be generally acquired in our research methods and 300-level lab classes, it can be argued that many of the skills in items A, C, and F (especially F) are acquired when students are involved in research outside of the classroom. Transfer students may not be getting the same opportunities that native students do, possibly because they have less time (e.g. only two years) to complete the sequence of required courses as well as get to know faculty well enough to become integrated into research programs.

4.   Actions Taken/Program Improvements

Because responses to the Psychological Skills Inventory were gathered at the end of the spring semester, no actions have been taken as a result of the 2005-06 assessment. The actions that are planned for 2006-07 include:

• We will communicate with new transfer students the importance of getting involved with research outside of the classroom, particularly if the students plan to attend graduate school. With help from the College of Arts & Sciences, we can assign all transfer students to the same academic advisor. The Transfer Advisor can easily identify and contact all new students (through the improved MyAdvisees system on Blackboard) to communicate this information. Another way of “getting the word out” to all students would be to incorporate the skill inventory into the learning outcomes matrix for psychology majors in ePortfolios.

• We will develop a system for getting better response rates from underclassmen and transfer students. Last year’s plan to have all students who come to mandatory advising sessions to fill out the inventory was not well implemented. A web-based delivery system is a possibility.

• Armed with a bigger sample than we had last year, we will review the 57 items to see if we need to revise the inventory. As noted earlier, scores in the Engagement category are low: seniors checked an average of 2.2 of the nine items, or roughly 25% of the items in the category. In the other skill categories, seniors check roughly 70% of the items.

• We will explore the possibility of revising the inventory in such a way that the three items in each cluster represent a continuous progression of acquiring higher levels of skill. In its current form, any item checked is equivalent to any other item checked, which is not necessarily consistent with the notion that some skills are cumulative or hierarchical.

• We can explore ways of classifying the different types of psychology classes that we offer (e.g., research-oriented, small seminar, second-level introductory) so that we can use type of class as an input variable rather than class standing to get a better sense of what skills are acquired in which type of class.

The skills inventory deliberately focuses on acquisition of skill rather than of content per se. We assume, as do many cognitive scientists, that appropriate content (declarative) knowledge must be learned before a skill can be properly demonstrated. What is missing from the current instrument is some measure of the quality of skill being demonstrated. For example, a student who received a “C-“ on an APA-style research paper and another who received an “A” would both endorse the item “I have written an APA-style research paper,” but it would seem imprudent to consider them equally skilled as is currently the case. We will continue to work on improving our instruments so that we can be confident that we are assessing learning outcomes to the best of our ability.