Student Achievement Assessment Committee

1. Learning Outcomes in the Department of Psychology

At the completion of baccalaureate degree studies in Psychology, students will:

• Exhibit broad knowledge about human behavior from a variety of psychological perspectives (e.g., biological, cognitive, developmental, social).
  
• Have the necessary skills in research and other forms of inquiry in order to develop new knowledge about behavior.
  
• Be able to communicate their knowledge of psychology to others.
  
• Have the necessary skills and content knowledge to be an informed and critical consumer of existing knowledge.
  
• Be prepared for post-baccalaureate studies in psychology or related disciplines, or for entering the workforce in areas related to their training.
  
  This report describes the results of the assessment activities carried out in the Department of Psychology during the 2001-02 AY. There were two projects during this time period, one that focused on the last learning outcome (Preparedness) using our questionnaire-based assessment of student learning outcomes and another that examined the match between our learning outcomes and those that were recently proposed by the Education Directorate of the American Psychological Association.
  
  2. Assessment Activities.
  
  A) "Relevance for Career" Across Years in College. Over the past several years, we have been refining our Assessment of Student Learning (ASL) instrument, which is administered during the last week of class. This instrument asks students to rate the extent to which an item described their experiences in a given course. A sample item from the set of 35 is "I connected what I learned in this course with courses in the natural sciences (e.g., biology, chemistry, physics, geology)." Students would indicate whether that statement reflect their experiences Very Well, Well Adequately, Partly, or Not At All. These, then, are students' perceptions of what was learned in the class. (Copies of the Spring 01 version of the inventory are available upon request.)
  
  
  As reported in previous updates, analysis of the ASL instrument indicates that it is made up of six subscales, each of which reflects a different category of learning outcomes. The subscales are: (1) Scientific Inquiry and Problem Solving, (2) Inter- and Intrapersonal Growth, (3) Research Skills and Logical Thinking, (4) Integration, (5) Relevance for Career or Future, and (6) Working with Others. Last year we reported that there was a decrease on scores along the fifth subscale as students progress through the major, with seniors indicating that the courses they were taking had less relevance for their careers than did juniors, sophomores, and freshmen. We found this trend troubling and decided to look at this issue again this year.
  
  A total of 2099 usable responses to the ASL instrument were obtained for classes taught during the 2001-02 AY. As we found last year, scores on the Relevance for Career or Future factor systematically decreased with year in school (F = 4.88, p < .005). Unlike last year's findings, however, in which post hoc analyses indicated that senior psychology majors saw the courses that they took as less useful than those taken by freshmen, sophomores, and juniors, post hoc analyses of this year's findings revealed that seniors, junior, and sophomores found their courses equally relevant but at a lower level than did freshmen. That is, the mean rating on the Relevance for Career or Future factor for freshmen was 12.2 whereas for sophomores, junior, and seniors the mean ratings were 10.7, 10.9, and 10.7, respectively. Relevance does not diminish beyond the first year. Although it is a bit worrisome that two different results were obtained with two different data sets, it is difficult to know to what the differences should be attributed. There is a high degree of overlap in which courses that are offered each year, but there is probably sufficient variability in the courses that are offered, the students who take them, and the instructors who teach them to make it risky to make direct comparisons across samples. Considering that last year's findings were based on a sample of 746 responses received over three semesters and that this year's findings are based on 865 responses obtained over two semesters, however, we tend to put more faith in the more recent findings.
  
  After last year's findings were presented to the faculty, a faculty member proposed a possible reason for seniors finding the courses they were taking less relevant than did their more junior counterparts. This faculty member noted that students often wait until their senior year before taking the two required laboratory courses, most of which involve collecting, analyzing, and understanding data obtained in various sub-fields of experimental psychology (e.g., cognitive science, behavioral neuroscience, experimental social psychology). He suggested that because an overwhelming majority of our senior psychology majors has planned NOT to go on in experimental psychology - which is typical of the field - it could be that the lab courses are seen as less career-relevant. Considering that in last year's sample over 80% of the students enrolled in the (junior-level) lab courses were seniors, if lab courses are indeed seen as not being relevant to the modal psychology major's career, one would expect to see a decrease in career-relevance during the senior year.
  
  Although the finding that the above proposal was designed to explain was not replicated in the current sample, the tenets of the proposal are still worth examining. After all, we might expect that courses that one takes as a senior would seem more relevant for one's career than those taken earlier, not simply as relevant as the current sample indicates. That is, if we are adequately preparing students for life beyond college, we might expect scores on the Relevance for Career or Future factor to be higher for seniors than for sophomores or juniors instead of equal to, which is what was found. Note that the lab courses were not the only courses that seniors took. If, however, the lab courses were deemed less relevant than other courses, those low ratings could have masked a possible increase in relevance in the non-lab courses that seniors were taking. That is, the seniors' mean score of 10.7 on the relevance factor could reflect high scores on non-lab courses that are offset by low scores on the lab courses.
  
  We explored the possibility that lab and non-lab courses differ in their relevance for seniors by comparing scores on the Relevance for Career or Future factor. A total of 93 responses were obtained from seniors in lab courses and 214 from seniors in non-lab courses. The means for these two groups were 11.3 and 10.5, respectively. An independent groups ANOVA revealed that this difference was significant (F = 4.04, p < .05), indicating that lab courses were rated by seniors as more relevant for their future than were the non-lab courses. This runs counter to the proposal generated last year in which it was assumed that lab courses would be seen as being less relevant than others. (Strictly speaking, independent groups ANOVAs should not be computed on these data because it is likely that some of the same students provided responses in both conditions. Given that repeated-measures designs generally yield smaller estimates of within-groups variability than comparable independent groups designs - which would result in a larger F given the same between-groups variability - we treat the results reported here as conservative estimates of any differences between groups.)
  
  B) Comparison of BGSU's and APA's Student Learning Outcomes. The Education Directorate of the American Psychological Association has developed a draft version of learning outcomes for students who receive baccalaureate degrees in psychology. The APA lists 10 major categories of learning outcomes, five of which represent the knowledge, skills, and values that are consistent with"the science and application of psychology," with the remaining five being outcomes that are consistent with a Liberal Arts education and are further developed in psychology. Although the subscales of our ASL instrument correspond nicely with the university's general learning outcomes, it seemed important to determine how closely our learning outcomes correspond with those of the discipline. After all, regardless of how well our outcomes correspond with the university's, if our psychology majors are not learning what the field considers vital, we could not truly consider ourselves a psychology department. The second assessment activity that we report on is an investigation of the correspondence between the learning outcomes we have designated and those of the American Psychological Association.
  
  The APA's learning outcomes, accompanied with brief descriptions, are listed below. The outcomes that are thought to be specific to the psychology degree are marked with an asterisk.
  
  *1. Knowledge Base of Psychology
  Demonstrate familiarity with the major concepts, theoretical perspectives, empirical findings, and historical trends in psychology.
  
  *2. Research Methods in Psychology
  Understand and apply basic research methods in psychology, including research design, data analysis, and interpretation.
  *3. Critical Thinking Skills in Psychology
  Respect and use critical and creative thinking, skeptical inquiry, and, when possible, the scientific approach to solve problems related to behavior and mental processes.
  
  *4. Application of Psychology
  Understand and apply psychological principles to personal, social, and organizational issues.
  
  *5. Values in Psychology
  Value empirical evidence, tolerate ambiguity, act ethically, and reflect other values that are the underpinnings of psychology as a science.
  
  6. Information and Technological Literacy
  Demonstrate information competence and the ability to use computers and other technology for many purposes.
  
  7. Communication Skills
  Communicate effectively in a variety of formats.
  
  8. Sociocultural and International Awareness
  Recognize, understand, and respect the complexity of sociocultural and international diversity.
  
  9. Personal Development
  Develop insight into their own and others' behavior and mental processes and apply effective strategies for self-management and self-improvement.
  
  10. Career Planning and Development
  Pursue realistic ideas about how to implement their psychological knowledge, skills, and values in occupational pursuits in a variety of settings.
  
  Forty advanced undergraduates were given a sheet that described APA's ten learning outcomes and our ASL instrument. The students were asked to classify each of the 35 items of our instrument (e.g., "I learned to express myself in writing") into one of the 10 learning outcomes listed above. Although some of the items may legitimately be seen as belonging to more than one category, participants were instructed to classify items according to the category of best fit. The results of this sorting task are seen in the table on the following page.
  
  The top row of the table shows the numbers 1 through 10, designating the APA learning outcomes. The numbers down the leftmost column represent the item number on the ASL. (A copy of the most recent version of the instrument is included.) The numbers in each row represent the percentage of people who placed that item into a specific category. For example, 100% of the participants placed Item 1 in APA Learning Outcome 1 (The remarkable agreement seen here is due to this item being used as an example when the instructions were given.) Note that at least 15% of the sorters had to have picked a given category for it to entered in the table.

Recall that the six subscales of the ASL were arrived at through factor analysis, a statistical procedure for seeing how well items in the instrument cluster together based on lots of peoples' responses to the items. The current version of the ASL has gone through several revisions since its inception, and we have deemed it a useful assessment tool because the scores on these subscales make sense. That is, scores on the Research Skills and Logical Thinking subscale are very high for courses in statistics and research methods and are lower for the introductory psychology course. There is a high degree of internal consistency with the ASL, and it appears to be valid when using internal criteria. This sorting task, on the other hand, offers a validation of the ASL using external criteria. That is, by asking people to sort our items into APA's categories, we are essentially buying into their factor structure and seeing how well our items fit.

If the ASL and APA's learning outcomes matched well, one would expect to see one or perhaps two entries in each row, with the row totals hovering around 90%. In addition, one would also expect to see 3-5 entries down each column, with column averages hovering in the 80s. What we have here differs from the good match in several ways. First, there are a few items whose row totals are only around 60%: Items 16, 17, 18, and 21 stand out. The last three items are those that deal with "connecting" to other disciplines, something for which APA has no explicit category. (Indeed, the dominant category choice for these items was "Application of Psychology.") Item 16, however, seems a bit adrift, with unsure footing for APA's first and third categories only. It may be necessary to reword that item - and similar items for which there is no dominant choice (e.g., Nos. 7, 13, 17, 30, 31) or eliminate them. The second way that the match is less than perfect is that there are some outcomes deemed important by APA that the ASL does not currently assess. In particular, we have essentially ignored Outcome 5 (Values), and we have too few items that strongly capture Outcomes 2 or 8 (Research Skills and Sociocultural and International Awareness). All in all, though, the match between APA's and our learning outcomes looks pretty good, suggesting that there is a high degree of commonality between what we and what APA's Education Directorate believe to be important for psychology majors to learn.

3. Actions Taken

A) "Relevance for Career" Across Years in College.
At the risk of appearing glib, the actions taken as a result of the analyses reported above is to do further analyses. For example, having determined that the required laboratory courses are seen by senior psychology majors as being more relevant for their careers than other courses they take, the question that arises is: given the variety of courses that seniors take, which are seen as most and least relevant? The numbers are small, but the initial results of subsequent analyses suggest that the low-enrollment specialized, 400-level courses are seen as even more relevant for seniors' careers than the lab courses. In particular, combining the responses from senior psychology majors who are taking PSYC 415 Psychology of Religion and Spirituality, PSYC 440 General Seminar (e.g., psychology of language, cognitive science), or PSYC 455 Stress Factors of Work, we find that the score on the relevance subscale is 11.8, which is higher than what we saw with the lab courses. PSYC 415, 440, and 455 are all low-enrollment (20-25 students) courses in somewhat specialized areas of the field. The numbers are low for these courses simply because they are not offered with high regularity. If indeed we want our seniors to be taking courses that they feel are relevant, we should do what we can to increase the frequency with which these low-enrollment, specialized courses are offered.

B) Comparison of BGSU's and APA's Student Learning Outcomes.
The less than perfect match of the two sets of learning outcomes has forced us to re-examine our assessment instrument. We will reexamine the factor structure of the ASL and possibly eliminate some of our items while adding others that reflect APA's outcomes. In particular, we will examine the need to include Values outcomes in our instrument, as well as additional items that the APA refers to as Research Skills and Sociocultural and International Awareness. Finally, we will consider whether the university's outcome To Connect is consistent with those established by the APA, which in some sense is a higher authority for our discipline than is the university.
(The 32-page report from the Task Force on Undergraduate Psychology Major Competencies that was appointed by the APA's Board of Educational Affairs is available upon request.)

Please let me know if you would like additional details concerning the Department of Psychology's assessment efforts. Overall, I am pleased with the directions in which we are going, and I hope that we will continue to make progress in those areas. I look forward to feedback from the Student Achievement Assessment Committee