Student Achievement Assessment Committee

THE UNDERGRADUATE PROGRAM

Learning Objectives

The following four learning outcomes were adopted by the Department of Sociology in 1996 as learning objectives for its undergraduate program:

Outcome 1
Development of the sociological perspective--a recognition of the importance of culture and social structure as fundamental social forces that influence human behavior at the individual, group, organizational, institutional, and societal levels. An understanding of basic sociological constructs and theories relevant to particular subareas of the discipline (e.g., criminology, demography, social psychology), and the ability to apply these to various areas of social life (e.g., family, education, government, community, business).

Outcome 2
An understanding of the diverse ways in which sociologists gather, interpret, and evaluate data, with a particular focus on the measurement of sociological constructs, inferring causal relationships, generalizing from samples to populations, and performing basic statistical analyses.

Outcome 3
The ability to articulate sociologically informed opinions and arguments concerning social and behavioral phenomena, and the ability to critically read and understand an argument and to critically evaluate that argument.

Outcome 4
An understanding of the similarities and dissimilarities of behaviors, attitudes, values, beliefs, and opinions across diverse social groups, and an appreciation of how various aspects of the social experience (e.g., occupational opportunities, crime, fertility) are structured or influenced by such factors as race, ethnicity, age, gender, and social class.

These outcomes have received roughly equal emphasis in our undergraduate program over the past year. Our earlier assessments, suggest that our students are doing a good job of internalizing sociological knowledge (evidenced, in part, by remarkably good performance on the Sociology Graduate Record Examination), but that their greatest difficulties come in connecting sociological generalizations and theories to empirical evidence about real human behavior. In consequence, we have (1) encouraged our majors to take their methods and statistics courses as early as possible in their undergraduate careers; (2) encouraged more majors to take our optional course in sociological theory; (3) exposed students to empirical analyses of relevant social phenomena in virtually all of their substantive courses; and (4) emphasized the importance of approaching sociological research and writing critically rather than passively. These efforts have helped us to continue to make progress in improving the sociological literacy and abilities of our graduates.

Assessment Procedures

The original plan for evaluating the degree to which these objectives were being accomplished –- that is, the extent to which our graduates demonstrate the learning outcomes in their work –- involved assessment of a sample of term papers written by graduating seniors by both internal and external reviewers. An external reviewer (Prof. Louis N. Gray, Director of Graduate Studies, Department of Sociology, Washington State University) has evaluated samples of seniors’ term papers each year since 1996. In the first three years the samples consisted of 9-10 papers each, selected randomly from those written by graduating senior sociology majors in all of our courses, and copied (without the student’s name or other identifying information) prior to grading by the course instructor. Since 1999 the papers he has evaluated have been those written for our Senior Seminar (SOC 480), which has been offered since Spring 1999 as a “capstone course” for sociology majors.

The Senior Seminar was implemented as part of our learning assessment process. The objective of the course is to give the students direct exposure to ongoing sociological research by having each faculty member in the department make a presentation and lead a discussion on his/her current research project and/or interests. This is followed by another discussion on the general area in which this research falls led by the course instructor. Students are required to read and comment on a journal article written by the professor who makes the presentation. This process also helps to summarize some of the knowledge students have acquired in their other Sociology courses.

Students in this course do two things that contribute directly to learning assessment. First, the majority of their grade in the seminar comes from a term paper, which they do under the guidance of a department faculty member of their choice. The paper, which may deal with any issue in sociology, is to be a research paper. The papers done for this seminar are then evaluated by both internal and external reviewers to ascertain the degree to which they reflect the accomplishment of the learning outcomes noted above.

Second, as a final examination, students are given a recent version of the Graduate Record Examination in Sociology. National norms are available for this test, the content of which reflects a consensus in the discipline concerning the knowledge and skills that sociology graduates should possess. The disadvantage of the test is that it is normed on populations of students who expect to attend graduate school in sociology, a small and highly self-selected subset of all sociology majors. The norms are, therefore, higher than would be derived from a test administered to all sociology majors.

Currently the Senior Seminar is not a required course. This means that enrollment is small (the averaging about seven students), and the students are self-selected and probably not representative of all sociology majors. We would like to require the course for all our majors, but unless we taught it in lecture mode to fairly large classes, we do not yet have the faculty personnel to handle the demand. We intend to continue offering the course as an elective for the next several years, while encouraging as many students as possible, from across the entire spectrum of ability, to take it. We believe it is very important for pedagogical reasons to retain its identity as a seminar, although this makes it somewhat less useful as a learning assessment tool.
Results

Term papers were submitted in partial completion of the Senior Seminar in the Sociology Department, SOC 480, offered Fall Semester 2004 by Martha Mazzarella. Seven Sociology majors self-selected into this challenging course. Their papers were evaluated separately by an internal committed and an external reviewer in terms of the Sociology Department’s four learning outcomes. Some general observations and suggestions follow the assessment.

Internal Committee

Outcome 1—Development of the sociological perspective

The sociological perspective should emphasize, among other things, how an individual’s location in the social structure influences life choices and the patterning of behavior. All of these papers address important social phenomena and processes. These topics include: “Masculine Ritual in Sport Subculture: The Case of Rugby and Symbolic Deviance”; The Coming Out Process Among Gay and Lesbian College Students”; “The Juvenile Death Penalty: Debatable, but Unconstitutional?”; “Binge Drinking, Rape, and the Dangers of Alcohol: Is it Just a Greek Problem?”; “Capital Punishment from an International Perspective”; “The Crimes People Do Not Commit: An Analysis of Motivations for Avoided Crimes”; “A Literature Review of Homosexual Parents and the Outcomes of Their Children.”

Once again this outcome continues to be the most successfully achieved. These papers reflect a range of relevant sociological interests. They demonstrate an understanding of a variety of sociological perspectives and how sociologists approach a number of social and interpersonal issues. At least two papers take more of the symbolic interactionist, or micro-sociological approach. For example, by focusing on the “coming out process,” one student shows a very good understanding of the importance of significant others in terms of accepting an identity that is stigmatized in American society. Similarly, by emphasizing the motivation to avoid crimes, another student shows a keen understanding of the issue of agency; that is, individuals do have control over personal behavior including criminal behavior. Two papers focus on capital punishment, with one of the papers taking more of a cross-sectional approach, and the other connecting juvenile delinquency with American legal principles as reflected in the Constitution. The paper on masculine ritual in sport subculture is quite Durkheimian in its focus on ritual as one way of building a sense of community among men playing rugby. These are all very good examples of using a sociological perspective.

Most of the students draw on relevant sociological theory and recent academic research and highlight the importance and relevance of their chosen topic. If a shortcoming is identified in meeting this outcome it is that students have difficulty explicitly incorporating sociological theory. Because most of our seniors have not elected to take our undergraduate theory course, Dr. Mazzarella teaches sociological theory in the Senior Seminar. This points to a solution: require Soc 302 Introduction to Sociological Theory of all our majors.

Outcome 2—Understanding and using sociological research methodology

Overall, each of the papers draws on a large and relevant research literature. All meet the basic standard of competently weaving prior scholastic research into a coherent “story.” A number meet the even more impressive standard (especially given the academic stage and time constraints involved) of collecting and incorporating original data. The “Capital Punishment...” and “Crimes People Do Not Commit...” projects bear special mentioning in this regard.

In the paper entitled, “The Coming Out Process,” the student’s qualitative study began with the process of obtaining HSRB approval before conducting any interviews. After approval, the student used the “snowball” method (i.e., asked a gay friend, if he would be willing to participate and did he have any other gay friends that might be interested in participating). The student also attended a meeting of Vision, the gay and lesbian student organization on campus, presented her research proposal, and asked if students would be interested in volunteering for the study. With regard to the questions asked, her research focused on five questions which allowed the participants to explain in their own words, whom “they came out to,” and why, and well as age sex, and sexual identity. The approach was quite sound in the use and understanding of qualitative sociological research methodology. The student’s faculty mentor was terrifically pleased with the student’s seriousness regarding research protocol. For example, she met several times with the chair of the HSRB when she needed clarification on filling out the forms. Additionally, the student did not initially get approval and worked at modifying her proposal and consent forms in order to get HSRB approval. In sum, this paper shows an excellent understanding of sociological research methodology, including a brief discussion of the generalizability of the findings, as well as how the findings are consistent with other findings in the empirical literature. (For example, the empirical literature indicates that a large percentage of gay and lesbian young adults feel bad about their sexual orientation; this student’s research also reports this finding).

For the paper entitled, “The Crimes People Do Not Commit…,” the student creates, administers, and analyzes data based on a survey of 97 undergraduate students in SOC 101 classes. Survey research is the primary methodology used by sociologists; thus, the student’s approach is consistent with our departmental learning outcomes. The section of the paper entitled “Issues of Validity” is impressive because it shows that the student is thinking like a methodologist. For example, the student notes that the study is not based on a random probability sample; rather it is a convenience sample. These are the sorts of distinctions we hope that our students can make after taking our courses. We think it is noteworthy that this student will enter our graduate program in the fall.

The methodology used in the paper entitled “Juvenile Death Penalty…” can be viewed as an analysis of existing sources (i.e., the U.S. Constitution, and court cases). The approach is quite appropriate for scholars doing research in Sociology of the Law.

Many of the papers are quite impressive in terms of the literature covered and how these resources are used to support the papers’ statements and theses; others were disappointing. In summary, we think the use of qualitative interview data, survey data, and existing sources as reflected in the papers indicates that we are preparing students to do research using the methodologies that are appropriate to the discipline of Sociology.

Outcome 3—Articulation of sociological opinions and arguments

Here, the projects were more of a mixed bag. At the more impressive extreme (notably the “Capital Punishment...” and “Crimes People Do Not Commit...” projects) are papers that seem plausible as foundations for scholastic publication efforts. Less appealingly, at least a couple came across as too “preachy”—with empirical evidence taking a backseat to philosophical positions.

On a positive note, all of the papers did a good job of presenting sociological findings as part of their literature reviews. Journals cited included: Crime, Law, and Social Change, Sociology of Sport Journal, Acta Sociologica, International Review of Sport, Journal of Quantitative Criminology, Journal of Health and Social Behavior, Social Psychology Quarterly, Social Forces, Sociological Quarterly, Criminology, American Journal of Sociology, and American Sociological Review – what is important here, is that the students are using and citing the sources that provide the best examples of sociological opinions and arguments. In sum, we feel that the articulation of sociological opinions and arguments is appropriate given the level of students’ experiences.

Outcome 4—Understanding the influence of key sociological factors

All seven papers are satisfactory in this regard; all demonstrate appreciation of the potential relevance of such factors as reflected appraisals, attachments to conventional institutions, gender identity, peers, and sexual orientation.

For example, the paper “Coming Out . . .” focuses on the important role of supportive others in the development of self-identities; the paper on the “Juvenile Death Penalty” looks at age, youthfulness, and moral reasoning; the paper on “Binge Drinking, Rape” clearly articulates the notion of “problem behavior syndrome”; and the paper on “Motivations for Avoided Crimes” focuses on behavioral intentions to act. Thus, all of the papers show an understanding of key sociological factors including age, sexual orientation, social support, problem behaviors, and motivations.

General Assessment – Internal Committee

“As a group, these papers show substantial progress toward meeting the Sociology Department’s standards for senior-level papers. Overall, we continue to be impressed with the various topics chosen by the students in the senior seminar. The class of 2004 chose a number of important and socially relevant topics to explore in their research projects. These projects demonstrate an understanding of a variety of relevant sociological theories and perspectives as well as recent academic research, and, as a result, they illustrate how a sociological imagination is used to approach a number of social, cultural and interpersonal issues. And they clearly represent the investment of a tremendous amount of time and effort by the student authors.
“ We are pleased with the overall scope of the papers and their social relevance, as well as the theory, data and methodology drawn on in conducting these projects; we think the average quality of this year’s papers is much superior to last year’s batch, particularly in theory articulation. Further, these papers were much better edited and did not illustrate the major technical and stylistic problems that we have seen in other years, notably last year.
“ We are pleased that these students can engage successfully in sociological research that illustrates their own sociological imaginations—even the less successful papers illustrate creative imagining of sociological questions. And we would encourage students to extend their papers and imaginations, perhaps with a discussion of the implications of sociological research and solid suggestions for social policy.

“Ideally, the Senior Seminar should be available to all Sociology majors because it clearly has an important effect in improving students’ sociological reasoning, and their analytical and critical writing abilities. Although the papers evaluated here range in quality and with respect to the learning outcomes they exhibited, nevertheless, they continue to illustrate a higher level of research, analysis, and writing abilities than does the work of undergraduates who do not self-select this course.”

External Reviewer

Overall Assessment – External Reviewer

“This set of seven papers is generally mediocre. The best of them are reasonably good, but the worst fail to meet several of the department’s goals. Several of them are fairly “typical” student papers, some poorly edited, but seem to present their preferred outcome without real examination of evidence. That is, they often seem to be “position” papers rather than serious attempts to examine the chosen problem. With some exceptions there appears to be highly limited use of a sociological perspective in the papers. In general, the students seem to be fulfilling the requirements of a research paper without understanding what those requirements entail. There are numerous typographic errors and/or misspellings in some of the papers and incomplete references were common in some papers—and few of them were sociological for some papers. While the papers reflect the interests of their authors, these are not always problems to which the authors have given much thought. As a whole these papers represent a substantial return to the situation of earlier years, and the papers often show a low level of care in their development. As usual, I will address the desired outcomes in turn and evaluate the papers with respect to each.

Outcome 1—Development of the sociological perspective

Assuming the central focus of this outcome lies in the identification of cultural and structural factors that influence individual behavior, the papers do best when considered in light of this goal. While the papers do not all detail the extent to which cultural and structural factors influence human behavior, they do address several areas in which such effects can be identified, and, with few exceptions, the authors identify these factors. Most of the papers show an awareness that norms, parental behaviors, general cultural factors, etc., can alter the way in which individuals and groups relate to each other. There is general acknowledgement that individual characteristics alone do not determine the location of a person, group, or society in a more general social structure but there is also often a failure to treat these factors in other than simplistic terms. While problems emerge in their understanding of the interactive nature of these kinds of factors, they rarely involve more than a single pair of variables and larger-scale interactions do not seem to be considered. There is also a strong tendency to view most effects as unidirectional, rather than seeing them as the result of reciprocal interactions. These problems are not limited to these student papers, since they’re common in sociology but, again, such a restricted view characterizes too much sociological understanding. The technical aspects of sociological uses of language, which clearly characterized the last set of papers, seem to be inadequately understood by most of these students. There is evidence that the ability to take a sociological perspective is being developed, but I would hope to see more progress from seniors. With two exceptions, these students don’t appear to have adequately achieved this goal—though some progress in that direction is evident.

Outcome 2—Understanding and using sociological research methodology

This set of papers is largely qualitative in nature. Even reports of statistics from other sources tend to involve simple percentages and are rarely interpreted within a larger context. The authors also rarely did much in the way of seeking alternative explanations. This suggests to me that the department has not been successful in communicating a scientific approach to most of these students. While all the papers deal with issues of potential research, there continues to be a tendency to view published research uncritically. Everyone seemed to realize the value (and purpose) of social research, but few students adopted the critical view that is essential to good social science. While students at this level can rarely be expected to be research experts, their papers need to reflect a concern with the details of research and go beyond the repetition of class and other easily available material. Only two of the students appeared to have internalized this idea. This goal, as I interpret it, is designed to produce graduates who appreciate at least some of the finer points of research and the difficulties involved in understanding social phenomena. These students are capable of appearing to move in this direction, and may understand the reasons for doing so, but seem generally unable to locate material that contradicts their original position. When I am easily able to recognize conflicting material, while not being associated with any of the areas involved, it appears to me that the students understand what they should do, but don’t engage in the effort required to actually do it. The real progress evident over the last few years seems to have disappeared for most of these students.

Outcome 3—Articulation of sociological opinions and arguments

The level of theory dealt with in these papers is rather elementary, and only two papers really attempt to tackle potentially complicated theoretical issues. While no one ignores sociological theory entirely, the level is close to that presented in introductory courses. Only the papers on avoided crime and sport subculture (rugby) really attempt to get the reader to think more seriously about these complicated topics. Neither of these papers extends theory to the level possible, but the authors do seem aware of such extensions. The remaining papers, on binge drinking, homosexual parents, coming out, and capital punishment (two papers) show limited awareness of theory and no serious attempts to expand knowledge. While these kinds of skills are more likely to be developed in graduate students, we’d hope that we could move undergraduates in that direction. Usually their personal viewpoints are clear, but they often are characterized by overstatement or overgeneralization. The papers show some ability to work with sociological ideas, but much less than earlier groups. They seem to be getting some idea that sociological language is a technical, rather than an ordinary, language, and they do sometimes seem able to articulate elementary theoretical arguments. These students largely have not moved as much in the desired direction as recent others. Even the best papers clearly fall short of the graduate student level.

Outcome 4—Understanding the influence of key sociological factors

There is a clear awareness that a sociologically trained student is supposed to be cognizant of these notions, but I'm not sure of the degree to which they've actually been integrated into all the student’s perspectives on social issues. Several of the papers evidence an attachment to general cultural norms or international values, rather than specific ones. While the papers do show awareness that persons in different life circumstances may have different experiences and that these experiences can produce differences in behaviors, evidence of real understanding appears missing. Their understanding of the reasons for these differences seems undeveloped and there is a greater tendency to be judgmental than shown in other recent papers. There seems to be a tendency on the part of most of these students toward a rather absolutist approach to sociological ideas along with a corresponding reluctance to examine (or even expose themselves to) contrasting theory or evidence.

General Assessment—External Reviewer

“This set of seven papers represents a reversal of the improvement trend seen over the last set and its recent predecessors. I would say that the work of these students is similar to that of graduating seniors at WSU, but, unlike our typical distribution, appears to contain no particularly promising students. While the students are on the road to a sociological perspective and imagination, they have barely begun their journey and I suspect that few, if any, will reach that goal. None of their papers reach the level of those of WSU’s graduate applicants. This is the first set for which I’ve had to make that statement so I wonder if these papers aren’t somehow atypical. BGSU’s undergraduates, especially the best ones, have been more than competitive with Washington State University’s (and your graduate students are generally superior). The weaknesses for this group showed up mainly in their inability to critically analyze arguments regarding human social behavior and structures and fully appreciate the methodological and statistical problems faced by social scientists. Along with the tendency to ignore contradictory evidence and a trend toward overstatement, these papers often failed to even minimally meet the goals the department has set for itself.

“Based on my recent experience with undergraduates at WSU I can now say there’s a high degree of similarity in the two schools. Perhaps the most difficult goal of our teaching is the development of a willingness to examine contrary evidence by our students and, at the same time, avoidance of overgeneralization. To the extent that we can get our students to question their beliefs and stereotypes, we’ve gone a long way toward achieving our goals. For some reason BGSU seems have had more difficulty with this set of papers than in the recent past.

“My experience at WSU this last fall was similar, though I had a few standout students. I have the feeling that I’ve had more success in getting students to question things in the past though, than I had this year. That appears to be the case at BGSU as well. Perhaps my negative response to this set is due to the small number and high quality of papers in the previous set. If that is the reason, then future sets should resume the level of improvement seen in the past, but if there’s been a change in the nature of the students or in the way faculty approach them, that needs to be determined as well. I think it may be necessary to review the kinds of course exposure seniors have to see if their trajectories through the system may provide an explanation. Then, of course, this group may just be an aberration.”

Integration

The external and internal reviewers disagree substantially on the degree to which the papers produced in our SOC 480 course this year reflect accomplishment of each of the four major outcomes, almost as though the reviewers read completely different sets of papers. The internal reviewers, overall, are quite complementary of the accomplishments represented in the papers, while the external reviewer is consistently, severely critical of the same papers. This is the first year we have experienced such divergence between internal and external reviews, and we have difficulty reconciling the inconsistency. A conservative interpretation is that the papers do show that the students were able to produce sociological research papers that demonstrate modest progress toward the goals we set for them. According to the external reviewer, but not the internal committee, these papers do not show a consistently satisfactory level of accomplishment. If we give more weight to the external reviewer’s evaluation, a reasonable conclusion is that this year’s crop of student papers may represent a random, temporary decline in the steady improvement of the last few years. Regardless which set of reviews are more accurate, we will have to remain vigilant to ensure that we resume that pattern of improvement.

Evaluation of Graduate Record Examination Scores

All seven of the students in this year’s Senior Seminar took the Graduate Record Examination in Sociology. This examination has been discontinued, but we use the latest available version, score it ourselves, and compare the raw scores to the published national norms.

Further evidence of the success of our students is that, while the best students have scored in the 99th percentile, the least successful have always scored above the 55th percentile (this year the seven scores ranged from the 56th to the 96th percentile). This is a rigorous course and the students who successfully complete it develop the background necessary for future work in graduate and professional education.

Actions

Because our students’ performance this year shows some decline compared to last year, we believe we need to reinforce the efforts we have already initiated. Our planned actions are similar to last year’s, and represent our intent to remain consistent in these efforts.

1. We continue to encourage our majors to take their methods and statistics courses early in their studies, before taking a large number of our substantive courses, so the skills taught in these classes will be available to them. As noted in previous reports, however, this is difficult for several reasons, including the late declarations of many of our majors and the fact that they are attracted to our substantive rather than our methodological courses. We must also contend with the fact that a high proportion of students in even our upper-division courses are non-majors, so the content of those courses must be adjusted accordingly. Early mastery of methods and statistics will help our majors to grasp the connections between ideas and evidence in our substantive courses.

2. We are encouraging more of our majors to take our course in sociological theory (SOC 302), which is currently offered as an alternative to Social Psychology (SOC 301), taken by most students instead of theory. We do not yet have sufficient faculty members to staff the theory course if it were required of all majors. All are majors are, however urged to take theory. Furthermore, we should include a component in the course on theory construction to help the students learn better how to connect data or evidence to ideas.

3. Undergraduates are exposed to statistics and statistical reasoning in virtually all of their substantive courses. Development of skill in statistical reasoning will be facilitated if we can encourage more students to take the statistics course (SOC 369) earlier in their careers; but even in the absence of a formal course, most students can grasp the logic of statistical analysis in the context of substantive courses if it is presented cogently. Our primary learning objective is not that students understand which statistic is most appropriate for each particular analysis, but rather that they understand how and why statistics are used at all.

4. Sociological theories and explanations are presented to students not as revealed truth, but rather as useful tools for furthering our understanding of the social world and as “works in progress,” subject to criticism, revision, and possibly refutation. Both the internal and external reviewers of the Senior Seminar papers note frequently that students almost uniformly seem to treat sociology as a body of knowledge to be internalized rather than as an ongoing process of analysis and explanation. We must encourage students to engage the material actively, to think critically about the evidence and explanations we present to them, and to understand the value of competing explanations. At the same time, we caution them that sociological explanations are different than opinions; we do not “choose to believe” one explanation over others, but are instead convinced of the relative merits of explanations by the weight of logic and evidence. This is an issue we must address in all our courses.

5. We need to emphasize to our students the importance of learning and applying appropriate “production values” to their work. This includes issues of basic literacy in writing, but also organization of papers, citation styles, effective ways of presenting data, etc. This is partly a task for the faculty supervisors of the Senior Seminar papers, but we need to pay more attention to students’ writing abilities throughout the curriculum.

Back to top

GRADUATE PROGRAM

LEARNING OUTCOMES:

The Sociology Department’s goals for doctoral students are that they realize the learning outcomes enumerated below prior to completion of the Ph.D. degree. Each learning outcome is associated with a particular area of training. For M.A. students, the department’s goals are that students develop learning outcomes A1-A5 (in major area of concentration only), C1-C2, and D2, D3, and D5.

A. Major and Minor Areas of Concentration
1. An understanding of major theories within area of concentration.
2. Knowledge of major and recent research within area of concentration.
3. An understanding of methods appropriate for research within area of concentration.
4. An ability to apply appropriate theories and methods to one’s own research within the area of concentration.
5. An appreciation of policy implications of major theories and research in the area of concentration.

B. Theory
1. An understanding of major sociological theoretical themes and perspectives.

C. Quantitative Methodology
1. An understanding of major research techniques and quantitative methodology.
2. An ability to design and implement a research program.

D. Professionalization
1. An ability to design and effectively teach an undergraduate course.
2. An ability to effectively present research results orally, as in a seminar or at a conference.
3. An ability to effectively present research results in writing.
4. Personal experience with the process of submitting and publishing research articles.
5. Socialization into the profession. This can range from attending a conference, organizing a session, reviewing a journal article, to getting a job in one’s area of training.

Five different methods are used to assess the degree to which M.A. and Ph.D. students are successfully meeting departmental learning goals:
1. Annual review of selected theses and dissertations by two readers in the area of concentration and a reviewer selected from outside the sociology department.
2. Annual review of student vitae.
3. Teaching evaluations.
4. Annual graduate student survey and exit interview.
5. Job placement survey of graduates.
ANNUAL REPORT

1. Learning Outcomes Assessed this Year

The Department of Sociology Graduate Program Assessment Plan calls for some of our assessment activities to take place during the fall semester of each year, others during the spring semester. To this end, a number of assessment procedures were implemented during AY 2004-05: internal and external review of M.A. theses and Ph.D. dissertations; the annual assessment survey of continuing students; exit interviews with those completing our program at either the M.A. or Ph.D. level; evaluation of graduate student teaching; and the job placement survey of those having completed our program within the past six years.

2. Assessment Methods and Procedures

Review of Theses and Dissertations. Toward the end of the spring 2005 semester, each masters thesis and Ph.D. dissertation completed during the previous calendar year was evaluated by departmental faculty committees in the students’ areas of concentration (criminology and deviance, demography, family sociology, and social psychology). By design, none of the evaluators were the major advisors of the students. The assessment included the reviewers’ ratings (on a 5-point scale from “excellent” to “poor”) of how well each learning outcome noted above (A1-A5, B1, C1-C2, and D3) was demonstrated in the student’s work. The evaluation form for theses and dissertations is included at the end of this report in Appendix 1.

In addition to these internal evaluations, reviewers at other universities read and evaluated a sample of theses and dissertations completed during 2004 with respect to how well the department’s desired learning outcomes were exemplified in these works. Three theses and one dissertation, each representing one of our four areas of concentration, were sent out for review. The external reviewers evaluated the theses and dissertations using the same evaluation form used by the internal readers.

During calendar year 2004, three M.A. students and two Ph.D. students successfully defended their theses/dissertations in the area of Criminology/Deviant Behavior. All the theses were consistently rated by department faculty as “Excellent” or “Very Good” on all seven dimensions of our rating scale, with no dimension standing out as particularly strong or weak. The external reviewer’s assessment of the thesis was slightly lower, but all dimensions were still rated “Good” or higher. The external reviewer was particularly complementary of the methods and statistics used, especially considering that the student had received only M.A. level training. The reviewer recommended greater consideration be given to the policy implications of the results.

Ratings of the dissertation were more varied from the internal reviewers, from “Good” to “Excellent,” with the strongest ratings given for “understanding major theories within area of concentration” and “knowledge of major recent research with area of concentration.”

Overall, these evaluations provide reassurance that our graduate students are acquiring the knowledge and skills taught in our program. In producing theses and dissertations of the high quality represented in this year’s sample, graduates of our program have demonstrated that they are competent scholars of criminology and deviant behavior.

During 2004 three students graduated from the department’s program with a specialization in some area of Demographic Studies. Two of these three earned M.A. degrees with concentrations in Applied Demography. The sole Ph.D. recipient specialized in Population Studies. In the opinion of one internal reviewer, the Ph.D. graduate did quite well. On the seven assessment items there were no scores lower than very good, the second highest category. The two remaining scores were excellent, the highest possible outcome.

Each M.A. candidate was evaluated by two internal reviewers. One of the two students was graded as outstanding. That is, this person was scored as excellent in every category by the first reviewer. The other reviewer was less generous, but still no score was lower than very good, the second highest category. The second M.A. graduate mainly received a combination of excellent and very good scores, however, this person also received a score of good, the third highest score, for one item.

The external evaluator rated the thesis she reviewed as “very good” across the seven assessment dimensions. She found it to be very well written and organized, and a piece of work that expanded scholarship in the area by employing a new analytic strategy. The theoretical orientation was judged to be adequate, while the level of the quantitative analysis was seen as exceeding that typically found at the M.A. level. In fact, this student’s use of a complex data set and her rigorous analysis and presentation of results was viewed by the reviewer as meeting the standards of high quality sociology/demography journals.

In sum, the demography students are doing quite well according to both internal and external evaluators, and most scores on the assessment instrument were either excellent or very good.

Three students earned degrees from the Family Sociology specialization component of our graduate program. One of these students earned a M.A. and two earned their Ph.D.’s and left our program to enter university settings as entry-level professionals.

In the opinion of one internal reviewer, one Ph.D. graduate did quite well. On the seven assessment items, the student scored “excellent” on six and “very good” on the remaining item. The reviewer believed that the student was excellent in understanding major theories and bodies of research in the area of concentration, had a firm understanding of the methods appropriate for research, and was able to design, implement, and complete a useful research project that contributed to the literature. This student has several publications and has served in many capacities in the department during her time in the program.

In the opinion of another internal reviewer, the second Ph.D. graduate was not as strong. The student graduated with only a “fair” understanding of the major research in the area, “fair” understanding of the relationship between theories and methods for research, “fair” ability to design and implement a research project, and “fair” writing skills. The student received two “good” scores for their understanding of theories and methods generally, and a “very good” score for their understanding of the policy implications of their research.

In the opinion of two reviewers of our M.A. graduate, the student ranged from “good” to “very good” across all the items, with one reviewer rating the student’s ability to design and implement a research program as “excellent.” This student is continuing in our Ph.D. program.

The external review of one of the M.A. theses in the family area rated the research as “very good” on four of the seven evaluative criteria. The thesis was judged to be very thorough in its review of the research findings and skilled in the use of appropriate statistical techniques. Two weaknesses were identified: the lack of a detailed theoretical perspective guiding the research, and the failure to discuss the policy implications of the research. Nonetheless, the reviewer believed the student did a very competent job.
In sum, the family sociology students are doing well, as most scores were either good or very good with one student demonstrating true excellence across most dimensions. Our Ph.D. graduates this year are somewhat uneven in performance, with one student excelling in the completion of the dissertation and related work and another performing with some difficulty, though doing fairly good job.

Three students earned degrees in the Social Psychology specialization of our graduate program this past year. Two students earned their M.A. degrees and one earned her Ph.D. With respect to employment, the student who received her PhD is employed in private industry in Columbus as a data-user specialist. One M.A. student joined the Peace Corp, which was an important life goal for this individual. The other student is employed in marketing research in Portland Oregon.

In the opinion of one internal reviewer, the Ph.D. graduate wrote a very good dissertation. The reviewer emphasized that her criticisms were minor. In fact, this dissertation won the university-wide distinguished dissertation award. With respect to the departmental assessment instrument, the student scored extremely well. The student is well-published for someone so early in her career. For example, she has co-authored articles in the American Journal of Sociology, Social Psychology Quarterly, and the Journal of Health and Social Behavior; these are the premier outlets for sociologically-trained social psychologists. Additionally, this student also has a ‘revise and resubmit’ from the Journal of Health and Social Behavior in which she is the first author. This student also has numerous paper presentations.

The external reviewer rated this dissertation as “excellent” or “very good” across the seven evaluative dimensions. She noted that the student demonstrated a thorough understanding of the theory underlying the research, conducted a sophisticated analysis of the data in evaluating several alternative predictive models, and produced a very good, high caliber dissertation. She was disappointed that the student did not address the policy implications of her research, but also noted that this was outside the agenda of the dissertation.

With respect to the first MA student, the department assessment of the quality of the thesis was quite high. Additionally, the department nominated this thesis for the Distinguished Thesis Award. Written comments about the thesis emphasize its high quality, and especially the clarity of the writing. After two years in the Peace Corp, this student anticipates completing the Ph.D. degree.

Regarding the third candidate, the departmental assessments were very good. In the opinions of the reviewers, the quality of the thesis was very good, with perhaps, the only weakness being a lack of appreciation of policy implications. This student will not be continuing her graduate training.

In summary, the quality of the written work of these three students ranges from extremely high to very good. We are pleased with the strength of our social psychology students’ work and will continue to encourage students to choose important topics and appropriate methods for their research.

Annual Graduate Student Survey. Students were given the opportunity to provide feedback regarding their graduate experience via an on-line graduate student survey that was conducted during the spring 2005 semester. This survey was anonymous and was accessed via a secure Internet web site managed by Senicio Software in Bowling Green. Students had two weeks to complete the survey. Responses were submitted to the secure web site, and were subsequently tallied and summarized in the aggregate by an ad hoc faculty committee. Item-by-item summary results are reported in Appendix 2 of this document.

A total of 32 current graduate students (based on a total of 39 full-time funded students this is an excellent response rate of 82 percent) responded to the survey that assessed views and perspectives on the departmental climate, attitudes of the faculty towards students, graduate student climate, nature of faculty-student interaction, department chair, graduate advisor, major area adviser, courses and adequacy of preparation for a career of teaching and research. The survey included statements to which the students could give responses ranging from strongly agree (5) to strongly disagree (1). In general, the ratings of these distinct facets of the graduate student experience were in the favorable range. The chair, graduate advisor and major area advisor scores were particularly favorable (an average of 4.53, 4.87, and 4.84, respectively). Graduate students’ responses concerning faculty’s attitudes towards students, graduate student climate, and faculty-student interaction were also quite favorable (an average of 4.01, 4.32 and 4.37, respectively). Although students generally agreed that the courses taken are a good preparation for preliminary exams, and are good learning experiences, one area of concern related to the availability of courses offered each semester (2.88). The department was also perceived as sensitive to gender issues, but scores for racial/ethnic diversity were not as uniformly favorable (2.88). As shown in Table 1, there is little difference between this year’s student evaluation and last year’s. The three items for which the change from 2004 and 2005 reached statistical significance using a t test (p < 0.05) are "Q46. I get along well with my major advisor," "Q48. My major area advisor makes himself/herself available to me," and "Q49. I am satisfied with the amount and quality of time spent with my major area advisor." Students have a more favorable evaluation of their major area advisor this year than they did last year.

Students also had the opportunity to describe, through an open-ended format section of the interview schedule, those aspects of the program that they felt were the strongest, to provide their own descriptions of areas of change they would like to see in the department, as well as in the training they receive. The positive open-ended comments emphasized faculty-student relationships, including positive learning atmosphere, faculty availability and mentoring, and quality of the faculty itself. The section on changes students would like to see included several comments about the need for a greater course selection, a more balance training on both research and teaching, better graduate assistantship assignments, and some who felt that faculty could show more respect for and be more available to students.

Exit Interviews. Upon exiting the program (i.e., subsequent to defending the dissertation or terminal thesis, or leaving the program without a degree), students were interviewed by the Director of Graduate Studies for the purpose of gathering information on a variety of issues: whether our program has prepared them for the job world or subsequent graduate education; the strengths and weaknesses of the teaching and research training they received while in our program; their assessment of the department’s professionalization and mentoring efforts; and the major strengths and limitations of our graduate program. Students who were majors in the Director of Graduate Studies’ area were interviewed by the Department Chair or a member of the department’s Graduate Committee. The exit interview schedule is included at the end of this report in Appendix 3.

The purpose of the exit interview is to ask students if they were encouraged and given the opportunity to acquire the department’s desired learning outcomes. At the time of submission of this report, four students who completed the program during AY 2004-05 have been interviewed. It should be noted that this small number of interviewees matches that of last year’s interviews. There are three primary explanations for why only a minority of those who graduate participate in the exist interview. First, the interview is voluntary and it is becoming evident that many students believe they have already provided this information in the on-line graduate student survey (and to some extent, they are correct in this judgment). Second, interviews are not conducted until the student has formally completed the degree. This typically means the end of the semester, when students are busy meeting Graduate College deadlines, relocating to other parts of the state or country, or beginning a job. Finally, there is the possibility, although we think it very unlikely based on the anonymous on-line survey responses, that some students are reluctant to talk about program weaknesses and limitations in a face-to-face setting.

The four students interviewed this academic year (three M.A. and one Ph.D.) had overall positive evaluations of their training. All believed that our graduate program adequately prepared them for their current job or short-term career plans. The major strengths of our program as reported by the interviewees included the positive departmental atmosphere, availability of faculty and their willingness to help students, the opportunity to work on research projects with faculty, our professional development seminar, the rigorous statistical and research training we provide, the teaching of problem solving and writing skills, strong advising and mentoring, the cooperative atmosphere among students, the inclusion of students in faculty discussions of programmatic issues, and the autonomy and responsibility students are given in their teaching and research assignments. Noted weaknesses included insufficient preparation for teaching and relatively little supervision of graduate student syllabus development and teaching, the lack of training in qualitative research methods, the relative lack of preparation for job interviews, the failure to make students aware of the many existing data sets that are available for theses, dissertations and other research (the department does some of this, but needs to do more), the almost exclusive focus on preparation for academic careers with little attention paid to non-academic and applied careers, and the tendency for some students to be diverted to a teaching track while others follow an exclusive research track in their assistantship assignments.

Assessment of Graduate Student Teaching. The department’s Undergraduate Committee conducted evaluations of six of the eight graduate students who had sole responsibility for teaching one or more undergraduate courses this past year. The committee noted that it is difficult to generalize about the quality of teaching by our graduate students.

On the face of the scores from the end-of-semester student evaluations (the same forms used to evaluate faculty teaching in the department) from fall 2004, these instructors are generally well respected and liked in their classrooms. In all but four of the fifteen sections taught by graduate students, the numerical evaluation of overall instructor quality is higher than the departmental average (3.42). From the committee’s individual observational assessments of separate classes, these evaluations were often shown to be right on the mark. The majority of our graduate instructors is comfortable in the classroom and quite competently conveys their enthusiasm for the material and for their students. Most of these teachers are comfortable and proficient using a combination of lecture and discussion coordinated with overheads, black/whiteboard use, and/or power point slides.

This said, there are a few concerns. In several of the classes observed, the material that was covered came directly from the textbook assigned with no additional material or no added explanation. The latter is a particular concern, especially when sociological theory is being taught. In our experience, most undergraduates need a better grounding in theory than any undergraduate introductory text supplies. Further, when it becomes apparent to students that the material presented in class is identical to that in their texts they often respond in one of two ways: they either don’t come to class or they don’t read the text. This type of pedagogy may be one of the factors that has led to the current concern at the departmental level with the large distribution of the grades of A and B in sections of Soc 101, Principles of Sociology.

The two syllabi that that were examined by the committee were judged to be excellent. However, the committee suggests that instructors keep their own syllabi in mind as they progress through the course material in order to be sure they maintain the focus on their own vision of the course mandate. Faculty know from our own experiences in teaching that one pitfall is to focus on facts and examples from the text and fail to make sure students understand the relevance of these examples to the course and within the discipline of sociology.

Other substantive suggestions are diverse and generally relate to the fact that these students are just learning to teach. Several of these instructors need to be reminded to keep classes for the entire time allotted for each class meeting. They need to learn how to differentiate between students’ and instructors’ “opinions” and how to use each to illustrate and explain the specific material under discussion. Further, our graduate student teachers should focus a bit more on classroom discipline with the goal of eliminating cell phone use and newspaper reading at the back of large classrooms.

We are very proud of our graduate student teachers and with better mentoring from regular teaching faculty the department believes we can help these instructors to be more productive in the classroom, thus enhancing the reputation of our department as well as these individuals’ success on the job market.

Job Placement Survey. During the past six years 61 students earned graduate degrees from the BGSU Sociology Department. Numerous attempts were made during the spring 2005 semester to locate all of these students. Once located, graduates were asked to offer a brief evaluation of the training they received while in the department and to respond to several questions about their current place of employment. The discussion that follows is based on the 37 students who replied to our questionnaire.

Of the students responding, 22 graduated with M.A. degrees and 15 with Ph.D.s. As Table 1 indicates, Criminology had the most graduates, followed in almost equal numbers by the other three areas. Although most areas had more M.A. than Ph.D. respondents, in Social Psychology there was an equal division.

All students were asked if their current job was in the area they were trained in at BGSU. Over 83% indicated that this was the case. Among the Ph.D. students more than 93% indicated they were working the area they were trained, as only one individual indicated that this was not the case. Among M.A. students, however, nearly 24% indicated that they were not working the area they were trained.

Graduates were also asked to evaluate the quality of training that they received using a five point Likert scale that ranged from “excellent” (1) to “poor” (5). The mean score for all respondents was 1.6, somewhere between “excellent” and “very good.” More than 54% selected “excellent,” 30% “very good,” and the remainder “good.” When controlling for degree received, we again see some important differences. Among Ph.D. students only 47% said that their training was “excellent,” while among M.A. students the corresponding figure was 60%. Apparently this suggests that Ph.D. students are more demanding and/or critical than M.A. students.

Graduates also indicated their current professional activities. Many M.A. graduates are currently completing their Ph.D. degrees (35.1%). One M.A. graduate recently left the workforce to have a baby and one Ph.D. student is currently between jobs. All others are gainfully employed. Over 40% of the respondents are university teachers and/or researchers. The remaining 19% are working in non-academic professions. Only one of these latter individuals has a Ph.D. degree.

3. Inferences from Assessments

In general, the results of the AY 2004-05 assessment of our graduate program are quite gratifying. As was the case last year, all indicators suggest that we are accomplishing the learning objectives set forth for graduate education in sociology. The internal review of theses and dissertations suggests that our students are conducting top notch research and do a fine job of presenting the results of this research in written form. The graduate student survey and exist interviews confirm what those of us in the department have known for quite some time: the quality of our faculty and the learning atmosphere created and fostered by that faculty are important factors, probably the most important factors, influencing students’ choices to pursue their graduate education at Bowling Green. Our students get along well with one another and with the faculty. This makes for an environment that is conducive to both learning and top notch scholarship. Undergraduate student and faculty evaluations show that our graduate students are performing very effectively in the classroom as teachers. Finally, our job placement survey continues to show that the overwhelming majority of our graduates are working in areas for which they were trained and that the training they received in our program prepared them well for the marketplace, be it academic or non-academic.

4. Actions Taken/Program Improvements

The Department Graduate Committee will study this fall the assessment results reported herein and take the necessary actions, via recommendations to the graduate faculty, to shore up identified weaknesses and build on identified strengths. We will, for example, continue our steady progress of recruiting and funding only the strongest of students. In addition, we currently are addressing what are perhaps the biggest shortcomings, relatively speaking, of our program: teaching preparation and course offerings.

While we have offered a “Teaching Introductory Sociology” course for many years, it has become clear over the past few years that it was not, in its old form, fully achieving its intended goals. Consequently, we changed the structure and delivery of the course this past year and it is clear that the modifications have paid dividends. However, it also it clear that further fine tuning is necessary, and we will continue to do this in response to both student and faculty identified problems and gaps in teaching training.

One of the consequences of our success in securing extramural funding for our research is the effect faculty release time has had on the offering of graduate seminars. Both faculty and students are concerned with this issue, and even though some faculty have willingly taught seminars out of load, the problem persists. As a result, the department will discuss this coming fall the various alternatives we have to make certain we offer a variety of graduate level courses on a regular basis.

Beyond the two issues, the Graduate Committee will continue to make faculty aware of other program shortcomings and gaps identified through our assessment procedures. We believe that simple awareness of these issues is likely to result in their quick rectification. In addition, we can and will make both structural and substantive changes to addresses some of the limitations. For example, our Professional Development Seminar can devote more time to interview preparation, the chair and graduate advisor can redouble their efforts to make certain students get both teaching and research experience in their assistantship assignments, and all faculty can be more cognizant of the fact that some of our students seek non-academic and applied careers and adjust our courses and advising accordingly. Finally, although we will remain a quantitatively oriented department that trains our students accordingly, we will also continue to work at integrating training in qualitatively analysis into our courses.

APPENDIX 1

APPENDIX 2

APPENDIX 3

Back to top