Student Achievement Assessment Committee

Learning Outcomes:

The Sociology Department’s goals for doctoral students are that they realize the learning outcomes enumerated below prior to completion of the Ph.D. degree.  Each learning outcome is associated with a particular area of training.  For M.A. students, the department’s goals are that students develop learning outcomes A1-A5 (in major area of concentration only), C1-C2, and D2, D3, and D5.

A. Major and Minor Areas of Concentration

1. An understanding of major theories within area of concentration.
2. Knowledge of major and recent research within area of concentration.
3. An understanding of methods appropriate for research within area of concentration.
4. An ability to apply appropriate theories and methods to one’s own research within the area of concentration.
5. An appreciation of policy implications of major theories and research in the area of concentration.

B. Theory

1. An understanding of major sociological theoretical themes and perspectives.

C. Quantitative Methodology

1. An understanding of major research techniques and quantitative methodology.
2. An ability to design and implement a research program.

D. Professionalizations

1. An ability to design and effectively teach an undergraduate course.
2. An ability to effectively present research results orally, as in a seminar or at a conference.
3. An ability to effectively present research results in writing.
4. Personal experience with the process of submitting and publishing research articles.
5. Socialization into the profession. This can range from attending a conference, organizing a session, reviewing a journal article, to getting a job in one’s area of training.

Five different methods are used to assess the degree to which M.A. and Ph.D. students are successfully meeting departmental learning goals:

1. Annual review of selected theses and dissertations by two readers in the area of concentration and a reviewer selected from outside the sociology department.
2. Annual review of student vitae.
3. Teaching evaluations.
4. Annual graduate student survey and exit interview.
5. Job placement survey of graduates.

Annual Report

1.   Learning Outcomes Assessed this Year 

The Department of Sociology Graduate Program Assessment Plan calls for some of our assessment activities to take place during the fall semester of each year, others during the spring semester.  To this end, a number of assessment procedures were implemented during AY 2005-06: review of M.A. theses and Ph.D. dissertations; the annual assessment survey of continuing students; exit interviews with those completing our program at either the M.A. or Ph.D. level; evaluation of graduate student teaching; and the job placement survey of those having completed our program within the past six years. 

2.   Assessment Methods and Procedures

Review of Theses and Dissertations .  Toward the end of the spring 2006 semester, each masters thesis and Ph.D. dissertation completed during the previous calendar year was evaluated by departmental faculty committees in the students’ areas of concentration: criminology and deviance, demography, family sociology, and social psychology.  By design, none of the evaluators were the major advisors of the students.  The assessment included the reviewers’ ratings (on a 5-point scale from “excellent” to “poor”) of how well each learning outcome noted above (A1-A5, B1, C1-C2, and D3) was demonstrated in the student’s work. The evaluation form for theses and dissertations is included at the end of this report in Appendix 1. 

In addition to these internal evaluations, we typically enlist the services of reviewers at other universities to read and evaluate a sample of theses and dissertations completed during the academic year with respect to how well the department’s desired learning outcomes were exemplified in these works.  Usually, three theses and one dissertation, each representing one of our four areas of concentration, are sent out for review.  The external reviewers evaluate the theses and dissertations using the same evaluation form used by the internal readers. Unfortunately, we were unable to request the external reviews in a timely manner this year, and thus decided to forego them entirely.  However, we will return to this assessment mechanism next spring.

During calendar year 2005, three M.A. students and three Ph.D. students successfully defended their theses/dissertations in the area of Criminology/Deviance .  On average, the theses were rated by department faculty as “Excellent” or “Very Good” on all seven dimensions of our rating scale, with no dimension standing out as particularly strong or weak.  No raters evaluated any dimension less than “Good.” 

Mean ratings of the dissertations were slightly lower than the thesis ratings, although most dimensions were rated as “Very Good.”  The strongest ratings (“Very Good to Excellent”) were given for “knowledge of major recent research within area of concentration.”  The weakest ratings (but still averaging “Very Good to “Good”) were given for “understanding of methods appropriate for research within area of concentration.

The faculty ratings of theses and dissertations in Criminology/Deviance this year are remarkably similar to last year’s ratings.  Overall, the evaluations provide reassurance that our graduate students are acquiring the knowledge and skills taught in our program.  In producing theses and dissertations of the high quality represented in this year’s sample, graduates of our program have demonstrated that they are competent scholars of criminology and deviant behavior. 

During 2005, four students graduated from the department’s program with a specialization in some area of Demographic Studies . Three of these four earned M.A. degrees with concentrations in Applied Demography. The sole Ph.D. recipient specialized in Population Studies. In the opinion of both internal reviewers, the Ph.D. graduate did quite well as 10 of his 14 scores were excellent, the highest score possible and the remainder were in the very good or second highest category.  Furthermore, both internal reviewers suggested that this dissertation be nominated for the best dissertation of the year award. Unfortunately, that award is now only offered every two years, and as such, this candidate must wait two years to enter this challenge. 

All three M.A. candidates were evaluated by two internal reviewers.  The average score across the seven assessment dimensions ranged from “very good” to “good” to “fair.”  All three students showed in their work that they were well trained and found quick employment in government and industry as extremely desirable employees

During calendar year 2005, two students successfully defended dissertations in the Family Studies area of concentration. Average ratings from the internal evaluators ranged from “very good” to “excellent” to “good.” Areas that were rated most highly across both dissertations were “understanding of major theories within area of concentration,” “knowledge of major recent research within area of concentration,” and “understanding of methods appropriate for research within area of concentration.” The area receiving the lowest ratings across both works was “appreciation of policy implications of major theories and research in the area of concentration.” However, as most students in the department focus on basic, rather than applied, research, they are not as attuned to policy issues as they might be in a different academic milieu.  Overall, however, the raters were quite satisfied with both students’ work on their respective projects.

Two students earned degrees in the Social Psychology specialization in 2005.  One student earned her M.A. and another earned her PhD.  With respect to employment, one student is a long-term full time instructor at a technical college.  At this time, she does not have plans to pursue the Ph.D.  The other former student is a full time instructor at a prominent university, which specializes in her areas of interest.  While not tenure track, this student actively sought this position, and declined a tenure track position at a less prestigious university.

Regarding the M.A. thesis, both reviewers believe that the student’s use of theory was very good, and that she did a good job on incorporating recent social psychological research into her work.  The student’s use of appropriate methods, understanding of policy implications, and ability to design a research program were also rated in the very good-good range.

Regard the quality of the Ph.D. dissertation, again, reviewers rated the student’s use of theory and recent empirical research as very good.  The student’s use of methods was appropriate for the study, and she expressed a good understanding of the policy implications of her work. 

Annual Graduate Student Survey .  Students were given the opportunity to provide feedback regarding their graduate experience via an on-line graduate student survey that was conducted during the spring 2006 semester.  This survey was anonymous and was accessed via a secure Internet web site managed by Senecio Software in Bowling Green.  Students had two weeks to complete the survey.   Responses were submitted to the secure web site, and were subsequently tallied and summarized in the aggregate by an ad hoc faculty committee.  Item-by-item summary results are reported in Appendix 2 of this document.

A total of 27 current graduate students responded to a survey that assessed views and perspectives on the departmental climate, attitudes of the faculty toward students, graduate student climate, nature of faculty-student interaction, department chair, graduate advising (department and major area adviser), mentoring, courses, and adequacy of preparation for a career of teaching and research. The survey included a series of positive statements concerning each area, to which respondents could strongly agree (5) to strongly disagree (1).

Overall mean ratings of the departmental climate, the faculty, the graduate students, faculty-student interaction, graduate advising, and major area advising were comparable to or higher than in 2004 and 2005, and were quite positive. For example, ratings of faculty-student interaction, graduate advising, and major area advising, were 4.32, 4.69, and 4.51, respectively. These scores suggest that students, on average, agreed to strongly agreed that the department was performing its mission well in these areas. Overall mean ratings of the departmental climate, the faculty, the courses offered, and student preparation for their careers were somewhat lower, but still quite acceptable. For example, the lowest of these ratings were 3.83 for student preparation and departmental climate. This suggests that students mostly agreed that they were being adequately prepared for the job market and that the departmental climate was positive. The overall rating of the departmental chair dropped, compared to the two previous years (3.67, versus 4.46 and 4.53). However this was most likely influenced by the temporary change in the chairship over that period, and its attendant disruption in departmental routines.

Students also had the opportunity to describe, through an open-ended format, those aspects of the program that they felt were the strongest, as well as to provide their own descriptions of areas of change they would like to see in the department, as well as in the training they receive. The positive open-ended comments emphasized faculty-student relationships, the strengths of each substantive area and the quantitative training offered in the department, the high quality of faculty availability and mentoring, and the quality of the faculty itself. Although students generally agreed that the courses taken are a good preparation for preliminary exams, and are good learning experiences, one area of concern related to the availability of courses offered each semester.  The department was also perceived as sensitive to gender issues, but scores for racial/ethnic diversity were not as uniformly favorable.  Faculty were given high marks in general, but there was less than perfect agreement that “faculty give students equal opportunity for research.” This suggests the need for additional attention to strategies that ensure that all graduate students have access to a variety of research experiences within the department. Finally, several students expressed the desire for more training in qualitative methods.  However, our new hire in social psychology, who will join us in the fall, is well trained in this area and has expressed an interest in offering such a course.

Exit Interviews .  Upon exiting the program (i.e., subsequent to defending the dissertation or terminal thesis, or leaving the program without a degree), students were interviewed by the Director of Graduate Studies for the purpose of gathering information on a variety of issues: whether our program has prepared them for the job world or subsequent graduate education; the strengths and weaknesses of the teaching and research training they received while in our program; their assessment of the department’s professionalization and mentoring efforts; and the major strengths and limitations of our graduate program.  Students who were majors in the Director of Graduate Studies’ area were interviewed by the Department Chair or a member of the department’s Graduate Committee.  The exit interview schedule is included at the end of this report in Appendix 3.

The purpose of the exit interview is to ask students if they were encouraged and given the opportunity to acquire the department’s desired learning outcomes.  At the time of submission of this report, six students who completed the program during AY 2005-06 have been interviewed.  It should be noted that this small number of interviewees matches that of last year’s interviews.  There are three primary explanations for why only a minority of those who graduate participate in the exist interview.  First, the interview is voluntary and it is becoming evident that many students believe they have already provided this information in the on-line graduate student survey (and to some extent, they are correct in this judgment).  Second, interviews are not conducted until the student has formally completed the degree.  This typically means the end of the semester, when students are busy meeting Graduate College deadlines, relocating to other parts of the state or country, or beginning a job.  Finally, there is the possibility, although we think it very unlikely based on the anonymous on-line survey responses, that some students are reluctant to talk about program weaknesses and limitations in a face-to-face setting.

The six students interviewed this academic year (five M.A. and one Ph.D.) had overall positive evaluations of their training.  All believed that our graduate program adequately prepared them for their current job or short-term career plans. 

The major strengths of our program as reported by the interviewees included the positive departmental atmosphere, opportunities to gain teaching experience, excellent research experience and opportunities to work with faculty on their projects, our professional development seminar, rigorous training in research methods and statistics, excellent faculty, accessibility of faculty, helpful and cooperative student peers, strong advising and faculty mentoring, strong support system from other students and faculty, solid training in the basics of the discipline, training in data analysis that impresses potential employees, internship opportunities, and the many services and opportunities provided by the CFDR.

Noted weaknesses included insufficient preparation for teaching and relatively little supervision of graduate student syllabus development and teaching, the lack of opportunity to collaborate with students in other departmental areas as well as those in other disciplines, poor computer maintenance in the student lab, lack of sufficient physical space for work and study, the infrequency of some course offerings, the dearth of  summer classes, a need for better communication between faculty and students, lack of support for students who want to pursue teaching rather than research careers, lack of training in qualitative research methods, lack of SAS training early in program, and the belief that mentorship of students is not taken seriously enough by some faculty who appear to view students as employees rather than junior colleagues.

Assessment of Graduate Student Teaching .  The department’s Undergraduate Committee conducted evaluations of six of the eight graduate students who had sole responsibility for teaching one or more undergraduate courses this past year.  The committee notes that it is difficult to generalize about the quality of our graduate students in the classroom. 

From the Fall 2005 semester, we had faculty evaluations of five of the ten graduate students who had sole responsibility for teaching one or more undergraduate courses.  From the Spring 2006 semester we had letters for two of seven.

These evaluations indicate that our student instructors are generally doing an excellent job in the classroom and that they will develop into superb teachers.  Laudatory comments are common.  One student teacher was “clearly invested in lecturing and quite energetic” as she moved about the room and “encouraged student participation.”  Another student teacher was told “I’m really at a loss at this point to make any recommendations as to how you might improve on your performance. . . . you’re doing an excellent job of teaching. . . . your classroom has a nice feel to it.”  Another comment: “the instructor was well-prepared and engaging, the students thoughtful.”  Another instructor was “urge[d] to do more of what he is doing.  Students were clearly motivated to discuss the theories being presented.”  Many evaluations noted the successful use of humor to teach as well as to lighten the classroom mood occasionally.

A few problems are addressed in the in-class evaluations.  Two of the classes evaluated ended five to ten minutes early.  At least one of our young teachers overly relied on lecture in Soc 101; this is not surprising, however, because most of these sections enroll at least 60 – 80 students and it takes time to develop the skills necessary to catalyze successful discussions or even participation in classes of this size. Another graduate teacher had trouble maintaining order in the back of the lecture/theater style rooms on the second floor of Olscamp Hall.  Several suggestions were made to this instructor including walking up and down the side isles while lecturing or calling on students, and giving small graded in-class assignments.  Another instructor was advised to cover fewer topics in greater depth during a 50-minute class session.

The end-of-semester student evaluations support our belief that these instructors are well respected and liked in their classrooms.  The range in aggregate instructor scores is 3.87 to 3.25 for the Fall 2005 semester and 3.82 to 3.19 for the Spring 2006 semester. The high scores represent an excellent achievement and are comparable to the scores achieved by the best tenured professors in the department. While we are concerned about the lowest scores, we note that they are in relatively large sections of Soc 101 which are a challenge to even well seasoned teachers.
We are very proud of our graduate student teachers and with better mentoring from regular teaching faculty we can help these instructors to be more productive in the classroom thus enhancing the reputation of our Sociology Department as well as these individuals’ success on the job market.

Job Placement Survey . During the past six years 68 students earned graduate degrees from the BGSU Sociology Department.  Recently numerous attempts we made to locate all of these students.  63 were contacted by e-mail.  Once located, on February 7, 2006 graduates were asked to complete a brief evaluation of the training they received while in this department and to respond to several questions about their current place of employment.  An individual email follow-up was sent on February 16, 2006.  As of February 28, 2006, 49 graduates responded to our survey (~76%).  The discussion that follows will be based on the 48 students who did reply to our questionnaire.

Of the students responding 31 graduated with M.A. degrees and 18 with Ph.D.s. As Table 1 indicates, Criminology had the most graduates, followed by demography, social psychology, and family.  All areas had more M.A. than Ph.D. respondents, and criminology had the most Ph.D. respondents.

All students were asked if their job was in the area they were trained in at BGSU.  Over 85.7% indicated that this was the case.  Among the Ph.D. students 100% indicated they were working the area they were trained.  Among M.A. students, however, 76.7% indicated they were working in the area they were trained.

Graduates were also asked to evaluate the quality of training that they received using a five point Likert scale that ranged from excellent (1) to poor (5).  The mean score for all respondents was 1.73, somewhere between excellent and very good.  About 47% selected excellent, 36.7% very good and the remainder good or fair (2 respondents reported “fair”). Among Ph.D. students only 44% said that their training was excellent, while among M.A. students the corresponding figure was 46.7%.  The corresponding percentages from the 2005 survey were 47% and 60%.  The drop in ratings may reflect a change in the perceptions of the quality of the MA-level training received at BGSU.  More likely, however, is that the change derives from the fact that the 2006 survey had a better response rate than the 2005 survey.  In 2005, 15 Ph.D. students responded (compared with 18 in the 2006 survey) and many fewer MA students responded—only 22 compared with 31 in the 2006 survey. 

Graduates also indicated their current professional activities.  Many M.A. graduates are currently completing their Ph.D. degrees (13 out of 33).  Four MA students are instructors at a university or college, and 12 are working in an applied job (such as in a marketing firm or for the Census Bureau).  Only one is not currently employed.  Among the 18 Ph.D. graduates, 14 are assistant professors at colleges or universities and the remaining 4 are instructors working in an academic institution.

Inferences from Assessments

In general, the results of the AY 2005-06 assessment of our graduate program are quite gratifying.  As was the case in previous years, all indicators suggest that we are accomplishing the learning objectives set forth for graduate education in sociology.  The internal review of theses and dissertations suggests that our students are conducting top notch research and do a fine job of presenting the results of this research in written form.  The graduate student survey and exist interviews confirm what those of us in the department have known for quite some time: the quality of our faculty and the learning atmosphere created and fostered by that faculty are important factors, probably the most important factors, influencing students’ choices to pursue their graduate education at Bowling Green.  Our students get along well with one another and with the faculty.  This makes for an environment that is conducive to both learning and top notch scholarship.  Undergraduate student and faculty evaluations show that our graduate students are performing very effectively in the classroom as teachers.  Finally, our job placement survey continues to show that the overwhelming majority of our graduates are working in areas for which they were trained and that the training they received in our program prepared them well for the marketplace, be it academic or non-academic.

 Actions Taken/Program Improvements 

The Department Graduate Committee will study this fall the assessment results reported herein and take the necessary actions, via recommendations to the graduate faculty, to shore up identified weaknesses and build on identified strengths.  We will, for example, continue our steady progress of recruiting and funding only the strongest of students.  Along these lines we will, for example, continue sending information on our programs to the 86 Alpha Kappa Delta (the undergraduate sociology honorary society) advisors in the surrounding 6-state region.  We have for many years sent general information about our programs to approximately 1,600 sociology departments nationwide, but found that last year’s more detailed and personal AKD mailings resulted in an increase in student inquires and applications that were directly attributable to these mailings. 

In addition, we currently are addressing what are perhaps the principal shortcomings of our program: teaching preparation and course offerings. While we have offered a Teaching Introductory Sociology course for many years, it has become clear that it was not, in its old form, fully achieving its intended goals.  Consequently, we changed the structure and delivery of the course this past two years and it is clear that the modifications have paid dividends.  However, it also it apparent that further fine tuning is necessary, and we will continue to do this in response to both student and faculty identified problems and gaps in teaching training.  To provide further teacher training, we began this past year assigning faculty mentors to our Ph.D. students who have full grade book responsibility for the courses they are teaching.  The faculty mentors advise the student teachers on matters such as syllabus construction, techniques to facilitate classroom discussion, and how best to respond to such problems as classroom disruptions.  The faculty mentors also visit the classroom and write evaluation reports that are useful to our young teachers as they work to hone their skills in the classroom.

One of the consequences of our success in securing extramural funding for our research is the effect faculty release time has had on the offering of graduate seminars.  Both faculty and students are concerned with this issue, and even though some faculty have willingly taught seminars out of load, the problem persists.  As a result, the department will discuss this coming fall the various alternatives we have to make certain we offer a variety of graduate level courses on a regular basis.  Such discussion are simultaneously occurring within our four areas of departmental concentration to make certain that student are receiving the training we have promised and that they deserve.

Beyond the two issues, the Graduate Committee will continue to make faculty aware of other program shortcomings and gaps identified through our assessment procedures.  We believe that simple awareness of these issues is likely to result in their quick rectification.  In addition, we can and will make both structural and substantive changes to addresses some of the limitations.  For example, our Professional Development Seminar can devote more time to interview preparation, the chair and graduate advisor can redouble their efforts to make certain students get both teaching and research experience in their assistantship assignments, and all faculty can be more cognizant of the fact that some of our students seek non-academic and applied careers and adjust our courses and advising accordingly.  Finally, although we will remain a quantitatively oriented department that trains our students accordingly, we will also continue to work at integrating training in qualitatively analysis into our courses.

Appendices to the 2006 Sociology Graduate Program report are included in the downloadable version of the complete report. Click here to download the full report.