Student Achievement Assessment Committee

The purpose of the ACS graduate student assessment plan is to aid our students in successfully completing their program of study and gaining the skills they need to successfully compete on the academic job market.  The plan will help us to ensure this outcome by providing, (1) several objective measures we can use to determine the success or failure of our students in the particular activities we have identified as essential to their success, (2) opportunities for us to review our students’ performance in their graduate coursework and to convey to our students the faculty’s assessment of their strengths and weaknesses along with suggestions for how to improve their performance, and (3) opportunities for our students to give us their assessment of the program, which will help us in continuing to review and  fine-tune the structure of the ACS graduate program.

Each learning outcome is stated below, followed by the measure achieved for this learning outcome during the 2005-06 academic year.  We will conclude with a short conclusion evaluating this year’s graduate student assessment.  

Learning Outcome:  Students will demonstrate mastery of their academic field.

Measure:  Preliminary examination for 3rd year doctoral students: 90% will pass examination.

Outcome:  The ACS Ph.D. preliminary examination was given in October 2005 to all seven students in the third-year doctoral student cohort.  All seven students passed the examination, although one student was required by the examination committee to rewrite the second portion of his examination, delaying his completion of the exam until January 2006.

Measure:  Comprehensive examination for 1st year MA students: 90% will pass examination.

Outcome:  Five MA students took the MA Comprehensive examination during the 2005-06 academic year, choosing to complete the Plan II non-thesis route to the MA degree.  All five students successfully passed the MA Comprehensive exam.

Discussion:  Our students met our achievement outcomes for both the doctoral preliminary and MA comprehensive exams, in both cases exceeding the benchmark criteria established by the program faculty.  We conclude that the curriculum of the ACS MA and Ph.D. programs are adequately preparing students to pass their comprehensive/preliminary examinations, thus displaying mastery of their academic
field.

Learning Outcome:  Students will demonstrate the ability and motivation to participate successfully in professional activities in their academic field.

Measure:  MA students—in each academic year, 50% of full-time second year students will present a paper at an academic conference.

Outcome:   In the academic year 2005-06, 7 second-year MA students presented papers at an academic conference, out of a total of 14 full-time second-year students—50%.

Measure:  Ph.D. students—in each academic year, 50% of full-time doctoral students (years 1-4) will present a paper at an academic conference.

Outcome:  In the academic year 2005-06, 24 Ph.D. students presented papers at an academic conference, out of a total of 36 full-time doctoral students—67%.

Measure:  Ph.D. students—by the end of their 4th year of full-time study, 100% will have presented one paper at an academic conference; 75% will have presented two papers; 50% will have presented three or more papers.

Outcome:  Below is a list presenting the total number of academic conference presentations compiled by our seven fourth-year doctoral students during their four years of study in the ACS Ph.D. program.  The number of conference presentations is listed for each individual student:

Student 1:         0
Student 2:         2
Student 3:         3
Student 4:         5
Student 5:         5
Student 6:         6
Student 7:         9

In summary, 86% of fourth-year ACS doctoral students have presented at least one conference paper during their time in the program; 86% have presented at least two papers; 72% have presented at least three papers.

Measure:  Ph.D. students—by the end of their 4th year of full-time study, 50% will have published an article in a professional academic venue (had an article accepted for publication in an academic journal—either paper or on-line—or a book chapter/essay in an edited volume (excluding book reviews). 

Outcome: Below is a list presenting the total number of academic publications compiled by our seven fourth-year doctoral students during the four years of study in the ACS Ph.D. program.  The number of academic publications (either in print or in press) is listed for each individual student:

Student 1:         0
Student 2:         0
Student 3:         2
Student 4:         2
Student 5:         2
Student 6:         2
Student 7:         3

In summary, 72% of fourth-year ACS doctoral students have published at least one academic journal article or book chapter during their time in the program—and in fact all 72% of doctoral students have published at least two articles/book chapters.

Discussion:  Our MA students met our expectations for professional conference activity, but just barely, with 50% of second-year students not presenting a conference paper during the academic year.  Our doctoral students, however, far exceeded our expectations for professional conference activity, with two-thirds of doctoral students presenting conference papers this year.  Four ACS doctoral students presented two papers during the year; two students presented three academic conference papers.  Additionally, we did not include presentations at the ACS Graduate Student Conference, held on the BGSU campus March 31-April 1, in this count.  Eighteen ACS graduate students presented papers at this conference.  

Our forth-year doctoral students, completing their eligibility for graduate assistance, also demonstrated a level of professional activity in terms of both conference activity and academic publication that exceeded our expectations, with multiple academic conference presentations and more than one academic publication during doctoral study proving to be the norm for this group of students.  In general, we conclude that both ACS MA and Ph.D. students have demonstrated the ability and motivation to participate successfully in professional activities appropriate to their level of academic study.

Learning Outcome:  Students will demonstrate the ability to participate successfully on the professional/academic job market.

Measure:  Ph.D. students—within one year of completion of degree, 50% of graduates will have successfully taken appropriate full-time professional employment; within two years of completion of degree, 75% of graduates; within three years of completion of degree, 100% of graduates.

Outcome:  Each of the last four years in which doctoral students completed their Ph.D. degrees is listed below, along with the statistics for current employment.

2005:  4 or 6 students completing the Ph.D. degree in this year are in full-time professional employment (66%); 3 of these jobs are tenure-track academic positions.

2004:  8 of 9 students completing the Ph.D. degree in this year are in full-time professional employment (88%); 4 of these jobs are tenure-track academic positions; another one is an academic librarian position.

2003:  4 of 4 students completing the Ph.D. degree in this year are in full-time professional employment (100%); 1 of these jobs is a tenured academic position.

2002:  6 of 6 students completing the Ph.D. degree in this year are in full-time professional employment (100%); 4 of these jobs are tenure-track academic positions; another one is an academic librarian position.

Measure:  Ph.D. students—within three years of completion of degree, 25% of graduates will have received a book contract to publish their dissertation.

Outcome:  Four students from the three recent graduating classes of 2002, 2003, and 2004 have received a book contract to publish their dissertation.  The four presses are: University of Wisconsin Press, Indiana University Press, University of Philippines Press, and Duke University Press.  One book is out:  Ilana Nash, American Sweethearts: Teenage Girls in Twentieth Century Popular Culture, Indiana University Press, January 2006.  A fifth former student’s dissertation is under advanced review by the University of Illinois Press.  This is out of a total of 19 graduates from this period—27%.

Discussion:  In general, we are pleased with the success our doctoral graduates have had in securing appropriate professional employment.  The data presented above exceed our minimal expectations for our students in the academic job market.  The one area of concern that we have is that only about one-half of our graduates for the four years under review have secured a tenure-track position at an academic institution.  Tenure-track academic work is the goal of our graduate placement, and we would like to improve this statistic over the next few years.

Closely tied to success in the academic job market is success in the publication field.  We would note that all four of our students who have successfully published their dissertations in academic presses are currently in tenure-track academic positions.  We intend to continue to encourage our doctoral students to attempt to publish their dissertations as an integral part of a successful academic career pattern.

Learning Outcome:  Students will demonstrate timely completion of their degree.

Measure:  MA students—by the end of their 2nd year of full-time study, 90% of continuing students will have completed their degree.

Outcome:  Out of the 18 students in the 2003 ACS MA entering class, 17 students have now completed the MA degree; the one remaining student is making final revisions in her MA thesis and is expected to graduate in December—a 94% graduation rate to date, expected to reach 100% by the end of Fall semester 2006.

Measure:  Ph.D. students—by the end of their 4th year of full-time study, 30% of continuing students (students who have completed all course work and attained ABD status) will have completed their degree; by the end of their 5th year of study, 60% will have completed their degree; by the end of their 6th year of study, 90% will have completed their degree.

Outcome:  For the year of each entering ACS doctoral class, the number of students in each entering class is given, followed by the number of students in that class who completed all course work and passed their preliminary examination, attaining ABD status, finally followed by the number of students who have completed their dissertations and graduated.  Finally the percentage for each entering class of ABD students who have graduated is indicated.

2001 entering class (12): 10 ABD, 6 Ph.D. (60%)
2000 entering class (11): 10 ABD, 7 Ph.D. (70%)
1999 entering class (7): 3 ABD, 2 Ph.D. (66%)
1998 entering class (6): 5 ABD, 5 Ph.D. (100%)

Discussion:  As part of the ACS program review internal report, which is being written during summer 2006, we have reviewed graduation rates for our MA and Ph.D. programs for the last ten years.  The high graduation rate of the 2003 MA entering class is no anomaly.  The ACS MA program has historically enjoyed high graduation rates of 80% or higher.  Our goal to elevate these rates to 90%+ for each graduating class is a realistic one.

The situation for the ACS doctoral program is more complicated because often doctoral students delay completion of their dissertation and graduation for seven or more years into their doctoral program.  We are very serious about reducing the time to degree and improving the graduation rate for ACS doctoral students, and these statistics indicate that we have made improvements in the graduation rates of recent entering classes, which have historically hovered around the 60% mark. 

We note the discrepancy between the number of students entering the doctoral program each year and the number achieving ABD status.  In our view, it is normal for one or two students to drop out of each entering doctoral class, as a result of changing goals, failure to satisfactorily complete degree requirements, etc.  We would rather see students who will be unable to complete the requirements for the doctoral degree drop out of the program early in their graduate studies rather than later.  This said, the unusually high drop out rate for the 1999 class we hope will prove to be an anomaly. 

Ongoing Assessment Activities

Assessment Activity:  MA students—All MA students are asked to provide a written evaluation of their experience in the ACS MA program at the time of graduation (evaluations will be collected and considered by the ACS MA Executive Committee).  Dimensions of assessment will include: appropriateness and quality of: coursework, academic advising, professional development, thesis mentoring (where appropriate), preparation for job market/further graduate study.

Outcome:  In late May 2006, an ACS Graduate Program Evaluation document was sent to the eighteen 2005-06 academic year graduates of the ACS MA program.  To date, we have not had any of the evaluations returned.  Next year we will attempt to find a better way of conducting this portion of our assessment, as our graduates’ evaluation of the ACS program is seen by us as providing important information for future program development.

Assessment Activity:  Ph.D. students—In January of each year, the ACS Ph.D. Executive Committee will conduct a review of each 1st-year doctoral student’s progress during their first semester of full-time study.  This review will be held in consultation with other ACS faculty members, especially those faculty members who have taught these students in their required first-semester classes.  The purpose of this review will be to assess the perceived strengths and weaknesses of each student at this early stage in their doctoral study.  The director of the ACS program will meet individually with the students to convey the substance of the committee’s review to them and to suggest any remedial action that may be deemed appropriate at that time (activity initiated in January 2004). 

Outcome:  For the second year the ACS Ph.D. Executive Committee conducted this mid-year evaluation of first-year doctoral students.  In general, the committee felt that students were performing at an acceptable level.  One student was identified with some deficiencies in skill levels, a second student with some deficiencies in background in cultural studies.  The ACS director met with both students and suggested additional course work and work in these areas of deficiency.  Both students have shown improvement in the areas of concern during their Spring 2006 semester course work.

Assessment Activity:  Ph.D. students—A mid-second year assessment of each student’s professional development will be written by the professor of the ACS 745-Publication and Professional Development seminar, a required class for all second-year doctoral students.  This report will assess each student’s professional development to that point.  Copies of the report will be provided to the student, the ACS director, the ACS Ph.D. Executive Committee, with a copy in each student’s file.

Outcome:  This assessment will be initiated in Fall 2006.

Assessment Activity:  Ph.D. students—All Ph.D. students are asked to provide a written evaluation of their experience in the ACS Ph.D. program at the time of graduation (evaluations will be collected and considered by the ACS Ph.D. Executive Committee). Dimensions of assessment will include, appropriateness and quality of: coursework, academic advising, professional development, dissertation mentoring, and preparation for professional/academic job market.

Outcome:  In late May 2006, an ACS Graduate Program Evaluation document was sent to the nineteen recent graduates of the ACS MA program (2004-06).  To date, only two of these evaluations have been returned.  Both graduates returning surveys expressed strongly positive evaluations of the program, but this number is too small to allow us to draw any conclusions.  We will continue to attempt to encourage our doctoral graduates to fill out and return this program evaluation, and next year we will attempt to find a better way of conducting this portion of our assessment, as our graduates’ evaluation of the ACS program is seen by us as providing important information for future program development.

Actions Based on Assessment Findings

Although this is our first year of graduate program assessment, we have anticipated the two areas of concern revealed in this program assessment—timely completion of degree (for doctoral students) and successful participation by our doctoral graduates on the academic job market—and have taken several steps over the last two years to address these issues. 

Actions to Address Timely Completion of Degree:  We have noted with some concern the historic failure of many ACS doctoral students to complete their Ph.D. degree in a timely fashion.  The Graduate College places a ten-year limit for completion of all doctoral degrees from date of initial enrollment in program.  Individual ACS graduate students have often stretched this time period to its limits in delaying the completion of their doctoral dissertations, and thus delaying and often seriously harming the initiation of their academic careers.  We have identified a four to six year period from initial enrollment to graduation as optimal, and have initiated two major changes in the ACS doctoral program to encourage more timely completion of the degree. 

The first of these changes has been the transition from an individually structured Ph.D. Preliminary Examination administered by each student’s dissertation committee to a common preliminary exam administered by the ACS program.  The Ph.D. program is structured so that students should complete all course work by the end of their second year of full time study.  In the past, doctoral students would in some cases postpone taking their preliminary examination until the beginning or even the end of their fourth year of full-time study.  This exam is now administered to all third-year doctoral students in October of each year, at the beginning of their third year of full-time study.  The first common preliminary exam was given in October 2004, the second in October 2005.  All students passed the 2004 exam (seven), although one student withdrew from the program just prior to the examination date; all students but one passed the 2005 exam (six), with the seventh student eventually passing the exam after being required to retake the second half of the exam.

The second change to encourage more rapid completion of degree was initiated in the Spring semester 2006.  All third-year doctoral students were required to take a dissertation-writing workshop, taught by the ACS program director.  This course was, in our view, a great success.  By the end of the semester, two students had better than half of their dissertations written in first draft, with most of this material also run through a second draft; four additional students had produced at least two dissertation chapters, with at least one chapter in second draft; the final student had completed and defended his dissertation proposal, with some additional material written in first draft.  It should be noted that better than half of these students had already successfully passed their dissertation proposals before the beginning of the Spring semester (and the dissertation writing workshop) in January.

As a result of these two program changes, we anticipate that this 2005-06 class of third-year doctoral students (the 2003 entering class) should all have completed their dissertations and graduated by the end of their fifth year of full-time study, with half having completed their dissertations and graduated by the end of their fourth year of full-time study. 

Actions to Address Successful Participation of Graduates on the Academic Job Market:  The most important steps to be taken to improve our graduates’ ability to secure good academic positions are in the areas of improvement of graduate training and student recruiting.  The reorganization of the ACS doctoral program over the last five years (creation of the two interdisciplinary tracks in Ethnicity/Gender and Film/Media, etc,) has been carried out with the primary goal of improving our graduates’ position in the academic job market.  We will need several more years before we can adequately determine the success of this strategy.

In the short term, we have taken several steps aimed at improving the competitiveness of individual ACS doctoral students as they enter the job market.  We have added job market-related modules to two of the required courses for our doctoral students (ACS 745-Publication and Professional Development—required for all second-year doctoral students—and ACS 782-Dissertation Workshop—required for all third-year doctoral students) on such topics as preparing a resume/curriculum vita, writing a letter of application, preparing a teaching portfolio, preparing for a job interview, applying for non-academic employment, etc.).  We encourage advanced doctoral students to prepare their academic job talk and test it well ahead of any job interviews with an audience of ACS faculty members and graduate students. 

Additionally, last Fall semester, ACS faculty member Madeline Duntley organized an afternoon-length workshop on academic job application and interview training.  This workshop was, in our view, very successful.  We intend to offer a similar, somewhat more elaborate seminar early in the Fall 2006 semester, which we hope will be attended by all advanced ACS doctoral students.  Again, while academic training, quality of dissertation, and professional development experience in the areas of successful teaching, conference presentation, and publication are the most important criteria for successful participation on the academic job market, training and preparation in the more mundane areas of vita preparation, letter writing, and proper preparation for an academic interview and job talk can be crucial and even decisive for individual doctoral students.