Student Achievement Assessment Committee

Contents of This Report
• Introduction
• Phase I: Revision of Learning Outcomes
• Phase II: Development of Three-Year Assessment Plan
• Phase III: Implementation of Year 1 Plan
• Phase IV: Recommendations and Actions

Introduction

The following observations or assumptions guide our assessment planning and execution

• Marketing specialists are BSBA majors.
Marketing students are BSBA majors who specialize in Marketing. Our Marketing learning outcomes are designed to complement the BSBA outcomes and assessment.

• Focus on what is realistic.
We have a limited window in which to influence Marketing specialists. Most Marketing specialists take their first Marketing course in their Junior year. Our learning outcomes are aimed toward conveying the knowledge and skills upon which we believe we can have some effect in the 21 hours we have to influence them.

• Life-long learning.
Our Departmental Mission includes a commitment to lifelong learning. The macroenvironments that affect all marketing decisions are highly dynamic. As a result, marketing practices are in constant flux. We believe that we should offer students opportunities to learn basic concepts, frameworks, to learn to ask certain questions as opposed to having all the answers, and to acquire certain skills that transcend the increasingly high rate of change in the business world and other types of organizations that utilize marketing.

• Need for a flexible framework.
In our courses we aim to overcome the misconceptions that Marketing is just advertising used by profit-oriented, business organizations. Instead, Marketing is applicable to any organization that wishes to reach and influence a group of people with their product, service, or idea. We need to offer students the opportunity to understand a basic framework that will apply in a variety of settings. This includes influencing particular behaviors (e.g., voluntary blood donation, anti-smoking programs, university promotion).

Phase I: Revision of Learning Outcomes

The Marketing Framework

Our original learning outcomes, established several years ago, had proven themselves to be difficult to work with for assessment purposes. The current Undergraduate Curriculum Committee developed a "Marketing Framework" and then developed and proposed a revised set of learning outcomes. These were approved by the department faculty in 2000.

Our knowledge learning outcomes flow from a basic concept known as customer value. The model that follows shows the centrality of the customer value concept and how marketing fundamentals (and coursework) lead back to this central concept. Our learning outcome statements listed below articulate the relationships depicted in the model.

Learning Outcomes:

1. Marketing students will communicate a working knowledge of the basic marketing framework (see model). In particular, marketing students will demonstrate an understanding of the following.

a. Information acquisition, management, and use;
b. Customer analysis; and
c. Marketing mix design and marketing strategy development for purposes of creating customer value leading to a sustainable competitive advantage.

2. Marketing students should be able to demonstrate the ability to:

a. Work effectively as a member of a larger team.
b. Communicate clearly and concisely in both written and oral form.
c. Develop and evaluate creative solutions, while solving problems.

Example Topics:

Phase II: Three-Year Assessment Plan

3-YEAR ASSESSMENT PLAN

Purpose: to develop a multi-year plan for assessing our learning objectives (see attached list).

Goal of Assessment: to improve our program for marketing specialists, based on learning outcomes. Assessment happens when a team of faculty draw useful conclusions and make suggestions for program improvement. For example, assessment is like the results and discussion/conclusion section of a manuscript. Just collecting the data is not assessment.

Criteria for selecting assessment methods:

• realistic; minimizes faculty time required to implement and uses assistance from GA’s, secretarial staff, and other services (e.g., data processing).
• does not place undue burden on a few faculty, but involves everyone.
• whenever possible, uses activities or assignments students are already engaged in as part of a class. Does not place undue burden on MKT 460, our capstone course, as the logical data collection point.

Overview of Proposed Assessment Plan

Explanation of Methods

Oral Communication Skills Assessment. (Once every 3 years)
Our learning outcomes state that we expect students to demonstrate oral communication skills. Oral presentation skills is a subset of oral communication (listening, articulating, etc.).

Method: senior Marketing specialists will be asked to record a short (e.g, 10 minutes) presentation on video tape. A sample of these tapes will be rated. A panel of at least three judges will rate each students’ presentation skills, using the Oral Presentation Evaluation form our committee has developed. Assessment will include the search for specific areas of oral presentation skills on which students are especially strong or weak. Program recommendations can follow. We will conduct this assessment every 3 years for comparisons across time.


Alumni Survey
Method: survey mailed (or sent electronically) to recent alumni (between 1 and 5 years out) to obtain their perceptions of the strengths and weaknesses of the Marketing program. Depending upon the size of the data base at the time, it may make sense to take a sample of the population.


Knowledge Application Assessment (Once every 3 years)
This assessment refers to our list of knowledge learning outcomes (see attached). This method is not dependent upon the content of those outcomes; if we change the outcomes, the basic method can remain the same.

Method: Individual written Case Analysis required in 460. To be evaluated by a panel of at least three faculty members. The panel would identify strengths and weaknesses of students’ knowledge and application. The Curriculum Committee will need to develop a set of criteria and a rating form to be applied. We are recommending a change in the learning objectives. Once we settle on learning objectives, then we can proceed with this assessment method.


Senior Surveys (at least once a year; can be each semester)
Method 1: CBA Graduating Senior Survey
Method 2: Department of Marketing Graduating Senior Survey

These surveys obtain students’ perceptions of program quality, skill coverage, etc. Both surveys are already in place. They require a relatively small amount of MKT faculty time to implement and assess the results. A graduate assistant can be asked to run statistical analyses on the data, if desired.


Student Learning Survey (once every 3 years)
Method: survey MKT 460 instructors to obtain perceptions of strengths and weaknesses of marketing specialists. Areas covered: all learning objectives.


Method A: Student Learning Survey

Purpose: to survey Marketing 460 (senior capstone course) instructors to obtain their perceptions of strengths and weaknesses of marketing specialists with respect to their knowledge and skills described in the Department of Marketing Learning Outcomes. All learning outcomes were covered in this survey.

Procedures: by means of a questionnaire using open-ended questions, instructors of MKT 460 (Summer 2000, Fall 2000, and Spring 2001 instructors) were queried with respect to the students in their most recent (or current) class. Each instructor described, in writing, his or her observations regarding students' knowledge and skill-related development for each of our learning outcomes.

Analysis: responses to each question were compiled (see below). Members of the Curriculum Committee studied the responses and identified themes with respect to each learning outcome. These results were presented for discussion to the department faculty in April 2001.

Results: described below for each learning outcome.

Analysis: responses to each question were compiled. Members of the Curriculum Committee studied the responses and identified themes with respect to each learning outcome. This information was reported to the department faculty for discussion.

Results of Survey

Learning Outcome 1a: Students’ understanding of information acquisition, management, and use for purposes of creating customer value leading to a sustainable competitive advantage.

Students’ Strengths:
• Seem to understand information acquisition
• Decent computer skills
• Great improvement in getting more information, especially from Internet sources

Students’ Weaknesses:
• Know little about managing and using information.
• Knowing where/how to get information
• Knowing how to interpret/use once they do get information
• Understand the difference between analysis and description
• Lack the ability to digest and synthesize the obtained information

Learning Outcome 1b: Students’ understanding of customer analysis for purposes of creating customer value leading to a sustainable competitive advantage.

Students’ Strengths:
• Understand customer analysis in the sense of marketing research
• Understand terms/concepts in a general, non-specific form
• Knowledgeable of various terms and concepts
• Understand the need to perform SWOT analysis

Students’ Weaknesses:
• Vague about connecting analysis with anything
• Little ability to apply or integrate the concepts
• Little ability to articulate how, for example, strong positioning leads to a competitive advantage
• Trouble discriminating between closely related concepts (and we have lots of those)
• Lack of a framework to tie things together
• Terms and concepts are remembered as concepts independent of each other

Learning Outcome 1c: Students’ understanding of marketing mix design and marketing strategy development for purposes of creating customer value leading to a sustainable competitive advantage.

Students’ Strengths:
• Better at working with individual elements of the marketing mix
• Again, know the basic terms/concepts in a general sense
• Understand to include all 4 ps in a proposed marketing program

Students’ Weaknesses:
• Not good at integrating the elements of the marketing mix
• Really don’t grasp the idea of strategy
• Promotion seems to be the element most focus on
• Some students still don’t know that you need to define the target market before you propose other elements
• Lack of integration among marketing mix
• Overall, I feel that our students are good at reciting, but not applying. The depth of understanding is somewhat shallow. In other words knowledge is good. Comprehension fine. Higher level (analysis, application, syntheses, etc.) Not adequate.

Learning Outcome 2a: Students’ ability to work effectively as a member of a larger team.

Students’ Strengths:
• Seem to do fairly well at this
• On rare occasions a team really functions well as a team
• At least don’t complain

Students’ Weaknesses:
• Delegation
• Planning
• Project management
• Leadership
• Not good at motivating team members to do their best

Learning Outcome 2b: Students’ ability to communicate clearly and concisely in both written and oral form.

Students’ Strengths:
• Most better at oral
• Presentation skills are pretty good as a whole
• Oral presentation is fine; very good at “packaging”

Students’ Weaknesses:
• Not as strong in written as oral (though 20% are just the opposite)
• Written skills are weak especially with respect to organization, structure, proper support, and basic writing skills
• Good at writing a descriptive report, but not an analytical one
• Good at talking at a general level (know all the big words); not good at specifics; also not good at providing quantitative evidence to support his/her points

Learning Outcome 2c: Students’ ability to develop and evaluate creative solutions, while solving problems.

Students’ Strengths:
• Fairly creative in limited instances in presentation style (e.g., PPT presentations)

Students’ Weaknesses:
• Only 1 or 2 students per class exhibit any level of creativity in case solutions
• Not very creative in thinking as a general rule
• Need very clear and specific guidelines/direction
• Weak in being able to integrate knowledge across courses but also across marketing courses to solve bigger problems
• Generally weak in problem-solving ability altogether
• Not common to see an “outside of the box” type of thinking – for example, dimensions used to segment a market tend to be very traditional ones, such as demographic, geographic, etc.

Method B. Marketing Senior Survey

Purpose: to obtain Marketing seniors' perceptions of their Marketing coursework, skills developed, and department service quality.

Sample: MKT 460 students (senior marketing specialists). We compiled responses across several semesters to obtain a large enough sample for stable result patterns.

Questionnaire: The items include closed ended and open-ended questions (see Appendix A).

Procedures: each MKT 460 instructor administered the survey in his or her respective classes toward the end of the semester.

Analysis: the data were analyzed by frequency counts and means for the closed-ended questions. Open-ended question responses were compiled so we could identify themes.

Results of Senior Survey Related to Learning Outcomes:

1. Students perceive that more emphasis is placed on market segmentation, understanding customers, and marketing strategy than upon marketing functions (product, promotions, place, and pricing). This is consistent with our intent (learning outcomes 1a and 1c). We do not currently ask a question about their understanding of information acquisition, management and use. We should, as this is one of our learning outcomes (1b).

To what extent did your overall course work in marketing emphasize the following areas?

Means
Market segmentation 3.53
Marketing strategy 3.41
Understanding customers 3.37

Product 2.91
Promotions 2.82
Place 2.72
Pricing 2.50

Scale: Not at all (0) to A great deal (4)

2. Our skill-related learning outcomes stress that students should be able to demonstrate the ability to work effectively as a team, communicate clearly and concisely in both written and oral form, and develop creative problem solving solutions. Students perceive that their marketing courses, internships, and participation in organizations have enhanced their skills and abilities quite a lot in working as a team member, listening to others, speaking in small groups, creatively solving problems, and making oral presentations. They report somewhat smaller improvement in resolving conflict and writing, and some in reading.

Looking back over your entire marketing program (courses taken at BGSU, marketing internships, and student organizations in marketing), to what extent have you developed or improved the following skills and abilities?

Means
Working as a Team Member 3.40
Listening to Others 3.19
Speaking in Small Groups 3.18
Creatively Solving Business Problems 3.15
Leading Others 3.09
Making Oral Presentations 3.06

Resolving Conflict 2.87
Writing 2.68
Reading 2.29

Scale: Not at all (0) to A great deal (4)


3. Students perceive the overall quality of the marketing specialization, teaching in marketing, and their overall marketing experiences to be very good. Courses in marketing, advising by faculty, and assistance in the marketing department office are rated as good. We would prefer to find that students rate advising by marketing faculty higher. In Fall 2000, the department implemented a new advising program, the effects of which are likely to emerge in surveys in future semesters.

Please rate the overall quality of each of the following:

Means
The marketing specialization 3.12
Teaching in marketing 3.12
Overall experiences 3.12
Assistance in the marketing department office 3.07
Courses in marketing 2.93
Advising by marketing faculty 2.68

Scale: Not at all (0) to A great deal (4)

Includes responses from those graduating in Spring 1999 (26.5%), Spring 2000 (42.6%), Summer 2000 (4.4%), Fall 2000 (13.2%), Spring 2001 (2.9%). [Survey responses were collected in Spring 2001, but are not reported here.]
------------------------

Method C. Student Presentation Videos

Purpose: to analyze students’ knowledge of marketing content (learning objectives 1a-1c) and oral communication skills (learning objective 2b).

Sample: MKT 460 students in Summer 2001 class.

Procedures: students will develop and record a short (5-10 minutes) presentation on video tape. The topic of the presentation is assigned: to discuss the essence of what Marketing is about. A panel of three judges will use the Oral Presentation Evaluation rubric to rate each presentation for oral communication skills. Evaluation of presentation content will examine specific areas on which students are especially strong or weak.

See Appendix B for the materials given to the MKT 460 instructor and students to prepare them for participating in this assessment activity.

Conclusions Based on Combined Results

• Knowledge Related: (Learning Outcomes 1a, 1b, and 1c)

Strengths: students are generally knowledgeable of terms and concepts with respect to knowledge learning outcomes. It appears that students are mastering descriptive -knowledge, including some procedural knowledge such as how to find certain types of information. Comprehension of material seems to be generally adequate. The students perceive their courses as emphasizing fundamental marketing principles more than functional areas, which fits with our Marketing Framework.

Weaknesses: students are neither connecting/integrating nor discriminating concepts adequately; they do not have an integrative framework to articulate; weak on understanding what analysis is; weaker on application and knowing what to do with information. Higher level reasoning could be stronger: analysis, interpretation, synthesis.

• Skills Related (Learning Outcomes 2a, 2b, 2c)

Strengths: students can complete a team project; oral presentation skills are good and may include creativity in presentations; writing descriptive reports sufficient. Students generally perceive their coursework as having developed their learning outcome related skills. It should be noted that students reported perceptions of improvement of skills; this does not necessarily mean their skills are where we would like them to be in all areas. Forthcoming methods of assessment will better calibrate the students' level of performance.

Weaknesses: team skills need work (e.g. planning, delegation, motivating); majority of students are not strong writers, especially when it comes to analytical writing; weak in creative thinking related to problem solving and general problem solving could be better.

Phase IV: Committee Recommendations and Department Actions

Recommendations Concerning Assessment Methods.

Our current assessment methods are accomplishing the objectives set for them. However, we have two recommendations for improving methods.

1. The Senior Survey should be revised to include questions that ask directly about the new learning outcomes.
Resources needed: no additional resources.

2. The Student Learning Survey conducted with instructors of our capstone MKT460 course was especially useful for identifying broad areas of strengths and weaknesses with respect to our learning outcomes. We should continue to use this method every three years. As follow up to the results, we should identify or develop rubrics for evaluating team work skills, writing skills, and creative problem solving skills. We already have a rubric for assessing oral communication skills (see Appendix B). These rubrics would be useful not only for measuring students’ skills more precisely, but could be used to identify specific dimensions of each skill area that need the most work. To improve students’ skills, it is important to identify these specific dimensions to make the task of improvement less overwhelming and more doable.
Resources needed: possibly one additional faculty member added to the Undergraduate Curriculum Comnmittee.

3. Continue to use the Marketing Framework model to guide assessment and curriculum changes. The Framework has already been useful in identifying a hole in our curriculum; that is, in the area of market information use (see Learning Outcome 1a). The Framework also provides a conceptual foundation to guide faculty in helping students to integrate and synthesize course material and what they learn from internships, co-ops, and club activities.
Resources needed: no additional resources.

Recommendations Concerning Curriculum Changes

Immediate Objectives to Implement in 2001-02

1. Objective: to help students improve their ability to integrate knowledge by applying the Marketing Framework and the central concept of customer value in all required (and preferably elective) undergraduate marketing courses.
a. Strategy: provide faculty with resources about integrating the Framework and customer value concept in their course.
b. Strategy: discuss in small groups of faculty who teach a particular course (e.g., MKT 300: Principles of Marketing).
c. Indicators: (1) Student Learning Survey given to MKT 460 instructors and (2) content of Videotaped Student Presentations about the essence of marketing (see Appendix B for materials).
d. Resources needed: no additional resources

2. Objective: to help students improve their team work skills
a. Strategy: provide faculty with resources (handouts, pedagogical ideas, other information) to help students work more effectively in teams.
b. Indicators: (1) Student Learning Survey given to MKT 460 instructors and (2) Senior Survey questions about team work related skills.
c. Resources needed: graduate assistant to help obtain informational resources for faculty use and to make this information available in appropriate form (e.g., handouts) for faculty use.

Curriculum Changes to Consider in Fall 2000:

In addition to the changes already discussed (see departmental actions below), the department faculty should consider prioritizing learning objective needs to systematically address curriculum development needs.

Department Actions

It is important to note that the department went through the University’s Program Review process in 2000-01. We decided to wait until we received feedback from that process before making any major plans for curriculum changes. We have agreed to take up curriculum revision at our department faculty retreat in August 2001. At that time we will take into account information obtained from our assessment activities and Program Review feedback to outline objectives and methods for achieving them. The following actions are preliminary steps already taken by the department faculty to address our curriculum needs.

1. Curriculum discussions in Fall 2000. The department faculty had approved the revised learning outcomes and Marketing Framework proposed by the Curriculum Committee. The new framework, learning outcomes, and faculty experience teaching certain courses aided us in identifying a hole in our curriculum. This hole is in the information technology area. The department faculty met several times to discuss several preliminary proposals. In Fall 2000 we spent considerable time discussing potential curriculum revisions and decided to make any final decisions until after the Program Review process was completed (end of Spring 2001). We will reconvene these discussions in Fall 2001.

2. Advising system changes. To more effectively guide students in course selection, scheduling, and career decisions, in Fall 2000 the department moved to a new advising system. Prior to this time, each department faculty member advised a group of marketing specialists. Now each marketing specialist is assigned to one of three faculty members who specializes in advising. Each of these faculty members are given course release time for advising.

3. New course offerings considered.

a. Marketing Plans: a hands-on course designed to give students the opportunity to learn how to create an entire marketing plan. Students would be required to collect and analyze customer and other primary and secondary data and to apply this information to design a marketing plan for a real business client. The course would be especially good for developing students’ analytical skills, their use and application of information skills, and integration of knowledge. It could also be designed to give attention to team work and analytical report writing, two skills areas identified through assessment methods that need development.

a. Marketing Information and Technology: a course designed to update our curriculum with regards to changes in information technology and its use for marketing decision making. The course could include such topics as: secondary data use, relationship marketing via data bases, and/or Internet marketing. Alternatively, instead of concentrating on one course, the department could implement a faculty development program to help infuse information technology coverage across the Marketing curriculum.