Portfolio > 4b > 1. Assessment > Assignment
Search this site

4. A collaborative consultant with skills in:
(b) Evaluation and assessment

Andrews University

Name:

David Heise

Student No:

86197

Course:

EDUC689 Seminar: 
Assessment and Evaluation

Instructor Dr. James Tucker

Assignment:

Evaluative Questions and Corresponding Methodologies

Due Date:

Summer, 1998

Evaluative Questions and Corresponding Methodologies

Contents

Introduction - What kind of questions?
1. Instructor Load and Productivity - Deans' Statistics
2. Strategic Market Position - Market Segment Analysis
3. Attracting Students - Using the Web
4. Retaining Students - Cohort and Retention Analysis

EVALUATIVE QUESTIONS AND CORRESPONDING METHODOLOGIES

Introduction - What kind of questions?

A very fundamental "what kind of..." question was asked at the 1997 Fall Fellowship meetings - "What kind of a Christian University should we be?"  Searching for answers to this question leads to questions about what we should be doing, and follow-on questions of why and how.  I believe that some of the "how" questions have answers that are dependent on technology.  It is my vision, both as Chief Information for the University and as an overall personal goal as a participant in the Leadership program, to provide administrators and managers with the tools for both tactical and strategic decision support, through the use of data warehousing technologies.

So, the questions I would like to work on finding answers to are those where technology can provide a tool that would assist administrators in leading the University towards meeting some fundamental goals.  Many of these goals are interrelated, such as goals of improved financial viability and academic excellence.  For instance, it has been pointed out that low wages for faculty can threaten the quality of the academic program.  Evaluative questions can be posed in at least the following areas:

  1. Instructor load and productivity
  2. Strategic market position
  3. Attracting students
  4. Retaining students

The process for designing and building a data warehouse is based on building models using key performance indicators that are suggested from a study of critical success factors which themselves come from the strategic plan.  This approach of using evaluative questions will serve well in helping to obtain clear definitions of the information needed for analysis and decision support in these areas.

1. Instructor Load and Productivity - Deans' Statistics

The office of Institutional Research produces various statistical reports for Deans, showing credits generated and other data by course, instructor, department, school, etc.  Prior to Banner, some of these reports were easier to generate than they are now, and were more informative.  The Deans must do considerable amounts of manual processing beyond what can be provided by Institutional Research from Banner, to get to the figures they need to support decisions relating to load balancing, credits and revenue generated, instructor productivity, minimum viable class size, etc.

1(a) What statistics and analysis tools best serve the needs of academic administrators in choosing policies for course offerings and staffing to optimize the efficiency and effectiveness of the academic experience at Andrews University?

Methodology

  1. Make presentations to the Deans' Council and meetings of chairs and department heads to describe the objectives of data warehousing, and demonstrate the kinds of data analysis that is possible.
  2. Have members of the Data Warehouse team present to individuals and groups what is already understood about the data requirements, and conduct interviews and joint application development sessions to clarify and refine that understanding, and gain additional insight.
  3. Design and build the necessary tables in an academic Data Mart, and develop transfer procedures for taking the data from Banner, summarizing them appropriately, calculating derived items, and loading them into the Data Mart.
  4. Populate the tables with historical information from Institutional Research reports where the information is not available in digital format.
  5. Set up an electronic information deliver system, preferably accessible over the web via the Intranet.
  6. Train academic administrators and/or their support staff in the use of this tool.
  7. In all of this, remember to develop the model iteratively, maintaining close consultation with the target user groups.  This needs to be a joint development project between ITS and those who will use these tools.  It needs participation and input from the users to ensure it meets their needs, and to gain their acceptance.  An "If we build it, they will come" attitude is certain to fail.

2. Strategic Market Position - Market Segment Analysis

In the September 1998 meeting of the Strategic Planning Committee, Jack Stout presented a market segment analysis methodology based on a mathematical model developed from research by the National Center for Postsecondary Improvement (NCPI).  This research "provides a tool that institutions can use to describe that market, find their place within it, and identify what they need to do in the future." (Zewmsky, 1997)  After feeding a number of statistical values into the model, a ranking from 1 to 7 is produced.  At the left hand end of the scale, rankings of 1-3 are considered to be in the "medallion" or "name brand" part of the market, and 6-7 at the other end of the scale are considered to be in the "convenience/user friendly" part of the market.  The 1994 statistics for Andrews University yielded a ranking of 5, which is a highly competitive and undesirable place for a private university of our size to be.

2(a) On the basis of the NCPI "Mapping the Market" study, and in relation to the mission of the University, what changes in strategy could lead to "a move to the left" in the market segment model?

Methodology

  1. Determine if it is appropriate to break Andrews into separate types of schools (name brand versus convenience/user friendly) for this analysis, and if so, where the divisions should be made.
  2. Locate a source for the data elements that are used for this analysis.
  3. Develop a system for automating the collection of the data, and for performing the calculations for the model.
  4. Perform the analysis for the various divisions identified in step 1, and observe if these different divisions are characterized as belonging to different market segments.
  5. Develop a model for generating data forecasts, and use this model to produce data for the Market Segment Analysis model.  This model would need to be well researched, and have wide input from faculty skilled in this area.
  6. Simulate changes in variables and observe the effect on the ranking that is generated.  Study the effects on the University as a whole as well as on the separate divisions.  This will hopefully yield useful information about the kinds of achievable changes that would result in "a move to the left" in the market segment model.
  7. Apply the model to data that has been collected over the past, and study any changes over time that are found in the Market Segment ranking.

2(b) Does the implementation of these changes in strategy produce the desired change in ranking over time?

Methodology

  1. Identify the indicators for the various changes that are implemented, and record their values and meanings.
  2. Compute the market segment ranking as each year passes, for the University as a whole, as well as for the separate market divisions.
  3. As data is collected over time, perform trend analyses, and search for correlations.
  4. Search for explanations for any correlations that are found, or not found when expected.
  5. If there is in fact "a move to the left" in the market segment model over time, attempt to relate it to the changes in strategy that were implemented.
  6. Refine and revise the strategy to enhance the positive changes, and diminish those having deleterious effects.

2(c) What changes in enrolment patterns, if any, occur in conjunction with these changes in strategy (that is, is there a resulting increase in enrolment)?

Methodology

  1. Identify the data elements that might be needed to support data analysis and statistical "data mining" research into changes in enrolment patterns.  Consult with the office for Institutional Research, the Vice President for Academic Administration, Deans and Faculty Chairs.
  2. Research the tools and consulting services available for the various aspects of Data Warehousing.  For this project, statistical data mining tools should be evaluated.
  3. Using appropriate Data Warehousing tools, specify the source locations for these data elements, the data transformations and quality checks, and the data extraction schedules for populating the Data Warehouse.
  4. Apply the data mining tool to answer the question.

3. Attracting Students - Using the Web

This is the subject of another project on campus, not directly related to data warehousing, and is not dealt with in this paper.  Registration via the Web, as well as student information lookup, such as account balance, grades, transcripts, course catalog, class schedules, etc, are being implemented through the Banner Web for Students product.

4. Retaining Students - Cohort and Retention Analysis

A number of studies on retention have been done, either formally or informally, and they all conclude that Andrews needs to improve its retention rate.  But since there is no automated system that computes retention, the studies have to be repeated manually every time they are done.  Some studies are not as exhaustive as others, and there are differences in the assumptions that are made.  Being able to analyze retention according to different parameters, conveniently and with clearly defined assumptions, will facilitate the discovery of factors that have the greatest impact on retention.  In addition, we will soon need to provide retention information in order to continue receiving certain types of federal aid.

4(a) What are the identifying characteristics of groups of students most at risk of not completing their study program, and what can we learn about their reasons for leaving?

4(b) What are the identifying characteristics of groups of students who complete their study program, and what can we learn about why they stayed?

Methodology

Since these questions are closely related, the methodologies will mostly be the same for each of them.

  1. Locate all those at Andrews who have done formal or informal studies of student retention in the past, and determine the data elements and data models that were used, as well as the assumptions and purposes of the model.
  2. Coordinate a meeting of all those interested in the project to brain storm about possible factors to include in the study, and to reach a common understanding of assumptions and meanings of terms.
  3. Perform the necessary steps to provide the necessary data in the Data Warehouse, including the various derived measures of retention that will be used.
  4. Develop the models that will be used for analyzing retention in relation to other data variables.
  5. Develop surveys where needed to elicit reasons why students withdrew or why they stayed.
  6. Implement the models for those users who request it, and consult with them about ways of refining the model as it is used.

References

Zemsky, R., Shaman, S., Iannozzi, M. "The Landscape - In Search of Strategic Perspective: A Tool for Mapping the Market In Postsecondary Education", Change, November/December 1997. p23


Return to 4b1 EDUC689 Assessment and Evaluation

Created: Tuesday, January 04, 2000 05:15 PM
Last Modified: Thursday, January 8, 2004 12:34 PM