Abstract

    Because admission to higher education, and particularly to university, remains limited and competitive, the question of admissions policy is a perennial one. Conflicting institutional and societal interests exist in attempting to balance the demand for a highly educated populace against the performance expectations required of those pursuing the highest levels of education.

    In Canada, the primary determinant of admission to university is high school grades. In some provinces, these grades consist of a blend of school and external "public" examinations, while in others only school grades are used. A comprehensive system of public examinations existed in Newfoundland until 1994. Since that time, the system has been more unstable, with public examinations being lost in 1994 due to a labour dispute, restored in 1995 and discontinued in 1996. Over the same period, Memorial University has moved from a 60 percent to a 70 percent high school average as the minimum requirement for admission.

    This paper examines the question of decision errors and cut points in light of the shifts in grading systems. Basic descriptive data on high school graduates and university admissions are presented, and prediction equations developed corresponding to changes in high school grades and in admissions standards over the period 1993 to 1997. Results show that, although predictive power is fairly high, the presence or absence of public examinations is associated with first-semester success rates for marginal students, and that an increase in university admission average has been largely offset by increases in high school averages in years when there have been no public examinations.

Background

    This article is the first in a proposed series of papers examining the effects of changes in a high school grading system and in university admission requirements on first-semester university performance. Until 1994, the province of Newfoundland maintained a comprehensive public examinations system. Final year grades for high school graduation consisted of a 50-50 blend of school and public examination marks. In that year, the public examinations were abandoned at short notice because of a labour dispute. After being restored in 1995, they were again discontinued in 1996, this time because of a dispute over a proposal to have teachers mark the exams during the regular school year instead of in the Summer. Complicating matters even further, the province's only university, Memorial University of Newfoundland, was, during the same period, in the process of phasing in an increase in the minimum admission average from 60 to 70 percent. This process had to be delayed in 1994 when it was judged that reliable grades were not available. In that year, all applicants receiving a minimum of a passing grade (50%) in the last year of high school were admitted. The new minimum was restored in 1995 and has been in place ever since. This paper presents basic descriptive data and prediction equations for the years 1992 to 1997 when these changes were occurring.

Conceptual Framework

    Although universal access to primary and secondary education is now largely taken for granted in most developed societies, admission to higher education has remained competitive. For this reason, the problem of how best to select students for admission to higher education has been a long-standing one. From the perspective of public policy, the issue is whether all students should be given a chance at higher education, even if this means high failure rates, or whether institutions should be selective enough so that only those with a high probability of success should be admitted.

    In most jurisdictions, either high school grades or scores on some form of academic aptitude test are the primary determinants of admission. Although institutions have used a variety of other selection devices, such as recommendations, student essays, interviews and the like, as well as scholastic aptitude tests, the research on admissions has consistently shown that high school grades are the best single predictors of success at the post-secondary level, while aptitude tests such as the SAT show moderate but lower predictive value (Jenkins, 1992; Smyth, et. al., 1990; Chase & Jacobs, 1989).

    The admissions problem may be conceptualized in terms of errors of false acceptance and false rejection. Because prediction is imperfect, some students who are admitted will fail (false acceptances) and others who are rejected would succeed if given the opportunity (false rejections). Moving the cutoff point up or down will affect the proportion of each type of error. The policy problem is that of determining the cutoff point, taking into account both types of errors. The error rate, of course, depends on the predictive power of the grades being used for admission purposes.

    The situation is depicted graphically in Figure 1. The actual data take the form of a scatter diagram, in which each student has a high school and a university average, and can therefore be represented as one point on the graph.

    The broadest possible approach to estimating the magnitude of the decision errors involves determining the probabilities that an individual will fall into one of the four quadrants of the decision schematic, given particular values of the high school cutoff value and the university passing value. These probabilities can be found if the correlation between the variables is known and certain statistical assumptions are met. Expectancy tables (Morgan, 1988) or nomographs (Saupe, 1992) may be used for this purpose. However, these are difficult to produce and are less useful in practice than a modified approach involving estimates of the proportions of passes and fails at particular marginal cutoff points which may be contemplated. This is because, given the size of the correlations at hand, the overall proportion of correct decisions is quite high for candidates who are substantially off the marginal values, and hence these candidates are not of particular interest.  As an aside, it is noted that it would make an interesting experiment to actually relax admission requirements order to provide direct data on the error probabilities. On the surface, it might appear as if this situation exists here because admission requirements have changed over the period in question, particularly in 1994, when essentially all applicants were admitted. Unfortunately, the changes in admission average are confounded with simultaneous changes in the high school grading system and the two effects cannot be disentangled. Nevertheless, it is possible to estimate from the regression equations probabilities of success for certain hypothetical cut points, and to examine these more directly using crosstabulated data. This is essentially the approach taken here.

Data Base and Analysis Methods

    The data base used in this analysis was constructed from the high school certification records of the Department of Education and from the registration system of Memorial University. The underlying population consisted of all students graduating from high schools in the Province of Newfoundland from 1993 to 1997. The relevant sub-population consisted of all those from this source admitted to Memorial University in Fall Semesters during this period. School and public examination grades in thirteen final year high school courses used for computation of university admission average were selected from the high school data base. These were matched with first-semester course grades from university files.

    In practice, two common statistical approaches have been taken to the prediction problem. The most common is the development of a simple bivariate regression equation, in which high school average is the independent variable and first-term university average is the dependent variable. Variations on this approach would involve using different composites for both the independent and dependent variables ( e.g. specific courses included or excluded in calculating the high school average) or the use of multiple linear regression, in which more than one independent variable (e.g. grades in several high school courses as independent variables) is used in predicting university performance.

    Although regression equations are intended to provide optimal predictive power (under the statistical assumptions and rules used), this approach is not the most useful in addressing the essential policy question, namely that of what cutoff point should be used in determining admissions. Typically, a simpler statistical approach, based on crosstabulations, has been used for this purpose. Although the crosstabulation approach is less powerful overall, it does allow for the direct examination of the success rates of students above and below particular cutoff points which might be contemplated. This is essentially the approach used in the studies leading to the decision in 1991 to increase the minimum required high school average for Memorial University from 60 to 70.

    Data representing both of these approaches are presented here. In addition, in accordance with the conceptual model, estimates are presented on the effect of changing the cut point for university admission on the probability of success for students at the margin. Although conceptually plausible, the effects of changing the cut point for university pass is not considered here.

Descriptive Statistics

    Table 1 gives provincial high school graduation data for the six years from 1992 to 1997. The number of graduates has been relatively stable, with increased high school participation rates offsetting a general decline in the age groups. The 1994 anomaly
reflects the effects of the labour dispute, with students obviously being given the benefit of the doubt in assigning final grades that year. Years without public examinations are clearly distinguished by averages of four to five points higher than for years with public examinations. Considering that grades for public examination years are blended, it should be noted that the difference between average school and public grades is more like eight to ten points.

    University admission data for the same period are presented in Table 2. As is evident from that table, first-year admissions have been stable for the last three years, following a decline from the 1994 peak. The minimum admission average has increased from 60 to 70 in accordance with established policy, with 1994 again being an anomaly. Several offsetting trends are apparent from these data.

Notes:
¹New high school graduates only
²Normal admission requirement waived because of teacher strike

    First, it should be noted that the number of students admitted has not increased, despite an increase of four to five points in high school grades. This suggests that changes in high school grades have been offset by the increase in the minimum entrance standard. Although it is possible that these are causally linked, there is no way of determining this from the data at hand. Similarly, the overall average of students admitted has increased by about the amount of the increase in high school average, while the average first-semester grade has changed very little.

    Overall, it seems likely that the university is admitting students of about the same calibre as in the period before 1994, despite the changes in both the minimum entrance standard and the high school grades themselves. It must be recognized, of course, that the majority of students admitted during the whole period in question have had average grades above the current minimum, and that the only students seriously affected by all of the changes are those at the margins of both the admission standard and the passing grade in university courses. This leads to the crucial issue, which the remainder of the analysis is designed to address.

Prediction of Success and Decision Errors

    Table 3 gives the results of the bivariate regression analysis for the years 1992 to 1997. The correlations in this case are, of course, simple Pearson product-moment correlations. The coefficients of determination represent the proportion of variance of university average predicted by high school average. The standard errors represent the standard deviation of the distribution of university averages around any point predicted by the regression equation.

    The table shows slightly lower correlations and higher standard errors for years with no public examination results. The data for 1997 are particularly notable because this is the first year in which high school averages included no public examination components. (this is because students typically take some of their final year courses while in Grade 11 because of high school schedules). The decrease in the coefficient of determination is of particular concern here because this is the most direct measure of the predictive power of the high school grades.

    Given a particular value of the high school cut point, it is possible not only to predict the university average for a student at this point but also to determine the distribution of university averages around the predicted value as a mean, on the assumption of normality (which happens to hold quite well for university but not for high school grades). Once this is done, the proportions of passes and fails can be found by locating where the line representing the university passing grade intersects this distribution. This situation is depicted graphically in Figure 2, for a cut point of 70 using 1996 data. Table 4 shows the general pattern of predicted proportions passing for the years in question for several hypothetical cut points.

    Figure 2
Proportion Passing for Selected High School Cut Point (70%)

    The most obvious point of contrast in this table is between the results for 1992 and 1993 and the more recent results. Clearly a substantial shift in predicted proportion passing occurred after 1993, when the public examinations were first eliminated. This is consistent with the general increase in high school grades, assuming no major shift in university grading practices. A more specific comparison may be made by examining the predicted pass rates for students at the current cut point of 70. This has declined from near 60% in the years with public examinations to an average of about 45% in more recent years.

    Because the predicted proportions are hypothetical ones, based on the regression equation, and because of the distributional assumptions underlying such estimates, it is instructive to compare the results from this analysis with actual results. This can be done, of course, only for those above the cut points, except for 1994 when there was effectively no cut point. To avoid the problem of distributional assumptions, the comparison uses crosstabulations of high school and university grades.

    These data are presented in Table 5. The general pattern revealed by this table is similar to that of Table 4. Again before 1994, the proportion passing at the 70% cut point was more than 60% while the comparable figure since 1994 had declined to less than 50%.

    Table 5 also shows the overall proportion of passes for the years in question. As can be seen the move to a 65% admission standard in 1993 was associated with an increase from 79% to 84% in the overall pass rate. 1994 saw a drop back to the 1992 rate of 79%. It is worth noting here that while the 1994 waiver did not result in any substantial influx of students at the lower end of the high school grade scale, those admitted in the low 60's range had a very low probability of success. Imposition of the 70% requirement in 1995 was associated with a substantial improvement in pass rates. Effectively students with less than a 50% probability of succeeding were eliminated that year. Since1994, with a constant 70% admission standard, the pass rate has continued to decline overall as well as at the specific cut points.

Discussion

    Keeping in mind the difficulties in making causal inferences from correlational data, the observed patterns are reasonably clear. First-semester success rates vary with both admission requirements and changes in the high school grading system. Success rates for any particular cut point are lower for non-public examination years than for years in which public examinations were in place. This is clearly linked to the increases in average high school grades occurring in the years without public examinations. Success rates for the last three years also show a pattern of erosion even in the presence of a constant admission standard. While there appears to have been no general grade inflation other than that associated with the changes in the examination system, the reduced success rates indicate that a given high school grade conveys an increasingly lower probability of university success.

    Although the results appear compelling, caution should be used in drawing strict causal conclusions from the data reported here. One alternative to the grade inflation hypothesis is the possibility of a systematic shift in university grading standards. While it is clear that there have been no policy changes at the university level comparable to those affecting high school grades, it is possible that individual professors or departments respond to increased admission standards by a tacit increase in expectation. It is planned, as part of this research project, to pursue this hypothesis through a more comprehensive analysis of university grading, and particularly by following student year-groups through their university careers.

    Assuming for the moment stability in the university grading system, what policy implications may be drawn from the results presented here? First, it seems clear that high school grades retain a reasonable level of predictive power, even in the absence of a common measure. Nevertheless, the large standard errors of prediction at the individual level remind us that even with fairly accurate predictability of proportions, attempting to predict success or failure for any individual student is hazardous. This leads to the question of which type of error, false acceptance or false admission, is considered most serious. This, in turn is related to the issue of whether post-secondary education, and university education in particular, should be broadly accessible or open only to the "best" students.

    The first position is supported by the view that a modern society requires large numbers of individuals who are educated to the highest possible levels, and by the well known relationship between level of education and individual economic success. The second position finds expression in the annual Maclean's Magazine ratings of universities, in which high admissions standards contribute to positive ratings. Both views can, of course, be accommodated if we assume a university system of two or more "tiers" in which some institutions focus on high selectivity and others on broad access. In the end, in a system of publicly-supported universities, this is a matter of public policy as much as of internal university policy. Nevertheless, it must be recognized that if university education is to have some meaning in terms of knowledge, experience, analytical capabilities or other attributes, an argument can be made that differential inputs should not translate into differential outputs. Under these circumstances, universities which choose to admit on the basis of minimizing errors of false rejection will have to adjust their programs, operate remedial services, or live with higher attrition rates or lower average performance.

    The second policy question raised by the data is whether, all things being equal, a university should adjust its entrance standards in response to known changes in the criteria used for admission. Comparing 1993 with 1997 actual results, for example, would lead to the argument that Memorial University should raise its entrance standard by as much as an additional five points in order to achieve the same overall or marginal pass rates obtained in 1993. While it is possible, of course, that this would simply set off another round of high school grade inflation, the overall pattern suggests that the observed grade inflation is more related to the abolition of public examinations than to university admission policy. (School grades themselves have not changed much over the period in question).

    One approach that can be taken to this problem is to think of admission standards more directly in terms of probabilities of success than of actual grade averages. A university could decide, for example, to adjust its minimum entrance requirement each year, based on the cut point which would yield a 50% (or some other acceptable) probability of success. While this would have to be done with a one-year time lag, it would have the advantage of removing any systemic incentive towards grade inflation. It would also convey a message as to the "degree of selectivity" being practised by the institution and would make it clear to incoming marginal students what the chances of success really are. In fact, it would be interesting to speculate that this might have the desired side effect of motivating those students who wish to do well to the greater effort required and/or of discouraging those on the margin who are less likely to engage in such effort.

    It must be recognized, of course, that in a system with no common measures, high school grades are the product of a large number of individual teacher decisions. One can speculate that in any such system, some teachers will want to ensure that a high proportion of their students are admitted to university or that teachers will believe that their own students are generally better than average, and hence will grade in a manner which helps contribute to a general increase in grades. This, then, leads to the question of differences in grading practices across teachers and schools. This of course is a matter of equity rather than of predictive power of high school grades. Because of the separation of school and public grades, the data base at hand can be used to address the question of equity. This question will be the topic of a subsequent paper in this series.
 
 

Chase, C.I., & Jacobs, L.C. (1989). Predicting academic success: the utility of high school achievement average based on only academic courses. College and University, 64,403-408.

Jenkins, N.J. (1992). The Scholastic Aptitude Test as a predictor of academic success: a literature review. ERIC Document ED 354 243.

Morgan, G. (1988). Expectancy tables. In J.P. Keeves, (Ed.) Educational research, methodology and measurement: an international handbook. Elmsford, NY: Pergamon Press.

Saupe, J.L. (1992). A technique for producing a double-entry expectancy nomograph from observed proportions without distributional assumptions. Research in Higher Education, 33(1), 85-98.

Smyth, G. K. (1990). Using the EM algorithm to predict first-year university performance. Australian Journal of Education 34(2), 204-224.