More on predictive validity

METHOD OF ANALYSIS PREDICTIVE VALIDITY

Pieter Schaap

The validity of an entire battery can be determined by computing the multiple correlation [R] of the criterion and the battery. This correlation presents the highest predictive value that can be obtained from a battery, as each measure is given an optimum weight for predicting the criterion in question. In certain instances, a measure that does not correlate with the criterion while correlating highly with other measures, can act as a suppressor variable and can eliminate irrelevant variance in other measures included in the battery.

The relationship between a measure and the criterion is consequently enhanced and the outcome is an increase in the validity of the battery as a predictor. Regression analysis eliminates serious cases of predictor duplication and assigns greater weight to measures contributing uniquely to predicting the criterion. The optimum weights are determined by a regression equation and can be successfully applied in predicting success on the criterion (McCormick E.J. & Ilgen D, 1985: Anastasi A 1990: Cascio 1987).

In a first study multiple regression analysis was performed on 365 cases using the stepwise method to identify the variables contributing most to the predictive validity of the PIB indices 4, 5, 10, 11, 15 and 16 (Schaap 1996).

In a second study performed by Helena Kriel (Kriel 1997) at Technikon Pretoria the same procedure was applied and the predictive validity of different PIB and VPIB indices was determined on a sample of 5 071 PIB cases and 1 580 VPIB cases respectively.

RESULTS OF THE FIRST STUDY (Schaap, 1996) In this particular study, performance appraisal results and percentage salary increases were used separately as criterion measures. The inter-correlation between the measures is 0.59 which indicates a certain amount of overlap. This means that additional factors contribute to salary increase and that an increase is not exclusively based on performance appraisals. Both criterion measures were transformed to a standard ten point scale to ensure that the criterion in different sections is equalized with the same mean and standard deviation.

With reference to the performance appraisals as criterion of work success, the following PIB Indices have predictive validity for the total group.

Indices 4 and 11 made the highest contribution to this value. The predictive validity coefficient obtained for the PIB battery relates well to other research findings. According to meta-analysis procedures whereby different research findings were summarized, it was found that the average predictive validity coefficient for cognitive related instruments for managerial success ranged from .25 to .30 (Cascio 1987).

However, when these correlations were statistically corrected for criterion unreliability and for range restriction due to pre-selection, the validity increased to .43 and .53. It must be emphasized that a predictive validity coefficient as low as .20 already justify inclusion in a selection program (Anastasi A 1990).

There is evidence of subjectivity in the performance appraisals used as criteria in this study. Large differences were found in mean values and the variance in scores of rates from different sections in the organization. The extent to which the appraisal system represents actual work performance is unknown, although subjective factors would have had an influence on the validity and reliability of the criterion.

Pre-selection would have had a negative influence on the magnitude of the multiple R coefficient obtained, as selection was not made at random. It can be expected that the multiple R coefficient will be higher if all applicants were included in its determination, and the unreliability of the criterion measure statistically controlled.

RESULTS OF THE SECOND STUDY (Helena Kriel, 1997)

In this particular study, students’ performance on academic tests and examinations were used as criterion measures. These scores included practicals, tests and formal exam marks.