PibSpeex in Training

1. Introduction

The main purpose of the following two studies, was to determine the predictive validity of PIB SpEEx in training environments. Ritson (1999) states that the rationale for using psychometric tests in selection processes, lies in the purported ability of the testing instruments to accurately and objectively assess an applicant’s potential to perform the tasks required in a specific situation.

In the first of the two studies, the predictive validity of the PIB SpEEx as selection instrument for a post-matric bridging programme at a semi-government institution was examined. The second of the studies was performed at Technikon Pretoria, using a sample of 3327 students admitted to the institution, via a potential assessment process in which PIB SpEEx was used, during the period 1999 to 2001.

2. Post-matric bridging programme for the purpose of this study a non-probability convenience sample was selected. The subjects were the total number of Technology students enrolled for the bridging programme in 2002, for whom both psychometric and academic data were available. This constituted a sample of 53 respondents. Because of the sample size, the results of this study must be viewed as preliminary and cannot be generalised to other situations. As the results are in line with previous findings in other training environments, the findings of this study remain important though.

In this regard, reference can be made to the findings of Muller and Scheepers (2003), who examined the predictive validity of the battery used for the selection of junior leaders in the South African National Defence Force. In this study sub-scales of PIB SpEEx were included in the battery and these researchers found a R2 value of 0.40. Kriel (2001) in her study of the predictive validity of PIB SpEEx in the selection of engineering technology students at Technikon Pretoria, found R2 values ranging between 0.26 and 0.65.

If the following comment of Wood and Payne (1998) this does not mean that ability tests are excellent at prediction, at best they predict something like 25% of the variance in job performance. is taken cognisance of, the context of this study becomes more clear and the results thus even more useful.

The statistical analysis in this study involved the following steps:

1. The mean and standard deviation for all predictive variables were calculated. A frequency distribution for each variable was drawn. Attention was given to the median, as well as the skewness and kurtosis of each distribution.

2. A multiple regression analysis was performed with the students. psychometric results as predictor variables and academic performance as criterion variables.

2.1. Results

Table 1 contains a summary of the descriptive statistics of the results obtained by the respondents on the indices included in the potential assessment battery for prospective students in the bridging programme.

Table 1: Descriptive statistics of results obtained by the respondents on the indices of the PIB SpEEx battery. When interpreting the descriptive statistics, it is important to keep in mind that these statistics describe the scores obtained by a pre-selected group. Only the scores of those students, who were finally admitted to the programme, were included in the analysis.

Table 2 gives the results for the multiple regression analysis performed on the data. The results of the various academic subjects were used as criterion for the analysis.

Highlighted predictors have individual value of higher than 0.20

* F Ratio statistically significant on 0.5% level

** F Ratio statistically significant on 0.1% level

Table 2: Result of multiple regression analysis performed on data. From Table 2 the validity coefficient (R) for each criterion variable can be seen and the percentage (R2) of the variance in the criterion variable explained by the variance in the predictor variable included in the prediction model, can be deducted. Furthermore, the standard error of estimate indicates the accuracy of the prediction.

The F Ratio indicates whether the relations between the actual and predicted scores are statistically significant.

2.2 Conclusion

This preliminary analysis yielded results that, to a certain extent, were to be expected, as the strong relationship between cognitive potential, as measured by PIB SpEEx, and training outcomes, has in the past been proven. It is important to keep in mind that due to the fact that the sample consisted of a pre-selected group, the validity coefficients might have been influenced by the restricted range effect. The results of this study should not be generalised as the sample is small and fairly homogeneous as far as age and level of training is concerned.

3. Technikon Pretoria

At Technikon Pretoria, after a prospective student applied for admission and met the minimum requirements set, such a candidate is invited to make an appointment for the Technikon Pretoria Potential Assessment (TPPA). The TPPA consists of a battery of tests compiled through a specified process, as stipulated by the developer of the PIB SpEEx. These tests are administered on groups of students by student counsellors and/or psychometrists by means of a paper and pencil application. As the tests all consist of multiple-choice questions, the answer sheets are scored by an Opscan- system. Results are the imported into an Excel-spreadsheet, per academic programme. The candidates are ranked in order of performance and a recommendation for every individual is noted in a designated column. The recommendations are forwarded to lecturing staff responsible for the final selection of students. These members of staff will then integrate all information on candidates before making the final decisions.

A turn over rate of 8 working hours is maintained, reckoned from the moment the candidate enters the test venue, until the results are received by the responsible lecturing staff. On average 6 500 applicants are assessed in this manner per calendar year. According to Zaaiman (1998), an efficient selection process, operate optimally under logistical constraints, such as time limitations and numbers of applicants. From the above, it can be deducted that the process at Technikon Pretoria is efficient. This specific research project then focussed on the effectiveness of the recommendation made by the student counsellors in the prediction of student success.

3.1 Methodology

In order to perform the study, a convenience sample of 3221 records of first time entry students between 1999 and 2001 were selected. The sample consisted of those students for whom on both MIS (Management Information System) and at the Bureau for Academic Support a student number was available and for whom a selection outcome as well as academic results were available on the MIS.

The performance of students recommended for study by the student counsellors, on grounds of their performance on the potential assessment, were compared with the results of those conditionally recommended, as well as those not recommended, but admitted. Criteria used were Degree Credits Passed (DCP) and Average Academic Performance (AAP). The data analysis was performed by the Statistical Support Department at Technikon Pretoria, making use of the SAS-programme.

3.2. Results

Table 3 contains a summary of the descriptive statistics of the criteria variables, with regard to the students that were recommended for study (Group 1) in their particular field of choice.

Table 3: Descriptive statistics of criteria variables, with regard to the students that were recommended for study (Group 1) in their particular field of choice.

Table 4 contains a summary of the descriptive statistics of the criteria variables, with regard to the students that were not recommended for study (Group 2) in their particular field of choice, but still admitted for studies.

Table 4: Descriptive statistics of criteria variables, with regard to the students that were not recommended for study (Group 2) in their particular field of choice.

Table 5 contains a summary of the descriptive statistics of the criteria variables, with regard to the students that were conditionally recommended for study (Group 3) in their particular field of choice.

Table 5: Descriptive statistics of criteria variables, with regard to the students that were conditionally recommended for study (Group 3) in their particular field of choice In order to determine the statistical significance of the difference found between the mean scores obtained by the various groups SAS’s GLM procedure was performed, including Duncan’s Multiple Range Test.

Table 6 illustrates the results obtained when testing for the significance of the difference between the means of the three groups on the criterion variable Degree Credits Passed (DCP). Table 7 refers to Duncan’s Multiple Range test for DCP.

Table 6: GLM Procedure: Dependent Variable DCP.

Table 7: Duncan’s Multiple Range Test for DCP.

Table 8 illustrates the results obtained when testing for the significance of the difference between the means of the three groups on the criterion variable Average Academic Performance (AAP).

Table 9 refers to Duncan's Multiple Range test for AAP.

Table 8: GLM Procedure: Dependent Variable AAP.

 Table 9: Duncan’s Multiple Range Test for AAP

3.3 Conclusion

From the results obtained in this study, it can be seen that there is a statistically significant difference between the academic performance of the candidates who were recommended on grounds of their performance on the PIB SpEEx indices and those students who were not recommended but for some reason admitted to the various academic programmes. This indicates that the selection decisions made on grounds of the performance of students on the PIB SpEEx can be deemed valid in this situation.

4. List of references

Kriel, H. 2001. The predictive validity of a potential assessment battery for engineering technology students. Unpublished MA thesis. University of Pretoria: Pretoria Muller, J. & Scheepers, J. 2003. The predictive validity of the selection battery used for junior leader training within the South African National Defence Force. Accepted for publication, South African Journal for Industrial Psychology, Volume 3, 2003.

Ritson, N.A. 1999. The validity of a test battery used in the selection of apprentice electricians. Unpublished MA Thesis. University of Natal: Durban.

Wood, R. & Payne, T. 1998. Competency based recruitment and selection: A practical guide. Chichester: Wiley. Zaaiman, H. 1998. Selecting students for Mathematics and Science: The challenge facing higher education in South Africa. Unpublished Ph.D thesis: Vrije Universiteit, Amsterdam.