Situation Specific validity
HELENA KRIEL
M.A.
INTRODUCTION
The current government subsidy policy for tertiary institutions favours academically successful students. It is therefore financially beneficial for tertiary institutions to have a mechanism in place by which only those applicants who are potentially successful students are admitted to such institutions. Furthermore, the high drop-out rates of especially first year students can be prevented if a proper selection procedure is in place.
At the Technikon Pretoria the admissions to 67 academic courses are managed by a potential assessment process. According to legal and professional guidelines such selection procedures need to be supported by thoroughly researched and documented validity evidence used as a foundation for selection decisions. By 14 August 1997 the PIB records used for this purpose amounted to 5 071 and the VPIB records to 1 580. Continuous research is being done on this data bank in order to be able to present empirically researched evidence for the situation-specific reliability and validity of our assessment tools.
The main objective of this study was to determine the situation-specific validity of the PIB in predicting the academic success of students at the Technikon Pretoria.
METHOD
One way of determining the validity of an instrument, or battery of instruments, is to relate scores obtained on such instrument(s) to performance on a relevant criterion and to analyze such relationship by multiple regression. Multiple regression analysis is a statistical technique that can be used to analyze the relationship between a single dependent (criterion) variable and several independent (predictor) variables. The objective of multiple regression analysis is to use the independent variables whose values are known to predict the single dependent value the researcher wishes to know. The result is a variate, a linear combination of the independent variables that best predicts the dependent variable. The variables are weighed in the process, the weights denoting their relative contribution to the overall prediction. In this way, regression analysis ensures that the analyst is provided with the maximal prediction in a format that also facilitates interpretation as to the influence of each factor in making the prediction (Hair et al, 1992).
A specific selection battery was composed at the Technikon Pretoria, for every specific course evaluated psychometrically for admission purposes. The predictive validity for each of these batteries is now being established by comparing the results obtained by students on the PIB-indices included in the specific battery, with the performance of the same students on academic tests and examinations. For the purpose of this study half-year marks in different subjects were used as criteria. The marks consisted of practical, test and formal exam marks.
PIB RESULTS
National Diploma: Journalism
With the mathematically calculated average of the nine (9) first year subjects of this course as criterion a multiple regression analysis was performed, using the stepwise method. The following indices have predictive validity for the whole group first year journalism students:
PIB indices –
- Mental Alertness (5)
- Reading Comprehension (3)
- Motivation (15)
- Spelling (22)
- Creativity (2)
Multiple R = .76
R2 = .58
Multiple R is the correlation coefficient for the regression of the independent and dependent variables. It has no plus or minus sign because in multiple regression the signs of the individual variables may vary, so this coefficient reflects only the degree of association. (Har et al, 1992). R square (R2) is the correlation coefficient squared, also referred to as the coefficient of determination. This value indicates the percentage of total variation of the dependent variable explained by the independent (variable(s)) (Hair et al, 1992).
According to the forward stepwise regression procedure, the indices which best predict academic success, for first year Journalism students at the Technikon Pretoria, as measured by their mathematically calculated average, are Mental Alertness (Index 5) and Reading Comprehension (Index 3).
With reference to the Political Science (major subject) as criterion the following variables had the highest predictive validity:
PIB indices –
- Mental Alertness (5)
- Spelling (22)
- Reading Comprehension (3)
- Motivation (15)
Multiple R = .75
R2 = .56
Index 5 (Mental Alertness) and Index 22 (Spelling) made the highest contribution to this value.
With Practice and Journalism (major subject) as criterion the following indices have predictive validity:
PIB indices –
- Spelling (22)
- Mental Alertness (5)
- Assertiveness (21)
- Motivation (15)
- Self-image (10)
Multiple R = .76
R2 = .58
The performance of the first year Journalism students on their second major subject was best predicted by Spelling (Index 22), Mental Alertness (Index 5) and Assertiveness (Index 21).
Pre-Tech Course in Mechanical Engineering
The Pre-Tech course in Mechanical Engineering is a bridging course presented by the Technikon Pretoria’s Department for Mechanical Engineering for potentially successful students with weak school marks in especially Mathematics and Physical Science, due to environmental deprivation. The students must obtain a pass mark in all six (6) subjects in order to be allowed to continue with the National Diploma: Mechanical Engineering.
With the mathematically calculated average of the six (6) subjects as criterion a multiple regression analysis was performed, again using the forward stepwise method. The following indices have predictive validity:
PIB indices –
- Creativity (2)
- Reading Comprehension (3)
- VPIB indices Spatial Reasoning and Perception (6 & 7)
Multiple R = .65
R2 = .42
According to the forward stepwise regression procedure Indices 2 and 3 of the PIB and indices 6 and 7 of the VPIB are the best predictors of a mathematically calculated average for Pre-Tech: Mechanical Engineering students at the Technikon Pretoria.
With Mechanical Engineering Drawing (a practical subject) as criterion the following indices have predictive validity:
PIB indices –
- Creativity (2)
- Reading Comprehension (3)
- Mental Alertness (5)
Multiple R = .67
R2 = .45
Index 2 of the PIB (Creativity) had the highest predictive validity towards this value. With regard to Applied Technology as criterion the next indices have predictive validity:
PIB indices –
- Creativity (2)
- VPIB indices Spatial Reasoning and Perception (6 & 7)
Multiple R = .67
R2 = .45
Indices 6 and 7 of the VPIB (Spatial Reasoning and Perception) made the highest contribution to this value. When the subject Mechanics was used as criterion the following indices had predictive validity:
PIB indices –
- Creativity (2)
- Reading Comprehension (3)
- VPIB indices Composition of Wholes (4)
- Spatial Reasoning and Perception (6 & 7)
Multiple R = .76
R2 = .57
The Pre-Tech: Mechanical Engineering students’ performances on this criterion are best predicted by Index 2 of the PIB (Creativity).
National Diploma: Drama
The formal half year progression reports of the first year Drama students were used as source and their academic performance were once again used as criteria for a stepwise multiple regression analysis. For the subject History of Costume, the following indices have predictive validity:
PIB indices –
- General Knowledge (1)
- Motivation (15)
- Mental Alertness (5)
Multiple R = .63
R2 = .40
Index 1 (General Knowledge) made the highest contribution to this value. With reference to Voice Production as criterion the following indices have predictive validity:
PIB indices –
- General Knowledge (1)
- Self-Image (10) [Inverted]
- Mental Alertness (5)
- Stress Management (18) [Inverted]
Multiple R = .80
R2 = .63
According to the forward stepwise regression procedure the indices that best predict the mentioned independent variable are Index 1 (General Knowledge) and Index 10 (Self-image). It is interesting to notice that, in support of previous research results, the profile of a successful drama student manifests a low self-image and high stress level (Kriel & Erasmus 1997).
For the subject Practical Exercises: Speech Sounds the following indices have predictive validity:
PIB indices –
- Reading Comprehension (3)
- Mental Alertness (5)
- Stress Management (18) [Inverted]
- Coping Skills (24)
Multiple R = .82
R2 = .68
Index 3 (Reading Comprehension) and Index 5 (Mental Alertness) made the highest contribution to this value. The finding towards high stress levels is confirmed. With reference to Practical Interpretation: Prose and Poetry the following indices have predictive validity:
PIB indices –
- Reading Comprehension (3)
- General Knowledge (1)
- Motivation (15)
- Mental Alertness (5)
- Coping Skills (24)
Multiple R = .75
R2 = .57
The first year Drama students’ performances on this criterion are best predicted by Index 3 (Reading Comprehension) and Index 1 (General Knowledge).
National Diploma: Civil Engineering
The academic performance of a group of S3 (third semester) Civil Engineering students were used as criteria for a stepwise multiple regression analysis. With reference to the subject Geo Technical Engineering the following indices have predictive validity:
PIB indices –
- Abstract Reasoning (7)
- Reading Comprehension (3)
- VPIB indices Spatial Reasoning and Perception (6 & 7)
- Composition of Wholes (4)
Multiple R = .55
R2 = .31
Indices 6 & 7 of the VPIB made the highest contribution to this value. For the subject Management: Civil the following indices have predictive validity:
PIB indices –
- Mental Alertness (5)
- Reading Comprehension (3)
- VPIB indices Spatial Reasoning and Perception (6 & 7)
- Composition of Wholes (4)
- Numerical Ability (3)
Multiple R = .81
R2 = .66
The performances of the Civil Engineering students on this criterion are best predicted by PIB Index 5 (Mental Alertness) and VPIB Indices 6 and 7 (Spatial Reasoning and Perception). With regard to Transport Engineering the following indices have predictive validity:
PIB indices –
- Mental Alertness (5)
- Reading Comprehension (3)
- VPIB indices Spatial Reasoning and Perception (6 & 7)
- Composition of Wholes (4)
Multiple R = .63
R2 = .39
According to the forward stepwise regression procedure the performance of third semester Civil Engineering students at Technikon Pretoria on the subject Transport Engineering are best predicted by VPIB Index 4 (Composition of Wholes).
CONCLUSION
The main aim of this research done by the Bureau for Student Counseling at the Technikon Pretoria was to give an indication of the situation-specific validity of the PIB in predicting the academic success of students at the Technikon Pretoria. As far as the mentioned results are concerned it can clearly be said that the PIB has a course specific predictive validity for students in different courses at the Technikon Pretoria. It is extremely important to realize that these results are situation-specific and cannot be generalized to other institutions, albeit similar ones. It is also important to realize that most situations are dynamic and therefore research into situation-specific validity and reliability should be a continuous process.
REFERENCES
HAIR, ANDERSON, TATHAM & BLACK, 1992. Multivariate Data Analysis with readings. VSA: Macmillan
KRIEL, H. 1997. The Predictive and Construct Validity of the PIB in an Academic Institution. In: Erasmus, P.F. (ed) The Unfair Labour Practice the Worker and Assessment: What every Human Resources Practitioner should know. Johannesburg: RNATA
