Skills testing

First and foremost, skill or proficiency tests should be judged in terms of the content and face validity of a specific skill area. This is what makes the test relevant and valid in terms of assessing job related skills and not as much the reliability issue. Reliabilities coefficients depend on the content validity, the length of the test and the specific data that was used. 

Reliability may vary from one situation to the next depending on context of use. For example, if a test is used as a criteria references test to determine the level of mastery of specific skills, the reliability will drop the higher the mastery level of the sample (Anastasi 1997). The reason for this is that reliability coefficients are dependent on score variability. This would not mean that the test is not a reliable measure of the specific skill and situation. (In a certain sense skill testing is similar to academic proficiency testing in schools and higher institutions. Why is the reliability of academic testing and exams never questioned? Although, content validity is often questioned).