Pols logo

Validation in Psychiatry

In psychiatry the situation is different, at least in those cases that no validation by physicochemical means is possible. There are several possibilities for confirming a hypothesis about the diagnosis: expansion of the anamnesis, heteroanamnesis, conversation with the patient and those who are close to him together, and psychological testing. In addition, much research has been done as to the reliability of the diagnosis by comparing diagnoses made by different psychiatrists regarding the same patient (“observer agreement”), by comparing the frequency of certain diagnoses in comparable patient populations (“frequency agreement”), and by comparing diagnoses in the same patient at different times (“consistency”). It was found that factors such as whether or not symptoms and diagnoses were precisely defined, the psychiatrists’ school of theory, and the setting in which the examination transpired, were very important, alongside other variables. Kendell mentions research by Beck in 1962 regarding an out-patient population in which four psychiatrists reached “observer agreement” in 54% of the specific diagnoses, and 82% when the alternative diagnoses were included. Kreitman et al found “observer agreement” of 63% in similar research. Improvement in the results can be expected when examination instruments are utilized such as questionnaires for patients to complete themselves, questionnaires for the examiner to complete, behavioral scales, and structured interviews. Improvement can also be expected from better classification systems: the DSM-III seems to offer such. It is not my intention to elaborate on the efforts to increase the reliability of psychiatric diagnoses.

When reading these research reports the impression constantly made is that it is more difficult to “extract” as it were, psychiatric disorders from a person, than is the case in physical illness. After all, determining a diagnosis is an effort to abstractly separate the person and his disease. In psychiatry, who someone is – or was, before his illness – constantly influences the phenomena of being ill. A psychiatric disorder is something one “has” and at the same time something one “does.” Furthermore, in my opinion the influence that the diagnosis itself can have on the diagnosed person and his environment, which in turn influences the predictive value, should also be constantly considered.

Assessment of reliability and predictive validity in psychiatry can be summarized as follows. Rooymans researched the literature as part of his dissertation about judgment and prejudice in psychiatric diagnoses. He mentions an “observer agreement” percentage which at the level of the main categories is usually between 60 and 70%, and at the level of specific diagnoses between 40 and 55%. The highest percentages are to be found in the main category of the organic disorders, the lowest in the category of neuroses. His conclusion is that the reliability of diagnoses is usually disappointingly low and that also the predictive validity is low. Kendell concludes regarding research on reliability done since 1950 that “reliability is often very low, and generally lower for neuroses and personality disorders than for psychoses and organic states.” Furthermore Kendell states that there is a certain predictive validity but that it is smaller than in most other branches of medicine.

Bakker’s dissertation can provide us with an example of research on the value of psychiatrists’ prognostic assessment. He requested the treatment teams of clinically treated patients who were about to be released to predict a number of matters regarding the first half year after release. Among the matters to be predicted were general conditions, progress, rehospitalization, suicide, and employment. These predictions were compared to the actual situations. A value of 0.12 to 0.27 was found for the ordinary kappa* , meaning that these predictions scarcely materialized, if at all. Although Bakker admonishes us to not generalize the results of his research too much, he did reasonably demonstrate that psychiatrists’ prognostic judgment based on their findings is far from dependable. Excessively optimistic predictions were made in particular when the patient was young, when his condition at the time of release was reasonable or good, and when the physician-patient relationship was judged to be “usual” or “better than usual.” Excessively pessimistic predictions were made in particular when the patient was 45 years or older, if he was not recovered at the time of release, and when the physician-patient relationship was judged to be “less well than usual.” In addition, it was shown that there was more pessimism than optimism.

In summary, there is little to applaud in psychiatry regarding reliability, predictive validity of diagnoses, and the making of prognoses. It should not be overlooked that this problem is significant in psychiatry only since the sixties of the twentieth century. Since then there have been concerted efforts to improve classification, to design evaluation instruments, and to achieve operational, well explainable concept definitions. There can be no other conclusion than that the reliability and predictive validity of psychiatric diagnoses are meager. On the other hand it has become clear that psychiatric validation is possible to a certain extent, and that the results are better than random. This means that psychiatric diagnoses are not, as Szasz asserts, purely subjective and random, but that they rise above that to a certain degree.

*This is a statistical measure in which, in addition to the observed correspondence, the coincidental correspondence is also figured, as well as the nature of and relationship between the variables. The value is set between 1 = perfect correspondence and 0 = no correspondence. – J.P.
Table of Contents