The friends and family test is unfit for purpose: A well-designed survey can be a powerful tool but the single-question A&E survey being introduced in hospitals is unclear, ambiguous and will not stand up to meaningful analysis.
In the wake of the Stafford hospital scandal and widespread concern about the treatment received by hospital patients, there is no doubt that new initiatives to improve care will be welcomed. Surveys can be powerful tools to quantify what goes wrong, how often, for whom and in what circumstances. Such information can be invaluable in helping to target resources and improve performance. But will the new friends and family test have the desired effect?
The survey has just one question. But if an A&E patient would not recommend this hospital, what would he or she recommend instead? To travel 50 miles to the next hospital? To avoid medical assistance completely? To wait until their GP has an appointment?
The question is hypothetical. It assumes a choice, but does not state the alternative which the patient is supposed to use as a means of comparison.
And it is unclear what patients are supposed to be assessing. Is it the standard of care by nurses? The standard of treatment? Comfort? Cleanliness? Food? It will be interpreted in vastly different ways.
The variation in how patients react to the question could distort comparisons between hospitals or wards.
Of course, these criticisms can be aimed at many survey questions, but hopefully – at least in the case of serious government or academic surveys – to a much lesser extent. The criticisms matter in this case because different hospitals, and different wards within a hospital, have different kinds of patients. The official guidance to trusts on implementing the test states that the question wording has been informed by independent research. It does not mention that the research commissioned by the Care Quality Commission concluded that the test is "not appropriate for use in an NHS setting".
Even if the question were clear, unambiguous and universally understood, the subject has connotations of social desirability. Some people think it impolite to criticise and will therefore answer "yes" to the question, even if they received worse care than someone else who answers "no". And they will be particularly likely to do so if asked by an interviewer rather than in an anonymous self-completion mode.
NHS trusts are allowed to administer the question in whatever way they see fit: by telephone, online, by SMS, at kiosks in the hospital lobby, by handing the patient a card, or in any other way. This introduces a systematic bias into comparisons between hospitals. A hospital's score will be artificially inflated if it uses interviewing rather than self-completion, for example. The NHS's own research recommended that guidance should be given both on the mode of delivery of the question and on the setting in which it should be delivered. Both these recommendations have been ignored.
Trusts are given free rein regarding how to encourage patients to take part. They are told only that the minimum response rate is expected to be around 15%. With such low response rates,
there is scope for huge variation in the type of patient who responds.
Then we come to the question of how the data are to be used. The intention is that trusts and wards will be compared on the proportion of respondents answering that they would be "likely" or "extremely likely" to recommend the department/ward. The problem with comparing simple proportions is that this takes no account of differences between the trusts/wards in the patient mix, the reasons for admission and the outcomes of treatment.
A ward specialising in a treatment that is nearly always successful and immediately reduces pain will score more highly than one dealing with more difficult problems where the effect of treatment may not yet be evident at the time of discharge, even if care standards are identical in the two wards.
Satisfaction (conceptualised in the friends and family test as the likelihood of recommending the ward to others) should be compared on differences between the inputs (the types of patients and their reasons for being in hospital). This could be done using statistical methods that control for patient characteristics, symptoms/disease and treatment. But the friends and family test will not be collecting any of this vital information. This makes it impossible to control for differences between hospitals and wards, or even to assess the extent to which controlling might matter.
There are survey organisations who know how to design and carry out a regular survey that would meet the state objectives of the friends and family test. If you think it would be expensive to bring in outside expertise, just think how much time and effort NHS trusts must be putting in to set up the test, to invite every patient to take part and to collect, analyse and publish the data. I doubt that the true costs would be greater if a single expert organisation would run the test with a genuinely standardised methodology across all trusts, with a larger set of questions, and with meaningful analysis. And the outcome might just be a system that is much more likely to identify the trusts and wards where improvements are really needed.
Professor Peter Lynn is a survey expert from the Institute for Social and Economic Research (ISER) at the University of Essex Guardian Professional.
In the wake of the Stafford hospital scandal and widespread concern about the treatment received by hospital patients, there is no doubt that new initiatives to improve care will be welcomed. Surveys can be powerful tools to quantify what goes wrong, how often, for whom and in what circumstances. Such information can be invaluable in helping to target resources and improve performance. But will the new friends and family test have the desired effect?
The survey has just one question. But if an A&E patient would not recommend this hospital, what would he or she recommend instead? To travel 50 miles to the next hospital? To avoid medical assistance completely? To wait until their GP has an appointment?
The question is hypothetical. It assumes a choice, but does not state the alternative which the patient is supposed to use as a means of comparison.
And it is unclear what patients are supposed to be assessing. Is it the standard of care by nurses? The standard of treatment? Comfort? Cleanliness? Food? It will be interpreted in vastly different ways.
The variation in how patients react to the question could distort comparisons between hospitals or wards.
Of course, these criticisms can be aimed at many survey questions, but hopefully – at least in the case of serious government or academic surveys – to a much lesser extent. The criticisms matter in this case because different hospitals, and different wards within a hospital, have different kinds of patients. The official guidance to trusts on implementing the test states that the question wording has been informed by independent research. It does not mention that the research commissioned by the Care Quality Commission concluded that the test is "not appropriate for use in an NHS setting".
Even if the question were clear, unambiguous and universally understood, the subject has connotations of social desirability. Some people think it impolite to criticise and will therefore answer "yes" to the question, even if they received worse care than someone else who answers "no". And they will be particularly likely to do so if asked by an interviewer rather than in an anonymous self-completion mode.
NHS trusts are allowed to administer the question in whatever way they see fit: by telephone, online, by SMS, at kiosks in the hospital lobby, by handing the patient a card, or in any other way. This introduces a systematic bias into comparisons between hospitals. A hospital's score will be artificially inflated if it uses interviewing rather than self-completion, for example. The NHS's own research recommended that guidance should be given both on the mode of delivery of the question and on the setting in which it should be delivered. Both these recommendations have been ignored.
Trusts are given free rein regarding how to encourage patients to take part. They are told only that the minimum response rate is expected to be around 15%. With such low response rates,
there is scope for huge variation in the type of patient who responds.
Then we come to the question of how the data are to be used. The intention is that trusts and wards will be compared on the proportion of respondents answering that they would be "likely" or "extremely likely" to recommend the department/ward. The problem with comparing simple proportions is that this takes no account of differences between the trusts/wards in the patient mix, the reasons for admission and the outcomes of treatment.
A ward specialising in a treatment that is nearly always successful and immediately reduces pain will score more highly than one dealing with more difficult problems where the effect of treatment may not yet be evident at the time of discharge, even if care standards are identical in the two wards.
Satisfaction (conceptualised in the friends and family test as the likelihood of recommending the ward to others) should be compared on differences between the inputs (the types of patients and their reasons for being in hospital). This could be done using statistical methods that control for patient characteristics, symptoms/disease and treatment. But the friends and family test will not be collecting any of this vital information. This makes it impossible to control for differences between hospitals and wards, or even to assess the extent to which controlling might matter.
There are survey organisations who know how to design and carry out a regular survey that would meet the state objectives of the friends and family test. If you think it would be expensive to bring in outside expertise, just think how much time and effort NHS trusts must be putting in to set up the test, to invite every patient to take part and to collect, analyse and publish the data. I doubt that the true costs would be greater if a single expert organisation would run the test with a genuinely standardised methodology across all trusts, with a larger set of questions, and with meaningful analysis. And the outcome might just be a system that is much more likely to identify the trusts and wards where improvements are really needed.
Professor Peter Lynn is a survey expert from the Institute for Social and Economic Research (ISER) at the University of Essex Guardian Professional.
No comments:
Post a Comment