Quality assurance is a pre-requisite for operational forensic genetic laboratories and professional organisations such as ENFSI, SWGDAM or ISFG. These organisations provide guidelines/recommendations for the methods used (both for analysis and evaluation), and for reporting. Aspects regarding analyses will generally be found in the accreditation scope, contrary to evaluation. Moreover, proficiency testing on how to evaluate and report results is still in its infancy. This is problematic as the quality of forensic science services for the administration of justice crucially depends on how results are evaluated and conveyed.
In this paper, we use examples of written and oral statements to illustrate how accreditation could improve and ensure that guidelines on interpretation are properly implemented, reviewed and maintained. We underline the need for certifying and validating the knowledge of the experts in that specific field. Indeed, analytical techniques have for many years been the focus of accreditation, rather than the soundness and logic of communication of the value of the results obtained with these techniques. It is now time for a shift and to provide means that ensure that forensic scientists’ conclusions are as justifiable as their analytical methods.
The worst consequence associated with problematic evaluation and reporting of DNA evidence is a wrongful conviction. Notorious cases involving DNA have been reported in the literature [
Inside the Cell: The Dark Side of Forensic DNA.
], in the news, in official reports [
Inquiry into the Circumstances That Led to the Conviction of Mr Farah Abdulkadir Jama.
Report Into the Circumstances of a Complaint Received From the Greater Manchester Police on 7 March 2012 Regarding DNA Evidence Provided by LGC Forensics. Mr Andrew Rennison MSc 17 September 2012 FSR-R-618 Crown Copyright.
]. Raising public awareness of the complications associated with evaluation and reporting is important, but is not the best forum to foster improvement. Quality assurance measures can also be put in place to prevent such threats and ensure that the best quality service is provided to the justice system. These measures are part of an accreditation scheme. But the heart of the matter is that there are still many laboratories for which evaluation and reporting is not in their scope of accreditation. Detailed procedures guiding the evaluation of the results (statistically or otherwise) and reporting are absent. It is as if opinions and interpretations (O&I in ISO jargon) were left out of the equation. In this short communication, we critically expose the reasons why it is important to make evaluative reporting part of the accreditation scope of all forensic DNA laboratories. We show examples of problematic statements provided in court or in reports. We then illustrate how quality assurance measures could mitigate the potential drawbacks associated with such a behaviour. We conclude by discussing practical aspects on how to ensure that the value of the results is communicated in court according to the best standards.
2. Why are there still only very few laboratories that include evaluation and reporting in the scope of their accreditation?
An idea persisting in forensic science since decades is that evaluation and reporting are personal and do not need to follow prescribed methods (contrary to analytical methods). This is illustrated by the fact that there are recommendations for evaluation and reporting, there are guidelines, but there are very few binding standards. This is also reflected in proficiency testing where, often, the evaluation part is optional. On the other hand, there are collaborative exercises and so-called interpretation challenges that testify to the importance of the post-analytical phase. We note that the degree of participation to collaborative exercises is still lower for evaluation than for the analytical exercises. For example [
GHEP-ISFG collaborative exercise on mixture profiles (GHEP-MIX06). Reporting conclusions: results and evaluation.
], in the collaborative exercise of the laboratories from the Spanish and Portuguese Speaking Working Group (GHEP-ISFG), 25 laboratories participated to the collaborative exercise on reporting and 120 participated to the analytical exercise. For the GEDNAP, about 50% participated to the evaluation exercise and for the English-speaking group of the ISFG in 2017, about 80% of the laboratories participated to the interpretation exercise, which is more encouraging.
No laboratory would consider (nor be allowed) not to be accredited for daily methods of DNA extraction and analysis. Quality assurance is a pre-requisite for forensic genetic laboratories and for professional organisations such as ENFSI, SWGDAM or ISFG. But, why not make evaluation and delivery of the end product mandatory part of the accreditation scope? We argue here that these last steps are too important to be left to the discretion of the reporting officers and, thus, should be placed under a proper quality assurance umbrella.
It is important to stress that while some laboratories have added these components to their scope, the practice is patchy. Forensic laboratories operating in an adversarial judicial system may be more prone to deal with the issue than laboratories operating under an inquisitorial system. It is as if the efforts towards quality were more dictated by the prospect of being challenged in court than by the pursuit of professionalism.
3. Examples of problematic reporting
One answer to the question “Why isn’t evaluation and reporting part of the scope?” could be that evaluation and reporting do not have a high impact in the case and that the dangers associated with problematic evaluation and reporting are minimal. This argument does not hold, however, as there have been cases of wrongful convictions or misleading DNA evidence – despite the fact that DNA has also proved to be a very powerful exculpatory tool. Illustrative examples are the widely known Adam Scott case, the Jama case, or the case involving the late David Butler. A very common point of miscommunication is when scientists give their opinion on the source of the DNA, instead of giving their opinion on the results only. This fallacy has been known for more than 30 years. It is disconcerting, thus, that one still encounters it in statements or in the reported results of proficiency tests.
Not having any mandatory requirements for proper reporting scheme is the open door to claims that are unscientific, but intriguingly convincing. A typical example for this is the following, drawn from the official report on the Jama case:
“It was 800 billion times more likely that the sample originated from the accused rather than an unknown person. In other words, it would appear to be necessary to search well beyond this planet and conceivably this galaxy to find a match.” [2, p. 40].
Another example of this fallacy, but in the context of activity level propositions, can also be found in casework, as shown in the example below:
"The expert evidence was that the likely reason for the defendant's DNA profile being on the door handle was that he had touched it". [
R.v Tsekiri, EWCA Crim 40 (17 February 2017).
4. Can technology help towards consistency, harmonisation and quality?
With the generalisation of probabilistic genotyping software, collaborative exercises have been initiated [
STRmixTM collaborative exercise on DNA mixture interpretation.
Euroforgen-NoE collaborative exercise on LRmix to demonstrate standardization of the interpretation of complex DNA profiles.
], guidelines and even standards have been generated (ISFG, For. Reg., SWGDAM). This is a good side-effect of technology; it enables standardisation and facilitated change and progress.
There are now a few laboratories that, in their accreditation scope, specify how they are to assess mixed and unmixed DNA profiles given sub-source level propositions using dedicated software.
How to monitor the performance of probabilistic genotyping software needs to be addressed by organisations who provide proficiency tests (PT). Professional organisations should act as an advisory group to liaison with PT providers. Indeed, guidance regarding the criteria against which labs would be judged, as well as the objective of the PT and expected outcomes will be needed to ensure that these tests are appropriate for accreditation purposes. PT providers should also ensure that the certificates and tests provided fulfil the criteria needed to achieve accreditation [e.g., [
Accreditation of the GHEP-ISFG proficiency test: one step forward to assure and improve quality.
5. The new norm: assessing the risks of problematic reporting
The standard ISO/IEC 17025: 2017, in §8.5.1 states that “The laboratory shall consider the risks and opportunities associated with the laboratory activities
” and in §8.5.3 that “Actions taken to address risks and opportunities shall be proportional to the potential impact on the validity of laboratory results
”. Cases have shown that if the results are not assessed and reported in a logical way, then this can pose a serious danger. It threatens the integrity and reputation of institutions, and places both individual defendants and the society as a whole into a vulnerable position. While this point should be obvious, the 'UKAS Guidance on the Application of ISO/IEC 17025:2017 Dealing with Expressions of Opinions and Interpretations' has indicated, in §2.3, that expression of opinions and interpretations relating to results is considered to be an inherent part of testing/calibration. Thus, it is of great importance that standards - rather than recommendations or guidelines - should be adopted as early as possible [
The need for reporting standards in forensic science Law.
]. That way, technical assessors can ensure that results are reported according to appropriate standards.
6. Education and certification
In court, threats are even more imminent, as errors of interpretation will usually not be detected by the parties. Having personnel with appropriate knowledge and evaluative expertise should help ensure that the value of the results is properly communicated. ISO/IEC 17025:2017, §220.127.116.11, specifies that “When opinions and interpretations are expressed, the laboratory shall ensure that only personnel authorized for the expression of opinions and interpretations release the respective statement.
” For the personnel to be authorised, there is a need for formal education in evaluation and reporting [
E-learning initiatives in forensic interpretation: report on experiences from current projects and outlook.
]. While it may be difficult to have proficiency testing that would ask for an entire report, or for court-like testimonies, these aspects can be monitored through formal education and audit. Some countries rely in addition on expert registers (e.g., NRGD [
Quality Management in Forensic Science.
]) and are exploring ways to certify experts in specific fields (such as reporting given activity level propositions).
To ensure that courts are given appropriate and robust information regarding the value of DNA results, it is necessary to have ‘Opinions and interpretation’ as part of DNA accreditation scope. We acknowledge that some forensic laboratories (especially when operating in an adversarial system) have already made significant steps towards that proposal. Our point is that it should become mandatory for all forensic providers (both when considering sub-source and activity level propositions) regardless of the nature of their judicial system.
By doing so, forensic laboratories will fully embrace the risk eliciting and mitigating approach at the core of the new ISO/IEC 17025:2017 standard.
As a practical implication, this proposal also highlights the need for (i) technical assessors who are proficient in DNA interpretation and (ii) proficiency testing that is fit for purpose both from a scientific point of view and from an ISO normative perspective.
Alex Biedermann gratefully acknowledges the support of the Swiss National Science Foundation through grant No. BSSGI0_155809."
Inside the Cell: The Dark Side of Forensic DNA. Nation Books,
Inquiry into the Circumstances That Led to the Conviction of Mr Farah Abdulkadir Jama. Victorian Government Printer,
Report Into the Circumstances of a Complaint Received From the Greater Manchester Police on 7 March 2012 Regarding DNA Evidence Provided by LGC Forensics. Mr Andrew Rennison MSc 17 September 2012 FSR-R-618 Crown Copyright.2012
GHEP-ISFG collaborative exercise on mixture profiles (GHEP-MIX06). Reporting conclusions: results and evaluation.Forensic Sci. Int. Genet. 2018; 35: 156-163
R.v Tsekiri, EWCA Crim 40 (17 February 2017).
STRmixTM collaborative exercise on DNA mixture interpretation.Forensic Sci. Int. Genet. 2019; 40: 1-8
Euroforgen-NoE collaborative exercise on LRmix to demonstrate standardization of the interpretation of complex DNA profiles.Forensic Sci. Int. Genet. 2014; 9: 47-54
Accreditation of the GHEP-ISFG proficiency test: one step forward to assure and improve quality.Forensic Sci. Int. Genet. Suppl. Ser. 2015; 5: e515-e517
The need for reporting standards in forensic science Law.Law Probab. Risk. 2015; 14: 169-173
E-learning initiatives in forensic interpretation: report on experiences from current projects and outlook.Forensic Sci. Int. 2013; 230: 2-7
Quality Management in Forensic Science.1st edition. Academic Press,
Published online: September 26, 2019
© 2019 Elsevier B.V. All rights reserved.