Reliability and relevance evaluations of REACH data

Ellen Ingre-Khans; Marlene Ågerstrand; Anna Beronius; Christina Rudén;
2019 | Toxicol. Res.

Regulatory authorities rely on hazard and risk assessments performed under REACH for identifying chemicals of concern and to take action. Therefore, these assessments must be systematic and transparent. This study investigates how registrants evaluate and report data evaluations under REACH and the procedures established by the European Chemicals Agency (ECHA) to support these data evaluations. Data on the endpoint repeated dose toxicity were retrieved from the REACH registration database for 60 substances. An analysis of these data shows that the system for registrants to evaluate data and report these evaluations is neither systematic nor transparent. First, the current framework focuses on reliability, but overlooks the equally important aspect of relevance, as well as how reliability and relevance are combined for determining the adequacy of individual studies. Reliability and relevance aspects are also confused in the ECHA guidance for read-across. Second, justifications for reliability evaluations were mainly based on studies complying with GLP and test guidelines, following the Klimisch method. This may result in GLP and guideline studies being considered reliable by default and discounting non-GLP and non-test guideline data. Third, the reported rationales for reliability were frequently vague, confusing and lacking information necessary for transparency. Fourth, insufficient documentation of a study was sometimes used as a reason for judging data unreliable. Poor reporting merely affects the possibility to evaluate reliability and should be distinguished from methodological deficiencies. Consequently, ECHA is urged to improve the procedures and guidance for registrants to evaluate data under REACH to achieve systematic and transparent risk assessments.

Improving structure and transparency in reliability evaluations of data under REACH: suggestions for a systematic method

Ellen Ingre-Khans; Marlene Ågerstrand; Christina Rudén; Anna Beronius;
2019 | Hum. Ecol. Risk Assess.

The goal of identifying hazardous chemicals registered under the Registration, Evaluation, Authorization and restriction of CHemicals (REACH) Regulation and taking appropriate risk management measures relies on robust data registrations. However, the current procedures for European chemical manufacturers and importers to evaluate data under REACH neither support systematic evaluations of data nor transparently communicate these assessments. The aim of this study was to explore how using a data evaluation method with predefined criteria for reliability and establishing principles for assigning reliability categories could contribute to more structured and transparent evaluations under REACH. In total, 20 peer-reviewed studies for 15 substances registered under REACH were selected for an in-depth evaluation of reliability with the SciRAP tool. The results show that using a method for study evaluation, with clear criteria for assessing reliability and assigning studies to reliability categories, contributes to more structured and transparent reliability evaluations. Consequently, it is recommended to implement a method for evaluating data under REACH with predefined criteria and fields for documenting and justifying the assessments to increase consistency of data evaluations and transparency.

Toxicity studies used in REACH ‐ How accurately are they reported?

Ellen Ingre‐Khans; Marlene Ågerstrand; Anna Beronius; Christina Rudén;
2019 | Toxicol. Res.

Toxicity studies on chemicals registered under REACH are provided as summaries instead of submitting a full study report. Since the registration data are used by regulatory agencies to identify chemicals of concern, the study summaries must accurately reflect the information in studies. A “study summary” should include sufficient information on the objectives, methods, results and conclusions in the full study report to determine the relevance of the study. Sometimes a “robust study summary” is required, which should contain more detailed information to enable an independent assessment of the study. The aim of this investigation is to examine how well published toxicity papers were reflected in study summaries submitted by registrants under REACH. Summaries of 20 published studies (peer‐reviewed studies including one abstract) were examined and broad categories of various types of observed differences were derived. As seen, the extent to which information in the published studies was reported in the study summaries varied. It also varied how accurately the information was reflected. Differences between the published studies and the summaries included simple typing errors, unclear and incomplete reporting as well as the omission of information on for example study design, results or interpretation of the results, which in some of the cases could be considered relevant for the risk assessment. This raises concerns regarding the accuracy of study summaries and their use for decision‐making. Moreover, the possibility for third parties to independently assess and scrutinise the summaries is limited. Considering that we rely on REACH registration data for chemical safety, all data used for risk assessment should be accessible for thorough examination and fully independent assessment.

Improving environmental risk assessments of chemicals: Steps towards evidence-based ecotoxicology.

Martin Olwenn; Adams; Beasley; Belanger; Breton; Brock; Buonsante; Galay Burgos; Green; Guiney; Hall; Hanson M; Harris; Henry; Huggett; Junghans; Laskowski; Maack; Moermond; Panter; Pease; Poulsen; Roberts; Christina Rudén; Schlekat; Schoeters; Solomon; Staveley; Stubblefield; Sumpter; Warne; Wentsel; Wheeler; Wolff; Yamazaki; Zahner; Marlene Ågerstrand;
2019 | Environ Int | 128 (210-2017)

A call for action: Improve reporting of research studies to increase the scientific basis for regulatory decision‐making

Marlene Ågerstrand; Sofie Christiansen; Annika Hanberg; Christina Rudén; Lars Andersson; Sjur Andersen; Henrik Appelgren; Christine Bjørge; Ian Henning Clausen; Dag Markus Eide; Nanna B. Hartmann; Trine Husøy; Halldór Pálmar Halldórsson; Marianne van der Hagen; Ellen Ingre‐Khans; Adam David Lillicrap; Vibe Meister Beltoft; Anna‐Karin Mörk; Mari Murtomaa‐Hautala; Elsa Nielsen; Kristín Ólafsdóttir; Jaana Palomäki; Hinni Papponen; Emilie Marie Reiler; Helene Stockmann‐Juvala; Tiina Suutari; Henrik Tyle; Anna Beronius
2018 | J Appl Toxicol | 1-3

This is a call for action to scientific journals to introduce reporting requirements for toxicity and
ecotoxicity studies. Such reporting requirements will support the use of peer‐reviewed research
studies in regulatory decision‐making. Moreover, this could improve the reliability and reproducibility
of published studies in general and make better use of the resources spent in research.

Transparency of chemical risk assessment data under REACH

Ellen Ingre-Khans; Marlene Ågerstrand; Anna Beronius; Christina Rudén
2016 | Environ. Sci.: Processes Impacts (12)

The aim of this study is to scrutinize the transparency, i.e. the accessibility and comprehensibility, of information on substances registered under REACH. Data on repeated dose toxicity and hazard assessment conclusions were extracted for 60 substances from the REACH registration database available on the ECHA website. The data were compiled in a database for systematically evaluating the transparency of information related to the conclusions on hazard or risk. In addition, CSRs were requested from ECHA for five substances. The transparency of information on the hazard and risk of substances was found to be limited for several reasons. First, certain information was removed due to confidentiality and certain fields were not published because they could contain confidential information although the information had not been claimed confidential. Also, the extent to which registrants reported information varied, and the presentation of some data and certain terminology required further clarification. In addition, the data source for the majority of the key and supporting studies could not be identified due to confidentiality. Since registrants are only required to summarise studies, it cannot be verified whether all relevant information from non-public industry reports have been reported. Lastly, certain information related to the hazard and risk assessment were only reported in the CSR which is only available upon request; a time-consuming and work-intensive process. As information on registered chemicals is currently provided to the public, it is difficult to follow steps that are undertaken in the hazard and risk assessment. This limits the possibility for a third party to evaluate the assessment.

Assessing the relevance of ecotoxicological studies for regulatory decision making

Christina Rudén; Julie Adams; Marlene Ågerstrand; Theo CM Brock; Veronique Poulsen; Christian E Schlekat; James R Wheeler; Tala R Henry
2016 | Integr Environ Assess Manag

Regulatory policies in many parts of the world either recognize the utility of, or mandate, that all available studies be considered in Environmental Assessment (ERA) of chemicals, including studies from the peer-reviewed literature. Consequently, a vast array of different studies and data types need to be considered. The first steps in the evaluation process involve determining whether the study is relevant to the ERA and sufficiently reliable. Relevance evaluation is typically performed using existing guidance but involves application of 'expert judgement' by risk assessors. In this paper we reviewed published guidance for relevance evaluation and based on the practical experience within the group of authors we identified additional aspects and further developed already proposed aspects that should be considered when conducting a relevance assessment for ecotoxicological studies. The overarching key aspect of relevance concerns the ability to directly or indirectly use the study in ERA with the purpose to address specific protection goals and ultimately regulatory decision-making. Since ERA schemes are based on the appropriate linking of exposure and effect estimates, important features of ecotoxicological studies relate to exposure relevance and biological relevance. Exposure relevance addresses the representativeness of the test substance, environmental exposure media and exposure regime. Biological relevance deals with the environmental significance of the test organism and the endpoints, the ecological realism of the test conditions simulated in the study, as well as a mechanistic link of treatment-related effects for endpoints to the protection goal identified in the ERA. In addition, uncertainties associated with relevance should be considered in the assessment. A systematic and transparent assessment of relevance is needed for regulatory decision-making. The relevance aspects also need to be considered by scientists to facilitate the studies use in ERA.

Combining web‐based tools for transparent evaluation of data for risk assessment: developmental effects of bisphenol A on the mammary gland as a case study

Linda Molander; Annika Hanberg; Christina Rudén; Marlene Ågerstrand; Anna Beronius
2016 | J Appl Toxicol

Different tools have been developed that facilitate systematic and transparent evaluation and handling of toxicity data in the risk assessment process. The present paper sets out to explore the combined use of two web-based tools for study evaluation and identification of reliable data relevant to health risk assessment. For this purpose, a case study was performed using in vivo toxicity studies investigating low-dose effects of bisphenol A on mammary gland development. The reliability of the mammary gland studies was evaluated using the Science in Risk Assessment and Policy (SciRAP) criteria for toxicity studies. The Health Assessment Workspace Collaborative (HAWC) was used for characterizing and visualizing the mammary gland data in terms of type of effects investigated and reported, and the distribution of these effects within the dose interval. It was then investigated whether there was any relationship between study reliability and the type of effects reported and/or their distribution in the dose interval. The combination of the SciRAP and HAWC tools allowed for transparent evaluation and visualization of the studies investigating developmental effects of BPA on the mammary gland. The use of these tools showed that there were no apparent differences in the type of effects and their distribution in the dose interval between the five studies assessed as most reliable and the whole data set. Combining the SciRAP and HAWC tools was found to be a useful approach for evaluating in vivo toxicity studies and identifying reliable and sensitive information relevant to regulatory risk assessment of chemicals.

A proposed framework for the systematic review and integrated assessment (SYRINA) of endocrine disrupting chemicals

Laura N Vandenberg; Marlene Ågerstrand; Anna Beronius; Claire Beausoleil; Åke Bergman; Lisa A Bero; Carl-Gustaf Bornehag; C Scott Boyer; Glinda S Cooper; Ian Cotgreave; David Gee; Philippe Grandjean; Kathryn Z Guyton; Ulla Hass; Jerrold J Heindel; Susan Jobling; Karen A Kidd; Andreas Kortenkamp; Malcolm R Macleod; Olwenn V Martin; Ulf Norinder; Martin Scheringer; Kristina A Thayer; Jorma Toppari; Paul Whaley; Tracey J Woodruff; Christina Rudén
2016 | Environ Health | 15 (74) (1-19)

Background
The issue of endocrine disrupting chemicals (EDCs) is receiving wide attention from both the scientific and regulatory communities. Recent analyses of the EDC literature have been criticized for failing to use transparent and objective approaches to draw conclusions about the strength of evidence linking EDC exposures to adverse health or environmental outcomes. Systematic review methodologies are ideal for addressing this issue as they provide transparent and consistent approaches to study selection and evaluation. Objective methods are needed for integrating the multiple streams of evidence (epidemiology, wildlife, laboratory animal, in vitro, and in silico data) that are relevant in assessing EDCs.

Methods
We have developed a framework for the systematic review and integrated assessment (SYRINA) of EDC studies. The framework was designed for use with the International Program on Chemical Safety (IPCS) and World Health Organization (WHO) definition of an EDC, which requires appraisal of evidence regarding 1) association between exposure and an adverse effect, 2) association between exposure and endocrine disrupting activity, and 3) a plausible link between the adverse effect and the endocrine disrupting activity.

Results
Building from existing methodologies for evaluating and synthesizing evidence, the SYRINA framework includes seven steps: 1) Formulate the problem; 2) Develop the review protocol; 3) Identify relevant evidence; 4) Evaluate evidence from individual studies; 5) Summarize and evaluate each stream of evidence; 6) Integrate evidence across all streams; 7) Draw conclusions, make recommendations, and evaluate uncertainties. The proposed method is tailored to the IPCS/WHO definition of an EDC but offers flexibility for use in the context of other definitions of EDCs.

Conclusions
When using the SYRINA framework, the overall objective is to provide the evidence base needed to support decision making, including any action to avoid/minimise potential adverse effects of exposures. This framework allows for the evaluation and synthesis of evidence from multiple evidence streams. Finally, a decision regarding regulatory action is not only dependent on the strength of evidence, but also the consequences of action/inaction, e.g. limited or weak evidence may be sufficient to justify action if consequences are serious or irreversible.

The dilemma in prioritizing chemicals for environmental analysis: known versus unknown hazards

2016 | Environ. Sci.: Processes Impacts

Science in Risk Assessment and Policy (SciRAP): an online resource for evaluating and reporting in vivo (eco) toxicity studies

Linda Molander; Marlene Ågerstrand; Anna Beronius; Annika Hanberg; Christina Rudén
2015 | Hum. Ecol. Risk Assess. | 21 (3) (753-762)

(Eco)toxicity studies conducted according to internationally standardized test guidelines are often considered reliable by default and preferred as key evidence in regulatory risk assessment. At the same time regulatory agencies emphasize the use of all relevant (eco)toxicity data in the risk assessment process, including non-standard studies. However, there is a need to facilitate the use of such studies in regulatory risk assessment. Therefore, we propose a framework that facilitates a systematic and transparent evaluation of the reliability and relevance of (eco)toxicity in vivo studies for health and environmental risk assessment. The framework includes specific criteria to guide study evaluation, as well as a color-coding tool developed to aid the application of these criteria. In addition we provide guidance intended for researchers on how to report non-standard studies to ensure that they meet regulatory requirements. The intention of the evaluating and reporting criteria is to increase the usability of all relevant data that may fill information gaps in chemical risk assessments. The framework is publically available online, free of charge, at the Science in Risk Assessment and Policy (SciRAP) website: www.scirap.org. The aim of this article is to present the framework and resources available at the SciRAP website.

Bad reporting or bad science? Systematic data evaluation as a means to improve the use of peer-reviewed studies in risk assessments of chemicals

2014 | Hum. Ecol. Risk Assess. | 20 (6) (1427-1445)

In this study we assess the applicability of a set of reliability criteria proposed by Ågerstrand et al. This was done by evaluating the reliability of 12 non-standard peer-reviewed ecotoxicity and toxicity studies for Bisphenol A. There was an overall agreement between the evaluator and the authors of the papers regarding the result of the evaluations. This suggests that the criteria offer enough guidance to be a useful and consistent evaluation tool. It provides a transparent and structured approach, and ensures that a minimum and similar set of criteria is used. The evaluation of the peer-reviewed ecotoxicity and toxicity studies concludes that important information is sometimes missing, and therefore the studies do not always meet common regulatory requirements regarding reporting. Whether this is due to insufficient reporting or due to poorly performed studies is not known. To improve the reporting, and thereby promote reliability and reproducibility, researchers, reviewers, and editors are recommended to use the suggested criteria as a guideline. In conclusion, in order to improve the reliability of peer-reviewed studies, and to increase their use in regulatory risk assessments of chemicals, the dialog between regulators, researchers, and editors regarding how to evaluate and report studies needs to be strengthened.

Contact information

Visiting addresses:

Geovetenskapens Hus,
Svante Arrhenius väg 8, Stockholm

Arrheniuslaboratoriet, Svante Arrhenius väg 16, Stockholm (Unit for Analytical and Toxicological Chemistry)

Mailing address:
Department of Environmental Science and Analytical Chemistry (ACES)
Stockholm University
106 91 Stockholm

Press enquiries should be directed to:

Stella Papadopoulou
Science Communicator
Phone +46 (0)8 674 70 11
stella.papadopoulou@aces.su.se