Applied Ecology Resources promote evidence-based decision making by curating a wide range of information sources including grey literature – but how can researchers and practitioners use this resource in their work? AER Chair Marc Cadotte provides the following advice.
Information is key for evaluating the evidence base underpinning scientific understanding, and for developing effective and efficient management of ecosystems. Yet, the primary sources of information used varies across key stakeholders; from a near exclusive reliance on peer-reviewed research articles by scientists to a diversity of informal knowledge networks used by practitioners for applied management.
With the exception of the peer-reviewed literature, these diverse sources of information are available from a variety of platforms including government websites, industry newsletters and NGO annual reports, and they are currently poorly captured in repositories, generally not searchable and frequently disappearing from webpages (i.e. link rot).
This is what Applied Ecology Resources aims to provide, a repository of applied ecology and conservation grey literature that is permanent, discoverable and searchable.
What is grey literature | Why use grey literature | Evaluating grey literature | Incorporating grey literature as an evidence base
What is grey literature exactly?
Grey literature is a commonly used term referring to factual and research-based material produced outside of traditional commercial and academic communication channels (i.e. journals and books), and, importantly, captures a huge diversity of material1,2. According to GreyNet International3, there are dozens of different types of documents or information sources that qualify as grey literature, including booklets, blogs, datasets, conference proceedings, newsletters, preprints, proposals and reports, to name a few.
It is important to note that grey literature is usually not peer-reviewed – though in some cases it is (e.g. panel review, review by inter-agency scientists, etc.) – and so puts greater onus on the user to evaluate the validity of the grey literature source (see: Evaluating grey literature). And while most scientists rightly disseminate their research outputs through peer-reviewed venues, there might be reasons to use non-peer-reviewed options at times.
Why use grey literature?
Grey literature often contains extremely valuable information that can provide an important evidentiary basis to evaluate applied actions, methodologies and policy4. There are a number of benefits associated with including grey literature in literature reviews and evidence assessments, including:
1. Grey literature might not have the same time-lag that results from the peer-reviewed process and so can supply cutting-edge or time sensitive information (e.g. a large amount of COVID-19 research is being published on preprint servers and cited because of the immediacy of the need for information5).
2. Grey literature is less likely to suffer from publication biases that are persistently present in peer-reviewed literature, where researchers are more likely to work on projects and publish results that exhibit stronger and more definitive findings6. This means that including grey literature into reviews and meta-analyses might provide less biased evidence.
3. Grey literature is much more likely to advance and evaluate the effectiveness of methodologies, especially for applications in non-experimental settings, than peer-reviewed literature.
4. Grey literature more commonly explores and develops policy implications and tools.
Evaluating grey literature
It is essential to always evaluate and assess the validity of the information before using it, whether in developing management plans, policy, hypothesis generation, literature reviews or meta-analyses, there is an especially important onus on the user to carefully evaluate grey literature.
Since grey literature has usually not been peer-reviewed (to the same extent as journal articles; though there might have been interagency review, which is important to note), the user, in some ways, needs to think like a reviewer and critically assess methodology and inferences, as well as the management and policy implications to ensure that there aren’t obvious biases or agendas. Further, finding and incorporating grey literature requires more time and effort than with traditional peer-reviewed articles4, but the payoff can be substantial.
Hopefully, with the advent of Applied Ecology Resources, finding applied ecology and conservation grey literature will be easier and repeatable.
Fortunately, there are methodologies to guide the evaluation of grey literature. For example, the AACODS checklist7 provides a great framework to guide a user’s assessment of grey literature. AACODS stands for Authority, Accuracy, Coverage, Objectivity, Date and Significance. The checklist lists a series of questions under each of these headings, and requires the user to assess aspects like whether the author is from a reputable organisation, whether the aims and methodology are clearly stated, and whether the date is included.
Examples of the questions a researcher should assess, according to the AACODS checklist include:
- Who produced the document? Is it from a reputable organisation?
- Does it cite other sources?
- When was it produced? How recent is the information?
- Does it clearly state transparent methodology?
- Does the work seem balanced in its presentation?
At the end of the day, the onus is on the user to assess and evaluate the validity of the grey literature material they wish to use in reviews or policy development. The organisation and the goals of the document should be clearly presented.
For example, a report purporting to dispel the myths of climate change alarmists and subsequent web searches fail to uncover who funds the publishing organisation should cast serious doubt on the validity of the material. A report that clearly states the goal of evaluating the effectiveness of different control measures to reduce invasive plant abundance from a government agency tasked with environmental management should be seen as credible.
To help users perform these checks, all documents archived on AER clearly list the source (who produced it), the date and users can also browse and filter by our member organisations.
Incorporating grey literature as an evidence base
Even when the AACODS checklist supports the use of grey literature articles, this material still should be handled a bit differently than peer-reviewed material. Whether evaluating grey literature for policy development, systematic review or meta-analysis, the following steps should be followed as best practice for incorporating grey literature into your work:
1. The methodology you employ should be clearly described. How grey literature was found (e.g. search terms, databases used, etc.) and the criteria used to appraise validity (e.g. AACODS checklist) need to be recorded.
2. You should include a supplemental file that lists the materials used. This includes not only bibliographic details, but also a DOI or other permanent link, and perhaps a scoring system (e.g. adding up AACODS questions) that quantifies reliability (e.g. a score of ‘5’ as very reliable and ‘1’ as questionable).
3. Make sure the resource is searchable and accessible. If grey literature was found through informal information networks, or the material obtained is not contained in a permanent repository, then the researcher should endeavour to make this material available, by working with the authoring agency or researchers to upload it to a searchable and free-to-access resource, like Applied Ecology Resources. This would generate a DOI or permanent link that then could be included in the bibliographic information.
4. Include sensitivity analysis. Results/interpretation should include sensitivity analysis where inferences are compared to information subsets without grey literature and grey literature only to evaluate how this material alters outcomes and interpretation.
5. Provide a qualitative reflection about the perceived value/concerns from your review of grey literature in the discussion section of your article/report. Much like how it is becoming standard practice for researchers reporting a review or meta-analyses to discuss possible biases that have emerged (e.g. geographical biases), the use of grey literature should similarly be reflected upon. Further, suggestions for improved discoverability or processing of grey literature could be included.
Final thoughts
Grey literature, for too long, has been accumulating on the periphery of research and policy communication. But the realisation that there are many thousands of such documents being produced that contain useful and valuable information means that researchers and practitioners should be evaluating it for a more thorough evidence base upon which to develop applied actions. Much like the medical sciences, applied ecology and conservation research and policy needs a more open and systematic way to access and incorporate grey literature.
The full searchable Applied Ecology Resources database will be launching later this year. Find out more about AER and sign up now to become a Founding Member organisation.
References
1Penn Libraries. 2019. Grey Literature in the Health Sciences: Overview, https://guides.library.upenn.edu/c.php?g=475317&p=3254238.
2Monash University Library. 2020. Grey Literature, https://guides.lib.monash.edu/grey-literature/home.
3GreyNet International. 2004. Document Types in Grey Literature, http://www.greynet.org/greysourceindex/documenttypes.html
4Adams, J., F. C. Hillier-Brown, H. J. Moore, A. A. Lake, V. Araujo-Soares, M. White & C. Summerbell (2016) Searching and synthesising ‘grey literature’and ‘grey information’in public health: critical reflections on three case studies. Systematic reviews, 5, 164.
5Latif, S., M. Usman, S. Manzoor, W. Iqbal, J. Qadir, G. Tyson, I. Castro, A. Razi, M. N. K. Boulos & A. Weller (2020) Leveraging Data Science To Combat COVID-19: A Comprehensive Review.
6Rothstein, H. R., A. J. Sutton & M. Borenstein (2005) Publication bias in meta-analysis. Publication bias in meta-analysis: Prevention, assessment and adjustments, 1-7.
7Tyndall, J. 2010. The AACODS checklist. Flinders University.
The Applied Ecologist 