When Science is Bad Science
So you managed to enter the magic world of science and you are happier than ever. As you realize your dream has come true, the celebrations still continue. Your friends and family rejoice in your great enthusiasm, even though they do not fully understand how one can aspire to dedicate the majority of their days formulating problems that unveil new problems — a cascade of inquiries that leads to a cloud of uncertainties rather than solutions, yielding more questions than answers. Surprisingly for them, this is exactly what you love about your job. You are driven by passion, sincere curiosity and intellectual honesty. You have the privilege of being paid to ponder (yes, to ponder!), a delight that brings a large smile to your face as you step into your new office. You tell your friends that science uncovers the equations governing the world, embarking on an audacious journey that rejects dogma and preconceptions, embracing doubt just as Bertolt Brecht said. In the magic world of science, a theory remains valid until proven otherwise, thank you Dr. Popper! Scientists, in your portrayal, resemble a kind of ‘overmen’ (and maybe, if you’re unbiased enough, ‘overwomen’, or something akin), these enchanting beings driven by curiosity, impervious to the temptation of wealth and success, standing as heroic figures who correct their beliefs when proven wrong and readily offer apologies when needed. Then you read about the resignation of Stanford University president Marc Tessier-Lavigne and you suddenly woke up: data falsification in articles he co-authored; his reluctance to rectify or withdraw the implicated articles; his inclination towards favoring the ‘winners’ (conveyors of positive results) over the ‘losers’ (conveyors of negative results) within his lab. An H-index of 133 should guarantee a certain credibility, isn’t it? Oh, I know how you feel. You are now tempted to consider this event as an isolated incident. Don’t. The sooner you understand that bad science is a realty, the better (not convinced yet? check here). The sooner you learn to identify bad scientists, the better. Why? Well, because if you know them you avoid them (spoiler: not easy at all!).
Below is a brief guide to bad science practices. I will continuously update it as there is no limit to the human fantasy when it comes to bad practices.
Manipulation of results
Photoshop can do the magic with a plot of data. Apparently, the manipulation of results stands out as one of the most pervasive unethical practices within the scientific community. Robert Schneider is an investigative science journalist specialising in academic publishing and research integrity. His website meticulously documents instances of misconduct within academia. Among the array of tags, ‘manipulation’ stands out as the most frequently referenced in its blog posts. That says a lot. A simple click on this tag reveals an extensive compendium of published cases involving manipulated data (below is an example).
Lack of scientific rigor/Sensationalistic headlines
Let’s take the earthquakes. It is highly probable that successfully predicting them would warrant the Nobel Prize. To provide context: the scientific problem of describing brittle fracture phenomena — and thus the rupture process at the origin of the earthquake process— incorporates even greater complexities than those involved in the theory of fluid turbulence, one of the seven mathematical problems of the millennium. Thus, when you see an article titled something like “Eureka! X is an earthquake precursor!” or “Wow! Y triggers earthquakes!”, go in the kitchen, take a cup of coffee or tea and recall that sensationalism contradicts the very essence of scientific humility. If any of these scientists had really found an earthquake precursor, they would have revealed time, location and magnitude of the impending earthquakes (because ‘predicting’ means to state that something is gonna happen BEFORE it actually happens). Go on Google Scholar and type ‘earthquakes X’, where X is a precursor of your choice (solar activity, tides, lunar phases, radon etc.). Many of these studies (note: they are highly cited articles) have been revealed to lack statistical rigor (i.e. [1], [2], [3], [4]). Often, correlation is confused with causation (a touch of self-citation, to smoothly introduce you to the next bad practice). NOTE: no nihilism involved here. We have both the capability and the responsibility to mitigate the impact of large earthquakes [5]. Also, research about earthquake precursors must persevere for both its scientific and societal importance. Certain reported anomalies, derived from rigorous and unbiased analyses, need closer scrutiny. Keep in mind that this post focuses on flawed science only.
Self-citations
This occurs when a scientist references their own work or encourages others to cite their article ([6] and references therein). While not inherently wrong, this behavior turns into a negative practice when self-citations are unnecessarily abundant, or when it becomes a coercive requirement imposed to others (quite common in imbalanced supervisor/student relationships or during the peer-review process). A notable case emerges from Italy, where researchers have started to resort much more often to the self-citations practice after the implementation of national regulations in 2011 that introduced new bibliometric metrics ([7], [8]). It’s worth noting the role of solidarity (!) into this game: to help each other navigating the challenging terrain of bibliometric indicators, some Italian researchers started to resort to the ‘citation-cartels’ as well (also known as ‘citation-clubs’). Jump to the next bad practice to know more about them.
Citations cartels
A citation cartel is a group of authors that cite each other significantly more compared to the citations they get from authors outside of their group. Not so easy to be spotted, Dr. Iztok Fister Jr gives a hint:
If you see a paper authored by X that cites an author or a group of authors 10 times, and then you look at those 10 papers and see that they are citing previous papers of author X equally ferociously, it is worth a deeper inspection.
This practice is especially prevalent in smaller, self-referential scientific domains. Intuitively, it stands out as one of the simplest methods to boost one’s citation count. Good news is that new tools are becoming available to discover citation cartels [9]. An example is shown in the figure below.
Salami-slicing and duplicate publication
A significant challenge faced by the scientific community is the exponential growth of the volume of scientific literature (a growth that does not translate into an exponential increase in our understanding of the world). This issue is exacerbated by two bad practices: salami-slicing publications and duplicate publications. The first one consists in fragmenting research outcomes into multiple publications. Instead of delivering one publication only, some scientists split it into a collection of “minimal publishable units” [10]. Instances of salami-slicing encompass scenarios such as sharing identical scientific questions and methodologies, presenting closely related scientific questions that could have been addressed together, reporting only a single outcome from a set of experiment results, or making minor, sometimes trivial, modifications to specific aspects of the study, such as research questions, methodologies, or subsets of data.
On the other hand, duplicate publication occurs when a researcher substantially duplicates content from a previously published work, typically without proper acknowledgment. Duplication can manifest as republishing the same paper in a different journal or reusing specific text, datasets and plots from a published work to create a new paper (more at this link).
Intuitively, researchers resort to both of these bad practices in an effort to increase their record of published publications and enhance the chances of receiving citations (including, as we’ve seen, self-citations).
Recommended action by COPE for Journal Editors:
- Suspected redundant (duplicate) publication in a submitted manuscript
- Suspected redundant (duplicate) publication in a published article
Cherry-picking
Cherry-picking typically arises when a particular (often limited) dataset is chosen to validate a model, substantiate a hypothesis or conduct a statistical test [12]. The inherent complexity of the described phenomenon is intentionally ignored if the experiment’s limitations — the dataset’s small size and the underlying assumptions — are disregarded when interpreting the results. Those who extrapolate conclusions from a limited set of measurements to validate the success of their experiment are not doing good science. A hypothesis supported solely by outliers or localized trend inversions lacks true validation. A statistical test based on 10 data is not a statistical test, is crap. A data-driven model that fits the data used to build it is not a model, is crap.
Ghostwriting
When a scientist attaches their name to an article they haven’t authored directly, the ‘true’ author is then called the ‘ghostwriter’. Ghostwriting is evident in instances where articles are entirely penned by industry representatives, often with the aim of promoting products and aiding regulatory submissions [13]. This form of content generation serves industry goals. In another scenario, mentors or advisors may aid their protégés, typically early-career scientists, in writing or rewriting their crappy articles to boost their visibility and scientific standing. This scenario illustrates a potentially harmful interdependence, where the supervisor owes something to the supervised and vice versa.
The researcher bias
Imagine you have built your entire career around Model X. Your Google Scholar profile is full of articles proclaiming that Model X not only works but it works pretty well. Imagine now that a bunch of newly arrived articles prove your model wrong, unapplicable or applicable to a very small set of data or under very strict assumptions (how dare they ?!). You hold a special reverence for Popper: his books are on your shelf and his picture has adorned many of your presentations at conferences worldwide. So you know what to do: take a step back and make a public declaration in which you state you’ve changed your mind on Model X. But you have built your entire career on Model X! Chances are — and they are high- that you will continue championing it until the end of your day, regardless of mounting evidence against it. Well, if that’s the case, you are not doing good science. You are affected by the ‘researcher bias’, which basically implies that you believe to be right even when you are proven wrong. That’s a bad practice. Understandable, but still bad.
Why scientists should feel bad about bad practices
Many researchers who perpetrate unethical practices often attempt to justify their actions using the “publish or perish” imperative. However, such justification doesn’t prove they are less wrong, particularly if one considers that numerous other scientists, who are also under the pressure of the “publish or perish” imperative, do not resort to these bad practices. If we base our considerations on the H-index only (questionable but useful to simplify the problem), many present-day scientists would surpass the performance of Albert Einstein, who had only 1564 citations over 147 articles at the time of his death! So, there’s something weird going on within the scientific community.
One might argue that this is all about narcissists and liars. After all, such traits exist everywhere, so why should the scientific community be an exception? However, the intent of this post isn’t to depict the psychological traits of scientists. Rather, is to spotlight bad practices in science together with their consequences. Yes, you guessed it well, you’ve got another list.
Consequences of bad practices in science
- impact on metrics: practices like self-citations, citation cartels, and salami-slicing artificially inflate researchers’ bibliometrics, corroding trust within the scientific community. Some people are questioning the use of H-index and they may have a point [14].
- resources draining: the costs of publication multiply, considering that publishing multiple articles is much more expensive than publishing a single one. If the publication costs rely on public funding (and that's often the case) this raises ethical concerns regarding an improper use of fundings.
- misleading literature: scientific literature is becoming incredibly full of redundant information, making it difficult for other researchers to identify meaningful findings.
- loss of public trust: bad practices in science can erode public trust in the scientific community and in research findings. In the era of fake news, where charlatans obtain more visibility than ever, this is incredibly dangerous.
- conflict of interests: when it comes to the researcher bias, a prominent researcher or professor may direct funding exclusively towards projects aligning with their own research record. Research proposals deviating from their viewpoint may struggle to pass peer review.
- undermining reproducibility and reliability: data manipulation, cherry-picking and other similar bad practices can lead to irreproducible results. Results published on open-access journals and generated with open-source codes are useless if they cannot be replicated or if the code cannot be effectively reused.
- negative impact on early-career researchers: unethical practices can create a hostile environment for early-career researchers. They may feel the pressure to conform to such practices in order to advance their careers. If they don’t, they may face marginalization within their research community.
- hampering innovation: unethical practices can discourage innovative thinking and exploration of new ideas. Those who dare could be marginalized. Check what’s happening to Prof. Avi Loeb, Baird Professor of Science and Institute director at Harvard University:
Since bad practices in science have consequences both within and outside the scientific community, it is a matter of responsibility to face this problem. Recognizing it is the first step for contributing to a better science.
Now go back to your work, you have so many problems to solve and so many questions to answer! As long as you keep the enthusiasm and the euphoria, the honest curiosity, you’re doing good science. And that’s kind of magic, you’re perfectly right.
REFERENCES
[1] Vidale, J. E., Agnew, D. C., Johnston, M. J., & Oppenheimer, D. H. (1998). Absence of earthquake correlation with Earth tides: An indication of high preseismic fault stress rate. Journal of Geophysical Research: Solid Earth, 103(B10), 24567–24572.
[2] Kagan, Y., 1999. The universality of the frequency-moment relationship, Pure appl. Geophys., 155, 537–574.
[3] Jordan, T., Chen, Y. T., Gasparini, P., Madariaga, R., Main, I., Marzocchi, W., … & Zschau, J. (2011). Operational Earthquake Forecasting: State of Knowledge and Guidelines for Implementation. Annals of Geophysics.
[4] Kato, M. (2019). On the apparently inappropriate use of multiple hypothesis testing in earthquake prediction studies. Seismological Research Letters, 90(3), 1330–1334.
[5] Kagan, Y. Y. (1997). Are earthquakes predictable?. Geophysical Journal International, 131(3), 505–525.
[6] Ioannidis, J. P. (2015). A generalized view of self-citation: Direct, co-author, collaborative, and coercive induced self-citation. Journal of psychosomatic research, 78(1), 7–11.
[7] Baccini, A., De Nicolao, G., & Petrovich, E. (2019). Citation gaming induced by bibliometric evaluation: A country-level comparative analysis. PLoS One, 14(9), e0221212.
[8] Seeber, M., Cattaneo, M., Meoli, M., & Malighetti, P. (2019). Self-citations as strategic response to the use of metrics for career decisions. Research Policy, 48(2), 478–491.
[9] Fister Jr, I., Fister, I., & Perc, M. (2016). Toward the discovery of citation cartels in citation networks. Frontiers in Physics, 4, 49.
[10] Broad WJ. The publication game: getting more for less. Science. 1981;211:1137–9.
[11] Bornmann, L., Haunschild, R., & Mutz, R. (2021). Growth rates of modern science: a latent piecewise growth curve approach to model publication numbers from established and new literature databases. Humanities and Social Sciences Communications, 8(1), 1–15.
[12] Morse, J. M. (2010). “Cherry picking”: Writing from thin data. Qualitative health research, 20(1), 3–3.
[13] Yadav, S., & Rawal, G. (2018). Ghostwriters in the scientific world. Pan African Medical Journal, 30(1).
[14] Engqvist, L., & Frommen, J. G. (2008). The h-index and self-citations. Trends in ecology & evolution, 23(5), 250–252.
