Journal Club: The tricky business of research on medicine quality

Hodges and Garnett describe the tricks academics use to tell stories about fake medicines, though evidence is slim. (They use all the same tricks in their own paper.)

Journal Club: The tricky business of research on medicine quality

Catching up on reading about medicine quality, I today ploughed through a paper by Sarah Hodges and Emma Garnett, about evidence gaps in research around fake drugs. It’s got a lovely title: “The ghost in the data: Evidence gaps and the problem of fake drugs in global health research“, and it’s a fascinating read, on many levels.

The paper claims that:

  • Most academic and grey literature relating to fake drugs underlines that the evidence base is very poor;
  • After recognising the poor evidence, most of those same papers nonetheless go on to assert that fake drugs are a major problem;
  • Most people who do academic research on fake drugs spend very little time thinking about sources of evidence other than the academic literature and the media (for example data held by industry), let alone about why they can’t get hold of those data. (“In particular, we note that global health scholarship is not, in the main, asking questions about the conditions that foreclose access to data.“)

I agree wholeheartedly with all of these points, which form the core of the paper. I think it is really important to understand how “facts” are created and circulated, and how the construction of facts (by whom? to answer what need or serve what interest? at what point in time? when what other things are happening in the world?) shapes the global health landscape. So important, indeed, that I once wrote a whole book about it (Chapters 1, 6 and 9 of The Wisdom of Whores are especially relevant to this discussion), as well as academic papers on the subject, like this one, co-authored by MedsWatch’s Maarten Kok.

 But I’m also fascinated by the way the authors, in this paper, play every single trick that they criticise others for. Some of these they acknowledge, others not. To see what they are doing, you might have to brush up on social-science-speak (SSS), a language that the authors didn’t see fit to tone down for a global health audience. A prize example of SSS:

“We trace how the material discursive work of scholarly practice constitute wider rhetorical patterns and are part of a rich discursive ecology that structures research of the contemporary global circulation of pharmaceuticals…. Based on a review of published findings we apply the methodological tools of ‘close reading’ and ‘reading against the grain’ that characterise much research on what has come to be seen as the ‘politics of knowledge’”

Tentative translation: “The way academics think about their research influences their findings about the medicine trade…. Like other academics who think about research, we read a bunch of papers critically.”

They also actually bothered to follow up references to see how robustly they supported the assertions in the paper that quoted them – a vital and underused strategy in gauging the bollox-factor in peer reviewed articles. Their paper provides an elegant illustration of why it’s important to do that.

Contention: researchers imply that, because there is antimicrobial resistance, there must be fake drugs. Supporting quote:

‘Scientific theory and common sense thus both suggest an inevitable link with [fake drugs and] antimicrobial resistance’ (Pisani, 2015, p. 12).

Actual quote, from Pisani, 2015, p 12:

The laboratory analyses described below confirm that falsified and substandard medicines — including those with sub-therapeutic levels of API and/or formulations that inhibit dissolution and restrict bioavailability — are common in countries with weak regulatory systems. Many of those countries also have high prevalence of infectious diseases. Scientific theory and common sense thus both suggest an inevitable link with antimicrobial resistance.

When others do it, Hodges and Garnett describe that trick as “refashioning data” — the selective re-interpretation of facts that can happen, almost by implication, when you take something out of its original context and mash it up with other information. They recognise that the unclear use of definitions is a problem in research around medicine quality. Then, conveniently supporting their own narrative, they take a reference to medicines that actually do contribute to antimicrobial resistance (i.e. antimicrobials that don’t deliver enough medicine to kill all pathogens, thus allowing the resistant strains to survive and multiply), and refashion it into a claim about “[fake drugs]”. In fact, most fake drugs contain no active ingredient or the wrong active ingredient, and are thus unlikely to contain sub-therapeutic levels of API and/or formulations that inhibit dissolution and restrict bioavailability in ways that directly drive antimicrobial resistance. Using information about fake medicines instead of information about substandard medicines to help explain antimicrobial resistance is like using the score of football matches instead of data on footballers salaries to help explain inflation in the sports industry. I have to agree with Hodges and Garnett: refashioned data can be badly misleading.

Another great example of using the tricksters’ tricks: the paper correctly criticises researchers’ tendency to point to limited or extreme cases (what they call “small cases”) to make a more generalizable point. They make the case against small cases with a single small case:

“rather than reflect on the problem-definition or methodology of research into fakes, the small-case was often framed as evidence in the absence of better evidence. Indeed, one major briefing report referred to a survey of medicines on sale at a large bazaar in New Delhi which found that only 7.5% were genuine but where this percentage came from was unclear (Stevens & Haja Mydin, 2013).”

Snippiness aside, I think this paper raises important issues. It’s certainly true that the small community of people who try to better understand the potential effects of poor quality medicines are making a lot of claims about the potential threats posed by fake drugs, without sufficient evidence. (The same is true for substandard medicines, which in my opinion are potentially even more of a threat in many low and middle income settings, but then we’re part of the small community making those claims.) It’s true, too, that the evidence may be limited because the problem really isn’t that big after all, so there’s nothing much to provide evidence of. But I would not be as dismissive as Hodges and Garnett are of the conclusions I and others draw from the evidence gaps. (“Instead of treating ‘evidence gaps’ as evidence of a lack of evidence, gaps themselves also became evidence of the need to generate more evidence.”)

For clarity, I’m reading this as Instead of treating ‘evidence gaps’ as evidence of a lack of a major problem…” If that reading is correct, two responses. Firstly, some academic groups, including that led by Paul Newton at Oxford, are developing lot quality assurance methods for post market surveillance, which basically work precisely on the principle of demonstrating that there is no problem (or at least not one big enough to hyperventilate about). Secondly, we are required to draw on academic evidence when writing academic papers if we want to avoid sniffy comments such as “demonstrating the presence of fakes involved the inclusion of undocumented sources, particularly with media reports and various journalistic accounts of fakes“. However, lack of academic evidence does not necessarily translate into a lack of evidence to inform our thinking, only to a lack of evidence that can be verified through close, against the grain reading of published academic literature. Some of us have access to other evidence bases, ones that we can’t publish in the academic literature, but which are certainly robust enough to signal a significant problem. These data encourage us to do more of the type of research that we will be able to publish.

Why don’t we just publish what we have and be done with it? The answer lies, perhaps, in the discipline of public health research, which is (and in our opinion should be) more wrapped up in the vicissitudes of politics and action than in epistemology. Yes, we aim to conduct rigorous research. But we also want the results of our research to be used, where warranted, to improve people’s lives. That means working with the potential end users of research, including policy makers, regulators and sometimes even (sharp intake of breath) industry, to identify which questions are most urgent and most potentially actionable, and to generate politically robust evidence to inform that action. It’s not smart to blow the trust of the very people who could use our research, just for the sake of another academic paper for the cv. So we can’t publish all the data we have, but we can use it to guide the design of research that we can publish.

As the authors of the Ghost paper point out, testing medicines is fiendishly expensive. Though they claim that “fake drugs have become a widely accepted concern in global public health research”, research funders invest remarkably little in well-designed studies that would allow us to produce the sort of knowledge that would allow us to put questions around fake and substandard medicines to bed. That’s certainly one reason that people who do know of data indicating there’s a real problem tend towards hyperbole in describing the likely impact of poor quality meds: we want research funders to share our sense of urgency, so that they’ll support the generation of evidence that is not wrapped up in non-disclosure agreements. Ugly, but true.

The MedsWatch family is planning a couple of field studies using new methods (one tries to develop measures of patient exposure to medicines, another uses market data to predict which products are at highest risk of being substandard or fake), and we’re trying to find cash for a couple more. If that rigorous research discovers that all is well in the medicine markets of the low and middle income countries where we work, we will say so loudly and clearly, and move on to researching something with more potential impact. And we shall certainly reference the Hodges and Garnett paper as a prescient harbinger of a world less worrisome that suggested by undocumented sources, (including those that pop up across the Medicine Quality Globe).

This post is a trial balloon for a MedsWatch Journal Club: We’d like to invite other academic or public health groups working on topics related to medicine quality, access, supply chains etc to take the lead on a rotating basis on picking a paper and scrutinising it every now and then. If you’re up for it, please contact us at info@medswatch.org.

Indonesia, USA: medicine pricing causes common headaches

The US and Indonesia are very different, but medicine reimbursement policies in both countries sometimes encourage pharmacies to limit stocks of cheaper drugs

As COVID-related news, research reports, papers, blog-posts and general pontification flood across my desk, I’ve been trying to fish out the bits most likely to signal existing or up-coming assaults on the quality of medicines. And one of the things I find most interesting is that the same risk-indicator fish seem to be swimming in very different bodies of water.

One example: reimbursement policies that incentivise legitimate pharmacies not to dispense the most affordable medicines, thus potentially sending patients to dodgier outlets where they are more likely to get fake medicines.

We see this in both the US and Indonesia, even though the two countries differ on virtually every health-care related measure. The US, one of the richest countries in the world, accepts that it’s fine to allow tens of millions of citizens to live without health insurance. Those that are insured have to choose between thousands of different private plans, each with different reimbursement policies. Meanwhile, lower-middle income Indonesia (which comes right after the US in population size, with close to 300 million citizens) has been working to provide at least basic health care to the whole population through one, mighty, national insurance system. The purchasing power that that confers has allowed Indonesia to do much more to bring down the price of medicine in the public sector (to the point where some medicines are arguably dangerously cheap). The fragmented US market seems to accept that, in general, medicine manufacturers should be allowed to charge whatever they can get away with; insurers and patients then decide what they can afford.

This very interesting analysis of variation in reimbursement rates in the US, from 3Axis Advisors [full report available at link], highlights the amazing range that insurance plans are willing to pay for the same medicine. For the same dose of generic azithromycin, for example, some insurers pay as little as US$0.11 while others pay $3.53 — over 30 times as much, for a product that consistently cost pharmacies between 53 and 69 cents to buy (HT @EdSilverman via Pharmalot).

A fair bit of the time, the reimbursement rate is less than the pharmacy paid for the medicine. Pharmacists may make up the loss with cross-subsidisation from other insurers (recouping the underpayment of the 11 cent insurers with cash sloshed their way by the three-dollar plans) or from other medicines. But where reimbursement is consistently below cost price, pharmacies don’t have much of an incentive to stock a medicine. And, as 3Axis observed (and we noted in our earlier blog post about the effect of COVID on medicine quality) the pandemic will likely exacerbate the problem, because producers will push up the price of medicines more quickly than payers will raise their reimbursement rates, making it even more unprofitable to dispense those meds.

The effect of this may be to push pharmacies to simply “short” the medicines which cost them money to dispense, offering patients instead the medicines they make money on. “Sorry, we’ve no generic azithromycin in stock, but I can offer you Zithromax, for cash…”

This is a dynamic we also see in Indonesia (though it will only apply to the handful of medicines which are reimbursed). The net effect of pushing patients towards out of pocket payment for more expensive medicines is to push them to buy medicines online, or in other unregulated outlets. That makes them easier prey for falsifiers.

In their blog post summarising the research, 3Axis listed some of the ways in which COVID-19 will push up the prices of generic medicines:

(Sound familiar?)

But nowhere do they address the question of reasonable or fair prices for generics, or any other kinds of medicines. The list price to pharmacies of 250mg azithromycin in the Indonesian market ranged from 29 US cents to two dollars in 2019 (compared with the 53-69 cent range actually paid by US pharmacies, according to 3Axis). The public procurement price for a pill with twice as much active ingredient (500 mg) was half as much — just 1778 rupiah, or 11 cents. Yet in the Indonesian market, Pfizer’s Zithromax 500mg sold to pharmacies at US$4.00 a pill — close to 40 times the public procurement price, and about the same as the “regular” cash price in the US, after applying the 90+ percent discount from the list price, according to the price comparison site Good Rx. But just take a look at the list price for something Good Rx describes as “an inexpensive drug” — US$ 434.65 for a product that the Indonesian government buys for 66 cents.

That’s an insane difference, and one that threatens quality from both sides. The Indonesian price is probably too cheap — it’s really hard to make and distribute a good quality antibiotic for 11 cents, even when you’re guaranteed a really big market. (New Zealand, which is notoriously good at driving a hard bargain for quality-assured medicines, pays 28 US cents for the same product.) But pricing medicines too high attracts falsifiers (who think just like pharma companies: why sell something cheap when you could sell the same thing more expensively?) And it drives patients who can’t afford the regular pharmacy list price into unregulated on-line stores, where everything is negotiable.

The COVID pandemic increases the chance that your other medicines won’t work.

For a couple of years now, the World Health Organization has been warning that fake and substandard medicines are on the rise globally. Until recently, the problem has been concentrated largely in countries where most people pay cash for medicines, and where medicine regulation is weak or non-existent. But the hyper-globalised pharmaceutical market is changing that. And the Coronavirus pandemic, an equal-opportunity screwer-up of business-as-usual, is about to bring poor quality medicines to your medicine cabinet, wherever in the world you live.

Not just COVID-related meds, you understand. Rising demand, falling supply, public panic, knee-jerk nationalism and distracted regulators will increase the risk of getting bad medicines for diabetes, heart disease, depression, rheumatism, cancer and virtually everything else. In virtually every country.

For a couple of years now, I’ve been working with researchers based in universities and data firms in Indonesia, Singapore, and the Netherlands to figure out why legitimate pharma companies make bad (substandard) medicines, and why people buy and take them. We look at fakes, too. How do the criminals that make them choose what to imitate, and who to sell to? What we’ve learned allows us to predict how Coronavirus will affect medicine quality globally.

It’s no great surprise that the answer to most of the questions above boil down to money, though political pressures have a role to play, too. The details in academic format here (or in this short video). To summarise:

Criminals thinking about making falsified medicines are looking for opportunities to make profits, while minimising their risk of getting caught and punished. Market opportunities are richest when there’s a local shortage of a legit product that people desperately want.

The likelihood of getting away with selling fakes is greatest when:

  • buyers (at any point in the supply chain) deviate from tried, trusted and effectively regulated sources, or worse, buy online.
  • the regulator in the country of sale doesn’t look too closely at medicines once they are already out in the market
  • the judicial system is not good at prosecuting or jailing offenders.

Legitimate pharma companies (from big-name multinationals to small generics producers in poorer countries) are also in the business to make money. Squeezed profit margins encourage legitimate manufacturers to cut costs, compromising quality and leading to substandard production.

These substandard meds are most likely to reach patients if:

  • the regulator in the country where they are made doesn’t do a good job of overseeing production (sometimes because politicians care more about promoting industry than they do about assuring quality, especially of medicines that will go to patients in other countries);
  • the regulator in the country where they are consumed doesn’t do a good job of checking product quality (sometimes because politicians care more about being seen to provide affordable medicines than they do about ensuring that the medicines actually work).

Most better-off countries are subject to some of these factors, but rarely all of them at once. COVID-19 is changing that. We believe the pandemic is going to put all these risk factors on steroids, for lots of products, in lots of markets. Here’s how it will happen:

Falsified medicines

Shortages will create new market opportunities because…

Demand will rise…

1. Demand for products needed for critical care, or that have actual or rumoured therapeutic effects against COVID-19 will rise rapidly in all areas (indeed, it already is!). For critical care products and (hopefully soon-to-be-)proven prophylactics/therapies, this demand will come from institutional buyers (including global health and national procurement agencies) as well as retailers and patients. For products rumoured to be effective, demand will rise more rapidly in the retail sector.

2. Both institutions (possibly including national medical stores) and individuals will buy and stockpile medicines not related to COVID-19. Medicines for life-threatening, chronic conditions, including HIV, diabetes and cardiovascular diseases are especially likely to be affected.

… while supply will fall (at least for some markets)

3. Wherever your medicines are made, the chances are they contain at least some ingredients that come from China, or a very small handful of other countries. For a few active ingredients, there’s only one producer, supplying every manufacturer of finished product worldwide. The supply of active ingredients is threatened by the pandemic (and the quality may be, too.) China’s Zhejiang province, the world’s largest producer of active ingredients, was second worst hit by COVID-19. The impact of COVID-19 related disruption was likely delayed because stocks are routinely built up before the Chinese New Year holiday, when the shut-down began. Northern Italy, another major centre of active ingredient production, has also been very badly hit.

4. China and other exporters of active ingredients will restrict exports to meet their own national needs, further restricting supply to manufacturers in other countries. These restrictions may be applied selectively for political reasons. The US is particularly vulnerable because of the ongoing trade dispute with China; the products that are in highest demand domestically in China, India and other active ingredient-producing countries will be worst hit.

5. Countries that manufacture finished medicines will also restrict exports of medicines in order to meet national demand. India, which is the world’s largest exporter of finished medicines, has already started restricting exports. That’s in part because India, which imports around two thirds of its active ingredients, is having a hard time getting the raw materials it needs. Last year, about 70% of its raw pharma imports came from China, and a further 11% from Italy. Export restrictions from India and other low-cost manufacturers will have a huge knock-on effect in the US, Europe and elsewhere.

6. Countries and large institutions will rebuild stockpiles, and profiteering distributors might hold on to medicines too, trying to drive the price up. This will cause localised shortages. In some countries, where public budgets are especially strained, medicines may be drained out of the public health systems to fatten the cushion under the private sector.

7. Manufacturers may face political pressure to switch production capacity to COVID-19-related medicines, disrupting the supply of other essential medicines. We’ve already seen car companies pressed into service to produce ventilators, but that’s fine, we can still drive last year’s car. When a maker of cancer drugs switches to making antivirals, the cancer patient dies.

8. A system-wide focus on COVID-19 coupled with disruptions to normal workflows will derail routine public procurement systems in some countries, leading to auction failures and shortages in the public sector.

9. Supply restrictions will push prices up. Countries with limited budgets for health may cut the number of medicines they buy, leaving patients to pick up the tab. Faced with buying more, higher-priced medicines, families may choose to “go cheap” — to buy medicines on the internet or from less reputable sources, contributing to risk 14, below.

10. Restricted transport, especially air traffic, will push up the cost of distribution, and reduce the timely delivery of medicines, creating localised shortages. Products such as vaccines, which need to be kept cold and transported quickly, will be especially at risk.

Meanwhile, profits for falsifiers will rise

11. The cost of making a fake pill is more or less the same, whatever you’re pretending is in it. Falsifiers usually target higher-end meds, because they make more money per pill, for roughly the same amount of risk (though they avoid hospital-only products, as well as those they know are most effectively policed by manufacturers). As shortages push prices up, the profit margin for falsifiers will rise on lower-priced medicines, encouraging more falsification of high-volume items that people are desperate to get hold of. These include medicines for high-prevalence chronic conditions, including diabetes and cardiovascular diseases, and in some markets HIV, TB and malaria medicines.

Risk of getting caught will fall because…

More buyers will step out of regulated and/or secure supply chains…

12. Shortages (1–10), disruptions to public procurement (8), and strong political pressure to be seen to be providing medicines will oblige more institutional buyers to source medicines from previously untested suppliers. This increases opportunities for falsifiers to introduce products to the supply chain.

13. When products are in short supply, it’s a seller’s market. Quality-assured distributors are going to serve their best clients first, and to them, their best clients are the ones that pay top dollar, on time. In lots of countries, national health systems are strapped for cash as well as strapped up in red tape, and are particularly bad at paying bills on time. They’re going to be dumped by their regular suppliers, and may have to look elsewhere for stocks.

14. With people stuck at home, or going to the health centre or pharmacy and facing empty shelves or sticker-shockingly high prices, sales of medicines on the internet will increase. The internet is an open door for falsifiers.

… and regulatory oversight over supply chains will be eroded

15. As governments concentrate human and financial resources on limiting the spread and impact of the pandemic, regulatory attention and laboratory capacity will be diverted. Together with restrictions on movement, this will disrupt oversight of supply chains, and reduce the already infinitesimally small proportion of products whose quality gets checked on import, or once it’s in the supply chain.

Substandard medicines

Profit margins will be increasingly squeezed because…

Production and distribution costs will rise

16. As we said earlier, the cost of raw materials will rise. They will be pushed up further for importing nations by restrictions on transport, and in some cases by falls in the local currency.

17. Increased illness, absenteeism or death in the workforce will push up costs to producers and distributors, especially in countries hardest hit by COVID-19.

18. Restricted transport, especially air traffic, pushes up the cost of distribution. (There’s also a direct effect on quality, because medicines, especially those which are more volatile, can degrade if they sit around in shipping containers or in poorly-cooled warehouses.

…while income may fall

19. Shortages of active ingredients will reduce production volumes for some producers; this may not be offset by rising prices for finished product. Producers that are locked in to longer-term contracts with large buyers will suffer most.

20. Recession, diversion of health spending to COVID-19-related products, or a general squeeze on public finances may reduce public funding available for other medicines.

21. Export restrictions may restrict sales to clients in higher-paying markets.

22. Political pressure to produce COVID-19 related products for public procurement may divert production from more lucrative products. This will especially be a problem in countries with large, state-affiliated medicine producers.

23. Sale of some relatively lucrative over-the-counter products may be reduced as a result of restrictions on movement, or simply because people don’t have spare cash to buy any medicines that they think are not a matter of life or death.

The squeeze on margins will lead to strong pressure on producers to cut costs where possible. The relatively high and rising price of active ingredients, coupled with shortages, suggest that reductions in the quality and quantity of active ingredient may be a common cost-cutting response.

Technical capacity won’t always be up to scratch

24. In many countries, the pandemic has sparked a lot of political grandstanding about national resilience and self-reliance, and this is in some places being translated into a push for rapid development or scale-up of local pharmaceutical production, even when the expertise, machinery or ingredients available locally are inadequate. That’s a good recipe for substandard medicines (we’re already seeing it in protective equipment).

25. Rapid scale-up of newly-developed COVID-19-specific vaccines or therapies may be especially vulnerable to production errors in the early stages, simply because there’s not much experience making them yet, especially on a large scale. We’ve seen this recently, in the case of substandard test kits shipped from China to Spain.

But errors are less likely to be spotted by regulators

26. The COVID-19 response is likely to disrupt routine regulatory functions. While the spotlight is on medicines and equipment needed to respond to the pandemic, regular inspections of factories and distribution chains for other, more common medicines may be winnowed down. So if any of the factors above are eating into quality, the problem is less likely to be spotted.

27. Political pressure to ramp up domestic production, as well as public pressure to deliver new medicines, may oblige regulators to fast-track products, or discourage them from being too strict on companies that are being called on to help out in a crisis.

What can we do about it?

Most of these problems are rooted in two fundamental (and intertwined) mismatches in the global economy. First, we increasingly look to governments to ensure that demand for affordable medicines is met, while relying on profit-seeking companies to supply that demand. That leads to procurement and production practices that incentivise cost-cutting and undermine product quality.

Second, we want both the price-lowering (and pollution-outsourcing) efficiencies of a globalised supply chain, while simultaneously demanding security of supply at the national level. As the current pandemic is teaching us, you can’t have it both ways.

Fixing those fundamental flaws in the global economic model is way above the pay grade of our research team. But in the meantime, we can use our analysis at least to flag up which medicines are most at risk, right now. None of the factors above is, all by itself, a clear indicator that a particular medicine is overly likely to be substandard or falsified. But when factors 1–15 are combined into an index:

Indicators of rising demand and indicators of restricted supply and indicators of good potential profits and indicators of low risk of getting caught; for a specific product (and sometimes even a specific brand) in a specific market…

… then we have a clear indication that the product might well be a fake. Regulators can use the index as an early warning system to trigger inspections, and to warn pharmacists and the public to view particular products or sources of supply with caution.

Ditto for factors 16–27; if a product ticks a lot of those boxes simultaneously, it’s at high risk of being substandard. If the national regulator is unable to respond by ramping up inspections and closing non-compliant producers, either because they are politically hamstrung or because they don’t have the capacity, a flagging system can at least help buyers to make better choices.

In Indonesia, where some of us are based, we were (before the pandemic struck) working with the national medicine regulator to try out a system that uses market data to flag products at high risk of being substandard or falsified (we call it MedsWatch). We’re still in the early stages, but it looks like it works pretty well. Well enough, indeed, to justify retooling our indicators for the 27 risk factors listed above, to develop an early warning system for the quality of medicines affected by the COVID-19 pandemic.

Watch this space. And in the meantime, please don’t stockpile meds (which will contribute to lots of problems for other people), or buy your pills from unregulated online stores (which may well end up being a problem for you yourself)

Thanks to Steven Harsono

Written by Elizabeth Pisani