ADHA: You can’t manage what you won’t measure

Who invests more than $260 million a year on a venture where there is no meaningful measurements of return on investment?

Who invests more than $260 million a year on a venture where there is no meaningful measurements of return on investment?

In March 2018  the Medical Republic attempted to score the Australian Digital Health Agency’s (ADHA) My Health Record (MHR) project on what it thought were its most important tasks, those listed below in ‘the 10 commandments’. That analysis is here if you want to check it. Much of it is still applicable today, which is not a good sign.

Some might belittle this attempt at measurement. It was quite anecdotal in some respects. But that’s the point. There weren’t and there still isn’t much in the way of  base performance criteria against which anyone can assesses the progress of the ADHA and its ROI.

Last month the Agency released its annual report 2018/19. Before that it announced that it was intending  to commit to measuring and monitoring the effectiveness of the MHR and making the information available via a publically accessible dashboard. They indicated that part of the reasoning was an attempt to show people how well things were going in order to get faster traction on what has been a somewhat controversial and divisive project. But why has such measurement never been in place anyway?

Performance benchmarking and governance are the very essence of most annual reports.

If the  CEO of the ADHA Tim Kelsey announces in a press release that  “The 2018–19 Annual Report shows the Agency is making significant progress toward the key priorities set out in the National Digital Health Strategy”, then should we not expect ‘significant progress’ to be defined by starting points, end points and where we think we are on that spectrum?

But the report has  no meaningful benchmarking of performance. Outside of senior management and board salaries , a balance sheet and a $22.7mm loss (what ever that means) the only numbers on performance we see are numbers inserted into various commentaries, without any  reference as to what they might mean for progress or not. They often appear as a net big number eg, “this many thousand pharmacies have…” or “one billion uploads of ….” , but invariably the numbers are quoted without  any reference point that gives the number meaning.

The CEO’s review of each strategic pillar of the ADHA, early in the annual report,  is  telling in terms of his deference (lack thereof) to numbers and benchmarking progress. Each pillar has a commentary but no target number, or a past target number that they had estimated or hit. It’s vague and random almost. Progress is impossible  to reliably decipher.

Some numbers are referenced of course. 90% of Australians are now opted in to the MHR. But quoting this number which does seem definitive and progressive, says a lot about how the numbers are spun at us in this annual report. How many people are really using the MHR today? How many should be  at this point of time do we think? What was our estimate a year or so back of where we should be now?  How many GPs are using it – really? How many people just don’t care? How many people who are managing their record are managing it well enough for doctors to use confidently? How many lives has it saved – really.? And so on.

“A major focus has been to improve access to clinical information in My Health Record. By June 2019, more than 1 billion documents had been uploaded, ” says Kelsey in one part of his review. How very Dr Evil of him. What does one billion mean to a punter like me?

Yes, one billion is a lot, but not if a gazillion is a good performance.

Why after more than three years of spending about $260m per year,are there no meaningful goals of performance being followed in  numbers? Surely such an approach would be seen as more transparent and trustworthy. As things stand this could go on for years. Who invests $260m a year and doesn’t meaningfully measure return?

It’s hard to call someone out on performance if they don’t provide a means by which you can measure them.

In the absence of any benchmark numbers for measurement and in the tradition started by The Medical Republic in trying to assess ADHA ROI on taxpayers money, Wild Health wants to stimulate some discussion by thinking a little more about the key numbers and events that surround each of the ADHA’s strategy pillars, some which we could use to measure our investment in them better.

Pillar 1: The My Health Record

Before we start there’s a somewhat largish elephant sitting in the corner who thinks that putting numbers around this Pillar and measuring it, might be a waste of time. That is, the MHR might never really do what it was meant to do, and no amount of opting people in without their knowledge, getting chemists excited that they might supplant GPs a bit more by adopting it, press releases on miracle MHR saves (there have been a few in fact,  but that isn’t  putting the project in the context of good government ROI, is it?), and various other measures, will mean much in the long term.

A centralised medical record, which is supposedly controlled by patients hasn’t worked anywhere else in the world. It’s a concept fraught with issues which the ADHA is running into now and which within a year or two might reveal the MHR project as farce.  Eg, patients who are opted in don’t care that they’re opted in or the ones that do care, only care every now and then…the resulting record is never reliable as far as the medical community is concerned. Even bigger, doctors aren’t using it much, except to get paid by a government incentive scheme.  All these statements should annoy the ADHA, but if they don’t put out any figures that counter the strong sense that these scenarios are unravelling, what are we to think over time?

Outside of this one small consideration let’s check some numbers and actual possible progress. To do this we have to have some Pillar  outcome goals which the Agency hasn’t established so we have for them here:

• Halve the number of medication mismanagement errors
• Halve the cost of private pathology to the system
• Bring efficiencies to a networked system of doctor data to the tune of say, $1billion within five years

Note: in Senate estimates a ‘savings’ number is quoted, which we have quoted in the past. But we can’t find it anymore. We think it was about $300m over five years.

With those outcomes, what numbers need to move meaningfully to indicate reasonable progress? Here’s a few suggestions:

• Regular GP users
• Engaged patient users who are meaningful to the system (chronic disease patients we guess)
• Hospital ER and discharge use, with a confirmed end point at the GP in the case of discharge summaries
• The number of private pathology tests uploaded, referenced to the entire volume that are uploaded by private pathology
• Medication error numbers (tracked reliably and benchmarked historically)

None of these measures are meaningfully reported at this stage by the ADHA. They do attempt a figure of engaged GPs, Pharmacists and patients, but they are not trustworthy measures of meaningful engagement yet unfortunately. And they are low numbers still. By the way, if the ADHA thinks they do have meaningful numbers of engaged doctors and patients, please report those numbers, report how much they are growing each month, and report very clearly, why they think these clients of the MHR are actually engaged (ie, tell us precisely how you deduce that they like and use the MHR).

On the subject of GPs, that elephant in the corner is smirking because the ADHA put the whole GP community offside during the ‘opt out’ process. With their arms around RACGP and AMA apparatchiks, agency personnel assured us doctors were onside. But they weren’t and they aren’t. Most GPs remain hostile to the MHR because of patient consent and work in progress time issues. They don’t see a meaningful return for them or their patients. It’s a giant fail that people are ignoring, and the agency seems to try to patch the issue up with figures around pharmacy uptake of the MHR.

It would be very informative if someone tried to explain to us why pharmacists are so bought into the MHR and GPs are not. Yes, a lot of GP surgeries are registered but that is incentivised by ePIP – they get paid to sign up and send patient summaries.  The situation has the whiff of GPs not trusting the system, not having time to do the paperwork and being miffed at being overridden on ‘patient consent’, whereas pharmacists, whose business model is rapidly being dismantled by the internet and ‘chains’, look like they are seeking to grab on to anything that can make them an extra buck in increasingly difficult trading conditions. Or, is that that pharmacists are just far more community minded than GPs?

The private pathology goal here has been very slow as private pathology doesn’t really have a stake in making the system flow better in the short term. Public hospital pathology numbers are emphasized mostly by the ADHA, but they aren’t meaningful really in saving money, the volume of private is. Private pathology numbers aren’t  meaningfully reported and where it is it is still very low.

Scores out of 10:

• 6 for effort and execution
• 2 for effectiveness
• 2 for potential usefulness to doctors and patients into the future

Pillar 2: Secure Messaging

The ADHA recognized the potential value of getting this Pillar moving from day one after listening to stakeholders and has put a lot of effort into attempting to sort out what is a complicated history of commercial entanglements and missed opportunity in Australia. They have put that effort in systematically and intelligently (that’s not a number I know, but we will get there), but if we’re at the footy and someone shouts “look at the scoreboard”, it’s still not very good after all these years.

Once again, the ADHA tends to manage perception in this Pillar by PR, anecdotes and pilots. Two pilots in particular are quoted around interoperability and the web sharing standard FHIR, and both, as pilots, were important and interesting in demonstrating that “it’s not the technology stupid”. But in the case of technology like FHIR, which was piloted, the experts have always maintained that the technology was never the issue. The issue is commercial  and political between big global and small local vendors who simply can’t afford to step out of the current paradigm of how they manage data and develop systems.

What would meaningful numbers look like in a Pillar that is not moving because of micro economic conditions of the healthcare software vendor market?

The key number would be something like number of vendors who are actually complying to a set of development and implementation standards and perhaps, a common governance framework.

After stepping in boots ready to kick at the beginning of this process,  the ADHA has actually taken an intelligent stand on this Pillar by stepping back, acknowledging the complexity of what is going on commercially, especially for small local software vendors, and taking a role of facilitation of the process in order to expedite what is currently snail pace change.

The only number you might get the ADHA to quote here is that they think they have got most of the commercial sector on side. I think what they mean is that they’ve got most of the commercial sector prepared to play the game of looking to be onside in order to keep a low profile with a government which has in the past proven bullish and destructive in how they attempt to affect change .

You might take an approach here of hard ass government regulation. Ie, here’s your grandfather clauses, and here’s your date to get up to these standards and if you don’t make the cut off,  you aren’t legal anymore. Such approaches rarely work and given the ADHA’s track record and trust bank account , such an approach would almost surely fail.

Scores out of 10:

• 8 for execution and effort
• 5 for effectiveness (applying a degree of difficulty of over 9)
• 5 for potential future usefulness for patients and doctors (it might be that interoperability is not about secure messaging as we know it going forward)

Pillar 3: Interoperability

Pillar 2 is first base for Pillar 3, therefore you know you’re in for the long hall on this Pillar no matter what. Others, notably FHIR founder Grahame Grieve, would probably argue that secure messaging in the way we currently are trying to solve it,  isn’t that relevant to how interoperability should be unfolding in an open systems world.

Either way you look at it, this is one Big Hairy Audacious (Ass)  Goal – commonly termed BHAG. Sometimes when faced with a BHAG and politics all around you, it’s a good idea to stall by doing a  bit of customer consultation. But the ADHA can’t be accused of this, if you accept that secure messaging is the first big issue of interoperability to solve. They tried to parallel run the hardest nut of Interoperability from the start. They just haven’t gotten very far.

For numbers the Agency says that it has so far held 15 co-design workshops and 33 community meetings in their consultation phase so far on interoperability.

I wonder whether such activity might end up lifting expectations and confusing people too much and whether there might be a bit more parallel work on the topic, as we’ve seen already with secure messaging. In other words, do more things now.

Such work might include more work on FHIR as a standard here, and the local iteration of its integration cousin Argonaut. In the US and parts of Europe both these technologies are already forming the basis of the future infrastructure of ‘interoperability’, while in Australia, we seem to be waiting to see what happens overseas first.

Scores out of 10:

• 2 for execution and effort (and that’s from Pillar 3)
• 1 for effectiveness (not good)
• 10 for potential future usefulness (interoperability is what the ADHA should do wholly)

Pillar 4: Medicines Safety

There is an infamous business rule called the “Rule of 3”, which means simply, never try to do any more than three things at once if you really want to get stuff done properly. Immediately in Pillar 4 you get a sense of ‘now you’re just adding stuff so people think you’ve got enough to do’. Medicines safety will be achieved primarily though Pillars 1 to 3, so why make it one itself after the fact?

In Kelsey’s review summary of this Pillar his main point of achievement and progress is Medicines View, which is an MHR initiative. It is the internal bit of smarts in the MHR which grabs medications records from a couple of spots, simplifies them, and puts them near the top of the MHR in an easy to comprehend format.

It was a very neat addition to MHR. It perhaps for the first time demonstrated that useful information can populate most records in the system. But it is part of Pillar 1 still.

There  is some great work on medicines safety that the Agency is involved in, outside of the first three Pillars. You could say it’s important tangential work. This is in the area of helping to facilitate a digital ecosystem for eScripts. Legislation has passed which will lay a foundation for scripts to be legal as an electronic only document on our mobiles and this is likely to lead to lots of good changes to the system, including clamping down on doctor shopping and avoiding errors that occur on paper. The ADHA has a taken a role here in publishing the solution architecture and conformance requirements for prescribing and dispensing software, partnering with the MSIA and some provider groups. Such work might even encourage the start of some new software companies, or provide opportunity for existing ones.

Scores out of 10:

• 5 for execution and effort (but remember, most is from preceding Pillars)
• 4 for effectiveness (a lesson here: do less and you might do better)
• 9 for usefulness to patients and doctors into the future

Pillar 5: Enhanced models of care

A quick and possibly too cursory assessment of this Pillar is we are too far beyond the “Rule of 3”. This is a Pillar we see in just about every healthcare organization these days from the RACGP to the AIHW, the MSIA and the AHHA. It’s too big to understand what the ADHA is really doing about it and looking at the word count on Kelsey’s review you suspect this Pillar is falling by the wayside. Maybe it should, as others have it well in hand, and if you pull off Pillars 1 to 3 you are surely going to facilitate enhanced models of care.

Scores out of 10

Not marked. No meaningful data

Pillar 6: Workforce and Education

This is a very important goal, but it maybe does not sit all that well in the hands of the Agency. As with Pillar 5, they already have a  massive job in Pillars 1 to 3  so giving them more is likely to dilute their chance of success in the big ones.

The politics of digital health education inside doctor based organisations is particularly bad so arguably there is some  role . But organisations like HISA and ACHI are doing a great job already. They just need more funding to go faster.

There are also some problems of potential conflict with how the ADHA is educating the workforce if you are in the camp of thinking that the MHR might be a dud. The ADHA spent a fortune last year on PHNs in order to educate GPs on the MHR, but some would say a lot of what they were pitching was near propaganda for what the ADHA would like the MHR to be, not what it is. That doesn’t help anyone in the end.

The agency is promising a ‘workforce roadmap’ in this coming year. That sounds interesting and promising. It is aimed at bringing together the workforce activities on education and training across jurisdications. The digital health education community is already very community minded and it’s already well facilitated by HISA and ACHI and some others. Maybe the ADHA should just give HISA and ACHI more money to do most of this job and come to their help where there are serious jurisdiction disputes.

Scores out of 10

• 3 execution and effort (the ADHA did a bad job of education around the MHR)
• 3 for effectiveness (see above)
• 8 for importance to the future of patients and doctors (they can’t do much with patients but they can help in educating doctors)

Pillar 7: Driving Innovation

This should be in the first three Pillars of the Agency, and that they’ve left it until 7, and written one small par with some pithy references to tiny pockets of funding and developer assistance suggests that the Agency is missing a big opportunity to help here.

Rule of 3 says the Agency should just have three priorities. We say only two:

1. Interoperability
2. Innovation

Healthcare is notoriously held hostage by large global EMR vendors, who are now being threatened themselves by the badly behaving global digital platform vendors (the four healthcare data platform horsemen: Google, Amazon, Apple and Microsoft). The large vendors do what big business do. They protect themselves and their business models until someone smarter and eventually more powerful disrupts them. This is all in play now. Cerner, Allscript, Epic, et al, are all under a lot of pressure to migrate to open systems overseas by governments and by the sort of network power they are likely to encounter from the four horsemen.

In this world of change Australia’s software vendors need  help. Such help will benefit the economy and patients because Healthcare is still a sector where local knowledge counts.

The vendors need either funding,  or tax breaks, and they need a  formalized and secure standards and governance regime. They need love.

MedicalDirector, one of our largest PMS vendors wants to break out and go global by starting in the UK. This could be ground breaking for the company and Australian doctors as the UK is the in future in many respects for some of how our local software should work and report. The ADHA should think much more carefully about the priority they give too and the  effort they put into this Pillar.

Scores out of 10:

• 2 for execution and effort
• 1 for effectiveness
• 10 for Importance to the future of patients and doctors

Accountability and ROI

There is no question that  ADHA has a very difficult  task identifying and executing policy and projects which will facilitate the Australian healthcare system to be optimised through the sensible application of digital technology. Projects, such as the MHR and secure messaging, are complex and at times inescapably messy, so being able to identify starting points and end points, and where we should be in between, was never going to be easy.

But as things stand today, we spend upwards of $260m and more each year and there is no meaningful attempt to measure progress or value for money for our investment. No one would do that in the private sector. Why do we think it’s OK to do it with taxpayers money?

By the way, governance starts at the top. It isn’t the  ADHA’s fault necessarily that it isn’t reporting progress and ROI properly. Agencies like the ADHA have strict constitutions and governance set by their masters. In this case the masters are COAG and the Department of Health.