Advancing education outcomes for First Nations girls

00:00:00

Welcome to Social Ventures Australia's workshop on outcomes-based contracting. This workshop is specifically designed for organisations who are thinking about preparing a tender for an outcomes-based contract. The examples in this workshop focus particularly on achieving better education outcomes, but we think this workshop material is relevant to anyone who's thinking about preparing an outcomes-based contract.

00:00:42

Module one - an introduction to outcomes-based contracting. This workshop is going to be presented in six different modules and the next 4 modules will build on our framework for developing an outcomes-based contract defining the beneficiary group, designing the solution, measuring outcomes and building the funding arrangement.

00:01:05

At the end of this workshop, you should have an understanding of the principles underpinning outcomes-based contracts and understanding of the process and stages involved in designing and delivering an outcomes-based contract and an understanding of the key risks for service providers.

00:01:21

You should take the opportunity while you're doing this workshop material to try applying the outcomes contracting theory to your organisations own priorities and programs, and we hope by the end of this workshop you'll have a better view of whether an outcomes-based contract is right for your organisation.

00:01:44

My name is Kirsten Armstrong. I'm a director of impact investing at Social Ventures Australia (SVA). SVA is a not-for-profit organisation that works with partners to alleviate disadvantage. Working towards in Australia where all people and communities thrive, we deliver our work through a range of different approaches. We have a number of ventures which are small pilot programs.

00:02:05

Testing the best evidence around a number of challenging different areas, particularly in education and employment.

00:02:12

We have a consulting practise that delivers and works closely with not for profits across Australia to help them deliver better impact impacts through their organisations. We have a policy and advocacy team that works closely with government and other stakeholders to try to get the policy settings right across Australia to reduce disadvantage and helped achieve greater social impact.

00:02:42

And we have an impact investing team which I'm part of which includes social impact bonds and outcome-based contracting practise as well as a number of investment funds where we invest on behalf of a number of investors across Australia to help achieve social impacts.

00:02:53

Let's look at the landscape of outcome-based contracting in Australia over the past decade, there's been over 30 outcomes-based contracts delivering social impact in a range of different areas in Australia, including seventeen social impact bonds, which will explain a little bit more throughout the module today.

00:03:14

NSW has been a particular leader in delivering outcome-based contracts, with some 11 contracts and more recently the Commonwealth government has engaged more heavily. They've been particularly involved in the employment space, so you'll see a large number of contracts in the employment space and a lot of those have been areas that the Commonwealth government has been involved.

00:03:36

State governments across Australia have been more involved in other areas where it's linked to the services that they deliver within the state. So, for example, out of home care homelessness, education. Recidivism (So reducing return to prison rates) and health, mental health and Aboriginal economic delivery development.

00:04:02

Despite the fact that outcome-based contracts have been going for some 10 years in Australia, you'll see that actually many of them are still in the delivery phase and that reflects the fact that outcome-based contracts tend to be longer term contracts perhaps seven years, and only a handful have actually matured or terminated to date.

00:04:23

So what are the benefits of an outcomes-based approach? We often hear that government likes outcomes-based approaches because it gives them more accountability for the money that they're spending and hopefully helps them to achieve better value for money.

00:04:37

What about service providers? What are the benefits for them? So we see outcomes-based contracting as a way to support innovation because it provides more flexible funding, so less focus on how money is actually spent and more flexibility to better meet the needs of individuals receiving support so that you can help to achieve those outcomes

00:04:59

It provides more accountability for outcomes, and that means there's an opportunity to better build the evidence base about what works.

00:05:08

So we see the opportunity for innovation, better evidence and also collaboration with new and different partners who are really interested in seeking the same sort of outcomes as you are.

00:05:15

We have gone this far without actually talking about what outcomes-based contracting actually is. It's an approach that governments use to Commission services for a clearly defined cohort of people with clearly defined outcomes.

00:05:34

So that means part of the government funding is dependent on what outcomes are achieved and it can come in many different forms. There's a whole range of different terms you'll hear in Australia and overseas, including terms like social impacts investment, payment by results, payment by outcomes, pay for success. In Victoria specifically, you'll hear about partnerships addressing disadvantage, and you might also hear about social impact bonds, which are a special type of outcomes-based contract involving investors.

00:06:09

So how do outcomes-based contracts compare to grants?

00:06:13

On the left hand side of this diagram, you'll see what a traditional grant agreement looks like in terms of payments to service providers.

00:06:21

Typically there's a government payment, a fixed government payment made in advance, which service providers then use to deliver their program. It's typically short term funding, perhaps 18 months, sometimes longer, and service providers need to prepare acquittal. Acquittals for any sort of underspend.

00:06:42

In contrast, an outcomes-based contract will typically involve a smaller payment, a small advance payment made by government to help kick start the program, and sometimes that small advance payment is also complemented by a payment by investors.

00:07:00

Later, after the program is delivered and outcomes are measured, there'll be an outcome or a series of outcome payments, although it's important to know that that outcome payment is dependent on what outcomes are achieved and could actually be.

00:07:15

So what that means is there's some sort of sharing of risk and returns between the stakeholders and the contract. Oftentimes, though, the useful thing is that often means a longer term funding cycle, and we find that contracts for outcomes-based contracting tend to be 4, 5, 6 even 7 years.

00:07:35

Into a little more detail about outcomes-based contract, this diagram shows the typical contracting approach, so step #1, a service provider enters into an outcomes-based contract with government and that contract defines how payments will be made on the basis of measured outcomes.

00:07:54

The service provider, Step 2, then delivers the program to the service users and hopefully that leads to better outcomes.

00:08:02

Step 3, those improved outcomes generate savings for government. They don't always, but this is often why governments have been interested in engaging in outcomes-based contracts, because investment early helps to lead to other savings for government down the track.

00:08:20

Lastly, Step 4, Governments pay service providers if outcomes are achieved according to the outcomes-based contract.

00:08:31

We mentioned social impact bonds as a special type of outcomes-based contract. They're very similar to the outcomes-based contract we talked about in the last slide, but with the added bonus of investors. So investors provide upright capital so that the investor rather than the service provider bears the risk of poorer outcomes, but also gets the financial benefit if outcomes are better than expected.

00:09:01

So a lot of the process looks similar to that first slide, but let's step it through. So Step 1, the service provider enters into an outcomes-based contract with government to pay on the basis of measured outcomes. And Step 2 is where the investors get involved. They provide the upfront capital to help Co fund the provision of services.

00:09:21

And then things continue from there. The service provider delivers the program. Hopefully the program generates savings for government and the government pays based on the outcomes. But see Step 6, the outcome payments provide investors with a performance linked return.

00:09:39

What all that means is that different types of outcomes-based contracts have different levels of risk for service providers.

00:09:47

So a grant agreement on the left hand side that traditional government funding has relatively clear risk for providers, relatively low service risk for providers, because fixed funding is provided upfront, there's not that great accountability in terms of outcomes, but a lot of accountability about program costs, the way money is actually spent.

00:10:10

Unfortunately though it may not generate evidence of what works.

00:10:15

At the other end of the spectrum, an outcomes-based contract involves greater risk, greater financial risk, typically for service providers, because of that link between outcomes payments and the outcomes achieved.

00:10:30

There's, there's typically lower accountability for program costs because the focus is really much more on outcomes and they're more likely to generate good evidence of what works.

00:10:41

In terms of risks to service providers, social impact bonds sit somewhere in between. There's often some risk borne by their service provider in terms of the financial outcomes being linked. Sorry, the financial payments to service providers being linked to outcomes and they can be a little more complicated because of the additional contracting involved by bringing investors into the mix.

00:11:07

So the challenges and benefits of an outcomes-based contract. It's important to understand that they can be challenging and so they're not for everybody. The development phase can be long, perhaps 1 to 3 years and it requires a lot of focus and effort to really get the design and contracting right, start thinking about operational processes, data finance and executive senior leadership support.

00:11:32

The other thing that it's important to be aware of is that it can put your organisation in the spotlight, so results are often quite transparent, be they good or bad, so it's important to be ready to be a little bit more on the spotlight.

00:11:47

Having said that, the hard work is often rewarded so a longer contract term is really quite typical four to seven years giving that financial and operational certainty to a service provider. Typically, we see larger scale programs compared particularly to philanthropic funding.

00:12:08

And more flexible funding, so it's possible for a service provider to adapt and change how they deliver the program if it means they can deliver better outcomes.

00:12:19

And there's really often broad learnings and sometimes those are quite unexpected learnings about what really needs to happen to build to, to deliver better impact. And that means building a much stronger evidence based on what works.

00:12:36

So this slide shows an illustrative development process or a timeline for what typically happens in first developing or designing and outcomes-based contract. So initially there'll be an application and assessment process which might typically be 3 months. It just depends on the request for tender process that the government's running.

00:12:59

And then there's what's called a joint development phase. And that's where all the really nuts and bolts are really sorted out. So terms are negotiated, program guidelines, developed, contracts signed and invested capital, if it needs to be raised, is also raised. But that can be quite an intensive phase. So depending on how much work has been done of the various parties in advance, somewhere between six months and two years, and it's really important to be aware of that timeline.

00:13:30

One to three months to set up and then again that longer service delivery period that we typically see.

00:13:40

So the inputs and outputs of that whole development process, the inputs what the service provider really needs to put in the effort that they need to put in, includes accountable and well qualified people who are able to be creative in coming up with solutions.

00:14:00

A good financial model and a clear program design time, good data, goodwill and flexibility from everyone involved and a real project mindset to make sure you can actually deliver the various pieces of the puzzle needed to make an outcomes-based contract work.

00:14:16

Work and the outputs of all of that include clear definitions of all the aspects of the program, clear measures and data sources to help measure your outcomes, a financial model, process maps and, of course, legal structure and contracts.

00:14:33

So throughout the rest of the workshop modules, we're going to use 2 examples that we think will be of interest and have good lessons for those considering developing an outcomes-based.

00:14:44

Fact the first of those is the  Side by Side contract, which is a social impact bond that is currently being delivered by in Victoria by Berry St and Vacca, and it is a contract that was commissioned through the Victorian Department of Education and Training.

00:15:04

It’s a 6-year program so 2021 through to 2027 and it's a social impact bond that has some risk sharing with investors. So $5 million of capital was raised from investors to support the social impact bond.

00:15:20

The program supports children in years one to four who have high absenteeism from school or are behind in their educational attainment. So the goal is to support 300 students over five years of referrals.

00:15:36

Aboriginal and Torres Strait Islander students have been prioritised.

00:15:40

Those students are referred into the program by the Department of Education and Training, with schools selected for inclusion based on some clear eligibility criteria.

00:15:52

The program includes a mix of different elements that focus directly on the student, also on their family and also on the school and the teachers working in those schools. So it's got these three different elements that all work together and provide 12 months of individualised support and then in the 2nd 12 months, some additional but scaled back support over that.

00:16:18

Outcomes measurement focuses on the number of days which the student is absent from school, and the outcome metrics which we'll talk about in later modules have all been designed focusing largely on that absentee level.

00:16:37

The second case study we have is the Play2Learn+ payment by outcome contract, which is being delivered at the moment by 54 Reasons previously called Save the Children. It is a Tasmania based program that was commissioned by the Commonwealth Department of social services and also received additional funding from the Paul Ramsey Foundation. It's currently in delivery, expecting to deliver through to the end of 2026.

00:17:05

How the program supports 3 to 4 year old children for families in Tasmania that hold a concession card and who are not currently engaged or have limited engagement in learning.

00:17:16

The goal of the program is to connect children and their families with early learning in the year before kindergarten, providing children and the carers with targeted playgroups and one to one support both before they enter kindergarten and in the first 3 months of kindergarten to ensure a smooth school transition.

00:17:36

Children are referred into the program by 54 Reasons own contacts as well as through a range of different community based services and healthcare services.

00:17:51

Change of module 1 so please tune in for module 2 where we'll be talking about the first element of our framework defining the beneficiary group.

00:00:00 

Welcome to module two on our workshop on outcomes-based contracting. So this module focuses on the 1st element of our framework for developing an outcomes contract, so defining the beneficiary group. 

00:00:15 

So who are the target beneficiary group? What we mean by a target beneficiary group is everyone who might potentially be eligible for the program. So it's not actually the people who finally join your program, just thinking about everyone out there who might be eligible for the program as you define it. 

00:00:35 

Now, it's common to be quite specific in an outcomes-based contract about who might be that Target group. 

00:00:40 

Common criteria include gender, age, geographic location, the occurrence of an event in that person's life, for example, they were previously in out of home care, or an amount of time since a prior events, for example, whether a person has received a welfare payment for a certain period of time to date. 

00:01:03 

Sometimes there's also an assessment or a classification made, for example, whether or not the person is able to commit to the program. 

00:01:11 

What's important about the target beneficiary group is that ideally, it would be objective, so it would be clear to everyone that that person, yes, is definitely in the target beneficiary. 

00:01:24 

It's inclusive, but also, oftentimes we think it's better for the target beneficiary group to be large. Of course, that's dependent on the size of the impact of your program. The larger the impact of your program, the less important it is that you've got a larger group you can sort of still see the impact for a smaller group. We'll talk about that a little bit more in this module. 

00:01:48 

So what are important considerations for eligibility criteria? 

00:01:57 

If you have flexible or wide or generic terms for your eligibility criteria, you'll end up with a larger eligible pool. It'll be simpler to determine if someone is eligible, but what that also means is that value for money for government is a little less clear, cause perhaps not everybody in that group is really going to get value out of the program you're planning on delivering. 

00:02:22 

And there's a higher risk of what we call cherry picking. So the people that end up in your program are people who might have done well anyway in the even in the absence of your program. 

00:02:34 

On the flip side, you can actually go for very objective and narrow criteria and the result of that is you might end up with a smaller eligible pool. 

00:02:44 

It can be more complex to determine whether or not someone is eligible, but they for money for government may be clearer because you might be really targeting really high needs people or people who are really gonna achieve a lot from engaging with your program. There's also a lower risk of that cherry picking that I talked about previously. 

00:03:04 

Practising, we recommend just balancing theory and reality so to be eligible for support.Keep it simple, a participant must meet the eligibility criteria. 

00:03:18 

So the enrolment process is probably something we're all quite familiar with, but for outcomes-based contracting, there's some special tweaks that often happen, and usually there's a need to be a little bit more stringent in the enrolment process and ensuring that all the people who are in there are meant to be part of that program.  

00:03:39 

So thinking about identifying and referring people into the program, we often see quite a few different approaches taken here. So sometimes the service provider can be responsible for identifying and bringing people into the program. 

00:03:52 

And other times we see that government might actually take the lead and be responsible for referrals and the referral process links a lot to assessing eligibility because, as we talked about in the last slide, there's important considerations for who's eligible to join the program and if the government is responsible for determining eligibility and referring people into the program, then sometimes it takes away some of that stress about who can and can’t be in the program based on whatever criteria have been set. 

00:04:27 

So it's just something to think about in, in the process is who is actually going to be responsible for finding people and then who is actually really going to be responsible and will everyone be confident that eligibility has been assessed appropriately? 

00:04:43 

It's really important to obtain participant consent not just to being part of the program, but also potentially to having their outcomes measured and payment made on the basis of outcomes. So there's a few extra tweaks that happen often in gaining participant consent. 

00:05:00 

And we're finally happy that that person is eligible and has consented to be part of the program. They need to be formally enrolled and it's important for an outcome-based contract that the date, the actual date of enrolment, is clearly recorded, because oftentimes that's when measurement starts for that participant. 

00:05:19 

How will eligible participants be identified and connected to your program. 

00:05:24 

There are three different ways that we typically see that this referral process happens. So the first is a more systematised approach. So that's where government data is used to develop a list of eligible individuals. There's typically no human judgement at all. Whoever is on the list is eligible and referred. 

00:05:44 

And often every single referral is directly enrolled in the program. An example of that that we've seen is the RESOLVE program, which is a NSW Health department related program where the NSW Health Department develops a list of people who meet specific criteria, so they've been in hospital for a particular period of time for a mental health condition and every single person is referred into that program.  

00:06:12 

So having an approach like that I think helps to manage some risk because there's some confidence that you'll get the right number of people joining the program. 

00:06:24 

Sometimes we see government initiated referrals, so government is responsible for using those eligibility criteria for identifying and referring those people into the program. So sometimes it's data based or sometimes it's based on and on the ground assessment. 

00:06:42 

And typically when we see government as having a key role in deciding who is referred into the program. Government in the outcomes-based contract process, so in the payment process, will actually take on the risk that the number of people who are referred into the program are different to what was planned. 

00:07:02 

The last option is thinking about using your own referral network, so it might be a network of community providers or referral partners that you work with, but it's important if you're going to do that to think about whether you have already got those referral connections on the ground or whether you're going to have to take up time to build a network in that situation. 

00:07:22 

We often see that if someone is looking to expand their program, so deliver a program in a new area, it's actually quite challenging to build the referral network and it's important to account for and plan for the time that it'll take to develop that up. 

00:07:38 

We talked about the fact that it is important to have a reasonable size in terms of your beneficiary group and the reason is that the overall size of your beneficiary group, the potential beneficiaries, determines really how many people could potentially get referred into the program and become your measurement group and your measurement group is really important for determining how your outcomes and payments will be determined. 

00:08:07 

This slide shows an example of a group of people, the Blue Group, who received no program at all, and let's imagine another group, the Green Group, who receive your program. 

00:08:21 

And you can see that the outcomes achieved by those two different groups in this hypothetical scenario are quite different. So there's a big gap between the average outcomes shown by the dotted lines achieved by the NO program group and the average outcomes achieved by your program. 

00:08:43 

But as you can see, participants could achieve a wide range of outcomes, and there's sort of overlap in the two groups. So some of the peoples in your program might actually achieve a lower outcomes than some of the people who achieve no program at all.  

00:09:02 

We will only know the outcomes of those people that are actually measured and it's important to understand the size of the measurement group will affect how reliably your program’s performance can be measured. 

00:09:15 

So in this example we've got a small measurement group and as you can see, there's only a small number of people from the Blue group who've been measured for their outcomes, and a small number of people from the Green Group who have been measured for their outcomes. 

00:09:28 

And in this example you know because there's quite a bit of overlap between the potential outcomes of the two groups, it's actually really difficult to observe where there's been any difference at all. So it's really, if we look at the observed average down the bottom, those two arrows, there's not a lot of difference between the blue and green arrows. So in this example, with a small measurement group your impact may simply not be clear. 

00:09:55 

However, with a bigger measurement group, it's possible to get a clearer sense of what the true impact of the program is. So you'll see in this scenario that the observed average for both the NO program group and your program group, those two areas down the bottom, are actually very similar to the actual average of the programs. 

00:10:18 

And there's a really clear gap between the NO program and your program group. So with the bigger measurement group, the difference, the true difference between the two programs, the impact of the program, is more likely to be able to be measured and be clearer. 

00:10:35 

In this last slide, we show the impact of a program that actually has a bigger effect. So what you'll see is that Green Group, the YOUR program group, is much further to the right up the scale and there's not a lot of overlap in the potential outcomes between the Blue Group and the Green Group at all. 

00:10:53 

So what that means is, even if we choose to measure a small number of people, chances are we're probably going to see an impact from that program. You can see that in the bottom row, the observed average outcomes, there's actually a clear difference between the blue group and the green group. 

00:11:11 

What's also interesting in this program, though, is you still have that problem is that it may not really be able to see the true effect of the program. So the observed average in that bottom row is actually still a lot less than the true impact of the program. So. 

00:11:28 

So it's important to think about whether you'll be able to see an impact at all with a smaller group and also important to think about the size of the impact and how confident you are you'll see that impact with the size of the group that you're able to support. 

00:11:46 

We would typically expect to have, you know perhaps 200 or 300 people in a cohort. They don't all have to be enrolled in one year. And so if you've got a group that is somewhat smaller that you might be enrolling in a particular year, you can actually start to see impact by enrolling people over a period of years. So perhaps 3, 4, 5 years of enrolment will start to give you the scale of measurement group that gives you more confidence around your impact. 

00:12:14 

So this slide shows the two examples we've been showing, the  Side by Side social impact bond and the Play2Learn + payment by outcome contract in Tasmania. So if you look at the  Side by Side social impact bond, you'll see that an eligible child is defined as someone who is enrolled at an agreed school. 

00:12:36 

Those terms are capitalised, so agreed school, anything that is capitalised. Here is something that is a defined term and gets defined very specifically in the contracting for these sorts of program projects. I thought it was worth highlighting that because there's a lot of detail that goes into defining a whole bunch of elements in an outcomes-based contract. 

00:12:59 

So the focus is this program is looking at people who've been absent from school for 21 or more days and it was done over the first three terms of the referral year. So that's a year prior to them entering the program. The reason it was done over the first three terms is that assessment is made in the fourth term so that they can be enrolled effectively, in the next year. 

00:13:22 

And again, the term absent is capitalised, which means absent had to be defined specifically as to whether it was, you know, absent for the entire day part of the day and so on or absent for a particular cause. 

00:13:34 

So again, just noting the specificity that happens in these sorts of contracts. For the  Side by Side social impact bond, the Department of Education has actually initiated the referrals. So that's not something that Barry St and VACCA have to drive. It comes from the department itself. 

00:13:55 

In the case of the Play2Learn + payment by outcome contract in Tasmania. Again you can see a number of specified terms, so the student resides in a catchment area for a specified school. 

00:14:09 

The parent needs to hold a concession card. The child will be turning 3 years old and has not been engaging in more than 10 hours of childcare per week. So initially the thinking was to have children who had not engaged in any sort of childcare at all,  

00:14:29 

but actually, that requirement was flexed a little so that children who were just in and out of childcare a little bit here and there were still eligible for the program because it was felt like they hadn't really engaged in that early learning yet. 

00:14:45 

54 Reasons is largely responsible for driving the referrals through their own networks, and of course through external services like community based services and healthcare providers. 

00:00:00

So welcome to module 3, which is all about defining your program. So really just getting into the nuts and bolts on what your program design is all about and really we want to focus in this section on what's different in your outcomes for when you're thinking about outcomes-based contracting. So what's the extra things you need to think about in your program design because you're doing an outcomes-based contract.

00:00:26

So when you sit down and define your program, you need to think through all the usual sorts of things. The why, the what, the who, the where, the how, the how much. We already talked a lot about the who in the last module, but in this one we're going to talk about all the other elements.

00:00:46

So let's talk about the why the why you're doing your program and that really duels down in your into your program logic and your program logic is

00:00:55

also called a theory of change that tells that story, it provides that framework for how the activities and inputs that go into your program help to deliver the outputs, outcomes and the impact that you're trying to achieve.

00:01:10

And really what I want to highlight here is that in an outcomes-based contract, being confident in your program logic or theory of change is just so important. It's really important because you're paid based on outcomes not based on inputs. So knowing and being confident that the activities and inputs you're going to deliver

00:01:29

can achieve those outcomes is critical. So what that means is that evidence is also critical.

00:01:38

So you need evidence. you need to track down evidence that helps you demonstrate that your program logic is actually going to deliver the outcomes or impacts that you're seeking.

00:01:49

And oftentimes, we find that the level of evidence we might ideally want just isn't available. So we look at the top of this list. So has a program been tried before? Let's say you're thinking about an established program. You might be able to find some really good quality evidence, even a published randomised control trial

00:02:09

that helps to articulate the impact of those sorts of programs.

00:02:13

But really, quite often in social impact bonds and outcome-based contracts, we're not talking about just repeating established programs. We've got, we're looking at programs that have some sort of element of innovation. It might be drawing on a program that's been implemented somewhere else in Australia or overseas

00:02:33

and moving it to another part of Australia. It might be tweaking one of your existing programs to really extend and address some of the problems you've found in the past.

00:02:42

And what happens is that the quality of evidence you're able to find to actually demonstrate or explain the impact of that program just gets harder and harder to find, and we often find that for particularly innovative programs, you might actually be relying only on anecdotal evidence.

00:03:00

But what I'd encourage you to do is think hard about whether what you're doing, whether what you're proposing, is truly innovative, or if there's actually some similar programs or components that you can find somewhere else that will help you build that evidence about what the potential impact of your program might be

00:03:20

Next element is the what, we need to define our services. Clearly you need to do this for all your programs. So really what I think is worth highlighting here is that oftentimes when we're doing an outcomes-based contract, we're trying to reduce some of the problems that we've seen in our existing programs

00:03:40

because we know there's some gaps or whatever that help will help us to lead to better outcomes. If we can just address those. So that's th comment at the bottom, is it your program as is or is? Is it with a twist? Are you trying to address the piece of the puzzle that you've never been able to address

00:03:55

that you know, or you're convinced it's gonna help you to achieve that or impact. So think hard about exactly what your services are going to be and also the different cohorts that you might be including as participants. So are there different phases or levels of support you're going to provide? How can you describe that? How will people move

00:04:15

between those phases and how will people get measured during those phases?

00:04:20

Do the services, the volume of services differ by your different cohorts and segments? And really important to think about is how your service is going to complement existing support for participants. We'll talk about that a little bit more when we talk

00:04:36

about the how you deliver the program.

00:04:41

So let's talk.

A little bit more about the who

00:04:42

We've we talked in a previous module about the target beneficiary. So all the people who might be potentially eligible for your program and meet all those eligibility criteria.

00:04:56

Then you've got a referral process that gets people into your measurement

00:05:00

group and oftentimes, if you look at the colours of those dots, this is not quite obvious, but you look at the colours of the dots in the target beneficiaries and the measurement group in this diagram, you'll see they're very similar. We really want to make sure that the referral process works well, so that the measurement group is reflective

00:05:20

of your target beneficiaries. So the people that are getting in there are actually reflective of your target beneficiaries.

00:05:28

It's really important though, to understand that your measurement group might not be the same as the participating group. People might drop out of your program and so you are going to get measured on that measurement group, but you're participating group might be the ones that you are actively involved and you can really deliver change for. There might also be the

00:05:48

group the number of people you think about in terms of when you're trying to cost out the program, so it's really important to to distinguish between.

00:05:56

Who's going to be measured? And then who might actually participate throughout both of the groups really need some clear rules.

00:06:05

So let's talk about the when. We're really talking about the length of your service delivery and this is something that's often quite restricted in an outcome-based contract because it influences when and how you can measure an outcome. And that's just really terribly important. So what we often find is that

00:06:25

it's really important to think about setting a maximum time that each person could be involved in the program.

00:06:32

And then when you're thinking about how much the program is going to cost, you need to work out what's the average time. If that's your maximum, well, are people gonna finish before? Drop it before or whatever. So plan you're costing around the average amount of time a person might spend in the program. It's important to understand what does it look like when someone exits.

00:06:52

So are they going to exit into another program? Are you going to have a process for ensuring that you say yes, they've exited?

00:07:00

So and when you're thinking about what the total contract length might be, it'll be the number of years you're enrolling people. So you might, you know, enrol people over a three year period for example. And then the number of years of service delivery, so you might be delivering services for two years.

00:07:20

So you'll end up with a four year.

00:07:21

Program overall and then you need a little bit of time at the start and a little bit of time at the end to make sure that you've allowed to be your setup and your scale up time. And then separately for you scale down and also measurement time because often time for the final measurements of people and their outcomes happens

00:07:39

a little bit after the end of the service delivery period.

00:07:47

Next, it's important to think about how you're going to deliver the program and the extension which you're going to have to collaborate with other service providers to achieve your impact. So you might think that you couldn't deliver the program as a single provider, which is fantastic.

00:08:05

But have a think about whether or not you've got the capacity to go to scale, whether you've got all the referral networks and contacts in a new region. If you're thinking to go to new region, for example.

00:08:18

And just the practicalities of how it might work.

00:08:22

If you think that you need to collaborate, I think it's important to understand that

00:08:28

collaboration can be really positive. It can increase the potential innovation in a program and the potential breadth of services that you can provide.

00:08:39

But it's also increasingly complex, and we've spoken to other providers service providers involved in outcomes-based contracts who, who sort of say it's really difficult to try to get into a partnership in an outcomes-based contract. So if you're going to do an outcomes-based contracts and you're going to get involved in partnerships in some sort of way,

00:09:01

it's really going to be much better if you already know those people.

00:09:04

There's a lot of complexity, a lot of moving parts in an outcomes-based contract. So you know, we'd usually encourage you to think about people that you were, organisations that you've already got a good existing relationship with and that you'd be confident to work with over the long term.

00:09:21

If you are going to be collaborating with others, think about who's sort of got the lead. So who's driving the overall design of the program? Who's playing that key role in delivering and got that primary accountability? What's the practicalities of how you work together? How will client information be shared and coordinated?

00:09:42

Are there differences in risk appetite between the different organisations? So can you both or all of you know share in the potential downside or potential upside if the outcomes aren't achieved and who's really on the cards for achieving those outcomes?

00:10:02

How will you share program costs, risks and benefits over time?

00:10:08

So there's a bit to think about if you are going to use a partnership, but you really need to think about whether when you're thinking about how it's, it's another place to think about whether or not you are capable or able to deliver all the services that this group of people might need, and if not, do you need to partner with someone else or?

00:10:26

And you can refer them other existing sort of services. You can refer them to to deliver those services. It might also help you think about your

00:10:38

target beneficiary group. So for example, if you make a decision that you don't have the skills and confidence to deal with a particularly difficult group, maybe it's a group that's got drug addiction or something like that. You might make a call to exclude them from your program or this particular outcome-based contract.

00:10:58

So you can see how all these pieces of the puzzle are quite iterative and it's really important to think through how you'll deliver the program.

00:11:07

So next, how much and it's really important that you are cognizant of all the costs that might be involved in delivering a program like this because it's a longer term contract and if you get the estimates wrong, then you know you're sort of living with the under estimate of costs over a longer period of time. And that can be quite challenging.

00:11:28

Thing extra things that we think are really important to think about. So think about your setup costs. You know, if you're going to be scaling up very time, if you've got a lot of recruitment to do, think about factoring in enough funding to cover those things, acquiring new capability. This is really very important. So we often find that

00:11:48

there's a need to implement new systems, particularly data systems or client management systems, to help to record and track clients.

00:11:58

And record and track their outcomes over a period of time.

00:12:04

And in fact, we've seen in a couple of programs that one of the things that has been costed into those programs is a data analyst of some sort, as well as some data, improved data systems because data is just so fundamental to outcomes-based thinking.

00:12:21

Apart from that, just be cognizant that overheads for some of these contracts can sometimes be a little higher than normal. You know, you might have a lot more senior leadership time involved in thinking through the program and perhaps more back office

00:12:37

support, you know, finance and so on to help you to, to actually deliver this program. So just make sure you've factored in your ongoing overheads.

00:12:49

So just returning to our case studies for today. So the  Side by Side, social impact bond, the why. They were focused on improving engagement and learning outcomes of primary school students. The challenge for them in their program logic was they found it difficult to find evidence to demonstrate that this program or what the extra impact of the program might be.

00:13:12

Part of the reason for that was both VACCA and Berry St had data on the performance of their programs, but they actually weren't measuring the same sorts of impacts or outcomes, and so they had very limited evidence to do that. And so the program from here is going to draw on evidence held by the Department of Education around participation rates.

00:13:37

So I think that was just a little bit tricky at the start for VACCA and Berry St to really settle on what the potential impact of their program might be for play. The Play2Learn+ in Tasmania, the why was focusing on how to improve child development and well being for at risk children and their families.

00:13:57

And again, they struggled with the limited evidence base.

00:13:59

Although it's important to know that 54 reasons already had a number of programs, so they had a Play2Learn program in other states, which they were sort of using that program to build on to expand. So they had some data from their existing programs but used a bit of guesswork I suppose to think about what the

00:14:19

outcomes and impact might be of this refined program moved to Tasmania.

00:14:27

So looking at the what. So  Side by Side, they've got a model that we showed in module one that uses a mixture of student and family support at home, educational case management, professional learning and as well as mentoring for teachers. So they had to make sure that they factored all those

00:14:46

elements into their into their costs, and actually worked out who VACCA or Berry St was leading on all those different components.

00:14:53

Play2Learn+ targeted goal-oriented group work and home visit and virtual support. So that was that was pretty clear.

00:15:04

The Who? It's interesting both Side by Side and Play2Learn+ are targeting around about 300 children in their respected programs. But of course you can see that they've defined those children and who's eligible to eligible to access the program a little bit differently.

00:15:25

So the where?  Side by Side had a target to work in nine different schools in those three regions in Victoria and they had a staggered school commencement, which enabled them to sort of gradually scale up their program and also refine a few important elements for the first couple of schools so they could get it right for the subsequent schools.

00:15:47

For the Play2Learn program, it was more of a quick ramp up and that proved challenging because 54 Reasons didn't have as good a footprint in terms of a referral network already existing in Tasmania as they might have had in some other states. So it just took a little bit longer to get

00:16:07

moving and I think that's really important to think about. If you're taking something to a new area, make sure you know how you're gonna build up the referral network over a period of time.

00:16:17

So the when? Again you can just see how both programs have thought through the period of time that they're going to provide some support, you know, so 12 months   12 months support for the  Side by Side program, for example, five annual intakes. So adding that extra year

00:16:36

Takes it to a total of 6 years of service delivery and of course their contract is around about seven years. Having up some scale up time at the start as well as measurement time at the end.

00:16:47

The how so the two service providers in the case of Side-by-Side VACCA and Berry St so important things to think about in terms of the partnership and in the case of 54 reasons in in Tasmania, just the one service provider.

00:17:03

See end of module 3. So in the next module module 4, we're finally gonna be talking about outcomes and how to measure outcomes. Thanks.

00:00:01 

Module four of our workshop on outcomes-based contracting and it might seem strange that it's taken all the way to module 4 to really start to talk about measuring outcomes, cause it seems so central to what we're talking about, but this is hopefully gonna be an interesting module for everybody. 

00:00:20 

So today, measuring outcomes, we're gonna be talking about two elements. So outcome metrics and also how you might think about the extra impact of your program. 

00:00:31 

So what is an outcome? An outcome is the changes that occur for people, organisations, systems or communities during or after a program. 

00:00:42 

And so if you think about your program logic, it's really that piece that is, after all, your activities and outputs, but not perhaps 

00:00:51 

not as far away as your impact, so examples like reductions in reoffending, sustained employments, those sorts things. 

00:00:57 

And really the key thing in an outcomes contract is that it's really essential that outcomes are measurable. 

00:01:06 

So some thoughts on selecting an outcome metric. 

00:01:10 

You first of all, it's really important that your outcome metric has alignment with the impact you're hoping to achieve. So we think it's best to have one or two simple metrics can which can be considered a proxy for the improved outcomes you're seeking. 

00:01:25 

And it's important to recognise that you may not be able to see the long term impact of your program, so you may need to be happy to flex to include short term or intermediate outcomes as your metrics, because they're more easily observable in the lifetime of the program. 

00:01:43 

It's got to be practical to implement, so it's important to think about what data might be available and ideally there will already be existing data collected about your outcome metric that will help think about historical data as well. 

00:01:58 

And if there isn't that good quality data, think about what will be needed to access and create some new data. So what time or costs or consents or approval process is going to be involved to access that data. 

00:02:11 

There's also this issue of objective versus subjective outcome metrics. 

00:02:21 

An objective outcome metrics like for example the number of people or whether or not a person is employed or whether or not a person is participating for a certain number of days. Those are all fairly objective things to measure. 

00:02:39 

But there's a lot subjective outcome metrics too. So something like improvement in educational attainment might be considered a subjective metric, particularly if it's recorded on the basis of a teacher perspective, for example. 

00:02:56 

And our experience is that governments are often a lot more comfortable with objective measures because there's often historical data. Everyone can agree that, yes, something has or hasn't happened, whereas with subjective measures you've got to be careful to make sure that the way it's being measured is actually something that's fair, it's not being biased or influenced by the fact that there's an outcome payment all linked to this. 

00:03:21 

Lastly, it's important to think about whether the outcome metric that you select actually gives you the right incentives. So sometimes an outcome metric might be achievement of a certain number of performance indicators or achievement of a certain level of educational standard, for example. 

00:03:43 

And sometimes what happens with those sorts of things. So there's very black and white sorts of metrics is that if a person's either clearly missed that metric, so for example, they return to gaol, for example, or if a person has clearly already achieved that metric, sometimes they don't get continued support from a program and 

00:04:03 

I have to think about really whether that's fair or not. So it's really important to think about whether the incentives built into your metric are aligned to delivering the program in the best possible way. 

00:04:15 

Data is so important when you're thinking about outcome metrics, and there's a range of possible data sources that might be relevant. Often times the contracting government department that you might be working with, like the Department of Education for example, might hold data that is relevant to your program. So it's important to understand what they hold 

00:04:35 

or what they're able to access. For example, schools might have a different data sets that are not held by the Department of Education, but they could have some really useful data to help measure your outcomes. 

00:04:46 

So it's really important to understand who holds what and whether you can access that. There might also be third parties who have got useful data, like a childcare centre or an employer. 

00:04:58 

Sometimes other government departments that aren't actually the one you're contracting with can be thought of as a third party provider. 

00:05:06 

Who else has got data that might be relevant to your metric? 

00:05:08 

Sometimes you the service provider might be responsible for providing that outcome data and sometimes it's participant led data as well. 

00:05:21 

But the key observation for me is that as you head further down, that list becomes more difficult to gather that data. There's probably more cost in trying to generate it, so you might be conducting surveys for example, or, you know, teacher perspectives on outcomes, etc. So you really have to think about how you're going to get that data and it's also decreasingly robust. 

00:05:45 

What do we mean by that? There's a concern that sometimes you can get bias, particularly with participant or service provider data depending on the way that's captured because obviously there's an incentive for the service providers to try to demonstrate they've got a better outcome. 

00:06:01 

So my experience has been that governments prefer to see more objective metrics that are collected in data sets that they've seen are like their own data sets or a third party data set if at all possible. 

00:06:18 

There are different types of metrics that you might see. So sometimes these are binary metrics, so it might be a pass, fail sort of situation. So for example, a child either does or does not achieve 17 or more KDC checks. So that's the Play2Learn program that we've been talking about today. 

00:06:42 

Some of them can be a frequency with the numbers values over time. So for example the number of KDC checks, there's a maximum of 21 and you can achieve any number from 1 to 21. So that's a a frequency I suppose. 

00:06:58 

And the bottom one is a continuum, so a frequency continuum. So for example, the school attendance rate over years is broadly a continuum. So how to think about what your metric types are? Sometimes if you use a binary pass file metric, there can be a real tension 

00:07:18 

about whether or not that's the right outcome. So for example, if a child achieved 16 of those KDC checks, is that good or bad? Because if you've got a binary metric like the one described there, that would just be considered a fail. 

00:07:34 

And so really, is that appropriate? It's just something to think about within the context of your program. 

00:07:42 

OK so here are a number of examples in the employment arena that I think drive home how many different metrics could possibly be out there. And this is just one subject area employment and you'll see that the data source for a lot of these is the Australian government. 

00:08:01 

Now all of these sorts of metrics could be relevant, depending on what your program is trying to achieve and who your target, what is. So by way of example, if your target cohort is a group of people who are on welfare, that third metric down ‘off welfare’ may be appropriate. 

00:08:22 

That's a great example of a binary metric, so whether or not a person is off benefits for at least whatever it is 10 weeks, that is binary either yes or no they were or were not. 

00:08:36 

And but you know equally, there are other examples where you might get a frequency or a continuum of metrics. So income is a great example where the Australian Government would have access to some data about people's income through tax. We've sort of not really been able to access that data through programs I've been involved in because of privacy reasons, but it's important to know that that data does exist. 

00:09:05 

What you need to be careful of is metrics that end up actually being ambiguous about whether it's a good or a bad thing. So the last two examples that ‘off welfare’ example, are the incoming example. If those two metrics increase, well, that's a positive thing. So the aim is to increase those metrics. What about job placements, you know, is that 

00:09:26 

positive or negative thing? Is more job placements good or is it bad in terms of someone's employment outcomes? 

00:09:31 

So you have to be really careful to make sure that whatever metric you choose is clearly linked to the outcome you're trying to demonstrate. So it's not ambiguous what a change in that metric might mean. 

00:09:46 

Turning to our two illustrations for today. First the  Side by Side social impact bond has got two different outcome metrics. So the first of those is school attendance and this is the primary metric for the program. So more of the outcome funding is linked to improvements in school attendance. 

00:10:07 

And so the metric is the number of recorded days absent during each of the intervention year, so the first year that the child participates in the program, and the following year, which is called the monitoring year. And I think it's just worth highlighting that there's some capitalised terms there recorded days absent. 

00:10:27 

So they have been very specifically defined in the contract. 

00:10:32 

The second outcome metric is about educational attainment, and this is at this stage considered a more subjective measure and that's part of the reason that the metric attracts less outcome funding. So it is a level based on the english online interview or teacher judgement so. 

00:10:58 

Two different types of metrics. One of those had some good historic well has some historical data now. So that's the school attendance metric. And that's partly the reason why it has been more of a focus for the outcome payments. 

00:11:15 

So focusing on the Play2Learn+ payment outcome contract. So three different outcome metrics are involved. So first of all, there's an engagement one which is literally an enrolment are based metrics. So the number of children enrolled in the measurement group, you'll see lots of capital letters there. Again, lots of defined terms. 

00:11:36 

The second one is based on attendance in ‘launching into learning’ sessions, so this is a binary measure. So basically the proportion of children that attend 10 or more little sessions. And so if a child achieves 9 of those little sessions, unfortunately they don't meet the performance benchmark.  

00:12:05 

Similarly outcome 3 is defined in a similar way, so children that achieve 17 or more of those KDC cheques. So two are quite binary outcome metrics. 

00:12:16 

It's important with those two outcome metrics that we just mentioned, the Lil or ‘launching into learning’ metric and the KDC check metric is that both of those are something that was routinely collected by the Tasmanian government as part of the process. So it was data that there was historical data available for and it will be routinely. 

00:12:37 

Collected. So, for example, children who enter kindergarten have that KDC check done when they first enter and six months later. So there was good historical data to help understand performance and help 54 Reasons think about how their performance could change relative to what had been achieved today. 

00:12:58 

So we're going to move and talk about the next component of the puzzle, your program’sprograms, extra impact and why this matters is because your outcome payments are most likely to be linked to how much extra impact you're going to be able to achieve compared to what might have been achieved in the absence of your program. 

00:13:19 

So if we take this really simplified example where you've got 100 children participating in the program, and let's assume that the goal of the program is to increase the number of children who finish school. So your program looks successful because 75 of the 100 children finish school, so we can call that an absolute outcome. 

00:13:40 

But actually, you know, government might come back and say, well, the number of children would have expected to finish school, even in the absence of the program, was probably 50. 

Of the program was probably 50. 

00:13:50 

So what that means is your extra outcome is only 25 students, and that's by comparing it to what we call a baseline or a counterfactual. So those fifty students, what we would have expected in the absence of your program. So in most outcome-based contracts, 

00:14:10 

your impact is measured relative to some sort of baseline. So the estimation of what would have happened in the absence of the program. 

00:14:19 

Now there are lots of different ways to try to estimate that baseline and try to compare the impact of your program to that baseline. But, 

00:14:30 

what we really strongly recommend is thinking pragmatically about what's really possible, given the size and scale of the program you're planning to do. 

00:14:42 

So, you know at the right-hand end of the spectrum are things like randomised control trials, for example, to compare the outcomes for two different groups in a randomised way. 

00:14:54 

Very research, heavy approach, very costly approach and not an approach that's been used for outcome-based contracts, at least not in Australia. So while that's always considered the gold standard for evidence, it's just not practical oftentimes for outcomes-based contracting. 

00:15:15 

We have seen I have seen at least one what's called a match control group where there's really good historical data and we use that really good data to create in the group a match group that has got similar characteristics to the characteristics of the people that are in the program. So similar 

00:15:35 

Age, gender, socioeconomic status, all those sorts of things. So it's called propensity score matching and we compare what's happened to that group with what's happened to the group that is measured in your program. I've only seen that in one 

00:15:51 

Program off the top of my head, but that was a very large program. There are about expected to be about 7000 participants in that program over 7 years and it's a healthcare program, it's, you know, and there's very solid detailed data for that. 

00:16:11 

So again, that's to me is quite unusual. What we tend to see is more down the left-hand end of the spectrum where you might use 

00:16:22 

And educated guests, oftentimes the individuals pre-program experience so you know how? An educated guess, oftentimes the individuals pre-program experience. What was their attendance rate like last year or even we often see a fixed estimate. So you know, 50% of students might achieve this and that becomes your benchmark going forward. 

00:16:41 

So it's really important to think about what's feasible and the cost and complexity of trying to build that evidence. 

00:16:49 

It's really important to understand that it's not just the effect of your program which will influence the measured outcome, and particularly this happens when you've got a program like an outcome-based contract that can be delivered over multiple years. You know, often times you know six or seven years of contracting. 

00:17:09 

And lots of other things can happen in the external economic environment. You can have a COVID pandemic. You can have different demographic trends, you know, and policy settings can change. 

00:17:25 

And I always enjoy this this slide, which is really relevant here because you know I called you all here because I need you to stop helping people. It's really messing up my impact assessment. You know, oftentimes there are multiple programs going on at once that some of your some of the people involved in your 

00:17:45 

participant Group might actually be able to engage in all these different programs too. There's a lot of things going on the outside or people who are not part of your 

00:17:52 

program might suddenly get access to another new program and so when you're comparing yourself to them, suddenly there's a different there's a different baseline to compare to. So I think it's important to understand that there is a lot of complexity out there in measuring impact over time and particularly for these longer contracts and most contracts 

00:18:13 

do have some sort of lever, generally called a baseline review or a counterfactual review that helps to deal with the fact that all these external factors might actually influence the program. 

00:18:26 

So going back to our examples again, so the  Side by Side social impact bond there the baseline or what would have happened in the absence of the program is measured based on the number of days absent for each student in the year prior to referral. So if they had high number 

00:18:45 

of days absent in the year prior that influences the best estimate of what might happen in the intervention year, and there's a progression factor. So usually when people have a very high level of absence in the following year, it might be a little bit lower. So in that there is an analysis of historical data to see what the typical progression might be. 

00:19:05 

In the Play2Learn+ Tasmanian contract, 

00:19:14 

the different outcomes are measured in different ways. So for the engagement measure, which is the number of enrolments, that's just a straight absolute measure. So every enrolment counts. 

00:19:29 

For the launching and learning (LiL) attendance and also for the development milestones that's been based on fixed rates. So there is an analysis of historical data 

00:19:43 

for the target beneficiary group, so the target group of people that look similar to people who might end up in the measurement group. So there's analysis of historical data, and specifically in relation to those KDC development milestones, there were fixed rate set that were different for boys and for girls. 

00:20:03 

There was also an adjustment to consider what was actually happening as a trend over time. 

00:20:10 

So unfortunately it looked like over a number of years those KDC attainment levels were actually declining. So when kids were coming into kindergarten, they were actually having lower attainment rates. Now lots of assumptions about what might be happening in the external environment to drive that, is the use of screens, zones. All that sort of thing, that it was reducing that over time, not clear, but it was important to sort of and analyse the historical data to see what was happening and adjust for those trends. 

00:20:42 

So that's the end of module 4 measuring outcomes. The next module, module 5, will start to talk about the funding arrangement and pulling it all together. 

Outcomes-based Contracting Module 5 1.mp4

00:00:00

So welcome to module five of our workshop on outcome-based contracting and this module is about building the funding arrangement. So we're going to be looking at three different parts of that. So the payment structure, the contracting arrangements and risk sharing. So let's start with the payment structure. So first and foremost,

00:00:21

let's think about the considerations for your payment structure. So the boundaries that you're going to have to work within.

00:00:27

So 4 elements we think are important. So first of all, what's the available funding pool? What's the maximum government funding available for this program and are there other sources of funding you might be able to leverage like

00:00:42

philanthropy? The other thing

00:00:44

to look at is whether there's any specific requirements

00:00:47

that are in place around that funding pool. So for example, has the Government indicated that there's a maximum pool of funding available that they would be prepared to pay in advance or a maximum proportion of funding that they'd be prepared to pay in a fixed sense? So, you know, wanting to see a minimum amount

00:01:06

at risk.

00:01:07

The second element you need to think about is your program delivery cost. So how much do you think it's going to cost to deliver your program and actually what's the timing? When are those costs going to occur? So you wanna make sure your payments are linked broadly to when the costs of your program are going to occur or have some way to have external resources or

00:01:27

other resources within your organisations to back you up if that timing doesn't quite work out.

00:01:34

So the outcome metrics are really critical to thinking about the payment structure. So how long after enrolment will you actually be able to measure the outcomes? What's the flow of participants through your program over time? What's that preferred outcome metric? So the outcome metrics

00:01:54

are really critical

00:01:56

in defining your payment structure and also the timing of your payment.

00:02:01

Last but not least, we've talked about risk quite a lot. So it's really important when you're designing the payment structure to think about what's the biggest loss you could actually as an organisation tolerate and if that loss isn't so much, is there some other way someone else you could get to help to support you, philanthropists, investors and so on

00:02:21

to provide some Co funding to share that risk of loss.

00:02:27

So there are really three different payment types that typically happen in an outcome-based contract payment structure.

00:02:36

So the first of

00:02:37

those is the fixed component and that might be called a standing charge or the advance payment, often times like a grant, I suppose. It's paid quarterly in advance and sometimes that's payable on term, repayable on termination.

00:02:53

Typically this represents a fixed amount of your estimated budget budgeted program expenses.

00:02:59

The proportion varies a little. We've seen in social impact bonds that it might be as much as or as low as 50% of your costs are paid. That's a fixed amount.

00:03:11

The rest is at risk, but in other contracts, so in payment by outcome contracts which don't involve investors, for example, or might be smaller scale, it might be something closer to 70% so it just varies.

00:03:26

The second element is at risk or variable amounts, so these are amounts that are based on your measured outputs and outcomes.

00:03:35

So outputs linked payments might be linked to something like the number of enrolments or the number of participants, or the number of job placements. So this is more likely

00:03:47

to be paid

00:03:48

for all output, so rather than the extra impact that we talked about before.

00:03:53

Actually, it's typically paid

00:03:54

for all outputs.

00:03:57

And usually an output based payment is there as a way to mitigate what we call volume risk and provide some interim cash flow throughout the program while you're waiting for those outcomes to be observed and measured and paid on.

00:04:15

So the last piece of the at risk component is the outcomes-based payment which we've talked a lot about and that's where you really focus on the measured achievement of your outcome metrics.

00:04:28

And again, these payments are usually made

00:04:32

on a relative

00:04:32

basis, i.e. what's the extra impact of your project?

00:04:38

And how could at risk payments be structured? There's three typical options, and probably for this particular program it might be more likely to see the first of those two options. So a proportion of program costs,

00:04:54

So this is something we often see – so payments are expressed as a percentage of the target payments or the program costs, and that the percentage figure that you get, it might be 10% of that target or 20% of that

00:05:10

target or what have

00:05:11

you determined by the level of performance.

00:05:16

We've already talked a little bit about how performance is measured. Oftentimes, if you've got measurements year by year, so you've got, you know, say three years enrolment groups, you might measure the outcomes for each of those enrolment groups separately.

00:05:36

But oftentimes for

00:05:38

payment purposes, you actually look at the cumulative results for that group and again that goes back to the initial discussion we had in one of the earlier modules where we talked about how when you've got a larger measurement group, you can be more confident in the outcomes that you're seeing. So paying an accumulative way is a way

00:05:58

to give you more

00:05:58

confidence that the outcome that's being, seen is more is a true reflection of the outcomes of the program.

00:06:08

So the second approach is the payment by outcome approach. So for every person, for example, who achieves a good outcome, you get a payment of a certain amount, such as $5000 per job placement.

00:06:22

What that means is that the program is dependent both on the number of people to participate as well as your rate of success. So if you've got a target to achieve, for example 50 job placements, you can achieve that either by

00:06:40

getting more people in your program or by your program

00:06:44

being more successful.

00:06:46

So that's a good and bad thing, but it's important to understand that a rate card or payment by outcome approach means that you're on risk for both the number of participants or the number of referrals, as well as the rate of outcome success.

00:07:06

The last example is something we don't see so often, but it's we thought it was worth highlighting that payments can be expressed as a proportion of the value generated. So remember, Government is often interested in outcome-based contracts because they see it as a way to save money down the track. So for example,

00:07:25

it might be that the service provider gets 100% of the value of the 1st $10 million value generated for.

00:07:32

Government. And then there's a sharing between Government and the service provider about the value that's generated beyond that. This is something that has been of interest, for example, in healthcare costs where you can often see quite directly the savings in in hospital costs for example,

00:07:54

by keeping people out of hospitals.

00:07:59

Also it is important to think about what the payment levels might be depending on different levels of performance of your program, so it's not important you don't need to just need

00:08:10

to think about

00:08:10

what happens if you reach your target, but also about what happens if you achieve something below

00:08:18

or above that target.

00:08:20

There's three examples that we provide here. So one is a binary approach

00:08:25

Which i.e. either you achieve the 10% target in this example or you don't achieve that target and the entire payment amount is based around whether or not you achieve that 10% improvement target. And as you can see that is really quite a challenging way to set things up because if you get very close but just under if you get 9.5%

00:08:48

performance improvement when you are hoping to get 10%, will you get no outcome-based payment and that's gonna be a really difficult place to be. So we wouldn't typically encourage a binary approach.

00:09:00

But you could consider the next step down, which is bands, so you might actually have different performance bands. So for example, if you get a 0 to 5% improvement, you might not get a payment, but if you get 5 to 10% you get the next band and so on. So this is something we see more often, but it's really important to think about

00:09:20

what happens? How big the jumps is between those bands? So again in this example you know if you get 9.5% improvement versus 10.5% improvement, there's a very different amount of outcome payment that gets paid.

00:09:37

The last option we provide there is a linear option so you get a dollar amount based on a percentage of the target. So let's say you hope to achieve 10%, then you get you know your relative proportion compared to that 10%. So it's a linear

00:09:56

amount up to some maximum level and that helps to manage that sort of risk or challenge of just being just above or just below any of those performance bands.

00:10:07

So let's turn to our two illustration examples. So firstly, the  Side by Side social impact bond, you'll see that there's a standing charge payment. So a regular payment, a payment made up front, it's paid on a quarterly basis based on the cost of the program. So that's the fixed payment, fixed component of the payment for this

00:10:27

program. Then there's two outcome payments and

00:10:30

they're paid at

00:10:31

slightly different times, so the attendance outcome payment, that's an annual payment which is made once we start once we get to the

00:10:41

end of a school

00:10:42

year and the attendance outcomes for that cohort can be measured

00:10:47

so it's also paid on a cumulative basis.

00:10:53

As the measurement group continues the actual improvement, the cumulative improvement is re estimated by adding each new group into the total and the amount of payment is based on a percentage of the target amount based on the overall average percentage improvement

00:11:12

in attendance that's been recorded for that group. So it's a linear amount, so there's not so much risk of jumping between performance bands.

00:11:23

There's an attainment outcome, so that's linked to improvements in educational attainment for people in this group. There's a smaller relative amount that goes to this particular type of outcome payment because there was it was a new tool that was being used for this program, so less confidence in the use of that

00:11:43

tool to set outcome performance payments.

00:11:47

It's paid at the end of the contract and it's got that tiered approach to payment. So there's four different tiers based on the cumulative improvement in educational outcomes.

00:11:58

So the Play2Learn+ program is a little more complex I suppose. So it does have that fixed payment, quarterly payment that is the fixed component or the standing charge. In addition, it's got three different types of output or outcome payments. So there's an enrolment payment which

00:12:15

is, I guess, a rate card a dollar amount

00:12:18

per person that's enrolled and that's paid when those enrolments happen

00:12:22

And then there are two different types of outcome payments, so

00:12:26

the first is

00:12:27

the Launching into Learning outcome payment and that's payable based on the percentage of children who achieve 10 or more or participate in 10 or more Launching into Learning

00:12:43

sessions and similarly the KDC outcome payment is based on the percentage of children achieving or 17 or more KDC1 markers.

00:12:56

And both of those are payments actually are tiered payments. So there's several different tiers, just depending on the proportion of children who achieve each of those outcomes. So there's some quite steep jumps in between those things. So it makes it really important to be sharp and clear

00:13:15

around how well those outcomes are achieved.

00:13:25

So I hope

00:13:25

it's been clear

00:13:27

that when you set performance targets, it's really important to be confident that it's possible to achieve those targets and their data that you're using or the evidence that you're using to help you set what targets, what performance you can possibly achieve

00:13:44

is really appropriate, so we'd encourage you to refer to the available evidence or similar programs or your own data, and think hard about whether that evidence is actually relevant to the current context.

00:13:56

Be aware of

00:13:57

the limitations of your own data. One particular tricky thing we see is that oftentimes when people look at their own data, they only look at the outcomes for people that stayed in their program from start to finish and actually forget that perhaps 10-20% of people dropped out of the program and actually oftentimes in an outcome

00:14:16

based contracts, outcomes for those people are measured too, so your impact might be less than you than you realise if you apply it to this outcome-based contracting world.

00:14:26

And balance that desire to sell the program, i.e. set higher performance target and being realistic so you don't set the program up for failure.

00:14:36

And lastly, it's really important to deeply understand what the implications are for your organisation if you miss your targets. So look at the spectrum of possible or results, not just whether you reach the target, but whether you underperform or over perform and think hard about what assumptions you've made along the way.

00:14:57

So building the funding arrangement, we've talked about payment structures and now we're going to talk about contracting arrangements.

00:15:06

So, in terms of contracting structures, oftentimes a service provider will contract directly with Government and the contract may look very similar to a grant agreement with a range of additional terms that help to define outcomes. And of course, all that extra detail that we've already talked about in terms of defining who's eligible, how performance will be measured and so on.

00:15:28

There may also be a range of subcontracts between the lead service provider and other partner organisations, and there may be contracts around that help to access special data either from other Government departments or from other service providers who have important data for your program.

00:15:46

In contrast, a contracting with the SIB may actually be somewhat more complex, and we've got an example straight up after this. The reason they can be more complex

00:15:55

is that there may

00:15:56

be special contracts with investors that set out their terms and then maybe you need to set up a special purpose vehicle to aggregate money from investors and also from government and ensure it's paid out to the all the relevant parties at the right time and place.

00:16:14

And so as you can see, the contracting structure for the  Side by Side social impact bond is somewhat more complicated than you know typical grant agreement

00:16:25

because of all the different parties involved and we would typically encourage that if you're going to get involved in a social impact bond or bring investors into your outcome-based contracting in any way, then we'd probably encourage you to consider engaging with an intermediately like Social Ventures Australia.

00:16:45

For others to act as an intermediary and help negotiate the right arrangements, this contracting structure, as you can see, Berry St is the lead partner with an outcomes-based contract and implementation agreement with the Victorian State government.

00:17:02

But they've also got a subcontract with VACCA for delivering their services, a loan agreement with the special purpose entity that was established so that  Side by Side trust, and then investors have a contract in terms of the way they will contribute funding to that and be paid investment returns if outcomes are achieved.

00:17:23

So there's a little bit more complex structure involved in social impacts bonds.

00:17:32

The last piece of the puzzle

00:17:33

on building the funding arrangement is risk sharing.

00:17:41

We've highlighted throughout the presentation that there are a range of different risks involved with outcomes-based contracts and we just wanted to run through some of the ones that are quite specific to outcomes-based contracts over and above what might be a risk for you typical program.

00:17:57

So first of all, thinking about that program logic or your

00:18:01

theory of change,

00:18:03

there's a risk that the program, even if delivered as designed, doesn't deliver the planned outcomes, because actually the program logic is flawed. It's really important to understand deeply your program logic and the assumptions you've made along the way

00:18:18

and to see if there's any of those assumptions, you can test along the way, because as you recall, it's not your inputs that you're getting paid on, but you're actually your outcomes. And so being confident that your program logic can deliver that is really critical.

00:18:32

Implementation risk is important, so even if your program logic is important is perfect, you've got to make sure you've got the service delivery capability to deliver the program as intended. So think about your staffing, process, control, quality control, all those sorts of things to make sure you can deliver the program as planned.

00:18:53

Volume risk is also important. So we see this often that the measurement group size ends up being lower than anticipated because you have lower number of referrals and it's really important in your payment model to think about who bears that best. So if Government is responsible for making sure referrals

00:19:13

come to you, then you know, perhaps it is appropriate for government

00:19:17

to bear the risk

00:19:19

that the volume is less than planned, but if you're responsible for finding potential participants through your own network, you might be responsible, you might bear the risk of volume being lower than planned.

00:19:36

Disengagement is important, so remember that if someone drops out of your program and they're included in the measurement group, well, you might actually end up having poor outcomes, even though there's a lot of people that

00:19:49

you weren't able to support.

00:19:51

Data reliability. So how confident are you in the data that is being used to measure outcomes? Is it complete? Is it accurate? Did you pick the right baseline? Are you comparing yourself to a fair benchmark and is that benchmark staying the same over time or is something changing in the external environment? And again, the policy is also really important to understand if there's something happening in the policy environment.

00:20:16

And lastly, there's statistical error, so making sure that your measurement group is of a sufficient size, that the outcomes that you measure are give you a real true sense of what the impact of the program is. So statistical area is important, particularly if you've got a smaller cohort group.

00:20:39

So what should

00:20:39

you do about that risk, so you know what that whole risk that you may not be able to achieve the outcomes you planned and that you may end up with lower outcome payments than you hoped for?

00:20:52

There's a couple of different ways that you might be able to seek funding to support you with that. So first you can consider using your own reserves, but it's really important to think about how much financial risk your organisation really can bear, and making sure your organisation and your board is comfortable with that.

00:21:11

You might be able to use philanthropy to fund it, looking at grants, or perhaps whether a philanthropist is willing to make an insurance, I guess, contribution so contributing extra to you if our performance doesn't go as planned and if you achieve losses, of course that's

00:21:30

a little bit of a difficult conversation with the philanthropist to ask them to pay

00:21:35

if your if your achievements are not up

00:21:38

to scratch.

00:21:39

It’s actually kind of a difficult conversation, so it might be easier to negotiate a grant up front. There's a possibility of an impact loan, which starts to bring investors into the mix. So looking for sub commercial rate loans, so philanthropic investors

00:21:56

who are willing to loan make a loan to you at a rate lower than a typical market

00:22:00

return and even loans that are performance based. And then lastly, obviously you can think if that risk that financial risk linked to the outcomes is too high, you might want to think about going for a social impact bond type structure. So, raising capital from private investors and asking them to

00:22:22

bear most of the financial risk of outcomes and performance don't turn out as planned. Of course, if they are bearing the downside risks, they'll also take the upside risk so that if outcomes are better than planned, they'll be hoping to take get an extra investment return for that.

00:22:40

So looking at

00:22:41

how the Side-by-Side social impact bond and the Play2Learn+ program have managed this. For the  Side by Side, I think it's really important to note that there's quite a bit of risk sharing from the this Victorian Government themselves through the amount of fixed payment they pay and also through the retention of that volume or enrolment risk.

00:22:59

Because it's a social impact bond, investors share in the risk as well through a $5,000,000 social impact bond contribution and their investment returns being

00:23:10

linked to the performance of the program. Interestingly enough, in this one both Berry St and VACCA have all also acted as investors in the program, so they will bear some of that, you know, financial investment return, either upside or downside depending on the performance of the program.

00:23:31

And then Berry Street got some additional residual risk in terms of the outcome payments. So if they bear some of the risk, if outcome payments are lower than planned.

00:23:43

For the Play2Learn+ contract, there's there are some material volume and performance risk borne by both the Department of Social Services and the Paul Ramsey Foundation, who's contributed to this program and then some volume risk is also borne by 54

00:24:02

Reasons because they had that engagement payment.

00:24:05

So it's important to note that in this case, because 54 Reasons was is responsible largely responsible for enrolling people and referring and finding people to participate in the program. They bear some of the risk that the volume of participants is

00:24:25

more or less than planned. And of course that the performance risk in terms of outcomes being better or worse, no investors involved in that one. So literally just borne by the different parties there.

00:24:38

And for each of these programs there's some checks and balances, I suppose. So if performance seems to be moving in an unexpected way compared to the baseline, and if there's a sense that perhaps the baseline is moving too, there's an opportunity to do a counterfactual review or a baseline review to see and ensure that the baseline is still

00:24:59

fair.

00:25:02

So that's the end of module 5. So our final module, module 6 will just look at bringing it

00:25:08

all together.

00:00:01 

So this is module 6, the last of 6 modules in our workshop on outcomes-based contracting and it's called bringing it all together. So really just some really key findings, 

00:00:12 

key themes for you to 

00:00:13 

think about as you develop your outcomes-based contract proposal. 

00:00:19 

So summing it all up what is included in an outcomes contract? So what we think are things that need to be thought about carefully as you progress are your eligibility criteria, your enrolment process, 

00:00:33 

how that's going to happen and who's going to be responsible for that and who's gonna bear the risk for that in the outcome payments. What your measurement group is so who are all the people who are going to be enrolled in that measurement group and how will they be measured? And it's likely that that group will include people who disengage from your program. 

00:00:53 

So think about that when you're thinking about your 

00:00:55 

impact. What's the scale of the program? So think about the scale of your measurement group and we think about scale in two ways. One is about thinking about whether or not the measurement group is actually sufficient to show the impact of your program, but also you know, it's clear that there's a lot of work involved in sort of designing an outcome 

00:01:15 

based contract. So it's important that this effort is worth it. So the scale of your program, you know, probably needs to be of enough effort to enough to justify all the effort that you're going thinking you're putting into this whole outcome-based approach. 

00:01:29 

Outcome metrics obviously clearly very, very critical 

00:01:33 

to the whole cycle. 

00:01:34 

And lastly, thinking about your baseline, so measuring the impact of your program relative to some sort of baseline. What would have happened in the absence of your program? 

00:01:49 

It's important to think about what capability you as a service provider are going to need to design and deliver on this program. We've identified 4 key themes. First of all is strategic alignment, so your organisation across the board needs to be clear on why you're engaging in an outcome-based 

00:02:08 

contract and have an appetite to be in the spotlight, cause often these can be quite transparent and the information well shared. You need to have really good governance and financial controls. So think about your corporate back office and whether they're they could upskill to understand what an outcome-based 

00:02:28 

contract involves. 

00:02:30 

You really need good client management systems, databases and processes 

00:02:36 

to help you 

00:02:36 

with that referral process to reporting or the client information, their needs, the services and also helping you to track those lead indicators that will lead potentially to outcomes. 

00:02:51 

We also think you need to be, you know, think about your staff in terms of whether they're adaptive and innovative because in an ideal outcome-based contract, your organisations will be looking at how well you're tracking toward those outcomes and adapting and responding your program to make sure that you can deliver what you hope to, what you plan to do. 

00:03:10 

And of course you need capacity to implement, so how will you access the right staff? How will you make sure you've got the right data and reporting and analysis to understand how you're tracking and really good financial analysis as well? 

00:03:28 

So a perspective from our experience on the success factors for something like that. So firstly, in our experience, time kills deals, we've talked about that long joint development phase and 

00:03:40 

it can be really quite difficult for organisations to last the distance through that phase, so really we'd encourage a really intense and concentrated effort to make those sorts of engagements and thinking and you know, contracting and negotiation as succinct as possible. 

00:03:58 

Agree the principles upfront with everyone involved. What is 

00:04:01 

it that we are aiming 

00:04:02 

for? What is the most 

00:04:04 

important thing that's gonna happen? 

00:04:05 

Out of this program, because it's difficult sometimes to achieve everything, so really agree, what's the most important thing? Is it about getting really rock solid outcomes or is it about testing a method 

00:04:17 

for measuring outcomes? Is it about saving money for government? Is it about something else? I think it's important to be clear for everyone around the table to be clear on 

00:04:27 

what really is the important principle. 

00:04:31 

Accountability. It's really important that every there is consistent involvement at key decision makers and that includes both from you as a service provider but also from the different government agencies and stakeholders involved. It's really important that you got decision makers at the table who are solution focused. 

00:04:50 

Understand the politics and the cultural challenges and really critically, critically understand and embrace the data and data is not always sexy but is absolutely essential to an outcomes-based contract. 

00:05:05 

So the potential challenges in an outcomes-based contract, it can be long in terms of the development period, it can be large in terms of effort to make it all happen, it can feel rigid. There's a whole lot of contracting that needs to happen and it 

00:05:22 

can feel a little bit niche 

00:05:25 

but on the   side, it's important to bear in mind the positives have come from all of that it can 

00:05:30 

be long in terms of the contract length as well 

00:05:33 

and it can 

00:05:34 

be large in terms of scale, the scale of program, the reach that you're able to make. 

00:05:38 

Like while the contract feels rigid sometimes, often times it's flexible in terms of the activities and how you deliver them. As long as you focus on that long term outcome. And lastly, it can be broad in terms of the learnings, it's where everyone is focused on outcomes. It's really helps everyone learn what works and what doesn't 

00:05:58 

to achieve real change. 

00:06:01 

So if you're 

00:06:02 

interested in learning more, we'd encourage you to focus on a number of different really good resources that are out there that collect information across the globe. So NSW Government has got its own website on the Office of Social Impact Investment which has a great knowledge hub with a whole range of tools and information 

00:06:21 

to help you. The Government Outcomes Lab up in the UK is a fantastic resource and the Brookings Institute collects information about what's happening in outcome-based contracting globally. 

00:06:33 

Social Ventures Australia has also got a range of useful resources, and while many of our resources have been focused on social impact bonds, it's important that the lessons are really relevant to a wide range of outcomes-based contracts. So I draw your attention specifically to the last group of material, the information memorandums, the annual reports, the legal documents and so on. So for every social impact bond 

00:06:56 

that we've been involved in and outcomes-based contract that we've been involved in, we put in all a lot of detail on the website so that you can learn more about the detailed design of these programs. 

00:07:08 

As well as evaluations, what have 

00:07:10 

we learned along the 

00:07:11 

way. So I really encourage you to 

00:07:12 

look at those programs. 

00:07:14 

Last of all, if you've got any questions, please feel free to reach out to the Social Ventures Australia team – myself, Patrick and Connor are all involved in the social impact bond, outcomes 

00:07:27 

based contracting space, and we'd love to hear from 

00:07:29 

you. Thank you.