1. Welcome and Introduction to BCFG
Angela Duckworth introduces the Behavior Change for Good initiative and its mission, and Alison Buttenheim introduces the panelists and session logistics.
welcome everyone it is just noon eastern time uh i'm angela duckworth and i'd like to open up today with allison i want to give you a sense of who behavior change for good is to begin then i'm goingto pass the mic to allison the behavior change for good initiative is actually a consortium of scientists world-class scientists from many different disciplines and what unites these academics is a desire to understand the science of behavior change andin particular when we think about the pandemic we recall even early maybe in the first weeks of the pandemic that it was clearly human behavior change that was an essential part of overcoming uh the virus so not justbiomedical breakthroughs i mean think about mask wearing as an example and then to take the example that we're going to talk about today vaccination once we had vaccines against covid 19 and once there was adequate supply at least
here in the united states we had a demand problem and anticipating this the scientists at behavior change for good wanted to ask the question and did ask the question how can we encourage adults in the united states toget vaccinated in anticipation of the covid vaccine this is now going back one year so last fall we got together and we did a mega study a massive trial where multiple scientific teams tried multiple and different ideas toencourage vaccination in that case one year ago against the flu so that we could understand messaging that would work to encourage vaccination more broadly and so uh that is what we're going to be discussing today you know everythingthat uh behavioral scientists have learned about this critical topic and with that let me pass the mike to uh allison buttenheim a good friend and of course a member here as well thanks angela hi everyone i'm alison buttonheim
i'm on faculty at the school of nursing at penn also scientific director at the center for health incentives and behavioral economics aka chive tribes of leading research center nationally applying behavioral science and behavioral insights to health behavior changeof all types many clinical domains on both the individual side and the provider side in clinical encounters and in populations and in the community we love partnering with bcfg and you'll hear about some of those collaborations today andwe'll also wrap up the session with kevin volp our founding director for some for some synthesis and and next steps one of the reasons i'm really excited about today is um you know for over a year policymakers ofall types from local to national have been asking us for evidence on what works to support this vaccine rollout and at the beginning you know a year ago we had a lot of punches and hypotheses from previous vaccine
rollouts from previous public health emergencies but there really wasn't a playbook for this vaccine and this rollout and it's been terrific in partnership with ecfg to help generate some of that evidence and to do it with rigor butto do it rapidly and importantly to have it be policy responsive to answer questions that policymakers had you know almost in real time about how to support vaccine demand a couple housekeeping details and then i'll turn it overto our panelists um please as the panelists speak use um our q a function to throw questions in that you want discussed during um during our q a portion and feel free to upvote other questions that you'd alsolike answered um but we also really encourage people to use the chat function as well to comment to have conversations with each other um some robust back and forth um in the chat would really would really be welcome
um so i'm just gonna highlight very briefly sort of a sentence each um who you're gonna hear from and we'll also drop links um to our speaker bios in the chat as we go we're gonna first hear fromkatie milkman who um along with angela directs behavior change for good and is also a wharton professor um after that lynya gandhi who's a doctoral student at wharton um will speak and finally hersha tiramurti who is an associateprofessor in our medical ethics and health policy department at the school of medicine um at penn and is also one of our associate directors at tribe where he directs the global health portfolio um you'll be hearing from harshaas well then we'll do a q a um and as i mentioned kevin vault will wrap up um so with that let me pass the mic to katie movement thank you alison and thank you to allison and angela
2. Mega Studies on Vaccine Text Messaging
Katie Milkman presents two mega studies with Penn Medicine, Geisinger, and Walmart testing text message strategies, finding that 'reserved for you' ownership language was the top performer for boosting flu vaccination.
for setting the stage so nicely for what we're going to share today um i'm going to share some slides and excited to start telling you about some data we collected here we go all right so i'm gonna tellyou about two mega studies that angela previews previewed that we ran in the fall of 2020 with the goal of generating insights based on what messaging worked to encourage flu vaccination that would be portable to immediately deploy anduse to encourage covid19 vaccination we wanted to have insights that were really ready to go and um just to preview what may be on many minds as you're thinking well gosh the flu is really different than kovan 19i want to highlight that some of our amazing team scientists led by heng chen dai of ucla and sylvia secardo of carnegie mellon took the best performing insights we we ended up with that i'll tell you about today
and then tested those as tools to encourage covet 19 vaccination and indeed found that they were effective so that was really exciting work that um that this queued up okay so we ran two mega studies in the fallto try to understand what would be the best messages to use if we wanted to promote vaccination and again we're looking at the flu we wanted to look at actual behavior because we know from past research that justasking people will this change your decision is not necessarily actually correlated with what people really follow through on um a mega study is a massive randomized controlled trial that instead of testing a single idea tests dozens of ideasum or or tens of ideas or many more than one and that way we can see in a head-to-head comparison what is the best performer and understand cost effectiveness and so on um we felt that this was the
perfect moment to use this tool to gain insights about what would encourage vaccination so the first mega study we ran was with two local health systems pen medicine and geysinger health both um large health systems in the northeastwith millions of patients and in these um cases what we were interested in was could we send messages to people who had a healthy doctor's visit coming up they were going to see their doctor in person that mightchange decisions those patients would make when offered a flu vaccine at their doctor's appointment so um the design of this study was to test 19 different text messages sent in the three days leading up to that in-person appointmentencouraging people to accept a vaccine when it was offered to them by their doctor and we had a giant team of collaborators economists psychologists computer scientists doctors who designed these messages with that goal of encouraging flu vaccination and
with the goal of designing messages that could then be redeployed to encourage covid19 and they tested all sorts of ideas i won't list all of them but they range from encouraging people to dedicate their flu shot to aloved one we dedicate our marathon runs and and many other activities to loved ones maybe dedicating a flu shot to a loved one would increase your uh altruism and your motivation um we tried telling people a joke aboutthe flu have you heard the one about the flu don't spread it around that was one of my favorites and the idea was that it would sort of give you an ear bug you'd be thinking about it you'dhave a positive attitude maybe you'd even tell other people to joke and you wouldn't forget and you'd follow through um some messages encouraged you to get a flu shot to protect other people because we know sometimes um thinking
about others rather than your own health can be more motivating to change behavior and then one team tested um the idea that a flu shot has been reserved for you and i actually want to shout out to allisonbuttenheim who helped develop this idea with gretchen chapman um highlighting that essentially it's a default it's available it's set aside for you um there's some ownership language in there and what we found when we tested all of thosemessages head-to-head uh first thing i want to highlight is if we look at text messages as a whole and this is just a graph showing on the x-axis the regression estimated increase in vaccination against a baseline which issort of business as usual we don't message you specifically via text about the flu vaccine that we're messaging you about your appointment that led to a 42 vaccination rate roughly and we get that vaccination rate all the way
up to a 46.4 rate with our very best performing message but if we pull all of the text messages just sending text message reminders did effectively increase vaccination um and all of the dots in red were messages thatsignificantly outperformed control and i'm sure many of you thinking about multiple comparisons issues and when we adjust for that we can still say um we see robust results where you know many more treatments are working than you'd expectby chance um the best performer significantly outperforms the control and that was telling people that a flu vaccine has been reserved for you and by the way the flu shot joke was not apparently as successful as i thoughtit would be we're actually quite bad at predicting what would work um i just want to highlight to you what this reserve for you messaging looked like that was our top performer which therefore leads us to wanna and
suggest that as a tool we could use in the future a really very basic language that looks like what you'd expect to get from a clinician telling you you have an appointment coming up there's an available flu vaccinein the first message and then 24 hours beforehand a much shorter message saying you know please make sure that you ask your doctor to receive this flu vaccine that's been reserved for you and i also want to notethat um of the 19 messages tested the three top performers all used reserve for you languages and none of the other messages did so that was a very strong signal of the effectiveness of this so big takeaway thisreserve for you language seemed incredibly promising based on this study and when we did an attribute analysis to try to understand what were the things that really rose to the top and this was led by angela and linea
who are both on this call as well they found that framing things in this in this term these reserve for you terms was important but it was also important that the message be congruent with the kind of thingyou expect to get from your provider so when there was interactivity when jokes are told when we're sending you pictures people did not seem to respond very well to that and the more sort of simple messages that you'reused to getting from a health provider were more effective okay the second mega study we ran was with walmart pharmacies and you might think that encouraging vaccination from a pharmacy would be a pretty different decision than accepting avaccine when you're already seeing your healthcare provider and they offer one and so we thought another study in that setting was warranted we teamed up with walmart pharmacies and 700 000 of their customers were messaged in this trial
here we developed and tested 22 different text messaging strategies thanks to the ideas submitted by team scientists and a sample of what we tested these were different messages than when we tested in the other setting i just describedbecause it's a different situation you have to get in your car drive to the pharmacy asked for a vaccine we sent these messages on september 25th of 2020 and then for three days afterwards there might have been follow-upmessages and we tested some of the same psychology but some different things too um get a shot to protect your family and friends getting people to make a commitment and text back and say yes i plan to geta shot we were actually not permitted to use the reserve for you language there was an anxiety that if we said that and then customers came and there was a stock out it could be um upsetting and so
instead we went with language a shot is waiting for you to convey that to convey that sort of ownership sentence and a couple of the different treatment arms and here what we found first of all we have hugestatistical power because of the 700 000 person sample vaccination rates at baseline were about 30 percent of the population every text message outperformed that significantly so again really strong evidence that just texting people reminders to get a flushot works it moves the needle pun intended but there was significant variability in what and how well different messages performed and that's important too so we can optimize the messaging once we know we have a useful reminder tooland the very best messages again use this ownership language a flu shot is waiting for you at walmart outperformed the rest of the messages that we tested though many other things again worked that that rose to the top
and here's what that looked like again really simple messages um it's flu season you can get a flu shot we'll send you another text and then remember a flu shot is waiting for you at walmart three days laterkey takeaway here uh these these top performers are so consistent that it's really uh encouraging i think to suggest that this ownership language is important and then here key things that rose to the top when we did anattribute analysis were multiple messages outperformed one so that nagging really seemed to work here and also that this this language of ownership was important so as i mentioned at the beginning we feel so lucky that uh science workthe way we always want science to work which is that these findings get out there last december we had an inkling of what was working um a couple of members of our team sort of took the baton ran
with it and did an amazing study a randomized control trial last january and february with ucla medical system and proved that the same kind of ownership messaging was effective in increasing vaccination against covet 19.i also want to mention that there were some follow-up work done in rhode island in um may and june suggesting that in a population where there were far far fewer that actually were taking up vaccines it wasn't enoughso um whereas 20 to 30 percent of the people if i'm remembering the numbers correctly in this trial at ucla were getting vaccinated when they got messages in rhode island something like two percent were and then that youknow when you're getting such low rates it didn't matter if you use ownership language or not so you have to be in a setting where there's some potential to change behavior for this to work but it seems like
we should be trying these ownership language messages in any population where there's hope of moving the needle and booster shots are a great candidate so let me stop there and turn this over to linea who's going to talk
3. Philly Vax Sweepstakes Lottery Experiment
Linnea Gandhi describes the Philadelphia vaccine sweepstakes, a regret lottery with priority zip codes given higher winning odds, and reports a precise null result on vaccination rates, highlighting the role of awareness.
about the philly vaxx sweepstakes great thank you katie thank you so much um let me just share my screen here
all right um so i love the work that katie just presented all around messaging for boosting vaccination we're gonna switch gears now and talk about using money to help boost vaccination rates um and specifically i'm going to sharewith you the results of a study run this past summer called the philadelphia vac sweepstakes and there were many different collaborators and organizations helping out many of the individuals you see on this slide are actually on this calltoday and i felt very fortunate to be part of the team with them the main experiment that i'm going to speak about was one where we tried to see if raising the odds of winning in this lottery wouldin turn raise the motivation of individuals to go out and get a covid19 vaccine now the philly vac sweepstakes was one of many different vaccine lotteries that you may have been aware of over this past summer what i
want to call out at the beginning here are two key design features that are different about the phillyvac sweepstakes from any of the other vaccine lotteries at least to my knowledge that have been running at the state levelthe first one to make note of is that what we ran was a regret lottery in all of the other vaccine lotteries that have been running they're really traditional lotteries meaning for your name to be drawn to wina prize in this lottery you had to have already gotten the covet 19 vaccine in a regret lottery which is what we ran all you had to be was a resident of philadelphia so whether or not you gota coven 19 vaccine your name could be drawn from the pool of people and we operationalize this by having a residential database um as well as the potential for people to manually register as a safeguard now how does
this actually work then well when your name gets drawn to win a potential prize in this lottery we find out whether or not you actually got vaccinated and if you didn't you don't get to claim your prize anda new name is drawn this is the beauty of the regret lotteries this feeling of getting so close to winning but then not getting that prize and the hope is that with the regret lottery design you're then ableto give additional boost additional motivation to get your vaccination because you don't want to be in that position so that's the first design feature to really highlight about this lottery in contrast to the other lotteries you may havebeen familiar with the other one and this was the focus of the research was our ability to embed an experiment inside the lottery where some individuals had higher odds of winning a prize than others and we wanted to
see if those higher odds actually boosted motivation to go out and get your copic 19 vaccine now we had four hundred thousand dollars from a very generous donor to allocate across three drawings and in each of those drawingswe gave away two fifty thousand dollar prizes four five thousand dollar prizes and six one thousand dollar prizes had we run this like any other lottery we would have just drawn names from anywhere in philadelphia and everybody wouldhave had the same odds of winning a prize instead we split those prize levels down the middle such that half of the prizes at each level at each category went to individuals whose names were drawn from one specificzip code in philadelphia and the other half came from names drawn from anywhere else in philadelphia which meant if you're a resident of this priority zip code you have much higher odds of winning a prize and that's how
we operationalize the differences in odds now a really salient question is how do you pick what zip code gets that priority this actually gave us an opportunity to focus the benefits of this lottery towards the most underserved vulnerablepopulations in philadelphia at the time which meant for us drawing the 20 zip codes with the lowest vaccination rates per capita in late may that became our pool of zip codes who were eligible to become priority in anyone of those three drawings and in each drawing we randomly chose one zip code from that pool of 20 to be priority so i'll just show you what that looked like on june 7th we announced the whole sweepstakesand we also drew our first priority lottery which happened to be 19126 so if you are a resident in 19126 you knew at the subsequent drawing on june 21st that you would have higher odds of winning which meant
if you haven't yet gotten your covet 19 vaccine you're gonna use the next two weeks to go out and hopefully get it so that if your name is drawn you'll be able to claim your prize on june 21stwe announced the winners of the first lottery and we did the whole thing over again where we drew another zip code from that pool of 20 this time it was 19133 and they had priority odds so if youlived in that zip code and you hadn't yet gotten vaccinated you would hopefully be more motivated to go out and get it over the next two weeks so you could take advantage of your higher odds on july 6thand then we've repeated the third time for lottery three how much though did these odds go up i'm going to show you here on this bar chart the absolute odds of winning a prize in a drawing for the
priority zip which is in blue and all other zips which is in red now this is actually pretty meaningful if you think about relative odds in drawing number one the priority zip code had almost a hundred times theodds of winning a prize compared to everybody else in philadelphia in drawing two and drawing three it was a little lower around 60 times the odds this is kind of like taking someone and saying instead of having aone in two hundred thousand chance of winning you actually have a one in two thousand or three thousand chance of winning any one of those prizes in the drawing so we tried to get the word out about thisbecause it's not worth it if people don't understand and know their odds through a variety of marketing and social media campaigns so i'll just show you a couple here this is an example from a press release that we
put out during the launch of the sweepstakes when notably we had the most national and local attention we also put out different text messages this is an example from an emergency text message system uh called ready philadelphia andour partner penn medicine put out text messages to their patients in the philadelphia area all trying to highlight hey if you're in one of these priority zip codes you've got some higher odds of winning go get your vaccinenow the question then is did it work so to evaluate if it worked uh we did a difference in different analysis comparing our three priority zip codes which i'm showing you here in dark green so priority zip codeone two and three we compared these three zip codes to our 17 controls these are the other zip codes from that pool of 20 who didn't get drawn but were eligible to become priority and we're going to look
at the first dose vaccination rates by week comparing the priority zip codes to these control zip codes and see if there's a difference in the difference over time so i'll show you the results with one zip code ata time so here i'm going to show you data across all the weeks of this summer and we're going to focus on what happens during that key two week period of time when the first zip code that waspriority knew that if they didn't get their vaccine by now they wouldn't be able to take advantage of higher odds on june 21st so we should presume that we could see a boost in first dose vaccination between thosetwo lines this is our data we are going to focus on what is happening with that blue dot which represents our priority zip code compared to our red dot the 17 control zips and if you focus between the
two purple lines you actually can see it looks like there is a pretty good lift there that there's a boost in the first dose vaccination rates for our priority zip compared to our 17 control zips our ols regressiondoes estimate a positive coefficient but unfortunately it's just not quite statistically significant so while this data is quite optimistic we can't conclusively say that for sure these higher odds boosted vaccination rates how about zip codes two and threei'll show you zip code two on the left and three on the right remember these had slightly lower odds of 60 times the rest of philadelphia we are now going to focus on the purple lines that represent thetwo-week period of time for each of these zip codes when they had priority odds for our second zip code when you eyeball the graph it's hard to see that there's any difference and the model reflects that uh looking
like a null result that there wasn't a difference in the difference in terms of vaccination rates and for our third zip code if you look between those two purple lines it indeed looks a bit like the blue dotwent a little bit down which is echoed by our ols regression uh having a negative coefficient but like these other ones it's not statistically significant so we can't confidently say either way when we pull the data across zipcodes one two and three we're actually able to estimate a pretty precise null result meaning raising these odds giving these zip codes higher odds of winning in their respective drawings did not seem to impact first dose vaccination rateseither way now stepping back instead of just thinking of this only as a null result you could also go back to the first zip code and say well we think we see a boost there there might be an
overall effect if you were optimistic in reading into it you might say well let's say i rolled this out what is the upper bound that i could expect looking at our confidence interval here and we can estimate that'sabout a nine percent boost so if you're optimistic and looking at the upper bound of our confidence interval we can say maybe in high odds vaccine lottery can boost vaccination rates by nine percent in our data we canalso suggest whether or not that's worth it so it translates to about 53 incremental first dose vaccinations at a cost of about 3 500 each so again this is our upper bound of what we might be able toexpect given the data we have and one of the things we've been thinking about as a team is that given the cost involved it's probably not worth it at this time of the pandemic there are probably more cost
effective tools available to policy makers so thinking about takeaways that is the first and main one is that maybe high odds vaccine lotteries could have worked earlier on when vaccines were first made available when maybe it was moreabout ease and speed to get vaccinated but at this point in the pandemic they don't seem to be as effective the second takeaway that we've really been chewing on is how we can disentangle odds from awareness we canmake odds higher all day long but if nobody is aware of them then it's hard to say that they're going to work either way and we we don't have good empirics for this but we do have a niceproxy which is registrations um on that manual registration website that i mentioned now this wasn't required for people to do but it is a good proxy for awareness we might think that if you're in a priority zip code
you would sign up for this you would register more than if you were in a control zip code and we can use that to see how aware these different zip codes were so just quickly as we wrap uphere i'll show you that data which we've just been pulling together i'm showing you here is the percent of people in a zip code who manually registered and what i've highlighted darker here is the two week period oftime for lottery one uh when the blue the priority zip code was priority and you can see greater percentage of people sign up this also is echoed for lottery 2 and lottery 3 of the blue bar being higherthan the red bar indicating that people in a priority zip code are more likely to manually register during the two-week period when they are priority um but the overall takeaway from this slide we think is to look at
that first lottery if you recall from the data this is where we saw the most optimistic results we also had the greatest local and national media coverage and we see the greatest uptake in registration so it might suggestthat high odds vaccine lotteries could work if we could really mobilize a lot of attention across channels to make sure people were aware of it so um the last thing i want to suggest or share before i moveon to the next speaker is that i think the most exciting part of this research is that we were able to pull off an experiment in partnership with the city of philadelphia in a policy rollout um we hopethat this gives uh inspiration to other policymakers and partners out there that it can be done and it's a great way to be able to evaluate high stakes policies while helping underserved communities and with that i'll thank my
collaborators and turn it over to harsha
4. Statewide Incentive Programs National Analysis
Harsha Thirumurti presents an observational difference-in-differences analysis of statewide vaccine incentive programs, finding no overall effect on vaccination rates and discussing subgroup findings and possible explanations.
thanks so much uh and hi everyone i hope you can see my slides here so um having seen the results from the randomized control trial that katie presented and then philly's backsweep six i'm going to zoom out tothe national level and present to you an observational data analysis that my colleagues and i have done looking at the effect of these statewide incentive programs on in promoting uh kobit 19 vaccinations so this is work with severalco-authors actually many of them are here in this in this event um and so i want to acknowledge their contributions here as well as the research assistants who supported this project so we're all aware that um many statesum in fact around in the country have been making an effort to increase covet 19 vaccinations um and and this as angela alluded to this at the top of the the hour um is really around mid april when
uh there was a recognition that um daily vaccination rates were declining even though vaccination coverage overall had not yet reached the target levels and so around this time um as as vaccination rates started to decline and in factthey that kept happening throughout most of the summer we had a number of states that experimented um so to speak with incentive programs that were motive that they hoped would motivate people to seek uh kobit 19 vaccinations nowthe highest profile example of this statewide incentive program is the one that was introduced in ohio many of you have probably heard about it it got a lot of press it was the first vaccine lottery implemented at thisat the entire state level and what ohio did in this case is offer um uh people who did get vaccinated a chance to win grand prizes of up to a million dollars um and also for for um adolescents
um and younger people it did offer the chance to win college scholarships got a lot of press um not just in terms of national coverage but also within the state it was closely watched in terms of the thedrawings that were held every week in which a winner of a million dollars was announced um what was interesting to us as we read about ohio's experience is that many different states um in the weeks and months thatfollowed introduced very similar programs so louisiana had a similar vaccine lottery and so did the state of michigan and indeed on and on a number of states did very similar things and had these statewide incentive programs that reallyat the end of the day were lotteries uh that enabled people who did get vaccinated to have a chance to win um these big prizes and also interestingly there were some other states in the country that didn't go
the route of offering lotteries which many economists and behavioral scientists have have hypothesized can be really motivating to people uh in terms of adopting a certain behavior but they these other states uh opted to offer uh what wecall guaranteed rewards which is that every individual who got vaccinated is guaranteed to to um be able to get some type of reward um in uh some states like the state of new jersey for a short while therewas a reward in the form of a free uh free beverage at participating uh restaurants so that's an example of a very small guaranteed reward but there were other states where the guaranteed rewards were slightly larger in sizeso in the state of west virginia there were 100 savings bonds four people in certain age groups um and then even in the state of california um there were 50 gift cards for newly vaccinated individuals so at the
end of the day what we're interested in is understanding um if we take stock of all the data from the states in the u.s whether these statewide incentive programs were effective in increasing vaccination rates and there's actually beena number of other studies that have looked at this question um the state that's been most carefully studied is ohio probably because it's the first to introduce these types of programs and the evidence overall has been mixed there'sbeen some studies showing that in ohio the incentive program was effective and there are others um that found that there was no effect at all and um in contrast to the statewide incentive programs and those evaluations there havebeen a few randomized controlled trials from different parts of the world also looking at this question about paying people to get vaccinated and here i put up two studies two randomized trials of guaranteed incentive programs now we wanted
to zoom out as i said to the national level and see if we could come come with some broad takeaways in terms of whether these intensive programs were effective or not and what we did is collect the dataover the summer um assemble essentially all the information about statewide incentive programs that were introduced in the u.s looking at data that was made available by the national governors association doing google news searches and then we linked thatup with the daily vaccine vaccinations um uh taking place in each state in the u.s um from the cdc so we focus on this period during uh between april uh first and july 29th when there was the sustaineddecline in vaccination rates and what we're doing is essentially running a difference in differences analysis comparing the states that did introduce vaccination incentives to states that didn't have those incentive programs we're adjusting for the trends that are taking
place in the country at this time as you'll see in one of the figures that i'll show you vaccination rates were declining throughout in all of the states at this time and so it's really important to make surethat we're adjusting for what's happening in in the background here um and we're also going to look at some of the short-term responses to these statewide incentive programs so just to give you a a a snapshot of whatthese data look like there were 24 states um interestingly that introduced statewide incentive programs um during this period um starting in in late april going all the way through the um through mid-july there were a number of statesthat introduced incentives um and given that these incentive programs came relatively late in the rollout of vaccines it's notable that the median population that was percent the median percentage of the population that was vaccinated here was 44 so
in many states when incentive programs are being introduced about 40 percent of people have already been vaccinated um what i said about the trends is important here just if you look at this figure what we're showing is umthe uh the vaccine doses administered every day um per every day in states that did introduce incentives those are shown in the blue dots and states that didn't introduce incentives and you can see that overall there's this declinethat's persistent in this period in both of those states both of those types of states and so what the analysis is going to do is really adjust for the fact that these these trends are declining and it's goingto ask the question are the vaccination incentive programs able to either slow down or reverse the decline um once the programs get introduced so here's the main results from the the paper um and i should say this is
a preprint right now that's under review um you can share a link to it in the chat but this figure essentially shows you the trend in vaccination rates the difference in the trends between incentive program states and thecomparison states that didn't have incentive programs and it shows you these patterns in the 14 days prior to the start of an incentive program and the 14 days after now if these incentive programs were effective in increasing vaccinationrates what you would start to see is on the right side of the vertical bar you'd start to see a positive gap emerge between the two the two states and clearly the the dots here you can see thatgenerally speaking they hover around zero and um the the confidence intervals uh include the the null effect of there being no no change and no difference in vaccination trends between the two types of states it's also notable um
when we do these types of analysis it's important to compare what was happening prior to the vaccination incentive period and it's in fact here reassuring that the states look very similar in their trends prior to those programs beingintroduced so this is one of the main kind of visual displays of the results that shows you that by and large we're able to conclude from this study that there is no association between the incentive programs being introducedand vaccination rates we can also and this is in the paper uh we can also zoom out and look at the entire period when incentive programs were affected were in effect not just those first 14 days and whatwe're seeing here is that um the there's no significant association and if anything the association here is slightly negative but not statistically significant so there's a decline of nine daily vaccinations per 100 000 individuals which equates to about
a two percent decline not statistically significant um uh once vaccination incentive programs are introduced it's also important to think about what kind of statistical power we have here to to to to consider what conclusions we can then drawabout the effectiveness of these programs and what we're able to rule out with with a lot of confidence in this analysis is that these incentive programs we can say um with certainty that they did not increase vaccination incentiverates by 10 or more and that's notable when you compare the results to some some other uh trials of guaranteed incentive programs that have been out there we did a few additional subgroup analysis and i'll mention those beforei close one one question we have going into this is whether the guaranteed incentive programs were uh different in their effectiveness than the lottery incentive programs in our data set what we observe in general is that lottery incentive
programs are much more common there's only a few states that have guaranteed rewards that they provided but nonetheless we don't see a difference between the the effectiveness of the two types of programs we see some evidence that whenprograms are introduced relatively later which is in late may or june or july those programs tend to be less effective than programs that were introduced earlier i think that's important to keep in mind when we compare studies thatwere that evaluate the experience of ohio relative to some other states perhaps and then another interesting finding here is that the incentives do appear to be more effective in republican-leaning states where they actually increase vaccination rates by 10or more so this increase of 57 doses administered per 100 000 individuals every day equates to a 10 increase relative to their underlying vaccination rates whereas in democratic leaning states um there's actually uh a negative and significant um
decline so one concern that many people have had and one reason why incentives sometimes are controversial is that people often worry sometimes that they may be counterproductive in these cases and our main result doesn't show that they hadany effect either in favor or against um that increasing vaccination rates but certainly these results the subgroup analyses suggest that there are some cases in which they're what there is a negative association um so finally why were incentiveprograms not effective now this is not a question we can definitively answer but we have some hypotheses uh one is that the probabilities of winning these lottery prizes may have been too small as linnea mentioned how you designthese rewards and what kinds of probabilities you offer can can make a difference but at the statewide level many cases the probabilities were extremely small and also the guaranteed rewards that were offered uh tended to be fairly small
as well so ranging from five to fifty dollars um the other factor of course is that we're we're dealing with a population that um is uh is more resistant or more hesitant to get vaccinated and so that makesthe need for larger incentives even greater potentially so that could be one other reason and so and finally the fact that these incentive programs are being introduced at the same time that people are concurrently being supposed to misinformationi think also makes it much harder for the program to succeed so in conclusion you know we can say from our analyses that there is no association between incentive programs and vaccination rates um we do think that largerrewards may be necessary for increasing vaccination rates it's notable that the biden administrations call for rewards of 100 or more to be offered it's not a question that has been tested in any of the studies yet but there's
a hypothesis that they may be more likely to succeed and then finally if we set aside incentives alternative approaches like mandates are likely to be more effective so i'll stop there and open this up for discussion thank you
5. Panel Q&A on Subgroups and Incentives
The panelists answer audience questions about subgroup effects, lottery design trade-offs, the state of evidence on incentives, and implications for vaccinating children ages 5-11.
great thanks harsha and uh gonna bring back our panelists there are there's about three hours worth of really interesting questions and topics for debate going on i wish we had the three hours to talk about this together butencourage everyone to to read through the questions and comments katie i'm gonna start with a question to you about um sort of subgroup effects but also timing do you know is there anything we can conclude from the megastudies about whether these messages work better for some groups versus others and work better at different moments at least in the flu shot uh season yeah great questions um what i will say is i was surprised by howliteral little heterogeneity we saw in the results meaning we'd try you know slicing subgroups by age by all sorts of different um characteristics and we didn't see huge differences now i think one difference that you saw in your
own sub study allison was based on whether or not people had gotten a vaccine in a prior year which seemed to boost effect size maybe a little bit maybe those folks were had higher intentions and so closing thataction and tension gap was more feasible with a simple reminder but there wasn't much there when we looked at things like age and gender and one result we saw on race which was only present in the penn geysingerstudy walmart's data didn't have great racial data with um unfortunately simply smaller effects for minority populations than white populations and there's a host of reasons why that might be the case you know from trust to access but thatwas disappointing certainly um and then uh i guess the other thing that i would say is on timing and and uh i think and i might be wrong about this but my suspicion about when we look at the
trials that tried to take our insights to covet and saw good effects in january february not so great in may i think that the baseline vaccination rate changed so dramatically that having the statistical power to detect any effectessentially became impossible by the time those other trials were launched um so i think you know that there's certainly no evidence that these very light touch low-cost tools aren't wise to use in combination with other things but toexpect large returns from them in a late stage of rollout is probably unwise they might be really useful at this moment for the five to eleven five to twelve age group i have a five year old can't waitto get him vaccinated and um boosters so that i think is where we probably see the the best results great thanks um linear lots of questions around the the design of the lottery one interesting one i thought maybe
you could speak to is you know we did the targeting based on zip code and you know what different results might we have expected if that targeting had been on a different um either geographic basis like neighborhood whichis different from zip code or even like social identity or occupational identity yeah that's a great question uh i i would be pure speculation to suggest where those would lead us but i think yeah i think it's helpfulthough to think about the trade-offs in these designs and there were a lot of trade-offs we considered early on as we were thinking about how to design the lottery and one of those was we really want to beable to randomize but at what unit at what level um some of us you know we were talking about focusing just on people who weren't yet vaccinated and randomizing in some way within just the non-vaccinated community but with
that you potentially create a perverse incentive when boosters come along or these god forbid another pandemic where people are waiting for the incentive waiting to be special and so hold off on getting vaccinated so we knew we didn'twant to focus there as for other socioeconomic or demographic variables i think we just wanted to very much avoid appearing unfair like the whole point was to be as equitable as possible while still focusing on the underserved communitiesthat didn't have as much vaccination or access to vaccines earlier on in the rollout um so that's kind of how we arrived at that but i love the idea of neighborhood that would be great if we could operationalizeit because you could imagine how you could do more local marketing to really raise awareness so if there's a policy maker out there who's interested maybe think about neighborhoods because it might be easier to marry those odds with
awareness i think there's easier marketing when it's fair amount versus 1913.exactly um harsha um you know there's been lots of debate you know certainly on twitter where some of us you know spend some time in the in the bubble um about uh the accumulating evidence on lotteries and incentivesand some folks pretty quick to say if they didn't work um were other folks saying like are we really ready to say that there's some individual incentive program studies that are coming out there's some other synthesis work likethe study you presented how do we know when we're sort of ready to say these worked or these didn't um where where else is sort of the r d on on future research that's needed to have a morekind of precise and and maybe robust answer to that question yeah um allison i i think um you know the researcher and me says like we we really should be empirically studying this question i i really like the
trials that were done uh in addition to the philly vac sweepstakes i like the trials that were done in sweden the one that was done with uh medicaid managed care beneficiaries in california um because i feel like thoserapid studies can at least tell us in a specific context in a specific setting um do incentives work i think the conclusion where we're drawing from our paper is that there's going to be tremendous heterogeneity like it maynot work in some places but in other places it may be what's necessary it comes down to some of these subgroup analyses i think are informative because they tell us like where which are the populations and settings inwhich these incentives are more likely to be effective so i think in terms of um in terms of how how we should approach this going forward so let's say next year there's another phase of the vaccine rollout during
which we need to determine whether incentives or other interventions are being introduced i think the subgroup analyses from these various studies can be informative in suggesting whether some settings and population should be offered these types of rewards andother rapid trials where you can do where you can say take a small smaller population and rapidly assess whether there's movement in the vaccination rates or not because that could be a useful way as well that's a greatway to think about it um our clock is ticking down so i'm going to do one last lightning round with maybe just a 15 or 30 second response from all of you uh panelists and you can actually pickthe prompt to respond to either what have we learned here that we can apply to five to elevens as the vaccine rolls out there is there anything we've learned or um really interesting question in the q a around
you know why aren't altruism messages working here like isn't that the rational response that i should just get this vaccine because it's the right thing to do for society and will protect other people um that person feeling somesome tension between that and like trying a lottery that pays people so you can one no i think one of those is harder than the other to answer and you're welcome to pick um katie we'll start with youstart with someone else
okay i'll bite um i think given i was closer to the vaccine lottery i will comment on the application to five to eleven-year-old since wasn't really much altruism involved in what we were doing um i have no ideaif a five to eleven-year-old will respond to a lottery maybe their parents will for sure uh you you could potentially experiment with different incentives like access to meet the creators of paw patrol or something else that pulls wellwith five-year-olds today but but all jokes aside i mean i think katie made this point that some of these techniques messaging or lotteries or incentives are going to work at different stages we are at an early stage withthat population so maybe this is the time to try a lottery again
yeah i mean i think on the basis of just looking at the analysis that we did in our paper i would suggest not going forward that way because i don't see the evidence in the in the data yetto support that all right katie sorry for being slow um i mean as i just said in the chat i actually i think i'm less hesitant than harsha to say sorry to use the hesitant word since that hasdouble lots of other meetings here but i'm i'm less hesitant to say we need more research on incentives in the u.s at this point and more ready to conclude that okay they worked in sweden but it's a reallydifferent climate and we have i haven't seen a single bit of evidence for their efficacy in the united states at the levels that you know have been tested so we don't know what i don't want to rule out
things like a thousand dollars a month or even a thousand dollars one time but you know up to a couple hundred bucks which is what i guess i i have seen doesn't seem to do anything and it's expensiveso i think we should be looking at other strategies and um certainly the evidence on mandates is extremely strong right now it's going to be really interesting as the 5 to 11s roll out you know we already haveexisting mandates in every state for school entry and whether and how the covet vaccine is added to those mandates and both the political and the epidemiological implications of that are going to be really interesting um let me hand
6. Kevin Volpp's Synthesis and Closing
Kevin Volpp synthesizes the findings, emphasizing the value of mega study methodology, the limits of light-touch nudges amid polarization, and the growing role of mandates in public health.
over to kevin to wrap us up um this hour went very fast as we knew it would um kevin vault all right well thank you to our panelists and to our audience for a spirited dialogue i think it'simportant to pause here on the context where vaccination rates had started to plateau most of the u.s population weren't vaccinated that's where the interest in using incentives came in nobody was using incentives as part of early rollouts andi think it's important to recognize that in many states i think incentives were being used in the hope that they would be effective and more paternalistic approaches like mandates would not be necessary so i think what what thestudies have shown is that while incentives have been shown to increase healthy behaviors in many contexts that did not appear to be the case in the context of covid leaving this study in sweden aside and that was true
here both in terms of incentives with small expected value direct incentives lottery type incentives but all of them have pretty small expected values and increasing the odds of winning as lynnea described didn't seem to really make much differencewe do see evidence that low-cost text message nudges can increase vaccination rates particularly if they draw on notions of scarcity and ownership with reserve for you type language and i want to highlight the methodological innovation here in termsof the mega studies in terms of the speed of learning i think this is in itself quite important typically researchers test one or two ideas at a time against a control and in both of these studies roughly 20ideas were tested at the same time really accelerating the learning process and and creating a paradigm that could be used for future innovation efforts but i think it's also important here to acknowledge that the most effective interventions increased
rates by about four to five percentage points on a base of about 42 so it's about a 10 percentage point increase it's a low-cost way to get lots of additional people vaccinated but clearly that in itself wasn't goingto solve our challenges in the us in terms of vaccinate vaccine hesitancy so i think what we've realized from a lot of this work is that lighter touch nudges of the kind described here in themselves are unlikely toaddress vaccine hesitancy in the u.s given how polarized things have been come and for that reason we've needed to use much stronger approaches various conditional incentives in terms of requiring vaccination for access to restaurants in some cities sportingevents sometimes workplaces and that's why mandates in essence have become required in so many settings and as we move forward with this and other public health emergencies there's going to be a lot of ongoing challenges balancing individual choice
and the impact of those decisions on the well-being of others and the economy that leaders in both the private and public sector are going to have to grapple with and it's going to be an ongoing challenge to figureout how hard should we push people in context like vaccination where behaviors both affect your own individual risk and the risk to others so let me conclude by thanking all of our panelists and thank thank you to allof you for joining us today