A Re-Analysis of ACE's Leaflet Study

Now that I’ve been doing some research blogging, it’s time to continue the trend with something big. Maybe you’ve seen ACE’s leafleting study analysis and was like “woah, that’s long, no way I’m going to read all that”. Instead, my reaction was “that’s long, but is there any way I can make it longer?” So I did. Sixteen hours later, I emerge with this:


Animal Charity Evaluators (ACE) is a non-profit with the goal of using research to find ways to help more save nonhuman animals with the same amount of resources. Given that the pro-nonhuman movement is quite small, any increase in our “bang for our buck” is quite important, and I welcome this research and aim to contribue to it myself.

Vegan Outreach and The Humane League are two pro-nonhuman non-profits that aim to do direct advocacy on behalf of non-human animals. One thing they do is deliver leaflets to people aiming to convince people about the horrors of factory farming and convince them to eat less meat. ACE claims that this is one of the most cost-effective ways to help non-human animals, but wants to put this to the test.

Throughout the latter half of 2013, Joey Savoie and Xio Kikauka, in cooperation with Animal Charity Evaluators, implemented a study of the impact of two Vegan Outreach leaflets on diet change. In March 2014, ACE released their analysis in cooperation with Statisticians Without Borders.

I was personally somewhat skeptical of the analysis and though there was a decent amount of information still being left on the table. I wanted to take advantage of the publicly provided dataset to do my own independent analysis to see what I could personally learn.

I did my best to check for errors, but it’s quite possible errors in my analysis may still remain.


This blog post in one sentence: ACE’s claims are broadly correct, though perhaps overstated.

I do not feel like the study provides any positive evidence for vegetarian conversion as a whole, so ACE’s claim that “[w]e found support for claims that distributing leaflets from Vegan Outreach (or similar leaflets published by other groups) causes a small percentage of respondents to go vegetarian” is overstated. I also think there is more nuance affecting some of their conclusions than ACE’s analysis admits, though to ACE’s credit, they are pretty self-skeptical about their own results.

I think ACE should be appluaded for having a good degree of self-skepticism, a high degree of transparency, and doing great with going through pre-registration and other checks to make sure they wouldn’t be fooling themselves.


Overall, one’s attitude toward these results depends really on how skeptical and negative one wants to be:

Reasons for positivity

Reasons for negativity

Setting up the Dataset

To start, I went to the survey dataset, downloaded a copy as a csv, removed the first row, and imported the file into R. The provided survey questions and guide to the data were very useful in interpreting the dataset, and I thank ACE for making it so easy to do this re-analysis. It reflects really well on their value for transparency.

My R script for doing the re-analysis is here.

What should the dependent variable be?

Once I set up the dataset, I looked into how we should represent the dependent variable – the amount of change in diet caused by the survey.

Meat Elimination

Question 1 asked about the amount of foods people eat now and Question 3 asks about the amount of foods people ate three months ago. I summed up the total ranks across red meat, poultry, and fish/seafood for both present and three months ago. I then compared these totals and looked for people who had no meat consumption at present but did consume meat three months ago (meat eliminator), people who consumed meat both in present and three months ago (no change), and people who consumed meat at present but did not consume meat three months ago (meat adders).

Meat Eliminator No Change Meat Adder N
4 (0.8%) 532 (99%) 1 (0.2%) 537

Note that while 0.8% new vegetarians might look low, we still have not yet compared that population to the leaflet-receiving population, and thus still cannot make any conclusions about how many people given they receive a leaflet, go vegetarian – that percentage could still be much higher (or lower).

Meat Reduction

I’m not sure how much I trust the Food Frequency Questionnaires to provide accurate and precise measures of people’s diets, so I’m a bit uncomfortable trying to tease meat reduction out of the dataset (as is the ACE analysis). However, there is also a problem that outright elimination happens too infrequently (only 0.8% of the data set), so we might want a variable that is more easily changed.

Therefore, I compared the past and current consumption of meat and this time assigned categories for Reduced, Stayed the Same, and Increased, looking at relative changes rather than outright elimination. I do not think this is a variable ACE looked at.

Meat Reducer No Change Meat Increaser N
174 (32.4%) 243 (45.3%) 120 (22.3%) 537

Red Meat Elimination  

ACE spent a lot of their analysis looking at how consumption of individual foods changed as a way to measure meat reduction. I was particularly interested at people’s elimination of red meat, as this is a type of meat frequently eliminated first.

Red Meat Eliminator No Change Red Meat Adder N
13 (2.3%) 539 (96.4%) 7 (1.3%) 559

Poultry Elimination

As ACE found positive results for poultry as well, I looked at that too.

Poultry Eliminator No Change Poultry Adder N
8 (1.4%) 547 (98%) 3 (0.6%) 558

Some of the people even eliminated both. Six people were found to eliminate both poultry and red meat, with nine people eliminating just one (seven for just red meat, two for just poultry).

Change in Declared Vegetarianism and Veganism

In addition to food frequency questionnaires, the survey also asked people if they identified as vegetarian or vegan at present and if they identified as vegetarian or vegan three months ago.

Omnivore (3mo ago) Vegetarian (3mo ago) Vegan (3mo ago) Omnivore (present) Vegetarian (present) Vegan (present)
578 32 13 570 39 14
Vegan to Omnivore Vegetarian to Omnivore Vegan to Vegetarian No Change Vegetarian to Vegan Omnivore to Vegetarian Omnivore to Vegan
3 3 1 572 1 10 4

Change in Declared Pescetarianism and Intent to Reduce Meat

As we were looking at whether people were dropping red meat and poultry, I was also curious if this would match up with people personally identifying as pescatarians.

Reducer (3mo ago) Pescatarian (3mo ago) Reducer (present) Pescatarian (present)
35 28 44 33
Pescatarian to Omnivore Reducer to Omnivore Pescatarian to Reducer No Change Reducer to Pescatarian Omnivore to Reducer Omnivore to Pescatarian
12 10 1 527 2 19 15

Challenges to These Variables  

Why look at eight different dependent variables? One reason was that we were interested in both partial meat elimination and full meat elimination. But another reason is that it’s really hard to figure out who exactly is vegetarian…

First Problem: Vegetarians and Vegans Eat Meat  

Weird, I know. But it seems like people report themselves as “vegetarian” or “vegan” without knowing what it really means. The good news is that vegetarians and vegans do eat less meat. But the weird thing is that they still do eat meat. Therefore, we may have someone who claim they became vegan, but all that might mean in reality is that they merely reduced their meat consumption.

Mean “Ranks” in Meat Eating by Diet Status (Reverse coded so higher = more meat eating)
  Red Meat All Meat All Animal Products
Omnivore 2.97 / 7 8.2 / 21 14.30 / 35
Vegetarian 0.97 / 7 2.6 / 21 8.51 / 35
Vegan 0.43 / 7 1.7 / 21 3.69 / 35

Second Problem: Pescetarians Underreport Their Pescatarianism  

If you think people have a hard time with the “vegetarian” label, then you must think people have huge problems with the less popular “pescetarian” label. Here’s a table with whether people label themselves pescetarian and how they respond on the food frequency questionnaires about red meat and poultry (they never ate the two, they no longer eat the two as of the post-survey, or they still eat the two despite self-identifying as pescetarian).

  Not Pescetarian Pescetarian
Never Ate 28 7
No Longer Eat 5 1
Still Eat 525 15

As you can see, the pescetarian label is quite untrustworthy.

Third Problem: Food Frequency Questionnaires Seem Troublesome

Quick! How many servings of cheese did you eat three months ago?

Don’t know the answer?

…Can you ballpark it?


Me neither.

But yet this is the alternative method of figuring out whether people go vegetarian or reduce meat consumption – compare their actual diets from three months ago to now. These food frequency questionnaires are standard to the nutrition science industry, so they might be the be best we’ve got. But I’m skeptical of them.


That being said, it’s possible that these food frequency questionnaires are reliable in the cases of elimination as opposed to reduction, as zero is a pretty hard number to misremember.

What should the relevant treatment and control groups be?  

So we’re looking to five different dependent variables to track changes in diet. But we’re really interested in the impact of leafleting. The study was designed well – there was a control leaflet and two different treatment leaflets.

Leaflets Recognized by Follow-up Survey Respondents
EIYLM (treatment) CC (treatment) CBTC (control) Fictional Flyer No Leaflet
95 125 44 5 428

EIYLM = Even if You Like Meat, CC = Compassionate Choises, CBTC = Cruelty Behind the Cuteness. Note: The same respondent may be in multiple categories.

Problems with the Control Group

It seems like it might be easy to sort people into a treatment group and a control group. However, because people received multiple leaflets, a lot of people who received the control leaflet also received the treatment leaflet, which means that they’re in the treatment group.

Leaflets Recognized by Follow-up Survey Respondents
Treatment Leaflet Group Control Leaflet Group No Leaflet
178 17 428

This makes a control group that is much smaller than we need to use. The ACE analysis decides to use the No Leaflet group as a secondary control group. This seems sensible as a point of comparison, though it’s possible that there could be selection biases if the type of people who would take a leaflet are more predisposed to go vegetarian.

Problems with Leaflet Recall

The fictional flyer refers to participants erroneously identifying receiving a flyer that was never handed out – at least five people were duped by this, so there’s probably other errors with people remembering which leaflet they actually received.

Also, ACE mentions that more than sixty people also mention receiving a treatment leaflet that was not actually handed out at their location. This means some people may have sorted themselves into the wrong treatment or control group, messing with our results.

What changes reportedly occurred as a result of the leafleting?  

So now that we have eight potential dependent variables and three groups to look at, how did things actually turn out?

  Treatment Flyer Control Flyer No Flyer
Meat Elimination Elim: 1/158 (0.6%)
None: 156/158 (98.7%)
Add: 1/158 (0.6%)
Elim: 0/15 (0%)
None: 15/15 (100%)
Add: 0/15 (0%)
Elim: 3/364 (0.8%)
None: 361/364 (99.2%)
Add: 0/364 (0%)
Meat Reduction Dec: 62/158 (39.2%)
None: 67/158 (42.4%)
Inc: 29/158 (18.4%)
Dec: 3/15 (20%)
None: 8/15 (53.3%)
Inc: 4/15 (26.7%)
Dec: 109/364 (29.9%)
None: 168/364 (46.2%)
Inc: 87/364 (23.9%)
Red Meat Elimination Elim: 7/162 (4.3%)
None: 155/162 (95.7%)
Add: 0/162 (0%)
Elim: 0/16 (0%)
None: 16/16 (100%)
Add: 0/16 (0%)
Elim: 7/381 (1.8%)
None: 368/381 (96.6%)
Add: 6/381 (1.6%)
Poultry Elimination Elim: 5/165 (3%)
None: 160/165 (97%)
Add: 0/165 (0%)
Elim: 0/15 (0%)
None: 15/15 (100%)
Add: 0/15 (0%)
Elim: 3/378 (0.8%)
None: 372/378 (98.4%)
Add: 3/378 (0.8%)
Net Vegetarians and Vegans Added 3/178 (1.7%) 0/17 (0%) 5/428 (1.2%)
Net Pescetarians Added 4/178 (2.5%) 1/17 (5.9%) 0/428 (0%)
Net Meat Reducers Added 4/178 (2.5%) 0/17 (0%) 5/428 (1.2%)

Are these differences statistically significant?  

Just looking at the table, it appears that a Vegan Outreach flyer does produce more meat reduction, red meat elimination, and net vegetarians/vegans added, relative to the control and no flyer groups, though it also seems to produce less meat elimination than the control / no flyer groups.

But the differences across the groups could just be due to chance. To be sure, let’s compare across the groups with T-tests. We’ll compare the population of those who received the treatment flyer to the population receiving the control flyer and then the wider population of everyone sampled who did not receive the treatment flyer. I think treatment vs. non-treatment is by far the better case to look at, given the small size of the actual control group, though it does have possible opportuniteis for bias.

  Treatment vs. Control Treatment vs. Non-treatment
Meat Elimination t = 0.78, p = NS t = 0.00, p = NS
Overall Meat Reduction t = -1.44, p = NS t = -2.21, p = 0.03
Red Meat Elimination t = -2.69, p = 0.01 t = -2.48, p = 0.01
Poultry Elimination t = -2.26, p = 0.02 t = -2.05, p = 0.04
Net Vegetarians / Vegans Added t = -1.00, p = NS t = -0.30, p = NS
Net Pescatarians Added t = 0.59, p = NS t = -0.99, p = NS
Net Meat Reducers Added t = -1.15, p = NS t = -0.51, p = NS

Based on this, the statistically significant effect of treatment flyers is muddled, but existant. They do have statistically significant effects on meat reduction (for treatment vs. non-treatment), red meat elimination, and poultry elimination, but there is no corresponding statistically significant effect with declared pescatarianism or declared intent to reduce meat. And no other variables are statistically significant.

Furthermore, it’s also worth noting that ACE did not do any sort of corrections for doing multiple statistical tests. When we do a Bonferroni correction here, having done fourteen separate analyses, we reduce our threshold for statistical significance from p = 0.05 to p = 0.05/14, or 0.003, which no tests met. Even a less stringent test, such as starting from p = 0.1 before Bonferroni corrections is not met.

How much can these changes be attributed to the flyers?  

One way we can look a little bit closer is with variables that (a) self-report how much of the leaflet was actually read, (b) self-report how influential the leaflet was, and (c) self-report the reason for changing diets.

How much of each leaflet was read?

  Did not read Glanced Read some Read all
EIYLM (treat) 9 (9%) 26 (27%) 34 (36%) 26 (27%)
CC (treat) 12 (10%) 33 (27%) 44 (35%) 35 (28%)
CBTC (control) 6 (14%) 15 (35%) 16 (37%) 6 (14%)
Fictional flyer 0 (0%) 4 (80%) 1 (20%) 0 (0%)
No flyer 155 (82%) 17 (9%) 7 (4%) 9 (5%)

Interesting that reading patterns for the treatment leaflets are statistically identical (p = 0.11), but statistically distinct from the control flyer (p = 0.05).

…Also of problematic interest is how many people claimed never to receive a flyer, yet claimed to have read a flyer anyway.

How influential were the leaflets?

  Leaflet caused diet change Thinking differently about farming Thinking differently about pets Possible effect No effect
EIYLM (treat) 7 (7%) 18 (19%) 6 (6%) 26 (28%) 37 (39%)
CC (treat) 11 (9%) 28 (23%) 2 (2%) 29 (24%) 53 (43%)
CBTC (control) 2 (5%) 6 (1%) 4 (10%) 13 (31%) 17 (40%)
Fictional flyer 0 (0%) 1 (25%) 0 (0%) 2 (50%) 1 (25%)
No flyer 0 (0%) 8 (4%) 10 (5%) 39 (22%) 124 (69%)


  • 6 people who received the treatment flyer self-reported becoming a vegetarian or vegan. Of these…

  • 5 out of the 6 reported reading all of the leaflet.

  • 1 out of the 6 report having their diet changed by the leaflet.

  • 2 out of the 6 report the leaflet having zero impact on their diet.

  • 2 out of the 6 report changing their diet for animal cruelty or ethical reasons.


We can also use this to dig a little bit deeper at red meat elimination, one of the dependent variables credited by ACE as being affected by the leaflet.

  • 7 people who received the treatment flyer self-reported eliminating red meat. Of these…

  • 2 out of the 7 reported reading all of the leaflet.

  • 4 out of the 7 report having their diet changed by the leaflet.

  • 1 out of the 7 report the leaflet having zero impact on their diet.

  • 4 out of the 7 report changing their diet for animal cruelty or ethical reasons.


This is good as a check of our results above – we have found that some of the people who appear converted by the leaflet also agree that they were converted. Moreover, I was concerned that people switching to eliminate red meat would be doing so for health reasons alone, and this does not appear to be the case, nearest as we can tell.

Though take this all with a grain of salt – if you have middling confidence in people’s ability to self-report their diet, like I do, you probably should also have middling confidence in people’s ability to understand how much the leaflet affected them. I do think you can justify slightly higher confidence in these numbers, as it’s more salient and potentially easier to recall than precise numbers about your diet, but people are still generally bad at retrospecting on causal influences on their life.

What are preliminary lower and upper bounds for flyer conversion rates?

Lower Bound

178 treatment flyers were given out, and it appears quite plausible (as clear as can be determined by the survey, given that you trust self-reported data) that precisely one person was lead to become a (genuine) vegetarian specifically based on having received the treatment flyer. One could consider this as establishing a preliminary lower bound on conversion rates at 0.56%, though I would be careful not to generalize from solely one person.

Upper Bound  

It’s worth noting that the small sample size of this study means that it might have trouble detecting a statistically significant difference in diet change. This study had a sample size of 623, with 178 having received a flyer. The past Facebook study had a sample size of 104 and a similar leafleting study had a sample size of “about 500”.

Small changes in diet change can be drowned out by statistical noise. To figure out what sample size we can use, we can use an “a priori sample size calculator for the two group chi-squared test”. For this, we need a desired power level (chance of detecting the effect given that the effect exists), a probability level (our threshold p-value), and an estimate of the proportion of people who become vegetarians (or vegans, or reduce meat, etc.) in the control group and treatment group.

Any desired sample size is going to be a trade-off between wanting more statistical power and wanting to spend less money collecting the sample. An additional consideration is that the desired sample size is the number of people we need to take the follow-up survey, which might be a smaller proportion of the total number of people that see the ads, due to inefficiencies in our conversion process.

Here’s a table with some estimations with the classic probability level = 0.05 and desired power level = 0.8, and our treatment-control proportions found in this survey, here is how much larger our population would need to be to detect the effect with 80% chance, given our current study size and ratios between the treatment and control group:

Control % Treatment % Difference Population Needed
1% 8% +7% 0.8x larger
1% 7% +6% 1.01x larger
1% 6% +5% 1.3x larger
1% 5% +4% 1.7x larger
1% 4% +3% 2.5x larger
1% 3% +2% 4.4x larger
1% 2% +1% 12.4x larger
1% 1.5% +0.5% 39.4x larger

Based on this table and our sample size, it’s possible that the leaflets could be converting people at a rate of 7% (relative to a 1% baseline) and we’d have about an 80% chance of detecting it. So therefore, I feel confidant placing a preliminary upper bound at 6%. Based on this data, it’s highly unlikely that flyers convert at a rate higher than 6%.

Also of note is that if we think the difference between treatment and control is quite small (like 0.5%), we’d need a much larger study to detect it – 39.4x larger than our current study. Though if we had an even split between treatment and control, it could be detected 80% of the time with just 16,294 people, or 26x our current population.


What are the implications for ACE’s Analysis?

ACE concludes in their analysis that “[w]e found support for claims that distributing leaflets from Vegan Outreach (or similar leaflets published by other groups) causes a small percentage of respondents to go vegetarian or give up eating specific types of meat.” This claim is tepid, but appears somewhat substantiated by my analysis.

Statistical tests were able to connect the treatment flyers to the cessation of eating red meat and poultry, but these tests ceased to be statistically significant once Bonferroni corrections were made. Additionally, they did not connect with any statistically significant finding for declared pescatarianism or meat reduction.

Additionally, statistical tests were not able to reject the null hypothesis that leaflets do not cause wholesale shift to a vegetarian diet. Therefore, the part of ACE’s claim that leaflets cause “a small percentage of respondents to go vegetarian” may not be established at this time. However, less robust (but still plausibly reliable) were able to detect one person who responded as we would expect a vegetarian who had been changed by the leaflet to respond.

How bad is it that there was no connection between pescatarianism or meat reduction?  

It seems potentially problematic that while there was a reduction of red meat and poultry consumption, there was no corresponding change in people declaring themselves “pescatarian.” However, given how bad people appear to be with just the common “vegetarian” label (e.g., how self-identified “vegetarians” eating meat), I wouldn’t trust them one bit to understand the “pescatarian” label. And once we do the analysis and see that people who don’t eat red meat or poultry still don’t call themselves pescetarian), it’s clear this variable is untrustworthy.

How bad is it that the tests were not able to pass Bonferroni corrections?  

It’s true that a statistically robust conclusion should be able to pass Bonferroni corrections. However, given the sample size of this study compared to how large the sample size would have to be, it’s unlikely that we would be able to detect such robust effects even if they did exist, so I’m not particularly worried. I’d be inclined to accept that the effects on meat reduction, red meat elimination, and poultry elimination are genuine effects, but that no other real effects exist (e.g., other kinds of meat, or shift to vegetarianism).

What are the implications of this study for prior studies of veg outreach?

It’s curious that this study finds a much smaller amount of vegetarian change attributable to flyers than a past study on leafleting, which found a conversion rate of 2% of leaflet takers becoming pescatarian (CCC Flyer Study), which is very similar to our study.

The Need for Some Point of Comparison, If Not a Control Group

The CCC Flyer Study sampled everyone, but only permitted those who received a leaflet to take the survey. Of course, the language of the survey was strong, asking people to directly attribute their diet change to the leaflet. But I imagine that this didn’t work as intended, and had they sampled people who did not receive the leaflet, or sampled at a college that had no leaflets distributed, it’s possible they would have found a similar conversion rate. Therefore, it’s good that this study was done with ACE with a point of comparison, so we can see that the 2% figure does make sense in context. Including these valuable pseudo-control groups is an important step forward.

Recollection Errors  

Prior flyer research had attempted to navigate non-response bias (systematic biases in who fills out the survey) and social desirability bias (systematic increases in vegetarianism as people aim to please or look more virtuous). Much has been done recently to try to work against these two biases.

However, another important aspect of the ACE Study is drawing our attention to a third form of bias – recollection errors. ACE concludes that “[w]e do not know whether this pattern [of decreased meat consumption in our data] reflects real decreases in consumption or only systematic errors in recollection.”

This recollection error also undermines the food frequency questionnaires used for meat reduction statistics – if they can’t remember receiving a leaflet, how can we trust them to remember intricate details about the number of servings of cheese they ate three months ago? This seems difficult to deal with and suggests that new designs will be necessary for future studies.

Survey Design Errors  

Another problem was the design of the survey. According to Joey and Xio, who designed the study and helped hand out many of the questionnaires, people were struggling with taking the survey as it was too long and complicated. One large problem was that people were skimming and had trouble understanding that the two food frequency questionnaires were different items.

How important is reducing red meat and poultry, but not anything else?  

It’s certainly good that the leaflets are doing something, most notably getting about 4% of respondents to give up red meat and another 4% to give up poultry, with no corresponding increases in meat eating elsewhere. But how important is this?

In my essay “How Much Suffering is in the Standard American Diet”, I provide figures that show if we take a suffering-per-kilogram point of view to our averting animal products, giving up red meat has 13% of the impact a typical vegan (not vegetarian) conversion would have, giving up poultry has 33.4% of the impact of veganism, and giving up both has 46% of the impact.

This means that a leaflet that converts 2% of people to become pescatarian (giving up red meat and poultry) would have essentially the same impact as a leaflet that converted 0.92% of people to become vegetarian, at least prima facie. Given that these leaflets appear to have this impact on poultry elimination, that isn’t so bad!

Vegetarian Recidivism

One potential problem, though, is that it’s unclear whether people would stick with poultry elimination with the same fervor as one would stick to vegetarianism. It seems intuitively less likely to me, because there’s less of a culture and moral force surrounding it.

The 100 Yard Line

On the other hand, perhaps we could look to Brian Tomasik’s 100 Yard Line Model for vegetarian conversion, which is summarized with the observation that perhaps pushing people to eliminate poultry and/or red meat is just one stepping stone to getting them to become full vegetarian soon. It’s definitely a big first step!

Why might the leaflets only be reducing red meat and poultry, and not other food groups?  

Aside from the philosophical question about what the value of reducing red meat and poultry is, we might also be curious as to the empirical question – why are people only eliminating red meat and poultry, and not other food groups?

Unfortunately, we can only speculate, but it seems quite plausible that the reason is that the two treatment leaflets (“Compassionate Choices” and “Even if You Like Meat”) both focus significantly on the meat of pigs, chickens, and cows, and do not focus much at all on dairy, eggs, or fish.

What future research should be done?

I think we are getting very close to understanding the impact of leafleting, but I don’t think we’re quite there yet. We could use further research (as always).

I think the most important future research should focus on addressing the methodological issues here, trying to come up with better dependent variables or better ways to mitigate the drawbacks that have been discovered with these dependent variables. For example, I’d propose further research into how the public understands terms like “vegetarian” or “pescetarian” or more research into leaflet recall after a few months. Perhaps we could make use of more reliable panel studies, where we know we’d have access to the same participants after a few months.

Additionally, the next step to settle this question as much as it can be settled would be to do a large sample size study with a control group. Having a couple thousand people in each group should do the trick and place a worthwhile upper bound on what the size of the effects would be, even if we don’t yet know the true lower bound (because it might remain too small to detect).

Lastly, something still worthwhile would be to run this study again though with a functional control group (that receives control leaflets) or to run this study in different contexts (e.g., online ads).