Author: Marshall Schott
Being deeply knowledgeable on one subject narrows one’s focus and increases confidence, but it also blurs dissenting views until they are no longer visible, thereby transforming data collection into bias confirmation and morphing self-deception into self-assurance.
~ Michael Shermer ~
It’s a bit baffling to me when a person proclaims their evaluation of something was completely untarnished by preconceived notions, prior experiences, and expectations of a particular outcome. Indeed, science recognizes the ubiquitous nature of bias and has developed myriad methods to assist in its reduction. When someone claims to be unbiased, I admit to immediately assuming two things– they don’t understand bias very well and they likely fall on the high end of the bias spectrum. I could be wrong.
I’ve always been interested in the stuff that makes us tick, all that goes on under the surface to influence the way we think and behave. Perhaps inevitably, this interest, which motivated my career choice, has intersected with my hobby. Rather than brewing for brewing’s sake, I make beer with the intent to test shit out, to the point I now view my hobby as “homebrew experimentation.” The exBEERiments have been a lot of fun and have opened my eyes to a new way of thinking about brewing, but the results tell us little about the participants… or those reading and interpreting the findings.
Last week, I asked people to complete a brief survey with questions focused primarily on self-perceived ability levels. To be clear, the results of this survey are not super scientifically valid, but rather intended to shine a light on something I find absolutely fascinating about the human condition. I trust some will develop arguments against these findings, while others may use them to confirm their own beliefs, both of which can be explained by the very phenomenon this survey was designed to evaluate– cognitive bias.
| Survey Results & Discussion |
A total of 1,188 people completed the survey, a sample consisting of craft beer drinkers and homebrewers of varying experience levels.
Respondents were asked to rate their beer tasting abilities compared to others. Given the way averages work, it’s expected that somewhere around 68% of people are actually in the average range, with less being somewhat above or below average and even fewer being much better or worse than average. But this didn’t measure actual tasting ability, rather it asked people to rate how good of a beer taster they think they are when compared to others. Did things line up?
Not so much. In fact, a staggeringly disproportionate 56% of respondents believe their tasting ability is better than average, which is a logical and statistical improbability– how can a majority of people be better than average? To add to the absurdity, a mere 6.2% of participants reported their tasting ability to be some degree worse than average. One plausible explanation for these results, and what inspired the question in the first place, is the Dunning-Kruger Effect, a cognitive bias that occurs when “relatively unskilled persons suffer illusory superiority, mistakenly assessing their ability to be much higher than it really is.” It would seem, based on this data, that beer drinkers are just as susceptible to said bias as anyone else, with most thinking they are better beer tasters than others.
I asked stats guru Justin Angevaare from OnBrewing.com to analyze the data to see if there were any relationships between perceived abilities and experience level. When it came to length of time as a craft beer drinker, responses were fairly spread out, though the highest proportion of those rating their ability as being somewhat above average reported being a craft beer drinker for 10+ years; however, when looking at homebrewing experience, those who reported having 1-4 years under their belt seem to think a tad more highly of their abilities than more experienced homebrewers.
Respondents were then asked to rate their ability to detect certain commonly discussed off-flavors in beer. The majority reported their abilities to be in the average range for all off-flavors, with fairly large portions also endorsing not sensitive at all. While it’s possible those who believe they’re very sensitive actually are, I can’t help but wonder how many have ever really tasted the off-flavors they claim to be sensitive to.
The final component of the perceived abilities portion of the survey involved respondents indicating how they think they would have done on previously completed exBEERiments if they’d participated in the triangle tests. The results were rather intriguing.
To put things into perspective, here’s a table of actual results on the same xBmts:
| xBmt | Sample Size | Actual Correct | Percentage Correct | P-Value |
|---|---|---|---|---|
| Lager Ferm Temp | 26 | 12 | 46.2% | 0.083 |
| Mash Temp | 20 | 9 | 45.0% | 0.134 |
| Boil Length (Pils) | 18 | 6 | 33.3% | 0.588 |
| Water Chem: Minerals | 29 | 18 | 62.1% | 0.001 |
| Hot Side Aeration | 12 | 4 | 33.3% | 0.610 |
It’s interesting to me that respondents thought the variable I’ve most often heard referred to as “myth,” hot side aeration, as being the most difficult to distinguish, while fairly large numbers thought they would have been able to perform better than our participants on xBmts that failed to achieve significance. To me, this suggests one of two potential things– either people don’t really trust the xBmt findings for one reason or another, which is a-ok, or they think they are more capable tasters than the people we include on our panels. This, to me, is fascinating.
| CONCLUSION |
Us humans certainly are an interesting lot, eh? We’ve evolved the capacity to consider the past, make fairly rational decisions about the present, and predict future outcomes with a decent degree of accuracy. And yet still to be influenced by something deeper, an inexplicable and seemingly omnipresent force. I’m compelled to believe our vulnerability to cognitive biases and logical fallacies serves a self-preserving function, regardless of the context. Superficially, beer is just beer, but for those of us so invested, it’s much more than that. Beer and brewing are meaningful outlets into which many of us plug our ideas of self. Whether overtly acknowledged or not, we view a good palate as a status symbol of sorts, something we ultimately derive meaning from. It makes sense to me that we might hold our own abilities in unrealistically high regard while viewing others as being less able, whether it’s illusory or not, it makes us feel special, like we matter.
I suppose for me, the problem arises when this overly positive self-perspective is used, subconsciously of course, to deride others. In the time I’ve been performing the xBmts, many of which have called into question purportedly necessary methods used to make beer, I’ve come to one conclusion: our abilities are not as good as we think. To take that one step further, what I’m saying is, despite how good of a taster you believe yourself to be, chances are your performance on the xBmts would be no better (or worse) than the participants.
I expect to field some complaints that the survey didn’t specifically ask about perceived abilities relative to a specific population. That was intentional, the idea being to get a glimpse of what people think about themselves in general. Personally, I think the imaginary population to which a person chose to compare their abilities is nearly as interesting as the final results, if only there were a good way to parse out that data. Again, this survey wasn’t designed to be as scientifically valid as a grad school dissertation, but rather provide some potentially interesting insights. I also can’t resist occasionally brülosophizing.
Finally, given the somewhat presumptive tone of this article, I want to make clear that I wasn’t talking about you, I was talking about us. In no way do I think that, just because I study and write about this stuff, I’m somehow less vulnerable to engaging in such self-delusion. I know I’m not and I’ve the evidence to prove it. I still notice insecurities about my brewing, tasting abilities, and even writing skills manifesting in odd ways from time to time, it’s only human.
Thanks to everyone who took the time to complete the survey, I truly appreciate it! If you’re wondering about responses to the last question regarding preferences for different Brülosophy projects, the large majority of respondents view most of what we do very positively, though the xBmts are the most popular.
Please share your thoughts in the comments section below. And for more on the Dunning-Kruger Effect and many other forms of self-delusion, I highly recommend checking out the You Are Not So Smart blog and podcast!
New Brülosophy Merch Available Now!
Follow Brülosophy on:
FACEBOOK | TWITTER | INSTAGRAM
If you enjoy this stuff and feel compelled to support Brulosophy.com, please check out the Support page for details on how you can very easily do so. Thanks!
75 thoughts on “Under The Surface: Results from the Self-Perceived Tasting Abilities Survey”
You guys are awesome! That is all.
You’re awesome!
Being a bit of a stats Hitler here, but I should point out it is entirely possible for the majority of a population to be above average in some parameter, but only if the distribution of that parameter is such that the mean and median do not lie at about the same point. I suggest replacing ‘average’ with ‘median’ in this write up in order for it to be accurate, as most would consider the ‘average’ to be the mean.
Cool stuff, though.
I see what you’re saying, but it would be wholly inaccurate to replace “average” with “median” since, for one, they are different, and secondly, I didn’t actually measure the median. I fully expected for this to be an issue, hence the paragraph about it. I also consulted with a very keen stats person who agreed with my assessment that unless every one of the nearly 1200 respondents truly were that much better and truly did compare themselves to a non-beer interested population, the results are more accurate than not.
Ok, fair enough, but this sentence isn’t currently accurate as it stands: ‘56% of respondents believe their tasting ability is better than average, which is a logical and statistical impossibility’.
Funny you should bring that sentence up, as that was the main thing I discussed with Justin. I have to disagree, it is accurate relative to the survey that was taken, which is what I was commenting on. I get that people are going to nitpick this piece and can understand why.
~90% of the population still drinks macro. Most, if not all of the people reading this article and taking part in your survey should have said “better then most” when asked “compared to others, how good of a beer taster are you?”. 😉
If they compared themselves to macro drinkers AND if their abilities have been tested to actually be better.
I think the problem here is population comparison, did people believe they were comparing themselves to other beer drinkers, others homebrewers, other readers of the site? The question was vague enough that we can’t really tell.
I address this in the article, it was intentional. And like I said there, the population to which a person chose to compare themselves is nearly as, if not more, interesting than the data itself, in my opinion.
HAHA I didn’t read the last few paragraphs
Yep when I answered the question “rate their beer tasting abilities compared to others” I was raking my self to everyone in general not just “Craft” beer drinkers.
Your decision to do so presumably driven by something deeper. Also, the belief there is that general beer drinkers actually aren’t as good at tasting, yet where’s the evidence to support that?
Beer is an acquired taste. I would say that when I first started drinking I thought all beer tasted roughly the same. Now, I even have preferences within the macro sphere.
You asked to compare to “others”. Logically that would be other beer drinkers and there are a LOT of beer drinkers who don’t drink craft or homebrew. It is also quite logical to assume that IF you have been exposed to a wide variety of beer flavors and have some understanding of what those flavors are and where they come from that you have a more educated palate. That works for cooks and wine experts so I don’t see why anyone would assume that someone who only drinks miller lite would be as sensitive and knowledgable about what they are tasting as someone who has tasted a wide variety, and has at least attempted to break down the flavor profile. Again I know this is true in wine tasting.
So even if everyone in your survey said they were great at tasting it still would not be even a microscopic portion of the hundreds of millions of beer drinkers.
IF you had said compared to others taking the survey that would be a different matter.
As is true for most things the question is often more important than the answer. The way the question was asked make the results fairly meaningless since you have no idea what question people were answering. And your conclusion is wholly unfounded. Though of course that doesn’t mean it’s not accurate, just that the survey doesn’t support your conclusion that people over rate there tasting abilities.
Which BTW is kind of confirmed by peoples assessment of how well they pick out various flaws.
The BeXperement scoring is also hard to draw much from. These were all tests that were done and reported. It’s safe to assume that many people read and knew the details and results of those tests. So it is probably a decent look at how many folks think they can do better at something than others. I find this esspecially true with people who have not done X in the past and so have no real understanding of the difficulties. Kind of that “how difficult can X be?” syndrome.
The most interesting part I found was your experience VS score graphic. Brewers with less experience feeling better about their perceptions than those with more years. That also matches my experience. It takes awhile to lean what you don’t know and are not very good at. And when your a “young stud” you think you know it all, or at least most…
I guess since we are speaking about a qualitative scale then the concept of average is pretty abstract as it is, but if we were speaking about a numerical top-heavy distribution then we could find that the majority of people were above average tasters. I am not suggesting that is the case, just saying that there is nothing in statistics or logic which disallows 56% of a population from falling above the average in some measure.
Anyway, I am not trying to start an argument and I love all the work you guys do, keep it up!
I love these types of chats and completely see where you’re coming from!
“In fact, a staggeringly disproportionate 56% of respondents believe their tasting ability is better than average, which is a logical and statistical impossibility- how can a majority of people be better than average?”
At first I was going to argue my case on this point stating I answered based on my observations in my homebrew club where we do a monthly competition and select a winner. The score sheet is simple and we write down aromas and flavors we perceive then, if possible, give feedback to improve the beer. I then compile this data and give it back to the brewers (all anonymous). I hear people say they smell or taste something, I then determine if I am tasting the same thing they just put a name to for me or I think they’re just wrong. Based on this experience is how I answered the survey with the fact in mind that most the club members are newer brewers.
I was also going to point out that the people reading this blog and taking your survey aren’t going to represent your typical homebrewer. We are the people who are always learning and not satisfied with the Status Quo. We are hard on ourselves and our own brews. We don’t take compliments well about our homebrew unless we ourselves think it’s a stellar brew which are few and far between. We know we’re not perfect but are always striving to be better.
Then you said these two things and I decided I didn’t have to make the above arguments 😉
“Superficially, beer is just beer, but for those of us so invested, it’s much more than that.”
“I expect to field some complaints that the survey didn’t specifically ask about perceived abilities relative to a specific population.”
Thank you for the making us all feel like a bunch of arrogant pricks first thing in the morning. I’m going to cry in the corner now.
I laughed when the results showed we think pretty highly of ourselves! Why cry, smile!
Interesting exercise, but given the audience, you perhaps should have included the question “are you drinking craft beer right now”? Similar to the Dunning-Kruger effect, we have the alcohol induced “I am awesome, and can fight those four 6′ 4″ nightclub bouncers after 10 IPA’s” effect. You may find we’d all had a few too many home brews before taking the survey 🙂
Ha, good point… this data is garbage!
Haha and hear, hear
I lied to make myself seem cool. I thought you were going to post specific data regarding each person’s answers. Oh well…
Seriously though, this is the kind of stuff that interests me the the most about your blog. I am always anxious to see the data from your exBEERiments, not so much for the exBEERiments sake but for the human component. Keep it up, this is truly a unique avenue you’re paving the way for.
There should have been a question at the end if “Were you lying?”. Hahaha
After all the apparent rationalization, which is completely to be expected, I wish I would have had the question: “To whom did you choose to compare your abilities?”
I am reading this quickly between calls at work (and as a non native English speaker), but couldn’t the responses re being able to distinguish different xbmts be viewed in another way – i.e. that people do trust the xBmt findings but tend to believe that they are somewhat more capable testers than the people in the panels.
When asked if I believed I would have been able to distinguish e.g. hot-side aeration, I asked myself “do I believe that it is a factor at all”? Since the answer is no, I answered no. The data would suggest that most people did the same as me re HSA and boil lengths.
When asked about the other xbmts, my answered was more based on if I would have been able to distinguish that factor (which I believe to exists at least in extreme values) when prompted to or if it had too little of an impact to do so given the specific nature of the xbmt (my only yes was re chemicals for what its worth). For lager temp, mash temp and chemicals people seem to have answered pretty close to the actual result. I am in no way stating that everyone responded in the same way as me, neither that you can compare the response percentage with the ones who were able to distinguish the correct one in the test, but read from this perspective I do find that there is some correlation (granted that most people in the survey probably not answered yes factoring in a 1/3 chans that the answer would have been correct based on sheer luck).
I four one miht have had a few befor etaking the Survey
Keep up this excellent work Marshall.
I guess my issue with your analysis is that I don’t consider that, for the group people who responded to your survey, 56% above average as a beer taster is anywhere near “staggeringly disproportional”. If fact, I’m surprised the number was so low. Readers of this blog typically represent the more educated among beer drinkers, and it is certainly reasonable for 56% of them to consider themselves above the 50th percentile when it comes to beer tasting ability when “compared to others”.
I think your analysis takes a lot of liberties in the assumptions you are making. Maybe it was not your intent, but your tone comes off as chiding the respondents for thinking so highly of themselves. You can’t take such a strong stand on the “compared to others” question, when you freely admit later in the conclusion that the question was intentionally vague.
I’m sure I am not the only reader who is going to be rubbed the wrong way by your analysis here. Again, that may not have been your intent, but that is certainly how it comes across to me.
I’m with Eric on this one. I like a lot your work, but I think you are undoing you accomplishments with the tone of this article. (This coming from someone who put he was below average)
I appreciate your feedback. The purpose wasn’t necessarily to rub anyone the wrong way, though it doesn’t surprise that some people might be uncomfortable with the analysis– this happens often in the social sciences, it doesn’t feel good to consider these types of things.
One thing I want to point out, which I said in the last paragraph of the article, is that I completely include myself in the group I’m talking about, I’m not pointing the finger, I’m looking in the mirror.
Cheers and thanks again for sharing your thoughts!
I’m a bit skeptical about your conclusions re: “Compared to others, how good of a beer taster are you?”
I think your survey and its results reflect a type of selection bias. Your question in the survey was “Compared to others, how good of a beer taster are you?”. What you have done here is capture the general population as your population in the survey. Most people who home brew only know a few others who also home brew, but certainly know a huge number of general layabouts who swill crappy mass produced beer. It’s easy to expect respondents in your survey to think they know more about beer tasting than the majority of their friends and associates.
Your sample has inevitably captured people who home brew. Even more specifically, people who appreciate some scientific process and who follow you and your site on social media. The sample you obtained is not representative of the population you intended to analyze.
Making comments like “a staggeringly disproportionate 56% of respondents believe their tasting ability is better than average, which is a logical and statistical impossibility- how can a majority of people be better than average?” and then being perplexed by the results is just a reflection of a poorly designed survey.
Its easy to expect that most of participants in your study captured by your method of distribution (people who follow you on social media and love your site – myself included) would think themselves better than the average population in tasting beer. Your site followers are home brewers who are actively trying to improve the quality of home brew they produce. Compared to their mates who love shitty Aussie (or US) water flavoured beer, most of the respondents to your survey definitely would be a better judge of beer taste!
Thanks for all the content, your site is sensational.
> What you have done here is capture the general population as your population in the survey.
This can only be assumed.
> It’s easy to expect respondents in your survey to think they know more about beer tasting than the majority of their friends and associates.
But do we, in actuality? Seems we certainly think we do 🙂
I see what you’re saying, don’t get me wrong, it’s just that that’s sort of what I was trying to get at!
>But do we, in actuality? Seems we certainly think we do
Probably but we don’t know for certain.
If the survey were done with “normal” beer drinkers rather than just self-selecting homebrewers, I’d suspect that you’d see a lot more “What is that” answers to questions about sensitivity to off-flavor compounds.
The group that self selected to take the survey might not be better tasters (I don’t think I am and don’t know that I answered above average to any of the questions) but they know more about beer and off-flavors. This *likely* makes us more likely to think we’re better tasters. “Normal” beer drinkers are less likely to know what to look for in their tasting of beers.
Personally, I know that I could improve my tasting with more focused practice but in the end, I just like making and drinking beer. I mostly enjoy where I’m at with the hobby. I’m on the far end of the experience spectrum (20 years) despite not being that old which is another self-selecting feature of the poll. This poll skewed towards younger folks, more likely to be frequently on the internet. My 65 year old dad and brew partner would be very unlikely to visit the site or take the survey.
I agree with others who indicate that the terms “impossibility” and “absurdity” do not apply when you’re: A) limiting the population of those surveyed to those people who obviously have a taste for beers and homebrews other than the mainstream Budmilloors, and B) limiting the population of those surveyed to those who have an interest in taking surveys about how awesome their tasting skills are.
Everyone thinks their crap doesn’t stink 🙂 In full disclosure I voted as “Somewhat Above Average” and I am Certified BJCP who scored an 85 on the tasting exam. I’d like to think I was accurately answering the question.
That was a fun read, thanks! As I answered those questions I was thinking roughly along the same lines.
My conclusion I think has to do with your reader base. It seems fairly logical that as readers of your blog, compared to most, we have a bit more sophisticated understanding of the topics which in turn probably makes us more susceptible to the bias discussed.
Interesting survey, not sure I find it particularly surprising. I would imagine readers of this blog are a pretty well informed and educated group when it comes to craft beer and home brewing. I wouldn’t find it out of the ordinary to find out that this group has better sensory abilities than the population in general.
From my own, extremely anecdotal, experience among my friends (craft and non-craft drinkers alike) shows that they really don’t have much sensory knowledge when it comes to beer. I don’t think I’d do any “better” than the tasting panels in past experiments, but I am very confident I would do better than the population in general. I find non-craft beer drinkers can’t even pick out the difference between malt and hops in a beer let alone diacetyl, DMS and acetaldehyde.
I wouldn’t say its sensory ability that sets us apart but rather experience. I can pick out the taste of those of these things because I’ve had them in my home brew before. Then begins the effort to identify these flavors and correct them for the future. Without that experience I would just think it was beer.
Great feedback! I guess what I wonder is whether less experienced drinkers simply can’t describe the differences they taste as well as the more experienced. I’ve looked and can’t seem to find anything suggesting experience actually improves one’s sensory abilities, though learning to speak the language certainly helps us get our point across better, which in turn might make it look as though we have better abilities.
I’ve a feeling I’ll be typing this quite a bit today, but I honestly have no clue and could be completely off in my assessment.
No, I think you nailed it. I believe that “regular” drinkers simply lack the proper language to describe what they are tasting. I’ve found with my wife that if I point out something that I perceive as an off flavor she (says) that she can then pick it out as well. It may simply be that she isn’t aware that what she is tasting is an off flavor at all, let alone be able to exclude that flavor from the rest in the beer and describe what it is.
I think that’s the crux of the argument here. I don’t think you can actually improve the number of your taste buds with experience. But I firmly believe that the processing of the data that they send to the brain improves with practice.
Tasting is like any other skill. There is definitely a fixed limit based on a person’s biology (i.e., number and type of taste buds, for example), but it takes practice to be able to use this to the best of one’s ability. Someone who is biologically a “supertaster” is not necessarily a better “beer taster” than average just by virtue of biology.
My wife, who isn’t a homebrewer and drinks beer occasionally, could easily be a better beer taster than I. I have no clue! It’s both a biological and philosophical question: who has the better taste buds? That’s empirical. I’ve heard that women can smell better than men, on average, for example. Secondly, how does learning about homebrew effect the way we talk about beer. I hand her a Belgian and say “this tastes like bubble gum,” right? That’s what I want it to taste like, that’s what it’s supposed to taste like. She says: no it doesn’t and gives some other description. She could be right!
Also men are stupid and generally full of themselves. ha.
My wife, who prefers her alcohol to taste like soda, is definitely more sensitive to smells and tastes than me. There are times I’ll ask her to smell a beer and tell me what she picks up, after the predictable “it smells like beer,” she’ll almost always give a couple descriptors that I didn’t even think of. Who’s better? I don’t know.
Language would be one part but the other is actually tasting what they are drinking. I drank wine for many years before meeting someone who was a bit appalled at my “taste” in wine. She was right. Around then I started actually tasting wine rather than just drinking it. Some time after that I had some of the language to go with those tastes.
So on a pure perception basis probably the homebrew folks are closer to average with probably a lower sensitivity to bitter than joe sixpack. But Joe doesn’t “taste” his beer he drinks it. So I actually think that there is little doubt that the folks who responded to your survey should mostly be in the better tasters end compared to the average beer drinker.
It is also quite possible that if you had asked how well you taste compared to other home-brewers you would have gotten results that showed we over rate our abilities.
That would have actually been an interesting comparison, to rate you ability VS everyone, craft beer drinkers, and to other home-brewers.
This is a great post and a perfect example of the Lake Wobegon Effect (aka Illusory superiority bias), a pretty well known effect in social psychology. It would have been surprising NOT to have found it. This is a great reminder of our biases, and I’m going to point students in my research methods class to this post (I teach psychology).
I especially liked the correlational analysis of #years brewing vs perception of tasting abilities. Interesting stuff.
One thing I’ve never liked in forums, or comment sections, is the idea that people who drink macro beer have bad taste, or bad palates. Taste is subjective, right? I live in the hometown of Labatt Brewing (London ON), don’t care for the stuff myself (for macro beers in Canada, Molson Canadian actually tastes pretty good, but mention that to most craft beer drinkers and be prepared for the condescension), but have toured the brewery many times and know the care they take to ensure that their product tastes the same batch to batch. Labatt Blue is pretty damn clean (and yes, not exciting to me, but who cares). If it suddenly tasted of banana, like Bud Light does (also brewed here in London), I’m pretty sure a good number of the macro drinkers would ask themselves “what the fuck?” because they noticed the ester. Just because people choose to drink a macro beer does not mean they don’t actually notice what they are tasting.
FWIW, I am a craft beer drinker (though I kind of dislike the term “craft” beer-there’s plenty of shitty “craft” beer made) but won’t turn away “macro” beer unless it is Corona.
And yes, I overuse brackets.
Spent 9 months in Guelph in the mid 90s and I agree with both your Canadian vs Labatt analysis as well as Canadian being pretty good beer.
Mainly just wanted to say “Good day, eh”
I like Labatt Blue…
Made me laugh. Very Bob and Doug McKenzie.
Out of curiosity, did you ever actually hear anyone say “Good day, eh?” while here?
Still can’t get behind Blue, even though it is the hometown brew.
Heard that ALL the time. Could have been very Guelph specific, I guess. Lots of other uses for “…, eh?” as well.
I only made it down to London once, and that was only for a couple hours while I pulled a bunch of ISP equipment out of a strip mall for Hookup Communications.
Thanks for the continued work and insight, all fascinating stuff. All *your* exbeeriments have made *me* a better brewer.
I have taken a couple of Cicerone prep classes at a local brewery. Highly recommend that everyone look in to those, whether you plan to get certified or not. Part of the class was style tasting as well as off flavor tasting, and was why I took the class multiple times. What was made painfully clear to me, was that I can generally tell you if the beer is good or bad, but I have a hard time telling you exactly *what* I am tasting or why it is good. (This is not a preference thing, ie i hate lambics, but I can generally tell you if I am sampling a good lambic) For me, I found this to be both a language problem as well as perception problem. I seem to be unable to pick out specific flavors from the whole.
I rated myself on the low end of the scales in this survey. However, before the classes, I would have said I had an average or slightly above average palate. I wonder if the results would change if more of the pool were exposed to a more regimented tasting framework and evaluation, essentially forcing them to confront their own shortcomings.
I personally would argue that if someone is unable to pick out a specific off-flavor from the whole, then the flavor really isn’t “off”.
With home brewers there seems to be an over-emphasis on finding flaws, rather than just saying what they LIKE about something (at least in forums). If you really have to concentrate to pick out a little pear flavor or whatever, its not really worth dwelling on. Just my opinion.
I can’t say I disagree with you.
Marshall, you are a true empiricist! And that’s why I love and support this website.
This study definitely has its limitations (as all do), but there is value nonetheless in the data:
Reading this article definitely made me realize that I am guilty of these biases. I rated myself as being better than average; however, I have ZERO data/evidence to support that rating. Shocking! But how could I not be better than most when I care so much my craft!? I think that anything which forces me to identify how agnostic I truly am is valuable, so thank you!
I would note that this study’s design was cross-sectional which only permits one to understand the descriptors of a target population, establish a metric’s prevalence, and generate hypotheses for future testing. As such, inferences drawn from this study about the target population are based on opinion and not statistical probability as hypotheses were not intentionally tested. I think that your decision to keep wording vague regarding comparisons ultimately hurts the validity of your claim about biases since we now potentially have a number of different comparisons being combined into one aggregate result. Further, I think your ability to demonstrate bias would have been strengthened if you had correlated self-perceived ability with legit credentialing, which was unfortunately not included.
Hopefully you don’t mind my critique- I’m not trying to be a dick, but rather love thoughtful conversation about data! Keep up the amazing work!
I’m just as guilty as you, my friend. Thanks for the great feedback, not only do I not mind the critique, I appreciate it!
I wonder if part of the “slightly above average taster” responses from homebrewers were generated by the “You are your own worst critic” effect. Maybe they really truly believe they are slightly above average taster because when drinking their own beer they’re always looking for off flavors.
“I’ve always noticed when I had diacytal/dms/astringency/overly estery/overly phenolic in my own homebrewed beers, so by extension I am a slightly above average taster of beer because I can pick out my own flaws.”
I don’t know what this affect would be called but, a bias definitely.
Very interesting comment responses. In regards to the ‘off flavors’ the results seem to fall generally in line with a Gaussian distribution (bell shaped curve). Not perfect but good enough for government work.
Personally, I marked average for most. I don’t consider myself a connoisseur but I do know good beer from bad beer and know great beer only as it relates to my personal likes and dislikes.
Great stuff!
As others will undoubtedly remark, when I answered these survey questions, I put them in the perspective of “Average against all humans” not “average against other homebrewers and craft beer drinkers.
The “Average” human being says things like “this tastes like beer”. They don’t even know how to articulate differences in taste and wouldn’t know a malty beer from a hoppy beer.
In this light, almost all of the readers of Brulosophy are probably a little above “average” in almost every tasting perspective, compared to “Joe Six Pack” who couldn’t tell the difference between an American Light Lager and a Munich Helles.
The results of the various Brulosophy experiments primarily use these “above average” tasters (homebrewers and craft beer lovers) as the test group, not the “average Joe Six Pack”.
Thanks for the great response. Now I wonder if I’d be able to tell a difference between a Light Lager and a Munich Helles…..
That sounds like an idea for an exBEERiment.
I find that even if you go for “average of entire populations sensitivity”, weren’t most of us “regular beer drinkers” before we go into the world of craft beer and homebrewing? So my guess is your palate hasn’t changed much, unless you quit smoking or had a major surgery that affects your body’s hormonal balance, your palate is probably largely the same as when you drank BMC pre-craft-awakening.
If anything since becoming a craft beer enthusiast, and a homebrewer, I’ve found my sensitivity has greatly decreased. What has improved is my capability to describe flavors, and my vocabulary to describe those flavors. My fiance’ on the other hand can pick out stuff I don’t notice at all, but I have to prompt her with sensory descriptors and words that normally aren’t used “do you taste green apple? or do you get hints of canned cream corn?” For example with my weizenbock I’ve recent completed I can *barely* pick out the banana-ester quality, barely, I’m almost angry how little I can pick out the banana. Though multiple people who are not generally craft beer drinkers, or homebrewers, all describe a big upfront banana character in the aroma and the flavor.
Now is it more likely I am a “below average sensitivity taster” and my sensitivity as decreased as time has gone on as a craft beer drinker, or is it more likely the non-craft beer enthusiasts are full of crap and are not actually tasting banana-esters in my weizenbock?
Doesn’t the dunning-kruger theory imply that the folks with the most experience and ability tend to underestimate their relative ability, or to rank themselves appropriately, while the people who are low ability (I believe lower quartile was the cut off point in the study – thank you wikipedia) tend to significantly over estimate their ability? Your snippet on the effect singled out the propensity for ‘relatively unskilled’ people to overestimate ability, without qualifying that that effect is most significant for the truly shit while the awesome folks would tend to reliably place themselves at an appropriate or lower level. Is the belief that this effect could skew the self assessment distribution to such an extent a manifestation of your own bias about the kind of hoi polloi who take part in the tests? I jest of course, but the fact that taste is such a subjective matter just makes this kind of discussion even more fun.
Shame you couldn’t have tested all those respondents’ actual ability to reliably detect faults / taste to determine where the folks were in terms of actual ability vs self assessment to determine the extent of the self deception in the brulosophy typical respondents. After all, there’s nothing more satisfying than pointing out the shortcomings of others is there… Love the blog by the way.
I’m a big fan of yours but I have to say I was frustrated with all but the first two questions. 😉 In fact, I nearly quit and closed the survey multiple times.
Why not define “beer taster”? Now that I read this post and responses I’m thinking you meant like BJCP taster? That’s not what I thought when I took the survey
Also, and I know it’s been beat to death above already, but without context (compared to what “others”?) the results of the 3rd question to me are dead on. Think of it this way, my answers to each of the following questions might have been quite different:
1. “Compared to all beer drinkers in the US, how …” Well jeez, compared to someone who drinks Blatz, what am I supposed to answer here? I’d answer better than most.
2. “Compared to other readers of brulosophy.com..”, I would have answered “worse than most”. Sorry, I think appropriate results are exactly what you got.
Now, what would be funny is if you simply asked the 2nd one I listed. IOW, “how do you as a reader of my blog compare to other readers of my blog?”.
Yet, the main thing that frustrated me was the off-flavors. You have 3 options about sensitivity then 1 that says “what’s this”? Do you really think there’s nothing between there? How about, “not sure what this tastes like”? Because I’ve heard of and try my best to prevent all of these off-flavors by doing all the things you’re supposed to do in brewing. So I ended up picking average for all of them without any thought to each particular one as I suspect many people did because “what’s this?” just wasn’t the right answer. Kind of wasted all of those questions, IMHO.
I’m a fan and hate being critical but really feel like you got bad (or fairly useless) data from wording critical aspects of most of the questions poorly.
Yes, I know you wrote about the “compared to others” and prepared to receive flak for it but that doesn’t render the question usefully written. 🙂
How about a redo?!
Damn, one more thing. Regarding the off-flavors, again, I sure wish you had something besides “what’s this?” – I’ve been brewing for nearly four years and only last month had my first taste of another homebrew (well, at least since 2004 or so) and vice-versa. And I’ve never had any training of any sort about what these off-flavors would taste like.
So, “I’ve heard of it but don’t know if I’ve tasted it” or something like that would have been much better.
It was a silly, largely invalid blog survey. I get it.
Dude, you just had your first try of someone else’s homebrew after 4 years?! Argh! Some of the best beers I’ve ever had have been homebrew. No club nearby?
No redo. A minority of people took it the wrong way, I didn’t expect that, I’m not very interested in proving anything.
“It was a silly, largely invalid blog survey. I get it.”
I think my tone came out much more harshly than intended. Sorry about that. I didn’t think it silly by any means. I’ve said it before and I’ll say it again – I’ve learned a GREAT deal about brewing from your website, including things to NOT get too worried about. This website has reduced my brew-day stress level significantly with absolutely no decrease in the taste of my beers. 🙂
RE: other’s homebrew. LOL – well, I’m a bit of a loner. Frankly most people aren’t very interesting to me and I’m not that sociable (though fortunately I’m happily married). I know that sounds like a pretty rude thing to say. BUT! Consider where I spend my time instead of with them – places like your website, so it’s truly a compliment to people like you.
Having said that, I just had that first home-brew last month because I finally broke down and joined our state-wide society of home-brewing.
Come to HomebrewCon! It’s going to be awesome!!!!
I rated myself very low on everything because I always perceive all (craft) beer as great and if there is one I don’t particularly like, I blame it on myself and drink it anyway…and then order another one.
Lucky me!
Marshall – keep up the good work. No survey is perfect and the results are what they are from whom they were taken. Personally, I know that I am below average in tasting abilities among those in my BJCP class, because we tasted flaws on spiked beers and I frequently had difficulty perceiving the flaw – we were not told about a flaw being present, by the way – and other younger palates picked up the flaw readily. And yet I scored an 80 on the tasting exam – better than most others in the class. Why? Vocabulary. Does that make me a better judge? Perhaps, because I describe fairly well what I perceive. But as far as tasting abilities, I am what I have been given (or, sadly, not blessed with).
Intesting post as always!
Being a Homebrewer in the 1 to 4 year category, I know I personally was comparing myself to the non-beer drinking crowd, and I hypothesis the 1 to 4 crowd are less connected in the homebrewing world but still have experience, and therefore are more likely to think of all their non-beer drinking friends who describe a heffy as roastyvor whatnot. Just a hypothesis. Also, I think a lot of people view being able to associate a certain beer with a certain style as being a good taster of beer, which admittedly, has all to do with their knowledge of styles and little to do with actual tasting, but nevertheless as it is often what we do when we analyize beers, I think it must be associated in our minds when someone asks the question: are you good at tasting beer. We may not be claiming to have better taste buds, but rather, simply acknowledging our knowledge on the subject of tasting beer.
I’m in a Siebel sensory training class at Raleigh Brewing right now. Couldn’t smell the green apple in the Acetaldehyde vial 🙁
Have just had my sh*ty palate confirmed. Couldn’t perceive the aroma/flavor of diacetyl or DMS, either. No wonder I think my brews taste so good! Can’t perceive any off flavors that might be present
Perhaps some work with a trained sensory panel might be a useful thing
Here’s my thought: I selected myself as a better than average taster of beer, because to a certain extent I can identify ingredients, sometimes specifically and sometimes more generally in a beer when I drink it.
However, I’m not sure that this is because I am a better taster than others, rather because I have learned what things taste like and have gotten better at describing what I taste.
Just some food for thought, comes down on the same sort of rationale for BJCP judges not being able to better distinguish different beers than regular folks
Pretty normal findings to me. Overconfidence is a normal aspect of life. Thanks for the research