Authors: Marshall Schott & Greg Foster
With help from: Scott Janish & Justin Angevaare
Since starting the xBmt series, we’ve received critiques regarding the fact so many have returned non-significant results, many of which seem quite valid, such as the p-value being imperfect or our sample sizes too small. However, there’s one complaint we get more than any other, I can only imagine as a mostly benign knee-jerk response to data that doesn’t jibe with one’s convictions. I refer to it as the “shitty (bad) palates” argument and it entails presuming/accusing the xBmt participants of not being able to reliably distinguish a difference because they aren’t very good at tasting stuff.
Totally understandable, I’m guilty of making such assumptions, even though I know most of the people on my tasting panels and trust their palates more than my own. One of the wonderful aspects of science as opposed to other “systems” is that questioning of results isn’t only expected, it is encouraged! Hypotheses aren’t presumed to be correct based on hunches, but rather put under the proverbial microscope with the hope of gaining a deeper understanding of whatever it is being investigated. I’m reminded of something the great biochemist, Issac Asimov, once said:
The most exciting phrase to hear in science, the one that heralds new discoveries, is not “Eureka!” but, “That’s funny…”
Inspired by the search for something funny and curious whether our results were indeed impacted by our participants’ shitty palates, we compiled the data from over 30 of the most recent xBmts to see whether this argument holds any water. With the help of Scott Janish, we put together the following infograph to display the performance of participants based on their level of experience. Check this out.
Whaaa?! This took me by surprise, no doubt. I reached out to our statistician friend, Justin Angevaare, who blogs at On Brewing, to see if he might shed some light on the performance differences between BJCP judges and non-judges. Here’s what he had to say:
Logistic regression can be used to determine if there is a statistically significant link between successful performance on the triangle test and participant experience level. This method allows us to model a binary outcome on the basis of one or more explanatory variables, then the statistical significance of these explanatory variables can be tested. Here, the odds of being correct on the triangle test would be modeled by taster experience level.
Using this method, the odds of selecting the correct sample in the triangle test (the odd-beer-out) by participants holding a BJCP rank of provisional or higher were compared to the same odds for tasters who hold no BJCP rank. To make this comparison accurately, differences among individual xBmts had to be accounted for in the model, which was deemed necessary because the xBmts varied in terms of difference detectability and participant demographics.
In the end, there was insufficient evidence to suggest a difference between BJCP judges and those without such ranking in terms of odds of success on the triangle test (p<0.05; p=0.7698).
Interesting, eh? While it might be easy to assume a person who has invested the time and energy to become a BJCP judge has an extraordinary palate, these results appear to suggest they possess tasting abilities similar to, well… everybody else. Rather, what seems more plausible is BJCP judges are perhaps better at describing what they’re experiencing than others, much the way a sound technician is better than a veterinarian at talking about sound stuff. They know the language and what they should be looking for. It should be noted the triangle test is designed to examine only one’s ability to distinguish between samples based on flavor, aroma, and mouthfeel, participants aren’t asked to describe their experience, thus arguably placing judges and non-judges on a level playing field.
Are you saying there’s no value in becoming a BJCP judge?
Absolutely not! Many people find great meaning in the competitive and evaluative aspects of brewing. That’s awesome, I’d never suggest folks not do what it is that makes them happy. I’m a judge myself, and while I’ve always been highly skeptical of the between-judge reliability when it comes to scoring, it is a lot of fun and contributes to the building of a rad community, that’s what matters to me. More than anything, I think this data demonstrates the shitty palates argument is largely futile, which may come across as a bit self-affirming. Fine. I also think it serves as justification for what we’re doing, not that it increases the validity of the xBmt results in any way, but maybe, hopefully, it’ll reduce the worry some have that the non-significant results are more a function of the tasters’ inability to accurately detect differences because of lack of experience rather than the more likely scenario the beers just aren’t that different.
Finally, this is not conclusive proof of anything. Although the data seems to support the notion that those trained to identify and describe beer character are no better than general beer drinkers at distinguishing between the samples included in the 34 xBmts analyzed, it’s wholly possible this is a function of some factor we’re missing. And this is where opinion comes into play, as one’s beliefs about what it means to be a BJCP judge, and the convictions they hold about their own tasting abilities, is most certainly going to shade the manner in which this information is digested. And that’s okay, it’s to be expected, we’re only human.
Huge props to Greg for compiling and organizing the data, to Scott for helping put it all in a visually pleasing and easy-to-read format, and Justin for his willingness to lend his statistical genius. Cheers!
Support Brülosophy In Style!
All designs are available in various colors and sizes on Amazon!
Follow Brülosophy on:
FACEBOOK | TWITTER | INSTAGRAM
| Read More |
18 Ideas to Help Simplify Your Brew Day
7 Considerations for Making Better Homebrew
List of completed exBEERiments
How-to: Harvest yeast from starters
How-to: Make a lager in less than a month
| Good Deals |
Brand New 5 gallon ball lock kegs discounted to $75 at Adventures in Homebrewing
ThermoWorks Super-Fast Pocket Thermometer On Sale for $19 – $10 discount
Sale and Clearance Items at MoreBeer.com
If you enjoy this stuff and feel compelled to support Brulosophy.com, please check out the Support Us page for details on how you can very easily do so. Thanks!
46 thoughts on “Investigating the Bad Palates Argument | A Graphical Look At xBmt Performance Based on Experience Level”
Great article. I have often wondered about this when reading your xbrmts. I also wonder if it says something about the consistency and experience of the brewers brewing for the xbrmts? There are so many variables that are impacted by so many factors, systems, and practices, I never expect brulosophy to become the law of home brew, but I appreciate that you are willing to questions established canon.
I’m not sure I accept anything as the law of homebrew, haha. And to be honest, it’s our consistency and experience gives our data a tad more validity, which to me is a good thing. Perhaps it’s the inconsistency and lack of experience of some homebrewers that are responsible for bad beer?
This sort of transparency is really critical when conducting and publicizing any experiment. I really appreciate what you do. It inspires me to consider marketing my brewing company as being “Data Driven”. I’m not sure how that would look like, and it’s years down the line, but you’ve definitely planted a seed with your continued empiricism regarding homebrewing.
Nice work and nice infographic. I think you’ve laid to rest the part of the shitty-palates argument that rests on experience level, but the argument that palates could be muddled by drinking multiple samples in quick succession still stands. It’s possible there are subtle but perceptible differences that vanish when two beers mix in the mouth or when senses are fatigued by repeated exposure (in the same way that you can feel socks when you first put them on but then can’t feel them for the rest of the day).
Having said that, 12 significant results out of 35 sounds like a non-random outcome to me, so keep up the good work.
Our participants are usually served a glass of water with their samples and encouraged to palate cleanse if they think it’d help.
Wow, pretty awesome analysis! I questioned your tasters in one post I’m sure. This is pretty surprising on the face of it. If you sit and think a minute though, it does seem to ring true that someone could have great training but have inferior ability to taste things or smell things. You’d think solid training would overcome that. maybe the training is not as good as it could be?
I’ve heard that tasters for some companies are super trained and very good, etc. however, i also talked to an ex-taster for Pepsi. he said they were always desperate to find tasters and made it a part of some people’s job profiles. he had tasted gallons of product over the years (and gotten bad teeth as a result!) He is a pretty avid cigarette enthusiast, so I immediately questioned their process. If you’re letting smokers taste for you, you must not care much about the results!
I would have thought that also (RE:smoking) but when you realize how many great chefs were/ are smokers it makes one wonder. But I assume that taste works like sight and hearing in that you don’t see with your eyes or hear with your ears. They collect data that your brain sifts and filters and presents to you what it feels is important. This is the basis of optical illusions and tons of sound work. Also you can train your brain to consider things important that it wouldn’t normally. In sound recording this is critical since the very things that will ruin a recording are things that your brain is wired to ignore. The “training” can compensate for a lot of noise in the signal so I’m guessing the same is true for taste. If you train to taste certain things as a sign of X defect, that combo of taste data is encoded to trigger your perceiving that defect. So even if your mouth also has a coating of tar that data set when it shows up gets through because your brain is looking for it.
The one thing the brain is hardwired to detect reliably is differences. A “change” in data is often a sign that something is not right and you should pay attention. A not uncommon sound “trick” to put an audience on edge is to have a quiet sound bed like country crickets in the background. After a bit nobody actually hears them anymore since the brain is wired to ignore things that don’t change. When you want to put them on edge stop the background. They weren’t hearing the crickets so they don’t realize what changed but the brain just sent a red alert to pay attention.
So it’s not surprising that non trained folks are about as good at detecting a change as trained ones. But I bet the analysis of exactly what change is better with the trained folks.
We’ve found this to be the same in our triangle tests. The processes for performing a triangle test and evaluating a beer against style are different. Most BJCP folks spend so much time evaluating each beer that the lose their flavor memory and decrease pallet sensitivity for the other beers in the triangle test. After a few drinks, a one smoky beer isn’t quite as smoky, or a hoppy beer isn’t quite as hoppy (why no one wants their homebrew entry at the end of the flight!). The best method for triangle tests is to ignore style and move quickly between beers, which is opposite of the BJCP training. A triangle test is more like a mini-BOS or BOS judging; move quickly, be decisive. The skill sets developed in BJCP training are mostly unnecessary for a successful triangle test. Without an advantage from the training, then the null hypothesis will be rejected, which is what we have observed and what you observed.
I believe that if you want folks to critically evaluate a beer and style (like with a homebrew competition), then put them through BJCP training. If you want people to do better on the triangle test, then provide them an SOP that teaches them how to effectively perform the test. You don’t need to know what acetylaldehyde is to determine one beer with it tastes different from a beer without it. Each is its own specialized skill.
Good points. I think it’s also important to note judges are evaluating a beer and comparing it to a given standard. Give a judge a Dunkles but tell him it’s a Porter, she will almost certainly experience the beer as a Porter… and like it my case, may end up giving it the same 39 point score she gave the very same beer as a Dunkles 🙂
Yeah, unfortunately that is the case with many judges. 0-50 seems to be a more of a scale of how much they like it instead of how much it fits style. However, I’ve experienced both. I had one BOS panel of judges, granted they were all grandmasters, that called me out on my schwarzbier tasting too much like a porter. I had used Jamil’s winning recipe instead of his authentic (Kostritzer) recipe, and they were spot on.
I had a wonderful experience with this: I once organized a comp in which I also submitted two beers. I mixed up my entries and submitted the Double IPA to both Saison and Double IPA flights. One of the judges on both flights was the same person but gave completely different written comments for idential beers. I told her afterward what had happened, and she litterally didn’t believe me. So I pull the two opened bottles and gave her samples again. Yep, she agreed, same beer. Context of evaluation is huge in our sensory experience.
I’m BJCP recognized and I love judging beers, but this really confirms what I already believed about beer judging.
I really think that what experience does for tasters is not make their sensory thresholds better. Instead, it makes them better at identifying their experiences as like other experiences they’ve had and gives them words for that experience. Hence, an experienced taster can often put a somewhat standardized vocabulary on the difference between two beers and sometimes identify a compound that makes the difference (e.g., they recognize DMS or phenol) while an inexperienced taster grasps for words that convey the difference. But experienced tasters don’t learn to detect what they couldn’t detect before anymore than people who spend a lot of time with crayola boxes can see different colors than the rest of us. Experienced tasters are probably also a little more capable of sorting out difference aspects of their experience too; to continue the metaphor of color a little, they are quicker to see what the components of a shade are, as an experienced painter can quickly blend yellow, blue and white to get the perfect pastel green for a leaf but I would stuggle to blend the right one at all… but once again, the painter can’t see a different color, they just have a better skill for translating their experience for others.
“The best method for triangle tests is to ignore style and move quickly between beers, which is opposite of the BJCP training”
I very much agree with this.
Later on you also mentioned mini-BOS – I adhere to this method there as well. First time through is just a yea or nay for general process and style parameters. I often do a + or – for beers. Get rid of most of them and then drill down. Otherwise I feel you cannot accurately judge 25 beers.
The results conclude yet one more time, that the interior of one mouth and nose tell the truth to the brain just as someone who is color blind has no ability to select true red. Attempting to teach the brain that bitter is sweet and vice-versa provides for the erratic results on score sheets.
Until we realize that all judges palates are not created equal; simply different, relying on a judge to effectively score a series of beer styles is futile.
As a BJCP Grand Master, Certified Cicerone and Pro Brewer, I’m not the least bit surprised by your findings. The beautiful thing about triangle tests is that they reduce evaluation to a simple yes or no question. “Is one sample different?” You’re also correct that the main product of a BJCP judge is a description of the beer and that skill alone is what constitutes rank. In my experience the most sensitive and accurate tasters are those who are learning to describe beer. Often the BJCP exam is their motivation to learn. Luckily, it isn’t the exclusive motivation for beer lovers.
In most exbeeriments there are taste qualifiers as well, so “simply different” is not a true disqualifier of the results provided here. As Gordon Strong said recently – a near quote – “entering a competition is a crap shoot, just keep entering and hopefully you’ll get a string of judges who like your beer”.
The data really show that your non-BJCP tasters are as good as your BJCP tasters, not that the average beer drinker has as good as a palate as a BJCP judge. I suspect your non-BJCP tasters are quite knowledgeable, more so than the average beer drinker.
Interesting results nonetheless!
Great article although I have to argue with you on one point. By showing these results you make the hypothesis of bad palates substantially less likely. This, almost my definition, adds some level of validity to your xbeeriments. Which is awesome!… Even if they may be off for some other reason. Keep up the good work!
I think that one thing that separates this website from published research is that there is an incentive for professional researchers to make their hypothesis true. If they don’t find a correlation, they don’t submit it for publication. If you are a researcher, you have to publish. If you don’t get the data that you want maybe you hold out for more data. It is refreshing to see that so many things that are held up as gospel do not make that much difference.
Interesting analysis. I think your sound guy analogy is a good one. As a trained audio engineer I never assume my hearing is any better than any body else, it is simply that I’ve learned ‘where’ to listen and ‘what’ to listen for when mixing etc. Keep up the good work, cheers.
BJCP Provisional here, my test is about a month out. Being hip deep in the process, I can’t say I can taste any better now than I did before. I was (and still am) a pretty huge beer geek, and being able to pick out a different-tasting beer is nothing that BJCP training has helped with. I can’t say my palate is any better, either. As a matter of fact, I’m deathly afraid that I’m in way over my head. I know textbook explanations of off flavors, I’m getting the new style guidelines down pat, but I’m still afraid that when I taste a beer, I can’t pick out vegetal or DMS or whatever the case may be. I actually feel really influenced by the guidelines as I taste, like I’ll see that style X is supposed to have toffee notes and then “oh yeah, you can really taste the toffee”. Sometimes I think the whole thing is BS – you read about things where someone tastes cheap white wine doctored to be red and told it’s a $100 bottle and they think it’s amazing. You can’t trust what your brain tells you.
Anyway, I’m just rambling now. My final comment is this – I don’t think this says anything about BJCP, I think it speaks to the caliber of the non-BJCP folks you recruit, that their palates are just as capable of picking out a different sample as anyone else. They may not know a toffee note from DMS, but if you have two the same and one different, that is noticeable.
I think that additional piece of information is needed when a Xbeeriment is performed. There is a general description of experiences of the tasters, but it would be better to know how many tasters are BJCP judges, craft beer enthusiasts, general beer drinkers, BJCP in training, and homebrewers, for each xbeeriment tasting. This information is collected as shown by the infographs, but I think it would be better to add this information to the results of each xbeeriment, so that we can compile the same information, if desired.
Do you still feel that way? If the make up people doesn’t seem to matter?
My wife hates beer but she’s one of the best beer tasters I know. She identifies minute things she hates in every beer (particular hop flavours, esters, malt flavours, anything). I could see how judges could actually become insensitive to many of these things that make samples different in a triangle test.
Most of these comments baffle me, as Brewbag points out. These results are perfectly consistent with the actual results of the actual exbeeriments; namely, that there really wasn’t a perceptible difference for most of them (e.g. 30 minute boil, high vs low mash), but there really was for some of them (e.g. water). It doesn’t matter how trained a person is, if there really is no perceptible difference. This is not rocket science. It’s a simple experiment with one variable being tested each time. This is not confusing.
Marshall and the gang: keep doing these awesome experiments and don’t pay too much attention to irrational doubts and snobbish skepticism (for example, “I think that additional piece of information is needed when a Xbeeriment is performed” — no, it’s not, really). Keep using a mix of people. What you’re doing is truly one of a kind and we appreciate it. I cannot wait to read the next one.
-matt
Well said and pretty much what I posted right after you. You guys are truly providing a gift to the homebrewing community in proving that we don’t have to get every detail perfect and that many of the things that used to be gospel turn out to hardly matter at all. Bravo and thanks!
Thank you!
I really appreciate the response and you chiming in. For the record though – I do not feel the referenced post is irrational per se.
“I think that additional piece of information is needed when a Xbeeriment is performed”
I may occasionally be glib for the sake of being snarky but we should, IMO, foster doubt. Readers are welcome to challenge, further question, and make requests. Most all are discussed internally, some are addressed, some are heeded, some are not.
Instead, may I suggest we ALL, graciously, challenge each other’s assertions.
Everyone is talking about BJCP judges vs. non judges, but to me this validates all the experiments with that show no (measurable difference). I only just found this site a month or so ago but all these null results have helped me to realize that when Charlie Papazian said, “Relax, don’t worry, have a home brew.”, he was right. For example, the next time I miss my mash temp by a degree or two I’m pretty sure I’ll never taste the difference. Charlie was right all this time and now, thanks to you, we have science to prove it.
At the end of the article, I pictured you doing a “mic-drop”.
BTW, I’m colorblind so I had my wife interpret the chart for me. She read the whole article and said, “Beer sommeliers? Cool”.
Keep the xBmts coming!
After a little more thought I think I’d like to add one more thing. The audio analogy got me thinking. My father is a real-deal audiophile and when you listen to classical music (which is what he listens to) on his system you are blown away with the soundstage, imaging etc. However, even he’d generally agree that if he made one small change (line level cabling for example) few people if any could tell a difference. Not surprising. However, it’s the sum of all of the changes and upgrades that are responsible for the drastic improvement compared to a much cheaper system.
To the point: while I said above that if I miss my mash temp a little I’m not going to freak out that’s analogous to having just changed one component. I do believe that applying all of the accumulation of the things that I’ve learned (and the upgraded equipment) has made a difference in my beer.
Said another way, just because the beer still seems to taste the same if I mess up just one part of my brew day (mash temps, for example) doesn’t mean the beer will still taste as good if I were to mess up a bunch of things all in one day.
Here’s an experiment: compile as many of the “no difference tasted” experiments’ variables all into one batch and see what happens. I bet THAT beer will stand out. 🙂
“I bet THAT beer will stand out.” I bet it will taste just fine David. Because I do them. I have adopted many of these practices – 30 m mash and boil, no chill, no aerate, no starter – and my beer tastes as good or better than it used to. Just my experience, anecdotal, but something to consider and try for yourself.
Doing two or three variable exbeeriments would be logistically hard to plan, and I’m sure they have some planned, and they might turn out different, but I bet the beer turns out just fine.
” just because the beer still seems to taste the same if I mess up just one part of my brew day (mash temps, for example) doesn’t mean the beer will still taste as good if I were to mess up a bunch of things all in one day.”
This is exactly the argument that I made on a previous experiment. When you have a solid process, a significant difference in one variable might not be enough to expose itself against all of the other constants. I would love to see experiments where several variables are pushed closed to the edge while forcing the main test variable to an extreme.
The shitty palate argument was always kind of dumb. If a panel of beer drinkers (the people whose opinions actually matter) cannot perceive the beers as different, then it really does not matter whether or not the beers ARE chemically different as judged by mass spec or some other analytical means.
In fairness to that argument, when p is kinda small but greater than .05, you might start to wonder if there’s a real effect but something is keeping it from significance. Bad palates aren’t a bad place to look for noise.
I see what you’re saying and agree with the idea it’s a potential “noise maker,” though I’d reckon some other factor is more likely the culprit.
I keep reading this as The Bad Pilates Argument and wondering what Pilates has to do with homebrewing
And at first I was reading it as PIRATES….. Argh.
While I am not surprised by the results here, I am a bit surprised that no one else has noted two possible ideas that seem very obvious to me from the last two bar charts:
1. Homebrewers are more likely to be able to distinguish differences
2. BJCP-in-training folks are less likely to distinguish differences
If these observations are statistically significant, possible explanations might be that homebrewers better understand the processes and ingredients, and their impact on the finished beer; and BJCP in training folks are prone to “overthinking” the differences as they work to calibrate their perceptions and understanding, and correlate this to their understanding of cause/effect.
That makes sense if you assume that none of the BJCP-in-training folks were homebrewers and none of the homebrewers were BJCP-in-training. Kind of a big assumption that does not jibe with my experience – all the BJCP judges I know [in training or otherwise] are homebrewers.
First, I really enjoy your exbeeriments and find the negative results as helpful as the positive ones. (Often enough you turn up a myth we can forget about that makes brew day easier)
The question of “Shitty Palate” is real though and it might increase the power of your results by trying to clean them out. The wine industry really has this figured out and there are people with highly sophisticated abilities to identify wines in blind tasting. In Champagne, the key to consistency is the taster who blends several wines to make the consistently blended cuvee. Similarly the blending of Scotch depends on a taster being able to mix a variety of Scotches to make a consistent product. The fact that beer is carbonated makes a lot of difference. I also make wine, so I can tell you that wine that has not been “Degassed” tasted horrible, (you may well have been served this by a poor but enthusiastic home vintner) but take the same wine, shake it and get the CO2 out and it is entirely different – Carbonation when not only completes beer’s flavour profile, but the carbonic acid covers a lot of faults, which some tasters may be more or less sensitive to.
There are many things which affect sense of taste and smell – genetics – medications – smoking – dry mouth/nose – allergies – colds etc etc – If you were to Google “Poor sense of smell” and “Poor sense of taste” you will find a lot of causes. Perhaps if you were to screen your panel by saying they must qualify by distinguishing between brands of apple juice or apple juice from white grape juice, or apple juice from the same apple juice with an additive ( ? vanilla / a spice etc) would insure that your participants had a minimum ability to distinguish 2 similar beverages on the given day (colds & allergies come and go)
Also, it might be interesting to compare uncarbonated or degassed beer results since carbonation makes such a difference to tastes.
What might be interesting is to identify a panel of those judges that consistently do get it right on triangle tests–BJCP or not, and have those panelist do an additional judging on experiments. Perhaps most of us can’t detect when one thing is different about a beer. But the beer becomes bad to us only when two or three things are bad. If you find that the super-tasters panel detects something, one may well be concerned about the method variation, even if us mere mortals can’t detect it.
https://brulosophy.com/2016/01/21/investigating-the-bad-palates-argument-a-graphical-look-at-xbmt-performance-based-on-experience-level/
I thought the same, data shows that BJCP are no better than non-BJCP, but is there an individual that really stands out everybody else? Maybe he is just lucky to get it right more often than other on xBmt, but who knows? Anyway, the thing is: xBmt has been quite a relief for some of my laziness when brewing, comforting me when I skip some procedures or don’t give too much thought on certain aspects. After all a lot of preconcepts are put in question, breaking the notion that we should do everything “as told”.
Interesting experiment, but probably confounded by unintentionally bad design, since the study failed to take into account degree of brewing and judging experience among the tasters.
First, speaking as a very experienced National level BJCP judge, with plenty of grading, teaching, and judging experience, and will all due respect to the program, not all BJCP judges are created equal.
Being a “provisional” BJCP judge means very little.
The real skill level breaks come at Provisional/Apprentice (i.e., hasn’t yet taken the tasting exam or failed it with a score of 70 score on the written exam, but only moderate levels of competition experience, and “Experienced National/Master/Grand Master” (80-100 on the beer tasting exam, ~>70 score on the written exam, extensive to incredible levels of competition experience).
Furthermore, due to changes in the exam structure over the years, judges who took the test(s) under the new exam structure face much greater challenges than those who passed the exam in the very early days of the program. As a result, they’re better trained with better knowledge of modern styles and modern brewing techniques, and better-developed sensory analysis skills, right out of the gate.
I’d gladly take a shiny new Recognized judge with a freshly-minted score of 90 on the tasting exam and 3-5 years of homebrewing experience over a crusty old-school National judge who hasn’t brewed seriously in 15 years, only likes a limited range of beers, and who has failed to keep up with the times.
Second, the term “Non-BJCP judge” is meaningless since it can describe any degree of gustatory/olfactory sensitivity and sensory analysis training from “first craft beer I’ve ever had” to “Weihenstephaner doctorate yeast lab manager.”
When judging with a non-BJCP ranked judge, the first thing I try to assess is how many batches of beer my partner has brewed (50+ AG batches and 5+ years of brewing experience is best). After that, I ask whether or not they have any formal or informal sensory analysis training (good answers are “cook,” or “wine/beer connoisseur”) and any brewing industry experience (The best answers I’ve gotten so far are “beer sales rep” and “sensory analysis panel director at a legacy national brewery.”)
Very roughly, the break points for measuring brewing experience are 0 batches of beer brewed, 10+, 50+, and 100+ assuming commensurate increases in style knowledge and brewing skill. In HB competitions and sensory analysis panels, the break points are 0, 5, 20, 50, and 100+ competitions judged. Maybe add a level or so for some serious chops not related to homebrewing.
FWIW, I’ve discovered that folks with professional cooking or brewing experience and/or a STEM degree, 50+ batches of HB, and 5+ years HB/craft beer experience are in the catbird seat when it comes to doing well on the BJCP exam and advancing to the higher levels of the BJCP program.
Finally, the triangle test is designed to be a simple test which can be performed by non-specialists. It depends solely on the ability to detect off characteristics rather than describe them and/or determine their causes and cures.
Given that our senses of smell and taste are more or less “hard-wired,” a novice sensory-analysis panel participants with acute sensitivities to certain flavor or aroma compounds will always beat an experienced panel participant whose sensitivities aren’t as acute.
Were the experiment redesigned to take experience levels into account, and the taste criteria were broadened to “hits/misses the style” or “list desirable/undesirable features in this beer,” the results would be more meaningful.
Experiments in consistency of scoring and ability to recognize identical samples of beer over time would probably yield the same results as those done on wine sommeliers, where even Master Sommeliers couldn’t predictably determine same/different wine, and failed to score the same wine consistently, when presented with blind samples.
I have a related thought when reading some of the xbmts:
It would be fun to do an xbmt on 3 or more things that have shown no difference previously, just to see if the combined differences would lead to blind tasters being able to tell a difference. Since differences below a threshold of perception could stack at some point to reach the level to where people could taste a difference (sort of like adding a pinch of salt to 5 gallons, it won’t be perceptible, but if you added 2 pounds of salt it would be obvious, there is some threshold there).
A great professional brewer once told me that they do a lot of very tiny things that many others don’t do when making lager beers and they believe that the difference in their beer is a result of those minor differences stacking up.