How Not to Do a Survey

Recently, Trump called the mainstream media - by which he evidently means anything that isn't an alt-right, dog-whistling, fake news machine (we'll get to that later) - the "enemy of the American people". (You can imagine how the press reacted - but you don't have to, because Internet.) Besides being frighteningly autocratic, it's dangerously undermining confidence in the Fourth Estate, which exists to keep the people informed so that they can be responsible, informed, good citizens.You cannot have a strong democracy without good information access, which is why we have a free press. It is a necessary - though not sufficient (we'll get to that later) - component of democratic society. We need a free press to mediate between the government and the populace. The press not only keeps us informed of what government officials say and do, but puts information in context, balancing official statements and rulings against expert opinions and commentary. They expose corruption, so that crooked individuals have a harder time taking and abusing power. Are they perfect? No, of course not. Nothing human is perfect (not even Misha Collins, though he comes awfully close, but we're not getting into that today).

So, yes, media accountability is an important issue, and we should be wise news consumers, and how to do that is something I'll address shortly, I promise. But let's start by saying Trump's "Mainstream Media Accountability Survey" is not how it's done.

First of all, you have to figure out which MMAS you're doing. I've found two, and I'm not even looking.I saw one link on a friend's Facebook post. Later, I went looking for it to show MP, and pulled up the other. "Wow, my memory is going wonky - I thought it was slightly different," I said to myself. My memory is fine; there are multiples. Here's ONE. Here's TWO.  How many others are there? I don't know - maybe there's only two, maybe there's lots. Why would there be multiples at all? Maybe I'm too cynical, but I'm guessing it's so they can cherry-pick the results to match what they want to say.

Then there are the questions themselves.Most are ridiculously misleading - "Do you agree with the President’s decision to break with tradition by giving lesser known reporters and bloggers the chance to ask the White House Press Secretary questions?" states one of the surveys. Well, what's wrong with giving "lesser known reporters and bloggers" the same opportunities as the Big Fish? Is anyone honestly going to say, "No! Only the known names should be involved!"

But context is important. Who are these Small Fry Intrepid Journalists? Breitbart and the like. In fact, in the other survey, we're asked:








Do you know what all those sites have in common? They're all alt-right, discredited, clickbaity, dog-whistling, and/or fake news nonsense. (See, I told you we'd get to it.) Six of the eight listed are on Melissa Zimdars's False, Misleading, Clickbait-y, and/or Satirical “News” Sources. The other two aren't - yet - but both are in the same general cluster, as far as what they produce. Not a single one is known for fair, balanced, in-depth, actual news coverage. Since the rest of the survey goes on to attack reputable news sources, I can only assume this question is actually product placement. Do I think actual journalists from reputable producers should be shushed in order to give hacks from National Review the chance to spread misinformation? Absolutely not!

Then there are the unanswerable questions - or, rather, the answers can be construed to mean whatever the person tabulating the data wants them to mean. For instance, one of the surveys states, "The media has not done its due diligence to expose ObamaCare’s many failures." Your choices in answer are 'yes', 'no', and 'no opinion'. (Unlike the other survey, this one does not let you given your own answer on anything.) So, what do those answers mean? 'Yes, the media hasn't done so' seems obvious - but it could also be taken as 'Yes, it has - the media has actually covered the flaws in the ACA'. 'No' could mean, 'No, the media hasn't done its due diligence!' or 'No, I don't agree with this statement at all! ObamaCare is awesome!' or even 'No, the media has done a fair and exacting job covering flaws in the ACA!'

Then there are statements that are misleading and unanswerable, like "Were you aware that a poll was released revealing that a majority of Americans actually supported President Trump's temporary restriction executive order?" What are the possible answers to that? 'Yes - I was aware of such a poll! It's a disgrace!' or 'Yeah, but there are lots more credible polls that show the majority didn't support it.' Or even, 'Yes - but that doesn't mean it was right or constitutional.' Or there's 'No, I wasn't aware! Let me be outraged!' or 'No, and I don't believe that poll exists or is accurate if it does.'

A couple of the questions on one survey are open-ended, but both are just asking for a list of what sources you use; you can, on this survey (which is apparently intended only for Republicans), input data for the 'other' answers on most of the questions. The other survey has no open-ended questions and does not allow you to answer with clarifications or qualifications.

The questions are all biased and inflammatory, and the surveys are obviously intended to enrage true believers and denigrate the press. One of the surveys is littered with references to "our Party," meaning the survey intentionally singled out Republicans - but would the gathered data have been reported that way? Or will the multiple surveys be conflated, so that the data swings far Right without acknowledgement of bias?

There is zero attempt to make the survey representative of the American people - not just with political demographics, either. Very little demographic data is gathered - only a zip code - so there can be no critical appraisal of how answers (such as they are presented) break down according to age, class, ethnicity, etc. - or how biased the data may unintentionally be due to these and like factors.

Moreover, and a bigger problem in my opinion, is the lack of anonymity. In order for your answers to count, you have to enter your name and email address and click either 'Record My Vote!' on the in-your-face Republican version or 'Submit' for the other. What happens then? Are you sorted out to different lists based on your answers? That seems conspiracy theory-ish. More likely, you're added to a mailing list of some sort, and I don't want to be on a mailing list of some sort. So, this tactic would discourage people who don't particularly want to hear any more propaganda, even if it is disguised as a survey. If you want to read more about this ridiculous survey from a reputable news site (I'm not a news site, and I'm not sure how reputable I am, either), I suggest NPR's coverage: The Trump Media Survey Is Phenomenally Biased. It's Also Useful

We the People, on the other hand, need to learn how to sift out actual news from propaganda, especially when we have an administration so devoted to pushing propaganda. Like I said earlier (see, I get back to everything eventually), a free press, while necessary for democratic society building and maintenance, is not sufficient - the people also have to be good at consuming the news. Even reputable, generally trustworthy sites can sometimes get it wrong. (Have you watched The Newsroom? You should definitely watch The Newsroom. No Misha, but, hey - can't be perfect.) It's our duty as good citizens to try and learn as much as possible, so we can be informed participants in a democratic society. Voting without making sure we have good information is a lot like going to pick up a pet without learning how to take care of it beforehand - it's your fish, now, and if you kill it, you're responsible. (One big difference - generally, a dead fish doesn't take you down with it.) Reading multiple reputable news sources and taking the time to think critically about what they're saying is important - you aren't a good citizen, otherwise. Learning how to tell what is and is not a good source is also important. I'm not going to teach you how to do that, since I've already built a wall of text here, and my fingers are tired of typing, and you can take this whole course from Poynter News University, which is awesome! It's free, it'll only take you an hour or two to go through, and it will help you be a better news consumer and producer. Go forth and do the thing. (For a more advanced course that should be available soon, check this out: In age of alternative facts, a college course on calling out scientific crap.)

Oh, also, here's how you do a survey. Someone pass this on to the White House, yeah?

Meanwhile, as a reward for digging through all my lecture above, here's a pretty picture:



Links!

Popular posts from this blog

Pi-Tastic!

How Small Is His...Er, Hand?

Where Are We Going, And Why Are We In This Handbasket?