Take the Survey

surveyCreating a good survey, one that gives you robust results, takes skill. In a former life I worked for a data analytics company where a team worked on creating surveys for consumers where I gained an appreciation of the skill. I have sincere worked with online surveys. Here are some aspects of survey design to consider.

Sample Size

Imagine you want to know whether Dutch people prefer dark or milk chocolate. The population of the Netherlands is 16.8 million. How many of them do you need to ask?

It turns out, not that many. If I collected data from 1067 people I could be 95% sure that my answer as correct with a margin of error of 3%. That means that if 70% choose milk chocolate the answer in the general population will lie between 67 and 73%. So if you’re a chocolate manufacturer you now know to make most of your flavours based on milk chocolate.

You can be more sure of the answer the further the outcome is from 50%. For the chocolate maker an answer of 47-53% would still be useful, but it’s problematic if you’re predicting political outcomes.

Once upon a time I knew the maths behind these calculations, now I just use an online calculator

Sample Selection

Your sample should reflect your target population as much as possible. This may involve excluding some people from  participating – if you are researching hair care products you don’t need bald men in your sample. For wider. issues it is more likely that you will try to construct a sample that mirrors the total population in terms of gender, race, age, income, family status, religion, location, gender identity and sexuality. That’s not easy. The further you are from your target group the less reliable the outcome of your survey.

Method Bias

Your method of collecting data may introduce bias, if you are collecting data by calling domestic numbers during working hours you exclude working people. If you collect data online you exclude those not on the Internet, and limit respondents to the small group that find your website.

If you are collecting data online you need to control for bots, and you may want to limit the number of times a respondent can answer.

Question construction

To get useful data from your survey you need to construct your questions to be neutral, unambiguous, not leading and specific.

Neutral

“Do you smoke cigarettes?” Is neutral

“Are you a filthy smoker?” Is not.

Unambiguous

It should be clear what information you ar seeking in your question; there are two traps to avoid here.

  • Asking two things in one question

“how friendly and helpful was your customer agent today?” Asks two things, and it’s impossible to decide how to answer if your customer agent solved the problem but was grumpy on the phone with you. You need to split this into two questions.

  • Using negatives

“Do you disagree that raising taxes won’t create jobs?” Is confusing. Rewrite this to ask “Do you agree that…  ?” to simplify it

Avoid Leading Questions

Leading questions contain details that indicate the expected answer.

“When will you start offering free upgrades?” assumes that you will offer free upgrades.

Specific

You will get more accurate and useful data if you ask specifics.

“Do you eat chocolate regularly?” doesn’t tell you much since ‘regularly’ means different things. Much better to ask “how often do you eat chocolate?” and give people a series of ranges to choose from.

What led to this post? A friend posted a strange survey from the President of the United States that breaks every single one of these rules, and a few others.

Here’s the title page of the survey, given that it was sent out after the press conference where the press was repeatedly called “Fake news” the title is clearly priming you to doubt the accountability of the media.

screen-shot-2017-02-20-at-16-45-37

The survey was sent to known Republican supporters, yet the President represents all Americans. The questions are certainly not neutral, and some are just confusing. Here’s the most confusing;
screen-shot-2017-02-21-at-16-05-14

And here’s the most ironic, given that we have already seen that the President uses “alternative facts“, misleading statements and untruths.

screen-shot-2017-02-20-at-16-41-47
 All of which is to say that when the Presidential PR machine talks about having data showing how people don’t trust mainstream media remember his data collection is flawed and the results cannot be trusted.

Images; What?  |  Véronique Debord-Lazaro  |   CC BY-SA 2.0

At the going down of the sun and in the morning We will remember them.

Holocaust MemorialSo in amongst the one hundred and ninety eleven crazy things coming out of the White House was the President’s statement regarding Holocaust Remembrance Day. (I’d link to the official version but it seems to be removed from the White House site). The statement managed to omit any mention of Jews, genocide or anti-semitism. This is not an “honest mistake”, and no Trump spokesperson has since corrected the error. As Deborah Lipstadt wrote in the Atlantic;

The de-Judaization of the Holocaust, as exemplified by the White House statement, is what I term softcore Holocaust denial.

(It’s worth reading the whole of her article)

Indeed the White House statement refers to “innocent people”, here’s a screen grab I took from the video in the Times article linked above.

screen-shot-2017-02-01-at-16-03-28

We all know that Jews were not the only people the Nazis sought to persecute and kill, the list of victims includes Roma, journalists, trade unionists, homosexuals, anarchists, priests, intellectuals, the disabled, and many others.  The total death toll of the Holocaust is usually given as 11 million; roughly equivalent to the combined population of New York and Chicago.  The White House seems to be trying to whitewash its message as more “inclusive”, but it cannot be beyond the abilities of White House staffers to write an inclusive message of remembrance and mention Jews. This was a conscious attempt to remove Jews from Holocaust Remembrance Day.

I write this sitting in Amsterdam, a city with a nickname “Mokum” that comes directly from Yiddish.  The legacy of the World War II Nazi occupation of Amsterdam is visible throughout the city. Of the approximately 80,000 Jews living in Amsterdam in 1941 an estimated 80% were killed in Nazi death camps. I am a five minute walk from the house of the most famous resident to share that history; Anne Frank. The National Holocaust Museum is in this city, housed in a former theatre that the Nazis used as a holding centre for Jews about to be deported. There are monuments and subtle memorials around the city, on the Nieuwe Keizersgracht the names of those removed from their homes on the opposite side of the canal are set into the pavement. This is known as the “Schaduwkade”or Shadow Quay, alongside each name is the name of where they died; Auschwitz, Sobibor, Buchenwald. Places famous for their terror.

The near complete destruction of Amsterdam’s Jewish community is so well etched into the city’s history that I give a silent cheer when I see a menorah, or lights at Hannukah.

There is no forgetting here.

If the White House wants to remind us to remember other victims we can do that.

Under the Third Reich the Nazis;

  •  controlled the media
  • censored the arts
  • burnt books
  • implemented a police state in which arrests were arbitrary.
  • eliminated freedom of speech
  • eliminated freedom of the press
  • removed the functioning judicial system and established a court system that would deliver verdicts as instructed.
  • arrested and killed those who opposed them, often without a fair trail (yeah, that happens in a police state).
  • incarcerated millions of people and forced those they didn’t kill to live in starvation.
  • labelled those incarcerated with a coded triangle to represent their “crime”.
  • saw a mass exodus of Jews and other minorities who felt at risk from every territory they invaded, (and the world did not always accept those refugees, those who worked to get people to safety are remembered as heroes)

So yes Mr Trump and your cronies, on Holocaust Memorial Day we remember what was destroyed under the Nazi regime. All of it.

There will be other memorial days, most countries have their own war memorials and many Jewish communities around the world observe Yom HaShoah. The world is watching the US right now, in horror. I watch and hope that our descendants do not need to create new memorials for the outcomes of this current regime.

 

Post Script; I usually avoid politics and religion, but as I sat down to write a serious post on communication and technology this was the only thing that I could write about. Normal posting will resume next week. Maybe. 

 

Image: Holocaust Memorial 2  |  Ian Southwell  |  CC BY-NC-ND 2.0

In Our Own Bubble

2016June Bubble

The information superhighway took a turn for the worse, we now travel down it in our own comfortable, insulated and isolated bubble.

We can now get any information about any subject at any time online. There’s so much information available that we cannot consume it all, so we make selections. There are more than 500 million tweets per day, but only about 20,000 make it into my twitter timeline, and I only see a subset of those. There are 420,000,00 status updates on Facebook each day, a few hundred of those make it into my feed and I read only a few of those. Then there is Linkedin, YouTube, RSS (yes I still use RSS), and general news outlets.

It’s way too much, so we apply filters. A big part of the filter is who I follow or connect to, in general I follow people who have similar interests or views. As my major news sources are now online I’m unconsciously applying a filter to what news I get.

But there’s another filter being applied that we might not be aware of. The major platforms are also filtering what lands on our screen in our Facebook feed, and (coming soon) our Twitter feed, and our search results. Meaning that Google results are customised based on your search history, your browser, your language choice, your computer. Here’s how it works.

We know that news shapes our world view; in this TED talk Alisa Miller talks about the amount of time given to various news stories. As news organisations reduce costs and dismantle their international news bureaus the international coverage has reduced. She’s speaking from a US perspective, but a similar dynamic plays out in other countries.

If you add together the distortion in what is published, the “customised” news presented in social media and search, and our own filters in choosing who to follow and what to read, it’s fair to say that we’re living in a bubble. Throw into the mix the human tendency for confirmation bias and it’s easy to see that people become increasingly entrenched in their views, both less likely and less willing to hear evidence that doesn’t support their view.

In the last few weeks I’ve seen emotional discussion on politics from both sides of the Atlantic as the US heads into a presidential election later this year and Britain heads to a referendum, dubbed “Brexit“, later this week. It’s not pretty, in both cases it’s a polarised discussion.

It’s because of the level of polarisation, and the anger I’ve seen that I started digging into this. I’ve long thought that social media platforms were poor places for serious discussion for five reasons;

  1. Clutter; Facebook is a blend of photos of cuteness, personal confessions and travel photos. Right next to a photo of my niece walking a tightrope doesn’t seem to be the best place to compare a candidates track record on gun violence.
  2. Godwin’s Law; sooner or later someone is going to drop the N-word. Either of them.
  3. Reading Comprehension; sooner or later someone is going to misunderstand you, perhaps willfully.
  4. Not in Person: in person I could read the person’s body language to pick up on sarcasm or irony (better than in an online discussion)
  5. Asynchronous; nothing worse that waiting hours for a reply to your well-formed attack on a person’s point-of-view. (This should be understood as a tongue-in-cheek comment, see no. 3 above).

So I wasn’t surprised to discover that there is a known phenomenon called the “political spiral of silence“, which means that nuanced, thoughtful points-of-view which are likely to cover some of the middle ground are lost in the noise of social media.

The outcome is a debate so polarised that it’s destructive. How can we change this? What would it take to make your social media and search results more inclusive?

Start by reading opposing views, and having open discussions. We can agree to disagree, can’t we?

Image: Bubbles  | Michael Carson  |  CC BY-NC 2.0