Degrees of separation

New Scientist tells us about Facebook’s analysis of the friend relationships in their social network. Only four degrees of separation, says Facebook, goes the New Scientist headline. Here’s their summary:

A few months ago, we reported that a Yahoo team planned to test the six degrees of separation theory on Facebook. Now, Facebook’s own data team has beat them to the punch, proving that most Facebook users are only separated by four degrees.

Facebook researchers pored through the records of all 721 million active users, who collectively have designated 69 billion "friendships" among them. The number of friends differs widely. Some users have designated only a single friend, probably the person who persuaded them to join Facebook. Others have accumulated thousands. The median is about 100.

To test the six degrees theory, the Facebook researchers systematically tested how many friend connections they needed to link any two users. Globally, they found a sharp peak at five hops, meaning that most pairs of Facebook users could be connected through four intermediate people also on Facebook (92 per cent). Paths were even shorter within a single country, typically involving only three other people, even in large countries such as the US.

The world, they conclude, just became a little smaller.

Well, maybe. There are a lot of things at play here, and it’s not simple. It is interesting, and it’s worth continuing to play with the data, but it’s not simple.

They’re studying a specific collection of people, who are already connected in a particular way: they use Facebook. That gives us a situation where part of the conclusion is built right into the study. To use the Kevin Bacon comparison, if we just look at movie actors, we’ll find closer connections to Mr Bacon than in the world at large. Perhaps within the community of movie actors, everyone’s within, say, four degrees of separation from Kevin Bacon. I don’t know any people in the movie industry directly, but I know people who do, so there’s two additional degrees to get to me. We can’t look at a particular community of people and generalize it to those outside that community.

There’s also a different model of friends on Facebook, compared with how acquaintance works in the real world. For some people, they’re similar, of course, but many Facebook users have lots of friends whom they don’t actually know. Sometimes they know them through Facebook or other online systems, and sometimes they don’t know them at all. Promiscuous friending might or might not be a bad thing, depending upon what one wants to use one’s Facebook identity for, but it skews studies like this, in any case.

People would play with similar things in the real-life six degrees game. Reading a book by my favourite author doesn’t count, but if I passed him on the street in New York City, does that qualify? What about if we went into the same building? If he held the door for me? If I went to his book signing, and he shook my hand and signed my copy of his book? Facebook puts a big e-wrinkle on that discussion.

But then, too, it’s clear that with blogs and tweets and social networking, we have changed the way we interconnect and interact, and we have changed how we look at being acquainted with people. I know people from the comments in these pages, and from my reading and commenting on other blogs. Yes, I definitely know them, and some to the point where I call them friends in the older, pre-social-network sense. But some I’ve never met face to face, nor talked with by voice.

So, yes, the world probably is a little smaller than it used to be. It didn’t just get that way suddenly, of course; it’s been moving in that direction for a while. Everything from telephones and airplanes to computers and the Internet have been taking us there.

Equal-Pay Day

Today, 12 April 2011, is Equal-Pay Day in the U.S. If you took the median-salary American man and the median-salary woman, and started paying them both on the first of 2010, today is the day when the woman will have finally earned what the man took in through 31 December, about 14 weeks ago.

Of course, it’s not that simple. You can’t just take any man and any woman and make that comparison. The figure that’s used for this is the median income: take all the men’s annual salaries, list them in order of lowest to highest, then pick the one in the middle. Do the same for women’s salaries. Compare. The median of the women’s salaries is about 78% of the median of the men’s. We could use the average (mean) instead of the median, but for these sorts of economic comparisons it’s typically the median that’s used, because it doesn’t suffer from skewing by the extremes at the edges.

The problem is that the majority of the gap comes from the fact that men and women are not equally represented in all the different jobs... and the jobs that employ primarily men just so happen to pay more than the ones that employ primarily women. I can’t imagine how that happened, but, well, there it is. Nurses earn less than doctors. Beauticians earn less than plumbers. Teachers earn less than corporate executives. And so on.

And it doesn’t stop there: what about college-educated women? What about those with PhDs? Because another fact is that more women than men are finishing college, these days, and more women than men are completing PhD programs. Doesn’t that fix it?

No. For one thing, when we look at the fields that women are getting degrees in, we find the same thing: the fields that attract women more tend to be the less lucrative ones.

But also, when we break it down by field we still find differences. In April of 2007, the American Association of University Women released a study titled Behind the Pay Gap (PDF). The study showed that female biological scientists earn 75% of what their male colleagues do. In mathematics, the figure is 76%; in psychology, 86%. Women in engineering are almost there: they earn 95% of what the men do. But less than 20% of the engineering majors are women.

The other argument for why there’s a pay gap is that women and men make different decisions about their lives. Women choose motherhood, a bigger hit against career advancement and salary opportunities than fatherhood. More women work part time. And so on.

The AAUW study looked at that. They controlled for those decisions, and they compared men and women who really could be reasonably compared. They looked at people in the same fields, at the same schools, with the same grades. They considered those of the same race, the same socio-economic status, the same family situations. They didn’t just compare apples to apples; they compared, as economist Heather Boushey puts it, Granny Smith apples to Granny Smith apples.

And they found that even in that case, there’s an unexplained pay gap of 5% the year after college, which increases to 12% ten years later. From the study:

The pay gap between female and male college graduates cannot be fully accounted for by factors known to affect wages, such as experience (including work hours), training, education, and personal characteristics. Gender pay discrimination can be overt or it can be subtle. It is difficult to document because someone’s gender is usually easily identified by name, voice, or appearance. The only way to discover discrimination is to eliminate the other possible explanations. In this analysis the portion of the pay gap that remains unexplained after all other factors are taken into account is 5 percent one year after graduation and 12 percent 10 years after graduation. These unexplained gaps are evidence of discrimination, which remains a serious problem for women in the work force.

It has gotten better: if today the general pay gap is about 20%, 15 years ago it was 25%, and 30 years ago, 35%. The improvement is good news.

But the speed of the improvement is not. The disparity of pay between male-dominated fields and female-dominated ones is not. The gap in pay between highly trained men and women in the same field is not. And that unexplained 5-to-12 percent is certainly not.

Let’s keep pushing that date back, and look for the year when equal-pay day is December 31^st.

Follow-up on placebos

I wanted to do a follow-up to Friday’s post about a ridiculously flawed study of the effect of placebos. I have two follow-up points.

Point 1: The study is not blinded, but that is not a problem, as I see it. There are two types of blinding used in studies:

1. Blinding relative to the subjects: the subjects don’t know which group they’re in (and sometimes don’t know specifically what’s being studied). This reduces the influence of the subjects’ expectations and biases on the results.
2. Blinding relative to the researchers: the researchers don’t know which group a given subject is in. This reduces effects that the researchers’ behaviour toward the subjects will have on the results.

For this study, the whole point was not to blind it to the subjects. And since the subjects are either being treated or not, there’s nothing to blind relative to the researchers, either. The problem with the study isn’t the lack of blinding, but the bias that the instructions to the subjects provides.

Point 2: I see a reasonable way to evaluate what these researchers want to evaluate, which fixes the bias problem — or, more accurately, which makes that bias part of the study itself. It involves what I’ll call half blinding relative to the researchers.

Get, say, 300 subjects, and divide them into three groups of 100. Group A gets no treatment, just as one group got in the existing study. Groups B and C each get identical bottles labelled PLACEBO. Each bottle contains identical pills. Each bottle contains an instruction sheet that only the subjects see, and each subject is told to read his sheet and not to discuss with anyone what it says.

The instruction sheets for group B would say, This bottle contains placebo pills made of an inert substance, like sugar pills, that have been shown in clinical studies to produce significant improvement in IBS symptoms through mind-body self-healing processes.

The instruction sheets for group C would say, This bottle contains placebo pills made of an inert substance, like sugar pills, that have no effect whatsoever. They will not affect your symptoms in any way.

Now, if groups B and C have similar results, we can legitimately say that placebos have their effect even when the subjects know they’re getting them. But if, as I suspect, group C has results comparable to those of group A, then what we’ve shown is that telling people the placebos are effective is the same as not calling them placebos — setting up the expectation of effectiveness is what gives the results.

If you want to call that mind-body self-healing processes, that’s fine with me. Well, almost fine: it’s not healing; it’s symptom relief. That your mind can help you feel better is no surprise. But it’s not going to heal your cancer, repair your liver damage, or kill viruses. Unless, perhaps, someone can show, with a properly designed study, that it can.

Someone’s high on placebos

On Wednesday, this paper, published in PLoS ONE, hit the popular news in the medicine/science category, with articles such as this one from MedPage Today and this, from Reuters. The headlines are consistent, implying that the study has shown that the placebo effect works even when patients know that they’re getting placebos.

From the MedPage article:

While this may seem counterintuitive, the results suggest that physicians don’t have to resort to deception to harness the power of the placebo effect, Kaptchuk explained in an interview.

Conventional wisdom says that in order for a placebo to work you have to convince the patient that they are taking a real drug, he told MedPage Today.

However, the brain appears to respond to placebo in a way that conditions like IBS are susceptible to, Kaptchuk noted.

Counterintuitive, indeed, and enough so that it merits some serious scrutiny. My first thought was that the headlines are misrepresenting the study and the claims of the researchers, a common problem in the reporting of research in the popular press. But it seems that they are, indeed, reporting exactly what the researchers are saying, here.

I’ll note that the paper appears in a Public Library of Science (PLoS) journal, rather than in some other, more respected medical journal (such as NEJM or BMJ). I’ll also note that the lead researcher, Ted Kaptchuk, is associated with Harvard’s Division for Research and Education in Complementary and Integrative Medical Therapies, a connection the press just puts forth as Harvard, downplaying the fact that he does complementary medicine.

Those aren’t reasons to ignore his research or his results, of course. But they are points that should make us ask questions. On the other hand, we should ask questions with any study; that’s what science is about.

Now, the study is on IBS, which is a very subjective, catch-all condition of unknown etiology. They mention in the article that they expect this effect to work also with conditions such as fibromyalgia and chronic pain — also subjective conditions of unknown etiology — and depression and anxiety, more highly subjective stuff. And the study itself is entirely subjective, using self reporting exclusively, and not measuring anything.

That could be OK. If we’re looking for cures, we need to measure; if we’re looking for symptom relief, well, if you think you feel better, then you feel better. Where I have to scratch my head, though, is in wondering what their hypothesis was.

When you go into a study, you go in with a hypothesis, which your study might support or disprove. Even if it seems like you’re just wildly trying any drug that might work, you have a hypothesis: This drug might work. What’s the hypothesis, here? Placebos might work even if the patients know they’re placebos, is a valid hypothesis, I suppose, but to whom would it occur to even try that?

The answer shows up in the study: it would occur to people who think they can show mind-body self-healing processes. OK... again, a valid thing to consider. But that’s where we get into some problems.

We start with some very typical problems with medical studies... again, from MedPage:

Limitations of the trial included a relatively small sample size, too short duration to examine long-term effects, and the possibility of self-selection bias in that the trial may have selectively attracted IBS patients interested in mind-body interventions.

Many, many studies are too small, too brief, and suffer from various sorts of selection bias, and this one is no different. But to get the real kicker, we have to go into the paper itself and see how the placebos were presented to the patients:

Patients were randomized to either open-label placebo pills presented as placebo pills made of an inert substance, like sugar pills, that have been shown in clinical studies to produce significant improvement in IBS symptoms through mind-body self-healing processes or no-treatment controls with the same quality of interaction with providers.

Read that again. The patients were not told just that they were getting placebos. They were not told that what they were getting is an inert substance with no medical effect. They were told that what they were getting has been shown in clinical studies to produce significant improvement in IBS symptoms.

In other words, these pills are having exactly the same placebo effect as is well documented in pretty much every other medical study that involves placebos. And, to put the point forth directly, far from making this deception-free, they are deceiving the patients in the same way patients are deceived in every other administration of placebos.

Let’s line it all up:

1. You recruit people who have a condition that makes them feel horrible, that no one can tell them the cause of, and that no one has effective treatment for.
2. You give them a pill and you tell them that this has been shown to provide significant improvement for their condition.
3. They report that the pills did, indeed make them feel better.
4. You claim that you did not deceive them, because you told them they were getting placebos.

Bullshit. Whether or not you put the label placebo on it, you told them they were getting effective treatment. That biased the outcome, and the result should be no surprise to anyone. And it says... nothing. This study is worthless garbage.

Bias is bad for science

The New York Times recently published an article about bias against women and minorities in science fields (and schooling). I’ve written about this before, and that was about a study from 2004. We’re not getting much better at this — or, if we are, it’s not fast enough.

Consider this:

The report found ample evidence of continuing cultural bias. One study of postdoctoral applicants, for example, found that women had to publish 3 more papers in prestigious journals, or 20 more in less-known publications, to be judged as productive as male applicants.

And this:

In a separate survey of 1,200 female and minority chemists and chemical engineers by Campos Inc., for the Bayer Corporation, two-thirds cited the persistent stereotype that STEM fields [Science, Technology, Engineering, and Mathematics] are not for girls or minorities as a leading contributor to their underrepresentation.

Other studies have shown that papers written by women are more likely to be accepted by journals if they list their authorship by initials, rather than using a feminine first name. And, of course, women still are paid, on average, only a little more than ¾ of what men are.

We weren’t raised by wolves; why can’t we fix this? Smart, successful, technically adept women do not pose a threat to men. Quite the opposite, they add to the pool of qualified developers, researchers, and educators. And the same goes for minorities. We need them. We should be encouraging them, instead of behaving in ways such as this:

Many in the Bayer survey, also being released Monday, said they had been discouraged from going into their field in college, most often by a professor.

“My professors were not that excited to see me in their classes,” said Mae C. Jemison, a chemical engineer and the first African-American female astronaut, who works with Bayer’s science literacy project. “When I would ask a question, they would just look at me like, ‘Why are you asking that?’ But when a white boy down the row would ask the very same question, they’d say ‘astute observation.’ ”

A few years ago, there was a series of advertisements about diversity — public service announcements, really — that aired on PBS. One depicted a job applicant ending what appeared to be a pleasant and successful interview. The applicant was clearly “of colour”, and he was talking with two white men. After he left the room, the older man crumpled his application and tossed it in the waste bin. “I think we have enough ‘diversity’ around here, don’t you?”, he said.

The younger man reached into the bin, took the wad out, flattened it on the desk, looked at the older man, and said, “No. I don’t.”

Let’s move in that direction now, and let’s all help. As the tag line from the ad goes: One voice can make a difference.

I am God’s co-pilot

A new study gives us an unsurprising result: that when believers are asked to characterize God’s political, social, and moral views, what the respondents say God thinks closely matches their own opinions. I can’t find the paper on the web site of the author nor that of the journal, but according to New Scientist:

“Intuiting God’s beliefs on important issues may not produce an independent guide, but may instead serve as an echo chamber to validate and justify one’s own beliefs,” writes a team led by Nicholas Epley of the University of Chicago in Proceedings of the National Academy of Sciences.

The first thing that comes to mind, here, is that it’s obvious, but backward: that it’s not that we attribute our views to God, but that we derive our views from our parents and our religious training, so it’s natural that our own moral and socio-political opinions match what we think God wants.

Of course, the researchers looked into that, and that’s the part that makes this study interesting:

Next, the team asked another group of volunteers to undertake tasks designed to soften their existing views, such as preparing speeches on the death penalty in which they had to take the opposite view to their own. They found that this led to shifts in the beliefs attributed to God, but not in those attributed to other people.

Play that back: when people were steered toward softening their own views, they correspondingly softened what they thought God’s views were. That looks like people are using God to support what they, themselves think, and that’s much less obvious (if still unsurprising to some of us). The researchers also used brain scans to collect more (and less subjective) evidence of this effect.

Of course, we see this taken to extremes all the time, when preachers and other leaders with strong religious leanings claim that God supports whatever it is that they’re trying to push. We see that, and we often recognize it for the manipulation that it is. But do we really understand that people are doing that all the time, every day, without even knowing it themselves?

The majority of the subjects of this study were Christian, and all “professed beliefs in an Abrahamic God.” I’d like to see whether there’s any difference with Hindus, Buddhists, and followers of other religions and philosophies. I suspect so: we’re very strongly inclined to imagine that what we, ourselves believe is closely tied to the “correctness” of the universe around us. It seems a form of self-validation that likely keeps us emotionally centered.

This also appears to be related to what researchers have seen from people who believe in reincarnation: the past lives they describe closely match what they already believe about past lives. Those who believe, for example, that your past lives can only be as non-human animals only remember past lives as non-human animals. Those who believe otherwise remember past lives as humans. It’s all spiritual confirmation bias.

Tasers again: this time, a study

New Scientist reports on a study that shows that “[u]sing a Taser to subdue a violent suspect is safer than police batons and fists.”

The team examined over 24,000 cases where police had used force, including almost 5500 incidents involving a Taser. After controlling for factors such as the amount of resistance shown by the suspect, they found that Taser use reduced the overall risk of injury by 65 per cent.

Despite the cases of deaths and serious injuries from Tasers, I have no doubt of the conclusion here: beatings are obviously likely to cause injury or death also, and are probably harder to keep under control.

But the key phrase here is “to subdue a violent suspect.” Not to coerce an uncooperative person. Not to quiet someone who’s being loud or boorish. And certainly not to punish someone who has, well, you know, just annoyed the officer, nor because the officer can’t figure out how to non-violently subdue a ten-year-old girl.

We see Tasers used over and over for those other purposes, purposes for which they aren’t meant. I would hate to see someone read an article about this study and conclude that such uses are, therefore, justified.

Is carrying a gun risky?

I honestly do wish the press — even the science press — would stop reporting on preliminary studies and incomplete results. Or maybe the fault is with the scientists who talk to the press about such results, and the publicity departments of their research organizations, which put out press releases.

New Scientist, which offers a mixed bag of good science reporting and stuff that the editors should have thrown in the rubbish bin, has just given us one in the latter category: “Carrying a gun increases risk of getting shot and killed”.

Packing heat may backfire. People who carry guns are far likelier to get shot — and killed — than those who are unarmed, a study of shooting victims in Philadelphia, Pennsylvania, has found.

It would be impractical — not to say unethical — to randomly assign volunteers to carry a gun or not and see what happens. So Charles Branas’s team at the University of Pennsylvania analysed 677 shootings over two-and-a-half years to discover whether victims were carrying at the time, and compared them to other Philly residents of similar age, sex and ethnicity. The team also accounted for other potentially confounding differences, such as the socioeconomic status of their neighbourhood.

The result? Well:

Overall, Branas’s study found that people who carried guns were 4.5 times as likely to be shot and 4.2 times as likely to get killed compared with unarmed citizens. When the team looked at shootings in which victims had a chance to defend themselves, their odds of getting shot were even higher.

Anyone staring at these pages for a while will know that I dislike guns. I’d love nothing more than to be able to take that result at face value, and to quote it far and wide. I’d love to have a definitive study showing such statistics.

This is not that study, and these results are useful only to prompt further study. As they stand, we can’t conclude anything from them.

The article itself does point out some of the problems, but many readers will miss them. First, this is not a randomized trial, nor even a review of other scientific work. They started with people who were shot. The article points out that practicality and ethics make it difficult to assign people to groups, but perhaps a study that selected people at random and then looked at what happened to them would have a better chance. As it is, the methodology here makes confirmation bias likely.

Second, while they attempted to control for factors such as age, sex, ethnic background, and socioeconomic status, they have not controlled for some major factors, not least of which involve the attitudes and behaviours of the subjects. We can’t say this enough: correlation does not imply causation. Even if we accept that they have shown a high correlation between carrying a gun and being shot, there is no sense in which they’ve shown any cause. Again, the article does note that, but not in so many words, and only in passing.

It’s entirely possible, for example, that the causation is exactly the other way around. It’s possible — I have no data to support this; it’s just hypothetical — that the people who were carrying guns were doing so because they often go places where they’re likely to be shot.

Third, they studied a city in the northeastern U.S., which has a certain view of guns. The results could be very different in, say, Dallas, where the gun culture is very different. Geographically diverse studies would be needed to account for regional differences in how we think about guns, and in the laws that regulate them.

None of this is to say that the work isn’t good, isn’t useful. It’s just that we can’t deduce anything directly from it. The value of studies like this is that they uncover apparent correlations and show us things that we can then go off and study more rigorously.

Unfortunately, it’s likely that many people will read reports of studies like this and won’t understand the limitations on interpreting the results. And in this case, trumpeting this study at the NRA’s gates would be a mistake, because its so easy to shoot it down (if you’ll excuse the metaphor).

On the other hand, I look forward to future studies that pursue the questions this one raises.

Anti-studies

In the issue dated 10 October, Science News reports on a study that suggests that peer reviewers prefer positive results:

Peer reviewers for biomedical journals preferentially rate manuscripts with positive health outcomes as better, a new study reports.

Now, at first blush this might seem like a “Duh!” moment, but it’s not. We obviously would like to see positive results when we’re studying a new medicine, but there’s a great deal of value in publishing negative results as well. It tells us what medicines don’t work. It tells researchers what direction to take, exposing some of the blind alleys. It’s critically important information, and that’s true in other fields, as well.

Consider studies of acupuncture, of astrology, and so on. There are a great many people who think those work — certainly enough to warrant a serious look with controlled studies. And controlled studies have been done. They show that astrology doesn’t work at all. They show that acupuncture works as a placebo: “fake” acupuncture is as effective as “real” acupuncture. These are useful results.

Consider herbal remedies: we know that herbs do have active substances in them, but there are lots of claims and we’d like to sort them out. Does Ginkgo biloba help with dementia? Does Echinacea reduce cold infections? Is Valerian effective as a sedative? Does St John’s wort work against depression? Studies say no, no, maybe, and yes, respectively. And the “no” results are arguably just as important as the others.

But it’s not just in medicine that we see a preference for favourable results. It’s true in computer research, as well. In fact, while it would be quite important to see, say, methods of spam filtering that seemed like good ideas but fell flat, we rarely see people submitting them, and I’m quite certain that reviewers would lean toward rejecting them in favour of “more interesting” papers with “better” results.

Probably one significant reason for the lack of submissions is that people aren’t eager to document “failure”. That means that it’s incumbent upon the review and publication system to actively encourage the publication of good ideas that didn’t work out. The “good ideas” part of it is key, of course: there’s plenty of work that went nowhere, but that wasn’t promising to begin with. There’s limited value in publishing that stuff.

On the other side, reviewers should be looking at the value a study has for teaching or for directing future work, and for confirming or overturning common theories. A paper that shows definitively that something we expected to work doesn’t... is arguably more important than one with a partial result in the expected direction.

Public misunderstanding of studies

Over at Bioephemera, Jessica Palmer agree with Language Log’s Mark Liberman in his admonition against the use of “generic plurals” in science reporting. Language Log:

This would lead us to avoid statements like “men are happier than women”, or “boys don’t respond to sounds as rapidly as do girls”, or “Asians have a more collectivist mentality than Europeans do"” — or “the brains of violent criminals are physically and functionally different from the rest of us”. At least, we should avoid this way of talking about the results of scientific investigations.

The reason? Most members of the general public don’t understand statistical-distribution talk, and instead tend to interpret such statements as expressing general (and essential) properties of the groups involved. This is especially true when the statements express the conclusions of an apparently authoritative scientific study, rather than merely someone’s personal opinion, which is easy to discount.

The problem, in case you don’t see it from what’s quoted above, is this (I’m going to make some details up, just to give an example):

Suppose some researchers do a study in which they ask people how happy they are, on a scale of 1 to 10. Suppose that they ask 50 men and 50 women, and the average happiness rating for the men is 7.3, while the average score for the women is 7.1. Now suppose that the study is reported in the news with the statement that “men are happier than women.”

Or let’s be even more straightforward: suppose the 50 men and 50 women are simply asked, “On the whole, are you happy?” 37 of the men and 36 of the women say, “Yes.” And the newspapers report that, according to a recent study, “men are happier than women.”

Of course, George reads that over his morning coffee, and says, “Hey, Martha. It says here that I’m happier than you. Ha! I always knew there was something wrong. Maybe you need some of that Prozac stuff.”

But we can’t generalize a finding based on average aspects of a group... to particular individuals in the general population. Martha may be far happier than George, and the study doesn’t say otherwise. George just doesn’t understand.

Of course, the problem isn’t limited to generic plurals with no statistics behind them. We could report that a study shows that “men are 50% more likely than women to get into traffic accidents,” but that wouldn’t mean that I am 50% more likely, just because I’m a man. There are other reasons, which the study might or might not go into, that are the causes of the difference, and the study just shows one correlation.

So it’s important to word these reports in a way that doesn’t invite that sort of misinterpretation. It’s important for a number of reasons:

• The media already often get the details wrong in reporting scientific studies. It makes it worse to compound that with confusing reporting.
• The media often highlight the wrong bits, in efforts to get catchy headlines and “interesting” copy.
• Readers don’t understand statistics, and misinterpretation is likely even when the stats are there. Don’t make it worse by eliminating them.
• Readers are prone to generalize results beyond what’s valid, and they’ll likely apply a group trent to specific individuals, as in the example above.
• Readers don’t understand the limitations of studies. Reporters should try to talk about one or two key limitations.

The first two are nicely demonstrated by the British newspaper The Telegraph. Back in June, they reported on work done by a student, Sophia Shaw, at the University of Leicester. The preliminary findings, according to Ms Shaw: “We can see from the results that sexually experienced men are more likely to coerce women in sexual situations; even more so if they believe the women to be sexually experienced.” But the Telegraph reported (the article has since been removed from their web site after the criticism of it, but you can read discussion of it) that the work “found that the skimpier the dress and the more outgoing the woman, the less likely a man was to take no for an answer.”

In The Telegraph’s competition, The Guardian, Ben Goldacre seemed to enjoy tearing the former’s report apart:

Women who drink alcohol, wear short skirts and are outgoing are more likely to be raped? “This is completely inaccurate,” Shaw said. “We found no difference whatsoever. The alcohol thing is also completely wrong: if anything, we found that men reported they were willing to go further with women who are completely sober.”

We often say that the public needs to be better educated with respect to science and critical thinking. This is a good place to start... and the news media need to be among the educators.

Is your wallet safe?

Recently, BoingBoing pointed us to an interesting article about a study involving “lost” wallets:

Hundreds of wallets were planted on the streets of Edinburgh by psychologists last year. Perhaps surprisingly, nearly half of the 240 wallets were posted back. But there was a twist.

Richard Wiseman, a psychologist, and his team inserted one of four photographs behind a clear plastic window inside, showing either a smiling baby, a cute puppy, a happy family or a contented elderly couple. Some wallets had no image and some had charity papers inside.

The results surprised me. First, 42% of the wallets were returned — surprising enough to someone from the New York City area. But then an amazing 88% of the wallets containing the baby picture came back, and only 15% of the ones with no picture at all did. That’s a hell of a difference! (It’s also important to note that none of them contained money or credit cards.)

The researchers’ hypothesis for the reason for the difference is evolutionary — an innate predisposition to protect children. They don’t explain the difference in return rates for the wallets with the other sorts of photos, though. In any case, they offer some advice:

Whatever the scientific explanation, the practical message is clear, said Dr Wiseman. “If you want to increase the chances of your wallet being returned if lost, obtain a photograph of the cutest baby you can find, and ensure that it is prominently displayed,” he said.

That’s amusing enough, I guess. But the whole study just cries out for a bunch of follow-up questions. Of course, there’s the obvious one of varying the ethnic identity of the people in the photos. Make the baby white, black, south-Asian, east-Asian, and so on, and see what changes. Make the family photo show different races, different types of dress. Include interracial and same-sex couples. Within that last, does a pair of men inspire a different rate of return from a pair of women?

Would the presence of money change things? Would the money outweigh all the photos equally? Would there be a difference between Edinburgh and, say, New York City? (I should think so, but....) Assuming that there is, would the relative return rates still be the same, nonetheless? Maybe not; maybe New Yorkers prefer puppies or grandparents to babies.

I’d then want to try it with different sorts of work IDs. Strew some wallets about that indicate that the owner is a teacher, a lawyer, a plumber, a car salesman, a firefighter, a waiter, or a banker. What do you think? I wonder where a computer geek falls.

Or try different medical specialties. One might think an obstetrician’s wallet would be more likely to be returned than that of a proctologist. How would cardiology compare with oncology? Would a neurosurgeon do better or worse than a dermatologist in the wallet-borne sympathy vote?

I think this is a fascinating way to study our unconscious preferences and prejudices, and I hope this is just the start of a batch of these studies worldwide.

Well, yesterday we wanted to find out just how honest the people in our audience were. Are they as honest as they like to say? We chose a lady from the audience and gave her ten $10 bills, each one in a separate envelope, each one marked with her name and address. We asked her to drop them around the studio to see how many would be returned. Well, we found out how honest people are: the lady cut out with the hundred dollars.

— George Carlin, “Daytime Television”
from “Take-Offs and Put-Ons” (1967)

Confidence!

It’s always been clear that we like it when people are firm, not wishy-washy. John Kerry lost the 2004 presidential election in large part because he was derided for having changed his vote.[1] Where sense might tell us that changing your mind after serious consideration of new information is a good thing, and a sign of intelligence and reason, we have something working against that, which made the label “Flip-Flop!” stick like a “Kick me!” sign on his back.

This isn’t news, of course, and isn’t, in itself, interesting. What is interesting is that we have studies (you knew we would) that show that people consistently prefer confidence even when it’s consistently wrong:

The research, by Don Moore of Carnegie Mellon University in Pittsburgh, Pennsylvania, shows that we prefer advice from a confident source, even to the point that we are willing to forgive a poor track record. Moore argues that in competitive situations, this can drive those offering advice to increasingly exaggerate how sure they are. And it spells bad news for scientists who try to be honest about gaps in their knowledge.

Indeed, the studies explain our real-work experience with leaders in politics and business who make strong, confident statements about things they know nothing about; who make wild, confident predictions about things they can’t possibly predict with any hope of accuracy; who make bold, confident decisions, and then stand by them even when they turn into disasters.

Mission accomplished!

The problem, though, is that honesty is considered weakness, and serious consideration of facts becomes a bad thing. And that’s dangerous. It’s also how the sorts of people who are firmly confident in fantasy garner support. Religious fanatics, conspiracy theorists, denialists of AIDS and climate change and the Holocaust, are all vehemently confident. Scientists, at least when we’re being honest about it, must always admit to some amount of uncertainty, however small. We don’t know everything there is to know about evolution, about viruses, about cancer, or about the global climate... but when we admit to a gap in what we know, we fall victim to this effect. We are uncertain, so we lose ground.

An example of this is something I quoted in these pages last fall, talking about the Large Hadron Collider. The fear-mongers have latched onto a crazy notion that the LHC will create a black hole that will destroy the world. Scientists are as sure as we can be that this is ridiculous, and, “No, it can’t happen,” would not be an unreasonable way to answer the question. And, yet, physicist Janna Levin answers by saying, “Well, it’s interesting, ’cause you can never say ‘never,’ actually, and the best things you can say are that it’s incredibly, ridiculously, extremely unlikely that anything like that can happen. [...] But can I say it’s a physical impossibility? I can not.”

I like the way New Scientist ends the article, with Dr Moore demonstrating the very effect we’re talking about:

So if honest advice risks being ignored, what is a responsible scientific adviser to do? “It’s an excellent question, and I’m not sure that I have a great answer,” says Moore.

---

[1] Well, that and because he ran a crappy campaign. But, hey.

Brain damage in football players

Examinations of the brains of six deceased former National Football League players have shown damage indicating chronic traumatic encephalopathy, a degenerative condition caused by repetitive head trauma:

Using techniques that can be administered only after a patient has died, doctors have now identified C.T.E. in all six N.F.L. veterans between the ages of 36 and 50 who have been tested for the condition, further evidencing the dangers of improperly treated brain trauma in football.

“It’s scary — it’s horribly frightening,” said Randy Grimes, who played center next to McHale on the Buccaneers for several years. “I’ve had my share of concussions, too. More than my share. My wife says I have short-term memory loss. It’s really scary to think of what might be going on up there.”

News flash: getting your head battered about for years... is not good for your brain. Who knew?

Well, actually, technically, we still don’t know. This isn’t a controlled scientific study, and doctors on the NFL payroll remind us of this, and tell us that as far as they know, if your head gets kicked in every few games, there’s no cumulative effect as long as it’s “managed properly” each time.

Now, I’m usually among the first to look critically at incomplete, flawed, or hasty studies, and to question judgments made with insufficient evidence. I often point out when questionable conclusions are drawn from such evidence. And it’s absolutely true that this needs to be studied more in order to be sure of what’s going on.

But there’s another set of points here: the suspected cause is so closely connected to known causes of the condition, the damage is sufficiently severe that quick action is warranted, and a properly controlled study would take too long to carry out and would be ethically questionable. Let’s look at each of those:

1. We know that repeated brain trauma brings on CTE. We know these players are suffering repeated brain trauma. What we don’t know is whether prompt and correct treatment of the individual traumas is enough to ward off long-term damage, as the NFL doctors claim, so we don’t know that the players’ concussions are the cause of their CTE pathology.

   But we’re not talking about wondering whether the players’ diet of, say, more red meat than the average person is what’s causing the encephalopathy. If that were the question, I’d agree that the data don’t support it. We’re talking about a known connection, for which I think the existing data is sufficient cause to take action, pending more study.

2. As Lisa McHale says in the article, “We’re not talking about turf toe — we’re talking about a significant brain injury that has huge implications in terms of people’s health.” The fact that this is addressing a very serious, debilitating, non-reversible condition means that taking action even before studies are completed is warranted.
3. Because the condition develops over many years, and because even after it develops it’s degenerative over many more years, a controlled study would take decades to complete. What’s more, since they can only be sure of what’s happened post mortem, the decades would stretch out into the lifetimes of the studied players. It could well be fifty years or more before a properly controlled study could produce results. And the ethics of leaving players untreated for suspected brain pathology are questionable, to say the least.

Sure, study this more, by all means. But there’s enough evidence here as it stands to require action now, from an ethical point of view, even before further study.

Carrie Bradshaw might get you pregnant

Or maybe it’s Samantha Jones; we can’t be sure.

Researchers have recently published a study that shows that teens who watch TV shows with sexual content are more likely to become involved in a pregnancy. The Washington Post seems to do a good job of summarizing the study; unfortunately, the full article at the American Academy of Pediatrics is behind a pay-wall.

From the AAP:

EXPOSURE TO SEXUAL CONTENT ON TELEVISION LINKED TO TEEN PREGNANCY

For the first time, a new study links teen exposure to sexual content on television with pregnancy. In “Does Watching Sex on Television Predict Teen Pregnancy? Findings from a National Longitudinal Survey of Youth,” researchers used data from a national survey of teens, ages 12 to 17, to assess whether exposure to television sexual content predicted subsequent pregnancy (girls), or responsibility for pregnancy (boys) over a three-year period. Teens exposed to high levels of televised sexual content (in the 90th percentile) were twice as likely to experience a pregnancy during the three-year period, compared to teens with lower levels of exposure (10th percentile). Limiting teen exposure to sex in the media and balancing portrayals of sex with information about possible negative consequences might reduce the risk of teen pregnancy, according to the study authors.

Here’s what the WaPo reports:

Chandra and her colleagues surveyed more than 2,000 adolescents ages 12 to 17 three times by telephone from 2001 to 2004 to gather information about a variety of behavioral and demographic factors, including television viewing habits. Based on a detailed analysis of the sexual content of 23 shows in the 2000-2001 TV season, the researchers calculated how often the teens saw characters kissing, touching, having sex, and discussing past or future sexual activity.

Among the 718 youths who reported being sexually active during the study, the likelihood of getting pregnant or getting someone else pregnant increased steadily with the amount of sexual content they watched on TV, the researchers found. About 25 percent of those who watched the most were involved in a pregnancy, compared with about 12 percent of those who watched the least. The researchers took into account other factors such as having only one parent, wanting to have a baby and engaging in other risky behaviors.

Fifty-eight girls reported getting pregnant and 33 boys reported being responsible for getting a girl pregnant during the study period. The increased risk emerged regardless of whether teens watched only one or two shows that were explicit or surfed many shows that had occasional sexual content, Chandra said.

If you’ve read some of my other complaints about survey-based studies, you’ll know what my complaint about this one is: they have shown correlation, but not causality.

It’s entirely likely that teens who are inclined to be sexually active are also inclined to want to watch TV shows that talk about or depict sex. Inversely, teens who would prefer not to have sex are likely to be more put off by titillating TV.

It’s not just me, of course:

Several experts questioned whether the study had established a causal relationship.

“It may be the kids who have an interest in sex watch shows with sexual content,” said Laura Lindberg of the Guttmacher Institute. “I’m concerned this makes it seem like if we just shut off the TV we’d dramatically reduce the teen pregnancy rate.”

And that is exactly the problem with these sorts of studies. It’s not that they have no value; they certainly do. It’s that they lead to silly conclusions and unwise decisions — often to unwise public policy. These sorts of studies should be used as ticklers for further study. They uncover correlations that can be further explored, but they do not uncover “truths”.

And then there’s this odd recommendation, which makes me wonder about the genesis and funding sources of this study:

Programmers should also include more-realistic portrayals of the risks of sex, such as sexually transmitted diseases and pregnancy, the researchers said.

Hm. Do we really think we want more TV shows depicting pregnant teenagers and teens with STDs? Somehow, I think not. And imagine how silly something like Sex and the City would have been if the 30-something ladies kept warning Samantha that she was going to get the clap if she didn’t take precautions.

One could do a better-controlled study like this, but it’d be hard: it would mean controlling what the kids watched on TV (and perhaps making them watch things they don’t want to), so it’s not clear how well it would work out, really. But they’d be volunteering.

Here’s (roughly) what I’d do:

1. Get a bunch of volunteers and have them fill out questionnaires. Do not, of course, tell them what you’re researching.
2. Based on the questionnaire, divide them into several groups of similar composition — each group has the same balance of religious kids, kids with a single parent, kids raised by wolves, and so on.
3. Assign a slate of TV watching to each group. You must watch the programs in your group’s list to the best of your schedule/ability. You must avoid other TV as much as you can.
4. Interview them periodically, and adjust the TV schedule, keeping the sexual content approximately constant.

It’d be tough to keep that up for three years, but that might give you some valid conclusions about causality. It might give you a better idea of whether it’s watching TV programs that relates to behaviour... or the inclination to watch that does.

A SIDS study

The idea of babies dying in their sleep, unexpectedly and without explanation, frightens and horrifies us. Of course, the idea of adults dying that way does too, or should... but when it’s babies, few-month-old bundles of cuteness and love, we find it especially jarring. So we give it a name — SIDS, Sudden Infant Death Syndrome — and we grope for explanations, trying to find out what the problem is and to stop it.

One problem with doing that is, of course, that we’ve just given a name to something vague. We may talk of a SIDS “diagnosis”, but it’s not so in the same sense as “liver cancer”. A diagnosis of SIDS means, “Your baby died, and we don’t know why,” a non-diagnosis, really. We assume that there are enough cases that are from the same cause that we can isolate at least one cause of at least some cases.

But it makes studying it difficult. And, of course, making it more difficult are other factors, such as the relative infrequency of SIDS and the fact that there are serious ethical issues preventing many of the ways medical issues are typically studied.

The result is that the studies we have are anecdotal, and we’re using a trial-and-error approach to it. Or better to call it a trial-and-progress approach, because progress has been made: there was a 56% reduction in the incidence of SIDS between 1992 and 2003.

There’s a recent study, published in the Archives of Pediatrics & Adolescent Medicine that adds itself to the mix. Unfortunately, one needs a subscription to read it, so I have to go by the news articles, which are notoriously spotty on reports of scientific studies. The New York Times says this, for example:

Since 1992 the rate of SIDS deaths has dropped by more than half, to about one death per 2,000 live births from 2.4 per 1,000. The decline is linked to a national “Back to Sleep” campaign that promotes putting babies on their back instead of their stomach, which has been shown to lower the risk of sudden death.

That’s a reduction from .24% to .05%. In what sense is that merely “more than half”? I suspected that it’s an error, and that at least one of the numbers is wrong.

Checking other reports turns up that the Times article tried to normalize the numbers and made an error in the process: it should have said 2.4 per 2000, not 1000. So the correct numbers are .53 SIDS deaths per 1000 live births in 2003, and 1.2 per 1000 in 1992, .053% vs .12% eleven years earlier.

Putting that in other terms, it means that in 1992 the rate was one SIDS death in about 833 live births, and in 2003 it was one in about 1887. We’ll come back to this after looking at what WebMD says about the study:

One theory is that SIDS is caused by the buildup of carbon dioxide when infants with inadequate sleep arousal responses re-breathe exhaled air trapped by bedding or proximity to other sleeping family members.

With this idea in mind, Li and colleagues from the research division of the California-based managed health care group Kaiser Permanente hypothesized that increasing airflow near a sleeping baby would help protect against sudden infant death syndrome.

To test the theory, they interviewed the mothers of 185 babies who died of SIDS in Northern California and Los Angeles County from 1997 to 2000. The average length of time between the SIDS death and the interview was 3.8 months.

The mothers of 312 children matched for age, area of residence, and socioeconomic and ethnic background to the SIDS victims were also interviewed.

Sleeping in a room with an open window was found to reduce the risk of SIDS by 36%, while sleeping with a fan in the room was associated with a 72% reduction in risk.

And the NY Times adds that the benefits of a fan along with the previously identified “safe sleeping environment” are “not statistically significant”, though there’s a 16% reduction in risk. How can that be?

It’s that statement that really points out what the problems with this sort of study are, and where its limitations lie. Let’s go back to the numbers I computed earlier: in 2003 you’d have to look at 1887 live births to statistically expect one SIDS death.

The researchers interviewed parents of 185 SIDS victims in the late 1990s. Even if we assume that the rate of death was 1 in 1000 then (somewhere between the 1992 and 2003 rates), that means we’d need 185,000 live births to expect that many SIDS deaths. If we looked at 18,500 live births, we’d expect to find only 18 deaths — still enough to do statistics on this. With 1,850, two deaths would be against the odds.

But that’s still six times the number of survivors they actually looked at.

If you talk with only 312 parents of living children, even if they had ignored all the “safe sleeping environment” recommendations, even if that group had the 1992 death rate applied to it, you’d only expect a 37.5% probability of a single SIDS death.

That means that things have to be pretty glaring before they have any statistical significance with a sample that small, and even then I wouldn’t call it a “reduction in risk”. At best, we can say that there’s a correlation — but we have to be careful about confusing correlation with cause.

So, should we be running fans in babies’ rooms when they sleep? Sure, it seems harmless and there’s some small indication that it might help. Mostly, it needs more study, but the only way to study it is to recommend it, have parents do it widely for a few years, and then see what happens to the SIDS rate by, say, 2011.

We’ll get there... but until we can actually confirm causes of this, it’s going to remain difficult to study and hit-and-miss to prevent.

I'm shocked... shocked, I tell you

For the latest installment in the “studies with obvious results” series, we turn to a study by the National Center on Addiction and Substance Abuse at Columbia University, which finds that online drug sales often bypass prescriptions:

A large majority of 365 Internet sites that advertise or sell controlled medications by mail are offering to supply the drugs without a proper prescription, according to a new study. The online trade is stoking the rising abuse of addictive and dangerous prescription drugs, the authors and federal officials say.

[...]

“Nevertheless, anyone of any age can obtain dangerous and addictive prescription drugs with the click of a mouse,” Mr. Califano said. The center is issuing the study, the latest of five annual surveys, on Wednesday.

This should surprise no one who’s received any amount of spam on the subject, spam with subjects such as these, from my spam logs:

90% drugs discount.
Your medical Faciiitation
LowerPrice Pharmacy! Save your medical bills up to 90%
76% Savings PLUS Free Shipping...3 Days Only
From $2/pill:- Ambiem, Xamax, Valiun, Phentermime & many hard available
get your hydrocodone, vicodin, phentermin online WithoutPrescription
RXOnline-70-80% OFF!
OnlinePharmacy - WholesalePrices
Bestt Pharmaccy Online!
BestQualityInternationalPharmacyForCustomers
Broad Choices Cheap ~Can a dian~ Medicament deals
on line Phar macy NO Prescription.
New online Pharm acy shop save up to 90%
SALE 85% OFF on Pfizer
your Farmaccy Prescripiton
Thanks for your Pharrrmacy order

It also should not be a surprise because of what online shopping gives us access to.

Federal law bars dispensing dangerous medications without a prescription from a doctor who has a bona fide relationship with the patient. But officials have had a hard time catching up to rogue Internet pharmacies that sometimes ship the drugs from foreign countries in disguised packages.

Short of having every package opened, there’s little we can do. Obviously, when officials get wind of something — someone within the US who they can arrest, or a source outside the US they can track and block — they do something about it, and new laws may help there. But it can be very hard:

For the last several years, the Drug Enforcement Administration and others have worked to halt the illegal trade and prosecute involved doctors and suppliers, with limited success.

“One of the main problems is that the sites can literally open up for a week, close and open up under a different name,” said Michael Sanders, a spokesman for the Drug Enforcement Administration.

For the study, the researchers did their own searches, using popular search engines — they didn’t respond to spam. They found 159 sites that sold drugs directly, and 135 of those would sell drugs without a prescription (or, equivalently, with just the buyer’s say-so that a prescription was in hand).

An interesting aspect of this is that some of the sites selling pharmaceuticals online are advertising not just through spam, but also by placing ads on those legitimate web sites and search engines — Google, Yahoo!, MSN, and others. Let’s see how much luck they have in shutting that down.

As I’ve said before, while I might make fun of studies that produce obvious results, that doesn’t mean that It’s not worth doing these sorts of studies. They validate what we know, they provide references to concrete data for other work, and, in cases like this where the study is repeated periodically, they demonstrate trends and changes over time.

The text of this year’s study is here.