Big Science is Broken

Forum adverts like this one are shown to any user who is not logged in. Join us by filling out a tiny 3 field form and you will get your own, free, dakka user account which gives a good range of benefits to you:

No adverts like this in the forums anymore.
Times and dates in your local timezone.
Full tracking of what you have read so you can skip to your first unread post, easily see what has changed since you last logged in, and easily see what is new at a glance.
Email notifications for threads you want to watch closely.
Being a part of the oldest wargaming community on the net.

If you are already a member then feel free to login now.

**djones520**

http://theweek.com/articles/618141/big-science-broken

I read this the day after I read a story about Oregon schools banning books that cast doubt on man made climate change. Thought it was fitting.

Science is broken.

That's the thesis of a must-read article in First Things magazine, in which William A. Wilson accumulates evidence that a lot of published research is false. But that's not even the worst part.

Advocates of the existing scientific research paradigm usually smugly declare that while some published conclusions are surely false, the scientific method has "self-correcting mechanisms" that ensure that, eventually, the truth will prevail. Unfortunately for all of us, Wilson makes a convincing argument that those self-correcting mechanisms are broken.

For starters, there's a "replication crisis" in science. This is particularly true in the field of experimental psychology, where far too many prestigious psychology studies simply can't be reliably replicated. But it's not just psychology. In 2011, the pharmaceutical company Bayer looked at 67 blockbuster drug discovery research findings published in prestigious journals, and found that three-fourths of them weren't right. Another study of cancer research found that only 11 percent of preclinical cancer research could be reproduced. Even in physics, supposedly the hardest and most reliable of all sciences, Wilson points out that "two of the most vaunted physics results of the past few years — the announced discovery of both cosmic inflation and gravitational waves at the BICEP2 experiment in Antarctica, and the supposed discovery of superluminal neutrinos at the Swiss-Italian border — have now been retracted, with far less fanfare than when they were first published."

What explains this? In some cases, human error. Much of the research world exploded in rage and mockery when it was found out that a highly popularized finding by the economists Ken Rogoff and Carmen Reinhardt linking higher public debt to lower growth was due to an Excel error. Steven Levitt, of Freakonomics fame, largely built his career on a paper arguing that abortion led to lower crime rates 20 years later because the aborted babies were disproportionately future criminals. Two economists went through the painstaking work of recoding Levitt's statistical analysis — and found a basic arithmetic error.

Then there is outright fraud. In a 2011 survey of 2,000 research psychologists, over half admitted to selectively reporting those experiments that gave the result they were after. The survey also concluded that around 10 percent of research psychologists have engaged in outright falsification of data, and more than half have engaged in "less brazen but still fraudulent behavior such as reporting that a result was statistically significant when it was not, or deciding between two different data analysis techniques after looking at the results of each and choosing the more favorable."

Then there's everything in between human error and outright fraud: rounding out numbers the way that looks better, checking a result less thoroughly when it comes out the way you like, and so forth.

Much more at the link, I don't like pasting full stories.

**AlmightyWalrus**

If the system is being used to investigate older studies and making sure they're not wrong and they find studies that are wrong, how is that not the system self-regulating? The claim that the self-regulating mechanism is broken because the self-regulating mechanism is self-regulating is so staggeringly ignorant that I don't know where to start.

**Monkey Tamer**

When people say they have a study I always remember that not too long ago homosexuality was in the DSM. Scientific opinions change.

**Jihadin**

I'm old school. The crap we do as humans dos have an effect on our environment.

**Ahtman**

djones520 wrote:
a must-read article in First Things magazine

One Second of Googling wrote:First Things is an ecumenical, conservative and, in some views, neoconservative religious journal aimed at "advanc[ing] a religiously informed public philosophy for the ordering of society".

Shocking, just shocking that an ecumenical, conservative journal would say such things.

**Frazzled**

Actually the argument that science is suffering from nonreplicability follows reports of multiple difficulties in the replicability of social science journal articles. Thats a bit of a scandal in higher education circles right now.

EDIT: I have no idea about the OP article.

**Kap'n Krump**

Last week tonight had a good section on this sort of thing:

https://www.youtube.com/watch?v=9a7WLaYlQJ8

**Ahtman**

Frazzled wrote:
Actually the argument that science is suffering from nonreplicability follows reports of multiple difficulties in the replicability of social science journal articles. Thats a bit of a scandal in higher education circles right now.

And yet the article originates from an incredibly untrustworthy source, which is sort of the problem. If it is a real problem I imagine there won't be a problem finding stories that don't originate in religious, conservative sources. Would you trust a story about how great cigarettes are for you when the original article comes from a tobacco manufacturer?

**Silent Puffin?**

Frazzled wrote:

EDIT: I have no idea about the OP article.

Meh. There will always be error and bias, that's what happens with fallible humans. That's the entire reason why the scientific method and the peer review process exists and the staggering advancement in human knowledge over the last century in particular is testament to its effectiveness.

Saying that "Science is broken" is ridiculous.

**Frazzled**

Ahtman wrote:
Frazzled wrote:
Actually the argument that science is suffering from nonreplicability follows reports of multiple difficulties in the replicability of social science journal articles. Thats a bit of a scandal in higher education circles right now.

And yet the article originates from an incredibly untrustworthy source, which is sort of the problem. If it is a real problem I imagine there won't be a problem finding stories that don't originate in religious, conservative sources. Would you trust a story about how great cigarettes are for you when the original article comes from a tobacco manufacturer?

I did not base my statement on the OP, just noting that this is an issue in some fields. I tend to keep up because of the Boy's career track, and I'd like to be an adjunct when I retire. I make no alliance with the group this comes from.

Automatically Appended Next Post:

Silent Puffin? wrote:
Frazzled wrote:

EDIT: I have no idea about the OP article.

Meh. There will always be error and bias, that's what happens with fallible humans. That's the entire reason why the scientific method and the peer review process exists and the staggering advancement in human knowledge over the last century in particular is testament to its effectiveness.

Saying that "Science is broken" is ridiculous.

Nor did I. I said there have been problems in replicating studies recent and its a bit of an issue in certain fields right now. Because of that it wouldn't surprise me if the issue was in more fields than originally believed. The issue is not a plot or something, its more to do with you don't get research money and tenure CV material by replicating studies.

**dogma**

Scientific claims rest on the idea that experiments repeated under nearly identical conditions ought to yield approximately the same results, but until very recently, very few had bothered to check in a systematic way whether this was actually the case.

Really? I have 3 degrees in social sciences, and bitching about lax methodology pretty much got me the third one. Hell, it is a well established fact that psychological and sociological research is suspect due to sample bias.

Looking at sixty-seven recent drug discovery projects based on preclinical cancer biology research, they found that in more than 75 percent of cases the published data did not match up with their in-house attempts to replicate.

Because cancer is a complicated thing which afflicts many parts of the body, all in different ways.

When a study fails to replicate, there are two possible interpretations.

No, there are many possible interpretations; person that clearly hasn't read anything about Kuhn or Popper.

**Frazzled**

Agreed on all points Dogma.

**Peregrine**

It actually is a problem, not just the conservative outrage machine at work. It comes down to two things:

1) Media reporting on science sucks. You see it over and over again, someone publishes an entirely reasonable paper with very conservative conclusions and a suggestion for future research, and the media reports it as "SCIENCE PROVES THAT ALIENS EXIST" or whatever. It's clickbait "journalism" at its worst, and because "nope, guess our preliminary results weren't accurate after all" isn't good clickbait the part where science does its job and investigates further is never reported.

2) Certain fields have a problem with replicating results. It's kind of the science journal equivalent of clickbait: you get published and the grants/attention/etc that come from being published by getting interesting results, not by repeating someone else's experiment. So someone does an experiment once, exaggerates the conclusions a bit to make it more attention-worthy, and then nobody ever runs the experiment again to confirm if the results were legitimate or mostly due to random noise in the data. And the result is that a study that really says "here's an interesting thing that might be worth investigating more" becomes the final word on a subject.

This is NOT likely to be a problem with things like climate change, because those studies have been repeated over and over again and analyzed in every possible way. The verification issue is with studies that don't attract much attention, which obviously excludes the big controversial stuff.

**Frazzled**

This is NOT likely to be a problem with things like climate change, because those studies have been repeated over and over again and analyzed in every possible way. The verification issue is with studies that don't attract much attention, which obviously excludes the big controversial stuff.

Exactly.

**jreilly89**

Peregrine wrote:
It actually is a problem, not just the conservative outrage machine at work. It comes down to two things:

1) Media reporting on science sucks. You see it over and over again, someone publishes an entirely reasonable paper with very conservative conclusions and a suggestion for future research, and the media reports it as "SCIENCE PROVES THAT ALIENS EXIST" or whatever. It's clickbait "journalism" at its worst, and because "nope, guess our preliminary results weren't accurate after all" isn't good clickbait the part where science does its job and investigates further is never reported.

2) Certain fields have a problem with replicating results. It's kind of the science journal equivalent of clickbait: you get published and the grants/attention/etc that come from being published by getting interesting results, not by repeating someone else's experiment. So someone does an experiment once, exaggerates the conclusions a bit to make it more attention-worthy, and then nobody ever runs the experiment again to confirm if the results were legitimate or mostly due to random noise in the data. And the result is that a study that really says "here's an interesting thing that might be worth investigating more" becomes the final word on a subject.

This is NOT likely to be a problem with things like climate change, because those studies have been repeated over and over again and analyzed in every possible way. The verification issue is with studies that don't attract much attention, which obviously excludes the big controversial stuff.

Shamelessly stolen from SMBC, a fantastic comic. Peregrine, I agree on all points. Science is not broken, the journalism regarding science is broken.

**Smacks**

I don't have an issue with scientists being wrong sometimes, that's expected. I don't even have an issue with there being a systemic problem with "big science"... in a world where scientists have to compete for funding, that is also expected.

What I do have an issue with though, are the people who are going to use this to make the fallacious argument that because "some" (or even "many") scientists have made errors, that therefore proves all science is questionable, even solid theories which have been corroborated over and over. Then make the even more extraordinary leap that their ridiculous idea about cavemen riding dinosaurs, is therefore supported somehow.

I'm happy to accept that there is a lot of bad science out there, but no matter how bad it gets, even the most obviously corrupt and flawed scientific study still has way more credibility IMO, than silly superstitions and 3000 year old hearsay. So lets not give up on science just yet.

**sebster**

I'm not sure about other fields but I know that in economics there is a problem called p-hacking. With modern computing power you can quickly examine data for patterns - while this is a great step forward, doing in minutes what might have once been a whole thesis work, it does come with a big problem. Because with a big enough data set with enough variables, all kinds of weird relationships will appear that'll beat the confidence interval and appear to be a real thing. If you try to offset this by increasing the confidence interval you risk excluding real but subtle patterns. What people should do is establish expected patterns before hand and only consider those, but that's both hard and risks all the work being junked if no relationship is shown. While it doesn't 'break' science or anything like what the original article claimed, it is an unfortunate limit on what people had hoped for this new world of big data.

As to the actual examples given in the article, I can only comment on the two economics ones cited. Levitt's work did include a simple arithmetic error, but it also included some really dodgy data selection. To his credit Levitt did accept these technical issues when presented with them. However, he then went a found another method that still produced a conclusion very similar to his own. Rather than technical errors that strikes me as the bigger issue - researchers finding a method to fit their conclusion.

And yeah, Rogoff and Reinhardt did produce a sloppy work with a basic excel error, and once that error was corrected the claimed relationship between debt and growth disappeared. But the greater point is that it was a very minor paper, no-one had picked up the error because minor works don't get checked. The paper wasn't in the ten biggest publications Rogoff and Reinhardt made that year, it certainly wasn't important to the greater field of economics. But for the GFC it probably would have been forgotten to history... but with the GFC we saw the rising political force of austerity, and those guys went looking for any kind of publication that'd give an economic cover to their political aims. So they ignored that this was a very small work with few data points, and instead treated it like a major, very important finding. And of course they ignored the major institutional voices that told them this work looked unlikely, and continued to ignore them when the work was revealed as junk. This strikes me as the other big problem in science - academics produce their work in a bubble, and don't protect themselves or their work when it is claimed by various political movements.

**Easy E**

Peregrine wrote:
It actually is a problem, not just the conservative outrage machine at work. It comes down to two things:

1) Media reporting on science sucks. You see it over and over again, someone publishes an entirely reasonable paper with very conservative conclusions and a suggestion for future research, and the media reports it as "SCIENCE PROVES THAT ALIENS EXIST" or whatever. It's clickbait "journalism" at its worst, and because "nope, guess our preliminary results weren't accurate after all" isn't good clickbait the part where science does its job and investigates further is never reported.

2) Certain fields have a problem with replicating results. It's kind of the science journal equivalent of clickbait: you get published and the grants/attention/etc that come from being published by getting interesting results, not by repeating someone else's experiment. So someone does an experiment once, exaggerates the conclusions a bit to make it more attention-worthy, and then nobody ever runs the experiment again to confirm if the results were legitimate or mostly due to random noise in the data. And the result is that a study that really says "here's an interesting thing that might be worth investigating more" becomes the final word on a subject.

This is NOT likely to be a problem with things like climate change, because those studies have been repeated over and over again and analyzed in every possible way. The verification issue is with studies that don't attract much attention, which obviously excludes the big controversial stuff.

A perfect example is the research on assuming a "Power Posture" leading to confidence and better life outcomes.

What a bunch of Gak made up and used to sell crap to people.

**jreilly89**

Easy E wrote:
Peregrine wrote:
It actually is a problem, not just the conservative outrage machine at work. It comes down to two things:

1) Media reporting on science sucks. You see it over and over again, someone publishes an entirely reasonable paper with very conservative conclusions and a suggestion for future research, and the media reports it as "SCIENCE PROVES THAT ALIENS EXIST" or whatever. It's clickbait "journalism" at its worst, and because "nope, guess our preliminary results weren't accurate after all" isn't good clickbait the part where science does its job and investigates further is never reported.

2) Certain fields have a problem with replicating results. It's kind of the science journal equivalent of clickbait: you get published and the grants/attention/etc that come from being published by getting interesting results, not by repeating someone else's experiment. So someone does an experiment once, exaggerates the conclusions a bit to make it more attention-worthy, and then nobody ever runs the experiment again to confirm if the results were legitimate or mostly due to random noise in the data. And the result is that a study that really says "here's an interesting thing that might be worth investigating more" becomes the final word on a subject.

This is NOT likely to be a problem with things like climate change, because those studies have been repeated over and over again and analyzed in every possible way. The verification issue is with studies that don't attract much attention, which obviously excludes the big controversial stuff.

A perfect example is the research on assuming a "Power Posture" leading to confidence and better life outcomes.

What a bunch of Gak made up and used to sell crap to people.

That's the problem I have with Psychology and most social sciences. They're extremely fascinating, but the tests are usually A) hard to replicate or B) filled to the brim with bias.

**Ketara**

dogma wrote:

Really? I have 3 degrees in social sciences, and bitching about lax methodology pretty much got me the third one. Hell, it is a well established fact that psychological and sociological research is suspect due to sample bias.

This. Anyone who works in any academic field knows that crap journal publications slip through the net all the time. Why? Because often your job relies on publishing for various reasons (minimum publishing requirements, trying to beat the REF, justifying funding, etc), and that means you publish whether stuff is of any good or not. Heck, there are journals out there you can pay just to take your article so you can put it in your CV.

Seriously, this isn't news, isn't an 'original' finding about academia, and is a poor attack on the current research paradigm. It certainly doesn't justify 'abolishing the PhD' as the writer goes on to demand. This article is practically guilty of what it's complaining about.

Then again, considering it comes from a hack with nothing more than a MsC in Management who works for the 'Ethics and Public Policy Centre', an organisation that apparently exists to 'deal openly and explicitly with religious and moral issues in addressing contemporary issues', I'm really not surprised. It probably is big news to him, and this wouldn't be the first time a religious bloke (he's a blogger on Catholic theology as well) tried to find fault with the scientific establishment.

**Smacks**

Easy E wrote:
A perfect example is the research on assuming a "Power Posture" leading to confidence and better life outcomes.

What a bunch of Gak made up and used to sell crap to people.

That wasn't entirely crap. The repeat study found that subjects did report "feeling" more confident, they just weren't able to corroborate the more objective claims like increased hormone levels and risk taking.

I think research into how things like smiling, exercise, posture, etc affect mood, is quite interesting. The brain is very complicated, and subtle things certainly do have a big impact on people's moods.

**jreilly89**

Smacks wrote:
Easy E wrote:
A perfect example is the research on assuming a "Power Posture" leading to confidence and better life outcomes.

What a bunch of Gak made up and used to sell crap to people.
That wasn't entirely crap. The repeat study found that subjects did report "feeling" more confident, they just weren't able to corroborate the more objective claims like increased hormone levels and risk taking.

I think research into how things like smiling, exercise, posture, etc affect mood, is quite interesting. The brain is very complicated, and subtle things certainly do have a big impact on people's moods.

Agreed. I think there's definitely some positive things to be found, but I'd say A) it varies person to person and B) I'd hesitate to say it does more than simple mood elevation.

**Easy E**

jreilly89 wrote:
Smacks wrote:
Easy E wrote:
A perfect example is the research on assuming a "Power Posture" leading to confidence and better life outcomes.

What a bunch of Gak made up and used to sell crap to people.
That wasn't entirely crap. The repeat study found that subjects did report "feeling" more confident, they just weren't able to corroborate the more objective claims like increased hormone levels and risk taking.

I think research into how things like smiling, exercise, posture, etc affect mood, is quite interesting. The brain is very complicated, and subtle things certainly do have a big impact on people's moods.

Agreed. I think there's definitely some positive things to be found, but I'd say A) it varies person to person and B) I'd hesitate to say it does more than simple mood elevation.

IIRC correctly, the authors basically p-hacked the Holy Hand Grenade out of the data by peeling off sample of a sample of a sample size. Basically, digging down so deep the findings became meaningless.

**dogma**

Smacks wrote:

I think research into how things like smiling, exercise, posture, etc affect mood, is quite interesting. The brain is very complicated, and subtle things certainly do have a big impact on people's moods.

Not to mention media. I don't know an athlete who doesn't have a pregame playlist and, speaking only for myself, listening to really aggressive music helps me write.

**whembly**

dogma wrote:
Smacks wrote:

I think research into how things like smiling, exercise, posture, etc affect mood, is quite interesting. The brain is very complicated, and subtle things certainly do have a big impact on people's moods.

Not to mention media. I don't know an athlete who doesn't have a pregame playlist and, speaking only for myself, listening to really aggressive music helps me write.

I listen to metal when I'm working... particularly if I'm writing lines of codes.

\m/

**Kilkrazy**

The placebo effect is a well proven facte of medical science. It's not unreasonable to suppose that similar effects might show up in other psychological/physiological interactions. IDK how much research has yet show such an effect in operation.

**AegisGrimm**

Nope, definitely cannot replicate research finding of how doing this to the enviroment is either man-made OR bad for the future......

Spoiler:

**Chute82**

Monkey Tamer wrote:
When people say they have a study I always remember that not too long ago homosexuality was in the DSM. Scientific opinions change.

40+ years ago...not sure if you where alive then. DSM2

**sebster**

Ketara wrote:
This. Anyone who works in any academic field knows that crap journal publications slip through the net all the time. Why? Because often your job relies on publishing for various reasons (minimum publishing requirements, trying to beat the REF, justifying funding, etc), and that means you publish whether stuff is of any good or not. Heck, there are journals out there you can pay just to take your article so you can put it in your CV.

Your complaint is valid, but just to point out university and research groups are aware of the problem of staff spamming junky papers. There's a general move to new KPIs like measuring the numbers of times a person's work is cited. Nothing is ever perfect, and there are ways of gaming these measures too, but that's probably true of any KPI system.

**NuggzTheNinja**

In the social sciences (especially social psychology) there are gross misconceptions regarding the actual hypotheses, assumptions, and appropriate conclusions associated with null hypothesis testing. This isn't isolated to social psychology though - see http://schatz.sju.edu/downloads/research/schatzstatsacn.pdf for a review of inappropriate uses of statistical tests in neuropsych.

Or one of my favorite posters - a dead salmon producing significant results in an fMRI study: http://prefrontal.org/files/posters/Bennett-Salmon-2009.pdf

P-hacking (just run more subjects until your P value drops below alpha), inappropriate use of null hypothesis testing (interpreting a null result as proof of a null phenomenon), failure to correct for inflated alpha, failure to satisfy the assumptions of tests - all of these things result from poor knowledge of math. People report partial eta squared as a measure of effect size mainly because SPSS spits it out at you, but it isn't nearly as intuitive to interpret as something like Cohen's D.

Even the methods that promise the best 'global picture' suffer. The file drawer problem really kills the validity of meta analyses. These factors all pushed me away from traditional experimental methods toward modeling & simulation and cognitive architectures.

Basically, traditional experimental papers don't really blow my skirt up anymore unless they're really ground breaking and worth reproducing, which they usually aren't.