How good are studies?

It's good to be a skeptic.

In 2016 only 39% of published psychology studies were able to be replicated.

in 2012 out of 53 "landmark" studies in oncology and hematology only 6 were able to be replicated.

Estimate that $28 billion dollars a years is spent on irreproducible preclinical reasearch.

This is scary given the amount of policies in all areas that is made based on published studies.

I remember getting sucked into Vit D and Selienium as well as pomegranate in slowing PC by studies that could never be reproduced.

It's not as bleak as all that, at least where cancer studies are concerned. See this recent article on the ongoing Reproducibility Project: Cancer Biology > Cancer reproducibility project releases first results, Nature, 18 Jan 2017

The drift of the article is that while skepticism is warranted and healthy, many factors can interfere with reproducibility, hence a single failure to reproduce a study doesn't necessarily invalidate the study's results. So far, the Project has reported on 5 studies, of which 2 were substantially reproduced.

Of note: "In 2012, researchers at the biotechnology firm Amgen in Thousand Oaks, California, announced that they had failed to replicate 47 of 53 landmark cancer papers. That was widely reported, but Amgen has not identified the studies involved."

Billy,
Sorry, I meant vit E and Selenium, which was the rage in 2009. All my doctors recommended it for PC control.

Tall Allen said...
BTW- The latest on Vitamin D3 - a review and meta-analysis - shows no value in supplementing for any purpose:

/www.aafp.org/afp/2018/0215/p226.html

So, based on all of that info, have you ceased vitamin D supplementation and testing?

Yep, this one was the usual tiny dose (tiny parachute syndrome) approach, from reference 13:
www.nejm.org/doi/full/10.1056/NEJMoa055218?query=recirc_curatedRelated_article

Somebody said...
We randomly assigned participants to receive 1000 mg of elemental calcium as calcium carbonate with 400 IU of vitamin D3 daily or placebo.

400 IU? Really? Like I have said so often, sometimes it seems like many studies are designed to fail. Folks can get 10,000 from 30 minutes at the pool in the southern summer time. When I took 2000, it only got my blood work barely up to 30 from however low it was. What would 400 have done for me? But despite the tiny dose, rather than zero benefit, there was this:

Somebody said...
Hip bone density was 1.06 percent higher in the calcium plus vitamin D group than in the placebo group....... Intention-to-treat analysis indicated that participants receiving calcium plus vitamin D supplementation had a hazard ratio of 0.88 for hip fracture

That sounds at least a little valuable, doesn't it, since people die from complications of hip fractures?

It is wise to be a skeptic on studies, as John T. said.

Post Edited (BillyBob@388) : 4/17/2018 8:58:27 PM (GMT-6)

And from the last quoted study, looking at a table, I guess they attempted to look just at the vitamin D effect, the table shows that the odd ratios for hip fracture were 50% higher for vitamin D blood levels of 43-60 nmol/l(17-24 ng) compared to those over 60 nmol/L(24ng). A measly >24ng was a good bit better than <24ng! That is an odd result to be included n a list of references showing that vitamin D is worthless. Gee, I wonder how they might have done if they had got them over 30 or 40 ng with some adequate doses?

www.nejm.org/doi/full/10.1056/NEJMoa055218?query=recirc_curatedRelated_article

And can some one tell me what this means from the above quoted study:

Somebody said...
Censoring data from women when they ceased to adhere to the study medication reduced the hazard ratio for hip fracture to 0.71

Does that mean once they adjusted for non-adhering patients, the risk of hip fracture dropped even further to .71? So hip fractures dropped by over 1/4? ( or do I have t backwards?) Remember, all of this is from the references given in the meta-analysis showing Vitamin D is good for nothing.

Post Edited (BillyBob@388) : 4/17/2018 9:20:32 PM (GMT-6)

F8 said...
so say you took that stuff and it made you feel like you were doing something for yourself and you are in remission all theses years later. also three things I know that are bad for you one year and good for you the next -- eggs, red wine and coffee.

Very true. Again, this raises a question about the studies and how much we can trust them, whoever they are from. And you really must dig into a study for the details to see if the data matches the headline or conclusion.

you should only read studies I write.

Just a personal anecdote. about 3 years before my RRP my PCP added Vit-D to my routine blood lab. It came back on the low side - about 24 ng/ml. So she told me to take 2000 units of Vit-D3 a day, which I did. 6 months later I was back for another lab to recheck my PSA and Vit-D levels. My PSA fell from 8.3 to 7.5 in those 6 months and my Vit-D level rose from 24 to 35 ng/ml. This was the very first time my PSA decreased at any time for a 10-15 year period. There was ALWAYS a very slow gradual rise. Then 6 months later PSA returned to about 8.4. So I unfortunately regained the lost ground

But the fact that my PSA dropped in that initial 6 month period made a believer out of me so I continue to take D3. I live in a place where the sun shines regularly - so I was surprised my Vit-D level was so low when first checked. Maybe I have a genetic predisposition for a deficiency in that particular vitamin, which is another reason I've continued. I'd like to keep it at a level of 50 to 70 ng/ml. or so.

Amen jmadrid,
This was exactly the point I was trying to make. It not only affects the medical sector, but major policies that effect all of our lives such as social issues educational policies, climate change ect. It is amazing how many policy decisions are made based on studies that cannot be reproduced by other scientists.

jmadrid said...
Even when a publication only offers statistics (as it usually happens in medical fields) the same requirement stands: the statistics should be reproducible. If it only can be applied for given, local, non-reproducible, conditions, the conclusions are not real science.

John T said...
Amen jmadrid

I was ok with the opening post, but I'm not so sure I'm on the same page with where this thread is going with the comment and apparent agreement, above. I will note that it is important to observe the type/quality of publication that information is published in—is it People magazine, or the New England Journal of Medicine. This is incredibly fundamental, but that wasn't a point that's been brought up here (yet). For the rest of this post, I'm only talking about peer-reviewed medical journals, which I believe was the intended reference to medical field publications in the comment above.

First point, I have never seen a medical publication "only offer statistics." Never. If you have any example whatsoever, please post the link here in reply so that we can all be simultaneously stunned.

Second, and tied to the first point, let's review what it means to be published in a peer-reviewed publication: a) Author must submit the article to the journal editor who forwards the article to experts in the same field for review. b) An impartial review panel evaluates the quality, accuracy and validity of the research methodology and procedures. If it's "only statistics," it's not going to get published.

Third point, when there are issues (like small sample size/low statistical power, as a very common example) with a study (and not all—or any—studies are perfect in the ways we'd like them to be), it's not like the issues are invisible. Too many people, however, fall victim to reading only the headlines. I have no sympathy for those. You have to read and make an attempt to understand the content if you are going to comment or forward a study, in my humble opinion.

Fourth point, is merely a comment on replication; our current system represents an expert initial judgement of whether enough information has been given to replicate the study...but actual replication attempts might then be reserved—from a practical standpoint—for particularly important or controversial or unexpected results.

Is our current system perfect? No. Is it pretty good? Despite the existence of "publication bias," I think it is pretty good and moreover I believe that there is no way to make it bullet-proof.

One aspect of the apparent "contradictions" noted in some of the earlier posts of this thread are, I believe, just the way that the progress of science unfolds in front of us in real time. We are smarter today than yesterday. I'm not so quick to suddenly reverse course on the multiple older opinions of Vitamin D, for example. In my view, we simply may not fully understand, yet, the complete set of variables (perhaps not yet controlled) which influence the results in our very complex bodies. We know there is a correlation between low Vit D and lots of bad things, but we don't even understand (yet) which came first...the chicken (low Vit D) or the egg (bad things). The best we can really say is that there is conflicting data which hasn't yet identified all the key variables.

Are we having a crisis of confidence on medical studies? Is the sky falling? I don't think so.

How good are studies in peer-reviewed medical journals? Pretty good.

Post Edited (NKinney) : 4/18/2018 12:10:22 PM (GMT-6)

jmadrid said...
All I can say is that I have been reading articles (I would add more than that) in peer reviewed journals on physics or chemistry for many years and, unfortunately, I read many of in peer reviewed medical journals now. Many contributions can be helpful, I do not deny that. I admit they can be clasified as "pretty good". But few of them reach the reproducibility level that it is required in what I consider science. Of course, I will not go into details about specific articles that mainly offer statistitics togheter with a few lines of some unproved speculations about what the statistics results mean. Anyway, medicine is, in a good part, based in statistics and stititical results can be science, for instance the reports on well designed trials. It was not my intention to be critical about articles reporting statitistics.
On the other hand, the thread is about reproducibility. I remembered a previous one on science and maybe I mixed both of them too much.

Jmad, re: reproducing results: there are those that can not be reproduced. But what about those where there is no attempt to reproduce? Or, at least no attempts by different people? I am thinking of the 2012 study out of USC that I just quoted over in the brand new vit d thread. The one that had fairly impressive results on follow up biopsy after 1 year on 4000 iu/day Vit D. That is 6 freaking years ago. Why hasn't there been a follow up on that study yet, if nothing else by the people who would delight in proving it wrong? You would think people would be all over that. Because if a valid study this would be very helpful news for PC patients. But there would also be a group that this might be bad financial news for, and I would expect them to be very interested in proving those results bogus. Why haven't they at least tried, though i doubt you could get an unbiased study out of a group trying to prove something with no financial benefit for them? Still, there should be some even if biased. Where are the clinical trials?

There should have been plenty of clinical trials with high dose vitamin D by now to tell us unequivocally whether it is good, useless or even harmful. But in fact I have yet to see them, or very, very few.

So, I agree with you on a need for to reproduce results before we can call it science. But what if no one will even try? Well, in the absence of evidence for harm, if no one will even go to the trouble to prove Hollis/USC wrong (or at least w don't know)by failing to reproduce USC's results with an unbiased study, I am going to consider the results of that USC study valid. Many will differ with me on that, no doubt. But we will all either die of old age or PC waiting on these guys to actually do the studies on things that will not generate financial gain, or might even do the opposite.

Post Edited (BillyBob@388) : 4/19/2018 12:49:44 PM (GMT-6)

Billybob-

I have tried to explain to you before that there is a standard that medical science sets for levels of evidence. I know you are as disinclined to learn about it as I am to read long, meandering posts with extensive quotes in the body of it. But I did actually look at the study you are talking about - it was very small (n=44) and non-randomized. How would any ever know what outcomes would have been if they had not taken the supplement?

In short, the study that you think is so important, actually shows nothing. It does suggest that if there is an effect, there cannot be very much of an effect. The hypothesis -- is there a benefit or a risk of supplementing Vitamin D in patients on active surveillance? -- cannot be answered by that study design. That question can only be answered by a large randomized clinical trial. Because there are much more important questions to answer, I doubt that such an RCT will be conducted anytime soon.