Welcome to the Utopia Forums! Register a new account
The current time is Thu Dec 13 00:09:03 PST 2018

Utopia Talk / Politics / Peer reviewed cronyism
Nimatzo
iChihuaha
Tue Mar 13 16:40:07
http://www.mitpressjournals.org/doi/abs/10.1162/REST_a_00666

This paper employs a unique data set on articles, authors, and editors of the top general interest journals in economics to investigate the role of social connections in the publication process. Ties between editors and authors are identified based on their academic histories. About 43% of the articles published in these journals are authored by scholars connected to one editor at the time of the publication. Ph.D. students and faculty colleagues of an editor also improve their publication outcomes when this editor is in charge of a journal.

The best researched connections gets to publish!
TJ
Member
Tue Mar 13 17:10:32
"There are three kinds of lies: lies, damned lies, and statistics." The Twain Mark
Renzo Marquez
Member
Tue Mar 13 18:06:02
http://www...writer-for-some-safety-reviews

Monsanto Was Its Own Ghostwriter for Some Safety Reviews

Academic papers vindicating its Roundup herbicide were written with the help of its employees.
Nimatzo
iChihuaha
Wed Mar 14 03:23:12
Economy is right there in the borderland between science and the type of psuedo science that will obliterate entire nations. We all know cronyism is the bread and butter of bankers and finance people, well the tradition goes all the way to the fountain head where their ideas and theories are pooped out. Garbage in garbage out and the self-correcting mechanism out of play.
Seb
Member
Wed Mar 14 05:43:51
Peer review is the start, not the end, of review processes.

Basically it is the no-brainer "is this worth publishing". It is not designed to catch fraud.

In this case, it is not clear whether authors are getting a free ride in sift stage (i.e. editor decides the subject matter is a better fit for coverage) or at review stage (papers scope and content isn't worth publishing or contains obvious errors).

The former is a problem in terms of giving some people an undue career leg up, but not necessarily a problem for the body of knowledge as a whole (incidentally, shall we talk about sex and racial bias at this stage?)

The latter is a much bigger problem in terms of corrupting the body of knowledge.

The best journals do name blind submissions so neither editors nor reviewers know the authors (though of course, this isn't full proof).

But as I said, peer review is a very basic sense check akin to "Is this published in the washington post, or is this published in the weekly world news?".

What happens next is other researchers read the piece, and then will publish their own articles contradicting or supporting their findings (and occasionally directly rebutting them if very poor).

Journals and Articles are measured by impact factor, which takes into account things like "how often is this piece of work cited" etc.

Generally, journals that are publishing shit bits of work will lose their overall impact factor, command lower subscription and publication fees.

IT's simply not correct to extrapolate from this one article that the corrective processes are broken.

The fact you don't like some things that are published doesn't mean you should be so free in discrediting the most effective process for divining what is true and what is not that humanity has come up with.

It's Putinesque - "If my opnions are not not considered objectively true, I shall poison the well. Then everything is possible and nothing is true, and my Truthiness will be safe from attack."
jergul
large member
Wed Mar 14 06:32:36
There probably is significant selection bias if article authors have previously collaborated with editors.

I would suggest the primary mechanism would be "rework" instead of "reject"

The topic would still have to be relevant and the method solid, but with weaknesses in form or language.

A flaw with the method is that someone good at publishing would tend to have a much wider academic network than someone poor at publishing.

The likelihood of having a relationship is higher for an accomplished researcher, than it is for a novice or inept research paper author.
jergul
large member
Wed Mar 14 06:33:01
Puinesque? Wow.
Nimatzo
iChihuaha
Wed Mar 14 08:57:59
"IT's simply not correct to extrapolate from this one article that the corrective processes are broken."

There are two types of people:
1. Those that can extrapolate from incomplete data.

But in this case I am connecting the dots or "interpolating" from the several articles and criticism I have read on this topic and anecdotally from the numerous people (yourself included) who have talked about the amount of politics there is involved in getting published. At the time you explained that this was one of the reasons you left.

The corrective processes are not "broken", they have flaws. As with anything else, the players learn and adapt their play to whatever game rules there are to further their personal interests and get points. In this game, the goal is to get published.

Just imagine if the company you worked for, 43% of the employees had previous personal affiliation with the CEO or head of HR. This is your "old boys club".

Now this isn't a company, this is the field of economy, they hand out noble prizes for this, people listen to economist for advice. It is one of the fields along with other social sciences that politicians regularly use as arguments and basis for policy. So to the extent that your research field can do damage, my sensitivity towards potential shenanigans go up.

"doesn't mean you should be so free in discrediting the most effective process for divining what is true and what is not that humanity has come up with."

And you are obviously here in the capacity of some sort of high priest defending the honor of his religion. This act is utterly meaningless to me. If I care about something, I care to make it better and to make things better you have to identify flaws, understand them and finally solve them. You seem to have problem digesting this concept, as you do when I criticize my own country.

Think about that last paragraph slowly. I don't engage in back slapping and circle jerks.
Seb
Member
Wed Mar 14 10:24:31
Nim:

"But in this case I am connecting the dots or "interpolating" from the several articles and criticism I have read on this topic and anecdotally from the numerous people (yourself included) who have talked about the amount of politics there is involved in getting published. At the time you explained that this was one of the reasons you left."

Yup, but it doesn't follow that what is published is of a poor quality.

What it means is that it creates a number of large bad incentives on how science is conducted and how scientists are employed.

For example, it leads to people publishing in dribs and drabs and the same work in miunte variations in multiple places. It means some really good researchers get squeezed out because they don't publish frequently enough, or engage in mutual citation and co-authorship partnerships.

But none of those kinds of behaviours leads to work that is fundamentally incorrect no getting published.

And those incentives are far more likely to encourage publication of controversial pieces attacking recieved wisdom (which is actually a good thing). From a journal perspective it generates hits and citations. Ditto from the researchers point of view.

The challenge is, such contrarian views need to be watertight - flimsy and unsupportable contrarian views in a field will be easily dismissed and lead to people concluding the journal isn't worth subscribing to.

So while I have said myself that one reason for getting out of academia was the impact of publish or perish models, I don't think that means published material and scientific outputs can't be trusted.

It means that you don't necesarily get to do what I would really like to be doing - running experiments and figuring things out - and instead spend a lot of time writting the same thing in fifteen different ways, even when it's not a very interesting finding, to maximise publication count.


Jergul raises some good points of uncontrolled factors that would lead you to expect such correlation even when things work.
Seb
Member
Wed Mar 14 10:27:49
" 43% of the employees had previous personal affiliation with the CEO or head of HR"

Editor of a journal isn't the same as CEO or Head of HR though, and yes, in my field, most succesful people know each other because that is the nature of working in an industry over a long period of time. Succesful people do keep coming across each other again and again.

"Garbage in garbage out and the self-correcting mechanism out of play."

My point is that this statement is an extrapolation the data doesn't bare.
Seb
Member
Wed Mar 14 10:30:18
The likley consequence of people knowing editors being more likely to get published (assuming that persists when ccorrecting for career and experience) would be that two people wanting to publish similar articles, the more connected one gets published. It does not follow that their findings would be substantially different.

Particularly as consensus tends to be driven by lots of people repeatedly getting the same or equivalent results.

What you describe seems more a problem for individual researchers and their careers than a systemic problem over the trustworthiness of what is published.
Nimatzo
iChihuaha
Wed Mar 14 11:47:56
I never said anything about the quality of the research. I have not read them all. Generally as a blanket statement I piss a little on economy as a field. I think I have earned it a little, given that I have a degree in it.

"What it means is that it creates a number of large bad incentives on how science is conducted and how scientists are employed."

We agree. Flaws in the system.

"It means some really good researchers get squeezed out because they don't publish frequently enough, or engage in mutual citation and co-authorship partnerships."

We agree. Flaws in the system. Are you getting published because of merits in your work or because you know or rub the right people.

"But none of those kinds of behaviours leads to work that is fundamentally incorrect no getting published."

I make valuable and necessary distinctions between different fields. Every field is not created equal, there are more than slight nuances even in publication traditions and output. Employing people you like and agree with, is the foundation for an echo chamber, it limits the search space, that in turn degrades quality.

^Generally most of my criticism is towards social sciences, this study specifically about Economy. Again as I said to the extent that these fields can cause real life trouble.

"And those incentives are far more likely to encourage publication of controversial pieces attacking recieved wisdom"

Cronyism creates incentives to attack the establishment and status quo? I have a very difficult time imagining this.

"I don't think that means published material and scientific outputs can't be trusted."

It depends on the field. Obviously the social sciences suffer from many flaws, which I have posted about several times. This study points to one in the peer review process that is new to me, which you agree creates a whole set of bad incentives.

Social sciences are important I think, not least because studies from these fields are used as basis for the administration of our societies. I think we should have higher expectations and level of scrutiny.

"It means that you don't necesarily get to do what I would really like to be doing - running experiments and figuring things out - and instead spend a lot of time writting the same thing in fifteen different ways, even when it's not a very interesting finding, to maximise publication count."

And this is just an awful waste. How many budding scientist stay in this awful game and get good at it being boring and uninteresting?

"Editor of a journal isn't the same as CEO or Head of HR though"

They hold about the same power over your career move.

"most succesful people know each other because that is the nature of working in an industry over a long period of time. Succesful people do keep coming across each other again and again."

That is the fair side of the coin, sure. The ugly side of that, specially in the more fuzzy fields, is that you can "research" nonsense over and over disconnected from the real world, you can be very successful doing that. Success as measured by your peers and publication ranking.

I live where things have measurable effects, so for me it would be tempting to dismiss fuzzy science, but I am not. I think we can do better. And these rants of mine are about that. I am not unlike others *cough* declaring entire fields as bunk. I am pointing to specific issues, but stopping short of giving suggestions for improvement. That would require much more effort and insight to be of any value, and for someone to pay me.

"the self-correcting mechanism out of play."

And this may sound hyperbolic, but I try to look at the entire body, from Academia to markets (application), econ contains a lot of outright harmful garbage.
Seb
Member
Wed Mar 14 13:55:28
Nim:

"ditions and output. Employing people you like and agree with,"

It's not demonstrated here that this is happening. Many senior academics in the same dept have fierce rivalries and disagree.

Editors are not hiring managers, and the fact people know each other and may be more willing to see eachothers work published based on respect for the individual doesn't necessarily mean the respect is founded on agreement. In my experience, quite the opposite.
KreeL
Special Member
Wed Mar 14 20:45:38
I would really really like to play Seb a game of checkers.
KreeL
Special Member
Wed Mar 14 20:46:12
...maybe even chess.
KreeL
Special Member
Wed Mar 14 20:47:07
Of course... he is probably too busy to enjoy a little relaxation.
Seb
Member
Thu Mar 15 10:23:54
How about a name blind game of diplomacy
?
jergul
large member
Thu Mar 15 10:31:43
Seb
You can drop the name-blind. It would not take long to figure out who you were.
jergul
large member
Thu Mar 15 10:32:10
Ah, I see. Pseudoname game of diplomacy.
TJ
Member
Thu Mar 15 10:38:03
The clever opponent imposes will on his challenger, but does not allow his will to be imposed on him. :)
Seb
Member
Thu Mar 15 19:33:57
Not necesarily I've done it with people who I did PhD with and several times we have not been able to figure it out.

It does require a fair bit of discipline in communication though.
show deleted posts

Your Name:
Your Password:
Your Message:
Bookmark and Share