Bogus versus extremely low-quality, Sullins edition

https://flic.kr/p/e6sfpP

Photo by CTBTO from Flickr Creative Commons (modified)

Calling a study “peer-reviewed” gives it at least some legitimacy. And if a finding is confirmed by “many peer-reviewed studies,” that’s even better. So the proliferation of bogus journals publishing hundreds of thousands of “peer-reviewed” articles of extremely low quality is bad news both for the progress of science and for public discourse that relies on academic research.

Two weeks ago I briefly reviewed some articles published by D. Paul Sullins, the anti-gay professor at Catholic University, on the hazards of being raised by gay and lesbian parents. I called the journals, published by Science Domain International (SDI), “bogus,” but said you could make an argument for extremely low quality instead.

After that Sullins sent me an email with some boilerplate from the publisher in defense of the journals, and he accused me of having a conflict of interest because his conclusions contradict one of my published articles. After correctly pointing out that a sting operation by Science failed to entrap an SDI journal with a bogus paper about cancer research, he said:

SDI is a new and emerging publisher. … While I would not say SDI is yet in the top tier, and I don’t like their journal names much either [which mimic real journal titles], for the reasons listed above I submit that this publisher is far from ‘bogus.’

How far from bogus?

Since that post, the reviews on the third of Sullins’ papers have been published by Science Domain and its journal, the (non-) British Journal of Education, Society & Behavioural Science. So we have some more information on which to judge.

The paper, “Emotional Problems among Children with Same-sex Parents: Difference by Definition,” was reviewed by three anonymous reviewers (from the USA, Brazil, Nigeria) and one identified as Paulo Verlaine Borges e Azevêdo, from Brazil. I summarize them here.

Anonymous USA

This reviewer only suggested minor revisions (nothing in the “compulsory revision” section). These were the suggestions: Avoid the first person, clarify the race of study participants, discuss the results in more detail, don’t use the word “trivial,” add citations to several statements, grammar check.

Anonymous, Brazil

This review demanded compulsory revisions: Clarify the level of statistical significance used, explain acronyms, clarify use of “biological parents” when discussing same-sex parents. And some minor revisions: one typo, one font-size change, standardize number of decimal places.

Anonymous, Nigeria

This reviewer included compulsory revisions: mention instrument used in the abstract, clarify measures used in previous studies on children’s well-being, test all four hypotheses proposed (not just three), clarify use of instrument used, shorten the discussion. Minor revisions: check for typos.

Paulo Verlaine Borges e Azevêdo, Brazil

This reviewer requested reorganizing the text, like this:

Would be better to redistribute the lengths of results (lessened), discussion (up) and conclusion (down) sections. In many moments, in the Result section the author deal with I believe would be better located in the Discussion (e. eg., between lines 345 and 355). I suggest that the subsections of Results would be reviewed by author and parts that discuss the results be transferred to the Discussion section … Strengths and Limitations would be better located in the discussion section too.

A few additional minor text modifications were included in the marked up manuscript.

Round two

Upon revision, Sullins was subjected to a punishing second round of reviews.

This included an interesting if ultimately fruitless attempt by Anonymous Brazil to object to this somewhat nutty sentence by Sullins: “biological parentage uniquely and powerfully distinguishes child outcomes between children with opposite-sex parents and those with same-sex parents.” What he meant was, when he controlled for the biological relationship between children and their parents — since hetero parents are more likely to have any biological parentage (and they’re the only ones with two bio parents) — it statistically reduced the gap in children’s mental health between married hetero versus same-sex parents. Although the exchange was meaningless in the decision whether to publish, and Sullins didn’t change it, and the reviewer dropped the objection, and the editors just said “publish it,” you would have to say this was a moment of actual review.

OK then

That’s it. None of this touched on the obvious fatal flaws in the study — that Sullins combines children in all same-sex families into one category while breaking those currently with different-sex parents into different groups (step-parents, cohabitors, single parents, etc.) — and that he has no data on how long the children currently with same-sex couples have lived with them, or how they came to live with them. So it leaves us right where we started on the question of same-sex parenting “effects” on children.

Of course, lots of individual reviews are screwed up. So, is this journal bogus or merely extremely low quality? Do we have a way of identifying these so-bad-they’re-basically-bogus journals that is meaningful to the various audiences they are reaching?

This matters is because journalists, judges, researchers, and the concerned public would like some way to evaluate the veracity of scientific claims that bear on current social controversies, such as marriage equality and the rights of gay and lesbian parents.

7 Comments

Filed under Research reports

7 responses to “Bogus versus extremely low-quality, Sullins edition

  1. The open access movement has always been plagued by the problem of entry barriers and evaluation. Virtually all of the open access journals are bogus. They do not merit “peer review” status. Indeed, even in the legitimate journals, the identification of peer-review with “truth” is problematic, to say the least. From a practical point of view for your academic readers, if your work doesn’t come out in TOP peer reviewed journals, it doesn’t count (at Vanderbilt, none of the specialty journals in my area “counted” for tenure). If you’re tenured, you may not care (but you should). If you’re a student or untenured, pay attention. Don’t waste your time with shit journals.

    Like

    • vijay

      This is, unfortunately, true. How I wish for open access, or low cost, journals, particularly in engineering! Our journals cost over 1000 $ a year.

      Like

  2. Wait, what? I am pretty sure that Not! British Journal says on their website that there are no anonymous reviewers, that all reviewers identities are published. I’m pretty sure that is what I read. Why then is Dr. Cohen writing about an anonymous USA reviewer? Can’t he contact the publisher and get the identities?

    Like

  3. The Unexpected Harm of Same-Sex Marriage: A Critical Appraisal, Replication and Re-Analysis of Wainright and Patterson’s Studies of Adolescents with Same-Sex Parents

    This study….What do you think?
    http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2589129

    Like

  4. Why did you ignore me…

    Like

Comments welcome (may be moderated)

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s