TreeBASE and Dryad

It is now generally expected that scientists, unless working on commercial or otherwise confidential projects, make the data underlying their scientific publications freely and publicly available, so that the studies can be replicated if necessary and so that others can use the data for further research.

Sometimes the data are submitted as supplementary material to be published on the journal website, together with the article itself. Some research organisations have their own data repositories. In many cases, however specialised databases are used. GenBank, for example, is a repository of DNA sequence data. Further down the analysis pipeline, I have in the past used TreeBASE to make available sequence alignment matrices and phylogenetic trees, and in one case I have reanalysed other people's data after obtaining them from there.

Recently I had reason to submit another such set of data matrices and phylogenetic trees to a database, and I thought I would go back to TreeBASE. Somehow it did not work out as well as it did a few years ago.

I was able to log in, I created a new submission, I submitted my files, and I described our analysis. The latter process is rather clunky, but okay, it works. Then it turned out that we needed to redo one of the phylogenetic analyses minus one sequence, so I had to delete one of the matrices and one of the trees and replace them with updated versions. That is when the fun started.

Although googling around a bit suggests that other people can do so, I find it impossible to delete anything in TreeBASE. There is no delete button next to anything except co-authors and submissions (i.e. the entire studies). Being unable to change data in a submission, I decided to delete the entire submission and start from scratch. That is surely not how it is meant to work, and it is a lot of extra effort, but what can I do?

As it turns out, not even that. When I ask that a submission be deleted, the web interface thinks for a bit an then throws a Java error at me. I now have three submissions under identical names and cannot delete the first two. Hurray.

At some point I thought I could maybe try out the alternative data repository Dryad.  Perhaps that would work more reliably? At least I have seen it used in several publications lately. I have now twice submitted my eMail address on their 'sign up for a new account' form, been told twice that a confirmation eMail has been sent, and days later neither I nor my spam folder have received any such message.

Perhaps the journal will accept our manuscript without us having the matrix and trees in a public repository? This process is becoming somewhat off-putting.

Update: After a mere four days I have now finally been sent a confirmation link by Dryad. Will see how that repository works.

Manuscript submission then and now

When I started in science, back in the dark ages, submitting a manuscript to a journal was still quite simple, if perhaps a bit inefficient:

1. Print the manuscript in triplicate.
   
2. Write a cover letter and print it.
   
3. Put everything into an envelope and send it off to the editor.

And that was that.

The first innovation was that you only had to send the manuscript to the editor as an eMail attachment, which was actually faster and saved a lot of paper. Unfortunately, however, things have changed again since then.

This is how it works today:

1. Log into the editorial management software of the journal of my choice. If I do not have an account with that journal yet, create one first.
   
2. Go to the author interface, click new submission.
   
3. Select the type of article.
   
4. Paste the title and abstract into an online form.
   
5. Select key words or topics that supposedly help the journal to assign editors and/or reviewers. Click 'save and continue'.
   
6. Upload main manuscript file, generally as an MS Word document.
   
7. Upload all the figures as separate files, generally as TIF or EPS, although JPGs may be acceptable at the review stage. Paste figure legends and write 'link texts' into the form fields.
   
8. Upload all the supplementary data files. If necessary, update the order of the files. Click 'save and continue'.
   
9. Next, the authorship page. As the corresponding author with an account at that journal I am already in, but I may be asked to link my ORCID. (I have no idea if anybody actually uses it for anything - I only ever look people up with their ResearcherID or Google Scholar.)
   
10. Search for my co-authors by name or eMail. I find the second co-author, great. The first and third co-authors aren't in the system, so I create entries for them. Click 'save and continue'.
    
11. Error: No telephone number provided for second co-author. But he was in the system, so you accepted him before! Also, will any editor really ever want to use it? Argh. Let's look up his number. Okay, edited. Click 'save and continue'.
    
12. Suggesting an editor for the manuscript. Oh dear, that's a long list. Hm. I know this guy hates one of the methods we used, he is out. This one is highly qualified but he will probably require us to add this other analysis that he likes. Ah well, worse things could happen. This one is also very qualified, but she works at a university I have a connection to - is that already a conflict of interest? Well, they can always choose somebody else, done.
    
13. Okay, suggesting peer reviewers and providing their contact information. This guy is an obvious choice as he is the expert for one of the analyses we used, but darn, he is currently between institutions. Let's google his name. No, that's outdated. This one too. Ah, I'm lucky: he has an updated CV on this third page I found, complete with the new phone number and eMail address. Okay, now for reviewer suggestion number two. She is another obvious choice as one of the world experts on our study group. Easy to find her information on a staff page, so that's good. Who else? Maybe two more experts on the study group? Ah yes, she would be interested in this, and I have her contact details. And then this other guy from Europe. Google. Darn, nothing, despite the unique name. Perhaps there is contact info on recent papers. No, he is too senior, the corresponding authors are always others. Ah, wait, here? No, an eMail address from 2012 going "director@institution.org" sounds fishy, most likely somebody else is director now. More Google. Ah, finally, was able to click myself through to a staff website, well hidden and not in English. Ye gods. Four qualified reviewers, that should be enough to get going. Click 'save and continue'.
    
14. Long, complicated page with miscellaneous information and declarations. First, write or upload cover letter. Done.
    
15. Next, declare that we have not submitted this manuscript elsewhere. Okay.
    
16. Is this a resubmission? No.
    
17. Declare that we have followed protocol so-and-so on ethical collection practices. Yes.
    
18. Declare that we have added a section on data availability. Wait, was that in the instructions to authors? Don't remember that. Argh. Save. Open manuscript file. Add data availability section. Back to file upload. Delete manuscript file. Re-upload manuscript file. Reorder files. Click 'save and continue'. Back to declaration. Yes, we now have a section on data availability.
    
19. Declare no conflicts of interest. Okay. Click 'save and continue'.
    
20. Large summary page. Check everything I entered so far. Down at the bottom: have to check PDF proofs before being allowed to submit. Click button, wait while the editorial manager bundles everything into a PDF.
    
21. Open PDF. One of the EPS figures does not display. Argh. Argh. Argh. Back to file upload. Delete offending figure. Re-upload figure - as a TIF this time, that should be foolproof. Reorder files. Click 'save and continue'. Back to summary page.
    
22. Re-check everything I entered so far. Click button, wait while the editorial manager generates a new PDF. Looks good this time.
    
23. The big moment is there: click here to submit. "Are you certain? This will submit your manuscript." Yes!

Yay, progress?

How the sausage is made: peer reviewing edition

One of the aspects of working as a scientist that I find most intriguing is peer reviewing each other's work. The main issue is that while how to write the actual manuscripts is explicitly and formally taught and further supported by style guides, helpful books and journals' instructions to authors, there is much less formal instruction on how to write a reviewer's report.

Essentially one is limited to (1) relatively vague journals' instructions to reviewers usually on the lines of "be constructive" or "be charitable", (2) deducing what matters to the editor from the questions asked in the reviewer's report form, and (3) emulating the style of the comments one receives on one's own papers. Apart from generic, often system-generated thank you messages there is generally no feedback on whether and to what degree the editors found my reviewer's reports appropriate and helpful or on how they compared with other reports.

In other words, most of it is learning by doing; after years of practice I now have a good overview of what reviewer reports in my field look like, but as a beginner I had very little to go by.

It is then no surprise that the style and tone in which reviewers in my field write their reports can differ quite a lot from person to person. There is a general pattern of first discussing general issues and broad suggestions and then minor suggestions line-by-line on phrasing, word choice or typos, and there is clearly the expectation of being reasonable and civil. But:

• Some reviewers may summarise the manuscript abstract-style before they start their evaluation, while others assume that the editor does not need that information given that they have the actual abstract of the paper available to them.
• Some stick to evaluating the scientific accuracy of the paper, while others obsess about wording and phrasing and regularly ask authors who are native speakers of English to have a native speaker of English check their manuscript.
• Some stick to judging whether the analysis chosen by the authors is suitable to answer the study question, while others see an opportunity to suggest the addition of five totally irrelevant analyses just because they happen to know they are possible. And sometimes they recommend cutting another 2,000 words from the text despite suggesting those additions, as if those would come without text.
• Some unashamedly use the reviewer's report for self-promotion by suggesting that some of their own publications be cited, relevant or not.
• Some use a professional tone and make constructive suggestions on the particular manuscript in question, but others apparently cannot help disparaging the authors themselves. Luckily that behaviour is rare.
• Some write one paragraph even when they recommend major revision (meaning they could have been more explicit about what and how to revise), others write six pages of suggestions even when their recommendation is rather positive.

Certainly then scientists in my field will have very different ways of approaching the task right from the start. Nonetheless, and for what it is worth, this is how I generally find it useful to do it.

First, I like to print the manuscript - I am old-fashioned like that. I try to begin reading it fairly soon after I accept the job, and for obvious reasons I also try to read through more or less over one day. Often I will read when I need a break from some other task like computer work, on a bus or in the evening at home.

Already on the first read I attempt to thoroughly mark everything I notice. Using a red or blue pen I mark minor issues a bit like a teacher correcting a dictation, while making little notes on the margins where I have more general concerns ("poorly explained", "circular", "what about geographic outliers?").

Usually the following day I order my thoughts on the manuscript and start a very rough report draft by first typing out all the minor suggestions. (I would prefer to use tracked changes on a Word document for that, but unfortunately we generally only get a PDF, and I find annotating those even more tedious than just writing things out.) Then I start on the general concerns, if any, merely by writing single sentences on each point but do not expand just then.

In particular if the study is valuable but has some weaknesses I prefer to sleep over it at this stage for 2-3 nights or, if the task has turned out to be a bit unpleasant, even a few days more, and then look at it again with fresh eyes. That helps me to avoid being overly negative; in fact I tend to start out rather bluntly and then, with some distance, rephrase and expand my comments to be more polite and constructive.

That being said, if the manuscript is nearly flawless or totally unsalvageable I usually finish my review very quickly. If I remember correctly my record is something like 45 min after being invited to review, because the study was just that deeply flawed. In that case I saw no reason to spend a lot of time on trying to develop a list of minor suggestions.

More generally I have over the years come to the conclusion that it cannot be the role of a peer reviewer to check if all papers in the reference list have really been cited or to suggest language corrections in each paragraph, although some colleagues seem to get a kick out of that. If there are more than a handful of language issues I would simply say that the language needs work instead of pointing out each instance, and if there are issues with the references I would suggest the authors consider using a reference manager such as Zotero, done. Really from my perspective the point of peer review is to check if the science is sound, and everything else is at best a distant secondary concern.

At any rate, after having slept over the manuscript a bit I will return to it and write the general comments out into more fluent text. I aim to do the usual sandwich: start with a positive paragraph that summarises the main contribution made by the manuscript and what I particularly like about it. If necessary, this is followed by something to the effect of "nonetheless I have some concerns" or "unfortunately, some changes are required before this useful contribution can be published".

Then comes the major stuff that I would suggest to change, delete or add, including in each case with a concrete recommendation of what could be done to improve the manuscript. I follow a logical order through the text but usually end with what I consider most important, or repeat that point if it was already covered earlier. To end the general comments on something positive I will have another paragraph stressing how valuable the manuscript would be, that I hope it will ultimately be published, etc. Even if I feel I have to suggest rejection I try to stress a positive element of the work.

Finally, and as mentioned above, there is the list of minor suggestions. Most other reviewers I have run into seem to structure their reports similarly.

When submitting the report, however, one does not only have to provide the text I have discussed so far, although it is certainly the most useful from the authors' perspective. No, nearly all journals have a field of "reviewer blind comments to the editor", which I rarely find necessary to use, and a number of questions that the reviewer has to answer. The latter are typically on the following lines:

• Is the language acceptable or is revision required?
• Are the conclusions sound and do they follow logically from the results?
• Are all the tables and figures necessary?

And so on. The problem I usually have is that these questions are binary, but I would like to write something like "mostly yes, except for that instance here which really needs to be dealt with".

The taxonomic impediment as illustrated by journals' criteria for the acceptance of manuscripts

About two weeks ago I learned from a co-author, who in that case is the corresponding author, that a certain systematic botany journal would consider our manuscript unacceptable no matter how much we improved it simply because it was out of scope. You see, our work was only "revisionary", as in dealing with species delimitation, and it would have to be a phylogenetic study to be acceptable. A few thoughts:

I do understand why higher-profile systematics journals do not accept descriptions of taxonomic novelties that take a qualitative approach like "hey, that looks different to that other species", or papers that merely validate taxonomic changes based on evidence presented elsewhere. But I completely fail to understand what the problem is with papers that, as in our case, use integrative, quantitative analyses of morphological, genetic and environmental data to resolve difficult species complexes. I would love to understand how a phylogenetic study is more serious than that. The conservation impact is, for example, much higher in studies finding a previously unrecognised, rare species than in those that only change the circumscription of a genus.

The journal in question is TAXON. Think about it: a journal literally called "taxon" has decided to accept no more taxonomic studies going forward. No word on when Evolution will stop accepting studies dealing with evolutionary biology, or when Heredity will reject all manuscripts dealing with genetics.

Note also that TAXON is still the go-to journal for nomenclatural suggestions in botany. In the latest issue as of writing, for example, we find Brownsey & Perrie, "Proposal to conserve the name Asplenium richardii with a conserved type" and Dorr & Gulledge, "Request for a binding decision on whether Briquetastrum Robyns & Lebrun (Lamiaceae) and Briquetiastrum Bovini (Malvaceae) are sufficiently alike to be confused". Those papers are important and need a forum, and it is good that TAXON is that forum. But the same is true for revisionary studies, and I cannot help but feel that in terms of editorial policy accepting nomenclatural suggestions like these but not evidence-based revisionary studies is the equivalent of saying, "we don't serve alcohol to minors, but we make an exception if you are under six months old."

The general problem is that there are quite a few systematics journals that have made the same decision over the last few years. I have thought about what journals there are in my field, and I cannot at the moment think of one with an impact factor of more than approximately one that would still accept revisionary studies. Most of the options are local journals published by university or state herbaria, usually named after a 19th century taxonomist or a plant genus, that either do not have an IF or one that is around 0.3-0.7. As valuable as those outlets are for publishing new species or smaller taxonomic revisions they just do not seem to be the right venue and have the right audience for a two-year study using complex analyses of genomic data. Surely if we have molecular phylogenetics journals with IFs of 2 to 5 it should be possible to have journals in that range that publish what might be called molecular taxonomy? If not, why not?

If we do not have journals like that, if the only option for a researcher doing species delimitation with cutting edge, expensive methods is to publish in journals that a job or promotion committee might consider to be a liability to publish in, then it is no wonder that fewer and fewer people will be willing to figure out how many and what species there are on our planet, and that those who are willing to do it will find it hard to get a job in academia. That is known as the taxonomic impediment: There are still many species to be discovered before we are even in a position to know what we need to conserve, but the number of people, institutions and resources assigned to that task is dwindling.

Which brings me to the final point. A year and a half ago I wrote about a study published in Systematic Biology that claimed to have disproved (!) the citation impediment to taxonomy. The authors actually mentioned the non-acceptance of taxonomic papers by high impact journals as one of the arguments underlying the citation impediment, but then argued the latter does not exist. As I wrote at the time, my interpretation of their paper is that they reached their conclusion based on defining phylogenetic studies that happen to include a taxonomic act as taxonomic papers, and then comparing them against phylogenetic studies that do not include a taxonomic act. For example, they had the Botanical Journal of the Linnean Society in their data, which at that moment had officially stopped accepting taxonomic papers for several years. In other words, the study's approach seems to have been the equivalent of examining discrimination against women by comparing men who grow a beard with men who do not grow a beard.

In the light of my recent experience, that paper now seems even more upsetting.

Journal citation styles

Among the minor frustrations of publishing scientific papers is formatting, in particular of reference lists. Every journal has its own format, with only sister journals published by the same society generally using the same. Here just a sample of the same paper as formatted for the reference lists of four journals I have published in:

Breitwieser I, Ford KA, Smissen RD (2010) A test of reproductive isolation among three sympatric putative species of Craspedia (Asteraceae: Gnaphalieae) at Mt Arthur in New Zealand. New Zealand Journal of Botany 48, 75–81.

Breitwieser, I., Ford, K.A., Smissen, R.D. 2010. A test of reproductive isolation among three sympatric putative species of Craspedia (Asteraceae: Gnaphalieae) at Mt Arthur in New Zealand. New Zealand Journal of Botany. 48, 75–81.

Breitwieser I., Ford K.A., & Smissen R.D. (2010) A test of reproductive isolation among three sympatric putative species of Craspedia (Asteraceae: Gnaphalieae) at Mt Arthur in New Zealand. New Zealand Journal of Botany, 48, 75–81. 

Breitwieser, I., Ford, K.A. & Smissen, R.D. (2010) A test of reproductive isolation among three sympatric putative species of Craspedia (Asteraceae: Gnaphalieae) at Mt Arthur in New Zealand. New Zealand Journal of Botany 48: 75–81.

The problem is, of course, that it would be extremely fiddly to reformat every journal name into italics or every volume number into bold every time that needs to be done. Wouldn't it be nice if publishers agreed on a standard, at least across the same field of science?

This is where reference managers come in. Perhaps the best known currently is the commercial software EndNote, and I could get it at work. But because I want something that I can also use on my own computer at home I prefer to use the open source software Zotero, which works through the browser. It has got very user friendly and flexible, and as far as I can tell there are only two downsides left.

The first is that when I need an in-line reference I only have the options of inserting "(Smith 2010)" or "(2010)", the latter by ticking the 'suppress author' option. It would be neat if it could also offer the option of "Smith (2010)", as EndNote apparently does.

The second is quite simply that the number of available citation styles - see above - is still a bit limited. For example, the Zotero Style Repository does not currently appear to have:

• Botanical Journal of the Linnean Society
• Systematic Botany
• Taxon

These are not exactly obscure under any measure, they are three of the most important plant systematics journals globally. Admittedly the Bot. J. has the same one as the Biological Journal published by the same society, but Taxon for example has a very unique style.

There appear to be two ways of building one's own style. As they have to be coded in the CSL format, one can learn that format and 'program' a style. Obviously that is a bit of a barrier; it amounts to learning what is pretty much a super-specialised, extremely cryptic scripting language that the average scientist will not need for anything else, ever. I at least do not really consider learning all the commands an efficient use of my time.

Luckily, a Visual CSL Editor has been made available (apparently by Mendeley and Columbia University?), and yesterday I experimented with it a bit. The idea is to start by searching the style database of the site for an existing style closely matching the desired one, changing it as desired, and exporting it to disk as a CSL file. It can then simply be dragged into an open browser window to install it in Zotero (if Zotero is what you are using).

This is much easier than scripting a style but also has its hurdles to overcome. The thing looks like this:

In the upper right field we have the way the example citations would be formatted under the style, so that is clear.

Below that is a field showing all the current settings for the part of a reference that we are currently working on. This is where it gets a bit more difficult. In the above screenshot, we can guess that delimiter is what separates author names, and that prefix and suffix are elements that would precede or follow any author name. In other cases it is harder to figure out what something means.

But the main problem is navigation. Looking at the field on the left we see a bewildering tree of fields that may or may not need to be edited. What is more, many of them have the same name, for example "names". Here we realise, for example, that our guess from the previous paragraph was wrong. Although we are in a field called "names" (directly under another field called "names", nested two levels under yet another field called "names"!), the delimiters, prefixes and suffixes are actually for the collective of all authors of a paper. Why do they have to have a delimiter between each other when they are actually delimited by year of publication, name of paper, journal name, volume number, page numbers, and line break? God and the people who came up with the CSL format will know.

In other words, there are so many more fields that I can ignore that I find it difficult to find the ones I do need, and they have the same names and are often directly next to each other. On the other hand, I was unable to figure out how to tell the style that it should use "& al." instead of "et al.", as required by the journal Taxon. Maybe that is impossible, or maybe I just can't find the option?

Of course, this is available for free and much, much better than scripting. I am extremely grateful to have this option. But still, it is clear why few people would find themselves able and the time to write their own styles...

A fast alert or a slow editor?

Okay, that is a first. My co-author and I just learned that we got a paper accepted not from a message from the editor but from my Google Scholar Alert on the topic we are working on. Apparently the journal published the abstract as "just accepted" on its website before even telling us that the manuscript was accepted. Weird.

Cracked explains what is wrong with science - or tries to

Sometimes I really wish people who try to write very confidently about a subject would at least make a token effort at trying to research the topic first. Or, failing that, perhaps because they don't have the time, at least try not to sound so damn Dunning-Kruger about it.

The listicle website Cracked recently tried to explain why science is broken, in six bullet points (hat tip: RationalWiki):

6. Publication bias towards positive results.

5. Scientists not having to make the data underlying their studies available.

4. Scientists having to pay publication fees, which creates perverse incentives and restricts access to publication venues for scientists from poorer countries.

3. Big publishing companies earning obscene profits by charging journal subscription fees in addition to publication fees, thereby restricting access to the results of research.

2. Scientists being forbidden from giving out reprints of their papers because they signed over the copyright to aforementioned publishers.

1. Predatory journals publishing nonsense science.

Except for the first point (#6), all of this is either misleading or self-contradictory.

Variants of peer review

In my field of science, peer review generally takes the following form: You submit a manuscript to a journal, it is reviewed by two to three colleagues, and then the editor of the journal decides whether to publish, whether to request changes before re-examining the paper for publication, or whether to reject. The editor and the peer reviewers know who wrote the paper, but you don't usually know who the reviewers were.

There are other ways of doing this; they have their own advantages and disadvantages, but I definitely know how I would change things if I could.

One often heard suggestion lately is that peer review should be "open" (or public); that is, the authors should be informed who the reviewers were. As far as I can see, there are two main considerations behind this. First, that this will make it harder to be unfair and rude, which is of course much easier to be under our current system. Second, a general and in this particular case unwarranted infatuation with the concept of openness, as in open source software or open access publishing.

Because there is one very simple problem: If the author can see who suggested their paper be rejected, will that not have a severely chilling effect? Imagine a postdoc reviewing the manuscript of a very influential professor, for example. Will they dare to say something negative if they know their name will be connected with it, be it ever so justified? I am fairly sure that I at least would say no considerably more often if I were to asked to review papers under an open system. Who knows when I would offend somebody who has to decide about my grant application a year later?

Scientific fraud doesn't make sense to me

Having just read about an alleged data manipulation scandal in political science in the USA, I wonder once more why people are committing scientific fraud. I get plagiarism in the humanities, although of course the risk of getting caught is growing ever greater considering the ease with which text searches can be conducted these days. Still, I realise why people might be tempted into doing it, especially if they are not pursuing a career in scholarship, just want a few extra letters to appear on their business card, and are unwilling or unable to make the necessary effort. Inventing or manipulating empirical data as a career scientist, however, I don't really understand.

A metric of Twitter and blog hits for scientific articles

On this blog I have sometimes mentioned, and complained about, the influence that the obsession with article citations and journal impact factors (IF) has on scientific publishing, scientific careers, and the hiring choices of scientific institutions.

In short, instead of actually reading and understanding scientific papers, many people 'assess' their value by looking at how often they have been cited. Instead of reading and understanding their work, many people, even members of search committees or advisors of funding agencies, 'assess' scientists by looking at how often their papers have been cited. And instead of reading and understanding the articles published in them, many people 'assess' scientific journals by looking at how often their average article is cited within the first two years after publication.

And as mentioned before, this approach systematically favours areas of science that have a quick turn-around and lots of practitioners able to cite each other whereas it systematically disadvantages areas of science where many publications are written for long term use, published in books as opposed to journals, and while very useful to many people may not even be meant to be cited. Such as the floras and monographs produced by taxonomists, for example.

But of course, once you think you know what bad looks like somebody will introduce you to something worse.

Scottish journal of EVERYTHING

One of the things often pointed out by Jeffrey Beall is that questionable publishers like to give their for-profit journals very wide scope. They presumably do so in the hope of being able to earn article processing fees from as many areas of research as possible.

Yesterday I received another scientific spam e-mail that takes the cake in this regard. Behold the call for papers from the Scottish Journal of Arts, Social Sciences and Scientific Studies:

Okay, the first thing one notices is the green ink. But apart from that, the funniest aspect is the ridonculously long list of areas of research that is given as the scope of the journal. It is an alphabetic list of everything they could come up with. Botany and dance; philosophy and finance. Yup, that sounds legit, where do I sign up?

The name that they invented for the supposed managing editor is also pretty interesting, by the way. Robest Pual Ashcraft. I guess they were going for Robert Paul Ashcroft but put a typo into every word? So I may be mistaken but this could be an indication of something being off; would not be surprised to learn the journal is really run out of a garage in Pakistan or Nigeria.

More editorial blindness please

Apart from phylogenetic systematics, another topic that I seem to be writing a lot about is scientific publishing, and there especially about peer review and the trade-offs between the system we have now and the potential problems of open access.

Just today I had a short conversation at work that made me think about what I would change if I were made the dictator of science publishing. Really there are two answers, one dealing with the commercial publishing versus open access issue and one dealing with the gate-keeping and peer review issue.

The first one is simple. If I could reorganize scientific publishing, I would run it all through non-profit public utilities. There would be no publication fees for authors and no access fees for the reader. Everything would be open access and financed directly from taxes. Note that scientific publishing is, with the exception of a few advertisements in the journals, also entirely financed from taxes now, only it happens indirectly, through tax-funded libraries and tax-funded scientists paying publication costs. So the only difference would be that we would stop wasting taxpayer money to provide profits for commercial publishers. (Sad, I know.) As an added bonus from the open access perspective, with no author fees there would also be no incentive to accept crappy papers.

That would still leave the second issue, which is essentially one of quality control. If everybody could get published for free, how do you know what research is sound? As mentioned before, I do not think much of post-publication review because it is not clear how an author would be motivated to correct a manuscript that has already been published. It would also make it harder for newcomers to get their work noticed. Under the current system, they can get a good manuscript into an influential journal and people will read it. In a system relying on post-publication review, anybody without a famous name is likely to get lost in the flood of useless papers using the same key words.

So I would, if I were asked to reorganize things, probably try to keep our current peer review practices pretty much in place: Have a few colleagues assess the manuscript and make an editorial decision whether it (a) is sound science and (b) has been submitted to the right journal or archive. It isn't perfect but it allows me to put more trust into a publication in a high ranking journal than, say, into what somebody wrote on their blog.

However, one thing that bugs me mightily about the procedures in my area of science is that while the authors of a manuscript generally do not know who the peer reviewers are, the peer reviewers and editors can see who the authors are. Some people even call for the abolition of anonymous peer review but that would make it harder for young researchers to critically evaluate the manuscripts of influential older colleagues, especially those having a say in funding decisions.

I would go exactly the other way: peer review should be double-blind. If the peer reviewers and the editors don't know who the authors are, many possible sources of bias disappear. They will not as easily be able to wave through a sub-par manuscript from a big name in the field. They will not be able to be negatively predisposed towards articles coming from specific countries or specific universities. They will not be able to see whether an author has got a female or foreign sounding name. And so on.

But is that not technically difficult? It should not be, anymore. Most journals these days have online submission systems for manuscripts. Of course the manuscript management software needs to know the contact details of the authors to communicate with them, but there is no reason why even the chief editor of a journal should know it before they have decided whether to accept the manuscript. The following set of rules would appear workable:

• The manuscript management software hides the names and affiliations of the authors from the editor(s) in charge of deciding whether the manuscript ultimately gets accepted. It organizes all communication between editor(s), reviewers and authors anonymously.
• To ensure that the manuscript is not sent out for review to the authors themselves or to their close collaborators, a different editor or editorial assistant, in other words somebody who does not decide on ultimate acceptance, chooses the peer reviewers.
• Authors may not disclose their name on the manuscript. If they do so, it is rejected outright. They can be invited to resubmit the manuscript without their names on it, but then a different editor is assigned to it who has not yet seen their names.

None of that is all that hard to achieve; it would merely be a matter of changing journal policies.

Inflation in citation metrics

On Jeffrey Beall's Scholarly Open Access blog I recently got involved in a short discussion on steadily rising citation metrics. He discussed a citation metric that he considers suspicious, and one of the reasons he mentioned was that it always seems to be going up. It is well possible that the metric in question is manipulated - or not, I would not know and have no opinion either way - but it is important to realize that some degree of inflation is unsurprising and not necessarily, on its own, an indication that something is off.

The most highly regarded of them all, the Impact Factors (subsequently IF) from Thompson Reuters' Journal Citation Report, show the same trend. Yes, there are some losers and a lot of stagnation especially at the lower end of the spectrum, probably because the editor of a very small journal is only willing to invest so much time into it, and if you only compare across two years or so you will get a lot of noise. But if you take a bunch of decent, mid to high level journals from my field and compare their current IFs with what they had a few years back you will see slight increases very nearly across the board.

Here a few semi-randomly chosen journals from my field and their change over five years - meaning I looked up the IF 2007 and 2012 for a few well-regarded journals whose names immediately popped into my head:

Am. J. Bot. +0.074; Aust. J. Bot. +0.217; Aust. Syst. Bot. +0.488; Bot. J. Linn. Soc. +1.514; Flora +0.559; Folia Geobot. +0.432; Syst. Bot. -0.345; Taxon +0.258; Trends Plant Sci. +2.813

The only one bucking the trend here is Systematic Botany, which does seem to have moved from publishing a lot of phylogenies to a lot of alpha taxonomy lately, and the latter unfortunately and unjustly does not bring in a lot of citations. Conversely, the Botanical Journal went up a lot since they stopped accepting purely alpha taxonomic papers and they published the last Angiosperm Phylogeny Group update, a landmark paper that was guaranteed to get a ton of citations. Trends also looks like it got a big increase but it is not that much relatively speaking; as a review journal it was always by far the highest on this short list.

So how is that possible? Why do the citation metrics appear to consistently increase over the years? Is there some manipulation going on?

What makes reviewing a paper enjoyable?

One more about peer review. To get our papers published, we all have to go through it, and the pleasantness of that experience can vary depending on the reviewers you run into.

But the same goes for the other side. In my role as peer reviewer, I find myself sometimes enjoying it very much and sometimes hating it. Apart from the obvious fact that it is no fun to do it if you have too much work already, what makes the process enjoyable?

Well, papers that are fun to review are the ones that inspire, that present new methods or insights, that make one think "I wish I had come up with that", while bad papers are often formulaic and uninspiring. As for quality, it is a pleasure to review a well written, concise and coherent, high quality paper because it is just a good read and you do not have a lot to criticize.

But interestingly, at least to me there is another peak on the fun/quality graph at the bad end of the spectrum. In a perverse way, reviewing becomes enjoyable again when the manuscript is sufficiently atrocious.

So really the papers that I enjoy reviewing least are somewhere around the middle. The ones that contain reasonably competent but completely uninteresting research, the ones that could actually be good if the authors knew how to write a coherent sentence and develop a clear line of thought, and, crucially, the ones containing overall sound science but a suffocating number of individually minor mistakes. In other words, those that have potential merit but force you to write pages and pages of suggestions on how to make them publishable.

The really hopeless manuscripts make for short reviewer's reports, just like the really good ones. If the science is completely bonkers, it is easy to decide what to recommend. If the entire text is beyond repair, you don't need to bother with making suggestions for individual sentences any more. But most importantly, if a paper is sufficiently bad, reading it can be a very invigorating experience. You may laugh, you may pull your hair, but at least you emote instead of getting bogged down in drudgery.

Failures and virtues of peer review

For a few days now, scientists have been discussing a small experiment conducted by a Science correspondent. He invented several variations on a bogus study in cancer research that contained several fundamental flaws and sent them out to 304 open access journals. 255 of them made a decision, and of those, 157 (or slightly more than 60%) accepted them.

Predictably, this experiment is being discussed along what we might call partisan lines. Those skeptical of the OA model see this as one more illustration of the problems of a system in which journals earn their money based on the number of accepted papers, pretty much regardless of their quality. Those advocating the OA model consider the affair a piece of propaganda for the traditional commercial publishing industry and stress that the experiment did not feature a control group of subscription based journals to see if they would have a lower acceptance rate.

I have already written about my concerns about the OA model so I will not repeat myself here. Note simply that I would very much prefer a system in which the rainbow press of science would be seen for what it is, in which all major journals are run as non-profit public utilities, and in which the results of scientific research are as available to the public as access to publishing venues is available to those who cannot afford $1,300 in article processing fees. I am merely worried about the incentives publishers operate under when using the OA model.

Instead, I wonder about the general issue of peer review. To express his opinion that the problem is not with OA but with peer review in general, Michael Eisen uses his linked blog post to call peer review a "joke", and similar sentiments are being expressed elsewhere. Others stress that this process is the gold standard that allows us to distinguish real scholarship from crankery. What gives?

Manuscript writing: what to avoid

Previously I wrote a post about peer reviewers from hell. But of course I am not only an author of papers but also a peer reviewer myself, so perhaps I should write something from that perspective. What are the things that annoy me about manuscripts? What mistakes or faults do I appear to encounter particularly often in my field?

Do as I say, don't do as I do

Lately awareness seems to be rising that impact factor mania has reached ridiculous proportions, and interestingly it appears as if one of the highest impact scientific journals is trying to hop onto the train. The American Association for the Advancement of Science, the publisher of the journal Science, is one of the signatories of the San Francisco Declaration on Research Assessment that quite sensibly urges researchers and their institutions to stop judging fellow researchers by the impact factor of the journals they publish in.

Perhaps to promote the declaration, the editor-in-chief of Science followed up with an editorial making the same argument, explaining how destructive the obsession with impact factors is and that they were never meant to evaluate people. Now another editorial has been published in Science bemoaning the emphasis on impact on a more general level, such as the increased push for applied as opposed to fundamental research, and the short-term-ism and risk avoidance that come with such a focus.

Basically I wholeheartedly agree with all of this. But here is the thing: I take issue with being lectured about impact mania from the pages of a journal that embodies all that those complaints are about. It's about the messenger, not the message. This is a journal that is well known for accepting publications not primarily on quality but on expected impact, and for rejecting everything out of hand that is unlikely to be a big thing over the next few years.

Don't get me wrong - I am not saying they don't have a generally good peer review, or that they publish more bad papers than would be expected from rare and unavoidable failures of the review process as they can happen anywhere. No, what I mean are those many, many papers, probably the majority of the submissions, that are never even reviewed because some managing editor turned them down on sight. Does anybody really assume they do that because they can immediately see that the paper is poor quality, or that the analyses are wrong? No, they do that because the topic is not sparkly enough, because if they published high quality papers that aren't sparkly enough then their rank as one of the two or three highest impact journals might suffer. Just you try submitting the most excellent taxonomic revision or seed germination experiment you have ever seen to Science, and see how it fares against cancer research, the discovery of a new exoplanet or even just the discovery of a new hominid bone fragment. Go on, I dare you. That is the name of the game.

And so it is rather galling to read these editorials in, of all places, Science, because to me they sound like "do as I say, don't do as I do". And honestly, gems like this, where the editorial argues against impact factor mania with the observation:

And it wastes the time of scientists by overloading highly cited journals such as Science with inappropriate submissions from researchers who are desperate to gain points from their evaluations.

Cry me a river. A journal that advertises itself with a tag line as nauseating as "The World's Leading Journal of Original Scientific Research, Global News, and Commentary" on its website does not really get to complain if people consider it a feather in their cap to get an article in there.

Authorship of scientific papers

One topic that comes up among scientists ever so often is that of paper authorship. There are several issues here but clearly they all revolve around the underlying question of what it means to deserve a co-authorship. First, the situation.

Where to submit a manuscript

In the past few weeks I have repeatedly poked fun at science spam advertising poor quality for-profit "journals", and I have cited an article that claimed it would become difficult for some universities to evaluate scientists' resumes due to the meteoric rise of legit-sounding pretend-journals. I really do not "get" any of this.

That is to say, I do not get why a significant number of people would be willing or foolish enough to submit their manuscripts to pretend-journals, by extension how these pretend-journals can be profitable enough to the spammers for them to continue spamming (many of these "journals" have few to no articles and sometimes the few articles they have are plagiarized from elsewhere so presumably they cannot earn a lot of fees), and finally why anybody would find it hard to know the good journals in their field.

When you start out as a scientist (or scholar) you generally obtain a decent university education and write your thesis under a hopefully competent supervisor. The supervisor will be able to teach you not only how to write a decent paper but also to what journal to submit it. By extension, they will teach you what the good journals in the field are and what kind of "publication strategy" is expected of you if you want to become a career scientist/scholar in the field. In addition to your supervisor, can talk with other colleagues and learn from them. Finally, the papers you needed to read to prepare for your thesis as well as the reference list of said thesis will also give you a clear idea of where the relevant literature in your field is published.

Reference managers

So far I have lived in the grey zone where the investment that would have been necessary to start using a reference manager appeared just a wee bit larger than the hassle of not using one. But after this Monday I am coming round to it.

The problem that reference managers solve is not so much having to add references to a manuscript per se - one still has to remember which was which to selected the right one, after all - but rather constantly having to format and reformat reference lists at the end of manuscripts. Every journal has its own idea of how a reference should look like. A few examples:

Smith A, Miller B, Ngyuen C, Kim D, 2009. Some paper title. Journal of Research 15: 42-48.

Smith A, Miller B, Ngyuen C, Kim D (2009) Some paper title. Journal of Research 15: 42-48.

Smith A, Miller B, Ngyuen C, Kim D (2009) Some paper title. J. Res. 15:42-48.

Smith, A., Miller, B., Ngyuen, C., Kim, D. (2009). Some paper title. Journal of Research 15: 42-48.

Smith, A., Miller, B., Ngyuen, C., Kim, D. (2009). Some paper title. Journal of Research, 15, 42-48.

Smith, A., B. Miller, C. Ngyuen & D. Kim (2009). Some paper title. Journal of Research 15: 42-48.

Smith A et al., 2009. J. Res. 15: 42-48.

And all possible combinations of the above and more! You can imagine that having to manually format all this for dozens of references gets tedious at some point. A reference manager does all that automatically for you, but of course you first have to import into it all the references you need and you have to provide it with the correct style information for the journal you want to submit a given manuscript to. And because I hopped so much from one study group to the next and from one methodology to the next during my career it never seemed profitable to import references and double check them to use them only in one, two or perhaps three manuscripts.

Because I now expect to work on the same plant family for a long time, and after having gone through a particularly annoying bit of reformatting, I have decided to start using the freeware reference manager Zotero with which I had already dabbled a bit a few years ago. Still, the trouble is the same as what put me off then:

If you don't want to enter all references manually, the easiest way is to import them from a website. Now Zotero is actually really clever at this: if your browser displays search results from Web of Knowledge or Google Scholar, it can open a list of boxes for you to tick and import several references at the same time. Unfortunately, Web of Knowledge finds only a ridiculously small percentage of the references I need and Google Scholar is stupid. I spend a distressing amount of time correcting what is imported from GS because it may provide first names as part of the family name, cities as publishers, paper titles or author names in ALL CAPS, and so on. Still, if I use the same references often enough it will hopefully be worth the effort.

The second problem is styles. Here it would probably pay off if I just invested the money to get EndNote instead, but I want to give Zotero a chance, and the Zotero style repository is simply missing most systematic botany and plant taxonomy journals. I assume I will in time have to figure out how to edit styles for my own purposes.