Over the last few years I’ve had to review tens of software application papers.  I’ve also been involved in writing a few of these (and have also conciously decided not to write papers for some of our software due to the amount of hassle involved).

In many ways software papers are some of the worst to tackle as a reviewer.  If you’re going to do your job properly you will need to get the software working, run it on some real data and evaluate the results.  For many application types this can be an arduous process, potentially involving setting up an entire development system and involving lengthy data processing so it’s not something to undertake lightly.

During the process of reviewing many applications I’ve found authors hitting the same problems over and over, and I’ve retrospectively realised that some of our previous software papers had also failed on these points, so this seemed like a good opportunity to write up some of these, both to remind myself in future, and because they might be of use to others.

What is the overall aim here?

The overall aim in a software review is to make the reviewer’s life as easy as possible.  You want to discourage them from giving up and simply saying your software doesn’t work.  As a reviewer you are often coming to new software with little to no prior knowledge, and many packages you review end up being very poor quality, so having a package which makes a good first impression can make a big difference.  As a reviewer I won’t ever approve a paper for a piece of software I couldn’t get to work, and I really don’t want to approve anything which I don’t think most other people could get to work, so your aim as a software developer is to convice me that you’ve thought about others when developing your package.  If you can make me feel well disposed to your software then the rest of your paper review will go much better for you.

Getting the software

The first thing anyone is going to have to do is to download your software from the project page.  This might be a github page, or a dedicated website for your program – it doesn’t really matter.  This is your first chance to create  good impression with the reviewer so getting this right gets you off to a good start.

Make the page welcoming and friendly

Put some effort into the front page of your project.  Being faced with a default github repository page with a brief, text-only README.md is a sure sign that the authors haven’t thought about other users when writing their software.  Your front page doesn’t need to be complex – it just needs to be friendly, welcoming and functional.

Things you should have on your front page:

1. The name of the project somewhere big and at the top.  Make it obvious that they’re in the right place
2. A brief description of what the program does and who it’s for.
3. A really obvious pointer to where and how to get the software.
4. A really obvious pointer to how to contact you with questions or problems.

Make it really obvious which file to download

If you offer several files to download on your project page make sure it’s really obvious which one they need.  Put the most common ones (ie binary distributions) at the top, and hide the ones which are less likely to be needed further down (eg source distributions) and label them clearly.  In our source distribution for FastQC we even have the launcher script check whether someone is trying to run the source distribution as if it was a binary and throw out a custom error message!

If you have current releases, old releases and development version all available then put the one you’d want most people to get front and centre.  People looking for something different will hunt it out, but make sure most people will trip over the right download.

Installation

By far the most common place that software falls down is that the user simply can’t get it running.  As an author you therefore want to think about the ways in which you can make life easier for the end user who just wants to use your software.

Provide pre-compiled, platform specific binary packages if possible

The best experience for an end user is if the installation process is as simple as downloading a tar.gz (linux), zip (windows) or dmg (OSX) file, uncompressing it and extracting the contents and that’s it.  Automated installations also work well (pip install yourprogram for example).  This level of simplicity obviously isn’t possible in all cases, but you want to try to get as close to this as you can.  Assuming you can’t get to quite this level of simplicity then think about the other steps people are going to have to go through and what you can do to make the install easier.

Make your dependencies really clear, and minimise them where possible

Modular coding and code re-use are one of the cornerstones of large project development which reduce the burden on your as a coder, but as an end user they can also be one of the most difficult things to deal with.  Installing a package which depends on loads of external libraries, packages and tools will create a barrier to the installation of your software which some users will not get over to try to make life easy for them.

Remove dependencies where you can

Some dependencies are going to be absolutely necessary, but a surprising number are somewhat trivial.  Major functionality will often require the use of external libraries, but this can carry too far.  Recently loads of major javascript based projects were brought down due to the removal of an 11 line package which provided a function to pad a string.  When writing your code try to think about the dependency choices you make, and consider if you can eliminate them, or choose a package which might be distributed by default.

Consider bundling dependencies with your project

Depending on the nature of the environment you’re working in it may be possible to either embedd or bundle the dependencies for your project in your binary distributions.  This gives you the advantage of modular coding, but makes life easier for the end user in that they don’t have to worry about finding and installing the dependencies for themselves.  In our SeqMonk application we rely on picard, apache commons math and jama, but we put a copy of all of these libraries in with our distribution so the end user doesn’t have to deal with finding and installing them separately.  We drew the line at bundling in a Java environment (which we could have done) and end users do need to do that, so there is a balance to strike here.

Make your dependency versions as lax as possible

The cluster we run at our institute is based on ROCKS6 (the latest version of ROCKS), which is itself built on CentOS6, which is now a few years old, so we often don’t have the latest versions of libraries and compilers available to us.  There’s nothing worse than looking at an install document which lists very recent versions of a load of libraries and tools, and knowing that you don’t have any of these immediately available and that you’re going to have to build a custom environment just for this tool to work.  It’s more frustrating when you find that the versioned dependencies listed were simply what the author had on their machine, and not the absolute minimum which was required.  Be clear about what is absolutely required and what you would recommend.

When writing your software please check whether you really need the latest versions of a given library, or whether the old version will just work, or could be made to work with minor coding changes.  If you absolutely need a newer version then so be it, but remember that this could be a pain for people who, for very good reasons, are working on older operating systems.

Minimise configuration and try to put in sensible defaults

Ideally try to make your application functional as soon as it’s installed.  If it requires conguration try to work this out automatically.  If your application needs post-install configuration then try to keep this as simple and obvious as possible.  Keep all of your configuration in a single place and make sure any configuration file designed to be edited by the end user is well commented to describe which bits they need to change and what the acceptible values for each option are.

Where possible try to make sensible defaults, or find a way to auto-configure as much as possible based on the users system and the installation location.  Don’t make every user set something up where nearly every time people will put in the same values.

If the user runs your program without the configuration being set up correctly be friendly.  Point them to the file they need to edit, or even have a setup process go through the configuration interactively with them.

Get someone else to test your installation instructions on a clean system

It’s really easy to miss out some vital steps in an installation because you don’t need to do them on the system you’ve been developing on.  Installation instructions only work if you can reproduce them on a completely blank system.  Ideally, get someone else to try the instructions for you an report back if they had any problems getting them to run.

Running the software

Having got the software installed you are pretty close to making the reviewer happy, but there are still a raft of things which can go wrong when trying to actally run the program.  You can’t always prevent people from providing incorrect data, or otherwise messing up your program, but you can make the experience either good or bad when problems arise.

Make the program do something sensible when you run it with no arguments

Often the first thing I’ll do after I’ve installed a program is to just run it to see what happens.  I’m really only testing that it’s found and executes, but you should try to make sure that your program deals with this nicely.  It’s somewhat disconcerting if what you get from running the program with no arguments is a crash or a stack trace.  Even doing nothing isn’t very friendly – give the user a clue.

One option in this situation is to display a help page, but this might be a bit much.  My preferred option would be to put out a short error saying that essential options were missing, showing a short usage summary and saying what flags to add to get the full help page.

Have command line help accessible from within the program

If your program takes command line arguments then make sure that a help file describing these is available from within the program.  Generally this should be accessed by running program -h or program –help.  It should have a command summary at the top showing which arguments are mandatory before showing a full list of the optional arguments, for each one saying what it does, why you might want to change that and what the default action is if that options is not specified.

Try to catch as many predictable errors as you can and have friendly error messages

Poor error handling is the bane of many bioinformatics projects.  In our own code a surprising amount of code is simply defensive coding against things going wrong.  The moment you release your software beyond your own group you will inevitably find conditions you never expected and your software will fail.  Again, in and of itself this isn’t a problem (up to a point), but how you handle this is the key.

The first thing to say is don’t let the reviewers be the first external people to ever run your software.  Things will go wrong when other people try to run it and you want to get those ironed out with friendly faces who are prepared to go back and forth a few times with you until it’s working.  In general reviewers are prepared to expend a bit more effort than a normal user to make things work, but are probably a bit less likely to contact you about problems they have than a normal user.

Where possible try to make your error messages informative to the end user.  Make it clear whether the error they’re getting is something they can fix by ammending their input or options, or is something which is probably internal and your fault.  Make sure the error you want them to see appears at the end of any text which is produced.  We often see programs which print the help page if a command has a problem, but this can push the original error message off the top of the screen so the user never sees it.

Try to provide the user with an option to report the error to you – this could be an email address, a pointer to a bug rerpoting system, or even an automated submission tool.  We’ve actually put automated bug submission tools into some of our software and they’ve proved hugely useful in making the code more robust and fixing real bugs.  Users are usually amazed and hugely appreciative if you actually reply to them with a fix for a problem they reported.

Try to make it clear what your program is doing

Having a program which you run and then just sits there silently for an extended period without giving you any indication that it’s working is quite disconcerting.  If there is an output file you can watch growing then at least you know something is happening, but many programs can sit for quite a while with no obvious sign of activity.

I would recommend having enough progress messages from your program that the user doesn’t get concerned that it’s just hung and be tempted to kill it.  You don’t want thousands of messages scrolling past, but some occasional progress messages to say what the program is doing, and to show any relevant information (parsing file3.txt for example) will help them to be confident that it’s all going well.  As a bonus this type of information will also be invaluable if and when they hit a problem so you can easily see how far the program got, and whether it appeared to have done anything sillly before the point at which it finally failed.

It’s a good idea to allow a flag (–quiet) which supresses all non-error output if the user doesn’t want it, and maybe another one (–debug) to give much more verbose output, but keep the default at a respectfully chatty level.

Make sure your program uses error states appropriately

It seems trivial, but make sure that if your program completes successfully that it returns a 0 exit status and if it exist in an error state that the exit is non-zero.  It’s up to you whether you want to document different numerical exit codes, but at least making the program play nicely in a pipeline or cluster setup is good manners.

Providing information to make reviewing easy

Aside from coding and distributing your program well, there are some other things you can do to make it easy to review.

Always provide an example dataset

Your reviewer might not have a suitable dataset to hand to run your software, or they might find that when they try it with their own data it doesn’t work, but they still want to be able to go through the process of running the program successfully, or just validate that it’s their data and not the program itself which is the problem.  To this end you should always provide an example dataset which can easily be run through the program and which provides useful and interesting results to be able to interpret.  If this is small you can bundle it with the download, or if it’s too big for that (try not to make it too big) then you can put it as a separate download.  In either case it means there is no excuse for the reviewer to not have been able to run your program.  Try to keep the example fairly minimal so it doesn’t take long to download or run through the software.

Always provide example output

You should make it easy for someone coming to your program to see what it will generate without having to run it for themselves.  Providing some pre-calculated output is an easy way to let people review the results without having to set anything up.  If your software provides something more complex like a LIMS system or other database then have an example system set up (read-only if you prefer) so that they can poke around in it before having to commit to setting it up for themselves.

Have good documentation on the project page

Having technical documentation descibing the program setup and options is a minimal requirement.  What is far more useful for someone new to a program is to have a more in-depth user manual which not only covers the technical parts of how to run the program, but also walks them through a typical use case, illustrating the reasons for any choices made and discussing the pros and cons of different options.  You should try to include screenshots or text capture from the program alongside the instructions to give a clear walk though of the process.  If you are able to supplement this with screencast videos or more interactive illustrations of the software’s use then this all makes it far more appealing to all potential users, reviewers or otherwise.

Be responsive to feedback

It’s hard to get people to send feedback.  If someone tries your program and it doesn’t work then the easiest thing for them to do is just delete it and move on.  If they go to the trouble of contacting you then try to send them a timely response.  Even if you can’t immediately fix the issue then even an acknowledgement that you’re looking at it will be gratefully received.  Bug reports and suggestions from external users have been one of the biggest factors in improving many of the packages we have produced and understanding the needs of people outside your immediate circle will often help you improve your software in ways you would not otherwise have thought about.

Summary

None of these points will guarantee success with your software – ultimately it has to do something useful, but too often we see potentially useful software failing because of relatively simple issues which could easily have been addressed.  Thinking about the points above may get you through a review, they may save you a round of re-review or they may just make your software friendier to new users, but hopefully they will provide a better experience for both you and your users.