We recently hit a problem when trying to run a fairly simple script through grid engine. The script used an internal redirection to merge together STDIN and STDOUT before doing a final transformation on the result.
A simplified version of what the script did is something like this:
echo hello 2>&1 | sed s/^/prefix:/
Which should put “prefix:” in front of any output from either STDIN or STDOUT. Running this from the command line worked, but trying to run it through qsub failed.
$ echo "echo Hello 2>&1 | sed s/^/prefix:/" | qsub -cwd -V
Your job 201149 ("STDIN") has been submitted
$ ls -tr
$ more STDIN.e201149 Ambiguous output redirect.
After many experiments we found the cause for this, which is actually pretty simple, but since it didn’t turn up in a google search I thought I’d document it. The reason this type of redirection fails under qsub is that the default shell used by qsub is csh rather than bash, and the normal 2>&1 notation is a bash specific extension. You can therefore make this work by either using csh redirection, or specifying bash as your preferred shell. Either of the commands below would work:
echo "echo Hello |& sed s/^/prefix:/" | qsub -cwd -V echo "echo Hello 2>&1 | sed s/^/prefix:/" | qsub -cwd -V -S /bin/bash