Bash Co-Processes

One of the new features in bash 4.0 is the coproc statement. The coproc statement allows you to create a co-process that is connected to the invoking shell via two pipes: one to send input to the co-process and one to get output from the co-process.

The first use that I found for this I discovered while trying to do logging and using exec redirections. The goal was to allow you to optionally start writing all of a script's output to a log file once the script had already begun (e.g. due to a --log command line option).

The main problem with logging output after the script has already started is that the script may have been invoked with the output already redirected (to a file or to a pipe). If we change where the output goes when the output has already been redirected then we will not be executing the command as intended by the user.

The previous attempt ended up using named pipes:

#!/bin/bash

echo hello

if test -t 1; then
    # Stdout is a terminal.
    exec >log
else
    # Stdout is not a terminal.
    npipe=/tmp/$$.tmp
    trap "rm -f $npipe" EXIT
    mknod $npipe p
    tee <$npipe log &
    exec 1>&-
    exec 1>$npipe
fi

echo goodbye

From the previous article:

Here, if the script's stdout is not connected to the terminal, we create a named pipe (a pipe that exists in the file-system) using mknod and setup a trap to delete it on exit. Then we start tee in the background reading from the named pipe and writing to the log file. Remember that tee is also writing anything that it reads on its stdin to its stdout. Also remember that tee's stdout is also the same as the script's stdout (our main script, the one that invokes tee) so the output from tee's stdout is going to go wherever our stdout is currently going (i.e. to the user's redirection or pipeline that was specified on the command line). So at this point we have tee's output going where it needs to go: into the redirection/pipeline specified by the user.

We can do the same thing using a co-process:

echo hello

if test -t 1; then
    # Stdout is a terminal.
    exec >log
else
    # Stdout is not a terminal.
    exec 7>&1
    coproc tee log 1>&7
    #echo Stdout of coproc: ${COPROC[0]} >&2
    #echo Stdin of coproc: ${COPROC[1]} >&2
    #ls -la /proc/$$/fd
    exec 7>&-
    exec 7>&${COPROC[1]}-
    exec 1>&7-
    eval "exec ${COPROC[0]}>&-"
    #ls -la /proc/$$/fd
fi
echo goodbye
echo error >&2

In the case that our standard output is going to the terminal then we just use exec to redirect our output to the desired log file, as before. If our output is not going to the terminal then we use coproc to run tee as a co-process and redirect our output to tee's input and redirect tee's output to where our output was originally going.

Running tee using the coproc statement is essentially the same as running tee in the background (e.g. tee log &), the main difference is that bash runs tee with both its input and output connected to pipes. Bash puts the file descriptors for those pipes into an array named COPROC (by default):

  • COPROC[0] is the file descriptor for a pipe that is connected to the standard output of the co-process
  • COPROC[1] is connected to the standard input of the co-process.

Note that these pipes are created before any redirections are done in the command.

Focusing on the part where the original script's output is not connected to the terminal. The following line duplicates our standard output on file descriptor 7.

exec 7>&1

Then we start tee with its output redirected to file descriptor 7.

coproc tee log 1>&7

So tee will now write whatever it reads on its standard input to the file named log and to file descriptor 7, which is our original standard out.

Now we close file descriptor 7 with (remember that tee still has the "file" that's open on 7 opened as its standard output) with:

exec 7>&-

Since we've closed 7 we can reuse it, so we move the pipe that's connected to tee's input to 7 with:

exec 7>&${COPROC[1]}-

Then we move our standard output to the pipe that's connected to tee's standard input (our file descriptor 7) via:

exec 1>&7-

And finally, we close the pipe connected to tee's output, since we don't have any need for it, with:

eval "exec ${COPROC[0]}>&-"

The eval here is required here because otherwise bash thinks the value of ${COPROC[0]} is a command name. On the other hand, it's not required in the statement above (exec 7>&${COPROC[1]}-), because in that one bash can recognize that "7" is the start of a file descriptor action and not a command.

Also note the commented command:

#ls -la /proc/$$/fd

This is useful for seeing the files that are open by the current process.

We now have achieved the desired effect: our standard output is going into tee. Tee is "logging" it to our log file and writing it to the pipe or file that our output was originally going to.

As of yet I haven't come up with any other uses for co-processes, at least ones that aren't contrived. See the bash man page for more about co-processes.

______________________

Mitch Frazier is an Associate Editor for Linux Journal.

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Syntax Error

Anshul Narula's picture

It is a valid syntax and it works .Its called process substitution.Here you redirect the standard output to the file descriptor created by bash from which tee reads from, The standard output of tee is not being redirected so in the end tee writes to the standard output and the log file.

Syntax Error

Mitch Frazier's picture

I'm familiar with process substitution, what I missed in your example was the space between the two greater than signs when I tested it.

echo hello

set +o posix

if test -t 1; then
    # Stdout is a terminal.
    exec >log
else
    # Stdout is not a terminal.
    exec > >(tee log)
    #     ^ WHITESPACE REQUIRED HERE
    #ls -la /proc/$$/fd
fi
echo goodbye
echo error >&2

One important note is that process substitution is not enabled if bash is running in posix mode (in which case you will get a syntax error). Add a set +o posix at the start of your script to turn off posix mode.

That's a great solution. Thanks Anshul.

Mitch Frazier is an Associate Editor for Linux Journal.

Excellent example of using

Anonymous's picture

Excellent example of using coproc!
The following can be shortened from:

exec 7>&-
exec 7>&${COPROC[1]}-
exec 1>&7-

to:
exec 1>&${COPROC[1]}

Shorter Code

Mitch Frazier's picture

Your shorter code does work but it leaves some files (pipes actually) open that don't need to be open. Running my solution with an "ls -la /proc/$$/fd" after the above code shows:

dr-x------ 2 ...  0 2010-08-06 09:22 .
dr-xr-xr-x 7 ...  0 2010-08-06 09:22 ..
lrwx------ 1 ... 64 2010-08-06 09:22 0 -> /dev/pts/1
l-wx------ 1 ... 64 2010-08-06 09:22 1 -> pipe:[442781]
lrwx------ 1 ... 64 2010-08-06 09:22 2 -> /dev/pts/1
lr-x------ 1 ... 64 2010-08-06 09:22 255 -> .../coproctest.sh
lr-x------ 1 ... 64 2010-08-06 09:22 63 -> pipe:[442780]

Using your code it shows:

dr-x------ 2 ...  0 2010-08-06 09:22 .
dr-xr-xr-x 7 ...  0 2010-08-06 09:22 ..
lrwx------ 1 ... 64 2010-08-06 09:22 0 -> /dev/pts/1
l-wx------ 1 ... 64 2010-08-06 09:22 1 -> pipe:[442803]
lrwx------ 1 ... 64 2010-08-06 09:22 2 -> /dev/pts/1
lr-x------ 1 ... 64 2010-08-06 09:22 255 -> .../coproctest.sh
l-wx------ 1 ... 64 2010-08-06 09:22 60 -> pipe:[442803]
lr-x------ 1 ... 64 2010-08-06 09:22 63 -> pipe:[442802]
l-wx------ 1 ... 64 2010-08-06 09:22 7 -> pipe:[442799]

The following code (thanks to your push) closes things and is slightly shorter:

exec 7>&-
exec 1>&${COPROC[1]}-

Mitch Frazier is an Associate Editor for Linux Journal.

Thanks, Mitch, I realized my

Anonymous's picture

Thanks, Mitch, I realized my mistake, but it was too late, the button was already pushed.
Thanks for the excellent high tech tips! GB

LJ Account

Mitch Frazier's picture

If you create a linuxjournal.com account and login to leave comments you should be able to edit them afterwards and avoid the "oh crap" moments.

Mitch Frazier is an Associate Editor for Linux Journal.

I think you can do this

Anshul Narula's picture

I think you can do this instead:
exec > >(tee log)

Syntax Error

Mitch Frazier's picture

No that doesn't work. It's not even valid syntax.

Mitch Frazier is an Associate Editor for Linux Journal.

What does this step do?

Sree Pratheep's picture

In the following step what is that -(minus) for? Is it a typo or does it have any special meaning?

we move our standard output to the pipe that's connected to tee's standard input (our file descriptor 7) via:
exec 1>&7-

The -(minus) closes file

Anonymous's picture

The -(minus) closes file descriptors after duplicating. From man:

Moving File Descriptors
The redirection operator

[n]<&digit-

moves the file descriptor digit to file descriptor n, or the standard
input (file descriptor 0) if n is not specified. digit is closed after
being duplicated to n.

Similarly, the redirection operator

[n]>&digit-

moves the file descriptor digit to file descriptor n, or the standard
output (file descriptor 1) if n is not specified.

To be honest, to me the

Ronald Brindl's picture

To be honest, to me the traditional first example looks very much simpler and more understandable.
I find it quite difficult to really understand what the advantage of coproc really is, but after all, i am not doing this kind of stuff a lot.

White Paper
Linux Management with Red Hat Satellite: Measuring Business Impact and ROI

Linux has become a key foundation for supporting today's rapidly growing IT environments. Linux is being used to deploy business applications and databases, trading on its reputation as a low-cost operating environment. For many IT organizations, Linux is a mainstay for deploying Web servers and has evolved from handling basic file, print, and utility workloads to running mission-critical applications and databases, physically, virtually, and in the cloud. As Linux grows in importance in terms of value to the business, managing Linux environments to high standards of service quality — availability, security, and performance — becomes an essential requirement for business success.

Learn More

Sponsored by Red Hat

White Paper
Private PaaS for the Agile Enterprise

If you already use virtualized infrastructure, you are well on your way to leveraging the power of the cloud. Virtualization offers the promise of limitless resources, but how do you manage that scalability when your DevOps team doesn’t scale? In today’s hypercompetitive markets, fast results can make a difference between leading the pack vs. obsolescence. Organizations need more benefits from cloud computing than just raw resources. They need agility, flexibility, convenience, ROI, and control.

Stackato private Platform-as-a-Service technology from ActiveState extends your private cloud infrastructure by creating a private PaaS to provide on-demand availability, flexibility, control, and ultimately, faster time-to-market for your enterprise.

Learn More

Sponsored by ActiveState