Bash Redirections Using Exec
If you've used the command line much at all you know about I/O redirection for redirecting input to and/or output from a program. What you don't see all that often or that you may not be familiar with is redirecting I/O inside a bash script. And I'm not talking about redirections that you use when your script executes another command, I'm talking about redirecting your script's I/O once it has already started executing.
As an example, let's say that you want to add a --log option to your script and if the user specifies it you want all the output to go to a log file rather than to the stdout. Of course, the user could simply redirect the output on the command line, but let's assume there's a reason why that option doesn't appeal to you. So, to provide this feature in your script you can do:
#!/bin/bash echo hello # Parse command line options. # Execute the following if --log is seen. if test -t 1; then # Stdout is a terminal. exec >log else # Stdout is not a terminal, no logging. false fi echo goodbye echo error >&2
The if statement uses test to see if file descriptor number one is connected to a terminal (1 being the stdout). If it is then the exec command re-opens it for writing on the file named log. The exec command without a command but with redirections executes in the context of the current shell, it's the means by which you can open and close files and duplicate file descriptors. If file descriptor number one is not on a terminal then we don't change anything.
If you run this command you'll see the first echo and the last echo are output to the terminal. The first one happens before the redirection and the second one is specifically redirected to stderr (2 being stderr). So, how do you get stderr into the log file also? Just one simple change is required to the exec statement:
#!/bin/bash echo hello if test -t 1; then # Stdout is a terminal. exec >log 2>&1 else # Stdout is not a terminal, no logging. false fi echo goodbye echo error >&2
Here the exec statement re-opens stdout on the log file and then re-opens stderr on the same thing that stdout is opened on (this is how you duplicate file descriptors, aka dup them). Note that order is important here: if you change the order and re-open stderr first (i.e. exec 2>&1 >log), then it will still be on the terminal since you're opening it on the same thing stdout is on and at this point it's still the terminal.
Perhaps mainly as an exercise, let's try to do the same thing even if the output is not to the terminal. We can't do what we did above since re-opening stdout on the log file, when it's currently connected to a file redirection or to a pipeline, would break the redirection/pipeline that the user specified when the command was invoked.
Given the following command as an example:
bash test.sh | grep good
What we want to do is manipulate things so that it appears that the following command was executed instead:
bash test.sh | tee log | grep good
Your first thought might be that you could change the exec statement to something like this:
exec | tee log & # Won't work
and tell exec to re-open stdout on a background pipeline into tee, but that won't work (although bash doesn't complain about it). This just pipes exec's output to tee, and since exec doesn't produce any output in this instance, tee simply creates an empty file and exits.
You might also think that you can try some dup-ing of file descriptors and start tee in the background with it taking input from and writing output to different file descriptors. And you can do that, but the problem is that there's no way to create a new process that has its standard input connected to a pipe so that we can insert it into the pipeline (although see the postscript at the end of this article). If we could do this, the standard output of the tee command would be easy since by default that goes to the same place the main script's output goes to, so we could just close the main script's output and connected it to our pipe (if we just had a way to create it).
So are we at a dead end? Ahhhh no, that would be a different operating system you're thinking of. The solution is actually described in the last sentence of the previous paragraph. We just need a way to create a pipe, right? Well let's use named pipes.
#!/bin/bash echo hello if test -t 1; then # Stdout is a terminal. exec >log else # Stdout is not a terminal. npipe=/tmp/$$.tmp trap "rm -f $npipe" EXIT mknod $npipe p tee <$npipe log & exec 1>&- exec 1>$npipe fi echo goodbye
Here, if the script's stdout is not connected to the terminal, we create a named pipe (a pipe that exists in the file-system) using mknod and setup a trap to delete it on exit. Then we start tee in the background reading from the named pipe and writing to the log file. Remember that tee is also writing anything that it reads on its stdin to its stdout. Also remember that tee's stdout is also the same as the script's stdout (our main script, the one that invokes tee) so the output from tee's stdout is going to go wherever our stdout is currently going (i.e. to the user's redirection or pipeline that was specified on the command line). So at this point we have tee's output going where it needs to go: into the redirection/pipeline specified by the user.
Now all we need is to get tee reading the right data. And since tee is reading from a named pipe all we need to do is redirect our stdout to the named pipe. We close our current stdout (with exec 1>&-) and re-open it on the named pipe (with ezec 1>$npipe). Note that since tee is also writing to the redirection/pipeline that was specified on the command line, our closing the connection does not break anything.
Now if you run the command and pipe it's output to grep as above you'll see the output in the termimal and it will also be saved in the log file.
Many such journeys are possible, let the man page be your guide!
p.s. There's another way to do this using Bash 4's coproc statement, but that'll wait for another time.
Mitch Frazier is an Associate Editor for Linux Journal.
Practical Task Scheduling Deployment
July 20, 2016 12:00 pm CDT
One of the best things about the UNIX environment (aside from being stable and efficient) is the vast array of software tools available to help you do your job. Traditionally, a UNIX tool does only one thing, but does that one thing very well. For example, grep is very easy to use and can search vast amounts of data quickly. The find tool can find a particular file or files based on all kinds of criteria. It's pretty easy to string these tools together to build even more powerful tools, such as a tool that finds all of the .log files in the /home directory and searches each one for a particular entry. This erector-set mentality allows UNIX system administrators to seem to always have the right tool for the job.
Cron traditionally has been considered another such a tool for job scheduling, but is it enough? This webinar considers that very question. The first part builds on a previous Geek Guide, Beyond Cron, and briefly describes how to know when it might be time to consider upgrading your job scheduling infrastructure. The second part presents an actual planning and implementation framework.
Join Linux Journal's Mike Diehl and Pat Cameron of Help Systems.
Free to Linux Journal readers.Register Now!
- Stunnel Security for Oracle
- SourceClear Open
- SUSE LLC's SUSE Manager
- Murat Yener and Onur Dundar's Expert Android Studio (Wrox)
- My +1 Sword of Productivity
- Managing Linux Using Puppet
- Tech Tip: Really Simple HTTP Server with Python
- Non-Linux FOSS: Caffeine!
- Google's SwiftShader Released
- Doing for User Space What We Did for Kernel Space
With all the industry talk about the benefits of Linux on Power and all the performance advantages offered by its open architecture, you may be considering a move in that direction. If you are thinking about analytics, big data and cloud computing, you would be right to evaluate Power. The idea of using commodity x86 hardware and replacing it every three years is an outdated cost model. It doesn’t consider the total cost of ownership, and it doesn’t consider the advantage of real processing power, high-availability and multithreading like a demon.
This ebook takes a look at some of the practical applications of the Linux on Power platform and ways you might bring all the performance power of this open architecture to bear for your organization. There are no smoke and mirrors here—just hard, cold, empirical evidence provided by independent sources. I also consider some innovative ways Linux on Power will be used in the future.Get the Guide