There are two devices for each serial port, a tty device (e.g., ttyS0) and a callout device (e.g., cua0). Historically, the tty device was used for incoming connections, and the callout version was used for outgoing connections. Hence, any attempt to open the callout device will fail if the associated tty device is already open. While this behavior still persists, many people prefer to use the tty device for all connections. When concerned that two programs may interfere with each other by accessing the device at the same time, lock files are used. So, to use /dev/ttyS0, you would create a file /var/lock/LOCK..ttyS0 containing the process ID of the program that is using the device; unless, of course, the file already exists, in which case you wait until the program using the device removes the lock file. Many communications programs are already designed to use lock files.
There is some controversy about how this works, so I looked into it some more, and here's what I found in the Linux PPP HOWTO:
9.2 Serial Port Names: Historically, Linux used cuaX devices for dial out and ttySx devices for dial in. The kernel code that required this was changed in kernel version 2.0.x and you should now use ttySx for both dial in and dial out. I understand that the cuaX device names may well disappear in future kernel versions.
However, I did some looking in the kernel source (both 2.0.29 and 2.1.84), and it turns out that the HOWTO documentation is a bit misleading. If you have the tty version of the device open, any attempt to open the callout version will result in an EBUSY error. I verified this, and it does, indeed work that way. So it would seem that it would work perfectly fine to use the tty device for incoming calls and the callout device for outgoing calls, provided you also use lock files or only make one attempt to place an outgoing call at a time.
I tried to set up my box to both answer and place phone calls using the two separate devices, and I found that getty had to have the tty device open to listen for an incoming call, so attempts to place outgoing calls on the callout device failed (EBUSY). It seems that this is an interesting problem, but beyond the scope of the article.
So for most people, it doesn't matter which device they use, as long as they are consistent and use lock files if there is a chance of multiple attempts to use the device at the same time. However, it is probably useful to understand the subtle difference between the callout and the tty devices.
The lock file protocol is documented in the FHS at: http://www.pathname.com/fhs/2.0/fhs-5.5.html
Also, I found another discussion of the differences between the two devices in a list archive: http://www.uwsg.indiana.edu/hypermail/linux/kernel/9607.0/0363.html. This discussion suggests that callout devices are just there for historical compatibility, and everything should now use tty devices with lock files.