Set up KDE to ask for your SSH passphrase on login.
I use pam_ssh. It sets up PAM so that at the primary login, you type your SSH passphrase instead of your normal password. The fact that the passphrase successfully decrypts your private key proves your identity just as surely as a unix password (probably more surely). pam_ssh then starts up an SSH agent for you with your key already loaded. Best of all, because it's integrated with PAM, this works not just in KDE, but from any login prompt, including the console. It's completely integrated and out of the way - you just think of your SSH passphrase as your password.
I'm not sure all of the required components are included by default in all distros. For example, I use Kubuntu Intrepid Ibex, and I found that ssh-askpass (which is the X GUI that passes the keyphrase back to ssh-add) wasn't installed.
A quick "sudo aptitude install ssh-askpass" fixed me right up. YMMV.