When the Administrator walks...

We never like to see our co-workers leave. In most cases, though, we are are happy for them because they are going on to bigger and better things. But occasionally they are not leaving under their own power. And that is when things can get...well...messy. So before you are tasked with the job of putting it all back together, why not take a moment and prepare.

I have discovered that there are two types of administrators – those who are shepherds of the systems, guarding them, nurturing them, and returning them to their employers in better shape than they found them and those who are owners of the systems, jealously guarding them, babying them and only reluctantly returning them to their employers, usually when they are shown the door. It is this last group that will cause you the most pain and suffering and it is this last group who you have to prepare for, prevent against and stand watch over.

While there are a number of ways to get into a system after the fact (and we will briefly touch on some of them), if you can avoid having to use those tools, so much the better – for you, for the systems, and for your employers. The first step is make sure there are policies and procedures in place. Which means getting the Human Resources department involved at the beginning. Remember that HR is there primarily to protect the company, so anything that helps them in this mission is generally regarded as a good thing, especially if it includes the systems they have come to depend on.

When new people start, what sort of background checks are you running on your SysAdmin people? Is it a simple credit check? Should it be more, such as an NCIC (or your local equivalent)? What about contractors? Personally, I have always thought we should be bonded, given the incredible access to corporate information many of us have. When it is time to leave, what are the policies for departure? Is there a clause that will suspend or withhold the final paycheque if all of the pertinent data is not turned over – especially root passwords – upon termination. Are there penalty clauses in contracts for outsourced systems, or contractors?

How about indemnification from unlawful access for password recovery? If you are thinking your company is too small or you are just a cog in the machine and do not have to concern yourself with these issues, think again, because it could be you the General Counsel comes to. Remember, in many parts of the world, cracking a password or a system is illegal, even if you are being contracted to do it. There is enough case law against you. So make sure the protections exist before hand.

There are a myriad of other things you can include to ensure a smooth transition. One of the biggest of course is making sure that a team of trusted people know the root password(s) and it takes that team to change it. This requires more hardware and scripting skills, but is a much better way to maintain password security. Of course, once you set the root password you should forget it and have proper least access procedures, but we all know that in a shop of one or two, or two hundred, what is best practice is not always followed.

Which brings us to the systems. If you have more than one server, do yourself a favor and use a directory service. Whether it is NIS and its derivatives or LDAP, it will save you pain and suffering in the long run if you set it up now and fully document its installation, implementation, configuration and use. And then use it!

But, despite all of your best efforts and preparations, the day will come when the boss and a lawyer (or two) will be standing at your desk, with a writ, or a warrant, or a letter of authorization (or all of the above) and asking you to help clean up. And unless you really like digging through trash, this will become the worst day of your life, even if the task only takes a couple of minutes. I can categorically say, that other than browsing cache files, cracking systems is the least enjoyable part of my job. But sometimes you have to do it. And this is where you need the tools.

So what sort of tools do you need? Well, besides a screwdriver, with an assortment of screw bits (I keep a flat head, Phillips and a torx bit in common sizes), you might also want a knife and a a pair of pliers. You may need cable crimping tools too, depending on your department and your gear. You will also need some software. The hardware is for extraction. Depending on those policies you set up, it may be necessary to image off the primary disk prior to doing any cracking. And depending on the size and make up of your hardware, that might mean removing the disk from the case. Now we have come a long way in ten years, to the point were almost any system can be hooked up to an external terabyte, or larger standalone, USB disk, thus reducing the need to remove the core disk from the system, but there are some systems out there that are still in service that you cannot just hook up a disk. Know your policies in this area and do not let yourself be rushed. And document, document, document every step. This will help you when you get hauled into court to testify. (I would suggest that if you do not own a suit, if you find yourself being asked to do this sort of thing, that you make sure you can buy a suit...).

And then, there is the software. I am not going to go through the steps for cracking a system. Sorry, there are lots of sources out there and there are lots of ways to break the egg. Each situation is different but here are some general recommendations. First, make sure you have a LiveCD of your favourite distribution. Fedora, Ubuntu, even Knoppix have a number of tools already baked in that you will find you will need. If possible, write your LiveCD out to a USB stick and add additional packages so they are there when you need them and you will not have to rely on an external connection to the Internet. Depending on the system you are being asked to access, it is best to assume you will not have access to the outside world. So bring it with you. Second, it is good to have a handful of Window's tools, such as chntpw and others. There are a number of them available that all run on Linux for tasks such as mounting disks, accessing and changing the passwords, and accessing and changing the registry. And you should know how to use them. Finally, I cannot stress enough that you document, document, document and do not let yourself be rushed.

It is sad when our friends and co-workers leave us. But it does not have to be a catastrophe. Because at the end of the day, we still have to manage the systems, and if we cannot do that, we could well be the next to leave.

Follow up [20100615]: For those who have an interest in this, here is a perfect case in point that I had forgotten about. Terry Childs held the city of San Francisco essentially hostage because he controlled the passwords to the routers and switches. He claims he was doing it as a service to the City. He is currently looking at a 5 year jail sentence, although he could be released shortly if time served is considered.


David Lane, KG4GIY is a member of Linux Journal's Editorial Advisory Panel and the Control Op for Linux Journal's Virtual Ham Shack


Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Big mess

macaroni's picture

It's a very big mess when the admin leaves followed by a change in some of the products you're using which only he would have accommodated. That is what we ran into when our credit card processing switched their encryption level, leaving us crippled because no one got the three messages which were sent before the change took effect. Yep... sometimes looking important people turns out a lot worse that one could imagine.


A couple things not optimal with this article

Sum Yung Gai's picture

There are a couple of things that are, shall we say, "not optimal" with this article. First is the title, "When the Administrator Walks," but you end up actually talking about "When the Administrator Is Fired."

The second one is the glossing over, and I'd say even outright ignoring of, the fact that administrators who "leave in a huff" and do so "under their own power" may well have good reason to do so. Companies who mistreat their people are only going to keep them until whatever recession in place at that time is gone. The dot-com boom taught us that in spades.

Companies that mistreat their people may well turn the "shepherd" type that you describe into the "owner" type that you also mention. Unfortunately, there's a lot of mistreatment of people that happens, especially during a recession. That's not how you engender loyalty, and I think you would do well to remember that.


Bad advice

Anonymous's picture

Passwords and systems should never be cracked, ever. It doesn't matter if your employer's business depends on it. If they allowed a situation to develop where one guy had all the passwords, and he left, then too bad! You just need to wipe everything out and start over, from backups if you're lucky. Suffer the consequences.

Any company stupid enough to allow one guy to ruin it doesn't deserve to be in business.

Say what?

David Lane's picture

There are two things wrong with your statement. First, when you are served with a warrant, you have little choice but to turn over the data. All the data. There may be a few extenuating circumstances, but that is for the lawyers to debate. If an executive of my company is standing on my desk telling me, with a member of the legal team and a member of law enforcement standing behind them, to open the system, I am opening the system (with the get out of jail free card and documenting every step. I have been there.).

Secondly, you are assuming that there are backups to restore from.

In every company and federal agency I have worked for, the hardware is the property of the employer. And there are a number of examples of companies that have let their systems be compromised by bad administrators. Suffering the consequences may not be financially or legally a smart business decision.

David Lane, KG4GIY is a member of Linux Journal's Editorial Advisory Panel and the Control Op for Linux Journal's Virtual Ham Shack

You yourself pointed out in

Anonymous's picture

You yourself pointed out in your article:

"Remember, in many parts of the world, cracking a password or a system is illegal, even if you are being contracted to do it. There is enough case law against you."

So cracking is not an option, ever. If it's illegal, don't do it. If the company is too stupid to manage their systems right, then they need to suffer the consequences.

What are you talking about.

venomfang's picture

If the company can prove that they own the systems in question and have proof about this, and background checks have been done. Then locally cracking the system password for that said employer should not be a problem.

Things to ask for before doing this though should be: a proof of purchase ( an invoice with serial number included, as well with payment type.), network documentation listing information about the system (hostname, static-ip / reserved-ip, mac address of NICs, operating system, ect...). If the supposed owner can't come up with any proof of purchase then don't reset/crack the password. Give them the option of replacing the hard drives and reinstall the system, but do not crack the password.

If law enforcement is involved, ask for a copy of the writ or warrant. Yes you do have to give the data to law enforcement, if you can't crack the password, depending on the circumstance, they will take the system back and have there forensics team get it cracked, but these are extreme cases. You can not go to jail for following a court order, you may go to jail if you refuse to do so, and are charged with failure to follow a court order or hindering a police investigation.

Huh? We're not talking about

Anonymous's picture

Huh? We're not talking about providing the passwords to the executives and LEOs, we're talking about cracking the system (i.e., you don't have the passwords, and you're trying to break in). That's illegal. Even if the cops are telling you to do something illegal, you don't have to do it.

If there are no backups to restore from, then too bad. Should have had backups. That's the price you pay for stupidity.

If you allow your system to be compromised by a bad administrator, that's your own fault. It's no excuse for illegal behavior (cracking the system).

I agree with most of this.... but there's more too it.

venomfang's picture

Don't get me wrong these are all good points to bring up, and yes "sigh", unfortunately in our industry there are a lot of bad sysAdmins out there.

There is more to it than just the server or workstation of the sysAdmin. IT departments deal with all divisions and people in a company, more than any other division, except for maybe janitorial and custodian.

These are some things that should be looked at immediately:

-Security systems: remove the sysAdmin from the premises security system.

-VOIP: Check the phone system, you be surprised how often people overlook this.

-VPNs: rebuild all VPN-tunnels with new keys.

-RAS(Remote Access Server) + Terminal Services: Change all remote access user passwords immediately.

-Directory Server: Check it for stale accounts, and set all users to change password at next logon.

-Wireless: Redo all pre-shared keys, if any. If a radius system redo the shared key for radius authentication (between your server and AP's).

-Firewall: Check for ports that should not be open.

After you are sure that the company's infrastructure is secure, then start looking into the issues with the sysAdmin's workstation. Once you are sure that the infrastructure is secured from the outside world, that's one less potential threat you have to worry about.

Also I find that most of these so called sysAdmin's have very poor security protocol's implemented system wide; especially at the user level, week passwords, people not logging off there machines at the end of the day, ect...

uh no

Anonymous's picture

the intent of the article wasn't to make a checklist

the author was going outside of the typical checklist, and giving the bigger picture into the mindsets of systems admins... and the implications.

your lists don't work for my organization.

each organization will have long complex lists, all you have to do is get the to ten IT workers together and brainstorm what will have to be changed in the event of a large shakeup.

the lists will be wildly varied from organization to organization...

Yes and no.

David Lane's picture

Yes, you are correct in that my over arching thrust was to raise awareness of the issues that most of us do not even consider until it is too late, especially when it comes to managing the situation after the fact (avoid managing it after the fact - manage it before hand, it will make your life easier).

But there should be a set of procedures. Some organizations live on checklists, and there is certainly nothing wrong with them for doing the tasks they are best suited for, but in the end, it is the procedures, which should be easy to follow and implement, vetted through the legal and HR processes and enforceable both at the corporate level and under the laws of the country you live in.

What venomfang was pointing out was that I overlooked other areas where a SysAdmin might have had access to (and I could have tossed in switches, routers, physical security databases for CAC cards etc...). I did not overlook them as much as ignore them outright in making my point.

Most of us spend our time with our heads down, keeping the systems alive. Very few of us have had to look at some of the larger issues that go into managing the systems, the people and the business processes that are wrapped around them.

I could have delved into the intricacies of the documentation or the procedures for breaking into a system or provided some sample HR policies, but by doing so, I run the risk of missing items that are important in your environment and distracting from the important issue that bad things happen and more than anything else, we need to be as prepared for them as we are for any other disaster scenario.

Perhaps we here at Linux Journal need to revisit the Penguins for Suits column that we once had, that looked at Linux from the business perspective to address some of those management issues that the average SysAdmin does not deal with, but that as your career progresses, you find yourself, more and more often dealing with, rather than the actual nuts and bolts of individual commands and wiring issues.

Let me know your thoughts.

David Lane, KG4GIY is a member of Linux Journal's Editorial Advisory Panel and the Control Op for Linux Journal's Virtual Ham Shack


David Lane's picture

I was leaving these things out on purpose, but certainly, there needs to be a good, consolidated check in/check out procedure that covers the gamut of services (and I will throw in phones, both analogue and portable to complement the VoIP).

The procedures should be a solid HR process that is then sent (via workflow is best) to the appropriate IT people to shut things down in an orderly fashion.

David Lane, KG4GIY is a member of Linux Journal's Editorial Advisory Panel and the Control Op for Linux Journal's Virtual Ham Shack

Good point

Bhaskar Chowdhury's picture

Hey David,

Thanks for reminding of documentation procedure...which I feel absolute must to avoid doing more mistakes in the rush.Proper documentation is "The" way to go in is business.

I do prefer "wiki" thing...nowadays almost all the good and better concerns have had their internal wiki pages and made available to different department with different rights.

Cheers mate!



JohnMc's picture

Though I can relate to the thrust of this article, having more than once had to gain control of a box after a disgruntled employee left in a huff. But there are preparations to be made before you hire --

* An iron clad agreement covering all the systems the employee is responsible for and a required list of all current id's and passwords upon termination of employment. And check the list before they leave!

* Always reup the employee data sheet that HR uses, twice a year. If you really have to issue a warrant to force release of id's and keys, it becomes rather embarrassing to be held up because you thought the person lived in LA but now lives in Pomona.

* Thy data space shall be separate from thy os space, always. Mounting the data space with a fresh os will be quicker in most cases than hacking into a unified system.

* Thy need for root shall be confined and the account data recorded.

Whether it is NIS and its

Josh's picture

Whether it is NIS and its derivatives or LDAP, it will save you pain and suffering in the long run if you set it up now and fully document its installation, implementation, configuration and use. And then use it!

I'm always a little bit frustrated by articles like this, because they never talk about the specifics as to how to document those processes. Should the procedures be in paper or electronic? If electronic, what format? What tool is used to build the policies and procedures? In my department, people prefer to have actual checkboxes to check off, how should that be done? Can any examples of those system docs be provided?

This might be a silly little point, but for formatting/documentation nazis like myself, it's a big one. :)

Documentation is nice

David Dreggors's picture

Documentation is nice when...

1. It can be accessed by any employees that need it at any time
2. It is accessible on or off the clock (as on-call staff)
3. Is easy to search for what you need in a hurry

For this I like a wiki. Not only can you reach it through your office vpn while off the clock (if you have a vpn) but you can setup some so that they are read only except for logged in admins or users with enough permissions. Even better only viewable by logged in users and only editable by admins.

This means that no-one can edit the rules/policies and only the people you choose can see them!

This goes for documenting procedures and best practices as well, since most wikis today allow for code blocks and syntax highlighting you can format your pages to have really nice content and bring peoples eyes right to important notes or command line sections to be preformed.

There is no easy way to do this on paper, you will spend WAY to long trying to get formatting, organizing, and/or referencing correct and slow your actual documenting to a crawl in my opinion.

I have my wiki open and copy/paste right into it as I work, then just a few wiki tags and viola! I have a nice page that already links to material I have already written and all formatting was done for me in most cases or with ease and on the fly in other cases.

Just my .02¢

Fair Enough

David Lane's picture

The problem is that each organization has their own way of doing things. One organization might want check lists, on paper, in a binder, updated quarterly, while other shops are comfortable with electronic wikis that are updated as needed. I have seen and used both, and others in between, for example an electronic document maintained under electronic version control like subversion and more formal change management control boards that had to sign off on the changes.

While providing these sorts of things could be done, it really is a local decision. The important thing is to have something.

David Lane, KG4GIY is a member of Linux Journal's Editorial Advisory Panel and the Control Op for Linux Journal's Virtual Ham Shack


W4BSD's picture

I love seeing a ham call in the sig.... :)


David Lane's picture

You should hop over to the forums, or drop by the IRC too. Lots of us lurking around. Even Tux has an EC's badge on!

David Lane, KG4GIY is a member of Linux Journal's Editorial Advisory Panel and the Control Op for Linux Journal's Virtual Ham Shack