Cooperating Geeks
A few years ago, on a BBS I used to frequent, in a tech forum long since forgotten, one of the users asked a bunch of the regulars where they got their experience. "How does one learn routing, mail server administration or just heavy duty Perl?" That's a good question.
Most of us had learned it working at universities, where we learned admin skills on the job. However, this user's situation was slightly different--he was unable to go back to school and learn these skills. And without them there was little chance he could get a job at an ISP.
An idea popped into my head that I quickly suggested: a geek cooperative. Geeks forming their own ISP and learning those skills on the job.
The idea gained acceptance, but no one took on the idea. About a year later, I decided to pursue the idea with some local geeks I was affiliated with, and we've learned a lot more than simple system administration. Through various ups and downs, we've achieved more than I could have imagined, and impressed quite a number of people. It's an idea that I think could well serve a lot of geeks.
Any cooperative's foundations are quite simple: a group of people who have a common need and find that they can achieve their goals through collective action. Each member contributes time and sometimes money, and in return they become part of a business or an enterprise. Services or goods are sold, usually at a higher rate to non-members than to members, to help bring in revenue.
This is essentially the premise of our geek cooperative, named The Geek Empire, or TGE (http://www.thegeekempire.net/). We formed with approximately ten people who donated equipment and some basic technical knowledge. Everyone brought strengths to the group, including DNS, SMTP, networking and web server skills, but these definitely needed to be polished before we could get a customer. Using a friend's basement as our NOC, broadband from a local provider (we were sure to get a contract that allowed us to resell access), we were in business.
To begin with, we amassed spare equipment, including workstations from HP, Sun and old PC equipment, along with some hubs, NICs and lots of cat 5 cabling, and mounted the core systems in an old rack. After a few evening's worth of software setup, we were ready to get off the ground.
Our webmaster set up a site, our hostmaster hooked us up with OpenSRS to do domain name registration (http://www.opensrs.org/, a very low cost, full service TLD registrar any decent organization can affiliate themselves with), and we started working.
At first, we had to fund our rather large bill for broadband access. Each core member donated $50 a month, which helped cover network access, various setup fees and some hardware we had to purchase. This was a bit of a turnoff for some people, but it helped us get off the ground. And this helped us do it all without one dollar of venture capital.
After about a year and a half of operation, TGE provides the following commercial services at a level that meets the requirements for most small businesses and individuals. We provide mail hosting, domain name registration and hosting (with off-site secondaries), web site hosting and management and even some colocation services. For the typical small business or private individual, the levels we provide are more than adequate, and the prices are significantly more competitive than even local ISP companies.
Since our founding, we've been able to rid ourselves of the "membership dues", which has helped to attract new members. Our revenue stream is enough to keep us in the black, as well as ready to purchase new, higher capacity equipment should things fail.
What's even more satisfying is we do almost all of the work on free UNIX variants. Between OpenBSD and, of course, Linux, we've managed to bring all this together using free software. We use Sendmail mail servers, BIND name servers (nicely secured, of course), the Apache web server, all of which you'll find in use at the best commercial network providers. It's been "almost all" free software because we inherited some HP 9000 workstations, and the HPPA UN*X ports are nowhere near production ready. As such, they still run HPUX.
Immediately several non-technical issues hit us. Trouble ticketing was one challenge, such as when a server goes down and someone is out of a service. We have yet to really get a system in place, but currently a tight core of members will respond to a problem report and tag it for who's responsible for fixing it. Followup can be a bit tricky, though, so we should probably integrate a real trouble ticket system.
Accounting and financial issues also became a potentially large scale problem. Keeping track of incoming money, and the sources, along with where it is being spent, required that someone learn basic accounting skills. One member, Jodie, took this on and has kept everything working smoothing. We accept PayPal payments or traditional checks; credit cards required too much overhead for our interests.
Management skills have also been developed, rather quickly, by the cooperative members. We're geeks and we were more focused, initially, on our technical areas. We found that personnel management by our "Emperor" and others had to be developed to chart directions, tasks and other associated responsibilities. Through time, these skills have been developed nicely.
Closely related, customer relations skills were learned most definitely in a trial-by-fire setting. When our network provider folded (more below), we scrambled to get the word out about what was happening, and what we were doing about it. Most customers were very understanding, and their requirements were met by our temporary measures. We learned to route everything through one member, who serves, in addition to a vital technical role, as our customer relations contact. We have found that this leads to less confusion about who is taking care of the problem and what is being done about it for the customer.
Time commitments have been difficult to hold, as well. Most of us were students when this began, and our schoolwork takes precedence. As such, when a busy semester, work or disinterest appear, we have to hand off our duties to someone else. These transitions can sometimes be a bit sticky, as, say, hostmaster duties don't get clearly delegated.
Lastly, going into business with your friends is certainly a dangerous area to tread. Between the issues of money and mistakes, feathers can get ruffled and feelings hurt. As with any business venture, keeping personal and professional motivations separate is good sense.
Our biggest challenge, the one that caused us the most pain, has been the failure of our initial ISP. We were using SDSL when the network provider filed for Chapter 11 status and, within a few hours, closed down the network. We quickly scrambled to move up an order we had put in before, but that still took two months to fill.
We wound up moving from an SDSL line of over 400kbps to a 56k dialup account during this crunch time. What was worse was moving from a /27 network to 1 IP address. Even changing these name server records with Network Solutions required we hijack our own domain through some crafty faxes one Friday night.
We wound up solving this crisis through skillful virtual hosting for mail and web services, and making heavy use of our secondary, off-site name servers. You never know what Sendmail rewrite rules can do until you hand craft a few! We even learned the intricacies of Apache virtual domains and the joys of port-level redirection.
This situation existed for about two months, and we're grateful that our customers were as patient as they were during this process. Damage control was perhaps the best lesson learned out of this.
Having stated some of the many problems we've encountered, and some of the great non-technical challenges we met, when we look at the goals of the coop, we find success overall, beyond what we anticipated.
First up, the main purpose of the coop was to have fun while geeking out. Between good friendships, the various parties we've thrown for all sorts of reasons, this goal has been achieved. Even with some of the less successful days and times, we do know how to throw good parties, with no less than two DJs on staff and an endless supply of friends who come by.
One of the most important reasons I designed the Geek Coop idea was to provide education for its members in ISP- and IT-related skills. Over the course of the past year and a half, at least three people have developed real world hostmaster and domain registrant skills. We've learned real world networking skills, and a host of other very marketable skills.
On the business side, we've been able to attract dozens of customers, many friends and fellow students at our local university, as well as family members. We provide needed services for the local community with a growing customer base, which we consider a testament to the fact that geeks, working together, can bring a business off the ground quickly.
Our advertising is mainly through business cards, word of mouth and flyers at geek conventions, such as DefCon and Rubicon. Unfortunately, the return on investment in this arena has been pretty low, so we're more focused on word of mouth, followed up by a business card.
While this may seem intuitive from the above description, it's actually a little more subtle than, "take a half dozen geeks, ten servers, some Linux experience and some broadband, stir and let simmer for six to eight weeks."
First, it helps to be associated with or near a university. The discard hardware alone makes it possible to start this without some form of venture capital. We've obtained several Sun and HP workstations, plenty of PC systems and accessories that form the core of our network. They're in great shape, just a bit underpowered for their former campus duties. They're more than adequate for serving mail or DNS, though.
Secondly, the right combination of geeks is needed, most of whom will have to have some strong technical skills; otherwise, getting started will be really difficult. They don't need to know how to do advanced work for servers, but a solid foundation is required. Newer members can learn from the basics, but your initial team should be skilled in software and server administration, as well as hardware. One evening, when a mail server failed, two geeks put together a PC running UN*X from spare parts in approximately ten minutes.
Thirdly, business skills will definitely be needed, every bit as much as the technical skills. In fact, some of the less technical savvy geeks can handily develop a strong foundation in managerial areas keeping things running. A clear organizational structure from the outset, together with sound delegation of responsibilities, will be key to the success of the coop.
Fourthly, decide what will be your core services and make sure that you are strong in them. Be sure to plan accordingly, as well. For example, if you plan to do web site hosting, make sure you either get enough bandwidth or charge a reasonable amount for the bandwidth users may need. It may be wise to slowly introduce services, rather than risk making big mistakes all at once.
Lastly, time will be your friend. All of this takes time, in the evenings after work, on a Friday night when your upstream provider folds and you have to get things working, and keep everything running smoothly. You will develop an appreciation for the complexities of running a large network. Treat this as any other business venture and you should be okay.
Going into business sounds like a lot of fun, but it isn't always, nor are you limited to enjoyment as your only success. By forming a cooperative ISP with fellow geeks, we've found a successful business model, a great technical education, and one of the best learning centers for real world business skills. While TGE isn't the only geek cooperative around, we hope to see more communities adopt this model.
The following resources have been invaluable in getting things set up and running smoothly.
DNS and BIND by Paul Albitz and Cricket Liu
Sendmail by Bryan Costales with Eric Allman
TCP/IP Network Administration by Craig Hunt
