the costs of living in a monoculture

I'm very interested in the costs associated with living in a computational monoculture. I recognize there are some benefits from being part of the monoculture, but for some people the cost side of the equation is becoming significant.

Most of the stories one hears are anecdotal like this one:

from my friend (name removed) (let go from Lucent a few years ago, he runs a small PC consulting firm out of his basement)

__

>Last week was Christmas in August for us. I was on vacation when the worms hit, but still at home. The phone started ringing off the
>hook from legal, dental and medical practices I have worked with or where someone came across my business card. These guys
>were frantic to get back in operation and had no idea what to do. They usually contract out their computer work, but only 10 to 40 hours
>a month and they don't have priority in emergencies.
>
>At first I really wanted to enjoy my vacation, but people were begging me for quotes. I told the first $500 for the morning and
>$200 an hour afterwards. It took $900 to get him up and running and I got a $100 tip. He acted like I had just saved his kid in surgery.
>
>The second was a call that came at 11pm that night. I told them I could come in at seven, but it would be expensive. "not good enough -
>I will give you $2000 if you can get my five PCs running before 5 am when I need to be online with London" .. I told him I had no idea
>what it would take, but would be willing to work for $400 an hour. When he heard that I could get to his place by midnight he said "good -
> $500 an hour.." I finished at 4:10 am and he pealed off twenty five $100 bills.
>
>I've never seen people so frantic. I did another job with a long time client and charged my normal rate - he gave me a $500 tip.
>
>I love Microsoft worms!

I have heard numerous reports of short term loss of business in companies of all sizes, the hosed schedules of computer staff (particularly at colleges with students arriving with unclean machines) and even people deciding it was best to just get a new Windows box.

If we continue to see problems surface one has to wonder which elements of the market will be impacted most. My suspicion is that small business and home users have the most to lose, but that is only speculation.

fascinating stuff!

Comments

[george]

This is interesting as I make a living supporting mostly Windows machines, but also some OS X and linux.

At the home level most people don't consider alternatives because they are afraid of not having support and they assume that anything else will be as bad as what they are using. In the small businesses we deal with there is also a fear of change, but also many of the programs they need are fairly custom and only support Windows (our largest client is locked into Windows 98 for this reason - moving to 2000 or XP has been impossible as they are dependent on a $50,000 custom program that is mostly a database).

My partner and I are completely burned out from the last two weeks and I'm taking a day off. We moved our normal dates with customers who didn't need work to work with people with emergencies. Our emergency rate is 200% during normal business hours and $300 on evenings and weekends.

We are making it up to our OS X and Linux users by giving them two free months of support paid for by the Windows users. Our hope is to migrate more of our users to something else -- probably OS X -- so we can branch out and offer more value by doing custom database work and other higher value jobs.

Our big customer is an interesting case. The Windows patch does not work with their customer program and Windows 98 ... They try to keep everything off of a network, but need to have some Internet access for some billing. We are trying to convince them to hire someone to rewrite their core software for linux or OS X. In the long run they would save money.

[steve]

The college situation is a current discussion on slashdot.

While /. doesn't have the highest signal to noise ratio, there are a few interesting comments:

the question

Recently with the outbreak of the MSBLASTER worm and the startup of the college semester here in the US we've been hit by a big problem here where I work. Many students are bringing in machines from home, often times infected. The infections are so bad that they bring the whole network to a crawl. Yes, you can install ACLs on edge routers and put a router between the dorms and the rest of your network, but it still brings the dorm to a crawl. You can make sure people install the patches, but what if someone re-installs windows, or brings in another machine, and what about NEXT year? From the Slashdot community, how have sysadmins out there delt with this? How can you manage each machine in a network such as a college, where people are bringing their own machines in from the outside? ACLs on routers... but what about for the segmented network?

comments on forced updates:

Then who supports them when the latest Windows update hoses thier machine? It happens less than it used to, but I have one client who lets auto updates run, and one patch in paticular (810577) has brought network browsing to a crawl. We have done literally hundreds of test and narrowed it down to this patch, but neith the knowledge base, user community nor a direct (and expensive call) to MS support can fix his issue. Now he has users screaming about slow network browses to files and folders, time outs hitting their home-brewed data base and his phone never stops ringing. Now mulitply that by the body of a college campus?

You'll need something more reliable than Windows if your plan is to mandate that sort of thing.

___

As the systems admin who will test those patches in a test lab before rolling them out to people, you will make sure that will not happen if you valuie that paycheck. Blindly checking off security updates for addition to the network is studipity no matter what the platform, wther you use up2date or MS AutoUpdate. For MS systems, having a SUS server helps centralize this process since you check off what you authorize to get pushed to the network. Active Directory policies can enforce this. Those that don't want to play in the domain can piss off. If you want to keep them off the network, there is always 802.1x.

___

There is a flaw in your thinking. If you don't trust Windows Update, then why do you trust MS enough to buy their product in the first place???

Intro to Computers 101 (n/c) - This is a manditory seminar for all students using Intel architecture computers in the campus network. This seminar will be held every 15 min in the largest lecture halls on campus and will provide all students with the knowledge needed to load a Knoppix distribution of Linux, and use OpenOffice.org to access, modify and distribute the documents that you created last year. Check the system listing at the door of each lecture hall to determin if your own system will be covered, and in which lecture hall it will be covered. A CD-Rom copy of Knoppix will be provided to all students. The only exemptions that will be allowed are for those students who bring their computer to the specified certification labs to demonstrate that the computer boots into Linux and has not option to boot into Windows.

Intro to Intel Linux - Lab - 102 (n/c) : This lab will allow you to bring your laptop or desktop computer to one of the specified computer labs. At this lab you will be assisted through the process of installing Linux on your computer, and given an opportunity to experiment with the applications that your professors have recomended using for the course you will be taking. This lab superseeds Intro to Computers 101.

Follow these up with automated software that periodically polls each computer to determine what OS is running, and admin down any port that is supporting a Windows PC. This software should also generate a memo to the students in the dorm room that that port supports requesting that they bring the offending computer to a certification lab to be validated and have the network port placed back into service.

Ok, It's not perfect, but would be an option that would protect the campus.

It's just an idea. I don't expect it to be implemented. One sticking point would be that the school would have to have an agreement with any computer vendor who sells computers on campus that they would certify that the computers they sell to students and faculty on campus support the desktop releases of Linux, (Knoppix, Morphix, Lindows, Mandrak 9.x+, SuSe 9.x+, RH 9.x+) and in all likelyhood the termination of any software contracts with Microsoft.

___

Case in point -- back in 2000, even though I had about four years Linux experience by then, I managed to bring down Internet access for an entire dorm (about 900 students) for a week.

It all started when I helped a friend install Linux on his new computer. Unfortunately, in addition to installing a DHCP client on his machine, I had accidentally flagged the DHCP server to install as well. What happened was that the DHCP server software on his new Linux box was challenging the Windows DHCP server that the dorm was using, and his machine won -- even though his DHCP server wasn't properly configured to hand out IP addresses to other clients. So, all of these other 900 students would turn on their computers, which would send out a DHCP request, and they would get a response from his computer instead of the real DHCP server, thus causing their computers to give up trying to connect to the network. Ironically enough, his computer connected to the internet fine, as it was the only one connecting to the real DHCP server (I guess that explains his super-fast connection during that week).

Anyway, we had no idea that any of this was happening until we headed back to his dorm room one day, and found three network services guys looking in bewilderment at the computer (they had never used anything but Windows, so they had no idea how to fix it). They claimed that it took them a week to isolate the problem to his machine. They explained what was happening, and it then hit me that the DHCP server was also running on his machine, so I logged in, apt-get removed it, and the problem was immediately fixed. Not in their eyes though, as they made us talk to the head guy at network services... He gave us fair warning that if we did that again, our access to the network would be revoked (and rightly so!).

The obvious moral of the story is, whereas most OSes give you just enough rope to tie a knot, Linux gives you enough rope to hang about 900 people. :^)

How about this one: What can a STUDENT at one of these schools do to help? I've tried teaching as many people as possible about computer safety (take a health classes' STD safety course, apply to computers basically), and I'm ineligable to become a dorm tech right now... anyone?

Write your own exploit of the vulnerabilities that patches them, and force feed it to any computer spamming you with virus-born packets ;)

___
Considering Blaster only affects 2000/XP/2003 machines, that means that the roughly 50 computers running those took 8 hours to clean? Something seems wrong here.

unfortunately not -- updating random systems is harder that it seems. When we got hit at our university i helped out cleaning a bunch of systems and I couldn't believe how long it took -- Win2k installs had to have Service Pack 4 installed before you could apply the security patch for the worm, other dependancies changed because of that, had to install and update the university verson of norton antivirus, which refused to install on many systems unless I started them in safe mode, etc. All in all, the half-dozen systems i cleaned up took several hours because of all the rebooting and screwing around that was necessary before the patch could even be applied.

___

50 computers over 8 hours = 9.6 minutes per computer, average. This time includes knocking on doors, explanations, going back to get rooms which were closed for some reason, booting up computers and rebooting them, loading the patches on to the machine and installing them, and all the regular crap that goes with handling 50 different computers with 50 different setups. Honestly I would say that 10 minutes per computer is simply amazing. These guys must be supermen to get a whole dorm patched in a day, unless they come in with an army of a dozen techs.

What can a student do? Preach alternative systems. Wean people off of Microsoft Windows entirely. I run 2 labs of a dozen Macintosh machines running Mac OS X and I haven't had to lift a finger to do much of anything for more than a year. The machines run perfectly and just laughed at all of the viruses, worms, trojan horses, and other problems that Windows computers have had to deal with. The same, I'm sure, is true of BSD and Linux based operating systems.

Take a look at the history of the Irish potato famine [ic.gc.ca]. The main cause of this horrible piece of history was a simple fungus. It spread so suddenly and completely because to grow potatoes quickly you can simply cut up one potato and plant the pieces. Each new plant is a genetic clone of the original potato. Thus when a disease hits one plant it quickly spreads and hits them all, turning a simple disease into an epidemic. The same is true of computers. A monoculture of Windows machines are much more vulnerable to the spread of computer infections than a mix of operating systems. Having one operating system dominate over 90% of the market is simply not healthy.

___

We are doing it now - fifteen of us at Utah State and we're averaging about 25 minutes a PC. A half dozen were destroyed by the software patches and the students are really mad.

how one school handles it

Here's an e-Mail that I got Friday that pretty much explains how my school, FGCU handled it: Student Housing Residents:

As you know, network access to student housing has been unavailable for the past few days. We are working on containing and controlling an outbreak of a virus that has been crippling our network. Unfortunately, there is a great deal of virus traffic coming from student residences. We have created CD's that contain Microsoft patches, Anti Virus software (trial or freeware versions) and specific fixes for known viruses (free fixes). We delivered the CD's to the Student Commons building so they may be loaned out to anyone who needs it. We have included instructions on the CD to check for the virus, update the OS and install the Anti Virus software. Please install or verify that your machine has these Microsoft updates and has at least one of the Anti Virus software packages loaded and running as soon as possible.

We have restored access to several residence halls on campus (A, B,E,F,M,Parts of phase 5). However, we are still seeing a large amount of virus activity coming from the residence units we have placed back on the network. It is imperative that everyone check their computer for virus's, install the updates and install an anti virus software package. We may have to turn off individual rooms in the residence halls in order to be able to restore access to the entire university-housing complex.

Once network access has been restored, make sure your Anti Virus software is set to automatically update its virus definitions or that you manually do it the first time and then set it up for automatic updates. Please keep your Microsoft Windows machines up to date by using the Windows update feature, found in all current versions of Microsoft Windows.

We encourage you to purchase an Anti Virus software package and maintain your subscription to the updates. The university uses McAfee Anti Virus, however, any well-supported Anti Virus software should be fine. Many Anti Virus software packages can be purchased locally at various stores such as Wal-Mart, Office Depot, Office Max, Circuit City and many others. You can even purchase and download the software off of the web from the manufacturers web site.

The current virus does not affect Macintosh or Linux/Unix computers, however, it is still paramount that all computer users install, maintain and run Anti Virus software on their computers.

We hope to bring the student residence buildings back on line soon. We appreciate your patience and understanding.

and from Washington

A real disaster. We're finding *all* of the student computers infested with viruses, worms and spyware. At this point about 5% of the machines are getting trashed by applying patches and we have some really mad students and faculty.

The only happy people are the OS X and Linux guys.

and paranoia

At our school the rumor is that this is the RIAA looking for downloaded music. Kids are frantically dumping their files.

[cindy]

Moving from one platform to another has some real costs. There is the money thing, moving your files and dealing with change. Most people hate change.

For me the move from XP (after a long string of Windows 2000, 98, 95 and 3.1) to OS X was fairly simple and refreshing because I was so fed up with Windows. I'm an astronomer, so the BSD nature of OS X was much closer to the research machines I deal with than Windows. And it just feels so elegant. I can't afford a $80,000 car, but I spend much more time in front of my laptop than driving, so the extra $1000 I spend on hardware an new software goes a long way.

Other people don't have an easy time. I've seen some struggle with Macs because they can't deal with the different. I suspect the great majority of users are in this boat.

Mitch Kapor has just switched from WIndows to OS X. It will be interesting following his notes. At this point it sounds like he has had a more difficult time than me.

I'm happy enough after six months with OS X that I'm getting rid of my old Dell desktop at work (I mostly cart around my 17" PowerBook these days anyway) and am replacing it with a dual processor G5 in a few months.

As a postscript I add that I was blissfully unaffected by last week. Our department secretary lost everything on her Dell when she tried to apply one of Microsoft's patches. Somehow she considers events like this part of what you have to go through to use a computer.