What is the Biggest IT Cockup You've Ever Made?

[BrilloPad]

I thought of it first

http://forums.contractoruk.com/gener...ml#post2425723

[BrilloPad]

Back in 1987 the bank I was working for had a great FX IT system. Positions were updated real-time : quite revolutionary then. I was helping on the release, being overseen by the expert. I had a TOPIC terminal and the expert was far more interested in the shares he had bought on some insider tipoff. At some point I got a drive letter wrong and the next day everything collapsed in a heap. There were no post-release checks.

FX desk could not trade next day. Being young and new I ended up having a few fingers pointed my way. However I stuck with the truth and it was a great learning experience.

[SeanT]

Ignored a disk hanging out of a server in a room I'd recently taken over in somewhat bad circumstances (long term hospitalisation as opposed to dismissal, of someone I have great respect for) for way too long. First sysadmin job after being on the helpdesk for a couple of years. Another disk started making funny noises and went red so I pulled it and swapped it expecting a rebuild to kick off. It was a RAID5 array but without the benefit of an initially configured hot spare of course! (never assume) It had been running without redundancy for months. I had to bring up the new box and do the restore on a Saturday night / all day Sunday (from home, thankfully). Less than a Terabyte but a mixture of home directories and shared drives so took forever to copy (from an NTFS volume used for the "warm" backup to the Novell NSS filesystem on a new server - hadn't even installed one into the tree before!) People were pissed about the timestamps on the directories on Monday as I just threw a copy command at it in Linux, but the permissions and the files were all back for start of business. Seems a bit tame now though!

[northernladuk]

I was working late night at a Tambrands site transferring all their drawings in to DXF format using various scripts and batch files. On one of the trips to the server room to set the next load off I though I could improve the throughput by killing some threads off. Worked a treat, the drawings were copying much faster but it was eerily silent. Went back on to the shop floor and some of the automated tampon making machines seemed to have stopped. (gulp).
Had to call the out of hours guy out to restart it all. Very uncomfortable day next day on site. Got my one and only written warning for that.

[SeanT]

You've never been properly sacked NLUK? You are full of surprises

[northernladuk]

You've never been properly sacked NLUK? You are full of surprises

No but I appear to have been offered voluntary redundancy more times than is normal

[jamesbrown]

No but I appear to have been offered voluntary redundancy more times than is normal

Kerching.

[AtW]

No but I appear to have been offered voluntary redundancy more times than is normal

By your accountant?

[fullyautomatix]

Ages ago I wrote a small web app for a local sandwich company which was essentially ordering a customised sandwich. The data was then emailed to a third party who would fax it to the kitchen. No database involved since they wanted it done cheaply and quickly.

Anyway, all done, tested on all environments and before go live I wanted to run some automated tests and decided that the best way was to comment out the line of code that sent the email, since I did not want the kitchen to start making sandwiches that I was testing.

On go live, I completely forgot about this and the site was launched and appeared to be busy. But no orders reached the kitchen and the phones went off the hook with customers angry that their sandwich never was delivered.

[grabri]

20yrs ago, trying to diagnose a memory corruption problem on a remote monitoring system I suggested that we would be able to get more information out of the system by turning off the watchdog timer and allowing the system to display the stack trace and allow us time to debug.

I was new in the company and the guy who had been assigned as my mentor suggested caution as there was a chance that the memory corruption could impact the telnet session that was our only way of communicating with this system. I decided to ignore this sage advice, disabled the wdog and sat back and waited for the issue to occur. After about 10 mins the system dumped a stack trace as expected and also trashed the telnet connection, as warned.

We made some progress on diagnosing the root cause of the problem, but had an uncomfortable call with the client late on a Friday afternoon to send someone on a 5hr round-trip to press the reset button.

I learned some valuable lessons that day.