Clusterf***
It's funny isn't it when a live system goes belly up, how people start off helpul and then switch into self preservation mode.
Take me for instance. A humble BA with lengthy techy background. Well respected within the team and now the leading light into the investigation. I conducted a first round of investigation with a view to getting a quick resolution and getting the system up asap.
The symptom is data corruption. The cause unknown. So I set about building a timeline from interviewing people, log file analysis, etc etc. A timeline with key events that lead to the system downfall.
Interestingly the data corruption has been going on for some time, but ignored it seems. Until recently when the problems got so bad it was not possible to ignore them.
Imagine my delight when one such interviewee told me a manual database backup failed because of table locks, and upon further investigation, structural problems with the table which he "fixed on the spot". Ha! A smoking gun. Better check what the database maintenance procedures are. You know, good old fashioned consistency checks, repairs, error reporting, optimisation.
None. None at all. Yipeee, a smoking gun and a witness statement, and a positive ID in a police lineout.
What to do? Repair the table. Repair the data. Audit the data. Regression test. Put live. Good idea. (Limited time to conduct a full investigation - this looks like probable cause)
So I get as far as the regression testing when my key witness has an attack of amnesia. He now denies there were any problems with the database at all. Whatsoever.
I have to now reopen the investigation and go back to square one.
For any smart arses on here, we wanted to identify the high probability causes quickly and proceed with a moderate risk level (mitigated with frequent backups and rollback points) with a view to getting the system up as soon as possible.
The point here being that Mr Amnesia has cost us a week of investigation, planning and rectifying that may well be for nothing.
As we now have no probable cause we have to go back to the drawing board and start the investigation pretty much from scratch. I have told this business this will take a lot longer. The noose is tightening.
Now I know the panel on here will laugh, laugh some more, slag me off, make out it's all my fault, laugh some more etc etc. But in among all that noise, all that dross and tomfoolery may be a germ of an idea, some pearl of wisdom that may invoke a thought process in me that leads to the solution.
As such I will personally put up a financial reward for anyone that does this. If multiple people share the idea in various posts you can either split the reward or give to charity. PM me for details.
Failing all else this will provide SASguru with some much needed mirth and pisstaking. So either way a total win win.
Thinking caps on chaps, Miss Marples, Inspector Cluseaus, Hercule Poirots.
Take me for instance. A humble BA with lengthy techy background. Well respected within the team and now the leading light into the investigation. I conducted a first round of investigation with a view to getting a quick resolution and getting the system up asap.
The symptom is data corruption. The cause unknown. So I set about building a timeline from interviewing people, log file analysis, etc etc. A timeline with key events that lead to the system downfall.
Interestingly the data corruption has been going on for some time, but ignored it seems. Until recently when the problems got so bad it was not possible to ignore them.
Imagine my delight when one such interviewee told me a manual database backup failed because of table locks, and upon further investigation, structural problems with the table which he "fixed on the spot". Ha! A smoking gun. Better check what the database maintenance procedures are. You know, good old fashioned consistency checks, repairs, error reporting, optimisation.
None. None at all. Yipeee, a smoking gun and a witness statement, and a positive ID in a police lineout.
What to do? Repair the table. Repair the data. Audit the data. Regression test. Put live. Good idea. (Limited time to conduct a full investigation - this looks like probable cause)
So I get as far as the regression testing when my key witness has an attack of amnesia. He now denies there were any problems with the database at all. Whatsoever.
I have to now reopen the investigation and go back to square one.
For any smart arses on here, we wanted to identify the high probability causes quickly and proceed with a moderate risk level (mitigated with frequent backups and rollback points) with a view to getting the system up as soon as possible.
The point here being that Mr Amnesia has cost us a week of investigation, planning and rectifying that may well be for nothing.
As we now have no probable cause we have to go back to the drawing board and start the investigation pretty much from scratch. I have told this business this will take a lot longer. The noose is tightening.
Now I know the panel on here will laugh, laugh some more, slag me off, make out it's all my fault, laugh some more etc etc. But in among all that noise, all that dross and tomfoolery may be a germ of an idea, some pearl of wisdom that may invoke a thought process in me that leads to the solution.
As such I will personally put up a financial reward for anyone that does this. If multiple people share the idea in various posts you can either split the reward or give to charity. PM me for details.
Failing all else this will provide SASguru with some much needed mirth and pisstaking. So either way a total win win.
Thinking caps on chaps, Miss Marples, Inspector Cluseaus, Hercule Poirots.
Comment