• Visitors can check out the Forum FAQ by clicking this link. You have to register before you can post: click the REGISTER link above to proceed. To start viewing messages, select the forum that you want to visit from the selection below. View our Forum Privacy Policy.
  • Want to receive the latest contracting news and advice straight to your inbox? Sign up to the ContractorUK newsletter here. Every sign up will also be entered into a draw to WIN £100 Amazon vouchers!

Accidentally nuked production database

Collapse
X
  •  
  • Filter
  • Time
  • Show
Clear All
new posts

    Accidentally nuked production database

    https://www.theregister.co.uk/2017/0...stantly_fired/

    Who would be that stupid? Accidentally nuking prod - we all make mistakes. Once you make a mistake, STFU. Don't go onto Reddit.....

    #2
    Originally posted by BrilloPad View Post
    https://www.theregister.co.uk/2017/0...stantly_fired/

    Who would be that stupid? Accidentally nuking prod - we all make mistakes. Once you make a mistake, STFU. Don't go onto Reddit.....


    The Chunt of Chunts.

    Comment


      #3
      Thats not their fault...

      The system admin team and especially networks need 100 lashes for abject ******* stupidity.

      1) Segregate DEV, Test and Production environments completely.

      2) Dev team should never know production passwords.

      3) Dev team should not have live user accounts.

      everything after the test environment should be handled by a deployment team NOT the developer team.

      The fact that they could sit on the Dev systems and still hit live shows a total lack of understanding...

      Comment


        #4
        Originally posted by bobspud View Post
        Thats not their fault...

        The system admin team and especially networks need 100 lashes for abject ******* stupidity.

        1) Segregate DEV, Test and Production environments completely.

        2) Dev team should never know production passwords.

        3) Dev team should not have live user accounts.

        everything after the test environment should be handled by a deployment team NOT the developer team.

        The fact that they could sit on the Dev systems and still hit live shows a total lack of understanding...
        Very true, true segregation is key in any well designed system.

        I generally deal with enterprise reporting software right across the board so have worked with most databases at some stage.

        One thing I always stress is having the right number of environments procedures and DR etc.
        I usually manage to "sell" it to be people that get it.

        i.e.

        Less risk
        Less stress
        Less down time
        Less money on expensive major incidents

        Funny though you see some of the most ludicrous things and not just in the smaller Blue Chips either.

        I wasn't particularly surprised to find a major gambling business with one "Prod" system complete with a centralized spreadsheet to patch in the data holes in their reporting.
        Obviously some clown left the spreddy open for the weekend causing 600 reports to have failed by the Monday.

        They then had to pay me to set up optimised batch jobs to play catch up during the week.
        Got a few more days billing out of that one
        The Chunt of Chunts.

        Comment


          #5
          Production DB access should be controlled more tightly than the bank accounts for many reasons.

          DB creation etc should be automated and only done by DBAs not Devs.

          Poor management
          Always forgive your enemies; nothing annoys them so much.

          Comment


            #6
            Originally posted by vetran View Post
            Production DB access should be controlled more tightly than the bank accounts for many reasons.

            DB creation etc should be automated and only done by DBAs not Devs.

            Poor management
            Lots of Devs have excellent knowledge of databases and are perfectly capable of maintaining a database. It's nothing special that can't be learnt.

            Comment


              #7
              Originally posted by bobspud View Post
              1) Segregate DEV, Test and Production environments completely.
              Racist!!!

              Comment


                #8
                Originally posted by woohoo View Post
                Lots of Devs have excellent knowledge of databases and are perfectly capable of maintaining a database. It's nothing special that can't be learnt.
                No thats not the point. The point is the DB is a crown jewel and no one especially a ******* first day dev should be able to get anywhere near it.

                Scripts that run against it should be peer reviewed to death and it should never be accessible from anywhere on the dev teams floor.

                Im not saying the dev team should not be able to do hourly release cycles or that Jenkins can't be made to run like the wind. Just that after you have the release candidate, the upgrade to live is handled by someone that respects the need for maximum up time.

                even with the best code team on the planet, we have all seen deployment scripts arrive with the wrong configuration settings and so for that reason you are an utter and complete moron if your dev team can ping a live instance from their terminals, let alone have the login details and permissions to execute code...

                Comment


                  #9
                  Originally posted by woohoo View Post
                  Lots of Devs have a little knowledge of databases and are perfectly capable of ballsing up a database. It's nothing special that they fail to learn.
                  FTFY.

                  qh
                  He had a negative bluety on a quackhandle and was quadraspazzed on a lifeglug.

                  I look forward to your all knowing and likely sarcastic and unhelpful reply.

                  Comment


                    #10
                    Originally posted by woohoo View Post
                    Lots of Devs have excellent knowledge of databases and are perfectly capable of maintaining a database. It's nothing special that can't be learnt.
                    They may have, however if you have a DBA you should segregate the duties. The only people who should create DBs in production are DBAs following detailed written instructions written on Dev, tested in QA and then migrated securely to Production. Its the only way to make sure migration goes well - trust me I have tried other ways.

                    No Dev should create a DB on Production they tend to 'play'. Anyway the production DB is owned by Support not Development.

                    No real need for a Dev to create one on QA. This is test for production.

                    A dev can have access to Dev and create DBs with pleasure. In fact I would expect them to do so and script them so the scripts can be tested by DBAs in QA & Production.

                    But smaller teams may do slightly different.
                    Always forgive your enemies; nothing annoys them so much.

                    Comment

                    Working...
                    X