Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

I once worked at a company which separated IT into 3 teams: developers, DB-sysadmin (ops), and QA (who also managed deployments). Releases were supposed to go in a waterfall model from the Dev group -> QA group -> Ops. QA wanted Dev to submit Word documents for each release with blanks to be filled in with server names. However Ops was so distrustful of Dev that it was not enough for them to lock us out of Prod using regular security tools, we were also not allowed to know the NAMES of servers in Prod or how currently deployed systems were grouped.

Every release was an Abbott-and-Costello "Who's on first?" routine. Do you have any idea how hard it is (especially in computing) to ask for something without being able to utter its name?

QA: "You left servername blank on this deployment document."

Dev: "I know; Ops won't tell me. Just ask them for where the service is currently."

QA: "Ops says there's 5 unrelated legacy services with that same project name, on different servers."

Dev: "5? I only knew about 3. You know, if I could query the schemas of the Prod DB, I could tell you in a jiffy which one it is."

Ops: "Pound sand. If you want look at databases that's what the dev DB server is for."

Dev: "Erm, OK well can I give you a listing of the Dev DB schema and you tell me if it looks like the one the Prod service is talking to?"

Ops: "Oh I see you want us to do your job for you now? You can compare the schemas."

Dev: "OK..."

Ops: "Just tell us which DB server you want the schema pulled for."

Dev: "But you won't tell me the server names."

Ops: "No."

My point is this is how bad communication can be when ops and dev are not on the same team.



I have trouble imagining the incentives driving Ops in that conversation.


Devs hardcoding things in their software in a rush making the software tougher to deploy and operate causing greater incident rates and therefore page-outs. Devs interested in greater resilience and stability in their software should be opting for dependency injection of pretty much every damn thing in the world around them whether it’s a network service or file system location. Otherwise, presume that it can go away at any time. A common pattern among developers trying to save time that costs more in the long run is to hardcode a path to an executable. A simple /use/local/bin/ buried in an infrequent job that is installed on developer machines but never in prod is all it would take to cause an incident in prod that costs the company millions. I say this both as someone that has written this and had to fix others committing the same error in their code and QA passing it along.

Ops tends to be where the brunt of technical debt is truly buried. Bad code is one thing but seeing the code in action with real world data is a different beast altogether.


The easiest way to ensure the stability of the systems that you own is to prevent anyone from changing things on them.


Ideally the guarding of valuable company (and incidentally customer) data.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: