For whatever reason, ntppool.org is blocked at my work.
And of course, you don't get the page that states why when the website is served via https. Not that I need to see the page to know it was either blocked for "hacking" or "entertainment", and I'm guessing it's not entertainment.
Edit: This probably explains why our clocks have been off by 45 minutes since Monday. I guess it will be entertaining to see how long it takes for IT to figure this one out.
I got told off for diagnosing issues in the past. The IT director is a megalomaniac and interprets it as a challenge to his power. The only time I offer suggestions now is when one of his employees specifically asks me for help.
Edit: I realize "got told off" didn't really capture what happened. I came in early one day and noticed we were having a dns issue. I manually refreshed my DNS cache and it started to work. I sent him an email to let him know that the DNS cache was expired. He told me I was out of line and complained to HR.
I had to go meet with HR, which was pointless since they think he is on a power trip as well. Anyways he added a line to the IT policy that specifically prohibits "performing a diagnosis on the network or any of IT managed systems."
I got into trouble at university when I was running a CAD session on X (Cadence VLSI design FTW, not). Some asshat had telnetted in and was trying to brute force root because it was a faster machine than the crappy sun4's dotted around. It was spewing all over the framebuffer. So I logged into another box and sent him an email saying pack it in and that I was trying to work and that I'd report him for AUP violation to the Sysops.
He complained (?!!!) and the next thing I was in front of a tutor getting a bollocking for it. No explanation was allowed to be returned or appealed, permanent black mark on my record.
And that's when I learned about university politics, gave them the fat middle finger and got a job and left a massive 11 page long diatribe about the charlatans at the place.
I sent a complaint to my boss and HR about IT recently that was around 30,000 words. It was basically 500 words describing the level of incompetence of the IT director, the remaining 29,000 words were just supporting evidence. I'd say it was a massive waste of time, but the IT director has basically made his expertise unassailable, and the only higher authority he recognizes is Microsoft.
I basically went through all my tickets and emails, took IT directors claims, read the MSDN articles on the topic, and pointed out all of the places where what he said was not only wrong, but grossly wrong. Things like "sometimes databases lose data." That's funny, cause I'm sure the team that built SQL Server 2012 would have something to say about that. Why don't we look at the documentation on ACID principles.
I've come to expect the IT director to be a moron at this point, and I had been trying to roll with the punches. However, my work has been going missing, and I got in trouble for it. He denied losing it, then he blamed the database for his incompetence. I was so angry I was awake for 3 days straight cross referencing everything.
The lost data wouldn't have even been a problem if he had just told me within a day or two. I only find out it's missing when we try finding it weeks or months later. Then I have to waste my time doing a forensic investigation in to the scope of the problem.
The most annoying this is that the last person in this role never had any of these issues because nobody ever audited our data. I have managed to instill the idea that data can be 100% correct, and we should always be checking our data to make sure it's right. So now I get blamed anytime information is missing or incongruous.
I know that otherwise well-reasoned emails seem like a rant once they reach a certain length. I kept the body of the email to 500 words, and included a pdf of supporting documentation. I let the email sit for a week before I sent it, and had another manager read it as well to make sure the tone was alright.
My entire argument was contained in the body of the email. I knew that was all it would take to get my boss on my side.
The PDF was aimed at HR. The IT director has a lot of power under the IT policy, and he has used it in a retaliatory manner in the past. My goal was to stop the IT director from retaliating so I could do my job while I look for a new one.
I write my emails as if they could be leaked to the general public at any time, and I'm certain that the email wouldn't reflect poorly on me, even out of context. Maybe there was a better course of action, but I don't think I did much to hurt myself.
As unfortunate as it is, most people, especially us engineering-types, are not good at seeing what drives peoples' decision making. That means we do things like this that only make us look worse under the belief that they will help (including the belief that if it doesn't help now, perhaps it will later; this is slightly more likely, but still extremely unlikely).
The important people often see those things, assume it's an incoherent screed from a disgruntled piece of crap, and delete it without reading (I know from personal experience; such completely accurate and valid diatribes have gotten me fired on the 3rd day of my 2-week notice and, on a separate occasion, a running joke among the big shots for months after my departure, where one would specifically talk about how the email came in while he was on the toilet, at which point he cackled and promptly deleted it without reading).
To anyone non-technical, that letter is all mumbo jumbo. It seriously might as well be in a foreign language. They are not going to check a few of the cross-references and see that you're obviously right. Even though you might hope they'll do this just one or two times, they won't.
They are not going to ask the people called out in your letter to account for your accusations.
They are going to write you off as an unhinged, angry, and worthless nothing-tier employee/student/whatever, make fun of you for a long time, and then forget all about it.
Humans base their decisions on personal trust/credibility. The way to win against an evil IT director is to obtain far more trust/credibility in the eyes of his bosses than he has, and then to use that credibility to your advantage.
That's a lengthy and difficult process, especially when you start out as a subordinate and the boss has a lot more access and ability to frame your efforts to his advantage.
I personally have never had the patience to undertake such political subterfuge seriously and I find such undertakings both incredibly frustrating and soul-crushingly phony and hypocritical, but I am now convinced it is the only reliable way to get real career success and mobility. Thus I accept that any career success I enjoy will be lucky/accidental.
Employment and career IS a popularity contest. Good software is somewhere between the 10th and 20th most important career concern for a developer. The number one concern for anyone trying to make it in white-collar America is to be as well-liked and popular as possible. Most of the time, love of colleagues and love of bosses are symbiotic and they feed off of each other (as long as you're sycophantic efforts aren't TOO obvious), but to the extent that a situation arises wherein someone's love has to be preferred, prefer to get the love of the bosses.
This is the sure path to career success. Disregard truth, objectivity, and practicality. No one cares about you or what you think, they don't care about what you judge to be practical or wise. They didn't really hire you for your experience or insight even though they want to pretend they did. They hired you because they thought you would make them feel and look good.
Not only bosses, but people in general, care only that your presence and actions are generating pleasant feelings for them. Do this reliably, disregard everything that is not this, put only as much energy into the tasks required by your actual job description as you must to be passable in the unlikely event of a performance audit, and pour the rest of your energy into social development. If you're going to make it as a company man, that's the only reliable path.
I don't disagree with you about the dishonesty of living a suck up life, but try actually being poor for a little while. Your tune will change to "Where would you like me to suck?" and "How hard?" very quickly.
I have the advantage to be able to leave a job and find another relatively easily, and I would definitely agree that I'd rather leave then suck up, but for those who don't have that kind of mobility, sucking up and playing politics can literally make the difference in making car / house payments, sending your kids to college, etc.
Yeah, this is also an element that's important not to discount, and can help describe why it's hard for engineers and other high-IQ roles to accept that social compliance is the #1 factor in all ongoing voluntary associations.
We're really spoiled in tech because the field is so abundant. We offend someone or get offended, and we're off to greener pastures within weeks if not sooner. I've come to believe that hopping around like this, which I've done for most of my career up to this point, is not healthy, but the availability means that technical people don't have to learn to conform as well as everyone else to survive. And while that means we may be able to keep a job, it's hard to move up.
These things that we struggle with are just normal life to a lot of people. They had to swallow these compromises early in life when it was apparent that good feelings were all they would be able to offer.
This contributes to the cycle because those people pay their dues to the establishment, go through the process, and get used to the circlejerk. They then expect everyone else to do the same.
When someone wants to come in and challenge some of their thoughts, opinions, or practices, even minor things, in what the challenger feels is a sporting way or a way to drive an interesting and inoffensive discussion, the "good feelings violation" siren fires off in the non-technical person's head.
This brings in a large flood of negative feelings and resentment, including but not limited to jealousy that you can express your thoughts openly while they've always had to kowtow, a sense that you're entitled for thinking you should be able to do this and "dictate from your expertise" (as was expressed about me once) instead of "climbing the ranks" the hard way and then quietly and subtly implementing your opinion after you've won the social clout like everyone else has to do, and a sense that you may represent a threat to the perceived competency of the challenged person (and those least competent will be the most aggressive protectors of this perception) in the minds of the people whose trust they live off, which is really everyone -- colleagues, subordinates, and superiors -- which means it's very difficult to overtly question or discuss anything done by anyone, even in what you believe is a polite or considerate way.
Exceptions are basically not made to this. The potential of substantial data loss, massive security holes, etc., are irrelevant. If someone grossly incompetent like this is on your team, the smart move is NOT to challenge or disprove, because again, no one evaluates proof on any basis except "which proponent do I trust more?". The smart move is to frame the situation such that his failures are opportunities for you to deduct from his social clout and add into your own, without ever firing an alarm in anyone's mind that you're trying to do this.
It's all an image game. As an individual contributor, you can avoid a lot of this game as long as you're non-threatening, churn out semi-reliable work, and are at least not annoying if not socially pleasant. Once you try to move up the ladder, even just one rung, image and likability goes from 65% of the equation to 99%.
Yeah, I definitely agree with that to a large extent. That's why I stated that any career success I personally enjoy will be based on luck and/or accident instead of political skill. I've tried to bring myself to find some way to reconcile these paths and still feel that I'm not throwing away my integrity, but I've been having a hard time doing that. Trying to find that reconciliation has actually been a large part of my personal struggle over the last year or so.
I think the thing is that there are degrees here. A lot of the things that we are socialized to consider "bad taste" are not actually immoral and unfair (for example, aggressive SEO), and you have to meet on that playing field if you ever intend to be competitive. But you have to identify the limits of what's just getting into the nitty gritty of business and what's crossing the line into being a cheater/liar/phony.
There also seems to be some people for whom social camouflage comes naturally. These people don't feel like frauds or phonies when they cater their preferences and likes to match those of the group around them. It's hard to compete with these people because they have no compunction about being yes-men and they have no malfeasance behind it, since they don't really even realize they're doing -- they only realize that they're making the people around them happy. They truly just have nothing original or important to contribute and don't realize that they're mimicking everyone around them, which, as we've discussed above, is a great skill to have on the path to career success.
I think that developing an active, interesting conversational style and taking care to frame arguments in the most emotionally influential way possible without altering, distorting, or seriously misrepresenting them is perfectly in bounds. Most people do this intuitively to some extent or another, and I believe that intuition can be improved with practice and training, and that that's a great thing for anyone to develop.
The problem is that that's about where I stop. Even if there is nothing immoral about dragging yourself to a baseball game with the bosses, it still feels painful and phony to do that kind of thing. Your unscrupulous and/or unaware competitors, however, will waste no time immediately making themselves as likable as possible by adopting all of the boss's favorite things and habits, refusing to criticize him or anyone whom he holds in high esteem no matter how grossly dangerous their actions are, etc.
It's a tough game. Sometimes I try to believe that there's some cove of people that aren't this way out there, but it always gets shot down as soon as I start believing it again.
I'm starting to think that the only bonds that allow real honest participation are permanent and non-revocable bonds like parent-child and sibling relationships. People still get offended in these and rarely they may even fully withdraw, but most of the time everyone accepts each other and has to get over the perceived slights. I'm not sure there are many voluntary relationships (certainly not relationships where the continued relation is predicated upon a regular payment) where people do that.
I encountered a similar IT manager in high school. I kept telling him that netsend wasn't locked down and that any user could run a .bat.
He told me I was wrong. So, I wrote a .bat with a netsend command and emailed it to all staff. Multiple staff clicked on the attachment.
Once they figured out it was me, they made me start a computer club with the IT manager as the supervisor of the club. First order of business was locking down .bat execution.
> Once they figured out it was me, they made me start a computer club with the IT manager as the supervisor of the club. First order of business was locking down .bat execution.
That seems like a great way to handle this situation. Some ignorant other schools might have kicked you to the curb because you were spreading "malware".
Im endlessly glad that my high school was extremely tolerant of us exploring the system and messing with things as long as we didn't try to cause harm (eg, deleting one file is okay, trashing a whole network drive is not) and reported what we found to the school IT manager. Most of us ended up as techs for the school and district as a student job. Several of those students went on to be whitehats. (Who knows where careers would've gone if they'd been discouraged and come to view the system as the enemy.)
They also were way cooler than they had to be about the several times we took down the network or broke the porn filters, or the time we port-scanned a district tech's machine, or had a whole collection of malware on the network drive, or....
I just worry about how students like me would fare in schools these days.
In secondary school I discovered Windows Registry - in particular, the LegalNoticeCaption and LegalNoticeText keys, which let you set a pre-login information dialog. I left a funny message on a computer in the school lab, saying something to the tune of "What a shame it isn't Linux".
Two hours later, I faced an angry teacher (who was also a math teacher and the lead teacher for our class). She said something about the next person after me freaking out that "the computer has viruses". Got a bad note for behaviour, spot-check of math homework leading to three F grades, and she also tried to take away my notebook with notes about Windows Registry -.-
Currently looking. I finally came to my last straw recently.
A coworker sent me an email because some data I was in charge of adding to the system was missing. I looked, and somehow data in our database had gone missing. I use the data to add information to another database, and that database had the information in it. So somehow he managed to lose information in SQL Server.
A few weeks later, my boss brought me in to his office because about 25k rows had incorrect information. I went to check my notes, and all of my notes from a period of 10 days were missing. I had been creating a changed file log (because files have been lost in the past), and I could see I had created notes during those dates, but they didn't exist anywhere.
The only evidence I had done my job at all during that time were the emails I sent to other people. (Thankfully Microsoft hosts these, so the IT director can't mess it up.) I had sent somebody an email that basically said "I found error [x], but I fixed it."
For a little while I was questioning whether I was insane. I mean, I would never have believed it was possible for data that you've added to an ACID database to just disappear. If you can't trust ACID principles, what can you trust?
I've seen this happen as well. Data "mysteriously" reverted and it turned out to be an unrequested database restore. They reverted to a recent backup but didn't mention it to anybody so a few days of data was missing.
That is definitely what happened. One manager found out about this and called him and ended up verbally tearing in to him for like an hour. She is really task-oriented and gets really frustrated with how often I want to change things just to see what happens. It felt really good to hear her step up to defend me, because she would only do that for someone who she really respects.
What kind of firm are you in (and perhaps more important: country/culture)? I've actively been a friendly pita wherever I worked and always found people to help search for answers, even if the question was not really in the line of work. Have a discussion related to tax and payrolling going on right now. Couldn't make sense of company policy even when reading / quoting the law (ianal). Still the surroundings stay friendly (or I'm in massive bubble). Getting into trouble for having a (truthful) opinion, sounds like a showstopper to me. Just don't email the big shot CEO if you haven't got your story straight, that would be my perferred level of openness.
I really like the term "friendly pita". When people complain about tech support being unhelpful, I've often found the opposite. Be genuinely friendly and relentless and I've found even incompentent people will fix things. Which I also genuinely appreciate.
Sincerity helps, both as a shield and as a way to diffuse the inevitable frustration that comes from troubleshooting tech.
Of course, I've got a strong force of will, too, so I tend to mix poorly with charlatains like the grandfather's post. I really feel for the souls who have to work under that jerk.
I'm definitely a friendly PITA. Most of my coworkers get it, though some definitely think that I'm just creating more work for myself. It's really only the IT director who is so defensive, and unfortunately he has the power to back it up. HR goes out of their way to find answers to questions, and my immediate coworkers are really driven.
If you want to sidestep your IT, you can report the site to your filter provider as miscategorized. I'd argue that ntppool.org is not related to hacking. Most of the filter providers have publicly available forms or email addresses for this stuff. There's not a high burden of proof to worry about either.
If a cached DNS record expired it would not be a problem. If a bad or old record was cached before TTL, clearing your cache works to update it. Maybe ITs ego didn't like your terminology of DNS cache being expired, and telling them theirs is- by definition expired records get looked up again because they no longer exist locally. As former IT it was mildly irritating to get unsolicited advice, But still it sounds like a redicilous over reaction on ITs part. I'm assuming you didn't actually clear a local IT managed server's local DNS cache.
Right, the DNS records weren't expired, they were no longer accurate. I'm not sure what the word for that is. The Windows DNS Cache on the local machines was inaccurate. Basically, I couldn't access any websites I had accessed before, but if I tried accessing a new site it worked fine. If I forced a DNS request for a site I had already visited, that site would start working but all the other sites would still be broken.
I knew what the problem was because I had run in to that issue a few years back with my own computers. After I updated my DNS, all the Windows computers were having issues, but none of the Linux ones were. That's when I learned that Linux doesn't typically cache DNS records on local machines.
It's called the resolver cache, and Linux does it too. Most commonly with nscd (use "nscd -i hosts" to clear it, keep in mind this is not specific to DNS requests). In the specific situation of a web browser it's more complicated and you'll find that the web browser caches these things too. If you run Chrome check out chrome://net-internals to get a glimpse of what's going on there.
Thanks for the tip on net-internals. I think I used a similar feature in Firefox a long time ago, but I've never used this tool in Chrome before.
I know Chrome in Linux definitely doesn't cache DNS requests. I believe Linux only keeps the DNS info around until the socket closes, but I don't know the actual implementation. I looked in to the ncsd man page, and it looks like DNS info isn't cached, only open sockets.
I know Chrome uses the system DNS cache in Windows, and my understanding was that all browsers in Windows used the system DNS cache.
I don't know a lot about how sockets are handled. I thought they were discarded as soon as they were closed, but they could function similarly to a DNS cache. I though a DNS cache stuck around for a lot longer though.
I'm currently looking for a role, but I've worked here for 3 years because I have been able to basically do whatever I want. I'm a combination of analyst, data scientist, and marketer. I'm in charge of our appeals, from strategy all the way to the money coming back in house.
We have over half a million constituents. It's really great if you are interested in testing. I've sent out mailings where I'm testing 10 or 11 different factors. I've tested basically everything: the size of the font, the weight of the letterhead, the format of the coupon, how the letter is mailed, the dimensions of the envelope, the structure of the ask, etc.
Even with the added expense of testing, I've decreased our cost to raise a dollar by over 20 cents.
On top of that, I've automated all sorts of processes. At this point, I think my processes save about a man-month per month.
So, I have a lot to feel good about, but I'm really being limited on the technology side of things at this point. Biggest barrier is that I haven't quite decided what I want my next step to be at this point. I'm afraid I'll end up with a job with a much narrower focus and get bored.
Good call. I thought it was possible to message me as long as the email field was filled in. I hadn't put my email in the description because it's my actual name at gmail. I've added it now.
you've got a great skillset. I think you could find a similar role at another company where you weren't being limited by a shitty IT department and internal politics.
Thanks, that's a real confidence boost. My formal education was in biology with an almost minor in computer science. I am a little self-defeating sometimes because I don't have a stats/marketing or CS degree, and a lot of job postings have a relevant degree as a requirement.
The IT may be terrible, but I'm very thankful for the opportunity I've had to develop these skills. It's really unfortunate how one or two people can ruin an organization. The rest of the people here are incredibly kind and incredibly driven people.
I don't know why an org would be blocking NTP. But ideally, IT would run their own NTP service (if the shop is big enough), block NTP to any other org, and sync their server with upstream. Similar to package management, DNS, etc. - self-host it all.
Blocking UDP 123 upstream is a common tactic to avoid participating in NTP DDoS botnets.
You're best off running your own NTP infrastructure that's isolated from the public Internet NTP servers if you want stable time sources. Put a CDMA NTP server in two locations, let them sync, and go. You can get cheap EndRun servers on eBay for a few hundred bucks.
It's tempting to believe telecom has got their act together, but I've found the public Internet NTP infrastructure to be more stable than CDMA/GSM refclocks. YMMV.
It's why you need multiple sources. The NTP protocol and implementation handles sudden skews from a source. When done properly, CDMA clocks can offer a much more stable source than the public internet, and you can filter out UDP 123 upstream.
Yes, but when your IT is incompetent they do the first half (block NTP to anywhere outside your network), then don't provide their own time source. Then try to blame users when they complain that their time is always wrong.
We do have our own time source. Our NTP source is locked to that server via GPO. NTP requests aren't blocked, because I can still set NTP on things like my printer to other NTP sources and receive updates.
I'm willing to bet they don't realize that they've blocked their server. What I want to know is how the drift got to be this bad in a matter of days. It's like whatever computer they are using as the NTP server doesn't have a real-time clock.
"It's like whatever computer they are using as the NTP server doesn't have a real-time clock."
It sounds like their domain controller holding the PDCE FSMO role is a virtual machine. This advice might be outdated but last time I checked it was better to host it on a physical machine. If they're restoring DBs from backups without notifications and managing their Active Directory environment like this I can only imagine how fragile your infrastructure really is. Good luck, you're going to need it.
>It sounds like their domain controller holding the PDCE FSMO role is a virtual machine.
Called it. I didn't even think about the fact that it was a virtual machine until you mentioned it. Everything makes so much sense now. The randomly disappearing files, random issues with file locking. I just assumed the problem was they had no idea what rules to use for file replication.
> I can only imagine how fragile your infrastructure really is.
Yeah, it's a nightmare. Ignoring the "files disappear and are never found" issue, files get overwritten with earlier versions all the time. I'm not entirely sure what their hosting situation is, but I suspect they are running virtual machines in virtual machines.
On top of that, most applications are only available on VMs through a Citrix connection. This isn't inherently a problem, but they have it set up so everyone is logged on to the same machine. This causes issues with most MS Office applications, because they assume one user per computer is accessing the files. So file locking is completely broken, because Windows looks at the file and says "Hey, that's me, I have that open!"
It's the biggest nightmare with Access applications, (which we have a ton of), because Access assumes that everyone has their own copy of the front-end (client) database, and each independently accesses the back-end (server) Access database. Optimistic locking doesn't work at all with the "everyone uses the same file from the same computer" approach, and pessimistic locking barely works. I've set up the client databases to automatically create a copy for each user, but as you can imagine it's a bit of a nightmare to keep everything updated.
Odds are high that someone set up a high-accuracy server on the local network and expects everyone to get their time from that. ...which is not a bad idea if they had the sense to make it known that they've done so.
AD and a few other systems get very snippy if the time various servers have isn't in sync, although they don't have much of a problem if every server has the wrong time (within reason).
And of course, you don't get the page that states why when the website is served via https. Not that I need to see the page to know it was either blocked for "hacking" or "entertainment", and I'm guessing it's not entertainment.
Edit: This probably explains why our clocks have been off by 45 minutes since Monday. I guess it will be entertaining to see how long it takes for IT to figure this one out.