Fairly large Nomad Enterprise user here, and I just want to say thanks for all of the work you and the team put in. I'm a big fan of Nomad and really appreciate the opportunities it has afforded me.
Regardless of the general sentiment, hoping for the best outcome for all of you.
Also a Hashistack enterprise customer and we have the same experience.
Thankfully we don't have to talk to Consul support very often, but each time we meet with Consul product people it's like they don't even know Nomad exists.
We're starting to adopt Consul Service Mesh with Nomad and I am not excited.
Totally agreed on being disappointed with Support. There are some diamonds in the rough, but I'd give them a hearty thumbs-down overall. Agreed on feature requests too. If it isn't already on the product's roadmap, good luck!
> They won't answer any questions explaining how given feature works, referring to solutions architects, which takes months to get access to them.
It sounds like your account team is really letting you down here. We meet with ours, including SAs, bi-weekly. Generally we can get answers to our questions/issues within a few business days, including feedback from product engineers.
> It sounds like your account team is really letting you down here. We meet with ours, including SAs, bi-weekly. Generally we can get answers to our questions/issues within a few business days, including feedback from product engineers.
Thanks. So I see that this is not normal, maybe I can escalate.
This is one of my favorite things about Thanos. We run Prometheus in multiple private datacenters, multiple AWS regions across multiple AWS accounts, and multiple Azure regions across multiple subscriptions. We have three global labels: cloud, region, and environment. With Thanos's Store/Querier architecture we have a single Datasource in Grafana where we can quickly query any metric from any environment across the breadth of our infrastructure.
It's really a shame that Loki in particular doesn't share this kind of architecture. Seems like Mimir, frustratingly, will share this deficiency.
I'm probably different from most people, but I really wish there were more rail options in the US (I'm in Texas). It's a 4 hour drive from where I live to my hometown. I really don't enjoy driving, and would happily take a train even if it took twice as long as driving. I'd visit home a lot more too.
If it's a high speed rail line at some distance it becomes faster to go by rail. Even slow rail. I once took the train from Glasgow to London and it was about 5 hours. Same drive I think is 7 hours.
Personally I like to remind myself when I'm driving that every one taking mass transit isn't on the road with me.
Three West Coast Main Line operates speeds up to 125mph/200kmph, so very much high speed rail in US terms - faster than the Acela if I recall correctly.
I’ve seen 155 mph on the Acela, but it only cruises at those speeds for a few minutes on the NY to Boston leg. On these routes it’s not so much freight priority as track quality, segment grading, turn radius, etc. The old Metroliner trains could run nearly this fast. It’s disappointing.
This is now our primary use of Ansible as well. We install and minimally configure systems in machine images, then use cloud-init to do the remaining 1% of config at VM build time.
A Task is an individual unit of work: a container, executable, JVM app, etc.
A Task Group/Allocation is a group of Tasks that will be placed together on nodes as a unit.
A Job is a collection of Task Groups and is where you tell Nomad how to schedule the work. A "system" Job is scheduled across all Nodes somewhat like a DaemonSet. A "batch" Job can be submitted as a one-off execution, periodic (run on a schedule) or parameterized (run on demand with variables specified at runtime). A "service" Job is the "normal" scheduler where all Task Groups are treated like ReplicaSets.
Placement is determined by constraints that can be specified on each Task Group.
This is ultimately my goal. Our current state has been prototyped iteratively over a year or so. The goal is to replicate and document the current state so that we can automate it such as with Terraform.
Unfortunately, unlike, say Kubernetes, which encourages a declarative approach, learning to integrate and use Vault comes with a lot of imperative operations, none of which is stored anywhere for reuse.
The Nomad Day N experience is pretty good. I help maintain 4 Nomad clusters running approx 40k Allocations (like Pods) for work. We have basically no problems with Nomad itself, and requires pretty much no day to day intervention. Upgrades are pretty painless too. We've gone from 0.7.x to 0.12.x with these same clusters, and will be going to 1.x soon.
Yep, we run the full stack. Consul for service discovery and as the storage backend for Vault. We use Vault for config, PKI, Nomad/Consul ACL auth, and we're just starting to experiment with MSSQL dynamic credentials.
Of the three systems, Vault probably takes the most of our time and effort, and that's probably only a few hours per month. We've struggled a bit with performance at least partially because the Consul backend is shared with service discovery.
All of the VMs are built and managed with Terraform using images built with Packer+Ansible. We also use the Nomad/Consul/Vault Terraform providers to apply and manage their configurations.
We have an SRE/Platform Engineering team of 12 (and hiring) that's responsible for the overall orchestration platform additionally including Prometheus/Thanos/Grafana for metrics and ELK for logs.
