Digestive Issues and DevOps: My Journey Building a Cluster with Hashicorp Tools

There has been a lot of stuff going on in the last week. I have been in less than optimal productivity zones physically and mentally this week. I made a poor decision of trying out a food place on Monday evening that I believe caused digestive issues and the impact of that continues till today. Moral of the story, have extremely high standards for what you choose to eat. The impact of one poor eating choice can trickle in to days of lost productivity.

Having said that, the main thing the consumed my time and propelled my learning was using Hashicorp tools to build my own cluster. I learned a ton about the tools and DevOps thinking in general. I also learned some better security practices from a Linux stand point and thought of new ways I can build and organize a cluster.

I setup a 2 node cluster and go the hang of standing it with nomad and consul working. I was also able to setup traefik to work as an edge router for services available within the cluster. There were some challenges in exposing the Web UI's exposed by nomad and consul (that run as binaries directly on the host) via traefik but I was able to sort that out for nomad. Consul was still a mystery. I was also pleased with having setup constraints so that traefik would only run on the primary node. This way I can configure DNS to route everything to single traefik instance rather than having a spin multiple traefik instances on each node. It took a while to figure out how map ports correctly, but these are just syntactic things that one would pick up as they get used to new tool.

The key hesitation I have at this point to start using nomad is the effort required to translate docker-compose files in to a nomad job. This impacts the time it will take me to provision a service. Particularly when most open-source or free tools available come with a docker-compose file that I can quickly run.

I do like the idea of using consul as a key value store for configuration management and vault as a secret management solution within the context of docker-compose. This will make things more secure and easier to manage. I will explore these ideas to see how vault and consul can be run in a docker-compose context.

For now on my cluster building ambitions, I think I'll revert to a simpler solution. I will run an additional isolated node with my secondary big application and configure both nodes to share a directory using NFS. I will run individual traefik instances on both which will use the same configuration file shared over NFS. This will avoid me having to also get into shared volume management with docker and nomad for now. In the meanwhile, I'll practice using the Hashistack on my single large node internally to develop best practices.

Lessons on Learning Well

Perhaps one of the biggest takeaways from the exercise was getting a better understanding of how I learn and what are better ways for me to learn.

The best way to learn anything is by doing. I think I spent way too much time reading blogs and the experiences of others that kept me from getting my hand dirty and actually producing results. Of course, there are many pros of having a look at what others did. It is great for discovery of new ideas and concepts. For example, the idea of NFS shares was not something I thought of, even though its a simple idea to execute. So reading about it in another blog made me aware of it and entertain it as a possibility for my solution.