Development Workflow

I haven't ever done software in a professional setting with best practices and all so I don't have a mature development and deployment workflow. I can quickly hack something together but to package and deploy it in a production setting is something I haven't mastered yet. I have been spending a lot of time configuring environments for my little python web apps and docker has helped a lot. Now I want to take it to the next level. Here's how I envision it.

Develop either locally or on a remote host. I prefer not to have to configure development environments on different machines. I would much rather connect to a remote development environment most of the time and have everything done their. Coder.com's code-server and sail projects seem very good for my needs but I've been having trouble setting up sail in my environment.

I've also done a fair amount of research on what is the right flow. Some interesting tools that could serve my needs include GitLab with CI/CD, CapRover, Kubernetes etc. But they seem to be overkill given my current bandwidth.

The current workflow I'm tinkering with is as follows:

Development environment:

Python installed
Conda installed [Optional]
Git installed

Development Workflow

Linux with miniconda installed
conda create -n template python=3.8
conda activate template
Install packages using pip install package not python -m pip install package because the later caused installation to host python install rather than the conda environment python install
conda env export > environment.yml pip freeze > requirements.txt
Create template of docker-compose.yml file with the right names and routing configuration (to be automated as a template)
Push the repository to github
Pull the repository on server
Run build docker build . -t base-template
Run docker-compose up -d

Note:

During the performance of above, I found the package I was trying to run would not work with conda. Basically if there is a python command line tool that needs to be invoked by CMD in the Dockerfile, it doesn't run because docker doesn't find it in the PATH. Probably something to do with how conda creates the environment within docker. Hence, I had to move from a conda docker base image to a python docker base image.

Append 2020-03-15

I have setup GitLab and now migrating my projects to git. This also required re-familiarity with best practice git workflows. I found the following tutorial on Git Collaboration helpful. The rest of the sections are also a good crisp read for quickly reviewing the concepts.