Resolving Python Requests TLS/SSL Certificate Verification Errors

Shuaib Mohammad -

You are working with the legendary python requests module and perform a routine GET request, when suddenly, this ugly message emerges out of the shadows and destroys the aesthetic on your spiffy screen.

requests.exceptions.SSLError: HTTPSConnectionPool(host='custom.host.com', port=443): Max retries exceeded with url: /endpoint?param=value (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)')))

There's a lot of underlying concepts to explain but let's quickly explore some approaches on how this can be resolved.

The quickest way is to disable certificate verification (not a secure workaround) by passing the verify=False argument to the request.

r = requests.get("https://custom.host.com/endpoint?param=value", verify=False)

Now you can go on with your life, but the following warning will appear every time you make a request.

InsecureRequestWarning: Unverified HTTPS request is being made to host 'custom.host.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings

The easy way out is to just suppress the warning.

This is done by adding the following in your python script.

import urllib3
urllib3.disable_warnings()

A little context. urllib3 is a non-standard, third-party library that is used by requests internally.

But lets say you don't want to take shortcuts. You want to fix the issue. Then, my friend, you need to obtain a full custom certificate chain and pass that path to it on to the verify argument in your request. I'll explain this bit a little later.

requests.get('https://github.com', verify='/path/to/certfile')
https://docs.python-requests.org/en/master/user/advanced/#ssl-cert-verification

But now you're needlessly adding this argument every time you make a request and it's making your eyes sore. Can't this be set globally? Well, yes it can!

Enter the REQUESTS_CA_BUNDLE environment variable.

Set this variable to point to the file system location of the certificate chain file (or a folder with such files, I'm told) and you will never have to look at those warning messages again!

How to get the full chain certificate file?

I could describe it here, but it's must more wonderfully described here. Follow the first part to obtain and download the certificate file.

Once you are in possession of a CER file, you then pass the string representation of it's absolute path to the verify parameter when making a request. This will make the SSL verification work smoothly.

[Todo: This still need some detail. Which certificate to copy? I will make a video tutorial for this]

Adding the certificate to your trusted certificate store

Configuring that for your system is another story. It will depend on your platform.

Ubuntu Linux

Convert the PEM file you obtained from the browser to a CRT file:

openssl x509 -in foo.pem -inform PEM -out foo.crt

Copy the file to the certificates directory:

sudo cp foo.crt /usr/local/share/ca-certificates/foo.crt

Make Ubuntu reference the certificate file in /etc/ca-certificates.conf

sudo dpkg-reconfigure ca-certificates

Make Ubuntu copy the actual file to /etc/ssl/certs

sudo update-ca-certificates

Some tactical topics I will add here are:

  • How to set the REQUESTS_CA_BUNDLE environment variable?

Learn More: