Hello Guys,
Here is another post on a scenario some of you working with telcos, banks and other organizations are familiar with. There is a server somewhere which offers a service usually available over VPN, sometimes not, that you need to connect to. That server has a self signed certificate and you need to deal with that. To simulate this, we will use a server on AWS, an Elastic IP (static public IP) as our server node and an Ubuntu vagrant box as the client node.
I could have used 2 vagrant boxes but I am using a slightly different scenario just to show that the practice is not for private networks only.I have opted for one public box (server) and one local box (client).
We will sign the same certificate twice. One for a day (which I did yesterday) another one for 10. The idea is that by signing the first certificate for a day by now it should expire. So we will end up with an expired and valid self signed certificate.
We have generated our static IP and it’s going to be 52.70.113.80. That will be our servername on the certificate is also going to be same. We will destroy this one we are done so don’t go try to hit this, I have locked it down to my IP as well.
Signing Certificate
This is really not an article for how to self signed certificate but I guess a little memory refresh on how to do it won’t be bad. I am using an old CompanyACA.crt ROOT CA I have already used in this article. We will use it to sign our certificates.
Generating Certificate Signing Request (CSR)
With the command below we will generate the key file to use with our certificate.
|
1 2 |
# $ openssl genrsa -out 52.70.113.80.key 2048 |
Now let’s issue the CSR.
|
1 2 |
# openssl req -new -key 52.70.113.80.key -out 52.70.113.80.csr |
Once the CSR ready we will use our ROOT certificat to sign it for a day.
|
1 2 |
# openssl x509 -req -days 1 -in 52.70.113.80.csr -CA ~/Dropbox/personal/certificatefortutorial/CompanyACA.crt -CAkey ~/Dropbox/personal/certificatefortutorial/CompanyACA.key -set_serial 01 -out 52.70.113.80.crt |
|
1 2 |
# openssl x509 -req -days 10 -in 52.70.113.80.csr -CA ~/Dropbox/personal/certificatefortutorial/CompanyACA.crt -CAkey ~/Dropbox/personal/certificatefortutorial/CompanyACA.key -set_serial 055442343 -out 52.70.113.80_10days.crt |
In the command above you will notice, I used a different serial number for the certificate. If I had kept it, browsers would have complained about it. The issue is that we will have same certificate signed by same Authority(ourselves) with 2 different validity days. To prevent that we need different serials for both for our browsers and components to be happy with.
Preparing Cetificate bundle for Nginx
This is also outside the purpose of the article, buuuuut for newbies I thought I would do some exception to show how to use the newly issued certificates on a server. A typical bundle certificate file used in nginx is of the following format:
|
1 2 3 |
<Certificate itself> <Intemediate Certificate> <Root Certificate> |
Let’s create our bundle file. I will only show one but we need to to it for 2 created certificates. We don’t have any intermediate certificate so this will be actual certificate and its root CompanyACA.crt.
|
1 2 |
# $cat ~/Dropbox/personal/certificatefortutorial/CompanyACA.crt >> 52.70.113.80_10days.crt |
Setting up SSL on our server
After installing nginx on our ubuntu xenial box quickly with : sudo apt-get install nginx . We also installed php with sudo apt-get install php-fpm to simulate our server.We then changed the default configuration in /etc/nginx/sites-available/default file to look like the following
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
# server { listen 80 default_server; listen [::]:80 default_server; listen 443 ssl; root /var/www/html; index index.php index.html index.htm index.nginx-debian.html; #I interchange the certificates here to use the 1 or 10 days certificate # ssl_certificate /srv/ssl/52.70.113.80.crt; ssl_certificate /srv/ssl/52.70.113.80_10days.crt; ssl_certificate_key /srv/ssl/52.70.113.80.key; server_name 52.70.113.80; access_log /var/log/nginx/nginx.vhost.access.log; error_log /var/log/nginx/nginx.vhost.error.log; location / { try_files $uri $uri/ =404; } location ~ \.php$ { include snippets/fastcgi-php.conf; # fastcgi_pass 127.0.0.1:9000; fastcgi_pass unix:/run/php/php7.0-fpm.sock; } } |
I am using a fake API build with php to take care of the call. So this is the file that is used on the server we will be calling from our vagrant box.
|
1 2 3 4 5 6 7 8 9 10 11 12 |
# <?php $date = date("d-m-Y h:i:sa"); $arrayData = json_decode(file_get_contents('php://input'), true); $arrayData['date'] = $date; header('Content-Type: application/json'); echo json_encode($arrayData); ?> |
Using 10 days Certificates
Making call to API: CURL
Our server is ready to respond to some requests.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
# vagrant@vagrant:~$ curl -XPOST -H "Content-Type: application/json" -d '{"name":"Joseph","sname":"Djomeda"}' https://52.70.113.80/test.php curl: (60) server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none More details here: http://curl.haxx.se/docs/sslcerts.html curl performs SSL certificate verification by default, using a "bundle" of Certificate Authority (CA) public keys (CA certs). If the default bundle file isn't adequate, you can specify an alternate file using the --cacert option. If this HTTPS server uses a certificate signed by a CA represented in the bundle, the certificate verification probably failed due to a problem with the certificate (it might be expired, or the name might not match the domain name in the URL). If you'd like to turn off curl's verification of the certificate, use the -k (or --insecure) option. |
Using the certificates in curl call
Doing this, I have 3 types of certificates in the vagrant box:
- CompanyACA.crt
- 52.70.113.80_10days.crt
- 52.70.113.80_10days_bundle.crt
Using Root certificate in the call: CompanyACA.crt
|
1 2 3 |
# curl --cacert CompanyACA.crt -XPOST -H "Content-Type: application/json" -d '{"name":"Joseph","sname":"Djomeda"}' https://52.70.113.80/test.php {"name":"Joseph","sname":"Djomeda","date":"22-10-2017 03:13:17pm"} |
Using certificate without its root in the call: 52.70.113.80_10days.crt
|
1 2 3 |
# curl --cacert 52.70.113.80_10days.crt -XPOST -H "Content-Type: application/json" -d '{"name":"Joseph","sname":"Djomeda"}' https://52.70.113.80/test.php {"name":"Joseph","sname":"Djomeda","date":"22-10-2017 03:12:56pm"} |
Using certificate with its root in the call: 52.70.113.80_10days_bundle.crt
|
1 2 3 |
# curl --cacert 52.70.113.80_10days_bundle.crt -XPOST -H "Content-Type: application/json" -d '{"name":"Joseph","sname":"Djomeda"}' https://52.70.113.80/test.php {"name":"Joseph","sname":"Djomeda","date":"22-10-2017 03:14:01pm"} |
Making call to API: HTTPIE
In case this is new to you, you could have a look at httpie documentation.
Making call with httpie
|
1 2 3 4 |
# $ echo '{"name":"Joseph","sname":"Djomeda"}' | http POST https://52.70.113.80/test.php http: error: SSLError: HTTPSConnectionPool(host='52.70.113.80', port=443): Max retries exceeded with url: /test.php (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)'),)) while doing POST request to URL: https://52.70.113.80/test.php |
Making httpie call with certificate without root
|
1 2 3 4 |
# echo '{"name":"Joseph","sname":"Djomeda"}' | http --verif=52.70.113.80_10days.crt POST https://52.70.113.80/test.php http: error: SSLError: HTTPSConnectionPool(host='52.70.113.80', port=443): Max retries exceeded with url: /test.php (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)'),)) while doing POST request to URL: https://52.70.113.80/test.php |
Making httpie call with certificate with root
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
# $ echo '{"name":"Joseph","sname":"Djomeda"}' | http --verif=52.70.113.80_10days_bundle.crt POST https://52.70.113.80/test.php HTTP/1.1 200 OK Connection: keep-alive Content-Type: application/json Date: Sun, 22 Oct 2017 17:56:00 GMT Server: nginx/1.10.3 (Ubuntu) Transfer-Encoding: chunked { "date": "22-10-2017 05:56:00pm", "name": "Joseph", "sname": "Djomeda" } |
Making httpie call with root certificate only
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
# $ echo '{"name":"Joseph","sname":"Djomeda"}' | http --verif=CompanyACA.crt POST https://52.70.113.80/test.php HTTP/1.1 200 OK Connection: keep-alive Content-Type: application/json Date: Sun, 22 Oct 2017 17:57:52 GMT Server: nginx/1.10.3 (Ubuntu) Transfer-Encoding: chunked { "date": "22-10-2017 05:57:52pm", "name": "Joseph", "sname": "Djomeda" } |
From this experience it looks like the ROOT certificate is the one needed for httpie.
Trusting the Certificate
To add the certificate to the trust store, we need to create a folder such as codingpains in /usr/share/ca-certificates/. We need to then copy the certificate to that folder and run the command dpkg-reconfigure ca-certificates.
|
1 2 3 4 5 6 |
# $ sudo mkdir /usr/share/ca-certificates/codingpains $ sudo copy 52.70.113.80_10days.crt /usr/share/ca-certificates/codingpains $ sudo copy 52.70.113.80_10days_bundle.crt /usr/share/ca-certificates/codingpains $ sudo copy CompanyACA.crt /usr/share/ca-certificates/codingpains $ sudo dpkg-reconfigure ca-certificates # select ask options |
Importing only 52.70.113.80_10days.crt
We imported the server certificate first and we will make calls with curl and httpie
|
1 2 3 |
# $ curl -XPOST -H "Content-Type: application/json" -d '{"name":"Joseph","sname":"Djomeda"}' https://52.70.113.80/test.php {"name":"Joseph","sname":"Djomeda","date":"22-10-2017 06:24:42pm"} |
|
1 2 3 4 |
# $ echo '{"name":"Joseph","sname":"Djomeda"}' | http POST https://52.70.113.80/test.php http: error: SSLError: HTTPSConnectionPool(host='52.70.113.80', port=443): Max retries exceeded with url: /test.php (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)'),)) while doing POST request to URL: https://52.70.113.80/test.php |
Importing only 52.70.113.80_10days_bundle.crt
- Curl call worked
- httpie call failed for httpie installed via pip
- httppie call worked for httpie installed via apt-get with warning
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
echo '{"name":"Joseph","sname":"Djomeda"}' | http POST https://52.70.113.80/test.php /usr/lib/python2.7/dist-packages/urllib3/connection.py:266: SubjectAltNameWarning: Certificate for 52.70.113.80 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.) SubjectAltNameWarning HTTP/1.1 200 OK Connection: keep-alive Content-Type: application/json Date: Mon, 23 Oct 2017 09:05:01 GMT Server: nginx/1.10.3 (Ubuntu) Transfer-Encoding: chunked { "date": "23-10-2017 09:05:01am", "name": "Joseph", "sname": "Djomeda" } |
Importing CompanyACA.crt
- Curl call worked
- httpie call failed for httpie installed via pip
- httppie call worked for httpie installed via apt-get with warning
Using 1 day, now expired Certificate
So to cut the long story short, all calls made to the server have failed because the certificate expired. Direct referencing of the ROOT CA and importing it into the OS trusted certificates stored all failed.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 |
# vagrant@vagrant:~$ curl -XPOST --cacert ~/52.70.113.80.crt -H "Content-Type: application/json" -d '{"name":"Joseph","sname":"Djomeda"}' https://52.70.113.80/test.php curl: (60) server certificate verification failed. CAfile: /home/vagrant/52.70.113.80.crt CRLfile: none More details here: http://curl.haxx.se/docs/sslcerts.html curl performs SSL certificate verification by default, using a "bundle" of Certificate Authority (CA) public keys (CA certs). If the default bundle file isn't adequate, you can specify an alternate file using the --cacert option. If this HTTPS server uses a certificate signed by a CA represented in the bundle, the certificate verification probably failed due to a problem with the certificate (it might be expired, or the name might not match the domain name in the URL). If you'd like to turn off curl's verification of the certificate, use the -k (or --insecure) option. vagrant@vagrant:~$ curl -XPOST --cacert ~/52.70.113.80_bundle.crt -H "Content-Type: application/json" -d '{"name":"Joseph","sname":"Djomeda"}' https://52.70.113.80/test.php curl: (60) server certificate verification failed. CAfile: /home/vagrant/52.70.113.80_bundle.crt CRLfile: none More details here: http://curl.haxx.se/docs/sslcerts.html curl performs SSL certificate verification by default, using a "bundle" of Certificate Authority (CA) public keys (CA certs). If the default bundle file isn't adequate, you can specify an alternate file using the --cacert option. If this HTTPS server uses a certificate signed by a CA represented in the bundle, the certificate verification probably failed due to a problem with the certificate (it might be expired, or the name might not match the domain name in the URL). If you'd like to turn off curl's verification of the certificate, use the -k (or --insecure) option. vagrant@vagrant:~$ curl -XPOST --cacert ~/CompanyACA.crt -H "Content-Type: application/json" -d '{"name":"Joseph","sname":"Djomeda"}' https://52.70.113.80/test.php curl: (60) server certificate verification failed. CAfile: /home/vagrant/CompanyACA.crt CRLfile: none More details here: http://curl.haxx.se/docs/sslcerts.html curl performs SSL certificate verification by default, using a "bundle" of Certificate Authority (CA) public keys (CA certs). If the default bundle file isn't adequate, you can specify an alternate file using the --cacert option. If this HTTPS server uses a certificate signed by a CA represented in the bundle, the certificate verification probably failed due to a problem with the certificate (it might be expired, or the name might not match the domain name in the URL). If you'd like to turn off curl's verification of the certificate, use the -k (or --insecure) option. vagrant@vagrant:~$ echo '{"name":"Joseph","sname":"Djomeda"}' | http --verify=~/52.70.113.80.crt POST https://52.70.113.80/test.php http: error: SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590) vagrant@vagrant:~$ echo '{"name":"Joseph","sname":"Djomeda"}' | http --verify=~/52.70.113.80_bundle.crt POST https://52.70.113.80/test.php http: error: SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590) vagrant@vagrant:~$ echo '{"name":"Joseph","sname":"Djomeda"}' | http --verify=~/CompanyACA.crt POST https://52.70.113.80/test.php http: error: SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590) |
Diagnose with php
From the result above it’s clear that there is something wrong. With the php code below, let’s use php curl to understand why the request is being rejected.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
# <?php $data = array("name"=>"Joseph","sname"=>"Djomeda"); $json = json_encode($data); $ch = curl_init(); curl_setopt_array($ch, array( CURLOPT_SSL_VERIFYPEER => true, CURLOPT_SSL_VERIFYHOST => 2, CURLOPT_VERBOSE => true, CURLOPT_CAINFO => '/home/vagrant/52.70.113.80_bundle.crt', )); // set URL and other appropriate options curl_setopt($ch, CURLOPT_URL, "https://52.70.113.80/test.php"); curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: application/json')); curl_setopt($ch, CURLOPT_HEADER, 0); curl_setopt($ch, CURLOPT_POST, 1); curl_setopt($ch, CURLOPT_POSTFIELDS, $json); curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 0); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); $ch_result = curl_exec($ch); echo "Result = ".$ch_result; curl_close($ch); ?> |
This actually assumes you installed php-curl module.
|
1 2 3 4 5 6 7 8 9 10 11 |
# vagrant@vagrant:~$ php phpcurl.php * Trying 52.70.113.80... * Connected to 52.70.113.80 (52.70.113.80) port 443 (#0) * ALPN, offering http/1.1 * Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH * successfully set certificate verify locations: * CAfile: /home/vagrant/52.70.113.80_bundle.crt CApath: /etc/ssl/certs * SSL certificate problem: certificate has expired * Closing connection 0 |
Basically it doesn’t work because the certificate has expired. The only thing to do in this case is to get the server to update the certificate or choose not to verify it. That can be done with –insecure on curl and –verify-no with httpie.
By not verifying it, it doesn’t mean the call is over plain http. It simply mean you are telling your app/program not to go all out finding who signed the certificate, when it’s going to expired or whether the name on the certificate tallies with the IP or servername of the server serving it.
I hope this has been useful to someone consistently facing issues like this. And oh, if you are using java, java store is actually different as well. I am yet to try making a java app look up certificate in system trusted store.