Docker – AWS S3 PHP Uploads – [curl] 6: Could not resolve host – DNS issues

Last night when I was trying to upload a new profile pic in our app, S3 uploads suddenly stopped working. Other S3 API methods are working fine except for the upload or PUT method. I have done many changes on my VM for the past couple of days that could have affected our app but it turns out to be just a DNS issue.

Background

I run our app inside a Docker container inside a VM. When I attempt to upload a file via AWS S3 (using the AWS SDK for PHP), I got this error:

[curl] 6: Could not resolve host: <my bucket>.s3.amazonaws.com; Unknown error [url] https://<my bucket>.s3.amazonaws.com/profile-pic-123.jpg

I tried to curl the url inside docker and it work just fine. Therefore, I suspect the issue is something else since the container can connect to the AWS hostname.

I just switched from bridged networking to NAT (VirtualBox) and assume that the issue was due to the NAT thing. However, when I switched back to bridged networking, the issue remains. Therefore, the issue is on the VM itself or the Docker container.

I also tried to disable my VPN connection but it turns out that VPN is not the issue as well. This leads me to conclusion that the issue could be a DNS issue.

DNS Settings

Another background, I used to use Google’s DNS servers. However, I noticed that Google hates my ISP that I have trouble resolving hostnames so I switched to OpenDNS. I used OpenDNS from my router settings, wifi adapter settings and even to the Docker daemon.

I remembered that Docker used Google’s DNS by default so I looked at the /etc/resolv.conf inside the container and found out that it used Google’s indeed. My settings have been reset! Well, I’m regularly rebuilding my Docker images due to upstream updates.

Then I checked the VM’s Linux guest DNS settings and found out that it still used OpenDNS. The last thing I checked was the file /etc/default/docker which now contains empty settings instead of my custom DNS setting. Beside the file was a file called /etc/default/docker.orig containing the original setting that I used to use.

Slackware users can relate to this. Remember the prompt after a slackpkg upgrade-all that asks how to handle new config files? I just choose the overwrite option and so it begins. Yes, I upgraded Docker and I wiped out the original setting without knowing or maybe I’m aware but just didn’t care. So I just restored the original settings instead.

Docker’s DNS settings

Here is the original content of my /etc/default/docker.

## Set defaults used by the docker daemon
## These are flags passed after `docker -d`
DOCKER_OPTS="--dns 208.67.222.222 --dns 208.67.220.220"

I restored the file. Did it fix it? No.

I tried manually editing the file /etc/resolv.conf inside the container but that’s silly and didn’t work anyway.

I restarted the container but still didn’t fix it.

Worried that I need to rebuild the Docker image, I read the docs about Docker’s DNS settings and it talks about the daemon. Therefore I conclude that I don’t need to rebuilt the image. I suspect that I need to restart the Docker daemon and at the same time, recreate the container to make it work.

So I stopped all the containers and restarted the docker deamon. After the daemon restart, I deleted the app’s container and recreated it and then start it. I then login into the container and looked at /etc/resolv.conf. Yes, it is now updated.

I also tested the upload and it is now working correctly. Finally!

This entry was posted in Web Development and tagged , , , , , , . Bookmark the permalink.

Related Posts

Leave a Reply

Your email address will not be published. Required fields are marked *