Step by step guide to use Ansible with Amazon EC2 – Part 1

For last few days, I am playing with Ansible as I want to automate few things. There are so many blogs and books but they don’t give you all the information about problems that you face when you start from scratch. This is my honest effort to help novice devops engineer so person does not have to go through what I went through learning Ansible.
Just a side note: I use Ubuntu 16.04 LTS as my control machine.

1) create “.aws” directory in your home directory.
e.g. my home directory is /home/polorumpus, so my “.aws” directory will be like:
/home/polorumpus/.aws

Then create file “credentials” in /home/polorumpus/.aws with following content:
——- Content starts from next line ————-
[polorumpus]
aws_access_key_id = my_aws_access_key_id
aws_secret_access_key = my_aws_secret_access_key
——- Content ends in above line —————-

Now define following environment variables:
polorumpus$ export AWS_ACCESS_KEY_ID=my_aws_access_key_id
polorumpus$ export AWS_SECRET_ACCESS_KEY=my_aws_secret_access_key
polorumpus$ export AWS_REGION=us-west-1

You can define them in some other ways:
– in file “~/.aws/credentials” as mentioned above
– in file ec2.ini as mentioned in step 3 below (note: not recommended as per Ansible comment in that file due to security risk)

2) You should be able to connect to aws instance with following command. I am in “/home/polorumpus/Ansible/playbooks/” directory.
This directory contains my pem file that I downloaded from AWS EC2.
polorumpus$ ssh -i “mykey.pem” ec2-user@ec2-52-87-199-88.compute-1.amazonaws.com

3) Now download ec2 dynamic inventory file and ec2 dynamic inventory .ini file as below.
https://raw.githubusercontent.com/ansible/ansible/devel/contrib/inventory/ec2.py and save it as ec2.py in “/home/polorumpus/Ansible/playbooks/” directory.
https://raw.githubusercontent.com/ansible/ansible/devel/contrib/inventory/ec2.ini and save it as ec2.ini in “/home/polorumpus/Ansible/playbooks/” directory.

4) EC2 Security group “For_Ansible”
Make sure you have appropriate ports open. If you want to make it secure, please read AWS documentation.

5) Try running dynamic inventory script as below:
——————————————————————————————————-
polorumpus$ ./ec2.py
{
“_meta”: {
“hostvars”: {
“52.87.199.88″: {
“ansible_ssh_host”: “52.87.199.88″,
“ec2__in_monitoring_element”: false,
“ec2_ami_launch_index”: “0″,
“ec2_architecture”: “x86_64″,
“ec2_block_devices”: {
“sda1″: “vol-12ab12e1″
},
“ec2_client_token”: “AbcdE1234567890123″,
“ec2_dns_name”: “ec2-52-87-199-88.compute-1.amazonaws.com”,
“ec2_ebs_optimized”: false,
“ec2_eventsSet”: “”,
“ec2_group_name”: “”,
“ec2_hypervisor”: “xen”,
“ec2_id”: “i-12345678″,
“ec2_image_id”: “ami-1234567a”,
“ec2_instance_profile”: “”,
“ec2_instance_type”: “t2.micro”,
“ec2_ip_address”: “52.87.199.88″,
“ec2_item”: “”,
“ec2_kernel”: “”,
“ec2_key_name”: “mykey”,
“ec2_launch_time”: “2016-09-14T14:38:23.000Z”,
“ec2_monitored”: false,
“ec2_monitoring”: “”,
“ec2_monitoring_state”: “disabled”,
“ec2_persistent”: false,
“ec2_placement”: “us-east-1a”,
“ec2_platform”: “”,
“ec2_previous_state”: “”,
“ec2_previous_state_code”: 0,
“ec2_private_dns_name”: “ip-172-30-0-15.ec2.internal”,
“ec2_private_ip_address”: “172.30.0.15″,
“ec2_public_dns_name”: “ec2-52-87-199-88.compute-1.amazonaws.com”,
“ec2_ramdisk”: “”,
“ec2_reason”: “”,
“ec2_region”: “us-east-1″,
“ec2_requester_id”: “”,
“ec2_root_device_name”: “/dev/sda1″,
“ec2_root_device_type”: “ebs”,
“ec2_security_group_ids”: “sg-56xxxx02″,
“ec2_security_group_names”: “For_Ansible”,
“ec2_sourceDestCheck”: “true”,
“ec2_spot_instance_request_id”: “”,
“ec2_state”: “running”,
“ec2_state_code”: 16,
“ec2_state_reason”: “”,
“ec2_subnet_id”: “subnet-2f123456″,
“ec2_tag_Name”: “”,
“ec2_virtualization_type”: “hvm”,
“ec2_vpc_id”: “vpc-123c5666″
}
}
},
“ami_1234567a”: [
"52.87.199.88"
],
“ec2″: [
"52.87.199.88"
],
“i-12345678″: [
"52.87.199.88"
],
“key_mykey”: [
"52.87.199.88"
],
“security_group_For_Ansible”: [
"52.87.199.88"
],
“tag_Name”: [
"52.87.199.88"
],
“type_t2_micro”: [
"52.87.199.88"
],
“us-west-1″: [
"52.87.199.88"
],
“us-west-1a”: [
"52.87.199.88"
],
“vpc_id_vpc_123c5666″: [
"52.87.199.88"
]
}
——————————————————————————————————-
6) Create file ansible.cfg in playbooks directory (/home/polorumpus/Ansible/playbooks).
[defaults]
hostfile = ./ec2.py
remote_user = ec2_user
private_key_file = /home/polorumpus/Ansible/playbooks/mykey.pem
host_key_checking = False
7) Try to ping EC2 instance that was created manually using AWS website as below:
————————— start of Ping command using ansible —————————————
polorumpus$ cd /home/polorumpus/Ansible/playbooks
polorumpus$ sudo AWS_PROFILE=polorumpus ansible -i ec2.py -m ping -u ec2-user all -vvvv
Password:
52.87.199.88 | SUCCESS => {
“changed”: false,
“invocation”: {
“module_args”: {
“data”: null
},
“module_name”: “ping”
},
“ping”: “pong”
}
——————————- End of Ping command using ansible——————————–

8) Now run RHEL7 play book for deploying NGINX:
————————————– start of running playbook command ————————
polorumpus$ sudo AWS_PROFILE=polorumpus ansible-playbook -i ec2.py -u ec2-user nginx-redhat7.yml -vvvv
Password:
Using /home/polorumpus/Ansible/playbooks/ansible.cfg as config file
Loaded callback default of type stdout, v2.0

PLAYBOOK: nginx-redhat7.yml ************************************************
1 plays in nginx-redhat7.yml

PLAY [Configure webserver with nginx] ******************************************

TASK [setup] *******************************************************************
ok: [52.87.199.88]

TASK [install nginx] ***********************************************************
ok: [52.87.199.88]

TASK [NGINX | Installing NGINX] ************************************************
changed: [52.87.199.88]

TASK [copy nginx config file] **************************************************
changed: [52.87.199.88]

TASK [copy index.html] *********************************************************
changed: [52.87.199.88]

TASK [restart nginx] ***********************************************************
changed: [52.87.199.88]

PLAY RECAP *********************************************************************
52.87.199.88 : ok=6 changed=4 unreachable=0 failed=0
————————————– End of running playbook ————————————-

9) Some useful information about NGINX directories and files on RHEL7:
[root@ip-172-30-0-15 ec2-user]# ls /etc/nginx/
conf.d fastcgi_params koi-utf koi-win mime.types modules nginx.conf scgi_params uwsgi_params win-utf

[root@ip-172-30-0-15 nginx]# cat nginx.conf
user nginx;
worker_processes 1;

error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;

log_format main ‘$remote_addr – $remote_user [$time_local] “$request” ‘
‘$status $body_bytes_sent “$http_referer” ‘
‘”$http_user_agent” “$http_x_forwarded_for”‘;

access_log /var/log/nginx/access.log main;

sendfile on;
#tcp_nopush on;

keepalive_timeout 65;

#gzip on;

include /etc/nginx/conf.d/*.conf;
}

You will find log files in /var/log/nginx as defined in nginx.conf file above.
[root@ip-172-30-0-15 nginx]# ls /var/log/nginx/
access.log error.log

10) One important thing about SELinux on RHEL7:
SELinux is a set of kernel modifications and user-space tools that have been added to various Linux distributions.
If you try running playbook and get following error:
“Unable to restart service nginx: Job for nginx.service failed because the control process exited with error code.”
You might hit policy security of SELinux.

polorumpus$ sudo AWS_PROFILE=polorumpus ansible-playbook -i ec2.py -u ec2-user nginx-redhat7.yml
Password:

PLAY [Configure webserver with nginx] ******************************************

TASK [setup] *******************************************************************
ok: [52.87.199.88]

TASK [install nginx] ***********************************************************
ok: [52.87.199.88]

TASK [NGINX | Installing NGINX] ************************************************
changed: [52.87.199.88]

TASK [copy nginx config file] **************************************************
changed: [52.87.199.88]

TASK [copy index.html] *********************************************************
changed: [52.87.199.88]

TASK [restart nginx] ***********************************************************
fatal: [52.87.199.88]: FAILED! => {“changed”: false, “failed”: true, “msg”: “Unable to restart service nginx: Job for nginx.service failed because the control process exited with error code. See \”systemctl status nginx.service\” and \”journalctl -xe\” for details.\n”}

NO MORE HOSTS LEFT *************************************************************
to retry, use: –limit @nginx-redhat7.retry

PLAY RECAP *********************************************************************
52.87.199.88 : ok=5 changed=3 unreachable=0 failed=1
———————————————————————————-

Try to find log and see what is happening. We are trying to bind NGINX service at port 60800. But because of SELinux, we get failure with “Permission denied error” as below from /var/log/nginx/error.log file.
———————————————————————————-
polorumpus$ ssh -i “mykey.pem” ec2-user@ec2-52-87-199-88.compute-1.amazonaws.com
[ec2-user@ip-172-30-0-15 ~]$ sudo su
[root@ip-172-30-0-15 ec2-user]# cd /etc/nginx/
[root@ip-172-30-0-15 nginx]# cat /var/log/nginx/error.log
2016/09/14 16:25:10 [emerg] 19129#19129: bind() to 0.0.0.0:60800 failed (13: Permission denied)
2016/09/14 16:26:52 [emerg] 19174#19174: bind() to 0.0.0.0:60800 failed (13: Permission denied)
2016/09/14 16:28:53 [emerg] 19185#19185: open() “/var/run/nginx.pid” failed (13: Permission denied)
2016/09/14 16:40:41 [emerg] 19666#19666: bind() to 0.0.0.0:60800 failed (13: Permission denied)
2016/09/14 16:41:28 [emerg] 20020#20020: bind() to 0.0.0.0:60800 failed (13: Permission denied)
———————————————————————————-

Because RHEL7 is using targeted policy by default, you can run following command to find which ports are available for use as http ports:
[root@ip-172-30-0-15 nginx]# semanage port -l | grep http_port_t
http_port_t tcp 80, 81, 443, 488, 8008, 8009, 8443, 9000
pegasus_http_port_t tcp 5988

As you can see from the output above with SELinux in enforcing mode http is only allowed to bind to the listed ports. The solution is to add the ports you want to bind on to the list
[root@ip-172-30-0-15 nginx]# semanage port -a -t http_port_t -p tcp 60800
[root@ip-172-30-0-15 nginx]# semanage port -l | grep http_port_t
http_port_t tcp 60800, 80, 81, 443, 488, 8008, 8009, 8443, 9000
pegasus_http_port_t tcp 5988

For more information about SELinux and RHEL7, read RHEL documentation at following link:

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/Deployment_Guide/sec-sel-enable-disable.html

Also read following:

http://serverfault.com/questions/566317/nginx-no-permission-to-bind-port-8090-but-it-binds-to-80-and-8080

11) some useful commands for RHEL7 to start & stop NGINX
Start an NGINX service:
[root@ip-172-30-0-15 nginx]# systemctl start nginx.service

Stop an NGINX service:
[root@ip-172-30-0-15 nginx]# systemctl stop nginx.service

12) To remove NGINX package:

[root@ip-172-30-0-15 nginx]# yum remove nginx
Loaded plugins: amazon-id, rhui-lb, search-disabled-repos
Resolving Dependencies
–> Running transaction check
—> Package nginx.x86_64 1:1.10.1-1.el7.ngx will be erased
–> Finished Dependency Resolution

Dependencies Resolved

=======================================================================================
Package Arch Version Repository Size
=======================================================================================
Removing:
nginx x86_64 1:1.10.1-1.el7.ngx @nginx 2.1 M

Transaction Summary
=======================================================================================
Remove 1 Package

Installed size: 2.1 M
Is this ok [y/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Erasing : 1:nginx-1.10.1-1.el7.ngx.x86_64 1/1
warning: /etc/nginx/conf.d/default.conf saved as /etc/nginx/conf.d/default.conf.rpmsave
Verifying : 1:nginx-1.10.1-1.el7.ngx.x86_64 1/1

Removed:
nginx.x86_64 1:1.10.1-1.el7.ngx

Complete!

 

Comments are closed.