Devops Ansible Tool

What is Ansible?

 Ansible is a free “configuration management” tool.
 Use cases:

•  It’s what you use when you’re tired of copying commands by hand from UNIX tutorials into terminal windows. (Or)
• When you want to set up identical servers that are actually identical.

Why it is required?

Problem # 1:

• When you want to execute series of commands on multiple servers.
• When we try to execute these commands manually by hand. While doing , if you miss any commands which may causes the different issues. 

 Problem # 2:

• Another common problem with managing multiple servers is keeping their configurations identical (or similar)
• When you set up a server (say server1), you tweak the configuration by hand and when you set up another server (server2) you do the same. But then there’s this setting you need to change on server2, which you then forget to change on server1. A different ecosystem was born.

                                               Simple Solution is use Ansible.
Ansible which does the automation or orchestration which makes life of the admin easy. 

Why Ansible is preferred than Puppet, Chef or Salt?

         Ansible uses just SSH ( no overhead of Agents or Agent less)

• The main difference between the other tools and Anisble, is that underneath Ansible is just SSH. Chef and Puppet both have dependencies that must be installed on the server before you can use them, Ansible does not. 
• It runs on your machine and uses SSH to connect to the servers and run the required commands.
• Ansible is agentless uses the PUSH technology. Whereas Puppet, chef needs to have agents to be installed on servers and uses the PULL technology.

Good nice article:[ Source :https://wiredcraft.com/blog/getting-started-with-ansible-in-5-minutes/]




From the above figure,

• Chef or Puppet work by installing an agent on the hosts they manage. This agent is pulling changes from a master host, using their own channel (usually not SSH).  
• Whereas Ansible on the other hand is simply using SSH to push changes from wherever it runs (a server or your own laptop).

Why not just use Bash script rather than Ansible? 

•  Ansible has an edge over Bash scripts because of its simplicity.
•  Ansible just uses a list of tasks to run in YAML format. 
•  Ansible also comes with idempotency out of the box.   That means you can run the same operation numerous times, and the output will remain consistent ( i.e. it won't do something twice unless you ask it to). You can write Bash scripts this way, but it requires quite a bit more overhead

Basic terms of Ansible:

PLAYBOOK:  

• In ansible, a definition of how a server should look  is called a playbook. 
• In a playbook, you define what actions ansible should take to get the server in the state you want. Only what you define gets done. (or)
• Ansible playbook is just a series of tasks that are performed in sequence, and a simple sequence fits entirely in one file

 YAML : YAML stands for "YAML Ain't Markup Language"

• (rhymes with camel) is a human-readable data serialization language. It is commonly used for configuration files [ Source from Wiki]
• Playbook are written in YAML language.
• YAML is not programming language and it is easy to write the .yml files.   e.g: first-playbook.yml
• YAML files should end in .yaml whenever possible.
• YAML is case sensitive.
• YAML does not allow the use of tabs. Spaces are used instead as tabs are not universally supported.

YAML Basic Data Types

•  Mappings (hashes / dictionaries), 
•   Sequences (arrays / lists), and  
•   Scalars (strings / numbers). 

Lists:
All members of a list are lines beginning at the same indentation level starting with a "- " (a dash and a space)

---
# A list of tasty fruits
fruits:
    - Apple
    - Orange
    - Strawberry
    - Mango
...

 Mappings (hashes / dictionaries)

 A dictionary is represented in a simple key: value form (the colon must be followed by a space):

# An employee record
martin:
    name: Martin D'vloper
    job: Developer
    skill: Elite

 More complicated data structures are possible, such as lists of dictionaries, dictionaries whose values are lists or a mix of both:

# Employee records
-  martin:
    name: Martin D'vloper
    job: Developer
    skills:
      - python
      - perl
      - pascal
-  tabitha:
    name: Tabitha Bitumen
    job: Developer
    skills:
      - lisp
      - fortran
      - erlang

Dictionaries and lists can also be represented in an abbreviated form if you really want to:

---
martin: {name: Martin D'vloper, job: Developer, skill: Elite}
fruits: ['Apple', 'Orange', 'Strawberry', 'Mango'] 

Scalars (strings / numbers):
They are the strings and numbers that make up the data on the page. A scalar could be a boolean property, like Yes, integer (number) such as 5, or a string of text, like a sentence or the title of your website. Scalars are often called variables in programming

integer: 25
string: "25"
float: 25.0
boolean: Yes 

property, like Yes, integer (number) such as 5, or a string of text, like a sentence or the title of your website.

Task  [Source: http://abregman.com/2015/12/25/ansible-write-and-run-your-first-playbook/]

A task  is simply the use of one of Ansible modules. Module implements specific functionality.  

For example, installing package would be task since it will require us to use the ‘apt’ module. 

There many modules, so task can be running service, fetching files, adding user and many more modules waiting for you to explore. 

- name: Ensure apache2 is installed

  apt: name=apache2 state=present

In the above example

• we use apt module to install package named ‘apache2’. The state is the action we are using on this package.so ‘present’ tells Ansible to make sure apache2 installed in the system. 
• There are additional states, as ‘latest’ which means ‘make sure latest package is installed’.

What is play in Ansible?

Play is a collection of tasks running against one or more hosts. It includes one or more task.

What is playbook?

Playbook composed of one or more plays.

Now, let us learn how to use the ANSIBLE in practical application.

Step 1: How to install Ansible in Ubuntu Linux machine?

Step 2: How to setup the hosts in /etc/hosts/ansible? 

Step 3: How to write the playbook?

Step 4: How to run the playbook using ansible commands?

How to install Ansible in Ubuntu Linux machine?

To run ansible, you need Python 2.6 or higher on the machine you run ansible from. Most current operating systems have this by default. 

sudo apt-add-repository -y ppa:ansible/ansible

sudo apt-get update

sudo apt-get install -y ansible

The easiest way to install these is to use pip. Let’s first install pip for your distribution.

    sudo apt-get install python-pip python-dev

    sudo pip install ansible

How to setup the hosts in /etc/ansible/hosts? 

Next step is Setup inventory or Manage servers or Setup Hosts in /etc/ansible/hosts file.

If the above file doesn't exists then create it using the below commands

   (sudo) mkdir /etc/ansible

   (sudo) touch /etc/ansible/hosts

else  Edit the /etc/ansible/hosts

Let’s say you have two nodes, named hostA and hostB. You can  simply add these two lines to ‘/etc/ansible/hosts

   hostA

   hostB

   [Group_Webservers]

   192.168.1.0

   192.168.2.0

   

   [my_group_of_hosts]

   hostA

   hostB

Define a group called as ‘Group_Webservers’. A group can be used in a playbook to run the playbook against a number of hosts. We’ve  added two hosts to the group (Group_Webservers). 

Source from : [http://abregman.com/2015/12/25/ansible-write-and-run-your-first-playbook/]

How to write the playbook? 

Now learn about writing playbook. Let's start play with the cool example.

So imagine our environment consists of two hosts:  hostA and hostB

We want to have two plays:

First play: one simple task – create file in /tmp named ‘apple_file’.

Second play: two tasks  – task to add user named ‘muni’ with bash as default shell, and another task to install the latest  ‘python’ package.

We know what we want to run and on what hosts we want to run it. Let’s start with the first play:

---

- hosts: hostB
  tasks:
      - name: Create file
        file: path=/tmp/apple_file state=touch

– hosts: hostB tells Ansible on which nodes to run the task in the above playbook. Here we mentioned as hostB, on which these task runs.

tasks: is a list of tasks, each using one of the available modules of Ansible that you would like to execute on remote hosts. In our case, hostB.

Our list contain only one task that uses ‘file’ module. ‘file’ module allows us to create/remove files and directories or modify their attributes. In the example above we simply create empty file. the ‘path=/tmp/apple_file’ used by the module to to create ‘apple_file’ file in /tmp dir. by “state=touch” we are actually telling Ansible we would like to ensure such file exists. If it exists, ansible will do nothing, but if it doesn’t, it will create it.

Let's extend the playbook,

---
- hosts: hostB
  tasks:
      - name: Create file
        file: path=/tmp/apple_file state=touch

- hosts: my_group_of_hosts
  sudo: yes
  tasks:
      - name: Create user
        user: name=muni shell=/bin/bash

      - name: Install python
        apt: name=python state=latest

   

We now have two plays. How do we know that? We can see end of tasks list of our firs play and you can notice a start of new play by the line that starts with “-hosts …’.

There are two tasks – First one is for creating new user. We used the ‘user’ module for that. We wanted him to have bash as default shell? that is the ‘shell=/bin/bash’.

Second task is about installing new package in your system. It using the ‘apt’ module with two options – name  of the package to be installed and state=latest. In the first example, we used ‘state=present’ to simply have the package installed in our system. ‘state=latest’ makes sure that if there is newer package available, it will be installed.

There is the ‘sudo: yes”. While for the first play you didn’t have to be root, because every user can create files in ‘/tmp’, for the second play you must be. Not every user can create new users in the system. So we ran the second play with sudo to make sure the play will not fail due to lack of permissions.

Below, you can find more examples about the Playbooks.

How to run the playbook using ansible commands

The command for running playbooks is pretty straightforward:

ansible-playbook <playbook name>

PLAY [hostB] ******************************************************************

GATHERING FACTS ***************************************************************
ok: [hostB]

TASK: [Create file] ***********************************************************
changed: [hostB]

PLAY [my_group_of_hosts] ***************************************************************

GATHERING FACTS ***************************************************************
ok: [hostA]
ok: [hostB]

TASK: [Create user] ***********************************************************
changed: [hostA]
changed: [hostB]

TASK: [Install python] **********************************************************
ok: [hostA]
ok: [hostB]

PLAY RECAP ********************************************************************
hostA                      : ok=5    changed=2    unreachable=0    failed=0
hostB                     : ok=3    changed=1    unreachable=0    failed=0

   

‘GATHERING FACTS’  is a step in which Ansible gather all kind of information about your remote system (memory, os distribution, ip addresses, etc). To see what information gathered in this step you can use:                                         

                                               ansible -m setup <hostname>

What is Regular Expressions? How to Use the regular Expressions in python?

Regular expression represents as "re"  [  Source from https://pymotw.com/2/re/) ]

What is Regular Expression? (re)

• Regular Expressions are generally described as regex, regexp. 
• These are mainly used for matching the text patterns.  
• A large number of parsing problems are easier to solve with a regular expression than by    creating a special-purpose lexer and parser. 
• Expressions can include literal text matching, repetition, pattern-composition, branching, and other sophisticated rules.  
• Unix Tools such as SED, grep, awk uses regular expressions internally for finding the particular pattern.

How to use Regular expressions in python?

Step 1: Import the Regular Expression module as  -  import re

Step 2: Design the Regular Expression to be used for your application.

Step 3: Use the appropriate Regular expression method to parse the text or string. 

What is RAW python strings?

When writing regular expression in Python, it is recommended that you use raw strings instead of regular Python strings. Raw strings begin with a special prefix (r) and signal Python not to interpret backslashes and special meta characters in the string, allowing you to pass them through directly to the regular expression engine.
This means that a pattern like "\n\w" will not be interpreted and can be written as r"\n\w" instead of  "\\n\\w" as in other languages, which is much easier to read.

Regular expression Methods:

    Generally there are three types of methods which are been used more in the regular expressions.

1.  re.match()
2.  re.search()
3.  re.findall()

re.match() - Matches at Beginning

It will match to the Beginning pattern of string.

Usage:
re.match(pattern, string, flags=0)
      If zero or more characters at the beginning of string match the regular expression pattern, return a corresponding MatchObject instance. Return None if the string does not match the pattern; note that this is different from a zero-length match.

    Note that even in MULTILINE mode, re.match() will only match at the beginning of the string and not at the beginning of each line.

Examples:
 re.match(r'c', "abcdef")              >> No match 
 re.match(r'cat', 'dog cat dog')     >> No Match
 re.match(r'dog', 'dog cat dog')    >> Match 

From the above examples, it is evident that re.match() method tried to find the given pattern at beginning of the string. If it matches, then it returns the Match object else NONE.  

re.search() - Matches at Anywhere

The search() method is similar to match(), but search() doesn’t restrict us to only finding matches at the beginning of the string, so searching for ‘cat’ in below example string finds a match: 

Examples #1:
re.search(r'cat', 'dog cat dog')     >> Match

Usage:
 re.search(pattern, string, flags=0)    Scan through string looking for the first location where the regular expression pattern produces a match, and return a corresponding MatchObject instance. Return None if no position in the string matches the pattern; note that this is different from finding a zero-length match at some point in the string.

Tip:
If you want to locate a match anywhere in string, use search() instead  match(). 


Examples #2:
 re.search(r'c', "abcdef")     >> Match

Regular expressions beginning with '^' can be used with search() to restrict the match at the beginning of the string:

Examples #3:
 re.search("^c", "abcdef")  >> No Match
 re.search("^a", "abcdef")  >>  Match

How about Multi line Matches? 

In MULTI LINE mode match() only matches at the beginning of the string. It means,  re.match() will only match at the beginning of the string and not at the beginning of each line.

Whereas using search() with a regular expression beginning with '^' will match at the beginning of each line.

re.match('X', 'A\nB\nX', re.MULTILINE)    # No match
re.search('^X', 'A\nB\nX', re.MULTILINE)   # Match

re.findall() - All Matching Objects

re.findall()  Shall return the list of all matching patterns in the string.

>>> re.findall(r'dog', 'dog cat dog')   >> Returns the list of strings that matches to the pattern
  ['dog', 'dog']

>>> re.findall(r'cat', 'dog cat dog')
  ['cat']

Note: 

re.search() and re.match() return the single instances of literal text strings.

Whereas re.findall()  method returns all of the sub strings of the input that match the pattern without overlapping 

re.compile() : Compile the Regular Expression Pattern

Compile a regular expression pattern into a regular expression object, which can be used for matching using its match() and search() methods

pattern="test"
string="testpatterntest"
prog = re.compile(pattern)
result = prog.match(string)


or
re.match(pattern, string)

Using re.compile() shall make  reuse of regular expression more efficient, when the expression  used several times in a single program.

Python program Examples

re.search() :
The basic rules of regular expression search for a pattern within a string are:

• The search proceeds through the string from start to end, stopping at the first match found
• All of the pattern must be matched, but not all of the string
• If match = re.search(pat, str) is successful, match is not None and in particular match.group() is the matching text 

Program #1    : In this example, match.group() is used to print the matching text

import re
patterns = ["this", "that"]
text= "Does this text match this string"
for pattern in patterns:
  match= re.search(pattern, text)
  if match is not None:
    print( "Found Match :", match.group())
  else:
    print('no match:', pattern )

Output:

  Found Match : this
  no match 

Program #2    : In this example, re.search() returns the match object if the pattern is matched or found.
From matchObject , you can get StartIndex, endIndex, string, pattern. Check out for more in program.

import re
patterns = ["this", "that"]
text= "Does this text match this string"
for pattern in patterns:
  matchObject= re.search(pattern, text)
  if matchObject is not None:
    startIndex = matchObject.start()
    endIndex =   matchObject.end()
    print('Found "%s" in "%s" from %d to %d ("%s")' %
          (matchObject.re.pattern, matchObject.string, startIndex,
           endIndex, text[startIndex:endIndex]))
  else:
    print('no match:', pattern )

 

Output:

Found "this" in "Does this text match this string" from 5 to 9 ("this")
no match: that 


Program #3 
findall() is probably the single most powerful function in the re module. Above we used re.search() to find the first match for a pattern. findall() finds *all* the matches and returns them as a list of strings, with each string representing one match.   

import re
testPattern ="abc"str="abcbbbabcbbbbabc"
listStr = re.findall('abc', str)
print (listStr)

Output:

['abc', 'abc', 'abc'] 

Program #4 

findall() - use "for" loop to display the strings.

import re
testPattern ="abc"str="abcbbbabcbbbbabc"
listStr = re.findall('abc', str)
for match in listStr:
   print ("The Match string :", match) 

Output: 

The Match string : abc
The Match string : abc
The Match string : abc

Program #5

findall() - With Files 

For files, you may be in the habit of writing a loop to iterate over the
lines of the file, and you could then call findall() on each line. 
Instead, let findall() do the iteration for you -- much better! Just 
feed the whole file text into findall() and let it return a list of all 
the matches in a single step (recall that f.read() returns the whole 
text of a file in a single string):  

import re
testPattern="ab"

# Open file
fp = open('testfile.txt', 'r')

# Feed the file text into findall(); 

# it returns a list of all the found strings

 
listStr = re.findall(testPattern, fp.read())
print(listStr)

Output:

['ab', 'ab', 'ab', 'ab', 'ab', 'ab', 'ab', 'ab', 'ab', 'ab', 'ab'] 

Program #6

 Use finditer() rather than findall() 

finditer() returns an iterator that produces Match instances 

instead of the strings returned by findall().

import re
testPattern="muni" 

text="munixxxmungggmunixxxmuniaaamuni" 

for matchIter in re.finditer(testPattern, text):
    startIndex= matchIter.start()
    endIndex= matchIter.end()

    print("StartIndex:", startIndex, "EndIndex:", endIndex, matchIter.group())

Output:

StartIndex: 0 EndIndex: 4   muni
StartIndex: 13 EndIndex: 17 muni
StartIndex: 20 EndIndex: 24 muni
StartIndex: 27 EndIndex: 31 muni  

Program #7 - re.compile

import re
regex_compiled_object = re.compile("this")
text= "Does this text match the pattern?"
if regex_compiled_object.search(text):
    print("found a match!")
else:
    print ("no match") 

Output: 

found a match!