Migrating my old blog

As part of migrating my older blog posts, this one was posted on July 27th, 2014.

Introduction

I know it has been a while since I put up my last post, but school was kept me busy. I wrote this one in a hurry and ask that you excuse any spelling and grammar mistakes. In particular, I've been taking CSC 373 - Algorithm Design, Analysis, and Complexity and have had several assignments to work on. In my free time, I've been working on my own Ansible script. Ansible is a tool used to deploy and update applications in an easy to use language, using SSH, with no agents to install on remote systems. The specifications for Ansible are all written in YAML, making them well structured and generally easy to follow. Back at Wave, Nathan wrote An Ansible primer blog post on the Wave Engineering blog. Here, I'll be taking you through my Ansible script as a practical example of what you could use Ansible for, outside of setting up remote servers.

Unfortunately, having recently had a hard drive failure on my laptop. I had to manually setup my machine just the way I like it. When setting up, I realized the colossal waste of time it was to lookup the ppa's I needed, the Sager specific driver I'd installed for my keyboard backlight and trying to recall what I used most. This time I'd done the smart thing and simply documented all my steps. There were a few missing things there involving using dconf, gconftool-2 and gsettings and a couple of steps I'd not yet written down. We'd been using Ansible at Wave and my last attempt to learn it met with confusion, ten VMs running in the background and an episode of The Sopranos. I had tried to follow Nathan's example (simplifying it along the way) as well as a bunch of other tutorials relating to Django. They all met with failure to get a successfully provisioned VM. I'd been making small contributions at work to our Ansible scripts in hopes to get the practice I needed for when I decided to try again. That's when Alex Tucker suggested I take my setup instructions and give it another go at Ansible. Now that you know how I got started, I'll get into the project.

The File Structure & Content

This is mostly self explanatory, I'll go into more detail about each section.

setup.yml                 # master playbook
HOSTS                     # HOSTS file which defines where the script will run. This points to localhost.
requirements.txt          # the requierments for running this playbook, including ansible
run.sh                    # one command script for install requirements, cloning the repo and provisioning.

roles/
    common/               # this hierarchy represents a "role"
        tasks/            #
            *.yml         #  <-- tasks files i'll talk about later
        files/            #
            *.*           #  <-- files for use with the copy resource
        vars/             #
            main.yml      #  <-- variables associated with this role

The HOSTS file

This file lets Ansible know where the machines are located, in this case all we'll need is localhost.

[local]
127.0.0.1

The Requirements File

Some on the Ansible modules have requirements, the Ansible docs provide them on each modules page.

python-apt==0.9.3.5
ansible==1.7.1

The Master Playbook

This is also pretty simple. Here we can define what we run where, here I'm just telling it to run the scripts, in my common role, on the local machine.

- name: a playbook to setup my local machine with my basic customizations
  hosts: local
  connection: local
  roles:
    - common

Common Roles: Variables main.yml

Here we can define variables which we can use with {{ variable name }} syntax. I use some of them to conditionally run tasks, to do them for a particular user and for downloading applications which are not available via ppa.

---
username: joseph
sager_laptop: false
vagrant_url: https://dl.bintray.com/mitchellh/vagrant/vagrant_1.6.3_x86_64.deb
virtualbox_url: http://download.virtualbox.org/virtualbox/4.3.14/virtualbox-4.3_4.3.14-95030~Ubuntu~raring_amd64.deb
github_username: JBKahn
bumblebee: false
uname_r: 3.13.0-32-generic

Common Roles: Files

For example, I place the autostart files in there so that I can copy them into the correct directory so applications will launch on boot. I also have configuration files for applications and my terminal.

[Desktop Entry]
Name=Variety
Comment=Variety Wallpaper Changer
Icon=/opt/extras.ubuntu.com/variety/share/variety/media/variety.svg
Exec=/opt/extras.ubuntu.com/variety/bin/variety
Terminal=false
Type=Application
X-GNOME-Autostart-Delay=20

Common Tasks

Here's where most of the coding is and I'll provide five examples, the rest you can checkout in the repository.

1) main.yml

---
  - include: pre.yml
  - include: sublime.yml
  - include: chrome.yml
  - include: numix.yml
  - include: plank.yml
  - include: keepass2.yml
  - include: laptop.yml
    when: sager_laptop == true
  - include: variety.yml
  - include: font.yml
  - include: base16.yml
  - include: work.yml
  - include: scm_breeze.yml
  - include: vim.yml
  - include: other_requirements.yml
  - include: dropbox.yml
  - include: bash_customizations.yml
  - include: bumblebee.yml
    when: bumblebee == true

'main.yml' is the file that the setup.yml common role calls, I just use it to conditionally call the other task files I use.

2) base16.yml

---
- name: base16 - checkout repo
  git:
    repo: https://github.com/chriskempson/base16-gnome-terminal.git
    dest: "/home/{{ username }}/setup/base16-gnome"

- name: base16 - copy monokai-dark
  copy:
    src: "/home/{{ username }}/setup/base16-gnome/base16-monokai.dark.sh"
    dest: "/home/{{ username }}/base16-monokai.dark.sh"
    mode: 0755

- name: base16 - install monokai dark
  command: "/home/{{ username }}/base16-monokai.dark.sh"

- name: base16 - set system font to source code pro
  gconftool-2:
    key: /apps/gnome-terminal/profiles/base-16-monokai-dark/font
    string: "Source Code Pro Semi-Bold 12"

- name: base16 - dont use default system font in terminal
  gconftool-2:
    key: /apps/gnome-terminal/profiles/base-16-monokai-dark/use_system_font
    bool: false

- name: base16 - set default terminal profile
  gconftool-2:
    key: /apps/gnome-terminal/global/default_profile
    string: "base-16-monokai-dark"

- name: base-16 - get base16-shell so that colors show up correctly in vim
  git:
    repo: https://github.com/chriskempson/base16-shell.git
    dest: "/home/{{ username }}/.config/base16-shell"

Here I make use of the git, file and command modules to checkout the amazing base-16 color scheme for my terminal, set it as the default and set the font (which I already setup in another task).

3) vim.yml

---
- name: horse vim - download vim
  apt:
    name: vim
    update_cache: yes
  sudo: yes

- name: horse vim - make sure bundle directory exists
  file:
    path: "/home/{{ username }}/.vim/bundle"
    state: directory
    mode: 0777
  sudo: yes

- name: horse vim - checkout vundle repo
  git:
    repo: https://github.com/gmarik/vundle.vim.git
    dest: "/home/{{ username }}/.vim/bundle/vundle"

- name: horse vim - checkout repo
  git:
    repo: https://github.com/JBKahn/horse.vim.git
    dest: "/home/{{ username }}/horse.vim"

- name: horse vim - copy .vimrc file
  file:
    src: "/home/{{ username }}/horse.vim/.vimrc"
    dest: "/home/{{ username }}/.vimrc"
    mode: 0755
    state: link

- name: horse vim - copy .vimrc.bundles file
  file:
    src: "/home/{{ username }}/horse.vim/.vimrc.bundles"
    dest: "/home/{{ username }}/.vimrc.bundles"
    mode: 0755
    state: link

- name: horse vim - make sure vim colors directory exists
  file:
    path: "/home/{{ username }}/.vim/colors"
    state: directory
    mode: 0777
  sudo: yes

- name: horse vim - copy color to config to avoid error
  copy:
    src: base16-monokai.vim
    dest: "/home/{{ username }}/.vim/colors/base16-monokai.vim"
    mode: 0755

- name: horse vim - run bundle install
  command: vim +BundleInstall +qall

- name: horse vim - add NeoVim ppa
  apt_repository:
    repo: 'ppa:rudenko/neovim'
    state: present
    update_cache: yes
  sudo: yes

- name: horse vim - install neovim
  apt:
    name: neovim
    update_cache: yes
  sudo: yes

- name: horse vim - symlink nvimrc
  file:
    src: "/home/{{ username }}/horse.vim/.vimrc"
    dest: "/home/{{ username }}/.nvimrc"
    mode: 0755
    state: link

- name: horse vim - symlink nvim
  file:
    src: "/home/{{ username }}/.vim"
    dest: "/home/{{ username }}/.nvim"
    mode: 0755
    state: link

Another part of my setup is to put Vim and all of the important packages on my machine. Unfortunately, when I had tried to install the packages it seemed that missing my color scheme file prevented me from downloading it through Vundle. Using Ansible's copy module I'm taking the current version (rather than doing a git pull for a more up to date one) to temporarily place the file there and have it replaced by running the setup.

4) work.yml

---
- name: download vagrant .deb package
  get_url:
    url: "{{ vagrant_url }}"
    dest: "/home/{{ username }}/setup/vagrant.deb"
    mode: 0755

- name: get currently installed vagrant version
  command: "vagrant --version"
  sudo: yes
  register: result
  ignore_errors: True

- name: install vagrant
  apt:
    deb: "/home/{{ username }}/setup/vagrant.deb"
  sudo: yes
  when: result.rc !=0 or result.stderr.find('A later version is already installed') != -1

- name: download virtualbox .deb package
  get_url:
    url: "{{ virtualbox_url }}"
    dest: "/home/{{ username }}/setup/virtualbox.deb"
    mode: 0755

- name: install virtualbox
  apt:
    deb: "/home/{{ username }}/setup/virtualbox.deb"
  sudo: yes

- name: install wave dependencies
  apt:
    name: "{{ item }}"
    state: installed
    update_cache: yes
  with_items:
    - python-pip
    - curl
    - nodejs
    - nfs-kernel-server
    - nfs-common
    - rpcbind
    - tmux
    - npm
    - libjpeg-dev
    - libfreetype6-dev
    - zlib1g-dev
    - libpng12-dev
    - python-dev
    - libpq-dev
    - python-dev
  sudo: yes

- name: get currently installed node version
  command: "node --version"
  sudo: yes
  register: result
  ignore_errors: True
  tags: nodejs

- name: checkout node
  git:
    repo: "git://github.com/ry/node.git"
    dest: "/setup/node"
    force: no
    accept_hostkey: yes
  when: result.rc !=0
  tags: nodejs
  sudo: yes

- name: configure
  sudo: yes
  command: "./configure"
  args:
    chdir: /setup/node
  when: result.rc !=0
  tags: nodejs

- name: make
  command: "make"
  sudo: yes
  args:
    chdir: /setup/node
  when: result.rc !=0
  tags: nodejs

- name: make install
  command: "make install"
  sudo: yes
  args:
    chdir: /setup/node
  when: result.rc !=0
  tags: nodejs

- name: install global nodejs packages
  npm:
    name: "{{ item.name }}"
    state: present
    global: yes
    version: "{{ item.version|default('') }}"
  with_items:
    - {name: 'bower'}
    - {name: 'coffee-script', version: '1.7.1'}
    - {name: 'grunt-cli'}
    - {name: 'less'}
    - {name: 'jshint'}
    - {name: 'coffee-jshint'}
  sudo: yes

- name: install tmuxp
  pip:
    name: tmuxp
  sudo: yes

- name: copy tmuxp config
  copy:
    src: .tmux.conf
    dest: "/home/{{ username }}/.tmux.conf"
    mode: 0755

- name: install ipython
  pip:
    name: ipython
  sudo: yes

There's a fair amount of overlap between things I use for my personal projects and those which I need for work. In here I've put the more development focused requirements making use of some of its wide variety of modules for installing packages through pip, .deb files and apt.

5) laptop.yml

---
- name: screen dimming - alter grub file
  lineinfile:
    dest: /etc/default/grub
    regexp: "^GRUB_CMDLINE_LINUX_DEFAULT="
    line: 'GRUB_CMDLINE_LINUX_DEFAULT="quiet splash video.use_native_backlight=1"'
  sudo: yes
  register: grubfile
  tags: sager

- name: screen dimming - update grub
  command: sudo update-grub
  sudo: yes
  when: grubfile|changed
  tags: sager

- name: keyboard colors - requirements
  apt:
    pkg: "{{ item }}"
    state: installed
  with_items:
    - build-essential
    - linux-source
  sudo: yes
  tags: sager

- name: keyboard colors - Make sure setup directory exists
  file:
    path: "/home/{{ username }}/setup"
    state: directory
  tags: sager

- name: keyboard colors - checkout clevo-wmi-code
  git:
    repo: git://git.code.sf.net/p/clevo-wmi/code
    dest: "/home/{{ username }}/setup/clevo-wmi-code"
    accept_hostkey: yes
  tags: sager

- name: keyboard colors - build clevo-wmi
  command: make
  args:
    chdir: "/home/{{ username }}/setup/clevo-wmi-code"
  tags: sager

- name: keyboard colors - link clevo-wmi to kernel check
  command: cat /etc/modules
  sudo: yes
  register: running_modules
  tags: sager

- name: keyboard colors - link clevo-wmi to kernel
  command: "sudo insmod /home/{{ username }}/setup/clevo-wmi-code/clevo_wmi.ko"
  sudo: yes
  # when: running_modules.stdout.find('clevo_wmi') == -1
  tags: sager
  ignore_errors: True

- name: keyboard colors - copy to kernel drivers
  copy:
    src: "/home/{{ username }}/setup/clevo-wmi-code/clevo_wmi.ko"
    dest: "/lib/modules/{{ uname_r }}/kernel/drivers/platform/x86/"
  sudo: yes
  tags: sager

- name: keyboard colors - handle dependencies
  command: sudo depmod -a
  sudo: yes
  tags: sager

- name: keyboard colors- add to etc/modules
  lineinfile:
    dest: /etc/modules
    line: clevo_wmi
  sudo: yes
  tags: sager

- name: keyboard colors - copy kbbl script
  copy:
    src: keyboard-color.sh
    dest: "/home/{{ username }}/keyboard-color.sh"
    mode: 0755
  tags: sager

One of the disadvantages of using linux can be that some things are hard to make work, although Ubuntu helps with a lot of it. For a long time I was unable to dim my backlight and my keyboard backlight was always on and always blue. Through the use of a small settings tweak, a custom driver and a custom script I wrote for changing the keyboard backlight, I'm not able to do all of these things. The task will only run contingent on the sager_laptop variable being true. One of the truly great modules in called 'lineinfile' and it does is ensure that a line exists in a file. That sounds pretty simple but it offers the ability to use a regular expression to identify the line (so that if it does not exist but it is meant to replace another line, it will) and the ability to create the file if it does not exist (configurable).

The script

sudo apt-get install python-setuptools

sudo easy_install pip

sudo apt-get install aptitude
sudo apt-get install git
sudo apt-get install python-dev libxml2-dev libxslt-dev

cd ~
mkdir -p setup
cd setup
git clone https://github.com/JBKahn/provisioning-local.git
cd provisioning-local

sudo pip install -r requirements.txt

echo -e "please enter your username, followed by [ENTER]" && read PROVISIONING_USER
sudo sed -i "s/^username: .*/username: $PROVISIONING_USER/" roles/common/vars/main.yml

echo -e "please enter your github username, followed by [ENTER]" && read PROVISIONING_GITHUB_USERNAME
sudo sed -i "s/^github_username: .*/github_username: $PROVISIONING_GITHUB_USERNAME/" roles/common/vars/main.yml

sudo sed -i "s/^uname_r: .*/uname_r: `uname -r`/" roles/common/vars/main.yml

ansible-playbook setup.yml -i HOSTS --ask-sudo-pass  --module-path ./ansible_modules

# currently unable to use ansible due to EULA that I can't seem to stub using debconf
sudo apt-get install steam

dropbox start -i &
/opt/extras.ubuntu.com/variety/bin/variety $
plank &

source ~/.bashrc
exit 0

By using:

bash wget -qO- https://github.com/JBKahn/provisioning-local/raw/master/run.sh | sudo bash

you can have this script download and run and have the machine setup in no time. It handles the requirements, and starts up three of the applications after the provisioning is completed.

The Output

joseph@Batcave-Ubuntu:~/dev/provisioning-local$ sudo rm -fr ../../setup/base16-gnome/
[sudo] password for joseph:
joseph@Batcave-Ubuntu:~/dev/provisioning-local$ ansible-playbook setup.yml -i HOSTS --ask-sudo-pass
sudo password:

PLAY [a playbook to setup my local machine with my basic customizations] ******

GATHERING FACTS ***************************************************************
ok: [127.0.0.1]

TASK: [common | base16 - checkout repo] ***************************************
changed: [127.0.0.1]

TASK: [common | set file permissions] *****************************************
changed: [127.0.0.1]

TASK: [common | base16 - install monokai dark] ********************************
changed: [127.0.0.1]

TASK: [common | base16 - set system font to source code pro] ******************
changed: [127.0.0.1]

TASK: [common | base16 - dont use default system font in terminal] ************
changed: [127.0.0.1]

TASK: [common | base16 - set default terminal profile] ************************
changed: [127.0.0.1]

PLAY RECAP ********************************************************************
127.0.0.1                  : ok=7    changed=6    unreachable=0    failed=0

Here after I run it, it will tell me how many of them at ran (changed) or did not require any more work (ok). There are some tasks (i.e. commands) which will always be 'changed' unless they are conditionally executed or a module is written to explain to Ansible how to check and see if something has changed.

That's It

That's all you need to get this up and running and provisioning your own machine. I hope to be able to write a blog post about using it with Vagrant and what the differences are. For now, I'll leave you with a few relevant links: