20200204 - CfgMgmtCamp 2020 - Lessons learned from testing over 200,000 lines of Infrastructure Code, Lev Goncharov

by Thierry de Pauw on


Lessons learned from testing over 200,000 lines of Infrastructure Code, Lev Goncharov @ultralisc

Many people pretend they have IaC but in fact they have ...
Infrastructure as Bash History

  • Reproducible: yes
  • Versioning: yes
  • Logging: yes
  • Sanity: no

IaC has all of them


Bash script


migrated Bash monolith to Ansible

  • @JDebois_: It's interesting to me how developers get slapped nonstop with conforming to the SOLID principles, yet a lot of sysadmins don't even know what it is #cfgmgmtcamp

Single Responsibility: as small pieces of roles

Open-Closed: ability to deploy to both VirtualBox and real prod infra

Liskov Substitution:

JDK -> Java
OpenJDK -> Java
Java -> Ansible playbook
Wildfly -> Ansible playbook

Interface Segregation: in the beginning, we tried to put everything in one playbook

Postgres, Wildfly, Mariadb -> one Anible playbook
=> roles

Dependency Inversion: split infrastructure into simple layers and create contracts between the layers
=> don't create a monolith infrastructure @ultralisc

Bus factor

=> Pair DevOpsing to share knowledge

it didn't really work for us: maybe because we were working remotely

But Mentoring did work for us

  • @bruvik: Using pair programming principles and mentoring to share knowledge about operations in a team #cfgmgmtcamp @ultralisc https://t.co/2T8ZALJ2hE

Code Review

  • @toshywoshy: RT @bruvik: When implementing code review, people started to argue about silly things #cfgmgmtcamp https://t.co/2K9DI3jmJS

=> introduce code style

Green Build Master

  • make infra changes on the branch :/
  • when green on the branch integrate in master

Lessons Learned:

  • build IaC from simple bricks
  • adopt XP practices

=> tests

IaC Testing Pyramid

from bottom to top

  • static analysis
    shellcheck (bash), ansible lint, pylint, rubocop, tflint

  • unit tests: Shunit2, Pytest, Rspec
    Ansible? Chef? SaltStack?

build infra from simple bricks: Common, OpenJDK, Wildfly

provision VM's (Virtualbox) or Docker containers with a simple brick and run tests against it

Ansible: Testinfra, Molecule (testing roles)
Chef: TestKitchen
Terraform: TerraTest

  • integration tests: unit tests are only for one brick in your infra combine several bricks, provision it and test if it integrates

- Lint playbooks
- Lint roles
- Check syntax
- Unit test roles
- Integration test playbook

  • e2e: Jenkins master + VM includes Jenkins slave + docker containers started by docker-compose + docker container with tests

one Ansible playbook to run the e2e tests: creates also the VM

Transcript of the talk: www.goncharov.xyz/iac