20200204 - CfgMgmtCamp 2020 - Lessons learned from testing over 200,000 lines of Infrastructure Code, Lev Goncharov

Lessons learned from testing over 200,000 lines of Infrastructure Code, Lev Goncharov @ultralisc

Many people pretend they have IaC but in fact they have ...
Infrastructure as Bash History

• Reproducible: yes
• Versioning: yes
• Logging: yes
• Sanity: no

IaC has all of them

DRY

Bash script

SOLID

migrated Bash monolith to Ansible

• @JDebois_: It's interesting to me how developers get slapped nonstop with conforming to the SOLID principles, yet a lot of sysadmins don't even know what it is #cfgmgmtcamp

Single Responsibility: as small pieces of roles

Open-Closed: ability to deploy to both VirtualBox and real prod infra

Liskov Substitution:

JDK -> Java
OpenJDK -> Java
Java -> Ansible playbook
Wildfly -> Ansible playbook

Interface Segregation: in the beginning, we tried to put everything in one playbook

Postgres, Wildfly, Mariadb -> one Anible playbook
=> roles

Dependency Inversion: split infrastructure into simple layers and create contracts between the layers
=> don't create a monolith infrastructure @ultralisc

Bus factor

=> Pair DevOpsing to share knowledge

it didn't really work for us: maybe because we were working remotely

But Mentoring did work for us

• @bruvik: Using pair programming principles and mentoring to share knowledge about operations in a team #cfgmgmtcamp @ultralisc https://t.co/2T8ZALJ2hE

Code Review

• @toshywoshy: RT @bruvik: When implementing code review, people started to argue about silly things #cfgmgmtcamp https://t.co/2K9DI3jmJS

=> introduce code style

Green Build Master

• make infra changes on the branch :/
• when green on the branch integrate in master

Lessons Learned:

• build IaC from simple bricks
• adopt XP practices

=> tests

IaC Testing Pyramid

from bottom to top

• static analysis
  shellcheck (bash), ansible lint, pylint, rubocop, tflint

• unit tests: Shunit2, Pytest, Rspec
  Ansible? Chef? SaltStack?

build infra from simple bricks: Common, OpenJDK, Wildfly

provision VM's (Virtualbox) or Docker containers with a simple brick and run tests against it

Ansible: Testinfra, Molecule (testing roles)
Chef: TestKitchen
Terraform: TerraTest

• integration tests: unit tests are only for one brick in your infra combine several bricks, provision it and test if it integrates

Pipeline:
- Lint playbooks
- Lint roles
- Check syntax
- Unit test roles
- Integration test playbook

• e2e: Jenkins master + VM includes Jenkins slave + docker containers started by docker-compose + docker container with tests

one Ansible playbook to run the e2e tests: creates also the VM

• @ivnovi: @ultralisc talking about infrastructure as code testing pyramid at #cfgmgmtcamp https://t.co/aaCOZWRbb2

Transcript of the talk: www.goncharov.xyz/iac