August 12, 2015

Applying Vagrant in Software Development

Hello, I am a freelance developer, trainer and DevOps enthusiast. I think Vagrant is a highly underestimated tool in software development and also deeply out shined by Docker. I want to place some spotlight on Vagrant in this article. In my opinion it cut costs in a highly efficient manner and nicely plays along with Docker. It's a virtual machine management console tool implemented and maintained by Hashicorp Inc. and I want to give some insight in how to apply Vagrant and make the best out of it in a developers daily work.

VAGRANT WILL CHANGE HOW YOU WORK

Currently, the developers machine is the holy kingdom in delivering the features and solutions of your company. In front there is the developers IDE and in the back the server, services, queue system and the database. In a multi project setup with branches there are now multiple servers, services and queue systems and databases or just one server and database. Either the developer switches services between projects or redeploys any time he switches the project's context.
In any way it seems cumbersome and it is cumbersome and especially it's a waste of time. Although starting a complex application server running on a developer's machine can be easy as typing:

$ vagrant init mycompany/websphere-7
$ vagrant up

SETUP, CONFIGURE, WORK

The installation and configuration complexity now plays out for each team member and just multiplies. Now let's put that effort inside vagrant boxes and distribute them throughout the team! Actually Backend servers and services should be installed and maintained inside vagrant boxes. I have seen this approach fail in teams. The reasons are manifold:

* The installation process is often not reproduce able and not transparent for team members. There is a lack of trust and team members tend to fall back to old procedures.

* The time to setup a virtual machine in a non automatic manner is very costly. It hurts if it breaks, corrupts or does not boot anymore. Next time team members will fall back to old procedures.

* The virtual machine is maintained and supported by someone that has a tight schedule. Team members can not wait for updates and fall back to old procedures.

* Discordance: Some team members like virtual machines done with Virtualbox but others would never work without VMWare.

VMWARE, VIRTUALBOX, DOCKER, HYPER-V

Vagrant is a tool where the installation process itself can be scripted or in Vagrant slang "Provisioned". Vagrant is a management console for creating, starting and stopping  "Providers" which can be popular virtual machine technologies like Virtualbox, VMWare, Docker and many more. Furthermore, Boxes can be maintained as composites, e.g. managing servers and databases as local clusters. What's left is a powerful host machine but that should be granted for a serious developer.

Realizing a Vagrant box for Websphere, JBoss, MySQL, DB2, PostgreSQL, MongoDB is no wasted effort. It can be easily scripted by console commands inside the Vagrantfile and put under source control and version management. It also supports popular scripting languages like Puppet, Chef, Salt and Ansible. This can be done by team members or operation specialists. The installation is transparent and highly editable.

The Box itself is handled like the software you produce. It is usually enough to simply provision the box by your team. You can also pay external specialists in order to do the provisioning for you and then use review tool in order to accept the installation procedure. Remember, you pay for transparent and readable Vagrantfiles and installation scripts.

BUILD, DOWNLOAD, WORK

Vagrant can be easily maintained as an on-premise solution. Vagrantfiles are written by team members or operation specialists and then checked-in inside the local version control system. Afterwards the boxes are build by Continuous Integration Systems for each checkin and updated using nightly builds. The latest version will be deployed inside an Artifact Repository for immediate download for each developer. The developer simply initializes a template Vagrantfile for his local machine and pulls the required box:

$ vagrant init myimages/mongodb 
$ vagrant up

Boxes can be versioned for any development scenario, projects and customers:

$ vagrant init myimages/mongodb https://myrespository/myimages/mongodb/versions/2.4.14
$ vagrant up

YOUR ENVIRONMENT, YOUR SOLUTION

The whole stack can be implemented inside your company, behind your firewall and in accordance with your operational guidelines. The tools are free to use and open source. My personal preference for an on-premise solution is the following technology stack:

  • Gitlab as the Git Source/Version Control System.
  • Jenkins as the CI- and Build-System with Build Slaves for Vagrant boxes.
  • Artifactory the Artifact Repository for Vagrant Boxes.
  • Virtualbox as the free Open Source VM-Technology.

SUMMARY

Vagrant boxes contain the developers backend systems: Application Servers, Queues and Databases. They are downloaded and fired up in about 30 seconds by any developer. They can be put together to composites, simulate system scenarios and enable system tests. The can be used for evaluation and bug hunting on a developers or operational team machine. Boxes can help to speed up setup work, building, maintaining systems and all in all enable proper DevOps. 

You can be sure it's a big help in keeping heads up in the complex world of IT!

Thanks for reading!

References

July 17, 2015

Jenkins - Getting Jenkins Cloud Ready - Part 1

Jenkins is a continuous integration tool for software development. You can define jobs, integration and deployment and schedule their execution. It's one of the most important tools in software development. The developer does not have to wait for a feedback but will be informed with detailed infos on how his new feature has fared inside the environments.

Jenkins tend to get bulky in time. Projects have to set up their unique demands, e.g. software versions and installations (Java 6-8, Tomcat, JBoss, Websphere).  As a result, a master-slave setup can ease the complexity of setting up Jenkins but setting up the master-slave environment itself remains.

The perfect solution is the cloud where you can setup any complex environment with containers and setup templates. Especially the Google Compute Engine looks interesting as you pay for use and you do not have any setup costs.

Therefore, I aim to define a setup template for cloud deployment and then explode my setup into the cloud. Which means setting up a secure Jenkins master with a scalable amount of build slaves. This will enable teams to setup a build environment per project rather than per team or company. The setup can be deployed and removed on demand and enable indefinite staging environments for software development.

The first step is building the appropriate cloud container using Docker. Docker is a container framework where you setup applications inside a linux container and being able to fire it up in any cloud environment. Jenkins is by its history an on-premise solution and setup is usually done by hand. A Jenkins instance is not by default able for deploy-and-use inside the cloud. Appropriate means:

  • The container must include a ready-to-use Jenkins.
  • The container must have a default security setup.
  • The container must accept slave connections.
  • The container must support backup & restore.

In my Github repository and project blacklabelops/jenkins I implemented a cloud-ready Jenkins master inside a Docker container. The project has elaborate documentation, feel free to roam and try it out. Managing Jenkins inside Docker required the following steps:

  1. Setting up a Dockerfile that installs Jenkins, Java and required software packages.
  2. Extending the Dockerfile by configuring Jenkins to use my pre-defined Docker Volume for persisting its settings and using it as the default workfolder.
  3. As a result I could write bash scripts in order to backup & restore the Docker Volume. Now I was able to resume my immutable container in any state.
  4. Define environment variables to be able to configure my container for specific use cases. Being able to control security setup, ports for slave connection and plugin installations.
  5. Environment variables are interpreted at startup and they use the docker-entrypoint script for applying their intend. The script itself is hooked inside the Dockerfile.
  6. Implementing several Groovy Scripts to execute the settings inside Jenkins, e.g. settings for security and the java virtual machine. Blacklabelops Groovy Scripts.
The container is build and deployed by the public Docker registry Docker Hub and checked by Continuous Integration by Circle-CI. Docker Hub and Circle-CI are free for public Github Repositories. You can always check my repository for the current state of my project as build badges provide a good overview of my implementation state.

Dockerhub Build Badge: 
Circle-CI Build Badge:


Dockerhub Info Badge:

This is now my cloud-ready Jenkins! I have already over 600 downloads, several clones of my repository and first positive feedback. Feel free to try! I would be happy to get some stars and comments on my Github and Dockerhub repositories!

The next article will be about deploying this container inside the Google Container Engine and I will have a look at the costs of its execution. Also I will have an article about securing Jenkins on the network layer with https. Last article will be about deploying and scale slaves in addition to Jenkins. Stay tuned!

Interesting? I also have a Jira cloud container! blacklabelops/jira

June 18, 2015

Github - Automatic VM Creation with Packer and Virtualbox


You always wanted to spin up VM's from latest available ISO images automatically? You want full control of what's installed on your VM? You like CentOS? Then this is the perfect place for you!

June 13, 2015

Release and Version Vagrant Boxes with Atlas

Vagrant is an amazing flexible management tool for VM (Virtual Machines). You can easily build, run and deploy Virtualbox, VMWare, AmazonWS and more VMs. Atlas is a platform for uploading and managing Vagrant boxes. I can build and upload new boxes and download my boxes to any environment and deploy them when needed.

I have several vagrant boxes under configuration and adjust them when necessary. New boxes will be rolled out and each box is checked daily by a nightly build. Old version will be still available and replaced seamlessly. In this tutorial I want to show how I implemented a release cycle for Atlas.

The release cycle is described in this Tutorial.

Project and code can be found in the Github repository blacklabelops/packercentos.

June 12, 2015

BlackLabelOps Automatic VM Creation

I finished my first Packer project! It creates a Virtualbox CentOS 7.1 Image for Vagrant on the fly!

Repositories:
I use this project as a base box for:
  • Creating a Vagrant base box for my examples.
  • Creating a Docker base image for my examples.
  • Testing versioning and releasing of VMs on Github and Atlas.
  • Testing continuous integration with Packer.

April 27, 2015

SAFe isn't Safe

May be you have heard about SAFe, may be you haven't. Does not really matter:

"SAFe tells corporate dinosaurs that those little mammals downstairs are nothing to worry about. Agile, on the other hand, thinks mammals are the answer."

The Article: Issues with SAFe

Coding Tutorials ;)

February 28, 2015

Cross Functional Teams in Munich

A good article by Stylight on their approach on cross functional teams!

February 11, 2015

Page Listing Technology Stacks: Stackshare.io

Discover what top companies are using Learn about the tech stack behind some of the world's best software companies. See which tools and services they use, and how they're using them. Stackshare.io

The Three Rules of the DevOps Game

New essay by Mary Poppendieck: The Three Rules of the DevOps Game

January 18, 2015

Non-Proper Problem Solving

If this happens to you then you just need help:

January 13, 2015

Lean Startup Board Game

I really liked the Kanban Board Game, hope this Lean Game will become a reality:

January 12, 2015

"Lean Startup" - Leicht verständlich

Ein sehr guter Foliensatz, der Lean Startup anschaulich anhand von Startups, Unternehmen und Enterprises darstellt: Lean Startup - Auch in erwachsenen Unternehmen?

January 10, 2015

Technical Design Tradeoffs

Development teams should not think of it as making technical design decisions but managing tradeoffs. There is no optimal decision and the tradeoff does not automatically lead to technical debt.

Why Deployment Freezes Don't Prevent Outages

Think about this, every time you say "Never touch a running system!":

The Tyranny of the Ever Decreasing Timebox

A classic reference, a must have read in Agile Development: The Tyranny of the Ever Decreasing Timebox - David J Anderson

December 17, 2014

ThoughtWorks Technology Radar

ThoughtWorks has started a monthly review on technologies: Platform, Frameworks etc.
The PDF format is perfect! Technology Radar PDF

November 04, 2014

Problem Altsoftware

Der folgende Zitat aus dem Artikel fasst das Problem "Altsoftware" sehr gut zusammen:
"Die unterschiedlichen Web-Apps ermöglichen den Einsatz verschiedener Technologien innerhalb des Java-Universums. Wirklich genutzt wurde dies jedoch nicht. Im Gegenteil: Bewährtes wurde gerne wieder verwendet und übergreifende Konzepte in geteilten Bibliotheken hinterlegt. Eine zunehmend enge Verzahnung war das Ergebnis. Kleine Änderungen wurden aufwändig. Innovationen, wie z. B. eine Java-8-Einführung oder ein Spring-4-Update, erzeugen viel zu hohe Aufwände. Solche Themen dann zeitnah oder „nebenbei“ anzugehen, fällt nahezu aus. Unzufriedenheit über die veraltete Software und ein Verlust an „Spaß bei der Arbeit“ sind die subtilen wie ernstzunehmenden Konsequenzen.

Darüber hinaus traten weitere Symptome monolithischer Anwendungen auf: Eine technisch enge Kopplung von fachlich entkoppelten Themen resultiert in steigenden Lern- und Einarbeitungskurven bei den Entwicklern. Die Konsequenzen einer Änderung abzusehen wurde stetig schwieriger. "

September 29, 2014

August 10, 2014