Scale Jenkins with Kubernetes on Google Container Engine

Last week, I had the pleasure to speak at the Jenkins Community Day conference, in Paris, organized by my friends from JFrog, provider of awesome tools for software management and distribution. I covered how to scale Jenkins with Kubernetes on Google Container Engine.


For the impatient, here are the slides of the presentation I’ve given:



But let’s step back a little. In this article, I’d like to share with you why you would want to run Jenkins in the cloud, as well as give you some pointers to interesting resources on the topic.


Why running Jenkins in the cloud?


So why running Jenkins in the cloud? First of all, imagine your small team, working on a single project. You have your own little server, running under a desk somewhere, happily building your application on each commit, a few times a day. So far so good, your build machine running Jenkins isn’t too busy, and stays idle most of the day.


Let’s do some bottom of the napkin calculations. Let’s say you have a team of 3 developers, committing roughly 4 times a day, on one single project, and the build takes roughly 10 minutes to go.


3 developers * 4 commits / day / developer * 10 minutes build time * 1 project = 1 hour 20 minutes


So far so good, your server indeed stays idle most of the day. Usually, at most, your developers will wait just 10 minutes to see the result of their work.


But your team is growing to 10 persons, the team is still as productive, but the project becoming bigger, the build time goes up to 15 minutes:


10 developers * 4 commits / day / developer * 15 minutes build time * 1 project = 10 hours


You’re already at 10 hours build time, so your server is busy the whole day, and at times, you might have several build going on at the same time, using several CPU cores in parallel. And instead of building in 15 minutes, sometimes, the build might take longer, or your build might be queued. So in theory, it might be 15 minutes, but in practice, it could be half an hour because of the length of the queue or the longer time to build parallel projects.


Now, the company is successful, and has two projects instead of one (think a backend and a mobile app). Your teams grow further up to 20 developers per project. The developers are a little less productive because of the size of the codebase and project, so they only commit 3 times a day. The build takes more time too, at 20 minutes (in ideal time). Let’s do some math again:


20 developers * 3 commits / day / developer * 20 minutes build time * 2 projects = 40 hours


Woh, that’s already 40 hours of total build time, if all the builds are run serially. Fortunately, our server is multi-core, but still, there are certainly already many builds that are enqueued, and many of them, perhaps up to 2-3 or perhaps even 4 could be run in parallel. But as we said, the build queue increases further, the real effective time of build is certainly longer than 30 minutes. Perhaps at times, developers won’t see the result of their developments before at least an hour, if not more.


One last calculation? With team sizes of 30 developers, decreased productivity of 2 commits, 25 build time, and 3 projects? And you’ll get 75 hours total build time. You may start creating a little build farm, with a master and several build agents. But you also increase the burden of server management. Also, if you move towards a full Continuous Delivery or Continuous Deployment approach, you may further increase your build times to go up to deployment, make more but smaller commits, etc. You could think of running builds less often, or even on a nightly basis, to cope with the demand, but then, your company is less agile, and the time-to-market for fixes of new features might increase, and your developers may also become more frustrated because they are developing in the blind, not knowing before the next day if their work was successful or not.


With my calculations, you might think that it makes more sense for big companies, with tons of projects and developers. This is quite true, but when you’re a startup, you also want to avoid taking care of local server management, provisioning, etc. You want to be agile, and use only compute resources you need for the time you need them. So even if you’re a small startup, a small team, it might still make sense to take advantage of the cloud. You pay only for the actual time taken by your builds as the build agent containers are automatically provisioned and decommissioned. The builds can scale up via Kubernetes, as you need more (or less) CPU time for building everything.


And this is why I was happy to dive into scaling Jenkins in the cloud. For that purpose, I decided to go with building with containers, with Kubernetes, as my app was also containerized as well. Google Cloud offers Container Engine, which is basically just Kubernetes in the cloud.


Useful pointers


I based my presentation and demo on some great solutions that are published on the Google Cloud documentation portal. Let me give you some pointers.


Overview of Jenkins on Container Engine

https://cloud.google.com/solutions/jenkins-on-container-engine


Setting up Jenkins on Container Engine

https://cloud.google.com/solutions/jenkins-on-container-engine-tutorial


Configuring Jenkins for Container Engine

https://cloud.google.com/solutions/configuring-jenkins-container-engine


Continuous Deployment to Container Engine using Jenkins

https://cloud.google.com/solutions/continuous-delivery-jenkins-container-engine


Lab: Build a Continuous Deployment Pipeline with Jenkins and Kubernetes

https://github.com/GoogleCloudPlatform/continuous-deployment-on-kubernetes


The latter one is the tutorial I actually followed for the demo that I presented during the conference. It’s a simple Go application, with a frontend and backend. It’s continuously build, on each commit (well, every minute to check if there’s a new commit), and deployed automatically in different environments: dev, canary, production. The sources of the project are stored in Cloud Source Repository (it can be mirrored from Github, for example). The containers are stored in Cloud Container Registry. And both the Jenkins master and agents, as well as the application are running inside Kubernetes clusters in Container Engine.


Summary and perspective


Don’t bother with managing servers! Quickly, you’ll run out of CPU cycles, and you’ll have happier developers with builds that are super snappy!


And for the record, at Google, dev teams are also running Jenkins! There was a presentation (video and slides available) given last year by David Hoover at Jenkins World talking about how developers inside Google are running hundreds of build agents to build projects on various platforms.


Serverless, Chatbots et APIs de Machine Learning au meetup TechBreak

Cette semaine, j'ai eu le plaisir d'inaugurer le meetup TechBreak lancé par Talan ! Merci à mon ami Jérôme Bernard pour l'invitation. En plus d'un accueil avec un chouette buffet champagne et petits fours, une tireuse à bière avait même été amenée pour l'occasion ! Une cinquantaine de personnes était venue pour parler de chatbots, de serverless, de machine learning, qui sont des sujets chauds du moment.

Aujourd'hui, je vous publie les slides, et nous devrions avoir également les vidéos bientôt sur YouTube.

Un bot pour gérer l'agenda de ta conférence

Et si vous tiriez parti d'un bot pour préparer et ajuster votre agenda pour la conférence ? Est-ce qu'il y a des présentations sur le Machine Learning, ou sur Docker, ou votre langage de programmation préféré ? Qui présente cette session ?Dans cette session, nous allons regarder comment construire notre propre assistant, avec les APIs de reconnaissance vocale et d'analyse de langage naturel de Google Cloud, avec les services de API.AI pour construire des bots intelligents, et Google Cloud Functions pour implémenter la logique métier nécessaire.


Les APIs de Machine Learning

Reconnaissance vocale or visuelle ? Analyse du langage naturel ? Compréhension des vidéos ? Il y a une API pour ça, chez Google Cloud. Dans cette présentation, nous ferons un tour d'horizon des différentes APIs disponibles, que vous pouvez intégrer dans vos applications.Nous évoquerons également brièvement TensorFlow, le framework Open Source de Deep Learning lancé par Google, ainsi que la plateforme Cloud ML Engine qui permet d'entrainer vos réseaux de neurones et de lancer vos prédictions dans le cloud. 

Jenkins Community Day: scaler Jenkins avec Kubernetes sur Google Cloud

A Paris, le 11 juillet, Jenkins, notre personnage open source préféré, sera là, pour accueillir des conférenciers bien connus comme Kohsuke Kawaguchi (le papa de Jenkins), Julien Dubois (le j-hipster bro), Quentin Adam (de nuage intelligent), Nicolas De Loof (la déjantée abeille nuageuse déguisée en canard) et bien d'autres.... dont... (roulement de tambour)... moi !!!


J'aurai le plaisir de voler dans les nuages avec Jenkins : combien faut-il de machines ou d'esclaves pour faire tourner vos builds rapidement ? Et bien il en faut toujours un de plus ! Nous découvrirons comment scaler son utilisation de Jenkins, en tirant partie de Google Cloud Platform, comment enchainer ses esclaves, ses machines virtuelles ou ses containers, pour leur faire obéir à tous ses désirs de build ! Pour cela, nous utiliserons Kubernetes et Container Engine.

Je suis impatient de plonger dans ce sujet avec vous, et j'espère vous voir nombreux à ma session ! Merci beaucoup à JFrog, nos amis grenouilles, pour m'avoir invité et m'accueillir à présenter à cet événement clé, si vous vous intéressez à l'intégration continue, au déploiement continu, et au DevOps en général. Merci JFrog ! JFrog fait partie de ces quelques entreprises qui ont bien compris les problématiques des développeurs, des ingénieurs, et qui soutient très activement le monde de l'Open Source. De nombreux projets Open Source comme Apache Groovy et bien d'autre bénéficient de l'infrastructure de JFrog pour leurs déploiements, en particulier en utilisant des produits comme Artifactory et Bintray.

Pour s'inscrire à la conférence, c'est par ici !

A year as a Google Cloud Developer Advocate

Time flies! Last week was my first "Googleversary": It's already been a year since I joined Google Cloud as a Developer Advocate. What a ride it's been so far!

I announced my move to Google in June last year. And since then got the chance to:
  • talk at more than 20 conferences or meetups
  • give 3 keynotes
  • write 36 articles
  • meet with a dozen customers or so
  • addressed literally thousands of developers
For some conferences, like Devoxx Belgium, I even spoke 5 times! Or for my trip to Singapore, I had 6 talks or workshops lined up!

There's a great level of freedom in this job, and you can decide where to put the cursor in terms of travel, or where to put the puck regarding the topics and products to cover. This is really wonderful!

I had the chance to cover things like Google App Engine (you know I've been a big fan of it since 2009!), Kubernetes / Container Engine, or Cloud Functions, on the compute side of the story, with a bit of Cloud Endpoints for the Web API aspect. And on the Big Data / ML side, I talked about the various Machine Learning APIs (Vision, Speech recognition, Natural Language processing, Video intelligence), or played with BigQuery for analyzing the Github dataset. And lately, my big hit at conferences and meetups, it's been the hot topic of chatbots + serverless, with API.AI and Cloud Functions for building conversational interfaces.

But there are just so many hours in a day, and there's so much more I want to work on and play with! As I often tell friends and developers that I meet here and there: working as a developer advocate for Google Cloud, it's a bit like being a kid entering a toy store where you'd be told: "hey, here are all the toys we have, feel free to play with everything you want!" But it's not only the toys... you also get the chance to play with other kids (err, sorry, fellow DAs) on all those things, to imagine the next epic game together!

I'm looking forward to telling you more about Google Cloud! Let's meet and chat at the next conference or meetup!


Trying out Apache Groovy's new Antlr4 parser with Java 8 support

Apache Groovy is coming up with a new parser, that supports the Java 8 syntax elements, as well as some new notation and operators of its own (like !in, !instanceof or ?[] for safe navigation with collections, or with ?= for Elvis assignment). I blogged recently about the fact that you can try this new flavor online on this forked Groovy Web Console version, without the need of installing everything. But today I'll tell you how to build it for yourself in order to run it on your machine.

It's still to be decided which is going to be the version number of the release containing the new "parrot" parser, but you can already play with this syntax today. As I'm the kind of guy living on the edge of Groovy, I always use Groovy from the firehose, using the master branch with the latest and greatest changes. But I always forget about the right parameters or environment variable to use to build Groovy with the new parser, and to activate it... although it's clearly explained in the documentation. So as a note to self, to know where to look at, I decided to write it down in this post!

Build Apache Groovy from the master branch with the Antlr4 parser:

./gradlew -PuseAntlr4=true installGroovy

It's going to build an installation of Groovy that you can point SDKman at:

$ sdk install groovy target/install/ dev
$ sdk use groovy dev

That way, you can use this "dev" version of Groovy in your shell. However, you still need to enable the Antlr4 parser, and you can do so with the following exported environment variable:

$ export JAVA_OPTS=-Dgroovy.antlr4=true

Then, when running groovyConsole or groovysh, you'll be able to try the new syntax. 

To learn more about the new parser, you can have a look at my presentation with some of the novelties provided by the "parrot" parser here:


And you can read Sergio's translation of Daniel's article on the new features here.
 
© 2012 Guillaume Laforge | The views and opinions expressed here are mine and don't reflect the ones from my employer.