Analyzing half a million Gradle build files

Gradle is becoming the build automation solution of choice among developers, in particular in the Java ecosystem. With the Github archive published as a Google BigQuery dataset, it's possible to analyze those build files, and see if we can learn something interesting about them!

This week, I was at the G3 Summit conference, and presented about this topic: I covered the Apache Groovy language, as per my previous article, but I expanded my queries to also look at Grails applications, and Gradle build files. So let's see what the dataset tells us about Gradle!

Number of Gradle build files and repositories

Instead of going through the whole Github dataset, I'm going to restrict the dataset by saving only the Gradle build files in my own, smaller, dataset:

SELECT * FROM [bigquery-public-data:github_repos.files] 
WHERE RIGHT(path, 7) = '.gradle'

This query returns only the files whose extension is .gradle. I'm saving the results in my [github.gradle_build_files] table.

But I also need the content of those files:

SELECT *
FROM [bigquery-public-data:github_repos.contents]
WHERE id IN (SELECT id FROM [github.gradle_build_files])

And I will save the content in the table [github.gradle_build_contents].

Let's start with a simple query to count the Gradle build files on Github:

SELECT COUNT(*) as count
FROM [github-groovy-files:github.gradle_build_files]

There are 488,311 Gradle build files! Roughly half a million.

This is the number of Gradle files: note that a project can contain several build files, that a repository can contain several projects, but also that the Github dataset only provides data on repositories for which it could detect an Open Source license. So it gives an idea of the reach of Gradle, but doesn't necessarily give you the exact number of Gradle-based projects in the wild! (and obviously can't even account for the projects hosted internally and elsewhere)

Since a repository can contain several build files, let's have a look at the number of repositories containing Gradle build files:

SELECT COUNT(repo_name) as repos
FROM (
  SELECT repo_name
  FROM [github-groovy-files:github.gradle_build_files]
  GROUP BY repo_name
)

There are 102,803 repositories with Gradle build files.

I was curious to see the distribution of the number of build files across projects. So I used the quantiles function:

SELECT QUANTILES(buildFilesCount, 101) 
FROM (
  SELECT repo_name, COUNT(repo_name) as buildFilesCount
  FROM [github-groovy-files:github.gradle_build_files]
  GROUP BY repo_name
  ORDER BY buildFilesCount DESC
)

I used a small increment (one percent), as the data was skewed towards some repositories with a huge amount of Gradle build files: essentially repositories like the Udemy course on Gradle for Android, or an online book about Android development, as they had tons of small build files or variations of build files with incremental changes for explanation purpose.

22% of the repositories had only 1 build file
85% of the repositories had up to 5 build files
95% of the repositories had less than 10 build files

The repository with the biggest amount of build files had 1333 of them!

Gradle vs Maven

You might also be interested in comparing Gradle and Maven, as they are often put against each other in holy build wars. If you look at the number of pom.xml files on Github:

SELECT count(*) 
FROM [bigquery-public-data:github_repos.files]
WHERE path LIKE '%pom.xml'

There are about 1,007,705 pom.xml files vs the 488,311 we counted for Gradle. So roughly twice as many for Maven.

But if you look at the number of repositories with Maven build files:

SELECT COUNT(repo_name) as repos
FROM (
  SELECT repo_name
  FROM [bigquery-public-data:github_repos.files]
  WHERE path LIKE '%pom.xml'
  GROUP BY repo_name
)

There are 131,037 repositories with Maven pom.xml files, compared to the 102,803 repositories with Gradle build files we counted earlier (about only 27% more). It seems Gradle is catching up with Maven!

Gradle build file names

Bigger projects tend to split their build tasks under different build files. I was curious to see which kind of split developers did by looking at the most frequent build file names:

SELECT f, COUNT(f) as count
FROM (
  SELECT LAST(SPLIT(path, '/')) AS f
  FROM [github-groovy-files:github.gradle_build_files]
)
GROUP BY f
ORDER BY count DESC


Of course, build.gradle comes first. Followed by settings.gradle. Notice the number of build files which are related to making releases, publishing / deploying the artifacts to a repository. There are also a few checking the quality of the code base, using checkstyle for style violations, JaCoCo for code coverage.

Gradle versions

Gradle projects often use the Gradle wrapper to help developers use a particular and consistent version of Gradle, without necessiting Gradle to be installed locally. For those developers who decided to commit their Gradle wrapper in Github, we can have a look at the breakdown of Gradle versions currently in the wild:

SELECT version, COUNT(version) AS count
FROM (
  SELECT REGEXP_EXTRACT(line, r'gradle-(.*)-(?:all|bin).zip') AS version
  FROM (
    SELECT SPLIT(content, '\n') AS line
    FROM [github-groovy-files:github.gradle_wrapper_properties_files]
  )
  WHERE line LIKE 'distributionUrl%'
)
GROUP BY version
ORDER BY count DESC


It looks like Gradle 2.4 was a big hit!

Gradle plugins

Gradle projects often take advantage of third-party plugins. You'll see plugins declared with the "id" syntax or applied with "apply plugin". Let's looked at both:

SELECT plugin, COUNT(plugin) AS count
FROM (
  SELECT REGEXP_EXTRACT(line, r'apply plugin: (?:\'|\")(.*)(?:\'|\")') AS plugin
  FROM (
    SELECT SPLIT(content, '\n') AS line
    FROM [github-groovy-files:github.gradle_build_contents]
  )
)
GROUP BY plugin
ORDER BY count DESC


Look at the big number of Android related plugins! Clearly, Android adopting Gradle as build solution gave a big boost to Gradle's adoption!

The plugins declared with "id" show another story though: 

SELECT newplugin, COUNT(newplugin) AS count
FROM (
  SELECT REGEXP_EXTRACT(line, r'id (?:\'|\")(.*)(?:\'|\") version') AS newplugin
  FROM (
    SELECT SPLIT(content, '\n') AS line
    FROM [github-groovy-files:github.gradle_build_contents]
  )
)
GROUP BY newplugin
ORDER BY count DESC


Here, we see a big usage of the Bintray plugin and the shadow plugin.

Build dependencies

Now it's time to look at dependencies. First, the "compile" dependencies:

SELECT dep, COUNT(dep) AS count
FROM (
  SELECT REGEXP_EXTRACT(line, r'compile(?: |\()(?:\'|\")(.*):') AS dep
  FROM (
    SELECT SPLIT(content, '\n') AS line
    FROM [github-groovy-files:github.gradle_build_contents]
  )
)
GROUP BY dep
ORDER BY count DESC


Again, there's a big usage of Android related dependencies. We also notice Spring Boot, GSON, Guava, SLF4J, Retrofit, Jackson.

For the test dependencies:

SELECT dep, COUNT(dep) AS count
FROM (
  SELECT REGEXP_EXTRACT(line, r'testCompile(?: |\()(?:\'|\")(.*):') AS dep
  FROM (
    SELECT SPLIT(content, '\n') AS line
    FROM [github-groovy-files:github.gradle_build_contents]
  )
)
GROUP BY dep
ORDER BY count DESC


No big surprise with JUnit coming first. But we have Spock, Mockito's mocking library, AssertJ assertions, Hamcrest matchers.

Summary

And this wraps up our analysis of Gradle build files, thanks to Google BigQuery and the Github dataset. It's interesting to see that Gradle has gained a very significant market share, coming pretty close to the Maven incumbent, and to see lots of Android projects are on Github with Gradle builds.

Analyzing Gradle, Grails, and Apache Groovy source code hosted on Github with BigQuery

A few months ago, I wrote an article about what you can learn from millions lines of Apache Groovy source hosted on Github, thanks to Google BigQuery. We answered a few questions like:

  • How many Groovy files are there on Github?
  • What are the most popular Groovy file names?
  • How many lines of Groovy source code are there?
  • What's the distribution of size of source files?
  • What are the most frequent imported packages?
  • What are the most popular Groovy APIs used?
  • What are the most used AST transformations?
  • Do people use import aliases much?
  • Did developers adopt traits?
At G3 Summit this week, I gave a presentation on this source code analysis, but decided to expand it a little bit, by also adding queries about Grails and Gradle.

For Gradle, here are the questions that I answered:

  • How many Gradle build files are there?
  • How many Maven build files are there?
  • Which versions of Gradle are being used?
  • How many of those Gradle files are settings files?
  • What are the most frequent build file names?
  • What are the most frequent Gradle plugins?
  • What are the most frequent “compile” and “test” dependencies?
And for Grails, here's what I covered:

  • What are the most used SQL database used?
  • What are the most frequent controller names?
  • What are the repositories with the biggest number of controllers?
  • What is the distribution of number of controllers?
I'll come back on those new queries in subsequent articles! But in the meantime, let me show you the slides I presented, and the results of those queries.


My G3 Summit Apache Groovy keynote

This week, I'm in Florida for the brand new G3 Summit conference, dedicated to the Apache Groovy ecosystem (Grails, Gradle, and more). I had the chance of giving the keynote, where I gave an overview of the Apache Groovy project's philosophy, history, and where it's heading. In the second part, I'm showcasing the new features, new syntax constructs, already there or coming in Groovy 2.4.x, in the future Groovy 2.5, and in Groovy 3.0 with the new parser.


Billions of lines of code in a single repository, seriously?

When I joined Google last June, I discovered a new world: tons of new acronyms or project code names to learn about, but also a particular environment for your source code. At Google, engineers work on a huge monolithic source code repository comprising of: 
  • 1 billion files
  • 9 million source files
  • 2 billion lines of code
  • 35 million commits
  • 86 terabytes of content
  • 45 thousands of commits every day.
Rachel Potvin, who's an engineering manager at Google, wrote an article for ACM about how Google handles such a huge repository, as well as the tools and practices around that. Wired also covered the topic in their article "Google is 2 billion lines of code and it's all in one place". And Rachel also presented this topic at the @Scale conference.

I had the chance to give here presentation at Devoxx Belgium

You can find the slide deck embedded below:

And the talk was also recorded, so you can view the video on Devoxx's YouTube channel here:


Binge streaming Web APIs, with Ratpack, Cloud Endpoints, App Engine Flex and Streamdata.io

At Devoxx last week, I had the chance to do a joint tools-in-action with my talented friend Audrey Neveu, titled "Binge streaming you Web API".

For the impatient, scroll down to the end of the article to access the slides presented, and view a recording of the video!

We split the presentation in two sections: 1) first of all, we obviously need to create and deploy a Web API, and then 2) to configure and use Streamdata.io to stream the updates live, rather than poll the API endlessly.

For the purpose of our demo, Audrey and myself decided to surf on the theme of the conference, by publishing ourselves an API of the conference content, listing all the talks and speakers available.

As the content is pretty much static, we needed some data that would evolve in real-time as well. We added a voting capability, so that users could click on a little smiley to say if they're enjoying the talk or not. With the streaming capability, as soon as votes are taking place, we update the UI with the new vote results.

Implementing my API with Ratpack

To build the API, I decided to go with the Ratpack framework. As the Ratpack website states:

Ratpack is a set of Java libraries for building modern HTTP applications.
It provides just enough for writing practical, high performance, apps.
It is built on Java 8, Netty and reactive principles.

You can use Ratpack with Java 8, but there's also a nice Groovy wrapper. So with my Apache Groovy hat on, I naturally chose to go with Groovy!

In the Groovy community, there's a tool called Lazybones which allows you to create template projects easily. And we're also using SDKman for installing various SDKs, including SDKman. 

If you don't have lazybones installed (but have SDKman), it's fairly easy to install:
sdk install lazybones
With both installed already on my machine, I just needed to create a new project with the Ratpack template:
lazybones create ratpack
And I had my template project ready!

Gradle to the rescue to build the app

My Gradle script is pretty straightforward, using the Ratpack Groovy plugin, the Shadow plugin, etc. Nothing really fancy:
buildscript {
    repositories {
        jcenter()
    }
    dependencies {
        classpath "io.ratpack:ratpack-gradle:1.4.4"
        classpath "com.github.jengelman.gradle.plugins:shadow:1.2.3"
    }
}
apply plugin: "io.ratpack.ratpack-groovy"
apply plugin: "com.github.johnrengelman.shadow"
apply plugin: "idea"
apply plugin: "eclipse"
repositories {
    jcenter()
}
dependencies {
    runtime 'org.slf4j:slf4j-simple:1.7.21'
    testCompile "org.spockframework:spock-core:1.0-groovy-2.4"
}
To build this app, I'll use the tar distribution target, which generates startup scripts to be launched from the command-line:
./gradlew distTar
To run the app locally, you can use:
./gradlew run
By default, it runs on port 5050.

The URL paths and methods

For developing my API, I created the following paths and methods:
  • GET /api : to list all the talks available
  • GET /api/{day} : to restrict the list of talks to just one particular day
  • GET /api/talk/{id} : to view the details of a particular talk
  • POST /api/talk/{id}/vote/{vote} : to vote on a particular talk (negative, neutral, positive)
  • POST /import : to import a JSON dump of all the talks, but it's just used by me for uploading the initial content, so it's not really part of my API.
Implementing the API

My Ratpack app implementing this API (plus a few other URLs) spans a hundred lines of Groovy code or so (including blank lines, imports, and curly braces). The implementation is a bit naive as I'm storing the talks / speakers data in memory, but I should have used a backend storage like Cloud Datastore or Cloud SQL, potentially with Memcache in front. So the app won't scale well and data will not be synchronized across multiple instances running in parallel. For the sake of my demo though, that was sufficient!
import ratpack.handling.RequestLogger
import org.slf4j.LoggerFactory
import static ratpack.groovy.Groovy.ratpack
import static ratpack.jackson.Jackson.json as toJson
import static ratpack.jackson.Jackson.fromJson
def log = LoggerFactory.getLogger('Devoxx')
def allTalks = []
ratpack {
    handlers {
        all(RequestLogger.ncsa())
        post('import') {
            log.info "Importing talks dump"
            byContent {
                json {
                    parse(fromJson(List)).onError { e ->
                        String msg = "Import failed: $e"
                        log.error msg
                        response.status(400)
                        render toJson([status: "Import failed: $e"])
                    }.then { talks ->
                        allTalks = talks
                        log.info "Loaded ${allTalks.size()} talks"
                        render toJson([status: 'Import successful'])
                    }
                }
            }
        }
        prefix('api') {
            prefix('talk') {
                post(':id/vote/:vote') {
                    def aTalk = allTalks.find { it.id == pathTokens.id }
                    if (aTalk) {
                        def msg = "Voted $pathTokens.vote on talk $pathTokens.id".toString()
                        switch (pathTokens.vote) {
                            case "negative":
                                aTalk.reactions.negative += 1
                                log.info msg
                                render toJson([status: msg])
                                break
                            case "neutral":
                                aTalk.reactions.neutral += 1
                                log.info msg
                                render toJson([status: msg])
                                break
                            case "positive":
                                aTalk.reactions.positive += 1
                                log.info msg
                                render toJson([status: msg])
                                break
                            default:
                                response.status(400)
                                msg = "'${pathTokens.vote}' is not a valid vote".toString()
                                log.info msg
                                render toJson([status: msg])
                        }
                    } else {
                        response.status(404)
                        render toJson([status: "Talk $pathTokens.id not found".toString()])
                    }
                }
                get(':id') {
                    def aTalk = allTalks.find { it.id == pathTokens.id }
                    if (aTalk) {
                        log.info "Found talk: $pathTokens.id"
                        render toJson(aTalk)
                    } else {
                        String msg = "Talk $pathTokens.id not found"
                        log.info msg
                        response.status(404)
                        render toJson([status: msg])
                    }
                }
            }
            get(':day') {
                def talksPerDay = allTalks.findAll { it.day == pathTokens.day }.collect {
                    it.subMap(it.keySet() - 'summary')
                }
                if (talksPerDay) {
                    render toJson(talksPerDay)
                } else {
                    response.status(404)
                    render toJson([status: "Invalid day, or no talks found for: $pathTokens.day".toString()])
                }
            }
            get {
                render toJson(request.queryParams.full ? allTalks : allTalks.collect {
                    it.subMap(it.keySet() - 'summary')
                })
            }
        }
    }
}

Some interesting points about this code:
  • use of an NCSA-compliant request logger to log all API calls (following the NCSA usual output pattern) and a dedicated logger for important events, and you're able to watch both those kind of logs in the Stackdriver logging in the cloud console
  • the use of prefix('...') to factor common path parts
  • Jackson is being used both for parsing (the input file containing the initial list of talks) as well as for output for rendering the JSON payloads
  • see how we use byContent / json to handle the requests coming up with content-type of application/json
  • the rest is essentially error handling, collection filtering, etc.
Containerizing the app

Ratpack requires JDK 8 to run, and it's not based on servlets, I went with Google App Engine Flex, which allows me to run JDK 8 / Java 8 apps which can be based on networking toolkits like Netty.

I mentioned we're using the distTar target to build a distribution of the application, and that's what we'll point our Dockerfile at, as App Engine Flex allows you to customize Docker images for bundling and running your app:
FROM gcr.io/google_appengine/openjdk8
VOLUME /tmp
RUN mkdir -p /app/endpoints
ADD service.json /app/endpoints
ADD build/distributions/devoxx-reactions.tar /
ENV JAVA_OPTS='-Dratpack.port=8080 -Djava.security.egd=file:/dev/./urandom'
ENTRYPOINT ["/devoxx-reactions/bin/devoxx-reactions"]
I'm using the dedicated Open JDK 8 image for App Engine, from the Google Container Registry. I'm specifying the port for my app, the entry point to the startup scripts. You'll notice the lines about "endpoints" and "service.json", and I'll come to it in a minute: it's because I'm using Google Cloud Endpoints to expose and manage my API!

At this point, to feel safer, you can double check that your app is running under docker with something like:
./gradlew distTar
docker build -t devoxx-reactions-image .
docker run -p 127.0.0.1:8080:8080 -it devoxx-reactions-image
The app is running on port 8080 of your localhost. We'll see later on how to test it with curl, and how to load the sample data that we've prepared in public/data/talks.json.

Gcloud, to set up our projects

As we're going to use Cloud Endpoints on App Engine Flex, it's time that I start setting things up with the gcloud SDK:
gcloud init
gcloud components update
gcloud components install beta
gcloud config set project [YOUR_PROJECT_ID]
About Google Cloud Endpoints

Google Cloud Platform offers a particular service for managing Web APIs called Endpoints. With Endpoints, you're able to monitor your API (see what endpoints, methods, are called, with some nice graphs and stats), to secure your API (with different kind of authentication like Firebase authentication, JSON Web Tokens, API keys, etc.), to scale it.

Speaking of scaling, I'm using App Engine Flex here as my deployment target, but it's possible to use Compute Engine or Container Engine as well. Interestingly, you can use Endpoints API management with an API hosted in a third-party cloud, as well as on premises too!

The management aspect of the API is done thanks to a proxy, called the Endpoints Service Proxy, which is implemented on top of NGINX (and which will be open sourced). And to continue on scaling aspect of the story, it's interesting to note that this proxy is living along your app (in its own container). If your API needs to scale across several machines, instead of one giant middle proxy somewhere, your ESP will be duplicated the same way. So there's no single point of failure with a gigantic proxy, but it also means that the latency is super low (below 1 ms usually), because the proxy is as close as possible to your API, without needing any additional costly network hop.

The last interesting aspect about Cloud Endpoints that I'd like to mention is that your API contract is defined using OpenAPI Specs (formerly known as Swagger). So it doesn't matter which language, framework, tech stack you're using to implement your API: as long as you're able to describe your API with an OpenAPI Specification, you're good to go!

Specifying the contract of our API with OpenAPI Spec

I mentioned before the various resources and methods we're using for our API, and we'll encode these in the form of a contract, using the OpenAPI Spec definition format. In addition to the various resources and methods, we should also define the payloads that will be exchanged: basically, a Talk, a Result, a list of talks. We must define the different status codes for each kind of response. Here's what my specification looks like:
---
swagger: "2.0"
info:
  description: "Consult the Devoxx schedule and vote on your favorite talks."
  version: "1.0.0"
  title: "Devoxx Reactions"
  contact: {}
host: "devoxx-reactions.appspot.com"
schemes:
- "http"
paths:
  /import:
    post:
      summary: "Import a dump of the lists of talks"
      operationId: "ImportTalks"
      consumes:
      - "application/json"
      produces:
      - "application/json"
      parameters:
      - name: "talkList"
        in: "body"
        required: true
        schema:
          $ref: "#/definitions/TalkList"
      responses:
        200:
          description: "Status 200"
          schema:
            $ref: "#/definitions/Result"
        400:
          description: "Status 400"
          schema:
            $ref: "#/definitions/Result"
  /api:
    get:
      summary: "Get the list of talks"
      operationId: "GetAllTalks"
      produces:
      - "application/json"
      parameters: []
      responses:
        200:
          description: "Status 200"
          schema:
            type: "array"
            items:
              $ref: "#/definitions/Talk"
  /api/talk/{talk}:
    get:
      summary: "Get a particular talk"
      operationId: "GetOneTalk"
      produces:
      - "application/json"
      parameters:
      - name: "talk"
        in: "path"
        required: true
        type: "string"
      responses:
        200:
          description: "Status 200"
          schema:
            $ref: "#/definitions/Talk"
        404:
          description: "Status 404"
          schema:
            $ref: "#/definitions/Result"
  /api/talk/{talk}/vote/{vote}:
    post:
      summary: "Vote for a talk"
      operationId: "VoteOnTalk"
      produces:
      - "application/json"
      parameters:
      - name: "talk"
        in: "path"
        required: true
        type: "string"
      - name: "vote"
        in: "path"
        description: "The vote can be \"negative\", \"neutral\" or \"positive\""
        required: true
        type: "string"
      responses:
        200:
          description: "Status 200"
          schema:
            $ref: "#/definitions/Result"
        400:
          description: "Status 400"
          schema:
            $ref: "#/definitions/Result"
        404:
          description: "Status 404"
          schema:
            $ref: "#/definitions/Result"
  /api/{day}:
    get:
      summary: "Get the talks for a particular day"
      operationId: "GetTalksPerDay"
      produces:
      - "application/json"
      parameters:
      - name: "day"
        in: "path"
        required: true
        type: "string"
      responses:
        200:
          description: "Status 200"
          schema:
            type: "array"
            items:
              $ref: "#/definitions/Talk"
        404:
          description: "Status 404"
          schema:
            $ref: "#/definitions/Result"
definitions:
  Result:
    type: "object"
    required:
    - "status"
    properties:
      status:
        type: "string"
    description: "Voting results"
  Talk:
    type: "object"
    required:
    - "day"
    - "fromTime"
    - "fromTimeMillis"
    - "id"
    - "reactions"
    - "speakers"
    - "talkType"
    - "title"
    - "toTime"
    - "toTimeMillis"
    - "track"
    properties:
      day:
        type: "string"
      fromTime:
        type: "string"
      fromTimeMillis:
        type: "integer"
        format: "int64"
      id:
        type: "string"
        description: ""
      reactions:
        type: "object"
        properties:
          negative:
            type: "integer"
            format: "int32"
          neutral:
            type: "integer"
            format: "int32"
          positive:
            type: "integer"
            format: "int32"
        required:
        - "negative"
        - "neutral"
        - "positive"
      room:
        type: "string"
      speakers:
        type: "array"
        items:
          type: "string"
      summary:
        type: "string"
      talkType:
        type: "string"
      title:
        type: "string"
      toTime:
        type: "string"
      toTimeMillis:
        type: "integer"
        format: "int64"
      track:
        type: "object"
        properties:
          title:
            type: "string"
          trackId:
            type: "string"
        required:
        - "title"
        - "trackId"
    description: "A talk representation"
  TalkList:
    type: "array"
    items:
      $ref: "#/definitions/Talk"
    description: "A list of talks"
You can write your API specifications using either JSON or YAML. I chose YAML because it's a bit easier to read, for the human eye, and still as much readable for the computer as well.

There's one extra step for instructing Cloud Endpoints about our API definition: I needed to convert my OpenAPI Spec into the service definition format used internally by Cloud Endpoints, thanks to the following command:
gcloud beta service-management convert-config swagger20.yaml service.json
Deploying on App Engine Flex

To deploy on App Engine and use Cloud Endpoints, we're going to use the gcloud command-line tool again:
gcloud beta app deploy
After a few minutes, your app / API should be available and be ready for serving.

Testing our API

I have a JSON file with the bulk of the talks and their details, so I uploaded it with:
curl -d @src/ratpack/public/data/talks.json -H 'Content-Type: application/json' http://devoxx-reactions.appspot.com/import
And then I was able to call my API with:
curl https://devoxx-reactions.appspot.com/api
curl https://devoxx-reactions.appspot.com/api/monday
curl https://devoxx-reactions.appspot.com/api/talk/HFW-0944
And to vote for a given talk with:
curl -X POST https://devoxx-reactions.appspot.com/api/talk/XMX-6190/vote/positive
Managing your API

When you're visiting your cloud console, you'll be able to see interesting statistics and graphs, about the usage of your API:



Streaming the API

For the streaming part of the story, I'll let Audrey cover it! I focused on the backend, how she focused on the Polymer frontend, with a custom Streamdata component, that used the Streamdata proxy to get the patches representing the difference between consecutive calls to the backend. So when the votes were changing (but the rest of the talk details were left unchanged), Streamdata would send back to the front only the diff. In addition to keeping ongoing data exchanges low (in terms of size), the proxy is also able to take care of caching, so it also helps avoiding hitting the backend too often, potentially helping with scalability of the backend, by keeping the cache at the proxy level.

Slides and video available!

You can watch the video online on Devoxx' YouTube channel:


And you can also have a closer look at the slides as well:

 
© 2012 Guillaume Laforge | The views and opinions expressed here are mine and don't reflect the ones from my employer.