Build Automation

Agenda

  • Introduction

  • Software Artifacts

  • Build Tools

  • Maven

  • GitLab CI

  • Conclusion

Definition

Build automation is the process of automating the creation of a software build and the associated processes including: compiling computer source code into binary code, packaging binary code, and running automated tests.

— Wikipedia
build big picture

Build Automation Goals

  • Eliminate redundant tasks

  • Reduce dependencies on key personnel

  • Log the history of builds and releases in order to investigate issue

  • Improve product quality

  • Accelerate the build process

  • Reduce build issues.

save time

Motivation

Build is just compilation…​

build compilation

Motivation (cont.)

and packaging …​

build packaging

Motivation (cont.)

and testing …​

build testing

Motivation (cont.)

and integrating …​

build integration

Motivation (cont.)

and evaluating the tests …​

build evaluation

Motivation (cont.)

and generating some code …​

build generation

Motivation (cont.)

and generating the documentation …​

build documentation

Motivation (cont.)

and deploying …​

build deployment

Motivation (cont.)

and also:

  1. Processing resources

  2. Keeping a history of builds

  3. Source code analysis

  4. Managing release versions

  5. and several other little things!

Agenda

  • Introduction

  • Software Artifacts

  • Build Tools

  • Maven

  • GitLab CI

  • Conclusion

Understanding Software Artifacts

In software construction, the term software artifact denotes a software that has been documented and stored in a repository so it can be retrieved upon demand.

software artifact

Artifact Dependencies

cd software artifact
Figure 1. Software Artifact
od software artifact
Figure 2. Software Artifact Example

Dependency Scope

dependency scope

Dependency Scope (Cont.)

Different Scopes:
Runtime

The dependency is needed during compilation, testing and execution

Testing

The dependency is only needed during testing

Compile

The dependency is only needed during compilation

Provided

The dependency is needed during compilation and testing, but should not be embedded in the package: it will be provided by the execution environment

Dependency Versions

dependency version

Version Ranges

RangeMeaning

(,1.0]

x ⇐ 1.0

1.0

It generally means 1.0 or a later version, if 1.0 is not available. Various Maven plug-ins may interpret this differently, so it is safer to use one of the other, more specific options.

[1.0]

Exactly 1.0

[1.2,1.3]

1.2 ⇐ x ⇐ 1.3

Version Ranges (Cont.)

RangeMeaning

[1.0,2.0)

1.0 ⇐ x < 2.0

[1.5,)

x >= 1.5

(,1.0],[1.2,)

x ⇐ 1.0 or x >= 1.2. Multiple sets are separated by a comma.

(,1.1),(1.1,)

This excludes 1.1 if it is known not to work in combination with the library.

Agenda

  • Introduction

  • Software Artifacts

  • Build Tools

  • Maven

  • GitLab CI

  • Conclusion

Apache Maven

maven logo
  • Java Build and Dependency resolution tool from Apache Foundation

  • Introduced several standards: project layout, artifact repository organization, build lifecycle, etc.

  • Most used Java build tool

Gradle

gradle logo
  • Open-source build tool, based on Maven standards.

  • Advantages over Maven:

    • Flexibility

    • Performance

    • Better user experience (DSL vs. XML)

GitLab CI

gitlab logo
  • Open-source, Continuous Integration and Continuous Delivery tool combined with GitLab.

  • Simple fast, scalable, and multi-language.

  • Supports Docker

SonarQube

sonarqube logo
  • Open-source platform used to measure and analyze the source code quality

  • Can analyze and manage code of more than 20 programming languages, including Java, c/c++, PL/SQL, and Cobol

  • Enables Continuous Code Quality Management

Agenda

  • Introduction

  • Dependency Resolution

  • Software Artifacts

  • Build Tools

  • Maven

  • GitLab CI

  • Conclusion

Foreword

maven logo

Maven was born of the practical desire to make several projects at the Apache Software Foundation(ASF) work in the same, predictable way.

Before, every project at the ASF had a different approach to compilation, distribution, and web site generation.

Ex. Build process for Tomcat was different from build process used by Struts. Turbine developers used a different approach to site generation than the developers of Jakarta, Commons, etc.

This lack of common approach to building software meant that every new project tended to copy and paste another project’s build system.

The barrier to entry for a project with a difficult build system was extremely high. Difficult to attract developer because it could take an hour to configure everything in just the right way.

Apache Maven

Maven is a:
  • Project management framework.

  • Build tool.

  • Set of standards.

  • Artifact repository.

  • Standard build life cycle.

Main Principles

Convention over configuration

Reuse of build logic

Conventions

  • Standard directory layout: sources, resources, config files, generated output, documentation, web site, etc.

  • A single Maven project produces a single output.

  • Standard naming conventions.

Features

  • A comprehensive software project model (POM) and tools that interact with this declarative model.

  • Allows developers to declare goals and dependencies and rely on default structure and plugin capabilities.

Benefits

  • Coherence: Maven allows organizations to standardize on a set of best practices.

  • Reusability (reusing the best practices).

Benefits

  • Agility: lowers the barrier to reuse not only of build logic but of components.

    • It is easier to create a component and integrate it to a multi-project build.

    • Easier for developers to jump between projects without the a steep learning curve.

  • Maintainability: stop building the build and focus on the application.

The Project Object Model

<project>
 <modelVersion>4.0.0</modelVersion>
  <groupId>com.mycompany.app</groupId>
  <artifactId>javaero</artifactId>
  <packaging>jar</packaging>
  <version>1.0-SNAPSHOT</version>
  <name>Javaero</name>
  <url>http://maven.apache.org</url>
  <description></description>
 <dependencies>
     <dependency>
       <groupId>junit</groupId>
       <artifactId>junit</artifactId>
       <version>4.12</version>
       <scope>test</scope>
     </dependency>
  </dependencies>
</project>
  • This pom will allow you to compile, test and generate basic documentation.

Build lifecycle

  • A series of phases where each phase can perform one or more actions, or goals, related to that phase.

  • For instance, the compile phase invokes a certain set of goals to compile set of classes.

Build lifecycle

  • If you tell Maven to compile, the validate, initialize, generate-sources, process-sources, generate-resources, and compile phases will execute.

Build lifecycle

  • Standard build life cycle consists of many phases and these can be thought of as extension points.

  • When you need to add some functionality to the build lifecycle you do so with the plugin.

Build lifecycle

  • Maven Plugins provide reusable build logic that can be slotted into the standard build life cycle.

  • Any time you need to to customize the way your projects builds you either employ the use of an existing plugin or create custom plugin for the task at hand.

Default build lifecycle

build lifecycle

Default build lifecycle (simplified)

  1. validate - validate the project is correct and all necessary information is available

  2. compile - compile the source code of the project

  3. test - test the compiled source code using a suitable unit testing framework. These tests should not require the code be packaged or deployed.

  4. package - take the compiled code and package it in its distributable format, such as a JAR.

Build lifecycle (cont.)

  1. integration-test - process and deploy the package if necessary into an environment where integration tests can be run

  2. verify - run any checks to verify the package is valid and meets quality criteria

  3. install - install the package into the local repository, for use as a dependency in other projects locally

  4. deploy - done in an integration or release environment, copies the final package to the remote repository for sharing with other developers and projects.

Dependencies

  • A dependency is a reference to a specific artifact that resides in a repository.

  • In order for Maven to attempt to satisfy a dependency, Maven needs to know what the repository to look in as well as the dependency’s coordinates.

  • A dependency is uniquely identified by the following identifiers: groupId, artifactId and version.

Dependencies are declaratives

  • In the POM you are not telling Maven where the dependencies physically, you are simply telling Maven what a specific project expects.

Dependencies

  • When a dependency is declared, Maven tries to satisfy that dependency by looking in all of the remote repositories that are available, within the context of your project, for artifacts that match the dependency request.

  • If matching artifact is located, it transports it from remote repository to your local for general use.

Dependency Declaration

<project>
(...)
<dependencies>
     <dependency>
       <groupId>junit</groupId>
       <artifactId>junit</artifactId>
       <version>3.8.1</version>
       <scope>test</scope>
     </dependency>
  </dependencies>
(...)
</project>

Repositories

repository
Types of repositories:
  1. Local: ~/.m2/repository

    • you must have a local repository in order for Maven to work.

  2. Remote

General pattern for the repository layout

repository pattern

If the groupId is a fully qualified domain name such as z.y.x then you would end up with a directory structure like the following:

directory pattern

Locating Dependency Artifacts

  1. Maven will attempt to find the artifact with groupId, artifactId and version in local repository -> ~/.m2/repository/…

  2. If this file is not present, it will be fetched from remote repository.

  3. By default, Maven will fetch an artifact from the central Maven repository at https://repo.maven.apache.org/

  4. If your project’s POM contains more than one remote repository, Maven will attempt to download an artifact from each repository in the order defined in your POM.

  5. Once dependency is satisfied, the artifact is downloaded and installed in your local repository.

Getting Started with Maven

Preparing to use Maven:

  1. Download Maven from http://maven.apache.org and unzip it to your desired installation directory. Ex. C:\maven in Windows, or /usr/local/maven in Linux.

  2. Add System variable M2_HOME as well as $M2_HOME/bin directory to your system path.

  3. Type the following in a terminal or command prompt:

mvn --version
  1. If you are behind a firewall, create a <your-home-directory>/.m2/setting.xml with the following content:

<settings>
  <proxies>
    <proxy>
      <active>true</active>
      <protocol>http</protocol>
      <host>proxy.ensinfo.sciences.univ-nantes.prive</host>
      <port>3128</port>
    </proxy>
  </proxies>
</settings>
  1. If there is an active Internal Maven proxy running. create a <your-home-directory>/.m2/setting.xml with the following content:

<settings>
  <mirrors>
    <mirror>
      <id>maven.mycompany.com</id>
      <name>University's Maven Proxy</name>
      <url><http://maven.mycompany.com/maven3</url>
      <mirrorOf>central</mirrorOf>
    </mirror>
  </mirrors>
</settings>

Creating your First Maven Project

  • To create the Quick Start Maven project, execute the following:

mvn archetype:generate -DgroupId=fr.unantes.datatypes
    -DartifactId=units
    -DarchetypeArtifactId=maven-archetype-quickstart
    -DinteractiveMode=false
  • New directory units has been created for the new project, and this directory contains your pom.xml which looks like the following file:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>fr.unantes.datatypes</groupId>
  <artifactId>units</artifactId>
  <packaging>jar</packaging>
  <version>1.0-SNAPSHOT</version>
  <name>units</name>
  <url>http://maven.apache.org</url>
  <dependencies>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>3.8.1</version>
      <scope>test</scope>
    </dependency>
  </dependencies>
</project>
units
Figure 3. Directory structure of your project after archetype generation

Maven Basic Commands

  • Compile the project: mvn compile

  • The output were placed in target/classes

  • Compile rests and sources and run unit tests: mvn test

  • Compile your test (but not execute them): mvn test-compile

  • Make a JAR file or the package: mvn package

  • take a look at the target directory and you will see the generated JAR file.

  • Install the artifact(JAR file) you’ve generated into your local repository so that it can be used by other projects: mvn install

<your-home-directory>/.m2/repository is the default location for the local repository

Test Execution

  • Surefire plugin (which executes the test) looks for tests contained in files with a particular naming convention. By default the following tests are included:

  • **/*Test.java

  • //Test.java

  • **/*TestCase.java

  • Conversely, the following tests are excluded:

  • **/Abstract*Test.java

  • **/Abstract*TestCase.java

Website/Reports Generation

  • Generate a basic website for the project:

  • mvn site

  • Clean the project (remove the target directory):

  • mvn clean

  • Create descriptor for the projects

  • for IntelliJ IDEA:

    • mvn idea:idea

  • for eclipse:

    • mvn eclipse:eclipse

Summary

  • Maven is a set of standards, a repository, a framework and is a software.

  • Maven is more than just downloading another JAR and a set of scripts, it is the adoption of processes that allow you to take your software to the next level.

Agenda

  • Introduction

  • Dependency Resolution

  • Software Artifacts

  • Build Tools

  • Maven

  • GitLab CI

  • Conclusion

GitLab CI

gitlab ci
  • Continuous Integration service fully integrated with GitLab.

  • Open source and included in any GitLab CE installation.

Runners

gitlab runner
  • In GitLab CI, Runners run your .gitlab-ci.yaml file.

  • A runner is an isolated (virtual) machine that picks up builds through the coordinator API of GitLab CI.

Containers

docker container
GitLab CI builds run inside one or more docker containers
  • Simple to get started (works out of the box)

  • Uses containers as throw-away VMs

Starting «.gitlab-ci.yaml»

  1. Choose a docker image for Maven: https://hub.docker.com/_/maven

  2. Create a simple job called test

    • Installs graphviz

  3. Runs on every push or MR

.gitlab-ci.yaml file
image: maven:3.6.1-jdk-8

test:
  script:
    - apt-get update -qq
    - apt-get install -y -qq graphviz

Jobs

Jobs can be run:
  • using Docker containers

  • using Docker containers and executing job over SSH

  • using Docker containers with autoscaling on different clouds and virtualization hypervisors

  • connecting to remote SSH server

image: maven:3.6.1-jdk-8

before_script:
  - 'apt-get update -qq && apt-get install -y -qq graphviz'

job:build:compile:
  script:
    - 'mvn --settings ./.gitlab.maven.settings.xml $MAVEN_OPTS -e $MAVEN_CLI_OPTS compile'

job:build:test:
  script:
    - 'mvn --settings ./.gitlab.maven.settings.xml $MAVEN_OPTS test'

Jobs (Cont.)

Why multiple jobs?
  • The smaller a job the easier parallelization across multiple runners is

  • Smaller jobs are easier to diagnose when they break

image: maven:3.6.1-jdk-8

before_script:
  - 'apt-get update -qq && apt-get install -y -qq graphviz'

job:build:compile:
  script:
    - 'mvn --settings ./.gitlab.maven.settings.xml $MAVEN_OPTS -e $MAVEN_CLI_OPTS compile'

job:build:test:
  script:
    - 'mvn --settings ./.gitlab.maven.settings.xml $MAVEN_OPTS test'

Pipeline

gitlab pipeline
  • Pipelines are user-defined models of a build process

  • Typically includes stages for building an application, testing it and then delivering it

Artifacts

Allows for transferring files from one job to the next within the same build
  • Why run apt-get every time?

  • Artifacts are passed between jobs

  • They may have an expiration (defaults to 1 day)

  • Dependencies determine a job can only run after some other job

job:build:artifacts:
  script:
- 'apt-get update -qq && apt-get install -y -qq graphviz'
  artifacts:
    expire_in: 1 day
    paths:
      - vendor/

job:build:compile:
  script:
    - 'mvn --settings ./.gitlab.maven.settings.xml $MAVEN_OPTS -e $MAVEN_CLI_OPTS compile'
  dependencies:
  - job:build:artifacts

Stages

The ordering of elements in stages defines the ordering of builds' execution:
  • Jobs of the same stage are run in parallel.

  • Jobs of the next stage are run after the jobs from the previous stage complete successfully.

image: docker-registry.univ-nantes.fr/sunye-g/docker-asciidoctor-revealjs

before_script:
  - apt-get update -qy
  - curl -sL https://deb.nodesource.com/setup_10.x | bash -

stages:
  - build
  - deploy

build:
  stage: build
  script:
   - bash ./compile.sh

pages:
  stage: deploy
  script:
    - mkdir -p public
    - rsync -r target/site/ public

Limiting builds

Sometimes we do not want certain jobs to always run:
  • Usually when they take lots of resources (time, compute)

    • Or when they do things like deploy to staging/production

  • Use only

    • Defines a list of git refs for which build is created

pages:
  stage: deploy
  script:
    - mkdir -p public
    - rsync -r target/site/ public
    - du -sh public
    - ls -la public
  artifacts:
    paths:
      - public
  only:
    - master

Agenda

  • Introduction

  • Dependency Resolution

  • Software Artifacts

  • Build Tools

  • Maven

  • GitLab CI

  • Conclusion

Conclusion

New perspective:
  • Software build is seen as a software artifact

  • It should be developed and maintained as any other artifact

Main benefits:
  • Eliminate redundant tasks

  • Reduce dependencies on key personnel

  • Log the history of builds and releases in order to investigate issue

  • Improve product quality

  • Accelerate the build process

  • Reduce build issues.

References