NetApp : Accelerate Continuous Integration (CI) with NetApp ONTAP

In this blog post, we explore how NetApp^®ONTAP^® data management features help prepare developer workspaces at scale with significantly less time and helps reduce infrastructure resource overhead.

• Predictable performance with NetApp AFF and data management efficiency with ONTAP allow more development workloads to run on a smaller infrastructure footprint.
• User and CI build times can be reduced with incremental builds for small code changes, so you can identify the bugs and issues in the code quickly.

ONTAP integrates into the CI workflow to provide better, more centralized control of the CI environment with security and management at the data layer. The NetApp-Jenkins framework 2.0 provides an automated build process using Jenkins pipelines that accelerates the CI workflow using ONTAP data management features with zero storage touch for developers. The framework uses stateful Docker containers with persistent ONTAP data volumes over NFS using RESTful APIs.

Reduced build time - In a NetApp integrated CI process, a git clone operation is performed once from the source repository to a data volume on ONTAP, at the beginning of the project. After the full build is complete, the build number is tagged to a Snapshot copy at the data layer.

The Snapshot copies are pointer-based, read-only copies of the data blocks that do not take any additional space except for the changed blocks. As the iterative CI process progresses after each code change submittal, these Snapshot copies are created and tagged to every build.

Apart from the baseline build, all the subsequent CI builds are incremental, thus saving a lot of time in the overall development process.

Predictable Software delivery - Depending on the size and the number of source code repositories, the number of builds generated in a day can be resource intensive and time consuming. AFF (along with ONTAP) provides predictable performance to complete parallel and distributed builds with a high degree of data efficiency for faster software delivery.

The NetApp integrated CI process allows developers to create their workspaces almost instantly at scale. Developers can use interactive desktop environment (IDE) tools such as Eclipse or IntelliJ to connect to their desktop workspaces for distributed development, with more data control for security and recovery from failures.

Enhanced engineering productivity - Developers select the last successful build and create a workspace. As mentioned earlier, every build is tagged with a Snapshot copy. NetApp FlexClone^® technology instantly creates a workspace with minimum data space on demand.

The NetApp clones are primarily sandboxes that contain the source code, artifacts, and compiled code. Developers can start writing code immediately, which improves their productivity.

All the stress that the git clone operation has on the Git server at scale is now offloaded to ONTAP, which provides more scalability with few resource constraints.

In many situations, developers must perform git merge to merge the changes in their workspaces with the latest CI build. The git merge operation traditionally triggers a full build that consumes resources and time. However, with FlexClone technology, git merge performs an incremental build that saves time and makes the CI process more efficient by eliminating merge conflicts.

Not all organizations have an established continuous delivery process in which any successful build is promoted to be released into production. However, this does not stop developers from generating a lot of builds during their workday. Most of the builds do not have to reside on expensive data platforms, but they need to be archived in different data tiers for retention.

Data Tier - Lower TCO by employing the flexibility & cost structure of the cloud. The FabricPool feature provides policy-based data tier capability to automatically move the builds to Amazon Simple Storage Service (S3).

Builds are generated during the CI process can use Fabricpool to reduce data footprint by tiering cold data in active filesystem to S3 endpoints. The builds are fully compiled copies of the source code and take up space. Moving the builds to S3 provides cost efficiency and allows business or asset owners to protect and reuse the binaries on demand.

Automating builds with ONTAP data management features using NetApp-Jenkins Framework accelerates the CI process dring the development workflow. As every development environment is different, the NetApp-Jenkins framework quantifiably improves developer productivity, reduces build time, and optimizes infrastructure efficiency. TR-4547 provides more information on the CI process with Jenkins, Git and JFrog Artifactory.