Technical Report
Data Protection for JFrog Artifactory on ONTAP 9 with SnapCenter 2.0 using RESTful APIs Application-Consistent Backup of MySQL Databases and Artifactory Repositories
Bikash Roy Choudhury, NetApp
Amarnath Rachapudi, NetApp
Jainish Shah, JFrog
July 2017 | TR-4607
Abstract
JFrog Artifactory is the most popular universal binary repository manager, supporting different software
packages created in any language. Developers write and test code using artifacts—libraries, compilers, and
tools—and deliver a successful final build. JFrog Artifactory is commonly used in DevOps processes as a
consolidated build repository. One of its key features is consistent version control for all the build packages
that development teams work on prior to production release.
Together with JFrog Artifactory, NetApp software manages all the build data that is created. In addition to
providing a robust and scalable data management plaftform, NetApp® ONTAP® helps developers meet
requirements such as data protection and data availability. By configuring JFrog Artifactory data volumes on
NetApp systems, developers can use NetApp SnapCenter® 2.0 to protect from data loss by taking
application-consistent snapshots of the database and the repository.
This report discusses the integration of JFrog Artifactory with MySQL, where the database and the checksum
repository is on ONTAP 9 a robust continuous integration/continuous deployment (CI/CD) pipeline.
SnapCenter 2.0 is used as a Microsoft Windows-based tool for taking application-consistent snapshots of the
database and the repository volumes in ONTAP.
In partnership with
2 JFrog Artifactory on ONTAP 9 Using SnapCenter © 2017 NetApp, Inc. All rights reserved. © 2016 NetApp, Inc. All rights reserved.
TABLE OF CONTENTS
1 Introduction ........................................................................................................................................... 3
2 ONTAP 9 and JFrog Artifactory .......................................................................................................... 4
3 SnapCenter 2.0 and Artifactory 5.1.4 .................................................................................................. 5
4 Summary ............................................................................................................................................... 9
5 Conclusion .......................................................................................................................................... 10
References ................................................................................................................................................. 11
Appendix .................................................................................................................................................... 11
LIST OF FIGURES
Figure 1) JFrog Artifactory in the continuous integration/continuous deployment (CI/CD) workflow. .............................3
Figure 2) Continuous integration workflow with JFrog Artifactory. ..................................................................................4
Figure 3) SnapCenter 2.0 integration with JFrog Artifactory. ..........................................................................................6
Figure 4) SnapCenter 2.0 and the MySQL plug-in. ........................................................................................................7
Figure 5) List of Artifactory volumes configured in SnapCenter 2.0. ...............................................................................7
Figure 6) A consistency group is enabled from SnapCenter 2.0 for Artifactory volumes. ...............................................8
Figure 7) SnapCenter 2.0 creates a snapshot with job ID 601. ......................................................................................9
Figure 8) Snapshot listing of the Artifactory volumes in NetApp OnCommand® System Manager. ...............................9
3 JFrog Artifactory on ONTAP 9 Using SnapCenter © 2017 NetApp, Inc. All rights reserved. © 2016 NetApp, Inc. All rights reserved.
1 Introduction
In a typical DevOps software development lifecycle process, the development languages used by development teams can require different artifacts in the form of repositories, tools, and libraries to compile and package the application dependencies. These dependencies come into play while the team is promoting the product to quality assurance/user acceptance testing and then releasing it into production. Managing different versions of binaries from different repositories created by various development teams is a challenge. JFrog Artifactory is a universal binary repository manager that supports different software packages created through different development tools, languages, and technologies.
As illustrated in Figure 1, Artifactory plays the role of a central hub for a binary artifact repository. It
manages builds from the CI pipeline and then promotes them to the CD process with well-defined security
policies for authorized personnel. The binary artifact repository is very different from the source code
repository: The source code repository manages version control for source code only, and does not
include prebuild artifacts such as run-time libraries, tools, and compilers.
During the CI process, several types of data can be generated each time a build is updated or revised.
The combination of binaries (final builds); the tools, libraries, and compilers used to develop the code;
and information about the software and systems used at the time an application was created are
collectively referred to as an artifact. The complexity of managing these artifacts increases with the size of
development teams, the increased frequency of builds, and the amount of 3rd-party dependencies (i.e.
programming languages, libraries, frameworks, etc).
This technical report focuses on a use case involving the implementation of JFrog’s Artifactory with
NetApp® ONTAP® 9 data management software. ONTAP provides data management, data protection,
and high data availability, and enables performance at scale. Also included in the implementation is
NetApp SnapCenter® 2.0. Together with ONTAP, SnapCenter 2.0 and storage-level RESTful APIs
provide monitoring and reporting, provisioning with service-level objectives, and data protection.
In the context of storage, JFrog Artifactory can be thought of as consisting of two primary components: a
database that stores the metadata pertaining to artifacts and builds, and the repository that stores the
files as checksums. JFrog uses Apache Derby DB as the default database, but it can connect with any
commercial SQL database for the metadata store.
Developer Workspace(Private Builds)
QA/Staging Deploy
Source Code Repository
Binary Artifact Repository
Continuous Integration (Integrate Builds)
1
24
5 6
1
2Pull artifacts, integrate Builds;
Push Builds to JFrog Artifactory
3 User Workspace for Private Builds
4 Commit Code Changes3
Figure 1) JFrog Artifactory in the continuous integration/continuous deployment (CI/CD) workflow.
4 JFrog Artifactory on ONTAP 9 Using SnapCenter © 2017 NetApp, Inc. All rights reserved. © 2016 NetApp, Inc. All rights reserved.
This report discusses the integration of Artifactory with MySQL, where the database and the checksum
repository are on ONTAP 9 for the CI/CD pipeline. SnapCenter 2.0 is used as a Windows-based tool for
taking application-consistent snapshots of the database and the repository volumes in ONTAP.
2 ONTAP 9 and JFrog Artifactory
NetApp ONTAP 9 is data management software that is optimized for many types of workloads, such as
databases, file shares, analytics, and application development. ONTAP supports these workloads in both
physical and virtual environments that can be on the premises or in a hybrid cloud. Along with data
reliability, efficiency, and availability, ONTAP provides horizontal scaling and performance required for all
supported workloads on a standard run-time system.
Because workloads are increasingly running on containers, NetApp now provides NetApp Docker Volume
Plugin to enable persistent data volumes to be mounted to Docker containers for databases, source code,
and binary artifact repositories. ONTAP RESTful APIs provide the flexibility to provision, load-balance,
and scale workloads in a single cluster or across clusters in multisite environments.
As illustrated in Figure 2, Docker Swarm manages resiliency for Docker containers by creating a multihost
cluster across virtual or physical machines. In the CI process, the sequence is as follows:
1. JFrog Artifactory is configured in Docker containers by using the docker-compose.yml
installation file. The installation provides private repositories for all the artifacts that are required
during the software build process and application packaging.
2. The data volumes for the Docker containers are then created by NetApp Docker Volume Plugin
before Artifactory installs the database and the repository.
3. GIT and Jenkins tools are set up and configured in Docker containers automatically. This occurs
when the respective data volumes are provisioned for source code version control, integrated
builds, and private builds through ONTAP APIs and mounted on Docker containers through
NetApp Docker Volume Plugin.
NetApp DockerVolume Plugin
CodeRepo1
CodeRepo2
CI Volume
for
Code Repo1
CI Volume
for
Code Repo2
Continuous Integrations (CI) Developer Workspaces
Integrated BuildsSource Code
Repository Manager Private Builds
Pre-Packaged
Source Code
& Pre-Build
ArtifactsArtifacts
& Builds
Universal BinaryRepository Manager
Snapshots Snapshots Snapshots
DatabasePre-Packaged
Source Code
& Pre-Build
Artifacts
Figure 2) Continuous integration workflow with JFrog Artifactory.
5 JFrog Artifactory on ONTAP 9 Using SnapCenter © 2017 NetApp, Inc. All rights reserved. © 2016 NetApp, Inc. All rights reserved.
The Artifactory plug-in for Jenkins allows a developer to pull prebuilt artifacts created during code
compilation and push an updated software build in the Artifactory repository.
The entire CI process with GIT, Jenkins, and JFrog Artifactory on ONTAP using APIs and Docker is
automated by NetApp.
Application-consistent checkpoints, or snapshots, for JFrog Artifactory are created for the following
reasons:
• Data can be backed up to a secondary storage tier like tape or replicated to a disaster recovery
site for protection.
• Data can be recovered and restored quickly if the database or the checksums in the Artifactory
volume become corrupted. if there is data corruption with the database or the checksums in the
Artifactory volume.
• An instant copy of the database can be spun up for testing patches or database refactoring
before new changes are pushed to production.
Developers can perform all of these data manageability functions by using SnapCenter 2.0 from a single
management pane. NetApp and JFrog have jointly worked to provide a seamless experience, allowing
developers to create application-consistent checkpoints from Artifactory whenever a change occurs in an
Artifactory data volume. Developers can create these checkpoints without the aid of IT administrators or
DevOps engineers.
3 SnapCenter 2.0 and Artifactory 5.1.4
NetApp SnapCenter software is a unified, scalable system for application-consistent data protection and
clone management. SnapCenter simplifies backup, restore, and clone lifecycle management with
application-integrated workflows. Using storage-based data management, SnapCenter increases
performance and availability and reduces testing and development times.
MySQL is a popular database choice because the LAMP (Linux, Apache, MySQL, PHP/Perl/Python) stack is one of the most popular open-source web-based platforms. MySQL is commonly used in mobile and web-based application development and deployment. Artifactory can connect to existing databases such as Oracle or SQL Server. In this report, a MySQL database is used for Artifactory. The code in this report creates artdb as one of
the databases in MySQL. SnapCenter 2.0 supports custom plug-ins for databases such as MySQL and DB2. More plug-ins will be available for NoSQL databases in the future.
SnapCenter 2.0 is installed on a Windows node, whereas JFrog Artifactory 5.1.4 components are
installed and configured on a Linux node. As shown in Figure 3 The Linux node is also configured with
the Docker engine and NetApp Docker Volume Plugin. The database component of Artifactory could be
Oracle, MySQL, or SQL for the artdb database store and can manage metadata of all the artifacts and
builds.
When docker-compose.yml is executed to install Artifactory, it not only installs Artifactory in the
Docker container, but it also creates persistent data volumes on ONTAP 9 and mounts them on the
containers by using NetApp Docker Volume Plugin. The installation process also creates artdb in
MySQL as a new database. The contents of the docker-compose.yml file are listed in the Appendix.
6 JFrog Artifactory on ONTAP 9 Using SnapCenter © 2017 NetApp, Inc. All rights reserved. © 2016 NetApp, Inc. All rights reserved.
Figure 3) SnapCenter 2.0 integration with JFrog Artifactory.
The MySQL database is set up on a physical machine for this validation. It can also run on a virtual
machine (VM). The following code shows Artifactory connecting with the MySQL database and lists the
artdb database after the installation.
BuildArtifacts
Snapshots
Database
Backup to Tape
Repository
Consistency Group
Snapshots
MySQL Agent
MySQL
SnapCenter2.0
NetApp DockerVolume Plugin
Dev/Test
Instantaneous Copies of DB Disaster Recovery
to Cloud
[root@linux-node6 ~]# mysql -u artifactory -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 1944
Server version: 5.7.17 MySQL Community Server (GPL)
Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights
reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input
statement.
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| artdb |
+--------------------+
2 rows in set (0.01 sec)
7 JFrog Artifactory on ONTAP 9 Using SnapCenter © 2017 NetApp, Inc. All rights reserved. © 2016 NetApp, Inc. All rights reserved.
To take an application-consistent snapshot, SnapCenter must communicate with the database.
SnapCenter identifies the MySQL plug-in during the host discovery phase. Figure 4 shows SnapCenter
remotely installing the agent for MySQL in the database location. When the agent is in a “running” state,
SnapCenter 2.0 is ready to create snapshots.
As part of the resource configuration process, the DevOps admin creates a MySQL database resource
that contains MySQL artdb details, the storage, and the Artifactory volumes.
Figure 4) SnapCenter 2.0 and the MySQL plug-in.
Figure 5) List of Artifactory volumes configured in SnapCenter 2.0.
8 JFrog Artifactory on ONTAP 9 Using SnapCenter © 2017 NetApp, Inc. All rights reserved. © 2016 NetApp, Inc. All rights reserved.
Next, a MySQL policy is created to specify the backups and the schedule. To provide the backup
replication details in the policy, a resource group with the MySQL resource is created and associate the
MySQL policy to that group.
Next, a MySQL policy is created to specify the backups and the schedule. The backup replication details
also have to be provided in the policy. Create a resource group with the MySQL resource and associate
the MySQL policy to that group. If the group complies with the business requirements, the resource
group is configured appropriately
The MySQL and Artifactory repository volumes are configured as a consistency group in SnapCenter 2.0,
as shown in Figure 6. With the help of the MySQL agent, SnapCenter 2.0 flushes all the logs and memory
buffers into the disk to take an application-consistent snapshot.
In Figure 6, a consistency group is enabled for both the Artifactory data volumes that were listed in Figure
5: mysql_vol and root_artdata. When a snapshot is triggered by SnapCenter 2.0, two snapshots are
created instantly for both the Artifactory volumes.
To prevent data loss, Artifactory does not update the metadata information in artdb until new or modified
checksums are written to the binary repository. Whenever SnapCenter 2.0 takes a snapshot, the
The artifactory.json file consists of the SnapCenter 2.0 API calls for a nonexpiring authorization
token followed by resource group and resource policy settings. For details about artifactory.json,
see the Appendix.
Here is how to create an application consistent snapshot from the Artifactory command line interface
(CLI).
Figure 6) A consistency group is enabled from SnapCenter 2.0 for Artifactory volumes.
9 JFrog Artifactory on ONTAP 9 Using SnapCenter © 2017 NetApp, Inc. All rights reserved. © 2016 NetApp, Inc. All rights reserved.
After the developer executes the curl command from the CLI of the Artifactory node, job ID 601 is
initiated. Figure 7 shows that SnapCenter 2.0 received the request to create an application-consistent
snapshot with job ID 601, and created snapshots for both the Artifactory data volumes.
Figure 7
Figure 8 lists the snapshots of the Artifactory data volumes on ONTAP 9 where the data volumes
physically reside.
4 Summary
[root@linux-node6 ~]# curl -u admin:password -H "Content-Type:
application/json" -X POST -d @artifactory.json
http://10.192.39.26:8081/artifactory/api/plugins/execute/snapCenter
[Backup:null, Context:null, DisplayCount:0, Job:[ApisJobKey:null,
Description:null, EndTime:null, Error:null, EventId:0, Id:601,
IsCancellable:false, IsCompleted:false, IsRestartable:false, IsScheduled:false,
IsVisible:true, JobTypeId:null, Name:Backup of Resource Group 'ART_RG' with
policy 'ART_Policy', ObjectId:0, Owner:null, ParentJobID:0,
PercentageCompleted:0, PluginCode:0, PluginName:null, Priority:0,
StartTime:2017-05-04T10:24:29.3567493-07:00, Status:5, Tasks:[]],
Result:[ErrorRecords:[], _errorCode:0, _message:], Results:[], TotalCount:0]
Figure 7) SnapCenter 2.0 creates a snapshot with job ID 601.
Figure 8) Snapshot listing of the Artifactory volumes in NetApp OnCommand® System Manager.
10 JFrog Artifactory on ONTAP 9 Using SnapCenter © 2017 NetApp, Inc. All rights reserved. © 2016 NetApp, Inc. All rights reserved.
The NetApp and JFrog joint integration with Artifactory and SnapCenter 2.0 allows developers to take
application-consistent snapshots on demand whenever they push modified or new artifacts and builds to
the repository. The developer has more control over the data (including the artifacts themselves), which is
available in a consistent state during the application development and deployment process. The
Artifactory plug-in gives developers more autonomy from the operations team, thus reducing time-
dependent communication challenges.
There is a one-time setup and configuration for SnapCenter 2.0 and Artifactory data volumes. After
SnapCenter 2.0 is implemented with the right resource group and resource policy settings, the developer
can use the Artifactory plug-in to make RESTful API calls to SnapCenter to take application-consistent
snapshots. The Artifactory plug-in is available in GitHub at https://github.com/JFrogDev/artifactory-user-
plugins/tree/master/netApp/snapCenter.
5 Conclusion
The integration of Artifactory and SnapCenter 2.0 not only gives developers more control over the data
generated during the CI process but also protects the data by enabling consistent checkpoint snapshots.
These snapshots are taken on the data volumes that provide persistent storage to Artifactory running on
Docker containers, and are instantaneous, thin provisioned, and highly efficient.
When SnapCenter 2.0 creates the thin-provisioned snapshots, developers can clone the data instantly to
test the applications during user acceptance validation and staging. Data can be replicated for disaster
recovery to a remote site or to public hypervisor cloud environments such as Amazon Web Services
(AWS) or Microsoft Azure. During development, developers can use Artifactory as the binary repository
manager on the premises; during deployment, they can use Artifactory in hyperscaler clouds like AWS.
Because SnapCenter and Artifactory integration enables a Data Fabric environment, application and
asset owners can move data to public hyperscaler clouds without vendor lock-in.
11 JFrog Artifactory on ONTAP 9 Using SnapCenter © 2017 NetApp, Inc. All rights reserved. © 2016 NetApp, Inc. All rights reserved.
References
NetApp SnapCenter datasheet
Appendix
The following shows the content of the Artifactory plug-in, artifactory.json.
JFrog Artifactory installation in a Docker container mounts persistent data volumes by using NetApp
Docker Volume Plugin. The following shows the content of the docker-compose.yml file.
# Version 2 without namespacing. Required env variables :
# ART_LICENSES
# ART_LOGIN (optional, default=admin)
# ART_PASSWORD (optional, default=password)
#
version: '2'
services:
artifactory-node1:
image: jfrog-docker-reg2.bintray.io/jfrog/artifactory-ha-primary:5.1.4
ports:
- 8081:8081
- 10017:10017
volumes:
- ~/data/artifactory/node1/etc:/var/opt/jfrog/artifactory/etc
- artdata:/var/opt/jfrog/artifactory/data
- ~/data/artifactory/node1/logs:/var/opt/jfrog/artifactory/logs
environment:
- DB_TYPE=mysql
- DB_HOST=10.192.39.26
- DB_PORT=3306
- HA_IS_PRIMARY=true
- HA_MEMBERSHIP_PORT=10017
[root@linux-node6 ~]# cat artifactory.json
{
"url": "https://10.192.39.22:8146",
"token": "",
"policy": "ART_Policy",
"resourcegroups": "ART_RG"
}
12 JFrog Artifactory on ONTAP 9 Using SnapCenter © 2017 NetApp, Inc. All rights reserved. © 2016 NetApp, Inc. All rights reserved.
- DB_USER=artifactory
- DB_PASSWORD=Netapp1!
- ART_LICENSES=" "
restart: always
artifactory-node2:
image: jfrog-docker-reg2.bintray.io/jfrog/artifactory-ha-primary:5.1.4
ports:
- 8082:8081
- 10018:10017
volumes:
- ~/data/artifactory/node2/etc:/var/opt/jfrog/artifactory/etc
- artdata:/var/opt/jfrog/artifactory/data
- ~/data/artifactory/node2/logs:/var/opt/jfrog/artifactory/logs
environment:
- DB_TYPE=mysql
- ART_PRIMARY_BASE_URL=http://artifactory-node1:8081/artifactory
- DB_HOST=10.192.39.26
- DB_PORT=3306
- HA_IS_PRIMARY=false
- HA_MEMBERSHIP_PORT=10017
- DB_USER=artifactory
- DB_PASSWORD=Netapp1!
- ART_LICENSES=" "
depends_on:
- artifactory-node1
restart: always
nginx:
image: jfrog-docker-reg2.bintray.io/jfrog/nginx-art:5.1.4
ports:
- "80:80"
- "443:443"
- "5000-5010:5000-5010"
environment:
- ART_PRIMARY_BASE_URL=http://artifactory-node1:8081/artifactory
volumes:
artdata:
driver: netapp
driver_opts:
snapshotDir: "false"
13 JFrog Artifactory on ONTAP 9 Using SnapCenter © 2017 NetApp, Inc. All rights reserved. © 2016 NetApp, Inc. All rights reserved.
Refer to the Interoperability Matrix Tool (IMT) on the NetApp Support site to validate that the exact
product and feature versions described in this document are supported for your specific environment. The
NetApp IMT defines the product components and versions that can be used to construct configurations
that are supported by NetApp. Specific results depend on each customer’s installation in accordance with
published specifications.
Copyright Information
Copyright © 2017 NetApp, Inc. All rights reserved. Printed in the U.S. No part of this document covered by copyright may be reproduced in any form or by any means—graphic, electronic, or mechanical, including photocopying, recording, taping, or storage in an electronic retrieval system—without prior written permission of the copyright owner.
Software derived from copyrighted NetApp material is subject to the following license and disclaimer:
THIS SOFTWARE IS PROVIDED BY NETAPP “AS IS” AND WITHOUT ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, WHICH ARE HEREBY DISCLAIMED. IN NO EVENT SHALL NETAPP BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
NetApp reserves the right to change any products described herein at any time, and without notice. NetApp assumes no responsibility or liability arising from the use of products described herein, except as expressly agreed to in writing by NetApp. The use or purchase of this product does not convey a license under any patent rights, trademark rights, or any other intellectual property rights of NetApp.
The product described in this manual may be protected by one or more U.S. patents, foreign patents, or pending applications.
RESTRICTED RIGHTS LEGEND: Use, duplication, or disclosure by the government is subject to restrictions as set forth in subparagraph (c)(1)(ii) of the Rights in Technical Data and Computer Software clause at DFARS 252.277-7103 (October 1988) and FAR 52-227-19 (June 1987).
Trademark Information
NETAPP, the NETAPP logo, and the marks listed at http://www.netapp.com/TM are trademarks of NetApp, Inc. Other company and product names may be trademarks of their respective owners.