Redgate Clone makes it easy to create disposable copies of SQL Server, Oracle, PostgreSQL and MySQL databases for development and test. The database copies are hosted in a centralized environment so you don't need database servers installed locally.

You can create, save and destroy your databases as often as you like, even spinning up multiple copies for different branches and CI pipelines. It uses a command line interface, making it simple to script and automate in DevOps pipelines.

Redgate Clone is designed to scale to meet your requirements making it easy to get started with minimal resources and increase these as you expand usage. This section will guide you through setting up a first proof of concept (POC), for more detailed information on installation please see the installation guide.

Contents

Supported database engines

Redgate Clone currently supports the following database engines:

Installing Redgate Clone for the first time

Redgate Clone is installed on a central server that users connect to via a command-line interface (CLI). It's designed to be scalable, so you can start off small and increase resources as usage grows.  As a starting point, we recommend 16GB RAM and 100GB of storage. This will allow for a single data-image from a small backup and several data-containers to be created and run in parallel. You will need to increase RAM and storage as you start working with more data-images and data-containers, and larger backups.

Redgate Clone architecture

Technical prerequisites

Virtual Machine

Redgate Clone is installed on a single server which will need to be scaled depending on usage. The specification below is a good starting point and will enable you to get Redgate Clone up and running and create your first clones. As you start creating more clones and working with larger backups, you will first need to increase RAM and disk(s) size, and we advise planning ahead for this – information on scaling is provided at the end of this guide.

File share

To create database copies from backups you will need to store the backup files in an accessible shared folder, which can be either an NFS or SMB file share. Configuring a file share is covered in part 2 of the installation steps below, but you may want to learn about backup storage in advance.

Network & Firewall

The following ports should be available on the host.

Other

Installation steps

Redgate Clone is a two-part installation. The first stage configures the virtual machine, and the second stage completes the setup of Redgate Clone. You should allow 40-60 mins for installation to complete.

Part 1: Configuring the VM

1. Make sure you have a machine ready based on the prerequisites above. You can check this manually by following our documentation or run a prerequisites check script with the following command:

   curl -sSL https://rd.gt/3RL6ioU | sudo bash

2. To begin configuration run the following command in the VM:

   curl -sSL https://k8s.kurl.sh/cloning-capability-app | sudo bash

   1. When asked about disabling swap to continue, press ‘Y’

   2. You will need the URL and temporary password generated by the installer for the second stage, so note these down. For example:
      
      
      

   3. Once completed, reload the bash shell:

      bash -l

3. Check that you can access the Kubernetes cluster on the VM through kubectl:
   1. Running kubectl get pods should return a short list of pods. 

   2. If you get an error like: error loading config file "/etc/kubernetes/admin.conf": open /etc/kubernetes/admin.conf: permission denied you should unset the KUBECONFIG  file by running the following command:

      echo unset KUBECONFIG >> ~/.bash_profile
      bash -l

4. Finish configuring the VM by running the following commands:

   kubectl label nodes redgate.com/infrastructure-node=true --overwrite --all
   kubectl label nodes redgate.com/cloning-node=true --overwrite --all
   kubectl label nodes redgate.com/cloning-mssql-node=true --overwrite --all
   kubectl label nodes redgate.com/cloning-oracle-node=true --overwrite --all

Part 2: Complete setup 

You'll use the Redgate Clone Admin Console to complete the setup, using the URL and temporary password you recorded from the previous steps. The Admin Console is also where you'll be able to monitor, configure settings, and install updates for Redgate Clone.

Go to the Admin Console web UI ( http://<IPv4 address>:8800 ) and follow the steps for completing installation of Redgate Clone:

1. Install the latest version of Redgate Clone
   1. You will first be prompted to enter the password you recorded previously
   2. Next, you will need to upload the application license file. Please contact sales@red-gate.com if you haven't received a link to download your Redgate Clone license.
2. From the Admin Console click on the Config tab from the navigation bar.
   1. In the Authentication Settings section, set the rgclone Access Token
      1. Create a secure password for the access token that will be used to secure communications to the CLI. This is defined by you.
   2. In the Data Image and Data Container Storage Settings section, specify a host directory and name pattern for the directory where data images and data containers are going to be stored. For more information see data storage.
   3. Configure the Communication security section
      1. Set your DNS (as defined previously)

      2. Choose certificate source. You can use a self-signed certificate to get started quickly, or provide a TLS certificate to ensure all communications are secure

         If you chose to use a self signed certificate, it will be generated for you. In a subsequent step you will need to export the certificate for distribution to end users.

          

3. You can optionally setup Admin Console TLS to enforce secure connections.

Linking your file share

From the Admin Console click on the Config tab from the navigation bar, then go to the Backup Storage Settings section.

To create data images from backups you will need to store the backup files in an accessible shared folder, which can be either an NFS or SMB file share. The file share will be mounted into the application and the backup files will be read directly from there to create a data image.

You can learn how to configure NFS and SMB files shares by following these links:

• Setup an NFS file share

• Setup an SMB file share

You will be able to get started without a file share as Redgate Clone enables the creation of empty data images. An empty data image is a running instance with no content in it, which you can update through creating a data container.

Setting up OIDC (Optional)

From the Admin Console click on the Config tab from the navigation bar, then go to the OpenID Connect section.

If you want to use OIDC, check OIDC Enabled.To enable redgate Clone to use an OIDC-enabled identity provider, click on the Config tab from the navigation bar in the Admin Console and select OpenID Connect (OIDC) Settings from the menu.

You’ll need to enter a few values that you’ll find in your OIDC-enabled identity provider’s configuration page.

1. The OpenID provider configuration path.  This is the URI pointing to the identity provider’s /.well-known/openid-configuration metadata. (Where to find: Okta, Azure AD).
2. The client ID of your identity provider application (Where to find: Okta, Azure AD).
3. The audience for your access token, contained within the aud claim of the token. (Where to find: Okta, Azure AD).
4. Any additional scopes that may be required for an authentication request. This is optional dependent on your identity provider. Azure AD, for example, requires an additional API scope, which would then be added here. Okta has no such requirement.
5. The username claim name that represents a unique username in your identity provider. The value contained in this claim within the access token will be attached to a resource, and will appear in any CLI output that shows a specific user (e.g. the image owner). Defaults to preferred_username. (For more information about claims: Okta, Azure AD).

Learn more about authenticating with OpenID Connect (OIDC).

Exporting an auto-generated TLS certificate (Optional)

If you chose to use a self-signed certificate, all end users of Redgate Clone will need to have their machine trust that certificate.

To do this, first go onto the VM and extract the certificate, saving it to a file called rgclone.crt :

kubectl get secrets/redgate-platform-ca -n redgate-clone-app -o jsonpath='{.data.tls\.crt}' | base64 --decode > rgclone.crt

Then, to install the certificate on an end user machine, first copy the rgclone.crt file to it from the VM. Then run:

# Requires an elevated Powershell session (i.e. "Run as Administrator")
Import-Certificate -FilePath rgclone.crt -CertStoreLocation Cert:\LocalMachine\AuthRoot

cp ./rgclone.crt /usr/local/share/ca-certificates/rgclone.crt
sudo update-ca-certificates

cp ./rgclone.crt /etc/pki/ca-trust/source/anchors/rgclone.crt
sudo update-ca-trust

Connecting to Redgate Clone

Installing the CLI

You interact with Redgate Clone using our Command Line Interface (CLI)  rgclone, which is hosted directly in the server and can be downloaded via our Web API (through a download endpoint). This also means that installation and upgrades are basically identical as they require this download to your local machine.

Supported operating systems (OS):

• windows-amd64
• linux-amd64

Select the instructions for the OS that you are installing the CLI onto here:

Authenticate rgclone using the access token you configured in the Admin Console. 

Once you have the rgclone CLI installed and configured on your local machine you're ready to start creating copies of your database instances.

Key concepts

Data image

A data image is the static template or blueprint for clones. It's a complete, point-in-time copy of a database instance, and can be configured with or without databases. The data image is defined in a single yaml file, which contains all the instructions necessary to create it You can create data images from backups or create an empty image with no content in it.

Example 1: Define a data image that mirrors your production instance:

dev-myApp.yaml

sourceType: backup
name: dev
engine: mssql
version: 2022
backups:
  - path: my-backups/backup1.bak
  - path: my-backups/backup2.bak

Example 2: Define a data image that is a completely empty instance:

dev-newApp.yaml

sourceType: empty
name: dev
engine: postgres
version: 15

Creating a data image is as simple as running the following command referencing the yaml file:

rgclone create data-image -f ./dev-myApp.yaml

Common values for data image definition

Data source. String: sourceType

Database engine. String: engine

Database engine version. String: version

Learn more about writing data image definition yaml .

Learn more about working with data images. 

Data container

A data container is a running instance of a data image that behaves like a regular database server that you can connect to and work with. Data containers are virtual clones, so they're quick to create and require only a tiny fraction of disk, yet provide access to all databases and data in the original image. From a single data image you can create as many data containers as you want very quickly, all isolated from each other, allowing you and your team members to make changes without worrying about impacting anything else.

Creating a data container is as simple as running the following command referencing the data image:

rgclone create data-container --image dev

Learn more about working with data containers. 

Learn more about Redgate Clone commands.

Scaling Redgate Clone

The specifications listed above are designed to help you get started but will only enable you to work with a single data image and a few data containers. As a guide, you will need to consider the following as you increase your usage of Redgate Clone:

Scaling	More RAM	More storage
More data containers
More data images
Larger backups

Creating more data containers

As each data container is a running instance it requires RAM – the more clones (data containers) you want to run in parallel the more RAM you will need to provision. As Redgate Clone runs as a central service you'll need to factor in the number of users and/or pipeline runs you'll want connecting at the same time. 

The tables below shows the default CPU and RAM requirements per database type. If required, these can be configured within the Admin Console.

Please also see the following FAQs:

• How much RAM does a data image consume and for how long?
• How much RAM does a data container consume and for how long?

Working with larger backups

When working with larger backups, you'll need to adjust the size of the attached disk(s) to provide enough storage for data images and any changes you make to data containers. Only the changes you make to data containers are stored by if you have lot of data containers running and make many changes, this can add up – the key is to treat data containers as ephemeral and small changes and frequently refresh them.

As each data image will be roughly the same size as the original database, the file share must have sufficient disk space available to store them.

For example: you may want to keep a data-image of your current backup while also enabling a data-image to be created of the latest backup when taken. In this example, you will require disks totaling 2x the size of your backup for the data-images, plus additional space for data-containers you create and changes you make to them. In this example, for a 500GB backup you should provision disks totaling 1.5TB.

You can increase storage either vertically (increase the size of the disk(s)) or horizontally (add more disks), and choosing which option is best will depend on your requirements.

Learn how to add more disks.

More information

The following pages provide additional information on installing Redgate Clone, but are geared towards a production rollout:

• Installation requirements

• Installation pre-requisites checks

• Installing Redgate Clone