About Command Line Automation
Published 06 January 2020
The command-line is installed alongside the Data Masker application. To see this in action, see our samples.
Usage: DataMaskerCmdLine [options] [commands]
If no command is specified, the run command will be applied to the given PARFILE
.
Options:
PARFILE=<parfile>
,--parfile <parfile>
(REQUIRED forrun
)- Path to the parameter file containing the details of the masking job to run, see the parameter file section for more info.
--version
- Show version information.
--help
,-?
,-h
Show help and usage information for the command-line.
Commands:
column-template build-mapping-file
build
run
For example:
> "C:\Program Files\Red Gate\Data Masker for SQL Server 7\DataMaskerCmdLine.exe" PARFILE=C:\MaskingSets\SamplePARFILE.txt > "C:\Program Files\Red Gate\Data Masker for Oracle 6\DataMaskerCmdLine.exe" PARFILE="C:\Masking Sets\Sample PARFILE.txt"
The column-template build-mapping-file
Command
Builds a default mapping file using information from SQL Data Catalog. By default it will mask all columns tagged with 'Static Masking' under the category 'Treatment Intent', using templates defined in the category 'Masking Data Set'. If a tag in this category is not recognised as a template, the default mapping for that datatype is used instead (see table under options).
Usage: DataMaskerCmdLine column-template build-mapping-file [options]
Options:
Option | Required | Description |
---|---|---|
--catalog-uri <catalog-uri> | Yes | URI to connect to SQL DataCatalog e.g. http://[Your SQL Data Catalog Server FQDN]:15156. |
--api-key <api-key> | No | API key to connect to SQL DataCatalog, if not provided Windows authentication for the current user will be used. |
--instance <instance> | Yes | The fully-qualified name of the SQL Server instance. For a named instance, this should take the form 'fully-qualified-host-name\instance-name' (e.g. myserver.mydomain.com\myinstance). For the default instance on a machine, just the fully-qualified name of the machine will suffice (e.g. myserver.mydomain.com). |
--database <database> | Yes | Database name to build the mapping file for. |
--information-type-category <information-type-category> | No | Category in SQL DataCatalog which defines information type, we expect 1-1 mapping between the tag and the column template name. Default value 'Masking Data Set'. |
--sensitivity-tag <sensitivity-tag> | No | The tag to identify whether we should mask the column. Default value 'Static Masking'. |
--sensitivity-category <sensitivity-category> | No | The category that contains the sensitive tag. Default value 'Treatment Intent'. |
--mapping-file <mapping-file> | Yes | Location to save the generated mapping file. |
--log-directory <log-directory> | Yes | The location to save log files |
--help , -? , -h | No | Show help and usage information for the column-template build-mapping-file command. |
Default templates (see mapping file section for more information on templates) for each datatype (used when a tag in information-type-category
is not recognised as a template):
Datatype | Template |
---|---|
bigint | Integer |
date | Date |
datetime | Date |
datetimeoffset | Date |
datetime2 | Date |
decimal | Decimal |
float | Decimal |
int | Integer |
money | Decimal |
ntext | Long string |
numeric | Decimal |
real | Decimal |
smalldatetime | Date |
smallint | Integer |
text | Long string |
contains "char" , max precision (e.g. varchar(max)) | Long string |
contains "char", precision under 6 (e.g. varchar(5)) | Random alphanumeric string |
contains "char", precision 6 or over (e.g. varchar(20)) | Short string |
The build
Command
Prepares an empty masking-set file for the given connection, unless a mapping-file is provided which will populate the masking-set with default rules.
Usage: DataMaskerCmdLine build [authentication] [options]
Authentication:
using-windows-auth
- Use Windows authentication.
using-sql-auth
- Use SQL authentication.
Options:
Option | Required | Description |
---|---|---|
--masking-set-file <masking-set-file> | Yes | The location to save the new masking set to be interpreted by Data Masker. |
--log-directory <log-directory> | Yes | The location to save log files. |
--instance <instance> | Yes | The SQL Server instance. |
--database <database> | Yes | The database name. |
--username <username> | SQL auth only | The user name. |
--password <password> | SQL auth only | The password. |
-s , --schema <schema> | No | A schema that you would like to mask. To add multiple schemas, specify this option for each schema. If not specified, all schemas that contain tables will be added to the masking set. |
-p , --parfile <parfile> | No | The location to save a parameter file to run the generated masking set. |
--mapping-file <mapping-file> | No | The column template mapping file in JSON format, see the mapping file section for more info. |
| No | Show help and usage information for the build command. |
The run
Command
Run a masking set using the specified PARFILE
.
Usage: DataMaskerCmdLine run [options]
Options:
PARFILE=<parfile>
,--parfile <parfile>
(REQUIRED)- Path to the parameter file containing the details of the masking job to run, see the parameter file section for more info.
--help
,-?
,-h
- Show help and usage information for the
run
command.
- Show help and usage information for the
The Parameter File
The parameter file can specify the following options:
Parameter | Required? | Description |
---|---|---|
MASKINGSET | Required | The full path of the masking set to be run. |
LOGFILEDIR | Required | The full path to the directory where log files will be placed. |
DATASETSDIR | Required | The full path to the directory containing the datasets. Unless you have created your own directory, the datasets are installed alongside Data Masker in the following locations:
|
REPORTSDIR | Required | The full path to the directory where the reports should be exported. |
INTERIM_REPORTS | Optional | Either If |
LOGINSUB | Optional | Add a LOGINSUB line to the file for each controller in the masking set that needs to be changed to point to a different database (or with a different user account). For more information about login substitution see: |
DMSSUBVALUExx | Optional | SQL Server Only. Currently only available in legacy performance mode. Data Masker will replace any instance of these parameter names with their value in any SQL. The xx is a two-digit numeric value. See substitution values for more details. |
DMOSUBVALUExx | Optional | Oracle Only. Currently only available in legacy performance mode. Data Masker will replace any instance of these parameter names with their value in any user-specified SQL. The xx is a two-digit numeric value. See substitution values for more details. |
DMSSUBVALUExx
(SQL Server version) / DMOSUBVALUExx
(Oracle version)
Up to twenty replacement strings can be specified in the parameter file. These runtime substitution values are designed to permit generic SQL statements to be dynamically adjusted at runtime for specific targets during batch operations.
Note: The keywords for these replacement strings are DMSSUBVALUE00
to DMSSUBVALUE19
. If the text %DMSSUBVALUExx%
is used in a Command rule or in the Where Clause of a masking rule, the value associated with that keyword will be replaced at runtime before the rule is executed.
For example, if a Command rule included a statement like:
TRUNCATE TABLE %DMSSUBVALUE05%_TEMP;
and the parameter file contained a line stating DMSSUBVALUE05=TESTUSER
then the actual statement that would be executed would be:
TRUNCATE TABLE TESTUSER_TEMP;
The value TESTUSER
would be substituted for the %DMSSUBVALUE05%
text in the SQL statement immediately before the statement was executed.
Examples
SQL Server PARFILE
MASKINGSET=C:\Program Files\Red Gate\Data Masker for SQL Server 7\Demo.DMSMaskSet LOGFILEDIR=C:\TEMP\Data Masker\ DATASETSDIR=C:\Program Files\Red Gate\Data Masker for SQL Server 7\DataSets REPORTSDIR=C:\TEMP\Data Masker\ INTERIM_REPORTS=false LOGINSUB=placeholder-login@DEV-DB[DataWarehouse]||DataMasker/SecurePassword@STAGING\SQL2019[DataWarehouseMasked] -- Substitution values DMSSUBVALUE00=DM_INVOICE_LINE_HISTORY DMSSUBVALUE05=DM_INVOICE_LINE_HISTORY
Oracle PARFILE
MASKINGSET=C:\Program Files\Red Gate\Data Masker for Oracle 6\Demo.DMSMaskSet LOGFILEDIR=C:\TEMP\Data Masker\ DATASETSDIR=C:\Program Files\Red Gate\Data Masker for Oracle 6\DataSets REPORTSDIR=C:\TEMP\Data Masker\ INTERIM_REPORTS=false --LOGINSUB=DummyLogin@DummyServer[DummyDatabase]||DataMasker/DataMasker@DEMODB[DMTest] --LOGINSUB=XXXLogin@XXXServer[XXXDatabase]||DBSSourceUser/DBSSourceUser@N2KBUILD-PC\N2KTEST1[DBSSource] DMOSUBVALUE00=DM_INVOICE_LINE_HISTORY DMOSUBVALUE05=DM_INVOICE_LINE_HISTORY
The Mapping File
Maps columns in the database to pre-defined templates that the build command uses to create a masking set.
Currently available templates are:
Template Name | Description |
---|---|
Given name | Male + Female first names. |
Family name | Surnames. |
Full name | Male + Female full names. |
Title | Titles, e.g. Mrs, Mr |
Date of birth | Randomly offset dates within roughly two months in either direction. |
Email address | Generated email addresses, e.g. sus.syy@massa.com |
Street address | Generated street addresses, e.g. 272 Getchell St. |
City | These templates are correlated - entries on the same row will form a coherent address. Addresses are US addresses. |
State | |
ZIP code | |
County | |
Country | Countries. |
Full address | A complete address in the form: 123 Zoffer St. Lake Charles LA 70609 |
Debit/credit card number | Digits only VISA numbers. |
Debit/credit card expiry date | Card expiry date, e.g. 06/20 |
Bank account number | Bank account numbers - digits only, e.g. 304340718 |
Bank sort code | UK bank sort codes - digits only, e.g. 930184 |
SWIFT-BIC | SWIFT-BIC Codes. |
Phone number | North American telephone numbers - digits only, e.g. 5062699083 |
Gender | Genders. |
Nationality | Nationalities. |
Passport number | Passport numbers - digits only, e.g. 234001394 |
Social Security number | Unique social security numbers |
Occupation | Job titles. |
Organization name | Company names |
URL | Generated URLs, e.g. http://hexagonal.net |
IP address | IP addresses. |
MAC address | Mac addresses. |
Password | 12 character passwords - e.g. a94tgh32lk09 |
Driving license number | Driving license numbers - digits only, e.g 940021042 |
Photo | Photos represented in binary |
Vehicle registration number | UK vehicle registration numbers. |
Integer | Random integers between 1 and 100. |
Decimal | Random decimal numbers roughly in the range of -125 to 570 with 2 decimal places. |
Short string | Random dictionary words. |
Long string | Paragraphs of gibberish with max chars 1000, min chars 1 and max word length 10. |
Date | Masked identically to Date of birth template. |
Example mapping file:
{ "source": "SQL Data Catalog", "mappings": [ { "template": "Given name", "independentlyMasked": [ { "schema": "dbo", "table": "People", "column": "F_NAME" }, { "schema": "dbo", "table": "Employees", "column": "F_NAME" } ] }, { "template": "Family name", "independentlyMasked": [ { "schema": "dbo", "table": "People", "column": "S_NAME" }, { "schema": "dbo", "table": "Employees", "column": "S_NAME" } ] } ] }
Exit Codes
The command-line will return an exit code to indicate success or failure; if the run of the masking rules was successful, then this exit code will always be zero. Any non-zero exit code implies that an error occurred while opening the masking set or while running the masking rules. If a non-zero exit code is returned, the log file should always be examined to determine the error that occurred. The defined error codes are:
Error code | Description |
---|---|
200 - EXITCODE_RULERUN_ERROR | There was an error executing one or more masking rules. See the log file for more information. |
110 - EXITCODE_PARAMFILESUB_ERROR | There was an error processing the LOGINSUB substitution keyword in the parameter file. |
109 - EXITCODE_MASKSETOPEN_ERROR | There was an error opening the masking set. See the log file for more information. |
108 - EXITCODE_MASKSETMISSING_ERROR | The masking set specified in the parameter file could not be found. See the log file for more information. |
107 - EXITCODE_REPORTSDIR_ERROR | The specified report directory could not be found or there were permissions issues. See the log file for more information. |
106 - EXITCODE_DATASETBUILD_ERROR | There was an error reading the datasets. See the log file for more information. |
105 - EXITCODE_DATASETSDIR_ERROR | The specified dataset directory could not be found or there were permissions issues. See the log file for more information. |
104 - EXITCODE_LOGFILEDIR_ERROR | The specified log file directory could not be found or there were permissions issues. See the log file for more information. |
103 - EXITCODE_CMDLINEARGS_ERROR | There was an error processing the command line. See the log file for more information. |
102 - EXITCODE_LOGCREATE_ERROR | There was an error creating the log file. |
101 - EXITCODE_LOGINIT_ERROR | There was an error initializing the logging sub-system. |
100 - EXITCODE_UNSET | Unspecified error. See the log file for more information. |
99 - EXITCODE_UNHANDLED_EXCEPTION | An exception occurred. See the log file for more information. |
0 - EXITCODE_SUCCESS | The run of the masking set completed successfully. The log file and reports will have full details. |
If an error occurs there will be more detail in the log that is created in the LOGFILEDIR
directory, some information will be written to the console output.
If running from the standard command-line then `echo %errorlevel%` can output the exit code. For example:
C:\Program Files\Red Gate\Data Masker for SQL Server 7>DataMaskerCmdLine.exe PARFILE=C:\workspace\data-masking\DMV6\DMS_TestSets\DebugPARFILE.txt ... C:\Program Files\Red Gate\Data Masker for SQL Server 7>echo %ERRORLEVEL% 200