2.0.5.7299 - Released on 21st November 2024

Fixes

• Fixed an issue when using desired size in conjunction with static data tables.

2.0.4.7180 - Released on 11th November 2024

Features

• Improved logging.

2.0.3.7037 - Released on 29 October 2024

Fixes

• Fixed an issue with self-referencing foreign keys.

2.0.2.6967 - Released on 23 October 2024

Fixes

• Fixed an issue where filter clauses can't begin with a 'WHERE'

2.0.1.6888 - Released on 21 October 2024

Features

• Renamed the command line executable from subsetter to rgsubset.
• Renamed the configuration file to options file.
• Moved some command line flags affecting the behaviour of subsetting to the options file.
• Moved static data tables to their own section instead of a flag on starting tables.
• Improved error logging when validating a subset.
• Added the ability to specify target size using percentages.
• Added a default target size of 10% (up to a maximum of 1GB) when running subsetting without specifying any target size or starting tables.

1.1.22.6378 - Released on 23 September 2024

Features

• Improved the output when source and target database have different schemas.
• Allowed subsetting to work on source tables with no rows.
• Reduced the working space required in the source database while subsetting.
• Added the ability (SQL Server only) to use temporary tables as a working space rather than tables in the source database. This can be configured using the UseTemporaryTables configuration option.

1.1.21.6082 - Released on 30 August 2024

Feature

• Internal library updates

Fixes

• Fixed an issue where disabling triggers failed due to unquoted trigger names
• Fixed an issue where schema validation failed due to relationships

1.1.20.5998 - Released on 21 August 2024

Fixes

• Logged the tables and relationships involved in a schema mismatch.

1.1.19.5974 - Released on 19 August 2024

Features

• Performance improvement.

Fixes

1.1.18.5918 - Released on 13 August 2024

Features

• Triggers on the target database are now automatically disabled while subsetting.

Fixes

• Fixed an issue where subsetter ignored values passed into the --include-tables-row-threshold  flag

1.1.17.5805 - Released on 5 August 2024

Features

• Added the ability to define excluded tables in the configuration file.

Fixes

• Made the auto detection of static data tables more robust.

1.1.16.5647 - Released on 26 July 2024

Fixes

• Fixed an issue with not being able to pass the --include-tables-row-threshold parameter on the command line.

1.1.15.5636 - Released on 26 July 2024

Fixes

• Fixed an issue where subsetting sometimes fails when working on tables that have computed columns.

1.1.14.5531 - Released on 16 July 2024

Features

• Added the ability to define starting tables as static data tables.
• Automatically included tables without outgoing relationships based on a row threshold.
• Added the ability to configure the row threshold for automatic table inclusion.

Fixes

• Fixed an issue where the subsetter tries to break relationships for cycles that are not in the final subset.
• Improved the way table size is measured for SQL Server to make the target size feature more accurate.
• Fixed an issue when trying to subset a table using more than one inward relationship.

1.1.13.5255 - Released on 25 June 2024

Features

• Added support for subsetting with cyclic relationships.

Fixes

• Added more detailed logging when validation errors occur.

1.1.12.0 - Released on 7 June 2024

Fixes

• Internal library updates.

1.1.11.0 - Released on 23 May 2024

Features

• Replaced the in-product date-based timebomb with a Redgate license check.
• Logged the tables involved when subsetting cannot proceed due to cyclic references.

Fixes

• Fixed an error when subsetting a database with a computed column with a unique constraint.

1.1.10.0 - Released on 14 May 2024

Fixes

• Improved reporting of errors for more useful error messages.

1.1.9.0 - Released on 7 May 2024

Warnings

• Breaking change: Multiple starting tables now subsets each starting table individually and merges the results.

Features

• Added support for subsetting multiple starting tables without primary keys for SQL Server, PostgreSQL, and Oracle.

Fixes

• Fixed --tartget-size  issue with empty parent tables.

1.1.8.0 - Released on 30 April 2024

Warnings

• Breaking change: CLI parameter --filter-table has been renamed to --starting-table to better reflect its purpose.

Features

• Performance improvements.

1.1.7.0 - Released on 27 March 2024

Warnings

• Breaking change: FilterTable is now named StartingTable in the YAML/JSON configuration file.

Features

• Added the ability to use multiple starting tables.

Fixes

• Fixed an issue where log files were being capped at 20MB.

1.1.6.0 - Released on 8 March 2024

Fixes

• Fixed issue with foreign keys with the same name across schemas.

1.1.5.0 - Released on 29 February 2024

Features

• Improved support for partitions in PostgreSQL.

1.1.2.0 - Released on 16 January 2024

Features

• Improved performance of filter table validation in source database (particularly impactful for large tables with many rows).
• Improved performance of row counting used by the target size feature, so that it will be much faster for large tables.

Fixes

• Improved validation for table identifiers such as the filter table.
• The subsetter is now able to accept filter clauses that start with the WHERE keyword.
• Fixed known high severity vulnerability CVE-2024-0056.

1.1.1.0 - Released on 20 December 2024

Features

• System version temporal tables in SQL Server are now supported. The subsetter copies the temporal tables but leaves the history tables empty.
• Improved command line validation for manual relationships in the configuration file.

1.1.0.0 - Released on 11 December 2024

Warnings

• A few known limitations in current and past versions:
  • The subsetter can't copy columns with User-Defined Types (UDTs) in Oracle.
  • The subsetter may fail on PostgreSQL databases if the source and target database are on machine with different architectures.

Features

• Significant performance improvements in all 4 supported database engines (SQL Server, Oracle, MySQL and PostgreSQL).
• Updated Redgate's Windows Authenticode digital certificate signature in the CLI executable (previous one expired on 6th of December 2023).

1.0.0.0 - Released on 5 December 2023

Warnings

• First General Availability (GA) release.

Features

• Added support for manual relationships in the configuration file.
  • This allows the user to specify foreign key relationships that are not actually in the database design, but that should be used during subsetting.
  • Note that this version of the subsetter only carries out limited validation of this part of the configuration file, so users are recommended to take care when completing it.
• Added experimental support for Azure SQL Databases.
• Added an application icon for the subsetter CLI executable.

Fixes

• Fixed a bug that made subsetting fail with MySql when the target and source are on the same server but using different ports.
• The subsetter now only starts the row counting process when an output file is explicitly requested (--output-file), which means it will run faster in cases where the file is not requested (which is impactful in larger source databases).

0.8.0.0 - Released on 1 November 2023

Warnings

• Breaking change: CLI parameter --starting-table has been renamed to --filter-table to better reflect its purpose. This also impacts our YAML/JSON configuration file where input field StartingTable is now named FilterTable.
• Breaking change: CLI parameter --filter has been renamed to --filter-clause to better reflect its purpose. This also impacts our YAML/JSON configuration file where input field Filter is now named FilterClause.
• Breaking change: Exit codes have been updated to the following values:

• • Ok = 0
  • GenericFailure = 1
  • UnhandledException = 2
  • FailedInitialization = 3
  • CliInvokedIncorrectly = 4
  • InvalidConfiguration = 5
  • FailedTableExtraction = 6
  • FailedTablePreprocessing = 7
  • FailedUpdatingTargetDatabase = 8

Features

• Improved logging setup error checking during application initialization.
• Several improvements to logging and error reporting during subsetting, making the information more human friendly.
• We no longer ship the subsetter CLI as a Docker Linux Alpine image. See our installation guide for more information on our release binaries.
• The subsetter CLI executable is now digitally code signed using dotnet sign (and Microsoft Authenticode technology) in Windows and SHA-256 hash checksum files are provided with the Linux distributables.

Fixes

• The subsetter ensures all SQL connections are closed at the end of execution, regardless of the outcome.
• Fixed issue where subsetter would only cleanup the temporary tables generated in the source database and restore the constraints on the target database if the overall operation was successful. This is now done regardless of the outcome.
• Fixed issue where sometimes database structure validation could fail claiming that columns were in a different order in the source and target.
• Made more culture invariant by no longer assuming we are running in British UK compatible environmental settings.
• Subsetter no longer uses SQL Server stored procedure sp_MSforeachtable
• Fixed a bug related to extracting tables with names containing unbalanced brackets.
• Improved error message when the tool doesn't work because the database contains cyclic references.

0.7.0.0 - Released on 6 October 2023

Warnings

• The subsetter now fully supports MySql and Oracle, increasing the total number of supported database engines to 4.

Features

• Runtime execution issues and exceptions are now reported to Redgate for analysis.

Fixes

• The subsetter can now handle tables whose fully qualified names would be ambiguous if unquoted, e.g. If there is a schema called A with a table called B.C and a schema called A.B with a table called C that will now work fine.
• Fixed an issue that could cause a log line to still be written to disk when --file-log-level=None was provided.

0.6.0.0 - Released on 19 September 2023

Warnings

• Two command-line arguments have been removed. Scripts or workflows that pass these command-line arguments will now fail with an error. The arguments should be removed.
  • The command-line argument --target-to-source-connection-string has been removed. This optional argument allowed the provision of a second connection string for use in remote queries. The subsetter now uses streaming rather than remote queries, and only uses one connection string for each database.
  • The environment variable corresponding to this command-line argument has been removed: i.e. the variable REDGATE_SUBSET_TARGET_TO_SOURCE_CONNECTION_STRING. The environment variable will now be ignored.
  • The command-line argument --ole-db-provider has been removed. This optional argument was only valid for Sql Server. It allowed the user to specify the database driver to use with remote queries. The subsetter now uses streaming rather than remote queries.
• The subsetter now modifies the source database. It creates tables to hold intermediary data during subsetting. It deletes them at the end of subsetting. Users are strongly advised not to run the subsetter against a live production database. It is recommended to restore a backup, and run the subsetter against the restored database.

Features

• Now uses a streaming approach to transfer subset data to the target database. Previous versions of the subsetter used remote queries to link the target database to the source database.
  • The subsetter no longer uses remote queries, so there is no longer any requirement to enable them. Previously OPENROWSET had to be enabled for Sql Server, and DB Link had to be enabled for PostgreSql.

Fixes

• Used the correct page size for Sql Server which was slightly wrong, causing a small error in the target size feature.

0.5.1.0 - Released on 4 September 2023

Fixes

• Fixed issue where mixed case tables didn't work in --target-database-write-mode=Overwrite mode for PostgreSql.
• Fixed ArgumentException thrown when using a relative output file living in the same location as the subsetter executable (e.g. --output-file=subsetting.json).

0.5.0.0 - Released on 31 August 2023

Warnings

• Breaking change: Optional --mode parameter has been renamed to --target-database-write-mode to better reflect its purpose.
• Breaking change: JSON output file schema has changed. sourceDbSize has been renamed to sourceDbSizeInBytes and targetDbSize has been renamed to targetDbSizeInBytes.

Features

• Improved processing and validation of the --output-file CLI parameter:
  • Folder structure will be created if it doesn't exist (e.g. C:\A\B\output.json in Windows will create folders A and B if they don't exist).
  • If the JSON output file cannot be created for any reason, tool will fail early (instead of only at the end of the subsetting operation).
  • The subsetter will fail if the output file value corresponds to an existing folder.
  • A warning will be issued if the output file value doesn't have an extension or if the extension doesn't match a JSON (.json) one.

Fixes

• Fixed issue where tables in the source database with special characters, reserved keywords or spaces in their names would cause the subsetter to crash during execution (regardless of whether they were being used in the subset operation or not).
• Fixed issue where a slow connection to the source or target databases could lead to runtime execution timeouts and a failed subsetting operation.

0.4.1.0 - Released on 16 August 2023

Features

• Results of a subset run can now be output as a JSON file using --output-file <file path> CLI parameter, where <file path> is a relative or absolute path to a file.

0.4.0.0 - Released on 11 August 2023

Features

• There is now an experimental --target-size CLI option which allows you to specify the desired size of the subset in the format x(B|KB|MB|GB|TB) (e.g. --target-size=5GB) instead of setting a starting table and filter.
  • NOTE: While the program does its best to choose a sample of the source database that is close to the target size, it's not guaranteed to work well on all databases.

Fixes

• Fixed an issue where copyright and version information was missing from the win-64 subsetter CLI executable.

0.3.1.0 - Released on 17 July 2023

Features

• Introduced the --mode optional parameter. Supported values are Strict and Overwrite. In Strict mode the subsetter stops if the target tables are not empty. In Overwrite mode it truncates the target tables if they have any rows. Defaults to Strict.
• Anonymous telemetry events are now reported during execution of the subset.

Fixes

• The validation of the target database is now less strict: Only the tables in the target database that are going to receive the subset are validated.
• It's now possible to subset SQL Server databases that use varchar keys (in particular with the SQL_Latin1_General_CP1_CI_AS collation).

0.3.0.0 - Released on 29 June 2023

Warnings

• Breaking change: SQL Server only CLI parameter --provider has been renamed to --ole-db-provider to better reflect its purpose.
• Breaking change: CLI parameter --database-engine value Postgres has been renamed to PostgreSQL to match official database engine name (and to ensure consistency across Redgate's toolset). This also impacts our YAML/JSON configuration file.

Features

• Composite foreign keys are now supported.

• In SQL Server, the default behavior if a custom OLE DB provider is not given via the --ole-db-provider CLI parameter has been changed. Instead of using a single default MSOLEDBSQL and exiting if not registered in the target database, the tool now tries several default OLE DB providers in order (MSOLEDBSQL, MSOLEDBSQL19, SQLNCLI and SQLOLEDB) and only fails if none work in the target database.
• Related with the above, the subsetter will now suggest a course of action if no suitable OLE DB provider is found and will also link to our troubleshooting documentation in that scenario.
• Added several improvements to SQL Server connection string (source, target and target-to-source) CLI validations (e.g. usage of preferred field aliases, replacement of unsupported field values by compatible supported ones) to ensure compatibility with remote connections via OPENROWSET.

Fixes

• A warning will be printed to standard out if the CLI is given a custom OLE DB provider via --ole-db-provider for any database engine that is not SQL Server. The subset operation will still be executed.
• The subsetter will no longer try to subset views in PostgreSQL.

0.2.1.0 - Released on 14 June 2023

Features

• Improved support for self references. Previously if a table contains a foreign key to itself, that column would always be set to null. Now, that column will be set to null only if it refers to a row not in the subset. Note self reference columns will not affect which rows are included in the subset.
• Added a runtime warning that output subset may not contain the parents for the hierarchyid SQL Server data type.

Fixes

• Fixed an issue where the logging output location wasn't always being created if it didn't exist.
• Fixed an issue where some target database validation errors were being silently swallowed by the CLI.

0.2.0.0 - Released on 2 June 2023

Warnings

• Important - This version introduces two breaking changes:
  1. To our YAML/JSON configuration where the source and target connection strings have been removed from the configuration schema.
     • The source and target connection strings can now either be passed in the command line or stored in environment variables. Please check the CLI reference for more details.
  2. To our connection string parameters to append a -string suffix to follow Redgate's convention:
     • --source-connection becomes --source-connection-string.
     • --target-connection becomes --target-connection-string.
     • --target-to-source-connection becomes --target-to-source-connection-string.

Features

• Add initial support for PostgreSQL database engine.
• Several improvements to the subsetter CLI parameters:
  • Added new parameter --log-folder to override the default log folder location.
  • You can now disable file and console logging with the special None log level (e.g. --file-log-level=None).
  • Several enhancements to command line validation checks.

0.1.0.0 - Released on 11 May 2023

Features

• First internal release of the Redgate subsetter. Only SQL Server is supported.