Skip to content

Latest commit

 

History

History
519 lines (410 loc) · 18.6 KB

CHANGES.md

File metadata and controls

519 lines (410 loc) · 18.6 KB
Raw

Changelog

2.20

  • Fix another unnecessary index re-creation case.
  • Fix some compiler deprecations.

2.19

  • Fix unnecessary index re-creation when another unique index has been created at the 'from' end. Thanks @flux-ricky.

2.18

  • Fix startup on ARM Linux.

2.17

  • Wrap filter expressions in type-casts to match the column to ensure that they hash correctly.

2.16

  • Automatically create/replace PostgreSQL enum types that are used in the tables being synced.

2.15

  • Fix build for later Homebrew libpq installations.
  • Upgrade BLAKE3 to 1.3.3.

2.14

  • Fix compilation regression on M1 macs.

2.13

  • Fix compilation on ARM Linux.

2.12

  • Upgrade BLAKE3 to 1.2.0.
  • Upgrade yaml-cpp to 0.7.0.

2.11

  • Remove OpenSSL as a dependency.
  • Apple Silicon compilation fixes. Thanks @oobles.
  • Fix parsing of timestamp(n) with/without time zone on PostgreSQL.
  • Compatibility fixes to the test suite.

2.10

  • Support syncing from a MySQL database to a specific named schema on PostgreSQL, vice versa, or between two PostgreSQL schemas.

2.9

  • Don't throw errors when syncing MySQL -> MySQL and tinyint(1) columns have non-boolean values (though note MySQL has deprecated integer display widths for anything except boolean columns). tinyint(1) columns will however continue to be interpreted as boolean when syncing MySQL -> PostgreSQL.

2.8

  • When syncing from MySQL/MariaDB to PostgreSQL, add a suffix number to indexes to make them unique across the schema if there are any conflicts.

2.7

  • Create keys before dropping keys when possible, in case they're in use for an FKC.
  • Performance optimisation: allocate heap memory less for small objects in keys.

2.6

  • Add --version option. Thanks @proby.
  • Revert 2.0 change 'Optimise inserts on MySQL/MariaDB by turning off unique_key_checks while syncing', suspected as the cause of a replication problem user field report.

2.5

  • Support MySQL/MariaDB's nonstandard FLOAT(M,D) and DOUBLE(M,D) syntax.

2.4

  • Draft support for multiple schemas on PostgreSQL, including syncing tables that have the same name in multiple schemas, and creating them in the appropriate schema.
  • Fix resetting sequences on PostgreSQL tables that need quoting.

2.3

  • Support specifying the SSH port using --via servername:port syntax.
  • Fix incompatibility with v1 'from' ends when using multiple workers.
  • Work around compatibility problem when using the MySQL client library with MariaDB servers, causing problems determining schema defaults etc.

2.2

  • Fix regression in LONGBLOB/MEDIUMBLOB support for MySQL & MariaDB.

2.1

  • New default hash algorithm BLAKE3, giving a significant performance improvement, particularly on CPUs supporting AVX2 and above.
  • Update test suite to run on Ruby 2.7.0. Thanks @proby.

2.0

  • Change the default --commit option to --commit=often. You can still use --commit=success to get the old default behavior.
  • For PostgreSQL set session_replication_role to 'replica' at the "to" end (when connected as a superuser) to better avoid foreign key constraint errors.
  • Support tables with no primary key and no usable substitute unique key, clearing and reloading that table if necessary so that at least the other tables can still be efficiently synced.
  • Support MySQL spatial types. Thanks @Safranil.
  • Support PostGIS spatial types (geometry and geography).
  • Support syncing spatial data between MySQL and PostGIS, including binary format conversion.
  • Support enumerated column types (ENUM).
  • Support JSON column types on PostgreSQL and MySQL 8+, and the MariaDB equivalent (LONGTEXT with a check constraint) on 10.2+.
  • Support fractional seconds in time/date-time columns on MySQL 8+ and MariaDB 10.0+.
  • Support VARBINARY(n) and BINARY(n) types on MySQL and MariaDB.
  • Support syncing unsupported (non-portable) column types when the same type is accepted at each end (ie. when using the same database server at each end).
  • Support default expressions for MySQL 8+ and MariaDB 10.2+, and fix bug in existing PostgreSQL support.
  • Support generated columns (GENERATED BY DEFAULT ... STORED) on PostgreSQL 12+.
  • Support generated columns (GENERATED BY DEFAULT ... STORED and GENERATED BY DEFAULT ... VIRTUAL) on MySQL 5.7+ and MariaDB 10.2+.
  • Support identity columns (GENERATED BY DEFAULT AS IDENTITY and GENERATED ALWAYS AS IDENTITY) on PostgreSQL 10+.
  • Use identity columns (GENERATED BY DEFAULT AS IDENTITY) instead of old-style sequence columns when converting MySQL/MariaDB AUTO_INCREMENT columns on PostgreSQL 10+.
  • Fix support for creating old-style sequence columns when creating tables with quotes in their names on PostgreSQL before 10.
  • Fix parsing of integer column types on MySQL 8.0.19+.
  • Fix key extraction on postgresql when tables have foreign key contraints (closes #78). Thanks @hsenot.
  • Fix column and key extraction on postgresql when there is another table with the same name in a different schema.
  • Fix support for mapping PostgreSQL's character varying with no length specification to LONGTEXT on MySQL/MariaDB.
  • Fix setting a default and changing a column to be non-nullable at the same time on MySQL/MariaDB.
  • Optimise adding a non-nullable column on MySQL/MariaDB to avoid them performing an unnecessary table rewrite.
  • Optimise inserts on MySQL/MariaDB by turning off unique_key_checks while syncing.
  • Recognise PostgreSQL's treatment of DEFAULT NULL, which it considers slightly different to no default, so we don't try and alter the schema each time.
  • Don't choose keys containing columns with replace: filters as substitute primary keys; complain if attempting to replace: an explicit primary key.
  • Stop warning about unsupported database flags. We didn't warn about other column type conversions, so there was little value.
  • Minor efficiency improvements.

1.17

  • Support PostgreSQL's numeric types with no scale or precision specification.

1.16.1

  • Add missing test file to fix build.

1.16

  • Support quote characters in column and table names.

1.15

  • 15-30% performance improvement for syncing into empty databases or inserting any new rows at the end of the table.

1.14

  • When using --alter to create tables, don't add a primary key if the original table doesn't have one.
  • Point users to SCHEMA.md when they hit an unsupported column type error.
  • Fix syncing tables with composite primary keys.

1.13

  • Fix handling of MySQL tinyint(n) columns for n > 1 (as previously, we assume n = 1 is used for boolean values).
  • Small performance improvement for hashing PostgreSQL bytea columns.

1.12

  • Support postgresql column type character varying without a length specification.
  • Small performance improvement for XXHASH64 (--hash XXH64).

1.11

  • Avoid producing confusing Connection closed errors from the 'from' end when aborting the sync due to schema mismatches etc.
  • Make the schema mismatch statement dumps more copy-and-pasteable by adding a semicolon to each statement.

1.10

  • Share the hash-checking work on individual tables when there's no more unprocessed tables left to work on, resulting in a substantial improvement in multi-core concurrency when some tables take much longer to process than others.
  • Support bracketed IPv6 addresses when parsing URLs.

1.9

  • Fix FTBFS due to missing test file.

1.8

  • Don't be Christmassy when stdout is not a TTY. Thanks @aclemons.
  • Correctness fixes.

1.7

  • Support setting MySQL_INCLUDE_DIR, MySQL_LIBRARY_DIR, PostgreSQL_INCLUDE_DIR, and PostgreSQL_LIBRARY_DIR as cmake -D options.
  • Abort the build (by default) if no database client libraries are found, rather than producing a probably-useless build. See INSTALL.md for more.

1.6

  • Warn during the build if no database client libraries are found, rather than producing a probably-useless build. This will become a fatal build error in the next version.
  • Write out Boost as a dependency.
  • Always use the vendored-in version of yaml-cpp instead of using PkgConfig to look for it on the system; previously these were optional "dependencies".
  • Add CentOS 7 install guide. Thanks @XavRsl.
  • Better error messages if the appropriate binaries are not found or if an SSH attempt fails.
  • Work around settings initialization regression in MySQL 8.0.
  • Remove some code that provided compatibility with versions before 1.0.

1.5

  • Fix error handling when there's an unknown column type, should be reported to the user gracefully.
  • Update CMakefile to find newer PostgreSQL client library versions to compile against.

1.4

  • Support for PostgreSQL's uuid type.

1.3

  • Fix support for timestamp field CURRENT_TIMESTAMP default functions on later versions of MariaDB.
  • Fix schema alteration support for removing a column that's in the primary key on later versions of MariaDB.

1.2

  • When using --via make an single initial SSH connection to give users a chance to accept host key prompts, and to give simpler error messages. Thanks @dylanmckay.
  • Documentation improvements and forward compatibility improvements.

1.1

  • Ignore views on MySQL, as for PostgreSQL. Thanks @Safranil.
  • Fix crash when running the sub-commands manually. Thanks @dylanmckay.
  • Usage example documentation fix. Thanks @pawelgradziel.
  • Minor efficiency tweaks. Compiler must now support C++14.

1.0

  • Behavior change: when using --via and the --filter option, the filter file is read locally rather than at the --via end.

0.99.2

Third release candidate for 1.0

  • Only --debug logging will output timestamps, not --verbose
  • Forward compatibility with forthcoming behavior changes to filtering
  • Forward compatibility with later msgpack versions and with setting up the test suite with later versions of mysql

0.99.1

Second release candidate for 1.0

  • Fix algorithm problem in a special case involving unique keys and the last row of the table being replaced by a later one
  • Tune some block size numbers

0.99

Release candidate for 1.0

  • Rewrite protocol and sync algorithm to hash concurrently at the two ends, and pipeline commands where possible.

0.60

  • Don't complain about unsupported column types on ignored tables or tables that are going to be dropped anyway.
  • Reduce max block size in order to reduce rework when mismatches with certain patterns are detected.
  • Drop support for the protocol used by version 0.45 and earlier.

0.59

  • Initial support for MySQL's special timestamp type and for DEFAULT CURRENT_TIMESTAMP and ON UPDATE CURRENT_TIMESTAMP behaviors on timestamp and datetime columns.
  • Support PostgreSQL's time with time zone and timestamp with time zone column types.
  • Fix support for PostgreSQL's default expressions.

0.58

0.57

  • Change default SSH cipher to aes256-ctr for compatibility with more platforms. Thanks @soundasleep and @aclemons.
  • Don't complain about tables that are missing a primary key (or suitable unique key) if they are ignored.
  • Stop printing debugging backtraces on SQL errors.
  • Document schema support.

0.56

  • Fixes to compilation on OpenBSD and FreeBSD. Thanks @aclemons.

0.55

  • Interpret postgres:// URLs to mean postgresql:// for compatibility. Thanks @aclemons.

0.54

  • Add an option to change the SSH cipher, and default to aes-256-gcm for compatibility with OpenSSH 7.2 and better speed. Thanks @normanv.

0.53

  • Look for MariaDB client libraries in mariadb-specific directories as well for compatibility with Ubuntu 16.04. Thanks @normanv.
  • Declare license (MIT).

0.52

  • Look for MariaDB client libraries as well.
  • Enhanced readme.

0.51

  • Add --progress option to print progress dots, which used to come on automatically with --verbose mode, not always useful in scripts.
  • Add progress dots during the long series of inserts at the end of a table.

0.50

  • Add --structure-only option, usually used with --alter to populate a database's schema. Thanks @dylanmckay.

0.49

  • Avoid PostgreSQL 9.3 syntax to add support for 9.2 (and partial support for earlier versions).

0.48

  • Work around CMake not finding PostgreSQL unless the server development header files are present.

0.47

  • Improve error messages from exceptions.
  • Start converting the endpoints to take their startup info from the environment rather than the command line.

0.46

  • Add a simple progress meter with --verbose. Thanks @soundasleep.
  • Implement optional xxHash64 support.
  • Add an option to override the path the 'from' binaries are in when using --via.

0.45

  • Fix pthreads link problems on newer Linux distros.

0.44

  • Fix ks_mysql or ks_postgresql not being found on some installs because PATH was not used in all cases.

0.43

  • Fix --debug listing arguments it wouldn't actually use if --via was not given.
  • Fixes URL examples in usage documentation.

0.42

  • Limit the maximum block size for a single hash command to reduce rework.
  • Optimisations from testing of the 0.40 and 0.41 changes.

0.41

  • Fix regression in 0.40 where a DELETE query could be run while an unbufferred SELECT query was run on MySQL.

0.40

  • Better algorithm for applying row changes that avoids using large amounts of memory when the 'from' end has deleted a large set of rows.
  • Default long_query_time to a higher value on MySQL to reduce annoying noise in logs.

0.39

  • Better forward compatibility with future versions that may add new entries to the database schema definitions.
  • Look for PostgreSQL version 9.5 too, even if not included in your platform's cmake module.
  • Fix various compiler warnings reported by users.

0.38

  • Remove unnecessary OpenSSL dependency on OS X (we use Common Crypto since 0.2).

0.37

  • Build configuration changes to fix using a system-wide copy of yaml-cpp. Thanks @pvenegas.

0.36

  • Don't attempt to add non-nullable columns or make nullable columns non-nullable if there are any unique keys defined on the columns, as this will inevitably hit duplicate key errors.
  • Fix MySQL 5.6.5+ compatibility. Thanks @bagedevimo.

0.35

  • Add new columns (if --alter is used) to the end of the columns list without recreating the table.

0.34

  • Replace the --partial and --rollback-after option with a more general --commit option and add options to commit after each table or insert batch. --commit often is useful to avoid large rollback segments on MySQL.
  • Fix a potential invalid memory access in the case where table columns don't match.
  • Boost's program_options library is no longer a dependency. Boost is still used at compile time.

0.32

  • Send quit commands to the 'from' endpoints as soon as possible.

0.31

  • Change column nullability (if --alter is used) without recreating the table.

0.30

  • Look for PostgreSQL version 9.4 too, even if not included in your platform's cmake module.
  • Fix dropping multiple columns on a table in one pass.
  • Debugging: comment briefly on what part of tables doesn't match.

0.29

  • Debugging: dump the worker arguments if run with --debug.

0.28

  • Drop extra columns (if --alter is used) without recreating the table.

0.26

  • Fix dropping defaults on non-nullable columns on MySQL.

0.25 - Christmas edition

  • Reset sequences after syncing table rows on PostgreSQL.

0.24

  • Implement bytea encoding for PostgreSQL.

0.23

  • Implement auto-creation of PostgreSQL sequences for serial columns.
  • Numerous fixes to PostgreSQL table creation and cross-compatibility with MySQL.

0.22

  • Set or clear column defaults (if --alter is used) without recreating the entire table.
  • Add missing include to fix GCC compilation errors.

0.21

  • Add or remove keys (if --alter is used) without recreating the entire table.

0.20

  • Documentation. Moar documentation.

0.19

  • Automatically figure out what tables need to be restructured to make the database schema match.
  • Implement the --alter option to automatically apply them.

0.18

  • Protocol schema additions to prepare for 0.19: extract and check whether columns are auto_increment/serial.

0.17

  • Give feedback to the user if protocol versions are incompatible
  • Fix no default on boolean columns getting parsed as default false on MySQL

0.16

  • Docs now tell you to make install rather than just make
  • Internal refactoring

0.15

  • Extract and check default values on columns
  • Fix a redefinition warning on yosemite's version of clang
  • Parse tinyint(n != 1) and mediumint MySQL column definitions for completeness
  • Give more helpful output if tests fail due to an EOFError while reading a command
  • Hopefully fix intermittent failures due to stderr output from the snapshot import/export test
  • Use msgpack 0.5.9 gem for Ruby tests

0.14

  • Use YamlCPP from the system if already present and found by pkgconfig. Thanks @proglottis.

0.13

  • Fix URL decode calls, need to be from class scope. Thanks @pvenegas for patch.

0.12

  • URL-decode the parts of the database URLs before using them.

0.11

  • Implement --set-from-variables and --set-to-variables options.

0.10

  • When sending rows, limit the queries to 10000 rows at once, to reduce slow query warnings and buffering.

0.9

  • Substantially reduce to_descriptor_list_start, as it turns out the OS X default getdtablesize() is a mere 256.

0.8

  • Fix filter_file argument getting trampled, use strings everywhere so the argvs have definitely been copied out before any status messages are written.

0.7

  • Set the current table name as the argv[1..] of the from endpoints so admins can see what they're doing.

0.6

  • Use NO_WRITE_TO_BINLOG in the MySQL FLUSH TABLES so downstream slaves aren't affected.

0.5

  • Use execvp instead of execv so ks can run from say /usr/local/bin without explicitly specifying the path.

0.4

  • Decode PostgreSQL bytea strings before hashing or sending in response to a rows command; decode booleans and pack as msgpack boolean type for portability.
  • Improvements to the protocol packing format and internal improvements to memory formats.

0.3

  • Add install recipe.
  • First version used in anger by Powershop.

0.2

  • GCC compatibility fixes.
  • Use Common Crypto instead of OpenSSL on OS X to fix deprecation warnings.
  • Use the database-specific quoting methods instead of a hardcoded standard version.
  • Use Bundler to install the gems needed to run the Ruby test suite.
  • Implement column type and length extraction/serialization/checking, supporting the basic set of mostly-portable types.
  • Numerous internal and protocol improvements.

0.1

  • First release for testing.