Release Notes Fivetran HVR Version 6.2.5/1 (solaris_10-x64-64bit) 2024-12-17 COMPATIBILITY FOR PLATFORM SOLARIS_10-X64-64BIT ----------------------------------------------- Fivetran HVR version 6.x is not compatible with the versions 5.x and 4.x. So, ensure that the HVR version is 6.x on the Hub machine and Agent machine. - Operating System: Solaris for Intel/AMD, version 10 and higher - Oracle: version 11.2, 12.1 and 12.2 - Ingres: version 10.2 - File replication: local, FTP/FTPS and SFTP - S3: supported - Google Cloud Storage: supported - Installation: Agent only, no Hub Server API VERSIONS SUPPORTED FOR REMOTE CLI ------------------------------------- /api/v0 /api/v6.0.5 /api/v6.0.5.1 /api/v6.0.5.2 /api/v6.1 /api/v6.1.0.3 /api/v6.1.0.4 /api/v6.1.0.5 /api/v6.1.0.6 /api/v6.1.0.7 /api/v6.1.0.11 /api/v6.1.0.15 /api/v6.1.0.26 /api/v6.1.0.36 /api/v6.1.5 /api/v6.1.5.2 /api/v6.2.5 /api/v6.2.6 HOSTED DATABASE SUPPORT POLICY ----------------------------- Often HVR supported source and target locations that can be installed on premises will be hosted by third parties such as cloud providers including (but not limited to) Alibaba, Amazon, Google, Microsoft and others. These hosted systems are generally supported by HVR as sources and targets insofar as they are compatible with their downloadable counterparts. The certification of compatibility is incumbent on the hosting provider, not HVR. Platforms have been separately documented where compatibility is not comparable and HVR has been able to implement interoperability code changes (for example, Amazon Aurora PostgreSQL). If a provider claims compatibility then they must offer the same SQL and API calls and responses that support the features as indicated in the HVR documentation for the corresponding location class requirements in the appropriate source and target sections. In cases where compatibility is not complete then support is not guaranteed. Any issue that may exhibit itself on the hosted platform must be reproducible on an equivalent downloadable version to be considered a bug in the HVR software. INSTALLING A FIVETRAN HVR UPGRADE ------------------------- Often all machines are upgraded at the same time. Current HVR version are fully compatible with two previous major versions; it is not necessary to upgrade all machines in a channel at once. Instead it is possible to only upgrade certain machines. E.g. only the hub machine, agent machine, capture (could be the hub or an agent), or the integrate machine (could be the hub or an agent). If this is done, it should be understood that each HVR release fixes some bugs and/or contains new features. Each bug fix or feature is only effective if the specific machine(s) indicated for it are upgraded. New features should not be used until all machines that are specified for that feature are upgraded, otherwise errors can occur. For example, if a new HVR release fixes an integrate bug, then that release must be installed on the machine(s) which do integrate. So if only the hub machine is upgraded, then there will be no benefit. To decide whether each machines needs to be upgraded, see the descriptions in the release notes below. Each description ends with a line says which machine must be upgraded for that specific bug fix or feature to be effective. LOG BASED CAPTURE SUPPORT ------------------------- - Ingres Log based capture is supported for all Ingres versions supported by HVR. No support for dual or partitioned DBMS log-files. No support for page-spanning rows. DDL replication (using action AdaptDDL) is not supported. - Oracle Log based capture is supported for all Oracle versions supported by HVR. Oracle RAC (cluster) and Oracle ASM are supported. For Oracle version 9, log-based capture is not supported for LOBs or tables without a primary key. Compressed tables are only supported for Oracle 11.2 and higher. Log based capture is supported from Data Guard standby database for Oracle 11 and higher. HVR can also capture from an database that was previously a Data Guard target. If HVR was capturing changes from one primary Oracle database and a role transition occur (so that a different Data Guard target becomes the new primary) then HVR can continue capturing from the new primary, including capturing any changes which occurred before the transition. This process is automatic, providing that the HVR location is connecting to Oracle in a way which 'follows the primary'. Log based capture from tables that are encrypted using Oracle TDE is supported for Oracle version 11 and higher. Both tables in an encrypted tablespace and tables with encrypted columns are supported. HVR supports software and hardware (HSM) wallets. If the wallet is not configured as auto-login (Oracle internal file cwallet.sso), the password for the software wallet or HSM needs to be set on the HVR Live Wallet port using the hvrlivewallet command. The software wallet can be in ASM or in a local filesystem. Capturing from Oracle TDE is not supported on HP-UX. Platforms that use OpenSSL version 3 do not support auto-login wallets for Oracle version 11. - SQL Server Log based capture is supported for SQL Server 2008, 2012, 2014, 2016, 2017, 2019 and 2022. Data Compression is supported. Capture from compressed backup transaction log files is supported. Capture from encrypted backup transaction log files is supported. Capture from memory optimized tables is not supported. Capture from tables with XML_COMPRESSION = ON is not supported. Capture of typed XML columns containing xsd:float and xsd:double values is not supported. Log based capture from databases that are encrypted using the SQL Server TDE is supported. The log based capture is supported only for the databases whose Database Encryption Key (DEK) is protected by Certificates. Log based capture is not supported from databases whose DEK is protected by Asymmetric Key. Column level encryption is not supported. Always Encrypted feature is not supported. - DB2 on Linux, Unix, Windows (LUW) Log based capture is supported for DB2 on Linux, Unix, Windows (LUW). The following table types are supported - regular tables, multidimensional clustering (MDC) tables, insert time clustering (ITC) tables, uncompressed tables, row compressed tables (both static and adaptive) and value compressed tables (both static and adaptive). - DB2 for i Log based capture is supported for DB2 for i - DB2 for z/OS Log based capture for DB2 for z/OS is supported. The following table types are supported - regular tables, compressed tables, partitioned tables, history tables and archive tables. Capture from LOAD is not supported. - PostgreSQL Log based capture is supported for all PostgreSQL versions supported by HVR. - HANA Log based capture is supported on Linux (not Windows) for HANA: version 1.0 SPS 11 and 12, and version 2.0 SPS 00 up to SPS 07 Only column-store tables are supported (no row-store tables). - MySQL and MariaDB Log based capture is supported for all MySQL and MariaDB versions supported by HVR. FIVETRAN HVR NETWORK ENCRYPTION ---------------------- An HVR connection to a remote location can be configured so that all communication over the network is encrypted. For network encryption, HVR uses OpenSSL, which is developed by the OpenSSL Project (http://www.openssl.org). HVR uses OpenSSL version 3.2.1 On UNIX, HVR's encryption will attempt to exploit the entropy (randomness) generation capability supplied by the '/dev/urandom' device. For optimal security, it is recommended that this Operating System option is installed, especially on the machine used to generate public/private key pairs CREDITS ------- - License Agreements, Copyrights, and Notices for the third-party software are listed in the hvr.3rdparty file in HVR_HOME directory NEW FEATURES IN HVR 6.2.5/1 (2024-12-17) ---------------------------------------- [T-783428] UPDATED DATA TYPE HANDLING FOR TARGET DBMSES THAT DO NOT SUPPORT TIME DATA TYPE To use this feature, upgrade HVR on all machine(s). HVR now defaults to using a VARCHAR data type for target databases that do not support the TIME data type. Previously, in such cases, HVR defaulted to using a TIMESTAMP data type on a target. This updated behavior applies to newly created channels and to cases where you modify the table DDL. Existing channels will continue with their current replication behavior. For example, if an existing channel replicates the TIME data type as a TIMESTAMP on the target, it will keep doing so. However, if you recreate tables on the target using a refresh or the AdaptDDL action, the default will switch to VARCHAR. You can override this new behavior by using the ColumnProperties action with the DatatypeMatch and Datatype parameters. [T-830301] ADDED RESILIENCE TO UNKNOWN PROPERTIES AFTER DOWNGRADE To use this feature, upgrade HVR on the hub machine(s). After a downgrade of HVR, there may be properties in the repository database or in files on disk. An older version of HVR will throw validation errors on unknown properties. This fix will instead print a warning and ignore them. Since properties are validated in various places, and HVR spawns many processes, many warnings may come out. These are safe to ignore, but are important to log because just ignoring properties may or may not cause further issues. [T-708437] DATABRICKS: ADD AN OPTION TO USE PARQUET FILES FOR STAGING TO DATABRICKS To use this feature, upgrade HVR on the integrate and hub machine(s). If a target location uses staging for loading the burst table during integrate, or the target table during refresh, the rows are stagind in csv files. With this change an option has been added to the Databricks location configuration that allows the user to use parquet files for staging. The new option is in the Source and Target Properties section of the Databricks location configuration and is titled "STAGING FILE FORMAT". [T-729900] GOOGLE BIGQUERY: SUPPORT USING PARQUET FILES WHEN STAGING DATA TO BIGQUERY To use this feature, upgrade HVR on the integrate and hub machine(s). This feature adds support for staging data using Parquet files when loading data into BigQuery. By default, HVR uses CSV files for staging data into a target database. To ensure accurate replication of DATETIME data in BigQuery, the TZ environment variable must be set to match the time zone of your BigQuery data warehouse. When using Parquet files for staging, dates prior to the year 1583 (the Julian to Gregorian calendar switch) will not be replicated correctly. For all staging file formats, timestamp values with fractional seconds in years before 1698 or after 2242 may be reported as mismatches by Compare, due to a rounding error during data querying. [T-808387] KAFKA: ADDED ABILITY TO SEND KAFKA HEADERS WITHIN KAFKA MESSAGE To use this feature, upgrade HVR on the integrate and hub machine(s). This feature introduces a new behavior for Kafka message generation. Added a new "MessageHeaders" parameter in the "Integrate" action, which enables Kafka messages to include a header section. [T-662534] POSTGRESQL: ADDED SUPPORT FOR COPY STATEMENT WHEN CAPTURING DATA FROM POSTGRES To use this feature, upgrade HVR on the capture machine(s). This change adds support for using COPY statement when capturing data from PostgreSQL source. Environment variable HVR_PQ_SELECT_USE_COPY has been added and is enabled by default. Additionally, the default value of the environment variable HVR_PQ_SELECT_USE_CURSOR has been changed to 0. [T-764064] POSTGRESQL: ADDED AUTOMATIC CONVERSION TO PGOUTPUT FOR EXISTING CHANNELS To use this feature, upgrade HVR on the capture machine(s). HVR now automatically converts existing channels to `pgoutput` logical decoding output plugin without requiring reactivation or refresh. Publications and replication slots are managed seamlessly when the capture job starts. [T-785880] POSTGRESQL: ADDED SUPPORT FOR SETTING REPLICATION SLOT NAME To use this feature, upgrade HVR on the capture machine(s). This change adds the Replication_Slot_Name location property to override the replication slot name. Replication slots are used for PostgreSQL logical replication (Capture_Method=SQL). [T-805555] POSTGRESQL: DISABLED DIRECT CAPTURE METHOD FOR POSTGRESQL To use this feature, upgrade HVR on the capture and hub machine(s). HVR has discontinued support for the DIRECT capture method for PostgreSQL. [T-249606] SAP HANA: ADD SUPPORT OF SAP DATA TYPES D16/34N,TIMN,DATN,UTCL FROM SAP DICTIONARIES To use this feature, upgrade HVR on the integrate and capture machine(s). HVR introduced support of SAP HANA datatypes D16/34N,TIMN,DATN,UTCL from SAP Dictionaries. [T-202643] SNOWFLAKE: ADDED SUPPORT FOR TRANSIENT TABLES To use this feature, upgrade HVR on the integrate and hub machine(s). HVR now supports transient tables when integrating to Snowflake. When setting the `TransientTable` TableProperties action, the corresponding target tables will be created as transient tables. Note that this only affects newly created tables and tables that are re-created due to schema changes. Specifically schema changes that result in a `ALTER TABLE` statement will not change the table type. PROBLEMS FIXED IN HVR 6.2.5/1 (2024-12-17) ---------------------------------------- [T-766320] REMOVED REDUNDANT LOG MESSAGES FROM MERGE REFRESH To fix this bug, upgrade HVR on the integrate machine(s). Fixed some confusing log messages in merge refresh jobs. One redundant message was removed, and in another message a section was removed in case we don't have data to show there. [T-816245] FIXED "TIMESTAMP HAS UNEXPECTED FORMAT" IN HVRSTATS. To fix this bug, upgrade HVR on the hub machine(s). Made hvrstats more resilient against unexpected timestamp formats. Previously, the stats job threw an F_JG231B error: "Timestamp has unexpected format". The fix is to skip processing such lines, adding them to the irregular output lines. [T-817227] FIXED CRASH LIKE F_JT04C6 FOR SOFTDELETE CHANNELS To fix this bug, upgrade HVR on the hub machine(s). Fixed an issue with Softdelete pipe injection where Softdelete pipe was injected once per table when customers used table_scope. [T-825975] FIX DUPLICATE ROW ERROR F_JD024D DURING INTEGRATION OF KEY CHANGE To fix this bug, upgrade HVR on the integrate machine(s). Fix duplicate row error F_JD024D during integration of multiple resilient key updates. [T-827122] FIXED F_JT1550 CRASH IN ORACLE ASM LOG-BASED CAPTURE To fix this bug, upgrade HVR on the capture machine(s). Fixed an issue causing crashes in Oracle ASM log-based capture worker processes. The issue produced errors such as: `Nov 04 04:41:27: hub-cap-src[bc0]: F_JG2166: Unexpected I/O detected on link to parent job process Nov 04 04:41:27: hub-cap-src: F_JT1550: Coprocess 'bc0' terminated with signal 9.` The specific process name 'bcXXX' may vary, but will always start with 'bc', and this fix only applies to such cases. [T-838145] DATABRICKS: FIXED F_JD1202 WHEN USING REFRESH MERGE OPTION WITH DATABRICKS EXTERNAL TABLES To fix this bug, upgrade HVR on the integrate machine(s). Fixed an issue that occured when refreshing data into a Databricks table that has an external location, and where the refresh "Merge into Target" option is selected. [T-844400] DATABRICKS: FIXED F_JG2919 THROWN DUE TO A NEGATIVE STRING LENGTH To fix this bug, upgrade HVR on the integrate machine(s). Fixed 'F_JG2919: Encountered internal error while encoding string from' thrown due to a negative length. Note that this change also fixes 'F_JG2228: Unable to allocate' with a very large number of bytes reported. [T-840417] GOOGLE BIGQUERY: FIXED ISSUE WITH BIGQUERY WHEN DATASET AND BUCKET ARE IN MULTI-REGION EU To fix this bug, upgrade HVR on the integrate machine(s). Fixed an issue where the error "F_JR1779 The BigQuery region 'EU' (from location property Bigquery_Region) is not the same as the discovered region 'eu-west-1' of the Google Cloud Storage" occurred when both the target dataset and the staging bucket were configured in the multi-region "EU". [T-811598] ORACLE: FIX DELETE ON ORACLE TABLE WITH EXTENDED VARCHAR2 COLUMN To fix this bug, upgrade HVR on the capture machine(s). This fix addressed a corruption on target table when Capture processed a delete operation on Oracle table with extended varchar2 column on a channel with soft-delete. The reason for the corruption is the inclusion of the LOB data header in the extended varchar2 data and the fix detects the extended varchar2 column and excludes the column data from the Delete record. [T-764064] POSTGRESQL: RESET CONNECTION INSTEAD OF STOP STREAMING REPLICATION To fix this bug, upgrade HVR on the capture machine(s). To improve handling of automatic conversion to the `pgoutput` output plugin for PostgreSQL, this change resets the connection instead of explicitly stopping streaming replication. It avoids potential error "no COPY in progress" with obsolete replication slot created for the `test_decoding` output plugin. [T-831999] SAP HANA: FIXED FALSE-POSITIVE THROWING F_JZ0A3F WHILE PARSING PARTIAL UPDATES FOR SAP HANA To fix this bug, upgrade HVR on the capture machine(s). Fixed F_JZ0A3F false-positive throwing when paritial update has more then 16 Million rows in one transaction. [T-793953] SQL SERVER: FIX CONNECTING TO AZURE SQL SERVER VIA OAUTH2 To fix this bug, upgrade HVR on the integrate and hub machine(s). Fixed an issue where HVR could fail with error F_JD21AE when attempting to connect to an Azure SQL Server using access tokens provided by OAuth2 authentication, due to incorrectly passing those credentials. Users facing either 'Login failed for user `` or `SPI Provider: No Kerberos credentials available` errors when connecting to an Azure SQL Server using OAuth2 authentication should upgrade to this version to resolve the issue. [T-826536] SYBASE ASE: FIXED READING OF PARTIALLY WRITTEN PAGES IN SYBASE BY ADDING CONSISTENCY CHECKS FOR RECORDS. To fix this bug, upgrade HVR on the capture machine(s). Fixed reading of partially written pages in Sybase by adding consistency checks for records. Added two additional checks while reading pages: 1. Page Number Consistency Check When the page number inside any record is higher than the current page number, it is considered a partially written page. The system will retry reading pages up to a maximum number defined by the environment variable `HVR_SY_READ_CORRUPT_PAGE_MAX` (default value: 10, maximum value: 255). If the number of retries reaches the maximum, an error `F_JZ1D02: The indicated page number ...` will be thrown, and the capture job will exit. 1. INSERT/DELETE/INSIND/DOL_INSIND Consistency Check Consistency of the page number and page timestamp within INSERT/DELETE/INSIND/DOL_INSIND operations is checked. If the page number or page timestamp of these operations is greater than the current page number or page timestamp, it is assumed to be a partially written page. Upon the first occurrence, a warning message, `W_JZ1D03: The indicated page number` ... or `W_JZ1D04: The page timestamp in the ...`, will be logged. The capture job will attempt to reread the page up to the maximum number of retries specified by the environment variable HVR_SY_READ_CORRUPT_OPERATION_MAX (default value: 10, maximum value: 255). If the maximum retries are reached without resolution, a warning `W_JZ1D05: Earlier, a record at address (%u, %hu) was encountered with an unexpected page number ...` will be logged, and the capture job will consider it a valid record and continue processing. [T-831523] SYBASE ASE: FIX THE OCCURENCE OF F_JZ270C - AN AFTER OPERATION IS SEEN BEFORE A BEFORE OPERATION To fix this bug, upgrade HVR on the capture machine(s). In some circumstances, UPDATEs to an "All Pages Locked" table will result in a fault similar to the following: `F_JZ270C: The [INSERT] record at (42156,10) will output a [update_after] operation, but the associated [update_before] operation has not been found.` This was due to a bug in identifying when update_before operations were output. As a result of this fix, the fault should no longer be observed. [T-833372] SYBASE ASE: FIXED THE OCCURRENCE OF F_JZ270C - DNOOP IS ALREADY MATCHED To fix this bug, upgrade HVR on the capture machine(s). In a scenario where a transaction affects multiple rows, we were encountering the error `F_JZ2709: The %s record at (%u, %hu)` in the logs. This occurred because the latest `DNOOP` was already matched with the previous `DELETE` record, but the `DNOOP` of the current `DELETE` record had not yet arrived. We mistakenly matched the `DNOOP` of the previous `DELETE` record with the current `DELETE` record. This issue has now been fixed. [T-837327] SYBASE ASE: FIXED THE OCCURRENCE OF F_JZ270A WHEN PAGE NUMBERS ARE REUSED AFTER ROLLING OVER TO OLD NUMBERS. To fix this bug, upgrade HVR on the capture machine(s). Fixed the occurrence of F_JZ270A when page numbers are reused after rolling over to old numbers. In addition, the following errors are now also continuable: `F_JZ0A91`, `F_JZ0A92`, and `F_JZ270A`. [T-833926] UI: FIXED THE REDEFINE BUTTON TO WORK ON VIEWS To fix this bug, upgrade HVR on the hub machine(s). Fixed an issue with the Redefine button on the UI that caused it to fail when used on a database view. [T-846665] UI: FIXED SNOWFLAKE CLIENT PRIVATE KEY DIRECTORY OPENS IN STAGING DIRECTORY INSTEAD OF HUB DIRECTORY To fix this bug, upgrade HVR on the hub machine(s). Fixed a UI bug where specifying an integrate staging directory on Snowflake caused the "CLIENT PRIVATE KEY" in "Database Connection" to open the integrate staging directory instead of the hub directory. NEW FEATURES IN HVR 6.2.5/0 (2024-11-21) ---------------------------------------- [T-200579] ADDED ABILITY TO CREATE ISOLATE TABLE REFRESH To use this feature, upgrade HVR on all machine(s). Allow users to create isolate table refresh for either refresh events initiated by the users or refresh events created as part of an AdaptDDL operation. User-initiated refresh events that are created without isolation, would result in all relevant integ jobs to be suspended until the refresh completes. However, isolate table refresh, would keep integ jobs working on all tables that aren't part of the isolate table refresh. Once the isolate table refresh completes, integration of the isolated tables will continue. AdaptDDL-initiated refresh events, would result in the capture side to be waiting for the refresh to completes. However, isolate table refresh, would keep the capture side working on all tables (including the isolated tables), and make the integ side working on all tables that aren't part of the isolate table refresh. Once the isolate table refresh completes, integration of the isolated tables will continue. [T-306320] EXPOSE RESILIENT OPERATIONS IN ONLINE REFRESH. To use this feature, upgrade HVR on the integrate and hub machine(s). Currently we expose only a subset of hvr_ops in the {hvr_op} integrate expression. For tables with ColumnProperties /TimeKey, this can result in apparent duplicate rows. For example, if a row was inserted while an online refresh was executing. By returning the resilient hvr_op codes, the results in the target table can be more easily interpreted. Additionally, the compare function will work correctly with these codes, instead of throwing a 1+1 compare/coalesce error when this situation occurred. To turn on this new behavior, the HVR_EXPR_SUBST_HVR_OP_RESIL environment variable must be set to zero. The default behavior is to return the subset of hvr_ops as before, the user must opt-in to the new behavior with the environment variable. [T-732111] ADD THE ABILITY TO CUSTOMIZE {HVR_OP} VALUES To use this feature, upgrade HVR on the integrate and hub machine(s). Allow users to customize {hvr_op} values by creating a ColumnProperties action with the following parameters * /IntegrateExpression={hvr_op_str [custom mapping]} /Extra /Datatype=char /Length=2 * E.g. /IntegrateExpression={hvr_op_str 0>D;1>I;2>U} /Extra /Datatype=char /Length=2 * Or /IntegrateExpression={hvr_op_str} /Extra /Datatype=char /Length=2 * In this case, since the user doesn't define a custom mapping, we will use the default mapping which is 0>D;1>I;2>U;3>K;4>4;5>T DataType can be char or varchar. If an hvr_op value is not part of the mapping, it won't be mapped to anything but it will be used as is. Since hvr_op can be a double-digit number, the ColumnProperties parameter /Length should be no less than 2. [T-801413] UPGRADED BUNDLED FIVETRAN PROXY TO 1.0.20 To use this feature, upgrade HVR on the agent machine(s). Upgraded bundled Fivetran Proxy to 1.0.20 [T-821955] BUNDLE FIVETRAN PROXY 1.0.21 To use this feature, upgrade HVR on the agent machine(s). Upgraded bundled Fivetran Proxy to 1.0.21 [T-771320] DATABRICKS: ADDED SUPPORT TO USE GOOGLE CLOUD STORAGE AS STAGING To use this feature, upgrade HVR on the integrate and hub machine(s). Added support to use Google Cloud Storage as staging for Databricks. Make sure the Databricks cluster is properly configured to access the corresponding Google Cloud Storage bucket(s). [T-775861] DATABRICKS: ADDED SUPPORT FOR BC DATES IN DATABRICKS TIMESTAMP To use this feature, upgrade HVR on the integrate and hub machine(s). This change adds support for BC (Before Christ) dates in the Databricks TIMESTAMP data type. [T-763648] DB2 FOR I: ADDED SUPPORT FOR XML DATA TYPE To use this feature, upgrade HVR on all machine(s). HVR now supports the XML data type in Db2 for i. [T-763650] DB2 FOR I: ADDED SUPPORT FOR ROWID DATA TYPE To use this feature, upgrade HVR on all machine(s). HVR now supports the ROWID data type in Db2 for i. [T-763651] DB2 FOR I: ADDED SUPPORT FOR DATALINK DATA TYPE To use this feature, upgrade HVR on all machine(s). HVR now supports the DATALINK data type in Db2 for i. Note that only the URL is replicated. Any embedded comment is ignored. [T-805110] DB2 FOR I: ADDED SUPPORT FOR BINARY DECIMAL PHYSICAL FILE DATA TYPE To use this feature, upgrade HVR on all machine(s). HVR now supports the BINARY DECIMAL (B, BINDEC) data type in Db2 for i. [T-789914] GOOGLE BIGQUERY: ENABLE PARALLEL SLICED REFRESH FOR BIGQUERY To use this feature, upgrade HVR on the hub machine(s). This change enables running a sliced bulk load refresh in parallel when the target is BigQuery [T-766016] ORACLE: ADDED SUPPORT FOR CONFIGURABLE LOB PREFETCH SIZE To use this feature, upgrade HVR on the integrate and capture machine(s). This change allows you to configure the LOB prefetch size for Oracle databases at the session level. The new environment variable HVR_ORA_LOBPREFETCH_SIZE enables setting a prefetch size that applies to all LOB columns (such as CLOB and BLOB) in replicated tables. To maintain the current behavior, avoid modifying the new environment variable, and you will see no changes in your workflow. This change is available for Oracle versions 12.2 and newer. [T-779320] ORACLE: ENABLED CONCURRENT CHANNEL READING FROM SINGLE ORACLE DATABASE USING BFILE To use this feature, upgrade HVR on the capture machine(s). This improvement allows multiple channels to read from the same Oracle database using BFILE. Previously, each channel required a separate BFILE directory path to avoid conflicts when accessing archived sequences from ASM. This update automatically adds a unique pre/suffix to files, enabling multiple channels to safely read concurrently within a single BFILE directory path. [T-784052] ORACLE: OPTIMIZED ONE OF THE QUERY USED IN HVRADAPT FOR ORACLE To use this feature, upgrade HVR on the capture machine(s). This improvement optimizes one of the Oracle query used in hvradapt for getting the supplemental logging setting of a list of tables using the HVR_SYS_TAB that could take a considerable amount of time to complete, resulting in a huge delay in adding tables to the channel. [T-810525] ORACLE: OPTIONALLY TREAT PERMISSIONS ERROR ON FILE REMOVING AS A WARNING To use this feature, upgrade HVR on the capture machine(s). Optionally treat permissions error on file removing as a warning to allow capture to continue [T-662172] POSTGRESQL: SUPPORT YUGABYTEDB WITH LOGICAL REPLICATION CAPTURE METHOD To use this feature, upgrade HVR on all machine(s). Add support for YugabyteDB release starting 2024.1.1.0 where PostgreSQL Logical Replication Protocol Support is added. Add YugabyteDB as an option of source and target location type on UI. Add support for YugabyteDB exclusive REPLICA IDENTITY option CHANGE. Set default isolation level to REPEATABLE READ for YugabyteDB. [T-683294] POSTGRESQL: ADDED SUPPORT FOR POSTGRESQL LOGICAL REPLICATION WITH `PGOUTPUT` PLUGIN To use this feature, upgrade HVR on the capture machine(s). This new feature enables support for PostgreSQL logical replication with `pgoutput` plugin, which is the standard decoding output plugin alongside `test_decoding`, currently supported by HVR. `pgoutput` plugin has to work with publication management, allowing us only receive selected tables from PostgreSQL logical replication stream and improving overall capture performance. Please note that loopback detection with `pgoutput` is not available yet. [T-683297] POSTGRESQL: ADDED OPTION FOR MANAGING PUBLICATIONS FOR POSTGRESQL DATABASES To use this feature, upgrade HVR on the capture and hub machine(s). This update enables management of publications in PostgreSQL databases used as source locations. - Added a checkbox to the Activate Replication dialog for enabling/disabling publication management. - Introduced the option `-u` in the hvractivate command that enables the creation of a publication if it does not already exist. - Only selected tables are added to publications. - T-683297 - T-728737 - T-728743 - T-728747 - T-754211 [T-705418] POSTGRESQL: REMOVE STREAMING REPLICATION ALTERNATIVE IMPLEMENTATION To use this feature, upgrade HVR on the capture machine(s). HVR SQL capture for PostgreSQL has an alternative implementation to consume logical decoding changes using SQL level functions. This is enabled by setting HVR_PQ_USE_STREAMING_REPLICATION to 0. This implementation is inefficient and has many problems. This has been deprecated and made not necessary for any supported PostgreSQL version in HVR 6.1.5 releases. We are removing it altogether in 6.2.5. Complete list of removed settings by this change: - HVR_PQ_USE_STREAMING_REPLICATION - HVR_PQ_USE_REPLICATION_SLOT_ADVANCE - HVR_PQ_LOG_CURS_PREFETCH - HVR_PQ_LOG_READ_PAGE_GROUP [T-764063] POSTGRESQL: STOP USING TEST_DECODING AS DEFAULT PLUGIN To use this feature, upgrade HVR on the capture machine(s). Stop using test_decoding as default plugin when to create new channels, pgoutput plugin will be used. The default value of environment variable HVR_PQ_USE_TEST_DECODING is 0. [T-780218] POSTGRESQL: DISABLE HVR_PQ_STREAMING_REPLICATION_RESTART_AT_END_OF_WAL BY DEFAULT To use this feature, upgrade HVR on the capture machine(s). This change disables HVR_PQ_STREAMING_REPLICATION_RESTART_AT_END_OF_WAL by default for all versions of PostgreSQL. [T-780241] POSTGRESQL: DROP SUPPORT FOR POSTGRESQL 9.5 To use this feature, upgrade HVR on all machine(s). This change removes support for PostgreSQL 9.5. [T-800574] POSTGRESQL: ADDED SUPPORT FOR SETTING POSTGRESQL PUBLICATION NAME To use this feature, upgrade HVR on the capture machine(s). This change adds the HVR_PUBLICATION_NAME environment variable to override the publication name. [T-801374] POSTGRESQL: REMOVED FORCED TIMEOUT IN STREAMING REPLICATION To use this feature, upgrade HVR on the capture machine(s). HVR no longer forces timeouts in streaming replication for PostgreSQL versions 12 and earlier. Timeouts were needed to handle limitations that existed before the HVR_PQ_STREAMING_REPLICATION_RESTART_AT_END_OF_WAL setting. Now that this setting is available and disabled by default, this change simplifies the decision process and eliminates the need for special handling of older PostgreSQL versions. No additional user actions are required. [T-812232] POSTGRESQL: ADDED SUPPORT OF YBOUTPUT PLUGIN FOR YUGABYTEDB LOGICAL REPLICATION To use this feature, upgrade HVR on the capture machine(s). This new feature enables support for YugabyteDB logical replication with `yboutput` plugin, in addition to the existing support of `test_decoding` plugin. And by default it uses `yboutput` plugin. `yboutput` plugin has to work with publication management, allowing us only receive selected tables from YugabyteDB logical replication stream and improving overall capture performance. PROBLEMS FIXED IN HVR 6.2.5/0 (2024-11-21) ---------------------------------------- [T-689331] FIXED INCORRECT PROCESSING OF HANA UTCL DATATYPE, IMPROVED ERROR HANDLING To fix this bug, upgrade HVR on the capture machine(s). Replace SAP transport files with newest version. Delivers fixes to for T-689331 and T-734793: Fixed incorrect processing of SAP Hana UTCL datatype for Netweaver driver; Correctly sends error reports in case of SQL processing [T-778063] FIXED RESILIENT INTEGRATE OF NON-KEY COLUMNS WHEN A ROW WAS DELETED AND THEN INSERTED WITH THE SAME KEY To fix this bug, upgrade HVR on the integrate machine(s). Fixed resilient integrate of non-key columns when a row was deleted and then inserted with the same key. See also T-679974, which resolved this case only for key columns. [T-778090] FIX 'TRUNCATE' SUPPORT FOR POSTGRES TIMEKEY TABLES To fix this bug, upgrade HVR on the integrate machine(s). Fixed an issue where error F_JD2313 was thrown on refresh after a truncate on a timekey channel for Postgres. T-597752 - Fixed a similar issue for Redshift [T-784898] FIXED F_JR4328 WHEN IMPORTING CHANNEL WITH ENCRYPTED TDE CERTIFICATES To fix this bug, upgrade HVR on the hub machine(s). Fixed an issue where channel import would fail with error F_JR4328 if the channel contained a location with a TDE certificate and was exported using encryption with a transport key. The error was caused by a premature validation of the certificate's private key and is now eliminated by skipping the certificate validation if the private key is encrypted. [T-791278] FIXED ERROR F_JG2142 WHEN REPLICATING FROM SAP HANA To fix this bug, upgrade HVR on the integrate machine(s). Fixed error F_JG2142 when performing a DELETE operation during replication from SAP HANA. [T-791280] FIXED F_JR0A0E WHEN ACCESSING ENCRYPTED SQL SERVER LOCATIONS To fix this bug, upgrade HVR on the hub machine(s). Fixed F_JR0A0E errors that occurred when accessing encrypted SQL Server locations after upgrading to OpenSSL 3. The issue was caused by missing ciphers that were moved to a legacy module in OpenSSL. This fix now packages the legacy module with HVR, eliminating the dependency on the hub machine having a separate OpenSSL installation. [T-794592] FIXED F_JR032D DURING HVRDEFINITIONIMPORT FOR CASE-INSENSITIVE ACTION NAMES To fix this bug, upgrade HVR on the hub machine(s). Fixed an issue where error F_JR032D could be triggered during hvrdefinitionimport if the action name was correct but the case did not match. This could only occur when users manually created definition JSON data or migrated manually edited HVR 5 hub database. It is recommended to use canonical ActionName capitalization when creating a definition JSON or to sanitize JSON data migrated from a manually edited HVR 5 hub. [T-795635] FIXED FALSE DIFFERENCES IN ONLINE COMPARE WITH CUSTOM TASK NAME To fix this bug, upgrade HVR on the hub machine(s). Fixed an issue where correct behavior of online compare depended on the compare job using the default task name of "cmp". In some edge cases, compare jobs with custom task names could report incorrect results. [T-801179] FIX UNEXPECTED TIMESTAMP FORMAT WRITTEN BY CAPTURE JOB To fix this bug, upgrade HVR on the hub machine(s). Fixed capture writing timestamp in output file without timezone and in unexpected format (missing T). T-602157 [T-805282] FIXED INTEGRATEEXPRESSION PERFORMANCE OF CAST(CURRENT_TIMESTAMP() AS DATETIME) FOR BIGQUERY To fix this bug, upgrade HVR on the hub machine(s). Add `cast(current_timestamp() as datetime)` to the constsqlexpr.pat so whenever `cast(current_timestamp() as datetime)` is used in IntegrateExpression, there will be no performance overhead [T-810635] FIX INTEGRATEEXPRESSION NOT POPULATED FOR BEFORE-UPDATE RECORDS FOR SOFTDELETE FILE TARGETS To fix this bug, upgrade HVR on the hub machine(s). Fix IntegrateExpression not populated for before-update records for SoftDelete file targets [T-821576] FIXED F_JD5E12 (MISSING INPUT ENCODING FOR DATATYPE '106' DATALINK) WHILE RUNNING HVRDATATYPEMAP To fix this bug, upgrade HVR on the hub machine(s). This fixes hvrdatatypemap outputting mapping for data type datalink [T-826502] FIXED SIGSEGV CONDITION DURING CHANNEL ADAPT To fix this bug, upgrade HVR on the hub machine(s). A segmentation fault condition was present in the adapt code. This has been removed. [T-826645] FIXED SPACE IN HVR_HOME CAUSING F_JG057F WITH CONNECTION RESET BY PEER To fix this bug, upgrade HVR on the proxy and agent machine(s). Fixed space in %HVR_HOME% causing F_JG057F: TLS negotiation failed: I/O error occurred: Connection reset by peer. This fix specifically applies to a Connection reset by peer error: - When the customer has an HVR_HOME with whitespace in it - That exposes itself as F_JG057F: TLS negotiation failed, and explicitly not another F_J error code that may have the same underlying Connection reset by peer. [T-827139] ENFORCE MINIMUM SUPORTED DATABASE VERSION UPON CONNECTING To fix this bug, upgrade HVR on the integrate and capture machine(s). This bug fix forces a supported version check upon connecting to a database. If a database is not supported, the new error F_JT05DD is raised, with the suggestion to contact customer support. [T-830358] FIXED F_JG2459 IN HVRREFRESH AND HVRCOMPARE INVOCATIONS To fix this bug, upgrade HVR on the hub machine(s). Fixes F_JG2459: Get value of variable 'opt_E_is_time' failed. Variable not set. when calling hvrrefresh and hvrcompare on repository databases configured with overridden library environment (e.g. PostgreSQL_Pglib location property) [T-777038] DB2 FOR LINUX, UNIX AND WINDOWS: FIXED F_JD21B5 SQL0206N "TYPE" IS NOT VALID FOR TIME AND DATE COLUMNS WITH A DEFAULT DEFAULT To fix this bug, upgrade HVR on the capture machine(s). Fixed an issue appearing during HVR activate with error code F_JD21B5 SQL0206N "TYPE" is not valid. This can happen for TIME and DATE columns with a default value of DEFAULT. [T-615210] DB2 FOR I: FIX F_JT0451 PIPE TXGROUP ENCOUNTERED A COMMIT WITH SEQUENCE <= EARLIER SEQUENCE ERROR WITH DST CHANGE ON DB2 FOR I SOURCE WITH DB2I_LOG_JOURNAL_SYSSEQ=TRUE To fix this bug, upgrade HVR on the capture machine(s). When capturing changes from Db2 for i with DB2i_Log_Journal_SysSeq=true, HVR used Db2 for i system sequence numbers to sort changes. Those system sequence numbers are, however, time based which causes problems when switching from summer time to winter time. HVR, therefore, will no longer use system sequence numbers to sort captured changes. Another way to sort captured changes will be used instead. That also allows handling of sequence number resets. [T-793972] DB2 FOR I: DB2 FOR I BY DEFAULT CAPTURES ALL MEMBERS BUT SETTING HVR_DB2I_MULTI_MEMBER_ALL=0 CAPTURES ONLY FIRST MEMBER To fix this bug, upgrade HVR on the capture machine(s). By default all members from DB2 for i multi-member files are captured. But setting environment variable HVR_DB2I_MULTI_MEMBER_ALL to 0 will lead to only first member being captured. Note that if HVR_DB2I_MULTI_MEMBER_ALL is set to 0 and tables which are in the channel are removed from the source database, then those tables should also be removed from the channel, otherwise the capture job will fail. [T-793790] DB2 FOR Z/OS: FIXED MEMORY LEAK DURING HVR ADAPT To fix this bug, upgrade HVR on the integrate and capture machine(s). This fix addresses a memory leak during HVR Adapt with Db2 for z/OS [T-785500] KAFKA: FIXED REGRESSION "TARGET UNKNOWN (0X80090303)" WHILE USING KERBEROS AUTHENTICATION ON WINDOWS PLATFORM To fix this bug, upgrade HVR on the integrate machine(s). Fixed "Target unknown (0x80090303)" while using Kerberos Authentication on Windows platform: - librdkafka used for communication with Kafka broker is rebuilt to correctly use UNICODE versions of SSPI WinAPI [T-761689] ORACLE: FIX FAILURE IN CONNECTING TO A NON-ACTIVE ORACLE DATA GUARD INSTANCE To fix this bug, upgrade HVR on the hub machine(s). This fix addressed a failure in capturing from a non-active Oracle Data Guard physical standby instance, which requires a sysdba privilege. The issue was introduced by T-632414 [T-790201] ORACLE: FIXED REGRESSION IN HANDLING REMOVED FILES WHEN SCANNING ARCHIVES To fix this bug, upgrade HVR on the capture machine(s). Fixed regression in handling removed files when scanning archives [T-802234] ORACLE: FIXED PREFIX FOR TEMPORARY FILES TO ALLOW CORRECT AGENTLESS USAGE To fix this bug, upgrade HVR on the capture machine(s). Fixed prefix for temporary files to allow correct agentless usage [T-822430] ORACLE: FIXED ADDITIONAL SLASH IN THE FRONT OF THE WINDOWS DIRECTORY NAME To fix this bug, upgrade HVR on the capture machine(s). Fixed additional slash in the front of the windows directory name. [T-723684] POSTGRESQL: FIXED SUPPLEMENTAL LOGGING FOR YUGABYTEDB To fix this bug, upgrade HVR on the capture machine(s). Supplemental logging will perform "ALTER TABLE ... REPLICA IDENTITY FULL" for regular PostgreSQL if REPLICA IDENTITY is set to DEFAULT. This fix ensures that the same logic applies to YugabyteDB when REPLICA IDENTITY is set to CHANGE, which is YugabyteDB's default setting. [T-795641] POSTGRESQL: FIXED ISSUE WITH RESTARTING STREAMING REPLICATION AT THE END OF WAL SERVICE To fix this bug, upgrade HVR on the capture machine(s). Fixed an issue where HVR would not correctly follow the WAL position (LSN) when the HVR_PQ_STREAMING_REPLICATION_RESTART_AT_END_OF_WAL setting was enabled. This defect could occur in specific rare workloads, potentially impacting replication reliability. The fix ensures that the streaming replication connection to PostgreSQL restarts correctly at the end of the WAL, as expected. [T-801613] POSTGRESQL: FIX TABLE NAMES ON PUBLICATION TO USE BASENAME To fix this bug, upgrade HVR on the capture machine(s). Fixed to use tables basename when setting tables on PUBLICATION. It was using table names on channel which could be different from basenames. T-683297 [T-805911] POSTGRESQL: ADDED SUPPORT FOR SETTING POSTGRESQL PUBLICATION NAME To fix this bug, upgrade HVR on the capture machine(s). This change stops HVR hub from trying to create PUBLICATION on unsupported agents. [T-806818] POSTGRESQL: FIXED PUBLICATION CREATION FOR TABLES WITH NON DEFAULT SCHEMA To fix this bug, upgrade HVR on the capture machine(s). This change fixes creation of PUBLICATION for tables that have non default schema. [T-812112] POSTGRESQL: FIXED F_JZ6227 TEST_DECODING MESSAGE TYPE To fix this bug, upgrade HVR on the capture machine(s). Fixed F_JZ6227 error when it failed to detect the "message" type emitted by `test_decoding` plugin of PostgreSQL logical replication. The hardcoded "message" keyword has been fixed to allow proper detection and handling. [T-814502] POSTGRESQL: ADD SINGLE PARTITIONS TO PUBLICATION FOR POSTGRES VERSION LESS THAN 13 To fix this bug, upgrade HVR on the capture machine(s). This fix adds partitions of the root table to the publication for PostgreSQL version prior to 13. [T-820616] POSTGRESQL: FIXED NON-KEY UPDATE FOR YUGABYTEDB'S YBOUTPUT PLUGIN To fix this bug, upgrade HVR on the capture machine(s). Fixed non-key update for YugabyteDB's yboutput plugin. With yboutput plugin, update before may contain old tuple with all columns unchanged, this fix resolved the error to replicate non-key update. [T-597752] REDSHIFT: FIXED TRUNCATE SUPPORT FOR TIMEKEY TABLES To fix this bug, upgrade HVR on the integrate machine(s). Redshift ODBC driver incorrectly reports not-null columns to be nullable. Then, HVR tries to insert nulls into these columns which results in error. In particular, this happens while replicating a 'truncate' statement on source table to target TimeKey table. [T-690300] SAP HANA: FIXED F_JD2288 WHILE PROCESSING OF HANA NATIVE DATATYPES DATE/TIME/TIMESTAMP To fix this bug, upgrade HVR on the integrate, capture and hub machine(s). Fixed getting trucated data from SAP Hana database like '0001-01-0'. Occured only if there are any LOB column in table definition. Correctly recognize SAP Time format 'HHMMSS' as a time. Correctly recognize SAP UTC Timstamp format 'YYYYMMDDHHMMSSsssssssss" as a timestamp [T-775324] SAP HANA: FIXED F_JZ0A4D AND F_JZ0A4E WHILE SEARCHING FOR HANA LOG BACKUPS To fix this bug, upgrade HVR on the capture machine(s). Fixed an issue where the capture from HANA may fail with F_JZ0A4D or F_JZ0A4E. When searching for a HANA log backup, capture job checks backup catalog, primary location and alternate location in order. When a suitable log backup is not found but a newer than needed backup is found instead, capture job may have failed with either F_JZ0A4D or F_JZ0A4E error code prematurely before checking the other locations. [T-802201] SAP HANA: FIXED PROVIDING EMPTY CONFIGURATION PARAMETER VALUE FOR SAP HANA HDBBACKINT UTULITY To fix this bug, upgrade HVR on the capture machine(s). In case of using Microsoft backint implementataion (MSAWB) 'log_backup_parameter_file' value is not set in SAP Hana configuration. HVR ignores to use '-p' option for hdbbackint application if value is not provided neither in SAP Hana database nor in location configuration. [T-819323] SAP HANA: FIXED F_JZ0C0D DURING CAPTURE FROM SAP HANA To fix this bug, upgrade HVR on the capture machine(s). Fxied incorrect calculation expected LOB size. [T-691448] SAP NETWEAVER: FIXED DATATYPE MAPPING FOR THE HANA'S DECIMAL_FLOAT WHEN CONNECTING OVER NETWEAVER To fix this bug, upgrade HVR on the capture machine(s). When connecting to a HANA database over the NetWeaver, all HANA datatypes based on the DECIMAL_FLOAT internal datatype were incorrectly mapped to the HVR's internal datatype 19 (decimal), whereas it should be mapped to the HVR's internal datatype 8 (number). This may later lead to the capture job failing with the F_JZ0A33 error. This fix addresses this issue. [T-739543] SQL SERVER: FIXED SQL SERVER CAPTURE REWIND TO THE OLDEST TX WHEN SOME TABLES ARE NOT IN THE DEFAULT SCHEMA To fix this bug, upgrade HVR on the capture and hub machine(s). When looking for the beginning of the oldest running transaction on SQL Server, HVR only considers transactions that hold a lock on one of the tables in the channel. Due to a bug, owever, only tables in the source database's default schema were considered. This fix addresses the issue. The issue affects Activate with the "Rewind to Start of Source Databases' Oldest Transaction" option and Online Refresh with "Changes before refresh are skipped by both capture or integrate jobs" option. [T-816669] SQL SERVER: FIXED F_JD0A3D DURING CAPTURE FROM SQL SERVER To fix this bug, upgrade HVR on the capture machine(s). SQL Server log-based capture may fail with the F_JD0A3D error if one or more of the source tables contain DECIMAL SPARSE columns. This fix addresses the issue. [T-798206] SNOWFLAKE: FIXED LEAKING OF EXPIRED CREDENTIALS TO LOG FILE To fix this bug, upgrade HVR on the integrate machine(s). Fixed an issue where expired credentials are leaked into the logs. All credentials are now redacted when logging erroneous SQL queries. [T-690218] SYBASE ASE: ADDED MORE CHECKS TO HELP PREVENT READS OF PARTIALLY WRITTEN LOG PAGES WITH ASE LOG BASED CAPTURE To fix this bug, upgrade HVR on the capture machine(s). Since there is no coordination between Sybase ASE and HVR in the writing and reading of transaction log pages, HVR must perform various checks after reading a page to ensure it is self-consistent and that contents are as expected. Otherwise, HVR risks reading pages which are only partially written. This change introduces more page consistency checking while allowing a configurable number of inconsistencies when re-reading the same page before reporting a fault. This allows an opportunity for a partially written page to become completely written before a fault is reported. The number of re-reads is configurable using environment variable HVR_SY_READ_CORRUPT_PAGE_MAX. [T-710526] SYBASE ASE: FIXED F_JD0AD0 BY NOT MOVING THE TRUNCATION POINT IF THE START_ADDR IS "UNKNOWN" IN ASE To fix this bug, upgrade HVR on the capture machine(s). The truncation point is no longer moved if the start_addr is "unknown" in ASE. This avoids a F_JD0AD0 error being raised in this scenario. [T-717979] SYBASE ASE: ASE DIRECT CAPTURE - SCANNER ENCOUNTERED AFTER BEFORE BEFORE To fix this bug, upgrade HVR on the capture machine(s). NOOP/INOOP/DNOOP transaction log records were mishandled in some instances, causing the following error: `text F_JT0406: Scanner encountered 'after update' before the 'before update' change is ready` A combination of log records being mishandled is illustrated by this example of a single "deferred" UPDATE operation. ```text DNOOP 5373308, 1 (0x0051FD7C, 0x01) 01 01 0c 00 7b fd 51 00 0c 00 00 00 30 00 88 00 00 00 00 00 00 00 00 00 00 00 00 00 b6 e1 d4 5e 00 00 00 00 00 00 00 00 2b 50 39 00 07 00 00 dc NOOP 5373308, 2 (0x0051FD7C, 0x02) 01 02 0a 05 7b fd 51 00 0c 00 00 00 08 01 88 00 00 00 00 00 00 00 00 00 00 00 00 00 b6 e1 d4 5e 00 00 00 00 00 00 00 00 2f 07 4e 4e 58 36 37 30 37 37 20 23 db c3 ff 05 a2 b1 00 00 fa e4 bd 00 41 da c3 ff 05 db 00 4c 4f 41 4e 00 00 00 00 00 00 0c 40 4a 50 4d 34 32 32 30 30 37 45 53 46 53 45 43 9f a5 00 00 00 00 00 00 54 fe fb 05 b5 87 de 5e a9 99 8b 3f 82 b1 00 00 00 00 00 00 a6 b3 00 00 00 00 00 00 00 00 00 94 81 4a 98 41 4d 44 50 46 58 30 58 36 37 30 37 37 ba 49 0c 02 2b 07 10 40 45 53 46 53 45 43 6f 00 82 b1 00 00 00 00 00 00 82 b1 00 00 00 00 00 00 19 fc 08 00 78 00 7b 7c 01 00 4e 23 30 00 ce 02 a7 d2 46 00 63 f1 01 00 30 aa a6 a2 a0 9e 9d 9c 9c 9c 98 96 94 92 8a 82 82 80 7a 72 72 72 72 6c 6a 66 5e 56 56 56 56 56 56 4e 46 46 46 46 46 46 42 42 42 3a 34 2b 2b 23 1f 00 17 42 5f 2a DNOOP 5373308, 3 (0x0051FD7C, 0x03) Same content as DNOOP (5373308,5) 01 03 0c 00 7b fd 51 00 0c 00 00 00 30 00 88 00 00 00 00 00 00 00 00 00 00 00 00 00 b6 e1 d4 5e 00 00 00 00 00 00 00 00 92 ee 38 00 22 00 00 c6 INOOP 5373308, 4 (0x0051FD7C, 0x04) Same content as NOOP (5373308,6), References both DNOOP (5373308,3) and DNOOP (5373308,5) 01 04 0b 05 7b fd 51 00 0c 00 00 00 08 01 88 00 00 00 00 00 00 00 00 00 00 00 00 00 b6 e1 d4 5e 00 00 00 00 00 00 00 00 2f 22 4e 4e 58 36 34 45 30 42 20 23 c5 c1 ff 05 a2 b1 00 00 fa e4 bd 00 41 c4 c1 ff 05 db 00 4c 4f 41 4e 00 00 00 00 00 00 0c 40 4a 50 4d 34 31 33 31 39 35 45 53 46 53 45 43 dd a0 00 00 00 00 00 00 54 fe fb 05 62 23 bd 66 c1 dd b3 3f 82 b1 00 00 00 00 00 00 4e cb 00 00 00 00 00 00 00 00 00 a8 f8 f1 9d 41 4d 44 50 46 58 30 58 36 34 45 30 42 a8 c6 4b 37 89 41 10 40 45 53 46 53 45 43 5c 01 82 b1 00 00 00 00 00 00 82 b1 00 00 00 00 00 00 19 fc 03 00 68 01 63 06 08 00 4e 23 42 00 f7 02 a7 d2 46 00 6c 20 08 00 30 aa a6 a2 a0 9e 9d 9c 9c 9c 98 96 94 92 8a 82 82 80 7a 72 72 72 72 6c 6a 66 5e 56 56 56 56 56 56 4e 46 46 46 46 46 46 42 42 42 3a 34 2b 2b 23 1f 04 4f 06 10 00 DNOOP 5373308, 5 (0x0051FD7C, 0x05) Called out in error message. Same content as DNOOP (5373308,3) 01 05 0c 00 7b fd 51 00 0c 00 00 00 30 00 88 00 00 00 00 00 00 00 00 00 00 00 00 00 b6 e1 d4 5e 00 00 00 00 00 00 00 00 92 ee 38 00 22 00 00 c6 NOOP 5373308, 6 (0x0051FD7C, 0x06) Same content as INOOP (5373308,4) 01 06 0a 05 7b fd 51 00 0c 00 00 00 08 01 88 00 00 00 00 00 00 00 00 00 00 00 00 00 b6 e1 d4 5e 00 00 00 00 00 00 00 00 2f 22 4e 4e 58 36 34 45 30 42 20 23 c5 c1 ff 05 a2 b1 00 00 fa e4 bd 00 41 c4 c1 ff 05 db 00 4c 4f 41 4e 00 00 00 00 00 00 0c 40 4a 50 4d 34 31 33 31 39 35 45 53 46 53 45 43 dd a0 00 00 00 00 00 00 54 fe fb 05 62 23 bd 66 c1 dd b3 3f 82 b1 00 00 00 00 00 00 4e cb 00 00 00 00 00 00 00 00 00 a8 f8 f1 9d 41 4d 44 50 46 58 30 58 36 34 45 30 42 a8 c6 4b 37 89 41 10 40 45 53 46 53 45 43 5c 01 82 b1 00 00 00 00 00 00 82 b1 00 00 00 00 00 00 19 fc 03 00 68 01 63 06 08 00 4e 23 42 00 f7 02 a7 d2 46 00 6c 20 08 00 30 aa a6 a2 a0 9e 9d 9c 9c 9c 98 96 94 92 8a 82 82 80 7a 72 72 72 72 6c 6a 66 5e 56 56 56 56 56 56 4e 46 46 46 46 46 46 42 42 42 3a 34 2b 2b 23 1f 00 00 00 00 00 ``` DNOOPs are to be paired with INOOPs in forming the Before and After images of an UPDATE operation. Prior to this correction, the list of DNOOPs encountered was searched forwards to find a match for a given INOOP. In fact, the matching DNOOP for an INOOP seems always to be the immediately preceding operation so that this list should be searched backwards. In addition, after analysis, it seems that the NOOP records are being used to "cancel" the immediately preceding DNOOP records. Also, not illustrated, the INSIND record has been observed to be used to trigger the output of the Before and After images of a "deferred" UPDATE. Recently it has been shown that an INSERT record may be used instead of the INSIND for that same purpose. HVR now takes these insights into account in its log-scan activities. [T-717979] SYBASE ASE: FIXED FAILING UPDATE TO A PARTITIONED TABLE IN SYBASE ASE To fix this bug, upgrade HVR on the capture machine(s). Fixed failing UPDATE to a partitioned table in Sybase ASE whereby an UPDATE to a row moves it from one partition of the table to another. [T-720493] SYBASE ASE: ASE ARCHIVE LOG CAPTURE - BETTER HANDLING OF UNEXPECTED END-OF-FILE To fix this bug, upgrade HVR on the capture machine(s). When configured to read archived transaction logs, the ASE Capture job is unable to distinguish an archive that has been completely written from one that is in the process of being written. In the later case, the file may appear to be corrupted resulting in a failure message similar to the following: `F_JZ0C16: An unexpected end-of-file is encountered while trying to read 2048 bytes of data at an offset of 16384 bytes from the backup transaction log file 'my_tran_log_archive_1.trn'.` Eventually the job will restart and likely successfully read the archive upon a subsequent try. In an effort to more appropriately report the situation, with this change the capture job will now report a warning message similar to the following: `W_JZ2702: An unexpected end-of-file is encountered while trying to read 2048 bytes of data at an offset of 16384 bytes from the backup transaction log file 'my_tran_log_archive_1.trn'. Assuming a partially written file and will re-read shortly.` The capture job will report this warning for up to 5 times before reporting the failure, at which point the administrator may wish to stop the job and determine whether in fact, the archive file contains some corruption. [T-743384] SYBASE ASE: MORE THOROUGHLY CHECK FOR CONSISTENCY OF PAGES IN SYBASE TO AVOID READING PARTIALLY WRITTEN PAGES To fix this bug, upgrade HVR on the capture machine(s). In this change, Sybase transaction log pages are more thoroughly checked for consistency to avoid reading partially written pages. Two additional checks have been introduced while reading pages: 1. Page Number Consistency Check When the page number inside the `BEGIN` operation is not consistent with the page number in the header, a warning `W_JZ1D02: The indicated page number ...` will be logged. The system will retry reading pages up to a maximum number defined by the environment variable `HVR_SY_READ_CORRUPT_PAGE_MAX` (default value is 10, maximum value is 255). If the number of retries reaches the maximum, an error `F_JZ1D02: The indicated page number ...` will be thrown, and the capture job will exit. 1. Timestamp Regression Check Consistency of timestamps read from the logs is checked. If a timestamp regression is encountered (i.e., a new timestamp is older than a previously seen timestamp), it is assumed to be a partially written page. Upon the first occurrence, a warning `W_JZ1D00: Encountered a timestamp regression at address ...` will be printed. The capture job will retry reading the page up to the maximum number specified by the environment variable `HVR_SY_READ_TSTAMP_REGRESSION_MAX` (default value is 10, maximum value is 255). If the timestamp regression is resolved during the retries, a message `Earlier, a timestamp regression was encountered at address (%u,%hu), but it has been resolved ...` will be printed. If the maximum number of retries is reached without resolution, a warning `W_JZ1D01: Earlier, a timestamp regression was encountered at address (%u,%hu). It was not resolved after ...` will be logged, and the capture will assume it is a correct record and continue processing. [T-763583] SYBASE ASE: ASE DIRECT CAPTURE - DELETE RECORD FOR DNOOP RECORD IS NOT FOUND To fix this bug, upgrade HVR on the capture machine(s). INSIND records are used to "tie together" the Before and After images of some kinds of UPDATE operations. INSIND record processing makes adjustments for After images of UPDATEs but fails to account for Before images that may have been earlier produced, particularly when LOBs (and so ROWIMAGE records) are involved. With this change proper adjustments are made when processing INSIND records to account for the Before images of UPDATE operations and the user will no longer experience the error: `text F_JZ270B: The DNOOP record at (,) was expected to be matched to a DELETE record with data page and rowid (,), but that data row was not found.` [T-780679] SYBASE ASE: RESOLVED FAULT IN CAPTURING 'TRUNCATE TABLE PARTITION' OPERATIONS FOR SYBASE ASE To fix this bug, upgrade HVR on the capture machine(s). Fixed an issue where Sybase ASE Capture jobs would ignore executions of `truncate table partition` commands. Execution of these commands now correctly force a refresh of the table. It is recommended to check the consistency of any partitioned tables that may have had these commands executed against them. [T-782731] SYBASE ASE: FIXED INFINITE LOOP ON W_JZ1D00/W_JZ1D01 WARNINGS To fix this bug, upgrade HVR on the capture machine(s). Fixed infinite loop on W_JZ1D00/W_JZ1D01 warnings, which occurred during a sanity check of a page in Sybase ASE when multiple records on the page had timestamps earlier than the current timestamp. [T-799532] SYBASE ASE: FIXED CAPTURE LOG FLOODING CAUSED BY W_JZ1D00, W_JZ1D01, AND W_JZ1D02 WARNINGS To fix this bug, upgrade HVR on the capture machine(s). Fixed capture log flooding caused by W_JZ1D00, W_JZ1D01, and W_JZ1D02 warnings. [T-814303] SYBASE ASE: ASE DIRECT CAPTURE - DELETE RECORD FOR DNOOP RECORD IS NOT FOUND To fix this bug, upgrade HVR on the capture machine(s). A further correction on top of T-763583 which better supports deferred UPDATEs when LOB columns are involved. With this change proper adjustments are made when processing DNOOP records to account for the Before images of UPDATE operations when LOBs are involved and the user will no longer experience the error: `text F_JZ270B: The DNOOP record at (,) was expected to be matched to a DELETE record with data page and rowid (,), but that data row was not found.` [T-814453] SYBASE ASE: FIXED F_JT0268 IN A TRANSACTION THAT UPDATES MULTIPLE TABLES. To fix this bug, upgrade HVR on the capture machine(s). Fixed `F_JT0268` in a transaction that updates multiple tables. The issue occurred when a transaction affecting multiple tables triggered a special case: only the ROWIMAGE operation with status ZSY_RWI_XSTAT_WRITETEXT and ZSY_RWI_XSTAT_AFTER_IMAGE was logged for the first table, while the second table's operations were logged in the following order: DNOOP, INOOP, DELETE, and finally INSIND. The first ROWIMAGE caused us to emit `UPDATE_BEFORE`. Since we received DNOOP and INOOP for the second table before the DELETE operation, we did not emit anything on DELETE. Then, when INSIND occurred, we mistakenly believed `UPDATE_BEFORE` for this table had already been emitted, so we incorrectly emitted `UPDATE_AFTER`. NEW FEATURES IN HVR 6.2.0/0 (2024-09-25) ---------------------------------------- [T-763647] DB2 FOR I: ADDED SUPPORT FOR BOOLEAN DATA TYPE IN DB2 FOR I To use this feature, upgrade HVR on all machine(s). HVR now supports the BOOLEAN data type in Db2 for i. Note that some older Db2 for i ODBC drivers have limitations in describing BOOLEAN columns. As a result, HVR will coerce source data to CHAR(1) instead of BOOLEAN. Integrate will still work properly when the source data contains only valid BOOLEAN values, but it may result in: - integer values other than 0 or 1 get silently converted to 1. - a Db2 for i error "SQL0402" for non-integer values. Defining action TableProperties with CoerceErrorPolicy cannot prevent this behavior. [T-777706] POSTGRESQL: DROP SUPPORT FOR POSTGRESQL 9.4 To use this feature, upgrade HVR on all machine(s). This change removes support for PostgreSQL 9.4. PROBLEMS FIXED IN HVR 6.2.0/0 (2024-09-25) ---------------------------------------- [T-752851] FIXED CHANNEL ACTIVATION FAILING WITH F_JR0923 To fix this bug, upgrade HVR on the hub machine(s). Fixed channel activation failing with F_JR0923 even though ColumnProperties with parameter ExpressionScope has parameter Context set, since the Context should only affect refresh/compare, activation shouldn't have been impacted. This fix ensure that context is always considered during the activation checks for ExpressionScope. [T-755955] FIXED F_JX0021 IN SLACK ALERTS CAUSED BY IMPROPER ESCAPING To fix this bug, upgrade HVR on the hub machine(s). Fixed a bug in HVR Slack alerts. In a rare edge case, if the alert was truncated in such a way that left it ending with a backslash, it caused an F_JX0021 error. In addition, the system would fail in creating an event for the alert error because of bad escaping for backslashes and double quotes. This fixes both those problems. [T-761882] FIXED F_JT0D2C WHEN USING OPT-IN BEFORE IMAGE FOR SAP UNPACK To fix this bug, upgrade HVR on the integrate machine(s). Fixed `F_JT0D2C` when using opt-in before image for SAP unpack. Only applies for opt-in cases where `$HVR_SAP_UNPACK_USE_BEFORE` is set. Fixed the sorting of update pairs such that they are kept together when double sorting in some SAP unpack scenarios. [T-762287] IMPROVED DATA COMPARE WITH BOUNDARY SLICING FOR NON-NULLABLE DATATYPE To fix this bug, upgrade HVR on the hub machine(s). Fixed an issue the query of the last slice for data compare with range slicing did try to match for NULL values in its where clause for columns that where non-nullable. For some databases this lead to a full table scan. Now we do not match for NULL values for columns that are not nullable according to the channel definition. If the source database has a nullable key column, and the column is not nullable in the channel definition, the NULL key will now no longer be in any slice when using boundary slicing. [T-776910] FIXED THE DATA TYPE OF HVR_CAP_LOC COLUMN IN HISTORY TABLE To fix this bug, upgrade HVR on the hub machine(s). Fixed history table having hvr_cap_loc column as varchar(5) while it should have it as varchar(12) [T-784478] FIXED BURST TABLES IN CUSTOM SCHEMA ON ORACLE TARGET NOT DROPPING ON DEACTIVATE To fix this bug, upgrade HVR on the integrate machine(s). Fixed burst tables in custom schema on Oracle target not dropping on deactivate [T-784570] FIXED FAILING INTEGRATE ON MYSQL TARGET WITH CUSTOM BURST TABLE SCHEMA DEFINED To fix this bug, upgrade HVR on the integrate machine(s). Fixed failing integrate on MySQL target with Burst integrate and custom burst table schema defined. [T-778939] INGRES: FIXED F_JZ220A IN CASE OF UNEXPECTED TIMESTAMP To fix this bug, upgrade HVR on the capture machine(s). Implemented ZIZ_INGRES_Y2K38_USE_32BIT_TIMESTAMP to fix Ingres F_JZ220A in case of unexpected timestamp. This environment variable is used to choose either to ignore or not ignore the higher 4 bytes in the 8-byte timestamp value captured from Ingres log file. In case of ignore the date is limited by 2038 year. [T-597251] ORACLE: FIXED INVALID DATE WHILE TRYING TO CONVERT DATE TO NUMBER To fix this bug, upgrade HVR on the capture machine(s). Fixed an issue where HVR exception `F_JD20D7: Invalid date encountered while trying to convert date to number of seconds since 01.01.1970 for column` was thrown from an automated test. Issue is fixed now. Issue was introduced by: T-564881 [T-738837] ORACLE: IMPROVED ORACLE CHANGE CAPTURE FOR UNDO CHAINING To fix this bug, upgrade HVR on the capture machine(s). Fixed an issue where Oracle log parsing encountered the error - "F_JZ1122: Chained undo change vector does not have enough structures." This fix allows multiple undo segments to be collated, split blocks to be merged between segments, and ensures proper handling of straddling undo segments. Additionally, the `hvrlogdump` output is now consistent with Oracle's native dumping format. [T-767763] POSTGRESQL: ENHANCED ERROR REPORTING FOR DATABASE SUB-TRANSACTIONS To fix this bug, upgrade HVR on all machine(s). HVR could previously report an F_JD22D4 error without any accompanying details when it occurred within a database sub-transaction. This fix ensures that relevant error details are now included. [T-774367] POSTGRESQL: FIXED ENDING CYCLES IN THE BEGINNING OF CAPTURE To fix this bug, upgrade HVR on the capture machine(s). HVR PostgreSQL capture could fail to end the capture cycles when `HVR_PQ_STREAMING_REPLICATION_RESTART_AT_END_OF_WAL` is `1` which is currently the default for PostgreSQL versions below 16. This change fixes the problem. [T-779065] POSTGRESQL: FIXED ERROR F_JT0921 WHEN ADDING VIEWS TO CHANNEL To fix this bug, upgrade HVR on the hub machine(s). Fixed error F_JT0921 when adding views or materialized views to channel from PostgreSQL. This could be reproduced both on UI and CLI. [T-735358] SQL SERVER: FIXED MSSQL F_JD0A0E ERROR WHEN CAPTURING UPDATE TRANSACTIONS To fix this bug, upgrade HVR on the capture machine(s). Fixed an issue where HVR could fail with error F_JD0A0E when capturing updates for LOB columns, even though those updates had been correctly integrated on the target during a prior capture cycle. [T-717979] SYBASE ASE: FIXED ERROR F_JT0406 WITH TRANSACTION LOG RECORD HANDLING To fix this bug, upgrade HVR on the capture machine(s). Fixed an issue with the handling and pairing of NOOP, INOOP, and DNOOP transaction log records, which are used to form the before and after images of an UPDATE operation. Previously, mishandling these records in some instances caused the following error - "Error Code: F_JT0406 - Scanner encountered 'after update' before the 'before update' change is ready". The fix ensures that DNOOP records are correctly matched by searching backward for their corresponding INOOP records. Additionally, NOOP records are handled properly, canceling the preceding DNOOP records as expected. [T-763583] SYBASE ASE: FIXED ERROR F_JZ270B WHERE THE DELETE RECORD FOR DNOOP WAS NOT FOUND To fix this bug, upgrade HVR on the capture machine(s). Fixed an issue with INSIND record processing during Direct capture, which made adjustments only for After images of UPDATEs and failed to account for Before images that might have been produced earlier, particularly with LOBs and ROWIMAGE records. This issue caused the following error - "F_JZ270B: The DNOOP record at (,) was expected to be matched to a DELETE record with data page and rowid (,), but that data row was not found." This fix ensures that proper adjustments are now made for Before images of UPDATE operations. [T-775721] SYBASE ASE: FIXED REFRESH JOB FOR SYBASE TARGETS WITHOUT CASE-SENSITIVE NAMES. To fix this bug, upgrade HVR on the integrate machine(s). The `Refresh` job was failing for sources with Case-Sensitive Names enabled, and Sybase targets without Case-Sensitive Names. [T-757389] UI: FIXED MANAGED SECRETS OPERATION ON WINDOWS To fix this bug, upgrade HVR on the hub machine(s). Fixed Managed Secrets to use hvrmanagedpassword.bat script name on Windows. [T-775030] UI: FIXED UNEXPECTED ENVIRONMENT ACTIONS APPLIED IN TABLE SELECTION UI To fix this bug, upgrade HVR on the hub machine(s). Fixed unexpected Environment actions being applied when browsing DB schemas on table selection in UI, by passing channel name to the REST API. [T-777475] UI: FIXED REFRESH/COMPARE DIFFERENCE VIEWING PERMISSION REQUIREMENTS To fix this bug, upgrade HVR on the hub machine(s). Changed permission requirements from ReadExecRefresh to ReadExec for querying refresh/compare differences, as compare can be performed by a ReadExec user. [T-779939] UI: FIXED ALERTS NOT SAVING SMTP "SPECIFIC PORT" To fix this bug, upgrade HVR on the hub machine(s). An alert option "Specific Port" under "Advanced SMTP Configuration" section in UI could be cleared (removed, which also means resetting to the default value), if "Advanced SMTP Configuration" section is collapsed. This has been fixed.