Managing Delta Loads and Change Data Capture in Open Hub Destinations

Explanation

Open Hub Destinations are frequently used to feed downstream systems—data lakes, EDW platforms, or custom applications—with incremental changes rather than full reloads. Getting delta handling wrong is one of the most common production issues with Open Hub, leading to duplicate records in target systems, missing deltas after failed loads, or downstream reconciliation failures. When an OHD is fed by a DTP set to delta mode, the DTP tracks its own delta pointer against the source InfoProvider, independent of any other DTPs consuming the same provider. Each successful delta request extracts only records that changed since the last successful delta extraction from that specific DTP. This is important: multiple OHDs or DTPs reading from the same InfoProvider each maintain their own delta queue position, so a failed load in one does not affect the others, but it also means you cannot assume all downstream consumers are synchronized to the same point in time. A critical concept is OHD request status control. Records extracted into an Open Hub table are tagged with a request ID and a status field. Downstream consumers (whether custom ABAP programs, external ETL jobs, or file-based pickups) must read only requests marked as valid/green, and must respect the recommended pattern of processing complete requests only—never partial requests, since a request may still be loading when a downstream job triggers. Some implementations add a control table or 'ready flag' file that is written only after the OHD load and any post-processing complete successfully, giving consuming systems a stable signal for pickup. Delta initialization is another key operational step. Before the first delta load, you must run an init request (or full-with-init pattern) that establishes the delta baseline; this typically extracts full historical data once, then subsequent DTP executions pull only deltas. If the InfoProvider is reloaded, restructured, or the OHD is redefined with new fields, the delta must usually be reinitialized, which requires careful coordination with the downstream target to avoid duplicate ingestion of historical data—commonly the target must truncate and reload or run reconciliation before resuming delta consumption. Error recovery matters significantly for delta integrity. If a delta DTP request fails after partially writing to the OHD table, the request is typically left in red/error status; you should not simply delete and rerun blindly if the target has already partially consumed the erroneous data. Best practice is to fail the request cleanly (do not let downstream jobs consume red-status requests), investigate root cause (source data, mapping errors, connectivity), then repeat the DTP request, which reprocesses the same delta package. Repeating a delta request is safe from BW's perspective because it does not advance the delta pointer past a successfully completed extraction—but downstream idempotency (target-side deduplication or upsert logic) is still recommended as a safety net. For OHDs writing to database tables (Open Hub Destination type Database Table) versus flat files or third-party destinations, the operational pattern differs: file-based delta requires strict file archiving and naming conventions per request, while database table destinations may support append-only or overwrite-per-request patterns depending on configuration. Third-party destination type requires a custom class implementing the Open Hub service interface, and delta handling logic must be explicitly coded to interpret the request status and data package structure correctly—this is a common area for defects when custom ABAP developers are unfamiliar with OHD semantics. Monitoring delta health in production typically involves checking process chain logs for the OHD DTP step, reviewing request statistics (record counts per delta run—unexpected zero-record deltas over multiple periods can indicate a stalled source, while unexpectedly large deltas may indicate a reinitialization was triggered upstream), and periodically reconciling record counts or checksums between BW source InfoProvider and the downstream target to catch silent data quality drift.

Code example

* Example: ABAP logic pattern for a Third-Party Open Hub Destination* implementing the data transfer interface to control delta handling* (Illustrative only - actual method signatures depend on the* Open Hub Destination API version in your BW release) METHOD if_rsbo_service~transfer_data.  DATA: lv_request_status TYPE rsreqdone.   " 1. Read request status metadata from the Open Hub context  " Only process requests flagged as technically complete  CALL METHOD get_request_status    IMPORTING      ev_status = lv_request_status.   IF lv_request_status NE 'R'. " 'R' = fully processed / green    " Do not consume partially loaded or erroneous requests    RETURN.  ENDIF.   " 2. Read data package structure and apply upsert logic  " downstream to guard against reprocessed delta packages  LOOP AT it_data_package INTO DATA(ls_record).    PERFORM upsert_target_record(ls_record).  ENDLOOP.   " 3. Write a completion marker only after full package commit  PERFORM write_ready_flag( iv_request_id = it_data_package-request_tsn ).ENDMETHOD.

Real project scenario

A retail analytics team used an Open Hub Destination on a sales InfoCube to feed a downstream cloud data lake nightly via a database table destination and a custom pickup job. After a process chain failure mid-load, the support team deleted the failed request and reran the DTP without checking whether the pickup job had already partially read the table. This caused a partial duplicate day of sales data in the data lake. The fix involved adding a request-status check to the pickup job (only reading requests with green status), introducing a target-side upsert key on order line plus request timestamp, and documenting a runbook step requiring reconciliation of record counts before manually rerunning any failed delta request.

Common mistakes

• Allowing downstream jobs to consume Open Hub table data before the request status is confirmed green/complete • Deleting and rerunning a failed delta DTP request without checking whether downstream systems already partially consumed the data • Assuming all OHDs and DTPs on the same InfoProvider share one delta pointer, when each DTP tracks its own independently • Reinitializing delta after a source reload without coordinating a corresponding reload or reconciliation on the downstream target • Building custom third-party OHD classes without properly checking request completion status, leading to processing of partial data packages • Not implementing idempotent (upsert-based) logic on the downstream target as a safety net against reprocessed delta requests

Best practices

• Always gate downstream consumption on OHD request status (only process complete/green requests) • Implement target-side upsert or deduplication logic keyed on business key plus request identifier as a safety net • Document and rehearse the delta reinitialization procedure, including downstream reconciliation steps • Monitor delta record counts over time to catch silently stalled or abnormally large delta extractions • Keep each downstream consumer on its own dedicated DTP rather than sharing one DTP output across multiple unrelated targets • For custom third-party destination classes, explicitly implement and test request status handling rather than assuming full-package delivery

Interview angle

Interviewers assess whether you understand that Open Hub delta tracking is per-DTP, not global, and that downstream consumers must respect request status semantics rather than blindly polling target tables. Be ready to explain the difference between delta initialization and delta extraction, why request status checks matter for data integrity, and how you would design idempotent downstream consumption to tolerate DTP reruns safely.