Real-Time Data Integration

Real-Time Change Data Capture

The iWay Integration Suite's flexible solutions for Real-Time Data Warehousing enable you to:

  • Improve timeliness of information in a data warehouse
  • Detect real-time events from applications and databases as content for feeds
  • Dramatically reduce ETL run time by only processing changes
  • Avoid impact on source systems with a non-intrusive log-scraping technology
  • Simplify development efforts by using a CDC system completely integrated with ETL
  • Leverage existing investment in iWay and/or WebFOCUS

Service Manager provides real-time event listening capabilities for over 300+ sources including all applications, databases, cloud, and legacy systems. Each event can trigger a variety of business processes while feeding the data warehouse in real-time.

Features include:

  • Bi-directional capability to feed a warehouse or extract data
  • Advanced transformation and data dictionary capabilities
  • Integration with data quality services to cleanse data before it is fed into the warehouse
  • Master Data Management of data inside and outside the warehouse
  • Advanced message based SOA architecture for seamless integration with business processes

DataMigrator Change Data Capture (CDC) provides a real-time capability using database logs to read only the changes (inserts, updates, and deletions) made to tables in any of the major relational databases and delivers those changes to DataMigrator.

With shrinking batch windows and the need for up-to-date information, traditional batch-processing is sometimes unable to handle today's business needs. There is often too much data to do a full refresh of the data warehouse every night. At the same time, business analysts want their analytical databases to have the most current information. These needs can be addressed by using CDC to provide access to database changes almost as soon as they happen.

All the major relational databases log changes made to database tables for error recovery. While these logs (or journals) can be used to detect changes, every database uses a unique format. On IBM Mainframes there is nonative logging, so a subsystem is a provided that stores records in a logstream using standard VSAM journaling (JRNAD).

DataMigrator CDC makes database logs from disparate databases available in a common format so that they can be used as a data source. By using CDC data sources instead of full source tables, processing time can be greatly reduced, freeing up batch windows. Processing the changes as they occur lets a data warehouse provide near real-time access to operational data.

DataMigrator CDC is completely integrated and all configuration is done from DataMigrator's Data Management Console. A polling interval (how often to check for changes) can be specified as often as once a minute. A timeout interval (how long to keep checking) can also be specified.

Once a CDC source is configured, it can be read using SQL and used like other data sources. A Data Flow (ETL job) built for an initial load can be easily converted to perform CDC processing.