Some reasons for using a data pump include the following:
1. Protection against network and target failures:
In a basic Oracle Golden Gate configuration, with only a trail on the
target system, there is nowhere on the source system to store the data
operations that Extract continuously extracts into memory. If the
network or the target system becomes unavailable, Extract could run
out of memory and ABEND However, with a trail and data pump on the
source system, captured data can be moved to disk, preventing the
ABEND of the primary Extract. When connectivity is restored, the data
pump captures the data from the source trail and sends it to the target
system(s).
2. You are implementing several phases of data filtering or transformation.
When using complex filtering or data transformation configurations, you
can configure a data pump to perform the first transformation either on
the source system or on the target system, or even on an intermediary
system, and then use another data pump or the Replicate group to
perform the second transformation.
3. Consolidating data from many sources to a central target.
When synchronizing multiple source databases with a central target
database, you can store extracted data operations on each source
system and use data pumps on each of those systems to send the data
to a trail on the target system. Dividing the storage load between the
source and target systems reduces the need for massive amounts of
space on the target system to accommodate data arriving from multiple
sources.
4. Synchronizing one source with multiple targets. When sending data to
multiple target systems, you can configure data pumps on the source
system for each target. If network connectivity to any of the targets
fails, data can still be sent to the other targets.
Overview of Replicat
The Replicat process runs on the target system, reads the trail on that system, and then reconstructs the DML or DDL operations and applies them to the target database. Yo can configure Replicat in one of the following ways:
Initial loads:
For initial data loads, Replicat can apply a static data copy to target objects or route it to a high-speed bulk-load utility.
Change synchronization:
When configured for change synchronization, Replicat applies the replicated source operations to the target objects using a native database interface or ODBC, depending on the database type.
To preserve data integrity, Replicate applies the replicated operations in the same order as they were committed to the source database.
You can use multiple Replicate processes with multiple Extract processes in parallel to increase throughput. To preserve data integrity, each set of processes handles a different set of objects. To differentiate among Replicat processes, you assign each one a group name.
You can delay Replicate so that it waits a specific amount of time before applying the replicated operations to the target database. A delay may be desirable, for example, to prevent the propagation of errant SQL, to control data arrival across different time zones,or to allow time for other planned events to occur. The length of the delay is controlled by the DEFERAPPLYINTERVAL parameter.
Overview of trails
To support the continuous extraction and replication of database changes, Oracle Golden Gate stores records of the captured changes temporarily on disk in a series of files called a trail.
A trail can exist on the source system, an intermediary system, the target system, or any combination of those systems, depending on how you configure Oracle GoldenGate.
On the local system it is known as an extract trail (or local trail). On a remote system it is known as a remote trail. By using a trail for storage,
Oracle Golden Gate supports data accuracy and fault tolerance The use of a trail also allows extraction and replication activities to occur independently of each other. With these processes separated, you have more choices for how data is processed and delivered. For example, instead of extracting and replicating changes continuously, you could extract changes continuously but store them in the trail for replication to the target later, whenever the target application them. Processes that write to, and read, a trail The primary Extract and the data-pump Extract write to a trail. Only one Extract process can write to a trail, and each Extract must be linked to a trail.
Processes that read the trail are:
1. Data-pump Extract: Extracts DML and DDL operations from a local trail that is linked to a previous Extract (typically the primary Extract), performs further processing if needed, and transfers the data to a trail that is read by the next Oracle GoldenGate process downstream (typically Replicat, but could be another data pump if required)
2. Replicat: Reads the trail and applies replicated DML and DDL operations
to the target database.
Trail creation and maintenance :
The trail files themselves are created as needed during processing, but you specify a two-character name for the trail when you add it to the Oracle GoldenGate configuration with the ADD RMTTRAIL or ADD EXTTRAIL command. By default, trails are stored in the dirdat sub-directory of the Oracle GoldenGate directory.
Full trail files are aged automatically to allow processing to continue without interruption for file maintenance. As each new file is created, it inherits the two-character trail name appended with a unique, six-digit sequence number from 000000 through 999999 (for example c:\ggs\dirdat\tr000001).
When the sequence number reaches 999999, the numbering starts over at 000000.You can create more than one trail to separate the data from different objects or applications. You link the objects that are specified in a TABLE or SEQUENCE parameter to a trail that is specified with an EXTTRAIL or RMTTRAIL parameter in the Extract parameter file. Aged trail files can be purged by using the Manager parameter
PURGEOLDEXTRACTS.
To maximize throughput, and to minimize I/O load on the system, extracted data is sent into and out of a trail in large blocks. Transactional order is preserved. By default, Oracle GoldenGate writes data to the trail in canonical format, a proprietary format which allows it to be exchanged rapidly and accurately among heterogeneous databases.
However, data can be written in other formats that are compatible with different
applications.
Overview of extract files
In some configurations, Oracle GoldenGate stores extracted data in an extract file instead of a trail. The extract file can be a single file, or it can be configured to roll over into multiple files in anticipation of limitations onfile size that are imposed by the perating system. In this sense, it is similar to a trail, except that checkpoints are not recorded. The file or files are created automatically during the run. The same versioning features that apply to trails also apply to extract files.
Overview of checkpoints
Checkpoints store the current read and write positions of a process to disk for recovery purposes. Checkpoints ensure that data changes that are marked for synchronization actually are captured by Extract and applied to the target by Replicat, and they prevent redundant processing.
They provide fault tolerance by preventing the loss of data should the system, the network, or an Oracle GoldenGate process need to be restarted. For complex synchronization configurations, checkpoints enable multiple Extract or Replicat processes to read from the same set of trails.
Checkpoints work with inter-process acknowledgments to prevent messages from being lost in the network. Oracle GoldenGate has a proprietary guaranteed-message delivery technology.
Extract creates checkpoints for its positions in the data source and in the trail. Because Extract only captures committed transactions, it must keeptrack of operations in all open transactions, in the event that any of them are committed. This requires Extract to record a checkpoint where it is currently reading in a transaction log, plus the position of the start of the oldest open transaction, which can be in the current or any preceding log.
To control the amount of transaction log that must be re-processed after an outage, Extract persists the current state and data of processing to disk at specific intervals, including the state and data (if any) of long-running transactions. If Extract stops after one of these intervals, it can recover from a position within the previous interval or at the last checkpoint, instead of having to return to the log position where the oldest open long-running transaction first appeared.
Replicat creates checkpoints for its position in the trail. Replicat stores its checkpoints in a checkpoint table in the target database to couple the commit of its transaction with its position in the trail file. The checkpoint table guarantees consistency after a database recovery by ensuring that a transaction will only be applied once, even if there is a failure of the Replicat process or the database process. For reporting purposes, Replicat also has a checkpoint file on disk in the dirchk sub-directory of the Oracle GoldenGate directory. Checkpoints are not required for non-continuous types of configurations that can be re-run from a start point if needed, such as initial loads.
Overview of Manager
Manager is the control process of Oracle GoldenGate. Manager must be running on each system in the Oracle GoldenGate configuration before Extract or Replicat can be started, and Manager must remain running while those processes are running so that resource management functions are performed.
Manager performs the following functions:
1. Start Oracle GoldenGate processes
2. Start dynamic processes
3. Maintain port numbers for processes
4. Perform trail management
5. Create event, error, and threshold reports
6. One Manager process can control many Extract or Replicat processes.
Overview of Collector
Collector is a process that runs in the background on the target system when
continuous, online change synchronization is active.
Collector does the following:
Upon a connection request from a remote Extract to Manger, scan and bind to
unavailable port and then send the port number to Manager for assignment to the requesting Extract process. Receive extracted database changes that are sent by Extract and write them to a trail file. Manager starts Collector automatically when a network connection is required, so Oracle GoldenGate users do not interact with it.can be re-run from a start point if needed, such as initial loads.
Collector can receive information from only one Extract process, so there is one
Collector for each Extract that you use. Collector terminates when the associated Extract process terminates.
Collector can be run manually, if needed. This is known as a static Collector (as opposed to the regular, dynamic Collector). Several Extract processes can share one static Collector; however, a one-to-one ratio is optimal. A static Collector can be used to ensure that the process runs on a specific port.
By default, Extract initiates TCP/IP connections from the source system to Collector on the target, but Oracle GoldenGate can be configured so that Collector initiates connections from the target. Initiating connections from the target might be required if, for example, the target is in a trusted network zone, but the source is in a less trusted
zone.
Overview of process types
Depending on the requirement, Oracle GoldenGate can be configured with the
following processing types.
1. An online Extract or Replicat process runs until stopped by a user. Online processes maintain recovery checkpoints in the trail so that processing can resume after interruptions. You use online processes to continuously extract
and replicate DML and DDL operations (where supported) to keep source and target objects synchronized. The EXTRACT and REPLICAT parameters apply to this process type.
2. A source-is-table Extract process extracts a current set of static data directly from the source objects in preparation for an initial load to another database. This process type does not use checkpoints. The SOURCEISTABLE parameter applies to this process type.
3. A special-run Replicat process applies data within known begin and end points. You use a special replicat run for initial data loads, and it also can be used with an online Extract to apply data changes from the trail in batches, such as once a day rather than continuously. This process type does not maintain checkpoints, because the run can be started over with the same begin and end points. The SPECIALRUN parameter applies to this process type.
4. A remote task is a special type of initial-load process in which Extract communicates directly with Replicat over TCP/IP. Neither a Collector process nor temporary disk storage in a trail or file is used. The task is defined in the Extract parameter file with the RMTTASK parameter.
Overview of groups
To differentiate among multiple Extract or Replicat processes on a system, you define processing groups. For example, to replicate different sets of data in parallel, you would create two Replicat groups.
A processing group consists of a process (either Extract or Replicat), its parameter file, its checkpoint file, and any other files associated with the process. For Replicat, a group also includes the associated checkpoint table.
You define groups by using the ADD EXTRACT and ADD REPLICAT commands in the Oracle GoldenGate command interface, GGSCI.
All files and checkpoints relating to a group share the name that is assigned to the group itself. Any time that you issue a command to control or view processing, you supply a group name or multiple group names by means of a wildcard.
Overview of the Commit Sequence Number (CSN)
When working with Oracle GoldenGate, you might need to refer to a Commit Sequence Number, or CSN. A CSN is an identifier that Oracle GoldenGate constructs to identify a transaction for the purpose of maintaining transactional consistency and data integrity. It uniquely identifies a point in time in which a transaction commits to the database. The CSN can be required to position Extract in the transaction log, to reposition Replicat in the trail, or for other purposes. It is returned by some conversion functions and is included in reports and certain GGSCI output.
Supported Databases
