Skip to main content

Ingest SAP CDC (Full Load Only) as Parquet

Copy data from SAP CDC (Full Load only) to parquet format in Azure Data Lake Storage Gen2.

Category: Ingest to Lakehouse | Tags: Ingestion

How it works

Extract object '<<OdpName>>' against SAP CDC and ingest as parquet into Data Lake location 'raw/<<DataLakeSystemFolder>>/<<DataLakeDatasetFolder>>'

To use this activity within the API, use an ActivityCode of SAP-CDC-FULL-ADLS.

Available Connections​

SourceConnection:

TargetConnection:

Example JSON​

An example of what the Task Config would look like for a task using this activity. Some of these variables would be set at the group level to avoid duplication between tasks.

{
  "SourceConnection": "MY-SOURCE-CONN",
  "OdpContext": "ABAP_CDS",
  "OdpName": "",
  "DataLakeSystemFolder": "my_folder",
  "DataLakeDatasetFolder": "data",
  "TargetConnection": "MY-TARGET-CONN",
  "DeltaSchemaName": "example_schema",
  "DeltaTableName": "my_table"
}

Variable Reference​

The following variables are supported:

  • DataLakeDatasetFolder (Required) - Name of the folder in the Data Lake containing the dataset.

  • DataLakeSystemFolder (Required) - Name of the parent (System) folder in the Data Lake containing the dataset.

  • DeltaSchemaName (Optional) - The name of the Schema this transformation lives in.

  • DeltaTableName (Optional) - The name of the Table representing this transformation.

  • ElevateToDelta (Optional) - Ingest directly to Lakehouse Table

  • IsFederated (Optional) - Makes task available to other Insight Factories within this organisation.

  • Links (Optional) - NULL

  • MaximumNumberOfAttemptsAllowed (Optional) - The total number of times the running of this Task can be attempted.

  • MinutesToWaitBeforeNextAttempt (Optional) - If a Task run fails, the number of minutes to wait before re-attempting the Task.

  • OdpContext (Required) - The context of the ODP data extraction.

  • OdpName (Required) - The name of the data source object to extract.

  • RetainHistory (Optional) - Should the raw files be saved to the History Container to preserve them?

    Show more details

    **Retain History? ** By default, this flag is set to the value assigned in the Configuration item SaveRawFilesToHistory (signalled by the double triangle brackets around the Configuration item name e.g. &lt;&lt;SaveRawFilesToHistory&gt;&gt;). This default behaviour can be overridden here.

  • SizeOfSparkCompute (Optional) - The size of compute used in the Spark cluster.

  • SourceConnection (Required) - Source connection to use.

  • TargetConnection (Optional) - Target connection to use.