Skip to main content

Export Notebook Results to Dataverse Online (via Service Principal)

Export data from a Databricks notebook to Dataverse Online (accessed via Service Principal).

Category: Export Data from Lakehouse | Tags: Export

How it works

Run Databricks Notebook '<<NotebookPath>>' and copy results to Dataverse Online entity '<<EntityName>>'

To use this activity within the API, use an ActivityCode of DELTA-DATAVERSE.

Available Connections​

SourceConnection:

TargetConnection:

Example JSON​

An example of what the Task Config would look like for a task using this activity. Some of these variables would be set at the group level to avoid duplication between tasks.

{
  "SourceConnection": "MY-SOURCE-CONN",
  "NotebookPath": "/Users/fred.nurks@example.com/MyRepo/My Notebook",
  "TargetConnection": "MY-TARGET-CONN",
  "EntityName": "",
  "NotebookParameters": { "Param1": "Value1", "Param2": "Value2" }
}

Variable Reference​

The following variables are supported:

  • AlternateKeyName (Optional) - Specify the alternate key name defined on your entity to upsert records.

  • DatabricksClusterId (Optional) - The Databricks Cluster to use for this task.

  • DIUsToUseForCopyActivity (Optional) - Specifies the powerfulness of the copy executor. Value can be between 2 and 256. When left at default, the Data Factory dynamically applies the optimal DIU setting based on the source-sink pair and data pattern.

  • EntityName (Required) - Name of the entity within Dataverse (Common Data Service for Apps).

  • IgnoreNullValues (Optional) - Indicates whether to ignore null values from input data during write operation.Allowed values are: true, and false. - true: leave the data in the destination object unchanged when doing upsert/update operation, and insert defined default value when doing insert operation. - false: update the data in the destination object to NULL when doing upsert/update operation, and insert NULL value when doing insert operation.

  • IsFederated (Optional) - Makes task available to other Insight Factories within this organisation.

  • Links (Optional) - NULL

  • MaximumNumberOfAttemptsAllowed (Optional) - The total number of times the running of this Task can be attempted.

  • MinutesToWaitBeforeNextAttempt (Optional) - If a Task run fails, the number of minutes to wait before re-attempting the Task.

  • NotebookParameters (Optional) - Parameters for use in the Databricks Notebook. This is JSON format e.g. { "Param1": "Value1", "Param2": "Value2" }.

  • NotebookPath (Required) - The relative path to the Databricks Notebook.

  • SaveExportToHistory (Optional) - Save the exported data to history in the Lakehouse?

  • SourceConnection (Required) - Source connection to use.

  • TargetConnection (Required) - Target connection to use.

  • WriteBehaviour (Optional) - The method of writing data to the target.