Create View Notebook Template

Input Parameters

All Notebook Parameters (if any) are contained in the dictionary variable 'params'. There are two ways to get an individual parameter from params, in both cases the parameter name is case-sensitive:

1. Use dot-notation - refer to the example below:

   params = { "Name": "Test", "Values": { "Title": "Results", "Results": [ { "Definition": "Core Sample", "Outcome": "Prospective" }, { "Definition": "Follow-up", "Outcome": "For review" } ] } }
   
   params.Name produces 'Test'
   params.Values.Title produces 'Results'
   params.Values.Results[0] produces { "Definition": "Core Sample", "Outcome": "Prospective" }
   params.Values.Results[1].Definition produces 'Follow-up'

2. Use the search_dictionary function as follows:

   var1 = search_dictionary(params, "parameter-name")

   There is an optional third parameter to this function: value_to_return_if_not_found - this is the value to return if the particular parameter is not found in params.

   Note that value_to_return_if_not_found can take on any type (string, int, boolean, struct, ...) e.g., search_dictionary(params, "IncorrectlyNamedParameter", False) will return the boolean False if "IncorrectlyNamedParameter" is not found in params.

CAUTION: There is another dictionary variable, 'config', that contains all of the configuration sent to this notebook. In most cases you will have no use for 'config', but if you choose to use 'config' in this notebook, note the following:

• Access the individual parameters within config by using the search_dictionary function e.g., search_dictionary(config, "ParameterName"). Dot-notation access does not apply to 'config'.
• Heed this WARNING - The individual parameter names within 'config' are subject to change outside of your control, which may break your code.

Adding your Code

Add your code in cells located between the markdown cells *** BEGIN Code Block *** and *** END Code Block ***.

Terminology

• A view that references datasets wholly contained in the same catalog that the view is defined in is referred to as a contained view.
• A view that references datasets in a catalog that is different to the catalog that the view is defined in is referred to as a cross-catalog view. Cross-catalog views are an important sharing mechanism in multi-domain businesses (where each business domain controls its own catalog) whereby data that resides in one catalog can be referenced by a view that is defined in another. In this scenario, the catalog where the data resides is referred to as the provider catalog and the catalog where the view is defined is known as the consumer catalog. It is important to note that for cross-catalog views, the permissions applied are based on the owner of the view.

Pre-requisites For Cross-Catalog Views

If you are creating a cross-catalog view, there are some permissions that must be assigned in the provider catalog prior to creating the view, as follows.

• USE CATALOG on the provider catalog
• USE SCHEMA on the provider schema
• SELECT on the provider table(s)

These permissions must be assigned to the owner of the view (or a group that the owner is in).

Note: It is also useful to assign these same permissions to the consumer catalog's developers group to enable the developers in the consumer catalog read access to the provider catalog datasets.

Guidance

1. Explicitly Declare All Columns

As a best practice, explicitly declare all columns you wish the view to return.

In your view definition, avoid the temptation to use an asterisk (*) to indicate all columns of a source table. The column list represented by * is baked in when you create or replace the view and will not be automatically refreshed if the structure of the source table changes. Additionally, you will have no visibility of which columns the view will return until you use the view.

2. Restrict use of Three-Part-Naming

When writing the view definition for a contained view, do not use three-part-naming. Simply use the two-part-naming syntax schema.table for the datasets you need to reference.

The only scenario where the usage of three-part-naming is required is when you are creating cross-catalog views. In this scenario, refer to the tables in the provider catalog using full three-part-naming syntax provider-catalog.schema.table. Use the notebook variable provider_catalog in your view definition (provider_catalog comes the metadata item Provider Catalog in your Task) to refer to the datasets in the provider catalog. For example:

view_definition = f"""
SELECT
   col1
   ,col2
FROM {provider_catalog}.schema.table;
"""

3. Employing Governance Logic

As views will consider the user context of the view owner rather than the view consumer, you may need to apply governance logic to the view to appropriately manage what view consumers can and cannot see. This is sometimes referred to as a dynamic view - you can read more in the Databricks dynamic views documentation.

Variables for Cross-Catalog Views

The provider catalog that you will reference in your cross-catalog view is stored in the variable provider_catalog. This variable is set in the cell below (please do not edit or delete this cell). Use this variable when you create your script for the view definition.

Returning metrics to record against the Task Run

If you would like to return one or more metrics regarding the running of the code in this notebook, simply declare a variable 'run_output' and populate it with a valid JSON string containing your metrics. At the end of the execution of this notebook, the value of run_output will be recorded against the Task Run record. For example, to record the version number of the model that is used to run inference, you might do something like:

run_output = '{ "model_version_number": 5 }'

Running this notebook directly in Databricks

This notebook can be run directly from your Databricks Workspace. If the notebook relies on Notebook Parameters, please read the following instructions:

1. Add this line of code to a cell at the top of your notebook and run that cell.

   dbutils.widgets.text('ParametersJSON', '{ "NotebookParameters": { "param1": "value1", "param2": "value2" } }')

2. This will add a parameter to the notebook. Simply replace (or remove) the pre-canned parameters, 'param1', 'param2', and their values with your own.
3. When you have finished running this notebook directly in Databricks, comment out the line of code you added or delete the cell entirely.

You can add any testing or debugging logic you like to any cells below the markdown cell Testing/Debugging Zone. These cells will be skipped when this notebook is run via Insight Factory.