Skip to main content

v7.3 Release Notes

Here's what's new in version 7.3.

New Features​

Task Run Queue​

A new Task Run Queue page gives you a single, clear view of the tasks that are currently running or scheduled to run. It's now much easier to see what's in progress and what's waiting to start — all in one place.

Multi-step Tasks​

Some ingestion and export activities are made up of several underlying steps — for example, activities that use both Azure Data Factory and Databricks. Previously these ran as a single step, so if either part failed, the whole activity had to run again, which could result in data being ingested more than once. These activities now run as independent steps, with each step retried on its own. If the ingestion step succeeds but a later step fails, only that failed step is retried — not the entire activity — which avoids re-ingesting data that was already loaded.

Foreman Notifications​

Foreman, the tool we use to manage Insight Factory upgrades and monitor your underlying infrastructure, can now keep you informed directly. You'll receive a notification when your Insight Factory is about to be upgraded, or if there's a cloud outage affecting your environment.

Table Metadata​

When defining a Databricks table in Insight Factory, you can now declare primary key columns, set up foreign key relationships to other tables, add tags, define comments at both the table and column level, and configure Liquid clustering — all directly from the UI. As with Databricks itself, primary and foreign keys are informational and aren't enforced at runtime.

This metadata is included in deployments, so anything you add to a table in your Development environment can be promoted through to Test and Production. We've also added the ability to sync these definitions from Databricks back into Insight Factory. For example, it's common to use Genie within Databricks to generate table comments based on the data; those comments can now be synced back to Insight Factory and deployed to the next environment.

Databricks Compute​

You can now view your Databricks compute directly within the Insight Factory UI, and assign a concurrency threshold to control how many runs use a given compute resource at once.

Lakehouse Editor​

The new Lakehouse Editor brings generic create, read, update, and delete management capabilities directly into Insight Factory, giving you a simpler way to manage your data records from within the platform. Lakehouse Editor is currently in private preview and available for use on a case-by-case basis.

Agentic Observability​

You can now view the inner workings of your agentic workflows directly within the Insight Factory UI. Insight Factory surfaces observability data from your agents so you can see the cost and latency of each run, follow the decisions the agent is making, and inspect the structure and steps of a workflow as it executes.

Enhancements​

  • Scheduling Improvements: You can now schedule task runs with options, and run variables are now resolved at schedule time rather than calculated on the fly, giving you more predictable scheduled runs.
  • Improved Task Groups: Editing Task Groups no longer requires working directly in JSON. A simpler form lets you add and edit items by selecting properties from a drop-down, with the expected data type, a description, and help text shown for each one.
  • Customer Email Address Support: If you log in to Databricks with a different email address than your Insight Factory one, you can now provide that Databricks email address to Insight Factory. This ensures features that rely on your Databricks identity — such as notebook selection and Set Working Folder — work correctly.

Bug Fixes​

  • Fixed an issue that prevented users from deleting permission groups in the UI.
  • Fixed a problem where non-mandatory integer task properties (such as Last Modified Hours) couldn't be cleared back to empty.
  • Fixed a crash that occurred when Admin or Reader users right-clicked an inactive node in a production line graph.
  • Fixed an issue that stopped users from cloning Task Groups.
  • The notebook editor no longer closes when you accidentally click outside the modal, so your work is no longer lost.
  • Fixed deployment so that .py files included as additional notebooks are now imported as files rather than notebooks.
  • Fixed the "Add Production Line" modal for Solutions so that production lines already attached to a Solution no longer appear in the list.

New Activities​

  1. Refresh Databricks Dashboard
    • Refreshes the data behind a Databricks dashboard, and can be triggered automatically once an upstream task completes — just like the existing Power BI refresh activity.

Decommissioned Features​

The standalone Concurrency Thresholds page has been removed. Concurrency thresholds still apply — they're now set as a simple integer directly on the Connection, Integration Runtime, and Databricks Compute resources, rather than managed as separate named thresholds.