Skip to main content
Version: Next

Mode

Certified

Important Capabilities​

CapabilityStatusNotes
Asset Containers✅Enabled by default
Column-level Lineage✅Supported by default
Descriptions✅Enabled by default
Detect Deleted Entities✅Optionally enabled via stateful_ingestion.remove_stale_metadata
Extract Ownership✅Enabled by default
Platform Instance✅Enabled by default
Table-Level Lineage✅Supported by default

This plugin extracts Charts, Reports, and associated metadata from a given Mode workspace. This plugin is in beta and has only been tested on PostgreSQL database.

Report​

/api/{account}/reports/{report} endpoint is used to retrieve the following report information.

  • Title and description
  • Last edited by
  • Owner
  • Link to the Report in Mode for exploration
  • Associated charts within the report

Chart​

/api/{workspace}/reports/{report}/queries/{query}/charts' endpoint is used to retrieve the following information.

  • Title and description
  • Last edited by
  • Owner
  • Link to the chart in Metabase
  • Datasource and lineage information from Report queries.

The following properties for a chart are ingested in DataHub.

Chart Information​

NameDescription
FiltersFilters applied to the chart
MetricsFields or columns used for aggregation
XFields used in X-axis
X2Fields used in second X-axis
YFields used in Y-axis
Y2Fields used in second Y-axis

Table Information​

NameDescription
ColumnsColumn names in a table
FiltersFilters applied to the table

Pivot Table Information​

NameDescription
ColumnsColumn names in a table
FiltersFilters applied to the table
MetricsFields or columns used for aggregation
RowsRow names in a table

Authentication​

See Mode's Authentication documentation on how to generate an API token and password.

Mode does not support true "service accounts", so you must use a user account for authentication. Depending on your requirements, you may want to create a dedicated user account for usage with DataHub ingestion.

Permissions​

DataHub ingestion requires the user to have the following permissions:

  • Have at least the "Member" role.

  • For each Connection, have at least"View" access.

    To check Connection permissions, navigate to "Workspace Settings" → "Manage Connections". For each connection in the list, click on the connection → "Permissions". If the default workspace access is "View" or "Query", you're all set for that connection. If it's "Restricted", you'll need to individually grant your ingestion user View access.

  • For each Space, have at least "View" access.

    To check Collection permissions, navigate to the "My Collections" page as an Admin user. For each collection with Workspace Access set to "Restricted" access, the ingestion user must be manually granted the "Viewer" access in the "Manage Access" dialog. Collections with "All Members can View/Edit" do not need to be manually granted access.

Note that if the ingestion user has "Admin" access, then it will automatically have "View" access for all connections and collections.

CLI based Ingestion​

Starter Recipe​

Check out the following recipe to get started with ingestion! See below for full configuration options.

For general pointers on writing and running a recipe, see our main recipe guide.

source:
  type: mode
  config:
    # Coordinates
    connect_uri: http://app.mode.com

    # Credentials
    token: token
    password: pass

    # Options
    workspace: "datahub"
    default_schema: "public"
    owner_username_instead_of_email: False
    api_options:
      retry_backoff_multiplier: 2
      max_retry_interval: 10
      max_attempts: 5

sink:
  # sink configs

Config Details​

Note that a . is used to denote nested fields in the YAML recipe.

FieldDescription
password ✅
string(password)
When creating workspace API key this is the 'Secret'.
token ✅
string
When creating workspace API key this is the 'Key ID'.
workspace ✅
string
The Mode workspace username. If you navigate to Workspace Settings > Details, the url will be https://app.mode.com/organizations/<workspace-username>. This is distinct from the workspace's display name, and should be all lowercase.
connect_uri
string
Mode host URL.
Default: https://app.mode.com
exclude_restricted
boolean
Exclude restricted collections
Default: False
ingest_embed_url
boolean
Whether to Ingest embed URL for Reports
Default: True
owner_username_instead_of_email
boolean
Use username for owner URN instead of Email
Default: True
platform_instance_map
map(str,string)
tag_measures_and_dimensions
boolean
Tag measures and dimensions in the schema
Default: True
env
string
The environment that all assets produced by this connector belong to
Default: PROD
api_options
ModeAPIConfig
Retry/Wait settings for Mode API to avoid "Too many Requests" error. See Mode API Options below
Default: {'retry_backoff_multiplier': 2, 'max_retry_interva...
api_options.max_attempts
integer
Maximum number of attempts to retry before failing
Default: 5
api_options.max_retry_interval
One of integer, number
Maximum interval to wait when retrying
Default: 10
api_options.retry_backoff_multiplier
One of integer, number
Multiplier for exponential backoff when waiting to retry
Default: 2
api_options.timeout
integer
Timout setting, how long to wait for the Mode rest api to send data before giving up
Default: 40
space_pattern
AllowDenyPattern
Regex patterns for mode spaces to filter in ingestion (Spaces named as 'Personal' are filtered by default.) Specify regex to only match the space name. e.g. to only ingest space named analytics, use the regex 'analytics'
Default: {'allow': ['.*'], 'deny': ['^Personal$'], 'ignoreC...
space_pattern.ignoreCase
boolean
Whether to ignore case sensitivity during pattern matching.
Default: True
space_pattern.allow
array
List of regex patterns to include in ingestion
Default: ['.*']
space_pattern.allow.string
string
space_pattern.deny
array
List of regex patterns to exclude from ingestion.
Default: []
space_pattern.deny.string
string
stateful_ingestion
StatefulStaleMetadataRemovalConfig
Base specialized config for Stateful Ingestion with stale metadata removal capability.
stateful_ingestion.enabled
boolean
Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or datahub_api is specified, otherwise False
Default: False
stateful_ingestion.remove_stale_metadata
boolean
Soft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled.
Default: True

Code Coordinates​

  • Class Name: datahub.ingestion.source.mode.ModeSource
  • Browse on GitHub

Questions

If you've got any questions on configuring ingestion for Mode, feel free to ping us on our Slack.