Prerequisites

Alation Cloud Service Applies to Alation Cloud Service instances of Alation

Customer Managed Applies to customer-managed instances of Alation

dbt Cloud

Set Up New a Project

Create a project with required models. For more information on setting up a project see dbt Project. Also, you must execute required jobs and tests for the connector to extract. To know about models, jobs, and tests see dbt models and Tests.

Generate a Service Account Token

A Service token specifies a unique access token of an account used to assign specific set of permissions to manage access. Service tokens are the preferred method to enable access on behalf of the dbt Cloud account.

To generate a service account token, see Generate Service Account Tokens.

Important

In the New service token window, you must select Read-Only and All projects in the Permission set and Project fields, respectively.

Note

Optionally, instead of a service token you can obtain a Personal Access Token (PAT). A PAT specifies a unique access token of an account.

Enable ELT Source in Alation

To enable an ELT source in Alation, contact Alation Support.

dbt Core

Set Up the dbt Projects

You must have the following file structure in your dbt project:

For single environment

The <project_name> is a placeholder that represents a specific dbt project.

Note

Alation doesn’t support ODBC project type.

Example

Alation considers the production environment by default if you don’t specify an environment.

Example

For multiple environments

Note

Use this project structure only if you plan to catalog development or staging sources or targets in Alation.

Example

The table below describes the various files that you must have under dbt project or environment within the project:

File	Description	How to Generate
manifest.json	Fetches the dbt model information and also builds table level lineage	Using the dbt build command
run_results.json	Fetches the job run and test run information to build data health	Using the dbt build command
catalog.json	Fetches the dbt model column information and also builds column-level lineage	Using the dbt docs generate command
env_details.json	Fetches the host and port information to build source system information and table or column level lineage	manual or custom build

Alation recommends using the dbt build command to generate the manifest.json and run_results.json files. However, if you choose to use the dbt run or dbt test commands to create these files (manifest.json using dbt run and run_results.json using dbt run or dbt test) and place the files in the project folder structure. Additionally, if there are multiple run_results.json files (one each from dbt run and dbt test), rename them and place them in the folders as shown below:

You can manually create the env_details.json file. Here’s an example of the env_details.json file created manually for Databricks Unity Catalog.

{
    "host":"dbc-25e69bfd-44ed.cloud.databricks.com",
    "port": 443
}

Note

If you are using Databricks without Unity Catalog, after creating the env_details.json file, you must set unityCatalog as false in the env_details.json file.

{
    "host":"dbc-25e69bfd-44ed.cloud.databricks.com",
    "port": 443,
    "unityCatalog": false
}

Important

If a project is not in the required structure, the connector will skip metadata and lineage extraction for that project.

Set Up Projects on Amazon S3

1. Create an S3 bucket in your AWS account or use an existing one.

2. Create an AWS IAM user with the following permissions. Replace BUCKET_NAME with the actual name of your S3 bucket.

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "VisualEditor0",
               "Effect": "Allow",
               "Action": [
                   "s3:GetObject",
                   "s3:ListBucket"
               ],
               "Resource": [
                   "arn:aws:s3:::BUCKET_NAME/*",
                   "arn:aws:s3:::BUCKET_NAME"
               ]
            }
       ]
   }
   

For more information on creating an IAM user with the required permissions, see Create an IAM User in Your AWS Account.

Set Up Projects on GitHub

1. Create a GitHub repository or use an existing one.

2. Create a GitHub access token. For details on how to create a personal access token, see Managing your personal access tokens.

3. In your GitHub repository page, go to Settings > Developer Settings > Personal Access tokens > Tokens(classic) and add the repo access to the token.

   

Set Up Projects on Azure Blob Storage

1. Create an Azure Blob Storage account in your Azure account or use an existing one.

2. Create a storage access key or Shared Access Signature. For details on how to create a storage access key, see Use the account access key and for Shared Access Signature, see Create a storage SAS.

   The storage access key must have full access to the storage account. Similarly, the Shared Access Signature must have permissions as shown in the image below:

   

Upload the Project

After you set up the project structure, upload the projects with the prepared files to GitHub, Amazon S3, or Azure Blob Storage based on your preferred storage location.

Note

If you already have projects with required files in GitHub, Amazon S3, or Azure Blob Storage, you can use the connector to extract metadata and lineage, provided the projects are in the required structure.

Enable ELT Source in Alation

To enable an ELT source in Alation, contact Alation Support.