Script Manifest (YAML) Files

This page provides an outline of how to setup a Script Manifest File within the Daasity App to configure the transformation that will execute as part of a workflow

What are script manifest files?

Script manifests are files used in workflows to dictate which transformation scripts should run when. They are YAML files that list a sequence of SQL scripts that should be executed in a particular order.

Script manifests can be used to run SQL scripts located in your custom code repository or in one the Daasity shared script repositories.

The daily_incremental.yml template file

When you get set up with Daasity, you are automatically set up with a daily workflow that is tied to a daily_incremental.yml script manifest file. This file by default is blank (except for a set of comments).

If you want to get a jumpstart on setting up your daily incremental transformation code, you can copy and paste some or all of the code from our template daily_incremental.yml file (located in our shared code repository) into your custom code repo's daily_incremental.yml file.

A snapshot of our template daily_incremntal.yml file

Other template manifest files

Daasity will maintain script manifest files in the following folder https://github.com/Daasity/platform-sql-shared/tree/master/scripts/script_manifest_files which you can reference to help create your own versions.

This folder currently has the following sample script manifest files:

• daily_incremental.yml: this script manifest file contains all the scripts needed to run the entire workflow and includes the code for every integration that has base code. You can remove the appropriate lines to build your version as well as reference this file to find the most recent additions / changes

• attribution_2_0.yml: this script manifest file contains all the scripts needed to run the Attribution 2.0 feature

• attribution_2_0_reset.yml: this script manifest file contains all the scripts needed if you have changed the Attribution Setting in the BSD and need to reset the tables to reflect the new attribution settings

Working with script manifest files

Structure

Here is a basic example of a daily_incremental.yml. This example file would run a set of scripts if you have a Shopify integration and another set of scripts if you have a Google Analytics integration:

daily_incremental.yml

version: "2.0.0"

sections:
  shopify_stuff:
    conditional_integrations: "shopify" # Makes it so the scripts in this section will only run if you have an active Shopify integration
    scripts: 
      - "some_shopify_script_in_your_repo.sql"
      - "github://platform-sql-shared/scripts/some_shopify_script_in_the_Daasity_shared_repo.sql"
      
  google_analytics_stuff:
    conditional_integrations: "google_analytics" # Makes it so the scripts in this section will only run if you have an active Google Analytics integration
    scripts: 
      - "some_google_analytics_script_in_your_repo.sql"
      - "github://platform-sql-internal/scripts/some_google_analytics_script_in_the_Daasity_internal_repo.sql"
      
    

This example has the following important components to every script manifest file:

• version - The current version number. This should not be changed from "2.0.0".

• sections - This key must be included, and should only be included once. Each section within it is just a way to group a set of scripts together.

  • <section_name> - Identifier for a group of scripts. Must be one word, or multiple words in snake case. There can be as many sections in your manifest file as you see fit.

    • conditional_integrations * - A boolean statement that specifies which integrations must be active and authorized in your Daasity account for the scripts in the current section to run. The statement should be enclosed in single or double quotes. You can use AND and OR logic in this statement, and you can use parentheses to group the statements. For example: conditional_integrations: "google_analytics AND (bigcommerce or shopify)" will make it so that the scripts in the section will run only if there is a Universal Analytics integration and either a Shopify or BigCommerce integration in the account. Learn about using proper integration references here. The conditional_integrations tag is optional. If you leave it out, all of the scripts in the section will run no matter what.

      • * There is a legacy integrations tag that works similarly and is still supported. The legacy tag supports a list of integrations, all of which are treated with OR logic, i.e. if any of the integration in the list is active and authorized in the account, the scripts in the section will run. You should not combine an integrations and a conditional_integrations tag in the same section.

    • scripts - A list of SQL script files that will be executed in sequential fashion, top to bottom. Read the Referencing script files section below to help you understand how to reference different types of files.

YAML formatting

It is important to pay attention to your file formatting or your script manifest file may not work.

• Indentation in the file matters!

• Each level of indentation needs to be 2 spaces (not tabs)

  • version: & sections: should not be preceded by any spaces

  • The section name should be indented 2 spaces from the beginning of the line

  • integrations: and scripts: should be indented 4 spaces from the beginning of the line

  • Individual script and integration list items should be indented 6 spaces from the beginning of the line

• Comments may be specified using a hash symbol #

Referencing script files

In the example file snippet above, you can see that there are two different types of file paths for the SQL scripts being referenced.

Local script files

- "some_shopify_script_in_your_repo.sql"
- "some_google_analytics_script_in_your_repo.sql"

This is how you would reference files that exist in your merchant repo. In this example, both files are in the top-level folder of your merchant repo. If you are going to be referencing custom scripts, they will look like these examples.

Remote script files

- "github://platform-sql-shared/scripts/some_shopify_script_in_the_Daasity_shared_repo.sql"
- "github://platform-sql-internal/scripts/some_google_analytics_script_in_the_Daasity_internal_repo.sql"

This is how you would reference base Daasity transform scripts located in the Daasity shared and internal repos.

References to script files that exist outside of your custom code repository must use a specific syntax. The syntax can be composed of the following pieces:

• GIT repository provider: currently only GitHub is supported (github://)

• Git repository name (e.g.: platform-sql-shared)

• file folder path (e.g.: /scripts/)

• file name with the explicit file extension .SQL (e.g.: some_shopify_script_in_the_Daasity_shared_repo.sql)

• optionally a branch name: this option should be rarely used and should never be used in a production deployment. If a branch is not specified it assumes master

Referencing integrations

When adding an integration dependency, you must use a valid integration reference. If you have access to our shared code repo, you can find the full list of valid integration references in the config/integrations_map.yml file.

Manifest files must be in root folder

Your script manifest files must remain within the root folder of your custom code repository, but your transformation code can be in any folder.