The Customizable Business Logic feature allows merchants to build upon the Daasity transformations scripts. Daasity is opening up the data model and some transformations so merchants can leverage these shared scripts and inject customizations.
PLEASE NOTE: This Feature is For Daasity PRO Merchant Accounts ONLY and requires a GitHub user account.
Please reach out to support@daasity.com to enable this feature for you.
KEY TOPICS
Click on the links below to take you to the section to learn more.
- Feature Description
- What is the Merchant Repo for my Account?
- Team Management
- Customizing the YAMLs
- What is platform-sql-shared repo
- Adding custom scripts to my repo
- Testing customized YAML files
- Publish / Go Live for Daily Workflow
- Pro Workflows: Workflow Manifest File
Feature Description
All Pro accounts have their own specific Merchant Repository at GitHub. The Customization team at Daasity uses these repositories to make updates on behalf of the Merchant.
The Merchant may also request access to their repo through the Daasity App.
Once approved, Users have Access to Read and Write to their Repo and will be able to View the "Shared" Daasity Transformations used during their Daily Workflows.
A Merchant Repo is created during the Account Provisioning Process. A private repository is created on the Daasity GitHub account and access is granted to Daasity teams for customization.
A Merchant may Login to Daasity and Request Access.
The Repository Name is generated from the system during provisioning. For Example: A New Account named: Daasity Test, would have the GitHub repo: https://github.com/Daasity/merchant-daasity-test.
The daily_incremental.yml file describes a Sequence of SQL scripts to execute during the Standard Daily Workflow.
Once Extraction and Data Loading are completed, transformations start. The daily_incremental.yml is loaded, dependencies are resolved, and then a SQL Manifest is generated in top-down order. The SQL Manifest will then execute against the warehouse transforming the data.
It may be necessary to "bootstrap" the Warehouse Tables from scratch. The reset.yml file is provided to customize this process should additional steps be necessary due to customization.
Team Management
The Daasity Support Team will enable this feature for Merchants within the Daasity App. Once Merchants have access, you can follow the steps below.
PLEASE NOTE: A User Must Have an Active GitHub Username and Agree to License and Terms of Service to Gain Access.
To request access:
- Login into your Daasity app
- Navigate to the Data > Workflows section of the Left-Nav.
- When presented with the GitHub connection information Click the Button to Request Access.
After requesting access, a Daasity support team member will be notified of the request and processed in the order received.
A User will have Access to both the "shared" Daasity scripts and their Private Merchant Repo at GitHub once Access is granted.
Example of the Github Connection section with the "Request Access" button.
Example of the "Accept License and Terms" popup when requesting access.
Example after requesting access, will remain "Pending" until a Daasity agent processes the request.
Once the user has been granted access, they may view the Daasity shared code repository. It is located in GitHub at: https://github.com/Daasity/platform-sql-shared.
Customizing the YAMLs
The new Workflows stored individually in each Merchant Repo may reference a number of scripts from inside their own repos as they already do on the platform-sql-v2 repository.
They may also reference script files present in any other Daasity repository.
The format used to reference internal files stays the same:
- scripts/some_folder/some_script.sql
The new format allows users to reference files that are NOT present in their own repo (and also select files present even in different branches).
PLEASE NOTE: When branch is NOT passed, the default branch master is assumed:
- github://platform-sql-internal/path/to/super_special_script.sql?branch=master
The platform-sql-internal repository (located at https://github.com/Daasity/platform-sql-internal) is a repository that holds internal sql scripts that touch on specific client data.
This repo should NOT be accessible, except for internal Daasity DevOps teammates.
The platform-sql-internal repo deals with Internal scripts from each merchant that may contain sensitive information. The repository content may ONLY be accessible by internal DevOps Daasity teammates to protect our Merchants Privacy.
The platform-sql-shared repository (located at https://github.com/Daasity/platform-sql-shared) is a repository that holds common sql scripts that are commonly used by multiple merchants. This repo is accessible to every merchant.
To Add Custom scripts to your repo:
Clone the repository locally in your machine and commit/push the new scripts to the repository.
All teammates approved on the Internal Merchant repo have permission to push files into it.To Test Custom Scripts on your Custom Repository:
Create new SQL manifest that point to this new repo (the new repo is available for use on SQL manifests as soon as it is created).
For example: You can validate your Daily Workflows files are still functioning after converting them to the new syntax protocol (v2).
You are ready to enable the Github merchant feature via the UI once your SQL scripts and yaml files are up to date.
To enable the Github Merchant Feature:
Click on the Enable new Repository Access
Now, The Daily Workflows files will be searched for in your own Custom Repo.
Pro Workflows: Workflow Manifest File
This section describes in more detail how a script manifest file (aka, YML file, YAML file, Workflow manifest file, Workflow Script file, ...) is processed and the format of the files.
Almost all Pro accounts have at least one SQL manifest file. That file is the daily_incremental.yml It is the file which is used when running the daily workflow for Pro merchants.
The script manifest file syntax is a standard YAML file (YAML definition). It consists of a version and sections.
- YAML syntax is positional sensitive
- the indentation must be spaces (not tabs)
- the indentation must be two (2) spaces
- comments may be specified using a hash symbol #
Version
The current version number (as illustrated above) is 2.0.0 and should not be changed.
Sections are simply a way to group a set of scripts together. They contain the following elements:
- <section_name>
- must be one word, if not one word, it should be snake case.
- scripts:
- is a list (an array) of "file path names" that will be executed in sequential fashion, top to bottom
- the "file path names" typically refer to a file in one of three repositories:
- platform-sql-shared: owned by Daasity - viewable by merchant users
- platform-sql-internal: internal to Daasity - not merchant viewable
- merchant-repo-name: merchant specific code - Daasity team and merchant may update, Daasity team may work with merchant on setting up Github workflows - branch management etc.
- integrations:
- is an array of "friendly names" of integrations that must be "runnable" for that section of scripts to run
- the logic is simple, if any of the specified integrations are "runnable" then the scripts will be run; think of it like this if IntegrationA OR IntegrationB OR IntegrationC are runnable the scrips will be run.
A "runnable" integration means that the account has an integration configured and the integration is active. For example, we have an account that contains two (2) integrations:
- "Shopify - US" which is activated;
- "TikTok - US" which is deactivated
In this example,
- when the script processor encounters a section that requires a shopify integration, it will execute the scripts because we have an active "runnable" Shopify Integration.
- when the script processor encounters a section that only requires a tiktok integration, it will not execute the scripts because we have an inactive "non runnable" TikTok integration.
Friendly names are just that, a more human readable name for a specific integration type. Instead of having a user know all the class names, we can create one friendly name that hides the deals of the integration, but also allows us to group them together. For example: the shopify includes both types of Shopify integrations, Shopify and Shopify Plus. By doing this the user may just specify one name, that will run if either type of Shopify integration is runnable.
In true YAML fashion, the friendly names are all snake case format.
The list of friendly names master list is located in the platform-sql-shared repository. It is a YAML file that contains the mapping of the friendly names to the integration classes. The file name is: integrations_map.yml
What is the" file path name" syntax?
The format used to reference "local files" - that are specific to a specific merchant is simply the full path to the file. The files may be organized in any way, but the name must include the full path along with the file name. The syntax looks like the following.
- some_folder/some_script.sql
References to script files not in a merchant repository must be specified in a specific syntax. The syntax allows the following options:
- GIT repository provider: currently only GitHub is supported
- Git repository name
- file folder path
- file name with the explicit file extension .SQL
- optionally a branch name: this option should be rarely used and should never be used in a production deployment. If a branch is not specified it assumes master
- github://platform-sql-shared/folder_name/sub_folder/some_daasity_script.sql?branch=master
- Once extraction and data loading are complete, transformations are started.
- The manifest file, which describes a sequence of SQL scripts to execute during a workflow, is loaded.
- While trying to load, if the file is "bad", typically invalid syntax, the user will be notified of the error.
- Once successfully loaded the processor runs a validation phase, which ensures that all the specified files exist.
- Assuming all files exist the processing runs in a sequential, top down manner, running section by section.
- When processing each section, the script processor looks for any runnable integrations that may be specified in the integrations section.
- If no integrations are specified, the processor will will run all the scripts of that section
- if no runnable integrations are specified, the processor will not run any of the script of that section if any (at least one) runnable integration is present, the process will run all the scripts of that section.
Final Notes
When modifying a script file or adding/editing/deleting any script files, it is best practice to TEST your changes on a branch before committing the changes to the production master branch.