Part 2 : Applying 12 factor app principle to data engineering

 

2nd principle is handling dependencies The Twelve-Factor App (12factor.net)

 

Explicitly declare and isolate all dependencies

 

Reference : https://developer.ibm.com/developer/default/articles/creating-a-12-factor-application-with-open-liberty/images/images02.png

 This principle states that the source code maintained in source control should contain only code that is relevant and unique to the application.

 External dependencies (like Node JS, python libraries, .NET add-ins etc..) should not reside within source  control but those should be identified, isolated and made sure it is available during runtime when the application  / service is trying to do reference.

 Lets take this principle and fit it in our data engineering /  data integration scenario.

 Handling dependencies in Databricks 

 In your python based solution, create a requirements.txt file including all dependency packages along with their versions:

• Numpy==1.24.3
• Pandas==2.0.2
• Py==3.1.0
• Py4 ==0.10.9.7
• Xlrd==1.2.0

to list a few.

These Python libraries are installed in Databricks cluster from DevOps pipeline downloading it from external artefacts repository. Alternatively, we could also have it stored in Azure DevOps -> Artefacts and download from there from pipeline if there is a need to restrict internet access in pipeline or when working with a self-hosted deployment agent.

 Practical Tip:

If we don't specify explicit version of packages, from Azure DevOps pipeline when it downloads from respective store, it gets the latest version. In our project, this lead to unexpected issues with data pipeline. Since these packages are Open source and community driven, it keeps evolving with new versions. High chance a particular feature might not work in a higher version. Hence it is always recommended to specify exact version with which you have done your development and testing. 

Following Microsoft article explains steps involved in creating a DevOps pipeline to package python libraries and to deploy libraries to Databricks cluster with use of Databricks-connect CLI tool to securely connect to cluster from DevOps agent.

https://learn.microsoft.com/en-us/azure/databricks/dev-tools/ci-cd/ci-cd-azure-devops

 In addition to external libraries listed above, our python code needs to be packaged, versioned and deployed to cluster for the data pipeline to work.

Link to blog on writing python script in notebooks vs writing software engineering approach-based Python classes in Databricks https://bharaniblogs.blogspot.com/2023/06/writing-python-script-in-notebooks-vs.html

 Initially, in our project we were hard-coding python code version every time we were doing a build and deploy during development phase. As part of continuous improvement, we introduced dynamic versioning using Versioneer.

 Link to blog on how to create and apply dynamic version to python packages and deploy to Databricks from DevOps

Bharani blogs: Dynamically generate version and apply python package from DevOps 

 

Part 1 : Applying DevOps, GitOps principles for data engineering / analytics use-cases

In this series, we will take 12 factor app principles and apply it to a typical data engineering use-case. 

First principle is Codebase. This principle states that one codebase (branch, typically) is to be maintained which would take various configuration settings, parameters, secrets as reference and deploys to multiple environments as shown in the figure below.

reference : developer.ibm.com

  

Often software engineering & DevOps perspective is overlooked for most of the analytics use-cases. Most Data engineers, Data scientists gives more priority on data and how to create data pipelines that would transport data from one layer to another to carry out activities cleansing, validation, transformation, enrichment but DevOps aspect is seen as if it is only applicable for an application workload.

It could be stemming from the thinking that the pipeline can be replicated with less effort from environment to another in a much less-hassle way compared to deploying a microservice or a web app from one to another environment. Thanks to many of the UI framework based ETL / ELT tools in the market. (Talend, Azure Data Factory etc..)

Not only data engineers, scientists in this case even clients or product owners or any relevant stakeholder of a data use-case does not really recognize the value DevOps, GitOps can bring to achieve quality, consistency, automation of deployment. 

Another factor attributes to this is the less availability of DevOps framework and tools for data analytics workloads compared to the toolset available for web or services based workloads.

Even if a team has taken this DevOps into consideration, in most cases it might not follow GitOps deployment model.

Having multiple codebase (branches) for each environment (dev, test, prod) with hard-coded environment specific parameters makes it difficult to do merge from Dev to Test or from Test to Prod branch. It violates the first principle of 12 factor app.

Trunk-based development model is the go-to option to achieve this principle of having single Codebase. This method defines one of the branch (usually 'Main') as trunk and other branches (feature/<feature name or developer name>) or short-lived branch which are created and destroyed post merge to trunk and feature release.

  

Environment specific branching 

 

Trunk-based branching approach

Trunk-based development model is used in Kubernetes and such cloud native development / deployment model when dealing with microservices or web apps like workloads. In Data related use-cases could also follow similar model to bring in best practices when it comes to continuous integration and deployment.

In Azure DevOps, such a trunk-based branching can be followed and to do the environment specific substitutions one could leverage variable groups (one per environment) of library section which maps to key vaults created per environment in their own resource groups / subscriptions based on how different environments are segregated.

  

Library section in Azure DevOps with environment specific variable groups pointing to key vault of respective environment

From DevOps pipelines, these variable groups can be used to substitute environment specific configurations or secrets from the deployment stages designated per environment.

Database deployment task in DevOps

When it comes to database deployment, one could dacpac tasks that are available in Azure DevOps by default. While deploying changes from dev to test or from test to prod (with or without data, based on need), database username and password, database names could be referred from key vault secrets mapped to variable groups. These variables could then be used in DevOps pipelines.

SQL Database deployment task 

Same goes for other data services like Data factory, Databricks, Analysis services etc. 

Data factory deployment task in DevOps

In case of data factory, it is a bit more complicated as one has to substitute many objects

• global parameters
• Linked connections to services
• Datasets
• Pipelines
• etc..

Adftools is one great DevOps tool to use for Data factory DevOps.

Databricks deployment task in DevOps

 In case of Databricks deployment, packaging of code and deploying it as library in cluster can be done using Azure DevOps.

Databricks DevOps

 In upcoming blogs, we would see each of this deployments in detail. This blog is just to iterate on DevOps & GitOps practices for data engineering use-case.