DAAS-9403 AzureSQL Profiling Pipeline

[ankurs-delphix]

Problem

Publish the AzureSQL to AzureSQL profiling pipeline.

Note

This PR only adds support of profiling AzureSQL source. Support for masking pipeline and related resources is tracked by DAAS-9358 and will followup soon.

Solution

Create the deployment template and the manifest file along with its readme.

Here is the workflow sequence diagram (Also added in the README)

Testing Done

Verified template Zip was created

➜  dcs-for-azure-templates git:(azuresql-profiling) ✗ docker compose up
[+] Running 0/1
 ⠼ Container dcs-for-azure-templates-build-template-artifacts-1  Recreated                                                 0.4s
Attaching to build-template-artifacts-1
build-template-artifacts-1  | Reading package lists...
build-template-artifacts-1  | Reading package lists...
build-template-artifacts-1  | Building dependency tree...
build-template-artifacts-1  | Reading state information...
build-template-artifacts-1  | The following additional packages will be installed:
build-template-artifacts-1  |   unzip
build-template-artifacts-1  | The following NEW packages will be installed:
build-template-artifacts-1  |   unzip zip
build-template-artifacts-1  | 0 upgraded, 2 newly installed, 0 to remove and 13 not upgraded.
build-template-artifacts-1  | Need to get 350 kB of archives.
build-template-artifacts-1  | After this operation, 930 kB of additional disk space will be used.
build-template-artifacts-1  | Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB]
build-template-artifacts-1  | Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB]
build-template-artifacts-1  | debconf: delaying package configuration, since apt-utils is not installed
build-template-artifacts-1  | Fetched 350 kB in 3s (135 kB/s)
build-template-artifacts-1  | Selecting previously unselected package unzip.
(Reading database ... 4393 files and directories currently installed.)
build-template-artifacts-1  | Preparing to unpack .../unzip_6.0-26ubuntu3.2_amd64.deb ...
build-template-artifacts-1  | Unpacking unzip (6.0-26ubuntu3.2) ...
build-template-artifacts-1  | Selecting previously unselected package zip.
build-template-artifacts-1  | Preparing to unpack .../zip_3.0-12build2_amd64.deb ...
build-template-artifacts-1  | Unpacking zip (3.0-12build2) ...
build-template-artifacts-1  | Setting up unzip (6.0-26ubuntu3.2) ...
build-template-artifacts-1  | Setting up zip (3.0-12build2) ...
build-template-artifacts-1  |   adding: dcsazure_adls_to_adls_prof_pl/README.md (deflated 67%)
build-template-artifacts-1  |   adding: dcsazure_adls_to_adls_prof_pl/dcsazure_adls_to_adls_prof_pl.json (deflated 89%)
build-template-artifacts-1  |   adding: dcsazure_adls_to_adls_prof_pl/manifest.json (deflated 78%)
build-template-artifacts-1  |   adding: dcsazure_adls_to_adls_mask_pl/README.md (deflated 67%)
build-template-artifacts-1  |   adding: dcsazure_adls_to_adls_mask_pl/dcsazure_adls_to_adls_mask_pl.json (deflated 88%)
build-template-artifacts-1  |   adding: dcsazure_adls_to_adls_mask_pl/manifest.json (deflated 78%)
build-template-artifacts-1  |   adding: dcsazure_Snowflake_Legacy_to_Snowflake_Legacy_prof_pl/README.md (deflated 65%)
build-template-artifacts-1  |   adding: dcsazure_Snowflake_Legacy_to_Snowflake_Legacy_prof_pl/dcsazure_Snowflake_Legacy_to_Snowflake_Legacy_prof_pl.json (deflated 83%)
build-template-artifacts-1  |   adding: dcsazure_Snowflake_Legacy_to_Snowflake_Legacy_prof_pl/manifest.json (deflated 80%)
build-template-artifacts-1  |   adding: dcsazure_Snowflake_Legacy_to_Snowflake_Legacy_mask_pl/README.md (deflated 68%)
build-template-artifacts-1  |   adding: dcsazure_Snowflake_Legacy_to_Snowflake_Legacy_mask_pl/dcsazure_Snowflake_Legacy_to_Snowflake_Legacy_mask_pl.json (deflated 89%)
build-template-artifacts-1  |   adding: dcsazure_Snowflake_Legacy_to_Snowflake_Legacy_mask_pl/manifest.json (deflated 79%)
build-template-artifacts-1  |   adding: dcsazure_Snowflake_to_Snowflake_prof_pl/README.md (deflated 63%)
build-template-artifacts-1  |   adding: dcsazure_Snowflake_to_Snowflake_prof_pl/dcsazure_Snowflake_to_Snowflake_prof_pl.json (deflated 82%)
build-template-artifacts-1  |   adding: dcsazure_Snowflake_to_Snowflake_prof_pl/manifest.json (deflated 80%)
build-template-artifacts-1  |   adding: dcsazure_Snowflake_to_Snowflake_mask_pl/README.md (deflated 68%)
build-template-artifacts-1  |   adding: dcsazure_Snowflake_to_Snowflake_mask_pl/dcsazure_Snowflake_to_Snowflake_mask_pl.json (deflated 90%)
build-template-artifacts-1  |   adding: dcsazure_Snowflake_to_Snowflake_mask_pl/manifest.json (deflated 79%)
build-template-artifacts-1  |   adding: dcsazure_Databricks_to_Databricks_prof_pl/README.md (deflated 65%)
build-template-artifacts-1  |   adding: dcsazure_Databricks_to_Databricks_prof_pl/dcsazure_Databricks_to_Databricks_prof_pl.json (deflated 87%)
build-template-artifacts-1  |   adding: dcsazure_Databricks_to_Databricks_prof_pl/manifest.json (deflated 78%)
build-template-artifacts-1  |   adding: dcsazure_Databricks_to_Databricks_mask_pl/README.md (deflated 69%)
build-template-artifacts-1  |   adding: dcsazure_Databricks_to_Databricks_mask_pl/dcsazure_Databricks_to_Databricks_mask_pl.json (deflated 88%)
build-template-artifacts-1  |   adding: dcsazure_Databricks_to_Databricks_mask_pl/manifest.json (deflated 81%)
build-template-artifacts-1  |   adding: dcsazure_AzureSQL_to_AzureSQL_prof_pl/README.md (deflated 66%)
build-template-artifacts-1  |   adding: dcsazure_AzureSQL_to_AzureSQL_prof_pl/dcsazure_AzureSQL_to_AzureSQL_prof_pl.json (deflated 81%)
build-template-artifacts-1  |   adding: dcsazure_AzureSQL_to_AzureSQL_prof_pl/manifest.json (deflated 80%)
build-template-artifacts-1 exited with code 0

➜  dcs-for-azure-templates git:(azuresql-profiling) ✗ ls -ltr releases/*Azure*
-rw-r--r--@ 1 ankur.srivastava  staff  16207 Oct  7 17:01 releases/dcsazure_AzureSQL_to_AzureSQL_prof_pl.zip

Imported the pipeline into the data-factory and run debug tests.

Discovered Tables

{
	"count": 12,
	"value": [
		{
			"specified_database": "DCSforAzureTraining",
			"specified_schema": "SalesLT",
			"identified_table": "Address"
		},
		{
			"specified_database": "DCSforAzureTraining",
			"specified_schema": "SalesLT",
			"identified_table": "Customer"
		},
		{
			"specified_database": "DCSforAzureTraining",
			"specified_schema": "SalesLT",
			"identified_table": "CustomerAddress"
		},
		{
			"specified_database": "DCSforAzureTraining",
			"specified_schema": "SalesLT",
			"identified_table": "CustomerMasked"
		},
		{
			"specified_database": "DCSforAzureTraining",
			"specified_schema": "SalesLT",
			"identified_table": "CustomerMaskedSrusia"
		},
		{
			"specified_database": "DCSforAzureTraining",
			"specified_schema": "SalesLT",
			"identified_table": "Product"
		},
		{
			"specified_database": "DCSforAzureTraining",
			"specified_schema": "SalesLT",
			"identified_table": "ProductCategory"
		},
		{
			"specified_database": "DCSforAzureTraining",
			"specified_schema": "SalesLT",
			"identified_table": "ProductDescription"
		},
		{
			"specified_database": "DCSforAzureTraining",
			"specified_schema": "SalesLT",
			"identified_table": "ProductModel"
		},
		{
			"specified_database": "DCSforAzureTraining",
			"specified_schema": "SalesLT",
			"identified_table": "ProductModelProductDescription"
		},
		{
			"specified_database": "DCSforAzureTraining",
			"specified_schema": "SalesLT",
			"identified_table": "SalesOrderDetail"
		},
		{
			"specified_database": "DCSforAzureTraining",
			"specified_schema": "SalesLT",
			"identified_table": "SalesOrderHeader"
		}
	],
}

Profiling Result (Stored in metadata)
dataflow-debug-07092024.csv

[sumeetdas-dlpx]

I think the PNG image you shared in the description has transparent background, because of which I'm unable to read text. This is how I'm seeing it: