Skip to content

Commit

Permalink
docs(metadata-models-custom): add example script to show producing cu… (
Browse files Browse the repository at this point in the history
  • Loading branch information
shirshanka authored Apr 19, 2022
1 parent 9ecb785 commit c5b8f57
Show file tree
Hide file tree
Showing 2 changed files with 57 additions and 0 deletions.
9 changes: 9 additions & 0 deletions metadata-models-custom/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,15 @@ results in
Update succeeded with status 200
```

The `scripts/insert_custom_aspect.py` script shows you how to accomplish the same using the Python SDK. Note that we are just using a raw dictionary here to represent the `dq_rule` aspect and not a strongly-typed class.
```console
cd scripts
python3 insert_custom_aspect.py
```
results in
```console
Successfully wrote to DataHub
```

## Advanced Guide

Expand Down
48 changes: 48 additions & 0 deletions metadata-models-custom/scripts/insert_custom_aspect.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
import json

from datahub.emitter.rest_emitter import DatahubRestEmitter
from datahub.metadata.schema_classes import (
ChangeTypeClass,
GenericAspectClass,
MetadataChangeProposalClass,
)

dq_aspect = {
"rules": [
{
"field": "my_event_data",
"isFieldLevel": False,
"type": "isNull",
"checkDefinition": "n/a",
"url": "https://github.com/datahub-project/datahub/blob/master/checks/nonNull.sql",
},
{
"field": "timestamp",
"isFieldLevel": True,
"type": "increasing",
"checkDefinition": "n/a",
"url": "https://github.com/datahub-project/datahub/blob/master/checks/increasing.sql",
},
]
}

emitter: DatahubRestEmitter = DatahubRestEmitter(gms_server="http://localhost:8080")

dataset_urn = "urn:li:dataset:(urn:li:dataPlatform:hive,logging_events,PROD)"
mcp_raw: MetadataChangeProposalClass = MetadataChangeProposalClass(
entityType="dataset",
entityUrn=dataset_urn,
changeType=ChangeTypeClass.UPSERT,
aspectName="customDataQualityRules",
aspect=GenericAspectClass(
contentType="application/json",
value=json.dumps(dq_aspect).encode("utf-8"),
),
)

try:
emitter.emit(mcp_raw)
print("Successfully wrote to DataHub")
except Exception as e:
print("Failed to write to DataHub")
raise e

0 comments on commit c5b8f57

Please sign in to comment.