Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(ingest/dbt): add support for urns in add_owner directive #11221

Merged
merged 1 commit into from
Aug 23, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion metadata-ingestion/docs/sources/dbt/dbt.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,11 @@ We support the following operations:
1. add_tag - Requires `tag` property in config.
2. add_term - Requires `term` property in config.
3. add_terms - Accepts an optional `separator` property in config.
4. add_owner - Requires `owner_type` property in config which can be either user or group. Optionally accepts the `owner_category` config property which can be set to either a [custom ownership type](../../../../docs/ownership/ownership-types.md) urn like `urn:li:ownershipType:architect` or one of `['TECHNICAL_OWNER', 'BUSINESS_OWNER', 'DATA_STEWARD', 'DATAOWNER'` (defaults to `DATAOWNER`).
4. add_owner - Requires `owner_type` property in config which can be either `user` or `group`. Optionally accepts the `owner_category` config property which can be set to either a [custom ownership type](../../../../docs/ownership/ownership-types.md) urn like `urn:li:ownershipType:architect` or one of `['TECHNICAL_OWNER', 'BUSINESS_OWNER', 'DATA_STEWARD', 'DATAOWNER'` (defaults to `DATAOWNER`).

- The `owner_type` property will be ignored if the owner is a fully qualified urn.
- You can use commas to specify multiple owners - e.g. `business_owner: "jane,john,urn:li:corpGroup:data-team"`.

5. add_doc_link - Requires `link` and `description` properties in config. Upon ingestion run, this will overwrite current links in the institutional knowledge section with this new link. The anchor text is defined here in the meta_mappings as `description`.

Note:
Expand Down
6 changes: 6 additions & 0 deletions metadata-ingestion/src/datahub/emitter/mce_builder.py
Original file line number Diff line number Diff line change
Expand Up @@ -244,6 +244,12 @@ def make_tag_urn(tag: str) -> str:


def make_owner_urn(owner: str, owner_type: OwnerType) -> str:
if owner_type == OwnerType.USER:
return make_user_urn(owner)
elif owner_type == OwnerType.GROUP:
return make_group_urn(owner)
# This should pretty much never happen.
# TODO: With Python 3.11, we can use typing.assert_never() here.
return f"urn:li:{owner_type.value}:{owner}"


Expand Down
14 changes: 10 additions & 4 deletions metadata-ingestion/tests/unit/test_mapping.py
Original file line number Diff line number Diff line change
Expand Up @@ -186,7 +186,7 @@ def test_operation_processor_advanced_matching_owners():
def test_operation_processor_ownership_category():
raw_props = {
"user_owner": "@test_user",
"business_owner": "alice",
"business_owner": "alice,urn:li:corpGroup:biz-data-team",
"architect": "bob",
}
processor = OperationProcessor(
Expand Down Expand Up @@ -222,18 +222,24 @@ def test_operation_processor_ownership_category():
assert "add_owner" in aspect_map

ownership_aspect: OwnershipClass = aspect_map["add_owner"]
assert len(ownership_aspect.owners) == 3
assert len(ownership_aspect.owners) == 4

new_owner: OwnerClass = ownership_aspect.owners[0]
assert new_owner.owner == "urn:li:corpGroup:biz-data-team"
assert new_owner.source and new_owner.source.type == "SOURCE_CONTROL"
assert new_owner.type and new_owner.type == OwnershipTypeClass.BUSINESS_OWNER

new_owner = ownership_aspect.owners[1]
assert new_owner.owner == "urn:li:corpGroup:test_user"
assert new_owner.source and new_owner.source.type == "SOURCE_CONTROL"
assert new_owner.type and new_owner.type == OwnershipTypeClass.DATA_STEWARD

new_owner = ownership_aspect.owners[1]
new_owner = ownership_aspect.owners[2]
assert new_owner.owner == "urn:li:corpuser:alice"
assert new_owner.source and new_owner.source.type == "SOURCE_CONTROL"
assert new_owner.type and new_owner.type == OwnershipTypeClass.BUSINESS_OWNER

new_owner = ownership_aspect.owners[2]
new_owner = ownership_aspect.owners[3]
assert new_owner.owner == "urn:li:corpuser:bob"
assert new_owner.source and new_owner.source.type == "SOURCE_CONTROL"
assert new_owner.type == OwnershipTypeClass.CUSTOM
Expand Down
Loading