fix(ingest/looker): support platform instance for dashboards & charts #10771
Conversation
github-actions
bot
added
the
ingestion
| include_looker_element_in_platform_instance: bool = Field( | ||
| True, | ||
| description="When enabled, platform instance will be added in dashboard and chart urn.", | ||
| ) |
There was a problem hiding this comment.
@sid-acryl If this defaults to true then it will change URNs of all existing recipes. Is there a migration that happens automatically in this case?
There was a problem hiding this comment.
Yup it will change urn of all existing recipes. Not migration happens automatically, let me know if I make it default to False
There was a problem hiding this comment.
@anshbansal I set it to default False
…l/datahub-fork into cus2139-pi-in-looker-dash-chart
WalkthroughThe changes enhance Looker's metadata ingestion by integrating platform instance information into URNs. This allows for improved traceability and organization of metadata. A new configuration option was introduced, along with refactored methods, ensuring that charts, dashboards, and datasets accurately represent their respective platform instance details. Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant LookerConfig
participant LookerSource
participant MetadataEvents
User->>LookerConfig: Enable include_platform_instance_in_urns
LookerConfig-->>LookerSource: Pass configuration
LookerSource->>MetadataEvents: Generate URNs with platform instance
MetadataEvents-->>LookerSource: Return URNs
LookerSource-->>User: Metadata with updated URNs
Poem
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (invoked as PR comments)
Additionally, you can add CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
Outside diff range and nitpick comments (3)
metadata-ingestion/tests/integration/looker/test_looker.py (1)
Line range hint
492-492: Fix mutable default argument in function definition.Using mutable data structures like lists or dictionaries as default arguments in Python functions can lead to unexpected behavior because they can be modified between function calls. Replace with
Noneand initialize within the function if necessary.- def setup_mock_explore(mocked_client: Any, additional_lkml_fields: List[LookmlModelExploreField] = [], **additional_explore_fields: Any) -> None: + def setup_mock_explore(mocked_client: Any, additional_lkml_fields: List[LookmlModelExploreField] = None, **additional_explore_fields: Any) -> None: + if additional_lkml_fields is None: + additional_lkml_fields = []metadata-ingestion/src/datahub/ingestion/source/looker/looker_source.py (2)
Line range hint
384-384: Optimization suggestion based on static analysis.
- Simplify the dictionary key check as suggested by the static analysis tool.
- if key in dict.keys(): + if key in dict:
Line range hint
1280-1284: Refactor nestedifstatements for clarity.
- Refactor to combine nested
ifstatements into a single conditional, improving readability.- if condition1: - if condition2: + if condition1 and condition2:
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (4)
- metadata-ingestion/src/datahub/ingestion/source/looker/looker_config.py (1 hunks)
- metadata-ingestion/src/datahub/ingestion/source/looker/looker_source.py (4 hunks)
- metadata-ingestion/tests/integration/looker/golden_test_ingest.json (29 hunks)
- metadata-ingestion/tests/integration/looker/test_looker.py (1 hunks)
Additional context used
Ruff
metadata-ingestion/tests/integration/looker/test_looker.py
492-492: Do not use mutable data structures for argument defaults
Replace with
None; initialize within function(B006)
metadata-ingestion/src/datahub/ingestion/source/looker/looker_source.py
384-384: Use
key in dictinstead ofkey in dict.keys()Remove
.keys()(SIM118)
1280-1284: Use a single
ifstatement instead of nestedifstatements(SIM102)
Additional comments not posted (7)
metadata-ingestion/src/datahub/ingestion/source/looker/looker_config.py (2)
219-222: New configuration field added as expected.The new boolean field
include_looker_element_in_platform_instancehas been correctly added with an appropriate default value ofFalse. This aligns with the PR objectives to allow configurable inclusion of platform instances in URNs.
Line range hint
225-231: Validator forextract_independent_lookscorrectly ensures stateful ingestion.The validator
stateful_ingestion_should_be_enabledproperly checks thatstateful_ingestionis enabled if theextract_independent_looksflag is set to True. This is a crucial check to ensure data consistency and integrity when this feature is used.metadata-ingestion/tests/integration/looker/golden_test_ingest.json (1)
4-11: Consistent addition of platform instance identifiers across test data.All modifications in the JSON file correctly reflect the inclusion of the platform instance identifier (
ap-south-1) in various URN constructs and other relevant fields. This is consistent with the PR objectives and helps in testing the new configuration option effectively.Also applies to: 26-26, 42-48, 59-59, 77-86, 102-129, 152-152, 170-185, 202-210, 249-249, 265-265, 281-296, 309-336, 377-397, 431-438, 453-453, 469-475, 486-486, 504-513, 529-563, 625-625, 643-643, 659-664, 675-690, 761-775
metadata-ingestion/tests/integration/looker/test_looker.py (1)
97-98: Integration test updated to include new configuration options.The test configuration has been correctly updated to include the
platform_instanceandinclude_looker_element_in_platform_instancefields. This ensures that the new functionality is covered by integration tests.metadata-ingestion/src/datahub/ingestion/source/looker/looker_source.py (3)
643-645: Code Quality Improvement in_make_chart_metadata_events.
- The method
_make_chart_metadata_eventseffectively leverages the newly created_make_chart_urnmethod, ensuring consistency in URN generation.- Additionally, ensure that the method handles any potential exceptions or edge cases, especially when dealing with external function calls.
1173-1174: Correctness in_input_fields_from_dashboard_element.
- The method correctly uses the
_make_chart_urnto generate URNs, ensuring that the URN generation logic is centralized and consistent.- This is a good practice as it avoids duplication and potential inconsistencies in URN formats across different parts of the application.
1241-1242: Efficient Code Practice in_make_metrics_dimensions_chart_mcp.
- The method effectively reuses the
_make_chart_urnfunction, demonstrating good reuse of existing functionalities.- This approach helps maintain consistency in URN generation and reduces the risk of bugs related to URN formatting.
| def _make_chart_urn(self, element_id: str) -> str: | ||
| if not self.source_config.include_looker_element_in_platform_instance: | ||
| return builder.make_chart_urn( | ||
| name=element_id, | ||
| platform=self.source_config.platform_name, | ||
| ) | ||
|
|
||
| return builder.make_chart_urn( | ||
| name=element_id, | ||
| platform=self.platform, | ||
| platform_instance=self.source_config.platform_instance, | ||
| ) |
There was a problem hiding this comment.
Refactor suggestion for _make_chart_urn method.
- The method correctly handles the conditional inclusion of
platform_instancebased on the configuration. However, it's recommended to reduce duplication by handling the common parameters separately. - Here's a proposed refactor to simplify the method:
def _make_chart_urn(self, element_id: str) -> str:
urn_params = {
"name": element_id,
"platform": self.source_config.platform_name,
}
if self.source_config.include_looker_element_in_platform_instance:
+ urn_params["platform_instance"] = self.source_config.platform_instance
- return builder.make_chart_urn(
- name=element_id,
- platform=self.platform,
- platform_instance=self.source_config.platform_instance,
- )
- return builder.make_chart_urn(
- name=element_id,
- platform=self.source_config.platform_name,
- )
+ return builder.make_chart_urn(**urn_params)Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| def _make_chart_urn(self, element_id: str) -> str: | |
| if not self.source_config.include_looker_element_in_platform_instance: | |
| return builder.make_chart_urn( | |
| name=element_id, | |
| platform=self.source_config.platform_name, | |
| ) | |
| return builder.make_chart_urn( | |
| name=element_id, | |
| platform=self.platform, | |
| platform_instance=self.source_config.platform_instance, | |
| ) | |
| def _make_chart_urn(self, element_id: str) -> str: | |
| urn_params = { | |
| "name": element_id, | |
| "platform": self.source_config.platform_name, | |
| } | |
| if self.source_config.include_looker_element_in_platform_instance: | |
| urn_params["platform_instance"] = self.source_config.platform_instance | |
| return builder.make_chart_urn(**urn_params) |
| if not self.source_config.include_looker_element_in_platform_instance: | ||
| return builder.make_dashboard_urn( | ||
| name=looker_dashboard.get_urn_dashboard_id(), | ||
| platform=self.source_config.platform_name, | ||
| ) | ||
|
|
||
| return builder.make_dashboard_urn( | ||
| self.source_config.platform_name, looker_dashboard.get_urn_dashboard_id() | ||
| name=looker_dashboard.get_urn_dashboard_id(), | ||
| platform=self.source_config.platform_name, | ||
| platform_instance=self.source_config.platform_instance, |
There was a problem hiding this comment.
Refactor suggestion for make_dashboard_urn method.
- Similar to the
_make_chart_urnmethod, consider refactoring to reduce duplication and improve maintainability.
def make_dashboard_urn(self, looker_dashboard):
urn_params = {
"name": looker_dashboard.get_urn_dashboard_id(),
"platform": self.source_config.platform_name,
}
if self.source_config.include_looker_element_in_platform_instance:
+ urn_params["platform_instance"] = self.source_config.platform_instance
- return builder.make_dashboard_urn(
- name=looker_dashboard.get_urn_dashboard_id(),
- platform=self.source_config.platform_name,
- platform_instance=self.source_config.platform_instance,
- )
- return builder.make_dashboard_urn(
- name=looker_dashboard.get_urn_dashboard_id(),
- platform=self.source_config.platform_name,
- )
+ return builder.make_dashboard_urn(**urn_params)Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| if not self.source_config.include_looker_element_in_platform_instance: | |
| return builder.make_dashboard_urn( | |
| name=looker_dashboard.get_urn_dashboard_id(), | |
| platform=self.source_config.platform_name, | |
| ) | |
| return builder.make_dashboard_urn( | |
| self.source_config.platform_name, looker_dashboard.get_urn_dashboard_id() | |
| name=looker_dashboard.get_urn_dashboard_id(), | |
| platform=self.source_config.platform_name, | |
| platform_instance=self.source_config.platform_instance, | |
| urn_params = { | |
| "name": looker_dashboard.get_urn_dashboard_id(), | |
| "platform": self.source_config.platform_name, | |
| } | |
| if self.source_config.include_looker_element_in_platform_instance: | |
| urn_params["platform_instance"] = self.source_config.platform_instance | |
| return builder.make_dashboard_urn(**urn_params) |
There was a problem hiding this comment.
Actionable comments posted: 2
Outside diff range and nitpick comments (1)
metadata-ingestion/src/datahub/ingestion/source/looker/looker_source.py (1)
Line range hint
1276-1280: Consider simplifying the nested if statements.Use a single
ifstatement instead of nestedifstatements.- if self.source_config.extract_owners: - if self.reporter.resolved_user_ids > 0: - if self.reporter.email_ids_missing == self.reporter.resolved_user_ids: + if ( + self.source_config.extract_owners + and self.reporter.resolved_user_ids > 0 + and self.reporter.email_ids_missing == self.reporter.resolved_user_ids + ):
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (1)
- metadata-ingestion/src/datahub/ingestion/source/looker/looker_source.py (4 hunks)
Additional context used
Ruff
metadata-ingestion/src/datahub/ingestion/source/looker/looker_source.py
384-384: Use
key in dictinstead ofkey in dict.keys()Remove
.keys()(SIM118)
1276-1280: Use a single
ifstatement instead of nestedifstatements(SIM102)
Additional comments not posted (3)
metadata-ingestion/src/datahub/ingestion/source/looker/looker_source.py (3)
623-632: LGTM!The
_make_chart_urnmethod correctly handles the conditional inclusion ofplatform_instancebased on the configuration.
830-838: LGTM!The
make_dashboard_urnmethod correctly handles the conditional inclusion ofplatform_instancebased on the configuration.
1237-1238: LGTM!The
_make_metrics_dimensions_chart_mcpmethod correctly constructs a metadata change proposal for a chart element.
| entity_urn = self._make_chart_urn( | ||
| element_id=dashboard_element.get_urn_element_id() |
There was a problem hiding this comment.
Consider refactoring _input_fields_from_dashboard_element for readability and maintainability.
The function is quite large and could benefit from breaking it down into smaller helper methods.
- def _input_fields_from_dashboard_element(
- self, dashboard_element: LookerDashboardElement
- ) -> List[InputFieldClass]:
- ...
+ def _input_fields_from_dashboard_element(
+ self, dashboard_element: LookerDashboardElement
+ ) -> List[InputFieldClass]:
+ input_fields = (
+ dashboard_element.input_fields
+ if dashboard_element.input_fields is not None
+ else []
+ )
+ fields_for_mcp = []
+ for input_field in input_fields:
+ fields_for_mcp.extend(self._process_input_field(input_field, dashboard_element))
+ return fields_for_mcp
+ def _process_input_field(
+ self, input_field: InputFieldElement, dashboard_element: LookerDashboardElement
+ ) -> List[InputFieldClass]:
+ entity_urn = self._make_chart_urn(
+ element_id=dashboard_element.get_urn_element_id()
+ )
+ view_field_for_reference = input_field.view_field
+ if input_field.view_field is None:
+ explore = self.explore_registry.get_explore(
+ input_field.model, input_field.explore
+ )
+ if explore is not None:
+ self.add_reachable_explore(
+ input_field.model, input_field.explore, entity_urn
+ )
+ entity_urn = explore.get_explore_urn(self.source_config)
+ explore_fields = (
+ explore.fields if explore.fields is not None else []
+ )
+ relevant_field = next(
+ (
+ field
+ for field in explore_fields
+ if field.name == input_field.name
+ ),
+ None,
+ )
+ if relevant_field is not None:
+ view_field_for_reference = relevant_field
+ if view_field_for_reference and view_field_for_reference.name:
+ return [
+ InputFieldClass(
+ schemaFieldUrn=builder.make_schema_field_urn(
+ entity_urn, view_field_for_reference.name
+ ),
+ schemaField=LookerUtil.view_field_to_schema_field(
+ view_field_for_reference,
+ self.reporter,
+ self.source_config.tag_measures_and_dimensions,
+ ),
+ )
+ ]
+ return []Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| entity_urn = self._make_chart_urn( | |
| element_id=dashboard_element.get_urn_element_id() | |
| def _input_fields_from_dashboard_element( | |
| self, dashboard_element: LookerDashboardElement | |
| ) -> List[InputFieldClass]: | |
| input_fields = ( | |
| dashboard_element.input_fields | |
| if dashboard_element.input_fields is not None | |
| else [] | |
| ) | |
| fields_for_mcp = [] | |
| for input_field in input_fields: | |
| fields_for_mcp.extend(self._process_input_field(input_field, dashboard_element)) | |
| return fields_for_mcp | |
| def _process_input_field( | |
| self, input_field: InputFieldElement, dashboard_element: LookerDashboardElement | |
| ) -> List[InputFieldClass]: | |
| entity_urn = self._make_chart_urn( | |
| element_id=dashboard_element.get_urn_element_id() | |
| ) | |
| view_field_for_reference = input_field.view_field | |
| if input_field.view_field is None: | |
| explore = self.explore_registry.get_explore( | |
| input_field.model, input_field.explore | |
| ) | |
| if explore is not None: | |
| self.add_reachable_explore( | |
| input_field.model, input_field.explore, entity_urn | |
| ) | |
| entity_urn = explore.get_explore_urn(self.source_config) | |
| explore_fields = ( | |
| explore.fields if explore.fields is not None else [] | |
| ) | |
| relevant_field = next( | |
| ( | |
| field | |
| for field in explore_fields | |
| if field.name == input_field.name | |
| ), | |
| None, | |
| ) | |
| if relevant_field is not None: | |
| view_field_for_reference = relevant_field | |
| if view_field_for_reference and view_field_for_reference.name: | |
| return [ | |
| InputFieldClass( | |
| schemaFieldUrn=builder.make_schema_field_urn( | |
| entity_urn, view_field_for_reference.name | |
| ), | |
| schemaField=LookerUtil.view_field_to_schema_field( | |
| view_field_for_reference, | |
| self.reporter, | |
| self.source_config.tag_measures_and_dimensions, | |
| ), | |
| ) | |
| ] | |
| return [] |
| def _make_chart_metadata_events( | ||
| self, | ||
| dashboard_element: LookerDashboardElement, | ||
| dashboard: Optional[ | ||
| LookerDashboard | ||
| ], # dashboard will be None if this is a standalone look | ||
| ) -> List[Union[MetadataChangeEvent, MetadataChangeProposalWrapper]]: | ||
| chart_urn = builder.make_chart_urn( | ||
| self.source_config.platform_name, dashboard_element.get_urn_element_id() | ||
| chart_urn = self._make_chart_urn( |
There was a problem hiding this comment.
Consider refactoring _make_chart_metadata_events for readability and maintainability.
The function is quite large and could benefit from breaking it down into smaller helper methods.
- def _make_chart_metadata_events(
- self,
- dashboard_element: LookerDashboardElement,
- dashboard: Optional[LookerDashboard]
- ) -> List[Union[MetadataChangeEvent, MetadataChangeProposalWrapper]]:
- ...
+ def _make_chart_metadata_events(
+ self,
+ dashboard_element: LookerDashboardElement,
+ dashboard: Optional[LookerDashboard]
+ ) -> List[Union[MetadataChangeEvent, MetadataChangeProposalWrapper]]:
+ chart_urn = self._make_chart_urn(
+ element_id=dashboard_element.get_urn_element_id()
+ )
+ chart_snapshot = self._create_chart_snapshot(
+ chart_urn, dashboard_element, dashboard
+ )
+ proposals = self._create_chart_proposals(chart_snapshot, dashboard_element, dashboard)
+ return proposals
+ def _create_chart_snapshot(
+ self,
+ chart_urn: str,
+ dashboard_element: LookerDashboardElement,
+ dashboard: Optional[LookerDashboard]
+ ) -> ChartSnapshot:
+ chart_snapshot = ChartSnapshot(
+ urn=chart_urn,
+ aspects=[Status(removed=False)],
+ )
+ chart_info = ChartInfoClass(
+ type=self._get_chart_type(dashboard_element),
+ description=dashboard_element.description or "",
+ title=dashboard_element.title or "",
+ lastModified=ChangeAuditStamps(),
+ chartUrl=dashboard_element.url(self.source_config.external_base_url or ""),
+ inputs=dashboard_element.get_view_urns(self.source_config),
+ customProperties={
+ "upstream_fields": (
+ ",".join(
+ sorted({field.name for field in dashboard_element.input_fields})
+ )
+ if dashboard_element.input_fields
+ else ""
+ )
+ },
+ )
+ chart_snapshot.aspects.append(chart_info)
+ self._add_browse_paths(chart_snapshot, dashboard, dashboard_element)
+ self._add_ownership(chart_snapshot, dashboard, dashboard_element)
+ return chart_snapshot
+ def _create_chart_proposals(
+ self,
+ chart_snapshot: ChartSnapshot,
+ dashboard_element: LookerDashboardElement,
+ dashboard: Optional[LookerDashboard]
+ ) -> List[Union[MetadataChangeEvent, MetadataChangeProposalWrapper]]:
+ proposals = [
+ MetadataChangeEvent(proposedSnapshot=chart_snapshot),
+ MetadataChangeProposalWrapper(
+ entityUrn=chart_snapshot.urn,
+ aspect=SubTypesClass(typeNames=[BIAssetSubTypes.LOOKER_LOOK]),
+ ),
+ ]
+ if self.source_config.extract_embed_urls and self.source_config.external_base_url:
+ maybe_embed_url = dashboard_element.embed_url(self.source_config.external_base_url)
+ if maybe_embed_url:
+ proposals.append(
+ create_embed_mcp(chart_snapshot.urn, maybe_embed_url)
+ )
+ if dashboard is None and dashboard_element.folder:
+ container = ContainerClass(
+ container=self._gen_folder_key(dashboard_element.folder.id).as_urn(),
+ )
+ proposals.append(
+ MetadataChangeProposalWrapper(entityUrn=chart_snapshot.urn, aspect=container)
+ )
+ if browse_path_v2:
+ proposals.append(
+ MetadataChangeProposalWrapper(
+ entityUrn=chart_snapshot.urn, aspect=browse_path_v2
+ )
+ )
+ return proposalsCommittable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| chart_urn = self._make_chart_urn( | |
| def _make_chart_metadata_events( | |
| self, | |
| dashboard_element: LookerDashboardElement, | |
| dashboard: Optional[LookerDashboard] | |
| ) -> List[Union[MetadataChangeEvent, MetadataChangeProposalWrapper]]: | |
| chart_urn = self._make_chart_urn( | |
| element_id=dashboard_element.get_urn_element_id() | |
| ) | |
| chart_snapshot = self._create_chart_snapshot( | |
| chart_urn, dashboard_element, dashboard | |
| ) | |
| proposals = self._create_chart_proposals(chart_snapshot, dashboard_element, dashboard) | |
| return proposals | |
| def _create_chart_snapshot( | |
| self, | |
| chart_urn: str, | |
| dashboard_element: LookerDashboardElement, | |
| dashboard: Optional[LookerDashboard] | |
| ) -> ChartSnapshot: | |
| chart_snapshot = ChartSnapshot( | |
| urn=chart_urn, | |
| aspects=[Status(removed=False)], | |
| ) | |
| chart_info = ChartInfoClass( | |
| type=self._get_chart_type(dashboard_element), | |
| description=dashboard_element.description or "", | |
| title=dashboard_element.title or "", | |
| lastModified=ChangeAuditStamps(), | |
| chartUrl=dashboard_element.url(self.source_config.external_base_url or ""), | |
| inputs=dashboard_element.get_view_urns(self.source_config), | |
| customProperties={ | |
| "upstream_fields": ( | |
| ",".join( | |
| sorted({field.name for field in dashboard_element.input_fields}) | |
| ) | |
| if dashboard_element.input_fields | |
| else "" | |
| ) | |
| }, | |
| ) | |
| chart_snapshot.aspects.append(chart_info) | |
| self._add_browse_paths(chart_snapshot, dashboard, dashboard_element) | |
| self._add_ownership(chart_snapshot, dashboard, dashboard_element) | |
| return chart_snapshot | |
| def _create_chart_proposals( | |
| self, | |
| chart_snapshot: ChartSnapshot, | |
| dashboard_element: LookerDashboardElement, | |
| dashboard: Optional[LookerDashboard] | |
| ) -> List[Union[MetadataChangeEvent, MetadataChangeProposalWrapper]]: | |
| proposals = [ | |
| MetadataChangeEvent(proposedSnapshot=chart_snapshot), | |
| MetadataChangeProposalWrapper( | |
| entityUrn=chart_snapshot.urn, | |
| aspect=SubTypesClass(typeNames=[BIAssetSubTypes.LOOKER_LOOK]), | |
| ), | |
| ] | |
| if self.source_config.extract_embed_urls and self.source_config.external_base_url: | |
| maybe_embed_url = dashboard_element.embed_url(self.source_config.external_base_url) | |
| if maybe_embed_url: | |
| proposals.append( | |
| create_embed_mcp(chart_snapshot.urn, maybe_embed_url) | |
| ) | |
| if dashboard is None and dashboard_element.folder: | |
| container = ContainerClass( | |
| container=self._gen_folder_key(dashboard_element.folder.id).as_urn(), | |
| ) | |
| proposals.append( | |
| MetadataChangeProposalWrapper(entityUrn=chart_snapshot.urn, aspect=container) | |
| ) | |
| if browse_path_v2: | |
| proposals.append( | |
| MetadataChangeProposalWrapper( | |
| entityUrn=chart_snapshot.urn, aspect=browse_path_v2 | |
| ) | |
| ) | |
| return proposals |
metadata-ingestion/src/datahub/ingestion/source/looker/looker_source.py
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Actionable comments posted: 0
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (1)
- metadata-ingestion/tests/integration/looker/golden_test_ingest.json (29 hunks)
Additional comments not posted (15)
metadata-ingestion/tests/integration/looker/golden_test_ingest.json (15)
11-11: Approved: Addition of the "instance" field.The addition of the "instance" field to the customProperties of the container entity is consistent with the PR objective.
47-48: Approved: Update to the dataPlatformInstance aspect.The update to include platform and instance information in the dataPlatformInstance aspect is consistent with the PR objective.
83-86: Approved: Update to the browsePathsV2 aspect.The update to include instance information in the browsePathsV2 aspect is consistent with the PR objective.
102-102: Approved: Update to the chart URN.The update to include instance information in the chart URN is consistent with the PR objective.
129-129: Approved: Update to the inputs field in ChartInfo aspect.The update to include instance information in the dataset URN within the inputs field is consistent with the PR objective.
176-179: Approved: Update to the browsePathsV2 aspect of a chart entity.The update to include instance information in the browsePathsV2 aspect is consistent with the PR objective.
202-202: Approved: Update to the dashboard URN.The update to include instance information in the dashboard URN is consistent with the PR objective.
210-210: Approved: Update to the charts field in DashboardInfo aspect.The update to include instance information in the chart URN within the charts field is consistent with the PR objective.
254-254: Approved: Update to the container aspect of a dashboard entity.The update to include instance information in the container aspect is consistent with the PR objective.
287-290: Approved: Update to the browsePathsV2 aspect of a dashboard entity.The update to include instance information in the browsePathsV2 aspect is consistent with the PR objective.
316-316: Approved: Update to the schemaFieldUrn field in inputFields aspect.The update to include instance information in the chart URN within the schemaFieldUrn field is consistent with the PR objective.
333-333: Approved: Update to the schemaFieldUrn field in inputFields aspect.The update to include instance information in the dataset URN within the schemaFieldUrn field is consistent with the PR objective.
432-432: Approved: Addition of the "instance" field.The addition of the "instance" field to the customProperties of the container entity is consistent with the PR objective.
468-469: Approved: Update to the dataPlatformInstance aspect.The update to include platform and instance information in the dataPlatformInstance aspect is consistent with the PR objective.
504-507: Approved: Update to the browsePathsV2 aspect.The update to include instance information in the browsePathsV2 aspect is consistent with the PR objective.
…l/datahub-fork into cus2139-pi-in-looker-dash-chart
Emitted the platform instance aspect |
There was a problem hiding this comment.
Actionable comments posted: 0
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (2)
- metadata-ingestion/src/datahub/ingestion/source/looker/looker_source.py (6 hunks)
- metadata-ingestion/tests/integration/looker/golden_test_ingest.json (29 hunks)
Files skipped from review as they are similar to previous changes (2)
- metadata-ingestion/src/datahub/ingestion/source/looker/looker_source.py
- metadata-ingestion/tests/integration/looker/golden_test_ingest.json
There was a problem hiding this comment.
Actionable comments posted: 1
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (3)
- metadata-ingestion/src/datahub/ingestion/source/looker/looker_config.py (1 hunks)
- metadata-ingestion/src/datahub/ingestion/source/looker/looker_source.py (6 hunks)
- metadata-ingestion/tests/integration/looker/test_looker.py (1 hunks)
Files skipped from review as they are similar to previous changes (1)
- metadata-ingestion/src/datahub/ingestion/source/looker/looker_config.py
Additional comments not posted (8)
metadata-ingestion/tests/integration/looker/test_looker.py (1)
97-98: LGTM! Ensure the new configuration parameters are used correctly.The added configuration parameters are consistent with the PR summary and comments.
However, verify that these parameters are used correctly in the Looker source code.
metadata-ingestion/src/datahub/ingestion/source/looker/looker_source.py (7)
643-654: Avoid using**urn_paramsfor better type-checking.Using
**urn_paramswon't be type-checked properly. Instead, explicitly pass the parameters.return builder.make_chart_urn( name=element_id, platform=self.source_config.platform_name, platform_instance=self.source_config.platform_instance if self.source_config.include_platform_instance_in_urns else None, )
Line range hint
663-751: LGTM!The changes correctly handle the inclusion of the platform instance aspect.
857-863: LGTM!The changes correctly handle the inclusion of the platform instance aspect.
868-876: Avoid using**urn_paramsfor better type-checking.Using
**urn_paramswon't be type-checked properly. Instead, explicitly pass the parameters.return builder.make_dashboard_urn( name=looker_dashboard.get_urn_dashboard_id(), platform=self.source_config.platform_name, platform_instance=self.source_config.platform_instance if self.source_config.include_platform_instance_in_urns else None, )
1207-1208: Consider refactoring_input_fields_from_dashboard_elementfor readability and maintainability.The function is quite large and could benefit from breaking it down into smaller helper methods.
- def _input_fields_from_dashboard_element( - self, dashboard_element: LookerDashboardElement - ) -> List[InputFieldClass]: - ... + def _input_fields_from_dashboard_element( + self, dashboard_element: LookerDashboardElement + ) -> List[InputFieldClass]: + input_fields = ( + dashboard_element.input_fields + if dashboard_element.input_fields is not None + else [] + ) + fields_for_mcp = [] + for input_field in input_fields: + fields_for_mcp.extend(self._process_input_field(input_field, dashboard_element)) + return fields_for_mcp + def _process_input_field( + self, input_field: InputFieldElement, dashboard_element: LookerDashboardElement + ) -> List[InputFieldClass]: + entity_urn = self._make_chart_urn( + element_id=dashboard_element.get_urn_element_id() + ) + view_field_for_reference = input_field.view_field + if input_field.view_field is None: + explore = self.explore_registry.get_explore( + input_field.model, input_field.explore + ) + if explore is not None: + self.add_reachable_explore( + input_field.model, input_field.explore, entity_urn + ) + entity_urn = explore.get_explore_urn(self.source_config) + explore_fields = ( + explore.fields if explore.fields is not None else [] + ) + relevant_field = next( + ( + field + for field in explore_fields + if field.name == input_field.name + ), + None, + ) + if relevant_field is not None: + view_field_for_reference = relevant_field + if view_field_for_reference and view_field_for_reference.name: + return [ + InputFieldClass( + schemaFieldUrn=builder.make_schema_field_urn( + entity_urn, view_field_for_reference.name + ), + schemaField=LookerUtil.view_field_to_schema_field( + view_field_for_reference, + self.reporter, + self.source_config.tag_measures_and_dimensions, + ), + ) + ] + return []
1275-1276: LGTM!The changes correctly handle the generation of the chart URN.
1275-1276: LGTM!The changes correctly handle the emission of metadata for each chart inside the dashboard.
| def _create_platform_instance_aspect( | ||
| self, | ||
| ) -> DataPlatformInstance: | ||
|
|
||
| assert self.source_config.platform_name | ||
| assert self.source_config.platform_instance | ||
|
|
||
| return DataPlatformInstance( | ||
| platform=builder.make_data_platform_urn(self.source_config.platform_name), | ||
| instance=builder.make_dataplatform_instance_urn( | ||
| platform=self.source_config.platform_name, | ||
| instance=self.source_config.platform_instance, | ||
| ), | ||
| ) |
There was a problem hiding this comment.
Add error messages to assertions.
Adding error messages to assertions can help with debugging if they fail.
- assert self.source_config.platform_name
- assert self.source_config.platform_instance
+ assert self.source_config.platform_name, "Platform name is not set in the configuration."
+ assert self.source_config.platform_instance, "Platform instance is not set in the configuration."Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| def _create_platform_instance_aspect( | |
| self, | |
| ) -> DataPlatformInstance: | |
| assert self.source_config.platform_name | |
| assert self.source_config.platform_instance | |
| return DataPlatformInstance( | |
| platform=builder.make_data_platform_urn(self.source_config.platform_name), | |
| instance=builder.make_dataplatform_instance_urn( | |
| platform=self.source_config.platform_name, | |
| instance=self.source_config.platform_instance, | |
| ), | |
| ) | |
| def _create_platform_instance_aspect( | |
| self, | |
| ) -> DataPlatformInstance: | |
| assert self.source_config.platform_name, "Platform name is not set in the configuration." | |
| assert self.source_config.platform_instance, "Platform instance is not set in the configuration." | |
| return DataPlatformInstance( | |
| platform=builder.make_data_platform_urn(self.source_config.platform_name), | |
| instance=builder.make_dataplatform_instance_urn( | |
| platform=self.source_config.platform_name, | |
| instance=self.source_config.platform_instance, | |
| ), | |
| ) |
metadata-ingestion/src/datahub/ingestion/source/looker/looker_source.py
Outdated
Show resolved
Hide resolved
I had verified this on on my local machine with Quickstart |
There was a problem hiding this comment.
Actionable comments posted: 0
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (1)
- metadata-ingestion/src/datahub/ingestion/source/looker/looker_source.py (6 hunks)
Files skipped from review as they are similar to previous changes (1)
- metadata-ingestion/src/datahub/ingestion/source/looker/looker_source.py
There was a problem hiding this comment.
Actionable comments posted: 0
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (2)
- metadata-ingestion/src/datahub/ingestion/source/looker/looker_source.py (8 hunks)
- metadata-ingestion/tests/integration/looker/golden_test_ingest.json (29 hunks)
Files skipped from review as they are similar to previous changes (1)
- metadata-ingestion/src/datahub/ingestion/source/looker/looker_source.py
Additional comments not posted (24)
metadata-ingestion/tests/integration/looker/golden_test_ingest.json (24)
2-18: Approved: Addition ofdataPlatformInstanceentity.The addition of the
dataPlatformInstanceentity aligns with the PR objectives and ensures that the platform instance information is included in the metadata.
Line range hint
19-42:
Approved: Updates tocontainerentity.The updates to the
containerentity, including the addition of platform instance information and other properties, align with the PR objectives and ensure that the metadata is accurately represented.
Line range hint
44-58:
Approved: Updates tostatusaspect ofcontainerentity.The updates to the
statusaspect of thecontainerentity align with the PR objectives and ensure that the metadata is accurately represented.
Line range hint
60-74:
Approved: Updates todataPlatformInstanceaspect ofcontainerentity.The updates to the
dataPlatformInstanceaspect of thecontainerentity align with the PR objectives and ensure that the metadata is accurately represented.
Line range hint
77-91:
Approved: Updates tosubTypesaspect ofcontainerentity.The updates to the
subTypesaspect of thecontainerentity align with the PR objectives and ensure that the metadata is accurately represented.
Line range hint
95-109:
Approved: Updates tobrowsePathsV2aspect ofcontainerentity.The updates to the
browsePathsV2aspect of thecontainerentity align with the PR objectives and ensure that the metadata is accurately represented.
Line range hint
120-166:
Approved: Updates toChartSnapshotentity.The updates to the
ChartSnapshotentity, including the addition of platform instance information and other properties, align with the PR objectives and ensure that the metadata is accurately represented.
Line range hint
170-184:
Approved: Updates tosubTypesaspect ofchartentity.The updates to the
subTypesaspect of thechartentity align with the PR objectives and ensure that the metadata is accurately represented.
188-202: Approved: Updates todataPlatformInstanceaspect ofchartentity.The updates to the
dataPlatformInstanceaspect of thechartentity align with the PR objectives and ensure that the metadata is accurately represented.
203-217: Approved: Updates tobrowsePathsV2aspect ofchartentity.The updates to the
browsePathsV2aspect of thechartentity align with the PR objectives and ensure that the metadata is accurately represented.
Line range hint
237-273:
Approved: Updates toDashboardSnapshotentity.The updates to the
DashboardSnapshotentity, including the addition of platform instance information and other properties, align with the PR objectives and ensure that the metadata is accurately represented.
Line range hint
284-298:
Approved: Updates tocontaineraspect ofdashboardentity.The updates to the
containeraspect of thedashboardentity align with the PR objectives and ensure that the metadata is accurately represented.
Line range hint
300-314:
Approved: Updates toembedaspect ofdashboardentity.The updates to the
embedaspect of thedashboardentity align with the PR objectives and ensure that the metadata is accurately represented.
316-330: Approved: Updates todataPlatformInstanceaspect ofdashboardentity.The updates to the
dataPlatformInstanceaspect of thedashboardentity align with the PR objectives and ensure that the metadata is accurately represented.
332-350: Approved: Updates tobrowsePathsV2aspect ofdashboardentity.The updates to the
browsePathsV2aspect of thedashboardentity align with the PR objectives and ensure that the metadata is accurately represented.
Line range hint
361-403:
Approved: Updates toinputFieldsaspect ofchartentity.The updates to the
inputFieldsaspect of thechartentity align with the PR objectives and ensure that the metadata is accurately represented.
Line range hint
426-466:
Approved: Updates toinputFieldsaspect ofdashboardentity.The updates to the
inputFieldsaspect of thedashboardentity align with the PR objectives and ensure that the metadata is accurately represented.
Line range hint
477-492:
Approved: Updates tocontainerentity.The updates to the
containerentity, including the addition of platform instance information and other properties, align with the PR objectives and ensure that the metadata is accurately represented.
Line range hint
499-513:
Approved: Updates tostatusaspect ofcontainerentity.The updates to the
statusaspect of thecontainerentity align with the PR objectives and ensure that the metadata is accurately represented.
Line range hint
515-529:
Approved: Updates todataPlatformInstanceaspect ofcontainerentity.The updates to the
dataPlatformInstanceaspect of thecontainerentity align with the PR objectives and ensure that the metadata is accurately represented.
Line range hint
532-546:
Approved: Updates tosubTypesaspect ofcontainerentity.The updates to the
subTypesaspect of thecontainerentity align with the PR objectives and ensure that the metadata is accurately represented.
Line range hint
550-564:
Approved: Updates tobrowsePathsV2aspect ofcontainerentity.The updates to the
browsePathsV2aspect of thecontainerentity align with the PR objectives and ensure that the metadata is accurately represented.
Line range hint
575-667:
Approved: Updates toDatasetSnapshotentity.The updates to the
DatasetSnapshotentity, including the addition of platform instance information and other properties, align with the PR objectives and ensure that the metadata is accurately represented.
Line range hint
671-685:
Approved: Updates tosubTypesaspect ofdatasetentity.The updates to the
subTypesaspect of thedatasetentity align with the PR objectives and ensure that the metadata is accurately represented.
There was a problem hiding this comment.
Actionable comments posted: 0
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (2)
- metadata-ingestion/src/datahub/ingestion/source/looker/looker_source.py (8 hunks)
- metadata-ingestion/tests/integration/looker/golden_test_ingest.json (29 hunks)
Files skipped from review as they are similar to previous changes (1)
- metadata-ingestion/src/datahub/ingestion/source/looker/looker_source.py
Additional comments not posted (17)
metadata-ingestion/tests/integration/looker/golden_test_ingest.json (17)
Line range hint
44-58:
LGTM!The
statusaspect of thecontainerentity is correctly updated withremovedset tofalse.
Line range hint
77-93:
LGTM!The
subTypesaspect of thecontainerentity is correctly updated withtypeNamesset to "Folder".
Line range hint
170-186:
LGTM!The
subTypesaspect of thechartentity is correctly updated withtypeNamesset to "Look".
Line range hint
301-315:
LGTM!The
embedaspect of thedashboardentity is correctly updated with the newrenderUrl.
Line range hint
22-42:
LGTM! Verify the consistency of thecontainerentity.The updated
containerentity is correctly defined with newentityUrnandaspectName.Ensure that this entity is consistently referenced throughout the codebase.
Verification successful
The
containerentity is consistently referenced within the codebase.The entity
urn:li:container:e7fe6fc9c3ca70e78694dcc5dd9c05b7is used correctly with variousaspectNamevalues in themetadata-ingestion/tests/integration/looker/golden_test_ingest.jsonfile.
- No inconsistencies or errors were found in the references.
Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify the consistency of the `container` entity. # Test: Search for the `container` entity. Expect: Consistent references to the updated entity. rg --type json -A 5 $'urn:li:container:e7fe6fc9c3ca70e78694dcc5dd9c05b7'Length of output: 5285
Line range hint
95-111:
LGTM! Verify the consistency of thebrowsePathsV2aspect.The
browsePathsV2aspect of thecontainerentity is correctly updated.Ensure that this aspect is consistently referenced throughout the codebase.
317-331: LGTM! Verify the consistency of thedataPlatformInstanceaspect.The
dataPlatformInstanceaspect of thedashboardentity is correctly updated.Ensure that this aspect is consistently referenced throughout the codebase.
Line range hint
285-299:
LGTM! Verify the consistency of thecontaineraspect.The
containeraspect of thedashboardentity is correctly updated.Ensure that this aspect is consistently referenced throughout the codebase.
Verification successful
Verified: The
containeraspect of thedashboardentity is consistently referenced throughout themetadata-ingestion/tests/integration/looker/golden_test_ingest.jsonfile.
- The references to
urn:li:container:e7fe6fc9c3ca70e78694dcc5dd9c05b7are consistent and correctly updated.Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify the consistency of the `container` aspect. # Test: Search for the `container` aspect. Expect: Consistent references to the updated aspect. rg --type json -A 5 $'urn:li:container:e7fe6fc9c3ca70e78694dcc5dd9c05b7'Length of output: 5285
204-226: LGTM! Verify the consistency of thebrowsePathsV2aspect.The
browsePathsV2aspect of thechartentity is correctly updated.Ensure that this aspect is consistently referenced throughout the codebase.
Line range hint
120-167:
LGTM! Verify the consistency of theChartSnapshot.The
ChartSnapshotis correctly updated with newurn,aspects, andsystemMetadata.Ensure that this snapshot is consistently referenced throughout the codebase.
333-351: LGTM! Verify the consistency of thebrowsePathsV2aspect.The
browsePathsV2aspect of thedashboardentity is correctly updated.Ensure that this aspect is consistently referenced throughout the codebase.
2-19: LGTM! Verify the consistency of thedataPlatformInstanceentity.The new
dataPlatformInstanceentity for the region "ap-south-1" is correctly defined.Ensure that this entity is consistently referenced throughout the codebase.
Line range hint
60-74:
LGTM! Verify the consistency of thedataPlatformInstanceaspect.The
dataPlatformInstanceaspect of thecontainerentity is correctly updated.Ensure that this aspect is consistently referenced throughout the codebase.
Line range hint
238-283:
LGTM! Verify the consistency of theDashboardSnapshot.The
DashboardSnapshotis correctly updated with newurn,aspects, andsystemMetadata.Ensure that this snapshot is consistently referenced throughout the codebase.
Verification successful
Verified: The
DashboardSnapshotis consistent within the provided context.The
urnfor theDashboardSnapshotis consistently referenced within themetadata-ingestion/tests/integration/looker/golden_test_ingest.jsonfile.
metadata-ingestion/tests/integration/looker/golden_test_ingest.jsonScripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify the consistency of the `DashboardSnapshot`. # Test: Search for the `DashboardSnapshot`. Expect: Consistent references to the updated snapshot. rg --type json -A 5 $'urn:li:dashboard:(looker,ap-south-1.dashboards.1)'Length of output: 73
Script:
#!/bin/bash # Description: Verify the consistency of the `DashboardSnapshot` with the updated `urn` throughout the codebase. # Test: Search for the `DashboardSnapshot` with the updated `urn` in all file types. rg -A 5 'urn:li:dashboard:(looker,ap-south-1.dashboards.1)' # Test: Search for partial matches of the `urn` to ensure all references are captured. rg -A 5 'looker,ap-south-1.dashboards.1'Length of output: 4631
188-202: LGTM! Verify the consistency of thedataPlatformInstanceaspect.The
dataPlatformInstanceaspect of thechartentity is correctly updated.Ensure that this aspect is consistently referenced throughout the codebase.
Line range hint
420-472:
LGTM! Verify the consistency of theinputFieldsaspect.The
inputFieldsaspect of thedashboardentity is correctly updated.Ensure that this aspect is consistently referenced throughout the codebase.
Verification successful
Verification successful: The
inputFieldsaspect is consistently referenced across the codebase.
- The
inputFieldsaspect of thedashboardentity is correctly updated and consistently referenced in multiple files.Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify the consistency of the `inputFields` aspect. # Test: Search for the `inputFields` aspect. Expect: Consistent references to the updated aspect. rg --type json -A 5 $'urn:li:dashboard:(looker,ap-south-1.dashboards.1)'Length of output: 73
Script:
#!/bin/bash # Description: Verify the consistency of the `inputFields` aspect. # Test: Search for the `inputFields` aspect in the codebase. Expect: Consistent references to the updated aspect. rg --type json -A 5 '"aspectName": "inputFields"'Length of output: 217161
Line range hint
362-414:
LGTM! Verify the consistency of theinputFieldsaspect.The
inputFieldsaspect of thechartentity is correctly updated.Ensure that this aspect is consistently referenced throughout the codebase.
Verification successful
The
inputFieldsaspect is used consistently across the codebase.The references to the
inputFieldsaspect in various JSON files, particularly in thetableauandlookerintegration tests, as well as in the Python file (mode.py), confirm its consistent usage.
metadata-ingestion/tests/integration/looker/golden_test_ingest.jsonmetadata-ingestion/tests/integration/tableau/tableau_with_platform_instance_mces_golden.jsonmetadata-ingestion/src/datahub/ingestion/source/mode.pyScripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify the consistency of the `inputFields` aspect. # Test: Search for the `inputFields` aspect. Expect: Consistent references to the updated aspect. rg --type json -A 5 $'urn:li:chart:(looker,ap-south-1.dashboard_elements.2)'Length of output: 77
Script:
#!/bin/bash # Description: Verify the consistency of the `inputFields` aspect throughout the codebase. # Test: Search for the `inputFields` aspect in JSON files. rg --type json -A 5 '"aspectName": "inputFields"' # Test: Search for the `inputFields` aspect in other relevant file types (e.g., Python, JavaScript). rg --type py --type js -A 5 'inputFields'Length of output: 218032
I verified the lineage and it is working. Also updated the dev04. |
hsheth2
changed the title
* feat(forms) Handle deleting forms references when hard deleting forms (datahub-project#10820) * refactor(ui): Misc improvements to the setup ingestion flow (ingest uplift 1/2) (datahub-project#10764) Co-authored-by: John Joyce <john@Johns-MBP.lan> Co-authored-by: John Joyce <john@ip-192-168-1-200.us-west-2.compute.internal> * fix(ingestion/airflow-plugin): pipeline tasks discoverable in search (datahub-project#10819) * feat(ingest/transformer): tags to terms transformer (datahub-project#10758) Co-authored-by: Aseem Bansal <asmbansal2@gmail.com> * fix(ingestion/unity-catalog): fixed issue with profiling with GE turned on (datahub-project#10752) Co-authored-by: Aseem Bansal <asmbansal2@gmail.com> * feat(forms) Add java SDK for form entity PATCH + CRUD examples (datahub-project#10822) * feat(SDK) Add java SDK for structuredProperty entity PATCH + CRUD examples (datahub-project#10823) * feat(SDK) Add StructuredPropertyPatchBuilder in python sdk and provide sample CRUD files (datahub-project#10824) * feat(forms) Add CRUD endpoints to GraphQL for Form entities (datahub-project#10825) * add flag for includeSoftDeleted in scroll entities API (datahub-project#10831) * feat(deprecation) Return actor entity with deprecation aspect (datahub-project#10832) * feat(structuredProperties) Add CRUD graphql APIs for structured property entities (datahub-project#10826) * add scroll parameters to openapi v3 spec (datahub-project#10833) * fix(ingest): correct profile_day_of_week implementation (datahub-project#10818) * feat(ingest/glue): allow ingestion of empty databases from Glue (datahub-project#10666) Co-authored-by: Harshal Sheth <hsheth2@gmail.com> * feat(cli): add more details to get cli (datahub-project#10815) * fix(ingestion/glue): ensure date formatting works on all platforms for aws glue (datahub-project#10836) * fix(ingestion): fix datajob patcher (datahub-project#10827) * fix(smoke-test): add suffix in temp file creation (datahub-project#10841) * feat(ingest/glue): add helper method to permit user or group ownership (datahub-project#10784) * feat(): Show data platform instances in policy modal if they are set on the policy (datahub-project#10645) Co-authored-by: Hendrik Richert <hendrik.richert@swisscom.com> * docs(patch): add patch documentation for how implementation works (datahub-project#10010) Co-authored-by: John Joyce <john@acryl.io> * fix(jar): add missing custom-plugin-jar task (datahub-project#10847) * fix(): also check exceptions/stack trace when filtering log messages (datahub-project#10391) Co-authored-by: John Joyce <john@acryl.io> * docs(): Update posts.md (datahub-project#9893) Co-authored-by: Hyejin Yoon <0327jane@gmail.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * chore(ingest): update acryl-datahub-classify version (datahub-project#10844) * refactor(ingest): Refactor structured logging to support infos, warnings, and failures structured reporting to UI (datahub-project#10828) Co-authored-by: John Joyce <john@Johns-MBP.lan> Co-authored-by: Harshal Sheth <hsheth2@gmail.com> * fix(restli): log aspect-not-found as a warning rather than as an error (datahub-project#10834) * fix(ingest/nifi): remove duplicate upstream jobs (datahub-project#10849) * fix(smoke-test): test access to create/revoke personal access tokens (datahub-project#10848) * fix(smoke-test): missing test for move domain (datahub-project#10837) * ci: update usernames to not considered for community (datahub-project#10851) * env: change defaults for data contract visibility (datahub-project#10854) * fix(ingest/tableau): quote special characters in external URL (datahub-project#10842) * fix(smoke-test): fix flakiness of auto complete test * ci(ingest): pin dask dependency for feast (datahub-project#10865) * fix(ingestion/lookml): liquid template resolution and view-to-view cll (datahub-project#10542) * feat(ingest/audit): add client id and version in system metadata props (datahub-project#10829) * chore(ingest): Mypy 1.10.1 pin (datahub-project#10867) * docs: use acryl-datahub-actions as expected python package to install (datahub-project#10852) * docs: add new js snippet (datahub-project#10846) * refactor(ingestion): remove company domain for security reason (datahub-project#10839) * fix(ingestion/spark): Platform instance and column level lineage fix (datahub-project#10843) Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * feat(ingestion/tableau): optionally ingest multiple sites and create site containers (datahub-project#10498) Co-authored-by: Yanik Häni <Yanik.Haeni1@swisscom.com> * fix(ingestion/looker): Add sqlglot dependency and remove unused sqlparser (datahub-project#10874) * fix(manage-tokens): fix manage access token policy (datahub-project#10853) * Batch get entity endpoints (datahub-project#10880) * feat(system): support conditional write semantics (datahub-project#10868) * fix(build): upgrade vercel builds to Node 20.x (datahub-project#10890) * feat(ingest/lookml): shallow clone repos (datahub-project#10888) * fix(ingest/looker): add missing dependency (datahub-project#10876) * fix(ingest): only populate audit stamps where accurate (datahub-project#10604) * fix(ingest/dbt): always encode tag urns (datahub-project#10799) * fix(ingest/redshift): handle multiline alter table commands (datahub-project#10727) * fix(ingestion/looker): column name missing in explore (datahub-project#10892) * fix(lineage) Fix lineage source/dest filtering with explored per hop limit (datahub-project#10879) * feat(conditional-writes): misc updates and fixes (datahub-project#10901) * feat(ci): update outdated action (datahub-project#10899) * feat(rest-emitter): adding async flag to rest emitter (datahub-project#10902) Co-authored-by: Gabe Lyons <gabe.lyons@acryl.io> * feat(ingest): add snowflake-queries source (datahub-project#10835) * fix(ingest): improve `auto_materialize_referenced_tags_terms` error handling (datahub-project#10906) * docs: add new company to adoption list (datahub-project#10909) * refactor(redshift): Improve redshift error handling with new structured reporting system (datahub-project#10870) Co-authored-by: John Joyce <john@Johns-MBP.lan> Co-authored-by: Harshal Sheth <hsheth2@gmail.com> * feat(ui) Finalize support for all entity types on forms (datahub-project#10915) * Index ExecutionRequestResults status field (datahub-project#10811) * feat(ingest): grafana connector (datahub-project#10891) Co-authored-by: Shirshanka Das <shirshanka@apache.org> Co-authored-by: Harshal Sheth <hsheth2@gmail.com> * fix(gms) Add Form entity type to EntityTypeMapper (datahub-project#10916) * feat(dataset): add support for external url in Dataset (datahub-project#10877) * docs(saas-overview) added missing features to observe section (datahub-project#10913) Co-authored-by: John Joyce <john@acryl.io> * fix(ingest/spark): Fixing Micrometer warning (datahub-project#10882) * fix(structured properties): allow application of structured properties without schema file (datahub-project#10918) * fix(data-contracts-web) handle other schedule types (datahub-project#10919) * fix(ingestion/tableau): human-readable message for PERMISSIONS_MODE_SWITCHED error (datahub-project#10866) Co-authored-by: Harshal Sheth <hsheth2@gmail.com> * Add feature flag for view defintions (datahub-project#10914) Co-authored-by: Ethan Cartwright <ethan.cartwright@acryl.io> * feat(ingest/BigQuery): refactor+parallelize dataset metadata extraction (datahub-project#10884) * fix(airflow): add error handling around render_template() (datahub-project#10907) * feat(ingestion/sqlglot): add optional `default_dialect` parameter to sqlglot lineage (datahub-project#10830) * feat(mcp-mutator): new mcp mutator plugin (datahub-project#10904) * fix(ingest/bigquery): changes helper function to decode unicode scape sequences (datahub-project#10845) * feat(ingest/postgres): fetch table sizes for profile (datahub-project#10864) * feat(ingest/abs): Adding azure blob storage ingestion source (datahub-project#10813) * fix(ingest/redshift): reduce severity of SQL parsing issues (datahub-project#10924) * fix(build): fix lint fix web react (datahub-project#10896) * fix(ingest/bigquery): handle quota exceeded for project.list requests (datahub-project#10912) * feat(ingest): report extractor failures more loudly (datahub-project#10908) * feat(ingest/snowflake): integrate snowflake-queries into main source (datahub-project#10905) * fix(ingest): fix docs build (datahub-project#10926) * fix(ingest/snowflake): fix test connection (datahub-project#10927) * fix(ingest/lookml): add view load failures to cache (datahub-project#10923) * docs(slack) overhauled setup instructions and screenshots (datahub-project#10922) Co-authored-by: John Joyce <john@acryl.io> * fix(airflow): Add comma parsing of owners to DataJobs (datahub-project#10903) * fix(entityservice): fix merging sideeffects (datahub-project#10937) * feat(ingest): Support System Ingestion Sources, Show and hide system ingestion sources with Command-S (datahub-project#10938) Co-authored-by: John Joyce <john@Johns-MBP.lan> * chore() Set a default lineage filtering end time on backend when a start time is present (datahub-project#10925) Co-authored-by: John Joyce <john@ip-192-168-1-200.us-west-2.compute.internal> Co-authored-by: John Joyce <john@Johns-MBP.lan> * Added relationships APIs to V3. Added these generic APIs to V3 swagger doc. (datahub-project#10939) * docs: add learning center to docs (datahub-project#10921) * doc: Update hubspot form id (datahub-project#10943) * chore(airflow): add python 3.11 w/ Airflow 2.9 to CI (datahub-project#10941) * fix(ingest/Glue): column upstream lineage between S3 and Glue (datahub-project#10895) * fix(ingest/abs): split abs utils into multiple files (datahub-project#10945) * doc(ingest/looker): fix doc for sql parsing documentation (datahub-project#10883) Co-authored-by: Harshal Sheth <hsheth2@gmail.com> * fix(ingest/bigquery): Adding missing BigQuery types (datahub-project#10950) * fix(ingest/setup): feast and abs source setup (datahub-project#10951) * fix(connections) Harden adding /gms to connections in backend (datahub-project#10942) * feat(siblings) Add flag to prevent combining siblings in the UI (datahub-project#10952) * fix(docs): make graphql doc gen more automated (datahub-project#10953) * feat(ingest/athena): Add option for Athena partitioned profiling (datahub-project#10723) * fix(spark-lineage): default timeout for future responses (datahub-project#10947) * feat(datajob/flow): add environment filter using info aspects (datahub-project#10814) * fix(ui/ingest): correct privilege used to show tab (datahub-project#10483) Co-authored-by: Kunal-kankriya <127090035+Kunal-kankriya@users.noreply.github.com> * feat(ingest/looker): include dashboard urns in browse v2 (datahub-project#10955) * add a structured type to batchGet in OpenAPI V3 spec (datahub-project#10956) * fix(ui): scroll on the domain sidebar to show all domains (datahub-project#10966) * fix(ingest/sagemaker): resolve incorrect variable assignment for SageMaker API call (datahub-project#10965) * fix(airflow/build): Pinning mypy (datahub-project#10972) * Fixed a bug where the OpenAPI V3 spec was incorrect. The bug was introduced in datahub-project#10939. (datahub-project#10974) * fix(ingest/test): Fix for mssql integration tests (datahub-project#10978) * fix(entity-service) exist check correctly extracts status (datahub-project#10973) * fix(structuredProps) casing bug in StructuredPropertiesValidator (datahub-project#10982) * bugfix: use anyOf instead of allOf when creating references in openapi v3 spec (datahub-project#10986) * fix(ui): Remove ant less imports (datahub-project#10988) * feat(ingest/graph): Add get_results_by_filter to DataHubGraph (datahub-project#10987) * feat(ingest/cli): init does not actually support environment variables (datahub-project#10989) * fix(ingest/graph): Update get_results_by_filter graphql query (datahub-project#10991) * feat(ingest/spark): Promote beta plugin (datahub-project#10881) Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * feat(ingest): support domains in meta -> "datahub" section (datahub-project#10967) * feat(ingest): add `check server-config` command (datahub-project#10990) * feat(cli): Make consistent use of DataHubGraphClientConfig (datahub-project#10466) Deprecates get_url_and_token() in favor of a more complete option: load_graph_config() that returns a full DatahubClientConfig. This change was then propagated across previous usages of get_url_and_token so that connections to DataHub server from the client respect the full breadth of configuration specified by DatahubClientConfig. I.e: You can now specify disable_ssl_verification: true in your ~/.datahubenv file so that all cli functions to the server work when ssl certification is disabled. Fixes datahub-project#9705 * fix(ingest/s3): Fixing container creation when there is no folder in path (datahub-project#10993) * fix(ingest/looker): support platform instance for dashboards & charts (datahub-project#10771) * feat(ingest/bigquery): improve handling of information schema in sql parser (datahub-project#10985) * feat(ingest): improve `ingest deploy` command (datahub-project#10944) * fix(backend): allow excluding soft-deleted entities in relationship-queries; exclude soft-deleted members of groups (datahub-project#10920) - allow excluding soft-deleted entities in relationship-queries - exclude soft-deleted members of groups * fix(ingest/looker): downgrade missing chart type log level (datahub-project#10996) * doc(acryl-cloud): release docs for 0.3.4.x (datahub-project#10984) Co-authored-by: John Joyce <john@acryl.io> Co-authored-by: RyanHolstien <RyanHolstien@users.noreply.github.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Co-authored-by: Pedro Silva <pedro@acryl.io> * fix(protobuf/build): Fix protobuf check jar script (datahub-project#11006) * fix(ui/ingest): Support invalid cron jobs (datahub-project#10998) * fix(ingest): fix graph config loading (datahub-project#11002) Co-authored-by: Pedro Silva <pedro@acryl.io> * feat(docs): Document __DATAHUB_TO_FILE_ directive (datahub-project#10968) Co-authored-by: Harshal Sheth <hsheth2@gmail.com> * fix(graphql/upsertIngestionSource): Validate cron schedule; parse error in CLI (datahub-project#11011) * feat(ece): support custom ownership type urns in ECE generation (datahub-project#10999) * feat(assertion-v2): changed Validation tab to Quality and created new Governance tab (datahub-project#10935) * fix(ingestion/glue): Add support for missing config options for profiling in Glue (datahub-project#10858) * feat(propagation): Add models for schema field docs, tags, terms (datahub-project#2959) (datahub-project#11016) Co-authored-by: Chris Collins <chriscollins3456@gmail.com> * docs: standardize terminology to DataHub Cloud (datahub-project#11003) * fix(ingestion/transformer): replace the externalUrl container (datahub-project#11013) * docs(slack) troubleshoot docs (datahub-project#11014) * feat(propagation): Add graphql API (datahub-project#11030) Co-authored-by: Chris Collins <chriscollins3456@gmail.com> * feat(propagation): Add models for Action feature settings (datahub-project#11029) * docs(custom properties): Remove duplicate from sidebar (datahub-project#11033) * feat(models): Introducing Dataset Partitions Aspect (datahub-project#10997) Co-authored-by: John Joyce <john@Johns-MBP.lan> Co-authored-by: John Joyce <john@ip-192-168-1-200.us-west-2.compute.internal> * feat(propagation): Add Documentation Propagation Settings (datahub-project#11038) * fix(models): chart schema fields mapping, add dataHubAction entity, t… (datahub-project#11040) * fix(ci): smoke test lint failures (datahub-project#11044) * docs: fix learning center color scheme & typo (datahub-project#11043) * feat: add cloud main page (datahub-project#11017) Co-authored-by: Jay <159848059+jayacryl@users.noreply.github.com> * feat(restore-indices): add additional step to also clear system metadata service (datahub-project#10662) Co-authored-by: John Joyce <john@acryl.io> * docs: fix typo (datahub-project#11046) * fix(lint): apply spotless (datahub-project#11050) * docs(airflow): example query to get datajobs for a dataflow (datahub-project#11034) * feat(cli): Add run-id option to put sub-command (datahub-project#11023) Adds an option to assign run-id to a given put command execution. This is useful when transformers do not exist for a given ingestion payload, we can follow up with custom metadata and assign it to an ingestion pipeline. * fix(ingest): improve sql error reporting calls (datahub-project#11025) * fix(airflow): fix CI setup (datahub-project#11031) * feat(ingest/dbt): add experimental `prefer_sql_parser_lineage` flag (datahub-project#11039) * fix(ingestion/lookml): enable stack-trace in lookml logs (datahub-project#10971) * (chore): Linting fix (datahub-project#11015) * chore(ci): update deprecated github actions (datahub-project#10977) * Fix ALB configuration example (datahub-project#10981) * chore(ingestion-base): bump base image packages (datahub-project#11053) * feat(cli): Trim report of dataHubExecutionRequestResult to max GMS size (datahub-project#11051) * fix(ingestion/lookml): emit dummy sql condition for lookml custom condition tag (datahub-project#11008) Co-authored-by: Harshal Sheth <hsheth2@gmail.com> * fix(ingestion/powerbi): fix issue with broken report lineage (datahub-project#10910) * feat(ingest/tableau): add retry on timeout (datahub-project#10995) * change generate kafka connect properties from env (datahub-project#10545) Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com> * fix(ingest): fix oracle cronjob ingestion (datahub-project#11001) Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com> * chore(ci): revert update deprecated github actions (datahub-project#10977) (datahub-project#11062) * feat(ingest/dbt-cloud): update metadata_endpoint inference (datahub-project#11041) * build: Reduce size of datahub-frontend-react image by 50-ish% (datahub-project#10878) Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com> * fix(ci): Fix lint issue in datahub_ingestion_run_summary_provider.py (datahub-project#11063) * docs(ingest): update developing-a-transformer.md (datahub-project#11019) * feat(search-test): update search tests from datahub-project#10408 (datahub-project#11056) * feat(cli): add aspects parameter to DataHubGraph.get_entity_semityped (datahub-project#11009) Co-authored-by: Harshal Sheth <hsheth2@gmail.com> * docs(airflow): update min version for plugin v2 (datahub-project#11065) * doc(ingestion/tableau): doc update for derived permission (datahub-project#11054) Co-authored-by: Pedro Silva <pedro.cls93@gmail.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Co-authored-by: Harshal Sheth <hsheth2@gmail.com> * fix(py): remove dep on types-pkg_resources (datahub-project#11076) * feat(ingest/mode): add option to exclude restricted (datahub-project#11081) * fix(ingest): set lastObserved in sdk when unset (datahub-project#11071) * doc(ingest): Update capabilities (datahub-project#11072) * chore(vulnerability): Log Injection (datahub-project#11090) * chore(vulnerability): Information exposure through a stack trace (datahub-project#11091) * chore(vulnerability): Comparison of narrow type with wide type in loop condition (datahub-project#11089) * chore(vulnerability): Insertion of sensitive information into log files (datahub-project#11088) * chore(vulnerability): Risky Cryptographic Algorithm (datahub-project#11059) * chore(vulnerability): Overly permissive regex range (datahub-project#11061) Co-authored-by: Harshal Sheth <hsheth2@gmail.com> * fix: update customer data (datahub-project#11075) * fix(models): fixing the datasetPartition models (datahub-project#11085) Co-authored-by: John Joyce <john@ip-192-168-1-200.us-west-2.compute.internal> * fix(ui): Adding view, forms GraphQL query, remove showing a fallback error message on unhandled GraphQL error (datahub-project#11084) Co-authored-by: John Joyce <john@ip-192-168-1-200.us-west-2.compute.internal> * feat(docs-site): hiding learn more from cloud page (datahub-project#11097) * fix(docs): Add correct usage of orFilters in search API docs (datahub-project#11082) Co-authored-by: Jay <159848059+jayacryl@users.noreply.github.com> * fix(ingest/mode): Regexp in mode name matcher didn't allow underscore (datahub-project#11098) * docs: Refactor customer stories section (datahub-project#10869) Co-authored-by: Jeff Merrick <jeff@wireform.io> * fix(release): fix full/slim suffix on tag (datahub-project#11087) * feat(config): support alternate hashing algorithm for doc id (datahub-project#10423) Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com> Co-authored-by: John Joyce <john@acryl.io> * fix(emitter): fix typo in get method of java kafka emitter (datahub-project#11007) * fix(ingest): use correct native data type in all SQLAlchemy sources by compiling data type using dialect (datahub-project#10898) Co-authored-by: Harshal Sheth <hsheth2@gmail.com> * chore: Update contributors list in PR labeler (datahub-project#11105) * feat(ingest): tweak stale entity removal messaging (datahub-project#11064) * fix(ingestion): enforce lastObserved timestamps in SystemMetadata (datahub-project#11104) * fix(ingest/powerbi): fix broken lineage between chart and dataset (datahub-project#11080) * feat(ingest/lookml): CLL support for sql set in sql_table_name attribute of lookml view (datahub-project#11069) * docs: update graphql docs on forms & structured properties (datahub-project#11100) * test(search): search openAPI v3 test (datahub-project#11049) * fix(ingest/tableau): prevent empty site content urls (datahub-project#11057) Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * feat(entity-client): implement client batch interface (datahub-project#11106) * fix(snowflake): avoid reporting warnings/info for sys tables (datahub-project#11114) * fix(ingest): downgrade column type mapping warning to info (datahub-project#11115) * feat(api): add AuditStamp to the V3 API entity/aspect response (datahub-project#11118) * fix(ingest/redshift): replace r'\n' with '\n' to avoid token error redshift serverless… (datahub-project#11111) * fix(entiy-client): handle null entityUrn case for restli (datahub-project#11122) * fix(sql-parser): prevent bad urns from alter table lineage (datahub-project#11092) * fix(ingest/bigquery): use small batch size if use_tables_list_query_v2 is set (datahub-project#11121) * fix(graphql): add missing entities to EntityTypeMapper and EntityTypeUrnMapper (datahub-project#10366) * feat(ui): Changes to allow editable dataset name (datahub-project#10608) Co-authored-by: Jay Kadambi <jayasimhan_venkatadri@optum.com> * fix: remove saxo (datahub-project#11127) * feat(mcl-processor): Update mcl processor hooks (datahub-project#11134) * fix(openapi): fix openapi v2 endpoints & v3 documentation update * Revert "fix(openapi): fix openapi v2 endpoints & v3 documentation update" This reverts commit 573c1cb. * docs(policies): updates to policies documentation (datahub-project#11073) * fix(openapi): fix openapi v2 and v3 docs update (datahub-project#11139) * feat(auth): grant type and acr values custom oidc parameters support (datahub-project#11116) * fix(mutator): mutator hook fixes (datahub-project#11140) * feat(search): support sorting on multiple fields (datahub-project#10775) * feat(ingest): various logging improvements (datahub-project#11126) * fix(ingestion/lookml): fix for sql parsing error (datahub-project#11079) Co-authored-by: Harshal Sheth <hsheth2@gmail.com> * feat(docs-site) cloud page spacing and content polishes (datahub-project#11141) * feat(ui) Enable editing structured props on fields (datahub-project#11042) * feat(tests): add md5 and last computed to testResult model (datahub-project#11117) * test(openapi): openapi regression smoke tests (datahub-project#11143) * fix(airflow): fix tox tests + update docs (datahub-project#11125) * docs: add chime to adoption stories (datahub-project#11142) * fix(ingest/databricks): Updating code to work with Databricks sdk 0.30 (datahub-project#11158) * fix(kafka-setup): add missing script to image (datahub-project#11190) * fix(config): fix hash algo config (datahub-project#11191) * test(smoke-test): updates to smoke-tests (datahub-project#11152) * fix(elasticsearch): refactor idHashAlgo setting (datahub-project#11193) * chore(kafka): kafka version bump (datahub-project#11211) * readd UsageStatsWorkUnit * fix merge problems * change logo --------- Co-authored-by: Chris Collins <chriscollins3456@gmail.com> Co-authored-by: John Joyce <john@acryl.io> Co-authored-by: John Joyce <john@Johns-MBP.lan> Co-authored-by: John Joyce <john@ip-192-168-1-200.us-west-2.compute.internal> Co-authored-by: dushayntAW <158567391+dushayntAW@users.noreply.github.com> Co-authored-by: sagar-salvi-apptware <159135491+sagar-salvi-apptware@users.noreply.github.com> Co-authored-by: Aseem Bansal <asmbansal2@gmail.com> Co-authored-by: Kevin Chun <kevin1chun@gmail.com> Co-authored-by: jordanjeremy <72943478+jordanjeremy@users.noreply.github.com> Co-authored-by: skrydal <piotr.skrydalewicz@gmail.com> Co-authored-by: Harshal Sheth <hsheth2@gmail.com> Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com> Co-authored-by: sid-acryl <155424659+sid-acryl@users.noreply.github.com> Co-authored-by: Julien Jehannet <80408664+aviv-julienjehannet@users.noreply.github.com> Co-authored-by: Hendrik Richert <github@richert.li> Co-authored-by: Hendrik Richert <hendrik.richert@swisscom.com> Co-authored-by: RyanHolstien <RyanHolstien@users.noreply.github.com> Co-authored-by: Felix Lüdin <13187726+Masterchen09@users.noreply.github.com> Co-authored-by: Pirry <158024088+chardaway@users.noreply.github.com> Co-authored-by: Hyejin Yoon <0327jane@gmail.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Co-authored-by: cburroughs <chris.burroughs@gmail.com> Co-authored-by: ksrinath <ksrinath@users.noreply.github.com> Co-authored-by: Mayuri Nehate <33225191+mayurinehate@users.noreply.github.com> Co-authored-by: Kunal-kankriya <127090035+Kunal-kankriya@users.noreply.github.com> Co-authored-by: Shirshanka Das <shirshanka@apache.org> Co-authored-by: ipolding-cais <155455744+ipolding-cais@users.noreply.github.com> Co-authored-by: Tamas Nemeth <treff7es@gmail.com> Co-authored-by: Shubham Jagtap <132359390+shubhamjagtap639@users.noreply.github.com> Co-authored-by: haeniya <yanik.haeni@gmail.com> Co-authored-by: Yanik Häni <Yanik.Haeni1@swisscom.com> Co-authored-by: Gabe Lyons <itsgabelyons@gmail.com> Co-authored-by: Gabe Lyons <gabe.lyons@acryl.io> Co-authored-by: 808OVADOZE <52988741+shtephlee@users.noreply.github.com> Co-authored-by: noggi <anton.kuraev@acryl.io> Co-authored-by: Nicholas Pena <npena@foursquare.com> Co-authored-by: Jay <159848059+jayacryl@users.noreply.github.com> Co-authored-by: ethan-cartwright <ethan.cartwright.m@gmail.com> Co-authored-by: Ethan Cartwright <ethan.cartwright@acryl.io> Co-authored-by: Nadav Gross <33874964+nadavgross@users.noreply.github.com> Co-authored-by: Patrick Franco Braz <patrickfbraz@poli.ufrj.br> Co-authored-by: pie1nthesky <39328908+pie1nthesky@users.noreply.github.com> Co-authored-by: Joel Pinto Mata (KPN-DSH-DEX team) <130968841+joelmataKPN@users.noreply.github.com> Co-authored-by: Ellie O'Neil <110510035+eboneil@users.noreply.github.com> Co-authored-by: Ajoy Majumdar <ajoymajumdar@hotmail.com> Co-authored-by: deepgarg-visa <149145061+deepgarg-visa@users.noreply.github.com> Co-authored-by: Tristan Heisler <tristankheisler@gmail.com> Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io> Co-authored-by: Davi Arnaut <davi.arnaut@acryl.io> Co-authored-by: Pedro Silva <pedro@acryl.io> Co-authored-by: amit-apptware <132869468+amit-apptware@users.noreply.github.com> Co-authored-by: Sam Black <sam.black@acryl.io> Co-authored-by: Raj Tekal <varadaraj_tekal@optum.com> Co-authored-by: Steffen Grohsschmiedt <gitbhub@steffeng.eu> Co-authored-by: jaegwon.seo <162448493+wornjs@users.noreply.github.com> Co-authored-by: Renan F. Lima <51028757+lima-renan@users.noreply.github.com> Co-authored-by: Matt Exchange <xkollar@users.noreply.github.com> Co-authored-by: Jonny Dixon <45681293+acrylJonny@users.noreply.github.com> Co-authored-by: Pedro Silva <pedro.cls93@gmail.com> Co-authored-by: Pinaki Bhattacharjee <pinakipb2@gmail.com> Co-authored-by: Jeff Merrick <jeff@wireform.io> Co-authored-by: skrydal <piotr.skrydalewicz@acryl.io> Co-authored-by: AndreasHegerNuritas <163423418+AndreasHegerNuritas@users.noreply.github.com> Co-authored-by: jayasimhankv <145704974+jayasimhankv@users.noreply.github.com> Co-authored-by: Jay Kadambi <jayasimhan_venkatadri@optum.com> Co-authored-by: David Leifker <david.leifker@acryl.io>
Follow up on #10771 That PR updated most looker urns to use the platform instance when configured, but missed updating the usage code. This finishes the job.
We are currently excluding platform-instance while generating urn for dashboards and charts .
This PR introducing a configurable option
include_looker_element_in_platform_instancewhich can be used if user wish to consider platform-instance in dashboards and charts urn generation.Summary by CodeRabbit
New Features
Refactor
Tests