fix(ingest/snowflake): missing view downstream cll if platform instance is set #8966
Conversation
github-actions
bot
added
the
ingestion
mayurinehate
changed the title
| upstream_column_info.table | ||
| ).get_dataset_name() | ||
| if self.config.platform_instance and upstream_table_id.startswith( |
There was a problem hiding this comment.
How does this happen? Perhaps we should separate platform instance out in the sql parser instead? cc @hsheth2
There was a problem hiding this comment.
imo the ideal outcome would be for the sql parser to respect platform_instance if one is passed in to the schema resolver
There was a problem hiding this comment.
I thought it already does that, but might be wrong
There was a problem hiding this comment.
So the sql lineage returned has
- dataset urn - with platform_instance
- column to column lineage where each column is represented by (dataset urn, column_name)
What's missing here is that - we don't have raw dataset name (or fully qualified name) without platform instance in the upstream result.
There was a problem hiding this comment.
@hsheth2 @asikowitz any thoughts on how to proceed here ?
There was a problem hiding this comment.
I think, this bug in snowflake view lineage can also be solved by converting sql parsing result to fine grained lineage directly instead of raw snowflake data models again. Let me attempt that.
There was a problem hiding this comment.
Done. @hsheth2 please check now.
maggiehays
added
the
hacktoberfest-accepted
mayurinehate
force-pushed
the
snowflake_view_downstream_cll_for_platforminstance
branch
from
9dcd287
to
dab2eed
Compare
hsheth2
added
the
merge-pending-ci
|
@mayurinehate looks like there's a conflict on this PR |
Resolved. |
Checklist