refactor(ingest/bigquery): Standardize audit log parsing and make TopKDict a DefaultDict #7738
Conversation
…t creation; standardize audit log parsing; make TopKDict a DefaultDict
github-actions
bot
added
the
ingestion
| lineage_extractor: BigqueryLineageExtractor = BigqueryLineageExtractor( | ||
| config=self.config, report=self.report | ||
| ) |
There was a problem hiding this comment.
No clue why we were doing this... was in this code from the very first bigquery lineage PR. The only instance variables are self.config and self.report, which are passed thru anyway, so I think this should be safe to remove
| raise e | ||
|
|
||
| def compute_bigquery_lineage_via_catalog_lineage_api( | ||
| def lineage_via_catalog_lineage_api( |
There was a problem hiding this comment.
Renamed compute_bigquery_lineage_via_catalog_lineage_api to be a bit more concise
| def _get_bigquery_log_entries( | ||
| self, client: GCPLoggingClient, limit: Optional[int] = None | ||
| ) -> Union[Iterable[AuditLogEntry], Iterable[BigQueryAuditMetadata]]: | ||
| ) -> Union[Iterable[AuditLogEntry]]: |
There was a problem hiding this comment.
Think this is a more accurate typing. Technically it doesn't matter because both AuditLogEntry and BigQueryAuditMetadata are Any
| logger.warning( | ||
| f"Unable to parse log missing {missing_entry}, missing v2 {missing_entry_v2} for {entry}", | ||
| ) |
There was a problem hiding this comment.
Seemed more fitting as a log warning, rather than in the report as an error
| self.error( | ||
| logger, | ||
| "lineage", | ||
| f"{project_id}: {e}", |
There was a problem hiding this comment.
This is to replace the self.error calls in the deleted compute_bigquery_lineage_via_gcp_logging and compute_bigquery_lineage_via_audit_metadata. Since this is catching unexpected errors and has a relatively clean message, I think this is valid to put in the report.
| @@ -492,7 +487,7 @@ def _extract_operational_meta( | |||
| else: | |||
| return None | |||
|
|
|||
| def _create_operation_aspect_work_unit( | |||
| def _create_operation_workunit( | |||
Bit again by not separating out some changes into a smaller PR. Takes changes from #7663 and extends them for lineage.py, i.e.:
get_sanitized_table_refcalls to ReadEvent / QueryEvent creationget_sanitized_table_refcallsAlso. makes
TopKDictinherit fromDefaultDictand cleans up calls around that.Checklist