feat(ingest/bigquery): Respect dataset and table patterns when ingesting lineage via catalog api

[ANich]

Similar to when ingesting lineage via GCP audit logs:

datahub/metadata-ingestion/src/datahub/ingestion/source/bigquery_v2/lineage.py

Lines 733 to 742 in 06ef2ad

	if not is_schema_allowed(
	self.config.dataset_pattern,
	destination_table.table_identifier.dataset,
	destination_table.table_identifier.project_id,
	self.config.match_fully_qualified_names,
	) or not self.config.table_pattern.allowed(
	destination_table.table_identifier.get_table_name()
	):
	self.report.num_skipped_lineage_entries_not_allowed[e.project_id] += 1
	continue

Currently, when using extract_lineage_from_catalog=True a request is issued for every table in the project.

Checklist

• The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
• Links to related issues (if applicable)
• Tests for the changes have been added/updated (if applicable)
• Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
• For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

[mayurinehate]

self.config.table_pattern needs to be checked against qualified table name <project>.<dataset>.<table> . I would suggest moving the table variable definition
table = f"{project_table.project}.{project_table.dataset_id}.{project_table.table_id}"
before if condition and using it here.

[ANich]

Thank you, I didn't notice that table patterns expected qualified names. I'll have to update my own recipes as well. Updated in 5cbb1b1

[mayurinehate]

LGTM