fix(metadata-io): in Neo4j service use proper algorithm to get lineage #8687
Conversation
github-actions
bot
added
product
|
Nice job! |
|
APOC needs to be enabled in Neo4j for this change to work. In particular, for deployment via Helm chart, this PR is needed: The dependency should be communicated via change logs. |
lix-mms
force-pushed
the
fix/metadata-io/use-proper-algorithm-to-get-lineage
branch
from
d01fa6c
to
5dd0dc7
Compare
lix-mms
force-pushed
the
fix/metadata-io/use-proper-algorithm-to-get-lineage
branch
from
6c185b9
to
8678241
Compare
|
Trying to figure out why |
* this bypasses dependency issues with apoc:all:4.4.0.9
|
I made a fix and (Needed to use a slightly higher version of neo4j apoc as a test dependency to avoid some dependency issue on snakeyaml.) |
maggiehays
added
the
community-contribution
…se-proper-algorithm-to-get-lineage # Conflicts: # metadata-io/build.gradle
|
Can you please make a note of this breaking change in updating-datahub.md? Thanks! Otherwise LGTM :) |
|
@RyanHolstien Thanks! I'll update it as soon as acryldata/datahub-helm#353 is merged. Btw, should we also add a breaking change tip line for the replacement of the Neo4j chart (PR: acryldata/datahub-helm#365)? It looks like the existing chart overrides will not automatically work with the new chart structure. |
|
@RyanHolstien I added a breaking change note to cover this PR change and the change introduced in acryldata/datahub-helm#365. Please check 😋 |
|
Not sure why the build checks start to fail after merging master and the errors do not seem to be relevant to this PR. Any help is appreciated! |
|
Only test |
|
Tests are passing. Mind a final check for merge? 😋 @RyanHolstien |
lix-mms
deleted the
fix/metadata-io/use-proper-algorithm-to-get-lineage
branch
The existing
shortestPathalgorithm supporting the lineage queries has performance issue when the graph database has a high amount of dataset nodes (say >100K). According to Neo4j Cypher PROFILE, it looks like the shortestPath query tries to enumerate all the nodes in the graph database to compute a shortestPath from the start node, which is unnecessary since usually the reachable nodes from the start node are only a very small portion of the whole set of nodes.Such a performance issue basically leads to that the lineage features in the UI of our DataHub do not work at all. (GraphQL queries
searchAcrossLineageorgetLineagealways time out, but in the background the execution of the query continues until it finishes after a long time.)After extensive investigations, we found that apoc.path.spanningTree is a better alternative to the existing algorithm. It is uses Breadth-First-Search to expand the lineage paths from the start node, efficiently returns collected paths and does not explore any non-reachable nodes.
With a fix via this PR, in our case, the query time given a node having >200 downstream lineages reduced from
90s to roughly 13s or even less, which is a substantial improvement.In addition, I restructured how the query statements are formatted, which should lead to better readability and easier maintenance.
Checklist
Links to related issues (if applicable)Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub