dataset field level Lineage support?

[clojurians-org]

i want to parse the sql to get the dataset field level Lineage by some open-source tool:
queryparser [https://github.com/uber/queryparser] for [hive, vertica, presto]
hssqlppp [https://github.com/JakeWheat/hssqlppp] for [oracle, postgresql]

i make the kafka avro message, and i want to push it.
but it don't find the Lineage for dataset field level.

i can just find the related information by upstreams from MetadataChangeEvent.avsc.

what's the adjustments i need to do it for such requirement?

[keremsahin1]

Hi @clojurians-org

DataHub doesn't support field level lineage at this point. We only have a support for dataset level lineage. You can see upstreams or downstreams of a specific dataset such as a mysql table. It's on our roadmap to have this but can't give timeline estimates for that. But, seems like we have fair amount request for this feature. We'll definitely consider this while defining our roadmap.

Thanks

[clojurians-org]

i already finish the table level field extract by parsing hive sql if you're intested.
i will add oracle stored procedure parsing later.

field-level lineage is very easy to support by backend parsing. but it need ui model to display.

  cd contrib/metadata_etl
  cat metadata_sample/hive_1.sql | bin/lineage_hive_generator.hs

https://github.com/clojurians-org/simple-datahub/blob/master/contrib/metadata-etl/bin/lineage_hive_generator.hs

[mars-lan]

Let's concentrate the discussion in this issue: #1731