feat(models) : Joins (Datasets) schema, resolvers and UI #8325

poorvi767 · 2023-06-28T15:37:30Z

Checklist

The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
Links to related issues (if applicable)
Tests for the changes have been added/updated (if applicable)
Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

poorvi767 · 2023-06-28T15:44:26Z

We have closed previous PR: #7961 by @rtekal and created this PR with additional changes containing updated resolvers and UI.

chriscollins3456

nice! this is looking solid, thank you so much for your effort here we're really appreciative.

I left some comments, some more serious than others.

In my first review I only had time to get through the modeling and backend changes. I'll take more of a look at datahub-web-react soon!

chriscollins3456 · 2023-07-10T22:02:17Z

datahub-graphql-core/src/main/java/com/linkedin/datahub/graphql/GmsGraphQLEngine.java

@@ -701,6 +710,9 @@ private void configureDataPlatformInstanceResolvers(final RuntimeWiring.Builder

    private void configureQueryResolvers(final RuntimeWiring.Builder builder) {
        builder.type("Query", typeWiring -> typeWiring
+            .dataFetcher("join", new AuthenticatedResolver<>(


AuthenticatedResolver is actually deprecated and an authentication check is handled elsewhere so if the user isn't logged in they can't get to any of these endpoints! so feel free to remove it from here as well as below

also i see duplicate "join" entries here in configureQueryResolvers so I think you can just remove this one entirely.

chriscollins3456 · 2023-07-10T22:07:10Z

metadata-models/src/main/resources/entity-registry.yml

+      - container
+      - globalTags
+      - glossaryTerms
+      - browsePaths


hmm i'm not sure I see browsePaths on joins right now since they won't have their own entity profile page. also unsure if we need container or institutionalMemory aspects either

At the present, I removed BrowsePaths and Container
InstitutionalMemory is needed since we are providing a place for the documentation.

chriscollins3456 · 2023-07-10T22:09:30Z

datahub-graphql-core/src/main/java/com/linkedin/datahub/graphql/types/join/JoinType.java

+  private com.linkedin.common.urn.JoinUrn getJoinUrn(String urnStr) {
+    try {
+      return JoinUrn.createFromString(urnStr);
+    } catch (URISyntaxException e) {
+      throw new RuntimeException(String.format("Failed to retrieve data product with urn %s, invalid urn", urnStr));
+    }
+  }


this method and getUrn above look to be duplicated. Also we have this method getUrn to do just this in a generic UrnUtils class you can use!

still a nit but I don't think you need this method but can just use the UrnUtils.getUrn instead

chriscollins3456 · 2023-07-10T22:13:02Z

datahub-graphql-core/src/main/resources/entity.graphql

+    """
+    List all Join
+    """
+    listJoin(input: ListJoinInput!): ListJoinResult


hm I don't see this implemented anywhere in GmsGraphQLEngine - are you still planning on creating this endpoint?

and when would we list all joins in general? won't it always be in the context of a dataset?

Addressed, In the beginning made stubs for all endpoints, yeah not needed.

chriscollins3456 · 2023-07-10T22:13:43Z

datahub-graphql-core/src/main/resources/entity.graphql

+    """
+    Edges extending from this entity grouped by direction in the lineage graph
+    """
+    lineage(input: LineageInput!): EntityLineageResult


is there lineage on Joins? I don't think so right? there are Relationships

Correct - no lineage on Joins but relationships between datasets and joins. Removed lineage

chriscollins3456 · 2023-07-10T22:48:50Z

datahub-graphql-core/src/main/resources/entity.graphql

+    """
+    Display name of the Dataset
+    """
+    name: String
+


hm is this intended? I don't think we're trying to allow dataset names to be editable

Oops - removed.

chriscollins3456 · 2023-07-10T22:52:36Z

li-utils/src/main/pegasus/com/linkedin/common/JoinUrn.pdl

+  "owners" : [ "urn:li:corpuser:fbar", "urn:li:corpuser:bfoo" ],
+  "fields" : [ {
+    "name" : "joinId",
+    "doc" : "Join native name e.g. <db>.<table>, /dir/subdir/<name>, or <name>",


this will be a UUID right? otherwise we wouldn't be able to have two joins with the same name and that could be problematic between all of the joins and datasets out there

chriscollins3456 · 2023-07-10T22:58:15Z

metadata-models/src/main/pegasus/com/linkedin/join/FieldMap.pdl

+  @Searchable = {
+    "fieldType": "TEXT_PARTIAL",
+    "enableAutocomplete": true,
+    "boostScore": 10.0
+  }
+  afield: SchemaFieldPath
+
+  /**
+   * All fields from dataset B that are required for the join, maps to aFields 1:1
+   */
+  @Searchable = {
+    "fieldType": "TEXT_PARTIAL",
+    "enableAutocomplete": true,
+    "boostScore": 10.0
+  }


let's make these TEXT fields instead of TEXT_PARTIAL since that's what we have on other schema fields. also we don't want to enableAutoComplete so we can just remove that. also i think you can remove boostScore here

chriscollins3456 · 2023-07-10T23:04:08Z

metadata-models/src/main/pegasus/com/linkedin/join/JoinProperties.pdl

+      "fieldType": "DATETIME"
+    }
+  }
+  lastModified: optional TimeStamp


how about using AuditStamp instead of TimeStamp? it just gives us some more optional fields to be more flexible in the future and requires the actor field to be populated

chriscollins3456 · 2023-07-10T23:04:56Z

metadata-models/src/main/pegasus/com/linkedin/metadata/key/JoinKey.pdl

+  @Searchable = {
+    "fieldType": "TEXT",
+  }


do we need this to be searchable? it looks like right now it's just a random UUID and we're making the name searchable anyways for joins

Yeah agreed... removed.

chriscollins3456

got through most of the frontend code as well! I'm going to pull this down and test locally later this week and may have some more feedback then as well. In general, frontend code could be broken down into smaller, more consumable chunks. I'm pretty concerned about the number of queries we're making and all of the updates to our .graphql files as well that seem unrelated (changing these files can have big impact on the performance of our queries)

chriscollins3456 · 2023-07-11T20:31:47Z

datahub-web-react/src/app/entity/dataset/DatasetEntity.tsx

@@ -96,6 +97,10 @@ export class DatasetEntity implements Entity<Dataset> {
                    name: 'Schema',
                    component: SchemaTab,
                },
+                {
+                    name: 'Relationships',
+                    component: RelationTab,


maybe we can rename this component RelationshipsTab?

chriscollins3456 · 2023-07-11T20:33:02Z

datahub-web-react/src/app/entity/shared/tabs/Dataset/Relationship/RelationTab.tsx

+    // const fieldSortInput: FieldSortInput = {
+    //     field: 'lastModifiedAt',
+    //     sortOrder: Sort.Desc,
+    // };
+    const tabs = [
+        {
+            key: 'joinsTab',
+            tab: 'Joins',
+        },
+        // {
+        //     key: 'pkFkTab',
+        //     tab: 'PK/FK',
+        // },


can we remove this commented out code? or do we still want this tab here?

Removed commented code.

chriscollins3456 · 2023-07-11T20:37:04Z

datahub-web-react/src/app/entity/shared/tabs/Dataset/Relationship/RelationTab.tsx

+        joinData = joins.search.searchResults.map((r) => r.entity as Join);
+    }
+
+    const joinPreview = (record: Join): JSX.Element => {


this RelationTab is getting pretty big and complicated. I think moving this joinPreview function with all of its rendering logic below out to its own component would be nice! or you can simply move this name logic into the existing JoinPreview component

Refactored.

chriscollins3456 · 2023-07-11T20:39:12Z

datahub-web-react/src/app/entity/shared/tabs/Dataset/Relationship/RelationTab.tsx

+                    )}
+                    {loadingJoin && (
+                        <div>
+                            Joins <LoadingOutlined />{' '}


minor: I think we usually just show <LoadingOutlined />

chriscollins3456 · 2023-07-11T20:44:50Z

datahub-web-react/src/app/entity/shared/tabs/Dataset/Relationship/RelationTab.tsx

+    const { data: table2Dataset } = useGetDatasetQuery({
+        variables: {
+            urn: selectDataset || 'urn:li:dataset:(urn:li:dataPlatform:snowflake,testData,DEV)',
+        },
+    });
+    const { data: table2Schema } = useGetDatasetSchemaQuery({
+        variables: {
+            urn: selectDataset || 'urn:li:dataset:(urn:li:dataPlatform:snowflake,testData,DEV)',
+        },
+    });


hmm we're hardcoding these urns as backup here? also i believe we'll be immediately firing these queries when we load this page. I think these should probably make these lazy queries to only fire once they select a dataset

also why do we need to call useGetDatasetQuery and useGetDatasetSchemaQuery? we're already getting the dataset when we search for it right? we could store that in state. I get needing to make a query to show the new dataset's schema.

in general we want to minimize the number of queries we make and only make queries that are (1) necessary and (2) when we need them (as opposed to on page load)

Both graphql queries removed and utilized lazy queries.

chriscollins3456 · 2023-07-11T20:55:16Z

datahub-web-react/src/app/entity/shared/components/styled/search/SearchSelect.tsx

-                    selectedEntities={selectedEntities}
-                />
-            </TabToolbar>
+            {!singleSelect && (


hmm hiding this bar based on singleSelect feels off. maybe we need another optional prop for this component saying that we should hide this select bar if it's true

chriscollins3456 · 2023-07-11T20:57:12Z

datahub-web-react/src/app/entity/shared/tabs/Join/JoinTab.tsx

+export const JoinTab = ({ properties }: { properties?: any }) => {
+    const { data: entityData } = useGetJoinQuery({ variables: { urn: properties } });


i'm confused - why is properties any type and then we pass that into the query as the urn parameter?

Thank you @chriscollins3456 for review and valuable feedback!
While we are working on other, for this one:

i'm confused - why is properties any type and then we pass that into the query as the urn parameter?

Here JoinTab is called from JoinEntity and tab type is 'EntityTab' with definition as below:
export type EntityTab = {
name: string;
component: React.FunctionComponent<{ properties?: any }>;
display?: {
visible: (GenericEntityProperties, T) => boolean; // Whether the tab is visible on the UI. Defaults to true.
enabled: (GenericEntityProperties, T) => boolean; // Whether the tab is enabled on the UI. Defaults to true.
};
properties?: any;
};
Component can support 'React.FunctionComponent<{ properties?: any }>;' this signature only.
This has properties to pass any argument to Tab and I wanted to get urn from it.
Please feel free to add any suggestion here. Thank you!

Addressed as suggested- utilized useEntityData from EntityContext.

chriscollins3456 · 2023-07-11T20:59:11Z

datahub-web-react/src/graphql/dataset.graphql

+    incoming: relationships(
+        input: { types: ["joinA", "joinB"], direction: INCOMING, start: 0, count: 100 }
+    ) {
+        ...fullRelationshipResults
+    }


so we will then be doing a relationships query here for all dataset queries - can we move this out and create a whole new query for getting joins associated with this dataset? also, aren't we making a query to get joins where a given dataset is either joinA or joinB?

We removed this fragment and we are fetching joins with graphql search query.

chriscollins3456 · 2023-07-11T21:00:27Z

datahub-web-react/src/graphql/search.graphql

+fragment searchResultFieldsWithRelationships on Entity {
+    urn
+    type
+    ... on Dataset {
+        name
+        origin
+        uri
+        platform {
+            ...platformFields
+        }
+        dataPlatformInstance {
+            ...dataPlatformInstanceFields
+        }
+        editableProperties {
+            name


why are we adding all of this stuff in our graphql files?

Removed entries which are not required.

chriscollins3456 · 2023-07-11T21:02:22Z

datahub-web-react/src/graphql/lineage.graphql

+            name
+            description
+        }
+        platform {
+            ...platformFields
+        }
+        ownership {


also confused why this lineage.graphql file is being updated? Joins aren't lineage, they are relationships, but we'll never get them in a lineage specific query

Vt pl join UI code review

…in aspect

Rtekal pl join UI er model relation

rtekal · 2024-03-19T12:42:57Z

Yep Chris. Took care of all your review comments.
Please let me know. Thanks.

chriscollins3456

thank you so much for this!! also thank you so much for being so helpful on working with me on these requested changes - you all are real troopers no doubt about it.

I have one tiny final change that I'd like to request just to update the casing in one field in the model, which I checked should require only updates to a couple other places.

However, i'm approving this to get CI going to make sure everything is green before we merge this in! that's the only real blocker now to make sure tests and linting are passed. I'll help you guys out and point out changes to make CI pass

Unfortunately I don't have write access to poorvi's repo so I can't push code to get CI green with you unless I get access. Either way I'm happy to help guide you on anything that we might need to do :)

chriscollins3456 · 2024-03-19T14:48:22Z

metadata-models/src/main/pegasus/com/linkedin/ermodelrelation/ERModelRelationshipProperties.pdl

+  /**
+   * ERModelRelationFieldMapping (in future we can make it an array)
+   */
+  relationshipfieldMappings: array [RelationshipFieldMapping]


one more tiny one! would you mind updating the casing of this model to be relationshipFieldMappings with a capital "F"? then i checked the code and there's just a couple of places that would have to be changed to reflect then then it's good!

chriscollins3456 · 2024-03-19T14:51:58Z

but it actually looks like you recently merged in master and the CI is looking solid! so really we should only have to worry about the casing request in that final comment and then we're merging this bad boy

I went ahead and merged in master one more time to make sure we're up to date

Rtekal pl join UI er model relation

rtekal · 2024-03-19T19:05:49Z

Sure Chris. Thank you. I went ahead and did the last review changes.
If you find issues with 'tests and linting' please let me know.
Tomorrow I will ask Poorvi to add you as collaborator, in case you commit any changes to accelerate things faster.

Thank you again for all your help and extreme patience with our contribution.

chriscollins3456

nice! i think we're good to merge :)

…ject#8325) Co-authored-by: Raj Tekal <rtekal@gmail.com> Co-authored-by: Raj Tekal <rtekal@users.noreply.github.com> Co-authored-by: Raj Tekal <varadaraj_tekal@optum.com> Co-authored-by: Chris Collins <chriscollins3456@gmail.com>

rtekal and others added 14 commits May 3, 2023 22:26

Initial version - Joins Schema

da18715

Joins slight model changes

bdf2bc6

Added resolvers

eeba4cb

Graphql change and extraneous code removed

89e8782

Merge branch 'master' into master

0c22848

Remove Snapshot and Aspect

35ea6ca

Merge branch 'master' into master

5ff65c5

Merge - Sync with master

03319c1

Join resolver updates

a819bfa

Additional changes to graphql files

0cd6323

join ui changes

f84a576

minor correction

badcb44

more updates to join ui

996bcce

Merge branch 'master' into pl-join-ui

0b9246a

github-actions bot added the product PR or Issue related to the DataHub UI/UX label Jun 28, 2023

vercel bot deployed to Preview June 28, 2023 16:05 View deployment

chriscollins3456 reviewed Jul 10, 2023

View reviewed changes

chriscollins3456 reviewed Jul 11, 2023

View reviewed changes

poorvi767 and others added 5 commits July 14, 2023 19:19

review comments incorporate for UI

6d123c1

Review changes 1

2550337

Merge branch 'pl-join-ui-review' into vt-pl-join-ui-codeReview

7d3680b

Merge pull request #1 from rtekal/vt-pl-join-ui-codeReview

1a4967b

Vt pl join UI code review

placed new entries at the end

6173ce4

anshbansal added the community-contribution PR or Issue raised by member(s) of DataHub Community label Jul 17, 2023

poorvi767 and others added 5 commits July 17, 2023 19:03

graphql fragments update

2c7e901

WIP- create and update join resolvers separate implementation

a3c8f0c

fetched unr from envt for update join

60ee9e8

CreatedBy is an entity

fca944c

create and update resolver refactoring done

8c1d018

rtekal added 10 commits March 14, 2024 09:52

Changed ermodelrelation to ermodelrelationship for entity name and ma…

1dc9631

…in aspect

Rename ermodelrelationId -> id

97817f3

ERModel relation -> relationship; code compiles

5ce698f

Few more name changes

36ace22

Add Cardinality based on relations

0690c2d

More Web files names changed to relationship

4e569d2

One more rename to relathipship

eda02f7

Some more renames to relationship

94220c5

Merge branch 'master' into rtekal-pl-join-ui-ERModelRelation

15b7334

Merge pull request #21 from rtekal/rtekal-pl-join-ui-ERModelRelation

6f4d1b1

Rtekal pl join UI er model relation

vercel bot deployed to Preview March 18, 2024 20:47 View deployment

rtekal added 2 commits March 18, 2024 17:33

Merge branch 'master' into rtekal-pl-join-ui-ERModelRelation

ce7d553

Merge pull request #22 from rtekal/rtekal-pl-join-ui-ERModelRelation

40da935

Rtekal pl join UI er model relation

vercel bot deployed to Preview March 18, 2024 22:18 View deployment

chriscollins3456 approved these changes Mar 19, 2024

View reviewed changes

Merge branch 'master' into pl-join-ui

004eac0

vercel bot deployed to Preview March 19, 2024 15:08 View deployment

rtekal added 6 commits March 19, 2024 11:49

Merge with main and some renames

9ebb047

few more renames

5b54652

Some more name changes

49d9e55

Merge pull request #23 from rtekal/rtekal-pl-join-ui-ERModelRelation

4d50ae2

Rtekal pl join UI er model relation

Merge branch 'master' into rtekal-pl-join-ui-ERModelRelation

79694d7

Merge pull request #24 from rtekal/rtekal-pl-join-ui-ERModelRelation

99ded17

Rtekal pl join UI er model relation

vercel bot deployed to Preview March 19, 2024 19:31 View deployment

chriscollins3456 approved these changes Mar 20, 2024

View reviewed changes

chriscollins3456 merged commit c1f6efa into datahub-project:master Mar 20, 2024
60 checks passed

		export const JoinTab = ({ properties }: { properties?: any }) => {
		const { data: entityData } = useGetJoinQuery({ variables: { urn: properties } });

feat(models) : Joins (Datasets) schema, resolvers and UI #8325

feat(models) : Joins (Datasets) schema, resolvers and UI #8325

Conversation

poorvi767 commented Jun 28, 2023

Checklist

poorvi767 commented Jun 28, 2023

chriscollins3456 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chriscollins3456 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

poorvi767 Jul 31, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rtekal commented Mar 19, 2024

chriscollins3456 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chriscollins3456 commented Mar 19, 2024 • edited Loading

rtekal commented Mar 19, 2024

chriscollins3456 left a comment

Choose a reason for hiding this comment

poorvi767 Jul 31, 2023 •

edited

Loading

chriscollins3456 commented Mar 19, 2024 •

edited

Loading