feat(ingestion): Adds more advanced configurations for runtime debugging #8998
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
ingestion
pedro93
force-pushed
the
ps-advanced-ingestion
branch
from
41f7852
to
154ed9b
Compare
datahub-web-react/src/app/ingest/source/IngestionSourceList.tsx
Outdated
Show resolved
Hide resolved
datahub-web-react/src/app/ingest/source/IngestionSourceList.tsx
Outdated
Show resolved
Hide resolved
datahub-web-react/src/app/ingest/source/IngestionSourceList.tsx
Outdated
Show resolved
Hide resolved
Comment on lines
+207
to
+239
| <Form.Item label={<Typography.Text strong>Extra Enviroment Variables</Typography.Text>}> | ||
| <Typography.Paragraph> | ||
| Advanced: Set extra environment variables to an ingestion execution | ||
| </Typography.Paragraph> | ||
| <Input | ||
| data-testid="extra-args-input" | ||
| placeholder='{"MY_CUSTOM_ENV": "my_custom_value2"}' | ||
| value={retrieveExtraEnvs()} | ||
| onChange={(event) => setExtraEnvs(event.target.value)} | ||
| /> | ||
| </Form.Item> | ||
| <Form.Item label={<Typography.Text strong>Extra DataHub plugins</Typography.Text>}> | ||
| <Typography.Paragraph> | ||
| Advanced: Set extra DataHub plugins for an ingestion execution | ||
| </Typography.Paragraph> | ||
| <Input | ||
| data-testid="extra-pip-plugin-input" | ||
| placeholder='["debug"]' | ||
| value={retrieveExtraDataHubPlugins()} | ||
| onChange={(event) => setExtraDataHubPlugins(event.target.value)} | ||
| /> | ||
| </Form.Item> | ||
| <Form.Item label={<Typography.Text strong>Extra Pip Libraries</Typography.Text>}> | ||
| <Typography.Paragraph> | ||
| Advanced: Add extra pip libraries for an ingestion execution | ||
| </Typography.Paragraph> | ||
| <Input | ||
| data-testid="extra-pip-reqs-input" | ||
| placeholder='["sqlparse==0.4.3"]' | ||
| value={retrieveExtraReqs()} | ||
| onChange={(event) => setExtraReqs(event.target.value)} | ||
| /> | ||
| </Form.Item> |
There was a problem hiding this comment.
so im trying to get better at enforcing small component because that keeps the frontend so much cleaner and easier to reason about. I'd move each of these inputs into their own new small file and component where in each of those the getter and setter logic can live. ie. retrieveExtraReqs and setExtraReqs can live in its own component where you render the input for Extra Pip Libraries
metadata-models/src/main/pegasus/com/linkedin/ingestion/DataHubIngestionSourceInfo.pdl
Show resolved
Hide resolved
maggiehays
added
the
hacktoberfest-accepted
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Checklist
Adds advanced configurations for specifying environment variables, DataHub plugins and pip packages at runtime for managed ingestion.
This feature is specific to managed ingestion as it uses logic in datahub actions.
Tested manually.
Here is a screenshot of how it looks: