feat(ingest/dagster): Dagster source #10071
Conversation
github-actions
bot
added
ingestion
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| cache: "pip" | ||
| - name: Install dependencies | ||
| run: ./metadata-ingestion/scripts/install_deps.sh | ||
| - name: Install dagster package and test (extras ${{ matrix.extraPythonRequirement }}) |
There was a problem hiding this comment.
[actionlint] reported by reviewdog 🐶
property "extrapythonrequirement" is not defined in object type {python-version: number} [expression]
| - name: Install dependencies | ||
| run: ./metadata-ingestion/scripts/install_deps.sh | ||
| - name: Install dagster package and test (extras ${{ matrix.extraPythonRequirement }}) | ||
| run: ./gradlew -Pextra_pip_requirements='${{ matrix.extraPythonRequirement }}' :metadata-ingestion-modules:dagster-plugin:lint :metadata-ingestion-modules:dagster-plugin:testQuick |
There was a problem hiding this comment.
[actionlint] reported by reviewdog 🐶
property "extrapythonrequirement" is not defined in object type {python-version: number} [expression]
| if: always() | ||
| run: source metadata-ingestion-modules/dagster-plugin/venv/bin/activate && pip freeze | ||
| - uses: actions/upload-artifact@v3 | ||
| if: ${{ always() && matrix.python-version == '3.10' && matrix.extraPythonRequirement == 'dagster>=1.3.3' }} |
There was a problem hiding this comment.
[actionlint] reported by reviewdog 🐶
property "extrapythonrequirement" is not defined in object type {python-version: number} [expression]
| token: ${{ secrets.CODECOV_TOKEN }} | ||
| directory: . | ||
| fail_ci_if_error: false | ||
| flags: dagster-${{ matrix.python-version }}-${{ matrix.extraPythonRequirement }} |
There was a problem hiding this comment.
[actionlint] reported by reviewdog 🐶
property "extrapythonrequirement" is not defined in object type {python-version: number} [expression]
...ngestion-modules/dagster-plugin/src/datahub_dagster_plugin/example_jobs/advanced_ops_jobs.py
Outdated
Show resolved
Hide resolved
...ngestion-modules/dagster-plugin/src/datahub_dagster_plugin/example_jobs/advanced_ops_jobs.py
Outdated
Show resolved
Hide resolved
reverting path_spec change
There was a problem hiding this comment.
are we actually using tox?
There was a problem hiding this comment.
do the example jobs need to be packaged into the datahub_dagster_plugin code?
we did it for airflow because they autogenerated docs based on them, but I think a more standard dagster-plugin/examples directory would be better here
- Removing tox.ini
Co-authored-by: shubhamjagtap639 <shubham.jagtap@gslab.com>
Checklist