ci: create a nightly build workflow #3553

jordanrfrazier · 2024-08-26T18:34:09Z

This PR is the beginning of nightly builds. It adds the following:

Creates a daily scheduled job that tags the current codebase with a dev tag: X.Y.Z.devN
Runs a nightly_release workflow that
- Runs FE / BE unit tests
- Publishes nightly builds to pypi: langflow-nightly and langflow-base-nightly

Follow ups are noted in comments:

Creating appropriate docker images (may do this in this PR if time permits)
Notifications - slack/discord
General simplification of the workflows
Testing!

I've tested that I can download the latest nightly build and run langflow as normal. langflow-nightly depends on the version of langflow-base-nightly at the time of release.

github-actions · 2024-08-26T18:34:51Z

Pull Request Validation Report

This comment is automatically generated by Conventional PR

Whitelist Report

Whitelist	Active	Result
Pull request is a draft and should be ignored	✅	❌
Pull request is made by a whitelisted user and should be ignored	❌	❌
Pull request is submitted by a bot and should be ignored	✅	❌
Pull request is submitted by administrators and should be ignored	❌	❌

Result

Pull request does not satisfy any enabled whitelist criteria. Pull request will be validated.

Validation Report

Validation	Active	Result
All commits in this pull request has valid messages	❌	✅
Pull request does not introduce too many changes	❌	✅
Pull request has a valid title	✅	✅
Pull request has mentioned issues	❌	✅
Pull request has valid branch name	❌	✅
Pull request should have a non-empty body	✅	✅

Result

Pull request satisfies all enabled pull request rules.

_{Last Modified at 26 Aug 24 18:34 UTC}

aws-amplify-sa-east-1 · 2024-08-26T18:35:32Z

This pull request is automatically being deployed by Amplify Hosting (learn more).

Access this pull request here: https://pr-3553.dmtpw4p5recq1.amplifyapp.com

.github/workflows/nightly_build.yml

nicoloboschi · 2024-08-27T09:38:18Z

.github/workflows/nightly_build.yml

+      build_docker_main: false
+      nightly_tag: ${{ needs.create-nightly-tag.outputs.TAG }}
+
+  # slack-notification:


I wonder if we should push on Discord instead of Slack, since the community is there. This would encourage external contributiors to add integration tests when they add new components and make them as involved as the langflow core maintainers

Agreed. Will do this either in a follow up, or here if it's not reviewed in the next few days.

scripts/ci/pypi_nightly_tag.py

.github/workflows/nightly_build.yml

pyproject.toml

github-actions · 2024-08-27T20:42:33Z

Detected 54 changes to dependencies in Poetry lockfile

Added aenum (3.1.15)
Added jsonrpcclient (4.0.3)
Added magicattr (0.1.6)
Updated anthropic (0.34.1 -> 0.34.2)
Updated astra-assistants (2.1.0.12 -> 2.1.0.19)
Updated attrs (23.2.0 -> 24.2.0)
Updated bce-python-sdk (0.9.19 -> 0.9.21)
Updated certifi (2024.7.4 -> 2024.8.30)
Updated cffi (1.17.0 -> 1.17.1)
Updated cohere (5.8.1 -> 5.9.0)
Updated dspy-ai (2.4.13 -> 2.4.14)
Updated e2b (0.17.2a52 -> 0.17.1)
Updated executing (2.0.1 -> 2.1.0)
Updated fastavro (1.9.5 -> 1.9.6)
Updated google-api-python-client (2.143.0 -> 2.144.0)
Updated google-cloud-aiplatform (1.64.0 -> 1.65.0)
Updated google-crc32c (1.5.0 -> 1.6.0)
Updated gotrue (2.7.0 -> 2.8.0)
Updated graphql-core (3.2.3 -> 3.2.4)
Updated groq (0.10.0 -> 0.11.0)
Updated grpcio (1.63.0 -> 1.66.1)
Updated ipython (8.26.0 -> 8.27.0)
Updated langchain (0.2.15 -> 0.2.16)
Updated langchain-community (0.2.14 -> 0.2.16)
Updated langchain-core (0.2.36 -> 0.2.38)
Updated langchain-text-splitters (0.2.2 -> 0.2.4)
Updated langfuse (2.46.0 -> 2.46.3)
Updated langsmith (0.1.106 -> 0.1.114)
Updated litellm (1.44.8 -> 1.44.18)
Updated llama-cpp-python (0.2.89 -> 0.2.90)
Updated more-itertools (10.4.0 -> 10.5.0)
Updated nvidia-nvjitlink-cu12 (12.6.20 -> 12.6.68)
Updated onnxruntime (1.19.0 -> 1.19.2)
Updated openai (1.42.0 -> 1.43.0)
Updated optuna (3.6.1 -> 4.0.0)
Updated packaging (24.1 -> 23.2)
Updated posthog (3.6.0 -> 3.6.3)
Updated pydantic (2.9.0b1 -> 2.9.0)
Updated pydantic-core (2.23.0 -> 2.23.2)
Updated pymilvus (2.4.5 -> 2.4.6)
Updated pyright (1.1.378 -> 1.1.379)
Updated rapidfuzz (3.9.6 -> 3.9.7)
Updated ruff (0.6.2 -> 0.6.4)
Updated safetensors (0.4.4 -> 0.4.5)
Updated setuptools (74.0.0 -> 74.1.2)
Updated sqlalchemy (2.0.32 -> 2.0.34)
Updated supabase (2.7.3 -> 2.7.4)
Updated torch (2.4.0 -> 2.4.1)
Updated transformers (4.40.2 -> 4.44.2)
Updated tree-sitter-python (0.23.0 -> 0.23.2)
Updated types-redis (4.6.0.20240819 -> 4.6.0.20240903)
Updated types-requests (2.32.0.20240712 -> 2.32.0.20240905)
Updated types-setuptools (73.0.0.20240822 -> 74.0.0.20240831)
Updated yarl (1.9.4 -> 1.9.11)

(3 added, 0 removed, 51 updated, 426 not changed)

jordanrfrazier · 2024-08-30T19:05:34Z

scripts/ci/pypi_nightly_tag.py

+    #     + ".dev"
+    #     + "0"
+    #     + datetime.now(pytz.timezone("UTC")).strftime("%Y%m%d")
+    # )


Note this. The current versioning uses X.Y.Z.devN - this is a lot easier to work with when iterating, as you can just increment N, but I see the appeal of the date-based name once builds are stable here. We can push this for now, ensure it's working in the next few iterations, yank the .devN builds, then move to date-based.

I like the date-based as well and agree that we can change that later

jordanrfrazier · 2024-08-30T19:06:20Z

scripts/ci/update_lf_base_dependency.py

+    #   the langflow-base dependency in pyproject.toml _again_. This is redundant, but
+    #   necessary because the release workflow relies on that script to update the base
+    #   dependency.
+    pattern = re.compile(r'langflow-base = \{ path = "\./src/backend/base", develop = true \}')


This is worth noting. Didn't want to make changes to the release workflow in this PR since it's already a mess, but we can simplify some of this.

By using uv instead of poetry? hahahah

Just kidding.

jordanrfrazier · 2024-08-30T19:06:45Z

scripts/ci/update_pyproject_version.py

+        raise ValueError(f"Invalid build type: {build_type}")
+
+
+if __name__ == "__main__":


I should add tests for these scripts :)

src/backend/base/langflow/__main__.py

jordanrfrazier · 2024-08-30T19:09:13Z

src/backend/base/langflow/__main__.py

-            package_name = "Langflow Base"
-        except ImportError as e:
-            logger.exception(e)
-            raise e


Logic moved into version.py and simplified to handle the four build types now

src/backend/base/langflow/__main__.py

jordanrfrazier · 2024-08-30T19:17:39Z

src/backend/base/langflow/api/utils.py


+    # TODO: FRAZ - Figure out what I want the behavior to be for nightly builds


TODO: Based on the code, I suspect we are just logging a warning that will say:

Your version of langflow (<some nightly version>) is outdated. Please update to <latest stable>

Which, doesn't really make sense - we probably want it to say "there's a new nightly build to try" or even just ignore this since it's already known you're on a nightly build, but low priority either way I believe. Not a blocker for this PR.

I'd say we just ignore it

nicoloboschi

LGTM

dosubot bot added the size:L This PR changes 100-499 lines, ignoring generated files. label Aug 26, 2024

jordanrfrazier requested review from zzzming and nicoloboschi August 26, 2024 18:34

dosubot bot added the enhancement New feature or request label Aug 26, 2024

github-actions bot added build and removed enhancement New feature or request labels Aug 26, 2024

jordanrfrazier marked this pull request as draft August 26, 2024 18:46

github-actions bot added build and removed build labels Aug 26, 2024

jordanrfrazier changed the title ~~build: create a nightly build workflow~~ ci: create a nightly build workflow Aug 26, 2024

jordanrfrazier mentioned this pull request Aug 26, 2024

ci: add empty nightly workflow #3559

Merged

jordanrfrazier force-pushed the create-nightly-build branch from ac543dc to 36b4ae3 Compare August 26, 2024 21:48

nicoloboschi suggested changes Aug 27, 2024

View reviewed changes

jordanrfrazier force-pushed the create-nightly-build branch from 9ea0a26 to 484517d Compare August 27, 2024 21:31

jordanrfrazier commented Aug 30, 2024

View reviewed changes

jordanrfrazier requested a review from ogabrielluiz August 30, 2024 19:25

jordanrfrazier marked this pull request as ready for review August 30, 2024 19:31

dosubot bot added size:XXL This PR changes 1000+ lines, ignoring generated files. enhancement New feature or request and removed size:L This PR changes 100-499 lines, ignoring generated files. labels Aug 30, 2024

nicoloboschi approved these changes Sep 2, 2024

View reviewed changes

dosubot bot added the lgtm This PR has been approved by a maintainer label Sep 2, 2024

jordanrfrazier force-pushed the create-nightly-build branch from cba272d to 3bfcc73 Compare September 5, 2024 17:53

jordanrfrazier added 3 commits September 5, 2024 10:53

test poetry install

78e7c8a

Add nightly builds workflow

3566839

remove old comments and fix poetry

5d03c88

jordanrfrazier force-pushed the create-nightly-build branch from 3bfcc73 to 5d03c88 Compare September 5, 2024 18:10

remove old debug statement

c2c5c29

jordanrfrazier force-pushed the create-nightly-build branch from 7a5b818 to c2c5c29 Compare September 5, 2024 18:55

[autofix.ci] apply automated fixes

36a0359

jordanrfrazier enabled auto-merge (squash) September 5, 2024 19:00

jordanrfrazier merged commit 7b3e51f into main Sep 5, 2024
30 checks passed

jordanrfrazier deleted the create-nightly-build branch September 5, 2024 19:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: create a nightly build workflow #3553

ci: create a nightly build workflow #3553

jordanrfrazier commented Aug 26, 2024 •

edited

Loading

github-actions bot commented Aug 26, 2024

aws-amplify-sa-east-1 bot commented Aug 26, 2024

nicoloboschi Aug 27, 2024

jordanrfrazier Aug 30, 2024

github-actions bot commented Aug 27, 2024 •

edited

Loading

jordanrfrazier Aug 30, 2024

ogabrielluiz Aug 30, 2024

jordanrfrazier Aug 30, 2024

ogabrielluiz Aug 30, 2024

jordanrfrazier Aug 30, 2024

jordanrfrazier Aug 30, 2024

jordanrfrazier Aug 30, 2024

ogabrielluiz Aug 30, 2024

nicoloboschi left a comment

		raise ValueError(f"Invalid build type: {build_type}")


		if __name__ == "__main__":


		# TODO: FRAZ - Figure out what I want the behavior to be for nightly builds

ci: create a nightly build workflow #3553

ci: create a nightly build workflow #3553

Conversation

jordanrfrazier commented Aug 26, 2024 • edited Loading

github-actions bot commented Aug 26, 2024

Pull Request Validation Report

Whitelist Report

Validation Report

aws-amplify-sa-east-1 bot commented Aug 26, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Aug 27, 2024 • edited Loading

Detected 54 changes to dependencies in Poetry lockfile

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nicoloboschi left a comment

Choose a reason for hiding this comment

jordanrfrazier commented Aug 26, 2024 •

edited

Loading

github-actions bot commented Aug 27, 2024 •

edited

Loading