Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(api): adds migration for tracking vector store indexing status #830

Merged
merged 116 commits into from
Sep 5, 2024
Merged

feat(api): adds migration for tracking vector store indexing status #830

merged 116 commits into from
Sep 5, 2024

Conversation

CollectiveUnicorn
Copy link
Contributor

@CollectiveUnicorn CollectiveUnicorn commented Jul 24, 2024 •
edited
Loading

  • Enables Supabase realtime for the vector_store_file table
  • Adds new Python dependency to test supabase-realtime
  • Status is currently being set during indexing so listening to this table should provide status updates on the file indexing process.

leapfrogai/src/leapfrogai_api/backend/rag/index.py

Line 41 in 30b8ba2

class IndexingService:

@CollectiveUnicorn CollectiveUnicorn linked an issue Jul 24, 2024 that may be closed by this pull request
feat(api): vector store indexing status #788
Closed
@netlify Netlify
Copy link

netlify bot commented Jul 24, 2024 •
edited
Loading

✅ Deploy Preview for leapfrogai-docs canceled.

Name Link
🔨 Latest commit b30efd8
🔍 Latest deploy log https://app.netlify.com/sites/leapfrogai-docs/deploys/66d8b5c0e1a29f0008c40747

CollectiveUnicorn and others added 27 commits July 24, 2024 13:38
@CollectiveUnicorn
Removes enforcement of status
8504585
@CollectiveUnicorn
Simplifies sql query to focus on vector_store_file
7c87604
@CollectiveUnicorn
Pulls supabase-realtime image from official instead of bitnami
0f4125a
@CollectiveUnicorn
Reverts supabase realtime image replacement
53e44a5
@CollectiveUnicorn
Adds separate supabase realtime config for websocket/api
f652a16
@CollectiveUnicorn
Updates the default config with necessary vars
4b3fd69
@CollectiveUnicorn
Adds init container placeholder for seeding tables via curl
f6d4166
@CollectiveUnicorn
Moves configmap changes to extra configs
775339b
@CollectiveUnicorn
Renames variable for compatbility with supabase version
e6d5394
@CollectiveUnicorn
Ensures configmap is mounted in container
427ffce
@CollectiveUnicorn
Specify the variables manuallY
7ba8566
@CollectiveUnicorn
Fixes env var typo
62a2793
@CollectiveUnicorn
Adds script to seed the realtime instance
86f0b4b
@CollectiveUnicorn
Moves script somewhere with permissions
4d9bbd9
@CollectiveUnicorn
Moves script to values file
39cc4d5
@CollectiveUnicorn
Updates url and var for realtime init container
dca8c65
@CollectiveUnicorn
Replace initContainer with sidecars
c254ee3
@CollectiveUnicorn
Adds wait for realtime container
811cefb
@CollectiveUnicorn
Replaces os-shell with busybox
1ec9ad8
@CollectiveUnicorn
Replaces busybox container with a curl container
2829b41
@CollectiveUnicorn
Replaces nc wait with curl wait
4a30a4b
@CollectiveUnicorn
Try to replace how the realtimme seed is happening
ffa7ffa
@CollectiveUnicorn
Merge branch 'main' into fix-supabase-realtime
a8fe333
@CollectiveUnicorn
Replaces job with inline curl call
3677383
@CollectiveUnicorn
Replaces wget commands with curl
395454c
@CollectiveUnicorn
Replaces invalid health endpoint on realtime
f99eda0
@CollectiveUnicorn
Moves the curl command to one line
0327095
gphorvath
gphorvath reviewed Aug 28, 2024
View reviewed changes
tests/e2e/test_supabase.py Outdated Show resolved Hide resolved
tests/e2e/test_supabase.py Outdated Show resolved Hide resolved
@CollectiveUnicorn CollectiveUnicorn changed the base branch from main to background-tasks August 29, 2024 21:16
gphorvath
gphorvath previously approved these changes Sep 3, 2024
View reviewed changes
Copy link
Contributor

@gphorvath gphorvath left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me once you resolve the conflict with main

Base automatically changed from background-tasks to main September 3, 2024 21:13
@CollectiveUnicorn CollectiveUnicorn dismissed gphorvath’s stale review September 3, 2024 21:13

The base branch was changed.

jalling97
jalling97 approved these changes Sep 4, 2024
View reviewed changes
Copy link
Contributor

@jalling97 jalling97 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You may need to update this with main, if that dismisses the review I'll check it again 👍

@CollectiveUnicorn CollectiveUnicorn merged commit eee3ed7 into main Sep 5, 2024
24 checks passed
@CollectiveUnicorn CollectiveUnicorn deleted the 788-featapi-vector-store-indexing-status branch September 5, 2024 16:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gphorvath gphorvath gphorvath approved these changes

@jalling97 jalling97 jalling97 approved these changes

@justinthelaw justinthelaw justinthelaw approved these changes

@YrrepNoj YrrepNoj Awaiting requested review from YrrepNoj

@alekst23 alekst23 Awaiting requested review from alekst23

Assignees

@CollectiveUnicorn CollectiveUnicorn

Labels
enhancement New feature or request
Projects
None yet
Milestone
Current - RAG UX Enhancements | Model...
Development

Successfully merging this pull request may close these issues.

feat(api): vector store indexing status
7 participants
@CollectiveUnicorn @justinthelaw @gphorvath @YrrepNoj @alekst23 @jalling97 @andrewrisse