Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Prepro rework #154

Merged
merged 209 commits into from
Sep 29, 2023
Merged

Prepro rework #154

merged 209 commits into from
Sep 29, 2023

Conversation

floschne
Copy link
Member

This PR includes major changes!

  • Complete rework of the preprocessing pipeline and celery worker
  • We now have one worker for general background jobs such as export, preprocessing, and crawling.
  • We do not load any ML model in the backend API worker or bg worker but call the ray worker which uses ray serve to automatically load and scale the models
  • We are now using weaviate as vector db for simsearch and got rid of the FAISS Index Service
  • many minor changes

@floschne
fixed backend bug that lead to errors when a project contains no docu…
67e9682 
…ments
@floschne
fixed backend bug that wrongly excluded password serialization from p…
fb73b1f 
…ydantic dtos
@floschne
black formatting
fa6c600
@floschne
minor changes and formatting
f079f95
@floschne
implemented crud methods for preprocessing jobs in redis service
4aa47f5
@floschne
fixed optional bug for project id parameter to get all crawler jobs i…
a537eec 
…n redis service
@floschne
fixed typo in var name for crawler job in redis service
280287b
@floschne
fixed copy paste typo in crawler job
b1d81dd
@floschne
implemented endpoints for prepro jobs
7549be4
@floschne
added exception handler for files that are already in the backend repo
7a221e1
@floschne
fixed import order in backend api
15e113e
@floschne
changed payload dict to list in dto
12b0a4f
@floschne
creating and returning a preprocessing job when importing documents t…
8fefb77 
…o a project
@floschne
implemented first sketch of prepro service
2f31f6c
@floschne
using prepro service in project endpoint
7053af7
@floschne
added project id to prepro job payload and formatted code with black
641ad52
@floschne
added status to prepro payload and added init status
63203d4
@floschne
implemented method to get an url for a file in the repo
7573d14
@floschne
removed unnecessary exposed ports and tidied up the .env files
d2fc913
@floschne
fixed prepro endpoints
7edb828
@floschne
implemented method to update single payloads in a PPJ
e6e1fe6
@floschne
excluding none values when updating PPJs
b3fa075
@floschne
isort and black
a5e8252
@floschne
renamed prepro payload file path to file name
b098694
@floschne
minor commit to improve log
1230a31
@floschne
flushing redis when resetting data in startup
982582a
@floschne
renamed file path to file name in prepro pipeline functions
4fbe35b
@floschne
fixed filename type to str
5e83902
@floschne
removed old code to import uploaded documents
78f5b71
@floschne
updated the interface of prepro methods to accept prepro job payloads
01ca04b
floschne and others added 28 commits September 27, 2023 21:31
@floschne
removed old docprepro code
9736466
@floschne
implemented ORM models for preprojob
b713208
@floschne
implemented CRUDs for ORM models for preprojob
7910d0f
@floschne
implemented and updated DTOs for preprojob
884cfd2
@floschne
added prepro ORMs to SQLService
78d0ae3
@floschne
updated endpoint to use sqls service for prepro
4a5cf67
@floschne
replaced redis preprojob code with sql
b273e9c
@floschne
pool size parameters for sqlservice
08b70f6
@floschne
re-added missing ffmpeg dep to bg jobs worker
0d022e9
@floschne
added prepro pipeline num worker parameters in config
b691e28
@floschne
improved entrypoint script for bg jobs worker and fixed issues for lo…
6a2299e 
…ng running celery worker jobs
@floschne
updated package json lock
52d7ce6
@floschne
updated frontend code to comply to new backend api
66c785e
@floschne
removed unnecessary call to receive sdoc id from prepro jobs
61a36fe
@floschne
added missing preprojob payload model in frontend
b4c03d9
@floschne
read methods in crud ppj and ppj payload
7666bc8
@floschne
new prepro status route
b8918b9
@floschne
updated frontend code to comply to new backend api
6225908
@floschne
added missing weaviate env vars and volume
bf77173
@floschne
added cupy to ray deps
0fb9bdb
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
2fc2d69 
for more information, see https://pre-commit.ci
@floschne
ray test config file
ad5827f
@floschne
updated github test actions
ede8059
@floschne
added free diskspace action to gh test actions
502e4c4
@floschne
updated github test actions
2c884b1
@floschne
updated github test actions
fc6117d
@floschne
updated github test actions
523d9cb
@floschne
added req file for tools
8ef72d4
@floschne floschne merged commit aae93c9 into mwp_v1 Sep 29, 2023
5 of 7 checks passed
@floschne floschne deleted the prepro-rework branch November 18, 2023 06:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@floschne @Alienmaster