Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Content Search Powered by Meilisearch #378

Open
qasimgulzar opened this issue Aug 15, 2024 · 8 comments
Open

Content Search Powered by Meilisearch #378

qasimgulzar opened this issue Aug 15, 2024 · 8 comments
Assignees

Comments

@qasimgulzar
Copy link

Abstract

The Open edX platform is enhancing its search functionality by implementing Meilisearch, a fast and memory-efficient search engine. This project aims to improve search performance, reduce resource usage, and provide a more developer-friendly experience.

Detailed Product Proposal

https://openedx.atlassian.net/wiki/spaces/OEPM/pages/4411031558/Proposal+Content+Search+Powered+by+Meilisearch

Context & Background (in brief, if a Product Proposal is linked above)

The Open edX platform currently implements many different forms of search. For
example, users can search for course content, library content, forum posts, and
more. Most of the search functionality in the core platform is powered by the
Elasticsearch search engine (though other functionality developed by 2U, such as
in edx-enterprise, is powered by Algolia).

Most uses of Elasticsearch in Open edX use
edx-search which provides a partial
abstraction over Elasticsearch. The edx-search library formerly used
django-haystack as an abstraction
layer across search engines, but "that was ripped out after the package was
abandoned upstream and it became an obstacle to upgrades and efficiently
utilizing Elasticsearch (the abstraction layer imposed significant limits)"
(thanks to Jeremy Bowman for this context). Due to these changes, the current
edx-search API is a mix of abstractions and direct usage of the Elasticsearch
API, which makes it confusing and difficult to work with. In addition, each
usage of edx-search has been implemented fairly differently. See
State of edx-search
for details (thanks to Andy Shultz).

Scope & Approach (in brief, if a Product Proposal is linked above)

The initiative to implement Meilisearch within the Open edX platform aims to enhance the search functionality by providing a more efficient, scalable, and user-friendly search experience. This includes the development of a new generalized search API capable of supporting multiple search engines, the integration of Meilisearch as a primary search engine, and the transition from the existing Elasticsearch implementation.

Value & Impact (in brief, if a Product Proposal is linked above)

Measurable Improvements to the Open edX Platform

Enhanced Search Performance:

Faster search response times will lead to a more efficient and satisfying user experience for course authors, instructors, and learners.
Reduced memory usage will make the platform more cost-effective and capable of handling larger datasets without significant resource strain.

Improved User Experience:

A more intuitive and responsive search functionality will make it easier for users to find relevant content quickly, enhancing their overall experience on the platform.
Direct search requests from users to Meilisearch, bypassing the need to route through Django, will reduce latency and improve real-time search performance.

Scalability and Flexibility:

The new generalized search API will provide the flexibility to integrate with multiple search engines, ensuring the platform remains adaptable to future technological advancements.
Strong multi-tenancy support will enable the platform to scale more effectively across multiple small Open edX sites, improving resource allocation and performance.

Operational Efficiency:

Offloading search processing to Meilisearch will reduce the load on the LMS and other IDAs, improving the overall system performance and stability.
Simplified maintenance and implementation processes will free up developer resources, allowing for more focus on enhancing other aspects of the platform.

Milestones and/or Epics

  1. integrate openedx-search-api into edx-platform
  2. Implemented LMS/CMS content classes to migrate index from elasticsearch to meilisearch
  3. Update learning MFE to implement existing courseware search feature with openedx-search-api

Named Release

Sumac

Timeline (in brief, if a Product Proposal is linked above)

We are trying to make it part of Sumac release of openedx.

Proposed By

Edly

Additional Info

Under this PR we have implemented a POC

Copy link

Thanks for your submission, @openedx/openedx-product-managers will review shortly.

@qasimgulzar
Copy link
Author

@jmakowski1123 is there any update regarding this proposal?

@jmakowski1123
Copy link

@jmakowski1123 is there any update regarding this proposal?

This thread contains the latest in terms of enabling via Tutor, and it would be helpful to channel any developer feedback that comes from that into the central forum thread. I'm also cc-ing @ormsbee who is going to be shepherding the implementation decisions for Sumac and Teak.

@ormsbee
Copy link

ormsbee commented Aug 29, 2024

@qasimgulzar: I think that moving forward with openedx-search-api makes sense, though that would also involve bringing it into the openedx GitHub org. I haven't looked over the code since the first PRs you linked in discussions. As for Meilisearch as the backend–it's going to be the backend we'll use for at least the Sumac launch, but I want to schedule some time to evaluate TypeSense as well, to see if it could better address @blarghmatey's concerns w.r.t. High Availability, as he explained here.

@qasimgulzar
Copy link
Author

qasimgulzar commented Sep 3, 2024

@ormsbee I would be happy to assist you. do we have any actionable so far?

@qasimgulzar
Copy link
Author

I can see the ticket is closed, but I believe there are still some pending actionable like moving openedx-search-api to openedx organisation as well as opening integration PR once repo will be move to openedx org.

@ormsbee ormsbee reopened this Sep 24, 2024
@ormsbee ormsbee self-assigned this Sep 24, 2024
@ormsbee
Copy link

ormsbee commented Sep 24, 2024

Assigning this to myself for more followup.

@regisb
Copy link

regisb commented Oct 15, 2024

I'm taking over this project, here's the corresponding PR: openedx/edx-platform#35650

@regisb regisb self-assigned this Oct 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Abandoned
Development

No branches or pull requests

4 participants