Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Drop offsets when tokenizing with RegularParser #111

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

funkjedi
Copy link
Contributor

Testing using my sample string that was causing memory issues for me memory usage was reduced by 46% using refactor.

Super simple test, showing peak usage after processing (aka. memory_get_peak_usage()) minus peak prior to processing.

new code existing code
110MB 258MB

Also referenced in #110


Note on changes. Reversed order of cases in switch as when for example marker is matched any subsequent captures (ex, separator) won't be present in matches array.

@thunderer thunderer force-pushed the memory-usage-optimization branch 2 times, most recently from 5840350 to 4e3a622 Compare June 3, 2024 18:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants