Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

buffer: add buffer.isUtf8 for utf8 validation #45947

Merged
merged 6 commits into from
Dec 25, 2022

Conversation

anonrig
Copy link
Member

@anonrig anonrig commented Dec 22, 2022

With the introduction of simdutf, we can finally have a fast way of validating utf-8 on the userland. This pull request introduces buffer.isUtf8.

This was initially requested by @KhafraDev while implementing undici websockets.

@nodejs-github-bot nodejs-github-bot added buffer Issues and PRs related to the buffer subsystem. c++ Issues and PRs that require attention from people who are familiar with C++. needs-ci PRs that need a full CI run. labels Dec 22, 2022
@anonrig anonrig mentioned this pull request Dec 22, 2022
4 tasks
@anonrig
Copy link
Member Author

anonrig commented Dec 22, 2022

CC @nodejs/buffer @nodejs/performance @nodejs/undici

lib/buffer.js Outdated Show resolved Hide resolved
src/node_buffer.cc Outdated Show resolved Hide resolved
lib/buffer.js Outdated Show resolved Hide resolved
@targos
Copy link
Member

targos commented Dec 23, 2022

Can you add a test with a detached array buffer ?

doc/api/buffer.md Outdated Show resolved Hide resolved
lib/buffer.js Outdated Show resolved Hide resolved
Copy link
Member

@mcollina mcollina left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

src/node_buffer.cc Outdated Show resolved Hide resolved
test/parallel/test-buffer-isutf8.js Outdated Show resolved Hide resolved
@anonrig anonrig requested review from KhafraDev, RafaelGSS, targos, lpinca and bnoordhuis and removed request for KhafraDev December 23, 2022 17:34
@anonrig anonrig added request-ci Add this label to start a Jenkins CI on a PR. author ready PRs that have at least one approval, no pending requests for changes, and a CI started. notable-change PRs with changes that should be highlighted in changelogs. labels Dec 23, 2022
test/parallel/test-buffer-isutf8.js Outdated Show resolved Hide resolved
test/parallel/test-buffer-isutf8.js Outdated Show resolved Hide resolved
test/parallel/test-buffer-isutf8.js Outdated Show resolved Hide resolved
@github-actions github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Dec 23, 2022
targos pushed a commit that referenced this pull request Jan 1, 2023
PR-URL: #45947
Reviewed-By: Robert Nagy <ronagy@icloud.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Rafael Gonzaga <rafael.nunu@hotmail.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
RafaelGSS added a commit that referenced this pull request Jan 2, 2023
Notable changes:

buffer:
  * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947
deps:
  * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803
  * add simdutf dependency (Yagiz Nizipli) #45803
http:
  * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778
net
  * add autoSelectFamily global getter and setter (Paolo Insogna) #45777
os:
  * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895
util:
  * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803

PR-URL: TBD
@RafaelGSS RafaelGSS mentioned this pull request Jan 2, 2023
RafaelGSS added a commit that referenced this pull request Jan 2, 2023
Notable changes:

buffer:
  * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947
deps:
  * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803
  * add simdutf dependency (Yagiz Nizipli) #45803
http:
  * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778
net
  * add autoSelectFamily global getter and setter (Paolo Insogna) #45777
os:
  * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895
util:
  * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803

PR-URL: #46061
RafaelGSS added a commit that referenced this pull request Jan 2, 2023
Notable changes:

buffer:
  * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947
deps:
  * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803
  * add simdutf dependency (Yagiz Nizipli) #45803
http:
  * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778
net
  * add autoSelectFamily global getter and setter (Paolo Insogna) #45777
os:
  * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895
util:
  * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803

PR-URL: #46061
RafaelGSS added a commit that referenced this pull request Jan 2, 2023
Notable changes:

buffer:
  * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947
deps:
  * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803
  * add simdutf dependency (Yagiz Nizipli) #45803
http:
  * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778
net
  * add autoSelectFamily global getter and setter (Paolo Insogna) #45777
os:
  * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895
util:
  * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803

PR-URL: #46061
RafaelGSS added a commit that referenced this pull request Jan 2, 2023
Notable changes:

buffer:
  * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947
deps:
  * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803
  * add simdutf dependency (Yagiz Nizipli) #45803
http:
  * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778
net
  * add autoSelectFamily global getter and setter (Paolo Insogna) #45777
os:
  * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895
util:
  * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803

PR-URL: #46061
RafaelGSS added a commit that referenced this pull request Jan 3, 2023
Notable changes:

buffer:
  * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947
deps:
  * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803
  * add simdutf dependency (Yagiz Nizipli) #45803
http:
  * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778
net
  * add autoSelectFamily global getter and setter (Paolo Insogna) #45777
os:
  * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895
util:
  * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803

PR-URL: #46061
RafaelGSS pushed a commit that referenced this pull request Jan 4, 2023
PR-URL: #45947
Reviewed-By: Robert Nagy <ronagy@icloud.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Rafael Gonzaga <rafael.nunu@hotmail.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
RafaelGSS added a commit that referenced this pull request Jan 4, 2023
Notable changes:

buffer:
  * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947
deps:
  * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803
  * add simdutf dependency (Yagiz Nizipli) #45803
http:
  * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778
net
  * add autoSelectFamily global getter and setter (Paolo Insogna) #45777
os:
  * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895
util:
  * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803

PR-URL: #46061
RafaelGSS added a commit that referenced this pull request Jan 4, 2023
Notable changes:

buffer:
  * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947
deps:
  * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803
  * add simdutf dependency (Yagiz Nizipli) #45803
http:
  * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778
net
  * add autoSelectFamily global getter and setter (Paolo Insogna) #45777
os:
  * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895
util:
  * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803

PR-URL: #46061
RafaelGSS added a commit that referenced this pull request Jan 4, 2023
Notable changes:

buffer:
  * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947
http:
  * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778
net
  * add autoSelectFamily global getter and setter (Paolo Insogna) #45777
os:
  * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895
util:
  * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803

PR-URL: #46061
RafaelGSS added a commit that referenced this pull request Jan 4, 2023
Notable changes:

buffer:
  * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947
http:
  * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778
net
  * add autoSelectFamily global getter and setter (Paolo Insogna) #45777
os:
  * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895
util:
  * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803

PR-URL: #46061
RafaelGSS pushed a commit that referenced this pull request Jan 5, 2023
PR-URL: #45947
Reviewed-By: Robert Nagy <ronagy@icloud.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Rafael Gonzaga <rafael.nunu@hotmail.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
RafaelGSS added a commit that referenced this pull request Jan 5, 2023
Notable changes:

buffer:
  * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947
http:
  * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778
net
  * add autoSelectFamily global getter and setter (Paolo Insogna) #45777
os:
  * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895
util:
  * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803

PR-URL: #46061
RafaelGSS added a commit that referenced this pull request Jan 5, 2023
Notable changes:

buffer:
  * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947
http:
  * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778
net
  * add autoSelectFamily global getter and setter (Paolo Insogna) #45777
os:
  * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895
util:
  * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803

PR-URL: #46061
RafaelGSS added a commit that referenced this pull request Jan 6, 2023
Notable changes:

buffer:
  * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947
http:
  * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778
net
  * add autoSelectFamily global getter and setter (Paolo Insogna) #45777
os:
  * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895
util:
  * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803

PR-URL: #46061
Copy link

@bentley1985 bentley1985 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks

juanarbol pushed a commit that referenced this pull request Jan 26, 2023
PR-URL: #45947
Reviewed-By: Robert Nagy <ronagy@icloud.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Rafael Gonzaga <rafael.nunu@hotmail.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
juanarbol added a commit that referenced this pull request Jan 28, 2023
Notable changes:

* buffer
  * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947
* deps:
  * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803
  * add simdutf dependency (Yagiz Nizipli) #45803
  * upgrade npm to 9.1.3 (npm team) #45693
* util:
  * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803

PR-URL: TBD
@juanarbol juanarbol mentioned this pull request Jan 28, 2023
juanarbol added a commit that referenced this pull request Jan 28, 2023
Notable changes:

* buffer
  * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947
* deps:
  * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803
  * add simdutf dependency (Yagiz Nizipli) #45803
  * upgrade npm to 9.1.3 (npm team) #45693
* util:
  * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803

PR-URL: #46396
juanarbol added a commit that referenced this pull request Jan 28, 2023
Notable changes:

* buffer
  * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947
* deps:
  * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803
  * add simdutf dependency (Yagiz Nizipli) #45803
  * upgrade npm to 9.1.3 (npm team) #45693
* util:
  * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803

PR-URL: #46396
juanarbol added a commit that referenced this pull request Jan 30, 2023
Notable changes:

* buffer
  * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947
* deps:
  * add simdutf dependency (Yagiz Nizipli) #45803
  * upgrade npm to 9.1.3 (npm team) #45693
* util:
  * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803

PR-URL: #46396
juanarbol pushed a commit that referenced this pull request Jan 31, 2023
PR-URL: #45947
Reviewed-By: Robert Nagy <ronagy@icloud.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Rafael Gonzaga <rafael.nunu@hotmail.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
juanarbol added a commit that referenced this pull request Jan 31, 2023
Notable changes:

* buffer
  * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947
* deps:
  * add simdutf dependency (Yagiz Nizipli) #45803
  * upgrade npm to 9.1.3 (npm team) #45693
* util:
  * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803

PR-URL: #46396
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
author ready PRs that have at least one approval, no pending requests for changes, and a CI started. buffer Issues and PRs related to the buffer subsystem. c++ Issues and PRs that require attention from people who are familiar with C++. commit-queue-squash Add this label to instruct the Commit Queue to squash all the PR commits into the first one. needs-ci PRs that need a full CI run. notable-change PRs with changes that should be highlighted in changelogs. semver-minor PRs that contain new features and should be released in the next minor version.
Projects
None yet
Development

Successfully merging this pull request may close these issues.