enhancement: read grok aliases from file #194

itkovian · 2023-04-17T10:33:16Z

This PR aims to address (in part) #89, allowing GROK aliases to be defined in one or more files. I am not sure

Files are assumed to contain JSON, with a single outer Object.
Duplicate patterns are not handled
Needs proper tests.

(recreated from vectordotdev/vector#14914)

fuchsnj

Added a few comments, overall it seems to be reasonable though. There was a large refactoring recently that moved most of the files from lib/* to src/*. The merge conflicts you are seeing are probably from that.

fuchsnj · 2023-05-26T18:51:43Z

lib/compiler/src/function.rs

@@ -519,18 +520,22 @@ pub enum Error {

    #[error(r#"mutation of read-only value"#)]
    ReadOnlyMutation { context: String },
+
+    #[error(r#"invalid alias source"#)]
+    InvalidAliasSource { path: PathBuf },


function specific errors should not go here. I would suggest just re-using the existing InvalidArgument error above.

fuchsnj · 2023-05-26T18:55:44Z

lib/stdlib/src/parse_groks.rs

+                        expr,
+                    })?
+                    .try_bytes_utf8_lossy()
+                    .expect("filename not bytes")


This should not panic, since nothing is making sure the inputs are strings here. The argument type is ARRAY, but the elements can be any type. This should check that all arguments are strings, and return a InvalidArgument if it's not.

I realize you probably just copied this from aliases above, but that is also a bug (that I just opened).

OK. Will change.

fuchsnj · 2023-05-26T19:12:33Z

lib/stdlib/src/parse_groks.rs

+                    "_status": "%{POSINT:status}",
+                    "_message": "%{GREEDYDATA:message}"
+                }),
+                alias_sources: Value::Array(vec![]),


I would like to see a test with a least 1 real alias file loaded. If it's easier, you can use the more general VRL testing framework at lib/tests/tests/functions/... and include a file containing an alias.

…error type

itkovian · 2023-06-05T09:28:24Z

@fuchsnj I tackled the other alias thing as well, can you check if this is what you expect to happen?

fuchsnj · 2023-06-27T14:14:27Z

@fuchsnj I tackled the other alias thing as well, can you check if this is what you expect to happen?

Yes, this looks great, thanks!

fuchsnj · 2023-06-27T14:23:41Z

Looks good overall. Please add a line to CHANGELOG.md for these changes. It also looks like the wasm32 CI check is failing. Make sure imports or other functionality that was added that isn't supported on wasm goes in the non_wasm module in this file so the conditional compilation works correctly. Let me know if you need any help with this.

pront · 2023-08-02T19:32:00Z

Hello @itkovian, let us know if you have time to updated based on this feedback or if you want to hand this off. And thanks again for submitting this!

itkovian · 2023-08-28T11:12:43Z

I was on holidays, will check asap.

itkovian · 2023-09-04T13:58:31Z

Looks good overall. Please add a line to CHANGELOG.md for these changes. It also looks like the wasm32 CI check is failing. Make sure imports or other functionality that was added that isn't supported on wasm goes in the non_wasm module in this file so the conditional compilation works correctly. Let me know if you need any help with this.

I think I fixed the wasm thing, let me know if anything else is needed. Thx!

pront · 2023-09-06T20:13:56Z

Hi @itkovian, there are some test failures. You can iterate on these locally by running ./scripts/checks.sh from the repo root.

itkovian · 2023-09-19T16:30:57Z

Tests fail for the main branch as well?

The following warnings were emitted during compilation:

warning: error: unable to create target: 'No available targets are compatible with triple "wasm32-unknown-unknown"'
warning: 1 error generated.
warning: error: unable to create target: 'No available targets are compatible with triple "wasm32-unknown-unknown"'
warning: 1 error generated.

error: failed to run custom build command for `zstd-sys v2.0.8+zstd.1.5.5`

Caused by:
  process didn't exit successfully: `/Users/ageorges/.cargo/build-cache/debug/build/zstd-sys-c0f0ce56419d5513/build-script-build` (exit status: 1)
  --- stdout
  cargo:rerun-if-env-changed=ZSTD_SYS_USE_PKG_CONFIG
  cargo:rustc-cfg=feature="std"
  cargo:rerun-if-changed=wasm-shim/stdlib.h
  cargo:rerun-if-changed=wasm-shim/string.h
  TARGET = Some("wasm32-unknown-unknown")
  OPT_LEVEL = Some("0")
  HOST = Some("aarch64-apple-darwin")
  cargo:rerun-if-env-changed=CC_wasm32-unknown-unknown
  CC_wasm32-unknown-unknown = None
  cargo:rerun-if-env-changed=CC_wasm32_unknown_unknown
  CC_wasm32_unknown_unknown = None
  cargo:rerun-if-env-changed=TARGET_CC
  TARGET_CC = None
  cargo:rerun-if-env-changed=CC
  CC = None
  cargo:rerun-if-env-changed=CFLAGS_wasm32-unknown-unknown
  CFLAGS_wasm32-unknown-unknown = None
  cargo:rerun-if-env-changed=CFLAGS_wasm32_unknown_unknown
  CFLAGS_wasm32_unknown_unknown = None
  cargo:rerun-if-env-changed=TARGET_CFLAGS
  TARGET_CFLAGS = None
  cargo:rerun-if-env-changed=CFLAGS
  CFLAGS = None
  cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
  CRATE_CC_NO_DEFAULTS = None
  DEBUG = Some("true")
  cargo:rerun-if-env-changed=CC_wasm32-unknown-unknown
  CC_wasm32-unknown-unknown = None
  cargo:rerun-if-env-changed=CC_wasm32_unknown_unknown
  CC_wasm32_unknown_unknown = None
  cargo:rerun-if-env-changed=TARGET_CC
  TARGET_CC = None
  cargo:rerun-if-env-changed=CC
  CC = None
  cargo:rerun-if-env-changed=CFLAGS_wasm32-unknown-unknown
  CFLAGS_wasm32-unknown-unknown = None
  cargo:rerun-if-env-changed=CFLAGS_wasm32_unknown_unknown
  CFLAGS_wasm32_unknown_unknown = None
  cargo:rerun-if-env-changed=TARGET_CFLAGS
  TARGET_CFLAGS = None
  cargo:rerun-if-env-changed=CFLAGS
  CFLAGS = None
  cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
  CRATE_CC_NO_DEFAULTS = None
  cargo:rerun-if-env-changed=CC_wasm32-unknown-unknown
  CC_wasm32-unknown-unknown = None
  cargo:rerun-if-env-changed=CC_wasm32_unknown_unknown
  CC_wasm32_unknown_unknown = None
  cargo:rerun-if-env-changed=TARGET_CC
  TARGET_CC = None
  cargo:rerun-if-env-changed=CC
  CC = None
  cargo:rerun-if-env-changed=CFLAGS_wasm32-unknown-unknown
  CFLAGS_wasm32-unknown-unknown = None
  cargo:rerun-if-env-changed=CFLAGS_wasm32_unknown_unknown
  CFLAGS_wasm32_unknown_unknown = None
  cargo:rerun-if-env-changed=TARGET_CFLAGS
  TARGET_CFLAGS = None
  cargo:rerun-if-env-changed=CFLAGS
  CFLAGS = None
  cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
  CRATE_CC_NO_DEFAULTS = None
  cargo:rerun-if-env-changed=CC_wasm32-unknown-unknown
  CC_wasm32-unknown-unknown = None
  cargo:rerun-if-env-changed=CC_wasm32_unknown_unknown
  CC_wasm32_unknown_unknown = None
  cargo:rerun-if-env-changed=TARGET_CC
  TARGET_CC = None
  cargo:rerun-if-env-changed=CC
  CC = None
  cargo:rerun-if-env-changed=CFLAGS_wasm32-unknown-unknown
  CFLAGS_wasm32-unknown-unknown = None
  cargo:rerun-if-env-changed=CFLAGS_wasm32_unknown_unknown
  CFLAGS_wasm32_unknown_unknown = None
  cargo:rerun-if-env-changed=TARGET_CFLAGS
  TARGET_CFLAGS = None
  cargo:rerun-if-env-changed=CFLAGS
  CFLAGS = None
  cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
  CRATE_CC_NO_DEFAULTS = None
  running: "clang" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "--target=wasm32-unknown-unknown" "-I" "wasm-shim/" "-I" "zstd/lib/" "-I" "zstd/lib/common" "-fvisibility=hidden" 
"-DXXH_STATIC_ASSERT=0" "-DZSTD_LIB_DEPRECATED=0" "-DXXH_PRIVATE_API=" "-DZSTDLIB_VISIBILITY=" "-DZSTDERRORLIB_VISIBILITY=" "-o" "/Users/ageorges/.cargo/build-cache/wasm32-unknown-unknown/debug/build/zstd-sys-3ee678ded
c1a7237/out/zstd/lib/common/debug.o" "-c" "zstd/lib/common/debug.c"
  cargo:warning=error: unable to create target: 'No available targets are compatible with triple "wasm32-unknown-unknown"'
  cargo:warning=1 error generated.
  exit status: 1
  running: "clang" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "--target=wasm32-unknown-unknown" "-I" "wasm-shim/" "-I" "zstd/lib/" "-I" "zstd/lib/common" "-fvisibility=hidden" 
"-DXXH_STATIC_ASSERT=0" "-DZSTD_LIB_DEPRECATED=0" "-DXXH_PRIVATE_API=" "-DZSTDLIB_VISIBILITY=" "-DZSTDERRORLIB_VISIBILITY=" "-o" "/Users/ageorges/.cargo/build-cache/wasm32-unknown-unknown/debug/build/zstd-sys-3ee678ded
c1a7237/out/zstd/lib/common/entropy_common.o" "-c" "zstd/lib/common/entropy_common.c"
  cargo:warning=error: unable to create target: 'No available targets are compatible with triple "wasm32-unknown-unknown"'
  cargo:warning=1 error generated.
  exit status: 1

  --- stderr


  error occurred: Command "clang" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "--target=wasm32-unknown-unknown" "-I" "wasm-shim/" "-I" "zstd/lib/" "-I" "zstd/lib/common" "-fvisi
bility=hidden" "-DXXH_STATIC_ASSERT=0" "-DZSTD_LIB_DEPRECATED=0" "-DXXH_PRIVATE_API=" "-DZSTDLIB_VISIBILITY=" "-DZSTDERRORLIB_VISIBILITY=" "-o" "/Users/ageorges/.cargo/build-cache/wasm32-unknown-unknown/debug/build/zst
d-sys-3ee678dedc1a7237/out/zstd/lib/common/debug.o" "-c" "zstd/lib/common/debug.c" with args "clang" did not execute successfully (status code exit status: 1).


warning: build failed, waiting for other jobs to finish...

vrl on 🌱 main [$] is 📦 v0.6.0 via 🦀 v1.72.0 took 19m55s 
✗ rustup target add wasm32-unknown-unknown
info: component 'rust-std' for target 'wasm32-unknown-unknown' is up to date

pront · 2023-09-19T21:45:00Z

Tests fail for the main branch as well?

Hi @itkovian, checks are passing on the current HEAD (ad17a96).

The failure you posted above seems like an environment setup issue. Two commands to try:

rustup target add wasm32-unknown-unknown
cargo clean

If this is blocking you, you can just pick and choose what to run:

./scripts/checks.sh clippy tests vrl_tests

pront · 2023-10-10T19:14:20Z

Hi @itkovian, just following up on this PR. Are you still interested in driving this to completion?

Cargo.toml

This reverts commit 5b00a80.

itkovian · 2023-12-12T12:56:02Z

@pront Yes, please :) Wasm build/tests should be fixed.

pront · 2023-12-12T13:16:03Z

CHANGELOG.md

@@ -47,6 +47,7 @@
 - fixed type definitions for side-effects inside of queries (https://github.com/vectordotdev/vrl/pull/258)
 - replaced `Program::final_type_state` with `Program::final_type_info` to give access to the type definitions of both the target and program result (https://github.com/vectordotdev/vrl/pull/262)
 - added `from_unix_timestamp` vrl function (https://github.com/vectordotdev/vrl/pull/277)
+- added the `alias_sources` parameter for `parse_groks` to read sources from files


Please move this line under "unreleased".

pront

Thank you @itkovian. Two things remain before we can merge this PR:

We are missing a real test.
I left a question about security.

Feel free to ignore the nits.

src/stdlib/parse_groks.rs

pront

Thank you for this contribution @itkovian!

Left one comment and also some CI checks are failing (running cargo fmt should fix them).

lib/tests/tests/functions/test.json

jszwedko · 2024-03-27T14:47:03Z

Just realizing we never documented this addition in the Vector documentation. If you feel inspired, would you mind doing that @itkovian ? See vectordotdev/vector#19081 for an example.

itkovian · 2024-04-15T14:29:42Z

@jszwedko Sure.

parse_groks: optional parameter for alias sources

8c861cf

bruceg mentioned this pull request Apr 18, 2023

chore: RFC for defining Grok alias source files vectordotdev/vector#15979

Closed

fuchsnj reviewed May 26, 2023

View reviewed changes

Merge branch 'main' into grok-alias-sources

9e6d428

itkovian force-pushed the grok-alias-sources branch from 90923b7 to 9e6d428 Compare June 1, 2023 19:50

itkovian added 2 commits June 2, 2023 20:20

fix: use InvalidArgument instead of newly defined InvalidAliasSource …

ed01eda

…error type

fix: return InvalidArgument when data cannot be parsed as a string

023bf61

itkovian added 2 commits June 5, 2023 13:42

test: failure when parsing an alias with a non-string expression

5290e36

fix: clippy

1e72efd

itkovian mentioned this pull request Jun 5, 2023

parse_groks aliases are not type checked #261

Open

Merge branch 'main' into grok-alias-sources

0dce623

itkovian added 3 commits August 28, 2023 13:31

Merge branch 'main' into grok-alias-sources

df444ab

changelog: mention the new alias_sources keyword in the unreleased list

a2c20cc

fix: make sure the alias_sources test does not run on wasm builds

efa8ee4

pront requested a review from fuchsnj September 6, 2023 15:13

Merge branch 'main' into grok-alias-sources

1d23d15

itkovian added 3 commits October 19, 2023 11:45

fix: pass the state when resolving constants

d62d939

Merge branch 'main' into grok-alias-sources

326ec36

fix: version for dns-lookup

5b00a80

pront reviewed Oct 19, 2023

View reviewed changes

Cargo.toml Outdated Show resolved Hide resolved

itkovian added 2 commits November 27, 2023 10:35

Revert "fix: version for dns-lookup"

f9a80ef

This reverts commit 5b00a80.

fix: do not import things that have no wasm support

3aa540b

pront reviewed Dec 12, 2023

View reviewed changes

itkovian added 2 commits December 13, 2023 10:34

fix: move alias_sources to unreleased section in the changelog

e4ec754

Merge branch 'main' into grok-alias-sources

2b2c26c

pront requested changes Dec 13, 2023

View reviewed changes

src/stdlib/parse_groks.rs Outdated Show resolved Hide resolved

src/stdlib/parse_groks.rs Outdated Show resolved Hide resolved

src/stdlib/parse_groks.rs Outdated Show resolved Hide resolved

src/stdlib/parse_groks.rs Show resolved Hide resolved

src/stdlib/parse_groks.rs Show resolved Hide resolved

itkovian added 2 commits December 19, 2023 15:55

fix: remove extra variable and clone whenever needed

66e0944

test: add alias_sources test reading from file.

63d305d

pront approved these changes Dec 20, 2023

View reviewed changes

lib/tests/tests/functions/test.json Outdated Show resolved Hide resolved

itkovian added 2 commits December 21, 2023 10:21

fix: rename groks alias source file to something more descriptive

fa35ded

fmt: fix style

18e6a67

itkovian mentioned this pull request Dec 22, 2023

Add a safe mode that disable functions who have access to more than event data #602

Open

pront enabled auto-merge December 23, 2023 08:57

pront added this pull request to the merge queue Dec 23, 2023

Merged via the queue into vectordotdev:main with commit a1279c8 Dec 23, 2023
9 checks passed

jszwedko mentioned this pull request May 20, 2024

feat(stdlib): add parse_etld function #669

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

enhancement: read grok aliases from file #194

enhancement: read grok aliases from file #194

itkovian commented Apr 17, 2023

fuchsnj left a comment

fuchsnj May 26, 2023

fuchsnj May 26, 2023

fuchsnj May 26, 2023

itkovian May 31, 2023

fuchsnj May 26, 2023

itkovian commented Jun 5, 2023

fuchsnj commented Jun 27, 2023

fuchsnj commented Jun 27, 2023

pront commented Aug 2, 2023

itkovian commented Aug 28, 2023

itkovian commented Sep 4, 2023 •

edited

Loading

pront commented Sep 6, 2023

itkovian commented Sep 19, 2023

pront commented Sep 19, 2023 •

edited

Loading

pront commented Oct 10, 2023

itkovian commented Dec 12, 2023

pront Dec 12, 2023

pront left a comment

pront left a comment

jszwedko commented Mar 27, 2024

itkovian commented Apr 15, 2024

enhancement: read grok aliases from file #194

enhancement: read grok aliases from file #194

Conversation

itkovian commented Apr 17, 2023

fuchsnj left a comment

Choose a reason for hiding this comment

fuchsnj May 26, 2023

Choose a reason for hiding this comment

fuchsnj May 26, 2023

Choose a reason for hiding this comment

fuchsnj May 26, 2023

Choose a reason for hiding this comment

itkovian May 31, 2023

Choose a reason for hiding this comment

fuchsnj May 26, 2023

Choose a reason for hiding this comment

itkovian commented Jun 5, 2023

fuchsnj commented Jun 27, 2023

fuchsnj commented Jun 27, 2023

pront commented Aug 2, 2023

itkovian commented Aug 28, 2023

itkovian commented Sep 4, 2023 • edited Loading

pront commented Sep 6, 2023

itkovian commented Sep 19, 2023

pront commented Sep 19, 2023 • edited Loading

pront commented Oct 10, 2023

itkovian commented Dec 12, 2023

pront Dec 12, 2023

Choose a reason for hiding this comment

pront left a comment

Choose a reason for hiding this comment

pront left a comment

Choose a reason for hiding this comment

jszwedko commented Mar 27, 2024

itkovian commented Apr 15, 2024

itkovian commented Sep 4, 2023 •

edited

Loading

pront commented Sep 19, 2023 •

edited

Loading