variable as value for grok pattern

[camilisette]

I have this VRL where the grok pattern for the log is based on the value of .fields.application. However, I am getting an error when using a variable for the pattern arguement.

Log sample:

{
  "message": "INFO - Application started successfully.",
  "fields": {
    "application": "app2"
  }
}

VRL:

dictionary = [
{ "parsing_type": "grok", "application": "app1", "pattern": "%{DATA:first_field},%{DATA:second_field},%{DATA:third_field}"},
{ "parsing_type": "grok", "application": "app2", "pattern": "%{LOGLEVEL:loglevel} - %{GREEDYDATA:message}"},
{ "parsing_type": "grok", "application": "app3", "pattern": "%{IPORHOST:ip}:%{NUMBER:port}"}
]

filter = filter(dictionary) -> |_index, value| {
    value.application == .fields.application
}

if length(filter) > 0 {
    config = filter[0]

    if config.parsing_type == "grok" {
        .log,err = parse_grok(.message, config.pattern)
    }

}

The error that I'm getting:

2023-09-18 18:06:55 15 │         .log,err = parse_grok(.message, config.pattern)
2023-09-18 18:06:55    │                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-09-18 18:06:55    │                    │
2023-09-18 18:06:55    │                    unexpected expression for argument "pattern"
2023-09-18 18:06:55    │                    expected: literal
2023-09-18 18:06:55    │                    received: query

Tried different things with no luck. Any suggestions on how I can work around this?

[jszwedko]

Hi @camilisette !

At this time, parse_grok requires a literal since compiling grok expressions at runtime is expensive. In your case the compiler could grow to be smart enough to realize that the variable will map to a literal that can be precompiled, but it isn't currently. Instead I think you'll need to use if/else statements rather than a dictionary lookup. I think #151 is tracking improvements to the compiler that might enable the dictionary lookup you have.

[camilisette]

Just posting an update after trying this again with v0.33.0 of Vector which I believe should have #151 part of it. It still doesn't work when you have the pattern as part of a dictionary.