Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adds backtick for the quoted string token lexer. #2095

Merged
merged 4 commits into from
May 19, 2020
Merged

Adds backtick for the quoted string token lexer. #2095

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
bd2bba6
Fixes unit in stats request log. (#2093)
cyriltovena May 19, 2020
8e6b0a2
Add backtick for the quoted string token lexer.
cyriltovena May 19, 2020
3a81a17
Fixes newlines.
cyriltovena May 19, 2020
bbce678
Add back missing test.
cyriltovena May 19, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions docs/logql.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,9 @@ the labels passed to the log stream selector will affect the relative
performance of the query's execution. The filter expression is then used to do a
distributed `grep` over the aggregated logs from the matching log streams.

> To avoid escaping special characters you can use the `` ` ``(back-tick) instead of `"` when quoting strings.
For example `` `\w+` `` is the same as `"\\w+"`, this is specially useful when writing regular expression which can contains many backslash that require escaping.

### Log Stream Selector

The log stream selector determines which log streams should be included in your
Expand Down Expand Up @@ -51,6 +54,7 @@ Examples:

- `{name=~"mysql.+"}`
- `{name!~"mysql.+"}`
- `` {name!~`mysql-\d+`} ``

The same rules that apply for [Prometheus Label
Selectors](https://prometheus.io/docs/prometheus/latest/querying/basics/#instant-vector-selectors)
Expand All @@ -64,6 +68,7 @@ regex:

- `{job="mysql"} |= "error"`
- `{name="kafka"} |~ "tsdb-ops.*io:2003"`
- `` {name="cassandra"} |~ `error=\w+` ``
- `{instance=~"kafka-[23]",name="kafka"} != kafka.server:type=ReplicaManager`

In the previous examples, `|=`, `|~`, and `!=` act as **filter operators** and
Expand Down
2 changes: 1 addition & 1 deletion pkg/logql/lex.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ func (l *lexer) Lex(lval *exprSymType) int {
lval.str = l.TokenText()
return NUMBER

case scanner.String:
case scanner.String, scanner.RawString:
var err error
lval.str, err = strconv.Unquote(l.TokenText())
if err != nil {
Expand Down
2 changes: 2 additions & 0 deletions pkg/logql/lex_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@ func TestLex(t *testing.T) {
expected []int
}{
{`{foo="bar"}`, []int{OPEN_BRACE, IDENTIFIER, EQ, STRING, CLOSE_BRACE}},
{"{foo=\"bar\"} |~ `\\w+`", []int{OPEN_BRACE, IDENTIFIER, EQ, STRING, CLOSE_BRACE, PIPE_MATCH, STRING}},
{`{foo="bar"} |~ "\\w+"`, []int{OPEN_BRACE, IDENTIFIER, EQ, STRING, CLOSE_BRACE, PIPE_MATCH, STRING}},
{`{ foo = "bar" }`, []int{OPEN_BRACE, IDENTIFIER, EQ, STRING, CLOSE_BRACE}},
{`{ foo != "bar" }`, []int{OPEN_BRACE, IDENTIFIER, NEQ, STRING, CLOSE_BRACE}},
{`{ foo =~ "bar" }`, []int{OPEN_BRACE, IDENTIFIER, RE, STRING, CLOSE_BRACE}},
Expand Down
51 changes: 35 additions & 16 deletions pkg/logql/parser_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,25 @@ func TestParse(t *testing.T) {
exp Expr
err error
}{
{
// raw string
in: "count_over_time({foo=~`bar\\w+`}[12h] |~ `error\\`)",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this be:

"count_over_time({foo=~`bar\w+`}[12h] |~ `error\`)",

With only a single backlash because it's inside the new backticks?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, unless you have an alternative we can't escape backtick here, basically foo |~ error`` is interpreted as "foo |~" + identifier(error) + empty raw string . So I need to use " quoted query. and that requires escape. I call this go quote inception. Not sure I explained this correctly.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, ok, i was more just asking if this was a mistake or if it was a requirement of how to get the value in the test... no worries LGTM

exp: &rangeAggregationExpr{
operation: "count_over_time",
left: &logRange{
left: &filterExpr{
ty: labels.MatchRegexp,
match: "error\\",
left: &matchersExpr{
matchers: []*labels.Matcher{
mustNewMatcher(labels.MatchRegexp, "foo", "bar\\w+"),
},
},
},
interval: 12 * time.Hour,
},
},
},
{
// test [12h] before filter expr
in: `count_over_time({foo="bar"}[12h] |= "error")`,
Expand Down Expand Up @@ -658,10 +677,10 @@ func TestParse(t *testing.T) {
},
{
in: `
sum(count_over_time({foo="bar"}[5m])) by (foo) ^
sum(count_over_time({foo="bar"}[5m])) by (foo) /
sum(count_over_time({foo="bar"}[5m])) by (foo)
`,
sum(count_over_time({foo="bar"}[5m])) by (foo) ^
sum(count_over_time({foo="bar"}[5m])) by (foo) /
sum(count_over_time({foo="bar"}[5m])) by (foo)
`,
exp: mustNewBinOpExpr(
OpTypeDiv,
mustNewBinOpExpr(
Expand Down Expand Up @@ -720,10 +739,10 @@ func TestParse(t *testing.T) {
{
// operator precedence before left associativity
in: `
sum(count_over_time({foo="bar"}[5m])) by (foo) +
sum(count_over_time({foo="bar"}[5m])) by (foo) /
sum(count_over_time({foo="bar"}[5m])) by (foo)
`,
sum(count_over_time({foo="bar"}[5m])) by (foo) +
sum(count_over_time({foo="bar"}[5m])) by (foo) /
sum(count_over_time({foo="bar"}[5m])) by (foo)
`,
exp: mustNewBinOpExpr(
OpTypeAdd,
mustNewVectorAggregationExpr(newRangeAggregationExpr(
Expand Down Expand Up @@ -781,10 +800,10 @@ func TestParse(t *testing.T) {
},
{
in: `sum by (job) (
count_over_time({namespace="tns"} |= "level=error"[5m])
/
count_over_time({namespace="tns"}[5m])
)`,
count_over_time({namespace="tns"} |= "level=error"[5m])
/
count_over_time({namespace="tns"}[5m])
)`,
exp: mustNewVectorAggregationExpr(
mustNewBinOpExpr(OpTypeDiv,
newRangeAggregationExpr(
Expand Down Expand Up @@ -812,10 +831,10 @@ func TestParse(t *testing.T) {
},
{
in: `sum by (job) (
count_over_time({namespace="tns"} |= "level=error"[5m])
/
count_over_time({namespace="tns"}[5m])
) * 100`,
count_over_time({namespace="tns"} |= "level=error"[5m])
/
count_over_time({namespace="tns"}[5m])
) * 100`,
exp: mustNewBinOpExpr(OpTypeMul, mustNewVectorAggregationExpr(
mustNewBinOpExpr(OpTypeDiv,
newRangeAggregationExpr(
Expand Down