config: Stop server startup if an unrecognized option is found in a config file #9855

kolbe · 2019-03-21T22:31:44Z

What problem does this PR solve?

What is changed and how it works?

Previously, tidb-server would simply ignore invalid config options encountered in a config.toml file. This can cause big problems if a user puts an option in the wrong section of the config file, or if they mis-spell an option name. This could conceivably lead to data loss (if, for example, data is not written to the correct device), security issues (if listeners are not bound to the correct interface), or administration headaches (if someone puts an option in the wrong section and cannot figure out why it isn't taking effect).

This change uses the metaData returned by toml.DecodeFile to identify any items not decoded. A list of them is made and is returned as an error from Config.Load, which causes server startup to abort.

Check List

Tests

Unit test
Yes, check added to config/config_test.go to make sure that an unrecognized option in a config file throws the correct error.

Code changes

Has exported function/method change
No
Has exported variable/fields change
No
Has interface methods change
No
Has persistent data change
No

Side effects

Possible performance regression
No
Increased code complexity
No
Breaking backward compatibility
Yes, if someone is using a config file in product that has unrecognized options.

Related changes

Need to cherry-pick to the release branch
No
Need to update the documentation
Yes
Need to update the tidb-ansible repository
Maybe, if it has a strange config file with invalid options!
Need to be included in the release note
Yes

…-config

…olbe-undecoded-config

codecov · 2019-03-21T22:41:22Z

Codecov Report

Merging #9855 into master will decrease coverage by 0.3011%.
The diff coverage is 100%.

@@               Coverage Diff                @@
##             master      #9855        +/-   ##
================================================
- Coverage   77.5172%   77.2161%   -0.3011%     
================================================
  Files           403        405         +2     
  Lines         81827      81641       -186     
================================================
- Hits          63430      63040       -390     
- Misses        13689      13932       +243     
+ Partials       4708       4669        -39

morgo · 2019-03-21T23:06:17Z

For context: this is the mysql server behavior with parsing config files. If an item is discovered that is unknown, an error is returned. As Kolbe pointed out, the logic is:

This could conceivably lead to data loss (if, for example, data is not written to the correct device), security issues (if listeners are not bound to the correct interface), or administration headaches (if someone puts an option in the wrong section and cannot figure out why it isn't taking effect).

winkyao

mind compatibility

winkyao · 2019-03-25T02:24:23Z

@jackysp PTAL

jackysp · 2019-03-25T02:39:54Z

/run-all-tests

jackysp · 2019-03-25T03:08:20Z

It seems that there are some illegal configurations in the test, which causes TiDB to fail to start.

xiekeyi98 · 2019-03-25T08:51:25Z

/run-all-tests

xiekeyi98 · 2019-03-25T11:40:02Z

/run-all-tests tidb-test=pr/764

xiekeyi98 · 2019-03-25T11:46:05Z

/run-unit-tests tidb-test=pr/764

xiekeyi98 · 2019-03-25T11:52:19Z

/run-mybatis-tests tidb-test=pr/764

zz-jason · 2019-03-25T12:12:36Z

config/config.go

+		for _, item := range undecoded {
+			undecodedItems += fmt.Sprintf("  %s \n", item.String())
+		}
+		err = errors.Errorf("config file %s contained unknown configuration options:\n%s", confFile, undecodedItems)


error generated by errors.Errorf() is attached with a call stack, we don't need to wrap a errors.Trace() in line 391.

That's fine with me, but the trace is not my code (it was there before, so that a non-existent config file for example would be traced for some reason).

config/config.go

zz-jason · 2019-03-25T12:20:25Z

config/config.go

@@ -371,10 +372,22 @@ func GetGlobalConfig() *Config {

 // Load loads config options from a toml file.
 func (c *Config) Load(confFile string) error {


When upgrade from old version tidb to a newer version, some old config item maybe is not used in the newer version tidb. The rolling upgrade may meet error in this case, then the rolling upgrade failed.

How about only check the config items when -config-check is provided?

I don't like this suggestion. I think the behavior should be that tidb-server simply refuses to start if there are unknown configuration options, for the reasons I provided above.

A couple points:

Now that there's a --config-check option, it can be a part of the recommended procedure for a rolling upgrade, so that the only people running into the issue you describe are those not following the recommended procedure.

The benefit of a rolling upgrade is that you never have to take the entire cluster offline. When upgrading the first node, it will refuse to start with this error. That's a fine time to address the problem, apply the change to the config files, get that first node upgraded, and then proceed with the rolling upgrade.

@morgo, your thoughts?

I think because our own CI broke, we may need to ease this change in slowly. Here would be my suggestion:

Keep --config-check as is, it is independently very useful.

Change from refuse to start to print a WARN about each incorrect config item.

Add a new flag for --config-strict, which defaults to FALSE. We can document it that enabling it is a best practice, and we may change the default to TRUE in the future. This will result in errors for invalid options.

It is not bulletproof of course, because the default log level is INFO it is possible that the warnings may not be seen.

I think the very fact our own CI broke is evidence of the value of this behavior change. Even we weren't keeping track of whether our config files were well-formed!

Your suggestion above is difficult, because the logging system isn't set up yet when the config file is parsed. Either we have to keep around the list of undecoded items and check them later after logging is possible, or we have to pass some other string/structure/flag around, or we have to simply print something to stderr, or ...? @coocood Morgan mentioned that you may have some thoughts on this?

I dislike this. Adding this new flag means that people may start relying on it, and then we have to keep it around as a no-op if we later make that behavior the default.

I do agree with Morgan's first and second suggestions. Our users may use old version ansible to upgrade, that doesn't pre-check the config using tidb-server --config-check, we need to keep the cluster stable when rolling upgrade.

Alright, it sounds like there's a consensus. Can someone offer a suggestion of the best way to technically implement a "warning" before the logging system is set up?

@kolbe I will create a pull request to your branch later.

kolbe#1 @kolbe PTAL

@winkyao I added some comments on your PR. I re-wrote my own interpretation of how I see this working and pushed it to my branch of my fork, please take a look.

Basically I added a kind of unsettling amount of additional stuff to create a custom error in config.go, then use that in the case of failed config validation. That enables checking for the specific error type in main.go, and if configStrict is not enabled, it'll get the string from the warning and keep that until ater logging has been set up.

This is kind of a lot of extra work and adds some ugly things, but the whole idea behind strict config checking (and the --config-strict option) is that this is a temporary situation until we make --config-strict behavior the default (and hopefully only!) behavior of TiDB Server in the future. I think at that point we'd simply remove all this extra stuff with the custom error and strange handling in the server.

Thoughts?

xiekeyi98 · 2019-03-25T12:23:36Z

/run-all-tests tidb-test=pr/764

…-config

… defer a warning from loadConfig until after logging has been set up. Virtually all of this should be removed after configStrict is made the default behavior of TiDB Server!

tidb-server/main.go

winkyao · 2019-04-02T11:21:50Z

Ci failed by

config_test.go:71:
    c.Assert(conf.Load(configFile), IsNil)
... value *config.ErrConfigValidationFailed = &config.ErrConfigValidationFailed{err:"config file /go/src/github.com/pingcap/tidb/config/config.toml contained unknown configuration options: tikv-client.token-limit"} ("config file /go/src/github.com/pingcap/tidb/config/config.toml contained unknown configuration options: tikv-client.token-limit"

The test case is wrong, you should replace (https://github.com/pingcap/tidb/blob/master/config/config_test.go#L48)

_, err = f.WriteString(`[performance]
[tikv-client]
commit-timeout="41s"
max-batch-size=128

token-limit = -1
`)

with

_, err = f.WriteString(`
token-limit = 0
[performance]
[tikv-client]
commit-timeout="41s"
max-batch-size=128
`)

…-config

… Fixed test.

winkyao

LGTM, @zz-jason PTAL

winkyao · 2019-04-03T03:41:15Z

/run-all-tests tidb-test=pr/764

codecov-io · 2019-04-08T03:07:49Z

Codecov Report

Merging #9855 into master will decrease coverage by 0.5034%.
The diff coverage is 100%.

@@               Coverage Diff                @@
##             master      #9855        +/-   ##
================================================
- Coverage   78.0225%   77.5191%   -0.5035%     
================================================
  Files           404        403         -1     
  Lines         82016      81932        -84     
================================================
- Hits          63991      63513       -478     
- Misses        13324      13711       +387     
- Partials       4701       4708         +7

xiekeyi98 · 2019-04-08T03:08:12Z

/run-all-tests tidb-test=pr/764

xiekeyi98 · 2019-04-08T03:35:07Z

/run-common-test tidb-test=pr/764

xiekeyi98

LGTM

…onfig file (#9855) (#10096)

kolbe added 2 commits March 21, 2019 13:22

Stop server startup if unrecognized options are found in the config file

ff42683

Added test for unrecognized options in a config file throwing an error

d5c8f8a

kolbe requested review from shenli and IANTHEREAL March 21, 2019 22:31

kolbe added 3 commits March 21, 2019 15:31

Merge branch 'master' into kolbe-undecoded-config

3be74b0

Merge branch 'master' of github.com:pingcap/tidb into kolbe-undecoded…

7f526b0

…-config

Merge branch 'kolbe-undecoded-config' of github.com:kolbe/tidb into k…

b4207f7

…olbe-undecoded-config

morgo changed the title ~~Stop server startup if an unrecognized option is found in a config file~~ config: Stop server startup if an unrecognized option is found in a config file Mar 21, 2019

kolbe added 3 commits March 22, 2019 09:09

Merge branch 'kolbe-undecoded-config' into kolbe-config-check

46f9037

Added --config-check flag

3238d1e

Clarified behavior of new --config-check flag in --help output

6266fd8

kolbe changed the base branch from master to rc4 March 22, 2019 18:51

kolbe changed the base branch from rc4 to master March 22, 2019 18:52

winkyao added the status/DNM label Mar 23, 2019

winkyao reviewed Mar 23, 2019

View reviewed changes

winkyao added component/server and removed status/DNM labels Mar 25, 2019

Merge branch 'master' into kolbe-undecoded-config

344e3d7

Merge branch 'master' into kolbe-undecoded-config

1062aff

xiekeyi98 closed this Mar 25, 2019

xiekeyi98 reopened this Mar 25, 2019

zz-jason reviewed Mar 25, 2019

View reviewed changes

Merge branch 'master' into kolbe-undecoded-config

5de01e9

shenli added the status/all tests passed label Apr 1, 2019

kolbe added 2 commits April 1, 2019 21:00

Merge branch 'master' of github.com:pingcap/tidb into kolbe-undecoded…

f65f452

…-config

Added new error type and some kind of circuitous handling to save and…

eb2a90a

… defer a warning from loadConfig until after logging has been set up. Virtually all of this should be removed after configStrict is made the default behavior of TiDB Server!

winkyao reviewed Apr 2, 2019

View reviewed changes

tidb-server/main.go Show resolved Hide resolved

kolbe and others added 3 commits April 2, 2019 08:59

Merge branch 'master' of github.com:pingcap/tidb into kolbe-undecoded…

1d8d5ec

…-config

Added handling for case of --config-check=true --config-strict=false.…

bb0ec3d

… Fixed test.

Merge branch 'master' into kolbe-undecoded-config

573f82a

winkyao reviewed Apr 3, 2019

View reviewed changes

morgo added the status/LGT1 Indicates that a PR has LGTM 1. label Apr 3, 2019

Merge branch 'master' into kolbe-undecoded-config

2ec9439

xiekeyi98 approved these changes Apr 8, 2019

View reviewed changes

winkyao merged commit f514a32 into pingcap:master Apr 8, 2019

xiekeyi98 mentioned this pull request Apr 10, 2019

config: Stop server startup if an unrecognized option is found in a config file (#9855) #10096

Merged

ngaut pushed a commit that referenced this pull request Apr 10, 2019

config: Stop server startup if an unrecognized option is found in a c…

276699f

…onfig file (#9855) (#10096)

lichunzhu mentioned this pull request Jul 23, 2019

Return error when loading some undefined configs from local pingcap/tidb-binlog#687

Merged

kennytm mentioned this pull request Aug 4, 2019

*: deny unknown fields when deserializing config TOML tikv/tikv#5190

Merged

lichunzhu mentioned this pull request Aug 12, 2019

Return error when loading some undefined configs from local pingcap/tidb-binlog#708

Merged

morgo mentioned this pull request Mar 3, 2022

*: Remove deprecated streaming #32765

Merged

12 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

config: Stop server startup if an unrecognized option is found in a config file #9855

config: Stop server startup if an unrecognized option is found in a config file #9855

kolbe commented Mar 21, 2019

codecov bot commented Mar 21, 2019 •

edited

Loading

morgo commented Mar 21, 2019

winkyao left a comment •

edited

Loading

winkyao commented Mar 25, 2019

jackysp commented Mar 25, 2019

jackysp commented Mar 25, 2019

xiekeyi98 commented Mar 25, 2019

xiekeyi98 commented Mar 25, 2019

xiekeyi98 commented Mar 25, 2019

xiekeyi98 commented Mar 25, 2019

zz-jason Mar 25, 2019

kolbe Mar 25, 2019

zz-jason Mar 25, 2019

kolbe Mar 25, 2019

morgo Mar 25, 2019

kolbe Mar 25, 2019

winkyao Mar 26, 2019 •

edited

Loading

kolbe Mar 26, 2019

winkyao Apr 2, 2019

winkyao Apr 2, 2019

kolbe Apr 2, 2019

xiekeyi98 commented Mar 25, 2019

winkyao commented Apr 2, 2019 •

edited

Loading

winkyao left a comment

winkyao commented Apr 3, 2019

codecov-io commented Apr 8, 2019

xiekeyi98 commented Apr 8, 2019

xiekeyi98 commented Apr 8, 2019

xiekeyi98 left a comment

		@@ -371,10 +372,22 @@ func GetGlobalConfig() *Config {

		// Load loads config options from a toml file.
		func (c *Config) Load(confFile string) error {

config: Stop server startup if an unrecognized option is found in a config file #9855

config: Stop server startup if an unrecognized option is found in a config file #9855

Conversation

kolbe commented Mar 21, 2019

What problem does this PR solve?

What is changed and how it works?

Check List

codecov bot commented Mar 21, 2019 • edited Loading

Codecov Report

morgo commented Mar 21, 2019

winkyao left a comment • edited Loading

Choose a reason for hiding this comment

winkyao commented Mar 25, 2019

jackysp commented Mar 25, 2019

jackysp commented Mar 25, 2019

xiekeyi98 commented Mar 25, 2019

xiekeyi98 commented Mar 25, 2019

xiekeyi98 commented Mar 25, 2019

xiekeyi98 commented Mar 25, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

winkyao Mar 26, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xiekeyi98 commented Mar 25, 2019

winkyao commented Apr 2, 2019 • edited Loading

winkyao left a comment

Choose a reason for hiding this comment

winkyao commented Apr 3, 2019

codecov-io commented Apr 8, 2019

Codecov Report

xiekeyi98 commented Apr 8, 2019

xiekeyi98 commented Apr 8, 2019

xiekeyi98 left a comment

Choose a reason for hiding this comment

codecov bot commented Mar 21, 2019 •

edited

Loading

winkyao left a comment •

edited

Loading

winkyao Mar 26, 2019 •

edited

Loading

winkyao commented Apr 2, 2019 •

edited

Loading