Lexer chokes on certain kinds of whitespace

[catern]

The presence in a Rust source file of unusual but useful kinds of whitespace, such as ASCII 0x0C (form feed), leads to the following error:

src/main.rs:1:1: 1:2 error: unknown start of token: \u{c}
src/main.rs:1 
              ^

I have a specific use case for form-feeds in source files. But I think in general it is nice to ignore the same whitespace that every other programming language and file format ignores; it lessens confusion for people coming from other languages and backgrounds.

My specific use case is the long-standing, but somewhat uncommon use of the form-feed character (which semantically is a separator between pages of text) as a way to group together especially closely related functions or blocks in a file of source code. Text editors or IDEs such as vim, Emacs or XCode provide convenience features to display these form-feeds in aesthetically pleasing way, move between form-feed-delimited pages, and restrict editing to one form-feed-delimited page at a time. It's just a simple convenience feature, but it would really be nice to support it.

[rntz]

+1, I also use this feature and was disappointed when I found Rust didn't treat it as whitespace.

[Aatch]

Looks like a fairly simple change could be made to the lexer so it uses char::is_whitespace instead of limiting to ' ', '\n', '\t', '\r'. The only think I can think of is that the is_whitespace function in lexer/mod.rs has been around since before we had a better is_whitespace function and nobody has changed it since then.

[steveklabnik]

/cc @rust-lang/lang , do we want to accept all kinds of whitespace?

[nikomatsakis]

I believe we should, yes.