Auto merge of #21619 - brson:featureck-tidy, r=alexcrichton

Builds on my [feature staging PR](#21248) to clean up the tidy scripts a bit, and make them much faster (6s vs ~40s).

Adds make rules 'tidy-basic', 'tidy-binaries', 'tidy-errors' and 'tidy-features'.

This is the output of `make tidy` here:

```
cfg: version 1.0.0-dev (a8c878d 2015-01-25 01:49:14 -0800)
cfg: build triple x86_64-unknown-linux-gnu
cfg: host triples x86_64-unknown-linux-gnu
cfg: target triples x86_64-unknown-linux-gnu
cfg: host for x86_64-unknown-linux-gnu is x86_64
cfg: os for x86_64-unknown-linux-gnu is unknown-linux-gnu
cfg: good valgrind for x86_64-unknown-linux-gnu is 1
cfg: using CC=gcc (CFG_CC)
cfg: enabling valgrind run-pass tests (CFG_ENABLE_VALGRIND_RPASS)
cfg: valgrind-rpass command set to "/usr/bin/valgrind" --error-exitcode=100 --soname-synonyms=somalloc=NONE --quiet --suppressions=/home/brian/dev/rust3/src/etc/x86.supp  --tool=memcheck --leak-check=full
cfg: no lualatex found, deferring to xelatex
cfg: no xelatex found, deferring to pdflatex
cfg: no pdflatex found, disabling LaTeX docs
cfg: no pandoc found, omitting PDF and EPUB docs
cfg: including test rules
cfg: javac not available, skipping lexer test...
check: formatting

* linted .rs files: 4948
* linted .py files: 27
* linted .js files: 2
* linted .sh files: 5
* linted .pl files: 0
* linted .c files: 28
* linted .h files: 3
* other linted files: 0
* total lines of code: 481217
* total non-blank lines of code: 423682

check: binaries
check: extended errors

* 249 error codes
* highest error code: E0315

check: feature sanity

* advanced_slice_patterns         lang    unstable    None    
* alloc                           lib     unstable    None    
* asm                             lang    unstable    None    
* associated_types                lang    stable      1.0.0   
* box_syntax                      lang    unstable    None    
* collections                     lib     unstable    None    
* concat_idents                   lang    unstable    None    
* core                            lib     unstable    None    
* default_type_params             lang    stable      1.0.0   
* globs                           lang    stable      1.0.0   
* hash                            lib     unstable    None    
* if_let                          lang    stable      1.0.0   
* import_shadowing                lang    unstable    None    
* int_uint                        lang    unstable    None    
* intrinsics                      lang    unstable    None    
* io                              lib     unstable    None    
* issue_5723_bootstrap            lang    stable      1.0.0   
* lang_items                      lang    unstable    None    
* link_args                       lang    unstable    None    
* link_llvm_intrinsics            lang    unstable    None    
* linkage                         lang    unstable    None    
* log_syntax                      lang    unstable    None    
* macro_rules                     lang    stable      1.0.0   
* main                            lang    unstable    None    
* managed_boxes                   lang    unstable    None    
* non_ascii_idents                lang    unstable    None    
* old_impl_check                  lang    unstable    None    
* old_orphan_check                lang    unstable    None    
* on_unimplemented                lang    unstable    None    
* opt_out_copy                    lang    unstable    None    
* optin_builtin_traits            lang    unstable    None    
* os                              lib     unstable    None    
* path                            lib     unstable    None    
* phase                           lang    unstable    None    
* plugin                          lang    unstable    None    
* plugin_registrar                lang    unstable    None    
* quad_precision_float            lang    unstable    None    
* quote                           lang    unstable    None    
* rand                            lib     unstable    None    
* rust1                           lib     stable      1.0.0   
* rustc_diagnostic_macros         lang    unstable    None    
* rustc_private                   lib     unstable    None    
* rustdoc                         lib     unstable    None    
* simd                            lang    unstable    None    
* simd_ffi                        lang    unstable    None    
* slicing_syntax                  lang    unstable    None    
* staged_api                      lang    unstable    None    
* start                           lang    unstable    None    
* std_misc                        lib     unstable    None    
* struct_inherit                  lang    unstable    None    
* struct_variant                  lang    stable      1.0.0   
* test                            lib     unstable    None    
* test_accepted_feature           lang    stable      1.0.0   
* test_removed_feature            lang    unstable    None    
* thread_local                    lang    unstable    None    
* trace_macros                    lang    unstable    None    
* tuple_indexing                  lang    stable      1.0.0   
* unboxed_closures                lang    unstable    None    
* unicode                         lib     unstable    None    
* unsafe_destructor               lang    unstable    None    
* visible_private_types           lang    unstable    None    
* while_let                       lang    stable      1.0.0   
```

There's a lot of informational output now, which comes after things like 'NOTE's.

Makefile.in

@@ -59,6 +59,9 @@
 #   * check-stage$(stage)-$(crate) - Test a crate in a specific stage
 #   * check-stage$(stage)-{rpass,rfail,cfail,rmake,...} - Run tests in src/test/
 #   * check-stage1-T-$(target)-H-$(host) - Run cross-compiled-tests
+#   * tidy-basic - show file / line stats
+#   * tidy-errors - show the highest rustc error code
+#   * tidy-features - show the status of language and lib features
 #
 # Then mix in some of these environment variables to harness the
 # ultimate power of The Rust Build System.

mk/main.mk

@@ -72,6 +72,7 @@ endif
 # numbers and dots here
 CFG_VERSION_WIN = $(CFG_RELEASE_NUM)
 
+CFG_INFO := $(info cfg: version $(CFG_VERSION))
 
 ######################################################################
 # More configuration
@@ -179,6 +180,7 @@ endif
 
 ifndef CFG_DISABLE_VALGRIND_RPASS
   $(info cfg: enabling valgrind run-pass tests (CFG_ENABLE_VALGRIND_RPASS))
+  $(info cfg: valgrind-rpass command set to $(CFG_VALGRIND))
   CFG_VALGRIND_RPASS :=$(CFG_VALGRIND)
 else
   CFG_VALGRIND_RPASS :=

mk/tests.mk

@@ -162,7 +162,8 @@ $(foreach file,$(wildcard $(S)src/doc/trpl/*.md), \
 ######################################################################
 
 # The main testing target. Tests lots of stuff.
-check: cleantmptestlogs cleantestlibs check-notidy tidy
+check: cleantmptestlogs cleantestlibs all check-stage2 tidy
+	$(Q)$(CFG_PYTHON) $(S)src/etc/check-summary.py tmp/*.log
 
 # As above but don't bother running tidy.
 check-notidy: cleantmptestlogs cleantestlibs all check-stage2
@@ -235,57 +236,24 @@ cleantestlibs:
 ######################################################################
 
 ifdef CFG_NOTIDY
+.PHONY: tidy
 tidy:
 else
 
-ALL_CS := $(wildcard $(S)src/rt/*.cpp \
-                     $(S)src/rt/*/*.cpp \
-                     $(S)src/rt/*/*/*.cpp \
-                     $(S)src/rustllvm/*.cpp)
-ALL_CS := $(filter-out $(S)src/rt/miniz.cpp \
-		       $(wildcard $(S)src/rt/hoedown/src/*.c) \
-		       $(wildcard $(S)src/rt/hoedown/bin/*.c) \
-	,$(ALL_CS))
-ALL_HS := $(wildcard $(S)src/rt/*.h \
-                     $(S)src/rt/*/*.h \
-                     $(S)src/rt/*/*/*.h \
-                     $(S)src/rustllvm/*.h)
-ALL_HS := $(filter-out $(S)src/rt/valgrind/valgrind.h \
-                       $(S)src/rt/valgrind/memcheck.h \
-                       $(S)src/rt/msvc/typeof.h \
-                       $(S)src/rt/msvc/stdint.h \
-                       $(S)src/rt/msvc/inttypes.h \
-		       $(wildcard $(S)src/rt/hoedown/src/*.h) \
-		       $(wildcard $(S)src/rt/hoedown/bin/*.h) \
-	,$(ALL_HS))
-
 # Run the tidy script in multiple parts to avoid huge 'echo' commands
-tidy:
+.PHONY: tidy
+tidy: tidy-basic tidy-binaries tidy-errors tidy-features
+
+endif
+
+.PHONY: tidy-basic
+tidy-basic:
 		@$(call E, check: formatting)
-		$(Q)find $(S)src -name '*.r[sc]' \
-		    -and -not -regex '^$(S)src/jemalloc.*' \
-		    -and -not -regex '^$(S)src/libuv.*' \
-		    -and -not -regex '^$(S)src/llvm.*' \
-		    -and -not -regex '^$(S)src/gyp.*' \
-		    -and -not -regex '^$(S)src/libbacktrace.*' \
-		    -print0 \
-		| xargs -0 -n 10 $(CFG_PYTHON) $(S)src/etc/tidy.py
-		$(Q)find $(S)src/etc -name '*.py' \
-		| xargs -n 10 $(CFG_PYTHON) $(S)src/etc/tidy.py
-		$(Q)find $(S)src/doc -name '*.js' \
-		| xargs -n 10 $(CFG_PYTHON) $(S)src/etc/tidy.py
-		$(Q)find $(S)src/etc -name '*.sh' \
-		| xargs -n 10 $(CFG_PYTHON) $(S)src/etc/tidy.py
-		$(Q)find $(S)src/etc -name '*.pl' \
-		| xargs -n 10 $(CFG_PYTHON) $(S)src/etc/tidy.py
-		$(Q)find $(S)src/etc -name '*.c' \
-		| xargs -n 10 $(CFG_PYTHON) $(S)src/etc/tidy.py
-		$(Q)find $(S)src/etc -name '*.h' \
-		| xargs -n 10 $(CFG_PYTHON) $(S)src/etc/tidy.py
-		$(Q)echo $(ALL_CS) \
-		| xargs -n 10 $(CFG_PYTHON) $(S)src/etc/tidy.py
-		$(Q)echo $(ALL_HS) \
-		| xargs -n 10 $(CFG_PYTHON) $(S)src/etc/tidy.py
+		$(Q) $(CFG_PYTHON) $(S)src/etc/tidy.py $(S)src/
+
+.PHONY: tidy-binaries
+tidy-binaries:
+		@$(call E, check: binaries)
 		$(Q)find $(S)src -type f -perm +a+x \
 		    -not -name '*.rs' -and -not -name '*.py' \
 		    -and -not -name '*.sh' \
@@ -300,11 +268,16 @@ tidy:
 		| grep '^$(S)src/libbacktrace' -v \
 		| grep '^$(S)src/rust-installer' -v \
 		| xargs $(CFG_PYTHON) $(S)src/etc/check-binaries.py
-		$(Q) $(CFG_PYTHON) $(S)src/etc/errorck.py $(S)src/
-		$(Q) $(CFG_PYTHON) $(S)src/etc/featureck.py $(S)src/
 
+.PHONY: tidy-errors
+tidy-errors:
+		@$(call E, check: extended errors)
+		$(Q) $(CFG_PYTHON) $(S)src/etc/errorck.py $(S)src/
 
-endif
+.PHONY: tidy-features
+tidy-features:
+		@$(call E, check: feature sanity)
+		$(Q) $(CFG_PYTHON) $(S)src/etc/featureck.py $(S)src/
 
 
 ######################################################################
@@ -639,7 +612,6 @@ CTEST_COMMON_ARGS$(1)-T-$(2)-H-$(3) := \
 
 ifdef CFG_VALGRIND_RPASS
 ifdef GOOD_VALGRIND_$(2)
-$(info cfg: valgrind-path set to $(CFG_VALGRIND_RPASS))
 CTEST_COMMON_ARGS$(1)-T-$(2)-H-$(3) += --valgrind-path "$(CFG_VALGRIND_RPASS)"
 endif
 endif

src/etc/errorck.py

@@ -15,6 +15,10 @@
 import os
 import re
 
+if len(sys.argv) < 2:
+    print "usage: errorck.py <src-dir>"
+    sys.exit(1)
+
 src_dir = sys.argv[1]
 errcode_map = {}
 error_re = re.compile("(E\d\d\d\d)")
@@ -54,8 +58,10 @@
             print("{1}: {2}\n{3}".format(*entry))
         errors = True
 
-print("{0} error codes".format(len(errcode_map)))
-print("highest error code: " + max(all_errors))
+print
+print("* {0} error codes".format(len(errcode_map)))
+print("* highest error code: " + max(all_errors))
+print
 
 if errors:
     sys.exit(1)

src/etc/featureck.py

@@ -20,6 +20,10 @@
 
 import sys, os, re
 
+if len(sys.argv) < 2:
+    print "usage: featurkck.py <src-dir>"
+    sys.exit(1)
+
 src_dir = sys.argv[1]
 
 # Features that are allowed to exist in both the language and the library
@@ -234,10 +238,8 @@
 
 lines.sort()
 
-print
-print "Rust feature summary:"
 print
 for line in lines:
-    print line
+    print "* " + line
 print
 

src/etc/licenseck.py

@@ -22,11 +22,6 @@
 \\1 except according to those terms.""")
 
 exceptions = [
-    "rt/rust_android_dummy.cpp", # BSD, chromium
-    "rt/rust_android_dummy.h", # BSD, chromium
-    "rt/isaac/randport.cpp", # public domain
-    "rt/isaac/rand.h", # public domain
-    "rt/isaac/standard.h", # public domain
     "libstd/sync/mpsc/mpsc_queue.rs", # BSD
     "libstd/sync/mpsc/spsc_queue.rs", # BSD
     "test/bench/shootout-binarytrees.rs", # BSD

src/etc/tidy.py

@@ -51,78 +51,155 @@ def do_license_check(name, contents):
     if not check_license(name, contents):
         report_error_name_no(name, 1, "incorrect license")
 
-
-file_names = [s for s in sys.argv[1:] if (not s.endswith("_gen.rs"))
-                                     and (not ".#" in s)]
-
 current_name = ""
 current_contents = ""
 check_tab = True
 check_cr = True
 check_linelength = True
 
+if len(sys.argv) < 2:
+    print "usage: tidy.py <src-dir>"
+    sys.exit(1)
+
+src_dir = sys.argv[1]
 
 try:
-    for line in fileinput.input(file_names,
+    count_lines = 0
+    count_non_blank_lines = 0
+
+    interesting_files = ['.rs', '.py', '.js', '.sh', '.c', '.h']
+
+    file_counts = {ext: 0 for ext in interesting_files}
+    file_counts['other'] = 0
+
+    def update_counts(current_name):
+        global file_counts
+        _, ext = os.path.splitext(current_name)
+
+        if ext in file_counts:
+            file_counts[ext] += 1
+        else:
+            file_counts['other'] += 1
+
+    all_paths = set()
+
+    for (dirpath, dirnames, filenames) in os.walk(src_dir):
+
+        # Skip some third-party directories
+        skippable_dirs = {
+            'src/jemalloc',
+            'src/llvm',
+            'src/gyp',
+            'src/libbacktrace',
+            'src/libuv',
+            'src/compiler-rt',
+            'src/rt/hoedown',
+            'src/rustllvm',
+            'src/rt/valgrind',
+            'src/rt/msvc',
+            'src/rust-installer'
+        }
+
+        if any(d in dirpath for d in skippable_dirs):
+            continue
+
+        def interesting_file(f):
+            if "miniz.c" in f \
+            or "jquery" in f \
+            or "rust_android_dummy" in f:
+                return False
+
+            return any(os.path.splitext(f)[1] == ext for ext in interesting_files)
+
+        file_names = [os.path.join(dirpath, f) for f in filenames
+                      if interesting_file(f)
+                      and not f.endswith("_gen.rs")
+                      and not ".#" is f]
+
+        if not file_names:
+            continue
+
+        for line in fileinput.input(file_names,
                                 openhook=fileinput.hook_encoded("utf-8")):
 
-        if "tidy.py" not in fileinput.filename():
+            filename = fileinput.filename()
+
+            if "tidy.py" not in filename:
+                if "TODO" in line:
+                    report_err("TODO is deprecated; use FIXME")
+                match = re.match(r'^.*/(\*|/!?)\s*XXX', line)
+                if match:
+                    report_err("XXX is no longer necessary, use FIXME")
+                match = re.match(r'^.*//\s*(NOTE.*)$', line)
+                if match and "TRAVIS" not in os.environ:
+                    m = match.group(1)
+                    if "snap" in m.lower():
+                        report_warn(match.group(1))
+                match = re.match(r'^.*//\s*SNAP\s+(\w+)', line)
+                if match:
+                    hsh = match.group(1)
+                    date, rev = snapshot.curr_snapshot_rev()
+                    if not hsh.startswith(rev):
+                        report_err("snapshot out of date (" + date
+                            + "): " + line)
+                else:
+                    if "SNAP" in line:
+                        report_warn("unmatched SNAP line: " + line)
+
             if cr_flag in line:
                 check_cr = False
             if tab_flag in line:
                 check_tab = False
             if linelength_flag in line:
                 check_linelength = False
-            if "TODO" in line:
-                report_err("TODO is deprecated; use FIXME")
-            match = re.match(r'^.*/(\*|/!?)\s*XXX', line)
-            if match:
-                report_err("XXX is no longer necessary, use FIXME")
-            match = re.match(r'^.*//\s*(NOTE.*)$', line)
-            if match and "TRAVIS" not in os.environ:
-                m = match.group(1)
-                if "snap" in m.lower():
-                    report_warn(match.group(1))
-            match = re.match(r'^.*//\s*SNAP\s+(\w+)', line)
-            if match:
-                hsh = match.group(1)
-                date, rev = snapshot.curr_snapshot_rev()
-                if not hsh.startswith(rev):
-                    report_err("snapshot out of date (" + date
-                               + "): " + line)
-            else:
-                if "SNAP" in line:
-                    report_warn("unmatched SNAP line: " + line)
-
-        if check_tab and ('\t' in line and
-                          "Makefile" not in fileinput.filename()):
-            report_err("tab character")
-        if check_cr and not autocrlf and '\r' in line:
-            report_err("CR character")
-        if line.endswith(" \n") or line.endswith("\t\n"):
-            report_err("trailing whitespace")
-        line_len = len(line)-2 if autocrlf else len(line)-1
-
-        if check_linelength and line_len > cols:
-            report_err("line longer than %d chars" % cols)
-
-        if fileinput.isfirstline() and current_name != "":
-            do_license_check(current_name, current_contents)
-
-        if fileinput.isfirstline():
-            current_name = fileinput.filename()
-            current_contents = ""
-            check_cr = True
-            check_tab = True
-            check_linelength = True
-
-        current_contents += line
+
+            if check_tab and ('\t' in line and
+                              "Makefile" not in filename):
+                report_err("tab character")
+            if check_cr and not autocrlf and '\r' in line:
+                report_err("CR character")
+            if line.endswith(" \n") or line.endswith("\t\n"):
+                report_err("trailing whitespace")
+            line_len = len(line)-2 if autocrlf else len(line)-1
+
+            if check_linelength and line_len > cols:
+                report_err("line longer than %d chars" % cols)
+
+            if fileinput.isfirstline():
+                # This happens at the end of each file except the last.
+                if current_name != "":
+                    update_counts(current_name)
+                    assert len(current_contents) > 0
+                    do_license_check(current_name, current_contents)
+
+                current_name = filename
+                current_contents = ""
+                check_cr = True
+                check_tab = True
+                check_linelength = True
+
+            # Put a reasonable limit on the amount of header data we use for
+            # the licenseck
+            if len(current_contents) < 1000:
+                current_contents += line
+
+            count_lines += 1
+            if line.strip():
+                count_non_blank_lines += 1
 
     if current_name != "":
+        update_counts(current_name)
+        assert len(current_contents) > 0
         do_license_check(current_name, current_contents)
 
 except UnicodeDecodeError as e:
     report_err("UTF-8 decoding error " + str(e))
 
+print
+for ext in file_counts:
+    print "* linted " + str(file_counts[ext]) + " " + ext + " files"
+print "* total lines of code: " + str(count_lines)
+print "* total non-blank lines of code: " + str(count_non_blank_lines)
+print
 
 sys.exit(err)