bpo-28837: Fix lib2to3 handling of map/zip/filter

[stuarteberg]

The 2to3 tool will automatically wrap calls to map, zip, and filter with list():

# Original:
zip('abc', '123')

# After 2to3:
list(zip('abc', '123'))

...but the tool misses some cases, if the call is followed by a "trailer" such as [0]:

# Skipped by 2to3:
zip('abc', '123')[0]

This PR augments the lib2to3 "fixers" for map, zip, and filter to correctly handle such cases.

http://bugs.python.org/issue28837

[codecov]

Codecov Report

Merging #24 into master will increase coverage by <.01%.

@@            Coverage Diff             @@
##           master      #24      +/-   ##
==========================================
+ Coverage   82.37%   82.37%   +<.01%     
==========================================
  Files        1427     1427              
  Lines      350948   351009      +61     
==========================================
+ Hits       289088   289144      +56     
- Misses      61860    61865       +5

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e7ffb99...5cdd2db. Read the comment docs.

[stuarteberg]

@benjaminp: Are you the right person to ping about lib2to3 PRs?

[benjaminp]

What's the reason for this change?

[stuarteberg]

What's the reason for this change?

On line 100 (below), I need to clone the argument list:

new = Node(syms.power, [Name("map"), args.clone()])

If I leave the above line unchanged, then I could have written this, instead:

new = Node(syms.power, [Name("map"), ArgList([args.clone()])])

...but there's a problem with that: It doesn't clone the ( and ) tokens -- it creates them from scratch. As a result, one of the tests fails, due to an unpreserved prefix before the ) token:

======================================================================
FAIL: test_prefix_preservation (Lib.lib2to3.tests.test_fixers.Test_map)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/bergs/Documents/workspace/cpython-git/Lib/lib2to3/tests/test_fixers.py", line 3036, in test_prefix_preservation
    self.check(b, a)
  File "/Users/bergs/Documents/workspace/cpython-git/Lib/lib2to3/tests/test_fixers.py", line 3031, in check
    super(Test_map, self).check(b, a)
  File "/Users/bergs/Documents/workspace/cpython-git/Lib/lib2to3/tests/test_fixers.py", line 36, in check
    tree = self._check(before, after)
  File "/Users/bergs/Documents/workspace/cpython-git/Lib/lib2to3/tests/test_fixers.py", line 32, in _check
    self.assertEqual(after, str(tree))
AssertionError: "x =    list(map(   f,    'abc'   ))\n\n" != "x =    list(map(   f,    'abc'))\n\n"
- x =    list(map(   f,    'abc'   ))
?                               ---
+ x =    list(map(   f,    'abc'))

Hence, I think it's best to capture and clone the whole trailer. But if I'm missing something, please let me know.

[benjaminp]

Okay, thanks for the explanation.