Test project on Windows with MINGW git (conda2.7&3.4/cpy-3.5) #519

ankostis · 2016-09-25T14:22:50Z

Use Appveyor integration servers;
Copy part of the travis logic, to speed-up test-time (no flake8, site & coverage), the only extra is wheel-packaging;
Limit combination of Anaconda, CPython and MINGW/Cygwin in only 6 variations.
Replaced poll/select buffer-assembly code with pump-threads (Win TCs faling 90-->20!) and rework unicode handling in many places.
Popen() procs it with subprocess.CREATE_NEW_PROCESS_GROUP so that they can be killed.
Invoke MINGW git-daemon (instead of git daemon); otherwise it detaches (unix-ism here) from parent git cmd, and then CANNOT DIE!
Stop reading directly Popen().proc.stdout/stderr from the launching-thread, it freezes due to GIL/buffering.
- Unslurp diff consumption code (but not raw-patch) to handle it line-by line to support arbitrary big output.
Ensure read-only files do delete on Windows.
+Rework GitCommandError.str() and add TCs on unicode handling.
Fixed TCs on config-reader/writer:
- Modify lock/read-config-file code to ensure files closed.
- Use with GitConfigarser() more systematically in TCs.
- Clean up any locks left hanging from prev TCs.
- Util: mark lock-files as SHORT_LIVED; save some SSDs...
Fixed TestRepo.test_submodule_update():
- submod: del .git file prior overwrite; Windows denied otherwise!
FIX TestRepo.test_untracked_files():
- In the git add <file> case, PY2 fails when args are unicode.
  Must encode them with locale.getpreferredencoding() AND use SHELL.
cmd: add shell into execute() kwds, for overriding USE_SHELL per
command.
repo: replace blocky communicate() in _clone() with thread-pumps (UNNECESSARY, see this later discussion).
test_repo.py: unittestize (almost all) assertions.
Fixed file-access problems by using with... construct (not complete).
- Replace open --> with open for index (base and TC).
Enabled some TCs, more remain dormant (ie the Hook TC never runs on Linux).
- test_index.py: Enabled dormant fake-symlink assertion.
Various encoding issues.
Debug-log all Popen() calls, to collect cmd usage.

Apveyor results

Py27: FAILED (SKIP=3, errors=1)
Py34: FAILED (SKIP=3, errors=3)
Py35 : FAILED (SKIP=3, errors=3)

Byron · 2016-09-25T14:35:35Z

@ankostis Totally awesome ! Even though I won't be of much help, I do hope that fixing the windows issues work out in the end. That would restore windows platform support and keep it, after all these years of supporting it just on a best-effort basis, which meant not to obviously use non-windows APIs.

+ Del extra spaces, import os.path as osp

[travisci skip]

ankostis · 2016-09-25T16:38:58Z

Yes, that's a start.
I wish that today this PR become "ready", but currently, it keeps on failing,
and adding gc.collect() in various places...

ankostis · 2016-09-25T22:05:03Z

@Byron the git.testtest_git:TestGit.test_handle_process_output() TC hangs almost on every combination under Windows.
Can you help unstuck it?

+ Pump once, so when input larger than `mmap.PAGESIZE`, stream is not emptied.

+ Pump code reads only once streams per `_read_lines_from_fno()` invocation, so when input larger than `mmap.PAGESIZE`, bytes are forgotten in the stream.

+ The code in `_read_lines_from_fno()` was reading the stream only once per invocation, so when input was larger than `mmap.PAGESIZE`, bytes were forgotten in the stream. + Also set deamon-threads.

+ The code in `_read_lines_from_fno()` was reading the stream only once per invocation, so when input was larger than `mmap.PAGESIZE`, bytes were forgotten in the stream. + Replaced buffer-banging code with iterate-on-file-descriptors. + Also set deamon-threads.

+ The code in `_read_lines_from_fno()` was reading the stream only once per invocation, so when input was larger than `mmap.PAGESIZE`, bytes were forgotten in the stream. + Replaced buffer-building code with iterate-on-file-descriptors. + Also set deamon-threads.

ankostis · 2016-09-26T00:49:16Z

@Byron that was a tough one!
The pump code in cmd:handle_process_output() was crafted for the select/poll and was reading the stream only once per _read_lines_from_fno() invocation; so when input was larger than mmap.PAGESIZE, bytes were forgotten in the stream, and the child-process hang.

I replaced the buffer-assembly code with iterate-on-file-descriptors. Actually the code is so simple that it maybe worth to replace the select-poll with the 2-pumps code, for all OSs?

Many ~~more~~ bugs remain in Windows, but no hangs.
[edit]Actually with this bug fixed, the failed TCs on Windows dropped from ~90-->20!!

A side question:
In cmd.handle_process_output() you write:

NOTE: Just joining threads can possibly fail as there is a gap between .start() and when it's
actually started, which could make the wait() call to just return because the thread is not yet
active

Is that possible? I would expect any language allowing such "gap" to be considered outright broken.
But I couldnt google anything relevant?

…`assertRaisesRegexp`

ankostis · 2016-10-01T18:50:56Z

Just pushed one last commit with those 3 TCs appropriately skipped.
-- ALL GREEN now --

Especially the one with the incompatible type appears like something that is relatively easy reproduce and fix via smmap.

Actually quite the contrary: I can only reproduce it when i run the full harness; if i launch these 2 TCs alone, they pass! So practically, I cannot enter debug-mode - I have to wait 2' just to reach this point.

PS: You have been invited to become a contributor/maintainer.

Have not received an invitation yet.

Byron · 2016-10-01T18:56:32Z

This is odd:

I have just re-invited you, maybe it comes through. Otherwise you would just have to visit https://github.com/gitpython-developers to see the invitation.

Your last commit was not part of my merge, I will fetch it separately.

+ Just to see Apveyor all green and merge; the TCs HAVE TO BE FIXED.

Byron · 2016-10-01T19:11:03Z

@ankostis I think it worked ! The last commit has been cherry-picked and AppVeyor is on its way to all-green ! Even though it's not the real thing just yet, I call it an impressive piece of work! Thanks so much !

Right now I wouldn't know how to tackle the smmap issue, just as it's hard to fix anything in a project that is installed as a dependency. Probably it would be easiest if it would be reproducible locally, which I can't even do.
Just so you know, I will probably go to read-only mode for the next days due to my travels, but should be back with replies by the coming weekend.

ankostis · 2016-10-01T19:22:52Z

@Byron many of the fixes in this PR can be applied also on gitdb, and as you said (haven't checked) on smmp. Is there a chance to tackle these issues in these projects? Or are they "frozen" somehow?

I mean, a lot of problems just go away if resource classes are retrofitted as with .... context resources - but this means API forward compatibility is broken for future clients (existing code should work with the modified libs though).

What do you think, is it worth the effort (but cna't promise anything)?

…f cmds

…rs#519 + Regression introduced in d84b960, by a wrong comment interpretation.

ankostis · 2016-10-02T23:05:52Z

For reference, reporting the CORRECT Appveyor results after #521 fixes by yarikoptic & ankostis restored skipped TCs:

PY27: 5 errors:
PY34: 4 errors:
PY35: 7 errors:
conda35: 7 errors:

…preferredencoding()` + On Windows, this is the default encoding (usually, the mate-encoding `mcbs' which points to the current code-page (i.e. `cp1252` for Europe).

Byron · 2016-10-22T09:23:08Z

@ankostis With the latest upgrade of gitdb and smmap to 2.0, it's finally possible again to make new releases. This would allow the changes you talk about to be retro-fitted indeed, and given all the subtle problems GitPython has, it would certainly be worth doing that.

In the git.diff.Diff class, the _index_from_patch_format and _index_from_raw_format methods' docstrings had still described them as reading from streams, even though they have instead read from processes (and taken "proc", not "stream", arguments) since gitpython-developers#519 when the change was made in a5db3d3 to fix a freezing bug. This updates the docstrings to reflect that they read from processes.

Byron added the acknowledged label Sep 25, 2016

ankostis force-pushed the appveyor branch 2 times, most recently from 1774f98 to d773d40 Compare September 25, 2016 16:31

ankostis added 2 commits September 25, 2016 18:35

test, deps: FIX mock deps on py3.

7842e92

+ Del extra spaces, import os.path as osp

apveyor: Wintest project with MINGW/Cygwin git (conda2.7&3.4/cpy-3.5)

1210ec7

[travisci skip]

ankostis force-pushed the appveyor branch from d773d40 to 8fe4fee Compare September 25, 2016 16:35

ankostis force-pushed the appveyor branch 3 times, most recently from 5899c59 to d12fdca Compare September 25, 2016 22:01

ankostis added a commit to ankostis/GitPython that referenced this pull request Sep 25, 2016

apveyor, gitpython-developers#519: FIX incomplete Popen pump

64fc749

+ Pump once, so when input larger than `mmap.PAGESIZE`, stream is not emptied.

ankostis force-pushed the appveyor branch from 64fc749 to 4b1249e Compare September 25, 2016 23:48

ankostis added a commit to ankostis/GitPython that referenced this pull request Sep 25, 2016

apveyor, gitpython-developers#519: FIX incomplete Popen pump

4b1249e

+ Pump once, so when input larger than `mmap.PAGESIZE`, stream is not emptied.

ankostis force-pushed the appveyor branch from 4b1249e to 040f549 Compare September 25, 2016 23:57

ankostis added a commit to ankostis/GitPython that referenced this pull request Sep 25, 2016

apveyor, gitpython-developers#519: FIX incomplete Popen pump

040f549

+ Pump once, so when input larger than `mmap.PAGESIZE`, stream is not emptied.

ankostis force-pushed the appveyor branch from 040f549 to 022ce82 Compare September 26, 2016 00:21

ankostis force-pushed the appveyor branch from 022ce82 to 148959c Compare September 26, 2016 00:29

ankostis force-pushed the appveyor branch from 148959c to 747a0ff Compare September 26, 2016 00:31

ankostis force-pushed the appveyor branch from 747a0ff to b3aaf7b Compare September 26, 2016 00:47

ankostis added a commit to ankostis/GitPython that referenced this pull request Sep 26, 2016

tesr, gitpython-developers#519: FIX failures in py2.6 due to missing …

7810405

…`assertRaisesRegexp`

ankostis force-pushed the appveyor branch from 7810405 to c785a03 Compare September 26, 2016 09:26

ankostis added a commit to ankostis/GitPython that referenced this pull request Sep 26, 2016

tesr, gitpython-developers#519: FIX failures in py2.6 due to missing …

c785a03

…`assertRaisesRegexp`

ankostis force-pushed the appveyor branch from c785a03 to de320f3 Compare September 26, 2016 09:32

Byron pushed a commit that referenced this pull request Oct 1, 2016

TC, #519: DISABLE failing tests

b3b9c02

+ Just to see Apveyor all green and merge; the TCs HAVE TO BE FIXED.

ankostis mentioned this pull request Oct 1, 2016

[Windows] Create_submodule fails due to use of hidden files #516

Closed

ankostis changed the title ~~Test project on Windows with MINGW/Cygwin git (conda2.7&3.4/cpy-3.5)~~ Test project on Windows with MINGW git (conda2.7&3.4/cpy-3.5) Oct 1, 2016

ankostis added a commit to ankostis/GitPython that referenced this pull request Oct 2, 2016

FIX regression by gitpython-developers#519 on reading stdout/stderr o…

08f7d17

…f cmds

ankostis added a commit to ankostis/GitPython that referenced this pull request Oct 2, 2016

FIX regression by gitpython-developers#519 on reading stdout/stderr o…

df5c1cb

…f cmds

ankostis mentioned this pull request Oct 2, 2016

BF: remove a submodule with a remote without refs + misc fixes around #521

Merged

ankostis added a commit to ankostis/GitPython that referenced this pull request Oct 2, 2016

FIX config-lock release early regression caused by gitpython-develope…

8a01ec4

…rs#519 + Regression introduced in d84b960, by a wrong comment interpretation.

ankostis mentioned this pull request Oct 3, 2016

Fix "known" Windows errored test cases #525

Open

ankostis mentioned this pull request Oct 11, 2016

Issues with AutoInterrupt in conjunction with sys.exit on python 3.5 #463

Closed

ankostis added the tag.leaks label Oct 11, 2016

ankostis mentioned this pull request Oct 11, 2016

issue with diff method #501

Open

ankostis modified the milestone: v2.1.0 - proper windows support Oct 15, 2016

ankostis mentioned this pull request Oct 16, 2016

Support Cygwin's Git on Windows #533

Merged

ankostis mentioned this pull request Nov 28, 2017

BF (codename "happy travis"): trying to address lints etc to make Travis green again #702

Merged

aarondav mentioned this pull request Jun 28, 2018

Require gitpython>=2.1.0 mlflow/mlflow#98

Merged

EliahKagan mentioned this pull request Nov 4, 2023

is_darwin is always False (os.name is never "darwin") #1731

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test project on Windows with MINGW git (conda2.7&3.4/cpy-3.5) #519

Test project on Windows with MINGW git (conda2.7&3.4/cpy-3.5) #519

ankostis commented Sep 25, 2016 •

edited

Loading

Byron commented Sep 25, 2016

ankostis commented Sep 25, 2016

ankostis commented Sep 25, 2016

ankostis commented Sep 26, 2016 •

edited

Loading

ankostis commented Oct 1, 2016

Byron commented Oct 1, 2016

Byron commented Oct 1, 2016

ankostis commented Oct 1, 2016 •

edited

Loading

ankostis commented Oct 2, 2016

Byron commented Oct 22, 2016

Test project on Windows with MINGW git (conda2.7&3.4/cpy-3.5) #519

Test project on Windows with MINGW git (conda2.7&3.4/cpy-3.5) #519

Conversation

ankostis commented Sep 25, 2016 • edited Loading

Apveyor results

Byron commented Sep 25, 2016

ankostis commented Sep 25, 2016

ankostis commented Sep 25, 2016

ankostis commented Sep 26, 2016 • edited Loading

ankostis commented Oct 1, 2016

Byron commented Oct 1, 2016

Byron commented Oct 1, 2016

ankostis commented Oct 1, 2016 • edited Loading

ankostis commented Oct 2, 2016

Byron commented Oct 22, 2016

ankostis commented Sep 25, 2016 •

edited

Loading

ankostis commented Sep 26, 2016 •

edited

Loading

ankostis commented Oct 1, 2016 •

edited

Loading