Get LLVM using jemalloc and inline FNV hashing #31460
Conversation
|
Do you have any numbers for this? Also, I seem to recall a huge discussion the last time the prefixes were to be dropped. Has that been resolved or is this a different change? |
|
We should put in a release note if the jemalloc prefix changes. If anyone is experimenting with |
|
@dotdash ah yeah sorry I forgot to copy them over into the PR description, but the two commits have: jemalloc un-prefixing:
fnv inline:
@bluss good point! |
alexcrichton
added
the
relnotes
|
@alexcrichton Thanks! You missed the prefix issue though. The old PR was #18678 which, besides the long discussion that led to it not being merged, also had it so that on iOS/OSX and Windows, the prefix is still present (which is the default for jemalloc). |
|
Aha yes, thanks for finding a link to that! I should have hunted that down and thrown that in here. When that discussion happened we did not have the same support for swapping out allocators that we have today. The RFC for custom allocators was merged long after that discussion, and the early conclusions also pointed towards using jemalloc unprefixed in executables. The major difference between now and #18678 is that we only use jemalloc in executables, no static library or dynamic library in Rust uses jemalloc by default (they use the system allocator). As a result, due to the change in landscape, many of the concerns raised in #18678 have since been solved. One of the major motivational factors for RFC 1183 was indeed to land this change, I just forgot to get around to it until now! |
|
@bors r+ |
|
📌 Commit 0e5c112 has been approved by |
|
⌛ Testing commit 0e5c112 with merge 9b24f54... |
|
💔 Test failed - auto-mac-64-opt |
|
Oops! |
|
@bors r- Seems that bors was a bit too happy to see this restored |
alexcrichton
force-pushed
the
supafast-rustc
branch
from
0e5c112
to
51e0ee8
Compare
|
@bors: r=brson 51e0ee8 ok, let's try that again |
|
⌛ Testing commit 51e0ee8 with merge 59bff40... |
|
💔 Test failed - auto-linux-musl-64-opt |
|
Looks like the failure on musl was legitimate, and it happened because:
This meant the linker found itself trying to include the same symbol twice, and rightly exited with an error. I believe that the compiler here is erroneous in passing |
|
ah and re-r? @brson |
|
@bors r+ |
|
📌 Commit e93b127 has been approved by |
alexcrichton
force-pushed
the
supafast-rustc
branch
from
96429b6
to
00bf071
Compare
|
@bors: r=brson 00bf071 |
|
⌛ Testing commit 00bf071 with merge 5978690... |
|
@bors: retry force |
|
⌛ Testing commit 00bf071 with merge e733a31... |
|
💔 Test failed - auto-mac-64-nopt-t |
|
@bors: retry On Thu, Feb 11, 2016 at 9:53 PM, bors notifications@github.com wrote:
|
|
⌛ Testing commit 00bf071 with merge 37ab094... |
|
💔 Test failed - auto-linux-64-x-android-t |
|
@bors: retry On Fri, Feb 12, 2016 at 4:51 AM, bors notifications@github.com wrote:
|
|
⌛ Testing commit 00bf071 with merge 27e3610... |
|
💔 Test failed - auto-linux-64-x-android-t |
|
Whelp, shoulda known that any segfault from unprefixing jemalloc is indicative of mixing allocators. Looks like on Android allocator replacement in libc somehow doesn't work. We call I'll just re-prefix the symbols on android for now. |
Right now the primary hashing algorithm of the compiler isn't actually inlined across crates, meaning that it may be missing out on some crucial optimizations in a few places (perhaps unrolling smaller loops, etc). This commit made the hashing function disappear from a profiled version of the compiler, but that's likely because it was just inlined elsewhere. When compiling winapi, however, this decreased compile time from 18.3 to 17.8 seconds (a 3% improvement).
Back in 9bc8e6d the linking of rlibs changed to using the `link_whole_rlib` function. This change, however was only intended to affect dylibs, not executables. For executables we don't actually want to link entire rlibs because we want the linker to strip out as much as possible. This commit adds a conditional to this logic to only link entire rlibs if we're creating a dylib, and otherwise an executable just links an rlib as usual. A test is included which will fail to link if this behavior is reverted.
Now that we properly only link in jemalloc when building executables, we have far less to worry about in terms of polluting the global namespace with the `free` and `malloc` symbols on Linux. This commit will primarily allow LLVM to use jemalloc so the compiler will only be using one allocator overall. Locally this took compile time for libsyntax from 95 seconds to 89 (a 6% improvement).
alexcrichton
force-pushed
the
supafast-rustc
branch
from
00bf071
to
e3b414d
Compare
|
@bors: r=brson e3b414d |
|
⌛ Testing commit e3b414d with merge a888333... |
Looking at some profiles of rustc recently, these seemed like two pretty easy wins we could get in terms of performance on the table.
Looking at some profiles of rustc recently, these seemed like two pretty easy wins we could get in terms of performance on the table.