Merge darwin/libdispatch-1324.41.2 to main #583

rokhinip · 2021-11-06T01:12:29Z

Merge tag darwin/libdispatch-1324.41.2 on darwin/trunk.

This includes the latest Apple source drop libdispatch-1324.41.2 (up from libdispatch-1121) on top of darwin/libdispatch-1121 (merged in #447)

Signed-off-by: Rokhini Prabhu <rokhinip@apple.com>

This is needed for proper shared linkage to the BlocksRuntime (e.g. when building SourceKit). We now build BlocksRuntime shared or static as requested. Signed-off-by: Kim Topley <ktopley@apple.com>

build: honour `BUILD_SHARED_LIBS` Signed-off-by: Kim Topley <ktopley@apple.com>

Adjust the build for Windows to permit the isa pointer to successfully link. They will be off by a level of indirection and need to be patched up at runtime. Signed-off-by: Kim Topley <ktopley@apple.com>

Windows BlocksRuntime Signed-off-by: Kim Topley <ktopley@apple.com>

…overlay. Because we are linking libdispatch through clang rather than swift, we need to explicitly add swiftrt.o, which is needed by ELF and COFF to register metadata sections. Fixes rdar://problem/44941707. Signed-off-by: Kim Topley <ktopley@apple.com>

[CMake] Make sure to link swiftrt.o when building the Swift Dispatch overlay. Signed-off-by: Kim Topley <ktopley@apple.com>

This fixes an error from the swift compiler: error: cannot convert value of type 'UInt32' to expected argument type 'UInt' Signed-off-by: Kim Topley <ktopley@apple.com>

swift: cast to the appropriate type Signed-off-by: Kim Topley <ktopley@apple.com>

queue: clean up a covered switch Signed-off-by: Kim Topley <ktopley@apple.com>

This unblocks building on 32-bit Linux using the i686 triple. Signed-off-by: Kim Topley <ktopley@apple.com>

Linux i686 Build Support Signed-off-by: Kim Topley <ktopley@apple.com>

This function is declared as accepting a `dispatch_function_t` callback, which is a function pointer with a `void *` argument. However, the implementation and Swift overlay declare the callback without arguments, causing a conflict which Clang warns about. Change the function signature to accept the correct type. Signed-off-by: Kim Topley <ktopley@apple.com>

Fix the signature of _dispatch_install_thread_detach_callback() Signed-off-by: Kim Topley <ktopley@apple.com>

Use the exported target from swift rather than recreate the paths locally. This almost works to replace the use of the paths. Unfortunately, swiftrt is not currently exported. Signed-off-by: Kim Topley <ktopley@apple.com>

build: support static builds for Swift Signed-off-by: Kim Topley <ktopley@apple.com>

dispatch_c99: remove unnecessary __printflike() definition Signed-off-by: Kim Topley <ktopley@apple.com>

Change from a symlink to a copy. This is more portable and fixes the distribution aspect. When the installation occurs, the symbolic link is not followed and a symbolic link is installed. Rather if we copy, we can get the contents. The files are small so the cost is relatively low. Signed-off-by: Kim Topley <ktopley@apple.com>

build: always copy the modulemaps Signed-off-by: Kim Topley <ktopley@apple.com>

Update the swift support CMake rules from XCTest. This adds support for additional features like partial module compilation which improves incremental builds. It allows for executable and library builds. Signed-off-by: Kim Topley <ktopley@apple.com>

Enhance add_wift_target to support building static libraries. We would always generate shared libraries previously. Signed-off-by: Kim Topley <ktopley@apple.com>

This splits out the SDK overlay component and libdispatch runtime itself. Doing so enables the re-use of libdispatch with and without swift and makes the behaviour similar across Darwin and other platforms. Signed-off-by: Kim Topley <ktopley@apple.com>

Signed-off-by: Kim Topley <ktopley@apple.com>

Split Swift SDK Overlay Signed-off-by: Kim Topley <ktopley@apple.com>

Remove the explicit /usr/include search path for the swift library build. This allows us to build for Windows as well. Signed-off-by: Kim Topley <ktopley@apple.com>

build: remove "/usr/include" for swift Signed-off-by: Kim Topley <ktopley@apple.com>

Treat windows like Linux and android and remove the BSD interfaces on these targets. This allows us to mostly build the SDK overlay for Windows. Signed-off-by: Kim Topley <ktopley@apple.com>

SDK: guard BSD specific paths against windows Signed-off-by: Kim Topley <ktopley@apple.com>

Provide an alias for the import symbol as the ObjC runtime is normally in a separate DLL and the compiler will annotate the function as being DLLImport. This allows us to resolve the symbols when building the SDK overlay. Signed-off-by: Kim Topley <ktopley@apple.com>

DispatchStubs: make more Windows friendly Signed-off-by: Kim Topley <ktopley@apple.com>

Signed-off-by: Rokhini Prabhu <rokhinip@apple.com>

Use an auto-reset event for the dispatch queue on Windows. On Linux, `eventfd` is used, which is auto-reset unless `EFD_SEMAPHORE` is specified. This mirrors that behaviour. The test suite continues to pass after this change. Signed-off-by: Rokhini Prabhu <rokhinip@apple.com>

dispatch: use auto-reset event for dispatch queue Signed-off-by: Rokhini Prabhu <rokhinip@apple.com>

The `DispatchIO` constructor maps to `dispatch_io_create`. The value on Windows for the `fd` (which is actually a `HANDLE`) is being truncated since `dispatch_fd_t` is actually `Int` not `Int32` on Windows. Use the `dispatch_fd_t` to ensure that the right size is always passed along. Signed-off-by: Rokhini Prabhu <rokhinip@apple.com>

IO: match the signature for `DispatchIO` with `dispatch_io_create` Signed-off-by: Rokhini Prabhu <rokhinip@apple.com>

When compiling statically, we have to link against DispatchStubs in addition to the other dependencies, so we are defining a separate modulemap for that. Signed-off-by: Rokhini Prabhu <rokhinip@apple.com>

Add modulemap for static compilation Signed-off-by: Rokhini Prabhu <rokhinip@apple.com>

_dispatch_runloop_queue_xref_dispose is declared in src/queue_internal.h but the declaration is hidden behind DISPATCH_COCOA_COMPAT. This means the call to _dispatch_runloop_queue_xref_dispose must also be put behind this preprocessor symbol. Signed-off-by: Rokhini Prabhu <rokhinip@apple.com>

Add missing DISPATCH_COCOA_COMPAT preprocessor symbol. Signed-off-by: Rokhini Prabhu <rokhinip@apple.com>

These changes include: * hw_config.h: prevent referring to sysctlbyname on OpenBSD, as this is not available on all platforms. * transform.c: these stanzas referring to FreeBSD or Linux also apply to OpenBSD. * tests/dispatch_apply.c: stanza applying to Linux applies to OpenBSD and also tweak style for consistency. * tests/dispatch_io_net.c, tests/dispatch_test.h: stanzas applying to FreeBSD also apply to OpenBSD. Signed-off-by: Rokhini Prabhu <rokhinip@apple.com>

Make some preliminary porting changes. Signed-off-by: Rokhini Prabhu <rokhinip@apple.com>

There is a small typo in src/shims/lock.h. Should read `transition` rather than `transiton`. Signed-off-by: Rokhini Prabhu <rokhinip@apple.com>

docs: fix simple typo, transiton -> transition Signed-off-by: Rokhini Prabhu <rokhinip@apple.com>

Android doesn't have a separate librt, it's just part of libc. Also, the static build wasn't working for armv7-a, because the test executables wouldn't link with the multiple definition errors listed in android/ndk#176, so use the workaround given there. Signed-off-by: Rokhini Prabhu <rokhinip@apple.com>

[android] Put in fixes for librt and armv7-a Signed-off-by: Rokhini Prabhu <rokhinip@apple.com>

The Windows 10 SDK Version 2104 (10.0.20348.0) which adds support for Windows 11 introduces new enumerated values for the logical processor configuration. Only one of the two is documented at MSDN. https://docs.microsoft.com/en-us/windows/win32/api/winnt/ne-winnt-logical_processor_relationship: ~~~ `RelationNumaNodeEx` Introduced in TBD - Release Iron. Requests that the full affinity be returned. Unlike the other relation types, `RelationNumaNodeEx` is not used oninput. It is simply a request for `RelationNumaNode` with full group information. ~~~ Treat this enumerated value as `RelationNodeNode`. It is unclear what the details of `RelationProcessorDie` is currently. For now, we leave that value ignored, though it is likely that we may have to address it in the future once the value is explained. Signed-off-by: Rokhini Prabhu <rokhinip@apple.com>

shims: adjust the Windows path for Windows 11 Signed-off-by: Rokhini Prabhu <rokhinip@apple.com>

Signed-off-by: Rokhini Prabhu <rokhinip@apple.com>

rokhinip · 2021-11-06T01:13:25Z

@swift-ci please test

das · 2021-11-06T01:58:17Z

CMakeLists.txt

@@ -40,6 +40,20 @@ int main(int argc, char *argv[]) {
  if(DISPATCH_HAVE_EXTENDED_SLPI_20348)
    add_compile_definitions(DISPATCH_HAVE_EXTENDED_SLPI_20348)
  endif()
+


not sure what's up with the github diff viewer but this difference isn't real, this file is identical between the two branches

3405691582 · 2021-11-09T01:40:55Z

os/workgroup_base.h

+#include <string.h>
+#include <stdlib.h>
+
+#include <mach/port.h>


Unprotected mach/port.h is a portability break. Availability.h appears to be missing, and os/base.h doesn't exist. os/object.h and os/clock.h does exist, but os/clock.h also refers to os/base.h, so something is missing here.

Seems like os/base.h should switch on os/generic_*_base.h. os/generic_*_basae.h needs to also #define away SPI_UNAVAILABLE...

3405691582 · 2021-11-09T02:04:48Z

There's other Mach-specific stuff that needs to be untangled too that I haven't felt up to untangling right now, but that'll trip up on supported non-Mach platforms too...

compnerd

I'll try to run through a build of this in a the next day or two.

compnerd · 2021-11-08T03:25:46Z

os/workgroup_base.h

+	OS_SWIFT_NAME(swift_name) \
+	OS_OBJECT_SHOW_SUBCLASS(name, super, name, ## __VA_ARGS__)
+
+#if defined(__LP64__)


I think that this needs a || defined(_WIN64). Windows64 is not a LP64 but rather a LLP64.

compnerd · 2021-11-12T04:34:47Z

There is a lot of use of mach_port_t, but I don't yet understand the use of them here. I wonder if we need to provide equivalences for those APIs on Linux/Android/Windows.

MadCoder · 2021-11-12T04:36:43Z

There is a lot of use of mach_port_t, but I don't yet understand the use of them here. I wonder if we need to provide equivalences for those APIs on Linux/Android/Windows.

the workgroup stuff is not relevant for non Darwin platforms and should be entirely hidden behind HAVE_MACH

compnerd · 2021-11-12T15:53:48Z

Thanks @MadCoder! That actually was very helpful. I do have a handful of patches to upload that improves stuff but is insufficient on its own. What would you prefer @rokhinip @MadCoder @das? I can push to the same branch, or create a separate branch that adds them. They aren't particularly freestanding, so I cannot just create PRs for main to address these items.

das · 2021-11-12T16:46:41Z

thanks @compnerd for looking at this!
It would be very useful to have a separate merge commit for the followup changes to this PR that we can cleanly split out, so I think a separate branch initially created off-of this one would be ideal so that we can either merge it to this branch or just merge the two to main one after the other (after a rebase)

compnerd · 2021-11-12T19:05:44Z

@das sounds good, I'm going to be pushing a series of changes to the same branch and have created a PR for ease of review. Expect a large number of changes, since I want to keep them tiny and targeted so we can deal with the individual issues. I've CC'ed the people here on #584.

rokhinip and others added 30 commits November 5, 2021 16:32

Merge libdispatch-1186

68c3553

Signed-off-by: Rokhini Prabhu <rokhinip@apple.com>

build: honour BUILD_SHARED_LIBS

f49eafb

This is needed for proper shared linkage to the BlocksRuntime (e.g. when building SourceKit). We now build BlocksRuntime shared or static as requested. Signed-off-by: Kim Topley <ktopley@apple.com>

Merge pull request #396 from compnerd/build-type-blocks

ac83a4a

build: honour `BUILD_SHARED_LIBS` Signed-off-by: Kim Topley <ktopley@apple.com>

build: enable usage on Windows

83b5a20

Adjust the build for Windows to permit the isa pointer to successfully link. They will be off by a level of indirection and need to be patched up at runtime. Signed-off-by: Kim Topley <ktopley@apple.com>

Merge pull request #362 from compnerd/shared

cf9fd05

Windows BlocksRuntime Signed-off-by: Kim Topley <ktopley@apple.com>

Merge pull request #398 from DougGregor/link-swiftrt

a79e5c1

[CMake] Make sure to link swiftrt.o when building the Swift Dispatch overlay. Signed-off-by: Kim Topley <ktopley@apple.com>

swift: cast to the appropriate type

8e746aa

This fixes an error from the swift compiler: error: cannot convert value of type 'UInt32' to expected argument type 'UInt' Signed-off-by: Kim Topley <ktopley@apple.com>

Merge pull request #399 from compnerd/casting

9fbd493

swift: cast to the appropriate type Signed-off-by: Kim Topley <ktopley@apple.com>

Merge pull request #400 from compnerd/cleanup

1ce501c

queue: clean up a covered switch Signed-off-by: Kim Topley <ktopley@apple.com>

Linux i686 Build Support

f2e0ebf

This unblocks building on 32-bit Linux using the i686 triple. Signed-off-by: Kim Topley <ktopley@apple.com>

Merge pull request #402 from Kaiede/i686Support

cad27f8

Linux i686 Build Support Signed-off-by: Kim Topley <ktopley@apple.com>

Merge pull request #403 from adierking/detach

8cefcf3

Fix the signature of _dispatch_install_thread_detach_callback() Signed-off-by: Kim Topley <ktopley@apple.com>

build: use the exported target instead of reconstructing paths

42e6931

Use the exported target from swift rather than recreate the paths locally. This almost works to replace the use of the paths. Unfortunately, swiftrt is not currently exported. Signed-off-by: Kim Topley <ktopley@apple.com>

Merge pull request #397 from compnerd/static-stdlib

8470f24

build: support static builds for Swift Signed-off-by: Kim Topley <ktopley@apple.com>

Merge pull request #406 from adierking/printflike

174bb90

dispatch_c99: remove unnecessary __printflike() definition Signed-off-by: Kim Topley <ktopley@apple.com>

Merge pull request #412 from compnerd/copy

6d24e9c

build: always copy the modulemaps Signed-off-by: Kim Topley <ktopley@apple.com>

cmake: support static and shared swift libraries

9012286

Enhance add_wift_target to support building static libraries. We would always generate shared libraries previously. Signed-off-by: Kim Topley <ktopley@apple.com>

build: split out the SDK overlay

9bc407f

This splits out the SDK overlay component and libdispatch runtime itself. Doing so enables the re-use of libdispatch with and without swift and makes the behaviour similar across Darwin and other platforms. Signed-off-by: Kim Topley <ktopley@apple.com>

build: correct dependency tracking

242e725

Signed-off-by: Kim Topley <ktopley@apple.com>

Merge pull request #401 from compnerd/split-sdk-overlay

370b38f

Split Swift SDK Overlay Signed-off-by: Kim Topley <ktopley@apple.com>

build: remove "/usr/include" for swift

7bb4a40

Remove the explicit /usr/include search path for the swift library build. This allows us to build for Windows as well. Signed-off-by: Kim Topley <ktopley@apple.com>

Merge pull request #410 from compnerd/include

516be48

build: remove "/usr/include" for swift Signed-off-by: Kim Topley <ktopley@apple.com>

SDK: guard BSD specific paths against windows

7694808

Treat windows like Linux and android and remove the BSD interfaces on these targets. This allows us to mostly build the SDK overlay for Windows. Signed-off-by: Kim Topley <ktopley@apple.com>

Merge pull request #413 from compnerd/bsdism

e570606

SDK: guard BSD specific paths against windows Signed-off-by: Kim Topley <ktopley@apple.com>

Merge pull request #415 from compnerd/stubs

0ba34b4

DispatchStubs: make more Windows friendly Signed-off-by: Kim Topley <ktopley@apple.com>

MadCoder and others added 19 commits November 5, 2021 16:33

Merge pull request #541 from buttaface/rpath

e94c8ea

Signed-off-by: Rokhini Prabhu <rokhinip@apple.com>

Merge pull request #542 from compnerd/auto-reset

252434b

dispatch: use auto-reset event for dispatch queue Signed-off-by: Rokhini Prabhu <rokhinip@apple.com>

Merge pull request #539 from compnerd/IO

9683049

IO: match the signature for `DispatchIO` with `dispatch_io_create` Signed-off-by: Rokhini Prabhu <rokhinip@apple.com>

Add modulemap for static compilation

88e7baa

When compiling statically, we have to link against DispatchStubs in addition to the other dependencies, so we are defining a separate modulemap for that. Signed-off-by: Rokhini Prabhu <rokhinip@apple.com>

Merge pull request #544 from drexin/wip-static-modulemap

753ca52

Add modulemap for static compilation Signed-off-by: Rokhini Prabhu <rokhinip@apple.com>

Merge pull request #537 from 3405691582/MissingDispatchCocoaCompat

77590c1

Add missing DISPATCH_COCOA_COMPAT preprocessor symbol. Signed-off-by: Rokhini Prabhu <rokhinip@apple.com>

Merge pull request #536 from 3405691582/NoSysctlbyname_AddOSSymbol

3df74e6

Make some preliminary porting changes. Signed-off-by: Rokhini Prabhu <rokhinip@apple.com>

docs: fix simple typo, transiton -> transition

7a97c24

There is a small typo in src/shims/lock.h. Should read `transition` rather than `transiton`. Signed-off-by: Rokhini Prabhu <rokhinip@apple.com>

Merge pull request #553 from timgates42/bugfix_typo_transition

0e37faf

docs: fix simple typo, transiton -> transition Signed-off-by: Rokhini Prabhu <rokhinip@apple.com>

Merge pull request #554 from buttaface/droid-arm

98bfc28

[android] Put in fixes for librt and armv7-a Signed-off-by: Rokhini Prabhu <rokhinip@apple.com>

Merge pull request #562 from compnerd/windows-11

1b3d2a5

shims: adjust the Windows path for Windows 11 Signed-off-by: Rokhini Prabhu <rokhinip@apple.com>

Merge libdispatch-1324.41.2

3136e15

Signed-off-by: Rokhini Prabhu <rokhinip@apple.com>

merge darwin/libdispatch-1324.41.2

4981dfd

Signed-off-by: Rokhini Prabhu <rokhinip@apple.com>

rokhinip requested a review from compnerd November 6, 2021 01:12

das reviewed Nov 6, 2021

View reviewed changes

3405691582 reviewed Nov 9, 2021

View reviewed changes

compnerd reviewed Nov 10, 2021

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Merge darwin/libdispatch-1324.41.2 to main #583

Merge darwin/libdispatch-1324.41.2 to main #583

rokhinip commented Nov 6, 2021

rokhinip commented Nov 6, 2021

das Nov 6, 2021 •

edited

Loading

3405691582 Nov 9, 2021

3405691582 Nov 9, 2021 •

edited

Loading

3405691582 commented Nov 9, 2021

compnerd left a comment

compnerd Nov 8, 2021

compnerd commented Nov 12, 2021

MadCoder commented Nov 12, 2021

compnerd commented Nov 12, 2021

das commented Nov 12, 2021

compnerd commented Nov 12, 2021

Merge darwin/libdispatch-1324.41.2 to main #583

Are you sure you want to change the base?

Merge darwin/libdispatch-1324.41.2 to main #583

Conversation

rokhinip commented Nov 6, 2021

rokhinip commented Nov 6, 2021

das Nov 6, 2021 • edited Loading

Choose a reason for hiding this comment

3405691582 Nov 9, 2021

Choose a reason for hiding this comment

3405691582 Nov 9, 2021 • edited Loading

Choose a reason for hiding this comment

3405691582 commented Nov 9, 2021

compnerd left a comment

Choose a reason for hiding this comment

compnerd Nov 8, 2021

Choose a reason for hiding this comment

compnerd commented Nov 12, 2021

MadCoder commented Nov 12, 2021

compnerd commented Nov 12, 2021

das commented Nov 12, 2021

compnerd commented Nov 12, 2021

das Nov 6, 2021 •

edited

Loading

3405691582 Nov 9, 2021 •

edited

Loading