Whistling while we work

Worker threads

Addressing a long standing feature request, we have added a straight-forward facility for managing a small pool of worker threads in Core.Program.Worker. You can put work items on a queue, then have a group of threads pull items from that queue. This allows you to set a rate limit of the number of concurrent workers but otherwise taking maximum advantage of Haskell's excellent concurrency facilities.

• Introduce worker threads by @istathar in #194
• Add timeout helper function by @istathar in #191
• Re-throw exceptions when racing threads by @istathar in #197
• Enable launching child processes by @istathar in #185

Telemetry improvements

Robustness improvements continue based on experience and problems encountered in the wild. There is one behaviour change, which is that when forking a new thread the telemetry state from the parent thread is inherited, rather than cleared. This moves the API version of core-program to 0.7.0

• Ensure metrics evaluated before adding by @istathar in #184
• Retry if Honeycomb is having a transient problem by @istathar in #190
• Modify application state in-place by @istathar in #193
• Don't reset telemetry when forking a thread by @istathar in #201

Ongoing upkeep

We support GHC 9.4.x now, and have made various small changes as needed to keep the family of packages working properly. There have also been a few bug fixes in the help output subsystem and preliminary support for getting information about the Git commit you're building from at compile time.

• Support GHC 9.4.5 by @istathar in #195
• Defensive upper bound on http2 package by @istathar in #186
• Soften requirement of http2 by @istathar in #187
• Tweak help messages by @istathar in #192
• Fix argument summary in help output by @istathar in #198
• Add Git repository information to TemplateHaskell splice by @istathar in #182
• Document querying of Git repository information in fromPackage by @istathar in #183

Thanks to @justinlynn and @dmvianna for their thoughtful feedback and code review as we worked on the code leading up to this release.

Full Changelog: core-program-0.6.6.0...core-program-0.7.0.0

Refining Telemetry

Telemetry improvements

Enhancements arising from ongoing use of core-telemetry in production:

• Allow for overriding the default honeycomb host by @carlosdagos in #171
• Allow override of dataset in Honeycomb telemetry by @istathar in #180
• Render numbers properly by @istathar in #177

Webserver improvemenets

Wasn't needed by people who do secure sockets termination in the load balancers, but if you need expose a webserver directly then you need to be able to give it a certificate so it can server HTTPS:

• Support TLS when launching webservers by @istathar in #179

Bug fixes and general improvements

Always be fixing Haddock, y'know? And a obscure but significant improvement to developer experience for anyone using the Haskell REPL:

• Fix spurious timeout when using execute in GHCi by @istathar in #178
• Fix Haddock links by @istathar in #174
• Fix forward links, again by @istathar in #175

Full Changelog: core-program-0.6.5.0...core-program-0.6.6.0

Changing Things when Things need a changin'

Changing Program

The original design of core-program's configure function had you specifying the initial value of the top-level application state τ. This has proved to be unwieldly, as it seems we all want to do things like querying the options and arguments in order to compose that starting value.

A number of us simultaneously his the same question, and independently came up with the same solution: enable the developer to change their Program τ to Program υ , where υ is the new top-level application state type. The new function enabling this is changeProgram. This means you can start your program in Program None and then switch to Program Settings or Program This,That, or TheOtherThing once you've worked out which way you're going (this latter example has emerged as a way to give each different command in your program its own top-level type. Interesting!). Thanks to @guaraqe and @dmvianna for validating this use case and approving the implementation!

• Enable changing application state type in sub programs by @istathar in #168

Changing Process

Some internal surgery and re-arranging allows us to introduce execProcess_, a function which will result in an exec(2) system call to replace your running program with a new one. It's important to use this helper function (rather than calling the Posix functions in unix directly) so that you can be confident that output logging and telemetry have been sent before switching to the the new binary.

• Invoke (and replace with) external processes by @istathar in #169

While the trailing underscore convention is appropriate here, the function previously named execProcess used for running a child and reading its output was misplaced, and so we've renamed it to readProcess. Not sure what we were thinking there, but the underlying name from upstream is clearly better and so we return to presenting our wrapper in the same style.

Bug fixes and general improvements

Introduce a new pair of functions for constructing Configs that allow adding top-level precis descriptions for --help output, simpleConfig' and complexConfig', each taking an additional parameter supplying the descriptive text to be printed above the "Usage:" section. It's just cosmetic, but we've been wanting to add this for a long time; makes for a nice touch to have a short description of the program before getting into the nitty-gritty of command-line options and arguments.

• Add top-level program descriptions to help by @istathar in #170
• Use stackage lts-20.10 at GHC 9.2.5 by @justinlynn in #166
• Fix documentation by @istathar in #167

New Contributors

• @justinlynn made their first contribution in #166

Full Changelog: core-program-0.6.2.2...core-program-0.6.5.0

Experimental effects support

Getting out of an effect system once you've gotten into one

Effect systems are being actively explored as ways to structure Haskell programs in the way that numerous competing efforts were exploring the streaming I/O space eight to ten years ago. We have found ourselves in a situation where we want to use Program τ as the base monad from within a codebase that has been heavily encumbered with the effect system from the effectful package, so have introduced a new core-effect-effectful package which provides experimental support for this.

We're hopeful that this new paradigm will mature and that the community will converge on a particular approach to effects that can be standardized and made robust. In the meantime, we're frankly not convinced that attempts to replace mtl style separation of concerns are well founded because building your application as stacks of monad transformers, though widely used, turns out not to have been a good idea in the first place. In terms of ergonomics, simplicity, and ease of use, stacks of effects don't appear much better than monad transformers, but we look forward to continued progress by the many teams working on this.

In the meantime, having a top-level application state that contains the settings and resources needed to perform various actions in the external world appears more than sufficient for many applications, and so we continue to encourage interoperabiltiy via the Program τ monad provided by this library. To that end, this new package introduces a new ProgramE τ effect that allows lifting and unlifting from effectful's Eff es to core-program's Program τ, so that applications needing to use a complicated effect system can still get access top-level program utilities such as concurrency, logging, and telemetry.

Documentation for Core.Effect.Effectful is on Hackage.

• Adapter for effectful effects library by @guaraqe in #162

Bug fixes and general improvements

Add a check at startup to ensure that the program was linked against the threaded runtime (using the -threaded GHC option). It turned out that sending telemetry was failing (probably due to the network library being FFI, of course) when the program was built with the old single-threaded runtime which unfortunately is the default of you don't tell GHC to do the right thing.

This isn't really a unbeliever thing, because really it's a Haskell thing. The notion that the original single-threaded I/O manager is a good idea because it's the stable original established choice is at least a decade out of date. The default really should be -threaded so you get the parallel garbage collector, parallelism for your concurrent code, and actual benefit from the multiple-cores on any modern system. Our position is that the library should just work at all times, so we now check for it having been built with the multi-threaded runtime.

• Ensure threaded runtime present by @istathar in #164

Finally, a really nice improvement to the documentation for the Honeycomb exporter in Core.Telemetry.Honeycomb is also on Hackage.

• Add some documentation about missing Honeycomb spans by @dmvianna in #163

Full Changelog: core-program-0.6.2.1...core-program-0.6.2.2

Moar Instances

Web routing

This release previews a simple web router, allowing you to serve basic endpoints from the warp server without needing to use servant. The idea is that a straight forward term-level declaration of the routes required and the handlers for each is sufficient for many use cases. See Core.Webserver.Router for a more detailed explanation.

• Routing mechanism for webservers by @istathar in #154

Instances

Otherwise a minor release mostly about adding some new instances.

• add back linkThread for backwards compatibility by @dmvianna in #153
• Formatting by @istathar in #156
• Parse Externalizable values from environment variables by @istathar in #158
• Add MonadUnliftIO instance by @istathar in #159

New Contributors

• @dmvianna made their first contribution in #153

Full Changelog: core-program-0.6.0.1...core-program-0.6.2.1

Structured Concurrency

Structured Concurrency

In this release we have dramatically simplified and improved the internal implementation of concurrency, removing a number of long standing defects arising from inconsistencies in async. Significant thanks are owed to the team behind ki for introducing a very powerful structured concurrency model. For our purposes, bi-directional exception propagation has been our goal and we've landed it using a simplified version of their approach.

In the case where you have waitThreaded for a forked child you will now get the expected behaviour of both the child passing the exception . Here we define a "scope" as a simple Set of ThreadIds whose only purpose is to be able to cancel (kill) them if/when the scope exits. You can enter a new scope with the new createScope function. We now have the correct behaviour when racing threads together, when using ^C to interrupt the program, and when a scope containing threads exits: an exception leaving the scope will cause the scope to kill its child threads. You can wait for a single thread or a list of threads (assuming they have the same return type). The helper functions for running two threads to completion or racing two threads now work in the face of either alternative having forked child threads.

This change requires a major API version bump due to the change to the internal representation (and so a change to the return type of internal function unThread), deprecation of the linkThread function, and overall changes of behaviour. The major version is now core-program 0.6.

• Fork forever threads by @istathar in #144
• Structured concurrency re-implementation by @istathar in #152

Robustness improvements for telemetry

Considerable work has been done in recent months to increase the robustness and polish of the observability implementation used to send telemetry to Honeycomb.

• Send "meta.span_type" field with root spans by @istathar in #127
• Fix overriding span start time by @istathar in #139
• Add instances to telemetry for dates and optionals by @koolray in #142
• Fix documentation to expose Telemetry class by @istathar in #141
• Enabling clearing trace if necessary by @istathar in #143
• New Telemetry instances for UTCTime and Maybe by @istathar in #145
• Preserve service name when clearing trace state by @istathar in #146
• Instance for sending UUIDs as metric values by @istathar in #147
• More instances relating to identifiers by @istathar in #148

Formatting and parsing of external data

New adapter class Externalize used to get data in and out of canonical external textual representations.

• Support for externalizing data by @istathar in #138

Bug fixes and general improvements

• Fix global help by @istathar in #136
• Fix assignment of values to required arguments by @istathar in #137
• Fix technique used to close stdin of child process by @istathar in #140
• Remove elements from Set and Map by @istathar in #151

New Contributors

• @koolray made their first contribution in #142

Full Changelog: core-program-0.5.0.2...core-program-0.6.0.1

Time for Time

Introduce Time timestamp type

We have a common and pervasive requirement for timestamps. We have been sourcing this simple type from another package but have now relocated this code here to core-data and added a new compatibility typeclass Instant similar to the Textual, Binary etc in this family of libraries.

• Relocate timestamp type as Time by @istathar in #133

Improve exception handling

Re-exporting the usual exception handling functions was working care of the MonadCatch, MonadThrow, and MonadMask instances for the Program τ monad but was proving awkward for developers new to Haskell. Since the pattern we promote is to run pure code in Either err and then when reaching the I/O layer to continue or throw, users of this library are now offered catch, throw, etc. specialized to the Program τ monad.

• Specialize exception handling functions to Program monad by @istathar in #132

As the original re-exports have been removed from Core.System this change has resulted in a major API bump to 0.5.0

Enhance telemetry capabilities and performance.

Significant Telemetry improvements, notably adding compression and chunking so we don't overwhelm the peer observability platform when sending to them. Add support for the traceparent: header so that traces can continue when requests cross service boundaries.

• Use traceparent header to set parent trace and span by @istathar in #113
• Adjust logging of telemetry by @istathar in #117
• Prohibit empty metric field names by @istathar in #118
• Work toward allowing override of Span identifier by @TheWizardTower in #126
• Limit the rate which telemetry is read from queue by @istathar in #134
• Compress events before sending to Honeycomb by @carlosdagos in #135

Improvements to telemetry being sent by webserver layer:

• Only send query string value if present by @istathar in #129
• Don't clear metrics before running handler by @istathar in #131
• Add context to prepareRoutes by @jack-davies in #121
• When sending errors include HTTP status code by @istathar in #123
• Refine HTTP request fields in telemetry by @istathar in #128

Other Changes

• Fix argument order by @istathar in #120
• Fix Haddock links by @istathar in #122
• Fix a typo by @TheWizardTower in #124

Full Changelog: core-program-0.4.5.3...core-program-0.5.0.2

List available environment variables in help output

Added environment variables (declared with Variable in Options) to the console output if a user runs a program with --help.

Fixed the escaping of strings when serializing JSON. We hadn't noticed this for a long time, but some strange characters in Show instances tripped us up when sending telemetry to Honeycomb. Turned out to be improper escaping. Twice. It's fully conformant with the RFC spec now.

Added unconsRope (long overdue) and breakRope (which is really just a convenience wrapper around findIndexRope and splitRope). Made some related if minor performance and correctness improvements inside core-text.

What's Changed

• Output known environment variables by @istathar in #110
• Escape backslashes in addition to quotes by @istathar in #112
• Fix json escaping again by @istathar in #115
• Fix reading metadata from .cabal files by @istathar in #116

Full Changelog: core-telemetry-0.1.9.3...core-program-0.4.5.3

Fix exception handling in spans

Bug fix for encloseSpan.

What's Changed

• Ensure a span is sent if an exception is thrown by @istathar in #109

Full Changelog: core-telemetry-0.1.9.2...core-telemetry-0.1.9.3

Supress colour output if not terminal

We ran into a problem that some log aggregating services don't deal with (ie, don't strip) ANSI escape sequences resulting in quite an amount of gibberish obscuring the output. This was particularly a problem with AWS CloudWatch, so much so that we (admittedly reluctantly) have done the usual trick of detecting whether the stdout handle is a terminal and deciding whether to colour or not based on that.

Thanks to @Imogi for his help while preparing the patch that enabled this improvement!

What's Changed

• Only colour log output if attached to a terminal by @istathar in #107

Full Changelog: core-program-0.4.5.0...core-program-0.4.5.1