lnwallet: aux signer batching fixes #9074

jharveyb · 2024-09-09T11:12:29Z

Change Description

Description of change / link to associated issue.

Addresses lightninglabs/taproot-assets#1114 . Improves the safety of cancel channel usage, as the cancel signal can now be sent by tapd as well as lnd.

Steps to Test

Steps for reviewers to follow to test the change.

Unclear; not sure exactly which events led to the original user issue that caused the stack trace from lightninglabs/taproot-assets#1114 .

Pull Request Checklist

Testing

Your PR passes all CI checks.
Tests covering the positive and negative (error paths) are included.
Bug fixes contain tests triggering the bug to prevent regressions.

Code Style and Documentation

The change obeys the Code Documentation and Commenting guidelines, and lines wrap at 80.
Commits follow the Ideal Git Commit Structure.
Any new logging statements use an appropriate subsystem and logging level.
Any new lncli commands have appropriate tags in the comments for the rpc in the proto file.
There is a change description in the release notes, or [skip ci] in the commit message for small changes.

📝 Please see our Contribution Guidelines for further guidance.

coderabbitai · 2024-09-09T11:12:38Z

Important

Review skipped

Auto reviews are limited to specific labels.

Labels to auto review (1)

llm-review

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
-- I pushed a fix in commit <commit_id>, please review it.
-- Generate unit testing code for this file.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
-- @coderabbitai generate unit testing code for this file.
-- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
-- @coderabbitai generate interesting stats about this repository and render them as a table.
-- @coderabbitai read src/utils.ts and generate unit testing code.
-- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
-- @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Other keywords and placeholders

Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

lnwallet/channel.go

jharveyb · 2024-09-11T10:48:48Z

Updated to sort in the correct spot; we lose the index info when processing the job, so this sort should be earlier.

Migrating to slices.SortFunc is the recommended change:

https://pkg.go.dev/sort#Slice

And in this case, - is the direct replacement for <:

https://pkg.go.dev/slices@go1.21.4#example-SortFunc-CaseInsensitive

Will add more changes to this PR following this sketch:

lightninglabs/taproot-assets#1114 (comment)

guggero · 2024-09-11T12:01:39Z

lnwallet/channel.go

@@ -4648,7 +4648,12 @@ func (lc *LightningChannel) SignNextCommitment() (*NewCommitState, error) {
 			newCommitView.txn,
 		)
 		if err != nil {
-			close(cancelChan)
+			select {


I took another look at how exactly the cancelChan is used here and in tapd. And I think the original idea was that it would only ever be closed in SignNextCommitment(). And only directly before a return, so it was guaranteed to only happen once.
And in any other place than SignNextCommitment, we would only ever select on the channel to abort (since that would be in a goroutine).

Therefore, I don't think we need to change anything here, really. We just need to follow the same pattern in tapd and never close() the channel ourselves but instead only select on it whenever we block somewhere (e.g. when writing to sigJob.Resp.

I think the description of the Cancel channel is what got me to implement this incorrectly in tapd:

lnd/lnwallet/aux_signer.go

Line 53 in 4960ead

// abandon all pending sign jobs part of a single batch.

// Cancel is a channel that should be closed if the caller wishes to // abandon all pending sign jobs part of a single batch.

This should read something like this:

// Cancel is a channel that is closed by the caller if they wish to // abandon all pending sign jobs part of a single batch. This should // never be closed by the validator.

And maybe we should make it <-chan struct{} to disallow it being closed in the first place by the verify job?

Okay, I decided to go with the "read only channel" approach in PART3:
8a2a097

This makes it clear that the purpose of the Cancel channel is purely to listen on for the abort signal.
So the main work will need to be done in tapd, where we need to make sure that we always also listen on this channel wherever we try to send to the response channel or otherwise have a blocking operation.
That should make sure we never run into a "close of closed channel" panic, as we prevent closure in the verifier at compile time.

Those changes on the tapd since were merged with lightninglabs/taproot-assets#1118 .

lnwallet/channel.go

jharveyb · 2024-09-11T15:25:28Z

Not sure what's up with that linter error, but I observe the same error locally. Seems like imports are already properly ordered.

guggero · 2024-09-11T15:27:16Z

Not sure what's up with that linter error, but I observe the same error locally. Seems like imports are already properly ordered.

That's a false positive with the slices package. Should be fixed in master, the linter is just out of date in this branch.

guggero

LGTM 🎉

guggero · 2024-09-11T15:36:53Z

lnwallet/channel.go

@@ -4622,6 +4622,17 @@ func (lc *LightningChannel) SignNextCommitment() (*NewCommitState, error) {
 	if err != nil {
 		return nil, err
 	}
+
+	// We'll need to send over the signatures to the remote party in the


Took this commit over into part3, added this commit message:

To make sure we attempt to read the results of the sig batches in the same order they're processed, we sort them _before_ submitting them to the batch processor. Otherwise it might happen that we try to read on a result channel that was never sent on because we aborted due to an error. We also use slices.SortFunc now which doesn't use reflection and might be slightly faster.

Awesome!

I'm wondering now how we should handle these multiple lnd branches, if we want to cut a release with this included, before 0-19-staging gets fully reviewed and pulled into master.

Basically we're only keeping the 0-19-staging in order for us to be able to cut a release before the rebased versions land in master. So as long as we don't have everything in master, we'll keep adding bugfixes here and then reference this branch in litd, using the experimental suffix.
Once everything landed in master, we'll cut lnd v0.18.4-beta and a non-experimental litd release.

Ah ok, so the key insight here seems to be:

Otherwise it might happen that we try to read on a result channel that was never sent on because we aborted due to an error.

If that's the case though, why isn't the solution to start to select on the error/cancel chan here:

lnd/lnwallet/channel.go

Lines 3920 to 3925 in 750770e

// With the jobs sorted, we'll now iterate through all the responses to

// gather each of the signatures in order.

htlcSigs = make([]lnwire.Sig, 0, len(sigBatch))

for _, htlcSigJob := range sigBatch {

jobResp := <-htlcSigJob.Resp

Though I still don't see why this change is needed at all.

Note that in master, we don't sort anything here:

lnd/lnwallet/channel.go

Lines 3867 to 3874 in 750770e

sigBatch, cancelChan, err := genRemoteHtlcSigJobs(

keyRing, lc.channelState, leaseExpiry, newCommitView,

lc.leafStore,

)

if err != nil {

return nil, err

}

lc.sigPool.SubmitSignBatch(sigBatch)

This seems to have been added only in the side branch.

We don't need to sort anything here as things are already in the correct order, as we iterate over a slice which is ordered according to the htlc ID:

lnd/lnwallet/channel.go

Lines 3105 to 3111 in 750770e

// For each outgoing and incoming HTLC, if the HTLC isn't considered a

// dust output after taking into account second-level HTLC fees, then a

// sigJob will be generated and appended to the current batch.

for _, htlc := range remoteCommitView.incomingHTLCs {

if HtlcIsDust(

chanType, true, lntypes.Remote, feePerKw,

htlc.Amount.ToSatoshis(), dustLimit,

When we go to validate, we iterate over the TxOut, then use an index to locate the HTLC that maps to a given output index:

lnd/lnwallet/channel.go

Lines 4523 to 4528 in 750770e

// If this output index is found within the incoming HTLC

// index, then this means that we need to generate an HTLC

// success transaction in order to validate the signature.

case localCommitmentView.incomingHTLCIndex[outputIndex] != nil:

htlc := localCommitmentView.incomingHTLCIndex[outputIndex]

So unless we have a failing unit test to show this actually fixes something, I think we should drop it from the diff.

Ah no, I'm wrong re the above, we sort here, right before we read the response:

lnd/lnwallet/channel.go

Lines 3913 to 3918 in 750770e

// We'll need to send over the signatures to the remote party in the

// order as they appear on the commitment transaction after BIP 69

// sorting.

sort.Slice(sigBatch, func(i, j int) bool {

return sigBatch[i].OutputIndex < sigBatch[j].OutputIndex

})

With that said, I think if we're going to make this change, then we need a test to demonstrate the concrete impact. Also based on the commit message at the top of this comment thread, it seems the correct solution would be to thread through either a context or use the quit channel.

With that said, I think if we're going to make this change, then we need a test to demonstrate the concrete impact.

Fair enough; seems like the TestMaxAcceptedHTLCs test could be forked to add a fallible aux signer that causes the issue I'm describing. Working on that now.

Also based on the commit message at the top of this comment thread, it seems the correct solution would be to thread through either a context or use the quit channel.

I think that would be a helpful change, though still separate from the sorting issue I've described. AFAICT the correct quit channel to propagate would be channelLink.quit:

lnd/htlcswitch/link.go

Line 386 in fb66bd2

quit chan struct{}

Which is the only non-test caller of SignNextCommitment. I'll add that in to this PR.

A test to demonstrate the issue and validate the fix is now in the second commit. To test:

git checkout jharveyb/aux_signer_batching_fixes git checkout HEAD~3 # This test run should time out, and show that SignNextCommitment was stuck on reading the sig job response make unit pkg=lnwallet case=TestAuxSignerShutdown timeout=10s git checkout jharveyb/aux_signer_batching_fixes # This should pass make unit pkg=lnwallet case=TestAuxSignerShutdown timeout=10s

I also added the propagation of the quit channel in here as the last commit, though I'm not sure that's the best way to actually add that parameter.

I did test in the playground that passing a nil channel in the tests is safe, and that arg is not nil outside of tests. It seems like setting that quit channel when constructing a LightningChannel would be better, but also a more invasive change. Open to doing that a different way.

Roasbeef · 2024-09-12T22:26:09Z

lnwallet/channel.go

@@ -4622,6 +4622,17 @@ func (lc *LightningChannel) SignNextCommitment() (*NewCommitState, error) {
 	if err != nil {
 		return nil, err
 	}
+
+	// We'll need to send over the signatures to the remote party in the


Ah ok, so the key insight here seems to be:

Otherwise it might happen that we try to read on a result channel that was never sent on because we aborted due to an error.

If that's the case though, why isn't the solution to start to select on the error/cancel chan here:

lnd/lnwallet/channel.go

Lines 3920 to 3925 in 750770e

// With the jobs sorted, we'll now iterate through all the responses to

// gather each of the signatures in order.

htlcSigs = make([]lnwire.Sig, 0, len(sigBatch))

for _, htlcSigJob := range sigBatch {

jobResp := <-htlcSigJob.Resp

Roasbeef · 2024-09-12T22:31:35Z

lnwallet/channel.go

@@ -4622,6 +4622,17 @@ func (lc *LightningChannel) SignNextCommitment() (*NewCommitState, error) {
 	if err != nil {
 		return nil, err
 	}
+
+	// We'll need to send over the signatures to the remote party in the


Though I still don't see why this change is needed at all.

Note that in master, we don't sort anything here:

lnd/lnwallet/channel.go

Lines 3867 to 3874 in 750770e

sigBatch, cancelChan, err := genRemoteHtlcSigJobs(

keyRing, lc.channelState, leaseExpiry, newCommitView,

lc.leafStore,

)

if err != nil {

return nil, err

}

lc.sigPool.SubmitSignBatch(sigBatch)

This seems to have been added only in the side branch.

We don't need to sort anything here as things are already in the correct order, as we iterate over a slice which is ordered according to the htlc ID:

lnd/lnwallet/channel.go

Lines 3105 to 3111 in 750770e

// For each outgoing and incoming HTLC, if the HTLC isn't considered a

// dust output after taking into account second-level HTLC fees, then a

// sigJob will be generated and appended to the current batch.

for _, htlc := range remoteCommitView.incomingHTLCs {

if HtlcIsDust(

chanType, true, lntypes.Remote, feePerKw,

htlc.Amount.ToSatoshis(), dustLimit,

When we go to validate, we iterate over the TxOut, then use an index to locate the HTLC that maps to a given output index:

lnd/lnwallet/channel.go

Lines 4523 to 4528 in 750770e

// If this output index is found within the incoming HTLC

// index, then this means that we need to generate an HTLC

// success transaction in order to validate the signature.

case localCommitmentView.incomingHTLCIndex[outputIndex] != nil:

htlc := localCommitmentView.incomingHTLCIndex[outputIndex]

So unless we have a failing unit test to show this actually fixes something, I think we should drop it from the diff.

lnwallet/channel.go

In this commit, we make sig job handling when singing a next commitment non-blocking by allowing the shutdown of a channel link to prevent further waiting on sig jobs by the channel state machine. This addresses possible cases where the aux signer may be shut down via a separate quit signal, so the state machine could block indefinitely on receiving an update on a sig job.

dstadulis · 2024-09-19T15:42:29Z

Investigating if a rogue quit signal is occurring.

guggero

Nice unit test reproduction! Looks very close to me, just a couple suggestions.

guggero · 2024-09-20T07:26:23Z

lnwallet/channel_test.go

+	for i := 0; i < numHTLCs; i++ {
+		htlc, _ := createHTLC(i, htlcAmt)
+		if _, err := aliceChannel.AddHTLC(htlc, nil); err != nil {
+			t.Fatalf("unable to add htlc: %v", err)


nit: use require.NoError()?

guggero · 2024-09-20T07:30:04Z

lnwallet/channel_test.go

+	require.NoError(t, tlvStream.Encode(&sigBlobBuf))
+
+	auxSigner.On(
+		"SubmitSecondLevelSigBatch", mock.Anything, mock.Anything,


Maybe leave a comment here that this only influences the value returned from SubmitSecondLevelSigBatch but the mock jobs are still passed to the failAuxSigJob? This confused me for a second...

guggero · 2024-09-20T07:30:14Z

lnwallet/channel_test.go

@@ -3469,6 +3470,88 @@ func TestChanSyncOweCommitmentAuxSigner(t *testing.T) {
 	require.NotEmpty(t, sigMsg.CustomRecords)
 }

+// TestAuxSignerShutdown tests that the channel state machine gracefully handles
+// a failure of the aux signer when signing a new commitment.
+func TestAuxSignerShutdown(t *testing.T) {


guggero · 2024-09-20T07:35:03Z

lnwallet/channel_test.go

@@ -111,7 +111,7 @@ func testAddSettleWorkflow(t *testing.T, tweakless bool,
 	// we expect the messages to be ordered, Bob will receive the HTLC we
 	// just sent before he receives this signature, so the signature will
 	// cover the HTLC.
-	aliceNewCommit, err := aliceChannel.SignNextCommitment()
+	aliceNewCommit, err := aliceChannel.SignNextCommitment(nil)


An alternative approach would be to add WithQuitChannel to *LightningChannel, then use that. Then need to make sure we pass a parent quit chan into every call to NewLightningChannel(). So not sure if that ends up resulting in a smaller or larger diff...

guggero · 2024-09-20T07:36:49Z

lnwallet/channel_test.go

@@ -129,7 +129,7 @@ func testAddSettleWorkflow(t *testing.T, tweakless bool,
 	// This signature will cover the HTLC, since Bob will first send the
 	// revocation just created. The revocation also acks every received
 	// HTLC up to the point where Alice sent here signature.
-	bobNewCommit, err := bobChannel.SignNextCommitment()
+	bobNewCommit, err := bobChannel.SignNextCommitment(nil)


The nil looks slightly weird to me... Probably because it hides what type it should be.
We could just do make(chan struct{}) to make it a bit more explicit (but with the same behavior).
Slightly longer so might cause the diff to be even larger, so non-blocking and only if you agree.

guggero · 2024-09-20T07:41:54Z

contractcourt/chain_watcher_test.go

@@ -145,7 +145,7 @@ func TestChainWatcherRemoteUnilateralClosePendingCommit(t *testing.T) {

 	// With the HTLC added, we'll now manually initiate a state transition
 	// from Alice to Bob.
-	_, err = aliceChannel.SignNextCommitment()
+	_, err = aliceChannel.SignNextCommitment(nil)


With the previous unit test in place, should we also add a test case where the aux signer doesn't send any responses at all but we can still shut down when sending on this new quit channel?

Roasbeef reviewed Sep 9, 2024

View reviewed changes

lnwallet/channel.go Outdated Show resolved Hide resolved

lnwallet/channel.go Outdated Show resolved Hide resolved

dstadulis assigned jharveyb Sep 10, 2024

jharveyb mentioned this pull request Sep 11, 2024

tapchannel - [bug]: potential deadlock when signing second level HTLCs lightninglabs/taproot-assets#1114

Closed

jharveyb force-pushed the aux_signer_batching_fixes branch from ee97c4a to def4704 Compare September 11, 2024 10:46

guggero reviewed Sep 11, 2024

View reviewed changes

jharveyb force-pushed the aux_signer_batching_fixes branch from def4704 to d52b456 Compare September 11, 2024 14:53

guggero approved these changes Sep 11, 2024

View reviewed changes

jharveyb marked this pull request as ready for review September 11, 2024 16:54

jharveyb requested a review from Roasbeef September 11, 2024 16:54

Roasbeef requested changes Sep 12, 2024

View reviewed changes

Roasbeef reviewed Sep 12, 2024

View reviewed changes

lnwallet/channel.go Outdated Show resolved Hide resolved

jharveyb and others added 5 commits September 18, 2024 08:11

lnwallet: allow mock aux signer to fail sig jobs

fb4e6cf

lnwallet: test aux signer shutdown handling

68dadf5

lnwallet: sort sig jobs before submission

5d75f91

lnwallet: clarify usage of cancel and response channels

5801c1c

jharveyb force-pushed the aux_signer_batching_fixes branch from d52b456 to c383799 Compare September 18, 2024 13:57

guggero mentioned this pull request Sep 19, 2024

[custom channels 5/5]: merge custom channel staging branch into master #8960

Open

guggero reviewed Sep 20, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

lnwallet: aux signer batching fixes #9074

lnwallet: aux signer batching fixes #9074

jharveyb commented Sep 9, 2024

coderabbitai bot commented Sep 9, 2024 •

edited

Loading

Review skipped

Chat

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

CodeRabbit Configuration File (`.coderabbit.yaml`)

Documentation and Community

jharveyb commented Sep 11, 2024

guggero Sep 11, 2024

guggero Sep 11, 2024

jharveyb Sep 17, 2024

jharveyb commented Sep 11, 2024

guggero commented Sep 11, 2024

guggero left a comment

guggero Sep 11, 2024

jharveyb Sep 11, 2024

guggero Sep 11, 2024

Roasbeef Sep 12, 2024 •

edited

Loading

Roasbeef Sep 12, 2024

Roasbeef Sep 12, 2024

Roasbeef Sep 12, 2024

jharveyb Sep 17, 2024 •

edited

Loading

jharveyb Sep 18, 2024

jharveyb Sep 18, 2024

Roasbeef Sep 12, 2024 •

edited

Loading

Roasbeef Sep 12, 2024

dstadulis commented Sep 19, 2024

guggero left a comment

guggero Sep 20, 2024

guggero Sep 20, 2024

guggero Sep 20, 2024

guggero Sep 20, 2024

guggero Sep 20, 2024

guggero Sep 20, 2024

	// With the jobs sorted, we'll now iterate through all the responses to
	// gather each of the signatures in order.
	htlcSigs = make([]lnwire.Sig, 0, len(sigBatch))
	for _, htlcSigJob := range sigBatch {
	jobResp := <-htlcSigJob.Resp

	sigBatch, cancelChan, err := genRemoteHtlcSigJobs(
	keyRing, lc.channelState, leaseExpiry, newCommitView,
	lc.leafStore,
	)
	if err != nil {
	return nil, err
	}
	lc.sigPool.SubmitSignBatch(sigBatch)

	// For each outgoing and incoming HTLC, if the HTLC isn't considered a
	// dust output after taking into account second-level HTLC fees, then a
	// sigJob will be generated and appended to the current batch.
	for _, htlc := range remoteCommitView.incomingHTLCs {
	if HtlcIsDust(
	chanType, true, lntypes.Remote, feePerKw,
	htlc.Amount.ToSatoshis(), dustLimit,

	// If this output index is found within the incoming HTLC
	// index, then this means that we need to generate an HTLC
	// success transaction in order to validate the signature.
	case localCommitmentView.incomingHTLCIndex[outputIndex] != nil:
	htlc := localCommitmentView.incomingHTLCIndex[outputIndex]

	// We'll need to send over the signatures to the remote party in the
	// order as they appear on the commitment transaction after BIP 69
	// sorting.
	sort.Slice(sigBatch, func(i, j int) bool {
	return sigBatch[i].OutputIndex < sigBatch[j].OutputIndex
	})

lnwallet: aux signer batching fixes #9074

Are you sure you want to change the base?

lnwallet: aux signer batching fixes #9074

Conversation

jharveyb commented Sep 9, 2024

Change Description

Steps to Test

Pull Request Checklist

Testing

Code Style and Documentation

coderabbitai bot commented Sep 9, 2024 • edited Loading

Review skipped

Chat

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

CodeRabbit Configuration File (.coderabbit.yaml)

Documentation and Community

jharveyb commented Sep 11, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jharveyb commented Sep 11, 2024

guggero commented Sep 11, 2024

guggero left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Roasbeef Sep 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jharveyb Sep 17, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Roasbeef Sep 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dstadulis commented Sep 19, 2024

guggero left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coderabbitai bot commented Sep 9, 2024 •

edited

Loading

CodeRabbit Configuration File (`.coderabbit.yaml`)

Roasbeef Sep 12, 2024 •

edited

Loading

jharveyb Sep 17, 2024 •

edited

Loading

Roasbeef Sep 12, 2024 •

edited

Loading