-
Notifications
You must be signed in to change notification settings - Fork 2.3k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
gopls/doc/design: rewrite the architecture overview
Change-Id: Ice7f00c2109f999bcc3619acb979ae265caadf28 Reviewed-on: https://go-review.googlesource.com/c/tools/+/555977 Auto-Submit: Alan Donovan <adonovan@google.com> Reviewed-by: Robert Findley <rfindley@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
- Loading branch information
Showing
2 changed files
with
162 additions
and
34 deletions.
There are no files selected for viewing
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,46 +1,173 @@ | ||
| # gopls implementation documentation | ||
|
|
||
| This is not intended as a complete description of the implementation, for the most the part the package godoc, code comments and the code itself hold that. | ||
| Instead this is meant to be a guide into finding parts of the implementation, and understanding some core concepts used throughout the implementation. | ||
| # Gopls architecture | ||
|
|
||
| ## View/Session/Cache | ||
| Last major update: Jan 16 2024 | ||
|
|
||
| Throughout the code there are references to these three concepts, and they build on each other. | ||
| This doc presents a high-level overview of the structure of gopls to | ||
| help new contributors find their way. It is not intended to be a | ||
| complete description of the implementation, nor even of any key | ||
| components; for that, the package documentation (linked below) and | ||
| other comments within the code are a better guide. | ||
|
|
||
| At the base is the *Cache*. This is the level at which we hold information that is global in nature, for instance information about the file system and its contents. | ||
| The diagram below shows selected components of the gopls module and | ||
| their relationship to each other according to the Go import graph. | ||
| Tests and test infrastructure are not shown, nor are utility packages, | ||
| nor packages from the [x/tools] module. For brevity, packages are | ||
| referred to by their last segment, which is usually unambiguous. | ||
|
|
||
| Above that is the *Session*, which holds information for a connection to an editor. This layer hold things like the edited files (referred to as overlays). | ||
| The height of each blob corresponds loosely to its technical depth. | ||
| Some blocks are wide and shallow, such as [protocol], which declares | ||
| Go types for the entire LSP protocol. Others are deep, such as [cache] | ||
| and [source], as they contain a lot of dense logic and algorithms. | ||
|
|
||
| The top layer is called the *View*. This holds the configuration, and the mapping to configured packages. | ||
| <!-- Source: https://docs.google.com/drawings/d/1CK6YSLt7G3svRoZf7skJI-lxRol2VI90YOxHcYS0DP4 --> | ||
|  | ||
|
|
||
| The purpose of this layering is to allow a single editor session to have multiple views active whilst still sharing as much information as possible for efficiency. | ||
| In theory if only the View layer existed, the results would be identical, but slower and using more memory. | ||
| Starting from the bottom, we'll describe the various components. | ||
|
|
||
| ## Code location | ||
| The lowest layer defines the request and response types of the | ||
| Language Server Protocol: | ||
|
|
||
| gopls will be developed in the [x/tools] Go repository; the core packages are in [internal/lsp], and the binary and integration tests are located in [gopls]. | ||
| - The [protocol] package defines the standard protocol; it is mostly | ||
| generated mechanically from the schema definition provided by | ||
| Microsoft. | ||
| The most important type is DocumentURI, which represents a `file:` | ||
| URL that identifies a client editor document. It also provides | ||
| `Mapper`, which maps between the different coordinate systems used | ||
| for source positions: UTF-8, UTF-16, and token.Pos. | ||
|
|
||
| Below is a list of the core packages of gopls, and their primary purpose: | ||
| - The [command] package defines Gopls's non-standard commands, which | ||
| are all invoked through the `workspace/executeCommand` extension | ||
| mechanism. These commands are typically returned by the server as | ||
| continuations of Code Actions or Code Lenses; most clients do not | ||
| construct calls to them directly. | ||
|
|
||
| Package | Description | ||
| --- | --- | ||
| [gopls] | the main binary, plugins and integration tests | ||
| [internal/lsp] | the core message handling package | ||
| [internal/lsp/cache] | the cache layer | ||
| [internal/cmd] | the gopls command line layer | ||
| [internal/debug] | features to aid in debugging gopls | ||
| [internal/lsp/protocol] | the types of LSP request and response messages | ||
| [internal/lsp/source] | the core feature implementations | ||
| [internal/memoize] | a function invocation cache used to reduce the work done | ||
| [internal/jsonrpc2] | an implementation of the JSON RPC2 specification | ||
| The next layer defines a number of important and very widely used data structures: | ||
|
|
||
| [gopls]: https://github.com/golang/tools/tree/master/gopls | ||
| [internal/jsonrpc2]: https://github.com/golang/tools/tree/master/internal/jsonrpc2 | ||
| [internal/lsp]: https://github.com/golang/tools/tree/master/gopls/internal/lsp | ||
| [internal/lsp/cache]: https://github.com/golang/tools/tree/master/gopls/internal/lsp/cache | ||
| [internal/cmd]: https://github.com/golang/tools/tree/master/gopls/internal/cmd | ||
| [internal/debug]: https://github.com/golang/tools/tree/master/gopls/internal/lsp/debug | ||
| [internal/lsp/source]: https://github.com/golang/tools/tree/master/gopls/internal/lsp/source | ||
| [internal/memoize]: https://github.com/golang/tools/tree/master/internal/memoize | ||
| [internal/lsp/protocol]: https://github.com/golang/tools/tree/master/gopls/internal/lsp/protocol | ||
| [x/tools]: https://github.com/golang/tools | ||
| - The [file] package defines the primary abstractions of a client | ||
| file: its `Identity` (URI and content hash), and its `Handle` (which | ||
| additionally provides the version and content of a particular | ||
| snapshot of the file. | ||
|
|
||
| - The [parsego] package defines `File`, the parsed form of a Go source | ||
| file, including its content, syntax tree, and coordinary mappings | ||
| (Mapper and token.File). The package performs various kinds of tree | ||
| repair to work around error-recovery shortcomings of the Go parser. | ||
|
|
||
| - The [metadata] package defines `Package`, an abstraction of the | ||
| metadata of a Go package, similar to the output of `go list -json`. | ||
| Metadata is produced from [go/packages], which takes | ||
| care of invoking `go list`. (Users report that it works to some extent | ||
| with a GOPACKAGESDRIVER for Bazel, though we maintain no tests for this | ||
| scenario.) | ||
|
|
||
| The package also provides `Graph`, the complete import graph for a | ||
| workspace; each graph node is a `Package`. | ||
|
|
||
| The [settings] layer defines the data structure (effectively a large | ||
| tree) for gopls configuration options, along with its JSON encoding. | ||
|
|
||
| The [cache] layer is the largest and most complex component of gopls. | ||
| It is concerned with state management, dependency analysis, and invalidation: | ||
| the `Session` of communication with the client; | ||
| the `Folder`s that the client has opened; | ||
| the `View` of a particular workspace tree with particular build | ||
| options; | ||
| the `Snapshot` of the state of all files in the workspace after a | ||
| particular edit operation; | ||
| the contents of all files, whether saved to disk (`DiskFile`) or | ||
| edited and unsaved (`Overlay`); | ||
| the `Cache` of in-memory memoized computations, | ||
| such as parsing go.mod files or build the symbol index; | ||
| and the `Package`, which holds the results of type checking a package | ||
| from Go syntax. | ||
|
|
||
| The cache layer depends on various auxiliary packages, including: | ||
|
|
||
| - The [filecache] package, which manages gopls' persistent, transactional, | ||
| file-based key/value store. | ||
|
|
||
| - The [xrefs], [methodsets], and [typerefs] packages define algorithms | ||
| for constructing indexes of information derived from type-checking, | ||
| and for encoding and decoding these serializable indexes in the file | ||
| cache. | ||
|
|
||
| Together these packages enable the fast restart, reduced memory | ||
| consumption, and synergy across processes that were delivered by the | ||
| v0.12 redesign and described in ["Scaling gopls for the growing Go | ||
| ecosystem"](https://go.dev/blog/gopls-scalability). | ||
|
|
||
| The cache also defines gopls's [go/analysis] driver, which runs | ||
| modular analysis (similar to `go vet`) across the workspace. | ||
| Gopls also includes a number of analysis passes that are not part of vet. | ||
|
|
||
| The next layer defines four packages, each for handling files in a | ||
| particular language: | ||
| [mod] for go.mod files; | ||
| [work] for go.work files; | ||
| [template] for files in `text/template` syntax; and | ||
| [source], for files in Go itself. | ||
| This package, by far the largest, provides the main features of gopls: | ||
| navigation, analysis, and refactoring of Go code. | ||
| As most users imagine it, this package _is_ gopls. | ||
|
|
||
| The [server] package defines the LSP service implementation, with one | ||
| handler method per LSP request type. Each handler switches on the type | ||
| of the file and dispatches to one of the four language-specific | ||
| packages. | ||
|
|
||
| The [lsprpc] package connects the service interface to our [JSON RPC](jsonrpc2) | ||
| server. | ||
|
|
||
| Bear in mind that the diagram is a dependency graph, a "static" | ||
| viewpoint of the program's structure. A more dynamic viewpoint would | ||
| order the packages based on the sequence in which they are encountered | ||
| during processing of a particular request; in such a view, the bottom | ||
| layer would represent the "wire" (protocol and command), the next | ||
| layer up would hold the RPC-related packages (lsprpc and server), and | ||
| features (e.g. source, mod, work, template) would be at the top. | ||
|
|
||
| <!-- | ||
| A dynamic view would be an interesting topic for another article. | ||
| This slide deck [requires Google network] | ||
| The Life of a (gopls) Query (Oct 2021) | ||
| https://docs.google.com/presentation/d/1c8XJaIldzii-F3YvEOPWHK_MQJ_o8ua5Bct1yDa3ZlU | ||
| provides useful (if somewhat out of date) information. | ||
| --> | ||
|
|
||
| The [cmd] package defines the command-line interface of the `gopls` | ||
| command, around which gopls's main package is just a trivial wrapper. | ||
| It is usually run without arguments, causing it to start a server and | ||
| listen indefinitely. | ||
| It also provides a number of subcommands that start a server, make a | ||
| single request to it, and exit, providing traditional batch-command | ||
| access to server functionality. These subcommands are primarily | ||
| provided as a debugging aid (but see | ||
| [#63693](https://github.com/golang/go/issues/63693)). | ||
|
|
||
| [cache]: https://pkg.go.dev/golang.org/x/tools/gopls@master/internal/lsp/cache | ||
| [cmd]: https://pkg.go.dev/golang.org/x/tools/gopls@master/internal/cmd | ||
| [command]: https://pkg.go.dev/golang.org/x/tools/gopls@master/internal/lsp/command | ||
| [debug]: https://pkg.go.dev/golang.org/x/tools/gopls@master/internal/lsp/debug | ||
| [file]: https://pkg.go.dev/golang.org/x/tools/gopls@master/internal/file | ||
| [filecache]: https://pkg.go.dev/golang.org/x/tools/gopls@master/internal/filecache | ||
| [go/analysis]: https://pkg.go.dev/golang.org/x/tools@master/go/analysis | ||
| [go/packages]: https://pkg.go.dev/golang.org/x/tools@master/go/packages | ||
| [gopls]: https://pkg.go.dev/golang.org/x/tools/gopls@master | ||
| [jsonrpc2]: https://pkg.go.dev/golang.org/x/tools@master/internal/jsonrpc2 | ||
| [lsp]: https://pkg.go.dev/golang.org/x/tools/gopls@master/internal/lsp | ||
| [lsprpc]: https://pkg.go.dev/golang.org/x/tools/gopls@master/internal/lsp/lsprpc | ||
| [memoize]: https://github.com/golang/tools/tree/master/internal/memoize | ||
| [metadata]: https://pkg.go.dev/golang.org/x/tools/gopls@master/internal/cache/metadata | ||
| [methodsets]: https://pkg.go.dev/golang.org/x/tools/gopls@master/internal/lsp/cache/methodsets | ||
| [mod]: https://pkg.go.dev/golang.org/x/tools/gopls@master/internal/mod | ||
| [parsego]: https://pkg.go.dev/golang.org/x/tools/gopls@master/internal/lsp/cache/parsego | ||
| [protocol]: https://pkg.go.dev/golang.org/x/tools/gopls@master/internal/lsp/protocol | ||
| [server]: https://pkg.go.dev/golang.org/x/tools/gopls@master/internal/server | ||
| [settings]: https://pkg.go.dev/golang.org/x/tools/gopls@master/internal/settings | ||
| [source]: https://pkg.go.dev/golang.org/x/tools/gopls@master/internal/lsp/source | ||
| [template]: https://pkg.go.dev/golang.org/x/tools/gopls@master/internal/template | ||
| [typerefs]: https://pkg.go.dev/golang.org/x/tools/gopls@master/internal/lsp/cache/typerefs | ||
| [work]: https://pkg.go.dev/golang.org/x/tools/gopls@master/internal/work | ||
| [x/tools]: https://github.com/golang/tools@master | ||
| [xrefs]: https://pkg.go.dev/golang.org/x/tools/gopls@master/internal/lsp/cache/xrefs |