Skip to content

Commit

Permalink
Merge branch 'release/0.1.1'
Browse files Browse the repository at this point in the history
  • Loading branch information
evadne committed Aug 31, 2019
2 parents e810fef + b7bdb7d commit 64b1e7f
Show file tree
Hide file tree
Showing 15 changed files with 291 additions and 175 deletions.
44 changes: 36 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,11 @@

## Overview

Etso is an [ETS](http://erlang.org/doc/man/ets.html) adapter, allowing you to use [Ecto](https://hexdocs.pm/ecto/Ecto.html) schemas with ETS tables.
Etso is an [ETS][erlang-ets] adapter, allowing you to use `Ecto` schemas with ETS tables.

Within this library, a bare-bones Ecto Adapter is provided. The Adapter transparently spins up ETS tables for each Ecto Repo and Schema combination. The tables are publicly accessible to enable concurrency, and tracked by reference to ensure encapsulation. Each ETS table is spun up by a dedicated Table Server under a shared Dynamic Supervisor.

For a detailed look as to what is available, check out [Northwind Repo Test](https://github.com/evadne/etso/tree/master/test/northwind/repo_test.exs).
For a detailed look as to what is available, check out [Northwind Repo Test][northwind-repo-test].

## Highlights & Benefits

Expand All @@ -24,13 +24,25 @@ Key points to consider when adopting this library are:

The following features are working:

- Basic query scenarios (C / R / U / D): insert all, insert one, get by ID, where, delete by ID.
- Basic query scenarios (C / R / U / D)
- Insert all (`c:Ecto.Repo.insert_all/3`, etc.)
- Insert one (`c:Ecto.Repo.insert/2`, etc.)
- Get all (`c:Ecto.Repo.all/2`, etc.)
- Get by ID (`c:Ecto.Repo.get/3`, etc.)
- Get where (`Ecto.Query.where/3`, etc.)
- Delete by ID (`c:Ecto.Repo.delete/2`, etc.)
- Selects
- Assocs
- Preloads

The [Northwind Repo Test][northwind-repo-test] should give you a good idea of what’s included.

### Missing Features

The following features, for example, are missing:

- Composite primary keys
- Dynamic Repos (`c:Ecto.Repo.put_dynamic_repo/1`)
- Aggregates (dynamic / static)
- Joins
- Windows
Expand All @@ -44,33 +56,49 @@ Using Etso is a two-step process. First, include it in your application’s depe

defp deps do
[
{:etso, "~> 0.1.0"}
{:etso, "~> 0.1.1"}
]
end

Afterwards, create a new [Ecto.Repo](https://hexdocs.pm/ecto/Ecto.Repo.html), which uses `Etso.Adapter`:
Afterwards, create a new `Ecto.Repo`, which uses `Etso.Adapter`:

defmodule MyApp.Repo do
@otp_app Mix.Project.config()[:app]
use Ecto.Repo, otp_app: @otp_app, adapter: Etso.Adapter
end

Once this is done, you can use any struct against the Repo normally, as you would with any other Repo. Check out the [Northwind modules](https://github.com/evadne/etso/tree/master/test/support/northwind) for an example.
Once this is done, you can use any Schema against the Repo normally, as you would with any other Repo. Check out the [Northwind modules][northwind] for an example.

## Utilisation

Originally, Etso was created to answer the question of whether ETS and Ecto can be married together. It has since found some uses in production applications, serving as a Data Repository for pre-defined nested content. This proved invaluable for rapid iteration.

*If you have other Use Cases for this library, please add it here with a Pull Request.*

## Further Note

This repository is extracted from a prior project [ETS Playground](https://github.com/evadne/ets-playground), which was created to support my session at ElixirConf EU 2019, [*Leveraging ETS Effectively.*](https://speakerdeck.com/evadne/leveraging-ets-effectively)
This repository is extracted from a prior project [ETS Playground][evadne-ets-playground], which was created to support my session at ElixirConf EU 2019, [*Leveraging ETS Effectively.*][evadne-ets-deck]

Specifically, this library was created to illustrate the point that ETS can serve as a scalable storage layer for data which changes infrequently. Check out the [Northwind Importer][northwind-importer] for an example.

## Acknowledgements

This project contains a copy of data obtained from the Northwind database, which is owned by Microsoft. It is included for demonstration and testing purposes only, and is excluded from the distribution. The Author thanks Microsoft Corporation for the inspiration.

The Author also wishes to thank the following individuals:

- [Wojtek Mach](https://github.com/wojtekmach), for the [inspiration](https://github.com/wojtekmach/ets_ecto) regarding creation of an Ecto adapter for ETS.
- [Wojtek Mach][wojtekmach], for the [inspiration](https://github.com/wojtekmach/ets_ecto) regarding creation of an Ecto adapter for ETS.

- [Steven Holdsworth](https://github.com/holsee), for initial concept proofing and refinement.

- [Igor Kopestenski](https://github.com/laymer), for initial reviews.

- [David Schainker](https://github.com/schainks), for initial reviews, and for finding uses for this library.

[erlang-ets]: http://erlang.org/doc/man/ets.html
[northwind]: https://github.com/evadne/etso/tree/master/test/support/northwind
[northwind-importer]: https://github.com/evadne/etso/tree/master/test/support/northwind/importer.ex
[northwind-repo-test]: https://github.com/evadne/etso/blob/master/test/northwind/repo_test.exs
[evadne-ets-playground]: https://github.com/evadne/ets-playground
[evadne-ets-deck]: https://speakerdeck.com/evadne/leveraging-ets-effectively
[wojtekmach]: https://github.com/wojtekmach
1 change: 1 addition & 0 deletions dialyzer-ignore-warnings.exs
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
[]
8 changes: 7 additions & 1 deletion lib/etso.ex
Original file line number Diff line number Diff line change
@@ -1,3 +1,9 @@
defmodule Etso do
@moduledoc false
@moduledoc """
Top-level module for Etso.
"""

@type repo :: Ecto.Repo.t()
@type schema :: module()
@type table :: :ets.tab()
end
38 changes: 29 additions & 9 deletions lib/etso/adapter.ex
Original file line number Diff line number Diff line change
@@ -1,36 +1,56 @@
defmodule Etso.Adapter do
@moduledoc """
Used as an Adapter in Repo modules, which transparently spins up one ETS table for
each Schema used with the Repo, namespaced to the Repo to allow concurrent running
of multiple Repositories.
Used as an Adapter in Repo modules, which transparently spins up one ETS table for each Schema
used with the Repo, namespaced to the Repo to allow concurrent running of multiple Repositories.
The Etso Adapter implements the `Ecto.Adapter.Schema` and `Ecto.Adapter.Queryable` behaviours.
To use the Etso Adapter in your application, define a Repo like this:
defmodule MyApp.Repo do
@otp_app Mix.Project.config()[:app]
use Ecto.Repo, otp_app: @otp_app, adapter: Etso.Adapter
end
"""

@behaviour Ecto.Adapter
@behaviour Ecto.Adapter.Schema
@behaviour Ecto.Adapter.Queryable

defmacro __before_compile__(_opts), do: :ok

@doc false
def ensure_all_started(_config, _type), do: {:ok, []}

@doc false
def init(config) do
{:ok, repo} = Keyword.fetch(config, :repo)
child_spec = __MODULE__.Supervisor.child_spec(repo)
adapter_meta = %{repo: repo}
adapter_meta = %__MODULE__.Meta{repo: repo}
{:ok, child_spec, adapter_meta}
end

@doc false
def checkout(_, _, fun), do: fun.()

@doc false
def loaders(:binary_id, type), do: [Ecto.UUID, type]
def loaders(:embed_id, type), do: [Ecto.UUID, type]
def loaders(_, type), do: [type]

@doc false
def dumpers(:binary_id, type), do: [type, Ecto.UUID]
def dumpers(:embed_id, type), do: [type, Ecto.UUID]
def dumpers(_, type), do: [type]

defp get_table(adapter_meta, schema) do
__MODULE__.TableRegistry.get_table(adapter_meta.repo, schema)
end
for module <- [__MODULE__.Behaviour.Schema, __MODULE__.Behaviour.Queryable] do
for {name, arity} <- module.__info__(:functions) do
args = Enum.map(1..arity, &{:"arg_#{&1}", [], Elixir})

use __MODULE__.Behaviour.Schema
use __MODULE__.Behaviour.Queryable
@doc false
def unquote(name)(unquote_splicing(args)) do
unquote(module).unquote(name)(unquote_splicing(args))
end
end
end
end
95 changes: 47 additions & 48 deletions lib/etso/adapter/behaviour/queryable.ex
Original file line number Diff line number Diff line change
@@ -1,51 +1,50 @@
defmodule Etso.Adapter.Behaviour.Queryable do
defmacro __using__(_) do
quote do
@behaviour Ecto.Adapter.Queryable

def prepare(:all, query) do
{:nocache, query}
end

def execute(adapter_meta, _, {:nocache, query}, params, _) do
{_, schema} = query.from.source
ets_table = get_table(adapter_meta, schema)
ets_match = Etso.ETS.MatchSpecification.build(query, params)
ets_objects = :ets.select(ets_table, [ets_match])
{length(ets_objects), ets_objects}
end

def stream(adapter_meta, _, {:nocache, query}, params, options) do
{_, schema} = query.from.source
ets_table = get_table(adapter_meta, schema)
ets_match = Etso.ETS.MatchSpecification.build(query, params)
ets_limit = Keyword.get(options, :max_rows, 500)
stream_start_fun = fn -> stream_start(ets_table, ets_match, ets_limit) end
stream_next_fun = fn acc -> stream_next(acc) end
stream_after_fun = fn acc -> stream_after(ets_table, acc) end
Stream.resource(stream_start_fun, stream_next_fun, stream_after_fun)
end

defp stream_start(ets_table, ets_match, ets_limit) do
:ets.safe_fixtable(ets_table, true)
:ets.select(ets_table, [ets_match], ets_limit)
end

defp stream_next(:"$end_of_table") do
{:halt, :ok}
end

defp stream_next({ets_objects, ets_continuation}) do
{[{length(ets_objects), ets_objects}], :ets.select(ets_continuation)}
end

defp stream_after(ets_table, :ok) do
:ets.safe_fixtable(ets_table, false)
end

defp stream_after(_, acc) do
acc
end
end
@moduledoc false

alias Etso.Adapter.TableRegistry
alias Etso.ETS.MatchSpecification

def prepare(:all, query) do
{:nocache, query}
end

def execute(%{repo: repo}, _, {:nocache, query}, params, _) do
{_, schema} = query.from.source
{:ok, ets_table} = TableRegistry.get_table(repo, schema)
ets_match = MatchSpecification.build(query, params)
ets_objects = :ets.select(ets_table, [ets_match])
{length(ets_objects), ets_objects}
end

def stream(%{repo: repo}, _, {:nocache, query}, params, options) do
{_, schema} = query.from.source
{:ok, ets_table} = TableRegistry.get_table(repo, schema)
ets_match = MatchSpecification.build(query, params)
ets_limit = Keyword.get(options, :max_rows, 500)
stream_start_fun = fn -> stream_start(ets_table, ets_match, ets_limit) end
stream_next_fun = fn acc -> stream_next(acc) end
stream_after_fun = fn acc -> stream_after(ets_table, acc) end
Stream.resource(stream_start_fun, stream_next_fun, stream_after_fun)
end

defp stream_start(ets_table, ets_match, ets_limit) do
:ets.safe_fixtable(ets_table, true)
:ets.select(ets_table, [ets_match], ets_limit)
end

defp stream_next(:"$end_of_table") do
{:halt, :ok}
end

defp stream_next({ets_objects, ets_continuation}) do
{[{length(ets_objects), ets_objects}], :ets.select(ets_continuation)}
end

defp stream_after(ets_table, :ok) do
:ets.safe_fixtable(ets_table, false)
end

defp stream_after(_, acc) do
acc
end
end
103 changes: 49 additions & 54 deletions lib/etso/adapter/behaviour/schema.ex
Original file line number Diff line number Diff line change
@@ -1,58 +1,53 @@
defmodule Etso.Adapter.Behaviour.Schema do
defmacro __using__(_) do
quote do
@behaviour Ecto.Adapter.Schema

def autogenerate(:id), do: :erlang.unique_integer()
def autogenerate(:binary_id), do: Ecto.UUID.bingenerate()
def autogenerate(:embed_id), do: Ecto.UUID.bingenerate()

def insert_all(adapter_meta, schema_meta, _header, entries, on_conflict, returning, options) do
%{schema: schema, source: source} = schema_meta
ets_table = get_table(adapter_meta, schema)
ets_field_names = Etso.ETS.TableStructure.field_names(schema)
ets_changes = Etso.ETS.TableStructure.entries_to_tuples(ets_field_names, entries)
ets_result = :ets.insert_new(ets_table, ets_changes)
if ets_result, do: {length(ets_changes), nil}, else: {0, nil}
end

def insert(adapter_meta, schema_meta, fields, on_conflict, returning, options) do
%{schema: schema, source: source} = schema_meta
ets_table = get_table(adapter_meta, schema)
ets_field_names = Etso.ETS.TableStructure.field_names(schema)
ets_changes = Etso.ETS.TableStructure.fields_to_tuple(ets_field_names, fields)
ets_result = :ets.insert_new(ets_table, ets_changes)
if ets_result, do: {:ok, []}, else: {:invalid, []}
end

def update(adapter_meta, schema_meta, fields, filters, [] = returning, _options) do
%{schema: schema, source: source} = schema_meta
[key_name] = schema.__schema__(:primary_key)
[{^key_name, key}] = filters
ets_updates = build_ets_updates(schema, fields)
ets_table = get_table(adapter_meta, schema)
ets_result = :ets.update_element(ets_table, key, ets_updates)
if ets_result, do: {:ok, []}, else: {:error, :stale}
end

def delete(adapter_meta, schema_meta, filters, _options) do
%{schema: schema, source: source} = schema_meta
[key_name] = schema.__schema__(:primary_key)
[{^key_name, key}] = filters
ets_table = get_table(adapter_meta, schema)
ets_result = :ets.delete(ets_table, key)
{:ok, []}
end

defp build_ets_updates(schema, fields) do
ets_field_names = Etso.ETS.TableStructure.field_names(schema)

for {field_name, field_value} <- fields do
position_fun = fn x -> x == field_name end
position = 1 + Enum.find_index(ets_field_names, position_fun)
{position, field_value}
end
end
@moduledoc false

alias Etso.Adapter.TableRegistry
alias Etso.ETS.TableStructure

def autogenerate(:id), do: :erlang.unique_integer()
def autogenerate(:binary_id), do: Ecto.UUID.bingenerate()
def autogenerate(:embed_id), do: Ecto.UUID.bingenerate()

def insert_all(%{repo: repo}, %{schema: schema}, _, entries, _, _, _) do
{:ok, ets_table} = TableRegistry.get_table(repo, schema)
ets_field_names = TableStructure.field_names(schema)
ets_changes = TableStructure.entries_to_tuples(ets_field_names, entries)
ets_result = :ets.insert_new(ets_table, ets_changes)
if ets_result, do: {length(ets_changes), nil}, else: {0, nil}
end

def insert(%{repo: repo}, %{schema: schema}, fields, _, _, _) do
{:ok, ets_table} = TableRegistry.get_table(repo, schema)
ets_field_names = TableStructure.field_names(schema)
ets_changes = TableStructure.fields_to_tuple(ets_field_names, fields)
ets_result = :ets.insert_new(ets_table, ets_changes)
if ets_result, do: {:ok, []}, else: {:invalid, []}
end

def update(%{repo: repo}, %{schema: schema}, fields, filters, [], _) do
{:ok, ets_table} = TableRegistry.get_table(repo, schema)
[key_name] = schema.__schema__(:primary_key)
[{^key_name, key}] = filters
ets_updates = build_ets_updates(schema, fields)
ets_result = :ets.update_element(ets_table, key, ets_updates)
if ets_result, do: {:ok, []}, else: {:error, :stale}
end

def delete(%{repo: repo}, %{schema: schema}, filters, _) do
{:ok, ets_table} = TableRegistry.get_table(repo, schema)
[key_name] = schema.__schema__(:primary_key)
[{^key_name, key}] = filters
:ets.delete(ets_table, key)
{:ok, []}
end

defp build_ets_updates(schema, fields) do
ets_field_names = TableStructure.field_names(schema)

for {field_name, field_value} <- fields do
position_fun = fn x -> x == field_name end
position = 1 + Enum.find_index(ets_field_names, position_fun)
{position, field_value}
end
end
end
9 changes: 4 additions & 5 deletions lib/etso/adapter/meta.ex
Original file line number Diff line number Diff line change
@@ -1,8 +1,7 @@
defmodule Etso.Adapter.Meta do
defstruct repo: nil, meta_table_name: nil
@moduledoc false

@type t :: %__MODULE__{
repo: module(),
meta_table_name: term()
}
@type t :: %__MODULE__{repo: Ecto.Repo.t()}
@enforce_keys ~w(repo)a
defstruct repo: nil
end
Loading

0 comments on commit 64b1e7f

Please sign in to comment.