Merge pull request #134 from GrappigPanda/issue-133

Fixes #133

README.md

@@ -33,9 +33,35 @@ intended to be part of a distributed torrent tracker network, but it is no
 longer a worthwhile pursuit, as distributed hash tables (as a part of the
 Bittorrent network) are able to be edited (upon BEP acceptance).
 
+## Running an Olivia Node
+Real quick and easy steps to run a node:
+```
+  1. git clone https://github.com/GrappigPanda/Olivia
+  2. cd Olivia
+  3. build/install_deps.sh
+  4. go build
+  5. ./Olivia
+```
+
+These 5 commands will get a **base level** node running, where the node runs,
+accepts commands, and will allow incoming connections from other Olivia nodes.
+To run a network of nodes, please see the config file and change a grand total
+of _3_ variables.
+
+An example workflow.
+```
+  $ nc 127.0.0.1 5454
+  SET key1:value1,key2:value2,key3:value3
+  :SAT key1:value1,key2:value2,key3:value3
+  GET key2
+  :GOT key2:value2
+```
+This is a really simple workflow, but it shows that setting keys/retrieving
+keys is simple. 
+
 ## Contact Maintainer
 
-[open an issue](https://github.com/GrappigPanda/notorious/issues/new)
+[open an issue](https://github.com/GrappigPanda/Olivia/issues/new)
 
 [tweet me](http://twitter.com/GrappigPanda)
 

bloomfilter/README.md

@@ -9,8 +9,22 @@ Bloom filters are used in Olivia to aide in selecting a peer to retrieve a key
 not found in the current node. So, if a key is requested from an node and the
 node doesn't contain the value, the operating node will search all known peers
 for the key. Since latency is a thing which exists, bloom filters are used to
-improve searches. 
+improve search times so we can only query nodes which **probably** have the
+key.
 
 Once a node is made aware of another node, each node will send a copy of their
-bloom filter (which will continued to be updated).
+bloom filter (which will continued to be updated). Moreover, on a timed
+interval (default of 30-seconds), a new updated bloom filter will be
+transferred between each node. 
 
+This does mean there's a 30 second window where any updates to bloom filters
+are not known by remote nodes. This is a **problem** and potential solutions 
+are being thought out. The currently courted (as of writing this) idea/solution
+is to send a list of updated hashes with each heartbeat. This allows quicker
+updating the bloom filters by simple O(1) insertions.
+
+The bloomfilter is backed by a third-party library
+(https://github.com/willf/bitset) and will continue to be so for the
+foreseeable future. To cut down on network burden, all bloom filters are
+marshalled to JSON and then run-length encoded. This tends to heavily cut down
+on total size of data being transmitted.

cache/README.md

@@ -12,6 +12,6 @@ cache and return. While I'm updating the cache, I don't need to lock, I can just
 wait until the new cache is completely constructed and then lock, change
 pointer, unlock. Yeah, I should do that.
 
-Beyond that, I want to allow key expirations. I can use my binary heap that I
-created to order by expiration time and on each heartbeat update, check the
-root node. If it's at our current time or before, we can expire the key.
+Beyond that, I want to allow key expirations. I plan on provindg key
+expirations via a Treap or my current binary heap implementation (which is
+Treap-like).

config/README.md

@@ -0,0 +1,10 @@
+## Config
+
+This is easily the least interesting part of the entire project. It just loads
+the config file found in the root directory.
+
+I'm a huge fan of the library used to do this (https://github.com/spf13/viper)
+and I essentially use it in every Go project. However, at this current
+iteration (0.1.x), I am not making full use of it. I would like to add support
+for environmental variables in the future, and will do so. 
+

dht/README.md

@@ -11,3 +11,8 @@ important nodes are set on a quick heartbeat, whereas each other node will have
 an artery clogged hearbeat every minute. Each peer operates on a FSM with
 multiple states. If a peer is continually not responding to queries, it will be
 set to a timed out state, but will continued to be beaten with our heart.
+
+Peer lists are typically used for heartbeats and peer-wide operations: for
+instance, whenever a request for a key not found in the current node is made,
+we'll iterate through each peer in the peerlist and see if that probably has
+the key.

network/README.md

@@ -11,6 +11,10 @@ Inside each goroutine, the connection will be handled by a connection
 processing finite state machine (FSM). The FSM operates in several states
 allowing varying states and processing paths in each goroutine.
 
+Upon a remote node connecting, a `REQUEST connect` command will be sent and any
+operations which are necessary will happen: currently (0.1.x) we just
+send/request bloom filters.
+
 # How the Network Layer works for outgoing connections.
 
 There's two ways. We have the ability to just send a normal command through

network/incoming/README.md

@@ -15,6 +15,7 @@ in the network/ folder.
     - Allows a remote node/client to request a bloom filter from a remote node.
   - Connect:
     - Allows a remote node/client to request a connection from a remote node.
+    - Upon acceptance, both nodes will exchange bloom filters.
   - Peers:
     - Allows a remote node/client to request a peer list from a remote node.
   - Disconnect:

network/message_handler/README.md

@@ -6,3 +6,8 @@ channels around so that I can do callbacks from (mostly) the receiver.
 Essentially what happens is we receive a response in the receiver, look up
 the hash value in the the message receiver by using a channel, and we get a
 callback channel. We can send the response through the callback channel.
+
+This allows us to send requests and _eventually_ handle the response rather
+than wait. There's a high likelihood I rip this code out in the future, but at
+the same time, it's extremely useful for sending requests which I expect to
+take an indefinite amount of time.

network/receiver/README.md

@@ -3,3 +3,5 @@
 Essentially all this does is wait for a response on a channel, look up the
 hash value in the response channel, and then call the message handler to
 handle callbacks for the response.
+
+This is the other half of the message handler.