2,508,232 events, 1,233,430 push events, 1,967,094 commit messages, 149,361,509 characters
Replace epee http client with curl-based client
In short: epee's http client is garbage, standard violating, and unreliable.
This completely removes the epee http client support and replaces it with cpr, a curl-based C++ wrapper. rpc/http_client.h wraps cpr for RPC requests specifically, but it is also usable directly.
This replacement has a number of advantages:
-
requests are considerably more reliable. The epee http client code assumes that a connection will be kept alive forever, and returns a failure if a connection is ever closed. This results in some very annoying things: for example, preparing a transaction and then waiting a long tim before confirming it will usually result in an error communication with the daemon. This is just terribly behaviour: the right thing to do on a connection failure is to resubmit the request.
-
epee's http client is broken in lots of other ways: for example, it tries throwing SSL at the port to see if it is HTTPS, but this is protocol violating and just breaks (with a several second timeout) on anything that isn't epee http server (for example, when lokid is behind a proxying server).
-
even when it isn't doing the above, the client breaks in other ways: for example, there is a comment (replaced in this PR) in the Trezor PR code that forces a connection close after every request because epee's http client doesn't do proper keep-alive request handling.
-
it seems noticeably faster to me in practical use in this PR; both simple requests (for example, when running
lokid status) and wallet<->daemon connections are faster, probably because of crappy code in epee. (I think this is also related to the throw-ssl-at-it junk above: the epee client always generates an ssl certificate during static initialization because it might need one at some point). -
significantly reduces the amount of code we have to maintain.
-
removes all the epee ssl option code: curl can handle all of that just fine.
-
removes the epee socks proxy code; curl can handle that just fine. (And can do more: it also supports using HTTP/HTTPS proxies).
-
When a cli wallet connection fails we know show why it failed (which now is an error message from curl), which could have all sorts of reasons like hostname resolution failure, bad ssl certificate, etc. Previously you just got a useless generic error that tells you nothing.
Other related changes in this PR:
-
Drops the check-for-update and download-update code. To the best of my knowledge these have never been supported in loki-core and so it didn't seem worth the trouble to convert them to use cpr for the requests.
-
Cleaned up node_rpc_proxy return values: there was an inconsistent mix of ways to return errors and how the returned strings were handled. Instead this cleans it up to return a pair<bool, val>, which (with C++17) can be transparently captured as:
auto [success, val] = node.whatever(req);
This drops the failure message string, but it was almost always set to something fairly useless (if we want to resurrect it we could easily change the first element to be a custom type with a bool operator for success, and a
.errorattribute containing some error string, but for the most part the current code wasn't doing much useful with the failure string). -
changed local detection (for automatic trusted daemon determination) to just look for localhost, and to not try to resolve anything. Trusting non-public IPs does not work well (e.g. with lokinet where all .loki addresses resolve to a local IP).
-
ssl fingerprint option is removed; this isn't supported by curl (because it is essentially just duplicating what a custom cainfo bundle does)
-
--daemon-ssl-allow-chained is removed; it wasn't a useful option (if you don't want chaining, don't specify a cainfo chain).
-
--daemon-address is now a URL instead of just host:port. (If you omit the protocol, http:// is prepended).
-
--daemon-host and --daemon-port are now deprecated and produce a warning (in simplewallet) if used; the replacement is to use --daemon-address.
-
--daemon-ssl is deprecated; specify --daemon-address=https://whatever instead.
-
the above three are now hidden from --help
-
reordered the wallet connection options to make more logical sense.
morty, what the hell were you thinking? of course we need mac, what do I have to do everything myself? - yeah, whatever Rick, you're welcome by the way.
"9:25am. I am up. Yesterday I finished the GGO novels and quite enjoyed them. Then I started playing Doom + Project Brutality again. Unleashed 2048 is a pretty good map pack. Somehow I completely missed that many weapons have alternate firing modes while playing through the entirety of Nova.
So yeah, I am really enjoying myself even though my focus should be on programming.
9:35am. Damn it, today I want to focus on just the ranges. I want to make a list of all the things that I did yesterday (because I can barely remember) and re-evaluate all the items on it.
I need proper self criticism of the code that I wrote rather that just putting it all in the shit pile.
9:40am. Once I decide what is right and not I'll start feeling better about it.
Well, for now let me just chill for a bit first.
10:15am. I should start, but let me chill for a while longer. After that I'll dedicate myself thoroughly to getting through today's challenge.
10:25am. Let me start. I need to get a little into it.
First of all - let me bring up yesterday's diff.
10:30am. Actually, I do not hate a lot of the work that I did yesterday at all.
let case_if_then_else d =
let i = col d
let inline f' keyword = range (skip_keyword keyword >>. next)
let inline f keyword = indent i (<=) (f' keyword)
(pipe4 (f' SpecIf) (f SpecThen) (many (f SpecElif .>>. f SpecThen)) (opt (f SpecElse))
(fun cond tr elifs fl ->
let f cond tr = function
| Some fl -> fst fl, RawIfThenElse(range_combine (fst cond) (fst fl),snd cond,snd tr,snd fl)
| None -> fst tr, RawIfThen(range_combine (fst cond) (fst tr),snd cond,snd tr)
let fl = List.foldBack (fun (cond,tr) fl -> f cond tr fl |> Some) elifs fl
f cond tr fl |> snd)) dI'll want to reevalutate the way I do ranges here, though this spot is not troublesome as far as I am concerned.
let case_match =
let clauses d =
let bar = bar (col d)
(optional bar >>. sepBy1 (root_pattern .>>. (skip_op "=>" >>. next)) bar
>>= fun l _ ->
match patterns_validate (List.map fst l) with
| [] -> Ok l
| e -> Error e
) d
(range (skip_keyword SpecFunction >>. clauses) |>> fun (r,l) ->
let pat_main = r, " main_arg"
RawInl(r,PatVar pat_main,RawMatch(r,RawV pat_main,l)))
<|> (range ((skip_keyword SpecMatch >>. next .>> skip_keyword SpecWith) .>>. clauses) |>> fun (a,(b,c)) -> RawMatch(a,b,c))This rewrite is 100% what is making me sick of this code. The issue is that I am lying about the ranges here. Internally, I might just go back to making RawInl acting like RawFunction. I had this before, removed it, but I should just put it back in.
10:40am. The diffing capability is really great here. It makes it a lot easier to review my code.
There were a lot of changes, but now that I am looking at it, I am fine with most of them.
let case_join_point = skip_keyword SpecJoin >>. next |>> fun x -> let r = range_of_expr x in RawInline(r,RawJoinPoint(r,x))This is a troublesome spot. I should get rid of placing RawInline during the parsing stage. Previously it was fine, but not anymore.
Let me backtrack a little. I though I hated the way ranges are handled paired symbol patterns? How did I miss that?
let symbol_paired d =
let i = col d
let next = forall
(pipe2 (range (read_symbol_paired_big .>>. next)) (many (indent (i-1) (<=) read_symbol_paired .>>. next)) (fun (r,a) b ->
let k,v = List.unzip ((to_lower (fst a), snd a) :: b)
let b = Text.StringBuilder()
List.iter (fun (x : string) -> b.Append(x).Append('_') |> ignore) k
List.reduceBack (fun a b -> RawTPair(range_combine (range_of_texpr a) (range_of_texpr b),a,b)) (RawTSymbol (r,b.ToString()) :: v))
<|> next) dThis is not even correct. Why I am setting the range of the symbol to be the expression part as well? This definitely needs re-evaluation.
No, I do not like returning only the range of the first case on an error. I am going to make these symbol paired pattern a special case just so I provide proper errors, no to mention stop lying about the ranges.
Let me go back a commit. I see that root_pattern stuff I did earlier. This is why I missed all of this.
11:05am.
let pat_symbol_paired =
(many1 (read_symbol_paired' .>>. opt root_pattern_pair) |>> fun l ->
let f ((o,a,r),b) = (o, a), Option.defaultWith (fun () -> r := SemanticTokenLegend.variable; PatVar(o,a)) b
match l |> List.map f |> List.unzip with
| (k :: k' as keys), v ->
let body keys =
let b = Text.StringBuilder()
List.iter (fun (_,x : string) -> b.Append(x).Append('_') |> ignore) keys
List.reduceBack (fun a b -> PatPair(range_combine (range_of_pattern a) (range_of_pattern b),a,b)) (PatSymbol (fst k, b.ToString()) :: v)
if Char.IsLower(snd k,0) then body keys else body ((fst k, to_lower (snd k)) :: k') |> fun x -> PatUnbox(range_of_pattern x,x)
| _ -> failwith "Compiler error: Should be at least one key in symbol_paired_process_pattern"
)
<|> pat_pairApart from using IsLower here, I am actually fine with this.
Well, no...
List.iter (fun (_,x : string) -> b.Append(x).Append('_') |> ignore) keysIn just how many places do am I doing this? I should factor this out.
11:10am. Moving on.
let symbol_paired d =
let next = operators
let i = col d
let pat = indent (i-1) (<=) read_symbol_paired' .>>. opt next
((many1 pat |>> fun l ->
let a,b = List.unzip l
let b = List.map2 (fun (o,a,r) b -> Option.defaultWith (fun () -> r := SemanticTokenLegend.variable; RawV(o,a)) b) a b
let (o,x,_),x' = match a with x::x' -> x,List.map (fun (_,x,_) -> x) x' | _ -> failwith "Compiler error: Expected `many1 pat` to produce an element."
let a = List.map (fun (_,x,_) -> x) a
let is_upper = Char.IsUpper(x, 0)
let a = if is_upper then to_lower x :: x' else a
let a =
let sb = Text.StringBuilder()
a |> List.iter (fun x -> sb.Append(x).Append('_') |> ignore)
sb.ToString()
let f a b = RawPairCreate(range_combine (range_of_expr a) (range_of_expr b),a,b)
if is_upper then RawApply(o,RawV(o,a),List.reduceBack f b)
else List.reduceBack f (RawSymbolCreate(o,a) :: b))
<|> next) dLet me paste this here. All of the symbol_paireds are questionable.
11:20am.
| ValueSome expr' -> ValueSome (process_statements (let r = range_of_expr expr in RawMatch(r,RawErrorNonUnit(r,expr),[PatE r, expr'])))I missed this is operator.
| "." -> Ok(p,a,fun (a,b) ->
let ra, rb = range_of_expr a, range_of_expr b
let r = range_combine ra rb
RawMatch(r,RawErrorNonUnit(ra,a),[PatE o,b])
)This rewrite is extremely questionable. Instead of doing it like this, I will remove RawErrorNonUnit and replace all of this with RawSeq or something like that.
11:25am.
type TopStatement =
| TopAnd of Range * TopStatement
| TopInl of Range * (VarString * RawExpr)
| TopUnion of Range * (TypeVar list * RawTExpr list)
| TopNominal of Range * (TypeVar list * RawTExpr)
| TopPrototype of Range * (VarString * VarString * TypeVar list * RawTExpr)
| TopType of Range * (VarString * TypeVar list * RawTExpr)
| TopPrototypeInstance of Range * (VarString * VarString * TypeVar list * RawExpr)I am almost done with the review. I really got lazy here. I need to do this properly.
11:25am.
let range_combine (a,_) (_,b) = a,bJust one last thing. I use this everywhere, but the name is too damn long which ruins the aesthetics of the code I've written.
let (+.) (a,_) (_,b) = a,bFor some reason the rename does not work and there are 36 occurences of this, so I'll have to rename it all by hand. Let me do it.
11:40am. Ok, done. That improved the quality of the whole file by at least a point.
It might be the case that the range_of functions have names that are too long, but I am not sure how to abbreviate them. So I'll leave this as is.
It is important that I slimmed this down.
Usually, I would do this kind of slimming right away, but yesterday I was just plowing straight on without regards for anything.
11:45am. With this I am done with the review.
11:50am.
(range (skip_keyword SpecFunction >>. clauses) |>> fun (r,l) ->
let pat_main = r, " main_arg"
RawInl(r,PatVar pat_main,RawMatch(r,RawV pat_main,l)))Let me start with this. I'll change RawInl so it takes in a list of patterns just like it had originally.
| RawInline of Range * RawExpr // Acts as a join point for the prepass specifically.Let me get rid of this while I am at it.
| RawInl of Range * (Pattern * RawExpr) listThis should be like so.
Let me make it RawFun.
11:55am. Ok, those are two issues down. I really do feel a lot better now that I am now lying around the ranges in places.
Even if what I had before would have worked that is besides the point. Once you start lying it takes very real effort to store things inside the mind. Lies are hard to keep track of. They will manifest as bugs eventually.
I want to avoid greasy, slippery codebases at all cost.
What is next?
RawMatch(r,RawErrorNonUnit(ra,a),[PatE o,b])
Let me get the easy stuff out of the way first.
| "." -> Ok(p,a,fun (a,b) ->
let ra, rb = range_of_expr a, range_of_expr b
let r = ra +. rb
RawSeq(r,a,b)
)This will now be thus.
(RawSeq(range_of_expr expr,expr,expr')))Now that I am simplifying the second case, I see that I got the range wrong originally.
| ValueSome expr' -> ValueSome (process_statements (RawSeq(range_of_expr expr +. range_of_expr expr',expr,expr')))There. I've streamlined this.
type TopStatement =
| TopAnd of Range * TopStatement
| TopInl of Range * (VarString * RawExpr)
| TopUnion of Range * (TypeVar list * RawTExpr list)
| TopNominal of Range * (TypeVar list * RawTExpr)
| TopPrototype of Range * (VarString * VarString * TypeVar list * RawTExpr)
| TopType of Range * (VarString * TypeVar list * RawTExpr)
| TopPrototypeInstance of Range * (VarString * VarString * TypeVar list * RawExpr)Now let me do this normally.
12:10pm. Ok, done with that. It was easy. I really shows how impatient I got yesterday that I did not do that then.
(case_value + case_default_value + case_var + case_join_point + case_symbol
+ case_typecase + case_match + case_typecase + case_rounds + case_record
+ case_if_then_else + case_fun + case_forall + case_unary_op) dAh, crap, I just realized something.
let op = indent (i-1) (<=) opThere was no need for this indent (i-1) thing after all.
inl f =
a + b
+ c
The statements in this form would never happen. The same goes for terms. I can afford to be permissive here.
let term = indent i (<=) next
let op = indent (i-1) (<=) opLet me get rid of this.
(pipe2 (range (read_symbol_paired_big .>>. next)) (many (indent (i-1) (<=) read_symbol_paired .>>. next)) (fun (r,a) b ->Now I am still doing this thing in symbol_paired.
12:15pm. Those 3 cases and if_then_else is next on the list. Actually, if_then_else is fine.
So that leaves symbol_paired in the 3 root functions.
Let me take a break here. I'll focus on the after that."
base: force-enable permissions hub regardless of what google says we can and can't do
:( fuck you google
Change-Id: I4343d97b462f36ff631251beda1a2910b6f2eaab Signed-off-by: Sipun Ku Mahanta sipunkumar85@gmail.com
Merge pull request #25 from IHaveThatPower/master
Revised to work with Inkscape 1.0+
(sorry for the delay, and thank you all for your patience, life has been a little crazy)
"1:30pm. Done with breakfast. Let me finish what the Melissa chapter and I'll resume what I was doing before.
https://www.youtube.com/watch?v=uh5im0rRX6U Frank Klepacki & The Tiberian Sons - Command & Conquer Remastered FULL Soundtrack
This came out recently and it is pretty good.
1:45pm. Ahhh, damnit. Let me just start.
1:50pm. Focus me, focus.
1:55pm.
(pipe2 (range (read_symbol_paired_big .>>. next)) (many (indent (i-1) (<=) read_symbol_paired .>>. next)) (fun (r,a) b ->I am thinking about the paired symbol syntax. Now that I've adjusted the operators to be indentation insensitive, I need to reevaluate this as well. Here is what I am going to do.
(pipe2 (range (read_symbol_paired_big .>>. next)) (many (indent i (<=) read_symbol_paired .>>. next)) (fun (r,a) b ->Forget that -1. I'll have it be regular.
let symbol_paired d =
let pat = indent (col d) (<=) read_symbol_paired' .>>. opt operatorsThis is the way to do it.
Ok...
2:05pm. I feel distracted. Focus me, focus. Do what is needed. This is the last stretch. Take care of the paired symbol cases.
let read_symbol_paired_big d =
try_current d <| function
| p, TokSymbolPaired(t',r) ->
if Char.IsUpper(t',0) then skip d; Ok t'
else Error [p, SymbolPairedShouldStartWithUppercase]
| p, _ -> Error [p, ExpectedSymbolPaired]Let me remove this.
2:10pm.
let read_symbol_paired d =
try_current d <| function
| p, TokSymbolPaired(t',r) -> skip d; Ok(p,t')
| p, _ -> Error [p, ExpectedSymbolPaired]
let read_symbol_paired' d =
try_current d <| function
| p, TokSymbolPaired(t',r) -> skip d; Ok(p,t',r)
| p, _ -> Error [p, ExpectedSymbolPaired]I've changed the first one.
2:20pm.
let symbol_paired d =
((many1 (indent (col d) (<=) read_symbol_paired .>>. forall) >>= fun l _ ->
match List.unzip l with
| [], _ -> failwith "Compiler error: Single item expected."
| (r,x) :: k', v ->
if Char.IsUpper(x,0) then
let k = (r,to_lower x) :: k'
let b = Text.StringBuilder()
List.iter (fun (_, x : string) -> b.Append(x).Append('_') |> ignore) k
List.reduceBack (fun a b -> RawTPair(range_of_texpr a +. range_of_texpr b,a,b)) (RawTSymbol (r,b.ToString()) :: v) |> Ok
else
Error [r, SymbolPairedShouldStartWithUppercase]
)
<|> next) dThis is not so bad, but I need to get rid of that StringBuilder part. Let me factor that part out.
let symbol_paired_concat k =
let b = Text.StringBuilder()
List.iter (fun (_, x : string) -> b.Append(x).Append('_') |> ignore) k
b.ToString()Let me do this.
2:25pm.
| (r,x) :: k', v ->
if Char.IsUpper(x,0) then
List.reduceBack (fun a b -> RawTPair(range_of_texpr a +. range_of_texpr b,a,b))
(RawTSymbol(r,symbol_paired_concat ((r,to_lower x) :: k')) :: v) |> Ok
else
Error [r, SymbolPairedShouldStartWithUppercase]...I think I am satisfied with this now.
Let me move to the next one.
let pat_symbol_paired =
(many1 (read_symbol_paired' .>>. opt root_pattern_pair) |>> fun l ->
let f ((o,a,r),b) = (o, a), Option.defaultWith (fun () -> r := SemanticTokenLegend.variable; PatVar(o,a)) b
match l |> List.map f |> List.unzip with
| (k :: k' as keys), v ->
let body keys =
let b = Text.StringBuilder()
List.iter (fun (_,x : string) -> b.Append(x).Append('_') |> ignore) keys
List.reduceBack (fun a b -> PatPair(range_of_pattern a +. range_of_pattern b,a,b)) (PatSymbol (fst k, b.ToString()) :: v)
if Char.IsLower(snd k,0) then body keys else body ((fst k, to_lower (snd k)) :: k') |> fun x -> PatUnbox(range_of_pattern x,x)
| _ -> failwith "Compiler error: Should be at least one key in symbol_paired_process_pattern"
)
<|> pat_pairHere is the pattern one. Let me redo it.
Right all of these are in different styles and it is annoying.
let pat_symbol_paired =
(many1 (read_symbol_paired' .>>. opt root_pattern_pair) |>> fun l ->
let f ((o,a,r),b) = (o, a), Option.defaultWith (fun () -> r := SemanticTokenLegend.variable; PatVar(o,a)) b
match l |> List.map f |> List.unzip with
| (r,x) :: k', v ->
List.reduceBack (fun a b -> PatPair(range_of_pattern a +. range_of_pattern b,a,b))
(PatSymbol(r,symbol_paired_concat ((r,to_lower x) :: k')) :: v)
|> fun l -> if Char.IsUpper(x,0) then PatUnbox(range_of_pattern l,l) else l
| _ -> failwith "Compiler error: Should be at least one key in symbol_paired_process_pattern"
)Much better.
Yeah, now this is good.
2:40pm. What else is there?
let symbol_paired d =
let pat = indent (col d) (<=) read_symbol_paired' .>>. opt operators
((many1 pat |>> fun l ->
let a,b = List.unzip l
let b = List.map2 (fun (o,a,r) b -> Option.defaultWith (fun () -> r := SemanticTokenLegend.variable; RawV(o,a)) b) a b
let (o,x,_),x' = match a with x::x' -> x,List.map (fun (_,x,_) -> x) x' | _ -> failwith "Compiler error: Expected `many1 pat` to produce an element."
let a = List.map (fun (_,x,_) -> x) a
let is_upper = Char.IsUpper(x, 0)
let a = if is_upper then to_lower x :: x' else a
let a =
let sb = Text.StringBuilder()
a |> List.iter (fun x -> sb.Append(x).Append('_') |> ignore)
sb.ToString()
let f a b = RawPairCreate(range_of_expr a +. range_of_expr b,a,b)
if is_upper then RawApply(o,RawV(o,a),List.reduceBack f b)
else List.reduceBack f (RawSymbolCreate(o,a) :: b))
<|> next) dHere is the value one. Now this is a mess. First, I do not need that let pat. Let me inline it.
let pat_symbol_paired =
(many1 (read_symbol_paired' .>>. opt root_pattern_pair) |>> fun l ->
let f ((o,a,r),b) = (o, a), Option.defaultWith (fun () -> r := SemanticTokenLegend.variable; PatVar(o,a)) b
match l |> List.map f |> List.unzip with
| (r,x) :: k', v ->
List.reduceBack (fun a b -> PatPair(range_of_pattern a +. range_of_pattern b,a,b))
(PatSymbol(r,symbol_paired_concat ((r,to_lower x) :: k')) :: v)
|> fun l -> if Char.IsUpper(x,0) then PatUnbox(range_of_pattern l,l) else l
| _ -> failwith "Compiler error: Should be at least one key in symbol_paired_process_pattern"
)
<|> pat_pairAh, whops, this is not supposed to be root_pattern_pair.
let pat_symbol_paired =
let next = pat_pair
(many1 (read_symbol_paired' .>>. opt next) |>> fun l ->
let f ((o,a,r),b) = (o, a), Option.defaultWith (fun () -> r := SemanticTokenLegend.variable; PatVar(o,a)) b
match l |> List.map f |> List.unzip with
| (r,x) :: k', v ->
List.reduceBack (fun a b -> PatPair(range_of_pattern a +. range_of_pattern b,a,b))
(PatSymbol(r,symbol_paired_concat ((r,to_lower x) :: k')) :: v)
|> fun l -> if Char.IsUpper(x,0) then PatUnbox(range_of_pattern l,l) else l
| _ -> failwith "Compiler error: Should be at least one key in symbol_paired_process_pattern"
)
<|> nextLet me do it like this.
2:45pm. Wow, I really had that error everywhere. That will teach me a lesson to mess with manual inlining.
Let me stop here for a bit so I can do the chores.
3:15pm. I am back. Let me resume.
let symbol_paired d =
let next = operators
((many1 (indent (col d) (<=) read_symbol_paired' .>>. opt next) |>> fun l ->
let a,b = List.unzip l
let b = List.map2 (fun (o,a,r) b -> Option.defaultWith (fun () -> r := SemanticTokenLegend.variable; RawV(o,a)) b) a b
let (o,x,_),x' = match a with x::x' -> x,List.map (fun (_,x,_) -> x) x' | _ -> failwith "Compiler error: Expected `many1 pat` to produce an element."
let a = List.map (fun (_,x,_) -> x) a
let is_upper = Char.IsUpper(x, 0)
let a = if is_upper then to_lower x :: x' else a
let a =
let sb = Text.StringBuilder()
a |> List.iter (fun x -> sb.Append(x).Append('_') |> ignore)
sb.ToString()
let f a b = RawPairCreate(range_of_expr a +. range_of_expr b,a,b)
if is_upper then RawApply(o,RawV(o,a),List.reduceBack f b)
else List.reduceBack f (RawSymbolCreate(o,a) :: b))
<|> next) dI need to redo this thing.
3:25pm.
let symbol_paired d =
let next = operators
((many1 (indent (col d) (<=) read_symbol_paired' .>>. opt next) |>> fun l ->
let f ((r,a,re),b) = (r, a), Option.defaultWith (fun () -> re := SemanticTokenLegend.variable; RawV(r,a)) b
match l |> List.map f |> List.unzip with
| (r,x) :: k', v ->
let name = r, symbol_paired_concat ((r,to_lower x) :: k')
let body = List.reduceBack (fun a b -> RawPairCreate(range_of_expr a +. range_of_expr b,a,b)) v
if Char.IsUpper(x,0) then RawApply(r +. range_of_expr body, RawV name, body)
else RawPairCreate(r +. range_of_expr body, RawSymbolCreate name, body)
| _ -> failwith "Compiler error: Should be at least one key in symbol_paired_process_pattern"
)
<|> next) dQuite a difference isn't it when I program with the wind at my back and when I just plow on. Isn't the above just so much better than the original?
Indeed, it is vastly more readable.
3:30pm. Ah, yes, there is just one thing I've yet to do before I am done. I think there is not an error message case missing.
| DuplicateVarsInRecordType l -> sprintf "Duplicate variables in record type: %s" (String.concat ", " l)3:35pm. Let me take a little breather...
I need to think of what is next.
Actually, in the absence of anything better, let me run the thing and try it. Maybe my redesign knocked something loose.
Apart from that I think that now I trully am well and done with the parser. The next step would be to actually move to typechecking. Now that I have ranges in the parsed AST, there is nothing at all stopping me from moving to the next step. Let me commit here."
Created Text For URL [www.nation.co.ke/kenya/life-and-style/dn2/i-d-like-to-marry-my-girlfriend-ex-boyfriend-hindrance-1906402]
Fix and nerf hygienebots (#51307)
About The Pull Request
This is a three-part PR:
Firstly, it fixes bugs for hygienebot construction. There were two present that mostly cancelled each other out: first, temporarilyRemoveItemFromInventory expects that the item passed to it will be deleted shortly after. It removes the item from mob's hands but keeps loc in the mob. Hygienebots are using a stack in their construction, so the assumption was not correct. The tubing stack appeared deleted (it was actually hidden). Second, do_after was written wrongly and would allow people to build hygienebots without consuming any tubing, if not for the first bug removing it. Their sprites were also moved to other bots.
Secondly, I added a proximity sensor into construction. I know bots are powered by magic and coder tears, but building it out of two metal sheets, a hole, and a tube is ridiculous even for those standards. Also literally every single bot uses proximity sensors. This change will make them less spammable, because I've seen upwards of 10 being build for the funzies.
Thirdly, I nerfed the hygienebots themselves. First, they no longer start slipping everything in their path when they're chasing someone for 5 seconds. This is stupid, no other bot is as disruptive as that, it doesn't make sense to give them slipping everything in their path as a part of normal operation. Combined with hygienebot spam, this was just shitty all-around. They still slip the tile they clean, but no more than that. Second, I made them less spammy; instead of a flat 60% chance to yell at people, it slowly increases to that number. I removed the vasectomy line (WTF?). Finally, they will now drop their target if it gets too far; previously they'd chase it forever.
Fixes #50159 Fixes #50339 Why It's Good For The Game
It requires about 0.7 brain cells and autolathe access to make slipper go brr, and there's no abundance of greyshirts in bloody shoes running around and turning station's hallways into a mass slip'n'slide. Changelog
🆑 fix: Hygienebot construction has been fixed. balance: Hygienebots now require a proximity sensor between welding and tubing. balance: Hygienebots no longer slip their path while chasing someone. balance: Hygienebots are less spammy. /🆑
Updating: 7/27/2020 5:00:00 AM
- Added: An alleged Nintendo leak has unearthed early game prototypes | VGC (https://www.videogameschronicle.com/news/an-alleged-nintendo-leak-has-reportedly-unearthed-early-game-prototypes/)
- Added: Exploring the Worker Thread API in Node (https://www.telerik.com/blogs/exploring-the-worker-thread-api-in-node)
- Added: Sponsor @chrissainty on GitHub Sponsors (https://github.com/sponsors/chrissainty?sc=t&sp=sfmskywalker)
- Added: Relâchement des gestes barrière chez les jeunes : «Le pire qu’on aura, c’est une grosse grippe» (https://www.leparisien.fr/societe/relachement-des-gestes-barriere-chez-les-jeunes-le-pire-qu-on-aura-c-est-une-grosse-grippe-26-07-2020-8358786.php)
- Added: Wicked problems (https://no-kill-switch.ghost.io/wicked-problems/)
- Added: Primer: The Job Guarantee (https://seekingalpha.com/article/4360842-primer-job-guarantee)
- Added: Tech unicorn Dave admits to security breach impacting 7.5 million users | ZDNet (https://www.zdnet.com/article/tech-unicorn-dave-admits-to-security-breach-impacting-7-5-million-users/)
- Added: Pixelles Creator Fund | Pixelles (Montreal) (http://pixelles.ca/pixelles-creator-fund/)
- Added: Lets stop being stupid about security (http://www.adambourg.com/security/sql/2020/07/26/Lets-stop-being-stupid-about-security.html)
- Added: Finding Joy in Making Happy Little Computer Videos on YouTube (https://www.hanselman.com/blog/FindingJoyInMakingHappyLittleComputerVideosOnYouTube.aspx)
- Added: What’s the point? (https://kangabru.xyz/2020/07/25/whats-the-point)
- Added: Developers Should Abandon Agile (https://ronjeffries.com/articles/018-01ff/abandon-1/)
- Added: How C# Records will change my life (https://josef.codes/how-csharp-records-will-change-my-life/)
- Added: New ‘Meow’ attack has deleted almost 4,000 unsecured databases (https://www.bleepingcomputer.com/news/security/new-meow-attack-has-deleted-almost-4-000-unsecured-databases/)
- Added: Small Mailserver Best Current Practices (https://bridge.grumpy-troll.org/2020/07/small-mailserver-bcp/)
- Added: Understanding the SynchronizationContext in .NET with C# (https://www.codeproject.com/Articles/5274751/Understanding-the-SynchronizationContext-in-NET-wi)
- Added: Create Your Own Logging Provider to Log to Text Files in .NET Core (https://www.roundthecode.com/dotnet/create-your-own-logging-provider-to-log-to-text-files-in-net-core)
- Added: Compilers Do Static Analysis, They Just Don't Tell You · Peter Dimov (https://pdimov.github.io/blog/2020/07/24/compilers-do-static-analysis/)
- Added: Virtual Azure Community Day - July 28th, 2020 (https://azureday.community/)
- Added: Amazon gets priority while mail gets delayed, say letter carriers (https://www.pressherald.com/2020/07/21/first-class-and-priority-mail-delayed-in-favor-of-amazon-parcels-according-to-portland-letter-carriers/)
- Added: Writing more succinct C# – in F# (Part 1)! (https://www.compositional-it.com/news-blog/writing-more-succinct-c-in-f-part-1/)
- Added: S&P Best of Times Wost of Times (https://www.vizlit.com/finance/2020/07/01/sp-performance.html)
- Added: Twilio Security Incident Shows Danger of Misconfigured S3 Buckets (https://www.darkreading.com/cloud/twilio-security-incident-shows-danger-of-misconfigured-s3-buckets/d/d-id/1338447)
- Added: Naming things (https://kodare.net/2020/07/26/naming-things.html)
- Added: Reverse Engineering Tools Review (https://www.pelock.com/articles/reverse-engineering-tools-review)
- Added: Super Troopers (2/5) Movie CLIP - The Cat Game (2001) HD (https://www.youtube.com/watch?v=1rlSjdnAKY4)
- Added: Tip 272 - Azure Security Best Practices (https://microsoft.github.io/AzureTipsAndTricks/blog/tip272.html)
- Added: LunrCore, a lightweight search library for .NET (https://weblogs.asp.net/bleroy/lunrcore)
- Added: Fresh Mozzarella From Scratch Recipe (https://www.seriouseats.com/recipes/2015/10/print/how-to-make-fresh-mozzarella-from-scratch-recipe.html)
- Added: Copy/Pasta Driven Development® (https://blog.grahamyooll.com/2020/07/26/copy-pasta-driven-development.html)
- Added: andrewducker | Facebook has blocked Dreamwidth (https://andrewducker.dreamwidth.org/3861716.html)
- Added: Slowsort - a pessimal sorting algorithm | Arpit Bhayani (https://arpitbhayani.me/blogs/slowsort)
- Added: Tyrrrz/CliFx (https://github.com/Tyrrrz/CliFx)
- Added: It’s Time to Get Serious About Research Fraud (https://undark.org/2020/07/23/cracking-down-on-research-fraud/)
- Added: BenchmarkDotNet 0.12.1 (https://www.nuget.org/packages/BenchmarkDotNet)
- Added: microsoft/vscode (https://github.com/microsoft/vscode/wiki/Performance-Issues)
- Added: Headphones are collecting too much personal data (https://www.soundguys.com/headphones-are-collecting-too-much-personal-data-21524/)
- Added: Merging migrations in Entity Framework Core 5 – Michał Białecki Blog (http://www.michalbialecki.com/2020/07/24/merging-migrations-in-entity-framework-core-5/)
- Added: Cosmos DB capacity pitfall: When more is less (https://mijailovic.net/2020/07/25/cosmosdb-throughput/)
- Added: Index (https://ebookfoundation.github.io/free-programming-books/free-programming-books.html)
rosy: dts: Overlay changes
With overlay:
- All stock .dts files are compilable.
- Rosy's code is isolated from other .dtsi files.
Note that, to successfully overlay rosy's changes specifically, we have to use an ugly hack. It involves predefining essential nodes for msm8953 dtsi (to evade "node not found" errors), and also involves deleting a node that conflicts with msm-pmi8940. The true fix would be CAF fixing their own shit to not conflict when some specific headers are included. This patch improves 78f9bfa0b4e14b2f050b4675f0ff8aff22aad585.
Signed-off-by: Dimitar Yurukov mscalindt@protonmail.com
unfold_invented/3 will now preserve theory constants.
- Previously, unfold_invented/3 would variabilise constants in a theory. This was the result of calling lift_program/2 in library lifting to variabilise an unfolded program. lift_program/2 did not make any provisions for lifting theories with constants so it would just indiscriminately variabilise every term in each literal in a program's clauses. Since unfold_invented/3 operates on encapsulated programs, this would also variabilise predicate symbols in encapsulated literals. To get around this, unfold_invented/3 first excapsulated an unfolded program, then lifted it, then encapsulated it again (so it could be processed correctly by the rest of dynamic_learning/5). This was bloody awful and this commit corrects the awfulness.
- As part of the changes for this purpose, lifted_program/2 in library lifting is now an alias for lifted_program(Ps,Cs,Ls) where Cs is a new argument that holds a set of theory constants that must not be variabilised.
- unfold_invented/3 now doesn't need to excapsulate and re-encapsulate a program. So it doesn't do that. Not anymore. No more.
- Constants in an encapsulated program, to be passed to lift_program/3, are collected by a new program, encapsulated_signature/2. This looks butt ugly but it works and it's not too slow. Still, unfold_invented/3 is not the most efficient way to do this work and it sure is convoluted and difficult to understand, if someone saw it that wasn't me. I've added a metric shitton of documentation that should hopefully help users orientate themselves correctly in the logic of the predicate.
PROJECT REPORT
1 SAFE DRIVING CHALLENGE ML Project Report BACHELOR OF TECHNOLOGY IN COMPUTER SCIENCE & ENGINEERING SUBMITTED BY NAME OF THE STUDENT ROLL NO Ms. M SUSMITHA 17WH1A0591 Ms. M NAGA PRAVALLIKA 18WH5A0517 Ms. T BINDHU BHARGAVI 18WH5A0520 Department of Computer Science and Engineering BVRIT HYDERABAD College of Engineering for Women (Approved by AICTE, New Delhi and Affiliated to JNTUH, Hyderabad) Bachupally, Hyderabad – 500090 2 Department of Computer Science and Engineering BVRIT HYDERABAD College of Engineering for Women (Approved by AICTE, New Delhi and Affiliated to JNTUH, Hyderabad) Bachupally, Hyderabad – 500090 Acknowledgement Firstly, I would like to express my immense gratitude towards BVRIT HYDERABAD College of Engineering for Women, which created a great platform to attain profound technical skills in the field of Computer Science though this industry enabled learning WISE. I would like to extend my sincere thanks and gratitude to Dr. K V N Sunitha, Principal, BVRIT HYDERABAD College of Engineering for Women and WISE team of college for their meticulous planning and conduction of this learning program. I would also like to extend my sincere thanks to WISE & Team of Talentsprint for enabling us with this unique learning platform. M Susmitha(17WH1A0591) M Naga Pravallika(18WH5A0517) T Bindhu Bhargavi(18WH5A0520) 3 INDEX S.NO Contents Page No. 1 Abstract 4 2 Introduction 5 3 Problem statement 6 4 Approach and Statistics of code 7 5 Data Sets 8-9 6 First Model 10 7 Feature Engineering 11 8 PCA 12-13 9 Neural Network 14-15 10 Second Model 16 11 Random forest and Naïve Bayes 17 12 Comparisons of Models 18-20 13 Result 21 14 Reference Link and Project Link 22 LIST OF FIGURES S.NO Name of the figure Page No. 1 Screen plot of 30 features 12 2 Histogram of mean alertness per trail 15 3 ROC curve of two models 20 4 ABSTRACT In this project we introduce a classifier which takes in multidimensional data consisting of realworld measurements of physical, environmental and vehicular continuous features obtained from number of driving sessions. We will show that using Naive Bayes classifier which assumes the data distribution to be Gaussian distribution we can make a prediction weather the driver is alerted or not while driving and achieve reasonable low misclassification rate for the given data. We will inspect how insight into relevant features were obtain by using Principal Component Analysis (PCA) and simple correlation matrix. We were able to obtain a misclassification rate as low as 12.03 % and 27.07 % for the test and training data respectively. 5 INTRODUCTION With a training and test set consisting of 33 features from real time measurements test we want to use that information to predict if a certain driver is alerted or not alerted while driving. Here our goal is to construct a binary classifier which will predict a binary target value using the whole or a subset of the 33 features and give a prediction as Predictions = ( 1 if the driver is alert 0 if the driver is not alert (1) A. Datasets The datasets are gained from the website www.kaggle.com and consist of one training set and one test set. The datasets include measurements from total of 510 real time driving session where each driving session takes 2 minutes. This gives a new measurement of the each of the 33 features every 100ms. The headers in the datasets are listed in table I below. The size of the training set is a measurement set of 510 driving sessions done by 100 people. This results in a 604330×33 as the size of the training set. The test data have fewer observations and has 6 Problem Statement Driving while distracted, fatigued or drowsy may lead to accidents. Activities that divert the driver's attention from the road ahead, such as engaging in a conversation with other passengers in the car, making or receiving phone calls, sending or receiving text messages, eating while driving or events outside the car may cause driver distraction. Fatigue and drowsiness can result from driving long hours or from lack of sleep. The data for this Kaggle challenge shows the results of a number of "trials", each one representing about 2 minutes of sequential data that are recorded every 100ms during a driving session on the road or in a driving simulator. The trials are samples from some 100 drivers of both genders, and of different ages and ethnic backgrounds. The files are structured as follows: The first column is the Trial ID - each period of around 2 minutes of sequential data has a unique trial ID. For instance, the first 1210 observations represent sequential observations every 100ms, and therefore all have the same trial ID The second column is the observation number - this is a sequentially increasing number within one trial ID The third column has a value X for each row where X = 1 if the driver is alert X = 0 if the driver is not alert The next 8 columns with headers P1, P2 , …….., P8 represent physiological data; The next 11 columns with headers E1, E2, …….., E11 represent environmental data; The next 11 columns with headers V1, V2, …….., V11 represent vehicular data; 7 APPROACH • Initially, we have analyzed train and test datasets • Imported the required libraries • By using Data preprocessing, logistic regression, feature engineering, PCA, Support vector regression, Neural network we have predicted the output. STATISTICS OF THE CODE • We have used google Collaboratory to predict the output. 8 SAFE DRIVING CHALLENGE 1 INTRODUCTION The objective is to design a classifier that will detect whether the driver is alert or not alert, employing data that are acquired while driving. This report is meant to illustrate the process of building a predictive machine learning model of the Machine Learning. 2 DATASETS There are 604,329 instances of data in the training dataset and 120,840 instances of data in the test dataset. The data for this challenge shows the results of a number of” trials”, each one representing about 2 minutes of sequential data that are recorded every 100ms during a driving session on the road or in a driving simulator. The trials are samples from some 100 drivers of both genders, and of different ages and ethnic backgrounds. The files are structured as follows: The training data was broken into 500 trials, each trial consisted of a sequence of approximately 1200 measurements spaced by 0.1 seconds. Each measurement consisted of 30 features; these features were presented in three sets: physiological (P1...P8), environmental (E1...E11) and vehicular (V1...V11). Each feature was presented as a real number. For each measurement we were also told whether the driver was alert or not at that time (a Boolean label called Is Alert). No more information on the features was available. 3 EXISTED MODEL In order to summarize existed work and formulate a plan in order to build an outperformed machine learning predictive model. Similar machine learning techniques are applied to this dataset. The techniques most participants used limited to Nave Bayes, Logistic Regression, Support Vector Machine, Neural Network, and Random Forest. But the performances oftheirmodels are totally different, asthey preprocessedtheoriginaldata in differentways, especially in their feature engineering. Thus, I will mainly focus on the feature engineering methods applied by the participants, instead of how they choose parameters of algorithms in the summary part. The highest score (AUC = 0.861151) was reached by a logistic regression model. As the dataset consists of sequential data recorded every 100ms for 2 minutes in each trial, the partitions of the data by trials (Trial ID) rather than randomly partition. The Means and Standard Deviations of each trial were computed as new features (include the target feature Is Alert). After- wards, feature selection based on diagnostics of the logistic regression was conducted and three strong features were chosen for modelling (sdE5, V11, and E9). How- ever, this model applies future observation (The mean and standard deviation can only be calculated when a trial is finished), thus inapplicable for real-life situations. A running Mean and Standard deviation were applied to training instead and the AUC has dropped slightly, from 0.861151 to 0.849245). We focus on the instances at the initial moment the driver lost alertness, the dataset is reduced significantly in this way and he highlighted the factors change significantly between status change forfeature 9 selection. E4, E5, E6, E7, E8, E9, E10, P6, V4, V6, V10, and V11 are selected for building a Neural Network. This model reaches an AUC of 0.84953 & also attempts to aggregate data from each trial and calculate means and standard deviations as additional features. After tossing up correlated feature and other feature engineering, a logistic regression model trained from feature selected data reaches an AUC of 0.80779. Fourier generates around 600 new features to the dataset (The inverse, the square, and the cube of each features, all the combinations of 2 columns, time interval variables). It reaches the highest AUC by applying forward search to select predictive features. A Nave Bayes model trained by these selected features reach an AUC of 0.844. We trained an epsilonSVR, RBF kernel model with parameters c = 2, g = 1/30, and p = 0.1, which reaches an AUC of 0.839 and applies a random forest with 199 trees and min node size of 25, the correlated features are tossed out beforehand. This predictive model reaches an AUC of 0.81410. 3.1 SUMMARY OF EXISTED MODEL An important feature for this dataset is that it contains sequential data. For each trial, the dataset records data every 100ms. Thus, all the participants shuffle the dataset by trials for the purpose of preserving this sequential feature. Aggregating data within a trial to generate means and standard deviations as new features for modelling is proofed as a useful method of data preprocessing. Another useful method of data preprocessing is to choose the instances close to the moment the driver lost alertness, which re- duce time to train the models significantly. Multiple methods of feature selection are applied, the mean/standard deviation of existed features, inverse, the square, the cube, and a combination of 2 columns are viewed as potentially useful new features. Correlated, remain constant features are always tossed out. As for the choice of predictive machine learning algorithms, there is no valid proof that one algorithm outperforms all the others in this specific situation. Generally, Nave Bayes, Logistic Regression, Random Forest, Support Vector Machine, and Neural Network all reach a good performance in this case. 4 MODEL BUILDING PLANS Even though many existed models have already had a decent performance, it’s still possible to improve the model. A plan for building a new predictive model is outlined in this section. 4.1 GAP IDENTFICATION The predictive model with the highest AUC value is trained from 20% of the training dataset. What’s more, the means and standard deviations of each trial are future observation features. Those make this predictive model inapplicable to a real-life situation. An AUC value of 0.861151 also means there are still rooms for improvement. Another noticeable point within most of the existed work is that most of the models are evaluated by either AUC score or classification accuracy. For this specific situation, it’s obviously more important to identify those not alert instances as driving while not alert can be deadly. Failing to identify ’not alert’ can lead to worse consequences compare to failing to identify ’alert’. Thus, true negative rate (TN / (TN + FP)) can also be a valuable measure of evaluation as it shows the percentage of ’not alert’ instances successfully identified. Furthermore, as all the models’ classification accuracies are above 50%, which makes building an ensemble model to reach a better performance possible as if the recalls 10 and the specificities of all the models can reach above 50% at the same time for all the models. 4.2MODEL BUILDING PLAN Firstly, those existed models with good performance will be reproduced, includes the way they preprocessthe data and the parameters they choose to build predictive models. Secondly, a local evaluation will be conducted on these models. The recalls and specificity will be used for evaluation, apart from classification accuracy and AUC score. Then to group those models with recall and specificity both higher than 50% to build an ensemble model, aims to reach a better performance than all the existed models. 5 SOLUTION DEVELOPMENT Python is used as the developing environment for this project. Scikit-learn is the machine learning tool applied. Missing data is identified as 0 throughout all the dataset. 5.1 FIRST MODEL The first predictive model is built by the data preprocessing method. We were concerned that using the entire data set would create too much noise and lead to inaccuracies in the model. The final goal of the system is to detect the change in the driver from alert to not alert so that the car can self-correct or alert the driver. So, we decided to just focus on the data at the initial moment when the driver lost alertness. According to this, I subset the dataset to the moment when the driver lost alertness. The rows with the feature ’Is Alert’ == 0 and the last rows with the feature ’Is Alert’== 0 are chosen, along with 5 rows before and after each (100ms of time between each observation, 5 rows before means focus on the data recorded 0.5s before and after the driver lost alertness).After sub setting, 37421 instances without duplication are chosen to build the predictive model. 11 5.1.1 FEATURE ENGINEERING There are 30 features included in the dataset, thus filter those features with higher impact could not only save computational resources but also potentially improve the performance of the predictive model. Principle Component Analysis(PCA) is applied asthe feature engineering technique in this case. For PCA, the dataset is standardized firstly, then the fraction of variances of each feature is calculated to identify those features have higher impact on the result. 12 図 1: Scree Plot of 30 features We can see that the first 14 attributes contribute 80.95% of the total variance, the number of features selected for modelling is decreased from 30 to 14 in this way. 13 5.1.2 MODELLING As the size of subset is relatively smaller, stratified 10-fold cross-validation is applied as data evaluation method to make full use of the dataset. Naive Bayes, Logistic Regression, Random Forest, Support Vector Machine, and Neural Network models are built from this dataset. Gaussian Nave Bayes model performs a validation ac- curacy of 61.74%. Logistic Regression with optimization algorithm of ’liblinear’ reaches a validation accuracy of 64.6%. Multiple models in Support Vector Machine family include Linear-SVC, Nu-SVC, C-SVC are applied as well. Their validation accuracies varied from 65% to 78% A Neural Network with 5 neurons in the first hidden layer and 2 neuronswiththe second hidden layersreaches a validation accuracy of 65.01%. I tried to use neural networks with different architectures, another neural network with 5 hidden layers and 14, 14, 12, 10, 5 neurons in each layer. The activation function is also changed from RELU to logistic regression. Unfortunately, the performance of the new neural network does not change mach. 14 15 Model Accuracy recall Specificity AUC Sc Logistic Regression 64.60% 94.32% 25.25% 0.5978 Nave Bayes 61.74% 89.47% 25.02% 0.5724 Random Forest 93.54% 97% 90.08% 0.9354 Linear-SVC 64.55% 94.85% 24.44% 0.5958 Nu-SVC 78.63% 92.36% 60.45% 0.7640 C-SVC 67.55% 98.13% 27.06% 0.626 Neural Network1 65.01% 94.31% 26.21% 0.6026 Neural Network2 65.96% 97.02% 24.83% 0.6092 Performances of algorithm on PCA dataset It can be found from the performance diagram that all the models perform pretty well on predicting those drivers ’in alert’. However, most models cannot reach a decent result when it comes to identifying drivers not in alert, which is more important in this specific situation. On the other hand, the Random Forest model outer form all other models, especially when it comesto specificity, which makesit a part of our final ensemble model. The Nu-SVC model reaches a specificity of more than 50% as well, which means it can also be part of an ensemble model. 5.2 SECOND MODEL Unfortunately, we did not get a good predictive ma- chinelearningmodel bythefirstdata preprocessing method (apart from the Random Forest with 50 trees model). I decided to conduct an exploratory analysis on the dataset in order to provide a guidance of data preprocessing 5.2.1 EXPLORATORY ANALYSIS AND FEATURE ENGINEERING ON THE DATASET We calculate the average Is Alert value per trial and plot the result on a histogram. 図 2: Histogram of mean alertness per trial 16 It is found that for most drivers, they either stay alert or not alert throughout the 1200ms trial. Thus, the characteristic of each driver, recorded in the mean and standard deviation of each attribute, can be helpful for predictive analysis. On the other hand, it is impossible to get the mean and standard deviation of a trial at the beginning of each trial, which makes using stable means and standard deviations of each feature unpractical in real-life situation. More- over, using stable means and standard deviations cannot record the change of the driver’s behavior within a trial, which may be constantly changing overtime. For these reasons, we decided to use rolling means and standard deviations of each features as new features in- stead of simply using stable means and standard deviations in order to make full use of the sequential feature. The rolling window is set to 5, as for every 5 instances (500ms), it calculates the mean and standard deviation for them, then the algorithm drops the first instance and add a new instance, etc. 5.2.2 MODELLING Similarly, we applied algorithms mentioned above to this preprocessed dataset. As the size of the dataset is big enough, we use 80%-20% to train-test split the dataset in- stead of crossvalidation. Firstly, we tried Random Forest algorithm, the one per- forms the best in the last feature selected dataset, to see if there’s any improvement compare to the other feature selecting method. The Random Forest has 50 trees, the parameters are the same as the one applied before. It reaches a decent performance on the validation dataset, with a validation accuracy of 98.91%. Algorithms of the Support Vector Machine family all fail to converge within a specific period of time. A neural network with four hidden layers, each layer has 90, 70, 50, 30 neurons respectively also applied, reaches a validation accuracy of 80.76%. Furthermore, Nave Bayes and Logistic Regression have not improved much compare to the preview models. Generally, Neural Network and Random Forest performs better than other models in this situation, and Random Forest performsfar betterthanNeuralNetworks. 17 18 Model Accuracy recall Specificity AUC Sc Logistic Regression 61.21% 75.21% 41.96% 0.5858 Nave Bayes 62.86% 45.28% 87.02% 0.6615 Random Forest 98.91% 98.58% 97.55% 0.9873 Neural Network 80.76% 96.40% 59.26% 0.7783 Performances of algorithm on rolling mean std dataset 5.2.1COMPARISON OF MODELS Comparing the performances of models trained from data preprocessed by different methods, it is found that algorithms logistic regression, Support Vector Machine, and nave Bayes are not suitable for this problem. While Neural Network can reach a good performance in the dataset preprocessed by generating time sequential feature, it is not the model fits the dataset the best. The Random Forest Algorithm generates the best result on predictive analysis, either trained from data preprocessed by PCA or from data preprocessed by other feature engineering techniques. Another interesting finding isthat most models perform better when it comes to predicting ’alert’ drivers than to predicting ’not alert’ drivers, apartfromthe NaveBayesmodel. Considering two values are basically equally distributed (alert: 349785, not alert: 254544), it’s hard to say one label is over represented than the other, which makes the unbalanced predict result hard to explain. 19 As a result, I choose three models for local evaluation, which are two Random Forest models and a Neural Net- work Model. 6 Local Evaluation We use the data’solution.csv’ to evaluate the final models. 6.1 MODEL 1 The first model is the Random Forest trained by the data with features selected from PCA. Predict = 0 Predict=1 Actual = 0 22571 7343 Actual = 1 63616 27310 AUC = 0.52744 Though this model only reaches an accuracy of 41.28% on the test dataset, it identifies many not alerted drivers correctly. Overall, this model is not good enough, no matter evaluated by which method. 6.2 MODEL 2 The second model is the Random Forest trained from the data with added features of rolling mean and standard deviation. Predict = 0 Predict = 1 Actual = 0 16671 13243 Actual = 1 8679 82247 AUC = 0.7309 This model reaches a good performance, with classification accuracy of 81.86% on the test data. It has a good performance in predicting alert drivers, with recall = 90.45%, precision = 86.13% and F1-score = 88.24%. However, for this specific situation. The model is expected to predict ’not alert’ drivers precisely, specificity (The percentage of Actual = 0 is predicted correctly) should be the evaluation method we focus on for this rea- son. The specificity of this model only reaches 55.73%, which still has lots of room to improve. Overall, the AUC value of this model is 0.7309, not as good as the work I referenced, but still an improvement. 6.3 MODEL 3 The thirdmodel isthe Neural Network trained fromthe data with added features of rolling mean and standard deviation. It has four hidden layers with 90, 70, 50, 30 neurons in each layer. We were meant to use the first layer to grab all the original features and the coming layers to process and predict the output, thus the number of neurons for the first layer is 20 as many as the number of the features. Predict = 0 Predict = 1 Actual = 0 12886 17028 Actual = 1 361 90565 AUC = 0.71340 This model reaches a good performance as well, com- pares to the first model. It successively predicts most of alert drivers (recall = 99.6%, precision = 84.17%, F1- score = 91.24%). However, the model fails to predict many not alert drivers correctly (specificity = 43.08%), which is the more important evaluation method for this predictive model. 6.4 COMPARISON OF MODEL 2 AND MODEL 3 Both two models perform better on predicting alert drivers than identifying not alert drivers as the true positive rate are both higher than their true negative rate in their confusion matrix, though the main goal of this predictive model is to predict not alert drivers. The curve reaches 100% true positive rate firstly isthe neural network, the other curve is the random forest. It also can be found that the random forest model performs better than the neural network model. However, the neural network predicts most alert drivers correctlyandwhen it predicts a driver as not alert, it’s correct at the most of times. 図 3: ROC curve of two models 21 The random forest model reaches a significantly higher result on identifying not alert drivers from all the drivers than the neural network model, though when it predicts a driver as not alert, it gets 34.25% chance of being wrong. The Random Forest model would be a better choice in this situation, but the architecture of the neuralnetwork model can be optimized to reach a higher performance. 7 RESULT REFLECTION AND COMPARISON 7.1RESULT CONCLUSION This project was meant to build a supervised learning model to predict not alert drivers, the model with the best performance is achieved by Random Forest with 50 trees in it. It predicts 16671 of 29914 not alert drivers correctly in the test data. It reaches a classification accuracy of 81.86% and AUC value of 0.7309. 7.2 RESULT COMPARISON When it compares to the results of those in the leader- board,there are lots of participants’models reach a higher performance. The best model reaches an AUC value of 0.86115, though it applies means and standard deviations of each trial as new features. Almost 20 participants’ models reach AUC scores over 0.8, which issignificantly higher than mine. Refers that there is still large room to improve my model. 7.3 DISCUSSION AND FUTURE WORK The existed predictive model for this problem is far from perfection. There are a few perspectives that can improve the performance of the model. 7.3.1 DATA PREPROCESSING METHOD The rolling means and the standard deviation is proved to be a good method to preserve the sequential attribute of the data. However, rolling means for every 0.5s could be too short to grab the driving pattern of a driver. Expand the rolling window to produce rolling means and standard deviations in a longer period could be considered as a useful method to introduce the long-term driving patterns of drivers. Better performance is believed can be achieved by model learns not only from drivers’ behaviors in a short time (0.5s) but in a long time as well. 7.3.2 MODEL OPTIMIZATION AND SELECTION Though the neural network model fails to produce a better performance than the random forest model, it is still not convincing that random forest is alwaysthe best option for this problem. Neural network still shows great potential to produce good result. Further work could to optimize the architecture of neural networks. 22 REFERENCE LINK • https://www.kaggle.com/c/stayalert/data PROJECT LINK https://github.com/bindhu520/Safe-driving-Challenge-ML-PROECT-
Merge #1162 #1342 #1374 #1383
1162: DifferentialEquations.jl-Based ODE Solvers r=simonbyrne a=ChrisRackauckas
This sets up "two" ODE solvers: DiffEqJLSolver and DiffEqJLIMEXSolver. These solvers allow taking in a DEAlgorithm which can be any DiffEq common interface ODE solver that satisfies the common integrator interface, which includes OrdinaryDiffEq, Sundials, and a few others.
The only kernel calls are done through fused broadcasts, so in theory there shouldn't be any overhead on large problems as long as we are only using the right number of kernels. In practice, since this is our first super larger scale application since we plan to track, there may be an extra kernel call somewhere, and so I assume that after we get attempt 1 done here we may need to make an upstream change. The standard RK methods (including the SSP methods, which includes some methods with enhanced stability specifically for DG discretizations) should all be optimized already. I think we may need to do something special for the low-storage RK methods.
As for implicit and IMEX methods, I setup the interface for how to hook into them, though we may need to setup a linear solver for DiffEq. That can be done via https://docs.sciml.ai/latest/features/linear_nonlinear/#Linear-Solvers:-linsolve-Specification-1 , so we can just make the solvers call your linear solvers.
There are a few things we're looking at for how this will benefit both our own research and Clima. To list a few:
- Our methods for numerical uncertainty quantification will allow for understanding the error induced by the time stepping methods. There's other (undocumented) UQ methods being published soon which could be useful too.
- Checkpointed adjoint sensitivity analysis for parameter fitting and UQ is probably a big feature Clima would gain from this. More work might be required to complete the integration of this, but I think there could be a lot of uses.
- Stochastic differential equation models would be pretty accessible from this form, so if there are reasonable extensions to SPDEs they can be considered and high order adaptive methods can be applied.
- Generalized Physics-Informed Learning: you can see from this talk that I want to start taking this to climate models for developing surrogates.
- Tons of new methods. Specifically:
- A bunch of SSP methods
- A bunch of low-storage RK methods
- A bunch of IMEX methods
- Whatever we build in the future.
The benefits to SciML are:
- We see that there's been some recent work in MRI methods. It would be nice to capture that momentum as something that can be added directly to OrdinaryDiffEq (or even just in SimpleDiffEq.jl or another common interface algorithm package) so that they can be used on an SplitODEProblem. This would enhance the Julia ecosystem beyond Clima and be something that can then be demonstrated as a benefit of the Clima project to other projects down the line. By creating this common interface ODE solver algorithm for Clima, I hope we can accelerate this effort because now, if the MRI methods are moved to a DiffEq interface compatible package, they should be usable in Clima without overhead which should be a good incentive to do so!
- We can start to use Clima as one of our canonical benchmarks, and hopefully overtime improve algorithm performance on Clima's type of problems and use it as a source of real-world problems in publications and grants (possibly we could look for a joint grant in this area).
- This serves as a good testing ground for MPI and GPU based array usage. It won't be perfect until we have a real test, so let's get a test!
So in short, this should help Clima get a few new features while improving SciML for large models to help out other PDE-based modeling projects beyond Clima.
There are a few small issues noticed in this PR:
- What test case should I be using? I just grabbed what was in the ODESolver tests, but what's a good demo climate simulation I can start using to test out CPU usage, GPU usage, and MPI+GPU usage? I'd like to have these three cases for development and to add to DiffEqBenchmarks.jl.
- When is
pdefined? In the DiffEq solvers, we wantpat the time of the construction of theODEProblemso we can concretely type the integrator. We at least need to know and have something of its type. The test just hasnothingparameters, so that works, but I wonder if this will be an issue. I addedpas a keyword argument to the solver's constructor. - The methods from Sundials and other C++/Fortran libraries are "technically" supported, but in reality they need the array type to be an
Array, which I assume will almost never be the case with Clima. Sundials does have some good MRI methods that would be interesting to try out here, but making them work would require making NVector wrappers which, it might just be easier to make Julia versions of all of that (and that is a summer project for students). - Is there any more structure in the ODEs that can be exploited? Semilinearity for exponential integrators or anything of that sort? Symplecticness or partitioning of the states?
- Dependencies: are you open to a DiffEqBase dependency? That's a whole lot smaller than OrdinaryDiffEq or DifferentialEquations.jl. A user would need to
using DifferentialEquations(or justusing OrdinaryDiffEq) to actually get solver methods.
If this is setup, then I would like to have some of our GSoC and MLH students optimize this interfacing and demonstrate results using it. That would then set the stage for less technical more mathematical students to just do pure algorithm development in OrdinaryDiffEq knowing that this would be efficient. I hope we can demonstrate that this connection ends up calling the optimal number of kernels by the end of the summer.
In 18.337 Scientific Machine Learning and Parallel Computing in the next fall, each student has final projects to do and, if this is all setup, I would like to point a few students towards implementing some new methods and demonstrating the results on Clima, so this is where new IMEX, MRI, etc. methods can get developed, improved, and tested.
1342: Em/agl/kd/land framework r=kmdeck a=kmdeck
This PR builds on the PR 1279, which added the src/Land/Model and test/Land/Model directories. PR 1279 created the LandModel framework (of which the SoilModel is a component, in turn composed of a water and a heat balance law). PR1279 also added SoilWaterParameterizations.jl, which contained the parameterized functions that we would need for the balance law for water in soil, and some unit tests for those functions (test_water_parameterizations.jl). This PR adds in the balance law components and a boundary conditions interface for the soil water model.
Our goal is to make it so that, within the soil model, one could run just the heat model, or just the water model, with different degrees of coupling (no coupling, or one way coupling in either direction). One way coupling is envisioned to occur when the user prescribes a function for e.g. water content as a function of space and time, instead of dynamically determining this from a PDE. This function could then be used to drive the heat model.
This PR adds in the balance law terms for the water component for two concrete types, inheriting from AbstractWaterModel. One is a PrescribedWaterModel, in which the user could supply a function of the water content as a function of space and time. There are no state variables or fluxes in this case. The other concrete type is SoilWaterModel, which has the necessary fluxes, etc, for Richard's equation. Along those lines, we created a function get_moisture_content(), which returns the state variables for moisture if a SoilWaterModel is used, or the value of the prescribed functions if a PrescribedWaterModel is chosen. When the balance law for heat is added, it will need to call that function, since moisture shows up in the equations for heat.
We also have a PrescribedTemperatureModel, along those same lines, and a get_temperature() function. Right now, all of the soil models use that AbstractHeatModel type. A second type, akin to the SoilWaterModel type, will be added in a future PR.
We've tested that (and have test cases showing that): (1) the two Prescribed models can be used together & the LandModel can still be integrated (nothing happens, no state variables, but the boundary condition calls dont give errors in this case). (prescribed_twice.jl) (2) The way we supply boundary conditions (as a structure with Dirichlet or Neumann conditions, for the top or bottom of the domain) works as expected. (test_bc.jl) (3) Richard's equation gives a solution which matches what we get from verified Matlab code (haverkamp_test.jl) (4) in steady state, the gradient of the hydraulic head goes to zero. This is in tutorials, because it is more of a sanity check/benchmark, compared with a true unit or integration test.
I have
- Written and run all necessary tests with CLIMA by including
tests/runtests.jl - Followed all necessary style guidelines and run
julia .dev/climaformat.jl . - Updated the documentation to reflect changes from this PR.
1374: Fix GCM benchmarks and add to Slurm CI r=kpamnany a=thomasgibson
The newly added GCM benchmarks on master are already broken. This PR hopefully fixes any lingering issues and adds all gcm tests to the Slurm CI suite (on GPU)
1383: Add flattened_tup_chain and chained getproperty/getindex methods r=charleskawczynski a=charleskawczynski
Debugging highly nested models (e.g., EDMF) is difficult because, at the moment, checking fields in all sub-models requires manually writing out all fields/subfields, or writing recursive getproperty/getindex methods, which is painful/overkill just for debugging. This PR adds a method flattened_tup_chain, along with a recursive and interleaved getindex/getproperty, to VariableTemplates so that users can iterate through every field in compute kernels in a convenient way, as demonstrated in a new test in the test suite:
ftc = flattened_tup_chain(st)
@test ftc[1] === (:ntuple_model, 1, :scalar_model, :x)
@test ftc[2] === (:ntuple_model, 2, :scalar_model, :x)
@test ftc[3] === (:ntuple_model, 3, :scalar_model, :x)
@test ftc[4] === (:vector_model, :x)
@test ftc[5] === (:scalar_model, :x)
# getproperty with tup-chain
for i in 1:N
@test v.scalar_model.x == getproperty(v, (:scalar_model, :x))
@test v.vector_model.x == getproperty(v, (:vector_model, :x))
@test v.ntuple_model[i] == getproperty(v, (:ntuple_model, i))
@test v.ntuple_model[i].scalar_model == getproperty(v, (:ntuple_model, i, :scalar_model))
@test v.ntuple_model[i].scalar_model.x == getproperty(v, (:ntuple_model, i, :scalar_model, :x))
endIf we change our initialization, as suggested by @trontrytel and discussed with @kpamnany, to initializing fields to NaNs, a very practical use-case of this might look like:
function init_heldsuarez!(balance_law, state::Vars{st}, aux::Vars{au}, coords, t) where {st,au}
# Initialize fields...
for tc in flattened_tup_chain(st)
@assert getproperty(state, tc) ≠ NaN
end
for tc in flattened_tup_chain(au)
@assert getproperty(aux, tc) ≠ NaN
end
# All fields in all balance_law submodels are now gauranteed to be initialized
endCo-authored-by: Chris Rackauckas accounts@chrisrackauckas.com Co-authored-by: Charles Kawczynski kawczynski.charles@gmail.com Co-authored-by: Thomas Gibson gibsonthomas1120@hotmail.com
Add files via upload
- Added one extremely important widget - Settings Check widget. It prevents the system from the most annoying bugs like sibling variables reset or missing Ralf links in the Barn. The second part of this widget will notify you with the pop-up window that Problem Solver just fixed the setting error. This widget is very important and still in progress yet, so help me with your feedback, please. I really need it :)
- Added a new non-consent scene. When you explore the city, there are equal chances to met a robber and a rapist. There is no Corruption check, by the way
- Fixed three last cheat codes for $24 tier supporters. Sorry for this
- Deleted "Lubricant", "Anal lubricant", "Vibrator", "Egg vibrator" from Lexena Market due to their uselessness
- As always, some writing improvements from Rachael. Thank you, Rachael
few more ugly hacks to make this shit working... why the fuck did I even move to norway?..
I just fucking wrote my own gui layout file format HOLY SHIT
wgengine/magicsock: deflake some tests with an ugly hack
Starting with fe68841dc7649162c43849beb2fcf9a2ad80ee7c, some e2e tests got flaky. Rather than debug them (they're gnarly), just revert to the old behavior as far as those tests are concerned. The tests were somehow using magicsock without a private key and expecting it to do ... something.
My goal with fe68841dc7649162c43849beb2fcf9a2ad80ee7c was to stop log spam and unnecessary work I saw on the iOS app when when stopping the app.
Instead, only stop doing that work on any transition from once-had-a-private-key to no-longer-have-a-private-key. That fixes what I wanted to fix while still making the mysterious e2e tests happy.