You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently the event (and lesser extent run) header table that still originates from the Virtex TOS ascii data days is kept as a Table[string, string] after parsing & until writing to the H5 file in raw_data_manipulation.
This is both error prone (in the sense of typos etc.) and very inefficient, as the majority of fields (but not all, e.g. "DateTime") are actually integers.
In addition for other data sources, e.g. Timepix3, data doesn't even arrive in string format. So we have an especially unnecessary back and forth conversions to string land.
Instead we should:
either define a typed object EventHeader
or have a table of something like Table[string, EventHeaderField]
with theoretical implementation ideas like:
typeEventHeader=
dateTime: DateTime
timestamp: intcase timepix: TimepixVersion# or TOS version field? ofTimepix1:
useHvFadc: bool...
or alternatively:
typeEventHeaderFieldKind=enum
ekString, ekBool, ekFloat, ekDateTime
EventHeaderField=objectcase kind: EventHeaderFieldKind of ekString: s: string of ekBool: b: bool...
etc.
I'd do it now, but it touches too many procedures etc. to quickly make the change.
This would also allow us to get rid of fully optional fields like useHvFadc in detectors / data sources for which it makes no sense etc.
The first solution has the advantage of being fully typed, while the second still allows for trivial extensions (i.e. adding a new scintillator field etc.).
The good thing about the second one is that at that point the code simplifies significantly as we have all type information ready in the table. So we don't have to hardcode the fields that are e.g. int or string anymore and can just check the kind of the field and generate a correct resulting H5Dataset based on that. Maybe that's the best of both worlds.
The text was updated successfully, but these errors were encountered:
In addition the timestamp information for Timepix3 data is of course much more precise than up to a single second resolution. Therefore in the current implementation we even throw away that precision for Tpx3. With a variant object in theory we could even have different timestamp types for Timepix1 and Timepix3 (but personally I'd probably just switch over from int to float for all of them.
Currently the event (and lesser extent run) header table that still originates from the Virtex TOS ascii data days is kept as a
Table[string, string]
after parsing & until writing to the H5 file inraw_data_manipulation
.This is both error prone (in the sense of typos etc.) and very inefficient, as the majority of fields (but not all, e.g.
"DateTime"
) are actually integers.In addition for other data sources, e.g. Timepix3, data doesn't even arrive in
string
format. So we have an especially unnecessary back and forth conversions to string land.Instead we should:
EventHeader
Table[string, EventHeaderField]
with theoretical implementation ideas like:
or alternatively:
etc.
I'd do it now, but it touches too many procedures etc. to quickly make the change.
This would also allow us to get rid of fully optional fields like
useHvFadc
in detectors / data sources for which it makes no sense etc.The first solution has the advantage of being fully typed, while the second still allows for trivial extensions (i.e. adding a new scintillator field etc.).
The good thing about the second one is that at that point the code simplifies significantly as we have all type information ready in the table. So we don't have to hardcode the fields that are e.g.
int
orstring
anymore and can just check thekind
of the field and generate a correct resultingH5Dataset
based on that. Maybe that's the best of both worlds.The text was updated successfully, but these errors were encountered: