event-representation

one may categorize event representation roughly into 4 modalities.

1. spike processing such as SNN
   • natively support sparse asynchronous data
   • difficult to train
   • require specialized hardware
2. analytical event representations
   • task-specific: do not generalize to a wide range of applications
3. intermediary representation
   • to be paired with machine learning methods in synchronous form.
   • be transformed into a proxy 2d image-like or 3d video frame-like representation——“proxy frames”
4. intensity image reconstruction
   • estimating intensity values for each pixel using events——ML methods

intermediary representation

Events occurring within a given temporal window are summarized to yield a proxy frame using one of the following techniques.

event accumulation/counting

events within a temporal window are separated by polarity and events are counted for each polarity type to yield two images of size $H\times W$

remark: while this approach retains polarity information, temporal information is lost.

event frames/polarity summing

computed by summing the polarity of events within a given time window.

polarity summing yields a single $H\times W$ proxy frame approximating the overall intensity changes that occurred within the given temporal window.

voxel grid

takes a temporal quantization approach, mapping the events to the nearest temporal grid.

remark: information is lost whenever multiple events are mapped to the same quantized pixel.

time surface

Time surface is a two dimensional representation that encodes time stamps as pixel values.

Surface of Active Events(SAE) time surfaces retain the timing for the latest event at each pixel location(for each polarity)

remark: not ideal for slow motion sequences or imaging in low light environments, as events timing becomes ambiguous or can be interrupted by noises

Graph-based

Graph-based methods transform events within a temporal window into a set of connected nodes, $U\times V$.

remark: such compact representation allows for a reduction in compute and memory resources

• constructing the graphs can be computationally intensive.
• discards fine temporal and spatial information during the transformation

bio-inspired

similarly, modeled after the behavior of Leaky Integrate and Fire(LIF) neurons, events are treated as spikes that decay over time exponentially in magnitude.

A general issue with time-windowed or event-windowed representations is that they induce latency.

• For temporal windowed representations with fixed frame rate, events that occur early in the window are not evaluated until the time window completes.