Sensor fusion is the process of combining data from multiple sensors into a unified picture that is more accurate, more complete, and more reliable than any individual sensor could produce alone. In operational systems it means correlating detections from radar, RF, EO/IR, acoustic, cooperative broadcasts, ISR, and whatever else you have into fused tracks with position, velocity, classification, and identity provenance. It is a physics problem with mathematical solutions. The math is decades old, battle-proven, and in some cases written in blood. It is not a problem to throw an LLM at, regardless of how much of the current marketing suggests otherwise.
For quick-reference answers on fusion algorithms, techniques, and related terms, see the Sensor Fusion FAQ.
The problem fusion solves
Every sensor lies. Not maliciously - but every sensor measurement contains noise, bias, and uncertainty. A radar return has range and bearing accuracy measured in meters and degrees. An RF direction-finding array has bearing uncertainty often in double-digit degrees. An EO/IR camera has pixel precision inside a narrow field of view. An acoustic array has range estimates with hundreds of meters of error. Cooperative broadcasts like Remote ID have asserted positions that can be completely fabricated.
No single sensor gives you the truth. Each gives you a noisy, partial, sometimes contradictory view of reality. The job of fusion is to combine those partial views into a picture that is closer to truth than any individual sensor can achieve - and, critically, to flag when the sensors disagree in ways that can't be reconciled so the operator sees the conflict rather than a silently-picked answer.
In operational terms: a radar sees something at range 1.2 km, bearing 047. An RF array hears a drone control link bearing northeast, somewhere in the 1-2 km range estimate. A cooperative RF sensor sees a Remote ID broadcast claiming to be a DJI Mavic 3 at 1.4 km, bearing 050. Are these three detections the same object, two objects, or three? Answering that question correctly, continuously, across potentially dozens of simultaneous detections, is the actual job.
Without fusion, an operator is staring at three separate screens mentally correlating three partial pictures. That works fine in a quiet environment with one or two objects. It collapses the moment a dense scene happens: drone swarm, contested airspace, a real incursion. Putting the three sensor feeds on one screen is not fusion. It is a dashboard. The operator is still the fusion engine, and the operator is overloaded.
Sensor fusion vs. intelligence fusion
"Fusion" covers a stack of related problems, and it helps to be precise about which one this page is about. The widely used JDL/DFIG data fusion model describes levels that run from raw signals up to situation, threat, and resource management.
Sensor fusion lives at the bottom of that stack. It answers physical questions about objects: how many are there, where are they, how are they moving, and what are they? Its inputs are measurements, and its outputs are tracks. The math is estimation, association, and probability.
Intelligence fusion builds on those tracks. It asks what the objects and events mean together: who owns this vessel, is this drone part of a pattern, does this emitter match a known threat, is this activity coordinated with something happening online? Its inputs include tracks plus reporting from other disciplines, and its outputs are assessments.
The two depend on each other. Intelligence fusion is only as good as the tracks underneath it, and sensor fusion without context produces accurate dots that nobody can interpret. This page covers the tracking layer. For the layers above it, see What is Intelligence Fusion?.
How fusion actually works
Fusion operates in three layers, each solving a different class of problem. They interact continuously, which is part of what makes getting this right non-trivial.
Layer 1: State estimation
The foundation of fusion is state estimation - maintaining a mathematical model of each tracked object's position, velocity, and (sometimes) acceleration over time. The workhorse algorithm is the Kalman filter. Variants handle specific problems: Extended Kalman (EKF) for nonlinear dynamics, Unscented Kalman (UKF) for highly nonlinear systems, and Interacting Multiple Model (IMM) for targets that transition between flight regimes.
The Kalman filter operates in a predict-update cycle:
Predict. Using the target's last known state and a kinematic motion model (constant velocity, constant acceleration, coordinated turn, etc.), project the state forward to the current time. The uncertainty in the estimate - represented by the state covariance matrix - grows during prediction because no new information has arrived.
Update. When a new measurement arrives from any sensor, compare the predicted state to the measurement. The residual (the "innovation") is weighted by the relative confidence in the prediction versus the measurement. An accurate sensor measurement pulls the estimate strongly. A noisy measurement pulls it weakly. The uncertainty shrinks after update because new information has been incorporated.
This cycle runs continuously, at the sensor's update rate, for every tracked object. A radar that updates every 2 seconds triggers a predict-update cycle every 2 seconds. An EO/IR tracker at 30 Hz triggers 30 cycles per second. Between updates from any given sensor, the filter predicts forward, maintaining the best estimate possible given the information available.
The Kalman filter naturally handles sensors with different update rates, different accuracies, and different measurement types. A radar reports range and bearing. An RF array reports bearing only. An EO/IR camera reports azimuth and elevation in pixel coordinates. The filter's measurement model translates each sensor's native output into the common state space and weights it appropriately.
IMM deserves special mention. Against maneuvering targets - a drone transitioning from cruise to attack dive, a cruise missile going terminal, an OWA executing evasive jinks - a single motion model will lag when the target's behavior changes. IMM runs several filters in parallel (typically a constant-velocity, a constant-acceleration, and a coordinated-turn model) and blends their outputs by how well each is currently explaining the measurements. The filter automatically shifts weight toward whichever model is tracking best. This is what keeps track quality high when targets don't fly in straight lines, which is to say, most of the time.
Layer 2: Data association
State estimation assumes you already know which measurement goes with which track. Data association is the problem of figuring that out, and it is where most fusion systems fail in ways that matter.
When a radar completes a scan and reports 15 detections, and you are already tracking 12 objects, which detection goes with which track? Some are updates to existing tracks. Some are new objects. Some are false alarms from clutter, noise, or multipath. Some tracks have no corresponding detection this scan - the target was in a null, or the sensor was pointed elsewhere, or the target was briefly masked by terrain.
Greedy (nearest-neighbor) association processes detections one at a time. For each detection, find the closest track within a gating threshold, assign it, move on. Fast, simple, and order-dependent. The first detection processed gets the best available match. The second gets whatever is left. In dense environments with tightly-spaced targets, the first assignment steals the correct match from a later detection, producing a cascade of mis-associations that propagates through the entire track picture.
Greedy works fine when targets are well-separated. It falls apart when they are not - which is exactly the scenario that matters. Drone swarms, missile salvos, dense commercial air traffic with threats embedded, coordinated approaches from multiple axes. The systems that claim to "fuse" via greedy matching are fine in the demo and fail in contact.
Optimal global assignment considers all detections and all tracks simultaneously. It formulates the association problem as an optimization: find the assignment of detections to tracks (including "new track" and "false alarm" hypotheses) that minimizes the total statistical distance across the entire scene. The Hungarian algorithm (Munkres) or auction algorithm solves this in polynomial time. The difference is not subtle. In a scenario with 30 targets tracked by 10 sensor classes, greedy produces several mis-associations per scan cycle that compound over time. Optimal global assignment produces the mathematically best assignment every cycle. The 25th detection gets the same association quality as the first.
Joint Probabilistic Data Association (JPDA) is an intermediate approach worth mentioning. Instead of committing to a single assignment, JPDA weights each track's update by the probability that each nearby detection is the correct one. It handles clutter well and is computationally cheaper than full multi-hypothesis tracking. For many operational contexts JPDA is enough. For dense, high-ambiguity scenes it isn't.
Layer 3: Multi-hypothesis tracking (MHT)
Even optimal global assignment commits to a single association per scan cycle. Multi-hypothesis tracking (MHT) goes further: it maintains multiple possible association hypotheses in parallel and defers commitment until the data resolves the ambiguity.
The canonical example: two aircraft in close formation. Their radar returns merge into a single detection. A single-hypothesis tracker must decide - Track A, Track B, or new track? Whichever it picks, it's wrong two-thirds of the time. When the aircraft separate, the tracker has corrupted whichever track it guessed wrong on. Track-swap. Broken picture. Operator confusion at the worst possible moment.
MHT maintains all three hypotheses in parallel, each with its own independent state estimate. When the aircraft separate and produce distinct detections again, the hypothesis that correctly maintained both tracks scores highest. The losing hypotheses get pruned. Both tracks emerge from the merge event with accurate state estimates. No track swap. No corrupted picture.
MHT is more computationally expensive than single-hypothesis tracking because the hypothesis tree grows. Practical implementations use pruning strategies (N-scan-back, hypothesis merging, aggressive gating) to keep the tree bounded. With reasonable pruning, MHT runs comfortably against dozens of targets at sensor update rates on modern edge hardware. It is not a mainframe-scale algorithm. It is table stakes for any system operating in a dense, contested, or adversarial environment.
Track management: birth, life, and death of a track
Estimation and association assume tracks already exist. Deciding when to create, trust, and remove them is a separate problem, and getting it wrong is one of the fastest ways to lose an operator's confidence.
Initiation. A single unassociated detection might be a new object, clutter, a bird, or multipath. Promoting every one to a track floods the picture with false tracks. Most systems start a tentative track and require confirmation, often with M-of-N logic (for example, three associated detections in five scans) or a sequential track score that accumulates evidence. The right threshold depends on the sensor: a radar with a high false-alarm rate in sea clutter needs stricter confirmation than a cooperative ADS-B report.
Confirmation and quality. Once confirmed, a track carries a quality measure based on how recently and how consistently it has been updated, and by which sensors. A track confirmed by two independent sensor types deserves more trust than one held up by a single noisy source.
Coasting. When updates stop, the filter keeps predicting forward and the uncertainty grows. Coasting lets a track survive brief gaps such as a radar null, terrain masking, or a missed scan, so the object is not dropped and re-created with a new identity.
Deletion. Coasting cannot last forever. Tracks are deleted when uncertainty grows too large, when too many expected updates are missed, or when the track score falls below a threshold. Delete too early and real objects flicker in and out. Delete too late and the picture fills with ghosts.
These decisions are highly sensor-dependent. Refresh rate, detection probability, false-alarm rate, measurement accuracy, and latency all change what the right thresholds are, which is why initiation and deletion logic should adapt to the sensors feeding each track rather than use one global setting.
Time alignment and latency
Every measurement has to be placed at the right moment in time. That sounds trivial until you fuse sensors that report with very different delays: a radar processed locally in milliseconds, a camera detection after a computer vision model runs, an API-driven sensor that batches results, or a node several hops away on a mesh network.
Two problems follow:
- Clock alignment. Sensors must share a common time reference, usually GNSS-disciplined time or a characterized offset. A few hundred milliseconds of clock error on a fast target produces position errors larger than the sensor's own accuracy.
- Out-of-sequence measurements (OOSMs). A late measurement can arrive after a newer measurement of the same target has already been processed. The tracker has to retrodict the state back to the late measurement's time, apply a dedicated OOSM update, or reprocess a short history. Ignoring the problem quietly corrupts tracks, and simply discarding late data throws away information from slower sensors.
Latency also affects association. A detection that is accurate but late can land outside the gate of the track it belongs to. Gating and track initiation need to account for each sensor's expected latency, not only its measurement noise.
Sensor registration and bias
Fusion assumes each sensor's measurements are unbiased. In practice they rarely are. A radar mounted a degree off true north, a camera with an imperfect pose estimate, or a sensor whose site survey is a few meters off will place every target slightly wrong in a consistent direction.
Uncorrected bias shows up as duplicate tracks: two sensors see the same object, but their systematic offsets push the detections apart, so the tracker concludes there are two objects. Registration (sometimes called gridlock in naval and air defense systems) estimates and removes those offsets, often by using objects that several sensors observe at once, such as cooperative aircraft broadcasting ADS-B, as reference targets. Registration is not a one-time setup step. Mounts shift, vehicles move, and calibration drifts, so it should be monitored continuously.
Fusion architectures: measurements or tracks
There are two broad ways to combine sensors, and most operational systems use both.
Centralized (measurement-level) fusion sends every detection to one tracker. It is the most accurate approach because the tracker sees all the raw information and can make the best association decisions. It also demands the most bandwidth and assumes every measurement reaches the tracker reliably and on time.
Distributed (track-level) fusion lets each sensor or node maintain its own tracks and shares tracks instead of detections. It uses far less bandwidth, tolerates intermittent links, and keeps working when nodes lose contact. The catch is double counting: two tracks derived partly from the same measurements are correlated, and treating them as independent makes the fused result overconfident. Techniques such as covariance intersection produce a consistent fused estimate without knowing the exact correlation, at the cost of some accuracy.
The practical pattern is hierarchical: fuse measurements centrally where sensors share a fast local network, then share fused tracks between sites, vehicles, and echelons over constrained links. That is also what makes fusion survivable in degraded communications.
Dense scenes and swarms
Every algorithm above becomes harder as density rises. Tracking three drones spread across a sector is a solved problem. Tracking dozens or hundreds of small, slow, closely spaced objects mixed with birds, ground clutter, and decoys is not.
Density stresses fusion in specific ways. Gates overlap, so association ambiguity grows quickly and greedy methods fail. Hypothesis trees in MHT grow and need aggressive but careful pruning. Track initiation has to balance catching real objects against promoting clutter, and the cost of a wrong choice multiplies. Sensor disagreements become the norm rather than the exception, which puts more weight on per-sensor trust and identity provenance. And because no single algorithm wins in every condition, robust systems monitor their own performance and fall back to methods suited to the sensors and data actually available.
Trust and identity provenance
Fusion isn't only about position and velocity. Operational systems must also fuse identity - what is this thing? - and manage the trust level of that identity assessment.
A radar classification says "air track, small RCS, micro-Doppler signature consistent with rotary-wing." An RF sensor says "control link matches DJI OccuSync 3." A Remote ID broadcast says "registered commercial drone, operator ID 12345, position X, Y." An EO/IR track says "quadcopter, no visible payload." An acoustic sensor says "multi-rotor propeller signature, four-blade."
These identity claims come from different sources with very different reliability characteristics. The radar classification is inferred from physics. The RF fingerprint is observed from the actual emitted signal. The Remote ID is claimed by the drone itself and can be spoofed by a $30 ESP32. The EO/IR classification is observed visually. The acoustic match is derived from pattern recognition on the sound signature.
A real fusion system maintains identity provenance: where each piece of identity information came from, how it was derived, and how much to trust it. Claimed identity (Remote ID) is treated differently from observed identity (RF fingerprint), which is treated differently from inferred identity (radar RCS + kinematic behavior). When claims conflict - the Remote ID says "friendly commercial platform" but the RF signature matches a known threat control stack, or the claimed kinematics violate the performance envelope of the claimed airframe - the fusion system surfaces the conflict. It does not silently pick one and move on.
Trust is not static, either. A track corroborated by three sensors five minutes ago might have only one sensor confirming it now. Trust level should degrade as corroborating sensors go stale, and should restore when sensors reacquire. It's a lifecycle, not a binary. The specifics of how a given platform encodes that lifecycle and how different evidence streams combine mathematically - this is where theories like Dempster-Shafer evidence combination, Yager's rule, and related evidential reasoning frameworks come in. The math is public and well-studied. The operational tuning - what weights to apply, how zones should affect trust policy, how to detect spoof conditions and react to them - is where real engineering lives.
Why fusion has to run at the edge
The physics argument for edge-deployed fusion is straightforward: latency. A rotating radar paints a target roughly every 2 seconds. The fusion engine has to process that detection, associate it with existing tracks, update the state estimate, and present the result to the operator before the next scan completes. If the fusion engine lives in a cloud data center, round-trip latency adds 200 milliseconds to 2 full seconds depending on the link. For an intelligence picture, that's acceptable. For a fire control solution against a target moving at Mach 3 with a closing engagement window, it isn't.
The operational argument is stronger: survivability. A cloud-hosted fusion engine is a single point of failure. When the satellite link drops - whether from adversary action, weather, or equipment failure - every unit depending on that fusion engine loses its operational picture simultaneously. Edge-deployed fusion means each unit maintains its own fused picture from its organic sensors. Connectivity enriches the picture. Its absence does not destroy it.
The computational argument kills the cloud case entirely. Fusion is computationally modest. Kalman filtering is matrix multiplication and inversion per track per update cycle. Optimal global assignment is polynomial in the number of tracks and detections. MHT with pruning is bounded. Full multi-hypothesis fusion against dozens of targets from many sensor classes runs comfortably on a COTS edge compute module drawing well under 100 watts. There is no computational reason to put fusion in the cloud. The only reason it ends up there is architectural inertia from platforms that were designed cloud-first, back when edge compute was less capable and bandwidth was cheaper than it has become in contested environments.
For the longer architectural argument, see Why JADC2 Needs Sensor Fusion at the Edge and What is JADC2?.
Data normalization: the unglamorous prerequisite
Before you can fuse anything, you must normalize everything. This is less glamorous than talking about Kalman filters and MHT, but it is where a depressing number of "integrated" solutions fall apart.
Every sensor vendor reports differently. Coordinate systems vary (WGS84 geodetic, MGRS, local Cartesian, ENU). Update rates range from sub-second to multi-second. Uncertainty is expressed inconsistently - some vendors give you a full covariance matrix, some give you CEP, some give you nothing and expect you to treat their detections as gospel. Transports are fragmented: one sensor streams JSON over MQTT, another uses a proprietary SDK with a gRPC interface, another pushes ASTERIX CAT-062 over UDP, another wants you to poll a REST API, another has a serial interface with a custom pinout. Schemas are lossy - Cursor-on-Target and Link 16 J-messages both have structural limitations that drop meaningful vendor metadata into unstructured extension fields or off the table entirely.
Before a single track can be correlated, all of this has to be ingested, parsed, timestamped to a common clock (or at least well-characterized time offsets), projected into a shared spatiotemporal frame, tagged with a normalized uncertainty envelope, and preserved with enough semi-structured context that classification and identity logic downstream has something to work with. Get the normalization wrong and everything downstream is downstream of garbage. Get it right and the rest of the fusion math has a chance.
Sensor fusion versus "AI-powered" fusion
A recurring complaint worth making explicit: the defense tech industry has started marketing "AI-powered sensor fusion" as if frontier LLMs are the right tool for multi-hypothesis tracking across dozens of targets at millisecond resolution. They aren't. LLMs are astonishingly useful for many problems - explainability, natural-language interaction with curated intelligence, summarization of structured inputs, agent-based workflow orchestration. They are not a replacement for the linear algebra and probability theory that has been solving the tracking problem for decades.
Throwing detections into an LLM and asking what it thinks is not sensor fusion. It is a tech demo. Render unto math what is rightfully a math problem. Use AI for what it's actually good at. The two are complementary, not substitutable.
Where Empyrean fits
Empyrean's Fusion & Decision Engine implements multi-hypothesis tracking with optimal global assignment, zone-scoped trust policy, identity-provenance lifecycle management, and spoof detection at the edge. It consumes fused track output into every other workspace on the platform: the Common Operational Picture, EMSO, SSA, Narrative Intelligence, CUAS and Force Protection, Cyber / Physical Convergence, and the Policy & Decision Layer that automates response chains across all of them. The Simulation & Wargaming engine runs the same fusion pipeline against synthetic sensor data so operator training transfers directly to operations. All of it runs at the edge on hardware that deploys with the unit, with no cloud dependency.
Going deeper
For the architectural case on why fusion has to run at the edge rather than the cloud, read Why JADC2 Needs Sensor Fusion at the Edge. For how sensor fusion applies specifically to counter-UAS operations in operational depth, read The Ultimate Guide to Counter-UAS Operations. For the broader JADC2 context, see What is JADC2?. For the intelligence layers built on top of tracks, see What is Intelligence Fusion?. For how Empyrean approached swarm-class fusion, see Release Log Vol. 2. For the counter-UAS reference page, see What is Counter-UAS?.
