Industrial Data Historian: Time-Series DBs & Compression

What an industrial data historian really is

An industrial data historian is a purpose-built time-series database for the high-frequency signals produced by industrial control systems — PLCs, DCS, SCADA, field instruments, IoT gateways — optimised along four axes that a general-purpose RDBMS systematically fails on: very high tag counts (tens of thousands to millions of distinct measurement points), very high sample rates (sub-second to hundreds of Hz per tag), very long retention horizons (typically 5–20 years for the warm tier, indefinite for regulated data), and very specific query patterns dominated by time-range scans, windowed aggregations, and trend reconstruction. The category was effectively defined by one product: OSIsoft PI, founded in 1980 by J. Patrick Kennedy, based on the conviction that the storage problem for process data was fundamentally different from the storage problem for business transactions and deserved its own engine. PI has been the category-defining system for forty-five years. Every modern alternative — InfluxDB, TimescaleDB, QuestDB, ClickHouse, Azure Data Explorer, AWS Timestream, VictoriaMetrics — is legible against PI as reference point, which is one of the strongest examples in industrial software of a product that so thoroughly solved a problem that its name became synonymous with the category. AVEVA's 2020 acquisition of OSIsoft rebranded it to AVEVA PI System, but the architectural DNA is unchanged.

I have been building SYMESTIC's time-series ingestion and storage layer since 2014 and the architectural choices we made — and the ones we consciously did not make — turn out to be almost entirely downstream of the tradeoffs the PI team worked out in the 1980s and 1990s. When you scale to 15,000+ machines across 18 countries publishing on the order of 100,000 structured signals per second at peak, the question is not whether you need the abstractions a historian provides. The question is whether you build them yourself on a modern time-series engine, license an established historian, or accept a hybrid where the historian covers process data and a modern TSDB covers the business-contextualised derivatives. This piece walks through the architectural decision space as it actually looks in 2026, not as vendor marketing describes it.

The tag-count arithmetic — where the storage problem comes from

The best way to understand why historians exist as a distinct category is to do the arithmetic that an RDBMS breaks on. A discrete-manufacturing plant with 50 production machines, each exposing roughly 200 signals (status bits, counters, setpoints, measured values, alarm tags), sampled at 1 Hz, generates the following volume:

Scale the same arithmetic to an enterprise deployment — 10 plants, an average of 100 machines, and sub-second sampling for the signals that matter (pressure, temperature, torque, current draw) — and the uncompressed raw feed exceeds 500 GB/day, or roughly 180 TB/year per customer. No general-purpose RDBMS is going to ingest that at the write rates required while simultaneously serving trend queries across 5-year windows without catastrophic index bloat and query latency. This is the arithmetic that forced the historian category into existence in the early 1980s and that still forces it today, even though the tooling has changed.

Swinging Door Trending — the compression algorithm that made historians possible

The single most important algorithm in the historian category is one that most of its users have never heard of by name. Swinging Door Trending was published by Edgar Bristol in 1990 ("Swinging Door Trending: Adaptive Trend Recording?", ISA Conference Proceedings, 1990) and became the default compression strategy in OSIsoft PI shortly thereafter. The idea is deceptively simple: the algorithm maintains a tolerance corridor around the trajectory of a signal as it is being recorded, and it only writes a new data point when the signal would exit the corridor if the previous point were held. The result is that smoothly varying signals — the overwhelming majority of industrial measurements — are represented by the minimum number of points needed to reconstruct the trend within a defined tolerance, while sharp transitions are captured faithfully. Typical compression ratios for industrial process data under well-tuned Swinging Door parameters run 10:1 to 50:1 depending on signal character; noisy high-frequency signals compress less, slowly-varying temperatures and flow rates compress more.

Four compression techniques are worth distinguishing, because plants that configure them without understanding the tradeoffs end up either with bloated storage (no compression) or with silent data loss (too-aggressive compression):

The Dead-Band Trap is the anti-pattern I see most often on customer sites that have inherited a historian configuration from someone who left the company five years ago. A dead-band value that is too loose silently loses signal transients that may be diagnostically critical — a 200 ms pressure spike on a hydraulic system, a brief temperature excursion during a changeover, an alarm-precursor drift that the maintenance team would have caught if the data had been recorded. Once the data is compressed away, it is gone; no amount of analytical sophistication downstream can reconstruct what was never written. Tuning compression parameters is a signal-engineering task that deserves the same level of engineering discipline as tuning a PID controller — and it is almost never treated that way. The guidance that works: default to tight compression (tolerances at 1-2% of measurement range, time windows under 60 seconds), audit compression ratios quarterly, and loosen parameters only when specific tags have been demonstrated to compress less than 5:1 under diagnostic-grade settings. The opposite failure — too-tight compression — produces storage bloat but never silent data loss, which is the failure mode you want.

Hot, warm, cold — the storage-tiering problem

A historian at scale is not one database; it is three, each optimised for different query patterns and operating on different storage media. Plants that treat it as a monolithic store either overpay for SSD capacity they do not use effectively or underperform on the recent-data queries that drive operational decisions.

The retention-window decisions are not aesthetic; they are driven by compliance and by the economics of storage. Regulated industries have specific minimum retention requirements that are non-negotiable: GxP environments (pharma, medical devices) typically require 10 years for batch records under EU Annex 11 and FDA 21 CFR Part 11; IATF 16949-governed automotive production requires 15 years for safety-relevant traceability data; EU ETS-regulated energy-intensive industries require historization of energy consumption for the full installation lifetime. The cold tier cost structure has to work for these windows or the system is not compliant, which in practice means compressed Parquet on object storage with column-level encoding optimised for the typical access pattern (time-range scans with tag-set projection). The warm-to-cold transition is the architecturally interesting one — done naively it breaks query continuity, done well it is invisible to the user.

The modern time-series database panorama — PI is not the only answer anymore

The twenty years from 1995 to 2015 were a near-monopoly for classical historians: OSIsoft PI, Honeywell PHD, GE Proficy, AspenTech IP.21, and a handful of others divided the industrial market with broadly similar architectures. The decade since 2015 has fundamentally changed that landscape with the arrival of a new generation of general-purpose time-series databases that originated outside the industrial domain — primarily from observability, IoT, and financial analytics — but that have architectural properties making them increasingly viable for industrial use. The seven that matter:

The practical decision framework: classical historians (PI) still dominate in heavily regulated process industries (pharma, oil & gas, utilities) where the 40-year feature depth around calculation engines, unit dictionaries, and compliance tooling is genuinely hard to replicate. Modern TSDBs dominate in cloud-native greenfield deployments where the workload profile is either very write-heavy (InfluxDB, QuestDB), analytics-heavy (ClickHouse, Kusto), or tightly coupled to business data that needs SQL JOINs (TimescaleDB). The hybrid pattern that most modern cloud MES platforms implement — including ours — uses a modern TSDB as the primary ingestion and storage engine while retaining historian-like abstractions (tag metadata, compression-aware querying, retention tiering, unit management) at the application layer. The result is a system that has 90% of what a classical historian delivers at 20% of the storage cost, with the specific features a manufacturing platform actually uses on top.

The Unified Namespace — the modern architectural alternative

The most consequential architectural idea in industrial data handling since the historian itself is the Unified Namespace (UNS), articulated publicly by Walker Reynolds from around 2020 onward and adopted rapidly in both greenfield and retrofit contexts. The core idea: instead of building a monolithic historian as the canonical source of process data, you publish every meaningful signal to a broker (typically MQTT with Sparkplug B payload) organised by a hierarchical namespace (enterprise / site / area / line / cell / device / signal), and every consumer — historian, MES, analytics platform, dashboards, AI models — subscribes to exactly the portion of the namespace it needs. The historian becomes one subscriber among many rather than the mandatory middleware through which all data must flow. The architectural consequences are significant enough that the UNS pattern deserves to be understood as a rival philosophy to the classical historian-centric architecture, not as a complement:

The practical pattern we implement: MQTT broker with Sparkplug B as the canonical real-time layer, and a modern TSDB (Azure Data Explorer in our cloud, with local edge buffering) as the historization subscriber. The UNS gives us the architectural flexibility to add new data consumers without changing the historian, and the TSDB gives us the multi-year query performance that a broker alone cannot provide. Anyone designing a greenfield industrial data architecture in 2026 should start from the UNS pattern and treat the historian as a subscriber rather than as the central hub; the retrofit pattern for existing PI deployments is typically to publish from PI to a broker to expose the data to modern consumers without replacing PI itself.

Store-and-forward at the edge — the availability requirement

The most operationally consequential requirement for an industrial data architecture is one that vendors rarely emphasise and that plants discover the hard way during the first network outage: what happens to the data stream when the link between the plant and the cloud historian drops. The answer is that if the architecture does not explicitly implement store-and-forward at the edge, the data is simply lost — and the gap in the historical record is permanent, which in regulated environments is a compliance violation and in operational environments is exactly the data that was most likely to be needed (network outages correlate with larger plant disruptions that generate exactly the trend data you want to analyse afterwards).

The pattern that works: every edge gateway buffers data locally with a retention window sized to the longest plausible outage (typically 7–30 days for industrial WAN links, 2–4 hours for fibre-served sites). When connectivity returns, the gateway replays buffered data to the historian in chronological order with original timestamps preserved, and the historian is engineered to accept out-of-order inserts gracefully. Sparkplug B implements this with birth/death certificates that signal the state transitions to all subscribers; classical OPC UA Client-Server implementations typically do not implement store-and-forward natively and have to be augmented with an edge buffer. The architectural question "does my historian accept out-of-order inserts without breaking compression or query continuity" is the single most important question to ask during vendor evaluation, and the answer separates systems that work from systems that will silently lose your data during the first week-long WAN outage.

From the SYMESTIC platform build, 2017–2019: the architectural decision I have spent more hours rethinking than any other was how to handle the ingestion layer for process-data signals at scale. The first-generation implementation we shipped in 2016 used straight PostgreSQL with a hand-tuned TimescaleDB configuration and no compression-aware ingestion — every signal from every machine landed as a raw row with full precision, indexed by (tag_id, timestamp), and we scaled it by throwing hardware at the problem. That worked up to approximately 1,200 connected machines across the customer base. At roughly 1,500 machines, we hit a wall that manifested as query latency degradation on recent-data dashboards — the standard OEE line view that loads the last 24 hours of status transitions started taking 8–12 seconds instead of the sub-second we were advertising, and the latency was caused by the query optimizer losing the ability to satisfy the window scan from memory once the hot-tier index exceeded about 40 GB. The obvious fix was Swinging-Door compression at ingestion, but implementing it in our architecture was not as simple as flipping a flag, because we had already made compliance commitments to customers about data-precision preservation that we did not want to violate. The architectural decision that took me about six weeks of analysis and that I would make identically today: dual-write every signal in two parallel streams — a compressed stream with aggressive Swinging-Door parameters into the hot/warm-tier TSDB driving all operational queries, and a raw uncompressed stream into Parquet files in Azure Blob Storage driving compliance, audit, and ML training. The storage cost penalty of keeping the raw stream is approximately 7x the compressed stream's storage cost, but object storage is cheap enough that the total infrastructure bill increased by less than 4% while query latency dropped to sub-500 ms for the worst dashboard workload. The second decision that has paid back harder than I expected: Sparkplug B as the canonical wire format between every edge gateway and the cloud ingestion endpoint, which gave us birth/death certificates and free store-and-forward semantics that we would otherwise have had to build ourselves. A customer in Mexico went offline for four days in late 2022 during a regional network incident; when the link came back, their edge gateway replayed 3.2 million buffered data points over seven hours and the historian reconstructed the gap with zero data loss and original timestamps preserved. The operator did not even notice; the dashboards showed a four-day gap that back-filled in the background. That is the architecture working the way it is supposed to. The third decision, which I regret not making earlier, was explicit tag-rationalisation tooling — the Tag Explosion Problem I describe below hit us internally before it hit our customers, and we spent about nine months building governance around what gets published into the time-series store versus what stays as ephemeral signal. Every manufacturing platform that grows past about 10,000 active tags hits that wall; the plants that survive it are the ones that treat tag discipline as a governance problem, not a storage problem.

The Tag Explosion Problem — and the governance that prevents it

The single most predictable failure mode of long-running industrial data deployments is The Tag Explosion Problem: a historian that was architected, scoped, and priced for 5,000 tags at go-live reaches 25,000 tags by year three and 100,000 tags by year five, and at each step the architectural assumptions that were correct at the previous scale break. The explosion has several reinforcing drivers:

1. Diagnostic creep. Every commissioning engineer who troubleshoots an unusual machine fault adds "just one more" tag to the historian to support future investigation. Multiply by hundreds of machines over years and the population expands linearly with troubleshooting activity.
2. OEM updates. A PLC firmware upgrade from the machine builder adds 200 new diagnostic tags to an existing 50-tag equipment class, and they flow into the historian by default because nobody reviewed the upgrade diff.
3. AI aspiration. Every machine-learning initiative wants "more data" and the path of least resistance is subscribing the historian to additional signals rather than deciding what to measure with discipline.
4. No deprecation discipline. Tags from decommissioned machines, retired product lines, and replaced sensors continue to consume storage and query namespace indefinitely because removing them feels risky and nobody owns the removal process.

The governance that prevents the explosion is not complicated but has to be implemented before the problem gets critical, not after. A tag-lifecycle state machine (proposed / active / deprecated / retired) with ownership per tag; quarterly tag-rationalisation reviews following the same pattern as alarm-management rationalisation; and explicit cost attribution so that each new tag has a visible marginal cost against the team or project that introduced it. Plants that implement this discipline from go-live keep their tag population at 3–5× the original scope over a decade; plants that do not typically see 20–50× expansion and the associated performance and cost consequences.

How this fits into the SYMESTIC platform

SYMESTIC implements the time-series storage layer on the dual-write architecture I describe in the amber section — a compressed operational store on Azure Data Explorer with Swinging-Door-parameterised ingestion for the dashboards and MES queries, and a raw Parquet stream on Azure Blob Storage for compliance, audit, and ML training that customers can export or query directly. The edge gateway layer uses MQTT with Sparkplug B as the canonical wire format to every cloud ingestion endpoint, with 7-day local buffering on every gateway for store-and-forward robustness; the worst network incident we have handled to date (4-day WAN outage, customer in Mexico, late 2022) produced zero data loss and back-filled into the historical record with original timestamps preserved. The UNS pattern is exposed to customers who want it — typically the more architecturally sophisticated ones with parallel analytics, AI, or third-party historian subscriptions — via a documented broker endpoint that mirrors the ingestion stream. Tag governance is built into the platform as first-class lifecycle state, because we have learned from both our own infrastructure growth and from customer deployments that the Tag Explosion Problem is the predictable five-year wall and the plants that implement governance from go-live are the ones that scale without re-architecture. The platform covers the four compression techniques described above (dead-band, box-car, box-car-back-slope, Swinging Door) with per-tag configurability so that signal-specific tuning is possible, defaulting to tight Swinging-Door parameters for safety. Retention tiering is automatic with configurable windows per tag class, aligned with the compliance requirements of the regulated industries we serve (GxP 10-year defaults for pharma customers, 15-year for automotive traceability, indefinite for ETS-governed energy data). The process data module is the customer-facing surface; the historian is the engine underneath it. See also alarm management for the event-layer analogue of the same pattern, shop floor control for the order-lifecycle layer that correlates historian data with business context, process documentation for the ISA-95 architectural framing, and predictive quality for the ML-training workload that the raw Parquet stream supports.

FAQ

What is an industrial data historian?
A purpose-built time-series database for the high-frequency signals produced by industrial control systems, optimised for very high tag counts (tens of thousands to millions), high sample rates (sub-second to hundreds of Hz per tag), long retention horizons (typically 5–20 years for the warm tier), and time-range query patterns. The category was effectively defined by OSIsoft PI, founded in 1980 by J. Patrick Kennedy, and PI (now AVEVA PI System after the 2020 acquisition) remains the reference product that every alternative is legible against. The core architectural distinction from a general-purpose RDBMS is the combination of specialised compression (Swinging Door Trending being the canonical algorithm), tiered storage (hot/warm/cold), and query engines optimised for time-range scans with tag-set projection.

What is the difference between a historian and a general-purpose database?
A general-purpose RDBMS like PostgreSQL or SQL Server breaks on industrial scale because of index bloat, insert latency under sustained high write rates, and absence of time-series-specific compression. At 10,000 tags × 1 Hz sampling, the raw volume is ~27 GB/day; at 100,000 tags × 10 Hz, it is >5 TB/day. No RDBMS ingests that while simultaneously serving trend queries across multi-year windows. A historian adds Swinging-Door-style compression (10–50× reduction on typical industrial signals), specialised tag metadata management, tiered storage, and query engines that execute time-range scans in memory without loading full indexes.

What is Swinging Door Trending?
A compression algorithm published by Edgar Bristol in 1990 that maintains an adaptive tolerance corridor around the trajectory of a signal as it is being recorded, writing a new data point only when the signal would exit the corridor if the previous point were held. The result is that smoothly-varying signals are represented by the minimum number of points needed to reconstruct the trend within a defined tolerance, while sharp transitions are captured faithfully. Typical compression ratios run 10:1 to 50:1 on industrial process data under well-tuned parameters. It became the default compression strategy in OSIsoft PI and is now the reference against which every other industrial compression scheme is evaluated.

What modern alternatives exist to classical historians like OSIsoft PI?
Seven modern time-series databases have meaningful industrial adoption: InfluxDB (2013, observability-origin, good for edge), TimescaleDB (2017, PostgreSQL extension, strong for SQL JOINs with business data), QuestDB (2014, financial-markets origin, extreme ingestion performance), ClickHouse (2016, OLAP-oriented, strong aggregation), Azure Data Explorer / Kusto (2015, cloud-native, Azure ecosystem), AWS Timestream (2020, serverless, AWS-native), and VictoriaMetrics (Prometheus-compatible). Classical historians still dominate in heavily regulated process industries where 40 years of feature depth around calculation engines, unit dictionaries, and compliance tooling is hard to replicate; modern TSDBs dominate greenfield cloud-native deployments and mixed operational + time-series workloads.

What is a Unified Namespace (UNS)?
An architectural pattern popularised by Walker Reynolds from 2020 onward where every meaningful industrial signal is published to a broker (typically MQTT with Sparkplug B payload) organised by a hierarchical namespace (enterprise / site / area / line / cell / device / signal), and every consumer — historian, MES, analytics, dashboards, AI — subscribes to the portion of the namespace it needs. The historian becomes one subscriber among many rather than the mandatory middleware through which all data must flow. The pattern produces linear rather than N-squared integration scaling, native edge autonomy via broker deployment at the edge, and dramatically reduced time-to-integrate for new data consumers. It is the dominant modern architectural alternative to the classical historian-centric pattern, not a complement.

What is store-and-forward and why does it matter?
The ability of an edge gateway to buffer data locally during network outages and replay it to the central historian with original timestamps preserved when connectivity returns. Without store-and-forward, the gap in the historical record is permanent — which in regulated environments is a compliance violation and in operational environments is exactly the data that was most likely to be needed (network outages correlate with the larger plant disruptions that generate the trend data you want to analyse afterwards). Sparkplug B implements this natively with birth/death certificates; classical OPC UA Client-Server does not and has to be augmented with an edge buffer. The question "does my architecture accept out-of-order inserts without breaking compression or query continuity" separates systems that work from systems that lose data during the first week-long WAN outage.

What is the Tag Explosion Problem?
The most predictable failure mode of long-running industrial data deployments: a historian architected, scoped, and priced for 5,000 tags reaches 25,000 tags by year three and 100,000 tags by year five, driven by diagnostic creep during troubleshooting, OEM firmware updates that add bulk-diagnostic tags, AI initiatives that subscribe to additional signals without discipline, and absence of a deprecation process for tags from decommissioned equipment. At each scale step, the architectural assumptions that were correct at the previous scale break. The governance that prevents it — tag-lifecycle state machine, quarterly rationalisation reviews, explicit cost attribution — has to be in place before the problem gets critical, not after.

How does a historian relate to MES, SCADA, and a data lake?
Four layers, distinct responsibilities. SCADA (ISA-95 Level 2) visualises and controls plants in real time with limited short-term history. The historian (ISA-95 Level 2 with Level 3 extensions) is the long-term time-series memory behind SCADA, storing signals at full granularity for years. MES (ISA-95 Level 3) consumes historian data for OEE, order correlation, quality, and maintenance workflows but typically does not store process data at raw granularity. A data lake is generic storage for mixed data types (documents, logs, telemetry, business records) and is usually an export destination from the historian rather than a replacement. Well-architected systems treat these as complementary layers with defined handoff points, not as competing products.

Related: MES: definition, functions & benefits · OEE: definition, calculation & practice · MES software compared · OEE software · Process documentation · Alarm management · Shop floor control (SFC) · Predictive quality · Schedule adherence · On-Time Delivery · Rolled Throughput Yield · Scrap rate vs. rework rate · Recipe management · Change control · MDE (machine data acquisition) · BDE (production data acquisition) · Process data module · Alarms module · Production metrics module · Automotive · Metal processing · Food & beverage · For maintenance managers · For production managers. External references: AVEVA PI System (successor to OSIsoft PI; reference product of the historian category) · OPC Foundation — OPC UA (including PubSub and Alarms & Conditions profiles) · Eclipse Tahu / Sparkplug B specification (MQTT payload standard for industrial pub/sub) · ISA-95 standards family (architectural framing for historian placement in the enterprise stack).

About the author

Mark Kobbert

CTO of SYMESTIC. Responsible for cloud-MES platform architecture since 2014. B.Sc. Business Informatics (SRH Heidelberg, dual programme). Built SYMESTIC's cloud-native platform from scratch — microservices on Microsoft Azure, IoT-gateway connectivity, real-time data processing for 15,000+ machines across 18 countries. Led the mid-2010s platform rebuild from on-premise to cloud-native. Expertise: Cloud-native MES architecture, Microsoft Azure, microservice architecture, OPC UA (including Alarms & Conditions and PubSub profiles), MQTT Sparkplug B, IoT-gateway development, edge computing, ISA-95 integration architecture, industrial connectivity, brownfield machine connectivity, REST APIs, C#/.NET, SQL, Docker/Kubernetes, time-series databases (Azure Data Explorer, TimescaleDB, InfluxDB), Swinging Door compression, Unified Namespace, store-and-forward architectures, real-time data processing, IT/OT convergence. · LinkedIn