A data lake sounds like a storage decision, but for upstream oil and gas operations the hard part is almost never storage, it is the four different protocols standing between your field data and anything queryable. SCADA speaks OPC UA, your historian speaks PI Web API, geological data lives in WITSML, and production systems expose REST APIs, and none of them were designed with the others in mind. Building a queryable, AI-ready data lake means bridging all four cleanly, and you can book a demo to see this architecture applied to your own systems.
How to Build a Unified Data Lake for Upstream Oil and Gas Operations
An architecture guide to ingesting SCADA via OPC UA, historian data via PI Web API, geological data via WITSML, and production data via REST into one queryable, AI-ready upstream data lake.
Four Protocols, One Data Lake, and No Shortcuts Between Them
Most upstream data lake projects underestimate the protocol bridging work because each individual connection looks straightforward in isolation. OPC UA to SCADA is a documented, standard interface. PI Web API to a historian is equally well documented. The difficulty appears when all four sources need to land in a common, well-indexed schema that an AI model or analyst can actually query without knowing which source system a given field originally came from.
The Four Data Sources and How Each One Connects
Each source system requires a different connection method and produces a different native data shape, and understanding those differences up front is what keeps the ingestion layer from becoming a permanent maintenance burden.
| Source | Protocol | Data Type | Typical Frequency |
|---|---|---|---|
| SCADA | OPC UA | Real-time flow, pressure, status tags | Seconds to minutes |
| Historian | PI Web API | Time-series trend and event data | Sub-minute to hourly |
| Geological Data | WITSML | Well logs, drilling and completion data | Event-based, per well |
| Production Systems | REST API | Allocation, test, and accounting data | Daily to monthly |
How the Data Lake Is Structured Once It Is Built
A well-built upstream data lake separates raw ingestion from queryable structure, so that new source systems can be added later without reworking everything built on top of the existing schema.
Raw Ingestion Zone
Data lands in its native structure directly from each protocol connection, preserved exactly as received so nothing is lost if downstream schema decisions need to be revisited later.
Normalized Schema Zone
Raw data is mapped into a common well-centric schema, with unit conversion, timestamp alignment, and well-ID resolution applied consistently across all four source types.
AI-Ready Query Zone
Normalized data is indexed and exposed for direct querying by analytics tools and AI models, with well-level, field-level, and time-range queries all returning consistent results regardless of source.
Skip the Months of Protocol Bridging Trial and Error
iFactory's platform already speaks OPC UA, PI Web API, WITSML, and REST, so the schema reconciliation work that consumes most build timelines is handled from day one. Book a demo and see the architecture applied to your own source systems.
A Step-by-Step Approach to Building Your Data Lake
The build sequence below reflects the order that avoids the most common cause of scope creep and schedule overrun: designing the normalized schema before ingestion starts rather than after.
Inventory Every Source System and Protocol Version
Catalogue every SCADA, historian, WITSML, and production system in use, including specific protocol versions, since older OPC UA and WITSML implementations can differ meaningfully from current standards.
Design the Normalized Schema First
Define the well-centric target schema, including well-ID conventions and unit standards, before any ingestion pipeline is built, so every connector is written against a fixed target rather than a moving one.
Build Ingestion Connectors by Protocol
Connect OPC UA, PI Web API, WITSML, and REST sources into the raw ingestion zone, validating data completeness against the source system before moving to normalization.
Apply Normalization and Validation Rules
Map raw data into the normalized schema with automated validation checks that flag unit mismatches, missing well-IDs, and timestamp misalignment before they propagate into the query layer.
Expose the AI-Ready Query Layer
Index the normalized data and expose it for analytics and AI model queries, confirming that well-level and field-level queries return consistent results regardless of originating source system.
How to Confirm the Data Lake Is Actually AI-Ready
A data lake that stores everything but cannot be queried reliably is not AI-ready, it is just a larger version of the fragmented storage problem it was meant to solve. The checklist below covers the validation points most builds skip until an AI model surfaces the gap downstream.
Well-ID Consistency
Every source system's well identifier resolves to the same canonical well record, with no orphaned or duplicate entries across SCADA, historian, WITSML, and production data.
Unit Standardization
Pressure, flow, and volume units are converted consistently across all sources, with no mixed-unit fields that would silently corrupt AI model training or query results.
Timestamp Alignment
Time zones and sampling intervals are normalized across sources, so a query joining real-time SCADA data with daily production data returns a coherent, correctly aligned result.
Query Performance at Scale
Well-level and field-level queries return results within an acceptable window even as historical data volume grows, validated under realistic multi-year data retention loads.
What Changes as You Add More Wells, Years, and Source Systems
A data lake architecture that performs well during initial build and testing does not automatically stay performant as data volume grows across additional wells, additional years of history, and additional source systems added after the initial build. Planning for that growth during the original schema design avoids a costly re-architecture later.
Historical data retention is often the first scaling pressure teams encounter, since multi-year SCADA and historian data volumes grow quickly once every well in a large portfolio is included. Query performance needs to be validated not just against current data volume but against the volume expected two or three years into the platform's life, particularly for time-range queries that span the full retention window.
Adding new source systems after the initial build, such as a newly acquired field running a different SCADA vendor, is far less disruptive when the normalized schema was designed to be source-agnostic from the start, since the new system only needs a new ingestion connector rather than a redesign of the schema itself.
Who Should Own the Data Lake Once It Is Built
A data lake without a clear owner tends to drift out of sync with the source systems it depends on, as protocol versions change and new wells come online without anyone responsible for keeping the ingestion layer current.
IT and Data Engineering
Typically owns ingestion pipeline health, schema versioning, and query performance, maintaining the technical infrastructure as source systems evolve.
Reservoir and Production Engineering
Defines the well-centric schema requirements and validates that normalized data accurately reflects what engineering teams need for analysis and AI model input.
What Becomes Possible Once the Query Layer Is Live
The value of a unified data lake is rarely visible during the build itself, since most of that work is invisible plumbing that only pays off once analysts and AI models can query across sources they previously could not touch together. A reservoir engineer investigating a production anomaly can query WITSML completion data alongside real-time SCADA pressure trends in a single request, rather than pulling each dataset separately and reconciling them manually, which is often where the earliest and clearest wins from the project appear.
Over time, the AI-ready query layer also becomes the foundation for more advanced applications, such as training predictive models across the full historical dataset or running field-wide comparative analysis that would have required a dedicated data engineering effort under the old fragmented system. The data lake, once built correctly, tends to keep paying off in ways the original project scope did not fully anticipate.
Common Questions From Data and Engineering Teams
Build Your Upstream Data Lake on Architecture That Already Works
Connect SCADA, historian, WITSML, and production data into one AI-ready query layer without months of protocol bridging trial and error.
