Introduction

Key Concepts

From Theory to Application

Metaform is a schema-free SQL query engine designed for integrating disparate data sources. Unlike traditional relational databases, Metaform was built from the ground up to query data where it lives—whether in files, cloud storage, NoSQL databases, or relational systems—without requiring centralized ingestion or rigid schemas. Understanding Metaform’s core concepts is essential to grasping how it achieves its flexibility, performance, and unique place in the modern data ecosystem.

Query Engine

At the heart of Metaform lies a distributed query engine. Instead of a single monolithic database server, Metaform operates as a cluster of cooperating nodes, each capable of parsing, planning, and executing portions of a query. This architecture allows Metaform to scale horizontally: the more nodes you add, the more data it can handle, and the faster queries can run.

Why it matters
Most enterprise data today is spread across systems too large for one machine to process efficiently. By distributing execution, Metaform enables parallel processing across heterogeneous data sources, delivering interactive performance over billions of records.

Schema-on-Read

Traditional SQL engines expect data to conform to predefined schemas before it can be queried. Metaform reverses this assumption with schema-on-read. Instead of enforcing structure upfront, Metaform interprets the structure of data at query time. A JSON file, for example, does not need to be normalized or loaded into a relational schema before you can analyze it.

Why it matters
This flexibility is crucial in modern analytics, where data often arrives in semi-structured formats, such as JSON. Schema-on-read eliminates the need for upfront modeling, thereby accelerating the path from raw data to insights.

Connectors

Metaform connects to a wide range of data sources through connectors. Each connector encapsulates the logic needed to communicate with a particular system—whether that’s a file system, a NoSQL database, or a relational database. From the user’s perspective, these sources appear as logical “workspaces” that can be queried with standard SQL.

Why it matters
Connectors make Metaform a true data federation layer. They allow anyone to treat disparate systems as a single, queryable fabric without having to move or transform the underlying data.

Schema Discovery

When Metaform encounters data, it dynamically discovers and infers its schema. For structured formats, such as a relational database, it reads the embedded metadata; for less structured formats, like JSON, it inspects the data itself. This discovery is performed at query time, and results are exposed as relational tables.

Why it matters
Dynamic schema discovery reduces the burden of schema management, which is often a bottleneck in traditional analytics environments. It also makes Metaform resilient in the face of evolving data, where fields may be added, removed, or nested over time.

Execution Model

Metaform’s execution model is built on a concept of query fragments distributed across a cluster. The system translates a SQL query into a directed acyclic graph (DAG) of operators—such as scans, joins, and aggregates—which are then executed in parallel on different nodes. Intermediate results are exchanged between nodes to produce the final answer set.

Why it matters
This model gives Metaform both flexibility and performance. It can optimize queries to push computation closer to the data source, leverage parallelism across nodes, and stream results incrementally to the client, enabling interactive exploration even on large datasets.

Metadata

Metaform provides system tables that expose metadata about clusters, queries, functions, and connectors. These tables function as virtual catalogs, enabling administrators and developers to inspect the internal state of Metaform using the same SQL interface they use for data analysis.

Why it matters
System tables close the loop between operational monitoring and data analysis, facilitating seamless integration. They provide visibility into how queries are executed, what resources are available, and how storage plugins are configured—all without leaving the SQL environment.

Putting It All Together

Metaform’s key concepts—distributed execution, schema-on-read, connectors, dynamic discovery, and system metadata—work in concert to deliver a unique approach for data integration. Instead of demanding rigid preparation, Metaform adapts to the data as it exists, wherever it resides. This approach makes it a powerful tool for organizations facing fragmented, fast-changing, or semi-structured data landscapes.

By combining the familiarity of SQL with the flexibility of schema-free querying, Metaform unlocks agile data integration at scale. Understanding these concepts is the first step toward mastering Metaform as both a platform and a mindset for working with modern data.

Quickstart

Install and Configure Metaform

Architecture

Anatomy of the Platform