Files

PCAP

Packet Capture Format File Connector

Introduction

Packet capture (PCAP) files are among the richest and most underused sources of operational data. They contain a complete record of network activity, including requests, responses, handshakes, and anomalies at the packet level. Traditionally, analyzing PCAP data requires specialized tools and workflows that isolate network analysis from the rest of your data.

The Packet Capture connector removes that barrier.

It allows you to query PCAP files directly using SQL, treating network traffic as structured data. There is no need to export, transform, or load data into another system. Each packet becomes a row, and attributes such as IP addresses, ports, timestamps, and protocol metadata become queryable fields.

This approach enables you to filter, aggregate, and join PCAP data with other sources like logs, transactions, or system metrics in place. You can investigate anomalies, analyze traffic patterns, and detect security issues using the same tools and queries you already use elsewhere.

Scenario

Imagine receiving packet capture files after a suspected security incident on a production system. At first glance, the data appears complete—every packet is present, including source and destination addresses, ports, timestamps, and payload fragments. But that completeness is deceptive.

As soon as you begin investigating, the structure breaks down. The data is fragmented across packets. Protocols shift from one exchange to the next. Sessions are implied but never defined. What looks like a coherent record is actually millions of low-level events with no obvious boundaries or narrative. Finding meaning in this data is not about access. It is about reconstruction.

The first question is simple but critical: which systems are involved?

Putting It Together

With Metaform, the friction described in the scenario disappears.

Once the PCAP connector is configured, each capture is exposed as a queryable, structured dataset. There is no need to reconstruct sessions manually or preprocess the file. You can move directly from raw packets to analysis using standard SQL.

SELECT DISTINCT
  src_ip,
  dst_ip
FROM pcap.`data.pcap`;

In a single query, you can identify the systems participating in the capture, turning millions of low-level packet records into a clear, high-level view of network activity.

Working Example

To explore how Metaform handles real-world PCAP data, like the packet captures in this scenario, walk through the steps below in order.

Each step introduces a key part of the workflow.

Installing the Connector

The PCAP connector is provided as a standalone JSON descriptor that registers a new connector with your Metaform instance. This descriptor declares the connector type, the file extensions it supports, and all available parsing and extraction parameters. Installing the connector is a one-time operation that loads these capabilities directly into Metaform’s storage subsystem.

To install the connector, run the command below for your operating system’s Docker Desktop environment.

# Download the connector JSON and pipe it directly into Metaform's connector API
curl -sSL https://docs.metaform.com/resources/examples/pcap-connector.json \
  | curl -X POST -H "Content-Type: application/json" -d @- http://localhost:8047/storage/pcap.json

This command retrieves the connector definition and registers it with Metaform through the REST API. After registration, the pcap connector appears in the Connectors tab and becomes available for immediate use in queries—no service restart or additional configuration required.

Download the Data

With the connector installed, you can now retrieve the example PCAP file.

Download it using the following URL:

https://docs.metaform.com/resources/examples/data.pcap

Save the file to a location accessible to your Metaform instance (for example, ~/data.pcap).

Review the Data

The example PCAP represents network traffic as a collection of packet-level records. Each row corresponds to an individual packet and includes fields such as source and destination IP addresses, ports, protocol, timestamps, and packet size.

Unlike traditional structured datasets, these records capture discrete events rather than complete transactions. Higher-level constructs such as sessions or flows are not explicitly defined and must be inferred through analysis.

Key attributes, most notably src_ip, dst_ip, protocol, and timestamp, can be queried, grouped, and aggregated to identify communication patterns, detect anomalies, and understand system interactions within the capture.

Query the Data

With the connector installed and the example PCAP downloaded, you can now execute the query in the Metaform Console to validate extraction and examine the resulting structure.

Copy the SQL statement below and paste it into the query field within the Query tab of the Metaform Console, then run the query to view the parsed dataset.

SELECT DISTINCT
  src_ip,
  dst_ip
FROM pcap.`data.pcap`;

What You Should See

If the connector is installed, the example PCAP file is available in your working directory, and the query executes successfully, Metaform will:

  • Detect the .pcap file through the registered PCAP connector
  • Expose packet-level fields such as src_ip, dst_ip, ports, protocol, and timestamps
  • Return a result set based on the query, such as distinct communicating IP pairs

At this point, you have validated the complete workflow: Metaform located the PCAP file, interpreted the packet structure, and exposed the capture as a queryable dataset.

Microsoft Excel

Microsoft Excel File Connector

PDF

Portable Document Format File Connector