Reference

API Reference

Metaform's REST API provides access to Structured Query Language (SQL) query execution, connector configuration and lifecycle management, and service health information and metrics. The Metaform API is served on the same host and port as the Metaform Console and does not require an extra process or extra configuration.

The REST API has the following characteristics:

All requests are written in Javascript Object Notation (JSON).
All responses are written in JSON, except for the response for canceling a running query which is plain text
All operations are stateless
All operations are synchronous
Request responses are not buffered
Returns HTTP error codes and detailed error responses in the response message body

Submit a Query

The query's computation is distributed across all of the Nodes in a cluster.

HTTP Request Structure: POST /query.json

Example curl command:

curl -X POST \
    -H "Content-Type: application/json" \
    -d '{"queryType":"SQL", "query": "SELECT * FROM dfs.`/data/*.txt` records", "autoLimit":0}' \
    http://localhost:8047/query.json

Curl Example Command Breakdown

-X POST specifies what kind of HTTP request to make
-H "Content-Type: application/json" defines the header of the HTTP request
-d specifies curl ASCII data which is comprised of three parts
- queryType - SQL, PHYSICAL, or LOGICAL are valid types. Use only "SQL".
- query - A SQL query that runs in across the Nodes
  - FROM dfs.`/data/*.txt dfs refers to the storage plugin being queried and the path enclosed within backticks is the path to the data to query within the storage plugin
- autoLimit - Limits the number of rows returned from the result set, 0 sets no limit
http://localhost:8047/query.json defines what host to send the curl request to

Example Request Body

This is the same text following -d but formatted onto multiple lines.

{
    "queryType" : "SQL",
    "query" : "<SQL query>",
    "autoLimit" : "<rows to be returned>"
}

Example Request Response

{
    "queryId": "201061a6-9e69-6f65-52fe-84dcea8ebcdc",
    "columns": ["fileHeader", "batches", "fileControl", "numberOfLines", "batchCount"],
    "rows": [{}],
    "metadata": ["MAP", "MAP", "MAP", "BIGINT", "BIGINT"],
    "queryState": "COMPLETED",
    "attemptedAutoLimit": 0
}

View Running Queries

HTTP Request Structure: GET /profiles.json

Example curl command:

curl http://localhost:8047/profiles.json

The default behavior for curl is a GET request, so we don't need to manually specify the request type.

Example Response

A JSON Object is returned containing the status information of all running and finished queries.

{
    "runningQueries": [],
    "finishedQueries": [
        {
            "queryId": "201060bc-9c22-4045-8260-cc55978bd34e",
            "startTime": 1609539394111,
            "endTime": 1609539394985,
            "time": "01/01/2021 17:16:34",
            "link": "http://localhost:8047/profiles/201060bc-9c22-4045-8260-cc55978bd34e.json",
            "foreman": "localhost",
            "query": "SELECT * FROM dfs.`/data/*.txt` records",
            "state": "Succeeded",
            "user": "anonymous",
            "totalCost": 30.0,
            "queueName": "Unknown",
            "duration": "0.874 sec"
        }
    ],
    "errors": [],
    "maxFetchedQueries": 100,
    "queriesPerPage": "10,25,50,100"
}

View a Specific Query

HTTP Request Structure: GET /profiles/{UUID}.json

Example curl command:

curl http://localhost:8047/profiles/201060bc-9c22-4045-8260-cc55978bd34e.json

The text before .json is the UUID of a query.

Parameters

queryid - The UUID of the query in standard UUID format

Example Response

{
    "id": {
        "part1": 2310452972030672965,
        "part2": -9052010583027887282
    },
    ...,
    "queryId": "201060bc-9c22-4045-8260-cc55978bd34e"
}

Cancel a Query

HTTP Request Structure: GET /profiles/cancel/{UUID}

Example curl command:

curl http://localhost:8047/profiles/cancel/201060bc-9c22-4045-8260-cc55978bd34e

Parameters

queryid - The UUID of the running query in standard UUID format

Response Example

Cancelled query 1c3670e0-1fa7-7309-037a-dd5c1ee095e0 on locally running node.

Managing Connectors

View Connectors

HTTP Request Structure: GET /storage.json

Example curl command:

curl http://localhost:8047/storage.json

This will return a JSON Object containing all of the connector configurations.

Example Response

[{
    "name" : "cp",
    "config" : {
        "type" : "file",
        "connection" : "classpath:///",
        "config" : { },
        "workspaces" : { },
        "formats" : {
            "image" : {
                "type" : "image",
                "extensions" : [ "jpg", "jpeg", "jpe", "tif", "tiff", "dng", "psd", "png", "bmp", "gif", "ico", "pcx", "wav", "wave", "avi", "webp", "mov", "mp4", "m4a", "m4p", "m4b", "m4r", "m4v", "3gp", "3g2", "eps", "epsf", "epsi", "ai", "arw", "crw", "cr2", "nef", "orf", "raf", "rw2", "rwl", "srw", "x3f" ]
            }
        },
        "enabled": false,
    }
}]

View a Connector

HTTP Structure: GET /storage/{name}.json

Example curl command

curl http://localhost:8047/storage/cp.json

This will return a JSON Object containing the configuration of the cp storage plugin.

Parameters

name - The assigned name in the connector definition.

Response Example

{
    "name" : "cp",
    "config" : {
        "type" : "file",
        "connection" : "classpath:///",
        "config" : { },
        "workspaces" : { },
        "formats" : {
            "image" : {
                "type" : "image",
                "extensions" : [ "jpg", "jpeg", "jpe", "tif", "tiff", "dng", "psd", "png", "bmp", "gif", "ico", "pcx", "wav", "wave", "avi", "webp", "mov", "mp4", "m4a", "m4p", "m4b", "m4r", "m4v", "3gp", "3g2", "eps", "epsf", "epsi", "ai", "arw", "crw", "cr2", "nef", "orf", "raf", "rw2", "rwl", "srw", "x3f" ]
            }
        },
        "enabled" : true
    }
}

Create a Connector

HTTP Request Structure: POST /storage/{name}.json

Example curl command:

curl -X POST \
    -H "Content-Type: application/json" \
    -d '{"name":"newplugin", "config": {"type": "file", "enabled": true, "connection": "file:///", "workspaces": { "root": { "location": "/", "writable": false, "defaultInputFormat": null}}, "formats": null}}' \
    http://localhost:8047/storage/newplugin.json

This will create a new connector called "newplugin".

Parameters

name - The name of the connector configuration to create or update.

Example Request Body

This is the same as the text following the -d option.

{
    "name" : "<name of storage plugin configuration>",
    "config" : {
        "type" : "<type of storage plugin>",
        "enabled" : <true or false>,
        "connection" : "<type of distributed file system and address/path>",
        "workspaces" : <null or unique workspace name and information>,
        "formats" : <file format definitions>
    }
}

Example Response Body

{
    "result" : "Success"
}

Delete a Connector

HTTP Request Structure: DELETE /storage/{name}.json

Example curl command:

curl http://localhost:8047/storage/cp.json

This will delete the specified connector.

Parameters

name - The name of the connector configuration to delete.

Example Response Body

{
    "result" : "Success"
}

View the Cluster

HTTP Request structure: GET /ensemble.json

Example curl command:

curl http://localhost:8047/ensemble.json

This will return a JSON Object containing the statuses of all of the nodes in the cluster.

Example Response

{
  "currentVersion" : "2.5.1-SNAPSHOT",
  "mismatchedVersions" : [ ],
  "userEncryptionEnabled" : false,
  "bitEncryptionEnabled" : false,
  "authEnabled" : false,
  "ensemble" : [ {
    "address" : "ce955a39dd87",
    "httpPort" : "8047",
    "userPort" : "31010",
    "controlPort" : "31011",
    "dataPort" : "31012",
    "version" : "2.5.1-SNAPSHOT",
    "current" : true,
    "versionMatch" : true,
    "state" : "ONLINE"
  } ]
}

Glossary

Metrics

Public Metrics