Pregel HTTP API

The HTTP API for Pregel lets you execute, cancel, and list Pregel jobs

See Distributed Iterative Graph Processing (Pregel) for details.

Start Pregel job execution

Start the execution of a Pregel algorithm

POST /_api/control_pregel

Request Body

  • algorithm (string, required): Name of the algorithm. One of:
    • "pagerank" - Page Rank
    • "sssp" - Single-Source Shortest Path
    • "connectedcomponents" - Connected Components
    • "wcc" - Weakly Connected Components
    • "scc" - Strongly Connected Components
    • "hits" - Hyperlink-Induced Topic Search
    • "effectivecloseness" - Effective Closeness
    • "linerank" - LineRank
    • "labelpropagation" - Label Propagation
    • "slpa" - Speaker-Listener Label Propagation

  • graphName (string, optional): Name of a graph. Either this or the parameters vertexCollections and edgeCollections are required. Please note that there are special sharding requirements for graphs in order to be used with Pregel.

  • vertexCollections (array of strings, optional): List of vertex collection names. Please note that there are special sharding requirements for collections in order to be used with Pregel.

  • edgeCollections (array of strings, optional): List of edge collection names. Please note that there are special sharding requirements for collections in order to be used with Pregel.

  • params (object, optional): General as well as algorithm-specific options.

    The most important general option is “store”, which controls whether the results computed by the Pregel job are written back into the source collections or not.

    Another important general option is “parallelism”, which controls the number of parallel threads that work on the Pregel job at most. If “parallelism” is not specified, a default value may be used. In addition, the value of “parallelism” may be effectively capped at some server-specific value.

    The option “useMemoryMaps” controls whether to use disk based files to store temporary results. This might make the computation disk-bound, but allows you to run computations which would not fit into main memory. It is recommended to set this flag for larger datasets.

    The attribute “shardKeyAttribute” specifies the shard key that edge collections are sharded after (default: "vertex").

To start an execution you need to specify the algorithm name and a named graph (SmartGraph in cluster). Alternatively you can specify the vertex and edge collections. Additionally you can specify custom parameters which vary for each algorithm, see Pregel - Available Algorithms.

Responses

HTTP 200: HTTP 200 is returned in case the Pregel was successfully created and the reply body is a string with the id to query for the status or to cancel the execution.

HTTP 400: An HTTP 400 error is returned if the set of collections for the Pregel job includes a system collection, or if the collections to not conform to the sharding requirements for Pregel jobs.

HTTP 403: An HTTP 403 error is returned if there are not sufficient privileges to access the collections specified for the Pregel job.

HTTP 404: An HTTP 404 error is returned if the specified “algorithm” is not found, or the graph specified in “graphName” is not found, or at least one the collections specified in “vertexCollections” or “edgeCollections” is not found.

Examples

Run the Weakly Connected Components (WCC) algorithm against a graph and store the results in the vertices as attribute component:

shell> curl -X POST --header 'accept: application/json' --data-binary @- --dump - http://localhost:8529/_api/control_pregel <<EOF
{ 
  "algorithm" : "wcc", 
  "graphName" : "connectedComponentsGraph", 
  "params" : { 
    "maxGSS" : 36, 
    "resultField" : "component" 
  } 
}
EOF

HTTP/1.1 200 OK
content-type: application/json
cache-control: no-cache, no-store, must-revalidate, pre-check=0, post-check=0, max-age=0, s-maxage=0
connection: Keep-Alive
content-length: 7
content-security-policy: frame-ancestors 'self'; form-action 'self';
expires: 0
pragma: no-cache
server: ArangoDB
strict-transport-security: max-age=31536000 ; includeSubDomains
x-arango-queue-time-seconds: 0.000000
x-content-type-options: nosniff

"70387"

Get Pregel job execution status

Get the status of a Pregel execution

GET /_api/control_pregel/{id}

Path Parameters

  • id (number, required): Pregel execution identifier.

Returns the current state of the execution, the current global superstep, the runtime, the global aggregator values as well as the number of sent and received messages.

Responses

HTTP 200: HTTP 200 is returned in case the job execution ID was valid and the state is returned along with the response.

  • id (string): The ID of the Pregel job, as a string.

  • algorithm (string): The algorithm used by the job.

  • created (string): The date and time when the job was created.

  • expires (string): The date and time when the job results expire. The expiration date is only meaningful for jobs that were completed, canceled or resulted in an error. Such jobs are cleaned up by the garbage collection when they reach their expiration date/time.

  • ttl (number): The TTL (time to live) value for the job results, specified in seconds. The TTL is used to calculate the expiration date for the job’s results.

  • state (string): The state of the execution. The following values can be returned:
    • "none": The Pregel run did not yet start.
    • "loading": The graph is loaded from the database into memory before the execution of the algorithm.
    • "running": The algorithm is executing normally.
    • "storing": The algorithm finished, but the results are still being written back into the collections. Occurs only if the store parameter is set to true.
    • "done": The execution is done. In version 3.7.1 and later, this means that storing is also done. In earlier versions, the results may not be written back into the collections yet. This event is announced in the server log (requires at least info log level for the pregel log topic).
    • "canceled": The execution was permanently canceled, either by the user or by an error.
    • "fatal error": The execution has failed and cannot recover.
    • "in error": The execution is in an error state. This can be caused by DB-Servers being not reachable or being non responsive. The execution might recover later, or switch to "canceled" if it was not able to recover successfully.
    • "recovering" (currently unused): The execution is actively recovering and switches back to running if the recovery is successful.

  • gss (integer): The number of global supersteps executed.

  • totalRuntime (number): The total runtime of the execution up to now (if the execution is still ongoing).

  • startupTime (number): The startup runtime of the execution. The startup time includes the data loading time and can be substantial.

  • computationTime (number): The algorithm execution time. Is shown when the computation started.

  • storageTime (number): The time for storing the results if the job includes results storage. Is shown when the storing started.

  • gssTimes (array of numbers): Computation time of each global super step. Is shown when the computation started.

  • reports (array of objects): This attribute is used by Programmable Pregel Algorithms (ppa, experimental). The value is only populated once the algorithm has finished.

  • vertexCount (integer): The total number of vertices processed.

  • edgeCount (integer): The total number of edges processed.

  • detail (object): The Pregel run details.

    • aggregatedStatus (object): The aggregated details of the full Pregel run. The values are totals of all the DB-Server.

      • timeStamp (string): The time at which the status was measured.

      • graphStoreStatus (object): The status of the in memory graph.

        • verticesLoaded (integer): The number of vertices that are loaded from the database into memory.

        • edgesLoaded (integer): The number of edges that are loaded from the database into memory.

        • memoryBytesUsed (integer): The number of bytes used in-memory for the loaded graph.

        • verticesStored (integer): The number of vertices that are written back to the database after the Pregel computation finished. It is only set if the store parameter is set to true.

      • allGssStatus (object): Information about the global supersteps.

        • items (array): A list of objects with details for each global superstep.

          • verticesProcessed (integer): The number of vertices that have been processed in this step.

          • messagesSent (integer): The number of messages sent in this step.

          • messagesReceived (integer): The number of messages received in this step.

          • memoryBytesUsedForMessages (integer): The number of bytes used in memory for the messages in this step.

    • workerStatus (object): The details of the Pregel for every DB-Server. Each object key is a DB-Server ID, and each value is a nested object similar to the aggregatedStatus attribute. In a single server deployment, there is only a single entry with an empty string as key.

      HTTP 404: An HTTP 404 error is returned if no Pregel job with the specified execution number is found or the execution number is invalid.

Examples

Get the execution status of a Pregel job:

shell> curl --header 'accept: application/json' --dump - http://localhost:8529/_api/control_pregel/70682

HTTP/1.1 200 OK
content-type: application/json
cache-control: no-cache, no-store, must-revalidate, pre-check=0, post-check=0, max-age=0, s-maxage=0
connection: Keep-Alive
content-length: 3048
content-security-policy: frame-ancestors 'self'; form-action 'self';
expires: 0
pragma: no-cache
server: ArangoDB
strict-transport-security: max-age=31536000 ; includeSubDomains
x-arango-queue-time-seconds: 0.000000
x-content-type-options: nosniff
Show response body

Get currently running Pregel jobs

Get the overview of currently running Pregel jobs

GET /_api/control_pregel

Returns a list of currently running and recently finished Pregel jobs without retrieving their results.

Responses

HTTP 200: Is returned when the list of jobs can be retrieved successfully.

  • (array): A list of objects describing the Pregel jobs.

    • id (string): The ID of the Pregel job, as a string.

    • algorithm (string): The algorithm used by the job.

    • created (string): The date and time when the job was created.

    • expires (string): The date and time when the job results expire. The expiration date is only meaningful for jobs that were completed, canceled or resulted in an error. Such jobs are cleaned up by the garbage collection when they reach their expiration date/time.

    • ttl (number): The TTL (time to live) value for the job results, specified in seconds. The TTL is used to calculate the expiration date for the job’s results.

    • state (string): The state of the execution. The following values can be returned:
      • "none": The Pregel run did not yet start.
      • "loading": The graph is loaded from the database into memory before the execution of the algorithm.
      • "running": The algorithm is executing normally.
      • "storing": The algorithm finished, but the results are still being written back into the collections. Occurs only if the store parameter is set to true.
      • "done": The execution is done. In version 3.7.1 and later, this means that storing is also done. In earlier versions, the results may not be written back into the collections yet. This event is announced in the server log (requires at least info log level for the pregel log topic).
      • "canceled": The execution was permanently canceled, either by the user or by an error.
      • "fatal error": The execution has failed and cannot recover.
      • "in error": The execution is in an error state. This can be caused by DB-Servers being not reachable or being non responsive. The execution might recover later, or switch to "canceled" if it was not able to recover successfully.
      • "recovering" (currently unused): The execution is actively recovering and switches back to running if the recovery is successful.

    • gss (integer): The number of global supersteps executed.

    • totalRuntime (number): The total runtime of the execution up to now (if the execution is still ongoing).

    • startupTime (number): The startup runtime of the execution. The startup time includes the data loading time and can be substantial.

    • computationTime (number): The algorithm execution time. Is shown when the computation started.

    • storageTime (number): The time for storing the results if the job includes results storage. Is shown when the storing started.

    • gssTimes (array of numbers): Computation time of each global super step. Is shown when the computation started.

    • reports (array of objects): This attribute is used by Programmable Pregel Algorithms (ppa, experimental). The value is only populated once the algorithm has finished.

    • vertexCount (integer): The total number of vertices processed.

    • edgeCount (integer): The total number of edges processed.

    • detail (object): The Pregel run details.

      • aggregatedStatus (object): The aggregated details of the full Pregel run. The values are totals of all the DB-Server.

        • timeStamp (string): The time at which the status was measured.

        • graphStoreStatus (object): The status of the in memory graph.

          • verticesLoaded (integer): The number of vertices that are loaded from the database into memory.

          • edgesLoaded (integer): The number of edges that are loaded from the database into memory.

          • memoryBytesUsed (integer): The number of bytes used in-memory for the loaded graph.

          • verticesStored (integer): The number of vertices that are written back to the database after the Pregel computation finished. It is only set if the store parameter is set to true.

        • allGssStatus (object): Information about the global supersteps.

          • items (array): A list of objects with details for each global superstep.

            • verticesProcessed (integer): The number of vertices that have been processed in this step.

            • messagesSent (integer): The number of messages sent in this step.

            • messagesReceived (integer): The number of messages received in this step.

            • memoryBytesUsedForMessages (integer): The number of bytes used in memory for the messages in this step.

      • workerStatus (object): The details of the Pregel for every DB-Server. Each object key is a DB-Server ID, and each value is a nested object similar to the aggregatedStatus attribute. In a single server deployment, there is only a single entry with an empty string as key.

Examples

Get the status of all active Pregel jobs:

shell> curl --header 'accept: application/json' --dump - http://localhost:8529/_api/control_pregel/

HTTP/1.1 200 OK
content-type: application/json
cache-control: no-cache, no-store, must-revalidate, pre-check=0, post-check=0, max-age=0, s-maxage=0
connection: Keep-Alive
content-length: 3520
content-security-policy: frame-ancestors 'self'; form-action 'self';
expires: 0
pragma: no-cache
server: ArangoDB
strict-transport-security: max-age=31536000 ; includeSubDomains
x-arango-queue-time-seconds: 0.000000
x-content-type-options: nosniff
Show response body

Cancel Pregel job execution

Cancel an ongoing Pregel execution

DELETE /_api/control_pregel/{id}

Path Parameters

  • id (number, required): Pregel execution identifier.

Cancel an execution which is still running, and discard any intermediate results. This immediately frees all memory taken up by the execution, and makes you lose all intermediary data.

You might get inconsistent results if you requested to store the results and then cancel an execution when it is already in its "storing" state (or "done" state in versions prior to 3.7.1). The data is written multi-threaded into all collection shards at once. This means there are multiple transactions simultaneously. A transaction might already be committed when you cancel the execution job. Therefore, you might see some updated documents, while other documents have no or stale results from a previous execution.

Responses

HTTP 200: HTTP 200 is returned if the job execution ID was valid.

HTTP 404: An HTTP 404 error is returned if no Pregel job with the specified execution number is found or the execution number is invalid.

Examples

Cancel a Pregel job to stop the execution or to free up the results if it was started with "store": false and is in the done state:

shell> curl -X DELETE --header 'accept: application/json' --dump - http://localhost:8529/_api/control_pregel/70240

HTTP/1.1 200 OK
content-type: application/json
cache-control: no-cache, no-store, must-revalidate, pre-check=0, post-check=0, max-age=0, s-maxage=0
connection: Keep-Alive
content-length: 2
content-security-policy: frame-ancestors 'self'; form-action 'self';
expires: 0
pragma: no-cache
server: ArangoDB
strict-transport-security: max-age=31536000 ; includeSubDomains
x-arango-queue-time-seconds: 0.000000
x-content-type-options: nosniff

""
❮  User-defined AQL functions

© ArangoDB - Graph and Beyond