Fastmap API
The Python SDK is the recommended way to use fastmap. It is designed to provide a drop-in replacement for
map and also handles the tricky bits like:
- Appropriately sizing batches
- Automatically extracting code / modules to upload
- Handling security and signatures
- Providing progress feedback
Warning: The API is experimental. Docs might not be up to date.
The fastmap API attempts to be as consistent as possible. Therefore, every endpoint shares several features:
- Every endpoint is a POST endpoint.
- Every endpoint expects its payload to be a MessagePack encoded object.
- The returned status will always be contained in the X-Status response header
- The response body will either be MessagePack encoded object (application/msgpack) or a gzipped & pickled binary object (application/octet-stream). You can tell the difference with the header. The latter is only for the /api/v1/map endpoint.
- If the response is a binary object, that response will also be HMAC signed with the latter 32 characters of your fastmap token.
- Authorization: "Bearer " + The first 32 characters of your fastmap token.
- X-Content-Signature: A SHA256 HMAC digest of your payload signed using the latter 32 characters of your fastmap token.
- X-Python-Version: E.g. "3.6.3"
POST /api/v1/auth
Convenience endpoint to check the token. The payload must be an empty dictionary.
Response statuses:
- OK - Everything works.
POST /api/v1/init
Upload a function along with its dependencies to the cloud. Because the maximum payload size is 20MB, this endpoint allows for the payload to be uploaded in parts.
Payload parts:
- payload_len - Number of payload parts. Optional. Default of 1.
- payload_part - Index of this part. Optional. Default of 0.
- func_hash - The first 16 characters of the sha256 hash of the dill pickled function. Required.
- func - Partial or full function dictionary. The function directory itself is a gzipped, MessagePack-encoded, dictionary of four items:
- func - The dill pickled function.
- local_sources - A dictionary of local dependencies with a str module_name -> str module_src format.
- installed_mods - A list of non-local module names. The server imports both these and local_sources before running the server.
- requirements - A list of requirements in
pip freezepkg==version format
- UPLOADED - Uploaded the part sent.
- EXISTS - The func dictionary which corresponds to the func_hash is in storage. Only triggered when the func is not sent.
- DOES_NOT_EXIST - The func dictionary which corresponds to the func_hash is NOT in storage. Only triggered when the func is not sent.
POST /api/v1/map
Map the provided batch over the function referenced by func_hash.
Payload parts:
- func_hash - The first 16 characters of the sha256 hash of the dill pickled function. Required.
- batch - The dill-pickled & gzipped iterable to map over. Required.
- label - An optional label for this map. Intended for tracking the same map instance over multiple executions. Optional.
- run_id - A unique id for this single map which is consistent across every batch. Use
secrets.token_hex()to generate this. Required.
- BATCH_PROCESSED - The batch was processed successfully and can be found in the "results" array. Various timing / credit data is also returned as headers.
- INITIALIZING - Server needs time to initialize worker. Retry in 5 seconds.
- INITIALIZATION_ERROR - Server needs time to initialize worker. Retry in 5 seconds.
- PROCESS_ERROR - Server was ready but something went wrong when running code.
POST /api/v1/done
Call this endpoint after processing is completed in order to free resources and stop charges. Resources are automatically freed after a few minutes but it's recommended that this endpoint be called once workers are no longer needed. Currently will always return DONE.
Payload parts:
- func_hash - The first 16 characters of the sha256 hash of the dill pickled function. Required.
- run_id - A unique id for this single map which is consistent across every batch. This should be the same run_id you passed into map.
- DONE - Resources are scheduled for shutdown or already shutdown.
- NOT_FOUND - Resources were not found. This might be because they were already shutdown and the record was deleted.