fastmap.io BETA
faq docs github

fastmap docs

Fastmap offloads arbitrary Python functions. It is designed to run exactly the same locally as it does on the cloud. To use fastmap on the cloud, you must deploy the fastmap cloud service (deployment can be done with one command). For more about fastmap's features, see the FAQ.

These are the docs of the Python client SDK which is the recommended way to run Fastmap. Docs for the cloud service REST API are also available.

installation

Installing the Fastmap client SDK is simple.

pip install fastmap

Fastmap runs the same locally as it does on the cloud. For local execution, the above pip install is all you need. To run using the cloud service, you'll need to deploy a cloud service. Don't worry, it's easier than it sounds like! There is a single deploy script.

quickstart

The first example uses exec_policy="LOCAL" so it doesn't touch the cloud. Instead, it runs in a subprocess on your computer. Everything in fastmap is designed to work the same locally as it does in the cloud.

"""Offload locally"""

import fastmap

fastmap.global_init(exec_policy="LOCAL")

def hello_world():
    print("Logging from a separate process!")
    return "Hello World!"

task = fastmap.offload(hello_world)
print(task.all_logs())  # Will output "Logging from a separate process!" amongst some server info
task.wait()  # Will wait for process to finish before printing "Hello World!"

        

Once you have an account, just change your initialization to run it on the cloud.

"""Offload to the cloud"""

import fastmap

# this line is the only difference from the previous example
fastmap.global_init(secret=YOUR_SECRET_API_TOKEN)

def hello_world():
    print("Logging from the cloud!")
    return "Hello World!"

task = fastmap.offload(hello_world)
print(task.all_logs())  # Will output "A bunch of logs  from the cloud!"
task.wait()  # Will output "Hello World!"

        

Here's an example of some of fastmap's more advanced features. Below, we are passing in parameters and using requests to download a wikipedia page. Requests is a third party pip package. Fastmap will automatically detect these dependencies and install them on the server for you. Notice also that we are returning two things from the function, the content and the page length. We can pass basically anything into functions and get basically anything out.

"""Dealing with parameters"""

# Be sure to actually import requests!
import fastmap
import requests

fastmap.global_init(secret=YOUR_SECRET_API_TOKEN)

def get_wikipedia_page(wiki_page):
    resp = requests.get("https://en.wikipedia.org/wiki/" + wiki_page)
    return resp.content, len(resp.content)

task = fastmap.offload(get_wikipedia_page, kwargs={'wiki_page': 'cloud'})

        

Fastmap never loses track of your tasks.

"""Using poll"""

import fastmap
import requests

fastmap.global_init(secret=YOUR_SECRET_API_TOKEN)

def get_wikipedia_page(wiki_page):
    resp = requests.get("https://en.wikipedia.org/wiki/" + wiki_page)
    return resp.content, len(resp.content)

task = fastmap.offload(get_wikipedia_page, kwargs={'wiki_page': 'cloud'})
del task

all_tasks = fastmap.poll_all()
"""
This returns a list of task dictionaries. A task will look something like:
{
    "type": "OFFLOAD",
    "task_id": "abc123",
    "func_name": "get_wikipedia_page",
    "task_state": "PROCESSING",
    "outcome": None,
    "runtime": 37.313993
    "label": "",
}
"""
task = fastmap.get_task("abc123")
        

Fastmap also has a simple-to-use CLI. You'll have to login first though.

$ fastmap login
Email: user@example.com
Password:
Login successful. Configuration is at /Users/default/.fastmap/default_config.json

$ fastmap poll
Found 1 task(s)
type     func_name            task_id    task_state    outcome    start_time           runtime    label    last_heartbeat
-------  -------------------  ---------  ------------  ---------  -------------------  ---------  -------  ----------------
OFFLOAD  get_wikipedia_page    abcd1234   PROCESSING              2021-07-08 03:12:17                      18 seconds ago
        

Once logged in, you'll no longer need to input your api token when working on your computer. Instead, you can call fastmap.global_init() without parameters.

For more see the machine learning tutorial.

sdk

Init functions:

FastmapConfig/fastmap functions: FastmapTask functions: Classes: Namespaces: Exceptions:

fastmap.global_init(config=None, secret=None, verbosity="NORMAL", exec_policy="CLOUD", machine_type="CPU_1", requirements=None) fastmap.init(config=None, secret=None, verbosity="NORMAL", exec_policy="CLOUD", machine_type="CPU_1", requirements=None)

Initialize fastmap. There are two options for how to do this. Using the fastmap.init() options returns a FastmapConfig object that you can call offload() on. Using fastmap.global_init() still allows you to use offload but without passing a configuration object around.

Parameters:
  • config - Either a JSON file path string or a config dictionary. Every subsequent parameter will alter this configuration. This is optional and if empty, only subsequent parameters will be used.
  • secret - Your API token. Treat this like a password. Do not commit it to version control! Failure to do so could result in man-in-the-middle attacks or your servers being used by others (e.g. cryptocurrency miners). If None, fastmap will run locally regardless of exec_policy.
  • verbosity - "SILENT", "QUIET", "NORMAL", or "LOUD". Default is "NORMAL". Consider using the Verbosity namespace.
  • exec_policy - "LOCAL" or "CLOUD". Default is "CLOUD". To understand how this works, see the ExecPolicy namespace.
  • machine_type - "CPU_1", "CPU_7", or "GPU_7". What kind of worker to launch in the cloud. The number indicates how many GB of memory will be available.
  • requirements - A list of packages to install in "package==1.2.3" style. By default, this is auto-detected. However, you might want to use this if you have dependency issues in the cloud or just to be more explicit.
Returns:

Example global_init usage:

import fastmap

fastmap.global_init(exec_policy="LOCAL")
task = fastmap.offload(func)
task.wait()

Example init usage:

import fastmap

config = fastmap.init(exec_policy="LOCAL")
task = config.offload(func)
task.wait()

fastmap.offload(func, kwargs=None, hook=None, label=None)

Start an offload execution and return a FastmapTask. This is an async operation so it returns long before the execution is completed. If you need something blocking, try using FastmapTask.wait().

This can also be called as a method from a FastmapConfig object.

Parameters:
  • func - Function to map against. This must be stateless and cannot access the network or the filesystem. This function may call other functions but the entire execution tree must be limited to the file where fastmap is instantiated.
  • kwargs - Named parameters (as a dict) to bind to the function. Optional.
  • hook - A function to run locally when execution completes.
  • label - Optional. Arbitrary string to track this task and differentiate it from others. This value is not consequential to the functionality of fastmap.
Returns:

fastmap.get_task(task_id)

Get a FastmapTask object given a task_id. This is lazily executed and if the task doesn't exist, an error will be raised only on the first FastmapTask method call.

This can also be called as a method from a FastmapConfig object.

Parameters:
  • task_id - Unique fastmap task_id string.
Returns:

fastmap.poll_all()

Poll for all non-cleared tasks. See the documentation for poll().

This can also be called as a method from a FastmapConfig object.

Returns:
    A list of task dictionaries. See poll() for details.

fastmap.clear_all()

Clear all DONE tasks. When a task is cleared, the result and logs are deleted and it will no longer be visible by default.

This can also be called as a method from a FastmapConfig object.

Returns:
    List of task dicts. See poll() for details.



FastmapTask.clear()

Clear a single DONE task. When a task is cleared, the result and logs are deleted and it will no longer be visible by default.

Returns:
    The task dictionary. See poll() for details.

FastmapTask.kill()

Start killing the task. Tasks are usually killed within a few seconds.

Returns:
    The task dictionary. See poll() for details.

FastmapTask.all_logs()

Get all the stdout and stderr from a single non-CLEARED task. Also see FastmapTask.new_logs().

Returns:
    A string of everything output to stdout and stderr.

FastmapTask.new_logs()

Get the new stdout and stderr from a single non-CLEARED task. Log position is tracked internally so that each subsequent call only gets the latest logs. Also see FastmapTask.all_logs().

Returns:
    A string of subsequent output to stdout and stderr.

FastmapTask.poll()

Return a dictionary detailing the task's status.

Returns:
    {
        "type": Currently, only OFFLOAD
        "func_name": The function that is running.
        "task_id": Unique fastmap task_id.
        "task_state": See TaskState.
        "outcome": See TaskOutcome.
        "start_time": Unix timestamp.
        "runtime": In seconds
        "label": User-provided label for the task.
        "last_heartbeat": Seconds since last heartbeat.
        "items_uploaded": Unused.
        "items_processed": Unused.
    }
                    

FastmapTask.retry()

Retry a DONE task with any outcome. This will create a new FastmapTask and leave the original unaltered.

Returns:

FastmapTask.return_value()

Get the return value of a DONE task with a SUCCESS outcome. This will have the exact same return type as the function which was run. This method raises an exception for tasks that are not DONE or do not have a SUCCESS outocme.

Returns:
    Whatever the function returns without alteration.

FastmapTask.traceback()

Get the traceback of a DONE task with an ERROR outcome. This method raises an exception for tasks that are not DONE or did not ERROR.

Returns:
    The traceback as a string.

FastmapTask.wait()

Wait for the task to finish and return the task's return_value on SUCESS. If the task is already DONE, this method will return immediately.

Parameters:
  • polling_interval - Time in seconds. Default is 3.
  • live_logs - If True, print logs as they arrive. Default is False.
  • raise_exceptions - If True, raise a FastmapException if the task outcome is not SUCCESS or the task_state moves into KILLING or CLEARED. Otherwise, just log the exception. Default is False.
Returns:
    The return value of the function.



fastmap.FastmapConfig

Returned from fastmap.init(). Use this when it is preferable to pass around a configuration object over a global configuration. Do not instantiate this directly.


fastmap.FastmapTask

Returned from fastmap.offload() or FastmapTask.retry() -->. Offload is asyncronous so you can think of FastmapTask like a "promise" object.




fastmap.ExecPolicy

Namespace for the exec_policy argument on fastmap.init() and fastmap.global_init().

Attributes:
  • CLOUD - The cloud execution policy will offload code to the cloud service. This is the execution policy you probably want in production or when running real workloads.
  • LOCAL - Local processing does not require an API token. It utilizes the multiprocessing module internally while working exactly the same as the cloud execution. This is the execution policy you probably want during development

fastmap.Verbosity

Namespace for the verbosity argument on fastmap.init() and fastmap.global_init().

Attributes:
  • SILENT - Do not log anything
  • QUIET - Log minimally
  • NORMAL - Log normally
  • LOUD - Very loud debug logging

fastmap.MachineType

Namespace for the machine_type argument on fastmap.init() and fastmap.global_init().

Attributes:
  • CPU_1 - Basic machine with 1GB memory
  • CPU_7 - Basic machine with 7GB memory
  • GPU_7 - Machine with attached GPU with 7GB memory

fastmap.TaskState

Namespace for the returned task_state in several functions like FastmapTask.poll().

Attributes:
  • PENDING - The task has not been started by a worker.
  • PROCESSING - The task has been started and is currently running.
  • KILLING - Either the user, a timeout, or some anomaly triggered a KILL on the task which the worker has not yet acted upon.
  • FINISHING - A brief state when the task is completed but the results are not ready.
  • DONE - The task is done and the results are ready.
  • CLEARED - The task has been cleared and logs / results have been deleted.

fastmap.TaskOutcome

Namespace for the returned outcome in several functions like FastmapTask.poll().

Attributes:
  • SUCCESS - Task completed successfully and the result is available.
  • ERROR - Task errored and a traceback is available.
  • KILLED_BY_REQUEST - The user killed this task.
  • KILLED_ZOMBIE - The controller killed this task.



fastmap.FastmapException

Generic and universal fastmap exception. Fastmap will throw this or a descendent of this for all known errors.