fastmap.io BETA
faq docs github

Fastmap docs

Fastmap is a distributed drop-in replacement for map. As a rule-of-thumb, it usually runs faster than the map builtin when execution would otherwise take more than one second. Fastmap can run both on the cloud and on your local machine via multiprocessing. For more about fastmap's features, see the FAQ.

installation

pip install fastmap

quickstart

Fastmap can run both locally and on a deployed cloud service.

"""Local example"""

import fastmap

fastmap_config = fastmap.init(exec_policy="LOCAL")

def sqrt(x):
    return x ** .5

results = list(fastmap_config.fastmap(sqrt, range(1000)))

To run using the cloud service, you'll need to deploy a cloud service. Don't worry, it's easier than it sounds like.

Fastmap may print a message saying that it didn't make sense to use fastmap here. This is true. Fastmap is useful for heavy computations and the square root of a number is one of the lightest computations possible.

sdk interface

Fastmap has exactly three public functions:

Additionally, three namespaces are available: And one custom exception:

fastmap.global_init(secret=None, verbosity="NORMAL", exec_policy="ADAPTIVE", confirm_charges=False, local_processes=None)
fastmap.init(secret=None, verbosity="NORMAL", exec_policy="ADAPTIVE", confirm_charges=False, local_processes=None)

Initialize fastmap. Using init returns a FastmapConfig object that you can call fastmap on. Using global_init allows you to use the fastmap method directly.

Parameters:
  • secret - The API token generated on fastmap.io. Treat this like a password. Do not commit it to version control! Failure to do so could result in man-in-the-middle attacks or your credits being used by others (e.g. cryptocurrency miners). If None, fastmap will run locally regardless of exec_policy.
  • verbosity: - "SILENT", "QUIET", "NORMAL", or "LOUD". Default is "NORMAL". Consider using the Verbosity namespace.
  • exec_policy: - "LOCAL", "CLOUD", or "ADAPTIVE". Default is "ADAPTIVE". To understand how this works, see the ExecPolicy namespace.
  • confirm_charges: - Manually confirm cloud charges. Default is False.
  • local_processes: - How many local processes to start. By default, fastmap will start a maximum of num_cores * 2 - 2.

Example global_init usage:

import fastmap

fastmap.global_init(exec_policy="LOCAL")
results = list(fastmap.fastmap(func, iterable))

Example init usage:

import fastmap

config = fastmap.init(exec_policy="LOCAL")
results = list(config.fastmap(func, iterable))

fastmap.fastmap(func, iterable, return_type="ELEMENTS", label=None)
FastmapConfig.fastmap(func, iterable, return_type="ELEMENTS", label=None)

Map a function over an iterable and return the results. By default, .fastmap is a generator and is executed lazily. To get the results as a sequence, wrap this in a list().

Parameters:
  • func - Function to map against. This must be stateles and cannot access the network or the filesystem. This function may call other functions but the entire execution tree must be limited to the file where fastmap is instantiated.
  • iterable - This may be either a sequence (list, tuple, etc.) or a generator.
  • return_type - "ELEMENTS" or "BATCHES". Default is "ELEMENTS". For more on how this works, see the ReturnType namespace.
  • label - Optional. Arbitrary string to track this execution and differentiate it from others on the dashboard. Only meaningful if some execution occurs on the cloud. This value is not consequential to the functionality of fastmap.
  • dependencies - Optional. Dictionary of dependencies to pass to cloud. If omitted, dependencies are automatically discovered. Dependencies include both local and 3rd-party modules. To pass up a local module, use the form {module_name: complete_file_source_string}. For a 3rd-party module, the format is {module_name: module_name==x.y.z}.

fastmap.FastmapConfig

Returned from the init function. Used when it is preferable to pass around a configuration object over a global configuration. Do not instantiate this directly.

Methods:
  • fastmap - Identical to the global fastmap function

fastmap.ExecPolicy

Convenience namespace class for the exec_policy argument on init and global_init.

Attributes:
  • LOCAL - Local processing does not require an API token and utilizes multiple processes on the host machine. It is similar to multiprocessing.Pool with a few additional advantages. Namely, fastmap handles generators and performs just-in-time computation. The local execution policy does not utilize the cloud and is therefore always free.
  • CLOUD - The cloud execution policy will offload code to a cloud service and will minimize its use of local processing power. This is helpful if you are running fastmap on a backend that you would otherwise like to keep performant for other tasks.
  • ADAPTIVE - The adaptive execution policy smartly distributes your job across both local and cloud resources. When running under the this policy, if the fastmap algorithm determines that it wouldn't make sense to run your code in the cloud, it won't. The ADAPTIVE policy is the default option and will usually be the fastest.

fastmap.ReturnType

Convenience namespace class for the return_type argument on the fastmap function.

Attributes:
  • ELEMENTS - The default. Using this causes fastmap to yield one element at a time. More intuitive but might be slower for certain workloads.
  • BATCHES - Causes fastmap to yield one batch at a time. Batches are what fastmap uses internally to divide work between multiple processes / machines. This eliminates the step of yielding individual batch elements and therefore, in isolation, causes fastmap to run marginally faster. Overall, however, this will only be faster if your code can work with batches directly.

fastmap.Verbosity

Convenience namespace class for the verbosity argument on init and global_init.

Attributes:
  • SILENT - "SILENT"
  • QUIET - "QUIET"
  • NORMAL - "NORMAL"
  • LOUD - "LOUD"

fastmap.FastmapException

Generic and universal fastmap exception. Fastmap will throw this if it cannot finish processing.