Fastmap is a distributed drop-in replacement for
map. As a rule of thumb, it usually runs faster than the
map builtin when execution would otherwise take more than one second. Fastmap can run both on the cloud via fastmap.io's cloud service and on your local machine via multiprocessing. For more about fastmap's features, see the FAQ.
pip install fastmap
Fastmap can run both locally and on fastmap.io's cloud service.
"""Local example""" import fastmap fastmap_config = fastmap.init(exec_policy="LOCAL") def sqrt(x): return x ** .5 results = list(fastmap_config.fastmap(sqrt, range(1000)))
To run using the cloud service, first obtain an API Token. Treat this like a password. Do not commit it to version control! Failure to do so could result in man-in-the-middle attacks or your credits being used by others (e.g. cryptocurrency miners). Then, try:
"""Cloud example""" import fastmap from config import FASTMAP_TOKEN fastmap_config = fastmap.init(exec_policy="CLOUD", secret=FASTMAP_TOKEN, confirm_charges=True) def sqrt(x): return x ** 0.5 results = list(fastmap_config.fastmap(sqrt, range(1000)))
Fastmap may print a message saying that it didn't make sense to use fastmap here. This is true. Fastmap is useful for heavy computations and the square root of a number is one of the lightest computations possible.
Fastmap has exactly three public functions:
- secret - The API token generated on fastmap.io. Treat this like a password. Do not commit it to version control! Failure to do so could result in man-in-the-middle attacks or your credits being used by others (e.g. cryptocurrency miners). If None, fastmap will run locally regardless of exec_policy.
- verbosity: - "QUIET", "NORMAL", or "LOUD". Default is "NORMAL". Consider using the Verbosity namespace.
- exec_policy: - "LOCAL", "CLOUD", or "ADAPTIVE". Default is "ADAPTIVE". To understand how this works, see the ExecPolicy namespace.
- confirm_charges: - Manually confirm cloud charges. Default is False.
- local_processes: - How many local processes to start. By default, fastmap will start a maximum of num_cores * 2 - 2.
Example global_init usage:
import fastmap fastmap.global_init(exec_policy="LOCAL") results = list(fastmap.fastmap(func, iterable))
Example init usage:
import fastmap config = fastmap.init(exec_policy="LOCAL") results = list(config.fastmap(func, iterable))
Map a function over an iterable and return the results. By default, .fastmap is a generator and is executed lazily. To get the results as a sequence, wrap this in a
- func - Function to map against. This must be stateles and cannot access the network or the filesystem. This function may call other functions but the entire execution tree must be limited to the file where fastmap is instantiated.
- iterable - This may be either a sequence (list, tuple, etc.) or a generator.
- return_type - "ELEMENTS" or "BATCHES". Default is "ELEMENTS". For more on how this works, see the ReturnType namespace.
- label - Optional. Arbitrary string to track this execution and differentiate it from others on the dashboard. Only meaningful if some execution occurs on the cloud. This value is not consequential to the functionality of fastmap.
Returned from the init function. Used when it is preferable to pass around a configuration object over a global configuration. Do not instantiate this directly.
- fastmap - Identical to the global fastmap function
- LOCAL - Local processing does not require an API token and utilizes multiple processes on the host machine. It is similar to
multiprocessing.Poolwith a few additional advantages. Namely, fastmap handles generators and performs just-in-time computation. The local execution policy does not utilize the fastmap.io cloud service and is therefore always free.
- CLOUD - The cloud execution policy will offload code to the fastmap.io cloud service and will minimize its use of local processing power. This is helpful if you have a backend server that you would otherwise like to keep performant for other tasks. This execution policy will consume fastmap credits but by default, you will get the option to confirm.
- ADAPTIVE - The adaptive execution policy smartly distributes your job across both local and cloud resources. When running under the this policty, if the fastmap algorithm determines that it wouldn't make sense to run your code in the cloud, it won't and you won't be charged. Like the CLOUD policy, by default, you will get the option to confirm charges before anything is done. The ADAPTIVE policy is the default option and will usually be the fastest.
Convenience namespace class for the return_type argument on the fastmap function.
- ELEMENTS - The default. Using this causes fastmap to yield one element at a time. More intuitive but might be slower for certain workloads.
- BATCHES - Causes fastmap to yield one batch at a time. Batches are what fastmap uses internally to divide work between multiple processes / machines. This eliminates the step of yielding individual batch elements and therefore, in isolation, causes fastmap to run marginally faster. Overall, however, this will only be faster if your code can work with batches directly.
- QUIET - "QUIET"
- NORMAL - "NORMAL"
- LOUD - "LOUD"
Fastmap will throw this exception if every process (worker), both locally and in the cloud, dies. If there is something wrong with your code, chances are good that this is the exception fastmap will throw.