Simple distributed computing
Fastmap is a drop-in replacement for map that makes your Python code run in parallel on the cloud.
global_init(...) to the top of your file, and replace every instance of map with fastmap. There are no servers to provision or code to upload. The SDK consists of only 3 functions.Stop fighting infrastructure.
Parallelize your code in minutes, not months.
- No credit card required.
- New accounts get 1000 free credits.
- 1 vCPU-hour = 10 credits
- 1 GB egress = 10 credits
- Refills available at $10 for 1000 credits.
Questions
▹ Why use fastmap?
As a data scientist: Most of your time is spent pre-processing data and managing servers. This wasn't what you were hired for. You were hired to analyze data and generate insights. Use fastmap to spend less time fighting infrastructure and more time doing things that matter.
As a backend engineer: Parallelizing code on the server is an essential yet non-trivial task. Instead of provisioning services, auditing security, and building release pipelines, why not use a service that requires no provisioning, works the same locally as it does in the cloud, and is simple enough for a summer intern to setup?
As an academic: You spend too much of your time reading OpenMP documentation and not enough time developing and simulating models. What matters is getting your research done - not configuring technology. Need more than the free tier but don't have grant money? Email me at scott@fastmap.io to ask about more free credits.
▹ When is fastmap appropriate to use?
Fastmap is best when map is too slow but setting up a Spark cluster or deploying Lambdas would be overkill.
As a rule-of-thumb, fastmap will speed up any call to map that would have otherwise taken more than one second. This is possible because, by default, fastmap algorithmically distributes work between local execution and the fastmap.io cloud service.
If in doubt, try running fastmap with a small test dataset. Fastmap is transparent and will inform you when using the service has made your code slower.
▹ How much does fastmap cost?
Fastmap is a prepaid service that uses "fastmap credits". These credits are available at a rate of $10 for 1000 credits. New accounts can request 1000 credits for free.
Then, credits are consumed at the following rates:
- 1 vCPU hour = 10 credits
- 1 GB egress = 10 credits
Credits are available in $10 increments and are not refundable.
▹ What versions of Python are available?
Fastmap works with Python 3.6+. Python 2 is not supported.
▹ What sort of functions can fastmap process?
- Stateless: The function should not use or alter mutable global state. In other words, it must be a pure function.
- No IO: The function cannot access the network or the filesystem. This is for security reasons. Eventually, expect some network access to be possible.
- Imports OK: Fastmap seamlessly handles secondary-function calls and imports of most local modules.
- Standard library + Numpy/Pandas: Any other installed third-party library will not work in the cloud. Eventually, expect fastmap to support any third-party library.
▹ What sort of iterables can fastmap process?
- Python sequences (lists, tuples, etc.)
- Python generators
▹ How do you compare to Spark / Hadoop?
Spark and Hadoop have dozens of features that you may or may not need. Setting up these frameworks as clusters can be time-consuming and expensive. By contrast, fastmap does exactly one thing - makes it easy to parallelize arbitrary code in the cloud.
▹ How do you compare to Lambda / Cloud functions?
For many common applications, fastmap can be a substitute for AWS Lambda and similar technologies. Fastmap's key advantage is that code is deployed at runtime. This means no need to manage separate repos, no infrastructure configuration, and no coordinated deploys. Also, because fastmap supports local execution, you can iterate faster and be more confident in your production environment.
▹ Are there memory limits?
Yes. Your code is placed in a sandbox which is limited to 1GB of memory. In practice, your code's memory limit will be slightly lower.
▹ How do the execution policies work?
Fastmap has three execution policies: LOCAL, CLOUD, and ADAPTIVE. These allow you to run your code in different environments. Running adaptively will generally be the fastest. If running locally, you will not consume fastmap credits. For more, see the ExecPolicy docs.
▹ What happens if you go bankrupt / are sold?
You depend on fastmap. If for any reason, in the future, fastmap.io must shut down, everything will be open-sourced.
▹ What about data security?
Fastmap has no interest in your code or data. We charge a reasonable rate to ensure that you are the customer - never the product.
- Fastmap does not persistently store your raw or processed data and never will.
- In the future, fastmap may store your code in encrypted form for caching purposes. This will be an opt-out policy.
- Actual execution is done in a sandboxed virtual service which is created fresh for you. Other customer's code is never run in the same environment.
- Your code, input data, and output data are all transferred with an HMAC signature. If the signature does not match the payload, the data will not be processed / unpickled.
- All network transfers, internal & external, are encrypted using standard https.
▹ How do you make money?
See pricing. Fastmap's prices are comparable to, but somewhat higher than, what you might spend if you ran your jobs on AWS, GCP, or Azure.
Over time, I think you'll save money by not building and maintaining your own distributed infrastructure. Remember the multiply-by-pi rule. Distributed computing is non-trivial and engineers are expensive. Let fastmap handle the parallelization so that you can focus on your core product.