Entry points overview
This page gives quick examples of how to use each of Meadowrun's entry points.
We will assume you've gone through either the quickstart or one of the tutorials for a
specific platform (AWS EC2, Azure VMs,
GKE, or Kubernetes). In the examples, we'll use
resources variables that we don't define here. You will need to define
these variables as they were defined (as parameters) in the platform-specific tutorials.
Runs a function remotely. Example:
meadowrun.run_function(lambda: sum(range(1000)) / 1000, host, resources)
Runs a command remotely. See Run a Jupyter notebook remotely for
a fully worked example of using
run_map is like the built-in
map function, but it runs in
parallel and remotely. This is a powerful tool for scaling computations: easily use
hundreds or thousands of cores in parallel to process large data sets.
meadowrun.run_map(lambda x: x ** x, [2, 3, 4, 5], host, resources)
In this case, resources applies per task.
The output is:
[4, 27, 256, 3125]
run_map, you will no longer see the standard output from the remote processes
locally. To see the logs from your processes you'll want to look at Stream logs on AWS
The Kubernetes implementation of
run_map currently has a major limitation which is
that the tasks are assigned to workers at the start of the job. This means that if
you have tasks where some take a long time and some finish quickly, the distribution of
tasks will be suboptimal.
run_map_as_completed is similar to
run_map but it returns an
than as simple list of results. Results will be returned as tasks complete. Example:
async for result in await meadowrun.run_map_as_completed( lambda x: x ** x, [2, 3, 4, 5], host, resources ): print(result.result_or_raise())
The output is:
27 4 256 3125
In general, there's no guarantee what order the results come back in.
run_map is recommended for most use cases. Consider using
your use case requires further processing of results locally, and you'd like to
interleave this processing with executing the tasks. This could be beneficial if the
processing is time-intensive.