TL;DR: Half 2 of two. Half 1 lined the one {hardware} thought you want and the 2 layers beneath (GKE and Ray Core). This half exhibits the libraries you really construct with, Ray Serve, Ray Knowledge, and Ray Prepare.
Recap
Fast recap if you happen to’re touchdown right here first. Operating Ray on TPU comes down to at least one caveat: TPU chips are wired into fastened teams known as slices (host VMs sharing a high-speed hyperlink known as the ICI), and a multi-host mannequin has to land on one complete slice or its staff cannot attain one another and the job hangs.
Google Kubernetes Engine (GKE) with the Ray Operator add-on provisions slices and labels their hosts, and a Ray Core primitive, slice_placement_group(), reserves an entire slice directly. You declare a topology (the slice form, like 4×4 for 16 chips, for instance) and the libraries beneath deal with the location for you.
With Core dealing with placement beneath, the libraries all observe the identical sample: declare a topology, let Core reserve the slice. What adjustments per library is barely what you declare it on. We’ll go within the order most groups undertake them, serving first.
Ray Serve on TPU
Serving is the place most groups begin from. A mannequin that wants a number of GPUs to suit can run on a single TPU host, and TPUs are sometimes a extra accessible and cost-effective choice for inference. Ray Serve offers you the standard autoscaling, load-balancing, and multi-model composition, and on TPU it serves LLMs by vLLM, a high-throughput engine.
The exhausting case is when a mannequin is just too massive for one host (say one sharded tensor-parallel throughout 16 chips). That is the place Serve clears it with a single further discipline, topology.
accelerator_type: TPU-V6E
accelerator_config:
type: tpu
topology: "4x4"Plain textual content
That one discipline is price understanding, as a result of getting it improper is the basic multi-host TPU failure. With topology set, Serve’s TPU backend skips its typical upfront placement group and defers to the duplicate, which creates a slice placement group at startup. That deferral is what retains a tensor-parallel mannequin’s staff on one shared ICI mesh. Go away it off and Serve falls again to per-chip bundles; on a multi-host mannequin these bundles can scatter throughout two slices, and since there is no ICI between slices, the employees by no means end their first collective. You aren’t getting a crash, you get a deployment that sits in DEPLOYING endlessly when you burn TPU-hours attempting to find a bug that is actually one lacking line of YAML. So bear in mind, topology discipline makes the distinction.
In observe you deploy a RayService (advisable over a uncooked RayCluster for manufacturing) on a printed vLLM TPU picture, look ahead to it to achieve Operating, and curl the endpoint. The official GKE tutorials cowl Llama 3 8B and Mistral 7B on v5e, Llama 3.1 70B on v6e, and Steady Diffusion. The serve step of the get-started instance walks the total deployment finish to finish.
Ray Knowledge on TPU: feeding the accelerators with iter_jax_batches
A quick accelerator is barely as helpful as the information you may preserve flowing into it, and TPUs are quick sufficient {that a} naive loader turns into the bottleneck. That is the issue iter_jax_batches() solves. It arms you batches which can be already JAX arrays and already device-sharded, so a coaching enter pipeline or a big batch-inference job pulls straight from a Ray Knowledge pipeline with no host-side NumPy-to-JAX copy stalling the step.
ds = ray.knowledge.read_parquet("gs://my-bucket/prepare/")
for batch in ds.iter_jax_batches(batch_size=1024):
# batch arrives as device-sharded JAX arrays, prepared for the coaching step
loss = train_step(batch)Python
iter_jax_batches API does the machine sharding for you, and it handles the ragged closing batch (the one which is not a clear a number of of your batch dimension) with an express alternative of drop, pad, or increase, as a substitute of a form error three hours right into a run.
You need to use it because the enter facet of a JaxTrainer job, and it is simply as helpful by itself for offline batch inference over an enormous dataset on a TPU slice. It landed lately in Ray, and the information step of the get-started instance makes use of it for dataset prep and batch inference.
Ray Prepare on TPU: distributed coaching with JaxTrainer
Coaching was the complicated a part of Ray on TPU, due to topology and having to account for the slice form in your code. JaxTrainer addresses that. It brings Ray Prepare’s coaching loop (checkpointing, fault tolerance, multi-slice scale-out) to JAX, Google’s array and autodiff library and the native framework for TPU. You hand it a coaching operate and a slice form and Ray launches one employee per host, wires them right into a single mesh, and runs your operate on every.
from ray.prepare import ScalingConfig
from ray.prepare.v2.jax import JaxTrainer
def train_loop_per_worker(config):
import jax # import jax INSIDE the employee fn (TPU requirement)
# ... your JAX/Flax coaching step runs right here, as soon as per host ...
coach = JaxTrainer(
train_loop_per_worker=train_loop_per_worker,
scaling_config=ScalingConfig(
use_tpu=True,
topology="4x4", # the slice form, NOT a chip rely
accelerator_type="TPU-V6E",
),
)
coach.match()Python
Two issues on this snippet you need to bear in mind to save lots of debugging time. The import jax lives inside train_loop_per_worker, not on the prime of the file, as a result of every employee initializes JAX in its personal TPU context; import it at module scope and you will combat cryptic device-init errors earlier than step one. And topology="4x4" is your complete placement declaration, the road that was a block of hand-written coordination code. Set subsequent to a GPU JaxTrainer or TorchTrainer, the one actual distinction is use_tpu=True and a topology as a substitute of a GPU rely.
The remainder it simply runs. It is because Ray Prepare owns the loop, you get checkpointing and fault-tolerant restarts, which is what makes lengthy TPU runs on preemptible capability really end, and topology scales to multi-slice (Ray wires the cross-slice coordination) when one slice is not sufficient. The prepare step of the get-started instance is a whole JaxTrainer DPO run.
Two closing extras: TPU Docker photos and dashboard metrics
As a part of the first-class accelerator help, Ray now publishes official rayproject/ray:*-tpu photos with the JAX/TPU stack (jax[tpu], flax, optax, orbax-checkpoint) and profiling tooling already put in, so you do not have to assemble a working TPU setting by hand. You possibly can simply base your picture on the tagged -tpu one.
And for monitoring, the Ray Dashboard, Ray’s built-in internet UI for cluster and job state, now exhibits TPU utilization and reminiscence subsequent to CPU and GPU on the Cluster tab, with ray.util.tpu.init_jax_profiler() exposing a per-worker JAX profiler the dashboard can connect to.
In Abstract
On this developer information on Ray on TPU, we lined the entire journey from how Ray runs on TPU to working AI workloads.
Half 1 confirmed that working Ray on TPU comes down to at least one caveat, preserving a multi-host mannequin on a single intact slice, and that GKE (by the Ray Operator add-on) and Ray Core (by slice_placement_group()) deal with that for you.
This half put the AI libraries on prime: Ray Serve gang-schedules a multi-host mannequin onto one slice with a single accelerator_config.topology discipline, Ray Knowledge feeds the slice JAX-native batches by iter_jax_batches(), and JaxTrainer runs a distributed coaching loop from one ScalingConfig. The identical Ray you already use on GPUs, now on TPU.
What’s subsequent
And extra is coming. The Ray staff on Google Cloud is widening TPU help from right here: deeper Ray Knowledge and Ray LLM TPU integration, SkyRL on multi-host TPU for reinforcement studying and post-training, and dynamic tremendous/sub-slice help are all on the roadmap.
To your personal subsequent step, my advice is: clone the get-started instance, arise the cluster, then run serve, knowledge, or prepare. Or simply allow --enable-ray-operator on a cluster and run one Ray activity on a small slice to see it work. You do not have to develop into a TPU professional to make use of one, simply give it a strive.
For now, thanks for studying! And when you have any further questions or suggestions, be at liberty to achieve out on socials (LinkedIn, X).
Joyful constructing!
Further assets
New right here? Half 1 explains slices, GKE, and Ray Core, the inspiration every part above builds on.
