supraj.dev

Interview questions & answers

GPU economics & capacity planning

Commitments, reservations, spot, utilisation, power and chargeback: GPU capacity that finance can sign off.

  • 19 questions
  • 12 medium
  • 7 hard

19 questions

Question 1MediumCommitments

Finance bought a three-year Compute Savings Plan to cover the company's GPU training. A month later the team cannot launch the H100 instances it needs: the launch fails for lack of capacity. What went wrong, and how should GPU capacity be secured on AWS?

#

The team runs 16 p5.48xlarge instances for most of the year and needs them reliably in one Availability Zone for training. Finance assumed the commitment reserved the machines.

Show answer

A Savings Plan is a billing commitment: you commit to a spend per hour and get lower prices on matching usage. AWS describes Compute Savings Plans as applying regardless of instance family, size, OS, tenancy or Region, with savings of up to 72% on compute. Nothing in that buys capacity.

Capacity comes from reservations: - On-Demand Capacity Reservations reserve instances in a specific Availability Zone. They are charged at the equivalent On-Demand rate whether or not you run instances in them, and Savings Plans and Regional Reserved Instance discounts apply to them. So the usual pattern is a Capacity Reservation for the machines plus a Savings Plan for the price. - Future-dated Capacity Reservations let you request capacity for a start date with a commitment period, during which you cannot reduce the count or duration. - Capacity Blocks for ML reserve GPU instances for a fixed period starting up to 8 weeks ahead, paid upfront.

Also check quotas: On-Demand quotas are set as a number of vCPUs per Region, and a p5.48xlarge has 192 vCPUs, so 16 of them need 3,072 vCPUs of quota before any launch can succeed.

Fix for this team: create a Capacity Reservation (or future-dated one) for 16 p5.48xlarge in the training zone, keep the Savings Plan to discount it, and raise the vCPU quota. Because the reservation bills whether used or not, track its utilisation; an idle reservation is pure cost.

Trade-offs

Reserving capacity guarantees launches but bills even when idle; relying on on-demand availability is cheaper until the day the capacity is not there.

Common mistake

Treating a discount commitment as a capacity guarantee.

Follow-up

The training schedule is only busy nine months a year. How would you combine reservations, Capacity Blocks and on-demand to cover it?

Question 2MediumShort-term capacity

Plan GPU capacity for a three-week fine-tuning campaign on 32 H100 instances using EC2 Capacity Blocks for ML. What constraints shape the plan?

#

The campaign starts in about six weeks. The job checkpoints every hour and takes about 20 days of compute if nothing goes wrong. A previous team lost the last day of work when their reservation ended.

Show answer

Constraints from the Capacity Blocks documentation: - Durations: 1-day increments up to 14 days, then 7-day increments up to 182 days. Three weeks (21 days) is a valid size; 20 days is not, so you would book 21. - Size: up to 64 instances per Capacity Block and up to 256 instances across Capacity Blocks, so 32 instances fit in one block. - Start: up to 8 weeks in the future, so a start six weeks out can be booked now. - Payment and changes: the price depends on supply and demand at purchase time, it is paid upfront, and Capacity Block cancellations are not allowed. - End: the termination process for instances in a Capacity Block begins at 11:00 UTC on the final day of the reservation.

Plan: - Book 21 days for 32 instances, starting a day or two before you need them so setup and validation (driver checks, a short multi-node test) happen inside the block. - Schedule the job to finish well before 11:00 UTC on the last day, and force a final checkpoint to durable storage before then; the earlier team lost a day because they planned to the calendar end of the reservation. - With 20 days of compute and 21 days booked, you have one day of slack for failures and restarts; if failure history suggests more, book 28 days or shorten the run. - Keep checkpoints and data in storage outside the instances, so a second block (or on-demand) can resume the job if it overruns.

Trade-offs

Capacity Blocks give guaranteed GPUs for a bounded period without a long commitment, but cannot be cancelled and are priced at purchase time; on-demand is flexible but may not be available at that scale.

Common mistake

Planning work up to the last calendar day of the block, when instance termination starts at 11:00 UTC that day.

Follow-up

The run is now expected to take 26 days. What are your options, and which would you choose?

Question 3MediumCommitments

On Google Cloud, a team wants a one-year committed use discount for its H100 GPUs but does not want to pay for idle reserved capacity. Is that possible, and how would you size the commitment?

#

The team uses A3 machines. Usage is 64 GPUs around the clock, rising to 128 GPUs for a few weeks each quarter.

Show answer

For resource-based commitments on GPUs, Google's documentation says attaching reservations is mandatory: you must reserve the committed GPUs and attach those reservations to the commitment, and you buy separate commitments per GPU type. (It notes an exception path for G2 and G4 GPUs.) So for H100s you cannot get the commitment discount without also holding the reserved capacity, and you pay for that capacity whether it is busy or not.

Sizing follows from that: - Commit and reserve only the steady baseline, 64 GPUs here, which runs around the clock anyway, so the reservation is rarely idle. - Cover the quarterly peak another way: on-demand if capacity is available, Spot VMs for interruptible work (Google states discounts of up to 91% for many machine types and GPUs, with preemption possible at any time), or flex-start VMs through Dynamic Workload Scheduler, which wait for capacity within a timeframe and then run for up to seven days. - Revisit the baseline before renewal; commitments last one or three years.

Track two numbers monthly: reservation utilisation (used GPU-hours divided by reserved GPU-hours) and the share of total GPU-hours covered by the commitment. Low utilisation means you over-committed; low coverage with high on-demand spend means you under-committed.

Trade-offs

A reserved, committed baseline is the cheapest per hour and guarantees capacity, but stranded capacity costs the full price; covering peaks with Spot or flex-start is cheaper but less certain.

Common mistake

Committing to the peak (128 GPUs) because it is cheaper per hour, and paying for 64 idle reserved GPUs most of the year.

Follow-up

How would you decide between a one-year and a three-year term for the baseline?

Question 4HardProvisioned throughput

Your team pays per token for a hosted model on Azure and suffers latency spikes at peak. Someone proposes provisioned throughput (PTUs). How do you decide, and what traps should you plan for?

#

Suppose (hypothetical numbers for the exercise) the workload needs 300 PTUs at peak, the hourly PTU rate is $1.00, and your current pay-per-token bill is $150,000 a month. Traffic is high for 12 hours a day and low at night.

Show answer

What the documentation says: - Provisioned deployments are billed hourly per PTU deployed, regardless of the number of tokens consumed; the meter runs from creation to deletion. - Azure Reservations are a financial discount on the PTU meter for a 1-month or 1-year commitment. They do not guarantee capacity: create the deployment first, then buy the reservation. - PTU quota does not guarantee capacity. Scaling down releases capacity back to the pool with no guarantee you get it back, so the docs advise against scaling provisioned deployments up and down with traffic to stay on hourly billing. - Provisioned types give guaranteed throughput and lower latency variance; Standard types are best effort.

Arithmetic with the scenario's numbers: 300 PTUs x $1.00 x 730 hours = $219,000 a month at the hourly rate, more than the $150,000 pay-per-token bill. A reservation discount would lower it, but at a daytime-only peak the PTUs are underused half the day.

A better shape: - Provision for the latency-sensitive baseline, not the peak: for example, enough PTUs for the steady daytime load that needs consistent latency. - Keep a Standard deployment for overflow and for night traffic (your gateway routes there when provisioned capacity is full). - Move non-urgent work to batch, which is billed per token at a discount.

Decide on cost and latency together: the value of provisioned capacity is the latency SLO it buys, so compare (provisioned cost + overflow cost) against (pay-per-token cost + the business cost of latency spikes). Measure PTU utilisation after launch; Cost Management shows reservation utilisation.

Trade-offs

Provisioned throughput buys predictable latency and throughput at a fixed hourly cost; pay-per-token is cheaper at low or spiky utilisation but has best-effort latency.

Common mistake

Sizing PTUs for peak and planning to scale them down at night, then finding the capacity is gone in the morning.

Follow-up

How would you measure how many PTUs the baseline really needs before you buy a reservation?

Question 5HardSpot and preemptible

A team wants to cut training cost by running a 70B fine-tune on Spot GPUs. What does the interruption notice on each cloud let you do, and how must checkpointing change?

#

The job uses 8 nodes of 8 GPUs with synchronous data parallelism. A full checkpoint (weights plus optimizer state) is roughly 1 TB and takes about 6 minutes to write to object storage.

Show answer

Notice windows: - AWS emits an interruption notice two minutes before stopping or terminating a Spot Instance, via instance metadata and EventBridge. - Google Cloud Spot VMs get a preemption notice through metadata; the preemption notice duration is 0 seconds by default or can be set to 120 seconds, followed by a best-effort shutdown period of up to 30 seconds. - Azure Spot VMs can receive Scheduled Events notifications, delivered best effort up to 30 seconds before eviction.

A 6-minute checkpoint does not fit in any of these windows, so the notice is only good for stopping cleanly, not for saving state. Checkpointing must be periodic and independent of the notice: - Write checkpoints on a schedule, asynchronously if the framework supports it, and keep the last two in durable storage. - Expected lost work per interruption is about half the checkpoint interval plus restart time (rescheduling, loading weights, warming up). With a 30-minute interval and a 10-minute restart, each interruption costs about 25 minutes of the whole job's 64 GPUs. - Synchronous training stalls when any one of the 8 nodes is preempted, so the interruption rate of the job is roughly the sum of the nodes' rates. Eight nodes on Spot are interrupted much more often than one.

Economics: compare (Spot price x GPU-hours x (1 + lost-work fraction)) with the on-demand or reserved price. If interruptions cost 20% extra GPU-hours, a discount below 20% is a loss. Measure the real interruption rate in your zones before committing the schedule.

Alternatives: Spot for evaluation and data processing, reserved or Capacity Block GPUs for the synchronous training itself, or elastic training frameworks that continue with fewer nodes.

Trade-offs

Spot cuts the hourly price a lot but adds lost work, restarts and schedule risk that grow with the number of nodes in a synchronous job.

Common mistake

Planning to write the checkpoint after the interruption notice arrives.

Follow-up

How would you choose the checkpoint interval if storage writes slow the training step?

Question 6MediumSpot and preemptible

Design a batch inference job (re-embedding 500 million chunks) that runs on Spot GPUs and finishes on time despite interruptions.

#

The job must finish within 10 days. Each GPU embeds about 2,000 chunks per second at the chosen batch size. Spot capacity varies by zone and instance type.

Show answer

Work units: split the corpus into units of a few minutes of work (for example, 500,000 chunks, about 4 minutes on one GPU). Each unit writes its output under a deterministic key (unit ID plus model version) and is marked done only after the write succeeds. An interrupted unit is simply re-queued; the most you lose is one unit per interrupted GPU.

Arithmetic: 500 million chunks at 2,000 per second per GPU is 250,000 GPU-seconds, about 69 GPU-hours. Ten days is not the constraint; even 4 GPUs finish in under a day. That changes the decision: run fewer GPUs, pick the cheapest pools, and accept interruptions.

Capacity: request several instance types and zones. On AWS, the price-capacity-optimized allocation strategy is the documented recommendation for EC2 Fleet and Spot Fleet; it prefers pools with spare capacity at low prices rather than only the lowest price. On Google Cloud, Spot VMs have no minimum or maximum runtime but can be preempted at any time.

Handling notices: on the interruption notice, stop taking new units and let the current one finish if it can in the window; otherwise drop it, since it will be retried.

Observability: units done, re-queued and failed; interruptions per pool; cost per million chunks; projected completion time versus deadline, with an automatic fallback to on-demand if the projection slips past it.

Trade-offs

Small units waste little work on interruption but add scheduling overhead; large units are efficient until an interruption throws away an hour.

Common mistake

Running the job as one long process per GPU that restarts from the beginning after an interruption.

Follow-up

Some units always fail on the same documents. How do you keep them from blocking completion?

Question 7HardBuy vs rent

Leadership asks whether to buy 64 H100 GPUs for an on-premises cluster or keep renting them in the cloud. Build the comparison.

#

All numbers are hypothetical inputs for the exercise: 8 DGX-class servers at $300,000 each, $400,000 for networking and storage, depreciation over 4 years, electricity at $0.12 per kWh, a facility PUE of 1.4, $150,000 a year for space and support contracts, and half an engineer at $200,000 a year fully loaded. Expected average utilisation is uncertain: somewhere between 30% and 70%.

Show answer

Annual cost of owning (with the scenario's numbers): - Servers: 8 x $300,000 = $2.4M over 4 years = $600,000 a year. - Networking and storage: $400,000 / 4 = $100,000 a year. - Power: NVIDIA lists the DGX H100 at 10.2 kW maximum, so 8 servers draw up to 81.6 kW. At an average of 70% of maximum that is about 57 kW of IT load; with a PUE of 1.4 about 80 kW at the facility. 80 kW x 8,760 hours = about 700 MWh a year, about $84,000 at $0.12 per kWh. - Space and support: $150,000. Staff: $100,000. - Total: about $1.03M a year.

Per useful GPU-hour: 64 GPUs x 8,760 hours = 560,640 GPU-hours a year. At 70% utilisation, about 392,000 useful hours, so about $2.64 each; at 30%, about 168,000 hours, so about $6.15 each. Utilisation moves the answer by more than any other input.

Compare with renting at the rate you can actually get (reserved or committed, not list on-demand) for the same useful hours, and add what the spreadsheet misses: - Lead time to buy, install and burn in (months), versus capacity now. - Obsolescence: a new GPU generation can make a 4-year depreciation schedule optimistic. - Failures and spares, and the engineering time to run drivers, firmware, schedulers and networking. - Elasticity: renting lets you burst for training campaigns and shrink afterwards. - Data gravity: where the training data and users already are.

A common answer is hybrid: own or commit to the steady baseline you are confident will stay busy, and rent the variable part.

Trade-offs

Owning is cheapest per hour at high, steady utilisation and gives control; renting costs more per hour but avoids idle capex, lead times and obsolescence risk.

Common mistake

Comparing the purchase price with on-demand list prices at 100% utilisation and ignoring power, staff and idle time.

Follow-up

Which single input would you try hardest to measure before deciding, and how?

Question 8HardUtilisation targets

Finance wants inference GPUs run at 90% or higher utilisation to justify their cost. Why does that conflict with latency targets, and what utilisation should you plan for?

#

An interactive chat service with a p95 time-to-first-token target. Traffic varies across the day. The fleet also has a backlog of offline jobs (evaluations, embeddings) that could run anytime.

Show answer

As a queue approaches full load, waiting time grows non-linearly. In the simplest queueing model (M/M/1, random arrivals and service times) the average wait is proportional to u / (1 - u), where u is utilisation: about 1 at 50%, 4 at 80% and 9 at 90%. Real LLM servers batch requests, which helps, but the shape is the same: the last few percent of utilisation cost the most latency, and spikes push a busy fleet over its SLO.

So plan by SLO, not by a utilisation number: 1. Load-test one replica with realistic traffic and find the highest request rate at which p95 time to first token still meets the target (goodput at SLO). 2. Size interactive replicas so the peak rate stays below that, plus headroom for a zone failure or a deploy. 3. Expect the interactive fleet to sit well below 90% on average, because of headroom and the daily cycle.

Then recover the economics with work that can wait: - Run offline jobs on the same GPUs at low priority when interactive load is low, preempting them when it rises; or keep a separate batch pool that grows into idle capacity. - Autoscale interactive replicas on queue length, so night-time capacity shrinks. - Report utilisation for the whole fleet including batch, and latency for interactive traffic, so finance and SRE look at the right numbers.

Measure utilisation carefully: a GPU that is "utilised" by the coarse metric can still be doing little useful work, so track tokens per second per GPU alongside it.

Trade-offs

Running hotter lowers cost per token but raises tail latency and reduces failure headroom; filling slack with interruptible batch work gets most of the cost benefit without hurting the SLO.

Common mistake

Setting a utilisation target for an interactive fleet without connecting it to the latency SLO.

Follow-up

How would you make batch jobs give GPUs back within seconds when interactive traffic rises?

Question 9HardCapacity planning

Forecast how many GPUs your inference fleet needs for the next two quarters, starting from product forecasts.

#

Product expects peak traffic of 400 requests per second next quarter and 600 the quarter after. Requests average 1,500 input and 300 output tokens and take about 6 seconds end to end. Load tests show one replica (4 GPUs) serves 25 requests per second within the latency SLO. Reservations need about 8 weeks of lead time.

Show answer

Replicas from goodput: - Next quarter: 400 / 25 = 16 replicas at peak. Add headroom for a zone failure, deploys and forecast error, for example 25%: 20 replicas, which is 80 GPUs. - The quarter after: 600 / 25 = 24 replicas, 30 with headroom, which is 120 GPUs.

Concurrency check with Little's law (requests in flight = arrival rate x time in system): 400 x 6 = 2,400 requests in flight at peak, which is 150 per replica across 16 replicas. Confirm a replica can actually hold 150 concurrent sequences of about 1,800 tokens in its KV cache; if not, the memory limit, not throughput, sets the replica count.

Inputs that change the answer: - Token mix: longer prompts (RAG, agents) raise prefill cost and KV memory; re-run the load test when the mix shifts. - Model changes: a larger or smaller model changes goodput per replica entirely. - Efficiency work in progress (quantisation, prefix caching, speculative decoding) that may raise goodput before the forecast date; do not count it until measured.

Procurement timing: with 8 weeks of lead time, the decision for the 120-GPU quarter must be made about two months before it starts. Commit to the part of the forecast you are confident in (the baseline), and cover the uncertain part with on-demand, short-term blocks or burst capacity.

Track forecast accuracy each quarter (forecast versus actual peak) and adjust the headroom based on how wrong past forecasts were.

Trade-offs

More headroom protects the SLO and absorbs forecast error, but every extra replica is idle most of the time; committing late keeps options open but risks no capacity.

Common mistake

Converting tokens per day into GPUs using a vendor peak-throughput figure instead of measured goodput at your latency SLO.

Follow-up

The product forecast doubles mid-quarter. Which levers do you pull in the first week?

Question 10HardAvailability

GPU capacity in your main cloud region is scarce, and launches fail at the worst moments. Design a multi-region or multi-cloud strategy for GPU availability.

#

Training data (2 PB) lives in one region. Inference serves global users. The team holds generous quotas and assumed that meant capacity.

Show answer

Start with the distinction the providers themselves make: - Quota is permission to request. Azure's provisioned-throughput docs say plainly that having quota does not guarantee capacity, and AWS On-Demand quotas are just a maximum number of vCPUs per Region. - Capacity is what is physically free when you ask. - Reservations (AWS Capacity Reservations and Capacity Blocks, Google reservations, Azure provisioned deployments) are how you turn a request into guaranteed capacity, at a price.

Strategy: - Separate workloads by how portable they are. Inference is portable if model weights, containers and configuration are replicated; training is anchored to its 2 PB of data, and moving it means paying data transfer and time, so train near the data. - For inference: run in at least two regions with reserved baseline capacity in each, route users by latency and health, and fail over between them. A second cloud adds availability but doubles the platform work (images, drivers, networking, identity, observability). - For training: reserve or book short-term capacity ahead in the data's region (Capacity Blocks on AWS, flex-start VMs or calendar-mode future reservations on Google Cloud), and keep a portable job definition so a campaign can move if a block is unavailable. - Keep the stack portable: containers, Kubernetes, the same inference engine and the same model artifacts everywhere, tested regularly in the secondary location.

Measure: launch failure rate by region and instance type, time to obtain capacity, reserved versus used capacity, and cost per useful GPU-hour per location.

Trade-offs

Spreading across regions and clouds improves the chance of getting GPUs but adds operational cost and data-transfer cost; staying in one region is simpler until capacity runs out.

Common mistake

Treating a quota increase as a capacity guarantee.

Follow-up

Which workloads would you never move to a second cloud, and why?

Question 11MediumChargeback

Design chargeback for a shared Kubernetes GPU cluster used by 20 teams, so costs are fair and teams have a reason to release idle GPUs.

#

Some teams run full 8-GPU training jobs, some run notebooks that hold one GPU all week, and some use MIG slices for small models. The platform has fixed costs (reservations, staff, storage) whether GPUs are busy or not.

Show answer

Choose the billing unit: - Allocation-based: GPU-hours requested by a team's pods (a full GPU, or a MIG slice priced as a fraction). This is simple, matches how capacity is blocked, and gives teams a reason to release GPUs they hold but do not use. - Usage-based: weighted by measured activity. This feels fairer to light users but rewards holding idle GPUs, so use it only as a discount on top of allocation, if at all.

Price per unit: total platform cost (reservations, on-demand spend, storage, network, staff) divided by expected allocated GPU-hours, published as a rate per GPU type and per MIG profile. NVIDIA's H100 page lists up to 7 MIG instances per GPU, so a slice can be priced at roughly 1/7 of a GPU-hour plus a small premium for fragmentation.

Idle capacity: decide explicitly whether unallocated GPUs are charged to the platform budget (an incentive for the platform team to right-size) or spread across teams by share (an incentive for teams to forecast). Most organisations start with platform-owned idle capacity.

Data: pod GPU requests over time from Kubernetes, joined with team labels; for an energy component, dcgm-exporter's default counters include a GPU energy counter (DCGM_FI_DEV_GPU_ENERGY_JOULES_TOTAL in the current default-counters.csv) and utilisation gauges for spotting idle allocations.

Rollout: show the numbers for two months (showback) before charging, then add policies such as idle notebook culling after a set period.

Trade-offs

Allocation-based charging is simple and discourages hoarding but can feel unfair to bursty users; usage-based charging feels fair but lets idle allocations go unpaid.

Common mistake

Charging only for measured utilisation, which makes holding an idle GPU free.

Follow-up

A team says their notebook GPUs are idle because people are thinking, not because they forgot them. How do you handle that?

Question 12MediumOn-premises

You are placing 32 DGX H100 systems in a colocation facility. Work out the power and cooling numbers you must agree with the facility before signing.

#

The plan is 4 systems per rack, 8 racks. The facility quotes a PUE of 1.4 and charges by contracted kW.

Show answer

Power per system: NVIDIA's DGX H100/H200 user guide lists 10.2 kW maximum, from six 3.3 kW power supplies in a 4+2 redundant configuration (the system keeps running with three PSUs, at reduced performance). Each H100 SXM GPU has a configurable maximum TDP of up to 700 W.

Per rack: 4 x 10.2 kW = 40.8 kW of IT load at maximum. Whole deployment: 32 x 10.2 kW = 326.4 kW of IT load at maximum.

Cooling: almost all electrical power ends up as heat, so the facility must remove about 40.8 kW per rack. In BTU per hour (1 kW is about 3,412 BTU/hr) that is about 139,000 BTU/hr per rack.

Facility load: PUE is total facility energy divided by IT energy; Google's explanation is that a PUE of 2.0 means one extra watt of cooling and power distribution for every watt of IT. At 1.4, 326.4 kW of IT needs about 457 kW at the facility.

What to agree with the facility: - Contracted power per rack at the maximum, not the average, with redundant feeds for the PSU redundancy to mean anything. - Cooling capacity per rack at that density, which many air-cooled rooms are not designed for; ask how they cool 40 kW racks. - Floor weight, network cross-connects to your storage and the rest of your cluster, and a remote hands service for GPU and PSU replacements.

If the facility cannot deliver 40.8 kW per rack, spread fewer systems per rack (which costs more racks and longer cables) or cap GPU power, measuring the throughput you lose.

Trade-offs

Dense racks save space and cable length but need more power and cooling per rack; spreading out is easier to cool but costs more space and network.

Common mistake

Planning power from the average draw you expect and finding the breakers trip during a full-load training run.

Follow-up

The facility offers 30 kW per rack. What are your options, and what does each cost you?

Question 13HardCapacity mix

Design a GPU purchasing portfolio for a fleet with a steady baseline, a daily peak and monthly training bursts. Which part goes on which pricing model?

#

Baseline: 200 GPUs busy around the clock (inference). Daily peak: 150 more GPUs for about 10 hours a day. Training: 512 GPUs for 10 days each month. Use a 30-day month for the arithmetic.

Show answer

GPU-hours per month by shape: - Baseline: 200 x 24 x 30 = 144,000 GPU-hours, every hour of the month. - Daily peak: 150 x 10 x 30 = 45,000 GPU-hours, busy 10 of 24 hours. - Training: 512 x 24 x 10 = 122,880 GPU-hours, concentrated in 10 days. - Total: 311,880 GPU-hours.

Match shapes to models: - Baseline (46% of hours, always busy): long-term commitment plus reserved capacity. This is the only shape where paying for every hour is efficient. - Daily peak (14%): reserving it would leave it idle 14 hours a day. Cover it with on-demand, Spot where the serving layer tolerates interruptions, or by shifting batch work into the night so the "peak" capacity does useful work around the clock (then it becomes baseline). - Training (39%): short-term reserved capacity booked ahead (EC2 Capacity Blocks run from 1 to 182 days and can start up to 8 weeks out; Google Cloud flex-start VMs run up to seven days), or a reservation shared between training and batch inference so it is not idle the other 20 days.

Check the alternative of committing to everything: reserving 862 GPUs (200 + 150 + 512) for the whole month is 862 x 720 = 620,640 GPU-hours paid for 311,880 used, so about half would sit idle; the commitment discount would have to exceed about 50% just to break even.

Review monthly: commitment utilisation, coverage, and the share of hours on each model. As the baseline grows, move stable parts of the peak into commitments.

Trade-offs

Commitments are cheapest per hour but only for hours you actually use; flexible models cost more per hour but nothing when idle.

Common mistake

Committing to peak capacity because the hourly rate looks cheaper.

Follow-up

The training team wants its 512 GPUs available at any time, not on a schedule. What does that cost, and how would you push back?

Question 14MediumCommitments

Should a company sign a three-year GPU commitment for inference, given that new GPU generations and more efficient models keep arriving?

#

The three-year rate is meaningfully cheaper than the one-year rate. Over the last year the team cut GPUs per request by half with quantisation and better batching, and a new GPU generation is due next year.

Show answer

What erodes a long commitment: - Efficiency gains: if GPUs per request halve again, a commitment sized for today's traffic is half idle unless traffic doubles. - New hardware: a newer GPU may serve more tokens per dollar, while your commitment pays for the old one. - Model changes: a move to a smaller model, or to a hosted API for some features, removes demand.

What protects you: - Commitment type. AWS Compute Savings Plans apply regardless of instance family, size or Region, so a newer instance family can still use the commitment; instance-specific or resource-based commitments (Google's GPU commitments are per GPU type and must have reservations attached) cannot move to new hardware. - Sizing: commit only to the floor you are confident about in year three, not to today's usage. - Laddering: buy several smaller commitments at different times, so they expire at different times and you can adjust as you learn.

Decision rule: estimate the probability-weighted usage in each year of the term. Sign the three-year term only for the part where (expected utilisation x three-year discount) beats the one-year option plus the value of flexibility. With efficiency gains of 2x a year, that part may be much smaller than today's usage.

Trade-offs

Longer terms lower the hourly rate but increase the risk of paying for capacity you no longer need; shorter terms cost more per hour but adapt to hardware and efficiency changes.

Common mistake

Sizing a three-year commitment from current usage without accounting for planned efficiency work.

Follow-up

How would you present this to finance, who see only the higher discount?

Question 15MediumWaste

Your cluster dashboard shows every GPU allocated, yet teams keep asking for more. How do you find and reclaim GPUs that are allocated but doing nothing?

#

A Kubernetes cluster of 256 GPUs with notebooks, dev servers, training jobs and inference. dcgm-exporter is installed. Nobody has looked at per-pod activity.

Show answer

Find idle allocations: - For each pod holding GPUs, join its allocation (from Kubernetes) with GPU activity from dcgm-exporter over a week. The current default counters include DCGM_FI_DEV_GPU_UTIL_RATIO and DCGM_FI_PROF_GR_ENGINE_UTIL_RATIO (ratio of time the graphics engine is active) and memory in use. - Classify: idle (activity near zero for days), lightly used (notebooks used a few hours a day), and active. Look at the 7-day maximum, not the average, so a nightly job is not flagged as idle.

Typical findings: notebooks left running, dev inference servers nobody calls, finished jobs whose pods were never cleaned up, and inference replicas pinned to minimum counts sized for a launch day.

Reclaim with policy, not tickets: - Idle notebook culling after a stated period, with a warning first. - Time limits on dev allocations, renewable with one click. - MIG slices or shared GPUs for notebooks and small models instead of full GPUs. - Chargeback by allocated GPU-hours, so holding an idle GPU shows up on a team's bill.

Arithmetic for the business case: if 40 of 256 GPUs are idle, that is 40 x 730 = 29,200 GPU-hours a month, about 16% of the cluster, recoverable without buying anything.

Keep reporting it: idle GPU-hours per team per week, and time from request to allocation (which should fall as waste falls).

Trade-offs

Aggressive culling frees GPUs quickly but annoys users who lose state; generous limits keep users happy but leave capacity stranded.

Common mistake

Reading cluster-wide average utilisation and concluding the cluster is full.

Follow-up

A team argues their inference replicas must stay at a high minimum for a rare traffic spike. How do you evaluate that?

Question 16MediumCost incidents

The monthly GPU bill rose 40% with no matching rise in traffic or training. Walk through how you find the cause.

#

The company uses AWS with Savings Plans, a few On-Demand Capacity Reservations, Capacity Blocks for training, and Spot for batch jobs. Cost data is in Cost Explorer and the Cost and Usage Report.

Show answer

Break the increase down along a few axes before guessing: 1. By purchase model: On-Demand, Savings Plan covered, reservation charges, Capacity Block upfront fees, Spot. A shift from covered to uncovered usage looks like a price rise with the same usage. 2. By resource and tag: which accounts, teams, instance types and Regions grew. 3. By time: a step change on one day points to a change event; a gradual rise points to growth or drift.

Common causes in GPU bills: - Idle reservations: Capacity Reservations are charged at the On-Demand rate whether or not instances run in them. A reservation kept after a project ended keeps billing. - Expired or under-sized Savings Plans, so usage that was discounted is now at On-Demand rates. - Capacity Blocks bought for a training campaign show as a one-time upfront payment, which can make one month look like a spike. - Spot fallback: batch jobs configured to fall back to On-Demand when Spot was scarce. - Forgotten resources: dev instances, large storage volumes and snapshots, data transfer between Regions. - Workload change: a new feature that doubled prompt length, or an autoscaler minimum raised during an incident and never lowered.

Fix and prevent: tag everything by team and purpose, alert on daily spend anomalies per account and purchase model, review reservation utilisation monthly, and make reservations and Capacity Blocks part of the same change process as infrastructure.

Trade-offs

Fine-grained tagging and anomaly alerts cost setup time but turn a month-end surprise into a next-day alert.

Common mistake

Assuming a bill increase must be traffic and asking teams to cut usage before checking idle reservations and coverage.

Follow-up

Which three alerts would have caught this within a day?

Question 17MediumShort-term capacity

A research team on Google Cloud needs 16 H100 GPUs for two to three days at a time, a few times a month, with no fixed schedule. Compare on-demand, Spot, reservations and flex-start VMs for them.

#

Their experiments can wait a few hours to start but should not be interrupted once running, because each run is a short multi-node job without elastic recovery.

Show answer

Options: - On-demand: no commitment, no interruption, but at scarce times the request may simply fail. - Spot VMs: much cheaper (Google states discounts of up to 91% for many machine types and GPUs) but can be preempted at any time, which these non-elastic multi-node runs cannot tolerate. - Reservations (with or without committed use discounts): guaranteed capacity, but paying for 16 GPUs all month for a few runs is mostly idle cost. - Flex-start VMs (Dynamic Workload Scheduler): Compute Engine tries to allocate the requested resources within a timeframe you set; if it succeeds, the VMs run uninterrupted for a run duration you set, up to seven days, after which Compute Engine stops or deletes them according to the VM's termination action. If capacity does not appear within the timeframe, the request fails.

Flex-start matches this team: they can wait a few hours to start, the run lasts 2 to 3 days (inside the 7-day limit), and they need no interruption once running. For work longer than seven days, or with a fixed start date, Google points to future reservations in calendar mode.

Practicalities: set the run duration with margin over the expected job time and checkpoint before it ends, since the VMs are stopped or deleted at the end of the run duration; keep data and checkpoints in durable storage; automate the request so researchers submit a job, not a VM.

Trade-offs

Flex-start trades a start-time guarantee for better access to scarce GPUs without paying for idle reservations; Spot is cheaper still but only for interruptible work.

Common mistake

Putting non-elastic multi-node jobs on Spot to save money and losing runs to preemption.

Follow-up

The team's runs grow to 10 days. What changes?

Question 18MediumEnergy

Sustainability and finance teams ask for the energy cost per million tokens served by your self-hosted model. How do you measure it, and how big is it compared with the GPU cost?

#

One 8-GPU server serves about 20,000 output tokens per second at steady load, with the GPUs drawing about 600 W each on average. The facility PUE is 1.3 and electricity costs $0.15 per kWh (both given for the exercise).

Show answer

Measure, do not estimate: dcgm-exporter's current default counters include DCGM_FI_DEV_GPU_ENERGY_JOULES_TOTAL (energy consumed since the driver was loaded) and DCGM_FI_DEV_BOARD_POWER_WATTS. Take the increase in the energy counter across all GPUs over a window, divide by tokens generated in the same window (from the serving engine's token counters), and you have joules per token for the GPUs. Add the rest of the server (CPUs, memory, fans, network) from server-level power readings if you have them.

Arithmetic with the scenario's numbers: - GPU power: 8 x 600 W = 4,800 W. - Energy per token: 4,800 J/s / 20,000 tokens/s = 0.24 J per token. - Per million tokens: 240,000 J = 0.0667 kWh at the GPUs. - With PUE 1.3: about 0.0867 kWh at the facility. - At $0.15 per kWh: about $0.013 per million tokens.

Perspective: compare that with the same server's hardware cost per million tokens. If the server costs, say, $20 an hour to own or rent, it produces 72 million tokens an hour at this rate, about $0.28 per million tokens, so energy is roughly 5% of the total here. Utilisation and hardware cost dominate; energy matters more for carbon reporting and power-limited sites than for the unit cost.

Caveats: energy per token rises at low load (idle power is spread over fewer tokens), differs between prefill-heavy and decode-heavy traffic, and should be reported per model and per traffic type.

Trade-offs

Counter-based measurement is accurate for GPUs but misses the rest of the server and the facility unless you add those readings; vendor TDP estimates are easy but overstate or understate real draw.

Common mistake

Multiplying GPU TDP by hours and calling it the energy cost, which ignores actual draw, load level and facility overhead.

Follow-up

How would you report energy per token for a fleet that runs several models with very different traffic?

Question 19MediumOn-premises

Your data hall has a hard power limit, and you could fit more GPU servers if each GPU ran at a lower power cap. How do you decide whether to cap, and at what level?

#

The H100 SXM GPUs in the fleet have a configurable maximum power. The hall is at its contracted limit, and the team wants 25% more servers.

Show answer

NVIDIA lists the H100 SXM maximum TDP as up to 700 W and configurable, so capping is a supported operating mode, not a hack. The question is what you lose.

Method: 1. Pick representative workloads: a training step, prefill-heavy inference, decode-heavy inference. 2. Run each at several power caps (for example, full power and two or three lower settings), measuring throughput (tokens per second or steps per second) and power from dcgm-exporter (DCGM_FI_DEV_BOARD_POWER_WATTS, plus the power-throttle time counter when enabled). 3. Plot throughput per GPU and throughput per watt against the cap.

Decide on fleet throughput within the power budget, not per-GPU speed: if a lower cap loses 10% throughput per GPU but lets you run 25% more GPUs in the same power, fleet throughput rises by about 1.25 x 0.9 = 1.125, a 12.5% gain, at the cost of the extra servers.

Caveats: - Workloads differ: memory-bound decode may lose little at a lower cap; compute-bound training or prefill may lose more. Measure, do not assume. - Synchronous training runs at the speed of the slowest GPU, so caps must be uniform and stable across a job. - Latency SLOs: a lower cap can raise time to first token; check it against the SLO, not only throughput. - Monitor throttling events so a cap set too low does not hide behind good averages.

Trade-offs

Power caps raise fleet throughput when power, not GPUs, is the constraint, but lower per-GPU speed and can hurt latency-sensitive or compute-bound work.

Common mistake

Assuming throughput falls in proportion to power, or capping without measuring your own workloads.

Follow-up

Would you apply the same cap to training and inference nodes? Why or why not?