Green AI engineering

As generative models, retrieval, and agents move into production, the cost of inference becomes an architecture question. Model choice affects latency and energy. Retrieval affects how much data moves and how large the context gets. Caching changes the cost of repeated work. Scheduling decides how well the infrastructure is used when demand varies.

Green AI engineering treats that as one systems problem: model efficiency, runtime, data architecture, cloud resources, orchestration, and measurement against a useful outcome. The aim is more useful intelligence per unit of computation, energy, time, and cost. It is not minimum consumption at any price. Waste goes. Accuracy, reliability, responsiveness, security, and business value stay.

Efficiency is a property of the whole path

A production request typically passes identity checks, retrieval, inference, tool calls, transformation, validation, and delivery. Each stage spends compute, network, and storage. Shrinking only the model leaves the rest untouched.

An internal assistant can spend its budget on too many model calls, a long context, weak retrieval, repeated embedding work, extra agent loops, data moving between storage and the model, idle accelerators, and retries after timeouts. Those interact. Poor retrieval lengthens the context, which raises inference cost and can worsen the answer. An unreliable workflow repeats stages and spends more without a better result.

A practical account of workflow cost adds inference, retrieval, orchestration, data movement, and validation. It is an accounting model, not a universal formula. Shared infrastructure must not be counted twice, and the allocation method should be stated. The same idea applies to energy. GPU energy alone misses CPU, memory, network, storage, and the rest of the path.

Match the model to the task

The largest model is not the best default. Classification, entity extraction, intent, routing, and field validation often need less reasoning than a multi-step analysis. Putting a premium model on every step raises cost and latency without a matching gain in quality.

Start from the workload. A small specialized model may fit intent classification if accuracy and throughput hold. Compact models may fit extraction if field-level precision holds. Embeddings, hybrid search, and reranking fit knowledge retrieval if recall and access checks hold. A more capable model fits complex reasoning if task success and consistency hold. These are starting points. Data, regulation, and operating conditions decide the implementation.

A model is eligible when it meets thresholds for task accuracy, structured-output validity, unsupported claims, policy compliance, tail latency, cost and energy per successful workflow, and reliability under realistic load. Among eligible options, pick the one that meets the operational requirement. Reduce computation only inside that quality and reliability envelope. In short: minimize energy subject to a quality floor, a latency ceiling, and a reliability floor, and add cost or privacy constraints when they apply.

Serving, caching, and scheduling

Serving efficiency depends on precision, memory, concurrency, sequence length, batching, and hardware fit. Quantization, including lower-precision weights where the hardware supports them, can reduce memory and raise throughput. It can also change reasoning, tool choice, or consistency. Compare the quantized model with its reference on the real tasks, including long context, structured output, and safety-sensitive cases.

For decoder-only models, the key-value cache avoids repeated attention work and consumes memory. Balance cache size, sequence length, batch shape, and concurrency. Continuous batching can help interactive traffic. Offline work can often wait for larger batches. Decide from load tests that match real request sizes and the latency objective.

Profile before you tune. Measure time to first token, time per output token, queueing, throughput at a stated concurrency, accelerator memory, host-to-device transfer, cache behavior, and failures. A model that looks fine on a single request can behave differently under production concurrency.

Caching avoids repeated inference, retrieval, embedding, and deterministic work. Two similar requests may still need different answers because of permissions, document updates, or business state. An embedding cache applies when the content, the model, and the preprocessing match. A retrieval cache applies when the index, filters, permissions, and document state match. A semantic response cache applies only when equivalence, freshness, authorization, and output constraints are established. A prefix or key-value cache applies when the runtime supports it and isolation holds. Each cache needs a key, an expiry, an invalidation rule, and a correctness contract. A cache must not let one user’s authorized result leak to another. Judge caches by avoided work, lookup cost, staleness, extra memory, latency, and cost per successful workflow.

Interactive inference is provisioned around latency and concurrency. Batch inference can use lower-priority capacity. Training needs its own allocation and interruption policy. Background agents should have deadlines and budgets. Indexing should process what changed. Autoscaling has to account for model load time, cold starts, and queues. Scaling down too hard reloads models and spikes latency. High utilization is a poor goal if it creates retries and a worse experience. Latency-sensitive systems may need headroom.

Data movement and cloud placement

Moving data between storage, CPUs, accelerators, retrieval, and regions costs latency, money, and energy. Filter and project before transfer. Ingest increments. Reuse embeddings for unchanged content. Avoid needless cross-region copies and repeated format conversion. In retrieval, a large set of weak chunks increases context and can lower answer quality. Hybrid retrieval, metadata filters, reranking, and deduplication exist to select a smaller evidence set that is still sufficient and permitted.

An agent that re-reads the same documents, calls tools without a purpose, or uses a model for a deterministic calculation is spending budget it does not need. Cap iterations, tool calls, tokens, and elapsed time. Call a function or a query when that finishes the step more efficiently.

The cheapest instance is not always the cheapest way to finish the task. It may deliver fewer tokens per second or use more energy per successful result. Compare cost per successful workflow, throughput at the required quality, accelerator fit, data-transfer cost, recovery needs, and data-residency rules. Regional energy characteristics belong in the comparison only where the data is reliable. CPU may be enough for preprocessing, rules, and some compact models. A GPU or a specialized accelerator may fit large parallel inference. Count the cost of keeping several execution paths.

Measure a useful outcome

Tokens per second and average inference cost do not say whether the result was correct. Define the unit as the business workflow: a claim classified correctly, a document with its required fields validated, a question answered with supporting evidence, an incident diagnosed with an actionable result, a financial workflow completed inside policy.

Report energy per successful workflow, cost per successful workflow, useful throughput, tail latency, tokens per success, retry rate, cache effectiveness, and accelerator utilization. Carbon per outcome, when you report it, needs a documented method, geography, time period, and a clear boundary. Energy and emissions are related and they are not the same number. Direct power measurement is stronger than an estimate from utilization alone. For shared infrastructure, state how energy was allocated, and whether the boundary includes host CPU, memory, network, storage, and cooling. Manufacturing emissions, if included, should be labeled as a lifecycle figure.

Put workload volume, model and runtime versions, hardware, concurrency, and the evaluation method next to the metric. Comparisons without that context mislead.

Change the system only inside a quality gate

Quantization, a shorter context, a more aggressive cache, and a scheduling change can all hurt quality. Compare a candidate with a baseline on representative data: correctness, retrieval, unsupported claims, structured output, adversarial cases, authorization, latency, failure behavior, and cost or energy under load. For nondeterministic systems, repeat the evaluation and look at the variation.

Roll out through staged release, shadow evaluation, canary traffic, or a controlled comparison, as the system allows. If cost falls and incorrect decisions, escalations, or retries rise, the saving is incomplete. Tie the release gate to the model, the prompt, the retrieval configuration, the runtime, the quantization settings, and the evaluation set. Keep a known-good configuration you can return to.

A practical sequence is: instrument a representative workflow and define the successful outcome; remove needless model calls, oversized context, and unbounded agent loops; benchmark smaller models and runtimes on the same suite; separate interactive and asynchronous work and tune scaling against measured traffic; put cost and energy into the release review; then use production evidence to find the next bottleneck.

Efficiency is acceptable when the system still meets its quality, reliability, security, and service levels. The useful system is the one that delivers the required outcome with the least justified computation.