AI Engineer Interview Questions

1. What is the difference between a model's context window and its effective memory — and how does that distinction affect how you design an application?

What a strong answer demonstrates: Understanding that longer context does not mean better recall — attention degrades across long windows, and models tend to perform better on content at the beginning and end. Strong candidates design around this, not against it.

Listen for: Whether the candidate recognizes the "lost in the middle" problem and describes concrete design responses — retrieval to surface the most relevant content, summarization to compress history, or structured prompting to prioritize key information. Candidates who treat context window size as equivalent to memory capacity will build applications that degrade in ways they do not anticipate.

2. How does tokenization affect the cost and latency of LLM API calls? What are the practical implications for how you design a system?

What a strong answer demonstrates: Practical awareness that token count drives both cost and latency — and the ability to reason about prompt compression, output length control, and model tier selection as cost levers.

Listen for: Concrete optimization strategies: trimming system prompt verbosity, using structured output formats that require fewer tokens, caching repeated prompt prefixes, routing simpler queries to lighter models. Candidates who have shipped LLM-powered features in production have almost always had to reason about token economics — candidates who have not will often skip this entirely.

3. Explain the difference between fine-tuning a model, using it with retrieval-augmented generation, and relying on prompting alone. How do you choose between them?

What a strong answer demonstrates: A clear-eyed view of the trade-offs — fine-tuning is expensive, requires data, and bakes knowledge into weights that become stale; RAG adds retrieval complexity but keeps knowledge fresh; prompting alone is fastest to iterate but limited by context and unreliable for factual recall.

Listen for: Whether the candidate frames the decision around the specific problem — the nature of the knowledge required, how often it changes, the volume of queries, and latency budget. Red flag: candidates who default to fine-tuning as the "serious" approach without assessing whether the use case warrants it. Most production AI engineer roles live primarily in the RAG and prompting layer.

4. What does the temperature parameter actually control, and when would you set it high versus low in a production feature?

What a strong answer demonstrates: A precise understanding of temperature as a sampling distribution modifier — not just "high temperature = more creative" — and the ability to connect it to specific use case requirements.

Listen for: Whether the candidate can name concrete use cases for each end of the range: near-zero temperature for structured data extraction, classification, or code generation where consistency matters; higher temperature for brainstorming, creative writing, or variety-seeking features. Also worth asking whether they test temperature settings systematically or choose them by intuition.

8. Walk me through how you would design a RAG pipeline for a production search or Q&A feature. What are the key architectural decisions?

What a strong answer demonstrates: End-to-end system thinking — document ingestion and chunking, embedding model selection, vector store choice, retrieval strategy (dense, sparse, or hybrid), re-ranking, and how retrieved documents are passed to the generation model.

Listen for: Whether the candidate starts by scoping the problem: What is the document corpus? How large? How frequently updated? What is the latency budget? Retrieval architecture decisions — chunking granularity, embedding model, re-ranking — all depend on these constraints. Strong AI engineers treat this as a system design problem, not a template to fill in.

9. How do you choose a chunking strategy for document retrieval? What are the trade-offs between different approaches?

What a strong answer demonstrates: Practical knowledge of the trade-offs between fixed-size chunking, semantic chunking, hierarchical chunking, and document structure-aware chunking — and the ability to match the approach to document type and query patterns.

Listen for: Whether the candidate connects chunking strategy to the specific failure modes it creates. Chunks too small lose context; chunks too large dilute relevance and waste context window. The best candidates also mention how they evaluate chunking quality empirically — not just reason about it theoretically.

10. How do you evaluate retrieval quality in a RAG system — separately from generation quality?

What a strong answer demonstrates: The ability to decompose RAG evaluation into its two separate components: retrieval precision/recall (are the right documents being retrieved?) and generation quality (is the model producing accurate, grounded responses from those documents?).

Listen for: Strong candidates describe metrics for each layer independently — hit rate, MRR, or NDCG for retrieval; faithfulness, answer relevance, and context precision for generation — and explain why evaluating them together makes debugging harder. This separation is often the difference between teams that iterate effectively on RAG quality and teams that cannot diagnose why their system is failing.

11. Your RAG system is returning retrieved documents that are technically on-topic but not useful for the specific query. What do you investigate?

What a strong answer demonstrates: Diagnostic thinking about retrieval failures — embedding space misalignment, poor chunking granularity, keyword/semantic retrieval mismatch, query rewriting needs, or re-ranking deficiencies.

Listen for: A structured diagnosis process rather than jumping straight to "try a different embedding model." Strong candidates consider: Is this a query representation problem (the query embedding doesn't match how the knowledge is stored)? A chunking problem (the relevant information is split across chunks)? A retrieval strategy problem (dense retrieval misses keyword-specific queries)? Each has a different fix.

16. Describe a time you reduced latency or cost in an LLM-based system without meaningfully degrading output quality. What levers did you pull?

What a strong answer demonstrates: Real optimization experience — model routing (sending simpler queries to cheaper models), prompt compression, output caching, streaming, or batching strategies — with measurable impact.

Listen for: A specific story with a before/after. Generic answers ("I would use a smaller model") are easy to give without any real experience. Candidates who have actually optimized production LLM systems can name the specific bottleneck they identified, the tradeoff they made, and the measurement they used to confirm the change worked. Also listen for whether they distinguished latency optimization from cost optimization — the levers are often different.

17. How do you set up observability for an LLM-powered feature in production? What do you track, and why?

What a strong answer demonstrates: A layered observability approach — request-level tracing (inputs, outputs, latency, token counts, costs), quality signals (user feedback, thumbs up/down, downstream engagement), and anomaly detection for unexpected output patterns.

Listen for: Whether the candidate distinguishes between infrastructure metrics (latency, error rate, token usage) and quality metrics (output quality, user satisfaction proxies). Strong AI engineers track both and know which one to look at when something goes wrong. Also listen for whether they log inputs and outputs for debugging — many teams do not, and it makes production diagnosis extremely difficult.

18. How do you handle context window limitations in a document processing pipeline at scale?

What a strong answer demonstrates: Practical knowledge of the approaches available — map-reduce patterns for long documents, rolling summarization, hierarchical indexing, or selective retrieval — and the ability to reason about when each is appropriate.

Listen for: Whether the candidate thinks about the quality implications of each approach alongside the technical mechanics. Map-reduce over a long document works for some tasks (summarization, extraction) but fails for others (reasoning across the full document). Strong AI engineers know which pattern preserves the information their specific use case needs.

19. What are the most common failure modes you have seen in LLM-powered production features, and how did you address them?

What a strong answer demonstrates: Hard-won production experience — the specific failure modes that only emerge under real traffic: latency spikes from upstream API issues, unexpected output format changes after model updates, context poisoning from bad retrieved documents, and user jailbreak patterns that bypass guardrails.

Listen for: Specificity and operational detail. Strong AI engineers can describe a real incident, how they diagnosed it, and what they changed to prevent recurrence. This question is one of the clearest proxies for whether a candidate has truly operated LLM systems in production. Candidates who describe only theoretical failure modes have likely not shipped anything under real load.