LLMs and APIs concepts

LLMs and APIs - Key Ideas

1. Large Language Models generate by predicting tokens

Key idea: An LLM generates a response by repeatedly predicting likely next tokens based on the input and context it receives.

A Large Language Model does not retrieve a complete prepared answer from a database. It generates a response step by step by predicting which token is likely to come next based on the input and context it has received.

This explains an important characteristic of LLMs: they can produce fluent and convincing text without guaranteeing that the information is correct. The model is generating a likely continuation, not performing a traditional database lookup.

Remember: An LLM is primarily a generation and prediction system, not a source of guaranteed facts.

2. Tokens and context define what the model can work with

Key idea: LLMs process tokens, and the context window limits how much information the model can work with at one time.

LLMs process text as tokens rather than directly as words or sentences. A token may represent a whole word, part of a word, punctuation, or another piece of text.

The model also has a limited context window. This is the amount of information it can consider when generating a response. The prompt, instructions, conversation history, documents, and generated output all consume context.

For developers, tokens matter because they affect how much information can be sent to the model and can also affect API usage and cost.

Remember: The model can only generate an answer based on the information available in its current context and what it learned during training.

3. The prompt is part of the application’s behavior

Key idea: Instructions, context, constraints, and desired output format can significantly affect the generated result.

The quality and structure of the input given to an LLM can significantly influence its output.

Compare:

Explain recursion.

with:

Explain recursion to a first-semester computer science student.  Use a simple Java example and keep the explanation below 100 words.

The second prompt provides a target audience, format, programming language, and length constraint. These instructions help guide the model toward the kind of response the application needs.

In an LLM-powered application, prompts and instructions should therefore be considered part of the application’s design and behavior rather than just questions typed by a user.

Remember: Good prompts provide useful context, clear instructions, constraints, and an expected output format.

4. An LLM API generates rather than simply retrieves

Key idea: Traditional APIs generally retrieve or calculate according to a defined contract. LLMs generate responses probabilistically.

A traditional API normally has a well-defined contract. If we request a particular resource twice and the underlying data has not changed, we generally expect the same result.

An LLM API is different because its main purpose is to generate a response. The output can vary even when the input is similar or identical, depending on the context and the model’s internal state and the temperature parameter (which controls the randomness of the output).

This means developers cannot always make the same assumptions they make with traditional APIs. Applications using LLMs need to handle variation, unexpected answers, and potentially incorrect content.

Remember: A traditional API usually gives us data according to a defined contract. An LLM API gives us generated output that may vary dependending on the input, context, and randomness in the model’s generation process.

5. LLM output should be treated as untrusted input

Key idea: Treat LLM-generated output as untrusted input. Valid JSON does not mean valid or correct information.

An LLM can produce an answer that looks convincing but is incorrect. This is often referred to as a hallucination.

Structured output does not remove this problem. For example:

{
  "price": -500,
  "quantity": 999999999
}

This is perfectly valid JSON, and Jackson could successfully convert it into a Java DTO. That does not mean the values make sense for the application.

For this reason, LLM output should be treated similarly to user input or data received from an external system. The application should validate important values before storing them, executing actions, or using them in business logic.

Remember: Successful deserialization tells us that the structure is valid. It does not tell us that the generated information is correct.

6. Structured output connects LLMs with normal application code

Key idea: Structured output creates a bridge between LLM-generated content and conventional Java application code.

Natural-language responses are useful when the result is intended for a human. Applications, however, often need predictable data structures.

Instead of receiving:

The capital of Denmark is Copenhagen.

an application may prefer:

{
  "capital": "Copenhagen",
  "country": "Denmark"
}

The structured response can be converted into a normal Java DTO:

public record CountryInfo(
    String capital,
    String country
) {}

The application can then continue using familiar programming techniques: DTOs, validation, business logic, databases, and other APIs.

Structured output therefore provides an important bridge between the probabilistic world of LLM generation and the deterministic parts of a traditional Java application.

Remember: Let the LLM handle language understanding or generation, then convert its result into validated structures that ordinary application code can work with.

Overall takeaway

Use deterministic code when the problem is deterministic. Use an LLM when language understanding or generation provides a meaningful advantage.

Using an LLM through an API does not replace normal software development.

A useful architecture is:

User input
    |
    v
Prompt / instructions
    |
    v
LLM API
    |
    v
Generated output
    |
    v
Structured data
    |
    v
Validation
    |
    v
Java DTOs
    |
    v
Application logic

The HTTP, JSON, DTO, and validation concepts are already familiar from traditional API development.

The important difference is that the LLM generates its output rather than simply returning predictable application data. That makes prompt design, structured output, validation, and appropriate trust especially important.