Introduction

In my previous comparison of Genkit Go and ADK Go 2.0 I covered the everyday tasks. This article is about the agent layer, and instead of comparing features one by one, I did what you would actually do at work: I built something complex in one framework and then tried to build the same thing in the other.

Same rules as always: every program here compiled and ran. Genkit Go v1.11.0 and ADK Go v2.1.0, gemini-3-flash-preview plus gpt-5-mini, 2026-08-05. Where I cite internals, they come from the released module sources.

One caveat upfront so it frames everything: Genkit’s agents API sits behind genkit.WithExperimental() and can change between minor releases, while ADK’s agent stack is GA. That is a real ADK advantage. Everything else in this article is about what you can build.

The build: a research team you can fire and forget

Here is the spec. It is deliberately the kind of thing a real product asks for:

1. An orchestrator agent that coordinates a researcher and a writer.
2. The sub-agents save their work as files (findings.md, draft.md) that end up in one place.
3. The system survives a model outage: if the orchestrator’s primary model fails, it falls back to a different vendor.
4. Dangerous tools need human approval before they run.
5. The client fires the task and disconnects; the result waits server-side.

In Genkit this is one program of 90 lines, and it ran on the first try. The interesting part is that requirements 1, 2 and 3 are not code at all. They are three entries in a middleware list:

orchestrator := genkitx.DefineAgent(g, "orchestrator",
    aix.InlinePrompt{
        // The primary model is BROKEN on purpose.
        ai.WithModelName("googleai/gemini-nonexistent-model"),
        ai.WithSystem("You coordinate a team. First delegate research to the researcher, then delegate a short draft to the writer. Finally reply with a one-sentence status."),
        ai.WithUse(
            &middleware.Fallback{Models: []ai.ModelRef{
                ai.NewModelRef("openai/gpt-5-mini", nil),      // req 3: cross-vendor healing
            }},
            &middlewarex.Agents{                                // req 1: the team
                Agents:           []aix.AgentRef{researcher.Ref(), writer.Ref()},
                MaxDelegations:   4,
                ArtifactStrategy: middlewarex.ArtifactStrategySession,
            },
            &middlewarex.Artifacts{Readonly: true},             // req 2: read the team's files
        ),
    },
    aix.WithSessionStore(localstore.NewInMemorySessionStore[any]()),
)

The researcher and writer are each ~8 lines: an InlinePrompt with a Gemini model, a system prompt, and &middlewarex.Artifacts{} so they get write_artifact tools. Requirement 5 is the client side:

conn.Send(&aix.AgentInput{
    Message: ai.NewUserTextMessage("Research why Go suits AI agents, then have the writer produce a short draft."),
    Detach:  true,
})
out, _ := conn.Output() // returns immediately
// later, from anywhere:
snap, _ := orchestrator.GetSnapshot(ctx, out.SnapshotID)

And the run, verbatim:

detached after 1ms, snapshot fc70253d-55f8-4890-a587-a6aa74d69438
settled: completed after 58s
artifact "researcher_1/findings.md" (5041 chars) from researcher
artifact "writer_2/draft.md" (3840 chars) from writer
orchestrator: I delegated the research task to the researcher (findings.md saved)
and the short draft to the writer (draft.md saved). Status: research and draft
are complete and saved as researcher_1/findings.md and writer_2/draft.md.

Read that output again, because a lot happened in those 58 seconds while no client was connected: the orchestrator’s broken Gemini model failed on every turn and healed onto OpenAI every time, it delegated to two Gemini-powered sub-agents through auto-generated delegate_to_* tools, their artifacts were merged into the orchestrator’s session with source metadata, and the whole thing settled into a snapshot I could read later from another process. A multi-vendor agent team, running unattended, in 90 lines.

Why this is easy: middleware composes with agents

The capstone works because of one architectural decision: in Genkit, a middleware is a unit that can inject tools, wrap the model call, and wrap tool execution, and the same unit plugs into a plain Generate call or an agent’s prompt with the same ai.WithUse(...) line. Capabilities become list entries.

The shelf in v1.11.0’s plugins/middleware: Retry, Fallback, ToolApproval, Filesystem (sandboxed file tools), Skills, plus the experimental Agents (delegation) and Artifacts. Two of them deserve their own demos, both of which I ran.

Cross-vendor fallback is configuration

The plugin ecosystem and the middleware system meet here. Any model from any plugin can be a fallback target for any other, so surviving a vendor outage is this, and nothing else:

g := genkit.Init(ctx, genkit.WithPlugins(
    &googlegenai.GoogleAI{},
    &openai.OpenAI{},
))

text, err := genkit.GenerateText(ctx, g,
    ai.WithModelName("googleai/gemini-nonexistent-model"), // broken
    ai.WithPrompt("Say hi in three words."),
    ai.WithUse(
        &middleware.Retry{MaxRetries: 1},
        &middleware.Fallback{Models: []ai.ModelRef{
            ai.NewModelRef("openai/gpt-5-mini", nil),
        }},
    ),
)

My run: without the middleware, Error 404, Message: models/gemini-nonexistent-model is not found. With it: Hello there, friend. from OpenAI. 37 lines total, and the same two lines work inside an agent, which is exactly how the capstone’s orchestrator heals itself.

Human approval is configuration too

ToolApproval wraps tool execution and interrupts any tool not on its allowlist. The tool itself knows nothing about approvals:

resp, _ := genkit.Generate(ctx, g,
    ai.WithPrompt("Delete the files in /tmp/scratch please."),
    ai.WithTools(deleteFiles),
    ai.WithUse(&middleware.ToolApproval{}), // nothing pre-approved
)
// resp.FinishReason == "interrupted"

part, _ := deleteFiles.RestartWith(p,
    ai.WithResumedMetadata[DeleteInput](map[string]any{"toolApproved": true}))
final, _ := genkit.Generate(ctx, g,
    ai.WithMessages(resp.History()...), ai.WithToolRestarts(part), /* ... */)

My run: finish reason: interrupted, then after approval, OK. I've deleted the files in /tmp/scratch. Governance became a deployment decision: you can add approval to an existing agent without touching a single tool.

The same build in ADK 2.0, piece by piece

Now the other side, requirement by requirement, with everything verified against the v2.1.0 sources or actually built.

The team (req 1): ADK has this, GA. SubAgents with the auto-injected transfer_to_agent tool, or agenttool for function-call delegation. No complaints; this is ADK’s home turf.

Approval (req 4): ADK has this too, GA, and the declaration is even simpler. functiontool.Config{RequireConfirmation: true} is one flag. I ran the full round-trip in both frameworks and they land within one line of each other (102 vs 101 lines). The difference is the shape: ADK’s wire flow is a synthetic adk_request_confirmation function call answered with a map[string]any{"confirmed": true} payload, while Genkit types the pause payload and the resume payload as Go structs and validates every resume against history. Pick your poison: a flag with an untyped protocol, or a typed protocol you wire yourself.

Artifacts (req 2): partial. ADK has an artifact service (in-memory or GCS) and a loadartifactstool. What it does not have is the merging story: sub-agent artifacts flowing into the parent session with source attribution is what ArtifactStrategySession gave me for free.

Cross-vendor fallback (req 3): you build it yourself, and it bites. ADK has no middleware and its callbacks cannot wrap the model call, so the seam is writing a custom model.LLM decorator. I built it: buffer the primary’s output, and on failure call the backup. There is a trap here, and my first version hit it. The backup vendor rejected the call with:

"The requested model 'gemini-nonexistent-model' does not exist."

ADK’s LLMRequest carries the model name inside the request, so my OpenAI backup was asked to run the Gemini model. The decorator also has to rewrite the request:

backupReq := *req
backupReq.Model = f.backup.Name()

After that fix it worked (Hi there, friend.). Final tally: 86 lines of decorator and ceremony versus two lines of configuration, and the 86-line version is per-agent, static, one-level-deep, and has to handle mid-stream failure buffering itself. This is what “no middleware” costs in practice.

Fire and forget (req 5): not possible. The v2.1.0 source is unambiguous. Runner.Run is a pull-based iterator: nothing executes unless the client consumes events, breaking the loop unwinds the stack, cancelling the context aborts the work, and the REST server passes the request context straight into the run. There is no context.WithoutCancel anywhere in the runner or agent paths. ADK can park at an interrupt and survive a process restart by replaying the session event log, which is genuinely robust, but pausing at a boundary is not continuing to work after the client leaves.

So the capstone scorecard: two requirements ADK meets as well or better (team, approval flag), one it meets partially (artifacts), one it makes you hand-build with sharp edges (fallback), and one it cannot do (detach). And each of the pieces it does have arrives through a different mechanism: a config field here, a toolset there, a plugin bundle, a hand-written decorator. Genkit’s version is one composition model for all of them.

The pieces in detail

The capstone leaned on agent features that deserve their own numbers. All of these ran, in both frameworks where both have them.

Typed session state

A Genkit agent’s state is a Go struct. A tool mutates it, every change streams to the client as a JSON-Patch, and the client reads it back typed:

type GameState struct {
    Score int      `json:"score"`
    Moves []string `json:"moves"`
}

// inside a tool
if s := aix.SessionFromContext[GameState](ctx); s != nil {
    s.UpdateCustom(func(st GameState) GameState {
        st.Score += in.Points
        st.Moves = append(st.Moves, in.Move)
        return st
    })
}

My run streamed [{"op":"replace","path":"","value":{"moves":["e4"],"score":10}}] and conn.Custom() returned score=20 moves=[e4 Nf3] as a struct. Prompts read the same state with ``.

ADK’s session state is Get(string) (any, error) / Set(string, any): a map, type assertions at every read, no schema. It works (my run: score=20 (int)), and instruction strings can template state with {score}, but nothing checks anything at compile time. The 2.0 graph engine does have typed parameter binding, but only for workflow function nodes, not for session state in a chat agent.

Custom agent loops

Both frameworks let you write a fully custom agent. In ADK you cannot implement the agent.Agent interface directly, since it is sealed and the compiler rejects any attempt with missing method internal; the sanctioned path is agent.New(agent.Config{Run: ...}), a full execution override, and it ran in my tests.

The difference is what you write. Genkit’s SessionRunner hands you a turn loop with history, per-turn tracing, snapshot writes and panic recovery built in; my draft-then-self-critique agent is 57 lines. The ADK version of the same agent is 96 lines, because a Run override means producing raw session.Events: dig the question out of the event log, call the raw model layer, construct and yield events yourself.

Agents defined in .prompt files

DefinePromptAgent binds an agent to a .prompt file by name; the model, config and persona are data, not Go:

pirate := genkitx.DefinePromptAgent(g, "pirate",
    aix.WithSessionStore[any](localstore.NewInMemorySessionStore[any]()),
)

26 lines of Go, and my run answered: Rewrite 'em in Python only if ye want yer agents movin' as slow as a barnacle-covered scow in a dead calm. Non-engineers edit the persona without touching code, and the same dotprompt format works across Genkit’s other language SDKs. ADK’s equivalent is an Instruction string or an InstructionProvider where you load files yourself; its YAML agent-config system exists but lives in an internal/ package only ADK’s own CLI can reach.

Where ADK’s agent stack is ahead

The list I would actually weigh on the other side, verified in the same source dive:

• GA stability, against everything above being on Genkit’s experimental track.
• The graph engine as the runtime: every ADK agent runs as a node in the workflow engine, and the node kit (typed function nodes, routing, per-node schema validation, retries, nested graphs) is public API.
• Schema-validated human input in workflows: a graph node can declare a JSON schema for the human’s answer, validated on resume. For form-like HITL inside a pipeline, that is ahead of anything Genkit ships.
• Task isolation: ModeTask sub-agents get their own conversation scope with an auto-injected finish_task tool and scope-filtered history.
• Live audio: RunLive with modalities, speech config and transcription has no Genkit Go equivalent.

The scorecard

Non-blank lines including imports. Every program ran.

Conclusion

The agent layer is where Genkit’s two architectural bets pay off together. The plugin ecosystem means models from different vendors, local runtimes and vector stores are interchangeable references. The middleware system means capabilities (delegation, artifacts, fallback, retries, approval, sandboxed files, skills) are composable list entries that work identically in a one-shot generate and inside an agent. Stack those two on top of typed sessions, snapshots and detach, and a fire-and-forget multi-vendor agent team is a 90-line program. I know because I ran it, and I also know what the same spec costs in ADK because I built the pieces: one requirement hand-written with real sharp edges, and one that cannot be built at all.

ADK’s agent stack remains the right pick when its strengths are your requirements: GA guarantees, graph workflows with schema validation, task isolation, live audio, and the Google Cloud platform around it. But if the question is “how hard is it to build something complex”, the answer I measured is: in Genkit, complexity composes; in ADK, each capability is its own project.

Further reading:

• Genkit Go vs ADK Go 2.0: a Hands-On Look at Boilerplate and Low-Level Control
• Genkit Go basic-agents sample
• Genkit middleware documentation
• ADK 2.0 release notes
• Genkit GitHub repository
• ADK Go GitHub repository

Introduction

Genkit and ADK (Agent Development Kit) are both Google-backed Gen AI frameworks with first-class Go SDKs, and I keep getting the same question: what is actually different when you sit down and write the code?

This is not a “which one is better” article. They point at different problems: ADK is an agent orchestration framework (agents, sessions, multi-agent transfer, and since 2.0 a graph-based workflow engine), while Genkit is a Gen AI application framework (model calls, flows, structured output, middleware). But they overlap on the everyday tasks every Go developer does (call a model, get typed output, stream, add tools, intercept requests), and on those tasks the developer experience diverges a lot.

So instead of comparing docs, I wrote the same eight programs twice and ran every single one against a live Gemini backend:

1. One prompt → one answer
2. A raw model call with no framework machinery
3. Structured output into a Go struct
4. Streaming
5. Tool calling
6. Intercepting / wrapping the model call
7. A custom model backend
8. Multi-turn conversation and HTTP serving

Everything below compiled and ran on Genkit Go v1.11.0 and ADK Go v2.1.0 (the google.golang.org/adk/v2 module, GA since June 30, 2026) with gemini-3-flash-preview, on 2026-08-05. Where I quote framework internals, they come from those released versions, not from the main branch. Nothing in this article is speculated from documentation.

TL;DR

The pattern that emerged: Genkit’s primitive is the model call; ADK’s primitive is the agent. When the thing you need is a model call, Genkit hands it to you directly and ADK makes you rent the whole agent apparatus. When the thing you need is a long-lived, multi-agent system, ADK’s apparatus stops being boilerplate and starts being the product.

The mental model

Genkit: the model call is the unit

Genkit Go has exactly one setup object (*genkit.Genkit) and everything else is a package-level function call:

g := genkit.Init(ctx, genkit.WithPlugins(&googlegenai.GoogleAI{}))

text, err := genkit.GenerateText(ctx, g,
    ai.WithModelName("googleai/gemini-3-flash-preview"),
    ai.WithPrompt("Why is Go a great language for building AI agents? One short sentence."),
)

That is the complete program (plus imports and an error check). Init doesn’t even return an error: configuration problems panic at startup, and from there on there is a single err to handle per generation. My runnable version is 21 non-blank lines with 2 error-handling lines.

ADK: the agent is the unit

ADK has no “just generate” entry point at the framework level. The unit of execution is an agent, and an agent runs inside a runner, and a runner requires a session service. This is the minimal equivalent program:

model, err := gemini.NewModel(ctx, "gemini-3-flash-preview", &genai.ClientConfig{
    APIKey: os.Getenv("GOOGLE_API_KEY"),
})
// if err != nil ...

a, err := llmagent.New(llmagent.Config{
    Name:        "assistant",
    Model:       model,
    Instruction: "You are a helpful assistant.",
})
// if err != nil ...

r, err := runner.NewInMemory("hello-app", a) // in-memory session/artifact/memory services
// if err != nil ...

msg := genai.NewContentFromText("Why is Go a great language for building AI agents? One short sentence.", genai.RoleUser)

var text string
for event, err := range r.Run(ctx, "user-1", "session-1", msg, agent.RunConfig{}) {
    if err != nil { /* ... */ }
    if event.LLMResponse.Content == nil {
        continue
    }
    if event.IsFinalResponse() {
        for _, part := range event.LLMResponse.Content.Parts {
            text += part.Text
        }
    }
}

49 non-blank lines, 8 error-handling lines for the same output. And to be fair, runner.NewInMemory is already ADK being nice: the explicit form makes you wire runner.Config{AppName, Agent, SessionService, AutoCreateSession} yourself.

Count the concepts you must hold for one prompt: a model, an agent with a Name (mandatory), an app name, a user ID, a session ID (both meaningless for a one-shot call, both mandatory positional arguments), a RunConfig, an event iterator, an IsFinalResponse() predicate, and manual concatenation of Content.Parts. There is no .Text() helper on events. Even ADK’s own console launcher spends ~45 lines turning events into printable text (cmd/launcher/console/console.go), because every consumer re-implements this loop.

None of this is bad design: sessions, users and apps are exactly the right vocabulary for a deployed multi-agent platform. It is simply the wrong altitude when what you wanted was an answer to a prompt, and there is no lower altitude offered (almost; see the escape hatch below).

Structured output: where the gap is widest

This was the experiment with the biggest difference, and the one where I initially got ADK wrong in an instructive way.

Genkit

You describe the shape as a Go type. The JSON Schema is inferred from the struct (via reflection over json/jsonschema tags), constrained decoding is enabled by default, and you get the struct back, parsed and typed:

type Recipe struct {
    Name        string   `json:"name"`
    PrepMinutes int      `json:"prep_minutes"`
    Ingredients []string `json:"ingredients"`
}

recipe, _, err := genkit.GenerateData[Recipe](ctx, g,
    ai.WithModelName("googleai/gemini-3-flash-preview"),
    ai.WithSystem("You generate recipes."),
    ai.WithPrompt("A simple tortilla de patatas recipe"),
)

fmt.Println(recipe.Name, recipe.PrepMinutes, recipe.Ingredients)

Output from my run:

Classic Spanish Tortilla de Patatas (15 min): 5 ingredients

28 lines total. The schema definition is the Go type.

ADK

ADK’s llmagent.Config.OutputSchema takes a *genai.Schema: a hand-built tree, field by field, with a hand-maintained Required list. There is no “infer from struct” path for agent output (curiously, ADK does have schema inference, but only for tool inputs):

recipeSchema := &genai.Schema{
    Type: genai.TypeObject,
    Properties: map[string]*genai.Schema{
        "name":         {Type: genai.TypeString},
        "prep_minutes": {Type: genai.TypeInteger},
        "ingredients":  {Type: genai.TypeArray, Items: &genai.Schema{Type: genai.TypeString}},
    },
    Required: []string{"name", "prep_minutes", "ingredients"},
}

a, err := llmagent.New(llmagent.Config{
    Name:         "chef",
    Model:        model,
    Instruction:  "You generate recipes.",
    OutputSchema: recipeSchema,
})

The schema enforcement worked perfectly in my runs: Gemini returned valid JSON every time. The interesting part is what happens next. My first attempt read the event’s Output field, which looks exactly like what you want. It was always nil.

Digging into the released v2.1.0 source: the LlmAgent does parse the structured output internally, and does attach it to the event, but the runner deliberately strips it before the event reaches you. In runner/runner.go, every non-partial agent reply gets cloned with clone.Output = nil. The parsed value exists to feed ADK 2.0’s graph workflow engine (typed routing between nodes), not for you. As a consumer of Runner.Run, you receive raw JSON as text and parse it yourself:

var jsonText string
for event, err := range r.Run(ctx, "user-1", "session-1", msg, agent.RunConfig{}) {
    // ... error handling, nil checks, IsFinalResponse, Parts loop ...
    jsonText += part.Text
}

var recipe Recipe
if err := json.Unmarshal([]byte(jsonText), &recipe); err != nil { /* ... */ }

(If you set OutputKey, the same raw string is also copied into session state, still not parsed for you.)

Two more sharp edges I hit in the source while verifying this:

• The OutputSchema doc comment in v2.1.0 still warns: “when this is set, agent can only reply and cannot use any tools, such as function tools, RAGs, agent transfer, etc.” The implementation has grown workarounds (on some backends it injects a synthetic set_model_response tool; on Vertex AI with Gemini ≥2.0 it can use native constrained output alongside tools), which means combining tools and output schema behaves differently depending on backend and model name, silently.
• Final tally: 68 lines vs Genkit’s 28, and the typed half of “typed output” is entirely yours to write.

Streaming: deltas vs partial events

Both frameworks stream. The difference is who assembles the result.

Genkit gives you a callback that receives deltas, and the final response arrives already assembled:

resp, err := genkit.Generate(ctx, g,
    ai.WithModelName("googleai/gemini-3-flash-preview"),
    ai.WithPrompt("Count from one to ten in words, one per line."),
    ai.WithStreaming(func(ctx context.Context, chunk *ai.ModelResponseChunk) error {
        fmt.Print(chunk.Text())
        return nil
    }),
)

My run printed streamed chars: 48, final response chars: 48. Chunks and final response account for the same text exactly once each.

ADK streams by setting agent.RunConfig{StreamingMode: agent.StreamingModeSSE} and then flags chunk events with LLMResponse.Partial. The gotcha: after all the partial events, ADK emits a final non-partial event that repeats the entire text. My run: partial (streamed) chars: 48, final event chars: 48. Print events naively and you output everything twice. Every ADK consumer must carry this filter:

for event, err := range r.Run(ctx, "user-1", "session-1", msg, agent.RunConfig{
    StreamingMode: agent.StreamingModeSSE,
}) {
    // ...
    if event.LLMResponse.Partial {
        fmt.Print(part.Text) // the streamed deltas
    }
    // the final event repeats the full text: skip it, or you print it twice
}

This isn’t hypothetical: ADK’s own example code and console launcher both contain hand-rolled deduplication (if text != prevText in cmd/launcher/console/console.go). It’s a small thing, but it’s a small thing every single ADK streaming consumer writes, and no Genkit consumer does. As a bonus, Genkit’s GenerateDataStream[T] even streams partially-parsed typed structs (your Recipe filling in field by field), which has no ADK equivalent.

Tool calling: closer than I expected

Here I expected a bigger gap and didn’t find one: ADK’s function tools are genuinely well designed. Both frameworks let a tool be an ordinary typed Go function, with the JSON Schema inferred from the input struct:

// Genkit
weather := genkit.DefineTool(g, "getWeather", "Gets the current weather for a city",
    func(ctx *ai.ToolContext, in WeatherInput) (string, error) {
        return fmt.Sprintf("22C and sunny in %s", in.City), nil
    })

resp, err := genkit.Generate(ctx, g,
    ai.WithModelName("googleai/gemini-3-flash-preview"),
    ai.WithPrompt("What is the weather in Valencia right now?"),
    ai.WithTools(weather),
)

// ADK: also nice! Generics, schema inferred, type params never spelled out.
weatherTool, err := functiontool.New(functiontool.Config{
    Name:        "get_weather",
    Description: "Gets the current weather for a city",
}, func(_ agent.Context, in WeatherInput) (WeatherOutput, error) {
    return WeatherOutput{Report: fmt.Sprintf("22C and sunny in %s", in.City)}, nil
})

Both ran the tool and produced the right answer. The differences are around the edges:

• Ceremony: the Genkit program is 29 lines; the ADK one is 66, because the tool still needs the agent/runner/session stack around it.
• Where the loop lives: Genkit’s tool loop runs inside Generate (with parallel tool execution and a WithMaxTurns bound), so tools work in a plain one-shot call. In ADK the loop belongs to the agent flow, which is consistent with its philosophy, but it means there is no way to use ADK tools without an agent.
• A public-API gotcha: ADK’s public tool.Tool interface is only Name()/Description()/IsLongRunning(). The executable contract (Declaration(), Run()) lives in an internal/ package you cannot reference, so you can’t hand-implement a runnable tool against a public interface; you go through functiontool.New. Genkit’s equivalent (ai.ToolFunc + DefineTool) is fully public.
• Composability limit: ADK documents (in its own examples/tools/multipletools) that you cannot mix built-in Gemini tools like geminitool.GoogleSearch{} with custom function tools on one agent; the workaround is wrapping each in a sub-agent.
• Genkit also has tool interrupts/resume as a stable API (pause generation from inside a tool, get an Interrupts() list, resume with RespondWith/RestartWith). ADK’s equivalents are tool confirmation flows wired through its HITL machinery.

Getting low-level

You can go deeper in Genkit than in ADK, and its middleware system is the reason: Genkit exposes the model call, the tool executions and the whole generation loop as wrappable layers, while ADK exposes fixed callback points and one wrappable interface. Here is what that means in code.

Wrapping the model call

Genkit middleware is a real onion: you get the request and a next() you control. Timing, retry, caching and request rewriting are all one closure, applied per call:

timing := ai.MiddlewareFunc(func(ctx context.Context) (*ai.Hooks, error) {
    return &ai.Hooks{
        WrapModel: func(ctx context.Context, params *ai.ModelParams, next ai.ModelNext) (*ai.ModelResponse, error) {
            start := time.Now()
            resp, err := next(ctx, params)
            log.Printf("model call: %d messages in, took %s",
                len(params.Request.Messages), time.Since(start))
            return resp, err
        },
    }, nil
})

resp, err := genkit.Generate(ctx, g, /* ... */, ai.WithUse(timing))

Ran, printed model call: 1 messages in, took 3.323s. There are WrapGenerate and WrapTool hooks at the other two altitudes (the whole tool loop, and each tool execution), middleware can inject tools, and the built-ins (Retry, Fallback, tool approval, filesystem sandbox) compose the same way.

ADK has callbacks, not middleware: BeforeModelCallback and AfterModelCallback with fixed signatures. They can short-circuit (return a response from Before and the model is never called, the documented caching hook) and they can replace the response, but they cannot wrap. There is no next(). To time a call I had to smuggle state between two separate functions:

var start time.Time

before := func(ctx agent.Context, req *model.LLMRequest) (*model.LLMResponse, error) {
    start = time.Now()
    return nil, nil // nil means "continue to the model"
}
after := func(ctx agent.Context, resp *model.LLMResponse, respErr error) (*model.LLMResponse, error) {
    log.Printf("took %s", time.Since(start))
    return nil, nil
}

This works (ran, took 1.631s) but it’s a workaround, it’s attached per-agent rather than per-call, and anything genuinely wrap-shaped (hedged requests, per-attempt timeouts, fallbacks across models) has to move to a different mechanism entirely: decorating the model. Which brings me to the thing ADK gets really right:

Custom model backends: local Gemma on Ollama

Nobody plugs in an echo model, so no toy examples here. The realistic scenario is a backend the framework didn’t bless for you: a local model server, an internal gateway, a provider without an official plugin. I tested two levels of this: running Google’s Gemma locally on Ollama, and writing a backend from scratch.

Genkit ships an official Ollama plugin, so the local model is a first-class provider:

oll := &ollama.Ollama{ServerAddress: "http://localhost:11434"}
g := genkit.Init(ctx, genkit.WithPlugins(oll))

gemma := oll.DefineModel(g, ollama.ModelDefinition{Name: "gemma3n:e4b", Type: "chat"}, nil)

resp, err := genkit.Generate(ctx, g,
    ai.WithModel(gemma),
    ai.WithPrompt("Why is Go a great language for building AI agents? One short sentence."),
)

23 lines, ran on the first try against gemma3n:e4b on my Mac, and the local model gets the same Generate, flows, middleware and structured output as any cloud model. The plugin covers embedders too.

ADK has no Ollama integration of its own. The path that works is the OpenAI-compatible adapter pointed at Ollama’s OpenAI endpoint:

gemma, err := openaimodel.NewModel(ctx, "gemma3n:e4b", &openaimodel.ClientConfig{
    BaseURL: "http://localhost:11434/v1",
    APIKey:  "ollama", // any non-empty value; Ollama ignores it
})

This also ran (47 lines with the usual agent, runner and session around it), and it is a documented path: the package README lists Ollama, LM Studio and vLLM. The difference is the shape of the support. In Genkit the local model is a provider with its own plugin; in ADK it rides on OpenAI compatibility, so anything Ollama offers outside that API surface is out of reach, and then you are writing the backend yourself, which is the next scenario.

And when there is no plugin at all

When the backend has no plugin and no OpenAI-compatible endpoint (an internal LLM gateway, a niche provider), you write the integration yourself. I did, in both frameworks: the same minimal OpenAI chat/completions client, roughly 45 lines of identical plain-net/http glue, run against the real OpenAI API.

ADK’s model.LLM interface is the smallest custom-backend contract I’ve seen in any framework:

type LLM interface {
    Name() string
    GenerateContent(ctx context.Context, req *LLMRequest, stream bool) iter.Seq2[*LLMResponse, error]
}

Two methods, and this interface is also ADK’s genuine wrapping seam: ADK’s own examples ship a resilientModel decorator doing retries and per-attempt timeouts this way. My OpenAI-backed model.LLM worked, with two costs on top of the shared HTTP glue. First, the vocabulary: requests and responses are google.golang.org/genai types, so you translate both ways, and some of it is non-obvious. The agent’s Instruction arrives in Config.SystemInstruction, not in Contents; miss that and your backend silently drops the system prompt. (ADK’s own OpenAI adapter is four files of translation code, which tells you how deep that rabbit hole goes. Also, parts of ADK sniff behavior from llm.Name(), so a renaming wrapper can silently change how output schemas are handled.) Second, the invocation: those two beautiful methods still need the agent, runner and session stack around them to run. Total: 116 lines.

The Genkit version of the same backend is the HTTP glue, a capability declaration, and nothing else:

func callOpenAI(ctx context.Context, req *ai.ModelRequest, cb func(context.Context, *ai.ModelResponseChunk) error) (*ai.ModelResponse, error) {
    body := oaRequest{Model: "gpt-5-mini"}
    for _, m := range req.Messages {
        role := string(m.Role) // "system" | "user" | "model"
        if role == "model" {
            role = "assistant"
        }
        body.Messages = append(body.Messages, oaMessage{Role: role, Content: m.Text()})
    }
    // ... plain net/http POST to api.openai.com/v1/chat/completions ...
    return &ai.ModelResponse{
        FinishReason: ai.FinishReasonStop,
        Message:      ai.NewModelTextMessage(out.Choices[0].Message.Content),
    }, nil
}

gpt := genkit.DefineModel(g, "myco/gpt-5-mini", &ai.ModelOptions{
    Supports: &ai.ModelSupports{Multiturn: true, SystemRole: true},
}, callOpenAI)

resp, err := genkit.Generate(ctx, g,
    ai.WithModel(gpt),
    ai.WithSystem("You are a helpful assistant."),
    ai.WithPrompt("Why is Go a great language for building AI agents? One short sentence."),
)

Total: 79 lines, and Generate calls it directly; the model immediately works with structured output, streaming, middleware and flows. The messages arrive as provider-neutral structs (ai.Message, with system prompt as a regular system-role message), so the translation is one role-mapping loop instead of a genai round-trip. One honest note: my first attempt failed at runtime because I passed nil options, since a Genkit model must declare what it Supports. That cost me one extra struct literal, and it’s the flip side of a real feature: the framework validates every request against the declared capabilities and even simulates a system prompt for models that don’t support one.

The raw call escape hatch

I didn’t want to write “you can’t go low-level in ADK” without trying it myself, so I tried. model.LLM is public, and this compiles and runs (33 lines in my experiment):

llm, _ := gemini.NewModel(ctx, "gemini-3-flash-preview", &genai.ClientConfig{APIKey: key})

for resp, err := range llm.GenerateContent(ctx, &model.LLMRequest{
    Contents: []*genai.Content{genai.NewContentFromText(prompt, genai.RoleUser)},
}, false) {
    // resp.Content.Parts ...
}

But notice what you’re holding: this is the bare genai transport with an ADK-shaped envelope. No instruction templating, no tool loop, no callbacks, no structured output processing, no telemetry: all of that lives inside the agent flow you just bypassed. There are no helpers (Text(), message builders) at this layer, and no example in the entire ADK repo uses it; only tests do. It’s an escape hatch, not a supported altitude.

That’s the crux of the low-level story: in Genkit, dropping down doesn’t cost you the framework. Generate is the low-level call: raw messages in, with middleware, tools, schemas and tracing still active. In ADK, the abstraction is a cliff: you’re either at agent altitude with everything, or at transport altitude with nothing.

Where ADK’s ceremony pays off

I also ran the experiments where I expected ADK’s extra structure to earn its keep. It mostly did.

Multi-turn conversation. Both frameworks carry the conversation for you; neither makes you thread history by hand.

In ADK, I called r.Run twice with the same session ID and the second turn knew my name and city. The session service is the conversation, and swapping InMemoryService for the database or Vertex AI implementation brings persistence with it.

In Genkit, the agents API does the same job with a session store. I ran the equivalent two turns on v1.11.0:

g := genkit.Init(ctx,
    genkit.WithPlugins(&googlegenai.GoogleAI{}),
    genkit.WithExperimental(),
)

assistant := genkitx.DefineAgent(g, "assistant",
    aix.InlinePrompt{
        ai.WithModelName("googleai/gemini-3-flash-preview"),
        ai.WithSystem("You are a helpful assistant."),
    },
    aix.WithSessionStore(localstore.NewInMemorySessionStore[any]()),
)

first, err := assistant.RunText(ctx, "My name is Xavi and I live in Valencia. Say hi in one sentence.")
// ...
second, err := assistant.RunText(ctx, "Where do I live? Answer with just the city name.",
    aix.WithSessionID[any](first.SessionID))

Both frameworks answered turn 2: Valencia in my runs, with zero history plumbing on either side. And Genkit’s session layer goes further than a message log: session snapshots, detach/resume (the client disconnects, the server keeps working, you resume from a snapshot) and artifacts, with in-memory and file stores shipping today plus a SessionStore interface for your own persistence. For conversation state, the two frameworks are at parity. (If you are outside an agent, composing plain Generate calls, ai.WithMessages(resp.History()...) threads history in one line, but that is a choice, not a limitation.)

The one difference I would plan around is the maturity label. ADK’s session services are GA and are the default path; Genkit’s agents API is on the experimental track behind genkit.WithExperimental() and can change between minor releases, which is also why the rest of this article compares the stable surfaces. If you are comfortable riding the experimental track, Genkit gives you all of it today; if you need the stability guarantee, ADK has it.

Multi-agent. ADK’s sub-agents get an auto-injected transfer_to_agent tool, there are sequential/parallel/loop workflow agents, and 2.0’s headline feature is a full graph workflow engine with typed routing between nodes, retries and human-in-the-loop pauses.

Genkit delegates too. The Agents middleware injects one delegate_to_<name> tool per sub-agent, discovers each sub-agent’s description automatically for the orchestrator’s prompt, forwards recent conversation history, and can merge sub-agent artifacts back into the orchestrator’s session. I ran an orchestrator delegating to a poet sub-agent:

orchestrator := genkitx.DefineAgent(g, "orchestrator",
    aix.InlinePrompt{
        ai.WithModelName("googleai/gemini-3-flash-preview"),
        ai.WithSystem("You are a coordinator. Delegate writing tasks to the right sub-agent using its delegation tool, then return its result verbatim."),
        ai.WithUse(&middlewarex.Agents{
            Agents:         []aix.AgentRef{poet.Ref()},
            MaxDelegations: 2,
        }),
    },
    aix.WithSessionStore(localstore.NewInMemorySessionStore[any]()),
)

43 lines, worked on the first run (I got my haiku back through the orchestrator), and the full researcher-plus-engineer version is in the official basic-agents sample. Same maturity note as sessions: this is the experimental track.

And ADK’s workflow layer, the sequential/parallel/loop agents and the 2.0 graph engine? Genkit’s answer is that the orchestration language is Go itself. Flows compose: a flow calls other flows, sequence is code order, parallel is a goroutine group, a loop is a loop, and every flow stays traced and individually servable. I ran a brief-writing flow that fans out to two sub-flows in parallel and then synthesizes:

brief := genkit.DefineFlow(g, "briefFlow",
    func(ctx context.Context, topic string) (string, error) {
        // Parallel: plain Go concurrency.
        var facts, structure string
        eg, egCtx := errgroup.WithContext(ctx)
        eg.Go(func() error {
            var err error
            facts, err = research.Run(egCtx, topic)
            return err
        })
        eg.Go(func() error {
            var err error
            structure, err = outline.Run(egCtx, topic)
            return err
        })
        if err := eg.Wait(); err != nil {
            return "", err
        }

        // Sequential: the next line of code.
        return genkit.GenerateText(ctx, g,
            ai.WithModelName(model),
            ai.WithPrompt("Write a two-sentence brief about %s using these facts:\n%s\nand this outline:\n%s", topic, facts, structure))
    })

53 lines including both sub-flows, ran on stable APIs, and genkit.Handler(brief) serves it over HTTP like any other flow. So the difference between the two workflow stories is representation, not capability. ADK reifies the pipeline as a data structure (edges, typed routes) executed by an engine that owns retries, pauses and resumability at the graph level; Genkit keeps the pipeline as ordinary Go and covers those concerns with its own pieces (tool interrupts for human-in-the-loop, the retry middleware, and durable execution through integrations like Temporal). Pick by taste: an inspectable graph object, or code you read top to bottom.

The serving platform. Genkit’s HTTP story is its philosophy in miniature. A flow becomes a handler on plain net/http:

mux := http.NewServeMux()
mux.HandleFunc("POST /greetingFlow", genkit.Handler(greetingFlow))
server.Start(ctx, "127.0.0.1:9090", mux)

$ curl -X POST localhost:9090/greetingFlow -d '{"data":"Xavi"}'
{"result":"Hello, Xavi!"}

ADK’s adkrest.NewServer is also a plain http.Handler (nice), but what it mounts is a platform: app listing, session CRUD, SSE runtime, artifacts, debug/trace endpoints. My curl transcript needed two calls (create the session, then run) and got back the full event envelope with usage metadata, state deltas and node info. On top of that, ADK’s launcher embeds the entire ADK Web UI into your binary via go:embed: go run . web webui gives you a chat UI with no npm and no separate process, which is honestly great. (Genkit’s Developer UI is a separate CLI process, though it’s also more than a chat UI: it traces every generate call, middleware hop and tool execution.)

These aren’t contradictions of the boilerplate findings; they’re the same design measured on its own terrain. ADK’s five constructors are the price of a system where conversations persist, agents transfer to each other, and a REST platform falls out of a config struct.

The plugin ecosystems

One more thing I checked in the released modules, because it changes the decision for real projects: what each framework integrates with out of the box.

Genkit Go v1.11.0 ships sixteen plugins in the module: model providers for Google (Gemini API and Vertex AI), Anthropic, OpenAI plus any OpenAI-compatible endpoint, and Ollama; vector stores for AlloyDB, PostgreSQL/pgvector, Pinecone, Weaviate and a local dev store; Firebase and Google Cloud telemetry; MCP; evaluators; and the middleware pack. And that is the Go module alone, before you count the wider JS ecosystem the framework shares its design with.

ADK Go v2.1.0’s integrations are a coherent Google Cloud story: Gemini and Vertex models, the OpenAI-compatible adapter, Apigee; sessions in memory, in a SQL database or on Vertex AI; GCS artifacts; Vertex AI memory; BigQuery agent analytics; MCP toolsets; and deploy tooling for Cloud Run and Agent Engine. What it does not have is a provider ecosystem beyond that: no Anthropic plugin, no vector store integrations, nothing aimed at other clouds.

If your stack is pure Google Cloud, ADK’s set covers it end to end, deploy included. The moment you need Anthropic next to Gemini, a Postgres vector store, or observability outside Google Cloud, Genkit has the plugin and ADK expects you to build it.

The scorecard

Numbers from my runnable programs (non-blank lines including imports; error-handling lines in parentheses):

Line counts are a blunt instrument; the interesting part is what the extra lines are. In every ADK program they are the same lines: construct model, construct agent, construct runner, invent user/session IDs, iterate events, filter partials, concatenate parts. It’s a fixed tax, roughly 25-30 lines, charged on every program regardless of whether sessions and events are relevant to it.

A concrete decision guide

I would reach for Genkit Go when:

• The unit of my product is a model call or a flow: an API endpoint, a pipeline step, a CLI, a backend feature.
• I want typed structured output without maintaining schema trees by hand.
• I want to wrap, retry, fall back, cache or rewrite model calls with composable per-call middleware.
• I’m integrating a custom or non-Google model backend and want the framework features to keep working on top of it.
• I value the Dev UI’s per-call tracing during development.
• I want sessions and agents too, and I’m fine with the experimental track while that API stabilizes.
• My stack mixes vendors: Anthropic or OpenAI next to Gemini, Ollama locally, or Postgres/Pinecone/Weaviate for retrieval. The plugin shelf already covers them.

I would reach for ADK Go 2.0 when:

• The unit of my product is a persistent, session-based agent: a chat product, a support agent, anything where conversation state is the point, and I want it on GA APIs today.
• I need multi-agent coordination: sub-agent transfer, workflow agents, or 2.0’s graph engine with typed routing and HITL pauses.
• I want the serving platform for free: session REST API, SSE, the embedded Web UI, A2A, Cloud Run / Agent Engine deploy tooling.
• My stack is Gemini/Vertex-first and the genai-typed vocabulary is a feature, not a translation burden.

And these compose better than you’d think: nothing stops you from building your model-centric services with Genkit flows and standing up an ADK agent where you genuinely need the agent platform. They’re both idiomatic modern Go (both lean on iter.Seq2 and generics nicely), and they’re solving different layers of the same problem.

Conclusion

After running everything, the facts are these: Genkit lets you get lower-level, and ADK charges more boilerplate. In detail:

• The boilerplate is real and remarkably consistent: ADK charges a fixed ~25-30 line agent tax on every program, because there is no supported altitude between “full agent” and “bare transport”. Genkit’s Generate occupies exactly that missing middle: raw request access with the framework still on.
• The low-level gap is sharpest in structured output (typed structs vs hand-built schema trees plus DIY parsing, and a runner that deliberately strips the parsed value it already has) and in middleware (a real next() vs fixed-signature callbacks).
• But ADK’s ceremony is not waste: it buys GA session persistence, multi-agent transfer, a graph engine and an embedded serving platform, and some of its individual designs (the two-method model.LLM, generic function tools) are excellent Go.

Pick the altitude that matches your product, and don’t confuse “more setup” with “more capable”, or “less setup” with “less serious.” They’re different tools that happen to share a language and a model provider.

Further reading:

• Genkit Go Get Started guide
• ADK 2.0 release notes
• Genkit GitHub repository
• ADK Go GitHub repository
• Genkit Go basic-agents sample (orchestrator, delegation, artifacts)
• Genkit Go flows documentation
• Stop Using Python for Gen AI: Genkit Go
• Vercel AI SDK Middleware vs Genkit Middleware

Introduction

AWS Secrets Manager has a lovely feature: automatic rotation. You attach a rotation Lambda, set a schedule, and every 30 days your RDS password is rotated for you. Secure by default, no human involved.

There’s just one problem, and it’s the one nobody mentions in the demo: your application has no idea it happened.

Your Go service read SecretString once, at startup, and stashed the password in a struct. Thirty days later Secrets Manager quietly rotates it, the old value stops working, your connection pool starts throwing auth errors, and you find out via a pager alert at 3am. The “fix” everyone reaches for is to redeploy on a schedule, or restart pods so the app re-reads the secret — which turns a security feature into an operational chore.

mamori is a Go library I built to close exactly this gap. It watches AWS Secrets Manager (and SSM Parameter Store) and reconciles a rotation into your running process — typed, validated, redacted — so a rotation triggers a connection-pool rebuild instead of a restart.

This is the AWS-focused companion to the launch article, Introducing mamori: Typed, Watchable Config & Secrets for Go. Read that first for the full model; here we go deep on AWS.

The two AWS schemes

The providers/aws module registers two ref schemes:

A few real refs — note the triple slash on Parameter Store, because SSM parameter names are absolute paths that start with /:

aws-sm://prod/api-key            # whole secret value
aws-sm://prod/db#password        # one key out of a JSON secret
aws-ps:///myapp/log-level        # SSM parameter (triple slash!)
aws-ps:///myapp/db#password      # one key out of a JSON SSM parameter

The rule of thumb: secrets (passwords, tokens, keys) belong in Secrets Manager (aws-sm), and plain, non-sensitive config (log levels, worker counts, feature toggles) belongs in the cheaper Parameter Store (aws-ps). mamori lets you mix both in the same struct.

Setup

go get github.com/xavidop/mamori
go get github.com/xavidop/mamori/providers/aws

Register it with a blank import for zero-config (it uses the standard AWS credential chain — environment, shared config, IRSA on EKS, EC2 instance role):

import _ "github.com/xavidop/mamori/providers/aws"

Or, when you need to pin a region or pass a custom client, construct the provider explicitly and hand it to Load/Watch with WithProvider — this takes precedence over the registry for that call:

import awsprov "github.com/xavidop/mamori/providers/aws"

cfg, err := mamori.Load[Config](ctx,
	mamori.WithProvider(awsprov.NewSecretsManager(awsprov.WithRegion("eu-west-1"))),
)

Your runtime’s IAM role needs the obvious permissions, scoped tightly:

{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow", "Action": "secretsmanager:GetSecretValue", "Resource": "arn:aws:secretsmanager:eu-west-1:123456789012:secret:prod/*" },
    { "Effect": "Allow", "Action": "ssm:GetParameter",              "Resource": "arn:aws:ssm:eu-west-1:123456789012:parameter/myapp/*" }
  ]
}

Load, then watch the rotation

Here’s a service that pulls its database password from Secrets Manager and its operational knobs from Parameter Store:

package main

import (
	"context"
	"log"
	"time"

	"github.com/xavidop/mamori"
	"github.com/xavidop/mamori/secret"

	_ "github.com/xavidop/mamori/providers/aws"
)

type Config struct {
	DBPassword secret.String `source:"aws-sm://prod/db#password"`
	APIKey     secret.String `source:"aws-sm://prod/api-key"`
	LogLevel   string        `source:"aws-ps:///myapp/log-level" default:"info" validate:"oneof=debug info warn error"`
	Workers    int           `source:"aws-ps:///myapp/workers" default:"4" validate:"gte=1,lte=256"`
}

func main() {
	ctx := context.Background()

	w, err := mamori.Watch[Config](ctx,
		mamori.WithPollInterval(60*time.Second),
		mamori.OnChange(func(ev mamori.Change[Config]) {
			if ev.Changed("DBPassword") {
				// A rotation just landed — rebuild the pool, no restart.
				pool.Rotate(ev.New.DBPassword.Reveal())
				log.Println("database password rotated and applied live")
			}
		}),
		mamori.OnError(func(err error) { metrics.Inc("config_error") }),
	)
	if err != nil {
		log.Fatal(err)
	}
	defer w.Close()

	cfg := w.Get()
	log.Printf("started with %d workers at %s", cfg.Workers, cfg.LogLevel)
}

DBPassword and APIKey are secret.String, so they redact themselves everywhere — log.Printf("%+v", cfg) prints [REDACTED], json.Marshal emits "[REDACTED]", and slog logs them as [REDACTED]. The plaintext only ever escapes through an explicit .Reveal(). mamori even ships a go vet analyzer that fails the build if you put an aws-sm:// value into a plain string instead of a secret.String.

Polling done right (jitter, batching, atomicity)

Secrets Manager has no push notification for value changes, so mamori polls — but it polls the way you’d want a fleet to poll:

• Jittered intervals. The poll interval is spread by ±20% by default, so 200 pods don’t all hit GetSecretValue on the same tick and blow through your API rate limit. You can tune the spread with WithJitter.
• Change detection is cheap. mamori compares the secret’s version (Secrets Manager’s VersionId) before doing anything, so an unchanged secret costs one lightweight check, not a full re-decode.
• Batching. The AWS provider resolves multiple refs from the same account in as few API calls as possible, automatically — so a struct with ten aws-sm:// fields doesn’t mean ten times the API traffic.
• Atomic + validated. When a rotation is detected, mamori re-validates the entire struct and only swaps it in if it’s valid. OnChange never fires with a half-updated or invalid config.

Compare this to the AWS Secrets Manager caching client: it caches and refreshes one secret at a time, with no struct composition, no validation, no diff-aware callback, and no config sources beyond Secrets Manager. mamori gives you the whole config surface — secrets and Parameter Store and env and files — in one typed struct with one reconciliation loop.

Tolerate a blip, page on a real outage

Secrets Manager occasionally throttles or has a transient error. You don’t want that to page anyone — a blip should just retry with backoff and keep serving the last good value. But if Secrets Manager is genuinely unreachable for ten minutes, that’s worth waking someone up. WithStale draws that line:

w, err := mamori.Watch[Config](ctx,
	mamori.WithStale(10*time.Minute),
	mamori.OnError(func(err error) {
		var stale *mamori.StaleError
		if errors.As(err, &stale) {
			pager.Page("AWS config stale >10m", stale)
			return
		}
		metrics.Inc("aws_transient_error") // blips: just a metric
	}),
)

Mixing sources in one struct

The real payoff shows up when your config isn’t all in one place. mamori doesn’t care that these live in four different systems — it’s one struct:

type Config struct {
	DBPassword secret.String `source:"aws-sm://prod/db#password"`     // Secrets Manager
	Workers    int           `source:"aws-ps:///myapp/workers"`       // Parameter Store
	LogLevel   string        `source:"env:LOG_LEVEL" default:"info"`  // environment
	TLSCert    []byte        `source:"file:///etc/tls/tls.crt"`       // local file
}

Wrapping up

Automatic rotation is only half a feature if your app has to restart to notice. mamori supplies the other half: it watches AWS Secrets Manager and Parameter Store and reconciles a rotation into your running Go service — typed, validated, redacted, no bounce.

go get github.com/xavidop/mamori/providers/aws

• Full introduction: Introducing mamori
• Docs: mamorigo.dev/docs
• Source: github.com/xavidop/mamori

mamori is MIT-licensed and at v0.1.0 — early days, so feedback and issues are genuinely welcome. Happy coding!

Introduction

Azure Key Vault is the well-behaved citizen of the Azure secrets world. Every secret is versioned, you get soft-delete and purge protection, access is governed by RBAC or access policies, and DefaultAzureCredential means your app authenticates the same way locally (az login) and in production (Managed Identity / Workload Identity on AKS) with zero code changes.

And yet the same old gap is there. Your Go service calls GetSecret once at startup, decodes the value into a struct, and treats it as immutable for the life of the process. When someone rotates the secret in Key Vault — a new version, the old one disabled — your service has no idea. It keeps using the value it read at boot until the next deploy or pod restart. The rotation was seamless in Azure and invisible to your app, which is exactly the wrong combination.

mamori is a Go library I built to fix this class of problem across every backend. Point it at a Key Vault secret and it watches for new versions, pulls the change, re-validates your whole config, and swaps it in atomically — while your service keeps running.

This is the Azure entry in the mamori launch series. For the full model, start with Introducing mamori: Typed, Watchable Config & Secrets for Go; here we focus on Key Vault.

The azure-kv scheme

The providers/azure module registers the azure-kv scheme. The grammar is azure-kv://<vault-name>/<secret-name>[#json-key][?version=<v>]:

azure-kv://my-vault/db-password              # latest enabled version
azure-kv://my-vault/api-key?version=abc123   # pin to a specific version id
azure-kv://my-vault/creds#password           # one key out of a JSON secret

One Key Vault quirk to keep in mind: versions are identified by opaque hex ids, not the sequential 1, 2, 3 you get in GCP Secret Manager. So ?version= here pins to an id like abc123def456..., not a number. Leave ?version off and mamori tracks whatever the latest enabled version resolves to — which is what you want for credentials that rotate.

Setup

go get github.com/xavidop/mamori
go get github.com/xavidop/mamori/providers/azure

Register it with a blank import. It authenticates through DefaultAzureCredential, so it picks up your az login locally, a Managed Identity on a VM/App Service, or Workload Identity on AKS — automatically, in that order:

import _ "github.com/xavidop/mamori/providers/azure"

The identity you bind needs read access to the vault’s secrets. With Key Vault’s RBAC model that’s the built-in Key Vault Secrets User role, scoped to the vault (or to individual secrets):

az role assignment create \
  --role "Key Vault Secrets User" \
  --assignee "<managed-identity-object-id>" \
  --scope "/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.KeyVault/vaults/my-vault"

If your vault still uses the classic access-policy model instead of RBAC, grant Get (and List) on secrets.

Load, then watch the rotation

package main

import (
	"context"
	"log"
	"time"

	"github.com/xavidop/mamori"
	"github.com/xavidop/mamori/secret"

	_ "github.com/xavidop/mamori/providers/azure"
)

type Config struct {
	// Tracks the latest enabled version: rotations flow in automatically.
	DBPassword secret.String `source:"azure-kv://my-vault/db-password"`

	// Pinned to an immutable version id: deliberate, reproducible.
	SigningKey secret.String `source:"azure-kv://my-vault/signing-key?version=abc123"`

	// One field out of a JSON secret.
	APIToken secret.String `source:"azure-kv://my-vault/creds#api_token"`
}

func main() {
	ctx := context.Background()

	w, err := mamori.Watch[Config](ctx,
		mamori.WithPollInterval(60*time.Second),
		mamori.OnChange(func(ev mamori.Change[Config]) {
			if ev.Changed("DBPassword") {
				pool.Rotate(ev.New.DBPassword.Reveal()) // live, no restart
				log.Println("Key Vault: db-password rotated and applied")
			}
		}),
		mamori.OnError(func(err error) { metrics.Inc("config_error") }),
	)
	if err != nil {
		log.Fatal(err)
	}
	defer w.Close()

	cfg := w.Get() // last VALID config, lock-free
	_ = cfg
}

Every secret field is a secret.String, so it redacts itself in fmt, slog, and JSON — a stray log.Printf("%+v", cfg) prints [REDACTED] — and only reveals plaintext through an explicit .Reveal(). mamori also ships a go vet analyzer that fails your build if an azure-kv:// value lands in a plain string instead of a secret.String.

What reconciliation guarantees

When a new version of db-password becomes the latest enabled one, mamori’s poll cycle detects the version change, fetches it, and then:

1. Re-validates the entire struct — a value that fails a validate rule is rejected atomically; Get() keeps returning the last good config and OnError receives a *ValidationError. A bad secret can’t break your service.
2. Swaps atomically — w.Get() moves from old to new in one lock-free store; no reader ever sees a half-updated config.
3. Fires a diff-aware callback — ev.Fields tells you exactly which fields moved, so you only rebuild what changed.

Key Vault has no push channel for “a new version was created”, so mamori polls — politely: the interval is jittered by ±20% (tunable with WithJitter) so a fleet of pods doesn’t hammer the vault on the same tick, and change detection compares the version id before pulling the payload, so an unchanged secret is cheap.

If Key Vault is briefly unreachable, mamori retries with backoff and keeps serving the last good value; wire WithStale(10*time.Minute) and check for a *mamori.StaleError in OnError if you want to page only on a prolonged outage rather than a transient blip.

Structured secrets with flatten

Storing a whole connection blob as one JSON secret? Decode it straight into a nested struct:

type Config struct {
	DB DBConfig `source:"azure-kv://my-vault/db-conn" flatten:"json"`
}

type DBConfig struct {
	Host     string        `mapstructure:"host"`
	Port     int           `mapstructure:"port"`
	Password secret.String `mapstructure:"password"`
}

One Key Vault call, one JSON document, cleanly typed — with the password still redacting itself.

When you don’t need this

If a secret only ever changes on deploy, a plain mamori.Load at boot is all you need — no watching, no poll loop. mamori earns its keep the moment a Key Vault secret rotates while your process is running and you’d rather rebuild a client than recycle a pod.

Wrapping up

Azure Key Vault rotates your secrets cleanly; mamori makes your Go service actually notice. It watches Key Vault, reconciles a new version into your running process — typed, validated, redacted — and lets you pin the versions you want frozen.

go get github.com/xavidop/mamori/providers/azure

• Full introduction: Introducing mamori
• Docs: mamorigo.dev/docs
• Source: github.com/xavidop/mamori

mamori is MIT-licensed and at v0.1.0 — early and moving fast, so issues and feedback are very welcome. Happy coding!

Introduction

Google Cloud Secret Manager has a clean, opinionated model: a secret is a container, and inside it live immutable, numbered versions. You never edit a version — you add a new one (1, 2, 3, …) and, when it’s ready, you move traffic to it, usually by resolving the latest alias (or by disabling the old version).

It’s a genuinely good design. But it collides with how most Go services consume secrets: you call AccessSecretVersion("latest") exactly once, at startup, and cache the result. From that moment your process is pinned to whatever latest meant at boot. Add version 4, disable version 3, and your service happily keeps using 3 until someone restarts it — which, at best, is a scheduled redeploy and, at worst, an incident when the disabled version stops resolving.

mamori is a Go library I built to make config and secrets watchable. Point it at a GCP secret and it notices when latest moves, pulls the new version, re-validates your whole config, and swaps it in atomically — while your service keeps running.

This is the GCP-focused companion to the launch article, Introducing mamori: Typed, Watchable Config & Secrets for Go. That one covers the full model; here we focus on Secret Manager’s version semantics.

The gcp-sm scheme

The providers/gcp module registers the gcp-sm scheme. The grammar is gcp-sm://<project>/<secret>[#json-key][?version=<v>]:

gcp-sm://my-project/db-password        # resolves the "latest" enabled version
gcp-sm://my-project/api-key?version=3  # pin to version 3, forever
gcp-sm://my-project/creds#password     # one key out of a JSON secret payload

That ?version= option is the important knob, and it maps directly onto GCP’s version model:

• No ?version → mamori tracks latest. When you add a new version and it becomes latest, mamori reconciles it in. This is what you want for rotating credentials.
• ?version=N → mamori pins to an immutable version N. It will never move. This is what you want for config you want to promote deliberately and reproducibly (think: “this deploy runs config version 7, full stop”).

Setup

go get github.com/xavidop/mamori
go get github.com/xavidop/mamori/providers/gcp

Register with a blank import; it uses Application Default Credentials, so on GKE it picks up Workload Identity automatically, and locally it uses your gcloud auth application-default login:

import _ "github.com/xavidop/mamori/providers/gcp"

The service account you bind needs roles/secretmanager.secretAccessor on the secrets it reads (and the ability to see version metadata to detect a new latest). Scope it to the specific secrets, not the whole project.

Load, then watch latest move

package main

import (
	"context"
	"log"
	"time"

	"github.com/xavidop/mamori"
	"github.com/xavidop/mamori/secret"

	_ "github.com/xavidop/mamori/providers/gcp"
)

type Config struct {
	// Tracks "latest": rotates in when you add a new enabled version.
	DBPassword secret.String `source:"gcp-sm://my-project/db-password"`

	// Pinned to an immutable version: deliberate, reproducible.
	StripeKey secret.String `source:"gcp-sm://my-project/stripe-key?version=7"`

	// One field out of a JSON secret.
	APIToken secret.String `source:"gcp-sm://my-project/creds#api_token"`
}

func main() {
	ctx := context.Background()

	w, err := mamori.Watch[Config](ctx,
		mamori.WithPollInterval(60*time.Second),
		mamori.OnChange(func(ev mamori.Change[Config]) {
			if ev.Changed("DBPassword") {
				pool.Rotate(ev.New.DBPassword.Reveal()) // live, no restart
				log.Println("db-password: new version reconciled")
			}
		}),
		mamori.OnError(func(err error) { metrics.Inc("config_error") }),
	)
	if err != nil {
		log.Fatal(err)
	}
	defer w.Close()

	cfg := w.Get() // last VALID config, lock-free
	_ = cfg
}

Every secret field here is a secret.String, which redacts itself in fmt, slog, and JSON — so nothing leaks into logs — and only reveals plaintext through an explicit .Reveal(). mamori’s go vet analyzer will even fail your build if you store a gcp-sm:// value in a plain string.

What reconciliation actually does here

When you rotate — add version 4 to db-password and let it become latest — mamori’s poll cycle notices that the resolved version changed, fetches it, and then:

1. Re-validates the entire struct. If the new secret somehow fails a validate rule, the update is rejected: Get() keeps returning the last good value and OnError gets a *ValidationError. A bad version can’t take your service down.
2. Swaps atomically. w.Get() transitions from old to new in one lock-free store — no reader ever sees a partially-updated config.
3. Fires a diff-aware callback. OnChange tells you which fields moved (ev.Fields) with old/new versions, so you only rebuild what actually changed.

Because Secret Manager has no push channel for “latest moved”, mamori polls — and it polls politely: the interval is jittered by ±20% so a large fleet doesn’t stampede the API on the same tick, and change detection compares version metadata before pulling the payload, so an unchanged secret is cheap.

Structured secrets with flatten

Teams often store a whole connection blob as one JSON secret. Decode it straight into a nested struct:

type Config struct {
	DB DBConfig `source:"gcp-sm://my-project/db-conn" flatten:"json"`
}

type DBConfig struct {
	Host     string        `mapstructure:"host"`
	Port     int           `mapstructure:"port"`
	Password secret.String `mapstructure:"password"`
}

One AccessSecretVersion call, one JSON document, cleanly typed — with the password still redacting itself.

Version pinning as a deploy strategy

The ?version= toggle enables a genuinely nice workflow. Keep operational secrets on latest so rotations flow in automatically, but pin config that changes behavior to explicit versions so a rollout is deterministic and rollback is trivial:

type Config struct {
	DBPassword   secret.String `source:"gcp-sm://my-project/db-password"`             // latest: auto-rotate
	FeatureFlags []byte        `source:"gcp-sm://my-project/flags?version=12"`         // pinned: promote on purpose
}

Bumping ?version=12 to ?version=13 becomes a reviewable, revertible change in your code — while credentials keep rotating themselves in the background.

Wrapping up

GCP Secret Manager’s immutable-versions model is great, but only if your app actually follows latest at runtime. mamori makes it do that: it watches the secret, reconciles a new version into your running Go process, validated and redacted, and lets you pin the fields you want frozen.

go get github.com/xavidop/mamori/providers/gcp

• Full introduction: Introducing mamori
• Docs: mamorigo.dev/docs
• Source: github.com/xavidop/mamori

mamori is MIT-licensed and at v0.1.0 — early and moving fast, so issues and feedback are very welcome. Happy coding!

Introduction

If you run Go services on Kubernetes, you know the ritual. You update a Secret or a ConfigMap, and then you do one of these:

• kubectl rollout restart deployment/my-app and eat the disruption.
• Install Reloader so it restarts your pods for you when a Secret changes.
• Mount the Secret as a volume and cross your fingers that your app re-reads the file (spoiler: most apps read it once at boot).

All three share the same flaw: the way you “pick up” a config change is to throw away the process and start a new one. For a rotating database password, that means dropping every in-flight request, cold connection pools, and a latency spike — all to change one string.

I built mamori partly because of this exact frustration. mamori watches your Kubernetes Secrets and ConfigMaps with native informers, and when a value changes it reconciles the new value into your running Go process — typed, validated, and atomically swapped — without restarting anything.

New to mamori? Start with the launch article, Introducing mamori: Typed, Watchable Config & Secrets for Go, for the full picture. This post is the Kubernetes-focused deep cut.

The two schemes you’ll use

mamori’s Kubernetes provider gives you two ref schemes:

Drop the #key and you get the whole Secret/ConfigMap as a JSON map, which you can decode into a nested struct with flatten. A few concrete examples:

k8s-secret://prod/db-creds#password   # one key from a Secret, base64-decoded
k8s-secret://prod/tls#ca.crt          # a certificate as []byte
k8s-cm://prod/app-config#log_level    # one key from a ConfigMap
k8s-cm://prod/app-config              # the whole ConfigMap as a JSON map

Setup

The provider lives in its own module. Note the import path is providers/k8s (not providers/kubernetes):

go get github.com/xavidop/mamori
go get github.com/xavidop/mamori/providers/k8s

Register it with a blank import — same convention as database/sql drivers. Its init() wires the k8s-secret and k8s-cm schemes into mamori:

import _ "github.com/xavidop/mamori/providers/k8s"

In-cluster, the provider uses the pod’s mounted ServiceAccount automatically. The only thing you owe it is RBAC: it needs to get, list, and watch the resources you reference in the namespaces you reference.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: mamori-reader
  namespace: prod
rules:
  - apiGroups: [""]
    resources: ["secrets", "configmaps"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: mamori-reader
  namespace: prod
subjects:
  - kind: ServiceAccount
    name: my-app
    namespace: prod
roleRef:
  kind: Role
  name: mamori-reader
  apiGroup: rbac.authorization.k8s.io

The watch verb is the important one — it’s what lets mamori use an informer instead of hammering the API server with polls.

Load once, then watch forever

Describe what you need as a struct. mamori resolves every ref, applies defaults, and validates — all before your main gets a config:

package main

import (
	"context"
	"log"

	"github.com/xavidop/mamori"
	"github.com/xavidop/mamori/secret"

	_ "github.com/xavidop/mamori/providers/k8s"
)

type Config struct {
	DBPassword secret.String `source:"k8s-secret://prod/db-creds#password"`
	LogLevel   string        `source:"k8s-cm://prod/app-config#log_level" default:"info" validate:"oneof=debug info warn error"`
	MaxConns   int           `source:"k8s-cm://prod/app-config#max_conns" default:"10" validate:"gte=1,lte=500"`
	CACert     []byte        `source:"k8s-secret://prod/tls#ca.crt"`
}

func main() {
	cfg, err := mamori.Load[Config](context.Background())
	if err != nil {
		log.Fatal(err)
	}
	_ = cfg // fully resolved, defaulted, validated
}

Notice DBPassword is a secret.String, not a plain string. That type redacts itself in logs, fmt, and JSON — so a stray log.Printf("%+v", cfg) prints [REDACTED] instead of leaking the password. You only ever see the plaintext when you explicitly call .Reveal().

Now the part that matters. Swap Load for Watch, and the moment someone runs kubectl edit secret db-creds, your process reacts:

w, err := mamori.Watch[Config](ctx,
	mamori.OnChange(func(ev mamori.Change[Config]) {
		if ev.Changed("DBPassword") {
			// Rebuild the pool with the rotated password — live, no restart.
			pool.Rotate(ev.New.DBPassword.Reveal())
		}
		for _, f := range ev.Fields {
			log.Printf("reconciled %s: %s -> %s", f.Path, f.OldVersion, f.NewVersion)
		}
	}),
	mamori.OnError(func(err error) { metrics.Inc("config_error") }),
)
if err != nil {
	log.Fatal(err)
}
defer w.Close()

cfg := w.Get() // lock-free snapshot; always the last VALID config

w.Get() is lock-free and always returns the last valid configuration. If someone pushes a ConfigMap with log_level: banana, the validate:"oneof=..." tag rejects it: the update is dropped atomically, Get() keeps returning the good value, and OnError receives a *ValidationError. A bad edit never takes your service into a broken state.

Informers, not polling

This is the difference between mamori and the mount-a-file-and-poll approach. The Kubernetes provider implements native watching backed by the client-go informer machinery, so change detection is push-based — the API server tells your process the instant a Secret changes, rather than your process asking every N seconds.

Practically, that means:

• Near-instant reconciliation. No polling interval to tune, no min(sync, interval) latency floor like the kubelet’s ~60s volume refresh.
• No API-server hammering. One watch connection per resource, shared, instead of a GET loop per pod.
• Graceful fallback. If a native watch fails to start for some reason, mamori automatically falls back to polling with jittered intervals, so you never lose reconciliation entirely.

Compare that to the alternatives:

Whole ConfigMaps and JSON blobs

Sometimes you keep a structured blob in one key. Decode it straight into a nested struct with flatten:

type Config struct {
	Redis RedisConfig `source:"k8s-secret://prod/redis-conn" flatten:"json"`
}

type RedisConfig struct {
	Addr     string        `mapstructure:"addr"`
	Password secret.String `mapstructure:"password"`
	DB       int           `mapstructure:"db"`
}

That reads the whole redis-conn Secret (a JSON document), decodes it, and — because Password is a secret.String — keeps the password redacted throughout.

A note on TLS certificates

Certificates rotate too, and a cert reload is the perfect no-restart use case. Because file and Secret updates for certs usually want to apply immediately (no coalescing delay), you can set ?debounce=0 on the ref:

type TLS struct {
	Cert []byte       `source:"k8s-secret://prod/tls#tls.crt?debounce=0"`
	Key  secret.Bytes `source:"k8s-secret://prod/tls#tls.key?debounce=0"`
}

Wire the OnChange callback to swap the *tls.Certificate your server hands out in its GetCertificate hook, and you’ve got zero-downtime cert rotation from a Kubernetes Secret.

When you don’t need this

Be honest with yourself: if your config is a log level from a ConfigMap that only ever changes on deploy, you don’t need runtime watching — a plain Load at boot is fine, and mamori is happy to do just that. And if you already mount a Secret as a file and genuinely re-read it, mamori’s built-in file:// provider (with fsnotify) covers that too, without any Kubernetes RBAC at all.

mamori earns its keep the moment a value changes while your process is running and you’d rather rotate a pool than recycle a pod.

Wrapping up

Kubernetes already tells you the instant a Secret changes — via informers. mamori just connects that signal to a typed, validated, redacted struct inside your Go process, so a rotation becomes a callback instead of a rollout.

go get github.com/xavidop/mamori/providers/k8s

• Full introduction: Introducing mamori
• Docs: mamorigo.dev/docs
• Source: github.com/xavidop/mamori

mamori is MIT-licensed and currently at v0.1.0, so issues and feedback are very welcome. Happy shipping!

Introduction

Every Go service I have ever shipped ends up growing the same organ. It starts small: a couple of os.Getenv calls. Then a secret needs to come from AWS Secrets Manager, so you add the SDK. Then someone wants that secret to rotate without a deploy, so you add a time.Ticker. Then the ticker needs a mutex because two goroutines read the config while it is being replaced. Then a bad value slips in and takes down production, so you add validation. Then a secret shows up in a log line during an incident, and now you are adding a redaction wrapper too.

By the end you have a bespoke ConfigManager in every repo: a loader, a fetcher, a refresh loop, a mutex, and a prayer. It is never quite the same twice, it is rarely tested well, and it is exactly the kind of code nobody wants to own.

I got tired of writing it, so I built a library that does it once, properly. It is called mamori, it is open source, and this article is the full tour: what it is, why it exists, how to use it end to end, and when you should (and shouldn’t) reach for it.

The restart-to-rotate problem

Here is the assumption baked into most Go config code: configuration is static. You read it at boot, decode it into a struct, and treat that struct as immutable for the lifetime of the process. If something changes, you redeploy.

That assumption was fine when “config” meant a log level and a port. It is not fine anymore. In a modern service, a meaningful amount of your config is dynamic by nature:

• Database passwords and API tokens rotate on a schedule (and increasingly, automatically).
• Vault hands you dynamic credentials with a lease that expires in an hour.
• Feature flags flip in the middle of the day, on purpose.
• TLS certificates get reissued.

The industry answer to “config is dynamic” has mostly been to push the problem into the platform. On Kubernetes, the External Secrets Operator syncs your cloud secret into a Secret, and then… you still restart the pod to pick it up, or you mount it as a file and hope your app re-reads it. The dynamic part gets flattened back into a static one at the worst possible layer.

mamori takes the opposite position: your process should be able to observe a change at the source and react to it, in memory, without bouncing. Rotating a password should rotate a connection pool, not recycle a pod.

What is mamori?

mamori (守り, Japanese for “protection” or “safeguard”) is an embedded Go library — not an operator, not a server, not a sidecar. It lives inside your process. Its tagline says it plainly:

Typed, watchable config & secrets for Go. Load configuration and secrets from anywhere into validated Go structs — and keep them reconciled at runtime, without a restart.

There are three concepts you actually touch, and they map cleanly onto three types:

1. Refs. Struct fields carry a source tag containing a URL-like reference to where the value lives — env:LOG_LEVEL, aws-sm://prod/db#password, vault://secret/app#token. A ref names what you want, not how to fetch it.

2. Providers. A provider resolves one scheme (aws-sm, vault, k8s-secret, …) into a value. Providers register themselves using the same pattern as database/sql drivers, so the core module carries zero cloud-SDK dependencies — you only pull in the providers you use.

3. The Reconciler. This is the interesting part. Watch resolves every ref once (fail-fast at startup), then keeps watching each source. When a value changes, it re-validates the entire struct, and only if the new snapshot is valid does it atomically swap it in and hand you a diff-aware callback. Your app never sees a half-updated, invalid config.

If you have used IOptionsMonitor<T> in .NET, w.Get() is the direct analogue: always the latest valid configuration, lock-free.

How mamori compares

I want to be honest about prior art, because none of these primitives are new — mamori’s contribution is composing them.

• gocloud.dev/runtimevar is the closest primitive. It watches a single variable beautifully, but it is one variable at a time, with no struct composition, no tags, no validation, no secret hygiene. mamori adds all of that on top of the same idea.
• Viper / koanf are excellent, but they are config-first with secrets bolted on. mamori is deliberately secrets-first, with config included: secret types redact by default, and there is a go vet analyzer that yells at you if you store a secret in a plain string.
• envconfig / caarlos0/env are load-once. No watching, env only.
• The External Secrets Operator is complementary, not competitive. mamori is for apps that want to skip the Kubernetes Secret hop entirely, or that do not run on Kubernetes at all. It keeps no persistent external state, so there is no finalizer lifecycle to babysit.

And to be equally clear about what mamori is not: it is not a secrets store (no encryption at rest, no server), not a sync engine between stores, not a general feature-flag platform, and not cross-language. It is a Go-idiomatic library and it stays in its lane.

Getting started

The core module has built-in env:, file://, and dotenv: providers, so you can do something useful with zero extra dependencies:

go get github.com/xavidop/mamori

Heads up: mamori requires Go 1.26 or newer. It leans on modern generics ergonomics, so this is a hard floor, not a suggestion.

The whole API is driven by a struct and its tags. You describe what you want; mamori figures out how:

package main

import (
	"context"
	"log"

	"github.com/xavidop/mamori"
	"github.com/xavidop/mamori/secret"
)

type Config struct {
	DBPassword secret.String `source:"aws-sm://prod/db#password"`
	LogLevel   string        `source:"env:LOG_LEVEL" default:"info" validate:"oneof=debug info warn error"`
	Workers    int           `source:"env:WORKERS" default:"4" validate:"gte=1,lte=256"`
	TLSCert    []byte        `source:"file:///etc/tls/tls.crt"`
}

func main() {
	cfg, err := mamori.Load[Config](context.Background())
	if err != nil {
		log.Fatal(err)
	}
	// cfg is fully resolved, defaulted, and validated.
}

The signature is exactly what you’d hope for:

func Load[T any](ctx context.Context, opts ...Option) (T, error)

Load resolves every ref once, applies defaults, validates, and returns your typed config. It fails fast: on any resolve or validation error it returns the zero value and a non-nil error. It never returns a partially-populated struct — you either get a complete, valid config or an error.

A few things worth knowing about the tags:

That default-only-on-not-found rule is deliberate and important: if AWS Secrets Manager is having a bad day, you want an error, not a silent fallback to a default password.

Watching and reconciliation

This is why mamori exists. Swap Load for Watch and your config becomes a living thing:

func Watch[T any](ctx context.Context, opts ...Option) (*Watcher[T], error)

You read the current config with a lock-free w.Get(), and you react to changes with an OnChange callback:

w, err := mamori.Watch[Config](ctx,
	mamori.WithPollInterval(30*time.Second),
	mamori.OnChange(func(ev mamori.Change[Config]) {
		if ev.Changed("DBPassword") {
			pool.Rotate(ev.New.DBPassword.Reveal())
		}
		for _, f := range ev.Fields {
			log.Printf("%s changed: %s -> %s", f.Path, f.OldVersion, f.NewVersion)
		}
	}),
	mamori.OnError(func(err error) { metrics.Inc("config_error") }),
)
if err != nil {
	log.Fatal(err)
}
defer w.Close()

cfg := w.Get() // always the last VALID config, no lock needed

The Change[T] you get in the callback carries the old value, the new value, and a per-field diff, plus a Changed(path) helper so you only do expensive work (like rebuilding a connection pool) when the field you care about actually moved.

The guarantees are the whole point, so let me spell them out:

• Atomicity. OnChange only ever fires with a fully re-validated snapshot. If an incoming value fails validation, the update is rejected: Get() keeps returning the last good config and OnError receives a *ValidationError. Your config never transitions into a broken state mid-flight.
• Push where possible, poll where not. Providers whose backend can push changes (Kubernetes informers, Vault lease watchers, Consul blocking queries, fsnotify for files) push natively. Everything else is polled on WithPollInterval, jittered by ±20% by default so a fleet of pods doesn’t stampede your secrets backend on the same tick. If a native watch fails to start, mamori quietly falls back to polling.
• Coalescing. Rapid-fire changes within a debounce window (500ms default, overridable per-field with ?debounce=) collapse into a single event.
• Lease-aware. If a provider reports an expiry (Vault dynamic creds), mamori schedules a refresh before the lease dies, rather than waiting for the next poll.
• First event. Watch resolves the initial config before it returns, so OnChange fires only on subsequent changes. Read the initial state from w.Get().

There’s a nice operational escape hatch too. Transient backend outages shouldn’t page anyone — but prolonged staleness should. WithStale draws that line for you:

w, err := mamori.Watch[Config](ctx,
	mamori.WithStale(10*time.Minute),
	mamori.OnError(func(err error) {
		var stale *mamori.StaleError
		if errors.As(err, &stale) {
			alert.Page("config stale", stale)
			return
		}
		metrics.Inc("config_transient_error")
	}),
)

A blip retries with exponential backoff and keeps the last good value. Ten minutes of failure escalates to a hard *StaleError you can actually page on.

Secrets that behave like secrets

Configuration and secrets are treated differently, on purpose. Any field that holds a secret should be a secret.String or secret.Bytes instead of a plain string or []byte. These types redact themselves everywhere a value normally leaks:

s := secret.NewString("hunter2")

fmt.Println(s)              // [REDACTED]
fmt.Sprintf("%v", s)        // [REDACTED]
fmt.Sprintf("%#v", s)       // [REDACTED]
json.Marshal(s)             // "[REDACTED]"
slog.Info("login", "pw", s) // pw=[REDACTED]

s.Reveal()                  // "hunter2"  <- the only way to read the plaintext

It works because the type implements fmt.Stringer, fmt.GoStringer, json.Marshaler, and slog.LogValuer. The single, greppable access point is Reveal(). That is a feature: in code review or an audit, grep Reveal shows you every place a secret is actually unwrapped. (secret.Bytes.Reveal() returns []byte rather than a string, for keys and certificates.)

And because “just remember to use secret.String” is exactly the kind of discipline that erodes under deadline pressure, mamori ships a go vet analyzer that enforces it:

go install github.com/xavidop/mamori/tools/reconcilevet/cmd/reconcilevet@latest
go vet -vettool=$(which reconcilevet) ./...

reconcilevet flags any field whose source points at a secret-bearing scheme (aws-sm, gcp-sm, azure-kv, vault, op, sops, k8s-secret) but is stored in a plain string or []byte. The leak becomes a build failure, not a postmortem.

Providers: more than 30 sources

The core stays dependency-free; each cloud backend is its own module you opt into. A ref’s scheme picks the provider:

go get github.com/xavidop/mamori/providers/aws         # aws-sm://  aws-ps://
go get github.com/xavidop/mamori/providers/vault       # vault://
go get github.com/xavidop/mamori/providers/gcp         # gcp-sm://
go get github.com/xavidop/mamori/providers/azure       # azure-kv://
go get github.com/xavidop/mamori/providers/k8s         # k8s-secret://  k8s-cm://
go get github.com/xavidop/mamori/providers/consul      # consul://
go get github.com/xavidop/mamori/providers/doppler     # doppler://
go get github.com/xavidop/mamori/providers/onepassword # op://
go get github.com/xavidop/mamori/providers/sops        # sops://

That is a small slice. All in, there are more than 30 providers: secret managers (AWS Secrets Manager, AWS Parameter Store, Vault, GCP, Azure Key Vault, Doppler, 1Password, SOPS), databases (Postgres, MySQL, SQLite, MongoDB, DynamoDB, Cosmos DB, Redis), KV stores (Consul, etcd), Kubernetes Secrets and ConfigMaps, Firebase/Firestore/Remote Config, object storage (S3, GCS, Azure Blob), and a whole shelf of feature-flag platforms (LaunchDarkly, Unleash, Flagsmith, ConfigCat, Split, GrowthBook, Flipt, GO Feature Flag).

Registration follows the database/sql convention — a blank import runs the provider’s init(), which calls mamori.Register(...):

import _ "github.com/xavidop/mamori/providers/vault"

When you need explicit configuration (a region, a custom client), construct the provider and pass it with WithProvider. That takes precedence over the registry for that call:

import awsprov "github.com/xavidop/mamori/providers/aws"

cfg, err := mamori.Load[Config](ctx,
	mamori.WithProvider(awsprov.NewSecretsManager(awsprov.WithRegion("eu-west-1"))),
)

The ref grammar is compact — scheme://path[#key][?opts] — with a couple of quirks worth internalizing:

• The #key fragment selects one field out of a JSON payload: aws-sm://prod/db#password.
• AWS Parameter Store, etcd, sops, and file preserve absolute paths, so they use a triple slash: aws-ps:///myapp/log-level, file:///etc/tls/tls.crt.
• Doppler is the one provider where the # fragment is required: doppler://backend/prd#STRIPE_API_KEY.

When a payload is a structured blob, you can decode it straight into a nested struct with flatten:

type Config struct {
	Redis RedisConfig `source:"aws-sm://prod/redis" flatten:"json"`
}

type RedisConfig struct {
	Addr     string        `mapstructure:"addr"`
	Password secret.String `mapstructure:"password"`
	DB       int           `mapstructure:"db"`
}

One AWS Secrets Manager call, one JSON blob, cleanly typed — with the password still redacting itself.

Writing your own provider

If your source of truth is homegrown — an internal config service, a weird legacy database table — you can teach mamori about it. The interface is intentionally tiny:

type Provider interface {
	Scheme() string
	Resolve(ctx context.Context, ref Ref) (Value, error)
}

A minimal implementation looks like this:

func (p *Provider) Resolve(ctx context.Context, ref mamori.Ref) (mamori.Value, error) {
	raw, err := p.fetch(ctx, ref.Path)
	if isNotFound(err) {
		return mamori.Value{}, mamori.ErrNotFound // MUST satisfy errors.Is
	}
	if err != nil {
		return mamori.Value{}, err
	}
	if ref.Key != "" {
		raw, err = mamori.SelectKey(raw, ref.Key) // implements #key against a JSON payload
		if err != nil {
			return mamori.Value{}, err
		}
	}
	return mamori.Value{
		Bytes:     raw,
		Version:   mamori.VersionHash(raw), // change detection; use a native revision if you have one
		Sensitive: true,                    // true for secret-bearing sources
	}, nil
}

func init() { mamori.Register(New()) } // database/sql pattern; panics on a duplicate scheme

The Version field is what powers cheap change detection: if your backend exposes a revision (a VersionId, a Vault version, an mtime hash), return it; otherwise mamori.VersionHash(bytes) gives you a stable FNV hash for free. Two optional interfaces let you do better when the backend supports it:

type WatchableProvider interface {
	Provider
	Watch(ctx context.Context, ref Ref) (<-chan Update, error)
}

type BatchProvider interface {
	Provider
	ResolveBatch(ctx context.Context, refs []Ref) (map[string]Value, error)
}

Implement Watch only if your backend can genuinely push — otherwise let mamori’s poller do the work, so behavior stays consistent across providers. Implement ResolveBatch if you can fetch many refs in one round trip (the AWS provider does this automatically).

Best of all, you don’t have to trust yourself to get the contract right. There’s a conformance kit — github.com/xavidop/mamori/providertest — that exercises resolution, not-found typing, version monotonicity, concurrency, context cancellation, native watching, goroutine hygiene, and even asserts you never accidentally log the payload. Point it at your provider and it tells you if you’ve broken the rules.

Middleware: cache, failover, audit, rate-limit

Because every provider is just a Provider, you can wrap one in another. The middleware package ships the decorators you’d otherwise reinvent:

They compose by nesting, so you can read the stack top-down:

import "github.com/xavidop/mamori/middleware"

mamori.WithProvider(
	middleware.Cache(5*time.Minute,
		middleware.Audit(logger,
			middleware.RateLimit(10,
				middleware.Failover(primarySM, replicaSM),
			),
		),
	),
)

Crucially, the shipped middleware preserves the WatchableProvider and BatchProvider capabilities of whatever they wrap — decorating a provider never silently downgrades it from push to poll.

Observability out of the box

The core takes no OpenTelemetry dependency (keeping it lean), but there’s a bridge module when you want metrics and traces:

go get github.com/xavidop/mamori/x/otel

import (
	"go.opentelemetry.io/otel"
	mamoriotel "github.com/xavidop/mamori/x/otel"
	"github.com/xavidop/mamori"
)

meter, _ := mamoriotel.NewMeter(otel.Meter("mamori"))
tracer := mamoriotel.NewTracer(otel.Tracer("mamori"))

w, err := mamori.Watch[Config](ctx,
	mamori.WithMeter(meter),
	mamori.WithTracer(tracer),
)

You get a mamori.resolve.duration histogram, mamori.refresh.count and mamori.watch.errors counters (all tagged by scheme), and a span per resolve. And because WithMeter/WithTracer take tiny internal interfaces, you can implement them against Prometheus, statsd, or a test recorder without pulling OTel into your binary at all.

A complete, runnable example

Here is the whole loop — load, watch, react — in one file you can run today (it’s the examples/basic program from the repo):

package main

import (
	"context"
	"fmt"
	"log/slog"
	"os"
	"time"

	"github.com/xavidop/mamori"
	"github.com/xavidop/mamori/secret"
)

const tokenPath = "/tmp/mamori-example-token"

type Config struct {
	LogLevel string        `source:"env:LOG_LEVEL" default:"info" validate:"oneof=debug info warn error"`
	Workers  int           `source:"env:WORKERS" default:"4" validate:"gte=1,lte=256"`
	APIToken secret.String `source:"file:///tmp/mamori-example-token"`
}

func main() {
	logger := slog.New(slog.NewTextHandler(os.Stdout, nil))
	_ = os.WriteFile(tokenPath, []byte("initial-token"), 0o600)
	defer func() { _ = os.Remove(tokenPath) }()

	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	w, err := mamori.Watch[Config](ctx,
		mamori.WithPollInterval(2*time.Second),
		mamori.OnChange(func(ev mamori.Change[Config]) {
			for _, f := range ev.Fields {
				logger.Info("config changed", "field", f.Path, "from", f.OldVersion, "to", f.NewVersion)
			}
			if ev.Changed("APIToken") {
				logger.Info("rotating clients with the new API token", "token", ev.New.APIToken)
			}
		}),
		mamori.OnError(func(err error) { logger.Warn("reconcile error", "err", err) }),
	)
	if err != nil {
		logger.Error("watch failed", "err", err)
		os.Exit(1)
	}
	defer func() { _ = w.Close() }()

	cfg := w.Get()
	logger.Info("loaded config", "logLevel", cfg.LogLevel, "workers", cfg.Workers, "apiToken", cfg.APIToken)

	// Simulate a rotation a few seconds in.
	go func() {
		time.Sleep(3 * time.Second)
		_ = os.WriteFile(tokenPath, []byte("rotated-token"), 0o600)
	}()

	time.Sleep(8 * time.Second)
	logger.Info("final config", "apiToken", w.Get().APIToken)
}

LOG_LEVEL=debug WORKERS=8 go run ./examples/basic

Notice the token field in the log output: it prints [REDACTED] every time, even though the program is actively rotating it. That’s the whole philosophy in one demo — live rotation, no restart, and nothing leaks.

When to reach for mamori (and when not)

Reach for it when:

• Your config or secrets genuinely change while the process runs, and you want to react in memory instead of bouncing pods.
• You pull from more than one source and are tired of a different SDK and refresh strategy for each.
• Secret hygiene matters and you want redaction and a go vet guardrail for free.
• You run outside Kubernetes, or you’re on Kubernetes but want to skip the Secret-sync hop.

Probably skip it when:

• Your config is a log level and a port from the environment, read once at boot. os.Getenv is fine — don’t add a dependency.
• You need a secrets store (encryption at rest, a server component) — that’s Vault’s job, not mamori’s.
• You’re syncing secrets between stores — that’s what ESO, vals, and teller are for.

Wrapping up

mamori is the library I wish I’d had every time I typed type ConfigManager struct for the fifth time. It takes the boring, error-prone, security-sensitive plumbing — load, validate, watch, redact, reconcile — and makes it a typed struct and one function call.

It’s open source under the MIT license and currently at v0.1.0 — early, pre-1.0, and moving fast, so feedback and issues are genuinely welcome. Give it a spin:

go get github.com/xavidop/mamori

• Docs: mamorigo.dev/docs
• Homepage: mamorigo.dev
• Source: github.com/xavidop/mamori

Provider deep-dives

If you live on a particular platform, I’ve written focused, hands-on companions:

• Hot-reloading Kubernetes Secrets and ConfigMaps in Go — no pod restarts
• Rotating AWS Secrets Manager secrets without restarting your Go service
• GCP Secret Manager, live-reloaded in Go
• Azure Key Vault secrets that reconcile themselves in Go

If you build something with it, or write a provider for a backend I haven’t covered yet, I’d love to hear about it. Happy coding!

Introduction

Anyone who has shipped a non-trivial Gen AI feature to production has hit the same wall: the happy path is fun to demo, but the unhappy path is brutal. Model providers throw 5xx errors. Rate limits kick in at the worst possible moment. A long-running agent gets halfway through a 12-step plan and the pod is restarted by Kubernetes. A user closes the tab in the middle of a streaming response and you have no idea what state the flow ended up in.

Genkit gives you a clean way to author those LLM-orchestrating pipelines as flows, but a Genkit flow, by itself, is just a function. It lives and dies with the process that invokes it. There is no durable history, no automatic retry policy, no built-in cancellation, no operations UI.

This is exactly the gap that Temporal was designed to close for general-purpose backend code, and it is the gap that the new genkitx-temporal plugin closes for Genkit flows.

In this article we will look at:

• What the plugin actually does.
• Why Temporal and Genkit are a particularly good match.
• How to use it end to end: define, register, run.
• When you should reach for it (and when you probably shouldn’t).

What is genkitx-temporal?

genkitx-temporal is a Genkit plugin that lets you execute any Genkit flow inside a Temporal Workflow, transparently. You keep writing flows the way you always have; the plugin takes care of:

• Registering the flow so a Temporal Worker can pick it up.
• Wrapping each execution in a deterministic Temporal Workflow.
• Running the non-deterministic LLM/tool/RAG work inside a Temporal Activity, so retries and timeouts are safe.
• Giving you helpers to start workflows from any client.

Under the hood, the plugin ships a generic, deterministic workflow called runGenkitFlow that invokes a single Temporal Activity called runGenkitFlowActivity. The activity looks up your Genkit flow by name in an in-process registry and runs it inside a full Node environment, where all the messy, non-deterministic things (network calls, model calls, tool calls, RAG lookups) are perfectly fine.

┌────────────┐  start workflow   ┌──────────────────────┐
│  Client    │ ────────────────▶ │  Temporal Server     │
└────────────┘                   └─────────┬────────────┘
                                           │ task
                                           ▼
                                ┌──────────────────────┐
                                │  Worker process      │
                                │  ┌────────────────┐  │
                                │  │ runGenkitFlow  │  │  (workflow, sandboxed)
                                │  │       │        │  │
                                │  │       ▼        │  │
                                │  │ runGenkit-     │  │  (activity, full Node)
                                │  │ FlowActivity   │  │
                                │  │       │        │  │
                                │  │       ▼        │  │
                                │  │  your Genkit   │  │
                                │  │  flow (LLM…)   │  │
                                │  └────────────────┘  │
                                └──────────────────────┘

This split is the whole trick. Temporal Workflows are required to be deterministic so that they can be replayed from event history after a crash, deploy or scale event. LLM calls obviously are not deterministic. Putting the LLM work inside an Activity is the canonical Temporal pattern, and the plugin does it for you so you don’t have to think about it.

Why Temporal and Genkit are a great match

It is easy to underestimate how much engineering is hiding behind “just call the model again if it fails”. Once you start building real agents, the list of things you want from your runtime grows quickly. Genkit gives you the authoring primitives, and Temporal gives you the runtime guarantees. Together:

• Automatic retries for transient errors. LLM providers regularly return 429s and 5xx. Tool calls hit the network. Vector stores time out. Temporal lets you express retry policies (exponential backoff, max attempts, non-retryable error types) declaratively, applied to every flow execution.
• Durable history. Every event in your flow’s execution is persisted by the Temporal Server. If your Worker pod is killed mid-flow, another Worker picks up the workflow exactly where it left off, with all prior activity results intact. No partial state, no double-charging the user, no orphan jobs.
• Timeouts, heartbeats and cancellation. Long-running agents that browse, plan and call tools for minutes are a nightmare to control with plain HTTP. Temporal models start-to-close, schedule-to-close and heartbeat timeouts as first-class concepts, plus explicit cancellation semantics you can wire to a user closing a tab.
• Operational visibility out of the box. The Temporal UI gives you a per-execution timeline of every workflow and activity. You can see inputs, outputs, retries, failures and stack traces, search by workflow id, terminate or signal running workflows. This complements the Genkit Developer UI nicely: Genkit’s UI is your local debugging tool during development; Temporal’s UI is your operational dashboard in staging and production.
• Horizontal scalability for free. Need more throughput? Run more Worker processes pointing at the same task queue. The Temporal Server load-balances workflows and activities across them. Your flows did not need to change.
• Long-running, human-in-the-loop friendly. Temporal workflows can sleep for days, wait for signals from external systems, and resume cleanly. Perfect for agents that need approval before executing a sensitive tool, or RAG pipelines that wait for a document indexing job to finish.

In other words, genkitx-temporal turns your Genkit flows from “smart functions inside a Node process” into first-class durable workloads without changing the way you write them.

Installation

npm install genkitx-temporal genkit

The Temporal SDK packages are peer-installed automatically as dependencies of the plugin.

You also need a running Temporal Server. For local development, the easiest path is:

brew install temporal
temporal server start-dev

This starts a Temporal Server on localhost:7233 and the UI on http://localhost:8233.

Usage end to end

The plugin exposes a small, focused API. Three things are enough to ship a durable Genkit flow: define it with the Temporal-aware helper, start a Worker, and execute it from a client.

1. Define a flow with defineTemporalFlow

defineTemporalFlow is a drop-in replacement for ai.defineFlow. The returned object is a normal Genkit flow, so you can still call it directly, expose it via the Developer UI, run it from tests, etc. The only difference is that it is also registered internally so a Temporal Worker can find it by name.

// flows.ts
import { genkit, z } from 'genkit';
import { googleAI } from '@genkit-ai/google-genai';
import { defineTemporalFlow, temporal } from 'genkitx-temporal';

export const ai = genkit({
  plugins: [
    googleAI(),
    temporal({ taskQueue: 'my-queue' }),
  ],
  model: googleAI.model('gemini-flash-latest'),
});

export const jokeFlow = defineTemporalFlow(
  ai,
  {
    name: 'jokeFlow',
    inputSchema: z.string(),
    outputSchema: z.string(),
  },
  async (subject) => {
    const { text } = await ai.generate(`Tell me a joke about ${subject}`);
    return text;
  },
);

A few things to notice:

• You configure the plugin like any other Genkit plugin, passing the Temporal task queue (and optionally address, namespace, etc.).
• The flow body is plain Genkit code. No Temporal-specific imports, no proxyActivities, no determinism gymnastics. The plugin handles all that for you.
• The flow’s name is also its Temporal registration key. Keep it unique within your Worker.

2. Start a Worker

Workers are the processes that actually execute your flows. A Worker imports your flows (so the registry is populated) and then calls startTemporalWorker:

// worker.ts
import './flows';   // side-effect import: registers the flows
import { startTemporalWorker } from 'genkitx-temporal';

startTemporalWorker({ taskQueue: 'my-queue' })
  .catch((e) => { console.error(e); process.exit(1); });

node ./dist/worker.js

You can run as many Worker processes as you want against the same task queue. Temporal will distribute work across them automatically. If a Worker dies mid-flow, another one picks up.

3. Execute a flow as a Temporal Workflow

From any client (an HTTP handler, a CLI, a cron job, another workflow), use executeTemporalFlow to start a workflow and await its result:

// client.ts
import { executeTemporalFlow } from 'genkitx-temporal';
import { jokeFlow } from './flows';

const result = await executeTemporalFlow(jokeFlow, 'cats', {
  taskQueue: 'my-queue',
});
console.log(result);

If you don’t want to block on the result, use startTemporalFlow instead. It returns the raw Temporal WorkflowHandle, which lets you query, signal, or cancel the running workflow later. This is the building block for human-in-the-loop scenarios, scheduled flows, fan-out/fan-in patterns, and so on.

Configuration

temporal(options) and every helper accept the same connection options. Anything you don’t pass falls back to environment variables, then to sensible defaults:

This makes it straightforward to run the same code locally against a dev server and in production against Temporal Cloud or a self-hosted cluster.

Advanced: combine with your own workflows and activities

The bundled runGenkitFlow workflow is enough for the common case. If you want to mix Genkit flows with your own existing Temporal workflows and activities, the plugin lets you bring your own:

await startTemporalWorker({
  taskQueue: 'my-queue',
  workflowsPath: require.resolve('./my-workflows'),
  activities: { ...require('./my-activities') },
});

// my-activities.ts
export { runGenkitFlowActivity } from 'genkitx-temporal/activities';
export async function myOtherActivity(/* ... */) { /* ... */ }

Re-exporting runGenkitFlowActivity keeps the built-in workflow working, so you can compose Genkit flows alongside hand-written activities for the parts of your system that aren’t AI.

API summary

The full public surface is tiny:

• temporal(options?) — the Genkit plugin.
• defineTemporalFlow(ai, config, fn) — define a flow and register it for Temporal execution.
• startTemporalWorker(options?) — start a Worker process.
• executeTemporalFlow(flow, input, options?) — run a flow inside a Workflow and await the result.
• startTemporalFlow(flow, input, options?) — same, but returns the WorkflowHandle for fire-and-forget / signalling.
• runGenkitFlowActivity — the underlying activity, re-exported so you can combine it with your own activities.
• registerTemporalFlow(name, flow) — manually register a flow that was defined elsewhere (useful when wrapping flows you don’t own).

A small API surface is the point. The plugin is intentionally a thin bridge between two well-designed systems; it does not try to reinvent either of them.

When to reach for genkitx-temporal

Not every flow needs a durable runtime. A streaming chat response that takes 800ms and either succeeds or is retried by the user is fine running on a plain HTTP handler.

You will feel the benefits the moment your flows look like one of these:

• Multi-step agents that orchestrate several model calls and tool invocations, where partial progress is expensive to throw away.
• Long-running pipelines (document ingestion, batch summarization, fine-tuning prep) where individual steps can take minutes and the process must survive deploys.
• Critical business workflows (refunds, account changes, contract generation) where you cannot afford to lose state or accidentally execute a step twice.
• Human-in-the-loop agents that need to pause for approval, an external webhook, or a manual review before proceeding.
• Anything you currently glue together with a job queue, a retry library, and a state machine. Temporal subsumes all three, and genkitx-temporal plugs Genkit straight into it.

If your team already runs Temporal for non-AI workloads, this plugin is a no-brainer: it lets your Gen AI features inherit all the operational maturity your platform team has already built around it.

How it pairs with the rest of the Genkit ecosystem

The thing I like the most about this plugin is that it composes cleanly with everything else Genkit gives you, not just one or two features:

• Genkit Developer UI. Because defineTemporalFlow returns a normal Genkit flow, you can still iterate on it locally with the Genkit Developer UI: fast feedback loop during development, durable execution in production.
• Genkit middleware. Middleware (in-call retries, fallbacks, skill injection, tool approval, prompt rewriting, etc.) runs inside the activity. You get two complementary layers of resilience: middleware for fine-grained in-call recovery, Temporal for whole-flow durability and replay.
• Tools and function calling. Tools defined with ai.defineTool are just normal flow code from Temporal’s perspective. Their calls, retries and outputs appear in both the Genkit trace and the Temporal event history.
• RAG primitives. Retrievers, indexers, embedders and rerankers all live inside the activity. That means heavy ingestion jobs (chunking, embedding, upserting into a vector store) inherit Temporal’s retry policies and survive restarts mid-batch.
• Evaluators and datasets. Genkit’s evaluators are flows like any other, so you can run eval jobs as Temporal Workflows, schedule them, fan them out across Workers, and inspect every run in the Temporal UI.
• Prompts and Dotprompt. Versioned .prompt files, prompt registries and structured outputs all work unchanged. The flow body is plain Genkit.
• Plugins and model providers. Any Genkit plugin (Google AI, Vertex AI, OpenAI, Anthropic, Ollama, local models, vector stores, etc.) plugs in as usual; the Temporal layer doesn’t care which provider is on the other side of the call.
• Telemetry and tracing. Genkit’s OpenTelemetry traces continue to be emitted from inside the activity, so they show up in whatever observability backend you already use, alongside Temporal’s own event history.
• Deployment surfaces. Flows can still be exposed as HTTP endpoints, Cloud Functions, Firebase callable functions or Express handlers; executeTemporalFlow is just one more entry point, and you can mix and match (e.g. quick chat requests over HTTP, long agent runs through Temporal).
• Multi-language story. Genkit is available in JS/TS, Go, Python (preview), Dart/Flutter (preview) and through a community Java SDK. This particular plugin targets JS/TS, but the architectural pattern — define your AI logic in Genkit, run it as a Temporal Workflow — is reusable across runtimes thanks to Temporal’s polyglot SDKs.

Conclusion

Genkit is a great way to author Gen AI logic. Temporal is a great way to run any long-running, failure-prone workload. genkitx-temporal is the missing adapter between the two: a small, focused plugin that turns every Genkit flow into a durable, retryable, observable Temporal Workflow without asking you to rewrite a single line of business logic.

If you are building anything more ambitious than a single-turn chat endpoint, give it a try:

• Source: github.com/xavidop/genkitx-temporal
• Docs: xavidop.github.io/genkitx-temporal
• Runnable example: examples/test-app

Your future on-call self will thank you the first time a model provider has a bad afternoon and your agents keep humming along regardless.

Introduction

If you have ever shipped a backend service without a debugger, a hot-reload dev server or decent logs, you remember how painful it was. Now imagine doing it in a world where your “function” is a non-deterministic black box that can rewrite its own output every time you call it. Welcome to building Gen AI applications.

The single biggest productivity multiplier I have found in the last two years of shipping AI apps is a good local debugging tool. Not a hosted dashboard. Not a cloud trace viewer with a 30-second propagation delay. A local tool that runs next to your code, picks up your edits, lets you replay a request with a tweaked prompt, and shows the full trace, tool call by tool call. We will get into the local-vs-hosted trade-off later in the article.

This article is about why that matters and how the three leading JS/TS Gen AI frameworks approach it:

• Genkit Developer UI
• Vercel AI SDK DevTools
• Mastra Studio

They are three very different products solving overlapping problems, and each one makes different trade-offs that are worth understanding before you commit to one.

Shift-left, applied to Gen AI

“Shift-left” comes from the world of testing and security: the earlier in the development lifecycle you catch a problem, the cheaper it is to fix. Bugs found in production cost orders of magnitude more than bugs found while you’re typing.

Gen AI apps make this principle existential, not just economical, because the failure modes are weirder:

• A prompt regression doesn’t show up as a stack trace. It shows up as the wrong tone, a hallucinated fact or a tool call that misfires once every twenty runs.
• A new model version can quietly change behavior across thousands of code paths with no warning.
• A tool that returns slightly different JSON can cause silent downstream breakage.

The only way to keep your sanity is to collapse the feedback loop to seconds. You want to be able to:

1. Change a prompt or a piece of orchestration code.
2. Re-run that exact unit, with that exact input.
3. See the model’s input, its output, every tool call, every retry, every token used.
4. Compare it to the previous run.
5. Decide if it’s better, worse or different.

If any one of those steps requires deploying, redeploying, opening a cloud console or grep’ing logs, you have already lost. Cycle time is the metric. A debugging tool is what makes that cycle time low enough to iterate productively.

Local vs hosted

It is worth being explicit about this trade-off, because all three tools in this article are primarily local:

• A local debugging tool runs on your machine, next to your code, with zero network latency between your edits and what you see. Iteration is fast, traces are immediate, and there is no risk of leaking prompts/responses to a third party. The downside is that it’s just for you.
• A hosted observability platform (Langfuse, LangSmith, cloud-native APM tools, etc.) is shared by your team, persists data long-term and is essential for production monitoring. The trade-off is propagation delay, configuration overhead and, in some cases, data residency concerns.

These are complementary, not competing. The argument here is that the local part of the loop is the one that’s still often missing, and that’s where shift-left lives.

What “good” looks like

After working with all three of the tools below, I would summarize the qualities of a good Gen AI debugging tool as:

• Zero-config or near-zero-config — it should pick up your code, not the other way around.
• Live reload — edit your code, save, hit “run” again. No restart.
• Interactive runners — invoke any prompt, tool, model, or higher-level primitive (agent, workflow, etc.) directly with arbitrary input.
• Full traces — every step of the generation loop, with input/output/usage at each node.
• Replay and modify — re-run any past trace with a tweak.
• Minimal code changes — the framework should be observable by default, ideally without you having to remember to wrap each model call.

Let’s see how each tool stacks up.

Genkit Developer UI

The Genkit team treats the Developer UI as a first-class part of the framework. It ships with the genkit-cli package and requires no code changes in your application to attach to your running process.

You install the CLI once globally:

npm install -g genkit-cli

And then start your app under it:

genkit start -- npx tsx --watch src/index.ts

That’s it. Genkit attaches to your running Node process, discovers every flow, prompt, model, tool, retriever, indexer, embedder and evaluator you have defined, and exposes them all in a local web app at http://localhost:4000.

Telemetry API running on http://localhost:4033
Genkit Developer UI: http://localhost:4000

What you actually get inside:

• Interactive runners for every primitive. Flows, prompts, tools, models, retrievers, indexers, embedders and evaluators all get an interactive panel where you fill out the input (validated against the Zod schema) and hit run.
• Full traces with the entire generation graph: every model call, every tool invocation, every middleware, with input/output/usage tokens.
• Live reload. Combined with --watch, edits to your code show up without restarting the UI.
• Prompt iteration. Tweak prompts and re-run inline.
• Evals. Run evaluators against datasets directly from the UI.
• Open by default. Add -o to auto-open in your browser.

What stands out here is that observability comes for free. You do not wrap your model and you do not add a middleware just to see traces. Anything defined with the Genkit primitives is automatically introspected and traced, which makes the “let me see what’s happening” cost essentially zero.

If you also use the middleware system, every middleware also shows up in the trace, which makes debugging things like retries and fallbacks a lot easier.

Vercel AI SDK DevTools

Vercel introduced the AI SDK DevTools more recently, and it is a meaningful step up from having nothing. The design philosophy is different from Genkit’s: it is a middleware-based capture tool that focuses tightly on the language-model layer, rather than an integrated development environment for the whole AI app.

You opt in by wrapping your model:

import { wrapLanguageModel, gateway } from 'ai';
import { devToolsMiddleware } from '@ai-sdk/devtools';

const model = wrapLanguageModel({
  model: gateway('anthropic/claude-sonnet-4.5'),
  middleware: devToolsMiddleware(),
});

And then run the viewer in another terminal:

npx @ai-sdk/devtools
# open http://localhost:4983

What it captures:

• Input parameters and prompts.
• Output content and tool calls.
• Token usage and timing.
• Raw provider request/response payloads.
• Multi-step interactions are grouped into “runs” with multiple “steps”.

Things to keep in mind when comparing it with the others:

• It is opt-in per model. Every model you want to observe has to be wrapped explicitly. In a real codebase with many models, this usually means a shared factory or repeated wrapping. If you forget on one path, you have a blind spot.
• It focuses on the language-model layer. Anything happening above that layer (your own application logic, custom orchestration, business steps) is not captured unless you add it yourself.
• It is a viewer, not an interactive playground. You cannot “re-run this with a different prompt” from the UI; you go back to your code, edit and re-call.
• It does not enumerate AI primitives because, in the AI SDK, those are just functions in your codebase. There is nothing to enumerate at runtime.
• Storage is a JSON file (.devtools/generations.json). The team is clear that this is local-only, never for production, and the middleware automatically appends .devtools to your .gitignore.

The upside is that it is probably the quickest of the three to drop into an existing project: install the package, wrap a model, run the viewer. If you live inside Next.js and the AI SDK and you mainly want a “what just happened?” panel for your model calls, it does the job well. If you are debugging a multi-step agent with custom orchestration, you will probably want to combine it with something that sees more of the picture.

Mastra Studio

Mastra Studio is the most ambitious of the three in terms of surface area. It is bundled with mastra dev and runs at http://localhost:4111, doubling as a local development tool and a deployable team console (you can ship Studio to production for non-developers).

Out of the box, it gives you:

• Agents tab — chat with your agents, hot-swap models, tweak temperature/top-p, view traces, attach scorers.
• Workflows tab — visualize workflows as graphs, run them step by step with custom inputs, watch the active step in real time, inspect tool calls and JSON outputs.
• Processors tab — see input/output processors and guardrails wired to each agent.
• Tools tab — run tools in isolation to debug them.
• Workspaces tab — file browser into the agent’s workspace filesystem, with a Skills tab listing discovered skills.
• MCP servers tab — list attached MCP servers and their tools.
• Request context — set runtime variables that flow into agent instructions/tools through DI, with schema-driven forms.
• Evaluation suite — Scorers, Datasets and Experiments tabs to run datasets through agents/workflows, attach scorers and compare experiments side-by-side.
• Observability with traces and logs.
• Editor integration for non-technical teammates to iterate on agents and version every change without redeploying.

This is more than a debugging tool — it is closer to a development environment plus a team-facing console. The natural trade-off is that Studio is tightly coupled to Mastra’s primitives (Agents, Workflows, Processors, Workspaces), so it shines the most when your application is structured the Mastra way.

If you build non-trivial agentic systems and you want evaluation and dataset management baked into the same tool, Studio is genuinely very good. For a quick “see my prompt and the model’s response” loop, it can feel like more than you need.

Side-by-side comparison

A quick note on the Vercel column: simplicity of integration is a real feature for many teams, and it is fair to call this the lightest-touch option. The trade-off is that observability is opt-in per model, so you have to be deliberate to avoid blind spots.

When I would pick which

After using all three, this is roughly how I’d think about it:

• Building a serious agent or multi-step pipeline with tools, retries, fallbacks and evals → Genkit Developer UI is hard to beat on pure iteration speed. Zero-config tracing of the full pipeline, every primitive runnable, and middleware shows up in the trace for free.
• Working primarily inside Next.js with the Vercel AI SDK and you mainly want better visibility into model calls → Vercel AI SDK DevTools is a quick win. Lightweight setup and it does what it says on the tin; pair it with something else if you need higher-level orchestration views.
• Building agentic systems with workflows, datasets and scorers, and you want a UI your PM or domain expert can also use → Mastra Studio. The evaluation suite and the deployability are real differentiators.
• Multi-language stack (JS/TS plus Go, Python or Dart) → Genkit is the only one of the three with first-party multi-language support; the local tooling story is consistent across runtimes.

The moral of the story: pick a debugging tool, any debugging tool

In a normal backend project, you can sometimes get away with a weak dev loop because the code is deterministic. Bad UX, but it works.

In an AI project, the code is not deterministic. Every iteration is also a small experiment. If your loop is slow or your visibility is poor, you don’t just iterate slower, you iterate worse, because you can’t tell whether your changes are improvements or regressions. Shift-left in this world is not a virtue, it is a survival strategy.

A good local debugging tool:

• Turns “I think the prompt got better” into “I can see the trace, the tokens, the eval score, side by side”.
• Catches regressions before they reach a user.
• Lets you onboard new engineers in hours instead of days, because they can see the system.
• Lets non-engineers (PMs, designers, domain experts) participate in iteration when the UI is friendly enough — Mastra Studio is explicit about this.

If you are starting a Gen AI project today and you are picking a framework, the local tooling story should weigh as much as the model abstraction or the tool API. It is the difference between writing AI code and engineering AI features.

Conclusion

All three tools are good in their lane:

• Genkit Developer UI is the most introspective of the three and requires zero code changes to be useful. It’s the one I reach for when I want to move fast and see every step of the pipeline.
• Vercel AI SDK DevTools is the lightest-touch option to drop into an existing AI SDK app, with a clear focus on the model-call layer. Think of it as a focused viewer rather than a full development environment.
• Mastra Studio is the most ambitious, closer to a full IDE-meets-console for agentic systems, with first-class evaluation, datasets and a deployable team UI. It’s at its best when your app is structured around Mastra’s primitives.

Pick the one that matches the shape of your project, but please pick something. Building Gen AI without a local debugging tool in 2026 is engineering with the lights off.

Further reading:

• Genkit Developer Tools
• Vercel AI SDK DevTools
• Mastra Studio overview
• Top JS/TS Gen AI Frameworks for 2026
• Genkit Middleware deep dive

Introduction

If you have been building anything non-trivial with Genkit, you have probably bumped into the same set of cross-cutting concerns over and over again: retrying transient model errors, falling back to a cheaper model when quota explodes, gating tool execution behind human approval, injecting filesystem access for coding agents, logging every request and response for observability…

Until now, you ended up either wrapping ai.generate() calls by hand or writing ad-hoc helpers that ended up duplicated across flows. The new Genkit Middleware changes that. It introduces a first-class, composable middleware layer for the generate() pipeline, with hooks for the model, the tool execution and the high-level generation loop, plus a small but very useful set of official middlewares published in the brand new @genkit-ai/middleware package.

This article is a practical tour of what the new middleware system gives you, the built-in middlewares you can drop in today, and how to write your own with generateMiddleware.

The official documentation lives at Genkit Middleware. All examples below assume the JavaScript/TypeScript SDK.

A quick reminder: although this article focuses on the JS/TS middleware API, Genkit is a multi-language framework. The official SDKs cover JavaScript/TypeScript (primary, stable), Go, Python (preview) and Dart/Flutter (preview), and there is a community-maintained Java SDK used in production. The middleware concepts described here are JS/TS-specific today, but the underlying generate() pipeline exists across all SDKs and the same patterns are landing on the other runtimes.

What is middleware in Genkit

Conceptually, Genkit middleware behaves like the middleware you already know from Express or Koa, only applied to the LLM lifecycle instead of HTTP requests:

1. A generate() call is intercepted before it reaches the model.
2. Each middleware can inspect or modify the request, decide whether to call next(), and inspect or modify the response on the way back.
3. Multiple middlewares can be chained. They run in the order they are declared and unwind in reverse order, exactly like an onion.

What makes Genkit’s design interesting is that it does not give you a single chokepoint, it gives you three orthogonal interception phases:

• model — wraps the call to the underlying model. Perfect for retries, fallbacks, request/response logging or response transformations.
• tool — wraps tool execution. Ideal for approvals, sandboxing, audit logs or input/output validation.
• generate — wraps the whole high-level generation loop (prompting, tool calling, output parsing). Best for things like injecting tools or system instructions before the loop starts.

You opt in per call via a use: array, which keeps things explicit and avoids global side effects:

const response = await ai.generate({
  model: googleAI.model('gemini-flash-latest'),
  prompt: 'Hello',
  use: [retry({ maxRetries: 3 }), loggerMiddleware({ verbose: true })],
});

Installation

The official middlewares ship in their own package, decoupled from the Genkit core:

npm install @genkit-ai/middleware
# or
pnpm add @genkit-ai/middleware

You still need genkit itself and a model provider plugin (for example @genkit-ai/google-genai).

The built-in middleware catalogue

Let’s go through the five middlewares the Genkit team ships out of the box.

filesystem — give the model a sandboxed file system

filesystem injects a standard set of file manipulation tools (list_files, read_file, write_file, search_and_replace) into the generation loop, restricted to a root directory of your choice.

import { genkit } from 'genkit';
import { googleAI } from '@genkit-ai/google-genai';
import { filesystem } from '@genkit-ai/middleware';

const ai = genkit({ plugins: [googleAI()] });

const response = await ai.generate({
  model: googleAI.model('gemini-flash-latest'),
  prompt: 'Create a hello world Node app in the workspace',
  use: [
    filesystem({
      rootDirectory: './workspace',
      allowWriteAccess: true,
    }),
  ],
});

Useful options:

• rootDirectory (required) — sandbox root, all paths are confined to it.
• allowWriteAccess — defaults to false. Read-only by default is a sane choice for safety.
• toolNamePrefix — namespace the injected tools to avoid collisions with your own.

This is essentially the building block for a “coding agent” pattern, without you having to write tool definitions or path validation logic.

skills — auto-load Markdown skills as system context

skills scans a directory for SKILL.md files (plus their YAML frontmatter), injects relevant ones into the system prompt, and exposes a use_skill tool the model can call when it needs more specific guidance.

import { skills } from '@genkit-ai/middleware';

const response = await ai.generate({
  prompt: 'How do I run tests in this repo?',
  use: [skills({ skillPaths: ['./skills'] })],
});

Think of it as a lightweight, file-based knowledge layer: every skill is a self-contained Markdown file with metadata, and the middleware decides when to surface them. It is a really clean alternative to ad-hoc system prompt soup.

toolApproval — human-in-the-loop for tool calls

toolApproval enforces an allowlist of tools the model is allowed to execute autonomously. Anything outside the list raises a ToolInterruptError, so you can pause execution, ask the user, and resume.

import { genkit, restartTool } from 'genkit';
import { toolApproval } from '@genkit-ai/middleware';

const response = await ai.generate({
  prompt: 'write a file',
  tools: [writeFileTool],
  use: [toolApproval({ approved: [] })], // empty list -> always interrupt
});

if (response.finishReason === 'interrupted') {
  const interrupt = response.interrupts[0];

  // ... ask the user, then mark the tool call as approved
  const approvedPart = restartTool(interrupt, { toolApproved: true });

  const resumedResponse = await ai.generate({
    messages: response.messages,
    resume: { restart: [approvedPart] },
    use: [toolApproval({ approved: [] })],
  });
}

This is exactly the pattern you want for any agent that touches the real world (filesystem writes, payments, sending emails). No more home-grown approval flags scattered across the codebase.

retry — exponential backoff with jitter for transient errors

The retry middleware retries failed model calls on transient status codes (UNAVAILABLE, DEADLINE_EXCEEDED, RESOURCE_EXHAUSTED, ABORTED, INTERNAL) using exponential backoff with jitter.

import { retry } from '@genkit-ai/middleware';

const response = await ai.generate({
  model: googleAI.model('gemini-pro-latest'),
  prompt: 'Heavy reasoning task...',
  use: [
    retry({
      maxRetries: 3,
      initialDelayMs: 1000,
      backoffFactor: 2,
    }),
  ],
});

Knobs you actually care about:

• maxRetries (default 3)
• statuses — which status codes to retry on
• initialDelayMs / maxDelayMs / backoffFactor
• noJitter — if you really want deterministic delays

This is one of those things every team writes once, badly. Having it in the framework is a very welcome change.

fallback — gracefully degrade to a different model

fallback switches to an alternate model when the primary one fails on configurable status codes. The classic use case is “try Pro first, fall back to Flash when quota is exhausted”:

import { fallback } from '@genkit-ai/middleware';

const response = await ai.generate({
  model: googleAI.model('gemini-pro-latest'),
  prompt: 'Try the pro model first...',
  use: [
    fallback({
      models: [googleAI.model('gemini-flash-latest')],
      statuses: ['RESOURCE_EXHAUSTED'],
    }),
  ],
});

You can chain multiple fallback models, and isolateConfig lets you decide whether the fallback inherits the original request configuration or starts clean (handy when the fallback model does not support the same options as the primary).

Building your own middleware with generateMiddleware

The same primitive that powers all the built-ins is exposed for you. The generateMiddleware helper gives you typed config schemas (via Zod) and access to the ai instance.

Here is the canonical “logger” example, straight from the docs but lightly annotated:

import { generateMiddleware, z } from 'genkit';

export const loggerMiddleware = generateMiddleware(
  {
    name: 'loggerMiddleware',
    description: 'Logs requests and responses',
    configSchema: z.object({
      verbose: z.boolean().optional(),
    }),
  },
  ({ config, ai }) => {
    return {
      // Phase 1: intercept the model call
      model: async (req, ctx, next) => {
        if (config?.verbose) {
          console.log('Request:', JSON.stringify(req));
        }
        const resp = await next(req, ctx);
        if (config?.verbose) {
          console.log('Response:', JSON.stringify(resp));
        }
        return resp;
      },
      // You could also add `tool: ...` and `generate: ...` hooks here.
    };
  }
);

Using it is identical to the official ones:

const response = await ai.generate({
  model: googleAI.model('gemini-flash-latest'),
  prompt: 'Hello',
  use: [loggerMiddleware({ verbose: true })],
});

A few patterns I have found very useful:

• PII redaction — implement a model hook that scrubs the request prompt and the response text against a regex/dictionary, returning the cleaned version.
• Cost accounting — wrap the model hook to read usage tokens from the response, and emit them to your metrics backend tagged by user/feature.
• Per-tenant quotas — use the generate hook to check a counter (Redis, Firestore…) before calling next(); throw your own custom error if the tenant is over quota.
• Caching — keyed on a hash of the model + request, return a cached response if hit, otherwise call next() and persist the result.

For more inspiration, the source of the official middlewares is open in the Genkit GitHub repository, and reading them is genuinely educational.

Composition: stacking middlewares

Middlewares compose in array order. A reasonable production stack might look like this:

const response = await ai.generate({
  model: googleAI.model('gemini-pro-latest'),
  prompt: userPrompt,
  tools: myTools,
  use: [
    loggerMiddleware({ verbose: false }),       // outermost: see everything
    retry({ maxRetries: 3 }),                   // recover from transient failures
    fallback({                                  // degrade if Pro is overloaded
      models: [googleAI.model('gemini-flash-latest')],
      statuses: ['RESOURCE_EXHAUSTED'],
    }),
    toolApproval({ approved: ['searchDocs'] }), // gate dangerous tools
  ],
});