Etsy is home to over 100 million listings from 5.6 million active sellers. Because the items for sale are unique and creative, there is no standard product catalog that tells us what they are. When someone searches for “light linen dress for summer,” our models must infer what the shopper means and what the listings contain. They do that using data from user visits and Etsy listings.

We use Kafka for streaming data. These data streams provide fresh features and embeddings that power our machine learning models. If a shopper favorites a hand-painted ceramic mug, that action can shape their recommendations within minutes.

But building these streaming applications hasn’t always been a smooth process. A new app requires templated code, configs, tests, validation, deployment steps, and careful review. That developer experience has been a strain on engineers across the company.

The Olden Days

Before coding agents were broadly available at Etsy, engineers turned machine learning designs into Kafka applications by hand. A streaming feature might require filtering events from terabyte-scale topics, writing transformation logic, and passing outputs through filtered topics to make them available in the feature store.

The challenge wasn't writing Java or YAML. It was choosing the right schema, identifying the correct base class, validating input, anticipating edge cases, and running local tests. While static templates could generate files, they couldn't streamline the iterative, time-consuming process.

Old process, human developer does everything Kafka Skills to the Rescue

Our team provides the infrastructure and platform support for Etsy engineers to create and manage new features and embeddings with Kafka, among other streaming data products. We decided to make this easier with Agent Skills, reusable playbooks that teach language models how to execute specific tasks. For each workflow, the agent uses the Skills and tools we built to generate code, validate configs, run apps locally, and create PRs.

Collaborative process between Human developer and Kafka agent

The Kafka Skills we created are conversational workflows that allow developers and LLMs to collaborate. They can be invoked with natural language or a slash command such as /new-streaming-feature.

Conversation with Claude using Kafka Skill to create a new ML feature

For each streaming feature pipeline, the agent creates a Java class that implements a shared interface. The YAML configuration files are fill-in-the-blanks. The agent uses scaffolding scripts, templated code generation, and other tools that we created to output deterministic code.

We saw an opportunity to have the Kafka Skills orchestrate all the steps to run an application locally. The human developer no longer needs to create test topics, build JARs, fill in manual commands and read from test topics. The agent will do all that, report back on findings, and generate all the relevant output links.

Pull request and verification links posted for the developer’s review Observability

We wanted to know which Kafka Skills are being used, how often they make it to pull requests, which environments and application types are involved, and where developers may still need support. To do this, we added lightweight telemetry around the workflow: when a Skill is invoked, a hook fires and a Python helper emits a started event. When the agent opens a pull request, a wrapper emits a completion event. Both events flow through an existing Etsy service that publishes a protobuf message to Kafka and stores it in a BigQuery table.

Recording developer use of Kafka Skills in BigQuery Build It and They Won't Come

We tested and demoed Kafka Skills and were very pleased with our “Agentic Improvements”. When we looked at adoption rates though, we were surprised to find that developers weren’t using them. They weren’t using the Skills to create features, generate embeddings, or even turn Kafka topic records into HTTP requests. Feeling a bit defeated, we asked devs: why? They responded that they simply didn’t know about them. The devs were using coding agents, but the agents never offered the Kafka Skills.

Before the Skills existed, the standard workflow for feature creation was to find the closest existing feature, copy the code for it, and modify the fields. Instead of starting the process by opening the LLM and saying "I'd like to create a new streaming feature," an engineer might engage with the agent halfway through the old process and ask to "copy this class and change it to count cart moves," or "edit this file" with a path lifted from a previous pull request.

Another challenge we found was that requests often arrive in whatever vocabulary the developer's team uses. When creating a new feature with Kafka, the dev might refer to a specific feature-store entry, an internal event name, a parent Java class name needed, or an output data type.

To address these issues, we enumerated entry points for different vocabulary, lifecycle stages, and copy-paste-shaped requests. Plus, we added repository-level routing so that touching related files is itself a trigger for the Kafka Skill. The developer describes the task however they naturally would, and the agent offers the paved path.

Guardrails

In developing the Kafka Skills, we hoped to prevent agents from inventing Kafka applications from scratch. The agent handles the ambiguous parts: understanding the developer's request, identifying source schemas and fields, and deciding which validation steps to run. The agent then passes structured inputs into versioned scaffolding scripts and validation tools. Anthropic describes this pattern in its tool-use documentation: the agent returns a structured tool call that application-owned code executes, and strict tool use can constrain those inputs to a JSON schema. The agent handles the orchestration, while our scripts and hooks generate the templated code.

Security research reinforced that decision. In a 2023 Stanford study, developers using an assistant wrote less secure code and were more likely to think their code was secure. BaxBench found a similar gap between code that runs and code that’s safe: across 392 backend tasks and 11 models, the best model was functionally correct 62% of the time, but no model produced code that was both correct and secure on more than 37% of the tasks. Researchers could exploit about half of the functionally correct programs.

Now that we had agents running commands and committing code on our behalf, we also found that we needed to tighten Kafka ACLs, repository permissions, and deployment controls. We designed the system around least privilege: agents only get the access they need for the task they are performing. This was to make sure that neither human nor agent could delete an important topic, write to an unexpected one, change an unrelated pipeline, or bypass the normal production review path.

What’s Next?

Kafka supports a wide range of critical data workflows at Etsy, including machine learning feature generation, embeddings, filtering, fan-out, and storing data. We’ve built Skills for seven Kafka workflows so far. Adoption is the near-term focus: we're continuing to add entry points so the Skills are offered no matter how a developer phrases the task. Among our earliest insights from the observability table will be whether the discoverability fixes described above worked, and we’ll continue to iterate as we learn.

We built the Kafka Skills to fix a developer-experience problem: too many steps stood between an idea for an ML model and a working pipeline with all the data. Our machine learning models shape the Etsy user experience for buyers and sellers. Each streaming feature that ships sooner helps our models understand a little earlier what shoppers mean and what listings contain. That means more fresh signals behind every search for a light linen dress, and behind the next hand-painted ceramic mug someone favorites.

Acknowledgements

A huge thanks to Kamya Shethia, Shannon Hoang, Nicollette Lui, Andrey Polyakov, and Virginia Poundstone for all their hard work designing and building Kafka Skills and agent tools, setting up observability, as well as locking down access control to make the world safer for robots and humans alike.