My Understanding of Current Thinking

As part of my development of seeds, I have begun to look for conversations surrounding how people are thinking about capturing the decisions and decision making process as they plan systems with the intent of having GenAI (Generative AI, e.g. agents) implement those systems. In the wake of Storey’s From Technical Debt to Cognitive and Intent Debt (the “triple debt” paper), people seem to be rallying around the term “intent debt” or rather the “absence of externalized rationale that developers and AI agents need to work safely with code.” Storey actually names three debts — technical, cognitive, and intent — and I’m setting cognitive debt aside here: that one is the internalized, shared understanding a team holds about a system, a different axis from the externalized deliberation seeds is built to capture. I like to think of intent as the “why” in “why did we choose this approach for the system”. A predominant attitude, captured well in Osmani’s The Intent Debt is “An agent can’t generate intent, because intent is the one input that has to come from you.” and that agents “will invent a confident-sounding reason, which is worse than admitting it doesn’t know.” The consensus seems to be that capturing intent and providing it to GenAI will help improve the decisions it makes while implementing a system. I agree with this notion. What seems to be lacking is consensus on how best to capture and represent intent, and that’s exactly the space I inadvertently started exploring when I began working on seeds.

When I introduced seeds, I described it as “deliberation software” but in a recent investigation into that terminology, I discovered that term is currently being applied to software that facilitates debate and decision making between sets of multiple AIs. That’s definitely not something seeds does, so at the moment, I see seeds as adjacent to “intent software” and will be exploring a more appropriate and permanent label for it. For now I will continue with using the term “deliberation” to represent “the messy, often non-linear, spaghetti-like process of exploring ideas about a system before deciding upon a particular approach to a problem.”

seeds is Upstream of Intent

To me, the conversation around intent is where/how to capture the sanitized, concise set of reasons behind the decisions that have gone into a system with the intent of helping keep GenAI on track. Indeed, the conversations I’ve seen about intent capture propose things like ADRs, BDD, AGENTS.md files, etc. All of these artifacts make the assumption that a particular approach has been decided and should now be recorded and enforced.

But for me, one of the most interesting and lossy parts of developing a system is discovering the approach to begin with, and that was the part I wanted seeds to capture.

I’d argue that intent isn’t even a complete picture. For instance, with seeds, my intention was to capture deliberation between a solo-dev and their agents. It’s a good intention. Now, how the heck was I going to do that?

I had to choose a language for implementation, a means of storing the ideas and deliberation, a way to structure that information, an interface for the agent to use, etc. Using Storey’s triple-debt framing, the closest approximation of what I’m describing is intent, but I can’t exactly see where all that deliberation is supposed to reside. I’m not going to pack an AGENTS.md file with something like this:

Well, I considered PostgreSQL, but that would require a user to set up a whole RDBMS system just to warehouse a couple hundred rows of data, so let’s use SQLite for now. Also, I thought about using Ruby because it’s my primary language, but ultimately decided to go with Python because LLMs seem most familiar with that language and this will be completely developed by GenAI. Also I think that a CLI over MCP is best because MCPs always take up context but a CLI can be interrogated on demand.

Some of that is intent, and some of that is vital historical information that has no place being loaded into my agents’ context each time it fires up. If we focus solely on “intent” we lose a lot of information I consider valuable. At the same time, if we capture all that information in a way that requires an agent to frequently review it and incorporate it into its context, we’re wasting tokens. Intention, the way people are talking about it and seem to want to use it as a means to inform GenAI’s behavior, should be streamlined, concise, and firm. But the process that gets us to that kind of intention is anything but.

I also find that intent is dynamic and emergent — something you often discover as you build, and revise as you go. Ball’s Building Software Is Learning does a great job explaining that building software is, in part, exploration and discovery — that you often can’t fully specify what you want until you’ve started building it. Particularly with seeds, I really had no idea if such a tool was feasible. My theory was that if beads could help an AI and human collaborate on implementation, perhaps I could build a tool that helped an AI and human collaborate on design and deliberation. I wanted to test this theory, so my original intention for seeds was to build an MVP of a tool that would then help me design the rest of that tool. My hope was that by using the tool to design more of the tool, I’d learn what was working, what was failing, and what was missing.

What I’ve Learned

Surveying the discourse surrounding intent debt has deepened my understanding of the problem space and given me some insights into where seeds fits:

The most surprising thing I learned — the intent was already reaching the agent. Intent can be (and, it turns out, is) a product of seeds, but it is not the sole focus of seeds. Across multiple posts, I saw authors postulate that by giving GenAI information about the intention behind what it was being asked to implement, GenAI would make better choices. As I mentioned in my previous post, I design a feature in seeds, then ask seeds to turn that design into a series of beads. I have never looked at the generated beads myself, because almost always GenAI was able to one-shot the feature I’d worked out in seeds. Why fix it if it ain’t broke.

But I became curious whether the intent in my seeds was actually reaching the agent, so I had an agent comb through how GenAI had been interacting with the beads. I figured that if the intent reached the implementer at all, it would be because the agent followed each bead’s back-references to the seeds it came from and read them. Or maybe I was doing such a thorough job speccing out every detail in seeds that GenAI had no need to make decisions at implementation time and never bothered with those back-links at all.

What the agent found surprised me: seeds had been quietly including a “## Why” section in each bead’s description, putting the intent behind the implementation directly into the bead. The GenAI had access to it all along — as plain tokens it reads by default — and that may very well be why it has been one-shotting with such regularity for me. I never would have thought to check before I learned about intent debt.

seeds is definitely a product of my environment - As I stated in the previous post, I am primarily a solo-developer. My boss is essentially my client. He and I communicate in either brief Slack messages or long, completely transcribed Zoom discussions. Any planning I do is in a conversation with an AI agent. All my deliberation about my software projects is therefore completely captured. This is essentially the world Haber describes in Everything Is Recorded Now — where you may as well assume everything you say at work is being recorded — except I’ve leaned into it on purpose. The common worry about that world is the “candor paradox”: once everyone assumes the meeting is recorded, the real, half-formed thinking tends to flee to the hallway and the DM, where no recorder is running. My setup doesn’t have those hallways; the candid conversation is the recorded one. That makes ingesting deliberation into seeds very, very low friction. If anything, I find myself calling out to AI in my conversations and Zoom discussions to “note this” when I want to flag something important for later ingestion into seeds.

I prefer live capture of intention over other approaches - Meta Engineering’s How Meta Used AI to Map Tribal Knowledge recently described inferring tribal knowledge — the conventions and constraints buried in code — from existing artifacts. Notably, that recovers what the code implies, not why the humans chose it. As I mentioned in the previous post, I have added seeds to a decade-old project, but I haven’t bothered to go back through and have seeds attempt to infer any intent. Capture from live sources feels like such a richer source of high-fidelity information and my plan for now is to simply capture deliberation on that project from here on out. Perhaps someday I will explore backfilling intent from other artifacts in the project, but it’s not interesting to me at this point.

seeds comes down on the “tokens” side of how to store and represent intent - Kellogg’s Agent Memory Patterns makes a pointed case that the only structure an LLM really needs is tokens — that knowledge graphs and SQL-backed schemas tend to fight the model, since its weights don’t know your schema. Now, seeds does store deliberation in a database, with a light amount of metadata and typed references between seeds — exactly the kind of structure Kellogg side-eyes. But that structure is storage-layer bookkeeping; it isn’t what the agent reasons over. What an agent actually works with is reassembled markdown — the body of each seed is a large blob of prose, pure tokens — so at the layer Kellogg cares about, seeds lands on his side of the argument. The structure just makes the right tokens findable, the way an issue tracker does. Whether even that light structure helps the agent or quietly fights it is a question I’m still chewing on. And fortuitously, seeds chose to generate beads with a simple “## Why” section, feeding that intent to the implementing agent as plain tokens in the bead’s description.

My survey of the conversation surrounding intent debt helped me better understand where seeds does and does not fit in the landscape, and turned up a couple of interesting avenues I might pursue. There’s a whole other side to keeping this upstream record — what you can do by going back and revisiting it later — but that’s a post for another day. For now, seeds continues to help me capture, organize, refine, and implement my thoughts about my various projects, and I hope you have a chance to try it out and see if it does the same for you. That, after all, is where I’ve planted seeds: upstream of intent, in the deliberation itself.

This post is about a small CLI tool I built called seeds. It is named in the same gardening direction as beads, and it exists because of what beads showed me was possible.

I built seeds because plan files were failing me. The deliberation that went into a plan was being silently overwritten every time the AI rewrote the plan file. Beads, meanwhile, was working — after several months of heavy use I had watched my AI agents pick up the beads CLI and wield it as a native extension of themselves, as a bridge between me and the work they were doing. That broke open what I thought tools could be. So I went looking for something that did the same trick for the deliberation behind decisions, not the decisions themselves. I couldn’t find one. So I built seeds.

The relationship with beads

Steve Yegge built beads — in his words, “an issue tracker. A special one,” and “a tool that AI has built for itself.” The problem he was solving: coding agents have, in his framing, “no memory between sessions — sessions that only last about ten minutes.” After installing beads, my agents started saying things like “I notice all your tests are broken … and I’ve filed issue 397 to get them working again” — flagging unrelated work without losing the current task, and trusting that the issue would still be there when somebody came back to it. The agent doesn’t forget. The work survives until I’m ready for it.

Seeds catches the work that comes earlier. Plans, deliberations, questions, half-baked ideas, things I haven’t decided are work yet, and may decide are not work at all. A seed is allowed to be deferred indefinitely. A seed is allowed to be abandoned (with a reason). A seed is allowed to spawn six children that disagree with each other and can be resolved at a later time. Beads, designed around an execution lifecycle, doesn’t appear to me to have a comfortable place for any of that.

Yegge has been explicit about beads’ scope. From Beads Best Practices: “Everyone is focused on making planning tools, and Beads is an execution tool.” The one product he names as a “planning tool” is OpenSpec. On the project’s GitHub discussions he put the same point a different way: “I love the idea of planning tools, but I think they belong in a completely separate layer from Beads.” And in Beads Blows Up he draws an even sharper line: “Finished issues and future issues don’t really belong in Beads — it’s best to keep them in a separate store.”

The way I think about this now: there are really three layers. Planning tools like OpenSpec, GitHub Spec Kit, and the various task-master flavors produce a spec or a PRD or a task list as their output. Execution tools like beads track what is being built right now. Deliberation tools capture the reasoning that decides whether something becomes plannable in the first place — and that’s the layer seeds is trying to be. Yegge’s complaint about “everyone making planning tools” is about the first bucket, not the third. I read it as agreement, not opposition.

Deliberation has always mattered, but now it’s cheap to track

One thing I have learned across two decades of maintaining long-lived software is that the decisions behind a piece of code are often just as important as the code itself. Why a feature behaves the way it does, why some other feature was considered and dismissed, what data we did and didn’t trust at the time — that information shapes every future decision we’ll make about the same project. I have lost count of the number of times I have sat in a meeting to discuss a feature, only to dimly remember that we considered and rejected it years ago for reasons I can no longer recall, and the meeting becomes an exercise in re-deriving conclusions we already had.

Historically, I never bothered to capture the majority of what went into those decisions. I was too busy participating in the discussion to take minutes. I was too busy implementing to record the thinking that went into the implementation. As a human, documentation is expensive. As a human, retrieval from a large pile of freeform text is also expensive. For a solo developer, the math never quite worked out.

Both costs have plummeted with LLMs. Agents will happily transcribe meetings, summarize decisions, and write copious documentation alongside the code. They will also happily go off and read every document, transcript, and issue in a project to pull together a coherent picture from all of it. Capture is now cheap. Retrieval is now cheap. The value proposition for keeping a real deliberation log is completely different than it was just a couple of years ago.

Why plan files don’t carry it

The oft-recommended plan-file-based workflow turned out to be a slow leak. For me the workflow looked like this: AI and I converge on a plan via a conversation. The AI writes the plan into a file. I give the AI feedback. The AI rewrites the file based on that feedback. The new file is cleaner and more confident, but silent about the alternatives we had considered, the ideas I rejected, the questions I asked along the way. The plan that lands on disk is the destination. The journey is in the chat scrollback, which is being summarized, compacted, and silently discarded after a default of 30 days.

Ultimately, I noticed I had started either avoiding feedback on, or maintaining ever-incrementing versions of, plan documents I cared about, because the feedback would overwrite the very history I wanted to preserve. Important information was slipping through my fingers, and I was distraught.

Looking for an existing tool

For me the question became: how do I capture and organize all these decisions? That was the wrong question, so I got the wrong answer: Architecture Decision Records. I read up on ADRs, played with the small set of CLI tools that exist for them, and got partway through building one of my own. ADRs are a great pattern. They were not the pattern I needed.

ADRs capture the decision after it has been made. They summarize, neatly and concisely, what was decided, along with a little of what was considered. To me that’s only one step removed from a plan file. The best software I have worked on came out of hour-long conversations that weren’t reviews of decisions but deep dives into the problem domain — the messy, almost philosophical, in-the-weeds part where the interesting solutions actually live. ADRs, as I have come to understand them, are not intended to capture that.

Instead I started asking a different question: how do people capture and organize all the discussion that goes into a decision? That is where I came across deliberation tools, decision software, IBIS, dialogue mapping. It is a rich topic and was exactly what I had been looking for. It is also a bit of a ghost town. There is a literature stretching back to the 1960s, lots of frameworks, but almost nothing in active use. The tools that exist are too formal to reach for in the moment, and the moment is where the actual thinking is happening. More importantly, the tools were written for humans — I needed a tool written for agents by agents.

I figured I’d put together a prototype to see how it went.

seeds, briefly

I called the tool seeds. It is, deliberately, a rhyming, gardening cousin of beads. It’s also about the metaphor: ideas are like seeds — they take time to germinate, they need to be nurtured and cared for, they can eventually sprout and blossom into something amazing.

A taste of what it looks like:

seeds jot "We should write native PostgreSQL-flavored SQL and then transpile it for other RDBMSes"
# Created seed-a1b2: We should write native PostgreSQL-flavored SQL and then transpile it for other RDBMSes

seeds ask "are there any good libraries for transpiling SQL" --seed seed-a1b2
seeds answer q-c3d4 "There are transpilation libraries but only the following are written in the language used by this project: ..."

seeds resolve seed-a1b2

There are more verbs (explore, defer, abandon, link, tree, prime), a SQLite store underneath, and a JSONL export so the deliberation graph stays git-trackable. The README has the rest. The point is that every seed has a body the agent can fill with its investigations and rationale, and that every state transition — resolving, deferring, abandoning — takes a reason. Both habits leave a trail of why.

Now an awkward admission. I have never personally invoked the seeds CLI. Not once. I have probably read less than one percent of the content in my own seeds databases. To be fair, I have the same approach with beads — after several months of heavy use, I do not believe I have ever run bd ready from a terminal. Both tools, for me, are magical black boxes mediated entirely through a session with an AI agent. The tools work well enough that I haven’t had to pop the hood and muck around.

This shaped the design of seeds more than anything else. The bar I set early on was: if seeds required me to use it, it was not going to get used. It had to feel like a natural extension of the AI’s working memory. Flag-based, atomic, no interactive prompts; every command does one thing and takes simple arguments. The interface is for the agent. The agent is the interface for me.

The workflow rhythm

For the last five months, this is the workflow I’ve fallen into with seeds. I use this workflow multiple times a day across about a dozen different projects. By the time I’m done with the workflow, more often than not I end up with a set of beads that my agents use to one-shot an entirely new feature for a given project. Every time it feels like magic.

Step zero is the brain dump. I hop into a project, fire up dictation, and talk through every stray thought I have about what I might want — the gist, the weird ideas, the directions I’m leaning, the parts that worry me. I don’t try to be organized.

Step one is “turn this into seeds.” The agent reads the dump and creates a batch of seeds — some ideas, some concerns, some open questions, some early decisions. Messy and partial and exactly what I wanted.

Step two is the agent interviewing me. It asks for clarification on things I said and only half-explained. It asks about things I didn’t say but should have considered. It flags places where my own thoughts contradict each other. I answer what I can, and ask questions when I don’t know the answer.

Step three is the magic part. The agent figures out which questions need to be answered before others can be opened up — which seeds gate the rest. It does this completely without my intervention or guidance. The agent builds parent-child relationships, tags things, links related seeds, and surfaces which open questions are foundational. Now I have a database of ideas that are ready for curation and further exploration.

Step four is selecting a few seeds to nurture. Sometimes I know what I want to start implementing first. Sometimes I ask my agent what it thinks a good candidate feature might be. We zero in on a handful of seeds to flesh out and refine. One of the things I love best about seeds is it helps the agent and me stay focused. We home in on seeds we want to address now, defer the ones that are related but not immediately relevant. Those seeds are still there and often the agent will take them into consideration as we deliberate, helping keep the new feature on track but also in alignment with other potential future work. But I’ve never been so focused in a planning session as I am with seeds.

Step five is cycling on nurturing until we’re done. In considering our selected seeds, sometimes the agent has questions so I answer them, and the agent updates the relevant seeds with my answers. Sometimes I have questions so the agent investigates by reading the docs, looking at the code, running a query against the data, searching the web, etc., and then answers me and updates the seed with its findings.

This is one of the aspects of seeds that gives me the most peace of mind. The conversation isn’t just in chat that evaporates at the next compaction; the thoughts and findings land in the seeds and stay there. Months later, when some future agent revisits a seed, it has the same information that was on the table when the decision was made. Not a summary. The actual findings.

Eventually the agent and I run out of things to consider. We have made virtually all the decisions we needed to make for this session, and we got there together, organically. So often agents are chomping at the bit to just get implementing long before I’m ready, but even with seeds my agent will eventually say: “hey, want me to make some beads so we can get to writing code?” But in my experience, after following all of the workflow above, my answer is often yes. We have spent enough time thinking. The agent has not prematurely jumped to implementation, and I have not lingered in planning past usefulness.

Step six is the handoff. “Make some beads out of these seeds.” The actionable seeds become bd issues with their seed-body context attached, and implementation begins.

Step seven is the loop, lighter than it used to be. More often than not, implementation goes off without a hitch — no revisiting, no weird surprises, not much going back to the drawing board. Certainly nothing like the churn I used to live in with plan files. When the drawing board is needed, I revise existing seeds or supersede them with new ones. Resolved seeds occasionally get reopened because reality disagreed. The deliberation feeds the next round of implementation.

The surprises

I built seeds because I wanted to capture deliberation. I got that. I also got four things I didn’t predict, and together they’re why I cannot go back to plan files.

Focus. Seeds aren’t a giant document inviting me to add another sentence. They’re discrete pieces of ideas, decisions, and questions. When the seeds I care about are resolved, I stop. Back when I was using plan files, an AI in a coding context was unreasonably satisfied with insufficient planning — pressuring me to jump into implementation when there was real thinking still to be done. Conversely, once I had nudged the AI into a planning context, it would happily stay there: plan, nitpick, over-plan, overthink, as far as I was willing to engage. I was already suffering scope creep in my plan files, before a line of implementation got written. With seeds, I no longer end up with plans ten times the size they should be.

Peace of mind. The deliberation isn’t being lost. The level of detail and permanence I want is the level I’m getting. I don’t feel the slow leak anymore. This is the surprise I appreciate most when I sit down to plan something: there’s no anxiety humming in the background that the thinking I’m doing right now is going to evaporate by next week.

Better implementations. Implementation goes off without a hitch more often than it used to — fewer surprises, less revisiting, less churn. I cannot disentangle this from the AI getting better at implementation in general — that is a real and ongoing trend — but the shape of the improvement is consistent with what the deliberation log is doing. The agent walks into implementation with the rationale attached to each beads issue, and it doesn’t have to re-derive it.

Tighter iteration. The cycle between planning and implementing has shortened. I’m no longer fighting an AI that wants to either start coding right now or plan forever. One pattern that has quietly become my default: when I’m planning a feature, I often have a pie-in-the-sky version in mind — what the feature could become three years out. Rather than fight that, I capture the pie-in-the-sky vision as one seed, with all its nuance and complexity intact. Then I capture the actually-needed-now version as a separate seed and implement that. The pie-in-the-sky seed gets deferred. It sits in the database with the thinking already underway; if the bigger version is ever called for, the deliberation has already started. The complexity gets parked, not lost. I get to be ambitious during planning without paying for it during implementation.

I don’t know how much of this is the tool, how much is me getting better at working with AI, and how much is AI itself getting better over time. Teasing that apart is intractable. What I can tell you is that the four together feel real, and feel different from what I had before.

Where this has shown up in real work

A few shapes seeds has taken in real projects, anonymized.

ETL design. I do healthcare-data ETL for a living. ETL is a parade of small, fiddly decisions: which incoming columns get used, which transformations apply to each, which rows get dropped or merged or backfilled, and why. The “why” is what a downstream consumer needs in order to trust the resulting data. I have watched groups try to capture that “why” in documentation, and there is never any guarantee that the documentation tracks the implementation, and the day-of decisions made by the people doing the work almost never bubble back up. In my last two ETL design sessions I used seeds to capture every characterization query, every decision, every compromise, with a level of fidelity I have never been able to maintain before. When my downstream consumers had questions about the data, seeds was there to tell me not only what had happened, but why the ETL was done that way.

Greenfield projects. I have not started a greenfield project in four months that didn’t begin with seeds. So far I have used the brain-dump → triage → resolve → handoff workflow to design two web apps and three smaller projects in support of those web apps. The most useful run was a complicated data-collection tool that will eventually ingest from forty-two distinct sources — seeds helped me prioritize which sources to explore first and figure out how to catalog and prepare each one for harvesting.

Decade-old codebases. I have recently been on the receiving end of another round of feature requests for one of my long-running projects — some genuinely useful, several that need real discussion before they go anywhere. I have captured each request as a seed, along with whatever discussion has been had about it, so the topics can be picked up later for further refinement. This is not yet the longitudinal archive of deliberation I ultimately want for every project, but it has demonstrated that seeds can slot into an existing project with very minimal friction.

Things that aren’t software. I was recently invited to join a tabletop role-playing game and I used seeds to brainstorm a character — names, attributes, personality quirks, a homebrewed skill tree. Brainstorming a character throws off a lot of possibilities, most of which get cast aside or refined. Seeds was a fine fit. I suspect there is a fair amount of universality to the planning-and-deciding shape that goes well outside software.

What seeds doesn’t do

Some honest limits, in case anyone is about to talk themselves into this.

Seeds doesn’t enforce completeness. It is a forest, not a checklist. There is no workflow that says “you have twenty source columns and only seventeen of them are resolved, finish the other three.” But it isn’t blind, either. If each column has its own seed, the agent can see that three remain unresolved and bring it up. Completeness is in how you use the tool, not enforced by a workflow.

Capture in the moment is still hard. Even with the CLI installed and a seeds prime command teaching the agent the deliberation context, the agent doesn’t always reach for seeds when a decision is implicit in the conversation. It is pretty good. It needs reminding. The capture-in-the-moment problem isn’t solved. I’m considering agent hooks and backfill commands but those are still just a couple of seeds I’m letting germinate.

Seeds quietly assumes a very small team. I built it for a solo developer and their agent coworker — at most a team of one or two humans. There’s no multi-user model, no concurrent-editor story, no permissions, nothing for coordinating a crowd around the same deliberation graph. The JSONL export is git-trackable, so in principle a couple of people could share a seeds database the way they’d share any other file in a repo, but I’ve never tried it and I designed nothing for it. If you’re picturing seeds as shared deliberation infrastructure for a large team, that isn’t the tool I built.

Seeds may not work the same for everyone. My agents are increasingly making notes, memories, and agent files for themselves to better tailor their interaction with me. Because I’ve been using seeds for many months, I can only imagine the amount of customization my agents have built up around working with seeds. One of my greatest concerns about releasing and promoting seeds is whether it will behave the same way for others as it does for me.

I cannot quantify any of this. Much of the AI world runs on vibes, and seeds is no exception. The vibes I get from seeds are a sense of security that the ideas and plans and designs I’m capturing are being kept at the level of detail and permanence I want them at, and a sense of wonder that an agent can wield the tool as masterfully as it does. None of that is science. For me, seeds passes the vibe check. I don’t know how to make it pass yours.

Where this goes from here

Software development has always been one branch of a much larger discipline: taking a hard problem, breaking it into smaller ones, finding solutions, and stitching the solutions back into a whole. Agents are very good at the implementation step now. The homing-in-on-the-solution step is what’s left for the rest of us. Tools that help with that step feel more critical to me, not less, in a world where implementation is cheap.

The deliberation space is a rich, criminally under-addressed area. Now that capture and retrieval are cheap, there is room — and reason — for more tools here. I would be surprised if seeds is the last word, or even the right word; I built mine because I needed it. If you can use mine, use it. If it inspires you to build something better, even better. I would like to see more emphasis put on this space. For my part, I’ve recently begun encoding my deliberation rhythm into a couple of agent skills, so the agent captures decisions and feedback as they happen instead of waiting for me to ask.

The bottom line

I built seeds for myself. The way I use it may not be the way you would use it. I have very modest plans for it going forward — it is genuinely a minimum viable product that an agent vibe-coded for me to start with — but at this point I cannot imagine planning or designing anything of any sophistication without it. I do not use plan files anymore. The cycle is seeds, then beads, then implementation, then back to seeds. Over and over and over.

Honestly: I don’t feel safe planning if I’m not planning in seeds.

If you give it a try, just ask your AI agent to set it up and run it for you. I’d be interested to see how well it works for you and your agents.

Try it

github.com/outcomesinsights/seeds. MIT licensed. Beta. Issues and pull requests welcome.

For a well-understood condition like hypertension, this might feel manageable — ACE inhibitors, ARBs, calcium channel blockers, beta-blockers, and diuretics are the obvious starting points. But even then, that’s five different ATC therapeutic groups to traverse. For something like rheumatoid arthritis, where the treatment landscape spans conventional DMARDs, biologics targeting different cytokines, JAK inhibitors, corticosteroids, and NSAIDs, the picture gets complicated fast. And for any condition where off-label use is common or where the standard of care has shifted recently, manual assembly is a recipe for gaps.

We needed something better for our work at Outcomes Insights. So we built it.

The Traditional Approach and Its Limitations

The standard pharmacological classification systems are organized around what a drug is, rather than what it does for a patient. ATC classifies drugs into a hierarchy based on the organ system they act on and their pharmacological mechanism. That’s useful for understanding drug classes, but it doesn’t directly answer the question a researcher typically starts with “what medications are used to treat this disease?”

MED-RT gets closer. It maintains explicit “may treat” relationships between drugs and conditions. But these relationships are curated at the ingredient level, coverage varies across therapeutic areas, and combination products are largely absent — of the roughly 3,800 combination ingredients in RxNorm, almost none have “may treat” links. If a patient is prescribed amlodipine/benazepril for hypertension, MED-RT knows about amlodipine and benazepril individually, but it doesn’t know about the combination product that actually appears on the claim.

The deeper problem is structural. Classification systems organize medications by chemistry and mechanism. Researchers and analysts need to find medications by therapeutic intent. These are related perspectives, but they aren’t the same thing — and the gap between them is where medications get missed.

Starting from What Drugs Actually Say They Do

We took a different approach. Instead of working backward from classification systems, we started with what drugs say they do — literally. Every FDA-approved medication comes with a Structured Product Label (SPL) that includes an “Indications and Usage” section. This is the authoritative statement of what the drug is approved to treat, in clinical language.

We built a pipeline that extracts structured therapeutic relationships from these indication statements across roughly 2,000 ingredients with available labels. Each extraction captures a condition-action-outcome triple:

• What condition does this drug address?
• What does it do?
• What’s the intended outcome?

An LLM reads the raw label text and produces these structured extractions, which are then validated against the source.

Here’s the key step: we encode each extracted indication as a vector embedding using BioLORD-2023, a biomedical language model. This transforms clinical statements like “treatment of primary hypercholesterolemia” and “reduction of elevated total cholesterol” into mathematical representations that capture their clinical meaning. Those embeddings go into a vector database — one entry per indication sentence, linked back to the originating ingredient.

This creates something that didn’t exist before: a searchable index of what medications actually do, expressed in the language of clinical practice rather than the language of pharmacological classification.

Searching by Disease, Not by Drug Class

With this index in place, a disease query — “type 2 diabetes,” “rheumatoid arthritis,” “treatment-resistant depression” — returns a ranked list of every ingredient whose approved indications are semantically close to the query.

The system uses the same class of embedding technology behind our ICD-10 Code Set Builder, but applied to a different corpus. Instead of encoding ICD-10 code descriptions, we’re encoding drug indication statements. The search doesn’t require exact keyword matches. A query for “diabetic medications for obesity” surfaces GLP-1 receptor agonists like semaglutide and liraglutide — drugs classified under ATC code A10BJ (“Blood Glucose Lowering Drugs / GLP-1 analogues”) whose labels describe weight management indications. It also picks up tirzepatide from a different ATC subclass (A10BX). These are diabetes drugs by classification, but the system ranks them by relevance to obesity because their indication text matches the query semantically, not lexically (i.e., as a text match). (Note that ingredients that are in the class, but have no indication information that is semantically relevant, are listed as “class only” and do not have any indication score.)

Results come back grouped by ATC therapeutic class, so the familiar pharmacological organization is preserved as context. The figure below shows this in action — the GLP-1 class appears with semaglutide, liraglutide, and others ranked by both class membership and indication relevance, with the system auto-selecting the most relevant ingredients.

But the tool also surfaces drugs from other ATC groups when their indications match the query — a statin with a specific diabetes-related indication, for example, or an antihypertensive with outcome data in diabetic patients.

This dual organization — searched by disease, displayed by drug class — bridges the gap between how researchers think about medications and how pharmacology organizes them.

Two Modes of Search

Not every question starts with a disease. Sometimes a researcher already knows the drug class of interest and needs to enumerate its members. The tool supports both modes:

Condition-based search encodes the disease query as an embedding and searches the indication corpus. This is the mode that captures therapeutic intent — it finds drugs based on what they treat, regardless of where they sit in the ATC hierarchy.

Class-based search encodes a pharmacological class name — “SGLT2 inhibitors,” “TNF-alpha inhibitors,” “SSRIs” — and searches a separate collection of ATC class embeddings. This is useful when the user already has a therapeutic class in mind and wants to enumerate its members quickly.

In practice, many queries blend both perspectives. A search for “GLP-1 agonists for obesity” triggers a class-focused search that identifies the GLP-1 receptor agonist class, then cross-references indication embeddings to rank members by relevance to obesity specifically — distinguishing between GLP-1 drugs approved for weight management and those approved only for diabetes.

From Ingredients to Billable Codes

Finding the right ingredients is only half the problem. The practical deliverable for most use cases is a set of billable codes — NDC codes for pharmacy claims and HCPCS J-codes for physician-administered drugs. The system follows each ingredient through the RxNorm hierarchy to its dispensable products and their associated NDC codes, then cross-references the CMS NDC-to-HCPCS crosswalk for injectable and infused medications.

Each ingredient in the results shows its NDC and HCPCS counts, available product forms, and representative indication text. Users can include or exclude individual ingredients, and the system pre-selects likely relevant drugs using a combination of semantic similarity scoring and an optional AI pre-check that reviews the selections in clinical context.

Why This Matters

Medication identification is foundational to a broad range of healthcare research — pharmacoepidemiology, outcomes studies, formulary analysis, health economics.

The traditional approach — browsing ATC hierarchies and curated drug-disease databases — works, but it organizes information around the drug rather than around the patient’s condition. Our approach inverts the starting point: begin with what you want to treat, and let the system find the medications that treat it. The pharmacological classification is still there, providing structure and context, but it’s no longer the only way in.

This is the second application we’ve built on this foundation. The first, our Code Set Builder for ICD-10-CM, uses the same semantic search principles to identify diagnosis codes. Both tools share a conviction that clinical intent (i.e., what am I looking for?) should be the starting point for code set construction, with classification hierarchies serving as guardrails rather than the primary navigation mechanism.

The traditional approach is some combination of clinical knowledge, keyword searches through code descriptions, and peer-reviewed literature. You open a reference table, search for “diabetes,” scroll through hundreds of results, and try to decide which codes belong and which don’t. It’s tedious, it’s error-prone, and two analysts working on the same condition can produce different code sets. That divergence matters — it can affect who ends up in the study cohort, and/or the inferences drawn from the results.

The Problem Is Bigger Than It Looks

ICD-10-CM has roughly 97,000 codes. They’re organized hierarchically, which helps, but the hierarchy is deep and full of clinical nuance. Take type 2 diabetes: the E11 family alone contains codes for ophthalmic, neurological, circulatory, renal, and dermatological complications, each with multiple subcategories. A keyword search for “kidney” won’t find codes described as “renal” or “nephropathy.” And even when you find the right family, you still have to decide which subcategories are relevant to your specific research question.

There are workarounds — reusing code sets from prior studies, borrowing from published literature, or building from institutional templates. They work, but they’re hard to audit, hard to reproduce, and tend to drift over time as analysts make small, undocumented adjustments.

Semantic Search Changes the Game

We’ve been building a tool that takes a fundamentally different approach. Instead of searching code descriptions with keywords, it searches code meaning with natural language.

The system encodes every ICD-10-CM code as a high-dimensional vector — a mathematical representation of its clinical meaning, informed by the code’s description, its position in the ICD-10 hierarchy, and LLM-generated clinical context. A query like “type 2 diabetes with kidney complications” returns codes that are semantically close to the intended meaning, not just codes that happen to contain matching words.

This is the same class of technology behind modern search engines and recommendation systems, applied to a very specific and important problem: identifying the right ICD-10-CM codes for a research project.

Not Just Search — a Feedback Loop

Semantic search is a good starting point, but a starting point isn’t a finished code set. The real value comes from what happens next.

The tool groups results into ICD10 code families defined by their 3 character group (e.g., E11). Then the tool scores them by relevance which is a measue of how close they are to your original query. Then, it identifies a natural cutoff between clearly relevant and probably irrelevant families. From there, the user reviews individual codes and marks them as include, exclude, or “unsure”. Each include or exclude decision feeds back into the algorithm through a process called Rocchio relevance feedback — the system adjusts its internal representation of the query based on those choices, pushing away from excluded codes and towards the included ones. It then re-searches with a refined understanding of what the user is looking for.

This creates an iterative refinement loop. With each round, the results get more precise. New families surface that the user might not have considered. Irrelevant ones drop away. The process is reproducible and auditable — a researcher can explain to a reviewer exactly how and why each code ended up in the set.

There is also a graphical feature that shows how close the codes are from each other in meaning, so you can get visual feedback on what is happening.

AI Assistance Where It Helps

We’ve layered optional LLM features on top of this core workflow. Before the user starts reviewing codes, an AI pre-check can classify subcategories as likely relevant, likely irrelevant, or uncertain — giving a head start on the manual review. A query advisor can flag unexpected code families and ask clarifying questions about intent. And an explain feature can break down what a specific code represents and why it might or might not belong.

These features are genuinely optional. The semantic search and Rocchio refinement work without any LLM. But when available, they reduce the cognitive load of sorting through hundreds of subcategories.

Harmonization: Comparing Code Sets

We also built a harmonization tool for a problem that sometimes arises in practice: there are two code sets for the same condition, built by different analysts or drawn from different sources, and there is a need to reconcile them. The tool shows the user what’s in both sets, what’s unique to each, and lets the user build a merged set with full visibility into the differences.

Case Study: Dementia in the Charlson Comorbidity Index

To make this concrete, consider a code set that thousands of researchers use routinely: the dementia algorithm from the Charlson Comorbidity Index.

The ICD-10 version of the Charlson index comes from Quan et al. (2005), a carefully conducted study that translated the original ICD-9-CM algorithms into ICD-10 using a multi-step consensus process across research groups in three countries. The dementia algorithm they published includes codes from four ICD-10 families: F00 (dementia in Alzheimer’s disease), F01–F03 (vascular, other, and unspecified dementia), G30 (Alzheimer’s disease), and G31.1 (senile degeneration of brain). It’s a reasonable list — and it’s been cited thousands of times.

But for people working with US claims data coded in ICD-10-CM, this list has problems.

First, F00 doesn’t exist in ICD-10-CM. The US clinical modification never adopted that code. In ICD-10-CM, Alzheimer’s disease with documented dementia is typically coded with a G30.- code for the underlying Alzheimer’s disease plus an F02.80/F02.81- code for the dementia manifestation. A researcher who takes the Quan codes at face value and searches for F00 in US claims data will find zero patients — not because there are no Alzheimer’s patients, but because the code doesn’t exist in the system they’re searching.

Second, and more consequentially, the algorithm misses entire categories of dementia that are now explicitly coded in ICD-10-CM:

• G31.0x — Frontotemporal dementia, including Pick’s disease and other frontotemporal variants. This is a clinically significant dementia subtype with its own family of codes.
• G31.83 — Neurocognitive disorder with Lewy bodies, which captures dementia with Lewy bodies as a distinct neurodegenerative disorder, and was introduced in ICD-10-CM after the Quan paper was published.

These aren’t obscure edge cases. Lewy body dementia accounts for an estimated 3–7% of all dementia cases. Frontotemporal dementia is a leading cause of dementia in people under age 60. Missing them means missing patients — exactly the kind of systematic gap that introduces bias into observational studies.

What the Tool Finds

When we run the query “dementia” through our Code Set Builder, semantic search surfaces exactly the families you’d expect: F01, F02, F03, and G30 — the core of the Quan algorithm. But it also identifies G31, scored highly because codes like G31.0x (frontotemporal dementia) and G31.83 (Neurocognitive disorder with Lewy bodies) are semantically close to the query. The tool doesn’t just match the word “dementia” in code descriptions — it understands that these conditions are dementias, even when the description reads “frontotemporal disease” or uses other clinical terminology.

From there, the AI pre-check step can help sort out which G31 subcategories belong (G31.0x, G31.83 — yes; G31.2, spinocerebellar degeneration — probably not), and iterative refinement allows fine-tuning based on the specific research question. The result is a code set that’s more comprehensive than Quan, specifically adapted to ICD-10-CM, and documented through every step.

This isn’t a criticism of Quan et al. — their work was rigorous and remains foundational. The point is that code sets built by manual translation twenty years ago inevitably have gaps, especially when the underlying coding system has continued to evolve. A tool that starts from semantic meaning rather than code-to-code translation can identify what manual processes miss.

Why This Matters

Code set construction is foundational work in observational research. It determines who’s in your study population and who’s not. Getting it wrong — in either direction — compromises everything downstream. Yet we’ve been treating it as an artisanal process, relying on individual expertise and ad hoc methods.

This tool doesn’t replace clinical judgment. It augments it with semantic understanding, algorithmic feedback, and AI assistance, producing code sets that are more comprehensive, more precise, and more defensible than what most of us can build by hand.

We’re using it internally at Outcomes Insights, and we’re excited about where it’s headed. In fact we were inspired to work on similar projects to identify medications (NDC and HCPCS codes) and procedures (CPT and HCPCS codes) based on disease areas.

Preview

Welcome to the new way to access data – the data enclave. This is a private data platform, designed and controlled by the data provider, on which researchers must do all their work. In this post we share some initial impressions from our experience with three separate data enclaves, and how we are moving forward in this new world.

High-level observations

One project is based on the Medicare Virtual Research Data Center (VRDC) and the other two are based on commercial data enclaves. The good news is that R and Python are available in all three. The bad news is that SAS, Stata, and RStudio are not universally available. Also, one provider offers Databricks, one uses Snowflake, and the last uses Athena - three different underlying data platforms.

The data documentation has been better than the documentation from providers who allow us to have the data on our servers. However, no data provider provides sufficiently detailed information about each column (variable) in the data. We still have to do a lot of data discovery on the raw data before we can use it (e.g., identify missing data, discover unexpected values, determine relationships among the tables, etc.)

For privacy reasons, all three enclaves have tight restrictions on what can be taken off the system. This makes it challenging for the research team to review the data, and adds a delay since all output has to be reviewed.

Opportunity

As many know, we have our Jigsaw software which can generate the SQL to create analysis-ready datasets from data in a common data model. We can generate SQL in most dialects used by data platforms, so we can readily adapt our research process to data enclaves.

But before we can construct cohorts using Jigsaw, we need to organize the data into a data model. This process is time-consuming even on an on-premises system. To allow us to support multiple data platforms in a reasonable timeframe, we started building tools to shorten the process from 6-12 weeks to something more like 1-3 weeks.

Side note: We’d like to say we can do it in hours, but we are being realistic given the heterogeneity of data, the platform architectures, and the time it takes to process large amounts of data, even with powerful hardware. The good news is that once the code is written for a data source, it is substantially faster to process another instance of the same data.

Progress

We have just completed our first version of tools to facilitate the process. At a high level these tools do the following:

1. Systematically explore every table and column in the raw data and characterize it in a Shiny app.
   • This includes missing values, ranges, typical values, unusual values (which are often data errors), and the identifiers (keys) that are used to connect records in different tables.
   • We also incorporate variable definitions from the data dictionary and show all relevant information in one interface.

2. Process the output of step 1 to create the mapping details required to move the raw data into the Generalized Data Model. This includes automating as much of the process as possible, and conducting logic checks to ensure that the mappings make sense.

3. Use the mappings in step 2 to automatically generate an SQL script to implement the complete transformation process on the target data platform.

Importantly, these tools are being built in R as much as possible so they can be brought into any data enclave.

Why go through all of this?

In terms of working on data enclaves, the following are some of the key benefits of a set of flexible tools for data management and cohort building:

1. The raw data stays on the data platform at all times, as required by data providers.
2. The data provider’s schema remains private, as required by some data providers.
3. The raw data is fully documented and can be compared to the reorganized data as part of a quality control process. This is important for research done for regulatory purposes, as well as being a good idea generally.

Implications

The most important implication is that we can use the same process for every project. We can organize the data into one data model. We can use one library of algorithms and one protocol builder for all research (Jigsaw). And because Jigsaw has its own data model for analysis-ready data, we can write efficient analysis code and standardize our analyses. In other words, we have the same path from start to finish regardless of whether the data is on premises or in a data enclave.

Another benefit is that all this infrastructure can be made open-source and available to anybody to use – competitors, data providers, commercial organizations, government researchers, etc. We are not ready for that yet, but that is where we are headed. If you want to know more, or to collaborate with us, don’t hesitate to contact us. 

Research using observational data (“real-world data”) has grown substantially over the last decade. New data sources have become available, and regulatory agencies have increasingly supported real-world evidence as part of their decision-making. However, the raw data available from data providers is virtually impossible to analyze in its original form. As a result, the enormous challenge of reviewing and manipulating the raw data creates a substantial barrier to conducting quality observational research.

The Real Miracle in Research

Borrowing a thought from Sydney Harris, the figure below is how most researchers view the process of structuring their data for research.

Breaking it Down

At a simple level, the manipulation of data can be broken into two tasks. The first is to organize the raw data by cataloging its contents and the relationships among the tables, examining it for completeness and correctness, and mapping it to a structure suitable to research. (For regulatory purposes, the organization process itself needs to be documented as well.) The second task is identifying the relevant cohort to study, extracting the relevant records, and creating analysis-ready datasets. Once this two-step data pipeline is implemented, researchers can then conduct the analyses of interest.

The Starting Point

For the first task, the only option for most researchers is to create an ad hoc data structure specific to the research project at hand. While adequate for one-off studies, this approach is prone to error, difficult to modify, and impossible to scale. More flexible approaches to data organization involve either writing a query translation layer or moving data into a common data model structure. Both approaches map the original data structure into a new, more efficient structure, but the common data model approach can be manipulated more efficiently, reliably, and repeatably by modern database platforms.

The Problem and the Solution

This highlights the fundamental problem for all researchers: there are no useful off-the-shelf, open-source toolkits that help researchers process raw healthcare data into a usable form for research. Hence, we are working to build such a toolkit and to make it available for others.

With Medicare data moving to the Virtual Research Data Center (VRDC) exclusively, researchers are being forced into a new way of conducting research. Some have expressed concerns that research could take longer, or that quality control could suffer. But these concerns simply highlight the long-standing struggle to write effective code for conducting observational research.

Part of the problem is that observational research is an insular process. To ensure patient privacy, data can not easily be shared. This makes collaboration difficult. And because 60-80% of a research project is simply wrangling raw data to create analysis-ready datasets, data management is labor-intensive.

To complicate matters further, most data management practices are still stuck in the technology of the 1990’s. One must look no further than to the ubiquity of SAS or Stata for large-scale data management, the minimal use of relational databases and SQL, and the use of double data programming instead of modern approaches for code testing and version control. Lastly, many researchers conflate cohort creation with data analysis by trying to use the same software and coding practices for both parts of a project.

In short, the limited adoption of appropriate data management practices has led to a dead-end in the era of data enclaves like the VRDC.

Where Do We Go?

With the advent of better tools for working with raw data, observational research needs to take advantage of fit-for-purpose technologies and apply them in appropriate ways. Standard data manipulation languages (e.g., SQL), relational databases (e.g., PostgreSQL), and data platforms (e.g., Databricks) make it efficient to extract analysis-ready datasets from terabytes of raw data.

Although tools exist to improve the efficiency of observational research, the solution is not as simple as changing software. The entire process needs to re-defined. And therein lies the primary issue – most observational researchers are not well-versed in the software development best practices that are required to move the field forward. Even more recent approaches to rethinking observational research (e.g., OMOP/OHDSI, FDA Sentinel, PCORnet, i2b2, etc.) still fail to address many of these challenges properly.

Software Development – Really?

By necessity, observational researchers dabble in software development as a byproduct of writing the code for a project. But code development is limited by the narrow scope of the project at hand. The iterative process of adapting code for each new project and evaluating the output to ensure correctness is second-nature to researchers. But the prospect of working on the VRDC exposes the inefficiencies and limitations of this approach.

The root cause is that researchers are increasingly out of their element when using modern software (e.g., Databricks on the VRDC). Writing reusable and testable code for modern data platforms requires an understanding of software libraries, programming paradigms, and testing frameworks. However, the fundamentals of these tools are not part of the training for most researchers and statistical programmers.

To move research forward, researchers need to reconsider their approach and adopt a software development mindset. Viewed through this lens, cohort-building can be reduced to a set of repetitive tasks that are identical across different observational study designs. In other words, researchers need to think in terms of building a “data pipeline”.

A data pipeline requires clear specifications for the inputs and outputs for each part of the process. There are many benefits to this approach. Importantly, this includes creating specifications for organizing the raw data. This can be accomplished in a variety of ways including using data models, database views, or other approaches. The key point related to the VRDC is that, once the structure of the raw data is defined, there is no need to touch the data to specify how to build the cohort and to create the analysis-ready datasets.

By analogy, it is like using Google or Apple Maps. Once we have a database of GPS coordinates for all roads and addresses, and an understanding of basic driving rules, we can write software to create detailed, optimized directions without ever having to get in the car.

How Will That Help with Platforms like the VRDC?

Working on the VRDC means that researchers need to develop their cohort-creation code outside the VRDC (see this blog post). Once researchers can generate the code to create their analysis-ready datasets without needing simultaneous access to the raw data, working on an enclave like the VRDC becomes easier, faster, and cheaper. In fact, this kind of “offline” approach enables researchers to work on data in a consistent fashion anywhere it is stored – a powerful idea.

That Is Not Realistic. Or Is It?

At this point it would be reasonable to say, “but surely everyone can’t build their own validated software stack for the few studies they might do in a year; that is neither possible nor efficient.” I can tell you that, as a very small company, we have built exactly this system so we know it is possible. I can tell you that, having learned how to work “offline”, we are excited to work on platforms like the VRDC. And I can tell you that, as I write this, we are working on some VRDC-related demonstration projects to prove that this approach works. So, yes, it can be done. But I can also tell you that it involved a lot of hard lessons and about 10 years of work.

That’s Nice. But How Does that Help Anyone Else?

One of our core principles is to share our work with others. The key components of our research process have been publicly available via GitHub since they were created. This includes the Generalized Data Model, which defines how we organize raw data, and ConceptQL, the language we built to create, store, and share algorithms. We also made our software, Jigsaw, and our algorithm library publicly readable as of last year (see this blog post). And we are actively searching for a way make Jigsaw usable by others to solve problems like working on the VRDC (see this blog post).

Conclusion

Observational researchers need to think outside the box that has enclosed observational research for the last 40 years. The tools are available, and we have laid much of the groundwork. We will continue to share our progress and help others conduct quality research in an evolving data landscape. If you have any questions, or if you want to collaborate, feel free to contact us.

Last year we made our algorithm library publicly available. In 2024, we are working to make our entire Jigsaw application freely accessible for creating and sharing protocols, creating and sharing algorithms, and generating the code to create analysis-ready datasets from observational data.

Doesn’t This Exist Already?

Solutions for streamlining observational research fall into one of the following approaches:

• Building an internal repository of implementation code
• Implementing and supporting an open-source software system internally
• Licensing access to a commercial platform

However, these options also have important limitations:

• In-house solutions are generally software-specific and have limited version control
• Open-source platforms require staff for installation, support, and updates
• Commercial platforms can be expensive black boxes with little visibility into underlying processes being implemented
• No solution allows researchers to collaborate across institutions that use different approaches.

In short, despite improvements in software capabilities, researchers still reside in their own silos, unable to collaborate efficiently with one another.

What is the Alternative?

The ideal solution is a freely accessible, cloud-based software application.

As an analogy, think about GitHub. For anyone unfamiliar with GitHub, the following is the one-sentence Wikipedia description:

GitHub, Inc. is an AI-powered developer platform that allows developers to create, store, manage and share their code.”

So, imagine if we alter that slightly for Jigsaw to read as follows:

Jigsaw is a cloud-based platform that allows observational researchers to create, store, manage and share their research protocols and to generate the code for implementing them against data.”

Conclusion

Despite thousands of hours and millions of dollars being invested in potential solutions, the fundamental problem remains – research methods are still too fragmented. It is about time we solved this problem. We think Jigsaw is the solution.

If an organization can store all the data it needs on its own servers, it can choose almost any software package or platform to manipulate the data for research purposes. But what happens if the organization isn’t allowed to store the data on its own servers? For example, most people who want to access the full Medicare data must do so within the CMS Virtual Research Data Center (VRDC) using either SAS or Databricks. Similarly, some countries limit their data to servers only accessible within their own country, and some commercial data providers limit access to their own virtual data centers. While this can improve data security and reduce patient privacy risks, it creates logistical challenges for researchers.

The Solution?

The solution is to bring project-specific software code to the data behind the firewall. But how does one write code for a project without touching the actual data? There are at least two options for solving the problem: synthetic data and code-generation software.

Synthetic Data

In theory, researchers can craft their project code against a synthetic, but realistic, copy of the data. This presupposes that someone will create and maintain an accessible, usable, privacy-protecting synthetic dataset for each data source of interest. Even then, working with large, unwieldy data can be a challenge. Imagine working with a synthetic version of the full Medicare data – that isn’t a job for anyone without access to suitable computing resources. All-in-all, synthetic data represents a possible solution, but it isn’t particularly efficient because it still relies on hand-writing code for manipulating potentially large datasets.

Code-Generation Software: Jigsaw

In our opinion, the better option is to use Jigsaw. It may not be obvious, but Jigsaw doesn’t need to touch the actual raw data to do its job. How is that possible? Jigsaw’s job is to write all the SQL queries for creating an analysis-ready dataset from the raw, organized data. As long as the data is organized using a known data model, Jigsaw can write a script containing the required SQL queries. A user can then run the script on a server containing the organized data, and the script can save the analysis-ready data sets on the same server in a location specified by the user.

Because Jigsaw itself can be Cloud-based, researchers can collaborate on the specification of the analysis-ready data from anywhere. The protocol and its algorithms remain publicly available and shareable. As before, all algorithms are explicitly documented in the protocol summary document that can be generated (and also shared).

Or Both?

In some scenarios, it could make sense to use both approaches. One could use Jigaw to create an analysis-ready dataset, and then create a synthetic version for developing the analyses themselves.

Challenges

In order for this work, the data on the server needs to be reorganized into a data model. We strongly prefer the Generalized Data Model because it brings together some of the best ideas of the i2b2 and OMOP/OHDSI data models. By this, we referring to combining the i2b2 idea of storing all clinical information in one central “fact” table combined with the OMOP/OHDSI vocabularies.

Once a transformation to a data model is created, it is relatively easy to share the code for others to transform their raw data into a data model. As of early 2024, we have code for the current SEER-Medicare linked data. Next, we will create the transformation code for the Medicare data on the VRDC. After that, we are open to suggestions.

Conclusion

Working on other data platforms doesn’t have to be challenging. We think Jigsaw can be a powerful tool for working with observational data wherever it is without compromising data security or patient privacy.

No Cluster Needed

Perhaps it was my bias from working with Apache Impala a few years back, but I just assumed that Spark was going to need Hadoop set up on a cluster of servers. I didn’t want to spend my time getting all that set up just to play around with Spark, so I never bothered with it before.

Turns out, Spark has a rather robust single-machine setup. Even better, there’s an R library that took care of all the set up for me.

sparklyr Makes Spark Simple

The R package sparklyr made my foray into Spark dead simple. The package happily installed Spark for me and provided me functions to easily start and stop a Spark instance from within my R scripts.

Pro tip: by default sparklyr limits Spark to a single core when it starts up an instance. You can change to multiple cores pretty easily and it makes a world of difference in terms of performance.

dplyr and Spark Is a Powerful Combination

sparklyr gave me access to the tables I loaded into Spark. dplyr gave me the ability to manipulate and query those tables via dbplyr.

dplyr is amazing. Rather than hand-writing Spark SQL, dplyr provides a set of functions that allowed me to join tables, add where clauses, and manipulate the columns returned from Spark.

Great Performance

My project was to explore replacing an existing part of our data pipeline. Using Spark, our processing time went from days to hours.

More Spark in the Future

After this successful venture into Spark territory, I’m pretty sure I’ll be employing Spark in future projects.