The traditional approach is some combination of clinical knowledge, keyword searches through code descriptions, and peer-reviewed literature. You open a reference table, search for “diabetes,” scroll through hundreds of results, and try to decide which codes belong and which don’t. It’s tedious, it’s error-prone, and two analysts working on the same condition can produce different code sets. That divergence matters — it can affect who ends up in the study cohort, and/or the inferences drawn from the results.

The Problem Is Bigger Than It Looks

ICD-10-CM has roughly 97,000 codes. They’re organized hierarchically, which helps, but the hierarchy is deep and full of clinical nuance. Take type 2 diabetes: the E11 family alone contains codes for ophthalmic, neurological, circulatory, renal, and dermatological complications, each with multiple subcategories. A keyword search for “kidney” won’t find codes described as “renal” or “nephropathy.” And even when you find the right family, you still have to decide which subcategories are relevant to your specific research question.

There are workarounds — reusing code sets from prior studies, borrowing from published literature, or building from institutional templates. They work, but they’re hard to audit, hard to reproduce, and tend to drift over time as analysts make small, undocumented adjustments.

Semantic Search Changes the Game

We’ve been building a tool that takes a fundamentally different approach. Instead of searching code descriptions with keywords, it searches code meaning with natural language.

The system encodes every ICD-10-CM code as a high-dimensional vector — a mathematical representation of its clinical meaning, informed by the code’s description, its position in the ICD-10 hierarchy, and LLM-generated clinical context. A query like “type 2 diabetes with kidney complications” returns codes that are semantically close to the intended meaning, not just codes that happen to contain matching words.

This is the same class of technology behind modern search engines and recommendation systems, applied to a very specific and important problem: identifying the right ICD-10-CM codes for a research project.

Not Just Search — a Feedback Loop

Semantic search is a good starting point, but a starting point isn’t a finished code set. The real value comes from what happens next.

The tool groups results into ICD10 code families defined by their 3 character group (e.g., E11). Then the tool scores them by relevance which is a measue of how close they are to your original query. Then, it identifies a natural cutoff between clearly relevant and probably irrelevant families. From there, the user reviews individual codes and marks them as include, exclude, or “unsure”. Each include or exclude decision feeds back into the algorithm through a process called Rocchio relevance feedback — the system adjusts its internal representation of the query based on those choices, pushing away from excluded codes and towards the included ones. It then re-searches with a refined understanding of what the user is looking for.

This creates an iterative refinement loop. With each round, the results get more precise. New families surface that the user might not have considered. Irrelevant ones drop away. The process is reproducible and auditable — a researcher can explain to a reviewer exactly how and why each code ended up in the set.

There is also a graphical feature that shows how close the codes are from each other in meaning, so you can get visual feedback on what is happening.

AI Assistance Where It Helps

We’ve layered optional LLM features on top of this core workflow. Before the user starts reviewing codes, an AI pre-check can classify subcategories as likely relevant, likely irrelevant, or uncertain — giving a head start on the manual review. A query advisor can flag unexpected code families and ask clarifying questions about intent. And an explain feature can break down what a specific code represents and why it might or might not belong.

These features are genuinely optional. The semantic search and Rocchio refinement work without any LLM. But when available, they reduce the cognitive load of sorting through hundreds of subcategories.

Harmonization: Comparing Code Sets

We also built a harmonization tool for a problem that sometimes arises in practice: there are two code sets for the same condition, built by different analysts or drawn from different sources, and there is a need to reconcile them. The tool shows the user what’s in both sets, what’s unique to each, and lets the user build a merged set with full visibility into the differences.

Case Study: Dementia in the Charlson Comorbidity Index

To make this concrete, consider a code set that thousands of researchers use routinely: the dementia algorithm from the Charlson Comorbidity Index.

The ICD-10 version of the Charlson index comes from Quan et al. (2005), a carefully conducted study that translated the original ICD-9-CM algorithms into ICD-10 using a multi-step consensus process across research groups in three countries. The dementia algorithm they published includes codes from four ICD-10 families: F00 (dementia in Alzheimer’s disease), F01–F03 (vascular, other, and unspecified dementia), G30 (Alzheimer’s disease), and G31.1 (senile degeneration of brain). It’s a reasonable list — and it’s been cited thousands of times.

But for people working with US claims data coded in ICD-10-CM, this list has problems.

First, F00 doesn’t exist in ICD-10-CM. The US clinical modification never adopted that code. In ICD-10-CM, Alzheimer’s disease with documented dementia is typically coded with a G30.- code for the underlying Alzheimer’s disease plus an F02.80/F02.81- code for the dementia manifestation. A researcher who takes the Quan codes at face value and searches for F00 in US claims data will find zero patients — not because there are no Alzheimer’s patients, but because the code doesn’t exist in the system they’re searching.

Second, and more consequentially, the algorithm misses entire categories of dementia that are now explicitly coded in ICD-10-CM:

• G31.0x — Frontotemporal dementia, including Pick’s disease and other frontotemporal variants. This is a clinically significant dementia subtype with its own family of codes.
• G31.83 — Neurocognitive disorder with Lewy bodies, which captures dementia with Lewy bodies as a distinct neurodegenerative disorder, and was introduced in ICD-10-CM after the Quan paper was published.

These aren’t obscure edge cases. Lewy body dementia accounts for an estimated 3–7% of all dementia cases. Frontotemporal dementia is a leading cause of dementia in people under age 60. Missing them means missing patients — exactly the kind of systematic gap that introduces bias into observational studies.

What the Tool Finds

When we run the query “dementia” through our Code Set Builder, semantic search surfaces exactly the families you’d expect: F01, F02, F03, and G30 — the core of the Quan algorithm. But it also identifies G31, scored highly because codes like G31.0x (frontotemporal dementia) and G31.83 (Neurocognitive disorder with Lewy bodies) are semantically close to the query. The tool doesn’t just match the word “dementia” in code descriptions — it understands that these conditions are dementias, even when the description reads “frontotemporal disease” or uses other clinical terminology.

From there, the AI pre-check step can help sort out which G31 subcategories belong (G31.0x, G31.83 — yes; G31.2, spinocerebellar degeneration — probably not), and iterative refinement allows fine-tuning based on the specific research question. The result is a code set that’s more comprehensive than Quan, specifically adapted to ICD-10-CM, and documented through every step.

This isn’t a criticism of Quan et al. — their work was rigorous and remains foundational. The point is that code sets built by manual translation twenty years ago inevitably have gaps, especially when the underlying coding system has continued to evolve. A tool that starts from semantic meaning rather than code-to-code translation can identify what manual processes miss.

Why This Matters

Code set construction is foundational work in observational research. It determines who’s in your study population and who’s not. Getting it wrong — in either direction — compromises everything downstream. Yet we’ve been treating it as an artisanal process, relying on individual expertise and ad hoc methods.

This tool doesn’t replace clinical judgment. It augments it with semantic understanding, algorithmic feedback, and AI assistance, producing code sets that are more comprehensive, more precise, and more defensible than what most of us can build by hand.

We’re using it internally at Outcomes Insights, and we’re excited about where it’s headed. In fact we were inspired to work on similar projects to identify medications (NDC and HCPCS codes) and procedures (CPT and HCPCS codes) based on disease areas.

Preview

Welcome to the new way to access data – the data enclave. This is a private data platform, designed and controlled by the data provider, on which researchers must do all their work. In this post we share some initial impressions from our experience with three separate data enclaves, and how we are moving forward in this new world.

High-level observations

One project is based on the Medicare Virtual Research Data Center (VRDC) and the other two are based on commercial data enclaves. The good news is that R and Python are available in all three. The bad news is that SAS, Stata, and RStudio are not universally available. Also, one provider offers Databricks, one uses Snowflake, and the last uses Athena - three different underlying data platforms.

The data documentation has been better than the documentation from providers who allow us to have the data on our servers. However, no data provider provides sufficiently detailed information about each column (variable) in the data. We still have to do a lot of data discovery on the raw data before we can use it (e.g., identify missing data, discover unexpected values, determine relationships among the tables, etc.)

For privacy reasons, all three enclaves have tight restrictions on what can be taken off the system. This makes it challenging for the research team to review the data, and adds a delay since all output has to be reviewed.

Opportunity

As many know, we have our Jigsaw software which can generate the SQL to create analysis-ready datasets from data in a common data model. We can generate SQL in most dialects used by data platforms, so we can readily adapt our research process to data enclaves.

But before we can construct cohorts using Jigsaw, we need to organize the data into a data model. This process is time-consuming even on an on-premises system. To allow us to support multiple data platforms in a reasonable timeframe, we started building tools to shorten the process from 6-12 weeks to something more like 1-3 weeks.

Side note: We’d like to say we can do it in hours, but we are being realistic given the heterogeneity of data, the platform architectures, and the time it takes to process large amounts of data, even with powerful hardware. The good news is that once the code is written for a data source, it is substantially faster to process another instance of the same data.

Progress

We have just completed our first version of tools to facilitate the process. At a high level these tools do the following:

1. Systematically explore every table and column in the raw data and characterize it in a Shiny app.
   • This includes missing values, ranges, typical values, unusual values (which are often data errors), and the identifiers (keys) that are used to connect records in different tables.
   • We also incorporate variable definitions from the data dictionary and show all relevant information in one interface.

2. Process the output of step 1 to create the mapping details required to move the raw data into the Generalized Data Model. This includes automating as much of the process as possible, and conducting logic checks to ensure that the mappings make sense.

3. Use the mappings in step 2 to automatically generate an SQL script to implement the complete transformation process on the target data platform.

Importantly, these tools are being built in R as much as possible so they can be brought into any data enclave.

Why go through all of this?

In terms of working on data enclaves, the following are some of the key benefits of a set of flexible tools for data management and cohort building:

1. The raw data stays on the data platform at all times, as required by data providers.
2. The data provider’s schema remains private, as required by some data providers.
3. The raw data is fully documented and can be compared to the reorganized data as part of a quality control process. This is important for research done for regulatory purposes, as well as being a good idea generally.

Implications

The most important implication is that we can use the same process for every project. We can organize the data into one data model. We can use one library of algorithms and one protocol builder for all research (Jigsaw). And because Jigsaw has its own data model for analysis-ready data, we can write efficient analysis code and standardize our analyses. In other words, we have the same path from start to finish regardless of whether the data is on premises or in a data enclave.

Another benefit is that all this infrastructure can be made open-source and available to anybody to use – competitors, data providers, commercial organizations, government researchers, etc. We are not ready for that yet, but that is where we are headed. If you want to know more, or to collaborate with us, don’t hesitate to contact us. 

Research using observational data (“real-world data”) has grown substantially over the last decade. New data sources have become available, and regulatory agencies have increasingly supported real-world evidence as part of their decision-making. However, the raw data available from data providers is virtually impossible to analyze in its original form. As a result, the enormous challenge of reviewing and manipulating the raw data creates a substantial barrier to conducting quality observational research.

The Real Miracle in Research

Borrowing a thought from Sydney Harris, the figure below is how most researchers view the process of structuring their data for research.

Breaking it Down

At a simple level, the manipulation of data can be broken into two tasks. The first is to organize the raw data by cataloging its contents and the relationships among the tables, examining it for completeness and correctness, and mapping it to a structure suitable to research. (For regulatory purposes, the organization process itself needs to be documented as well.) The second task is identifying the relevant cohort to study, extracting the relevant records, and creating analysis-ready datasets. Once this two-step data pipeline is implemented, researchers can then conduct the analyses of interest.

The Starting Point

For the first task, the only option for most researchers is to create an ad hoc data structure specific to the research project at hand. While adequate for one-off studies, this approach is prone to error, difficult to modify, and impossible to scale. More flexible approaches to data organization involve either writing a query translation layer or moving data into a common data model structure. Both approaches map the original data structure into a new, more efficient structure, but the common data model approach can be manipulated more efficiently, reliably, and repeatably by modern database platforms.

The Problem and the Solution

This highlights the fundamental problem for all researchers: there are no useful off-the-shelf, open-source toolkits that help researchers process raw healthcare data into a usable form for research. Hence, we are working to build such a toolkit and to make it available for others.

With Medicare data moving to the Virtual Research Data Center (VRDC) exclusively, researchers are being forced into a new way of conducting research. Some have expressed concerns that research could take longer, or that quality control could suffer. But these concerns simply highlight the long-standing struggle to write effective code for conducting observational research.

Part of the problem is that observational research is an insular process. To ensure patient privacy, data can not easily be shared. This makes collaboration difficult. And because 60-80% of a research project is simply wrangling raw data to create analysis-ready datasets, data management is labor-intensive.

To complicate matters further, most data management practices are still stuck in the technology of the 1990’s. One must look no further than to the ubiquity of SAS or Stata for large-scale data management, the minimal use of relational databases and SQL, and the use of double data programming instead of modern approaches for code testing and version control. Lastly, many researchers conflate cohort creation with data analysis by trying to use the same software and coding practices for both parts of a project.

In short, the limited adoption of appropriate data management practices has led to a dead-end in the era of data enclaves like the VRDC.

Where Do We Go?

With the advent of better tools for working with raw data, observational research needs to take advantage of fit-for-purpose technologies and apply them in appropriate ways. Standard data manipulation languages (e.g., SQL), relational databases (e.g., PostgreSQL), and data platforms (e.g., Databricks) make it efficient to extract analysis-ready datasets from terabytes of raw data.

Although tools exist to improve the efficiency of observational research, the solution is not as simple as changing software. The entire process needs to re-defined. And therein lies the primary issue – most observational researchers are not well-versed in the software development best practices that are required to move the field forward. Even more recent approaches to rethinking observational research (e.g., OMOP/OHDSI, FDA Sentinel, PCORnet, i2b2, etc.) still fail to address many of these challenges properly.

Software Development – Really?

By necessity, observational researchers dabble in software development as a byproduct of writing the code for a project. But code development is limited by the narrow scope of the project at hand. The iterative process of adapting code for each new project and evaluating the output to ensure correctness is second-nature to researchers. But the prospect of working on the VRDC exposes the inefficiencies and limitations of this approach.

The root cause is that researchers are increasingly out of their element when using modern software (e.g., Databricks on the VRDC). Writing reusable and testable code for modern data platforms requires an understanding of software libraries, programming paradigms, and testing frameworks. However, the fundamentals of these tools are not part of the training for most researchers and statistical programmers.

To move research forward, researchers need to reconsider their approach and adopt a software development mindset. Viewed through this lens, cohort-building can be reduced to a set of repetitive tasks that are identical across different observational study designs. In other words, researchers need to think in terms of building a “data pipeline”.

A data pipeline requires clear specifications for the inputs and outputs for each part of the process. There are many benefits to this approach. Importantly, this includes creating specifications for organizing the raw data. This can be accomplished in a variety of ways including using data models, database views, or other approaches. The key point related to the VRDC is that, once the structure of the raw data is defined, there is no need to touch the data to specify how to build the cohort and to create the analysis-ready datasets.

By analogy, it is like using Google or Apple Maps. Once we have a database of GPS coordinates for all roads and addresses, and an understanding of basic driving rules, we can write software to create detailed, optimized directions without ever having to get in the car.

How Will That Help with Platforms like the VRDC?

Working on the VRDC means that researchers need to develop their cohort-creation code outside the VRDC (see this blog post). Once researchers can generate the code to create their analysis-ready datasets without needing simultaneous access to the raw data, working on an enclave like the VRDC becomes easier, faster, and cheaper. In fact, this kind of “offline” approach enables researchers to work on data in a consistent fashion anywhere it is stored – a powerful idea.

That Is Not Realistic. Or Is It?

At this point it would be reasonable to say, “but surely everyone can’t build their own validated software stack for the few studies they might do in a year; that is neither possible nor efficient.” I can tell you that, as a very small company, we have built exactly this system so we know it is possible. I can tell you that, having learned how to work “offline”, we are excited to work on platforms like the VRDC. And I can tell you that, as I write this, we are working on some VRDC-related demonstration projects to prove that this approach works. So, yes, it can be done. But I can also tell you that it involved a lot of hard lessons and about 10 years of work.

That’s Nice. But How Does that Help Anyone Else?

One of our core principles is to share our work with others. The key components of our research process have been publicly available via GitHub since they were created. This includes the Generalized Data Model, which defines how we organize raw data, and ConceptQL, the language we built to create, store, and share algorithms. We also made our software, Jigsaw, and our algorithm library publicly readable as of last year (see this blog post). And we are actively searching for a way make Jigsaw usable by others to solve problems like working on the VRDC (see this blog post).

Conclusion

Observational researchers need to think outside the box that has enclosed observational research for the last 40 years. The tools are available, and we have laid much of the groundwork. We will continue to share our progress and help others conduct quality research in an evolving data landscape. If you have any questions, or if you want to collaborate, feel free to contact us.

Last year we made our algorithm library publicly available. In 2024, we are working to make our entire Jigsaw application freely accessible for creating and sharing protocols, creating and sharing algorithms, and generating the code to create analysis-ready datasets from observational data.

Doesn’t This Exist Already?

Solutions for streamlining observational research fall into one of the following approaches:

• Building an internal repository of implementation code
• Implementing and supporting an open-source software system internally
• Licensing access to a commercial platform

However, these options also have important limitations:

• In-house solutions are generally software-specific and have limited version control
• Open-source platforms require staff for installation, support, and updates
• Commercial platforms can be expensive black boxes with little visibility into underlying processes being implemented
• No solution allows researchers to collaborate across institutions that use different approaches.

In short, despite improvements in software capabilities, researchers still reside in their own silos, unable to collaborate efficiently with one another.

What is the Alternative?

The ideal solution is a freely accessible, cloud-based software application.

As an analogy, think about GitHub. For anyone unfamiliar with GitHub, the following is the one-sentence Wikipedia description:

GitHub, Inc. is an AI-powered developer platform that allows developers to create, store, manage and share their code.”

So, imagine if we alter that slightly for Jigsaw to read as follows:

Jigsaw is a cloud-based platform that allows observational researchers to create, store, manage and share their research protocols and to generate the code for implementing them against data.”

Conclusion

Despite thousands of hours and millions of dollars being invested in potential solutions, the fundamental problem remains – research methods are still too fragmented. It is about time we solved this problem. We think Jigsaw is the solution.

If an organization can store all the data it needs on its own servers, it can choose almost any software package or platform to manipulate the data for research purposes. But what happens if the organization isn’t allowed to store the data on its own servers? For example, most people who want to access the full Medicare data must do so within the CMS Virtual Research Data Center (VRDC) using either SAS or Databricks. Similarly, some countries limit their data to servers only accessible within their own country, and some commercial data providers limit access to their own virtual data centers. While this can improve data security and reduce patient privacy risks, it creates logistical challenges for researchers.

The Solution?

The solution is to bring project-specific software code to the data behind the firewall. But how does one write code for a project without touching the actual data? There are at least two options for solving the problem: synthetic data and code-generation software.

Synthetic Data

In theory, researchers can craft their project code against a synthetic, but realistic, copy of the data. This presupposes that someone will create and maintain an accessible, usable, privacy-protecting synthetic dataset for each data source of interest. Even then, working with large, unwieldy data can be a challenge. Imagine working with a synthetic version of the full Medicare data – that isn’t a job for anyone without access to suitable computing resources. All-in-all, synthetic data represents a possible solution, but it isn’t particularly efficient because it still relies on hand-writing code for manipulating potentially large datasets.

Code-Generation Software: Jigsaw

In our opinion, the better option is to use Jigsaw. It may not be obvious, but Jigsaw doesn’t need to touch the actual raw data to do its job. How is that possible? Jigsaw’s job is to write all the SQL queries for creating an analysis-ready dataset from the raw, organized data. As long as the data is organized using a known data model, Jigsaw can write a script containing the required SQL queries. A user can then run the script on a server containing the organized data, and the script can save the analysis-ready data sets on the same server in a location specified by the user.

Because Jigsaw itself can be Cloud-based, researchers can collaborate on the specification of the analysis-ready data from anywhere. The protocol and its algorithms remain publicly available and shareable. As before, all algorithms are explicitly documented in the protocol summary document that can be generated (and also shared).

Or Both?

In some scenarios, it could make sense to use both approaches. One could use Jigaw to create an analysis-ready dataset, and then create a synthetic version for developing the analyses themselves.

Challenges

In order for this work, the data on the server needs to be reorganized into a data model. We strongly prefer the Generalized Data Model because it brings together some of the best ideas of the i2b2 and OMOP/OHDSI data models. By this, we referring to combining the i2b2 idea of storing all clinical information in one central “fact” table combined with the OMOP/OHDSI vocabularies.

Once a transformation to a data model is created, it is relatively easy to share the code for others to transform their raw data into a data model. As of early 2024, we have code for the current SEER-Medicare linked data. Next, we will create the transformation code for the Medicare data on the VRDC. After that, we are open to suggestions.

Conclusion

Working on other data platforms doesn’t have to be challenging. We think Jigsaw can be a powerful tool for working with observational data wherever it is without compromising data security or patient privacy.

No Cluster Needed

Perhaps it was my bias from working with Apache Impala a few years back, but I just assumed that Spark was going to need Hadoop set up on a cluster of servers. I didn’t want to spend my time getting all that set up just to play around with Spark, so I never bothered with it before.

Turns out, Spark has a rather robust single-machine setup. Even better, there’s an R library that took care of all the set up for me.

sparklyr Makes Spark Simple

The R package sparklyr made my foray into Spark dead simple. The package happily installed Spark for me and provided me functions to easily start and stop a Spark instance from within my R scripts.

Pro tip: by default sparklyr limits Spark to a single core when it starts up an instance. You can change to multiple cores pretty easily and it makes a world of difference in terms of performance.

dplyr and Spark Is a Powerful Combination

sparklyr gave me access to the tables I loaded into Spark. dplyr gave me the ability to manipulate and query those tables via dbplyr.

dplyr is amazing. Rather than hand-writing Spark SQL, dplyr provides a set of functions that allowed me to join tables, add where clauses, and manipulate the columns returned from Spark.

Great Performance

My project was to explore replacing an existing part of our data pipeline. Using Spark, our processing time went from days to hours.

More Spark in the Future

After this successful venture into Spark territory, I’m pretty sure I’ll be employing Spark in future projects.

A few years later, the Arrow package for R came out and it had support for, much to my surprise, Parquet. Suddenly Parquet seemed to be freed from HDFS and could plunk huge swaths of data into cute little folders directly in my computer’s filesystem. What a powerful tool I suddenly had. It was a great alternative to all the other dirty data formats I dealt with in the past.

Since then, we’ve standardized our data pipeline tools on Parquet. It’s a great format for data storage and transfer. It has a true standard for storing data and their types, unlike CSV. It compresses data down very nicely, leaving the files smaller than most other proprietary or CSV formats we’ve seen. It is fast to read and write, making it a great intermediate storage for our data pipelines. And, thanks to Arrow, all the languages we use in our tools have first-class support for working with Parquet. Also, shout out to DuckDB for being the SQLite of Parquet.

This last winter, one of our data vendors actually offered to send us data in Parquet format. It was amazing. AMAZING. We downloaded the files from them and within a minute I was able to query the data, get counts of rows, types of columns, and confirm that we had, indeed, received all the records we expected. It was unlike any other data ingestion experience I’ve ever had. No unzipping, no CSV tools, no proprietary formats. These pre-ETL steps were almost completely unnecessary and we could move right into transformation of the data into GDM

When I first encountered it, I never thought I’d be such a fan of Parquet, but I now sincerely hope that it continues to become the standard for transferring claims data between vendors and researchers.

Background

From inside any algorithm in the library, people can create a stand-alone algorithm summary. This is a stand-alone webpage that can be downloaded and saved like any other document (click the green “Download” button at the top). Importantly, the webpage also contains a CSV file with all algorithm codes. The CSV file can be accessed via link at the bottom of the summary.

To create the summary from inside any algorithm in the library, simply click the “Action” button at the top, and select “Create Algorithm Summary”.

Two Algorithm Sharing Options

Link to the algorithm within the Jigsaw algorithm library

The first option is a link to the algorithm in the Jigsaw library itself. For example, below is a link to an algorithm in the library:

https://public.jigsaw.io/algorithms/646c44ce-be1a-4f04-b4e6-ecc5d7b44e13

From that link, people can view algorithm attributes, a full-screen algorithm diagram, and the ConceptQL statement that stores all the algorithm specifications. People can even generate an SQL implementation of the algorithm from within the diagram by choosing “Source” at the top of the screen.

Link directly to the stand-alone algorithm summary

The second option is a direct link to the stand-alone algorithm summary page. It is the same URL as the algorithm in the Jigsaw library except it ends in “/summary”. See the link below for an example:

https://public.jigsaw.io/algorithms/646c44ce-be1a-4f04-b4e6-ecc5d7b44e13/summary

Bonus points – Protocols Can Also Be Shared!

Pretty much everything above applies to protocols too. Not only can an entire protocol be shared from within Jigsaw’s protocol builder, but a stand-alone summary document can also be created, shared, and downloaded. The entire protocol specification, including the enrollment criteria, algorithms and how they were defined and used (e.g., inclusion, exclusion, outcome, etc.) can be created as a stand-alone, downloadable web page. For example, see the link below:

https://public.jigsaw.io/studies/b522827a-0ea3-4d22-add7-849c072a5717/summary

Why is this useful?

The reason we designed these features was to allow for sharing and for documentation purposes. Imagine being able to see someone else’s algorithms and protocols – the exact steps they followed to create cohorts and analysis data sets. Government agencies, academic researchers, and commercial companies could share their methods without having to share the actual code or data.

Also, simply having all the detailed specifications in one place allows us to retain them as an appendix to technical reports, and saves time and energy in writing study publications.

Why put this out there?

We have spent many years struggling with the same issues that all researchers face when trying to extract information from healthcare data. Algorithms are hard to find, poorly documented and difficult to implement consistently. It is time to solve the problem.

Who is funding this?

This project is completely self-funded. Please keep in mind that we have limited resources, and we may not be able to accommodate requests. It turns out that it is surprisingly challenging to make something publicly and freely available, so we appreciate your understanding.

Collaboration

If any organization is interested in collaborating with us on algorithms, we welcome it – even from organizations that might be considered “competitors”. Algorithms should be freely available and compatible with other systems than our own. We are willing to see if we can make them work with other systems, even proprietary ones.

Feedback

We welcome suggestions. We are happy to consider algorithm modifications if an algorithm can be improved in any way. We would like our algorithms to be usable by anyone. Along these lines, we are willing to consider adding other algorithms to our library, as long they can be made public.

Browser requirements

The algorithm library is designed and tested against Google Chrome because of its popularity. Other browsers may work, but we don’t provide cross-browser support at this time.

But wait, there’s more!

In addition to the high-level details above, we also provide more detailed technical information and limitations below, as well as an example algorithm.

Technical details

Limitations

Example algorithm

Technical Details

What is an algorithm library?

Like most libraries, the Jigsaw algorithm library stores algorithms, facilitates searching for algorithms, and contains tools for creating, editing, and versioning algorithms. The library enables algorithm sharing in two ways: as a stand-alone HTML file downloaded from the website, or via the algorithm’s unique URL.

How to use the Jigsaw algorithm library

Anyone interested can simply navigate to the algorithm section of the Jigsaw application to find algorithms. Users can then search for algorithms by different characteristics. Clicking on any algorithm’s name opens it, and the user is presented with algorithm attributes as well as a diagram of the algorithm specification. Users can click on the diagram to get more detailed information about the specifications.

We cover some of the basics below, and we will provide more granular instructions in the future.

What is an algorithm?

An algorithm has two parts – attributes and specifications.

Algorithm attributes include all of the information that makes algorithms friendly to researchers. This includes titles, tags, documentation, variable names, and even evidence of validity (e.g., sensitivity and specificity).

Algorithm specifications are comprised of the code(s) and operator(s) that are needed to identify information for an analysis dataset. The most basic operation of an algorithm is to “select” records with the code(s) of interest. This means that the simplest possible algorithm specification is simply to select all records using a single code (see Standardization for more on codes). However, some algorithms can have thousands of codes. Algorithms can also require operations related to the context in which the code was used (provenance), have temporal specifications (before, after, within, etc.), require relationships to other codes (co-reported), use filters (first, last, etc.), or specify other operations.

Sometimes algorithm specifications require many operations in sequence. For these use cases, we create special operators to make them easier to implement. The best example is an operator that looks for a code that occurs at least 1 time in the inpatient setting or at least 2 times in the outpatient setting within specified time intervals (sometimes referred to as a “one inpatient or two outpatient” operation). These operators make creating algorithm specifications much simpler.

What does an algorithm specification produce?

An algorithm specification is a representation of a table that is created and, potentially, modified. When implemented against data, every algorithm starts with a selection operator to create the first table based on the codes in the specification. This table can then be passed to subsequent operators for additional modification. The output of an algorithm is a table that includes all of the relevant records for each person as modified by the specified operations. As an example, a specification could first select all records with at least one diabetes code based on a list of diabetes codes, and then filter these records to retain only the first code for each person.

The core of each table includes a person id, details about the exact table and record that was returned, the start and end dates, the vocabulary (e.g., ICD-10-CM), and the specific code. Additional columns are dynamically added for selected types of records (e.g., prescription records will add additional columns specific to prescriptions, like dosing information).

How algorithms operate

We have defined all algorithm operations that are commonly used for research using ConceptQL (see ConceptQL blog post). Within the algorithm library, there is a hyperlink from each operator to our ConceptQL specification in Github that provides a more detailed description of how it functions.

Implicitly, all algorithm operations are conducted by person since almost all research is conducted on cohorts of people. All algorithms are read left to right, and only one table is ever produced at the end of any operation. This is true even if multiple tables are used as inputs to an operation. For algorithms with two sides to them, the left side takes the input of the previous operation and modifies it using information from the right side. Tables on the right side are never passed through and included in the output.

If the above doesn’t make any sense, don’t worry. We will provide more details about how algorithms operate in a future post.

Visual representation

We have found that a visual representation of an algorithm specification is extremely helpful for communicating it to others. As a result, an algorithm is specified using a visual algorithm builder that generates ConceptQL statements and encapsulates the algorithm specification. See below for an example.

Algorithm scope

Do we have every possible operation that might ever be needed? Probably not. Hence, it is possible to extend Jigsaw with new operators as the need arises. However, we expect the need for new operators to be relatively rare. Many times, the desired output requires some creativity using existing operations.

Alternatively, it may be that the desired operation is really a protocol specification, and not an algorithm specification. It is beyond this blog post to define the difference between an algorithm specification and a protocol specification precisely. However, in general, an algorithm specifies the variable of interest, and the protocol defines the context in which the algorithm operates. The separation between an algorithm and a protocol ensures that an algorithm is as reusable as possible.

To make this a bit more concrete, an algorithm might define a myocardial infarction. Or it might define diabetes diagnosed at least 30 days before a myocardial infarction. However, a myocardial infarction within 30 days of the index date, or as an inclusion or exclusion criterion, or as a censoring variable – those are ways of using a myocardial infarction algorithm in the context of a protocol. The same idea applies with clean periods or lookback periods – those are protocol-specific ideas that are not part of an algorithm’s scope.

As a side note, users may see some algorithms in our library that violate this principle. This is because our thinking about the line between algorithms and protocols has evolved over the years. As a result, we have added features to our protocol builder to help keep the distinction as clear as possible. But since we don’t readily remove algorithms we have used, these algorithms remain in the algorithm library.

It isn’t clear how algorithms are used to create analysis data sets

Protocols, not algorithms, specify the analysis datasets that need to be created. Protocols specify the index date, inclusion and exclusion criteria, baseline variables, lookback periods, clean periods, date restrictions, and outcome variables. Protocols use algorithms to specify each variable within the context of a protocol. It is much like assembling a jigsaw puzzle (protocol) using pieces (algorithms). Hence, our software is named “Jigsaw”.

As a simple example, the specification for myocardial infarction can be identical regardless of whether it is needed as a baseline variable, index event, or outcome. However, different records will be selected depending on which context required.

To illustrate how algorithms fit into protocols, we have made our entire Jigsaw application available, including our protocol builder, albeit in read-only form. There is a demonstration protocol available within the application.

Limitations

Disclaimer

While we built Jigsaw and the algorithms carefully, we make no guarantees of correctness or suitability for a specific study. Jigsaw is a tool to make research easier and more transparent; however, it still requires that users fully understand all aspects of the research they are conducting, their data source(s), and the limitations of real-world data.

Is every algorithm ready for research?

The algorithm library contains over 1,000 algorithms and spans many years of development and growth. Therefore, not every algorithm is ready to be used in its current state. Some were created as examples, some were created prior to ICD-10-CM data becoming available, and some were created or updated very recently. Most algorithms are classified as “Draft” even though they have been extensively used. Some “Finalized” algorithms should probably be updated. As mentioned above, this is a work in progress and we only update algorithms as we reuse them. We suggest looking at the date of the most recent update to get a sense of how current an algorithm is. Users should do their own due-diligence before using any algorithm.

Is an algorithm specific to a data model or software system?

The algorithm library is agnostic to both the data structure and the software used for implementation. The algorithm simply captures the specifications for an algorithm. For example, an algorithm can encapsulate the idea that we are looking for a “diabetes diagnosis at least 30 days before a myocardial infarction”.

As such, it is not written in SQL, SAS, R, Python, or any language that can be used directly on data. Instead, it is specified in JSON using the ConceptQL language. Jigsaw uses the specification to dynamically generate SQL and implement the algorithm against data. In theory, ConceptQL could generate something other than SQL, but practically, SQL is so widely available there has been no reason to consider other options. This approach allows us to generate SQL that is specific to the idiosyncrasies of different database implementations of SQL. For example, we can generate SQL specific to PostgreSQL, Impala, SQLite or most other database systems that follow SQL standards.

Furthermore, we can generate SQL that works directly against different data models, like the Generalized Data Model (GDM), OMOP, PCORnet, and Sentinel. (Note that support for OMOP, PCORnet, and Sentinel is not currently implemented because we have not had a use case for them.)

To see what the SQL code looks like for an algorithm, users can dynamically generate the necessary SQL from inside the algorithm diagram. The SQL is specific to the Generalized Data Model, and it produces PostgreSQL queries. See below for a complete example of an algorithm that includes the diagram, attributes, ConceptQL JSON statement, and the resulting SQL.

Why not make the entire application available?

While Jigsaw is copyrighted and remains our intellectual property, our hope is that we can make its full functionality available to other researchers. To do that, we must make ETL easier first, which is not a trivial issue. We have made progress on that front as well, and are willing to collaborate with organizations interested in adopting the Generalized Data Model and/or Jigsaw.

Example Algorithm

Below is an example algorithm that identifies diabetes diagnoses coming at least 30 days prior to a myocardial infarction. Both ICD-9-CM and ICD-10-CM codes are used to identify diabetes and myocardial infarction. Note that this is an example and may not represent the correct codes or a clinically meaningful event.

Algorithm diagram

Algorithm attributes

ConceptQL

Below is the ConceptQL statement that corresponds to the diagram above.

[
  "before",
  {
    "left": [
      "union",
      [
        "icd9cm",
        "250.00",
        "250.01",
        "250.02",
        "250.03",
        "250.10",
        "250.11",
        "250.12",
        "250.13",
        "250.20",
        "250.21",
        "250.22",
        "250.23",
        "250.30",
        "250.31",
        "250.32",
        "250.33",
        "250.80",
        "250.81",
        "250.82",
        "250.83",
        "250.90",
        "250.91",
        "250.92",
        "250.93"
      ],
      [
        "icd10cm",
        "E10.10",
        "E10.11",
        "E10.61",
        "E10.61",
        "E10.62",
        "E10.62",
        "E10.62",
        "E10.62",
        "E10.63",
        "E10.63",
        "E10.64",
        "E10.64",
        "E10.65",
        "E10.69",
        "E10.8",
        "E10.9",
        "E11.00",
        "E11.01",
        "E11.10",
        "E11.11",
        "E11.61",
        "E11.61",
        "E11.62",
        "E11.62",
        "E11.62",
        "E11.62",
        "E11.63",
        "E11.63",
        "E11.64",
        "E11.64",
        "E11.65",
        "E11.69",
        "E11.8",
        "E11.9",
        "E13.00",
        "E13.01",
        "E13.10",
        "E13.11",
        "E13.61",
        "E13.61",
        "E13.62",
        "E13.62",
        "E13.62",
        "E13.62",
        "E13.63",
        "E13.63",
        "E13.64",
        "E13.64",
        "E13.65",
        "E13.69",
        "E13.8",
        "E13.9"
      ]
    ],
    "right": [
      "union",
      [
        "icd9cm",
        "410.01",
        "410.11",
        "410.21",
        "410.31",
        "410.41",
        "410.51",
        "410.61",
        "410.71",
        "410.81",
        "410.91"
      ],
      [
        "icd10cm",
        "I21.01",
        "I21.02",
        "I21.09",
        "I21.11",
        "I21.19",
        "I21.21",
        "I21.29",
        "I21.3",
        "I21.4",
        "I21.9",
        "I21.A1",
        "I21.A9",
        "I22.0",
        "I22.1",
        "I22.2",
        "I22.8",
        "I22.9"
      ]
    ],
    "at_least": "30d"
  }
]

SQL

The code below is generated from the ConceptQL specification. This particular SQL code will create an output table from data stored in a PostgreSQL relational database based on the Generalized Data Model schema.

SELECT
    *
FROM ( WITH "before_22_3_24a2b6fb9b0f37c3735c654d1ffd3cab" AS MATERIALIZED (
        SELECT
            *
        FROM (
            SELECT
                "person_id",
                "criterion_id",
                "criterion_table",
                "criterion_domain",
                "start_date",
                "end_date",
                "source_value",
                "source_vocabulary_id",
                "label"
            FROM (
                SELECT
                    *
                FROM (
                    SELECT
                        *
                    FROM (
                        SELECT
                            *
                        FROM (
                            WITH "union_22_2_64ac7676c653ae02d50931398ee95903" AS MATERIALIZED (
                                SELECT
                                    *
                                FROM (
                                    SELECT
                                        "person_id",
                                        "criterion_id",
                                        "criterion_table",
                                        "criterion_domain",
                                        "start_date",
                                        "end_date",
                                        "source_value",
                                        "source_vocabulary_id",
                                        "label"
                                    FROM (
                                        SELECT
                                            "person_id",
                                            "criterion_id",
                                            "criterion_table",
                                            "criterion_domain",
                                            "start_date",
                                            "end_date",
                                            "source_value",
                                            "source_vocabulary_id",
                                            "label"
                                        FROM (
                                            SELECT
                                                *
                                            FROM (
                                                SELECT
                                                    "patient_id" AS "person_id",
                                                    "id" AS "criterion_id",
                                                    CAST(
                                                        'clinical_codes' AS text
) AS "criterion_table",
                                                    CAST(
                                                        'condition_occurrence' AS text
) AS "criterion_domain",
                                                    "start_date",
                                                    "end_date",
                                                    CAST(
                                                        "clinical_code_source_value" AS text
) AS "source_value",
                                                    CAST(
                                                        "clinical_code_vocabulary_id" AS text
) AS "source_vocabulary_id",
                                                    CAST(
                                                        NULL AS text
) AS "label"
                                                FROM
                                                    "clinical_codes"
                                                WHERE ((
                                                        "clinical_code_concept_id" IN (
                                                            SELECT
                                                                "id"
                                                            FROM
                                                                "concepts"
                                                            WHERE ((
                                                                    "vocabulary_id" = 'ICD9CM'
)
                                                                AND (
                                                                    "concept_code" IN (
                                                                        '250.00', '250.01', '250.02', '250.03', '250.10', '250.11', '250.12', '250.13', '250.20', '250.21', '250.22', '250.23', '250.30', '250.31', '250.32', '250.33', '250.80', '250.81', '250.82', '250.83', '250.90', '250.91', '250.92', '250.93'
)
)
)
)
)
                                                    OR (
                                                        "clinical_code_concept_id" IN (
                                                            SELECT
                                                                "id"
                                                            FROM
                                                                "concepts"
                                                            WHERE ((
                                                                    "vocabulary_id" = 'ICD10CM'
)
                                                                AND (
                                                                    "concept_code" IN (
                                                                        'E10.10', 'E10.11', 'E10.61', 'E10.61', 'E10.62', 'E10.62', 'E10.62', 'E10.62', 'E10.63', 'E10.63', 'E10.64', 'E10.64', 'E10.65', 'E10.69', 'E10.8', 'E10.9', 'E11.00', 'E11.01', 'E11.10', 'E11.11', 'E11.61', 'E11.61', 'E11.62', 'E11.62', 'E11.62', 'E11.62', 'E11.63', 'E11.63', 'E11.64', 'E11.64', 'E11.65', 'E11.69', 'E11.8', 'E11.9', 'E13.00', 'E13.01', 'E13.10', 'E13.11', 'E13.61', 'E13.61', 'E13.62', 'E13.62', 'E13.62', 'E13.62', 'E13.63', 'E13.63', 'E13.64', 'E13.64', 'E13.65', 'E13.69', 'E13.8', 'E13.9'
)
)
)
)
)
) -- #<ConceptQL::Operator::icd9cm @arguments=["250.00", "250.01", "250.02", "250.03", "250.10", "250.11", "250.12", "250.13", "250.20", "250.21", "250.22", "250.23", "250.30", "250.31", "250.32", "250.33", "250.80", "250.81", "250.82", "250.83", "250.90", "250.91", "250.92", "250.93"]>
) AS "t1"
) AS "t1"
) AS "t1" -- #<ConceptQL::Operators::Union @upstreams=[#<ConceptQL::Operator::icd9cm @arguments=["250.00", "250.01", "250.02", "250.03", "250.10", "250.11", "250.12", "250.13", "250.20", "250.21", "250.22", "250.23", "250.30", "250.31", "250.32", "250.33", "250.80", "250.81", "250.82", "250.83", "250.90", "250.91", "250.92", "250.93"]>, #<ConceptQL::Operator::icd10cm @arguments=["E10.10", "E10.11", "E10.61", "E10.61", "E10.62", "E10.62", "E10.62", "E10.62", "E10.63", "E10.63", "E10.64", "E10.64", "E10.65", "E10.69", "E10.8", "E10.9", "E11.00", "E11.01", "E11.10", "E11.11", "E11.61", "E11.61", "E11.62", "E11.62", "E11.62", "E11.62", "E11.63", "E11.63", "E11.64", "E11.64", "E11.65", "E11.69", "E11.8", "E11.9", "E13.00", "E13.01", "E13.10", "E13.11", "E13.61", "E13.61", "E13.62", "E13.62", "E13.62", "E13.62", "E13.63", "E13.63", "E13.64", "E13.64", "E13.65", "E13.69", "E13.8", "E13.9"]>]>
) AS "t1"
)
                            SELECT
                                *
                            FROM
                                "union_22_2_64ac7676c653ae02d50931398ee95903"
) AS "l"
) AS "l"
                    WHERE (
                        EXISTS (
                            SELECT
                                1
                            FROM (
                                SELECT
                                    *
                                FROM (
                                    SELECT
                                        "person_id",
                                        max(
                                            "start_date"
) AS "start_date"
                                    FROM (
                                        WITH "union_22_1_d3d7bbd424bca0fe3b3f966d4ee80692" AS MATERIALIZED (
                                            SELECT
                                                *
                                            FROM (
                                                SELECT
                                                    "person_id",
                                                    "criterion_id",
                                                    "criterion_table",
                                                    "criterion_domain",
                                                    "start_date",
                                                    "end_date",
                                                    "source_value",
                                                    "source_vocabulary_id",
                                                    "label"
                                                FROM (
                                                    SELECT
                                                        "person_id",
                                                        "criterion_id",
                                                        "criterion_table",
                                                        "criterion_domain",
                                                        "start_date",
                                                        "end_date",
                                                        "source_value",
                                                        "source_vocabulary_id",
                                                        "label"
                                                    FROM (
                                                        SELECT
                                                            *
                                                        FROM (
                                                            SELECT
                                                                "patient_id" AS "person_id",
                                                                "id" AS "criterion_id",
                                                                CAST(
                                                                    'clinical_codes' AS text
) AS "criterion_table",
                                                                CAST(
                                                                    'condition_occurrence' AS text
) AS "criterion_domain",
                                                                "start_date",
                                                                "end_date",
                                                                CAST(
                                                                    "clinical_code_source_value" AS text
) AS "source_value",
                                                                CAST(
                                                                    "clinical_code_vocabulary_id" AS text
) AS "source_vocabulary_id",
                                                                CAST(
                                                                    NULL AS text
) AS "label"
                                                            FROM
                                                                "clinical_codes"
                                                            WHERE ((
                                                                    "clinical_code_concept_id" IN (
                                                                        SELECT
                                                                            "id"
                                                                        FROM
                                                                            "concepts"
                                                                        WHERE ((
                                                                                "vocabulary_id" = 'ICD9CM'
)
                                                                            AND (
                                                                                "concept_code" IN (
                                                                                    '410.01', '410.11', '410.21', '410.31', '410.41', '410.51', '410.61', '410.71', '410.81', '410.91'
)
)
)
)
)
                                                                OR (
                                                                    "clinical_code_concept_id" IN (
                                                                        SELECT
                                                                            "id"
                                                                        FROM
                                                                            "concepts"
                                                                        WHERE ((
                                                                                "vocabulary_id" = 'ICD10CM'
)
                                                                            AND (
                                                                                "concept_code" IN (
                                                                                    'I21.01', 'I21.02', 'I21.09', 'I21.11', 'I21.19', 'I21.21', 'I21.29', 'I21.3', 'I21.4', 'I21.9', 'I21.A1', 'I21.A9', 'I22.0', 'I22.1', 'I22.2', 'I22.8', 'I22.9'
)
)
)
)
)
) -- #<ConceptQL::Operator::icd9cm @arguments=["410.01", "410.11", "410.21", "410.31", "410.41", "410.51", "410.61", "410.71", "410.81", "410.91"]>
) AS "t1"
) AS "t1"
) AS "t1" -- #<ConceptQL::Operators::Union @upstreams=[#<ConceptQL::Operator::icd9cm @arguments=["410.01", "410.11", "410.21", "410.31", "410.41", "410.51", "410.61", "410.71", "410.81", "410.91"]>, #<ConceptQL::Operator::icd10cm @arguments=["I21.01", "I21.02", "I21.09", "I21.11", "I21.19", "I21.21", "I21.29", "I21.3", "I21.4", "I21.9", "I21.A1", "I21.A9", "I22.0", "I22.1", "I22.2", "I22.8", "I22.9"]>]>
) AS "t1"
)
                                        SELECT
                                            *
                                        FROM
                                            "union_22_1_d3d7bbd424bca0fe3b3f966d4ee80692"
) AS "t1"
                                    GROUP BY
                                        "person_id"
) AS "r"
) AS "r"
                            WHERE ((
                                    "l"."person_id" = "r"."person_id"
)
                                AND (
                                    "l"."end_date" < CAST((
                                            CAST(
                                                "r"."start_date" AS timestamp
) + make_interval(
                                                days := - 30
)
) AS date
)
)
)
)
)
) AS "t1"
) AS "t1" -- #<ConceptQL::Operators::Before @upstreams={:left=> #<ConceptQL::Operators::Union @upstreams=[#<ConceptQL::Operator::icd9cm @arguments=["250.00", "250.01", "250.02", "250.03", "250.10", "250.11", "250.12", "250.13", "250.20", "250.21", "250.22", "250.23", "250.30", "250.31", "250.32", "250.33", "250.80", "250.81", "250.82", "250.83", "250.90", "250.91", "250.92", "250.93"]>, #<ConceptQL::Operator::icd10cm @arguments=["E10.10", "E10.11", "E10.61", "E10.61", "E10.62", "E10.62", "E10.62", "E10.62", "E10.63", "E10.63", "E10.64", "E10.64", "E10.65", "E10.69", "E10.8", "E10.9", "E11.00", "E11.01", "E11.10", "E11.11", "E11.61", "E11.61", "E11.62", "E11.62", "E11.62", "E11.62", "E11.63", "E11.63", "E11.64", "E11.64", "E11.65", "E11.69", "E11.8", "E11.9", "E13.00", "E13.01", "E13.10", "E13.11", "E13.61", "E13.61", "E13.62", "E13.62", "E13.62", "E13.62", "E13.63", "E13.63", "E13.64", "E13.64", "E13.65", "E13.69", "E13.8", "E13.9"]>]>, :right=> #<ConceptQL::Operators::Union @upstreams=[#<ConceptQL::Operator::icd9cm @arguments=["410.01", "410.11", "410.21", "410.31", "410.41", "410.51", "410.61", "410.71", "410.81", "410.91"]>, #<ConceptQL::Operator::icd10cm @arguments=["I21.01", "I21.02", "I21.09", "I21.11", "I21.19", "I21.21", "I21.29", "I21.3", "I21.4", "I21.9", "I21.A1", "I21.A9", "I22.0", "I22.1", "I22.2", "I22.8", "I22.9"]>]>}>
) AS "t1"
)
    SELECT
        *
    FROM
        "before_22_3_24a2b6fb9b0f37c3735c654d1ffd3cab") AS "t1"