data-definition-engine

Data Definition Engine

under development License: MIT License: CC BY 4.0

The Data Definition Engine (DDE) is an open-source tool that generates CDISC study artifacts — Define-XML and ODM Case Report Forms (CRFs) — from a machine-readable clinical trial protocol expressed in the Unified Study Definitions Model (USDM). It is developed as part of the CDISC 360i Program by the Define-XML/CRF Generation Project Team.

USDM is a TransCelerate / CDISC standard that represents a clinical trial’s complete protocol in a machine-readable JSON format: objectives, arms, visits, eligibility criteria, the schedule of activities, and the Biomedical Concepts each activity collects. It was co-developed through a formal partnership between CDISC and TransCelerate BioPharma as part of TransCelerate’s Digital Data Flow (DDF) initiative, and it is the DDE’s input — the “source of truth” for the study.

The DDE reads USDM 4.0 from both known producers, the CDISC USDM Excel-to-JSON tool (E2J) and the SoA Workbench. Both declare usdmVersion 4.0, so the loaders tell them apart by the shape of the data rather than by version.

Table of Contents

Background

Clinical trials submitted to regulatory agencies (such as the FDA) must include standardized metadata files that describe the structure, content, and meaning of all datasets. Producing these files — most notably Define-XML — and the CRFs used to collect the data has traditionally been a manual, error-prone, and time-consuming process.

The CDISC 360i program aims to automate this process end to end: starting from a machine-readable study protocol (USDM), the DDE derives the metadata needed both to collect the data (CRFs) and to submit it (Define-XML). It closes the gap between protocol design and data submission by using the same source of truth throughout the study lifecycle.

How It Works

The DDE is built as loaders → DDS JSON → generators. Loaders extract metadata from the protocol and the CDISC Library and write it into the Data Definition Specification (DDS), a JSON model. Generators read the DDS and write study artifacts. The DDS file is the contract between the two halves.

flowchart LR
    USDM["USDM 4.0 JSON<br/>(E2J or SoA Workbench)"]
    LIB[("CDISC Library<br/>BCs · Specializations<br/>CDASHIG · CT")]
    CSV["CRF Specializations<br/>(draft CSV)"]
    DICT["Defaults dictionary<br/>(JSON)"]

    DL["Define loader<br/>create_define_json.py"]
    CL["CRF loader<br/>crf_loader.py"]
    DDS[/"Define DDS"/]
    CDDS[/"Combined DDS<br/>Define + CRF metadata"/]
    PR["Pagerefs loader<br/>aCRF pages → Origin"]
    DFL["Defaults loader<br/>well-known defaults"]
    FDDS[/"Final DDS"/]

    DG["Define generator<br/>define_generator.py"]
    CG["CRF generator<br/>crf_generator.py"]
    DX["Define-XML v2.1<br/>+ HTML"]
    ODM["ODM 1.3.2 / 2.0 CRF<br/>blank + annotated, + HTML"]

    USDM --> DL
    LIB --> DL
    DL --> DDS
    DDS --> CL
    USDM --> CL
    CSV --> CL
    LIB --> CL
    CL --> CDDS
    DDS --> DG
    CDDS --> CG
    CG --> ODM
    ODM -- "crfpdf: acrf.pdf + page refs" --> PR
    CDDS --> PR
    PR --> DFL
    DICT --> DFL
    DFL --> FDDS
    FDDS --> DG
    DG --> DX

Why a central JSON model? No single source has all the metadata needed for a full submission. The DDS acts as an aggregation layer, combining protocol content, biomedical concept definitions, controlled terminology, and any manually filled gaps into one validated model.

See documents/Solution_Overview.md for the full architecture and rationale.

Study Artifacts and Status

Artifact Format Produced by Status
SDTM Define-XML v2.1 .xml, .html Define loader → Define generator ✅ Implemented
ODM CRF v1.3.2 and v2.0 (blank CRF and aCRF in one file) .xml Define loader → CRF loader → CRF generator ✅ Implemented, from draft CRF Specializations
CRF / aCRF HTML rendering .html crf_generator.py --html with stylesheets/ ✅ Implemented, based on Phase 1 stylesheets
ADaM Define-XML .xml — 🔜 Planned
Trial Design datasets (TA, TD, TE, TI, TM, TS, TV) Dataset-JSON — 🔜 Planned
Dataset-JSON shells .json — 🔜 Planned
Lab Data Transfer Agreement → SDTM model LinkML src/dta2sdtm/ 📝 Draft model, no code

Incremental loading with metadata provenance tracking, and quality and conformance checks on the DDS, are also planned.

This project is provided “as is” without warranty or guarantee of suitability for any particular purpose. Expect breaking changes as the DDS model evolves and additional generators are developed.

Prerequisites

Installation

git clone https://github.com/cdisc-org/data-definition-engine.git
cd data-definition-engine

python3 -m venv .venv
source .venv/bin/activate

# One environment for all four programs
pip install -r src/define-xml/requirements.txt \
            -r src/loaders/crf/requirements.txt \
            -r src/generators/define/requirements.txt \
            -r src/generators/crf/requirements.txt

Each program has its own requirements.txt if you only need one of them.

Set your CDISC Library API key. Create a .env file at the repository root; both loaders find it there:

# .env  (git-ignored)
CDISC_API_KEY=your_api_key_here

You can instead set CDISC_API_KEY in your environment, or pass --cdisc_api_key on the command line. Both loaders also accept --base_api_url to point at a CDISC Library proxy or caching mirror.

Windows: activate with .venv\Scripts\activate, use a backtick ` instead of \ for line continuation in PowerShell, and \ as the path separator.

Quick Start

This runs the whole pipeline on the study in the repository, NCT01797120, a SoA Workbench export. All output goes to the repository’s git-ignored output/ directory.

The pipeline driver runs every stage in order, from the study’s pipeline.yaml, and ends with the review items still open. Its stages D (crfutils crfvalidate/crfpdf), E (pagerefs loader) and F (defaults loader) are not in the step-by-step version below:

cd src
python -m pipeline --study ../data/protocols/NCT01797120/pipeline.yaml   # stages A-G

The steps below run the four main programs one at a time.

The Define programs must run from their own directory (see The Programs), so each step cds from the repository root.

# 1. Define loader: USDM -> Define DDS
cd src/define-xml
python create_define_json.py \
  --usdm_file ../../data/protocols/NCT01797120/NCT01797120-latest.json \
  --output_template ../../output/NCT01797120-dds.json \
  --sdtmct 2026-03-27 \
  --patch_file ../../output/define-patch.yaml
cd ../..

# 2. CRF loader: Define DDS + USDM -> combined DDS
cd src/loaders/crf
python crf_loader.py \
  --dds_in ../../../output/NCT01797120-dds.json \
  --usdm_file ../../../data/protocols/NCT01797120/NCT01797120-latest.json \
  --dds_out ../../../output/NCT01797120-dds-crf.json \
  --crf_spec_file ../../../data/crf_specializations/cdisc_crf_specializations_draft.csv \
  --cdashct 2026-03-27 \
  --patch_file ../../../output/crf-refinement.yaml
cd ../../..

# 3. Define generator: Define DDS -> Define-XML v2.1
cd src/generators/define
python define_generator.py \
  -t ../../../output/NCT01797120-dds.json \
  -d ../../../output/define.xml
cd ../../..

# 4. CRF generator: combined DDS -> ODM CRF (2.0, then 1.3.2)
cd src/generators/crf
python crf_generator.py -t ../../../output/NCT01797120-dds-crf.json \
  -o ../../../output/crf-2.0.xml --odm-version 2.0 --validate
python crf_generator.py -t ../../../output/NCT01797120-dds-crf.json \
  -o ../../../output/crf-1.3.2.xml --odm-version 1.3.2 --validate
cd ../../..

# 5. Render HTML
xsltproc src/generators/define/define2-1.xsl output/define.xml > output/define.html

To render the CRFs, install saxonche and add --html ../../../output --stylesheet ../../../stylesheets/odm_2-0_transform.xsl to the 2.0 command in step 4 (or odm_1-3-2_transform.xsl for 1.3.2). It writes crf-2.0_crf.html (blank) and crf-2.0_acrf.html (annotated).

What to expect. Both ODM files pass odmlib and XSD validation, though some form text is still __PLACEHOLDER__ and every visit collects no forms (see the CRF loader below). The Define-XML is a draft: values the loader cannot derive from USDM, such as variable Length and Origin, are written as __PLACEHOLDER__, so it will not pass schema validation yet. To finish it, fill in output/define-patch.yaml and re-run step 1 with --apply_patch ../../output/define-patch.yaml added, then step 3 with -v. The pipeline fills most of these values for you: the pagerefs loader writes CRF-evidenced origins, the defaults loader fills well-known lengths and origins, and the define-patch.yaml beside pipeline.yaml is left listing only what neither could supply.

The Programs

Each program has a README with the full argument reference. The Define loader and Define generator use flat top-level imports and must be run from their own directory. The CRF loader and CRF generator are regular Python packages: they also run as python -m loaders.crf.crf_loader and python -m generators.crf.crf_generator from src/.

Define loader — src/define-xml/create_define_json.py

USDM → Define DDS. Calls the CDISC Library for Biomedical Concepts and SDTM Dataset Specializations. Full reference.

Option Purpose
--usdm_file, --output_template Input USDM and output DDS paths (required)
--sdtmct SDTM Controlled Terminology package date, yyyy-mm-dd (required)
--sdtmig SDTMIG version (default 3.4)
--patch_file / --apply_patch Write a YAML file listing every value still __PLACEHOLDER__; fill it in and re-run with --apply_patch pointing at it. --apply_patch requires --patch_file, so the file is refreshed after each run.
--validate + --validation_report Validate the DDS against the LinkML schema (define.yaml) and write an Excel report; the two are used together
--debug Write intermediate debug_*.json files to the working directory

CRF loader — src/loaders/crf/crf_loader.py

Define DDS + USDM → combined DDS. Selects one CRF Specialization per Biomedical Concept and fills it out from CDASHIG and CDASH CT. Full reference.

Option Purpose
--dds_in, --usdm_file, --dds_out Input Define DDS, input USDM, output combined DDS (required)
--crf_spec_file The CRF Specializations .csv/.xlsx; see its provenance
--cdashct / --cdashig CDASH CT package date (required) and CDASHIG version (default 2.3)
--implementation_option Denormalized (default) or Normalized, for concepts with several CRF variants
--patch_file / --apply_patch The refinement YAML: forms, sections, visit bindings, CRF variant choices, uncovered concepts
--validate Validate against the CRF profile JSON Schema (schemas/dds/profiles/crf)
--no_cache Bypass the on-disk CDISC Library cache (.cache/cdisc_library), which makes re-runs fast

The refinement loop matters most for visits. SoA Workbench exports carry no activity-to-visit links, so every study event comes out with no forms. The refinement YAML lists candidate forms for each visit; bind them under studyEvents and re-run with --apply_patch.

Define generator — src/generators/define/define_generator.py

DDS → Define-XML v2.1, using odmlib. Full reference.

Option Purpose
-t, --template Input DDS JSON (required)
-d, --define Output Define-XML path. Always pass it: the default is tests/fixtures/define-360i.xml, a committed test fixture.
-v, --validate Schema-validate the output; exits 1 on failure
-s, --submission Set def:Context to Submission (default Other)
-x, --sas_xpt Use .xpt dataset extensions instead of .ndjson

Render the result as HTML with xsltproc and the bundled define2-1.xsl, or with python -m defineutils.definehtml.

CRF generator — src/generators/crf/crf_generator.py

Combined DDS → ODM CRF, version 1.3.2 or 2.0 from the same input. Requires odmlib ≥ 0.2.1. Full reference.

Option Purpose
-t, --template Input combined DDS (required)
-o, --odm Output ODM XML path (default crf.xml)
--odm-version 1.3.2 or 2.0 (required)
-v, --validate Validate with odmlib and against the ODM XSD
--html / --stylesheet Also render the blank CRF and aCRF as HTML (needs saxonche)
--forms Emit only these Form OIDs (comma-separated)

Defaults loader — src/loaders/defaults

DDS → DDS with well-known defaults (pipeline stage F). Fills lengths, origins and display formats that are still unfilled from one or more dictionaries (data/defaults/, schema schemas/defaults/), then rewrites define-patch.yaml with what is still open. Full reference; design and seed tables in DEFAULT_METADATA_LOADER.md.

Option Purpose
--dds_in, --dds_out Input DDS (the pagerefs loader’s) and output DDS (required)
--defaults One or more dictionaries, later files win (default data/defaults/well_known_defaults.json)
--patch_file The define patch: its answers are applied, then it is rewritten with what is still open plus every answered value
--report Write defaults-report.yaml: what was filled, dataType conflicts, Collected/Investigator origins without aCRF pages, unmatched entries

Pipeline — src/pipeline

Runs stages A-G for one study from its pipeline.yaml: the Define loader, CRF loader, CRF generator, crfutils crfvalidate/crfpdf, the pagerefs loader (src/loaders/pagerefs), the defaults loader and the Define generator. --from/--to re-run part of it, --strict fails while a review item is open, and --dry-run prints the commands. Full reference.

Project Structure

data-definition-engine/
├── data/                          # Inputs (see data/README.md)
│   ├── protocols/NCT01797120/     # USDM 4.0 study from SoA Workbench
│   ├── crf_specializations/       # Draft CDISC CRF Specializations (CSV) + PROVENANCE.md
│   ├── defaults/                  # Well-known defaults dictionaries for the defaults loader
│   └── metadata_xlsx/             # SDTM/ADaM metadata spreadsheets and the Define Excel template
├── documents/                     # Solution overview, glossary, presentations
├── schemas/                       # Vendored schemas (see schemas/README.md)
│   ├── dds/                       # DDS LinkML model, and profiles/crf (CRF profile)
│   ├── defaults/                  # JSON Schema of the defaults dictionary (DDE-owned)
│   ├── define/                    # Define-XML 2.0 and 2.1 XSDs
│   └── odm/                       # ODM 1.3.2 and 2.0 XSDs
├── stylesheets/                   # ODM 1.3.2 and 2.0 CRF stylesheets (XSLT 2.0)
├── output/                        # Generated files (git-ignored)
└── src/
    ├── define-xml/                # Define loader: USDM -> Define DDS
    ├── loaders/
    │   ├── common/                # Shared loader utilities (DDS I/O, USDM, Library cache, patching)
    │   ├── crf/                   # CRF loader: Define DDS + USDM -> combined DDS
    │   ├── pagerefs/              # Pagerefs loader: aCRF page references -> Define Origin
    │   └── defaults/              # Defaults loader: well-known defaults for what is still unfilled
    ├── pipeline/                  # The driver: stages A-G for one study
    ├── generators/
    │   ├── define/                # Define generator: DDS -> Define-XML v2.1 (+ define2-1.xsl)
    │   └── crf/                   # CRF generator: DDS -> ODM 1.3.2 / 2.0 (targets/ holds version differences)
    └── dta2sdtm/                  # Draft LinkML model for lab Data Transfer Agreements -> SDTM

Running the Tests

None of the suites call the network. Run each from its own directory:

cd src/define-xml         && pytest tests
cd src/loaders/crf        && pytest
cd src/loaders/pagerefs   && pytest
cd src/loaders/defaults   && pytest
cd src/pipeline           && pytest
cd src/generators/define  && pytest
cd src/generators/crf     && pytest

Known Limitations

Key Concepts

Term Definition
USDM (Unified Study Definitions Model) A TransCelerate / CDISC standard that represents a complete clinical trial protocol as structured, machine-readable JSON. It is the primary input to the DDE.
SoA Workbench A CDISC tool for authoring a study’s schedule of activities that exports USDM JSON.
CDISC 360i A CDISC initiative to make the full clinical trial lifecycle — from protocol to submission — machine-readable and interoperable.
DDS (Data Definition Specification) The central intermediate JSON model in the DDE pipeline. It aggregates metadata from all sources and is the single input for all generators.
Combined DDS A DDS carrying both Define-XML and CRF metadata, written by the CRF loader. Each generator reads the sections it owns.
Defaults dictionary A JSON file of well-known default values (lengths, origins, display formats) the defaults loader writes where no other loader supplied a value. Sponsor-editable; see data/defaults/.
Define-XML An XML file submitted alongside clinical trial datasets that describes their structure, variables, permitted values, and controlled terminology. Required by the FDA. Define-XML v2.1 is an extension of ODM v1.3.2.
ODM (Operational Data Model) The CDISC XML standard for exchanging clinical study metadata and data, including CRF definitions. The DDE generates CRFs in ODM v1.3.2 and v2.0.
aCRF (annotated CRF) A CRF annotated with the SDTM variables each field maps to. The DDE emits one ODM file that renders as either the blank CRF or the aCRF.
Biomedical Concepts (BCs) Standardized, reusable definitions of clinical observations (e.g., “Heart Rate”) maintained in the CDISC Library.
Dataset Specializations (DSSs) CDISC Library mappings that describe how a Biomedical Concept is represented in a specific SDTM domain.
CRF Specializations Mappings that describe how a Biomedical Concept is collected on a CRF: its fields, prompts, codelists, and SDTM targets.
CDASHIG / CDASH CT The CDISC standard for CRF data collection fields, and its controlled terminology. The CRF loader uses both to fill in CRF Specializations.
CDISC Library CDISC’s REST API providing access to controlled terminology, standards, Biomedical Concepts, and Dataset Specializations.
odmlib A Python library for creating and parsing CDISC ODM and Define-XML documents, used by both generators.
LinkML A modeling language used to define and validate the DDS JSON schema.
VLM (Value Level Metadata) Metadata that applies to specific values within a variable (e.g., rules that only apply when VSTESTCD = "SYSBP").

More terms are defined in documents/glossary.md.

Contributing

Contributions are welcome. Please read CONTRIBUTING.md before submitting pull requests. All contributions must follow the Code of Conduct and will fall under the project licenses below.

License

Code and Models

Licensed under the MIT License.

Content (documentation, etc.)

Licensed under CC-BY-4.0.

When re-using content, please cite as:

Content based on Data Definition Engine (GitHub) used under the CC-BY-4.0 license.

References

This site is open source. Improve this page.