• Introduction
• Top-level Metadata Attributes
• Column Metadata
• • Supported Column Data Type Combinations
  • Date/Time Variables
  • Decimal Variables
• Row Data
• A Full Example of a Dataset-JSON File
• NDJSON Representation of Dataset-JSON
• A Full Example of an NDJSON Dataset-JSON File
• Glossary and Abbreviations

Introduction

Dataset-JSON is a data exchange standard for sharing tabular data using JSON. It is designed to meet a wide range of data exchange scenarios, including regulatory submissions and API-based data exchange. Each Dataset-JSON dataset can optionally reference a Define-XML document containing more complete metadata for the dataset. One aim of Dataset-JSON is to address as many of the relevant requirements in the PHUSE 2017 Transport for the Next Generation paper as possible, including the efficient use of storage space.

Dataset-JSON uses lowerCamelCase notation for attribute names.

The JSON standard does not allow specifying or controlling the order of attributes. However, since most JSON encoders and decoders allow control over attribute order, it is strongly recommended to follow the attribute order documented in this specification. Due to the potentially large size of Dataset-JSON datasets, adhering to the specified attribute order allows software using streaming approaches to read the file more efficiently and quickly.

Dataset-JSON must contain only 1 dataset per file. Dataset-JON uses the file extension .json.

Although adapted from the Dataset-XML Version 1.0 specification, Dataset-JSON uses the JSON format and includes many enhancements.

The LinkML model representation, the JSON schema, both for JSON and NDJSON representations, and examples can be found at the GitHub repository for the Dataset-JSON Version 1.1 standard.

The specification and user's guide are for both clinical and non-clinical research data.

Top-level Metadata Attributes

The following table summarizes the technical and dataset attributes at the top level of the Dataset-JSON object.

The following example illustrates the basic Dataset-JSON structure.

Column Metadata

columns is an array of basic information about dataset variables. The order of the elements in the array must be the same as the order of variables in the described dataset.

The following example shows the columns metadata structure.

Supported Column Data Type Combinations

The JSON data type mentioned in the "Supported Column Data Type Combinations" table is the data type of the actual data in the "rows" arrays in the Dataset-JSON file. This JSON data type is defined by the Dataset-JSON Schema, but only includes "string", "integer", "boolean", number", and null. It does not include "array" or "object".

Date/Time Variables

Timing variables (datetime, date, time) are stored as ISO 8601 strings in the JSON format. The targetDataType attribute needs to be specified when different from dataType attribute or the JSON data type.

For example, consider the following AE dataset. The AESTDTC and AEENDTC variables have "date" as their logical data type (dataType attribute), and since the values are already ISO 8601 strings in the data, no conversion is needed to a string in JSON format. The targetDataType for the date and datetime variables does not need to be mentioned for SDTM datasets as the logical type is the same as the JSON data type.

For ADaM datasets, the targetDataType must be set to integer. Consider the following example of ADAE data. The targetDataType attribute is mentioned, as the logical data type is different from the JSON data type. Also note that the displayFormat attribute has been specified as "E8601DA" so that SAS can display these numeric values in a user-friendly way.

Decimal Variables

Some decimal variables can be exchanged as a string in JSON to represent terminating decimal fractions without rounding. Hence, the targetDataType is "decimal" and JSON data type is "string".

For example:

Row Data

rows is an array of records with variables values. Each record itself is also represented as an array of variables values.

Missing values are represented by null. Empty strings are represented by "".

Example:

A Full Example of a Dataset-JSON File

NDJSON Representation of Dataset-JSON

The purpose of the NDJSON, or new-line delimited JSON, representation of Dataset-JSON is to simplify streaming large datasets. With NDJSON, a dataset can easily be read or written one row at a time without loading the entire dataset into memory. The NDJSON and JSON dataset content are the same. In a data-exchange scenario, the sender and receiver determine whether to use the JSON or the NDJSON representation of Dataset-JSON.

NDJSON is a standard for delimiting JSON in stream protocols. In NDJSON, each line is valid JSON. JSON is delimited by the new-line character (\n or 0x0A), which may be preceded by a carriage return character (\r or 0x0D). UTF-8 encoding is expected. NDJSON uses the file extension .ndjson.

The Dataset-JSON NDJSON format is created from the Dataset-JSON standard by:

• Row 1. Create 1 JSON object that contains the metadata, including the dataset attributes and column definitions.
• Row 2-n. Create 1 JSON array per data row

A Full Example of an NDJSON Dataset-JSON File

Each row in an NDJSON file can be parsed and processed as stand-alone JSON.

Glossary and Abbreviations