Chapter 5 Data Pipeline and Reproducibility

The data pipeline architecture, versioned file inventory, and verification procedures supporting the quantitative findings in Chapter 5 are documented here. Reproducibility is a foundational requirement for design science research artefacts: the serialisation schema and its associated analysis pipeline must be transparent in their inputs, transformations, and outputs such that an independent researcher could, given access to the same source materials, reproduce the findings reported in the evaluation. This appendix also identifies the methodological limitations that constrain full reproducibility. The full data package, including all analysis scripts and versioned outputs, is deposited as the Chapter 5 artefact bundle described in Appendix: Chapter 5 Evaluation Results and Data Package, where the deposit location and canonical artefacts are enumerated.

Pipeline Architecture

The analysis pipeline proceeds through four stages, each producing versioned outputs that serve as inputs to the subsequent stage.

Stage 1: Source Extraction. The SDA Design Standard (2019) produces two parallel extraction channels: a text channel extracting clause-based requirements from the standard’s prose (611 total entries, 140 classified as design requirements), and a figure channel extracting diagram-based requirements from the standard’s 19 figures (48 sub-figures; 406 total entries, 189 classified as design requirements). Both channels preserve the full source text alongside structured metadata (figure references, clause references, field types, applicability markers).

Stage 2: Normalisation and Triple Extraction. Figure-channel entries are encoded in a structured, consistently field-typed and applicability-tagged form. Design requirements were decomposed into subject-predicate-object triples using pattern-based extraction, producing 189 triples with 56 unique canonical entities and 17 unique predicate phrases. Entity resolution was performed to map variant surface forms to canonical identifiers; for example, 20 surface variants of “Door” were mapped to the canonical entity Door. The decomposition quality audit reports 187 clean extractions (98.9%), 1 fallback (0.5%), and 1 failure (0.5%).

Stage 3: Multi-Dimensional Analysis. Four parallel analyses are applied to the normalised corpus: (a) five-dimensional resolution-rate scoring (documented in Appendix: Chapter 5 Figures-Channel Ambiguity Analysis); (b) deontic force classification and inventory (documented in Appendix: Chapter 5 Predicate Coverage and Deontic Force); (c) polysemy assessment (documented in Appendix: Chapter 5 Polysemy Burden Assessment); and (d) text-figure cross-validation (documented in Appendix: Chapter 5 Cross-Channel Validation Results). Each analysis was executed by a dedicated, version-controlled analysis program, producing both a machine-readable data export and a human-readable narrative report.

Stage 4: Unified Data Assembly. The outputs of Stage 3 are assembled into a single unified explorer dataset containing all 56 entities, 189 triples, and associated metadata in one queryable structure. This dataset serves as the canonical data source across all four evidence domains. An interactive, self-contained browser explorer was generated from it to support entity browsing, triple inspection, and design-category filtering. In summary, the four-stage pipeline produces a transparent, versioned chain of evidence from source standard to quantitative findings, and the next section describes the named datasets that make each stage auditable.

Named Datasets by Pipeline Stage

The tables below describe the named source, intermediate, and output datasets produced at each pipeline stage, each with its record scope and approximate data volume. All datasets are deposited in the Chapter 5 artefact bundle whose deposit location is recorded in Appendix: Chapter 5 Evaluation Results and Data Package; the bundle’s own data dictionary documents the field-level schema of each.

Stage 1: Source Datasets

Stage 2: Normalised Datasets

Stage 3: Analysis Datasets

Each Stage 3 analysis was produced by a dedicated, version-controlled analysis program documented in the bundle’s data dictionary.

Stage 4: Unified Dataset

Verification Procedures

Each pipeline stage incorporates internal verification that allows the findings to be cross-checked against multiple independent representations of the same underlying data.

Stage 1 Verification. The figure extraction was validated against source images. A figure-extraction comparison record, deposited in the bundle, documents discrepancies between the initial and revised extractions, with all discrepancies resolved in the revised pass.

Stage 2 Verification. The normalisation stage produces a statistics dataset that independently reports field-type distributions (context: 67, Design requirement: 189, Applicable to: 107, Note: 43, total: 406), enabling cross-check against the raw extraction counts. A normative-voice audit within this dataset confirms that 181 design requirements contain “shall” and 8 contain “permitted,” with zero design requirements lacking a normative verb.

Stage 3 Verification. Each analysis produces both a machine-readable data export and a human-readable narrative report. The narrative reports present the same quantitative findings in prose, enabling manual review against the data exports. The parity assessment explicitly tracks gaps between the text and figure analysis channels, recording three gaps closed and three remaining minor gaps. Building on these stage-level checks, Stage 4 verification confirms that the unified dataset preserves a complete provenance chain from source material to final output.

Stage 4 Verification. The unified dataset is built by a deterministic assembly procedure that reads from the Stage 3 datasets. The dataset’s metadata records source identification, entity count (56), triple count (189), and version, providing a provenance chain from source to final output. Every quantitative claim in the evaluation appendices is traceable to a named field in a named dataset described above.

Limitations on Reproducibility

Five limitations constrain the reproducibility of this analysis. An independent researcher attempting replication should be aware of each.

Manual entity resolution. The 56-entity vocabulary records a specific set of resolution decisions that are defensible but not uniquely determined, because the mapping of 20 surface variants of “Door” (and similar variant sets for other entities) to canonical identifiers requires manual adjudication. A different researcher might draw canonical boundaries differently, for instance, treating “Door circulation spaces” and “Door sizes” as distinct entities rather than variants of Door. The 56-entity vocabulary reflects a specific set of resolution decisions that are defensible but not uniquely determined.

WordNet polysemy scope. WordNet sense inventories provide general-language sense counts but do not capture domain-specific technical meanings. An entity like Bollard is flagged as monosemous despite having a specific access-control meaning in the SDA context, while entities like Tap inherit 20 general-language senses mostly irrelevant to building standards. Polysemy burden figures should be interpreted as upper bounds on general-language ambiguity rather than as precise measures of domain-specific ambiguity.

Source standard access. The SDA Design Standard (2019) is a copyrighted instrument administered by the National Disability Insurance Agency. The source document cannot be redistributed with the research materials. An independent researcher would need to obtain it through authorised channels to reproduce the Stage 1 extraction.

Extraction subjectivity. The figure extraction process requires interpreting graphical annotations, dimensional callouts, and spatial relationships from rasterised diagram images. While the extraction prompt template and format specification provide detailed instructions, some degree of interpretive judgement is inherent in reading dimensions from scaled diagrams.

Pipeline iteration history. The text channel produces 7 analysis iterations (cycles 0-6), while the figure channel produces a base analysis plus a deepening pass. All final outputs derive from a single consistent extraction snapshot, so that the deposited datasets form one coherent version for verification. Earlier intermediate outputs may differ from the final versions. Overall, these five limitations bound the reproducibility claim in a transparent manner: they identify where independent replication requires either access to the source standard, tolerance of alternative resolution judgements, or awareness of WordNet’s general-language scope. The next section summarises the pipeline’s aggregate scope and the traceability structure that holds across all four evidence domains.

Summary

The data pipeline from source standard to evidence appendices is traceable across 4 stages and roughly two dozen versioned datasets totalling approximately 6.9 MB of structured data. Every quantitative claim in the evaluation appendices is traceable to a named field in a named dataset, and every analysis is executed by a documented, version-controlled program operating on versioned inputs. Within the constraints identified above, the pipeline provides a transparent and auditable chain of evidence from source material to quantitative findings. Therefore, the reproducibility infrastructure documented here meets the transparency requirements that design science research imposes on artefact-based evaluation and supports the Chapter 5 claim that the evaluation findings are independently verifiable within the stated constraints.