Document Type
Student Coursework
Publication Date
Fall 10-31-2026
Abstract
While municipal crime statistics frequently report broad, citywide downward trends, aggregate metrics often mask localized divergence at microgeographic scales. Concurrently, data engineering pipelines designed to ingest, process, and forecast municipal incidents must navigate fundamental trade-offs between distributed computing overhead, low-latency streaming ingestion, and the extrapolation limitations of machine learning estimators. This paper presents the design, end-to-end cloud implementation, and empirical evaluation of a dual-path batch and streaming Big Data architecture deployed over four contiguous Philadelphia neighborhoods (West Kensington, East Kensington, Fishtown, and Port Richmond). Historical incidents (309,106 records spanning 2006–2026) were processed via Apache Spark and MapReduce backends, cross-validating an identical 1,749-key aggregation space (a 177:1 dimensionality reduction). Near-real-time ingestion via Structured Streaming achieved steady-state micro-batch throughput of 615 records/second at an average latency of 1,117 ms. Two critical empirical findings are highlighted: (1) at sub-million record volumes, single-node MapReduce (1.80 s) matched distributed Spark DataFrame execution (1.66 s), demonstrating that distributed cluster coordination overhead offsets parallelization benefits below defined data thresholds; and (2) ordinary least squares linear regression significantly outperformed an ensemble Random Forest (R² = 0.674 vs. R² = 0.345) under chronological hold-out validation, exposing the fundamental inability of tree-based partitioners to extrapolate negative trends outside training bounds. Finally, spatial clustering (K-Means, k = 3, silhouette = 0.689) and neighborhood-level disaggregation revealed a 53.5-percentage-point divergence in incident trajectories within a single contiguous corridor, confirming that centralized municipal reporting obscures hyper-local safety dynamics.
Program or Discipline Name
Computer and Information Sciences
Recommended Citation
Jiang, Y., & Cha, S. (2026). Evaluating Scalability, Extrapolation Limits, and Microgeographic Divergence: A Hybrid Batch–Streaming Architecture for Urban Crime Analytics. Retrieved from https://digitalcommons.harrisburgu.edu/other-works/37
Start Page No.
1
End Page No.
8
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License.
Submission aggreement