Skip to content

Data Processing

  • Structured Streaming – Framework for incremental and continuous data processing.
  • Auto Loader – Incremental file ingestion tool using structured streaming with exactly-once guarantees.
  • COPY INTO – SQL command for batch-based incremental ingestion of files.
  • Streaming Checkpoints – Metadata storage for fault tolerance and exactly-once semantics.
  • Trigger Modes – processingTime, once, availableNow for controlling streaming intervals.