Data Engineering Framework For Production Pipelines — Prompt-Spark

The Highly Efficent and Modular Spark Data Engineering Framework

Motivation Behind Development

1. Why PROMPT ?

2. How it is diffrent from other Frameworks

Hurdles To Overcome with PROMPT

1. Functional Smoothness!

1. PROMPT provides data modelling concept back in the world of Big Data.
2. Dont care about the setup of pipeline code.
3. Helper API's to make things generic.

2. Technical Smoothness

Framework In Details

── com
└── promptscalaspark
└── framework
├── api
│ ├── LoaderHelper.scala
│ └── ModellerHelper.scala

├── functionalModel
| ├── FunctionalModelSchema.scala
│ └── FunctionalModeller12.scala

├── io
│ ├── ioSchema
│ │ └── InputSchema.scala
│ │
│ ├── loader
│ │ └── DummyInputFileData1Loader.scala
│ │
│ └── writer
│ └── DSWriter.scala

├── jobs
│ └── PromptBatchJob.scala

├── modeller
│ ├── ModellerSchema
│ │ ├── Sample1ModellerSchema.scala
│ │ └── Sample2ModellerSchema.scala
│ │
│ ├── SampleModeller1.scala
│ └── SampleModeller2.scala

└── visualiser
└── preVisualiserProcessor.scala

1. Modules In Details

a. io — Input/Output (SCALA Object)

Loader in PROMPT reads data in SPARK Data Structres like RDD or Datasets, now it can behave diffrentls in case of varied kind of input data but the output of loader should be bounded with some case class structure.

Writer in PROMPT writes data from SPARK Data Structres like RDD or Datasets, that is already bounded with ioSchema case classes.

b. api (SCALA Object)

ModellerHelper is the object where helper methods will reside to model data according to the needs of either models or modelProcessors.

c. modeller (SCALA Object)

ModellerSchema in PROMPT consists of multiple case classs that will define and bound with input and output data.

d. functionalModel (SCALA Trait)

e. Jobs (SCALA Object with main)

--

--

Software Engineer

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store