Telecom Use Case Part 2 ETL Practice

#etl #scala #dwh #spark #Spark_In_Arabic #Big_Data #Hadoop #هادوب #بالعربي #Big_Data_In_Arabic #Code
In this episode, we explain the following topics.
- (00:00) Introduction to ETL Life Cycle
- (01:20) ETL best practices review
- (02:45) Code practices
- (03:06) Project structure and dependencies
- (04:30) Input source scenario analysis
- (05:34) Pre-processing step
- (07:35) ETL step rules
- (08:26) Job configuration analysis with example
- (14:34) ETL pipeline
- (17:27) Data Design
- (22:22) Dev Notes
- (29:45) Quality checks
- (30:24) Coding exercise
- (30:38) Writing unit test to validate JSON string parsing as case class
- (34:33) Writing function to parse JSON string as a static case class
- (40:48) Writing function to parse JSON string as any case class using generics
- (45:45) Write unit test to validate JSON string parsing from a file
- (50:35) Writing function to read file content and pass it to JSON parser
- (59:30) Video recap
----------------------------------------------------------------------------------------------------------------------
"Big Data Engineering in Depth": https://gability.com/en/courses/big-data-in-depth/
----------------------------------------------------------------------------------------------------------------------
Repo: https://github.com/garage-education/spark-etl-demo
Documentation: https://docs.google.com/document/d/11MDJCHMxUZE7EAYIBJluq6HJv3upYgV06gYMNApUNdc/
----------------------------------------------------------------------------------------------------------------------
- About Garage Education: https://gability.com/en/#about
----------------------------------------------------------------------------------------------------------------------
Follow us:
Twitter: https://twitter.com/garageeducation
Facebook: https://www.facebook.com/GarageEducationCourses/
Youtube: https://www.youtube.com/c/GarageEducation
Website: http://gability.com/