A basic book on data engineering. It covers all aspects of designing and operating various data architectures, as well as the complete lifecycle of working with data: preparation, ingestion, transformation, cleaning, combination, orchestration, and pipeline creation. Special attention is given to cloud technologies, load distribution, and secure data storage. Techniques for query formation, working with databases, and tracking data flows are discussed. The book provides a comprehensive understanding of designing systems for processing heterogeneous data, maintaining these systems in an orderly and accessible manner, and the trends in the development of the modern data industry. Over the past decade, data engineering has been rapidly developing, leading to an increasing demand among programmers, researchers, and data analysts for a comprehensive consideration of data engineering as a coherent paradigm. From the book, you will learn how to design and create enterprise systems for data processing. The book helps evaluate the various technologies available within data engineering and understand the data lifecycle as a product. The authors demonstrate the intricacies of data engineering in motion, explaining how to integrate and combine different cloud data processing technologies to maximally satisfy user interests. It shows how to organize the generation, ingestion, orchestration, transformation, storage, and pipelining of data and their management in any environment, regardless of the underlying technology it is based on.
