ARCHITECTURAL OPTIMIZATION OF CLOUD-NATIVE DATA PROCESSING PIPELINES: A SYSTEMATIC ANALYSIS OF SERVERLESS COMPUTING PARADIGMS

Authors

  • Seshendranath Balla Venkata Comcast, USA Author

Keywords:

Serverless Computing, Data Processing Pipelines, Cloud Architecture, Infrastructure Optimization, Data Analytics, Event-driven Computing

Abstract

This article comprehensively analyzes enterprise-scale serverless data processing compute platforms - AWS EMR Serverless, Google Cloud Dataproc Serverless, and Azure Synapse Serverless - going beyond traditional static resource compute jobs and lightweight serverless computing to address large-scale data analytics needs. The article examines how these specialized platforms enable complex batch processing, near real-time streaming jobs, ETL operations, and distributed machine learning workloads on petabyte-scale datasets while maintaining serverless benefits of dynamic scaling and consumption-based pricing. Through a systematic evaluation of enterprise implementations, this study investigates architectural patterns and operational strategies specific to big data processing, including distributed computation frameworks, data lake integration approaches, and performance optimization techniques for data-intensive workloads. Case studies from organizations demonstrate the practical benefits and challenges of migrating traditional big data workloads to serverless platforms, providing insights into performance improvements, cost optimization, and operational efficiency gains in processing large-scale datasets with flexibility. The findings reveal that serverless big data architectures, when properly implemented, can significantly reduce infrastructure costs and operational complexity while maintaining processing performance and reliability for enterprise-scale analytics workloads. This article contributes to the growing body of knowledge on cloud-native data processing by providing a structured framework for evaluating and implementing serverless big data pipelines and recommendations for addressing common challenges in large-scale data processing environments.

References

B. Madupati, "Serverless Architectures and Function-As-A-Service (Faas): Scalability, Cost Efficiency, And Security Challenges," International Journal of Research in Management, IT & Economics, 2023. [Online]. Available: https://www.ijirmps.org/papers/2023/2/231326.pdf

A. Dutta and B. J, "Impact of Serverless Computing on Scalability and Cost-Effectiveness in Cloud-based Applications," International Journal of Research Publication and Reviews, 2024. [Online]. Available: https://ijrpr.com/uploads/V5ISSUE3/IJRPR23549.pdf

J. Gilbert and E. Price, "Software Architecture Patterns for Serverless Systems: Architecting for innovation with events, autonomous services, and micro frontends," IEEE Press, 2021. [Online]. Available: https://ieeexplore.ieee.org/book/10162936

Safeer Cm, "Architecting Cloud-Native Serverless Solutions: Design, build, and operate serverless solutions on cloud and open source platforms," Packt Publishing eBooks, 2023. [Online]. Available: https://ieeexplore.ieee.org/book/10251275

S. Nayak, "Exploring Serverless Computing: Advantages, Limitations, and Best Practices," CloudOptimo, 2024. [Online]. Available: https://www.cloudoptimo.com/blog/exploring-serverless-computing-advantages-limitations-and-best-practices/

Cloudflare "Why Use Serverless Computing?," 2024. [Online]. Available: https://www.cloudflare.com/learning/serverless/why-use-serverless/

P. Kava and C. Gong, "AWS Serverless Data Analytics Pipeline Reference Architecture," AWS Big Data Blog, 2020. [Online]. Available: https://aws.amazon.com/blogs/big-data/aws-serverless-data-analytics-pipeline-reference-architecture/

Zahra Shojaee Rad & Mostafa Ghobaei-Arani "Data Pipeline Approaches in Serverless Computing: A Taxonomy, Review, and Future Directions," Journal of Big Data, SpringerOpen, 2024. [Online]. Available: https://journalofbigdata.springeropen.com/articles/10.1186/s40537-024-00939-0

M. Golec, G. K. Walia, M. Kumar, F. Cuadrado, S. S. Gill, and S. Uhlig, "Cold Start Latency in Serverless Computing: A Systematic Review, Taxonomy, and Future Directions," ACM Computing Surveys, 2024. [Online]. Available: https://arxiv.org/abs/2310.08437

H. Shafiei, A. Khonsari, and P. Mousavi, "Serverless Computing: A Survey of Opportunities, Challenges and Applications," IEEE Access, vol. 8, pp. 28369-28381, 2020. [Online]. Available: https://arxiv.org/pdf/1911.01296v3

AWS "AWS Lambda Case Studies," AWS, [Online]. Available: https://aws.amazon.com/lambda/resources/customer-case-studies/

Dashbird "Serverless Framework: The Coca-Cola Case Study," Dashbird, 2020. [Online]. Available: https://dashbird.io/blog/serverless-case-study-coca-cola/

E. Van Eyk, L. Toader, S. Talluri, L. Versluis, A. Uta, and A. Iosup, "Serverless is More: From PaaS to Present Cloud Computing," IEEE Internet Computing, vol. 22, no. 5, pp. 8-17, 2018. [Online]. Available: https://ieeexplore.ieee.org/document/8481652

Published

2024-12-19

How to Cite

Seshendranath Balla Venkata. (2024). ARCHITECTURAL OPTIMIZATION OF CLOUD-NATIVE DATA PROCESSING PIPELINES: A SYSTEMATIC ANALYSIS OF SERVERLESS COMPUTING PARADIGMS. INTERNATIONAL JOURNAL OF COMPUTER ENGINEERING AND TECHNOLOGY (IJCET), 15(6), 1492-1508. https://mylib.in/index.php/IJCET/article/view/1754