Blog posts around Data Engineering | TO THE NEW Blog

From Naive RAG to Production-Grade Agent: The 6-Stage Architecture of an Enterprise AI Assistant

Krishna Jain — Mon, 10 Aug 2026 05:48:20 +0000

Introduction Large Language Model (LLM) applications are evolving beyond single prompt-and-response setups. While basic Retrieval-Augmented Generation (RAG) scripts work well for simple Q&A over documents, real-world enterprise environments demand a much broader set of capabilities.In production, enterprise data is inherently fragmented: HR policies reside in unstructured PDFs, workforce metrics live in relational databases (SQL), and […]

The post From Naive RAG to Production-Grade Agent: The 6-Stage Architecture of an Enterprise AI Assistant first appeared on TO THE NEW Blog.

How should organizations evaluate native Databricks capabilities when DBT Cloud is already part of their data platform?

Rahul Kumar Jha — Mon, 27 Jul 2026 08:08:34 +0000

Introduction Modern data platforms have evolved significantly over the last decade. Organizations that once relied on traditional ETL systems are now adopting cloud-native architectures built around scalable processing engines, cloud storage, and open data formats. Technologies such as Apache Spark, Delta Lake, Databricks, and DBT have played an important role in this transformation by enabling […]

The post How should organizations evaluate native Databricks capabilities when DBT Cloud is already part of their data platform? first appeared on TO THE NEW Blog.

CrewAI for Agentic AI and Multi-Agent Workflows

Muazzam Munawwar Sayyed — Fri, 03 Jul 2026 10:39:49 +0000

Introduction Artificial Intelligence is moving beyond simple prompt-and-response interactions. Earlier AI applications were often built around a single prompt sent to a Large Language Model. The model generated an answer, and the workflow ended there. That works well for summaries, brainstorming, and content generation. But real enterprise problems often need planning, tool usage, validation, collaboration, […]

The post CrewAI for Agentic AI and Multi-Agent Workflows first appeared on TO THE NEW Blog.

Moving Beyond the Messy Middle: How I Automated Regulatory Data Pipeline with Snowflake and Streamlit

Karishma Singhal — Thu, 04 Jun 2026 05:47:28 +0000

Every data engineer knows the dread of a process that looks perfect on paper but relies entirely on manual intervention. Not long ago, I was managing a critical regulatory compliance data pipeline for a client. The raw ingredients were complex, multi-format regulatory list updates provided in Excel and CSV files. The destination was Snowflake, where […]

The post Moving Beyond the Messy Middle: How I Automated Regulatory Data Pipeline with Snowflake and Streamlit first appeared on TO THE NEW Blog.

Solving ETL Dependency Bottlenecks with GitHub Actions

Keha Gupta — Thu, 12 Feb 2026 11:06:00 +0000

Introduction In modern data platforms, ETL pipelines are rarely independent. They are deeply interconnected—one pipeline’s output becomes another pipeline’s input. In one of our production projects, we faced a classic orchestration challenge. Pipelines were scheduled based on time rather than actual completion i.e real-time problem with predictable execution time of pipelines. This article will walks […]

The post Solving ETL Dependency Bottlenecks with GitHub Actions first appeared on TO THE NEW Blog.

What I Learned Integrating Data with Airbyte

Isha Vason — Tue, 10 Feb 2026 11:34:46 +0000

Like many data engineers, I’ve spent a good chunk of my time dealing with a problem that sounds simple on paper but is messy in reality: reliably moving data from source systems into an analytics platform. In one of my recent projects, I worked on setting up data integration using Airbyte, and this post is […]

The post What I Learned Integrating Data with Airbyte first appeared on TO THE NEW Blog.

How GenAI Is Transforming Data Engineering

Shashvat Vashishtha — Tue, 06 Jan 2026 07:41:15 +0000

Introduction Data engineering, once dominated by manual coding, SQL development, and repetitive operational tasks, is entering a new era. With Generative AI (GenAI), data teams are automating ingestion workflows, accelerating data modeling, writing code faster, improving quality checks, and generating documentation instantly. GenAI isn’t just an add-on—it is fundamentally transforming how modern data platforms are […]

The post How GenAI Is Transforming Data Engineering first appeared on TO THE NEW Blog.

Accelerating Data Transfer with Apache Arrow Flight

Rahul Pupreja — Mon, 15 Sep 2025 08:21:54 +0000

In the modern data ecosystem, speed and efficiency are paramount. Whether you’re building real-time analytics pipelines or scaling distributed systems, the bottleneck often lies in data serialization and transport. Enter Apache Arrow Flight—a high-performance RPC framework designed to move large datasets efficiently using the Arrow memory format. What is Apache Arrow Flight? Apache Arrow […]

The post Accelerating Data Transfer with Apache Arrow Flight first appeared on TO THE NEW Blog.

From tasks and streams to tranquility: My first real project with Snowflake dynamic tables

Isha Vason — Wed, 23 Jul 2025 08:50:55 +0000

Let me tell you about the moment I realized I’d been overcomplicating things for years. I was working on a pipeline in Snowflake. You know the type — a multi-stage transformation process where a few base tables feed into intermediate tables, some reconciliation happens, and eventually it all lands in a final reporting layer. I’d […]

The post From tasks and streams to tranquility: My first real project with Snowflake dynamic tables first appeared on TO THE NEW Blog.

Using AI to build my first Android app: What worked and what didn’t

Abhishek Chauhan — Sun, 25 May 2025 12:37:20 +0000

Initial Thoughts Having spent over a decade working in software development and data engineering, I thought, where is AI right now? Is it capable of eliminating the developer or is there still some time? So, I challenged myself with building an Android app. That was an unfamiliar area for me. While it intrigued me, there […]

The post Using AI to build my first Android app: What worked and what didn’t first appeared on TO THE NEW Blog.

Automating Code Reviews Using OpenAI and GitHub

Kedhar Praveen Natekar — Mon, 05 May 2025 10:31:11 +0000

The State of Code Reviews in Today’s Development Landscape: In today’s fast-moving world of software development, AI has made remarkable progress. It can write code, debug errors, and even help design architectures. But let’s be honest, we’re not quite at a point where AI can take over the entire development process. Human developers are still […]

The post Automating Code Reviews Using OpenAI and GitHub first appeared on TO THE NEW Blog.

Snowflake Document AI : Unlocking Insights from Unstructured Data

Akshay Vijay Girulkar — Sun, 09 Mar 2025 04:27:28 +0000

Fun fact! Around 80%-90% of the world’s data is unstructured. I was shocked when I read this fact. The unstructured data contains images, emails, PDF files social media posts, and other formats. Even though it is widely present 70% of data is not being used to drive insights and get analytics. As a Data Engineer, […]

The post Snowflake Document AI : Unlocking Insights from Unstructured Data first appeared on TO THE NEW Blog.