Building Data Platforms That Transform Complex Data Into Intelligent Action.
I am Santosh Pothnak, a Lead Data Engineer & AI Solutions Architect with 16+ years of expertise. I bridge the gap between high-scale Data Engineering, BI Architecture, and Generative AI by designing PySpark pipelines, Medallion Lakehouses, and autonomous AI/Data agents.
Profile
For over a decade and a half, I have partnered with enterprise organizations to solve their most demanding data scale and modeling challenges.
My expertise centers on architecting highly optimized PySpark distributed pipelines, Medallion-structured Lakehouses, and enterprise Power BI/Fabric semantic models. Today, I am focused on the next frontier: deploying autonomous AI and Data Agents. By combining LLM Orchestration (using frameworks like LangChain and LlamaIndex) with modern vector databases, I help teams unlock conversational corporate intelligence directly on top of their big data repositories.
Work Experience
BI & Data Architect (Data & AI Platform)
- Architecting the migration of legacy operational database layers to an Azure Cloud PGFS lakehouse for United Health Group (UHG) / Optum, leveraging PySpark to transform raw PostgreSQL sources into highly optimized Medallion semantic tables.
- Designing and deploying metadata-driven Data Agents inside Microsoft Fabric to automate healthcare data asset discovery, cataloging, and structural lineage across multi-terabyte environments.
- Bridging big data with analytics by engineering Fabric Gold-layer architectures optimized directly for Power BI Embedded reporting and downstream RAG AI models to support conversational healthcare analytics.
Application Development Team Lead (Data & AI)
- Developed a Learning Spend AI Agent using Python and LangChain that autonomously queried Power BI dataset APIs to isolate spend patterns and generate natural language MIS narratives for senior executives.
- Designed and deployed Retail Sales & Operations Dashboards for Marks & Spencer using Power BI to monitor retail sales performance, freezer temperature compliance, and network-wide operational metrics, reducing product spoilage by 15% through proactive temperature alerts.
- Automated end-to-end data ingestion pipelines, integrating Power Apps, Power Automate, and Power BI Embedded, cutting manual report processing overhead by 40% while preserving strict 99.9% system SLA compliance.
Technology Lead (Data Engineering & BI)
- Re-engineered legacy siloed databases into unified enterprise data pipelines, enhancing dashboard and report refresh times by 50% through advanced SQL query tuning and Python-based automation.
- Programmed custom analytical data parsers in Python to identify market shifts, converting raw unstructured data into clean structured elements for corporate decision-making frameworks.
- UNIPER (Germany): Led cloud reporting migration, designed comprehensive migration data models, and managed universes in SAP BusinessObjects.
- Anglian Waters (UK): Managed a reporting team of 4 to migrate legacy Crystal reports to Web Intelligence (Webi), completing critical data model updates.
- E.ON (Germany): Supervised regression testing of 800+ upgraded reports during a major migration from SAP BO 3.1 to 4.1.
- CA Technologies: Migrated enterprise databases from SQL Server to Teradata, fixing integrity issues and writing custom Webi reports.
SAP BI Consultant
- Engineered complex universe structures and information models using SAP Business Objects, establishing robust semantic definitions that served as the predecessor for modern cloud metadata warehouses.
- Wildlife Conservation Society: Developed analytical dashboards for donation tracking and donor engagement, completing extensive data profiling and cleansing.
- Schwan Food Company: Replicated Cognos reports and calculations in SAP BO 3.1 universes and developed Webi and Crystal reports using stored procedures.
- Hyderabad Chemicals Limited: Designed operational universes and built sales KPI dashboards with multi-level drill-down capabilities, integrating non-SAP data via Data Services.
- Sesa Goa Limited: Designed custom mining operations universes and production dashboards using SAP BI 7.0 for management tracking.
Software Engineer
- Set up software infrastructure and supported reporting requirements using SAP BOBJ for production-readiness verification.
- Cornell University: Executed data profiling, cleansing, match, and consolidation tasks to improve data quality using BODS 3.2.
- Verifone: Tested Web Intelligence (Webi) reports after database migration from Oracle R11 to R12 to ensure data consistency.
Software Developer
- Debugged application code across Java, HTML, and PHP to ensure secure system integrations and proprietary features.
- Employee Management System: Built a console-based CRUD system in Core Java implementing file-based persistent storage, exception handling, and object-oriented design.
Projects & System Architectures
Key client engagements and technical platforms built across my career.
Fabric Metadata Data Agents
Designed active scanning agents within Microsoft Fabric to automate data asset discovery and cataloging, while constructing semantic Gold layers optimized for downstream conversational RAG models.
- Deployed metadata-driven Fabric agents to automate schema and lineage mapping.
- Engineered semantic architectures integrated with downstream Vector DBs.
- Bridged corporate lakes with conversational analytics interfaces.
Learning Spend AI Agent
Built an autonomous AI agent using LLM orchestration to query enterprise BI dataset APIs, isolate anomalous spend patterns, and generate natural language MIS reports for corporate executives.
- Developed agent flows using LangChain and advanced Python scripting.
- Queried Power BI Embedded dataset APIs directly to extract key metrics.
- Generated conversational narrative reports with zero human intervention.
Retail Sales & Operations Dashboards
Designed retail performance dashboards monitoring network-wide sales, freezer compliance, and operational check compliance to prevent inventory loss.
- Visualized retail sales performance by region and categories.
- Created temperature check trackers and compliance alert alerts.
- Reduced network product spoilage by 15% through proactive alerts.
Azure Cloud PGFS Lakehouse
Led the migration of legacy operational reporting environments to a modern Azure-hosted Medallion Lakehouse using distributed PySpark clusters.
- Transformed raw PostgreSQL databases into high-performance semantic layers.
- Architected Gold-layer structures using Bronze-Silver-Gold partitions.
- Optimized distributed pipelines for analytical dashboard ingestion.
BusinessObjects Cloud Migration
Aimed at migrating UNIPER’s existing reporting environment to the cloud, creating a sustainable data model, and ensuring seamless integration of reports.
- Designed a comprehensive data model for the migration activity.
- Migrated reporting systems to the cloud and resolved integration issues.
- Developed and maintained reports and universes in BusinessObjects.
AWS Reporting Solution
Adapted existing reporting systems to support updated data models while developing new reports and maintaining system integrity during migration.
- Migrated Crystal reports to Webi after significant data model updates.
- Managed a team of four to develop reports and address user requests.
- Coordinated with the ETL team for accurate data integration.
One2Two Regression Testing
Ensured a smooth transition during the major upgrade of trading reporting systems from SAP BO 3.1 to 4.1, minimizing errors and maximizing performance.
- Conducted regression testing for 800+ upgraded reports.
- Created and executed test cases, ensuring data accuracy and functionality.
- Delivered development support for Webi reports.
EIW Migration & Development
Focused on high-scale database migration from SQL Server to Teradata, resolving data integrity errors and improving performance.
- Migrated databases from SQL Server to Teradata and fixed integrity errors.
- Developed Webi reports and enhanced system performance post-migration.
- Supported UAT deployment and handled end-user issues.
Sales and KPI Dashboards
Developed dashboards that provided sales and performance analysis, enabling data-driven decision-making for regional and product-specific trends.
- Designed mock-ups and developed multi-level drill-down capabilities.
- Brought non-SAP data into BOBJ via Data Services and designed universes.
- Created KPI dashboards using column, bar charts, and scorecards.
Data Quality & Cleansing
Enhanced data quality and integrity for Cornell University by performing comprehensive data cleansing and transformation processes.
- Executed data profiling and created jobs for cleansing and parsing.
- Performed match and consolidation tasks to improve data quality.
- Coordinated with stakeholders and uploaded reports to management console.
Cognos to BO Replacement
Replaced legacy Cognos reports with modern Webi and Crystal reports over RDBMS, supporting sales, expense, and route performance lines.
- Analyzed and replicated Cognos calculations and functionalities.
- Created Universes in BO 3.1 tailored to Cognos reports.
- Developed Webi reports using stored procedures and advanced logic.
Wildlife Conservation Dashboard
Created an operational reporting dashboard for tracking global donations and donor engagement, enhancing data transparency.
- Developed dashboards for donation analysis and reporting.
- Conducted data profiling and cleansing for accurate reports.
Verifone R11 to R12 Migration
Ensured data consistency and reporting functionality after migrating databases to the upgraded Oracle environment.
- Tested and validated Webi reports after database migration from Oracle R11 to R12.
Employee Management System
Developed a simple console-based Employee CRUD system focusing on practical exposure to Core Java principles like OOP, exception handling, and file storage.
- Developed basic CRUD operations for employee records using Core Java.
- Designed and implemented console-based user interface.
- Used file handling to store and retrieve data persistently.
Community & Leadership
Evangelizing data technologies, hosting developer forums, and training the next generation of engineers.
Global Education & Mentorship
- Instructor of popular online courses focusing on enterprise BI, including "Getting Started with Power BI" and "Understanding Key DAX Functions".
- Created a specialized course enabling Project Managers and Scrum Masters to connect Power BI with Azure DevOps (ADO) data via OData feeds and Analytics Views.
- Maintained high course ratings while answering technical queries and coaching learners globally on data architecture.
Speaking & Community Impact
- Core Member of the Hyderabad Data & AI Community, driving local tech outreach, developer networking, and Fabric adoption.
- Microsoft Fabric Bootcamps: Delivered sessions at Microsoft Offices in Hyderabad (300+ attendees) and Bangalore (400+ attendees) on semantic modeling and end-to-end data platform scenarios.
- Conference Speaker: Guest speaker at MACC3 (Microsoft Analytics Community Conference) and Data Toboggan 2025, presenting Microsoft Fabric Data POC architectures.
- Academic Mentorship: Conducted Power BI workshops at IIM Raipur (2 sessions) and PSG College of Arts & Science; guest lectured at Acharya Nagarjuna University UGC National Seminar.
- Industry & Hackathon Panelist: Served as a panelist at Siva Sivani Institute of Management (SSIM) and invited back as a Hackathon Jury Member at CBIT (Oct 2025).
Technical Skills
A cleanly categorized mapping of my technologies, architectural focus, and credentials.
Data Engineering
- PySpark
- Microsoft Fabric
- Databricks
- Snowflake
- Medallion Architecture
- ETL/ELT Pipelines
- PostgreSQL
- Azure Synapse
AI & Automation
- Autonomous AI Agents
- LLM Orchestration
- RAG
- Vector Databases
- Python
- Power Automate
BI & Semantic Layer
- Enterprise Power BI Architecture
- DAX Modeling
- Power BI Embedded
- Semantic Layer Modeling
- SAP Business Objects
- MicroStrategy
Programming & Web
- Python
- Java
- SQL
- HTML & CSS
- PHP
- MySQL
Professional Certifications
Education
Bachelor of Engineering (ECE)
Muffakham Jah College of Engineering & Technology, Hyderabad
Class of 2005 – 2009High School Diploma / Intermediate (MPC)
Vikas Junior College (aka Holy Cross Co-operative Junior College)
Class of 2003 – 2005SSC (Regulars)
Mt. Helicon Public School
Class of 1993 – 2003Get In Touch
Have a pipeline to optimize, an AI agent to architect, or a data model to structure? Let's connect.
Contact Information
Feel free to reach out through the form or directly via the details below.