
Mastering CI/CD in Azure Databricks: From Dev to Production
In the world of data engineering, moving from manual execution to automated deployment is a massive leap in maturity. Many engineers master the logic of building a Medallion architecture.
Data Engineer & Microsoft Fabric Architect with 8+ years building governed lakehouse pipelines (ADF, Databricks, Fabric) and enterprise Power Platform automation. I deliver data trust—CI/CD-ready delivery, PII-aware governance, and measurable operational impact.
Get to know me better
I’m Charles McFail—a Data Engineer & Microsoft Fabric Architect who thrives where data engineering meets real-world business impact. Over the past 8+ years, I’ve built “digital backbones” for both public-sector scale (K–12 education) and global enterprise (Omin), turning fragmented systems into trusted, governed, decision-ready data ecosystems.
My north star is simple: architecting trust in data—so leaders can act fast, teams can self-serve confidently, and platforms stay secure as they scale.
My Approach: design and deliver data systems the way high-performing engineering teams operate:
What I Do: end-to-end data + automation that scales
Data Engineering (Azure + Databricks + Fabric)
Power Platform (Strategy + Build + Adoption)
Business Intelligence (Power BI that drives action)
What people say about working with me

Cloud Engineer at IPG Omnicom
I have had the pleasure of working with Charles many times, over the last few years. Most recently, we partnered to develop Power Platform solutions for internal data classification reports. Charles is a partner who is thoughtful in his approach, listens as well as he speaks and takes the time to understand the dynamics (both political and technical) in a project. Looking forward to working with him again.
A comprehensive overview of my technical proficiencies and tools I work with daily
My professional journey
IPG Omnicom
•full-timeArchitecting enterprise-scale data ecosystems and automation frameworks within a high-volume, multi-tenant environment. Serving as the Application Developer and "Internal MSP," delivering governed, AI-ready products that bridge the gap between complex Azure infrastructure and executive-level business intelligence.
Houston Independent School System
•full-timeLed district-wide digital transformation for the 8th largest school system in the U.S., supporting 15,000+ employees. Focused on building resilient data frameworks and communication systems where accuracy and stakeholder alignment were mission-critical.
Freelance
•freelanceCollaborated with a range of clients to design and deliver custom websites and web applications, owning projects end-to-end—from discovery and scoping through build, launch, and post-launch support.
Spring Independent School District
•full-timeBuilt the communication and coaching foundation that powers my technical leadership today—translating complex systems into clear, usable understanding for diverse audiences.
My academic background

Prairie View A&M University
Biology
Under the mentorship of Dr. Lane in the department of Biology, created Prairie View’s first pilot program for the student utilization of iPads in the classroom in a 1-to-1 setting. Presented the findings of the iPad pilot program at Texas A&M’s Teaching With Technology Conference.

Morehouse College
Biology
Research assistant at the Morehouse School of Medicine, I researched the genetic drivers behind Vibrio cholerae biofilm production. I analyzed mutant strains to understand how quorum sensing triggers the phenotypic shift from single cells to attached communities.
Some of my best work


Enterprise-grade analytics platform for data-driven decisions
Professional credentials and certifications
December 5, 2023
for
Microsoft Applied Skills: Create and manage canvas apps with Power Apps

Microsoft
Credential ID:
ABCFE81585AC788F
November 14, 2023
for
Microsoft Applied Skills: Create and manage automated processes by using Power Automate

Microsoft
Credential ID:
74B2D41E7A7EF787
May 4, 2021
for
Linux knowledge that provides real-world practice with the command line and virtual machines.

Linux Professional Institute
Credential ID:
99cl7bttff
Milestones and recognitions
Developer Conference (dev.events)
Created the web application, Smart Eye, intended for health professionals to use a secure streaming environment with chat functionality, and two-way live video by utilizing Rest APIs from Wowza Media Systems and Yubikeys from Yubico for hardware authentication.
Learn More
Texas A&M’s Teaching With Technology Conference
February 2013
Created Prairie View’s first pilot program for the student utilization of iPads in the classroom in a 1-to-1 setting. Presented the findings of the iPad pilot program at Texas A&M’s Teaching With Technology Conference.
What I can do for you

Design and implement a governed, scalable lakehouse that turns fragmented sources into trusted, decision-ready data products.
I build the “digital backbone”: ingestion → transformation → curated outputs, with security, observability, and repeatable delivery patterns baked in.
Pricing
Market-based package pricing derived from senior DE contract economics (often ~$80+/hr on freelance benchmarks), plus the additional premium for architecture + governance + production hardening.
Timeline
6–14 weeks depending on source complexity, volume, and governance requirements

I build Power BI dashboards tied to Power Apps/Automate on governed Fabric data, so teams act on insights fast.
This service turns reporting into a closed-loop system: reliable data → clear insights → automated action. I align on outcomes (KPIs, bottlenecks, compliance), then build governed data and consistent metrics with secure, high-performance Power BI. Finally, I connect insights to Power Apps and Power Automate so teams can trigger approvals, escalations, and exception handling directly from the dashboard.
Pricing
Aligned to senior build + rollout cycles and consistent with market DE/analytics contracting benchmarks
Timeline
4–10 weeks depending on workflow count, data readiness, and security model
Thoughts, tutorials, and insights

In the world of data engineering, moving from manual execution to automated deployment is a massive leap in maturity. Many engineers master the logic of building a Medallion architecture.
Essential TypeScript patterns and practices for building maintainable, type-safe applications at scale.
Wherever you are in the world, let's work together on your next project.