Executive Summary

In the three months of Q4 2025, Pilotly’s product development team transitioned from a traditional engineering kanban to an AI native, agentic development model. As VP of Product & Engineering, I was tasked with championing this rapid adoption across engineering, design, and product.

Pilotly is a SaaS market research platform that tests TV shows, film, and other media for major studios like Amazon, Netflix, and Disney. A small company of 20, Pilotly’s competitive advantage has always been technical innovation, so we knew fast adoption of AI would be critical to maintaining our industry leadership.

Problem

Prior to Q4 2025, Pilotly’s product development process resembled that of most high-functioning start-up SaaS organizations: A small rockstar team overwhelmed with business priorities juggling features with tech debt. Pilotly had a strong team, but we were slow & reluctant to embrace AI due to competing priorities and tech debt. This was hurting velocity and limiting future opportunity for the company.

Strategy

Despite all the technical advances of AI, the barriers to maximizing enterprise value remain human ones. Engineers, PMs, and designers are learning completely new ways to build, and to motivate the team, I approached them with empathy and empowerment:

1. Aligning the team with a mission-driven objective
2. Strategically organizing tasks and supporting each team member personally with early wins
3. Fostering collaboration and autonomy to iterate on new workflows, rather than dictating process

Successful leaders build the right team and get out of the way. To quote one of my favorite TV shows:

“Success is not about the wins and losses. It’s about helping these young fellas be the best versions of themselves.”
— Ted Lasso

Results

The team went from zero AI usage to a fully agentic model within three months. Engineering began developing with Claude within 2 weeks, while product and design upskilled their roles throughout the quarter.

AI didn't just help us ship more, it changed how fast we could explore, iterate, and deliver the right solutions. Across every performance metric, the team saw massive gains:

• 3x product velocity based on features shipped
• 200K+ lines of AI code per month
• 30% more weighted output
• 43% less hotfixes
• 5x coding velocity

Setting Objectives & Key Results

As in most start-ups, CEO’s have the vision, and they often see the potential of a new market or technology before the rest of the company. This was no different at Pilotly. James Norman is an accomplished engineer himself, who wrote much of the initial codebase at Pilotly. He recognized that AI was going to change the game, and he immediately began cooking up new products and pushing new tools on the team.

When it came time to create our OKRs and roadmap for Q4, we knew we wanted the team to start building with AI. So our initial plan was to create a huge product objective of “complete new dashboard” which would require the team to adopt AI to build faster. But looking at this objective I realized I had it backwards. My objective in the quarter was not a product, it was the AI adoption itself:

So I rearranged the OKRs so our main objective was an organizational one, rather than a product one. This helped to set the tone with the team that the destination was not the objective, the journey was.

The AI Transformation

Phase 1: Alignment and Early Wins

While several engineers had begun using AI, it was limited to personal experimentation using ChatGPT and Cursor. To kick off our Q4 objective, I wanted to unite the team to our mission with intention. Often engineers respond best to other engineers, so to help set the direction, I brought in a former colleague and engineering executive whom I hugely respect: Jono Spiro. Jono has helped hundreds of early stage start-ups in his role as Head of Engineering at Fractal Software. In addition to his technical expertise, he is effective through empathy by speaking with teams, not at them.

We officially kicked off our OKR effort in October with a workshop from Jono where he discussed the benefits of agentic development and shared his set-up. Discussing his daily productivity was eye opening to the team:

“I log in and assign tasks to my team of agents over my morning coffee. Then I walk my dog, go to the gym, and come back to review their work over lunch. I don’t write code anymore. I manage my agents like I would a scrum team.”

The whole team was engaged and energized. Over the course of the next several months Jono was a valuable resource to the team, helping everyone with their Claude configurations, refining our rules markdowns, and joining our new AI-dedicated slack channel.

Once the full team was set-up with Claude in terminal, I lined up some very basic and simple tasks for each engineer to do. This was a critical strategic step to get those early wins and help each engineer have their own personal “ah ha” moment. For one engineer, that was a simple HTML style update. For another, it was asking Claude to review the codebase for potential performance improvements.

The first meaningful milestone was the delivery of a feature largely built with AI assistance: adding export capability to tune-out open-ends. The feature was completed faster than expected with no bugs, validating the potential of AI-assisted development. These early wins gave the team confidence, and helped us look beyond “can I do it?” to “now let’s do it, faster.”

Phase 2: Building Momentum and Redesigning Workflows

Following team alignment, adoption accelerated rapidly. By the end of October, all engineers were actively using AI tools as a core part of their workflow. Team milestones started to rack up:

• October 15: First feature developed with AI
• October 20: Entire team submitting PR’s written by Claude
• October 28: Complex API integration written completely with AI (~18K LOC)

These milestones demonstrated that AI could not just help us code faster, but complete projects at production quality. I extended our Friday stand-up to a one-hour “Claude 9” working meeting where we could all celebrate wins, share best practices, and iterate on our workflows. This helped accelerate the shift from primarily writing code to:

• Structuring problems and prompts for AI systems
• Iterating our rules markdowns
• Creating documentation to give AI better context
• Refining agents and running them in parallel

This shift resulted in a fundamental change in how work was approached. The team moved from task execution to orchestrating solutions. This reduced dependency on specific roles for execution and gave the team more flexibility. AI empowered the team to have more ownership and agency. For example: engineers did not lose time waiting if design had forgotten to spec an error state - AI could fill in the gap.

My role drastically changed during this time as well. I no longer spent time writing out detailed product specs or scoping with the engineers. I found myself busy just keeping up with all the work that was being done! The team was off and running, and I was along for the ride. An early sprint retrospective captured the excitement across the team:

Phase 3: Expanding AI with Design & Product

“Writing code faster” is just one aspect of an AI native organization. To truly increase product velocity you need efficiencies across the entire development workflow, and those became apparent as design and product adopted AI in their daily work.

In November, the team conducted its first review of an AI-generated design. Sentiment is one of the most visited pages in the Pilotly dashboard, and up to this point it was a read-only page. CRUD UX/UI was added through a prompt, rather than a figma file. The quality and thoroughness of the output gave our lead Product Designer her “ah ha” moment, and built confidence in using AI more broadly to direct frontend builds.

With AI acting as an intermediary, design began iterating directly within production code rather than static mockups. This reduced the need for traditional handoffs and enabled faster cycle times. By January 2026, design took one further step and began submitting and merging their own pull requests! This marked a structural shift in the dev process. Design was no longer dependent on engineering to implement refinements, significantly reducing latency between concept and execution. We all celebrated Masumah’s achievements as it demonstrated how far the team has evolved in just three months.

Parallel to design cutting their first PRs, product management was also expanding their scope and capabilities. AI tools enabled PMs to engage more directly with the codebase, improving their ability to:

• Investigate and isolate bugs
• Validate edge cases
• Participate more actively in QA processes
• Automate manual operations

As a result, PMs became more effective contributors to overall product velocity. They were able to resolve or narrow down issues before involving engineering, reducing context-switching and accelerating resolution times. Additionally, AI helped product to automate several manual tasks further improving both speed and quality across the team.

A few weeks later, PMs began shipping their own code, marking the team’s full transition to an AI-native development model. While systems still need to be expanded and refined, the leap from zero AI-generated production code to designers and product managers independently delivering features in just a few months underscores a fundamental shift: when teams are truly empowered with AI, the pace and ownership of innovation accelerate dramatically.

Results

The team adopted a fully agentic workflow within three months, tripling product velocity and exceeding all OKRs in Q4:

AI adoption accelerated faster than expected, with all engineers developing with Claude within 2 weeks. Lines of code (LOC) committed with AI increased from 39k in October to over 200K in February and March.

Performance Outcomes

3x More Features Shipped

Following the introduction of AI-assisted workflows, the team tripled product velocity, measured by number of features shipped quarter-over-quarter. The team impacted every area of the product, shipping so many new features and enhancements that it became difficult to capture them all on a single slide during the company’s OKR review:

Because feature count alone does not tell the full story, we also evaluated output using weighted ticket analysis. Across every metric, comparing Q3 2025 (pre-adoption) and Q1 2026 (post adoption) shows increased engineering performance:

• 30% more weighted output
• 78% more complex tickets
• 43% fewer hotfixes

Every engineer saw a boost in their individual output, including Leo who was out for weeks on vacation. NOTE: Leo’s agents were inactive on vacation, he was fully unplugged 🤙

Underlying both product and engineering performance was a ~5× increase in total raw code throughput (58K → 208K lines changed), highlighting the scale of acceleration in overall development speed.

AddedRemovedTotal Changed

Product Results

Several of the key product launches in Q4 focused around the new sentiment UX/UI in dashboard, which is one of the most valuable pages for our clients. Currently features are spread across the old “classic” dashboard and Pilotly’s all new “delta” dashboard. Q4 brought the new dashboard up to parity with legacy, resulting in more usage and positive feedback from clients. Page analytics improved substantially:

Improvements in dashboard performance was another major win for the company in Q4. Pilotly’s dashboard was not initially designed for the current scale of the company, so frequent CPU spikes caused slowdowns and delayed the research team completing their work. AI identified many improvements that resulted in removing approximately all CPU spikes.

NOTE: Spikes observed after AI improvements are due to system deployments.

Reflections

Key Success Factors

Several factors were critical to the success of this AI transformation:

1. Bottom-Up Adoption: Giving engineers the space to experiment and have their personal “ah ha” moment created early proof points. This enabled broader adoption based on demonstrated value rather than a dictated mandate.
2. Cross-Functional Empowerment: Extending AI beyond engineering to product & design further increased velocity by removing systemic bottlenecks.
3. Willingness to Redefine Roles: The team did not attempt to preserve traditional role boundaries. Instead, responsibilities evolved naturally as new capabilities emerged.

Conclusion

Pilotly’s transition to agentic development demonstrates that meaningful gains from AI are not limited to just building faster. When adopted broadly and integrated into workflows across functions, AI can fundamentally reshape how product teams operate. While further opportunities remain—particularly in dev ops and system-level automation—the current state already represents a step-change in both velocity and capability.

Reflecting on the past several months, my main takeaways from championing this team through their transition are twofold:

1. AI adoption is dependent on a culture shift. The main challenges are personal and organizational, not technical.
2. AI’s impact is maximized not when it accelerates existing workflows, but when it enables teams to redesign them entirely.

I could not be more proud of the team and all their progress over the past several months. While the AI journey remains ongoing, the team has truly been empowered, and seeing their personal growth has provided me with my own “ah ha” moment.