Neon + Databricks
Issue
Neon + Databricks
Distilling Deep AI Insights from Leading Podcasts

Neon: A Serverless And Developer Friendly Postgres
Data Engineering Podcast by Tobias Macey · 57 min
Listen to episode →The podcast episode explores Neon, a cloud-native serverless version of Postgres designed to scale elastically and optimize the developer experience. Nikita Shamgunov, Neon’s founder, discusses the motivation behind creating Neon, emphasizing the need for a Postgres alternative that supports serverless cloud usage without sacrificing compatibility with Postgres’s extensive ecosystem of plugins and extensions. Neon achieves this by running an unmodified Postgres engine, intercepting its local storage calls, and redirecting them to a custom-built, scalable networked storage layer via a thin API, enabling separation of storage and compute. This architecture supports zero-cost branching, allowing developers to create isolated development and staging environments quickly, connected directly to GitHub PRs, which significantly accelerates modern development workflows. Neon targets small, agile developer teams who lack dedicated DevOps resources, focusing on simplicity, automatic scaling, and minimized operational overhead. Key engineering innovations include running Postgres in dynamically resizable virtual machines with live migration to maintain stateful TCP connections and implementing a serverless driver alongside a connection balancer to overcome Postgres’s traditional connection limitations, especially for edge environments like Cloudflare workers. The episode also addresses challenges like managing latency introduced by network hops in decoupled architectures and the complexities of Postgres version upgrades, highlighting Neon's ongoing efforts toward automated upgrade tooling. Neon’s contribution and support for AI-related features such as the pgvector extension underscore its commitment to embedding AI capabilities within its platform. Finally, the conversation situates Neon within the competitive landscape of cloud-native and open-source databases, explaining its cloud-only strategy and reflecting on lessons learned from other systems like SingleStore, with a focus on sustainable engineering and developer productivity.
Key Takeaways
- 01
Neon maintains full compatibility with Postgres by running the core engine unmodified and replacing only the local storage with a networked storage layer via a thin API that intercepts WAL and page requests, enabling serverless Postgres without breaking the extensive Postgres plugin ecosystem.
This approach is critical because Postgres's ecosystem is vast and many users rely on a wide range of plugins and extensions that assume traditional local disk storage. By intercepting storage interactions at the transaction log (WAL) and page level, Neon ensures seamless compatibility, allowing developers to use all existing Postgres features without modification. This design cleverly balances the innovation of serverless architecture with maintaining ecosystem trust and developer workflow continuity. It addresses broader industry challenges where modern cloud systems must integrate legacy technology without fragmentation. Implementing this required precise engineering 'surgery' on Postgres internals, given that Postgres was not originally designed for pluggable or remote storage engines. The significance lies in making legacy technology cloud-native while preserving investments made by developers and organizations in Postgres tooling and operational practices.
- 02
Focusing exclusively on a cloud-only, serverless Postgres product enables Neon to optimize user experience and scalability specifically for small, agile development teams that lack traditional DevOps resources, avoiding the dilution of effort found in hybrid cloud/on-prem strategies.
This strategic decision allows Neon to differentiate itself from legacy databases and competitors who support multiple deployment paradigms but face complexity managing divergent user needs. Targeting small teams acknowledges the rising market segment requiring turnkey, self-service cloud databases with no manual provisioning or scaling. By focusing purely on serverless cloud, Neon can build features such as automatic VM resource scaling with live migration, connection balancers, and seamless branching tailored to cloud-native workflows. This aligns with broader trends in cloud infrastructure and AI platforms emphasizing managed services that dramatically reduce operational burdens. The focus also supports faster iteration cycles for startups and agile teams, enabling speed and simplicity over supporting complex legacy on-prem architectures.
- 03
Neon's serverless implementation runs Postgres inside dynamically sized virtual machines with live migration to maintain stable, stateful TCP connections, addressing the challenge of scaling a traditional stateful database in a serverless environment.
Postgres is inherently stateful, maintaining persistent TCP connections which complicate elasticity and scaling in serverless contexts where compute resources often change dynamically. Neon’s innovative use of VM live migration allows compute resources—CPU and memory—to be adjusted on the fly without interrupting active connections or user sessions. This requires layered expertise in VM management, security isolation, and operating system integration to ensure seamless user experience. Such sophistication illustrates the complexity involved in delivering a truly serverless Postgres service and highlights how deep infrastructure engineering supports modern cloud-native database services. For businesses, this means Neon provides elastic scalability and operational transparency without sacrificing reliability or session continuity.
- 04
Neon introduces a serverless Postgres driver and a connection balancer (PG balancer) to overcome Postgres's well-known challenge of managing numerous connections, especially addressing edge cases like Cloudflare workers that lack TCP support.
Postgres traditionally struggles with scaling a large number of simultaneous connections, and environments like Cloudflare workers exacerbate this by limiting TCP support. Neon’s serverless driver abstracts connection management and employs PG balancer to pool and scale connections automatically, packaging complexity away from developers. This greatly simplifies integration, increases reliability, and assures that connections never run out, which is critical for serverless and edge-native applications. This solution reflects ongoing industry trends to cater to developer productivity by reducing operational complexity around database connections. Examples highlighted include simplifying application integration in edge functions and making Postgres accessible for modern serverless workloads.
- 05
The separation of storage and compute enables Neon's zero-cost branching feature that allows developers to create instant, isolated copies of databases tied to GitHub pull requests, facilitating advanced development workflows and rapid feature testing.
This architectural decoupling means that database branches are cost-effective because they rely on metadata operations rather than duplicating large amounts of data. Developers gain the ability to safely modify data and schema in their own isolated environments without affecting production or other branches. The branches can be promoted downstream through staging and production environments, integrating with schema migration tools like popular ORMs (Prisma, Drizzle, TypeORM). This innovation represents a significant advance in database-backed application development, reducing conflicts, accelerating iteration, and fitting neatly into CI/CD pipelines and modern JavaScript ecosystems, including frameworks like Next.js and deployment tools like Vercel.
- 06
Neon manages the latency impact from network hops introduced by storage and compute separation through optimizations leveraging existing Postgres synchronous replication paradigms and using the Paxos protocol within their Safekeepers component to ensure reliability without significant performance degradation.
While separating storage and compute naturally adds network round-trips compared to monolithic architectures, Neon argues that this is comparable to traditional Postgres setups using networked storage like AWS EBS volumes. The Paxos consensus protocol employed by Safekeepers guarantees data durability and consistency across replicas, mitigating reliability concerns. This design yields nearly infinite IO throughput due to Neon’s multi-tenant storage, which can scale to large databases demanding high throughput. The slight latency trade-off is acceptable against these throughput and scalability advantages. This reflects continuing tension in cloud database design between latency sensitivity and scalability, informing adoption decisions for different workload profiles.
- 07
Neon is actively developing automated and simplified Postgres upgrade processes, recognizing the historical complexity in Postgres upgrades involving multiple versions and storage format migrations, and drawing lessons from simpler, more reliable processes like those in SQL Server.
Upgrading Postgres has traditionally required careful orchestration, including running multiple versions concurrently and manual intervention to maintain storage compatibility, which poses operational risks and downtime threats. Neon treats upgrade automation as a core feature under development to alleviate these burdens, aiming to provide seamless, failure-resistant upgrades in a serverless cloud context. This is significant for increasing operator confidence, reducing labor costs, and aligning Postgres with cloud native practices. The comparison to SQL Server’s atomic restart upgrade, which guarantees failure-free upgrades, highlights industry benchmarks Neon aspires to meet. This capability is critical for user trust and long-term platform viability.
- 08
Neon contributes directly to Postgres’s open-source ecosystem, particularly supporting the pgvector extension for embedding and similarity search, demonstrating its commitment to AI workloads and cloud optimization while fostering community collaboration.
Rather than building proprietary AI embedding vectors, Neon supports and contributes code to the widely used pgvector extension, enhancing its performance on Neon’s platform. This dual role as both a cloud service provider and open source contributor exemplifies modern business models leveraging open innovation. Given the growing importance of AI and semantic search in database usage, Neon’s alignment with open standards and ecosystem tools positions it well for adoption among developers integrating AI into their applications. It reflects a broader industry trend where integrating AI capabilities into traditional databases is a strategic imperative, and performance optimizations in cloud environments provide competitive differentiation.
- 09
Neon's engineering updates to Postgres have been kept minimal and modular via extensions and limited forking, enabling easier merging with future upstream versions and ensuring platform sustainability as Postgres anticipates native support for pluggable storage engines.
Minimizing core Postgres modifications while pushing functionality into extensions preserves compatibility with the broader Postgres community and reduces maintenance overhead. Neon anticipates that upcoming Postgres releases (versions 18 or 19) will natively support pluggable storage engines, aligning with Neon's vision and easing future integration. This strategy balances innovation with ecosystem continuity, ensuring Neon remains stable, upgradable, and aligned with developer expectations. It highlights an important trend in database evolution toward modularity and pluggability, which are essential for cloud-native architectures and serverless systems. For businesses and developers, this means confidence in the platform's longevity and ability to adapt as Postgres evolves.

Databricks Founder Ion Stoica: Turning Academic Open Source into Startup Success
Training Data by Sequoia Capital · 60 min
Listen to episode →In this episode, Ion Stoica, co-founder of Databricks, discusses the transition of academic open-source projects like Apache Spark and Ray to successful ventures in the tech industry. Stoica emphasizes the foundational role of open-source software in fostering innovation and trust, allowing enterprises to customize tools for their needs. He highlights the importance of strategic partnerships, particularly with Microsoft, which significantly accelerated Databricks' growth and established Spark as a dominant tool for data engineers and AI practitioners. Stoica shares insights for aspiring founders, urging them to focus on identifying emerging problems, as he believes addressing issues that will grow in importance is key to long-term success. The conversation also navigates the complexity of AI infrastructures and the challenges businesses face in maintaining effective data governance. Stoica points out the potential commoditization of AI models and the implications that could have for innovation. Ultimately, the episode reveals the continuous interplay between academic research and commercial success, showcasing how foundational technologies can evolve into mainstream solutions to tackle pressing challenges in the AI landscape.
Key Takeaways
- 01
Open-source as a strategic foundation for innovation.
Ion Stoica elucidates that open-source software is vital for Databricks, promoting transparency and enabling enterprises to modify tools according to their requirements. This approach cultivates trust among users and invites community contributions, resulting in more effective and advanced solutions. For example, tools like Spark and MLflow have benefitted from this collaborative development, allowing businesses to adapt quickly to market changes. In the current AI landscape, this trend aligns with the push for democratized technology access, fostering a culture of innovation while addressing customization needs.
- 02
The importance of strong partnerships in startup growth.
Stoica stresses that strategic partnerships, notably with Microsoft, were monumental in Databricks' ascent within the tech landscape. This alignment not only provided essential resources but also integrated Databricks' offerings into a larger ecosystem, facilitating a broader reach in the marketplace. The collaboration with Microsoft has enhanced Spark's adoption among data scientists, illustrating the crucial nature of partnerships in scaling a startup's operations. As trends show increased competition in AI, startups will need to seek similar alliances to augment their visibility and credibility.
- 03
Transforming academic research into marketable technologies.
Stoica highlights the smooth transition of open-source projects from academic settings to commercially viable products through Databricks. This shift underscores the significant role academic contributors play in the tech ecosystem by turning theoretical advances into practical applications. By creating a platform optimized for Spark, Databricks secured its position as a leader in the data engineering space, showcasing how leveraging academic insights can lead to transformative business models. In a time where continuous innovation is paramount, this trend encourages universities to engage more closely with industry dynamics.
- 04
Identifying long-term challenges as a pathway for innovation.
Stoica emphasizes that entrepreneurs should focus on problems expected to grow in importance, as this gives them a competitive edge. By constructing solutions that address future needs, founders can establish themselves as pioneers in emergent markets. Such foresight is crucial in fast-evolving arenas, like AI and data governance, where proactive measures are often more beneficial than reactive strategies. This proactive mentality is increasingly critical for success amid rising expectations in the tech sector regarding agility and innovation.
- 05
Navigating complexity in AI infrastructures.
The podcast identifies the complexity arising within AI ecosystems and its implications for businesses. Stoica articulates how these intricate infrastructures demand that companies develop better data management and governance tools, as they grow more intertwined with business operations. Technologies, such as Unity Catalog for data governance, emerge as responses to these challenges. As businesses grapple with adherence to data privacy regulations, resolving the complexity in these systems is vital for sustainable operations and consumer trust.
- 06
The potential commoditization of AI models.
Stoica discusses the risk that widespread adoption of AI models could diminish unique competitive advantages, leading to a marketplace where even advanced functionalities are widely available. He contrasts open-source models with proprietary solutions, suggesting that while open-source democratizes access, it may discourage significant investments in R&D. This shift could challenge startups as they aim to carve out unique offerings in a crowded field. Companies must thus strategize to maintain value differentiation as AI resources become ubiquitous.
- 07
Role of large language models in advancing AI capabilities.
The conversation delves into how advancements in large language models, exemplified by architectures like GPT-4, possess significant implications for the future of AI applications. Stoica points out that the capabilities of such models enable new use cases across various industries, ultimately transforming how businesses leverage AI for enhanced effectiveness. As the demand for comprehensive AI solutions rises, understanding and integrating these larger models into offerings will be vital for companies seeking to maintain a competitive edge in technological landscapes.
- 08
Emphasizing the need for control and visibility in AI applications.
Stoica notes that enterprises increasingly demand visibility and auditability regarding data governance in AI applications. This pressure arises from growing concerns about data privacy and compliance, where stakeholders seek assurance that AI models operate within ethical and regulated frameworks. The importance of solutions that provide this control reflects broader trends in corporate governance, wherein transparency is integral to maintaining consumer trust. Companies that prioritize these aspects in their AI deployments will be advantageous in meeting modern consumer expectations.
- 09
The necessity of adaptability in tech entrepreneurship.
Stoica argues that technology entrepreneurs must retain adaptability to navigate the rapidly changing AI landscape. His insights highlight the importance of recognizing shifts in user needs and market demands, emphasizing the need for systems that are not only innovative but also aligned with the evolving industry environment. Startups that embrace this adaptability can better position themselves to leverage emergent opportunities, thereby fostering sustainable growth amid competition. The dynamic nature of the AI industry amplifies this need, as new challenges continuously arise, requiring agile responses.
Questions answered in this issue
What motivated Neon to focus exclusively on a cloud-only serverless Postgres rather than supporting on-premises deployments?
Nikita explains that they learned from SingleStore’s experience that supporting both on-prem and cloud workloads diluted focus and prevented them from competing strongly in cloud analytics. They observed that serverless cloud deployment requires optimizations that differ from on-prem and that focusing solely on cloud products enables a sharper user experience and scalability. They also found no existing open source alternative to Aurora, which inspired them to build Neon as a cloud-only, serverless, Postgres-compatible alternative. This strategy allows them to prioritize developer experience and cloud-native scalability.
How does Neon maintain Postgres compatibility given that Postgres does not have a pluggable storage engine?
Neon runs an unmodified Postgres engine but replaces the local disk storage with a networked storage layer via a thin API layer. They intercept Postgres' storage calls, redirecting WAL records and page read requests over the network to Neon’s storage system. This approach lets them keep the full Postgres engine intact, preserving ecosystem compatibility and plugin functionality. While some surgical modifications were needed, the overall Postgres behavior and protocol remain unchanged, which is crucial because any break in compatibility leads to customer problems and loss of ecosystem support.
How does Ion Stoica view the relationship between academic research and industry?
Stoica believes that the boundary between academia and industry is increasingly blurred, with academic projects like Spark and Ray transitioning into viable commercial solutions. He emphasizes that this synergy is instrumental in addressing real-world problems, reflecting a practical approach to research. This transition allows for innovative ideas to be tested and implemented effectively in the market, leading to significant advancements in technology and improved solutions.
What are Stoica's thoughts on the current state of AI and its implications for businesses?
Stoica observes that AI ecosystems are becoming complex, which presents both opportunities and challenges for businesses. He notes that companies must navigate this complexity to harness AI's full potential effectively. This environment requires businesses to be agile and well-informed about multiple AI models and techniques to stay competitive. Therefore, keeping pace with advancements and understanding their implications is crucial for leveraging AI successfully.