Emerging Open Source Database Technologies

Last week, I came across two new table format projects — and it made me pause. So I went back to revisit the “open data stack” diagram I had once built live with another OSS founder, to see how things have evolved… and what’s still painful. Here’s what it looks like today 👇 🗂️ File formats: We often talk about Parquet and ORC as the backbone of the modern data lake. But reality is: many still ingest and store JSON, CSVs, and other semi-structured formats. And now, with AI workloads rising, we’re seeing new columnar and hybrid formats like Lance, Nimble, and Vortex — built for wide tables, random access, and unstructured data that Parquet can’t serve well. 📦 Table formats: Beyond the “big three” — Apache Hudi, Apache Iceberg, Delta Lake — we’re seeing experimentation again. Paimon brings an LSM-style storage layout, Lance blends blobs and vectors. New entrants like MSFT’s Amudai (already storing exabytes internally) and CapitalOne’s IndexTables, focusing on search and indexing, validate that there's still a lot of ground to cover here. 🔄 Interoperability: This is where the real work lies. Unification to “one format to rule them all” is a pipe dream — but interoperability is achievable. Projects like Apache XTable (Incubating) and DeltaUniform help bridge open formats and sync them into commercial engines (Onehouse, Databricks, EMR) and closed warehouses (Snowflake, Redshift, BigQuery). Let’s not forget: 70–80% of enterprise data still sits in closed formats today. 🧭 Open Metastores: We’re finally seeing viable Hive replacements — Polaris, UnityCatalog OSS, Gravitino, LakeKeeper. But there’s an important distinction: Open APIs (e.g. Iceberg REST) ≠ Open Metastore servers (operational catalogs). Most projects/vendors are building their own APIs to cover gaps, while exposing their tables as Iceberg using tools like XTable. We’re heading towards a messy N×M world of catalogs vs. formats. 🚧 Open problems: 1: Many Iceberg REST APIs are format-specific; we need multi-format APIs to future-proof interoperability. 2: Even beyond table formats, catalog data (policies, permissions, relationships) is still stored in proprietary databases — we need at least an open interchange format here. 3: The ecosystem still lacks actual, vendor-neutral, meritocratic bodies like the IETF or JCP community that can steward truly open data standards. So… what’s the takeaway? 👉 Unification is a myth. 👉 Interoperability is the path forward. 👉 OSS and users alike have their work cut out — to build a flexible, sustainable data ecosystem that allows innovation without fragmentation. #OpenData #DataLakehouse #ApacheHudi #XTable #Iceberg #DeltaLake #DataEngineering #OSS #Interoperability #TableFormats #Metastore #Data #BigData

Open Source is Eating the Data Stack. What's Replacing Microsoft & Informatica Tools? I've been reading a great discussion about replacing traditional proprietary data tools with open-source alternatives. Companies are increasingly worried about vendor lock-in, rising costs, and scalability limitations with tools like SQL Server, SSIS, and Power BI. The consensus is clear: open source is winning in modern data engineering. 💡 What's particularly interesting is the emerging standard stack that data teams are gravitating toward: • PostgreSQL or DuckDB for warehousing • dbt or SQLMesh for transformations • Dagster or Airflow for orchestration • Superset, Metabase, or Lightdash for visualization • Airbyte or dlt for ingestion As one data engineer noted, "Your best hedge against vendor lock-in is having a warehouse and a business-facing data model worked out. It's hard work but keeping that layer allows you to change tools, mix tools, lower maintenance by implementing business logic in a sharable way." I see this shift every day. Teams want the flexibility to choose best-of-breed tools while maintaining unified control and visibility across their entire data platform. That's exactly why you should be building your data platform on top of tooling that integrates with your favorite tools rather than trying to replace them. Vertical integration sounds great, if you enjoy vendor lock-in, slow velocity, and rising costs. Python-based, code-first approaches are replacing visual drag-and-drop ETL tools. We all know SSIS is horrible to debug, slow and outdated. The modern data engineer wants software engineering practices like version control, testing, and modularity. The real value isn't just cost savings - it's improved developer experience, better reliability, and the freedom to adapt as technology evolves. For those considering this transition, start small. Replace one component at a time and build your skills. Remember that open source requires investment in engineering capabilities - but that investment pays dividends in flexibility and innovation. Where do you stand on the proprietary vs. open source debate? And if you've made the switch, what benefits have you seen? #DataEngineering #OpenSource #ModernDataStack #Dagster #dbt #DataOrchestration #DataMesh

🧠 Postgres + Vectors: Building the Brain Inside Your Database Postgres isn’t just surviving the AI wave — it’s leading it. For 30+ years, Postgres has outlived every trend: 🔢 Relational 🔄 NoSQL ☁️ Cloud 🤖 And now… AI-native data architectures Today, the magic comes from pgvector, a lightweight yet powerful extension that lets Postgres store and query vector embeddings — mathematical representations of meaning. 🧩 Why Vector Embeddings Matter Embeddings turn your data into semantic fingerprints — numbers that capture context, not just text. 📌 Instead of searching exact words, you search similar meaning. Try asking Postgres: “🔍 Find tickets similar to this customer complaint.” No keyword hacks. No Boolean gymnastics. Just pure semantic similarity. Behind the scenes, Postgres compares vectors and returns conceptually related matches — powering: ✨ Recommendation engines ✨ Semantic search ✨ AI copilots ✨ RAG (Retrieval-Augmented Generation) pipelines ✨ Intelligent CRM + support workflows All using plain SQL. 🧠 Your Database Just Got a Brain The best part? No separate vector DB. No new infra. No integration glue. Your existing Postgres cluster becomes the context engine of your AI stack. For startups → 🚀 Huge cost and ops savings For enterprises → 🏗️ Architectural simplicity and compliance confidence For the Postgres community → 💙 Proof that open source doesn’t follow trends… it shapes them 🌐 The database that powered the web is now powering intelligence. If you’re building AI-native apps, Postgres + pgvector is no longer optional — it’s foundational. 🔖 #Postgres #pgvector #AI #SemanticSearch #RAG #OpenSource #Database

Microsoft just moved #DocumentDB to The Linux Foundation—aiming to make document databases an open standard. I sat down with Kirill Gavrylyuk, VP of Azure Cosmos DB at Microsoft, to unpack the why and the what-next. His take was clear: “Document databases are critical for AI apps,” and the industry needs a “vendor‑neutral, open source” path—hence DocumentDB at LF. He also revealed Amazon Web Services (AWS) is joining as a co‑maintainer, with a steering committee that already includes Yugabyte DB , Rippling, and AB InBev. Two quotes that jumped out: — “We don’t believe one vendor should control the standard. The core goal is developer freedom.” — Kirill Gavrylyuk — “It will stay pure Postgres and MongoDB‑compatible—those principles are in the charter.” Roadmap: richer MongoDB compatibility, Kubernetes operator for easy deploys, scale‑out and HA, and AI‑driven capabilities (including broader vector indexing). Also notable: Microsoft uses DocumentDB internally for Cosmos DB vCore and will keep contributing upstream. https://lnkd.in/evTtZmzG #OpenSource #LinuxFoundation #Postgres #DocumentDB #MongoDB #AI #CloudNative #DeveloperExperience #Microsoft #CosmosDB

Forkable infrastructure is emerging as the next primitive in data infrastructure. We’re seeing more and more customers ask for more than snapshots or backups. They want true branches of their database — forks they can spin up instantly, test safely, and (sometimes) merge back. Supabase and Neon are both pushing in this direction, but with different philosophies: ▪️ Supabase Native Migrations: tightly integrated with GitHub and CLI workflows. Open a PR → spin up a branch → run SQL migrations → merge applies them to production. ▪️ Neon ORM-Powered Migrations: instant copy-on-write forks, but migrations are left to your ORM or toolchain (Prisma, Drizzle, Flyway, etc.). The nuance: not all schema changes are equal. — Adding tables or columns touches application code: natural for an ORM to own. — Adding an index or constraint rarely requires code changes: you want to test these directly in the DB, often for performance reasons, before committing. AI coding agents may push this divide further. If your program already uses an ORM, an agent will almost certainly stay within that framework: models + migrations together. That works for schema tied to code. But it doesn’t address database-only optimizations, the kind of changes developers want to experiment with directly inside a fork. Forkable infra sits right at this intersection: a tool for both human creativity and agent-driven automation. I’ve been thinking a lot about developer workflows for data infrastructure. Should forkable infra be opinionated about migrations, or should it stay raw and flexible, letting humans and agents layer their own workflows on top? Or perhaps somehow both? Curious to hear your thoughts. What do you think? #ForkableInfra #DataInfrastructure #DatabaseBranching #Postgres #DeveloperExperience #AICodingAgents #DataEngineering

After a massive labor of love, I'm thrilled to share my new blog post on the vast and ever-evolving world of data stores! https://lnkd.in/g725V_d4 The database landscape has undergone a "Cambrian explosion" over the last two decades. We've moved far beyond the traditional monolithic RDBMS, and today's choices include everything from in-memory data grids to specialized graph, time-series, and vector databases. This piece is a comprehensive guide to navigating this complex ecosystem. I cover: ✅ The core types of data (transactions, interactions, observations) ✅ A detailed taxonomy of data stores ✅ A look at how AI and modern workloads are driving the next wave of innovation Whether you're an architect, a developer, or a data leader, this blog will help you understand the nuances and make the best choice for your next project. Please let me know what you think A quick note on the products mentioned: The examples in this guide are representative, not comprehensive. The data store landscape is vast and constantly changing, so always refer to the latest product documentation for your specific use case. #Data #Databases #AI #Technology #DataEngineering #Cloud #NoSQL #RDBMS #DBMS #GraphDB

Introducing DB25: A Modern HTAP Database with Computational Storage Meet DB25, an open-source Hybrid Transactional/Analytical Processing (HTAP) database system that unifies OLTP and OLAP workloads in a single platform, eliminating traditional ETL bottlenecks.  Key Features:  • Real-time analytics on operational data  • Intelligent workload routing (OLTP/OLAP)  • Computational storage integration for near-data processing  • PostgreSQL-compatible SQL interface  • C++17 implementation with vectorized execution  Current Status:  • Complete query processing pipeline (parsing → optimization → execution)  • Working HTAP architecture foundation  • Comprehensive 33-page technical documentation  • Educational framework for database systems courses  Roadmap:  • Phase 7-9: Production HTAP features  • Advanced computational storage optimization  • Real-time streaming analytics  • Cross-workload performance optimization  DB25 bridges the gap between academic database research and practical HTAP implementation, serving both as a learning tool and foundation for next-generation database systems.  Perfect for database researchers, systems engineers, and students exploring modern data architectures!  Full documentation & code: https://lnkd.in/gdpxHpFH  #Database #HTAP #OpenSource #SystemsEngineering #DataEngineering #ComputationalStorage #DatabaseResearch  What are your thoughts on unified HTAP architectures? Share your experiences below!