Databricks 2024 Developments and Announcements

16.06.2025
8 min read

The global Databricks Data + AI conference held two weeks ago in San Francisco included a long list of innovations and announcements. Unlike last year, this year Snowflake and Databricks held their conferences in the same conference hall (Snowflake first and Databricks a week later), giving data professionals and enthusiasts the opportunity to catch up, learn and attend an action-packed couple of weeks of announcements, expert presentations and networking.

Some figures from the conference:

16,000 visitors
600 professional sessions
Representatives from 140 countries
200 data teams featuring different customer success stories
One Nvidia CEO in a leather jacket on stage

The conference included a long list of updates on a variety of topics and it’s practically impossible to cover them all in a single post, however all sessions were recorded and available for viewing on the conference website. We will try to give you a small taste of the conference and briefly review the most significant updates in our opinion:

1. Changing format rules

In early June, Databricks announced the acquisition of Tabular, whose founders invented the Iceberg data format. It’s with no surprise that with this acquisition, Databricks wants to develop their products with the two most common open-source data management formats: Iceberg and Delta Lake. Also with the acquisition, Databricks announces Delta Lake UniForm, which is essentially full and transparent support for both formats (as well as Apache Hudi) enabling data teams to choose their preferred format without limitation and need for conversions.

2. Serverless Everything

Until recently, the main service available in serverless configuration within the Databricks ecosystem was SQL Warehouse. Serverless are “hot” servers that Databricks manages and when a customer chooses to use these, they are allocated for their benefit, with the advantage being that the server is immediately available. At the conference, Databricks announced that all the different services are available from now on in a serverless configuration, so that the provision and spin-up time of the service if you chose to work in this configuration is significantly shortened to a few seconds.

3. Pack me some models on sale

In July 2023, Databricks acquired MosaicML, which specializes in training LLM models. Since the acquisition, the folks at Databricks have been working diligently to implement Mosaic’s capabilities throughout the platform, During the conference, Databricks introduced the principle of Compound AI Systems. What does this actually mean? We are all trying these days to leverage and implement GenAI capabilities in the various work processes, according to Databricks, the right way to do this is to connect several different models into a unified flow that brings a variety of capabilities to solve business challenges. To illustrate this, a demo of a business scenario was presented, creating a personalized Instagram campaign for the marketing team based on a sequence of GenAI models and connecting to integrated functions and data sources.

We highly recommend to see the recording of this demo, first of all the awesome product manager on stage debugs the demo in real time in front of a hall full of spectators, but beyond that in our view there is nothing less than a revolution in the operating concept of LLM models ahead of us. More so in the way that traditional work processes can be transformed from end to end and present a real and tangible ROI to the organization (and not just hype).

Additional innovations from Mosaic presented at the conference:

Built-in ability to fine tune an open source or commercial model within Databricks through training on the customer’s internal data.
Shutterstock ImageAI – the world’s largest image exchange partners with Databricks and makes a text-to-image model accessible. The model generates custom images for the needs of business users.
Mosaic AI Agent Framework – built-in ability to package organizational information in RAG and submit it to LLM models, including governance integrated using Unity Catalog.
Mosaic AI Tool Catalog – built-in set of tools for use within Databricks for common LLM tasks. In this way, data, functions, models and more can be packaged for reuse plus assembly of “puzzles”, as part of the Compound Systems concept.

4. Let’s not forget the DWH

In the data warehousing space, Databricks released significant updates with the understanding that this is an essential core capability for organizations that is also particularly expensive. In an effort to reduce the complexity of migration processes, Databricks complements a broad set of functional capabilities and actually provides support for all the goodness we’re used to from traditional data warehouses. Did anyone ask for materialized views? Now you have it, SQL and Python UDF? Yes that as well, you get the idea…

In addition, Databricks harnesses the ability of Compound Systems for the benefit of its internal platform and actually runs AI-based optimization engines behind the scenes. How does it manifest itself in practice? Here are some examples:

There is no need to define partition or clustering of a table, Databricks learns the usage patterns on its own based on queries and automatically generates auto clustering, and the same goes for indexes.
Databricks claims that through a combination of AI optimization mechanisms, they were able to achieve a 73% improvement in enterprise query runtimes over the past two years.

Additional updates in the DWH space include built-in LLM query writing, the ability to call AI functions in SQL code, and run data retrieval from vector databases.

5. Add some AI on top of my BI

Databricks does not neglect analysts or data consumers and launches a completely new service called AI/BI. In fact, it is a combination of two capabilities, one of which includes on-the-fly creation of dashboards adapted to the user’s needs based on free text requests. The second is called Genie and is an LLM prompt that allows you to ask questions about the data directly or from dashboards.

According to Databricks, one of the challenges of organizations is their business terminology, which is not generic and therefore LLM finds it difficult to give reliable answers in these situations, with GEnie this problem is also addressed because the model can be taught what the definition is, for example, of “churn ratio” or any other business term, and from that point on it will be able to calculate and present this figure in response to analysts’ questions. How does Genie achieve this? Once again, incorporating Compound AI as a core component into the platform, this time to help Genie familiarize itself with all the different data assets in order to give more accurate answers.

6. Cross-org data catalog

One of the hottest topics at the conference was in the domain of governance, and data cataloging as part of it, of course. Databricks has made a lot of investments over the past year in Unity Catalog, its built-in data catalog as part of the platform. Development continues with a number of interesting announcements, promoting Unity as the main catalog for organizations due to it’s integration with the entire data stack.

First, it opens and makes Unity accessible as an open-source component, which means that even customers who do not currently use data can make use of Unity. This capability opens up the opportunity to build integration capabilities and connectors for Unity. To top that, Databricks gives users some of these connectors already today using a capability called Lakehouse Federation, which means that you can already connect additional systems to Unity today such as Redshift, Power BI and – Snowflake

Another interesting update is the ability to manage business KPIs in Unity Catalog. Databricks announces an object called Metrics and integration with common third-party systems for managing business metrics. This was accompanied by other announcements of new Data Sharing and Data Clean Rooms capabilities, which we will expand on in a separate post.

7. Data flows in the palms of your hands

One of the most important announcements was saved for the end of the conference. Databricks announces a new product called LakeFlow, an ETL tool managed within the platform addressing ingestion, transformation processes and business logic, as well as orchestration of these processes. This capability is based on an acquisition that Databricks made in the past of Arcion and includes built-in connectors for common sources, CDC capability, a graphical interface for transformations and more. This announcement challenges the ecosystem of Databricks partners in these areas (such as Fivetran, DBT, Airflow) and it will be interesting to see if customers choose to put all their eggs in one basket and migrate processes in these areas to Databricks.

In conclusion: Databricks continues to establish its status as a unified organizational data infrastructure that provides a solution for all data assets, all data workloads, and all functionaries involved in the various data processes in the organization. In this respect, the conference was another demonstration of capabilities in the battles for control against Snowflake and a clear signal to Microsoft’s Fabric that the gap relative to it in capabilities remains significant and continues to grow. One thing is certain: the cloud data space is certainly one to watch over the next few years.

More to Explore

Financial Crime Digest

See how leading institutions achieved results through our tailored compliance and financial crime solutions.

Extending AML Regulations to Investment Advisors

On August 28, 2024, the Financial Crimes Enforcement Network (FinCEN) issued a final rule that extends Anti-Money Laundering (AML) and Countering the Financing of Terrorism (CFT) requirements to Registered Investment Advisers (RIAs) and Exempt Reporting Advisers (ERAs). This rule, effective January 1, 2026, mandates that RIAs and ERAs develop and implement comprehensive AML/CFT programs, aligning them with other financial institutions under the Bank Secrecy Act (BSA).

2024 Year in Review: Financial Crime Compliance, and Regulatory Trends

In 2024, regulatory enforcement increased, financial crime risks evolved, and scrutiny of both traditional and emerging financial institutions heightened. As regulatory bodies worldwide intensify their focus, organizations must adapt to a rapidly changing compliance landscape.

Cloud AI and RPA

The fight against financial crimes has been on for decades. Do you remember Michael Night? Those of us who grew up in the good old days of the 80s were probably huge fans of the American action crime drama TV series, Knight Rider.

FINTRAC’s requirements – Armored cars

The Financial Transactions and Reports Analysis Centre of Canada (FINTRAC) is Canada’s financial intelligence unit and anti-money laundering and anti-terrorist financing supervisor. Its mandate is to facilitate the detection, prevention and deterrence of money laundering and the financing of terrorist activities, while ensuring the protection of personal information under its control.

Using Generative AI in Combating Financial Crimes

Amazon recently launched a feature where hundreds, and sometimes thousands, of reviews are summarized into a concise and simple paragraph. This auto-generated summary allows the customer to instantly get an overall review of a product’s capabilities as well as overall customer satisfaction. This is a great example of using Large Language Models (LLM) to process and produce text that resembles that of humans. These models can understand language structures, grammar, context, and semantic linkages since they have been trained on enormous amounts of text data.

Evolution of DeFi Amid Regulatory Uncertainty

Emerging technologies are reshaping the financial services industry. On one end of the spectrum, initiatives such as Real-Time Payments and ISO20022 are modernizing existing payment infrastructure, making it faster and more efficient. On the other end, blockchain and distributed ledger technologies (DLT) are laying the foundation for an alternative ecosystem involving digital assets and cryptocurrencies.

Balance Open Banking Enthusiasm with Caution

Digitalization and Open Banking are two most prominent trends in banking industry in recent time. While the former was initiated by changing customer behavior, the latter was driven by regulatory and market forces.

Open Banking is a new kid on the block with a lot of promise and fanfare, but it can present new challenges for financial services. Rather than being swayed by its exuberance, a cautious approach is required for its implementation.

Combating Fraud – A Journey From Good to Great

Financial Institutions are in a similar situation. A lot has changed in the last few years. Fraud activities and losses are on the rise. FIs need to act or face the consequences – higher fraud losses, loss of public trust and declining customer loyalty. Given the risk, it is imperative that risk leaders adjust their fraud strategies to adapt to the new reality.

Banking & Compliance Post Pandemic

Traditional banking and the wind of change are not inherently linked, but now more than ever, it seems that the change offered by modern technology is not only adopted, but due to the pandemic is happening at an accelerated pace. From the demand for more speedy processes, through the increased trend of closing branches, and all the way to new possibilities that the digital arena offers (including AI and Machine Learning). All this without even mentioning Crypto and the challenges it posts to traditional banks. Weather we like it or not, change is all around us. Here are some highlights from our recent webinar where we discussed banking and compliance post pandemic:

Understanding Economic Sanctions

Imposing economic sanctions is a powerful foreign policy tool used by countries and international organizations that can include travel bans, asset freezes, arms embargoes, and trade restrictions with countries, individuals, and entities. The US Department of the Treasury’s Office of Foreign Assets Control (OFAC) “administers and enforces economic and trade sanctions based on US foreign policy and national security goals.” The names of individuals, entities, aircraft, vessels, and countries are incorporated into OFAC’s list of Specially Designated Nationals and Blocked Persons (“SDN list”), which blocks U.S. persons from transacting with both them and their assets. The SDN list is updated ad-hoc, with new additions to and removals from the list.

Modern Challenges and Innovative Tools for Sanctions Compliance

Financial institutions need to meet growing sanctions compliance demands without disrupting customer services, incurring an exorbitant overhead, and being exposed to regulatory fines.

Navigating Emerging Risks and Regulatory Changes with a Robust AML Compliance Program

Regulators have fined financial institutions close to 20 billion dollars for Anti-Money Laundering (AML) shortcomings and regulatory violations in the last 6 years.

Matrix-IFS and Quantifind Partner to Complement Financial Crimes Solutions and Services

Quantifind, a provider of a SaaS platform used by banks to help automate financial crimes risk screening and investigations, today announced its partnership with Matrix-IFS, the leading provider of financial crime advisory and implementation services. Quantifind’s Graphyte™ platform brings best-in-class risk assessment and entity resolution accuracy to Matrix-IFS’ comprehensive services for KYC/CDD/EDD, transaction monitoring, sanctions screening, and case management. Graphyte includes integrations with leading case management platforms that seamlessly incorporate advanced risk intelligence directly within analysts’ familiar tools and workflows.

The Benefits of AML Cloud Convergence – AML as a Service

Many Chief Compliance Officers at midsized financial institutions sense that streamlining their anti-money laundering systems will lead to increased efficiency and effectiveness across their AML programs, but they wonder where to start.

How Mid-sized FIs Can Turn 3 Industry Trends into AML Opportunities

Midsized financial institutions (with $1 billion – $10 billion in assets) play an important role in our financial system, especially in the U.S. where banks with less than $10 billion in assets represent 14% of the market and 97% of the total number of banks.

Step 3 – Entity Resolution (Account Matching & Linking)

Welcome back to Matrix AML Academy. In case you missed our previous post, this is the 3rd part of an 8-part educational program on how to improve tour AML program and systems

Step 2 – Proactive Data Quality Automation

Welcome back to Matrix AML Academy. In case you missed our previous post, this is the 2nd part of an 8-part educational program on how to improve tour AML program and systems

Step 1 – Transaction Monitoring Implementation Best Practices

Welcome to Matrix Academy! A place you can hone in on your AML skills, gain valuable knowledge, learn insider tips of the trade and keep up to date with current technologies and methods

COVID-19 Cyber Security Risks & Remedies

As COVID-19 continues to spread, phishing lures related to the CoronaVirus continue to appear. Some instances of “Casebaneiro Banking Trojan”, “HawkEye” and “WSH RAT” all using COVID-19 in phishing lures or executable names were spotted.

The NY DFS 500 Cyber Security Regulation Requirements Checklist

The New York State Department of Financial Services (“DFS”) has been closely monitoring the ever-growing threat posed to information and financial systems by nation-states, terrorist organizations and independent criminal actors.

Matrix-IFS Expands Its Financial Crime Advisory Practice

Matrix-IFS, a specialized financial crime and compliance solution provider, announces the expansion of its Advisory Services with the acquisition of a leading NYC-based Advisory consulting firm, Alius.

Matrix-IFS Named “10 Most Trusted Risk Management Solution Providers” in 2019

With millions of accounts containing people’s life savings, security has always been one of the largest concerns for financial institutions and their customers. As cybercriminals become more sophisticated in their hacking techniques, institutions should adopt more advanced cybersecurity and fraud prevention systems. Although new technologies provide more advanced security options, knowing which ones to use and how to implement them is a challenge many institutions face today.

Hunter – The Historical Transaction Lookup Digital Investigator (RPA BOT)

Looking up historical transaction data is a task every Investigator knows and dreads; constantly going to the upstream data systems, collecting historical data, merging it, only to try and make sense how the customer behavior looks like. Imagine a world where he wouldn’t have to do all that and simply focus on investigations. Sounds like a dream, right? Not anymore. With Robotics Process Automation it is now a reality, one that could easily and quickly be adapted.

Unveiling Hunter, Kaycee & Lexi – AKA, the Digital Investigators

On October 10th, we had the honor of speaking at the European Banking Forum in London in front of 100 compliance & financial crime Senior Managers, where we announced our purpose-built Digital Investigators into the world.

A Word About Cost Conscious Compliance

Last week, we had the honour and pleasure to engage in a fruitful conversation with a roomful of Compliance leaders from the financial sector, to share ideas and findings as to practical approaches to reducing compliance overheads.

Crime Tourism – ATM Skimming Operations

We’ve all been there, choosing our next travel destination. What should be a fun and exciting experience, can sometimes be stressful and overwhelming as there are many factors that can impact on our choice of a travel destination, to name a few: budget, travel companions, timing, weather and popular attractions.

Matrix-IFS is Sponsoring ACAMS Europe in Berlin (June 12-13 )

On June 12th, Matrix-IFS will be taking part in one of the most prestigious Financial Crime Conferences in Europe – ACAMS Europe, which will take place at the Berlin Congress Center GmbH. This 2-day conference brings together vendors, industry thought leaders and various financial institutions from across Europe.

The 5th AML Directive Readiness Checklist

During 2019-2020, EU countries will pass laws that introduce the 5th AML Directive (5MLD) into their respective national laws. Now is the time for your organization to invest in improving and optimizing your existing AML solutions to meet the increased challenges and regulator demands before it’s too late.

Best Practices to Transaction Monitoring Implementation

There are four key phases in transaction monitoring (TM) implementations and how a bank should design and execute these phases. Successful implementation of rule selection, data prep, segmentation, tuning and operational optimization will determine the success of the overall TM implementation in your organization.

Matrix-IFS is Attending ACAMS Florida (April 15-17)

On April 15–17 Matrix-IFS will be taking part in one of the largest Financial Crime Conferences in the US – ACAMS Florida, which will take place at the Diplomat Resort & Spa in Hollywood, Florida. This 3-day conference brings together vendors, industry thought leaders and various financial institutions from across the world.

The Problem with AML Today & How to Fix It: Part I – The AML Problem

The latest discoveries around the massive scale of money laundering at Danske Bank and ING are just two of the most recent examples of an underlying problem with the Anti-Money Laundering (AML) discipline. Despite increasing efforts and investments focusing on the AML problem, money laundering techniques continue to evolve and evade the controls implemented by Financial Institutions (FIs). More and more industry voices decry the efficiency and effectiveness of the current AML approach – which entails running all transactions through a series of automated checks to spot anomalies based on a large set of pre-defined typologies provided by experts.

Applying Robotic Process Automation in AML & FIU Operations

Learn how RPA can increase efficiency, lower risk, and trim your overhead

Matrix-IFS Hosts 4 Anti-Financial Crime Webinars

Matrix-IFS introduces a series of timely and informative webinars on new issues, trends and solutions in Anti-Financial Crime. Webinars are designed to update and inform risk, compliance, control room and fraud professionals working in financial institutions and capital markets.

Supercharge your FIU’s Operations with RPA at ACFE Las Vegas

During The 29th Annual ACFE Global Fraud Conference Anshul Arora , Head of FL Delivery Center will present Matrix-IFS’s RPA solution for improved operations and increased efficiency. During his session, Anshul will explore a new vision of a modern FIU department, which incorporates Artificial Intelligence, Machine Learning and advanced analytics to address and reduce alerts as well as how robotics and automation can play part in reducing risk and simplifying the work of the Investigators.

Top 6 Deal / List Systems Challenges and How to Solve Them

You know the challenges associated with your Deal Management (DMS) and List Management (LMS) systems & processes. You may even be aware of the full burden and cost they pose to your department and your organization. Now benefit from the fresh approach firms are taking with their Deal and List Management systems to gain efficiency and save time and resources.

How to Overcome AML Operations Growing Pains Using Cutting-edge Technology? | Find Out at ACAMS Europe

During ACAMS Annual Europe Conference taking place 30 May – 1 June in Amsterdam, Matrix-IFS will participate in a panel session on the subject of AML Operations and how to fight growing pains using cutting-edge technology where we will share options to enhance performance by replacing limiting rule-based solutions with far more efficient intelligent solutions that operate within existing infrastructure.The panel will discuss combining Graph Analytics, AI/Machine Learning, and Scenario Authoring on big data to improve the quality of detection, prevention and reporting of financial crimes.

Future-Proofing Financial Crime Compliance: A Technology Blueprint for Smarter Risk Management

The evolution of threats to the financial system is outpacing advancements in Financial Crime Compliance (FCC) regulations and technology used by financial institutions (FI). In this situation, FI can save considerable funds, by creating or evolving their business architecture with a view of adapting to emerging threats and the advanced tech that will be required to tackle them.

Matrix-IFS Named “Top 10 Risk & Compliance Solution Providers 2018”

Since 2006, Matrix International Financial Services (Matrix-IFS) has been helping financial institutions strengthen business compliance and address financial crimes and fraud issues, with a goal to satisfy both regulators and clients with effective, efficient and cost-effective solutions.

Deciphering Multi-Faceted Venezuelan Sanctions – Top Ten Practical Tips to Stay Compliant

The recently imposed Venezuelan sanctions issued by the U.S., the E.U., and Canada have placed heavy burdens on sanctions compliance programs. This has made it…

Quick answers. Real solutions.

Enter your contact details