Unitycatalog: the first look

Let's start with one of the most important parts: persistent storage. Apache Ranger supports several backend RDBMSs for storing policies and accesses (MySQL, PostgreSQL, etc.). During configuration, you'll need to provide an existing database, and Ranger will create all its tables as part of the initialization process. Databricks Unitycatalog has both an advantage and a disadvantage. The upside is that Unity comes with an included (as a dependency) H2DB. The disadvantage is that Unity currently only supports H2DB. And this H2DB file is created in /etc/, so you cannot easily separate the policy server from the policy store. Migrating and backing up the DB is also difficult.

server/src/main/java/io/unitycatalog/server/persist/HibernateUtil.java:

configuration.setProperty("hibernate.connection.url", "jdbc:h2:file:./etc/db/h2db;DB_CLOSE_DELAY=-1");
configuration.setProperty("hibernate.hbm2ddl.auto", "update");
LOGGER.debug("Hibernate configuration set for production");

I hope the persistent part will be changed soon and the current implementation with H2 is just for demonstration purposes. Because without the ability to choose a real RDBMS to store policies in, the solution is far from being ready for production, more like a playground from my point of view.

The most promising and interesting part of Unitycatalog for me is the extension of governance to ML/AI objects. This is definitely a huge advantage over Apache Ranger, which does not offer anything like this. It is also a huge step forward because MLFlow, for example, has very limited authentication and permission management capabilities. There is no integration with something like Active Directory out of the box. And I was looking for Unitycatalog with the hope that it will finally be a unified solution that covers not only data in the data lake but also ML models/experiments. Unfortunately, the current OSS version does not offer anything related to ML…

I hope that Databricks will expose additional APIs related to ML/AI governance in future releases. Without that, it is hard to call the Unitycatalog truly unified.

But the Functions API within the Unitycatalog looks like something really new and fresh. I don't know of anything competitive in the existing big data governance frameworks.

It provides two main functionalities:

1. Register SQL function
2. Register external function that can be in any language.

server/src/main/java/io/unitycatalog/server/model/CreateFunction.java

public enum RoutineBodyEnum {
  SQL("SQL"),

  EXTERNAL("EXTERNAL");
}

Such a function may also have dependencies. A dependency is either a table or another function:

public class Dependency {
  public static final String JSON_PROPERTY_TABLE = "table";
  private TableDependency table;

  public static final String JSON_PROPERTY_FUNCTION = "function";
  private FunctionDependency function;
}

The part of the documentation related to the Functions API is a little unclear, but from the code it looks like functions are bound to catalogs. But it is important that function names are unique, because names are keys, there is nothing like adding a hash or generating an internal ID. At least I have not found anything like that.

Anyway, the functionality looks really cool! I can imagine a lot of uses for it. For example, you could register something like audit functions, or delta-vacuum functions, or some SQL routines. Finally, a nice tool to unify access management for table functions!

A huge advantage of Unitycatalog over Apache Ranger!

Table management is a functionality offered by almost all of Unitycatalog's competitors, including Apache Ranger. So let's see what the Databricks solution brings to the table.

Both external and managed tables are supported, so it is cool!

server/src/main/java/io/unitycatalog/server/model/TableType.java

public enum TableType {

  MANAGED("MANAGED"),

  EXTERNAL("EXTERNAL");
}

In the marketing promo, Databricks claims that Unitycatalog is a unified tool that can be used with more than just delta tables. But in fact it cannot. This is because you cannot create an external table and map it to existing data in Iceberg/Hudi formats in your datalake:

server/src/main/java/io/unitycatalog/server/model/DataSourceFormat.java

public enum DataSourceFormat {

  DELTA("DELTA"),

  CSV("CSV"),

  JSON("JSON"),

  AVRO("AVRO"),

  PARQUET("PARQUET"),

  ORC("ORC"),

  TEXT("TEXT");
}

So, at the moment only Delta is supported. Apache Iceberg or Apache Hudi are supported only by extending metadata to delta-compatible one. It may be achieve via Apache XTable (incubating) or via Delta UniForm, of course. But still disappointing a little. For the comparison, Apache Ranger works with a Hive Metastore where you can register any of Delta, Iceberg and Hudi.

What really puzzles me is why Uniticatalog does not have a separate field for partitioning information in io.uniticatalog.server.model.TableInfo. Maybe for a DWH-like workload, hive-style partitioning is kind of outdated these days. But in my experience of a data engineer working in ML team, for batch ML-like/MLOps processing, old-school functional data engineering is still a perfect fit. I'm disappointed that Unitycatalog does not support information about partitioning. Especially considering that it was announced for ML/AI governance.

I'm actually wondering now, does Unicatalog actually support hive-style partitioning?

Technically, Apache Ranger supports the management of unstructured data in the same way as tables. This can be achieved in Ranger by registering a policy on a path. Unitycatalog provides a more user friendly API in my opinion. This API is called "Volumes". The support of both managed and external volumes is claimed:

server/src/main/java/io/unitycatalog/server/model/VolumeType.java

public enum VolumeType {

  MANAGED("MANAGED"),

  EXTERNAL("EXTERNAL");
}

But at the moment you can actually creates only external one…

server/src/main/java/io/unitycatalog/server/persist/VolumeRepository.java

if (VolumeType.MANAGED.equals(createVolumeRequest.getVolumeType())) {
    throw new BaseException(ErrorCode.INVALID_ARGUMENT, "Managed volume creation is not supported");
}

Volumes are defined by path or storage_location. Also Unitycatalog generates unique IDs for each volume, so it looks like that names may be not unique:

volumeInfo.setVolumeId(UUID.randomUUID().toString());