The connector.databricks package provides a convenient interface for accessing and interacting with Databricks volumes and tables directly from R.
This package is meant to be used with connector package, which provides a common interface for interacting with various data sources. The connector.databricks package extends the connector package to support Databricks tables and volumes through three connectors:
-
connector_databricks_sql(): tables, using a SQL warehouse through the bricksterDatabricksSQL()driver. This is the preferred way of working with tables. -
connector_databricks_table(): tables, using a cluster through theodbc::databricks()driver. -
connector_databricks_volume(): files in Unity Catalog volumes, using brickster.
Installation
You can install the connector.databricks from CRAN package using the following command:
# Install from CRAN
install.packages("connector.databricks")Development version
To get a bug fix or to use a feature from the development version, you can install the development version of connector.databricks from GitHub.
pak::pak("novonordisk-opensource/connector.databricks")Usage
library(connector.databricks)
tables <- connector_databricks_sql(
warehouse_id = "your-warehouse-id",
catalog = "my_catalog",
schema = "my_schema"
)
volume <- connector_databricks_volume(
catalog = "my_catalog",
schema = "my_schema",
path = "path-to-file-storage"
)
tables |>
write_cnt(mtcars, "mtcars")
tables |>
read_cnt("mtcars")
volume |>
write_cnt(iris, "iris.csv")Credentials are read from the environment by the brickster package, for example DATABRICKS_HOST and DATABRICKS_TOKEN. See Authentication for how to set this up locally, on Posit Workbench and on Posit Connect.
To work with tables through a cluster with the odbc::databricks() driver, use connector_databricks_table(). See Choosing between ConnectorDatabricksSQL and ConnectorDatabricksTable for how the two table connectors differ.
The connectors can also be created from a configuration file with connector::connect(). See the getting started vignette for an example.