Apache OpenDAL: The Universal Data Access Layer

data management

Apache OpenDAL provides a universal Open Data Access Layer, simplifying interaction with diverse storage services. It offers unified and efficient access to object, file, and cloud storage across multiple programming languages.

Apache OpenDAL (pronounced "OH-puhn-dal") is an Open Data Access Layer designed to provide seamless interaction with a diverse range of storage services. It embodies the vision of "One Layer, All Storage," striving to unify data access complexities under a single, robust interface.

OpenDAL's core principles guide its development:

Open Community: Fostering collaborative growth and widespread adoption.
Solid Foundation: Ensuring reliability and stability.
Fast Access: Optimizing performance for data operations.
Object Storage First: Prioritizing support and capabilities for object storage systems.
Extensible Architecture: Allowing for easy integration of new services and functionalities.

Further details on the project's vision can be found in the OpenDAL Vision documentation.

Language Bindings

OpenDAL offers bindings for various programming languages, enabling developers to integrate its capabilities into their preferred environments.

Note: Each binding maintains its own independent version number, which may differ from the Rust core version. Always refer to the specific binding's version for compatibility and updates.

Rust Core: The foundational implementation.
C Binding
Cpp Binding
D Binding
Dart Binding
Dotnet Binding
Go Binding
Haskell Binding
Java Binding
Lua Binding
Node.js Binding
OCaml Binding
PHP Binding
Python Binding
Ruby Binding
Swift Binding
Zig Binding

Tools and Utilities

OpenDAL provides specialized tools for common data access scenarios:

oli: A command-line interface for data access, serving as an alternative to tools like s3cmd, s3cli, and azcopy.
ofs: An implementation for accessing data via the POSIX file system API, similar to s3fs.

Ecosystem Integrations

OpenDAL integrates with other popular projects and libraries to extend its utility:

dav-server-opendalfs: An implementation of dav-server-rs utilizing OpenDAL for storage.
object_store_opendal: An object_store implementation powered by OpenDAL.
unftp-sbe-opendal: An unftp storage backend implementation built with OpenDAL.
parquet_opendal: Offers efficient I/O utilities for Parquet data formats.

Supported Storage Services

OpenDAL supports a comprehensive range of storage services, categorized as follows:

Standard Storage Protocols: ftp, http, sftp, webdav
Object Storage Services: azblob, cos, gcs, obs, oss, s3, b2, openstack_swift, upyun, vercel-blob
File Storage Services: fs, alluxio, azdls, azfile, compfs, dbfs, gridfs, hdfs, hdfs-native, ipfs, webhdfs
Consumer Cloud Storage Services: aliyun-drive, gdrive, onedrive, dropbox, koofr, pcloud, seafile, yandex-disk
Key-Value Storage Services: cacache, cloudflare-kv, dashmap, memory, etcd, foundationdb, persy, redis, rocksdb, sled, redb, tikv
Database Storage Services: d1, mongodb, mysql, postgresql, sqlite, surrealdb
Cache Storage Services: ghac, memcached, mini-moka, moka, vercel-artifacts
Git Based Storage Services: huggingface

Resources

Examples demonstrating OpenDAL's usage are available.

The official documentation can be found at: https://opendal.apache.org

Contribute to OpenDAL

OpenDAL is an active open-source project that welcomes contributions from users and developers. Here's how you can get involved:

Refer to the Contributing Guide.
Submit Issues for bug reports or feature requests.
Initiate Discussions for questions or ideas.
Engage directly with the community on Discord.
Report security vulnerabilities via the private mailing list.

Branding Guidelines

For any individual use, such as on webpages, handouts, or slides, the first and most prominent mention of the project must use its full form: "Apache OpenDAL™". The full name should be used sufficiently often to clearly convey the association of both the OpenDAL project and its software product with the Apache Software Foundation (ASF) as the parent organization.

For detailed guidance, please consult the Apache Product Name Usage Guide.

License and Trademarks

OpenDAL is licensed under the Apache License, Version 2.0. You can review the license details at: http://www.apache.org/licenses/LICENSE-2.0

"Apache OpenDAL", "OpenDAL", and "Apache" are registered trademarks or trademarks of the Apache Software Foundation.