Working with delivery methods
Data Product Hub provides several options for delivering data products to the consumer. Each item in a data product can be delivered by using different methods. The connector type determines the available delivery methods.
Types of delivery methods for Data Product Hub
You can configure your data product to be delivered by the following delivery methods, depending upon the connector type:
- Downloading directly from a URL
- Downloading a data extract from a target connected data source
- Opening a URL
- Accessing a data product with Flight service
- Access in watsonx.data
- Deliver as a table in watsonx.data
| Name | Description | Notes |
|---|---|---|
| Download | Consumers receive a URL to download a data product from a connection. | Requires a data source connection that supports download. |
| Data extract | Consumers can download a data product as a file extract from a target connection. | Requires two connections: A source connection and a target connection. Also requires a default project. |
| Open URL | Consumers receive a URL to directly access a data product. | Does not require a data source connection. |
| Live access with Flight service | Consumers receive a code snippet to connect with a data source and receive a data product. | The provided code snippet can be added to notebooks and other applications. |
| Access in watsonx.data | Consumers can access the data product created from the watsonx.data instance. | Requires access to watsonx.data and a Presto engine |
| Deliver as a table is watsonx.data | Consumers access their data product as a table in watsonx.data. | Does not support FIPS enabled cluster in 5.2.0 |
Download
The Download delivery method generates a URL that allows consumers to download directly from a connected data source.
Open URL
The Open URL delivery method provides a URL that points to a website containing the items in a data product.
Data extract
You can extract data from a database and deliver the extract as a file to consumers. The data extract delivery method requires two connections, one for connecting to the database you are extracting from (the source) and another connection for delivering the file to the consumer (the target). To view which connectors support source and target connections for data extracts, see Delivery methods for connectors.
The data extract delivery method requires a default project. The default project is created automatically when you create a bucket in Configurations and settings>Storage.
The target connection is a file storage location that is used exclusively for storing data extracts. This connection delivers the extract to consumers through a download URL. Since data extracts can be large, a file storage location with adequate capacity is recommended. The target connections require read/write credentials.
Create two connections for the data extract delivery method:
- Source connection: The source connection is the location where you extract data.
- Target connection: The target connection is the location where the extracted file is stored and where consumers download a data extract from a URL.
The data extract delivery method supports these file types: Avro, CSV, Delimited text, Excel, JSON, ORC, Parquet, SAV, and XML.
Live access with Flight service
The Flight service provides real-time, read/write access to many data sources through a common open source API. The Flight service provides a single interface for accessing many different data sources. With the Flight service, Data Product Hub no longer needs a connection asset to the data source with personal credentials. The Flight service retrieves the credentials transparently. To determine if your data source supports the Flight service, see Delivery methods for connectors.
When you subscribe to a data product and select delivery by the Flight service, a Flight URL is provided. You download a code snippet in your preferred language (Python or R). The code snippet contains an asset ID and a catalog ID to connect to a data source to receive delivery of the items in a data product. The code snippet can be inserted into a notebook and other applications.
You can also access the data programmatically by using an Arrow client. Arrow libraries are available for C, C++, C#, Go, Java, JavaScript, Julia, MATLAB, Python, R and Ruby. See Apache Arrow for instructions on installing the libraries for each language.
See Flight client example for an example of how to access your data product using Python.
For more information about Arrow Flight RPC, see Arrow Flight RPC.
Access in watsonx.data
The access in watsonx.data delivery method provides consumers access to a data product containing data assets from a watsonx.data lakehouse.
Pre-requisites for data providers
A data provider must have the following roles to create a data product from data assets in watsonx.data.
- Required roles:
- Data Product Hub community role: Editor or Administrator
-
- watsonx.data lakehouse roles:
- watsonx.data instance role: User
- Catalog role: Admin
-
- Presto engine role: User
- If you want to grant consumers access to the engine, you must have Admin role.
A data provider can enable this delivery method using the following steps:
- Create a watsonx.data Presto connection in Data Product Hub. The credentials in the Presto connection will be used to create an access policy in watsonx.data to grant consumers access to a specific table.
- Create a data product by including data assets using the watsonx.data Presto connection.
- Select the Access in watsonx.data delivery method. During subscription, the consumer is granted access to the table in watsonx.data lakehouse.
- Optional: Allow consumers access to a Presto engine.
- Publish the data product.
Prequisites for data consumers
A data consumer must have the following roles to subscribe to a data product and access the data in watsonx.data.
- Required roles in Data Product Hub community:
- Community roles: Editor or Viewer
- watsonx.data lakehouse role: User
Data consumers can subscribe to a data product with this delivery method using the following steps:
- watsonx.data console link: For easy access to the delivered data in watsonx.data.
- Data asset: Specific catalog, schema and table details which can be viewed in the Data manager view in watsonx.data
- Presto engine: If the data provider enabled access to a Presto engine during data product creation, the consumer can use the engine to run SQL queries on the table.
The details can be obtained from the subscription in the My Subscriptions page.
Deliver as a table in watsonx.data
The deliver as a table in watsonx.data delivery method allows consumers to access their data product as a table within their watsonx.data instance. It works by storing the data as a CSV file in an IBM Cloud Object Storage bucket, and then using an ingest job in watsonx.data to load the CSV into the specified table. Users with the appropriate permissions can create new tables or append data to existing ones. The deliver as a table in watsonx.data delivery method requires a default project.
Pre-requisites for data consumers
- Required roles
-
Data Product Hub community role: Editor
-
watsonx.data lakehouse roles:
-
Service access roles: MetastoreAdmin and DataAccess
-
Platform access role: Administrator
-
- Target iceberg catalog: User
- You can also create an access policy that gives some permission to use the catalog. For more information, see Managing data policy in the watsonx.data documentation.
-
Spark engine connected to the target catalog: User
-
Presto engine role: User
-
Storage used for the target catalog: Writer or above
-
Cloud Object Storage: Add as a component in the watsonx.data lakehouse. The data extract will land here before being ingested to watsonx.data.
Data consumers must create two connections in Data Product Hub:
- Cloud Object Storage: Add a connection using the same connection as watsonx.data. You must use HMAC credentials to configure the Access key and Secret key or the Service credentials authentication method.
- IBM watsonx.data Presto: Copy the JSON snippet provided in the watsonx.data Presto engine instange and pasting it when creating a new connection.
Learn more
Delivery methods for connectors
Parent topic: Publishing a data product