Utilities for using Azure Storage Accounts with Dagster. This is mostly aimed at Azure Data Lake Storage Gen 2 (ADLS2) but also contains some utilities for Azure Blob Storage.
The storage account name.
Resource containing clients to access Azure Data Lake Storage Gen2.
Contains a client for both the Data Lake and Blob APIs, to work around the limitations of each.
Default Value: None
Stateful mock of an ADLS2Resource for testing.
Wraps a mock.MagicMock. Containers are implemented using an in-memory dict.
Logs op compute function stdout and stderr to Azure Blob Storage.
This is also compatible with Azure Data Lake Storage.
Users should not instantiate this class directly. Instead, use a YAML block in dagster.yaml
such as the following:
compute_logs:
module: dagster_azure.blob.compute_log_manager
class: AzureBlobComputeLogManager
config:
storage_account: my-storage-account
container: my-container
secret_key: sas-token-or-secret-key
default_azure_credential:
exclude_environment_credential: true
prefix: "dagster-test-"
local_dir: "/tmp/cool"
upload_interval: 30
storage_account (str) – The storage account name to which to log.
container (str) – The container (or ADLS2 filesystem) to which to log.
secret_key (Optional[str]) – Secret key for the storage account. SAS tokens are not supported because we need a secret key to generate a SAS token for a download URL.
default_azure_credential (Optional[dict]) – Use and configure DefaultAzureCredential. Cannot be used with sas token or secret key config.
local_dir (Optional[str]) – Path to the local directory in which to stage logs. Default:
dagster._seven.get_system_temp_directory().
prefix (Optional[str]) – Prefix for the log file keys.
upload_interval – (Optional[int]): Interval in seconds to upload partial log files blob storage. By default, will only upload when the capture is complete.
inst_data (Optional[ConfigurableClassData]) – Serializable representation of the compute log manager when newed up from config.
ADLS Gen2 file system name.
ADLS Gen2 file system prefix to write to.
Default Value: ‘dagster’
Lease duration in seconds. Must be between 15 and 60 seconds or -1 for infinite.
Default Value: 60
Persistent IO manager using Azure Data Lake Storage Gen2 for storage.
Serializes objects via pickling. Suitable for objects storage for distributed executors, so long as each execution node has network connectivity and credentials for ADLS and the backing container.
Assigns each op output to a unique filepath containing run ID, step key, and output name. Assigns each asset to a single filesystem path, at “<base_dir>/<asset_key>”. If the asset key has multiple components, the final component is used as the name of the file, and the preceding components as parent directories under the base_dir.
Subsequent materializations of an asset will overwrite previous materializations of that asset. With a base directory of “/my/base/path”, an asset with key AssetKey([“one”, “two”, “three”]) would be stored in a file called “three” in a directory with path “/my/base/path/one/two/”.
Example usage:
Attach this IO manager to a set of assets.
from dagster import Definitions, asset
from dagster_azure.adls2 import ADLS2PickleIOManager, adls2_resource
@asset
def asset1():
# create df ...
return df
@asset
def asset2(asset1):
return df[:5]
defs = Definitions(
assets=[asset1, asset2],
resources={
"io_manager": ADLS2PickleIOManager(
adls2_file_system="my-cool-fs",
adls2_prefix="my-cool-prefix"
),
"adls2": adls2_resource,
},
)
Attach this IO manager to your job to make it available to your ops.
from dagster import job
from dagster_azure.adls2 import ADLS2PickleIOManager, adls2_resource
@job(
resource_defs={
"io_manager": ADLS2PickleIOManager(
adls2_file_system="my-cool-fs",
adls2_prefix="my-cool-prefix"
),
"adls2": adls2_resource,
},
)
def my_job():
...
The storage account name.
The credentials with which to authenticate.
SAS token for the account.
Shared Access Key for the account.
{}
ADLS Gen2 file system name
Default Value: ‘dagster’
FileManager that provides abstract access to ADLS2.
Implements the FileManager API.
ADLS Gen2 file system name.
ADLS Gen2 file system prefix to write to.
Default Value: ‘dagster’
Lease duration in seconds. Must be between 15 and 60 seconds or -1 for infinite.
Default Value: 60
( deprecated ) This API will be removed in version 2.0.
Please use ADLS2PickleIOManager instead..
Renamed to ADLS2PickleIOManager. See ADLS2PickleIOManager for documentation.
The storage account name.
The credentials with which to authenticate.
SAS token for the account.
Shared Access Key for the account.
{}
Resource that gives ops access to Azure Data Lake Storage Gen2.
The underlying client is a DataLakeServiceClient.
Attach this resource definition to a JobDefinition in order to make it
available to your ops.
Example
from dagster import job, op
from dagster_azure.adls2 import adls2_resource
@op(required_resource_keys={'adls2'})
def example_adls2_op(context):
return list(context.resources.adls2.adls2_client.list_file_systems())
@job(resource_defs={"adls2": adls2_resource})
def my_job():
example_adls2_op()
Note that your ops must also declare that they require this resource with required_resource_keys, or it will not be initialized for the execution of their compute functions.
You may pass credentials to this resource using either a SAS token, a key or by passing the DefaultAzureCredential object.
resources:
adls2:
config:
storage_account: my_storage_account
# str: The storage account name.
credential:
sas: my_sas_token
# str: the SAS token for the account.
key:
env: AZURE_DATA_LAKE_STORAGE_KEY
# str: The shared access key for the account.
DefaultAzureCredential: {}
# dict: The keyword arguments used for DefaultAzureCredential
# or leave the object empty for no arguments
DefaultAzureCredential:
exclude_environment_credential: true
ADLS Gen2 file system name.
ADLS Gen2 file system prefix to write to.
Default Value: ‘dagster’
Lease duration in seconds. Must be between 15 and 60 seconds or -1 for infinite.
Default Value: 60
Persistent IO manager using Azure Data Lake Storage Gen2 for storage.
Serializes objects via pickling. Suitable for objects storage for distributed executors, so long as each execution node has network connectivity and credentials for ADLS and the backing container.
Assigns each op output to a unique filepath containing run ID, step key, and output name. Assigns each asset to a single filesystem path, at “<base_dir>/<asset_key>”. If the asset key has multiple components, the final component is used as the name of the file, and the preceding components as parent directories under the base_dir.
Subsequent materializations of an asset will overwrite previous materializations of that asset. With a base directory of “/my/base/path”, an asset with key AssetKey([“one”, “two”, “three”]) would be stored in a file called “three” in a directory with path “/my/base/path/one/two/”.
Example usage:
Attach this IO manager to a set of assets.
from dagster import Definitions, asset
from dagster_azure.adls2 import adls2_pickle_io_manager, adls2_resource
@asset
def asset1():
# create df ...
return df
@asset
def asset2(asset1):
return df[:5]
defs = Definitions(
assets=[asset1, asset2],
resources={
"io_manager": adls2_pickle_io_manager.configured(
{"adls2_file_system": "my-cool-fs", "adls2_prefix": "my-cool-prefix"}
),
"adls2": adls2_resource,
},
)
Attach this IO manager to your job to make it available to your ops.
from dagster import job
from dagster_azure.adls2 import adls2_pickle_io_manager, adls2_resource
@job(
resource_defs={
"io_manager": adls2_pickle_io_manager.configured(
{"adls2_file_system": "my-cool-fs", "adls2_prefix": "my-cool-prefix"}
),
"adls2": adls2_resource,
},
)
def my_job():
...