Using Branch Deployments with Gitlab CI/CD#

In this guide, we'll walk you through setting up Branch Deployments with Gitlab CI/CD.

Using this approach to branch deployments may be a good fit if:

• You use Gitlab for version control
• You want Dagster to fully automate Branch Deployments

If you don't use Gitlab for version control or want full control over your setup, check out the Branch deployments with the dagster-cloud CLI guide.

Prerequisites#

To complete the steps in this guide, you'll need:

• Organization Admin permissions in Dagster+
• The ability to run a new agent in your infrastructure. This isn't required if you're using Serverless deployment.
• The ability to configure Gitlab CI/CD for your project. This isn't required if you used the Dagster+ Gitlab app to connect your project as a code location.

Step 1: Generate a Dagster+ agent token#

In this step, you'll generate a token for the Dagster+ agent. The Dagster+ agent will use this to authenticate to the agent API.

1. Sign in to your Dagster+ instance.
2. Click the user menu (your icon) > Organization Settings.
3. In the Organization Settings page, click the Tokens tab.
4. Click the + Create agent token button.
5. After the token has been created, click Reveal token.

Keep the token somewhere handy - you'll need it to complete the setup.

Step 2: Create and configure an agent#

While you can use your existing production agent, we recommend creating a dedicated branch deployment agent. This ensures that your production instance isn't negatively impacted by the workload associated with branch deployments.

Step 3: Add Gitlab CI/CD script to your project#

In this step, you'll add the required Gitlab CI config file to your Gitlab project. The file can be found in our CI/CD workflow repository or Serverless quickstart repository, depending on the agent you are using.

Copy the following files to your project (the linked files are shown for Hybrid repos, there are equivalents for Serverless repos).

• dagster_cloud.yaml
• .gitlab-ci.yml

In the next step, you'll modify these files to work with your Dagster+ setup.

Step 4: Configure the Gitlab project#

In this section, you'll:

1. Add the agent registry to dagster_cloud.yaml
2. Configure Gitlab CI/CD variables
3. Update Gitlab CI/CD script
4. Verify CI/CD pipeline runs

Step 4.1: Add the agent registry to dagster_cloud.yaml#

In the dagster_cloud.yaml file, replace build.registry with the registry used by the agent you created in Step 2.

For example:

# dagster_cloud.yaml

locations:
  - location_name: example_location
    code_source:
      python_file: repo.py
    build:
      directory: ./example_location
      registry: 764506304434.dkr.ecr.us-east-1.amazonaws.com/branch-deployment-agent

Step 4.2: Configure Gitlab CI/CD variables#

1. In your project, click the Settings tab.
2. In the CI/CD section of the sidebar, expand Variables.
3. Click Add variable.
4. In the Key field, enter the name of the variable. For example, API_TOKEN
5. In the Value field, paste the value of the variable.
6. Click Add variable.

For Hybrid deployments, repeat steps 3-6 for each of the secrets required for your registry type:

Step 4.3: Update Gitlab CI/CD script#

In this step, you'll update the Gitlab CI/CD config to set up Docker registry access.

In the .gitlab-ci.yml file, uncomment the step associated with your registry. For example, for the Gitlab container registry, you'd uncomment the following portion of the .gitlab-ci.yml file:

build-image:
  ...
  before_script:
    # For Gitlab Container Registry
    - echo $CI_JOB_TOKEN | docker login --username $CI_REGISTRY_USER --password-stdin $REGISTRY_URL

Save and commit the files to the project.

Step 4.4: Verify pipeline runs#

The last step is to verify that the Gitlab pipeline runs successfully.

1. On the project page, click the CI/CD tab.

2. In the list of pipelines, locate the latest branch deployment run. For example:

   

If the run finished successfully, that's it!

Step 5: Access the branch deployment#

Now that Branch Deployments are configured, you can check out the preview in Dagster+, by accessing the branch deployment from the deployment switcher: