Skip to main content
Version: devel View Markdown

Playground

note

Use of the dltHub platform and toolkits is subject to a commercial dltHub License.

The Playground is a zero-config destination managed by the dltHub platform. When you run pipelines on the platform, you can load data with destination="playground" without configuring any credentials or storage of your own. The platform provisions isolated storage for each workspace and writes your data as Delta tables to a managed S3 bucket.

It is meant for quick testing, demos, and a fast first run on the platform.

warning

The Playground is intended for demos and a quick start, not for production. For production workloads, use a destination you own and control, such as Delta, Iceberg, Snowflake Plus, or any of the dlt destinations.

Why use it

When you run a pipeline on the platform with a local-style destination such as duckdb, the data is written to the dltHub's ephemeral storage, which is erased after the run, so it can't be explored afterwards. The Playground instead persists your data in managed storage, so you can query it from the platform UI once the run finishes.

Use it for:

  • A quick first run on the dltHub platform without setting up a bucket or credentials.
  • Running examples and demos.
warning

Do not load sensitive or confidential data into the Playground.

Prerequisites

  • A dltHub workspace. If you don't have one yet, scaffold it with uvx dlthub-start@latest (see the installation guide).

  • You must be logged in and connected to a workspace:

    uv run dlthub login
    uv run dlthub workspace connect <workspace-name>
  • The deltalake package must be installed in your project. The Playground writes Delta tables, and the runtime image installs your project dependencies, so add it to your pyproject.toml:

    uv add deltalake

Usage

Set the destination to playground in your pipeline and declare it as a job so the platform can deploy and run it. No bucket_url, credentials, or other configuration is required.

Decorate your pipeline function with @run.pipeline in data_pipeline.py:

import dlt
from dlt.hub import run

@run.pipeline("pipeline")
def load_data():
pipeline = dlt.pipeline(
pipeline_name="pipeline",
destination="playground",
dataset_name="data",
)
pipeline.run([{"id": 1}], table_name="items")

Declare the job in __deployment__.py so the platform can discover it:

from data_pipeline import load_data

__all__ = ["load_data"]

Deploy the workspace, then run the job on the platform:

uv run dlthub deploy
uv run dlthub pipeline run pipeline -f

See deployments for more on the __deployment__.py manifest, @run.pipeline, and deploying jobs on the platform.

Working with the data

Once a run completes, open the platform dashboard to explore the persisted data. It includes a SQL query editor against your dataset:

uv run dlthub dashboard
note

The platform dashboard is itself a deployed job, provisioned when you run dlthub deploy with a __deployment__.py manifest. The local dashboard (uv run dlthub local show) does not require a manifest.

How it works

The Playground behaves like the Delta destination: it is a filesystem destination that writes Delta tables to a dltHub-managed S3 bucket. Each workspace gets its own isolated prefix (s3://.../<org_id>/<workspace_id>/...), so data from different workspaces never mixes. Storage and write dispositions follow the behavior of the Delta destination.

dlt is destination-agnostic, so anything you prototype against the Playground can later be moved to any destination you own with minimal changes. You swap the destination and provide your own storage and credentials.

This demo works on codespaces. Codespaces is a development environment available for free to anyone with a Github account. You'll be asked to fork the demo repository and from there the README guides you with further steps.
The demo uses the Continue VSCode extension.

Off to codespaces!

DHelp

Ask a question

Welcome to "Codex Central", your next-gen help center, driven by OpenAI's GPT-4 model. It's more than just a forum or a FAQ hub – it's a dynamic knowledge base where coders can find AI-assisted solutions to their pressing problems. With GPT-4's powerful comprehension and predictive abilities, Codex Central provides instantaneous issue resolution, insightful debugging, and personalized guidance. Get your code running smoothly with the unparalleled support at Codex Central - coding help reimagined with AI prowess.