The Datafye CLI

The Datafye CLI (datafye) is the command-line tool you use to provision, manage, and control your Datafye environments. It's the primary interface for creating private cloud deployments, whether you're setting up a development Foundry or a production Trading Environment.

The Datafye CLI officially supports macOS and Linux. Windows users can run the CLI via WSL (Windows Subsystem for Linux).

New to Datafye deployments? Start with Private Cloud Deployments to understand deployment models, then review Descriptors to learn about configuration files.

What the CLI Does

The Datafye CLI bridges the gap between your configuration (descriptors) and your infrastructure (private cloud deployments). Here's the flow:

1. You create descriptors — YAML or JSON files specifying what data, algos, and broker connections you need

2. You run CLI commands — The CLI reads your descriptors and provisions the infrastructure

3. Datafye creates your environment — A complete, isolated deployment with the services you specified

The CLI handles all the complexity: spinning up services, configuring networking, connecting to data providers, deploying containers, and ensuring everything works together.

Core Workflow

Every Datafye environment follows the same lifecycle, managed through CLI commands:

1. Provision

Create a new environment from descriptors. The command structure varies by deployment environment:

# Standalone Foundry (single deployment per machine)
datafye foundry local provision --descriptor foundry.yaml

# Distributed Foundry on AWS (requires name and cloud-specific params)
datafye foundry aws provision \
  --name my-dev-foundry \
  --descriptor foundry.yaml \
  --profile my-aws-profile \
  --cidr 10.0.0.0/16

# Standalone Trading environment
datafye trading local provision --descriptor trading.yaml

# Distributed Trading environment on AWS
datafye trading aws provision \
  --name production-trading \
  --descriptor trading.yaml \
  --profile my-aws-profile \
  --cidr 10.1.0.0/16

What happens during provisioning:

• CLI validates your descriptors (schema, semantics, dependencies)

• Infrastructure is allocated based on your deployment model (Standalone, Distributed Self-managed, or Distributed Datafye Managed)

• Services are deployed (Data Cloud, Algo Container, Backtesting Engine, Broker Connector — depending on scenario)

• Network connectivity is established

• Data providers and brokers are connected using your credentials

• Health checks verify everything is running correctly

Provisioning time:

• Local deployments: 5-10 minutes depending on internet connection and system resources

• AWS deployments: 10-20 minutes

Do not interrupt the provisioning process. If provisioning fails, deprovision and try again.

2. Start

Start a provisioned environment that's currently stopped:

# Standalone environments (no name needed - only one per machine)
datafye foundry local start
datafye trading local start

# Distributed environments (specify which deployment to start)
datafye foundry aws start --name my-dev-foundry
datafye trading aws start --name production-trading

Starting an environment is much faster than provisioning (typically under a minute) because the infrastructure already exists — services just need to resume.

3. Stop

Stop a running environment without deleting it:

# Standalone environments
datafye foundry local stop
datafye trading local stop

# Distributed environments
datafye foundry aws stop --name my-dev-foundry
datafye trading aws stop --name production-trading

Stopping preserves your environment configuration and data while freeing up compute resources. This is useful for:

• Saving costs when you're not actively using the environment

• Performing maintenance or updates

• Pausing development work

4. Deprovision

Completely remove an environment and all its resources:

# Standalone environments
datafye foundry local deprovision
datafye trading local deprovision

# Distributed environments
datafye foundry aws deprovision --name my-dev-foundry
datafye trading aws deprovision --name production-trading

Warning: Deprovisioning is irreversible. All deployment data will be permanently deleted, including databases, configurations, and all infrastructure. Ensure you have backed up any critical data before proceeding.

How the CLI Uses Descriptors

The CLI's power comes from its descriptor-driven approach. Instead of remembering dozens of command-line flags and options, you declare what you want in a descriptor file.

Descriptor Input Methods

Single combined descriptor:

datafye foundry provision --descriptor complete.yaml

Where complete.yaml contains data, algo, and broker sections:

data:
  equities:
    symbols: ["AAPL", "GOOGL"]

algo:
  name: "my-strategy"
  image: "myrepo/strategy:v1"

broker:
  provider: "alpaca"
  mode: "paper"

Separate descriptor files:

datafye foundry provision \
  --data-descriptor data.yaml \
  --algo-descriptor algo.yaml \
  --broker-descriptor broker.yaml

This modular approach lets you:

• Reuse common descriptors across environments (e.g., same data descriptor for dev and prod)

• Version control each descriptor type independently

• Share descriptor fragments with team members

Learn more: See Descriptors for complete details on descriptor format and organization.

Deployment Model Selection

The CLI uses environment-specific commands to handle the differences between deployment models. The environment is specified as part of the command (e.g., local, aws, azure, gcp), and each environment accepts different parameters.

Note: Developers can provision Standalone (local) and Distributed Self-managed (aws/azure/gcp) deployments using the CLI. Distributed Datafye Managed deployments (required for live trading) can only be provisioned by Datafye Ops. To get a managed deployment, contact Datafye to request provisioning.

Standalone Deployment

All services run on your local machine (or a single cloud VM). Commands use the local environment:

datafye foundry local provision --descriptor foundry.yaml

Key difference: Local deployments do not use a --name parameter since only one deployment is allowed per machine. You provide descriptor file(s) to specify what to provision.

See CLI Command Reference for complete parameter lists.

The CLI:

• Uses Docker to run all services as containers on your machine

• Sets up Docker networks for inter-service communication

• Binds ports to localhost for API access

• Manages container lifecycle (start, stop, health checks)

Best for: Development, learning, paper trading on local hardware

Distributed Self-managed

Services spread across multiple machines in your cloud account. Commands use cloud provider names (aws, azure, gcp):

datafye foundry aws provision \
  --name my-dev-foundry \
  --descriptor foundry.yaml \
  --profile my-aws-profile \
  --region us-east-1 \
  --cidr 10.0.0.0/16

Key differences: Distributed deployments require:

• --name — A unique name for the deployment (multiple deployments can exist in the same cloud account)

• --profile — The AWS CLI profile to use for provisioning

• Descriptor file(s) specifying what to provision

• Optional cloud-specific parameters like --region and --cidr

See CLI Command Reference for complete parameter lists.

The CLI:

• Provisions cloud resources (VMs, networking, storage) in your account

• Deploys services across multiple machines for scalability

• Sets up private networking (VPCs, subnets, security groups)

• Configures monitoring and logging

• Uses native cloud infrastructure (no Docker/containerization)

Best for: Production-scale development, paper trading with cloud resources, teams with cloud expertise

Distributed Datafye Managed

Datafye Ops Only: Distributed Datafye Managed deployments can only be provisioned by Datafye Ops. Developers cannot provision these environments directly using the CLI. To request a managed deployment, contact Datafye support.

Services spread across multiple machines in a Datafye-operated cloud account, provisioned and managed by Datafye Ops:

What Datafye Ops does:

• Provisions infrastructure in a dedicated account under Datafye's root

• Deploys services across multiple machines for scalability and resilience

• Handles all operations, monitoring, updates, and compliance

• Uses native cloud infrastructure (no Docker/containerization)

• Configures for live trading requirements when applicable

Best for: Live trading (required for live), production environments, teams wanting fully managed infrastructure

To request: Contact Datafye with your descriptors and deployment requirements.

Learn more: See Private Cloud Deployments for detailed comparison of deployment models.

Scenario-Specific Provisioning

Different scenarios require different combinations of descriptors:

Foundry: Data Cloud Only

Descriptors required: Data only

Standalone:

datafye foundry local provision --data-descriptor data.yaml

Distributed (AWS example):

datafye foundry aws provision \
  --name my-data-foundry \
  --data-descriptor data.yaml \
  --profile default \
  --region us-east-1 \
  --cidr 10.0.0.0/16

What gets provisioned: Data Cloud services for historical and live replay data.

You bring: Your own algo containers that connect to Data Cloud APIs.

Foundry: Full Stack

Descriptors required: Data + Algo

Standalone:

datafye foundry local provision \
  --data-descriptor data.yaml \
  --algo-descriptor algo.yaml

Distributed (AWS example):

datafye foundry aws provision \
  --name my-full-foundry \
  --data-descriptor data.yaml \
  --algo-descriptor algo.yaml \
  --profile default \
  --region us-east-1 \
  --cidr 10.0.0.0/16

What gets provisioned: Data Cloud, Algo Container runtime, Backtesting Engine, MCP Server for AI-assisted vibe coding.

You bring: Nothing — complete stack provided by Datafye.

Trading: Data Cloud + Broker

Descriptors required: Data + Broker

For paper trading - Standalone:

datafye trading local provision \
  --data-descriptor data.yaml \
  --broker-descriptor broker.yaml

For paper trading - Distributed Self-managed (AWS example):

datafye trading aws provision \
  --name paper-trading \
  --data-descriptor data.yaml \
  --broker-descriptor broker.yaml \
  --profile default \
  --region us-east-1 \
  --cidr 10.1.0.0/16

For live trading: Contact Datafye Ops to provision a Distributed Datafye Managed deployment.

What gets provisioned: Data Cloud services for live market data, Broker Connector for trade execution.

You bring: Your own algo containers that connect to Data Cloud and Broker Connector.

Trading: Full Stack

Descriptors required: Data + Algo + Broker

For paper trading - Standalone:

datafye trading local provision \
  --data-descriptor data.yaml \
  --algo-descriptor algo.yaml \
  --broker-descriptor broker.yaml

For paper trading - Distributed Self-managed (AWS example):

datafye trading aws provision \
  --name paper-full-stack \
  --data-descriptor data.yaml \
  --algo-descriptor algo.yaml \
  --broker-descriptor broker.yaml \
  --profile default \
  --region us-east-1 \
  --cidr 10.1.0.0/16

For live trading: Contact Datafye Ops to provision a Distributed Datafye Managed deployment.

What gets provisioned: Data Cloud, Algo Container runtime, Broker Connector, MCP Server for AI-assisted vibe coding.

You bring: Nothing — complete trading stack provided by Datafye.

Learn more: See Choose Your Path for detailed scenario descriptions.

CLI Configuration

The CLI stores configuration in ~/.datafye/config.yaml:

# Default deployment model
default_deployment: standalone

# Cloud provider settings for distributed deployments
cloud:
  provider: aws
  account_id: "123456789012"
  region: us-east-1

# Datafye account credentials
credentials:
  api_key: "your-api-key"

# Default descriptor paths
descriptors:
  data: ./descriptors/data.yaml
  algo: ./descriptors/algo.yaml
  broker: ./descriptors/broker.yaml

This configuration provides defaults so you don't have to specify common options on every command.

Environment Variables

The CLI supports environment variable substitution in descriptors, keeping secrets out of version control:

In your descriptor:

broker:
  provider: "alpaca"
  account:
    api_key: "${ALPACA_API_KEY}"
    api_secret: "${ALPACA_API_SECRET}"

When you run the CLI:

export ALPACA_API_KEY="your-key"
export ALPACA_API_SECRET="your-secret"
datafye trading provision --descriptor trading.yaml

The CLI reads environment variables and substitutes values before provisioning.

Validation and Error Handling

Before provisioning, the CLI validates your descriptors to catch errors early:

Validation Checks

Schema validation:

• All required fields present

• Correct data types (strings, numbers, arrays)

• Values within valid ranges

Semantic validation:

• Referenced resources exist (container images, data symbols)

• Combinations are valid (e.g., can't request live trading in Standalone deployment)

• Resource requests are feasible (enough memory, valid CPU counts)

Dependency validation:

• Data providers support requested asset classes

• Broker supports requested permissions

• Container images are accessible from the deployment environment

Error Messages

When validation fails, the CLI provides clear, actionable error messages:

Error: Invalid descriptor
  File: algo.yaml
  Field: resources.memory
  Problem: Value "32GB" exceeds maximum allowed "16Gi" for Standalone deployment
  Fix: Reduce memory request or use Distributed deployment model

This prevents provisioning failures and saves time troubleshooting.

Status and Monitoring

Check the status of your environments:

# List all local environments (shows the single deployment if it exists)
datafye foundry local list
datafye trading local list

# List all AWS environments (shows all named deployments in the account)
datafye foundry aws list
datafye trading aws list

# Get detailed status
datafye foundry local status
datafye foundry aws status --name my-dev-foundry

# View logs
datafye foundry local logs
datafye foundry aws logs --name my-dev-foundry

Status output shows:

• Environment ID and name

• Deployment model (Standalone, Distributed Self-managed, Distributed Datafye Managed)

• Current state (provisioning, running, stopped, failed)

• Services running (Data Cloud, Algo Container, etc.)

• Resource usage (CPU, memory, network)

• Health check status

Common Workflows

Development Workflow

1. Create minimal descriptors for fast iteration

2. Provision Standalone deployment on your laptop

3. Develop and test your algo locally

4. Stop environment when not in use to save resources

# Morning: start work
datafye foundry local provision --descriptor dev.yaml
# ... develop and test ...

# Evening: stop to save resources
datafye foundry local stop

# Next morning: resume work
datafye foundry local start

Testing Workflow

1. Create full descriptors with production data

2. Provision Distributed Self-managed for cloud resources

3. Run comprehensive backtests with parallelization

4. Validate results and optimize parameters

5. Deprovision when testing is complete

datafye foundry aws provision \
  --name backtest-env \
  --descriptor backtest.yaml \
  --profile default \
  --region us-east-1 \
  --cidr 10.0.0.0/16

# ... run backtests and optimization ...

datafye foundry aws deprovision --name backtest-env  # Clean up resources

Production Workflow

1. Create production descriptors with live data and broker

2. For paper trading: Provision Standalone or Distributed Self-managed to validate

3. For live trading: Contact Datafye Ops to provision Distributed Datafye Managed deployment

4. Datafye Ops handles all operations and monitoring for managed environments

# Start with paper trading in self-managed environment
datafye trading aws provision \
  --name production-paper \
  --descriptor production-paper.yaml \
  --profile production \
  --region us-east-1 \
  --cidr 10.2.0.0/16

# ... validate performance with paper trading ...

# For live trading: Contact Datafye Ops with your production-live.yaml descriptor
# Datafye Ops will provision a Distributed Datafye Managed environment

Prerequisites and Installation

The Datafye CLI requires Java 17+ and Docker (for local deployments). For AWS deployments, you'll also need the AWS CLI v2 configured with your credentials.

Complete installation guide: See CLI Installation for detailed prerequisites, installation instructions, and troubleshooting.

Command Reference

This page provides a conceptual overview of the CLI. For detailed command syntax, parameters, and examples, see the CLI Command Reference.

Next Steps

• Install the CLI — Follow CLI Installation for step-by-step installation instructions

• Understand deployment models — Read Private Cloud Deployments to understand Standalone, Distributed Self-managed, and Distributed Datafye Managed deployments

• Learn about descriptors — Review Descriptors to understand how to configure your environments

• Get started with a scenario — Choose your path in Choose Your Path

• See command details — Browse CLI Command Reference for specific command syntax

---

Last updated: 2025-10-22